the code4lib journal – extra editorial: on the release of patron data in issue 58 of code4lib journal mission editorial committee process and structure code4lib issue 58, 2023-12-04 extra editorial: on the release of patron data in issue 58 of code4lib journal we, the editors of the code4lib journal, sincerely apologize for the recent incident in which personally identifiable information (pii) was released through the publication of an article in issue 58. the article “bringing it all together: data from everywhere to build dashboards” linked to two microsoft power bi files containing circulation data. timeline this is a summary of events: on monday dec 4, 2023, 11:28 pm est, the article was published as part of issue 58 of the code4lib journal. on tuesday dec 5, 2023, 02:02 pm est, a concerned reader emailed code4lib journal to flag the inclusion of the files using the email address that appears on the journal’s website. unfortunately, unknown to the current editors, that email account was not operational and, hence, no action was taken. on tuesday dec 5, 2023, 02:40 pm est, the author of the paper contacted its editor about the files; the editor removed the links to the files from the article at 02:49 pm est. on tuesday dec 5, 2023, 03:09 pm est, the author of the paper asked its editor to remove the files from the server; the editor – who did not have email access at that time – removed them at 04:15 pm est. on friday dec 8, 2023, around 11:40 am est, another editor informed the editorial board about an open letter about the incident, which was announced on mastodon. at the time of writing, the letter has 161 signatories. statistics during the time the files were online, they were accessed from 7 different ip addresses, with several accesses coming from the same ip addresses: almabibliographic holdings data.pbix 2023-12-04: 4 successful accesses (gets with return code 200, and no failed access attempts) 2023-12-05: 6 successful accesses (gets with return code 200), 3 failed attempts (gets with return code 404) almacirculation statistics.pbix 2023-12-04: 5 successful accesses (gets with return code 200, and no failed access attempts) 2023-12-05: 19 successful accesses (18 gets, 1 head, with return code 200), 3 failed attempts (2 heads, 1 get, with return code 404) the files were found not to have been cached by google, bing, yandex, yahoo, ecosia, duckduckgo, and internet archive. context the released files were in a proprietary file format, microsoft power bi, with which none of the editors have experience. since this article did not describe the actual content of the files, there was no immediate reason to believe they would contain pii. this was an erroneous assumption that the code4lib editors take full responsibility for. the current editors were also unaware that the email account listed on the code4lib journal website was not operational, slowing the notification of the editorial board, and causing the files to remain online for a longer period of time. this is another error that the editors take full responsibility for. the editorial board has since re-established access to this email address. going forward we are determined to take greater measures to prevent similar incidents from occurring in future journal issues by improving the editorial process. code4lib operates without a budget, and with volunteer editors. this means that fully addressing the feedback we have received in a responsible and sustainable way will take time. effective immediately and until we can establish policies and procedures that better safeguard personal information, code4lib journal will not accept or publish papers that utilize individuals’ personal data. we will describe this change on the journal’s website and in the call for papers. we invite colleagues who are knowledgeable in establishing relevant policies and procedures to support the code4lib journal by using their expertise to recommend sustainable guidelines that are informed by existing best practice, either independently or in the form of a journal article. we are grateful to all of those who worked to raise this important issue and look forward to collaborating with the community on best practices going forward. monday february 5, 2024. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – editorial: just enough of a shared vision mission editorial committee process and structure code4lib issue 43, 2019-02-14 editorial: just enough of a shared vision what makes a vibrant community? a shared vision! when we live into a shared vision, we can accomplish big goals even when our motivations are not completely aligned. by peter murray, issue 43 coordinating editor i love my job, and i love this profession. that is, i get excited about my job as the open source community advocate at index data and am an eager participant in the library technology profession because we freely share our knowledge and experiences. i will take as a given that i have both a myopic view and rose-colored glasses. [1] (myopic because there may be other professions that have similar inclinations to share expertise and experiences; if so, i’d love to compare notes! rose-colored…well, you can be the judge.) let me explain. in my day job working on the folio project, i alternate between astonishment at how well the community works towards the same goal and unease at not knowing how-and-why it works the way it does. (or worse: that one misstep could bring the project’s community crashing down.) this isn’t to say that the project doesn’t have rough edges; there have been disagreements over priorities, technical approaches, and other concerns. instead, the community seems to get stronger as it works through those disagreements and adds more partners. the same can be said about the code4lib community. sixteen years ago some colleagues got together to shared their expertise and experiences – first on a mailing list, then an irc channel. out of that sprang national meetings (america and japan that i know of), numerous regional meetings, this journal, and a slack team. (did i miss anything?) we’ve been through our struggles — figuring out if we’ve grown into needing a fiscal agent and who that fiscal agent would be, for example. people have left the community, and new people have joined. why does this work? can preconditions be set to strengthen the chance that a community will succeed? much has been written and talked about best practices for healthy communities, and i want to add to that body of thought. so here is my take — let’s call it the “just enough of a shared vision” theory. the just-enough-of-a-shared-vision theory i think there is a crucial need for a common understanding of what a community is about. this common understanding needs to be ingrained so deeply in the community that the participants are guided by this shared vision when they are not consciously thinking about it. and further, that there is a close alignment of one’s personal goals, one’s organizations’ goals, and the community goals. or, put another way, all three (the person, the organization, and the community) are getting benefits for the effort. what might this look like? one year’s conference organizers generously share their knowledge with the next year’s organizers. a beginner’s question on a mailing list is answered by half a dozen people with personal stories of hard-won wisdom. and yes, a new author is inspired by previous writings in the code4lib journal and wants to share their own experiences. (or become a volunteer editor for the journal!) viewed this way, we might be able to work out how to create a shared vision for a vibrant community. i think it comes in three parts. openness to sharing the vision. this openness takes the form of community members being willing to live into the community’s vision and an innate acceptance of those who say they are living into the same vision. openness to being wrong. while sharing the vision, the community members know that the community is not perfect; that there are misunderstandings, blind spots, and inadequacies. openness to new ideas. the building of the community is never done; it is a journey of experiments in doing things better and learning from each other. the community’s vision is attractive to people and organizations. people grow in experience and personal connections. library patrons are better off through services improved by that experience and personal connections. organizations take from the community in proportion to what they give to the community. and the community moves forward. in a community consisting of libraries and non-profits, i think much of this comes naturally. when commercial ventures are added to the mix, community members can wonder about motivations. is the company going to put into the community in proportion to the benefit they receive? it is here that we lean on the “just enough” part of the theory. the goals of the community and of the organizations involved do not need to align perfectly, and they probably never will. but there needs to be close enough alignment and openness in communication so the rest of the community members understand the alignment. this means that decisions are made in such a way that the goals of a participating organization are met and the goals of the community are met. that when decisions are out of balance between the organization and the community, the community has an instinctive reaction to guide the process back to balance. and i get that while this is easy to say, it is hard in practice and in specific circumstances. if the shared vision is strong enough and inclusive enough, though – wow, that is a place i’d love to be. introduction to issue 43 issue 43 has seven articles. developing weeding protocols for born digital collections authored by athina livanos-propst and edited by junior tidal: when you have 100,000 resources how do you construct a sensible way to evaluate the quality of your collection? this article describes one way to approach the challenge. content dissemination from small-scale museum and archival collections authored by avgoustinos avgousti, georgios papaioannou and feliz ribeiro gouveia, and edited by sara amato: extending the description of specialized collections – historic coins, in this case. never best practices: born-digital audiovisual preservation authored by julia kim, rebecca fraimow and erica titkemeyer, and edited by rebecca hirsch: three case studies of how three libraries with different needs and goals approach digital preservation. scope: a digital archives access interface authored by kelly stewart and stefana breitwieser, and edited by eric hanson: this collaboration extends the description of disparate digital objects. making the move to open journal systems 3 authored by mariya maistrovskaya and kaitlin newson, and edited by ron peterson: the university of toronto libraries had a tall task to update their hosted ojs installations, and in this article they describe how they accomplished it. improving the discoverability and web impact of open repositories authored by george macgregor, and edited by péter király: the university of strathclyde at glasgow test changes to their eprints site to improve search engine optimization and offer suggestions for any site looking to improve repository visibility. a systematic approach to collecting student work authored by janina mueller, and edited by ron peterson: the harvard university graduate school of design describes the technical and social issues behind their efforts to archive student works out of their learning management system. inspired by something you see here? please consider submitting an idea for an article to the code4lib journal. thank you, carol in the course of putting together this issue, code4lib journal editor carol bean finished retiring from the journal. (andrew said goodbye in his editorial in the last issue, but it has taken this long to complete the process!) carol volunteered as an editor for the journal’s first issue, and over the course of the next 42 issues she eagerly offered her insights and keen copyeditor eye. over the last few months she has transferred her knowledge and her responsibilities to other members of the editorial committee, and with a full-throated voice we say “thank you, carol!” endnotes [1] “rose-colored glasses” — an optimistic perception of something; a positive opinion; seeing something in a positive way, often thinking of it as better than it actually is. (see wiktionary definition.) i’m also grateful for colleagues like filip that make me think of broader geographic and cultural communities, and so i now think about including explanations of idioms when i use them. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – column: 700 dollars and a dream : take a chance on koha, there’s very little to lose mission editorial committee process and structure code4lib issue 1, 2007-12-17 column: 700 dollars and a dream : take a chance on koha, there’s very little to lose i truly believe that the meekest amongst us has a special duty and a special circumstance that fosters innovation. ours is not the culture of red tape entrenched tradition, but rather the atmosphere of the pioneer. no one will notice a failed experiment in the middle of nowhere, but they’ll certainly notice a cataloguer someplace in edema making a dent in backwards standards. by bws johnson one of the sentiments that i try to infect the library science populace with is the notion that innovation can come from anywhere at all. this is especially true of rural libraries. i truly believe that the meekest amongst us has a special duty and a special circumstance that fosters innovation. ours is not the culture of red tape entrenched tradition, but rather the atmosphere of the pioneer. no one will notice a failed experiment in the middle of nowhere, but they’ll certainly notice a cataloguer someplace in edema making a dent in backwards standards. so much of this field is not about the money—technology is definitely in that basket, particularly with open source making a fierce showing of things. this was the argument i took to my board: let me take a small portion of our state aid to public libraries money and try out this new fangled thing. the software’s free. yes really, we don’t pay anything for it, i just go out and grab it. nope, it won’t be more than $1,000 so we’ll still have emergency money. if it doesn’t work out, i can just reformat the drive, make it a regular old public access terminal, and we’re good to go. nope, it’s not stealing. everyone’s got something to give back to the community, and when i set things up and figure out what does what, i’ll write about it, and that’ll be our share. it’s like a barn raising. we had nothing to lose—our library wasn’t automated yet. ours was the perfect test environment. my board was receptive, my patrons weren’t attached to any system whatsoever, and my staff were behind the move. my loving husband was willing to install this for us, as well as physically assemble a custom server, although that’s not necessary—koha will run on just about everything. as with any other product, the more you put into your hardware, the better the result until you top out at a certain point. on koha, that point is frighteningly low, making my $700 box a ferrari testarossa. with the proliferation of linux user groups out there, it oughtn’t be too hard for just about anyone to approach the geeks that be and walk away with a functional server in a matter of a few hours. we’re mostly done with bibliographic input now—we’ve got just over 7,000 items catalogued of about 8,500. our patrons are in the database. we could circulate now if we wanted to. we’ve tested the basic features that we need and they work well enough for our purposes. when we started out, we were looking for the barest minimum of functionality. we got a whole lot more than we bargained for. koha is far more reliable than many commercial ils options. this was certainly a factor with me. it seemed as though things would be down every other month for a few days of unscheduled time with a few of the commercial products i’ve had the displeasure of experiencing. our server has been down twice in about 3 years of testing, with the box running 24/7. once was when my roommate inadvertently unplugged the server to charge his mobile phone. the second time was a catastrophic hardware failure. the power supply essentially caught fire. i was terribly worried my data was toast. it wasn’t. i had backups, but i didn’t need to use them. koha is far better at keyword searching than anything i’ve ever seen. something in the way it ranks search results really ends up giving you highly relevant items first. it also loots and pillages its way through a marc record so that those notes fields everyone tires over are searched through, too. the support is astounding. i have yet to pay money for support, yet i’ve had developers bend over backwards to program in a feature i’ve wanted, in a remarkably short span of time. a basic reports module came to me free of charge inside of a couple of days from across the ocean in france. an irc channel dedicated to koha tends to have someone on it most of the time. with heavily involved developers in the united states, france and new zealand the project doesn’t sleep. i can’t imagine the results you would see if you had a few thousand dollars to give to a developer for your feature. at a recent demo, i was eating lunch and chatting with the other librarians about which developer was responsible for what feature. one of the other librarians stopped me and said of their product, “wow, this is so great. you know the names of the developers. we’re lucky to even get through to support on the telephone!” with koha you get something you don’t get with any other product. you have compleat control over what your catalogue looks like, you don’t have to wrestle with a vendor to get your data to do what you want it to do, and if the product doesn’t have a feature you need, you can programme it or pay someone to add it. the rate of development and improvement over the past few years has been nothing short of astounding. when i started using koha, it was very wooden and very ugly. it’s come a long way since then. the current out of the box release is on par with at least a handful of commercial products. when the templates are customised for a given library, the product can meld seamlessly and aesthetically with a library’s website. the horowhenua library trust catalogue can give you a taste of the aesthetics: (http://www.library.org.nz/cgi-bin/koha/opac-main.pl) the upcoming version 3 looks quite like the athens county catalogue: (http://search.athenscounty.lib.oh.us/) since koha was developed in new zealand, connectivity issues caused the developers to make a product that would be very easy to access regardless of the speed of a person’s connection. i was able to access the catalogue which resided at home in albany, new york from my library in western massachusetts with no noticeable wait time for searching and data input over an incredibly crummy connection. (it was allegedly a 56k connection, but the plain old dial up telephone line connections routinely ran faster.) it’s not for everyone, however. installation is still difficult. unless you’ve someone in the area who is very comfortable with linux administration, this project will be a difficult set up. on the other hand, one can pay for a preinstalled box. cataloguing for a large institution would be tough. holdings information is a bit bodged at the moment. the cataloguing module is certainly clunky to use. the interface is tabbed with each marc field getting its own text box. as a result, either a librarian ends up sticking all of the fields in one tab for a really long screen of many, many boxes, or fields are missed by sloppy cataloguers that don’t switch tabs. it is possible to set up frameworks that anticipate necessary fields for a given material type, but this entails a good deal of planning during setup. the good news in this department is that thanks to google summer of code, a powerful new tool is being worked on to make things much nicer for cataloguers everywhere, and functionality should be vastly improved with version 3. reports are also getting a massive workover thanks to sponsorship from the british national health service. these can be tricky from a programmer’s perspective thanks to each client wanting a different data set. the new module will guide a user through the process of selecting which sets they’d like in order to produce the table or chart they’d like to pull from the raw data. because koha came from the mind of a computer programmer, there are creature comforts that librarians take for granted that could be absent or less fleshed out than one might like. increasingly, this is less true as the developers address new feature requests and the project gathers fans, and thus steam, along the way. the positive side of this is that it rapidly assimilates neat new web 2 innovations, for instance tag clouds are going to be featured in the new opac. like everything out there, there are bugs. developers do work to keep this down to a minimum, but i don’t want anyone to think i promised perfection. users are encouraged, and yea, even thanked when they submit problems to the project’s bug tracker, bugzilla (http://bugs.koha.org/cgi-bin/bugzilla/index.cgi). it’s far from perfect, but i can’t name a commercial product that has it all. ask yourself—what does my library have to lose? why not run an open source catalogue redundantly to your current system to discover the differences for yourself? if you do like koha, imagine how much you do have to lose in terms of that nasty annual license fee. you can choose to either have the product supported at an affordable rate or you can just set everything up yourself and never pay a thing except for the cost of the hardware. the model that koha is based on is very similar to national public radio or the corporation for public broadcasting. open source is out there waiting to be enjoyed by everyone, regardless of financial status. just as local programming is developed in your backyard and contributed back to the national efforts, individual libraries can customise their installation. when some flavours are contributed back, like the nelsonville templates, they prove to be very popular and are widely accepted in turn, like fresh air or nova. not everyone supports their local affiliate in a fund drive, and not everyone can afford to financially support the koha project. when libraries choose to pay for support or new features, everyone benefits since good reliable features can be selected out and then rolled into the product. even small contributions of time and labour end up making large differences in making the product better through collective effort. there is further information and a demo on the koha web site: http://www.koha.org and nicole engard has a blog entry about koha: http://www.web2learning.net/archives/1165 about the author: bws johnson is a graduate of the graduate school of library and information science at the university of illinois urbana – champaign and was the director of the hinsdale public library in hinsdale, ma. she was recently honoured to serve as president of the western massachusetts regional library system. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – infomaki: an open source, lightweight usability testing tool mission editorial committee process and structure code4lib issue 8, 2009-11-23 infomaki: an open source, lightweight usability testing tool infomaki is an open source “lightweight” usability testing tool developed by the new york public library to evaluate new designs for the nypl.org web site and uncover insights about our patrons. designed from the ground up to be as respectful of the respondents’ time as possible, it presents respondents with a single question at a time from a pool of active questions. in just over seven months of use, it has fielded over 100,000 responses from over 10,000 respondents. by michael lascarides introduction in november 2008, in anticipation of an upcoming home page redesign, the new york public library’s digital experience group ran a traditional online survey using the popular web-based tool surveymonkey.com, which we linked to from a one-line text banner at the top of our nypl.org homepage. it was a regular, please-answer-these-questions pitch comprised of 19 questions about web usage habits spread across 8 pages. over 14 days, that survey received 7,341 individual answers to questions from 520 respondents, just 60% of whom completed the whole survey. about the same time, we discovered the five second test (fivesecondtest.com), a web service built by an australian design firm based on an idea proposed by usability expert jared spool. the five second test (as the name implies) involves showing a visual design to a user for five seconds and then asking them to recall specific features, or asking them which of two designs (each shown for two and a half seconds) they liked better. we even considered using the fivesecondtest.com service to evaluate new nypl.org designs, but it lacked an easy way to redirect users back to our site once they were finished. the contrast between the two tools got us wondering if there wasn’t a way to make surveys and usability testing more painless (and dare we say fun?) for our users in order to maximize the number of responses received. to be sure, the library has strategic questions that require a lot of setup and deep knowledge about the respondents. we have a lot of those questions, and we are asking them in all their properly-sampled, audience-segmented glory, often with the assistance of consultants and our strategy department. but during the day-to-day process of designing a web site, what is often needed is just a reassurance that our team is on the right track. we contacted the fivesecondtest.com developers, who graciously gave us their blessing to adapt and expand on their ideas, and set about coding a prototype. ruby on rails was chosen as the development platform due to its flexibility and rapid prototyping capabilities, its full open source codebase, and the fact that our team had already successfully built several rails-based sites. in february 2009, we launched our solution: infomaki, an open source web application that incorporates ideas from both the five second test and traditional surveys. in its first 48 hours of public use, infomaki collected over 6,900 responses from 840 respondents, almost exceeding the entire total from the two-week traditional survey. design and implementation infomaki is a “lightweight” usability testing tool developed to evaluate new designs for the nypl.org web site and uncover insights about our patrons. designed to act as a “one question” survey, it presents respondents with a single question randomly selected from a pool of active questions. initially, two types of questions were supported: multiple choice and “where would you click to…?” (attached to a screenshot or other image). recently, we have added five-second tests for comparing two designs and for testing  recall of a design’s features. response times for each answer are captured as well. infomaki was designed from the ground up to be as respectful of the respondent’s time as possible. to this end, all of the language used in the project is geared towards lowering the cognitive load on the respondent. the link from our main web site to the tool reads, “answer a single question and help us improve our web site!” the “sales pitch” makes it clear that even if you only answer one question, it will be welcomed. figure 1 infomaki public home page figure 2 infomaki sample public page responding to a question only takes one click, and the “thank you” page that follows immediately (but politely) asks the respondent to answer another. as such, we’re finding that even with the “one question” pitch, an astonishing 90% of respondents answered more than one question, and the average number of questions answered per respondent is almost 11. it seems to be the potato chip of surveys: no one can eat just one. this has made an appreciable difference in our approach to surveys: rapid feedback leads to rapid turnover. we’re mining the vast middle ground between putting a full survey in the field with full protocols and methodologies, and asking people in the office “does this look right to you?” infomaki is not intended to be a formal research tool; rather, its strength lies in lowering the turnaround time between formulating a question and getting a response to that question from the general public. to this end, care has been taken to make it as easy as possible for staff to add questions to the system. designers here have already gotten into the habit of adding questions on friday night and returning monday morning to several hundred responses on their latest designs from weekend visitors. internally, the application is optimized to store all results from varied types of questions in a single common database table, which makes it extremely easy to analyze response statistics and ensure that no respondent sees the same question more than once. response data is displayed in tables and histograms (for multiple choice-type questions) and heat maps (for “click on this”-style questions). heat maps can show up as individual clicks or a percentage grid overlay, and colors are adjustable for contrast with different designs.  we welcome those from outside the nypl who would like to analyze the collected data; feel free to contact us. figure 3 infomaki sample heat map results page figure 4 infomaki sample histogram results page results the “lightweight” level of engagement on the part of the user has led to stellar response rates. in its first seven months of intermittent use, infomaki has captured 111,823 responses to 231 design, language and demographic questions from 10,203 individual respondents. that’s an average of 484 responses per question posted and 10.96 questions answered per person. when the banner is posted on nypl.org, roughly 1% of visitors click through from the main site, and over 90% of respondents answer more than the “one question” we asked them for. surprisingly, given that it’s essentially just a survey tool, users have called infomaki “fun,” “like a video game,” and “addictive”. more than one person has reported wanting to “find the end” by answering all of the active questions. in fact, the first improvement implemented as a result of user feedback was a way to skip the thank you page and keep answering questions without interruption. by testing designs with infomaki and in-person usability tests in tandem, we have been able to uncover a number of insights and potential pitfalls. ambiguities in navigation language were especially plentiful; for example, shortening the link to our fundraising page from “support the library” to “support” led to confusion with “technical support” (we reverted to “support the library”), and using the label “community” for a page with links to social networking tools had the unforeseen effect of siphoning clicks away from users seeking information on their local branch library (as much as 40% in one test; we changed the link to “interact with us”). more broadly, the iterative testing process has made abundantly clear the degree to which changes in a single element have an effect on other parts of the page. on one recent web page design, the main navigation was working acceptably, and when we added a search bar, response times went up precipitously. analysis showed that the search bar components looked too much like the other navigation links, increasing the (apparent) number of choices that users were required to cognitively parse. adding a background tint to the navigation and other design cues to create a distinction with the navigation returned the response times to acceptable levels. figure 5 navigation design changes figure 6 example of recall test drawbacks and criticism we have identified a few problems with the infomaki approach. first, by linking to the survey from the main web site, we don’t get a rounded profile of all library users. it’s safe to assume that infomaki respondents are among our more web-savvy patrons. but as long as we’re aware of that limitation, we’ve determined that it’s not a detriment to have a bias towards web users since most questions posted directly relate to the web site. a more pressing issue is the identification of particular “user segments” or “personas,” groups of users who are deemed to have the same general behavior patterns (such as researchers, recent immigrants, and so on). when we recently announced a new round of tests on twitter, usability expert craig tomlin tweeted back, “ouch, they don’t know the persona of the tester!” we had not deemed segmentation as critical for this particular test, since we were a) mainly testing global navigation which needed to apply to everyone, and b) testing so iteratively that problem areas would become apparent even though we might not know who was having the problem. but tomlin’s comment spurred us to add new tools to mark the referral source of the respondent. we also plan to add cross-segmenting based on the subset of users who answered a demographic question during the same session (for example, show only clicks from respondents over 50 years old). there are definite order biases that can creep in by presenting questions randomly. sometimes, when asking for user feedback in a text field, we will find users responding with the exact language that was used in a previous question. this may be mitigated in the future by presenting questions in a preferred order rather than a truly random one (for example, the “five-second recall” test works best when the user hasn’t already seen the same design in a different kind of question). however, we feel strongly that frequent, high-volume iterations of testing, combined with smaller volumes of more formal, segmented testing should give us a well-rounded view of potential problem areas with web designs. open source release the infomaki source code was released to the community under the gnu general public license on may 5, 2009. the current release is a “throw it over the side and see if it swims” release. to get it running, one needs to be familiar with the ruby on rails programming framework. it has spotty-to-nonexistent test coverage, a bit of vestigial code, and possibly some dependencies on a rubygem or two that we forgot to package. some non-user-friendly features remain in the administrative interface, such as the fact that it’s possible to delete screenshots that are in use, causing errors. the nypl plans to update the code to include more user-friendly administrative features by the end of the year. for more information on deploying infomaki from the current codebase including step-by-step “quick start” instructions, see this blog post: http://labs.nypl.org/2009/05/06/infomaki-goes-open-source/ since the open source release, we have been alerted to a couple of similar projects (see usabilla and chalkmark in the “references” section), the developers of which have been gracious in sharing ideas with us. to the best of our knowledge, infomaki is the only full open source tool in its class. roadmap we’ve added a number of generic demographic questions to the mix (how old are you, where do you live, etc.) and the hope is that in future versions, we will be able to segment responses to one question based on the answers to another question. for example, we can test familiarity with certain terms in one question and segment out those responses by age (for any respondents who answered both questions). behind the scenes, there are definitely some improvements that need to be made. it’s becoming clear that a frequent pattern of use is to test the same question (”where would you click to…?”) over screen-shots of several variant designs. right now, one must enter the same question repeatedly to get these comparisons. a future redesign of the administrative interface may allow us to build a suite of questions and simply upload a new screenshot to that suite to launch a new battery of tests and compare it to previous versions. we have also been thinking about ways to score accuracy, perhaps by adding values to particular click locations. since the tool is already capturing response times, a scatterplot chart with time on one axis and accuracy on the other would be a compelling illustration of which designs are performing the best. ideally, we’d like to work out a way that this tool can be “baked in” to the new nypl.org redesign so that user feedback becomes an ongoing, always-on process. we are considering ways of displaying the feedback banner based on context, such as only displaying the banner to a small, random percentage of visitors, or only to those visiting certain pages or searching for certain terms. as of this writing, we already have prototype versions of some of these features running locally and will be folding them into the public source code within a few weeks. we encourage everyone to download the infomaki source and let us know how your experience goes! references infomaki project page at sourceforge http://sourceforge.net/projects/infomaki/ infomaki on twitter http://twitter.com/infomaki infomaki’s launch announcement http://labs.nypl.org/2009/02/16/introducing-infomaki-bite-sized-usability-testing/ infomaki’s open source release announcement http://labs.nypl.org/2009/05/06/infomaki-goes-open-source/ jared spool’s original post on the five second test concept http://www.uie.com/articles/five_second_test/ five second test web service, from the australian firm angry monkeys http://fivesecondtest.com/ usabilla, a similar web service (free beta) http://usabilla.com/ chalkmark, another similar web service (fee-based) http://www.optimalworkshop.com/chalkmark_alt.htm the design firm volkside was one of the first users of infomaki after its public release http://www.volkside.com/2009/07/usability-test-with-infomaki/ about the author michael lascarides is the user analyst for the digital experience group of the new york public library and a member of the mfa computer art faculty at the school of visual arts. subscribe to comments: for this article | for all articles 2 responses to "infomaki: an open source, lightweight usability testing tool" please leave a response below: dennis van der heijden, 2010-10-29 besides ny library is someone else using this? like to see some working example the ny lib is not working anymore jesus tramullas, 2011-01-16 a reference to this work (in spanish), http://tramullas.com/2010/11/02/analisis-de-usabilidad-con-infomaki/ leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – developing an online platform for gamified library instruction mission editorial committee process and structure code4lib issue 35, 2017-01-30 developing an online platform for gamified library instruction gamification is a concept that has been catching fire for a while now in education, particularly in libraries. this article describes a pilot effort to create an online gamified platform for use in the woodbury university library’s information literacy course. the objectives of this project were both to increase student engagement and learning, and to serve as an opportunity for myself to further develop my web development skills. the platform was developed using the codeigniter web framework and consisted of several homework exercises ranging from a top-down two-dimensional library exploration game to a tutorial on cleaning up machine-generated apa citations. this article details the project’s planning and development process, the gamification concepts that helped guide the conceptualization of each exercise, reflections on the platform’s implementation in four course sections, and aspirations for the future of the project. it is hoped that this article will serve as an example of the opportunities–and challenges–that await both librarians and instructors who wish to add coding to their existing skill set. by jared cowing introduction undergraduate students at woodbury university are required to take a 1-unit information literacy class, information theory and practice, which is taught by librarians. each librarian is allowed flexibility in customizing their course sections to suit their teaching style and to try new ideas. a librarian wishing to make such customizations might find themselves frustrated with functional limitations presented by moodle, woodbury’s course management system. those possible frustrations include the rigidity in how online activities may be structured: instructors can create a multiple-choice quiz or prompts for the student to write an answer to a question, but it is not possible to create deeper interaction through the recall and manipulation of previous answers or through more advanced logic to determine custom reactions to student responses. another frustration might be the visual experience of using moodle; assignment content competes with other visual information that lines all four sides of the browser window, increasing the cognitive load of users. this visual information ranges from layer-upon-layer of navigation bars to moodle announcements to footer disclaimers. while moodle provides us with deep functionality in many areas that is invaluable, the overall user experience of moodle could be described by few as ‘fun’ or ‘inspiring curiosity.’ upon encountering these frustrations, i began a project to develop a web platform that could host more customized and interactive class assignments. initial experiment and obstacles the goal of this project started out simple: to create an interface that could accommodate multiple-choice questions or written answers, and that allowed for more flexible responses to user input. just as importantly, the interface needed to be as visually clean as possible, containing the absolute minimum of noise necessary. the hope with this visual approach was that it might reduce cognitive load and help users focus more on the single question being presented to them at any given time. such an interface was relatively simple to create with basic html/css, javascript, and php. it was indeed a cleaner visual experience. however, it became quickly apparent that despite some visual benefits, the functionality was not unique enough to justify a departure from moodle. editing the questions required directly entering them in the php code, which was certainly no faster than editing questions through a graphic user interface in moodle. additionally, there was no authentication or database system in place to securely identify and store student answers. to justify any time spent pushing this idea further, i would need to stop and think more clearly about my goals and the technology needed to achieve them. as someone with novice coding skills, these initial efforts represented an effort to have some fun coding a basic interactive tool. to progress any further, the project’s more personal goals would need to take a back seat to the realistic needs of a web platform intended for a real classroom environment. figure 1. this might look cleaner than moodle, but is this quiz that much more engaging, or is it just reinventing the wheel? getting serious about having fun taking the time to stop and think about where this project was going revealed a clearer set of goals and requirements: a platform or framework was required that allowed for basic functions like user authentication and sessions, the storage/retrieval of information from a database, and usage of a templating system. this platform could not be so overly complex that i would need to spend my time learning how to do things ‘the drupal way,’ for example, and in so doing lose some ability to be flexible and improvise. from a pedagogical standpoint, i required the ability to create assignments that were interactive, engaging, personalized, and that could vary widely in structure from week-to-week. the answer to these technological needs, codeigniter, came through the suggestion of a colleague. the answer to the pedagogical needs came through the concept of gamification. codeigniter codeigniter is a php-based web framework that is lean, efficient, and highly extensible. it is built on mvc architecture (model-view-controller) and represented the ideal balance of functionality with the simplicity that allowed for rapid development. the documentation is up to date and extremely clear, and is written in a way that allows for the flexibility to either take advantage of codeigniter’s mvc structure or ignore it completely. for a beginner, it represents the ideal sandbox to work in with just the right amount of fool proofing to keep one out of trouble. gamification gamification has only recently gotten a name but has been a trend in education for some time. at its core, it is “using game-based mechanics, aesthetics, and game thinking to engage people, motivate action, promote learning, and solve problems” (kapp 2012). it does not need to be the literal use of games in the classroom; rather, it can be any effort that utilizes motivational mechanisms that also exist in games. done well, it should not come off as a gimmick or as an excuse to play games. rather, gamification in the classroom should be used with clear learning objectives in mind. to dig deeply into how or why it might work, it is necessary to delve into game theory and the psychology of motivation. many such theories attempt to break down the individual mechanisms present in most games (kapp 2012). other theories detail the various personality types that gravitate to different game mechanisms (bartle 1996). still more theories attempt to define the very concept of “fun” or the conditions necessary for a person to experience motivation (deci and ryan 2000). working result utilizing codeigniter and the principles of gamification in a new round of development, the result was a platform containing five separate homework assignments. to access each assignment, the student would click on a link in moodle that took them to a login screen specific to that assignment. each student’s login credentials were set manually in code at the onset of the course. after logging in, the student would get an introductory screen letting them know what they could expect. after that point, the nature of each assignment diverged considerably. the first two homework assignments were largely multiple choice and text-answer based, and utilized much of the pre-codeigniter code. questions were mostly stored and supplied through a php script that existed outside of codeigniter’s mvc structure and were dynamically inserted into the page through jquery animations. figure 2. the beginning of homework #2, which includes custom paths based on the student’s choice of major. the third homework was the first one conceptualized after codeigniter and gamification became integrated into the project. the objective of this assignment was to prepare students for a physical library tour that would be taking place the following class. the intent was to have students start the tour already curious about various shelving locations and materials in the library. to achieve that end, the homework assignment became a top-down game in which students had to move around a representation of the library’s floor plan to discover all our shelf locations–along with several easter egg locations. at the end of the game, students receive a prompt to go to the library in person, find some of these shelf locations, and write some observations in a moodle forum. the game was built mostly using javascript and css, and consists of several html tables used to simulate the floor plan of each library space. when a player presses an arrow key to move, their current location and desired movement direction are checked against an array of 1’s and 0’s that dictate whether a player can or can’t move in that direction. this array also helps determine whether an event will be triggered upon entering the destination ‘tile.’ curious readers can try the game here using ‘code4lib’ as both the username and password. the objective of the fourth assignment grew out of my own classroom metaphor that research is like assembling a team of superheroes, in that one must think about how each source complements the strengths and weaknesses of the others, and how each helps to accomplish the ‘mission’ at hand. this metaphor is to counter a common student temptation to find several similar sources that all provide the same information. to encourage students to practice this ‘team building’ mentality, the objective of this assignment was to have students assemble a team of ‘heroes’ (sources) to help answer a previously identified research question. the homework consists of a workspace in which they can fill in basic information on each hero to demonstrate what role it plays in their research. students can also see how far along they are using a progress bar. when they select a type of source to categorize their hero under, a corresponding insignia is revealed to show that they have ‘recruited’ a new source to their team. figure 3. the fourth homework assignment on this site, labeled here as #7 because assignments 4-6 were separate activities completed outside of this site. making this assignment work required much more constant communication with the mysql database of student responses, and so the mvc architecture of codeigniter was leveraged much more in order to access its native database functions. while there are advantages to the visual continuity provided when using jquery to dynamically insert new content on a page, properly taking advantage of codeigniter’s database driver tools required sending information to a controller php script and–in so doing–loading a new page using a coded url. receiving information from the php script and displaying it on the page also necessitated the reloading of pages. i took from this assignment a much better appreciation of the power available through more fully utilizing codeigniter’s core architecture. the final homework assignment on this site involved citations and apa formatting. few people would call this their favorite topic, and so the challenge in developing this assignment was to make it engaging and to prompt students to pay closer attention to how and why citations are used. the result was an assignment containing a mix of small exercises including a ‘seek the citation errors’ mini-game. it also would recall and display students’ answers from previous assignments to prompt them into thinking about what information they would need to look the source up again. reception, aspirations, and advice this platform was developed mostly through the winter of 2015/16 and has so far been used by myself while teaching four sections of information theory and practice–one in spring 2016, one in the summer, and two this fall. some takeaways became clear early on, while others have taken some time to think over. student reactions the most positive reaction seemed to come from the top-down game. students would begin the following physical library tour asking about parts of the library–such as our loft space–that they did not previously know existed. my speculation is that the positive reaction came in part because this assignment is the most game-like of them all. additionally, the fact that the game’s setting was in our own library may have made the experience more personal and relatable. the version of this platform that was used in the spring and summer did not contain the functionality to recall a student’s previous answers. if they restarted a homework assignment, any answers already submitted would not populate in text boxes (except for the “hero recruiting” assignment, which required the recall and display of stored answers to work properly). while this shortcoming was disclaimed in the assignment instructions, it understandably led several students to think that their assignments had been erased or never received for grading. this was a frustrating user experience resulting from my having omitted a critical feature. in the time between the summer and fall semesters, the functionality necessary to display and modify a student’s prior answers was added. aspirations for the project while the result of this project–a gamification of my homework assignments–was worthwhile, the next goal is to think about ways that the platform could be extended to further infuse those concepts into class lectures and in-class activities. those gamified in-class activities that have been used–such as an online drag and drop call number sorting exercise i developed–appear fairly successful in engaging students. another aspiration is to develop assessment measures to better gauge the platform’s effectiveness. it is difficult to gauge a student’s reactions while they complete a homework assignment beyond looking at their homework answers and their overall class performance. in addition, student responses in the standard university course evaluations were usually more general in nature and made few distinctions between the gamified class elements being tested and the core elements that were shared by all sections of information theory and practice. to better measure the success of these efforts, a new assessment method will be needed that prompts more specific feedback from students. researching the theories behind gamification has provided new lenses through which to analyze each class assignment and look for ways to engage different types of learners. delving deeper into the literature on gamification may reveal new ways that different personality types could be motivated when completing these assignments. these efforts will help to ensure that my own attempts at gamification are not limited by personal preferences for what features are the most engaging and instructive. one possible step might be to empower students who are more motivated by social or competitive game mechanisms; this could be done through features like allowing students in the top-down library exploration game to drop notes and items on the library floor for their other classmates to find. in teaching a graded course, more control is offered in that students become partly responsible for observing the technological requirements of each assignment. this represents a departure from the realities experienced by professional web developers, who must be prepared to serve users of any device, operating system or internet browser. this platform made heavy use of css3 and jquery animations that rendered older versions of internet explorer useless, and it came as a great surprise just how many students were still using these older browsers. looking forward, one final goal is to explore making this web platform more accessible for users of different browsers and devices, and also for users with visual disabilities. advice for others in addition to the other stated goals of this project, it also served as one possible example of how an instructor or librarian might–or might not–be able to use basic coding skills as a tool in their pedagogical toolkit. a project of this nature could be rewarding for both the instructor and students, but with major cautions. one caution is that for any experimental product created by a novice coder, students may find bugs or encounter the occasional user experience difficulty. expect some inquiries on weekends and in the late of the night, and be prepared to offer quick responses to address any problems right away. students may need to be offered the benefit of the doubt, and some flexibility, if they say that they ran into trouble. no student wants to be penalized for a technical issue. beginning coders should be encouraged to let their imaginations run wild and not be afraid to build things that are flawed in conception and wildly inefficient in execution. it is easy to allow the rigid and complex structures of professional coding practice to dim the light of inspiration that will drive a novice to learn. it is that messy sort of trial and error that can nurture a long-lasting love for coding, and it may produce the occasional innovation worth keeping. nevertheless, if the intent is to finish with a polished product used by others, one must be prepared to scrap much of what has been done in order to build again with a clear strategy and outcomes in mind. for learning purposes, there should be no shame in having to do something twice if unique insight was gained each time–provided that no deadlines were missed along the way. conclusion i plan to continue using this platform for the near future, with a stronger focus on assessment and a willingness to continually make strategic improvements. i’m especially interested in getting feedback from more seasoned developers and instructors alike about what they think could be improved or done differently. while this project was far from groundbreaking in the technical or pedagogical sense, the hope is that it represents a realistic example of what can be produced by a librarian or instructor who is interested in trying their hand at coding. references bartle r. 1996. hearts, clubs, diamonds, spades: players who suit muds. journal of mud research. deci el, ryan rm. 2000. self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. american psychologist. 55(1). kapp km. 2012. the gamification of learning and instruction: game-based methods and strategies for training and education. san francisco: pfeiffer. further reading de jong t. 2010. cognitive load theory, educational research, and instructional design: some food for thought. instructional science [internet]. [cited 2016 nov 18];38(2):105-134. available from: http://link.springer.com/article/10.1007/s11251-009-9110-0 ke f. 2009. a qualitative meta-analysis of computer games as learning tools. in: ferdig re, editor. effective electronic gaming in education. hershey (pa): information science reference. p. 1-32. madigan j. 2016. getting gamers: the psychology of video games and their impact on the people who play them. lanham (md): rowman & littlefield. sitzmann t. 2011. a meta-analytic examination of the instructional effectiveness of computer-based simulation games. personnel psychology. 64(2): 489-528. about the author jared cowing is the systems librarian and an assistant professor at woodbury university in burbank, ca. his professional interests include the gamification of libraries and the development of more interactive, intuitive library discovery interfaces using rich library metadata. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – developing an academic image collection with flickr mission editorial committee process and structure code4lib issue 3, 2008-06-23 developing an academic image collection with flickr a group at lewis & clark college in portland are in the process of developing an educational collection of contemporary ceramics images using the photo sharing site flickr as a back end. this article discusses the evolution of the project, flickr machine tags, and the concept of flickr as an application database layer. the article includes code samples for creating and querying machine tags using the flickr api. by jeremy mcwilliams introduction academic visual resources are in the midst of a shift from traditional slide libraries to reliance upon digital collections. rather than loading the slide tray for a class, instructors are turning to digital image collections like artstor, james madison university’s mdid, and collection software like insight, for teaching. such resources tend to have higher quality and better-described images than what one might get from a google image search. yet resources like mdid and artstor are closed data silos and can be difficult to work with due to proprietary presentation software and copyright restrictions on the images themselves. and despite typically lower quality images found via google image search, instructors often use those images because they’re easy to find and use (if not necessarily legal). in the summer of 2007, a group at lewis & clark college in portland, oregon, decided to create an alternative image resource collection for education. specifically, the goal was to develop a collection of contemporary ceramics images, as no such resource existed. but rather than gathering images in a closed platform like mdid or artstor, we wanted to develop a collection that had high quality images, was open to anyone, included a distributed model for adding and cataloging images, and was mobile/remixable in the spirit of web 2.0. it became clear that the photo sharing site flickr was an intriguing, if somewhat experimental, solution to achieve these goals. flickr already has a ‘group’ model, in which users can contribute images toward a shared collection. a flickr group can also be moderated, so a curatorial board of designated administrators can accept/reject images submitted to the group. flickr also allows users to assign a creative commons license to images they own, which permits the images to be used with fewer copyright restrictions. in addition, flickr’s impressive application programming interface (api) lets developers easily create web sites with flickr images and data. with these ideas in mind, we decided to take the plunge and attempt to build a contemporary ceramics image collection for education with flickr as the primary back-end. we figured it could end up as a failed r&d experiment, or it could provide a revolutionary model for academic image resource collections. our results thus far are at accessceramics.org. this article will discuss the site design and evolution of accessceramics, flickr machine tags, and the concept of flickr as the database layer for an academic image collection. accessceramics.org: initial design our working group consisted of ted vogel (assistant professor of art, program head in ceramics, department of art, lewis & clark college), margo ballantyne (visual resources curator), mark dahl (assistant director for systems and technical services), and myself, jeremy mcwilliams (digital services coordinator). we didn’t really have any extra time, resources, or additional staff to devote to the creation of the image collection, though our expertise in different areas helped to distribute the workload fairly evenly. ted and margo developed the metadata schema and worked directly with artists, while mark and i handled the technological aspects, including plenty of testing within flickr, and the development on the code base. we hoped to rely as much as possible on the existing flickr infrastructure for collection organization, metadata storage, and user authentication. the idea was to create a lightweight, mobile site that was little more than a thin technological layer on top of flickr. the initial site consisted of php, css, and the jquery javascript library, and handled all data storage within flickr via the api (figure 1). essentially, we wanted flickr to be the database. figure 1: accessceramics initial model [view full-size image] during our initial development, i created a test flickr group, and wrote php code to create an interface that interacted with flickr using its api. the site was designed to work with basic flickr api functions, including flickr authentication, viewing flickr image sets on our interface, adding tags to images, and submitting images to flickr groups. once development and test phases were completed, we invited local ceramicists to create free flickr accounts, upload images of their works, and join our flickr group. the artists then used our interface to catalog and submit their images to the collection. to do this, an artist would log in to flickr through our site, select an image from their personal flickr collection, and then add metadata to a cataloging form to describe the image (figure 2). upon submission, php scripts converted the metadata to tags, added them to the image on flickr, and placed it in the flickr group queue, where it awaited approval or denial by a group administrator. figure 2: accessceramics cataloging form[view full-size image] yet our code did more than just convert metadata to tags. in order to create useful metadata for images, the cataloged data was converted to machine tags, a relatively new type of tag structure that we hoped could make the ‘flickr as database’ concept a reality. flickr machine tags one of flickr’s most popular features is the ability for users to describe images with tags. this creates an environment for social photo sharing, as flickr users can easily view sets of images tagged with common keywords. but tags alone don’t provide the depth of metadata description that some image collections might require. users of such collections should be able to search and browse by different fields, and keyword tagging simply doesn’t provide that framework. recognizing that need, flickr launched machine tags in january of 2007. machine tags have the format namespace:field=value, enabling complex image descriptions. not only can machine tags allow field-value relationships, but similar relationships can be grouped together with a common namespace. geotagging is perhaps the most common use of machine tags, as a simple keyword tag of ‘45.12234’ won’t have the same meaning as geo:lat=45.12234. and while there was some initial discussion to regulate the use of machine tag namespaces, the selection of a machine tag namespace in practice is largely arbitrary. while flickr users can create and view machine tags in the flickr interface, they are intended primarily for use in the flickr api. since a user on flickr is not likely to add a tag like ‘image:color=red’, it makes more sense for code in an external application to take user inputted metadata and convert them to machine tag syntax. in the case of accessceramics, we developed a cataloging interface for artists that takes form values, converts them to machine tags, and uses the api to tag the targeted image on flickr. similarly, if a user wanted to browse images in which the glaze is electric oxidation, the application should convert the query to machine tag syntax, perform the query through the api, and return a formatted results set. in other words, the existence and use of machine tags should be invisible to the user, just as sql queries are in common lamp applications. flickr api machine tag code samples flickr’s api has over 100 methods for a variety of purposes, utilizes a rest-style format, and requires an api key. each method has a thorough demonstration page in which users can enter sample queries, and receive xml responses (here’s an example). developers have also created a number of api kits in a variety of languages to further simplify the api interaction process (we use dan coulter’s phpflickr). for the purpose of these examples, we’ll use language-independent rest url queries, with xml responses. we hope they will provide some insight on our attempts to store and retrieve image metadata in flickr. adding a machine tag to add a tag or machine tag to a photo in flickr, flickr.photos.addtags is the appropriate method. in this example, we’ll add a machine tag to an image for a fictional image collection of dogs. to designate a ‘breed’ field as ‘cocker spaniel’, the machine tag could read dogs:breed=’cocker spaniel’. below is the structure of the rest url to add the machine tag to the flickr image through the api: http://api.flickr.com/services/rest/?method=flickr.photos.addtags&api_key=xxx&photo_id=2411163173&tags=dogs:breed='cocker+spaniel'&auth_token=xxxx&api_sig=xxx this particular action requires an auth_token and api_sig to verify the write permission to add the tag (more information about flickr api authentication can be found on the authentication api page for flickr services). also, quotes are required around cocker spaniel, as it is a multi-word tag seperated by spaces. quotes aren’t required if the value is a single word. the response xml is: which just confirms the addition of the machine tag. the screenshot below shows the addition of the machine tag in flickr. notice that machine tags are grouped separately from normal tags. retrieve results via machine tag an effective method to perform a machine tag search is flickr.photos.search (for more information on machine tag query syntax visit the flickr.photos.search page). this method has an optional machine tag parameter, and can also be narrowed to a group. here is a query of the accessceramics flickr group for images in which the object type is a wall piece: http://api.flickr.com/services/rest/?method=flickr.photos.search&api_key=xxx&machine_tags=ã“machine_tagsã“+=>+ã“ceramics:object_type=wall_pieceã“&group_id=511711%40n24 the machine_tags value in the url is slightly different than the rest of the parameter / value pairs. also, notice the underscore between wall and piece. it is important to note that this is not machine tag query syntax, but rather a decision we made for accessceramics in handling multiple word machine tag values. this practice seemed to yield better success, and we used php to convert underscores to spaces when preparing the html view. the quotes around machine_tags and ceramics:object_type=wall_piece are part of the rest syntax for machine tags used in flickr.photos.search. this yields the response: <?xml version="1.0" encoding="utf-8" ?> <rsp stat="ok"> <photos page="1" pages="1" perpage="100" total="16"> <photo id="2319784593" owner="24450244@n03" secret="88ec9f4fe0" server="2199" farm="3" title="still life with yellow chick" ispublic="1" isfriend="0" isfamily="0" /> <photo id="2320601654" owner="24450244@n03" secret="0dcf641ce1" server="3012" farm="4" title="storm cloud monkey" ispublic="1" isfriend="0" isfamily="0" /> <photo id="2320594270" owner="24450244@n03" secret="84aa5ba8f4" server="3062" farm="4" title="seeds" ispublic="1" isfriend="0" isfamily="0" /> <photo id="2319787077" owner="24450244@n03" secret="f6df10ca19" server="3273" farm="4" title="turkey bird" ispublic="1" isfriend="0" isfamily="0" /> <photo id="2319775315" owner="24450244@n03" secret="8d5937fa27" server="2227" farm="3" title="ghost (ascending)" ispublic="1" isfriend="0" isfamily="0" /> <photo id="2319770783" owner="24450244@n03" secret="ff80d3c3cb" server="3230" farm="4" title="ghost (welcoming)" ispublic="1" isfriend="0" isfamily="0" /> <photo id="2304747377" owner="9379045@n05" secret="e175505f03" server="2005" farm="3" title="kylix on pedestal" ispublic="1" isfriend="0" isfamily="0" /> <photo id="2304496875" owner="9379045@n05" secret="26fdd69e4d" server="2361" farm="3" title="3 vessels" ispublic="1" isfriend="0" isfamily="0" /> <photo id="2305280568" owner="9379045@n05" secret="083726cfeb" server="3003" farm="4" title="untitled" ispublic="1" isfriend="0" isfamily="0" /> <photo id="2177056856" owner="64194819@n00" secret="e55793c0ce" server="2165" farm="3" title="cy and chocolate" ispublic="1" isfriend="0" isfamily="0" /> <photo id="2152075599" owner="64194819@n00" secret="9ec34040be" server="2254" farm="3" title="shortbus" ispublic="1" isfriend="0" isfamily="0" /> <photo id="1800274987" owner="64194819@n00" secret="89c46a7fb0" server="2176" farm="3" title="marley's halloween mural" ispublic="1" isfriend="0" isfamily="0" /> <photo id="1748821816" owner="15388497@n02" secret="bee45ef0ca" server="2103" farm="3" title="silver branch" ispublic="1" isfriend="0" isfamily="0" /> <photo id="1747965855" owner="15388497@n02" secret="b4f53d6853" server="2150" farm="3" title="black sky" ispublic="1" isfriend="0" isfamily="0" /> <photo id="1747972271" owner="15388497@n02" secret="caca4951e7" server="2017" farm="3" title="white branch" ispublic="1" isfriend="0" isfamily="0" /> <photo id="1748817406" owner="15388497@n02" secret="1db24ed5ab" server="2275" farm="3" title="stump roost target" ispublic="1" isfriend="0" isfamily="0" /> </photos> </rsp> when parsed, reformatted, and augmented with additional data using the flickr api method flickr.photos.getinfo, a results screen like this can be created (figure 3): figure 3: formatted results screen from machine tag query [view full-size image ] ‘flickr as database’ flickr has an excellent infrastructure for developing image collections, both on the site itself, and with external applications using the api. however, in some ways, it may not be quite ready as an exclusive database layer for an academic image resource. by default, anyone can add tags to images in flickr, and changing tagging permissions is not entirely a straightforward process. this is analogous to permitting anyone to have ‘write access’ to the database. perhaps in the context of sites like wikipedia, this isn’t necessarily a deal breaker. nonetheless, it’s difficult to ignore potential cases of sabotage. for example, if a user didn’t like a particular piece of art in our collection, he/she could find the image on flickr, and tag it with ceramics:title=’i_am_not_fond_of_this’. while machine tags have potentially expanded flickr as a resource, they still lack a couple important features. perhaps most important is the inability to perform truncated machine tag searches. for example, a search of ceramics:material=clay will return only exact matches; machine tags disallow wild card variables in the value portion of the tag. creating a search interface without this feature would likely create a frustrating experience for the user. also, as mentioned previously, there is currently no authority to regulate machine tag use. this is probably fine for now, but could become an issue if more developers use machine tags. the ‘flickr as database’ model also lacks a degree of centralized control to handle tedious details, like spelling variations on metadata tags. by exclusively using flickr to store metadata, artists would be required to make corrections to their own tags in order to adhere to bibliographic control. this isn’t very practical, as tasks like this should be performed centrally. in our case, we’re indebted to our artists for taking the time to upload and catalog their images; we don’t want to hassle them with nitpicky metadata problems. while the flickr api is quite possibly the web services standard by which other apis should be judged, it lacks certain methods that would be useful for our project. if we wanted to create a ‘browse by field’ screen, for example, there currently isn’t an api method to gather all possible machine tag values in an image collection for a given field. it would require an api call to retrieve a list of images in the collection, another api call per image to retrieve the machine tags, some code to select the desired tags via regular expressions and place them in an array, and finally some processing of that array prior to displaying on the interface. this sequence of events just isn’t practical. a comparable action to retrieve the same result set from a mysql database would require just a single query and some basic conversion of the results to html. because of these various issues, we ultimately decided to abandon using flickr as the exclusive database layer, and began storing metadata in a mysql database (figure 4). not only does this give us more control over the data, but it will make the development of site tools easier, as we won’t necessarily have to depend upon the existence of certain flickr api methods to add functionality. in our new model, artist-entered metadata is stored in the mysql database, and machine tags are generated by an accessceramics ‘super user’ flickr account. we’re still creating machine tags on flickr images, with the hope that functionality will improve in the near future, or that flickr will build a true collection feature to fit cases like ours. the shift to a mysql database will give the site administrator more control of the metadata, make the site run faster, and will catalyze the development of site tools and functionality. figure 4: accessceramics current model [view full-size image] future directions as of spring 2008, accessceramics has a little over one hundred images contributed by about a dozen artists. we’re hoping that volume might be a typical weekly yield in the very near future. we are also attempting to procure grant funding to accelerate the development of the project; up to this point, all work on the project has been squeezed in amongst our other various duties. with additional funding, we would hire a coordinator to help artists with the image submission process, and enlist technical expertise to better design and develop the site. aside from increasing the volume of the collection, we plan to develop tools to facilitate educational use of the images. our short term laundry list includes the creation of better searching and browsing capabilities, and facilitating the use of the images and metadata in slideshow and presentation software. we also hope ceramics educators will use the accessceramics collection in the flickr interface, as flickr has a track record of unveiling new tools for users, some of which may be useful in an educational setting. while flickr has ruffled some feathers by now supporting video, the development has added some intriguing possibilities for enhancing accessceramics and similar flickr-based collections. our site could eventually include videos showing comprehensive views of a given piece of art, tours of ceramics exhibits, interviews with contributing artists, and actual lectures or teaching tips that could be useful for education. and videos work seamlessly with the flickr api; this post by flickr’s kellan elliott-mccrea includes further description and code samples for embedding flickr videos within a web application. we hope other developers will continue to discover ways to use flickr for education and digital collections. while some in the academic community might view flickr as little more than a variation of myspace, perhaps the recently added library of congress collection will help change perceptions and encourage more experimentation and development with flickr. peter brantley of the digital library federation and mark dahl from our accessceramics group have discussed the notion of an ‘academic flickr’, theoretically provided as a sub-service of flickr itself. while this may or may not come to fruition, we should take advantage of what flickr has already offered: a free set of wonderful tools to help us redefine what a visual resources collection can be. acknowledgments: thanks to ted, margo, and mark for your hard work, to watzek library director jim kopp for letting us library people work on this, to the contributing artists, and to flickr for being flickr. note: accessceramics is not an open source project, though we would be happy to share our code. email jeremy (jeremy2443@gmail.com) if you’re interested. about the author jeremy mcwilliams is the digital services coordinator at lewis & clark college’s watzek library in portland, or. he has been at lewis & clark for 10 years, and enjoys creating public and staff-side library web applications. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – lantern: a pandoc template for oer publishing mission editorial committee process and structure code4lib issue 53, 2022-05-09 lantern: a pandoc template for oer publishing lantern is a template and workflow for using pandoc and github to create and host multi-format open educational resources (oer) online. it applies minimal computing methods to oer publishing practices. the purpose is to minimize the technical footprint for digital publishing while maximizing control over the form, content, and distribution of oer texts. lantern uses markdown and yaml to capture an oer’s source content and metadata and pandoc to transform it into html, pdf, epub, and docx formats. pandoc’s options and arguments are pre-configured in a bash script to simplify the process for users. lantern is available as a template repository on github. the template repository is set up to run pandoc with github actions and serve output files on github pages for convenience; however, github is not a required dependency. lantern can be used on any modern computer to produce oer files that can be uploaded to any modern web server. by chris diaz motivations open educational resources (oer) are free teaching and learning materials that are available online for unlimited use, consultation, adaptation, and distribution. typically, oer are distributed under a creative commons license [1]. while they can be downloaded and used for free, maintaining an oer support infrastructure is an expensive endeavor. for example, academic libraries provide services to faculty focused on oer awareness, adoption, and creation. these services require staffing, training, coordination, technology, and marketing. institutional oer grants and faculty stipends are a popular method for incentivizing and supporting the creation of new oer (santiago & ray 2020). however, in order for the public to reap the benefits, the oer needs to be published. libraries also support the publication of oer by making the oer content discoverable, accessible, preservable, and reusable. the supporting infrastructure that libraries provide for oer raises questions about sustainability. lantern was developed with these questions in mind. the sustainability of an oer depends on the oer’s ongoing ability to meet its educational purpose. for oer initiatives in libraries, there are two primary sustainability concerns to keep in mind: (1) the production and access of oer and (2) the use and reuse of oer by the public (wiley 2006). both of these parts require people, workflows, and technologies. by minimizing the costs of digital infrastructure and maintenance, libraries can increase investments in people and workflows for oer production and access. lantern was designed to reduce the technical complexity of technology stacks necessary for the production, sharing, use, and re-use of oer by meeting these sustainability criteria, adapted from wiley (2006): oer is created in a format that operates equally well across hardware and operating systems oer is available to the public in such a way that edits and adaptations can be made for teaching and learning in a variety of contexts these criteria for the sustainability of oer can be aligned with the principles underlying minimal computing, a framework developed by digital humanists for designing systems that only use the hardware and software resources that are necessary for the task (gil 2015). this thought exercise helped reduce the technology stack lantern uses to create, host, and archive oer. lantern’s stack is focused on structured plain text, static web technologies, version control, and open source software. lantern figure 1. lantern workflow overview. lantern’s workflow begins with common word processing software and ends with a multiformat oer publication. lantern provides a folder template and bash script for using pandoc to convert between manuscript (.docx or .odt), source (markdown, yaml, bibtex), and publication formats (html, pdf, epub) [2]. it is intended to make using pandoc, a comprehensive document conversion tool, easier to use for oer creators and publishers who are generally new to command line programs. lantern aims to teach fundamental digital skills in plain text editing, static web technologies, and open source software in order to encourage the use of minimal technology stacks in digital library projects. “minimal technology stacks” refers to the intentional constraints around the technology components required for a computing process. pandoc is a command line tool that converts an input file of one file format into an output file of another file format. both the input and output files need to be represented in a structured markup language. at the time of this writing, pandoc (version 2.17.1) can read from 39 input file formats and write to 62 output formats, with varying levels of accuracy. each conversion can take zero or dozens of options from 96 that are available. the lantern.sh bash script, files, and folders within the lantern template repository simplify the level of customization available to pandoc users for oer production use cases. for example, the bash function responsible for generating the pdf looks like this: pdf() { # combine all markdown files into one pandoc text/*.md -o _temp/chapters.md # convert markdown to html to pdf pandoc _temp/chapters.md \ --defaults settings/pdf.yml \ --output $output_directory/$output_filename.pdf echo "the pdf edition is now available." } the pdf function first calls pandoc to combine all of the markdown files within the /text/ subfolder to make up the body of the oer, ordering them by filename. the function then calls pandoc again to convert the concatenated markdown file into a pdf using the settings specified in a pandoc defaults file [3]. the defaults file specifies a selection of pandoc options, including the metadata, templates, and pdf settings we want to apply [4]. each output format has its own function within the script following the basic workflow but referencing different defaults files and options. ideally, the lantern template repository serves as an approachable foundation from which users can build their own customizations and features for their projects. structured plain text most people write and edit text using a rich text editor, like those found in microsoft word, google docs, wordpress, and email programs. rich text editors display the style elements of the document, but obscure the semantic elements of the document’s underlying structure. this leads people to use alignment and bold fonts to signal that a specific text element is a heading, which sighted people may understand visually, but machines (like screen-readers) might miss. to avoid this pitfall, lantern provides tips on tagging manuscript files in .docx format with proper headings and styles using word processing software, like microsoft word. document structure is essential for accessibility, formatting, and portability. lantern uses markdown and yaml as the structured plain text representations of an oer’s content and metadata. plain text offers numerous advantages for library-based oer publishers, as tenen and wythoff (2014) explain: plain text both ensures transparency and answers the standards of long-term preservation. ms word may go the way of wordperfect in the future, but plain text will always remain easy to read, catalog, mine, and transform. furthermore, plain text enables easy and powerful versioning of the document, which is useful in collaboration and organizing drafts… plain text is backwards compatible and future-proof. lantern also provides a file structure for an oer project. an example project folder contains a lantern.sh script file, a metadata.yml file, a subfolder (/text/) for markdown files, and several other subfolders containing the templates, styles, and configurations. this structure enforces a separation between content and style; lantern users only need to use the metadata.yml and /text/ subfolder. markdown provides many advantages for academic writing, oer production, and preservation. john gruber, one of its inventors, explains the philosophy of markdown this way (2004): markdown is intended to be as easy-to-read and easy-to-write as is feasible. readability, however, is emphasized above all else. a markdown-formatted document should be publishable as-is, as plain text, without looking like it’s been marked up with tags or formatting instructions. here’s an example of markdown syntax: # chapter title introductory paragraph text with **bold** and *italic* text. ## section heading introductory paragraph for the section. another paragraph, but with a [link to a website](https://example.com). ### subsection heading more content, but in list form: list item list item list item markdown is useful for digital publishers and preservationists because it is human-readable in its raw form and machine-readable for converting to html and dozens of other markup formats. lantern mostly follows pandoc’s markdown syntax for textual elements, with additional support for numbered references (for equations, figures, and tables), callout boxes, and exercise questions. lantern organizes the content of an oer project as one or more markdown files in a `/text/` subfolder. each file is named according to its numerical order within the larger project: 001-preface.md 010-introduction.md 020-theory.md lantern uses yaml as its primary metadata format. like markdown, yaml was selected for its readability and ability to be transformed into json and several other structured data formats. the metadata file contains bibliographic metadata fields represented in yaml syntax, for example: title: lantern subtitle: an oer publishing toolkit author: name: chris diaz affiliation: northwestern university name: lauren mckeen mcdonald affiliation: northwestern university keywords: textbooks oer digital publishing github for oer publishing perhaps one of the most powerful advantages for using plain text to organize and produce oer is the ability to use the git version control system and the github ecosystem of collaboration and automation tools. the management, collaboration, and preservation benefits of git and github for library technology projects are well documented (davis 2015, giorgio et al. 2019; becker et al. 2020). lantern demonstrates the benefits of git and github for oer projects. lantern is a template repository on github. it is intended to make it easy for anyone to generate their own oer projects using the same repository structure and files. in practice, a user would login to github, visit the lantern repository, and generate a new repository for their oer project. they would then add their own project’s content and metadata and use lantern’s preconfigured settings to produce their multi-format oer for free. lantern’s pre-configurations take advantage of github actions [5] and github pages [6]. github actions is a programmable workflow automation tool and github pages is a static website hosting service. these features are especially useful for oer publishing. lantern provides documentation to help users prepare manuscripts in common file formats (.docx) and github actions to convert them to markdown using pandoc. the basic workflow involves the following steps: user generates a github repository using lantern as a template repository user uploads .docx files to the /original/ subfolder user makes a commit using the github web interface: “add files via upload” this triggers a github actions workflow that performs the following tasks: provision a hosted virtual machine running ubuntu 20.04+ lts install pandoc 2.17+ checkout the main branch of the github repository convert each .docx file to a markdown file using pandoc move the markdown files to the /text/ subfolder commit this change back to the main branch of the github repository figure 2. logs from using github actions to convert manuscript files with pandoc. after this process, the user is ready to check the markdown files for any conversion errors and make necessary changes using github’s web interface for editing and previewing markdown. figure 3. github’s web interface for code (i.e. markdown) editing. figure 4. github’s web interface for previewing markdown rendered as html. lantern adopts a lightweight continuous integration / continuous deployment approach to oer publishing. lantern is preconfigured to build and deploy the html version of an oer project by default. other output formats, such as pdf and epub, are needed to be enabled by making a change in a configuration file. each time the user makes a commit to either the metadata.yml file or any of the markdown files in the /text/ subfolder, another github actions workflow will be triggered, executing the following tasks: provision a hosted virtual machine running ubuntu 20.04+ lts install pandoc 2.17+ and other lantern dependencies checkout the main branch of the github repository run the lantern.sh script, which builds a static html website for the oer by default deploy the website files to the gh-pages branch once the user enables the github pages feature on their repository, the website files contained within the gh-pages branch of the repository will be made publicly available at a github.io url. from then on, each new commit in a lantern oer’s repository will trigger a rebuild and redeployment of the oer website and output formats. users can disable the public accessibility of their in-development oer project by disabling github pages in their repository and re-enable it whenever they are ready to publish. static web technologies lantern builds a static website for the oer’s metadata, full-text, and downloadable assets (e.g. pdf, epub, etc.). static websites are faster, cheaper, simpler, and more secure than dynamically generated websites because they remove the authentication, database, and application layer typically used by content management systems (newson 2017; varner 2017; diaz 2018). static websites are read-only and require minimal maintenance in order for the public to visit and use the website. their reduced complexity makes them an attractive option for oer publishers and digital archivists (rumianek 2013). lantern transforms the metadata.yml and /text/ subfolder into a multi-page static website using pandoc and bash scripting (lantern.sh). if the user decides to produce pdf, epub, and docx versions of the oer project, each of those documents will be linked from the website available for download. here is output directory for a real-world example of an oer website built with lantern hosted on github pages [7]: css/ js/ .nojekyll cname index.html 010-intro.html 020-casual-inference.html 030-theory.html 040-data.html 050-hypothesis-testing.html 060-surveys.html 070-experiments.html 080-large-n.html 090-small-n.html 100-social-networks.html 110-machine-learning.html 120-conclusion.html 900-appendix-math.html clipperton_emps.docx clipperton_emps.epub clipperton_emps.pdf static websites are compelling options for well-scoped web publications, like oer, monographs, scholarship, digital collections, and exhibits, that libraries hope to maintain in perpetuity. lib-static provides models, concepts, and community around leveraging minimal digital infrastructures and static web technologies for library projects [8]. websites that require content management systems and server-side application software in order to function can become costly and difficult to maintain. oer publications in particular may require years of stability, even if the content is no longer updated. static websites provide that stability. portability with open source software library-based publishers, scholarly communications specialists, and open education advocates developed a keen interest in advancing an open infrastructure for scholarly publishing after the news that bepress, a provider of proprietary institutional repository software, was acquired by elsevier (schonfeld 2017; schonfeld 2019). this news generated new investments in open source software development for libraries and non-profits involved in digital publishing, among many other initiatives (lewis 2017; invest in open infrastructure). this momentum provided the motivation to prioritize the use of portable, cross-platform, open source software as the foundation from which lantern was developed. lantern requires the following software programs: pandoc: a command-line document converter pagedjs: a pdf generator for html styled with css any unix shell interpreter with grep, awk, and sed utilities any text editor git (a source code version management system), pandoc-crossref (a filter for handling cross-references to equations, figures, and tables), and latex (a pdf typesetting system), are not required but can be useful for collaborative or mathematically-rich oer projects. all of these programs are open source and compatible on macos, windows, and linux operating systems. open source software was a requirement for lantern’s design because it is less likely to produce the problem of vendor lock-in, a phenomenon in which a customer becomes dependent upon a vendor’s products (maxwell et al. 2019). lantern teaches the fundamental skills of using markdown, yaml, and command line programs necessary to use other software that performs similar functions if and when lantern’s software dependencies become unusable for any reason. markdown and yaml can be parsed and converted by hundreds of other software libraries in dozens of programming languages. github is provided as a convenience, but it is not required for use. lantern users can download the template files from github, run the software on their own files, and upload the output files to any web hosting service. conclusions reducing unnecessary overhead for the setup and maintenance of systems will ultimately lower technology and labor costs. an oer support infrastructure within an academic library is composed of people, workflows, and technologies. if given a budget to design and implement an oer program, it would be reasonable to think that the administrative roles and editorial processes for oer creation (i.e. the people and workflows) should be the highest priority, with promotion and discovery for oer adoption (i.e. people and workflows) following closely behind. lantern was designed to enable librarians to have a robust publishing workflow with the fewest technology maintenance expenses in order to devote more resources to the labor of oer creation and adoption. by adopting approaches like lib-static and minimal computing, librarians can focus on developing transferable skills rather than learning specific platforms. it is not the goal for lantern to become a “publishing platform” for oer. the goal is to demonstrate how fundamental digital skills with structured plain text, version control, and open source software can help librarians design and deploy sustainable web products for their communities. notes [1]: overview of oer licensing here: https://en.wikipedia.org/wiki/open_educational_resource#licensing_and_types [2]: git repository of lantern on github: https://github.com/nulib-oer/lantern [3]: pandoc documentation on using defaults files for configuration management: https://pandoc.org/manual.html#defaults-files [4]: example of a pandoc default’s file used for managing pdf output settings: https://github.com/nulib-oer/lantern/blob/main/settings/pdf.yml [5]: github actions is a continuous integration and continuous deployment service: https://github.com/features/actions [6]: github pages is a static website hosting service: https://pages.github.com/ [7]: example of the website files generated from lantern (https://github.com/nulib-oer/emps/tree/gh-pages) and the final website (https://emps.northwestern.pub/). [8]: lib-static community website: https://lib-static.github.io/ bibliography becker d, williamson e, wikle o. 2020. collectionbuilder-contentdm: developing a static web ‘skin’ for contentdm-based digital collections. the code4lib journal [internet]. (49). [accessed 2022 feb 23]. available from: https://journal.code4lib.org/articles/15326. davis, r. 2015. git and github for librarians. behavioral & social sciences librarian 34.3. 159–164. available from: https://academicworks.cuny.edu/jj_pubs/34/. diaz c. 2018. using static site generators for scholarly publications and open educational resources. the code4lib journal [internet].(42). [accessed 2022 feb 23]. available from: https://journal.code4lib.org/articles/13861. giorgio s, et al. 2019. what is git/github? – librarycarpentry/lc-git: library carpentry: introduction to git. [internet]. available from: http://doi.org/10.5281/zenodo.3265772. gil a. 2015. the user, the learner, and the machines we make. minimal computing: a working group of go: dh [internet]. [cited 2022 february 24]. available from: https://go-dh.github.io/mincomp/thoughts/2015/05/21/user-vs-learner/. gruber j. 2004. markdown: syntax. daring fireball [internet]. available from: https://daringfireball.net/projects/markdown/syntax. invest in open infrastructure (page 1). invest in open infrastructure. [accessed 2022 feb 23]. available from: https://investinopen.org/about/. lewis dw, goetsch l, graves d, roy m. 2018. funding community controlled open infrastructure for scholarly communication: the 2.5% commitment initiative | lewis | college & research libraries news. doi: https://doi.org/10.5860/crln.79.3.133. [accessed 2022 feb 9]. available from: https://crln.acrl.org/index.php/crlnews/article/view/16902. maxwell jw, hanson e, desai l, tiampo c, o’donnell k, ketheeswaran a, sun m, walter e, michelle e. 2019. setting context. in: mind the gap: a landscape analysis of open source publishing tools and platforms. [accessed 2022 feb 23]. available from: https://mindthegap.pubpub.org/pub/gei072ab/release/2. newson k. 2017. tools and workflows for collaborating on static website projects. the code4lib journal [internet].(38). [accessed 2022 feb 23]. available from: https://journal.code4lib.org/articles/12779. rumianek, m. 2013. archiving and recovering database-driven websites. d-lib magazine [internet]. [cited 2022 february 23]. available from: http://www.dlib.org/dlib/january13/rumianek/01rumianek.html. santiago a, ray l. 2020. navigating support models for oer publishing: case studies from the university of houston and the university of washington. reference services review. 48(3):397–413. doi:10.1108/rsr-03-2020-0019. schonfeld rc. 2017. elsevier acquires institutional repository provider bepress. the scholarly kitchen. [accessed 2022 feb 23]. available from: https://scholarlykitchen.sspnet.org/2017/08/02/elsevier-acquires-bepress/. schonfeld rc. 2019. invest in open infrastructure: an interview with dan whaley. the scholarly kitchen. [accessed 2022 feb 17]. available from: https://scholarlykitchen.sspnet.org/2019/06/12/invest-open-infrastructure/. tenen d & wythoff g. 2014. sustainable authorship in plain text using pandoc and markdown. the programming historian [internet]. available from: https://doi.org/10.46430/phen0041. varner s. 2017. minimal computing in libraries: introduction. minimal computing [internet]. [accessed 2022 feb 23]. available from: https://go-dh.github.io/mincomp/thoughts/2017/01/15/mincomp-libraries-intro/. wiley d. 2006. on the sustainability of open educational resource initiatives in higher education. oecd’s centre for educational research and innovation [internet]. [cited 2022 february 24]. available from: https://www.oecd.org/education/ceri/38645447.pdf. about the author chris diaz (https://chrisdaaz.github.io) is the digital publishing librarian at northwestern university. he is an avid user of static site generators for library-based publishing projects, such as journals, monographs, exhibits, and open educational resources. lantern received financial support from the association of research libraries’ venture fund program in 2020. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – editorial mission editorial committee process and structure code4lib issue 58, 2023-12-04 editorial issue 58 of the code4lib journal is bursting at the seams with examples of how libraries are creating new technologies, leveraging existing technologies, and exploring the use of ai to benefit library work. we had an unprecedented number of submissions this quarter and the resulting issue features 16 articles detailing some of the more unique and innovative technology projects libraries are working on today. this issue features several articles on how libraries are using programming tools and related technologies to enhance work in technical services. enhancing serials holdings data: a pymarc-powered clean-up project discusses the use of the alma api and the python library pymarc to conduct a post-migration clean-up project on serials holdings data. the use of python to support technical services work in libraries features research detailing the various ways libraries are using python in technical services, and pipeline or pipe dream: building a scaled automated metadata creation and ingest workflow using web scraping tools describes the use of python and apis to automate the collection of documents and data online, while a practical method for searching scholarly papers in the general index without a high-performance computer discusses how the r programming language can be used to build a bibliography and visualizations from the general index. in addition to the use of coding, other articles describe using various technology tools to enhance library work. using scalable vector graphics and google sheets to build a visual tool location webapp describes the use of google sheets and svgs, while bringing it all together – data from everywhere to build dashboards and real-time reporting using the alma api and google apps script discuss how the authors used tools like powerbi, powerautomate, and google apps to gather disparate data for dashboards and reporting. other technology tools being used include airtable and aviary, discussed in using airtable to download and parse digital humanities data and leveraging aviary for past and future audiovisual collections respectively. standing up vendor-provided web hosting wervices at florida state university libraries: a case study describes how florida state university libraries is using reclaim hosting’s domain of one’s own web-hosting service to provide web domains to fsu faculty, staff, and students. also included in this issue are articles on technology being used for archives and digital collections, including islandora for archival access and discovery on how unlv implemented islandora 2, and developing a multi-portal digital library system: exploring the technical decision-making in developing the new university of florida digital collections system, about how uf created their own digital collections system using a combination of python, apis, elasticsearch, reactjs, postgresql and more. jupyter notebooks and institutional repositories: realities, opportunities and exploring a path forward discusses how institutional repositories can be used to preserve scholarship housed in jupyter notebooks. issue 58 also includes several articles that explore the potential use of ai in libraries, including the use of chatgpt in beyond the hype cycle: experiments with chatgpt’s advanced data analysis at the palo alto city library and automated speech recognition technologies in comparative analysis of automated speech recognition technologies for enhanced audiovisual accessibility. finally, using event notifications, solid and orchestration for decentralizing and decoupling scholarly communication describes koreografeye, an automated assistant prototype that can enhance scholarly infrastructure, providing value-added services to institutional repositories. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – applying lessons from 8 things we hate about it to libraries mission editorial committee process and structure code4lib issue 13, 2011-04-11 applying lessons from 8 things we hate about it to libraries book review of 8 things we hate about it with commentary on how susan cramm’s points can be applied to libraries. cramm, susan. 2010. 8 things we hate about i.t. boston (ma): harvard business press. coins by timothy m. mcgeary introduction in 8 things we hate about it susan cramm discusses frustrations business leaders have with it and strategies to remove those frustrations in order to form effective partnerships.  while this book focuses on solving the traditional corporate struggle between business leaders and their it group, library technologists are in a unique position in this struggle: right in the middle. the vast majority of academic libraries fall into two categories: libraries that are separate from central it and libraries that are merged with it in one organization. in both of these examples, librarians are the “business leaders” bringing projects, ideas, and requirements to library technologists, the latter of which fit the classic role of it.  but within the larger organization, library technologists turn around seeking the same support from central it.  often this role is a no-win situation, due to lack of power and difficulties in communication.   even though the examples susan cramm offers are focused on the private sector, the principles are just as important to discuss and implement in libraries.  all librarians and library technologists would be wise to heed the lessons in ms. cramm’s book, including those from libraries that either completely control or outsource their entire it infrastructure and support. cramm lays out the following battles with it: service or control, results or respect, tactics or strategy, expense or investment, quickness or quality, customization or standardization, innovation or bureaucracy, goodness or greatness.  for each battle, i will briefly discuss ms. cramm’s descriptions of the battles and suggestions for overcoming these challenges, while providing examples and commentary on their applications to the library.  before going any further, it is important to define a few key terms that ms. cramm repeats frequently: term definition business or line leader generally non-technical, high-level staff who come up with ideas that promote a business idea or end-user goal; in the library, this can be applied to library directors, librarians, or other library staff not directly building or supporting technology. it leader generally technical staff, but sometimes refers to management of technology teams or units, and must make decisions consistent with the enterprise; in the library, this can be applied to a systems librarian who is the one-stop source of all technical solutions. enterprise the entire it infrastructure and technological foundation of the business.  this includes staffing, servers, software, operations costs, recovery, new development, and support.  when enterprise is applied to a people group (also referred to generally as it), it can be assumed a cio or cto is top personnel of the enterprise, and thus the enterprise people group works in the best interest of the cio/cto office. 1) service or control it leaders want to please business experts because they get excited  solving big problems, but it leaders answer to both business leaders and the enterprise, which may make it difficult for it leaders to fully meet the expectations.  business leaders have a present problem and want a present solution, but the enterprise requires long-term sustainability. the line leader wants service the enterprise wants control i have a great idea for a project. yes, but it’s not good enough. the systems need to do a lot. yes, but you can spend only a little. i know what the systems need to do. yes, but you need to make sure others agree. i have a vendor that i would like to work with. yes, but you need to use approved vendors. there’s a package that does everything we need. yes, but you need to comply with standards. i’d like to get started right now. yes, but you have to wait for resources. i’d like the project to get done faster. yes, but you need to follow processes. the project just needs a little more time (or money). yes, but your time is up, and you need to put the project out of its misery. the system’s ready to be rolled out. yes, but you need to comply with the security, regulatory compliance, and business continuity policies. the system has generated great business. yes, but you haven’t increased your p&l targets. the system needs to be upgraded (or enhanced). yes, but you aren’t using what you have. table 1-1 service or control (p. 17) there are four parts to what cramm describes as a balanced approach for organizations to have successful partnerships between business and it leadership.  these are: realizing values (investing in projects with highest roi and holding those investments accountable); serving the business (balancing innovation as an art not a science, thus projects are most successful with clearly defined outcomes, limiting time frame, funding in stages, scope is managed appropriately, and the right resources are assigned); running efficiently (simplifying and standardizing technologies, vendors, operations, and creating efficient automated processes); securing the future (leverage existing it solutions to support strategic business requirements, thus eliminating the multitude of applications and systems needing support, upgrades, or overhauls in the future).  (p. 18-19) the corresponding key to finding a balance in libraries is acting in partnership with both business and it.  it’s not service or control, but service and control.  partners back each other, and look to work together for a successful venture.  librarians need to realize solutions must be sustainable, not just fill a need.  technologists need to realize they can’t solve a problem on their own just because a problem has been stated.  both groups need to think strategically together about how a potential project benefits their larger organization. early in my career, as the lone library technologist in my organization, i made four of the eleven business-leader statements listed in table 1-1 when communicating with central it staff about a project.  in fact, i was so entrenched in the library business requirements that i lost sight of both the relationship dependency i had with central it and the enterprise.  the result was a very vocal argument with their cubicle farm.  regardless of any shared blame, i was both too naïve and too inexperienced to make such demands on the enterprise.  while the hard knocks experience taught me a great deal, i would have rather been given this advice before that project, rather than spending the greater part of the next year re-building the respect i had lost. 2) results or respect the business leader wants results and generally does not understand the it requirements those results depend on.  most frequently the solution has been to go around it directly to a vendor if the business leader does not believe it will produce their required results.  but often it is still required to intervene, at a minimum, or take over entirely.  (p. 29-30) cramm urges business leaders, and i believe this is just as true for library leaders, to consider focusing on building connections and relationships with it instead of merely focusing on the results. (p. 30) make no mistake – this is hard: personalities clash, experiences vary, and alignments differ.  consider: it people are more aligned and/or affiliated with it and their technology than their library organization at large. it staff have different backgrounds or experiences within libraries and library business that makes communication difficult. business and library leaders focus on getting a project done now; it leaders focus on getting the project done right, to sustain and support it. (p. 31) but it is important to realize that both business and it staff hate the control of bureaucracy.  this common point of pain can often be a jumping point in starting to build a sustainable, and respectful, relationship between library leaders and it. another consideration for library leaders in building respect with library it is this statistic: on average, 55% of it human resources are required to maintain, support, or operate it solutions.  10% are typically dedicated to planning, with 35% left over for new development. (p. 34) mileage will vary, but my opinion is that within the library, it technologists spend closer to 70% on maintenance, operation, and support, leaving 30% for planning and new development.  often planning gets an even shorter stick, or maybe no allocation at all, to the detriment of both future development and especially the inevitable, and more difficult, support, thus straining the next cycle of new development. library business leaders can change the rate of results by changing their focus from results to respect.  add an it leader on your team, elevate it to a true partner status, and develop interpersonal connections with it members who are working on your projects.  these are excellent ways to ensure greater rates of success for your projects.  last, seek their input about what they expect a library business leader (you) to accomplish. (p. 38-40) 3) tactics or strategy strategies are not goals, annual objectives, or key initiatives.  strategy is also not about meeting short-term demands that overwhelm our time and resources.  strategy, as cramm defines it, is the foundation used to make daily, monthly, quarterly, and annual decisions, and the tactics concerning goals, objectives, and initiatives. (p. 43-45) because of short staffing in many, if not most, libraries, strategic planning tends to lose out to demands, objectives, and goals.  library leaders decide that demand for applications, and the innate desire to fulfill that demand, suffices for it strategy.  what is often not realized is that building strategy is an ongoing process, and not finite events that occur every three to five years. (p. 60) annual goals lists, while effective for planning, should be checked against and prioritized by your it strategy.  if your library doesn’t have one, now is the time to start making one. but who initiates strategy?  typically both business and it wait for the other, as shown in the following table. business says it leaders should… it says that business should… “partner more with the business to identify how technology can be strategically deployed for the business.” “involve it as early as possible when contemplating changes to the business or new initiatives.” “set common goals for long-term business initiatives and the technology supporting them.” “involve it early and strategically.” “get more involved in the early stages of developing strategy.” “bring in it early; initiate frequent strategy discussions.” “be involved in the strategic meetings within all aspects of the business.” “collaborate with it to develop strategic and operating plans.” “offer visioning workshops to broaden the minds of business leaders.” “involve it in strategy discussions.” table 3-1: who should involve whom? (p. 45) the key is to work together.  it will often not know the business problem, especially in the library, which needs to be solved with an it-based strategy.  likewise ordering it solutions, as one-off applications, will reveal your ineffective it-based strategy and, likely, an unwilling it participant. (p. 46) while this chapter deals with complex strategic positioning, librarians and technologists should follow a similar path when developing it-enabled business strategy, as cramm describes in the figure below: understanding the fundamentals industry, competitive environment, key trends business fundamentals (business outlook, economic model, key metrics, long-term targets) key initiatives brainstorm how to influence performance customers (number, channels, features, pricing) capital (facilities, equipment, inventory, etc.) resources (people, raw materials, energy, etc.) cycle time (sales, order, fulfillment, replenishment, etc.) articulate business objectives as they relate to cost focus, value differentiation, flexibility, agility, growth, and human resources articulate it objectives understand it fundamentals (strengths/weaknesses, key metrics, long-term targets, key initiatives) for each business objective, articulate the role of it and the implication to process, information, and it architecture identify it-enabled initiatives articulate key initiatives, timing, and success measures figure 3-1: deriving it-enabled business strategy (p. 55) the key to all this is gaining quality commitment to the strategy beyond the quality of the idea. (p. 60) ideas come and go; yet even good ideas require strong commitment for a project to become successful.  without commitment from both the library leader and it, a good idea could easily become a failed project.  by building a strong strategic foundation based on respectful communication of both it and library/business initiatives, projects will have a much better chance for successful implementation and on-going support. 4) expense or investment of the 8 things we hate about it, this might be the most complicated to translate to library land because it has such a corporate feel to it.  but in reality, the pieces of demand management, which is the focus of deciding if a project is an expense or an investment, is just as important within library technology projects.  whether demand management is actually performed or not will likely vary based on the size and it-maturity of the library.  whatever level of demand management is implemented plays an important, if not vital, role in successful technology ventures. so what is demand management?  cramm defines it as the “process of allocating limited resources to the overall benefit of the enterprise.” (p. 64) continuing in the definition, demand management is a cycle and a process involving the following steps: strategic planning – as addressed above, this provides the context for prioritization of investing in it-enabled projects portfolio management – provides an on-going guidance on project investment decision and project review decision rights – the principle that business leaders have authority to decide what it is needed and it leaders have authority to decide how it is delivered financial planning determines amount of funding or price it costs for business to find external funding prioritization – occurs parallel to and in conjunction with the four above steps value management – accountability for defined business value and monitoring of actual results (p. 64-65) of all enterprises, libraries can surely understand the definition, and impact, of limited resources; therefore it is even more critical for libraries to implement demand management.  without using pieces like strategic planning and portfolio management, libraries can become easily overwhelmed with too many projects on-going or too many applications to support long-term.  at the same time, it is important that each side, library business leaders and it leaders, be responsible for their areas of expertise.  not only does this maintain balance within project-planning relationships, but it also keeps accountabilities balanced when evaluating outcomes. of all areas, value management is the step most often ignored or implemented poorly in all businesses that use it.  we easily attempt to predict or define the value of a project beforehand, but do we honestly seek to evaluate whether that value was returned?  libraries rarely have a financial return on investment, so usage statistics are often the most reliable return, albeit sometimes misleading.  regardless of the metric used to rate the return on investment, it is prudent for libraries to take the time to evaluate whether existing projects are as valuable in operation as they were predicted during planning. 5) quickness or quality only half of it projects deliver successfully or on time, on budget, and on spec.  12% deliver too little or are canceled, 33% too late (and by an average of 71% longer than scheduled), and more than one-third are over budget (by an average of 41%). (p. 85-86) the biggest reason is that defining specifications and requirements is very hard, and iterations take time to get it right.  this naturally results in a frustration between “done” and “done enough.”  (p. 86) the balance is between getting it done and getting it done right.  otherwise, an organization may find that being out of balance results in standalone solutions that are difficult to integrate or over-engineered solutions that miss the business requirements.  cramm suggests, however, that project success can be increased through these steps (p. 89): define a clear objective for the business and people on the front lines engage the intellectual, emotional, and tangible aspects of the people affected by this new project integrate and streamline business processes leverage existing technology to the fullest use fast-cycle approaches, using best human resources and delivering every 3-6 months beyond cramm’s suggestion of delivering every 3-6 months, there is an increase in investment within higher education on agile (also known as scrum) project methodology.  some colleges and universities have joined together to create an organization called scrumu (http://www.scrumu.edu) to support implementations of scrum project management.  i have implemented agile project management at lehigh with some initial success. [1] agile allows both a quick delivery of solutions with the ability to revise requirements and specifications before going too far down the wrong path.  as always, results will vary based on the implementation, but it can provide a jumpstart to solving the dilemma cramm is describing in this section. 6) customization or standardization cramm compares moving a software solution into production to moving into an exclusive community – it must measure up to the existing standards. (p. 108) how “gated” the higher education and library production environments are varies widely.  much commentary has been made recently about enterprise versus toy projects in the code4lib community (see [2] hellman, 2010, for example), but in reality a production environment there are existing standards that must be met.  these existing standards usually (and should) contain requirements to provide adequate documentation on how the application was built, proves the application was thoroughly tested, and that it meets the organization’s requirements of compliance, security, and service.  the benefit of such scrutiny, once accepted, is that your project acquires security, backup, performance monitoring, and detailed support. (p. 108) but cramm adds that you still need to do some more work (p. 109-110): train users thoroughly to use the full application resolve technical issues quickly maintain and enhance the application secure funding/resources to support all of the above the rest of the chapter cramm focuses on the need to reduce the total cost of running technology.  it is important, as the business leaders in the relationship between the library and central it, to consider the impact our new project has on the it enterprise.  do we really need a separate server?  can an existing database server be leveraged?  can we piggyback on a storage solution?  are we utilizing enough of the servers and storage already provisioned for us?  libraries should think carefully about these questions and their respective answers when considering the deployment of our new technology. 7) innovation or bureaucracy cramm states that 37% of it leaders and 50% of business leaders agree: “it is overly bureaucratic and control oriented.” (p. 125) technologists would rather have business leaders marvel at a successful project than complain about why it says it can’t be done.  but it struggles daily with an overwhelming amount of project requests, on top of supporting existing enterprise solutions. (p. 126) three key areas bog down it (p. 126-127): dealing with existing complexity of mixed and matched solutions, unstandardized legacy systems, and standalone applications.  reducing this complexity would significantly reduce cost. promoting enterprise interest – once it finds and implements standardized technology, it becomes a strategic initiative to leverage it where possible, often at the frustration of business leaders looking for new innovations. lack of resources – it is often not proactive in supporting the organization or driving business decisions because it lacks resources to be proactive while standardizing and supporting the current enterprise. to create more innovation, library leaders need to take more responsibility to be innovative themselves.  this requires a more hands-on approach in partnership with it.  this removes the “reading of the mind” aspect of the business-it relationship.  requirements documents do not do this – person-to-person activity does. (p. 129) secondly, ask it to create a solution that allows you to do more without their need to intervene. gain their trust by asking them to build the functionality for you to do more on your own, while admitting that any harm you do will have serious consequences in a system clean-up.  take accountability for that privilege and power. (p. 129-130) finally, forge a new partnership, as cramm shows in table 7-1 from page 139: helping it help the business helping the business help it embrace enterprise interests clarify and streamline decision making strengthen relationships with it create a business-savvy and service-oriented it organization develop it-enabled business strategies facilitate the development of it-enabled business strategies and enterprise architecture generate value from it-enabled investment ensure that it-enabled value is realized deliver quickly and with quality deliver quickly and with quality focus customizations on necessary differences drive down year-over-year lights-on expenses invest in it smarts enhance business partner self-sufficiency assume permanent accountability for the it assets that fuel your business fully democratize it table 7-1: leadership principles of the new partnership 8) goodness or greatness cramm hopes that by addressing the first seven issues, business leaders can promote it from being just good to great.  this will happen from encouraging more it-smart business leadership to treating it as a partner, not as a service, as well as following the entire organizational strategic chain of planning, investing, innovating, delivering, and operating. (p. 141-143) confronting the difficult facts of supporting the entire enterprise, cramm asks us, as business leaders to our it, to answer these questions (p. 143-144): what have i learned? what frustrates me about it? how am i contributing to my frustrations with it? what am i going to do about it? but while we may think we know what we need to change, cramm advises that it is important to do a reality check with our it partners.  get feedback and own up to it. (p. 146) cramm gives excellent examples about how good it can be, and realistic examples of how bad it can get without the proper leadership and partnership. conclusion it is the responsibility of library leaders to set the tone of partnership with it, and our role as library technologists to see both sides of the coin.  library technologists have a unique opportunity to serve library organizations as both it providers to librarians and business leaders to the larger supporting it organization.  susan cramm offers excellent advice, strategy, and most importantly examples that can be used to position library technology in effective partnerships within our organizations.  it is not an exaggeration, in the ever-increasing technological era we live in, to describe library technology as a keystone for our organizations.  but in taking on that position, it is even more important to heed the strategic advice of leading effectively between the business of libraries and the enterprise of it. references: [1] mcgeary, timothy m.  january 2011. implementing agile at lehigh university. team dynamixhe community column [internet]; available from: http://community.teamdynamix.com/columns/73/implementing-agile-at-lehigh-university [2] hellman, eric. february 8, 2011. toys and tools vs. the enterprise at code4lib. go to hellman [internet]; available from: http://go-to-hellman.blogspot.com/2011/02/toys-and-tools-vs-enterprise-at.html about the author tim mcgeary serves as the team leader of library technology at lehigh university libraries, and is responsible for the technology infrastructure, specifically focusing on leading the development of efficient and dynamic solutions to connect library collections to users.  tim has presented nationally and regionally on library technology development, managing electronic resources, open source solutions, and the kuali open library environment (ole).  tim currently serves as kuali ole functional council member for lehigh university, the code4lib journal editorial committee, the lyrasis advisory panel for open source initiatives, and the niso erm data standards & best practices review committee. subscribe to comments: for this article | for all articles 2 responses to "applying lessons from 8 things we hate about it to libraries" please leave a response below: jenn riley, 2011-04-12 fantastic writeup tim. your insights applying this model to libraries strongly resonate with me, highlighting issues i and many others are currently struggling with. thanks for this. timmcgeary, 2011-04-14 you are very welcome, jenn. it is my pleasure. thank you for your kind comments. leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – a video digital library to support physicians’ decision-making about autism mission editorial committee process and structure code4lib issue 23, 2014-01-17 a video digital library to support physicians’ decision-making about autism a prototype digital video library was developed as part of a project to assist rural primary care clinics with diagnosis of autism, funded by the national network of libraries of medicine. the digital video library takes play sample videos generated by a rural clinic and makes it available to experts at the autism spectrum disorders (asd) clinic at the university of alabama. the experts are able to annotate segments of the video using an integrated version of the childhood autism ratings scale-second edition standard version (cars2). the digital video library then extracts the annotated segments, and provides a robust search and browse feature. the videos can then be accessed by the subject’s primary care physician. this article summarizes the development and features of the digital video library. by matthew a. griffin, mlis, dan albertson, ph.d., and angela b. barber, ph.d. introduction a prototype video digital library was developed as part of a partnership between primary care medical clinics and the university of alabama, a project funded by the national network of libraries of medicine, southeastern/atlantic region (nnlm sea). this article summarizes the current functionality of a prototype video digital library aimed at supporting physicians treating potential autism cases and describes the developmental processes and implemented system components. caregivers of children who fail an autism screener at the primary care clinics partnering with this project may be asked if trained staff at the clinic can conduct a video-recorded structured play-sample for further analysis at the university of alabama autism spectrum disorders (asd) clinic. these videos (children playing) can be uploaded to the video digital library, which in turn makes them available to the asd clinic, where teams of experts can select full play-sample videos to examine within a secure area of the video digital library. observations, scores, and other clinical notes can be attached or appended to the video (i.e. annotated) using an integrated version of the childhood autism ratings scale-second edition standard version (cars2). the video digital library then uses evaluation data and other input from autism experts to segment the full videos, which would otherwise be inefficient for a child’s primary care physician to use within the context of patient care or to navigate to the meaningful observations within a video. researchers in this ongoing project anticipate that physicians will be able to use the processed shorter video clips, generated from the full play-sample videos, and corresponding embedded feedback from autism experts, to make decisions about patient care. the digital library ultimately enables physicians and autism experts alike to search and browse usable individual video clips of patients by score, test type, gender, age, and other attributes, using an interface designed around the envisioned users (i.e. health professionals) and basic human computer interaction (hci) metrics. having the processed video clips accessible through the digital library allows physicians to compare and contrast different patients and observations across a larger video collection, which can, ultimately, enhance understanding about autism. the video digital library is a secure, web accessible, video retrieval system that uses an apache web server and ssl certificates for information transmission over hypertext transfer protocol (https). php, mysql, javascript, jquery, python, and ffmpeg are the current developmental tools used for implementation and ongoing maintenance of the video digital library: mysql is the database system of the video digital library. php enables database connectivity and added security. python programs execute the video processing functions of the video digital library. the python functions segment and index the contributed video files by handling the naming and iterative functions. ffmpeg manipulates the video files. details of the video library’s implementation is presented in this article, along with plans for future development, which will further examine certain characteristics of clinical videos for improving search, browse, and application of video as a resource in the context of patient care. technical innovations of this project include the design of user interfaces to collect, order, and effectively present different information formats for supporting clinical tasks and decision-making. key innovations of video digital library in this section, two important pages of the digital library with unique interface features will be detailed. the design and functionality of the annotation and search pages contain key innovations of the video digital library. annotation page the annotation page enables experts of the ua asd clinic to score children and their behaviors while watching a video play session in the digital library. this page incorporates a digital adaptation of the childhood autism ratings scale-second edition standard version (cars2) for conducting patient assessments (schopler, et. al., 2013)]. definitions of scores and categories were inserted by the developers into the annotation page so that the autism experts could quickly reference the meanings of each score across all categories. each category (e.g. imitation, object use, relating to people, etc.) of the cars2 has a function that allows the autism experts to indicate up to three time points, along with an accompanying notes field for times indicated. the autism experts, using the video digital library, manually input time points in mm:ss format and input text in the notes field. the form does not allow for saving drafts; however, since the autism expert must watch the full video to conduct assessments and complete the annotation form, this was deemed to not be a necessary function. once the form is submitted, a record for the video is transmitted to the database, and the full video is marked for processing by programs used to extract the video clips. full videos are processed, and individual clips of notable tests are extracted and created. therefore, corresponding search results of individual clips are available on the site within 15 minutes of the annotation form being completed and submitted. figure 1. screen capture of the annotation page. search page the search functions of the video digital library comprise the other innovative page. first, a user can narrow search results dynamically by using the picnet table filter (tapia g … [updated 2013]), described further below. the filter creates a dropdown menu that the user can use to limit the search results by hiding the content of filtered table elements. in addition to the dynamic content, such as age or patient id being filtered, the static definitions of the scores displayed can be refined by keyword search. all limiters of the picnet table filter can be used in combination with each other to limit results shown. in the event of a system error, the clear selection button placed at the top of the results page will refresh and initiate a new session. as important as these two pages are for users of the video digital library, the overall functionality is dependent on its systematic ability to segment video files, i.e. extract video clips. figure 2. screen capture of the search page with picnet filter. development of video digital library the video digital library is a new application that relies on a number of open source tools. open source tools used for implementation, other than those briefly mentioned above, are described in this section. open source tools used with the assortment of processes required to build this interactive video digital library, the primary developer looked for open source tools that would provide the means to develop and implement the needed functionality. the smarty template engine design formed the building blocks for the individual webpages (basic syntax … [updated 2013]). smarty “tags” allow variables to be securely shared across multiple webpages, which helped avoid having to store variables in the address bar or in cookies. smarty tags also provide the compartmentalization of the development process, and the smarty template engine permits each page, specifically the annotation and search pages, to have the required flexibility of unique programming and functionality. adminer (formerly phpminadmin), an open source program, is used as a database management tool. adminer minimalized installation steps and requirements (given it is a single php file) and provided a clean interface for the database manager. picnet table filter, as presented above, provided an open source solution for the search page, providing the ability to filter results once the main search query is submitted. picnet was developed with jquery, and thus can dynamically change the results displayed through keyword or dropdown menu. having the ability to search and browse by both known and exploratory criteria is crucial for a digital library type of retrieval tool. security security of the video digital library was a significant consideration throughout development and implementation; several security measures and methods were implemented. these barriers were designed to make it unlikely that an unauthorized user or program could access information, such as the videos, clinic notes, and scores, from the video digital library. the primary security features included the php templates to hide webpage extensions in an attempt to camouflage the language in which webpages were written. in addition, secure sessions were also implemented, which were tested with several hacking scenarios by the primary developer. these tests helped suggest several security approaches and enhancements. one specific change, implemented by the primary developer, was the addition of a conditional statement to test the user’s ip-address, saved from the current login session. the temporarily saved ip-address will block a user if their ip-address changes during a session. when this happens, the user is presented with a login screen for authorized users to reenter their credentials. video being the most important aspect of the video digital library, we used caution in designing the mechanism for delivering video. we decided to use flash early on in development process so that older web browsers would be able to play the video. but without a video streaming server, flash would download a copy of the video to the user’s computer causing a serious security issue for the video digital library. we researched several methods to provide video streaming for the video digital library, and module h264 was found to fit our needs. after installing h264 several changes to the coding were necessary. this was an easy task due to the video digital library referencing one function in order to play video. making the necessary changes there allowed the h264 module to function for the entire video digital library. full directions on h264 available from http://h264.code-shop.com/trac. to prevent sql injection during user searches, the system uses php data objects (pdo) extensions (achour m … [updated 2013]). pdo extension is a database driver for php that is database specific. for this project we used pdo_mysql, but several others are available at: www.php.net/manual/en/pdo.drivers.php pdo also cleans any sql queries of any trouble caused by characters such as: ‘)’, ‘ ;’, ‘}’, and ‘]’. the example code below shows how we used pdo to insert records into the database. $database; try { $database = new pdo('mysql:host=127.0.0.1;dbname=`database_name`, `user`, `password`); //solution to connection error by adding these following lines. $database -> setattribute( pdo::attr_default_fetch_mode, pdo::fetch_assoc ); $database -> setattribute( pdo::attr_persistent, true ); } catch (pdoexception $error) { $database = new pdo('mysql:host=localhost;dbname=`database_name`, `user`, `password`); $database -> setattribute( pdo::attr_default_fetch_mode, pdo::fetch_assoc ); $database -> setattribute( pdo::attr_persistent, true ); } figure 3. connection to the database using pdo //prep the query to the database the appropriate info about clips $query = $database->prepare(<<<hd insert into clips (sessionsid, patientid, ratingid, path, key_frame) values (:sessionid, :patientid, :ratingid, :path, :key_frame); hd ); figure 4. preparing database query using pdo //load the clean variables and write queries to the database $successful &= $query->execute(array( ':sessionsid' => $_sessions['sessionsid'], ':patientid' => $patientid, ':ratingid' => $ratingid, ':path' => "$/$filename.mp4”, ':key_frame' => "$/$filename.jpg”, ':user_id' => $_session['userid’] )); figure 5. writing to the database using pdo as seen above, the pdo function was called to connect to the database and use the :table_field => $php_variable to insert each record. this approach prevents a user from conducting a search for “child play; truncate table patient;” and causing everything in the patient’s table to be erased. video processing the program to automatically segment the video into individual clips required multiple iterations, or versions. one such version did not use python, but instead called ffmpeg directly from php. however, in order to quickly develop a script to control for file naming and check for file creation, we used python. the main function of the python script is to call ffmpeg for the clip creation and keyframe extraction, from the clips, along with creating filenames and file paths. clip extraction ffmpeg enables video clip extraction. the correct syntax to create a usable video file was discovered by trial and error. the best solution was to use the application hand brake, which uses ffmpeg to process video. as a function of the software, handbrake displays the command line parameters used when converting video. the resulting code from handbrake is “ffpmeg, ‘-y’,’-i’, filename,’-vcodec’,’libx264’,’-s’,’320×240’…” this is then adapted to use in the python script as seen below. def cut(movie, start, clip_name): #calls ffmpeg to slice video if(subprocess.popen(['../ffmpeg', '-y', '-i', movie, '-ss', str(start), '-t', '15', "-vcodec", "libx264", "-s", "320x240", "-crf", '20', "-async", '2', "-ab", "96", "-ar", "44100", clip_name])): sleep(120) #solves racing condition when clips are long figure 6. primary python function as seen in the comments of the code, a race condition was discovered late in the development process, and the quick fix was to add a wait command. it is bad practice to depend upon a wait command to fix a bug. however, as this script is running on the server, the user will never be aware of this wait time. in addition, this fix appears to be one of the few ways to fix this race condition known as, “time of check to time of use,” or tocttou. tocttou explains the race condition which occurs in the fraction of a second between the check for a file and the use of that file. the race condition is in that fraction of a second when the file can be deleted, moved, or accessed by another program causing the file to be inaccessible (mathias et. al 2012). since this is not a unique problem to the python language, the effect can only be mitigated (pilgrim m … [updated 2013]). the race condition is compounded by the twelve ranking categories, each one having the possibility of three video clips being created. so, at most, thirty-six video clips and keyframes are created. having the program wait was the best solution, as there is no reliable method to check that a file is not in use or readable prior to calling ffmpeg. keyframe extraction keyframes are individual frames, i.e. still images, used to represent the visual contents of a video to a user through a user interface. for this project, once a video clip is created, a keyframe (i.e., visual surrogate) is needed to represent the clip to users in the search results page. the keyframes as extracted and employed for the video digital library are stored as jpegs in order to keep file size down. keyframes are selected by simply choosing and extracting the middle frame number from a video clip and designating that as the visual surrogate. for example, if a clip has 100 frames, approximately the 50th frame in the clip would be extracted and designated as the “keyframe.” this keyframe selection approach has proven to be just as effective as other more complex image processing approaches, previously evaluated for detecting the “best” keyframe through content-based comparisons. extracting the middle frame from the video was made a bit more complex in this context since there was no control over the length of the video uploaded to the website. the best solution was redirecting the standard output of ffmpeg using the parameter ‘stdout = pipe”, shown in the following code sample. the output contains detailed information of the video file created by ffmpeg. another function writes the output to a text file in order to generate the duration of the video. extracting duration is detailed below. def logfile(clip_name): result = subprocess.popen(['../ffmpeg', '-i', clip_name], stdout = pipe, stderr = stdout) return [result.stdout.readlines()] figure 7. python function for redirecting the standard output of ffmpeg. b'ffmpeg version n-43574-g6093960 copyright (c) 2000-2012 the ffmpeg developers\n' b' built on aug 15 2012 05:18:13 with gcc 4.6 (debian 4.6.3-1)\n' b" configuration: --prefix=/root/ffmpeg-static/64bit --extra-cflags='-i/root/ffmpeg-static/64bit/include -static' --extra-ldflags='-l/root/ffmpeg-static/64bit/lib -static' --extra-libs='-lxml2 -lexpat -lfreetype' --enable-static --disable-shared --disable-ffserver --disable-doc --enable-bzlib --enable-zlib --enable-postproc --enable-runtime-cpudetect --enable-libx264 --enable-gpl --enable-libtheora --enable-libvorbis --enable-libmp3lame --enable-gray --enable-libass --enable-libfreetype --enable-libopenjpeg --enable-libspeex --enable-libvo-aacenc --enable-libvo-amrwbenc --enable-version3 --enable-libvpx\n" b' libavutil 51. 69.100 / 51. 69.100\n' b' libavcodec 54. 52.100 / 54. 52.100\n' b' libavformat 54. 23.100 / 54. 23.100\n' b' libavdevice 54. 2.100 / 54. 2.100\n' b' libavfilter 3. 9.100 / 3. 9.100\n' b' libswscale 2. 1.101 / 2. 1.101\n' b' libswresample 0. 15.100 / 0. 15.100\n' b' libpostproc 52. 0.100 / 52. 0.100\n' b"input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/videos/demo01/1/1/3.mp4':\n" b' metadata:\n' b' major_brand : isom\n' b' minor_version : 512\n' b' compatible_brands: isomiso2avc1mp41\n' b' encoder : lavf54.23.100\n' b' duration: 00:00:15.03, start: 0.000000, bitrate: 522 kb/s\n' b' stream #0:0(und): video: h264 (high) (avc1 / 0x31637661), yuv420p, 320x240 [sar 1:1 dar 4:3], 385 kb/s, 29.97 fps, 29.97 tbr, 30k tbn, 59.94 tbc\n' b' metadata:\n' b' handler_name : videohandler\n' b' stream #0:1(und): audio: aac (mp4a / 0x6134706d), 44100 hz, stereo, s16, 128 kb/s\n' b' metadata:\n' b' handler_name : soundhandler\n' b'at least one output file must be specified\n' figure 8. raw output of ffmpeg as written in the text file. the script iterates over each line of the ffmpeg output, as seen above, checking for the string “duration” and then extracts the duration of the full video by slicing the string. a conditional test for duration and using string slicing calls the process for retrieval of the full duration video. the appropriate timing to extract the keyframe was computed by converting the given duration of a clip into seconds and dividing it by two. with the middle of the clip calculated, python names the clip iteratively with the regular expression (‘%s’ % (count)). the clip number or file name matches the resulting keyframe name. for example, python names the video full_video, then each extracted video clip numbered sequentially (e.g. 1.mp4, 2.mp4). the filename for the keyframe matches that of the video clip with a .jpg file extention. the full video, extracted clips and representing keyframes for each submitted play sample are all saved in a separate folder. the file path for each play sample (i.e. session for a child) is built using the patient id and the session id. for example, the path for the second session, “play sample,” of patient 1 would be ‘/data/1/2’. using python in conjunction with ffmpeg was challenging, but the end result is a digital library application able to process video and extract shorter clips according to input from the autism experts. future development at the conclusion of the primary developmental stages, system developers compiled a list of tools that would further enhance implementation and functionality of the video digital library. the annotation and search pages, discussed above, were notably complex to design and implement. future development of the system may potentially benefit by incorporating ajax (asynchronous javascript and xml) for making the annotation form more dynamic and user friendly, considering ajax allows webpage content to change and update without reloading the full webpage (ajax tutorial … [updated 2013]). one obstacle remains for any future development of this system. when patient age is used to search for clips, several clips are returned, but one of the clips returned does not match search criteria. it is not an error with the database entry and must be an error in the php code. further development of the search page is at the top of the developers’ priorities for future development due to the importance of being able to search the digital library using different user criteria. search is further complemented by browse functions, e.g. for browsing search results, which is significant for visual collections, as users expect visual feedback and surrogates to peruse a collection and to base relevance judgments. conclusion open source tools and platforms were combined to make a stable, secure, and innovative video digital library. lessons learned from the development of this system motivate future research and experimentation. this project demonstrates potential for enhancing clinical services in underserved areas. furthermore, the prototype video digital library can be used to streamline patient assessments by autism experts and provide enhanced information back to the patient’s primary physician for making clinical decisions. future research with this project can also employ the prototype presented here to examine how video digital libraries enhance understanding of autism among physicians and physician/caregiver communications. in addition, findings from future research may lead to health literacy centered designs of video digital libraries. references achour m, betz f. php manual [internet]. [updated 2013 nov 15]. www.php.net.; [cited 2013 nov 17]. available from: http://www.php.net/manual/en/ back ajax tutorial [internet]. [updated 2013 nov 17]. www.w3schools.com.; [cited 2013 nov 17]. available from: http://www.w3schools.com/ajax/ back bellard f. ffmpeg documentation [internet]. [updated 2013 nov 17]. www.ffmpeg.org.; [cited 2013 nov 17]. available from: http://www.ffmpeg.org/documentation.html basic syntax [internet]. [updated 2013 nov 13]. www.smarty.net.; [cited 2013 nov 13]. available from: http://www.smarty.net/docs/en/language.basic.syntax.tpl back mathias p., gross t. protecting applications against tocttou races by user-space chacing of file metadata. 2012. acm sigplan notices; 47(7):215-226. available from: http://libdata.lib.ua.edu/login?url=http://search.ebscohost.com/login.aspx?direct=true&db=edswsc&an=000308657200020&site=eds-live&scope=site back pilgrim m. 2009. [internet]. 2. apress.; [cited 2013 nov 17]. available from: http://www.diveinto.org/python3/ back schopler e, van bourgondien m. childhood autism rating scale, second edition (cars2) [internet]. [updated 2013 nov 15]. pearson.; [cited 2013 nov 15] . available from: http://www.pearsonclinical.com/psychology/products/100000164/childhood-autism-rating-scale-second-edition-cars2.html. back tapia g. picnet table filter [internet]. [updated 2013 nov 17]. www.picnet.com.au.; [cited 2013 nov 17]. available from: http://www.picnet.com.au/picnet-table-filter.html back about the authors matthew a. griffin is a first year doctoral student in the college of communications and information science, university of alabama where he recently received an mlis. matthew is interested in researching digital libraries as part of his doctoral program. dan albertson is an associate professor in the school of library and information studies, university of alabama. his work can be found in places such as the journal of the american society for information science and technology, journal of documentation, journal of information science, journal of education for library and information science, and others. dan’s primary research interest is interactive information retrieval; he holds a ph.d. in information science from indiana university, bloomington. angela barber is an assistant professor in the department of communicative disorders, university of alabama. her research focuses on early identification and intervention with young children who have autism. she holds a ph.d. in communication disorders from florida state university. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – archidora: integrating archivematica and islandora mission editorial committee process and structure code4lib issue 39, 2018-02-05 archidora: integrating archivematica and islandora “archidora” is shorthand for the publicly available integration between the open source software packages archivematica and islandora. sponsored by the university of saskatchewan library, this integration enables the automated ingest into archivematica of objects created in islandora. this will allow institutions that use islandora as a digital asset management system, particularly for digitized material, to take advantage of archivematica’s standards-based digital preservation functionality, without requiring staff doing digitization to interact with archivematica. this paper outlines the basic functionality and workflow of archidora; provides an overview of the development process including challenges and lessons learned; and discusses related initiatives and possible future directions for development. by tim hutchinson project background and institutional context the university of saskatchewan library has a long and productive history of digitization projects, with the first “virtual exhibit” launched in 1995, and over 50 thematic projects, both institutional and consortial, completed since then. until 2013, the university of saskatchewan archives and the special collections unit of the university of saskatchewan library were organizationally separate, but collaborated on projects and both received systems/it support from the library. starting in 2005, the university archives moved to a home-grown database for digital projects, allowing us to more systematically create metadata records and store high resolution images. in the meantime, special collections had started using contentdm for various collections. the initiation of the saskatchewan history online (sho) project in 2011 allowed the university library to move towards a more programmatic approach to digital initiatives. sho, originally dubbed the saskatchewan multitype digitization initiative, was a three-year provincial project funded by saskatchewan’s ministry of education – the government home of both the provincial library and the multitype library board. the university of saskatchewan library was contracted to coordinate the project, with a committee of the multitype library board, the saskatchewan digital alliance (sda), serving as an advisory body to the project. sda planning as well as discussions within the university library helped identify digital preservation as an important component of the sho project. for too long, our projects had produced extensive and unique digital content, but without any preservation plan or infrastructure beyond basic backups. a consultation with artefactual systems (lead developers for archivematica) in january 2012 set the wheels in motion towards our selection of archivematica as a preservation system to ultimately manage digital content from various sources and systems. in addition to islandora, which was selected as the digital asset management system for the sho project (and as a replacement for contentdm), we were also using dspace for an institutional repository of electronic theses and dissertations. of course, we also had material from many different digital projects (managed in legacy systems or as flat files) – not to mention born digital records. we thought that addressing the preservation of digitized material would be a good way to establish an infrastructure for digital preservation that could later be used for other systems like dspace, as well as the more complex area of born digital records. development process the main development work on the archivematica/islandora integration was completed between 2013 and 2014, with additional enhancements and bug fixes since then. overall, the project has moved much more slowly than anyone had anticipated – during the design, development, and deployment stages. discussions with artefactual systems started later in 2012, with design requirements partly informed by an islandora camp session that summer. the first quote was received in october 2012. however, this focused on archivematica to islandora workflow. further discussion about requirements – especially the requirement to have this process be automated, with staff doing digitization not needing to interact with archivematica – led to a more detailed project outline, and the clarification that development of islandora would also be required. artefactual took the lead on the project, coordinating with discovery garden, which would ultimately be the subcontractor for the development on the islandora side. the complexities of the technical interactions between the two systems, clarifications relating to university of saskatchewan workflows, and time constraints for all parties meant that a substantially revised quote did not arrive until may 2013. a contract was approved for work to take place between october 2013 and january 2014. ultimately this main phase of work was completed in september 2014. testing during the first phase of development revealed some limitations in the archidora module, i.e. desired functionality which had not been considered. in particular, we wanted to add a configuration option to suppress archivematica ingests for a given collection; to add better handling for compound objects and books; and to add the ability to delete obj datastreams (i.e. the master objects) in bulk. we undertook a separate contract with discovery garden, during fall 2014, to address this. since then, progress on moving into production has been slow, and we are now targeting early 2018 for this milestone. however, i am risking becoming the boy who cried wolf: i have declared that deployment was close at least a few times over the last couple years. there have been a number of factors contributing to this. both archivematica and islandora are under active development, and have many moving parts. on a few occasions, a change to another part of the code, or even configuration changes, have led to regressions or unexpected behaviour. in some cases it has been difficult to quickly assess whether the source of the problem is in islandora or archivematica. and resolving one issue has sometimes surfaced others. our lead in-house islandora developer made some changes to the archidora module in order to allow recursive use of the drush script. there was an understable learning curve involved, and the usual competing priorities for time. in tying up loose ends on the archivematica side, we did not initially formalize or schedule the work required. this work got done through a combination of goodwill on the part of artefactual (especially for a few items out of scope of the original contract), and the university library’s support contract with artefactual. as a result, timelines were a lot longer than would have been the case with a paid contract. design and functionality in general terms, objects are first ingested into islandora. these are added to a deposit in the archivematica storage service. after each deposit is finalized (reaching a certain size or time limit), they are sent to the archivematica dashboard to be processed, with an archival information package (aip) as the final output. this schematic from artefactual outlines the detailed workflow as initially developed. it remains largely accurate. figure 1. islandora/archivematica integration workflow, as originally designed [1]. the islandora/archivematica integration was achieved through the development of an islandora module called archidora; and through code changes to both the archivematica dashboard and storage service. on the archivematica side, a key component, which will facilitate integrations with other systems, was the development of a sword api, part of the storage service’s rest api. the sword api “allows 3rd party applications to automate the process of creating transfers” (sword api [updated 2017]). the following walks through the basic configuration, and the workflow for an object starting with its ingest into islandora. configuration – islandora configure the archidora module in islandora, with basic settings like storage service url, username and api key. you can also configure the maximum size of the transfer and the length of time (maximum age) before the transfer will be finalized (the settings are labelled aip but it’s really the deposit/transfer). after a transfer is finalized, no more objects will be added to it, and the processes on the archivematica side can begin. settings for the size and maximum age will largely depend on institutional workflows as well as server capacity (e.g. size of transfers that archivematica can handle). maximum size is likely more important, but the maximum age ensures that a transfer will not get abandoned if enough objects are not added to it. the cron time setting helps account for the ingest of compound objects, so that the xml file documenting the relationships between objects is ingested along with the objects. figure 2. archivematica integration configuration in islandora configuration – archivematica in the storage service, add a location … figure 3. configuration of location for fedora deposits in archivematica storage service … and a corresponding space for fedora deposits figure 4. configuration of space for fedora deposits in archivematica storage service the pipeline also needs to be configured as for any archivematica deployment, but to ensure that transfers are approved automatically, the api username and key need to be populated. figure 5. configuration of pipeline in archivematica storage service you can also configure a post-store callback, so that archivematica will update islandora with information about objects that have been ingested and therefore can be deleted from islandora. this generates a list in islandora’s “manage” tab for the relevant collection for manual processing, with an interface allowing either individual or bulk deleting (see figure 18). institutions may want to use this functionality to avoid redundant storage of the master digital objects, if the master no longer needs to be accessible in islandora. figure 6. configuration of post-store callback in archivematica storage service. in archivematica, it is important to make sure that the processing configuration is set up not to require any user intervention. figure 7. archivematica processing configuration if your digitization workflow ensures that you’re already reliably creating preservation quality files, you may also want to disable certain default normalization rules. for example, if a tiff file is identified in the format policy registry as a preservation format, archivematica will still normalize it if a normalization rule is enabled for that format (which is the default for tiff files). this will result in both the original file and the normalized file being saved in the aip. the ability to disable a rule was added as part of archidora development, but an arguably more intuitive enhancement would be to automatically skip the normalization task if an object is already in a preservation format. figure 8. archivematica preservation planning entry islandora workflow now we’re ready to bring some objects into islandora. if you use the zip importer, then it’s best to configure it to use the filename as the datastream label (e.g. filename.tif rather than obj). if you use a generic datastream label, archivematica will still be able to ingest the objects, but there may be microservices that return errors due to the lack of a file extension. in particular, ffmpeg has known issues. figure 9. configuration of islandora’s zip importer you can use any method to ingest objects into islandora. archivematica interacts with the fedora repository, not the drupal frontend. there is also a drush script available to do batch transfers of objects already in islandora. for this example, we have a small collection of three tiff files. figure 10. objects ingested into islandora. archivematica workflow on the archivematica side, the deposit initiated in islandora is first sent to the storage service. initially the islandora mets files are saved, and archivematica fetches the corresponding objects and mods files. figure 11. downloaded fedora deposit, showing submissiondocumentation folder. the objects are saved in separate subdirectories. this covers situations where there may be duplicate filenames in the same transfer – especially the “obj” naming mentioned earlier. figure 12. downloaded fedora deposit, showing top level. figure 13. downloaded fedora deposit, showing one object subfolder. once the deposit is finalized (following one or more cron calls in islandora), a transfer is initiated on the archivematica dashboard. there is an automation tool available to automatically remove successfully completed transfers and ingests from the dashboard.  the transfer name is automatically generated from the mods title (using the first record, if there are multiple objects in the deposit); where needed it will be sanitized to deal with spaces, diacritics, etc. figure 14. archivematica dashboard, showing transfers in progress. completed aips can be browsed and searched on the archival storage tab. in addition to the usual search functionality, you can search for an islandora pid (e.g. islandora:1234) via the identifiers field. the full mods is not indexed, since it’s assumed users will do any detailed searching in islandora. figure 15. archivematica dashboard, archival storage tab the aip structure mirrors that of the deposit. figure 16. downloaded aip, showing objects folder. islandora troubleshooting and follow-up back in islandora, the manage | archivematica tab provides information about the status of individual objects. you can also use the “send to archivematica” button to initiate a new deposit. this is useful for testing, but would also be required in the case of a failed transfer or if an object in an existing record is replaced (archivematica will only fetch new objects). figure 17. islandora administrator interface, showing manage | archivematica tab log reports are also available under reports | recent log reports. these are primarily useful in the case of a failed deposit. at the collection level (or compound object/book level), the manage | archivematica tab provides an interface for deleting objects that have been ingested by archivematica, if the callback is configured. figure 18. islandora administrator interface, showing manage | archivematica tab at the collection level challenges and lessons learned planning while code regressions and limitations on people’s time (as outlined above) are hard to avoid, a key lesson learned is that the design requirements should have been much clearer from the outset. indeed, the initial requirements were developed by artefactual and discovery garden following a series of e-mails and meetings; the university library did not submit any formal documentation. this lack of concrete direction contributed to the false start we experienced, with a quote for an integration assuming workflow would start in archivematica. further, at the outset of the project, we had only recently adopted islandora; and as project lead i had very little hands-on experience with islandora before we started to do quality assurance, let alone during the development of requirements. for example, we missed some simple things, such as object naming conventions for the default batch import routine (resulting in duplicate filenames), or detailing how different object types (e.g. compound objects) should be handled. since this development work was part of the sho project, it was not actually managed within the library as a separate project, which might have helped mitigate some of these planning shortfalls. sustainability an ongoing challenge relates to ongoing maintenance and sustainability of the code. this initially resulted from a lack of understanding about the different development models for archivematica (maintained by artefactual systems) and islandora. artefactual systems is the “lead developer” of archivematica. as such, any development work done for clients is pursued with open source release, and more general utility, in mind. as artefactual describes on its website: “… although we may develop new features for you, we will not be creating a custom application that will need to be maintained by your organization. instead, we will incorporate the new customizations into later releases of the software and support them independently of your organization so that others can benefit from them” (artefactual systems – services – development). the development relating to the archivematica/islandora integration has been incorporated into the most recent public releases of archivematica, and is stable as of archivematica 1.6.1 and storage service 0.10.0. my experience had been with atom, the other software package maintained by artefactual systems. so i had mistakenly assumed that the development model for islandora, and the approach of discovery garden, would be similar. indeed, we incorporated language in the development contract with artefactual to allow both companies to publish the code under their respective open source licenses. (the university’s contract was with artefactual, which subcontracted with discovery garden for the work on the islandora module.) contrary to archivematica, islandora’s code is owned by the islandora foundation. during the process of arranging for the necessary approvals to transfer the archidora module to the foundation, we discovered one stumbling block: the lack of a volunteer component manager to shepherd its incorporation into the islandora code base [2]. since the development had been contracted out, we were not in a position to identify an in-house component manager. islandora guru mark jordan (simon fraser university) kindly volunteered for that role. however, a more important barrier was that the development of the module had not been done with general release initially in mind. the islandora foundation has accepted the module, but it is currently being hosted at islandora labs, and will require further development and testing to be considered for the core release (archidora module [updated 2015]). until this module is deployed by a broader sector of the islandora community and is ready for incorporation into the general release, it will essentially be a customization that the university of saskatchewan needs to maintain. future development opportunities there are a number of possible improvements to the archivematica/islandora integration. i will discuss just a few here. artefactual systems’ lead archivematica developer mentioned several other possibilities in his open repositories 2015 talk (simpson, 2015). a two-way street while this not a use case currently important for the university of saskatchewan, there has been a lot of interest expressed in integration of archivematica and islandora in the opposite direction. that is, ingest of objects first into archivematica, which would generate the packages required for either automated or manual upload to islandora – similar to the integrations for atom and contentdm, described above. this is of interest especially for institutions that are using islandora as the access system for both digitized and born-digital material. mark jordan’s presentation at islandora camp in 2012 described both use cases, and sketched out preliminary development strategies (jordan, 2012). a few institutions have reported on local customizations and experimentation. for example, michigan state university (msu) libraries transformed the mets file from archivematica into a version for ingest into fedora (collie and mak, 2013). a poster presentation also by msu (collie, higgins, mak and nicholson, 2014) further makes the case for islandora/archivematica integration, by highlighting elements of the ndsa levels of preservation (national digital stewardship alliance, 2013) that each system addresses. more recently, a couple threads in the islandora google group have captured interest on working on the archivematica to islandora integration, and reported on on possible development approaches. as outlined by mark jordan, “islandora is ready for this, via the islandora rest module. the work required to have archivematica produce dips for public access in islandora needs to be done on the archivematica side” [3]. the zuse institute berlin may have taken the archivematica to islandora integration the furthest, reporting at ipres 2015 on an implementation involving archivematica, fedora/islandora, and irods (klindt and amrhein, 2015). this code does not appear to be publicly available at this time. there is also potential for integration between archivematica and fedora, rather than islandora per se. indeed, the current integration is primarily between the archivematica storage service and the fedora repository, even though this is achieved through an islandora module. storage of aips in a fedora repository is one area of interest; this has been achieved, for example, at the universities of york and hull as part of an ongoing research data project (mitcham et al, 2016). integration of islandora metadata the mods files are saved in the aip as part of submissiondocumentation; and at least for images, mods and dc metadata become part of the archivematica mets file via exif tool output. but this descriptive metadata is not fully searchable, and the dmdsec section of the archivematica mets file is not populated, as it would be for imported metadata (archivematica documentation: import metadata). this kind of integration was not a priority for our initial development; we assume that most searching will take place in islandora, with the islandora pid sufficient to pull the master object from archivematica. however, adding more integrated metadata would introduce possibilities for richer integration with other systems, for example for dissemination information packages (dips) generated for atom or other access systems. there is also potential to take greater advantage of islandora’s available digital preservation functionality. the islandora premis module provides the capability to generate xml and html representations of premis data in islandora, currently including fixity checks, agent information, and rights metadata from the descriptive record [4]. this module was not on our radar during the initial development of archidora, and we have not currently implemented it. clearly, however, there would be advantages to including islandora’s premis data in the packages generated by archivematica. at least part of the premis output from islandora – fixity information – is currently included in the fedora mets file. development would likely be needed to configure how this data is included in the archivematica mets file, or to pull in the full premis xml file from islandora. re-ingesting objects/aips currently, the archivematica process is only triggered for new – not updated – objects in islandora. arguably a new aip should be created if digital objects are replaced; decisions about what extent of changes to metadata warrant a new aip are more challenging [5]. a manual process is available, through the “add to archivematica” button, but this is obviously subject to user error. another potential downside to this manual process is that this creates an aip with just that object, rather than the larger aips generated as part of normal workflow; and this option is not available at the book or compound object level. for larger sets of objects needing re-ingest, the drush script would be another option, but this also currently needs to be run manually. conclusion the archivematica/islandora integration adds to a growing set of integrations between archivematica and other digital preservation, access and repository software packages, including atom, contentdm, dspace, archivesspace (eckard, pillen and shallcross, 2017), and lockss. responding to a suggestion that archivematica and islandora might be competing for the same users, then artefactual president peter van garderen tweeted, “we don’t compete, we integrate” (van garderen, 2012). an artefactual analyst elabourated on this philosophy in her talk at open repositories 2015, focusing on endpoints and handoffs from source systems (mumma, 2015). while islandora has some digital preservation functionality, our preference is to use a system with digital preservation as a specialization, and take advantage of islandora’s specialization as an access and digital asset management system. since islandora is actively used by multiple users, archivematica also provides a better option for reliable preservation of the master objects. ultimately, as described in the background section, other systems and sources will feed into archivematica, so that we are not managing preservation in multiple systems. we are always glad to hear about interest in adopting and developing archidora [6]. over time, we are hopeful that wider adoption will result in archidora (in both archivematica and islandora) achieving a status as community owned and maintained software, integrating archivematica and islandora with workflows in both directions. references archidora module, islandora labs [internet]. [updated (last commit) 2015 march 2]. github [cited 2017 october 23]. available from: https://github.com/islandora-labs/archidora archivematica documentation: import metadata [internet]. artefactual systems: archivematica documentation, version 1.6 [cited 2017 october 19]. available from: https://www.archivematica.org/en/docs/archivematica-1.6/user-manual/transfer/import-metadata/ artefactual systems – services – development [internet]. artefactual systems website [cited 2017 october 23]. available from: https://www.artefactual.com/services/development/ collie a, higgins d, mak l, nicholson s. 2014. furthering the community: integrating archivematica and islandora [internet]. library information and technology association (poster presentation), january 2014. [cited 2017 october 18]. available from: https://figshare.com/articles/furthering_the_community_integrating_archivematica_and_islandora/899883 collie a, mak l. 2013. incompatible or interoperable? a mets bridge for a small gap between two digital preservation software packages [internet]. alcts metadata interest group, ala midwinter meeting; seattle, washington: 2013 january 27. [cited 2017 october 18]. available from: http://connect.ala.org/node/199172 eckard m, pillen d, shallcross m. 2017. bridging technologies to efficiently arrange and describe digital archives: the bentley historical library’s archivesspace-archivematica-dspace workflow integration project. code4lib journal [internet]; issue 35 (2017 january 30). [cited 2017 october 25]. available from: http://journal.code4lib.org/articles/12105 jordan m. 2012. integrating islandora and archivematica [internet]. charlottetown, canada: islandora camp, 2012 august 2. [cited 2017 october 18]. available from: http://summit.sfu.ca/item/10873 klindt m, amrhein k. 2015. one core preservation system for all your data – no exceptions! proceedings of the 12th international conference on digital preservation, chapel hill, north carolina, 2015 november 2-6 [internet]. [cited 2017 october 25]. available from: https://phaidra.univie.ac.at/detail_object/o:429551 mitcham j, awre c, allinson j, green r, wilson s. 2016. filling the digital preservation gap: a jisc research data spring project; phase three report [internet], 2016 october. [cited 2017 october 25]. available from: https://dx.doi.org/10.6084/m9.figshare.4040787 mumma c. 2015. archivematica: handshaking towards comprehensive digital preservation workflows. or2015: 10th international conference on open repositories, indianapolis, indiana, 2015 june 9 [internet]; [cited 2017 october 25]. available from: http://program.or2015.net/mumma-archivematica_integration-174_a.pdf national digital stewardship alliance, ndsa levels of preservation, version 1 (2013?) [internet]. [cited 2017 october 18]. available from: http://www.digitalpreservation.gov:8081/ndsa/activities/levels.html simpson j. 2015. archidora: leveraging archivematica preservation services with an islandora front-end [internet]. or2015: 10th international conference on open repositories, indianapolis, indiana, 2015 june 9. [cited 2017 october 23]. available from: http://program.or2015.net/simpson-archidora-229.pdf sword api [internet]. [updated 2017 march 23]. artefactual systems: archivematica wiki [cited 2017 october 25]. available from: https://wiki.archivematica.org/sword_api van garderen p. 2012 april 25 [internet]. twitter [cited 2017 october 25]. available from: https://twitter.com/pjvangarderen/status/195206083806113792 notes [1] from improvements/islandora [internet]. [updated 2016 march 17]. artefactual systems: archivematica wiki [cited 2017 october 20]. available from: https://wiki.archivematica.org/improvements/islandora. further technical documentation is also available on this page. a key difference between the schematic and the functionality as it now exists is that the post-store call back prompts archivematica to list the object(s) in the aip as ready to be deleted, but this is not done automatically; rather, the objects can be selected for individual or bulk deletion in the manage tab of the relevant collection/book/compound object. [2] the islandora foundation licensed software acceptance procedure [cited 2017 october 23; available from: http://islandora.ca/developers/lsap) appears to have been fleshed out since our contribution of archidora. an archived version of the page dated march 2015 (https://web.archive.org/web/20150316194238/http://islandora.ca/developers/lsap) does not include a reference to the component manager requirement. that element had been added by september 2015, with a more thorough revision as recently as march 2017. [3] islandora and archivematica [internet]. islandora users group, 2017 august 16 [cited 2017 october 23]. available from: https://groups.google.com/d/msg/islandora/5qxsz3vwbvw/ognos7lkagaj. see also aip, dip, sip generation [internet], islandora users group, 2017 august 30 [cited 2017 october 23]. available from: https://groups.google.com/d/topic/islandora/w7uee1wb0v4/discussion [4] islandora premis [internet]. github [cited 2017 october 19]. available from: https://github.com/islandora/islandora_premis. for more details and discussion of future directions, see jordan m., mclellan e., premis in open-source software: islandora and archivematica. in: dappert a., guenther r., peyrard s. (eds) digital preservation metadata for practitioners [internet]. springer, cham, 2016 [cited: 2017 october 18]. available from: https://doi.org/10.1007/978-3-319-43763-7_16 [5] a presentation at access 2017 outlined some of these issues, notably the fact that objects can be saved multiple times as part of a single ingest. see weiwei shi, shane murnaghan, and matt barnett, the way leads to pushmi-pullyu, a lightweight approach to managing content flow for repository preservation at uofa libraries [internet], saskatoon, canada: access 2017, 2017 september 28, [cited 2017 october 19]. available from: https://drive.google.com/file/d/0b8b5fxybn_3_tws0um1nmln1vu1iuzrvcuduqtqxeupyvhbr/view [6] beyond some informal inquiries, we are aware of archidora’s use in testing workflows for research data. see research data canada, rdc federated pilot for data ingest and preservation, 2015 january 9 [internet]. [cited 2017 october 25]. available from: https://www.rdc-drc.ca/the-rdc-federated-pilot-for-data-ingest-and-preservation/ about the author tim hutchinson has been an archivist at the university of saskatchewan since 1997. he was appointed as university archivist in 2004, and as head of university archives & special collections in 2013 (he is currently on sabbatical); and has been active in a range of activities and developments in the areas of digital archives and preservation, digitization, archival descriptive standards, and technology for archives more generally. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – a practical method for searching scholarly papers in the general index without a high-performance computer mission editorial committee process and structure code4lib issue 58, 2023-12-04 a practical method for searching scholarly papers in the general index without a high-performance computer the general index is a free database that offers unprecedented access to keywords and ngrams derived from the full text of over 107 million scholarly articles. its simplest use is looking up articles that contain a term of interest, but the data set is large enough for text mining and corpus linguistics. despite being positioned as a public utility, there is no user interface; one must download, query, and extract results from raw data tables. not only is computing skill a barrier to use, but the file sizes are too large for most desktop computers to handle. this article will show a practical way to use the gi for researchers with moderate skills and resources. it will walk though building a bibliography of articles and a visualizing yearly prevalence of a topic in the general index, using simple r programming commands and a modestly equipped desktop computer (code is available at https://osf.io/s39n7/). it will briefly discuss what else can be done (and how) with more powerful computational resources. by emily cukier introduction the general index (gi) is a massive corpus of text data derived from published articles; its stated purpose is to open scholarly literature to public inquiry. the gi contains tables of keywords [1], ngrams, and associated metadata extracted from a corpus of over 107 million journal articles drawn from general and specialized journals [2]. this enables full-text searching of a wide swath of academic literature in a single shot, without limitation by vendor platform or paywalls. the gi is the output of a mass journal article accumulation project spearheaded by open information advocate carl malamud and conducted at jawaharlal nehru university (jnu) in new delhi [3]. public resource, a nonprofit founded by malamud, released the first public version of the general index in october 2021 [4]. as of this article’s publication, it contained articles published as recently as 2020. the gi could be a powerful aid to librarians and digital humanists for tracking concepts through scholarly literature. one clear application is in literature review and systematic searching: one can query search terms simultaneously across more articles and disciplines than contained in even the largest vendor discovery platforms, and without permission to access full text [5]. with minimal manipulation one can link term instances to publication years, enabling diachronic analysis of word usage over time. this allows for visualization of trends in journal articles, much like google ngram viewer does for books. but unlike google, the gi links keyword and ngram data to the document of origin, which enables identification and retrieval of the works. a substantial obstacle to using the gi is lack of an exploratory interface. the user must download the raw data tables, then devise methods for querying the records. even though the tables are split into slices, reading them requires more disk space and memory than found in typical desktop computers. with some deliberate programming choices and tricks, it is still possible to use portions of the gi in a limited computing environment. this paper will present one workflow for querying gi keywords in a desktop computing environment using the r programming language. it will show how to create a bibliography of literature on a topic of interest and plot its yearly prevalence in gi articles. it will also discuss adapting the method to a high-performance computing environment, which enables analysis of gi ngrams. contents the general index comprises three data tables of keywords, ngrams, and metadata parceled into sixteen slices apiece and formatted as .sql files. slices are compressed to facilitate downloads. the compressed files take up a total of 4.7 terabytes of space and expand to a total of 37.9 terabytes. table 1. general index compressed and expanded file sizes compressed slices compressed total unzipped slices unzipped total metadata – single file enhanced (version 10.21) n/a 24.2 gb 4.2-4.4 gb 70 gb keywords (version 10.18) 21-23 gb 355 gb 95-102 gb 1.6 tb ngrams (version 10.18) 262-288 gb 4.5 tb 2.1-2.3 tb 36 tb records are allocated to slices according to the first digit of a field labeled “dkey”. the dkey is a unique identifier for each article consisting of 40 hexadecimal digits determined by applying a hash algorithm to the file that contains it. this common field enables linking keyword and ngram records to their corresponding metadata (table 2). file names are standardized to include the data type (“keywords”, “ngrams”, “info” or  “meta”) and dkey first digit – e.g., the 11th slice of ngram data is named “doc_ngrams_b.sql”. ngrams (1-5 grams) were determined using spacy, a cython-based natural language processing tool. keywords were determined using the python package yake (yet another keyword extractor) [6]. yake ranks the importance of words within a document according to text features and statistics like word frequency, position in the document, relation to context, and capitalization. the keyword and ngram tables each contain fields for the keyword or ngram found, its lower-case equivalent, and number of tokens (words) it contains. each also has a field that describes the term’s representation within the article: the yake score for keywords (smaller values correspond to greater importance), and term frequency (number of occurrences in the document) for ngrams. the metadata table contains metadata derived from both outside the file (document metadata) and extracted from full text. of particular interest for citation tracking and analysis are fields for doi, isbn, journal title, article title, publication date, and authors. a partial list of fields and their descriptions can be found in the gi readme file. table 2. selected general index metadata fields keywords ngrams metadata dkey dkey dkey keywords ngram doi keywords_lc ngram_lc isbn keyword_tokens ngram_tokens journal keyword_score ngram_count title term_freq pub_date author the corpus is heavily weighted towards recent publications, with the bulk of the articles with determinable publication years (39%) (45844188 of total) published in 2017-2019. five percent of articles did not have readily extractable publication years. 1544 (0.001%) had publication years falling outside of the feasible values of 1600-2020. figure 1. number of articles per year. figure 2. number of articles per year, logarithmic scale. the log scale makes several features of the data visually apparent: the absolute numbers during lower-volume publication years, the onset and duration of an exponential growth phase, and the publication burst in the most recent few years. the earliest publications in the gi (with realistic publication years) were produced in the 1660s. thence forward, the gi contains on the order of 100 articles per year until 1880, when the article volume shows steady exponential growth through 2016. article volume jumps by orders of magnitude in 2017. low article volume in 2020 suggests that article accumulation ended in this year. local workflow: literature indexing the following diagram demonstrates the steps for compiling bibliographic information and determining yearly prevalence of a term in the gi. figure 3. gi workflow diagram. functions and packages are shown in italics. i performed these steps on a hp computer running windows 10 with a 3.4 ghz intel i3 cpu, 8 gb ram, 256 gb internal hard drive, and an external 1 tb hard drive. scripts were written in r and run in rstudio. part i: file preparation download & unzip general index files are available for download at https://archive.org/details/generalindex. downloading each keyword file and the single enhanced metadata file took roughly 6 hours apiece at a download speed of approximately 1 mb/sec. the zipped gi files require decompression before manipulation. to unzip the combined metadata file, i used the free utility 7-zip (available at https://www.7-zip.org) twice: once to unpack the file from .gz to .tar, and again to yield a set of 16 .sql files. to prepare the keyword files, i created an rstudio script that would individually decompress each file, search for terms of interest, retain the matches, and remove the expanded file before decompressing the next. this limited the amount of disk space needed to store keyword files to 457 gb at any given time (355 gb for all zipped slices + 102 gb for the largest uncompressed file). if disk space is truly at a premium, it would be possible to download a single keyword file at a time, unzip it, extract records of interest, and delete the file before moving on to the next. part ii: compiling a bibliography term search drosera is a genus of carnivorous plants (commonly known as sundews) found on six continents [7]. to demonstrate literature indexing, i chose to search the gi keyword files for mentions of sundew or drosera. the smallest decompressed keyword file was about 95 gb, which was too large to read into rstudio. however, the r function fread() (part of the data.table package) can presearch a structured file and read in only the lines that match a particular pattern. this is done using the element “cmd” to execute an operating system command on the file before importing it into the rstudio environment. in this case, i used the pattern-searching command grep [8] to select lines that contained either ‘drosera’ or ‘sundew’. r could readily handle this data subset. to reduce exclusion of valid hits, i used a case-insensitive search and placed no restrictions on characters before or after each search string. # -------creating a bibliography for search terms-----------# ## ---------unzip each file and read in keyword hits -------## keyword_path = "d:/gi_files/keywords" keyword_files = dir(keyword_path, pattern="*.sql.zip", full.names = true) keyword_headers = c("dkey", "keywords", "keywords_lc", "keyword_tokens", "keyword_score", "doc_count", "insert_date") keyword_tbl <data.frame() # -i makes the grep query case-insensitive # -e makes the grep query an inclusive or query = str_c("-i -e 'drosera' -e 'sundew'") for (i in keyword_files) { expanded_file <unzip(i, exdir="d:/gi_files") command = str_c("grep ", query, " ", expanded_file) current_tbl <fread(cmd = command, encoding="utf-8", quote="", header=false, col.names = keyword_headers, fill=true, keepleadingzeros=true) keyword_tbl <rbind(keyword_tbl, current_tbl) file.remove(expanded_file) progress <str_c("file ", i, " finished: ", sys.time()) print(progress) } write_tsv(keyword_tbl, "sundewkeywords.tsv") ## ------remove spurious hits, remove repeat articles ----## dkey_list <keyword_tbl %>% filter(!keywords %in% "sundewall") %>% distinct(dkey) the algorithm took roughly 50 minutes to process each slice (30 for unzipping and 20 for term matching), less than 14 hours total. it retrieved 9675 matching keyword records across the slices, which required less than 1 mb of disk storage to save as a .tsv file. at this stage it is prudent to manually inspect a sample of records to ensure the query behaves as intended. after looking at the results in openrefine [9], i added lines of code to filter the surname “sundewall”, a spurious hit caught by my original query. i also included a line to remove duplicate instances in the dkey field. duplicate article matches can occur even when searching for a single word, as gi keyword records may include the same term on its own and as part of one or more phrases. after these filtering steps, 4,565 unique articles remained. match metadata the next steps to turn this occurrence list into a useful index are to load in the gi metadata corresponding to the identified articles, using the common “dkey” field. i was unable to load the >4gb metadata slice files into rstudio due to memory limitations. instead, i made similar use of fread() and grep as with the keyword files, here reading in only the metadata records with dkey identifiers that matched the keyword hits. since the search for each individual dkey occurred serially, this took considerable time – about 36 hours for the 4,565 unique sundew dkeys. metadata were found for 4336 (95.0%) of them. ## ------read in matching article metadata ---------------## metadata_path = "d:/gi_files/metadata/doc_info.2022.12.update/doc_meta/doc_meta/" meta_headers = c("dkey", "raw_id", "meta_key", "doc_doi", "meta_doi", "doi", "doi_flag", "doi_status_detail", "doi_status", "isbn", "journal", "doc_title", "meta_title", "title", "doc_pub_date", "meta_pub_date", "pub_date", "doc_author", "meta_author", "author", "doc_size", "insert_date", "multirow_flag") term_metadata <data.frame() dkeys_by_digit <split(dkey_list, f=substr(dkey_list$dkey,1,1)) slice_list <c(0:15) for (slice in slice_list) { file_path = str_c(metadata_path, "doc_meta_alt_", as.hexmode(slice), ".sql") dkey_vec <as.data.frame(dkeys_by_digit[[slice+1]], col.names="dkey") for (i in dkey_vec$dkey) { query = str_c("-m 1 '^", i, "'") command = str_c("grep ", query, " ", file_path) current_tbl <fread(cmd = command, encoding="utf-8", quote="", header=false, col.names = meta_headers, sep="\t", colclasses="character", fill=true, keepleadingzeros=true) term_metadata <rbind(term_metadata, current_tbl) } ## optional show progress bar for each slice progress <str_c("slice ", as.hexmode(slice), " finished: ", sys.time()) print(progress) } table 3. first ten records of gi papers containing the terms “drosera” or “sundew”, reproduced as-is. some metadata elements are hidden to improve readability. the full set is available at https://osf.io/s39n7/. title author pub_date journal isbn doi edit page 20 n/a 8/7/2015 \n \n n/a mycorrhizal status of knieskern’s beaked sedge (rhynchospora knieskernii) in the new jersey pine barrens john dighton, ted gordon, rosa mejia and marilyn sobel 2013 bartonia 10.2307/41955392 reviews of british and foreign literature. the natural history of digestion a lockhart gillespie; london; walter scott 2/16/2019 \n \n n/a notes on the flora of block island w. w. bailey 1893-06-17 bulletin of the torrey botanical club 10.2307/2475918 cytological analysis of ginseng carpel development silva, jeniffer; kim, yu-jin; xiao, dexin; sukweenadhi, johan; hu, tingting; kwon, woo-saeng; hu, jianping; yang, deok-chun; zhang, dabing 2/2/2017 protoplasma 10.1007/s00709-017-1081-4 caryophyllales: a key group for understanding wood anatomy character states and their evolutionb oj_1095 342..393 sherwin carlquist fls 8/11/2017 \n \n n/a frederick burkhardt; james secord;\\r          <i>et al.</i>\\r              (editors).\\r              <i>the correspondence of charles darwin</i>\\r          . volume 20. xxxix + 862 pp., illus., bibl., index. cambridge: cambridge university press, 2013. â£90 (cloth).frederick burkhardt; james secord;\\r          <i>et al.</i>\\r              (editors).\\r              <i>the correspondence of charles darwin</i>\\r          . volume 21. xxxvii + 784 pp., illus., bibl., index. cambridge: cambridge university press, 2014. â£90 (cloth). ghiselin, michael 12/2/2015 isis 10.1086/684657 structure elucidation of plumbagin analogues from\\r          <i>dionea muscipula</i>\\r          and their\\r          <i>in vitro</i>\\r              immunological activity on human granulocytes and lymphocytes kreher, b.; neszmelyi, a.; polos, k.; wagner, h. 1989-2 planta medica 10.1055/s-2006-961890 trading in genes. development perspectives on biotechnology, trade and sustainability van damme, patrick 2006-12 economic botany 10.1663/0013-0001(2006)60[394b:tigdpo]2.0.co;2 flowers of july and august edward h. day 1879-08 the art amateur 10.2307/25626849 the article metadata show inconsistent formatting, sporadic inclusion of markup, and missing information. nonetheless, in most cases there is sufficient information to identify each article. one could ask what advantages this method has over searching for papers in google scholar. the analogous query (drosera* or sundew*) yielded 6,140 results (search performed on jun 2, 2023) – admittedly a larger number than found in the general index. however, google scholar records cannot be exported in bulk. also, not all articles and citations in google scholar have searchable full text [10]. since neither database fully encompasses the scholarly literature, searching the general index could complement searches in google scholar. part iii: yearly term prevalence select publication dates reporting the number of articles per year containing a given term is not useful for determining the term’s prevalence over time, as it does not account for the total number of articles in the general index for that year. finding the total is challenging in a limited computing setting, because each single metadata slice is too large to read fully into r. i overcame this limitation with another application of fread(). i used a command to accomplish two tasks on each line of the file before reading in the data: select lines that contain metadata records (to avoid reading in sql header information) and read in only the columns needed for the current task. to accomplish this, i settled on a statement in the awk [11] programming language [12]. the statement selects lines that begin with a 40-character hexadecimal number (corresponding to a valid dkey), then returns the fields corresponding to the dkey and publication date [13]. this allowed reading in dkey/publication date pairs at a rate of 2-3 minutes per slice, or about 40 minutes total. # -----calculate yearly term prevalence --------------# ## ------get years for all metadata -----------------## date_headers = c("dkey", "pub_date") slice_list <c(0:15) count_tbl <list() for (slice in slice_list) { file_path = str_c(metadata_path, "doc_meta_alt_", as.hexmode(slice), ".sql") expression = str_c("/^[0-9a-f]{40,}/") # find lines starting with 40 hex characters command = str_c("awk ' begin {fs=ofs=\"\t\"} $1 ~ ", expression, " {print $1, $17} ' ", file_path) current_tbl <fread(cmd = command, encoding="utf-8", quote="", header=false, col.names = date_headers, sep="\t", fill=true, keepleadingzeros=true) # detect four consecutive integers in the publication date field to determine year current_tbl <current_tbl %>% mutate(pub_year = str_extract(pub_date, '[0-9]{4,}')) count_tbl[[slice+1]] <count(current_tbl, pub_year) ## optional show progress bar for each slice progress <str_c("slice ", as.hexmode(slice), " finished: ", sys.time()) print(progress) } meta_dates <do.call("rbind", count_tbl) %>% count(pub_year, wt=n) i encountered memory constraints when trying to combine the dkey/publication date information across all slices. instead, i had the script count the number of gi articles for each year for each slice individually without retaining the raw dkey/publication date information, and later totaled the year counts across slices. spurious hits are unlikely at this stage. the dkey identifier is assigned to articles by applying a hash algorithm to yield a 40-digit hexadecimal number. thus, there are 1640 (1.5 * 1048) possible unique results of this hash. the probability that the set of 1.1*108 articles contains at least one pair of different articles with the same hash is roughly 3 * 10-23 [14]. determine year the gi metadata table contains three fields with information about publication year: doc_pub_date (publication date extracted from the document text), meta_pub_date (publication date given in document metadata), and pub_date (date from metadata if available, text if not). i chose to determine dates from the pub_date field to give the greatest likelihood of successful extraction. date formats are not standardized in the gi – months and days are not always given, and the order and separating characters may vary. i focused on the four-digit year to avoid confusion among ways to format dates and provide a convenient unit for diachronic analysis. for both the metadata and keyword bibliographic data, i used the r stringr function str_extract() to detect when four consecutive digits were present in the pub_date field and add them to a new field, pub_year. count & normalize the rstudio package dplyr contains convenient functions to count() occurrences in data columns and mutate() columns to perform arithmetic operations. i used count() on the pub_year field to determine the total number of articles per year in the gi, and number of articles per year in the keyword hit subset. i joined these results into a new table by the common field of publication year, then calculated a new field called “prevalence” by dividing the count of articles containing the term by the count of articles in the gi for that year. ## ------determine term publication years, ## -----count and normalize -------------------------## term_pub_year <arranged_results %>% mutate(pub_year = str_extract(pub_date, '[0-9]{4,}')) %>% count(pub_year) %>% left_join(meta_dates, join_by(pub_year), suffix=c(".t", ".m")) %>% mutate(prevalence = n.t/n.m) %>% arrange(pub_year) visualize  ## ------plot prevalence by year --------------------## prevalence_plot <ggplot(data=term_pub_year, aes(x=as.integer(pub_year), y=prevalence)) + scale_y_break(c(0.0015, 0.013), space = 0.8) + labs(x="publication year", y="prevalence (percentage)") + geom_point() prevalence_plot plotting the prevalence term over time is a proxy for interest in the term over time. the plot of sundews shows the species has not had a resurgence of interest since a series of upticks in the 1870s and the early 1900s. figure 4. prevalence of the terms “sundew” or “drosera” in the general index. starting from 1850 onward, sundews see the greatest representation around 1875, with spikes of interest around 1910 and 1925. from then their prevalence stays relatively consistent, suggesting little special attention paid beyond a baseline interest. earlier data are more noisy, and sparse hits before 1825 suggest too little data to interpret. these results may appear similar to a google ngram viewer plot, but they measure publication volume in different ways, making direct comparison inappropriate. in the above, prevalence shows the proportion of articles in the gi containing at least one mention of the desired term. google ngram viewer plots frequency, the proportion of occurrences of a word/term normalized by total number of words in the corpus per year. a more comparable method is to download raw data from the google ngram corpus, which allows a similar calculation of yearly prevalence – as the proportion of volumes in google books containing a term, compared to the total volumes in the google books corpus. however, this is only possible if looking for a single term of interest; in the example above, volumes containing both “sundew” and “drosera” would be double-counted. hpc adaptations i performed my first explorations of the general index on kamiak, the washington state university high performance computing cluster. only after becoming familiar with the data set and potential workflows was i able to devise the local approach described above, which i hope will be accessible to more researchers. working in a high-performance computing environment confers multiple advantages that boil down to: doing more things, faster. an hpc can manipulate the ngram files, run multiple queries in parallel, and accomplish individual tasks faster than less powerful computers. scripts used to process and analyze the data can be more straightforward, as they do not require memory restriction workarounds. figure 5. hpc workflow for creating a yearly ngram prevalence plot kamiak includes 500 gb of storage space per lab with a free-tier account and temporary (up to two weeks) scratch storage of 10 tb. access to this scratch storage let me serially download, unzip, and query ngram slices, then save the results to permanent storage. kamiak runs a linux operating system that i accessed using the windows subsystem for linux ubuntu application. in this environment one can use scripts to perform simple file and table manipulations, perform multiple tasks in sequence, and run tasks in parallel using job arrays. this makes it easy to search for multiple keywords or ngrams at the same time and later combine and compare the results. i downloaded files directly from the gi using the linux command wget and extracted them with the commands unzip (for .zip files) and tar (for .tar.gz metadata files). in a series of bash scripts, i appended metadata years to all the metadata records, used grep and regular expressions to select ngram records that contained my search terms of interest. i used r scripts similar to those described in the desktop sections to count occurrences of publication years, normalize by article years across the corpus, and plot the results. i ran the r scripts on kamiak when the search result files were too large to read into rstudio locally. i intend to present more details on these processes in a future publication. discussion this paper shows that working with the general index is feasible in a desktop computing environment, so long as we: limit the analysis to the keyword and metadata files retrieve a sufficiently small number of keyword hits employ r packages that can selectively read subsets of file data into memory. applied to the gi, these methods can create bibliographies and diachronic analyses of keywords used throughout the history of academic publishing. though beyond the scope of this work, one could imagine performing a more nuanced analysis by incorporating the yake (“meaningfulness”) score to include a dimension of relative importance of the keyword in each article. if working in an hpc environment, analyzing ngrams would allow querying longer phrases and greatly increase the number of results retrieved – hits would not be limited by their importance in the document. using these longer phrases would also facilitate the use of additional text and data mining analytic methods, such as colocation and topic modeling. this paper presents one method for searching the general index, but it is by no means the only way. all the tasks i have described are straightforward [15] file and text manipulations confounded by the size of the data set. any software or method that can perform similar manipulations is fine for the purpose. i hope this paper will inspire others to explore and experiment with this resource. for those that do, here are a few considerations: when possible, subdivide large data sets and work on smaller chunks. test code on very small data subsets to avoid long wait times to verify the desired effect. strive for consistency in file formats and encoding. choose conventions for importing and exporting files (e.g., utf-8 encoding, tab-separated file format), and use them throughout the workflow. general index files can also be downloaded via torrent, which could be faster and more resilient to interruption. local campus policy prohibited me from testing this. the r package sqlite may offer other methods and shortcuts for querying large files in r. i did not test this approach due to unfamiliarity. admittedly, the limitations of the general index go beyond its lack of interface. even my sample metadata records show a substantial lack of data cleaning, missing fields, and inconsistent formats and standards. five percent of article identifiers in the sundew sample lacked corresponding metadata altogether. one should take this uncertainty into account when choosing analyses to conduct and interpreting results. there is also little transparency in how well the articles in the gi constitute a representative sample of the entire universe of published articles. it is possible to determine the distribution of publication dates, but it is much harder to characterize the articles in other ways. articles in languages other than english are present but sparse. the methods used to collect the articles could overor underrepresent different article types, disciplines, or knowledge domains. it is difficult to infer the direction and possible extent of bias without more information about the origins of the articles within the gi, and this limits the extent we can responsibly interpret and draw conclusions from the data. the stability of the resource is also unclear, as the database’s legality has not yet been tested. malamud maintains that the gi does not infringe copyright despite containing information derived from paywalled papers, because the text is only retrievable in short fragments [16]. however, the legality of amassing the articles in the first place is unclear [17]. currency is also a factor in the gi’s utility. according to carl malamud, the database is planned to be updated yearly at most [18]. these caveats are substantial, but i still consider the gi highly valuable because it opens up the scholarly record to an extent of inquiry that was previously impossible, and does so freely to the public. it can aid in comprehensive literature searching, following concepts over time, and text mining. by focusing on local computing workflows, i hope this paper can reach a broad and diverse audience of researchers, who will in turn ask a broad variety of questions of the scholarly record. i am eager to collaborate with others interested in working with this database and happy to offer advice for using it. acknowledgements this paper would not be possible without the hard work of carl malamud and public resource to create the general index in the first place, and making it available to the public. i offer many thanks to elizabeth blakesley and jay starratt for encouraging me to follow my research interests, wherever they may roam. thanks also to elizabeth blakesley, suzanne fricke, ann dyer, clara del junco, and xanda schofield for helpful conversations on this work and resource, and all my wsu library colleagues for moral support. many thanks to the kamiak hpc/wsu center for institutional research computing (circ) https://hpc.wsu.edu for unhesitatingly providing access to the hpc cluster that enabled this work and for kind troubleshooting assistance. this work was not supported by grant funding. about the author emily cukier (orcid id: 0000-0002-3972-5506) has been a science librarian at washington state university since october 2021. references and notes [1] in this context, “keywords” do not refer to key terms/phrases assigned by an author or editors to aid finding, but words or short phrases identified by an importance-scoring algorithm. [2] the general index. 2021. [accessed 2023 jul 7]. http://archive.org/details/generalindex. [3][5][17] pulla p. 2019. the plan to mine the world’s research papers. nature. 571(7765):316–318. doi:10.1038/d41586-019-02142-1. [4][16] else h. 2021. giant, free index to world’s research papers released online. nature. doi:10.1038/d41586-021-02895-8. [accessed 2023 apr 24]. https://www.nature.com/articles/d41586-021-02895-8. [6] campos r, mangaravite v, pasquali a, jorge a, nunes c, jatowt a. 2020. yake! keyword extraction from single documents using multiple local features. inf sci. 509:257–289. doi:10.1016/j.ins.2019.09.013. [7] barthlott w, ashdown m. 2007. the curious world of carnivorous plants : a comprehensive guide to their biology and cultivation. english language ed. portland, or: timber press. [8] the grep command is used with regular expressions and can perform far more sophisticated pattern-matching than is described here. one good tool for learning and experimenting with regular expressions is available at https://regexr.com. [9] openrefine is a free data viewing and wrangling tool available at https://openrefine.org/. [10] lopez-cozar ed, orduna-malea e, martin-martin a, ayllon jm. 2018. google scholar: the “big data” bibliographic tool. doi:10.48550/arxiv.1806.06351. [accessed 2023 may 30]. http://arxiv.org/abs/1806.06351. [11] awk (named after creators alfred aho, peter weinberger, and brian kernighan), is a scripting language highly suited for pattern matching in structured data. [12] aho av, kernighan bw, weinberger pj. 1988. the awk programming language. reading, mass: addison-wesley pub. co. (addison-wesley series in computer science). [13] these fields correspond to the first and seventeenth columns in metadata file version 10.21. the field indexes may require updates if later metadata file versions change the order or number of columns. [14] calculation performed at https://www.bdayprob.com with d=1640 (entered as 2128) and n=1.1*108 (approximated and entered as 227) [15] i admit that straightforwardness is in the eye of the beholder. [18] carl malamud, personal communication subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – ontology for voice, instruments, and ensembles (onvie): revisiting the medium of performance concept for enhanced discoverability mission editorial committee process and structure code4lib issue 54, 2022-08-29 ontology for voice, instruments, and ensembles (onvie): revisiting the medium of performance concept for enhanced discoverability medium of performance—instruments, voices, and devices—is a frequent starting point in library users’ search for music resources. however, content and encoding standards for library cataloging have not been developed in a way that enables clear and consistent recording of medium of performance information. consequently, unless specially configured, library discovery systems do not display medium of performance or provide this access point. despite efforts to address this issue in the past decade in rda, marc, and the linked data environment, medium of performance information continues to be imprecise, dispersed across multiple fields or properties, and implied in other data elements. this article proposes revised definitions for “part,” “medium,” “performer,” and “ensemble,” along with a linked data model, the ontology for voice, instruments, and ensembles (onvie), that captures precise and complete medium of performance data reflecting music compositional practices, performance practices, and publishing conventions. the result is an independent medium of performance framework for recording searchable and machine-actionable metadata that can be hooked on to established library metadata ontologies and is widely applicable to printed and recorded classical, popular, jazz, and folk music. the clarity, simplicity, and extensibility of this model enable machine parsing so that the data can be searched, filtered, sorted, and displayed in multiple, creative ways. by kimmy szeto introduction medium of performance—the instruments, voices, and devices at the library users’ disposal—is a frequent starting point in their search for music resources, and an identifying element of musical works and expressions (ostrove, 2001). however, the limited availability and functionality of medium of performance as an access point has persisted from the physical card catalog, through the marc era, into current developments of linked data ontologies. uniquely important to music resources, medium of performance refers to the tools involved in expressing a musical work. historically, the term “medium” is not well defined in cataloging rules, and some instructions and practices are incompatible with linked data modeling practice today. medium of performance has been recorded as supplemental information in the title area (coyle, 2011) and in the subject area (subject analysis committee, 2017), and did not receive much attention as an independent data element until the release of the current generation of library cataloging standard resource description and access (rda) in 2010. inheriting the basic outline from earlier standards, rda instructs catalogers to record a list of instruments, voices, and ensembles, followed by the number of parts and total number, and, when necessary, further identifying characteristics. cataloging practice continues to treat medium of performance as a single entity. the reality, however, is not as simple. tracing a musical work from its composition to performance, subsumed in what is broadly termed “medium of performance” is a network of relationships involving the composer (musical parts), the publisher (published scores and parts), the instruments/voices (mediums), the performers, and the ensembles. from the library users’ standpoint, the single list of instruments/voices and numbers in the data does not always match the score and parts in the actual publication, and often does not provide enough information about the specific instruments, devices, and players required for planning a reading or a performance. while these more complex relationships pose a challenge for the marc format to encode, linked data models offer the capability. linked data —a set of technologies and practices that foster publishing and connecting structured data on the web— have been increasingly embraced in the past decade by the library community as a means to provide open access to its richly curated bibliographic catalogs. although some attempts have been made to refine the medium of performance element in linked data models, such as in the performed music ontology (pmo), so far, no library linked data model has been built out beyond alignment with rda/marc format and conversion of existing data. in the context of expanding the conception of the medium of performance element, this article lays out broadened definitions for “part,” “performer” and “ensemble” as described in szeto (2017), proposes a revised definition for the term “medium,” and, based on these revised definitions, presents a new data model, the ontology for voices, instruments, and ensembles (onvie). onvie gives medium of performance additional dimensions by simultaneously capturing and relating the composer’s intended performing forces, published parts, specific instruments, voices, and devices required/used, as well as specific performers. the clarity, simplicity, and extensibility of this model enable more nuanced machine parsing so that the data can be searched and displayed in multiple, creative ways. modeling medium of performance this paper will first discuss the development of onvie in the context of rda, the current content standard, as encoded in marc field 382. the next section will turn to the performed music ontology, an extension for music resources specifically developed as an extension for bibframe, the general linked data model being developed for library bibliographic data to replace marc. recent cataloging practice prior to the 2000s, the tripartite division of author, title, and subject of the card catalog resulted in cataloging rules and decades of practices that embedded medium of performance information in the uniform title when it is necessary to distinguish between identical titles and in subject heading form subdivisions (elmer, 1960). the marc format did offer field 048 for medium of performance, but the fields held only codes and numbers for voices, instruments, and ensembles, which were not readily decipherable to catalogers and users alike. this coded field fell into disuse, and users mainly relied on a free text note and clues from the title and subject headings. the dispersal of structured data complicated display and indexing, resulting in a limited ability for users to search directly or filter medium of performance search results (subject analysis committee, 2017). this is particularly problematic for compilations, vocal music, folk music, jazz, and recorded popular music where medium of performance is generally not explicitly stated bibliographically (newcomer et al., 2013, section ii.d). these limitations led to coordinated efforts to raise the visibility of this data element. in 2007, the library of congress, collaborating with a range of stakeholders, initiated developments in three areas: a dedicated faceted vocabulary library of congress medium of performance thesaurus (lcmpt) was launched in 2014 (library of congress, 2014), a new marc field 382 for an expanded encoding of medium of performance data based on rda instructions was established in 2010 (library of congress, 2020), and programmatic changes that extracted existing authority data from the uniform title to be placed into the 382 field took place on worldcat (library of congress, 2012). with these three areas in place, programmatic retrospective implementation of faceted vocabulary in both authority and bibliographic records continues to this day (mullin, 2018; subject analysis committee, 2022). below is an excerpt from a marc record on worldcat (oclc number 989164116) that illustrates how medium of performance information is recorded in the uniform title (field 240) and the subject headings (field 650) and coded in fields 048 and 382. it might not be immediately apparent, from the way the uniform title and subject headings are constructed and structured, that this record is based on a musical score for solo harpsichord with piano accompaniment. (there is no string orchestra!) what might (or might not!) help library users is the note in field 500, a phrase typically used for this situation in music cataloging practice. unfortunately, the note is not present in the actual worldcat record, but was added here by the author to illustrate the difficulties posed to human readers and computer algorithms alike. the issue with clarity of semantics was identified and critiqued in coyle (2011), and the retrospective implementation of lcmpt and field 382 to address this issue was detailed in mullin (2018). 048 ## $b kc01 $a ka01 240 10 $a concertos, $m harpsichord, string orchestra, $n bwv 1052, $r d minor; $o arranged 382 01 $b harpsichord $n 1 $a piano $n 1 $s 2 $2 lcmpt 500 ## $a acc. arr. for piano. 650 #0 $a concertos (harpsichord with string orchestra) $v solo with piano. data modeling considerations while these efforts have enabled structured medium of performance data in bibliographic records using lcmpt as the controlled vocabulary, the implementation remains less than ideal. rda inherited the broad outlines from earlier standards—”the instrument, instruments, voice, voices, etc. for which a musical work was originally conceived” (rda 6.15.1.1), and then “record each instrument…” (rda 6.15.1.4) or “record an/the appropriate term…” that groups instruments and voices by family or into an ensemble (rda 6.15.1.6 to 6.15.1.10). these terms, depending on their characteristics, are encoded in marc 382 subfields $a, $d, $d, and $p. after each term, rda instructs the cataloger to record the number of parts (rda 6.15.1.3). this number is then encoded in marc 382 subfields $e, $n or $r, which are defined for the number of performers or ensembles. in addition, marc 382 offers subfields $s and $t for the total numbers of performers and ensembles. this practice creates some data subfields that are not unambiguous in some cases and not atomic in some others. in the course of resolving ambiguity and atomicity issues, two fundamental principles of linked data design for the semantic web, it became apparent that separate definitions for part, medium, performer, and ensemble could offer the overarching solution. ambiguity and atomicity issues the rda instruction to record the number of “parts” which is then encoded in the “number of performers” subfield $n in marc field 382 creates a semantic ambiguity. the onvie model will treat part and performer as two separate concepts. another conflation occurs in the usage of terms such as “percussion” and “continuo” as mediums, when these mediums are actually parts referring to a group of instruments. the onvie model will provide clarity by allowing the individual instruments to be linked to these parts. another ambiguous practice is conflating individual instruments/voices with ensembles, and encoding them in the same marc subfields, even though ensembles are not themselves instruments or voices. however, it is not always clear what exactly forms an ensemble. is the string quartet an ensemble of instruments? is it a group of performers? or is it a group of parts? in the onvie model, individual parts are used as the starting point, and a group of parts will form an ensemble. this way, all of the ensemble’s constituent parts will always be known, as will the mediums be linked to each part. separate instructions are given in rda for music intended for one performer to a part as opposed to more than one performer to a part. the attributes of “solo” and “accompaniment” are also treated separately. however, these are not characteristics of the mediums, but are characteristics of the parts. (for example, the instrument violin itself cannot possess the quality of “solo” but a violin part can.) in onvie, the number of performers to a part, the solo status, and the accompaniment status will all be treated as refinements to the part, rather than as separate classes of entities. more specifically, rda instructs the cataloger to omit the accompanying keyboard instrument (such as a piano) in a classical song for solo voice (rda 6.15.1.11). while the piano is not an unreasonable assumption among musicians, such an arbitrary exception causes the discovery system to return incomplete results, which could complicate a library user’s search, especially for a vocalist looking for repertoire regardless of the accompaniment. special issue: number of hands rda instructs the “number of hands,” if other than two, to be recorded (rda 6.15.1.5.1). this poses an impossibility in the encoding mechanism in marc field 382. while the number of performers is encoded in subfield $n, there is nowhere to encode the number of instruments, or which hand is being used. while “piano, 4 hands,” where two pianists use all four hands on a single piano, is a fairly common genre, piano duets—two pianos and two pianists—is not uncommon either, especially in more recent repertoire with the two pianos tuned a quarter tone apart (for example, chiaroscuro for two pianos (one tuned down a 1/4 tone) by john corigliano (published 2011)). yet these are encoded identically as “$a piano $n 2,” and library users will need to inspect the free text notes, the subject heading, or the title. suppose a piece of music was composed for two pianos with two players, each using only the left hand, or, suppose another piece of music was composed for two pianos with two players, one using both hands, the other using only the left hand, the result is still the same: “$a piano $n 2.” figure 1, adapted from szeto (2016, slide 22), further illustrates this issue with combinations of two pianos and six hands. meanwhile, a piece of music composed for two pianos and one player (for example, trois hommages for 2 pianos (tuned a quarter tone apart) 2 hands by georg friedrich haas (published 2009)) would be encoded as “$a piano $n 1,” identical to solo piano music (one piano, one player). figure 1. combinations of pianos and hands that cannot be differentiated in rda descriptions or encoded in marc field 382. at present, when these distinctions, or any other distinctions discussed above, cannot be expressed in the marc 382 field, the cataloger is instructed to provide clarification in a free text note in subfield $v. in the onvie model, separate treatments for part and performer will remove this issue. additional rda medium of performance characteristics rda includes instructions for three more attributes: pitch and range of instruments, doubling, and alternative. pitch and range the “pitch” of the instrument here refers to musical instruments that are “transposed,” where playing with the same set of fingering and sound production techniques on the instrument yields a different set of pitches (for example, clarinet in b-flat). what rda referred to as “range” refers to instruments that come in common sizes (for example, the saxophone family includes soprano, alto, tenor, and baritone). pitch and range are recorded only when the cataloger considers them important for identification and access (rda 6.15.1.5.1) and appears in marc field 382 as a free text note in subfield $v. the onvie model will include further refined properties for these, applied separately to parts and mediums. doubling instrumental doubling refers to a musical part that instructs a single performer to use more than one instrument. when recorded, the medium is encoded in marc 382 subfield $d. this structure seems straightforward for a traditional orchestral doubling, such as a flute player who also plays the piccolo. this is not so straightforward for other parts that are less clearly defined, for example, an orchestral percussion part where multiple players share a set of percussion instruments, or where a timpani player occasionally joins other percussionists (timpani is recorded separately from percussion). currently, the instruction is to record the term “percussion” alone. neither rda nor marc offers a solution for this entanglement between performers, mediums, and parts. separating them in the onvie model will remove this issue. alternative rda instructions for “alternative instruments” (rda 6.15.1.5.3) invite further refinement. in current practice, “alternative” is used for a range of situations where one or more mediums deviate from the original. while some composers expressly indicate that the identical part may be used on more than one instrument (for example, the “flute or violin” part in the jazz composition out of the cool by david heath (as published in 1986)), in most cases, what are also considered alternatives range from slightly different (for example, clarinet music transcribed for the viola; see swanson, 2003, pp. 13-15 for the process of compiling the repertoire list which could have been aided by a direct medium of performance search), to quite different (for example, a continuo part realized for piano), to drastically different (for example, orchestra music in operatic works arranged for piano with no change in the vocal parts). the onvie model will be able to precisely indicate the first case where the part is identical, and a hook is provided for future extensions that capture the various degrees of change. there are also situations where the alternative involves a role change in one or more parts (for example, the arrangements of the scherzo movement in robert schumann’s second symphony (as published in szeto, 2010a and 2010b) where the first violin part could be a solo part or an ensemble part; the arrangement of the overture to leoš janácek’s from the house of the dead (as published in szeto, 2013), where the violin part changed to solo and the trumpet part could optionally be offstage). currently, data cannot be simultaneously encoded as alternative (in $p) and as solo (in $b) in marc field 382, whereas onvie imposes no restrictions on these refinements. equally important is the consideration of whether the alternative is so different that the music should be considered a separate musical expression or an entirely new musical work. this is outside the scope of this article but was discussed in more detail in szeto (2017), and the data modeling is an active area of investigation under various keywords such as music information retrieval, arrangement, versions, and annotation (weiß et al., 2021; lewis et al., 2022). more specifically, the library of congress policy statements for rda 6.18.1.4 instructs catalogers whether or not to consider alternatives or arrangements in a range of conditions, such as slight alterations to instrumentation in an orchestral piece, early music composed before 1800 performed on modern instruments, and a change from vocal to instrumental in the popular music idiom. while such practices to make an either/or judgment are essential for the classification and collocation of physical materials, the onvie model will remove much of these artificial criteria by simply allowing all parts and mediums to be recorded completely and precisely. another common situation is alternative voices (such as a song for soprano sung by a tenor, or for alto sung by a bass). this is an ambiguity issue where the voice part is the same but the property of the medium differs. in the onvie model, soprano/alto/tenor/bass can be indicated as refinements to the medium “voice.” bibframe and the performed music ontology the bibliographic framework initiative (bibframe) is a library of congress initiative to develop a linked data alternative to marc. the library of congress initially worked with the music library association to model the medium of performance element but later abandoned the effort. instead, the bibframe model left “hooks” for a full model to be developed by a third party (szeto et al., 2016, p. 32). in response, the performed music ontology (pmo, 2021) was developed as an extension of the bibframe ontology with a focus on describing performed music. the medium of performance portion of the pmo model tracks closely with rda and marc practices. pmo begins with making the distinction between the declared medium, which is stated by the composer or in a reference source, and the performed medium, which was used in the actual performance and not necessarily the same as the declared medium. the pmo has separate properties for individual and ensemble mediums, a separate class for “part,” as well as properties to connect performers to their instruments, voices, and dramatic roles. however, pmo follows rda instructions where solo, alternative, and doubling are separately addressed, while other attributes are recorded as literals (free text) in a single catch-all “part type” property. the onvie model will break away from the rda/marc structure, and will provide an extensible structure for characteristics to be encoded as linked data with clear semantics. unlike pmo where medium of performance classes depend on linking to the parent ontology at multiple points (bf:notatedmusic for pmo:declaredmedium, bf:audio for pmo:performedmedium, bf:contribution for the pmo:individualmop and pmo:musicpart), the onvie model exists in a self-contained, independent space requiring only a single “hook” to the musical resource being described. the first contact on this hook is “parts.” by beginning with this layer, the onvie model no longer requires differentiation between mediums that were notional or actual. a new data model for medium of performance ontology for voice, instruments, and ensembles the primary motivation, and innovation, of the onvie model is capturing the relationship between part, medium, performer, and ensemble, which form the four main classes of the ontology. the first three main classes follow a loop: part connects to medium, medium connects to performer, and performer loops back to part. the class ensemble forms a branch by grouping one or more part entities. these relationships are shown in figure 2. figure 2. class relationships in the ontology for voice, instruments, and ensembles. all further refinements to the model fall within this framework. the list of refinements presented here consists of data that are currently recorded in cataloging practice, fixes to known issues, with many more properties that cover characteristics of specifical musical settings, parts, instruments, voices, devices, etc. onvie’s simple framework allows it to be readily extensible through additional refinements. definitions part class: musicpart scope: a series of musical events, abstracted from a musical work/expression, generally independent from other such abstractions from the same musical work/expression, which holds a consistent association with one or more mediums and/or performers note: a part is not equivalent to, but may be related to, a “printed part” (sheet music for an individual voice/instrument that does not contain notation for other voices/instruments of an ensemble), or a “voice” or “line” (individual melodies of polyphonic musical composition) examples: “violin” ; “percussion” ; “mezzo-soprano” ; “horns” medium class: musicmedium scope: a tool of sound production for a musical work/expression note: mediums include human voices, musical instruments, devices, and other means examples: “violin” ; “marimba” ; “voice” ; “saxophone” ; terms in lcmpt and unimarc codes for individual voice, instruments, and devices performer class: musicperformer scope: an agent responsible for expressing or actuating one or more parts note: performers include humans, machines, computers, and other entities examples: “violinist” ; “percussionist” ; “vocalist” ; “saxophonist” ; entities referring to actual persons in vocabularies such as lcnaf and viaf ensemble class: musicensemble scope: a group of parts in a musical work/expression note: in this model, only part groupings are used, even for ensembles traditionally known for the personas or the instruments (for example, string quartet) examples: “orchestra” ; “percussion quartet” ; “children’s choir” ; “jazz combo” ; ensemble terms in lcmpt and unimarc codes ; lcnaf, viaf, or other vocabularies for names classes and properties the onvie model is built with the library users’ starting point in mind: the instruments, voices, and devices (class musicmedium). the connecting layer between the medium and the music resource is the concept of “part” (class musicpart). the term “part” has been occasionally used in definitions and instructions, and occasionally conflated with voices, instruments, and performers, as discussed above. here, a new definition has been developed for this layer, distinct from musicmedium and musicperformer. the onvie model begins with at least one musicpart. a musicpart can link to other musicpart entities when alternatives or further subdivisions are present, or can link to printed published parts which could differ. every musicpart is required to have at least one musicmedium, even if it is unmediated (such as a spectator). musicpart can link to more than one entity in musicmedium, such as in the case of instrumental doubling or a single percussion part calling for multiple instruments. for musical scores, it ends here. for performed music, including popular, folk, and jazz that exist only as sound recordings, performers (class musicperformer) are linked from musicmedium, and each musicperformer loops back to one or more entities in musicpart. ensembles are not considered mediums, but are a separate class (class musicensemble), formed by an aggregate of individual parts. this departure from current cataloging practice prevents ensemble terms from being used as mediums, but rather encourages a complete accounting of mediums involved. the four main properties of this model connect these classes. refinements are provided to describe each in further detail. the list of refinement properties presented here was drawn from the current models as well as from the author’s own experience as an ensemble librarian, and is by no means exhaustive. what is not included in this model are the numbers. rather than requiring catalogers and metadata creators to supply the number of parts, performers, and ensembles, the granularity of the model enables linked data interpreter software to perform the counting. this method not only provides the flexibility to produce separate counts for parts, mediums, performers, and ensembles, it also removes the uncertainty and detailed analysis (or guesswork!) required to arrive at the number of performers needed for group-oriented parts such as percussion and continuo. by interpreting the refinements, machine counting can also provide the total, as well as further numerical breakdowns for soloists, accompanying performers, ensembles, and voice/vocal parts. the ontology table 1. classes. class subclass of mediumofperformance musical works/expressions musicmedium mediumofperformance musicpart mediumofperformance musicperformer mediumofperformance musicensemble mediumofperformance   table 2. properties. property use with expected value hasmusicpart mediumofperformance ; musicpart musicpart ismusicpartof musicpart musicpart ; mediumofperformance hasmusicmedium musicpart musicmedium ismusicmediumof musicmedium musicpart hasmusicperformer musicmedium musicperformer ismusicperformerof musicperformer musicmedium isresponsibleformusicpart musicperformer musicpart isperformedby musicpart musicperformer   table 3. refinements. property use with expected value notes examples rdfs:label mediumofperformance ; musicpart ; musicmedium ; musicensemble literal xml:lang mediumofperformance ; musicpart ; musicmedium ; musicensemble uri language code of the term source mediumofperformance ; musicpart ; musicmedium ; musicensemble uri source of information sourcetype mediumofperformance ; musicpart ; musicmedium ; musicensemble “transcribed” ; “recorded” ; “published” ; “inferred” ; “editorial” ; “programmatic update” ; etc. sourcenote mediumofperformance ; musicpart ; musicmedium ; musicensemble literal “first page of music” alternative mediumofperformance ; musicpart ; musicmedium ; musicensemble “is alternative” ; “is not alternative” ; “performer’s choice” ; “unspecified” ; “unknown” alternativetype mediumofperformance ; musicpart ; musicmedium ; musicensemble uri * hook for the full consideration of types of musical alternation/arrangement/transcription/adaptation alternativenote mediumofperformance ; musicpart ; musicmedium ; musicensemble literal “identical” ; “arranged for viola” ; “transcribed for solo piano” [concerto, jazz] ; “piano reduction” [orchestral accompaniment] ; “adapted for the violin” [folk music] ; “combined percussion part for one player” partnumber musicpart whole number >=0 “violin 1” ; “percussion 2” playertoapart musicpart “specified” ; “multiple” ; “performer’s choice” ; “unspecified” ; “unknown” playertoapartnumber musicpart whole number >=0 solo musicpart ; musicmedium ; musicensemble “is solo” ; “is not solo” ; “performer’s choice” ; “unspecified” ; “unknown” accompaniment musicpart ; musicmedium ; musicensemble “is an accompaniment” ; “is not an accompaniment” optional musicpart ; musicmedium “is optional” ; “is not optional” ; “performer’s choice” ; “unspecified” ; “unknown” ad lib musicpart ; musicmedium “is ad lib” ; “is not ad lib” ; “performer’s choice” ; “unspecified” ; “unknown” offstage musicpart ; musicmedium “is offstage” ; “is not offstage” ; “performer’s choice” ; “unspecified” ; “unknown” obligato musicpart ; musicmedium “is an obligato part” ; “is not an obligato part” ; “performer’s choice” ; “unspecified” ; “unknown” amplified musicpart ; musicmedium “is amplified” ; “is not amplified” ; “performer’s choice” ; “unspecified” ; “unknown” prerecorded musicpart ; musicmedium “is prerecorded” ; “is not prerecorded” ; “performer’s choice” ; “unspecified” ; “unknown” periodinstrument musicpart ; musicmedium “is a period instrument” ; “is not a period instrument” ; “performer’s choice” ; “unspecified” ; “unknown” periodinstrumentnote musicpart ; musicmedium uri or literal “baroque” [flute] fingeringsystem musicpart ; musicmedium uri “german” [recorder] tuningsystem musicpart ; musicmedium uri “just intonation” ; “pythagorean” ; “equal temperament” tuningreferencepitch musicpart ; musicmedium uri the pitch name of the tuning reference pitch. “a” tuningreferencefrequencyhz musicpart ; musicmedium number >=0 the frequency of the tuning reference pitch in hertz. “432” scordatura musicpart ; musicmedium “is tuned scordatura” ; “is not tuned scordatura” ; “performer’s choice” ; “unspecified” ; “unknown” tuning of a western string instrument which deviates from the standard tuning. tuningnote musicpart ; musicmedium literal “piano is tuned quarter tone flat” ; “drop d” [guitar] handsnumber musicpart ; musicmedium whole number >=1 number of hands playing an instrument handsside musicpart ; musicmedium ; musicperformer “left” ; “right” ; “performer’s choice” ; “unspecified” ; “unknown” which hand is being used for playing an instrument handsnote musicpart ; musicmedium ; musicperformer literal “piano (2), 3 hands” doublebasscextension musicpart ; musicmedium “requires a c extention” ; “does not requires a c extension” ; “performer’s choice” ; “unspecified” ; “unknown” flutebfoot musicpart ; musicmedium “requires a b foot” ; “does not require a b foot” ; “performer’s choice” ; “unspecified” ; “unknown” instrumentmute musicpart ; musicmedium uri instrumentmutenote musicpart ; musicmedium literal instrumentdimension musicpart ; musicmedium uri * hook for measurements of musical instruments instrumentdimensionnote musicpart ; musicmedium literal “26 inch” [timpani] instrumentsize musicpart ; musicmedium uri * hook for instrument sizes instrumentsizenote musicpart ; musicmedium literal “three-quarter” [guitar] ; “concert” [ukelele] instrumentpitch musicpart ; musicmedium uri pitch of single-pitched musical instruments “c4” [crotale] instrumenttransposition musicpart ; musicmedium uri key of transposing instruments “a” [clarinet] instrumenttranspositionnote musicpart ; musicmedium literal “clarinet in a” instrumentrangenumber musicpart ; musicmedium whole number >=0 pitch range of a musical instrument in number of half steps “60” [marimba] instrumentrangelowest musicpart ; musicmedium uri lowest pitch of a musical instrument “c2” [marimba] instrumentrangehighest musicpart ; musicmedium uri highest pitch of a musical instrument “c7” [marimba] instrumentnote musicpart ; musicmedium literal “5 octaves” [marimba] voicetype musicpart ; musicmedium uri “mezzo soprano” ; “contralto” voiceweight musicpart ; musicmedium uri “spinto” ; “soubrette” voicetessitura musicpart ; musicmedium uri “high” ; “medium” ; “low” voicepitchlowest musicpart ; musicmedium uri lowest pitch required of the vocalist voicepitchhighest musicpart ; musicmedium uri highest pitch required of the vocalist voicenote musicpart ; musicmedium literal technicalrequirement musicpart ; musicmedium uri * hook for computer / recording carrier / playback device information technicalrequirementnote musicpart ; musicmedium literal “requires an 8 track player” next steps with the onvie model laid out conceptually in this paper, the next steps would be to formalize the ontology with modeler software, and, with the help of the library community, integrate into library linked data editors and test a range of use cases to further fine tune semantics, data constraints, and documentation. in the course testing, standardized vocabularies can be developed for the many characterizations in the list of refinements, possibly in alignment with open platforms such as wikidata. some suggestions for these “hooks” can be found in the “notes” column in table 3. finally, pathways to publishing and maintaining the ontology can be explored. further analysis can be made in relation to the unimarc encoding standard, as well as to library-adjacent ontologies such as doremus, developed for analysis and visualization of music data, the music ontology, which focuses on capturing production of musical events, and musicbrainz, which is widely used for sound recordings. as none of these ontologies currently includes a model built out for medium of performance, developing a mechanism to hook onvie on to them would be a worthwhile investigation. another potentially fruitful area of study would be to align musicparts in onvie with other ontologies where the concept is also used, such as the “observations” object in the “musicological objects” layer in lewis et al. (2022). conclusion although linked data are designed to be machine-actionable, it is humans who ultimately employ the mediums with their voices, instruments, and other tools, to create music. it is also humans who are ultimately responsible for expressing and actuating each part of a musical work. this new ontology for voices, instruments, and ensembles, when hooked on to linked data bibliographic systems, will enable a medium of performance access point at a fine level of precision and completeness. library users will be provided a more straightforward path not only toward identifying and selecting music resources, but also toward discovering additional insights into the evolution of performing forces in the history of music making, a whole new area of humanistic studies previously hidden in plain sight. bibliography corigliano, j. (2011). chiaroscuro: for two pianos (one tuned down a 1/4 tone) (ed 4435, hl 50490191). g. schirmer. coyle, k. (2011). marc as metadata: a start. code4lib journal, 14. https://journal.code4lib.org/articles/5468. elmer, m. (1960). the music catalog as a reference tool. library trends, 8(4), 529-538. https://www.ideals.illinois.edu/bitstream/handle/2142/5907/librarytrendsv8i4f_opt.pdf. haas, g. f. (2009). trois hommages: für zwei klaviere (im vierteltonabstand gestimmt) zu zwei händen (1982/84) (ue 34 693). universal edition. heath, d. (1986). out of the cool: for flute (or violin) and piano (ch55693). chester music. lewis, d., shibata, e., saccomano, m., rosendahl, l., kepper, j., hankinson, a., siegert, c., and page, k. (2022). a model for annotating musical versions and arrangements across multiple documents and media. in 9th international conference on digital libraries for musicology (dlfm2022) (pp. 10-18). association for computing machinery. https://doi.org/10.1145/3543882.3543891. library of congress. (2012). summary of programmatic changes to the lc/naco authority file: what lc-pcc rda catalogers need to know. https://www.loc.gov/aba/rda/pdf/lcnaf_rdaphase.pdf. library of congress. (2014). library of congress launches medium of performance thesaurus for music. https://www.loc.gov/catdir/cpso/medprf-list-launch.html. library of congress. (2020). content designator history. in 382 – medium of performance. https://www.loc.gov/marc/bibliographic/bd382.html. mullin, c. (2018). an amicable divorce: programmatic derivation of faceted data from library of congress subject headings for music. cataloging & classification quarterly, 56(7), 607-627. https://doi.org/10.1080/01639374.2018.1516709. newcomer, n. l., belford, r., kulczak, d., szeto, k., matthews, j., & shaw, m. (2013). music discovery requirements: a guide to optimizing interfaces. notes, 69(3), 494-524. http://dx.doi.org/10.1353/not.2013.0017. online computer library center. (2022, may 24). oclc 989164116. a publicly accessible version of this record as imported by university of california, berkeley can be retrieved at https://search.library.berkeley.edu/discovery/sourcerecord?vid=01ucs_ber:ucb&docid=alma991033194509706532&recordowner=01ucs_network. ostrove, g. e. (2001). music subject cataloging and form/genre implementation at the library of congress. cataloging & classification quarterly, 32(2), 91-106. https://doi.org/10.1300/j104v32n02_08. performed music ontology: documentation (2021, may 6). https://github.com/ld4p/performedmusicontology/tree/main/documentation. rda toolkit. (2022). american library association. https://access.rdatoolkit.org/. subject analysis committee. (2017). a brave new (faceted) world: towards full implementation of library of congress faceted vocabularies. association for library collections & technical services. http://hdl.handle.net/11213/8146. subject analysis committee. (2022). retrospective implementation of library of congress faceted vocabularies. association for library collections & technical services. http://hdl.handle.net/11213/17998. swanson, c. m. (2003). adding to the viola repertoire by arranging: a study on methods of arranging music for viola from clarinet, with an original arrangement of the saint-saens clarinet sonata in e-flat, op. 167 (umi number: 3107045) [doctoral dissertation, university of arizona]. proquest information and learning. http://hdl.handle.net/10150/280389. szeto k. (2010a). concerto for violin and chamber ensemble arranged after robert schumann’s scherzo from symphony no. 2. alexander street press classical scores library, volume iv. https://search.alexanderstreet.com/preview/work/bibliographic_entity%7cscore%7c3643958. szeto k. (2010b). robert schumann’s symphony no. 2 in c major arranged for chamber ensemble. alexander street press classical scores library, volume iv. https://search.alexanderstreet.com/preview/work/bibliographic_entity%7cscore%7c3643956. szeto k. (2013). leoš janácek: ouvertüre zur oper “aus einem totenhaus” für solovioline und ensemble (ue 36 249). universal edition. szeto, k. (2016, october 15). untangling medium of performance for the linked data environment [presentation]. new york state-ontario chapter of the music library association fall meeting, university of toronto, ontario, canada. https://academicworks.cuny.edu/bb_pubs/1251/. szeto, k., adams, a. d., billet, k. e., busselen, c., kishimoto, k. s., loprete, a. a., mcfall, l., rondeau, s., snyder, t. l., soe nyun, j. l., vanden dries, w. r., & vermeij, h. (2016). report of the cmc bibframe task force to the board of the music library association. https://cmc.wp.musiclibraryassoc.org/wp-content/uploads/sites/5/2019/02/201602task_force_report.pdf. szeto, k. (2017). the mystery of the schubert song. notes, 74(1), 9-23. https://doi.org/10.1353/not.2017.0071. weiß, c., zalkow, f., arifi-müller, v., müller, m., koops, h. v., volk, a., & grohganz, h. g. (2021). schubert winterreise dataset: a multimodal scenario for music analysis. journal on computing and cultural heritage, 14(2), 1-18. https://doi.org/10.1145/3429743. about the author kimmy szeto is associate professor and metadata librarian at baruch college, city university of new york, where he manages metadata for digital resources. his recent research focuses on the technical and conceptual tensions between cataloging practice and the linked data environment. outside the library and academia, kimmy can be heard as a chamber arranger of symphonic works and as a collaborative pianist in concert halls and theaters around new york city. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – leveraging the rbms/bsc latin place names file with python mission editorial committee process and structure code4lib issue 48, 2020-05-11 leveraging the rbms/bsc latin place names file with python to answer the relatively straight-forward question “which rare materials in my library catalog were published in venice?” requires an advanced knowledge of geography, language, orthography, alphabet graphical changes, cataloging standards, transcription practices, and data analysis. the imprint statements of rare materials transcribe place names more faithfully as it appears on the piece itself, such as venetus, or venetiae, rather than a recognizable and contemporary form of place name, such as venice, italy. rare materials catalogers recognize this geographic discoverability and selection issue and solve it with a standardized solution. to add consistency and normalization to imprint locations, rare materials catalogers utilize hierarchical place names to create a special imprint index. however, this normalized and contemporary form of place name is often missing from legacy bibliographic records. this article demonstrates using a traditional rare materials cataloging aid, the rbms/bsc latin place names file, with programming tools, jupyter notebook and python, to retrospectively populate a special imprint index for 17th-century rare materials. this methodology enriched 1,487 machine readable cataloging (marc) bibliographic records with hierarchical place names (marc 752 fields) as part of a small pilot project. this article details a partially automated solution to this geographic discoverability and selection issue; however, a human component is still ultimately required to fully optimize the bibliographic data. by kalan knudson davis background in 2018, the university of minnesota (umn) libraries cataloging and metadata group (cmg), an alma governance group that creates and maintains system-wide cataloging and metadata policy and procedures, approved a new written policy for rare book cataloging. these rare book cataloging guidelines stipulated the use of hierarchical place names to aid discoverability for contemporary place names as well as more faithfully transcribed imprint locations, but there were hundreds of thousands of extant umn libraries bibliographic records for rare materials that did not have this hierarchical place name field populated. indexing the hierarchical place name field was problematic as only a small subset of recently cataloged rare materials could utilize it. a solution needed to extend much further than updating cataloging policy and reindexing fields in umn’s primo discovery layer. redescribing all rare materials imprint statements to reflect current cataloging standards and transcription practice was out of scope due to other cataloging priorities and rare materials staffing limitations. a purely manual record-by-record method of adding contemporary place names retrospectively to rare materials records would be time consuming, error prone, and an impractical solution. i set out to develop an automated approach by testing and implementing a small pilot project. geographic selection illustrated in marc the orbis latinus lists eleven variations for the place name venice, italy: venetia, s. marci civ., portus venetus, venecia, veneciae, venetiae, veneticis, venetie insula, venetorum civ., venetus, and venitia (grässe 1971) [1]. these entries for venice are among the thousands of potential rare materials place names contained in the medieval geography reference. for many legacy rare materials, a contemporary form of place name, such as the search keyword venice, italy, appears nowhere on the bibliographic record. to add consistency and discoverability to imprint locations, rare materials catalogers utilize the hierarchical place name, or the marc 752 field, to create a special imprint index (library of congress 1999 [updated 2019 november] [2]). the marc 752 field has been redefined as recently as 2017 with several marc discussion papers and proposals regarding its use (library of congress 2004 [3], library of congress 2004b [4], and library of congress 2004c [5]). this recently refined field in a mature bibliographic repertoire results in many legacy cataloging records that do not currently utilize the marc 752 field to its fullest capacity. rare materials catalogers optimize the place of publication transcribed in the marc 260/264 subfield $a for the identification of bibliographical variants in manifestations via a more faithful transcription ([library of congress 1999b [updated 2019 november] [6]). to further illustrate the geographic selection problem with marc examples, consider the following transcribed place names for venice, italy. 260 \\ $a venetiis : $b ex typographia remondiniana ..., $c mdcclxiii [1763] 260 \\ $a impressum venetijs: $b per jacobu[m] pentiu[m] de leucho : $b arte impensa vero juncte de junctis florentini, $c anno d[omi]ni 1508 vltimo octobris [31 oct. 1508] 264 \1 $a in venetia : $b nella stamperia di giouanni salis, $c m. dc. xvi. [1616] from those same bibliographic records illustrated above, rare materials catalogers optimize the following marc 752 field for rare materials geographic selection tasks (library of congress 1999 [updated 2019 november] [7]). adding the marc 752 makes these records discoverable by a contemporary and normalized place name. 752 \\ $a italy $d venice. although the fuller, more faithful transcriptions of rare materials allow users to clearly identify and distinguish among the variants of manifestations, the variety of transcription methods found in the imprint statements of bibliographic records and the orthographic and graphic changes throughout time can obscure geographic selection, which is remedied by the addition of normalized and contemporary hierarchical place names. literature review rare materials descriptive needs the introduction to the descriptive cataloging for rare materials (books) (dcrm(b)) emphasizes the rare materials’ need for “more faithful transcriptions and more accurate physical descriptions” and that this practice “facilitate[s] differentiation between manifestations and reveal[s] the presence of bibliographic variants among seemingly identical items” (bibliographic standards committee 2011 [8]). rare materials have artifactual value; therefore, rare materials cataloging addresses them both as a text and as a physical object. the cataloging of rare materials requires “additional details of description in order to identify significant characteristics” (bibliographic standards committee 2011 [9]) and an accurate representation of the entity as it describes itself. in order to achieve these more faithful transcriptions, rare materials cataloging standards and transcription practices, much like graphical representation of language and orthography, have changed and evolved over time. where needed, current rare materials cataloging standards deviate from other bibliographic standards to meet unique descriptive needs (bibliographic standards committee 2011 [10]). for grammatically separable elements, dcrm(b), allows for the noted transposition of elements (bibliographic standards committee 2011 [11]), but the use of contractions or transcribed phrases such as en venetia and impressum venetiis add an additional layer of imprint location complexity as well as indication of role. transcribed place name abbreviations such as venet. or lugduni batauor. are common. the article “transcription in rare books cataloging” discusses the differences in cataloging and transcription practices between dcrm(b), international standard bibliographic description (isbd), and resource description access’ (rda) ‘take what you see’ principle for early printed books (sjökvist 2006 [12]). sjökvist discusses transcription and normalization in the terms of fulfilling user tasks: identification and retrieval (i.e. selection), respectively. although the article concentrates its arguments on title access rather than imprint locations, its conclusion of using two different marc fields, each optimized to a different ifla lrm user task, is similar to how rare materials catalogers currently handle imprint locations. leslie and griffin in their 2013 discussion paper “transcription of early letter forms in rare materials cataloging” note that “there is no clearly superior solution to the [transcription] problem; our task is finding the least painful solution” (leslie and griffin 2013 [13]). the article discusses the evolution of letter forms; spelling conventions; and contractions, ligatures, and digraphs usage, both archaic and modern, while thoroughly addressing the 23-letter roman alphabet’s journey to our modern 26-letter system. leslie and griffin evaluate several different transcription systems by ensuring user convenience, representation of the entity as it describes itself, accuracy of descriptions, principle of sufficiency and necessity, and the principle of standardization. transcribed imprint statements illustrate the numerous difficulties presented to rare materials catalogers in graphical and orthographical usage, both modern and archaic, as the “gradual differentiation between u and v as representing different sounds, in particular u (and i) as vowels, and v (and j) as consonants” occurred over time (leslie and griffin 2013 [14]). this leads to further place name spelling variations in the geographic selection puzzle as characters may have variations in transcription due to the cataloging rules and title page language. for example, transcribing basileæ or basileae depends on whether the title page is in english or french. the article “rda and rare books cataloging” discusses imprint treatments between the marc 260 and marc 264 subfield $a, in particular where elements of the imprint statement are grammatically inseparable (burns 2019 [15]). burns notes that “publication statements on rare books can present some of the most challenging transcription issues for rda rules” (burns 2019 [16]). indeed, rda publication statements will surely bring many more transcribed place name variations and more transcription practices to bibliographical records. place names and imprints in rare book cataloging orbis latinus, recently revised in 1971, has been the standard reference resource for medieval geography since its original publication in 1861 (grässe 1861 [17]). frederick benedict edited a second edition in 1909 with special considerations for medieval and modern latinities (grässe 1909 [18]). r.c. schmidt published a third edition in 1922 (grässe 1922 [19]). helmut plechl in cooperation with günter spitzbart helmed a fourth edition in 1971. this most recent publication has been expanded to over 579 pages (grässe and plechl 1971 [20]). the first efforts to digitize the 1909 resource took place at columbia university under karen green in 2000 (bibliographic standards committee 2015 [21]). another important place name resource, peddie’s place names in imprints, first published in 1932, often finds a home within the workstations of rare materials catalogers. the preface notes, “one of the difficulties of the cataloguer and the bibliographer has been to identify the place of printing or publication when disguised in latin, or in some dialect form. this index is intended to help in the solution of this problem” (peddie 1932 [22]). the rbms/bsc latin place names (lpn) file is a listing of latin place names found in the imprints of materials printed before 1801 and their vernacular equivalents in resource description and access (rda) form. the file has been compiled from r.a. peddie’s place names in imprints and grässe’s orbis latinus. robert l. maxwell created the rbms/bsc lpn file in 1997 with the assistance of karen larson (bibliographic standards committee 2015 [23]). this online resource has been maintained since that time as a rare book manuscript section / bibliographic standards committee cataloging aid. the introductory section of rbms/bsc latin place names (lpn) file states, “this database was compiled from the imprint information in cataloging records of several anglo-american research libraries. because these records were created over a long period of time and under different standards and rules of transcription, the orthography of the place names with respect to i/j and u/v/w does not necessarily reflect what was found in the original [materials]” (bibliographic standards committee 2015 [24]). out of necessity, the rbms/bsc has standardized the file; therefore, rare materials imprint statements contain many more place name variations than are listed explicitly in the file. in the case of venice, italy, for example, one of the eleven place names listed in grässe’s orbis latinus, venetiis, has been transcribed in bibliographic records as venetijs. recent developments in web encoding standards are another complexity inherent in leveraging the rbms/bsc lpn file. html5 utilizes unicode utf-8 as, generally, does current bibliographic practice. however, the introduction to the rbms/bsc lpn file also notes, “at the time that the latin place names file was developed, it was only possible to enter ascii characters in html files. this limited the availability of diacritical marks to the most common. therefore, some rda forms, especially those for eastern european cities, are missing their diacritics” (bibliographic standards committee 2015 [25]). the twenty plus variations transcribed place names of nuremberg, munich, and zurich (e.g.. nürnberg, münchen, and zürych) illustrate this. use of python in cataloging workflows frank’s article “augmenting the cataloger’s bag of tricks : using marcedit, python, and pymarc for batch-processing marc records generated from the archivists’ toolkit” provides an overview of the ongoing trend that “combines traditional cataloging standards and programming, in order [for catalogers] to efficiently manage their workflows” (frank 2013 [26]). frank specifically mentions the pymarc python library created by gabriel farrell, mark matienzo, and ed summers (pymarc [updated 2020 april] [27]) as a workflow adaptation, with the one-by-one record approach no longer being a sustainable practice. in their “leveraging python to improve ebook metadata selection, ingest, and management,” thompson and traill discuss the use of the pandas python module (pandas … , [updated 2020 march] [28]) to create a dataframe to analyze and evaluate the quality of marc records, as well as to write out comma separated value (csv) reports for output and further human remediation tasks (thompson and traill 2017 [29]). thompson and traill’s use of dataframes not only to execute evaluation of marc records, but also to implement workflows within those records themselves heavily influenced the development of the methodology and script described in this article. uris in marc the “task group on linked data best practices final report” details recommendations for practical experimentation and guidance to catalogers for coding marc subfields $0, $1, and $4 (program for cooperative cataloging 2019 [30]). the report also recommends an appropriate subfield order for the uri and non-uri marc subfields. of particular interest are the multiple options and recommendations regarding the marc subfield $4: the use of option three “if a relationship uri or code is given, it is encouraged but not required to also provide the corresponding label ($e, $i, or $j depending on the field)” (program for cooperative cataloging 2019 [31]). the pcc task group’s final report, along with the marc object table (program for cooperative cataloging 2019b [updated 2019 august 31] [32]) and the “pcc task group on uris in marc’s formulating and obtaining uris: a guide to commonly used vocabularies and reference sources” (program for cooperative cataloging 2019c [33]), informed the choice, construction, and order of the uris and enriched marc 752 fields for this pilot project. method the script’s purpose is to check each record for existing marc 260/264 subfield $a values and 008 marc country code value against the dataframe version of the rbms/bsc latin place names file, supply normalized geographic data in the marc 752 field when possible, and output the results into three separate spreadsheets for review. turning the rbms/bsc latin place names file into a pandas dataframe to create and test the python scripts, i extracted the 1,487 marc bibliographic records for 17th-century rare materials as a sample from the university of minnesota libraries’ alma library services platform. this dataset exemplified many of the characteristic orthographic, geographic, and graphical variations in the imprint locations of rare materials, while remaining a small subset of manageable size for a pilot project. the marc bibliographic record dataset contained 323 unique strings between the marc 260/264 subfield $a. i added these unique strings, along with an additional 1,411 entries extracted from the rbms/bsc latin place name file, to a comma separated value (csv) file. each row within the csv file was also populated with the various values comprising the marc 752 subfield components, including the country ($a), the first-order political jurisdiction (i.e. state or province) ($b), city ($d), relator term ($e), relator term uri ($4), and real world object uri ($1). table 1. an excerpt of the csv input for the rbms/bsc latin place names file dataframe. venetii it italy$dvenice, italy no value venice, venetiis it italy$dvenice, italy no value venice, venetio it italy$dvenice, italy no value venice, venetis it italy$venice, italy no value venice, venetiuis it italy$dvenice, italy no value venice, the csv file is an essential component of the python script’s iterative process. peddie, grässe, and the bibliographic standards committee did not design their geographic resources to account for every place name variation, phrase, scribal contraction, and abbreviation. the bibliographic records themselves account for every possible use case. the script user feeds additional locations and place name variations into the script for processing making for a flexible, locally customizable, and responsive design. the python script feeds the csv file into a pandas dataframe. table 2.the enriched rbms/bsc latin place name file is fed into pandas dataframe. #create pandas dataframe masterchart = pd.read_csv('masterchartcleaner.csv') #index the publication_place column idx = pd.index(masterchart.publication_place) evaluating marc records– the good, the bad, and the okayish marc record evaluation and dataframe querying takes place before a record is sent on to one of the python script’s three pymarc writers. the evaluation process splits the original marc file into one of three possible paths based on evaluation of the data contained within the records. option 1: 752s are already present on marc record recently cataloged rare materials may already have a marc 752 field populated with a hierarchical place name. in these cases, the script identifies these records and writes them to a separate file for manual review. the file reviewer may wish to further enrich the existing marc 752 subfields with additional real world object uris, if these are lacking. table 3.the script writes rare materials that already have a populated marc 752 field to a file for review. #if there is already a 752 in the record, send to the #okayish writer for manual review. else: #create dictionary of values for okayish csv file mmsid = [record['001']] oclcno = [record['035']] ccfrom008 = countrycode field260 = [record['260']] field264 = [record['264']] field752 = [record['752']] #create a dictionary of lists dict = {'alma mms id': mmsid, 'oclc number': oclcno, 'country code': ccfrom008, '260': field260, '264': field264, '752': field752} okayishdf = pd.dataframe(dict) #saves marc information to the okayish dataframe okayishdf.to_csv('reportmarcalreadywith752s.csv', mode='a', header=false) writerokayish.write(record) option 2: 752s are added to marc record for marc records containing only one marc 260/264 subfield $a/$e value, the script queries the pandas dataframe to find a relevant match. the python script assigns the marc country code in the 008 field and the marc 260/264 subfield $a to variables. the pandas dataframe is essentially filtered twice, matching both the 008 country code and marc 260/264 subfield $a values. table 4.the enriched rbms/bsc latin place name file dataframe is filtered to match the bibliographic record’s 008 country code and marc 262/264 subfield $a. #this filters the original masterchart dataframe for marc #country code and publication place queriedchart = masterchart.query('countrycode == @countrycode and publication_place == @clean264a', inplace = false) #this returns the size of the queriedchart, in case there #aren't any matches. size = queriedchart.size at this stage, the script takes special care to assign the appropriate relator terms (subfield $e) and relator uris (subfield $4) for marc 264 fields according to the various indicator values. as noted in the dcrm(b), “the roles of publishers, printers, and booksellers were not clearly delimited in the hand-press period. statements relating to printing frequently appear prominently on early printed materials, reflecting the tendency of printers to function as more than solely manufacturers” (bibliographic standards committee 2011 [34]). therefore, defining relator terms for the marc 260 field is not as straightforward, but follows current dcrm(b) practice: “if the publication bears only a statement relating to manufacture, or multiple such statements, generally assume the manufacturer(s) to also be functioning as publisher(s), distributor(s), etc.” (bibliographic standards committee 2011 [35]). not every relator relationship can be programmatically accounted for. although the script takes special care to evaluate the nature of the place name’s relationship to the manifestation, a rare materials cataloger may wish to evaluate and scan the supplied marc 752 subfield $e and subfield $4 on retrospectively enriched records. table 5.the script assigns relator terms and uris according to indicator values. #assign value to 752 subfield e and subfield 4 based on 264 #indicators. for imprintrole in record.get_fields('264'): #assign 752 subfield e and 4 for production place if imprintrole.indicator2 == '0': field752e = "production place." field7524 = "http://id.loc.gov/vocabulary/relators/prp.html" #assign 752 subfield e and 4 for publication place elif imprintrole.indicator2 == '1': field752e = "publication place." field7524 = "http://id.loc.gov/vocabulary/relators/pup.html" #assign 752 subfield e and 4 for distribution place elif imprintrole.indicator2 == '2': field752e = "distribution place." field7524 = "http://id.loc.gov/vocabulary/relators/dbp.html" #assign 752 subfield e and 4 for manufacture place elif imprintrole.indicator2 == '3': field752e = "manufacture place." field7524 = "http://id.loc.gov/vocabulary/relators/mfp.html" #handle records that do not have indicators with 264, #unlikely but possible. else: #sends problem records with 264s with no indicators #to the badwriter. indexchecker = false if a match within the rbms/bsc latin place name file dataframe exists, the script compiles the subfield components for the marc 752 field and the script adds the field to the marc record. table 6.the script adds the compiled marc 752 field to the record. if indexchecker is true and size > 0: #assign the value of term3 which gives the index of the row #in the masterchart. term3 = queriedchart.index[0] #assign the values of the 752 subfields that weren't already #assigned above. field752a = masterchart.loc[term3,"752a"] field752b = masterchart.loc[term3,"752b"] field752d = masterchart.loc[term3,"752d"] field7521 = masterchart.loc[term3,"geonames_uri"] #add the assembled 752 subfields to the marc record. record.add_ordered_field( field( tag = '752', indicators = [' ',' '], subfields = [ 'a', field752a, 'b', field752b, 'd', field752d, 'e', field752e, '4', field7524, '1', field7521, ])) the script writes all records with added 752 fields to a separate file for manual review. table 7.the script writes records with populated marc 752 field to a file for review. #the marc record is written to the goodwriter after the #752 is populated and added to the record. #create dictionary of values for goodwriter csv file mmsid = [record['001']] oclcno = [record['035']] ccfrom008 = countrycode field260 = [record['260']] field264 = [record['264']] field752 = [record['752']] #create a dictionary of lists dict = {'alma mms id': mmsid, 'oclc number': oclcno, 'country code': ccfrom008, '260': field260, '264': field264, '752': field752} gooddf = pd.dataframe(dict) #saves marc information to the good dataframe gooddf.to_csv('reportmarcwith752s.csv', mode='a', header=false) writergood.write(record) option 3: 752s not added to marc record by design, the script does not add a marc 752 field for some use cases. briefly, i will describe some of these situations, characteristics, and script handling. they can be broadly categorized into the following five groups: the script identifies potential errors present in the marc record. the script identifies an imprint location that could refer to several places and the cataloger will need to evaluate the record manually and in greater context. the script can not identify the imprint place. the script user needs to add additional imprint information to the csv file. the marc record lists two places of imprint. errors are commonplace in legacy marc bibliographic records. mismatches between the 008 marc country code and place of imprint are possible where marc country codes or geographic headings are obsolete and now require evaluation and maintenance. geographic and typographic errors are also common. table 8.excerpt of mismatches between 008 marc country code and marc 260/264 values. ne =260 \\$ahafniae :$bimpensis salomonis sartorii,$c1624. ne =260 \\$ahafniae :$bliteris matthiae godicchenii,$c1664. ne =260 \\$ahafniae :$bsumptibus petri hauboldi, excudebat petrus hakius,$c1658. ne =260 \\$ahafniae :$btypis matthiae godiccheni sumptibus petri haubold,$c1666. the rbms/bsc lpn file indicates several names that refer to one or more places and/or to fictitious places. when the script can’t resolve a match based on the 008 marc county code and the latin place name, instead of adding a marc 752 field, the script places the marc record in the “752s not added to marc record” writer queue instead. table 9.example from the rbms/bsc latin place names file dataframe requiring manual resolution. lpn master file augustae * most frequently augustae vindelicorum but also argentorati augustae treuirorum londini tarracone taurini and tubingae xx manual input unfortunately, not all bibliographic mysteries are solvable. sometimes rare materials catalogers cannot determine an imprint location. the script does not add a marc 752 field for items traditionally denoted as “s.l.”, abbreviation for sine loco, or more recently as “[place of publication not identified].” table 10.examples of [place of publication not identified] as imprint locations. xx =260 \\$a[place of publication not identified],$c1613. xx =260 \\$a[place of publication not identified] :$b[publisher not identified],$c1648. xx =260 \\$a[place of publication not identified],$c1680. finally, where there is less than one or more than one of any combination of marc 260 and marc 264 and when there is more than one subfield $a/$e value, the record is kicked out for manual review. a match on a place name alone without also a match of a 008 marc county code could very easily lead to an erroneous result. a rare materials cataloger can assign only one 008 marc country code per bibliographical record, which may or may not be the country code for all imprint locations listed on the rare material. the script alone can’t resolve these situations. a rare materials cataloger needs to resolve and verify, in further consultation with bibliographic utilities and with research on the typical locations of the publisher, manufacturer, etc. the script contains a simple logical true/false variable, which accounts for many of these edge cases and marc record errors, and will write the record to the “752s not added to marc record” queue. table 11.the script sends the marc record for manual resolution. else: indexchecker = false message = "more than one 260 subfield a/e is present in record. resolve manually." table 12.the script sends a marc record for manual resolution. #if there is more or less than one 260/264 tag combo, #send it to the badwriter. else: indexchecker = false message = "more than one 260/264 fields are present in record. resolve manually." csv review files as the script writes marc records to the three writer queues, the python script publishes additional pandas dataframes as csv files. the dictionary contains important marc record identifiers, such as the 001 and 035 control numbers, the 008 marc country code, the entire 260/264 string, if assigned, the 752 string, and a status message. the script writes the csv files for human consumption and at a glance can help further diagnose errors within the marc records and/or verify various components of the marc 752 fields (locations and relator terms, alike). rare materials catalogers can evaluate the csv files to ensure that the script assigned marc 752 subfields appropriately. this is especially helpful for examining and resolving marc records that have fictitious, incorrect, grammatically inseparable elements, and/or multiple imprint locations. table 12.an excerpt from a csv file with marc records requiring further review. gw =260 \\$afrancofurti,$alipsiae :$bprostant apud nicolaum scipionem,$c1685. gw =260 \\$ahannover$aund wolffenbüttel :$bh. grentz,$c1690. gw =260 \\$ahanover ;$agvelpherit :$bsumptibus gothofredi freytagii,$c1696. gw =260 \\$aargentorati :$btypis j. caroli,$c1618. results marc records of the 1,487 marc bibliographic records for 17th-century rare materials used as an initial dataset, only five marc records were previously populated with the marc 752 field. this constitutes less than one percent of all records in the dataset. as predicted, these items were recently acquired by the umn’s collections and were cataloged within the last three years. these records illustrate option one as described above, where marc 752 fields were already present on records. i manually reviewed the marc 752 fields and enriched them with uris. from the original dataset, 1,410 marc bibliographic records were automatically populated with the hierarchical place name field, due to a confirmed queried pandas dataframe for the 008 marc country code and imprint location in the marc 260/264 field subfield $a. this constitutes 94% of all records in the dataset and illustrates option two as described above. after further review, i manually resolved 72 of the 1,487 marc bibliographic records. this constitutes five percent of all records in the dataset. these records illustrate option three as described above, where the script did not add 752s to marc records due to errors, multiple places of imprint, or other reasons. out of the 72 records, i identified 32 records that contained errors. the most common error was a mismatch between the 008 marc country code and marc 260/264 subfield $a. this included simple geographic errors, such as copenhagen (denmark) being associated with the netherlands, or toulouse listed in germany rather than france. it also included simple typographical errors, such as ne (for the netherlands) being mistyped re (for réunion). another common error occurred when a publisher name was mistaken as a place of publication. four records contained “place of publication not identified” as an imprint location and i did not evaluate them further. i identified twenty records as correct and after minor tweaks to the pandas dataframe will process correctly in the future– these were mostly due to previously unaccounted for diacritics and special characters. finally, i identified only sixteen records as having more than one marc 260/264 subfield $a. one should be cautious in correcting potential errors and manually assigning marc 752 fields keeping a vigilant eye out for false imprints, reproduction statements, fictitious imprints, or other edge use cases using human/cataloger judgment to fully optimize the data. conclusion i intended the 1,487 marc records as a pilot project, with the intention to fully populate the special imprint index using the python script for all umn libraries rare materials dated 1450 to 1800. as the pandas dataframe is constantly used, updated, and enriched, it will grow to account for as many imprint use cases as possible. the python script has the potential to be of great use to other libraries with rare materials collections who would like to retrospectively populate a special imprint index using the marc 752 field. the code is available on github for broader use and adaptation (davis [updated 2020 april] [36]). the uri enriched imprint pandas dataframe contains many important pieces of the linked data puzzle and has broad implications for the history of the book. enriching the marc records with geonames uris enables exciting bibliographical associations with larger scale geospatial data. the data collected in the pandas dataframe could inform future developments for rbms/bsc lpn file as linked open data. the tabular data could be used to export and populate future applications of owl and rdf semantic relationships for imprint places. references [8] [9] [10] [11] [34] [35] bibliographic standards committee. 2011. descriptive cataloging of rare materials (books). rare books and manuscripts section, association of college and research libraries; in collaboration with the policy and standards office of the library of congress. washington, dc: library of congress, cataloging distribution service [cited 2019 december 9]. available from: https://rbms.info/files/dcrm/dcrmb/dcrmb3.pdf [21] [23] [24] [25] bibliographic standards committee. 2015. rbms/bsc latin place names file. chicago, il: american library association, rare book and manuscript section, bibliographic standards committee [cited 2019 december 11]. available from: http://rbms.info/lpn/ [15] [16] burns, m. 2019. rda and rare books cataloging, part 2. library resources & technical services. [cited 2019 december 12]; 63(1): 4-28. available from: http://search.ebscohost.com.ezp2.lib.umn.edu/login.aspx?direct=true&authtype=ip,uid&db=aph&an=134098616&site=ehost-live [36] davis, k. k. [updated 2020 april]. rbms-bsc-lpn-file-dataframe. available from: https://github.com/ladylazarus3/rbms-bsc-lpn-file-python-dataframe [26] frank, h. 2013. augmenting the cataloger’s bag of tricks : using marcedit, python, and pymarc for batch-processing marc records generated from the archivists’ toolkit. code4lib journal [internet], issue 20 [cited 2019 december 11]. available from: https://journal.code4lib.org/articles/8336 [17] grässe, jgt. 1861. orbis latinus, oder, verzeichniss der lateinischen benennungen der bekanntesten städte etc., meere, seen, berge und flüsse in allen theilen der erde. dresden: g. schönfeld buchhandlung (c.a. werner) [cited 2019 december 9]. [18] grässe, jgt. 1909. orbis latinus, oder, verzeichnis der wichtigsten lateinischen ortsund ländernamen. new york: e. steiger & co [cited 2019 december 11]. [19] grässe, jgt. 1922. orbis latinus, oder, verzeichnis der wichtigsten lateinischen ortsund ländernamen. berlin: r.c. schmidt [cited 2019 december 11]. [1] [20] grässe, jgt and plechl, h. 1971. orbis latinus, lexikon lateinischer geographischer namen. braunschweig, germany: klinkhardt & biermann [cited 2019 december 9]. [13] [14] leslie, dj and griffin, b. 2013. transcription of early letter forms in rare materials cataloging. a discussion paper prepared for the dcrm working conference [cited 2019 december 11]. available from: http://rbms.info/files/dcrm/dcrmb/wg2lesliegriffin.pdf [2] [7] library of congress. 1999 [updated 2019 november]. marc 21 format for bibliographic data, 752. washington, dc: library of congress, network development and marc standards office [cited 2019 december 9]. available from: https://www.loc.gov/marc/bibliographic/bd752.html [6] library of congress. 1999b [updated 2019 november]. marc 21 format for bibliographic data, 260. washington, dc: library of congress, network development and marc standards office [cited 2019 december 9]. available from: https://www.loc.gov/marc/bibliographic/bd260.html [3] library of congress. 2004. marc discussion paper 2004-dp02. washington, dc: library of congress, network development and marc standards office [cited 2019 december 11]. available from: https://www.loc.gov/marc/marbi/2004/2004-dp02.html [4] library of congress. 2004b. marc discussion paper 2016-dp21. washington, dc: library of congress, network development and marc standards office [cited 2019 december 11]. available from: https://www.loc.gov/marc/mac/2016/2016-dp21.html [5] library of congress. 2004c. marc proposal no. 2004-07. washington, dc: library of congress, network development and marc standards office [cited 2019 december 11]. available from: https://www.loc.gov/marc/marbi/2004/2004-07.html [28] pandas: python data analysis library [updated 2020 march]. available from: http://pandas.pydata.org [22] peddie, ra. 1932. place names in imprints: an index to the latin and other forms used on title pages. london: grafton & co. [cited 2019 december 12]. [30] [31] program for cooperative cataloging. 2019. task group on linked data best practices final report. washington, dc: program for cooperative cataloging, library of congress [cited 2019 december 12]. available from: https://www.loc.gov/aba/pcc/taskgroup/linked-data-best-practices-final-report.pdf [32] program for cooperative cataloging. 2019b [updated 2019 august 31]. marc object table. washington, dc: program for cooperative cataloging, library of congress [cited 2019 december 12]. available from: https://docs.google.com/spreadsheets/d/1met87ymzjiwjrh_psatjpaecxv_30hw4c1skbrzq8ra/edit#gid=55309217 [33] program for cooperative cataloging. 2019c. task group on uris in marc. formulating and obtaining uris: a guide to commonly used vocabularies and reference sources. washington, dc: program for cooperative cataloging, library of congress [cited 2019 december 12]. available from: https://www.loc.gov/aba/pcc/bibframe/taskgroups/formulate_obtain_uri_guide.pdf [27] pymarc [updated 2020 april]. available from: https://gitlab.com/pymarc/pymarc [12] sjökvist, p. 2016. transcription in rare books cataloging. cataloging & classification quarterly [internet]. [cited 2019 december 12]; 54:5-6. available from: https://doi.org/10.1080/01639374.2016.1192079 [29] thompson, k. and traill, s. 2017. leveraging python to improve ebook metadata selection, ingest, and management. code4lib journal [internet], issue 38 [cited 2019 december 11]. available from: https://journal.code4lib.org/articles/12828 about the author kalan knudson davis is the special collections metadata librarian at the university of minnesota libraries. she provides intellectual access for rare materials held in the wangensteen historical library, the james ford bell library, and the andersen horticultural library where she performs specialized and rare book cataloging using a variety of descriptive standards including resource description access (rda) and the descriptive cataloging of rare materials (dcrm) manuals; provides subject access; and applies various classification schemas. although classically-trained as an anglo-american cataloging rules revision 2 (aacr2) cataloger, she aspires to become one of jay jordan’s information jedi. kalan actively feeds her keen interest in book history, codicology, paleography, bibliographical description, bookbindings, provenance, and other related disciplines, which in turn, informs her cataloging practice. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – archiving the web: a case study from the university of victoria mission editorial committee process and structure code4lib issue 26, 2014-10-21 archiving the web: a case study from the university of victoria the university of victoria libraries started archiving websites in 2013, and it quickly became apparent that many scholarly websites being produced by faculty, especially in the digital humanities, were going to prove very challenging to effectively capture and play back. this article will provide an overview of web archiving and explore the considerable legal and technical challenges of implementing a web archiving initiative at a research library, using the university of victoria’s implementation of archive-it, a web archiving service from the internet archive, as a case study, with a special focus on capturing complex, interactive websites that scholars are creating to disseminate their research in new ways. by corey davis introduction while “paper survives benign neglect for a long time” (rosenthal, 2010a), such is not the case with materials on the web. digital information is far more fragile than print. “electronic resources are profoundly unstable,” and unlike paper, they are not generally intelligible following even a modest amount of wear and tear (cohen & rosenzweig, 2005). websites are especially problematic because of the ephemeral nature of content on the web itself. the average half-life of a webpage has been estimated at two years (koehler, 2004), which is almost certainly too generous, and those that do survive tend to change frequently (dougherty et al., 2010). so while the web’s impact on society has continued to grow alongside an explosive growth of the content available, this rich trove of informational resources remains very much at risk. roger highfield (2014) of the telegraph captures the feeling of many librarians and archivists involved in web archiving when he states: “[i]f we’re not careful, historians will know more about the beginning of the past century than the start of this one.” it is this sense of urgency and obligation that in great part animates web archiving initiatives throughout the world. web archiving web archiving involves collecting data from the web, storing that data and preserving it at some level, and making it accessible to users (niu, 2012). julien masanès of the internet memory foundation, and former director of web archiving at the bibliothèque nationale de france (bnf), has identified three main technical approaches to web archiving (pennock, 2013): transactional, server-side, and client-side. “[t]ransactional archiving records content… presented to a user on a given date and time” by creating a record of all http requests and responses (pennock). this requires server-side configuration, which means the cooperation of a host. sitestory (http://mementoweb.github.io/sitestory) is an example of a transactional web archiving tool. developed at the los alamos national laboratory, sitestory enables a server, when it receives a request from a browser: …to send the response to that browser as expected, but also [to] push it into an associated web archive. as a result, all versions of all resources that are being requested by browsers are also available in the archive (van de sompel, 2012). server-side archiving also requires support from content hosts because it involves copying files directly from a server. server-side archiving can be employed when content, such as a database, cannot easily be captured via http requests. deeparc is an example of a server-side web archiving tool. developed by the bnf, it “transform[s] relational database content into xml for archiving purposes” (aubry & hafri, 2005). by far the most common form of web archiving is client-side, also referred to as remote harvesting. in this method, web crawlers use the http protocol to collect content directly from a server by crawling all the links associated with a specific ‘seed’ url. crawl behavior is determined by providing the crawler with instructions about crawl parameters, such as crawl depth (pennock). the internet archive the most well known client-side archiving technology was developed at the internet archive (ia), a remarkable organization founded in 1996 by mit graduate and silicon valley entrepreneur brewster kahle, with the express purpose of “build[ing] an internet library” (internet archive, n.d.a). it was only after kahle and his group’s groundbreaking work that “national libraries and archives around the world also began to see the importance of preserving this global resource [i.e. the web]” (grotke, 2011). as such, many national libraries have established web archiving initiatives. for example, as of 2012, the bnf’s archive consisted of 370 terabytes, or 18 billion pages. in 2013 the british library conducted the first crawl of all .uk websites, which led to the capture of 30 terabytes of data (webber, 2014). the library of congress had collected 250 terabytes by 2011, and library and archives canada’s collection consists of over 7 terabytes [1]. these and other national collections still pale in comparison to the ia, which contains over 2 petabytes of data and over 150 billion web pages, and the ia corpus grows by approximately 100 terabytes a month (internet archive, n.d.b)—larger than the entirety of most national library collections. to provide a sense of scale, the web archives of the library of congress—the world’s largest library—grows at the relatively meager rate of 5 terabytes a month (taylor, 2011). and while the ia has developed an incredibly important resource—the wayback machine gets close to 700,000 unique ip visits per day (internet archive, 2014)—in the words of clifford lynch, “terrifyingly, this major chunk of our cultural memory basically relies on the good offices and good spirit of brewster kahle” (lynch, 2014). the ia and its archiving technologies dominate the web archiving landscape. teams at ia developed the widely-used heritrix tool, a java application that crawls the web and stores content specifically for the purposes of web archiving (donovan, 2014). that content is stored in either the arc or warc format [2], both of which “specif[y] a method for combining multiple digital resources into an aggregate archive file together with related information” (library of congress, 2009). [3] this means that in addition to, for example, the html text, other files—images, javascript, media files, etc.—are combined into a single w/arc file, which also contains crawl-specific metadata to indicate “the time of harvesting, the ip address of the harvesting machine, internet media type (mime type) and response code for the harvest transaction, the purpose of harvesting, etc.” (library of congress, 2009). w/arc files can then be replayed by entering the original page’s url into an instance of the wayback machine, a java servlet application “that retrieves, displays, and indexes archived internet content” (donovan). the heritrix web crawler and the wayback machine are both open source [4], and have been extensively implemented at institutions ranging from the library of congress (n.d.) to the subscription-based web archiving service (was) of the california digital library (n.d.). warc is an iso standard (28500:2009), and w/arc files can be indexed for full-text search using open-source tools like nutchwax (developed by the internet archive and the nordic web archive), solr, or elasticsearch, all of which are based on lucene [5]. to serve the web archiving needs of smaller institutions, several saas tools have been brought to market in recent years. in 2005, the ia launched archive-it, a subscription-based service based on its core web archiving technologies, which it “developed, in particular, for memory institutions and state archives” in order to create curated collections not captured by larger web archiving efforts (information today, 2005). the california digital library’s web arching service (was) is another saas tool using heritrix, wayback, and nutchwax—all technologies developed initially at the ia¬—and is used by many institutions, including stanford and berkeley. archivethenet (also in saas) is offered by the internet memory foundation (imf) and is used, among others, by the british library to create the uk government web archive. university of victoria’s web archiving initiative with the ia and various national libraries and other organizations archiving the web, why would a research library want to get involved? to start with, ia crawls are broad and focus on top-level domains, and national library collections tend to emphasize government sites and country-specific domains (.ca, .fr, .uk, etc.). this approach needs to be paired “with deep, curated collections by theme or by site, tackled by other cultural heritage organizations” (grotke). in this way, web archives can be seen as an extension of existing collection activities meant to support faculty, student, and community research: …broad national web archive collections, which often only have limited accessibility for legal and technical reasons, may not meet the dispersed needs of individual researchers, and be in danger of providing a “one-size fits nobody” solution. archiving and providing access to individual historical web resources is the basic “must-have” of a web archive (hockx-yu, 2012). in 2013, the university of victoria libraries decided to subscribe to the archive-it service as part of a consortial license negotiated by the council of prairie and pacific university libraries (coppul). up to this point, while some at the library recognized the importance and possibilities of archiving websites—especially in support of archives and special collections—there was no clearly-articulated demand for such a service by librarians, archivists, or faculty. as such, when the coppul offer was announced and the library decided to participate in the archive-it subscription, it was without a preconceived vision of what a web archiving initiative would look like, or how it would fit into the library’s emerging strategies around digital scholarship. in fact, it took almost eight months before we identified specific websites and started initial test crawls using archive-it. it has only been since the start of 2014 that we have begun building collections in earnest. these currently include: an events-based collection on the 50th anniversary of the university; several thematic collections in support of our extensive archival and special collections holdings on anarchist and transgender issues, hyper-local news, and environmental organizations; collections of local government websites; and, digital humanities websites. the ia hosts and stores all of the data and provides a web-based interface through which relatively non-expert users can create collections. some organizational challenges considered library vs. archive while the web is often referred to as “the world’s largest library”, this analogy is disingenuous. academic library collections—or at least library acquisitions budgets—still tend to be dominated by books and journal articles (and the platforms and discovery services that make them accessible in either print or electronic form). relationships between these kinds of materials—most of which are considered discrete information objects—are not rigorously represented, and they tend to be grouped together based on some shared subjective quality, such as subject matter. this is not the case with archives, which consist largely of primary source materials organized according to very different principles: respect des fonds, provenance, and original order. at most universities, the administrative functions for library and archival acquisitions are distinct. websites do not respect such organizational arrangements. articles, reports, newsletters, and other ‘discrete published items’ are common on websites, but website structure—the relationship between documents—is also important. unlike items on a library shelf, “[w]eb documents at the page level (but also the site level) hardly ever make sense alone….[t]hey are mingled in a larger document network” (masanès, 2005). in this way, they are more like archival material, where original order—i.e. site structure—is essential to understanding the whole. and much web content, such as blogs and wikis, is of a kind whose print analogs would usually be found in collections of personal or organization documents in an archive. furthermore, the use of the term “archive” vis-à-vis web archiving, is also problematic, because in this context the term itself is “much more in keeping with the computing usage of archive as a back-up copy of information then the disciplinary perspective of archives” (owens, 2014). the organizational lines that often separate libraries from archives suggest that for web archiving initiatives to be successful, participants will have to be willing to work across traditional divides. jurisdictional issues—if not addressed upfront—can stall a web archiving initiative. in order to address this challenge, our initiative at the university of victoria is overseen by an archive-it working group with broad representation. this working group consists of people with technical and collections expertise, but also the associate university librarian for learning and research resources, the head of collections management services for the library, as well as the director of special collections and the university archivist. the ownership conundrum websites as objects of acquisition can blur the organizational lines between library and archive. but they are also problematic in the same way that a significant amount of open access journal literature is. the advent of open access content is a relatively new thing for librarians. we tend to focus on materials we have paid for. services like portico, lockss, and scholar’s portal attempt to tackle issues of preservation, but are mostly focused on subscription-based resources. and while work is currently being done by the public knowledge project (pkp) to address the long-term preservation of open journal system (ojs) content using lockss (public knowledge project, 2014), open access literature—especially from smaller publishers—tends to be more vulnerable to loss than expensive subscription-based content, simply because no single library necessarily has the incentive to invest in long-term preservation. websites tend to fall into this same category. unlike governmental organizations that have a mandate to collect certain types of websites, such as state or national libraries and archives, university libraries and archives rarely have such obligations. for individual selectors, the focus is generally on expending funds for books, journals, research databases, and other paid-for resources. there is often little incentive to build collections of archived websites, which involves a considerable amount of effort, especially when more pressing collection-building activities related to acquisitions are at hand. as such, any web archiving initiative will have to carefully consider how to engage librarians in the time-intensive process of collection building. legal issues an article for wired uk about archiving britain’s web contains the subtitle: “the legal nightmare explored” (scott, 2010). this is not far off the mark, and as of this writing, the university of victoria libraries is still in the process of finalizing our policy around web archiving. as such, we have not yet made the majority of our collections available to the public. basically, there are two approaches to dealing with the issue of copyright: opt-in and opt-out. with the opt-in approach, the institution undertaking the web archiving contacts the appropriate content owners to get explicit permission to capture their site. the opt-out approach simply captures sites of interest while respecting the robots.txt protocol, and take-down and other requests are handled after the fact on a case-by-case basis. the opt-in approach is best characterized by the british library’s approach to the creation of its open uk web archive prior to 2013. response rates to permission requests were just 24 percent. the ceo of the british library at the time, dame lynne brindley, considered this approach unsustainable because it would mean the capture of perhaps only one percent of uk websites (scott). all of this changed, of course, when legislation for legal deposit for non-print resources passed in the uk parliament in april 2013 (british library, 2013). as of this writing, the british library is working on its second annual capture of the entire uk domain (webber). the internet archive uses an opt-out approach, “that allows a website owner/content provider to remove access from the archive, and/or prevent their content from being captured by putting up a robots.txt exclusion on their website” (scott). if requested by the content owner, the ia will remove a site from public access via the wayback machine—but not, importantly, from its central index. because the scale of the web is so vast, and response rates to permission requests are so low, many consider the opt-out approach the only workable solution to web archiving. in the canadian context, this is the approach used by the university of alberta libraries (2013), which states: “when a site owner authorizes communication of their work to the public by telecommunication without technological restrictions, we view this as their implicit consent to the indexing and caching of their site.” this is the general approach the university of victoria is planning to take. and while this approach has not been tested in the courts, many universities in the u.s. and canada are conducting web archiving under the auspices of fair use (or its canadian equivalent, fair dealing), as outlined in the association of research libraries code of best practices in fair use for academic and research libraries (2012). this document states quite forcefully that: [web archives] represent a unique contribution to knowledge and pose no significant risks for owners of either the sites in question or third-party material to which those sites refer. in the absence of such collections, important information is likely to be lost to scholarship….it is fair use to create topically based collections of websites and other material from the internet and to make them available for scholarly use. the library of congress seems to strike a balance between the two perspectives. while they might seek explicit permission for some sites, “some notice at a minimum must be provided to the site owner” before their site is crawled and captured (library of congress collection development office, 2013). because of the lack of explicit legal precedence and the varying approaches to web archiving across multiple institutions in multiple jurisdictions, it is absolutely central to tackle the issue of permissions as early on as possible, often in consultation with legal counsel. technical issues the web has not only grown in size, but also in complexity. it was originally envisioned by berners-lee as a collection of documents, and up until the early 2000s, this mostly held true: in the 1990’s and even moving into the early 2000’s, most web pages and site designs were relatively simple. many used “hybrid” site models that enabled publishers to separate out resources into separate directories and to optimize for different usage scenarios and workflows, but the majority of the emphasis was on more traditional load balancing for scale vs [sic] integration across diverse hosts and services (netpreserve.org, 2012). things have certainly changed. as negulescu and rosenthal (2013) aptly put it, “it’s not your grandfather’s web anymore.” most websites are anything but simple. for example, an august 28, 2014 archive-it test crawl of the new york times homepage captured 235 urls from 61 individual hosts, including 85 images (53 jpegs, 18 gifs, and 14 pngs) and 35 javascripts. table 1: a file types report from archive-it for the homepage of the new york times, august 28, 2014. the file types report shows how many urls were captured for each specified file type. file type urls text/dns 62 image/jpeg 53 text/html 47 text/plain 36 application/x-javascript 24 image/gif 18 image/png 14 unknown 14 application/javascript 8 text/css 6 application/json 4 text/javascript 3 image/x-icon 2 text/json 2 application/x-shockwave-flash 1 application/xml 1 image/svg+xml 1 text/xml 1 the trouble with ajax websites are becoming increasingly reliant on technologies like javascript, xml, json, and ajax/j, which can be problematic for a traditional crawler like heritrix. a good example is the website for colonial despatches, a “digital archive [that] contains the original correspondence between the british colonial office and the colonies of vancouver island and british columbia” (holmes, n.d.). created by the humanities computing and media centre at the university of victoria, the site is based on a large exist database containing xml documents marked up in tei, and serves as a good example of some of the technical issues facing web archiving tools that use the client-side harvesting approach. while the website looks relatively simple, it is not. for example, the browse section (http://bcgenesis.uvic.ca/docsbydate.htm) uses ajax techniques to convert tei/xml called from the database, processes it via xslt into xhtml, and embeds it in the page using javascript and css. image 1: a simple version of the application structure of the colonial despatches website (http://bcgenesis.uvic.ca/development.htm). when users go to the browse page, they are presented with a javascript-driven list which expands as they click on list items: image 2: when users click on the final ‘link’, the javascript executes a call to the exist database to get the data related to a specific date. image 3:  this content is then embedded dynamically into the page. before the advent of ajax and similar functionality, dynamically generating xml content for display in a single page was not possible, and users would most likely be taken to another page via an anchor tag (< a>) containing a url with specific database query parameters. this anchor tag would enable the heritrix crawler to execute the specific database query via http, get the associated document, and capture it. in the above example, < span> tags are used in place of < a> tags, perhaps because <span> is more flexible in terms of display (most browsers have default display rules which can affect the appearance of < a> tags, whereas < span> is a kind of tabula rasa). here is the code for the last bullet in the list: <li>     <span class="getdocsbydate" onclick="getdocsbydate(this, '1846-09-07')"></span>     <div></div> </li> when the javascript is executed, the < div> tags are filled in with xslt-transformed and css-formatted tei/xml data from the exist database. because heritrix can only discover content by following links to that content from an initial ‘seed’ url, and because in this case the < span> tag is used in place of an anchor < a> tag, heritrix will not recognize this as a link and will therefore not execute the javascript necessary to create a link to query the database and generate the appropriate content for archiving. in fact, it’s not clear that this kind of dynamic menuing—even if < a> tags were used—would result in effective capture, because urls are generated through user interaction, and are therefore not a part of the page content or the associated javascript. the problems with dynamic, interactive content this example hints at one of the major limitations of all current crawlers, where “web pages that are generated via a database in response to a request from the user” are simply not captured (pennock). and this limitation is particularly serious as database-driven websites become the norm, and where content is increasingly generated dynamically based on user-initiated http requests, either through form-based queries or via ajax-driven user interactions. for example, the index page for the diary of robert graves 1935-39 and ancillary material (http://graves.uvic.ca/graves/site/index.xml) can only be accessed via form-based query, which makes it very problematic to access via the wayback machine [6]. more and more website content is being generated dynamically from database content called via user-initiated javascript. this development is almost certain to continue as html5 is more widely adopted in order to provide an app-like experience via mobile web browsers (i.e. websites will behave more like single-purpose apps, but without the constraints levied, for example, by apple’s app store). as rosenthal (2011) states, “the key impact of html5 is that, in effect, it changes the language of the web from html to javascript, from a static document description language to a programming language.” this means that traditional web crawlers like heritrix will have increasingly more trouble archiving materials effectively: “a major effort to develop techniques capable of collecting and preserving ajax and html5 is urgently needed” (rosenthal, 2011). no one knows this more than the folks at archive-it: web sites have become increasingly reliant on client-side script to render pages and to ensure an optimal viewing experience to users….for example, as a user navigates within a facebook page, content is delivered on demand through javascript when the user scrolls to an un-viewed section of their timeline. displaying content dynamically through client script allows sites to optimize the user experience and reduce the load on their servers. these optimizations, however, make it difficult for heritrix to discover resources that are necessary for optimal capture and display of archived content (reed, 2014). as of june 5th, 2014, archive-it has deployed a tool called umbra, to work alongside heritrix when crawling seed urls. according to archive-it: when a crawl is run using umbra, designated seeds are sent by heritrix to a separate process that mimics the way a browser would access the seed urls. this allows client-side script to be executed so that previously unavailable urls can be detected for heritrix to crawl. umbra also gives heritrix a flexible way to imitate human interactions with web sites that were previously not possible, such as executing javascript through clicking or hovering the mouse over different web page elements and scrolling down a page (reed). this is heartening, but the results so far have been mixed. crawls at the university of victoria since june 5th have effectively captured popular social media sites like facebook and twitter (for which umbra is optimized), but the above-mentioned issues with ajax in the colonial despatches site occurred during crawls in august 2014, two months after umbra was released. again, in this particular instance, the tag might be the problem, and an effective capture might result from using the < a> < a>tag instead, but this is unclear, and the basic issue remains: “crawlers are not able to see any content that is created dynamically” (google, 2014, june 14). these kinds of challenges are an issue for search engines like google, whose crawlers also have trouble with dynamic content. their webmaster guidelines talk about how to deal with some of these issues: use a text browser such as lynx to examine your site, because most search engine spiders see your site much as lynx would. if fancy features such as javascript, cookies, session ids, frames, dhtml, or flash keep you from seeing all of your site in a text browser, then search engine spiders may have trouble crawling your site (google, 2014). because of google’s overwhelming role in content discovery, content creators have an incentive to make their stuff as visible as possible to google’s spiders. as such—and especially for larger, more commercially-oriented sites—decisions are made early in the creation of sites that allow google to more effectively index content, such as the creation of site maps in xml. and while seo is very different than web archiving, the principles are the same. if the library is able to somehow influence decisions at an early stage in the development of websites, these sites can be optimized for web archiving. as such, we have recently started a series of conversations with humanities computing and media centre on issues of web archiving and digital preservation. we are discussing the possibility of creating some built-in functionality for dynamic sites that would automatically write content into flat html for capture by archive-it, perhaps similar to techniques for generating html snapshots for indexing ajax applications in google (google, 2014, june 18). this raises another important issue: what exactly are we trying to capture? in the case of the colonial despatches website, even if a perfect representation of the look-and-feel, functionality, and interactivity could be achieved via the use of a tool like archive-it, the exist database containing the marked-up tei/xml of the original transcribed documents would not be preserved. this database—which represents the majority of the project’s human effort—arguably has more value than the website itself. we have talked with the humanities computing and media centre about possible capture strategies that include a virtual machine snapshot of the entire os. in some cases the use of a tool like deeparc might be an option. our use of archive-it has stimulated these kinds of conversations, and it’s clear that this is an area in need of considerably more investigation and testing. access and digital preservation another area of consideration is digital preservation and access. warc files captured by archive-it are stored in the united states (hanna, 2014). for canadian institutions, this is problematic because the notice-and-takedown provisions of the american digital millennium copyright act (dcma) differ significantly from the notice-and-notice regime under canada’s copyright modernization act. this situation could potentially see the removal of content by the internet archive where that content might not have been removed had a similar take-down request been issued in canada. this is even more troubling considering tools like archive-it are used to capture sites where content owners might have compelling reasons beyond copyright to see that captured content is removed (politically-oriented and government sites, for example). where a convincing argument can be made to keep public access available to captured content, having that content located in the u.s. is problematic. digital preservation is another key issue for web archives. capturing websites in warc format for playback and full-text search is only a part of what is needed for true digital preservation. warc files backed up by the internet archive are susceptible to corruption. as david rosenthal has recognized, the ia can be thought of as a statistical sample of the whole web. some data loss is to be expected, and this loss introduces an acceptable amount of noise into a very large sample (rosenthal, 2010b). this is not necessarily problematic at scale, but for the purposes of a single institution, it is concerning. archive-it does provide users with warc files on request, which can then be managed locally. at the university of victoria, we are considering obtaining copies of our warc files for processing by archivematica and storage in the coppul private lockss network (coppul pln). archivematica is an “open-source digital preservation system that is designed to maintain standards-based, long-term access to collections of digital objects” (archivematica, n.d.). archivematica can ingest warc files, which are run through a series of processes based on the open archival information system reference model (iso-oais), which allows for future preservation planning based on format identification and other information. archivematica creates archival information packages (aips) from warcs using bagit. these ‘bagged’ files can then be uploaded to the coppul-pln, where a single copy, if damaged, can be repaired by other copies in the network. none of this potential archivematica/lockss integration development work is trivial, especially the creation of a manifest for lockss to harvest aips from archivematica, but it is important to recognize that jurisdictional issues and considerations of digital preservation may require the local management of warc files to ensure ongoing access and long-term preservation. conclusion the collections being created by the university of victoria libraries are unique in the world, and even though the act of capturing websites is a messy affair, there is real value in capturing content that would almost certainly not be around by the time today’s students become tomorrow’s researchers. the importance of the web as part of the human record cannot be overstated, but there are significant challenges any organization faces in starting a web archiving initiative. organizational, legal, and technical challenges all need to be addressed. notes [1] lists of website archiving initiatives are maintained at http://netpreserve.org/about/archivelist.php and http://en.wikipedia.org/wiki/list_of_web_archiving_initiatives [2] for sample arc and warc files from the internet archive, see https://archive.org/details/examplearcandwarcfiles [3] “the (w)arc format is a revision of the arc file format [arc_ia] developed in the mid-1990’s that was first used to store web crawls as sequences of content blocks harvested from the world wide web.” https://webarchive.jira.com/wiki/display/arih/archive-it+storage+and+preservation+policy [4] see https://webarchive.jira.com/wiki/display/heritrix/heritrix and http://archive-access.sourceforge.net/projects/wayback/ [5] “solr or elasticsearch may prove more versatile than nutchwax, which is designed to be implemented in large multi-server environments.” https://www.archivematica.org/wiki/websites [6] see https://wayback.archive-it.org/4376/20140807180501/http://graves.uvic.ca/graves/site/index.xml references archivematica. (n.d.). what is archivematica? retrieved from https://www.archivematica.org/wiki/main_page association of research libraries. (2012, january). code of best practices in fair use for academic and research libraries. retrieved from http://www.arl.org/storage/documents/publications/code-of-best-practices-fair-use.pdf aubry, s., & hafri, y. (2005). deeparc. retrieved from http://deeparc.sourceforge.net/ british library. (2013, april 4). click to save the nation’s digital memory. retrieved from http://pressandpolicy.bl.uk/press-releases/click-to-save-the-nation-s-digital-memory-61b.aspx california digital library. (n.d.). web archiving services faq. retrieved from http://webarchives.cdlib.org/faq cohen, d. j., & rosenzweig, r. (2005). the fragility of digital materials. in digital history: a guide to gathering, preserving, and presenting the past on the web. retrieved from http://chnm.gmu.edu/digitalhistory/preserving/1.php donovan, l. (2014, april 25). archive-it (ait) 4.5 technical overview. retrieved from https://webarchive.jira.com/wiki/display/arih/archive-it+%28ait%29+4.5+technical+overview dougherty, m., meyer, e. t., madsen, c., van den heuval, c., thomas, a., & wyatt, s. (2010). researcher engagement with web archives: state of the art. retrieved from http://repository.jisc.ac.uk/544/1/jisc-rewa_stateoftheart_august2010.pdf google, inc. (2014). webmaster guidelines. retrieved from https://support.google.com/webmasters/answer/35769?hl=en google, inc. (2014, june 14). what the user sees, what the crawler sees. retrieved from https://developers.google.com/webmasters/ajax-crawling/docs/learn-more google, inc. (2014, june 18). how do i create an html snapshot? retrieved from https://developers.google.com/webmasters/ajax-crawling/docs/html-snapshot grotke, a. (2011). web archiving at the library of congress. computers in libraries, 31(10). retrieved from http://www.infotoday.com/cilmag/dec11/grotke.shtml hanna, k. (2014, february 12). archive-it storage and preservation policy. retrieved from https://webarchive.jira.com/wiki/display/arih/archive-it+storage+and+preservation+policy highfield, r. (2014, january 6). can dna reign supreme in the digital dark age? the telegraph. retrieved from http://www.telegraph.co.uk/science/10553626/can-dna-reign-supreme-in-the-digital-dark-age.html hockx-yu, h. (2012, december 19). digital humanities and the study of the web and web archives. retrieved http://britishlibrary.typepad.co.uk/webarchive/2012/12/digital-humanities-and-the-study-of-the-web.html#sthash.cswkltsl.dpuf holmes, m. (n.d.). development of this site [colonial despatches]. retrieved from http://bcgenesis.uvic.ca/development.htm information today. (2005, november 28). internet archive offers new archive service. retrieved from http://newsbreaks.infotoday.com/digest/internet-archive-offers-new-archive-service-16063.asp internet archive. (2014, august). number of unique ip addresses per day. retrieved from https://archive.org/stats/ internet archive. (n.d.a). about the internet archive. retrieved from https://archive.org/about/ internet archive. (n.d.b). internet archive projects. retrieved from https://archive.org/projects/ koehler, w. (2004). a longitudinal study of web pages continued: a consideration of document persistence. information research, 9(2). retrieved from http://www.informationr.net/ir/9-2/paper174.html library of congress collection development office. (2013). library of congress collections policy statements supplementary guidelines: web archiving. retrieved from http://www.loc.gov/acq/devpol/webarchive.pdf library of congress. (2009). warc, web archive file format. retrieved from http://www.digitalpreservation.gov/formats/fdd/fdd000236.shtml library of congress. (n.d.). web archiving: technical information. retrieved from http://www.loc.gov/webarchiving/technical.html lynch, c. a. (2014, january 30). challenges of stewardship at scale in the digital age. retrieved from http://youtu.be/rfvllq2nzj0 masanès, j. (2005). web archiving methods and approaches: a comparative study. library trends, 54(1). retrieved from http://muse.jhu.edu/journals/library_trends/v054/54.1masanas.html negulescu, k. c., & rosenthal, d. s. (2013). not your grandfather’s web anymore. retrieved from http://www.cni.org/topics/digital-preservation/not-your-grandfathers-web-any-more/ netpreserve.org. (2012, may 17). iipc future of the web workshop: introduction & overview. retrieved from http://netpreserve.org/sites/default/files/resources/overviewfuturewebworkshop.pdf niu, j. (2012). an overview of web archiving. d-lib magazine, 18(3/4). retrieved from http://dlib.org/dlib/march12/niu/03niu1.html owens, t. (2014, february 27). what do you mean by archive? genres of usage for digital preservers. retrieved from http://blogs.loc.gov/digitalpreservation/2014/02/what-do-you-mean-by-archive-genres-of-usage-for-digital-preservers/ pennock, m. (2013). web archiving. retrieved from http://dx.doi.org/10.7207/twr13-01 public knowledge project. (2014, june 3). announcing forthcoming pkp lockss pln. retrieved from https://pkp.sfu.ca/announcing-forthcoming-pkp-lockss-pln/ reed, s. (2014, june 5). introduction to umbra. retrieved from https://webarchive.jira.com/wiki/display/arih/introduction+to+umbra rosenthal, d. s. (2010a). how green is digital preservation? retrieved from http://www.digitalpreservation.gov/meetings/documents/othermeetings/rosenthal.pdf rosenthal, d. s. (2010b). how are we ensuring the longevity of digital documents? retrieved from https://www.youtube.com/watch?v=h53dmtbuxsk rosenthal, d. s. (2011, august 22). moonalice plays palo alto. retrieved from http://blog.dshr.org/2011/08/moonalice-plays-palo-alto.html scott, k. (2010, march 5). archiving britain’s web: the legal nightmare explored. wired magazine. retrieved from http://www.wired.co.uk/news/archive/2010-03/05/archiving-britains-web-the-legal-nightmare-explored taylor, n. (2011, june 6). web and twitter archiving at the library of congress. retrieved from http://eventsarchive.org/sites/default/files/webandtwitterarchivingatthelibraryofcongress-110614202159-phpapp01.pdf university of alberta libraries. (2013). university of alberta libraries web archiving policy. retrieved from http://www.library.ualberta.ca/aboutus/collection/policy/web%20archiving%20policy.pdf van de sompel, h. (2012). sitestory transactional web archive software released. d-lib magazine, 18(9/10). retrieved from http://www.dlib.org/dlib/september12/09inbrief.html webber, j. (2014, june 12). how big is the uk web? retrieved from http://britishlibrary.typepad.co.uk/webarchive/2014/06/how-big-is-the-uk-web.html about the author corey davis has been with the university of victoria since 2012, where he works as systems librarian. he has a particular interest in the long-term preservation of born-digital materials. prior to joining uvic, he was head of technical services at royal roads university library. subscribe to comments: for this article | for all articles one response to "archiving the web: a case study from the university of victoria" please leave a response below: andy jackson, 2014-11-04 great article. just a minor correction – we (the british library) do not run the uk government web archive. the national archives of the uk does that. thanks. leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – drying our library’s libguides-based webpage by introducing vue.js mission editorial committee process and structure code4lib issue 55, 2023-1-20 drying our library’s libguides-based webpage by introducing vue.js at the kingsborough community college library, we recently decided to bring the library’s website more in line with dry principles (don’t repeat yourself). we felt we this could improve the site by creating more concise and maintainable code. dryer code would be easier to read, understand and edit. we adopted the vue.js framework in order to replace repetitive, hand-coded dropdown menus with programmatically generated markup. using vue allowed us to greatly simplify the html documents, while also improving maintainability. by mark e. eaton keeping it dry a common goal among programmers is to write code that is dry, in other words, code where you don’t repeat yourself. this is usually motivated by the insight that computers can often effectively automate repetitive tasks, making it unnecessary to repeat yourself in code. taking advantage of the efficiencies of automation is widely regarded as a best practice among programmers. however, html, when written by hand, is unfortunately not terribly suited to dry practices. html is particularly declarative: all elements of the page are explicitly laid out by the programmer, so as to fully describe its structure. the problem with this is that it means that hand-written webpages are often not very dry. even those of a relatively modest amount of complexity can quickly grow into very long html documents. this can be problematic, for a few reasons: it can become difficult to conceptualize the structure of a whole page when it stretches out over hundreds of lines. even relatively trivial aspects of coding, such as indentation, can become difficult with the deeply nested html structures of a large page. it is easy to introduce syntax errors or formatting problems into long html documents, because typos can be easily overlooked. this is especially problematic in cases where there is no built-in linting or validation.[1] at our college these challenges were familiar to us at kingsborough community college, a college of the city university of new york. our homepage, built on libguides cms, ran to over 500 lines, not including the <head> or <footer> sections. much of this was owing to repetitive dropdown menus: our page relies heavily upon bootstrap-based dropdown navigations to provide easy access to many of our services from the library homepage. these hand-coded menus, structured as lists of links, accounted for much of the length of the page’s source code. included below is the original code for our hamburger menu, which, despite its length, was in fact the shortest and simplest dropdown menu on our page: <div class="dropdown" id="hamburger-container"> <button class="btn btn-default dropdown-toggle" type="button" data-toggle="dropdown" aria-haspopup="true" aria-expanded="true" id="hamburger"> <i class="fas fa-bars" style="font-size: 2em;"></i> </button> <ul class="dropdown-menu fade" aria-labelledby="hamburger" id="hamburger-ul"> <li> <a class="searchmenu" aria-label="onesearch" href="https://library.kbcc.cuny.edu/onesearch"> <div class="highlight-menu-item bigger-fancy-text"> <i class="fas fa-search fa-fw bigger-icon" aria-hidden="true"></i> <strong>onesearch</strong> </div> </a> <a class="searchmenu" aria-label="databases a to z" href="https://library.kbcc.cuny.edu/az.php"> <div class="highlight-menu-item bigger-fancy-text"> <i class="fas fa-database fa-fw bigger-icon" aria-hidden="true"></i> <strong>databases a-z</strong> </div> </a> <a class="searchmenu" aria-label="research guides" href="https://library.kbcc.cuny.edu/guides"> <div class="highlight-menu-item bigger-fancy-text"> <i class="fas fa-telescope fa-fw bigger-icon" aria-hidden="true"></i> <strong>research guides</strong> </div> </a> <a class="searchmenu" aria-label="faq" href="https://library.kbcc.cuny.edu/faq"> <div class="highlight-menu-item bigger-fancy-text"> <i class="fas fa-question-circle fa-fw bigger-icon" aria-hidden="true"></i> <strong>faq</strong> </div> </a> <a class="searchmenu" aria-label="hours" href="https://library.kbcc.cuny.edu/calendar"> <div class="highlight-menu-item bigger-fancy-text"> <i class="fas fa-clock fa-fw bigger-icon" aria-hidden="true"></i> <strong>library hours</strong> </div> </a> <a class="searchmenu" aria-label="site map" href="https://library.kbcc.cuny.edu/sitemap"> <div class="highlight-menu-item bigger-fancy-text"> <i class="fas fa-location-arrow fa-fw bigger-icon" aria-hidden="true"></i> <strong>site map</strong> </div> </a> </li> </ul> </div> abstracting away some of that repetition was, in some important ways, an obvious win for the maintainers of the library webpage. there were clear benefits to abstraction. specifically, drying the page would: provide increased simplicity and maintainability; align us more with contemporary best practices in web development; allow us to write more aesthetically pleasing code; allow us to adopt and learn a modern javascript framework; raise the technical bar for what we are attempting to accomplish with our webpage. in brief, it would make the site better, and make life easier for the maintainers. these improvements were not undertaken without some hesitation. our library has non-technical librarians who work with libguides daily, and who may also want to edit our webpage. we were worried that adding another layer of abstraction might be confusing to them, as they would no longer be able to “see” the full html document object model (dom) to their satisfaction, and therefore no longer be able to properly understand and manipulate it themselves. this was an important concern. on the other hand, reducing the html devoted to dropdowns might in fact make other parts of the website more legible to our non-technical colleagues, because it would reduce the amount of noise that a non-expert user would need to filter through to accomplish their goals. in this sense, simplifying is also a way to improve access to the code. we decided to proceed because we felt that, in sum, the benefits out-weighed the drawbacks. the tradeoff is that it will make the page more maintainable for some, while it is perhaps of mixed benefit to others. this project was the best way we could find to address these issues in a balanced way, while making sustained progress on the further development of the site. selecting and using vue.js the tool we chose to do this work was vue.js (referred to as vue in the text that follows). vue is what is referred to as a “progressive” javascript framework, in that it aims to scale up, as well as scale down. scaling down was important to us, as our use case was not complicated, and we did not need the overhead of the complex build systems that are common to many javascript frameworks. we wanted something we could use within our cms. helpfully, it is possible to use vue in this way. we were able to import vue as a library with a simple call to a content delivery network (cdn), which allowed us to use it much in the same way that we would use other common libraries like bootstrap or jquery. we had access to many of vue’s abstractions by simply including <script src=”https://cdn.jsdelivr.net/npm/vue@2.7.8″> in our page, without necessitating other complex overhead. vue provides a very useful templating system to build html programmatically. we were familiar with html templating from previous work that we had done with python’s flask framework and its templating engine, jinja. jinja is somewhat conceptually similar to vue’s templating system, which helped us wrap our head around some parts of vue. however, in our opinion, vue provides added functionality beyond what is possible with jinja, such as two-way binding and an even more broad-based control of the dom. vue allowed us to write directives such as v-for, which is basically a for loop for constructing part of the dom. constructing a long list of links with a v-for loop is a huge improvement over typing out many lines of html. the content of the individual items, such as links, text, icons, and so on, can be stored as a javascript object in our vue constructor. vue’s directives allow us to draw content from this object, while structuring the html with the templating syntax. this approach gives us the full power of javascript and vue when building our html. it is truly much more powerful and accurate than patiently typing out html by hand. here is the html for the same hamburger menu as shown in the previous code snippet, but now dryed with vue: <div class="dropdown" id="hamburger-container"> <button class="btn btn-default dropdown-toggle" type="button" data-toggle="dropdown" aria-haspopup="true" aria-expanded="true" aria-label="hamburger menu" id="hamburger"> <i class="fas fa-bars" style="font-size: 2em;"></i> </button> <ul class="dropdown-menu fade" aria-labelledby="hamburger" id="hamburger-ul"> <li> <a v-for="item in hamburger" v-bind:key="item.id" class="searchmenu" v-bind:aria-label="item.description" v-bind:href="item.link" v-bind:target="item.target_blank"> <menus-component v-bind:item="item"></menus-component> </a> </li> </ul> </div> the corresponding vue component that makes this v-for loop happen looks like this: const menuscomponent = { template: `<div class="bigger-fancy-text"> <i v-bind:class="item.icon" aria-hidden="true"></i> <strong>{{ item.description }}</strong> </div>`, props: ['item'] } and the vue constructor: new vue({ el: '#app', components: { 'menus-component': menuscomponent … }, data() { return { hamburger: [ { link: "https://cuny-kb.primo.exlibrisgroup.com/discovery/search?tab=everything&vid=01cuny_kb:cuny_kb&lang=en", icon: "fas fa-search fa-fw bigger-icon", description: "onesearch", target_blank: "", id: 1 }, … we uploaded our vue code as a single file to the “upload customization files” section of the libguides cms admin interface. while this was quite effective, nonetheless, there were a couple of notable downsides: the “upload customization files” section of libguides cms is a bit hidden away in the admin interface. this is not our preference, but it is a design decision by springshare, the maker of libguides. the result of this is that someone new to the project, or new to libguides, might not immediately know where to look for the configuration files that are essential to rendering the page. it is very important that the uploaded javascript be valid, since formatting errors will mean that the data won’t load when the page loads, which will cause major problems. indeed, bad javascript often results in parts of the page not being rendered at all. the solution we adopted is to simply remember to validate our code before uploading it. we used babeljs (https://babeljs.io) for this purpose. babeljs allows us to paste in our code – for example, our vue constructor, including the data object – and it will flag any syntax errors. this is clearly not the most automated workflow, but it was simple enough to be an effective strategy for us.[2] assessing our approach this project was not an entire re-write of our library webpage in the idiom of vue.js. that was not our goal, and was far beyond the scope of this project. we simply took parts of the page that were easily dry-able and used vue to render them. for the most part, this process consisted of replacing the numerous hand-coded lists of links that produce our site’s menus. we focused on these lists because they were the principal offenders that made our page source too long and unmanageable. we tested this proposed approach ahead of time, by implementing vue’s v-for directive on the web librarian’s personal projects page. this trial run worked surprisingly seamlessly, with basically no problems of any significance. this gave us the confidence to move ahead implementing vue on the library homepage. and if there were major problems, we knew we could always use version control to roll back to a previous version, if needed (we use git and github for version control). nonetheless, the transition was not entirely without problems. our initial, most naive approach introduced new <div>s to the page, which broke the existing css. we initially (and mistakenly) thought that these problems were on account of vue, but they were in fact due to our css not behaving as expected on account of the new dom structure. in hindsight, it should have been obvious to us that changing the page structure would break the css. the good news was that our vue code was fine and was more or less working as expected. tweaking the html created by vue – so as not to break our css – was entirely doable. we solved the problem by configuring our v-for loops so as to faithfully recreate the original dom structure, which allowed the css to work properly and as expected. in this way, the project was completed without needing to rewrite any of our css. moreover, we found that besides iterating through individual menus using vue, we could dry the next level of our webpage’s hierarchy by having vue iterate through the list of menus (in other words, through the entire nav bar), so as to create another layer of abstraction, automation, and benefit to the maintainers. while there appears to be more than one way to tackle this problem, we settled upon creating a second vue component to handle the second-order drying logic. this higher-order abstraction is new to us, and we expect to make it better and more efficient over the coming weeks and months. these limited goals and constrained scope for this project meant that the changes that we attempted were not too overwhelming to implement. we rolled out our improvements within a few weeks, without disrupting other, existing workflows. we did this work in our “sandbox” libguides group, before moving the code over to our production group. we did this to avoid breaking, even temporarily, the production homepage. one especially pleasing part of this project was that, from the perspective of our users, there was no change at all to our website. from a user’s point of view, the site remained identical. we were able to increase the maintainability of the site while causing no end-user effects, which was a big win. to look at these changes quantitatively, our homepage was initially 501 lines of html, and after applying our drying techniques, was 279 lines. this is a difference of 44%. while our goal is not to play code golf, we felt that this alone made this project a worthwhile endeavor. of course this appealing headline number is somewhat offset by the added cognitive load (and lines of code) of the vue components and constructor, but nonetheless it is fair to say that the page is more concise. the new logic builds much of the dom automatically, with less room for human error. conclusion the result of our work is a more manageable and concise html document, which provides efficiencies in maintainability. adopting vue for our limited use case turned out to be a good decision. we hope to find compelling reasons to go further with vue’s more advanced features in the future. for example, as a next step, we intend to move more of the page’s logic to vue methods and computed properties. one possible outcome of this project is that it may eventually lead to a full rewrite of the webpage which more fully embraces the vue idiom. the chance to adopt vue as the principal organizing framework for our page offers many exciting new possibilities beyond our current bootstrap-oriented setup. vue could expand the functionality available to us, and truly move the needle when it eventually comes time to fully redesign our webpage. drying our project is a small first step in that direction. endnotes [1] the platform we use, libguides cms, does not provide built-in linting or validation. [2] interestingly, this is a case where a more sophisticated build system would be an advantage, as linting and validation could be included as part of the build. this is something for us to consider in the future, if we decide to adopt vue further. about the author mark e. eaton is a reader services librarian (associate professor) at kingsborough community college (city university of new york). subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – patron-driven expedited cataloging enhancement to webpac pro mission editorial committee process and structure code4lib issue 18, 2012-10-03 patron-driven expedited cataloging enhancement to webpac pro this article outlines the development of an integrated patron-driven expedited cataloging feature in the catalog of the connecticut state university library system (consuls). the proposed enhancement to the library’s innovative millennium ils provides users with a direct method for obtaining newly-arrived library materials and allows the cataloging & metadata services departments at the four connecticut state university campuses a way to better identify priority materials in their queues. while the project was developed with a single ils in mind, the idea behind it can easily be implemented on most any other integrated library system. two versions of the enhancement are covered: one for standalone libraries and one for libraries that share a union catalog. the source-codes for both versions of the enhancement are provided, as are instructions for implementing the enhancement on any millennium installation. by steven jay bernstein backlogs, frontlogs, and patron driven expedited cataloging in the world of catalog librarianship, backlogs are an unfortunate fact of life. as solomon so eloquently stated three thousand years ago, “of making many books there is no end” (eccles. 12:12; jps). and of those with the solomonic wisdom to catalog and classify those hoards of books there is practically no beginning. in libraries across the world a not-so-insignificant fraction of recently and formerly acquired materials lies dormant, waiting for their overburdened stewards to provide the description and access that will connect them with their readers. allowing for those connections to be made means reducing the backlog, a task for which there are many approaches. one popular approach is the transformation of the backlog into a “frontlog”, or a backlog in which quick-and-dirty minimal-level bibliographic records have been created for the as yet to be fully cataloged materials (nof 2011). records for frontlog materials often consist solely of descriptive data relating to the book’s author, title, imprint, edition, and international standard book number as well as a sequential accession number and/or barcode for inventory purposes. more often than not, these brief records—colloquially referred to as “baby bibs”—are created at the time of purchase or receipt by acquisitions staff and require no further effort on the part of the cataloging department until they are retrieved from the frontlog for full-level cataloging. although baby bibs often lack—among other things—subject analysis and authorized access points, they are still accessible through keyword searching provided that a) the book’s title adequately describes the book’s subject matter; or b) the user has foreknowledge of the author, title, or isbn she is seeking. this is perhaps the greatest benefit of the frontlog model: the ability of the library user to find uncataloged materials in the catalog and the ability of the cataloging department to then rush-catalog those materials upon request. jumping on the bandwagon of the recent buzz surrounding pda (patron driven acquisitions), we call this benefit patron driven expedited cataloging. the benefit of pdec has one missing link, however: the disconnect for the user between finding the item in the catalog and being able to access the item itself in the real world. whereas records for fully cataloged items contain classification numbers for locating the item in the stacks, records for frontlog items often contain nothing more than a note indicating that the item is in processing. there is no signpost for the user telling her how to retrieve such an item. and let’s face it, she’s not going to ask; she’s just going to give up. in the off chance that she does not give up she may discover a feature in the catalog for requesting a hold or a recall on the item in question. this function would fall short of providing her with prompt access to the item, however, since the request feature would not generate an alert to library staff when her request was placed. such items would only be identifiable through happenstance upon retrieval from the frontlog by a cataloger. without an alert, the user’s request might not be acted upon until long after she no longer needed the item. in an effort to bypass this dead end, central connecticut state university developed a rush cataloging request form that would notify the cataloging staff when a user needed an item from the frontlog. the form was placed on the cataloging department page of the library website. however, since no direction was provided to the library user as to how to get to the form from within the catalog it was seldom used. this project began as a response to that problem. background of the library environment at ccsu central connecticut state university is a regional, comprehensive public university dedicated to learning in the liberal arts and sciences and to education for the professions.  it is one of four universities in the connecticut state university system.  ccsu’s elihu burritt library shares a union catalog with the libraries at its three sister schools (eastern, southern, and western) as well as the connecticut state library.  the connecticut state university library system, (consuls) uses the millennium ils by innovative interfaces incorporated, release 2009b 1.2. project development stage one: appending a link in its early stages, the aim of the project was merely to append to ccsu’s in-processing notes a link to the rush cataloging request form so that the notes would read: 1 copy being processed for ccsu. click here to expedite processing to accomplish this we first needed to assign a unique identifier to the element in the html of the record display that contained the in-processing notes.  the element happened to be a table element with a class attribute having the value of “biborder”.  we modified the element by going into the web options tab of the administration module of millennium and altering the “record displays > tableparam_bib_order” parameter so that it would include the following: id=”biborder” assigning a unique identifier to this element allowed us to access and modify its contents using the following javascript which we included in the “toplogo.html” file: <script type="text/javascript"> function rushccsu() { <span style="color: #ff6600;">// function to append a link to ccsu in-processing notes, which-when clicked--brings the user to the rush cataloging form.</span> try { for (var i = 0; i < document.getelementbyid('biborder').rows.length; i++) { // for every row of order notes notes... var rush = document.getelementbyid('biborder').rows&#91;i&#93;.cells&#91;0&#93;; // ...declare a variable called "rush" to hold the cells in that row. if (rush.innerhtml.search('ccsu') == -1 || rush.innerhtml.search('processed') == -1) { } // if the html contained in any instance of "rush" does *not* contain the text string "ccsu" or "processed" then do nothing. else { // if, however, the html contained in any instance of "rush" *does* contain those text strings, then... rush.innerhtml += ' <a href="http://www.library.ccsu.edu/services/rushcat.php">click here to expedite processing</a>'; // ...append to that instance of "rush" a link to the rush cataloging form which will have the display text "click here to expedite processing". } } } catch(err) { } } window.onload=rushccsu; // loads the rushccsu function when the page loads </script> like the cataloging done for frontlog materials, this solution was quick-and-dirty. while it got the job done, it was inelegant. the style of the page on which the rush cataloging form appeared did not match the style of the catalog itself. more importantly, linking to the form on a separate page required the user to remember the details about the item she was requesting. having to navigate back and forth between the record and the form was an exercise in user unfriendliness. these perceived shortcomings gave rise to the second stage of the project in which the form was integrated into the catalog itself as a popup. stage two: integrating the form into the catalog in order to integrate the form into the catalog as a popup, the requisite html for the form was added to the toplogo.html file.  this html was placed inside an absolutely positioned div element whose display was set to ‘none’ and modified so that the style of the form would fit with the style of the opac.  as we were making these changes, we also replaced the underlying script that controlled the e-mail submission of the form with the freely available huggins’ email form script by james s. huggins.  this php script file, which—due to the millennium server’s inability to process php—was placed on ccsu’s webserver, provided a much more robust set of options for securing and customizing the form than the original script.  this customizability came in quite handy in the final stage of the project when the enhancement was modified for use by all four connecticut state university libraries. <div id="rushformccsu" style="display:none; position:absolute; top:65px; border:2px solid #00437f; color:#000000; background-color:#ffffff; width:500px; left:50%; margin-left:-250px; padding-bottom:5px;"> // the container for the form. it is not displayed by default; it appears when the user clicks on the link to display the form. <div style="font-weight:bold; color:#00437f; background-color:#d7e3ff; height:20px; padding:5px;"> // the title bar for the form. <span style="float:left;">rush cataloging request form (<span style="color:red;">ccsu only</span>)</span> // the title of the form, indicating that this form is for use by library users affiliated with central connecticut state university and not any other csu campus. <span style="float:right; height:20px;"> <a href="javascript:showhide('about', 'inline')" title="about the rush cataloging request form" style="display:inline-block; font-size:8pt; line-height:12pt; text-decoration:none; color:#00437f; background-color:inherit; border:1px solid #00437f; padding:1px;">&nbsp;?&nbsp;</a> // an "about" button for toggling the display of information about the project, including its version number and a link to the project page of the php script that was used for the form. &nbsp;<a href="javascript:showhide('rushformccsu', 'none')" title="close" style="display:inline-block; font-size:8pt; line-height:12pt; text-decoration:none; color:#00437f; background-color:inherit; border:1px solid #00437f; padding:1px;">&nbsp;x&nbsp;</a> // a "close" button to hide the form if the user does not wish to submit the request. </span> </div> <div style="padding:5px; text-align:justify;">fill out the form below to expedite the processing of this item. cataloging and metadata services will rush catalog it and you will be notified via e-mail when it is available. the item will be held at the circulation desk under your name for 10 days.</div> // the instructions for the form. <form action="huggins-email-form-script-v2.2.3.php" method="post" id="formccsu" style="display:inline;"> // the form. <input name="formechouser" type="hidden" value="yes"/> // a parameter for the form instructing it to send a confirmation e-mail to the submitter. <input name="formemailfieldlist" type="hidden" value="email_address, confirm_email_address"/> // a parameter for the form indicating which field(s) contain the submitter’s e-mail address. <input name="formfieldnamelabelpluslist" type="hidden" value= "labelline, item to be cataloged..., record, record, y, 0, 0 | author, author, n, 0, 0 | title, title, n, 0, 0 | publisher, publisher, n, 0, 0 | isbn, isbn, n, 0, 0 | blankline, labelline, person for whom the cataloging is to be done..., name, name, y, 0, 0 | id, bluechip id, y, 8, 8 | email_address, email address, y, 0, 60 | confirm_email_address, confirm email address, y, 0, 60 | test78, are you human?, y, 0, 10 | status, status, y, 0, 0 | blankline, labelline, technical details about form submission..., ip, ip address, n, 0, 0 | browser, browser information, n, 0, 0 |"/> // a parameter for the form which specifies the names of all the fields in the form, the label that should be used for each field in the resulting e-mail message, whether the field is required, and its minimum and maximum length. <input name="formfieldnameeditlist" type="hidden" value="test78, captcha, hot, ~"/> // a parameter for the form indicating the required answers for various fields in the form. <input name="formnamefieldlist" type="hidden" value="name"/> // a parameter for the form indicating which field(s) contain the name of the submitter. <input name="formnexturl" type="hidden" value="rush%20cataloging%20form%20landing%20page.htm"/> // a parameter for the form indicating the url of the landing page that will be displayed upon successfully submitting the form. <input name="msg1addrlist" type="hidden" value="john smith, johnsmith, college.edu, jane doe, janedoe, college.edu"/> // a parameter for the form indicating the e-mail address(es) of individual(s) who will receive the completed form. <input name="msg1fieldnameexcludelist" type="hidden" value="confirm_email_address, test78"/> // a parameter for the form indicating which fields are to be excluded from the resulting e-mail message. <input name="msg1subject" type="hidden" value="rush cataloging request form submission"/> // a parameter for the form indicating the subject line of the resulting e-mail message. <input name="msg1textbottom" type="hidden" value="please complete cataloging within one business day. thank you."/> // a parameter for the form indicating the text that is to appear in the resulting e-mail message below the form data. <input name="msg1texttop" type="hidden" value="a rush cataloging request has been submitted for the following item:"/> // a parameter for the form indicating the text that is to appear in the resulting e-mail message above the form data. <input name="msgechofieldnameexcludelist" type="hidden" value="record, confirm_email_address, test78"/> // a parameter for the form indicating which fields are to be excluded from the confirmation e-mail sent to the submitter. <input name="msgechofromaddr" type="hidden" value="jane doe, janedoe, college.edu"/> // a parameter for the form indicating the "from" address for the confirmation e-mail sent to the submitter. <input name="msgechosubject" type="hidden" value="rush cataloging request form confirmation"/> // a parameter for the form indicating the subject line for the confirmation e-mail sent to the submitter. <input name="msgechotextbottom" type="hidden" value="thank you for submitting your rush cataloging request. cataloging and metadata services will rush catalog the item you have requested and you will be notified via e-mail when it is available. the item will be held at the circulation desk under your name for 10 days. do not reply to this e-mail."/> // a parameter for the form indicating the text that is to appear in the confirmation e-mail message below the form data. <input name="msgechotexttop" type="hidden" value="you have submitted a rush cataloging request for the following item:"/> // a parameter for the form indicating the text that is to appear in the confirmation e-mail message above the form data. <table style="margin-left:5px; margin-right:5px;"> // a table to layout the form. <tr> <td colspan="3"><hr style="width:100%; height:2px; background-color:#00437f; color:#00437f; border-style:none;"/></td> </tr> <tr> // the form field into which the user inputs the author of the item being requested. <td style="background-color:#d7e3ff; font-size:10pt; font-weight:bold; font-style:italic; width:240px; height:30px; text-align:right; padding-right:10px;"><label for="authorccsu">author</label></td> <td colspan="2"><div style="margin-right:5px;"><input type="text" id="authorccsu" name="authorccsu" style="width:100%; padding:0px;"/></div></td> </tr> <tr> // the form field into which the user inputs the title of the item being requested. <td style="background-color:#d7e3ff; font-size:10pt; font-weight:bold; font-style:italic; width:240px; height:30px; text-align:right; padding-right:10px;"><label for="booktitleccsu">title</label></td> <td colspan="2"><div style="margin-right:5px;"><input type="text" id="booktitleccsu" name="booktitleccsu" style="width:100%; padding:0px;"/></div></td> </tr> <tr> // the form field into which the user inputs the publisher of the item being requested. <td style="background-color:#d7e3ff; font-size:10pt; font-weight:bold; font-style:italic; width:240px; height:30px; text-align:right; padding-right:10px;"><label for="publisherccsu">publisher</label></td> <td colspan="2"><div style="margin-right:5px;"><input type="text" id="publisherccsu" name="publisherccsu" style="width:100%; padding:0px;"/></div></td> </tr> <tr> // the form field into which the user inputs the isbn of the item being requested. <td style="background-color:#d7e3ff; font-size:10pt; font-weight:bold; font-style:italic; width:240px; height:30px; text-align:right; padding-right:10px;"><label for="isbnccsu">isbn</label></td> <td colspan="2"><div style="margin-right:5px;"><input type="text" id="isbnccsu" name="isbnccsu" style="width:100%; padding:0px;"/></div></td> </tr> <tr> // the form field into which the user inputs his or her name. <td style="background-color:#d7e3ff; font-size:10pt; font-weight:bold; font-style:italic; width:240px; height:30px; text-align:right; padding-right:10px;"><label for="nameccsu">your name</label></td> <td colspan="2"><div style="margin-right:5px;"><input type="text" id="nameccsu" name="name" style="width:100%; padding:0px;"/></div></td> </tr> <tr> // the form field into which the user inputs his or her id number. <td style="background-color:#d7e3ff; font-size:10pt; font-weight:bold; font-style:italic; width:240px; height:30px; text-align:right; padding-right:10px;"><label for="idccsu">your 8-digit bluechip id no.</label></td> <td colspan="2"><div style="margin-right:5px;"><input type="text" id="idccsu" name="id" style="width:100%; padding:0px;"/></div></td> </tr> <tr> // the form field into which the user inputs his or her e-mail address. <td style="background-color:#d7e3ff; font-size:10pt; font-weight:bold; font-style:italic; width:240px; height:30px; text-align:right; padding-right:10px;"><label for="emailaddrccsu">your e-mail address</label></td> <td colspan="2"><div style="margin-right:5px;"><input type="text" id="emailaddrccsu" name="email_address" style="width:100%; padding:0px;"/></div></td> </tr> <tr> // the form field into which the user confirms his or her e-mail address. <td style="background-color:#d7e3ff; font-size:10pt; font-weight:bold; font-style:italic; width:240px; height:30px; text-align:right; padding-right:10px;"><label for="confirmemailaddrccsu">confirm your e-mail address</label></td> <td colspan="2"><div style="margin-right:5px;"><input type="text" id="confirmemailaddrccsu" name="confirm_email_address" style="width:100%; padding:0px;"/></div></td> </tr> <tr> // the form field into which the user inputs his or her status. <td style="background-color:#d7e3ff; font-size:10pt; font-weight:bold; font-style:italic; width:240px; height:30px; text-align:right; padding-right:10px;"><label for="statusccsu">your status</label></td> <td style="font-size:10pt; width:120px;"><input type="radio" value="fac/staff" checked="checked" id="facstaffccsu" name="status"/><label for="facstaffccsu">faculty/staff</label></td> <td style="font-size:10pt; width:120px;"><input type="radio" value="student" id="studentccsu" name="status"/><label for="studentccsu">student</label></td> </tr> <tr> // a captcha field for weeding out spam. <td style="background-color:#d7e3ff; font-size:10pt; font-weight:bold; font-style:italic; width:240px; height:30px; text-align:right; padding-right:10px;"><label for="humanccsu">are you human?</label></td> <td colspan="2"><span style="float:right;">winter is cold; summer is: <input type="text" id="humanccsu" name="test78" style="display:inline" size="1"/></span></td> </tr> <tr> <td colspan="3"><hr style="width:100%; height:2px; background-color:#00437f; color:#00437f; border-style:none;"/></td> </tr> <tr> // the submit button for the form. <td colspan="2" style="color:red; font-size:10pt;">after you have clicked 'submit' you can click your browser's back button to return to this record.</td> <td colspan="1" style="text-align:right;"><button type="submit" style="display:inline-block;">submit</button></td> </tr> </table> </form> </div> <div id="about" style="display:none; z-index:1; position:absolute; top:200px; border:2px solid #00437f; color:#000000; background-color:#ffffff; width:350px; left:50%; margin-left:-175px; padding-bottom:5px;"> // the container for the "about" popup. it is not displayed by default; it appears when the user clicks on the about link on the form. <div style="font-weight:bold; color:#00437f; background-color:#d7e3ff; height:20px; padding:5px;"> // the title bar for the "about" popup. <span style="float:left;">about the rush cataloging request form</span> // the title of the "about" popup. <span style="float:right; height:20px;"><a href="javascript:showhide('about', 'none')" title="close" style="font-size:8pt; line-height:1.66em; text-decoration:none; color:#00437f; background-color:inherit; border:1px solid #00437f; padding:1px;">&nbsp;x&nbsp;</a></span> // a "close" button to hide the "about" popup after the user is done. </div> <div style="padding:5px; text-align:justify;">version 1.1 beta<br/><br/>the rush cataloging request form was developed at central connecticut state university by steven bernstein and uses <a title="huggins’ email form script" target="_blank" href="http://www.jamesshuggins.com/huggins-email-form-script">huggins’ email form script</a>.<br/></div> // the contents of the "about" popup, including its version number and a link to the project page of the php script that was used for the form. </div> one of the features of huggins’ e-mail form script was the ability to specify a landing page for the user to serve as a confirmation that the form has been successfully submitted. we created such a page as follows and placed the html file for the page in the screens directory of the webpac server: figure 1. the landing page. we next had to add a function to the javascript that would toggle the display property of the div element in which the form was contained. function showhide(divid, state){ // function to show or hide the form. document.getelementbyid(divid).style.display=state } finally, the rushccsu() function was modified so that the link that was to be appended to the in-processing notes would call the showhide() function for displaying the form rather than linking to the original form. <script type="text/javascript"> function rushccsu() { // function to append a link to ccsu in-processing notes, which--when clicked—brings the user to the rush cataloging form. try { for (var i = 0; i < document.getelementbyid('biborder').rows.length; i++) { // for every row of order notes notes... var rush = document.getelementbyid('biborder').rows&#91;i&#93;.cells&#91;0&#93;; // ...declare a variable called "rush" to hold the cells in that row. if (rush.innerhtml.search('ccsu') == -1 || rush.innerhtml.search('processed') == -1) { } // if the html contained in any instance of "rush" does *not* contain the text string "ccsu" or "processed" then do nothing. else { // if, however, the html contained in any instance of "rush" *does* contain those text strings, then... rush.innerhtml += ' <a href="javascript:showhide(\'rushformccsu\', \'inline\'); fillin()">click here to expedite processing</a>'; // ...append to that instance of "rush" a link that will call the showhide() function and which will have the display text "click here to expedite processing". } } } catch(err) { } } figure 2. the pop-up form. the resulting form had a much more consistent appearance and felt more like a natural part of the opac and less like an add-on to it. despite the fact that the form was now on the same screen as the record, a potential problem remained: the user would still need to remember the exact details about the item she was requesting. this was due to the fact that when the form was displayed, there was the possibility that the content of some of the record’s fields would be hidden behind it. our initial thought regarding how to remedy this was to make the div element in which the form was contained movable on the screen. on further reflection, we decided instead to circumvent the issue of user input of the record’s metadata entirely by adding a javascript to the project that would automatically fill in those form fields meant to contain bibliographic data. stage three: adding auto-fill the idea to have the form fields for bibliographic data automatically filled in came about after playing with another enhancement that had already been implemented in consuls for sending an item’s title and call number to one’s mobile device via short message service (i.e. texting). the “send sms from your iii catalog” script was developed by adam brin at bryn mawr college. examining the workings of his javascript, we were able to write a function to suit our needs: function fillin() { // function to cull the bibliographic information from the record for inclusion in the form fields. var author = ''; var title = ''; var publisher = ''; var isbn = ''; // declare empty eponymous variables to hold the author, title, imprint, and isbn of the record. var bibid = document.getelementbyid('recordnum').innerhtml.replace("http://www.consuls.org:80/record=","").replace("~s16","") + 'a'; // declare an eponymous variable to hold the bibid of the record, taken from the record's persistent link. try { var tr = document.getelementsbytagname('tr'); for(i = 0; i < tr.length; i++) { // for every row in the record... var x=tr&#91;i&#93;.getelementsbytagname('td'); // ...declare a variable called "x" to hold the cells in that row. if (x.length == 2 && x&#91;0&#93;.innerhtml == "author") { // if the row has 2 cells and the first cell contains the text string "author"... author = x&#91;1&#93;.innerhtml.replace(/(<(&#91;^>]+)>)/ig,""); // ...then strip of any markup from the text string contained in the second cell of that row and assign it to the "author" variable. } if (x.length == 2 && x[0].innerhtml == "title") { // if the row has 2 cells and the first cell contains the text string "title"... title = x[1].innerhtml.replace(/(<(&#91;^>]+)>)/ig,""); // ...then strip of any markup from the text string contained in the second cell of that row and assign it to the "title" variable. } if (x.length == 2 && x[0].innerhtml == "publisher") { // if the row has 2 cells and the first cell contains the text string "publisher"... publisher = x[1].innerhtml.replace(/(<(&#91;^>]+)>)/ig,""); // ...then strip of any markup from the text string contained in the second cell of that row and assign it to the "publisher" variable. } if(x.length == 2 && x[0].innerhtml == "isbn") { // if the row has 2 cells and the first cell contains the text string "isbn"... isbn = x[1].innerhtml.replace(/(<(&#91;^>]+)>)/ig,""); // ...then strip of any markup from the text string contained in the second cell of that row and assign it to the "isbn" variable. } } } catch (e) {} document.getelementbyid('authorccsu').value = author; document.getelementbyid('booktitleccsu').value = title; document.getelementbyid('publisherccsu').value = publisher; document.getelementbyid('isbnccsu').value = isbn; document.getelementbyid('bibidccsu').value = bibid; // assign the text strings held by the "author", "title", "publisher", "isbn", and "bibid" variables as values for the corresponding fields in the form. } both this function and the rushccsu() function needed to be called at the time of the page load. to accomplish this, another function was created called start() that would call both functions at once. the global event handler that called rushccsu() on the page’s load was changed to call the start() function instead. function start() { rushccsu(); fillin(); } window.onload=start; one of the benefits of having the form filled in automatically was that it removed the possibility of transcription error on the part of the library user. to further ensure that no typographical errors would creep into the metadata submitted by the form, we set the author, title, and publisher fields to be read-only and modified their styles so that their contents would look like text on the page rather than data in a form field. another benefit of having the form filled in automatically was that we could cull certain metadata from the record about which the user need not be concerned and do so behind the scenes. of particular interest to us was the ability to include the record’s bibliographic record number in the e-mail that the cataloging staff would receive. we added the “bibid” to the form as a hidden field. <tr style="display:none;"> <td style="background-color:#d7e3ff; font-size:10pt; font-weight:bold; font-style:italic; width:240px; height:30px; text-align:right; padding-right:10px;"><label for="recordccsu">bibid</label></td> <td colspan="2"><div style="margin-right:5px;"><input type="text" id="bibidccsu" name="record" style="width:100%; border:none; padding:0px;" readonly="readonly"/></div></td> </tr> <tr> <td style="background-color:#d7e3ff; font-size:10pt; font-weight:bold; font-style:italic; width:240px; height:30px; text-align:right; padding-right:10px;"><label for="authorccsu">author</label></td> <td colspan="2"><div style="margin-right:5px;"><input type="text" id="authorccsu" name="author" style="width:100%; border:none; padding:0px;" readonly="readonly"/></div></td> </tr> <tr> <td style="background-color:#d7e3ff; font-size:10pt; font-weight:bold; font-style:italic; width:240px; height:30px; text-align:right; padding-right:10px;"><label for="booktitleccsu">title</label></td> <td colspan="2"><div style="margin-right:5px;"><input type="text" id="booktitleccsu" name="title" style="width:100%; border:none; padding:0px;" readonly="readonly"/></div></td> </tr> <tr> <td style="background-color:#d7e3ff; font-size:10pt; font-weight:bold; font-style:italic; width:240px; height:30px; text-align:right; padding-right:10px;"><label for="publisherccsu">publisher</label></td> <td colspan="2"><div style="margin-right:5px;"><input type="text" id="publisherccsu" name="publisher" style="width:100%; border:none; padding:0px;" readonly="readonly"/></div></td> </tr> <tr style="display:none;"> <td style="background-color:#d7e3ff; font-size:10pt; font-weight:bold; font-style:italic; width:240px; height:30px; text-align:right; padding-right:10px;"><label for="isbnccsu">isbn</label></td> <td colspan="2"><div style="margin-right:5px;"><input type="text" id="isbnccsu" name="isbn" style="width:100%; border:none; padding:0px;" readonly="readonly"/></div></td> </tr> stage four: modifying the project to include the other campuses at this point, the project was pretty much complete. however, when word about the enhancement got around to the other consuls libraries, the other three campuses wanted in as well. the connecticut state library, however—having circulation policies that differed considerably from those of the libraries at the four connecticut state universities—was not interested in participating. the final stage of the project involved modifying the enhancement to append form links to the in-processing notes of the other three campuses and creating variant forms for each. before this revision could be started the following information needed to be gathered from each library that was to be involved: 1) the library’s policy with regard to the turnaround time for requested items. 2) the library’s policy with regard to the length of time that requested items would remain at the circulation desk. 3) the email address(es) of the library staff member(s) to whom rush requests would be submitted. 4) the singular e-mail address from which the library user who submitted the form would receive her confirmation. 5) the location code for those collections for which rush cataloging would not be made available (if any). once this information had been gathered we were able to incorporate it into three new forms. the new forms were nothing more than copy-and-pastes of the original ccsu form, but with the following changes: 1) textual edits were made to brand each form so as to make it clear that the form was meant for use by that campus’ constituents only. 2) both the e-mail addresses to which the form was to be submitted and the e-mail address from which the user was to receive her confirmation were replaced with the e-mail addresses specified by each campus. 3) most importantly, the values of and references to the id attributes for the various html elements in the form were changed from including ‘ccsu’ in their strings to including ‘ecsu’, ‘scsu’, or ‘wcsu’, respectively. thus: id="rushformccsu" became id="rushformecsu" id="publisherccsu" became id="publisherecsu" javascript:showhide('rushformccsu','none') became javascript:showhide('rushformecsu', 'none') next, three copies of the rushccsu() function were made in which references to ccsu were replaced with references to the particular campus for which the function was intended. the rushwcsu() function required a bit more reworking as western connecticut state university indicated that they did not want items destined for certain collections available for rush cataloging. the resulting function was as follows: function rushwcsu() { try { for (var i = 0; i < document.getelementbyid('biborder').rows.length; i++) { var rush = document.getelementbyid('biborder').rows&#91;i&#93;.cells&#91;0&#93;; var wcsu = "wcsu"; var order = "processed"; if (rush.innerhtml.search(wcsu) == -1 || rush.innerhtml.search(order) == -1) { } else if (rush.innerhtml.search("wcsu h archives theses") != -1 || rush.innerhtml.search("wcsu h music scores") != -1) { } else { rush.innerhtml += ' <a href="javascript:showhide(\'rushformwcsu\', \'inline\');">click here to expedite processing</a>'; } } } catch(err) { } } there were now four different functions for appending links to in-processing notes, all of which needed to be called when the record was loaded. therefore, calls to the functions were added to the start() function. function start() { rushccsu(); rushecsu(); rushscsu(); rushwcsu(); fillin(); } window.onload=start; the fillin() function now needed to fill fields in all four forms and was amended to do so with the addition of the following lines of code to the end of the function: document.getelementbyid('authorecsu').value = author; document.getelementbyid('booktitleecsu').value = title; document.getelementbyid('publisherecsu').value = publisher; document.getelementbyid('bibidecsu').value = bibid; document.getelementbyid('isbnecsu').value = isbn; document.getelementbyid('authorscsu').value = author; document.getelementbyid('booktitlescsu').value = title; document.getelementbyid('publisherscsu').value = publisher; document.getelementbyid('bibidscsu').value = bibid; document.getelementbyid('isbnscsu').value = isbn; document.getelementbyid('authorwcsu').value = author; document.getelementbyid('booktitlewcsu').value = title; document.getelementbyid('publisherwcsu').value = publisher; document.getelementbyid('bibidwcsu').value = bibid; document.getelementbyid('isbnwcsu').value = isbn; the final change that needed to be made was to modify the landing page to include the turnaround and circulation desk retention policy information for all four campuses. while each campus could have had its own landing page it was much easier to just modify the single page to serve all four libraries. stage five: testing the first step in testing the script was to do a cross-browser compatibility test, which was done on a windows 7 machine in chrome 20, ff12, ff13, ie7, ie8, and ie9; and on a macos 10.6 machine in chrome 20, ff4, and safari 5. the script was also tested on the native browsers of various mobile platforms, including android, ios, and windows phone. there was a minor display problem with the bottom border of the “close” and “about” buttons in internet explorer 7, but otherwise the form worked and displayed in a consistent manner on all the browsers on which it had been tested with no noticeable impact on the performance of the opac. the heads of the catalog departments of the four campuses were then asked to test the enhancement and provide feedback. while most of the feedback consisted of requests for minor textual changes, there was one piece of feedback that proved to be invaluable insofar as the actual functioning of the form was concerned. as it turned out, the captcha question of “winter is cold; summer is ___.” could be answered with the antonym “hot” (as we had imagined) or a negation of “not” (an option we had not considered). the formfieldnameeditlist field, which contained parameters for the form indicating the required answers for particular fields was edited in each of the four forms so as to allow for both answers. old:<input name="formfieldnameeditlist" type="hidden" value="test78, captcha, hot, ~"/> new:<input name="formfieldnameeditlist" type="hidden" value="test78, equal, hot, not, ~"/> considerations for further enhancements to the project after extensive testing, the enhancement went live on the opac’s staging server. while being pleased with the end result of the project, those of us with a cataloger’s perfectionist mindset were still not satisfied. there were still a few things that we would have liked to have added to the final product: implementation of the idea that was briefly considered in stage two of development for providing movability of the form on the page as well as greying-out the record in the background so as to give more focus to the form itself. these changes were mainly issues of style and would be quite easy to implement at some future date. incorporation of the opac’s login system into the form for added security. this would make the enhancement that much more integrated into the fabric of the opac, but would require a programming ability that we perceived to be beyond our current intermediate level. the inclusion in the e-mail message sent to the cataloging staff of a link to a canned search of oclc’s worldcat for a potential record for the item being requested. this would be easy enough to accomplish as it would only require a concatenation of the item’s isbn with a query string for searching worldcat. however, because such a canned search could be highly inaccurate it would be important to make note in the e-mail that any record retrieved through clicking the link should be examined with a critical eye. the inclusion in the e-mail message sent to the requestor of the expected date that the item would be ready for her at the circulation desk as well as the date after which the item would be sent to the stacks if not claimed. this too would be easy enough to accomplish by manipulating a date object in accordance with each library’s turnaround and retention policies. the idea was floated around to alter the circulation policy for frontlog items such that we would allow the user to borrow the item while it was being cataloged. it was decided that this would not work in our situation, however, because our frontlog items do not possess item records but are instead called through the accession numbers in their bibliographic records. although this proposed policy change was rejected by the connecticut state university library system, it was included here for the benefit of other institutions that may wish to employ it as part of their implementation of the project. conclusions the patron-driven expedited cataloging enhancement to webpac pro is set to go live in late autumn of 2012, at which point it can be seen in records for frontlog items in consuls at http://www.consuls.org. by integrating a form for requesting in-process materials into the very structure of the catalog, it is our hope that the gap that the user may have encountered between finding frontlog items in the catalog and gaining access to them will be bridged. we believe that this enhancement will also allow our cataloging departments to better prioritize those materials in their queues that the user will need today above those materials that the user might need tomorrow. if executed hand-in-hand with a progressive approach to cataloging, we foresee the patron-driven expedited cataloging enhancement greatly increasing the efficiency and impact of our cataloging output. however, until the enhancement has been active for a significant amount of time we can only guess and hope as to the effect that it will have on our users’ ability to request in-processing materials and on how that will influence our departments’ cataloging. over the course of the project’s first year of operation we plan to monitor its usage and compare the resulting statistics with the usage statistics of our previous rush request form. if a significant increase in use is observed we hope to implement some of the enhancements that are proposed above so as to add a few extra bells and whistles to our project. implementing the enhancement in your library you can download the source code and instructions for implementing the patron-driven expedited cataloging enhancement in your webpac pro installation here. the instructions cover the implementation of the enhancement on union catalogs as well as on catalogs for standalone libraries. how you chose to implement the enhancement in your catalog is up to you. you may decide to include some of the improvements proposed above or you may even choose to scrap some of the features we’ve included in this first version. whatever you choose to do, we ask that when you apply the enhancement to your catalog, you keep the credits that appear in the “about the rush cataloging request form” popup intact. figure 3. about the rush cataloging request form pop up. while this project was developed on an innovative interfaces incorporated millennium system, the concept behind it can easily be implemented on pretty much any other integrated library system provided that the ils allows for the customization of the opac. the technique for implementing this enhancement on an opac other than webpac pro would likely vary a great deal and would require some reengineering so as to leverage the customization features unique to that system. it is our hope that this article will have provided you with inspiration and ideas for implementing a patron-driven expedited cataloging enhancement in your opac and we would be very interested in learning about your experiences in doing so. sources cited nof j. 2011. frontlog cataloging: using in-process records to reveal backlogged collections. judaica librarianship 16/17:93-112. huggins’ email form script [internet]. [updated 2010 aug 1]. houston (tx): james s. huggins’ refrigerator door; [cited 2012 jul 19]. available from: http://www.jamesshuggins.com/h/hefs/huggins-email-form-script.htm sending sms from an iii catalog [internet]. bryn mawr (pa): bryn mawr college; [cited 2012 jul 19]. available from: http://trilogy.brynmawr.edu/trico/sys/sms.html author biography steven bernstein is an assistant catalog librarian at central connecticut state university. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – automatic preparation of etd material from the internet archive for the dspace repository platform mission editorial committee process and structure code4lib issue 8, 2009-11-23 automatic preparation of etd material from the internet archive for the dspace repository platform a big challenge associated with getting an institutional repository off the ground is getting content into it. this article will look at how to use digitization services at the internet archive alongside software utilities that the author developed to automate the harvesting of scanned dissertations and associated dublin core xml files to create an etd portal using the dspace platform. the end result is a metadata-rich, full-text collection of theses that can be constructed for little out of pocket cost. by tim ribaric introduction the library where i work recently embarked upon creating an institutional repository. the ultimate goal of the repository is to be the authoritative, archival source of information produced by my institution, brock university. during the selection process to choose a software platform, many different products, both open source and proprietary, were evaluated. after a thorough search the selection came down to the dspace platform [1]. from what we saw, dspace offered the best set of features and being an open source product allowed us to add extra functionality to it. the dspace platform has a large body of users which has created an extensive pool of expertise to draw from. the only drawback is that the official documentation is a bit lacking and doesn’t shed light on everything that dspace is capable of. the project i embarked upon was not to modify the platform itself but to find a way to populate it with institutional data. i was searching for an automated way to ingest a collection of dissertations created by the university. the start of the etd portal like most institutions with graduate programs, my library possessed a collection of printed and bound theses that dated back to the inception of the school. the plan was to create an etd (electronic theses and dissertations) portal within dspace. the total list of graduate dissertations numbered somewhere around 1400. this proposed etd portal created two distinct challenges: first how to digitize this vast quantity of material, and second how to ingest this information into the fledgling dspace instance. the answer to the first question turned out to be twofold. some theses would be shipped out to the internet archive for digitization while some were to be digitized in house. the internet archive [2] site summarizes its aims quite succinctly: the internet archive is a 501(c)(3) non-profit that was founded to build an internet library, with the purpose of offering permanent access for researchers, historians, and scholars to historical collections that exist in digital format. [3] while best known for hosting a variety of digital content, the internet archive foundation also provides a digitization service. paper materials can be sent to the offices of the archive (at the university of toronto in our case) to be digitized. this service delivers a pdf document with full ocr mark up and information about the material itself in the form of metadata. the material is then hosted on the internet archive site, presumably indefinitely, to be accessed by anyone. the digitization service offered by the internet archive solved half of the problem. the end result was well-crafted digital representations of our theses. unfortunately they were made available in a way that wasn’t the most useful for us. that is to say they existed in an online form that was not locally hosted or integrated into our already existing repository. the next step in the process was to locally load these theses into the dspace site. this boiled down to two complementary processes, each of which were automated with a web application. ia_scraper the first utility i created, ia_scraper [4], was an intelligent rss feed monitor that would (with help from the php curl class) monitor our collection of newly digitized theses and download the required materials. every item posted on the archive is displayed with a fairly predictable layout. the url follows a succinct predetermined pattern and for each digitized item there are a base set of files that are created. an example record of one of our theses on the archive, with associated xml and marc record files, can found at http://ia331419.us.archive.org/1/items/descartesmeditat00yurkuoft/: figure 1. directory of a scanned thesis at archive.org the extension of the file names show exactly what is bundled for each item. in my particular case i was looking for the pdf file and the xml dublin core file. furthermore, each distinct set of material found on the archive is organized in a unique ‘collection’. in my case all the items from brock university were sorted under http://www.archive.org/details/brockuniversity. by browsing the collection unique to my institution and by selectively searching for the word ‘thesis’ in the description of the item it was possible to put together a list of our digitized theses. in addition, the internet archive offers an rss feed that reports the items as they get added to particular collections. putting these two features together i was able to monitor our institute-specific rss feed and fetch the theses as they were added. the end result is a collection of xml and pdf files of the digital versions of our theses. ds_ingestor the ds_ingestor [5] completes the final steps of converting the internet archive data into something that is dspace ready. dspace creates a standardized directory and file structure for each imported item. more information about this formatting can be found in the dspace manual [6]. simply pointing the ds_ingestor at the data gathered from the ia_scraper will process titles into ready to ingest numbered sets (as defined in a config file). the most challenging part of the ds_ingestor process was modifying the xml produced by the internet archive into the format that dspace expects. example of the differing xml syntax the internet archive provides metadata for its items in a variety of different formats. i used the ocr full-colour pdf as our repository object and the dublin core xml file associated with the item. dublin core was chosen because it has been adopted widely and it is natively supported by dspace. below is the dublin core metadata that i retrieved from the internet archive for one of our digitized theses located at: http://www.archive.org/details/descartesmeditat00yurkuoft. (updated, see correction note). <?xml version="1.0" ?> <oai_dc:dc xmlns:oai_dc="http://www.openarchives.org/oai/2.0/oai_dc/" xmlns:dc="http://purl.org/dc/elements/1.1/"> <dc:title>descartes' meditations : can the idea of god be derived from a meditation on the will and substance? /</dc:title> <dc:creator>yurkewich-liddell, kellyann.</dc:creator> <dc:type>text</dc:type> <dc:publisher>st. catharines, ont. : brock university, department of philosophy,</dc:publisher> <dc:date>2004.</dc:date> <dc:language>eng</dc:language> <dc:description>thesis (m.a.)--brock university, 2004.</dc:description> <dc:description>includes bibliographical references (l. 94-98)</dc:description> <dc:subject>descartes, ren&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;eacute;, 1596-1650.</dc:subject> <dc:subject>god</dc:subject> </oai_dc:dc> the xml native to the internet archive is written against the openarchives.org namespace with metadata elements enclosed within dublin core colon-delimited tags. dspace, however, requires things to be formatted a bit differently. here is the same information from our example record after it has been converted in the dspace format by ds_ingestor. continuing with our previous example now found at: http://dr.library.brocku.ca/handle/10464/2298 <dublin_core> <dcvalue element="title" qualifier="none">descartes' meditations : can the idea of god be derived from a meditation on the will and substance? /</dcvalue> <dcvalue element="contributor" qualifier="author">yurkewich-liddell, kellyann.</dcvalue> <dcvalue element="type" qualifier="none">text</dcvalue> <dcvalue element="publisher" qualifier="none">st. catharines, ont. : brock university, department of philosophy,</dcvalue> <dcvalue element="date" qualifier="none">2004.</dcvalue> <dcvalue element="language" qualifier="iso">eng</dcvalue> <dcvalue element="description" qualifier="none">thesis (m.a.)--brock university, 2004.</dcvalue> <dcvalue element="description" qualifier="none">includes bibliographical references (l. 94-98)</dcvalue> <dcvalue element="subject" qualifier="none">descartes, ren&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;eacute;, 1596-1650.</dcvalue> <dcvalue element="subject" qualifier="none">god</dcvalue> </dublin_core> the dspace xml itself is a nested set of ‘dcvalue’ tags with a series of attributes, one for element, and another for qualifier. the basic metadata like ‘author’ and ‘title’ is listed, but there is also a ‘subject’ tag, which is the lc subject headings that have been assigned to the thesis when it was being digitized. by incorporating this information into the dspace record of each thesis another facet of categorization and search is added to the etd portal. the completed workflow figure 2. sample thesis at archive.org the complete workflow for our example would look something like this: the thesis is digitized and made available on the archive at url http://www.archive.org/details/descartesmeditat00yurkuoft ia_scraper downloads the required information by examining an rss feed and saves it locally as follows: ia_scraper_data_dir/descartesmeditat00yurkuoft/ descartesmeditat00yurkuoft.pdf descartesmeditat00yurkuoft_dc.xml ds_ingestor runs and adds the files to a bulk directory with other titles: ds_ready_dir/batch_2009-09-14-01-19/ ... descartesmeditat00yurkuoft/ contents dublin_core.xml descartesmeditat00yurkuoft.pdf ... ds_ingestor reports back that data is ready along with the final command line instruction that needs to be entered that commits the data to the repository. (this will vary depending on dspace configuration.) for this example it might end up being (on a unix style system): /dspace/bin/import --add --eperson=person@repository.edu --collection=1111/1000 --source=/var/www/dsingestor/ds_suitable/batch_2009-09-14-01-19/ --mapfile=/home/dspace/ingest_files/batch_2009-09-14-01-19.map.ingest briefly looking at the components: /dspace/bin/import – the location of the dspace binaries directory where the import command is located. --add – the switch to tell the importer you are adding items --eperson=person@repository.edu – the email address of the person that has permission to deposit items into the repository --collection=1111/1000 – is a combination of your handle id [7] and the id of the collection you are adding your content to --source=/var/www/dsingestor/ds_suitable/batch_2009-09-14-01-19/ – where the information downloaded by ia_scraper and processed by ds_ingestor is held. -mapfile=/home/dspace/ingest_files/batch_2009-09-14-01-19.map.ingest – is where the generated mapfile of the input process will be kept. this is handy to keep because it will tell you what the id number is of every item that was imported. this is also discussed in the dspace manual [6]. figure 3. final dspace page at brock university the end result and the limits of dspace the final repository populated with the internet archive content ends up being a very useful and attractive product. since the pdf’s have been ocr’ed it is possible to have a full text searchable database of theses. this can easily be accomplished by adding the full text of repository items to the search indexes that dspace maintains. the specific details can be found in the dspace manual [8]. by also doing some refinements in how things are categorized it is possible to have the theses organized by rank and discipline so that easy browsing is facilitated. as previously mentioned, the inclusion of lc subject headings allows for even more entry points into the content. our finalized dissertations and theses portal can be found at: http://dr.library.brocku.ca/handle/10464/4 dspace in general takes some time to get used to. the fact that it runs in a tomcat environment using a postgresql database is a stark contrast to the traditional lamp stack that most popular open source applications can be classified under. this difference of infrastructure created a real barrier to learning how to use dspace effectively. as mentioned previously, official documentation always seemed out-of-date and not at all thorough. however, dspace is a platform well worth implementing and supporting. in fact, all open source oai products should be encouraged. these platforms become the basis of huge collaborative initiatives like oaister [9] that will eventually be crawling all of the world’s digital repositories and unifying all the content under one search mechanism. a brief word on the etd-ms as previously mentioned, dspace out of the box supports the dublin core metadata format. dublin core is a great schema that is widely applicable to many different publication types but it does not describe some essential pieces of information that pertain to dissertations. that is why the etd-ms [10] was created. proponents of other repository platforms point out that dspace should support this schema. in fact , in order to have theses canada regularly harvest your digitized theses, support for the etd-ms is a requirement [11]. this shouldn’t be seen as a drawback to implementing dspace. previous versions of dspace had user contributed software that allowed for the mapping of the dspace dublin core schema to etd-ms. with luck this mapping will once again be possible in current and future versions of the dspace platform. final words the ia_scraper and ds_ingestor are generalized enough to use for any internet archive content, and are not restricted simply to dissertations. quite recently i have used the packages to download and locally archive a collection of out of copyright material that is of local interest. by reusing the metadata and digitized items created by the internet archive i was able to locally archive a collection of about 300 books in a matter of minutes. the combination of digitization services offered by the internet archive, the ia_scraper and ds_ingestor software, and dspace platform allows anyone to create a fully functional open source digital repository. the primary benefit is an end product that requires no licensing fees and only costs staff time and a negligible scanning fee to get online. references [1] dspace homepage. http://dspace.org [2] internet archive homepage. http://archive.org [3] internet archive: about ia. http://www.archive.org/about/about.php [4] ia_scraper. http://elibtronic.ca/software/ia_scraper [5] ds_ingestor. http://elibtronic.ca/software/ds_ingestor [6] dspace item importer and exporter. http://www.dspace.org/1_5_2documentation/ch09.html#n13795 [7] the handle system. http://handle.net [8] configuring lucene search indexes. http://www.dspace.org/1_5_2documentation/ch05.html#n11980 [9] oaister. http://oaister.org [10] etd-ms: an interoperability metadata standard for electronic theses and dissertations. http://www.ndltd.org/standards/metadata/etd-ms-v1.00-rev2.html [11] theses canada requirements for harvesting electronic theses and metadata. http://collectionscanada.ca/thesescanada/s4-262-e.html about the author with a bsc. in computer science and a mlis, tim ribaric has been the digital services librarian at brock university, located in the niagara region of ontario canada, since 2006. his areas of interest include openurl resolvers, the dspace platform, and finding a way of using twitter effectively in the library environment. he blogs at http://elibtronic.ca and can be reached at tribaric (at) brocku.ca. correction november 25th, 2009: the two code blocks in the example of the differing xml syntax were inadvertantly switched. this error is now corrected and the surrounding paragraphs slightly rewritten to reflect this change. the original text of this section read: below is the dublin core metadata that i retrieved from the internet archive for one of our digitized theses located at: http://www.archive.org/details/descartesmeditat00yurkuoft. <dublin_core> <dcvalue element="title" qualifier="none">descartes' meditations : can the idea of god be derived from a meditation on the will and substance? /</dcvalue> <dcvalue element="contributor" qualifier="author">yurkewich-liddell, kellyann.</dcvalue> <dcvalue element="type" qualifier="none">text</dcvalue> <dcvalue element="publisher" qualifier="none">st. catharines, ont. : brock university, department of philosophy,</dcvalue> <dcvalue element="date" qualifier="none">2004.</dcvalue> <dcvalue element="language" qualifier="iso">eng</dcvalue> <dcvalue element="description" qualifier="none">thesis (m.a.)--brock university, 2004.</dcvalue> <dcvalue element="description" qualifier="none">includes bibliographical references (l. 94-98)</dcvalue> <dcvalue element="subject" qualifier="none">descartes, ren&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;eacute;, 1596-1650.</dcvalue> <dcvalue element="subject" qualifier="none">god</dcvalue> </dublin_core> the xml itself is a nested set of ‘dcvalue’ tags with a series of attributes, one for element, and another for qualifier. the basic metadata like ‘author’ and ‘title’ is listed, but there is also a ‘subject’ tag, which is the lc subject headings that have been assigned to the thesis when it was being digitized. by incorporating this information into the dspace record of each thesis another facet of categorization and search is added to the etd portal. dspace, however, requires things to be formatted a bit differently. here is the same information from our example record after it has been converted in the dspace format by ds_ingestor. continuing with our previous example now found at: http://dr.library.brocku.ca/handle/10464/2298 <?xml version="1.0" ?> <oai_dc:dc xmlns:oai_dc="http://www.openarchives.org/oai/2.0/oai_dc/" xmlns:dc="http://purl.org/dc/elements/1.1/"> <dc:title>descartes' meditations : can the idea of god be derived from a meditation on the will and substance? /</dc:title> <dc:creator>yurkewich-liddell, kellyann.</dc:creator> <dc:type>text</dc:type> <dc:publisher>st. catharines, ont. : brock university, department of philosophy,</dc:publisher> <dc:date>2004.</dc:date> <dc:language>eng</dc:language> <dc:description>thesis (m.a.)--brock university, 2004.</dc:description> <dc:description>includes bibliographical references (l. 94-98)</dc:description> <dc:subject>descartes, ren&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;eacute;, 1596-1650.</dc:subject> <dc:subject>god</dc:subject> </oai_dc:dc> with the inclusion of a name space and some strict xml encoding, dspace xml is a bit more rigorous than the internet archive’s format. nevertheless, every digitized thesis that passed through ds_ingestor was able to be converted, indicating that the xml from the archive was well-formed even without the addition of a namespace. subscribe to comments: for this article | for all articles 3 responses to "automatic preparation of etd material from the internet archive for the dspace repository platform" please leave a response below: karen coyle, 2009-11-24 tim, i hadn’t ever seen the “dcvalue” form of dc, so i was curious… when i go to the ia dc.xml file, i get pretty standard looking dc: descartes’ meditations : can the idea of god be derived from a meditation on the will and substance? / yurkewich-liddell, kellyann. text st. catharines, ont. : brock university, department of philosophy, 2004. eng thesis (m.a.)–brock university, 2004. includes bibliographical references (l. 94-98) descartes, rené, 1596-1650. god source: http://ia331419.us.archive.org/1/items/descartesmeditat00yurkuoft/descartesmeditat00yurkuoft_dc.xml has something changed? did you use a different file? karen coyle, 2009-11-24 oops, need to escape all of the &lt: <?xml version=”1.0″?> <oai_dc:dc xmlns:oai_dc=”http://www.openarchives.org/oai/2.0/oai_dc/” xmlns:dc=”http://purl.org/dc/elements/1.1/”> <dc:title>descartes’ meditations : can the idea of god be derived from a meditation on the will and substance? /</dc:title> <dc:creator> yurkewich-liddell, kellyann. </dc:creator> <dc:type>text</dc:type> <dc:publisher>st. catharines, ont. : brock university, department of philosophy,</dc:publisher> <dc:date>2004.</dc:date> <dc:language>eng</dc:language> <dc:description>thesis (m.a.)–brock university, 2004.</dc:description> <dc:description>includes bibliographical references (l. 94-98)</dc:description> <dc:subject>descartes, rené, 1596-1650.</dc:subject> <dc:subject>god</dc:subject> </oai_dc:dc> tim ribaric, 2009-11-25 hello karen, thanks for the astute observation. as it turns out there was a typo with the xml code blocks, (they were inverted) and that created the confusion. this has now been corrected and the article reads as it was intended to. leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – need help with your code? piloting a programming and software development consultation service mission editorial committee process and structure code4lib issue 34, 2016-10-25 need help with your code? piloting a programming and software development consultation service in the spring 2016 semester, george washington university libraries (gw libraries) undertook a pilot to provide programming and software development consultation services for the university community. the consultation services took the form of half hour appointments conducted by librarians with software development expertise, similar to other reference services offered by gw libraries. the purpose of this paper is to provide an overview and assessment of the pilot project. by laura wrubel, daniel kerchner, justin littman inception gw libraries piloted a programming and software development consultation service for the university community in spring 2016. the consulting service originated with the scholarly technology group (stg) at gw libraries, a unit composed of librarians and staff with technical expertise and responsibilities for supporting academic work at the university. the idea had been developing within stg for a number of years, prompted by faculty and students who would find their way to stg with technical questions. it gathered further motivation by members of stg listening to the needs of researchers at events like thatcamp or the annual digital humanities showcase hosted by gw libraries. we sensed there was a gap in campus support for researchers and students, particularly in the social sciences and humanities, who wanted to get started with projects that require digital tools, and that we in the library were well-situated and prepared to fill that gap. in september 2015 we began preparing a project proposal — a short document describing the goals, scope, benefits, risks, and implementation details. the project proposal incorporated feedback from gw reference librarians, select faculty, and gw libraries’ senior leadership. in addition, members of nyu’s digital scholarship services were consulted for guidance. we also attempted to determine any overlap with services offered by other parts of the university, and did not identify any. we were given approval to start offering the service in the spring 2016 semester. based on the success of the pilot and with support from senior leadership, we are permanently adding software development consultations to the growing list of services offered by gw libraries. goals, scope and risks the primary goal of the consulting service is to support academic or scholarly inquiry which requires coding. this goal is intended to be reasonably broad in scope and audience, which includes faculty, students, staff, and researchers. some of the areas that are in scope include: guidance on coding/software development/scripting/programming, including: recommending a learning path and coding tutorials advice on starting a new coding project code review/debugging tools selection assistance with: working with data markup/encoding (e.g., xml, json, csv, rdf) retrieving data from websites/apis data cleansing/manipulation databases (e.g., table design, querying, optimizing, loading) fulltext searching online exhibits data visualization the consulting service is not without limits. in particular, there were a number of activities that we specifically placed out of scope: computer science (or similar) assignments writing entire programs or doing all work for researchers tutoring filling function of ta for a class in addition to supporting stg’s mission of academic work, our secondary goal was to increase interaction between stg staff and the gw community in support of gw libraries’ strategic objective of being “the nexus for cross-disciplinary collaboration on campus.” in that vein, we thought that this interaction might provide insight into topics for future workshops to be led by team members. during the project proposal phase, we identified a number of risks. foremost among these was that the consulting service would excessively consume library staff time, especially if a consultation continued beyond the initial meeting. further, there was a concern that library staff would be in the position of having to regularly deny researcher requests either because they were out of scope or exceeded the limited resources. implementation for the pilot, a total of two hours of consultations were made available per week in half hour meetings, using the libraries’ existing research calendar. gw faculty and students could select an appointment from this calendar, which is also used to schedule reference, gis, statistics, and similar appointments. the appointment form requests the course name, topic/research question, and resources already consulted. members of the gw community were also welcome to email us if the scheduled time slots were not convenient or if a remote (webex) consultation was preferable. the consultation service is staffed primarily by the three authors, who work as software development librarians. though we have fairly wide-ranging technical expertise, we identified gaps in our skills and connected with others in the libraries, including our web developer and a digital operations manager with r skills. when a staff member received an appointment request, she made sure the topic for the consultation was within scope. depending on the topic, she might contact the requester by email to clarify the need, conduct some research in preparation, or find a different staff member with relevant expertise. in several cases, the original staff member would sit in as a learning opportunity. we held the consultation meetings in the same offices on the gelman library’s main floor used for other research appointments. these rooms were equipped with external monitors, which proved useful for collaboration. for tracking purposes, a ticket was created in our helpdesk system for each appointment. the staff member would add notes on the appointment and update based on any follow-up. an estimate was kept on how long each consultation took. the ticket was used solely for internal recordkeeping and not for interacting with the student or faculty member. the helpdesk system was not the ideal fit for this task, but good enough for our limited use. a key part of the implementation was publicity of the consulting service. we were fortunate to have the assistance of robin delaloye, gw libraries’ director of communications and research, who has deep experience in marketing the libraries’ services to the gw community. some of the mechanisms used to publicize the service include: digital signage within the libraries announcements via twitter announcements in the libraries’ poster newsletter (“gelman exposed”) displays, flyers, and/or talks at relevant campus events web page providing additional details on the service briefing to reference and user services staff in addition, we received some secondary publicity including an article in the student newspaper and a blurb in the office of the vice president for research’s email newsletter. while we don’t have firm evidence, our intuition is that the digital signage provided the most exposure, as several students referenced the digital signage as the way they learned of the service. from a marketing standpoint, our biggest challenge was what to call the service. the merits of the terms “coding,” “programming,” and “software development” were considered before we settled on “programming and software development” consultation service. providing quality service while we had experience helping the gw community with software and technical projects on an ad hoc basis in the past, we had some anxiety about this new service. would we have enough knowledge about the topics to be helpful? would we be able to meet the expectations of students and faculty consulting with us? as we began meeting with students and faculty, we found these fears largely unfounded. the appointment request process helped address our concern about being prepared to help individuals. it provided a starting point for a conversation to understand the possible need, allowed us to identify tools and resources in advance, and to brush up on a particular tool if needed. this pre-meeting conversation also helped ensure the consultee could come prepared to get the most out of the meeting. for example, we might recommend they bring a laptop and install anaconda in order to have an easy-to-use environment for working with python and jupyter notebooks. similar to our colleagues’ reference interviews, consultations involved a conversation to explore the consultee’s need and seek an understanding of the best approach. sometimes the consultee requested help with a particular tool in her appointment request, but in conversation, it became clear a different approach would be better suited to reach her goal. for example, one student requested help scraping several web pages, but we found that the website provided an api. we were able to work with the student to get api keys and use python to request data and parse the response into a csv file. one of our concerns in embarking on this pilot was whether we had all of the knowledge and background to easily answer questions about a wide range of programming topics. we had strategies for coming to the consultation prepared to be helpful — from background research on the topic to polling colleagues and bringing them into the consultation. but what about during the course of the meeting? we found that our gaps in knowledge or memory turned out to be useful opportunities to model how to answer programming questions. we could show consultees that most code doesn’t work as planned at first, and debugging and checking reference materials is a typical practice for all software developers. outcome and lessons learned we provided 13 software development consults over the course of the semester. it took a good month before our first appointment, perhaps due to the ramp-up on the marketing and/or the tempo of the semester. activity was steady through the rest of the semester, with appointments tapering off at the end and the summer continuing with few appointments. there were typically 1-2 appointments per week. each appointment took, on average, 1.8 hours of staff time, with the greatest being 4 hours. overall, we were pleased with both the number of appointments and the reasonable draw on staff resources. the most popular technologies that assistance was requested with were html/javascript and python. fortunately, this played to our technical strengths. while consultees were affiliated with a number of departments across the university, the most popular were political science, international affairs, and computer science. most consultees were undergraduate or graduate students, but also included some research assistants and faculty. around two-thirds of the consultees had little or no programming experience. some of the more interesting topics included: scraping legislation from another country’s parliamentary website and security council resolutions from the united nations website troubleshooting a web application used in a biology lab for quantifying the behavior of roundworms troubleshooting a fluid mechanics visualization building an online health survey with custom logic unavailable in current online poll sites interview preparation for software development positions by far the most significant issue we faced (about a half dozen instances over the semester) was computer science students requesting assistance with class assignments. in each case, we referred the student to her ta or suggested other resources (e.g., lynda.com, to which the university subscribes). there were also some instances of requests for consultations on topics that should have gone to other library staff (e.g., for statistics consultations). we suspect that both of these issues could be handled through improvements to the research calendar and the information it provides users. the software development consult pilot also produced some unexpected benefits. the publicity around the coding consults raised the general visibility of the stg group and resulted in some members of the gw community consulting us about matters that were in the purview of stg, but not the coding consult service. we built awareness in the gw community, including the university administration, that the libraries had software development expertise and that libraries engage in software development as an activity supporting the university. as hoped, our increased interaction with the gw community provided us additional insight into needs we could potentially meet with workshops. it also provided each of us with learning opportunities, as we often had to sharpen our skills to provide assistance in technologies that we don’t use on a daily basis; additionally, we learned from each other as we would sometimes sit in on each other’s consultations. future plans given our experience with the pilot and suggestions we’ve received, we see some opportunities for improvement. in particular, we would like to: perform outreach to department chairs as a means of reaching faculty and students and look for additional avenues for publicity. consider a simple post-consultation survey and checking on outcomes with participants. collect more consistent data on the appointments. in particular, it would be helpful to explicitly collect consultees’ departments, roles (undergraduate, graduate student, etc.), and programming experience. feedback on our initiative or sharing the experience with similar services at other institutions is welcome. overall, the authors consider the programming and software development coding consultations pilot a solid success. it achieved the goals of supporting academic or scholarly inquiry at gw which requires coding, while getting us away from our desks to interact with the gw community. based on the pilot project, we plan to move forward with the software development consultations as an ongoing service of gw libraries. acknowledgments the authors would like to acknowledge the guidance and support of geneva henry, hannah sommers, robin delaloye, matthew mihalik, dan chudnov, the reference and user services librarians, and our stg colleagues. about the authors laura wrubel (lwrubel@gwu.edu) is a software development librarian at george washington university libraries. justin littman (justinlittman@gwu.edu) is a software developer and librarian at the george washington university libraries. daniel kerchner (kerchner@gwu.edu) is a senior software developer and librarian at the george washington university libraries. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – making patron data work harder: user search terms as access points? mission editorial committee process and structure code4lib issue 3, 2008-06-23 making patron data work harder: user search terms as access points? montana state university (msu) libraries are experimenting with re-using patron-generated data to create browseable access points for the electronic theses and dissertations (etd) collection. a beta querycatcher module logs recent search terms and the number of associated hits. these terms are used to create browseable lists and tagclouds which enhance access to the etd collection. gathering and reusing information about user behavior is an emerging trend in web application development. this article outlines msu libraries’ reasoning for moving towards a user-generated model and provides a complete walkthrough of the steps in building the application and example code. by jason a. clark introduction whether it is a respect for user privacy or a reluctance to give up control, libraries have traditionally shied away from harvesting or “reusing” data collected from our users. with the rise of social web 2.0 apps, some interesting applications of user data have begun to appear. in popular web sites such as flickr and del.icio.us, we have large-scale examples of user-generated content (e.g., the tagging systems or the social sharing functionalities that exist between the users of these systems) working to improve user access to content. within the library environment, we often see applications and organizational systems that focus more on the top-down approach of assigning access points a priori. at montana state university (msu) libraries, we started to ask ourselves: “does it have to work like this all the time? what kinds of access might we be restricting in forcing users to navigate and learn our strict knowledge hierarchies?” to this end, we’ve started to try to rethink the ways in which our collected user data can open up access to our collections. the initiative: querycatcher in his account of the rise of google, john battelle introduces the concept of a “database of intentions” to analyze the role that search engines play as the collectors of humanity’s curiosity, exploration, and expressed desires. [1] battelle is speaking about the search terms that we use and how these terms are recorded, logged, and then mined for insights by google. it was exactly this idea of collecting knowledge that made me think of a new possibility for our own library search engines. [2] in keeping with battelle’s concept, we introduced a new component to our msu electronic theses and dissertations (etd) web application (http://etd.lib.montana.edu/etd/view/). we created a search term recorder module, beta name “querycatcher,” that logs recent searches and uses those recent searches to create user access points in both a standard search list and “searchcloud” format. the search list view (seen in figure 1 below) shows the most recent 50 searches recorded into the system. it includes the number of hits for each search and the number of times that the term has been browsed or searched. you can see it in action at http://etd.lib.montana.edu/etd/view/searches.php. figure 1: search list view [view full-size image] the “searchcloud” view (seen in figure 2 below) shows the most recent 50 searches recorded into the system. it includes a visual weighting of the terms according to the number of times that the term has been browsed or searched. you can see it in action at http://etd.lib.montana.edu/etd/view/searchcloud.php. figure 2: “searchcloud” view [view full-size image] our idea was to provide more search transparency for our users and for ourselves. frequently, etd searching was restricted to exploratory, single term searches. some of this user behavior was a function of the traditional searching and browsing environment we had set up for the etd application. (the traditional search interface is a series of web forms and browseable links based on predetermined controlled vocabularies.) we wanted to enable users to see what search terms were getting results so they could refine them for their own purposes as well as make visible the most popular terms, which are invisible to our standard web analytics for the application. recording the actions of our users and turning that data into access points was an experiment in enabling our users to create their own browseable access to the collection. unobtrusive harvesting one of the experiments of the project has been the concept of releasing the user from any efforts to create the access points. the etd application already has a tagging module that has met with mixed success. we are finding that ownership and real participation in tagging items can be hard to foster in a local system. with this in mind, we decided to go in a different direction with querycatcher. it works in the background to collect users’ search queries and places the onus on our system to make things happen. the goal was to allow users to create value without really knowing it. while this technique might raise concerns about user privacy, it is possible to take advantage of patron data in an anonymous fashion (these concerns will be addressed as part of “complications and limitations“). the “invisibility” of this process is a large reason it has proven much more successful than our user tagging module. implementation: making it work at msu libraries, our library systems are built within the lamp stack, so i’m going to walk you through the steps needed to create the querycatcher application using php, mysql, css, and xhtml for markup. [3] i will describe the pieces of the application in terms of a data model layer, an application logic layer, and a display view layer. it’s my hope that in using these generic terms other developers will be able to understand the functions of each piece of the application and apply it to their programming environment of choice. 1. data model layer our first step is to create the database that will log the queries. think of this stage as setting up the data model layer. here’s the create table syntax for the table which will record the queries into the system. --table structure for table `log` -drop table if exists `log`; create table `log` ( `query_id` int(16) not null auto_increment, `query_data` varchar(255) default null, `query_results` int(6) default null, `query_time` timestamp not null default current_timestamp on update current_timestamp, `query_referrer` varchar(255) default null, primary key (`query_id`), fulltext key `full_index` (`query_data`) ) engine=myisam default charset=utf8; there’s nothing too unique here as the table contains fields to collect the query, the number of hits for the query, and a timestamp to help us organize the data for display. the syntax is sql as represented in mysql. [4] 2. application logic layer our second step is to create the logic and insert statement that will search and record query data. at this stage, we are talking about the application logic layer that provides the searching and recording action for the etd application. here’s a snippet of the processing code from the results.php file: case 1: if($bkeyword) { $result = mysql_query("select * from item where item_status='a' and match (dc_title, dc_creator, dc_description, dc_contributor) against ('+$keyword*' in boolean mode)"); $num_rows = mysql_num_rows($result); if($num_rows == 0) { nomatches(); } else { //log search term query, trim whitespace and store in "log" table of database $logquery = @mysql_query("insert into log set query_data='".trim($keyword)."', query_results='".$num_rows."', query_referrer='".$_server['http_referer']."'"); if (!$logquery) { die('<h2>error adding query to log: ' . mysql_error() . '</h2>'); } echo '<h3 class="results">your search for keyword <strong>'.stripslashes($keyword).'</strong> resulted in <strong>'.$num_rows.'</strong> match(es).</h3>'."\n"; } } in this example, we use a php switch statement to check for the existence of the keyword variable. if the keyword variable exists and retrieves results, we record the number of results in the $num_rows variable. next, we pass those values through some logic checks and trim functions (for cleanup) and insert into the “log” table to record the query data, referrer, and total hits. it should be noted that any variables passed to the application logic layer will be passed from a simple web form and we’ll build the form in the next section. 3. display view layer our third step is to create the display view layer, which includes hooking the etd search form into the “log” table and providing several displays for the recorded search data. first, the search form from the main page of the etd application must be hooked into the “log” table. we’re talking here about basic user interface (ui) design and one of the first components of the display view layer. we use post data from an xhtml form stored as variables and pass the variables along to our processing script. <form class="search" action="results.php" method="post"> <p> <label for="keyword"><strong>keyword:</strong></label> <input class="text" type="text" id="keyword" name="keyword" maxlength="50" size="50" /> <input type="submit" id="submit" class="submit" value="search" /> </p> </form> the important piece to note here is the <input name="keyword"> as it is the glue that binds the display view layer to the application logic layer. as a user enters a search term in the web form, the data is passed as the keyword variable to the results.php file which will allow for a search of the database and the recording of the query. the next piece of our display view layer involves showing the search query data. as mentioned earlier, we currently provide two views: a search list view that shows the number of hits and the number of times the term has been browsed or searched and a “searchcloud” view where terms are visually weighted according to the number of times they’ve been used. the xhtml markup for the pages is pretty straightforward – a <ul> unordered list tag set to display in columns for the search list view or a <p> paragraph tag with different type size settings for the “searchcloud” view. however, the php logic and the mysql queries are a bit more involved and worth noting. both the search list view and the “searchcloud” view use the same mysql query and php code: //get searches that start with the current variable, place hit number and count number $query = "select distinct(query_data), count(query_id) as query_count, query_results, query_time from log group by query_data order by query_time desc limit 0,50"; $result = mysql_query($query); here’s what’s happening with this code sequence: the $query variable in the code block is where we store the mysql statement that selects all unique queries (select distinct), places a count on the number of queries logged into the table (count(query_id) as query_count), grabs the number of results and the timestamp for the query, orders the results according to most recent timestamps (order by query_time desc), and limits the number of results to 50 (limit 0,50). finally, the $result variable coupled with the mysql_query() php function actually tells php to run the query and store the returned data for use later in the script. once we have made the call to the database and recorded all the different information, we need to loop through the info and print it out to the screen as xhtml. here’s the code from the search list view page that makes it happen: search list view (source: searches.php) while ($row = mysql_fetch_object($result)) { $search = rtrim(stripslashes($row->query_data)); $searchresults = $row->query_results; $searchcount = $row->query_count; if ($search != "") { echo '<li><h2><a href="./results.php?keyword='.urlencode($search).'">'.$search.'</a></h2>'."\n"; echo '<p>results: <strong>'.$searchresults.'</strong> hits <br/>term searched/browsed: <strong>'.$searchcount.'</strong> time(s)</p>'."\n"; echo '</li>'."\n"; } } in this particular code block, the first line uses a while loop coupled with the $result variable from the previous code snippet to provide the resulting database rows from the query. as the while loop is working, we take the different field values by using the php function mysql_fetch_object() and format them into different variables for display ($search, $searchresults, and $searchcount). before attempting to output, we check to make sure the $search variable is not empty using the simple if statement: if ($search != ""). finally, we print out the different variable values as xhtml using the php echo() function. the key here is the while loop which will execute until it evaluates to false; in this case, false occurs when there are no more query results to print out. [5] finally, the display needs to be finished with css. at this point, i assume you know what css is and does. if not, there are many excellent tutorials available online. [6] however, i wanted to point out a few special css rules that are used to create our final view of the query data. ul#block {float:left;width:100%;margin:0;padding:0;list-style:none;text-align:left;} ul#block li {position:relative;float:left;width:25em;margin:0;padding:15px 0 15px 15px;} ul#block li h2 {font:1.1em georgia,times,serif;color:#444;margin:0 0 15px 0;} p#tagcloud {padding:25px 0;} p#tagcloud a.size1 {font-size:.95em;} p#tagcloud a.size2 {font-size:1.1em;font-weight:bold;} p#tagcloud a.size3 {font-size:1.2em;font-weight:bold;} p#tagcloud a.size4 {font-size:1.3em;font-weight:bold;} p#tagcloud a.size5 {font-size:1.5em;font-weight:bold;} p#tagcloud a.size6 {font-size:1.6em;font-weight:bold;} p#tagcloud a.size7 {font-size:1.7em;font-weight:bold;} p#tagcloud a.size8 {font-size:1.8em;font-weight:bold;} p#tagcloud a.size9 {font-size:2.5em;font-weight:bold;} the first css snippet, ul#block, creates the columns effect for the search list view (source: ). the second css snippet, p#tagcloud and the a.size class, creates the wrapper element and font sizes for the “searchcloud” view (source: searchcloud.php). complications and limitations there are always complications in the process of implementation. msu libraries has not contacted parties to let them know of our “reuse” of the query data. we are careful not to log ip addresses or personal info in our recording processes, but some libraries may wish to be more transparent with their patrons. a simple note on the homepage of an application using this technique briefly explaining the process could go a long way to alleviating any patron concerns. another complication is the growing datasets that go along with logging each and every query into the system. since the introduction of querycatcher in october 2007, we have logged over 37,000 queries into the system. [7] as the mountain of data has grown, there became a need for a cronjob specifying a mysqldump of the log table along with a delete and new table create statement to start the process over again. we are currently running this cronjob to create a query log archive once a month. a related limitation is that the size of the recorded data informs the amount of data that can be displayed at any given time. we’re restricting the display views to the last 50 queries recorded to alleviate some of this pressure. updates and future work so, how is the experiment working out? there has been some measurable success in the use of the querycatcher pages to access the etd collection. both the searchcloud.php and searches.php have continually appeared in the top 15 urls for the etd site. both pages have also appeared as 2 of the top 8 entry pages to the etd collection for the last five months. this indicates that transparency of search terms can be a useful access point and the trends within our statistics suggest that continued use and growth is likely. continued growth brings new challenges and motivation to enhance these tools. one of our first steps will be to write a programming function to “de-dupe” the query log and only record unique searches. 37,000 logged queries shows some impressive use of the system, but the number of duplicate queries make for a lot of noise. we’d also like to create a more long term view of our most popular searches over time as a query archive for users to browse and use. other possibilities include using the collected data to suggest possible search terms in real time via an ajax autosuggest solution and building “did you mean?” functionality that looks for successful searches using similar terminology when users get few or no results. we feel we are just getting started and the possibilities for the query data remain to be fully tapped. conclusion libraries and their corresponding organizational systems and applications are complex because the information that we collect and represent is complex and varied. that’s a fact and it’s not going away. while we need to respect this complexity, it is also beneficial for us to recognize that giving up some control over “how” users access our collections can create added value that we couldn’t build on our own. experiments like querycatcher where users are allowed to contribute their own terms as browseable access points are small steps in this direction. let’s see what else our development community can imagine. code there are six files available. download zip (25.2 k) querycatcher.sql – the database create statements form.php – the xhtml form for querying the database results.php – the search processing and query recording instructions searches.php – the search list display view searchcloud.php – the searchcloud display view styles.css – the master styles notes [1] battelle, john. (2006). the search: how google and its rivals rewrote the rules of business and transformed our culture. portfolio trade. isbn: 1591841410. pp. 2-5. (coins) [2] the most perfect expression of battelle’s “database of intentions” is the google zeitgeist available at http://www.google.com/press/zeitgeist.html. it includes trends and analysis of queries logged into the google systems. [3] there are always other scripting languages and database engines. ruby on rails, asp, postgresql, even sqlite could be used as the backbone for this application. it’s really up to you. [4] see w3c schools for a nice intro to sql: http://www.w3schools.com/sql/. the mysql documentation is available at: http://dev.mysql.com/doc/ [5] i’m not including the code snippet from the “searchcloud” view as the similarities make it a bit redundant. the only real difference is the added $searchweight variable which is used to provide the different font sizes for the weighted “searchcloud”. here’s the code for those that are interested: “searchcloud” view (source: searchcloud.php) while ($row = mysql_fetch_object($result)) { $search_id = $row->query_id; $search = rtrim(stripslashes($row->query_data)); $searchcount = $row->query_count; $searchweight = $searchcount; if ($searchweight > 9) { $searchweight = '9'; } if($search != "") { echo '<a id="'.$search_id.'" class="size'.$searchweight.'" title="'.$search.' has been searched/browsed: '.$searchcount.' time(s)" href="./results.php?keyword='.urlencode($search).'">'.$search.'</a>'."\n"; } } [6] if you aren’t sure and want to learn more about css (cascading style sheets), visit the w3c tutorial at http://www.w3schools.com/css/ [7] data recorded on june 3, 2008. about the author jason a. clark builds library web applications and sets digital content strategies for montana state university libraries. he writes and presents on a broad range of topics ranging from open source applications to interface design and metadata. when he’s not thinking about apis and other geeky stuff, jason likes to hike the mountains of montana with his wife, jennifer, and their dog, oakley. if you would like to contact him, his web site is http://jasonclark.info/. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – batch metadata assignment to archival photograph collections using facial recognition software mission editorial committee process and structure code4lib issue 21, 2013-07-15 batch metadata assignment to archival photograph collections using facial recognition software useful metadata is essential to giving individual meaning and value within the context of a greater image collection as well as making them more discoverable. however, often little information is available about the photos themselves, so adding consistent metadata to large collections of digital and digitized photographs is a time consuming process requiring highly experienced staff. by using facial recognition software, staff can identify individuals more quickly and reliably. knowledge of individuals in photos helps staff determine when and where photos are taken and also improves understanding of the subject matter. this article demonstrates simple techniques for using facial recognition software and command line tools to assign, modify, and read metadata for large archival photograph collections. by kyle banerjee and maija anderson introduction archival photograph collections are often the centerpiece of digital initiatives. compelling images create powerful visual ties to institutional history, impressing administrators and funders while serving purposes ranging from public relations to intensive academic research. collected in analog formats or as born-digital objects, they come to the archives via institutional departments and individual donors by the hundreds and thousands. photographs appeal to diverse user groups, and researchers expect archival collections to be easily discoverable and keyword-searchable online. using traditional methods, providing this access requires specialized staff with knowledge of archival collections who are skilled in creating original metadata as well as historical research. when dealing with a collection of hundreds or thousands of photographs, balancing the expectations of researchers with staffing realities is challenging. without judicious project planning through all stages of digitization, projects to digitize archival photos become bottlenecked at the point of metadata creation. even managers experienced with other digitization projects often underestimate the time it takes to create specialized metadata for archival photographs — leading to a backlog of scanned images without descriptive metadata [1]. backlogs may also be generated through scanning for in-house use, exhibits, or researcher requests. backlogs of born-digital images grow as donors contribute collections of digital photographs, often with no description beyond a filename. to be made accessible, these scanned images require complete metadata as well as hosting on a public platform. without this support, images are stored on servers, external hard drives, networks, or staff computers where they are effectively hidden from staff as well as researchers. we suggest there are opportunities for archivists and systems staff to collaborate in automating metadata creation and harvesting metadata more efficiently. this article demonstrates how staff with modest technical skills can use simple automation techniques to leverage existing metadata in digital and digitized photographs and information obtained from facial recognition software to assign metadata more quickly, accurately and completely. things you need to know most systems that libraries use to store digital photographs do not take advantage of metadata within the digital image files. rather, this information is managed in an external database. however, even for institutions that use systems that depend on external metadata, there are several advantages to utilizing and creating metadata directly within the files themselves: automatic extraction of technical metadata such as creation date, dimensions, resolution, and bit depth automatic assignment appropriate access points such as subject and location to groups of photos images are more discoverable via search engines information associated with the image travel with it including use restrictions with the image however it is used enhanced portability of collections. migrating images to new digital asset management systems is simpler metadata can be created, edited or deleted for many images per second which is orders of magnitude faster than opening them individually in image processing programs such as photoshop and gimp. consequently, it is practical to use processes that focus on individual metadata elements across many images at once image metadata image metadata is stored directly in the image file. while format and location of metadata in binary files can vary depending on image format, it can often be read simply by viewing the image in a text editor. the following is an excerpt from the top of an image file: ii???@d??f?(1?2?? ??]bi?? ?ci?black and white photograph of esther pohl lovejoy and doctors elliot and moskovetz in athens in 1923.??['??['adobe photoshop cs2 windows2012:04:10 14:16:16<?xpacket begin="?" id="w5m0mpcehihzreszntczkc9d"?> [lines deleted] <rdf:description rdf:about="" xmlns:tiff="http://ns.adobe.com/tiff/1.0/"> <tiff:imagewidth>6046</tiff:imagewidth> <tiff:imagelength>4880</tiff:imagelength> [more lines deleted] <dc:subject> <rdf:bag> <rdf:li>lovejoy</rdf:li> <rdf:li>moskovetz</rdf:li> </rdf:bag> </dc:subject> [more lines deleted] </rdf:description> notice that the metadata is stored as plain human readable xml. though not visible in this example, metadata using several different schemes were stored in this photo. when this happens, it’s clearly labeled, so you can easily identify components you’re interested in. by familiarizing yourself with three complementary image metadata standards, you can maximize the portability of your metadata and develop better photo processing routines. it’s important to be aware that these standards are relatively new so they are inconsistently implemented across platforms. nonetheless, they are still very useful for processing and planning for the future. the most reliably supported image metadata format is exif (exchangeable image file format). virtually all scanners and digital cameras populate images with exif metadata which provides extensive technical information and supports geolocation as well as a number of tiff (tagged image file format) attributes (exchangeable image file format for digital still cameras … [updated 2012]). standard archiving procedures for digital photographs include recording numerous details including height, width, color depth, and resolution. since this information is typically embedded directly in the file, there is no reason not to simply extract it using a utility like exiftool. exif does not provide tags for many access points that are important to libraries, so this information is generally best included using iptc (international press telecommunications council) or xmp (extensible metadata platform) extensions. iptc metadata is built largely around the needs of professional photographers so it provides a number of useful fields such as named individuals in photos. iptc does not provide a number of fields needed by archivists and librarians, but xmp provides great flexibility including the ability to create previously undefined tags. working with multiple standards may sound confusing. however, the process of writing each type of metadata to images is nearly identical as will be demonstrated later. the main thing to remember is to use exif for what you can (mostly technical information), then iptc for some descriptive and administrative information not covered by exif, and xmp for everything else. utilizing facial recognition with archival photographs a number of facial recognition technologies exist and describing them is beyond the scope of this article. there is variation in the accuracy of these technologies, but differences in lighting, facial expression, and viewing angle are problematic for all two dimensional forms of facial recognition (grother et al. 2011). having said that, accuracy is still high enough for facial recognition software to be of significant assistance to those processing digital photographs. facial recognition is useful for a number of reasons: for many photos, identifiable people represent the most important access point by its nature, it provides authority control and prevents multiple variations of the same name from inadvertently being used in a collection identification of individuals can help staff determine when and where a photo was taken as well as the subject matter of the photo even when the software cannot determine who is in the image, extraction of faces simplifies manual identification and organization non-specialist staff can do more metadata work a variety of software applications and services that perform facial recognition are available — even most smartphones have this functionality. for purposes of assigning metadata in a library setting, google’s picasa works well because it is free, fast, and stores information about people in two files that can easily be read. however, any facial recognition software that stores information it finds in a parsable format would be practical in a library setting. staff will want to recognize things as well as people, so be aware that facial recognition is designed to work with people. it is not effective for animals, geographic features, buildings, and artwork. figure 1: picassa facial recognition figure 1 demonstrates the mechanism picasa uses to identify people. an algorithm detects faces which are matched against known individuals. when a probable match is found, picasa provides checkboxes that allow staff to confirm or reject the match. matching becomes more accurate as picasa learns more about each person. staff can assign rejected matches and previously unknown faces to any name desired. accuracy of matching depends on many factors including pose, lighting, facial expression, aging, orientation of photo, image resolution, number of photos previously matched, total number of faces it must compare, and genetic similarity of people in photos. accuracy at oregon health and science university has been well over 90%. however, it is still essential to manually review matches. picasa does not guess when it cannot determine who an image may be. also, if it determines that a face could match multiple probable identities, it lists only those which narrow the list of possibilities staff must consider. like most photo management software, picasa supports tagging. picasa stores tagging information within the exif metadata in the image itself so other programs can utilize this information. however, exif provides no mechanism to indicate where people appear in photographs and who they are so picasa stores this information in .picasa.ini and contacts.xml. as the name implies, contacts.xml keeps track of people. for the example in figure 1, the corresponding entries in contacts.xml are as follows: <contact id=”c0ef2256901bfbb6” name=“esther pohl lovejoy” modified_time=”2012-11-26t09:48:04-08:00″ local_contact=”1″/> <contact id=”c2c65f903b3150cb” name=”joseph matarazzo” modified_time=”2012-11-30t15:02:10-08:00″ local_contact=”1″/> the identifier in red text is used as a key to identify the individuals in .picasa.ini. in the examples above, the key is only linked with a name for the person. if more extensive information is added to the contacts database, it will appear here. the .picasa.ini file contains information describing which individuals were identified in photos and where they appeared. consider the example entries below: [lovejoy-esther_portrait_nd.jpg] faces=rect64(135a175de074cd8b),c0ef2256901bfbb6 backuphash=23375 [matarrazo-joseph_2001.jpg] faces=rect64(3407026fe607ac00),c2c65f903b3150cb backuphash=33 each entry begins with the filename in brackets. the faces entry are four 16-bit hexadecimal entries which define the points of a rectangle containing the face. the backuphash tag is not relevant for purposes of inserting information about people into photo metadata. the important thing to notice that that every person identified in a photo receives an entry in .picasa.ini that points to an individual in contacts.xml. this makes creating a list of all files and associated people trivial in virtually any language, though there is free software that can do this for you (picasa face detection to lightroom … [updated 2010]). incorporating facial recognition data into image metadata the easiest way to interact with metadata in image files is using exiftool, a free perl library that supports a wide variety of open and proprietary metadata standards (harvey … [updated 2013]). during the authors’ tests, it consistently read, modified, and wrote metadata as expected. other software can be used to interact with image metadata, but all programs should be tested with local procedures to ensure no undesirable modification occurs. adding metadata headings to photos is simple. for example, if we want to add a person, we can use an iptc extensions because exif does not define a person tag but iptc does: exiftool -xmp-iptcext:personinimage+=”doe, john” myimage.tif to add a dublin core (dc) subject to a photograph, we use an xmp extension because neither exif nor iptc defines a subject tag: exiftool -xmp-dc:subject+=‘my new heading’ myimage.tif in addition to having a command line interface, bindings for exiftool can be found for many programming languages. exiftool is very useful for synchronizing metadata that is managed by an external image with metadata within the image. extracting image metadata for use in external systems most image management systems do not use image metadata. rather, they store this information in an external database. most libraries encode technical metadata (dimensions, size, color space, etc) by having staff open images manually and extracting this data. this process is slow and error prone. if useful metadata within the file is automatically transferred to spreadsheets or xml files that can be loaded, processing speed will improve while reducing error rates. extracting information from files using exiftool such as technical metadata is simple. labeled display can be extracted with: exiftool filename xml output can be obtained with: exiftool –x filename tab delimited can be generated with: exiftool –t filename embedded metadata within images lacks elements that many libraries want, so it’s best to use it as a starting point to populate spreadsheets, templates, or simply be made available to staff via copy and paste. if staff modify metadata in an external system in accordance with predominant library practices, these changes will not be reflected in the embedded metadata within the image unless specific measures are taken to do so. libraries that write routines that transfer this data automatically to image files should be aware that file checksums will change, which will require integrity monitoring routines need to be modified accordingly. exploration and future directions facial recognition software is a powerful tool for archivists trying to digitize and organize large collections of archival photographs. our case demonstrates that staff with modest programming skills can significantly accelerate creation of accurate item-level metadata using free consumer-oriented software such as picasa. using facial recognition software in conjunction with creative approaches to automation offers intriguing possibilities for making social metadata a more reliable source of descriptive information. staff and researchers alike sometimes recognize individuals who are not identified in photographs. if the user interface further engages users — game interfaces have already been shown to generate more extensive metadata than casual crowdsourced tagging (flanagan and carini, p. 532) — the software can be taught to recognize additional images containing that same individual. combining information from facial recognition software with methods to read and write metadata directly to and from images allows libraries to provide more reliable and extensive metadata for more photographs than can be accomplished with staff and crowdsourced tagging alone. by leveraging the expertise of users who have a high level of knowledge and personal engagement with collections, individuals in photographs can be identified throughout collections which in turn can improve the usability of collections. creating metadata for archival collections will never be an entirely automated process – the interpretation and organization of content as well as legal and ethical issues will always require an archivist’s judgment. while metadata for digital archival collections can not be entirely automated, our example illustrates that more of it can be automated than one might assume. this is a critical consideration for archives faced with large image backlogs requiring detailed item-level metadata. archivists would be well-served to work with systems staff to identify ways in which metadata creation can be automated, giving archives staff time to focus on those areas where human judgment is most valuable. notes [1] since 2011, staff in ohsu historical collections & archives have been working to create item-level metadata for a backlog of several thousand digital images. benchmarking for this project suggests that with some automation and a refined workflow in place, creating metadata for a single image requires an average of 20 minutes of staff time. references exchangeable image file format for digital still cameras: exif version 2.3 [internet]. standard of japan electronics and information technology industries association. camera & images products association. [cited 2013 apr 4]. cipa dc-008-translation-2012. jeita cp-3451c. 185 p. p. 9. available from: http://www.cipa.jp/english/hyoujunka/kikaku/pdf/dc-008-2012_e.pdf extensible metadata plaform (xmp) [internet]. [cited 2013 apr 3]. available from http://www.adobe.com/products/xmp/ flanagan, mary & carini, peter (2012). “how games can help us access and understand cultural artifacts.” american archivist 75(2), pp 514-537. grother, patrick j, quinn, george w., and phillips, jonathon p. 2011. report on the evaluation of 2d still-image face recognition algorithms [internet]. [cited 2013 apr 2]. [cited 2013 apr 4]. nist interagency report 7709. available from: http://www.nist.gov/customcf/get_pdf.cfm?pub_id=905968 harvey, phil. exiftool. [cited 2013 apr 3]. available from: http://www.sno.phy.queensu.ca/~phil/exiftool/ is picasa 3 silently modifying exif info and dropping exif data???!!! google product forums – picasa . [cited 2013 apr 4]. available from: http://productforums.google.com/forum/#!topic/picasa/32jkt3gqsow metadata games. [cited 2013 may 30]. available from http://metadatagames.com picasa. [cited 2013 apr 4]. available from: http://picasa.google.com/ picasa face detection to lightroom. gregor’s blog. [cited 2013 apr 8]. available from: http://blog.gregerstoltnilsen.net/2010/07/picasa-face-detection-to-lightroom/ riecks, david. iptc standard photo metadata (july 2010) : adobe cs5 file info panels user guide [internet]. [cited 2013 apr 4]. available from: http://www.iptc.org/std/photometadata/documentation/iptc-cs5-fileinfo-userguide_6.pdf about the authors kyle banerjee (banerjek@ohsu.edu) is the digital collections and metadata librarian at oregon health & science university. maija anderson (andermai@ohsu.edu) is the head of historical collections & archives at oregon health & science university. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – content dissemination from small-scale museum and archival collections: community reusable semantic metadata content models for digital humanities mission editorial committee process and structure code4lib issue 43, 2019-02-14 content dissemination from small-scale museum and archival collections: community reusable semantic metadata content models for digital humanities this paper highlights the challenges in content dissemination in cultural heritage (ch) institutions by digital humanities scholars and small museums and archival collections. it showcases a solution based on community reusable semantic metadata content models (rm’s) available for download from our community website. installing the rm’s will extend the functionality of the state of the art content management framework (cmf) towards numismatic collections. furthermore, it encapsulates metadata using the resource description framework in attributes (rdfa), and the schema.org vocabulary. establishing a community around rm’s will help the development, upgrading and sharing of rm’s models and packages for the benefit of the cultural heritage community. a distributed model for community reusable semantic metadata content models will allow the community to grow and improve, serving the needs and enabling the infrastructure to scale for the next generation of humanities scholars. by avgoustinos avgousti, georgios papaioannou, and feliz ribeiro gouveia introduction this paper addresses a fundamental gap and challenges incontent disseminationin cultural heritage institutions.  with the advent of the internet, the digital age has influenced almost every aspect of human activity and has transformed it in a revolutionary way never seen before. in particular, the domain of digital cultural heritage has gained in popularity during the last decade. the scientific community has shown new possibilities for integrated access to museum and archival collections. cultural heritage organizations are increasingly eager to cooperate and provide the best possible access to their collections through the web. access to cultural heritage assets provides users with a wide range of opportunities to gain new knowledge.  memory institutions (stainforth, 2016) were probably the first to digitize information, creating online databases to which access was granted only locally through on-site servers. since then, the internet has allowed people to access knowledge from their home computers and increased the number of users drastically.  nowadays the digitization process is essential in the cultural heritage domain, and its accessibility is improved continuously due to open-source communities and technological advancements (cimadomo et al., 2013).  however, museum and archival collections known outside of the area of their discipline are usually supported by large consortiums, expensive technology, and generous funding awards, e.g. europeana[1], the metropolitan museum of art collections[2], the british museum[3] and so forth. (meghini et al., 2017) and (dombrowski, 2016). problem statement unfortunately, the potential of new methodologies and tools to have a transformative impact on individual research and smaller projects is higher than the grant funding available to support extensive, expensive, and extravagant technical undertakings. although there are plenty of ideas for creating online museum collections, there are also many limitations. currently the majority of small museums willing to follow large cultural heritage institutions digitization steps are usually not able to obtain expensive technology.  furthermore museums do not usually hire experts to plan, develop, deploy and maintain a digital collection for them, but delegate them to museum humanities scholars who are often limited in their technological skills. the recruitment of experts with experience and skills is costly, leading to the cost problem mentioned above. many digital humanities scholars and small museums and archives are unable to complete digital projects on their own.  further, even if they do manage to digitize their projects, they are often stored in isolated data silos incompatible with automatic processing and unable to search for related information. these limitations can be addressed by organizing and publishing data using standard formats and by adding structure and metadata to the content of the web pages and linking related data. however, this is not a trivial task, and many humanities scholars with non-technical background usually do not have the technical support to undertake such intricate work. often this extra step requires complex setups and in many cases use of sophisticated, unfamiliar software. these are some of the primary reasons that prevent the dissemination of cultural heritage content and over the web, with the result that content “knowledge” remain unused and hidden. many attempts and much research has, and continues to be, undertaken attempting to provide solutions to these problems. still, the solution remains elusive, and as such, will be the focus of this paper. literature review there are many popular open-source dissemination solutions in the cultural heritage domain. in recent years, there has been an effort, dedicated to cultural heritage, in tailoring to a particular functionality of popular content management systems (cms). a great example is a work by kim christen and craig dietrich (michael, 2014), developers of the mukurtu cms[5], an open source platform. the result is a flexible cms to meet the needs of diverse communities that manage and share their digital cultural heritage online. the platform was developed on the leading open source drupal[6] content management framework. further, it expands the functionality of the software towards the cultural heritage domain. the platform also supports international metadata standards, such as encoded archival description (ead), machine readable cataloging (marc) and dublin core (dc). the platform has been used in many indigenous archiving communities, as in a case study by hall (2016) “tribal archives, traditional knowledge, and local context: why the “s” matters” (christen, 2015), christen explains how to use the system to archive and manage digital assets, and how to help local museums, archives, and libraries to manage, preserve, and reuse their digital cultural heritage. a similar platform is opennumisma developed by a.avgousti at the cyprus institute (cyi): reusable web-based platform that merges digital imaging and data management and offers new opportunities for research and dissemination of knowledge and data (avgousti et al., 2017). the development of opennumisma aims to create a digital web framework for numismatic collections, and it is implemented by small museums to digitize their numismatic collections. in 2018, the platform was deployed for the creation of a numismatic collection for the medieval coins from the bank of cyprus cultural foundation. one of the unique features of this platform is the application of reflectance transformation imaging (rti), a computational photography method that offers tremendous image analysis possibilities for numismatic research (alzua at al., 2005). furthermore, system metadata is based on the popular schema.org vocabulary, a semantic markup supported by major search engines that improves the sharing, discovery, and reuse of the data. it has also been used in the cultural heritage domain (wallis, 2017), (nuno et al., 2017, and ronallo, 2012). the platform was developed on top of the drupal cms and it is available on github. a similar idea was presented by illias at al., (2012) in their research paper “a drupal module for managing museum collections.”, where they present the problem of small museums unable to reach the public through the web. some of the reasons mentioned are the cost and the complexity of the development and customizations of such applications. in his paper ilias, he recommends a different approach and the development of a module that will extend the functionality of a general purpose cms towards museum collections. they also mention the lack of generic cms which uses cidoc-crm[7] and dublin core metadata. alongside, the drupal cms is a powerful open source content management platform that is popular among humanities scholars (dombrowski, 2016). the platform has core website functionality, such as taxonomy search and multilingual capabilities. drupal also has a relatively high learning curve for non-coders who plan to build a site using the user interface. drupal also has been used in support of digital collections and is the first major cms that supports semantic web technologies in its core (havlik, 2011). lenneth varnum’s book, drupal in libraries, covers how to build and maintain a library website using drupal (varnum, 2012). matt ostercamp described the implementation of drupal as a cms at trinity international university rolfing library. along these lines, quinn dombroski recently published a book, drupal for humanists demonstrating how to use drupal to develop small humanities projects (dombroski, 2016). athanasios velios in his research paper “off-the-shelf crm” (velios, 2016) introduced a new method of using the drupal cms in cultural heritage institutions to publish data in both human-understandable and machine-understandable formats using the resource description framework in attributes (rdfa) a flavor of the resource description framework (rdf). such data is suitable for machine consumption. further, athanasios velios (velios, 2011) in his research paper “the john latham archive: an online implementation using drupal” demonstrates how the cms can be used to create an archive system. nowadays, drupal powers an extensive infrastructure of digital cultural heritage projects (dombrowski, 2016), like the louvre website. the smithsonian libraries are using drupal for the management and presentation of their content (thompson et al., 2013), as are the qatar digital library, the new york public library (anderson et al., 2018) and many others. moreover, constantinos sophocleous (sophocleous et al., 2017) describes in “medici 2: a scalable content management system for cultural heritage datasets” a content management system for cultural heritage datasets. it is specialized for management analysis and visualization of ch research. it was developed by the national center for supercomputing applications (ncsa) at the university of illinois at urbana champaign, the library of alexandria (ba) and the cyprus institute (cyi). medici 2.0 can handle datasets in a variety of formats (3d, rti) that are widely used in cultural heritage research. for metadata, the cms used dublin core (dc) where metadata can be shared with the scientific community. also under the category of collection management systems is the work of the roy rosenzweig center for history and new media (rrchnm), and george mason university the developer of omeka, a system developed specifically for scholars, librarians, archive, museums to create their digital exhibitions. omeka was developed with the objective of helping institutions and organizations to manage and collaboratively discover their digital assets using recommended best practices. omeka uses the dublin core for storing metadata about the objects. one of the most famous scholarly extensions of omeka is neatline, a plugin that adds geolocation functionality to an omeka archive and allows the user to tell visual stories with maps and timelines. the recently released next-generation omeka s is more oriented to connecting digital cultural heritage collections using linked open data principles. omeka has been used in many memory institutions and libraries (hardesty, 2014). however, compared to large and robust ecosystems that are available for other large cmss like wordpress, joomla, and drupal, omeka[8] is fairly moderate and with small community support, plugins, developers, and users. another cms similar to omeka is wordpress. it started as an easy to use blogging system and became the system of choice for large and more complex websites (vandegrift, 2013). it is free and open source and has a relatively low learning curve. nowadays, wordpress powers the majority of the sites on the web (martinez et al., 2018). extensions of wordpress can be used to consume museum collections management data. a great number of plugins have been developed by digital humanities to extend wordpress in more advanced ways beyond a simple web presence. dh press is a plugin that allows the users to create events, documents, people and develop links with a location map. other plugins include commentpress, a great text annotation tool for humanities researchers, and the cultural object plugin, which is more oriented to online museum collections and allows the easy pulling and consuming of museum collections management data. yet another choice for a cms is the islandora platform that was developed by the robertson library of the university of prince edward island’s, (moses et al., 2013) and now has its implementation and maintenance under the responsibility of a growing international community (martins et al., 2018). its architecture is built on a base of other software such as drupal cmf, fedora, and solr. islandora[9] offers a set of solution packages that allow its users to manipulate various types of digital objects, such as images, videos, pdf, among others, and fields of knowledge (perkins, 2015). these solution packages enable integration with other software, publishers, data-processing applications, and metadata schemas. at vassar college, islandora offers the students a unique experience by hosting students diaries from the early days of the college (pasquale et al., 2013). islandora is also beneficial if the institution wants to create an image repository. however, the development time is extensive. the islandora system’s complexity and advanced customization usually require support from professional it staff. justification of technology our prototype applications are developed on top of the state of the art content management framework drupal (cmf) and the schema.org ontology. due to the reasons outlined below, drupal will be the cmf to use for our prototype system. however, the implementation of our methodology can apply to any other cmf/cms. the drupal cmf was one of the first major systems to support semantic technology in its core. more specifically, drupal supports rdfa[10] which allows expression of structured data in markup languages. also, it requires limited technical knowledge which allows non-technical individuals to easily publish, manage, and organize a variety of content. over the past few years, drupal has become very popular among humanities projects. (dombrowski, 2016). moreover, it is open source software with the vital community of user and developers that usually comes with at no-cost, and some of the community groups are dedicated to humanities projects (avgousti, 2018). a comparison made by ibm also shows the benefits of using drupal among other cmss (weitzman et al., 2006) see figure 1. figure 1.content management systems comparison adapted from: (weitzman, et al, 2006) [full box = easy] [ empty box = difficult] [half box = medium ] why the drupal content management framework is appropriate to meet our goals: it has familiar tools with which non-technical researchers can publish data without expensive and extensive technical support it is a popular tool among humanities scholars and researchers a significant amount of digital humanities infrastructure is built upon drupal a significant amount of museum and archival collections are using drupal it is the first major cmf that supports semantic web technologies in its core it is free and open-source community priorities: free web experience without the control of big industry giants modular system, each module offering part of its functionality large distribution ecosystem philosophy: software freedom, community ownership and intersectional politics to co-develop technologies collaborative approach and large community support one of the most popular open-source general purpose content management frameworks next we describe how drupal is used by our methodology. community reusable semantic metadata content models (rm’s) nowadays, humanities scholars willing to digitized their research projects can accomplish it by using existing open source platforms and without the need of hiring a computer specialist (dombrowski, 2016). however, this process usually involves a steep learning curve, and a significant amount of time that generally humanities scholars do not have (ilias daradimos, 2015). furthermore, they rarely take the extra step to embed structured data on their online projects, and their research is often stored in isolated data silos (sikos, 2015). we aim to develop community reusable semantic metadata content models (rm’s). the models will be a collection of reusable elements which taken together satisfy a particular case related to cultural heritage institutions. examples: a) numismatic collections b) photographic collections c) works of art collections d) ancient graffiti collections, and so forth. it would be sufficient to package, reuse and improve this functionality once and reuse it again and share it with others in a larger community. the following image represents an example of community reusable semantic metadata content models (rm’s). this data model representing ancient graffiti, showing the complexity and the research required to create the data modeling and the systems architecture. figure 2.example of rm’s data model [representing ancient graffiti] adapted from : mia gaia trentin, post doc fellow at the cyprus institute science and technology in archaeology research center (starc)-ncsa university of illinois at urbana champaign the following list represents some of the fundamental advantages willing to achieve in cultural heritage domain by using community reusable semantic metadata content models (rm’s). to increase dissemination of content from small museum and archival collections to reshare, improve and upgrade rm’s to lower the complexity of the digitization process to reduce small scale museum and archives collections development complexity to increase the use of open-source applications in cultural heritage domain to reduce human errors to lower development costs to create labor-saving tools to minimize software complexity to expand semantic markup annotations in this paper, we are introducing our first rm’s that are ready to use and downloaded from our website (https://rms.cyi.ac.cy). the rm’s available for download and installed with a few clicks, and it will extend the functionality of the drupal cmf towards numismatic collections. further, it encapsulates metadata using the rdfa format and uses the schema.org vocabulary, supported by major search engines aiming to improve sharing, discovering and reusing of online content. in 2011 the major search engines (google, yandex, bing, yahoo) introduced the schema.org (jason, 2012 and (guha, 2015) an ontologies that is understandable to web crawlers and other machines (shayna, 2018). given the popularity of search engines, many content creators and site administrators will most probably integrate the schema.org on their projects. nowadays, the schema.org already implemented in millions of websites and used by cultural heritage organizations like europeana (wallis, 2017). figure 3. rm’s data modeling-numismatic collections further, by downloading and installing, the rm model will generate a pre-defined collection of data types known to drupal as “fields” which are related to each other by an informational context. the pre-defined collection of data types were developed to support an online numismatic collection data model. all the fields were selected in and with research, collaboration and in a joint effort with numismatic experts. while there is no single perfect data model for a given case study, this data model was developed very carefully, and with the confidence that it will cover a good percentage of the needs of a small numismatic collection project. furthermore, the option of extending and improving the model by adding more functionality it’s always welcomed, and it is the aim of our future community. each field also uses the schema.org ontology to add structured metadata embedded in every html page generated. with that, all major search engines (google, yandex, bing, etc) understand the published content more efficiently. figure 4. example of generated pre-defined collection of data types installing the rm’s will automatically generate and display a list of content that will be able to present all the collection in the form of a list or table. moreover, it creates filters for easy information retrieval where the user can filter down the information by using the artist, material, and issue authority. the rm also generating taxonomical vocabularies and allow the end user to populate them with terms, for example, axis, mint, weight, etc. taxonomic terms can be used to classify content, and in this case, probably the most important use is to relate content. for example, searching for all coins that have weight 1.73 mm the system will be able to locate coins based on the specific taxonomic term. also, it automatically creates a display template, where the images are shown on the left and the information about the images on the right (see fig: 6). with that users can easily read the content and at the same time view the images. figure 5. list of all coins collection sort by taxonomy (medieval coins from the bank of cyprus cultural foundation collection) figure 6. generated pre-defined collection of taxonomic categories figure 7. left: html page (medieval coins from the bank of cyprus cultural foundation collection). right: a real-time rdfa, data visualizer using rdfa play tool applicability the first successful implementation of community reusable semantic metadata content models (rm’s-numismatics) was implemented for the development of a kiosk website installation at the leventis municipal museum in nicosia, cyprus, hosting a numismatic collection of about 125 coins. (http://coins.cyi.ac.cy) the second request came from the fluminense federal university in brazil willing to see the possibility of using our rm’s-numismatic for their online collection. conclusion and future work establishing an online community the broader idea is to establish an online community “an online community is a group of people with common interests who use the internet (websites, email, instant messaging, etc.) to communicate, work together and pursue their interests over time”(“what is an online community?” 2018)”. some of the reason of the development of an online community for community reusable semantic metadata content models (rm’s) is to allow digital humanities scholars, site builders, computer scientists, digital humanities, site administrators, web developers, and so forth to create, develop, collaborate, improve, upgrade and share their models, packaged for the benefit of the cultural heritage community. a distributed model for community reusable semantic metadata content models will allow our community to grow and be more flexible, serve more needs and give the infrastructure to scale for the next generation of humanities scholars. step 1: develop community reusable semantic metadata content models that will cover other areas of museum and archives collections. example: photographic collections, folk collections, and so forth. step 2: we already created the first version (beta version) of our community website (https://rms.cyi.ac.cy) that will allow members of the community to share community reusable semantic metadata content models. in the site, users can learn about rm’s and able to download and use our first module describe in this paper. step 3: is to create a email-list, forum and discussion group, where members can share ideas and discuss the further development of new or existing rm’s. for example, development of a new rm that will cover a new area in cultural heritage, or improve an existing rm’s with more functionality either create a new one that will extend the functionality of an existing one. step 4: we are planning to have an upload session to register users, where they can upload their own rm’s, to share them with the rest of the community. the new uploaded rm’s will be first tested and only then published online. step 5: disseminate the existence of our community by attending conferences, events related to cultural heritage and technology conferences. dissemination: conferences and events attended icomon 2018 “building a digital platform for numismatic collections” acropolis museum, athens, greece drupal hack camp 2018 “drupal and schema.org in the semantic interoperability of cultural heritage knowledge” bucharest, romania library of alexandria 2017 “ digital library for cultural heritage” alexandria, egypt oxford university 2016 “platform for numismatic collections” oxford, england united nations headquarters 2015 “drupal for cultural heritage” new york, usa figure 8.: left: web-community reusable semantic metadata content models. right: rm’s package model about the authors avgoustinos avgousti (a.avgousti@cyi.ac.cy) is a research technical specialist at the cyprus institute at the science and technology in archaeology research center (starc). he is the web architect and lead developer of dioptra: the digital library for cypriot culture. he received his bachelor of science (bsc) from st. francis college in new york city in information technology/computer science and master of science (msc) in medical informatics from the state university of new york downstate medical center in new york city. he has more than 12 years of experience in information technology and 3 years of teaching computer science at a college level. furthermore, he participated in a number of eu and national funded projects. avgoustinos’ research activity has been focused in the last five years on digital humanities and digital cultural heritage. dr georgios papaioannou is an associate professor for the ma in museum and gallery practices university college london (ucl) qatar. his research interests lie in museology, archaeology of the eastern mediterranean and the arab world, education (including e-learning), tourism (including city tourism), cultural studies and it applications, including virtual reality, augmented reality, big data and mobile applications. he has lectured, excavated, led tours and conducted museum / cultural heritage work in the uk, greece, italy, albania, cyprus, sweden, australia, spain, egypt, syria, oman, turkey, yemen and saudi arabia. dr papaioannou is general-secretary of the hellenic society for near eastern studies, director of the museology lab in corfu (greece) and a member of icom. dr feliz ribeiro gouveia (fribeiro@ufp.edu.pt) is currently an associate professor of computer engineering at the faculty of engineering of university fernando pessoa, porto, portugal. he has worked in research and technology transfer projects for industry, in the areas of knowledge management and object-oriented databases for major european companies. he participated in several research projects in cultural heritage and digital humanities and has acted as consultant for organizations implementing cultural collection management projects. he graduated in electrotechnical engineering from the university of porto and holds a phd in computer science from the université de technologie de compiègne, france. notes [1] https://www.europeana.eu/portal/en [2] https://www.metmuseum.org/art/collection [3] https://www.britishmuseum.org/research/collection_online/search.aspx [4] https://html.com/semantic-markup [5] http://mukurtu.org [6] https://www.drupal.org [7] http://www.cidoc-crm.org [8] http://wiss-ki.eu [9] https://omeka.org [10] https://islandora.ca [11] https://rdfa.info references alzua-sorzabal, aurkene and linaza, maria teresa, and abad, marina, and arretxea, larraitz, and susperregui, ana. 2005. interface evaluation for cultural heritage applications: the case of ferrum exhibition. the 6th international symposium on virtual reality, archaeology and cultural heritage. vol. vast. doi:10.2312/vast/vast05/099-106. anderson, jennifer l, and edwin guzman.  2018. “adaptation: the continuing evolution of the new york public library’s digital design system.” code4lib journal,  issue 41.   https://journal.code4lib.org/articles/13657 avgousti, avgoustinos, andriana nikolaidou, and ropertos georgiou. 2017. “openumisma: a software platform managing numismatic collections with a particular focus on reflectance transformation imaging.” code4lib journal, issue 37. https://journal.code4lib.org/articles/12627 brawley-barker, tessa. 2016. “integrating library, archives, and museum collections in an open source information management system: a case study at glenstone.” art documentation: journal of the art libraries society of north america. doi:10.1086/685979. butkiewicz, michael. 2011. “understanding website complexity?: measurements , metrics , and implications categories and subject descriptors.” proceedings of the 2011 acm, 313–28. doi:10.1145/2068816.2068846. christen, kimberly (2015) “tribal archives, traditional knowledge, and local contexts: why the “s” matters,” journal of western archives: vol. 6 : iss. 1 , article 3.  https://digitalcommons.usu.edu/westernarchives/vol6/iss1/3 cimadomo, guido. 2013. “documentation and dissemination of cultural heritage: current solutions and considerations about its digital implementation.” proceedings of the digitalheritage 2013 – federating the 19th int’l vsmm, 10th eurographics gch, and 2nd unesco memory of the world conferences, plus special sessions fromcaa, arqueologica 2.0 et al. 1: 555–62. doi:10.1109/digitalheritage.2013.6743796. corlosquet, stéphane, and cyganiak, richard, and polleres, axel,  and decker, stefan.  2009. “rdfa in drupal: bringing cheese to the web of data.” ceur workshop proceedings.  https://aic.ai.wu.ac.at/~polleres/publications/corl-etal-2009.pdf daradimos, ilias, and vassilakis, costas vassilakis, and katifori, akrivi.  “a drupal cms module for managing museum collections.” conference paper, march 2015. di pasquale, j., and palmentiero, j., and streett, l. (2013), “using archives and metadata to uncover women’s lives: challenges and opportunities for scholarship through archives and digital libraries”, paper presented at the women’s history in the digital world conference, 22-23 march, at bryn mawr college, bryn mawr, pennsylvania, available at: https://repository. brynmawr.edu/greenfield_conference/papers/saturday/22/ dombrowski, quinn.  drupal for humanists. texas a&m university press, 2016. flake, gary william, and lawrence steve, and giles, c. lee.  “efficient identification of web communities.”  proceedings of the sixth international conference on knowledge discovery and data mining, august 20-23, 2000, pp. 150-160. doi: 10.1145/347090.347121  freire, nuno, and charles, valentine, and isaac, antoine.  “evaluation of schema.org for aggregation of cultural heritage metadata.”  gangemi a. et al. (eds) the semantic web. eswc 2018. lecture notes in computer science, vol 10843. springer, cham.  https://www.inesc-id.pt/publications/13560/pdf/ gasparotto, melissa. 2014. “search engine optimization for the research librarian: a case study using the bibliography of u . s . latina lesbian history and culture.” practical academic librarianship: the international journal of the sla academic division. volume 4, n. 1, june 2014, pp. 15-34.   https://journals.tdl.org/pal/index.php/pal/article/view/6971. guha, r. v.,  and brickley, dan and macbeth, steve. “schema.org: evolution of structured data on the web.” queue 13 (9), nov. 2015.  doi:10.1145/2857274.2857276. havlik d. 2011.  building environmental semantic web applications with drupal. in: h?ebí?ek j., schimak g., denzer r. (eds) environmental software systems. frameworks of eenvironment. isess 2011. ifip advances in information and communication technology, vol 359. springer, berlin, heidelberg.  doi: 10.1007/978-3-642-22285-6_42. kawano, h. 2012. digital archive system using cms and gallery tools –implementation of anthropological museum.  https://www.cut.ac.cy/euromed2012proceedings/shortpapers/125.pdf hardesty, juliet l. 2013. “exhibiting library collections online: omeka in context.” new library world 115 (3–4): 75–86. doi: 10.1108/nlw-01-2014-0013. martinez-caro, jose-manuel, and aledo-hernandez, antonio-jose, and guillen-perez, antonio and sanchez-iborra, ramon, and cano, maria-dolores. 2018. “a comparative study of web content management systems.” information 9 (2): 27. doi: 10.3390/info9020027. matt ostercamp, 2010 “a new way to create a website: using drupal to create a dynamic web presence.” american theological library association summary of proceedings 64 (2010): 139-147. moses, donald, and stapelfeldt, kirsta.  2013. “renewing upei ’ s institutional repository: new features for an islandora-based environment.” code4lib journal,  issue 21.  https://journal.code4lib.org/articles/8763 murugesan, san, and yogesh deshpande. 2005. “web engineering: introduction and perspectives.” web engineering – managing diversity and complexity of web application development 2016: 1–2. doi:10.1007/3-540-45144-7. pantoja, javier, and docampo, javier, and martín, ana, and maturana, ricardo alonso, and navalón, carlos (2016). the new prado museum website: a semantic challenge. retrieved april 6, 2018: https://mw2016.museumsandtheweb.com/paper/the-new-prado-museum-website-a-semantic-challenge/ perkins, stephen. 2015. “introduction to islandora.” infoset digital publishing. http://www.infoset.io/articles/introduction-to-islandora. ronallo, jason. 2012. “html5 microdata and schema.org.” code4lib journal,  issue 16. https://journal.code4lib.org/articles/6400 royce, dr. winston w. 1970. “managing the development of large software systems.” ieee wescon, no. august: 1–9. shepard, michael. “review of mukurtu content management system.” language documentation & conservation, vol. 8, 2014. pp. 315-325. http://hdl.handle.net/10125/24610 stainforth, elizabeth. 2016. “from museum to memory institution: the politics of european culture online.” museum & society. 14 (2): 323–37. samis p.s., johnson, l.f. and smith, r.s. (2007), “pachyderm: from multimedia to visual stories.” journal of computing in higher education, vol. 19 no. 1, pp. 3-25. scholz, martin, and guenther goerz. 2012. “wisski: a virtual research environment for cultural heritage.” frontiers in artificial intelligence and applications, 242: 1017–18. doi:10.3233/978-1-61499-098-7-1017. sikos, leslie f.  mastering structured data on the semantic web. 1st ed. apress, 2015. sophocleous, constantinos. 2017. “medici 2: a scalable content management system for cultural heritage datasets” code4lib journal, https://journal.code4lib.org/articles/12317 thompson, keri, and joel richard. 2013. “moving our data to the semantic web: leveraging a content management system to create the linked open library.” journal of library metadata, doi:10.1080/19386389.2013.828551. tongue, charles. 2018. “museum survey 2017 – vernon systems.” vernon systems. https://vernonsystems.com/museum-survey-2017/. vandegrift, micah, and stewart varner. 2013. “evolving in common: creating mutually supportive relationships between libraries and the digital humanities.” journal of library administration, 53 (1): 67–78. doi:10.1080/01930826.2013.756699. varnum, kenneth j.  drupal in libraries. london: facet, 2012. vavliakis, konstantinos n. and karagiannis, georgios and pericles, mitkas a. “semantic web in cultural heritage after 2020.” conference: 11th international semantic web conference 2012 (iswc 2012) velios, athanasios.  “the john latham archive?: an on-line implementation using drupal.”  art documentation: journal of the art libraries society of north america, vol. 30, no. 2, fall 2011, pp. 4-13. wallis, richard, and isaac, antoine and charles, valentine, and manguinhas, hugo. 2017. “recommendations for the application of schema.org to aggregated cultural heritage metadata to increase relevance and visibility to search engines: the case of europeana.” code4lib journal,  april 2017. http://journal.code4lib.org/articles/12330. weist, j. (2010), “implementing collective: access at the bruce high quality foundation university archive.” visual resources association bulletin, vol. 37 no. 2, pp. 23-26. weitzman,louis, and lewis-bowen, alister, and evanchik, stephen. using open source software to design, develop, and deploy a collaborative web site, part 1: introduction and overview.  july 2006.  https://web.archive.org/web/20070901123433/http://www-128.ibm.com:80/developerworks/ibm/library/i-osource1/index.html “what is an online community?”. 2018. common craft. https://www.commoncraft.com/archives/000208.html. whysel, noreen. 2014. “a survey of digital humanities skillshare applications.” doi:10.7916/d8571903 wiltshire, n g.   “the use of sahris as a state sponsored digital heritage repository and management system in south africa.” isprs annals of the photogrammetry, remote sensing and spatial information sciences, volume ii-5/w1, 2013. xxiv international cipa symposium, 2 – 6 september 2013, strasbourg, france. https://www.isprs-ann-photogramm-remote-sens-spatial-inf-sci.net/ii-5-w1/325/2013/isprsannals-ii-5-w1-325-2013.pdf subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – data munging tools in preparation for rdf: catmandu and lodrefine mission editorial committee process and structure code4lib issue 30, 2015-10-15 data munging tools in preparation for rdf: catmandu and lodrefine data munging, or the work of remediating, enhancing and transforming library datasets for new or improved uses, has become more important and staff-inclusive in many library technology discussions and projects. many times we know how we want our data to look, as well as how we want our data to act in discovery interfaces or when exposed, but we are uncertain how to make the data we have into the data we want. this article introduces and compares two library data munging tools that can help: lodrefine (openrefine with the deri rdf extension) and catmandu. the strengths and best practices of each tool are discussed in the context of metadata munging use cases for an institution’s metadata migration workflow. there is a focus on linked open data modeling and transformation applications of each tool, in particular how metadataists, catalogers, and programmers can create metadata quality reports, enhance existing data with lod sets, and transform that data to a rdf model. integration of these tools with other systems and projects, the use of domain specific transformation languages, and the expansion of vocabulary reconciliation services are mentioned. by christina harlow introduction with the expansion of digital and physical library platforms, as well the increasing need to both migrate and work with metadata between these platforms, library data munging has increased in importance. library data munging or wrangling—defined here as the work of remediating, enhancing, and transforming library datasets—creates a better understanding of the state of metadata and supports a more flexible and interoperable data ecosystem. resting at the intersection of library metadata, data modeling, and scripting, data munging can require specific but varied skills. the need to wrangle data as well as the number of skills required for efficient and accurate data work has increased along with the growing number of institutions beginning to experiment with and migrate to resource description framework (rdf) modeled data.[1][2] described here are some tools that support a team with many different skillsets to perform library data munging and work with rdf data. two such tools are discussed in this article: lodrefine and catmandu. both freely-available, open-source data tools help make data wrangling easier, although they solve different problems. because of this and the nature of data work[3], a person will probably use both of these tools for varying use cases. they are introduced here within the context of a data munging and migration workflow currently used at the university of tennessee knoxville (utk) libraries. metadata munging workflow context(s) at utk, we are migrating most of our marc and non-marc metadata and objects into new platforms. for digital collections, this involves going from a variety of platforms to either an instance of islandora or to a yet to be determined, fedora-based platform for a new institutional repository. for the marc catalog, utk moved recently from aleph to alma, involving the migration of some bibliographic records built for aleph-specific local procedures (such as acquisitions stubs). now we need to review the migrated data to remove some defunct data practices and review possibilities for enhancements. to start this work, we first created utk metadata documentation, including interoperable metadata application profiles (maps) for the various platforms. the following, primarily descriptive-focused metadata workflow provides an overview of the work done to update, remediate, and support data consistency. part of this workflow includes generating reports that check and show the state of the original metadata, the changes made, and how these changes will improve discoverability. there was also the need to build a workflow that includes a wider number of internal stakeholders than has been involved before in metadata migration work – such as the content specialists that created the original metadata, catalogers who are re-skilling for working more with non-marc and batch metadata, and digital library developers who want to pull new-to-them metadata schema into indexing engines and platforms. these workflow steps also address preparation for rdf by both reconciling data with authority datasets available as linked open data as well as producing rdf descriptive metadata for experimentation and educational purposes. catmandu and lodrefine have become an essential part of this workflow. lodrefine/openrefine background openrefine (formerly google refine) is a tool that has gained a lot of traction in library data work in the past few years. built originally by david huynh and metaweb technologies, it has gone through a number of changes until in 2012 it became openrefine, an entirely community-sourced project.[4] one can read more about the tool, how to get involved, and any new developments on the openrefine github repository, on the openrefine website, or from the openrefine twitter account.[5][6][7] openrefine is described as “a power tool that allows you to load data, understand it, clean it up, reconcile it, and augment it…”.[8] built in java with jetty, openrefine runs locally, with editing occurring in a web-browser-based interface built with javascript and jquery, and all project data and associated work is stored locally and managed by jetty. along with the data munging abilities built into openrefine, there are also a growing number of community-sourced extensions that can be added. discussed here is the deri rdf extension developed by researchers at the linked data research centre, part of the former digital enterprise research institute (now the insight centre for data analytics) in galway, ireland.[9][10] a fork of openrefine, called lodrefine, comes with these deri rdf extensions and a few others included in the install.[11] at the time of the writing of this article, lodrefine as well as the deri rdf extension are no longer being actively developed, however they are still widely used. lodrefine is primarily what is used in the examples given below. getting started with openrefine/lodrefine there are a number of ways to install lodrefine. for the most recent version of openrefine, follow the instructions on the openrefine github wiki.[12] the download for the deri rdf extension discussed here can be found on the deri refine site; follow the instructions on the wiki to install it as an extension in openrefine.[10][13] alternatively, lodrefine can be installed as a single unit from its github repository.[11] since hosted versions of openrefine do not provide all the functionalities described below, each person at utk works with lodrefine installed locally on their computer(s) there is great documentation on the openrefine github wiki and elsewhere on the web for getting started with the core features of openrefine/lodrefine.[14][15] the real strength of openrefine/lodrefine is that it gives the user a graphical user interface (gui) for data normalization, enhancement, and mapping to json, xml and rdf (and other). this allows users with less comfort working in a command line interface (cli) or scripting to still be involved in the data munging workflow. additionally, the openrefine/lodrefine gui is built for browsing and visual review of metadata fields for data outliers and errors. lodrefine offers ways to perform data value clustering; batch edits using a domain-specific language called google refine expression language (grel); remediation against external datasets available via api, rdf document, or sparql endpoint; and the ability to build an rdf export skeleton for exporting each record as a node in a rdf graph serialized as n-triples or rdf/xml. in the context of this workflow, a weakness of lodrefine is that the user must convert existing metadata to a tabular model for working in lodrefine, then back into the preferred data model and format for the new platform or datastore. there are also performance issues for working with larger and more complex datasets (based on experiences at utk, this generally means ~50k mods records as a maximum) in lodrefine or for local document reconciliation work. additionally, working through similar mappings, review, or data transformations requires many steps in the gui. this is where catmandu supplements and builds out the data munging workflow from lodrefine. catmandu background catmandu is a data tool developed as part of the librecat project.[16] the librecat project aims to create library data workflows that can be consistently applied to different datastores, data types, and platforms within libraries – in particular, the similar but platform-dependent data munging needs of institutional repositories, digital collections, and more traditional library data workflows. catmandu, the core of the librecat project, is an open source, freely-available suite of library data wrangling modules currently supported and developed primarily by three european universities – the universities of lund, gent and bielefeld – and a team of developers.[16] catmandu works via command line interface or in perl scripting, both giving the user access to various perl modules for working with library data. catmandu works using an extract-transform-load (or etl, popular with many big data/data warehouse tools) workflow. this workflow is built to interact with items, the term for a basic unit of data in catmandu, like so: extract: a variety of catmandu importer modules extract data from remote sources and either import the data into an application or store it locally. transform: the user can then review, enhance, and transform that data using routines built off of the catmandu-specific fix language. the fix language is a key component to catmandu that was built to allow non-metadata specialists to perform data munging. load: the data extracted and transformed using fix routines can then be ingested into a datastore, whether indexing engine, database, or other. catmandu currently offers modules for pushing data to a datastore like mongodb [17] or an indexing engine like elasticsearch [18]; pulling data from or exposing data via oai-pmh [19] or via sru [20]; and pushing data into platforms such as dspace [21] or fedora [22]. for more information on working with catmandu, the librecat catmandu online handbook is recommended.[23] installation instructions for catmandu are available there.[24] getting started with catmandu because catmandu is a suite of perl modules, the core installation is getting the catmandu cpan module, along with all dependencies, installed in the working environment.[25] alternatively, there is a virtual machine with catmandu installed available for download from the librecat blog[26], and there is a docker image of a recent build of catmandu available from the docker repositories registry.[27] the user may need to install additional modules beyond catmandu for other data formator platform-specific functionalities. for those not using a virtualization option, install the cpan task::catmandu module, which comes with a set of recommended catmandu modules included. while catmandu can be used in perl scripts or via a command line interface (cli), the following workflow takes advantage primarily of the latter. working in the cli, users can configure their local installation of catmandu or specific catmandu projects to have default datastores, formats, importers, exporters, or other options; this just requires saving a catmandu configuration yaml file in the project’s directory.[28] the developers of catmandu created a very helpful cheat sheet for working with catmandu in a cli which can be used to review commands discussed below.[29] the strengths of catmandu are pulling data from a variety of platforms or sources, relatively easy conversion of these datasets to different formats better suited for different parts of a munging workflow, easy report generation, and ability to use the fix language, a catmandu-specific data wrangling language, in etl processes. the fix language works with paths that refer to specific parts of a data record or item, uses specified functions to edit that data, and can use conditionals and/or binds to better specify when to apply edits. fixes are meant to be used for easy data manipulation by non-programmers, and can be run in-line or via a saved fix file. the formal fix language grammar is given in the catmandu::fix::parser module, and the fixes cheat sheet contains a condensed summary.[30][31] the fix language is not used heavily at present in the following utk data munging workflow, as most of the normalization, enhancement, and remediation work is done by content specialists and catalogers in lodrefine. this is because lodrefine offers a user-friendly gui and more external data remediation options at present. a consideration for utk in using catmandu is that it requires knowledge of the cli, bash, and/or perl scripting basics; this creates a workflow involving people with many different technology comfort areas as well as vastly different original metadata sets. there is a goal to expand use of catmandu and lessen the use of lodrefine as metadata sets are normalized and the enhancement work needing more manual/visual review is completed, while concurrently some of the cataloging staff reskills to become more comfortable with the cli and using scripts. data wrangling workflow import/pull data some utk (soon-to-be) legacy platforms either do not give the option for a full data dump/export or the platform-supported data export methods creates poor-quality data (i.e. invalid xml feeds).thus, for non-marc metadata, utk uses standard oai-pmh feeds (built with pkp harvester) as the default option for extracting descriptive metadata. the oai-pmh dc/xml descriptive metadata records are pulled and checked to make sure there is a consistent method for linking the record to other parts of the digital object, in most cases, the digital asset file stored on one of the digital collection servers. most of the non-marc datasets were created in ‘collections’ that have various meanings according to the platform and the project. what collections share, however, is that the metadata can follow different standards, interpretations, and guidelines according to a number of factors (some unknown). much of the utk metadata migration work addresses these inconsistencies and aligns all the descriptive metadata across platforms to a core utk metadata application profile. in practice, this means the workflow described below for non-marc work is applied to each collection individually instead of to full platform feeds. marc data is exported from the utk alma ils using application-specific functionalities, then locally stored in smaller datasets of ~50,000 bibliographic records each (with the total number of bibliographic records currently around 2 million). there has been some experimentation with accessing the data via the alma api, but this is limited by the api requirement of using alma-specific record identifiers to export records (so the identifier lists to feed into the api would need to be built manually). pushing back enhanced data to alma via the api holds more promise. there are a number of supported formats and methods for importing data into an application or store accessible to catmandu. depending on the work required, the user might wish to extract data from an external source and save it locally, which can use the convert or export commands, or import the data to a local development datastore or indexing engine. the following convert command pulls a simple dc/xml metadata set from the utk oai-pmh feed (using the catmandu oai module) and saves the data locally as [oai set name].oai.dc.xml: $ catmandu convert oai \ --url http://dloai.lib.utk.edu/cgi-bin/xmlfile/dloai/oai.pl \ --metadataprefix oai_dc \ --set egy \ --handler oai_dc to xml > egy.oai.dc.xml the next command pulls a mods/xml metadata set from the utk oai-pmh feed for migrated mods metadata and saves the data locally as [oai set name].oai.mods.xml: $ catmandu convert oai \ --url http://dloai.lib.utk.edu/cgi-bin/xmlfile/dlmodsoai/oai.pl \ --metadataprefix mods \ --set kefauver \ --handler raw to xml > kefauver.oai.mods.xml note the use of convert with the oai command to pull data from an oai-pmh feed. some oai-specific flags, including url, metadataprefix, and set, target the data wanted. the command then uses the handler flag to specify that catmandu should use an oai_dc parser for the dc set.[32] there is also a specific handler for marcxml, but other formats and schemas, such as mods, must use either the generic struct (the default) or raw handlers to work with the xml as found.[33] this causes issues currently in pulling oai-pmh mods/xml metadata (discussed further below, though support for mods is being added at the time of the writing of this article). finally, to xml ends the convert command, and tells catmandu to change the data to xml instead of to its default of json. the user could also pull that same data from the oai-pmh feed, but convert to another format (like json), and store that locally instead: $ catmandu convert oai \ --url http://dloai.lib.utk.edu/cgi-bin/xmlfile/dloai/oai.pl \ --metadataprefix oai_dc \ --set egy \ --handler oai_dc > egy.oai.dc.json $ catmandu convert oai \ --url http://dloai.lib.utk.edu/cgi-bin/xmlfile/dlmodsoai/oai.pl \ --metadataprefix mods \ --set kefauver > kefauver.oai.mods.json the convert command can handle working with xml, rdf, yaml, marc, marc/xml, json, csv, excel, and other formats. when saving locally, it saves each record as a separate catmandu item in the same output file – resulting in invalid xml that may require file-splitting before further work. this method is useful, however, for sending these records to an application or database, as well as for loading this data into an application such as lodrefine. to import the oai-pmh or other dataset into a store, use the import command. the example extracts and stores an oai-pmh dc/xml set and an oai-pmh mods/xml set as json in an existing local mongodb database for use in data reports and visualizations. the count command then queries that mongodb database collection/bag and returns the number of objects stored. $ catmandu import oai \ --url http://dloai.lib.utk.edu/cgi-bin/xmlfile/dloai/oai.pl \ --metadataprefix oai_dc \ --set egy \ --handler oai_dc \ to mongodb \ --database_name viz_database \ --bag oaidc $ catmandu count mongodb --database_name viz_database --bag oaidc $ 231 $ catmandu import oai \ --url http://dloai.lib.utk.edu/cgi-bin/xmlfile/dlmodsoai/oai.pl \ --metadataprefix mods \ --set kefauver \ to mongodb \ --database_name viz_database \ --bag oaimods $ catmandu count mongodb --database_name viz_database --bag oaimods $ 315 mapping and review in catmandu catmandu’s convert and import can be used for quick review of a dataset and its fields in the cli, as well as transforming library datasets easily to other formats that work better with visualization tools or data applications like lodrefine. for the datasets retrieved in the previous section, typing in the following command will convert the json data to yaml shown in the cli for easy review: $ catmandu convert json to yaml < egy.oai.dc.json figure 1. screenshot of yaml output for first record in above convert command to have this conversion work with the xml data stored, it will need to be saved first as one well-formed (though not necessarily valid) xml document. the yaml output can be good for a quick browse of metadata fields. catmandu also allows the user focus on a particular field with simple in-line fix routines. the following command will convert the json data to yaml and only return the “title” field in the cli: $ catmandu convert json \ --fix 'retain(title)' to yaml < egy.oai.dc.json figure 2. screenshot of cli output for above command asking for title only the user can also generate a csv report. the following command generates a csv file in the cli from the json created, using the very simple, referenced fix file (shown below) that removes extra whitespace (or trims) then joins fields that may have multiple values for the same field (for easier viewing in tabular format): $ catmandu convert json \ --fix dctocsv.fix to csv < egy.oai.dc.json > egy.oai.dc.csv --dctocsv.fix --do visitor() trim(scalar) end join_field("_about", " | ") join_field("subject", " | ") join_field("relation", " | ") join_field("_setspec", " | ") join_field("title", " | ") join_field("type", " | ") join_field("creator", " | ") join_field("rights", " | ") join_field("source", " | ") join_field("description", " | ") join_field("publisher", " | ") join_field("coverage", " | ") join_field("identifier", " | ") join_field("contributor", " | ") additionally, the following command and fix file will take the oai-pmh dc/json and perform a preliminary mapping (saving the output as a new json file) of the data in preparation for loading into lodrefine: $ catmandu convert json \ --fix dctoopenrefine.fix egy.oai.dc.json > egy.pre-or.dc.csv --dctolodrefine.fix --unless all_match('_status', 'deleted') copy_field('_datestamp', 'record.recordcreationdate') copy_field('_id', 'record.recordidentifier') copy_field('_identifier', 'record.identifier.$append') copy_field('_setspec', 'record.oaicollection.$append') join_field('_setspec', ' | ') split_field('_setspec', ' | ') copy_field('contributor', 'record.namecontributor.$append') join_field('namecontributor', ' | ') copy_field('coverage', 'record.geographicsubject') copy_field('creator', 'record.namecreator.$append') join_field('namecreator', ' | ') split_field('namecreator', ' | ') copy_field('description', 'record.abstract') if any_match('identifier', 'http://\w+') copy_field('identifier', 'record.objectincontexturl') else copy_field('identifier', 'record.identifier.$append') end; if all_match('publisher', 'university of tennessee libraries') copy_field('publisher', 'record.recordsource') else copy_field('publisher', 'record.publisher') end; if all_match('relation', '^\d{2}//\d{2}//\d{2}$') copy_field('relation', 'record.date') else copy_field('relation', 'record.relateditem') end; copy_field('rights', 'record.accesscondition') copy_field('source', 'record.relateditemcitation') copy_field('subject', 'record.subjecttopical') copy_field('title', 'record.title') copy_field('type', 'record.typeofresource') end; retain('record') for working with the mods datasets, the same methods as above can be used, with the fix file being expanded for working with all the possible field names reviewed in the yaml file. however, mods records pulled in by catmandu from the oai-pmh mods/xml feed (using the raw handler) have encoding issues that need to be addressed before mapping. reviewing the mods/json output generated, note that some xml encoding remains in the json _metadata field. this can be repaired by applying first the fix xml_simple(_metadata), which changes the xml in that field to simplified xml (as if applying the raw handler then rerunning the struct handler on the heavily nested portions). the user can then proceed with reviewing and mapping/fix routines for the mods/json or mods/yaml in ways detailed above. the following series of commands takes the mods/json pulled in the previous section, converts the mods/json to json again while applying the xml_simple fix, then creates a yaml output in the cli: $ catmandu convert json \ --fix "xml_simple(_metadata)" \ < kefauver.oai.mods.json \ > kefauver_updated.oai.mods.json $ catmandu convert json to yaml < kefauver_updated.oai.mods.json catmandu can work directly with various types of marc data for transformation and review. the following command takes a set of binary marc records and converts these to yaml, as done above with the dc/json files: $ catmandu convert marc to yaml < file_0.mrc figure 3. screenshot of in-line targeting marc as with the dc/json sets, simple, in-line fix language can be used to target specific fields in a marc record, or a saved fix file can be created to map marc fields to another field name, then apply normalization work. the command below takes each identifier from the marc records and prints it as yaml in the cli: $ catmandu convert --fix 'retain(_id)' marc to yaml < file_0.mrc the next command uses a fix routine (shown below) to generate a report used at utk to display controlled access points and possible identifier (or $0 subfield) usage in marc records as a csv output. this fix file also shows more of the capabilities for metadata normalization using fix routines. $ catmandu convert marc --fix marc2csv.fix to yaml < file_0.mrc figure 4. metadata normalization using fix --marc2csv.fix --marc_map("001", recordidentifier) marc_map("020a",isbn.$append) join_field(isbn," | ") split_field(isbn, " | ") marc_map("245ab",title, join:" ") # subject identifiers? if marc_match('6**0', "\w+") set_field("subjectrecon", "exists") else set_field("subjectrecon", "does not exist") end # name identifiers? if marc_match('1**0','\w+') set_field("namerecon", "exists") else set_field("namerecon", "does not exist") end # primary name marc_map('100abcdegq024','primaryname.$append', join:" ") sort_field('primaryname',-uniq=>1) # corporate name marc_map('110abcdegq024','corpname.$append', join:" ") sort_field('corpname',-uniq=>1) # additional names marc_map('700abcdegq024','addlname.$append', join:" ") sort_field('addlname',-uniq=>1) # additional corporate names marc_map('710abcdegq024','addlcorpname.$append', join:" ") sort_field('addlcorpname',-uniq=>1) # geographic subjects marc_map('651*','subjectgeographic.$append', join:"--") replace_all('subjectgeographic.*','\.$','') sort_field('subjectgeographic',-uniq=>1) # name subjects marc_map('600abcdefuvxyz02','subjectname.$append', join:" ") marc_map('610abcdefuvxyz02', 'subjectname.$append', join:".") sort_field('subjectname',-uniq=>1) # topical subjects marc_map('650abcdefuvxyz02','subjecttopical.$append', join:"--") replace_all('subjecttopical.*','\.$','') sort_field('subjecttopical',-uniq=>1) # genre terms marc_map('655abcdefuvxyz02','genre.$append', join:" ") replace_all('genre.*','\.$','') sort_field('genre',-uniq=>1) remove_field('record') mapping and review in lodrefine at this point, depending on the size and needs of a metadata set, it will be loaded into lodrefine for more involved remediation by the cataloging and metadata staff. when working with lodrefine, one can import data as csv, excel, xml including rdf/xml, json and google sheets. some imported data formats (especially xml and json containing nested elements) require further massaging once in lodrefine to make working with the data more efficient. at core, lodrefine is creating a tabular view of the original data. this change brings with it all the inherent issues involved with going from a hierarchical, graph or other model to a tabular model (and eventually back to the desired data model and format), though catmandu helps with pre-transformation to lodrefine-friendlier data formats. once the lodrefine project is created, the user will edit the columns to fit the target metadata schema’s fields. all of the column functions are accessed via the arrow at the top of each column in lodrefine. to reorganize and delete all the columns in an openrefine project at once, click on the arrow on the first/all column, then choose re-order/remove columns. this opens a box for reviewing all the columns in a project, removing those columns selected, and re-ordering those the user wishes to keep. the number of options one can apply in batch to all or a group of columns is fairly limited. working with each column separately offers more options for renaming, removing, splitting, adding additional columns or performing some reconciliation work. before mapping the fields represented by columns to a new field/column name, the catalogers browse more closely a field’s values through faceting. by choosing a column, clicking on the top arrow, then selecting facet > text facet, a facet box will appear in the left-hand part of the lodrefine interface. that box facets and shows all of the text values that appear in the cells of that chosen column. figure 5. screenshot of the text facet box for a selected column in lodrefine the text facets show just how a field was used in the original dataset, highlight typos or errors, perform some field-specific normalization work, and review to group certain values in subsets of records. for that last option, by clicking on a facet, the lodrefine project screen updates to show only the records containing that value in the selected column. normalization in lodrefine, there are many metadata normalization capabilities available by using faceting and google refine expression language (grel). using the facet > text facet function on a column, the lodrefine column facet values box allows a user to not just review, but also edit specific facet values. the edits will apply to all cells in that column that contained that facet value. figure 6. screenshot of editing a text facet in openrefine additionally, the cluster button in the top right corner of a text facet box provides a way to pull out groupings of similar data values for that column. using the cluster and edit column interface in lodrefine, the user can decide to cluster facet values using different algorithms, as well as to merge the clustered values. figure 7. screenshot of lodrefine’s cluster & edit column interface for a specific column, the user can also facet values via numerical value, text length, or other options. there is additionally the ability to create a custom text facet, which allows the user to facet a column’s values according to a grel expression. figure 8. screenshot of creating a grel expression for a custom text facet in lodrefine figure 9. screenshot of a custom text facet applied to an lodrefine project grel, which somewhat mirrors javascript in form and function, is one of the better documented areas of openrefine.[34] with grel, the user can transform data values using simple conditional statements and regular expression matching, apply values from other columns, change encodings, and many other data normalization functionalities. the user can also run custom grel operations on the cells within a chosen column by selecting edit cells > transform…. this will open a custom text transform box where grel can be entered, previewed, and applied against a column. along with the openrefine site and the openrefine github wiki, the recent book using openrefine by ruben verborgh and max de wilde is a recommended introduction and explanation of the core functions of openrefine, including working with grel.[35] grel, as an intermediary between working in an application and working with the cli, has proven to be very accessible for cataloging staff previously uncomfortable with the idea of scripting. figure 10. screenshot of the custom text transform interface in lodrefine for working with datasets that are too complicated or too big currently for lodrefine, like marc catalog data dumps, normalization is performed using just catmandu and fix routines built off the examples given in the review and mapping section. reconciliation in lodrefine in lodrefine, data reconciliation means normalizing and enhancing local metadata by comparing it with external datasets exposed either through a restful api, rdf documents, or sparql endpoints. one can query external datasets within any basic openrefine interface by using the ‘add column by fetching url’ function. this uses the existing data to create a http call to the external data service, then saves the full response in a new column. a more efficient and user-friendly reconciliation method is available by using the standard openrefine reconciliation service api structure, which builds a http-based restful api handling json data between openrefine and external data sources.[36] there are some existing, community-sourced reconciliation service templates and examples constructed primarily in python and php, as well as shared reconciliation services for the community to use.[37][38][39] many library-specific reconciliation services built off of the openrefine standard reconciliation service api have been created and maintained. they include: a fast reconciliation service by ted lawless [40], a viaf reconciliation service by roderic d. m. page [41], another viaf reconciliation service by jeff chiu [42], which is also hosted [43], and a service integrating vivo instances and openrefine [44]. documentation on the openrefine wiki, although in need of updates, provides more information about these standard reconciliation services, as does a recent presentation by this article’s author.[36][45] to reconcile a column against an existing reconciliation service in lodrefine, the user goes to the top arrow of a column, then selects reconcile > start reconciling… figure 11. screenshot of starting reconciling on a column in openrefine next, the standard reconciliation service column box shows reconciliation services currently set up for the instance of lodrefine. to add a new service, click on add standard service… in the bottom left corner. in the box that appears, the user then adds the url for the service’s endpoint and clicks on add service. figure 12. screenshot of the add standard reconciliation service box in openrefine after adding a service and loading the service metadata, the new reconciliation service should appear in the left-hand options in lodrefine. depending on the metadata and configurations for that particular service, different entity types to search against in the external data source may appear. there will be at least one type for reconciliation, along with a few other options like auto-matching high confidence reconciliation candidates or using data in other columns to better inform the reconciliation process. after reviewing the settings, the user then clicks on start reconciling to reconcile that column against the external data source. figure 13. screenshot of reconciliation faceting once the service is done, the lodrefine interface left-hand area displays facet boxes for reconciliation match levels and options. the user can select or deselect facets to quickly review levels of matches for the cells in that column. within each cell, the user chooses the best match from the top choices. while reviewing the matches for each cell, one can click on the possible reconciliation candidates to then open the relevant external data web page for that term, as well as decide to apply a reconciliation match to just to that one cell or to all cells with the same value. figure 14. screenshot of reconciliation service matches and facets in lodrefine the deri rdf extension offers the ability to reconcile data in lodrefine with a rdf document or a sparql endpoint. using either lodrefine or an installation of openrefine with the deri rdf extension included, the user then has access to the rdf extension via a button in the top right corner of the interface. figure 15. screenshot of the lodrefine extensions area, with the deri rdf extension in use using the rdf extension menu, the user clicks on add reconciliation service > based on sparql endpoint… or based on rdf file…. this will open a dialog box for working with the relevant data source chosen, and requires the user to supply information about the sparql endpoint or the rdf file. setting up a sparql endpoint requires a service name, the url for the sparql endpoint, optionally a graph url for the vocabulary exposed via the sparql endpoint, a sparql server type, then the label properties that lodrefine will use to find the possible data matches between the local data and the external rdf data. a user can select from some common rdf label properties included by default, or add multiple other properties (such as http://www.loc.gov/mads/rdf/v1#authoritativelabel for working with some of the library of congress lod authorities using mads/rdf). adding multiple properties for matching can help improve accuracy (since the user can catch both preferred and alternative labels), but may also slow down performance. figure 16. screenshot of the sparql-based reconciliation service interface in lodrefine the rdf reconciliation service requires also a name, along with either a url to the rdf document or a document upload. the user then selects the file format and, as with the sparql service, adds the properties lodrefine should search to retrieve the labels to be matched against the local data (with the same option to add more). figure 17. screenshot of the rdf document-based reconciliation service interface in lodrefine the deri rdf site, although no longer actively updated, has documentation on using the sparql reconciliation options for working with non-library-specific data sources.[46] the free your metadata team has also written a good overview for using the deri rdf reconciliation service options with the sparql endpoint for the library of congress subject headings on a sparql server created by the site’s authors.[47] they expand this work in their book, linked data for libraries, archives and museums.[48] for using the rdf document reconciliation option, a walk-through by this article’s author on reconciling against the thesaurus of graphic material available as a rdf dataset downloaded from the library of congress linked data service is available.[49] the utk workflow described here mostly uses the standard reconciliation service options to work with geonames, fast, and viaf. rdf data reconciliation is used for reconciling against local ontologies (such as utk form terms and the database of the smokies vocabulary, both stored as simple skos rdf documents) and smaller lod sets (such as the library of congress thesaurus of graphic materials). the sparql endpoint reconciliation option is used rarely, due to some limitations of the deri rdf extension in working with current sparql endpoints of interest. as an example, there is currently no way to use the deri rdf extension sparql reconciliation service with the getty vocabularies sparql endpointan issue discussed in the getty lod documentation and in the openrefine google group.[50][51] reconciliation in catmandu in catmandu, there are some quickly expanding reconciliation options. the catmandu getjson module and lookup in the fix language offer two simple but effective methods. as an example of the latter, the following command finds the dcmi type value in the oai-pmh dc/json metadata set pulled above, looks it up in the dctomodstype.csv dictionary given below, then replaces the dcmi type value with the matched mods type value. if a match isn’t found, it replaces the value with a default value of ‘other’. $catmandu convert json \ --fix "lookup('type','dctomodstype.csv',-default=>'other')" \ < egy.oai.dc.json --dctomodstype.csv --key,value collection,mixed material dataset,"software, multimedia" event,other image,still image interactive resource,"software, multimedia" service,other software,"software, multimedia" sound,sound recording text,text catmandu getjson takes values in a local store, uses them to create a url, and gets the json response from the web service to which the url query points. this works similarly to the openrefine option of ‘add a column by fetching urls…’, but is much faster. catmandu developers are also exploring ways to create and support richer reconciliation services. there is a github issue currently open discussing this, as well as one method and script shared by a core developer of catmandu for reconciling values in local datasets with linked data fragments servers and/or sparql endpoints.[52][53] linked data fragments (ldf) is a project from ghent university and iminds (formerly the institute of broadband technology or ibbt). the project wants to make lod much more scalable and support the creation of lod endpoints that are as lightweight as possible to make them more effective and sustainable.[54] as part of their work, they have created open-source ldf servers and query client applications in a variety of languages/frameworks, enabling people to expand and better secure access to rdf data. at present, a local instance of the ldf open source server application built with node.js is set up on the utk metadata librarian’s workstation with hdt (header, dictionary, triples) files, a compact form of rdf data that still allows querying, generated from the library of congress name authority file, the library of congress subject headings and the getty art and architecture thesaurus.[55][56][57] this ldf server supports reconciliation using catmandu for vocabularies that pose reconciliation problems in lodrefine (due to size and/or sparql endpoint compatibility issues). a developer and primary committer to the catmandu project wrote a sample perl script for reconciling marc data with viaf stored in an example ldf server.[58] a script was built out by a non-developer at utk to take the marc data pulled in the previous sections of this article and query the local ldf endpoint, matching different vocabularies to different fields.[59] when there is an exact match, the matched external value uri is appended to the marc field in subfield $0, and the updated marc (as binary usmarc by default) created. adding repeating $2 fields for indicating the uri source, if it points to an authority or a real world object, or is an actionable uri is being explored, following work on this topic.[60] creating the updated marc records works because of how the catmandu iterator (used for working through larger datasets primarily, but here used to loop through one marc record at a time) makes each marc record into a set of perl hashes, with the record then becoming an array of arrays. that allows the script to use the perl push command to add the uri field to the end of a marc field containing the reconciled value. $ perl recon.pl almajuly2015.mrc $ 53,204 records processed $ catmandu convert marc \ --type usmarc \ to marc \ --type marcmaker < marc.mrc figure 18. screenshot of recon.pl marc output. notice headings with more than one match or characters that need to be escaped for urls don’t get reconciled currently. transform and export from lodrefine once normalized, reconciled, and enhanced, the data needs to get into the form required for ingest into various platforms. for datasets in lodrefine, clicking on the export button in the top right corner gives the available format and encoding options for data export. figure 19. screenshot of the lodrefine export menu before discussing rdf options, there are other core export options worth reviewing. first, the triple loader export option might seem of interest for rdf work, but this is no longer operational (as it was originally part of the now-obsolete freebase extension). the templating… option allows the user to create a json or xml data export template, with limitations occurring according to how rows and records are modeled in the openrefine project. however, it is an extremely useful option for those users who take the time to build a data-specific export template – see, for example, work done on exporting from openrefine to mods/xml.[61] figure 20. openrefine templating export interface at utk, non-marc metadata is currently exported as simple xml via the templating option to then be transformed to mods/xml using an existing xslt stylesheet. the mods/xml records are named following the element that matches the names of the digital asset files names (highlighted during export review phase), moved to the same environment as asset files, and then ingested in islandora. catmandu is being explored here for easier export to the fedora instance behind the utk digital collections islandora instance, but updated documentation and procedures for how and where to store the digital assets the metadata describes need to be completed first. [62] for rdf experimentation, lodrefine also has the ability to export to rdf/xml or turtle serializations. for either export option, the user must first set up a rdf skeleton, accessible through the deri rdf extension button in the top right corner of the openrefine interface. figure 21. screenshot of the edit rdf skeleton option in the openrefine deri rdf extension menu clicking on edit rdf skeleton…, brings up the rdf schema alignment box. there, the user can configure just how each column will be transformed into a node in the exported rdf document. the user can add prefixes to use with additional namespaces, which are then queried by lodrefine to offer type-ahead for properties and services. in the skeleton interface, one can also decide how the triples are generated from the project data and see a preview of the to-be-exported rdf document, among other functions. the deri rdf website has a short but useful introduction to this rdf export work.[63] figure 22. screenshot of openrefine deri rdf schema alignment interface as utk is migrating a lot of descriptive metadata from a number of schema and practices to mods/xml, there is interest in how mods/xml will be represented as mods/rdf (or mapped to other, existing properties). while work on this continues [64][65], the images below show two areas of exploration: 1. a rdf export skeleton for mods/rdf (with blank nodes and namespace lookups in use) as currently defined by the library of congress mods/rdf version 1 standard; then 2. the mods/xml metadata as dcterms in rdf. this work has both given many internal stakeholders a better understanding of what the change to rdf will mean, as well as examples to work with in crafting community discussions. note: for the type-ahead features to work with the mods/rdf version 1 namespace, the user must click add prefix, click on the advanced button, then upload the vocabulary terms from a file stored locally (for mods/rdf version 1, this owl document downloaded from the library of congress).[66] figure 23. lodrefine rdf skeleton with namespace lookups, using modsrdf v.1 figure 24. lodrefine rdf skeleton using dcterms figure 25. lodrefine rdf skeleton node options lodrefine will then export the dataset following the constructed rdf skeleton as one file with multiple statements/graphs in whatever serialization was chosen. transform and export using catmandu as mentioned previously, catmandu can convert data to various rdf serializations while working with aref (or another rdf encoding form) in the catmandu internal processing.[67] taking the marc data worked with so far and converting to rdf requires a fix routine. the following commands and fix routine will take marc data having been reconciled using the script above, make the subject of all the triples generated for a record the record identifier (or the _id value as created by catmandu working with marc), then map fields to objects of predicates from a variety of namespaces, very loosely guided by recent bibframe discussions. as one example, the following fix routine creates the start of a bibframe work description from the marc file shown: $ catmandu convert marc \ --type usmarc \ --fix marctordf.fix \ to rdf \ --type ntriples < marc.mrc > marc_dcterms.rdf --marc.mrc snippet (displayed as marc --type marcmaker): --=ldr 01684ajm 2200397u r4500 =001 994822350102311 =005 20150603214529.0 =007 sd\bsmenn\\\\\ =008 840611s1965xx\\\\\\\nn\\\\\\\\\\0\\ger\d =028 00$awer 60020$bwergo =035 \\$a(tu)000482235utk01 =035 9\$a508073 =035 \\$a(ocolc)10831384 =040 \\$atkn$ctkn =048 \\$aoa =048 \\$aca =049 \\$atknm =090 \\$am2021.p4$ba8 =100 1\$apenderecki, krzysztof,$d1933-$0(dlc)no98008167 =240 10$afluorescences$0(dlc)nr91007103 =245 10$afluorescences. stabat mater for drei a-cappella chore$h[sound recording] =260 \\$bwergo$cwer 60020.$c1965. =300 \\$aon side 2 of 1 disc.$b33 1/3 rpm. stereo.$c12 in. =440 \0$astudio-reihe neuer musik. =501 \\$awith his psalmy dawida. =500 \\$atape copy. =511 0\$awarschauer philharmonisches orchester (in the first work); chor der warschauer philharmonie; andrzej markowski, conductor. =650 \0$aorchestral music. =650 \0$achoruses, sacred (mixed voices), unaccompanied.$0(dlc)sh85095326 =650 \0$astabat mater dolorosa (music)$0(dlc)sh85127183 =700 12$apenderecki, krzysztof,$d1933-$tstabat mater.$0(dlc)no97072342 =700 1\$amarkowski, andrzej,$d1924-$4prf$(dlc)n83006447 =710 2\$afilharmonia narodowa (warsaw, poland)$4prf$0(dlc)n83194275 --marctordf.fix: --# based off of https://github.com/ld4l/ontology # @bfld : <http://bibframe.ld4l.org/ontology/> # @rdau : <http://www.rdaregistry.info/elements/u/#> prepend('_id','http://library.utk.edu/sample-bf/') append('_id','/work/') add_field('a','bfld:work') if all_match('ldr/6','a') add_field('a','http://bibframe.ld4l.org/ontology/text') end if all_match('ldr/6','c') add_field('a','http://bibframe.ld4l.org/ontology/notatedmusic') end if all_match('ldr/6','e') add_field('a','http://bibframe.ld4l.org/ontology/cartography') end if all_match('ldr/6','[ij]') add_field('a','http://bibframe.ld4l.org/ontology/audio') end if all_match('ldr/6','m') add_field('a','http://bibframe.ld4l.org/ontology/multimedia') end if all_match('ldr/6','r') add_field('a','http://bibframe.ld4l.org/ontology/threedimensionalobject') end marc_map('008_/7-10','dc_date') replace_all('dc_date','\d','0') marc_map('008_/35-37','dct_language') if all_match('dct_language','\w+') set_field('dct_language','und') end marc_map('050ab','bfld:classifiedas') marc_map('6500','dct_subject.$append') replace_all('dct_subject.*','\(dlc\)','http://id.loc.gov/authorities/subjects/') unless exists('dct_subject') marc_map('650','dct_subject.$append',-join=> '--') end marc_map('245ab','bfld:haspreferredtitle', -join=>" ") replace_all('bfld:haspreferredtitle', ' /$','') marc_map('1000','rdau:p60447') replace_all('rdau:p60447','\(dlc\)','http://id.loc.gov/authorities/names/') unless exists('rdau:p60447') marc_map('100','rdau:p60447') end marc_map(700ad,'rdau:p60398.$append',-join=>" ") marc_map('028','bfld:hasinstance.bfld:identifiedby') add_field('bfld:hasinstance.a','bfld:musicpublishernumber') remove_field('record') --marc_dcterms.rdf snippet: --<http://library.utk.edu/sample-bf/994822350102311/work/> <http://purl.org/dc/elements/1.1/date> "1965" . <http://library.utk.edu/sample-bf/994822350102311/work/> <bfld:hasinstance> _:b1 . _:b1 <bfld:identifiedby> "wer 60020wergo" . _:b1 <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <bfld:musicpublishernumber> . <http://library.utk.edu/sample-bf/994822350102311/work/> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <bfld:work> . <http://library.utk.edu/sample-bf/994822350102311/work/> <http://purl.org/dc/terms/language> "ger" . <http://library.utk.edu/sample-bf/994822350102311/work/> <http://purl.org/dc/terms/subject> <http://id.loc.gov/authorities/subjects/sh85095326> . <http://library.utk.edu/sample-bf/994822350102311/work/> <http://purl.org/dc/terms/subject> <http://id.loc.gov/authorities/subjects/sh85127183> . <http://library.utk.edu/sample-bf/994822350102311/work/> <rdau:p60398> "penderecki, krzysztof, 1933-" . <http://library.utk.edu/sample-bf/994822350102311/work/> <rdau:p60398> "markowski, andrzej, 1924-" . <http://library.utk.edu/sample-bf/994822350102311/work/> <bfld:haspreferredtitle> "fluorescences. stabat mater for drei a-cappella chore" . <http://library.utk.edu/sample-bf/994822350102311/work/> <rdau:p60447> <http://id.loc.gov/authorities/names/no98008167> . a similar process can be done for non-marc records being processed by catmandu, with the updated fix routine for that data. as catmandu::rdf is built off of rdf::ns (among other helpful perl modules for rdf data), many of the more popular namespace prefixes are already defined— so above, dct_ becomes <http://purl.org/dc/terms/> . other namespaces and their prefixes, such as rdau: for the rda (resource, description and access) properties need to be defined either in a perl script using rdf::ns or in the data store this rdf data is going to.[68] while catmandu can convert between rdf serializations (using the -type flag and with rdf/xml as the default), the core developers of catmandu recommend using other tools for this work. catmandu is better suited for transforming non-rdf data to rdf. conclusions and looking ahead there is a lot of utility in both catmandu and lodrefine for data work, and these tools complement each other while offering a lot of functionalities yet to be explored for the utk metadata workflows. as one example, catmandu’s ability to query connected datastores and pull those values into metadata during an etl process [69] is of particular interest for working with the datastores behind platforms such as vivo and skosmos and using those datasets to both normalize and enhance metadata across platforms.[70][71] the growth in use of domain-specific transformation languages, intended to make metadata transformation more accessible to non-developers and as seen with catmandu’s fix language, is also an area of continued exploration with tools such as metafacture and the digital public library of america’s krikri.[72][73][74] another area of experimentation is how to continue using existing data tools such as marcedit and new tools such as d:swarm (a middleware tool for data review and visualization) in a wider and more consistent metadata munging workflow.[75][76] despite the noted areas requiring further exploration and the pitfalls discussed above, the biggest success of this workflow and the tools used so far is the better integration of the staff members involved with various parts of the metadata ecosystem. instead of keeping marc catalogers, content specialists, digital projects managers, digitization staff, and developers functionally isolated, the above workflow is a step towards more interoperability, consistency, and better communication of metadata munging practices, whether marc or non-marc. as that occurs, the metadata being migrated will become more consistent, staff will have a better understanding of rdf (or metadata more generally) discussions, and utk’s collections discoverability will continue to improve. acknowledgements the author would like to thank the editors and readers for their comments and feedback. special thanks to patrick hochstenbach (digital architect at the ghent university library) and nick ruest (digital assets librarian at york university) for their review and suggestions that greatly improved this article. notes and references [1] more about resource description framework (rdf): http://www.w3.org/rdf/ [2] oclc international linked data surveys show the growth of libraries, archives and musuems working with rdf: http://www.oclc.org/research/themes/data-science/linkeddata.html#linked-data-survey [3] “like families, tidy datasets are all alike but every messy dataset is messy in its own way.” –hadley wickham, “tidy data”, journal of statistical software. http://vita.had.co.nz/papers/tidy-data.pdf [4] a short history of openrefine: http://openrefine.org/2013/10/12/openrefine-history.html [5] openrefine github repository: https://github.com/openrefine/openrefine [6] openrefine website: http://openrefine.org/ [7] openrefine on twitter: http://www.twitter.com/openrefine [8] quote taken from the openrefine github repository readme file: https://github.com/openrefine/openrefine/blob/master/readme.md [9] deri rdf extension for openrefine github repository: https://github.com/fadmaa/grefine-rdf-extension/downloads [10] deri rdf extension for openrefine website: http://refine.deri.ie/ [11] lodrefine github repository: https://github.com/sparkica/lodrefine [12] openrefine github wiki installation instructions: https://github.com/openrefine/openrefine/wiki/installation-instructions [13] openrefine github wiki extension installation instructions: https://github.com/openrefine/openrefine/wiki/installing-extensions [14] openrefine github repository wiki users’ documentation: https://github.com/openrefine/openrefine/wiki/documentation-for-users [15] introduction to openrefine written by owen stephens: http://www.meanboyfriend.com/overdue_ideas/2014/11/working-with-data-using-openrefine/ [16] to learn more about or get involved in supporting catmandu, visit the librecat github organization site (https://github.com/librecat), the librecat website (http://librecat.org/), or get in touch with one of the primary contacts listed on the librecat about page (http://librecat.org/about.html). [17] catmandu::store::mongodb: https://github.com/librecat/catmandu-store-mongodb [18] catmandu::store::elasticsearch: https://github.com/librecat/catmandu-store-elasticsearch [19] catmandu::oai: https://github.com/librecat/catmandu-oai [20] catmandu::sru: https://github.com/librecat/catmandu-sru [21] catmandu::dspace: https://github.com/librecat/catmandu-dspace [22] catmandu::fedoracommons: https://github.com/librecat/catmandu-fedoracommons [23] the catmandu handbook: http://librecat.org/catmandu/ [24] catmandu installation instructions: http://librecat.org/catmandu/#installation [25] catmandu core cpan module: https://metacpan.org/pod/catmandu [26] catmandu virtual machine plus instructions for use: https://librecatproject.wordpress.com/get-catmandu/ [27] catmandu docker image: https://registry.hub.docker.com/u/librecat/catmandu/ [28] local configuration options for catmandu: http://librecat.org/catmandu/#configuration [29] the most recent version of the catmandu command line interface cheat sheet: http://librecat.org/catmandu/#command-line-client-cheat-sheet [30] catmandu::fix::parser: https://metacpan.org/pod/catmandu::fix::parser [31] catmandu fixes cheat sheet: https://github.com/librecat/catmandu/wiki/fixes-cheat-sheet [32] catmandu::importer::oai: https://metacpan.org/pod/catmandu::importer::oai [33] more information on the catmandu oai parsers available: https://github.com/librecat/catmandu-oai/tree/master/lib/catmandu/importer/oai/parser – though as of the first draft of this article, support for mods has been added by the catmandu core development team: https://github.com/librecat/catmandu-oai/commit/7c94797504821a6a1fab8e4ab3e16dd0118c4480 [34] google refine expression language (grel) functions: https://github.com/openrefine/openrefine/wiki/grel-functions [35] using openrefine by ruben verborgh and max de wilde (2013, packt publishing) [36] documentation on the openrefine standard reconciliation service api (note that this documentation needs to be updated in parts): https://github.com/openrefine/openrefine/wiki/reconciliation-service-api [37] openrefine reconciliation service api examples: https://github.com/openrefine/openrefine/wiki/reconciliation-service-api#examples [38] sample openrefine reconciliation service api built in python/flask: https://github.com/mikejs/reconcile-demo [39] openrefine reconciliation service for viaf built with php: https://github.com/rdmpage/phyloinformatics/blob/master/services/reconciliation_viaf.php [40] ted lawless’ fast reconciliation service for openrefine: https://github.com/lawlesst/fast-reconcile [41] blog post by roderick d. m. page on his viaf reconciliation service for openrefine: http://iphylo.blogspot.com/2013/04/reconciling-author-names-using-open.html [42] another viaf reconciliation service for openrefine by jeff chiu: https://github.com/codeforkjeff/refine_viaf [43] the hosted version/endpoint for the viaf reconciliation service referenced above: http://refine.codefork.com/ [44] a vivo reconciliation service for openrefine: https://wiki.duraspace.org/display/vivo/extending+google+refine+for+vivo [45] a presentation by the author of this article on building openrefine reconciliation service apis: http://christinaharlow.com/openrefine-reconciliation-workshop-c4lmdc/ [46] deri rdf extension sparql reconciliation service documentation: http://refine.deri.ie/sparqlrecondocs [47] the free your metadata team’s explanation of reconciliation using lodrefine: http://freeyourmetadata.org/reconciliation/ [48] the above authors’ book expanding the lodrefine reconciliation work (among other topics): http://book.freeyourmetadata.org/ [49] walk-through of using the lodrefine rdf document reconciliation service: https://github.com/cmh2166/c4lmdcpres/blob/master/breakout_workflows/derirdfexample.md [50] getty lod documentation discussing limitations of working with lodrefine: http://vocab.getty.edu/queries#openrefine_reconciliation_service [51] openrefine google group post on trying to use lodrefine with getty sparql endpoint: https://groups.google.com/d/topic/openrefine/g7ljofkbymo/discussion [52] catmandu::rdf github repository issue on reconciliation services: https://github.com/librecat/catmandu-rdf/issues/23 [53] librecat project blogpost on using catmandu and linked data fragments (ldf) servers for reconciliation work with viaf: https://librecatproject.wordpress.com/2015/06/03/matching-authors-against-viaf-identities/ [54] the linked data fragments (ldf) website: http://linkeddatafragments.org/ [55] the ldf server.js (built with node.js) github repository: https://github.com/linkeddatafragments/server.js [56] the rdf/hdt website: http://www.rdfhdt.org/ [57] researchers involved with building a hdt-cpp library and tools used for generating and working with hdt files at utk: javier d. fernández ,miguel a. martínez-prieto, claudio gutiérrez, axel polleres and mario arias. “binary rdf representation for publication and exchange (hdt)” (2013), web semantics: science, services and agents on the world wide web. http://www.websemanticsjournal.org/index.php/ps/article/view/328. miguel a. martínez-prieto, mario arias and javier d. fernández. “exchange and consumption of huge rdf data” (2012), the semantic web: research and applications. pages 437-452. [58] patrick hochstenbach’s perl script for reconciling marc data with viaf in a development ldf server: https://gist.github.com/phochste/c87c81c79d8b8a6a2179 [59] this author’s draft perl script for reconciling marc data with ldf server containing lcnaf, lcsh, and getty aat vocabularies: https://github.com/cmh2166/catmandu_recon [60] uris in marc: a call for best practices by steven folsom: https://docs.google.com/document/d/1fuhvf8bxh7hldy_xj7f_xn2rp2dj8o-ca9jhhghieug/edit [61] “converting spreadsheets into modsxml using open refine” by sara allain: https://www.utsc.utoronto.ca/digitalscholarship/content/blogs/converting-spreadsheets-modsxml-using-open-refine [62] the catmandu::fedoracommons module: http://search.cpan.org/dist/catmandu-fedoracommons/ [63] deri rdf extension site’s introduction to rdf export: http://refine.deri.ie/rdfexportdocs [64] modsrdf version 2: https://github.com/blunalucero/mods-rdf [65] hydra mods and rdf descriptive metadata subgroup: https://wiki.duraspace.org/display/hydra/mods+and+rdf+descriptive+metadata+subgroup [66] mods/rdf version 1 owl document: http://www.loc.gov/standards/mods/modsrdf/v1/modsrdf.owl [67] another rdf encoding form (aref) website: http://gbv.github.io/aref/aref.html [68] rdf::ns (namespaces) perl module: https://metacpan.org/pod/rdf::ns [69] see slides 59-60 of this rdf in catmandu presentation to learn more about the weave_by_id and weave_by_query fix functions in catmandu: http://librecat.org/assets/20131113_workshop_slides.pdf [70] about vivo: http://www.vivoweb.org/about [71] about skosmos: http://skosmos.org/ [72] about metafacture: https://github.com/culturegraph/metafacture-core/wiki [73] section 5.5 of this article by the dpla technology team mentions in passing their interest in domain-specific languages for metadata aggregation, mapping and enhancement: matienzo, m., rudersdorf, a.. the digital public library of america ingestion ecosystem: lessons learned after one year of large-scale collaborative metadata aggregation. international conference on dublin core and metadata applications, north america, oct. 2014. available at: http://dcpapers.dublincore.org/pubs/article/view/3700/1923 [74] about krikri: https://github.com/dpla/krikri – for seeing the influence of catmandu’s fix language on this work, check out specifically their work on application-specific mapping language functionalities: https://github.com/dpla/krikri/blob/develop/lib/krikri/map_crosswalk.rb https://github.com/dpla/krikri/blob/develop/lib/krikri/mapper.rb https://github.com/dpla/krikri/blob/develop/lib/krikri/mapping.rb https://github.com/dpla/krikri/blob/develop/lib/krikri/mapping_dsl.rb [75] current utk e-resources vendor marc records review using marcedit: https://wiki.lib.utk.edu/display/cat/marcedit+e-resource+batch+preparation [76] about d:swarm: http://www.dswarm.org/en/datenmanagement/dswarm-hilfe/was-ist-dswarm/ subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – getting real in the library: a case study at the university of florida mission editorial committee process and structure code4lib issue 39, 2018-02-05 getting real in the library: a case study at the university of florida in the fall of 2014, the university of florida (uf) marston science library, in partnership with uf it, opened a new computer lab for students to learn and develop mobile applications. the mobile application development environment (made@uf) features both software and circulating technology for students to use in an unstructured and minimally-staffed environment. as the technological landscape has shifted in the past few years, virtual and augmented reality have become more prominent and prevalent, signaled by companies like facebook, google, and microsoft making significant financial investments in these technologies. during this evolution, made@uf has migrated to focus more on virtual and augmented reality, and we will discuss the opportunities and challenges that hosting and managing such a space has provided to the science library and its staff. by samuel r. putnam and sara russell gonzalez introduction in order to be fully prepared to enter the workforce or graduate school, students need the opportunity to explore new technology and develop new proficiencies. through the development of makerspaces, technology workshops and lending collections, libraries are positioning themselves on academic campuses as collaborative learning environments where students can experience emerging technologies and gain valuable expertise. made@uf (mobile application development environment) is a digital makerspace for students at the university of florida to develop and test mobile and virtual reality and augmented reality (vr/ar) applications. with two locations, in marston science library and the infinity residence hall, it was originally designed to solely focus on mobile app development; however, in response to the emergence of vr/ar technology and the demand of our stakeholders, made@uf has pivoted to focus primarily on providing a space for vr/ar development. background made@uf was originally conceived in 2013 as a sandbox for mobile application development by anne allen in the university of florida’s (uf) academic technology unit. allen had funding for technology through the uf technology fees, which are assessed to each student, but lacked a space to install made@uf. in 2014, marston science library (msl) completed a full renovation of its ground floor including the repurposing of a large classroom. originally contemplated as a physical makerspace, plans for the room changed when allen approached the library with the proposal to use the space for made@uf instead. this new lab’s emphasis on emerging technology fit with the science library’s objective of encouraging collaboration in its predominantly engineering and science student population. the original design of the space was a mixture of 3 collaboration pods and 5 individual workstations each with a mac mini. the collaboration pods accommodate 5-7 users and the large-scale monitor is connected with a crestron unit that allows users to change inputs between the mac mini or mirroring over wifi their own laptop screens. figure 1. collaboration pods in made@uf (enlarge) figure 2. 2014 made@uf floor plan with potential 2018 changes (enlarge) the mac minis were selected because early focus groups of students, faculty and alumni indicated that most students would want to develop ios applications and an apple computer is required for ios development. software installed included eclipse adt, xcode, android studio, brackets, and seamonkey. we made the decision early in made@uf’s development not to include microsoft office or other productivity software in an attempt to preserve the sandbox nature of the space and discourage general homework activity; this was not viewed as unduly hindering student work as there are another 90 workstations on the same floor outside of made@uf with the full range of software available to students. to support ios development, we also joined the apple ios developer university program (https://developer.apple.com/programs/ios/university/). this free program, open to degree-granting higher education institutions, permits students to begin developing in ios without first paying the required $100 developer fee. we felt this was important because one of made@uf’s major objectives was to encourage students who are new to app development and to eliminate barriers such as the initial developer fee. however, enrolling in this program also required the library to mandate that users would first sign a user agreement stipulating that students would comply with apple’s terms and conditions as well as general use policies. the logistics of this agreement also meant that only students could use made@uf because there was not a way to restrict software usage to different types of patrons, such as faculty, staff, and community patrons. along with the software development resources, students could also check out boxes of tablets of varying sizes and manufacturers to test their apps throughout the development process. these boxes could be checked out at the service desk on the main floor for 4 hours and contained an ipad 2, ipad mini, android nexus tablet, and an ipod touch as a proxy for an iphone. one of the first challenges was how to facilitate students wiping the memory and resetting each of the devices before returning the device box. one of the individual workstations was set aside for this need, unfortunately removing a workstation from general student usage. detailed instructions for resetting the devices was provided next to the station and it was the responsibility of the student to reset all devices before returning the box to circulation. after made@uf in msl was opened, the library was approached by housing it to query if we would like to partner with them on opening a second made@uf in a new residence hall being constructed a few blocks off campus. this new hall targeted students focused on entrepreneurship and included a ground floor with resources that students might need to bring a new invention or company to fruition. these services included a fabrication lab, graphics design studio, and start-up incubator; made@uf was seen as an ideal complement to support students developing digital projects. using the same uf technology fee support, the new made@uf at infinity was opened in 2015 with similar setup and technology as msl’s location. this was a deliberate attempt to keep the 2 spaces uniform so that students could move between either location easily. although infinity is approximately a 10-minute walk off campus, it does have the advantage of being open 24 hours a day, 7 days a week. the hall’s lobby staff monitor usage in the space, checkout technology, and disseminate room keys to the students. early stages activity early on was aimed at raising awareness of the space and encouraging students to begin app development. uf information technology (ufit) agreed to provide an experienced undergraduate to staff msl’s made@uf for 3 hours a day, monday through friday. the student was familiar with ios development and also could triage problems with the workstations. we quickly discovered that students rarely sought assistance; although, it was unclear if it was because they were already comfortable with the software or if they were doing work other than app development. after 2 semesters, ufit decided to discontinue the daily staffing and provide help as needed. marketing of made@uf was done mainly through events where we were allowed to set up tables and discuss made@uf with participants as well as presentations across campus. these events included new student orientations, the campus-wide it showcase, and department presentations to faculty. in addition to a handout explaining the basic concept of made@uf, students received a google cardboard kit, a do-it-yourself cardboard viewer that couple with a smartphone to create a vr headset,along with instructions on how to assemble it and links to available apps to try. also, made@uf obtained an oculus rift dk2 that was used for hands-on demonstrations; this invariably led to long lines of students excited to try out the new technology. even though students were impressed by the rift, they often seemed reluctant to use made@uf due to a lack of programming experience. part of the preliminary development of the made@uf concept was recognizing the need for basic-level tutorials and instruction to encourage students to explore building apps. part of the initial funds were earmarked to a computer science professor who agreed to create an online course leading beginners through development using phonegap, a cross-platform app builder. unfortunately, the professor did not follow through on finishing the online course so workshop activity changed to locating in-person instructors who would teach hands-on classes. it proved challenging to find speakers and instructors capable of leading workshops without a budget to compensate presenters and we did not have library staff with enough programming expertise to teach. we did invite a local app development company to speak about their work and later held a 3-part series designed to start students developing android apps. infinity’s made@uf had the advantage of an in-house coding school that held classes adjacent to their space. otherwise, students were encouraged to take advantage of uf’s lynda.com subscription that contains numerous programming tutorials covering both ios and android development. all workshops that were held in made@uf were heavily attended, highlighting that the demand existed. transition shortly after made@uf opened, we decided to expand app development to include wearable technology. smartwatches were discussed, such as the android-based pebble watch, but staff settled on introducing google glass and epson moverio, a headset aimed at business. the glasses were challenging for it to support because they required network access, necessitating either tethering to a phone or directly connecting to the campus wireless network. since the glasses could not connect to the campus network due to its authentication protocols, a separate hidden wireless network was installed in the msl location exclusively for using the glasses. these proved to generate the most excitement for made@uf at the beginning because most people had never seen or tried on a pair and were highly curious about the interface. however, after examining the circulation statistics for the glasses and moverios, we realized that made@uf served as a showcase to try out new equipment rather than its original purpose as a development lab. the google glasses led directly to purchasing the oculus rift development kit 2, a virtual reality headset, as well as the microsoft hololens, a mixed reality headset. issues with the oculus rift development kit 2 began when oculus dropped support of the rifts on macs. considering made@uf was an all-mac lab at the time, this effectively ended our vr circulation program. however, we utilized another uf technology fee to purchase windows computers in conjunction with oculus release of their new vr headset. oculus was chosen over the htc vive partially due to the rift’s minimal space requirements (htc vive requires a 6.5 feet by 5 feet of empty space with a minimum 16 feet between base stations). additionally, these vr-capable windows computers present a barrier for adoption because several libraries can afford the cost of an oculus rift or htc vive but not the computer that is required to run vr games and experiences. while most newer machines will meet the minimum requirements for cpu, ram, and ports, few off-the-shelf builds will contain the requisite graphics card to support vr. microsoft hololens provided another list of issues. first, the high cost of the hololens, starting at $3000, made staff wary of loaning the item and thus limited us to purchasing 1 for circulation. next, the development version does not work well with lending due to need for users to link their microsoft account to use. this requirement leaves circulation staff with the burden of ensuring patrons are logged off of the hololens, otherwise risking exposure of a patron’s microsoft account to a stranger. as an alternative, microsoft offers a hololens with enterprise options geared toward multiple users for $5000. the transition from mobile app development to vr/ar technology also reflected the increased investment in vr/ar by some of the largest technology companies in the world. in the past four years, facebook purchased the virtual reality company oculus, apple released the arkit for developing augmented reality applications on ios devices, google developed google cardboard as an affordable vr option, and sony released playstation vr to accompany their gaming platform, just to name a few notable examples. this increase of vr/ar development was mirrored by a rise in student interest and faculty research in using and creating new vr/ar content at uf. around this time, we began conducting informal whiteboard polls at the library entrance. the whiteboard polls would pose one question and invite the responses of library patrons. when the we asked questions related to technology in the libraries on the whiteboard polls, we received robust responses, typically between 75-100 in a 24-hour period. though informal, these polls helped us receive up-to-date opinions related to technology quickly from our patrons. recent developments the transition from mobile app development to vr/ar was accompanied by new partners in made@uf. gator vr, a student club on campus dedicated to creating vr/ar projects and holding vr/ar events, began conducting their general body meetings, officer meetings, and special events in the made@uf space. made@uf’s partnership with gatorvr has brought 15 students into the space who are using the vr equipment on a weekly basis. this partnership creates familiarity with a group of students that otherwise may not have been involved. beginning in february 2017, the george a. smathers libraries was a part of a marketing project funded by our regional library cooperative, the northeast florida library information network. for the marketing project, the libraries agreed to focus on marketing made@uf. the project team worked to refresh the made@uf website. this included updating information, synthesizing information to eliminate unnecessary pages, adding google analytics to the web page, and creating a newsletter sign-up form. google analytics allows us to see that our website has had 468 users from september 1, 2017 through november 30, 2017 with 90% of the users being new users. we have also had 39 email subscribers since the beginning of fall 2017. these subscribers receive emails about new technology as well as opportunities to “beta-test” via checkout before it is available to all library patrons. subscribers also receive information about upcoming events in made@uf as well as other vr/ar events on campus and in the community. the project team also crafted press releases accompanying new technology in the space and advertised for the space. the work of the marketing project team laid a foundation for future growth as made@uf seeks to increase its impact on campus. beginning in fall 2017, benjamin lok, a professor in computer and information science and engineering, created a class named “vr for the social good.” the class consists of 50 students who will break into groups of five to create a vr/ar project over the course of the semester. the enrollment will grow to 100 in spring 2018 and potentially larger in fall 2018. the course does not require students to be computer science majors. rather, students from non-computer science disciplines are encouraged to enroll as the first month of the course is dedicated to learning unity, a game engine used to create vr/ar applications. lok’s course does not have the infrastructure to support vr/ar development on this scale, which is where the partnership with made@uf began. made@uf is equipped with collaborative spaces for multiple student groups to work with oculus rifts and unity software in order to complete their projects. we have also blocked off time during mornings from 8:00 a.m. to 10:00 a.m. on monday through friday for students enrolled in vr for social good to receive priority in made@uf. the class also is able to reserve stations with vr/ar technology through use of the libcal room booking system. these accommodations allow for students to complete group work without risking interference from casual computer users. due in part to these new partnerships, msl’s oculus rifts have been circulated 112 times during the fall 2017 semester, a vast improvement over 77 loans in the spring 2017 semester. in order to provide the most appropriate vr technology for developers on campus, we surveyed the students in the vr for social good course (see appendix 1 for survey on vr/ar technology in made@uf). we will be able to use this survey data to make a more informed decision when we apply for the annual uf technology fee. the timing of the yearly application allows librarians to survey students from the fall and spring semesters of the course; in this funding cycle, we will be able to survey approximately 150 student developers for their input. in the future, we will survey the student club gatorvr to gain their input. in fall 2017, made@uf hosted one-hour workshops within the space taught by recently hired librarians and graduate interns. five workshops were planned during the semester, three focusing on unity, one focusing on aurasma, an augmented reality app, and one on git and github, which was unfortunately cancelled due to hurricane irma. the four workshops brought in 70 participants, a mark representing 50% capacity on average for the events. these hands-on workshops allowed students to use these programs and create a vr/ar project of their own. gatorvr sent a representative to each event to allow participants interested in vr/ar projects the opportunity to connect with other interested students. made@uf also hosted two open hours during the fall 2017 semester. one open house, which was marketed as “coffee and vr”, aimed to bring in faculty and graduate students who identified vr/ar within their research interests. upon bringing these interested parties into made@uf, the goal was to foster discussion amongst researchers from various fields and introduce them to the resources available in made@uf and marston science library. we emailed information about the first “coffee and vr” event to 90 interested parties. we tracked the emails and discovered 49 recipients opened the emails, while 5 clicked on links inside the emails. during the event, 7 participants arrived to enjoy coffee and talk vr/ar; 1 of the participants came as a result of the email. although this may seem less the ideal, the 1 participant was from the school of construction management, a department we did not have a relationship with prior to the email blast. since the event, we have been in discussions with this researcher about partnership opportunities. the other open house was designed to reach undergraduate students and excite them about the potential of vr/ar. we included gatorvr in this open house and had them demo the final version of the vr application they created during spring 2017. the two-hour event drew 40 participants including 1 who received the email blast. this graduate student brought his class to the event, a group of about 12 students. also, we have been in discussions with this graduate student about future partnerships with his college, the college of the arts, with which we previously did not have a relationship. conclusion made@uf has transitioned from a mobile app development space to a vr/ar focused space over the span of about two years. the transition echoes the emergence of vr/ar into the public consciousness and reflects new partnerships created between librarians at marston science library and vr/ar stakeholders on campus at the university of florida. in the future, made@uf hopes to continue to identify potential partners through open houses and workshops on campus as well as supporting faculty interested in utilizing the space for relevant courses. as an area of growth, we are interested in identifying potential opportunities for working with faculty and graduate students conducting research in vr/ar and enhancing assessment to understand our impact on student learning. about the authors samuel r. putnam (srputnam@ufl.edu) is an engineering librarian at the university of florida where he is the mechanical and aerospace engineering liaison and director of the made@uf development lab. his current research interests include virtual and augmented reality in libraries as well as innovative instruction and information-seeking behavior of students. sara russell gonzalez (saragonz@ufl.edu) is a science librarian at the university of florida where she is the physical sciences and mathematics liaison and former director of the made@uf development lab. her current research interests include emerging technologies in libraries, modeling and visualization of data, and scientific literacy instruction. appendix 1: survey on vr/ar technology in made@uf see the made@uf student survey from spring 2017. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – exposing library services with angularjs mission editorial committee process and structure code4lib issue 26, 2014-10-21 exposing library services with angularjs this article provides an introduction to the javascript framework angularjs and specific angularjs modules for accessing library services. it shows how information such as search suggestions, additional links, and availability can be embedded in any website. the ease of reuse may encourage more libraries to expose their services via standard apis to allow usage in different contexts. by jakob voß and moritz horn introduction the demand to open up library systems through web services has been known for years (breeding 2009). in particular, service-oriented architecture (soa) promised to better allow a continuous evolution of library automation and to better connect with external systems. nevertheless current library systems are rarely built of loosely-coupled parts that could be used independently. this reliance on monolithic systems results in a lack of open apis to library services. library services and apis services provided by a library or similar institution should be easy to use by anyone and in any form. most services, however, can only be used in the fixed context of a particular user interface (ui). if a service is also accessible via application programming interface (api), its functionality can be integrated and used in other applications, too. unfortunately the need to expose services via open apis is less obvious than uis. user interfaces are curated and revised by usability studies and user experience (ux) at best. in other instances the ui is simply judged with common sense by normal library staff and management. apis on the other hand, cannot simply be viewed, used, and judged by anyone. unlike the ui, an api is not a final application to make use of a service, but the basis for creation of service applications: without apis, applications are difficult to build and services can only be provided in limited form. without applications, however, it is difficult to justify the need for an api. to give an example, most library systems lack an api to query current availability of documents held by the library. as long as information about current availability was only displayed in local library opacs there was little motivation to create a public api for this purpose. with the need to display availability information in discovery interfaces such as vufind, the document availability information api (daia) was created at gemeinsamer bibliotheksverbund (gbv) (voß and reh 2014). a daia web service was created for pica lbs, the most common local library system in gbv. but little interest was shown by other libraries and system vendors as long as they did not require the api for internal use. the full benefit of an open api is not revealed until different applications by different parties make use of it. this article will demonstrate a possible strategy to increase visibility and use of library apis by providing client modules that facilitate the creation of applications by third parties. [1] the modules are based on the javascript framework angularjs which is getting more and more popular among developers of web applications. the general strategy is illustrated in the following diagram: figure 1. from library service to web application angularjs angularjs (https://angularjs.org) is a web application framework that aims to enhance the functionality of javascript. the framework is designed to support modularization on multiple levels. functionality of applications is broken into parts that can be tested and reused independently. application logic is first grouped in modules, each included with an html <script> tag. some popular modules are listed at the unofficial directory http://ngmodules.org/. modules may build on each other and define directives. these directives can be used in the form of custom html tags and attributes (“declarative html”). the core angularjs module contains directives for basic programming syntax in html, such as conditionals (ng-if) and loops (ng-repeat), among others. this extension of html is further enriched by a template syntax with angularjs expressions written in curly brackets ({{...}}). the most common use of these expressions is to dynamically display variables in html templates. in contrast to most other template systems, variables are bound two-way: the display is updated automatically when a variable is changed, and changes of the html document (e.g. by input forms) are reflected in the javascript variables. to avoid name collision and to better separate functionality of different directives, variables belong to scopes that act like namespaces in other programming languages. the following example illustrates the use of angularjs with scope variables, templates, and directives: [2] <html ng-app="myapp"> <head> <script src="angular.min.js"></script> <script> angular.module('myapp', []); function mycontroller($scope) { $scope.books = [ { title: "one thousand and one nights" }, { title: "where the wild things are", author: "sendak" }, { title: "the hobbit", author: "tolkien" }, ]; } </script> </head> <body ng-controller="mycontroller"> <ul ng-repeat="b in books | orderby:'title'"> <li> <i>{{b.title}}</i> <span ng-if="b.author">by {{b.author}}</span> </li> </ul> </body> </html> the example consists of an application module “myapp”. this module defines a controller “mycontroller” to set a list of three bibliographic items in the variable “books” of a given scope. the controller is later used in the html body to display a sorted list of books with an html template. the template makes use of standard angularjs directives (ng-repeat, ng-if) and expressions (| orderby:'title', b.title, b.author). the application logic to create such a list could also be packed in a new directive to be used as a “widget” in multiple places. modules for embedding library services the practical embedding of library services in websites with angularjs is illustrated in the following two examples. both are available as angularjs modules for easy reuse: the ng-suggest module provides access to search suggestions and links (voß and horn 2014a) and the ng-daia module provides access to availability information (voß and horn 2014b). both modules are hosted at public git repositories with api documentation, examples, and downloads (https://gbv.github.io/ng-suggest/ and https://gbv.github.io/ng-daia/). suggestions with ng-suggest the opensearch standard for search engines includes a specification for how to query search suggestions and autocomplete services via http (clinton 2006). suggestion services are provided by many search applications as “typeahead”. the method can also be used for instance by recommendation services (voß 2008) and to support tagging with controlled vocabularies (nagaya et al. 2011). figure 2. typeahead via opensearch suggestions the opensearch suggestions specification defines a query response as a json array with at least two elements (query string and a list of search completions): [ "moz", ["mozilla","mozilla firefox","mozart","mozilla thunderbird",...] ] optional elements can include descriptions and urls for each search completion. while processing of this simple format is not very complex, it still requires javascript skills to make use of a suggestion service. ng-suggest simplifies the embedding to two html statements. the following example code adds wikipedia typeahead features to an input form element: <html ng-app="myapp"> <head> <script src="angular.min.js"></script> <script src="ui-bootstrap-tpls.min.js"></script> <script src="ng-suggest.min.js"></script> <script> angular.module('myapp', ['ui.bootstrap','ngsuggest']); function mycontroller($scope) { $scope.api = "https://en.wikipedia.org/w/api.php?action=opensearch" + "&limit=10&namespace=0&format=json&search="; $scope.selectitem = function(item) { $scope.item = item; }; }; </script> <link href="bootstrap.min.css" rel="stylesheet" /> </head> <body ng-controller="mycontroller"> <input style="width:400px" class="form-control" typeahead-on-select="selectitem($item)" ng-model="input" suggest-typeahead="api" jsonp=1 placeholder="search wikipedia" /> <div ng-if="item">{{item.label}}</div> </body> </html> the resulting html page would look like this: figure 3. suggest wikipedia articles with ng-suggest similar suggestions can be provided for any open search suggestions service by just changing the service’s base url. among other features, responses can be embedded as simple lists (seealso recommender services, for instance related documents and related publications), and other json response formats can be mapped to suggestions format. availability with ng-daia the daia specification defines a data model and an http api for accessing information about the current availability of documents (voß and reh 2014). in contrast to complex internal library system apis, such as ncip and slnp, daia was designed to be used openly. the aim of daia is to provide a way for libraries to allow open and easy-to-use access to holdings information from their catalogs. this, in turn, enables the inclusion of document availability information in external applications and websites (catalogs, reference management, e-learning platforms, etc.). among other formats, daia provides availability information in json, the first choice for web applications written in javascript. the angularjs module ng-daia implements client code to execute and process a daia query and to display holding information in convenient form. below is an example of the html integration: <html ng-app="myapp"> <head> <script src="angular.min.js"></script> <script src="ng-daia.min.js"></script> <script>angular.module('myapp', ['ngdaia']);</script> <link href="ng-daia.css" rel="stylesheet" /> </head> <body> <div daia-api="http://daia.gbv.de/" daia-id="opac-de-ma9:ppn:685460711"> </div> </body> </html> the ng-daia module provides a directive (daia-api) to query a daia service with a given document identifier (daia-id). the recieved daia response is then fed to a customizable angularjs template, resulting in the following display: figure 4. full availability view with ng-daia default template the full availability view as implemented in the default templates of ng-daia reflects the nested structure of daia data model, consisting of an outer layer for institutional and general document information, and specific information for each document holding (voß and reh 2014). the default template of directive ng-api uses another directive for the display of a holding item (daia-item) and its item template can be customized as well. another directive is provided for the most compact display (daia-simple, see figure 5). daia simple is a flattened, aggregated form of availability information that covers typical use cases, such as short display in a result list (see section 6.1 of the daia specification). the ng-daia module includes functions to transform from full daia to daia simple as well. figure 5. minimal availability view with daia-simple all templates included in ng-daia can be customized with css. localization for display in other languages is already supported with the popular module angular-translate (precht 2014). thanks to two-way binding of angularjs variables, a simple statement such as $scope.language = 'de' can be enough to update the full availability display in another language. conclusions despite efforts to open up library systems via standard apis, for instance the ils-di recommendations (ils discovery interface task force 2008), the support of library services via open apis is rather low. if apis exist (e.g. ncip), they are often complex, vendor-specific, or available only for internal use. one reason for the lack of open apis may be the invisibility of benefits and usage examples. the examples given in this article demonstrate how library services (e.g. search suggestions, recommendations, document availability) can be used easily once they have been made available via standardized apis (e.g. open search suggestions and daia). the simple integration into web applications also requires client modules like ng-suggest and ng-daia for angularjs. modules for other standard apis relevant to libraries, such as opensearch and sru for search (hammond 2010) and paia for patron account interaction (voß 2014), shouldn’t be hard to implement. [3] most importantly, these client modules only have to be implemented once instead of having to build both server and client implementation for each particular library system. with a set of angularjs modules for the basic library services (search, availability, patron account) it should even be possible to create a custom opac interface in less than a hundred lines of html and javascript. even if angularjs is not the framework of your choice, it makes sense to provide client modules to your apis, as illustrated in figure 1. libraries should not only expose their services via openly specified apis but also provide client libraries to facilitate the integration of these services into web applications. to minimize the work of doing so, one should build on standardized apis independent from particular library systems. we hope to motivate more library developers in doing so. [4] references breeding m. 2009. opening up library systems through web services and soa: hype or reality? american library association. clinton d. 2006. opensearch suggestions. available from: http://www.opensearch.org/specifications/opensearch/extensions/suggestions hammond t. 2010. nature.com opensearch: a case study in opensearch and sru integration. d-lib magazine [internet] 16. available from: http://dlib.org/dlib/july10/hammond/07hammond.html hoskins j ed. 2014. angularjs modules. [internet]. available from: http://ngmodules.org/ ils discovery interface task force ed. 2008. technical recommendation. digital library federation. available from: http://old.diglib.org/architectures/ilsdi/ nagaya s, hayashi y, otani s, itabashi k. 2011. controlled terms or free terms? a javascript library to utilize subject headings and thesauri on the web. code4lib journal [internet]. available from: http://journal.code4lib.org/articles/5994 precht p. 2014. angular translate. [internet]. available from: http://angular-translate.github.io/ voß j, horn m. 2014a. ng-suggest. [internet]. available from: http://gbv.github.io/ng-suggest/ voß j, horn m. 2014b. ng-daia. [internet]. available from: http://gbv.github.io/ng-daia/ voß j, reh u. 2014. document availability information api (daia). available from: http://purl.org/net/daia voß j. 2008. seealso: a simple linkserver protocol. ariadne [internet]. available from: http://www.ariadne.ac.uk/issue57/voss/ voß j. 2014. patrons account information api (paia). available from: http://gbv.github.com/paia/ endnotes [1] this article is based on the assumption that libraries actually want to facilitate the use of their services. in some cases this assumption might be wrong. [2] all examples are available also as part of the article’s code repository at https://github.com/jakobib/angularjs2014/. [3] we are currently working on the module ng-skos (https://github.com/gbv/ng-skos) to interact with authority files and simple knowledge organisation systems. [4] one can also ask the vendor of library systems to implement standardized apis to the core functionality of its product, but this requires some pressure by libraries as customers. about the authors jakob voß works in research and development at the head office of the common library network (gemeinsamer bibliotheksverbund, gbv) in göttingen. moritz horn studies information management at the university of applied sciences and arts in hanover. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – pipeline or pipe dream: building a scaled automated metadata creation and ingest workflow using web scraping tools mission editorial committee process and structure code4lib issue 58, 2023-12-04 pipeline or pipe dream: building a scaled automated metadata creation and ingest workflow using web scraping tools since 2004, the fraser digital library has provided free access to publications and archival collections related to the history of economics, finance, banking, and the federal reserve system. the agile web development team that supports fraser’s digital asset management system embarked on an initiative to automate collecting documents and metadata from us governmental sources across the web. these sources present their content on web pages but do not serve the metadata and document links via an api or other semantic web technologies, making automation a unique challenge. using a combination of third-party software, lightweight cloud services, and custom python code, the fraser recurring downloads project transformed what was previously a labor-intensive daily process into a metadata creation and ingest pipeline that requires minimal human intervention or quality control. this article will provide an overview of the software and services used for the recurring downloads pipeline, as well as some of the struggles that the team encountered during the design and build process, and current use of the final product. the project required a more detailed plan than was designed and documented. the fully manual process was not intended to be automated when established, which introduced inherent complexity in creating the pipeline. a more comprehensive plan could have made the iterative development process easier by having a defined data model, and documentation of—and strategy for—edge cases. further initial analysis of the cloud services used would have defined the limitations of those services, and workarounds could have been accounted for in the project plan. while the labor-intensive manual workflow has been reduced significantly, the required skill sets to efficiently maintain the automated workflow present a sustainability challenge of task distribution between librarians and developers. this article will detail the challenges and limitations of transitioning and standardizing recurring web scraping across more than 50 sources to a semi-automated workflow and potential future improvements to the pipeline. by matthew krc and anna oates schlaack introduction the federal reserve bank of st. louis’s fraser digital library is composed of historical and contemporary publications, as well as other types of digitized and born-digital content that are curated and maintained by a team of librarians and supporting staff. the digital library staff capture and archive contemporary digital publications on a recurring basis as they are issued. these digital publications, like myriad information on the web that have archival value, are not presented as structured data that is easily retrievable through standard automated processes—e.g., metadata capture via an application programming interface (api). an institution that wishes to continuously collect such documents must either depend upon frequent, manual checks of source websites or on services that monitor and identify changes to a given web page. a challenge to automating this archival process of checking for new content is having a reliable system that can capture digital resources and readily available metadata with minimal human intervention. despite this core challenge and other potential obstacles, in 2022, the fraser digital library team embarked on a project to attempt automating elements of, what had previously been, a tedious, manual process. background since 2004, the fraser digital library has provided free access to publications and archival collections related to the history of economics, finance, banking, and the federal reserve system. the collections are maintained by a team of four library staff, and the digital library portal and digital asset management system (dams) are maintained by a team of three web developers. the library services staff and information technology staff ultimately report to the same division head, though through different reporting structures, and the two teams work together in an agile project management environment to address enhancements and maintenance for the application. physical collections for in-house or outsourced digitization are obtained through partnerships with external host repositories. digital content is primarily collected directly from government institutions’ websites. in 2006, digital library staff began collecting content as it was published [1]. over the years, the manual collection workflow saw several iterations to make the process more efficient. in 2015, staff began using distill, a tool that monitors web pages and sends alerts when a web page changes. this software was particularly valuable for reducing the extensive time required to survey sources for new content. while this improved the process by reducing the time required to evaluate whether a source had published new content, automated monitoring of web page changes was not a perfect process. the majority of alerts were false positives—i.e., distill identified that there was a change to the web page, but there was no new content. for example, if an icon moved to another location on the page or if new tweets appeared in the embedded twitter feed, the tool would send an alert. between february 2018 – july 2022, 46% of web page change alerts were false positives—resulting in additional time to filter out alerts which did produce new content. fraser collections range from government documents and data releases to academic publications and personal archives. each of those content types require different metadata and, in the case of the contemporary digital publications, the descriptive metadata provided by the source varies. some publications in fraser’s collection scope are captured from the government publishing office’s (gpo) govinfo, which serves rich, standardized metadata. other publications, such as the press releases of the u.s. department of the treasury, present only a title and date of publication on the source website. in addition to the challenge of disparate metadata availability, the unique publication schedules posed challenges to the manual workflow and any potential automation. the distill web monitoring tool was so effective for this reason–it offered the possibility of knowing when an irregular publication had been issued simply by checking for changes on the page. albeit with faults, automated monitoring streamlined the process of tracking when sources had been checked for new content. timeliness to ingest new publications in fraser is key to ensuring that the digital library is a trustworthy location for finding historic and contemporary documents on economics. as seen with the web monitoring tool, any attempt at automation could be thwarted by the possibility of dynamic websites constantly changing. as of 2021, the number of publications included in the recurring downloads workflow had increased significantly [2], and, with that growth, the time required to monitor alerts, collect, describe, and ingest content manually as it was published became a full-time position. to alleviate staff time required to complete the largely monotonous workflow, in 2019, digital library staff began exploring possibilities for automation. the most successful attempt at automation was a python program that ran daily to scrape a web page, check to see whether the captured elements already existed in the dams, and, if not, transform and ingest the metadata and file, and send an email notification to staff when the ingest was complete and ready to be made public. the program was customized for a single publication that was the most laborious to capture because of its high publication frequency, with new content published as often as every 15 minutes [3]. given the aforementioned challenges of variance in sources, automating harvesting for all of the publications was not achievable with a simple program but required an extensible web scraping framework. recurring downloads design phase informed by the successful python program pilot, a team of architects and the agile web development team that supports fraser began a design phase to ideate a pipeline that would scrape metadata and files from a set of more than 50 websites, transform and enhance the metadata, and handle ingest of the metadata and files into the dams. the automated recurring downloads process was broken into two components: data processing and data management. data processing would run entirely in the background and would be managed through a dashboard in the dams. the data processing had to be capable of: harvesting and transforming metadata to the local metadata object description schema (mods) implementation, following title-level metadata requirements handling when new content is available and filtering out existing content the ideal data management dashboard would include: a list of new records queued for quality assurance (qa) notifications for records missing required metadata fields or with metadata conflicts (e.g., the author of a work does not correspond to an existing name authority record) an interface in which to modify existing sources and add new sources to the pipeline an interface for selecting elements on a source web page that should be scraped an interface to schedule scrape jobs at the title-level (i.e., distinct scraping schedules for each publication) from the initial design sessions, the team tentatively decided on an infrastructure that: leveraged the openrefine api to handle metadata transformations so that digital library staff had ownership of modifying metadata transformations without requiring changes to the application code employed web scraping software that had out-of-the-box, point-and-click functions for selecting web elements to scrape used aws dynamodb for select administrative data that would be created exclusively for the automated workflow as evident in the many learning resources, including a library carpentry lesson and library juice academy course, openrefine is a tool widely used in libraries for metadata work, and is invaluable specifically for metadata normalization. with the pipeline, library staff hoped to automate repetitive tasks, including normalizing dates and formatting author names, so that metadata records would be public-ready when ingested into fraser. the openrefine api enables clients to run automatically a set of openrefine operations, functionally fulfilling the transformation operations needed for the pipeline. a set of operations can be created in the openrefine user interface, with which fraser library staff were already familiar. in this proposed pipeline implementation, library staff would create operations in the openrefine application, load the operations for use in the pipeline, and assign operations at the title-level. this would enable library staff to add and modify metadata transformations operations without requiring web developer support. prior to the recurring downloads project design phase, digital library staff identified parsehub and web scraper as two web scraping tools that are easy to use for library staff who do not have coding experience. building a custom web scraping tool that replicated the same functionality and user-friendly interface would not have added notable value and would have required more time for planning and investing more development resources to build. web scraper was ultimately selected for handling the web scraping element of the pipeline because it was the easiest output to process and was configurable to authenticate with other tools. in order to store metadata transformation templates that would be quickly and easily retrievable, the development team chose to use aws dynamodb for the ease with which users could access and add and edit dynamodb documents, or database records. dynamodb is an amazon web services (aws) managed nosql, non-relational database service that allows fast access to stored data. because the recurring downloads pipeline wouldn’t require complex queries on data that would be retrieved, storing the metadata transformation templates in the primary postgresql relational database was an unnecessarily complex approach. prior to the project discussions, the web development team had been interested in investigating dynamodb as a more lightweight option for certain data types, and this project was an optimal pilot. in the aws console, librarians also had access to edit dynamodb data in a manner more intuitive than typical sql database clients allow. figure 1. screenshot of aws dynamodb record form in aws console. openrefine, web scraper, and dynamodb were all new tools to the development team, and integrating openrefine in the pipeline was ultimately discarded because its integration in the pipeline would require additional developer analysis, planning, experimentation, and testing. ultimately, it was easier and, at the time, deemed more cost effective for the web developers to work with a familiar tool rather than learn to integrate one that wasn’t native to their existing webstack or a provided aws service. instead, digital library staff compiled a list of known metadata transformation operations required, which the development team would integrate into the new workflow as custom code to run in an aws serverless lambda service. pipeline implementation the recurring downloads pipeline was deployed in october 2022. due to timeline and resource constraints, the architecture and data flow were slightly different and pared down from the finalized design. the lack of some features removed from the released pipeline have posed pain-points for the digital library staff, which are detailed in the challenges section of this article. the final pipeline consisted of tools that fulfilled the central function of capturing files and metadata from a source, transforming the files and metadata to follow fraser requirements, and ingesting the files and metadata to the dams for qa before being made public. table 1. list of enhancements to existing infrastructure for the metadata pipeline tools function(s) web scraper browser extension (chrome extension, firefox add-on) sitemap creation web scraper cloud sitemap storage, scrape job scheduling, scrape job output, metadata preprocessing aws dynamodb metadata transformation template storage aws lambda metadata transformation, file transformation enhancement to dams new record queuing, record qa figure 2. diagram of recurring downloads pipeline architecture and workflow. central to the web scraper service are sitemaps. distinct from xml sitemaps that aid search engines in web crawling, web scraper sitemaps are json documents that define the website that should be scraped with a “starturl” parameter and what to scrape from the given website with “selectors.” selectors contain information that inform web scraper what and how to scrape. to set up a new source to be included in the pipeline, library staff create sitemaps using the web scraper browser extension and upload the sitemaps to web scraper cloud. { "_id": "c4l-journal", "starturl": [ "https://journal.code4lib.org/" ], "selectors": [ { "id": "issue-title", "multiple": false, "parentselectors": [ "_root" ], "regex": "", "selector": "h1.pagetitle", "type": "selectortext" }, figure 3. snippet of web scraper sitemap (see appendix 1 for full example). web scraper cloud allows users to store sitemaps and create scrape job schedules unique to each sitemap. by default, the application supports downloading scraped data output as csv, json, and xlsx. the scraped data can also be retrieved as json via the web scraper api, which was the preferred method to retrieve data for the recurring downloads pipeline. web scraper cloud supports interfacing with other applications with a webhook that sends an http post request to a specified endpoint with a corresponding authentication token and a scrape job id corresponding to the job that was kicked off. // this is the data that your server receives "scrapingjob_id": 1234 "status": "finished" "sitemap_id": 12 "sitemap_name": "my-sitemap" // optional, custom_id will be passed to post data only if set "custom_id": "your-custom-id" figure 4. web scraper webhook data included in the http post request. figure 5. screenshot of creating a sitemap to scrape articles from the current issue of code4lib journal using the web scraper browser extension for chrome. figure 6. screenshot of metadata preprocessing in web scraper cloud. among the enhancements to the existing fraser application, the development team created an http endpoint to accept the post data from web scraper and, if properly authenticated, invoke a serverless aws lambda instance with the scrape job id as an input parameter. the lambda then runs python code that requests web scraper data corresponding to the scrape job id, and performs metadata transformations and validation. the code queries aws dynamodb for a document that corresponds to the parent title record to which the scraped record would belong, and contains the specific transformations necessary to match the metadata payload that will eventually be inserted into the fraser database, using pre-formatted variable inserts, such as date values and issue numbers. these injected variable keys are represented in the dynamodb document, using a custom syntax developed for the project, with curly brackets used to represent variable names, such as {mm} for a zero-padded month number, and {month} for a month’s proper name. figure 7. screenshot of dynamodb metadata template. among the metadata values generated by the lambda is a filename that follows a standard naming convention, unique to each fraser title, usually containing a timestamp value corresponding to the date the document was published. to determine whether the record already exists in the fraser dams, the constructed filename is checked against the existing files in fraser’s file storage system in aws simple storage service (s3). if a file with the constructed name exists in s3, the file is considered not new, and the process terminates immediately. if a matching filename does not exist, the next step of the process kicks off. checking only for new content in this way does not handle edge cases where a file has been revised with new or changed content but the filename remains the same. several alternative options were considered but required exploration and experimentation that did not fit into the project timeline. recommended future work would modify the process to check other unique criteria, such as a file checksum, in order to account for revised documents that would otherwise generate the same filename. after file download and metadata transformation and validation completes, the lambda uploads the document to fraser file storage and inserts the metadata record into the fraser relational database, where pending records are accessed through the fraser dams. once metadata and files reach the dams, digital library staff conduct a qa review of the document to ensure that the record contains correct metadata. in this manual step, staff are able to clean up records that may have missing or inaccurate metadata and, in cases where a document could not be retrieved through the automated process, manually capture the file. potential future enhancements for the process include advanced and filtered logging and tracking of the web scraping and ingest processes. digital library staff have requested individual job logging in the dams dashboard to more easily track errors in the pipeline. web scraper records scrape job and webhook post data, including scrape job information for records which, ultimately, were not ingested because the content already existed in the dams. the web scraper logs do not represent failures in the lambda, where scraped content does not exist in fraser but the process ceases for another reason. notifications for and comprehensive logging of unsuccessful job runs would help staff quickly to address faulty runs in the future. challenges from the outset of development to the final implementation, the development team encountered obstacles with automating the existing recurring downloads workflow. the manual process of archiving documents had been long fraught by issues with false-positive alerts from the website monitoring software and tedious, mundane daily tasks. the workflow, however, was clearly driven by institutional knowledge, not possible to replace fully with machine processes. additionally, the development team met technical hurdles that slowed development and demanded revisiting and altering the planned architecture and workflow. several impediments that could have been anticipated and mitigated caused unanticipated disruptions. technology stack the developers supporting fraser exclusively build and maintain php applications. the aws lambda service is limited in what languages are supported. the team quickly began work on the lambda code using php and realized, after the bulk of the code had been completed, that lambda would not support php in a straightforward way. shifting from php to python was a necessary transition and required members of the team to work in an unfamiliar language while development was already in progress. more thorough preliminary analysis could have caught this earlier and allowed time for staff training in the new language, or more thorough exploration of a solution that would have supported languages that the team was already familiar with. while many of the born-digital documents are downloadable pdf files, some of the documents involved capturing semi-dynamic web pages as static pdf documents. to capture primarily textual content presented only as html, the manual archiving process depended on the print to pdf feature available in most modern web browsers. with several rounds of testing to automate this conversion process, google chrome’s headless print option stood out as the best option for producing a pdf that held a look-and-feel most similar to the original web page. while similar functions exist in python libraries, most still depend on a headless browser that emulates browser functionality without a graphical user interface (gui). python libraries that don’t use a headless browser produce pdfs with varying levels of quality, and for the most part, lack the fidelity of the original web page present in browser-based methods. using the headless browser necessitated including a google chrome binary with the deployed lambda, which proved technically difficult using the infrastructure planned at the outset of the project. because development was done locally on machines that had the google chrome binary and due to the nature of user testing, this critical flaw in the pipeline was not realized until after the project was deployed to production, at which time rearchitecting the pipeline was no longer an option [4]. institutional limitations on third-party software institutional requirements limited third-party software permissible for use. web scraper was a new software to the institution and was required to undergo thorough security audit and, as a cloud-based software, necessitated further evaluation and approval. the development team began implementation with the assumption web scraper would be approved for use. while the software ultimately was approved, plans to productionize the pipeline were dependent upon approval because a back-up solution that fully met the requirements set in the project planning had not been identified. additionally, if openrefine had been ultimately pursued as a component of the pipeline architecture, it would have also been scrutinized by the internal review process. project planning even with project planning, sufficient time was not allocated to planning and refining work and the iterative development and testing cycle needed for successfully working with the new toolkit. areas of the workflow that required additional, more nuanced consideration were neglected. many of the metadata transformation operations and templates were created ad-hoc, but, as a core function in the recurring downloads pipeline, this component of the workflow would have been greatly improved by standardizing a coherent schema, with corresponding documentation around the various metadata variables that would be scraped from the web sources. the lack of advance planning on metadata transformation led to confusion among the various stakeholders, including web developers and library staff, as to how the metadata transformation schema should be formatted, which led to on-the-fly decisions around basic requirements during development. pipeline responsibilities and sustainability early planning of the recurring downloads pipeline included considerations for library staff to have greater oversight of and responsibility for the pipeline. in the final product, the components that library staff maintained included the web scraper sitemaps and metadata templates stored in dynamodb. while these components gave a majority of workflow ownership to library staff, the product was very advanced and technical. the pipeline would have benefited from an approach that leaned less on technical staff and more on open source tools and the knowledge of library staff. one such approach, that was abandoned early on in planning, was the use of openrefine. the recurring downloads project was planned as a final business enhancement prior to a multi-year technical project that would leave applications in maintenance mode. bugs and inadequate functionality that resulted from insufficient planning and user testing would not be in scope for development until after the technical project completion. because the implementation of requirements were not fully designed as part of project planning, development would have required many iterations of testing and remediation. the goal to automate collecting content from source websites, transforming and enhancing the metadata, and ingesting the metadata and files into the dams ultimately was a success. the fraser team currently has a productionized workflow for gathering external metadata and documents and has ingested 500 new records through this process. false positive alerts, which previously took up a large portion of library staff time sifting through, have disappeared as a concern. in general, library staff time dedicated to the recurring download process has decreased. however, some of that time and effort has, in turn, shifted to technical staff, whose time has historically been more limited. any institution looking to embark on an automation effort such as this would be advised to consider the job descriptions for staff that are responsible for supporting the process or function. conclusion the fraser library staff and web developers embarked on a formidable goal in the recurring downloads project. the ingenuous plan would alleviate some of the workload and tedium involved in the daily manual process, while also optimizing the speed and efficiency with which new records could be added and increasing the scalability of collecting digital documents and publications. the project and resulting pipeline was successful in many ways: the amount of active time required to add new records was reduced – from getting an email notification with a url, reviewing the site to ensure new content was present, adding files to the s3 file store and finally creating a metadata record, to simply clicking two buttons in a single location to qa and approve a record. the number of false positive alerts of new archivable documents was reduced to zero. given constrained staff resources, the pipeline and simple dashboard made it possible to distribute the workload from a single employee to several staff without tedious training. however, the pipeline and its maintenance increased complexity for the recurring downloads workflow. instead of digital library staff owning the archival process entirely, it became distributed between technical staff and library stakeholders. much of the maintenance and additional work on the project had to be assumed by technical staff, whose capacity is consumed by other priorities outside of fraser. while this project to automate and streamline contemporary digital publication collection was not without trials, it was a valuable learning experience for stakeholders involved, will be informative for future ventures to automate workflows and tasks, and ultimately created efficiencies for digital library staff. notes [1] the first record created as part of what was later formalized as the recurring downloads workflow was the october 2006 issue of economic indicators https://fraser.stlouisfed.org/title/1/item/134. [2] from 2020-2022 the sources in scope for collection and archiving increased from ~20 to ~50. [3] fraser includes historical press releases from the u.s. department of the treasury going back to 1916 and collected ongoing from the u.s. department of the treasury press releases website. [4] a potential option would have been to use a container service such as docker in order to provide greater flexibility in pulling in external dependencies for a codebase that is otherwise limited in lambda. about the authors matthew krc (he/him) is a data engineer and product owner of the fred data desk team at the federal reserve bank of st. louis, where he contributes to and plans work on the data and metadata pipelines for the federal reserve economic data website. he has a professional background in library web application development and was previously part of the technical team that supports the fraser application. anna oates schlaack (she/her) is a product owner in the research division at the federal reserve bank of st. louis, where she leads a web development team in modernizing legacy information discovery applications. appendices appendix 1. example web scraper sitemap { "_id": "c4l-journal", "starturl": [ "https://journal.code4lib.org/" ], "selectors": [ { "id": "issue-title", "multiple": false, "parentselectors": [ "_root" ], "regex": "", "selector": "h1.pagetitle", "type": "selectortext" }, { "id": "issue-date", "multiple": false, "parentselectors": [ "_root" ], "regex": "", "selector": "h1.pagetitle", "type": "selectortext" }, { "id": "article-wrapper", "multiple": true, "parentselectors": [ "_root" ], "selector": "div.article", "type": "selectorelement" }, { "id": "article-title", "multiple": false, "parentselectors": [ "article-wrapper" ], "regex": "", "selector": "a", "type": "selectortext" }, { "id": "article-authors", "multiple": false, "parentselectors": [ "article-wrapper" ], "regex": "", "selector": "p.author", "type": "selectortext" }, { "id": "article-abstract", "multiple": false, "parentselectors": [ "article-wrapper" ], "regex": "", "selector": ".abstract p", "type": "selectortext" }, { "id": "article-url", "linktype": "linkfromhref", "multiple": false, "parentselectors": [ "article-wrapper" ], "selector": "a", "type": "selectorlink" } ] } appendix 2. example aws dynamodb document { "id": "c4l-journal", "sortkey": "c4l-journal", "cron": "0 22 * 2,5,8,11 5", "filenames": { "pdf": "c4l_{yyyy}_{nn}.pdf" }, "frequency": "q", "metadata": { "genre": [ "periodical" ], "language": [ "eng" ], "titleinfo": [ { "title": "{issue-title}", "titlepartnumber": "{issue-titlepartnumber}" } ], "origininfo": { "dateissued": "{month} {d}, {yyyy}", "issuance": "periodical", "sortdate": "{yyyy}-{mm}-{dd}" }, "typeofresource": "text", "physicaldescription": { "form": "electronic", "digitalorigin": "born digital" }, "tableofcontents": [ { "titleinfo": [ { "title": "{article-title}" } ] }, { "name": [ { "role": "creator", "namepart": [ "{article-author}" ] } ] } ], "relateditem": { "1": { "@type": "itemparent", "recordinfo": { "recordidentifier": [ "123" ] }, "titleinfo": [ { "title": "code4lib journal" } ] } } }, "s3_path": "/docs/publications/c4l/" } subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – creating library websites with joomla: not too big, not too small, just right mission editorial committee process and structure code4lib issue 12, 2010-12-21 creating library websites with joomla: not too big, not too small, just right many organizations, including libraries, are turning to content management systems to simplify the management of their websites. alfred university‘s herrick memorial library recently implemented a new website using joomla, an open-source content management system. while drupal has received significant attention in the library community, joomla may be a more practical choice for some libraries. the purpose of this paper is to share our experience with joomla so that other libraries can more easily evaluate its suitability to their environment. by ellen bahr and matt speed introduction alfred university is a small comprehensive university in western new york with a student population of about 2,400. herrick memorial library serves the college of business, the college of liberal arts and sciences, and the graduate school. a second university library, the samuel r. scholes library, serves the new york state college of ceramics, a statutory unit of the university. herrick has four librarians, including the library director, and seven support staff. figure 1. herrick’s joomla website recently, when faced with the task of redesigning our website, we chose to do it with joomla, an open source content management system. content management systems (cmss, also referred to as web content management systems or wcmss) are database-driven software systems for building and managing websites. the open source content management systems most frequently discussed in libraries today are drupal, wordpress, and joomla. drupal has a reputation for flexibility and extensibility but also for having a steep learning curve. wordpress, known for ease of use, began as a blogging platform and has evolved to incorporate more features of a cms. joomla, which emerged in 2005 as a fork in development of the mambo cms, has gotten relatively less attention in the library community but has a strong following in other sectors, both for-profit and not-for-profit. joomla framework like drupal and wordpress, joomla is written in php and stores data in a mysql database. unlike websites that rely on static html, sites built with content management systems like joomla serve content dynamically. when a user requests a page on a joomla site, the content is retrieved from the mysql database, formatted according to the site’s template, and displayed in the user’s browser. as depicted in figure 2, joomla is a “three-tiered system” [1] consisting of an extension layer, an application layer, and a framework layer. the entire joomla system interacts with the mysql database, which stores information about the site. the extension layer consists of modules, components, and templates that extend the functionality of the joomla system. some extensions are included with the joomla core installation and others can be added later. the application layer includes jinstallation for installing joomla, jadministrator which is responsible for the back-end administrative interface, jsite which is responsible for the front-end of the website, and an xml-rpc interface. the framework layer contains joomla’s programming foundation, including libraries needed by the framework and plugins that extend the functionality of the framework. figure 2. joomla software framework the core joomla installation includes all of the features needed to create a functioning website, including systems for managing users, media, languages, banners, contacts, polls, searching, web links, content, newsfeeds, menus, and templates. these features can be expanded through add-ons. help features, such as a glossary of joomla terms, links to joomla documentation, and a version checker are built into the web-based administrative interface. for a more complete description of the core joomla features see the joomla features overview [2]. the joomla extensions directory [3] provides access to thousands of add-ons, the majority of which are free. these provide additional functionality not included in the core installation. joomla’s extension manager (figure 3) makes installing extensions easy. there are three options for installing extensions: upload a package file, install from a directory, or install from a url. uploading a package file is probably the most common method. the extension archive file is downloaded from the website of the extension provider and saved locally. browse to where you saved the file and select upload file and install. at this point, joomla completes the installation. figure 3. joomla extension manager in the joomla extensions directory, extensions are classified as components (c), modules (m), plugins (p), languages (l), templates (t), or extension specific add-ons (s). many installation packages include extensions of more than one type. for example, in the screenshot below, the template toolbar extension contains only a template, while the jforms extension contains a component and a plugin. figure 4. joomla extensions directory version 1.5 of joomla includes five extension types: plugins are event handlers. “in the execution of any part of joomla, be it the core, a module or a component, an event can be triggered. when an event is triggered, plugins that are registered with the application to handle that event execute” [4]. joomla’s wysiwyg editor is an example of a plugin. content is stored in joomla’s mysql database as html. the wysiwyg editor plugin converts html to rich text for editing, and then back to html for storage. joomla plugins are similar to hooks in drupal [5]. while joomla’s documentation describes plugins as an extension type, joomla’s software framework diagram (figure 2) includes plugins in the framework layer. plugins operate mostly behind the scenes, sometimes in combination with components or modules to provide a user interface. modules are used to display chunks of content on web pages; joomla has a variety of module types to handle different kinds of content. each template has defined module positions, and multiple modules can be displayed on a given page. examples of modules include breadcrumbs, rss feeds, banners, menus, footers, simple polls, and user logins. joomla modules are similar to blocks in drupal [5]. components are mini-applications. they have more complex administrative settings than modules, and only one component can appear on a given page. an example of a component is chronoforms, which can be used to create interactive forms. joomla components are similar to modules in drupal [5]. language extensions are used to install additional languages. once installed, they are managed in the language manager. joom!fish, which supports multi-lingual websites, is an example of a language extension. templates control the design of the website. they can be custom-built or selected from among the many joomla templates available for purchase or for free. in drupal, templates are called themes. managing users joomla provides for three user types: unregistered front-end users, registered front-end users, and registered back-end users. unregistered (or public) front-end users are visitors to the site who have access to any content that doesn’t require a login. registered front-end users can be given additional privileges, including the ability to view restricted content, and to author, edit, or publish articles. there are three levels of back-end users: managers, administrators, and super administrators, each with increasing privileges. installation and configuration joomla requires a server with apache, mysql and php. the xamp installer is an easy way to install all components. at our institution, the library wanted to get some experience with joomla without having to install it locally. a relatively simple way to do this is to use a commercial hosting service. many hosting services offer “one-click” installation for joomla and drupal, putting installation within reach of those with minimal technical skills. the joomla resources directory [6] includes a list of potential hosts. we installed joomla and drupal on a hosted server in order to compare them, and the experience contributed heavily to our decision to use joomla. it was clear that joomla would require less customization than drupal, hopefully shortening our development timeline. further, it seemed unlikely that we could take advantage of everything that drupal had to offer, while joomla’s features were sufficient for our relatively straightforward needs. after talking with another academic library that had recently built a site with joomla, we decided to take the plunge. while we were experimenting with joomla on a hosted server, we were in touch with the university’s information technology services (its) department about eventually hosting the site locally. we continued to develop the site on the hosted server, with the understanding that we would contact its when we were ready to move our site to a local production environment. when we reached this point, the local installation and configuration of joomla was handled by an its network services support specialist. what follows is a brief description of the installation and configuration process at our institution. the specific steps will vary depending on a variety of factors in your local environment. refer to the joomla! 1.5 installation manual [7] for complete information. we use a physical (as opposed to virtual) server that runs oracle unbreakable enterprise linux 64-bit 5.5 with php version 5 and mysql 5. this os is very similar to red hat enterprise linux 5.5. sixty-four and 32 bit setups are identical. the server is a dell poweredge 2950 with a xeon e5405 2ghz and 8gb of ram. there are two ways to install joomla – manually or via a web interface. we installed joomla manually because of the additional options that it provides. the first step was to edit the configuration file. copy the sample configuration file and enter your local settings such as the site url and your database username and password. the next step was to create the mysql database that will be used with the site. (note: if you’re using the web-based interface, the database must be created prior to starting the setup.) to create the database in mysql, we used sqlyog [8] to execute the task. sqlyog is a simple mysql management tool that allows for the creation and management of the database server. once the database was created, we then opened the apache configuration file and added the site information (because we host more than one site on a single ip address, we created a virtual host). dns records then need to be created and configured by whoever manages your dns. in our case, this was done internally. once the apache configuration reloads the site becomes available. at this point, we needed to verify the file permissions. this requires a ssh connection to be open. once you have made a ssh connection to the server with an account that has the ability to change permissions, run the following commands from the joomla root directory (note: these commands work only in a *nix environment): find . -type f -exec chmod 644 {} \; find . -type d -exec chmod 755 {} \; these commands reset file and folder permissions for all files and folders in the joomla root recursively. finally, because our site was developed on a hosted server, we needed to move the mysql database and customized template files to the local server. this was done by connecting to the hosted server with sqlyog and downloading the sql database (sql dump). next, connect to the local server and replace the database. the customized template files were rsynced from the hosted site to the local site. backup a backup of the site consists of two parts: the joomla site files and the database. at alfred, we use an enterprise product called emc networker [9] to back up the joomla site files. if you don’t have an enterprise solution another option is to rsync, ftp, or sftp the data to another server for backup. this can be done with a crontab or windows scheduled task. simply copying the files to an external drive is another option. the second part of the backup involves the database. we use a cron job to do an sql dump of the database, which is then saved by the emc networker backup system. a joomla extension called akeeba backup can also be used for site backup [10]. upgrades and migrations in joomla, upgrading refers to the process of making updates within the same version of joomla (for example, updating from version 1.5.18 to 1.5.19) and migrating refers to the process of moving from one version of joomla to another (for example, migrating from version 1.0 to version 1.5). joomla updates are released on a fairly regular basis (every couple of months or so) while migrations are a much rarer occurrence (version 1.5 was released in 2008 and version 1.6 is currently in beta testing). the joomla documentation provides complete instructions for site updates and migrations. in the case of upgrades, an extension called update manager for joomla can be used to automate the process [11]. a good way to stay up-to-date with new releases is to subscribe to the announcements discussion in the joomla forums [12]. security joomla has some built-in security features that can be helpful to the person responsible for managing the server side of joomla. for example, it will let you know if the joomla configuration files or server configuration are incorrect or pose security risks. joomla recommends staying up-to-date with the most recent stable release of joomla and any additional extensions you’ve installed. as mentioned above, the announcement discussion topic in the joomla discussion forums provides a way to receive automatic notifications of new releases, as well as important security notifications. joomla’s security checklist [13] covers other important security topics. at alfred university we added an extra layer of security by putting a web application firewall (waf) in front of the server. we use an imperva x1000 waf, which inspects all of the traffic going into our web servers. it looks for potential exploits such as cross-site scripting, sql injection, and parameter violations. the firewall helps with blocking known problems and patching new problems as needed. when implementing the waf to the joomla site, the waf first needs to learn the site through web traffic. in our experience, this didn’t present any problems on the public website. the waf did, however, generate errors in joomla’s web-based administrative interface. some site changes generated false positives based on the way joomla handles its code through the /administrator/index.php and index2.php pages. errors were reported to information technology services staff and corrected by making changes to the imperva configuration. joomla’s administrative interface once joomla is installed, site administrators and developers access the site through a back-end, web-based administrative interface. figure 5 shows the control panel which functions as the homepage of the administrative interface. figure 5. joomla control panel in the control panel, a series of drop-down menus and shortcuts provide access to joomla’s administrative options. joomla’s official documentation [14] and the “learn to use joomla!” section of the absolute beginner’s guide to joomla! [15] are good resources for getting started with using some of the back-end options. some knowledge of html and css will be helpful if you want to edit template files. experience with php and mysql will open up additional opportunities for customizing the site. designing and building the site working with templates there are three basic approaches to designing a joomla site: find a template that you like and make only minor changes; find a template that approximates what you want and modify it to suit your needs; or design and build your own joomla template. while designing a template from scratch provides the most control, using an existing template will likely save time and reduce the need for in-house design and technical expertise. keep in mind that most templates will need at least some modification, even if it’s only replacing the default logo with your own. in herrick’s case, we found a template that had the right look and feel, and we modified it to suit our needs. we chose the “ja rutile” template, which can be purchased from joomlart. some sources for free and purchased templates are: joomlashack [16], joomlart [17], and rockettheme [18]. templates are installed and uninstalled through the web-based administrative interface. one of the first steps in editing templates is becoming familiar with a given template’s defined module positions. for example, figure 6 below shows the module positions of the default ja rutile template and figure 7 shows the module positions of our website. for any joomla site, you can view the module positions by adding “?tp=1” to the site url (http://herrick.alfred.edu/?tp=1 will display the module positions for our site). figure 6. module positions for the ja rutile template figure 7. module positions for the herrick homepage with an understanding of a template’s module positions you can then use the module manager to make basic changes to the front page. the module manager is where modules are created, edited, enabled or disabled, and assigned to defined template positions. for more complex modifications, the website developer will want to explore three files that work together to control the template: index.php, template.css, and templatedetails.xml. the joomla documentation is a good source of information on more advanced modifications, such as adding new module positions [19]. the table below shows the module names, types, and positions for herrick’s homepage. note that “rotating” is a new module position that wasn’t included in the ja rutile default template. module name module type module position top menu mod_mainmenu user3 search mod_search user4 library hours mod_custom left front rotating images mod_jw_sir rotating tabbed search mod_custom ja-news news mod_slick_rss right find mod_mainmenu user1 about mod_mainmenu user2 requests mod_mainmenu user5 help mod_mainmenu user6 herrick footer mod_footer footer content organization before building the site, it’s important to plan for the organization of its content. joomla uses a three-level hierarchical system, comprised of sections, categories, and articles to organize content. each section can contain many categories, and each category can contain many articles. a good resource for getting started with organizing content in joomla is the “learn joomla 1.5 fast!” tutorial [20]. it describes the steps involved in creating articles, organizing them with sections and categories, and using menus to display content in the site’s front end. useful extensions one of joomla’s strengths is the large number of free extensions developed by its user community. in comparison to drupal, joomla has fewer library-specific extensions. the joomla extensions directory includes some extensions for managing personal book collections [21] and j-car [22], a joomla extension for dspace. there is no joomla equivalent to sopac, a drupal module for integrating the library catalog. while joomla currently lacks library-specific extensions, it does have many useful extensions that can be used in developing a library website. following are some of the extensions and modules used in herrick’s joomla implementation: sourcerer – sourcerer [23] builds a lot of flexibility into the joomla site by allowing you to insert custom code (php or html including javascript and css) into articles and some other types of content. to use it, you simply put the sourcerer tags around the code in the wysiwyg editor: {source}your code{/source} wrappers – joomla’s wrapper module [24] can be used to display other web pages within the joomla site. this can be useful for bringing third-party content into the library website (serials solutions journal locator for example). scroll bars can be displayed if necessary. extplorer – this component [25] provides a web-based interface, accessed through joomla’s control panel, for managing and editing files on the web server. files can be edited in extplorer, or extplorer can be used to upload and download files. chronoforms – chronoforms [26] is one of many joomla extensions for creating web forms. jce – the default joomla wysiwyg editor is the tinymce editor. the jce editor [27], a component with its own configuration settings, is a more sophisticated editor than tinymce. googlemaps – this plugin [28] can be used to easily add maps to joomla sites. bigshot google analytics – this plugin [29] adds the google analytics tracker code across the site. solving problems big and small in the process of developing our library website with joomla, we occasionally ran into problems. fortunately, there are a variety of resources to turn to for help. the joomla documentation is a good first place to look for information. the joomla forums are the place to go to find answers to common questions, or to pose questions of your own. in our experience, the joomla user community is very supportive. the book, beginning joomla! [30], is recommended as a thorough and organized introduction to joomla. when it comes to editing templates, many template providers provide documentation and a forum where users can share information. there are a variety of ways to become further involved in the joomla community such as participating in forums and listservs or attending events like joomla! days [31]. the joomla in libraries [32] group, while not as active as the core joomla community, provides a place to share information about using joomla in a library setting. those who want a more systematic approach to learning joomla might investigate formal training opportunities. what’s not to like? while joomla has mostly met our library’s needs, we did discover some downsides: in our experience, there isn’t an easy way to create production and testing environments. our process will likely involve two joomla installations, and a scripted or manual process for dumping the mysql database from one site to the other. while joomla provided enough flexibility for our site, developers with strong coding skills, more ambitious goals, or a desire to control all aspects of the site might be happier with drupal, which provides a more open-ended framework. despite the large number of joomla templates available, in our experience it can still be difficult to find a really good template. many joomla templates have a similar look and feel about them. it would be great to have some joomla templates designed specifically for libraries. staffing and project timeline our project was carried out for the most part by three people: a systems librarian, a library assistant for web services and digitization initiatives, and a network services support specialist in the university’s information technology services department. teamwork was important since this was our first experience with joomla. we began to think about redesigning our website during the 2008-2009 academic year when we conducted a survey to gather input from library users. in the fall of 2009, we inventoried the content of our existing website in preparation for a complete reorganization. data from google analytics, which we had been collecting for about two years, were used to identify the most-used content. beginning in the fall of 2009, the library assistant for web services led the way in exploring drupal and joomla, while the systems librarian focused on navigation and content. because our existing site was about ten years old, the content needed reorganizing, editing, and in some cases rewriting. once we made a decision to use joomla, we selected a template and started to develop the homepage, with the goal of finalizing the homepage design and navigation before building lower-level pages. basic homepage mock-ups were used to gather feedback from faculty, students, and librarians. building the site was an iterative process, with frequent meetings and discussions between the systems librarian and the library assistant for web services. the rest of the library staff and the library director were informed and consulted at key points along the way. we completed the transition to our new site during the summer of 2010. because none of the staff worked on the website project exclusively, it’s difficult to estimate the total amount of time that the project required. during the 2009-2010 academic year, when the bulk of the work was completed, it’s estimated that the library assistant spent about 50% of his time on the project, the systems librarian spent about 20% of her time, and the network services support specialist spent about three days in total. depending on the library’s goals and staffing arrangement, a similar project could be completed in a shorter timeframe. for us, the most time-consuming parts of the project were investigating cms options and deciding to use joomla, learning how to use joomla, preparing the content and developing the navigation, and customizing the template. once these steps were completed, building the site went relatively quickly. measuring our success when we set out to redesign our library’s website, we identified the following goals, which have been mostly met by the redesign: a more modern design the ability to easily change site navigation better compliance with changing web standards cross-browser compatibility better separation of design and content so that future design changes can be more easily accommodated secure forms the ability for non-technical staff to edit or contribute content the project was also driven by an overarching need to “do more with less” and to design a site that would be easier to manage going forward. in redesigning the site, we eliminated two homegrown databases, one for displaying our research by subject guides and a-z databases list (these now reside in libguides) and the other for searching and browsing the library’s movie collection (patrons are now directed to the library catalog and static urls are used to display movie lists by genre). since moving to the new site, we have been watching our site analytics to see what impact the new site has had on use. a comparison of some site metrics for the period of september 1 through november 13, in 2009 and 2010, appears below. metric sept. 1nov. 13, 2009 sept. 1 – nov. 13, 2010 percent change visits 76,916 65,654 -14.64% page views 179,963 165,359 -8.12% pages/visit 2.34 2.52 7.65% bounce rate 68.00% 64.82% -4.68% average time on site 00:02:10 00:03:01 39.26% percent new visits 30.74% 32.31% 5.12% since there are some important differences between the old site and the new site (for example, the new site has fewer pages overall and a much flatter hierarchy) it’s somewhat difficult to make direct comparisons. and, without analytics from other libraries to compare to, the best we can do is to examine our own statistics over time. while it’s disappointing to see that overall visits and page views are down, it’s difficult to know the reason for the change; it could be that the library website is becoming less essential to research tasks than it used to be. on the other hand, it’s nice to see that other statistics have improved, including a slight reduction in the bounce rate, a significant increase in the amount of time spent on the site, and an overall increase in new visits. while we have not conducted any task-based user testing, anecdotal feedback from students about the new website has been positive. is joomla “just right” for your library? the question of whether joomla is right for any particular library depends to some extent on the library’s needs and the web developer’s skills and expectations. in our case, joomla provided a good match for our relatively modest needs, our staffing levels, and the skills of our librarians and staff. while library staff had some experience with coding, we were also willing to give up some customization in order to reduce the amount of time required to build the site and to lessen the overall demands of site management going forward. we chose a content management system because we anticipated that once the site was built it would be easier to manage, giving the systems librarian and web development assistant an opportunity to focus on other important projects. support from the university’s its department made it possible to host the site locally. in our opinion, while many libraries are turning to drupal, joomla will be a better fit for many smalland medium-sized libraries. in any case, getting hands-on experience with a variety of open source cms’s is highly recommended, as it will allow you to make an informed choice. as previously mentioned in this paper, commercial hosting services provide a low-investment way of trying open source content management systems. and, for those libraries that don’t want the responsibility of hosting the site locally, a hosted site can be a long-term option as well. lishost [33], for example, provides low-cost web hosting services specifically for libraries. in a recent email, they indicated that they currently host two joomla sites, about 80 drupal sites, and at least 100 wordpress sites (email from blake carver of lishost to the author, 2010 nov 11). when we set out to redesign our site, it was difficult to find information about using joomla in libraries. we hope, therefore, that the information presented in this paper will be helpful to other libraries considering joomla. bibliography [1] framework [internet]. [updated 2010 may 18]. joomla! documentation: [cited 2010 nov 11]. available from: http://docs.joomla.org/framework [2] joomla! features overview [internet]. c2005-2010. open source matters, inc.: [cited 2010 nov 11]. available from: http://www.joomla.org/core-features.html [3] joomla! extensions [internet]. c2005-2010. open source matters, inc.: [cited 2010 nov 11]. available from: http://extensions.joomla.org/ [4] joomla! extensions defined [internet]. [updated 2010 may 9]. joomla documentation: [cited 2010 nov 11]. available from: http://docs.joomla.org/joomla_extensions_defined [5] mort g. 2010. joomla vs drupal, a coder’s perspective [internet]. nyphp joomla sig: [cited 2010 dec 9]. available from: http://lists.nyphp.org/pipermail/joomla/2010-july/002895.html [6] joomla! resources directory [internet]. c2005-2010. open source matters, inc.: [cited 2010 nov 11]. available from: http://resources.joomla.org/directory/support-services/hosting.html [7] wallace a. 2007. joomla! 1.5 installation manual, version 0.5 [internet]. [updated 2007 oct 30]. joomla! user documentation team: [cited 2010 nov 11]. available from: http://downloads.joomlacode.org/docmanfileversion/1/7/4/17471/1.5_installation_manual_version_0.5.pdf [8] sqlyog [internet]. c2010. webyog: [cited 2010 nov 11]. available from: http://www.webyog.com/en/ [9] emc networker unified backup and recovery software. [internet]. c2010. emc corporation: [cited 2010 nov 11]. available from: http://www.emc.com/domains/legato/index.htm [10] akeeba backup [internet]. c2006-2010. akeeba developers: [cited 2010 nov 11]. available from: http://www.akeebabackup.com/ [11] update manager for joomla! [internet]. [n.d.] sam moffatt consulting: [cited 2010 nov 11]. available from: http://sammoffatt.com.au/os/joomla-15-products/3-jupdateman [12] joomla! discussions forums [internet]. c2005-2010. open source matters, inc.: [cited 2010 nov 11]. available from: http://forum.joomla.org/ [13] joomla! security checklist [internet]. [updated 2010 mar 10]. joomla! documentation: [cited 2010 nov 11]. available from: http://docs.joomla.org/category:security_checklist [14] joomla! official documentation [internet]. [updated 2010 oct 29]. joomla documentation: [cited 2010 nov 11]. available from: http://docs.joomla.org/ [15] absolute beginner’s guide to joomla! [internet]. [updated 2010 may 10]. joomla! documentation: [cited 2010 nov 11]. available from: http://docs.joomla.org/beginners [16] joomlashack [internet]. c2005-2010. joomlashack: [cited 2010 nov 11]. available from: http://www.joomlashack.com/ [17] joomlart [internet]. c2005-2010. joomlart.com: [cited 2010 nov 11]. available from: http://www.joomlart.com/ [18] rockettheme [internet]. c2010. rockettheme, llc: [cited 2010 nov 11]. available from: http://www.rockettheme.com/ [19] how do you add a new module position? [internet]. [updated 2010 sept 29]. joomla! documentation: [cited 2010 nov 11]. available from: http://docs.joomla.org/how_do_you_add_a_new_template_position%3f [20] bhide sr. [n.d.]. learn joomla! 1.5 fast! [internet]. [cited 2010 nov 11]. available from: http://help.joomla.org/files/visualguide15.pdf [21] books and libraries [internet]. c2005-2010. open source matters, inc.: [cited 2010 nov 11]. available from: http://extensions.joomla.org/extensions/living/education-a-culture/books-a-libraries [22] j-car [internet]. c2008-2010. wijiti pty ltd.: [cited 2010 nov 11]. available from: http://www.wijiti.com/projects/j-car [23] sourcerer [internet]. c2010. nonumber!: [cited 2010 dec 9]. available from: http://www.nonumber.nl/extensions/sourcerer [24] help16: extensions module manager wrapper [internet]. [updated 2010 jul 11]. joomla! documentation: [cited 2010 dec 9]. available from: http://docs.joomla.org/help16:extensions_module_manager_wrapper [25] extplorer [internet]. 2007 [updated 2010 jun 10]. soeren eberhardt: [cited 2010 dec 10]. available from: http://extplorer.sourceforge.net/ [26] what is chronoforms? [internet]. [n.d.]. chronoengine: [cited 2010 dec 10]. available from: http://www.chronoengine.com/component/content/article/1-latest/26-what-is-chronoforms.html [27] jce [internet]. c2010. joomlacontenteditor.net: [cited 2010 dec 10]. available from: http://www.joomlacontenteditor.net/ [28] plugin googlemap [internet]. c2005-2008 [updated 2010 jul 17]. open source matters, inc.: [cited 2010 dec 10]. available from: http://joomlacode.org/gf/project/mambot_google1/ [29] bigshot google analytics [internet]. [n.d.]. bigshot: [cited 2010 dec 10]. available from: http://www.thinkbigshot.com/kansas-city-marketing-services/205-free-joomla-extensions.html [30] rahmel d. 2007. beginning joomla! from novice to professional. berkeley (ca): apress. [31] joomla! days worldwide [internet]. c2005-2010. open source matters, inc.: [cited 2010 nov 11]. available from: http://community.joomla.org/events/joomla-days.html [32] joomla in libraries [internet]. [n.d.]. [cited 2010 nov 11]. available from: http://www.joomlainlibrary.com/ [33] lishost [internet]. [n.d.]. [cited 2010 nov 11]. available from: http://lishost.org/ authors ellen bahr is information systems librarian for herrick memorial library at alfred university in alfred, ny. email: bahr@alfred.edu. matt speed is network services support specialist for the information technology service department at alfred university in alfred, ny. email: speed@alfred.edu. acknowledgements ellen bahr would like to recognize brett arno’s contributions to the herrick library website and thank him for commenting on drafts of this paper. she would also like to thank kathryn frederick of the lucy scribner library at skidmore college for sharing her joomla experience with us. subscribe to comments: for this article | for all articles 7 responses to "creating library websites with joomla: not too big, not too small, just right" please leave a response below: noah, 2011-09-20 thanks for this article. very helpful. do you have any recommendations for integrating a library catalog into a joomla site? any advice would be much appreciated. thanks ellen, 2011-10-18 sorry for the delay – i just noticed that you left a comment here. this is the only joomla library catalog extension that i’m aware of: http://sourceforge.net/projects/obiblioopac4j/ there has been some discussion about this topic in the joomla in libraries forum: http://www.joomlainlibrary.com/index.php?option=com_kunena&func=view&catid=9&id=334&limit=6 in our case, we built some opac search boxes into our joomla site (this is easy to do) and we just link out to the library catalog. as far as i know, there isn’t much happening in the way of developing library catalog extensions for joomla in the same way that modules are being developed for drupal. so, if catalog integration is important to you, you may want to look at drupal. ellen noah, 2011-10-21 thanks ellen arthur, 2012-02-16 hello, can anyone share joomforest’s jf chrome template? it is here: http://www.joomforest.com/, please, someone share it, thank you build it, but will they come? | rocky mountain chapter (rmsla), 2012-03-27 […] might manage a website using drupal (http://groups.drupal.org/libraries), joomla (http://journal.code4lib.org/articles/4226) or some other content management system, but smaller organizations might find software […] glenn, 2014-06-27 ellen, in your article, you alluded to the lengthy learning curve associated with drupal (i agree); this may make joomla more favorable, but i didn’t see any specifics on why wordpress was not given serious consideration as your library’s site management platform. was wordpress security a major issue in 2009? philipp holz, 2014-08-06 for a new projects i also need a library functionality within my joomla site. do you have any suggestions for extensions which would be good a start for joomla 2.5 or 3.x? i have seen that the herrick library already uses joomla 3.x. do you use a component to manage these books or are they all separated articles? thanks in advance. leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – collecting virtual reference statistics with an im chat-bot mission editorial committee process and structure code4lib issue 3, 2008-06-23 collecting virtual reference statistics with an im chat-bot a perennial problem in libraries is capturing accurate statistics. this article addresses this problem with the creative use of web 2.0 tools: meebo and aol instant messenger. it describes the development and implementation of an instant messaging “stat-bot” that prompts staff to record virtual reference statistics via im. step-by-step guidelines and the perl script are provided. by mason r.k. hall introduction over the past several years, virtual reference has become a popular and valuable point of contact between patrons and libraries. with new web-based technologies like meebo [1], librarians can now be available for virtual reference anywhere they have internet access. because of this anywhere-access factor, keeping track of statistics from multiple employees located in multiple places is a problem. e-mail and manual statistics collection and compilation are successful only if the employee remembers to compile and send them after each session. unfortunately, our employees (myself included) often forget to perform this crucial step after virtual reference. at our institution, virtual reference is usually performed as a secondary task (usually in addition to the employees’ primary duties), and can be easily forgotten about if no questions are received for a length of time. how could we prompt employees to record virtual reference statistics? after weighing our options, we agreed that the best solution to this problem would be to create a system that actively pulled statistics from the employee each hour. the system would have the ability to query staff members for statistics, and receive the information as an instant message. creating the “statbot” removed the problem of statistics collection from both the supervisors and the employees and has provided a consistent and accurate stream of quantitative data from our virtual reference system [2]. this article will explain how to set up and use the statbot in your own library using the provided perl script. requirements for this project you’ll need command line access to a linux server with perl >= 5.8.5, and the ability to add a cron entry [3]. for this particular script, we’ll be using aol’s instant messaging network for sending and receiving statistics messages. this means whoever is manning your virtual reference service will need to be logged onto aol im. many libraries now use meebo to connect to multiple services at once [4]. this allows many users on many different chat systems to access that library’s virtual reference. for this project (and for virtual reference in general) i recommend using meebo. setup 1. create statbot account first, you’ll need to create your statbot account in aol instant messaging. aol provides a special account for bots, which you can sign up for here: http://dev.aol.com/aim/bot. a. click on the “bot my screenname” link under “what do i need to get started?” b. on the right side of the screen, under “screen name or e-mail” click on the “get an account” link. c. fill in all the appropriate information. i recommend documenting the security question in case the password is ever forgotten. the screenname can be anything, but should be something your employees can readily associate with the statbot e.g. fsustatbot, mylibstatbot, etc. d. once submitted, you’ll be asked to agree to aol’s terms of service. you must agree to continue. important: you’ll need to add this screenname to your virtual reference’s aol “buddy list” so that the statbot can talk to employees using it. aol doesn’t allow bots to initiate conversations without the bot being on that user’s buddylist. 2. put statbot script on server create or choose a directory on your server and place the statbot.pl script inside of it. from inside the directory, create a folder for the statlog by issuing the following command: “touch statlog.txt” 3. customize statbot script open the statbot.pl script in your favorite text editor. you’ll need to modify a couple of lines to customize your instance. make sure to only change the information within the single quotes. my $screenname = ‘statbotscreenname’; change this to the screenname you just created for the statbot in step 1. my $password = ‘statbotpassword’; change this to the password you created for the statbot in step 1. my $vrscreenname = ‘virtualreferenceaolscreenname’; change this to your library’s virtual reference (vr) aol screenname. my $timetoquery = ’52’; change this to the minute on the hour you would like the statbot to query vr for stats. if your vr staff changes at the end of each hour, it should ask right before the shift change (like the example). my $finishhour = ’22’; change this to the hour (in 24 hour time) that you would like the statbot to finish for the day and send its report. as is, statbot would compile and send all statistics, then shut-down at 10:00 p.m. my $statcollector = ‘sombodys@email.edu’; change this to the email address you would like the end-of-day statistics report sent to each night. my $statcollectorsmtp = ‘mail.smtp.edu’; change this to your local smtp server address. my $statlogloc = ‘/var/www/statbot’; change this to the directory on the server you have placed statbot inside of in step 2. 4. setup schedule for statbot you’ll want to set these variables to correspond to the times you would like the statbot to come online and go offline each day. use 24 hour time. if you don’t want statbot to come online that day use ‘99′ for both the on and off times. my $monon = ’12’ ; my $monoff = ’20’ ; my $tueon = ’12’ ; my $tueoff = ’20’ ; my $wedon = ’12’ ; my $wedoff = ’20’ ; my $thuon = ’99’ ; my $thuoff = ’99’ ; my $frion = ’12’ ; my $frioff = ’18’ ; my $saton = ’99’ ; my $satoff = ’99’ ; my $sunon = ’12’ ; my $sunoff = ’17’ ; this code was originally created for a virtual reference service that operated a limited number of hours each day. if your institution uses 24 hour vr service, set the on time as ‘0′ and the off time as ‘24.’ [5] 5. add a crontab entry the crontab is typically located in the /etc directory, but may vary depending on your setup. you should be able to find it by issuing a “whereis crontab” command at the command line. this is the recommended crontab entry. make sure to change the path to your statbot script and the path to perl if different from the default. 30 0 * * * root /usr/bin/perl /path/to/your/statbot.pl 6. install needed perl modules the script requires two perl modules. to install these, issue the following commands at the command line: perl -mcpan -e shell install net::oscar (answer ‘y’es to any required dependencies) install mail::sender (answer ‘y’es to any required dependencies) how it works once all the steps above are done, you should have a fully-functioning statistics-capturing im bot. while the script itself will be run once per day by the crontab, the bot will only show up online during the times you scheduled in step 4. each hour, the statbot will then send an im to the staff member who is on virtual reference, asking for the number of questions they have fielded that hour. the user then has 5 minutes to respond and relay their statistics to the statbot, who will then record those statistics in a daily flat file (text file). in the case that the employee does not respond, or is not logged into the library’s virtual reference screenname, the statbot will send an email to the supervisor stating the shift which failed to report. the statbot will also record this error in the text file. at the end of the day, the script will shut itself down and calculate the total and average number of questions for the day, and send all of those statistics as an email attachment to the stat collector (defined in $statcollector above). example of flat file output: 12am 0 1am 0 2am 0 3am 0 4am 0 5am 0 6am 0 7am 0 8am 0 9am 0 nl 10am 0 nr 11am 1 12pm 1 1pm 1 2pm 0 3pm 1 4pm 1 5pm 0 6pm 2 7pm 1 8pm 0 9pm 1 10pm 0 nr 11pm 0 total 9 average 0.375 there are two error codes which may be found next to some entries: nl: the staff member was not logged into virtual reference at this time. nr: the staff member did not respond to the statbot when asked for hourly stats. there are two other options that are available while the statbot is online: check stats for the day if you would like to see the total number of questions for the day (up to the current hour) instant message the statbot with ‘adminstats’ (no quotes). shut down statbot if the statbot is malfunctioning, you can shut it down prematurely by sending an instant message that reads ‘gotobednow’ (no quotes). conclusion the statbot code has made recording statistics for virtual reference a simpler process. the granularity of hourly reports has made informed staffing changes possible, and we can now present what we believe are more accurate statistics for yearly reports (like the association of research libraries report). previous collection methods tended to be far less immediate for the employees involved. because statistics were collected monthly from employees, “estimates” of the outcomes of the previous four weeks of vr sessions were often provided instead of exact statistics. the statbot script presents a unique framework for future statistical projects. expanding this system could result in a larger statistics collection process which could pull many different statistics from many different areas in the library. this “automatic” stat collection is not limited to quantitative data either–all types of information could be collected, sorted, analyzed and finally filtered to the necessary entities. code see attachment notes [1] see www.meebo.com. an online chat application that provides one interface for many different instant messaging services. [2] chat-bots have been in use since the early 1990’s, and have varying uses and levels of complexity: http://en.wikipedia.org/wiki/chat_bot. “stat-bot” is a very simple form of this technology, designed for the singular purpose of collecting statistics. [3] if you’re not familiar with cron or how to set up a cron entry, see http://www.unixgeeks.org/security/newbie/unix/cron-1.html for a fairly straightforward overview and introduction [4] meebo currently supports aim, yahoo!, google talk, msn, icq, and jabber [5] you may ask why statbot isn’t just on constantly. when initially testing the script, there were some minor security worries, which were allayed by only having statbot online when virtual reference was online. these “security issues” are now moot, but we find that this setup still provides the least maintenance. if the statbot or the aol network dies unexpectedly, it will be restarted the next day with the crontab, regardless. about the author mason r.k. hall is the electronic resource integration management librarian at florida state university. he currently holds the record for the longest job title in his library. his email address is mrhall@fsu.edu. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – building cyprusark a web content management system for small museums collections online mission editorial committee process and structure code4lib issue 54, 2022-08-29 building cyprusark a web content management system for small museums collections online this article introduces cyprusark, a work-in-progress solution to the problems that small museums in cyprus have in providing online access to their collections. cyprusark is an open-source web content management system for small museums’ online collections. developed as part of avgousti’s ph.d. thesis, based on qualitative data collected from six small museums in cyprus. by avgoustinos avgousti, georgios papaioannou, and feliz ribeiro gouveia 1. introduction cultural heritage digitization and online accessibility offer an unprecedented opportunity to democratize museum collections. museum collections online are usually presented via institutional websites, and they mirror the world’s culture and civilization. they form an increasing trend toward a world where information is digitally preserved, stored, accessed, and disseminated instantaneously through a global and concatenated digital network. however, while the web has enabled cultural heritage organizations to democratize information to diverse audiences, many small museums do not enjoy the fruits of this digital revolution. the majority of small museums are underfunded, understaffed, and lack technology infrastructures (oecd, 2020). as a result, many small museums cannot publish their collections online. due to the problems above, digital versions of small museum collections are mainly inaccessible and hidden. subsequently, humans have less access to information that can lead to new knowledge. 2. defining scales 2.1 what is a small museum museums are defined as small based on their annual budget, number of staff, collections size and physical size of the museum. the american association for state and local history 2007 (aaslh) defines museums as small if they have a yearly budget of fewer than 250,000 dollars and limited staff with multiple responsibilities. other factors such as the size of collections and the physical size of the museum could further categorize a museum as small. katz (1995) set the same budget and the staff number at five or less. honeysett and falkowski (2018) put the budget at 300,000 dollars and five or less employees. miller (2008) notes that the average small museum has just two full-time employees and a budget of less than 90,000 dollars. watson (2007) by contrast, defines small museums as ones that grew out of the community they serve. for the purposes of this article, a small museum is one with more than one but less than five full-time employees, not including museum custodians. categorizing a museum based on its budget is difficult and contentious since often museum staff are funded by another body, such as a municipality. 2.2 what is cyprusark cyprusark (github) is proposed as a solution to the challenges small museums in cyprus face in providing online access to their collections. cyprusark is an open-source web content management system for small museums’ online collections developed as part of avgousti ph.d thesis. it follows the kiss principle, “keep it simple, stupid”. according to the kiss principle, most systems work best when they are simple rather than complex in maintenance (u.s. navy, 1960). designed primarily to meet the needs of cyprus’s small museums, cyprusark can be used by other small museums all over the world. cyprusark was developed based on qualitative research of six small museums in cyprus using semi-structured interviews. cyprusark has been created using an open-source stack consisting of the leading-edge django, a high-level python web framework, postgres database, bootstrap, an open-source css framework, and running on virtualized docker containers. instances of cyprusark can be hosted on any infrastructure selected by the museum. cyprusark differs from other comparable systems, as described below, because it focuses on small museums and most specifically small museums in cyprus. 3. related works the mukurtu content management system is an open-source platform dedicated to meeting the needs of diverse communities through the subsequent sharing of their digital cultural heritage online (humanities for all). the platform was developed on the leading open source content management system drupal, and its advert has assisted substantially in the field of extending the functionality of the content management system towards the critical issue of preserving and disseminating the knowledge derived from cultural heritage. another similar system is the vimuseu content management system, which constitutes a project dedicated to the needs of small finnish museums and has been developed by the department of arts and cultural studies at the university of jyväskylä in finland in collaboration with the university’s museum. the vimuseo cms can be regarded as an online multimedia presentation tool mainly used for virtual exhibitions and projects. the vimuseu content management system’s utility is related to digital-free institutions that wish to be more active on the internet. this tool can be described as a web application content management system (web cms) co-developed by programmers and graphic designers. additionally, a similar project carried out by the cyprus institute (cyi) and the openumisma content management system avgousti et al. (2017). the opennumisma content management system can be perceived as a reusable web-based platform which consists of a merge of digital imaging and content that offers tremendous opportunities for research, as well as for the dissemination of knowledge and data. the platform’s development seeks to create a digital web framework concerning online numismatic collections. this kind of content management system has been implemented and used by small numismatic museums in cyprus. daradimos et al. (2015) presented the significant problem of small museums that are considered unable to reach the public through the web due to the cost of commercial software and the complexity of the development and customization of such software. in his article, he recommends a different approach, along with developing a general-purpose plugin (module) that will extend the functionality of the open-source drupal cms for small museum collections online. a similar idea was also proposed by (avgousti, papaioannou, and gouveia, 2019), a collection of reusable drupal modules aimed at small-scale museums and archival online collections. the getty foundation online scholarly catalog initiative (osci) is helping museums make web-based publications available to the public with the use of technology and the internet (marrow, 2017). the development of chicagocodex can be regarded as the basis for this tool since this kind of toolkit leaves the margin for institutions to deploy a microsite’s digital collections. however, based on marrow (2017), the parameters relative to the toolkit’s complexity combined with its augmented cost on account of its technical expert requirements can be regarded as certain of its disadvantages. accordingly, the wisski a content management system platform can be regarded as one other illustration that is aimed at the production of a virtual research environment for the digital humanities (scholz, 2012). the platform is based on the open-source content management system drupal and requires installing approximately thirty contributed plugins to implement the system. the default installation of the software, in same cases, may lead to difficulties with upgrading to new versions of the system because of its heavy dependence on many other modules (velios, 2017). in line with the above, the omeka, a content management system developed specifically for scholars, librarians, archives, and museums, targets the creation of digital collections. it uses the dublin core for storing the metadata of the objects. moreover, the omeka “s” is considered more oriented towards connecting digital cultural heritage collections through the consequential usage of linked open data principles (ontotext, n.a ). hardesty (2014) mentioned that omeka had been used in various memory institutions (flynn, 2018). although, compared to large and robust ecosystems available for other large content management systems such as wordpress, joomla and drupal dombrowski (2016) argues that omeka can be described as a fairly moderate cms with minor community support, plugins, developers, and users. lastly, the glamkit content management system, or web framework, is designed specifically for glam sector organizations (galleries, libraries, archives and museums) (weakley et al., 2009). it has tools for managing and displaying collections online and complex events; it can be focused on small institutions with limited technical staff (mansfield, 2017). glamkit is developed using the django framework and the wagtail content management system focusing on common museum needs (mansfield, 2020). 4. cyprusark architecture 4.1 content model in this section, we discuss the high-level development and design of the cyprusark prototype, using the django framework. currently, the cyprusark prototype has basic functionality. keeping the design simple will help with future testing of the technology to explore how useful and applicable it is within the small museums’ domain. the cyprusark content model consists of four content types: place (e.g., museum), bundle (e.g., collection), creative work (e.g., item), and a maker (e.g., person). cyprusark content types draw inspiration from the object id, dublin core, and the schema.org field recommendations, for modeling the information and creating the necessary content types. figure 1. cyprusark conceptual content model 4.2 place the first content type is the place, which can be used to make a landing page using cyprusark. the place content type includes fifteen fields that cover most of the information that describes a place (e.g. museum). we attempted to maintain the field titles as straightforward as possible to help structure these fields with rich semantic information for future development. the table below depicts the content type place along with its fields and expected attribute types, as well as a brief description of each field. attribute/field name attribute type description name text the name of the museum (e.g., pierides museum). classification drop down list museums, etc. working hours text the time that the museum is open to the public. address text the address of the place. city text the city where the place is located (e.g. athens, nicosia). country text the country where the place is located (e.g. cyprus, greece). image image a photo(s) of the place. phone integer the telephone number of the place (e.g., 22 55667788). logo image the logo of the place. description long text short description of the place. same as link an external link (e.g., wiki page, fb page). website link an external link to the place website if it exists. order integer the order in which the content will be displayed. online boolean field if selected, the content will be shown on the template front end, if not the content will live only on the backend. slug text is the url that the place will get in cyprusark. the slug field will be auto-generated by using the place name field table 1. place content-type (e.g., museum) 4.3 bundle the bundle content type is the second content type. the bundle content type describes a “museum collection(s)”. an example of the content bundle is: “terracotta collection from salamis cyprus”. a place is related to one or multiple bundles. for example, a museum is a place and the museum collection(s) are bundles of that place. the following image shows this type of relation. figure 2. bundle(s) relationship to a place it consists of eight fields to cover most of the bundle (e.g. collection) information. the table below depicts the content type bundle along with its fields and expected attribute types, as well as a brief description of each field. attribute/field name attribute type description name text the name of the bundle (e.g. portraits collection). size integer auto-generated, based on the creative works related to the bundle. photo imageobject an image representing this bundle. description long text a description of the bundle. what is it about? same as url url of a reference webpage. online boolean field if selected, the content will be shown on the template front end, otherwise the content will live only on the backend. slug url the url of the bundle. order integer the order in which the content will be displayed. table 2. bundle content-type 4.4 creative work the third content type is the creative work content type. the creative work content type can be used to create museum item(s) (e.g. paintings, sculptures). it consists of twenty two fields to cover most of the information that can be used to publish a creative work on the web. it is not necessary to use all those fields. the creative work content type is related to the place, and the bundle content types. the table below depicts the content type of creative work fields and expected attribute types, as well as a brief description of each field. attribute/field name attribute type description art form select list the art form of a creative work (painting, drawing, sculpture, photograph). art medium select list the material used (oil, pastel, pencil). art work surface select list the supporting materials for the artwork (canvas, stone, paper, wood). additional type url or text any additional type (icon, coin) linked to an external vocabulary such as getty aat controlled vocabulary. title text the object title by which it may be known and can be identified (e.g. mona lisa). date created text when the object was made (e.g. 1650). description long text description of the object. image image object image(s). dimensions text or number the size and/or weight of the object. credit line text example: gift of j. pierpont morgan, 1917. accession number text and or number identification of the object (e.g. number 1254a124d). sameas url url to the same item (e.g. wiki, db media). culture text the culture of the item (e.g. cypriot, american, arabic). example period text the period of the item (e.g. late bronze age). rights of reproduction text e.g. public domain. citation text citation or reference to scholarly articles, web page etc. slug text the url of the item that can become a uri. order integer the order in which the content it will be displayed. bundle reference the bundle (collection) in which the creative work belongs. makertype reference the role of a maker of this creative work (e.g. creator, editor). example a maker can be a creator in creative work one, at the same time can be the author in creative work two. maker reference the maker content type. online select boxes show on the front-end. table 3. creativework content-type a place is related to a bundle(s), and a bundle is related to creative work(s). for example, a bundle is a collection of creative works that belongs to a place. the image below shows the relationship between the place, bundle, and creative work. figure 3. place, bundle(s) and creative work(s) relationship 4.5 maker the maker is the last content type. the maker content type consists of twelve fields to cover most information that can be used to describe a maker (person). a maker can be an object’s creator, artist, penciler, editor, publisher, etc. the maker content type is related or not related to the creative work content type. for example, creative work (mona lisa) is related to a maker that is the creator (leonardo da vinci ). the image below shows the relationship between the creative work and the maker concerning the rest of the content model. figure 4. place, bundle(s), creative work(s) and maker(s) relationship attribute/field name attribute type description first name text the name of the maker. last name text the last name of the maker. gender text the gender of the maker. image image a photo of the maker. date of birth date date of birth of the maker. birth place date the place where the maker was born (e.g., athens). date of death date if the maker is dead, the date of death. death place date if the maker is dead, the place of death (e.g. nicosia). same as external link an external link (e.g. wiki page). example: https://dbpedia.org/page/leonardo_da_vinci table 4. maker content-type 5. collections online creation process in this section, we will describe the cyprusark back-end used by a login user for content creation, and how content will appear to the front user. we used data from the pattichion municipal museum to demonstrate this exercise. the first step is for a back-end user to log into cyprusark, using the username and password provided by the administrator. when the user is logged in, it will be auto-redirected to the cyprusark dashboard. all login users have specific permissions; they can create, retrieve, update and delete (crud) content. however, the administrator of cyprusark can give custom permissions to individuals or groups of individuals (e.g. museum curators can only add content). 5.1 adding a place the login user can create, retrieve, update and delete (crud) a place using the place content type web form. first, the user must fill out the web form consisting of the name, and subject (e.g. pierides museum). after filling out the fields, the place content will be created and appear on the cyprusark admin dashboard. it is important to note that the user can add only a single place. the following table shows how the login user crud functionality. the user can decide whether to have the place online or not. we added this functionality to give the user the option to have the content offline until it is fully completed and ready to be displayed online. the following image represents a content store in the system backend. figure 5. print screen place backend (click to enlarge) when the online checkbox is selected, the content will appear on the front end. 5.2 adding a bundle the second step is to create a bundle (e.g. collection(s)). in order to create a bundle, the user should complete some important fields such as the name, image description and so forth. after filling out the fields, the bundle content will be created and appear on the cyprusark admin dashboard. further, the login user can select in which order the content will be displayed. the following image represents three bundles,only two displayed on the front end. the following image represents bundles that are stored in the system. figure 6. print screen (click to enlarge) when the online checkbox is selected, the content will appear on the front end. the following image shows how the place and the two bundles display on the html page. figure 7. print screen bundle(s) front-end (click to enlarge) 5.3 adding a creative work the next step is to create a creative work(s) (museum items), using the creative work content type web form, and assign the creative work to a bundle. the same creative work can belong to a single or multiple bundles. figure 8. print screen creative work(s) management dashboard (click to enlarge) from this panel the login user can make direct changes to any creative work without visiting the detailed page of each item. further, they can filter down the information using the right menu or the search bar on the top. if the online checkbox is selected, the creative work will be displayed on the html front end page. the creative work uses two html templates. the first template is a list of all creative works in relation to a bundle (collection of items) and the second template is a detailed page of each creative work. the end-user can manipulate and filter the information using the drop down menu on the top of the page. figure 9. list of creative works that belong to a bundle (click to enlarge) at this point, a login user can create a maker (person) and relate a maker to creative work. if the item is not related to any maker simply the maker will not be displayed on the html page. 5.4 adding a maker (pearson) the maker is our last content type, and the same process is needed to crud a maker. the maker may be related to the creative work. if the creative work is related to a maker, the maker will show up on the screen. the same maker can have different roles on different creative works. for example, in creative work, one is the creator, and in creative work, two is the author. figure 10. detail view of creative works with a maker relation (click to enlarge) 5.5 adding slides the login user can also change the slider by adding customized images representing their own institution. furthermore, they can link the slider image to a specific page, which could be internal or external. the user can add as many slides as they like and can change the slider order (what is first, second, etc. ). figure 11. updated slider (click to enlarge) 6. conclusion and future steps in this article, we presented an overview of the cyprusark high-fidelity prototype architecture, and we then moved to the in-depth details of this prototype. we discussed the content types and the related fields used and added to cyprusark, presenting the content relationships in a detailed content model. further, we continued with the actual presentation of cyprusark. we explained how it works and the process of creating place, bundles, creative work, and makers, as well as presenting the back and front-end. future steps will include the implementation of the semantic markup based on schema.org vocabulary or ontology. moreover, we plan to test the cyprusark prototype and explore how useful and applicable it is within the small museums’ domain in cyprus. based on the results, we will add/change the functionality of cyprusark to better fit small museums’ needs. references aaslh. “what is the definition of a small museum? survey results,” 2007. http://download.aaslh.org/small+museums/small+museum+survey+results.pdf. avgousti, avgoustinos, georgios papaioannou, and feliz ribeiro gouveia. “content dissemination from small-scale museum and archival collections: community reusable semantic metadata content models for digital humanities.” the code4lib journal, no. 43 (february 14, 2019). https://journal.code4lib.org/articles/14054. avgousti, avgoustinos, andriana nikolaidou, and ropertos georgiou. “openumisma: a software platform managing numismatic collections with a particular focus on reflectance transformation imaging.” the code4lib journal, no. 37 (2017). https://journal.code4lib.org/articles/12627. daradimos, illias, costas vassilakis, and akrivi katifori. “a drupal cms module for managing museum collections,” 2015. https://www.academia.edu/29947679/a_drupal_cms_module_for_managing_museum_collectio dombrowski, quinn. drupal for humanists. texas a&m university press, 2016. https://drupal.forhumanists.org/book. flynn, bernadette. “making collections accessible.” federation of australian historical societies inc, 2018. https://www.history.org.au/wp-content/uploads/2018/10/makingcollectionsaccessible.pdf. humanities for all. “mukurtu cms: an indigenous archive and publishing tool.” accessed august 9, 2022. https://humanitiesforall.org/projects/mukurtu-an-indigenous-archive-and-publishing-tool. shepard, michael. “review of mukurtu content management system.” language documentation & conservation, september 2014. http://scholarspace.manoa.hawaii.edu/handle/10125/24610. katz, paul. “the quandaries of the small museum.” the journal of museum education 20, no. 1 (1995): 15–17. https://www.jstor.org/stable/40479486. l. hardesty, juliet. “exhibiting library collections online: omeka in context.” new library world 115, no. 3/4 (2014): 75–86. https://doi.org/10.1108/nlw-01-2014-0013. mansfield, tomothy. “small museum? want more from your website?” the interaction consortium pty ltd, 2017. https://interaction.net.au/articles/small-museum-want-more-from-your-website/. mansfield, tomothy. “glamkit is dead. long live glamkit!” the interaction consortium, 2020. https://interaction.net.au/articles/glamkit-dead-long-live-glamkit/. marrow, deborah. museum catalogues in the digital age (osci final report). the getty foundation, los angeles, 2017. https://www.getty.edu/publications/osci-report/assets/downloads/osci-report.pdf. miller, pamela wilder. “what we do best: quality collections care practices in small museums in utah.” doctoral thesis, utah state university, 2008. https://digitalcommons.usu.edu/cgi/viewcontent.cgi?article=1007&context=etd. oecd. “culture shock: covid-19 and the cultural and creative sectors,” 2020. https://www.oecd.org/coronavirus/policy-responses/culture-shock-covid-19-and-the-cultural-and-creative-sectors-08da9e0e/. ontotext. “what are linked data and linked open data?” accessed august 17, 2022. https://www.ontotext.com/knowledgehub/fundamentals/linked-data-linked-open-data/. scholz, martin, and günther görz. “wisski: a virtual research environment for cultural heritage.” in ecai, 2012. https://doi.org/10.3233/978-1-61499-098-7-1017. velios, athanasios, and aurelie martin. “off-the-shelf crm with drupal: a case study of documenting decorated papers.” international journal on digital libraries 18, no. 4 (2017): 321–31. https://doi.org/10.1007/s00799-016-0191-5. watson, sheila, ed. museums and their communities. leicester readers in museum studies. london ; new york: routledge, 2007. weakley, alastair, thomas ashelford, julien phalip, and greg turmer. “introducing glamkit – a free, open-source web framework for the glam sector,” 2009. https://www.museumsandtheweb.com/mw2010/abstracts/prg_335002451.html. about the authors avgoustinos avgousti (a.avgousti@cyi.ac.cy) senior research technical specialist in digital humanities and cultural heritage informatics at the cyprus institute at the science and technology in archaeology research center (starc). and, a ph.d. candidate at ionian university, school of information science & informatics, department of archives, library science and museology. dr. georgios papaioannou (gpapaioa@ionio.gr) associate professor at the ionian university, corfu, greece. his research interests lie in museology, digital humanities, archaeology of the eastern mediterranean and the arab world, cultural studies, and it applications including heritage documetation, big data and mobile applications. he has lectured, excavated, and conducted museum and cultural heritage work internationally (europe, arab world). he is the general-secretary of the hellenic society for near eastern studies, director of the museology research lab at the ionian university (greece), member of the pool of experts of the european museum academy, and a member of icom. dr. feliz ribeiro gouveia (fribeiro@ufp.edu.pt) professor of computer engineering at the faculty of engineering of university fernando pessoa, porto, portugal. he has worked in research and technology transfer projects for industry, in the areas of knowledge management and object-oriented databases for major european companies. he participated in several research projects in cultural heritage and digital humanities and has acted as consultant for organizations implementing cultural collection management projects. he graduated in electrotechnical engineering from the university of porto and holds a phd in computer science from the université de technologie de compiègne, france. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – from users to developers: ncsu’s involvement with an open source erm mission editorial committee process and structure code4lib issue 34, 2016-10-25 from users to developers: ncsu’s involvement with an open source erm coral, an open source electronic resource management tool, has been adopted by libraries around the world. the community manages the software development contributed to the open source codebase by independent organizations. ncsu libraries’ acquisition & discovery department started using coral to manage monograph orders at the end of 2013. since then, they have completed a series of developments to enhance coral functions for workflow management, streamlining the complex electronic resource acquisition process. this paper presents ncsu’s adoption and development of coral. it explains what prompted the development, shares the experience, from identifying internal resources to outsourcing development work, and identifies challenges and opportunities of the current mechanism of coral development. by xiaoyan song introduction the acquisition & discovery (a&d) department at north carolina state university libraries has three units: monographs, serials, and data projects and partnerships (dpp). a&d’s core functions include the acquisition, licensing, cataloging and access management of materials for all ncsu libraries, including the branches. the department employs a suite of commercial systems and home-grown tools to facilitate the core work. e-matrix, developed in-house, is used to support e-resource life-cycle management for e-journals and databases. e-matrix manages the complex relationships of e-resource titles, holdings, collections, organizations, contacts, licenses, administration information, and acquisition details. however, workflow support is not provided in e-matrix. the department has developed a set of workflow maps for daily core work related to the resource selection and acquisition process, which involves multiple departments and staff. purchase requests are often initiated by collection managers (cm), then sent to a&d, and finally routed to an appropriate workflow with corresponding staff. there were a couple of challenges with the process. first, the purchase requests came from a variety of methods including emails, phone calls, and shared google docs, and staff found it challenging to keep track of requests from mixed channels. and second, orders were distributed to staff in the two units via emails or excel files, but the sheer volume of firm orders was too great to keep track of. we needed a tool to support the diverse workflows and to address the needs for the resource selection and acquisition process. coral was chosen due to its flexible and powerful workflow function support. it is an open source electronic resource management (erm) system which supports electronic resource life-cycle management. it was initially developed by notre dame university library and was released in 2010. it includes five modules [1]: resources, licensing, organizations, usage statistics and management. the resource module stores resource records and provides support for e-resource life cycle management, including acquisition, access, cataloging, and workflow routing. the licensing module stores licensing agreements, terms and related files. the organizations module is used to manage organizations including names, contacts and accounts. the usage statistics supports counter and counter-like reports. the management module is a place to store documents such as policies, process, and procedures, and to preserve the history of the documents. all five modules are interoperable, though independent from one another. since its first release of the resources module in 2010, coral has been gradually adopted by libraries and other organizations globally. there are 61 confirmed institutions/organizations using coral presently[2]. coral users have shared their experience at a number of major conferences and these presentations have been gathered on the coral website. we noticed that many users have implemented and used coral but few performed significant functional enhancements as developers. to give back to the community, we want to share our journey to implement and develop coral, and the lessons we learned from our excursion. from users to developers to address the issues with the resource acquisition process, a&d implemented the coral resources module to manage monograph firm orders in dec. 2013. we created records for new resources to be ordered and assigned them to staff in coral. this module enabled us to keep the firm orders in one place and to keep track of such order requests. in conjunction with coral, a shared google document between a&d and cm was created to centralize all firm order purchase requests. the specialist in the monograph unit was notified whenever a new order was added to the spreadsheet, and she would then create a record in coral and use the workflow in coral to automatically assign it to a staff member, who would process the order following the corresponding workflow. this streamlined the acquisition process. when we set up these workflows in coral, we immediately realized the limitation of coral’s workflow function to distribute orders among staff members. in coral, workflows are created as global templates at the admin tab and they can then be associated with a resource record. users can make changes to the global workflows at the admin tab, but they cannot modify a workflow once it is linked to a local resource record. we identified two local workflow customization features to be added to coral workflow function. the first is to enable users to re-assign a workflow step to a different member if needed, and the second is to allow users to delete any unnecessary step in a local workflow. figure 1 shows that a workflow is linked to a resource record in coral. clicking on the pencil icon allows users to change the group assignment. to delete a step, users can click on the red x symbol. figure 1. workflow enhancement   these two new features enhanced coral’s workflow functionalities and led to the shared use of coral by both the monographs unit and the serials unit. the monographs unit expanded the use of coral to include firm orders, open access cataloging, and one-time purchases (a.k.a. end-of-year purchases). the serials unit established a set of continuing resources new order workflows in coral. see figure 2 for an overview of all our coral workflows. figure 2. coral workflows the libraries were determined to contribute these modifications back to the community. the coral community requires that any submitted code changes be reviewed by the coral steering committee (sc). this committee consists of several electronic resource librarians (erl)  and it staff from libraries, who are also coral users. we reached out to the committee chair and inquired about the process. it was not a formalized process then because we were the first non-sc members who showed interest in making code changes to improve coral functions. as suggested by the committee chair, we created a workflow enhancement document and shared it with coral sc, who reviewed and approved the enhancement specs. the coral sc was looking for justification for the enhancement and a description of the changes to be made.we included the use cases for the requested changes and mockups of the interface change. this round of review and approval process ensured a smooth code merge to the master code at a later stage. our it staff completed the development based on the the approved enhancement spec within several weeks. the code merge request was created on github and then reviewed and approved by coral sc before it was merged into coral codebase. while we were pleased with the enhancements, one unexpected opportunity landed in front of us – ncsu libraries announced an internal good idea mini-grant to encourage library innovation and collaboration in spring 2015. to take advantage of the opportunity, a&d and cm collaborated on a proposal for further coral improvements that focused on streamlining e-resource selection and the acquisition process across the two departments. excitingly, the proposal was approved by the good idea mini-grant committee and we received $21k for the project. two groups were formed: the admin group and the working group. the admin group, including two department heads from a&d and cm and an associate head from a&d, was charged with providing admin support and managing the project budget. the working group was led by an e-resource librarian (erl) and consisted of two specialists from a&d, one librarian from cm and one it staff member. this group was tasked with creating the spec and interacting with developers to provide testing feedback. due to the unavailability of it resources, the admin group decided to outsource the development work to provide some relief on internal competition for it resources. this required a request for proposal (rfp) process. the key component of the rfp was a detailed development specification. after the grant was announced, the working group immediately started working on the spec. within a couple of weeks, the group delivered the initial specs and shared it with the sc, who provided constructive feedback. because of the complexity of the enhancements, it took several iterations of communications between the team and the coral sc until the final spec was agreed upon. the finance & business department and the university purchasing department facilitated the rfp process. to hire an experienced development team, we proactively reached out to the coral sc and a couple of potential candidates. we received two proposals after the rfp was sent out. among the two, we chose our current development team, biblibre, a french software company, who provided a reasonable quote and had significant experience with open source tool development in the library environment. grant-funded coral developments for the grant-funded project, the team identified three areas for improvement. the first one was to build a tightly integrated process for resource selection and acquisition between a&d and cm. generally speaking, this could be a process involving acquisition staff and selectors in any library. the process had previously relied on emails and shared documents to help with decision-making. our team aimed to streamline the process and to make it more efficient. we created a propose new resource api, which allowed users to submit a resource through the api. the api was based on flight [3], a simple php framework enabling users to build restful applications. a client form, based on unirest [4], was created for users to fill in and submit a resource to coral. figure 3 shows the functions implemented in the api. for more information about the api and how to use it, refer to coral github issue #7 add an api to coral [5]. see screencast 1 to preview the completed development. figure 3. api functions unable to display content. adobe flash is required. screencast 1. propose new resource api. best viewed in full screen mode. the second area identified was management of the complex continuing resource renewal process in coral. this process involves a variety of activities and requires the authorization from our cm when there is significant price change for a renewal resource. we wanted to integrate the renewal authorization process to the workflow and to make the process transparent to all stakeholders. the team proposed a couple of enhancements to the workflow functions, allowing users to manage renewals: restart new workflow for a completed resource. coral users could not restart a new workflow if a workflow was completed for a resource. this allowed users to restart a new workflow, for example a renewal workflow, after a previous workflow was finished. see screencast 2 for a preview. archive a completed workflow. this enabled completed workflows to be archived by default, and users could show or hide the archived workflows. see screencast 2 for a preview. unable to display content. adobe flash is required. screencast 2. restart and archive a workflow. best viewed in full screen mode. add a new step to the in-progress workflow. for an on-going workflow, users could add a new step if needed. see screencast 3 for a preview. reminders for workflow deadlines. task reminders are often seen in project management applications or task management tools. this functionality would remind users to complete a step by a due date. see screencast 3 for a preview. unable to display content. adobe flash is required. screencast 3. edit the current workflow and email reminder. best viewed in full screen mode. email notes to a group/user regarding assigned resources. users had complained that the workflow assignment emails sent from coral did not contain much useful information. this would allow us to embed a concise message in the email about the assignment rather than the generic message generated by the system. see screencast 4 for a preview. unable to display content. adobe flash is required. screencast 4. embed a note to reassignment email. best viewed in full screen mode. last but not the least was to improve the import tool to support more diverse needs. at the moment, the import tool in coral only allowed a small number of fixed fields to be mapped, and this had limited the use of the tool. users could use the tool for different purposes, like loading a kbart file into coral, batch loading some cost data, or loading resources to be ordered and triggering a workflow once resources were added. to facilitate the various uses of the tool, it required more flexible data mapping. sirsidynix contributed some improvement to the coral import tool [6]. we reviewed sirsidynix’s work, collected feedback from the coral sc, and modified our original proposal to incorporate the sirsydynix contributions. this resulted in the following feature enhancements: configure/create import templates. allows users to create an import template by mapping incoming data elements to a desired field in the coral resources module. the template could be loaded and applied during import. preview the mapping before submit. the preview would provide a summary of data to be loaded, such as the total number of records loaded, the number of new records to be overlaid, etc. review history of uploads. users would be able to review the history of uploads, including the date of previous uploads, the file name, the link to the file, and the link to the imported resources. the import tool development is work in progress and the development started mid-september. challenges and opportunities as open source tool users, we are benefiting from community contributions to the project. however, as the community is growing, potential problems arise and challenge the community. during our journey, we experienced a few challenges, some may be unique to the ncsu libraries, but some may apply to others. the biggest challenge was resolving conflicts between local needs and global needs. since its first release in 2010, coral has gone through several evolutions. community members can identify and implement improvements and contribute changes to the coral codebase. enhancement requests originate from local instances. but to merge modifications to the codebase, the enhancements have to meet the community’s needs at large. it’s wonderful when local requests meet global needs. when they don’t, how do we resolve our local concerns? one way is to branch the code, which ensures local development fully responds to local needs. however, branching often means greater local tech support, and the possibility of being unable to use other desirable community contributed enhancements. in our case, our it did not favor the branch option. this left the team with no choice except to resolve the conflicts and to negotiate with the community. this required more time and effort to deliver the final product, but we were able to deliver a product for all. the negotiation process was all about seeking to understand each other and to be understood by others. since community developers can all add code to the code base, it becomes a big challenge to ensure code consistency and to maintain high quality. hurley [7] points out that “establishing a standard for code submissions, requiring acceptance of a common license, and implementing peer review are three ways in which good open source projects help to mitigate the risk of problematic code.” the coral sc is working to establish version controls and best practices to address this issue. whether or not to outsource development can be a question for any software development. the answer to that question largely depends on the availability of internal resources and on the availability of third-party resources. for our grant-funded project, we chose to outsource the development work to a third-party due to the unavailability of internal resources. networking and outreach ensured the success of finding the right outsourcing team. we realized, though, there were very few third-party developers in the us in this area of development. this may be because libraries have always relied on internal it or other internal resources to do such development work, or simply purchase available commercial products. the software industry in the us should respond to the growing needs of software developments in libraries. while working with non-us developers, it is important to take into account different cultures. the 12-week development cycle planned for our grant-funded project was extended several weeks longer because of a couple of “unknown” holidays and vacations that our french team took. it’s not a bad idea to clarify with your developers of upcoming holidays and vacations during the development cycle. this is especially important if the development is funded by state fund or other funds that have a strict spending timeline. conclusion coral has a bright future as an open source erm because of the strength of the software itself and of the strong community support. either as regular users or developers, community members undoubtedly benefit from all valuable contributions. challenges can become exciting opportunities for the software to evolve and for the community to exploit synergies. references [1] coral modules. retrieved from: http://coral-erm.org/modules/ [2] coral user map. retrieved from: http://coral-erm.org/about/user-map/ [3] flight. retrieved from http://flightphp.com/ [4] unirest. retrieved from http://unirest.io/php.html [5] add an api to coral (issue #7). retrieved from: https://github.com/coral-erm/coral/pull/41 [6] sirsidynix additions and style change. er&l 2016 coral user’s group meeting. retrieved from: http://www.slideshare.net/scottvieira/er-l-2016-coral-user-group-meeting [7] hurley, david. 2014. 12 challenges for open source projects. retrieved from: https://opensource.com/life/14/6/12-challenges-open-source-projects about the author xiaoyan song is an e-resource librarian at north carolina state university libraries. she holds an mls and a b.a. in information management and information system. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – trust, but verify: auditing vendor-supplied accessibility claims mission editorial committee process and structure code4lib issue 48, 2020-05-11 trust, but verify: auditing vendor-supplied accessibility claims despite a long-overdue push to improve the accessibility of our libraries’ online presences, much of what we offer to our patrons comes from third party vendors: discovery layers, opacs, subscription databases, and so on. we can’t directly affect the accessibility of the content on these platforms, but rely on vendors to design and test their systems and report on their accessibility through voluntary product accessibility templates (vpats). but vpats are self-reported. what if we want to verify our vendors’ claims? we can’t thoroughly test the accessibility of hundreds of vendor systems, can we? in this paper, we propose a simple methodology for spot-checking vpats. since most websites struggle with the same accessibility issues, spot checking particular success criteria in a library vendor vpat can tip us off to whether the vpat as a whole can be trusted. our methodology combines automated and manual checking, and can be done without any expensive software or complex training. what’s more, we are creating a repository to share vpat audit results with others, so that we needn’t all audit the vpats of all our systems. by melina zavala and matthew reidsma like many institutions, gvsu university libraries has made web accessibility a priority, especially in recent years after a 2016 investigation by the u.s. department of education over web accessibility was resolved (grand valley state university 2017; united states department of education 2017). while we have worked to improve the accessibility of our own online interfaces, most of our software is licensed from third-parties: discovery layers, link resolvers, opacs, and particularly our subscription databases. thankfully, vendors often self-report their own accessibility testing, since most of their customers are, like us, federally obligated to meet accessibility standards. the voluntary product accessibility template (vpat) is a document created by the information technology industry council to show how software meets the americans with disabilities act (ada) section 508 guidelines (voluntary product accessibility template [updated 2018 april]) . vpats are a useful window into the accessibility of web software, but they rely on the software vendor to self-report whether or not their software meets these guidelines (adams et al 2018). because the documents are voluntary and self-reported, others in the library world have noted that “completed vpats often lack detail and vendors can easily hide product deficiencies in this document” (adams et al 2018). this was also a concern for us at gvsu, after finding problematic accessibility issues in many of our tools that were not reflected in the product’s vpat. more recently, with the increasing adoption of vpats by vendors, several other publications have chronicled attempts to verify the vendor claims. in 2015, laura delancy published a study where she aimed to do a near-comprehensive automated test of complete vendor vpats, and found that the vpats she tested were around 20% incorrect (delancy 2015). a 2016 study of course management software accessibility in higher education found that automated testing only gets us so far – the results must still be interpreted by someone trained in accessibility (kumar and owston 2016). michael fernandez also used automated tools to audit vendor databases for accessibility but did not bother reviewing vpats, since “ there is no requirement for vendors to complete vpats, there is often a significant lack of consistency among the vpats that vendors do provide” (fernandez 2018). the tennessee board of regents, as part of their push to improve the accessibility of their resources, planned in 2018 to audit the vpats of all commercially licensed databases (adams et al 2018). more recently, rysavy and michalak (2020) published the results of an automated audit of their licensed databases for the goldey-beacom college library, where they needed to support a blind student. they noted the limitations of automated tools, in that automated tools help you identify where problems exist, but most automated reports still require interpretation by a knowledgeable person (rysavy and michalak 2020). because of the labor involved with individual libraries conducting tests of every licensed resource, several groups got together to run audits together to share the labor. after all, licensed subscription databases are the same for all libraries at a given time. in 2015, a group of uk libraries banded together to audit ebook databases (the ebook accessibility audit [2016]). their reports are available online, and often highlight accessibility issues that vendors had not mentioned in vpat disclosures. meanwhile in the united states, the big ten alliance conducted similar third-party reviews of subscription databases (library e-resource accessibility – testing [n.d]), and these reports are available to all libraries online as well. at gvsu, we wanted to make sure we were doing everything we could to improve the accessibility of our online offerings for our students. from reviewing all of these thorough reports published by other libraries and consortia, it was clear that library software vendors have made remarkable improvements over the past two decades. but from our own casual testing and user reports, we knew that there was still a lot to be done. while it has been encouraging that vendors have made improvements over the past two decades, there is still work to do. yet most libraries do not have the resources or expertise to do the kinds of full accessibility evaluations envisioned by delancey (2015) and fernandez (2018) in their studies. the studies themselves have a short shelf life, as well, given the increasing emphasis on accessibility improvements and the wide adoption of agile development methods, with frequent and continuous software releases. it also strikes us as problematic that libraries are already paying licensing fees that outpace inflation by several orders of magnitude, and then vendors also benefit from our thorough accessibility testing of these products? that feels like an additional accessibility tax which we were not willing to pay. in our case, we wanted a quick and dirty method for checking up on the accessibility claims of our vendors. if the introductory statements of our vendor vpats are to be believed, they have already done this detailed accessibility testing. the primary reason we want to verify that our vendors’ vpats are accurate is because accessibility is a priority at our institution. we strongly believe that all our users deserve equitable and seamless access to our resources. in addition, digital accessibility is the law. subscribing libraries, not commercial vendors, are responsible if the systems we subscribe to do not meet federal accessibility requirements (mulliken 2019). because of this, it is important to take this extra step to verify that our software is accessible for all users. we needed a way to identify the most common problems vendors face with accessibility, so that we could focus our spot checks on those items only. the in-depth published study of vendor vpats by delancey (2015) was over five years old at this point, and the article itself did not detail the errors every vendor faced. we also did not have the resources or time to conduct even a handful of large-scale accessibility reviews to determine this criteria. luckily, from 2015 through 2019 the big 10 academic alliance began conducting in-depth accessibility reviews on subscription databases, and publishing the results online (library e-resource accessibility – testing [n.d.]). we began this project with the assumption that most vendors suffered from the same common accessibility errors, and so we used the 65 reports available in february 2020 from the big ten alliance as a data set to look for the most common accessibility issues. reviewing all of the reports, we discovered that nearly every vendor product suffered from a combination of the same 5 or 6 accessibility issues. those issues are: improper heading use (no headings, skipped heading levels) lack of consistent image alt text (alt text missing, regardless of whether it is useful) improper form structure (missing or incorrect use of labels, fieldsets, and legends.) low color contrast keyboard navigation errors (logical order, visible focus, keyboard traps) these same issues were the most commonly reported accessibility issues in delancey’s 2015 study. so, while improvements have been made to increase the accessibility of library software, the major challenges are still in the same areas. in addition, many vendors did not make alerts or other messages that are created asynchronously available to users of assistive devices. to ensure screen readers are aware of messages that appear without a page load, developers must make use of accessible rich internet application attributes (aria). aria has a significant learning curve compared to the 5 accessibility issues listed above. developers have to understand how to incorporate aria attributes into their html to let assistive technologies know the semantic purpose of each section of the page, as well as integrate aria attributes into their scripts to ensure that dynamic content is properly recognized and announced to the reader. since our goal with this project was to make spot-checking vpats accessible to library staff without specialized training, we opted not to use it as a testing criteria. in every case from the big 10 alliance accessibility tests, vendors who failed to make messages and status updates accessible also failed the more basic criteria listed above. we then developed a simple method for testing vendor tools for these three items. first, we look over the most recent vendor vpat. many vendors now list their vpats on their support websites, although you may have to ask your sales representative for a copy [1]. the wcag 2.0 areas that relate to our five test areas of vendor vpats are as follows: improper heading use 1.3.1 information and relationships 1.3.2 meaningful sequence 2.4.6 headings and labels lack of consistent image alt text 1.1.1 non-text content 1.4.5 images of text (cannot be tested with automated tools) improper form structure 1.3.1 information and relationships 3.2.2 on input (cannot be tested with automated tools) 3.3.2 labels or instructions low color contrast 1.4.1 use of color (cannot be tested with automated tools) 1.4.3 contrast (minimum) keyboard navigation 2.1.1 keyboard 2.1.2 no keyboard trap 2.4.1 bypass blocks 2.4.3 focus order (cannot be tested with automated tools) 2.4.7 focus visible (cannot be tested with automated tools) 3.2.1 on focus (cannot be tested with automated tools) when looking at vendor vpats, reports of “partially supports” with details of what situations are not yet fully compliant with an accessibility guideline are encouraging. when vendors are doing a thorough accessibility test they will encounter errors that impede accessibility. if they are transparent enough to report that there are problems and that they are working on the issue, you can likely trust the results of the vpat. on the other hand, if a vendor claims they support every accessibility guideline that applies to them, this is a red flag. we have yet to encounter any vendor site that meets all accessibility guidelines. because larger vendors have more resources to support accessibility, they will likely have more accurate and detailed vpats (fernandez 2018). to test the first four items, we installed the wave accessibility testing extension in our web browser of choice [2]. we visited a page from the vendor, preferably in the way most of our users would encounter it, such as an article landing page linked from our discovery layer’s search results or the link resolver. we then clicked the wave extension to run the page through wave’s automated accessibility checker. while there are certainly drawbacks in using automated accessibility tests (rysavy and michalak 2020; delancey 2015; kumar and owston 2016), for the purposes of seeing how accurately vendors report the results of their own accessibility testing, automation is often enough. figure 1. screenshot of wave test results screen on taylor and francis database. wave will show a list of errors and alerts. at this point, we read through the results to look for accessibility issues that fall into one of the first four categories above. if there are images without alt text, for instance, we look carefully at the image to see if the image is decorative (in which case there should be no alt text). if the image is supposed to convey meaning, then missing alt text means the content of that image is not available to low-vision or no-vision users. if the vendor skips heading levels, or has empty headings on the page, we note those, as well as any color contrast or form issues. it is possible that wave will report accessibility errors that are not, in fact, errors. alt text is a great example. wave cannot tell if an image is decorative or not, and so it flags every image without alt text as an error. as rysavy and michalak (2020) note, “running your website through the wave tool does not result in a ‘pass’ or ‘fail,’ … [because] only humans can determine whether a web page is accessible.” finally, we test for the fifth area of concern: keyboard navigation. for this we use a simple manual test. we refresh the page (to remove the wave test results), and proceed to hit “tab” to navigate through the page. tab moves us forward, and shift + tab moves us back. the site should move us logically through every hyperlink on the page, from left to right and top to bottom. we should always be able to see where the focus is, often by the link being surrounded by a dotted line box. at no point should the focus visually disappear, and we should be able to tab through the whole page and then start over at the top again. we should never get trapped in a loop, or end up in a situation where keyboard users are “trapped” in a pop-up window or section of the site that we cannot escape. this manual test uncovers many issues. as oswal (2014) noted, “if one were to try to use these databases solely with a keyboard without a mouse, one would quickly discover a number of accessibility failures on their front search pages.” running the wave test and the manual keyboard test takes between three and five minutes, in our experience. because this test does not rely on an understanding of assistive technologies, or an understanding of writing accessible code, it can be done simply by nearly anyone in the organization. and because it is so quick, it can easily be updated as vendors release new updates that improve accessibility or add new features. although this is a quick method, we still looked for criteria to prioritize testing some vendor tools over others. in the end, we contacted our collection strategist librarian and acquisitions librarian to ask for any databases that were up for renewal this year, as well as a list of the top vendors in terms of cost to the university libraries. the list of vendors we tested on this first round was [3]: ebscohost [4] proquest [5] taylor & francis [6] wiley [7] elseivier (science direct) [8] springer [9] oxford [10] all total, even counting the time we took to take detailed notes for this article, we spent just under 90 minutes running these tests. after testing, there were clear discrepancies between what the vendor’s vpat claimed and the results of our basic accessibility tests of each platform. the main issues seen across most of the platforms were low color contrast and issues with form labels. while not all vendors exhibited the same problems, every vendor we tested had accessibility issues that were not reflected in the vpat. most of the vpats we reviewed contained sections that admitted that the software did not fully meet accessibility guidelines, detailing the issues that had been found, and at times, even offering a road map and estimated timeline for the issue to be resolved. for instance, proquest noted in their report of section 1.3.5, identifying input purpose, that as of the first quarter of 2020, they should have addressed issues with the autocomplete connecting to the form field and label that it is associated with. in addition, springer noted in several places that they “…are in the process of transferring all content from the older design to the newer design,” which would address certain problems with meeting accessibility guidelines. as we approach contract renegotiation, we will revisit those issues in order to see if they have been addressed. on the other hand, some vendors were honest about their issues, but perhaps not clear about their plans to resolve them. oxford noted in its vpat for sections 1.1.1 and 2.1.1 that it had no support for these basic areas of non-text content and keyboard navigation. they noted that they had “unlabeled and ambiguous form fields”, “missing alternative text” for images, and “non-descriptive alt text” for many of the images that did have the attribute. in addition, for keyboard navigation in their vpat, they admitted that some areas of their pages were inaccessible to keyboard users. in our test, we verified through a single page that there were orphaned form labels, redundant form labels and alt text (several images on the page had the same alt text), and that there were significant problems with keyboard navigation. several links did not receive visual focus when navigating with the keyboard, and in fact many “links” had no visual indication that they were links at all. one highlight was that the “skip to main content” link was hidden visually under normal circumstances, it immediately became visible and received focus when keyboard navigation began. while it is admirable that oxford documented these issues with their online tool, this vpat was finalized in 2017. we ran our test over 2 years later, and all of these issues remain. the vpat should not work as merely a confessional. vendors must work to improve the deficiencies in their accessibility as well. when we approach renegotiation with oxford at the end of our current contract, we will share the results of our audit as part of our negotiation. in another case, wiley listed many accessibility issues, but upon testing, we found a completely different set of issues with their user interface. they admit in their vat to missing alt text on images and struggling with keyboard navigation (wiley 2018), and these were for the most part still problems. (many of their images had alt text that said “image,” which is perhaps less useful than just leaving the alt text blank.) however, they reported that they handled visible focus well, and that they associated all input fields with the appropriate labels. in reality, though, we found that keyboard focus visually disappeared at times, and the results page we tested had several orphaned form labels, which will confuse assistive technologies. the larger vendors, proquest and ebsco, generally reported correctly the issues that they faced with accessibility. they both admitted to having limited issues with color contrast and alt text, as well as some issues where form labels were not associated with their input fields. both vendors had recent vpats (within the last six months), and at times suggested that they were working on a fix for specific issues listed in the vpats. we did find a few issues that were in the article content, rather than in the user interface. both vendors, as well as science direct, mentioned that they had limited ability to verify or improve the accessibility to content in their platforms. nevertheless, the issue of inaccessible content will be something we address in our next renegotiation. while most of the vendors at least admitted they had some issues, taylor & francis did not. their vpat claims its platform meets all of the accessibility guidelines with no issues., unfortunately, their platform was the one that had the most problems come up. they were missing alt text on many images, they had several “empty” heading and buttons (which are not visible to sighted users, but will be announced with no content to users of assistive technologies), unlabeled form elements, and an all around sloppy use of headings, skipping heading levels seemingly at will. these issues would make it hard for a screen reader to parse through and accurately tell the user what is on the page. for the non-automated part of the test, there were also a lot of issues; the skip link in the beginning of the page did not have a visible focus (it never appeared visually on the screen, so for a sighted user, it appeared that keyboard navigation was not possible). the search bar, perhaps the most useful element in the user interface, also did not have visible focus. when we finally made it to the content, the article itself was a pdf image, with alternative text that said “free first page,” but not the text of the article itself. (it is unclear whether we could get to an ocr version of the pdf by going deeper into the results.) this is a huge problem for the users who are not able to read the image itself and rely on screen readers to read it for them. what’s more, this result was shown to an authenticated user, not someone landing on the article page from the open web. the content should already be available to authorized users, right up front. to cap it off, taylor & francis traps keyboard users at the end of the page, looping between sharing and social media icons and the web browser’s user interface. once a keyboard user makes it to the share icons, the rest of the page, including article content, becomes inaccessible. these issues, as well as the incorrect vpat, will be passed along to our acquisitions and collections team to become part of our next licensing negotiation with taylor and francis. while we hold no illusions that we have captured a detailed accounting of the accessibility of our licensed databases through this simple exercise, we felt that it was time well spent. in less than two hours, we walked away with detailed notes that will enable our licensing team to press the vendors for more thorough responses, more decisive action, more transparent timelines, and perhaps even a more fair license for our library. since the process is relatively simple and quick, there is no need for a public repository of test results that will go out of date as soon as the vendor pushes the newest version live. most libraries have the ability to conduct five minute verifications of their vendors’ vpats. if we all approach the license negotiations with notes about inaccuracies in a vendor’s vpat, perhaps then we will make even quicker progress in ensuring our licensed materials are accessible to all our users. notes [1] you can also check laura delancey’s vpat website (delancey 2019), although many of these are 5 or more years old. [2] you might need to test with multiple browsers to make sure that these basic accessibility guidelines are met, but we’ve never had a situation where a vendor was perfect with one browser and not others. [3] sage was also on our list based on total cost per year, but we did not receive their vpat information in time to add them to the testing protocol. [4] ebsco’s vpat is available to subscribing libraries by emailing support@ebsco.com [5] proquest vpats are available from their accessibility directory at https://support.proquest.com/articledetail?id=ka140000000guogcac. [6] taylor and francis’ vpat is available from https://help.taylorfrancis.com/students_researchers/s/article/voluntary-product-accessibility-template-vpat. [7] wiley’s vpat is available by contacting your account manager. see https://onlinelibrary.wiley.com/accessibility for more details. [8] elsevier’s science direct vpat is available at https://www.elsevier.com/solutions/sciencedirect/support/web-accessibility. [9] springer’s vpat is available on github at https://github.com/springernature/vpat/blob/master/springerlink.md. [10] oxford university press’ vpat is available at https://academic.oup.com/journals/pages/about_us/legal/accessibility. references adams, sj., halaychik, c., and mezick, j. 2018. accessibility compliance: one state, two approaches. the serials librarian, 74(1-4): 163-169. byerley, sl., and chambers, mb. 2003. accessibility of web-based library databases: the vendors’ perspectives. library hi tech 21(3): 347-357. byerley, sl., chambers, mb., and thohira, m. 2007. accessibility of web-based library databases: the vendors’ perspectives in 2007. library hi tech, 25(4): 509-527. delancy, l. 2015. assessing the accuracy of vendor-supplied accessibility documentation. library hi tech 33(1): 103-113. delancey, l. 2019. vpat repository. [cited 2020 mar 17]. available from https://vpats.wordpress.com/. dermody, k. and majekodunmi, n. 2011. online databases and the research experience for university students with print disabilities. library hi tech 29(1): 149-160. the ebook accessibility audit. [internet]. [updated 2016]. ebook audit 2016. [cited 2020 mar 17]. available from https://sites.google.com/site/ebookaudit2016/home. grand valley state university resolution agreement ocr docket #15-16-2195. [internet]. [2017 march 13]. department of education office for civil rights. [cited 2020 mar 17]. available from https://www2.ed.gov/about/offices/list/ocr/docs/investigations/more/15162195-b.pdf. kumar, kl., and owston, r. 2016. evaluating e-learning accessibility by automated and student centered methods. educational technology research and development 64(2): 263-283. library e-resource accessibility – testing [internet]. [n.d.] champaign (il): big 10 academic alliance. [cited 2020 feb 28]. available from https://www.btaa.org/library/accessibility/library-e-resource-accessibility–testing mulliken, a. 2019. eighteen blind library users’ experiences with library websites and search tools in u.s. academic libraries: a qualitative study. college & research libraries 80(2): 152-168. oswal, sk. 2014. access to digital library databases in higher education: design problems and infrastructural gaps. work. 48:307-317. rysavy, mdt., michalak, r. 2020. assessing the accessibility of library tools & services when you aren’t an accessibility expert: part 1. journal of library administration 60(1):71-79. tatomir, j., and durrance, jc. 2010. overcoming the information gap: measuring the accessibility of library databases to adaptive technology users. library hi tech, 28(4): 577-594. united states department of education, office for civil rights region xv. [internet]. [n.d.]. department of education office for civil rights. [cited 2020 mar 17]. available from https://www2.ed.gov/about/offices/list/ocr/docs/investigations/more/15162195-a.pdf. voluntary product accessibility template (vpat). [internet]. [updated april 2018]. washington, d.c. section508.gov. [cited 2020 mar 17]. available from https://www.section508.gov/sell/vpat. about the authors melina zavala is an early career librarian that graduated from the university of illinois at urbana-champaign in 2018. they are currently working at grand valley state university as a digital scholarship librarian. matthew reidsma is the web services librarian at grand valley state university and the co-founder and former editor-in-chief of weave journal of library user experience. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – the dsa toolkit shines light into dark and stormy archives mission editorial committee process and structure code4lib issue 53, 2022-05-09 the dsa toolkit shines light into dark and stormy archives themed web archive collections exist to make sense of archived web pages (mementos). some collections contain hundreds of thousands of mementos. there are many collections about the same topic. few collections on platforms like archive-it include standardized metadata. reviewing the documents in a single collection thus becomes an expensive proposition. search engines help find individual documents but do not provide an overall understanding of each collection as a whole. visitors need to be able to understand what individual collections contain so they can make decisions about individual collections and compare them to each other. the dark and stormy archives (dsa) project applies social media storytelling to a subset of a collection to facilitate collection understanding at a glance. as part of this work, we developed the dsa toolkit, which helps archivists and visitors leverage this capability. as part of our recent international internet preservation consortium (iipc) grant, los alamos national laboratory (lanl) and old dominion university (odu) piloted the dsa toolkit with the national library of australia (nla). collectively we have made numerous improvements, from better handling of nla mementos to native linux installers to more approachable web user interfaces. our goal is to make the dsa approachable for everyone so that end-users and archivists alike can apply social media storytelling to web archives. by shawn m. jones, himarsha r. jayanetti, alex osborne, paul koerbin, martin klein, michele c. weigle, michael l. nelson editor’s note: this article makes use of robust links. next to each hyperlink the reader will discover a menu that allows them to visit an archived version of the linked resource in case the current version has changed or is no longer available. visit the robust links project for tools and more information on combating reference rot. web archive collections are too large to understand at a glance web archives are invaluable for a variety of research studies. historians have analyzed how humans interacted on extinct websites, like geocities. social scientists have used them to study the changes in social commerce over time. journalists can use web archive evidence to bring attention to questionable medical practices and document changes in government policy. some archivists create themed web archive collections by selecting web pages for preservation that support a topic. each web page, or original resource, can change over time. archivists capture these original resources at specific points in time, turning each observation into a memento. the date and time of capture is that memento’s memento-datetime. a timemap contains the set of mementos for an original resource. archive-it is a popular platform for creating themed web archive collections. themed collections also exist at the library of congress, conifer, the croatian web archive, the uk web archive, and the national library of australia’s (nla) pandora and trove collections. figure 1. archive-it has two collections about the south louisiana flood in 2016. east baton rouge parish library created the one represented by the left screenshot. old dominion university created the one on the right. (screenshots taken in june 2021.) such topical web archive collections are invaluable to those studying a topic, but it can be difficult for visitors to understand which collection they should begin exploring. figure 1 shows screenshots of two archive-it collections containing mementos about the south louisiana flood of 2016. each collection contains different metadata fields. which one should a researcher use in their project? alternatively, if a visitor uses archive-it’s collection-level search engine to find collections about a topic like “human rights,” they have 48 collections to review as of january 2022. like government of canada publications, some collections contain hundreds of thousands of documents but no metadata to help visitors understand the collection. in a 2019 study (preprint version), we evaluated all collections at archive-it and determined that collections with more original resources have less metadata to help visitors understand them. metadata is more consistent in the nla’s pandora collections than in archive-it’s collections, but the sheer size of some collections makes it difficult to understand them at a glance. figure 2 shows a screenshot of the page for the pandora subject politics. this subject is a collection in its own right and contains subcategories and collections so that visitors can view parts of the collection. it also includes a listing of 5,269 page titles. at a minimum, each pandora collection contains a collection title and page titles. some are divided into sub-collections, but a human would still need to review many documents (or at least page titles) to understand the collection. figure 2. the pandora subject politics contains subcategories and collections, as well as a list of page titles.   the dsa toolkit for summarizing corpora the dark and stormy archives (dsa) project helps visitors achieve collection understanding by finding solutions that summarize web archive collections through visualizations that provide understanding at a glance. we apply social media storytelling as that visualization because most visitors are already familiar with the paradigm and thus require no additional training to understand how to consume these visualizations. social media storytelling consists of surrogates that summarize individual pages. figure 3 shows the same page rendered as a surrogate in different web platforms. most web users understand how to interpret the cards from search results. many readers understand how to interpret social cards used by facebook, twitter, tumblr, and other platforms. these surrogates take different shapes, but each attempts to summarize an individual document. figure 3. different surrogates for the same web page rendered by different platforms. figure 4. a social media story on wakelet about the 2021 us capitol breach. figure 5. a screenshot of the iipc’s archive-it collection novel coronavirus (covid-19). when we combine a group of surrogates together as a unit, we create a story that summarizes a topic. figure 4 shows a story that summarizes the 2021 us capitol breach. this story contains surrogates, in the form of social cards, each describing a different web resource discussing the attack. archive-it and pandora have their own surrogates. in archive-it’s case, the surrogate consists of the url captured and any metadata added by the archivist. in pandora’s case, each surrogate is a page title. we want to extend this idea of the social media story further than is already accomplished by these web archiving platforms. figure 5 shows a 2019 screenshot of the international internet preservation consortium’s (iipc) archive-it collection novel coronavirus (covid-19). this collection contains more than 23,000 mementos, far more than a single human can review, let alone understand at a glance. we applied the dsa toolkit to produce a story summarizing this collection (figure 6). the archive-it collection contains metadata painstakingly provided by the archivist. we see page titles, countries of publication, languages, and, in some cases, descriptions of individual resources. there are even facets to help users explore different aspects of the collection. the story generated by the dsa toolkit has people in masks, pictures of the virus, headlines, dates, sources, names, places, page summaries, maps showing the virus spreading across the world, and links back to the collection so visitors can explore that collection further. figure 6. a story by the dsa toolkit summarizing iipc’s archive-it collection novel coronavirus (covid-19) using sampling algorithms and social media storytelling. the dsa toolkit is built on our model of five storytelling processes, as shown in figure 7. a user follows these processes to tell a story that summarizes a corpus. figure 7. the five processes for telling a story that summarizes a corpus. as a summarization, our story provides a subset of the collection to a user. this subset consists of exemplars – documents that represent the collection well. thus, we need to select exemplars from the collection to tell our story. selecting exemplars can be done by humans or by various sampling algorithms. we can choose exemplars that attempt to summarize the collection as a whole or select those that feature a particular aspect of the collection, such as a specific time period, person, or source. we generate story metadata to enrich our story for the visitor. story metadata can consist of the collection name, who created the collection, and other collection metadata. the story from figure 6 shows that story metadata can also contain entities, terms, images, and other content extracted from the collection. once our exemplars are selected, we generate document metadata to summarize each exemplar. in figure 6, each of the social cards shown in the story visualizes this document metadata. document metadata can take many forms, as needed by the storyteller. we can then visualize the story in the desired medium by applying our story and document metadata. most users will want a story that visitors can view on the web, and hence we have focused on providing visualizations that use html. we can also visualize stories in other media, such as video. finally, we can distribute the story for others to consume our visualization. distribution can be from an author’s website. it could also be via social media. alternatively, a user could potentially print the story and manually hand it to someone. the point is that visitors cannot consume a story if they cannot access it. as shown in figure 8, the dsa toolkit provides tools to help meet the goals of this model. hypercane helps users select exemplars and generate story metadata. mementoembed generates document metadata. raintale helps users visualize the story by accepting input from hypercane and mementoembed. not shown in figure 8 are two additional supporting packages. the library aiu helps programmers identify the mementos and collection metadata from a web archive collection. the otmt (short for off-topic memento toolkit) identifies off-topic mementos in a collection. this paper highlights these tools and the results of their recent pilot with the nla, thanks to a grant from the iipc. figure 8. how the dsa toolkit fits with the storytelling model shown in figure 7. other efforts for understanding web archive collections the dsa toolkit is one of many tools for helping users understand web archive collections. for users with access to the web archive (warc) files that make up a web archive collection, archivespark (by holzmann et al.) and the archives unleashed toolkit (by ruest et al.) offer an environment for exploring collections. a user can load warcs into archivespark and then perform a set of operations for generating data, such as extracting titles, calculating term distributions, discovering named entities, building a graph of outgoing hyperlinks, and extracting images. archivespark typically outputs a json file consisting of the desired data. the output of one operation in archivespark can feed into another, allowing users to chain them together and produce the desired result. it is up to the end-user to further process this data with a third-party tool. archives unleashed toolkit (aut) is built on warcbase (lin et al.). based on work with web archive researchers, lin et al. developed four steps for those trying to work with web archives: filter, analyze, aggregate, and visualize. aut implements these four steps. with aut, a user can filter based on various features, such as original resource url, language, domain name, memento-datetime, or url pattern. an aut user can also generate reports on top-level domains, named entities, or link structure. archives unleashed cloud (auk) integrates a subset of this functionality with archive-it, providing different reports for archivists to apply to their own collections. if an aut user wishes to visualize the output, they must use a third-party tool. these tools inspired the development of the dsa toolkit. hypercane allows users to chain together several web archive collection operations, just like archivespark and aut. raintale accepts the output of hypercane’s operations and visualizes it. aut and archivespark can potentially help archivists select exemplars and generate story metadata. they can also extract or generate some document metadata. their focus, however, is not storytelling, so they rely on third-party tools for visualization and distribution. the input type is another difference between the dsa toolkit and these tools. where archivespark and aut accept warcs as input, the dsa toolkit does not. because its goal is storytelling through summarization and visualization, the dsa toolkit expects that all resources to be shared are accessible to the visitor of the story. also, we wanted the dsa toolkit to be usable by those who did not have access to a collection’s warcs. thus, the dsa toolkit uses memento urls instead of warc files as its input. jatowt et al. may have been the first to visualize web pages over time. their page history explorer downloaded mementos from web archives and generated screenshots ordered by memento-datetime as part of a timeline. similarly, the tmvis project (by mabe et al.) accepts a single original resource url, applies alsum’s algorithm to find the most novel mementos of that page, allows the user to reduce that set further manually, and then renders those mementos as a set of screenshots. these efforts work with a single original resource url, not a collection of many different ones. however, they satisfy parts of our storytelling model by selecting exemplars, generating story metadata, generating document metadata (in the form of a screenshot), and visualizing the story – with jatowt using a timeline and tmvis providing several different visual arrangements. with tmvis, users can distribute parts of their stories in various forms such as embeds, animated gifs, and image sliders. padia et al. generated a set of visualizations for web archive collections. padia’s visualizations did not focus on a single original resource url but instead tried to consider different features of the mementos in the collection. they produced visualizations consisting of heat maps, timelines, word clouds, bubble charts, and treemaps. when possible, they leveraged the metadata provided by the collection, allowing them to cluster individual mementos as part of these visualizations. while they successfully created a set of visualizations to convey meaning about aspects of the collection, these visualizations required training for the viewer to understand them. thus, they did not satisfy our goal of conveying meaning at a glance. figure 9. with tmvis, users can explore a single original resource over time and visualize thumbnails for that resource. alnoamany et al. pioneered combining storytelling with web archive collections. at the time of her work, the most popular social media storytelling service was the now-defunct storify. she analyzed storify stories and compared them with archive-it collections to understand their similarities and differences. she identified that popular stories contain a median of 28 links, giving automated storytelling platforms a target number of items to select from a collection. she identified some of the operations necessary to select exemplars automatically. her algorithm filtered off-topic mementos, then duplicate mementos, and then non-english content from the collection. she also applied clustering, both by memento-datetime and by content. finally, she scored the resulting mementos by padia’s web page category, mccown’s url path depth, and brunelle’s memento damage. her algorithm, which the dsa toolkit implements as dsa1, is the first of many possible methods for selecting exemplars. she demonstrated (preprint version) that participants could determine the difference between randomly generated stories and those created by human archivists through a user study. equally important, she confirmed that participants could not distinguish between stories produced by her visualization algorithm and those created by human archivists. figure 10 shows a story generated by alnoamany’s proof-of-concept with storify. her story metadata was the collection name. her proof-of-concept generated document metadata to fit into this platform. storify handled distributing the story to visitors. her research project was named dark and stormy archives, and the current dsa project is a continuation of her work. figure 10. a story produced by alnoamany’s dark and stormy archives proof-of-concept from the archive-it collection russia plane crash. table 1. different web archive collection analysis efforts and suitability for summarizing collections with storytelling. archivespark archives unleashed toolkit page history explorer tmvis padia’s visualizations alnoamany’s dark and stormy archives dsa toolkit (current) storytelling model support select exemplars yes yes yes yes yes yes yes generate story metadata yes yes yes yes yes yes yes generate document metadata yes yes yes yes yes yes yes visualize the story no no yes yes yes yes yes distribute the story no no no yes not currently public, but could be yes yes can user customize part of storytelling model? select exemplars yes yes no yes no yes yes generate story metadata yes yes no no no no yes generate document metadata yes yes no no no no yes visualize the story n/a n/a no yes no no yes distribute the story n/a n/a no no no no yes supports more than one original resource yes yes no no yes yes yes allows analysis without warcs no no yes yes yes yes yes web archive platforms supported any warc files any warc files 1 3 1 1 any mementocompliant archive the dsa toolkit figure 11. the dsa toolkit workflow for producing a story relies heavily on hypercane and raintale. the dsa toolkit consists of five different software projects that satisfy our storytelling processes. figure 11 provides an overview of the workflow for these tools. a user executes hypercane to select exemplars and generate story metadata. then they feed that output into raintale to visualize their story. both hypercane and raintale consult the other components as needed. here we introduce each tool and highlight our recent improvements thanks to the pilot with nla. aiu aiu (formerly archive-it utilities) is a python library containing several classes that provide information about web archive collections. a programmer can invoke one of these classes and supply a collection identifier. the class includes several methods providing information like collection title, the urls of the original resources in the collection, and other metadata, if present. aiu applies apis and scraping html to gather its information. aiu initially provided this information only for archive-it collections. archive-it collections contain metadata and a list of preserved original resources. from these original resources, we find timemaps. from these timemaps, we discover mementos. aiu follows this path to find the mementos in an archive-it collection. below is an example ipython session where a user can extract information about an archive-it collection (we measure the length of the list returned by the list_seed_uris() method for brevity, but it does produce a list of original resource urls for the collection): in [1]: from aiu import archiveitcollection in [2]: aic = archiveitcollection(5728) in [3]: aic.get_collection_name() out[3]: 'social media' in [4]: aic.get_collectedby() out[4]: 'willamette university' in [5]: aic.get_description() out[5]: 'social media content created by willamette university.' in [6]: aic.get_collection_uri() out[6]: 'https://archive-it.org/collections/5728' in [7]: len(aic.list_seed_uris()) out[7]: 113 as part of our iipc grant, we extended aiu to service the three types of collections at the nla. as presented through their pandora and trove websites. the pandora archive is the curated selective web archive component of the larger australian web archive (awa). the nla’s entire web archive corpus (including pandora) is accessible through the trove discovery service. while the awa includes pandora’s thematic sub-collections, the more extensive pandora subject listings are only viewable through the stand-alone pandora archive website. pandora subjects contain other pandora subjects listed as “subcategories.” each pandora subject includes a list of page titles, as seen in figure 2. each page title represents an original resource and links to a title entry page (tep). the tep serves as a timemap that links to all mementos for that original resource. aiu will follow this chain to find similar metadata to archive-it, including original resource and memento urls. pandora subjects also contain pandora collections. like subjects, each pandora collection can include collections and page titles. each page title links to a tep which functions the same as with pandora subjects. aiu will follow these links to find metadata and urls for pandora collections. trove collections are slightly different. each trove collection fans out into subcollections. subcollections contain direct links to mementos, as shown in figure 12. now aiu includes a class that will follow these links, gather metadata, and capture urls as needed. please view aiu’s documentation for more information on applying it to gather collection information. figure 12. a screenshot of a trove collection. otmt the off-topic memento toolkit (otmt) is an application and library that supports different text similarity measures for identifying which mementos in a collection are on-topic or off-topic. it is used as a library by hypercane to perform this same task. the otmt received bug fixes as part of our iipc grant work. hypercane hypercane is a complex application that provides the user with many ways to select exemplars and generate story metadata automatically. hypercane exists in the command line application hc. a user selects exemplars with the sample action of that application. for example, to select exemplars from archive-it collection 694 using the dsa1 algorithm, a user types: # hc sample dsa1 -i archiveit -a 694 -o story-mementos.tsv generating story metadata is handled by hypercane’s report action. for example, to generate a list of named entities from a file containing a list of mementos, a user types: # hc report entities -i mementos -a story-mementos.tsv \ -o entity-report.tsv finally, to create a rich story, a user can synthesize the output of other hypercane commands into other formats. to create a rich json story file for raintale, a user types: # hc synthesize raintale-story -i mementos -a story-mementos.tsv \ --imagedata ${image_report} \ --title "my story for collection x" \ --termdata terms-report.tsv \ --entitydata entity-report.tsv \ -o mystory.json the sample, report, and synthesize actions provide hypercane’s top-level functionality. their capabilities are not limited to what we show in the examples above. the sample action currently supports fourteen different sampling algorithms, denoted by their name – e.g., a user can execute stratified random sampling with hc sample stratified-random, or they can run the dsa3 algorithm from our previous work with hc sample dsa3. likewise, the report action supports ten different types of reports that can produce helpful metadata for storytelling, like extracted collection metadata, image analysis, or the list of top phrases for the input. the synthesize action gives the user different output format options, like a raintale story file or warcs. as seen above, each hypercane command supports the arguments -i (for input type) and -a (for input argument). the input type instructs hypercane on the nature of the input argument. as seen above, if the input type is archiveit, then the input argument must be an archive-it collection identifier. if the input type is mementos, then the input argument must be a file containing a list of memento urls. hypercane currently supports the following input types: mementos – for a file containing a list of memento urls timemaps – the input argument is a file containing a list of timemap urls original-resources – the input argument is a file containing a list of original resource urls archiveit – the input argument is an archive-it collection identifier pandora-subject – the input argument is a pandora subject identifier, added as part of the iipc grant work pandora-collection – the input argument is pandora collection identifier, added as part of the iipc grant work trove – the input argument is a trove collection identifier, added as part of the iipc grant work hypercane supports all memento-compliant archives. thus, a user need not rely upon mementos from archive-it or nla. the mementos and timemaps input types allow users to submit urls from other archives, such as the internet archive’s wayback machine, archive.today, or arquivo.pt. every hypercane command also supports the -o argument to specify an output file. in most cases, the output file is a list of memento urls. this feature allows the user to chain hypercane commands together so that the output of one command can feed into another. hypercane’s sample action helps users execute existing sampling algorithms, but users also have access to the following algorithmic primitives that allow them to build their samples: identify – identifies the memento, timemap, or original-resource urls for a given input type filter – produces a list of mementos that match the provided criteria (e.g., are on-topic, have unique content, are written in a specific language) cluster – clusters the collection via a given algorithm, such as through lda topic modeling or k-means clustering by memento-datetime score – scores each memento in the input by a given function order – sorts the input by a given feature, like memento-datetime by combining these primitives, users can create powerful scripts for sampling from collections, allowing them to tell many types of stories. we will not describe all of these primitives in detail for brevity but instead, direct interested readers to hypercane’s official documentation and our recent publications. as part of the iipc grant, we also sought to make hypercane more approachable to administrators and end-users. hypercane is a python application and also depends on mongodb for caching. we are evaluating native linux installers for centos 8 and ubuntu 21.10 that handle installing dependencies and provide convenience scripts for administering the application. figure 13. a screenshot of hypercane’s web user interface landing page. we developed a new web user interface (wui) so that hypercane was approachable for users not familiar with scripting or working with command-line interfaces. figure 13 shows a screenshot of this wui. we wanted to create a web user interface while still preserving the existence and capabilities of the command-line application for scripting. we built hypercane’s wui with the wooey library, which creates web user interfaces from command-line applications, satisfying our needs. with wooey, a user can fill out a web form to submit a hypercane command as a job and come back later to retrieve the results. wooey also provides authentication and separation of jobs to protect users’ privacy as a django application. hypercane’s sample action allows users to run pre-existing sampling algorithms to get a list of mementos for their story. as noted above, it supports ten sampling algorithms, including alnoamany’s. but what if a user wants to run sample, execute all report actions necessary to generate different types of story metadata, and finally synthesize a json file for use in a rich raintale story? before the wui, we suggested that they write a script that executes these commands in order. the wui presents convenience recipes that help users run complex tasks with minimal input and no scripting. hypercane’s wui currently supports seven convenience recipes as web forms. figure 14. a screenshot of the hypercane web form for the recipe “summarize a collection as a raintale story file.” figure 15. a screenshot of a completed hypercane job. the user can click the download button (top right) and save the raintale story file. (the red text only indicates a warning.) most users just wish to leverage hypercane’s storytelling capabilities. the recipe “summarize a collection as a raintale story file” accepts a collection id for archive-it, pandora subject, pandora collection, or trove and executes the necessary sample, report, and synthesize commands to produce a raintale story file. a user will need to supply the collection type, collection id, sampling algorithm, and story title, as shown in figure 14. figure 15 shows the interface once the job is complete, allowing the user to download their raintale story file. some users already have a list of memento urls or wish to use mementos from a memento-compliant archive not listed above. the recipe “summarize archived pages as a raintale story file” accepts an input file containing a newline-separated list of memento urls and does the same as the recipe above. other users have already selected exemplars from web archives and do not need to execute the sample step. the recipe “create a raintale story file from archived page urls” accepts an input file containing a newline-separated list of memento urls and executes the necessary report and synthesize commands to produce a raintale story file. sometimes archivists or researchers just need a list of the memento page urls in a collection. the “generate archived page urls from a collection” recipe accepts a collection id for archive-it, pandora subject, pandora collection, or trove and executes the necessary identify command to find all memento page urls in the collection. similarly, a user may want the list of original resource urls in a collection. the “generate original page urls from a collection” recipe performs the same steps but outputs a list of original resource urls. the user could also run the “identify by collection id” web form, but these recipes present simplified language and options. additionally, there are cases where a user will desire warcs for a third-party application. the recipe “synthesize warcs from a collection” accepts a collection id and will execute the necessary identify and synthesize steps to convert the mementos in the collection into warcs that closely resemble what the archive initially crawled. hypercane leverages the memento protocol and its knowledge of accessing raw memento content from different web archives to make this happen. these warcs are an approximation built from what hypercane can access via the web and are not identical to the web archive’s holdings. still, they are sufficient for experimentation and analysis with tools like aut. likewise, a user can produce warcs from a list of memento urls with the recipe “synthesize warcs from archive page urls.” these recipes continue to make hypercane more approachable for end-users. though hypercane primarily exists to feed content to raintale, it has evolved into a much more promising toolkit of its own. mementoembed mementoembed is a surrogate generation tool that can create four different types of surrogates. it produces social cards like those seen on facebook and twitter, page screenshots (also called browser thumbnails), word clouds, and imagereels. imagereels are animated gifs (gif version 89a) of the top images found in a memento. figure 16 displays one of mementoembed’s social cards. mementoembed also supports a web api for generating document metadata. through this web api, a client can create surrogates. clients can also request specific document metadata, such as page titles, memento-datetimes, image ranking, and automatic text summaries. we developed mementoembed after analyzing more than 50 platforms and finding that none support mementos correctly. some platforms do not understand how to extract the original resource url from the memento. others could not differentiate between a web archive’s content and the memento content. still, others refused to generate surrogates for mementos. mementoembed is critical to the dsa toolkit. hypercane applies mementoembed’s functions that discover images (preprint version) and raw memento content. raintale is a client of mementoembed’s web api. most web archives augment their mementos with navigation elements and branding to aid visitors. unfortunately, these augmentations confuse natural language processing technology, like that implemented by hypercane and mementoembed. for example, when comparing frequent terms among a set of mementos, the word “trove” appears to be the most frequent word, even though it does not represent the actual memento content. for wayback and openwayback web archives, like archive-it, mementoembed simply inserts an id_ flag into their memento url to visit a page lacking these augmentations. trove mementos required different handling. figure 17a displays a screenshot of a trove memento. figure 17b blurs its augmentations to emphasize the actual memento’s content. as part of our pilot with the nla, we had to update mementoembed to discard these augmentations when processing trove’s mementos. finding the actual memento content required the following steps: change the domain name in the memento url from webarchive.nla.gov.au to web.archive.org.au insert the id_ flag into the memento url from here, mementoembed can now process the raw memento content. figure 16. a screenshot of a mementoembed social card containing a striking image, title, description, original resource domain, and favicons, as well as information about the archive, when the archive preserved the page, and links to other mementos. memento from trove the content of the memento exists in the unblurred area. figure 17. trove mementos consist of augmentations that aid users in navigation and provide additional information that confuse natural language processing systems. we also updated mementoembed to better handle creating screenshot surrogates of trove’s mementos. the additional augmentations required changing the timeouts on mementoembed’s screenshot capabilities. as with hypercane, we created native linux installers for mementoembed for centos 8 and ubuntu 21.10 to help system administrators quickly stand up mementoembed. for more information on mementoembed, please see its documentation. raintale hypercane focuses on selecting exemplars and generating story metadata. mementoembed focuses on generating document metadata for a single memento. raintale takes in all of this information and produces a story in a format that the storyteller desires. raintale is a command-line application executed by the tellstory command. to apply the rich story file generated in the hypercane section above, a user types: # tellstory -i mystory.json --storyteller template \ --story-template mytemplate.html -o mystory.html the -i argument specifies the input file containing the information for the story. the --storyteller argument indicates the type of story being told. because the above example suggests that we wish to use a template, the --story-template argument is needed to specify a file containing a template that raintale will use to format the story. finally, the -o argument indicates where to store the resulting story. the user heavily controls the format of raintale stories through templates. figure 18 shows a story that applies a bootstrap carousel template to a set of mementos to summarize the trove collection tourism. figure 18 shows a story that applies an nla-authored template to a group of hand-selected mementos about australian zoos, wildlife sanctuaries & aquariums. figure 18. a raintale story generated using a bootstrap carousel. figure 19. a raintale story generated using a template developed by the national library of australia. because raintale supports templates, users can construct stories that take many forms, from the covid-19 story in figure 6 to the stories shown in figures 17 and 18. raintale applies a modified version of the jinja2 template language that supports additional options for extracting specific document metadata for the output. this template language does not limit raintale to html. it permits many textual formats, like markdown, xml, json, and jekyll. as with hypercane, we recognized that a command-line application is not always approachable for all software users. as part of our pilot, we leveraged wooey to create a wui for raintale as well. figures 20–22 show screenshots of a user executing “create story from template,” which produces a templated story just like we demonstrated with the tellstory command above. the “create story from preset” option lets users forgo a template in favor of built-in story templates. with “tell story with twitter,” raintale will create a twitter thread from the mementos in the input. finally, “create video story” generates an mp4 file consisting of the top-ranked images and text from each memento in the story. figure 20. a screenshot of the landing page of the raintale wui. figure 21. a screenshot of the web form allowing a user to “create story from template.” figure 22. a screenshot showing a completed raintale job where the user can download the story. as with hypercane and mementoembed, we developed native centos 8 and ubuntu 21.10 installers for raintale. we presented raintale at the wadl 2020 workshop and iipc wac 2021. an early version of raintale helped us conduct user testing that showed that social cards probably provide a better understanding of web archive collections when compared to alternatives like page screenshots. for more information on installing and leveraging raintale, please consult its documentation. the future of storytelling with web archives we see a lot of potential for further development of these tools. this pilot project has helped us better understand the needs of our users. we continue to improve the error handling of these tools as we learn more from the community. overall, we want to improve the approachability of these tools. our native installers and web user interfaces are a start but only apply to linux. thanks to the experience of creating an installer for linux, we have some ideas for a macos dmg file that supports the applications of hypercane, mementoembed, and raintale. with wooey, we could preserve our existing command-line interfaces for scripting while also providing new web user interfaces. the authors of gooey promise this same functionality using the same methods but with a native non-browser graphic user interface. we are hopeful that our efforts with wooey might make it easier to apply gooey and provide a proper native gui interface. as we build upon these technologies, we are charting a path toward making the dsa toolkit a set of microsoft windows desktop applications. as a set of python applications, the dsa toolkit will have to contend with library installation issues on windows. once we resolve those issues, we will have reached the maximum number of users with the current code. we are also exploring additional functionality. aiu, and, by proxy, hypercane now support archive-it, trove, and pandora. we can expand it to support web archive collections from the croatian web archive, the uk web archive, the library of congress, and conifer. this way, users can apply our tools to summarize more collection types. of course, we would continue to support sets of memento urls as input because that would extend the reach to web archives that do not currently support collections. the hypercane recipes inspired by our collaboration are just the beginning. we are exploring the idea of a “recipe builder” where users can create their own hypercane scripts but in a much more user-friendly fashion than shell scripting. once this is in place, we hope that users will share their recipes. we could even facilitate a recipe exchange for interested archives that want to share hypercane solutions. raintale supports templates so that archivists can customize their stories. we see a future where archivists can share the look and feel of their stories with each other. in addition to templates, we are considering the idea of a “story builder” allowing users to create stories graphically in real-time, much like bloggers do with wordpress blocks. our collaboration between lanl, odu, and the nla has been fruitful but was only the beginning. either by executing these tools on your own or visiting someone else’s stories, what gems will you discover in a web archive collection? acknowledgements our many thanks go to the iipc for funding the dsa pilot documented here. we also thank the imls for funding the dsa project through their storytelling (lg-71-15-0077-15) and cedwarc (re-70-18-0005-18) grants. this research was supported by the information science and technology institute and by the laboratory directed research and development program of los alamos national laboratory (lanl) under project number 20210529cr. lanl is operated by triad national security, llc, for the national nuclear security administration of the u.s. department of energy (contract no. 89233218cna000001). about the authors shawn m. jones (0000-0002-4372-870x), los alamos national laboratory, is an information science & technology institute postdoctoral fellow working for lanl’s information sciences (ccs-3) division. shawn recently received his ph.d. from old dominion university and was advised by dr. michael l. nelson. he is also an alumnus of odu’s web science and digital libraries research group. he has contributed tools, data, and analysis to projects and frameworks such as memento, signposting, and robust links. in addition, shawn is the lead of the dark and stormy archives project, an initiative to summarize web archive collections through visualization techniques common in social media. more information about shawn is available at: https://www.shawnmjones.org/. himarsha r. jayanetti (0000-0003-4748-9176) is a ph.d. student at old dominion university working under the supervision of dr. michele c. weigle. she is also a member of the web science and digital libraries research group as a graduate research assistant. her research interests are in digital preservation, digital libraries, and social media. she graduated from gujarat technological university in india with a bachelor’s degree in computer engineering in 2017. more information about himarsha is available at: https://himarshaj.github.io/. alex osborne develops open source tools for web archiving and maintains the infrastructure of the australian web archive at the national library of australia. paul koerbin is assistant-director web archiving at the national library of australia. he has been involved with the development and operation of the nla’s web archiving program since its inception in the 1990s, including being part of the team that developed one of the first workflow systems for selective web archiving. he has published papers and given many presentations, in australia and overseas, on web archiving operations and practice. he holds a graduate qualification in library and information studies from the university of tasmania and a ph.d. from the university of western sydney. martin klein (0000-0003-0130-2097), los alamos national laboratory, is a scientist and lead of the prototyping team in lanl’s research library. in this role, he focuses on research and development efforts in the realm of web archiving, scholarly communication, digital system interoperability, and data management. he is involved in standards and frameworks such as memento, resourcesync, signposting, and robust links. martin holds a diploma in computer science from the university of applied sciences berlin, germany, and a ph.d. in computer science from old dominion university. michele c. weigle (0000-0002-2787-7166) is a professor of computer science at old dominion university. her research interests include web science, social media, digital preservation, and information visualization. she has published over 115 articles in peer-reviewed conferences and journals and has served as pi or co-pi on external research grants totaling almost $6m from a wide range of funders, including the national science foundation, the national endowment for the humanities, the institute of museum and library services, and the andrew w. mellon foundation. she currently serves on the editorial boards of the journal of the association for information science and technology (jasist) and the international journal on digital libraries (ijdl). dr. weigle received her ph.d. in computer science from the university of north carolina in 2003. michael l. nelson (0000-0003-3749-8116) is a professor at old dominion university and the virginia modeling, analysis, and simulation center (vmasc), and he co-leads the web science and digital libraries research group.  prior to joining odu, he worked at nasa langley research center from 1991-2002, where he created the nasa technical report server (ntrs).  more information about dr. nelson can be found at www.cs.odu.edu/~mln/ and twitter.com/phonedude_mln. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – the martha berry digital archive project: a case study in experimental pedagogy mission editorial committee process and structure code4lib issue 17, 2012-06-01 the martha berry digital archive project: a case study in experimental pedagogy using the martha berry digital archive project as an exploratory case study, this article discusses experimental methods in digital archive development, describing how and why a small project team is leveraging undergraduate student support, a participatory (crowdsourced) editing model, and free and open source software to digitize and disseminate a large documentary collection. by stephanie a. schlitz and garrick s. bodine introduction housed and maintained by the berry college archives at berry college in rome, georgia, the martha berry collection comprises over 160 file boxes, each containing somewhere between two and more than fifteen file folders. spanning the years 1885 to 1941, the collection, which is primarily epistolary, is organized chronologically by year and therein folders are indexed alphabetically by author. perhaps half of the documents were written by berry college’s founder, martha berry, the remainder by a profoundly diverse set of correspondents, including school children, business leaders, philanthropists, literary figures, friends, politicians, heads of state, and educators. as recently as spring 2011, the collection was extant exclusively in its original print format.  document-level bibliographic descriptions and metadata did not exist, and the inherited collection-level index, which consisted of a word processor document containing a serial listing of each file box and its approximate contents, was unreliable. the collection’s bibliographic state precluded broad access, and, materially, the documents were at considerable risk of deterioration. despite these challenges, stakeholders at all levels agreed the collection must be digitally preserved and disseminated. the fact that the digitization process can be technically demanding, time-consuming, and expensive is not only invisible to many stakeholders and end-users, but also to some extent immaterial. if we want our collections disseminated, discoverable, and used, we have little choice but to embrace and leverage the digitization process, perhaps especially when significant impediments exist. in the specific case of the martha berry digital archive (mbda) project, restrictive budgets, limited staff, and a large number of unedited documents presented us with an opportunity to explore methodological innovations and to pioneer new methods of archival development that would support not only the berry project but also creators and stewards of other collections who are confronted with similar resource challenges. we adopted an approach which designates experimentation and archival development as equivalent in importance to our project end goal, the digital archive itself. this case study describes three experimental, process-driven decisions we’ve made to advance development, including platform selection,  hiring and training undergraduates, and designing and developing a participatory (i.e. crowdsourced) editing model.[1] platform selection mbda required selection of an infrastructure that would be user-friendly and non-intimidating to project faculty, staff and students. project deliverables, which include preservation of the martha berry collection and publication of mbda, demanded selection of a mature, extensible, and well-maintained platform, a substrate we could customize and extend to serve as an archival hybrid comprising an accessible repository behind-the-scenes and a public-facing, interactive digital archive on the front-end. mbda's hybrid infrastructure requires the advantages of powerful, often complex, digital repositories (such as fedora commons) as well as those of common, easy-to-use web publishing platforms (such as wordpress or drupal) in order to empower non-specialists to make contributions to the underlying repository our repository short list included fedora commons, drupal, and hydra, but we ultimately selected omeka, “a free, flexible, and open source web-publishing platform for the display of library, museum, archives, and scholarly collections and exhibitions”.[2] although in itself not a digital object repository, omeka’s architecture and extensibility lend it to performing this role admirably. in contrast to robust but arguably less user-friendly digital object repository environments such as fedora commons, which does not currently have an extensively built out user interface and which does not have a public facing front-end, omeka advertises “five-minute setup,” facilitates web publication, is user-friendly, and comes with pre-configured publishing outputs and themes. the ability to harness existing omeka plug-ins such as dropbox, oai-pmh repository, and simple vocab further contributes to omeka’s appeal. yet perhaps more important to our venture is the fact that omeka has a relatively mature, open plug-in architecture and its layout is thoroughly themable. the omeka project’s open-source nature makes it possible, without having to start from scratch, to build features for mbda’s unique use cases that don’t currently exist in any of the archive and publishing systems we considered. developing new functionality as a plug-in, as opposed to simply modifying or forking the omeka codebase, allows us to program in parallel with omeka’s future directions in a more predictable and less conflicting way, i.e. we can continue to take advantage of future improvements to omeka’s core without contamination from our separate codebases, ensuring a relatively clear upgrade path for both the plug-in and the core omeka system it runs on. omeka’s api is exposed to plug-in developers via numerous functions, hooks, and filters documented on the omeka website’s documentation wiki. while from a programming perspective the documentation of particular features varies in scope from extensive to absent, there is an active community of developers and users whose work and listserv discussions remain a point of reference for mbda. and omeka’s cooperative documentation model allows us to reciprocate by filling in the gaps in documentation we’ve had to bridge along the way. imaging student-driven imaging model berry college is a small liberal arts school with a student population of less than 2000. the school’s comprehensive work study program, which offers meaningful, real-world employment to every student who is willing to work, is a hallmark of the berry experience and one of martha berry’s enduring legacies. training and paying undergraduate students to image their school’s eponymous collection was not merely a practical and affordable solution, it was a strategic one. our student-driven imaging model is designed to achieve mbda-specific goals and to expand them by supporting collaborative, experiential-based learning and an opportunity for undergraduates to create links between their study in subjects such as literature and history and their work with primary source documents in the martha berry collection. workflow documentation imaging can be demanding, tedious work, and its critical importance to document preservation and to the digital archive lifecycle cannot be overemphasized. because document scans are the backbone of mbda, attention to detail, accuracy and vigilant documentation during the scanning process are paramount. acknowledging the challenges inherent to imaging and seeking to ensure the success of our student participants, we created and vetted an imaging priority list which identifies a thematically coherent sub-set of the martha berry collection to serve as a preliminary ‘core sample’ for development, test, and initial publication and we devoted several weeks to hands-on, document-level imaging workflow design. design and documentation of every step within the imaging workflow was essential, and, among other details, our model includes: limiting scanning shifts to 1 to 2 hours use of a priority guide to identify which box and folder to scan how to handle fragile, hundred-year old documents filenaming conventions which create a one-to-one correspondence between a digital object and its material exemplar how to save to the project’s remotely accessible file share documenting for other members of the scanning team where a shift started and stopped in document terms the mbda imaging guide is both meticulous and concise, including step-by-step instructions, a screen capture to illustrate nearly every step of computer-based functions, and beginning and end of process reminders. students tested and critiqued the guide for ease of use (could they follow the instructions on their own?) and clarity (were there any ambiguities? did they need additional visuals?), and we edited accordingly. we also created ancillary materials including a spreadsheet for overall progress tracking and another for day-to-day documentation. page 2 of mbda’s imaging guide we monitor progress, quality, and accuracy, and a berry college library staff member provides general oversight. however, the berry college undergraduate imaging team has, since june 2011, been managing the imaging process largely independently, has continued virtually error free, and has successfully trained new student participants. from imaging to digital archive document scans are processed and uploaded to the mbda development site via a simple multi-step workflow. the workflow begins with a mogrification script which generates jpeg derivatives (compressed image files suitable for web-based archival usage) from the original tiff-formatted images created by the scanning team (and retained by the berry college archives as official, uncompressed digital representations of documents). once the images are mogrified, we ftp files from their storage location at berry college to the development server at bloomsburg university and change the file permissions to enable ingestion within the development site. together, mogrification and ftp batch upload subvert the need for individualized/manual image manipulation or upload. next, in order to edit and publish a scanned document within mbda, it must be converted to an item (item, in omeka parlance, is analogous to the frequently used term digital object) and assigned a unique identifier (item number) and item-specific constituents (e.g. dublin core metadata, tags, associated binary or media files, etc.). we use the dropbox plug-in to automate the conversion process so that once image derivative files are batch uploaded to the appropriate folder within the mbda database, we can use a simple ‘check box’ interface to select hundreds of files for simultaneous, automated conversion. as files are converted, each is assigned the requisite item number and constituents, as well as any universal tags we ascribe. because documents in the martha berry collection share several descriptive characteristics, we also modified the dropbox plug-in so that as files are converted to items, some single-value, archive-wide metadata fields are auto-populated. for example, metadata for the dublin core (dc) element fields language (all the documents in the collection are in english), identifier (the identifier for mbda items is based on their physical filing location within the collection, and the file name given to each scanned image is aligned with this location), source and publisher (both berry college) is automatically entered as items are created. in parallel with our student-driven imaging model, we designed a metadata editing scheme, developed detailed documentation, and hired and trained a small team of undergraduate students as digital editors. once document scans are ingested within the digital archive, student editors can review documents, contribute dc metadata, and match orphaned documents with their parents to ensure that multiple pages of the same document are reunited in the virtual world. students are able to reunite the pages of a document despite the fact that every document scan (most often in the case of mbda documents, this is a single side of a single page) is mogrified and then batch uploaded as an individual document even if it represents but one page of a lengthier multi-page document. we also created a flag for review element (discussed below) so that if editors identify an important document, an image problem, a filenaming error, or other issue while they’re editing, they can easily flag it for administrative review. experimental pedagogy platform selection, dev site development [3], and implementation of the imaging workflow served as the first steps toward liberating the martha berry collection from its locus in situ and behind closed doors (a significant milestone where stakeholders are concerned) and laid the foundation for more serious and more adventurous design and development, including a participatory editing model and accompanying omeka plug-in. why participatory editing? the martha berry collection contains many, many thousands of documents. although we had prioritized boxes for imaging, we were well aware that any box within the collection may prove as important as any other. the absence of comprehensive documentation to describe the collection meant that we couldn’t predict where the documentary gems may be hidden, and we recognized that our stage one project goals must include creating the kinds of exposure and interactivity that could also support document-level description. in short, we needed to leverage this preliminary stage to get the collection catalogued. while some in the academic community remain skeptical about crowdsourcing, for over a decade, crane has been making the case for the expansion of editing beyond academia, and projects like transcribe bentham, nasa’s map mars, investigate your mp’s expenses, and most recently the national archives citizen archivist project reveal the immense, sustained interest in finding ways to involve the larger community in activities which have traditionally been restricted to the domain of scholarship.[4][5] from our perspective, not only did mbda stand to gain from crowdsourcing, we considered our experimentation an opportunity to advance emerging participatory editing practices and to carve new inflections to expand their utility. leveraging and extending omeka consistent with the open archives initiative protocol for metadata harvesting, omeka supports the dc descriptive metadata standard by providing a pre-configured set of dc element fields for manual or auto-generated population. using the simple vocab plug-in, we developed controlled vocabularies for seven dc elements, including: source: martha berry collection publisher: berry college memorial library rights: creative commons, attribution, non-commercial, no-derivatives format: image-jpeg language: english identifier: mb75_4_1_1 coverage: mount berry, georgia these element fields are automatically populated during initial object ingestion as described above. for collections where existing metadata is already available, omeka does have an oai-pmh harvester plug-in, which could perform largely the same function. a sub-set of dc elements, however, requires individualized document review and manual description, including: title description creator date type to facilitate administrative oversight and document searching and to buttress later stage tasks for mbda (which, as noted above, is a collection of letters to and from martha berry), we defined a new omeka item type: correspondence, which contains the elements script type (e.g. typewritten or handwritten) and document recipient in addition to the existing omeka document-related metadata. our newly-developed plug-in, crowd-ed [6], also adds a new element set to omeka: crowdsourcing metadata. the crowdsourcing metadata element set includes new metadata features related to crowdsourcing, such as enable crowdsourcing (i.e. whether or not to enable crowdsourcing for a given item) and flag for review (i.e. community participants and project staff can flag an item for administrator review and identify either a specific or general reason for the flag). description of the dc elements title, description, creator, date, and type as well as flag for review, recipient and script type metadata are crucial yet minimally complex aspects of the editing process. inviting students and community members to edit these, we determined, would aid in cataloging the collection, drive archival development, and provide meaningful editing tasks for participants. the crowd-ed plug-in thus enables students and community members to participate in editing these eight elements (as shown in the screen capture below), respects the controlled vocabulary set by site administrators (using the simple vocab plug-in), and provides site administrators the ability to choose which other omeka elements are available for community editing. screen capture of working prototype of the crowd-ed participatory editing interface developing the crowd-ed plug-in in planning mbda’s participatory editing model, we identified two main additions to the archive’s underlying omeka infrastructure required for our use-cases: a plug-in to provide extensive crowdsourcing functionality, and a custom, grid-based theme to simplify layout design, especially as a basis for the crowdsourcing plug-in’s rather complex editing forms and layout requirements. like most repository and archival publishing applications, omeka has been limited to an administrator-driven editing model. this means that once documents are uploaded or ingested into an omeka instance, edits can be made to items and their constituents exclusively by administrators. in many archival use cases, this data separation model, which delivers item content unidirectionally to end-users, is ideal. but in a participatory editing use case, which relies on students and community members to populate a set of defined metadata categories, facilitated, bidirectional communication is essential. by mediating bidirectionally between the digital archive and its end-users, crowd-ed’s design model extends the archive’s utility to enable guided, community-based, participatory editing. from a user perspective, crowd-ed simply presents editable forms that make it possible for authenticated site users to contribute information to the archive metadata for the crowdsourcing enabled items they encounter while browsing or searching the archive. when the crowd-ed plug-in is installed and enabled, users who are logged in are presented three paths to engage with the archive from the archive’s home page: browsing all of the published items in the archive, searching for particular items using metadata categories, or participating in the development of the archive. screen capture of (in-progress) mbda development site home page (marthaberry.org) screen capture of browse items display display of individual item the crowd-ed plug-in creates links to an item’s crowdsourcing interface on each displayed item to allow an authenticated public user to contribute information at any time. if a user chooses to follow the participate path, further explanation is given regarding the purpose and intents of mbda crowdsourcing, as well as instructions on how the user may participate in the most urgent or essential aspects of the project. users who do not have logins are directed to the create account page where they must create a new account before participating in editing tasks. because the crowd-ed interface allows non-administrative users to add and edit information in the crowdsource-enabled fields, the mbda project had to create custom validation routines (using omeka’s built-in validation hooks where possible) to allow for security and content control.  we are currently developing audit trail and archiving functionality to complete the safety features and to ensure continuity should there be an issue with ‘crowd control.’ in order to provide this outer layer of functionality, crowd-ed takes advantage of another important aspect of omeka’s extensibility: the fact that it’s built on the very powerful open-source zend php framework and library. this means that any objects or methods available from the zend framework are also available to omeka plug-ins as native functionality. the functionality provided by the library enables us to record who changed what data from what value for which item at which time. it also provides site administrators with the ability easily to revert any changes made via the crowdsourcing plug-in. flash mobs? by scaling out the editing process to support community-contributed metadata, crowd-ed solves an important archival development problem. even so, we acknowledge that engaging part-time, voluntary editors as remote partners in an archival project introduces unique challenges where accuracy and continuity are concerned. but for mbda, crowdsourcing has never been synonymous with ceding control or relinquishing quality. by contrast, crowd-ed is being designed to guide contributors’ efforts and to prevent potential inconsistencies in keywording or encoding. we’ve defined editing categories that take advantage of non-specialist knowledge (e.g. identification of document author, recipient and date) and have restricted some participant choices via controlled vocabulary. and we are in the process of developing simple and interactive user access points and creating ‘how to’ tutorials. we also recognize that although crowdsourcing isn’t new, it is relatively new within the archival editing community, and the jury is still out on the value of participatory editing as a method within digital editing projects. while healthy skepticism and astute design schemes are among our best defenses against the unruly flash mobs some critics of crowdsourcing fear, we think it would be a mistake to dismiss the method untested and in doing so to dismiss the possibility of significant advancement and valuable community engagement. teaching digital editing = pedagogy although the ‘crowd’ employing our crowdsourcing effort to date has been a small one, our five-student, part-time imaging team has scanned more than 15,000 documents in less than a year, and our three-student, part-time digital editing team has matched thousands of documents and edited hundreds more. our students are learning about history, language, and politics (as well as their alma mater), and they are gaining valuable experience with a primary source collection and digital editing. their contributions have been integral to development of mbda, and the once moribund martha berry collection is thriving. while the real test of our methodology won’t begin until we advance beyond the development stage and begin public participatory editing this fall, mbda is already being discussed in college classrooms, explored by historians, and is serving as a pilot collection for innovations in ocr and transcription (up next on our to do list). notes [1] within the digital editing community, participatory editing is used as a de facto designation for the crowdsourcing of various editing tasks. this discussion uses participatory editing and crowdsourcing interchangeably to describe engaging students, community members, and the broader public in digital editing tasks. [2] about. [internet]. omeka. [cited 2012 april 5]. available from: http://omeka.org/about/ [3] in the spirit of transparency, to facilitate testing, and to permit pre-publication use within several college classes, we elected to leave the dev site open for the duration of the development lifecycle, despite the challenges doing so creates. [4] crane, gregory and jeffrey a. rydberg-cox. new technology and new roles: the need for corpus editors. in: acm 2000 digital libraries: proceedings of the fifth acm conference on digital libraries; 2000 june 2-7; san antonio, texas. p. 252-253. available from: http://hdl.handle.net/10427/57002 [5] crane, gregory. give us editors! re-inventing the edition and re-thinking the humanities [internet].  in: mcgann, jerome, editor. online humanities scholarship: the shape of things to come. connexions website. 8 may 2010. available from: http://cnx.org/content/col11199/1.1 [6] our first task was to architect and implement the crowd-ed plug-in. having done so, we’re currently working on somewhat less difficult (at least from a technical stance) though no less important (functionally) issues such as interface styling, testing, and end-user help text and tutorials. we anticipate the beta release of the crowd-ed plug-in during summer 2012. about the authors stephanie a schlitz, a berry college alumna, is an associate professor of linguistics at bloomsburg university of pennsylvania and director of the martha berry digital archive project. her email address is sschlitz@bloomu.edu. garrick s. bodine is an it manager for a software development team at pennsylvania state university-university park and lead programmer for the martha berry digital archive project. subscribe to comments: for this article | for all articles 3 responses to "the martha berry digital archive project: a case study in experimental pedagogy" please leave a response below: jonathan rochkind, 2012-07-20 while fedora et al are difficult to set up and use and often do not offer good ui’s — they, in theory, if i understand it right, have paid quite a bit of attention to digital preservation, to doing certain things to make sure your digital artifacts stay accessible and usable for, well, a while. (i could also be over-optimistic about fedora et al’s capabilities here, i do not have a lot of experience in this area). since you are calling your project an “archive”, did you consider issues of long-term digital preservation? does omeka offer anything relevant there? garrick bodine, 2012-07-30 thanks for the comment and question– in attempting to preserve *and* disseminate the documents in the berry collection, we considered both aspects pretty extensively to be sure. in our preliminary evaluations of potential software, we didn’t find omeka to purport to handle long-term preservation as one of its main use cases, and haven’t really found anything more in our initial implementation either. this forced a reassessment of how to handle long-term preservation, and we turned it into as a secondary requirement/aspect of the project given the limited resources and time available. our plan therefore basically breaks down as follows: sticking to very common digital imaging formats (tiffs, jpeg derivatives) we feel will provide enough long-term accessibility for the foreseeable future (at least until such time as another project can focus on long-term preservation as *primary*); as to the metadata, that too we feel is sufficiently open and static (textual data in a standard, documented database table in a popular, open-source rdbms) that it is foreseeably-future-proof until/unless additional resources are put toward converting it further. basically, i think that both of these cases leave us with data that can be easily (i.e. *programmatically*) transformed or exported into yet another format/configuration/implementation using an almost limitless number of methods or tools currently available. there really isn’t anything about omeka that locks the data or metadata into any particular form, and the forms we chose were based on their presumed availability moving forward. i hope that helps somewhat but if we can clarify any of these points or if you have additional insights, we’d welcome further discussion. stephanie schlitz, 2013-05-22 project update: mbda has moved to its permanent host. you can now find us at: https://mbda.berry.edu/ project code is available at: https://github.com/gsbodine?tab=repositories leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – using airtable to download and parse digital humanities data mission editorial committee process and structure code4lib issue 58, 2023-12-04 using airtable to download and parse digital humanities data airtable is an increasingly popular cloud-based format for entering and storing research data, especially in the digital humanities. it combines the simplicity of spreadsheets like csv or excel with a relational database’s ability to model relationships and link records. the center for digital research in the humanities (cdrh) at nebraska uses airtable data for two projects, african poetics (africanpoetics.unl.edu) and petitioning for freedom (petitioningforfreedom.unl.edu). in the first project, the data focuses on african poets and news coverage of them, and in the second, the data focuses on habeas corpus petitions and individuals involved in the cases. cdrh’s existing software stack (designed to facilitate display and discovery) can take in data in many formats, including csv, and parse it with ruby scripts and ingest it into an api based on the elasticsearch search index. the first step in using airtable data is to download and convert it into a usable data format. this article covers the command line tools that can download tables from airtable, the formats that can be downloaded (json being the most convenient for automation) and access management for tables and authentication. python scripts can process this json data into a csv format suitable for ingesting into other systems the article goes on to discuss how this data processing might work. it also discusses the process of exporting information from the join tables, airtable’s relational database-like functionality. join data is not human-readable when exported, but it can be pre-processed in airtable into parsable formats. after processing the data into csv format, this article touches on how cdrh api fields are populated from plain values and more complicated structures including markdown-style links. finally, this article discusses the advantages and disadvantages of airtable for managing data, from a developer’s perspective. by william k. dewey using airtable to download and parse digital humanities data airtable is a cloud-based, software-as-a-service platform for entering and storing data commonly used for academic research, including in the digital humanities.  it is commercial subscription-based software, using a proprietary format but with the ability to export into open formats. it can best be compared with spreadsheets (like google sheets or excel) and relational databases (like mariadb), as it combines aspects of both types of applications (airtable platform [updated 2023])[1]. although it is marketed to businesses, academic and in particular digital humanities projects are using airtable for its combination of spreadsheet-style data entry with relational modeling capabilities. airtable’s advantages include ease of use for students and project assistants, who can easily add information related to projects and link related records together. airtable compared with spreadsheets at first glance, airtable is quite like spreadsheet applications, as information can be entered into cells, rows, and columns. but its data model is not simply a flat text file like csv. fields can be populated with many data types, including dates, integers, rollup and lookup fields, and can include multiple values and controlled vocabularies (supported field types… [updated 2023])[2]. like excel, airtable provides programmatic functionality with formulas (formula field reference [updated 2023])[3]. for more involved programming involving multiple records and tables, there are automations, which might look up records and create other records based on them. it is not necessary to know a programming language to do this, because airtable provides a graphical interface to define the automation step by step (airtable automations [updated 2023])[4][5]. there are features more like a sql relational database than a spreadsheet, including the ability to link related records together using foreign keys and join tables. lookup fields provide the ability to transform data from linked records into more useful forms (linking records…[updated 2023])[6]. like many database platforms, airtable allows access privileges to be managed. the platform also provides the ability to create a workspace for an organization. within an organization, one can create a project base (a set of related tables), and give users different access privileges to it. one can grant view privileges only, editing privileges, or ownership permission to do tasks like create and delete tables. any collaborator can be given an api access token so that they can download airtable data from the command line or with scripts (airtable base collaboration…[updated 2023])[7]. airtable compared with relational databases airtable has advantages over relational databases. while sharing functionality like join tables, queries, defined data types, and programmatic capabilities, airtable requires much less coding knowledge. knowledge of sql or other programming languages is not necessary to enter or query records, create join tables or even to create formulas. the online interface provides functionality for entering data, linking records together, and adding formulas, although formulas still require some knowledge of markup and syntax (formula field reference [updated 2023])[8]. array functions, for instance, are one way to use lookup fields to redisplay joined records, and airtable’s gui will provide guidance in this transformation (using array functions… [updated 2023])[9]. unl’s center for research in the digital humanities and its technical stack airtable is used at the center for research in the digital humanities (cdrh) to serve the center’s unique needs in transforming data for the cdrh api. the cdrh is a digital humanities center based in the university of nebraska-lincoln libraries that, in collaboration with researchers, assists professors in creating web applications for projects with humanistic content and data (about the cdrh [updated 2023])[10]. when the center was developed, projects primarily involved working with xml documents, but the center now works with more varied types of data including tables, images, and maps. airtable data was structured and used so that it would be compatible with the technical stack at the center (see figure 1). the center’s local infrastructure is designed to ingest and store data so that it can be displayed in web applications. the first component of the stack is orchid, a template for front end web applications, written in ruby on rails. it follows the standard model-view-controller pattern of ruby on rails, but instead of creating data models from a sql database, orchid pulls data from the cdrh api. orchid sites provide faceted search and discovery, display api entries in a standardized way, organize data into sections, and allow for the creation of static html pages (orchid [updated 2022])[11]. the cdrh’s api is based on ruby on rails, as well. it is a front end to an elasticsearch index (a search engine based on lucene) and provides methods to post and query data, and to view the raw data in a browser. figure 1. technical stack of the center (dalziel [updated 2018][12].). when a query is made through the search bar of an orchid application, the api also queries elasticsearch. the underlying data in elasticsearch is in json format, so the api schema needs to be structured in a way to convert the data into json, with datatypes expected by the elasticsearch index (api [updated 2022])[13]. datura is the cdrh’s system for populating the api with data. it posts data into the elasticsearch index via the api from source files that are in a variety of formats. datura creates a json hash which is populated field-by-field with simple ruby functions, which take data from the document being ingested. the values of nested fields must themselves be in json format. ruby’s json libraries can help here but sometimes it is necessary to do more transformations to make sure data is parsable as json, such as removing illegal characters. an example of a nested field is “person”, which has subfields that include names (first, last, and alternate), birth and death date, and demographic information (sex, age, and race). most of the formats datura can ingest are xml based, commonly tei (text encoding initiative) documents. datura also has the capacity to ingest csv, a simple process of adding each row as a separate record into the api, with each column representing a field. although it would appear simpler to directly use the json from airtable directly (keeping in mind that it would still have to be converted to match the api schema), datura does not currently have the capacity to ingest json (datura [updated 2023])[14]. projects using airtable: petitioning for freedom and african poetics the first cdrh project in which airtable was used was petitioning for freedom. this project, funded by an nsf grant, focuses on petitions for habeas corpus in the american west from 1785 to 1920, and how they were used to challenge often unjust situations of confinement or coercion. although the principal investigator (dr. katrina jagodinsky) has her own interests (including critical legal theory, race, slavery, immigration, gender, and family law), the project is intentionally designed so that it is useful to scholars with other interests. a wide variety of cases were included from the states of iowa, washington, kansas, missouri, and nebraska (jagodinsky et al. 2023; see figure 2 for an example of a case page)[15]. apart from some xml-encoded case documents, most of the data consists of summaries and metadata about the cases in spreadsheet form. the first draft of petitioning for freedom involved a simple sql database, linked to ruby on rails with active record (the part of rails that creates ruby objects from the database), and seeded in the `db/seeds.rb` file from a csv file. seeding the database manually via ruby scripts is not a good model for data entry for even a relatively simple data set. such a script lacks much of the basic functionality of a spreadsheet application, like the ability to view the data in tabular form, sort it and add formulas. for this reason, the project transitioned to storing data in google sheets, from which data could be exported in csv format and then exported into the cdrh api (habeascorpus [updated 2023])[16] [17]. this solution made greater use of the cdrh’s existing functionality; in particular, using the existing api schema meant there was less of a need to customize the database and there was greater control over the data model. however, the csv-exported spreadsheets were not rich enough for the data and relationships dr. jagodinsky wanted to model, and promised to model in the related project grant. airtable provided much of the needed functionality like join tables, and it was an easy platform for data entry for both students and staff (see figure 3 for part of the “cases” table, including the entry underlying figure 2). formulas were worked out to export join data in readable form, although this proved to be an unwieldy process. figure 2 – petitioning for freedom front end, displaying information about a habeas corpus case. (in the matter of…[updated 2023])[18]. figure 3 – airtable back end, displaying the cases table, and information regarding individuals and the roles they played in cases, the underlying data behind figure 2. the second project for which airtable was used for data storage was the african poetics digital portal (adpd), a project funded by the ford and mellon foundations. the adpd is the digital portion of kwame and lorna dawes’ african poetry book fund dedicated to promoting african poetry and related scholarship (dawes 2023)[19]. airtable was most extensively used for the african poetics in the news project. it is a comprehensive account of coverage of african poets in british newspapers (most predominantly the times of london and sunday times) from 1865 to 1985 (see figure 4 for an example of a page on a poet). most of the data, aside from newspaper images, was in tabular or textual form. the extensive dataset included news articles, events, poets, and works, and select critical commentaries written by students (dawes et al., 2023)[20]. the data items are related to each other in complex ways, and the data model uses hidden tables containing data such as locations. originally this data was in an even more complex set of sql databases, and data entry was handled by a rails application which was not intuitive for most users. it was decided to simplify the data model and move it into airtable (figure 5), handling join tables in similar fashion to petitioning for freedom. scripts were written to download the data from airtable, process it for the cdrh’s api, and the front end was revised to use the new data schema (african_poetics [updated 2023])[21]. figure 4 – african poetics digital portal front end (delius, anthony ‘ronald’…[updated 2023])[22]. figure 5 – airtable back end for african poetics, showing some of the data for people   exporting data from airtable one may desire to export airtable data offline. a goal of cdrh projects is to be able to distribute data publicly, and update project data in a way that is fully automated, consistent and reproducible. however, as a proprietary cloud-based format, one cannot download or save an “airtable table” for offline use. the tables can only be viewed with an account or personal access token to airtable’s api. fortunately, it is possible to export airtable tables into more common, open-source formats like csv and json. the csv format is generally better for sharing data, as it is more human-readable, and most spreadsheet software can open it. however, csv can only be downloaded via the web interface and is not available programmatically (airtable views [updated 2023])[23]. other cdrh projects have used processes that involve manually exporting spreadsheets from an application, but such a process proves to be unreliable. downloading airtable tables to json can be automated, as json can be downloaded via the airtable api , either by making curl requests or using command line tools. json can also be transformed into the csv format needed for the datura scripts. for these reasons the cdrh preferred the json format for exporting from airtable. the ‘airtable-export’ python module is one way to download tables as json from the command line, requiring an airtable base id and a personal access token (willison 2023)[24][25]. the token can be generated from the website, [26]replacing the deprecated api key (zhao 2023)[27]. access tokens work in the same way, so this change posed no difficulties in using the `airtable-export` script (personal access tokens, updated 2023)[28]. the base id can be found after creating an account, by viewing the api documentation,[29] clicking on the name of the base, and finding the id after “the id of this base is.” (api reference, updated 2023)[30]. this tool is incorporated into the python scripts as shown below, [31] but it could also simply be used from a terminal command line. command = f"airtable-export source/json {airtable_base_id} cases people --key={api_key} --json" for the petitioning for freedom project, the base id and access token are stored as environment variables in a .env file (not committed to github) using the module python-dotenv (kumar 2023)[32]. similar processes are used in other projects. despite the wide variety of data types in the airtable table, the values of the cells in json generally come in one of three forms, a string, an array, or a coded reference to a linked record. linked records appear as meaningful names in the table but are encoded as alphanumeric strings in the exported json, in a key that is not available in any exported data, making it difficult to use the data outside airtable. for instance, the case roles column in the cases table (linked to the case roles join table), displays information about what individuals played what role in the case, but it looks like this when downloaded as csv: "case role [join]": [ "rec1gh0qf5um5q93j", "recvnhfc6zgtkwcsd", "recmhnoslta5rkzhz", "rectoen7otienziuq", "recwtdz7zpkwf7ny1", "reclcnp4ebbrglxqn", "recinbuvyvwvjvtud" ] it is possible, but not necessarily practical, to match up ids manually with the linked table. as will be discussed later, a better way to get around this limitation of the download format is to create new airtable fields based on the linked records. processing airtable data for the cdrh api the scripts to download and preprocess the data prior to the api ingest vary in complexity. the data must be converted to csv for the datura scripts, which will convert them to a json format that can be consumed by elasticsearch. in the case of african poetics, a lot of preprocessing was done in a simple python script, which constructed json and csv files from airtable api data [33]. instead of downloading json directly it was built up step by step by api requests to airtable. records = {} url = server+base_id+'/'+table headers = {'authorization':'bearer '+api_key} params = {'view':'grid view'} res = requests.get(url, params = params, headers = headers) data = res.json() for record in data['records']: record['fields']['airtableid'] = record['id'] records[record['id']] = record the json files were then written to csvs with the csv python module, using the hash values as the header. header = list(table_fields[table].keys()) if table in ["commentaries", "events", "news items", "people", "works", "contemporary poets"]: csvfile = open('source/csv/'+table+'.csv', 'w') else: csvfile = open('scripts/airtableexport/csv/'+table+'.csv', 'w') writer = csv.dictwriter(csvfile,fieldnames=header,extrasaction='ignore',quoting=csv.quote_all) the ingest into the api did not involve an especially complicated data model. at most the api expected nested fields like “person”, which took the shape { name: name, role: role, id: id }. on the airtable side, it was necessary to use airtable formulas to combine the join data into a field that was more easily readable, like [krog, antjie](apdp.person.000980). data processing with pandas scripts the petitioning for freedom project has a more complex data schema, so extensive data processing was needed to get the data into a shape that the cdrh api could consume. python’s pandas module, which is one of the most common tools for data manipulation and analysis, turned out to be one of the best tools for that purpose. it has strengths in converting json to csv and manipulating data frames [34]. after downloading the json files via ‘airtable-export,’ the files are read into a pandas dataframe, and, following all the necessary transformations, the dataframe is turned back into csv. the scripts are intended to modify the csv so that datura scripts can post the data into the api, without causing errors or storing undesired values. there were a lot of columns that contained unwanted airtable metadata and the script dropped them with cases_frame.drop(columns=[“encoding notes”, “last modified”,…]. some columns need to be made parsable by ruby’s json plugin. for instance, a list of values might appear as the string “[0, 1, 2, 3, 4, 5]” and ruby can parse it as json and turn it back into an array. a challenge in this process was that python introduces single quotes into such fields, which is not legal json. the solution is to apply json.dumps in the python script on every field that needs to be parsed as json. for label in ["petition type", "site(s) of significance", "tags", "petitioners", "rdf person role case (from case role [join])", "petition outcome", "fate of bound party(s)", "court name(s)", "source material(s)", "bound_party_age", "bound_party_race", "bound_party_sex", "repository", "case state"]: cases_frame[label] = cases_frame[label].apply(json.dumps) the conversion process creates blank and nan values, which can cause parsing errors or unwanted values in the api, so it is necessary to strip them out. cases_frame = cases_frame.fillna('') this code replaces nan values with an empty string, which will not be ingested into the api exporting data from join tables linked records and join tables pose further challenges for those who want to reuse airtable data, and additional data processing may be necessary to make the exported data usable. the airtable application groups records together in a way that expresses more relationships than an ordinary flat file spreadsheet. for instance, petitioning for freedom uses lookup tables for people and locations associated with a case. there are two join tables, “cases data” for roles played by persons in cases (e.g. petitioning attorney, judge), and “relationships” for person-to-person relationships (e.g. lawyer for, mother of, etc.). the cases data table also contains information on personal characteristics of petitioners (race, sex, and gender) from the original case records. the relationships table is linked to the people table twice to capture reciprocal relationships like mother and son. in african poetics, much of the original complex array of sql join tables, recording information like related works and associated locations, was made into airtable join tables. the challenge for programmers is that the linked records when downloaded appear as a meaningless list of alphanumeric strings, as described above. it is possible to download the join table itself and try to match records with ids, but such a process would be overly complex to script. a better solution is to create formulas to make new fields on the primary tables. airtable does provide rollup fields, which perform a simple function on linked records to create a readable field (rollup field overview [updated 2023])[35]. the petitioning for freedom and african poetics tables use the rollup feature and custom formulas to make fields on the primary tables that represent the linked records [36]. they are structured with delimiting characters (like pipes, semicolons, and so forth), which can be parsed by ruby and ingested into the api as arrays and hashes. two examples from african poetics illustrate what this parsable data looks like in airtable. a list of related people on the “people” table looks as follows: [campbell, roy](apdp.person.000514);;;[currey, ralph nixon 'r. n.'](apdp.person.000253);;;[madge, charles henry](apdp.person.000235);;;[plomer, william](apdp.person.000132);;; on the “works” table, with the addition of a role, the field looks like this: [brutus, dennis vincent](apdp.person.000779)|poet;;;[okigbo, christopher](apdp.person.001150)|poet;;;[peters, lenrie](apdp.person.000030)|poet;;;[clark, john pepper 'j. p.'](apdp.person.000828)|poet;;; markdown syntax ‘[name](id)’ is used to represent ids from the api and corresponding names [37] otherwise the strings are split on the characters “;;;” (for each person) and then “|” (separating the markdown from the role). the parsing script might look like this, with the variable people representing the data from airtable and the parse_ function being used to process the markdown: if people &amp;&amp; people.length &gt; 0 people = people.split(";;;") people.each do |person| data = person.split("|") if data[0] name = parse_brackets[data[0]] id = parse_parentheses[data[0]] role = data[1] result &lt;&lt; { name: name, role: role, id: id } end end end< here are more examples from petitioning for freedom, showing relationships between individuals, and the role an individual played in a case: "[\"pickard, aubrey\"](hc.pers.000945)|parent of|[\"pickard, leonard\"](hc.pers.000946)", [\"pickard, aubrey\"](hc.pers.000945)|unindicated|[in the matter of the petition of aubrey pickard for a writ of habeas corpus for leonard pickards](hc.case.ne.0735). however, this was a complex field to parse. when airtable formulas concatenate linked records as strings, they often put values in quotes and it is difficult to work around this behavior. the quotes also caused trouble at the stage of ingest into the cdrh api (where they were not desired in the data). the quotes were stripped out with pandas and python prior to the ingest, although another possibility would be using regular expressions in the ruby script. each of the fields with unwanted quotes was passed into a function to remove quotes: people_frame = remove_quotes(people_frame), 'person_case_year') the pandas function to remove the quotes was difficult to write, and some of the complexity arose from the fact that the values were nested within arrays such that they cannot readily be replaced. in the below pandas function, the arrays in the column are “exploded” (turning each array value into a separate row), and then quotes and nan values are replaced in each string. finally, the exploded arrays are aggregated again into their original places in the frame: def remove_quotes df[column]= df.explode(column)[column].astype(str).str.replace("\"", "").replace('nan', '').replace("none", '').groupby(level=0).agg(list) the complexity of this process ultimately arose from limitations in how airtable handles and exports data from join tables. assessing airtable as a data platform from the perspective of a backend developer, airtable has numerous advantages and disadvantages for storing and processing digital humanities data. it can model complex associations and relationships, like a relational database does, giving it an advantage over traditional spreadsheets. data entry does not require knowledge of sql or any other programming language, and is simple enough to be handled by non-programmers (like many cdrh students). the data can be authored and edited by a team working in collaboration. airtable makes it relatively easy to keep the data consistent, do deduplication, and so on. airtable tables can also be exported into open-source formats like csv and json, which can be imported into an api with simple data transformations. these advantages will mean that the center for research in digital humanities is likely to continue using it for its projects. airtable does have some disadvantages that might make a digital humanities center reconsider using it. it is a proprietary platform that can only be accessed through the website, by obtaining user permissions or an api key. airtable requires a paid subscription to start and maintain a base and add collaborators, which may be beyond the means of some institutions. with any proprietary platform there is a chance that it may change its terms in the future and make the platform less accessible and more costly. references between tables are not represented adequately in the exported data formats but shown as a list of unreadable random numbers. this accounts for much of the processing that must be done on the airtable side and backend scripts to make the data usable by apis and human readable. in addition, airtable does not have all the error checking/consistency checking features of a sql database, and many tasks (especially using formulas for transforming records) do require some programming knowledge. for this reason, it is better to use spreadsheets for simple projects that do not need linked records, and traditional relational databases if data entry can be handled by skilled professionals with programming expertise. the cdrh is exploring open-source alternatives to airtable including baserow and nocodb, although we have not yet ascertained to what degree they can replicate the functionality of airtable. future work will seek to discover and assess alternatives to airtable that are more sustainable and accessible. acknowledgements other members of the cdrh development team who worked on the petitioning for freedom and african poetics include karin dalziel (team lead and digital resources designer, who designed the airtable formulas and the schema for representing linked data), sarita garcia (who assisted with database design), andrew peterson (who managed the airtable system), greg tunink, erin chambers, and laura weakley. mike litwa of library it assisted in creating the download scripts for airtable. cory young served as project manager on petitioning for freedom. credit must also be given to the many student assistants who helped with data entry on both african poetics and petitioning for freedom. bibliography and notes [1] airtable platform [internet]. [updated 2023]. san francisco, ca: airtable [cited 2023 oct 1]. available from https://www.airtable.com/platform. [2] supported field types in airtable overview [internet]. [updated 2023 sep 18] san francisco, ca: airtable [cited 2023 oct 1]. available from https://support.airtable.com/docs/supported-field-types-in-airtable-overview [3] [7] formula field reference [internet]. [updated 2023].https://support.airtable.com/docs/formula-field-reference [4] however, formulas do require some knowledge of markup and programming, depending on their complexity. [5]getting started with airtable automations [internet]. [updated 2023 aug 23]. san francisco, ca: airtable [cited 2023 oct 1]. available from https://support.airtable.com/docs/getting-started-with-airtable-automations. [6]  linking records in airtable [internet]. [updated 2023 aug 23]. san francisco, ca: airtable [cited 2023 oct 1]. available from https://support.airtable.com/docs/linking-records-in-airtable. [8]airtable base collaboration overview [internet]. [updated 2023 aug 23]. san francisco, ca: airtable [cited 2023 oct 1]. available from https://support.airtable.com/docs/airtable-collaboration-overview. [9] using array functions in airtable [internet]. [updated 2023 sep 18]. san francisco, ca: airtable [cited 2023 oct 1]. available from https://support.airtable.com/docs/using-array-functions-in-airtable. [10]  about the cdrh [internet]. [updated 2023 jul 24]. lincoln, ne: university of nebraska, lincoln. center for research in digital humanities. [cited 2023 oct 1]. available from https://cdrh.unl.edu/about/aboutcdrh [11] orchid [internet]. [updated 2022 dec 19]. lincoln, ne: university of nebraska, lincoln. center for research in digital humanities. [cited 2023 oct 1]. available from <a href=”https://github.com/cdrh/orchid.”>https://github.com/cdrh/orchid.</a> [12] dalziel, karin. 2018. a new way to publish: the cdrh api [internet]. cdrh development blog. lincoln, ne: university of nebraska, lincoln. center for research in digital humanities. [cited 2023 oct 20]. available from https://cdrhdev.unl.edu/log/2018/api/. [13] [30]api reference [internet]. [updated 2023]. san francisco, ca: airtable [cited 2023 oct 1]. available from https://airtable.com/developers/web/api/introduction. [14] datura [internet]. [updated 2023 mar 23]. lincoln, ne: university of nebraska, lincoln. center for research in digital humanities. [cited 2023 oct 1]. available from https://github.com/cdrh/orchid. [15]jagodinsky, katrina, cory young, et al. 2023. petitioning for freedom. lincoln, ne: university of nebraska, lincoln. center for research in digital humanities. [cited 2023 oct 1]. available from https://petitioningforfreedom.unl.edu/. [16] habeascorpus. github [internet]. [updated 2023 sep 25] university of nebraska, lincoln. center for research in digital humanities. [cited 2023 oct 1]. available from https://github.com/cdrh/african_poetics [17] the github files are under the name habeas_corpus because an earlier working title for the project was habeas corpus. [18] in the matter of the application for a writ of habeas corpus for gwoo shee ah look. jagodinsky, katrina, cory young, et al. petitioning for freedom [internet]. [updated 2023 oct 21] lincoln, ne: university of nebraska, lincoln. [cited 2023 oct 22] https://cdrhdev1.unl.edu/habeascorpus/cases/item/hc.case.wa.0183 [19] dawes, kwame, et al. 2023. african poetry book fund [internet]. lincoln, ne: university of nebraska, lincoln. [cited 2023 oct 1]. available from africanpoetrybf.unl.edu. [20] dawes, kwame, and lorna dawes, et al. 2023. african poetry digital portal [internet]. lincoln, ne: university of nebraska, lincoln. center for research in digital humanities. [cited 2023 oct 1]. available from https://africanpoetics.unl.edu/. [21] african_poetics. github [internet]. [updated 2023 sep 25] university of nebraska, lincoln. center for research in digital humanities. [cited 2023 oct 1]. available from https://github.com/cdrh/african_poetics [22] delius, anthony ‘ronald st. martin’. african poetry digital portal [internet]. [updated 2023]. lincoln, ne: university of nebraska, lincoln. center for research in digital humanities. [cited 2023 oct 22]. available from https://africanpoetics.unl.edu/inthenews/poets/item/apdp.person.000001. [23] getting started with airtable views [internet]. [updated 2023 aug 23]. san francisco, ca: airtable [cited 2023 oct 1]. available from https://support.airtable.com/docs/getting-started-with-airtable-views. [24] willison, simon. 2023. airtable-export. github [internet]. [cited 2023 oct 1] available from https://github.com/simonw/airtable-export. [25] https://github.com/simonw/airtable-export. the tool can be downloaded vi pip [26] personal access tokens [internet]. [updated 2023 apr 20]. san francisco, ca: airtable [cited 2023 oct 1]. available from https://airtable.com/create/tokens (login required). https://airtable.com/create/token [27] zhao, frank. 2023. new api capabilities now in ga and upcoming api keys deprecation period. airtable community. [cited 2023 oct 1]. available from https://community.airtable.com/t5/announcements/new-api-capabilities-now-in-ga-and-upcoming-api-keys-deprecation/ba-p/141824?utm_id=recdxe5vjzz5vr0mh&utm_id=recdxe5vjzz5vr0mh [28] although the script references an “api token” which is now deprecated as a way of accessing airtable, personal access tokens work in the same way, so this change posed no difficulties in using the `airtable-export` script (personal access tokens, updated 2023). [29] see https://airtable.com/developers/web/api/introduction after logging in. [31] although the script references an “api token” which is now deprecated as a way of accessing airtable, personal access tokens work in the same way, so this change posed no difficulties in using the `airtable-export` script (personal access tokens, updated 2023). [32] kumar, saurabh. 2023. python-dotenv. github [internet]. available from https://github.com/theskumar/python-dotenv. [33] this script can be found at https://github.com/cdrh/data_african_poetics/tree/main/scripts/airtableexport [34] the script can be found at https://github.com/cdrh/data_habeascorpus/tree/dev/scripts/python/json_to_csv.py [35] rollup field overview [internet]. [updated 2023 jul 24] san francisco, ca: airtable [cited 2023 oct 1]. available from https://support.airtable.com/docs/supported-field-types-in-airtable-overview [36] the join table was not used directly in the ingest scripts, it is simpler to have each script parse a single table. [37] functions in datura can parse this syntax with regular expressions. for instance, to extract the name between the brackets: /\[(.*?)\]/.match(query)[1] if /\[(.*)\]/.match(query) about the author will dewey is programmer/analyst ii at the center for research in digital humanities in the university of nebraska, lincoln libraries subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – scope: a digital archives access interface mission editorial committee process and structure code4lib issue 43, 2019-02-14 scope: a digital archives access interface the canadian centre for architecture (cca) identified certain technological issues, namely extensive reference workflows and under-utilizing existing metadata, as significant barriers to access for its born-digital archives. in collaboration with artefactual systems, the cca built scope, a digital archives access interface. scope allows for granular fileand item-level searching within and across digital archives, and lets users download access copies of the collection material directly to a local machine. scope is a free, open-source tool. the beta version is available to the public, and a second phase is under-development as of spring 2019. by kelly stewart & stefana breitwieser introduction as archives increasingly have acquired born-digital material, traditional methods of collection access need to expand to include digital files. however, technical challenges can potentially frustrate archives’ best efforts at leveraging meaningful metadata for searching, as well as serving, these files to researchers. in response to these challenges, the canadian centre for architecture (cca) is launching scope, a browser-based access interface for digital material. scope has been developed in partnership with artefactual systems, a company based in new westminster, canada, and the lead contributor to two other open-source applications: archivematica and atom. as a free and open-source tool, we hope to offer the digital preservation community not only a potential solution for digital collections access, but also a case study for collaborative technical solutions. why scope? scope-ing out parameters for born-digital access the cca is an international research institution and museum focused on the study and practice of architecture; it produces exhibitions and publications informed by its extensive archives, library, photographs, and prints and drawings collections. beginning in 2012, the cca began investigating a new research question: “how did the introduction of digital technology affect architecture and architectural practice?” what followed was the archaeology of the digital project, culminating in two books, three museum exhibitions, and more than twenty-five e-publications. the acquisition of twenty-five archives with a significant digital component as a part of this project led to a different kind of research question for the cca archives staff: “how do we process, preserve, and make accessible more than 5tb of complex born-digital archival material?” preservation of these born-digital files was our starting point. the digital archives staff of five began to work through the five terabytes of born-digital archival material dating from 1988-2012, comprised of roughly one million files. the goal was not only to preserve and make accessible the material from these archives, but to also layout a blueprint for what an open archival information system (oais) compliant preservation system might look like at the cca. oais is the iso-standard for digital preservation, which requires institutions to accept materials from information producers, have control over the materials in such a way that ensures long-term preservation in a self-described or independently understandable way, follow approved policies and procedures, and finally make the materials available to the designated community of intended users (lavoie, 2014; schumann and recker, 2012). oais also defines different types of information packages that can be maintained over time: submission information packages (sips), or the material as it was transferred from the donor and prior to ingest in a digital preservation system; archival information packages (aips), or the preservation copy; and dissemination information packages (dips), or the access copy (lavoie, 2014). processing of the twenty-five archaeology of the digital archives followed the usual digital archival workflows: original digital storage media was disk-imaged using a bitcurator workstation. files were carved from the disk images, and then were arranged and described. tim walsh, cca’s former digital archivist, also developed a suite of open-source tools used for digital archival processing, including brunnhilde to characterize groups of files and to flag potential preservation issues, and cca tools to package sips in a uniform way and automate description. potential preservation issues and other archival processing considerations, like corrupted timestamps or duplicate files, were corrected or accounted for as much as possible through manual clean-up and minor bash scripting. automated description was then supplemented by the digital processing archivists, following the general international standard archival description (isad-g) and local standards. the metadata was then entered into cca’s database of record, the museum system (tms). following processing, the material was ingested into archivematica, the digital preservation system built by artefactual. archivematica enacts a number of preservation micro-services (including identifying files and generating related metadata), describes each of these micro-services as a premis event, and packages this information into a mets file stored with the aip. archivematica does fixity checking on stored aips to ensure the files remain the same over time, and generates dips for user access. figure 1. screenshot of micro-services performed by archivematica on a particular sip. premis metadata is generated for each of these services, and put into a mets file. however, even though the cca’s aips were well-preserved and granularly described, we found that providing meaningful researcher access to the files and their descriptions was difficult. we were confronted with two problems. first, the existing workflow for serving the files to researchers was becoming unsustainable in that it was time-consuming, labor-intensive, and required input from many stakeholders. that workflow was as follows: a researcher made a request for the digital material and reference staff forwarded the request to the digital archives team. an archivist queried the archivematica aip storage server to generate a dip on the local machine, moved the dip to a shared folder with the researcher, and finally unzipped the files. the archivist then notified the researcher and reference, which often resulted in a consultation on how to use the files. in instances where dozens of dips were requested, this workflow could take hours. though born-digital archival material has only seen modest use as a percentage of cca’s total researchers, use has still more than doubled in the two years we have made the material available with four external researchers in 2017 and ten in 2018. internally, cca’s interest in digital architectural materials has remained strong across its programming and publishing and as an institution has committed itself to increasing visibility and expanding scholarship around these materials. we saw this moment as an opportunity to improve access workflows for these materials in order to better accommodate research as it reaches its expected critical mass in the coming years. our second problem was that traditional archival research methods seem to apply less to digital materials. historically, archival research has been conducted in the top-down method dictated by finding aids, starting at the collection-level description and moving downwards until the appropriate material is found. this method works well for traditional qualitative research questions: “what drawings do you have for zaha hadid’s phaeno science center?” however, the digital humanities occasionally have shifted archival research to a more quantitative focus: “i’m trying to discover how 3d modeling software has evolved over time. can i look at every rhinoceros file made in 2002 from across all collections?” being able to answer these types of questions would require both a deep individual knowledge of our digital holdings and endless time pouring through finding aids, an increasingly unscalable expectation due to the growing volume of available digital material. aggregate description typically used in finding aids also provided a significant barrier to answering these extremely specific questions. looking at these two problems, we knew that we had to both streamline our workflow as well as take advantage of the granular file-level description created by archivematica. after assessing existing solutions, however, we felt that building a custom application was the only way forward. atom, the archival management system also developed and maintained by artefactual, was considered but ultimately decided against given that the cca’s description is currently in tms and it seemed unwise to manage description and files across two archival management systems. other digital repository systems, like dspace and islandora, were too heavy and often displayed information in a way that did not necessarily reflect archival hierarchies in a meaningful way. their interfaces were also often in english only, falling short of quebec legal requirements. we also considered adapting the existing cca website; however the millions of item-level descriptions for digital files would have appeared alongside library books and fonds-level description, creating a lot of unnecessary noise and cluttering search results. figure 2. sample dublin core metadata, written by a processing archivist, as it appears in the archivematica-generated mets file. figure 3. sample premis metadata, created by archivematica to document micro-services, as it appears in the archivematica-generated mets file. with all this in mind, tim walsh designed and built a proof-of-concept for scope in 2017, which has now evolved into a fully-functional browser-based access interface for dips developed by artefactual. scope allows archives staff to upload dips to the interface, which then uses the mets file in the dip to display folderand file-level metadata. researchers receive a login, search this metadata to find individual files or entire dips, and download the dips directly to the locked-down workstations in the cca study room via a link on the interface. researcher access to scope is only available on these workstations, where internet and usb access is blocked. material cannot be moved off of the workstations or accessed elsewhere. (note that dips can only be downloaded as a whole, not as individual files. this is for several reasons, particularly to maintain archival context and to not break files with external references, common with architectural formats.) figure 4. screenshot of the scope home page. figure 5. screenshot of a collection-level page in scope. figure 6. screenshot of a folder-level page in scope. note the “download zip file” button. reference staff and archivists no longer need to mediate the request, meaning the turnaround time is significantly faster. searching is also greatly improved. the file-level metadata in the mets file is now discoverable, and searches can be conducted across all collections at once. researchers have direct access to file-level metadata, including file name, file format, last modified date, and size, which we were not able to display previously given the format of our finding aids. it also allows for display of premis events, which makes archival processing more transparent to our researchers. figure 7. screenshot of the search results for “autocad drawing” in scope. figure 8. screenshot of item-level page in scope, including documentation of the premis events (titled “preservation metadata”). a team effort collaborating with artefactual was an easy, but necessary decision. the first reason for doing so was a practical one. though our archives staff has a reasonably high level of technical skill, building a full-fledged application would require the work of professional developers. secondly, because an important eventual feature of scope is integration with our archivematica storage, artefactual felt like a natural collaborator due to their expertise with their product and their understanding of archival practices and principles. the cca ended up sponsoring roughly 700 hours of artefactual’s developer and analyst time over two phases from summer 2018 to spring 2019. collaboration is embedded in artefactual’s work culture. typically, a development project includes an archivist/librarian (analyst), a software developer, and a project manager. the analyst is the subject specialist who has both who has the training and expertise in the field and can translate the client’s feature request to the software developer through various means, including wireframes and feature files. finally, the process is overseen by a project manager, who takes care of administrative tasks and makes sure that everyone across both organizations is moving towards a successful conclusion. it was also a collaborative effort across the cca. archivists and other collections staff worked to communicate their needs for this application, and with the help of artefactual, translated these into particular features for the developer. members of cca’s digital division also contributed valuable expertise in digital project management and user experience testing. two user experience sessions with cca staff during the first phase were able to inform development moving forward. these sessions made it clear that any final product needed to be not just functional, but also intuitive and easy to use. they also gave us the opportunity to help conceptualize certain functionalities, particularly filtering and faceting search results, with real users. in practical terms, collaboration across the two organizations centered on producing a beta version of an application that would enable researchers to access meaningful content in archivematica dips from within the cca’s study room. the cca provided user stories and wireframes that informed the initial conceptual stages of the project. artefactual then wrote feature files in the gherkin syntax (the translation document between user and developer) to more fully describe the cca’s desired functionality. from there, development proceeded as outlined below. ultimately, the beta version of scope was demonstrated at the cca with feedback from those sessions folded into discussions for a potential second round of development. development followed an iterative, agile process, with calls between the cca and artefactual taking place every other week. these calls provided an opportunity for the cca and artefactual to review progress since the last discussion, address questions, and review/modify upcoming priorities. the code is housed in github in the cca organization as a repository called dip-access-interface. artefactual staff were given access to github. in order to manage the project overall, we used waffle, the same tool that artefactual uses to manage archivematica. from waffle, the entire team could look at issues with their associated pull requests and track  progress as they were created, edited, reviewed, tested and completed. both github and waffle enable labels, and waffle allows filtering by label so that the user can select the label ‘help wanted’ and then view all issues and pull requests with that label. for example, for this project, ‘demo1’, ‘demo2’, and ‘demo3’ labels were created and associated with issues that needed to be complete in order to successfully run the first, second, and third demonstrations that cca ran internally to its user groups. scope’s waffle board was used extensively to organize work in progress, work ready for review, and upcoming tasks throughout all phases of the project. this approach allowed for on-the-fly re-scoping  and re-prioritizing, which resulted in a product that more closely addressed the overall intended outcomes. figure 9. the scope waffle board, ready to go for phase 2 of development. future work we are now looking at a second round of development that will take scope to ‘post-beta’. the first round was meant to deliver a useable product. the second will introduce new features and revise existing features. the most major update will be scope’s integration with archivematica’s dip storage. we are analyzing archivematica’s current dip creation workflow and comparing it to the cca’s use of automation tools (at) to generate dips to determine the best way forward (issue #117). it is possible that we will continue to use at to generate the type of specialized dip that the cca requires.  however, it’s also possible that we integrate the dip upload functionality to scope within archivematica itself. time will tell as we work through this second phase! the other major update will involve improved searching (including faceting and filtering). currently, researchers cannot search by date range or by file format, nor can they refine search lists or use tags (issue #91). this will represent another step forward as we take advantage of granular item-level metadata created by archivematica. additional features will also include reporting for reference and collection statistics, using google analytics and kibana respectively, to be able to better understand how collections are being used, what the digital archive consists of, and how it is growing over time. we will also be addressing the nearly thirty outstanding github tickets related to back-end updates and user experience. conclusion like any product, a software application needs to be maintained in order to stay relevant. there are dependencies within the application and external environmental or organizational factors that need attention. the great power of open source software is that ideally it’s the community which, through various means, contributes to the ongoing maintenance and continued relevance of the application. there is no vendor lock-in, but the trade-off is that those vested in the product must also accept responsibility for its ongoing care. it’s been said that an open source software application is like having a kitten. when someone is given a free kitten they don’t have to pay for the animal but there are vet bills to pay, food to prepare, and love to give to make sure the cute little kitten grows up to be a well-fed and contented cat. scope is that cute little kitten right now, or maybe, since we’re not out of beta, it’s still in utero. in order to make sure it grows into a dip lovin’ cat, scope needs to be housed, to be used, to be documented. those who are interested can download the application and try it out in conjunction with archivematica. when they do, they can contribute back to the project through many avenues: troubleshooting on a user forum, doing code review, writing code, and identifying or commenting on issues. in doing so, they are implicitly agreeing to become part of scope’s community of care which will ensure the continued longevity of the application. if you are interested in learning more about scope please contact the authors. you can also check out our waffle board or our github repository. we look forward to hearing from you! acknowledgments the authors would like to thank tim walsh (digital preservation librarian, concordia university libraries), bun ek (digital projects manager, cca), marc boucher (information technology, cca), martien devletter (associate director, collection, cca), and sophie couture (associate director, digital, cca) for their work on scope. additional thanks to digital processing archivists justine couture, alexandra jokinen, and mireille nappert, as well as digital archives technicians kyle dennis and anne-marie trépanier, for their work processing the archaeology of the digital collections. scope is generously funded by the montreal cultural development grant awarded by the city of montreal and the quebec department of culture and communications. about the authors kelly stewart (kstewart@artefactual.com) is director of archival and digital preservation services for artefactual systems. she holds a master of archival studies degree from the university of british columbia and has many years experience as a consultant and educator in archives and records management, specializing in policy work, business process analysis, facilitation and teaching. she has worked for artefactual since 2017. stefana breitwieser is the digital archivist at the canadian centre for architecture. she can be reached at sbreitwieser@cca.qc.ca with any questions related to scope or born-digital archives. bibliography and notes lavoie, brian. the open archival information system (oais) reference model: introductory guide (2nd edition). digital preservation coalition; 2014 [cited 2019 january 10]. available from: https://www.dpconline.org/docs/technology-watch-reports/1359-dpctw14-02/file schumann, natascha and astrid recker. demystifying oais compliance: benefits and challenges of mapping the oais data model to the gesis data archive [internet]. iassist quarterly; summer 2012 [cited 2019 january 10]. available from: https://iassistdata.org/sites/default/files/iqvol36_2_recker.pdf subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – arduino-enabled patron interaction counting mission editorial committee process and structure code4lib issue 20, 2013-04-17 arduino-enabled patron interaction counting using the arduino development board (http://arduino.cc) has become a very popular way to create hardware prototypes that bridge the divide between the physical world and the internet. this article outlines how to use an arduino, some off-the-shelf electronic parts, the processing programming language, and google documents to create a push-button reference desk transaction tally device. the design: plugged into a computer at the reference desk, staff members push the appropriate button on the device when a reference transaction occurs, and the action is instantly tallied in a google document. having a physical device on the desktop increases chances of proper collection of information since it is constantly visible and easily accessible, versus requiring staff members to click through a series of options in a piece of software running on the pc. the data can be tabulated in google documents or any other source that processes form-based html data. this article covers all of the major components of creating the project: – constructing the arduino circuit and programming it – creating the google docs form – creating the processing program that will listen for information from the arduino and send it to the google docs form by tim ribaric and jonathan younker the arduino microcontroller platform is popular with hobbyists for two main reasons: first, it is easy to use, requiring minimal knowledge of electronics and programming; second, it is very versatile in the way it can sense phenomenon from the physical world and translate it into signals that can be understood by computers. this basic understanding of the functioning of the arduino provides the basis of a patron interaction tabulating device created by the library systems and technologies department at brock university. traditionally, tracking patron interactions at the various service points in a library was conducted with paper-based tally sheets. each time a patron approaches the desk with an inquiry, a tick mark is placed in the appropriate category of the tally sheet. the data gathered can be used to keep track of how busy the desk is at specific points in time, to provide insights into what staffing levels are required, to get a general understanding of what the gaps in the service are by analyzing recurring questions, etc. most often, this data is transcribed into an electronic database or spreadsheet where further analysis can be done. at some institutions – like brock university – the paper tally form has been replaced with a software solution that allows service desk staff to enter this data electronically, using a piece of software or a web form to capture the interaction. the immediate downside to such a configuration is that it is often easy for staff to forget or to miss recording the interactions into the application when the service point becomes busy, the immediacy of paper and pencil is lost, and additional steps are introduced (opening the application, selecting the options, etc.) this project aims to address the shortcomings of each of these collection styles (physical and electronic), by creating a hybrid approach facilitated by the use of an arduino microcontroller. figure 1. tabulation device / controller box a tabulation device (seen above) is placed on the service desk and connected to a computer with a usb cable. when a patron asks a question, the staff member simply taps a button, and the interaction is added directly to a google docs spreadsheet (the data collection endpoint can easily be changed, if necessary). the immediate advantage of using this approach is that there is a lower chance of staff forgetting to tabulate the question, as the physical device would be difficult to overlook, the process is extremely quick and efficient, and the transaction is logged electronically so that it can be analyzed in a meaningful way. three distinct pieces are involved in the synthesis of this machine (known as the tabulatron): the construction and programming of the physical signaling device, a software component running on a computer to receive the signals from the physical device and to translate them to html post calls, and a google docs spreadsheet to collect the information. the signaling device the parts required to construct the device: an arduino microcontroller board (example) four pushbutton switches (example) one amber light emitting diode (led) (example) one usb cable to interface the device to a computer and to provide power(example) one enclosure to hold the constructed device (example) the example links are just suggestions and a wide variety of products could easily be substituted in their place. local electronic stores are a great resource and can easily supply all the required components. the total cost to construct a tabulatron is approximately $50. the arduino microcontroller board comes in a variety of sizes and configurations designed to be used in various different environments (http://arduino.cc/en/main/products). the arduino uno board was used for this project, however, other arduino boards could have been used instead. the only restriction is that the board chosen must have the required number of analog inputs to correspond to the number of desired buttons (smaller arduino boards often have less input/output ports). the following diagram, created in an open­ source package called fritzing (http://fritzing.org), shows a rendering of the completed circuit as a schematic. figure 2. schematic the wiring can be described as follows: the led is connected as an output on digital pin 13 and grounded to the arduino one terminal of each of the 4 buttons is connected as input to analog pin 0, analog 1, etc. the opposite terminals of the push buttons are wired together in series and grounded to the arduino this exact circuit, rendered with physical wires and buttons, takes shape as the following: figure 3. controller box with wiring programming the arduino microcontroller after the physical construction of the device is completed, it needs to be programmed via the arduino integrated development environment (ide). this ide is an open source package and can be downloaded from the arduino site (http://arduino.cc/en/main/software). once programmed, the arduino microcontroller will listen for button pushes and send a signal via the usb port to the connected computer. this connected computer is running a program that will be listening for these signals (outlined in the following section). for example, when the button connected to the analog 0 input is pushed, a ‘0’ character is sent to the computer via the usb port and then the led is told to flick on and off. this provides haptic feedback to the user that the button press has registered. when the button attached to analog 1 input is pushed, a ‘1’ character is sent to the computer via the usb port followed by a blink of the led, and so on. programming the arduino chip with the ide only needs to be done once. programs created with the ide are called sketches, and the chip will retain this uploaded sketch indefinitely, even after power cycling. the key function in this piece of code check_switches() was borrowed from the adafruit site (http://www.adafruit.com/blog/2009/10/20/example-code-for-multi-button-checker-with-debouncing/). adafruit is an online store for electronic components and also a resource hub for learning how to create with electronics. the following code is the complete program that the arduino needs: //tabulatron arduino code //will signal via serial whenever a button is pressed //circuit is pretty simple //-pushbutton on a0 //-pushbutton on a1 //-pushbutton on a2 //-pushbutton on a3 //-led on 13 //many,many thanks to adafruit industries where i blatantly lifted this code // http://www.adafruit.com/blog/2009/10/20/example-code-for-multi-button-checker-with-debouncing/ //buy something from there please //global variables // hash marked lines are not comments in the arduino ide as is common // with other languages, more details: http://arduino.cc/en/reference/define #define debounce 10 byte buttons[]={14,15,16,17}; // the analog 0-5 pins are also known as 14-19 #define numbuttons sizeof(buttons) byte pressed[numbuttons], justpressed[numbuttons], justreleased[numbuttons]; int led = 13; voidsetup(){ byte i; serial.begin(9600); pinmode(13,output); //make input & enable pull-up resistors on switch pins for(i=0;i<numbuttons;i++){ pinmode(buttons[i],input); digitalwrite(buttons[i],high); } start_led(); } //led blink sequence for when device starts up. voidstart_led() { digitalwrite(led,high); delay(1000); digitalwrite(led,low); delay(1000); } //led blink sequence for after a button is pressed //gives nice feeling that the machine is paying attention void flick_led() { digitalwrite(led,high); delay(250); digitalwrite(led,low); delay(250); digitalwrite(led,high); delay(250); digitalwrite(led,low); delay(250); digitalwrite(led,high); delay(250); digitalwrite(led,low); delay(250); } //this function does the heavy lifting of checking for button presses //and changing the values of the byte array to reflect button states void check_switches() { static byte previousstate[numbuttons]; static byte currentstate[numbuttons]; static long lasttime; byte index; if (millis() < lasttime) { //we wrapped around, lets just try again lasttime = millis(); } if ((lasttime+debounce) > millis()){ //not enough time has passed to debounce return; } //ok we have waited debounce milliseconds, lets reset the timer lasttime = millis(); for (index = 0; index < numbuttons;index++){ justpressed[index] = 0; //when we start, we clear out the "just" indicators justreleased[index] = 0; currentstate[index] = digitalread(buttons[index]); //read the button if (currentstate[index] == previousstate[index]) { if ((pressed[index] == low) && (currentstate[index] == low)) { //just pressed justpressed[index]=1; } else if ((pressed[index] == high) && (currentstate[index] == high)) { //just released justreleased[index] = 1; } pressed[index] =! currentstate[index]; //remember, digital high means not pressed } previousstate[index] = currentstate[index]; //keep a running tally of the buttons } } void loop() { check_switches(); for (byte i = 0; i < numbuttons; i++){ if (justpressed[i]) { serial.print(i,dec);//print the button number to the serial port so that the processing app can listen for it. serial.println(); serial.flush(); //always flush when you're done. flick_led();//an immediate couple of blinks of the led makes the user feel like it has done something } } } the arduino website has a clear and comprehensive tutorial on how programming the chip can be done. reading through the tutorial before programming the tabulatron will help clarify this process (http://arduino.cc/en/guide/windows | http://arduino.cc/en/guide/macosx ) setting up the reference workstation: software listening for the button press this piece of software is written using the processing ide (http://processing.org/), and its main purpose is to listen for the signals sent to the usb port from the tabulatron. processing is a java-based product that can be used for a multitude of applications. it is used here because it is very easy to draw graphical user interfaces, and it is complementary to the arduino ide, due to the fact that it has built-in support for reading signals sent through the usb connection (which is the functionality we developed in the previous section). this processing sketch is run on the computer that the arduino is connected to and always needs to be running so that it can listen for signals from the tabulatron. when run, this processing program first renders an image of the physical device to the screen with labels that indicate what each corresponding button push will do: figure 4. processing script display once the processing sketch receives a signal from the device, it completes a html post to a google form. as a further method of feedback, the rendered image on the screen will change the corresponding button that was just pressed to green. the ‘view form’ button will open a browser to the results page of the google form. the following code listing is the processing source code that completes this: //tabulatronapp // //listens for the data sent to the serial port from the arduino //post svalues to google doc // //written by tim ribaric(tim@elibtronic.ca) //@elibtronic import processing.serial.*; serial port; pfont f; string buttonsig; //a google doc form is required. //it should only have 1 question on it //that one question should be a dropdown box with only 4 values on it //example: https://docs.google.com/spreadsheet/viewform?formkey=dhdas1nkdethy2y5yknjadvfngfpske6mq#g id=0 string formkey = "dhdas1nkdethy2y5yknjadvfngfpske6mq"; // the public 'key' for the google document form (found in url of view mode) string revformkey = "0aicmfka7wo4fdhdas1nkdethy2y5yknjadvfngfpske"; // the private key for the google document form (found in the url of edit mode) string btn0 = "reference"; // the text of the first option in the dropdownbox string btn1 = "technical"; // the text of the second option in the dropdownbox string btn2 = "directional"; // the text of the third option in the dropdownbox string btn3 = "referral"; // the text of the fourth option in the dropdownbox char bsig; //clicking on the 'view form' button will open up the 'edit' mode in a browser //a login to google doc will be needed unless set to public void mousereleased(){ if(mousex > 75 && mousex < 300&&mousey > 325 &&mousey < 350) link("https://docs.google.com/spreadsheet/ccc?key="+revformkey+"#gid=0"); } //branding void viewform(){ rect(75,55,225,25); rect(75,325,225,25); ellipse(350,90,25,25); fill(0); text("viewform",150,345); text("tabulatron",150,75); fill(255); } //draw the 4 boxes and labels them according to the form values outlined above //once a button is pressed it will be drawn in green, as a form of feedback voidboxmaker(intc,intpos){ //1 draws a green box, everything else is white switch(c){ case1: fill(119,232,105); //greenish break; default: fill(255); } //four button, 4 rectangles switch(pos){ case0: rect(75,110,100,75); fill(0); text(btn0,90,155); break; case1: rect(200,110,100,75); fill(0); text(btn1,215,155); break; case2: rect(75,225,100,75); fill(0); text(btn2,83,270); break; case3: rect(200,225,100,75); fill(0); text(btn3,220,270); break; } fill(255); } //draws blank boxes void clearboxes(){ boxmaker(0,0); boxmaker(0,1); boxmaker(0,2); boxmaker(0,3); } //does the actual sending to google //will produce loads of errors in the console window //couldn't really fix these. (exceptions couldn't be caught) void tally(intbtn){ switch(btn){ case0: loadxml("https://docs.google.com/spreadsheet/formresponse?formkey="+formkey+"&amp;ifq&ent ry.1.single="+btn0); break; case1: loadxml("https://docs.google.com/spreadsheet/formresponse?formkey="+formkey+"&amp;ifq&ent ry.1.single="+btn1); break; case2: loadxml("https://docs.google.com/spreadsheet/formresponse?formkey="+formkey+"&amp;ifq&ent ry.1.single="+btn2); break; case3: loadxml("https://docs.google.com/spreadsheet/formresponse?formkey="+formkey+"&amp;ifq&ent ry.1.single="+btn3); break; default: } } voidsetup(){ f = loadfont("aharoni-bold-16.vlw"); textfont(f); size(400,400); background(128); boxmaker(0,0); boxmaker(0,1); boxmaker(0,2); boxmaker(0,3); viewform(); //at first run you'll have to figure out what comm port the arduino is on //change the array index to corresponding value println(serial.list()); port = newserial(this,serial.list()[2],9600); } //listens to serial port defined above and reads until linefeed //grabs the character and colors the box, sends the tally //works fairly well void draw(){ while(port.available() > 0){ buttonsig = port.readstringuntil(char(10)); try{ bsig = buttonsig.charat(0); } catch(exception e){ //sometimes reading from serial will produce a null value //break out and try again break; } switch(bsig){ case'0': clearboxes(); tally(0); boxmaker(1,0); break; case'1': clearboxes(); tally(1); boxmaker(1,1); break; case'2': clearboxes(); tally(2); boxmaker(1,2); break; case'3': clearboxes(); tally(3); boxmaker(1,3); default: } } } of interesting note is the function called tally() that begins on line 97. this is where the html post happens. the calls to loadxml() within tally() can be easily modified to work with a pre­existing library solution instead of google docs. an additional benefit of using processing is that it can create standalone executables for all of the major platforms (mac os, windows, linux). the processing program can then be placed in the ‘startup’ directory of a windows machine, or the appropriate location on other platforms, so that when the machine boots up the program automatically runs and sits waiting for signals from the tabulatron, requiring no intervention from the staff member at all. the google documents spreadsheet form the final component is the google documents spreadsheet form. the construction of this form needs to completed as follows: one multiple choice question with four answers, each one corresponding to the labels used in the preceding processing program. figure 5. google form the processing program emulates exactly what happens when a typical end user is filling data into to a google form via the web. a few values need to be set in the processing source code to complete this action. the formkey variable (line 18) is what is found in the ‘formkey‘ variable in the url of the publicly ­facing form. figure 6. google form key the revformkey variable (line 19) is the ‘key’ variable is found in the url of the the form when looking at it in ‘results’ mode. figure 7. google form key the form should also have ‘share’ options enabled so that it can be viewed by any when the ‘view form’ button is clicked. this is done by clicking on the ‘share’ button and enabling ‘anyone who has the link can view’. figure 8. google form share settings putting the pieces together once the signaling device is constructed and attached via usb to a computer, the processing program is run. each time a button is pressed on the tabulatron, a signal is sent via the usb port. the processing sketch listens for this signal and completes an html post of the appropriate value to the google form. the form will automatically add a timestamp to every submission, adding an extra dimension to the data. the process is interactive and instantaneous. a video demonstrating the tabulatron in action can be found here: http://youtu.be/f7hq6lvvdhi figure 9. device in use any library staff member can be looking at the web results and can watch the patron interactions tabulated in real­time. this information can then be used to signal backup staff to come out to the desk when large volumes of interactions are seen. if a library has existing patron question counting software, this code can be easily modified to post to that instead of a google form. figure 10. google form spreadsheet view as seen earlier, if the tally function is modified it can be used to create any web call, (e.g. loading a php file that increments a mysql database). while the tabulatron was only constructed with four buttons, it could easily be modified to have more or less, depending on the needs of the institution. the code required to construct this project is all available via github and can be found here: https://github.com/elibtronic/tabulatron conclusion the tabulatron was initially conceived as a proof ­of ­concept device, yet after demonstrations and an in­-house presentation, staff were immediately enthusiastic about using this in production at brock university library service points. while technically still a prototype, the device is fully­ functional; additional units can be easily constructed, using the exact same arduino code to program the microcontroller. we hope to deploy this prototype as a pilot project in the near future. using tools like fritzing, it would be possible to design and construct a custom printed circuit board (pcb) to reduce the size and eliminate unused features/functions of the arduino board, but at such low volumes and low setup costs to equip a library’s service points with these devices (not to mention an already small footprint), this really seems unnecessary at this point. the great advantage of the tabulatron is its customizability, simplicity, and ease of use. with a little imagination and a few additional lines of code, the tabulatron could be modified to serve a number of roles in the library. all at the push of a button. about the authors tim ribaric (tribaric@brocku.ca) is the digital services librarian at brock university located in st. catharines, ontario. he blogs at http://elibtronic.ca and is the subject specialist for computer science and philosophy as well as the administrator for the brock university digital repository. he has been working at brock since 2006. jonathan younker (jyounker@brocku.ca) is the head, library systems and technologies at brock university in st. catharines, ontario. prior to working at brock, he worked in public and academic libraries in illinois. jonathan is particularly interested in finding and developing creative solutions to ongoing library technology problems. subscribe to comments: for this article | for all articles one response to "arduino-enabled patron interaction counting" please leave a response below: donald moses, 2013-05-11 thanks for the article … i made this and have lots of ideas for other uses. a nice project and a great introduction to arduinos/processing and how we might use them. donald leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – tools and workflows for collaborating on static website projects mission editorial committee process and structure code4lib issue 38, 2017-10-18 tools and workflows for collaborating on static website projects static website generators have seen a significant increase in popularity in recent years, offering many advantages over their dynamic counterparts. while these generators were typically used for blogs, they have grown in usage for other web-based projects, including documentation, conference websites, and image collections. however, because of their technical complexity, these tools can be inaccessible to content creators depending on their level of technical skill and comfort with web development technologies. drawing from experience with a collaborative static website project, this article will provide an overview of static website generators, review different tools available for managing content, and explore workflows and best practices for collaborating with teams on static website projects. by kaitlin newson introduction in 2017, the ontario council of university libraries’ geo community wrapped up a 2-year digitization project of over 1,000 historical topographic maps of the ontario region. as part of this project, the team wanted to create an online presence for these maps to highlight this collection, promote the project, and make the maps more accessible to a wide audience of users. after evaluating different web technology options, the team decided to use a static website generator to create this online project space. static site generators static website generators are tools used to generate a website made up only of html, css, and javascript.  static websites, unlike dynamic sites, do not use databases or server-side scripting languages, and the website appears to the user as it exists on the web server.  a static site generator is typically made up of: a templating language for website layout and theming. a markup language for content creation (e.g. markdown). a local development server to preview and test the site before building. a compile process that builds the final site files into html, css, and javascript (rinaldi 2015). to those that have been working in web development for a while, the re-emergence of static websites may seem like a step backwards. in the beginning of the web, websites were all static, made up only of html, and eventually adding css and javascript. as web technologies developed, the limitations of static websites became apparent, and databases and server-side scripting languages began to fill the gaps in functionality left by the static web. with the emergence of large-scale content management systems (cms) like wordpress and drupal, dynamic websites became more popular as plugins and editing systems led to tools that could be used by content creators without the need for a depth of technical expertise. so, why has the shift back to static websites emerged? with the creation of the modern static site generator, static sites no longer need to be hand-coded from scratch. getting a static website live can be a straightforward process for those with the technical skill to create them. with services like github pages, there’s no need to worry about managing a server or a domain, although a custom domain can still be used – in fact, that is how code4lib’s own conference website is developed, using the jekyll static site generator. for many websites, the process of installing a cms, managing a database, and keeping the site up to date to protect against security vulnerabilities can be cumbersome and overly complex. static website generators offer a more straightforward approach, with pre-existing themes that can easily be used to create a polished and modern website with minimal need for maintenance. there are also many to choose from, with almost 200 options written in a variety of programming languages currently available according to staticgen. features while static site generators are not suitable for every website, they have a number of benefits that should be considered when evaluating systems to use for a new web project. static websites can be up to six times faster than their dynamic counterparts, primarily due to the challenges of caching dynamic content which are not a concern in a static environment (christensen 2015). these websites are also easier to secure since there is no database to be exploited or outdated plugins to create vulnerabilities. from a preservation perspective, static websites are much easier to preserve. since dynamic websites have content pulled from a database and can generate different views based on the user’s context, they can present more challenges for archiving, especially without direct access to the website’s back-end. in the case of a static website, preservation is much easier because the site is displayed as it exists on the web server (rumianek 2013). hosting is also much simpler with static websites. services like github pages have made hosting much more straightforward, with no need to manage a server or even have a registered domain. if you are using your own server, a static site means that you don’t need to worry about managing a database or working with server-side programming languages. however, static websites won’t meet the requirements of every web project. having no database or server-side scripting means that some commonly used website functionality like user accounts or input are more challenging, if not impossible, to implement. many static website generators have plugin options, but many of these plugins require integration with third-party services that may not be ideal depending on a project’s privacy requirements. for example, if you want to have a contact form on a static website, using a service like formspree is required, meaning that all of your data is passed through this service and the functionality of this feature depends on the reliability of this service. another challenge with static websites is their lack of a straightforward administrative environment like those found in cmss such as wordpress or drupal. a lack of editing environment can be a major barrier for content creators, who may not be comfortable writing content in markdown, using command line tools, or managing a git repository. however, solutions are gradually being developed to tackle this problem. content management tools one of the major disadvantages to static website generators is their lack of a content authoring environment, which can be a significant challenge for content creators that are not as familiar with web technologies. when working as a team on a project, this limitation may result in a higher workload for team members with web development expertise, and can ultimately slow down the project’s progress. since content creation for websites is frequently a collaborative process, there is a clear need to have user-friendly systems that allow users to edit content regardless of their level of technical skill or knowledge of web development. since static website content is written in markdown, many markdown editors, such as stackedit, can be excellent options for creating website content. unfortunately, many of these editors do not have the same level of integration with github as other tools or are missing important features that content creators may need, such as image import or interface elements for formatting text. markdown editors can be a valuable tool for content creators, but most do not go far enough to create a fully integrated environment that is accessible to users of different technical skill levels. as static website generators have grown in popularity, developers have begun to address the gap in editing environments by creating more robust cmss that can be used with static websites, with 27 options currently listed at headlesscms.org. while many of these services are paid, some are fully open-source and can be configured by a website administrator to be used by other project collaborators. netlifycms is emerging as a strong contender for a static site cms to bridge the gap between raw markdown editing and the wordpress-like editing environment that many are currently used to (fig. 1). this tool allows users to log into the administrative interface with their github credentials, manage different types of content, and write using a minimal and user-friendly interface. figure 1. the netlify cms, built from the demo available at https://www.netlifycms.org/docs/test-drive/ if a static website project is using github, one option is to use the built-in editor for content editing on their website (fig. 2); however, this does not give users straightforward options for text formatting, and a working knowledge of the markup language being used (e.g. markdown) is necessary. this approach is also missing functionality that content authors may need, like the ability to import images. while markdown is not a challenging syntax to learn, it’s important to make the content creation as simple as possible for users with a range of technical skill levels. by using more robust content management systems, users have more editing options available to them without requiring technical knowledge of web development. figure 2. the github editing environment prose (fig. 3) works with an existing public github repository to allow users to author content in a clean and simple environment. once editing is complete, content can be committed directly to the github repository, meaning that no working knowledge of git is required. prose has many standard text editing features, like font bolding and inserting headings, as well as the option to import images that will be inserted directly into the repository. prose also has minimal setup requirements, and only needs github authorization to access the project’s repository to get started with content management. figure 3. the prose editing environment it should be noted that the tools above rely on a project using a github repository. internal repository management tools like gitlab or bitbucket have a built-in web editor similar to that of github but do not integrate with the tools mentioned above without some code configuration. desktop editors for markdown writing may meet content editing needs in these cases but were not explored in depth for this article. the topographic maps project in 2017, members of the ontario historical topographic maps digitization project team developed their website using a static website generator. the historical topographic maps digitization project was a 2.5-year digitization initiative completed by the geography community in the ontario council of university libraries (ocul). the project involved the digitization and georeferencing of over 1,000 topographic maps across ontario. one of the final steps of the project was promotion, which involved the development of a website to showcase the project and the digitized maps. since the full resolution maps and metadata were being stored and displayed through scholars geoportal, this website was intended to promote the project and provide a user-friendly way to browse the collection, rather than to provide access to geospatial mapping tools, preservation capacity, or complete metadata records. figure 4. the historical topographic maps digitization project website, available at ocul.on.ca/topomaps the project team was made up of staff from across ocul universities and scholars portal. as the website lead, i had prior experience working with web technologies, including html, css, javascript, git, and content management systems. this prior experience gave me the necessary knowledge to handle the technical aspects of the project, while other team members focused on content. initially, we were unsure of what tool we wanted to use for the project’s website. we began by considering content management systems like drupal and omeka, but these seemed more complex than necessary for a website primarily made up of images and text. scholars portal had previously used a static site generator for another project, so i began investigating different functionality and theming options to determine if this was suitable for the new website. after creating a demonstration site and sharing it with the rest of the project team, we decided to develop the site using a static generator. hugo, a popular static generator built in the go language, was used to create the project website. hugo was selected because of the simple installation process, fast build speeds, and previous experience using the tool. the website is hosted on the ontario library research cloud (olrc), a cloud storage service built with openstack swift that has the ability to serve static content. this project serves the dual purpose of showcasing the collection and demonstrating a potential use case for institutions using the olrc. the theming for the website was based on the creative portfolio theme for hugo, which was originally created for designers to showcase their work, but works well to highlight any collection of images. the template was modified to add multilingual support for english and french, along with some css modifications to improve the colour contrast and font sizes for accessibility purposes. javascript and jquery were used for additional functionality, including magnifying the maps when a user hovers over them with their mouse (fig. 5), and filtering the list of maps based on text input. an inventory of the full collection of maps had been previously created as a google spreadsheet, so this data was converted to json which was then used to generate the listing of the collection. a google app script was used to export the spreadsheet data into usable json (fox 2013). juxtaposejs was used to create a comparison slider to demonstrate changes between map areas over time (fig. 6). the code for the website can be found in the project’s github repository. figure 5. the zoom feature of the topographic maps website (http://ocul.on.ca/topomaps/maps/map04/) figure 6. the map slider developed with juxtaposejs (http://ocul.on.ca/topomaps/highlights/) the initial draft for website content was created collaboratively in google docs, with different team members contributing individual sections. through team meetings, decisions were made about different aspects of the website, such as top-level navigation items, content structure, and the display of the maps. once the team agreed that the content was complete, it was sent for french translation. members of the project team then used prose to work with content in the github repository. as content was developed, some bugs were encountered when editing with prose which prevented changes from being committed to the repository, and the built-in github editor was used for content modifications in these cases. when changes were committed to the repository, the website lead would then pull down the changes, recompile the site, and update the development version of the site. once all of the content was in place, the team did a final review, referring any styling suggestions to the website lead, and fixing any content issues that were found. final configuration elements were added, including google analytics tracking, and the site was launched and promoted to the academic community. overall, the historical topographic maps project website was a success for the ocul geo community. the website serves as an attractive and intuitive way for users to engage with the maps that may not have been exposed to them otherwise. while more advanced geospatial data users may explore the maps in their preferred gis tool or through scholars geoportal, the website provides a space for a broader range of users to engage with the map collection. conclusions & lessons learned for the team at scholars portal, this project helped us to better understand what kinds of projects static generators are best suited for, and how they can be used collaboratively. although we do have sites built with drupal, these large content management systems are far too resource-intensive and complex for small and simple websites. static website generators still require technical knowledge to get them up and running but require far less long-term maintenance than something built in drupal or wordpress. once you understand how to use them, these tools can be incredibly efficient for getting a simple and attractive website up quickly, without needing to manage a database or write server-side code; however, an understanding of markup languages and command line tools is necessary. if you want to do customizations for your site, you may also need to understand css and javascript, and also be willing to learn the templating language that is used by the static generator. depending on your website hosting environment, you may also need knowledge of web server management. this technical knowledge is currently necessary to get a static site up and running, but hopefully future technologies will reduce this barrier to entry. static websites have seen a strong resurgence with the modern static generator, offering a number of benefits over dynamic websites in performance and security. while the editing environment isn’t yet on par with those found in content management systems like wordpress, tools are gradually emerging to fill this gap in the development process. static generators are not suitable for every kind of website, such as those that require dynamic elements like user input, but should be strongly considered as one possible option when evaluating systems to use for a new project. notes if you’re interested in making a site with jekyll and github pages, programming historian has an excellent tutorial to get you started. references christensen, mb. [updated 2015 nov 02]. why static site generators are the next big thing [internet]; [cited 2017 august 16]. available from:  https://www.smashingmagazine.com/2015/11/modern-static-website-generators-next-big-thing/ fox, p. exporting a google spreadsheet as json [internet]. [updated 2013 june 07]. [cited 2017 august 16]. available from: http://blog.pamelafox.org/2013/06/exporting-google-spreadsheet-as-json.html rinaldi, b. static site generators: modern tools for static website development. sebastopol (ca): o’reilly media inc.;  2015 [cited 2017 august 16]. available from: http://www.oreilly.com/web-platform/free/static-site-generators.csp rumianek, m. 2013. archiving and recovering database-driven websites. d-lib magazine [internet]. [cited 2017 august 16]. available from: http://www.dlib.org/dlib/january13/rumianek/01rumianek.html about the author kaitlin newson (kaitlin@scholarsportal.info) is the digital projects librarian at scholars portal, the technology service arm of the ontario council of university libraries. you can find her online at www.kaitlinnewson.com and on twitter as @kaitlinnewson. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – beyond the hype cycle: experiments with chatgpt’s advanced data analysis at the palo alto city library mission editorial committee process and structure code4lib issue 58, 2023-12-04 beyond the hype cycle: experiments with chatgpt’s advanced data analysis at the palo alto city library in june and july of 2023 the palo alto city library’s digital services team embarked on an exploratory journey applying large language models (llms) to library projects. this article, complete with chat transcripts and code samples, highlights the challenges, successes, and unexpected outcomes encountered while integrating chatgpt pro into our day-to-day work. our experiments utilized chatgpts advanced data analysis feature (formerly code interpreter). the first goal tested the search engine optimization (seo) potential of chatgpt plugins. the second goal of this experiment aimed to enhance our web user experience by revising our bibliocommons taxonomy to better match customer interests and make the upcoming personalized promotions feature more relevant. chatgpt helped us perform what would otherwise be a time-consuming analysis of customer catalog usage to determine a list of taxonomy terms better aligned with that usage. in the end, both experiments proved the utility of llms in the workplace and the potential for enhancing our librarian’s skills and efficiency. the thrill of this experiment was in chatgpt’s unprecedented efficiency, adaptability, and capacity. we found it can solve a wide range of library problems and speed up project deliverables. the shortcomings of llms, however, were equally palpable. each day of the experiment we grappled with the nuances of prompt engineering, contextual understanding, and occasional miscommunications with our new ai assistant. in short, a new class of skills for information professionals came into focus. by m ryan hess and chris markman i. introduction chatgpt and other large language models (llms) have sparked much curiosity recently among libraries and information professionals. as a forward-thinking library that likes to experiment with emerging tech, we were eager to explore whether these new ai systems could enhance our work. while other libraries in 2023 tested applications for gpt-3, we wanted to push further by utilizing the more advanced gpt-4 model within chatgpt. with patron privacy top of mind, we conducted hands-on experiments focused on seo optimizations and developing an improved website taxonomy. our tests uncovered a mix of successes and failures, revealing fundamental limits around accuracy, bias, and memory. though imperfect, we believe that with careful oversight, llms can augment certain workflows – acting as research assistants, content creators, data analysts and more. in this article, we detail our search engine optimization (seo) and website taxonomy experiments to advance the conversation around ai in libraries. we share tangible lessons learned, outlining key skills information professionals will need to selectively incorporate ai into their work. while more exploration is needed, our research highlights the tremendous potential of llms to enhance the noble mission of organizing knowledge. despite several pitfalls, our experiments represent early and exciting steps toward integrating ai thoughtfully into libraries. with the right human guidance, llms promise to boost productivity, creativity, and problem-solving. the path ahead may be winding, but the possibilities make it well worth exploring. ii. methodology seo experiment #1 – chatgpt seo plugins this first experiment with seo was based on an earlier success in 2023 using the free version of chatgpt. at palo alto city library (pacl) we use biblioweb as a content management system (cms) and make every effort to fully leverage its seo tools. as a result, this often means updating or reviewing html meta descriptions for “freshness” over time. one workflow issue with this seo process is that it not only takes time to do, but it is also not the most exciting task: imagine trying to summarize page content and thinking about keyword alternatives on a library website, while also trying to keep in mind various user groups or audiences at the same time. juggling all of this can be a mentally taxing exercise. chatgpt happens to be a champion at this one specific task—just copy/paste the content of any webpage into the chat and in less than 5 seconds it can crunch large blocks of text into short snippets or lists of alternative keywords. at the same time, it can take that same webpage “gist” and generate hundreds if not thousands of variations depending on the context you provide. it’s a prime example of the “cognitive offload” that llms can provide the plugin store based on this early success, in june of 2023, it seemed like there had to be something new that our newly acquired chatgpt pro account and suite of plugins could help us with. figure 1. chatgpt about plugins what we instead found was a slew of privacy and security concerns from plugin makers with zero accountability, and a cheeky “use at your own risk” message from openai. thankfully, some of these concerns were addressed in later updates to the chatgpt plugin store in late 2023 with the inclusion of developer information links and contact details. there are sometimes, but not always, privacy policies or terms of service pages. at first glance this does seem like an improvement, but at the time of writing this article, many of these links are essentially placeholders or contain very little information. some of these seo plugins could assist with ranking your site versus others in the same keyword space and search results list. however, they were asking for api keys and full access to google analytics. this leads to very interesting philosophical questions like “is it still considered a man-in-the-middle attack if you first ask users to opt-in?”. needless to say, the cost-benefit of this setup was not adding up. after browsing several options, it looked like in the few cases where these seo tools might be useful, google search console already had us covered. figure 2. the chatgpt pro plugin store seo options with developer info links in the fine print. why reinvent the wheel? these plugins could be useful as part of some other automation routine, perhaps in combination with other chatgpt plugins or features, but for data analysis and web content strategy, chatgpt pro already felt like overkill. around this same time, seo and marketing experts started to warn about the dangers of llm generated seo spam, which eventually lead to notable events like quora.com’s misinformation feedback loop bleeding into google search results [1] and more and more of this content creeping into top search results [2]. the moral of the story here is that while chatgpt can help with some aspects of seo, the overall landscape is full of landmines. proceed with caution! seo experiment #2 – working with sitemap.xml while the first experiment with plugins (a feature limited to chatgpt pro subscribers) suggested the platform was entering gartner hype cycle’s “trough of disillusionment”, our second experiment using the newly added advanced data analysis option did have some positive results. advanced data analysis mode is an excellent feature because it can write custom python code tailor made for your specific use case. like meta tags and meta descriptions in html, the core elements of the previous experiment, this second experiment was set up to test another basic element of seo: your sitemap. you might be wondering at this point, why use python at all? it’s important to note that python is the programming language of choice for advanced data analysis mode, so this was part of the experiment itself. while this mode can assist with other coding languages, under the hood it is essentially a plugin developed by openai that “graduated” from the plugin store and became embedded into the product itself. for example, if you upload a csv file containing a data table in advanced data analysis mode, it will use a python code library to generate a graph from that data (that is, if you ask it to). we should also note here that most modern cms will automatically create and update sitemap.xml files(s) to encourage a good data harvest from roaming web spiders. it’s now a universal process and a very behind the scenes element of most websites, but this wasn’t always the case (and in fact wasn’t a built-in wordpress feature until wordpress 5.5 released in 2020 [3]). in fact, for many years generating this xml document was one of the quickest ways to improve your site’s seo. knowing this, surely, there must be something cool that chatgpt pro could do with this vital data. for example, being able to quickly create a visual representation of your websites’ overall architecture for discussions with stakeholders or for the purpose of onboarding is extremely useful. analyzing a sitemap might also help you identify pages that are, for whatever reason, not being indexed. this was the initial idea that started experiment #2. xml surprises as it turns out, our sitemap, which spans over 100 pages of content, is in fact an index file that points to a collection of smaller, chronologically sorted xml files. the real goal was to work with the data inside those smaller xml files, but before chatgpt could do anything it needed to recombine this data set into a single file. this seemed easy enough for a computer to do—trivial in fact! but this is where the trouble really began. our process went as follows: load the data. test out what the new subscription-only advanced data analysis feature could do. maybe generate some interesting visualizations or see what key insights chatgpt would be able to glean from this odd assortment of timestamps, page titles, and urls that compose our website topography. easy, right? we had seen from demos that advanced data analysis could create python code on the fly. it sounded magical. what resulted from this experiment was the realization that chatgpt pro and its shiny new gpt-4 model, despite its potential for productivity gains, also has the potential to lead one astray. what should have been an easy task was not easy for a newbie programmer who had overestimated their python coding skills. goose games the second problem this experiment posed, aside from this initial dunning-kruger effect, was that chatgpt never suggested trying a different approach! compare this with an in-person reference interview, or even what a google search might provide: chatgpt has no knowledge of the person asking the question unless you tell it, its main goal is to answer the question directly and concisely, 100% of the time. without extra prompting, llms have no concept of paring down information based on expertise, the same way you might explain how to install an ebook app differently to a patron with more or less tech-savvy. essentially, chatgpt is far too willing to return what it considers the “top search result” (to go back to the google search comparison) without first asking if that initial question or search string is really going to solve your problem. this meant that each conversation about this coding problem had turned into a “wild goose chase” where subsequent questions or error messages not only overestimated the knowledge of a novice coder, but was also unable to detect how far these conversations had strayed from the original problem. the correct approach ended up being a python script that did everything for us in one go. it looked like this: import xml.etree.elementtree as et import requests # function to get xml tree from a url def get_tree_from_url(url): response = requests.get(url) response.raise_for_status() return et.elementtree(et.fromstring(response.content)) # parse the sitemap index file sitemap_index_url = 'https://library.cityofpaloalto.org/sitemap.xml' index_tree = get_tree_from_url(sitemap_index_url) index_root = index_tree.getroot() # create a root element for the combined sitemap combined_root = et.element('urlset', xmlns="http://www.sitemaps.org/schemas/sitemap/0.9") # iterate through the sitemap urls for sitemap in index_root.findall('{http://www.sitemaps.org/schemas/sitemap/0.9}sitemap'): loc = sitemap.find('{http://www.sitemaps.org/schemas/sitemap/0.9}loc').text tree = get_tree_from_url(loc) # download and parse individual sitemap root = tree.getroot() for url in root.findall('{http://www.sitemaps.org/schemas/sitemap/0.9}url'): combined_root.append(url) # write the combined sitemap to a new file combined_tree = et.elementtree(combined_root) combined_tree.write('combined_sitemap.xml') by the way, chatgpt wrote and provided comments for us. it was great. the initial misstep was asking how to do each individual step in isolation using methods that were already familiar rather than simply explaining the end goal to chatgpt first and letting it recommend the best solution. as it turns out, elementtree is a python library that exactly solves the problem of parsing and navigating xml documents. after several attempts to wget and recombine these xml files using brute force, it seemed like there had to be a better way. this was not a unique problem and someone, somewhere in the history of the internet must have found an elegant solution. indeed, they had, but getting to that point required taking a step back and first asking chatgpt to lay out some options. you can read the full chat transcripts in chatgpt directly [4], but here’s a quick overview of the conversation that struck gold. below each question are some notes about why they were asked in that specific order: what are some options to solve this data problem with code? three options came back: create a python script, use a specialized command line tool, or try an xml editor. before this i had already tried some command line options to no avail, and i remember enough about the quirks of xml editing software from library school homework assignments to quickly avoid that option. here are some details about my local code environment, how would i do this option? i picked the python option with some hesitation because i knew there would be some initial setup steps for macos, but i figured chatgpt could work through any error messages. sounds good, now explain how to do this step by step…please? this is where chatgpt starts to really shine. being able to ask follow-up questions based on its step-by-step instructions is much faster than sifting through help docs or forum posts. results of this experiment were mostly indirect benefits. parsing our sitemap did highlight pages in our website that had not been updated in a very long time, but much like the seo plugins, we already had a cms that could display this same information with a few clicks. the main difference was having a 3rd party point this out automatically through a chat interface. admittedly, this same insight could have been found through sorting by the last modified date in our cms, but the key difference is that chatgpt reported this information automatically through open-ended questions. more importantly, by the end of this seo experiment, we had gained the ability to take this sitemaps data further with additional python code tutorials and preexisting software libraries. chatgpt’s human counterpart took on a new workflow they would have never attempted otherwise, learning how to create, edit, debug, and run python scripts. biblioweb taxonomy experiment our second project involved generating new website taxonomies for topics and genres to facilitate better relevancy in the forthcoming personalized promotions feature in bibliocommons. personalized promotions is a feature of bibliocommons where marketing web content known as ‘cards’ is contextually presented to users in their catalog search results [5]. for example, if a user searches for ‘great films’ a card promoting staff recommendations would show up alongside the normal catalog search results of dvds and streaming movies. figure 3. a staff list triggered by a catalog search as it appears inside the catalog search results in bibliocommons. our existing taxonomies were rather simple. with the release of personalized promotions, however, we realized more robust taxonomies would enhance the relevance of cards that were displayed to our customers. enter chatgpt we saw potential in chatgpt plus to help analyze catalog usage data in order to develop a taxonomy that was related to actual customer interests. with a plus subscription, we would have access to gpt 4’s advanced data analysis mode, which provides a way to import data directly into chatgpt for analysis. essentially, this tool is like having a data analyst on staff who can help us understand patterns in the data, clean up data and even visualize the data. data anonymity it is important to point out that we anonymized all data provided to chatgpt. this is important because any data shared with chatgpt is also transmitted to openai for the purposes of improving their products.[6] obviously exposing personal customer data was not something any library would want, so we carefully reviewed each spreadsheet intended for use with chatgpt to ensure no personal information was included. search queries first attempt our first strategy did not go well. we first tried using search queries customers had entered into our catalog. to obtain these, we used google analytics, which captures such queries as part of its tracking of urls accessed by library catalog users. in this round, we provided chatgpt with a csv file with the following headers: searchterm | page views many of these queries were composed of known titles, author names and many stop words. also, we found that some of the queries included search parameter strings such as ‘nw:[0 to 180]’ or ‘formatcode:(ebook ) lx:[0 to 400]’. at first, our work with chatgpt went well as we attempted to clean up the data, normalizing empty values in our spreadsheet and having chatgpt remove format codes and other parameters. in fact, chatgpt was excellent at helping us clean the data. we then asked it to begin analyzing the remediated data and identify some categories. as the following screenshot demonstrates, chatgpt appeared to produce a useful taxonomy. figure 4. screenshot from chatgpt that demonstrates chatgpt appeared to produce a useful taxonomy. however as we looked closer at the distribution of searches under each of the taxonomy terms, we discovered that the vast majority fell under “other themes and genres”. we attempted to force chatgpt to break the other category into smaller chunks but this failed and we were left with a rather unhelpful, uncategorized data category representing 93.4% of all user searches. after many attempts at fixing the ‘other’ problem, we realized we had hit a dead end. table 1. distribution of searches under each of the taxonomy terms. category associated search terms percent of total other themes and genres 2747 93.4 literature & fiction 81 2.75 literature & fiction – book series/authors 25 0.8 education & children’s content 49 1.7 travel & geography 27 0.9 business & economics 6 0.2 mixed categories no data no data regrouping we spent some time re-strategizing our approach with chatgpt. upon reviewing the data we were submitting to chatgpt, we could see that there were many author names and known title searches. this seemed at least a partial explanation. since chatgpt is not connected to the internet, there is no way to have it cross-reference a data set like books in amazon or the library of congress in order to identify suitable subjects for a given author or title. in fact, chatgpt seemed to understand this. figure 5. screenshot from chatgpt. as you can see from the transcript above, chatgpt was pretty good at understanding its limitations and suggesting a solution. we considered its recommendations and then decided that an entirely different approach might work out better. rather than using search queries, we would just upload circulation data associated with each title and its associated subject headings. circulation data our circulation data was drawn from book titles only and was structured as follows: title | circulation | subject 1 | subject 2 | etc... each title had multiple columns for subjects, with a few items having over 20 subjects describing them. we reduced the size of the dataset substantially. in our previous work with chatgpt, we had begun to suspect that our massive dataset was maxing out chatgpt’s memory and this was impacting its ability to deliver quality analysis. with that in mind, we reduced our original circulation data of 40,000 titles to only those titles with 100 or more circulations each. this brought the total to 1,992 titles. since an llm like chatgpt is really good at summarizing content, we thought providing the explicit subject headings might make it easier for chatgpt to define a taxonomy that was both robust and relevant to our customer’s interests. it appears that we were right. ahead of having chatgpt generate taxonomy terms, we wanted to make sure it understood the structure of library of congress subject headings, such that subjects broken up with hyphens were shorthand for denoting hierarchy. we also explained that the circulation data was important and that it should weight decisions on our taxonomy based on the popularity of the book, that is its circulation numbers. in fact, it recommended separating the subjects into distinct terms where hyphens indicated hierarchy. figure 6. screenshot from chatgpt showing it recommending separating subjects into distinct terms where hyphens indicated hierarchy. we were ready for takeoff, but we paused to remind chatgpt that we were aiming for a concise list of 40 or so taxonomy terms that should represent a ‘flat’ non-hierarchical taxonomy derived from the subject terms. it confirmed it understood and then proceeded to generate the 40 terms! the 40 terms were very much the kinds of things you would expect. terms like fiction, juvenile fiction, folklore, biography, california, dogs, etc. however there were some unexpected terms including bears, technique and behavior, etc. given this, we determined that we had probably achieved the best outcome possible, and that human intervention would be required to carry this over the finish line. therefore, we decided to expand the list to 100 items to give our human staff a good set of terms to refine. interestingly, chatgpt had one final curveball for us, which was that it provided a list of 84 terms, not 100 as we had asked for. but who’s counting? the humans did meet a few weeks later to finish where chatgpt left off and we were able to generate our genre and topics taxonomies. one particularly helpful element to this manual labor was the vector map of topics we had asked chatgpt to generate. it turns out that when confronted with strange terms, the vector map helped us understand where this term was coming from. for example, the term mice was not really associated with other animals, but was related to christmas stories. iii. discussion based on our experience we are exploring the long term feasibility and sustainability of developing a chatgpt plugin that would make interfacing with ils data much easier. as we saw in the taxonomy experiment, chatgpt has some difficulty identifying or validating simple things like book titles or author names. the ability to combine this with other plugins could also open up many new workflows and possibilities. whether we do this through an api, or z39.950 interface, or good ‘ol web scraping, we don’t know yet, but the scope of the idea and possibility of allowing other libraries to also share these benefits seems like a win-win situation. a universal ils interface for chatgpt sounds great at first, but you might already be thinking about the privacy implications of a setup like this. there would of course be numerous drawbacks for library patrons vs an online catalog, but what we are talking about is not a complete replacement, but instead a parallel system available for pacl power users in the same way interlibrary loan services are slower than shelf browsing. for some, the pros of that setup will outweigh the cons. before we jump into the new set of skills that we think will help you avoid many of the issues we encountered. it’s important to mention there are several new features in the world of generative ai that were unavailable earlier this year but will likely be available when this article is published: multimodal support. very soon it will be possible to quickly move between photos (including optical character recognition and translations) and other formats and modes within chatgpt. multi-agent chat interfaces. imagine having multiple chatbots trained to do specific tasks, all working together within the same chat interface. better memory management. in the future your ai assistant will be able to recall past conversations or have access to a custom “knowledgebase” of reference documents, training material, or local practices. new skills while everyone awaits the seemingly inevitable leaps in llm utility, we invite you to consider the following four skills which our experiences suggest library organizations should develop in their staff as we move further into the ai age. skill #1 – maintaining freshness one of the disappointing but necessary limits of chatgpt is that out of the box, it does not have a memory of past conversations. imagine working with an assistant who has seemingly read every book in your library but has zero short term memory. in some ways this person is smarter than you in terms of recall, but they’re very sleepy. all the time. how can you quickly get them up to speed before the clock strikes midnight and they turn back into a pumpkin? thankfully you’re a librarian and providing “just in time” information is not a new concept. use that skill to your advantage. this became extra important during seo experiment #2 when trial and error was a major factor. if you tell chatgpt you’re working in macos, expect different results. it’s that easy. skill #2 – prompt engineering without getting into why this term is misleading and problematic (one could argue it just sounded cooler than alternative options like “llm whisperer” or “query expert”) this is the one we’re stuck with for now. it’s also a skill that is most likely to shift the most, as ai systems continue to evolve and better deduce what you were actually trying to say. prompt engineering is also largely dependent on the domain you’re working in, so for the purpose of this article let’s focus on the general concept of design thinking. one school of thought is that “one shot” prompting is the best option—try to build up your original prompt as much as possible to get instant results. this is extremely satisfying when you get it right, and is great for sharing results with other people (or bots) but could be compared to teaching someone only how to read but not write. for best results, you will probably need to use an iterative process over time to perfect your prompting skills. the best way to increase your prompting skills is to write lots of bad prompts and analyze those results. this is an iterative process, and you are part of that feedback loop. skill #3 – the “reverse reference” interview put skill #1 and #2 together and what do you get? we’re calling this “reverse reference” because in many ways a chatgpt conversation flips the script of a normal front desk interaction at a library. the difference is that this time, you’re both the patron and the librarian. in more practical terms, think about the extra context that might change the direction of the chatbot’s response, that in a normal reference interview happens towards the end of the conversation and try to build that into the front of the conversation. so for example, asking “what does this error message mean?” can create totally different results if you don’t first tell chatgpt a little about what you were trying to do in the first place. think about all those times you’ve talked to a patron who asked “where is the biography section?” without mentioning they have a homework assignment about a local author. well guess what, there’s an entire local authors section at a different library branch! do you see how that conversation could have gone smoother? you can anticipate those needs. words have meanings! be precise and prosper. skill #4 – data remediation since most data is dirty, staff using chatgpt will benefit from learning a few of the steps involved in data remediation. while we found gpt to be quite good at identifying some data cleanup needs on its own, it is incumbent on users to prep their data before handing it off to gpt to get better results. if you do use gpt to help you remediate your data, then be sure to start a fresh chat session to restore the tokens spent in the cleanup process. either way, understanding how to remediate data is a good skill to have, especially when using data with ai. we recommend considering the following data issues your remediation efforts should account for: bias. look out for cultural bias in the data that could lead chatgpt toward biased outputs. corruption. consider the impacts on quality when data is handed off from person to person, or merged between different datasets. with each touchpoint, the chances of error increase so look over the data to ensure data integrity and consistency. intent. ensure the data you are using was intended for how you want to use it and that your assumptions about that data match the findings you hope to draw from it. consistency. if you are working with long-term data, understand that standards that can change over time. again, check for consistency. controlled entries. watch out for a lack of standardized data-entry practices or free text fields. and if you are dealing with free text entries, woe be to you. good news all of the caveats aside, we found using chatgpt to be a major enabler for our team. with great ease, we could leverage expertise and skills that our team is not strong in, realizing previously unheard-of efficiencies in our work. moreover, using chatgpt, we were able to tackle problems that previously were beyond our ability to solve. essentially, we found a new super-star member in our team that could pivot between data analyst or debugger with ease. in our taxonomy project, our gpt data analyst was able to carry out critical and time-consuming tasks that would normally take hours or even days. and it did so in seconds. and when it came to checking our code, gpt offered us what felt like having a senior python engineer on hand to guide us. importantly, this would be a role our library would never be able to afford, so it was a game-changer in that respect. all of this meant that barriers to entry were lowered significantly. projects that would have been impossible suddenly looked possible. to take the taxonomy project as an example, we never would have put the time into analyzing thousands of rows of circulation data. in a team of two with far-flung responsibilities in our library, we would have scoffed at the suggestion, as we scurried off to put out fires with online resources or web content. but with chatgpt, we needed only to dedicate a few hours of time. in fact, having gone through this exercise, we could probably do similar projects in minutes next time. and then there is the wonder of solving highly technical problems with what feels like a very patient guru who you can confide in using plain english. in a technical sense, this is the best human computer-interaction interface imaginable. in our mind, this is the truly killer app of the technology and something we continue to be awed by. whether it’s asking it in natural language to break down an unfamiliar area of knowledge or grappling with complicated technical issues, chatgpt provides your team with a mentor who is patient, non-judgmental and seemingly very interested in having you succeed. as managers, we see this as especially important for removing obstacles for non-technical staff as they solve problems in their work. iv. conclusion based on our experience, we feel that library organizations need to begin developing ai skills in their staff and begin widespread application of this power into their work. however, as with the introduction of any new technology or process, the principles of change management should be applied to help staff adapt their mental models to this new paradigm. indeed, this particular technology is so groundbreaking, and therefore disruptive, that we highly encourage managers to refresh their understanding of moving staff through the change process. you can ask chatgpt for such a refresher or go and read the seminal work of leon festinger [7] on cognitive dissonance and review david kolb’s [8] findings on adult learning. the bottom-line is that you can spare your organization lost time due to change resistance if you manage the introduction of ai into the workplace appropriately. as part of this onboarding into ai, there should be careful attention paid to training staff on the important limitations of chatgpt. hallucinations are a real issue in the world of chatgpt. this was most encountered during the taxonomy experiment when our initial approach simply overloaded gpt by uploading far too many search terms at once, but can also be felt when any individual chat sessions went on for too long. in 2023, one needs to recognize when an llm is not performing well, like the last sputtering of a car running out of gas. cost is also a factor to consider. our experiments relied on an individual chatgpt pro license which costs $20 per month, but the estimated cost for an enterprise solution at our library would be well over $30,000 a year! this is a hefty price to pay without more reliable outputs that could balance these costs with efficiency gains. in sum, facts need to be scrutinized, findings analyzed, hallucinations identified and dealt with. and as we have noted previously, there are new skills one must gain to use the tool effectively. the productivity gains are unprecedented and our experience interviewing staff who have tried gpt is that they also appreciate its potential. even for a small library system like ours, we can see how a variety of work groups could be positively impacted by incorporating gpt into strategic areas of their jobs. that said, as our experiments have also shown, there is still work to be done to improve and fine-tune chatgpt for libraries. in particular, the hallucinations and untrustworthiness of its outputs greatly limits its use by novice users. and while it understands the library domain such as the structure of marc and even the purpose of libraries, its outputs could benefit from greater access and possibly more training on library data. we encourage libraries and particularly technically savvy library staff to begin introducing chatgpt into their workplaces, their projects and processes as long as there is clear understanding of the issues we have outlined above. we believe now is the best time to do so, in fact. llms present a radical reordering of the information landscape and as information professionals, we must keep up with this rapidly changing technology. there is no better way to develop this understanding than to experiment with it here and now, developing an intimate knowledge of its shortcomings but also its potential to overhaul how libraries do their business. references [1] edwards b. 2023 sep 26. can you melt eggs? quora’s ai says “yes,” and google is sharing the result. ars technica. https://arstechnica.com/information-technology/2023/09/can-you-melt-eggs-quoras-ai-says-yes-and-google-is-sharing-the-result/. [2] hays k. ai booster paul graham complains of rise in web content that’s “ai generated seo bait.” business insider. [accessed 2023 nov 2]. https://www.businessinsider.com/paul-graham-complains-web-content-ai-generated-seo-bait-2023-9. [3] building a sitemap for a site. 2020 aug 9. learn wordpress. [accessed 2023 nov 2]. https://learn.wordpress.org/lesson-plan/building-a-sitemap-for-a-site/. [4] chatgpt. chatgpt. [accessed 2023 nov 2]. https://chat.openai.com/share/b4c46ecd-ca1f-44ec-8d25-8f04c25a6ae9. [5] bibliocommons. 2022. 2022 year in review. https://www.bibliocommons.com/news/2022-year-in-review. [6] openai. 2023. privacy policy. openaicom. https://openai.com/policies/privacy-policy. [7] festinger l. 1957. a theory of cognitive dissonance. stanford: stanford university press. [8] kolb da, lewis lh. 1986. facilitating experiential learning: observations and reflections. new directions for adult and continuing education. 1986(30):99–107. doi:https://doi.org/10.1002/ace.36719863012. about the authors m ryan hess is digital initiatives manager at palo alto city library, delivering award-winning library experiences around technology and also a lecturer at san jose state university’s ischool. ryan speaks internationally on topics of digital literacy, web3 and blockchain, artificial intelligence and was the founder of bay area library ux. prior to coming to palo alto, he was the digital services coordinator at depaul university library and a former market researcher at adobe systems. ryan.hess@cityofpaloalto.org chris markman is the digital services program coordinator at palo alto city library, where he oversees library technology services and it systems. he previously served as senior librarian at mitchell park, publishing and presenting extensively on topics like information security, vr/ar, digital literacy, and the future of libraries. he holds an ms in information technology from clark university and an mlis from simmons college. chris.markman@cityofpaloalto.org appendix 1: how to prepare your library for the ai future gartner’s hype cycle contends that technologies move up a steep curve of hype as they impress us with their novelty, only to peak soon after and collapse into the trough of disillusionment. however, long-lasting technologies, gartner contends, eventually rise back as their utility works its way into business processes. in 2023, we viewed library use cases of chatgpt as falling into the trough of disillusionment. but we also see where it will eventually go and this makes us excited. in the meantime, what should libraries do? begin introducing chatgpt to staff, teaching them about it and discovering its power together. for us, the easy applications in this process of introducing chatgpt in libraries would include the following ‘safe’ use-cases. drafting writing of any kind from emails, blogs, web content and even introductions to journal articles (ha ha) are great ways to save time and generate content more productively. get staff involved in any kind of data manipulation possible. use chatgpt to both clean up data and summarize large datasets. while users must be cautious with this use-case, as we learned, using gpt with care and a critical mind can allow individuals to quickly save time and know more about library data than ever before. using gpt as a learning tool. for staff that coordinate training for others, it can quickly create lesson plans and course outlines for you. for staff that need to learn a new skill, you now have a jedi master waiting to show you the force. on the it side, you now have a senior code engineer who can guide library staff from novice to genius when it comes to writing, debugging and learning code. combine this with the training guides and online learning courses that are already part of your library’s collection. at the reference desk, llms can be a quick first pass for staff to consult before applying their own knowledge to a question, particularly in areas they are less familiar with. however, like with data analysis, because of hallucinations, bias and other issues with the current generation of ai, we recommend this always be used with care. note: article edited on 2023 december 5 to add the appendix. subscribe to comments: for this article | for all articles one response to "beyond the hype cycle: experiments with chatgpt’s advanced data analysis at the palo alto city library" please leave a response below: https://cryptolake.online/crypto7, 2025-06-18 https://cryptolake.online/crypto7 leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – how hard can it be? : developing in open source mission editorial committee process and structure code4lib issue 7, 2009-06-26 how hard can it be? : developing in open source in 2000 a small public library system in new zealand developed and released koha, the world’s first open source library management system. this is the story of how that came to pass and why, and of the lessons learnt in their first foray into developing in open source. by joann ransom with chris cormack and rosalie blake introduction look at it from our point of view: we were a very ordinary public library in new zealand, we had hardly any money and a library management system that was going to stop working on 1st january 2000 . . . . what else could we have done? and how hard could it be anyway? the librarians would tell the programmers how a library works and they would make it so. and we weren’t going to make a big deal of this, ok; 3 months is loads of time. and thus koha was born . . . ok, not quite but pretty much. from time to time over the years, we have been asked how koha got started, because it is pretty big now, so here is our story. context horowhenua district is a one hour drive north of wellington, the capital city of new zealand. it is medium in almost every way you can think of. it is of medium population–30,000. it is on the low side of medium in terms of socio-economic status. the district has one medium-sized town, three small towns and a collection of villages and beach settlements. there are libraries in three towns, as well as a tiny volunteer library. the libraries are heavily used and well loved by their community, functioning very much like community centres. but medium does not mean ordinary. in 1996, the horowhenua district council established the horowhenua library trust (hlt) and settled on the trust all the assets of the libraries, except the buildings. from 1 january 1997, the trust would run the libraries. a fixed budget meant the trust could only seek additional funding from council for exceptional circumstances. the move away from the local authority environment allowed a quite different culture to develop in horowhenua libraries. we experienced freedom to explore alternate avenues, to innovate, to take risks in ways that would have been difficult under the direct control of a district council. being medium is an asset here. we’re large enough to reach a critical mass, but not important enough for everyone to be watching every move we make. rosalie blake was appointed head of libraries for the hlt in 1997. she developed a culture within the district libraries of open communication and consensual decision making where everyone is encouraged to help solve problems, no matter what their status in the organisation. the trustees [1] also influenced the organisational culture. in 1999 we had 5 individuals with strong community and business backgrounds who were willing to take calculated risks in order to maximise gains. our partner in this venture was katipo communications, a web development firm based in wellington, new zealand, with whom we already had a long and trusted relationship. rachel hamilton-williams started the company in september 1996, and by january 1997 had hired her first programmer, chris cormack. since then katipo has continued to grow both in numbers and skills, servicing clients throughout new zealand and the world. hlt still works very closely with katipo, a business relationship spanning over 12 years. the situation in 1998 the trust undertook a complete review of all library activities. the report, towards superior service [2], included in-depth focus group interviews of patrons. our patrons made it clear that while they appreciated that computers were a necessary part of a modern library, they did not consider them the most important part. our patrons did not want to see a deterioration of traditional library services (books to borrow, pleasant spaces to sit and browse) in favour of it development, and they did not want new charges imposed to fund sophisticated computer gear. if it came to a contest between the library as a provider of information through technological development, and the library as a community centre which is safe, comfortable and "has the walls lined with books", the people preferred the latter. if we needed to fund new technology, this was a strong warning not to dip into the book budget. in 1999 with a 12 year-old system running on a 386 server, hlt needed to think replacement. while our much-loved dos system still performed very adequately, it looked old-fashioned and somewhat tired. the crunch, however, was that while the system would probably survive y2k, we were sure our networking would not. we asked council for an exceptional circumstances grant, and were successful in getting agreement for council to fund 50% of the cost of a replacement library system. with our management contract with council, we were on a fixed income and responsible for our own finances, which made us very conscious of costs. we had negotiated a favourable deal for the telecommunications link between our branch libraries and the district library, and we were very reluctant to change that. the open source world open source software (oss) is software in which both source code and binaries are distributed or accessible for a given product, usually for free. while "shareware" and "freeware" have been available since the earliest days of computing, oss had developed in the years leading up to 2000 on a different scale entirely. it was no longer confined to the realm of "hobby" programmes. oss projects were starting to produce software that matched or exceeded the quality of commercial products at the time, and linux was starting to challenge windows in very large-scale projects. objectives our overall objective was to source a library system which: could be installed before y2k complications immobilised us, was economical, in terms of both initial purchase and future license and maintenance support fees, ran effectively and fast by dial-up modem on an ordinary telephone line, used up-to-the minute technologies, looked good, and was easy for both staff and public to use, took advantage of new technology to permit members to access our catalogue and their own records from home, and let us link easily to other sources of information – other databases and the internet. if we could achieve all of these objectives, we’d be well on the way to an excellent service. a beginning we started, traditionally enough, by sending a request for a proposal (rfp) to suppliers of library systems. our initial rfp explored the possibility of a joint development in co-operation with our southern neighbours, kapiti coast district libraries. this was an idea worth exploring, but the cost of communication links rendered the joint system prohibitively expensive, and we reluctantly went our separate ways. examination of the proposals revealed: there are systems available that over-deliver, at a cost considerably higher than we wanted to pay, the systems which we could afford met only some of our needs, all the available systems had much more expensive communications solutions than we had been using and most had higher maintenance charges. there simply was no off-the-shelf package that met all our objectives. in commenting on the proposals we received, katipo staff observed that it was a pity none of the available systems used internet technology, since that would take care of the communications speed and costs. did such a system exist? we hunted, and found fragments, and work in progress, but no system that would fit us. "how hard can it be" katipo staff wondered, "to write a library system that uses internet technology?" well, not very, as it turned out. could we write our own? there were barely 15 weeks until d-day. the proposal would develop systems that used www protocols, which was pretty radical at the time, and for which development was significantly easier and faster than under windows (which was the favoured platform for commercial library system providers). could katipo deliver in time? although primarily web designers, katipo had already done lots of database work and an opac for wellington city library. it looked do-able, but it would be tight. katipo had an excellent reputation for attractive, workable web pages for a variety of clients. the brief was that if you could use a web browser, you’d be able to use a library programme which used web technology. the issue of providing external access was no problem at all—that’s what the world wide web was all about. the system would be released under the gnu general public license (gpl) [3], ensuring that the library system we were commissioning would be free/open source software. this was suggested by katipo as a way to ensure the project had longevity; they didn’t necessarily want to spend the rest of their days supporting a proprietary system. koha would thus be available to anyone who wanted to try it and had the technical expertise to implement it. we would encourage other people to use it—because they would improve and enhance it, add to it, fix it and join it up with other good programmes. the gpl would ensure that subsequent modifications and additions by other organisations were open source as well. demonstrating the difference between the trust environment and the local authority environment appealed to us. this project had the potential for establishing a reputation for the hlt as innovators, people who put library priorities in the right order, listened to their patrons and acted upon what they heard. our patrons told us that spending large sums on computers was not their top priority. libraries are it organisations, but we don’t have to buy into high-cost solutions. no-frills efficiency and co-operative development may produce far more in the long run, not just for horowhenua, but for other libraries as well. the undeniable fact that an open-source library system is a public good was not lost on us. a significant part of the cost of commercial library systems is license fees for other commercial software used as components in that system—e.g. the cost of development tools used in the construction of the system, and ongoing license costs for other software not necessarily written by the library system provider (database systems, remote control systems, operating systems). only a portion of the cost of a commercial library system is for code actually written by the library system company. the new system we were thinking about would have two interfaces—a web browser for most purposes, but a simple, fast telnet interface for issues and returns, especially at the branches. the software katipo used to build the new library system was already well proven. it was floss software so as well as the benefits of being fast and robust it came with all the freedoms that we were seeking: debian/gnu linux as the operating system, perl as the programming language and mysql as the relational database. regarding equipment purchases, katipo had a reassuring "let’s-try-the-old-ones-and-see" attitude, before they sent us out to order new pcs for every staff member. there was a potential saving if we didn’t have to buy all new machinery. the trustees invited katipo to prepare a quote, and it was acceptable. the name we struggled with a name for our developing programme before deciding on koha. koha is a maori word from the native people of new zealand meaning gift or donation—or perhaps more like “giving your specialty to the collective event”. there is a sense of quid pro quo or reciprocation about the concept, too. in traditional maori society (and still) you bring a koha (contribution) to an event like a funeral or wedding or big meeting, often food or the specialty of your region. when it’s your turn to hold an event all your guests will bring a koha, to ease the burden of catering for a lot of people. we chose koha as the name, because it’s free and because it’s our gift to the world. figure 1: koha logos the development process the proposed method was prototyping, rather than designing. it started with a fast analysis of what the present system did, and of what we would like it to do. it was the responsibility of hlt staff to ensure the software writers did not miss any key points in their fundamental understanding of the way libraries work. a rudimentary first cut was written quickly and presented, tested, then fixed, rewritten and tested again. librarians tested, explained, clarified their needs and tested again. there was a real onus on library staff to manage quality control. it had to be a partnership process. we saw a danger that hlt staff might get in over their heads—but we were fairly confident that we already had a high level of it competence right through the staff, a high level of understanding of what our current system did and did not do. the essence of this method is that it needs to be written in a short sharp burst of activity. expectations won’t change over a short-duration project. projects that take a long time get caught in expectation creep—end users change their expectations as they see the project develop and long projects get caught in technology creep—the writers just get it nearly finished when something new and different comes along and they have to start again. we saw the initial process as taking something like ten weeks. we finished up with something very tightly tuned to horowhenua’s, and the librarians’ needs. but our finishing point would be but a beginning in the amazing world which is open source software. progress looking once again at our objectives: objective 1: a product that could be installed before the y2k. we did not work on christmas day 1999. we allowed ourselves one day off. but every other day, whether a so-called "holiday" or not, found us hard at it. and on january 5, 2000, the first day back, we had koha installed and ready to go. the minimum (issues, returns, renewals and memberships) was achieved in the nick of time, truly, and by lunchtime we were starting to breathe easy; nothing like a go-live on incompletely tested software to test your mettle! further development continued at a more decorous pace, and by june acquisitions and cataloguing were sufficiently bug-free for us to feel confident about announcing it to the new zealand library world. objective 2: a product that is economical, in terms of initial purchase, license fees and maintenance support fees. this objective had been achieved on all counts. the programming we commissioned cost us about 40% of the purchase price of an average turn-key solution. there was no requirement to purchase a maintenance contract, and no annual licence fees. when the commissioning stage was completed, we hired katipo on an "as needed" basis, fixing bugs and developing minor enhancements. we were able to use many of our older pcs, even including some diskless 486s, by running lots of copies of netscape and telnet on one of our servers, and using these old machines as dumb terminals. only four 7-year old machines had to be discarded and replaced; all other purchases were enhancements to our network and services. objective 3: a product that runs effectively and fast by dial-up modem on an ordinary telephone line. katipo pulled out all the stops on this one. each branch library had a little local network of one issues/returns machine and one opac, and they both worked just fine sharing one ordinary (not isdn, not adsl, just ordinary copper wire analogue) phone line, with an ordinary dial-up modem. the branches worked at very acceptable speed. at levin, we experienced problems during very busy times at the circulation counter. after applying every fix we could think of we achieved some improvement, but the telnet interface still let us down occasionally. finally, the programmers went back to the beginning and rewrote the telnet interface using a different interface library. objective 4: a product that uses up-to-the minute technologies, looks good and is easy for both staff and public to use. up to the minute technology was designed in. looking good is subjective, but we liked it. it was easy to use; any patron with previous experience of the internet had no problems using the opacs at horowhenua libraries, and staff were happy that their part of the system was sensible and intuitive. but remember the internet was still quite new, and in horowhenua we have a predominantly senior population with many of them flummoxed by a mouse and by screens that assumed an understanding of "if it’s underlined and coloured, you can click it for more information". we held training sessions for anyone interested, and taught more people than we care to admit how to play solitaire, for mouse practice. objective 5: a product that permits members to access our catalogue and their own records from home as easily as in the library. the opac was initially accessed only over the intranet, but stage 2 followed within a few months and allowed patrons to access our database from home to check their loans, renew items and place reserves. objective 6: a product that lets us link easily to other sources of information – other databases and the internet. internet access is available at all three libraries, and stage 2 development saw links to other databases, including the past perfect database of the holdings of horowhenua historical societies. community koha was a world first, the first open source library management programme. although our initial development was not yet completed, we announced koha in new zealand in the july 2000 issue of library life. word of mouth in the open source community is very strong, and approval was swift. it was only a matter of hours before we were noticed around the world. we received emails from new zealand, then australia, and fiji, followed by north america and europe. figure 2: koha users as of 16 april 2009 an open source project is never finished. someone will always see something else to improve and we were counting on this. we wanted to encourage a supportive community around koha right from the start. katipo set up web pages for inspection, a listserv for discussion and the means for anyone interested to download the code. within the first month, there were 368 downloads of koha, the project webpage had 2,933 hits, and more than 50 people from around the world were on the mailing list. the first occasion when a technical question was asked, and it was answered by someone who was not working for katipo or the hlt, was a real thrill. very soon after the release of koha 1.0 a newspaper library in new zealand had the system fully installed for evaluation, libraries in poland and estonia had started to translate the front pages and students in america were suggesting additions that they might write as their contributions. since then koha has gone on to win multiple awards including the 3m award for innovation in libraries, the interactive nz award in 2000, and the trophées du libre in 2003. seven years after its release and a thriving project later, chris cormack won the open source contributor award at the nz open source awards in 2007. open source projects only survive if a community builds up around the product to ensure its continual improvement. koha is stronger than ever now, supported by active developers (programmers) and users (librarians) – and they actually talk to each other! a range of tools are employed to nurture the koha community: koha web site, mailing lists: developers and general, koha developers wiki, irc real time chat, koha documentation project, list of koha libraries by geographical region, koha extensions, blog aggregator, koha special interest groups (including spanish, french, italian), conferences and workshops, paid support vendors. support there are a range of support options available for koha, both free and paid, and this has contributed to the overall strength of the koha project. free support can be accessed from the website, and includes irc, mailing lists, faqs, and documentation. the site also lists vendors who provide paid support for those not confident enough with linux to go it alone. these are drawn from around the globe, including not only new zealand but also australia, china, india, pakistan, africa, france, the uk and usa. support has never been limited to third party vendors. steven tonnenson worked on non-acquisitions based cataloguing, then moved on to write a web-based circulation module. his work formed the core of release 1.1.0 (january 2001). pawel skuza’s work on translations formed the core of release 1.1.1 and when 1.3.0 was released in september 2002 multiple flavours of marc were also supported. many of the existing submitters have been involved since the beginning: biblibre’s paul poulain and henri-damien laurent; stephen hedges, owen leonard and joshua ferraro (nelsonville public library); and mj ray from turo technology. they all joined the community in 2002. vendors like anant, biblibre, bywater, calyx, catalyst, inlibro, indserve, katipo, kohaaloha, liblime, libsoul, nchc, osslabs, paklag, ptfs, sabinet, strategic data, tamil and turo technology take the code and sell support around the product, develop add-ons and enhancements for their clients and then contribute these back to the project under the terms of the gpl license. assessment and learning looking back from a distance of 10 years, there are a few things we should have done differently: participation firstly, we did not participate in the community. once we had our own koha up and running, and had made the code available for download, we basically abandoned it. we relied on katipo to represent us in the koha community when we should have taken responsibility for this ourselves. i guess we had scratched our own itch and simply moved on, but there was also an element of a medium-sized public library in new zealand developing something simple and then going head to head with big american libraries who needed it to be more. what we should have done was stayed active and joined in discussions, articulating and defending our approach and vision. koha was designed in what is now described as a frbr [5] arrangement, although of course it wasn’t called that 10 years ago, it was just a logical way for us to arrange the catalogue. a single bibliographic record essentially described the intellectual content, then a bunch of group records were attached, each one representing a specific imprint or publication. attached to the group records were any number of items or individual copies, and items inherited all the attributes of the group they were attached to, and groups inherited all the attributes of the bibliographic record they were attached to. this made it very easy to reserve the first item which became available; really good when you didn’t care if you got the collins 1999 copy of ‘a title’ with 100 pages or the random 2007 paperback edition of ‘a title’ with 124 pages. we should have had the courage of our convictions and also the confidence to defend the decisions we made. to this day i don’t think anyone understands that structure, although remnants of it remain in the current 3.0 release. the current rda [6] work is quite exciting for us and it feels like we will be coming full circle in due course. stay with the main trunk we also made the mistake of not carrying out regular upgrades in order to stay in the main development trunk. we had such a perfectly working, highly customized and very stable library management system that we really didn’t see any need to upgrade to something which we didn’t need. for example, koha was developed to handle multiple flavours of marc quite early in its development, something we had completely ignored when designing the system. we could see no obvious benefit to be gained from cataloguing in marc, preferring to continue cataloguing by filling out fields on a form. implementing marc into koha ensured the widespread appeal of the programme but it did make our frbr system work very awkwardly. basically koha forked and we remained on a rock solid but isolated branch with a system we had to maintain ourselves, which meant that every single upgrade forever into the future would require extensive customisation. this effectively meant we did not receive any of the financial gains of oss software development. we held out for 5 years before finally upgrading from v 1.x to v 2.2.4 in 2005. this was a massive upheaval and key modules which had worked perfectly for us for 5 years were irrevocably broken in our eyes; periodicals, simple acquisitions and catalogue maintenance were the ones we mourned for the longest! koha was essentially a whole new programme by this point, and we had absolutely no one to blame but ourselves. we are philosophical about it now; it’s the nature of open source. to control or shape development you have to participate: either contribute code yourself or fund development, but definitely join in the discussion. oss development is much more consensual than you would think. ideas and issues are tossed around until a clear path forward is agreed upon and decisions tend to be stronger for it. applying the learning: koha 3.0 the release of koha 3.0 in late 2008 brought koha completely into the web 2.0 age and all that entails. we are reconciled to taking a small step back for now, but the frbr logic is around and rda should see us back where want to be in a year or so – but with all the very exciting features and opportunities that koha 3 has now. this time we are determined to take control of our own system and not operate in isolation. while we weren’t quite up to downloading the code and getting it all working, we have taken responsibility for working through all the system preferences and settings, and working out the best arrangement of the collection within the database. there are so many ways to make koha work, not just one right way, and while this is a great thing it has taken a bit to get used to. we have invested many hours in trying to understand the new terminology, like what "item types" mean in koha 3.0 as opposed to "collection codes". it is different from how we used them in 1.0 and in 2.0 also. it’s definitely better but it has taken us time to understand the logic behind changes made from 1.0 to 2.0 then on to 3.0. we are getting involved again, benefiting from the generosity of the community in sharing learning and experience. every time we have asked for help, help has been at hand. in return we have made a conscious decision to participate fully on the lists, sharing the knowledge we have gained. in the early days, the koha list appeared to have been dominated by programmers but i have noticed a lot more librarians participating now. this is great as we do understand koha from a different perspective, but we need to continue to develop koha together. conclusion the declining global economic situation means libraries must review how they invest their operating budgets to maximise return to the communities they serve. to quote the recently published darien statements [7], we need to: "adopt technology that keeps data open and free, abandon[ing] technology that does not." the time is right for oss. it fits the library philosophy of sharing and accessibility and cooperation. developing and releasing in open source has worked well for horowhenua library trust. we started the ball rolling with a local solution but like a snowball it gained its real power through the momentum gained of a global community of brains and expertise. our name for our project may have been a premonition, but we’re proud to have offered our koha to the library world. e iti noa ana na te aroha small gift, given in love (mäori proverb) references [1] the trustees of horowhenua library trust in 2000 were: alan hercus (chair), george sue, heather birrell, louise robbie and peter hodge. [2] rosalie blake, towards superior service : a review of the operations of the horowhenua library trust (june 1998). available at: http://kete.library.org.nz/site/topics/show/16-towards-superior-service [3] http://www.gnu.org/copyleft/gpl.html [4] http://en.wikipedia.org/wiki/free_and_open_source_software [5] frbr : functional requirements for bibliographic records. a conceptual entity-relationship model developed by ifla that represents a more holistic approach to retrieval and access as the relationships between the entities provide links to navigate through the hierarchy of relationships. the model is significant because it is separate from specific cataloguing standards such as aacr2 or isbd. for more information: http://www.loc.gov/catdir/cpso/whatfrbr.html [6] rda: resource description architecture is the new cataloguing standard being developed to replace aacr2 rev. in 2009. it goes beyond earlier cataloguing codes in that it provides guidelines on cataloguing digital resources and a stronger emphasis on helping users find, identify, select, and obtain the information they want. rda also supports clustering of bibliographic records to show relationships between works and their creators. this important new feature makes users more aware of a work’s different editions, translations, or physical formats. underlying rda are the conceptual models frbr (functional requirements for bibliographic records) and frad (functional requirements for authority data). for more information: http://www.collectionscanada.gc.ca/jsc/rdaprospectus.html [7] the darien statements were written by john blyberg, kathryn greenhill and cindi trainor in the days following the "in the foothills: a not-quite summit on the future of libraries" at which participants were instructed to "come prepared to help sketch out the role librarians should play in defining the future of libraries". this was held in darien, usa, 2009. more information: http://www.blyberg.net/2009/04/03/the-darien-statements-on-the-library-and-librarians/ about the authors joann ransom is currently deputy head of libraries at horowhenua library trust, levin, new zealand. she has worked on two award winning open source projects: as one of the team who developed koha back in 1999-2000, and more recently leading development of kete, an open source project for building a community digital archive of articles, images, audio, video and documents. joann is a professional librarian with qualifications in english literature and computer network administration. she regularly blogs and tweets jransom, and leads a quiet life in a small coastal village with her menagerie of children and pets. christopher cormack has a bsc computer science and a ba mathematics and maori studies. while working for katipo communications he was the lead developer of the original version of koha, which went live at horowhenua library trust january 5, 2000. since then he has served various roles in the community, release manager, qa manager and currently translation manager. christopher believes in free software, and allowing users the freedom to innovate. rosalie blake has worked at horowhenua libraries since 1981, in a variety of positions which she keeps reinventing. currently head of libraries for the horowhenua library trust, which came into being in january 1997. loves computers, email, web pages . . . so it’s not surprising she was willing to take a punt with koha, the world’s first open source library system. she has served several terms as regional and national councillor for lianza, being responsible for the great new zealand tv turn-off (a library week promotion), and the 1995 edition of the public library standards for new zealand libraries. she has an absurdly large garden which she shares with a cat and a dog. subscribe to comments: for this article | for all articles 7 responses to "how hard can it be? : developing in open source" please leave a response below: joann ransom, 2009-06-26 correct link for towards superior service report: http://kete.library.org.nz/site/topics/show/16-towards-superior-service joão, 2009-07-04 a good history about koha ils and open source for libraries. thanks. verónica lencinas, 2009-08-06 “frbr arrangement”: it was one of the first things i liked when i saw the database structure of one of the first releases. i thought: wow, they implemented frbr! but there was no marc support and that was very bad for our team. (very happy koha 2.2.9 and koha 3 user from argentina) topic 6 task » oua imt study area, 2012-08-23 […] ransom, j., cormack, c., & blake, r. (2009). how hard can it be? : developing in open source. code4lib journal, (7). retrieved from http://journal.code4lib.org/articles/1638 […] topic 6 task » tis assessment one, 2015-02-12 […] ransom, j., cormack, c., & blake, r. (2009). how hard can it be? : developing in open source. code4lib journal, (7). retrieved from http://journal.code4lib.org/articles/1638 […] topic 6 task » information services foundation practicum, 2015-09-03 […] ransom, j., cormack, c., & blake, r. (2009). how hard can it be? : developing in open source. code4lib journal, (7). retrieved from http://journal.code4lib.org/articles/1638 […] topic 6 task – blog recycle test, 2015-12-30 […] ransom, j., cormack, c., & blake, r. (2009). how hard can it be? : developing in open source. code4lib journal, (7). retrieved from http://journal.code4lib.org/articles/1638 […] leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – code4lib: more than a journal mission editorial committee process and structure code4lib issue 2, 2008-03-24 code4lib: more than a journal it is a pleasure and an honor to be able to introduce this, the second issue of code4lib journal. code4lib is much more than a journal. it is a thriving community. by eric lease morgan something for everyone each article in this issue has a little bit of something for all who call themselves a librarian or work in a library. each identifies some sort of library problem to be addressed, and offers one or more solutions. many are complete with code snippets. after all, this is code4lib. for example, people in public service may be interested in edward m. corrado and kathryn a. frederick’s review of database-driven subject guide applications. kenneth furuta and michele potter describe a simple help system that brings librarians running to the reference desk. margaret mellinger and kim griggs explain how library resources can be organized into course pages without the need of html knowledge and yet sport web 2.0 features. nancy fried foster, nora dimmock, and alison bersani shed light on participatory design. for those of us who enjoy cataloging and metadata issues, jonathan gorman outlines how he modified vufind to exploit wikipedia and cataloging authority records to enhance information about authors in a library catalog. chris freeland, martin kalfatovic, jay paige, and marc crozier illustrate a different use of library of congress subject headings by integrating place names with google maps. carol jean godby, devon smith and eric childress describe a technique for crosswalking just about any metadata format into just about any other metadata format. for the systems librarian in you, dan scott and kevin beswick share how they used linux live cds customized as kiosk browsers to provide laptops as ‘quick lookup’ stations at their library. andrew darby takes advantage of the google calendar api to easily manage the display of library hours. jody deridder exploits google sitemap technology and static html pages to make content in the “deep web” more accessible. we hope you find these articles useful, stimulating, and relevant to your daily working lives. we’re growing! a few months ago we put out a call to solicit more code4lib journal editors. we received quite a number of well-qualified applicants, and selecting a few from the pool was very challenging. we are happy to announce that christine schwartz, andrew darby, and ryan wick will be joining the editorial committee. “welcome!” anyone else interested in helping with the journal, whether or not they are on the editorial committee, is welcome to participate on our public discussion list. the code4lib community code4lib is more than a journal. it is a community — a group of loosely federated problem solvers who work in libraries and exploit computers to solve library problems. the exact date when it all began is impossible to track down, but it can probably be traced back to november 2003 when ed summers, robert fox, chuck bearden, dan chudnov, and myself discussed the possibilities of starting up a new mailing list. ed stated it most succinctly: code4lib would be for *any* language, not just perl. people could discuss non opensource software (although they probably wouldn’t want to), and conversation will not be limited to xml. it will be a discussion list for programmers, who like programming in/for libraries or dealing with information sciencey things. thus the code4lib mailing list was born. [1, 2] presently the mailing list includes about 900 subscribers from all over the world, and it grows by about one person every other day. since then an irc channel has become an additional venue for discussion. [3] it is a “place” where real-time discussions occur. “what is the proper syntax for this command?” “have you heard about that new api?” “how do you suggest i tweak this script?” the channel is full of mis-spellings, any number of seemingly random thoughts, bad jokes, and simultaneous conversations whose participants talk right past one another. it is inhabited by a robot named zoia who tells you the weather, searches google, plays hangman, defines words, and constantly trawls the ‘net for relevant rss feeds. it is a place where “karma” is increased or decreased by incrementing and decrementing usernames or words through the use of double plus and minus signs (ie. books++ or taxes–). presently mike giarlo, ed summers, and ross singer have the highest karma. in this seemingly chaotic environment relationships are built, ideas are formulated, and problems are solved. more recently the code4lib conference has come into being. there have only been three conferences, but each one has been filled to capacity and greatly anticipated each year. code4lib is more than the discussion on a mailing list, more than the idiosyncrasies of text-based chat sessions, more than a conference, and more than journal articles. code4lib is a growing community exhibiting the characteristics of the internet. it is a decentralized organization with no defined leaders. there is very little formal governance and when consensus is not apparent voting is employed. processes are relatively transparent; we value collaboration. there are no dues. everybody contributes what they can and the resources just seem to flow. sociologists and anthropologists would have a field day if they studied the inner workings of code4lib. consider participating in code4lib. we would love to have you, and we are sure you have something significant to offer. you don’t need to know how to program. you don’t need to be an expert in computers. you just need to be a person who enjoys a collaborative working environment where diverse and sometimes conflicting ideas abound. code4lib++ notes and links [1] these quotes are buried deep in the code4lib mailing list archives. [2] the official home page of the code4lib mailing is http://dewey.library.nd.edu/mailing-lists/code4lib/. [3] find an irc (internet relay chat) client like irssi or a use gateway like http://mibbit.com, connect to chat.freenode.net, and /join #code4lib to share your ideas and become a part of the community. find out more at http://www.code4lib.org/irc. tags: code4lib journal, editorial introduction subscribe to comments: for this article | for all articles 3 responses to "code4lib: more than a journal" please leave a response below: mike giarlo, 2008-03-25 i won’t say the karma system is rigged, but that i am currently atop the pile exposes deep and fundamental flaws in it. great introduction to the issue and the community, eric! adam chandler, 2008-03-25 thank you for putting out a fine issue, eric. i wonder, though, does batching these articles up into an issue really add any value? why not release the articles through the rss feed as soon as they make it to the end of the editorial process? i would rather read one here and one there over the course of few weeks. eric lease morgan, 2008-04-02 adam, the editors discussed the pros and cons of publishing the journal in issues vs the continuous release model. the latter model is, in some ways, easier from the editor’s point of view. but, we came to agree that publishing issues seemed like the better choice for our particular group dynamic. we also thought that many of our readers still favored discrete issues and a predictable publication schedule. if, in the future, we get more article submissions than we feel we can deal with using this model, we may change our mind rather than adopting a more frequent publication schedule. nothing is irrevocably set. –elm leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – real-time reporting using the alma api and google apps script mission editorial committee process and structure code4lib issue 58, 2023-12-04 real-time reporting using the alma api and google apps script when the university of michigan library migrated from the aleph integrated library system (ils) to the alma library services platform (lsp), many challenges arose in migrating our workflows from a multi-tier client/server structured ils with an in-house, locally hosted server which was accessed by staff through a dedicated client to a cloud-based lsp accessed by staff through a browser. among those challenges were deficiencies in timely reporting functionality in the new lsp, and incompatibility with the locally popular macro software that was currently in use. while the alma lsp includes a comprehensive business intelligence tool, alma analytics, which includes a wide variety of out-of-the-box reports and on-demand reporting, it suffers from one big limitation: the data on which the reports are based are a copy of the data from alma extracted overnight. if you need a report of data from alma that is timely, analytics isn’t suitable. these issues necessitated the development of an application that brought together the utility of the alma apis and the convenience of the google apps script platform. this article will discuss the resulting tool which provides a real-time report on invoice data stored in alma using the google apps script platform. by david fulmer introduction a long-standing and important workflow in the university of michigan library’s acquisitions area involves exporting information about invoices from our integrated library system (ils)/library services platform (lsp) to the university of michigan’s central financial system. before the migration from aleph to alma by the university of michigan libraries in 2021, a well-established local workflow using a combination of electronic data interchange (edi), spreadsheets, macro software, and manual data entry prepared the invoice information in aleph for its export, processing, and importing into the university’s central financial system. but this workflow was not compatible with the new lsp, alma, and needed to be adapted on a tight deadline. alma has an integration to interface between alma and financial systems called the enterprise resource planning (erp) integration [1]. included in this integration is an alma job to ‘export invoices for payment’. invoices to be exported from alma are designated by being given the invoice workflow status “waiting to be sent”. the integration job runs nightly and changes the invoice workflow status of exported invoices in alma to “closed” and creates an xml file of the exported invoices as output. this exported xml file is processed locally into a format that can be understood by the university’s central financial system and then transmitted to the university’s central financial system via email. two key components of this export process are the presence and correct formatting of an invoice reference number, or batch number, and a properly formatted invoice note in each invoice exported so that each exported invoice can be included in the appropriate batch and imported into the university’s central financial system successfully. if an invoice lacks a batch number, or if the invoice does not have a note that conforms to the proper formatting, the export of the invoice information from alma and its import into the university’s central financial system will not be successful, requiring time-consuming manual intervention. as may be seen, because each invoice is exported the night that it is processed, any quality check of the invoices must be performed before the export of the invoices that evening. alma analytics [2] is the reporting platform built into alma and it provides extensive reporting capabilities in subject areas of alma including users, loans, fulfillment, electronic inventory, physical inventory, and acquisitions. however, the reports produced by alma analytics are based on a copy of the data from alma made daily for the purpose of querying, so a report that needs to have current data from alma cannot be produced by alma analytics. the alma analytics reporting platform cannot be used to review invoices before export as the changes made to the invoices throughout the day will not be reportable in alma analytics until after the full database refresh or extract, transform, and load (etl), as this happens nightly after the invoices have been processed. and double-checking invoices in the alma ui would be a time-consuming manual process consisting of many clicks. before migrating to alma, the library was able to use macroexpress, an application that could navigate around the aleph client in imitation of user actions, clicking on areas of the client or designating fields, and aiding in the invoice processing workflow by helping to format invoice notes correctly. but as the migration to alma approached, it was recognized that the browser based-ui of alma was not compatible with the tools developed for aleph. research into more modern information technology tools, particularly those that would be compatible with the alma apis, led to experiments with google apps script integrated with a google sheet as a potential replacement and improvement on the obsolete workflows. google apps script google apps script [3] is a cloud-based programming platform that is tightly integrated with the google workspace suite of applications like google sheets, google docs, and google drive. in a google spreadsheet, google apps script is just a menu click away and comes with its own integrated development environment for writing sharable scripts that interact with and modify the google files. it can also incorporate other apis, including the alma apis, to fetch data from outside sources and incorporate it into a google spreadsheet. google apps script uses a scripting language that is based on javascript with a few differences, mostly additional methods google provides to make the platform even more useful. there is an included integrated development environment with a script editor that has a variety of sophisticated programming tools like autocomplete, logging, and debugging tools, making it an excellent choice for either the seasoned programmer or a novice. it has been around for over ten years (as of 2023), and in 2020 the integrated development environment was completely redesigned [4]. google apps script makes it possible to have a google spreadsheet with a custom function that queries the alma api to gather current data about invoices in alma, formats that data, and then displays that data in the google spreadsheet. alma rest apis the alma lsp comes with a suite of apis [5] that allow direct, programmatic access to the data within alma. grouped into categories such as acquisitions, bibliographic records and inventory, and users and fulfillment, the apis make it possible to “read” data from alma at any given moment, along with updating many kinds of data in alma, from bibliographic records to patron accounts. to use the alma api, you must create an alma api key, and the permissions granted to any given key may be very granular. options include environment: production or sandbox, permission: read-only access to alma, or read/write access to alma, and api keys may be restricted to particular functional areas like acquisition, bibs, or users, among others. for our purposes, an api key with acquisition, production, and read-only parameters was sufficient to get data out of alma and report on it in a google spreadsheet. the report in order to create a report that would include real-time data from alma which could be used to double-check that all invoices due to be exported had a required batch number, and had a properly formatted note, it was necessary to create a google spreadsheet with special formatting and a custom function. the custom function script was written and deployed in the apps script extension of the google spreadsheet. library staff are able to use the google spreadsheet to call up a real-time report of invoice data from alma. the custom function script communicates with the alma api and returns information about invoices in the “waiting to be sent” invoice workflow status – those due to be exported that night – displaying information about each invoice in alma due to be exported along with conditional formatting that highlights invoices that need to be edited in order to avoid problems with the export from alma and import into the university’s central financial system. the report is designed to be used at the end of the day before the ‘export invoices for payment’ job runs automatically to check if all the invoices that are “waiting to be sent” have a batch number (an invoice reference #) and a properly formatted note. the enhanced google spreadsheet can identify these two types of problems so that invoices that have them can be corrected or put back in review status so they won’t be exported that day. the script the script used to create the report [6] queries the alma api. here is how it works. to add a custom function to a google spreadsheet, you simply click on extensions > apps script at the top of the spreadsheet, which opens the apps script code editor. this code editor allows you to add programming to the google spreadsheet to do a wide variety of things in a container-bound script attached to the spreadsheet. this script begins by declaring the api key as a global variable: var apikey = ‘your_api_key’; it will be used in the main function but could also be used in other functions were there to be more added later. the custom function doing all of the work of getting the data out of alma is called “getnotes”. ‘function’ is a keyword in javascript indicating where the function begins, “getnotes” is the name that is used for this custom function, ‘(input)’ is the input parameter for the function, and the first opening curly bracket shows where the function begins, and is matched by a closing curly bracket at the very end of the function: function getnotes(input) { if (input){ the content ‘=getnotes(b1)’ is placed in cell a8 of the sheet called “sheet1” so that when content is added or changed in cell b1, the function will execute. it may be helpful to think of the getnotes function analogously to a built-in function such as ‘sum’. were we to put ‘=sum(2+b1)’ into cell a8, then a8 would display ‘2’ if b1 were empty, ‘3’ if b1 had the value ‘1’, and so on. cell a8 would display the result of the sum function adding 2 to the contents of cell b1. getnotes is a custom function which, like sum, will return data, but in this case, it will return not the result of a math operation, but the result of a call to the alma api to retrieve invoice information which is then formatted and displayed in cell a8 and other adjacent cells. the next section of the script retrieves invoice information from alma. var limit = 100; var offset = 0; var url = 'https://api-na.hosted.exlibrisgroup.com/almaws/v1/acq/invoices/?limit='+limit+'&offset='+offset+'&invoice_workflow_status=waiting%20to%20be%20sent&apikey='+apikey; var xml = urlfetchapp.fetch(url).getcontenttext(); var document = xmlservice.parse(xml); var totalrecordcount = document.getrootelement().getattribute('total_record_count').getvalue(); xml = xml.replace(/<\?xml version='\"1.0\" encoding=\"utf-8\" standalone=\"yes\"\?><invoices total_record_count=\"\d+\">/, "")'; xml = xml.replace(/<\/invoices>/, ""); var fullresponse = xml; the first call to the api will retrieve information about the first 100 invoices that have the status “waiting to be sent”. the url in this code will vary based on location in the world (north america, europe, asia pacific, canada, or china) and the call to an alma api is directed to different urls depending on the location of your library. the alma api console documentation provides details about constructing api calls and in this case it is a “get invoices” request utilizing the “invoice_workflow_status” parameter equal to “waiting to be sent”. the reply is in xml format and the script edits this reply a little in case there are more than 100 invoices with this invoice workflow status. the next section of the script is a loop, only invoked when there are more than 100 invoices with the specified invoice workflow status. while (totalrecordcount > offset + limit) { offset += 100; var url = 'https://api-na.hosted.exlibrisgroup.com/almaws/v1/acq/invoices/?limit='+limit+'&offset='+offset+'&invoice_workflow_status=waiting%20to%20be%20sent&apikey='+apikey; var xml = urlfetchapp.fetch(url).getcontenttext(); xml = xml.replace(/<\?xml version='\"1.0\" encoding=\"utf-8\" standalone=\"yes\"\?><invoices total_record_count=\"\d+\">/, "")'; xml = xml.replace(/<\/invoices>/, ""); fullresponse = fullresponse.concat(xml); } this is not a common situation, but by changing the offset, it is possible to retrieve the information for more than 100 invoices, which is the default limit for the get invoices api request. next, the script edits the reply, combining all requests into one xml document, parses that document, and sets up the ‘invoices’ variable with the invoice information that has been retrieved from alma. fullresponse = '<?xml version="1.0" encoding="utf-8" standalone="yes"?><invoices total_record_count="">' + fullresponse + '</invoices>'; var documentb = xmlservice.parse(fullresponse); var invoices = documentb.getrootelement().getchildren('invoice'); at this point, it might be useful to review the sample of an xml-formatted get invoices request from the alma api. the api console inside the developer network is a development environment provided by ex libris that enables you to test out the alma apis on sample data or on your library’s data. figure 1. sample of part of an xml-formatted get invoices request response from the alma api. this screenshot shows the api console inside the developer network. next, a few variables are established to store data throughout the script and to be presented in the spreadsheet at the end. var invoicearray = []; var space = [" "]; var invoicearraybad = ["these below are not okay"]; the ‘invoicearray’ is used to gather information about all the invoices in the specified status, the ‘space’ variable is used for formatting, and the ‘invoicearraybad’ is just what it sounds like: this needs attention. the script then begins a loop through all the invoices. for (var i = 0; i < invoices.length; i++) { var invoicegoodorbad = 1; var invoicenumber = invoices[i].getchild('number').gettext(); var vendorcode = invoices[i].getchild('vendor').gettext(); var referencenumber = invoices[i].getchild('reference_number').gettext(); each invoice starts out with the ‘invoicegoodorbad’ variable set to 1, which means it needs attention, but if it is found to be acceptable this variable will be changed. then the invoice number, vendor code, and reference number or batch number are retrieved from the data sent by the alma api. next, there is a check for the reference number, and if there is none, a message, ‘no batch number’, the invoice number, and the vendor code of the invoice are added to the invoicearraybad array. if (!referencenumber) { invoicearraybad.push(["no batch number",invoicenumber," ",vendorcode," "]); referencenumber = 'none'; } then the script evaluates the invoice notes. var invoicenotes = invoices[i].getchild('notes'); if (invoicenotes) { var invoicenote = invoicenotes.getchildren('note'); for (var j = 0; j < invoicenote.length; j++) { var notecontent = invoicenote[j].getchild('content').gettext(); var patt = /^act......,.............-...,.,\[p\]$/; var goodorbad = patt.test(notecontent); if (goodorbad) { var pattern = "yes"; invoicegoodorbad = 2; } else { var pattern = "no"; } invoicearray.push([pattern,invoicenumber,referencenumber,vendorcode,notecontent]); } if (invoicegoodorbad < 2){ invoicearraybad.push(["no good note",invoicenumber,referencenumber,vendorcode," "]); } } else { invoicearray.push(["no lines",invoicenumber,referencenumber,vendorcode,"no lines"]); invoicearraybad.push(["no lines",invoicenumber,referencenumber,vendorcode,"no lines"]); } } the invoice notes are retrieved one by one and checked against the pattern required by the invoice processing system for successful exporting of the invoice to the university’s central financial system. if there is at least one note in the invoice that matches the pattern then the invoice is classified with the invoicegoodorbad value set to ‘2’, if not, the invoice gets added to the invoicearraybad array as an invoice flagged for attention. an invoice may have more than one note, but each invoice needs to have one note which matches the pattern necessary for invoice processing. the script finishes up by building up an object, ‘invoicearraye’, with all the output from the alma api response having been processed by the script to return the data formatted in a way that makes it easy for library staff to review the invoices according to the needs of the invoice processing system. the ‘invoicearraye’ is a combination of the invoicearraybad information, followed by a space, followed by another row with “these below are all notes”, followed by all the ‘waiting to be sent’ invoices with their accompanying notes. because “=getnotes(b1)” is placed in cell a8, all this information is displayed in cell a8, to the right of cell a8, and below cell a8. var okaymessage = ["these below are all notes"]; invoicearraye = invoicearraybad.concat(space).concat(okaymessage).concat(invoicearray); return invoicearraye; conditional formatting the final piece of the spreadsheet is the conditional formatting rules, which have been added to the sheet to highlight certain rows and cells of the report. this screenshot shows the display of three invoices that are currently in the ‘waiting to be sent’ status in alma with a variety of problems, and one invoice that passes all the quality check tests and is ready to be exported from alma and imported into the university’s central financial system without any problems. figure 2. the display of three invoices in the note check spreadsheet. invoice number 4000 has a batch number (‘c1234’) and a properly formatted note (‘act615590,vid0000652571-001,d,[p]’), so it does not appear on the first section of the report below the first green row, but it does appear below the second green row, which is a section of the report intended to show each invoice note. invoice number 4001 also has a batch number (‘z123’) and a properly formatted note (‘act615590,vid0000652571-001,d,[p]’), so it also does not appear on the top part of the report. however, the batch number does not start with the letter ‘c’. when it was recognized that this was a potential issue, a conditional formatting rule (“=not(regexmatch(c9,”^c”))”) was added to the sheet to highlight batch numbers without the proper format in purple, which may be seen in row 15, column c. figure 3. a conditional format rule in the spreadsheet. invoice number 4002 does not have a batch number and it does not have a properly formatted note, so it appears twice on the report, in row 9 and row 10, to flag these two problems. finally, invoice number 4003 does have one note which is shown in row 16, it also has a batch number in the correct format so cell c16 is not highlighted, but the note that it has is not properly formatted so it is flagged in row 11 with the note in cell a11 “no good note”. conclusion having this tool to help identify potential problems in the processing of invoices has helped immeasurably and has served the university of michigan library well for over two years. mistakes with formatting and data entry are an unfortunate part of any procedure involving processing thousands of documents and transmitting their salient information between two systems – the library lsp and the university’s central financial system. this report with its custom function overcomes the limitation of day-old data in alma’s integrated reporting tool, alma analytics, by utilizing the alma api to retrieve real-time invoice data. using the powerful, accessible tools of the google apps script platform, this data can then be checked and formatted in a report that flags issues before they develop into problems with the export of financial information from alma and into the university’s central financial system. it was also particularly helpful to library staff during the transition from the aleph ils to alma when workflows were being adapted, new processes needed to be learned, and older tools were not compatible with the new system. a new lsp necessitated the development of new information technology tools simply to carry on with the work that needed to be done. citations [1] alma – financial systems: https://knowledge.exlibrisgroup.com/alma/product_documentation/010alma_online_help_(english)/090integrations_with_external_systems/020acquisitions/010financial_systems [2] alma analytics: https://exlibrisgroup.com/products/alma-library-services-platform/alma-analytics/ [3] google apps script: https://developers.google.com/apps-script [4] google apps script release notes for december 07, 2020: https://developers.google.com/apps-script/docs/release-notes#december_07_2020 [5] alma rest apis: https://developers.exlibrisgroup.com/alma/apis/ [6] notes check app: https://github.com/dfulmer/notes-check about the author david fulmer is an applications programmer/analyst intermediate at the university of michigan library. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – scraping bepress: downloading dissertations for preservation mission editorial committee process and structure code4lib issue 47, 2020-02-17 scraping bepress: downloading dissertations for preservation this article will describe our process developing a script to automate downloading of documents and secondary materials from our library’s bepress repository. our objective was to collect the full archive of dissertations and associated files from our repository into a local disk for potential future applications and to build out a preservation system. unlike at some institutions, our students submit directly into bepress, so we did not have a separate repository of the files; and the backup of bepress content that we had access to was not in an ideal format (for example, it included “withdrawn” items and did not effectively isolate electronic theses and dissertations). perhaps more importantly, the fact that bepress was not sword-enabled and lacked a robust api or batch export option meant that we needed to develop a data-scraping approach that would allow us to both extract files and have metadata fields populated. using a csv of all of our records provided by bepress, we wrote a script to loop through those records and download their documents, placing them in directories according to a local schema. we dealt with over 3,000 records and about three times that many items, and now have an established process for retrieving our files from bepress. details of our experience and code are included. by stephen zweibel institutional context we at the gc share a bepress digital commons instance with the rest of the cuny libraries. we work entirely with doctoral and master’s students, who often have academic work that they submit to our digital commons, which we call cuny academic works. in fact, in order to graduate, students generally must submit their final thesis/capstone/dissertation to academic works. thus all or nearly all dissertations and theses published since we began working with bepress are on academic works. with corporate changes, including the acquisition of bepress by elsevier in 2018, we in the library began to feel like we were losing control of our students’ work. additionally, it became clear to us that academic works is not a complete archival solution, because it does not provide archival quality pdfs. in response, we decided to explore creating our own digital archive using archivematica. in order to accomplish this, we needed the documents and metadata that were stored in academic works. when we requested our data from bepress, it was supplied, but we found its format and organization to be unideal, and a difficult starting point for our work. for example, it included “withdrawn” items and did not effectively isolate electronic theses and dissertations. working with this dataset would have required a great deal of manual work confirming records and editing metadata, far too much for a team of three librarians exploring this in their quieter moments. so, instead, i began to look for programmatic methods to get our documents and metadata out of academic works ourselves. the most likely solution would have been an api (application programming interface), or so i thought. often, websites with a great number of documents, especially those with an academic focus, have an api that allows for programmatic searching and harvesting of records. bepress does indeed provide an api (https://www.bepress.com/reference_guide_dc/digital-commons-oai-harvesting/), but there is an important limitation. supplemental files are not exposed through bepress’s oai (open archives initiative) interface, so these files, which may include a database, set of images, or anything not part of the primary document, would be invisible to any simple harvester i might write. this would render any result incomplete, and thus not at all an archival copy. on top of that, bepress does not provide a sword (simple web-service offering repository deposit) api, which would have made downloading our documents a relatively simple task. problem the problem became a technical one, revolving around the difficulty of retrieving each student’s thesis/dissertation/capstone as well as all supplemental files they had uploaded to academic works. as far as we could tell, there was no method by which bepress had exposed the locations or existence of supplemental files. and yet, there they were, listed on the index page of each document of academic works! so close and yet so far. our final aims were: to download every thesis/dissertation/capstone in academic works, to match each one with its corresponding metadata, and to relocate those files and metadata to a set of directories with unique ids organized such that they could be imported into archivematica. at the moment i was stuck at step one, without a reliable method of getting the files we wanted (and only those files) out of the bepress repository. since the most likely solution to our troubles (a robust api) was not forthcoming, and asking nicely didn’t get us anywhere, the next method to turn to would be web scraping. scraping web scraping is often a last resort in situations where it is necessary to get data off of a website. when describing the technique, i often say that it is like creating a program to click through a website for you using an html map you’ve written for the program. this technique can be unreliable, in that when websites update their architecture, or even when they make small changes to their html, the web scraping script will break, since the map you’ve provided no longer applies. to build the “map” required for the web scraping script, i needed to know the url locations of each document we wanted to download. hence the problem: up to this point we had the url of the primary documents, but none of the supplementary files. figure 1. however, because i had resorted to scraping the pages, and each “index page”, as they are called, lists the supplemental files associated with the document (and, crucially, provides a link to download that file!), i was able to find everything according to a predictable schema and place each file in our predetermined file structure. the rest of this article will demonstrate the process to scrape a website according to a provided schema, or map. figure 2. the schema luckily, bepress provides a “content inventory” report (see image) that allowed us to just pull the metadata of the collections we were interested in, in this case all dissertations/theses, etc.. this would allow us to complete our second objective, pairing metadata to files. more importantly, for each row (which corresponded to a record), there was a corresponding url for the primary document’s index page (see image), where, again, each supplementary file is listed. at this point,i knew what was required: to write a script that would proceed line by line down the content inventory report, go to each index page, and download the primary document and any supplementary files found there. to start with, i needed my script to read the ‘content inventory’ csv, and turn each row into an object, like: import csv csv = open('bepress.csv') csv_reader = csv.dictreader(csv) then, looping through the csv: all_works = [] for item in csv_reader: author = item['author'] pdf = item['pdf_filename'] … together = {'filename': pdf, 'author': author, …} all_works.append(together) and so on. now i have a list of dictionaries that contain the metadata of each item in our collection. as i do this, i can transform each string of text into my desired format. for instance, an author’s name could go from “jane smith” to “smith, jane”. requests when i interact with websites, i turn to the requests library. it’s a library for python that simplifies getting the data from a website into a format that i can play around with. for instance: import requests r = requests.get('https://academicworks.cuny.edu') website_text = r.text the first line would get me the front page of academic works, and ‘r.text’ would contain the entire page as a string, ready for me to parse. this would work equally well for calling a web api–better, in fact, as that would be much simpler to parse. i wrote a function to get the files from academic works, simplified below: def fetch_url(entry): # define the location where i want the files to go path = "dissertations/"+ entry['last_name'] + "_" + str(entry['unique_id']) # the file url i got from the csv uri = entry['download_url'] os.mkdir(path) r = requests.get(uri, stream=true) # checking if the url works, and then starting the download if r.status_code == 200: with open(path + '/' + entry['filename'], 'wb') as f: for chunk in r: f.write(chunk) # getting the supplemental files, discussed below for index, part in enumerate(entry['supplementals']): get_supplementals(index, part, entry['article_number'][0], path) return path here we accomplish our third objective: to relocate our files and metadata to a set of directories with unique ids. the trick to getting the supplemental files is the only real web scraping necessary in this approach. as it turns out, the format bepress uses to create the urls for supplementary files is quite simple, provided you know how to read it. here is one below: https://academicworks.cuny.edu/cgi/viewcontent.cgi?filename=0&article=1784&context=gc_etds&type=additional the first part of that url should be familiar to any user of the web: https://academicworks.cuny.edu. we are interested in the part that comes after the question mark: those are known as query parameters. filename=0&article=1784&context=gc_etds&type=additional the query parameters follow the ? in the request, and are separated from one another by the & symbol. to determine what each parameter does, we can only experiment, because documentation is not available to us. the first query parameter, filename=0, gets the first supplementary file (therefore ‘filename=1’ would get the second!). the second parameter, article=1784, refers to the unique idof the primary document, which we know from our csv. ‘context’ is the particular collection we are looking in, and we don’t have any use for ‘type’! so, a hypothetical 5th supplementary file would have the url of: https://academicworks.cuny.edu/cgi/viewcontent.cgi?filename=4&article=1784&context=gc_etds&type=additional this sort of url is sometimes called an “internal api”, inasmuch as it is an undocumented api; we can still use it to get what we want. the function is as follows: def get_supplementals(which_one, filename, article_number, path): download_url = 'https://academicworks.cuny.edu/cgi/viewcontent.cgi' \ + '?filename=' + str(which_one) \ + '&article=' + str(article_number) \ + '&context=gc_etds' \ + '&type=additional' # download the file s_file = requests.get(download_url) if "error: no such file is available for this article" in s_file.text: return with open(path + '/' + filename, 'wb') as f: f.write(s_file.content) the variable ‘download_url’ is constructed for each entry out of the filename retrieved from the content inventory csv. the ‘article_number’ parameter is which supplementary file (1, 2, 3, etc.) as discussed above. in all of our metadata, we still don’t know how many supplementary files a document has, or whether a document even has any supplementary files! so, we loop through and try to download the first supplementary file, the second, and so on, until we get an error message, which in this case comes as a warning from bepress: “error: no such file is available for this article”. when we see this text, we stop. result now we have achieved our goals: to download every thesis/dissertation/capstone in academic works, we used the requests library and bepress’s internal api; to match each one with its corresponding metadata, we used the “content inventory” provided by bepress; and to relocate those files and metadata to a set of directories with unique ids organized in order to be imported into archivematica, we wrote a line in python that created and named a directory according to each document’s unique id. all this has allowed us to continue moving forward on our project to make a real archival backup of our dissertations, theses, and capstones. of course, this method has limitations, and a few potential improvements come to mind, for example, a way of ensuring that the downloaded files are complete, not corrupted or mistaken, for instance hash checking. (the way i accomplished this, which is not a complete solution, was to download the whole set multiple times and comparing the sets.) subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – generating standardized audio technical metadata: aes57 mission editorial committee process and structure code4lib issue 30, 2015-10-15 generating standardized audio technical metadata: aes57 long-term access to digitized audio may be heavily dependent on the quality of technical metadata captured during digitization. the aes57-2011 standard offers a standardized method of documenting fairly comprehensive technical information, but its complexity may be confusing. in an effort to lower the barrier to use, we have developed software that generates valid aes57 files for digitized audio, using output from fits (file information tool set) and a few fields of information from a tab-delimited spreadsheet. this article will describe the logic used, the fields required, the basic process, applications, and options for further development. by jody l. deridder long-term access to digitized audio may be heavily dependent on the quality of technical metadata captured during digitization.  in 2011, the audio engineering society (aes) released a new standard for technical audio metadata:  the aes57-2011 (audio engineering society [updated 2015]), which covers audio object structures for preservation and restoration (both for analog and digital). prior to 2011, the best standard available for audio technical metadata was audiomd (library of congress [updated 2011]), which is far less extensive. also released in 2011 was the aes60 (audio engineering society [updated 2015]) for descriptive metadata, which extends the unqualified dublin core fields to add an additional 4 fields (version, publication history, metadata provider, and entity type) (audio engineering society 2011). an excellent review of the aes57 standard while in development was provided by jane otto [otto 2010]. broad categories of information covered by this standard include physical properties and dimensions of the analog object, signal characteristics (such as playback speed, sound field, and noise reduction), digital file characteristics (such as sample rate, bit depth, and byte order); and comments about the condition of the object. each audio object is described by a separate aes57 document; thus a 2-reel performance (digitized into two separate audio files) would be described by two separate aes57 documents. this article will describe a method and accompanying perl script used to automatically generate the portions of aes57 technical metadata necessary for schema validation, as part of a digitization process.  the intent is to provide at least the minimal standard technical metadata necessary to ensure the ability to effectively manage digitized audio content long-term. by automating the generation of the aes57 documents, we increase the speed of metadata work while reducing the level of complexity for the digitizing staff. within the aes57, the audio object is described at multiple hierarchical levels: the audio object itself; the face levels within that object (such as the side of a vinyl record); the regions within the face; and the streams within each region.  within the spreadsheet, a line is created for each audio file (audio object) and for each region (or segment) within that file which we are extracting as a separate track for online delivery.  if there are no segments, there is a single line in the spreadsheet for the file.   since an intellectual item may span multiple files (such as a two-reel performance), our file naming system allows for sub-items (regions, or segments for web delivery) at two levels, and the level is indicated by the length of the file name, where the first segment of the file name (21 characters long) specifies the intellectual item, and the remainder indicates the sequence.  in a recording (of multiple musical segments) which spans two reels, for example, the intellectual item is identified by the first 21 characters, and the reel would be identified by the next 5 characters.  since the reel would have multiple musical segments, it would have sub-items, each of which would be identified by more than 26 characters.  had there been only one reel, the sub-items would be identified by the 5 characters following the first 21. if (length($id) > 26){ $itemid = substr($id, 0, 26); # collect all sub entries for this subitem push (@{$mykids{$itemid}}, $id); # an array of sub-sub-identifiers, keyed on the subitem id, in the hash %mykids } elsif (length($id) > 21){ $itemid = substr($id, 0, 21); # collect all sub entries for this item push (@{$mykids{$itemid}}, $id); # an array of sub-identifiers, keyed on the item id, in the hash %mykids } else{ $itemid = $id; } although the aes57 does not require information in the section of the schema designated for physical properties (physicalproperties), it does require a “type” attribute value (in formatlist/formatregion) that may be used to indicate information categorizing the original material with one of these values: analogdiscformatregiontype   (such as a long-playing record) opticaldiscformatregiontype  (compact disc or dvd) analogtapeformatregiontype  (cassette tape or reel-to-reel) digitaltapeformatregiontype  (digital tapes) cylinderformatregiontype (cylinders) wireformatregiontype  (wire) baseformatregiontype formatregiontype as this software (fits2aes) was written for content undergoing digitization, it tries to identify the physical form by parsing text from the spreadsheet “format” field, entered in appropriate rda cataloging form, such as: “1 sound tape reel: analog (10:56 min); 7 inches.” values that must exist in the format field for correct assignment include “tape”, “cassette” or “reel” (with or without “digital”); “disc” (with either “digital” or “analog”); “roll,” or “wire.”   if the type of content cannot be determined, the “formatregiontype” value is substituted for this attribute. undef $phys; if ($phystype =~ /tape/i || $phystype =~ /cassette/i || $phystype =~ /reel/i){ if ($phystype =~ /digital/i){ $phys = "digitaltapeformatregiontype";} else{ $phys = "analogtapeformatregiontype"; } } elsif ($phystype =~ /disc/i){ if ($phystype =~ /analog/i){ $phys = "analogdiscformatregiontype";} elsif ($phystype =~ /digital/i){ $phys = "opticaldiscformatregiontype";} } elsif ($phystype =~ /roll/i){ $phys = "cylinderformatregiontype";} elsif ($phystype =~ /wire/i){ $phys = "wireformatregiontype"; } else{ $phys = "formatregiontype";} all regions must have their identifiers referenced in the formatlist/formatregion@ownerref attribute. the aes57 offers many more fields for further description of the original object in the physicalproperties section, but as they are not required for validation, they have not yet been included.  additional fields would need to be added to the spreadsheet, but this additional information would be useful from an archivist’s perspective, to provide provenance information about the original object. most of the information necessary to capture about the audio object itself can be extracted from the fits (file information tool set (harvard 2015)) extract, though some requires reformatting.  the aes57 fields which we mapped to the fits (version 0.8.4) fields are as follows: aes57 field fits field reformatting required audioobject @id filename format identity@format audioobjecttype/byteorder byteorder little_endian = 0; big_endian = 1 audioobjecttype/audiodatablocksize blockalign audioobjecttype/audiodataencoding audiodataencoding (jhove tool version) audioobjecttype/filechecksum/checksumvalue md5checksum audioobjecttype/filechecksum/checksumcreatedate lastmodified transform to datetime format audioobjecttype/firstsampleoffset offset audioobjecttype/objectcreationdate lastmodified transform into datetime format* formatlist@label identity@mimetype (optional) formatlist/formatregion/wordsize wordsize formatlist/formatregion/soundfield channel 4=surround; 2=stereo; 1=mono formatlist/formatregion/bitdepth bitdepth starttime@editrate and duration@editrate samplerate face/region/numchannels channels duration duration (exiftool) transform into editunit frames* * to be discussed in more detail shortly. templated values: because the checksum type extracted from the fits is the md5 checksum, the script we developed will template “md5” for the value of audioobjecttype/filechecksum/checksumkind.  other options include sha-1 and crc. the audioobject/primaryidentifyer@identifiertype is templated as “file_name” in our software. other options would include shelf_number, umid, or other. since this script was developed for digital files, we template the audioobject@analogdigitalflag as file_digital. other options include analog and phys_digital (for digital content with a permanent physical structure). after installing fits (which requires java), the form of the system call used to generate the fits file on a linux/unix machine is: /path/to/fits.sh  -i   $audiofile   -o $fitsfile  >/dev/null 2>&1 for windows: c:\\path\\to\\fits.bat –i $audiofile –o $fitsfile in this line, “-i” precedes the full path and file name of the audio file, and “-o” precedes the full path and file name of the fits file being generated.  the last portion of the line (after $fitsfile) simply prevents output of information to the command line during the process. after generating the fits file, fits2aes opens the file to check whether the wav is well-formed and valid.  if it is not, the script will output errors identifying the fits file name and the problem found. for example, using perl: open (in, $fits) or die “can’t read in $fits\n”; while ($line = <in>){ chomp $line; # remove newline $line =~ s,\r,,; #remove windows carriage returns if ($line =~ /<well\-formed.*?>true<\//){ $wf = 1;} elsif ($line =~ /<valid .*?>true<\//){ $valid = 1;} } close(in); if (!$valid && ! $wf){ print "$fits: not valid and not well-formed\n";} elsif (!$valid){ print "$fits: not valid\n";} elsif (!$wf){ print "$fits: not well-formed\n";} } if the file generated is not valid and well-formed, correction should take place before proceeding further. the transformation of the fits lastmodified value (in the form yyyy:mm:dd hh:mm:ss-hh:mm) to datetime format (w3c 1997) (yyyy-mm-ddthh:mm:ss-hh:mm) is fairly straightforward: elsif ($line =~ /<lastmodified toolname=\"exiftool\".*?>(.*?)<\//){ $adate = $1; undef $ending; ($ending = $adate) =~ s,.*?(\-.*),\1,; # pull off offset value for reuse after transform $adate =~ s,\-.*,,; # remove offset such as -0500 from end $adate =~ s,([\d]{4})\:([\d]{2})\:([\d]{2}) ,$1\-$2\-$3 ,; # hyphens not colons in yyyymmdd $adate =~ s, ,t,; # add t between date and time in place of space if (! $ending){ $ending = "z";} # add z on end if no offset $datec = $adate.$ending; # add the offset back on } the remainder of the information needed for a validating aes57 can be provided via a tab-delimited export from a spreadsheet.  as noted above, this script supports entries for singletons (one region to be used from an audio file) to file entries followed by multiple regions. fields from the spreadsheet include file name, title, format, clip begin, clip end, region condition notes, condition notes for each of four possible streams, notes on prior or ending streams, direction, label, and speed correction. the aes57 can also accommodate security notes at the level of the region. we accommodate these by semicolon separation of the security note from any other region notes, and preceding the security note with “security: “.  also, as condition notes can optionally be targeted to specific clips of time within a region or stream, such clips of time can be included (in square brackets) prior to the appropriate note within the spreadsheet.  for example, to provide 3 comments (one about security) about portions of a specified region, the following formatting would be expected in the spreadsheet field for region condition notes, in the row dedicated to that region: [12:23-12:30] crackly; security: phone number; [13:01-14:12] background whine each condition note in the aes57 includes a creation date, which for our purposes is the same date as digitization.  modification of this would require either additional spreadsheet fields or additional formatting within the notes field above. sub printnotes{ my ($notefield,$samplerate, $speedcorrection, $datec, $cond) = @_; undef @mynotes; if ($cond =~ /;/){ # multiple condition notes are separated by semicolons @mynotes = split(";", $cond); } else{ push (@mynotes, $cond);} foreach $note (@mynotes){ undef $myclipbegin; undef $myclipend; undef $mystart; undef $myduration; undef $sec; # security note if ($note =~ /\[ *([\d\:]+) *\-([\d\:]+) *\] *(.*)/){ $myclipbegin = $1; $myclipend = $2; $note = $3; $mystart = &calcframes($myclipbegin, $samplerate, $speedcorrection); $myframes = &calcframes($myclipend, $samplerate, $speedcorrection); $myduration = $myframes $mystart; } if ($note =~ /(.*?)security *\:(.*)/i){ $note = $1.$2; $notefield = "securitynote"; print out " <$notefield>".$note."</".$notefield.">\n"; } else { $notefield = "conditionnote"; if ($note){ print out " <$notefield>\n <note>".$note."</note>\n"; if ($myclipbegin){ &printrange("timerange", $myclipbegin, $myclipend,$samplerate,$speedcorrection); } print out " <creationdate>$datec</creationdate>\n"; print out " </".$notefield.">\n"; } } } } time codes for clip begin and clip end fields are entered into the spreadsheet in hours:minutes:seconds format (used by other scripts to generate audio decision list files and to generate derivatives for web delivery).  these time codes are used with the sample rate and speed correction to determine the duration (and starttime, if not zero).  fields in aes57 that can have a time range element include face, region, and condition notes. the time range type (timerangetype) elements include required duration and starttime elements which are of type “editunit” (frames).  each of these requires an editrate and the optional fields factornumerator and factordenominator, all three of which are positive integers.   clarification of these fields comes from the ebucore metadata set (ebu 2013) from which they are drawn (as subelements of the aspectratiotype).  the intent is to describe the aspect ratio of the resource compared to the original recording, in order to indicate the amount of speedup or slowdown used to generate the digitized product. here the factornumerator is the numerator of the ratio and the factordenominator is the denominator, and the sample rate is used for the “editrate” value.  the sample rate is one of the fields extracted from the fits file, and represents the rate at which the audio was sampled, expressed in hertz, such as 22000, 44100, 48000, 96000, etc.  the formula (ebu 2013) used to determine the editunit is: editunit = 1 / (editrate * (factornumerator / factordenominator)) in the spreadsheet, the speed correction value is entered as whole numbers and decimals:  for example, 1.5 indicates the speed is one and a half times faster, whereas .5 means the speed is half as fast as the original.  this is translated into a numerator and denominator in the following way: if ($speedcorrection){ $numerator = $speedcorrection * 100; $denominator = 100; } so if the speed was doubled, the value “2” is entered in the spreadsheet, and the numerator becomes 200 while the denominator becomes 100, as 200/100 = 2. thus if the sample rate (editrate) is 44100 and no change has been made in the speed, then an editunit value is:  1/(44100*(1/1) = 0.00002267573696. however, if the speed correction value is 2, the editunit becomes 1/(44100 * (200/100)) = 0.0000113786848 indicating the speed was doubled during digitization, (2 * 100 / 100).  the duration (a value measured in frames) multiplied against the editunit should provide the amount of time in seconds; yet it is the seconds that we have captured in the spreadsheet. duration * editunit = seconds duration = seconds / editunit since the editunit is defined as:   1 / samplerate * speedcorrection, the previous statement is equivalent to: duration =  seconds * samplerate * speedcorrection therefore we obtain the duration value in frames by multiplying the number of seconds elapsed against the sample rate and speed correction, to provide the duration value in frames.  should the start time of the clip be other than zero, then the value for starttime is generated the same way. sub calcframes{ my ($duration, $samplerate, $speedcorrection) = @_; if ($duration =~ /.*?\:.*?\:.*?:/){ # if hours:min:sec:subsec ($hours, $min, $sec, $subsec) = split(":",$duration); $totalsec = ($hours * 3600) + ($min * 60) + $sec + ($subsec / 60); } elsif ($duration =~ /.*?\:.*?\:/){ # if hours:min:sec ($hours, $min, $sec) = split(":",$duration); $totalsec = ($hours * 3600) + ($min * 60) + $sec; } elsif ( $duration =~ /\:/){ # if min:sec ($min, $sec) = split(":",$duration); $totalsec = ($min * 60) + $sec; } else{ $totalsec = $duration;} # seconds only if (! $speedcorrection){ $speedcorrection = 1;} $frames = $totalsec * ( $samplerate * $speedcorrection); return $frames; } as described before, each audio object consists of one or more “faces,” each of which contains one to many “regions,” each of which contains one to many “streams” (this software supports up to four streams per region). examples of a “face” include a single side of a long-playing record; both tape tracks of a half-inch open reel tape containing stereo content; and a single portion of a quarter track four-track tape for which each flip of the tape contains a single stream that is monophonic.  additionally, each variation in the direction or speed correction value would require a separate “face” section.  for example, a reel-to-reel tape on which multiple entries were captured at different speeds (or in different directions) would require a separate “face” entry in the aes57 for each segment. directions supported include front, back, forward, reverse, a_pass, b_pass, c_pass, d_pass and none.  front and back may be used for two-sided objects, such as a long-playing record.  forward and reverse are most appropriate when audio is encoded along the length of the object, such as cassette or reel-to-reel tapes.  the a_pass, b_pass, etc. are more descriptive of the quarter track tape described above.  none would be used for born digital audio, which has no physical structure.  changes in direction, speed correction, or file name pattern (or end of file) indicate an end to a “face,” in this software.  thus, when the tab-delimited spreadsheet is first read in, preliminary parsing of the lines documents these changes, and notes “endface” and “startface” times for each audio file. open (log, $log) or die "can't read $log\n"; while ($line = <log>){ chomp $line; # remove linux newlines $line =~ s,\r,,g; # no new lines of any kind (this gets windows carriage returns) if ($line =~ /^ *([a-z]{1}[\d]{4}\_[\d]{7}[^\t]*) *\t(.*)/){ # identifier should be in first field $id = $1; $mystuff{$id} = $2; # collect everything on the line for this identifier $id =~ s, ,,g; #remove spaces # split up what we just collected, so we can parse it a bit ($title, $phystype, $clipbegin, $clipend, $regioncond, $stream1, $stream2, $stream3, $stream4, $priorstream, $endingstream, $direction, $label, $speedcorrection, @trash) = split("\t",$mystuff{$id}); # trash is any other fields in the log for other purposes # here we set the values for the first entry in the spreadsheet, to be compared against following entries if (! $lastspeed){ $lastspeed = $speedcorrection;} if (! $lastdirection){ $lastdirection = $direction;} if ($first){ # if this is the first line of the spreadsheet, $lastspeed = $speedcorrection; $lastdirection = $direction; undef $first; $face = 1; $startface{$itemid}{$face} = 0; # we always start the audio file at zero } elsif ($id !~ /$lastid/){ # new item, new face, new audio file $lastspeed = $speedcorrection; $lastdirection = $direction; $face = 1; $startface{$itemid}{$face} = 0; } elsif (($lastspeed != $speedcorrection) || ($lastdirection ne $direction)){ # new face! change clipend of last face to stop before clipbegin of this one $endface{$itemid}{$face} = $clipbegin; $face ++; # we're on the next face now, because the speed changed $startface{$itemid}{$face} = $clipbegin; # this new face starts here } $myface{$id} = $face; # use this to find out what face something is on $lastface{$itemid} = $face; # this keeps getting overwritten, so the last face encountered is listed here for this item $lastspeed = $speedcorrection; $lastdirection = $direction; $lastid = $itemid; $lastkid = $id; } the “startface” time is expected to be zero for all audio files, however, the clips may not begin at zero.  for these, preliminary regions are generated with the label “start of file” and any notes in the “priorstream” field of the spreadsheet for the first entry for a new file. if ($clipbegin && $clipbegin =~ /[1-9]+/){ # the first clip does not start at zero $priorid = $id.".begin"; $ownerref .= $priorid." "; $startframe = &doface(1, 0, $samplerate, $speedcorrection, $direction, $id, $title, $endframe, $endframe); $priorframes = &calcframes($clipbegin, $samplerate, $speedcorrection); $myend = $priorframes -1; $lastend = &printregion($priorid,$faceid,"start of file",$samplerate,0,$myend,$channels,$priorstream,$speedcorrection,$datec,$iam,); $starttime = $lastend + 1; } below is an example of the beginning of a “face” in an aes57 file, in which a “begin” region has been added prior to the first clip (or region) identified in the spreadsheet for web delivery: <face id="section_1" direction="forward" audioobjectref="u0008_2012038_0000007" label="alumni banquet dr. matthews and student speakers"> <timeline> <starttime editrate="22500">0</starttime> <duration editrate="22500">43312500</duration> </timeline> <region id="u0008_2012038_0000007_0001.begin" formatref="waveform_audio" faceref="section_1" label="start of file"> <timerange> <starttime editrate="22500">0</starttime> <duration editrate="22500">269999</duration> </timerange> <numchannels>2</numchannels> <conditionnote> <note>crowd noise</note> <creationdate>2015-07-31t15:54:21-05:00</creationdate> </conditionnote> <stream id="u0008_2012038_0000007_0001.begina" label="1" faceregionref="section_1"> <channelassignment channelnum="1" leftrightposition="0.0" /> </stream> <stream id="u0008_2012038_0000007_0001.beginb" label="2" faceregionref="section_1"> <channelassignment channelnum="2" leftrightposition="0.0" /> </stream> </region> also, the “duration” value for the entire file is drawn from the fits duration (exiftool entry) and this may differ from the duration as specified within the spreadsheet.  if the difference is more than a second (determined by the sample rate), an error is generated.  otherwise, if the difference is less than a second, the frame rate for the fits duration is substituted for the one provided in the spreadsheet. $endframe = &calcframes($clipend, $samplerate, $speedcorrection); if ($duration){ $endfile = &calcframes($duration, $samplerate, $speedcorrection); if ($endfile > $endframe){ $endframe = $endfile;} # if significantly different, may have left off content elsif ($endfile < $endframe){ $diff = $endframe $endfile; if ($diff > ($samplerate )){ print errors "duration of file ($clipend) differs ($diff) by more than a second from fits duration of $duration\n"; } else{ $endframe = $endfile;} # correct for single second differences } } else{ $endfile = $endframe;} regions within a face correspond to tracks or clips expected to be played separately.  examples would be each song on one side of a long-playing record, each musical piece in a performance, or the recording of each speaker during a segmented presentation.  compilation of all regions should account for the entire “face” of the object.  thus, segments (regions) that are not selected for extraction for web delivery (as indicated in the spreadsheet) need to be accounted for as well.  potential reasons for skipping segments might be:  extended silences; applause; security reasons or intellectual property rights; inadvertent capture of content unrelated to the intended material; or damaged or lost portions of the original.  to account for such segments, our spreadsheet includes fields for notes on prior or ending streams, and the time codes and duration for those are calculated from the time codes for the audio file and the entries for the regions (or segments) to be included in web delivery.  notes indicating the reason for suppression of the segment skipped should only be added in the priorstream field (unless it is the ending stream) on the row of the succeeding region. elsif ($kidnum > 1 && ($priorframes != ($lastend -1)) && ($priorframes != $lastend) ){ # need an intermediary clip here $priorid = $lastkid.".5"; $ownerref .= $priorid." "; # build the list for formatregion $starttime = $lastend + 1; $myend = $priorframes -1; # i should end one frame before the next clip; $lastend = &printregion($priorid,$faceid,"skipped portion",$samplerate,$starttime,$myend,$channels,$priorstream,$speedcorrection,$datec,$iam,); } each region contains one to many streams, each of which represents an individual channel of audio information, and each of which is assigned a different non-negative integer.  this script supports up to 4 streams, labeling them alphabetically in order; each must reference the region and face of which it is a part. face identifiers are generated by numbering faces sequentially, and preceding them with “section_”; singleton regions are identified by file identifier with “_0000” attached, and multiple region files simply depend upon the region identifiers as entered in the spreadsheet. regions added at the beginning of a file use the next identifier concatenated with “.begin”; those at the end use the last identifier concatenated with “.end”; and skipped regions added are identified in fits2aes by the last identifier concatenated with “.5”.   this allows the person digitizing to avoid entering lines in the spreadsheet for each segment of the audio file which is not being described and extracted for web delivery, and still generate valid aes57 files which define all segments, as required. additionally it is possible to assign “pan positions” within the audio sound stage that the stream should occupy during playback, on a left-right axis and/or a front-rear axis, however this optional encoding is not included yet in the software being described;  instead all are assumed to come from center stage (leftrightposition=“0.0”). sub printstreams{ my ($regionid,$id, $faceid,$datec, $channels, $samplerate, $speedcorrection, @streams) = @_; %numtoalpha = ( 1 => "a", 2 => "b", 3 => "c", 4 => "d" ); undef $numerator; undef $denominator; if ($speedcorrection){ $numerator = $speedcorrection * 100; $denominator = 100; } for ($i = 1; $i <= $channels; $i ++){ # must print streams for as many channels as we have. they may or may not have comments undef @mynotes; $alpha = $numtoalpha{$i}; $id = $regionid.$alpha; $label = $i; $mychannel = $i; $cond = $streams[$i-1]; print out " <stream id=\"$id\" label=\"$i\" faceregionref=\"$faceid\">\n"; print out " <channelassignment channelnum=\"$i\" leftrightposition=\"0.0\" />\n"; $cond = $streams[$i-1]; &printnotes("notetimerange",$samplerate, $speedcorrection, $datec, $cond); print out " </stream>\n"; } print out " </region>\n"; } an example of a region generated at the end of a file for a segment (with the endingstream comment of “applause”) following the last region (clip) entered in the spreadsheet would thus look like this: <region id="u0008_2012038_0000007_0007.end" formatref="waveform_audio" faceref="section_1" label="end of file"> <timerange> <starttime editrate="22500">43290001</starttime> <duration editrate="22500">22499</duration> </timerange> <numchannels>2</numchannels> <conditionnote> <note>applause</note> <creationdate>2015-07-31t15:54:21-05:00</creationdate> </conditionnote> <stream id="u0008_2012038_0000007_0007.enda" label="1" faceregionref="section_1"> <channelassignment channelnum="1" leftrightposition="0.0" /> </stream> <stream id="u0008_2012038_0000007_0007.endb" label="2" faceregionref="section_1"> <channelassignment channelnum="2" leftrightposition="0.0" /> </stream> </region> in summary, the aes57 is a fairly complicated schema for audio technical metadata, but it is attainable by generating fits files for each audio file, and creating a tab-delimited export of a spreadsheet containing appropriate information gathered during digitization. by documenting information about each portion of the archival audio file, even those segments not provided as web-accessible derivatives, future web delivery can be consistent with current content selections and can be informed by previous decisions regarding access. additionally, the generation of this type of technical metadata adds a layer of quality control to the digitization process, to ensure that the time entered for clips do not actually extend beyond the duration of the entire file, and also to verify that the original audio file is valid and well-formed prior to preservation. example aes57 files, their corresponding fits files, and an example tab-delimited spreadsheet are available with the script from http://www.lib.ua.edu/wiki/digcoll/index.php/fits2aes. bibliography audio engineering society [internet]. [updated 2015]. aes57-2011: aes standard for audio metadata – audio object structures for preservation and restoration.  [cited 2015 july 25]. available from: http://www.aes.org/publications/standards/search.cfm?docid=84 audio engineering society [internet]. [updated 2015]. aes60-2011: aes standard for audio metadata – core audio metadata.  [cited 2015 july 25]. available from: http://www.aes.org/publications/standards/search.cfm?docid=85 audio engineering society [internet]. 2011. aes standard for audio metadata – core audio metadata (preview). [cited 2015 july 25]. available from: http://www.aes.org/tmpfiles/aessc/20150725/aes60-2011-i.pdf dublin core metadata initiative [internet]. [updated 2015]. dublin core metadata element set, version 1.1.  [cited 2015 july 25]. available from: http://dublincore.org/documents/dces/ ebu operating eurovision and euroradio. 2013. tech 3293: ebu core metadata set (ebucore), version 1.4, pp. 29 &77. [cited 2015 july 29]. available from: http://citeseerx.ist.psu.edu/viewdoc/download;?doi=10.1.1.278.4772&rep=rep1&type=pdf harvard institute for quantitative social science [internet]. 2015. file information tool set (fits). [cited 2015 july 25]. available from: http://projects.iq.harvard.edu/fits/home library of congress [internet]. [updated 2011 oct 5].  audiomd and videomd – technical metadata for audio and video.  [cited 2015 july 25]. available from: http://www.loc.gov/standards/amdvmd/audiovideomdschemas.html otto, jane johnson. 2010. a sound strategy for preservation: adapting audio engineering society technical metadata for use in multimedia repositories.  cataloging & classification quarterly (48:5) and rutgers university community repository [internet]. [cited 2015 july 25]. available from: http://dx.doi.org/doi:10.7282/t3ww7g11 w3c [internet]. [updated 1997 sep 15]. date and time formats. [cited 2015 july 29]. available from: http://www.w3.org/tr/note-datetime about the author jody l. deridder (jody@jodyderidder.com) is head of metadata & digital services at the university of alabama libraries, where she manages digitization, metadata, software development, infrastructure, and preservation policies and procedures. previously, she developed digital libraries at the university of tennessee. she has an m.s. in both information science and computer science. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – editorial introduction: a peer network mission editorial committee process and structure code4lib issue 19, 2013-01-15 editorial introduction: a peer network code4lib, and the code4lib journal, considered as a peer network. by andrew darby the code4lib journal might, at first glance, seem to be in a client-server relationship with its readership: we publish an issue, you consume the contents. (this assumes you view the world in computer networking terms; you do, don’t you?) however, even the most inky of journals doesn’t work that way; the published article is a finished product, but also a starting point. people do something with the information. but i like to think that people can do more with our information, which suggests a different networking model. a peer network is decentralized, “a distributed application architecture that partitions tasks or workloads among peers [where peers] are equally privileged participants in the application” [1]. it’s not a perfect analogy, but rather an interesting way to consider the journal and indeed the whole code4lib universe or ecosystem or . . . network. the journal, conference, listserv, irc: each of these nodes are separate but feed into one another. a post on the listserv might transform into a presentation at the conference; a presentation might be elaborated as an article; an article might be referenced to solve a problem posed to the list. the data is disseminated, cited, copied whole, re-purposed or forked. this sort of decentralized and distributed architecture encourages doing. a newcomer to the overall code4lib network quickly discovers that there is no waiting for permission; if something doesn’t exist on a peer network, well, put it there. if it’s good, if it makes sense, if it’s necessary, it will prosper; a distributed network is pretty egalitarian. or is it? is code4lib really an egalitarian network? are we all peers? a recent opt-in survey of gender diversity found that 18% of respondents (and 22% of female respondents) didn’t consider themselves members of the code4lib community [2]. you don’t have to self-identify as a peer to participate in a peer network, but there is room for everyone. even if you are not a super-coder, there are ways to contribute, for example, writing documentation, beta testing, user testing, even editing articles! another feature of peer networks is some form of peer review: from a list moderator who can switch off misbehaving nodes to an up/down vote on an article or project to that holiest of holies, double-blind peer review. with each degree of review we move further away from a pure anarchic internet vision, but we gain something–the ability to zero in on those nodes with a greater probability of value or usefulness. so peer review, in whatever form, can chip away at the “just do it” ethic. casual peer review (thumbs up! thumbs down!) will slowly suppress some ideas and elevate others. double-blind peer review anonymously makes decisions that might seem arbitrary. at the journal, the entire editorial committee reviews proposals, and while there is an assigned and second editor for each article, it is not unusual for other members to read and weigh in on drafts. in this system, a peer reviewer can be a collaborator as easily as a gatekeeper; we work with the author to produce the best article possible, and like a good friend (or peer) we sometimes suggest that an idea is not quite ready, or would make more sense attached to a different network [3]. we have a dozen articles this issue–it’s a big one. topics and technologies range from apis to xslt. find the articles that interest you, leave a comment, ping the author, blog about the ideas, fork the project, heck, why not submit a proposal for an upcoming issue [4]? thanks finally, a big thank you to tom keays and mark pernotto for all their work in getting our wordpress install and plugins up-to-date and running smoothly again. notes [1] yes, this definition is from wikipedia: http://en.wikipedia.org/wiki/peer-to-peer [accessed jan 12, 2013]. [2] the survey was created and analyzed by rosalyn metz, and promoted on the code4lib listserv (so presumably respondents were list subscribers). the summary of results is available here: http://goo.gl/8wtkc. [3] so, is the journal peer reviewed? ed corrado discussed this in a previous editorial: http://journal.code4lib.org/articles/3277 and carol bean outlined our process in a recent article in in the library with a lead pipe: http://www.inthelibrarywiththeleadpipe.org/2012/open-ethos-publishing/. [4] http://journal.code4lib.org/call-for-submissions. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – parsing and matching dates in viaf mission editorial committee process and structure code4lib issue 26, 2014-10-21 parsing and matching dates in viaf the virtual international authority file (oclc online computer library center 2013) http://viaf.org is built from dozens of authority files with tens of millions of names in more than 150 million authority and bibliographic records expressed in multiple languages, scripts and formats. one of the main tasks in viaf is to bring together personal names which may have various dates associated with them, such as birth, death or when they were active. these dates can be quite complicated with ranges, approximations, bce dates, different scripts, and even different calendars. analysis of the nearly 400,000 unique date strings in viaf led us to a parsing technique that relies on only a few basic patterns for them. our goal is to correctly interpret at least 99% of all the dates we find in each of viaf’s authority files and to use the dates to facilitate matches between authority records. python source code for the process described here is available at https://github.com/oclc-developer-network/viaf-dates. by jenny a. toves and thomas b. hickey background viaf merges records from various national and trans-national authority files into clusters. these authority files provide identifiers and often standard names for people, titles, series and concepts. there are many challenges in matching across the files, including recognizing and parsing dates associated with personal names. bibliographic dates can be surprisingly complicated and over the years libraries have come up with a mélange of approaches to capture the aspects of dates that they are interested in.[1][2] dates can appear in a number of different fields, including name headings, separate date fields, and within notes. viaf needs to recognize the presence of dates and convert them into a uniform format that it can use when matching and comparing records about people. how viaf uses dates while contemporary files such as orcid (http://orcid.org/) place great reliance on email addresses and institutional associations, library files have in general eschewed such information and rely heavily on birth and death dates to disambiguate people. viaf tries to match records that describe the same entity in multiple files, and dates associated with people are important pieces of information when matching names across authority files. each viaf cluster that represents a person, maintains a date range (described below in storing and comparing dates).  if viaf can identify that two names are associated in some way (for example the forms are similar or have matching cross references) and share both a birth and death year, this is considered a very strong indication of a match, second only to more specific information such as matching unique identifiers.  while on the face of it matching dates would seem straight forward, there are some subtleties even after the dates have been parsed. most of the dates viaf needs are specifically coded as dates, so recognizing them is not difficult.  however viaf also needs to pick dates out of unstructured text, such as notes in authority records.    for this we use some fairly simple regular expressions [3] to recognize and parse these dates.  appendix 3 has examples of these. here are examples of some of the main variants we see: ranges.  typically used to indicate birth and death 1903-1993 flourished dates (when the person was active). may or may not be a range. 1130 fl. circa dates.  approximate dates ca. 1507-1584 other approximations 4./5. stol. 10..-11.. showing month and day 1942 june 24 1943 ún. 23.-? different scripts ٢٧٣م different calendars 604-672 of course, these can be combined in various ways 14..-1472 14?? –ca 1475 here are some ways we see december 3, 1949 entered: 1949 (december 3) 1949 dec. 3 1949 3 déc. 1949 december 3 1949 (dec. 3) 03.12.1949we find 27,000 ways to say what we interpret as flourished around 1850.  here are the patterns that occur more than a thousand times (the first character is the marc subfield code): f18..-19.. d18?? f18..-18.. d18..-19..  f18..-18..? f17..-18.. f18..-18.. d18..-18.., d18..-18.. d18..-…. fca 18– d17..-18.. f18..-19..? f17..-18..? f18..-19.. f17..-18.. although our previous system was able to share much of the code used to parse dates from various authority files we became aware that we had no way to make global changes in how dates were handled.  one change we wanted to make was how we handled flourished dates, and realized we needed to touch the dozens of routines that handle the details of each authority file. an interesting observation is that while the domain, that is how dates are represented, is almost limitless, the range of actual individual dates representing a single day is fairly small.  given two thousand years of publication there are only about 730,000 different dates of interest (2000 years x 365 days/year).   since most dates we encounter are after 1700, only about 115,000 dates will be encountered with any regularity and the range of the transformation from text to our stored dates is fairly small.  another observation is that while we characterized the domain as ‘almost limitless’, in fact, within date subfields we find 384,000 actual unique date strings in all the viaf authority source authority records, even before normalization (this is larger than the 115,000 individual dates both because of variant representations and because many of these strings represent a date range). a table [4] of all these date strings, ranked by frequency is available on github. identifying dates in authority and bibliographic records viaf is interested in various dates in both authority and bibliographic data.  we tend to divide dates into four categories publication dates personal dates in a field with a specified date format personal dates in free text notes personal dates in a heading the first two of these are fairly straight forward.  while publication dates can be complicated, this data is considered of low relevance for matching names so we simplify it by just using the equivalent of the marc-21 date1 field.[5]  this is a coded field, but even that can have a number of variant forms, such as 198u to indicate an approximate date in the 1980’s.  for the purpose of viaf we avoid publication dates that are indicated as questionable or are date ranges and discard dates that are not all digits, along with a few patterns, such as ‘0000’ and ‘9999’, which do not look like valid dates. when a date is in a field such as the marc-21 046 [6], parsing is also quite straight-forward since the iso 8601 standard on the representation of dates and times [7] is specified. picking up dates from free text notes fields can be a challenge, but prior to 2009, marc-21 authority records did not have a structured way of entering date, month and days associated with people, so this information is often embedded in notes and needs to be found.  with the many languages and scripts in viaf source authority records, identifying the type of date in free text can be difficult, e.g. whether it is a publication date, birth, death, or some other type of date.  again, we simplify the process by looking for key words and then picking up only the most straightforward of the dates. while all these dates are important we have put the most effort into parsing the dates contained in the personal headings themselves.  a typical heading looks like: ‡a  shakespeare, william, ‡d  1564-1616 ‡a  shakespeare ‡b  william ‡f  1564-1616 where the ‡d (marc-21) and ‡f (unimarc) indicate the date subfield with a hyphen indicating a birth-death range.  our approach to parsing these dates is the topic of the rest of the paper. storing and comparing dates each cluster in viaf representing a person has a date range stored as two dates (min and max) consisting of a year, month and day.  associated with the date range is an indication of what they represent.  we have three values: lived – the dates are birth and/or death dates flourished – the dates show when the person was active circa – the dates are approximate each section of the date (year-month-day) is set to 0 if not available.  the dates themselves are kept in the standard western calendar (often called the gregorian calendar), using negative dates for dates before 1 a.d./c.e., turning 1 b.c.e. into -1.  while best practice when using negative numbers for b.c.e. dates is to use astronomical conventions  [8] [1] which use a year 0, we do not, maintaining consistency with the common practice of not having a year 0 in b.c.e dates and reserving 0 to indicate no data. our most basic test for comparing date ranges is to see whether two sets of date ranges are compatible: if the both the min and max years are zero in both dates, there is no conflict for min dates after 1400: one max date cannot be less than the other min date max dates must be within 120 years of the other min date when present if either pair is marked flourished min and max dates must be within 100 years of each other if present if either pair is marked circa min and max dates must be within 10 years of each other if present otherwise both are lived min and max dates must be within 3 years of each other if present if both have months specified, they must match if both have days specified, they must match the 3 year range was arrived at pragmatically (we started at 0 and worked up).  for various reasons dates associated with people tend to vary by a few years, and allowing some variation in them helps bring records together. when comparing dates within a single authority file (e.g. to look for duplicates) the fully parsed dates essentially have to match exactly to not conflict. when only one of the min/max dates is present for comparing we get a single-date match which is much weaker than the earlier mentioned double-date match. mapping dates to patterns our original approach grew from parsing dates in two authority files to eventually three dozen.  although our processing for each file shared code, each had its own dedicated parsing routines.  driven by necessity, we evolved the current system that relies on mapping dates to patterns shared across all authority files that are then used to guide the parsing. here are some of the pieces we need to identify: type of date: flourished, circa, birth/death dates era, e.g. a.d., b.c., b.c.e for gregorian dates reign and era dates for japanese names century indications months alternate calendars, e.g. hijri (arabic) dates wild card characters separating the min and max dates (e.g. birth and death) parsing a date subfield is accomplished by morphing the input data into a pattern that guides parsing. there are four basic steps: normalization of the input data splitting the input data into min (e.g. birth) date and max (e.g. death) date sections if we have a han (japanese) date then resolve it via lookup table and we are done[9] if all parts of the date are century dates then resolve them and we are done resolving any wildcards parsing the min/max date sections as an example,  “1949 december 3-“ is converted to the pattern “nnnn month nn”.  this pattern is then matched to this regular expression: nnnnmonthnn = re.compile('(\d\d\d\d) +([^0-9. ]*)\.? ?(\d\d?)') which is then used to pick out the year, month and day from the input string. appendix 4 shows the 32 recognized patterns, and the code for all the patterns can be found in datefld.py.[10] normalization starts with lowercasing the data and decomposing unicode characters. next some non-latin digits and punctuation characters such as hyphens and question marks are converted to latin equivalents. digits are then converted to “n” and anything that looks like a month name is converted to “month”.  we have built these tables up from examples encountered in dates, augmented by month names and abbreviations for various locales known to the python date libraries.  here are some examples of the transformation: 1999 -> nnnn 1999 january 10 -> nnnn month nn 1947-1999 -> nnnn-nnnn 30 b.c. -> nn bc circa 1920 -> circa nnnn splitting tries to find a single hyphen and use that point to split the date into min/max sections. if no hyphen exists, then look for evidence of this being a death date (ie. “d 1946”). if there is no hyphen and no evidence of a death date then it is assumed to be a birth date. if there are multiple hyphens, then we have a manually constructed table of 75 common patterns[11] that can be split. an example is “nnnn-nnnn-“. the second hyphen is probably a typo so we ignore it and split the date at the first hyphen. here are some examples of the transformation: nnnn-nnnn -> (nnnn, nnnn) died nnnn -> (none, nnnn) nnth century-nnnn -> (nnth century, nnnn) nnnn -> (nnnn, none) resolving wildcards means finding things like “nnn?”, “nnn.”  and “nnnn or n” and modifying the pattern, the input string and the datetype. an example is the string “nnn?” in a birth pattern with a date string of “197?”. the input data is changed to ‘1979’. the pattern is changed to ‘nnnn’ and the date type is set to ‘circa’. ‘circa’ allows ±10 years to match so this birth date will match any other birth date from 1969 to 1989.  there are some patterns that we are unable to parse, usually because they are ambiguous.  appendix 1 has a list of these patterns. post parsing we apply hijri  and bc adjustments.  arabic dates are converted to their latin form. if both min and max dates are bc (or if only the max date is bc), multiply min and max by -1. the sample viaf code in github has a stub for the hijri code.  since it has its own licenses it needs to be obtained separately.[12] during this transformation, the dates undergo some ‘sanity’ checks, the most important being: max >= min date max date – min date <= 110 years month values must be between 01 and 12 inclusive day values must be appropriate for the given month testing when we receive a new authority file to add to viaf, one of our first tasks is to identify the dates in the records and decide how to parse them.  to do this we extract all the dates found in the structured fields, count them and rank them.  these strings can then be put through our existing date parsing routines and then visually compared to the input.  since we have the dates ranked by frequency we concentrate on the dates we will encounter the most, making sure that the interpretation looks correct and that we are handling at least 99% of the dates correctly.  the date patterns tend to be clustered, so this task is not as difficult as it might seem. since we now have a uniform approach to dates across the various authority files that contribute to viaf, we can also consolidate all the dates we have found across all of them.  this lets us spot patterns that, while fairly rare in individual files, across all of viaf are large enough to merit attention.  appendix 2 shows the most common patterns along with an indication of how often they are encountered. limitations we recently added japanese ‘reign’ dates to viaf.  while there are many other forms of dates in the world [13] [14], we are comfortable that we are covering the vast majority of dates we encounter. works cited [1] library of congress. “extended date/time format (edtf) 1.0 submission.” library of congress. january 13, 2012. http://www.loc.gov/standards/datetime/pre-submission.html (accessed 11 11, 2013). [2] dublin core metadata initiative. dcmi date working group. http://dublincore.org/groups/date (accessed 04 24, 2013). [3] wikimedia. “regular expression.” english wikipedia. http://en.wikipedia.org/wiki/regular_expression (accessed 11 11, 2013). [4] https://github.com/oclc-developer-network/viaf-dates/blob/master/datefields.txt [5] network development and marc standards. “marc 21 format for bibliographic data: 008:.” network development and marc standards. 09 01, 2011. http://www.loc.gov/marc/bibliographic/bd008a.html (accessed 04 26, 2013). [6] network development and marc standards office, library of congress. “marc 21 format for authority data: 046: special coded dates.” 04 05, 2011. http://www.loc.gov/marc/changes-rda-046.html (accessed 05 25, 2013). [7] wikimedia. “iso_8601.” wikipedia. 04 24, 2013. http://en.wikipedia.org/wiki/iso_8601 (accessed 04 26, 2013). [8] espenak, fred. “year dating conventions.” nasa eclipse web site. february 25, 2008. http://eclipse.gsfc.nasa.gov/sehelp/dates.html (accessed 02 10, 2014). [9] https://github.com/oclc-developer-network/viaf-dates/blob/master/handates.py [10] https://github.com/oclc-developer-network/viaf-dates/blob/master/datefld.py [11] https://github.com/oclc-developer-network/viaf-dates/blob/master/overrides.py [12] alsadi<alsadi@gmail.com>, muayyad saleh. “hijra/hijra.py.” http://github.com/ojuba-org. 2006-2008. https://github.com/ojuba-org/hijra/blob/master/hijra.py (accessed 08 26, 2014). [13] reingold, edward m., and nachum dershowitz. calendrical calculations. cambridge, uk: cambridge university press, 2001. [14] blinn, peter. “today’s date in over 400 more-or-less obscure foreign languages.” curious notions. 2014. http://www.curiousnotions.com/todays-date.asp (accessed 03 18, 2014). about the authors jenny toves is a software architect in oclc research. in addition to viaf, her current interests include the large scale frbr clustering of bibliographic records, which is central to oclc’s linked data program.  one aspect of this work is the algorithmic creation of authority records by mining the bibliographic data. she has an undergraduate degree in computer information systems and a m.s. in computer science. thomas hickey is chief scientist at oclc where he helped found oclc research.  current interests include metadata creation and editing systems, authority control, parallel systems for bibliographic processing, and information retrieval and display.  in addition to implementing viaf, his group looks into exploring web access to metadata, identification of frbr works and expressions in worldcat, the algorithmic creation of authorities, and the characterization of collections. he has an undergraduate degree in physics and a ph.d. in library and information science. appendix 1 unhandled date patterns with multiple hyphens. these patterns are not handled because the meaning is ambiguous or because we haven’t seen it before. this data is monitored to look for new overrides to add to overrides.py (https://github.com/oclc-developer-network/viaf-dates/blob/master/overrides.py). unhandled ca nn– 2330 unhandled ca nn – – 110 unhandled nn– 69 unhandled nn-..38 unhandled n– 36 unhandled ca nn — 34 unhandled nn-ה/nn-המאה ה 30 unhandled -cannn of -cannn 17 unhandled nn – – 17 unhandled -nnn or -nnn 16 appendix 2 this table summarizes the 40 most common date patterns covering 99.4% of the dates viaf encounters. see https://github.com/oclc-developer-network/viaf-dates/blob/master/appendices.py for code to generate this table. pattern occurences example parsed date date type % of total nnnn 19563491 d1947[1947, ”, ”] lived 94.19% . 372270 d(1947). [0, ”, ”] lived 95.99% nnn 179488 d900-talet [900, ”, ”] lived 96.85% nnnn? 145978 f1950?-…. [1950, ”, ”] circa 97.55% nnth century 126436 d20th century [1900, ”, ”] flourished 98.16% ca. nn. jh. 47126 dca. 20. jh. [1900, ”, ”] flourished 98.39% … 43903 d1977-… [0, ”, ”] lived 98.60% ca. nn./nn. jh. 17848 dca. 20./21. jh. [1900, ”, ”] flourished 98.69% nnth cent. 16349 d17th cent. [1600, ”, ”] flourished 98.76% nn 15008 d19-…. [19, ”, ”] lived 98.84% ca. 11056 dca. gegenwart [0, ”, ”] circa 98.89% nnnn month nn 10839 d1921 october 30[1921, ’10’, ’30’] lived 98.94% ? 10562 d?-…. [0, ”, ”] lived 98.99% nn. jh. 8852 d20. jh. [1900, ”, ”] flourished 99.04% nn./nn. jh. 6758 d20./21. jh. [1900, ”, ”] flourished 99.07% nnth cent 6620 d19th cent [1800, ”, ”] flourished 99.10% n… 6342 d18..-1… [1850, ”, ”] flourished 99.13% ca. n. jh. 5796 dca. 6. jh. [500, ”, ”] flourished 99.16% nne e. 5136 d18e e. [1700, ”, ”] flourished 99.18% nnnn month n 4886 d1956 november 7[1956, ’11’, ’07’] lived 99.21% ca. n. h. nn. jh. 4783 dca. 2. h. 20. jh. [1900, ”, ”] flourished 99.23% ca. nn.jh. 4016 dca. 20.jh. [1900, ”, ”] flourished 99.25% nn. stol. 3964 d19. stol. [1800, ”, ”] flourished 99.27% nth century 3787 dactive 9th century [800, ”, ”] flourished 99.29% nn.nn.nnnn 3437 d09.06.1703[1703, ’06’, ’09’] lived 99.30% n. jh. v. chr. 3213 d3. jh. v. chr. [-300, ”, ”] flourished 99.32% ?. 2836 d(1892-?). [0, ”, ”] lived 99.33% nne eeuw 2722 d18e eeuw [1700, ”, ”] flourished 99.34% n. jh. n. chr. 2563 d5. jh. n. chr. [400, ”, ”] flourished 99.36% ca nn– 2330 fca 18– [0, ”, ”] circa 99.37% ca. n. hälfte nn. jh. 2250 dca. 2. hälfte 17. jh. [1600, ”, ”] flourished 99.38% sec. xvi 2225 dsec. xvi [1500, ”, ”] flourished 99.39% ca. ende nn. jh./anfang nn. jh. 2105 dca. ende 20. jh./anfang 21. jh. [1900, ”, ”] flourished 99.40% nnth/nnth cent. 2041 d17th/18th cent. [1600, ”, ”] flourished 99.41% sec. xvii 1904 dsec. xvii [1600, ”, ”] flourished 99.42% n 1813 d1-1 [0, ”, ”] lived 99.43% nnth/nnth cent 1797 d17th/18th cent [1600, ”, ”] flourished 99.44% ca. n./n. jh. 1729 dca. 5./6. jh. [400, ”, ”] flourished 99.44% nnnn ? 1636 f1577 ?-1650 [1577, ”, ”] circa 99.45% ca. nn./nn.jh. 1415 dca. 20./21.jh. [1900, ”, ”] flourished 99.46% appendix 3 regular expressions used to find birth and death dates in free text note fields in marc authority records. ## dd. mm. yyyy. # ('', 'narozen', '31', '01', '1962') birth3 = re.compile('(^| )(narozen) (\d{1,2}) ?(\d{1,2}) ?(\d{4})') death3 = re.compile('(^| )(zemrel) (\d{1,2}) ?(\d{1,2}) ?(\d{4})') # dec 21, 1903 # ('', 'b.', 'jan 31 1962', 'jan ', '31 ', '1962') birth1a = re.compile('(^| |\()(b\.|n\.) ((\d{3,9} )?([0-9]{1,2},? )?([0-9]{4}))') death1a = re.compile('(^| |\()(m\.|d\.) ((\d{3,9} )?([0-9]{1,2},? )?([0-9]{4}))') birth1b = re.compile('(^| )(born|ne|nee|birth|dob) (([a-z]{3,9} )?([0-9]{1,2},? )?([0-9]{4}))') death1b = re.compile('(^| )(died) (([a-z]{3,9} )?([0-9]{1,2},? )?([0-9]{4}))') # 21 dec, 1903 # ('', 'b.', '31 jan 1962', '31', 'jan', '1962') birth2a = re.compile('(^| |\()(b\.|n\.) (([0-9]{1,2}) (\d{3,9}) ([0-9]{4}))') death2a = re.compile('(^| |\()(m\.|d\.) (([0-9]{1,2}) (\d{3,9}) ([0-9]{4}))') birth2b = re.compile('(^| )(born|ne|nee|birth|dob) (([0-9]{1,2}) ([a-z]{3,9}) ([0-9]{4}))') death2b = re.compile('(^| )(died) (([0-9]{1,2}) ([a-z]{3,9}) ([0-9]{4}))') appendix 4 there are 32 known patterns which map to 10 regular expressions that are used to parse the dates.  the month table (monthlookup in https://github.com/oclc-developer-network/viaf-dates/blob/master/datefld.py) is used to convert month names to 01-12. knowndatepatterns = { 'nnnn month nn' : parsennnnmonthnn, 'nnnn monthnn' : parsennnnmonthnn, 'nnnn month n' : parsennnnmonthnn, 'nnnn monthn' : parsennnnmonthnn, 'nnnn month' : parsennnnmonth, 'nn month nnnn' : parsennmonthnnnn, 'nn month nnn' : parsennmonthnnnn, 'n month nnnn' : parsennmonthnnnn, 'month nn nnnn' : parsemonthnn_nnnn, 'month n nnnn' : parsemonthnn_nnnn, 'nnnn nn month' : parsennnn_nnmonth, 'nnnn n month' : parsennnn_nnmonth, 'nnnn nn nn' : parsennnn_nn_nn, 'nnnn-nn-nn' : parsennnn_nn_nn, 'nnnn/nn/nn' : parsennnn_nn_nn, 'nnnn-nn-nn-' : parsennnn_nn_nn, 'nnnn n n' : parsennnn_nn_nn, 'nnnn/n/n' : parsennnn_nn_nn, 'nnnn/n/nn' : parsennnn_nn_nn, 'nnnn/nn/n' : parsennnn_nn_nn, 'nn.nn.nnnn' : parsenn_nn_nnnn, 'nn-nn-nnnn' : parsenn_nn_nnnn, 'nn/nn/nnnn' : parsenn_nn_nnnn, 'nn.n.nnnn' : parsenn_nn_nnnn, 'n.nn.nnnn' : parsenn_nn_nnnn, 'n.n.nnnn' : parsenn_nn_nnnn, 'nn.nnnn' : parsenn_nnnn, 'nnnnnnnn' : parsennnnnnnn, 'nnnn' : parsennnn, 'nnn' : parsennnn, 'nn' : parsennnn, 'n' : parsennnn, } subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – editorial introduction – issue 6 mission editorial committee process and structure code4lib issue 6, 2009-03-30 editorial introduction – issue 6 the intelligent use of technology in libraries continues to be one of our most crucial challenges. for those of us who became librarians because we loved to explore the book stacks, we are now finding new ways to explore both old and new content in digital form. with issue 6 of the code4lib journal we hope you will find new ways to explore, experiment, and bring to your library users what they want and need. by christine schwartz coordinating editor, issue 6 the intelligent use of technology in libraries continues to be one of our most crucial challenges. we watch the world of book and newspaper publishing struggle to find the right mix of print and digital. and we–as keepers of recorded knowledge–know that where publishing goes, we must follow. the intersection of technology, libraries, and the future is where the code4lib community “lives.”  however, many of us coming out of a traditional library background (myself included) are now having to learn things and develop skill sets we never thought we’d need. it’s not an easy transition.  so, the code4lib journal is a great continuing education tool for librarians who are trying to expand their skills. the informal, collaborative approach of both the journal and community makes it easy to get your toes wet on issues related to information technology and programming. for those of us who became librarians because we loved to explore the book stacks, we are now finding new ways to explore both old and new content in digital form. some of this requires experimentation. in our darker moments–when experiments fail–the library’s future looks bleak. but it is out of these digital experiments that libraries are staking their claim for a future that we hope will be exciting, interesting, and full of potential. growing our technological skills helps librarians to not just passively resign themselves to digital inevitability. these skills enable us to create the future library that our users want and need. we need to become conversant in a technological mindset in order to meet the challenge with confidence and an open, intellectual curiosity. to quote joanna dipasquale, one of the author in this issue: [code4lib] “wants libraries to win and is doing something about it.” [1] so, let’s do it. you know, someday little kids will think that being a librarian is a cool job, because it is. new code4lib journal design we have adelie design to thank for the new look of the journal for issue 6. this company designed the distinctive, new code4lib logo pro bono as a gift to code4lib community. also, we’d like to thank editorial committee member,  jonathan brinley for the new css that matches the logo. notes [1] dipasquale, joanna et al. “conference report: code4lib 2009.” the code4lib journal iss. 6 (march 2009). http://journal.code4lib.org/articles/998 subscribe to comments: for this article | for all articles one response to "editorial introduction – issue 6" please leave a response below: jai haravu, 2009-03-31 thanks for the great issue of code4lib journal. leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – building up a collaborative article database out of open source components mission editorial committee process and structure code4lib issue 12, 2010-12-21 building up a collaborative article database out of open source components members of a swiss, austrian and german network of health care libraries planned to build a collaborative article reference database. since different libraries were cataloging articles on their own, and many national health care journals can not be found in other repositories (free or commercial) the goal was to merge existing collections and to enable participants to catalog articles on their own. as of november, 2010, the database http://bibnet.org contains 45,000 article references from 17 libraries. in this paper we will discuss how the software concept evolved and the problems we encountered during this process. by markus fischer and stefan kandera introduction in a meeting of health care libraries in early 2009 in zurich, switzerland, we formed a working group with the goal of building up a collaborative article database.  within the network of health care libraries we already had two major database collections which could be used as a starting point for the project: the rudolfinerhaus in vienna had a collection of about 23,000 records and the pro senectute library in zurich had a collection of about 20,000 records. further growth of the database should be achieved by offering a cataloging tool to the participating libraries. there is only a small amount of funding for the project: the participating libraries are free to pay a small fee to support the project. from the beginning we strove towards open source components, which gave us the possibility to adapt the solutions to our needs. practical prerequisites the working group defined some minimal requirements the end product had to fulfill: the reference database should be freely available the software should be manageable for and by libraries open source solutions are preferred the data format should be marc21 and the rules for cataloging should be aacr2 the system and the bibliographic data should allow openurl requests tracing articles to institutional level is desirable, but not a necessity the project was built using 5 open source components: vufind as the discovery layer, doctor-doc as knowledge database and linkresolver, koha for cataloging, a set of script utilities to manipulate bibliographic data (“bibnet.org – dedupe utilities”) and drupal as cms.[1] indication of availability in vufind was achieved using the daia driver to query doctor-doc. some of these components turned out to be more suitable for our needs than others. technical prerequisites one of the most astonishing facts we were faced with was that marc21 does not specify separate fields for article level descriptions like volume, issue and page information. while these fields are essential for the most basic functions like building an openurl request or to create bibliographic citations, this data is normally stored in marc21 in the field 773g as freetext: 773 $g 50(2010), no. 3, p. 352-362 the situation we encountered got even even worse, as the marc data delivered to us came from different institutions with different “rules” on how to enter the information in 773g. it became obvious that it is almost impossible to parse 773g reliably on the fly in a production system out of such diverse data. we decided to first optimize the marc structure with additional subfields to hold the citation information, and, in a second step, to parse the data of each institution with an adapted algorithm before importing the marc21 data into vufind. this was achieved by analyzing the different contents we found in 773g and by adapting, step by step, the parsing conditions to these situations. the resulting code is included in a gui based form in the “bibnet.org – dedupe utilities” (see below). we found no official recommendation to solve this structural marc problem, except for a discussion paper from the library of congress[2] dating from 2003. we chose to use the proposed solution 4.1 from this paper, adding three subfields for volume (773v), issue (773l) and first page (773q). while this served the basic needs of our project with regard to openurl and indicating availability, we now see that even this solution has some shortcomings: for exporting a citation to generate bibliographies you also need the last page. we may address this in future improvements. since many open source library systems like koha or vufind are built around marc21, it is inevitable that these systems, as article databases, have the same shortcomings as the marc21 format itself: missing fields like volume, issue and first page in every record based function of the system. cataloging articles does not seem to be the traditional core business of libraries. machine readable accessibility to the data stored in the freetext field 773g turned out to be essential for our project. the solution we currently run is the following: $g 50(2010), no. 3, p. 352-362 $l 3 $q 352 $v 50 for the reasons explained above, rethinking these structural marc21 problems would be an important step in making bibliographical article data usable independently of its holding data, and finally, suitable for search systems supporting openurl. vufind http://sourceforge.net/projects/vufind/files vufind is a discovery layer for libraries. the system is based on a solr index and sets new benchmarks in discovering library collections. vufind is developed by villanova university. we found that vufind is easily expandable due to a very clean and modular design. we made a simple default installation of vufind and added the following changes: additional fields for volume, issue, startpage in every aspect of the view (overview, details, favorites, export record, cite this…) a location chooser to manually override the indication, by ip address, of availability of articles for an institution changed the daia-driver to send availability requests over openurl and add the ip of the requesting client speed up the availability requests by parallelizing the sequential requests, using curl location chooser the institutions listed in the location chooser, added in our installation below the existing language chooser, can be defined in vufind’s config.ini as a key value pair together with their ips. the default indication of availability is by ip address of the requesting client, but this can be overridden by selecting another institution in the location chooser. the code for the location chooser is placed in web/index.php and simply sets a cookie with the ip of the desired institution to view its availability: // setup locator $iprequest = $_server['remote_addr']; if (isset($_post['myloc'])) { $location = $_post['myloc']; setcookie('location', $location, null, '/'); } else { $location = (isset($_cookie['location'])) ? $_cookie['location'] :$iprequest; } // make sure the location code is valid. reset to default if not: $validlocations = array_keys($configarray['locations']); if (!in_array($location, $validlocations)) { $location = $configarray['site']['location']; } $interface->setlocation($location); there were other small changes involved. to make the location chooser appear as a select button in the gui you need to add it in layout.tpl: {if $showbreadcrumbs} <form method="post" name="locform" action=""> <select name="myloc" onchange="document.locform.submit();"> {foreach from=$alllocs key=loccode item=locname} <option value="{$loccode}"{if $userloc == $loccode} selected{/if}>{translate text=$locname}</option> {/foreach} </select> <noscript><input type="submit" value="{translate text="set"}" /></noscript> </form> {/if} in web/sys/interface.php you need to add function getlocation() { return $this->location; } function setlocation($location) { global $configarray; $this->location = $location; $this->assign('userloc', $location); $this->assign('alllocs', $configarray['locations']); } curl curl is a library to facilitate getting and sending files or requests using a url syntax. since php does not include threading in its core, curl might be used to create a multi-threaded type situation for sending and receiving requests over the web. to speed up the rather slow availability drivers in vufind we installed curl on the server and tweaked the daia driver to send “multi-threaded” openurl requests. the base code for this is from: http://www.phpied.com/simultaneuos-http-requests-in-php-with-curl/ by stoyan stefanov, which we adapted to our needs: private function multirequest($data, $options = array()) { // array of curl handles $curly = array(); // data to be returned $result = array(); // multi handle $mh = curl_multi_init(); // loop through $data and create curl handles // then add them to the multi-handle foreach ($data as $id => $d) { $curly[$id] = curl_init(); $url = (is_array($d) && !empty($d['url'])) ? $d['url'] : $d; curl_setopt($curly[$id], curlopt_url, $url); curl_setopt($curly[$id], curlopt_header, 0); curl_setopt($curly[$id], curlopt_returntransfer, 1); // post if (is_array($d)) { if (!empty($d['post'])) { curl_setopt($curly[$id], curlopt_post, 1); curl_setopt($curly[$id], curlopt_postfields, $d['post']); } } // are there any extra options? if (!empty($options)) { curl_setopt_array($curly[$id], $options); } curl_multi_add_handle($mh, $curly[$id]); } // execute the handles $running = null; do { curl_multi_exec($mh, $running); usleep(25000); // important to reduce unnecessary cpu-load! } while($running > 0); // get content and remove handles foreach($curly as $id => $c) { $xmlstring = $result[$id] = curl_multi_getcontent($c); curl_multi_remove_handle($mh, $c); } // all done curl_multi_close($mh); return $result; } daia http://www.gbv.de/wikis/cls/document_availability_information_api_%28daia%29 daia (document availability information api) is a data model to encode availability information for documents. the protocol was developed by the german consortias gbv[3], hebis[4] and the beluga[5] project. daia is a simple and lightweight model to query and return the availability of a given document. it works on the assumption that you have a unique identifier / id associated with the document. articles can hardly be traced for availability using a unique record id if they are not mapped to holdings. identifiers like dois or pmids are often not available for the kind of literature we are indexing. additionally we do not come from a consortial situation with one shared ils. typically there is no one-to-one relationship where a record has a unique id for one article related to holdings information. therefore, we were forced to obtain availability information by using the bibliographic information of issn, year, volume and issue, from the marc 773g field. to make daia work for our purposes, we send openurl requests to our daia server[6] and get back daia-xml answers. an institution’s ip address can also be appended as a parameter to the openurl request, to get the availability of a record for that institution. this allows vufind to distinguish between availability by ip address on the records overview and the general availability for all institutions in the details of a record. doctor-doc http://sourceforge.net/projects/doctor-doc/ doctor-doc is primarily a tracking tool for ill. it may also be used as a link resolver for online journals in connection with the german ezb[7]. journal print holdings can be uploaded[8] to doctor-doc, to be indicated in the link resolver and searched down to issue level through a daia interface[9] sending openurl requests to doctor-doc. there is one big challenge when trying to establish availability information by using the issn as your main identifier: a journal often does not have one, but several issns (like e-issn, p-issn and an issn for the cd-rom edition). often a journal gets a new issn when there is a small change in the title or when a journal changes its publisher. issn.org recently introduced the l-issn (linking issn) that should be used in openurl situations to create a more reliable linking situation. but many libraries and data providers still do not use l-issn. doctor-doc takes care of this messy issn situation by internally mapping an issn to any related issn. so a client may do an openurl request with the most recent version of an issn and still finds all holdings that use the old version of the issn. it turns out that for our situation, after having resolved the issue with multiple issn numbers, the issn is a reliable identifier. the issn data to achieve this comes primarily from the issn-to-issn-l[10] table of all issns assigned by issn.org, which has been freely available for download on their website. koha http://koha-community.org/download-koha/ since vufind is not a cataloging system but a highly optimized search interface using an index instead of a database, we had to find another system able to produce marc21 data for those libraries without a capable ils. we didn’t find any small system that would be appropriate for our needs, so we chose koha. koha is an integrated library system (ils or ilms). we are using the cataloging editor and the marc21 framework for about 400 journal templates and the authority file system. we run version 3.0.6. while koha is a very large and powerful system, it is not easy to customize due to a rather complex design. we had to fix various java-script bugs in connection with ie on the cataloging interface, which failed to open new windows, due to blank space in the wrong place in the js function. we needed to ensure that the cataloging tool works with browsers down to ie 7. there is an ongoing bug in the marc 008 field in koha which scrambles the content of the field. we have been forced to correct the data before importing into vufind, by using a custom de-dupe and data control tool (see “bibnet.org – dedupe utilities” below). a nice feature in koha is that cataloging templates can be generated for different materials. we created marc21 templates for about 400 journals with journal title and issn, mandatory, and repeatable or default values already defined. based on our experience with both commercial and open source ils, we realized that there is a high potential for optimizing cataloging articles. catalog records for articles contains a lot of repeated information which should be system generated rather than user entered: koha specific the leader and the system-id is system generated but needs an additional click each. the system-id should not be editable once generated, to avoid loss of record identity. copying an existing record should produce a different system-id. the code for the cataloging agency should be prefilled and not be editable. general the code for the publication place in 008 should be automatically prefilled for the selected journal. the language code in 008 should be automatically generated for the selected journal. publication date in 008 could be generated with the actual year. the volume in 773v could be calculated upon the frequency of the publication. 773g could be composed automatically from 260c, 773v, 773l and 773q. during the cataloging session the last entered issue information should be maintained for the next article to avoid reentering of 260c, 773v, 773l and 008. we are currently evaluating if we should develop a custom application for the specialized task of cataloging articles. bibnet.org – dedupe utilities http://sourceforge.net/projects/bibnet/ a major task in the project is de-duping marc21 data coming from different sources. vufind and koha both provide de-duplication mechanisms. however, we found neither solution was flexible enough for our situation. both approaches de-dupe either too much or too little. de-duplication is dependent on the structure of a given record. a record with full citation information may be de-duped more selectively than a record containing only a title and a year. so we developed a custom de-duping application, available as open source in a raw but functional beta stage. basically this de-dupe utility performs data control and de-dupes using different approaches depending on the machine readable information (issn, volume, issue, startpage) present in a record. possible duplicate matches are further checked and identified by using a fuzzy search on the title. the fuzzy search is accomplished by calculating the damerau-levenshtein distance after string normalization. it is essential to normalize the strings being compared before using this algorithm. librarians tend to index records with different punctuation, using different case sensitive characters and so on. we normalize the article title to lower case and remove all spaces and non-alpha-numeric characters before calculating the damerau-levenshtein distance. we found that using this approach results in highly reliable de-duplication for our needs. drupal http://drupal.org/ for a cms we chose drupal. drupal serves for coordination purposes, although any other cms could do as well: we create the internet presence of the project. we provide direct links from drupal to the cataloging templates of koha. we list the journals assigned to libraries to be cataloged. an internal ticketing system (drupal modul “support 6.x-1.3”) helps us coordinate different tasks (basically, to correct bugs and report new feature requests). conclusions http://bibnet.org/ the features required by bibnet.org could not be developed in a single architecture without significant investments. as a consequence we chose a modular architecture with different software components. not all components of bibnet.org work together without manual intervention at the moment. in particular, the export, de-dupe and import processes need further automation. we are confident we can achieve this after creating a more customized and automated cataloging situation with less possibility of cataloging errors. the concept of bibnet.org can be adapted to any pool of libraries willing to work together. we try to convey self-administration to any participating library with as little technical interference as possible. each library is invited to work with (or improve) each component. the code (and therefore the system itself) is entirely open: cms (drupal) for allocating libraries and journals, to coordinate cataloging and for documentation purposes ils (koha) for cataloging “bibnet.org – dedupe utilities” to prepare records for import opac (vufind) as discovery layer doctor doc as an issn and holdings knowledge base to denote availability (daia) and as a link resolver using the services of the german ezb the most challenging part is to change marc21 to be able to expose machine readable article reference data. as a consequence all incoming data from external resources have to be parsed by our “bibnet.org – dedupe and script utilities”. we think the gain, especially in regard to openurl functionality, is worth the effort. koha turned out to be less suitable for our needs because of incompatible behavior with ie (due to inappropriate use of java-script in the cataloging interface) and because of some other bugs like the 008 bug we mentioned earlier. in general we found that for cataloging articles the interface of koha is not optimized to the point it could be. as powerful as koha may be as a general full featured integrated library system, it seems less apt for our specialized needs. bibnet.org shows the potential libraries can have if they start to cooperate: the software components available for the library world today do allow building aggregate systems with a reasonable effort. we can recommend in particular vufind, which is easy to install and to adapt. vufind provides a powerful discovery system for a multi-institutional environment. future plans cataloging develop a streamlined and optimized cataloging solution for bibnet.org to produce adapted marc21 data. this could be achieved by expanding the existing code base of “bibnet.org – dedupe utilities”. while building a general ils is a huge and complex task, we believe that creating a specialized cataloging system for article references is rather easy to achieve. oai vufind recently integrated a functional oai harvester and import function. we plan to integrate oai connectors to import external data and to further expand the existing database. cairn.info[11], heclinet[12] and ccmed[13] could be valuable targets and partners for bibnet.org. alerts many publishers of the journals we index do not provide alert functionality for their publications. we are aiming to create such functionality within vufind or within the eventually expanded code base of “bibnet.org – dedupe utilities”. vufind recently added the possibility of tracking the index date of a record by providing a persistent entry in a database. this is a precondition to develop alert functionality for vufind, since the solr index is not an optimal place to permanently store this kind of information. notes [1]we leave out the evaluation process here, but other open source software solutions were tested as well. [2]http://www.loc.gov/marc/marbi/2003/2003-dp01.html [3]gbv -gemeinsamer bibliotheksverbund: http://www.gbv.de [4]hebis -hessisches bibliotheksund informationssystem: http://www.hebis.de/ [5]beluga: http://beluga.sub.uni-hamburg.de/ [6]daia server address of doctor-doc: http://www.doctor-doc.com/version1.0/daia.do [7]the “elektronische zeitschriftenbibliothek regensburg” is a freely accessible knowledge database and a-z list for scientific online journals. http://rzblx1.uni-regensburg.de/ezeit/ [8]information about the upload process for print holdings to doctor-doc: http://sourceforge.net/apps/mediawiki/doctor-doc/index.php?title=help:contents#upload_print_holdings [9]description of the daia interface in doctor-doc: http://sourceforge.net/apps/mediawiki/doctor-doc/index.php?title=help:contents#print_holdings_availability [10]http://www.issn.org/2-24117-download-the-issn-issn-l-table.php [11]cairn.info databse of human and social science journals: http://www.cairn.info/ [12]heclinet health care literature information network (stopped in 2000): http://www.dimdi.de/static/de/db/dbinfo/hn69.htm [13]ccmed: http://www.zbmed.de/ccmed.html about the authors markus fischer is head of hospital libraries “solothurner spitäler ag”. he is the main developer of doctor-doc.com and swissconsortium.ch (http://www.so-h.ch). he can be reached at: markus.fischer@spital.so.ch stefan kandera is librarian at “bibliothek und dokumentation pro senectute schweiz” (http://bibnet.org/?q=taxonomy/term/2). he graduated in philosophy at the university of constance (germany) and has a master of advanced studies in information science (htw chur, switzerland). he can be reached at: stefan.kandera@pro-senectute.ch subscribe to comments: for this article | for all articles one response to "building up a collaborative article database out of open source components" please leave a response below: bibnet.org: ???? ?????? ????? ??? ???? « the blog of meorero (oren maurer) – f.o.s.s and libraries, 2011-03-28 […] database out of open source components", code4lib journal (c4lj), issue 12, 2010-12-21, http://journal.code4lib.org/articles/4438. ????? ????? ???: ??? ?????? ?????? ?????? ???????, […] leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – editorial mission editorial committee process and structure code4lib issue 60, 2025-04-14 editorial welcome to the 60th issue of code4lib journal. we hope that you enjoy the assortment of articles we have assembled for this issue. by mark swenson in the week before finalizing the content for this sixtieth issue of the code4lib journal, i attended the computers in libraries conference in arlington, virginia.  in the midst of dozens of presentations on genai and what it means for today’s libraries one session touched on issues core to this publication and its aims: empowering libraries through code: future ready digital leadership.  the first presentation, from austin stroud at indiana university indianapolis, articulated the gulf between the widely held perception that learning a high-level programming language would improve career possibilities for mlis students and the reality that, historically, ala accredited programs do not provide this training.  following a second presentation, from scott hargrove of fraser valley regional library in british columbia on how their new strategic plan focuses on engaging the staff and public on programming and ai literacy, the question was posed to those attending the session: why do librarians and library staff need programming skills anyway? if you have found your way to the code4lib journal (dedicated to sharing coding solutions in libraries) you may already have many ideas of the ways that programming knowledge can be useful for library staff.  if not, this issue has eight articles that demonstrate how competencies in software development are intimately intertwined with modern library operations: from corinne chatnik and james gaskell at union college we have a description on how writing a program in python was able to improve the ability of undergraduate students to accurately enter data. karen coyle describes the openwemi specification of dublin core explaining how it can expand the use of frbr ideas into new non-library environments. jennifer d’souza (tib leibniz centre for science and technology) demonstrates using knowledge graphs to analyze large language models. halie kerns (binghamton university) and leah fitzgerald (amsterdam free library) describe how they made a video game to teach information literacy skills. aerith y. netzer (northwestern university) demonstrates a way to use large language models to turn plain-text citations to bibtex. wilhelmina randtke (georgia southern university) relates the journey they took to simplify and restructure a library system database that had become overcomplicated following a system merger and a migration. andrew weymouth (university of idaho) details the use of python and google apps script to text mine and tag the university’s oral history collections. olivia wikle (iowa state university) and evan peter williamson (university of idaho library) show their success in using the collectionbuilder framework and the static site generator jekyll for creating easy to maintain digital collections websites. these articles showcase a variety of perspectives and challenges tackled through modern computational methods.  the authors are united by their commitment to enhancing workflows and services, their eagerness to engage in innovative collaborations, and their dedication to acquiring new skills.  additionally, these contributions reflect a willingness to share knowledge and support the ongoing understanding of the skills needed in a contemporary library setting.  certainly not every library staffer needs to know how to write computer programs, but having someone on your library staff who understands computer programming and can apply that knowledge to library problems is something any library can use. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – case study: using perl and cgi scripts to automate a quality control workflow for scanned congressional documents mission editorial committee process and structure code4lib issue 17, 2012-06-01 case study: using perl and cgi scripts to automate a quality control workflow for scanned congressional documents the law library digitization project of the rutgers university school of law in camden, new jersey, developed a series of scripts in perl and cgi that take advantage of the open-source module perlmagick to automatically review the image quality of scanned government documents. by implementing these procedures, rutgers was able to save staff working hours for document quality control by an estimated 25% percent from the previous manual-only workflow. these scripts can be adapted by novice perl and cgi programmers to review and manipulate large numbers of text and image files using commands available in perlmagick and imagemagick. by doreva belfiore project background in 1997, the law library of the rutgers university camden campus began a project to provide open web access to digital legal materials from the state of new jersey, other u.s. states, and the united states federal government. the camden library has been registered as a depository library of the government printing office for many years. through an agreement with the gpo, the library is allowed to scan congressional documents to a set of mutually agreed upon standards (600 dpi minimum resolution, uncompressed tiff image format, and full-text searching, among others) for digitization and presentation via the rutgers camden digital law library website. the project began digitizing congressional hearings and committee prints dated from the mid 1970’s through 1999, and expects to continue and expand its scope in the coming years. the author was involved in the initial phase of the project described in this article, and her code is enclosed in the appendix. after accepting a position at another university, she is no longer working directly with the project. the project code is now being managed by john joergensen, reference librarian at rutgers camden law library. legacy processing for digitization, each printed congressional document is cut at the binding with a guillotine-blade cutter. every document is labeled with its corresponding library of congress control number. if no lccn exists, an internal 8-digit document identifier is assigned. multi-volume sets are divided into sections and labeled with either one or two trailing letters of the alphabet, i.e. 12345678a through 12345678az. panasonic kv-s2026c sheet-feed duplex scanners are used to scan individual pages and generate tiff image files of each document page. work-study students of the law school are charged with scanning bundles of cut documents and saving the resulting tiff images to a central network server. prior to this project, one full-time staff member from technical services and one part-time graduate student intern from a local library school were tasked with manually performing quality control on every scanned congressional document. the former quality control workflow included: reviewing thumbnail images for every scanned page in the document, confirming that the total number of scanned images corresponded with the number of pages in the print original, manually re-scanning any mis-scanned pages, missing pages, or oversize fold-out maps and supplementary materials, manually copying and pasting a marc record from the library catalog into a text file for use during post-processing. after checking, completed congressional documents are prepared for web presentation by a series of automated perl scripts which were run by the digital librarian: each scanned page is run through an ocr program to create a text file of its contents, tiff images are converted to pdf files for an alternative web format. a single multi-page pdf is also created containing the entire document, an rdf file is created for oai-harvesting and use on the semantic web, each document is uploaded to a web server directory and indexed using swish-e open-source indexing software. once processed, documents can be easily searched via the congressional documents page of the library website. problems the panasonic rtiv software can be adjusted to account for deskewing of the image, auto-cropping document edges, adjusting brightness levels and other minor corrections. however, despite regular cleaning and maintenance of the scanning equipment, problems occur which sometimes cause documents to fail a quality control check. dirt and wear on the scanner rollers can cause both horizontal and vertical lines on the scanned images. wear on the roller can impede the paper flow through the scanner and cause blank pages with a large horizontal drag mark on the top of the pages. failure to remove security tape (“tattle-tape”) or to cut off all of the binding glue during the guillotining process can cause pages to stick together, altering the final document page count. the presence of fold-out maps and supplementary materials can also skew the final page count. figure 1. example of a scanning error – dirt and wear on scanner rollers causing black horizontal lines we estimate that approximately 12 hours of combined staff time per week was dedicated to manual quality control procedures. even at that level of staffing, the number of scanned documents quickly exceeded the ability of the staff members to perform quality control, and the bottleneck created a large backlog of documents. this work grew out of a need to address such bottlenecks by developing procedures to automate as much of the quality control workflow as possible. solution imagemagick is a free image manipulation program that can be run on various platforms (windows, macintosh, unix). perlmagick is the corresponding perl module for scripting. the digital library project has previously used perlmagick for automated image brightening (cf. belfiore, 2011) and we hypothesized that its power for image evaluation could be harnessed for purposes of quality control. we developed a series of perl and cgi scripts as follows: first, we needed to ensure that documents were in a standard file and folder format for automatic processing. occasionally, students misnamed files or folders during the scanning process. in addition, there have been cases where the rtiv scanning software settings have been inadvertently changed and documents were scanned as pdf’s instead of tiff images. we created a perl script that uses regular expressions to detect any misnamed documents or folders or to detect the presence of pdf files and move the document folders to a “problem” directory for manual remediation. see appendix script 1 for the entire perl script. once the pre-check runs, we run a second perl script, which performs various functions. it first scans the folder for multivolume documents and identifies them by their naming convention of lc control number plus one or two alphabetic characters. it moves any multivolume folder to a secondary staging folder for review. because the marc record for a multi-volume set records the number of volumes and the total number of pages, it cannot be used to estimate the total number of pages per volume. a staff member (or student) must still gather the pagination information, but we developed an easier way to accomplish the task. the person tasked with this activity can launch a cgi script from any web browser that will prompt him or her to enter the first and last pages of the document, along with the number of prefatory pages. this data is stored in a very small temporary text file that is read in lieu of the marc record and used for pagination calculations. see appendix – script 2 for the entire cgi script. next, gdocpagecheckonly.pl checks if the congressional document folder already contains a valid marc record in a catalog file in the form of “.cat”. if not, it automatically gathers a marc record for the document from the library catalog via a wget command. for each document, if the document is not a numbered volume of a multivolume set, it then opens the catalog file and parses the 300 physical description field of the marc record to identify the correct number of pages in the document, and evaluates the document against the official page count. for multivolume sets, it reads the small test file created by the multivp.cgi script. in evaluating the number of pages, the script takes into account whether the number of pages of prefatory material is even or odd, in order to account for the extra blank sheet after an odd number of pages. likewise, it takes into account whether the number of total pages is even or odd, in order to account for a final blank sheet after an odd number of body pages. if, after the calculations, the total number of pages counted in the document folder matches the number on the marc record exactly, the document folder is moved to a second staging folder for the final quality control check. if the document folder contains more image files than the marc record states, the document is moved to a secondary staging folder for further checking. if the document folder contains fewer image files than is expected by the marc record, the document folder is moved into the “problem” directory for manual checking by a staff member. see appendix script 3 for the entire godcpagecheckonly.pl script. because the documents sometimes contain extra blank pages at the end of the content, the quality control process must account for folders that contain trailing blanks in the pagination calculations. by segregating documents with more pages than specified in the marc record, we can apply a third perl script to determine whether extra pages are acceptable blanks or mis-scanned pages. the script, gdocblankcheck.pl, first uses imagemagick to evaluate a small cross-section “slice” of a known reference image of a blank sheet and measures the total brightness of the file. the script then checks each document folder and repeats the same pagination check as in the previous script. for documents with an odd number of prefatory pages, the next even number page is checked via imagemagick for the presence of a blank. when a similar “slice” of the file is checked, if the brightness level of the suspect scanned page matches that of the known reference blank within a small tolerance range, that page is marked as a good “known blank” and entered into a hash for future reference. the same procedure is conducted for a suspect even page after an odd number of total document pages. but how does one account for extra blank pages that may occur at the end of a scanned document? the script has already calculated the total of number of pages that should be present and accounts for one possible blank file at the end of an odd-numbered final page. for an amount of n extra pages, the script loops through an imagemagick check routine n times to see if those pages are indeed blank. every time, it sets a variable flag to mark each pass. when the flag reaches the final number, if all of the pages are evaluated as “good blanks”, the script will allow that document to pass to the next phase of quality control checking. if, however, one of the n extra pages has a brightness value that is too dark and exceeds that of the reference sample (as in a page containing text), it is considered to be non-blank and fails the test. the folder will then be moved to the “problem” directory for manual evaluation by a staff member. in the final step of our automated quality control procedure, any documents that have passed the previous checks are located in the final staging folder, which for purposes of clarity is named imagecheck. the final script, gdocimagecheck.pl, begins by using imagemagick to take the reference values of multiple types of sample documents: blank white pages, very clean scanned text pages, and pages containing dark black lines. figure 2. example of a clean scanned text file used as a reference sample for the gdocimagecheck.pl script it continues with the exact same page counting steps as its predecessor. this step is redundant, but serves an important purpose. in this script, any blank pages found after an odd-numbered prefatory page, an odd-numbered final body page, or at the very end of the document are labeled as “good” or “known” blanks. these are stored in a perl hash table %goodblanks for later lookup. the script then cycles through each document and evaluates each scanned page against the “clean” reference sample of a page of text. if the brightness level falls within a small range of tolerance, it is deemed to be good. if the level is too light, the page may be blank. the page number is then checked against the values stored in the %goodblanks hash table. if the blank page is known to be appropriate, the script allows it to pass and continues. if, however, the blank page is not known, it is marked as a “suspect” file. if the brightness level is too dark, the file is also labeled as “suspect”. after the script cycles through all the files in a given document, if there are no “suspect” files, the document is cleared to be moved to a final “good” staging directory for further processing for indexing and web presentation. if there are “suspect” files, the document is moved to the “suspect” staging directory where the document can be manually evaluated by a staff member. see appendix – script 5 for the entire gdocimagecheck.pl perl script. complications and lessons learned in the course of this project, we experienced some issues which necessitated our breaking this script into modular parts. we initially began with one master script which performed all the actions from downloading the marc records to checking individual images. because the library fileserver, maintained by library it services, was running an older version of the red hat linux operating system, we were not able to upgrade to the latest stable version of the imagemagick software. we experienced high server cpu utilization rates when running any scripts containing imagemagick, even during non-business hours, and had to stop the use of imagemagick on that server. our solution was to break the script into parts and utilize workstations to run some of the scripts. we run the first two scripts, gdocprecheck.pl and gdocpagecheck.pl, on our production unix server. we then use an ubuntu linux box that connects to our server using an sshfs link to run the two scripts requiring imagemagick, gdocblankcheck.pl and gdocimagecheck.pl. in phase ii of the project, described later in this article, planned server hardware and operating systems upgrades allowed us to eliminate redundancies, re-integrate the scripts and run them from one production server. one benefit we found to modular script development is flexibility. our project estimates that it takes the script about 3 seconds to check each file. for large documents, it may take over 15-20 minutes per document depending upon server memory and processor speed. because these scripts are written in a modular fashion, each one can be run independently and multiple scripts could be run at once. this is helpful for a busy digitization project where multiple documents are being scanned and checked on a given day, so that the people scanning the documents are not impacted by the script processing. our scripts rely on correct aacr2 cataloging and isbd punctuation in order to parse downloaded marc records and to gather pagination information. it is important to note that for approximately 10% of documents, the gdocpagecheckonly.pl script cannot accurately parse the marc record which will cause the document to fail the page check and be moved to the “problem” directory. there are various potential causes for this: the marc record is inappropriately coded or is missing information. the marc record is missing isbd punctuation. the marc record has been coded for a multivolume set but for some reason the document was not labeled as a multivolume part. the marc record is in rda format. the marc record was not properly downloaded so the catalog file is missing or incomplete in some way. the marc download portion of the script could not match the document to an lccn number and therefore no catalog file was created. in any of these cases, the documents are reviewed by a staff member and the marc records can be re-downloaded or the catalog files re-created manually. in the future, script enhancements can be made that would make multiple passes at reading the 300 line of the marc file to correct for missing information. the shift from aacr2 to rda format for cataloging will necessitate future revisions. our goal, as always, is to limit the time required for staff members to manually review the congressional documents. occasionally, a page with very little text will read as a blank file and will be labeled as “suspect”. further testing and manipulation of the base gdocimagecheck.pl script could refine the size of the sampled “slice” of the page and adjust the brightness tolerance so that partial pages are labeled as “good”. project phase ii in the intervening months since the author took a position at another university, the library obtained a server hardware and os upgrade, such that the scripts are now able to run directly on a linux server. the digital library has created a regular schedule for automated script processing and document triage. john joergensen, of the rutgers camden law library, has streamlined the scripting of the original processing scripts to eliminate redundancies and improve functionality. for phase ii of this project, the digital library is testing a cgi quality evaluation script similar to amazon’s “mechanical turk” that can be used by non-experts (i.e. work-study students) to mark suspect scanned documents as “good” or “bad” and triage them appropriately. the script converts the original tiff image of a suspect file to a jpg on the fly using imagemagick and presents it to the user in context on a web page. the user can then view a suspect page along with the preceding and following page, and can navigate between pages like an online reader in order to make a decision about the suspect file. according to the viewer’s decision, the scanned document is automatically moved to a “good” directory for further processing for web presentation, or to a “rescan” directory to be sent for rescanning by work-study students. this cgi script can be run from any internet browser and will greatly reduce the time required from library staff members to review suspect documents. future projects and other ideas currently, this set of scripts provides a level of basic processing that identifies potentially problematic documents. the gdocimagecheck.pl script can be extended to handle specific problem cases and triage them appropriately. for example, a finer grain of brightness tolerance or the use of imagemagick’s image-matching function could be employed to specifically distinguish mis-scanned pages with roller drag-marks (see figure 3) and automatically send the document to be rescanned. this is useful for scanned congressional documents because there are frequent instances of blank divider pages found in the middle of a document which are not erroneous, and it would be very useful for a quality control script to be able to distinguish acceptable blanks from scanner errors. figure 3. example of a scanned page with a “roller drag” mark instead of textual content the author plans to continue to work with imagemagick and perlmagick to develop other scripts for digital image quality control that take advantage of the software’s power of image evaluation. conclusions by taking a fully manual process and automating image checking for large series of uniformly formatted text documents, rutgers camden digital law library was able to save approximately 25% of staff time in quality control procedures. the implementation of cgi scripts to review specific “suspect” pages within scanned documents allows the library to easily offload quality control work to non-expert users such as work-study students, which will further save person-hours from technical services and digital library staff. the library’s future goal is to save 50% of total staff time dedicated to quality control for the congressional documents digitization project. while these scripts may not be appropriate for the quality control and revision of archival materials, manuscripts, or photographs, we feel that they are useful for textual documents that are generally uniformly formatted, such as government documents. as the imagemagick and perlmagick software are powerful and highly customizable, their use could be extended to manipulate and evaluate more complex documents and image types than are demonstrated by this project. this software and these scripts are very suitable for libraries and archives that have smaller budgets and limited information technology resources, as they can be edited and developed by staff with a beginning knowledge of perl and cgi programming. references amazon.com [internet]. (updated 2012). amazon mechanical turk: artificial artificial intelligence; [cited 2012 april 8]. available from https://www.mturk.com/mturk/welcome. belfiore, d. 2011. using imagemagick to automatically increase legibility of scanned text documents. code4lib journal [internet]. [cited 2012 april 08]; 14:2011-07-25. available from: http://journal.code4lib.org/articles/5385. ciornii, alexandr. [internet]. (updated 2007?). roman-1.23. cpan; [cited 2012 april 08]. available from: http://search.cpan.org/~chorny/roman/lib/roman.pm. imagemagick studio llc. [internet]. (updated 2011). imagemagick: convert, edit and compose images; [cited 2012 april 08]. available from: http://www.imagemagick.org/script/index.php. imagemagick studio llc. [internet]. (updated 2011). perlmagick api; [cited 2012 april 08]. available from: http://www.imagemagick.org/script/perl-magick.php. joergensen, j.p. 2002. the new jersey courts publishing project of the rutgers–camden law library. law library journal. [cited 2012 april 08];94(4):673-689. pozkanzer, jef. [internet]. (updated 2009 july 12). tiftopnm; [cited 2012 april 08]. available from: http://netpbm.sourceforge.net/doc/tifftopnm.html. thyssen, a. [internet]. (updated 2011 march 15). examples of imagemagick usage; [cited 2012 april 08]. available from: http://www.imagemagick.org/usage/. acknowledgements the author wishes to thank john joergensen of the rutgers university camden school of law for his mentoring and guidance. appendix n.b. – the scripts below are original to the project except where enhancements are noted and credited. in certain cases, information regarding specific server addresses and paths has been omitted for security and should be adjusted for the specific needs of the institution. for more information about current scripts, please contact the rutgers camden digital law library. [back] script 1: normalization pre-check script – gdocprecheck.pl [back] script 2: cgi scripts producing web forms to automate the pagination of multivolume government document sets part a: first cgi script page to confirm document – multivp1.pl part b: second cgi script page to evaluate pagination – multivp2.pl n.b. – this file was edited by john joergensen to reduce redundancy and improve functionality. the author wishes to thank mr. joergensen for his assistance with this script. [back] script 3: initial marc record download and page check – gdocpagecheckonly.pl script 4: imagemagick script # 1 for checking for extra blank pages – gdocblankcheck.pl n.b. – this script is written to run under ubuntu linux with an established sshfs share called ./camlaw . adjust this share name to your institution’s needs. [back] script 5: image checking program for quality control – gdocimagecheck.pl about the author doreva belfiore received her masters of library and information science in 2011 from the drexel university ischool in philadelphia, pa. from 2009 to 2011 she worked as a digital library and circulation intern at the law library of the rutgers university school of law in camden, new jersey. she currently works as a bibliographic assistant in the digital initiatives department of temple university libraries in philadelphia, pennsylvania. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – data for decision making: tracking your library’s needs with trackref mission editorial committee process and structure code4lib issue 33, 2016-07-19 data for decision making: tracking your library’s needs with trackref library services must adapt to changing patron needs. these adaptations should be data-driven. this paper reports on the use of trackref, an open source and free web program for managing reference statistics. by michael carlozzi introduction as a technology librarian i am responsible for implementing technical solutions to enhance library experiences for both patrons and staff. since i also work the reference desk, i noticed an intersection of two seemingly unrelated problems: 1) we do not track informative reference statistics, and 2) i have trouble meeting patrons’ technology needs because of time spent covering reference. as i was soon to find, these problems were connected. in preparation for submitting annual reports, my library would traditionally sample reference statistics by using “clickers” or “hash marks” over a three-week period. the purpose of these samples was to meet state requirements and thus reported only the volume of transactions. as they provided no details on these transactions, several questions lingered: with what specific reference services did patrons need help? how could staff respond best to those needs? how should we staff reference in response to these data? canton public library employs three full-time librarians in the reference department—a reference services librarian, a teen librarian, and me. staffing the reference desk consumes a considerable amount of our time and prevents us from providing specialized services. we average 19 hours/week at the desk per staff member, although the amount varies due to overlap and availability; some weeks, for example, i spend over 30 hours covering reference. in those weeks especially, i am handicapped in my ability to provide technology services and to maintain and improve our library’s technical infrastructure. anecdotally, i felt that most of the reference transactions we processed were directional, capable of being handled by paraprofessionals and volunteers. the directional nature of reference work has been addressed extensively in the literature (aluri and st. claire 1978; summerhill 1994), with many librarians “transforming” reference in response to this and other pressures (meldrem et al. 2005; sonntag and palsson 2007). but without data, all i had were conjectures—and no clarity on how to proceed. in order to improve our traditional sampling methods, i decided to implement reference tracking software (rts). many librarians use rts such as libanswers (flatley and jensen 2012), desk tracker, and gimlet (chan and johns-masten 2014). librarians often record transactions to make organizational decisions: bravender, lyon, and molaro (2011), for instance, used libstats to conclude that “it is not cost effective to use reference librarians to answer [virtual] chat questions” (see also garrison 2010). thus rts seemed to solve my two problems by enabling us to collect informative reference statistics and to make informed organizational decisions based on those statistics. many rts programs come with downsides. some of the issues are: they may be proprietary, limiting customization and/or requiring a monetary commitment. they may be inconvenient; for instance, users may need to select multiple options from a drop down menu. as goodsett (2013) observed, “if your staff has to answer a slough[sic] of questions every time someone comes to the reference desk or sends an email, they may be discouraged enough to just skip recording reference data altogether.” they may be unsupported or unavailable; libstats, a popular solution in the mid-late 2000s (see jordan 2008), cannot be found easily, and its code — hosted on github – has not been updated in over four years. google forms has been used as rts to some acclaim, but it has been criticized for lacking customizability and “reporting functionality” (goodsett 2013). for example, google forms requires the manual entry of dates and times because its timestamp feature does not translate well to analytical platforms like excel, spss, sas, or r. finding the available rts unsatisfactory for the above reasons, i coded in php—and released under gnu’s general license 3.0—a web-based program (trackref) built on an sql database. this program seeks to address the above limitations by providing librarians with rts that is free, open source, convenient, and powerful. in february of 2016 i launched the program in my library and we have been logging transactions ever since. figure 1. trackref’s homepage. i wanted the program to facilitate efficient, comprehensive, and customizable data analysis. with that goal in mind, i programmed trackref to allow administrators the ability to manipulate whole categories of data (renaming, deleting, and creating) and to enable qualitative input. the ability to input qualitative data has proven useful. after data collection began, we formulated additional “research questions”—for example, 7% of all transactions within the first month related to “technology help.” but the kinds of technology help that people were seeking remained unclear. i could have added additional transactions for specific concerns (e.g. transactions of “email” and “browser printing”), but adding new transactions for each concern seemed unsustainable in the longer run. thus i coded trackref to enable a feature (called “detailed mode”) that allowed the entry of textual input for select transactions. now staff can identify the kind of “technology help” they provided when logging. what we learned from tracking reference statistics as suspected, we discovered a low frequency of bona-fide reference transactions; 94% of our transactions were directional. consistent with national trends, few patron requests required a librarian’s research skills. further, most research questions required basic fact retrieval: i.e. identifying local news, retrieving phone numbers, accessing websites. these results suggested that staffing the reference desk with three full-time librarians was not the best use of the librarians’ expertise. in response to these data, we began to implement a more “on-call” model of reference, where paraprofessional and volunteer staff answered most questions and directed difficult queries to librarians who otherwise spent their time on more specialized work. this approach is relatively uncommon; miles (2013) found that only 10% of surveyed academic librarians used an on-call model of reference. public library figures are likely lower than academic libraries’, as public libraries often lack the staffing surplus to allow their librarians “office time.” but our data show that most reference transactions can be handled by paraprofessionals; other researchers, even in academic contexts, have reached similar conclusions (see dinkins and ryan 2010; faix et al. 2010). we are fortunate that a library of our (small) size has three pages and numerous volunteers, who have helped to cover the reference desk and “free up” the librarians to work within their specializations. as an example, consider that patrons were not approaching the reference desk for assistance with ereaders (1.2% of trackref’s transactions), in stark contrast to national claims that patrons needed help in this area (library journal 2014). as the technology librarian, i was particularly interested in this problem since my job is to train patrons in using modern technology yet our patrons were not visiting the reference desk for ereader guidance. but with fewer hours at the reference desk, i have had time to market and offer guided technology instruction: each week i reserve two hours (30 minute appointments) for one-on-one assistance at times when i would ordinarily have worked reference. every timeslot has been filled through july, with many patrons clamoring for additional hours—which i have been adding slowly. of these appointments, 30% concern ereaders. i also aim to offer classes based upon the needs of those attending my one-on-one sessions. excel, for example, which accounts for 24% of booked appointments, suits the classroom environment in a way that ereaders—with so many idiosyncratic devices—do not. i have had more time to build a technology-centered curriculum, something for which many librarians unfortunately lack time, and i am also planning information literacy classes. although a core tenet of librarianship, instruction is too often neglected because of other concerns. even in academia, the predominant model of library instruction is the problematic “one-shot” (fister 2013; helms and whitesell 2013). with more time to build engaging lessons built on data-supported curriculum, i can implement stronger pedagogy. the library has also used trackref’s data to improve staff efficiency. we noticed that several patrons needed help “accessing” our computer network: “access” was the largest category of transactions in trackref at 16%. we discovered more efficient ways to grant computer access. rather than manually creating user sessions through our time management software, taking upwards of two minutes per action, we now batch print 90-minute guest passes. and although this does not solve the “problem” of needing to provide guest access (as we will always offer guest access for privacy-conscious patrons), it optimizes our process. we also created better signage and instructional materials to help patrons with common problems encountered when trying to access library resources, as identified by additional information entered alongside “access” transactions in detailed mode. trackref revealed that almost 25% of patrons needed help with office services (faxing, printing, copying, and scanning). we decided to upgrade our reference department’s infrastructure by purchasing an easy to use, touchscreen library document station that consolidates these functions. when we learned that so many transactions concerned the copy machine, i flagged the “photocopy” transaction to prompt detailed information (see the section “installation and configuration” for details). by flagging these transactions we found that most of the problems were related to: 1) a misleading “key card” error message on the display for which there was no fix; 2) very particular positioning requirements that resulted in photocopies coming out “wrong”; or 3) a coin-bill acceptor that only took coins. the library document station solves these problems by eliminating the confusing error messages, allowing successful copies regardless of where patrons place documents on the scanner, and installing a new currency acceptor that takes bills, as well as coins. figure 2. details on the “photocopy” transaction. finally, 13% of our reference transactions involved staff members booking the study room. in response, i am currently improving our in-house study reservation software by coding an online version which will allow patrons to book their own appointments. this way, reference staff will be responsible only for granting patrons access to the physical room (which we decided not to track separately in trackref). installation and configuration this section presents step-by-step instructions on installing, configuring, and launching trackref. instructions on using the program are covered in the following section, “using trackref: process and analysis.” 1) trackref requires a webserver. unless you already use an online webserver, or you plan to implement the program across multiple networks/subnets, i recommend installing wampserver. wampserver brings the program’s essentials: apache’s webserver (for hosting), mysql (for database storage), and php (for querying the database). mac users can use mamp, though it has not been tested with this software. the latest release of wampserver when i wrote this, 3.0, requires numerous windows visual c++ packages, and users on 64-bit machines must install both the 32-bit and 64-bit packages. this is quite a headache, and some users report that installing packages “out of order” causes the entire installation to fail. because wampserver 2.5 does not require nearly so many packages and has functionally similar to 3.0, i recommend using wampserver 2.5. additional configuration outside of selecting a default browser shouldn’t be required, although password protecting the sql database is recommended (that is beyond this guide’s scope). for further help on configuring wampserver, consult this thread on http://forum.wampserver.com/read.php?2,123606. 2) download trackref’s code from http://www.activelibrarians.com/trackref and extract it to your webserver’s directory: in wamp, the default is c:/wamp/www. as wampserver’s authors note, this directory should not be changed from default. figure 3. correctly extracted trackref folder in windows 7. 3) on the machine hosting trackref, test the installation by navigating in a web browser to http://localhost/trackref. trackref’s homepage should be displayed, not a “not found” error (sql connection errors are to be expected, as trackref has not yet been configured). because trackref is hosted on a webserver, all computers within the same network can access its webpage. for example, my library uses the 192.168.1.0/24 address space for its internal network. the particular computer hosting trackref has an ip address of 192.168.1.76. thus for other staff members to access trackref, they visit http://192.168.1.76/trackref. consult your library’s information technology representative if you are unsure of your library’s network layout. 4) once trackref is online, its settings must be configured. trackref’s folder contains numerous php files, each responsible for handling different program functions. the settings.php file is used to configure trackref. the settings.php satisfies many work environments—those that want quick, easy, and strictly quantitative data. if you want to use trackref in detailed mode, which allows staff to enter descriptive information for select transactions, you will need to replace the settings.php file with settingsdetailed.php. when choosing this option, first delete or rename the original settings.php and rename settingsdetailed.php to settings.php. this now becomes the file that you will edit in order to configure trackref’s settings. which file is for you? all else being equal, i recommend using detailed mode (settingsdetailed.php) unless you absolutely do not want textual input. with detailed mode, the administrator can “flag” transactions when needed (see below), allowing for additional data. librarians interested in purely aggregate data may prefer the simpler version. 5) trackref stores all of its data in an sql database and must have credentials to access that database. edit settings.php with your database’s login information, as you had determined when installing wampserver. by default, the username is “root,” the host is “localhost,” and the password is “” (blank). those using external web hosting can retrieve this information from their web host. web hosting may require changing the database’s name from “reference” to something else, in which case the administrator must change the database’s name in settings.php, settingsdetailed.php, and create.php (all the files that create a database connection). figure 4. database settings in settings.php. in settings.php you create the transactions you wish to track; the program supports up to 14 distinct transactions. with detailed mode enabled, staff members are prompted for additional qualitative (textual) input after recording transactions. this option is configured through an array called $total in the settings.php page (renamed from the settingsdetailed.php file): for each transaction type, “0” will not prompt additional information whereas “1” will. figure 5. the $total array in detailed mode: ‘1’ indicates detailed information requests. here, for example, “catalog” will not request additional information but “faculty” will. other options managed in settings.php are time zone, the ability for staff to delete their transactions, the times to appear on hourly transaction charts, and the requirement to confirm transactions. 6) once settings.php is saved, visiting http://localhost/trackref/create.php will run a script that creates a database called “reference” with all the required tables as well as an “admin” user. in case the script fails, trackref includes the database.txt file outlining the database’s structure. administrators can add these tables by using an interface like phpmyadmin, which comes with wampserver and can be accessed through http://localhost. see phpmyadmin’s documentation for further guidance. 7) with the database created, the program administrator can now log into trackref with the admin user, whose default credentials are “admin/admin”; the admin account has special privileges, enabling the creation of additional users and library locations as well as the ability to structure the database. after logging in with these credentials, the administrator should change the account’s password under the settings tab located in the top navbar; this is not the settings.php page. the admin should also create a library location and associate it with him/herself by clicking “update user settings.” trackref can be configured to add support for “roving reference” by including the following meta element on index.php:(you may need to tweak the content parameters for your devices). this addition makes the homepage mobile-friendly when recording transactions; it is not enabled by default because, absent customization, it may distort the homepage. using trackref: process and analysis on trackref’s homepage, staff members click a word to record its represented transaction; a javascript alert box (which can be disabled) asks for confirmation; when using detailed mode, they are instead asked to provide detailed information for flagged transactions. once confirmed, the transaction and its corollary information enter the database: year, day, time, month, transaction type, staff member, and location. staff can also enter data retroactively under the navbar tab manual entry. with manual entry, trackref stores the last-entered date so users do not need to keep changing it when, say, “catching up” at week’s end. in detailed mode, i recommend standardizing input because the program does not offer controlled vocabulary; for instance, instruction librarians may want to determine the distribution of classes visiting reference. if enough of one class visits the desk, then librarians may contact the relevant faculty and offer course-based instruction (if using this instructional model). data become easier to visualize and code elsewhere (such as in excel) when standardized. a listing of transactions with information like “enl 240,” “bus 101,” and “nur 331” is more manageable than one with “english 240 intermediate lit. class,” “introduction to business studies” and “upper-div nurse class.” the program offers two analytical methods: 1) graphs for quick and aesthetic analysis and 2) csv exportation for deeper analysis. the analytics tab uses fusioncharts to render data. by default, these charts display usage by month, weekday, and hour. basic customization is also allowed, where users can specify certain days of the week (or years), which are then analyzed on the custom.php page; for instance, my library opens from 10:00 am to 9:00 pm three days a week, so when analyzing hourly statistics i might compare only those three days. figure 6. fusioncharts’ rendition of our hours for days open from 10:00 am to 9:00 pm. for further analysis, use the program’s second analytical method: database extraction. under the settings tab, click the “export” button under “export database to csv.” this exports the transactions table to a csv file, where data can be freely manipulated. the graphs provided within trackref are meant to visualize common analytical functions and are not meant to be exhaustive. experienced users should not be discouraged from tweaking the source code to add additional graphs, but consider exporting the database to a csv file instead; analysis is often easier within a statistical package than through programming. issues encountered and further work the largest hurdle has been getting all staff members to log transactions. as other librarians have noted (flatley and jensen 2012), rts only works when used. once logging begins, forgetfulness becomes “declining usage”—everyone enthusiastically logs in september and then neglects to do so in november. thus i have had to qualify all analysis; for example, when comparing transactions by time of day i have to consider that some workers log more diligently than others. security may also be a concern. web-based (http) limitations apply to this program as they would to any other; absent encryption, data (e.g. login credentials) can be sniffed by network users. i do not regard this as a serious enough risk in terms of likelihood or severity to implement encryption in my own library network, but be aware of the risks. concerned librarians should be able to encrypt their networks with the help of computer support services. finally, as a work in progress trackref is far from perfect. its advantage over commercial products is that it is free, easy to use, open source, and customizable. unfortunately, as it is the work of one developer, it has some limitations, listed below: web hosting may break some of the code’s query structuring. although i have not tested this particular program online, other programs i have written failed when hosted online. in one case, i had to change every sql query from single to double quotation marks. in another, my web hosting’s peculiar php configuration required extensive troubleshooting. the programming for “hourly rate” statistics in the analytics tab is lackluster. trackref divides total transactions (for each hour block) by the number of distinct dates in the database, yielding a “per hour” rate statistic. this works poorly with unequal operating hours. for example, because my library closes at 5:30 on friday and saturday, the hourly rates for 5:00 pm to 9:00 pm will be slightly lower than reality. i am working on an improved hourly rate version, but for now that statistic represents an approximation, not the actual value. staff members can only be associated with one location at a time. this can be annoying when working physical and virtual reference simultaneously, so administrators might consider creating a shared account called “virtual” (because the program does not prevent multiple logins for one account, administrators can create the virtual account as they would any other). multiple staff members can then log into the virtual account to record transactions. on the custom.php page between lines 380 to 507 is some code which presents a pie chart of transactions received in a given hourly period. the code is “commented out” to prevent sql errors. librarians may wish to know at what times certain kinds of transactions come to the reference desk and thus staff accordingly. unfortunately this feature is not yet release-ready and requires manipulation of the source code. to change the current time from the default of 8:00 pm, edit the following code on line 390, changing the times (e.g. 7:00 pm would be $newtime > = 700 && $newtime < 759). this code currently does not work for mornings. figure 7. conclusion trackref has allowed our library to make data-driven organizational decisions. i have been able to offer more technological instruction and plan to provide even more in the future. we have also improved overall staff efficiency and have made a large (for our institution’s size) evidence-based purchase for the reference department: the library document station. going forward, i plan to use these data for organizational rather than strictly departmental analysis (e.g. our operating hours). not all of our goals have been met, unfortunately. full-time degreed librarians still primarily cover reference. our reference librarian in particular still spends considerable time staffing the reference desk and has not been able to develop a genealogy center as originally planned. she has, however, been relieved from the desk enough to reorganize our archival/local history room in preparation for launching such a center. nevertheless, data from trackref have helped us to move in a positive and progressive direction that will hopefully continue. references aluri, r., & st. clair, j. 1978. academic reference librarians: an endangered species? journal of academic librarianship: 82-4. bravender, p., lyon, c., & molaro, a. 2011. should chat reference be staffed by librarians? an assessment of chat reference at an academic library using libstats. internet reference services quarterly 16(3): 111-127. chan, t., & johns-masten, k. 2014. a study of gimlet use in reference transactions. internet reference services quarterly 19(2): 73-87. dinkins, d., & ryan, s. 2010. measuring referrals: the use of paraprofessionals at the reference desk. the journal of academic librarianship 36(4): 279-286. faix, a., bates, m., hartman, l., hughes, j., schacher, c., elliot, b., & woods, a. 2010. peer reference redefined: new uses for undergraduate students. reference services review 38(1): 90-107. fister, b. 2013. the library’s role in learning: information literacy revisited. library issues: briefings for faculty and administrators: 33(4). flatley, r., & jensen, r. 2012. implementation and use of the reference analytics module of libanswers. journal of electronic resources librarianship 24(4): 310-315. garrison, j. 2010. making reference service count: collecting and using reference service statistics to make a difference. the reference librarian 51(3): 202-211. goodsett, m. 2013. no more tallies: comparing web-based reference statistics tools. persona website [internet]. [cited 2016 may 4]. available from: https://mandigoodsett.com/2013/10/21/no-more-tallies-comparing-web-based-reference-statistics-tools/ helms, m., & whitesell, m. 2013. transitioning to the embedded librarian model and improving the senior capstone business strategy course. the journal of academic librarianship 39(5): 401-413. jordan, e. 2008. libstats: an open source online tool for collecting and reporting on statistics in an academic library. performance measurements and metrics 9(1): 18-25. library journal. 2014. ebook usage in u.s. public libraries. [cited 2016 may 8]. available from: https://s3.amazonaws.com/webvault/ebooks/ljslj_ebookusage_publiclibraries_2014.pdf meldrem, j., mardis, l., johnson, c. 2005. redesign your reference desk: get rid of it! currents and convergence: navigating the rivers of change proceedings of the twelfth national conference of the association of college and research libraries. ed. hugh a. thompson. chicago: association of college and research libraries, 2005. miles, d. 2013. shall we get rid of the reference desk? reference & user services quarterly 52(4): 320-324. sonntag, g., & palsson, f. 2007. no longer the sacred cow—no longer a desk: transforming reference service to meet 21st century user needs. library philosophy and practice 45(2): 104-107. summerhill, k. 1994. the high cost of reference: the need to reassess services and service delivery. the reference librarian 20(43): 71-85. about the author michael carlozzi (michael@activelibrarians.com) is the library director at wareham free library in massachusetts. previously, he was technology and information services librarian at the canton public library in massachusetts. he dreams of a world that takes cribbage more seriously. subscribe to comments: for this article | for all articles one response to "data for decision making: tracking your library’s needs with trackref" please leave a response below: beeresh, 2023-09-28 hi, mr michael carlozzi, greetings!! trackref is really helpful and appreciate your kind efforts in this regard. i am also trying to use tracref for my library. this is to say thanks to you., request you to mail me your further thoughts. i may need your help making use of this tool. thank you, beeresh, bangalore, india leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – introducing sage: an open-source solution for customizable discovery across collections mission editorial committee process and structure code4lib issue 52, 2021-09-22 introducing sage: an open-source solution for customizable discovery across collections digital libraries at research universities make use of a wide range of unique tools to enable the sharing of eclectic sets of texts, images, audio, video, and other digital objects. presenting these assorted local treasures to the world can be a challenge, since text is often siloed with text, images with images, and so on, such that per type, there may be separate user experiences in a variety of unique discovery interfaces. one common tool that has been developed in recent years to potentially unite them all is the apache solr index. texas a&m university (tamu) libraries has harnessed solr for internal indexing for repositories like dspace, fedora, and avalon. impressed by frameworks like blacklight at peer institutions, tamu libraries wrote an analogous set of tools in java, and thus was born sage, the solr aggregation engine, with two primary functions: 1) aggregating solr indices or “cores,” from various local sources, and 2) presenting search facility to the user in a discovery interface. by david b. lowe, james creel, elizabeth german, douglas hahn, and jeremy huff 1)  introduction a persistent challenge facing the digital library community is how to provide access to a large set of heterogeneous digital collections with disparate metadata. a widely adopted solution for searching these digital collections is apache solr. it is efficient and flexible open-source software with a broad user base in other industries. the solr discovery application programming interface (api) is a common solution in the library space with applications such as dspace, fedora, and blacklight. sage (solr aggregation engine) is a new solution from texas a&m university libraries for aggregating solr documents from across repositories.  sage is an open-source application that provides two feature sets: 1) aggregation and 2) discovery views. sage combines any number of solr indices (or “cores”), crosswalks the fields, and generates one (or more) aggregated index. the discovery view feature set can be utilized via the user interface (ui) or api and enables dynamic customization of search interfaces and search results for any given solr core. sage source code and documentation can be found on github. table 1. sage links. link type url open source code https://github.com/tamulib/sage collections in production from fedora https://u.tamu.edu/berger-cloonan https://u.tamu.edu/comrosters from dspace https://u.tamu.edu/cubamaps this article will provide an overview of the technical background of sage, suggest future development of sage, and discuss its current and future use within libraries. 2) brief historical look at library search tools while code4lib journal readers are undoubtedly familiar with the historical background of online searching in libraries, still there is a trend worth emphasizing that relates to the aggregational nature and function of sage.  first, libraries can boast a strong heritage as early implementers of information and communications technology (ict), with record #1 (the rand mcnally book of favorite pastimes) in the authoritative and collaborative oclc bibliographic database denoted as entered in may of 1969.  in the ensuing transition from card catalogs to online public access catalogs (opacs), the library community embraced technologies such as telnet, gopher, and wide area information server (wais) to share the records of their holdings and make them searchable.  while the card catalog had been an access system mostly limited to what books were on a given library’s shelves, forsaking it left some of the more nostalgic information seekers with nasty dispositions (see baker 1994); in the end, the scale of library holdings and their growth rates were destined to overwhelm our analog systems.  harnessing the unstoppable force of the web throughout the 1990s translated into momentum that could match that grand scale such that today, rare is the library that cannot be searched via browser online. the succeeding generation of library catalogs, dubbed “discovery services,” sought to bring not just the physical book collections but also the library’s subscription content accessible online within typeable reach of a single search box.  like sage, discovery services feature a unified index of all the items they cover.  they followed a foray into what were deemed “federated search” tools which covered essentially these same disparate resources, but the federated search tools attempted to access a heterogeneous array of indices in real time, so their comprehensive result sets were routinely thwarted by network traffic and other responsiveness factors.  for both federated and discovery services, the technical innovation shared with sage is a heterogeneous index or set of indices from multiple sources that can be brought together into a single, searchable unit for the end-user. in parallel with the rise of discovery tools, and in the wake of concurrent mass digitization efforts, ict implementers in libraries began to devote increasing attention to making their freshly digitized collections available.  another development intertwined here from the early 2000s was the rush to build institutional repositories, primarily in support of open access (oa) efforts by providing a platform for hosting faculty, graduate, and (usually honors-related) undergraduate student research.  the lines between strictly defined institutional repositories (containing theses, dissertations, and oa journal articles) and venues for digital collections scoped more broadly (historical photographs, letters, maps, etc.) often became blurred. also in the 2000s, the information landscape witnessed two more important, related trends:  mass book digitization and web search tool frameworks that made searching these new large text corpora possible in a timely manner.  in order to index not just the metadata now, but also the bulky full text from within these millions of books required new tools, and the root driving force for searching that mixture of metadata and content has been apache lucene, first written by doug cutting and released in 1999 (ingersoll 2007).  lucene still, as of july 2021, holds close to one third (31.66%) market share in the enterprise search category, according to datanyze (datanyze 2021).  and it has fruitfully borne successful offspring.  hot on lucene’s heels (currently at 31.46% market share) is solr, based on the lucene search library and created in 2004 by yonik seeley at cnet (thuma 2018), who donated it to the apache software foundation in 2006.  notably, hathitrust, as the largest (currently over 17 million items) full-text book searching operation yet achieved by libraries, relies on solr to power its searches.  to close this thread, two other library-related, lucene-based search tools worth noting include elasticsearch (with 7.43% market share), first released by shay banon in 2010 (banon 2010), and secondly xtf (extensible text framework), first released in 2005 (haye 2005) by the california digital library (cdl) and used in some digital collections and archival finding aid applications in research libraries.  it is solr, though, that is most relevant to the topic of this paper. to focus on digital library applications, it is fair to note that sage is not unique in the way it leverages these critical solr resources.  in fact, it owes much to the established blacklight software efforts.  blacklight traces its origins to 2009 at the university of virginia.  in the texas a&m university libraries context, as we implemented blacklight’s cousin spotlight as our digital exhibits presentation tool, we also considered blacklight, which is written in ruby on rails.  since our application development shop leans more toward java, it was only natural that our developers sought to manipulate this critical piece of our architecture more nimbly and fluently. 3) ecosystem context concerning the people involved, the digital initiatives unit at texas a&m university libraries includes an applications development team that produces custom software for the libraries and contributes to open-source products that advance the libraries’ mission.  past successes include a major role, in collaboration with the texas digital library, in the development of vireo  (https://github.com/texasdigitallibrary/vireo), a workflow tracking application for approving and depositing electronic theses and dissertations into institutional repositories.  vireo has been implemented widely among research universities thanks to its thoughtful design and ease of use.  among current projects, tamu developers are heavily involved in the new open-source integrated library system known as folio (for “the future of libraries is open;” see: https://github.com/folio-org).  a commitment to open-source is a part of the team ethos, and they pursue opportunities for collaborative development. concerning the technology involved, solr provides an http api to a backend implemented with the apache lucene search library.  as we have seen, the solr search platform is common in digital library applications such as catalogs, digital repositories, and other discovery tools.  the texas a&m university libraries run a number of solr-based applications, including instances of dspace, fedora, spotlight, vufind, and sage. in a single-search-box world, our library patrons have come to expect the ability to search across various sources regardless of the application. however, providing such a search service presents technical challenges.  the fact that all these applications employ solr offers us a way forward. the primary technical challenge in providing search across heterogeneous collections is the homogenization of their metadata.  for the most part, a metadata schema’s implementation tends to be specific to an institution.  most library search applications allow for customization of the metadata schema and how documents appear in a search index.  efforts to standardize schemas across institutions can impose some order, but it seems unlikely to us that the worldwide library community will settle on one schema to rule them all.  some efforts that have had considerable success in specific contexts include marc, dublin core, bibframe, and schema.org.  however, even broad multi-institution adoption of a schema standard (as has been the case with dublin core in dspace) does not render metadata interoperable – local practices for what goes in a particular field will vary and local customizations to the schema are all but inevitable. furthermore, even within an institution with consistent practices, the metadata will vary across collections of different types of items.  for example, photographs may not bear titles, but may require descriptive fields not applicable to print documents. the requirements and restrictions of a solr core afford sage a useful structural starting point.  solr cores consist of documents that themselves consist of keys and values in a flat structure.  as adventitious and heterogeneous as a set of records may be, if it has been indexed into solr, it must adhere to this structure.  at the heart of sage is a mechanism for defining how a key in one solr core could be mapped to a key in an aggregated index.  through this mechanism, a curator may choose how to reconcile and normalize the indices throughout the ecosystem.  furthermore, as we will see, a curator may choose different mappings for different ways of viewing the collections. 4)  aggregation  at the center of sage is aggregation, which may be considered analogous to an etl (extract, transform, load) process.  each aggregation process is referred to as a “job” and employs any number of each of these four elements: internal metadata:  a user-defined set of metadata keys or labels reader:  gathers data from a source solr core and employs a user-defined mapping from that source into sage’s internal metadata operator:  takes a value from a specified key from the source and performs pre-defined operations such as transformations (e.g., regex) or normalizations (e.g., date formatting) writer:  outputs the internal metadata, utilizing a user-defined mapping into an external solr core through this process it is possible to normalize or modify the data using sage’s user-specified operators so that a uniform search experience can be provided to end-users even though the original solr cores may be disparate and contain incomplete data.  sage provides administrators with a simple interface to configure the aggregation process.  in summary, the sage aggregation process, or job, reads a solr core, performs any data manipulation through the operators, transforms data into a standard metadata schema, and writes the data into the sage solr core. figure 1. aggregation: overview. internal metadata the internal metadata schema is the local storage that sage uses to preserve the data elements that are read from various solr cores.  it can be created and modified at any time. there is no limit to the number of metadata elements that can be created, and the schema should contain all the elements that might be needed for aggregation. in defining the internal metadata schema for texas a&m universities libraries we relied heavily on our libraries’ metadata guidelines for our digital collections:  https://hdl.handle.net/1969.1/175368. figure 2: aggregation: internal metadata. readers readers handle the process of running a query against a solr core and of harvesting the data to be stored in the internal metadata.  this process involves crosswalking the solr core data definitions to the internal metadata schema.  for texas a&m universities libraries it is not uncommon for our digital collections to exist in dspace and fedora solr cores.  below is an example of mapping the solr fields from dspace and fedora into the internal metadata schema based on our metadata guidelines.   dspace solr core element fedora solr core element sage internal metadata dc.title title title dc.subject subject_ss subject dc.creator creator_ss creator table 2:  aggregation: possible metadata crosswalk mappings used by sage readers operators a powerful characteristic of sage is the operator feature.  operators are built in transformation functions that can be applied to the data as it flows through the aggregation process.  these functions can be added to or expanded upon as needed.  for example, one operator may perform data normalization so that all known variations on a given field could be standardized ensuring the results are consistent. figure 3. aggregation: operators. writers writers are similar in setup and use to the readers except they take the internal metadata and push it out to a solr core which is used by discovery. moreover, multiple writers can push internal metadata with specific transformations to a single sage solr core.   figure 4. aggregation: writers. once the readers, operators, and writers are properly defined, it is possible to rapidly create multiple sage solr cores that can be used by the discovery view.  through this process the institution is able to easily highlight and promote disparate content and collections that normally may be buried deep in incompatible systems.   5) discovery views discovery view general description aside from the creation of solr cores from disparate sources, sage offers the discovery view feature set. this suite of features focuses on exposing the aggregate cores through an attractive discovery interface. these interfaces can be created by administrative users and can be highly customized within the context of a management dashboard. once created, they can be utilized by the end-users to conduct both keyword text and faceted searches of the aggregated data set. discovery views for end-users  figure 5. discovery view: results page. figure 6. discovery view: single item view. sage’s discovery view feature set is distributed between two modes of ui interaction: administrative and end-user. for the end-user, sage discovery views are landing pages featuring various form inputs which allow for several modes of search queries, and a results section that displays all records retrieved through the user’s query. after a search is executed, either through a keyword text query or a faceted search, the results are depicted in a list. each result displays a predetermined subset of the record’s metadata and serves as linked navigation to a single item view. the single item view for any record displays a more complete depiction of the selected record’s public metadata as well as rendering the digital object via a viewer appropriate to its resource type. discovery views for administrators figure 7. discovery view: single item view. figure 8. discovery view editing: results tab. for the administrative user, sage discovery views offer a management dashboard that allows for the creation and customization of these discovery views. each discovery view can expose any solr core known to the sage application. a discovery view can be configured to pre-filter the records it exposes by providing a canned solr filter query. this means that a discrete discovery view does not necessarily need to expose the entirety of the solr core upon which it is based. given this, multiple discovery views might be created for different subsets of the same solr core. for example, while there might be a single fedora solr core, there could be multiple discovery views based upon specific collections. the fields and the widgets used to interact with these facets can be selected through the administrative interface, as well as the keys which are available for the text-based searches. through this customization interface, the administrator can also select what metadata will be displayed within the result list as well as within the single item view. additionally, various elements of the landing page can be customized, including a thematic image, associated links, and introductory text. the rationale behind discovery views a driving force behind the creation of sage discovery views was a desire to achieve two primary conveniences. the first of these is the ability to deploy and maintain a single instance of an application which allows for the creation and customization of multiple search engine results page (serp), each with a unique configuration. if instead there is a one-to-one relationship between a serp’s configuration and the application providing that view, the multiplication of serps would result in an increase in resource expenditure both in maintenance time and computing resources. secondly, it was desirable that the configuration of discovery views be exposed through a user interface and not require developer intervention to create or modify. this would empower non-developers to both expose and maintain the records they wished to make available through discovery. potential future development for discovery views to date, sage is capable of rendering records with various viewers, including mirador 2, the browser’s native pdf viewer, and a basic text renderer. these viewers are easily expanded upon, and this represents one area where sage can grow. support for audio, video, map, and 3d object viewers are all plausible areas where sage is likely to see development. additionally, the facet widgets which sage exposes can be expanded. currently, sage allows for link-list facet and free text facet (with type-ahead support) to be configured for each facet field. additional facet widgets, such as range selection or geo-coordinate input, would greatly increase the value of sage discovery views. 6) broader implications implementing disparate data source types currently, sage has been developed to aggregate a single type of data source into a combined solr core, and that source type is itself the solr core. a possible future innovation in sage could be the addition of multiple source types. this would allow sage to create aggregate solr cores not only from other solr cores, but from disparate sources, such as: csvs; tsvs; various apis; or iiif collection metadata. the potential source types that might be implemented are practically limitless. fortunately, the groundwork for a feature of this nature was planned for in the earliest iterations of sage, reducing the development barrier for these disparate sources to be implemented. general library discovery the library discovery ecosystem comprises many systems and interfaces including legacy opacs, e-journal search platforms, database directories, and local repositories. sage has the potential to move the library discovery space to an extensible search model where new targets are able to be developed and incorporated into the libraries’ search strategy while maintaining a consistent search experience for users. the discovery views feature set enables non-developers to create and manage these experiences without the need to burden valuable developer time and resources. future work can include additional value-added services such as recommendation, spell check, and other automated search assistance at a view-by-view level and additional views for any solr-based system within the library’s discovery ecosystem and in the future, any source type. digital humanities and remixing at the most basic level, the discovery view capabilities unlock numerous potential applications for digital humanities (dh) and other text-focused research.  awareness of a digital object is an important first step in research, followed by the usual concerns while searching sets of things, such as precision and recall. the flexibility of solr enables all of the above.  within a result set, researchers can turn to manipulating sage’s facets for counts to use in statistical inquiries relevant to sets of metadata and full text.  as mentioned immediately above, the future direction of empowering the searcher to extend the greater index set would be another boon to dh in particular and text-based searching in general. bibliography always already computational – collections as data. “collections as data facets.” accessed january 8, 2021. https://collectionsasdata.github.io/facet1/. anant/awesome-solr. 2017. anant corporation, 2021. https://github.com/anant/awesome-solr. “apache solr.” accessed january 8, 2021. https://lucene.apache.org/solr/. baker n.  1994 “discards.” the new yorker, april 4, 1994. https://www.newyorker.com/magazine/1994/04/04/discards. banon, shay. “you know, for search.” elastic blog, february 8, 2010. https://www.elastic.co/blog/you-know-for-search. becker d, williamson e, and wikle o. “collectionbuilder-contentdm: developing a static web ‘skin’ for contentdm-based digital collections.” the code4lib journal, no. 49 (august 10, 2020). https://journal.code4lib.org/articles/15326. bolton, m, creel j, day k, hahn d, huff j, laddusaw r, savell j, and welling w. “user-configurable discovery across collections,” may 22, 2019. https://tdl-ir.tdl.org/handle/2249.1/156404. cartolano rt. “history of blacklight,” 2015. https://doi.org/10.7916/d8j38s9m. cole tw., and shreeves sl. “search and discovery across collections: the imls digital collections and content project.” library hi tech 22, no. 3 (january 1, 2004): 307–22. https://doi.org/10.1108/07378830410560107. datanyze. “apache solr market share and competitor report | compare to apache solr, apache lucene, swiftype.” datanyze. accessed july 5, 2021. https://www.datanyze.com/market-share/enterprise-search–287/apache-solr-market-share. ———. “enterprise search market share report | competitor analysis | apache lucene, apache solr, swiftype.” datanyze. accessed july 5, 2021. https://www.datanyze.com/market-share/enterprise-search–287. db-engines. “db-engines ranking.” accessed january 8, 2021. https://db-engines.com/en/ranking/search+engine. digital library services task force. “final report [of the] digital library services task force 2.” university of california libraries, 2011. https://libraries.universityofcalifornia.edu/groups/files/dlstf/docs/dlstf2_final_10may2011.pdf. duplain r, balser ds, and radziwill nm. “build great web search applications quickly with solr and blacklight.” in software and cyberinfrastructure for astronomy, 7740:774011. international society for optics and photonics, 2010. https://doi.org/10.1117/12.857899. european commission. joint research centre. institute for prospective technological studies. “enterprise search in the european union: a techno economic analysis.” lu: publications office, 2013. https://data.europa.eu/doi/10.2791/17809. gaona-garcía pa, martin-moncunill d, and montenegro-marin ce. “trends and challenges of visual search interfaces in digital libraries and repositories.” the electronic library 35, no. 1 (january 1, 2017): 69–98. https://doi.org/10.1108/el-03-2015-0046. gilbert h, and mobley t. “breaking up with contentdm: why and how one institution took the leap to open source.” the code4lib journal, no. 20 (april 17, 2013). https://journal.code4lib.org/articles/8327. goodmann e, matienzo ma, vancour s, and vanden dries w. “building the national radio recordings database: a big data approach to documenting audio heritage.” in 2019 ieee international conference on big data (big data), 3080–86. los angeles, ca, usa: ieee, 2019. https://doi.org/10.1109/bigdata47090.2019.9006520. haye, martin. “internationalizing xtf: bringing multilingual features to the extensible text framework,” may 2005. https://xtf.cdlib.org/wp-content/uploads/2010/08/internationalizing_xtf_v2.pdf. ho j, and stokes ck. “metadata guidelines for digital resources at texas a&m university libraries.” working paper, may 16, 2019. https://oaktrust.library.tamu.edu/handle/1969.1/175368. ingersoll, grant. “better search with apache lucene and solr.” 2007. https://web.archive.org/web/20120131154001/http://trijug.org/downloads/trijug-11-07.pdf. kelbert p, droege g, barker k, braak k, cawsey em, coddington j, robertson t, whitacre j, and güntsch a. “b-hit – a tool for harvesting and indexing biodiversity data.” plos one 10, no. 11 (november 6, 2015). https://doi.org/10.1371/journal.pone.0142240. “large-scale full-text indexing with solr | www.hathitrust.org | hathitrust digital library.” accessed july 5, 2021. https://www.hathitrust.org/blogs/large-scale-search/large-scale-full-text-indexing-solr. pham k, reyes f, and rynhart j. “building a library search infrastructure with elasticsearch.” code4lib journal, no. 48 (may 1, 2020). https://proxy.library.tamu.edu/login?url=https://search.ebscohost.com/login.aspx?direct=true&db=edsdoj&an=edsdoj.594fb9ee35e64189900db986330db681&site=eds-live. pretoro ed, de roock e, fremout w, buelinckx e, buyle s, and van der stede v. “optimizing elasticsearch search experience using a thesaurus.” the code4lib journal, no. 51 (june 14, 2021). https://journal.code4lib.org/articles/15749. seeley y. “a history of lucene and solr.” solr ’n stuff (blog), june 16, 2014. https://yonik.com/lucene-solr-history/. thuma, john. “what is apache solr.” medium, august 9, 2018. https://johnthuma.medium.com/what-is-apache-solr-a18a60004e70. weig ec, and slone m. “spokedb: open-source information management system for oral history.” digital library perspectives 34, no. 2 (january 1, 2018): 101–16. https://doi.org/10.1108/dlp-03-2017-0012. williams s. “better search through query expansion using controlled vocabularies and apache solr.” the code4lib journal, no. 20 (april 17, 2013). https://journal.code4lib.org/articles/7787. zhang h, durbin m, dunn j, cowan w, and wheeler b. “faceted search for heterogeneous digital collections.” in proceedings of the 12th acm/ieee-cs joint conference on digital libraries, 425–26. jcdl ’12. new york, ny, usa: association for computing machinery, 2012. https://doi.org/10.1145/2232817.2232924. acknowledgements the authors would like to acknowledge additional sage project developers michael bolton, kevin day, ryan laddusaw, rincy mathew, jason savell, and william welling. about the authors david b. lowe (https://orcid.org/0000-0003-2856-8629 ) is an assistant professor and the digital collections management librarian at texas a&m university libraries, where he was recently selected as one of eight 2021 texas a&m institute of data science career initiation fellows. his work responsibilities include policies and procedures for digital collections and the institutional repository in the libraries. his research interests include applications of artificial intelligence and machine learning techniques to text mining in a scholarly communication and digital humanities context, with digital collections as data. james creel is a software applications developer iv at texas a&m university libraries. james holds a master of science degree in computer science from texas a&m and has 15 years of experience developing and managing digital library applications. elizabeth german is an associate professor and the service design librarian at texas a&m university libraries where she focuses on bringing together user experience, project management, and accessibility in order to provide quality user experiences for researchers and learners. her research interests include library search behavior, accessibility inclusion, and assessment. douglas hahn is the director of library applications and integration at texas a&m university libraries with over 20 years’ experience in the computer information technology field. he received a master of science degree from university of north texas. his current interests are in the evolution of the web, and technology’s impact on society. jeremy huff is a software applications developer iii at texas a&m university libraries and has been involved in software development at the libraries for the past 8 years. in addition to development on sage, he has represented the libraries at texas a&m on several other open-source projects, including both folio and vireo. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – help! a simple method for getting back-up help to the reference desk mission editorial committee process and structure code4lib issue 2, 2008-03-24 help! a simple method for getting back-up help to the reference desk using the “net send” command, native to windows xp, librarians at the university of california, riverside created a “help button” for the reference desk. the simple script file sends a message to librarians’ workstations in their offices and logs the date and time of use. this paper describes that program. by ken furuta and michele potter introduction reference librarians are occasionally caught in the middle of time-consuming reference questions as the line of patrons needing service lengthens. leaving the current patron to call around for other librarians, or running around trying to grab one at random may take too much time and frustrate everyone. at the university of california, riverside (ucr) libraries, we faced this problem more and more as workload increases made double staffing less and less feasible. we needed a way to contact all of the librarians at the same time in a method similar to a pa message at a department store (i.e. “back-up librarians to the reference desk.”) looking for solutions in the library literature, we didn’t find anything to help us solve this problem. while there are papers on staffing policies and others on using statistics [1], none addressed the nuts and bolts of calling for back-up help efficiently. a search through the archive of a reference librarian listserv (libref-l) showed some questions on policies for backups, but no methods for calling it. what we decided to try, after some brainstorming, turned out to be completely free and has been working beautifully for a few years now. we are using the “net send” command, native to windows xp, to broadcast the message “need help at the reference desk” to the workstations in all of the librarians’ offices. the message and the recipients are contained in a short batch file on the desktop. the procedure requires only the click of a button to create a stampede of librarians heading for the reference desk. in our experience backups arrive within 45 seconds. if a stampede does arrive, it becomes a pleasant opportunity to stretch and briefly visit. if a stampede is not desirable, there may be some other possible refinements. for example, if librarian workspaces are close together, one librarian may offer to go. how it works the command “net send” was created for communication from system administrators to users by transmitting one-way messages to another machine on the network. the command’s syntax is: net send [name] message in the syntax, “[name]” is the name of the user or computer to which you would like to send the message, and “message” is the message you would like to send. to test the command or to send a single message to a single user, one can: click on the start menu click on run type cmd type net send [name] message this would work very well for inviting a colleague to lunch, but is hardly feasible for the reference desk, especially when the line is lengthening. however, as with all dos commands, one can create a batch file in order to execute a collection of commands all at once. in our case it looks something like this: net send librarian1 need help at the reference desk! net send librarian2 need help at the reference desk! etc… the batch file can be created in a text editor, such as notepad, and saved as a “.bat” file, e.g., “help.bat”. the message to the user looks like this and includes an audible alert: a shortcut to the batch file was placed on the desktop of all of the reference machines and works as a virtual help button. it is even possible to add coding to the file to keep track of when the button is used (see appendix). saving the batch file to a common, or shared, drive, instead of on each hard drive of every reference librarian’s pc, greatly simplifies maintenance. the file can be updated once and changes go into effect immediately. trouble shooting there are a few technical issues that may need to be resolved before the program will work smoothly. some are relatively minor while others may require you to work with your systems department to resolve. net send is a dos command, and as such it resides on your hard drive. if you see that the default drive for your command line interface is different (e.g., z:\>), you may need to add “c:” to the beginning of the batch file. if you are not sure of the user/computer names of your reference librarians, you can type “net view” in the command line interface to see a list of names (see below). the batch file progresses in order from user (or computer) name to user name. if one of the users is not logged in or if the computer is off, it takes a while for the timeout to get back to the originating computer. other users may not get the message for a few seconds to a minute. in order to partly resolve this issue, you can rearrange the list of recipients in order of the likelihood that they have logged in. another issue is the format of the user names. when our network was upgraded to sp2, the format of the user name changed from just the name to including the prefix “/domain:” (i.e. “net send /domain:librarian1 help is needed at the reference desk!”). the windows messenger service can be turned off on individual office workstations through the control panel. in this case the message will not appear on that machine. if you have the ability to see and alter your profile, this can be done in the administrative tools/services window. it is listed as “messenger”. in windows xp sp2, messenger is set to “disabled” as the default. it will need to be reset to “started.” if you cannot or do not want to mess with your settings, check with your systems department. it is also possible that the ability to receive net send messages can be turned off on a network because outsiders may also try to send net send messages (spam) to your computer. however, a network firewall can likely block these messages coming from the outside. this is an issue you should discuss with your systems department alternative method there has been some discussion at the ucr libraries about using msn messenger (formerly window messenger) or some other instant messaging program to call for help. in our case, msn messenger is already installed on our computers. msn messenger allows users to communicate and share files with others who are logged on at the same time. this may be a good alternative if you or your systems department is reluctant to enable microsoft messenger. however, we could not create a “canned” help message or list of recipients beforehand in msn messenger. thus the librarian at our desk would have to type a help message and designate the recipients when a backup is needed resulting in extra time calling for backup. other operating systems the windows vista operating system does not support the net send command. instead, vista users can use the “msg” command. the msg command is also available in xp, but is slightly more complicated to use. the syntax is “msg /server:[computer name] message”. it is worth noting that msg does not require the messenger service to be running on the target machines. it may even be possible to send messages between pcs and macs in mixed shops. however, we are unable to test this since ucr is a pc only shop. conclusion using a help button based on the native “net send” capabilities of most windows operating systems works well to send help requests to librarians’ offices when lines begin to form. the method may be most useful in departments where reference librarians’ offices are spread out, because the program sends a message to the various recipients no matter where they are located. note for example see deborah rinderknecht, “new norms for reference desk staffing adequacy: a comparative study,” college & research libraries 53 (1992): 429-36. appendix: help.bat file rem script created by michele potter. kenneth furuta added the log file. net send librarian1 need help at the reference desk! net send librarian2 need help at the reference desk! rem etc... rem code for the log file rem use the text parsing mode of the 'for' command and the /t (display only) rem switch on the date and time commands. rem "for in do" loops require 2 percent signs before variables, "%%variable_name" for /f 'tokens=1-4 delims=/ ' %%i in ('date /t') do ( set dayofweek=%%i set month=%%j set day=%%k set year=%%l ) rem only save the hour and minute. for /f 'tokens=1-2 delims=:' %%i in ('time /t') do ( set hour=%%i set minute=%%j ) rem create a comma delimited file. (note the comma between the variables). rem enclose variable names with percent signs. rem ">>" redirects the output from the screen to writing to a file. rem double angled brackets, ">>" appending it to the end of the file. echo %dayofweek%,%month%/%day%/%year%,%hour%:%minute% 1>> help_button_log.csv rem optional clean up variables (setting them to {nothing} deletes them) set dayofweek= set year= set month= set day= set hour= set minute= about the authors kenneth furuta and michele potter are both at the university of california, riverside university libraries. kenneth is the reference/information technology librarian in the rivera library (email: kfuruta@ucr.edu). michele is the engineering librarian in the science library (email: michelep@ucr.edu). the authors acknowledge vicki bloom’s (head, rivera reference department) contributions on this project. subscribe to comments: for this article | for all articles 4 responses to "help! a simple method for getting back-up help to the reference desk" please leave a response below: steve johnson, 2008-03-25 my network administrator noted that xp service pack 2 shuts down the net send command as a security risk. the command can be reenabled at the workstation level, as it will be here since the idea of a “help” icon seems quite popular. of course, we will also do a second icon, one with the message “call campus security to the reference desk.” the only drawback to the simple panic button message is that no information about the nature of the emergency is conveyed. bill helman, 2008-12-18 i would think that you could achieve a similar outcome, using msn messenger with a macro (from macro express, for instance). great idea, though! yoyosamo, 2010-09-10 hi, just wondering, will this work on windows 7 and what are the commands? ????? ???? ???????, 2011-07-07 this “help button” sure sounds like a great idea, especially as you said it would assist where reference librarians’ offices are spread out. nice thinking. :) leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – leveraging aviary for past and future audiovisual collections mission editorial committee process and structure code4lib issue 58, 2023-12-04 leveraging aviary for past and future audiovisual collections now that audio and video recording hardware is easy to use, highly portable, affordable, and capable of producing high quality content, many universities are seeing a rise in demand for oral history projects and programs on their campuses. the burden of preserving and providing access to this complex format typically falls on the library, oftentimes with no prior involvement or consultation with library staff. this can be challenging when many library staff have no formal training in oral history and only a passing familiarity with the format. to address this issue, librarians at the college of charleston have implemented avpreserve’s audiovisual content platform, aviary, to build out a successful oral history program. the authors will share their experience building new oral history programs that coexist alongside migrated audiovisual materials from legacy systems. they will detail how they approached migrating legacy oral histories in batch form, and how they leveraged aviary’s api and embed functionalities to present aviary audiovisual materials seamlessly alongside other cultural heritage materials in a single, searchable catalog. this article will also discuss techniques for managing an influx of oral histories from campus stakeholders and details on how to make efficient use of time-coded transcripts and indices for the best user experience possible. by tyler mobley and heather gilbert background and existing infrastructure the lowcountry digital library (lcdl) is a multi-institutional cooperative digital library that currently hosts several hundred collections consisting of almost 150,000 digitized images, objects, pages, and recordings of primary source materials and their associated metadata.  lcdl is built around the open-source resourcespace digital asset management system that serves as our central repository for metadata records and presentation-quality digital files ranging from still images to audio files and pdfs. to bring these collections to the public, we created a custom plugin that indexes records in the repository to apache solr, a java-based search platform. indexed records are then available for search and discovery via blacklight, lcdl’s primary catalog. blacklight is a ruby-based search and discovery platform that provides a graphical interface for interacting with data in the solr index. it provides full-text and faceted searching of all our materials, and the college of charleston libraries has relied on it for nearly a decade across both lcdl and the south carolina digital library (scdl), a provider hub for the digital public library of america.  figure 1. current lcdl technology infrastructure diagram. lcdl currently hosts over 600 oral histories in audio and video formats. our oral history collections have grown at a steady trickle over time, though we have noticed a substantial increase in interest in recent years from partners both within the college and local community thanks to the sudden ubiquity of smartphones and accessible audio visual options. as lcdl is focused on the collecting of historic cultural heritage materials related to the lowcountry region of south carolina, our hosted oral histories have naturally hewed to this subject area. however, we have increasingly been approached with oral histories covering topics and time periods that don’t necessarily fit the stated mission of lcdl. historically, oral histories in lcdl have been loaded to resourcespace in bulk as metadata and digital files. however, rather than serving these large files directly from resourcespace, we host copies of oral history files in amazon s3 storage and serve them for streaming with amazon cloudfront, a content delivery network (cdn) better suited for flexible streaming to browsers. the metadata records loaded to resourcespace contain a field referencing the file’s address on cloudfront. when indexed, that metadata field is served to blacklight to fill html embed code to embed the file on item pages for viewing. you can see an example of how that final item looks below. while this system allowed us to get oral histories online in a way that was adequately usable, it lacked in features and functionality. figure 2. existing lcdl oral history user interface.  selecting the “click here to read the embedded transcript.” assessing our options we began evaluating our options for a better oral history public interface in 2021.  we knew that we wanted to provide a more sophisticated experience for our patrons accessing oral histories. as mentioned above, lcdl employed an embedded pdf viewer for patrons to view transcripts while listening to an oral history, however we wanted the ability to offer value added features such as auto-scrolling of time-stamped transcripts and subject indexing.  as we considered our options, deciding factors also included sustainability, personnel support requirements, and pricing.  the college of charleston libraries has a history of working on a shoestring budget, but for sustainability purposes has been actively seeking out hosted solutions to specific digital projects with an effort to increase scalability and project longevity.  we also knew from previous migrations that for sustainability purposes, any new platform had to have the ability to import and export our content accurately and efficiently. additionally, we knew that adding advanced features to our oral history platform couldn’t require the need to hire additional staff, we had to work within our existing personnel infrastructure.  finally, we knew the solution had to be cost effective.  requesting recurring funding wasn’t out of the question, but the cost to benefit ratio had to be well supported. we assessed several different options including the university of kentucky’s oral history metadata synchronizer package known as ohms (including incorporating ohms into our existing digital library infrastructure), alma digital (college of charleston libraries migrated to alma in 2020), dspace (which at the time was the foundation for our existing albeit underused institutional repository), and aviary, which is a collaboration between avp and yale university’s fortunoff video archive for holocaust testimonies. ohms offered the most options for developing a sophisticated oral history viewing interface.  however, it would require self hosting, increasing our already hefty server footprint, advanced retraining of existing staff and, in all likelihood, the addition of personnel to support the extra work of file preparation, assuming we really wanted to take advantage of all ohms had to offer.  during our assessment, the barrier to sustainability was high enough that we didn’t need to assess this platform further using our other criteria. it was decided that this option was perhaps too sophisticated for both our needs and our capacity for support.   alma digital was an appealing option as we had just fully migrated to alma the year prior (2020), however we found the ex libris-provided documentation on alma digital lacking in specificity as to what the repository could actually do and after significant testing on our part we encountered problems having alma digital support our advanced mods metadata schema.  while alma digital could support both audio and video file formats, we found the user interface to be too simplistic for our needs. alma digital support documents indicated some import and export options, but with the platform’s simplicity, it wasn’t worth pursuing during our assessment. alma digital just didn’t suit our needs. similarly, we determined that dspace was an unsuitable option.  like alma digital, dspace has the ability to support audio and video file formats in their repository, but the interface was also lacking. dspace has robust import and export tools, but, ultimately that wasn’t enough to make it a good match for our needs. we decided that both alma digital and dspace didn’t provide any advantages over the existing lcdl display of oral histories.  aviary was our last option to evaluate, and we were pleasantly surprised by what we found.  aviary offered the advanced user interface options we were looking for, including auto scrolling of time-stamped transcripts and the ability to add custom indexing, while also providing a user-friendly back-end experience.  this made it easier for us to use existing staff who were more experienced with using a gui as opposed to a terminal or code-based command structure.  aviary was a hosted solution, so we wouldn’t be adding yet another digital project to our server support portfolio, and considering the modest size of our collection, it was relatively affordable.  finally, aviary offered us a lot of flexibility.  we could batch load collections or single load items, we could use some advanced features or use none of them without losing the level of interface we already provided in lcdl, we found we could embed aviary objects within our existing digital library for seamless discovery, and aviary offered a variety of permission and access levels that could support a wide range of use-case scenarios for digital collection access. furthermore, aviary provides a flexible api for interacting with resources and media files in the system. while the user interface provides multiple means of importing and exporting loaded oral history materials, the availability of an api to programmatically handle import and export in batches eased concerns about the possibility of future platform migrations and accidentally trapping our data in a closed system.  aviary’s flexibility has helped us prepare to serve a growing need for campus supported oral history projects alongside our legacy cultural heritage material. with aviary, new collections of oral histories not within the scope of lcdl can be loaded and presented through a styled search and discovery interface on aviary’s own platform. each institution’s collections are discoverable through a custom site to which a custom domain can be applied (in our case, https://lcdl.aviaryplatform.com), and the entire library of items hosted across all subscribers is also searchable as a federated catalog. our mission with aviary is to serve a new and growing audience in oral histories at our institution while also enriching the production and presentation of our existing collection of cultural heritage oral histories in lcdl. figure 3. the aviary platform interface provides discovery for all of our a/v collections whether they are outside the scope of lcdl (ewi society oral history collection) or within the scope of lcdl (college of charleston oral histories). configuration and batch loading aviary, as stated, is its own hosted platform, providing search and discovery for users alongside collection browsing functionality. in adopting aviary, we realized we would have to reconsider our workflows for oral histories at both a procedural and technical level. this was something of a quandary. should we now only load items straight to aviary instead of our backup-redundant repository? should we remove oral histories from lcdl’s primary catalog and direct users to aviary instead? for the purposes of item loading and record creation, we decided on a two-fold approach. metadata for lcdl collections would continue to be loaded to our resourcespace repository as before. these collections would not be indexed to the catalog from resourcespace, but they would remain together in a central location with other lcdl materials to ensure continuity in case of future platform migrations. these collections would then be additionally loaded to aviary via the system’s import job functionality. the batch-loading system employed by aviary allows an administrator to load csv files for metadata, media files, transcripts, and even indexes as a single job, and then aviary ties those components together to create complete resources. in this process, an oral history collection might be loaded with any number of rows representing individual resources in a ‘resources.csv’ file including a column for a unique identifier. the companion ‘media.csv’ and ‘transcripts.csv,’ representing av-related files and transcript files respectively, would hook on that primary key and create the relationships on the aviary back-end.  in order to import just over 600 legacy oral histories into aviary, we ran metadata exports of records per collection from our apache solr index to csv format. this could have also been done via resourcespace’s metadata export process, but our solr index more cleanly outputs extra data used for presentation like presentation filenames, transcript details, etc. this process provided us with a csv file for each oral history collection containing descriptive metadata for all records in that collection, one record per row. next, we pulled the cloudfront url from this csv into a separate ‘media’ csv file, again formatted one item per row. this file also included columns for both the record’s primary identifier and a new unique identifier for the media file. finally, in a third ‘transcripts’ csv file, we broke out the file path to each transcript per record as one record per row. this file additionally included a column referencing the media file’s unique identifier to ensure that they were associated upon load. these three csv files were then loaded in-browser to the aviary platform alongside a zip file of the referenced transcript files. once all necessary files were loaded, and the import job was assigned to a collection, the job ran in the background and records were added to aviary. while the details above might sound a little daunting, the process for our workflow ultimately involved taking one csv file, splitting it into three, and adding some additional administrative fields before uploading them all at once. since we usually only receive small batches of new oral histories at a time, our typical process going forward will most likely center around single item loading directly into aviary via traditional form-based data entry. however, it’s worth keeping in mind that for platform migrations or unusually large loads of oral histories, aviary’s import job functionality works effectively. in the event of future batch loads, we have since taken advantage of resourcespace’s plugin functionality and written a plugin that produces an aviary-compliant ‘resource’ csv file for use with the import job process. this should reduce some data manipulation steps and allow our staff, at least at a metadata level, to only worry about the initial metadata creation and load process. with aviary’s import functionality, we were able to load our bulk of legacy oral histories to aviary in about 12 total ‘jobs.’ most of these records, though successfully loaded, were not immediately made public as we then had to work through metadata remediation on the legacy metadata and transcripts associated with these oral histories. further, each collection in aviary had to be configured for appropriate display of our preferred metadata fields. content migration because of aviary’s enhanced feature set, we wanted to not only batch load new collections, but migrate existing lcdl collections into the new platform.  this would allow our legacy oral history collections to benefit from these features if project partners wanted to review these items and add augmented metadata such as time stamping and/or subject indices.  however, as with most migrations, this required remediation efforts.  while we were able to use batch loading to easily migrate all of our existing oral history collections into aviary, the content was not adequately formatted for public display.  we determined that, to get the most out of our aviary instance, we would need to manually configure metadata mapping for each aviary collection, reformat transcripts to meet aviary’s transcript requirements, load pdf transcripts in aviary’s “supplemental file” field for researchers to have downloadable access, and finally create subject indices (as time and budget allowed) for recordings that would best benefit from this feature. aviary comes standard with a robust set of dublin core metadata elements but it also allows for broad metadata field creation and customization. lcdl collections are described using the mods metadata schema, so batch importing the collections also imported our metadata elements as custom fields.  this worked well for us as it meant that we didn’t have to do much metadata rectification pre or post migration.  however, as the end goal for our lcdl aviary destined collections was reintegration within the lcdl platform, it was important for us to mirror the display order of the metadata elements between lcdl and aviary.  aviary allows for this level of customization at the collection level on the resource description menu but there is no way to customize metadata field order at the instance level (as far as we could determine at the time of this writing).  this is unfortunate for a project of this nature, as our display order is uniform across collections.  while the resource description page is easy to use with a drag and drop interface, it must be configured for each collection to adequately reflect the metadata display order we desire. while this isn’t overly time consuming, for as granular a product as aviary is, it feels like an oversight to not have the ability to set display preferences at the instance level.  however, aviary is being very actively developed, with each update providing major improvements, so it’s quite possible that this may be available in the future. figure 4. aviary allows for granular customization of metadata fields and display at the collection level. unfortunately, transcript migration is proving to be the most challenging part of our rectification project. our legacy oral history collections only required the submission of pdf transcripts.  aviary utilizes .doc, .docx, or .txt transcripts for their transcript display.  this is requiring us to convert pdfs to word documents and clean up formatting issues to conform with aviary import preferences.  as with all products, there are quirks.  if a transcript contains any text that is in hh:mm:ss or hh:mm format, even if it is within a sentence, aviary interprets this data as a timestamp and will link the sentence, inappropriately, to the time indicated.  this is easy enough to correct within transcripts, however, we’ve also noticed that aviary will treat any colon used within a sentence, even without numerical context, as a 00:00 timestamp and will wrongly link the sentence and/or paragraph.  this is something that we are addressing with future projects in a transcription style guide, however it still is adding time to our rectification process of legacy materials. some of our older oral history collections had their transcript text formatted into tables, therefore stripping out this formatting in the word document is necessary but time consuming. ultimately, we think the time involved will be worth it as we’ve developed robust find and replace practices that help with some of these formatting challenges.  additionally, we still get to make good use out of our original pdf transcripts.  aviary provides a “supplemental file” field for every record.  we’ve found that we can load the pdf transcript in that field and allow download access.  this way our patrons get the best of both worlds – the recordings can be listened to/watched with auto-scrolling enabled (assuming time stamps are in place) and they can download a human-readable transcript for later research use. figure 5. the aviary public user interface provides a tabbed viewing experience. the description tab displays item level metadata, the transcript tab allows for auto-scrolling of time stamped transcripts as well as multiple transcript selection support, and the supplemental files tab allows for downloadable pdf transcripts (in addition to other file formats). subject indexing is one of the areas that excited us most about aviary, but in the short term it is the feature we have used least.  we plan on leveraging this more post-migration as an optional, value-added feature that can be used to augment existing metadata or possibly be leveraged as an experiential learning opportunity within faculty/student partnership projects. aviary allows for batch importing of subject indices, but also provides a very easy to use back-end interface for one-off index creation.  from the aviary back-end, any recording can be reviewed, and a subject index added using a set of simple forms.  the index can be as complex as including the relevant portion of a transcript or as simple as adding a title and very brief description.  indices can also include keywords and subject headings, which are searchable within both the resource and the overall platform, gps data, and hyperlinks.  subject indexing is an area which we have just started to experiment with.  we’ve created some recommendations for indexing for our partners both to ensure best practices are being followed and to moderate expectations and requirements of staff time. in the future, we hope that this feature may lend itself to faculty/student collaborations, where a faculty member might want to assign a recording to a student and that student can then review and suggest appropriate subject indexing.  in early conversations, faculty have shown interest in this potential. figure 6. in addition to batch loading of subject indices, aviary also provides a friendly user interface for recording review and index creation. we anticipate being able to leverage this to create more interactive experiential learning opportunities for students on future projects. integration with existing platform in terms of presentation and discovery, we quickly decided that removing oral histories from the lcdl catalog would hinder search and discovery and undermine our promise to our partners that the items they had submitted to us would be in lcdl. we would need to find a solution that maintained a search and discovery experience that our partners and audience had come to expect while doing our best to integrate the benefits gained from leveraging aviary. to that end, we began work on integrating aviary items into our primary catalog by indexing aviary data into apache solr with the aviary api. aviary’s api was an attractive feature for us when we were reviewing the platform. it provides well-documented restful interaction methods for loaded resources and collections. this allowed us to leverage the api via a script in php. the script is fairly simple. it accepts a passed-in csv file of resource identifiers that correspond to items in aviary. the script then authenticates against the aviary api, iterates over each resource, and builds and commits a solr document for each resource to our apache solr index. in iterating over each resource in aviary, we do some modifications to records. first, since aviary’s api outputs metadata field names in their proper display format, the script crosswalks those field names to their solr counterparts. for instance, ‘date digital’ translates to ‘datedigital’ or ‘contributing institution’ to ‘contribinst.’ further, we inject new fields into the record for easier flagging and control in blacklight. for example, we inject a custom placeholder image url for a thumbnail for catalog display, and we also add an ‘is-aviary’ field as a simple flag to check when loading image viewers on oral history item pages. in our current transitory period of launch and migration, the flag allows the catalog pages to quickly slot in either legacy viewer code for cloudfront urls or a new embed snippet from aviary depending on the value. through this script, we can serve the primary lcdl catalog with results mixed from both our central repository and aviary in a single search. while integrating two data sources into a single index was straightforward, we had philosophical discussions about how to handle the item display of these mixed results. initially, we considered making aviary items in the catalog open a new tab to show the aviary resource page in full. of course, that presented the issue where we would be jettisoning users from the site and their own search patterns without a clean way to get them back into the catalog. ultimately, we opted to leverage aviary’s embedding feature directly into our item display. one of the big strengths of aviary that drew us to the platform was the richness of its viewing experience. as discussed earlier, aviary provides auto-scrolling transcript functionality, indexed timestamps, and other features that we do not have the time or capability to incorporate directly into lcdl ourselves. instead, legacy oral histories simply provided an audio/video player and an embedded pdf of a transcript alongside each other, previously shown in an example image above. to get the full benefit of our investment in aviary, we decided to use aviary’s embed functions on item show pages. the embed functionality provides a duplication of the full viewer as seen on aviary, and allows users to follow along with transcripts live, navigate an oral history via timestamps, and more. this functionality does rely on an iframe currently, but thanks to the fact that all our solr-indexed metadata is still included farther down the page as usual, we do not lose any helpful seo or readability on the record. an example of an aviary embed on the lcdl catalog can be seen below. figure 7. aviary hosted oral history viewed within the lcdl platform. note that transcript auto-scrolling is an option as are subject indices and supplemental files. workflows for new oral history projects oral history was identified as a growing need on the college of charleston campus several years ago, however many potential oral history projects did not fit the criteria of the lowcountry digital library. as discussed above, using aviary has provided us with significantly more options for supporting campus oral history projects as we can now provide a platform and federated search for all oral histories while still only integrating oral histories into lcdl that meet that platform’s curation criteria.   we’ve had early success working with academic faculty outside the library, using simple strategies to manage workflows while still collecting the data needed to upload and make their recordings accessible. these strategies include early intervention regarding project development, copyright, and informed consent, campus sponsored digital file storage, and data collection using simple online forms. setting expectations early with faculty regarding project development and the level of informed consent needed in oral history projects was vital to our early success.  prior to this strategy, the library was being approached with completed “oral history” projects with the incorrect expectations that:  we would accept any project into our special collections holdings and  we would rapidly provide anything from online access to a custom website, digital catalog, and submission portal. unfortunately, none of the above is true. our resources are limited and oftentimes these completed projects would not even include consent forms that adequately covered copyright and sharing permissions.  therefore, early intervention in the project development stage has been extraordinarily helpful.  this way we reach faculty prior to project start to answer any questions they may have about designing a project appropriate for library collaboration and can provide sample consent forms that cover a wide variety of permission and copyright options.  this early intervention also allows us to plan ahead for upcoming projects and minimize the number of “surprise” digital collections that can turn up on one’s desk. for our digital library partners outside of campus, our expectation is that collections are delivered to us either via the cloud or an external drive.  however, external partnership comes with metadata training and templates as well as some expectations and dedication on behalf of the partner organization’s staff.  we’ve tried this approach with our faculty in the past with mixed results. further, oral histories often have more moving parts than a straightforward digitized manuscript collection.  therefore, we’ve started setting up structured cloud drives for use in faculty oral history collections.  our campus uses microsoft teams and sharepoint for shared storage.  once we’ve discussed the upcoming project with the faculty partners, we set up a teams drive for the project which includes all of the documentation and consent forms discussed in the initial meeting.   every recording gets its own folder which should be named after the interviewee. each interviewee folder will eventually contain the recording file, the transcript, the signed consent forms, and any other information deemed necessary. this way there is no excuse for poor file management.  we also upload simple read me instructions into the drive so faculty partners know what to do within the drive and can stay on top of the process. this read me file includes links to a “submission form” that is used to capture as much data about the interview from the faculty partner as possible. metadata collection can be challenging from our faculty partners who have no metadata experience and no inclination to learn that skill.  by using a simple online form (we use google forms) we can collect the most salient points.  then, our digital projects librarian can quickly export the entries into a spreadsheet and massage the results into usable descriptive and administrative metadata. while these are very simple, common-sense techniques and strategies, we’ve found them to be most effective at quickly and efficiently aiding in the development of successful oral history projects while both managing campus expectations and being mindful of staffing restrictions. lessons learned this process has provided several learning opportunities for us that are specific to both digital asset migration and oral history management.  while integrating aviary into the traditional lcdl user experience proved straightforward from a technical standpoint, it became clear that this integration would cause a significant change to our staff’s workflows for oral histories. by continuing to add items to resourcespace for backup purposes alongside their presence in aviary, we were effectively doubling our commitment to the ingestion and management of oral histories for lcdl. though oral history collections rarely arrive in large batches, we realized that this process could become cumbersome for staff over time if we didn’t focus on developing methods to ease this new workflow. this lesson led us to the creation of the previously mentioned custom plugin in resourcespace that generates aviary-compliant csv files for batch imports. through this process, the initial work performed in loading items to resourcespace can also complete a portion of the work needed to load to aviary. it has also led us to plan for future work to leverage the resourcespace and aviary apis to automate the process of loading items more fully from one system to the other. additionally, the close integration of aviary items into a unified catalog introduced another layer of technical complexity to our infrastructure. the apache solr index feeding our catalog’s search results now houses the combined feeds of aviary and resourcespace. further, item display pages in the catalog now use hooks in those solr documents to correctly display aviary’s embed functionality. while it wasn’t difficult to make these changes to our infrastructure, they introduced new challenges for lcdl staff in how to approach troubleshooting. after all, even with years of training, we all occasionally make mistakes during the metadata creation and item loading process. furthermore, the public is often quick to find and point out errors that make it into production. the integration of aviary into this mix means we had to train and prepare staff for what to do when errors arise, and how and where to resolve them. in addition to the above technical workflow changes, we also learned that migration of legacy oral history transcripts can be very time consuming, especially if the format of the original transcript is dated or heavily formatted.  having a small reserve of funds set aside to pay for online transcription services is incredibly helpful in this situation.  while migrating a very early oral history, we discovered that the 50+ page transcript was available only as a pdf and was formatted as a multi-column table with inaccurate time-stamping.  manually removing this formatting and correcting the transcription errors would take at least a full day of work.  however, for less than what we would pay our oral historian, we were able to send these problem files out for re-transcription using an online service, and were returned transcripts that only required a brief review.  this was a cost effective solution that freed up the time of our oral historian so they could continue working on other projects. not to mention the frustration that this saved our staff. finally, as with any migration project, it is important to document everything.  the notes that were recorded while testing out migration techniques in aviary certainly weren’t fit to be shared broadly, but they were vital for the documentation we produced later.  while determining how to both add new content as well as migrate existing content into a platform of this nature, it is not uncommon to write ourselves little “cheat sheets” or notes.  don’t throw these away!  very often these things can quickly and easily be formatted and fleshed out into a manual or migration checklist that can be passed on to others to assist in completing the project. conclusion we are in the early days of our migration of legacy oral history materials to aviary. while we have largely thought through how we want to integrate new aviary materials into both the staff workflows and the discovery and presentation of lcdl, we still have extensive remediation of legacy materials to perform before we can bring them live on aviary and embed them into lcdl.  however, with aviary we now have the flexibility and potential to not only support our existing oral histories in the manner that we prefer, but also the capacity to take on new oral history projects.  we’ve developed manageable workflows to support faculty driven oral history projects and now have the capability to support a dedicated oral history platform.  as such, we are working on branding and launching the lowcountry oral history initiative (lohi) which is essentially a third pillar of the overall ‘lowcountry’ project alongside the lowcountry digital library and the lowcountry digital history initiative. under the banner of lohi we hope to cultivate relationships between the lowcountry digital library and various archival and community organizations to record, preserve, and make available audio and video recordings that document the memories of historically marginalized communities. through aviary and the lowcountry digital library, we can continue to service our partners and community. we look forward to steadily adding new and varied oral history collections to aviary while we ensure that new and legacy cultural heritage oral histories in lcdl are presented with a robust new look and feel. about the authors tyler mobley is the digital services coordinator for the college of charleston libraries. he serves as co-director of the lowcountry digital library and technical director of the south carolina digital library. he holds an mlis from the university of south carolina. his research interests include digital asset management and preservation, digital repository systems development, library information technology management, and digital scholarship. heather gilbert is the associate dean of collection and content services for the college of charleston libraries. she serves as associate director for the coastal region of the south carolina digital library. she holds an mfa from the pennsylvania academy of fine arts and an mlis from the university of south carolina. her research interests include academic content acquisition and management strategies, digital asset management and preservation, digital scholarship, metadata aggregation, and archival/cultural heritage repository management. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – works, expressions, manifestations, items: an ontology mission editorial committee process and structure code4lib issue 53, 2022-05-09 works, expressions, manifestations, items: an ontology the concepts first introduced in the frbr document and known as “wemi” have been employed in situations quite different from the library bibliographic catalog. this is evidence that a definition of similar classes that are more general than those developed for library usage would benefit metadata developers broadly. this article proposes a minimally constrained set of classes and relationships that could form the basis for a useful model of created works. by karen coyle introduction a model for the bibliographic entries in library catalogs was developed in 1998 that included a view of creative endeavors having four levels of abstraction, from the most abstract, “work,” to the actual physical instance, “item.” this model, developed solely within and for the library community, has found adherents among metadata practitioners in other communities. these non-library communities present significantly different use cases both among themselves and against the library use case the model was designed to address. these provide evidence that a set of classes that expresses this model but that is not constrained by the library catalog use case would be useful. each of the application-specific uses, including the one developed by libraries, would be an appropriate sub-class to a more general model describing the nature of created works. this article first introduces the concepts in the library model. it then describes the uses outside of the library area. this is followed by a proposal for a definition of entities with minimum semantic commitment that can be used as superclasses for metadata definitions in any environment describing created works. the frbr-lrm model in 1998, the cataloging division of the international federation of library associations (ifla) issued a document proposing a new conceptual model for the entries in library catalogs. (ifla 1998, ifla 2009) this metadata model, known by its acronym frbr (for functional requirements for bibliographic records), extrapolated entities and relationships from the metadata creation standards used for library catalogs. the model has three groups of entities, named solely as groups 1, 2, and 3. group 2 consisted of persons and groups of persons who have some agency over the cataloged works (such as creators and publishers). group 3 had the entities that represented topical aspects of the resource. group 1 was the most innovative, as it separated the description of the created resource into four levels of abstraction: work, expression, manifestation, and item (wemi). it is this latter set of entities that has found adherence beyond the library use case. figure 1. frbr group 1 (wemi) although the study group that created the frbr model was working exclusively on describing library data, their work revealed a missing concept in the description of created works far beyond the library context. the levels of abstraction from work to item has been seen to be useful for such diverse resources as dataset versions, legal documentation tracking, and fashion industry design flow. while these non-library parties are interested in making use of the wemi entities, those entities reflect the library catalog environment and its particular business rules. this article proposes the development of an unconstrained created work ontology that benefits from frbr’s innovation yet facilitates reuse and extension in situations differing from the original intended library use. the library use case library metadata serves a variety of use cases within the institution. the most mundane is that of an inventory of the owned items. for library users the catalog is the means to discover and locate resources at the level of specificity that serves their need. the multi-level description in frbr’s wemi was derived from important bibliographic relationships that could be of interest to library users. for example, one creative work can exist in multiple editions, translations, or arrangements (expression); one expression may be published in many books or other media (manifestation); and the end point of a search can be obtaining an individual copy owned by the library (item). the frbr model also followed general database criteria with a goal to eliminate any duplication of data elements across the entities. because the library community shares metadata records for items they own in common, the model promises an efficiency for the community at large. (coyle, 2016 p. 76) frbr has been superseded by a revised model called the library reference model (lrm). (riva, 2017) this model provides some significant changes in design and naming but the basic concepts of wemi remain. note that most of the extra-library discussion continues to refer to the model with the acronym “frbr” and some concepts introduced in the lrm are not yet widely discussed. the entities of wemi are defined today in the library reference model as: work: the intellectual or artistic content of a distinct creation. (riva, 2017 p.21) expression: a distinct combination of signs conveying intellectual or artistic content.(riva, 2017 p.23) manifestation: a set of all carriers that are assumed to share the same characteristics as to intellectual or artistic content and aspects of physical form. that set is defined by both the overall content and the production plan for its carrier or carriers. (riva, 2017 p.25) item: an object or objects carrying signs intended to convey intellectual or artistic content. (riva, 2017 p.27) each entity has specific relationships with the near entities, and the inverse relationship is assumed (riva, 2017 p.65-66): work is realized by expression expression is embodied in manifestation manifestation is exemplified by item in the library model, the relationships between the wemi entities are disjoint from each other and are fixed in order. being disjoint, the entities cannot share data elements if rdf reasoning will be applied. this is mainly a problem with sharing between metadata schemas that have distributed their data elements differently, such as the bibframe three-level model and the resource description and access (rda) adherence to the four-level models of frbr or lrm.(baker, 2014) in the frbr model all intervening entities are required: the only route from the manifestation to the work is through the link between manifestation and expression and between expression and work. (shotton, 2019) the relationships are therefore immutable and all “higher” entities must be present for a “lower” entity to be described. part 2: uses of wemi outside of libraries there is ample evidence that metadata designers outside of the library data silo are drawn to the concepts first introduced in the frbr report of 1998, and in particular they are drawn to the wemi entities. these entities provide a defined model of created “things” that acknowledges the abstract planes that we interact with separately from the physical embodiments. the examples that follow are a selection of uses of frbr in non-library applications. frbrcore at vocab.org when frbr was issued in 1998, there was no expression of the model in any information technology schema or code. this was in keeping with the declaration that frbr was a conceptual model that was not being introduced as a bibliographic record schema. this left open, however, the need for systems development using the frbr conceptual model. in 2004, library systems developers ian davis and richard newman created frbr core (davis, 2005) (revised in 2006), an rdf vocabulary encompassing all of the frbr entities and the frbr-defined relationships between them (e.g. “revisionof“, “expressionof“, “ownerof“). frbr core did not, however, follow precisely the description of frbr as given in the 1998 document. it added a top-level class, endeavor, to which the four entities of wemi are sub-classed which was not included in the frbr model. it also added a top-level class, responsibleentity, for the two entities of the second group, which are corporate body and person. (dunsire, 2008) the frbr document defined specific bibliographic relationships between entities, such as “successor” and “reproductionof,” which were also included in frbr core. frbr core follows the constraints on frbr entity relationships in that it declares each of the entities other than subject to be disjoint of all of the other entities. it also maintained the linear relationships of the entities, defining them as rdf classes in which each entity has only the relationships that maintain the wemi entities in a single order. frbr core is “core” because it consists only of the frbr entities and the relationships between them. it does not include properties for the attributes that frbr defined for the entities, such as title of the work, or publisher of the manifestation. it therefore leaves room for other vocabulary developers to describe the entities through the attributes best suited to their use case. frbr core is important to the story here because in the absence of a metadata schema definition of frbr or the lrm, frbr core has been used in metadata vocabularies implementing their own frbr-based metadata for non-library objects. frbr core was not, however, adopted by the ifla working group as an approved expression of frbr. this has created a fork in implementation between the community adhering to the ifla model and those outside of the library standards environment. fabio: semantic publishing and referencing one of the uses of frbr core is in the semantic publishing and referencing (spar) ontologies. (peroni, 2018) these ontologies define metadata for the description of documents and the publication workflow in the scholarly publishing domain, and provide a rich vocabulary for  citations, references, and agents. within that suite of ontologies is the frbr-aligned bibliographic ontology, fabio (peroni, 2012) (shotton, 2019). fabio defines sub-classes of the frbr core entities and gives them definitions with scholarly publishing semantics. for example, fabio work is defined as: fabio:work – a subclass of frbr work, restricted to works that are published or potentially publishable, and that contain or are referred to by bibliographic references, or entities used to define bibliographic references. fabio works, and their expressions and manifestations, are primarily textual publications such as books, magazines, newspapers and journals, and items of their content. however, they also include datasets, computer algorithms, experimental protocols, formal specifications and vocabularies, legal records, governmental papers, technical and commercial reports and similar publications, and also bibliographies, reference lists, library catalogues and similar collections. for this reason, fabio:work is not an equivalent class to frbr:scholarlywork. an example of a fabio:work is your latest research paper. the definition of the superclass to fabio work in frbr core is: “an abstract notion of an artistic or intellectual creation.”  the fabio classes are a reuse of the frbr concepts with narrower semantics. (peroni, 2017) the classes of wemi are further subclassed in fabio to describe the nature of the entity, such as: work subclasses: biography, critical essay, dataset expression subclasses: article, conference poster, instruction manual manifestation subclasses: analog manifestation, digital manifestation item subclasses: analog item, digital item by declaring a particular entity to be of the class fabio:biography using the facilities of rdf class relationships, one has also declared that entity to be a fabio:work and a frbrcore:work through the class relationships. fabio also expands the model provided in frbr core by allowing additional relationships between the wemi entities such as from work directly to item without passing through the intermediate entities of expression and manifestation. figure 2. fabio class relationships (http://purl.org/spar/fabio/) functional requirements for information resources (frir) members of the open government data community have proposed a frbr-based data model called frir (for functional requirements for information resources) for datasets. this also is based on the frbr core vocabulary. (mccusker, 2012) in this model, the use of abstraction, expressed information, and physical sources is needed to allow members of the community to combine data from different sources and to determine at what level the sources represent the same information. the authors apply the four wemi levels (which they refer to as the “four frbr levels of abstraction”) to digital information resources. they consider exact copies of files (the same bitstream) to be items of the same manifestation. different file formats with the same content, such as a csv file and an excel file, are different manifestations of the same expression. different expressions contain the same informational content but the files may differ in having more or less content, yet they are expressions of the same work because they express the same basic data. datasets that are derived from expressions but that do not contain the same information content are considered different works. wemi class designation for the datasets in the frir model can be determined computationally using algorithms that derive from the cryptographic digests of the datasets. as applied, these algorithms bring together files at the appropriate level of abstraction, logically building the wemi relationships from previously unlinked files. figure 3. frir (mccusker, 2012 p.4) the frir uses the frbr core vocabulary but not in any bibliographic sense. it does make use of some of the relationship properties such as successor, revision, and reproduction. although frbr was developed for bibliographic works, and is specifically appropriate to published materials, frir shows that it is possible to interpret wemi as representing relationships for very different kinds of information. legal documentation the use of wemi for legal documentation arises from the need in that community to keep a strict account of versions of documents as they develop in the juridical and legislative processes, and to make these available in a consistent way across jurisdictions. (boer 2002) the first to propose the use of wemi for legal documentation was joao alberto de oliveira lima, of the brazilian senate. (lima, 2006) he gives the example below which illustrates how the wemi structure can be used with legal documentation. the work in this example has two expressions: the original text and the amended text. the original text is published as a manifestation in the federal record and on the related senate website on subsequent days. the amended text is published in the record of the legislature one day later. w1 – “act n± 9 691, 07/22/1998 [creator = national congress]” e1:1 – “original text” m1:1:1 – “official publication in the federal journal, 07/23/1998” i1:1:1:1 – exemplar on paper in the national library i1:1:1:2 – exemplar on paper in the library of the federal senate m1:1:2 – publication in the federal senate web site, 07/24/1998” i1:1:2:1 – <http://www6.senado.gov.brlegislacao/ listatexto.action?id=127883> e1:2 – “rectifying text” m1:2:1 – “official publication in the federal journal, 07/24/1998” i1:2:1:1 – exemplar on paper in the national library figure 4. wemi relationships in legal documents this example shows how wemi can be used to provide a map of the history of document versions over time. (lima, 2008) actual legislative workflows can take place over months or years and can iterate through committees and legislatures multiple times while the interested parties negotiate amendments. a wemi analysis can be used to organize the version history of a single piece of legislation from first proposal to law. legal documents using wemi in this way are formatted in xml. one such xml schema is metalex (boer, 2008), another is the akoma ntoso schema (palmirani, 2008) (vitali, 2008) which is a standard under the oasis legaldocml technical committee. in the akoma ntoso schema, wemi metadata is embedded in the xml document itself, in a section called “identification.” the identification section creates a standardized description of the document. the entities are defined with data element names beginning with “frbr,” such as frbrtitle, frbrauthor. the akoma ntoso schema defines elements that are not found in the original frbr nor the lrm vocabularies; these support the xml structure of the akoma ntoso-coded document. for example, the identification section is linked to the legal document and its versions with the element frbrthis (which identifies the document to which the element relates). there are some properties, such as frbrdate and frbruri, that are common to each wemi entity, while other properties, such as frbrcountry and frbrfomat, are defined only for the appropriate entities. also note that in the xml schema nesting of entities obviates the need for specific relationship properties between the entities. </frbrwork> <frbrexpression> <frbrthis value="/akn/us/act/2011-11-29/112-61/eng@/!main"/> <frbruri value="/akn/us/act/2011-11-29/112-61/eng@"/> <frbrdate date="2012-05-09" name="generation"/> <frbrauthor href="#palmirani" as="#editor"/> <frbrlanguage language="eng"/> </frbrexpression> <frbrmanifestation> <frbrthis value="/akn/us/act/2011-11-29/112-61/eng@/!main.xml"/> <frbruri value="/akn/us/act/2011-11-29/112-61/eng@.akn"/> <frbrdate date="2012-05-09" name="generation"/> <frbrauthor href="#palmirani" as="#editor"/><preservation <mods:physicaldescription xmlns:exslt="http://exslt.org/common"> <mods:note type="source content type">deposited</mods:note> <mods:digitalorigin>born digital</mods:digitalorigin> <mods:extent>3 p.</mods:extent> </mods:physicaldescription> </preservation> <frbrformat value="xml"/> </frbrmanifestation> figure 5. snippet of akoma ntoso code the legal community’s use of wemi points out a concept that was missing from the original frbr but that has been added to the model in the lrm: the ability to treat time-based families of created works. the legal community has a need to include time as a central component of its analysis, and as such has turned to the use of time in the museum community’s model, frbroo, (bekiari, 2017) to satisfy this need. (lima, 2008) fast fashion design industry two ontologists engaged by the “fast-fashion design industries” to develop a metadata model and who were familiar with frbr found that they could apply concepts based on wemi to the work-flow of the imperial design company. there was the initial creative design effort (work) followed by a sketch of the design idea (expression), then a concrete manufactured product (manifestation) with individual sales units (items). (peroni, 2017) the vocabulary makes use of an owl2 dl implementation of the frbr core (ciccarese, 2018) to create a four-level description of the creation as defined in the fast fashion industry. in the place of work, expression, manifestation and item, the wemi classes for this industry were: style, item, stock-keeping unit and piece. (peroni, 2017) part 3: a new model these implementations for the use of wemi concepts for datasets, manufacturing, and non-library bibliographic data illustrate the utility of these concepts outside of the specific library cataloging context in which they originated. why a new model is needed given that wemi is defined in a standard issued by a professional body, it may not be clear why one should not simply use the entities as they have been defined there. the use cases above show that the specifics of the model designed for library catalog metadata vary significantly from the needs of other communities even though those communities would like to describe their created works with levels of abstraction analogous to wemi. although there is some room for interpretation in the library model, the library catalog’s business rules are built into the frbr/lrm standards by design. for example, the library model includes the primary bibliographic attributes of each entity. some of these are general in nature (“physical extent”) while others are specific to the descriptions produced for library catalogs (“script conversion”). the library use case also requires significant uniformity between metadata instances created separately in thousands of institutions of various types because libraries make use of commercial software packages that depend on adherence to the data standard. frbr and the lrm thus can be seen as conceptual models for application profiles that facilitate the use of library community-specific software. it would be unlikely to find another community that has the same needs. what follows is a proposal for a less constrained model and vocabulary for created works that responds to the demonstrated interest in a wemi-like set of classes for descriptive metadata. the proposed model will not be bound by library cataloging practices nor constrained by those business rules. because the goal of this new model is to support a wide range of views and uses, this proposal implements a theory of ontology development known as minimal ontological commitment. (gruber, 1993) in this approach, base vocabulary terms should be defined with as little built-in semantics as possible, with semantics in this sense being the axiomatic semantics of rdf/owl as well as meaningful semantics in the sense of human understanding. an ontology whose terms have high semantic definition, such as the library frbr/lrm model, will provide fewer opportunities for re-use because usage must adhere to the defined semantics in the original ontology. less semantic commitment in the base ontology means that desired semantics can be defined in specific implementations through the sub-classing of entities and definition of additional entities as needed. the only constraints on this model will be those that maintain necessary coherence between the entity definitions. it is important to note here that minimum semantics also means that the various community-supported vocabularies that derive from these terms may not be interoperable between them when their community-specific constraints are applied. like owl:thing, the created work entities are intended primarily to provide a foundation for more specific terms that are derived from them. proposal: a minimally defined wemi ontology to begin, we need a new name for this model since the commonly used acronym, wemi, may be too directly related to frbr and the lrm. the new ontology is neither the expression of a functional requirement (fr) nor is it directly related to bibliographic records (br). we also need to avoid confusion as people already refer to different models as “frbr” and often refer to the wemi entities as frbr when they are only a part of that model. unlike the library use case of modeling recorded knowledge, the proposed model is a model for all created works, that is, any agent-created “thing.” for the remainder of this document i will refer to the created work ontology, or cwo, but other names have been suggested, such as openwemi. modeling the ontology i propose that the cwo be initiated with the four entities first developed in frbr group 1, as these have been shown to have utility, although a community may decide that there is a need for other entities that describe aspects of the creation. the entities could have the following definitions: work: the essence of the created thing expression: a sign or series of signs that signify the created thing manifestation: the physical characteristics of a realization of the created thing item: an instance of the created thing in time and space another way is to view the entities as occupying general facets or planes: work: the idea or concept plane (“has concept/is concept”) expression: the plane of signs or signifiers (“has signs/is signs”) manifestation: the physical plane (“has physicality/is physical”) item: the location plane (“has location/is located”) i don’t recommend defining attributes for the entities at this level. it is inevitable that different communities will make different decisions in where they place specific metadata elements across the wemi planes. further experience with the model may surface universals for the entities, but it would be limiting to define attributes prematurely. as communities define their metadata with classes that are sub-classed to cwo, the properties that they create will serve to further explain the meaning of the created work in their environment. the relationships in cwo wemi is conceptually a technology stack with relationships between the members of the stack. whether the stack is organized from item to work, or work to item is a matter of a point of view. however, it is similar to a technology stack in that the order of the elements of the stack is fixed, and that order determines the possible relationships between the entities. this statement about the order is true even though not all elements of the stack are required: entities must retain their position in the stack to be semantically valid. a metadata description of a created resource does not require the existence of the full wemi stack, but it does require adherence to the logical order of the entities. the formal rules for cwo therefore need to maintain the wemi stack while allowing for flexibility. the relationships can be between any more abstract levels in the stack to those more concrete. this reads as: work is expressed in expression / expression expresses work is manifested in manifestation / manifestation manifests work is instantiated in item / item instantiates work expression is manifested in manifestation / manifestation manifests expression is instantiated in item / item instantiates expression manifestation is instantiated in item / item instantiates manifestation there are various reasons for not requiring the presence of every entity in the full stack for valid adherence to the model including the lack of omniscience on the part of metadata creators and the potential lack of completeness of a set of metadata at a given position in the data creation workflow. a person working on the description of an individual item may not know at the time whether it is one of a set of identical products that could be considered manifestations, nor may be in possession of the information about who created the work or even what the work represents. thus, at a particular moment in time, one may be creating metadata for an item that is orphaned from the rest of the wemi stack. knowledge is a process of constant building on prior knowledge, and even with a full wemi description one should be aware that new entries into the graph result in different knowledge about the creation. this is analogous to the open world assumption of rdf, which allows for the absence of information without negating the truthfulness of the information that is present. the cwo can be used for more than metadata creation. as an open model with few restraints it can be used to create the variety of views that meet the needs of the users that a community wishes to serve. a view may not be the same as stored metadata but may be a selection from a larger dataset. a separate work and expression in one application could be a single view in another. at this point it is also important to say that the definition of the entities does not entail that metadata making use of these concepts must describe each separately as structural elements of the metadata schema.these entities are not defined as disjoint, as they are in frbr. disjointness is a restriction that would limit the ways that the entities can be defined and whether they can share attributes. it also makes it difficult to combine metadata using different entity definitions, such as the difference between bibframe’s work-instance-item and rda’s work-expression-manifestation-item. to require all graphs to have a separation between entities would mean that the wemi model could not be used in concert with non-wemi metadata. (baker, 2014) such a requirement would create a data silo for any communities using wemi. having flexibility, within the necessary ontological commitment, is compatible with the need to share some metadata across communities as well as to facilitate ongoing evolution of information resources. there is also the possibility of creating other types of relationships between entities, such as work/work or expression/expression relationships. an interesting approach to this was taken by ross singer when he defined generic relationships for frbr core entities that could be used between any defined entities: commonendeavor, commonwork, commonexpression, etc. these properties are in a dormant area of the vocab.org namespace and perhaps should be reconsidered as part of the cwo effort. cwo in owl appended to this article is a first pass at an encoding of cwo in owl. it is also available at https://github.com/kcoyle/openwemi. it is expected that this is just a demonstration of the concept, and that, if deemed useful, will be modified and housed at a more stable namespace. conclusion perhaps inadvertently the library metadata community has developed a set of classes that resonate for metadata developers in a wide range of topic areas. this non-library use of the frbr concepts has created a fork of the group 1 classes of frbr, primarily through the use of frbrcore. these applications have, however, inherited some of the built-in constraints of frbr that were intended for library catalog metadata, and which could lead to unexpected difficulties if inferencing is applied to their metadata. a set of less-constrained classes that are not specific to the library data model should mitigate this problem and also allow for more flexibility in the use of the wemi concepts. if developers are able to sub-class their created work views to the cwo, we could see more communities embracing a multi-level view for their descriptive metadata. about the author karen coyle has been modeling metadata since 1980, and now interacts with dublin core and w3c in metadata standards development. bibliography baker t, coyle k, petiya s. 2014. multi-entity models of resource description in the semantic web: a comparison of frbr, rda and bibframe. library hi tech 32(4): 562-582. https://doi.org/10.1108/lht-08-2014-0081. bekiari c, doerr m, le bœuf p, riva p. 2017. frbr object-oriented definition and mapping from frbrer, frad and frsad. version 3.0 international working group on frbr and cidoc crm harmonisation. september 2017. https://www.cidoc-crm.org/frbroo/sites/default/files/frbroo_v3.0.pdf. boer a, hoekstra r, winkels r. 2002. metalex: legislation in xml. in: t. bench-capon, a. daskalopulu and r.winkels (eds.). legal knowledge and information systems. jurix 2002: the fifteenth annual conference. amsterdam: ios press, 2002, p. 1-10. boer a, winkels r, vitali f. 2008. metalex xml and the legal knowledge interchange format. in: casanovas p, sartor g, casellas n, rubino r. (eds) computable models of the law. lecture notes in computer science, vol 4884. springer, berlin, heidelberg. ciccarese p, peroni s. 2018. essential frbr in owl2 dl. version 1.0.1. http://purl.org/spar/frbr. accessed december 14, 2019. coyle k. 2016. frbr, before and after: a look at our bibliographic models. chicago : ala editions, an imprint of the american library association. davis i, newman, r. 2005 expression of core frbr concepts in rdf. http://vocab.org/frbr/core.html. accessed december 16, 2019. dunsire g. 2008. declaring frbr entities and relationships in rdf. https://www.ifla.org/files/assets/cataloguing/frbrrg/namespace-report.pdf. accessed november 18, 2019. gruber tr. 1993. toward principles for the design of ontologies used for knowledge sharing. in: international journal human-computer studies 43. p.907-928. international federation of library associations and institutions. 1998. functional requirements for bibliographic records: final report. münchen: saur. ifla study group on the functional requirements for bibliographic records. 2009. functional requirements for bibliographic records. den haag. http://archive.ifla.org/vii/s13/frbr/frbr_2008.pdf. lima jao. 2006. an adaptation of the frbr model to legal norms. in: biagioli c, francesconi e, sartor g. (eds.) proceedings of the v legislative xml workshop, 2006 european press academic publishing, italia 2007. lima jao, palmirani m, vitali f. 2008. moving in the time: an ontology for identifying legal resources. in: casanovas p., sartor g., casellas n., rubino r. (eds) computable models of the law. lecture notes in computer science, vol 4884. springer, berlin, heidelberg. mccusker, jp, lebo, t, chang c, mcguinness, dl, da silva, pp. 2012. parallel identities for managing open government data. ieee intelligent systems, 27(3), 55. http://tw.rpi.edu/media/2012/02/07/d641/ex_issi-2011-09-0138.r1_mccusker.pdf. palmirani, m., et al. 2018. akoma ntoso version 1.0 part 1: xml vocabulary. 29 august 2018. oasis standard. http://docs.oasis-open.org/legaldocml/akn-core/v1.0/akn-core-v1.0-part1-vocabulary.html. peroni s, shotton d. 2012. fabio and cito: ontologies for describing bibliographic resources and citations. in journal of web semantics, 17: 33-43. https://doi.org/10.1016/j.websem.2012.08.001. open access at: http://speroni.web.cs.unibo.it/publications/peroni-2012-fabio-cito-ontologies.pdf. peroni s, shotton d. 2018. the spar ontologies. in: vrande?i? d. et al. (eds) the semantic web – iswc 2018. iswc 2018. lecture notes in computer science, vol 11137. springer. peroni s, vitali f. 2017. interfacing fast-fashion design industries with semantic web technologies. the case of imperial fashion. journal of web semantics. 44:37-53. https://doi.org/10.1016/j.websem.2017.06.001. riva p, le boeuf p, žumer m. 2017. ifla library reference model: a conceptual model for bibliographic information. den haag, ifla. https://www.ifla.org/wp-content/uploads/2019/05/assets/cataloguing/frbr-lrm/ifla-lrm-august-2017_rev201712.pdf. shotton, d, peroni, s. 2019. fabio, the frbr-aligned bibliographic ontology. v 2.1 https://sparontologies.github.io/fabio/current/fabio.html. accessed december 26, 2019. vitali, f, palmirani, m, lima, jao. “moving in the time: an ontology for identifying legal resources ” in: casanovas p, sartor g, casellas n, rubino r. (eds) computable models of the law. lecture notes in computer science, vol 4884. springer, berlin, heidelberg, 2008. appendix: openwemi.ttl @prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> . @prefix owl: <http://www.w3.org/2002/07/owl#> . @prefix skos: <http://www.w3.org/2004/02/skos/core#> . <http://example.org/openwemi/> a owl:ontology ; rdfs:label "openwemi ontology" . <http://example.org/openwemi/endeavor> a owl:class ; rdfs:label "endeavour"@en ; skos:definition "the conceptual or intellectual aspect of a creation."@en ; rdfs:isdefinedby <http://example.org/openwemi/> . <http://example.org/openwemi/work> a owl:class ; rdfs:label "work"@en ; skos:definition "an abstract notion of an artistic or intellectual creation."@en ; rdfs:isdefinedby <http://example.org/openwemi/> ; rdfs:subclassof <http://example.org/openwemi/endeavor> . <http://example.org/openwemi/expression> a owl:class ; rdfs:label "expression"@en ; skos:definition "an expression of a work in signs."@en ; rdfs:isdefinedby <http://example.org/openwemi/> ; rdfs:subclassof <http://example.org/openwemi/endeavor> . <http://example.org/openwemi/manifestation> a owl:class ; rdfs:label "manifestation"@en ; skos:definition "the physical embodiment of one or more expressions."@en ; rdfs:isdefinedby <http://example.org/openwemi/> ; rdfs:subclassof <http://example.org/openwemi/endeavor> . <http://example.org/openwemi/item> a owl:class ; rdfs:label "item"@en ; skos:definition "an exemplar of a single manifestation."@en ; rdfs:isdefinedby <http://example.org/openwemi/> ; rdfs:subclassof <http://example.org/openwemi/endeavor> . <http://example.org/openwemi/responsibleentity> a owl:class ; rdfs:label "responsible entity"@en ; skos:definition "one responsible for the creation, production, distribution or maintenance of a created entity."@en . <http://example.org/openwemi/relatedendeavor> a rdfs:property ; rdfs:label "related endeavor"@en ; skos:definition "another endeavor that is related in some way to an endeavor."@en ; rdfs:isdefinedby <http://example.org/openwemi/> . <http://example.org/openwemi/expresses> a owl:objectproperty ; rdfs:label "expresses"@en ; skos:definition "an endeavor that expresses a work."@en ; rdfs:isdefinedby <http://example.org/openwemi/> ; rdfs:subpropertyof <http://example.org/openwemi/relatedendeavor> ; rdfs:domain <http://example.org/openwemi/expression> ; rdfs:range <http://example.org/openwemi/work> . <http://example.org/openwemi/manifests> a owl:objectproperty ; rdfs:label "expresses"@en ; skos:definition "an endeavor that manifests an expression or a work."@en ; rdfs:isdefinedby <http://example.org/openwemi/> ; rdfs:subpropertyof <http://example.org/openwemi/relatedendeavor> ; rdfs:domain <http://example.org/openwemi/manifestation> ; rdfs:range [ a owl:class ; owl:unionof ( <http://example.org/openwemi/work> <http://example.org/openwemi/expression> ) ] . <http://example.org/openwemi/instantiates> a owl:objectproperty ; rdfs:label "expresses"@en ; skos:definition "an endeavor that instantiates a manifestation, an expression or a work."@en ; rdfs:isdefinedby <http://example.org/openwemi/> ; rdfs:subpropertyof <http://example.org/openwemi/relatedendeavor> ; rdfs:domain <http://example.org/openwemi/item> ; rdfs:range [ a owl:class ; owl:unionof ( <http://example.org/openwemi/work> <http://example.org/openwemi/expression> <http://example.org/openwemi/manifestation> ) ] . subscribe to comments: for this article | for all articles one response to "works, expressions, manifestations, items: an ontology" please leave a response below: basma chebani, 2025-02-21 dear karen coyle, i enjoyed reading your article, but you didn’t mention the rda registry that uses the wemi model (lrm) in relationships as uri in marc records and elsewhere. i am heavily using these rda wemi in making the relationships between works and manifestations and expressions in the library catalog in american university of beirut libraries as a step ahead to linked open data marc enrichment. do you think what i am making since 2013 till now is obsolete and i need to use open wemi instead? . please advise. chebani.basma@gmail.com leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – editorial introduction: on libraries, code, support, inspiration, and collaboration mission editorial committee process and structure code4lib issue 25, 2014-07-21 editorial introduction: on libraries, code, support, inspiration, and collaboration reflections on the occasion of the 25th issue of the code4lib journal: sustaining a community for support, inspiration, and collaboration at the intersection of libraries and information technology. by dan scott this issue marks the 25th issue of the code4lib journal since it launched in 2007. at that time, jonathan rochkind (co-ordinating editor of the first issue), wrote “we want the immediacy of a blog, the usefulness of a professional conference, the reliable quality of a good scholarly journal, and the participatory nature of our online communities, all in one easy to read and easy to produce package.” we’re still running that experiment seven years later, with different co-ordinating editors each issue, and i’m delighted that this has been my opportunity to shepherd our community’s efforts… but all the credit goes to the authors for volunteering to take the time to share their findings with the community, and to the editors for helping refine those words and ideas for publication. reflections on an initial code4lib encounter i hope you will indulge the following reflections on the code4lib journal, its relationship to the code4lib community, and my own experiences at the intersection of information technology and libraries as we celebrate this 25th issue. code4lib as support group my own introduction to the code4lib community occurred in late 2006, the year i transitioned from a corporate software development environment to my current position as a systems librarian at laurentian university. facing a proprietary library system, highly restricted access to our servers, marc, and the deadly combination of a limited budget and unlimited expectations, within a few months i was very seriously considering a quick exit from this career choice. thankfully, two things happened: my colleague lise doucette recommended the access conference as an appropriate destination for a new systems librarian. i took her up on the suggestion and engaged in my first hackfest, which was organized by dan chudnov and ross singer, and attended by the likes of patricia williams, art rhyno, bess sadler, terry reese, birkin diana, and other ur-members of the code4lib community. this hackfest was immortalized in a one big library podcast episode. around the same time, art pointed me at the #code4lib irc channel, which i found filled with smart people commiserating (in a good-natured way) around the constraints of library standards and proprietary technology. yet these people were infused with a belief in the importance of libraries, and determined to overcome those barriers to achieve a greater good. no, really! those two occurrences gave me hope for the future of libraries, and helped me stick with my career decision. eight years later, when i’m frustrated or in a state of despair due to some hurdle or setback, the #code4lib channel still serves as a kind of support group: there are inevitably people there who “get it”. in her issue 5 editorial, emily lynema succinctly stated that the existence of the code4lib community offers the reassurance that “librarians working with technology are not working alone.” code4lib journal articles also support that goal: they let information technologists in libraries know that no matter how geographically or institutionally isolated they might feel, there are others who understand and tackle similar challenges, and who are sharing their findings in the hopes that their peers may benefit and build upon their work. if you look closely, written between the lines of these articles you may find the words there is hope for libraries, if you persevere! code4lib as inspiration over the span of a day at that hackfest in 2006, my teammates heather matheson, holly eggleston, scott nickerson and i built a proof-of-concept collection analysis tool named cartman. it was an incredibly refreshing exercise in liberating yourself from institutional and technical constraints and just working with really smart people to get things done. and yes, of course we decided on the acronym first and then figured out how to make the name fit. that sassiness and creative flair was an important part of the code4lib community spirit as well: we don’t just get things done, we get things done with style and attitude. through the seven years that the code4lib journal has been in existence, i suspect that many articles have inspired others to adopt new technologies and approaches that they would not otherwise have felt capable of mastering, or to further their own work. to share a personal example, issue 16 containing jason ronallo’s “html5 microdata and schema.org” article was published at the same time that my own baby steps with schema.org were incorporated into evergreen. jason’s article gave me a much broader insight into the possibilities for schema.org and provided some validation for my efforts (“no, i’m not entirely crazy pursuing this!”), and i have spent the bulk of my research time over the subsequent two years working in the domain. code4lib as collaboration returning to my reminiscence about access hackfest 2006: many of the participants would not have self-identified as programmers; they had signed up the hackfest because they could bring their skills and insight into metadata, or collection development, or public services requirements to the discussion and collaborate on building a project that was much richer for their participation. somehow the organizers and promoters of that hackfest were able to communicate that the event would be a safe space in which true collaboration and communication could occur, and teams like mine benefited from the wonderfully diverse perspectives and knowledge. previous editorials such as kelley mcgrath’s editorial introduction – a cataloger’s perspective on the code4lib journal and ron peterson’s the code4lib journal isn’t just for coders have emphasized a similar point: the code4lib journal, and the community in general, is a forum for communication by all those involved and interested in the future of libraries. overview of issue 25 this issue offers another broad set of articles that reflect the diversity of the code4lib community, from the “getting things done efficiently” approach of kristina m. spurgin’s “getting what we paid for: a script to verify full access to e-resources“, to an introduction to cutting-edge containerization technology in john fink’s “docker: a software as a service, operating system-level virtualization framework“, to terry reese’s hands-on review of oclc’s metadata and holdings apis “opening the door: a first look at the oclc worldcat metadata api“. digitization efforts continue to impact our work. kyle rimkus and kirk hess share their experiences wrangling jpeg2000 images into compliance in “hathitrust ingest of locally managed content: a case study from the university of illinois at urbana-champaign“. pieter de praetere tells the story of how he overcame it policy constraints to improve the efficiency of the scanning workflow at the provincial library of west-vlaanderen in “within limits: mass-digitization from scratch“. and matt weaver reports on his the community cookbook project to digitize local cookbooks of historical value in “ebooks without vendors: using open source software to create and share meaningful ebook collections“. data and gis librarians also have something to look forward to in this issue, with some core geoblacklight design considerations highlighted in darren hardy and kim durante’s “a metadata schema for geospatial resource discovery use cases“, and frank donnelly’s case study “processing government data: zip codes, python, and openrefine” providing a hands-on tutorial in massaging datasets to produce added value for their users. for the forensics-oriented librarians, archivists, and curators in the audience, we are delighted to offer misra, lee, and woods’ “a web service for file-level access to disk images“. while our community is generally familiar with the mysql relational database and the solr full-text search engine, arie nugraha suggests it might be worth looking at some alternatives in “indexing bibliographic database content using mariadb and sphinx search server“. finally, josh rompf brings some practical recipes for wrangling media transcoding, concatenation, and scripting challenges in “solving advanced encoding problems with ffmpeg“. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – developing a multi-portal digital library system: a case study of the new university of florida digital collections mission editorial committee process and structure code4lib issue 58, 2023-12-04 developing a multi-portal digital library system: a case study of the new university of florida digital collections the university of florida (uf) launched the uf digital collections in 2006. since this time, the system has grown to over 18 million pages of content. the locally developed digital library system consisted of an integrated public frontend interface and a production backend. as with other monoliths, being able to adapt and make changes to the system became increasingly difficult as time went on and the size of the collections grew. as production processes changed, the system was modified to make improvements on the backend, but the public interface became dated and increasingly not mobile responsive. a decision was made to develop a new system, starting with decoupling the public interface from the production system. this article will examine our experience in rearchitecting our digital library system and deploying our new multi-portal, public-facing system. after an environmental scan of digital library technologies, it was decided to not use a current open-source digital library system. a relatively new programming team, who were new to the library ecosystem, allowed us to rethink many of our existing assumptions and provided new insights and development opportunities. using technologies that include python, apis, elasticsearch, reactjs, postgresql, and more, has allowed us to build a flexible and adaptable system that allows us to hire developers in the future who may not have experience building digital library systems. by todd digby, cliff richmond, dustin durden, and julio munoz introduction as with the integrated library systems, where some libraries are moving through their third or fourth generation of ils’s, we are now starting to see these generational changes to our digital library and institutional repository systems. generally, our migrations to new digital library systems are not as seamless and straightforward as our cousin ils system. the university of florida’s digital library system, first launched in 2006, presented additional complexity in our ability to easily migrate to a new modern digital library system. the system was locally developed and was not based on more common digital library systems, like dspace or islandora. with the system architected using an aging microsoft sql/iis framework, our ability to adapt and make changes to the system became challenging, especially since the lead developer had left uf and our ability to find developers who were fluent in these technologies became increasingly difficult. a turnover of programmers and developers happened in 2020, which jump-started our decision to develop a new system using more commonly used development and technology platforms. this article will examine our experience in re-architecting our digital library system and deploying our new multi-portal, public-facing system. historical background the university of florida digital collections (ufdc) currently hosts local and international collections, housing over 18 million files of content, including all material types, including books, archival documents, newspapers, photographs, audio, video, museum objects, data sets, and maps, that have been published in many languages. during the 1990s, the uf libraries began experimenting with digitization of print materials for the purposes of preservation and broader access to collections. in 1999, the uf libraries formalized ongoing support for this digitization effort by creating the digital library center. these digitization efforts were further expanded with the creation of the university of florida digital collections (ufdc) system in 2006. the system was powered by the open-source sobekcm system (http://sobekrepository.org/sobekcm), which was originally developed at the university of florida, and is run on microsoft web and database-based servers. as the system developed, it grew in complexity to accommodate the unique characteristics of these various types of digital content and digitization workflows. the sobekcm system also powered the digital library of the caribbean (dloc, https://dloc.com), which the uf libraries are one of the founding partners and the technical host partner of. started in 2004, dloc has grown to include 70 partners and associate partners in the united states, the caribbean, canada, and europe. like many systems developed during this time period the sobekcm system is structured to power both the back-end production work, as well as serve as the public-serving front-end. over the past decade and a half, there have been minor incremental system updates, but the interface essentially stagnated. the system served the library and the users well for the past decade, but the age of it is showing and change was needed. need for change besides the system being developed using a technological framework that was over a decade and a half old, there were additional factors that led to the decision to undertake a major effort to re-architect and modernize the ufdc system. as the system was internally developed and not based upon other open-source digital library systems, having programmers that were knowledgeable in how the system was developed was key to being able to respond in a timely manner to resolve issues or in the development of new features. since the original developer of the system departed uf a number of years ago, it has become increasingly difficult for new team members to be able to understand the nuances of the system. additionally, since the system was based within the microsoft realm, including ms-sql, iis, and mostly written in c#, it became difficult to find individuals who have the required skill sets. configured to run on older windows server versions, there became an increasing concern that a future server os version may render the sobekcm system inoperable. with our server infrastructure hosted by campus it, we were not in a position to potentially be open to security vulnerabilities by having the system remain on a version no longer supported by microsoft. in recent years the instability of the system became more and more evident with more frequent interruptions. with both the digital production or back-end tools and the public-facing system on the same instance, if one system was having an issue and needed to be restarted, it would impact the entire system, including the staff processing and making additions to the system. another impetus that impacted our decision was hearing from granting agencies who were reluctant to allocated grant funds to a system that looked dated and was not fully mobile friendly and accessible for users today. decisions – build versus buy/implement open source we knew we needed to move to or develop a new digital library system, but we also needed to decide if we were going to develop the system on our own or were going to implement and then migrate our existing digital collection to one of the many systems available across the digital library system landscape of both commercial and open-sources systems. since the system was developed internally, it continued to change and accommodate new formats and varieties of digital content as they became available, unlike many other academic institutions which implemented systems that were designed around the type of materials being housed. for instance, libraries have historically implemented different digital library systems for collections like newspapers and photos, or act as their institutional repositories, which could respond to the unique search, metadata, and file types needed for those types of materials. an examination of the currently available systems led to the conclusion that it may be necessary to break up our content into unique systems or we may lose some of the customized interfaces that we had developed, such as our digital maps and aerial imagery collections where a customized coordinate-based map search system was implemented. additionally, with two major front-end discovery views (ufdc and the digital library of the caribbean) we needed a system that could replicate our existing system. with no clear available system, and a priority of library leadership to keep the collections housed within a single system, the decision was made to once again develop the system ourselves. the decision was also made to develop the new system and technological infrastructure by using open-source components and open, standards-based protocols and formats. designing the system before deciding on an architecture for our updated system, we needed to map out how we were going to stage our development efforts and what part of the system we should concentrate on first. working within the constraints of needing to keep our existing system fully available and also not stopping the ingestion of materials. at this time, we were ingesting large grant-funded digitized collections that could not be postponed because of the deliverables that were agreed upon as part of the various grants we were involved in. knowing that we had to keep our back-end system available for loading materials into the system, it was decided that we would first build our front-end digital library components. this decision was also made easier, given the fact that we wanted to update the system architecture to separate the component systems, such as the front and back-end systems, that would address whole system downtime issues. additionally, in our desire to have a more stable system, we wanted to build in redundancy and fail-over as part of the initial deployment process. as part of our environmental scan process, we examined other digital library system implementations that have large homegrown digital library systems. we met with the university of north texas (https://digital.library.unt.edu/), who offered valuable insights and directions on the structure of their locally developed system. technology stack with the decision to focus on the public-facing system, we could continue to use the data from the existing system to feed the new front end. using open source components gave us the opportunity to move from windows-based servers (iis and ms-sql) to a linux-based technology stack. in 2021 we were in a unique staffing situation with our digital development team, where we had an entirely new development team consisting of two front-end and two back-end developers, who had no previous experience with our existing digital library system or any prior digital library experience. this team, however, had expertise and experience with modern database and indexing toolsets and development frameworks. the technology stack that was decided on was implementing a postgresql database, with an elasticsearch search engine, and a nginx web server. to modernize and create a more flexible front-end, it was decided that it would be driven by apis. with the goal to have redundancy and fail-over built into the system design, we implemented a redundant production web server environment with load balancing to meet possible system demands. for the database and search engine, we implemented redundant servers. since we were only focusing on our front-end, we designed it in a way to continue to point at our existing files or resource server to pull the access files of our digitized materials. how the system is structured can be seen in figure 1. figure 1. ufdc system architecture diagram development as with any system development we needed to focus on target objectives for an initial release. to prevent feature bloat and continued refinement which may lead to a delay in release, we set out to define a minimum viable product (mvp) for initial release, with alpha and beta release stages. we needed to have a system that would replicate the main functions of our existing system, while at the same time gave us an opportunity to assess if any of our old features were still important to include. we stripped away many of the lesser used features, such as users being able to have their own bookshelf of items and the ability for users to self-submit within the front-end to our institutional repository (ir). during this development period, we made additional efforts to work with collection curators to undertake a clean-up of the ever expanding list of collections that were contained in the system. many of these collections had limited items or may have been created but not really used or expanded as expected. through this effort we went from having over 900 collections to under 500. for our minimum viable product (mvp), all development had to be done while the current system continued to operate with no interruptions and back-end production could continue without interruption. as we developed our production system, we also changed how we implemented our code testing and refinements. this meant implementing a full front-end test system, where the developers could release their code only available to be accessed by those within the development team. once code was through testing, we moved it to the staging environment. this staging environment is available to on-campus library employees to test features and functionality before it is moved to the full production servers. these test environments use the same search apis as the production environment and also have access to the access files available, so accurate testing can happen in this staging environment. with clear objectives defined, we also implemented an agile development framework, which set up clear staged deliverables for each team member. with a team that had limited experience working in libraries, it was decided that the library technology department leadership would attend the daily standup meetings, so that we were able to address any issues or questions from the team immediately. this streamlined the development significantly and enabled the whole team to come together. since this system was a high priority for library leadership and the fact that we had the ability to focus on building the new system, we were able to go from initial development starting around may 2021 to being able to release the system in beta to the public at the end of november of 2021. one of the major goals of developing the new system was to maintain the multiple portal approach to our digital collections. the system now consists of a ufdc portal interface that includes the ability to search the entire aggregate of the collections we house, and additionally includes different user interfaces into the dloc and florida digital newspaper library (fdnl, https://newspapers.uflib.ufl.edu) subcollections (figure 2). figure 2. ufdc collection / portal diagram the multiple portal approach allowed us to develop slightly different user front-ends that are customized to the types of materials being searched/displayed and to the potential audiences that are accessing the collections. this new development not only resulted in being able to replicate our two existing collection portals, but given the flexibility built into the new system, we were able to create a new portal dedicated to our florida digital newspaper library (figure 3). figure 3. florida digital newspaper library portal an example of the customizations that we were able to do within a customized portal is our title search/browse feature within the fdnl collection. since the newspaper titles within this collection are all limited to florida, we decided to add a map-based county facet/limiter to our title search section. this way if a user was interested in exploring titles only within a certain county, they could find them more easily (figure 4). figure 4. newspaper title search by county ongoing work and next stages once in beta and being used by the public, we were able to start to refine the user interface and various other search and structural changes. in order to get user-centered input, we established a library user resource discovery committee, that included a number of librarians and other library staff that worked with the public. this group met regularly and became our main vehicle for moving changes forward in the new system. another avenue for input was to regularly examine questions that were submitted in our ask a librarian service to see if any of the questions related to our new digital library system. this direct input quickly helped us address many issues that were not necessarily evident to our development team. because the new patron interface is focused on user needs, it’s the perfect way for us to best see the metadata that has highest value and impact for users, and prioritize work to best support the user experience. as the front-end systems are in place and we are still making incremental improvements, we are now focusing on our back-end production systems. we envision this process to happen at a slower pace than our front-end system development, but we expect most of this work to be completed within the next two years. we have already developed a starting web-based production toolkit (figure 5) environment where we can release the various component services and jobs needed by our digitization staff. figure 5. ufdc toolkit conclusion although we continue to make progress and there continues to be a lengthy list of enhancement requests, we are very satisfied with what we were able to accomplish. by focusing on a user-centered design, we were able to understand how our users were using the system and make necessary changes. in addition to being able to produce a new front-end system for our digital library, this effort also enabled us to change many of our internal development and project management processes which continues to benefit both our team and the libraries as a whole. about the authors todd digby is the chair of library technology services at the university of florida. he leads a department that researches, develops, optimizes, and supports advanced library information systems and technology services for the university of florida libraries. he has over 25 years working in technology in academic libraries. he holds an ed.d. from hamline university and a master of library and information studies from the university of alberta. cliff richmond is the digital development team supervisor within the library technology services department. cliff received a bs and ms in geology from the university of pittsburgh, and a bs in environmental engineering from the university of florida. he has over 35 years of information technology (it) experience and is a certified information systems security professional (cissp). julio munoz is a digital collections front-end web programmer in the digital development team within the library technology services department. julio received his master of science in computer information systems from the university of phoenix and his bachelor of science in computer engineering from havana university. he also attended the miami international university of art and design for a bachelor of fine arts in web design and interactive media. before starting at uf, julio worked at sharpspring in gainesville, prior to that, he worked at fine art biblio as a web developer in miami, florida, and prior to that, he oversaw the us navy fleet forces command (ffc) website. he has over 18 years of experience in web design and web development, is a microsoft certified it professional, and a certified engineer intern by the florida board of professional engineers. dustin durden is a digital collections application developer analyst in the digital development team within the library technology services department. dustin has a ba in computer science from the university of florida. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – matchmarc: a google sheets add-on that uses the worldcat search api mission editorial committee process and structure code4lib issue 46, 2019-11-05 matchmarc: a google sheets add-on that uses the worldcat search api lehigh university libraries has developed a new tool for querying worldcat using the worldcat search api.  the tool is a google sheet add-on and is available now via the google sheets add-ons menu under the name “matchmarc.” the add-on is easily customizable, with no knowledge of coding needed. the tool will return a single “best” oclc record number, and its bibliographic information for a given isbn or lccn, allowing the user to set up and define “best.” because all of the information, the input, the criteria, and the results exist in the google sheets environment, efficient workflows can be developed from this flexible starting point. this article will discuss the development of the add-on, how it works, and future plans for development. by michelle suranofsky and lisa mccoll background the lehigh university libraries and the lehigh university art galleries (luag) enjoy a collaborative relationship. the two areas of the university continued this relationship in march of 2019 when the staff of the libraries agreed to catalog a collection of approximately 2,000 books owned and housed by luag. these books were being used internally for reference purposes by the staff of luag, and were being stored on shelves in a storage facility on campus. however, a newly remodeled space for a reference reading room in the main art gallery facility was being planned. the staff at luag wanted these books to be available to anyone in the campus community who wished to visit that space. they also wanted the books to be discoverable through the library’s online catalog. while the library staff was eager to take on this project, we were also daunted by the task, knowing that the day to day demands of our job would continue without interruption. in consultation with the luag staff members we decided we could find a balance in getting the appropriate records into the catalog as quickly as possible, while keeping in mind that more granular details, if needed, could be applied to the records later. the opening for the new reading room was set for october 2019, and our goal became to have all of the books on the shelves, classified, and in the catalog in time for the opening reception. evaluating solutions as we began by considering where efficiencies in our typical cataloging workflow could be inserted for this project, automated oclc record lookups came to mind. we estimated that most of the books in the collection were not too old to have isbns or lccns, so we thought that data would be a good entry point for automated searches. we wanted a tool that would select a single “best” oclc record based on a cascading sequence of logic that mirrored as much as possible how we select a record manually. ideally, it would choose the record that we would have chosen in an individual lookup. although oclc’s connexion client does have a batch lookup tool, we knew it could not be customized to the extent we needed to achieve the results we were looking for. we began by looking at tools that utilized oclc’s worldcat search api [1] for record selection. two tools that were considered were cornell university’s ls tools [2] and yale university’s backlog_lookup open refine project. [3] our review showed both to have very rich feature sets. looking at yale’s backlog_lookup tool we got a sense of the large amount of logic and configurability built into it. with those rich features and configurability came a fairly steep learning curve that would have to be navigated in order to obtain the customizations we desired.  fortunately, while this review was taking place, the possibility of using a google sheets add-on [4] for a technical solution opened up after hearing a podcast that focused in part on that feature. steps in cataloging workflows often take place in spreadsheets so it seemed like a tool that existed within this environment would be a good fit. further investigation revealed other attributes of google sheets add-ons that met our needs: it has the ability to make calls to apis it has the ability to parse and construct xml. this meant the add-on could seamlessly parse the marc xml returned from the api calls – to inspect values and pull out values to write back to the spreadsheet. the code is written using plain javascript and is fairly intuitive to work with. google provides thorough documentation and coding examples for this platform the actual writing of the code can take place directly in a browser (using their ‘script editor’) so there is not much effort needed in setting up a development environment. this platform gives you the ability to easily share your projects.  during the development process, the add-on could be shared in a similar way to sharing a google sheet or google document.  this meant the code could be written, easily shared and tested providing an efficient feedback loop. iterations based on those positive attributes, we decided to move forward with creating a google sheets add-on proof-of-concept. we were pleased with how quickly a rudimentary prototype came together.  in the initial version, the record matching criteria was hard-coded. for our test case, we set the criteria to use the following sequence of logic for selection when given an isbn or lccn: return records first that are held by lehigh university (lyu) if no records are held by lyu, return if 040 contains dlc if no records are dlc, return if 042 contains pcc if no records are pcc, return the record that has the most holdings attached. the add-on was able to call the oclc worldcat api using isbns and lccns that were entered into a google sheet and then insert the values from the marc records contained in the api responses back into the sheet. during the next iteration of the project we added match criteria that would allow the user to retrieve records that represent a specific format. while the project for which this was being developed called for a return of records for print monographs, we began to think about future uses for this tool. we realized we would desire functionality from the add-on that would enable us to use it for a variety of formats. therefore we decided if format selection could be entered by the user and not hard-coded the tool would be more extensible. at first we considered using a form within the sidebar of the add-on to set the match criteria. then, as we began to realize that additional match criteria could be desired, another option was put into place. the setting of the criteria, instead of squeezed into the sidebar, was placed onto a new tab in the spreadsheet itself. with this setup a large amount of real estate became available that would allow a user to input fairly complex matching rules, directly in the spreadsheet, with no knowledge of coding needed. this spreadsheet criteria tab also became a place for the user to specify the bibliographic data they wanted returned in their results. testing with the selection logic and match criteria in place, it was time to test the add-on for accuracy. a good test subject came in the form of a donation of 83 math books that the library wanted to accession. each book was searched for in oclc’s connexion client, manually, and an oclc number was decided upon and recorded. then, in a google sheet, an isbn or lccn for each book was entered and the matchmarc add-on was run. we compared the column of manually selected oclc numbers with the column of machine selected oclc numbers. there were only two discrepancies between the human chosen and machine chosen records, and they were both the result of human error: the add-on selected the best record each time. encouraged by these results we began using the add-on immediately to select records for luag’s collection of books. the process of copy cataloging was sped up considerably not only by this new lookup process, but also by the native spreadsheet format, out of which the following workflow developed: after matchmarc results populated the spreadsheet: the column of call numbers returned was used to create spine labels. while a mail merge feature is available in the g suite, we decided to save the google sheet as a microsoft excel document and use ms word’s mail merge feature to complete this task. the same excel document used to create spine labels above was used to create brief marc records using marcedit’s delimited text translator function. local information placed in the spreadsheet, like local notes, barcodes, and call numbers, became part of this brief marc record. the brief marc record file created above was merged with the marc file from oclc, using marcedit’s merge records function. the oclc number was used in this process as the unique identifier on which to base the merge. as a final step before importing the marc file into our ils, a marcedit task removed unnecessary marc fields and marcedit’s rda helper was applied. the records were then imported into our ils. finally, the excel spreadsheet was used as a checklist to ensure barcodes and spine labels were placed on the correct books. using matchmarc matchmarc is free and publicly available. it can be obtained by going to the “add-ons > get add-ons” menu in any google sheet. a search for marc will show the add-on: figure 1. screen-shot showing the icon for matchmarc. once installed and launched, the user will see an “oclc lookup” sidebar. the add-on sidebar gives a field for an oclc api key to be entered. if you need a key, click on “request a key” on the worldcat search api page. the sidebar is also used to indicate which tab contains the isbn and lccn numbers and which tab contains the matching criteria.  this allows the user to work in one spreadsheet using multiple tabs. clicking the “start search” button initiates the search. to help first-time users get started with the add-on the side bar provides a button labelled “click to initialize sample tabs.”  this will generate two tabs: one with a few test searches and a second tab with sample matching criteria. it is important that the match criteria tab is laid out in this format because it is coded to look in specific rows and columns. a feature we added in the latest release of the add-on is the ability to have a marcxml file emailed to you that will contain all of the records it considered a ‘match’.  to receive the email, the user can enter an email address in field labeled “create marc record file and mail to” before clicking the “start search” button. figure 2. screen-shot illustrating the google sheets add-on pointing out the sidebar where the api key is entered, tab selections made and email address entered. matchmarc operates using two tabs inside a google sheet. the first tab is a place to input the information for the records that need to be searched.  it expects column 1 to contain an isbn. if it doesn’t find anything in column 1, it looks in column 2 for an lccn: figure 3. screen-shot illustrating the search criteria (isbn or lccn) in the first two columns of the google sheet. the second tab contains the matching criteria.  on this tab the user can indicate whether or not a search for local holdings is needed. if configured this way, attempting to find local holdings will be the first api call it makes.  if it finds a record, it writes the results to the spreadsheet. otherwise, it makes a second api call instructing the api to return results sorted by the number of library holdings. when it gets those results it starts at the top of the match criteria (in the second tab) to try to detect the preferred record. figure 4. screen-shot illustrating the google sheets tab that contains match criteria. when the add-on finds a record where all of the match criteria in one row is found, it considers that record a match and writes the record details into the first tab. figure 5. screen-shot illustrating the record details added to the spreadsheet. the desired bibliographic details from the marc record that are written to the spreadsheet are also configurable in the lower part of the second tab. figure 6. screen-shot illustrating the google sheets tab that contains match criteria calling out the section of the tab where you configure the fields you want written to tab #1. the add-on will compare the values in each row until it finds a match.  if it does not find a match based on the set criteria it defaults to the record with the highest number of holdings. limitations one limitation to using the google sheets add-ons platform is the execution time limit of 10 minutes. this add-on was developed for a cataloging project that leant itself well to smaller batches of lookups. when we tried to apply the add-on to a large list of ebooks supplied by a vendor, we ran into the time limit problem. another limitation is its reliance on isbns or lccns. this is very effective if a list of these numbers is provided, or if the user has monographs with isbns that are embedded in scannable barcodes. when isbns are not located in a barcode that can be scanned, the user can enter the numbers manually. we have found that even this method is a time saver over a manual lookup. many older monographs may not contain an isbn or lccn, thus making the add-on unusable in these situations. at times the isbns are incorrect, or duplicated. these cases, however, are relatively rare and can be detected with proper checks in place in the workflow. plans for development one exciting development that is planned for matchmarc is that it will have the ability to take marc fields and subfields and their values that a user adds to the results spreadsheet and apply them to the appropriate marc record. when the marc file is delivered by email, it will contain the full oclc records with fields that were added to the google spreadsheet added to each marc record. the field values can vary from record to record. this development arose from the fact that in our current workflow for the luag project, we add local information, like barcodes, call numbers, and local notes, directly on the spreadsheet. after receiving our emailed file of marc records from oclc, we use marcedit to turn the spreadsheet into a marc file, then transfer that local information that was originally in the spreadsheet to the oclc records, using marcedit’s merge function. once all of the information is in a single marc file, the file can be uploaded into our ils, populating the holdings and item record with the appropriate information. this planned development will enable us to skip the steps of transforming the spreadsheet to marc, then merging the files. we are hoping this will be useful for our daily acquisitions processes, since a “best” record could be found at the point of order by the add-on and the acquisitions data for that order could move from the spreadsheet to a delivered marc file seamlessly. also planned is a new application that will use google sheets in the same way but will exist outside of the google add-ons platform. this will provide a way to work around the execution time limit while continuing to leverage the advantage of managing the match criteria and lookup values inside of a google sheet.  we believe that expanding the functionality in this way will be useful for processing the large lists of ebooks we receive from vendors. the complete source code for this project can be found here: https://github.com/suranofsky/tech-services-g-sheets-addon about the authors michelle suranofsky (mis306@lehigh.edu) is a senior analyst on the library technology team for lehigh university. lisa mccoll (lim213@lehigh.edu) is a cataloging/metadata librarian for lehigh university. references [1] oclc developer network [internet]. dublin (oh): oclc headquarters; [cited 2019 sept 30]. available from: https://www.oclc.org/developer/develop/web-services/worldcat-search-api.en.html [2] solla, nancy [internet]. [updated 2019 may 14]. ithaca (ny): cornell university library technical services’s batch processing unit; [cited 2019 sept 4]. available from: https://confluence.cornell.edu/display/tsawg/library+services+tools [3] sugiyama, yukari [internet]. [updated 2019 mar 29]. github; [cited 2019 sept 9]. available from: https://github.com/ysugiyama3/backlog_lookup [4] extending google sheets with add-ons [internet]. g suite developer; [cited 2019 sept 30]. available from: https://developers.google.com/gsuite/add-ons/editors/sheets subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – testing three types of raspberry pi people counters mission editorial committee process and structure code4lib issue 38, 2017-10-18 testing three types of raspberry pi people counters the hudson county community college (hccc) library tested three different types of raspberry pi based people counters between 6/14/2017 and 7/9/2017. this article will describe how we created each type of counter, will compare the accuracy of each sensor, and will compare them to the college’s existing 3m 3501 gate counters. it will also describe why and how our team decided to make this project, discuss lessons learned, and provide instructions for how other libraries can create their own gate counters. by johnathan cintron, devlyn courtier, and john delooper introduction the initial idea for this project came from a talk delivered by jason griffey at the computers in libraries conference in 2013. griffey’s presentation, entitled “making libraries: getting into the hardware biz”, argued that libraries should use devices such as arduinos and raspberry pis to make better, cheaper, and potentially more accurate solutions for library technologies like people counters and heat maps. based on the ideas from this talk, our library discussed the possibility of creating our own electronic people/gate counter. we were frustrated that our existing counters required staff to manually check the machine and write down a visitor number three times a day. in our existing system, written numbers are transcribed once a month to a microsoft access database at one of our campuses, and to a microsoft word document at the other. in addition to errors caused by handwriting or data input mistakes, having staff remember to check the gate count was difficult to ensure, especially when the library was busy and staff were occupied responding to patrons and thus unable to leave the service desk. aiming to make a more accurate solution that did not require human intervention, we hoped to use griffey’s idea to test a fully automated system, using our north hudson branch campus as a test location. there were several challenges to setting up this project. our group had to decide which technology to use to best and most easily store and make the statistics available. we discussed whether we should use arduinos or raspberry pis, or a combination of both. as relatively new technologies, both were hard to acquire from the vendors that we usually use to purchase electronics, and thus there were several months of delays while we sought paperwork from new vendors who stocked these devices and made sure they were added to our purchasing systems. serendipitously, in 2014, we received eight intel galileos, which are arduino-compatible devices with additional processing power, and seed studio brand sensor kits as part of a grant from intel. this equipment was distributed as part of an intel effort to promote the use of their devices in stem curricula at educational institutions. as such, our initial efforts to create gate counters involved using the intel galileo. however, we did not have all the necessary sensors, and our group found that the galileo’s interface for storing and recording data was too difficult. therefore, we switched our efforts to raspberry pis, which, along with appropriate sensors, we were able to first acquire in 2016, thanks to a generous gift by then dean of libraries carol van houten. hoping to create a “set-it-and-forget-it” system, we also wanted to ensure each sensor’s accuracy. therefore, after over a year of research and testing, we settled on three different types of hardware to use: passive infrared sensor (pir), a laser and light dependent resistor (also called ldr or photo resistor,) and ultrasonic distance sensor. methodology: device placement one of the first steps in setting up the project was determining where to place the counters. our initial testing and calibration was performed in a staff office. however, for the final deployment, we had to consider several factors including proximity to power and data (ethernet) connections, patron safety, and potential to disrupt library services. with these factors in mind, we originally thought to place the counters near the inside of our main entrance door, as pictured below. figure 1. view of main entrance to library figure 2. close-up of library entrance door this door is the library’s primary point of entry, and would thus provide a good place to count patrons. unfortunately, placing our devices near the door would require running a series of long extension cords. we feared that deploying extension cords of this this length would be both unsightly and in our estimation, likely to cause trip hazards. we also thought they would have a high risk of being accidentally unplugged. with this in mind, we also wanted to be sure that our spot was against a wall, which would make each sensor easier to mount. this was especially important for the laser sensor, as it required aligning a resistor within a few feet of it. if the sensor was not properly aligned, the laser would lose its connection. space would also have to be sufficient so that the ultrasonic sensor was not recording a wall as a person either. in testing this issue, we found that the distance between walls in our entrance corridor was sufficient to avoid this type of input error. ultimately we chose to use a spot in our entrance hallway, next to our existing gate counter. this area had connections for both power and ethernet nearby, and we felt it would be minimally disruptive for our students as it was in an area with several other electronic devices including a photocopier, and the aforementioned 3m counter. this location was also an easier location for library students and staff, as it allowed the pis to be mounted relatively unobtrusively and minimized trip hazards. we also felt that this spot would provide the most accurate point for comparison with our existing gate counter, as it was adjacent to this device, and thus would measure who entered the library at nearly the exact same location. as the existing 3m counter doubles as an electromagnetic theft detection system, we recognized that this proximity may have made the raspberry pis or their sensors subject to electromagnetic interference, potentially compromising our project’s effectiveness. while we did not notice any issues that were directly attributable to the electromagnetic interference, it is an issue that we will have to keep in mind, and continue to study. the raspberry pis themselves were placed in cardboard boxes, along with the breadboards and attached sensors, and secured to walls with 3m command hooks. when we initially set up the devices, we used command hooks that have a capacity of two pounds. however, the two pound rated hooks proved unable to support the weight of our systems, so we replaced the original mounting strips that were included with the hooks with five pound capacity mounting strips. these higher capacity strips proved sufficient to keep the devices from falling during our testing period. photographs of the setup are below: figure 3. a view of the entrance corridor figure 4. this image shows a step in the setup of our sensors relative to the existing gate counter. the sensors were placed together on the left hand side of the picture. on the right hand side, a white cardboard box contains the laser and galileo. figure 5. once mounted, the pis were relatively unobtrusive when placed near the existing gate counter. methodology: hardware the first and easiest hardware for our staff to construct was the passive infrared or pir sensor, also known as a motion sensor. it required six male-female connector cables, a raspberry pi, a breadboard, and a pir sensor. in the illustration below, the red and black cables are power and ground while the blue cable connects to general purpose input output (gpio) 7. the black cable on the breadboard next to the blue cable is what transmits the information from the sensor to the pi. the other two cables in the sensor are the power and ground. figure 6. pir pi and breadboard setup with view of pin outs figure 7. view of the pir sensor figure 8. view of the pir sensor figure 9. view of the pir sensor for the ultrasonic sensor, we used eight male-female cables and three 1k resistors, although other libraries could choose to use a 1k and 2k resistor instead. the pi pinouts used are pin 5 (power), pin 6 (ground), pin 16 (gpio23), and pin 18 (gpio24). the first two connect the breadboard to the power and ground portion. pin 16 connects to the breadboard and is adjacent to another cable, both blue in figure 10, which connects directly to the sensor. this is how the sensor communicates with the pi. the charge on the pi needs to be reduced to receive accurate readings from the sensor, so the resistors are set up connecting the cable from the pi (red) to the cable from the sensor (yellow). the remaining two pins on the sensor are power and ground, respectively. figure 10. ultrasonic pi and breadboard setup with view of pin outs figure 11. close-up of ultrasonic sensor figure 12. raspberry pi 3 gpio (source: element14.com/raspberrypi) the last sensor, the laser sensor, utilizes a general purpose input output (gpio) ribbon cable, a 40-pin pi wedge, a light dependent resistor sensor, a laser diode, and an mcp 3008 chip, in addition to the other parts we have been using thus far — minus the resistors. another notable difference is that we needed to use a large quantity of male-male cables in this project instead of male-female. for this project, we needed to attach a wedge to the end of the breadboard so the pins are on the two sides of the breadboard. we used five of the pinouts on the wedge: the 3.3v, serial clock (sck), master input slave output (miso), master output slave input (mosi), and chip enable (ce0). the 3.3v connects to the power line of the breadboard. the remaining four cables are connected to ports on the mcp3008 chip. figure 13. mcp 3008 chip port display (note the small indent on the chip should match the display between 1 and 16 figure 14. laser pi and breadboard setup with view of pin outs the sck cable is connected to port 13, the miso is connected to port 12, the mosi is connected to port 11, and the ce0 is connected to port 10. port 16 connects to power on the breadboard, and port 14 is connected to ground on the breadboard. port 15 is connected directly behind the cable connecting to power on the chip. port 1 is connected from the chip to a blank space on the breadboard that has two empty spaces around it in all directions. directly next to that, one cable is connected to power and one to ground. (see below image) these cables are how the ldr sends data. to connect the ldr sensor to the board, three male-female cables are needed: one to connect the cable to the power source, one to connect the ground next to the cable for ground, and the last next to the remaining cable. once these are all set up, the ribbons should be connected to the pi on one end and to the wedge on the other. after this, the laser is connected to power, and you can use another raspberry pi to connect the ldr sensor to a ground and power port. this is because you will need a way for the laser to get power. in our setup, we originally used another raspberry pi, but swapped it for a galileo, as the galileo allowed us to pass through power directly without additional code or other steps required to make a raspberry pi operational, such as installing an operating system. figure 15. close-up of laser breadboard methodology: software our software code was modeled off of the instructions on electro18’s arduino-diy laser / ir person counter instructable (http://instructables.com/id/ir-laser-person-counter/) and adapted and rewritten in python. we chose to use python because it is a versatile scripting language that is easily installable on any linux distribution. as our main goal was to have to the raspberry pi write the statistics to a remote database, we used the mysqldb python library, and used the following sql query: # open database connection db = mysqldb.connect("hostname","username","password","database") # prepare a cursor object using cursor() method cursor = db.cursor() # prepare sql query to insert a record into the database. sql = "insert into ldrstats (date, time, gatecount) values ('%s', '%s', '%d')" % (curr_date, curr_time, count) this query inserted the date, time, and gate count statistics into each sensor variant’s own table in the database. each sensor type required its own customization. full details can be seen in the python scripts at https://github.com/squash-/hccc-library-gatecounter-scripts. troubleshooting the project required a great deal of troubleshooting to work properly. as we were worried about potential trip hazards caused by connecting the pis via ethernet, we initially connected the pis to our campus wireless network. however, this required using one of our staff’s credentials for the wpa2 enterprise setup, which made the project less maintainable. in addition, the pis would disconnect from our institution’s wireless network every night, so we had to use ethernet for the testing. there were also issues with determining how sensitive the sensors should be. this necessitated some tweaking of read intervals before we arrived at the final numbers found in our github code. finally, users who stood in the door often threw off the numbers for each type of sensor, which was a problem even with our original 3m counter. when this happened, we had to manually reboot and reset the system back to zero, and start the count again. finally, we had to abandon our original idea of entering a count of 1 to the database each time the pi recorded motion. doing this type of count, while potentially more accurate, was too difficult to program in the time we had available. findings we started recording data from the gate counters after they were set up on 6/14, and continued throughout the month of july. each week, we checked the data to make sure our mysql database was still receiving data from the raspberry pis. unfortunately, we failed to properly back up our system, and when we did the final data export, we lost all the data from after july 9. with that in mind, we still did record some interesting data, and have noted the results below. 3m 3501 gate counter and paper the campus on which we tested this project has used a 3m 3501 electromagnetic detection system both for security and for gate counts since 2011. the gate counter counts each time a person passes through the gates, and library staff manually record total gate tallies three times a day. by subtracting the number from our final day with the day we started to record, we find that we had 5440 visits over the course of one month. the 3m counter increments up as people both enter and leave, and is vulnerable to errors caused when students, faculty, or staff stand in the path of the gate, thus creating potential over counts. pir the pir recorded data until 6/16 and reset after 1721 visits. it recorded again until 6/19 and the machine rebooted and its numeric tally reset to zero after recording up to 428. after this, it continued recording until 7/3 and rebooted and reset at 1028. it recorded again until 7/9 and rebooted and reset at 1912. thus, the pir recorded a total of 5089 visitors, and required three resets. as with the 3m gate counter, it recorded motion in both directions, thus potentially over counting visitors. ldr the ldr sensor rebooted and reset on its first day of recording at 28 visitors. it jumped from 176 to 321 on 6/15, and had similar jumping issues on 6/19 at 13,639 and 7/3 at 4310. the worst jump was on 6/19, when the device actually fell from its wall mount and laid on the ground, at 7:40 a.m. on 6/19. at this time, the machine began to malfunction and went from 716 visitors to 2100, and climbed to 13,639 visitors by the end of the day. the final tally on 7/9 was 414. therefore, this machine recorded 18,391 total visits, or 5468 if the data from the day the device fell is removed, and we required three resets. as with the 3m and the pir, it recorded motion in both directions, thus potentially over counting visitors. ultrasonic the ultrasonic sensor rebooted and reset its numeric count on 6/14 at 733 visitors and again the same day at 230. it reset again on 6/19 at 248, and on 6/20 at 587 and 7/3 at 337. the final tally on 7/9 was 461. therefore, the ultrasonic sensor recorded a total of 2596 visitors, with 5 resets. this device also counted in both directions, and thus, as with our other sensors, probably over counted visitors. analysis as our project originally hoped to record the month of july, we were somewhat disappointed with the results. this was our own fault, as we did not make proper backups when we checked that the machine was functioning. for each device, other than the existing 3m device, we found resets were a problem, which we suspect was due to a memory leak in our programming. we were not able to verify this during the study. due to the reboot and resets, it was also difficult to obtain truly cumulative data. all of our devices were also uni-directional, and thus counted both entrances and exits, meaning all gate counts were probably twice as high as total visitors. based on the data, the laser/ldr appears more vulnerable to jumps from people standing in its path, and the ultrasonic appears to return the lowest total visitor count. until this issue is studied for a longer period of time, it is difficult to say which option would be the most accurate. future versions of this study could include some hour by hour sampling, or using hand counters to test the overall accuracy of all of the studied devices. related work libraries have long understood the importance of keeping statistics. some of the earliest literature ignores patron count (gerould, 1906) and suggests other metrics like finances and collection size to be most important for evaluating the delivery of library services. however, discussion of patron counting appears as early as chen (1978), and devices for electronically counting people have been around since at least 1965, when stephen hornung received a patent for developing a device to count users of elevators (hornung, 1965). several other technologies were developed afterward, including taylor et al. (1976), who used ultrasonic sensors to count bus passengers, and tsubota and satoru (1978), who used pressure sensitive mats to track movement. the earliest literature we found about library adoption of electronic people counters came in 1999, when 3m began to promote its model 3500 detection system, which upgraded their earlier security systems to include built-in people counters. the earliest instance we found discussing libraries using either raspberry pi or arduino based counters came with griffey (2013), but there are also several other examples of raspberry pi or arduino based counters or motion detectors published in the same time period, including nawrath (2012), one million monkeys (2012), and modmypi (2014). the library technology community has shown an interest in this topic as well on the lita-l listserv, which included discussions about gate counters and the raspberry pi in august 2016, and in february and march of 2017 (see http://lists.ala.org/sympa/arc/lita-l/2017-03/msg00089.html ). finally, there are several commercial providers, including sensource and 3m (now bibliotheca), who provide their own systems for tracking patron use through people/gate/door counters. these are usually sold either from the vendor directly (sensource, 2015), or from library specific vendors like demco (2017) or the library store (tls) (2017). conclusion and future research while the data gathered was useful, we found that the overall reliability of our sensors was not high enough to deploy as a “set it and forget it” type solution. while this project taught us a lot about the possibilities for using raspberry pis to record statistics, we feel that the technology has not yet fully matured. we will continue to refine these systems, and we hope that individuals from other libraries will read this and other literature on the issue, and work to improve on our findings to make more accurate, yet affordable solutions to track library usage statistics. acknowledgements: the authors of this paper would like to thank carol van houten, who purchased the original raspberry pis and sensors, and encouraged us to research this issue. we would also like to thank david hardgrove for supporting our trial leading up to and during the month of july 2017, and cynthia coulter for her encouragement and for helping us find space to test our project at the north hudson library. bibliography 3m. (2017). electromagnetic products. retrieved august 7, 2017, from http://solutions.3m.com/wps/portal/3m/en_ww/track_trace/home/products/one/two/?pc_z7_rjh9u52308du80i4ccrm093ii4000000_assettype=mmm_article&pc_z7_rjh9u52308du80i4ccrm093ii4000000_assetid=1180619853892&pc_z7_rjh9u52308du80i4ccrm093ii4000000_univid=1180619853892#z7_rjh9u52308du80i4ccrm093ii4 3m introduces digital identification system, model 3500 detection system; demonstrates pikiosk. (1999). information today, 16(8), 47.(coins) ching-chih, c. (ed.). (1978). quantitative measurement and dynamic library service. phoenix, az: greenwood press.(coins) electro18. (n.d.). arduino-diy laser / ir person counter. retrieved august 19, 2017, from http://www.instructables.com/id/ir-laser-person-counter/ friedman, m. (2011, july 18). jonesboro library installs thermal people counter. arkansas business, p. 12.(coins) gaven macdonald. (2013). ultrasonic sensor with the raspberry pi. retrieved from https://www.youtube.com/watch?v=xacy8l3lsxi gerould, j. t. (1906). plan for the compilation of comparative university and college library statistics. the library journal, 31(11), 761–763.(coins) griffey, jason. (2013, april). making libraries: getting into the hardware biz. presented at computers in libraries, washington, d.c. retrieved from http://jasongriffey.net/wp/2013/04/08/make-the-tools-that-measure-the-future/ hc-sr04 ultrasonic range sensor on the raspberry pi. (n.d.). retrieved august 21, 2017, from https://www.modmypi.com/blog/hc-sr04-ultrasonic-range-sensor-on-the-raspberry-pi inc, d. (2017). demco.com – patron counters. retrieved august 12, 2017, from http://www.demco.com/category/security/traffic-control/patron-counters/_/n-1a0 jedhodson. (n.d.). arduino and pi in harmony – as a sensor web server! retrieved august 21, 2017, from http://www.instructables.com/id/arduino-and-pi-in-harmony-as-a-sensor-web-server/ jones, j. l. (2011). using library swipe-card data to inform decision making. georgia library quarterly, 48(2), 11–13.(coins) making a laser tripwire with a raspberry pi. (n.d.). retrieved august 21, 2017, from https://www.raspberrypi.org/learning/laser-tripwire/ morton-owens, e., & hanson, k. l. (2012). trends at a glance: a management dashboard of library statistics. information technology & libraries, 31(3), 36–51.(coins) mrhobbyelectronics. (2015). raspberry pi – distance sensor. retrieved from https://www.youtube.com/watch?v=inxfadw0m9y nawrath, m. (2012, january). arduino frequency counter library. retrieved august 12, 2017, from http://interface.khm.de/index.php/lab/interfaces-advanced/arduino-frequency-counter-library/ normington, j. (2014). what are those people really doing in our library? incite, 35(10), 15–15.(coins) patron counters. (2017). retrieved august 12, 2017, from http://www.thelibrarystore.com/category/patron_counters_and_door_alarms patron counting analytics for the library industry. (2015). retrieved august 12, 2017, from http://www.sensourceinc.com/industries/library.htm perone, c. s. (2012, august 19). raspberry pi & arduino: a laser pointer communication and a ldr voltage sigmoid. retrieved august 21, 2017, from http://blog.christianperone.com/2012/08/raspberry-pi-arduino-a-laser-pointer-communication-and-a-ldr-voltage-sigmoid/ phillips, j. (2016). determining gate count reliability in a library setting. evidence based library and information practice, 11(3), 68–74.(coins) pi my life up. (2015). raspberry pi motion sensor using a pir sensor. retrieved from https://www.youtube.com/watch?v=mms7esi0sao raspberry pi gpio sensing: motion detection. (2014, february 19). retrieved august 21, 2017, from https://www.modmypi.com/blog/raspberry-pi-gpio-sensing-motion-detection raspberry pi laser tripwire. (2012, december 21). retrieved august 21, 2017, from https://1000000monkeys.wordpress.com/2012/12/20/raspberry-pi-laser-tripwire/ raspberrypiivbeginners. (2014). raspberry pi – gpio & python (9/9) – passive infrared sensor. retrieved from https://www.youtube.com/watch?v=cpr4vxngzew screechynutz. (2014). motion activated surveillance system using a raspberry pi. retrieved from https://www.youtube.com/watch?v=jhbgllftxi8 security tools. (1999). computers in libraries, 19(8), 16.(coins) stephen, a. h. (1965, september 21). 3207266. louisville, ky.: united states patent office. retrieved from http://www.google.com/patents/us3207266 szczys, m. (2011, april 19). laser trip wire – the bare essentials. retrieved august 21, 2017, from http://hackaday.com/2011/04/19/laser-trip-wire-the-bare-essentials/ taylor, w. r., linder, f. x., & clark, r. v. (1976, december 14). 3997866. silver spring, md. retrieved from http://www.google.com/patents/us3997866 thompson, s. (2015). using mobile technology to observe student study behaviors and track library space usage. journal of access services, 12, 1–13. https://doi.org/10.1080/15367967.2015.972754 tsubota, n., & satoru, s. (1978, october 24). 4122331. retrieved from http://www.google.com/patents/us4122331 we count people. (n.d.). retrieved august 12, 2017, from https://wecountpeople.com/ white, l. l. (1981, june 23). us4275385 a. retrieved from http://www.google.com/patents/us4275385 wright, a. (2006, february 15). ntlp – not typical library partners: gate counters. retrieved august 12, 2017, from http://ntrls.blogspot.com/2006/02/gate-counters.html about the authors johnathan cintron is an it technical support specialist at bergen community college. he holds an a.s. in computer science from hudson county community college, where he was a library technology associate from 2013 to 2017. he is an active participant in the android firmware development community, and his projects can be seen at http://pastebin.com/kfhycfd4. his github page is https://github.com/squash. his email address is johnathancintron@gmail.com. devlyn courtier is a library technology associate at hudson county community college, where he is also a student. he has written and presented about topics including using devices like raspberry pis to teach coding, hosting video game tournaments in academic libraries, and circulating electronics at academic libraries. his email address is dcourtier@hccc.edu. john delooper is director of library technology at hudson county community college, where he has worked since 2011. he holds an mlis from rutgers university, a b.a. in history from the george washington university, and recently completed an ms in information systems at baruch college. his email address is jdelooper@hccc.edu. appendix a: final tallies table 1. final tallies ultrasonic ldr pir 3m ldr – adjusted 733 28 1721 230 13639 428 716 248 4310 1028 4310 587 414 1912 414 337 461 2596 18391 5089 7966 5440 each time a sensor reset, its highest tally was recorded in the table above. more detailed tallies are available at www.hccclibrary.net/pi/pi_data.xlsx . appendix b: pictures picture 1. view of laser pi and breadboard with ldr sensor picture 2. close-up of ldr sensor picture 3. close-up of laser breadboard picture 4. view of ultrasonic pi, senor, and breadboard picture 5. alternative view of ultrasonic pi, senor, and breadboard picture 6. view of the pir sensor from various angles picture 7. view of all three mounted pis picture 8. laser in box subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – on two proposed metrics of electronic resource use mission editorial committee process and structure code4lib issue 52, 2021-09-22 on two proposed metrics of electronic resource use   the author has requested that this article be retracted. subscribe to comments: for this article | for all articles one response to "on two proposed metrics of electronic resource use" please leave a response below: becky yoose, 2021-09-26 [the following is a slightly edited and revised version of the feedback given to the code4lib journal editorial team within a 24-hour timespan. the input was requested late monday, with publication scheduled for the end of the workweek. therefore, the feedback below is not comprehensive of the privacy, technical, data quality, assessment, and ethical concerns present in the article as it is published. this is primarily a public record of what the editorial committee had in hand when they decided to publish after their last-minute request for privacy feedback.] the feedback below is a combination of technical and data privacy factors that makes this proposed method of electronic resource use assessment of particular concern if published without additional context around risk and potential privacy harms to patrons (if published at all). here’s a list of those factors in no particular order or rank of importance: 1. behavioral tracking – the article describes a process where several scripts clean up a daily log that (theoretically) is scrubbed daily and creates a csv file that is kept for reporting purposes. the data in the csv is not in the aggregate – the barcode or institutional user id links a patron’s use of a platform. the article describes keeping these csv files to create reports on the annual use of electronic resources, meaning that the library has a record of the daily behavior of identifiable patrons that otherwise would not exist. 2. lack of clear, explicit user consent and privacy policy issues – i have not had time to review york university libraries’ privacy policy or the university privacy policy. however, if the policy allows for statistical analysis of aggregated user data, this particular use case would fail to meet that reserved right to use the data due to the identifiable data in the csv. it could also be that patrons are not aware of behavioral tracking that is happening in this article. the article does not state if consent or notification was given to patrons. 3. unprotected identifiable data – this one is pretty self-explanatory. storing csvs that have identifiable information – barcode or institutional id – tied to the use of library resources and materials is a privacy risk. one way of mitigating this risk is through pseudonymization of the patron pii to create a unique id that breaks the direct link between behavior and identity; however, in this use case, i would argue that pseudonymization won’t do much in protecting patron privacy, leading to the next point… 4. re-identification risk due to assessment focus and characteristics of certain patron groups – we are looking at assessing the unique use of electronic resource platforms. electronic resource platforms range from multidisciplinary (jstor) to niche, specialized areas of research. even if you pseudonymize the barcode or institutional id data, i can with some level of certainty identify individuals based on behavioral patterns based on the reports combined with a public school directory that contains major/minor concentrations or other public data sources available through the university or elsewhere. this assessment’s granularity and focus make it nearly impossible to use de-identification methods to mitigate privacy risks. 4.1 on a side note, transforming the barcode to the institutional id makes this dataset even more identifiable and possible for misuse through secondary use of the dataset or improper access or combination of this dataset with another sensitive dataset on campus (and most likely without the student’s knowledge). 4.2 side note #2 – the statement “it produces anonymized aggregate data with all personally identifiable information gone.” isn’t really true, even if you pseudonymized the barcode/institutional id because the platform analysis is still granular enough that aggregation can’t protect the privacy of those using niche or specialized resources based on the number of patrons doing work in those areas. 4.3 side note #3 – this is the obligatory reminder that proper anonymization is nearly impossible https://georgetownlawtechreview.org/re-identification-of-anonymized-data/gltr-04-2017/ is an excellent place to start. i can also get into the specifics of how l-diversity and k-anonymity can also point out the privacy risks with the dataset, but i’ll spare you the math! 5. report data lifecycle – according to the privacy section of the article, “these reports can be kept and used after the source data has been wiped. whether they are shared outside the library depends on institutional openness about collection usage statistics, but there are no user privacy reasons preventing it.” we still have the daily csv reports hanging around. where are they stored, who has access to them, and how long are they retained? 6. choice of assessment harmful to both desired assessment outcome and patron privacy – this is more of a concern in the area of “using data to address an issue that can be more effectively addressed with other methods that are less privacy-invasive.” suppose we are interested in how patrons use specific resources or how different groups use these platforms. why are we not talking to these patrons about their use of said resources? there are better ways to assess collection use than violating patron privacy through the process described in the article. in addition, other more qualitative research methods, such as interviews with students and faculty, can produce more meaningful data that can more accurately reflect the value of the databases in question since the value is not very easily derived through data alone without making dangerous and inaccurate assumptions about the use data itself. flawed data leads to flawed assessment. leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – improving the discoverability and web impact of open repositories: techniques and evaluation mission editorial committee process and structure code4lib issue 43, 2019-02-14 improving the discoverability and web impact of open repositories: techniques and evaluation in this contribution we experiment with a suite of repository adjustments and improvements performed on strathprints, the university of strathclyde, glasgow, institutional repository powered by eprints 3.3.13. these adjustments were designed to support improved repository web visibility and user engagement, thereby improving usage. although the experiments were performed on eprints it is thought that most of the adopted improvements are equally applicable to any other repository platform. following preliminary results reported elsewhere, and using strathprints as a case study, this paper outlines the approaches implemented, reports on comparative search traffic data and usage metrics, and delivers conclusions on the efficacy of the techniques implemented. the evaluation provides persuasive evidence that specific enhancements to technical aspects of a repository can result in significant improvements to repository visibility, resulting in a greater web impact and consequent increases in content usage. counter usage grew by 33% and traffic to strathprints from google and google scholar was found to increase by 63% and 99% respectively. other insights from the evaluation are also explored. the results are likely to positively inform the work of repository practitioners and open scientists. by george macgregor introduction significant resource has been invested over the past decade to expose rich digital collections using a variety of repository technologies. this investment has resulted in unprecedented usage of institutional repositories, as evidenced in the uk by services such as irus-uk which, at time of writing, has recorded 146,398,650 counter compliant downloads from participating repositories since 2013 [1]. however, many institutions continue to demonstrate limited commitment to ensuring their scholarly content is exposed optimally. this also extends to a failure to ensure their repository is as usable as possible. in fact, many repositories have not undergone development beyond their original establishment and maintenance of its scholarly collection. the reasons for this inertia are complex and it is not the purpose of this paper to explore them. however, it is sufficient to state that such institutions may attempt to promote their repository content but if few attempts have been made to optimise for discovery, then these repositories may find themselves under exposed [2] and under used. significant future challenges are facing open access repositories, as well as the open science movement more generally [3]. competing scholarly platforms, many of which are proprietary, appear to be growing in popularity yet demonstrate poor support for open standards or prevalent open science technical protocols, as well as low levels of integration with open scholarly infrastructure. it is therefore imperative that user expectations of repositories are better met and improvements to the index penetration and exposure of their scholarly content demonstrated. only by doing this will scholarly open access repositories validate their continued relevance in open scholarly communication. in this contribution we experiment with a suite of repository adjustments and improvements performed on strathprints [4], the university of strathclyde institutional repository powered by eprints 3.3.13. these adjustments were designed to support improved repository web visibility and user engagement, thereby improving usage. although the experiments were performed on eprints it is thought that most of the adopted improvements are equally applicable to any other repository platform. following preliminary results reported elsewhere [5], and using strathprints as a case study, this paper will outline the approaches implemented, report on comparative search traffic data and usage metrics, and deliver conclusions on the efficacy of the techniques implemented. the results are likely to positively inform the work of repository practitioners and open scientists. background given the importance of institutional repositories in promoting open scholarly communication and the discovery of open research content, it is perhaps surprising to note that only a limited amount of prior work has been documented on repository discoverability approaches and their evaluation. many contributions note the importance of repository discoverability and report on some of the factors that should be addressed [6], but few then evaluate the impact of these factors. most recently, however, the code4lib journal published a contribution on the use of microdata within institutional repositories as a “low barrier” means of better exposing contents to google [7]. this work described the implementation of schema.org within dspace. it is a notable contribution owing to the fact that repository support for schema.org is a feature of the coar next generation repositories agenda [8]. pekala reported generally positive results but conceded that demonstrating its impact was difficult. kelly and nixon reported on the use of general seo techniques on three separate uk repositories [2]. this work relied on analytics services and tested early data indicating the importance of blogs in driving repository web traffic. the authors reported mixed results and therefore concluded that further work was required in order to refine their methodology and better understand search engine behaviour. in a poster presented at the 2017 repository fringe conference, the present author evaluated the preliminary results derived from a series of repository enhancements designed to improve web impact and discoverability. while some encouraging evidence was reported about the impact of specific repository enhancements, the small nature of the evaluation prohibited any wider conclusions to be drawn. others have focused on hypothesised impediments to repository discoverability. for example, tonkin et al. explored the significance of repository coversheets in disrupting the bot crawling potential of repositories in some cases [9], a practice also considered by anurag acharya of google scholar as undesirable [10]. better supporting google scholar indexing was addressed by arlitsch and o’brien, who noted variable indexing coverage of repositories on gs and evaluated the effects of adjusting in-page metadata on gs indexing penetration. arlitsch and o’brien highlight the dangers of paying insufficient attention to discoverability and propose corrective actions for repository managers to perform. promoting repository discovery whilst many of the prominent repository platforms (e.g. eprints, dspace, digital commons, ojs, etc.) now provide basic out-of-the-box support for discovery and interoperability with key academic tools, including meeting google scholar inclusion guidelines, there remains wide variation on the relative visibility and discoverability of repository content. the question of repository discoverability is therefore something which has attracted significant attention at the university of strathclyde as the institution seeks to ensure its internationally significant research [11], much of it available open access via strathprints, can be found easily. strathprints is powered by eprints (version 3.3.13). to improve repository web visibility and user engagement, thereby improving usage, a series of technical changes were made to strathprints in spring 2016 and their impact monitored during 2016/2017, and again in 2017/2018. process improvements were also implemented. the changes could be said to fall into one of two categories: improvements, and; adjustments. “improvements” were changes that resulted in substantive modifications to repository functionality, while “adjustments” included actions that sought to refine existing aspects of the repository. as noted below, much of the motivation for these improvements and adjustments came from the broader literature on web publication best practice and seo; although some were gleaned from the repository best practice literature [10]. figure 1. discovery paths for content stored in the strathprints repository. technical changes improvements the principal improvements made included: implementation of a refreshed strathprints user interface (ui). many repositories continue to demonstrate low levels of usability [12], [13]. low levels of usability can result in the users’ abandonment of a website or of system sessions [14], [15], [16]. an heuristic evaluation [17] of strathprints user interface (ui) was therefore undertaken in early 2016 to direct ui changes intended to improve usability and user engagement (figure 2 & 3); following heuristic evaluation, a “mobile first”, responsive re-engineering of strathprints was implemented, thereby triggering important signals in pagerank [18] and, later, heavier weighting in the google “penguin” updates [19] (figure 2 & 3); “white hat” improvements [20] to the way strathprints functions. this included improvements to internal linking (e.g. navigation, hyperlink labels, etc.) and content improvements including promotion of user interaction through support for the core recommender and altmetric. both of these improvements stimulate additional user interaction. for example, in the case of the core recommender this is achieved by referring users to alternative but related additional strathprints content, recommended to the user on the basis of the repository item they are currently browsing; improved integration with social tools, including growth in social interactions which are the result of tweets about recently deposited strathprints content; implementation of a “connector-lite” configuration actioned to cultivate strathprints as a full-text destination for users and machines alike [21]. within the currently scholarly communication landscape it is not uncommon for institutional repositories to now operate in parallel with the local current research information system (cris). this so-called “connected” configuration enables metadata and digital content exchange from the cris to the repository. it is a configuration that applies to strathprints, which is no longer a point of entry for staff wishing to deposit content in strathprints; instead users deposit via the cris which then automatically writes metadata and content to strathprints. “connector-lite”, however, enables greater control over what is written to strathprints by the institutional cris [21]. figure 2. strathprints ui (homepage). figure 3. strathprints ui (abstract pages). adjustments a series of adjustments were made to fine tune the search engine friendliness of strathprints and to enhance user experience. a number of these related to delivering page speed improvements for strathprints, in line with trends within search agents to factor speed in results rankings ([18], [19], [20], [22]). adjustments to the file-naming conventions used for deposited full-text files in order to render them more crawler friendly. descriptive file-names can lead to better and more effective crawling of files. moreover, words contained in file-names factor in retrieval algorithms and may be highlighted to users in results pages, so accurate naming is necessary to facilitate ‘known-item’ searching by users. a descriptive file naming convention with proactive use of hyphens to separate words in the filename [18] was therefore adopted. the broad approach to naming was as follows: {author surname(s)}{journal/conference acronym}{year of publication}{selected uninterrupted words from title of article using hyphens for spacing}.pdf so, for example, a file pertaining to the present article would be named: macgregor-c4l-2019-improving-the-discoverability-and-web-impact-of-open-repositories-techniques-and-evaluation.pdf gradual cleaning of broken links within strathprints thereby improving the “content health” of strathprints and, again, triggering important signals in pagerank [18]. like many repositories of its type, strathprints has been operating in one form or another for over 10 years and during that time has accumulated its fair share of “link rot”; “minification” of all relevant repository files (e.g. css, js, etc.) to deliver increased page loading speeds. minification refers to the process of removing superfluous or redundant data without affecting how the resource is processed by browsers, e.g. code comments, formatting, white space characters, unused code, using shorter variable names, etc. this superfluous data may aid the human readability of the code but is not needed for the code to execute efficiently. rationalisation of all css and javascript (js) files in order to remove unused rules and variables. this can be performed manually but there are automatic online tools (e.g. purifycss, uncss! online) which can analyse websites to determine which css rules are actually being applied to a given website, thus allowing redundant rules to be deleted. similarly, there are code quality tools for js (e.g. jshint). asynchronous loading of js resources: render-blocking js is probably the single most difficult obstacle to overcome when attempting to deliver repository speed improvements (see [23] for further details). a repository like strathprints, like most others, will require the loading of many js resources in order to deliver important functionality. for strathprints this includes native js resources but also third-party js such as the google jsapi, altmetric api, analytics from google analytics and addthis, as well as for any eprints plugins that have been installed from the eprints bazaar. however, some simple experimentation can deduce whether it is necessary for js to be loaded at the same time as the page itself since in many cases js can actually be deferred until after page rendering [23]. html5 introduced the async attribute to be used with <script>. this boolean attribute indicates that the browser should, if possible, execute the script asynchronously. for example: <script type="text/javascript" async="async" src="https://www.google.com/jsapi"><!--padder--></script> gzip compression: gzip is a file format and software application used for file compression and decompression. all modern browsers support and automatically negotiate gzip compression for all http requests and, where used, gzip can compress the size of the transferred response by up to 90%. this significantly reduces the amount of time needed to download resources, reduces data usage for users, and improves the first render time repository pages. enabling gzip, however, is an infrastructural task as it necessitates adjusting the repository server configuration so that it returns “gzipped” content to compliant browsers. gzip implementation is described in more details at [23]. revisiting image optimisation: the question of optimising images for delivery over the web will vary from repository to repository and, in fact, many repositories have very little visual content at all. strathprints uses large banner images which, when not sufficiently compressed, were found to negatively influence page loading times [22]. all image resources were therefore compressed and optimised accordingly. migration to innodb as the mysql storage engine in order to improve repository performance: eprints generally runs on mysql, using myisam as the default storage engine, but table locking was found to be a db performance issue thereby inhibiting the execution of simultaneous queries. innodb demonstrates concurrency, locking only the row(s) which are relevant to the db query, leaving the rest of the table available for crud operations. deployment of google data highlighter: we noted earlier that exposing contents to google could be improved through the implementation of schema.org [7]. it was not possible in this instance to re-engineer eprints in order to expose schema.org interoperable data, although this may be something to be explored in future. instead google data highlighter – a pattern matching tool for structured data on websites – was deployed as a substitute [24]. data and results the impact of the repository changes were monitored and measured using a variety of metrics, including search traffic data from google search console [25], counter compliant usage data from irus-uk [1], google analytics [26] tracking data and routine statistical data from strathprints itself. the periods examined were the year up to end july 2016 (year 1 – y1), prior to the changes being implemented; and the years up to end july 2017 (year 2 – y2), after improvements were deployed, and end july 2018 (year 3 – y3), after the adjustments were implemented. note that counter usage data [1] refers to the international counter ‘code of practice’, which sets standards on how electronic content usage is calculated thereby allowing content publishers to provide consistent, credible usage data. this data can then be used to accurately understand real world usage and provide usage comparisons across multiple services or websites. traffic web traffic, as measured by google analytics (ga), grew by 150,408 in y2 to 428,407, equivalent to a 54% improvement when compared to y1. a 52% improvement in unique traffic was also observed during the same period (figure 4(a)). an increase in traffic in y3 was less than y2 (6%) but was still in excess of y2 (n = 454,318), meaning that the total percentage growth in traffic during the entire reporting period was 63% and 65% for traffic and unique traffic respectively. as might be expected, google was found to be the largest referral source, accounting for 55% of all traffic in y3; but thereafter google scholar was found to be the most significant referral source, accounting for 25% of all web traffic in y3 and growing by 99% during the entire reporting period. traffic in y2 grew by 48% (n = 83,045) and 34% (n = 111,563) in y3. 77% of all this traffic in both y2 and y3 was unique. this is at variance with previously reported results emerging from a preliminary evaluation [3], in which gs traffic was found to have declined slightly as a proportion of total web traffic (by 3%). in fact, this present evaluation, using a more comprehensive dataset, found the percentage of total traffic to strathprints from gs to have increased by 5%, with almost all of these gains achieved during y3. thus, the percentage traffic gains achieved from gs during the reporting period (99%) grew even quicker than the broader gains achieved from other web traffic sources (63%). this can be observed data charted in figure 4(b). repositories serving more content enjoy deeper indexing by google scholar (gs) [10] and, combined with the other improvements and adjustments, may be a possible explanation for the gs improvements. figure 4(a). volume of referral traffic (views and unique views) as calculated by google analytics (ga) in y1, y2 & y3. figure 4(b). volume of referral traffic from google scholar (gs) for views and unique view, as calculated by google analytics (ga) in y1, y2 & y3. figure 4(c). volume of google & google scholar referral traffic (views) in y1, y2 & y3. the principal referral sources remained largely unchanged during the reporting period, with both google and gs referring the majority of the traffic. however, the proportion of the overall traffic referred to strathprints by google and gs grew by 18% between y2 and y3 such that 80% of all repository traffic was referred by either google or gs. the remaining 20% comprised a long tail of services. the nature of this traffic growth can perhaps be better observed in figure 4(c) when the data for figure (a) and (b) are overlaid, with gs demonstrating steeper growth relative to other traffic. table 1 summarises the top ten referral sources (with local sources excluded). a 29% decrease in bing referrals between y2 and y3 is noted, as is a larger decrease for yahoo! (which shares the bing index). reasons for this are suggested later in this paper but essentially relate to search interference arising from the institutional cris. however, given the overall small contribution to traffic made by bing and yahoo! – and the far larger increases in referral traffic from other sources (including within the long tail) – this decrease is more than cancelled out. an interesting observation relates to the increase in referrals from social sources, such as twitter and facebook. again, this traffic remains small in relation to the volume of total traffic but extraordinary percentage increases can nevertheless be observed. for example, traffic from twitter increased by 3700% between y1 and y3 as improved social media interaction opportunities were implemented. referral source y1 y2 y3 google 152890 185491 251705 google scholar 57319 83045 111563 bing 10794 10411 7405 twitter 173 1414 6556 android google search 0 0 2274 baidu 3234 2657 2209 glgoo 878 1048 2077 yahoo 3628 1351 1436 facebook 533 634 1108 ebsco (eds) 482 433 485 table 1. summarised web traffic referral sources as measured by ga with local sources excluded. discovery a more appropriate measure of repository discoverability lies in search metrics. google search console was used to gather search data during the reporting period, thereby allowing the effect of the repository changes to be examined on google search queries. search console makes the distinction between data pertaining to “impressions” and “clicks”. impressions are defined as occurring when “a link url record […] appears in a search result for a user”, while a click is “any click that sends the user to a page outside of google search” [25]. improvements in impressions and clicks were observed in y2 at 52% (n = 5,795,781) and 23% (n = 169,720) respectively when compared to the y1 period. this upwards trend continued in y3 at 61% (n = 9,357,582) and 25% (n = 212,148), and a general upwards trend in impressions and clicks can be observed in the graph profile of figure 5, with impressions and clicks demonstrating particular growth from early 2017 onwards. the total percentage growth in impressions and clicks during the entire reporting period was 146% and 53% respectively. figure 6 provides a summary of the increase in clicks, impressions and counter usage, with steeper increases in impressions and clicks noted between y2 and y3. figure 5. strathprints counter usage during y1, 2 & 3 and google clicks & impressions during the same period. figure 6. charted data on observed clicks, impressions & counter usage during reporting period. during the full period examined (i.e. y1-y3), strathprints demonstrated a 33% growth in counter compliant usage. this growth in usage was observed despite only a 19% growth in full-text deposits during the same period. the pattern of this usage appears more nuanced when considered on an annual basis. for example, y2 and y3 observed a 6% and 25% increase respectively in counter usage, with the number of deposits in y3 actually declining by 19% while increasing by a similar proportion in y2. usage therefore generally increased greater than the number of deposits but in the first year this was not observed, possibly owing to the latency of search tool indexes during y1. it is also noteworthy recalling that google search referrals and gs traffic demonstrated growth well in excess of the 19% full-text deposit rate, as per figure 4. in other words, the percentage of users being referred increased at a greater rate than the percentage growth rate of full-text. to determine whether a correlation between clicks and counter usage was present, pearson’s correlation coefficient was calculated for each year in the reporting period. pearson’s correlation coefficient provides a measure of the linear correlation between two variables by using a value between -1 and 1 to denote the strength of correlation. it can be reported that a correlation was detected, ranging from a weak relationship in y1 (r = 0.26) to a moderate positive correlation in y2 (r = 0.65). for those readers familiar with statistics, this correlation was confirmed via the t statistic (t = 2.68, df = 11, p < 0.05). a strengthening of the positive correlation was further observed in y3 (r = 0.97), also confirmed by the t statistic and a higher level of statistical significance (t = 12.72, df = 11, p < 0.001). computing the coefficient of determination (r2) allows us to better understand the proportion of the variance in the dependent variable (i.e. counter usage) which is predictable from the independent variable (i.e. google clicks). computing the coefficient of determination revealed data to be more nuanced (figures 7, 8 & 9). r2 was stronger in y2 (r2 = 0.419) than y1 (r2 = 0.069); clearly a significantly higher value but indicating that only circa 42% of the unique variance in counter usage could be directly attributed to google clicks. however, this variance narrowed considerably for y3 (r2 = 0.934) with a strong linear relationship between variables noted such that 94% of the unique variance in counter usage could be directly attributed to google clicks. this narrowing in variation can also be observed from figure 5, with data points grouping more closely to the regression line. figure 7. coefficient of determination (r squared) for y1 (clicks and counter usage). figure 8. coefficient of determination (r squared) for y2 (clicks and counter usage). figure 9. coefficient of determination (r squared) for y3 (clicks and counter usage). by exposing their content to disparate search services, and the nature of repository content itself, repositories encourage – and are conducive to – “horizontal” information seeking strategies [19]. these types of information seeking strategy typically correspond with the relatively high “bounce rates” that repositories experience. bounce rates are calculated by ga as “a session that triggers only a single request to the analytics server, such as when a user opens a single page on your site and then exits without triggering any other requests […] during that session” [26]. the bounce rate in this study remained relatively unchanged, fluctuating across reporting periods at circa 75%. however, the average time users spent on strathprints upon arrival increased, up from 01:13 in y1 to 01:54 in y2 and then 01:59 in y3. although users were continuing to bounce, they were typically spending longer on strathprints, indicative perhaps that improvements to the ui and strathprints functionality was enough to persuade users to defer their bounce. in other words, it was possible to improve users’ “dwell time” on strathprints by 61% between y1 and y3. dwell time therefore suggests itself as a more accurate indicator of repository engagement than bounce rates, which experienced only marginal change during the reporting period. bounce rates are not necessarily a reliable metric within models of information seeking behaviour. for example, a user might spend 25 mins reading content on a repository, taking notes and chaining references, but then they might leave. in this example the user “bounced” because they failed to navigate to another page on the repository. but, in repository terms, the user spent 25 mins consuming repository content and found that content sufficiently useful that they “dwelled” for 25 mins. dwell time is therefore critical to understanding repository engagement. interestingly, it is for this reason that many search services, google and bing included, factor “dwell time” into their relevance rankings [27], [28]. like pagerank more generally, the way in which search tools calculate dwell time, or the weighting it is assigned in computing algorithms such as pagerank, is unknown; but it is clearly a variable in calculating relevance and is therefore a metric institutions and repository managers should monitor. similarly, the significance of dwell time in this evaluation is impossible to calculate. it is only possible to state that it would have positively influenced the visibility of strathprints in the search results of services such as google and bing. conclusion and future work in this contribution we experimented with a suite of repository adjustments and improvements performed on an eprints powered repository. these adjustments were designed to support improved repository web visibility and user engagement thereby improving usage and should be considered within the wider context of the coar next generation repositories agenda. the evaluation provides persuasive evidence that specific enhancements to technical aspects of a repository can result in significant improvements to repository visibility, resulting in a greater web impact and consequent increases in content usage. the results suggest that both web and search traffic and counter usage can be significantly improved on the most important search and discovery tools, with strong correlations between google search visibility and repository counter usage demonstrated and variation narrowing particularly in y3. 94% of the unique variance in counter usage was found to be directly attributed to google clicks. strathprints also demonstrated a 33% increase in counter compliant usage during the years examined. across the entire reporting period total traffic to strathprints grew by 63%, with google impressions and clicks increasing by 146% and 53% respectively. gs traffic was also found to have generated a traffic growth 99%, accounting for 25% of all web traffic to strathprints in y3. user dwell time was also found to have increased, suggesting longer interaction sessions by users. of course, as with any experiments attempting to effect change on third party systems, it is impossible to control for all variables hypothesised to influence web visibility. it is not claimed that every known variable has been addressed in this instance. the approach adopted here of delivering repository adjustments and improvements was a holistic one, and was intended to address as many as possible. the approach could therefore be described as pursuing the accumulation of marginal gains; identifying numerous minor optimisations that can be implemented which, when taken in aggregate, effect further significant improvements. there are also limitations to be noted on the use of search data from google search console which, for obvious reasons, provides data on google searches only. however, as the majority of referral traffic to strathprints comes via google this seemed an acceptable compromise to be made in this instance. future similar studies should nevertheless explore additional sources of search data to improve the accuracy of conclusions drawn, especially as google cannot be relied upon to be the preeminent web search engine indefinitely. we intend to continue monitoring our data into y4 with the hope of exploring how additional adjustments could improve visibility on other search discovery tools, thereby providing the basis for greater longitudinal analysis. although the experiments were performed on eprints it is thought that most of the adopted improvements are equally applicable to most repository platforms. there is, in fact, potential for others to improve the impact of the approach. for example, it was noted in the literature that coversheets are considered to be disruptive to the bot crawling potential of repositories and it has been suggested that repositories disable such repository functionality [29]. based on local experimentation and the need to ensure accurate attribution of repository outputs, coversheets remained enabled in strathprints and continue to remain enabled. this therefore highlights a possible limitation. however, there are also potential additional improvements to be gained by other repositories willing to develop their own alternative approaches (e.g. watermarking attribution details) or disabling coversheets altogether. furthermore, owing to the existence of strathprints within a connected cris configuration, the present author noted issues of the cris front-end interfering with the visibility of strathprints in some cases. again, this interference was almost impossible to quantify and appeared to particularly affect bing and yahoo! searches; but for those repositories operating outside of a cris environment or functioning as the de facto cris front-end, considerable additional opportunities are available vis-à-vis promoting the discoverability and web impact of repository content. references [1] irus-uk [internet]. 2018 [cited 2018 jun 30]. available from: http://www.irus.mimas.ac.uk/ [2] kelly b, nixon w. seo analysis of institutional repositories: what’s the back story? in: open repositories 2013 [internet]. university of bath; 2013 [cited 2017 jul 19]. available from: http://opus.bath.ac.uk/35871/ [3] macgregor g. the long read: why do institutional repositories remain one of the only viable options for green open access? [internet]. open access @ strathclyde. 2016 [cited 2017 jun 29]. available from: https://perma.cc/g52j-2fsg [4] macgregor g. reviewing repository discoverability?: approaches to improving repository visibility and web impact. in: repository fringe 2017 [internet]. john mcintyre conference centre, university of edinburgh; 2017 [cited 2018 aug 3]. available from: https://strathprints.strath.ac.uk/61333/ [5] macgregor g. reviewing repository discoverability with strathprints [internet]. open access @ strathclyde. 2017 [cited 2018 aug 29]. available from: https://perma.cc/a3r9-w2jv [6] tmava am, alemneh dg. enhancing content visibility in institutional repositories: overview of factors that affect digital resources discoverability [poster] [internet]. iconference, 2013, fort worth, texas, united states. 2013 [cited 2018 aug 13]. available from: https://digital.library.unt.edu/ark:/67531/metadc146593/ [7] pekala s. microdata in the ir: a low-barrier approach to enhancing discovery of institutional repository materials in google. code4lib journal [internet]. 2018 feb 5 [cited 2018 aug 13];(39). available from: https://journal.code4lib.org/articles/13191 [8] coar. next generation repositories: behaviours and technical recommendations of the coar next generation repositories working group [internet]. göttingen: coar; 2017 nov. available from: https://www.coar-repositories.org/files/ngr-final-formatted-report-cc.pdf [9] tonkin el, taylor s, tourte gjl. cover sheets considered harmful. information services & use [internet]. 2013 jan 1 [cited 2018 aug 29];33(2):129–37. available from: https://doi.org/10.3233/isu-130705 [10] acharya a. indexing repositories: pitfalls and best practices [internet]. proceedings of open repositories 2015. 2015. available from: http://purl.dlib.indiana.edu/iudl/media/6537033b6s [11] shirlaw d. university of strathclyde research rankings rocket [internet]. glasgow city of science and innovation – news. 2014 [cited 2018 aug 14]. available from: https://perma.cc/9cnk-8z53 [12] zhang t, maron d, charles c. usability evaluation of a research repository and collaboration website. journal of web librarianship [internet]. 2013 jan 1; available from: http://docs.lib.purdue.edu/lib_fsdocs/51 [13] mckay d, burriss s. improving the usability of novel web software: an industrial case study of an institutional repository. in: web information systems engineering – wise 2008 workshops [internet]. springer, berlin, heidelberg; 2008 [cited 2017 jul 18]. p. 102–11. (lecture notes in computer science). available from: https://link.springer.com/chapter/10.1007/978-3-540-85200-1_12 [14] wang j, senecal s. measuring perceived website usability. journal of internet commerce [internet]. 2007 aug 8;6(4):97–112. available from: https://doi.org/10.1080/15332860802086318 [15] pendell kd, bowman ms. usability study of a library’s mobile website: an example from portland state university. information technology and libraries [internet]. 2012 jun 12 [cited 2018 aug 3];31(2):45–62. available from: https://ejournals.bc.edu/ojs/index.php/ital/article/view/1913 [16] everard a, mccoy s. effect of presentation flaw attribution on website quality, trust, and abandonment. australasian journal of information systems [internet]. 2010 mar 1 [cited 2018 aug 3];16(2). available from: http://journal.acs.org.au/index.php/ajis/article/view/516 [17] nielsen j, molich r. heuristic evaluation of user interfaces. in: proceedings of the sigchi conference on human factors in computing systems [internet]. new york, ny, usa: acm; 1990. p. 249–256. (chi ’90). available from: http://doi.acm.org/10.1145/97243.97281 [18] google. search engine optimization (seo) starter guide [internet]. 2018 [cited 2018 aug 3]. available from: https://perma.cc/8ct3-uav5 [19] kloboves k. continuing to make the web more mobile friendly [internet]. official google webmaster central blog. 2016 [cited 2018 jul 19]. available from: https://webmasters.googleblog.com/2016/03/continuing-to-make-web-more-mobile.html [20] moreno l, martínez p. overlapping factors in search engine optimization and web accessibility. 2013 jun [cited 2018 aug 3]; available from: https://e-archivo.uc3m.es/handle/10016/20175 [21] macgregor g. feeding the beast: workloads in a hybrid ir / cris environment [internet]. open access @ strathclyde. 2017 [cited 2017 jul 19]. available from: https://perma.cc/dl7u-9vce [22] wang z, phan d. using page speed in mobile search ranking [internet]. official google webmaster central blog. 2018 [cited 2018 aug 3]. available from: https://perma.cc/8qkp-ne5s [23] macgregor g. demonstrating the need for speed: improving page loading and rendering in repositories [internet]. open access @ strathclyde. 2017 [cited 2018 aug 3]. available from: https://perma.cc/dcm7-ts7b [24] google. about data highlighter [internet]. 2018 [cited 2018 aug 24]. available from: https://perma.cc/92dy-mfzp [25] google. google search console [internet]. 2018 [cited 2018 aug 10]. available from: https://www.google.com/webmasters/tools/home [26] google. google analytics [internet]. 2018 [cited 2018 aug 13]. available from: https://marketingplatform.google.com/about/analytics/ [27] microsoft. how to build quality content [internet]. 2011 [cited 2018 aug 24]. available from: https://perma.cc/x2u7-qmpj [28] shewan d. dwell time: the most important metric you’re not measuring [internet]. 2017 [cited 2018 aug 24]. available from: https://perma.cc/5h5e-bte2 [29] tonkin e, taylor s, tourte g, web-support@bath.ac.uk. cover sheets considered harmful. in: 17th international conference on electronic publishing [internet]. blekinge: university of bath; 2013 [cited 2015 sep 18]. available from: http://www.bth.se/com/elpub2013.nsf/ data statement data underpinning this work are available under a cc-by license at: https://doi.org/10.5281/zenodo.1411207 about the author george macgregor (g3om4c@gmail.com) is the institutional repository manager at the university of strathclyde in glasgow, scotland (uk). george’s interests are in structured open data (esp. within semantic web and repository contexts), information retrieval, distributed digital repositories and human-computer interaction (hci). web: https://purl.org/g3om4c orcid: https://orcid.org/0000-0002-8482-3973 subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – large language models for machine-readable citation data: towards an automated metadata curation pipeline for scholarly journals mission editorial committee process and structure code4lib issue 60, 2025-04-14 large language models for machine-readable citation data: towards an automated metadata curation pipeline for scholarly journals northwestern university spent far too much time and effort curating citation data by hand. here, we show that large language models can be an efficient way to convert plain-text citations to bibtex for use in machine-actionable metadata. further, we prove that these models can be run locally, without cloud compute cost. with these tools, university-owned publishing operations can increase their operating efficiency which, when combined with human review, has no effect on quality. by aerith y. netzer background and motivation northwestern university libraries publishes two peer-reviewed journals, the bulletin of applied transgender studies, and studies in russian philosophy, literature, and religious thought. northwestern’s journal publishing operates under tight economic constraints—direct and opportunity—and therefore must solve the same problems of corporate academic publishers with a fraction of the resources available [1][2]. one of these problems is reference metadata, i.e., machine-actionable references that are then used to count citations of articles. the act of capturing, counting, and using citations accurately enables funding agencies, universities, and publishers to make data-driven decisions for funding allocation, reviewers to validate the research of a manuscript, and faster literature review. an example the workflow for our university—a medium-size, elite university in the midwest united states—consists of receiving manuscripts from authors in a microsoft word file format. we then use pandoc [3] to transform this word document to a markdown file format, from which we can build pdf and web versions from a single source. but due to manuscript author’s primarily writing their manuscript in microsoft word, this meant looking up each source, adding them to a zotero [4] library, and then exporting the bibtex file for use as metadata in the web version of the article. as northwestern libraries’ journal-publishing operation is a one-woman show and quickly growing in complexity and scope, we found it necessary to find a way to find a faster way. there have been many projects aimed to converting plain-text to bibtex using programmatic means, but are often limited to certain languages [5] or are dependent upon external data [6]. as northwestern’s journal submissions often use non-latin languages, such as sources in studies in russian philosophy, this is a limitation that precludes many of the sources necessary for us to translate into machine-readable text [7]. as large language models grew popular, we originally reached for the most popular option — the gpt-3 and 3.5 api. however, due to these popular options being paid, using this method would not be scalable to many journals with hundreds of citations to process per volume/issue. further, we as an organization prefer transparency and replicability in our tools. as gpt is managed by openai, access to the model can be closed at any time, forcing us to move to a new system. while with open-source systems, we can upgrade or downgrade as needs arise, and we need not pay. thus, we reached for another, more open, tool—ollama [8]. limitations & concerns this analysis is limited to works that appear in the crossref api, creating a bias in the dataset against older works and academic monographs. while this limits the usefulness of this analysis to publishers whose specialty lies within fields where citations are limited to recent works (such as the physical sciences), future work can and should include plain-text citations of historical, non-digital, and non-academic works. along with the rapid growth in users of large-language models, so have concerns over the ecological sustainability of llm technology [9][10]. most of these concerns, however, can be alleviated with the use of “small” models such as those provided by ollama. further, there are concerns about the validity of large-language models, especially concerning their propensity to hallucinate. however, in combination with validity checkers such as bibtexparser and human review, we are confident enough in this system to be used in production of our journals [11]. future work in this area should include building scalable, verifiable workflows that require less human oversight. methodology data [12] was collected via the crossref api. using the sample function of the crossref api, we retrieved a random doi. then, using the crossref content negotiation endpoint, we were able to retrieve a plain-text formatted citation from a randomly selected citation style from the following: chicago author-date elsevier-harvard ecoscience apa mla ieee council of science editors using the crossref api, we pulled the bibtex citation, the plaintext citation, and the doi to create a dataset for our analysis. table 1 presents the variables and their descriptions. table 1: citation metadata variable description doi the digital object identifier of the requisite work. bibtex citation metadata about the work in bibtex format. plain text citation the cited work in a given citation style. plain text citation style the style in which the plain text citation is given. using this random assignment of citation formats, we achieved a roughly even distribution of each citation style in the dataset (see figure 1). figure 1. pie chart demonstrating the proportion of each citation style present in the dataset. analysis all language models (see table 2) were tested using the ollama toolkit using the quest [13] supercomputing cluster at northwestern university, running in a singularity container [14][15]. testing of all models took 14 hours to complete on two nvidia a100 graphical processing units, one node with eight cores, and 128 gigabytes of memory[16][17]. all code was written in python using an anaconda environment to aid in reproducible deployments of this code [18].  we used the plain text citation given by the crossref content negotiation api as a ground truth to which the model would aspire. we prompted each model with the same text: you are a professional citation parser. given the following plain text citation: {plain_text_citation} please convert this citation into a structured bibtex entry. include all relevant fields such as author, title, journal, volume, pages, year, etc. output only the bibtex entry, and nothing else. do not include any explanations, preambles, or additional text. the following models were prompted in this analysis codegemma:2b codegemma:7b llama2:7b llama3.3:70b llama3:8b mistral:7b starcoder2:3b tinyllama these models were chosen to represent a range of model sizes and training methods. the following variables were saved to the output file of the model: table 3: study variables variable description model the model being tested. one of the eight models listed above. plaintextcitation maps to plain text citation field table 1. timetogeneration time taken to generate the entry. actualbibtex bibtex entry retrieved from crossref. totalfields the number of bibtex fields being compared in generated and “ground truth” entries. matching fields the number of fields that have a match in both the generated and “ground truth” entries. percentage match (overall accuracy) matching fields / total fields this generated 8 csv files of approximately 3,000 lines each. each row corresponds to a single doi. these files were used for analyzing the efficiency and effectiveness of each model. model effectiveness figure 2. per field accuracy and valid bibtex of the model. unsurprisingly, llama3.3:70b, the most advanced and largest model of the chosen models, performed the best. further, starcoder2:3b failed to create any valid bibtex entries, whereas every other model created valid bibtex for every citation. figure 3.box and whisker plot for timetogeneration by model model time to generation model median time to generation (seconds) standard deviation (seconds) codegemma:2b 1.18 3.24 codegemma:7b 1.08 0.29 llama2:7b 0.96 0.27 llama3.3:70b 5.45 1.90 llama3:8b 1.00 0.31 mistral 1.00 0.27 starcoder2:3b 0.00 0.00 tinyllama 0.57 0.64 as every model except for starcoder2:3b created valid bibtex perfectly, we are primarily concerned with the accuracy of the fields. in this discussion, the validity of the bibtex simply means that if the bibtex can be parsed without errors, then the bibtex is valid. however, a well-formed bibtex entry can be valid but incorrect. meaning that the entry can be parsed, but the data in the entry is wrong. llama3.3:70b generated the most accurate bibtex entries. however, we should not assume that the model was necessarily incorrect, but was just different from how crossref represented the field. mistral and codegemma, though, are very close behind, especially with their parameter sizes (and therefore cost of compute) much lower than llama3.3:70b, it may be economical for some publishing operations to use smaller models, decreasing their cost, while keeping parity with the accuracy of the model. trading a .2% reduction in overall accuracy for, on average, a 5x faster computation is an effective strategy for this use case. figure 4.per-field accuracy by model all models were very accurate in producing volume, year, and journal entries in bibtex, while author, publisher, and school were the least accurate fields. this is because there is greater freedom and flexibility in how these fields are entered, and thus a correct and valid generated bibtex need not be exactly the same as crossref’s representation of the same data. future work should include creating a validator to identify equivalent author, publisher, and school names. for example, consider the following bibtex entries: {national academy of sciences, the} {the national academy of sciences} while these entries refer to the same entity, they cannot be identified as the same programmatically, and are thus penalized as “inaccurate.” thus, the results in figure 2 should be interpreted as the models’ accuracy when using crossref as the metric of accuracy. this analysis is useful because it shows which models are better-suited for this task, rather than the concluding 50% of the fields to be incorrect. recreating results on consumer hardware while we ran these models on a supercomputer to aid in analysis, models with parameter sizes of less than 10 billion can be run on current consumer hardware. we recommend a computer with a dedicated, modern gpu (verified to work on the author’s personal nvidia 3080ti, amd ryzen 6-core cpu, and 32 gb of ram; and an apple m3 max with 36 gb of memory) for unix systems, use curl to install ollama with one command: curl -fssl https://ollama.com/install.sh | sh then pull the model (we recommend mistral): ollama pull mistral then, start an ollama server listening on port 11434: def generate_text_with_ollama(model_name, prompt): url = 'http://localhost:11434/api/generate' payload = { "model": model_name, "prompt": prompt, "temperature": 0, # make output more deterministic "stop": ["\n\n"] } headers = { "content-type": "application/json" } response = requests.post(url, headers=headers, data=json.dumps(payload), stream=true) # handle streaming response generated_text = '' for line in response.iter_lines(): if line: data = json.loads(line) if data.get('done', false): break else: generated_text += data.get('response', '') return generated_text.strip() you can then pass any input you like to this function and return a generated bibtex key. full code sample is in the author’s github repo. conclusion for university-owned publishers, small, locally-available llms are capable of producing well-formed bibtex. these models can be used to create machine-actionable citation metadata, automating a step in the publishing process. as of the publication of this paper, 7 billion parameter models, especially mistral, are capable of running on the latest laptops, and provide acceptable performance at the least cost. data and code availability the author strives to adhere to the fair guiding principles. code and data used for this analysis is available on github. references and notes [1] association of college & research libraries. “the state of u.s. academic libraries: findings from the acrl 2023 annual survey.” chicago: association of college & research libraries, 2024. retrieved from https://www.ala.org/sites/default/files/2024-10/2023%20state%20of%20academic%20libraries%20report.pdf [2] relx. 2023. “market segments.” relx. [3] “pandoc – index.” n.d. https://pandoc.org/. accessed march 8, 2024. [4] “zotero your personal research assistant.” n.d. https://www.zotero.org/. accessed december 31, 2024. [5] “makino takaki’s page – writings – technical tips – generate bibtex entry from plain text (.en).” n.d. https://www.snowelm.com/~t/doc/tips/makebib.en.html. accessed march 9, 2024. [6] “text2bib.” n.d. https://text2bib.economics.utoronto.ca/index.php/index. accessed march 9, 2024. [7] williams, rowan. 2024. “sergeii bulgakov, socialism, and the church.” northwestern university studies in russian philosophy, literature, and religious thought. [8] “ollama/ollama.” 2025. ollama. [9] ding, yi, and tianyao shi. 2024. “sustainable llm serving: environmental implications, challenges, and opportunities : invited paper.” in 2024 ieee 15th international green and sustainable computing conference (igsc), 37–38. https://doi.org/10.1109/igsc64514.2024.00016. [10] (chien, andrew a, liuzixuan lin, hai nguyen, varsha rao, tristan sharma, and rajini wijayawardana. 2023. “reducing the carbon impact of generative ai inference (today and in 2035).” in proceedings of the 2nd workshop on sustainable computer systems, 1–7. boston ma usa: acm. https://doi.org/10.1145/3604930.3605705. [11] “about the journal – bulletin of applied transgender studies.” n.d. https://bulletin.appliedtransstudies.org/about/. accessed march 8, 2024. [12] [18] netzer, aerith. “aerithnetzer/biblatex-transformer.” 2025. https://github.com/aerithnetzer/biblatex-transformer. [13] “quest high-performance computing cluster: information technology – northwestern university.” n.d. https://www.it.northwestern.edu/departments/it-services-support/research/computing/quest/. accessed january 1, 2025 [14] kurtzer, gregory m., vanessa sochat, and michael w. bauer. 2017. “singularity: scientific containers for mobility of compute.” plos one 12 (5): e0177459. https://doi.org/10.1371/journal.pone.0177459. [15] singularity containers allow for reproducible environments for analysis. the singularity definition file used in this analysis can be found in the github repository. as llms are typically stochastic in nature, one can expect to have reasonably similar, but not exactly the same, results as presented in this paper. the singularity definition file primarily serves as a resource for readers to deploy this system in their home institution. [16]the script used to create the slurm job can be found in the github repository https://github.com/aerithnetzer/biblatex-transformer. [17] thank you to kat nykiel at purdue university for her assistance in building and deploying the singularity container. about the author aerith y. netzer is the digital publishing and repository librarian at northwestern university in evanston, illinois. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – editorial : the cost of knowing our users mission editorial committee process and structure code4lib issue 52, 2021-09-22 editorial : the cost of knowing our users some musings on the difficulty of wanting to know our users’ secrets and simultaneously wanting to not know them. by mark swenson for the second time in the past year we have an article that provokes concerns with patron privacy. despite some reasonable objections to printing this article because of the implied endorsement of a cavalier attitude toward patron data, the editorial committee has decided that the overall quality of the article merits its publication. we have added a note to the top of the article which acknowledges that there are considerable problems with elements of patron privacy. in this editorial i will mention some of the problems that come with using patron data in this way, as well as express some doubt that there are technical solutions to handling this data that could make it truly anonymized. as agencies with limited resources and a base of users that need specific information services, libraries are caught in a hard place when it comes to collecting data on user behavior to improve services. we have always collected data about our collections and users. many of the articles which run in this journal focus on how to dissect, clean, parse, and use that data. however, we have entered a pact with our users which states that what they individually do will never be divulged to anyone else. consequently library user data tends to be very broad and generic. for example, we know that a given book has been checked out 100 times, but we don’t have any idea if the book was checked out by 100 different people, by 50 people twice, or by one person 100 times. we know that our computer systems have the granular information that can answer this kind of question. so it’s possible to mine that and create new lists linking users and resources and get the kind of detailed use information that would be great for evaluating a resource. this is what this issue’s controversial article describes. in doing this, however, a library creates a record that theoretically makes it much easier for someone to misuse patron data and turn the question around: asking not what resources are being used by individuals but which individuals are using specific resources. when we create data like this it becomes difficult to control its future. we may have the best intentions in doing it, but it’s hard to be sure that everyone who can obtain it will respect the privacy of users equally. this is incredibly frustrating, so we keep delving into this data mine to figure out a way that maybe we can get that kind of information about users while somehow keeping their information anonymous. if we could do that, then maybe we could know if it would be worthwhile using limited resources to buy the next book in the series since 100 people will probably check it out too, or if it is more likely that we will just be buying it for the one person. as a thought experiment i want to briefly imagine exactly what we’d need to do to successfully have truly anonymized individualized user data, and why ultimately this probably isn’t possible. my goal here isn’t to sketch an actual viable plan, but rather to just show how hard this really is. because it’s a good example of a scenario where this is at least imaginable, i want to use the case of database access logs. from the logs we are taking only the following information: a user’s barcode, the name of the resource, and the date and time on which the resource was accessed. it is important to note here one of the many problems that libraries have with trying to maintain user privacy is that the logs in question are often not under our direct control. in such cases vendors may be collecting this information and not providing control to libraries as to what data is being collected, how long that data is kept, and how frequently that data is deleted. without control over that data, even with a hypothetically perfectly anonymized set of logs, the promise of anonymity granted by them is worthless. either the data could be obtained directly from the vendor or any anonymization rendered worthless by comparing the records to vendor data. in most cases with vendor-collected data i think that it should be a high priority for libraries to establish policies with vendors that value patron privacy first and foremost. for the purpose of this thought experiment we will imagine data which the library has complete control over. in reality, such a limitation would make doing the rest of the work to obtain the holy grail of anonymized data largely pointless, but it’s the only good starting point. let us also assume that we are working with a large population of over ten thousand users (i work in a suburban public library and a user base of this size is typical here) and a modest collection of resources. this scenario avoids problems one would be likely to encounter at a small academic institution, for example, where it might be possible to identify a user because the number of persons interested in a topic is small and there might be one resource that is almost exclusively used by them. first, i need to at least obfuscate the barcodes, the single piece of data that can be tied to an individual. if i am storing this in a database, and the database is hacked or requested as the result of a legal order, i don’t want to lose control of a list of barcodes that could then be used to figure out the identities of individual users. with modern technology, the best way to obfuscate some data is to use a cryptographic hash, turning my 14 digit barcodes into a much longer (maybe 32-72 character) random blob of letters and numbers. at this point things are looking good, but what if the person who gets the data wants to know if a specific person, whose barcode they already have, is listed in this database. it would be trivial for them to figure out which hashed value matches that barcode just by figuring out the hash algorithm, of which there are just a handful in common use. so to protect against that i’m going to need to add a salt (some extra random data) to the barcodes to make them harder to guess. it is here that the difficulty facing the person who wants to both protect and use this information becomes apparent. if i’m collecting these barcodes over a period of time and want later to be able to run a report that determines which resources are being used by multiple users and which ones are only being used by one, i need to always use the same salt. but if i use the same salt all of the time, that needs to be stored in a place where it could be seen by the same person who has a copy of database. i can generate a unique salt for every single record in the database and encrypt that salt with a public key. that should thoroughly obscure the information and it should be impossible for anyone to figure it out unless they got my private key. in theory, i can collect this data and only remove the encryption in a safe way to create a nice anonymized report. but who else has access to this private key? how can i keep it safe while not putting my library in a position where the ability to run reports on this set of data requires my secret knowledge. also, when i run a report and start to observe patterns in the data, even with a large population, certain users may be much easier to identify than i would expect. it might be obvious to a worker at a service desk that an anonymous user who used two dissimilar resources in a short timeframe would likely be a specific person. with that problem in mind, it becomes necessary to make the user id hashes unique for each resource. to do this, the barcode, the database name, and the salt all have to go into the hashing process, and then the salt needs to be encrypted and stored for later retrieval. other problems with the private key and data ownership remain, but we’ve maybe, finally gotten to a point where if the stars align correctly, and there are no flaws in the encryption, and the code is all written perfectly, this information’s pretty anonymized. that’s a really hard place to get to, and with the remaining problems with the private key and its ownership, a hard place to stay. on top of what’s already been mentioned in trying to get a balance between restricted access to the key and keeping it safe, it would still be vulnerable to potential warrant requests from law enforcement or vulnerable to unauthorized access on a hacked network. as much as we would really, really like this data, is it worth it to go here? i’m not so sure. subscribe to comments: for this article | for all articles 3 responses to "editorial : the cost of knowing our users" please leave a response below: kristin briney, 2021-09-22 this editorial fails to address the actual issue with the article in question: the violation of user privacy (namely, the privacy to access resources without surveillance). instead, the editorial lays out a technical solution to an ethical problem, except that anonymization isn’t actually a solution. it has been shown by researchers such as narayanan that anonymization simply doesn’t work. the larger issue here is that code4lib published an article that violates patron privacy and justified this violation with a theoretical technical exercise in how we can better protect data that should never have been collected and analyzed in the first place. patron data is not the new black or the new oil. as becky yoose says, data is glitter: it gets everywhere, it’s hard to clean up, and it’s best to never even let it into your house. under this analogy, code4lib just got glitterbombed and needs to clean up a mess. melissa belvadi, 2021-09-23 i strongly disagree with this statement: “it should be a high priority for libraries to establish policies with vendors that value patron privacy first and foremost”. rather i think that first and foremost we should establish such policies that value informed patron *choice*. let our patrons decide where each wishes to make the tradeoff between features and privacy. to adopt your value, we should be negotiating with every publisher to disable on their platforms the features that allow patrons to optionally create “my researcher” accounts, which usually involve giving their email address, which is a fairly unique identifier easily linked to their human existence. the generations that have widely adopted the use of facebook and the like have their own values about privacy and we have no business imposing our own more restrictive ones at their expense. becky yoose, 2021-09-27 “despite some reasonable objections to printing this article because of the implied endorsement of a cavalier attitude toward patron data, the editorial committee has decided that the overall quality of the article merits its publication.” any library worker making such a statement must reexamine their commitment to protecting a patron’s right to privacy at the library. the decision to sum up these critical privacy issues as an “implied endorsement of a cavalier attitude toward patron data” indicates that the editorial board places little to no value on privacy. instead, they chose to champion code over all else, patrons and professional ethics be damned. as the editorial states, this is the *second* article to have significant patron privacy issues. after the publication of the first article, the editorial committee made a statement that they would “pull in outside experts to comment on articles.” [1] when i was called to review the article with a very short turnaround (the email stated that the article was slated to be published later that week), i thought there was some progress in incorporating privacy checks in the review process. instead, the article was published as-is with one little note on the top of the article stating that there were concerns. in addition, it was implied later on that i would be more than welcome to clean up my feedback to turn it into a more formal, publishable article. i would say that the editorial process is broken, but perhaps it’s working by design. the more i learn about the editorial process in the journal, the more concerned i am about the ability of the editorial committee’s commitment in creating meaningful, substantial change to the process. it is standard for many library journals to ask authors to revise and resubmit articles. i recommended that the article be heavily edited to address the multitude of privacy issues if the article is published at all. [2] however, the current editorial review structure seems not to accommodate revise and resubmit. the implied expectation that i would contribute a rebuttal of my own in the form of article submission or other publication completely misses the point about why this article shouldn’t have been published as is in the first place. *it describes in detail how to violate professional ethics and patron privacy*. a debate in the scholarly literature doesn’t undo the harm of such an article in the scholarly record. even if the original article was framed in the context of privacy and ethics, any debate would not and must not be a substitute for an ethics review during the review process. disregarding a patron’s right to privacy seems to be becoming a pattern here, particularly when you brought someone to point out the privacy red flags only to ignore them. i have been aware that the editorial committee is revising guidelines for guest editors; however, i am not optimistic that this revision for “guests” would solve the root issues present in the overall editorial process. i ask that the editorial committee make the following changes: 1. create a code of ethics or ethical guidelines for submission authors – technology isn’t neutral. librarianship has a code of ethics. patrons have rights in the library. anyone working in library technology has to recognize all of this and reflect this in their scholarly output. 2. create a rubric or other mechanisms to evaluate submissions on their adherence to professional ethics, including the patron’s right to privacy – library technology journals are not neutral, either. what is and is not published here shapes the library technology discourse. people in the profession consider this journal as the prestige library technology journal. if this journal doesn’t reflect the profession’s ethics or protect the rights patrons have in the library, it codifies this disregard of ethics and patron rights in the scholarly record. 3. make more use of “revise and resubmit” during the review process, or make more decisions not to publish if the ethical issues in the original submitting ultimately cannot be resolved or adequately addressed in the submission. 4. revise the process for bringing in external reviewers – please, for the love of everything good and holy, do not bring in external reviewers four days before schedule publication, only to ignore their input. ideally, the ethics review would trigger a need for external review early in the editorial process. some people have called for the editorial committee to apologize to me for the treatment i received during this process. i do not want an apology. i want fundamental changes to the submission and editorial processes. if the editorial committee is committed to meaningful change, an excellent first step would be a full retraction of the article in question. only then any progress on ethical guidelines for submissions and reviews can start in earnest. [1] https://journal.code4lib.org/articles/15340#comment-2745195 [2] https://journal.code4lib.org/articles/16087#comment-2745444 leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – using book data providers to improve services to patrons mission editorial committee process and structure code4lib issue 6, 2009-03-30 using book data providers to improve services to patrons at paul smith’s college, i recently implemented a “new books” display using open apis and an image scroller. in this article i’ll give a brief overview of google book search, openlibrary and worldcat, explain how i created this new books widget using book cover data, and provide readers with some practical and simple code to show how to collect this data. this article will be of interest to anyone who wants to read about a brief overview of current state of free book data service providers. additionally, beginner programmers will likely find the examples at the end of the article helpful when getting started with projects of their own. by mike beccaria introduction with the advent of large book data repositories such as google book search, open library and worldcat along with the application programming interfaces (apis) that accompany them, programmers are quickly developing new tools to enhance patron user experiences at library web sites. at paul smith’s college, i recently implemented a “new books” display on the front of our web page using some of these open apis in conjunction with a very affordable 3rd party application used to create an image scroller. in this article i’ll give a brief overview of the some of the larger book data providers, show how i used some of this data to create a new books widget at paul smith’s college, and provide readers with some practical and simple code to show how to collect this data. this article will be of interest to anyone who wants a brief overview of current state of free book data service providers and some projects that were created using them. additionally, beginner programmers will likely find the examples at the end of the article helpful when getting started with projects of their own. introduction to 3rd party data providers a trend in online service providers has been to create application programming interfaces (apis) for software developers. apis are simple ways for computers to give and take data from each other, allowing developers to use the data and repackage it in different and creative ways. worldcat, google book search, librarything and open library have extensive data repositories with metadata on millions of books. each of them has an api that allows developers to query, collect and use only the information they need. additionally, some of these providers (namely google and open library) have been offering full or limited access to the actual scanned text of some books. libraries that recently had access to these books only in print now have the ability to point their users to online copies of these books and allow them to search and see them without having to travel to the book shelves. i recently queried open library with a subset of books we house in our library archives and found that between 30-50% of my queries reached a book that had been scanned, placed online, and was available for viewing or download in full-text. by simply adding a link in our catalog records for these books, we can allow our patrons to have access to books that were recently only able to access with special permission. let’s take a brief look at some of the major data providers and discuss some of the offerings of each. google book search api google book search api documentation can be found at http://code.google.com/apis/books/. google book search (gbs) api allows you to programmatically embed book previews on your websites, gather social information such a book reviews, ratings, labels, and user library data, and perform searches on the gbs database to get back detailed results on the books. this allows users to, for example, query the gbs database by keyword, isbn or oclc number and receive back general record information such as title, author, and publisher, etc. as well as whether google has a thumbnail cover image or a book preview is available. some uses of google book search many institutions and organizations are using gbs in creative ways to enhance the products and services they deliver. the strength of gbs that sets it apart from the other services providers is its embedded preview capabilities. perhaps the most notable example of gbs uses related to library services is found in oclc’s worldcat. when viewing an item level record in worldcat, users can see the “google preview” button located in the “get it” section of the page when a preview is available. figure 1: a screenshot from an item record in the worldcat catalog showing the google preview icon when a preview is available. [view full-size image] when clicked, the user is brought to a new worldcat page with the google preview embedded in the site. figure 2: google book preview embedded into the worldcat catalog website. users can search and view pages. [view full-size image] gbs limits the pages that users are able to see, but still allows access to a significant portion of the book. users are also able to search the full content of the book. worldcat search api worldcat search api documentation can be found at http://www.oclc.org/worldcatapi/. the worldcat api provides access to the worldcat database using restful uri queries that return the results in a variety of formats, including rss, atom, marc xml and dublin core. the worldcat api is free to all cataloging members with a subscription to connexion and requires member libraries to apply for an access key that is used when performing queries. the worldcat database contains over 100 million records written in 470+ languages from 112 countries in every conceivable physical and electronic format, all of which are accessible via the worldcat search api [1]. the clear strength of the worldcat api is the vast amount of information that is available and the flexibility of the api itself. some uses of worldcat search api perhaps the most notable demonstration of the api that illustrates its power and flexibility is a prototype online public access catalog (opac) that was built completely using api commands to worldcat. david walker, from california state university, developed this prototype and gave a presentation at code4lib 2008 titled “working with the worldcat api” [2]. david was able to create a completely customized user interface and populate it with data from worldcat. at paul smith’s college we use the worldcat api to populate our item level displays in our opac with bibliographic citations. figure 3: an item level view of the paul smith’s college opac showing bibliographic citations for the book downloaded via the worldcat api. [view full-size image] open library api open library api documentation can be found at http://openlibrary.org/dev/docs/api. open library, a project of the internet archive, has the goal of creating one page for every book ever published. they currently have 20 million available records and over 1 million books available in full-text [3]. the entire project is open, including the source code that runs the web page and all of the data in the database. open library has scanning centers that scan out-of-copyright books which are then added to the database. much of the data in open library comes from donated marc records from large university libraries. while still in beta, the open library api is a great source of data for projects that can leverage its api. some uses of the open library api while not as large as the worldcat database or providing as much copyrighted information as google, the strength of open library is found in its openness with the content it houses. it is the only source of bibliographic data that is both open source and open data, meaning all of its data and the software that displays it is available to download, use and reuse [4]. perhaps the most popular example of one of its uses is found in the openbookdata plugin for wordpress [5]. wordpress is a an open source blog software package and wordpress plugins are created by community members that allow users to extend the capabilities or wordpress. the openbookdata plugin allows users of wordpress to put book covers and book information from open library on their sites. for more information on the openbookdata plugin, please see john miedema’s article in issue 4 of code4lib journal [6]. at paul smith’s college, our thumbnails and some book information in the new books widget come directly from the open library api. librarything api librarything api documentation can be found at http://www.librarything.com/services/ and http://www.librarything.com/api. librarything is a for-profit company that lets book lovers create online catalogs of their books. librarything connects people who have similar reading interests by finding similarity between their libraries. users are able to rate books, write reviews and even have conversations about books online. librarything’s api comes in a variety of different packages and flavors ranging from web based xml queries to simple book thumbnail requests. their web services api allows users to download interesting and often unique information about books, and herein lies its most important strength. because a lot of librarything data is created by its users, it is often possible to collect some really unique information on books that might not be available elsewhere, such as first and last lines, awards and honors the book has received, important places in the book, and user book reviews. also of notable mention is the fact that librarything now has over 1 million user-uploaded book covers that are open and available to use by developers as well as data feeds that enable users to download certain sets of raw data and thumbnail images [7]. some uses of librarything api librarything provides javascript generation tools on their website to help users add librarything data on their web pages and blogs as well as getting librarything on your cell phone. in the librarything developers’ forum there is some talk of getting librarything applications for the iphone or ipod touch [8]. making a new books display at paul smith’s college at paul smith’s college, we recently undertook a major website redesign and we left a large center portion of our site available to advertise some of the services we offer to our patrons. one of the services we decided to start with was a new books widget that mirrored our new books display in the library. to make the library web page more interesting and useful to our users, we incorporated data that is freely available from open library and google book search to supplement the information already available in our library catalog. here are screenshots of our website before and after the new books widget was implemented: figure 4: paul smith’s college library website before and after the new books widget was implemented. [view full-size image] figure 5: paul smith’s college library website after the new books widget was implemented. [view full-size image] the user interface was created using a very inexpensive 3rd party javascript scroller creator called sothink javascript web scroller [9]. after modifying the scoller templates in the sothink product, i was able to collect data from a database populated with book information from our ils, google book search and open library. implementation the process between adding new books in the catalog to displaying them on the web page has many steps and components. here is an overview of the process that the new books application performs to create an automated new books widget on our library website: each night, our ils outputs a pipe delimited text file to a web accessible folder on the server. the file contains the title, author, and isbn of books that are newly added to our catalog. the web server downloads the created text file from the ils server. a python script reads the text file and, for each item, uses the 3rd party apis to download and store the relevant book data in a mysql database for use by the new books widget. if a thumbnail image of the book cover is found in the search process, it is also downloaded, renamed to match the isbn number of the book, and saved. the new books widget, created with the sothink javascript web scroller and a php script, queries the database for relevant new book information and populates the widget with the data as well as the saved thumbnails. when a user opens the library web page, the new books widget loads the thumbnail files and queries the required information from the database and displays it. i wanted to keep the book data separate from the tool so that if, down the road, we decide to change our web presence, the data would still be accessible to be used in a different form. additionally, keeping the data separate from the widget allows us to create other tools, such as rss feeds for new books; novel ways to display book covers and information, such as making a wall of books [10], creating a coverpop [11]; or using book data to support a digital signage display [12]. getting the book data the code for paul smith’s college’s new books widget will continue to be customized as our needs evolve. so, instead of delving into the specific details of how i made the widget, let’s look at some basic principles on how to get data from a couple of the content providers, namely google book search and open library, so that you can start making your own services to meet the needs of your users. one common search feature that each of these services has is the ability to query by isbn number. let’s walk through a python code sample of reading data from a text file that contains isbns and querying each of the services to retrieve book data and cover thumbnails. what you will need: a copy of python [13] installed on your computer. the code in the article has been tested on python version 2.5. (http://python.org/) the demjson python library. (http://deron.meranda.us/python/demjson/) note: at the time of writing this article, demjson does not currently work with python 3.0. a new version is in production. the process: our sample program will perform the following steps: read a text file that contains isbn numbers for each isbn, query open library and google book search api and retrieve back the data in a usable format. download and save book covers from open library and google book search if available. step 1: reading the isbn text file the first thing we have to do is load the python modules that we will be calling. #import some of the modules we will need to use to complete our tasks import urllib, urllib2, demjson, os the modules imported on the 1st line of the code snippet perform several different functions. the urllib [14] and urllib2 [15] modules provide functions that allow users to gather data from the internet and are used to query gbs and open library. the demjson module will allow us to quickly and easily take the json [16] data from gbs and open library and convert them into python variables that we can reuse in our programs. the os [17] module allows us to read and write to files which will be useful when we want to read the text file that contains isbn numbers as well as save the image thumbnail images that we collect. the urllib, urllib2 and os modules come with python 2.5. the demjson module needs to be downloaded and installed before you use it. given a text file of isbn numbers separated by carriage returns, we can read these items into a python list (also called an array in other programming languages). #create a function that will read lines from a text file and #store the contents in a list def read_newbooks_file(path): data = open(path) isbnlist = [] for isbn in data.readlines(): isbnlist.append(isbn.replace("\n","")) return isbnlist #read our text file and name the list "isbns" isbns = read_newbooks_file("c:\\newbooks.txt") #print our results print isbns if our text file found at c:\newbooks.txt contained the following isbns: 0618379436 9780700615582 9781593761288 9781405163354 this program would output the “isbns” list that looks like this: ['0618379436', '9780700615582', '9781593761288', '9781405163354'] we can now use this list to query the service providers for book information. step 2a: searching open library open library api documentation can be found at http://openlibrary.org/dev/docs/api the open library api allows users to query its database in a number of ways. the result of an isbn query returns unique ids to the book that are specific to the open library database. in order to get actual book data from open library, we first need to query the database for items that match our isbn. open library will return the unique id values back which we could then use to find book specific data for each title. open library returns json formatted data which, after it has been gathered, needs to be parsed and converted to python using a json library. there are several json libraries available to use for python, but i chose to use demjson. the following code cycles through our isbn list and queries open library to see if it has the books in their database. #cycle through each isbn in the list for isbn in isbns: #create the url string to query openlibrary url="http://openlibrary.org/api/search?q={%22query%22:%22(isbn_10:(" + isbn + ")%20or%20%20isbn_13:(" + isbn + "))%22}" #send the url request and name the json response "response" response=urllib.urlopen(url) #translate the json response into a python variable of dictionaries and lists book=demjson.decode(response.read()) #if the result set isn't empty (i.e. it returned a hit) if book["result"]!=[]: results = book["result"] #value of python dictionary value returns a list #print out the list print results #take the 1st item in our list, the open library key for the book and create a url to query again url = "http://openlibrary.org/api/get?key=" + results[0] #send the url request and name the json response "olresult" olresult=urllib.urlopen(url) #translate the json into a python variable data=demjson.decode(olresult.read()) #print it print data when you run this code, it should output a dictionary object with the book data embedded in it. here is sample output: [u'/b/ol7604987m'] {u'status': u'ok', u'result': {u'publishers': [u'houghton mifflin'], u'languages': [{u'key': u'/l/eng'}], u'subtitle': u'and other stories of intriguing kitchen science', u'key': u'/b/ol7604987m', u'title': u'how to read a french fry', u'number_of_pages': 320, u'isbn_13': [u'9780618379439'], u'isbn_10': [u'0618379436'], u'publish_date': u'september 8, 2003', u'last_modified': {u'type': u'/type/datetime', u'value': u'2008-04-29 13:35:46. 87638'}, u'authors': [{u'key': u'/a/ol2688000a'}], u'type': {u'key': u'/type/edition'}, u'id': 10350312, u'first_sentence': {u'type': u'/type/text', u'value': u'everyone loves deep-fried foods, as a glance at any fastfood menu will prove.'}, u'revision': 1}} step 3a: getting book covers from open library now that we have the book data, we need to download the book cover image. open library stores its book covers in small, medium and large formats and uses the open library unique ids in the url. you can request different book cover image sizes by changing the url of the image. ending the url with a –s, -m, or –l provides you with small, medium and large images respectively. the following urls point to thumbnails with an open library unique id of ol7604987m: small: http://covers.openlibrary.org/b/olid/ol7604987m-s.jpg medium: http://covers.openlibrary.org/b/olid/ol7604987m-m.jpg large: http://covers.openlibrary.org/b/olid/ol7604987m-l.jpg because we already have the open library unique ids from our previous code section, we simply need to add a few lines of code to download the book covers. when open library doesn’t have a book cover, it returns a small file with a length of 808 bytes. in the code below, we check this by downloading the file and checking its size (in the case of the code below, we check if it’s smaller than 1000 bytes). if the file is smaller than 1000 bytes we know we have an invalid thumbnail and delete the bad file. #the url to the thumbnails end with -s for small, -m for medium, -l for large imgurl = 'http://covers.openlibrary.org/b/olid/' + results[0][3:] + '-m.jpg' #retrieve the file and save it to c:\ with the name of the isbn as the name of the file imgfile = urllib.urlretrieve(imgurl, "c:\\" + isbn + ".jpg") #retreive the file size and if the file size is less then 1000 bytes, delete it fsize = os.path.getsize(imgfile[0]) if fsize < long(1000): os.remove("c:\\" + isbn + ".jpg") &#91;/sourcecode&#93; notice on the 1st line of the code i needed to add the "<code>[3:]</code>" to remove the 1st three characters from the open library ids, because the api returns the ids with a "<code>/b/</code>" appended to the front (i.e. "<code>/b/ol7604987m</code>") and we only need to use the id (i.e. "<code>ol7604987m</code>"). after running the program, i found three book thumbnails in my c:\ folder with isbn numbers as their name, indicating that one of our books doesn't have a thumbnail in open library: <p class="caption"><img id="figure6" title="figure 6" src="https://journal.code4lib.org/media/issue6/beccaria/figure_6.png" border="0" alt="figure 6" width="376" height="565" /> <strong>figure 6: three book thumbnails downloaded from open library.</strong> <h3>step 2b: searching google book search</h3> google book search api documentation can be found at <a href="http://code.google.com/apis/books/">http://code.google.com/apis/books/</a> google book search provides an api that allows users to get access to the metadata associated with books in its library. google has partners with a number of publishers and large academic libraries and has been scanning a large number of books and adding them to their collection [<a id="ref18" href="#fn18">18</a>, <a id="ref19" href="#fn19">19</a>]. depending on copyright and publisher permissions, google allows snippets, previews or full-access to books scanned into its database. at paul smith's college, we are using gbs information in our open source opac [<a id="ref20" href="#fn20">20</a>] as well as in our new books widget. if a google book preview is available, an icon is shown under the book title in the results page. in our new books widget, when a user hovers over a book cover with their mouse, a popup is shown with more detailed book information. for both tools, if a preview is available from google, an icon is shown with a link to the book on the google books site. <p class="caption"><a title="figure 7" href="https://journal.code4lib.org/media/issue6/beccaria/figure_7.png"><img id="figure7" src="https://journal.code4lib.org/media/issue6/beccaria/figure_7_tb.png" border="0" alt="figure 7" width="500" height="291" /></a> <strong>figure 7: when a user hovers over a book cover with their mouse, a popout window displays further information and, in this case, a google preview link.</strong> [<a href="https://journal.code4lib.org/media/issue6/beccaria/figure_7.png">view full-size image</a>] the gbs api allows you to perform searches and get back book information, reviews, ratings, labels, and user libraries, as well as embed google book previews on your website. the documentation on the google site is quite extensive and allows the potential for some great tools to be made. to get started, we are going to continue our book code sample by gathering data that indicates whether google book search has a preview and a cover image for a given book. gbs api allows http get requests to be made with oclc, isbn or lccn numbers and it returns data in json format. because we already have isbn numbers for our books, we will use those to search gbs in our example. here is a sample url request that will receive data back from google: <a href="http://books.google.com/books?bibkeys=isbn:0451526538&amp;jscmd=viewapi&amp;callback=mycallback">http://books.google.com/books?bibkeys=isbn:0451526538&amp;jscmd=viewapi&amp;callback=mycallback</a> google returns json-formatted data with these elements: <pre>jsonsearchresult { string bib_key; string info_url; string preview_url; string thumbnail_url; string preview; };</pre> starting where we are cycling through our isbns, let's take our example code and append a section that returns google book information. [sourcecode language='python'] #create parameters that will be used in the url sent to google. the results will be called gcallback. gparams = urllib.urlencode({'bibkeys': isbn, 'jscmd':'viewapi','callback':'gcallback'}) #open a new urllib2 handler and send the request opener = urllib2.build_opener(urllib2.httphandler()) request = urllib2.request('http://books.google.com/books?%s' % gparams) #we need to change the headers to trick google to think that firefox is sending the request opener.addheaders = [('user-agent', 'mozilla/5.0 (windows; u; windows nt 5.1; en-us; rv:1.9.0.3) gecko/2008092417 firefox/3.0.3')] #read the response from google g = opener.open(request).read() #print the results print g #if the results aren't empty, strip the "gcallback" parameter from the json results leaving only the relevant data if g != "gcallback({});": g = g[10:-2] #translate the json into a python variable gbookinfo=demjson.decode(g) #google only sends values that have data, so we need to check if that data #exists before we can print it or else python will throw an error. #print the url to the google page and the url to the thumbnail if gbookinfo[isbn].has_key("info_url"): print "gb info url: " + gbookinfo[isbn]["info_url"] if gbookinfo[isbn].has_key("thumbnail_url"): print "gb thumbnail url: " + gbookinfo[isbn]["thumbnail_url"] notice that i didn’t use the typical urllib python library in the same way that we did for open library. after some trial and error, i realized that google wasn’t returning results to me despite the fact that the program was structurally correct. google’s api would not accept the http headers that were being sent by the python module. i had to trick google into thinking the request was coming from the firefox web browser by changing the headers. also note that the gbs api only returns results when it actually has the data (i.e., it will not return any thumbnail url data if it doesn’t have any, versus returning an empty string). i had to check if the results included the key/value pairs that i was looking for (i.e., “info_url ” or “thumbnail_url”) or python will throw an error. the above code sample should output the url’s to the google books information page and the url to the book cover thumbnail for the book in question. e.g., the code outputs urls such as the following: gb info url: http://books.google.com/books?id=qd0qaaaacaaj&source=gbs_viewapi gb thumbnail url: http://bks8.books.google.com/books?id=qd0qaaaacaaj&printsec=frontcover&img=1&zoom=5&sig=acfu3u2mmlbje9kpba16k_rrafmukmn62q step 3b: getting book covers from google book search let’s see how to download the thumbnails and add them to our folder with the images from open library. because google doesn’t accept python headers when requesting objects, we need to send firefox headers again so google will allow us to download the image thumbnails. google book search returns the image url in the “thumbnail_url” value. here’s the code to complete our program: #we need to send a request to google and change the headers to imitate firefox opener = urllib2.build_opener(urllib2.httphandler()) request = urllib2.request(gbookinfo[isbn]["thumbnail_url"]) opener.addheaders = [('user-agent', 'mozilla/5.0 (windows; u; windows nt 5.1; en-us; rv:1.9.0.3) gecko/2008092417 firefox/3.0.3')] #and download it and save it with a -g appended to the end to indicate it came from google picfile = open("c:\\" + isbn + "-g.jpg", "w+b") picfile.write(opener.open(request).read()) the end product of the complete program should be book data stored as python objects that can be used easily in any programs you write, as well as downloaded thumbnail images, with those saved by gbs having a –g appended to the end of the file name. you are now only a few small steps away from creatively using this data. hopefully this sample got you started. figure 8: six thumbnail images, three from open library and three from google book search. [view full-size image] to make the sample easier to use, here is the complete version of the program with all of the comments removed. import urllib, urllib2, demjson, os def read_newbooks_file(path): data = open(path) isbnlist = [] for isbn in data.readlines(): isbnlist.append(isbn.replace(“\n”,””)) return isbnlist isbns = read_newbooks_file(“c:\\newbooks.txt”) print isbns for isbn in isbns: url=”http://openlibrary.org/api/search?q={%22query%22:%22(isbn_10:(” + isbn + “)%20or%20%20isbn_13:(” + isbn + “))%22}” response=urllib.urlopen(url) book=demjson.decode(response.read()) if book[“result”]!=[]: results = book[“result”] print results url = “http://openlibrary.org/api/get?key=” + results[0] olresult=urllib.urlopen(url) data=demjson.decode(olresult.read()) print data imgurl = ‘http://covers.openlibrary.org/b/olid/’ + results[0][3:] + ‘-m.jpg’ imgfile = urllib.urlretrieve(imgurl, “c:\\” + isbn + “.jpg”) fsize = os.path.getsize(imgfile[0]) if fsize < long(1000): os.remove("c:\\" + isbn + ".jpg") gparams = urllib.urlencode({'bibkeys': isbn, 'jscmd':'viewapi','callback':'gcallback'}) opener = urllib2.build_opener(urllib2.httphandler()) request = urllib2.request('http://books.google.com/books?%s' % gparams) opener.addheaders = [('user-agent', 'mozilla/5.0 (windows; u; windows nt 5.1; en-us; rv:1.9.0.3) gecko/2008092417 firefox/3.0.3')] g = opener.open(request).read() print g if g != "gcallback({});": g = g[10:-2] gbookinfo=demjson.decode(g) if gbookinfo[isbn].has_key("info_url"): print "gb info url: " + gbookinfo[isbn]["info_url"] if gbookinfo[isbn].has_key("thumbnail_url"): print "gb thumbnail url: " + gbookinfo[isbn]["thumbnail_url"] opener = urllib2.build_opener(urllib2.httphandler()) request = urllib2.request(gbookinfo[isbn]["thumbnail_url"]) opener.addheaders = [('user-agent', 'mozilla/5.0 (windows; u; windows nt 5.1; en-us; rv:1.9.0.3) gecko/2008092417 firefox/3.0.3')] picfile = open("c:\\" + isbn + "-g.jpg", "w+b") picfile.write(opener.open(request).read()) [/sourcecode] conclusion we have just begun to scratch the surface of gathering data from content providers and finding new ways to present it to our patrons. this article was meant to explain how we at paul smith’s college created a new books widget and added supplemental data to our book catalog using many of the same principles described in the provided code samples. we covered an overview of some of the larger content providers, gave an outline of how we used data from those content providers, and provided some building block code samples to get the beginning programmer started. book data downloader at google code if you are interested in delving further into downloading and storing book data from these repositories, you can get a head start by downloading the source code to the program i have made that runs the book data download portion of the new books widget at paul smith’s college. this program is designed to act much like the example in this article with additional components that store the downloaded data into a mysql database for future use. additionally, the program is organized in such a way that allows additional repositories to be added easily. the new books widget code can be downloaded from the google code repository at http://code.google.com/p/bookdatadownloader/. references [1] worldcat search api. <http://www.oclc.org/worldcatapi/default.htm> [2] david walker, working with the worldcat api, code4lib conference 2008. <http://code4lib.org/conference/2008/walker> [3] about the open library. <http://openlibrary.org/about> [4] about open librarianship. <http://openlibrary.org/about/lib> [5] john miedema. openbook book data wordpress plugin. <http://wordpress.org/extend/plugins/openbook-book-data/> [6] john miedema. openbook wordpress plugin: open source access to bibliographic data. the code4lib journal, issue 4. <http://journal.code4lib.org/articles/105> [7] librarything. cover images repository. <http://www.librarything.com/feeds/> [8] librarything forum. librarything on ipod touch. <http://www.librarything.com/topic/52201> [9] sothink javascript web scroller. <http://www.sothink.com/product/javascriptwebscroller/index.htm> [10] edward vielmetti. building a wall of books. higheredblogcon 2006. <http://www.higheredblogcon.com/index.php/building-a-wall-of-books/> [11] coverpop. <http://www.coverpop.com/> [12] greetsaver digital signage. ann arbor district library. <http://www.aadl.org/node/10631> [13] mark pilgrim. dive into python. <http://diveintopython.org/>. [14] urllib — open arbitrary resources by url python module. <http://docs.python.org/library/urllib.html> [15] urllib2 — extensible library for opening urls python module. <http://docs.python.org/library/urllib2.html> [16] douglas crockford. introducing json. <http://www.json.org/> [17] os — miscellaneous operating system interfaces python module. <http://docs.python.org/library/os.html> [18] google books library project. <http://books.google.com/googlebooks/library.html> [19] google book search partner program. <https://books.google.com/partner/> [20] paul smith’s college library catalog. <http://library.paulsmiths.edu/catalog/> about the author mike beccaria is the systems librarian and head of digital initiatives at paul smith’s college, a small baccalaureate institution located in the adirondack park in northern new york. subscribe to comments: for this article | for all articles 5 responses to "using book data providers to improve services to patrons" please leave a response below: matt weaver, 2009-03-31 i am using a drupal module called “book post”: http://drupal.org/project/bookpost in an online community for readers (just began testing). via book post, an isbn entered into a content type returns cover art and publishing info. the field is available in the views module, allowing for the slide show on the front page. matt weaver web librarian westlake porter public library (westlake, ohio) carol bean, 2009-04-11 i am a bit confused by the initial steps of the process. in the implementation section, it says, “1. each night, our ils outputs a pipe delimited text file to a web accessible folder on the server. the file contains the title, author, and isbn of books that are newly added to our catalog.” but in the process section, (step 1: reading the isbn text file), it states, “given a text file of isbn numbers separated by carriage returns, we can read these items into a python list (also called an array in other programming languages).” where does the text file of isbn numbers separated by *carriage returns* come from? is there another step which converts the pipe delimited file to a carriage return delimited file, or is this conversion handled by one of the python modules? thanks, carol course, 2009-05-19 could you provide me with some address that provides books mike beccaria, 2009-10-16 carol, the script in the article is a shortened and simplified script relative to the one i am using in my production setting. i wanted to make the article as simple to read and (re)use as possible, so i tried not to make it too complicated. modifying the script to take an input file from a specific ils or file format shouldn’t be too hard. the principles are the same. josh gachnang, 2011-10-13 one of the links at the bottom is broken. diveintopython.org is permanently down. it has been mirrored at diveintopython.net. please update your link. thanks! leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – from digital commons to oclc: a tailored approach for harvesting and transforming etd metadata into high-quality records mission editorial committee process and structure code4lib issue 33, 2016-07-19 from digital commons to oclc: a tailored approach for harvesting and transforming etd metadata into high-quality records the library literature contains many examples of automated and semi-automated approaches to harvest electronic theses and dissertations (etd) metadata from institutional repositories (ir) to the online computer library center (oclc). however, most of these approaches could not be implemented with the institutional repository software digital commons because of various reasons including proprietary schema incompatibilities and high level programming expertise requirements our institution did not want to pursue. only one semi-automated approach was found in the library literature which met our requirements for implementation, and even though it catered to the particular needs of the dspace ir, it could be implemented to other ir software if further customizations were applied. the following paper presents an extension of this semi-automated approach originally created by deng and reese, but customized and adapted to address the particular needs of the digital commons community and updated to integrate the latest resource description & access (rda) content standards for etds. advantages and disadvantages of this workflow are discussed and presented as well. by marielle veve introduction as many libraries have previously done, in 2013 the university of north florida (unf) library contemplated harvesting the etd metadata generated in their institutional repository, digital commons, to oclc with the goal of avoiding duplication of cataloging efforts and improving efficiencies. when the library explored the literature, we discovered many semi-automated and some completely automated approaches. however, most of these approaches could not be implemented to the digital commons platform for various reasons. the completely automated approaches, such as the worldcat digital collection gateway did not seem to work properly with the qualified dublin core (qualified dc) metadata feeds that originated from ir software that use proprietary metadata, such as digital commons and dspace[1] [2]. many of the semi-automated approaches we considered were rejected because they relied completely on proquest services to generate the initial etd metadata. unf, like some other libraries (mcmillan, halbert, & stark, 2013), does not subscribe to proquest for these services, as the practice of relying on proquest to generate initial etd metadata has been scrutinized recently in favor of student choice and open access options (clement, 2013). other approaches relied on tools that would require a high level of programming expertise to be implemented (park & brenza, 2015), a background many catalogers do not have. only one semi-automated approach was found that could be implemented in our institution’s case. this approach, developed by sai deng and terry reese (2009), is customized to address the specific needs of the dspace ir, but could be extended and implemented to digital commons software if further customizations are performed. these customizations would address the differences that exist in software capabilities, schema used, and element display differences between dspace and digital commons. the dspace workflow deng and reese’s workflow starts with the generation of descriptive metadata within the dspace ir software and exposes the result in qualified dc schema using the open archives initiative protocol for metadata harvesting (oai-pmh); the resulting metadata is then harvested and transformed into marc records with the assistance of an xslt stylesheet (extensible stylesheet language transformations) and the marcedit oai harvester. after the transformation, the generated marc records are imported into oclc by cataloging staff. we have been successfully using this approach to harvest etd metadata from our ir. the marcedit oai harvester tool has proved over time it can harvest metadata in any proprietary schema and from any ir software without any complications. designed with the library community in mind, this tool is user-friendly and can be implemented by anyone with or without programming expertise; plus its customer service is excellent. emails are answered within the same day by the tool’s creator or by other members in the discussion list. in addition, the tool has a huge community of followers and users who support its development in the long-term. even though deng and reese’s workflow can work smoothly with any ir that uses the dspace software, some major customizations were necessary to address the issues that arise when a different ir software is used. obstacles to digital commons ir software implementation in the case of unf, we encountered the following obstacles when trying to implement deng and reese’s workflow with the digital commons software: the qualified dc metadata exposed via oai-pmh by dspace is different from the qualified dc metadata exposed by digital commons. this is because each institution makes their own internal decisions regarding which metadata elements to use in their ir metadata template and which ones to expose via oai-pmh. in addition, dspace and digital commons have different capabilities for displaying the metadata elements. for example, dspace has the ability to display the etd advisors’ names in inverse order (deng, matveyeva, & wang, 2008) while digital commons cannot (digital commons representative, personal communication with author, october 13, 2015). the dspace ir uses a different set of qualified dc elements from the ones used by digital commons ir to map their etd metadata to oai-pmh. dspace uses elements from the default qualified dc in the dspace metadata registry while digital commons uses the bepress proprietary schema for qualified dc. the xslt stylesheet used in the deng and reese case study uses elements from the dspace proprietary schema and follows the older cataloging content rule standards, the anglo-american cataloging rules, 2nd edition (aacr2). in order to be implemented to the digital commons software, this xslt stylesheet would need to be adjusted to the bepress proprietary schema. we also needed to adjust it to reflect the current resource description & access (rda) content standards. given that the initial metadata generated by dspace will not be the same as the metadata generated by digital commons, we anticipated that an additional set of final edits would need to be performed on the marc records. even with these major obstacles, deng and reese’s approach still presents a good set of practical tools and ideas that can be useful to the digital commons ir community if special customizations and adaptations are implemented to address these differences, such as the use of a customized etd stylesheet and marcedit’s oai harvester. for that reason we included these tools as part of the workflow presented below, but with additional customizations performed at each stage of the harvesting workflow to address these issues. the workflow is also updated to integrate the current description standards for etds mentioned in rda. tailored approach for the digital commons platform: unf case study we designed the following workflow to transform our ir etd metadata into high quality, rda-compliant marc records that can be ingested by oclc. the transformed records have authorized headings (lcnaf) for the etd authors, college departments, and library of congress subject headings (lcsh). in addition, keywords supplied by the etd authors are included. we have designed this workflow to work out-of-the-box with minimal edits [3] so that it can be used by other digital commons ir users who do not want to rely on proquest services to produce the original etd metadata nor have a strong background in programming. the workflow consists of seven main steps we implemented with the assistance of a customized digital commons crosswalk for etd metadata (appendix a). digital commons workflow steps step 1. created list of fields desired in final etd record (a wishlist) — (columns a & b) the first step was to decide which descriptive metadata elements we wanted to display in our final marc records. examples of these fields are the authorized form of author, etd title, degree acquired, college, and name of advisors. appendix a, column a provides a list of these fields with column b displaying how they map to marc fields. step 2. customized digital commons etd submission metadata template — (column c) when looking at the original submission metadata template in digital commons, we noticed that some of the desired fields from the “wishlist” were already included there, while others were not. in order to include the additional desired fields that were not originally included, we created a new template and customized it to include as many of these desired fields as possible. creating this new template helps ensure quality metadata from the beginning, a good practice that helps reduce editing efforts and avoid future problems. column c in appendix a specifies the fields we added to the digital commons submission template—if not already there—while column d illustrates how these fields should be mapped to the qualified dc elements in bepress’s proprietary schema. examples of the additional fields we added to the template are: birth date lcnaf authors’ names lcnaf corporate body advisors’ names in inverted order controlled subject terms it is important to clarify at this point that most of these additional fields are only used to generate the final marc records sent to oclc. fields such as ‘birth date’ are hidden from the public display in digital commons but still mapped to the appropriate oai-pmh field. we added these fields to the etd submission template and mapped them to the elements specified in column d of appendix a by sending a request to the institution’s designated digital commons’ service representative. our new, customized digital commons template can be seen in figure 1. figure 1. customized digital commons etd template     step 3. mapped fields in etd submission template to oai-pmh — (columns c & d) digital commons exposes its metadata via oai-pmh in three schemas: “simple-dublin-core (dc), qualified-dublin-core (qdc), and oai_etdms” (bepress, 2015, p.2). from these three schemas, we selected qualified dc as the preferred schema to harvest the etd metadata for three main reasons. first, the metadata exposed in qualified dc provides the broadest set of elements with the highest level of flexibility for mapping customized fields. second, qualified dc is one of the most commonly used xml schemas in irs. according to park and tosaka (2010): “a trend of qualified dc being used (40.6 percent) more often than unqualified dc (25.4 percent) is noteworthy.” third, the qualified dc proprietary schema used by digital commons integrates the additional etd elements recommended by the networked digital library of theses and dissertations (ndltd) metadata standards (robertson, 2011). at this stage, we mapped the fields from the digital commons etd template (column c) to the appropriate bepress qualified dc elements in the bepress oai feed, requested from: http://digitalcommons.unf.edu/do/oai/?verb=listrecords&metadataprefix=qdc&set=publication:etd [4] appendix a presents our mapping of the elements in the digital commons template (column c) to their corresponding elements in the oai feed (column d.) for any of the fields that do not match this exact pattern, we requested a mapping re-configuration from our digital commons consulting services representative. some fields, like the marc fields 336, 337, 338, and 540, are constant for each record. for these fields (noted in column c with the value “will not be added at this point”), rather than mapping them in the oai, we added them later using an xslt stylesheet (column e). step 4. customized institution’s etd stylesheet (xslt) — (column e) the marcedit oai harvester tool provides a good set of generic xslt templates that can be used and customized to meet particular institutional repository data transformation needs. we used the xslt named “oaidctomarcxml” included in the marcedit 6 package. the stylesheet was adapted to etds and customized to include the additional fields desired in the final marc records and exclude the ones not wanted. examples of changes and additions performed to this original xslt and instructions on how to perform them are illustrated in column e of appendix a.  a copy of this finalized customized xslt stylesheet is available in appendix b. step 5. transformed qualified dc into marc metadata using an oai harvester & xslt we then use an oai harvester tool to transform the qualified dc metadata displayed in the oai feed to marc metadata with the assistance of the customized etd xslt stylesheet (appendix b). an excellent open access oai harvester is the marcedit oai harvester, which can crosswalk the files to marc records to be ingested by oclc. we use this tool to formulate a feed request for etd metadata in qualified dc from our digital commons repository, using the customized xslt stylesheet in appendix b. an example of this request is presented in figure 2. it is important to know that the “qdc” option used in this request example is not included in the default metadata drop-down menu in the marcedit oai harvester, however, typing “qdc” manually in the field will allow the harvester to complete the transformation. we selected the customized stylesheet in the crosswalk field and ran the crosswalk. this request will generate marc records that look like figure 3. figure 2. request for metadata using the marcedit oai harvester     figure 3. marc record generated for etds     step 6. performed final edits using the marc editor tool– (column f) even though most of the metadata at this point has been customized with the assistance of the initial ir template and the xslt stylesheet, there are still some final edits that need to be done. these edits are minor and can be performed in batch or by creating automated assigned tasks using marcedit’s marceditor. column f in appendix a contains a list of suggested final edits that can be performed. examples include separating the different lcsh that display in one 650 field into separate individual 650 fields (figure 4), fixing punctuation mistakes, substituting the word ‘yyyy’ in fields 502 and 008 for the actual year digits (figure 5), and removing keywords that only serve a local purpose and will not make sense outside the ir context (figure 6). figure 4. separating lcsh that display in one field into multiple 650 fields click to enlarge     figure 5. substituting the word ‘yyyy’ in fields 502 and 008 for digits     figure 6. automated assigned task to remove locally purpose keywords     step 7. imported records to oclc and perform final quality control — (column g) after final edits are performed with marcedit, we compile records back into marc-8 and import to oclc connexion. detailed instructions on importing and exporting records in batch to oclc are located in oclc’s documentation [5]. in oclc, we perform a final quality control of records by validating them. column g in appendix a contains a list of suggested things to look for during quality control. something that is of particular importance to check is that symbols in the abstract display well and that the abstract content was imported completely without errors. a common problem encountered with this particular xslt (appendix b) is that if the symbol “>” appears somewhere in the abstract, any data after the symbol will be dropped and will not be transferred to the new marc record during the harvesting process. this is a side effect of the xml coding “remove-html” that is used in xslt stylesheets to avoid transferring unnecessary html symbols to marc records. figure 7 final marc record of etd in oclc     advantages and disadvantages of this workflow: future improvements when implemented with digital commons repositories, the workflow presented in this paper will avoid duplication of efforts by generating high quality etd metadata from the beginning and in just one place. the time that is saved can be better allocated to perform other important etd tasks such as authority control for the etd authors and the assignment of controlled subject headings (lcsh) in addition to the subject keywords provided by the etd authors. for those interested in implementing this workflow, the tools are free and no programming or major xslt editing will be required, as most of the needed customizations have already been integrated. another advantage of this workflow is that it has the ability to separate multiple lcsh that may appear in one 650 marc field into individual separate 650 marc fields as well as separating the authors’ keywords (653s) from the controlled headings (650s). lastly, by avoiding the use of proquest metadata services and the oclc digital gateway, this workflow will enable other digital commons ir users to gain more control over the type and quality of etd metadata produced for these important and unique materials. in addition to this, bypassing proquest will channel access to the full-text of etds into one place, the institutional repository, instead of having multiple access routes to full-text that may divert traffic from the ir. on the other hand, this workflow presents two disadvantages. first, asking for authors’ birthdates in the etd metadata template may raise privacy concerns among some scholars, even though the practice of asking for birthdates during the etd submission process is not new to the profession and has been performed by libraries for a while. this privacy issue, can be addressed by either not requiring students to fill the birthdate field in the etd template or by blocking this field from the public display in digital commons. alternatively, using unique researcher identifiers such as the open researcher and contributor id (orcid) to identify etd authors has been suggested by some scholars. the orcid registry assigns a unique alphanumeric code to authors instead of using birthdates or authorized forms of headings. originally launched in october 2012, the registry offered a few mini-grants in 2013 for institutions that would be willing to implement and test the registry (tamu libraries, 2015). texas a&m university libraries and the university of missouri were two of these grant awardees that pioneered the orcid implementation into the etd domain in 2014. using orcid identifiers for etd authors, however, is a new concept that is still in its infancy and more research should be done to assess its applicability to bibliographic records in the marc schema environment. another disadvantage of this workflow is that names of the etd advisors have to be entered twice in the digital commons template. one form of the name is in direct order (firstname lastname) while the other one is in the inverted form (lastname, firstname). the reason for this duplication is that students enter their advisors’ names in direct order in the template while the inverted form is needed to generate the 700 fields for the marc records. when digital commons was approached by this paper’s author to display the etd advisors fields in the inverted order in the oai, they said they were unable to do so. the only solution would be to add an additional field to the etd template with the inverted form of the name [6]. an alternative solution to this problem is to include an xml coding in the stylesheet that can perform transformations for personal names, but doing so would be too complicated given all the possibilities a name can be displayed. reese included an example of this complex xml coding in one of his xslt stylesheets located at the oregon state university ir [7]. unfortunately, the coding for personal names used in this stylesheet cannot be used today without significant alteration, as it was created before the rda days and still follows the former aacr2 rule conventions. conclusion finally the workflow presented in this paper, even though it is customized and tailored to address the particular needs of the digital commons ir software, can be extended to other ir software through further customization to address the differences in ir software capabilities and schemata used. notes [1] s. wynne, s. mcintyre, & m. finn, discussion with author through metadatalibrarians mailing list, september 23, 2015. [2] a. harrell, w. robertson, & m. gibney, discussion at digitalcommons@google.groups, april 29, 2016. [3] the only customization that will be required is changing the institution’s name and place in the stylesheet. (fields 264, 502, 040, and 008 will be affected). [4] to formulate this type of request, see bepress’s documentation for more details. “digital commons and oai-pmh: harvesting repository records.” available at http://digitalcommons.bepress.com/reference/80 [5]“cataloging: export or import bibliographic records” available at https://www.oclc.org/content/dam/support/connexion/documentation/client/cataloging/exportimport/exportimportbib.pdf [6] digital commons representative, email communication with author, october 13, 2015. [7] “oregon state university electronic theses dspace (oai-pmh) to marc21xml crosswalk” available at https://ir.library.oregonstate.edu/xmlui/handle/1957/6300 references bepress. (2015). bepress proprietary schema for qualified dublin core. retrieved from http://www.bepress.com/assets/xsd/oai_qualified_dc.xsd bepress. (2015). digital commons and oai-pmh: harvesting repository records, 2-6. retrieved from http://digitalcommons.bepress.com/reference/80 clement, g. p. (2013). american etd dissemination in the age of open access: proquest, noquest, or allowing student choice. college & research libraries news, 74, (11), 562-566. retrieved from http://crln.acrl.org/content/74/11/562.short deng, s., matveyeva, s. & wang, t.m. (october 2008). customized mapping and metadata transfer from dspace/soar to oclc and voyager. paper presented at the ex-libris southcentral users group (elsug) meeting, wichita, ks. retrieved from http://soar.wichita.edu/handle/10057/1573 deng, s., & reese, t. (2009). customized mapping and metadata transfer from dspace to oclc to improve etd workflow. new library world, 110 (5/6), 255. doi: 10.1108/03074800910954271 dspace. (2015). default dublin core metadata registry (dc). retrieved from https://wiki.duraspace.org/display/dsdoc4x/metadata+and+bitstream+format+registries mcmillan, g., halbert, m. & stark, s. (july 2013). comprehensive study of national etd practices. paper presented at the annual meeting for the united states electronic thesis and dissertation association, claremont, california. retrieved from https://conferences.tdl.org/usetda/index.php/usetda/usetda2013/paper/view/666/318 oclc connexion client guides. (2014). cataloging: export or import bibliographic records. retrieved from https://www.oclc.org/content/dam/support/connexion/documentation/client/cataloging/exportimport/exportimportbib.pdf orcid. (2015). retrieved from https://orcid.org/about/what-is-orcid park, j.r., & brenza, a. (2015). evaluation of semi-automatic metadata generation tools: a survey of the current state of the art. information technology and libraries, 34, (3), 24. doi: 10.6017/ital.v34i3.5889 park, j.r. & tosaka, t. (2010). metadata creation practices in digital repositories and collections: schemata, selection criteria, and interoperability. information technology and libraries 29, (3), 108 and 114. doi: http://dx.doi.org/10.6017/ital.v29i3.3136 reese, t. (2009). automated metadata harvesting: low-barrier marc record generation from oai-pmh repository stores using marcedit. library resources and technical services, 53, (2), 129. doi: http://dx.doi.org/10.5860/lrts.53n2.121 robertson, w. c. (april 2011). repository metadata: challenges of interoperability. paper presented at the alcts institutional repository webinar series. retrieved from http://ir.uiowa.edu/lib_pubs/76/ tamu libraries. (may 2015). orcid integration at texas a & m. retrieved from http://guides.library.tamu.edu/content.php?pid=553864&sid=4564757 about the author marielle veve has been metadata librarian at the university of north florida since 2013. previously she worked as cataloger & metadata librarian at the university of tennessee from 2006 to 2013 and cataloger for latin american materials at tulane university from 2003 to 2006. appendix a digital commons crosswalk for etd metadata pdf available a b c d e f g description of desired fields for etd “wish list” marc fields (follow rda) digital commons submission template fields digital commons oai-pmh display (in bepress proprietary schema for qualified dc) customizations to xslt (bepress’ schema + rda) final edits using marcedit quality control in oclc author of etd authorized form 100 naco name & birth date (add in separate fields) <dc:creator> map to the 2nd creator element 100 will be mapped to dc:creator[2] and an additional constant will be added at the end of this field $e author. if $q shows, remove comma before $q. if there is a title, such as jr. or ii, add $c in front press “f11” to link the 100 field to the lcnaf record title 245 $a $b title  <dc:title> statement of responsibility 245 $c *will not be added at this point *will not show in oai mapping *will not be added at this point can be added at this point using automated task 245$c pub. place 264 $a *will not be added at this point *will not show in oai mapping will be added at this point. 264$a will be mapped to [jacksonville, florida]: publisher 264 $b *will not be added at this point *will not show in oai mapping will be added at this point. 264$b will be mapped to [unf digital commons], pub. date 264 $c year of publication (already there) <dc:date.created> instructions will be given to select last 4 digits (year) physical description 300/336/337/338 *will not be added at this point *will not show in oai mapping will be added at this point using constants: 300$a 1 online resource 336$atext$btxt$2rdacontent 337$aunmediated$bn$2rdamedia 338$a volume $bnc$2rdacarrier degree 502 degree name  <dc:thesis.degree.name> an additional constant will be added at the end of this field thesis ( )–university of north florida, yyyy. substitute the word ‘yyyy’ for actual digits. will change field 502 and 008. season pub (spring, summer or fall) 500 season of publication (needs to be added if not there)  <dc:date.issued> abstract 520 abstract <dc:description.abstract> check the content of all the abstract passed. for symbols and special characters not recognized by oclc, change using a character from oclc table. rights 540 *will not be added at this point *will not show in oai mapping a constant field will be added at this point $a all rights…. and mapped to 540 lcsh (controlled subjects) 650 controlled terms (need to be added)  will show in one element  <dc:subject.lcsh> generate a separate 650 field for each lcsh. author provided subjects (uncontrolled) 653 keywords  will show in separate elements  <dc:subject> remove local purpose keywords that you don’t wish transferred to final record thesis advisor(s) in inverted order 700 for each advisor: lcnaf or aacr2 form of (1st, 2nd, 3rd) advisor name (need to be added)  <dc:contributor.advisor> an additional constant will be added at the end of this field $e thesis advisor. yes. check diacritics, if any, passed univ & dept. who granted degree 710 $a $b naco controlled corp body (need to be added)  <dc:thesis.degree.grantor> an additional constant will be added at the end of this field $e degree granting institution. press “f11” to link the 710 field to the lcnaf url 856 *will not be added at this point  <dc:identifier> an additional constant will be added at the end of this field $z connect to this object online. verify url works. fixed fields ldr, 008,006,007,040 *will not be added at this point *will not show in oai mapping will be added at this point using constants: 008 006 007 040$afnp$beng$erda$cfnp substitute the word ‘yyyy’ for actual digits. will change field 502 and 008. appendix b appendix b is available as an xslt stylesheet. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – making user rights clear: adding e-resource license information in library systems mission editorial committee process and structure code4lib issue 29, 2015-07-15 making user rights clear: adding e-resource license information in library systems libraries sign a wide variety of licensing agreements that specify terms of both access and use of a publisher’s electronic collections. adding easily accessible licensing information to collections helps ensure that library users comply with these agreements. this article will describe the addition of licensing permissions to resource displays using mondo [1] by queen’s university and scholars portal (a service of the ontario council of university libraries) [2] . we will give a brief introduction to mondo and explain how we improved mondo to add the license permissions to different library systems. the systems we used are an ils (voyager), an openurl link resolver (360 link), and a discovery system (summon). however, libraries can use mondo to add the license permissions to other library systems which allow user configurations. by jenny jing, qinqin lin, ahmedullah sharifi and mark swartz introduction changes in canadian copyright law and in the way that universities approach copyright compliance, combined with the growing complexity of working with digital materials, have made copyright a priority for canadian higher education institutions. one factor that universities worldwide have to consider when establishing copyright compliance initiatives is the vast array of database licenses that libraries sign in order to access electronic resources (e-resources). these licenses specify terms that users must follow in order to legally use these materials. for example, some licenses may allow users to include materials on learning management system sites. others may allow libraries to loan e-resources to other libraries. this information is frequently only included in licensing agreements that are not accessible to users, or on website “terms of use” that are often ignored. possible outcomes of not complying with these agreements include loss of access to databases and legal action. as a result of the inaccessible nature of these licenses, and the serious consequences of having users ignore these terms, it is essential to make this licensing information accessible to all users in a format that is understandable and clear. this is why many institutions have created databases that make license permissions available to all. when i started working as the information systems librarian at queen’s university library in 2013, my first project was to migrate the library’s openurl link resolver from sfx to 360 link. a major task in this project involved linking the license permissions from the mondo database to the e-resources in the user interface of 360 link. we finished the project in two months and here is a sample record of an e-resource in 360 link and its license page. (figure 1). figure 1. the user interface of 360 link and license page the license links: before and after the migration queen’s university library uses mondo, an open source application used to manage the copyright and license information for e-resources. our mondo is hosted by scholars portal, a service provider of ontario council of university libraries (ocul). (figure 2). because the license pages for our e-resources are created in mondo, our task is to add the urls of those license links to our library systems, for example, 360 link. figure 2. the relationship between mondo and queen’s e-resource knowledgebase in sfx, this process is accomplished by inserting the license links into the “general note” field for an e-resource. because of the migration away from sfx, we needed to build a new model for adding license links to our e-resources in 360 link and other library systems. michael vandenburg, my supervisor, suggested using 360 link reset [3] and summon_click [4] to add the links to the user interface of these two systems. after discussing the priorities with my supervisor, my first step was to familiarize myself with mondo, the open source software we use to manage the license permissions for queen’s, as described in the next section. mondo license grinder mondo is an open source application developed by the university of british columbia library in 2009 (https://code.google.com/p/mondo-license-grinder/ ). it displays license permissions for online resources. (figure 3). a sample site is available at: http://queens.scholarsportal.info/licenses/ figure 3. customized mondo by scholars portal – a to z list and a license page the key functions and main workflow of mondo first, authorized library staff must create license records for each type of e-resources and parameters are set based on the copyright agreement between the library and the database provider. second, library staff copy the url of the license page of an e-resource in the mondo system and paste it in a target record in the link resolver system. for example, in 360 link the link can be added in the “public all titles note” field for the database “abi/inform global” (figure 4). then the “license terms of use” link for that e-resource will display in 360 link. figure 4. create license page in mondo and paste the url in a record in 360 link issues to overcome first, there were about 2,000 e-resources in our serials solutions’ knowledgebase and we needed to finish the migration from sfx to 360 link in several weeks. because 360 link is a hosted service provided by serials solutions, we couldn’t batch load the license links for those 2,000 records into the database tables. during the summer, there were no student or library staff who could help us enter the data manually. in addition, we needed to add the links to the license information to other library systems, such as voyager and summon. adding them in different library systems by myself would be very time-consuming. in order to finish this task by our deadline, we needed to find a new model for providing this service. second, because the urls of those license links are not in a predictable pattern (sample records in table 1), we could not automatically generate the license links and add them in the target library systems. table 1. the e-resource names in mondo and the license urls of the e-resources database names in mondo license link urls abi inform global http://queens.scholarsportal.info/licenses/abi_inform acm digital library http://queens.scholarsportal.info/licenses/acm_digital_library 19th century uk periodicals online http://queens.scholarsportal.info/licenses/seventeenth_eighteenth_century_burney finally, because we wanted to add the license links to our discovery system (summon) and there is no field that we could use to display that piece of data in summon, we needed to work out a solution which could display the license links in different library systems using a predictable pattern rather than manually entering the links. the solution: enhancements to mondo after reviewing all of the aspects of this project, we identified the following key issues/requirements to improve the workflow: we need to use a predictable, predefined pattern for the license urls created in mondo (for example, a base url, plus the database name in a standard format), so that we can generate the license links with the same pattern in the target library system’s user interface. we need to have access to the configuration files of the target library system’s user interface (for example, we can insert javascript codes into the summon search result page). we need to be able to read the e-resource names from the target library system’s user interface in order to generate the url (for example, in figure 1, the e-resource name is “abi/inform global”). we need to create a “general license” page for the e-resources which do not have a license page in our mondo database. based on the requirements we defined above, we made the following enhancements to mondo: first, we added a new function in mondo to automatically generate the urls for the license link pages in a predefined pattern “(rawurldecode($db_name))”. this allows us to generate the same pattern of the urls in different target library systems’ interfaces, so that we do not need to enter the license link urls manually in mondo. add a javascript file called “licenselink.php” in the mondo/admin folder to generate the license page url licenselink.php: //in index.php, get database name from url parameter $db_name = $_get['name']; //get license $license= $db->get_license(rawurldecode($db_name)); //or redirect to an information page if($license== null){ $license= "generic_license"; } //redirect to license view ('location: '.base_url.'index.php?license='.$license); modify the “add/edit a license” page in mondo second, in order to eliminate the need to manually enter the urls of the license link in mondo and save time, we modified mondo/admin/index.php and update.php to automatically generate the urls and save them in mondo. (figure 5). this also prevents typos resulting in bad links. (for example, the url for the license page of “bibliographie der deutschen sprach”, might be spelled wrong if the person who enters the url of the license page does not understand the language). the detailed changes we made in mondo are listed below: in mondo-license/admin/index.php, we added the following code: $vendor=select('vendor',$vendor,$db); //mondo code $consortium=select('consortium',$consortium,$db); // mondo code $databases=select('databases', $databases, $db); //we added this line … <form id="add-license" action="update.php" enctype="multipart/form-data" method="post"><!—this line is mondo code --> <!-we add --> <fieldset><label class="desc" for="database">databases</label> <span style="float: left; width: 100%;"> $databases <label class="fullwidth sub" for="sel_databases">databases</label> <input name="import" type="file" /> </span> </fieldset> figure 5. enter the database name and url of the license page will be generated automatically in admin/update.php, we added the following code to get uploaded/input databases: // "database name", which corresponds to this license and was uploaded by users, are saved in here //open the file in read mode. notice the 'import' is the name of input in admin/licenselink.php. $fp=fopen($_files['import']['tmp_name'],'r'); //the upload file needs to be csv format. //read in line while ( ($line = fgetcsv($fp)) !== false) { if($line[0] <> 'id'){ //skip title line //insert to 'license_db' table $db->insert_data_databases($line); } } two tables were added in the mondo database: databases: contains all databases.(columns could be db_id, db_name etc.). license_db: license – databases map (id, db_id, license_id). in db.inc.php, we added a function called insert_data_databases($line): public function insert_data_databases($line){ //assume in csv, comma ',' is the delimiter $field = explode(",", $line); $sql="insert into `db_id`, `db_name` set `db_id`=:db_id, `db_name`=:db_name"; $stmt=$this->preparedstatement('dblicref', $sql); //to simplify the code, we don't add error check here. $stmt->execute(array('db_id'=>$field[0], 'db_name'=>$field[1])); } add the license links to different library systems after changing mondo functions and automatically generating the license links in a predefined pattern, we were ready to add the license links to different target systems. the examples in this article will be using 360 link, voyager opac, and summon. openurl link resolver: 360 link there are three parts in the 360 link 1.0 search result page: citation display, content provider links, and custom links. in the content provider links section, under “resource” column, all the links to the providers’ sites (providers can be databases, publishers or journals) are listed. the licenses and copyrights are assigned based on the agreement our library has with the providers, so we need to assign our “license terms of use” links to the provider information. (figure 6). figure 6. sample license links in 360 link 1.0 user interface first, my supervisor suggested that we test 360 link reset [3] as the starting point. after reading the documentation and consulting with matthew reidsma, the author of the program, we learned that we could modify the 360 link 1.0’s user interface by inserting javascript in 360 link 1.0 admin > advanced options > page head > head html. second, we defined the pattern for the urls of the license links to:“http://baseurl+e-resourcename”, in order for us to add those urls to 360 link user interface. the baseurl is the http address our mondo service provider gave us. for example: “https://queens.scholarsportal.info/licenses/license/?name=”. the “e-resourcename” is the name of the e-resource. we used a javascript file called “360_get_names.js” to add the urls of the license links to 360 link user interface and it follows a two-step process outlined here: find the e-resource names in the 360 link page (the e-resource names are given as the elements of a css class called “ss_databasehyperlink”). add the license links to the e-resource using the same pattern we defined in mondo. (http://baseurl+e-resourcename). the javascript is listed at the following url: https://gist.github.com/happyrainb/81541e23049e30b281d2 360_get_names.js: $(document).ready(function() { $(".ss_databasehyperlink").each ( function() { $('<a target="_blank" href="https://baseurl/licenselink.php?name='+(urlencode($(this).text()))+'">[license of use]</a>').insertafter($(this)); $('<span>&nbsp;</span>').insertafter($(this)); } ); function urlencode(str) { / //discuss at: http://phpjs.org/functions/urlencode/ (function urlencode: this function was copied from phpjs.org and the last line of this function needs to be changed to “.replace(/\+/g, ‘%20’);”. this function resolves the linguistic difference between php and javascript’s url encoding functions. javascript uses the “%20” encoding to represent a spaces, while php uses “+.” the difference in urls would lead users to the general message page instead of the specific database license. replacing the “+” with “%20” while generating the url results in consistent urls.) the code above simply goes through each link class (“.ss_databasehyperlink”) and appends a license link to it with the url pattern (http://baseurl+e-resourcename) as we defined in mondo. the name parameter is also read from the same link. the new version of 360 link in the spring of 2015, a new version of the 360 link interface was released. comparing with 360 link 1.0, the new 360 link makes it easier to reference an external javascript file, and offers more customization options with configurable custom links. (according to eddie neuwirth, the product manager lead – discovery services at proquest, the new 360 link interface is loosely based on some of the design concepts from matthew reidsma’s 360 link reset but available out-of-the-box). in order to test the license links, we made a sandbox on our server and added the links to the 360 link (figure 7) by making a new version of the get-license.js, which we called get-license-2.js. figure 7. sample license links in 360 link new version user interface in the 360 link new version user interface, the e-resources all have the same css class name: ‘.resource-name’. we just need to get each of them and generate the license link to the e-resource using the baseurl and the pattern we defined in mondo. we list the code for the get-license-2.js below: get-license-2.js: jquery(document).ready(function() { // get all the elements that have a class name '.resource-name' var links = jquery(".resource-name"); // go through the elements and add a link after each one based on the text each element contains for(var i =0; i<links.length; i++){ // turn the current item into a jquery object var el = jquery(links[i]); // create a new jquery object which is the new link var new_link = jquery('<a target="_blank" href="https://baseurl/licenselink.php?name='+(urlencode($(this).text()))+'">[license of use]</a>'); new_link.insertafter(el); var empty_span = jquery('<span>&nbsp;</span>'); empty_span.insertafter(el); } }); voyager 8 and 9 after we added the license permissions to 360 link in the summer of 2013, we discussed this process with our partners at western university. we learned that they were able to add the license links to the opac of their ils and discovery system (taufique, 2015), then we tried to add the license links of the e-resource to our voyager opac. having read the voyager manual and consulted voyager-l, we searched the web and found a presentation about editing the voyager opac (guy, 2010) and a website with documents for customizing the voyager skin [5]. they helped us identify the file we needed to modify in voyager in order to add links to the skin (the display.xsl file in the contentlayout folder in webvoyage). after testing the changes in our voyager sandbox, we managed to add the license links to voyager opac using the same pattern that scholars portal defined in mondo. (figure 8) figure 8. sample license links in voyager opac – record page after we discussed this issue with our cataloger, we learned that we can insert the url of the license links to holdings record tag 856 subfield z. because we didn’t have library staff or student to help us add the license links manually in voyager catalogue module, we tried to add the license links in voyager in the opac configuration files. after testing in the webvoyage sandbox, we found the block which is responsible for the holdings 856 subfield z is in the template ”bmd3000?. the display.xsl file is listed in the following url: https://gist.github.com/happyrainb/71da337b043d02757691 display.xsl: <xsl:variable name="providerdata"> <xsl:value-of select="slim:subfield[@code='z']" /> </xsl:variable> <xsl:variable name="provider"> <xsl:choose> <xsl:when test="contains($providerdata, '')"> <xsl:value-of select="substring-after($providerdata, '')" /> </xsl:when> <xsl:when test="contains($providerdata, ',')"> <xsl:value-of select="substring-before($providerdata, ',')" /> </xsl:when> <xsl:otherwise> <xsl:value-of select="$providerdata" /> </xsl:otherwise> </xsl:choose> </xsl:variable> <xsl:variable name="baseurl">https://baseurl/licenselink.php?name= </xsl:variable> <a href="{$baseurl}{fn:encode-for-uri($provider)}">[license terms of use] </a> <br/> the xml above simply defines a baseurl variable and uses it in the newly added xml for the links. the provider name is extracted from the data in the voyager opac interface and the license links are inserted in the xml file under subfield “z”. summon 1.0: server-side data endpoint api after we discussed this process with our partners at the university of toronto, we learned that their team added the license information in summon 1.0 using summon api (taufique, 2015). after testing in december 2013, it seems that the only way we can add the license links to summon 1.0 is to use 360 link api, because we don’t subscribe to the summon api service. the license data that we want to show the user is generated by a php script which we refer to as the server-side data endpoint api. this is a server-side script that generates all the data we want to show the user when they “hover” over the license link for a specific search result. we can see an example of the popup in our testing site (figure 9): figure 9. sample license links in summon 1.0 – record page the data in the popup is loaded via ajax. the ajax call for each popup is made when the page loads all the search results. the ajax call makes a request to the server-side data endpoint api, using an http request, which then returns a response with the entire popup html populated with the relevant data. therefore, the interface must simply show that html to the user once the ajax call has completed successfully. add the popup to the summon interface the search result list from summon 1.0 needs to show the license information for each individual entry. because the library decided to show the license links only for articles, we need to find the journal article entries in the summon search result page first. the most easily deployable solution was to use client side functionality on the browser to dynamically add the links and respective popups for each search result. since the search results are generated in a pattern it is easy to find that pattern and use it to make the modifications we need. the following figure shows the html structure for the entries area we are interested in. (figure 10) figure 10. the html structure of summon 1.0 user interface – record display we can see that each search result is wrapped in a “div” element with the class “.document”. inside each of these “div” elements we have the necessary information to make our ajax request and obtain the relevant popup data to show the user when prompted. given the structure and patterns we have found, we could now write code to enable the computer to generate the links to help the user quickly access license information for any viable search result journal. as suggested by my supervisor, we modified the summon_clicks.js. [4] the complete javascript code (summon1-use360api.js) is listed at the following github url: https://gist.github.com/happyrainb/946e708a60373305375a for the purposes of this section, the important part of this script starts at line 94 of the file from the github link above. first, we must iterate over each search result (‘.document’) class element. for each search result we can read the metadata in order to make our ajax calls to our server-side endpoint api, explained in the previous section. essentially, there are two tasks performed for each search result which are listed below in order: extract issn and doi from the body after we read the 360 link api documents and two articles (durno, 2012 and talsky, 2008) , we decided to use the doi and issn as parameters to supply the ajax call and make a request to the server-side data endpoint api. it is important for us to get the issn and doi because we need to give these parameters to the server-side endpoint api in order to receive the correct information for an article, such as the e-resource names, the dates available, etc. the code block below is responsible for this:summon1-use360api.js: sections = $(this).find('div.previewdocumentcontent').find(".section"); var issn = ''; var doi = ''; $(sections).each ( function() { if ($(this).find('.title').text() == 'issn') { issn = $(this).contents().filter(function() { return this.nodetype == 3; }).text(); } if ($(this).find('.title').text() == 'doi') { doi = $(this).find('.title').next().text(); } } ); after we get the issn and doi data from the above code, we can send them to the php script to make the popup and display the article link, database link, holdings data, etc., in the mini-window shown in figure 9. make ajax call to server-side endpoint api with the above parameters and add resulting popup code to the html dom. summon1-use360api.js provider=''; $.post('summon1-use360api.php', { 'issn': issn, 'doi' : doi}, function(data) { $(publicationlocation).find(".summary").append(' <div style="display: none;">'+data+'</div> '); $(publicationlocation).find(".summary").children('div').first().mouseover(function(){$(this).stop(true,true).show();}).mouseout(function(){$(this).stop(true,true).fadeout();}); $(publicationlocation).find(".summary").children('div').first().mouseover(function(){$(this).stop(true,true).show();}).mouseout(function(){$(this).stop(true,true).fadeout();}); $(publicationlocation).find(".summary").children('div').first().find('.theme-close').click(function(){$(this).parents('div.summary').children('div').first().hide();}); }); $(publicationlocation).find(".summary").children('a').first().mouseover(function(){$(this).next('div').stop(true,true).show();}).mouseout(function(){$(this).next('div').stop(true,true).fadeout();}); when this code successfully executes, the user will be able to hover the license link and view the the license information for each of the e-resources within the popup. use 360 link api to get data and display them in the popup in the above javascript file we get the issn and doi of an article from summon user interface and provide them to a php script (summon1-use360api.php) to make the elements in the popup. the php script is listed at the following github url: https://gist.github.com/happyrainb/2608d28cfdcfdc54350a summon1-use360api.php: … $queensurl= xxx; //you can find this in your 360 link document provided by proquest $baseurl = yyy; //you can find this in your 360 link document provided by proquest $key=zzz; //you can find this in your 360 link document provided by proquest $mandatoryfields = array('version' => '1.0'); //this is required in 360 link api document $optionalfields = array('issn' => $issn, 'volumn' => $volumn,'doi' =>$doi); //make the query string $fields = array_merge($mandatoryfields, $optionalfields); $querystring = http_build_query($fields); $url = $baseurl.'?'.$querystring; //make the url of the query string combining the api key, baseurl, etc. $xml = get_xml_result($url); //the data is displayed in xml format $xmldomdoc = new domdocument(); $xmldomdoc->loadxml($xml); …. …. foreach ($xmldomdoc->getelementsbytagname('linkgroup') as $datagroup) //assign the value to each { //elements which will be displayed in the popup window $url_elements = $datagroup->getelementsbytagname('url'); $data=$datagroup->getelementsbytagname('holdingdata')->item(0); $urldata=$datagroup->getelementsbytagname('url'); $results[$counter]['databasename'] = $data->getelementsbytagname('databasename')->item(0)->nodevalue; $startdate = $data->getelementsbytagname('startdate')->item(0)->nodevalue; $enddate = ($data->getelementsbytagname('enddate')->item(0))?($data->getelementsbytagname('enddate')->item(0)->nodevalue):''; $results[$counter]['dateavailable'] = ($startdate. '-' .$enddate); $results[$counter]['databaseid'] = $data->getelementsbytagname('databaseid')->item(0)->nodevalue; we supply the doi and issn parameters to the ajax call which returns an xml result from which we can extract information about the article. we use fields such as “dateavailable”, “databasename”, and so on. it is important to note that we use the “databasename” field as the key to retrieve relevant license information for a given article; this key will be used in the next step. for the purposes of this section, the important part of this script starts at line 212 of the file from the github link above. summon1-use360api.php: … <tr> <td valign="middle" align="center" class="c1"> <div class="cl" id="articlecl"> <a target="_blank" href="http://<?php echo $key;?>.scholar.serialssolutions.com/log?l=<?php echo $key;?>&d=<?php echo $result['databaseid'];?>&p=link&u=<?php echo $result['article_url']; ?>"><img src="http://library.queensu.ca/images/xxx/article-sm.png"></a> </div> </td> <td valign="middle" align="center" class="resultsrow"> <div class="cl" id="journalcl"> <a target="_blank" href="http://<?php echo $key;?>.scholar.serialssolutions.com/log?l=<?php echo $key;?>&d=<?php echo $result['databaseid'];?>&p=link&u=<?php echo $result['journal_url'];?>"><img src="http://library.queensu.ca/images/xxxx/journal-sm.png"></a> </div> </td> <td valign="middle" class="c1" class="resultsrow"> <span><?php echo $result['dateavailable'];?></span> </td> <td valign="middle" class="resultsrow"> <div class="cl" id="databasecl"> <span> <a href="http://<?php echo $key;?>.scholar.serialssolutions.com/log?l=<?php echo $key;?>&d=<?php echo $result['databaseid'];?>&p=link<?php echo $result['source'];?>"><?php echo $result['databasename'];?></a> </div> </td> <td valign="middle" class="resultsrow"> <span><?php echo $result['license information'];?></span> <span><a href="https://baseurl/licenselink.php?name=<?php echo rawurlencode($result['databasename']);?>">license information</a></span> </td> </tr> the above code is responsible for displaying the elements in a table in the popup window. we make a mini-version of the 360 link search result page using 360 link api and assign the license links to each of the e-resources. this way we provide the complete information for permitted use of the e-resource and it’s easy for the users to understand because it’s similar to the 360 link user interface. summon 2.0: available methods to modify summon 2.0 modifying summon 2.0 to achieve similar functionality for license links was easier because we found a script called “dnl_summon_custom_step3.js” [6] from the articles “hacking summon 2.0 the elegant way” in the code4lib journal (bailey, 2014) and “exposing library services with angularjs” (voß, 2014). all of them helped us customize the summon 2.0 user interface. although all the techniques described in the code4lib article “hacking summon 2.0 the elegant way” are good and have their own merits, we decided to use “technique 2: direct dom manipulation” because it was closely related to the same methodology as the techniques we used to modify summon 1.0. the complete codes are listed at the following github url: https://gist.github.com/happyrainb/e0f546390436a1c26ac2 for the purposes of this section, the important part of this script starts at line 217 of the file from the github file listed above. add license links to the summon 2.0 user interface our approach to modify summon 2.0 was to identify the html structure that was used to store the relevant source name data and then to create links that directly send the user to the license information. compared to summon 1.0, summon 2.0 was easy because it uses angularjs, a standard mvc client-side framework that is documented and can easily be understood by a programmer. we decided to inject our code into a “watchcollection” observer, which is a facility provided by angularjs to detect changes made to a collection of data such as our search results, because we wanted to generate the links every time the list of search results or “feeds” were changed. the structure of the search results seemed more complex, but after hacking away on the javascript console, we were able to find the correct jquery selectors to get the elements that we need from the main item which can be seen below (figure 11): figure 11. main html element pattern which represents each search result the dynamically-generated links all have the same endpoint: a simple url pattern, shown in the pattern below: http://baseurl/licenselink.php?name={{source name}} url pattern: {{baseurl}} + {{source name}} so for example if we wanted to get information regarding source: “acm digital library” we would enter the following url into the browser: http://baseurl/licenselink.php?name=acm digital library. summon2-license.js: //watch feed change, add license links function watchfeedchange( ) { myscope.$watchcollection('feed', function(){ //delay 1 sec. to wait for dom to actually finish loading settimeout(function (){ ... link generation code ... },1000); }); } //watch feed change() we can see the feed change event handler above; this handler basically waits for search or feed changes and runs our link generation code when such changes are detected. after we learned about the summon 2.0 user interface, we found that the e-resource is called “source” in the detailed record popup on the right side of the screen. the next step for us was to get the data in this field and assign the license links for those e-resources. (figure 12). figure 12. sample license links in summon 2.0 preview server – record page the source names (“[abi/inform global]” and “[academic onefile]”in figure 12) we need are visible in the popup on the right and it appears when we mouse over the record listed in the search result. if we observe the figure above, we can see that the popup has information regarding the first record in the search results, which is where the user has “mouse-over”. dynamic link generation in this application we do not need to embed the links into the html. when the user clicks the “empty” license link, we dynamically build the appropriate url for those links and redirect the user to it when the user clicks any of the generated links. the “source: [license information]” link upon the “watchcollection” event triggering due to the search results changing, the first thing the javascript code (summon2-license.js) does is to add an “empty” link called “source: [license information]” at the bottom of each journal article. this will prompt the user to click if they require license information. the following code performs the task of adding these “empty” links: //for each content type display, do these $(".contenttype.ng-scope span[bo-bind='doc.content_type | translate']").each(function(){ //if it's journal article and hasn't been processed by our custom script if($(this).text().indexof('journal article') != -1 && $(this).not('.processed').size() > 0 ) { //add a class called "processed" so we can keep track, then add a custom link $(this).addclass('processed').parent().append(' source: <a class="mycustomlink" href="#" target="_blank"> [license information]</a>'); } }); first, we find all the <li> items shown in figure 11 by using jquery to select them; then we simply check if the text “journal article” is in the selected html, since we are only handling journal articles. if the item is an article, then we inject the “empty” “source: [license information]” links. extract the source name and define the pattern for the license link url after the user clicks the “source: [license information]” link, the license links of each e-resource for that journal article (for example, “[abi/inform global]” and “[academic onefile]” in figure 12) are displayed. (if the user does not click the “[license information]” link, the two license links won’t be displayed).in order to generate links to the relevant license information we need to extract the source data. an example of this data can be seen in figure 12 on the right side. because angularjs always keeps its data inside the models of the application, we need to access the model data from the javascript object named “myscope.preview.doc” in the summon 2.0 user interface.to extract the source data in the popup, we defined two javascript functions listed below: function getcontainervalue(){ if(myscope.preview.doc.databases.length > 0){ return myscope.preview.doc.databases[0].name; //extract the source name } else { return myscope.preview.doc.publisher; } } function buildurl(source_name){ //define the pattern for the url return "http://baseurl/licenselink.php?provider="+(urlencode(source_name)); } generate the url based on the extracted data and redirect the user to it. the source names are stored in the “myscope.preview.doc.databases” array. if any source names were found, we generate the relevant links for the user to click. if the source was not found for any reason then we redirect the user to a general license page (“http://library.queensu.ca/360 link/360sample.htm“), which informs the user to contact the copyright office for license information of that e-resource. $("a.mycustomlink").click( function(){ if( $(this).parent().children('div.license_popup').length ){ $(this).parent().children('div.license_popup').remove(); } else { if(myscope.preview.doc.databases && myscope.preview.doc.databases.length > 0){ var new_node = ' <div class="license_popup"> '; for (var i = 0; i < myscope.preview.doc.databases.length; i++) { new_node += '<span class="icon sprite-s format-sprite format--generic_16px format-journal_article_16px"></span>'; new_node += '<a class="mycustomlink2" href="' + buildurl(myscope.preview.doc.databases[i].name) + '" target="_blank"> ['+ myscope.preview.doc.databases[i].name +']</a>'; new_node += '<br />'; }; new_node += '</div>'; $(this).parent().append(new_node); } else { window.location.href = "http://library.queensu.ca/360 link/360sample.htm"; } } return false; } ); when the “empty” license link is clicked we check that source data is available by checking the “myscope.preview.doc.databases” array existence and length. if such array is available and has at least one source then we proceed to insert the html code to create the links for the user to click if needed. lessons learned hosted service provided by vendors 360 link is a hosted service provided by serials solutions. we could not load data into 360 link’s database, which means we could only enter data manually from the vendor’s client site. this made it difficult for us to modify the records efficiently. in addition, there is no testing environment in 360 link and summon. in voyager, we have a sandbox folder in which we can test our code, but in 360 link and summon we need to set up those testing environments ourselves. we also need to wait at least 24 hours to see the changes we made in 360 link and summon, which means if we made a mistake in the configuration, the interface will be displayed with this mistake for 24 hours. documentation and community support the biggest challenge for adding the license links in the voyager opac is that we didn’t find much documentation in the voyager manual about customizing the opac skin. through a google search, we found a number of useful presentation and documents made by other voyager users on how to do similar tasks, which helped us to understand and modify the configuration files in voyager. one key reason we were able to finish this project is the community support we got from the users groups. for example, the summon-clicks.js for summon 1.0 and dnl_summon_custom_step3.js for summon 2.0 and the articles from the code4lib journal had already solved several issues we faced. we built our solutions using those scripts and they saved us a lot of time to finish the project on time. consortia and partners because our service provider scholars portal hosted mondo for us, we could not edit the license page urls in the database table, or load all of the urls into mondo. we needed to work closely with scholars portal to modify the data in our mondo. fortunately, scholars portal helped us whenever we needed their support. we also learned a lot from other libraries in the ocul consortia, such as western university and university of toronto. they provided us with solutions for adding the license links in the systems that they use, and they shared their experiences with us. this is another key reason why we could successfully finish this project by the deadline. summary from the summer of 2013 to spring 2014, we worked with scholar portals and added the license links to the 360 link and voyager in production, as well as the summon 1.0 testing site. because summon 2.0 uses angularjs, which was new to us, and we did not have a plan for migrating our discovery system from summon 1.0 to summon 2.0 by the summer of 2015, we waited to add the license link to summon 2.0 after reading articles posted on code4lib and consulting with ahmedullah sharifi, an expert in angularjs. interpreting, managing and providing licence terms to library users is a complex task. integrating the information in the mondo database into tools like summon and 360 link allow users to access this information at their point of need, which helps libraries meet their legal requirements while making it easy for users to find this information. with access to this information, users may even discover that they can use items in ways that they didn’t realize were possible, for example, as a pdf upload on a learning management system course page or in a print course package of readings. throughout this project, we needed to work with different vendors, user communities and our partner scholars portal to find solutions for our needs. this is not a project accomplished by our group, but a project supported by the library community. we hope this article can help other libraries provide user right/license information to their e-resources and avoid the issues/problems we had during our implementation. all of the scripts/codes described in this article are avaliable at the following url: https://gist.github.com/happyrainb. libraries have started using more vendor hosted services than before. these vendor-hosted services make customizing library systems more difficult, especially when the vendors release new versions of their systems and change the user interface, system configuration, etc. we have to deal with different types of issues and be creative in solving them. in addition, we need to work with peers in consortial universities, members in the same user group and vendor support teams to find the solutions to our unique needs. what’s next? making license information accessible at points of need for users drives future integration possibilities for mondo. for example, information from mondo can be included in proprietary databases or in university libraries’ digital asset management tool or institutional repositories. by including the copyrights/access rights information like this in a digital asset management system (in the digital preservation process [7]) or in an institutional repository (in the submission process), libraries can showcase the flexibility of these resources for users as most of this material is open access and can be used for all of the purposes generally included in mondo. this can help encourage the open access movement, while also increasing the use of library provided tools. acknowledgements the following people helped us with this project. without their support we wouldn’t have been able to add the license links to our systems. we gratefully acknowledge their assistance. 360 link/summon: matthew reidsma, web services librarian, grand valley state university; wittawat meesangnil, systems librarian, dimenna-nyselius library, fairfield university mondo: teresa lee, e-resource & access librarian, university of british columbia our partners: amaz taufique, assistant director of systems and technical operations; marc lalonde, web coordinator, librarian, university of toronto; christina zoricic, acting head, metadata access, western university. our project team: michael vandenburg, associate university librarian; mark swartz, copyright specialist; anne brule & ellen symons, e-resource librarian; katie legere, web coordinator; andrew dacosta, web development technician; alex fletcher, application support technician. notes [1] 1. https://code.google.com/p/mondo-license-grinder/ [2] scholars portal is a service provider of ontario council of university libraries (ocul). it hosts the e-resource license database using mondo for queen’s university library, as well as the other ocul members’ license databases. [3] https://github.com/gvsulib/360 link-reset [4] https://github.com/gvsulib/summon-stats/blob/master/summon_clicks.js [5] http://systemsmgr.wikispaces.com/voyager_customizations [6] http://mlib.fairfield.edu/summonjs/dnl_summon_custom_step3.js [7] http://public.ccsds.org/publications/archive/650x0m2.pdf reference: bailey a and back g. 2014. hacking summon 2.0 the elegant way. the code4lib journal 26. [cited 2015 april 16]. available from: http://journal.code4lib.org/articles/10018 durno j. 2012. hacking 360 link: a hybrid approach. the code4lib journal 18. [cited 2015 april 16]. available from: http://journal.code4lib.org/articles/7308 guy l. 2010. voyager 7 tomcat webvoyage interface for dummies, or: what will it take to customize our new user interface and remain sane? eluna 2010, 11-13 may 2010, fort worth, texas. [cited 2015 april 16]. available from: http://documents.el-una.org/456/ voß j and horn m. 2014. exposing library services with angularjs. the code4lib journal 26. [cited 2015 april 16]. available from: http://journal.code4lib.org/articles/10023 talsky d. 2008. auto-populating an ill form with the serial solutions link resolver api. the code4lib journal 4. [cited 2015 april 16]. available from: http://journal.code4lib.org/articles/108 taufique a, zoricic c, jing j and lalonde m. 2015. transparent licenses: making user rights clear. ola super conference 2015, toronto, on. [cited 2015 april 16]. available from: http://www.slideshare.net/happyrainb/ola-2015-ourpresentation about the authors jenny jing has more than ten years of experience working as the systems librarian in different organizations including: queen’s university, harvard business school and memorial sloan kettering cancer center. her experience includes: migrating ils systems, building library ir and digital repositories. if you have questions/comments about 360 link, summon 1.0 and voyager solutions described in this article, please contact jenny at: jingjenny@hotmail.com qinqin lin is a graduate of the university of toronto’s computer science program and has worked for university of toronto since 2009 as a programmer. she works on a variety of scholars portal projects including enhancing ontario council of university libraries (ocul)’s new usage rights database, our and the scholars portal ebooks platform. if you have questions/comments about mondo functions described in this article, please contact qinqin at: qinqin.lin@utoronto.ca ahmedullah sharifi is currently the senior software engineer at vegrif group inc. in toronto, ontario. sharifi focuses on developing software technology to fully utilize computers and their endless potential in many areas of application. he has been published in the icme 2006 for his work with fluid interfaces at the university of toronto. if you have questions or comments about angularjs and summon 2.0 functions described in this article, please contact ahmedullah at: ahmed@vegrif.com mark swartz is the copyright specialist at queen’s university. in this position, he works with librarians, staff, faculty and instructors across all faculties and schools to develop web-based information and educational programs on copyright. mark has held two other positions at queen’s – as an education librarian and as the online course developer for the continuing teacher education department. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – creating a seamless cross-platform online experience for mobile users mission editorial committee process and structure code4lib issue 16, 2012-02-03 creating a seamless cross-platform online experience for mobile users in creating a mobile-optimized website for drexel university libraries, we have strived to preserve the seamless transition between platforms that our desktop users experience. we employ separate technology and coding solutions to make drupal, wordpress, and html sections mobile optimized, while continuously improving the mobile user experience in terms of design, usability, and site performance. this paper details how, through extensive research, design, and development, we found the best solution for creating a steady mobile experience for our users. by katherine lynch introduction according to google analytics, in september 2009, the drexel university libraries website (http://www.library.drexel.edu) received over 141,500 unique visitors. of those visitors, 23 were on mobile devices. in september 2011, nearly 800 unique mobile visitors used the drexel university libraries website on handheld and tablet devices, and that number continues to climb steadily each month. as mobile usage figures grow dramatically for many libraries, having a mobile-optimized web presence becomes nearly as important for reaching patrons and students as having a web presence at all. a mobile application or mobile-optimized version of a library’s website is beneficial to the institution in many ways, from reaching more users to creating a clean, semantic version of the site. the drexel university libraries website has benefitted in all of these ways with the development of a smooth, seamless user experience for mobile patrons across our multiple content management systems. in this paper, i will describe the process of design and development for the first mobile-optimized web presence for drexel university libraries. i will detail the challenges posed by working with the existing architecture of a website running on multiple content management systems, and the solutions we employed. i will also address ongoing improvements we made to our mobile presence, as mobile-optimization tools and techniques have emerged and matured. it is important to note that, when we decided to create a mobile version of the libraries website, we did so with the knowledge that this would require the development of more than one mobile optimization solution. the drexel university libraries website, as it is referred to in this paper, is comprised of the following: a website containing primarily resources, services, and location information, which includes personalized, dynamic, and interactive elements. this section is powered by the open-source content management system, drupal 6. blogs maintained by staff and liaison librarians for research and information, powered by wordpress. research guides maintained by liaison librarians, powered by wordpress. several small web-based projects and applications too small to require a separate content management system, written in html and css; for example, the drexel university libraries annual report web version, an online version of a widely-circulated print document roughly 20 pages long motivations in the summer of 2009, a few months before those first 23 mobile users graced our website with their presence, drexel university libraries undertook a usability study of the current website with in-depth user interviews of several drexel students from diverse groups. participating students were asked to give some details about their daily web use habits, including data on how often they went online each day, the websites they used the most, and the ways in which they used the drexel university libraries website. the sampling of students included undergraduates and graduates, with majors ranging from engineering to art history. two recurring themes emerged from these interviews. first, our students were beginning to need constant access to the internet for their studies, which would result in increased acquisition and usage of mobile devices by the student body. second, if we wanted to continue to offer all of the information and services available on the drexel university libraries website to our students as often as possible, it was no longer good enough for our full browser website to look passable on a mobile device. it was time to create a user experience specifically for mobile devices. our goal in taking this project on was not to create a stripped-down, application-like mobile version of the website, but rather to make all information and resources on the drexel university libraries website available, accessible, and readable on a wide range of mobile devices. with this in mind, the design process began. interface design the design of the new mobile presence stripped out all elements located in the peripheral areas of the sidebars of the full browser version of the website. sectional navigation deemed absolutely necessary was planned for inclusion at the base of each page. the horizontal navigation across the top of the full browser version was preserved in this design, as these large buttons were easy to use on touch screens as well as desktops, well recognized from the full browser design, and followed the lead of other successful button-centric mobile designs. figure 1: finalized design mockups for mobile-optimized website pages apart from stripping out most sidebar content, the information presented on pages with this design was to remain the same as the content presented in the full browser version. for the mobile site’s homepage, the elimination of content was a bit more radical. all content was stripped from the homepage, save the omnipresent, horizontal navigation, the resource and catalog search interface, three popular links, and the chat widget. in this way, we created a design that followed the stripped-down nature of mobile-only sites, but still allowed users, upon entering, to access any information available on the full browser version of the site. the mobile-optimized designs for the wordpress-powered pages were very much the same. the full browser version of the drexel university libraries website preserves the same aesthetic look and feel across drupaland wordpress-powered pages, and something similar was desired for the mobile-optimized version. figure 2: finalized design mockups for mobile-optimized blogs and research guides mobile optimization with drupal 6, wordpress, and media queries once the look and feel of the mobile site had been decided, it was obvious that one mobile solution would not extend far enough to cover every part of our site’s presence. we realized that three solutions were needed – one for drupal, one for wordpress, and one for any stray pages or side projects written in html. as in our desktop presence, the mobile site needed to seamlessly switch between platforms, without any indication to the user that the current solution had been withdrawn and a new one was in place. integration for a seamless experience the goal for integration was a coherent, consistent user experience, referring not just to the overall branding of the site but, more importantly, to the rule that a user would never be thrown out of the mobile-optimized version of the web presence, regardless of where they went within the site. each of the modular solutions provided this in a robust way, and the custom themes, while not all identical to the original designs, were solid representations of the mobile-optimized web presence for the whole site and excluded no necessary content. drupal in researching this issue, it was determined that there were two main ways to create mobile-optimized websites in drupal: the mobile tools module and the mobile plugin module. both offered a certain amount of plug-and-playability and seemed to favor similar methods for mobile conversion, specifically theme-switching. each module set the drupal theme for use in content display based on rules set within the module’s configurations for device groups including desktop, tablet, and handheld devices. the mobile tools module did not show much promise at first. there was very little helpful documentation to consult when something did not work as intended. the initial selection of “theme-switching” in the interface did nothing to change the theme once enabled and configured, and very little could be determined about what was wrong. multiple experiments were performed with site caching and php memory limits, but the module could not function as needed for theme switching. mobile plugin, on the other hand, seemed the better choice. once configured, theme switching worked like a charm on many mobile devices, including tablets and devices with non-standard operating systems. the configuration interface was not terribly user-friendly, to say the least, though with good reason. administrators could set device detection rules for each mobile device type and operating system that might hit the website. though it required more technical knowledge for entering rules, this would actually afford greater control, should the need arise to have even more different themes associated with other devices or screen resolutions. rules are entered as text by the administrator and dictate which devices the module treats as mobile. in the example below, devices in the group “mobile” are treated as mobile and devices in the group “no” are full browser. rules also use a “weight” variable to dictate which theme should be given precedence for devices that land in multiple groups. an example of the mobile plugin rules is shown below: figure 3: mobile plugin device detection rules configuration screen however, it was not without its quirks. by enabling mobile plugin on the drexel university libraries customized drupal installation, our site lost the ability to employ a separate theme in the administrative sections of the website. this option is built into every new drupal installation, and is generally beneficial in helping content editors better understand how to use and edit a site. the newest version of mobile plugin evidently overrode the drupal variable used to set the administrative theme internally, causing the theme to always default to the live one. while it was irksome to lose this feature, it was not a deal breaker when the reward was gaining a completely functional mobile solution for drupal. additionally, theme switching required drupal’s built-in site caching mechanism to be disabled. this would have been true of mobile plugin or mobile tools when using theme switching. theme switching for device detection in drupal requires an absence of drupal performance caching at all times. drupal performance caching consists of built-in content caching features that primarily utilize compression of site elements such as cascading style sheets and javascript files along with database query caching for minor to moderate site performance boosts. this type of caching allows drupal pages to be rebuilt from memory rather than from a live database lookup every time, and so provides a substantial speed increase on websites that use it. with drupal caching turned on, a theme could not switch fast enough—mobile users would still get the full browser site, likely exiting before ever seeing a change to the mobile version. therefore, drupal site caching had to be turned off to use theme switching through mobile plugin. while this was not an ideal situation, it was unavoidable for creating a mobile drupal presence and the performance was not bad enough to force us to scrap the entire mobile effort. even with these two small-to-moderate stumbling blocks, mobile plugin ensured a more robust front-facing user experience; thus it was the solution we chose to implement in the mobile pilot site. building the mobile theme the drupal mobile theme, named drx mobile, was built as a customized fork of nokia mobile, a contributed base theme (available at http://drupal.org/project/nokia_mobile). in the documentation, nokia mobile is the recommended starter theme to accompany sites using mobile plugin, and with good reason; it serves as a helpful and comprehensive framework of base html for mobile designs. to match our own mobile design, nearly all of the built-in look and feel were stripped out and replaced with custom php, html, and css. with the exception of some later additions of content and menu items, the drupal result was extremely similar to what had been planned. figure 4: unaltered nokia mobile theme on a sample site figure 5: drx mobile theme in action figure 6: drx mobile theme in action figure 7: drx mobile theme on a tablet device figure 8: drx mobile theme on a tablet device figure 9: drx mobile theme on a desktop device figure 10: drx mobile theme on a desktop device drupal performance caching the mobile optimization in drupal functioned well with theme switching from the mobile plugin module, but could not be considered a long-term solution as long as we were unable to utilize even basic, built-in caching for site performance in the same drupal installation and had no other caching solution in place for mobile, tablet, or desktop users. we researched some methods for controlling the cache outside of drupal through server configurations, but we needed help balancing the performance load of the high traffic on the drexel university libraries full browser drupal-based site pages. we first considered using a caching proxy such as squid with a reverse proxy setup on the web server. using reverse proxy, we could provide cached versions of popular pages on our site to unlimited users on one server. however, as we were running multiple content management systems, using squid for production would require extensive testing before rollout. this was not a deal breaker, however it seemed that there might have been a simpler solution to our caching problem related solely to the drupal installation. before pursuing squid, we searched out a simpler drupal-based solution, and found one. there was always another option for mobile optimization in drupal. rather than theme switching, we could employ domain switching, directing users to a separate url for the mobile site. our main reason for resisting this approach at the outset was our desire not to manage two separate sites—one mobile, and one desktop. creating two separate sites would mean several headaches. content duplication across the two websites would increase the risk of inconsistent or outdated information on one of them. measures to avoid this would mean twice the work for content editors and site maintainers. most importantly, two sites would create a potentially jarring transition for users between drupal and the rest of the website. the only way that this solution would work was if it allowed us to maintain a single base of information available in our web presence, and display it in an optimized way, responsive to the device of the user. additionally, this was only a drupal problem. as i will detail later in this paper, the wordpress and html pages had no caching problems. keeping the wordpress and html pages on the main website and pushing only the drupal solution off onto a separate mobile domain would create a problematic data silo that we wanted to avoid. therefore, we compromised. researching ways to manage multiple domains within one installation of drupal immediately led us to the domain access module, written for just this purpose. the module could be used to control multiple site instances with multiple backend databases, but most importantly, it could control multiple urls pointing to the same drupal installation, while displaying a different theme based on the url accessed. in other words, we would be able to have one site accessible from two urls, pulling all of the same content from one database, and the only difference would be which theme was displayed. we secured the mobile url of m.library.drexel.edu through the drexel university office of information resources & technology. then we configured the domain access module to pull all content from one database, use the full browser drupal theme for users visiting the site from www.library.drexel.edu, and use drx mobile as the theme when the site was viewed from m.library.drexel.edu. furthermore, the domain access module allowed separate caching configurations for each domain. the main website could therefore finally start using basic drupal performance caching again without a negative impact on mobile users. as an added bonus, within this solution we were also able to employ a separate configuration of drupal performance caching for the mobile theme, which gave it a nice performance boost as well. however, once we got the urls and themes distributed, we still needed a mobile detection mechanism. as we wanted to move away from mobile plugin’s favored method of theme switching for mobile detection, we reexamined the mobile tools module. an unexpected bonus to moving away from mobile plugin was that the administrative theme bug in the mobile tools module went away. we were again able to have a separate theme in the administrative side of drupal. figure 11: drupal theme shown to patrons and anonymous users on the live website figure 12: administrative theme in use on a page administration screen in drupal upon installation and configuration, there was still one relatively small problem. while tablet devices could be assigned a specific theme for theme switching in the mobile tools plugin, tablet devices were automatically classified as desktop devices for domain switching. upon contacting the developer, we were informed that this was because of a belief that tablet devices were closer to desktop devices, and therefore should use the full browser version of a site. for our own configuration, we needed a more flexible approach. we needed to direct tablet device users to the same mobile site as handheld device users.  we built a small custom module we built designed to catch the mobile tools detection variable and set it to “mobile” for users coming into the site from certain tablet devices to solve this problem. after this module was installed, and a few other minor configurations were made, we had a mobile drupal site that seamlessly moved between a mobile url and full browser url, able to blend well with our additional solutions while also taking advantage of caching for site performance. to summarize our drupal research, the following drupal modules were at some point used and/or tested: domain access allows one drupal installation and one shared database to power multiple domains.  http://drupal.org/project/domain domain analytics allows each domain on a single drupal installation to have separate google analytics account settings, which allows for separate analytics information to be collected for each domain.  http://drupal.org/project/domain mobile tools provides theme switching and domain redirection capability for mobile devices.  http://drupal.org/project/mobile_tools mobile plugin offers theme switching and domain redirection through administrator-defined device detection rules for mobile devices.  http://drupal.org/project/mobileplugin wordpress when this project began, the wordpress-powered blogs had already been experimentally employing a plugin called wptouch to make them mobile-friendly. the plugin was a set-and-forget type, which offered no real configuration and no way to preview the look and feel within the browser. it only optimized wordpress blogs, and did so in a very rigid way. the mostly unchangeable interface of the mobile optimized wordpress instances using wptouch resembled that of many news blogs and feed readers. the interface is suited for the delivery of information that changes rapidly over time, for which the context of date and time are integral. unfortunately, wptouch was not right for drexel university libraries’ more stable research guides running on wordpress. the research guides utilized pages in favor of here-today-gone-tomorrow posts, and required front pages with static information. while it was possible to create these in wordpress, it was not possible to display them this way using the wptouch plugin. another obstacle to the continued use of the wptouch plugin related to the fact that, as mobile technologies continued to advance, the wptouch module did too, but as wptouch pro, a plugin that the developers forked from their open-source version and were gearing up to sell commercially. this was signaled by the already-sparse configuration options for the open-source wptouch plugin becoming sparser, and losing some support in favor of the pro version. ultimately, the reason we chose not to upgrade to wptouch pro was the fact that the plugin could only provide a semi-seamless mobile web experience for 50% of our wordpress presence. another approach was required. enter the wp mobile detector plugin. as with wptouch, there were open-source and paid versions of this plugin available, but the open-source version already offered a wealth of configuration options for the creation of a mobile site in wordpress. we were able to drop our own custom wordpress theme into the file structure of the plugin, turn mobile site tracking on or off within the configuration, dynamically resize rich media to fit smaller screens, and, most importantly, create a fluid experience entering and exiting wordpress, for both the blogs and the research guides. in creating the custom drx mobile wordpress theme, based on the anakin mobile wordpress theme, the design changed slightly for both sections. in order to preserve the familiar mobile experience created by using the wptouch plugin, the homepages of the blogs were modeled similarly to their previous appearance. single blog post pages kept drexel university libraries branding and website color combinations, but gained an even more stripped-down look. the horizontal navigation was removed in favor of the bottom-up navigation utilized at the bottom of each mobile drupal page. the research guides gained static homepages for guide listings and the same stripped-down branding, while also keeping the drexel university libraries-consistent color scheme. these themes proved not only sleeker on handheld devices but also pleasantly easy to read on tablet devices. figure 14: mobile-optimized wordpress blog homepage and single post figure 15 and 16: tablet-optimized wordpress blog homepage and single post figure 17 and 18: mobile-optimized wordpress research guide homepage and single guide figure 19 and 20: tablet-optimized wordpress research guide homepage and single guide unlike our drupal solution, the wordpress mobile solution included minimal drama where caching was concerned. we used the popular w3 total cache plugin for site performance on desktop and mobile devices already. this plugin cached database queries, minified css/javascript files/rss feeds, and utilized browser caching for repeat page views for a performance boost. it also specifically claimed to work well with mobile devices by offering caching configuration options for separate device groups, similar to the mobile plugin drupal module. we were happily surprised to find that the authors of wp mobile detector had taken the w3 total cache plugin into consideration when creating their plugin. a tip rolled right into the wp mobile detector’s wordpress.org faq page (http://wordpress.org/extend/plugins/wp-mobile-detector/faq/) stated that the only configuration needed to make wp mobile detector work while w3 total cache was active was adding the wp mobile detector cookie into the w3 total cache “rejected cookies” list, as show below: figure 21: page cache configuration screen in the w3 total cache plugin, configured to exclude pages defined as mobile by the wp mobile detector plugin from desktop-optimized page caching by rejecting this cookie, the wordpress pages that were labeled “mobile” by wp mobile detector would not be subject to the page content caching of desktop pages through w3 total cache. database caching, external script and feed minification, and other caching mechanisms remained in place for mobile devices. device groups, referred to as “user agent groups” in the w3 total cache plugin, used rules set by administrators for which themes would display dependent on the user agent. any active theme could be selected for forced display on any user agent in an administrator-defined group. however, there was also the “pass-through” option, which, when selected, passed the specified user agent group through the w3 total cache module’s theme selection for caching and allowed another theme switching solution, such as the wp mobile detector’s theme switching mechanism, to take precedence. figure 22: configuration screen for sorting user agents into groups for specific theme and url redirection in the w3 total cache plugin the plugins used that were crucial to creating our mobile-optimized wordpress configuration are as follows: google analytics for wordpress allows wordpress blogs to be associated with google analytics accounts for analytics and metadata tracking.  http://wordpress.org/extend/plugins/google-analytics-for-wordpress/ w3 total cache caches wordpress sites through database query caching and minified css/javascript – also caches by device group to support mobile theme switching.  http://wordpress.org/extend/plugins/w3-total-cache/ wordpress mobile admin delivers mobile-optimized display of wordpress administrative pages for mobile content management.  http://wordpress.org/extend/plugins/wordpress-mobile-admin/ wp mobile detector detects mobile devices and displays an administrator-selected mobile-optimized theme based on the device.  http://wordpress.org/extend/plugins/wp-mobile-detector/ css and media queries with content under the drupal and wordpress content management systems optimized for mobile devices, the only pages left with a decided support gap were some semi-static pages written in html and css. there were not many of these pages, but any of them could cause a break in the mobile-optimized user experience, so a solution was put in place. cascading style sheets provided some of the easiest-to-implement solutions for creating mobile content. we used the technique of conditionally calling style sheets to hide certain non-optimized content, reveal mobile-only content, and change the look and feel of content existing in both realms. to conditionally call style sheets, we used a media query. a media query is a logical expression check performed by the user agent against the current media type—in this case “screen”—for the conditions or parameters of a certain feature of that media type. features can be one of many variable display items, including aspect ratio, color display, screen orientation, and device width. our css solution consisted of a combination of three style sheets – one for mobile content on the smallest screens, one for standard mobile screens, and one for overriding any mobile-specific css classes or content. to determine when to call each style sheet, we employed media queries as follows: <link rel="stylesheet" media="screen and (max-device-width: 480px)" href="micro-handheld.css" /> <link rel="stylesheet" media="screen and (min-device-width: 481px)" href="handheld.css" /> <link rel="stylesheet" media="all and (min-device-width: 1024px)" href="override-handheld.css" /> using this code, we were able to determine the maximum and minimum device width of any incoming users. this media query checked the width of the user’s device and called the style sheet only if width conditions were met. if a device with a screen width of 480 pixels or less called up the website, the “micro-handheld” style sheet was called. if a device with a minimum screen width of 481 pixels called up the site, the “micro-handheld” style sheet was not called and the regular “handheld” style sheet for handheld device classes was called instead. finally, if a device with a screen width of 1024 pixels or more called up the site, the handheld classes were overridden in the “override-handheld” style sheet. this css technique was used not only for side projects and the occasional html page, but has also come into use on content in both drupal and wordpress pages. data tables and wide-format content that display well in a full browser site but not on a mobile device have mobile-specific css classes assigned to their html elements. these css classes re-style the content for better readability on smaller screens. figure 24: full browser version of the drupal-powered hagerty library staff page figure 24: full browser version of the drupal-powered hagerty library staff page this css solution is by far the most reliably adaptive solution for optimizing web content for mobile users, changing mostly when the standards of actual device widths change. possibly our best example of its extensive use to date is in this past year’s drexel university libraries annual report 2009-2010, web edition. figure 26: full browser version of annual report homepage figure 27: tablet version of annual report homepage figure 28: handheld version of annual report homepage the handheld version of the annual report was designed after the full browser version was complete. in the full browser version, all of the report’s content was dynamically displayed using jquery with pagination effects. these effects suited a desktop user agent, but were inappropriate for the stripped-down look required for an easy-to-use mobile interface. as each section was loaded dynamically, the desktop version consisted of one index file, so we opted to create the mobile homepage’s content—a list and the report’s description—at the end of that index file. each item in the mobile homepage’s list linked to small, separate, mobile-optimized pages, one for each of the report’s sections. separate pages for each section made it easier for handheld and tablet users to bookmark sections of the annual report for later referral and sharing. media queries were used to stop the mobile version from appearing in the desktop version, and vice versa. from there, the tablet-optimized version’s css, which would override the mobile css, was called using an additional media query on the mobile version of the site’s html. we made a concerted effort to create specially optimized experiences for both handheld and tablet device users when building this application, because it was designed with the idea of allowing visitors from outside of the library to learn more about our institution and accomplishments in a friendly and intuitive way, a way encouraged by the social nature of information sharing on mobile devices. the result has afforded outside visitors a positive experience with this report, and has also made it far easier for our professional staff to refer to it in professional conversations and presentations. conclusion our current successful recipe for a seamless mobile web experience is as follows: drupal mobile url domain access module mobile tools module configured to use domain switching with the mobile url small custom module to augment tablet support wordpress wp mobile detector plugin custom theme based on the contributed anakin theme w3 total cache plugin, configured to exclude the detection cookie used by the mobile plugin html css media queries conditionally calling style sheets based on screen resolution each of these solutions requires minimum upkeep and maintenance from developers and creates no extra work for content editors. as we have been able to keep all of the content for each platform or technology coming from the same source as the full browser version, the daily workflow of editors and site maintainers remains the same. the most up-to-date web content is available at every point of access to drexel university libraries website users, whether they enter from a desktop computer, an ipad, an android, or any other device. these solutions have not only shown that we can create a reasonably responsive design for a site managed across multiple platforms, but have also allowed us to embrace new technology at the same pace as our student body and patron population. this gives them the ability to continue using the drexel university libraries website as a learning tool and educational environment wherever they go and gives us the ability to continue using our website as a tool for information and education for all. related links drexel university libraries: main web site – http://www.library.drexel.edu drupal domain access module (drupal) – http://drupal.org/project/domain mobile plugin module (drupal) – http://drupal.org/project/mobileplugin mobile tools module (drupal) – http://drupal.org/project/mobile_tools nokia mobile theme (drupal) – http://drupal.org/project/nokia_mobile wordpress google analytics plugin – http://wordpress.org/extend/plugins/google-analytics-for-wordpress/ mobile admin plugin – http://wordpress.org/extend/plugins/wordpress-mobile-admin/ mobile detector plugin – http://wordpress.org/extend/plugins/wp-mobile-detector/ mobile detector plugin faq – http://wordpress.org/extend/plugins/wp-mobile-detector/faq/ w3 total cache plugin – http://wordpress.org/extend/plugins/w3-total-cache/ about the author katherine lynch works as web developer for drexel university libraries in philadelphia, pa. she presented on drupal mobile development at drupal ig meeting annual 2011 at 2011 ala annual conference, and writes and presents extensively on drupal, mobile development, and web accessibility. more information can be found at her website, http://www.katherinelynch.org/. subscribe to comments: for this article | for all articles 2 responses to "creating a seamless cross-platform online experience for mobile users" please leave a response below: junior tidal, 2012-02-03 excellent article. right now we’re using a separate mobile site outside of our d6 installation and using the mobile tools module, but it doesn’t seem to be working too well. i think i’ll have to check out the mobile plug-in. joshua odmark, 2012-08-06 katherine, what a great writeup. i enjoyed the flow of the article as it followed your thought process. i wrote the wp mobile detector, so it was a pleasure reading how it has helped create a mobile website for drexel university libraries. cheers. leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – why purchase when you can repurpose? using crosswalks to enhance user access mission editorial committee process and structure code4lib issue 11, 2010-09-21 why purchase when you can repurpose? using crosswalks to enhance user access the mansfield library subscribes to the readex database u.s. congressional serial set, 1817-1994 (full-text historic reports of congress and federal agencies). given the option of purchasing marc records for all 262,000 publications in the serial set or making use of free access to simple dublin core records provided by readex, the library opted to repurpose the free metadata. the process that the mansfield library used to obtain the dublin core records is described, including the procedures for crosswalking the metadata to marc and batch loading the bibliographic records complete with holdings information to the local catalog. this report shows that we successfully achieved our goals of dramatically increasing access to serial set material by exposing metadata in the local catalog and discusses the challenges we faced along the way. we hope that others tasked with the manipulation of metadata will be able to use what we learned from this project. by teressa m. keenan introduction the university of montana-missoula (um) is a doctoral institution serving approximately 12,000 undergraduates and 2,000 graduate students. faculty and staff comprise a population of over 1,500. um is part of the montana university system with three affiliated campuses in dillon, helena, and butte. the maureen and mike mansfield library (ml) holds the largest collection of books and media in montana. ml’s collections exceed one million volumes, 125,000 maps, 100,000 photographs, 50,000 media items, over 30,000 print and electronic journals, and access to over 300 databases. however, like many institutions its size, the library has a limited budget and large expenditures must be carefully considered. thus when the library was faced with the option of purchasing vs. repurposing metadata to augment the catalog, they chose repurposing. in january 2007 ml subscribed to the readex database u.s. congressional serial set, 1817-1994, a collection of full-text historic reports of congress and federal agencies. access to the database was provided through a database a-z list and a subject-based libguide. when readex made bibliographic records available to subscribers, the library decided to add these to the local catalog in order to increase access to the database. readex offered subscribers the option of purchasing a set of marc format records for $25,000 or receiving a set of dublin core (dc) format records at no additional charge. the addition of a metadata librarian to ml’s technical services unit in september 2008 provided the personnel resource needed to repurpose the free metadata. the project the serial set project was initiated in order to achieve a cost-effective way of increasing patron access to and use of the serial set database. initial concerns centered on the quality of the bibliographic records and our ability to successfully crosswalk a less granular metadata schema such as dublin core into the more granular and structured rules of aacr2 and marc. a test was run on a small set of records; the data was collected, crosswalked and then loaded into the library’s test database. the results were then compared to the marc records available for purchase (see appendix a). after consultations between project team members, it was determined that the metadata in the dc records provided by readex was compatible with the standards set for the library’s bibliographic catalog. while it was acknowledged that the marc records available for purchase more closely followed the conventions of aacr2 than the crosswalked records and that cataloging each title individually following aacr2 would produce more robust and complete bibliographic records, we felt that the information provided from the dc records would display appropriately in our catalog, would be sufficient for our goals and would enhance our users’ ability to find, identify, and access these resources. the project was broken down into three tasks: collect the available dublin core records, transform the xml into a compatible format, and load the metadata into the library’s integrated library system (ils). collecting the data the first available segment of simple dublin core records provided by readex included data for every publication contained in serial set volumes 1 to 5377, approximately 168,800 records. the records were harvested over oai-pmh with marcedit, a windows-based freeware marc editing tool already in use at the library (reese, n.d.). the interface was straightforward once the correct server address was obtained from readex. the download process took approximately 10 hours. when complete, 6827 xml files, each containing data for approximately 40 individual titles, had successfully downloaded. transforming the data the first step in transforming the metadata into a form that was ingestible by the library’s voyager catalog was to create a crosswalk from dublin core to marc. existing dc to marc crosswalks (lc, 2008; dutta, 2003) were compared with the indexing capabilities of the local ils and a working crosswalk was created specifically for this project. (see appendix b) we edited the existing xslt in marcedit to match our project crosswalk specifications, but stumbled in our initial attempts to edit the script due to lack of experience with xml and xslt. for example, we could not separate the year from the rest of the date field in the dublin core record until we learned about the substring-before function. another issue we had with the bibliographic records after running the xslt script was the inclusion of unwanted whitespace within the subfields in the 1xx and 7xx marc fields. while the whitespace did not show in the public display of the catalog and did not appear to affect searching it was visible in the staff side display in the cataloging module of voyager. we were unable to remove the whitespace with xslt, but found success with find and replace in marcedit, so we added that step to its automated tasks. the following example contains extra whitespace in the date subfield: 710 10‡au.s. congress. house. committee on the judiciary ‡d (1813). here is the same field, with the whitespace removed: 710 10‡au.s. congress. house. committee on the judiciary ‡d(1813). additional bibliographic data not included in the dc record were incorporated into the final marc record through the xslt stylesheet and the batch editing features in the marcedit program. additional data included: the government documents classification number stem (=086 0\ ‡a y 1.1/2:), an authentication code to indicate that the record was converted from a simple resource description record in another syntax using the dublin core metadata element set, and that the content of the record (descriptive elements and headings) may or may not follow any cataloging standard (=042 \\ ‡a dc) a reproduction note indicating that the material was provided by readex and has restricted access (=533 \\ ‡a electronic reproduction ‡b chester, vt. ‡c newsbank, inc. ‡d 2005. ‡n available via the world wide web ‡n access restricted to readex u.s. congressional serial set subscribers) a local note indicating that the same material is also available in print (=590 \\ ‡a ml: this title is also available in print. see the “united states congressional serial set”, call number y 1.1/2:) a public note in the electronic location and access field (=856 40 ‡z connect to this title online.) holdings/item record information (=949 \\ ‡c uonetlib ‡d y 1.1/2: ‡t useminet) see appendix c for the full xslt stylesheet. batch loading the data in an effort to reduce the overall time and manual interaction with the transformation, editing and batch load process, we wrote a script to combine the numerous records into six batch files. once the data were transformed, voyager batch loading protocols were used to add the marc bibliographic, holdings and item records to the catalog. a load profile was established on voyager that defined the load parameters, including expected character set, fields to match on, how to handle matches (e.g., replace or merge), and location of holdings information. the batch files were loaded using parameters on the server, and system logs were reviewed for problems or errors upon completion each of these batch files, roughly 64mb in size, required 18 hours to finish loading. therefore, the timing of the load became an important consideration. the first file load was interrupted by our routine daily system back-up. once this collision was discovered, further file loads were timed to not coincide with the back-up process. as the process of loading the files continued, we discovered that each subsequent file took longer to load. we decreased the processing time somewhat by loading the files without keyword indexing, then re-indexed after the project was completed. in order to minimize load time in the future, records will not be processed with the above script. instead, we’ll use the batch edit feature of marcedit to combine the records into a number of smaller batches. this revised procedure will involve less manual interaction while avoiding the problems we encountered with very large files. project results (cost effectiveness & user access) when completed, over 262,000 bibliographic, holding, and item records were added to the library’s catalog. the process took approximately 46 staff hours and just over 260 computer hours to complete. based on these statistics, repurposing of the metadata allowed the database to be enriched by the addition of about 856 titles per hour. while the library did not have any initial purchase costs associated with the project (the metadata was freely available to database subscribers, and the software used was either open source or already in use for other applications), there were costs involved (table 1). most of the staff time was spent in planning, research, and the development of the metadata transformation scripts. staff time estimates include salary and benefits. approximations for computer time and other overhead costs such as electricity, etc. were not included in this analysis. repurpose purchase data acquisition $0.00 $25,000.00 staff time (systems support) $614.80 $412.46 staff time (metadata librarian) $514.25 $257.25 total estimated cost $1,129.05 $25,669.71 table 1. cost comparison chart system supplied statistics for use of the serial set database show an overall increase in use since the library subscribed in january 2007 (table 2). overall usage increased by 2,577 views in the nine months following the addition of individual title records to the catalog. the addition of bibliographic records to the library catalog is not the only factor affecting use of the database. for example, reference librarians include that database in their research classes. also, in april 2009 the library celebrated its 100th year as a federal depository, an event which helped focus attention on this database. initiating a better method of tracking use (including how users got to the database, rather than just the number of times a database was viewed), would provide better analysis opportunities for determining success of future projects intended to increase access to collections. year january february march april may june july august september october november december total 2010 2014 1151 1686 n/a n/a n/a n/a n/a n/a n/a n/a n/a 4851 2009 911 1925 4499 4956 1433 2513 285 587 1092 2497 620 304 21622 2008 427 579 631 887 154 656 62 136 759 383 1308 812 6794 2007 488 175 578 1745 735 672 618 553 165 349 1333 1377 8788 table 2. use statistics conclusions and future plans the repurposing of the pre-existing metadata to increase access to digital material was an overall success. repurposing the metadata proved to be a cost effective alternative to purchasing marc records, saving the library over $24,500. use statistics show a marked increase in database usage after the records were added to the catalog. because this project was the first of its kind at our library, the project team gained valuable knowledge that can be utilized in the future. because of inexperience with xslt and scripting in general, creating the crosswalk took a bit of trial and error. additionally, the marcedit software was new to members of the project team. the reference section of this report includes materials that were helpful in learning the basics of xslt and marcedit. according to the readex webpage, additional dublin core records will be available each month (readex, n.d.). the library plans to selectively harvest the new records, crosswalk them to marc, and add the data to our catalog using the workflows established in this project. future research plans include enhancing our understanding of xslt and consideration of additional ways to repurpose existing metadata to enhance our users’ experience. references dutta, b. (2003). cataloguing web documents using dublin core, marc 21. presented at the workshop on digital libraries: theory and practice, drtc, bangalore. (coins) library of congress, network development and marc standards office. (2008). dublin core to marc crosswalk. retrieved march 30, 2010, from http://www.loc.gov/marc/dccross.html open archives initiative – protocol for metadata harvesting – v.2.0. (n.d.). . retrieved march 30, 2010, from http://www.openarchives.org/oai/openarchivesprotocol.html otegem, m. (2002). sams teach yourself xslt in 21 days. indianapolis in: sams. (coins) readex. (n.d.). marc records. retrieved march 31, 2010, from http://www.readex.com/readex/index.cfm?content=296 reese, t. (2009a). youtube – add new metadata function. retrieved march 31, 2010, from http://www.youtube.com/watch?v=3x5ke81aoeu reese, t. (2009b). youtube – translating oai metadata to marc using marcedit. retrieved march 31, 2010, from http://www.youtube.com/watch?v=gvbrmvh6j7u reese, t. (n.d.). marcedit homepage: your complete free marc software. marcedit. retrieved march 30, 2010, from http://people.oregonstate.edu/~reeset/marcedit/html/index.php appendix a: metadata comparison the following example of the simple dublin core and marc records available through readex (readex, n.d.), followed by the crosswalked and edited marc record created by ml, demonstrates the similarities and differences in the granularity of the marc records: simple dublin core, serial set <record> <header> <identifier>oai:docs.newsbank.com/sset.0fdbe2664851d648</identifier> <datestamp>1817-12-08</datestamp> <setspec>sset</setspec> </header> <metadata> <oai_dc:dc xmlns:oai_dc="http://www.openarchives.org/oai/2.0/oai_dc/" xsi:schemalocation="http://www.openarchives.org/oai/2.0/oai_dc.xsd"> <dc:type> motions and resolutions</dc:type> <dc:date>1817-12-08</dc:date> <dc:coverage>congressional session number 15th congress, 1st session; session volume number 1</dc:coverage> <dc:description>senate document</dc:description> <dc:source>serial set number 2 s.doc. 3</dc:source> <dc:identifier>http://docs.newsbank.com/select/serialset/0fdbe2664851d648.html</dc:identifier> <dc:language>english</dc:language> <dc:title>in senate of the united states, december 8, 1817. mr. sanford submitted the following motion for consideration: resolved, that the committee of finance inquire what alterations or amendments may be requisite in the present system of collecting the duties charged...</dc:title> <dc:creator>sanford, nathan, 1777-1838, republican-jeffersonian (ny)</dc:creator> <dc:creator>u.s. congress. senate</dc:creator> <dc:subject>customs administration</dc:subject> <dc:subject>foreign trade</dc:subject> <dc:subject>imports</dc:subject> <dc:subject>tariffs and duties</dc:subject> <dc:description>1 p.</dc:description> </oai_dc:dc> </metadata> </record>   readex marc record =ldr 01911cam 22003371i 4500 =001 nb00000000008 =003 readex =005 20090622155353.1 =006 m\\\\\\\\d =007 cr\cn\|||||||| =008 070221s1817\\\\dcu\\\\\\s\\\f000\0\eng\d =035 \\ ‡a (readex)0fc9a3c9017886a0 =040 \\ ‡a readex ‡c readex =110 1\ ‡a united states. ‡b congress. ‡b senate. =245 10 ‡a in senate of the united states, december 8, 1817. mr. sanford submitted the following motion for consideration: resolved, that the committee of finance inquire what alterations or amendments may be requisite in the present system of collecting the duties charged... ‡h [electronic resource] =260 \\ ‡a washington, dc, ‡c 1817 =300 \\ ‡a 1 p. =440 \0 ‡a united states congressional serial set; ‡v serial set no. 2 =490 1\ ‡a senate document / 15th congress, 1st session. senate ; ‡v no. 3 =500 \\ ‡a title taken from opening lines of text. =533 \\ ‡a electronic reproduction. ‡b chester, vt.: ‡c newsbank, inc., ‡d 2005. ‡n available via the world wide web. ‡naccess restricted to readex u.s. congressional serial set subscribers. =540 \\ ‡a copyright 2007 by newsbank, inc. all rights reserved. =610 17 ‡a united states. ‡b congress. ‡b senate. ‡b committee on finance. ‡g (1816) ‡2 readex congressional thesaurus =650 07 ‡a customs administration. ‡2 readex congressional thesaurus =650 07 ‡a foreign trade. ‡2 readex congressional thesaurus =650 07 ‡a imports. ‡2 readex congressional thesaurus =650 07 ‡a tariffs and duties. ‡2 readex congressional thesaurus =655 \7 ‡a motions and resolutions. ‡2 readex congressional thesaurus =700 1\ ‡a sanford, nathan, ‡d 1777-1838. ‡u republican-jeffersonian (ny) =830 \0 ‡a senate document (united states. congress. senate) ; ‡v 15th congress, 1st session, no. 3 =856 40 ‡u http://docs.newsbank.com/select/serialset/0fdbe2664851d648.html   crosswalked & edited marc record =000 01927cam a 00361mi 00 =001 1552483 =005 20100704120842.0 =007 cr cn||||||||| =008 090331s1817 dcu|||||s||||f||| 0|eng|d =035 \\ ‡a (readex)0fdbe2664851d648 =040 \\ ‡a readex ‡c readex =042 \\ ‡a dc =086 0\ ‡a y 1.1/2:serial set number 2 =110 1\ ‡a united states. congress. senate =245 10 ‡a in senate of the united states, december 8, 1817. mr. sanford submitted the following motion for consideration: resolved, that the committee of finance inquire what alterations or amendments may be requisite in the present system of collecting the duties charged... ‡h [electronic resource] =260 \\ ‡a [washington, dc], ‡c 1817. =300 \\ ‡a 1 p. =490 \0 ‡a congressional session number 15th congress, 1st session; session volume number 1 =500 \\ ‡a title from opening lines of text. =520 \\ ‡a senate document =533 \\ ‡a electronic reproduction ‡b chester, vt. ‡c newsbank, inc. ‡d 2005. ‡n available via the world wide web ‡n access restricted to readex u.s. congressional serial set subscribers =540 \\ ‡a copyright 2005 by newsbank, inc. all rights reserved. =546 \\ ‡a english =590 \\ ‡a ml: this title is also available in print. see the "united states congressional serial set", call number y 1.1/2: =650 07 ‡a customs administration ‡2 readex congressional thesaurus =650 07 ‡a foreign trade ‡2 readex congressional thesaurus =650 07 ‡a imports ‡2 readex congressional thesaurus =650 07 ‡a tariffs and duties ‡2 readex congressional thesaurus =655 7\ ‡a motions and resolutions ‡2 readex congressional thesaurus =700 10 ‡a sanford, nathan, ‡d 1777-1838, ‡g republican-jeffersonian (ny) =786 0\ ‡n serial set number 2 s.doc. 3 =856 41 ‡u http://weblib.lib.umt.edu/redirect/proxyselect.php?url=http://docs.newsbank.com/select/serialset/0fdbe2664851d648.html ‡z connect to this title online. =949 \\ ‡c uonetlib ‡d y 1.1/2: ‡t useminet appendix b: dc to marc map the following table contains the mapping/crosswalk between the simple dublin core elements found in the readex serial set records and marc bibliographic data elements: dublin core marc title 245 10 ‡a (title statement/title proper) subject 650 07 ‡a (subject added entry – topical term) ‡2 (source of thesaurus) description 520 \\ ‡a (summary, etc. note) 300 \\ ‡a (physical description – extent) source 786 0\ ‡n (data source note) 086 0\ ‡a (government document classification number) language 546 \\ ‡a (language note) coverage 490 \0 ‡a (uncontrolled series statement) creator 110 1\ ‡a (main entry – corporate name) 100 1\ ‡a (main entry – personal name) 710 2\ ‡a (added entry – corporate name) 700 1\ ‡a (added entry – personal name) date 260 \\ ‡c (date of publication, distribution, etc.) identifier 856 40 ‡u (electronic location & access/url) type 655 \7 ‡a (index term – genre/form) appendix c: dc to marc xslt the following xslt stylesheet was used for dc to marc translation: <?xml version="1.0" encoding="utf-8"?> <xsl:stylesheet version="1.0" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:dcterms="http://purl.org/dc/terms/1.1" xmlns:oai_dc="http://www.openarchives.org/oai/2.0/oai_dc/" xmlns:xsi="http://www.w3.org/2001/xmlschema-instance" xsi:schemalocation="http://www.openarchives.org/oai/2.0/oai_dc/ http://www.openarchives.org/oai/2.0/oai_dc.xsd" xmlns:xsl="http://www.w3.org/1999/xsl/transform" xmlns="http://www.loc.gov/marc21/slim" exclude-result-prefixes="dc dcterms oai_dc"> <xsl:import href="marc21slimutils.xsl"/> <xsl:output method="xml" encoding="utf-8" indent="yes"/> <xsl:template match="/"> <collection xmlns:xsi="http://www.w3.org/2001/xmlschema-instance" xsi:schemalocation="http://www.loc.gov/marc21/slim http://www.loc.gov/standards/marcxml/schema/marc21slim.xsd"> <xsl:apply-templates /> </collection> </xsl:template> <xsl:template name="oai-pmh"> <xsl:for-each select = "listrecords/record/metadata/oai_dc:dc"> <xsl:apply-templates /> </xsl:for-each> <xsl:for-each select = "getrecord/record/metadata/oai_dc:dc"> <xsl:apply-templates /> </xsl:for-each> </xsl:template> <xsl:template match="text()" /> <xsl:template match="oai_dc:dc"> <record> <xsl:element name="leader"> <xsl:variable name="type" select="dc:type"/> <xsl:variable name="leader06"> <xsl:choose> <xsl:when test="$type='collection'">p</xsl:when> <xsl:when test="$type='dataset'">m</xsl:when> <xsl:when test="$type='event'">r</xsl:when> <xsl:when test="$type='image'">k</xsl:when> <xsl:when test="$type='interactive resource'">m</xsl:when> <xsl:when test="$type='service'">m</xsl:when> <xsl:when test="$type='software'">m</xsl:when> <xsl:when test="$type='sound'">i</xsl:when> <xsl:when test="$type='text'">a</xsl:when> <xsl:otherwise>a</xsl:otherwise> </xsl:choose> </xsl:variable> <xsl:variable name="leader07"> <xsl:choose> <xsl:when test="$type='collection'">c</xsl:when> <xsl:otherwise>m</xsl:otherwise> </xsl:choose> </xsl:variable> <xsl:value-of select="concat(' ',$leader06,$leader07,' 3u ')"/> </xsl:element> <datafield tag="035" ind1=" " ind2=" "> <subfield code="a"> <xsl:text>(readex)</xsl:text> <xsl:value-of select="substring(dc:identifier,43,16)" /> </subfield> </datafield> <datafield tag="042" ind1=" " ind2=" "> <subfield code="a">dc</subfield> </datafield> <datafield tag="040" ind1=" " ind2=" "> <subfield code="a">readex</subfield> <subfield code="c">readex</subfield> </datafield> <datafield tag="086" ind1="0" ind2=" "> <subfield code="a"> <xsl:text>y 1.1/2:</xsl:text> <xsl:value-of select="substring-before(dc:source,'s.')" /> <xsl:value-of select="substring-before(dc:source,'h.')" /> </subfield> </datafield> <xsl:for-each select="dc:creator"> <xsl:choose> <xsl:when test="(.!='') and (position()=1)"> <xsl:call-template name="persname_template"> <xsl:with-param name="string" select="." /> <xsl:with-param name="field" select="'100'" /> <xsl:with-param name="ind1" select = "'1'" /> <xsl:with-param name="ind2" select = "' '" /> <xsl:with-param name="type" select="'author'" /> </xsl:call-template> </xsl:when> <xsl:otherwise> <xsl:if test=".!=''"> <xsl:call-template name="persname_template"> <xsl:with-param name="string" select="." /> <xsl:with-param name="field" select="'700'" /> <xsl:with-param name="ind1" select = "'1'" /> <xsl:with-param name="ind2" select = "'0'" /> <xsl:with-param name="type" select="'author'" /> </xsl:call-template> </xsl:if> </xsl:otherwise> </xsl:choose> </xsl:for-each> <xsl:for-each select="dc:title[1]"> <datafield tag="245" ind1="1" ind2="0"> <subfield code="a"> <xsl:value-of select="."/> </subfield> </datafield> </xsl:for-each> <xsl:for-each select="dc:title[position()>1]"> <xsl:if test=".!=''"> <datafield tag="246" ind1="3" ind2="3"> <subfield code="a"> <xsl:value-of select="."/> </subfield> </datafield> </xsl:if> </xsl:for-each> <xsl:choose> <xsl:when test="dc:publisher"> <xsl:if test="translate(dc:publisher/.,'.,:;','')!=''"> <datafield tag="260" ind1=" " ind2=" "> <xsl:choose> <xsl:when test="dc:date"> <subfield code="a"><xsl:text>[washington, dc],</xsl:text></subfield> <subfield code="b"> <xsl:value-of select="dc:publisher[1]"/>, </subfield> <xsl:if test="translate(dc:date[1]/., '.,:;','')!=''"> <subfield code="a"><xsl:text>[washington, dc],</xsl:text></subfield> <subfield code="c"> <xsl:value-of select="substring-before(dc:date[1],'-')" />. </subfield> </xsl:if> </xsl:when> <xsl:otherwise> <subfield code="a"><xsl:text>[washington, dc],</xsl:text></subfield> <subfield code="b"> <xsl:value-of select="dc:publisher[1]"/>. </subfield> </xsl:otherwise> </xsl:choose> </datafield> </xsl:if> </xsl:when> <xsl:otherwise> <xsl:if test="translate(dc:date[1],'.,:;','')!=''"> <datafield tag="260" ind1=" " ind2=" "> <subfield code="a"><xsl:text>[washington, dc],</xsl:text></subfield> <subfield code="c"> <xsl:value-of select="substring-before(dc:date[1],'-')" />. </subfield> </datafield> </xsl:if> </xsl:otherwise> </xsl:choose> <xsl:for-each select="dc:coverage"> <xsl:choose> <xsl:when test="translate(., '0123456789-.?','')=''"> <!--likely;this is a date--> <datafield tag="500" ind1=" " ind2=" "> <subfield code="a"> <xsl:value-of select="."/> </subfield> </datafield> </xsl:when> <xsl:otherwise> <!--likely a geographic subject, we will print this later--> </xsl:otherwise> </xsl:choose> </xsl:for-each> <xsl:for-each select="dc:identifier"> <xsl:if test="position()!=last()"> <datafield tag="500" ind1=" " ind2=" "> <subfield code="a"> <xsl:value-of select="." /> </subfield> </datafield> </xsl:if> </xsl:for-each> <xsl:for-each select="dc:description"> <datafield tag="520" ind1=" " ind2=" "> <subfield code="a"> <xsl:value-of select="normalize-space(.)"/> </subfield> </datafield> </xsl:for-each> <xsl:for-each select="dc:rights"> <datafield tag="540" ind1=" " ind2=" "> <subfield code="a"> <xsl:value-of select="."/> </subfield> </datafield> </xsl:for-each> <datafield tag="533" ind1=" " ind2=" "> <subfield code="a">electronic reproduction</subfield> <subfield code="b">chester, vt.</subfield> <subfield code="c">newsbank, inc.</subfield> <subfield code="d">2005.</subfield> <subfield code="n">available via the world wide web</subfield> <subfield code="n">access restricted to readex u.s. congressional serial set subscribers</subfield> </datafield> <xsl:for-each select="dc:language"> <datafield tag="546" ind1=" " ind2=" "> <subfield code="a"> <xsl:value-of select="."/> </subfield> </datafield> </xsl:for-each> <xsl:for-each select="dc:subject"> <xsl:call-template name="subj_template"> <xsl:with-param name="field" select="'650'" /> <xsl:with-param name="ind1" select = "'0'" /> <xsl:with-param name="ind2" select = "'7'" /> <xsl:with-param name="string" select="." /> <xsl:with-param name="delimiter" select="';'" /> </xsl:call-template> </xsl:for-each> <xsl:for-each select="dc:coverage"> <xsl:choose> <xsl:when test="translate(., '0123456789-.?','')=''"> <!--likely; this is a date--> </xsl:when> <xsl:otherwise> <!--likely a geographic subject--> <datafield tag="490" ind1=" " ind2="0"> <subfield code="a"> <xsl:value-of select="." /> </subfield> </datafield> </xsl:otherwise> </xsl:choose> </xsl:for-each> <xsl:for-each select="dc:type"> <datafield tag="655" ind1="7" ind2=" "> <subfield code="a"> <xsl:value-of select="."/> </subfield> <subfield code="2">readex congressional thesaurus</subfield> </datafield> </xsl:for-each> <xsl:for-each select="dc:contributer"> <xsl:call-template name="persname_template"> <xsl:with-param name="string" select="." /> <xsl:with-param name="field" select="'100'" /> <xsl:with-param name="ind1" select = "'1'" /> <xsl:with-param name="ind2" select = "'0'" /> <xsl:with-param name="type" select="'contributor'" /> </xsl:call-template> </xsl:for-each> <xsl:for-each select="dc:source"> <datafield tag="786" ind1="0" ind2=" "> <subfield code="n"> <xsl:value-of select="."/> </subfield> </datafield> <datafield tag="590" ind1=" " ind2=" "> <subfield code="a"> <xsl:text>ml: this title is also available in print. see the "united states congressional serial set", call number y 1.1/2:</xsl:text> </subfield> </datafield> <datafield tag="949" ind1=" " ind2=" "> <subfield code="c">uonetlib</subfield> <subfield code="d"> <xsl:text>y 1.1/2:</xsl:text> <xsl:value-of select="substring-before(dc:source,'s.')" /> <xsl:value-of select="substring-before(dc:source,'h.')" /> </subfield> <subfield code="t">useminet</subfield> </datafield> </xsl:for-each> <xsl:for-each select="dc:relation"> <datafield tag="787" ind1="0" ind2=" "> <subfield code="n"> <xsl:value-of select="."/> </subfield> </datafield> </xsl:for-each> <xsl:if test="dc:identifier"> <datafield tag="856" ind1="4" ind2="1"> <subfield code="u"> <xsl:text>http://weblib.lib.umt.edu/redirect/proxyselect.php?url=</xsl:text> <xsl:value-of select="dc:identifier[last()]" /> </subfield> <subfield code="z">connect to this title online.</subfield> </datafield> </xsl:if> </record> </xsl:template> <!--subject template--> <xsl:template name="subj_template"> <xsl:param name="field" /> <xsl:param name="ind1" /> <xsl:param name="ind2" /> <xsl:param name="string" /> <xsl:param name="delimiter" /> <xsl:choose> <!-if a paren, stop at an opening semicolon --> <xsl:when test="contains($string, $delimiter)!=0"> <xsl:variable name="newstem" select="substring-after($string, $delimiter)" /> <datafield> <xsl:attribute name="tag"> <xsl:value-of select="$field" /> </xsl:attribute> <xsl:attribute name="ind1"> <xsl:value-of select="$ind1" /> </xsl:attribute> <xsl:attribute name="ind2"> <xsl:value-of select="$ind2" /> </xsl:attribute> <subfield code="a"> <xsl:value-of select="substring-before($string, $delimiter)" /> </subfield> </datafield> <!--need to do recursion--> <xsl:call-template name="subj_template"> <xsl:with-param name="field" select="'650'" /> <xsl:with-param name="ind1" select="'0'" /> <xsl:with-param name="ind2" select="'7'" /> <xsl:with-param name="string" select="$newstem" /> <xsl:with-param name="delimiter" select="';'" /> </xsl:call-template> </xsl:when> <xsl:otherwise> <datafield> <xsl:attribute name="tag"> <xsl:value-of select="$field" /> </xsl:attribute> <xsl:attribute name="ind1"> <xsl:value-of select="$ind1" /> </xsl:attribute> <xsl:attribute name="ind2"> <xsl:value-of select="$ind2" /> </xsl:attribute> <subfield code="a"> <xsl:value-of select="$string" /> </subfield> </datafield> </xsl:otherwise> </xsl:choose> </xsl:template> <xsl:template name="persname_template"> <xsl:param name="string" /> <xsl:param name="field" /> <xsl:param name="ind1" /> <xsl:param name="ind2" /> <xsl:param name="type" /> <datafield> <xsl:attribute name="tag"> <xsl:value-of select="$field" /> </xsl:attribute> <xsl:attribute name="ind1"> <xsl:value-of select="$ind1" /> </xsl:attribute> <xsl:attribute name="ind2"> <xsl:value-of select="$ind2" /> </xsl:attribute> <!-sample input: brightman, samuel c. (samuel charles), 1911-1992 --> <!-sample output: $abrightman, samuel c. $q(samuel charles), $d1911-. --> <!-will handle names with dashes e.g. bourke-white, margaret --> <!-capture primary name by looking for a paren or a dash or neither --> <xsl:choose> <!-if a paren, stop at an opening paren --> <xsl:when test="contains($string, '(')!=0"> <subfield code="a"> <xsl:value-of select="substring-before($string, '(')" /> </subfield> </xsl:when> <!-if a dash, check if it's a date or part of the name --> <xsl:when test="contains($string, '-')!=0"> <xsl:variable name="name_1" select="substring-before($string, '-')" /> <xsl:choose> <!-if it's a date remove it --> <xsl:when test="translate(substring($name_1, (string-length($name_1)), 1), '0123456789', '9999999999') = '9'"> <xsl:variable name="name" select="substring($name_1, 1, (string-length($name_1)-6))" /> <subfield code="a"> <xsl:value-of select="$name" /> </subfield> </xsl:when> <!-if it's not a date, check whether there is a date later --> <xsl:otherwise> <xsl:variable name="remainder" select="substring-after($string, '-')" /> <xsl:choose> <!-if there's a dash, assume it's a date and remove it --> <xsl:when test="contains($remainder, '-')!=0"> <xsl:variable name="tmp" select="substring-before($remainder, '-')" /> <xsl:variable name="name_2" select="substring($tmp, 1, (string-length($tmp)-6))" /> <subfield code="a"> <xsl:value-of select="$name_1" />-<xsl:value-of select="$name_2" /> </subfield> </xsl:when> <!-if there's no dash in the remainder, output it --> <xsl:otherwise> <subfield code="a"> <xsl:value-of select="$string" /> </subfield> </xsl:otherwise> </xsl:choose> </xsl:otherwise> </xsl:choose> </xsl:when> <!-no dashes, no parens, just output the name --> <xsl:otherwise> <subfield code="a"> <xsl:value-of select="$string" /> </subfield> </xsl:otherwise> </xsl:choose> <!-capture secondary name in parens for subfield q --> <xsl:if test="contains($string, '(')!=0"> <xsl:variable name="subq_tmp" select="substring-after($string, '(')" /> <xsl:variable name="subq" select="substring-before($subq_tmp, ')')" /> <subfield code="q"> (<xsl:value-of select="$subq" />) </subfield> </xsl:if> <!-capture date for subfield d, assume date is last item in field --> <!-note: does not work if name has a dash in it --> <xsl:if test="contains($string, '-')!=0"> <xsl:variable name="date_tmp" select="substring-before($string, '-')" /> <xsl:variable name="remainder" select="substring-after($string, '-')" /> <xsl:choose> <!-check second half for another dash; if present, assume that is date --> <xsl:when test="contains($remainder, '-')!=0"> <xsl:variable name="tmp" select="substring-before($remainder, '-')" /> <xsl:variable name="date_1" select="substring($remainder, (string-length($tmp)-3))" /> <!-check whether it has a number before it and if so, output it as date --> <xsl:if test="translate(substring($date_1, 1, 1), '0123456789', '9999999999') = '9'"> <subfield code="d"> <xsl:value-of select="$date_1" />. </subfield> </xsl:if> </xsl:when> <!-otherwise this is the only dash so take it --> <xsl:otherwise> <xsl:variable name="date_2" select="substring($string, (string-length($date_tmp)-3))" /> <!-check whether it has a number before it and if so, output it as date --> <xsl:if test="translate(substring($date_2, 1, 1), '0123456789', '9999999999') = '9'"> <subfield code="d"> <xsl:value-of select="$date_2" />. </subfield> </xsl:if> </xsl:otherwise> </xsl:choose> </xsl:if> <subfield code="e"> <xsl:value-of select="$type" /> </subfield> </datafield> </xsl:template> </xsl:stylesheet> about the author teressa m. keenan is the section head of metadata and continuing resources at the university of montana, mansfield library. tags: crosswalks, dublin core, marc, metadata, repurposing data subscribe to comments: for this article | for all articles one response to "why purchase when you can repurpose? using crosswalks to enhance user access" please leave a response below: dorothea salo, 2010-09-22 re killing whitespace in xslt: the normalize-space() function should do the trick nicely. you wind up with something like: <xsl:value-of select=”normalize-space(/whatever you already have here/)”/> leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – pythagoras: discovering and visualizing musical relationships using computer analysis mission editorial committee process and structure code4lib issue 51, 2021-06-14 pythagoras: discovering and visualizing musical relationships using computer analysis this paper presents an introduction to pythagoras, an in-progress digital humanities project using python to parse and analyze xml-encoded music scores. the goal of the project is to use recurring patterns of notes to explore existing relationships among musical works and composers. an intended outcome of this project is to give music performers, scholars, librarians, and anyone else interested in digital humanities new insights into musical relationships as well as new methods of data analysis in the arts. by brandon bellanti background i began this project as a capstone study for my graduate program in library and information science. i was interested in a digital humanities project that would bring together what i was learning related to data mining, database design, data analysis, and linked data. i chose music as a topic of interest because i studied it in my undergrad program and am especially interested in music theory and history. throughout music history, composers influenced and borrowed from one another, and these relationships are revealed through music analysis and theory. some relationships were between teachers and pupils, but others were between rival composers or even between composers and patrons. if composers use the same pattern in their works, there is some relationship represented. this overlap could be as elemental as the use of the same scale or as specific as a musical quotation or set of variations. in the project outlined here, i perform an analytical task – finding recurring patterns across works – that i’ve done many times before, but using a computer to find these patterns rather than finding them manually. i focused only on the most basic matches using intervals alone, but the same method could be expanded to include other facets of musical patterns as well. summary this paper describes an application of computer-aided musicology. patterns, or sequences of notes, recurring within data extracted from xml-encoded music scores are considered to represent some existing relationship between works and composers. the paper outlines my process of collecting, transforming, analyzing, and visualizing music score data with python and other tools. first, i converted musicxml files to csv files, which i loaded into a dataframe. i transformed the note pitches into intervals and converted the intervals to strings. i used a regular expression to find recurring sequences of intervals in the outputted strings, and i stored the returned sequences in a sqlite database along with their respective work titles and composers. finally, i queried the database and used matplotlib and musescore to visualize the patterns. this workflow is outlined in figure 1. figure 1. project workflow for pythagoras. method setting up the environment for this project, i primarily used python 3 in jupyter notebooks. i executed longer-running functions directly through the terminal in visual studio code. i imported several packages, some standard to python and others installed with anaconda. the main ones were lxml, pandas, numpy, matplotlib, and sqlite3. additionally, i used a substitute regex module (barnett, 2020) and the music21 toolkit (cuthbert et. al, 2019) for visualizations. the music21 toolkit works in conjunction with musescore, an open-source music notation software. most of my work was done on a 2015 macbook pro 13-inch computer running macos mojave; i also used a 2015 macbook pro 15-inch computer running macos catalina for the more resource intensive processes such as recursively searching for patterns with regular expressions and writing patterns to a database file. obtaining xml-encoded music score data my initial step was finding music score data in a format i could parse with python. i chose musicxml because of its wide availability. musicxml is an xml-based format primarily used for exchanging data between music notation software, meaning i was not limited to working in a specific application. musicxml files can be generated by manually inputting scores into a notation software or by scanning sheet music using optical music recognition (omr). since musicxml files are meant to exchange, publish, and archive music scores digitally, it is easy to find files online through collections like project gutenberg or the international music score library project (imslp). the scores i used for this project came from a large collection of files i received upon purchasing a license for capella scan, an omr software. the files i received were in a proprietary format (.cap), so i opened each score file in capella 8 and exported it as a musicxml file. i converted a total of 349 files (figure 1, steps 1–3), which included digitized piano, orchestral, and choral scores. parsing musicxml files using lxml after converting the scores to xml files, i used python’s lxml library to parse the files. the xml elements i used in this project are shown in figure 2. i also retrieved the values of the id attribute of the part element and the number attribute of the measure element. figure 2. tree structure of musicxml files used for this project. i wrote a function, xml_to_csv() in figure 3, which iterates over each note element in an xml file (figure 3a) and writes a line to a corresponding csv file (figure 3b) with values for part id, measure number, and note data. note data includes step, octave, duration, type, and accidentals; it also includes whether the note is a rest, grace note, dotted note, tied note, or cue note. three of my files had xml-validation errors from the capella 8 export, but i was able to convert 346 of the 349 xml-encoded scores to csv files (figure 1, steps 3–5). figure 3. converting an xml file to a csv file using xml_to_csv(). figure 3a. snippet from an xml file of mozart’s twelve variations on “ah vous dirai-je, maman”, k. 265/300e before conversion to csv figure 3b. snippet from the corresponding csv file of mozart’s twelve variations on “ah vous dirai-je, maman”, k. 265/300e after conversion from xml and interval calculation building a dataframe and writing to a string using pandas the next step was transforming my data to a format that could be searched for patterns (figure 1, steps 5–7). i planned to use regular expressions to find patterns, so i generated a string of pitches for each musical work using the function csv_to_strings() shown in figure 4. figure 4. reading a file as a dataframe and outputting strings using csv_to_strings(). i loaded each converted csv file into a dataframe using pandas’s read_csv() method, each row representing a single note. with all of the pitches in a series (column) in the dataframe, i used the series.to_string() method to write the pitches to a single, continuous string. i replaced the newline delimiters between the pitches with single dots using string.replace() to make the strings more readable for error checking. this process worked well, but if i had searched the string at this point, i would only find matching patterns if the pitches were exactly the same. regex searches could not recognize recurring patterns at lower or higher pitches or in a different key signature. i solved this issue by instead using intervals that showed the relative distance between the pitches. regardless of the pitch or key signature, the intervals between the steps of recurring patterns would always match. to calculate intervals, i assigned each note in the dataframe a piano keyboard value based on the pitch and octave. i created a dictionary with each pitch and octave combination as the dictionary key and the piano keyboard number as the value. for example, middle c is c4, which is key 40 on a piano keyboard. i assigned the notes these values using the series.apply() method. once each note had been given a numeric value, i used the series.diff() method to create a new series of intervals, calculating the difference between consecutive rows of keyboard numbers. the final step before converting the series to a string was to divide the dataframe by score part. i did this to prevent finding a pattern that occurred only once in each part, as my goal was to find patterns that recurred throughout a work. i iterated through the unique score parts, generated a string of intervals for each part using series.to_string(), and saved the strings to a dictionary. searching for patterns using regular expressions the core of this project is finding patterns in music score data, and i accomplished that through regular expression (regex) searches (figure 1, steps 7–9). instead of python’s standard regex library, i used an alternate regex package that could find overlapping matches. i wrote a regular expression that searches for a pattern of a minimum length that recurs a minimum number of times: `((?<=\.)[^.]+\.(?:[^.]*\.){%d,}[^.]+(?=\.))(?:.*\1){%d,}` the expression is built on the following rules: the values in the string are delimited with dots. the pattern must begin with a non-dot character immediately preceded by a dot; this is implemented through a lookbehind: `(?<=\.)`. the pattern must end with a non-dot character immediately followed by a dot; this is implemented through a lookahead: `(?=\.)`. the pattern must be at least a certain length; this is implemented by specifying a minimum repetition for a non-capturing group: `(?:[^.]*\.){%d,}`. the pattern must recur at least a certain number of times; this is implemented by a non-capturing group that includes a backreference: `(?:.*\1){%d,}`. the find_patterns() function in figure 5 takes as its first argument the string dictionary returned from csv_to_string_dict() function. the other arguments are the minimum length of the sequence and the minimum number of occurrences. the argument for minimum length is reduced by two to account for the first and last interval, which are already represented in the regular expression; the minimum occurrences argument is lessened by one to account for the initial occurrence of the pattern. figure 5. searching for patterns in strings with regular expressions using find_patterns(). i saved each returned list of patterns to a unique text file so i could retrieve the patterns without redoing the entire search process. some of the largest score files were skipped over because the regex searches took too long. i used the func-timeout package to set a five-minute time limit when i ran the find_patterns() function. i did this because a test pattern search through one of the largest files was still unfinished after three hours, though most searches finished well under the time limit. in the end, 313 of the 346 csv files were successfully converted to strings and searched (figure 1, steps 5–9). creating a database using sqlite3 after finding the patterns, i created a simple sqlite3 database to store the patterns and their corresponding works (figure 1, steps 9–11). the structure of the database is outlined in figure 6. i created three tables: one for musical works, one for patterns, and one bridge table. there are fields for work and pattern identifiers, the pattern strings, and the work title and composer. figure 6. pythagoras database logical model. i wrote the function add_to_db() in figure 7 to read in the text files that held the patterns returned by the find_patterns() function and add data to the three tables. when i had originally converted the capella files to musicxml files, i made sure that each file name followed the format: composer_name-work_title.file_extension. splitting the filename returned values for the composer and work title. based on the work’s title, i searched for its id, adding any missing works to the database. i iterated over the patterns in each text file, adding to the database any patterns that hadn’t already been added and retrieving their ids. i added a row in the bridge table for each work id and pattern id combination. figure 7. inserting values into the database using add_to_db() the database currently holds 346 works from 6 composers containing over 380,000 patterns. i wrote a short function query_db() in figure 8 which takes an sql query string and returns an iterable of rows from the query results. the sample query returns the pattern and associated work titles for any patterns that are at least 10 characters long and occur in 5 or more works. figure 8. querying the database using query_db() with a sample query. visualizing patterns using matplotlib and musescore the patterns stored in the database are sequences of intervals; they represent the relative distances between note steps, but not the steps themselves. this was necessary for finding patterns at different pitches or in different key signatures, but it meant that the interval pattern strings must be converted back to sequences of steps in order to visualize the original pattern. i included this conversion in the plot_pattern() function in figure 9. figure 9. plotting an interval string with matplotlib using plot_pattern(). first, i split the interval strings on the dot separators, returning a list of strings. i converted the strings to intervals using the map() method. i created a new list to hold steps and added an initial step of 0, since the first interval represents the relative distance of the second step from the first. for each interval in the interval list, i added its value to the previous step value and appended it to the step list. so, for example, the string ‘1.0.-1.1.2.-7’ was converted the interval list [1,0,-1,1,2,-7] which was used to build the step list [0,1,1,0,1,3,-4] according to the scheme current step + interval = next step: initial == 0 0 + 1 == 1 1 + 0 == 1 1 + (-1) == 0 1 + 1 == 1 1 + 2 == 3 3 + (-7) == -4 i plotted the list of steps using matplotlib, and labeled the steps with provisional pitches based on step 0 representing the pitch c (figure 10). figure 10. matplotlib visualization of an interval string, output of code in figure 9. i used the music21 toolkit developed by a team at the massachusetts institute of technology to plot the list of steps to a musical staff. figure 11 shows the output of passing the steps from the example pattern in figure 9 to music21’s converter.parse() method. although this visualization does not necessarily capture the pitches or durations of the original notes, it displays the pattern in a format that is easily recognizable to a musical audience. figure 11. musescore visualization in jupyter notebook using the same pattern as figure 10. challenges my first major challenge was collecting digitized scores, since optical music recognition software is not yet as functional as i had hoped. i began with the intent of scanning individual scores and exporting them as musicxml files but found the recognition to be prohibitively inaccurate. some programs were more accurate than others, but none were effective enough to be considered for this project. a second challenge was making sure the desired matches were being found by my regular expression searches. i tested simple strings like the pitches for the melody of twinkle, twinkle, little star before working with longer strings generated by my csv_to_strings() function. one of the issues i had to solve was making sure both leftand right-anchored matches were returned. the overlapping functionality of the alternate regex library only returned right-anchored matches. for example, a regex search to match four digits in the string ’123456’ would return ’123456’, ’23456’ and ’3456’, but not ’1234’, ’12345’ or ’2345’. to solve this issue, i iterated through the initial list of returned patterns and searched one-by-one through the strings to find any additional matches. those extra matches were then appended to the list of patterns, as shown in figure 5. another challenge i faced throughout the project was limited computing power. the major steps – converting xml files to csv files, searching for patterns, and adding patterns to the database – took many hours to run. results this project is still in progress, but my early findings from database queries are promising. i wrote a query for patterns at least 20 characters long and appearing in at least 5 works, and the results included the following: pattern: -1.-2.-2.-2.-1.-2.-2 composers: beethoven,schubert,handel,mozart appears in: 31 works this result is confirmation that the functions i’ve written are able to successfully parse and convert scores, find and store patterns of intervals, and show at least some relationship between works and composers based on those patterns. this particular sequence, when converted back from intervals to steps, is a descending major scale: 12-11-9-7-5-4-2-0, or c-b-a-g-f-e-d-c in pitches. a next step is to distinguish between commonly used patterns like scales and more unique patterns that would possibly represent a more interesting link between certain works or composers. further work the work presented in this paper constitutes an introductory approach to the larger goal of using technology to gain new insights into the arts, specifically music. there are several opportunities to expand on the work that i’ve described here. i will continue to add works and composers to my database. the composers and works included in the project so far represent only a narrow slice of music history. i am particularly interested to see how musical themes have been borrowed across distances in time and place. much of the future work has to do with accounting for the complexities of musical patterns. i have, so far, only considered patterns made up of specific intervals, but that is a most basic type of musical pattern. a pattern could be inverted, reversed, shortened, lengthened, or any combination of those variations. it could exchange major intervals for minor intervals, or use completely different intervals altogether and be based on rhythm instead. there is more information to include about the relationship between the pattern and its respective work. as of now, i can only say whether or not a pattern exists within the notes of the work, but not the number of times the pattern appears or the measure numbers where it began or ended. conclusion computer analysis can yield insights quickly on a large scale. it is no replacement for trained music theorists, music historians, and musicologists; but it is a useful tool for identifying musical characteristics and relationships that may otherwise go unnoticed. musical patterns are complex, and their nuance and variation is difficult to account for in the type of searches outlined in this paper. nevertheless, the results of this initial attempt to show existing relationships using patterns in music score data are promising. references barnett, m. (2020). regex (version 2020.4.4). pypi. https://pypi.org/project/regex/ cuthbert, m., ariza, c., hogue, b., & oberholtzer, j.w. (2019). music21: a toolkit for computer-aided musicology (version 5.7.2). massachusetts institute of technology. http://web.mit.edu/music21/ savannah, t. (2019). func-timeout (version 4.3.5). pypi. https://pypi.org/project/func-timeout/ about the authors brandon bellanti recently graduated from simmons university’s master of library and information science program with a concentration in information science & technology. he can be reached for questions or comments at brandonbellanti@gmail.com. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – recount: revisiting the 42nd canadian federal election to evaluate the efficacy of retroactive tweet collection mission editorial committee process and structure code4lib issue 37, 2017-07-18 recount: revisiting the 42nd canadian federal election to evaluate the efficacy of retroactive tweet collection in this paper, we report the development and testing of a methodology for collecting tweets from periods beyond the twitter api’s seven-to-nine day limitation. to accomplish this, we used twitter’s advanced search feature to search for tweets from past the seven to nine day limit, and then used javascript to automatically scan the resulting webpage for tweet ids. these ids were then rehydrated (tweet metadata retrieved) using twarc. to examine the efficacy of this method for retrospective collection, we revisited the case study of the 42nd canadian federal election. using comparisons between the two datasets, we found that our methodology does not produce as robust results as real-time streaming, but that it might be useful as a starting point for researchers or collectors. we close by discussing the implications of these findings. by anthony t. pinter and ben goldman introduction twitter is a social media platform launched in march 2006 that allows users to post messages in 140 characters or less, and share hyperlinks, images, and video. the potential historical value of content posted to twitter was recognized in 2010 when the library of congress announced an agreement with twitter to archive and make accessible every tweet. since that time, however, the library has been unable to meet its pledge of making all tweets available. in recent years, researchers and archivists have begun developing new strategies for curating topical datasets of tweets around culturally significant events as they happen. using twitter’s api, such researchers can harvest tweets going back seven to nine days. datasets of this variety are composed of thousands of ordinary users, offering researchers access to representative samples of public discourse by and between individuals during these significant events. however, the relative newness of tools like twarc and social feed manager means that researchers and librarians have only been able to develop such collections in the past five years, leaving a gap between when twitter was created (march 2006) and when collection tools became available. this gap presents collectors with a two-fold problem – we know that people were tweeting about events (and frequently, tweeting a lot), but conventional methods use twitter’s api for collecting, which limit collections to the past seven to nine days. if a collection is started while the event is not in this near past, or if the subject of interest lies outside of the near past, there is not a readily available, open-source method for accessing these tweets. as well, there can be data loss over time as twitter users delete tweets, whole accounts, or alter the privacy settings on their accounts. at penn state university libraries, these concerns were brought to the forefront when we considered the feasibility of retrospectively capturing tweets related to the jerry sandusky child abuse scandal which dominated the national u.s. news in late 2011. from the initial public revelation of the scandal in november 2011 to the death of penn state’s football coach, joe paterno, in january 2012, the twitter reaction was likely considerable, and we wondered how comprehensively we would be able to capture the dialogue found on this platform years after the tweets were first posted. in this paper, we offer the results of an investigation to determine how much twitter data can be retroactively captured using a potential workaround. by using twitter’s advanced search feature and in-browser javascript, we found it was possible to collect large numbers of tweets from beyond the seven-to-nine day limit imposed by twitter’s api. we outline our methodology and then test its efficacy by collecting tweets from the 42nd canadian federal election and comparing them to a dataset of tweets that was generated in real-time during that election. we close by discussing its limitations and the implications that these limitations have on our ability to collect from twitter’s far past. background in early 2016, nick ruest and ian milligan published “an open-source strategy for documenting events: the case study of the 42nd canadian federal election on twitter”, documenting a strategy for collecting tweets about a specific event via hashtag streams and searches. the researchers turned to open-source software, twarc, developed by ed summers, to conduct their streams and searches. over the course of 102 days, the researchers collected 3,918,932 tweets that used the hashtag, #elxn42. #elxn42 is a reference to the 42nd canadian federal election, which occurred in late 2015 and resulted in justin trudeau being elected prime minister of canada. using this generated dataset, ruest and milligan demonstrated various analytic tools that were included with twarc, supported by a separate tool called twarc-report, or that were available via command-line tools. this included outputs such as timelines, user-level analytics, hashtag-level analytics, and js visualizations of the data. ruest and milligan’s published dataset offers a concrete benchmark against which we can test the effectiveness of a method for collecting tweets that are weeks, months, or even years old. determining the efficacy of this methodology is important, because as ruest and milligan note, “once an event has happened, if a small window of time has passed – 7 to 9 days – the tweets become largely inaccessible on a large scale without considerable monetary resources.” offering a work-around to this time limitation will allow broad research efforts to be taken to examine twitter during significant events that fall outside of the near (7 to 9 day) past. methodology our methodology for collecting tweets beyond the seven to nine day range was built upon the methods employed by the trump twitter archive, which aimed to collect, organize, and make accessible all of the tweets sent by donald j. trump from 2009 to present. the major change we made was to alter the amount of information collected, as we only needed the tweet ids in order to hydrate using twarc. following, we describe the process and highlight the changes we made to the original collection strategy. the trump twitter archive was built using twitter’s advanced search feature to get the date codes in the url, and then manually changing the dates searched in the url. for example, (author emphasis on the relevant portions of the url): https://twitter.com/search?q=%23trump%20since%3a2016-12-19%20until%3a2016-12-21&src=typd the first highlighted portion is the search term. in our instance, we used the search term “#elxn42” to capture tweets containing that hashtag. the second and third highlighted portions set the beginning and end dates of the search, respectively. thus, our search url for the hashtag #elxn42 from september 2015 to november 2015 becomes: https://twitter.com/search?q=%23#elxn42%20since%3a2015-09-05%20until%3a2015-11-05&src=typd however, in our case, the size of the dataset prevents us from using just one url that spans the entire date range. to get around this issue, we broke the original date range into 102 individual days, pulling one search from each day of the original timeline. following the trump archive methodology, for each search we used the developer’s console feature to run an auto-scrolling javascript code, which went through all of the results of each hashtag search: setinterval(function(){ scrollto(0, document.body.scrollheight) }, 2500) the developer’s console feature can be found under the “develop” tab on most browsers. this console allows you to enter javascript code to run on the webpage that is currently open. in our case, we used the code described above to autoscroll to the bottom of each page, and used the subsequently described code to collect the tweet ids for all of the tweets on each page. in figure 1, we show an example of the developer’s console. figure 1. the developer’s console (in safari) as previously stated, our methodology differed from the trump archive’s process. the trump archive used javascript code that collected the date, id, text, retweet status, favorite status, and url of each tweet, and copied it to the computer’s clipboard. we quickly realized that much of this data was useless for our methodology, because we were going to rehydrate the tweets to collect the full json output associated with each tweet. thus, we changed the code to only collect the ids of the tweets, the piece of information needed for rehydration. the original javascript to accomplish this task was: var alltweets = []; var tweetelements = document.queryselectorall('li.stream-item'); for (var i = 0; i < tweetelements.length; i++) { var el = tweetelements[i]; var text = el.queryselector('.tweet-text').textcontent; alltweets.push({ id: el.getattribute('data-item-id'), timestamp: el.queryselector('a.tweet-timestamp').getattribute('data-original-title'), text: text, link: 'https://twitter.com' + el.queryselector('div.tweet').getattribute('data-permalink-path'), is_retweet: el.queryselector('.js-retweet-text') ? true : false, retweets: el.queryselector('.js-actionretweet .profiletweet-actioncountforpresentation').textcontent, favorites: el.queryselector('.js-actionfavorite .profiletweet-actioncountforpresentation').textcontent, replies: el.queryselector('.js-actionreply .profiletweet-actioncountforpresentation').textcontent, }); } copy(alltweets); while we used the following code: var alltweets = []; var tweetelements = document.queryselectorall('li.stream-item'); for (var i = 0; i < tweetelements.length; i++) { var el = tweetelements[i]; var text = el.queryselector('.tweet-text').textcontent; alltweets.push({ id: el.getattribute('data-item-id') }); } copy(alltweets); similar to the autoscroll script, this code was entered in the developer’s pane of an internet browser (described above). after running this code in the console, the tweet ids are copied to our machine’s clipboard. we pasted this collected data into a text-file and rehydrated using twarc. then, the files were concatenated, deduplicated, and unshortened, similar to ruest and milligan’s approach. we would like to note that this is an implementation that anyone can operationalize, regardless of programming ability. to do large scale searches such as ours, it can quickly become tedious. there is a script that automates this process but requires some programming knowledge to properly implement. we chose to present the low barrier version of the methodology to appeal to wider audiences and mention this more efficient method for those possessing the technical skills to explore. results & discussion in this section, we turn to the same analytical approach that ruest and milligan utilized to examine our dataset and compare it to the original dataset. this included comparing the results of user counts, hashtag counts, and url counts between our “new” dataset and ruest and milligan’s original dataset. by utilizing the same analytical approach, we were able to determine the efficacy and usefulness of our retroactive collection methodology. we excluded the other metrics originally used by ruest and milligan (namely retweets, geographic information, and images) because we felt they did not offer as great a means of comparison as username, hashtag, and url. a larger dataset would by default include a larger number of retweets, so this number would be disproportionately skewed. the geographic information and images were excluded because not every tweet contains them, making it hard to compare between two datasets. number of tweets collected using our methodology, we collected 250,781 tweets. this represented approximately 6.4% of the original dataset collected by ruest and milligan (3,918,932 tweets), who themselves acknowledged the inherent limitations of using twitter’s api to collect data instead of commercial platforms. at face, this low rate of overlap indicates that this methodology is not robust enough to stand on its own, a conclusion that was reinforced when we attempted to rehydrate the #elxn42 dataset from the publicly accessible list of tweet ids, which resulted in a collection of 64.5% of the original tweets from ruest and milligan’s dataset. the loss of nearly 1.5 million tweets from the original data collected is likely due to tweets or accounts being deleted, or users changing their privacy settings. but the loss of an additional two million-plus tweets suggested that our methodology was missing tweets that remained accessible online. users a deeper examination of the data somewhat clarified the gaps in our methodology. the original researchers identified 318,176 unique users in their dataset, for an average of approximately twelve tweets per account. in comparison, we collected 28,157 unique users with an average of approximately nine tweets per user. this represented 8.8% of the users found in the original set. surprisingly, the top tweeters from the two datasets widely differed, with only one account shared between the two lists. this account, davidmorrison17, topped the list in the original set and was second on our list, indicating some overlap between the two datasets. tables 1 and 2. original top ten users (left) versus new top ten users (right) tweets username 1. 21,423 davidmorrison17 2. 15,527 p_wog 3. 10,812 chuddles11 4. 10,051 444_nal4b 5. 8,871 joannecangal 6. 8,346 littleshasta 7. 8,316 madeincanada56 8. 8,114 lucmatte9 9. 7,360 frazzling 10. 7,019 stopharpertoday tweets username 1. 2,448 bergg69 2. 2,001 davidmorrison17 3. 1,704 muskokamoneybag 4. 1,490 billhillier 5. 1,344 rabbelca 6. 1,252 deepgreendesign 7. 1,194 worldwideherald 8. 1,036 alanabowker 9. 1,028 tomassorico 10. 1,019 cmusician it is promising to see a similar (12 vs. 9) tweet/user breakdown and a shared account finishing high on both lists (davidmorrison17), but the lack of further overlap merits further investigation. it was notable that p_wog does not appear in our top 10, as that account remains public and active, reinforcing the idea that deletions and privacy settings alone would not account for the considerable differences in tweets collected. hashtags there was a large amount of overlap between the hashtags present in the two datasets, both in the number of unique hashtags found and the top ten hashtags in each data set. the original dataset contained 70,112 unique hashtags, while our dataset had 21,411 unique hashtags. this meant our set had 30.5% of the hashtags found in the original set, a percentage much higher than the percentage of overlap we found in our unique users. in terms of comparing the top ten hashtags from each set, aside from the top hashtag (#elxn42, as expected), there were six other shared hashtags, including #2 – #6. the higher rate of comparative overlap between the number of unique hashtags and the top ten hashtags suggests this methodology could be useful in examining the hashtags that were being used, and using this information in tandem with other methodologies to paint a fuller picture of the social media activity at the time. tables 3 and 4. original top ten hashtags (left) versus new top ten hashtags (right) tweets hashtag 1. 3,685,885 #elxn42 2. 1,390,783 #cdnpoli 3. 164,339 #ndp 4. 139,070 #cpc 5. 129,082 #lpc 6. 89,303 #elxn2015 7. 68,387 #polcan 8. 64,718 #realchange 9. 62,282 #polqc 10. 61,700 #globedebate tweets hashtag 1. 250,781 #elxn42 2. 108,275 #cdnpoli 3. 11,649 #cpc 4. 11,044 #ndp 5. 9,833 #lpc 6. 7,255 #elxn2015 7. 4,277 #harper 8. 3,547 #onpoli 9. 3,535 #canada 10. 3,446 #realchange urls in the original dataset, there were 1,988,693 urls tweeted, representing 50.75% of the total collected tweets. in our dataset, we found 140,839 tweeted urls, or 55.34% of the total tweets. comparing unique urls, the original dataset contained 334,841 unique urls, while our set contained 100,789 unique urls or 30.4% of the original set.  in tables 5 & 6, we compare the top ten most-tweeted urls from the original set versus our “new” set. tables 5 and 6. original top ten urls (top) versus new top ten urls (bottom) tweets url 1. 11956 http://www.cbc.ca/includes/federalelection/dashboard/index.html 2. 9712 http://www.conservative.ca/ 3. 4562 http://www.votetogether.ca/ 4. 3983 http://www.cbc.ca/news/politics/macleans-debate-leaders-2015-1.3182000 5. 3926 http://www.elections.ca/scripts/vis/finded?l=e&qid=-1&pageid=20 6. 3104 http://www.elections.ca/home.aspx 7. 2812 http://www.theglobeandmail.com/try-it-now/?articleid=26875323 8. 2808 https://www.facebook.com/abu.nawaf.581/posts/10206977713713332?pnref=stor 9. 2757 http://dont-be-a-fucking-idiot.ca/ 10. 2707 https://www.mypayingads.com/index.php?ref=51826   tweets url 1. 365 http://looniepolitics.com 2. 311 http://www.votetogether.ca 3. 213 http://votecompass.cbc.ca 4. 185 http://www.macleans.ca/news/canada/vanishing-canada-why-were-all-losers-in-ottawas-war-on-data/ 5. 175 http://www.cpac.ca 6. 160 http://dont-be-a-fucking-idiot.ca/ 7. 152 http://looniepolitics.com/election_2015/ 8. 149 http://election.davidsuzuki.org 9. 119 http://bit.ly/healthycandidates 10. 113 http://www.votetogether.ca/ while there was a high percentage of shared urls between the two datasets, there was little overlap between the two top ten lists. this discrepancy likely indicates that the url metrics from datasets collected by our methodology should not be trusted on their own, but might be useful in conjunction with other sources. the opaqueness of twitter’s documentation on search and advanced search impacted our ability to fully understand the limitations of this methodology for retroactive collection, but it seems to suggest that twitter selectively filters search results based on a number of factors, including the existence of “duplicate or near duplicate content,” hashtag spamming, automated replies and bot-generated tweets, and even the use of 3rd party applications. it is also worth noting over the course of our experimentation, twitter’s advanced search was updated to remove options for filtering by factors such as positive and negative sentiment, and the inclusion of retweets. finally, twitter does acknowledge that search result algorithms display tweets in the advanced search feature in a specific manner, such as due to the popularity of the account or the number of views that specific tweet received, which would then skew datasets collected in this manner. conclusion in conclusion, we offer two principal takeaways from testing the efficacy of this work. first, this methodology might be useful to researchers or curators looking to identify research questions or build collections from twitter’s far past. while it is not nearly as powerful or comprehensive as collecting tweets in real-time or from the near past, it can still reveal trends in hashtag usage and link popularity, and it is a better alternative than not having access to data at all. additionally, datasets built from this methodology might be augmented with additional collection tools that capture other period information from online sources (news crawls, the wayback machine, etc.). the process might also be augmented in future work to be applied to other social media formats, such as facebook or tumblr, to collect posts and comments that occurred in the far past. this, in turn, offers several avenues for future work exploring how we might retroactively collect from these other social media platforms to paint a better picture of the discourse surrounding significant events from the far past. we believe this retroactive collection methodology, although imperfect, might offer a good start point for collectors and researchers to delve into past events and how the internet discussed them. second, the failings of this methodology highlight the importance of collecting in real-time around events. while we were able to build a sizeable collection of tweets from over 18 months ago, it is clear that we collected a drop in the bucket compared to what was collected in real-time during the event. thus, the results of this work highlight the importance of active collecting around societal events, and suggest that a best practice for archivists and curators is to collect in real-time as much as possible. while this will generate large amounts of data, this is preferable to losing tweets, and the meaning they convey, to the far past limitations of twitter’s api. in their original article, ruest and milligan concluded by stating: “in an era where web archiving and twitter collection can be seen as expensive luxuries, this article shows how, for a relatively small investment of computing power, bandwidth, and storage, people can create and analyze their own twitter archives.” while web archiving and twitter collection might be seen as an expensive luxury, our explorations into retroactive collection show that the expense is well worth the result – preserving a better picture of our societal reactions to events. references binkley, peter. twarc-report readme.md. https://github.com/pbinkley/twarc-report/blob/master/readme.md, last accessed 11 may 2017. brown, brendan. trump twitter archive. http://www.trumptwitterarchive.com/, last accessed 11 may 2017. brown, brendan. twitter_scraping. https://github.com/bpb27/twitter_scraping/blob/master/readme.md, last accessed 11 may 2017. ruest, nick, 2015, “#elxn42 tweets (42nd canadian federal election)”, hdl:10864/11270, scholars portal dataverse, v4 ruest, n., & milligan, i. (2016). an open-source strategy for documenting events: the case study of the 42nd canadian federal election on twitter. the code4lib journal, (32). http://journal.code4lib.org/articles/11358, last accessed 11 may 2017 the search api – twitter developers. https://dev.twitter.com/rest/public/search, last accessed 11 may 2017 summers, ed. twarc readme.md. https://github.com/docnow/twarc/blob/master/readme.md, last accessed 11 may 2017. twitter search rules and restrictions. https://support.twitter.com/articles/42646, last accessed 11 may 2017 zimmer, michael. “the twitter archive at the library of congress: challenges for information practice and information policy.” first monday 20.7 (2015). about the authors anthony t. pinter is a phd candidate in information science at the university of colorado boulder, where he studies identity in online spaces. he previously completed his b.s. and m.s. in information sciences and technology at the pennsylvania state university. ben goldman is the kalin librarian for technological innovations at penn state university libraries, where he oversees web archiving efforts. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – chrononlp: exploration and analysis of chronological textual corpora mission editorial committee process and structure code4lib issue 57, 2023-08-29 chrononlp: exploration and analysis of chronological textual corpora this article introduces chrononlp, a free and open-source web application designed to enable the application of natural language processing (nlp) techniques to textual datasets with a time-based component. this interactive python platform allows users to filter, search, explore, and visualize this data, allowing the temporal aspect to play a central role in data analysis. chrononlp makes use of several powerful nlp libraries to facilitate various text analysis techniques including topic modeling, term/tf-idf frequency evaluation, automated keyword extraction, named entity recognition and other tasks through a graphical interface without the need for coding or technical knowledge. by highlighting the temporal aspect of specific types of corpora, chrononlp provides access to methods of parsing and visualizing the data in a user-friendly format to help uncover patterns and trends in text-based materials. by erin wolfe introduction computational text analysis encompasses a range of methods and strategies employed by researchers to examine and derive meaningful insights from large amounts of text in a fast and systematic way. it has become an important tool across various fields, enabling researchers to uncover hidden information and gain deeper understandings from text-based materials. applying these methods to textual corpora with a time-based component can enable users to uncover valuable insights into ways that the content changes over time. this article introduces chrononlp [1], a free, open source web-based application designed to specifically facilitate application of natural language processing (nlp) techniques to texts of this type, by providing means for the temporal aspect to play a central role in the data analysis. background chrononlp is an interactive web-based python application designed by the author to provide the user with an interactive means of filtering, searching, exploring, and visualizing textual materials with a time-based component. this platform grew out of the author’s interest in “distant reading” of textual materials, a method of text analysis that treats text as data and employs computational approaches and analytical tools to uncover patterns, trends, and other features that may not be as easily revealed through traditional literary analysis. while there are a number of available very good open tools that do nlp actions [2], the intention of chrononlp is to provide an alternate tool for researchers to explore texts in which a time-based element can provide valuable insight into the data and data changes. specifically, chrononlp is designed to allow the user a graphical means to explore and highlight various aspects of the texts using a variety of popular powerful text analysis libraries without a need for coding or command line knowledge. the website is deployed using streamlit, an open-source python library that simplifies the creation of interactive web applications for data science and machine learning [3]. streamlit allows easy integration with any python-based library, including the various powerful nlp libraries used by chrononlp (which will be discussed in later sections), and its real-time data processing and rendering capabilities allow users to interactively explore and visualize the underlying data in an intuitive and user-friendly graphic interface. in addition, the streamlit community cloud service can deploy web apps directly from github at no cost, with code updates automatically reflected in the user interface. chrononlp offers a variety of nlp models and data visualization techniques to interact with textual datasets, including filtering data based on user-defined criteria, robust keyword searching and visualization tools, a framework for side-by-side comparison of different slices of the data, and other actions. in addition, chrononlp provides an optional method for identifying a secondary data source component (such as a publisher or author category), which can be further used to filter and explore the data. all graphs and tables are interactive and responsive and support zooming, panning, and other features to enhance data exploration and analysis. the data sample datasets although chrononlp is designed for users to upload their own corpora, there are sample datasets included that can be used to review aspects of the tool, including a (1) collection of news articles related to covid-19 from six online newspapers published in douglas county, kansas, between january 2020 and january 2022 (‘dataset 1’), [4] and (2) a collection of 757 posts from the blog of the history of black writing (hbw) project at the university of kansas, published between 2011 and 2021 (‘dataset 2’). [5] uploading user data in addition to the text itself, chrononlp requires minimal metadata to be included [6]. the only required element is a date, ideally formatted according to the iso 8601 standard (yyyy-mm-dd) or similar. additionally, three optional fields may also be included: (1) a unique identifier for each item (e.g., filename or uri), (2) a label for each item (e.g., title) and (3) a grouping column (“source”) that will vary based on the type of data (e.g., author, genre, publisher). including a source field will allow filtering and/or visualizing the data on this facet, such as comparing keyword usage between authors, relative frequency of publication by source, or other methods of grouping the dataset. data can be added to chrononlp either as (1) structured data in a csv file or (2) a zip file of individual text files along with a csv of metadata. on upload, the user has a chance to review the data and to map the csv columns to the correct values – any extraneous columns will be ignored. preprocessing once the data is ingested and verified, the full text of each article will go through a number of preprocessing steps to facilitate the subsequent analysis of the data. first, the entirety of the text is retained in full as retrieved during the initial harvest (“full text”). this is useful for searching for phrases in the unaltered text which may otherwise be missed (e.g., “the cdc”). chrononlp implements spacy 3 [7] and textblob [8] to process each text and create a cleaned version of the text, removing punctuation and stopwords [9] and converting all tokens (i.e., individual words) to lowercase. the text is analyzed for parts of speech and named entity identification, and some additional descriptive metrics are extracted. the preprocessing tasks may take several minutes, depending on the data size. the resultant processed dataset can be downloaded, allowing users to skip this step in subsequent visits. filtering the data at any stage of the data review process, users can apply filters based on various facets to examine specific subsets of the data. these filtering options include specifying date ranges, selecting data sources, and excluding or including texts that contain specific terms. the resulting filtered dataset can be downloaded as a csv file, and all subsequent nlp tasks can be applied exclusively to this subset of the data. this ability to create on-the-fly subsets of data can help users review the texts more closely and from different angles. by comparing data from different date ranges, users can study temporal patterns and trends, seeing how language, topics, or sentiments evolve over time. filtering by source can enable users to investigate the contributions and perspectives of specific publishers, authors, or platforms. it may help identify variations in content, biases, or writing styles, providing a more comprehensive understanding of the dataset and allowing a deeper analysis of the diverse perspectives present in the texts. filtering based on specific terms or phrases allows users to focus their analysis on articles that are most relevant to their research questions. this functionality helps eliminate noise and irrelevant information, enabling researchers to zoom in on the key aspects they want to explore. overall, the filtering functionality in chrononlp empowers users to tailor their analysis, saving time and effort by narrowing down the dataset to specific subsets that are most pertinent to their research goals. it enhances the efficiency and effectiveness of the analysis process, facilitating more targeted and insightful interpretations of the textual materials at hand. functionality data overview the data overview page provides a high-level look at the dataset, beginning with a stacked bar chart depicting the different sources’ number of items or word count over time (see figure 1). figure 1. stacked bar chart showing the number of articles in dataset 1 by source. a tabular summary of the data is shown, including item count, average word count, shortest and longest entries, and the date range for each source (see figure 2). the textdescriptives python library [10] is implemented in spacy to evaluate the readability of the corpus, using a variety of recognized tests. in this data, chrononlp reports the mean flesch reading ease readability score, a metric that uses the average sentence length (in words) and the average word length (in syllables) to calculate a measure of readability. the result is a number from 0-100 in which a higher number reflects a text that is easier to read, with a lower score indicating one that is more difficult to read. figure 2. high level summary of dataset 1 sources presented in a tabular format. parts of speech and named entity recognition part-of-speech (pos) tagging is a natural language processing (nlp) task that involves labeling words within a text with their corresponding grammatical part of speech, such as noun, verb, adverb, etc. named entity recognition (ner) tagging is a common nlp task that uses statistical predictions to identify and label a variety of named and numeric entities within a given text. chrononlp employs spacy 3’s default model and pipeline for pos and ner tagging [11], which evaluates each word in its context and position within a sentence to automatically assign these and other linguistic labels (see figures 3 and 4). figure 3. sample part-of-speech evaluation of a subset of dataset 2, including only entries that contain the word “novel.” figure 4. sample named entity recognition from a subset of dataset 2, including only entries that contain the word “novel.” term frequency and keyword searching one of the primary goals of chrononlp is to enable the visualization of word occurrences and frequency changes over time. exploring term specific data is facilitated in two ways: (1) a user-instigated keyword search and (2) various automated keyword extraction methods. analyzing word occurrences and frequency changes over time provides valuable insights into how language usage, trends, and topics evolve. it allows researchers and users to track the rise or decline of specific terms, identify emerging or fading concepts, and understand linguistic shifts within a given context. keyword search the chrononlp interface allows the user to enter one or more search terms and view the usage frequency fluctuation in a variety of ways. the user can search by exact match or use wildcards to find variations (see table 1). wildcard results can be returned as either a single combined result of all matches or as individual results for each match. specific terms can be also excluded from the results. search type example returns single term test all matches for exact match – “test” phrase test case all matches for “test case” wildcard test* a single count for all matches beginning with “test” (e.g., “test”, “testing”) wildcard, separated ^*test individual counts for each match ending with “test” (e.g., “test”, “detest”) combination test*, ^*virus* (1) a single count for all matches beginning with “test”, and (2) individual counts for all matches containing the string “virus” table 1. examples of keyword searching a count of all search terms is returned, which can be especially helpful when using wildcard searches (see figure 5). figure 5. summary of search results in dataset 1 for the search terms “test” (with no wildcard) and “mask*” (with wildcard). the results are then visualized in a few different ways. first, a line graph showing a simple raw usage count of the term(s) over time, or as a normalized count which is weighted against the percentage of the number of items in that time frame in the dataset, to show the relative frequency (see figure 6). data is graphed both by individual sources and as an aggregate, providing a comprehensive view of the textual patterns and trends. the results are also presented in a tabular format that can be downloaded as a csv file. figure 6. graphs depicting the results for the search “test” and “mask*” in dataset 1. clockwise from top left: raw count of all uses, normalized count of all uses, normalized usage by source, raw count of usage by source search terms are also presented as a keyword-in-context table, showing all results along with n terms on the left and right, allowing the user to gain additional context of the words as used in the text (see figure 7). separate tables calculate the 20 most frequent terms directly to the left and right of the keyword throughout the dataset, along with word clouds of the 200 most frequent terms (see figure 8). figure 7. sample keyword-in-context results from the search “test” and “mask*” in dataset 1. figure 8. most frequently occurring terms to the left and right of search results for the search “test” and “mask*” in dataset 1. if more than one keyword is searched, the user can select to view the combined results or individual results for each term (see figure 9). figure 9. chrononlp interface depicting the available options for reviewing the results for the search “test” and “^mask*” in dataset 1. note the carat and asterisk usage in “mask”. automated keywords in addition, chrononlp goes beyond user-instigated searching, by providing an interface to explore computationally derived keywords. the chrononlp interface prompts the user to select a date range, one or more sources, number of n-grams [12] (ranging from 1-3), and the preferred method of analysis. in order to reduce noise from common or less pertinent terms, the user can also input a custom list of terms to omit from the results. three methods of automated keyword analysis are available for the user to highlight different aspects of the text. all methods are applied to the cleaned text to remove very common words (such as “the”, “and”, “for”) from the results. for each of these inquiries, the user can see a word cloud composed of the top 200 terms, a list of the top ten terms with frequency and/or weight, and a graph showing the top five terms’ frequency over time. two inquiries can be run and displayed side-by-side to allow the user to easily compare two sets of results, such as the most frequent terms from two different time periods or sources. term frequency is the most basic and uses nltk’s freqdist module to identify the most commonly used n-grams in a given set of articles. tf-idf (term frequency-inverse document frequency) analysis is an algorithmic approach to text analysis that highlights potentially significant terms that occur less frequently than others in relation to the entirety of the texts. chrononlp employs the scikit-learn library to perform tf-idf analysis on the selected article set. [13] topicrank analysis is a dimensionally-based analysis performed in spacy using the pytextrank library that groups similar terms together and provides a count of each of these topics. table 2. sample results from automated keyword extraction for dataset 2, including most frequent terms (n=1 and 2), tf-idf terms (n=1 and 2), and topicrank topics and a sample of terms related to each topic. topic modeling topic modeling is an algorithmic process that evaluates a set of documents to computationally extract patterns and structures from textual data to identify topics contained within. a statistical model analyzes and clusters the words from the document set based on their relationship. using these clusters, the model generates a set of keywords for each topic, along with a probabilistic rate of occurrence for each keyword within that topic. individual documents are assigned a relative proportion of each topic (to total 100%) based on the content, which enables the representation of the documents in terms of their underlying topics. latent dirichlet allocation (lda) is a widely used method of topic modeling introduced in 2003 that is commonly used for nlp tasks [14]. lda is an unsupervised learning method (i.e., it does not require a manually created set of training data) that works by analyzing the words in the documents, applying a probabilistic framework for capturing the relationships between words and topics, and grouping the words into a specific number of topics. each document is then assigned to one or more of these topics based on its content, which can allow users to uncover the latent thematic structure in their textual data. chrononlp employs gensim [15], an open-source library that uses modern statistical machine learning for nlp tasks, to perform lda on a selected subset of the articles and to select the number of topics to generate. to determine the optimal number of topics for a given set of documents, the user can use chrononlp to review two language model evaluation metrics: perplexity and coherence. perplexity is a measure of how well a language model predicts unseen data, where a lower value indicates better performance. while this is a good metric for machine learning tasks, it may not be the most suitable measure for creating a human interpretable model. coherence is a measure of how well the topics within a model make semantic sense. a higher coherence score suggests that the topics are more interpretable and clear, and it is a more appropriate metric to use when creating a model that can be visually reviewed and understood by humans. these metrics help evaluate how well the topics align with the documents and how understandable they are. by comparing these results, the user can select a number of topics for the model (between 5 and 15) to achieve the best results for their specific dataset (see figure 10). figure 10. results of coherence and perplexity evaluation for a subset of dataset 1, including only entries that contain the word “community.” based on the coherence scores (the darker line), in this example, the optimal number of topics would likely be 5, 6, or 7. results are plotted over time, with larger bubbles representing a higher presence of the topic. results can be plotted to show the absolute article count or relative frequency (i.e in relation to other topics present at that time) of a given topic (see figure 11). figure 11. results of topic model graph using 7 topics for a subset of dataset 1, including only entries that contain the word “community,” showing both the absolute values (left) and the relative values (right). the user can then review more specific details about each of the topics. as noted above, lda is a topic modeling technique that clusters words into topics. lda assigns documents to relevant topics based on the words they contain, assigning each topic a weight between 0.0 and 1.0. the sum of weights for all topics in a document is equal to 1.0, representing the distribution of topics within that document. in the chrononlp interface, the user can see (a) a list of the top 10 keywords and (b) a word cloud based on the top 200 keywords for a given topic, as well as the number of texts included in that topic group (see figure 12). figure 12. sample of topics from a model using 7 topics for a subset of dataset 1, including only entries that contain the word “community.” limitations and future work this project could be expanded to allow exploration of a variety of different collections that include texts with a time-based component, including literature, twitter datasets, oral histories, or other content. some testing has been done using some of these materials, including the plays of shakespeare and historical speeches by u.s. presidents, but more work remains to facilitate wider application. the chrononlp interface does rely on a large number of external libraries and platforms. it will require periodically updating and migrating to newer versions of libraries or exploring alternative libraries to ensure the project remains up-to-date and compatible with the latest technologies. at this time, the tool has been tested with relatively small datasets, and the efficiency and reliability of working with larger datasets is not known at this time. future work will involve scalability, performance, and efficiency improvements, through code optimization, parallelization, hardware accelerators, and other means. in addition, the platform is currently limited to datasets in english. future work could expand this through the inclusion of additional libraries and models. conclusion this article has presented chrononlp, a web-based application that facilitates the application of natural language processing (nlp) techniques to texts with a temporal component. it offers a range of functionalities that enable users to filter, search, explore, and visualize textual materials in an interactive and user-friendly manner. integration with streamlit enhances the accessibility and real-time data processing capabilities, empowering researchers to tailor their analysis and focus on specific subsets of the dataset that are most relevant to their research goals. it is the author’s hope that this tool may help researchers to uncover hidden insights, track language usage over time, and gain a deeper understanding of the textual materials at hand. by eliminating the need for coding or command line knowledge, chrononlp makes text analysis accessible to a broader range of users and enhances the efficiency and effectiveness of the analysis process. data availability all code and underlying data described in this article is hosted in the author’s public github account: https://github.com/ewolfe1/chrononlp notes [1] wolfe, e. (2023). chrononlp. https://chrononlp.streamlit.app/ [2] some examples of other open nlp tools include: sinclair, s., & rockwell, g. (2016). voyant tools. http://libvoyant.unm.edu/docs/#!/guide/about; morgan, e., & abeysinghe, e., et al. (2019). the distant reader – tool for reading. pearc ’19, july 28-august 1, 2019, chicago, il, usa. https://doi.org/10.1145/3332186.3333260; anthony, l. (2013). developing antconc for a new generation of corpus linguists. proceedings of the corpus linguistics conference (cl 2013), july 22-26, 2013. lancaster university, uk, pp. 14-16. https://www.laurenceanthony.net/research/20130722_26_cl_2013/cl_2013_paper_final.pdf [3] streamlit inc. (2023). streamlit documentation. [accessed june 15, 2023] https://docs.streamlit.io/ [4] a collection of 3,753 news articles from six online publications in douglas county, kansas published between january 28, 2020 and january 31, 2022. articles were collected from the public web sites of three city publications (the lawrence journal-world, the lawrence times, and the eudora times) and three student university publications (the university daily kansan, the baker orange, and the indian leader). articles were identified as part of a university of kansas libraries web archiving project identifying online content relating to covid-19. [5] a collection of 757 posts from the blog of the history of black writing (hbw) project at the university of kansas, published between 2011 and 2021. learn more about the hbw at https://hbw.ku.edu/. [6] documentation for the user on data preparation is included on the site: https://chrononlp.streamlit.app/upload_dataset [7] honnibal, m., montani, i., van landeghem, s., & boyd, a. (2020). spacy: industrial-strength natural language processing in python. http://doi.org/10.5281/zenodo.1212303 [8] loriah, s. (2023). textblob: simplified text processing. https://textblob.readthedocs.io/en/dev/ [9] stopwords are very common words that generally do not contribute to the understanding of the text, such as “a”, “the”, “would”, etc. it is a common practice in nlp tasks to remove stopwords and punctuation as part of the pre-processing of the texts to help reduce noise in the results. [10] hansen, l., & enevoldsen, k. (2023). textdescriptives: a python package for calculating a large variety of statistics from text. arxiv preprint arxiv:2301.02057. [11] spacy provides four english language pre-trained pipelines (simply, a sequence of processing steps or algorithms that can be applied to text data) for performing basic nlp tasks. each of these varies in size, processing speed, required computational resources, and suitability for a given nlp action. chrononlp uses the smallest of these four, “en_core_web_sm”, as it is the fastest model and well-suited for the available streamlit resources, and it performs as well as the other larger models for the basic tasks of chrononlp, such as pos and ner identification. more information about the models and their functionality is available in spacy’s documentation: https://spacy.io/models and https://spacy.io/models/en/. [12] n-grams are simply a sequence of n words. the use of n-grams is very common in many nlp tasks. c.f. https://machinelearningknowledge.ai/generating-unigram-bigram-trigram-and-ngrams-in-nltk/ [13] pedregosa, f., varoquaux, g., gramfort, a., et al. (2011). scikit-learn: machine learning in python. journal of machine learning research, 12, 2825-2830.. [14] c.f. kapadia, s. (2019). evaluate topic models: latent dirichlet allocation (lda). towards data science. https://towardsdatascience.com/evaluate-topic-model-in-python-latent-dirichlet-allocation-lda-7d57484bb5d0 [15] rehurek, r., & sojka, p. (2010, may 22). software framework for topic modelling with large corpora. in proceedings of the lrec 2010 workshop on new challenges for nlp frameworks (pp. 45-50). valletta, malta: elra. http://doi.org/10.13140/2.1.2393.1847. see also https://radimrehurek.com/gensim/ about the author erin wolfe is the digital initiatives librarian at the university of kansas, where he works with the libraries on a variety of tasks involving the creation, description, and organization of digital materials for access and preservation, along with instruction and consultations in tools and practices for metadata management, dataset creation, digital preservation, and related topics. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – ebooks without vendors: using open source software to create and share meaningful ebook collections mission editorial committee process and structure code4lib issue 25, 2014-07-21 ebooks without vendors: using open source software to create and share meaningful ebook collections the community cookbook project began with wondering how to take local cookbooks in the library’s collection and create a recipe database. the final website is both a recipe website and collection of ebook versions of local cookbooks. this article will discuss the use of open source software at every stage in the project, which proves that an open source publishing model is possible for any library. by matt weaver introduction as libraries consider the impact/future of ebooks in libraries, so much of that energy is spent on the actions of vendors and publishers. digitization of local content holds a lot of potential for all libraries. vendor-driven solutions are expensive and can be overkill for many types of information that libraries might want to digitize and distribute. i came across some local cookbooks on my library’s shelves that represented a wide historical range, from a 2011 book produced by the city for westlake’s centennial to one produced by a local church’s sunday school in 1906, with several others in between. i wondered how we could capture that content, strip the recipes from their container, and create a database of local recipes. when looking for open-source digitization tools and processing tools, i realized that i could not only extract the recipes from the cookbooks, but also produce ebook copies. the result is the community cookbook, a recipe website and collection of ebook versions of local cookbooks. the project is entirely open-source based: homer project, suite of open source tools for the production of searchable pdfs when building projects from tiffs. sigil, an open-source epub editor calibre, an open-source, multipurpose ebook program used for converting epub ebooks into the kindle-compatible mobi format drupal, an open-source content management system. a project like this shows that libraries of any size or budget can use open-source software for meaningful content creation. because westlake porter public library hosts its own websites and owned all of the hardware used in digitization, the community cookbook project has been accomplished without any direct costs. what follows is an explanation of how wppl developed this resource, and of the decision-making processes with regards to hardware, software, copyright and site infrastructure. let’s begin with the hardware. hardware used in this project low-end pc bleeding edge technology is not required. the pcs that i have used for processing digitized images into ebooks were machines with average processor speeds at the time of their manufacture, running windows xp. hard drives the digitization process involves creating a large number of images. more images are created as part of the production process; the net result is large project files requiring adequate hard drive space. my current project file space for local ebooks exceeds 100 gigabytes for about a dozen ebooks. ebook data should also be appropriately backed up on an external hard drive. when a pipe burst in my office this winter, releasing the contents of the library’s sprinkler system, i was fortunate that my hard drives survived. scanner flatbed scanners can be painfully slow, and are inappropriate for traditionally-bound books. many cookbooks are published in spiral bindings so they can lay open when used in the kitchen. this binding makes them ideal for flatbed scanning. around the time i started this project, the library acquired multifunction copier/printer/scanners that scanned books rapidly. on such a machine i could scan even a large cookbook in about twenty minutes. the scanners supported the tiff format. dslr camera while there are different paths to ebook freedom, the method i use requires images to be in tiff format (more about formats later). a camera that shoots in tiff format is ideal, but raw format, which is more common, can easily be converted to tiffs using the open-source image-editing program gimp, or other common commercial products. digitization rig a digitization rig is needed for scanning with a camera. rigs can be elaborate and expensive, or downright elegant in their simplicity. figure 1 is a rig that i cobbled together using a couple of bookstands that are held together with rubber bands. it got the job done. figure 1. test rig used at wppl. deciding on a copyright strategy in creating local ebook collections, copyright is a primary consideration. when dealing with works covered by copyright, the society of american archivists posits the following principles [1]: multiple legal rationales may apply to a specific project or use; holdings in archival collections should be used, not left unused because of obscure ownership status; common sense should apply. copyright issues under section 108 of copyright law, libraries cannot distribute electronically digital copies made for preservation. section 104 of copyright law prohibits the creation of an ebook version of a physical book that is still under copyright without the rightsholder’s permission. i could use neither section 108 protections nor a fair use strategy for this project. to proceed with digitization, either the work would have to be in the public domain, or the library would have to get permission from the rights holder. few of these types of cookbooks were registered with the u.s. copyright office, and hardly any feature a copyright statement, which factors into an item’s copyright status. complicating matters is the fact that often i could not locate the rightsholder. some organizations that had produced cookbooks no longer existed, and it was difficult to determine who would have held the rights. copyright and recipes according to the u.s. copyright office [2], recipes can be subject to copyright: ingredient lists are not copyrightable, but instructions may be, if they amount to expression [3]. anecdotes about a recipe, accompanying images, and any other original content in cookbooks may be copyrightable. therefore, for the purposes of the community cookbook, in order to provide ebook copies of actual books, the only legitimate approach to copyright was to digitize titles that are public domain, and to seek permissions from the organizations that produced the cookbooks. when considering this collection, i was unclear about how copyright would apply to these recipes. even cookbook publishers inc., a popular producer of fundraising cookbooks, states in its documentation that “individual recipes cannot be copyrighted” [4]. however, some of the titles that we have had access to were published by cookbook publishers and always carry a copyright notice. i emailed the company to ask if the copyright notice applied to the recipe content. a member of the company’s customer service department responded that copyright only applied to the artwork and other content provided by the company [5]. therefore, if i received permission from the producing organization to digitize a cookbook published by cookbook publishers inc., i needed to make sure that no artwork, or other publisher-provided content would end up in the ebook versions of the cookbooks. i created a template for book cover art that i use for cookbooks from cookbook publishers, inc. researching copyright status in order to research the copyright status of books that are potential candidates for digitization, i turned to several online resources: the copyright office’s online catalog, which contains copyright records from 1978 to the present. stanford university’s copyright renewal database, which contains renewal records for books published between 1923 and 1963. as renewals became automatic by statute starting in 1964, this database is a valuable resource for assessing the copyright status of works. those renewals were received between 1950 and 1992. the copyright genie, from the ala office for information technology policy and michael brewer, which guides a user in determining whether an item is in the public domain, through a series of questions about the work, and makes an evaluation of its copyright status. the user can generate a pdf with information about the work to document efforts to ascertain its copyright status. the copyright slider, also from michael brewer and the ala office for information technology policy, is an easy-to-use interface for quickly determining a work’s copyright status based on its publication date, whether it contains a copyright notice, and some other factors. documenting research it is important to document all copyright research, and that research be as detailed as possible. if someone claims to be a rightsholder of a work that you have digitized, providing documentation about the process by which you decided to use the work can help prove the legality of your actions or at least show that you employed thoughtful, professional methods in your evaluation of the content contained in your project. according to the society of american archivists, “a documented history of the search for a copyright holder should help establish that a good-faith effort was made” [6]. consent agreements when i determined that a cookbook was still covered by copyright, i would seek out the current head of the organization and ask for permission to digitize it. since none of the cookbooks that i have digitized that are still subject to copyright had a single author, i determined that permission from the current head of the organization that produced the cookbook would be an appropriate indicator of permission. the consent agreement contains language that the signer of the agreement has the authority to approve our use of the cookbook. for each work published after 1923 that i determined to be in the public domain, i thoroughly documented the research process in an excel file in case i might have to justify my use of the content, and to document a good faith effort in attempting to identify rights holders. because, through this project, the library acts as a publisher, there had to be special consideration of how to handle access to the ebook as a digital object. my process for determining how users could access the content respected the concerns of the organizations that created the cookbooks without applying technologies that would both make the project cost-prohibitive and inject technological obstacles to patron access. access control there are two components to digital rights management (drm) that need to be considered in a project like this. technological protection measures (tpm) represent the end-user technological obstacles commonly referred to as drm. there is a second component to drm, however: copyright management information (cmi) [7]. there are no tpms used in the community cookbook ebooks. the content in this system is in most cases out of print, so it wouldn’t make any sense. this is clearly spelled out in the content agreement that is signed by representatives of organizations that produced cookbooks that are included in the collection. cmi, however, is an important part of the digitization process. data that comprises this category must be included in the digitization process, or it represents a violation of section 1202 of us copyright law [7] [8]. cmi for cookbooks is preserved both in the ebooks and in fields for both recipe and ebook content types in the community cookbook website. at this stage in the project’s development, only cardholders can access the full site. one collection, a text book of domestic science, a common home economics textbook from the early 1900s, which is in the public domain, is provided to all users largely as an example of how the site works. in the future, we may consider moving more titles beyond user authentication, but this will largely be determined by relationships with contributors. ebook formats epub has been established as a standard ebook format by the international digital publishing forum. there are two versions of epub. most ebooks are currently epub2 format; but the epub3 format, the current version of the standard [9], is gaining in popularity. the most significant difference between versions is that epub3 does not use the navigation center extended (ncx) file. this file, which is used for navigation within the ebook [10], can be left in an epub ebook file system so older ereading devices can still access the book. epub3 also offers improved support for mathml [11]. mobi is a kindle-compatible format. while most ebooks for amazon’s kindle device are in amazon’s proprietary azw3 format (which is mobi-based), mobi offers the broadest kindle device and app support. [12]. the ubiquitous pdf (portable document format) is a terrible ebook format for ereaders, smart phones and small tablets. designed to capture the exact layout of printed documents, the lack of adjustable text forces users of small tablets, smart phones and ereaders to pan and zoom in order to read a document. for users reading on a computer or large tablet, the pdf can provide an acceptable reading experience. the community cookbook workflow the content from the digitized cookbooks is stored on the community cookbook website in two ways: a recipe database; and a collection of ebooks in epub, mobi and pdf formats. at the core of the recipe database is the drupal recipe module, which provides functionality comparable to commercial recipe websites. users can browse recipes by ingredient or recipe name; and download recipes in three formats that can be uploaded into recipe management tools and websites: mastercook4, recipeml, and plain text. figure 3. digitization workflow. there are three main stages in the workflow for the community cookbook project (figure 3). stage one is digitization, using either a camera or flatbed scanner. optical character recognition is performed in stage two, beginning with tiff-format images from the digitization process and ending with production of a searchable pdf. stage three uses the text layer from the pdf, often after substantial editing, as the source of recipes which are uploaded into the recipe database, and to produce an epub ebook which is later converted to a mobi ebook. figure 4. sample recipe. key drupal modules apart from ‘article’ and ‘page” content types that are part of drupal out of the box, three others are at the core of the site’s functionality: recipes — produced by the recipe module, ebooks, and organizations. the ebook content type has three file fields, one for each ebook format. the organization content type has fields for an organization’s logo, website address, and description. i felt it was important to give the organizations that produced the cookbooks a visual presence on the site. the organization information and logo appear alongside every recipe from their cookbooks (figure 5). figure 5. ebook connected to organization content type. in terms of recipe metadata, the recipe module [13] does a lot of the heavy lifting with its built-in management of ingredient and measurement data (figure 4). ingredients can be entered in a simple interface with dropdown selectors for measurements and autofill textfields for ingredients (figure 6). figure 6. recipe ingredient interface. one criticism of the module is that ingredients are not stored as a taxonomy, but rather as arrays in tables for the module, preventing their availability to the views module. recipes can be imported individually in either plain text, or mastercook4 format from a simple text box, allowing copying and pasting from other sources (figure 7). figure 7. recipe import interface. an essential module for maintaining the site is views bulk operations (vbo) [14]. through views, vbo can be used to make changes to large numbers of content nodes in minutes. because a cookbook can contain hundreds of recipes, i have set the default published status of all recipes to “unpublished”. when an entire cookbook has been entered into the site, i use vbo to publish all recipes for that cookbook en masse. content access for the cookbooks and recipes that are restricted to library cardholders, the taxonomy access control lite (tacl) [15] module allows permissions to be set on content based on individual taxonomy terms. i created a “recipe collection” taxonomy, with terms — the titles of each cookbook — applied both to the ebook and recipe content types. out of the box, drupal’s files are set as “public”. to control access to the ebook files, i changed the setting at “admin/configuration/media/file system” to “private files”. unlike public files, which are accessed directly via the web server, private files are accessed via drupal path requests [16], providing access control to those files, which the tacl module handles. because some of the organizations whose cookbooks i had arranged to digitize did not want their content shared with non-cardholders, i had to allow users to create accounts on the site, which drupal handles out of the box. however, since the library already has a site that requires user accounts, i wanted to simplify account creation and site usage. the ils authentication module [17] allows users to create an account simply by using a library card number and pin. the module overrides the core user registration/login block (figure 8). there are costs associated with this, at least for libraries using sirsidynix symphony®. one has to subscribe to web services in order to be able to generate the client id required to use the module. as my library had already subscribed to sirsi web services, this module was easily added. out of the box, drupal provides user accounts. ils authentication provides convenience for the end user. figure 8. ils authentication module overrides drupal user registration/login. usage to capture usage data on recipes, i created a view that filters based on paths in the access log. all prints or downloads of recipes use a path that begins with “recipe/export”. i use the download count module [18] to keep track of ebook download statistics. this module tracks downloads of private files, and gives the option of filtering out downloads by the site administrator. since late october 2013, the site has had more than 1,500 ebook downloads and more than 20,000 individual recipes have been downloaded or printed, content that cost the library literally nothing to acquire. future improvements mapping – i have received cookbooks that are outside the westlake, ohio, community, and are of regional interest. i have started looking at how to map cookbooks and recipes collections (figure 9). the website already uses location as a facet for recipe data, so adding a map is a natural interface figure 9. testing mapping of collections. integration with local history content – usually every recipe features the name of its source. i would like to combine recipes with images in our local history, and perhaps other sources to collocate as many historical items as possible. adding recipes to images of people and families expands our understanding of their experience because we can see what people ate, beyond traditional historical information. developing a local publishing model – based on the skills that i have developed and via the website that i have established, i would like to grow this project into a publishing service. as small organizations have a need to produce books for fundraising, the library could work with organizations to help produce ebook copies. in such a model, starting with content in electronic format eliminates the digitization and ocr parts of the workflow, reducing editing time. to help patrons produce books that can be printed, the open-source desktop publishing program scribus [19] could be used. this program can produce pdfs, but also layouts that a printer would need to produce a physical book. the library would be able to help with content creation in every stage except printing. as organizations sell out of their print run, the library could attempt to negotiate the rights to those works. the value of the experiment developing the skills to digitize, reshape and distribute content has the power to change our thinking with regard to electronic content, leading toward greater independence among libraries. by building projects/communities, we develop expertise in our communities that vendors cannot possess. libraries of any type possess intimate knowledge of their communities. the system developed in this project has low technological and cost barriers, and represents the first step in the development of an open source publishing model for libraries. a logical next step would be to provide publishing assistance for organizations, families, churches, etc., that want to publish cookbooks for fundraising purposes and facilitate design and layout. appendix open source tools used the homer project [20](lupocos, 2011) was begun as a way of developing low-cost, open digitization tools for museums, libraries, or anyone who wants to digitize books. it combines a number of free and open-source tools to make book digitization easy and affordable. any of these components can be downloaded individually. if any of the following are already installed, they must be uninstalled before installing the homer software. imagemagick (for manipulation images) jpegtran (loseless jpeg transformation) jbig2 encoder (compression tool for bi-level images) tesseract-ocr – optical character recognition engine rubyinstaller (installs the ruby programming language) hpricot (html parser) rmagick (interface between the ruby programming language and imagemagick) pdfbeads (to create searchable pdf) cmdow.exe (command-line utility used in homer) scantailor (book page-processing tool) homer.sh (bash script: command-line interface for producing a searchable pdf) the homer bash script can be used to rename and rotate the scanned images. renaming can be an important step as, to a pc, page111.jpg will appear right after page11.jpg. many of these tools operate without any direct user interaction. the homer bash script uses tesseract-ocr and pdfbeads to create a searchable pdf. the searchable text layer created by tesseract-ocr/pdfbeads is what allows the recipes to be stripped from the cookbook and uploaded individually into the community cookbook site. that text layer also can be copied into an ebook editor to create other ebook formats. some caveats installation: when launching the windows installation executable, a number of open source tools are installed automatically. as part of the installation process, the usual installation displays will emerge and will proceed with the installation script taking complete control over the process. each installation process for the components that homer installs is controlled by the script, and progresses rapidly. even if there is a pause, with a window that appears to prompt you for approval to proceed, no user action is needed. figure 2. homer warning. network profiles – because the “cmd does not support unc paths as current directories” (figure 2) the homer project cannot be installed on a pc on a network’s domain controller, using a network profile. homer cannot write the final pdf to the desktop. cmdow – antivirus software might quarantine cmdow as a hacking tool “because it can hide windows.” the cmdow project site recommends checking the checksum if you have any concerns about the file’s authenticity. troubleshooting the installation – i have installed homer on several pcs. usually, all of the components are installed without a problem. if there are problems with the installation, the project can be uninstalled by running homer-uninstall.exe. even then, some components might not have been removed. the above list of tools installed by homer can be used with the windows software uninstaller in the control panel to see if any remain. most of the components installed as part of homer require no human interaction. for instance, jbig2 allows the creation of bi-level images so the text layer created by tesseract-ocr can be combined with the scanned image by pdfbeads to create the final searchable pdf. below i discuss in greater detail the components that the user interacts with, and some other tools in addition to homer that are part of the workflow. tesseract-ocr [21] is a powerful optical character recognition engine that is installed as part of the homer package. packages have been developed for a large number of languages, from afrikaans to vietnamese. scantailor [22], which is installed as part of the homer package, processes scanned images of book pages. if images of book pages were scanned in double, they can be split. the software also deskews and despeckles text. scantailor supports right-to-left writing systems. the software produces an output directory (named “out”) that contains images, and an html file that contains the text. this output directory is dragged and dropped directly onto the homer command-line interface. ocr is then applied via tesseract-ocr, and then pdfbeads produces the final searchable pdf. sigil [23] is an epub editor that allows users with little or no xhtml experience to produce epub2-compatible ebooks. sigil is not installed as part of homer and must be downloaded separately. calibre [24] is the swiss army knife for ebooks: one can use it to manage ebook collections, as a reader, and as a powerful tool to convert ebook books into a range of formats. for the community cookbook project, i have only used calibre for dealing with books in epub, pdf and mobi formats. kindlegen [25] is a program provided by amazon for converting documents in various formats to mobi. this program is not open source, but it is free. there are restrictions on its use [26]. i haven’t used this tool as part of the community cookbook project; but having experienced some problems with mobi-format ebooks that had been converted with calibre, i am looking into it. epub validator [27] is a website powered by the open-source epubcheck system [28] to validate epub ebooks. the site is for non-commercial use only. references [1] society of american archivists (january 12, 2009). orphan works: statement of best practices. retrieved on may 5, 2014 from http://www.archivists.org/standards/owbp-v4.pdf p. 2 (back) [2] u.s. copyright office. (feb. 6, 2012). fl-122: recipes. retrieved on may 5, 2014 from http://www.copyright.gov/fls/fl122.html. (back) [3] butler, joy. (jan. 29, 2008). are recipes copyrightable? – rights clearance observations about julie & julia, deceptively delicious, and the sneaky chef. guide through the legal jungle. accessed on may 11, 2014 from http://www.guidethroughthelegaljungleblog.com/2008/01/are-recipes-cop.html. (back) [4] cookbook publishers, inc. recipe management. retrieved on 4/22/2014 from http://www.cookbookpublishers.com/wp-content/uploads/2012/05/recipemanagementtips3.pdf. (back) [5] crosby, debbie. (october 2, 2013). email. (back) [6] society of american archivists (january 12, 2009). orphan works: statement of best practices. retrieved on may 5, 2014 from http://www.archivists.org/standards/owbp-v4.pdf p. 11. (back) [7] lipinski, tomas. (may 8, 2014). using copyright and licenses to your advantage in the public library setting [workshop]. ohio library council, columbus ohio. (back) [8] u.s. copyright office. (n.d.) copyright law of the united states of america, chapter 12 section 1202. retrieved on may 11, 2014 from http://www.copyright.gov/title17/92chap12.html#1202. (back) [9] international digital publishing forum (2012). epub validator. accessed on may 5th, 2014 from http://validator.idpf.org/. (back) [10] buse, jarret w. (2014). epub from the ground up: a hands-on guide to epub 2 and epub 3. mcgraw-hill education, new york, ny. p. 130 (back) [11] buse, jarret w. (2014). epub from the ground up: a hands-on guide to epub 2 and epub 3. mcgraw-hill education, new york, ny. p. 170 (back) [12] buse, jarret w. (2014). epub from the ground up: a hands-on guide to epub 2 and epub 3. mcgraw-hill education, new york, ny. p. 149, table 6-3. (back) [13] tzoscott. (sept. 28, 2003). recipe module. retrieved on may 5, 2014 from http://drupal.org/project/recipe. (back)22 [14] https://www.drupal.org/project/views_bulk_operations. (back)23 [15] dave cohen [sic]. (march 13, 2006). taxonomy access control lite module. retrieved on may 5, 2014 from https://drupal.org/project/tac_lite. (back)24 [16] arianek (2009, dec. 1). working with files in drupal 7. drupal.org. acquired on may 11, 2014 from https://drupal.org/documentation/modules/file. (back)25 [17] jsherman. (january 26, 2012). ils authentication module. retrieved on may 5th, 2014 from https://drupal.org/project/ilsauthen. (back)26 [18] wordfallz. (feb. 12, 2007). download count module. retrieved on may 5, 2014 from https://drupal.org/project/download_count. (back)27 [19] http://www.scribus.net/canvas/scribus. (back)28 [20] lupocos. (september 15, 2011). home page: homer book scanner. retrieved on may 5, 2014 from http://bookscanner.pbworks.com/w/page/40965440/frontpage. (back)13 [21] tesseract-ocr. (n.d.). project page. retrieved on may 4, 2014 from https://code.google.com/p/tesseract-ocr/. (back)14 [22] scantailor. (n.d.). project page. retrieved on may 5, 2014 from http://scantailor.org/. (back)15 [23] https://code.google.com/p/sigil/. (back)16 [24] calibre. (n.d.) project page. retrieved on may 5, 2014 from http://calibre-ebook.com/. (back)17 [25] amazon.com. (n.d.) kindlegen. retrieved on may 11, 2014 from http://www.amazon.com/gp/feature.html?docid=1000765211. (back)18 [26] http://www.amazon.com/gp/feature.html?docid=1000599251. (back)19 [27] http://validator.idpf.org/. (back)20 [28] https://github.com/idpf/epubcheck. (back)21 about the author matt weaver is the it manager at westlake porter public library (westlake, oh). his email address is mattrweaver1@gmail.com. you can connect with him at facebook.com/mattrweaver and twitter.com/mattrweaver, and see slides about this project at slideshare.net/mattrweaver. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – reporting from the archives: better archival migration outcomes with python and the google sheets api mission editorial committee process and structure code4lib issue 46, 2019-11-05 reporting from the archives: better archival migration outcomes with python and the google sheets api columbia university libraries recently embarked on a multi-phase project to migrate nearly 4,000 records describing over 70,000 linear feet of archival material from disparate sources and formats into archivesspace. this paper discusses tools and methods brought to bear in phase 2 of this project, which required us to look closely at how to integrate a large number of legacy finding aids into the new system and merge descriptive data that had diverged in myriad ways. using python, xslt, and a widely available if underappreciated resource—the google sheets api—archival and technical library staff devised ways to efficiently report data from different sources, and present it in an accessible, user-friendly way,. responses were then fed back into automated data remediation processes to keep the migration project on track and minimize manual intervention. the scripts and processes developed proved very effective, and moreover, show promise well beyond the archivesspace migration. this paper describes the python/xslt/sheets api processes developed and how they opened a path to move beyond csv-based reporting with flexible, ad-hoc data interfaces easily adaptable to meet a variety of purposes. by david w. hodges and kevin schlottmann introduction migrations bring out the best and worst of library activities. dedicated staff have invested years of practice and skill development around specific technologies, platforms, and software; and adapted to meet novel conditions and requirements. but after a while that cms or software platform is looking long in the tooth. or maybe its open-source maintainers have moved on to other things and there hasn’t been a new release in years. or maybe each library unit has developed its own tools and workflows on separate paths. then the time comes to bite the bullet and modernize. you may find that, while there may be resources out there for migrating from platform a to b, they don’t quite fit your particular flavor of a and/or b. or they don’t take into account your special edge cases and cleverly devised workarounds. improvisation is in your future, best-laid plans notwithstanding. that’s all just to say, when it comes to tools and methods for migration, we often have to wing it. this was on our minds as columbia university libraries (cul) embarked on a multi-phase project to migrate nearly 4,000 records describing over 70,000 linear feet of archival material from disparate sources and formats into archivesspace. in the process, we committed to regularizing and remediating data while we had it in view—marc records, accessions, and encoded archival description (ead) xml files created and updated over many decades by countless staff. some problems were known, such as that collection metadata managed for years in parallel legacy systems diverged, sometimes significantly. but other issues with the data were unknown or could only be guessed. how were we to review so much data and make effective use of archivists’ time and effort? which irregularities were isolated edge cases and which were project show-stoppers? to address these concerns we looked at how to streamline the extraction and reporting of important data and devised new tools and workflows using python, xslt, and the google sheets api to: analyze the corpus, pipe custom reports of targeted data directly to shared google sheets for distributed review, and in turn use information in the sheets to guide data remediation at scale. many institutions make use of workflows involving tabular data structures and tools such as the pandas python library or open refine to automate data cleanup (see e.g., bartczak and glendon 2017). however, the manual touch points of importing and exporting data to and from spreadsheets can prove burdensome and error-prone.  we found the efficiencies gained by dispensing with transitory csv (or excel, openoffice) documents and routing data directly to and from the collaborative space of google sheets made the additional effort worthwhile. in the process, we developed a reusable tool set with wider applicability beyond the migration. in effect we found that google sheets could serve as an ad-hoc database—a versioned, cloud-hosted, reliable one at that—for python-powered projects, with a flexible web gui as well. this paper describes the driving forces motivating this work at cul, the tools and workflows created, and the work done across archival processing and technology units that helped make the migration successful. background columbia’s main archival repositories reside in the rare books and manuscripts library (rbml), avery art and architectural library, starr east asian library, and the burke theological seminary library. the largest holdings by far are in rbml, which was also the first to explore archival management software (archivists toolkit, primarily to track accessions). in phase 1 of our migration, we moved around 4,000 accessions from archivists toolkit along with a similar number of collection-level marc records from our voyager ils into archivesspace. this initial migration left us in a hybrid state, with collection-level catalog records managed in archivesspace while finding aids remained for the time being in an exist xml repository, authored manually by archivists in oxygen xml editor, and published by a durable but aging custom php-based web application developed in 2008. the scripts and workflows around finding aids had served the organization well for many years, but the time for consolidation in archivesspace had come [1]. this led us to phase 2, the migration and integration of ead finding aids into archivesspace, and the retirement of the legacy web application and associated workflows. this entailed not merely importing but rather merging of data at different levels of description in the platform. it was in this phase where the scope and scale of review and quality control (qc) called out for automated solutions and efficient reporting mechanisms. defining the universe of migration a surprisingly difficult question to answer was what exactly we were migrating in this phase. we decided early on to focus on finding aids that were already in ead 2002 xml, thus setting aside for later an assortment of ur-legacy content—scanned pdfs, static html finding aids, and such [2]. the legacy xml system, built on an exist-db repository backend and an impressive array of associated scripts and tooling, was feature-rich but lacked data integrity safeguards. it had accumulated a large assortment of duplicate or orphaned content as well as “stub” records auto-generated from marc with no component description. the exist database held in the neighborhood of 7,300 records from six library repositories—clearly far more than we intended to migrate. de-duplicating and winnowing down this corpus to an authoritative data set was not a trivial task. we compiled an inventory of all xml data and, through some judicious sorting and filtering, were able to select a subset of records that (a) had component description (i.e., a non-empty element), (b) were currently (or soon to be) published as a finding aid, and (c) had a corresponding resource record in our newly operational archivesspace. in this way, out of the thousands ead files we defined our universe of migration to roughly 1,400 records. migration strategy having selected the archival description to migrate, we examined a few different approaches to actually getting the data into archivesspace. since we had already imported collection-level records and linked them with accessions in phase i, we couldn’t simply import the ead records and overwrite collection-level information. appending component description (the dsc section in ead files) to existing resources would get us closer to the desired result, but we lacked a means to do this at scale; the harvard excel importer plugin, while excellent for single file processing, lacks (as of this writing) a batch import feature.we also had to handle divergent description that predated the migration (described in more detail below). together with our hosting vendor lyrasis, we settled on a hybrid approach: export the archivesspace resource record as ead; create a complete ead from the best elements of the exist ead and the archivesspace ead; delete the archivesspace record; import the completed hybrid ead; relink accession, assessment, external document, and locally-defined fields directly in the database. this approach took advantage of the in-house expertise at cul across collection and digital libraries departments centered on archival description, ead, python, xslt, and xquery. divergent description in the old infrastructure, when a new collection-level record was first created in voyager, a stub ead was automatically spawned. from that point forward, the ead and marc records were no longer programmatically connected. thus, as a consequence of separate cataloging workflows for marc and ead, inconsistent manual updating, and automated changes such as authority control to the marc records in the ils; individual fields of collection-level records diverged significantly in unpredictable ways over the last decade. for example, a biographical note might have been updated with a death date in the voyager record but not in the ead record or vice versa. that same biographical note might also have later been expanded considerably in the ead by a project archivist, but was not brought into the ils record due to field size limitations. similarly, an expired restriction that was found and lifted programatically in the ead record using exist might not be reflected in the ils record. the immediate issue for phase ii migration was thus to identify whether each field in every one of 1,400 collection-level records was better in the ead or in the marc record. our goal was to not lose descriptive data in the process and reconcile the differences. it was very clear that cul archivists were best positioned to do the review, but we were unsure how to do so efficiently. workflow overview for the finding aid migration and remediation, we generally worked in an extract-transform-load (etl) model, exporting all data to local xml files and performing bulk and individual cleanup operations via xslt [3]. a recurring task was to iterate over a batch of ead files and report out selected data based on xpath expressions to tabular format. a typical way to do this is to use xslt to generate a pipe-delimited text string for each file; a script (bash, python, etc.) would read directory contents and feed files one-by-one to the stylesheet, and compose the output into csv format for import into excel, openoffice, or google sheets. but what if we could skip a step and write the data directly to the spreadsheet in a user-friendly format? enter the google sheets api, a great resource sitting right there in our organization’s google apps suite, little noticed. use is relatively straightforward to anyone familiar with apis and google provides decent documentation along with sample snippets in several code languages to help users get started. after some preliminary setup and oauth authorization, any google spreadsheet with requisite permissions (i.e., with edit permissions granted to the api user via the document “share” settings) can be addressed by script functions. a sheet then can be put to use as both a light-weight, versioned data store behind a python (or other language) process as well as a versatile collaborative interface for data review and remediation. to facilitate no-frills interaction with the api we abstracted some core functions into the lightweight python module sheetfeeder that has proven very useful. its functions can be imported in python in the usual way: from sheetfeeder import datasheet # define a datasheet instance my_sheet = datasheet('1yzm1ginagfotuiaoa21pspt8tkwtt9klgq','sheet1!a:z') its datasheet class defines a target sheet (based on the stable id in the url) and provides a number of straightforward methods for interacting with a sheet, for example, reading a whole table into a list: # get some data the_data = my_sheet.getdata() the_heads = the_data[0] selectively extracting data via pattern matching : # return value of col 1 for any row where col 0 matches id the_matches_by_id = my_sheet.lookup('423451', 0, 1) # return rows where id and title match patterns my_matching_rows = my_sheet.matchingrows([['id', '^123*'], ['title', '.*papers.*']]) or posting new data to sheet: # clear the sheet my_sheet.clear() # append new data my_data = [['a','b','c'],['d','e','f']] my_sheet.appenddata(my_data) the sheetfeeder module provides pythonic shorthand code that was very useful for our project , but readers are also encouraged to explore the google sheets api for themselves and try coding against it using whatever framework is most convenient. with our tooling in place, our new workflow thus took shape: ead data extracted from two sources (legacy and phase-1 migrated archivesspace) was parsed and pushed to a shared google sheet via python and the google sheets api using sheetfeeder; a user-friendly presentation allowed archivists to perform qc review; the qc results were read by another python process and fed into an xslt pipeline for merging and cleanup; and finally, the results were imported back into archivesspace and qc’d again there. figure 1.workflow diagram. reading ead data into a sheet although its verbose syntax and ponderous matching mechanism can be daunting, xslt is a great way to quickly and precisely grab a selected set of information from well-formed ead xml files. we found that, since advanced xslt 2.0 functions were not necessary for the extraction part of the process, we could use the native python lxml library to apply xpath 1.0 expressions and forego calls to an external saxon processor. namespace support can be idiosyncratic with lxml, so the xpath expressions were more verbose than we would have liked, but still easily comprehensible, as is shown in a list of query strings in ead_process.py: # default namespace for cul eads ns = {"ead": "urn:isbn:1-931666-22-9"} [...] # dict of elements and their xpath. asqs = { "bibid": "ead:archdesc/ead:did/ead:unitid[1]/text()", "repo": "ead:eadheader/ead:eadid[1]/@mainagencycode", "title": "ead:archdesc/ead:did/ead:unittitle[1]/text()", "status" : "ead:eadheader/@findaidstatus", "revisiondesc": "ead:eadheader/ead:revisiondesc", "altformavail": "ead:archdesc/ead:altformavail", "accruals": "ead:archdesc/ead:accruals", "accessrestrict": "ead:archdesc/ead:accessrestrict", "userestrict": "ead:archdesc/ead:userestrict", "acqinfo": "ead:archdesc/ead:acqinfo", "arrangement": "ead:archdesc/ead:arrangement", "bibliography": "ead:archdesc/ead:bibliography", "bioghist": "ead:archdesc/ead:bioghist", [...] } the script defines two file sources for each record: an ead file of the legacy finding aid (exported from exist) and a corresponding ead file exported from archivesspace. an obvious prerequisite was that there be two corresponding ead files for each record (identified by the ils’s unique bibliographic number, the bib id for short) [4]. since the data structure differed slightly in archivesspace-flavored ead than what we had used for years in our legacy system, xpath expressions were slightly different and needed to be stored in two parallel dictionaries. looping through each batch of files (first legacy, then archivesspace export), the script parses and extracts data via the xpath expressions, composes them into a list, and in turn compiles each list as a row in a nested list. once done iterating through the batch, the resulting data object is posted to the sheets api via sheetfeeder to populate the target tabs (one for legacy, one for archivesspace) in the spreadsheet. the result is two enormous tables of very unreadable data. however, this is where formulas in the spreadsheet take over! figure 2. raw import of ead elements into columns. data interface the desired workflow was for archivists to compare two versions of a single element (e.g., scopecontent, userestrict, etc.) for a record and determine which is authoritative (or more accurate, or just better). this was a binary choice—further textual refinement could happen later. to make a reasonable interface for this task, each element was broken out into a separate tab. for a given element (e.g., relatedmaterial), query() functions pulled in single columns from each of the large data tables and place them side by side for review. a column allowed archivists to indicate which to migrate for each record. work was distributed to a number of staff to work concurrently on different tabs and record sets according to their domain expertise. queries could be added as comments for review and resolution. figure 3. each element is filtered to a separate tab, where archival staff can review two versions side-by-side and determine which to migrate. “migrate” added in the notes column indicates that the legacy version (on left) should be migrated in the merge. data collation: the migration grid the archivists completed the review efficiently by working concurrently in the same document. each tab (24 in all), representing a collection-level element, now had a column indicating which version to migrate for each record. another tab called “migrate-grid” assembled the information from the 24 tabs into a grid, with each element represented by a column. the cells were populated by formula with boolean values based on the information added by archivists in the respective tabs. the end result was a layout of one row per record, with a series of boolean values indicating whether or not to overwrite the archivesspace values with those of the legacy data. figure 4. a migration grid collates selections from the various element views into a single row for ingest into the merge script. round trip back to python the next step was to merge the two versions of each record, grafting the component (dsc) section from the legacy data onto the collection-level data coming out of archivesspace (and by extension, the ils), and selectively replacing elements per archivists’ wishes as well. for this, another python script, ead_merge.py, needed to read the data from the google sheet and use it to inform the merge process. for this to work, the script needed to send the boolean values from the spreadsheet as parameters when executing the xslt transformation. again using the sheets api and sheetfeeder, ead_merge.py read the migrate-grid tab into an array of rows, including the heads for use in identifying element names. the values from each row were parsed out into a series of parameters with boolean values to be passed to the xslt pipeline executed in saxon [5], like so: m_userestrict=y m_arrangement=y m_scopecontent=y [...] the first stylesheet in the pipeline (ead_merge.xsl) prepared the merge by saving the requested elements (along with the dsc section) in a tree fragment variable and interpolating them into the appropriate locations in the result tree. <xsl:variable name="add_ins"> <xsl:if test="$m_revisiondesc = 'y'"> <xsl:copy-of select="//eadheader/revisiondesc"/> </xsl:if> <xsl:if test="$m_bioghist = 'y'"> <xsl:copy-of select="//archdesc/bioghist"/> </xsl:if> <xsl:if test="$m_scopecontent = 'y'"> <xsl:copy-of select="//archdesc/scopecontent"/> </xsl:if> <xsl:if test="$m_relatedmaterial = 'y'"> <xsl:copy-of select="//archdesc/relatedmaterial"/> </xsl:if> [...] </xsl:variable> on the xslt side, this task of merging data in various configurations in the two corresponding source trees proved complicated. the stylesheet had to test for a variety of scenarios where an identified element (e.g., a bioghist note) may replace one or more counterpart elements or may have no counterpart to replace and is inserted as a new element. the output was passed to two additional clean-up stylesheets performing a large number of global remediation actions. the end result was a valid, house-standard-compliant ead that included (a) collection-level data as known to the ils, (b) selected collection-level notes and other elements identified by archivists and brought over from the legacy ead, and (c) all component description from the legacy finding aid. figure 5. elements from two source trees are merged in a result ead document ready to import back into archivesspace. loading back into archivesspace the last step of the migration was to re-import the now-complete ead files into archivesspace. we realized when evaluating migration options that using an etl approach that relied on deleting and re-importing ead, rather than editing in place, would require extra work from our host vendor lyrasis to restore lost links to other archivesspace data structures [6]. however, careful analysis of existing links and database-level scripting allowed lyrasis to quickly relink accessions, assessments, and other data structures. over two months, we performed three rounds of data load and qc. 99% of our ead files imported successfully in each load, thanks to the extensive data cleanup. errors found in the qc process were mostly minor and due to either quirks in the archivesspace-ead importer, such as silently dropping uniform titles in controlled vocabularies, or unusual cul ead constructs. conclusion and further applications the archivesspace ead migration at cul was a complex project that forced us to countenance data variability and think through the implications of various strategies at some length. the tool chain discussed in this paper is best suited for scenarios where there are intermediate programming skills and a need to consolidate and accelerate reporting or qc processes on the cheap. institutions that already make heavy use of google docs and xml workflows, such as many academic libraries (colin et al. 2019), will find this a useful way to synergize these competencies. libraries that encourage staff to learn by doing (e.g., writing python scripts for one-off tasks like reporting on an ead collection) and to explore scripted solutions on a case-by-case basis may find this approach hits a sweet spot as well. building to the audience: staff ux matters too the visual design of internal reports and data interfaces is often an afterthought at best (and nonexistent for csv exports). but the principles of user experience are just as applicable, and anything that can be done to lower the cognitive load required to navigate and use an interface means users can more readily focus attention on the task at hand [7]. advantages of using google sheets for distributed data review and remediation are myriad, starting with the ubiquity and (likely) familiarity of the interface, along with ease of customization. being able to work concurrently in one document allowed our project to move forward quickly and complete dozens of person-hours of work in a few days. use of filtered views, formulas, and conditional formatting let us tailor the interface to our users and purpose. programmatic use of archivist-generated judgements on which field was correct to build the combined ead file meant we avoided the error-prone copy-and-paste method of data transfer. the automated population of the google sheet from the data sources meant that it was very easy to refresh data on demand and eliminated risk of version confusion that often emerges whenever csvs or other files are passed back and forth. most importantly, this approach allowed archivists to focus on the descriptive data in a familiar web-based interface, centering their skills and experience on the content question rather than wrestling with xslt or struggling with version control of multiple spreadsheets. while we made only modest attempts to innovate reporting interfaces for this particular project, we think this approach shows potential to iteratively improve report ux design at minimal cost, with potential to boost productivity, reliability, and general satisfaction. going “beyond the csv” in a collaborative reporting environment with high customizability thus presents advantages to institutions wishing to innovate around qc methods and practices. extension and limitations the python–google-sheets integration outlined here is easily adapted to other projects and reporting needs and has proved useful at cul for data import qc and reporting accessions and other data from the archivesspace api [8]. with the addition of some modest query and operator functions including the ability to select rows based on regular-expression matching, the toolset takes on broader applicability, and in effect enables a google sheet to serve as a light-weight, versioned, cloud-hosted database for any python project. not a fan of python? the sheets api offers good documentation for ruby, php, java, and other frameworks. in this way, we see this type of integration as holding potential for library projects at institutions where google apps is already the go-to collaborative platform and where spinning up reporting interfaces quickly and repeatedly is desired. it would however not be ideal for large-scale data manipulation and storage, and production applications would be better served with a high-availability mysql or other database if performance is of high importance. if complicated analysis and manipulation of tabular data sets is a requirement, users may do well to look at the pandas python library and the many tools built around it. acknowledgements the authors would like to thank their colleagues in the rare book and manuscript library, avery architectural library and archives, digital library and scholarly technologies, digital project management, and library information technologies, and at our host lyrasis, for their work on the migration described here. about the authors david w. hodges is special collections analyst in the digital collections and preservation services unit of the libraries, and served as project manager for the first phase of the archivesspace migration project. kevin schlottmann has been head of archives processing at columbia university’s rare book and manuscript library since 2018. among his first tasks was spearheading migration into archivesspace. previously, kevin was digital archives manager at the new york philharmonic and archival processing manager at the center for jewish history. notes [1] in the earlier workflow, new finding aids were manually seeded in exist by a digital librarian with basic data from voyager ils; archivists then edited and added description using oxygen xml editor. a php application pulled data from exist in real-time to build public-facing html content on the fly (catapano et al. 2008). the setup accrued many features over the years, including a staging/preview mode for unpublished content; custom xpath-based reporting; and cul-specific validation against a custom xml schema. it was a fine example of the custom-built solutions that archivists have built since the advent of ead 2002, in the absence of archives-specific software solutions. however, as time went by the system showed other hallmarks of complex bespoke solutions familiar to library professionals. the intricate xml and php codebase—layered with years of bug fixes and feature additions—required delicate handling for even minor changes. the broad permissiveness of ead 2002 emboldened archivists to push the limits of what finding aids could do, producing sometimes very divergent practices and and an assortment of display-oriented markup and ad-hoc workarounds to achieve desired outcomes. if that weren’t enough, the php application had been flagged by central it as a security risk. perhaps most importantly, the diy ethos it embodied was increasingly at odds with the trend toward common frameworks and open-source platforms in library technology. with advances in the archivesspace project over recent years, the coalescence of support in the archival community around the platform as the best-of-breed tool of archival description, and the emergence of community-developed add-ons like the excellent harvard excel import plugin (aspace-import-excel, [updated 2019]), the decision to migrate was an easy one. we have reached the point that mike rush called for in 2011, to “. . . move beyond our angle bracket fetish to develop and implement tools that allow us to focus on archival tasks. ” (hensen et al., 2011) [2] we did however take the opportunity to survey all non-ead archival description and lay the groundwork for future remediation efforts. [3] for specific ead cleanup and preparation for archivesspace import, we relied heavily on ground covered previously by our colleagues at other institutions. dave mayo and kate bower’s extensive description of ead data cleanup in “the devil’s shoehorn (mayo and bower 2017) proved extremely useful, as did mark custer’s yale as-prep schematron(custer 2015). [4] a separate script extracted eads from as based on bib id: https://github.com/cul/rbml-archivesspace/tree/master/ead_harvest [5] because of the complexity of the cleanup and merging activities performed on each record, we found we needed to break the xslt processing into three passes (ead_merge.xsl, ead_cleanup_1.xsl, and ead_cleanup_2.xsl). multiple stylesheets can be strung together in a single call to saxon, with the result tree of one becoming the input of the next. [6] this approach does not maintain as ids for resources. cul didn’t rely on these, but persisting these would likely require additional database-level work. [7] even minimal tweaks to shared sheets such as freezing header rows and write-protecting imported data columns make for much improved report ux. [8] our migration workflow did not make use of the as api since we were already working with ead xml and it was easier to merge xml with xml. however, a similar process could act on json data from api calls rather than xml via xslt; the python-google sheets approach described here is agnostic as to data interchange methods. references aspace-import-excel [internet]. [updated 2019 june 20]. harvard library of harvard university. available from: https://github.com/harvard-library/aspace-import-excel bartczak, j., & glendon, i. (2017). python, google sheets, and the thesaurus for graphic materials for efficient metadata project workflows. the code4lib journal, (35). available from: https://journal.code4lib.org/articles/12182 catapano, t., dipasquale, j., & marquis, s. (2008). building an archival collections portal. code4lib journal, (3). available from: https://journal.code4lib.org/articles/77 colin p., chassanoff a., lee c.a., rabkin a., zhang a., skinner k., meister s. (2019). digital curation at work: modeling workflows for digital archival materials. proceedings of the 19th acm/ieee joint conference on digital libraries, 39-48. jcdl ’19. urbana-champaign, il: ieee computer society press. doi:10.1109/jcdl.2019.00016. available from: https://colincpost.info/files/post-etal-digital-curation-at-work_final-preprint.pdf custer, m. (2015). validation scenarios – archivesspace @ yale. available from: https://campuspress.yale.edu/yalearchivesspace/2015/07/22/validation-scenarios/ huffman, n. (2016). getting things done in archivesspace, or, fun with apis. the digital collections blog. available from: https://blogs.library.duke.edu/bitstreams/2016/09/21/archivesspace-api-fun/ mayo, d., & bowers, k. (2017). the devil’s shoehorn: a case study of ead to archivesspace migration at a large university. the code4lib journal, (35). available from: http://journal.code4lib.org/articles/12239 ou, c., rankin, k. l., & shein, c. (2017). repurposing archivesspace metadata for original marc cataloging. journal of library metadata, 17(1), 19–36. available from: https://doi.org/10.1080/19386389.2017.1285143 hensen, s., landis, w., roe, k., rush, m., stockting, w., & walch, v. (2011). thirty years on: saa and descriptive standards (session 706). the american archivist, 74(supplement 1), 1–36. available from: https://doi.org/10.17723/aarc.74.suppl-1.15011hj3lg56t0t3 subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – respect my authority mission editorial committee process and structure code4lib issue 2, 2008-03-24 respect my authority some simple modifications to vufind, an open source library resource portal, improve the retrieval of both lists of works and information about authors from wikipedia. these modifications begin to address ways that current “next-generation” catalogs fail to fully harness all of the bibliographic tools available for indexing and presenting author information. simple methods such as those described in this article, which make use of full headings for authors, can offer marked improvements to these systems. by jonathan gorman introduction as current “next-generation” catalogs attempt to overcome the inadequacies of the previous generation, they have abandoned useful techniques that have evolved in the practice of cataloging. a good example is the display of search results for works by an individual author. many of the last generation of catalogs offered large browse lists of unique authors with little guidance for choosing between authors. the current generation lets you find all the books written by people with the same name but still offers little to those who want the works of just one author. relatively simple changes allow authority practice to be used more effectively in the next generation. this article will show a quick enhancement to the vufind system that improves use of its heading information to group books by particular authors. a similar technique could be applied to many of the next-generation catalogs. generations of catalogs and authority control many of us can still remember working with the card catalog. given a large enough collection, you would find yourself flipping through “johnson, james, 1705-“, “johnson, james, 1777-“,”johnson, james, 1835-“, and so on. each card would have the author’s name along with the information about a particular work. fanning the names gave a sense of what the person wrote or created. if the titles did not seem to match the search, you could just skip ahead until the pattern changed. one could learn to quickly scan the catalog in this manner, flipping the cards in a blur. then came the first generation of web interfaces. this generation typically used the concept of an author browse list. instead of seeing sequences like: johnson, james, 1705articles of visitation and enquiry johnson, james, 1705sermon preached before the right honourable ... johnson, james, 1777economy of health johnson, james, 1777tour in ireland: with meditations and reflections you saw: johnson, james, 1705johnson, james, 1777examples of such lists can be seen in many currently used systems: library of congress – voyager catalog urbana free library – horizon information portal. in theory, instead of having to look through a hundred cards to find an author, the searcher need only view a screen or two. in reality, the old card based system’s speed was limited by manual dexterity and mental agility. in newer systems, though, users must wade through the tedium of multiple distinct actions, and their speed is limited by the time spent waiting for pages to load. compounding the problem was a maze of see, see also, and scope notes, presented in a poorly explained and unfriendly interface. clicking an author link in an individual record may take you to a list of works by that author in some systems like voyager, while others like horizon took you to the appropriate place in the author browse. many next-generation catalogs take the opposite approach to the author browse list. works by authors who share a name get lumped together, usually derived from the subfield a of the main entry or added entries in the marc record. records for “carter, john, 1921-” will be interspersed with records for “carter, john, 1912-“. this can be seen in implementations of vufind, evergreen, worldcat.org, and koha. for smaller collections or less common names, this is not as big of a hurdle. there is one famous ray bradbury and getting all the books by him is relatively easy using just his name. however, as collections grow, more common names start colliding. in the case of large collections with works spanning centuries it becomes much more difficult to find what works a library may have by a single individual. your search can return hundreds of volumes with no obvious way to narrow the results down to the books of just one author. vufind’s author page can we do better in the next generation? i believe so. modifying vufind why use vufind? the interface pages use php, the indexing and searches use solr, and transforming the records for both indexing and display are done using xslt. i am already familiar with all of these technologies. vufind is under the gpl (version 2), so i can share my modifications. it’s easy to use as a stand-alone system. one just needs bibliographic records and minor modifications to the codebase to make it run without an ils, allowing for easy experimentation. there seems to be a surge of local interest in central illinois (where i work) about vufind. getting vufind currently, vufind is hosted at sourceforge. vufind 0.7, running on kubuntu 7.10, was used while writing this article, although release 0.8 is due soon with some potential changes (vufind is a young and active project, so details here may differ from the most current version). you can either download the files or check it out of svn (svn co https://vufind.svn.sourceforge.net/svnroot/vufind/releases/vufind-0.7 vufind). setting up vufind. you will want to follow the readme file that comes with the vufind distribution. see the wiki and check out the mailing lists if you have any issues. getting some records for testing this system a small sample of around 2,000 records was used (catalog.xml). they were combined from two different searches using the yaz z39.50 client [1] and querying the z39.50 server at carli [2], an academic consortium in illinois (see appendix i for details of this process). feel free to use the created catalog.xml file by placing it into the import directory. the file isn’t meant to be an exhaustive scientific test, just an exercise in seeing how we can treat authorities differently. now, if you’re following along, you should see how vufind normally uses the headings. take the catalog.xml file, put it in the import directory, and follow the vufind import steps (see the readme) and experiment with it. this will make sure you have the setup correct. however, the changes we are about to make will require re-indexing. the easiest way to do this again is to simply remove the existing indexes and records in solr. to do so, run the following curl commands while vufind.sh is running: curl http://127.0.0.1:8080/solr/update \ --data-binary '<delete><query>[* to *]</query></delete>' \ -h 'content-type: text/xml; charset=utf-8' curl http://127.0.0.1:8080/solr/update \ --data-binary '<commit />' \ -h 'content-type: text/xml; charset=utf-8' later, after modifying the index you will need to run “./vufind.sh restart” and re-run the import steps described in the readme. modifying the index to collocate all the records by a particular author, we’re going to take advantage of the system specified by aacr2 to allow people to distinguish headings. a heading in a catalog record, when created according to the rules of aacr2 chapter 22 [3], should provide a unique string for a given author to be used in all the records for works by that author or about that author. many libraries keep track of these headings by using authority files, one of the most frequently referred to being the library of congress authorities. the heading is derived mostly from the most well-known or commonly published form of an author’s name. information is also added to make sure the heading of a new author does not duplicate an existing heading. in a marc record these headings will be found in the main entry (100), used for the main author of a work; added entries (700), used to indicate people who contributed to a work but are not the main author; and subject added entries (600), used to indicate people the book is about. the marc record splits the heading into several subfields, such as the personal name (subfield a), titles (c), and date (d) [4]. vufind does index author information, but only a normalized version of the subfield a. it does not include the information used to distinguish between authors. when searching for authors we can search for authors that are jackson, michael or by just part of the author name, but we cannot get a particular author. this is fine for user searches where people are likely to be searching using names. however, for internal purposes something that uses a more unique identifier is needed so we can do things like describe just a particular author or show works by just one person. why not use the heading in the record which is already functioning as an identifier? to change vufind to use the full heading, it is useful to understand solr, the index engine underlying vufind. queries are done by passing an http get request to the index engine. that returns information in a variety of ways about the search and the search results. similarly, the index is constructed by uploading xml files consisting of name-value pairs via http post requests. an example might look like: <add> <doc> <field name="id">1</field> <field name="author">johnson, james</field> </doc> </add> the various fields solr will index in these uploaded documents are specified in a file called schema.xml (located in vufind at solr/conf/schema.xml). in order to allow the indexing engine to keep track of the full heading, we’ll add a line to the schema.xml file. find the fields element and add the line: <field name="authornaf" type="string" indexed="true" stored="true" /> now if the uploaded record contains the field name “authornaf” it will be added to the index. the type “string” is used since the “string” type requires a search to be an exact match in order to return a method. so search for an authornaf value of ajackson,_michael would not return a record that had ajackson,_michael,d1942-. currently, the “author” field uses “text”, which allows for partial matches in search. so searching for jackson would return the record “jackson, michael,”. next we will modify the import process that reads in the catalog.xml file and creates the upload files for solr. this process is driven by the import-solr.php script in import/. this script breaks each record element in the catalog.xml file into a string. each of these strings has an xslt transformation, marcxml2solr.xsl, applied to create solr upload files. these files are then posted to solr. to do this, i’ve added a section to marcxml2solr.xsl: original code <xsl:if test="//marc:datafield[@tag=100]/marc:subfield[@code='a']"> <field name="author"> <xsl:value-of select="//marc:datafield[@tag=100]/marc:subfield[@code='a']"/> </field> <field name="author-letter"> <xsl:value-of select="substring(//marc:datafield[@tag=100]/marc:subfield[@code='a'], 1, 1)"/> </field> </xsl:if> <xsl:if test="//marc:datafield[@tag=110]/marc:subfield[@code='a']"> <field name="author2"> <xsl:value-of select="//marc:datafield[@tag=110]/marc:subfield[@code='a']"/> </field> </xsl:if> modified code (added at about line 78) <xsl:if test="//marc:datafield[@tag=100]/marc:subfield[@code='a']"> <field name="author"> <xsl:value-of select="//marc:datafield[@tag=100]/marc:subfield[@code='a']"/> </field> <field name="author-letter"> <xsl:value-of select="substring(//marc:datafield[@tag=100]/marc:subfield[@code='a'], 1, 1)"/> </field> </xsl:if> <xsl:if test="//marc:datafield[@tag='100']"> <field name="authornaf"> <xsl:for-each select="//marc:datafield[@tag=100]/marc:subfield"> <xsl:value-of select="@code"/> <xsl:value-of select="translate(normalize-space(.),'abcdefghijklmnopqrstuvwxyz ','abcdefghijklmnopqrstuvwxyz_')" /> </xsl:for-each> </field> </xsl:if> <xsl:if test="//marc:datafield[@tag=110]/marc:subfield[@code='a']"> <field name="author2"> <xsl:value-of select="//marc:datafield[@tag=110]/marc:subfield[@code='a']"/> </field> </xsl:if> the added code joins all the subfields of the 100 field, after first removing any spaces before and after each subfield and converting the text to upper-case. for example, if $ajackson, michael, $d1942appeared in the 100 field of a a record, there would be a field in solr that would be searchable with the value ajackson,_michael,d1942if you’re following along, you’ll need to restart vufind.sh (./vufind.sh restart) and re-import (php import-solr.php) at this point. the author home page in vufind now, in order to demonstrate how the the index can be used to improve an interface, i’ll focus on the author page of vufind, which provides information from wikipedia about the author and a list of works by the author . currently, a record or search results page will link to the author page with a url like http://example.com/author/home?author=newman,%20paul. the php page resides in web/services/author/home.php, and takes in just one get parameter, author={processed subfield a}. for this information vufind uses subfield a of the 100 field, with some slight normalizations. this author information is used in two ways within the page. to create a url used to screen scrape a wikipedia page providing information about the author. $author = $_get['author']; if (substr($author, strlen($author) 1, 1) == ",") { $author = substr($author, 0, strlen($author) 1); } $author = explode(',', $author); // some unrelated code ... $url = 'http://en.wikipedia.org/w/api.php?action=query&amp;prop=revisions&amp;rvprop=content&amp;format=php&amp;titles=' . urlencode("$author[1] $author[0]"); you will notice it only uses the name, assuming it will find an entry in wikipedia for this particular author at “first name last name”. so, for any heading that has the personal name “jackson, michael,” you will get the information for the pop star since his wikipedia identifier is “michael jackson”. vufind’s display of author information from wikipedia to make a solr call to search for all books written by the author. // get records by this author $this->db = new solr($configarray['solr']['url']); $result = $this->db->query('author:"' . $_get['author'] . '"', null, 0, 20); //does some things with the resulting arrays to display the results this will collocate all books in the system written by authors with that name, even if they are different authors. results of an author search in vufind to improve both the wikipedia search and the list of author works, we need author/home.php to use our indexed full heading, authornaf, so we can refer to a unique bibliographic identity. later we will modify the author page to use the authornaf field, but first we have to pass it in. in places where the old url existed, it should now be formed to look like http://example.com/author/home?author=newman,%20paul&authornaf=anewman,_paul,d1921-. for an example of how existing urls in vufind should be changed, we will look at the display of an individual record. vufind uses the raw marcxml file and transforms it for display, similar to how it transforms it for the solr upload file. so in web/services/record/xsl/record-html.xsl, i modified a section that creates the url for the author in the record to also include the authornaf. original code (line 79) <xsl:template name="citation"> <table cellpadding="2" cellspacing="0" border="0" class="citation"> <xsl:if test="//datafield[@tag=100]"> <tr valign="top"> <th><xsl:value-of select="php:function('translate', 'main author')"/>: </th> <td> <a> <xsl:attribute name="href"><xsl:value-of select="$path"/>/author/home?author=<xsl:value-of select="//datafield[@tag=100]/subfield[@code='a']"/></xsl:attribute> <xsl:value-of select="//datafield[@tag=100]/subfield[@code='a']"/> </a> </td> </tr> </xsl:if> modified <xsl:template name="citation"> <table cellpadding="2" cellspacing="0" border="0" class="citation"> <xsl:if test="//datafield[@tag=100]"> <tr valign="top"> <th><xsl:value-of select="php:function('translate', 'main author')"/>: </th> <td> <a> <xsl:attribute name="href"> <xsl:value-of select="$path"/> <xsl:text>/author/home?authornaf=</xsl:text> <xsl:for-each select="//datafield[@tag=100]/subfield"> <xsl:value-of select="@code"/> <xsl:value-of select="translate(normalize-space(.),'abcdefghijklmnopqrstuvwxyz ','abcdefghijklmnopqrstuvwxyz_')" /> </xsl:for-each> <xsl:text>&author=</xsl:text> <xsl:value-of select="//datafield[@tag=100]/subfield[@code='a']"/> </xsl:attribute> <xsl:value-of select="//datafield[@tag=100]/subfield[@code='a']"/> </a> </td> </tr> </xsl:if> modifying wikipedia search we can now use the authornaf field to enhance the wikipedia search . we’ll do this using the following algorithm: use the authornaf to retreive the title field stored in solr for all records that contain that authornaf. find the two most common words in these titles after a simple stop list is used to eliminate common words like “the”,”of”, etc. create a wikipedia search url and use it to query wikipedia. extract the search results from the html page returned by wikipedia iterate through results in relevancy ordering until a url is found that has all the parts of the name use this url to retrieve the page and extract the first section. we’re assuming this page is about the author. this algorithm seems to work for many cases, but some further experimentation could involve: looking for a disambiguation page not counting parts of the first name as heavily using subject headings as search words using a local index of wikipedia or a local dbpedia for more complex searches doing this means we no longer get information about the king of pop everytime we search wikipedia; rather, we get information more suited to the individual or no information at all. improved results from wikipedia note: the relevant code for this can be found in appendix ii. modify the list of works by an author now, let’s focus on the second part of the author page, the list of all the author’s works. in this case, instead of doing a solr query that will get us records that have the same 100 subfield a, we’ll do a query that finds all records with the same normalized 100. so instead of looking for all works that have jackson, michael we will find all the bibliographic records that have a normalized 100 that looks like ajackson,_michael,d1942-. the result will be a list of just the works by the well-known beer and whiskey critic michael jackson. results of an authornaf search to accomplish this, we modify line 152 in web/services/author/home.php from $result = $this->db->query('author:"' . $_get['author'] . '"', null, 0, 20); to $result = $this->db->query('authornaf:"' . $_get['authornaf'] . '"', null, 0, 20); note: all of the changes from this article can be implemented with a patch file [5]. issues authority complexity a flaw in the use of the heading to collocate author works is that it makes the assumption that a person’s entire works will be represented by a particular heading. this assumption is fundamentally flawed. in certain cases there will be multiple headings for one person, as when an author writes under a pseudonym or a changed name, or writes as a government officer. since we do not examine the authority records, each of these headings will be treated in our modified vufind as if it is for a different author. take, for example, bill clinton, the 42nd president of the united states. he has one authorized heading as “clinton, bill, 1946-“, but he also has writings as: arkansas. governor (1979-1981 : clinton) arkansas. governor (1983-1992 : clinton) united states. president (1993-2001 : clinton) ideally, there would be some way to find all the writings of an individual but still be able to distinguish between writings done by separate “bibliographic identities”. this would require a much higher level of processing, including use of the authority records. the lack of descriptions regarding the nature of a connection between two authority headings is also problematic. us practice does not give enough information to identify the relationship between two authority records. if this information existed in a machine-readable form it would be possible to display, for example: material by stephen king writing under pseudonym richard bachman or works created as president by george w. bush lack of authority control. one of the largest stumbling blocks to implementing this system is the simple fact that not all libraries practice authority control. it appears a large majority of academic libraries do use some name authority control. according to a survey of libraries with a carnegie classification of either doctoral/research extensive or intensive level, 95% of those who responded (75%) do some form of authority control. for in-house work, 88% of new and maintence cataloging had the personal name headings verified. for vendor work 95% of new cataloging involved verifying personal name headings and 90% of maintence work verified personal name headings [6]. however, that leaves many libraries that do not practice any authority control. in fact, some may even practice a mixture by keeping the headings on records they import from external systems such as oclc but not adding any to their internal records. this would prevent the association of some books with their authors. in these cases, grouping all the records by authors with the same name may be the only solution. possible future directions: don’t use all the subfields the modification given in this paper is a simple algorithm that uses all the subfields of the 100 field of the bibliographic marc record. in the future, it would make sense to restrict the fields indexed to subfields a, b, c, d, q, and u. use added entry as well as main entry in the future, treat added entries and headings used as subjects in a similar way. this might allow for some more sophisticated interfaces. user testing far more user testing is needed, looking at when people look for items by particular authors and what can help them get appropriate results. with this technique we can take some different approaches than those currently offered by many next-generation catalogs. system testing what impact does having more unique headings have on the performance of large production systems? perhaps previous next-generation catalog developers have purposely avoided using the full heading for this reason. facets there is a need to try to improve the faceting of results. right now the interface will “group” all the authors with the same normalized subfield a together. however, this makes this particular facet useless when people are searching for books by an author’s name. if one clicks on a link in a record, they’ll just see the exact same search they performed. the identifier information could be used to populate the facet display with unique authors, even if they have the same subfield a. distinguished information could be added such as titles of works created by the individual or common subject headings assigned to the person. authority files briefly mentioned earlier, incorporating the authority files into a separate index could allow for interesting possibilities, such as multiple levels of searching by author (i.e., all works or just works written under a particular pseudonym). better linking to external projects if we can identify a person and the works they created, we can use that information to try to connect with other sources of information. we already link to wikipedia; we could also link to similar projects. static dumps of external projects wikipedia can be downloaded as a static file and re-indexed to focus more on information boxes, persons, and works. dbpedia and similar projects also offer rdf tags and a chance to search wikipedia information. using static dumps when possible allows for more intensive automated searching. for example, there are several works by michael jackson, an anthropologist in my current collection. the algorithm of using the most popular two words fails in this case. few of the words in his titles overlap and the highest ranking words do not appear in the wikipedia article. having a local source would reduce concerns of over-use and abuse of wikipedia bandwidth and other resources, allowing experimentation with different algorithms to find the most appropriate article. conclusion we have room for improvement in the next generation of catalogs. it may be that there are some cataloging pratices that could be changed to make automation easier, but there are also existing techniques that developers are not using to their full potential. the new generation of open systems brings new opportunities for experimentation. this does not have to be complex or intimidating; by changing just a few lines of code, we can create lists of the works of individual authors and improve the retrieval of author information from wikipedia. go experiment, and make our catalog interfaces better than they have ever been. notes this was run with the yaz-client version 2.1.18, from the ubuntu package. yaz-client is a client that is included with the yaz library consortium of academic and research libraries in illinois. for information on connecting to the z39.50 server hosted by carli, read “i-share via z39.50“. i-share is a catalog that contains all the records of most of the carli members. anglo-american cataloguing rules. second edition. revision 2002. american library association, chicago; 2002. (coins) for more information about the details of these marc fields, see the library of congress bibliographic data pages, in particular main entry — personal name, added entry — personal name, and subject added entry — personal name. or, see the oclc bibliographic formats and standards. apply the patch file to recreate the modifications to vufind described in this article. wolverton, robert e., jr. authority control in academic libraries in the united states: a survey. cataloging & classification quarterly. 41 (1), 2005. p 111-131. (coins) appendix i – getting records using yaz-client first, i issue the command to log into the i-share z39.50 server and start creating an outputfile. ($ indicates command line prompt). any records that i “show” during the session will also be recorded into the output file. $ yaz-client -u i-share auth.carli.illinois.edu:210/voyager -m somefile.marc if i wanted to narrow it just to a particular institution, i could use a different user name. for example, to search just urbana-champaign. $ yaz-client -u uiu auth.carli.illinois.edu:210/voyager -m somefile.marc i actually searched i-share for records with michael and jackson in the author fields, but only searched urbana-champaign for the graphic novels. it should be noted that results from running yaz-client are always appended to the filename given with the -m flag (somefile.marc in the example). it is not overwritten. this means you can build up a file from several different institutions and sessions. you will get a response that contains some information about the server and then another command prompt (z>). now you can do some searches and retrieve records. the search should return the number of hits, which you can then “show 1+number of hits” to retrieve. one word of warning, for larger return sets you may wish to do them in batches by doing “show 1+500; show 501+500” and so on. otherwise you risk timing out with busy servers. getting records by people named michael jackson z> find @and @attr 1=1003 jackson @attr 1=1003 michael sent searchrequest. received searchresponse. search was a success. number of hits: 483 records returned: 0 elapsed: 3.119026 z> show 1+483 getting records that might be comic books/graphic novels/comic strip collections z> find @and @attr 1=21 "comic books" @attr 1=21 strips sent searchrequest. received searchresponse. search was a success. number of hits: 5605 records returned: 0 elapsed: 4.772098 z> show 1+500 ... lots of files go zipping by z> show 501+500 ... lots of files go zipping by z> show 1001+500 ... lots of files go zipping by z> close z> exit finally, you will need the marc records to be in marcxml. another yaz tool can help here. yaz-marcdump -x somefile.marc > catalog.xml appendix ii – modified wikipedia code for retrieving author information the instructions below apply to web/services/author/home.php (see the original code): add the following function at line 29 (before the launch function): /* return: array consisting of word => number of times words appears input: $string the string to analize $words an array consisting of word => number of times words appear if there are existing values, they will be added to. this means you can pass in a series of strings and get the overall totals */ function countwords($string,$words) { foreach(explode(' ',$string) as $word) { //print("$word <br />"); $word = strtolower($word); $word = str_replace(array(':', ';', ',', "\'", '"', '(', ')', '|', '/', '?', '!', '@', '#', '$', '%', '^', '&amp;', '*', "\\", '.', '+', '=', '_', '~', '`', '"'), '', $word); /* not removed/kept out because hypens might be important probably could just focus on beginning and end of strings */ /* simple stop list */ if($word != '' &amp;&amp; $word != 'the' &amp;&amp; $word != 'a' &amp;&amp; $word != 's' &amp;&amp; $word != 'of' &amp;&amp; $word != 'on' &amp;&amp; $word != 'in' &amp;&amp; $word != 'an' &amp;&amp; $word != 'if' &amp;&amp; $word != 'to' &amp;&amp; $word != 'and') { $words[$word]++; } } return($words); } then replace lines 113-209 (after adding the above function): original code // clean up author string $author = $_get['author']; if (substr($author, strlen($author) 1, 1) == ",") { $author = substr($author, 0, strlen($author) 1); } $author = explode(',', $author); $interface->assign('author', $author); // connect to wikipedia if (!isset($_get['page']) || ($_get['page'] == 1)) { $url = 'http://en.wikipedia.org/w/api.php?action=query&amp;prop=revisions&amp;rvprop=content&amp;format=php&amp;titles=' . urlencode("$author[1] $author[0]"); $client = new http_request(); $client->setmethod(http_request_method_get); $client->seturl($url); $result = $client->sendrequest(); if (!pear::iserror($result)) { $body = unserialize($client->getresponsebody()); //check if data exists or not if(!$body['query']['pages']['-1']) { $body = array_shift($body['query']['pages']); $info['name'] = $body['title']; $body = array_shift($body['revisions']); $body = explode("\n", $body['*']); $done = 0; while(!$done) { if($body[0] == '') { array_shift($body); continue; } switch(substr($body[0], 0, 2)){ case "[[" : case "{{" : case "}}" : case "]]" : case "| " : //echo " sub : '" . substr($body[0], 0, 2) . "' "; $stpos = stripos($body[0], "image:"); if(!$stpos) $stpos = stripos($body[0], "image"); if($stpos) { $len = 4; $endpos = stripos($body[0], ".jpg"); if(!$endpos) { $len = 4; $endpos = stripos($body[0], ".gif"); } if($endpos) { $image = substr($body[0], $stpos, $endpos + $len $stpos); } } array_shift($body); break; default : $done = 1; break; } } $desc = ""; $done = 0; while(!$done) { if(substr($body[0], 0, 2) == "==") $done = 1; else { $desc .= $body[0]; array_shift($body); } } //create links to wikipedia $pattern = array(); $replacement = array(); $pattern[] = '/(\x5b\x5b)([^\x5d|]*)(\x5d\x5d)/'; $replacement[] = '<a href="http://en.wikipedia.org/wiki/$2">$2</a>'; $pattern[] = '/(\x5b\x5b)([^\x5d]*)\x7c([^\x5d]*)(\x5d\x5d)/'; $replacement[] = '<a href="http://en.wikipedia.org/wiki/$2">$3</a>'; // removes citation $pattern[] = '/({{)[^}]*(}})/'; $replacement[] = ""; $desc = preg_replace($pattern, $replacement, $desc); $info['image'] = $image; $info['description'] = $desc; $interface->assign('info', $info); } } } } modified code // clean up author string $author = $_get['author']; if (substr($author, strlen($author) 1, 1) == ",") { $author = substr($author, 0, strlen($author) 1); } $author = explode(',', $author); $interface->assign('author', $author); $authornaf = $_get['authornaf']; //we'll now search to see if we can find //a wikipedia article that seems associated with the //author by using common title words // connect to wikipedia if (!isset($_get['page']) || ($_get['page'] == 1)) { // get records by this author $this->db = new solr($configarray['solr']['url']); $result = $this->db->query('authornaf:"' . $_get['authornaf'] . '"', null, 0, 20); /* the result will have some information about the solr query and also information about each record. issue is this is an array of arrays, unless there's only one result, then it's just an array with values */ if (is_array($result['record'][0])) { $records = $result['record']; } else if (is_array($result['record'])){ $records = array($result['record']); } $titles = array(); $words = array(); for($i = 0;$i < count($records);$i++) { $words = $this->countwords($records[$i]['title'],$words); } asort($words); /* now the words should be sorted from most frequent to least */ $words = array_keys($words); /* now we search for the author words (from earlier processing) and the two most common words. why? some rouging testing seem to indicate this was a good number. */ $url = "http://en.wikipedia.org/w/index.php?title=special:search&amp;search=" . urlencode("$author[1] $author[0] " .array_pop($words) . " ". array_pop($words) ); //now we examine the results. $client = new http_request(); $client->setmethod(http_request_method_get); $client->seturl($url); $result = $client->sendrequest(); if (!pear::iserror($result)) { $xmlstring = $client->getresponsebody(); } else { print("<html><head><title>error</title></head><body>error</body></html>"); } //need to suppress warnings //errors about id $xmldoc = new domdocument(); //see http://www.mutinydesign.co.uk/scripts/problems-encountered-with-php-dom-functions---3/ on suppressing warnings -> bad html @$xmldoc->loadhtml($xmlstring); $docxpath = new domxpath($xmldoc); //for some reason i haven't quite yet figured out, //registering the namespace isn't working, //the dom class seems to ignore it in the source //document $query = '/html/body/div[@id="globalwrapper"]/div[@id="column-content"]/div[@id="content"]/div[@id="bodycontent"]/ul[1]/li/a'; $links = $docxpath->query($query); $goodlink = ''; //now, i'll iterate through the results //i'm looking for the first result that //has all the parts of the author name in it // //this could definitely be improved foreach($links as $link) { $firstname = $author[1]; $firstname = str_replace(array('.',','),'',$firstname); $firstname = trim($firstname); $lastname = $author[0]; $lastname = str_replace(array('.',','),'',$lastname); $lastname = trim($lastname); if (stripos($link->nodevalue,$firstname) > -1 &amp;&amp; stripos($link->nodevalue,$lastname) > -1) { //print("good link <br />"); $goodlink = $link->attributes->getnameditem('href')->nodevalue; break; } } $title = substr($goodlink,6); $interface->assign('info', $info); $url = 'http://en.wikipedia.org/w/api.php?action=query&amp;prop=revisions&amp;rvprop=content&amp;format=php&amp;titles=' . $title; //if we found something, display the wikipedia info //(in final version we'd want to have something displayed // if there wasn't a match or a more strict if ($goodlink != '') { $client = new http_request(); $client->setmethod(http_request_method_get); $client->seturl($url); $result = $client->sendrequest(); if (!pear::iserror($result)) { $body = unserialize($client->getresponsebody()); //check if data exists or not if(!$body['query']['pages']['-1']) { $body = array_shift($body['query']['pages']); $info['name'] = $body['title']; $body = array_shift($body['revisions']); $body = explode("\n", $body['*']); $done = 0; while(!$done) { if($body[0] == '') { array_shift($body); continue; } switch(substr($body[0], 0, 2)){ case "[[" : case "{{" : case "}}" : case "]]" : case "| " : //echo " sub : '" . substr($body[0], 0, 2) . "' "; $stpos = stripos($body[0], "image:"); if(!$stpos) $stpos = stripos($body[0], "image"); if($stpos) { $len = 4; $endpos = stripos($body[0], ".jpg"); if(!$endpos) { $len = 4; $endpos = stripos($body[0], ".gif"); } if($endpos) { $image = substr($body[0], $stpos, $endpos + $len $stpos); } } array_shift($body); break; default : $done = 1; break; } } $desc = ""; $done = 0; while(!$done) { if(substr($body[0], 0, 2) == "==") $done = 1; else { $desc .= $body[0]; array_shift($body); } } //create links to wikipedia $pattern = array(); $replacement = array(); $pattern[] = '/(\x5b\x5b)([^\x5d|]*)(\x5d\x5d)/'; $replacement[] = '<a href="http://en.wikipedia.org/wiki/$2">$2</a>'; $pattern[] = '/(\x5b\x5b)([^\x5d]*)\x7c([^\x5d]*)(\x5d\x5d)/'; $replacement[] = '<a href="http://en.wikipedia.org/wiki/$2">$3</a>'; // removes citation $pattern[] = '/({{)[^}]*(}})/'; $replacement[] = ""; $desc = preg_replace($pattern, $replacement, $desc); $info['image'] = $image; $info['description'] = $desc; $interface->assign('info', $info); } } } } } (final modified code for web/services/author/home.php) about the author jonathan gorman spends his days shifting bits and bytes through the voyager system, among several other duties, as a research information specialist for the university of illinois. his nights are spent playing around with library technologies in hopes of constructing tools he would actually enjoy using. jonathan is grateful for the love and support of his wife, colleen; this article would not have been possible without her. he can be contacted at jonathan.gorman at gmail dot com. subscribe to comments: for this article | for all articles 2 responses to "respect my authority" comments on this article are currently closed. daniel lovins, 2008-04-21 this article addresses a concern we’ve been discussing at yale for several weeks. thanks so much for sharing the results of your investigations. jonathan gorman, 2008-04-21 hi daniel, glad my article was of some help. in case you check back, do you mind giving a summary of the concern you have at yale? you made me curious. issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – quality control automation for student driven digitization workflows mission editorial committee process and structure code4lib issue 60, 2025-04-14 quality control automation for student driven digitization workflows at union college schaffer library, the digitization lab is mostly staffed by undergraduates who only work a handful of hours a week. while they do a great job, the infrequency of their work hours and lack of experience results in errors in digitization and metadata. many of these errors are difficult to catch during quality control checks because they are so minute, such as a missed counted page number here, or a transposed character in a filename there. so, a computer science student and a librarian collaborated to create a quality control automation application for the digitization workflow. the application is written in python and relies heavily on using openpyxl libraries to check the metadata spreadsheet and compare metadata with the digitized files. this article discusses the purpose and theory behind the quality control application, how hands-on experience with the digitization workflow informs automation, the methodology, and the user interface decisions. the goal of this application is to make it usable by other students and staff and to build it into the workflow in the future. this collaboration resulted in an experiential learning opportunity that has benefited the student’s ability to apply what they have learned in class to a real-world problem. by corinne chatnik and james gaskell introduction digitization of cultural heritage materials in academic libraries is important for increasing the access and visibility of its holdings, providing content and data for digital scholarship, and allowing for increased accessibility of library materials. the output of this work is valuable to researchers and the union college community, but it is also resource intensive and time-consuming. union college schaffer library experiences these challenges particularly in the context of a predominantly student-staffed digitization operation. the digitization program at schaffer library relies heavily on undergraduate student workers for the capture and quality control of the digital archival records. union college is a solely undergraduate institution and the student work hours are limited to a few hours per week with varying schedules. most undergraduates do not have prior experience with digitization and this is a new skill set for them that requires significant training. successful training is made difficult due to the differences in the students schedules resulting in ad hoc training sessions and sometimes days between each work shift. there is also currently no staff member or librarian dedicated to the digitization lab full time; instead, supervision of students is divided between two librarians and one staff member. with these limitations, it was recognized that more qc can’t be done but better qc can be achieved with automation. to do this, a collaborative project was initiated between the library’s digital collections and preservation librarian, corinne chatnik and a senior union college computer science student, james gaskell, who was initially hired and trained to do digitization. together they planned and developed a quality control automation application. by leveraging python programming and the openpyxl library, the aim was to create a tool that could systematically verify metadata consistency and file management accuracy, thereby reducing the burden on manual review processes. this approach was informed by direct experience with the digitization workflow and an understanding of common error patterns observed over time. the resulting application needed to be both sophisticated enough to catch subtle errors and user-friendly enough to be integrated into the existing workflow, particularly considering the varying technical expertise of student workers and staff. this paper details the journey in developing this quality control automation solution, from initial concept to implementation. it will explore how the application addresses specific challenges in the digitization workflow and the technical decisions that shaped its development. moreover, how this project served as an experiential learning opportunity, allowing james to apply classroom knowledge to solve real-world problems while contributing to the library’s digital initiatives. background in 1795, union college was established in upstate new york as the first non-denominational institution (not affiliated with a religious organization) of higher education in the united states. today, it is a small undergraduate, liberal arts college committed to the integration of arts and humanities with science and engineering. union college has a long history evidenced by the collections held by the schaffer library’s special collections and archives. these collections are not only valuable to the institution but to a broader audience. with the significance of these collections, schaffer library hopes to increase digitization. automating the quality control aspects of the workflow will help this initiative. the digitization program at union college has a relatively short history. schaffer library began digitizing small amounts of material from special collections in 2008. then a loss of funding resulted in a pause in digitization efforts. in 2014, library leadership recognized the importance of digitization for access and accessibility and plans were made to restart the program. a staff member with digital projects experience was hired to take over the digitization lab and complete all aspects of the digitization workflow. these projects were small boutique collections chosen based on faculty and librarian interests. the resources allocated for these projects were not conducive to scaling up to larger initiatives. taking advantage of various staff turnover, new positions were created with assembling a digital projects team in mind. an increase in technical skills and positions with digital projects responsibilities allowed schaffer library to start scaling up digitization efforts. this also came with the recognition that undergraduate student workers were a valuable resource in document imaging, so the library enacted plans to hire and train students for that purpose. with increased production, it identified some areas of the workflow that acted as bottlenecks to the process, quality control checks being one major area. this analysis served to signal to schaffer the need to improve quality control for increased efficiency and realized for some aspects, automation was possible. methodology quality control workflow prior to automation to consider automating the process, the digitization and quality control workflow was scrutinized. for the existing workflow, the digital projects librarian creates a metadata spreadsheet prior to pulling material for digitization. they then share the spreadsheet and pull boxes or folders of the corresponding records for the students to scan. while scanning, students read the metadata and make sure it matches the physical document. they should be verifying metadata fields like title, date, and creator match what they can see on the record. after they scan, they enter the extent, or number of pages of the document into the metadata spreadsheet. once they have completed the scanning portion they move on to quality control. for quality control, students should check work scanned by other students. this is so that they are not desensitized to the records they’ve already gone through. additionally, if they required more training, their errors would be caught by someone else. when students begin the quality check, they navigate to the folder of completed scans. each item has its own directory with the scans inside. so, going item by item in the metadata, they search for the item identifier and match it to a directory. they then open the folder to visually confirm that filenames in the parent directory match and all those filenames match the identifier in the spreadsheet. if there is anything that doesn’t match, they enter “fail” in the qc pass/fail column which is the last column of the spreadsheet. next, if the item is a multipage record, they make sure page count matches the extent column in the spreadsheet. if the values are not the same, they fail the check. if the document is less than 20 pages, they should look at each page individually. for expediency, if the document is longer than 20 pages, the students are to skim through the images, taking a random sample, approximately 10% of the total page count and verifying the quality of the images. when checking the images they evaluate the color balance, making sure the color looks approximately correct, not too blue or red, for example. next they observe the orientation, checking if images are mistakenly disorientated. overall, students are encouraged to use their best judgment. the role of quality control in the digitization workflow quality control is vital, not just for accuracy for descriptive metadata it also ensures that administrative and technical metadata are accurate and complete, making materials findable and usable within digital collections. additionally, poor quality digitization (like blurry text, missing pages, or incorrect color reproduction) can lead to misinterpretation or make materials unusable for research [1]. not only should digitization result in a high quality product, the data is vital to the success of the digital repository workflow. if some aspects of the metadata are not exact, it will cause the repository part of the workflow to fail. each batch of digitized records is over one hundred images and filenames. it’s easy to skim hundreds of filenames and miss a transposed identifier or misread a date. especially when the workflow for student workers ends with quality control and they don’t work with the files in the digital collections repository. in many cases, the data errors were not found until the upload process. several data points can act as a point of failure during upload, for instance, if the system is unable to find a file through the filepath, no digital object will display with the metadata. if the digital object file is too large, that will also cause display issues. another is if the date is not in iso format, the date facet will not work. the upload will also fail if required fields are not filled. these quality control failures will disrupt the workflow and if not caught before upload, undoing the work to repair it is time consuming and resource intensive. union college’s digital repository is archipelago, a flexible, customizable, open source repository created by metropolitan new york library council’s digital services team. the software is built on drupal with custom modules and indexed with solr [2]. for the upload process, the digital files are transferred via ftp from a onedrive directory to an amazon s3 server and the metadata spreadsheet is uploaded through the archipelago interface. the metadata contains the file’s full url path in addition to the other fields. the metadata is processed through a php template engine called twig and calls the files from the s3 path for display [3]. if the file path is not an exact match, the upload fails. additionally, there are other variables that, if not correct, will cause the upload or interface functionality to fail. part of the responsibilities of the digital collections librarian is uploading the metadata and scans to the digital repository. in doing so, corinne was encountering these errors during the upload process. after getting several completed batches of digitized records but still encountering some of the same issues, she wrote small scripts to check on those elements. this ad hoc process was sufficient until the digital projects team began ramping up digitization. it was then clear that these errors needed to be caught earlier and data checking could be integrated into the workflow. james was part of a cohort of students hired for digitization. with his background in computer science, he pretty quickly realized that some aspects of the workflow could be automated. especially in regards to saving time by eliminating some manual processes. with his hands-on experience implementing the workflow, he understood the goal of the application and how it could fit into the workflow going forward. together, they analyzed the workflow. corinne’s perspective was from the digital collections repository and what was going wrong during upload and indexing. james’s perspective was rooted in the actual digitization process. through this process, the following variables were identified that could be systematically evaluated both within the spreadsheet and by comparing the scans to the metadata. the biggest error encountered during the upload was mismatched or missing filenames. as a result, one of the first checks is determining if the filenames listed in the metadata, have a corresponding file in the scans directory. building on that, for archipelago, the full file path is required for upload and rendering. therefore, in addition to the filename, the metadata also has to have the full amazon web services filepath. both the filename to scan verification and the aws path check are pattern matching strings which is something scripting is really useful for automating. similarly, the number of pages in the extent field could be verified automatically. the input guidelines for that field are integer plus format; this means “x pages” with the rich text clarifying the integer represents a page count metric . so by identifying the extent field, if the “pages” string or other format string is stripped out, the integer can be isolated. then by locating the images, it can determine the actual number of pages that belong to each object. at that point it’s just a matter of comparing two integers. figure 1. an excerpt of a metadata spreadsheet in excel showing the identifier field, the aws path to the image, and the extent field with the page numbers. schaffer library’s metadata schema has required fields. so for those fields like label, type, ismemberof, and rights_statements it was just a matter of confirming a multi-character string was there. figure 2. an excerpt of a metadata spreadsheet in excel showing the required metadata fields for upload. the ismemberof field holds the identifier of the parent object in archipelago. the type field triggers archipelago processing to use a mapped iiif template. the label field is the same as the title but archipelago uses it to name the digital object. finally, rights_statements tells archipelago which https://rightsstatements.org/ icon to display. for the date_created field the qc check was a bit more complex. there is apache solr faceting functionality in archipelago for date faceting so the date needs to be formatted in yyyy-mm-dd (iso 8601) format. an even more complex match is when the filename is derived from the analog version of the record’s physical location within the collection. for example, an item located in box 2, folder 1, will have an identifier represented as zwu_sca0319.b02.f01 and those values need to be accurate. figure 3. an excerpt of a metadata spreadsheet in excel showing the more complex fields for the quality control check. physical location informs the identifier and date_created needs to be in yyyy-mm-dd (iso 8601) format for apache solr indexing and faceting. finally, with hundreds of scans and different people working on different digitization workstations and software, it is possible image file settings will get changed. archipelago doesn’t handle the display of massively large files well. this needs to be addressed prior to upload so it doesn’t overload the archipelago system. it was determined that if the size of the image file is over 500 mb it needed to be flagged and its size reduced. considering the metadata and image variables in this way laid the groundwork for automation. application design technical approach python was chosen as the programming language for this application because of shared knowledge, longevity, and python’s ability to interact with excel files, which is the format the metadata is stored in. it is also the programming language that james and corinne have in common. this is important because when james graduates he will no longer be able to support the software. but with corinne’s knowledge of python, the program can be maintained and updated. additionally, python is the most widely taught programming language at union college so there’s a greater chance future students can work with the application. python is also an attractive choice due to the vast array of packages available for data handling and i/o operations. openpyxl is the best choice for reading and writing to excel files. it allows for more complex formatting than most csv handlers and it is possible to highlight and change text formatting at a cellular level [4]. this is important since some discrepancies will be more easily fixed with a manual review and these features will indicate problematic records and fields while maintaining the metadata format. the package also reads into pandas dataframes with column headings rather than excel column references making it easier to extract and reference data. since a user interface is provided to make the program more friendly for work study students, tkinter is valuable for providing helpful error messages, and pyqt5 to provide a rich, user-friendly experience for the main functions of the program [5]. finally, python supports object oriented programming. since records in the metadata spreadsheet refer to physical holdings and digital copies of these holdings with attributes such as pagecount, permanent location and filename, it makes sense to store these records as objects. with a macro lens, sheets of a spreadsheet can be viewed as objects, and perhaps even individual spreadsheets should the scope of the project increase. technical architecture as discussed, it is imperative that the automated processes fit into the existing workflow to minimize disruption to the current digitization processes. further, the limitations of automation are understood. recording metadata and reviewing scanned images currently remain manual processes since the technology required to extract data or make human inferences from images is complex and inaccessible at this time. as such, james decided to split the program into three stages. all three of these functions are reliant on the same excel spreadsheet and file system and will adopt the same object structure and add more fields at each step in the process as they become necessary. the file structure is as follows: a spreadsheet contains multiple sheets. these sheets correspond to a physical box which has individualized items (the rows of the spreadsheet). these items represent both the physical holdings and the files in which the digital copies are stored. the items have attributes based on their spreadsheet data. attributes include date created, physical location, filename, page extent etc. data validation the first function of the program is data validation which should be initiated after the metadata spreadsheet is created. at this stage the file structure is created, the spreadsheet is represented as an object with a list of files, which are also objects with attributes like page count etc. to automate work with microsoft excel spreadsheets, james utilized openpyxl. openpyxl is a python library that can read and write to microsoft excel files. the program reads the spreadsheet into a dataframe using the pandas python package. pandas organizes each row of the excel spreadsheet into a table and has search functions that make the imported data easier to navigate [6]. the program goes through each record in the pandas dataframe and creates a file object then adds it to the spreadsheet wrapper. this allows a comparison of the physical location data entered for each object to the filename generated from that data which follows the name convention of “box.x.folder.y.item.z”. this function also checks if the date is in iso format, the international standard for representing dates, and checks the identifier and filepath columns for duplicate filenames. some errors and their solutions can be anticipated. most date errors are resolved by understanding alternate date formats like month dd, yyyy or mm/dd/yyyy. other variables, such as naming conventions and duplicate filenames, are more complicated and will require manual intervention. still, this will reduce the time needed to resolve the errors should they be discovered after the physical document has been scanned. file location and instant fails at the second step, assuming the initial discrepancies have been resolved and the scanning has been completed, the program will search for the scanned documents in onedrive. this process is one of the most time intensive and, until now, a manual process of copying the filename from the spreadsheet into the search bar of onedrive. at this stage, the application checks if the file exists, determines if the size of the file is below the 300mb threshold, and if the number of pages in the file matches the extent recorded in the metadata spreadsheet. this all occurs with a single click of a button. using the glob python package it recursively scans a user-selected folder with a much lower margin for error and much more quickly than the current manual method. glob takes a root directory, provided by the user through the ui, and the list of file objects from the sheet, then checks every subfolder for the file path [7]. if there are missing files, the item automatically fails the check, and a useful message is output to the spreadsheet to indicate to the user that the file could not be located. image quality checks after searching the file structure, the scanned document’s existence is confirmed or denied. if the file does exist, and thus can continue with the qc process, it also stores a file path that can be used to open the document from within the python application. this is much quicker than manually searching the file structure, opening the file and following the steps of the qc guidelines. further, since the python program has the ability to edit the metadata spreadsheet the application can mark the record as pass or fail, limiting human interactions with the metadata record which could introduce sources of error. user interface considerations and design principles command line scripting can seem overwhelming to those unfamiliar, and since this application needs to be integrated into a workflow primarily done by students, an interface was necessary. james designed a graphical user interface (gui) to make the program more usable to everyone who is part of the digitization workflow. the windows operating system is the primary os for schaffer library student computers and digitization workstations. with windows as an application parameter, tkinter will suffice for less detailed error messages. tkinter is a standard python interface package for tk gui. it is used to create simple interactive applications for python scripts [8]. the application also utilizes pyqt5, because it is more feature rich than tk and the drag and drop methods for designing the application in qtdesigner are more accessible to less experienced ui designers [9]. pyqt5 also benefits from css support allowing for aesthetic improvements and hover-over button information which should make the program even more user friendly. some effort has been made in this regard but it is still in its infancy, should there be a need for further aesthetic improvements to the gui, this could be done without much complexity by adding to, or altering, the current css. the gui is important for making sure the application is useful to as many people as possible and that it stays a sustainable part of the workflow even if the developers move on. where possible the program can utilize existing python gui architectures such as easygui for basic tasks like file selection. with a gui, the program has both a front-end and back-end that are, for the most part, independent of one another. as such, the singleton design pattern can be adopted to link the two, allowing for future independent changes to be made to both whilst ensuring a clean project structure for future development. the singleton, which is considered the main program, is a class guaranteed to only have one instance [10] and bridges the front-end and back-end operations. more importantly, there is a plan to implement new features and that can be done easily in this framework. when those features are written on the backend, modifying the interface is adding another button to the gui to trigger the new behavior. for example; initially it may appear as though a spreadsheet can also be a singleton with just one instance of the class, however, this design pattern guarantees expandability by allowing the tracking of multiple spreadsheets should the need arise due to changes in the workflow or increased system requirements. a change like this would require a change to the front end to allow the user to select a spreadsheet from a given list. integration with the current workflow implementing this application with the existing workflow will not create too large of a shift. though the data is read into the program and transformed for analysis, the program will convert the objects back into dataframes and output them back to the original spreadsheets. to the user, the spreadsheet seemingly remains unchanged after each check except for color coded flags to indicate errors. automatic failures due to extent and file size issues alongside files that cannot be located in onedrive will automatically be indicated in the pass/fail column and a useful error message produced. it is important to maintain the original data format and retain excel as the medium for data review. though a lot of tasks are automated, some variables require human intervention and this method changes that portion of the workflow very little. for the work-study students this application requires very little additional training to adapt to the new process. moreover, the user interface should be intuitive to use and the overall design is meant to reduce each process to a few mouse clicks. implementation the first step in creating the program is implementing the openpyxl read/write functionality. at both the read and write stages the file data is ported into rows of a pandas dataframe, so the object file structure described acts as an intermediary medium for processing. openpyxl handily uses the column headings for dataframe fields. furthermore, iteration over the rows of the dataframe discards unnecessary data and initializes abstract file objects for each row. at this stage the file objects have attributes; permanent location, filename, extent and date created, and the list of files is stored in the parent spreadsheet. this is also an object with a “sheetname” class variable to identify it in the full spreadsheet. importantly, each file also has an “errors” and a “failures” dictionary which can be used at each step to identify issues with the record. the dictionaries contain date, filename, duplicate filename, extent, filesize and existence flags which are all instantiated to false. updating these boolean flags indicates if an error has been found for the respective file for output to the spreadsheet. in order to translate these objects back into the data frame for writing, the filename can be considered a de-facto unique identifier. duplicates duplicate checks are the least complex but require the most human intervention to remediate. the duplicate check is implemented with a linear search. since the number of records on each sheet is never more than 200 this method suffices but could be scaled up if the metadata sets grow or for other libraries with higher digitization output. if the function finds a duplicate filename within the list of files, the ‘dupfilename’ flag is set to true – this is done for each instance of the duplicate. """checks for duplicate filenames in the list of files args: sheet: excel sheet containing a list of files """ def check_duplicate_filenames(sheet): for file in sheet.filelist: for comp_file in sheet.filelist: if file.filename == comp_file.filename and file != comp_file: file.errors['dupfilename'] = true sheet.errors += 1 figure 4. checking the spreadsheet for duplicate values in the filenames field. physical location each filename may not match its physical location. this is the first major check conducted, primarily because at this stage, it is the cause of most failures. the file location format in the spreadsheet is plaintext which is traditionally difficult to work with. location descriptors in the filename, such as “box” and “folder” are reduced to “b” and “f” respectively. increasing the complexity further, the location names also have item descriptors “bulletin”, “sheet” and “item” – when translating to filenames, bulletins are reduced to “bull”, sheets remain “sheet” and the item identifier is dropped entirely. since the aim is to maintain the existing workflow, the file naming conventions cannot change and these identifiers must be accounted for when matching locations with filenames. to effectively compare the location with the filename there are two options: one is to attempt to recover the filename from the location or vice versa. the former was chosen. the first part of the filename is computed as the most common prefix by taking the first string appearing before the period in the list of filenames in the metadata spreadsheet. importantly, this ensures the program can be used with different collections. the william stanley jr. collection which was used to inform the development of this program has the prefix “zwu_sca0319”. files that do not conform to this are marked as errors for manual review accounting for a rare edge case in which an item from a different collection may be grouped incorrectly. the algorithm below is an averager, using a dictionary to track the number of examples of each prefix and returning the mode prefix to the filename predictor subroutine. """determines the correct prefix for the files by finding the most common from the file means the script can be used for different collections args: list[scanfile]: list of file objects from excel sheet returns: string: most likely prefix given all the entries """ def find_file_prefix(filelist): filenamedict = {} #dictionary containing the prefixes in the document and their count for file in filelist: try: #ignore any funky filenames prefix = file.filename.split('.')[0] if filenamedict.get(prefix) == none: filenamedict[prefix] = 1 else: filenamedict[prefix] += 1 return(max(filenamedict)) #returns the most common prefix assumes this is correct except: pass figure 5. checking the filenames for the collection prefix. splitting the physical location field by commas, the program is able to discard filler words “box” and “folder” from physical locations thus converting the data into more usable, numeric form. the “bulletin” and “sheet” identifiers are retained by checking the full string for these substrings meaning item type identifiers are propagated through to the final filename prediction in format “bxx.fxx.bullxx” """checks that the filename matches the permanent location for each item in a given sheet reconstructs an expected filename from the location then compares it to what is recorded follows a precise naming convention. box xx, folder xx, item type xx records the error in the file's errors dictionary if there is a discrepancy args: sheet: excel sheet containing a list of files """ def check_location_filename(sheet): prefix = find_file_prefix(sheet.filelist) for file in sheet.filelist: pred_filename = prefix if not file.location == none: location = list(filter(none, file.location.translate(str.maketrans('', '', string.punctuation)).split(" "))) #ignore any funky filenames try: file.filename = file.filename.replace(" ", "") #removes any spaces that shouldn't be in the filename except: pass if "box" in location: pred_filename += ".b" + location[1].zfill(2) if "folder" in location: pred_filename += ".f" + location[3].zfill(2) elif len(location) == 4: pred_filename += "." + location[3].zfill(2) #accounts for items not in folders if "bulletin" in location: #assumes .bull. for bulletins pred_filename += ".bull." + location[5].zfill(2) elif "sheet" in location: #assumes .sheet. for sheets pred_filename += ".sheet." + location[5].zfill(2) elif len(location) >= 6: #assumes no identifier for items pred_filename += "." + location[5].zfill(2) if pred_filename != file.filename: file.errors['filename'] = true sheet.errors += 1 figure 6. checks that the filename assigned matches its permanent location value for each item. after comparing the prediction to the filename recorded in the file object, the program can deduce if there is a mistake and, if so, flips the boolean flag for “filename” in the error dictionary to true. date format the date formatter is designed to systematically attempt to convert from multiple expected date formats into iso. the most common issues with formatting are years without months and days, and the use of commas rather than periods or backslashes as separators. of course, some dates are already in the correct format so this is also accounted for. """checks the date format and attempts to format the date if in unexpected form args: sheet: excel sheet containing a list of files returns: boolean: true to verify the process was executed successfully """ def check_date_format(sheet): spell = speller(lang="en") for file in sheet.filelist: success = false if file.date != none: if not type(file.date) is datetime.datetime: try: date = (parse(file.date.rstrip())) success = true except: if type(file.date) is str and not success: success, date = attempt_format((file.date)) elif type(file.date) is int and not success: success, date = year_to_date((file.date)) if not success: #last ditch effort, successful if incorrect spelling in date try: date = (parse(spell(file.date.rstrip()))) date = date.strftime("%y-%m-%d") file.date = date success = true except: file.errors['date'] = true sheet.errors += 1 else: file.date = date.strftime("%y-%m-%d") else: file.date = file.date.strftime("%y-%m-%d") return true figure 7. checks the date format and attempts to format the date if in unexpected form. the main formatter method uses helper functions such as the one shown below with a success flag indicating whether the returned date could be successfully converted. since the methods use type casting and type specific operations, it is important to encapsulate them within exception handlers (in python try, except), without this the program would crash due to the variability of formats. should all the methods fail “date” is added to the errors list so the record can be manually reviewed. interestingly, a common date issue was the misspelling of written dates, again highlighting the abundance of human error. this was tackled by using the autocorrect speller package after which we are able to convert into iso using regular type casting. """converts dates from year to year and day. e.g. 1980 to 1980-01-01 args: date: date in year form returns: [boolean, date]: success flag and date in iso format """ def year_to_date(date): try: date_new = datetime.datetime(date, 1, 1) return [true, date_new] except: return [false, date] figure 8. converts dates written as a date range to iso format. file existance, size and extent the largest source of failure came from missing files within the onedrive file structure. since individual scans are saved as jpeg images and multi-page scans are saved as pdfs the program searches for both cases using rglob, a recursive search package that can find files within a larger file structure. the parent directory, selected by the user, is stored in the program singleton meaning the file structure can be searched by glob and the full file paths can be stored into the file object. the method uses easygui which in turn uses windows file explorer to again ensure a familiar user interface for the quality control student. """opens an easygui window to allow the user to select the file they want to parse returns: string: filepath of the selected file """ def get_file(): path = easygui.diropenbox() return path def find_file(folder): for path in path(folder).rglob('*.pyc'): print(path.name) return true figure 9. opens an easygui window to allow the user to select the file they want to parse. since the file structure is complicated and some files are at greater depths within subfolders, there is a moderate time requirement for this step. helpful print messages display, showing the process is in fact still running for the approximate 1 minute it takes to look for every file. if the file is found, the “exists” flag is set to true. while the program verifies the existence of each file, it also checks the file size. the size of the file in megabytes, is read into the file_size variable within the file object. if the extent falls above the preset size threshold of 300mb, the too_large variable is set to true, otherwise it is set to false. """conducts the preliminary qc checks checks if the file exists in the file structure, if the extent is correct and if the file size is less than 300mb adds the respective failures to the count and to the file's failure dictionary args: sheet: excel sheet containing a list of files parent_directory: the onedrive parent folder to search through """ def check_files(sheet, parent_directory): failures = 0 for file in sheet.filelist: try: for path in path(parent_directory).rglob(file.filename + '.[pdf jpg]*'): file.filepath = (parent_directory + "\\" + file.filename + ".pdf") # can i do this using path above? # we can't assume this will be a pdf as single pages stored as jpg file.exists = true # we perhaps can't assume this is correct since some don't have parent folders if file.extent == len(os.listdir(path.parent.absolute())) 1 or file.extent == none: file.failures['extent'] = false else: file.failures['extent'] = true failures += 1 if not (os.path.getsize(file.filepath) >> 20) < 300: file.failures['filesize'] = true failures += 1 except: pass if not file.exists: file.failures['existence'] = true failures += 1 sheet.failures = failures # add number of failures to the sheet object figure 10. checks if the file listed exists and the file size. checking page count proved to be more complex than initially anticipated. since the files are hosted using onedrive, to use the built-in page enumeration tools provided in python’s pdf handling packages, the pdfs would have to be downloaded. this is not possible given the size and number of files. fortunately the scanning software used at union college maintains a copy of each page in jpeg form inside each subfolder. utilizing this structure, the program is able to count the number of jpegs and compare this with the recorded value, but this process is highly tailored to the digitization process at the college and may need adapting elsewhere. for every type of failure a helpful error message is added to the data frame along with a “fail” in the necessary column. the program tracks the failure rates for each of the three categories which can be used to evaluate the accuracy of different parts of the digitization process. indicating errors as described, there are many errors that are too complex to be remedied by the program and must be highlighted for human review. with openpyxl we can do this literally by selecting a color, highlighting the problematic row, and overwriting the metadata spreadsheet. the program currently uses orange (hex #efbe7d) to indicate file naming issues, blue (hex #8bd3e6) to indicate duplicates, and yellow (hex #e9ec6b) to indicate date errors. these color codes are assigned but a new method of selecting colors is also being considered. the program will likely be run more than once at each stage, allowing for errors to be manually rectified and the application run again. as a result, the first step is resetting the color of rows within the spreadsheet. the automation program is only one source of highlighting since the digital projects and metadata librarian also use various colors to convey information about records. so those messages aren’t affected; the spreadsheet only resets the colors with the precise hex values used by the program. the function that does this is shown below. """removes specific highlight colors from the spreadsheet to allow the program to be run continually after each error is rectified args: excelfile: excel file to remove colors on contains the error colors dictionary wb: work book opened with openpyxl colors_to_remove: allows specification of individual colors so running spreadsheetchecks alone (for example) doesn't remove failure highlighting """ def reset_colors(excelfile, wb, colors_to_remove): fill_reset = openpyxl.styles.patternfill(fill_type=none) for sheet in excelfile.sheetlist: ws = wb[sheet.sheetname] for row in ws.iter_rows(): for cell in row: if cell.fill.start_color.index in colors_to_remove.values(): cell.fill = fill_reset """highlights rows with errors and/or failures with their corresponding hex color since the function pulls from both the error dictionary and the failure dictionary it can be used in both parts of the program args: excelfile: contains the failure/error information and the highlighting colors returns: boolean: true if the save is successful, false if not """ def highlight_errors(excelfile): xl_file = pd.excelfile(excelfile.filepath) wb = openpyxl.load_workbook(xl_file) reset_colors(excelfile, wb, excelfile.errorcolors) for sheet in excelfile.sheetlist: dt = pd.read_excel(xl_file, sheet.sheetname) ws = wb[sheet.sheetname] errors = sheet.getsheeterrordict() errors.update(sheet.getsheetfailuredict()) colors = excelfile.errorcolors colors.update (excelfile.failcolors) for file in errors: error_color = colors[errors[file]] if error_color != none: fill = openpyxl.styles.patternfill(start_color=error_color, end_color=error_color, fill_type="solid") for index, row in dt.iterrows(): try: if file == dt['filename'][index]: for y in range(1, ws.max_column+1): ws.cell(row=index+2, column=y).fill = fill except: pass try: wb.save(excelfile.filepath) wb.close() xl_file.close() return true except: wb.close() xl_file.close() return false figure 11. highlights rows with errors and/or failures with their corresponding hex color. since the failure and error dictionaries share the same keys as the color dictionaries, these can be referenced alongside one another to determine which color to highlight which record. the above function loops over each error and failure, selects the color, locates it in the workbook and fills the corresponding row. again, exception handling is of paramount importance – if the excel file is currently open openpyxl does not have the permissions required to edit it. in cases throughout the program, including this, tkinter windows are used to inform the user of errors since python’s error outputs are difficult to read for the untrained eye. figure 12. an unfortunate shift causing mismatched filenames has been caught and highlighted by the qc program. in some cases no user input is required – this is mainly the case for automatically fixed dates, which are not indicated with color on the spreadsheet, and automatic fails. to ensure consistency with the current workflow the program uses the “qc results”, “qc initials” and “qc comments” columns to indicate records that have failed with descriptive output messages and an auto flag to indicate this was an automated fail. similar output messages are used when the file size is too large or the pagecount is incorrect. the whole record for auto fails is highlighted red for immediate visibility after opening the spreadsheet. figure 13. two files with auto fails. one because the file cannot be located, the other for incorrect page count. results and evaluation in testing the quality control automation program’s first step, the application was run on a spreadsheet where the items had already been scanned. this immediately highlighted an issue where filenames were duplicated for different records. while this led to additional labor requirements to remediate the scanned batch, it may not have been caught in the upload because technically the filename was correct and had a scanned image but it would have been inaccurate for users of the repository. further, an unfortunate shift in cells, not obvious to the human eye, meant item n was saved with the filename for item n+1 which would cause issues when uploading items to archipelago. if the data type was inaccurate, like a date format in a field expecting a string, the upload would have failed. if the data type was acceptable, the upload would have carried on and the metadata would display in the wrong field or not at all, leading to inaccuracies and poor presentation. dates can be tricky in excel, even if entered correctly, if the format settings are off, excel will automatically change the date pattern. also if many people have different methods of recording the date and that can be an issue if metadata goes through many hands. the application was able to resolve 100% of dates that were in the incorrect format. while developing the program, james discovered that dates were usually in a predictable and readable format, hence the success rate, but not one that is useful for archipelago’s solr index. this application can reformat the dates to the iso 8601 standard automatically, so this feature reduces manual labor requirement for post processing. the second stage of the automation workflow highlighted a major issue. in the spreadsheet, 32% of items were marked as scanned but could not be located in the onedrive folder. evidently something went wrong in the digitization workflow and could mean scans are stored incorrectly in a different directory and as a result scanned multiple times unnecessarily. even though the solution to this issue requires manual intervention, this report still represents a massive time saving effort. prior to this, students were taking approximately 2 minutes to attempt to find a file, enter a fail, and a note in the spreadsheet for a supervising librarian to attempt the same check for 32% of the files. in the test file this equates to 131 failures, saving time and providing invaluable insight into how the qc process can be improved outside of this automation task. the program is able to identify files that are too large to be uploaded during this step. again, an error that requires manual intervention once discovered but eliminates the frustration of finding this out during the upload process. it also detects discrepancies in the spreadsheet’s extent column versus the scan’s actual number of pages. in one recorded case, this zeroed in on an item where a page had not uploaded correctly so the item needed rescanning. it may seem superfluous, but when the record is a pdf and remembering to check the number of pages while a document is open amongst the other checks, that item can fall by the wayside. table 1. preliminary errorrates table: representing 23 record failures across the spreadsheet. box failure rate 3 & 4 3.8% 5 8.3% 6 6.3% 7 2.8% the failure rate for the second step on the test data is 36.84%. this accounts for approximately 144 automatic fails and suggests there is a problem somewhere else in the workflow. the majority of these failures are due to missing files, but a handful are due to oversized files and incorrect page counts. conclusion the collaboration between the digital collections and preservation librarian and computer science major resulted in an application and outcomes that exceeded expectations. not only was a solid product developed to improve and expedite parts of the digitization workflow, but the process was a valuable experiential learning opportunity for the computer science student. the librarian brought the high level workflow needs and analysis and experience with the post digitization workflow issues, while the student provided more sophisticated programming skills and hands on digitization experience. through analysis the amount of human error in manual data entry is evident. of course, many aspects are better handled by people and it is a risk removing the human aspect of libraries by automating everything, but using it as a tool to check over manual work is highly valuable. it was noted that it is much easier to write software to do a job after having been trained on the manual workflow. often, computer programmers do not have extensive knowledge of the system requirements and therefore require a consultancy period. this method of development is an interesting model and can be considered in university libraries with cs students. this integration of the programmer into the original workflow highlighted the importance of fitting any new tool into the existing workflow and not trying to overhaul the entire process. this application is able to find things that are invisible to the human eye, and provide helpful statistics such as the 2-8% failure rate for filenames and duplicates and the 36% upload failure. it helps identify where errors are created with statistics to support it. the process ensures consistency and creates a common standard for quality control for every collection that passes through it. without, one wrong character could grind the process to a halt. the implementation of this application leaves room for further technological developments. other areas of quality control are being scrutinized for automation. this team is especially interested in some of the visual aspects of the checks like color balance, skewed images, and even cropping of images. the application itself is relatively new to the workflow but over time, schaffer library hopes to determine the real time cost savings in producing a system like this. it’s important to determine if the time taken to build the program was worth the quality control hours saved before more resources are put into expanding the application. references [1] federal agencies digital initiatives. 2023 may. technical guidelines for digitizing cultural heritage materials: third edition. u.s. national, archives and records administration. https://www.digitizationguidelines.gov [10] gamma e, helm r. 1994. design patterns: elements of reusable object-oriented software. 40th printed ed. boston: addison-wesley professional. [4][6] gazoni, e, clark, c. 2024. openpyxl – a python library to read/write excel 2010 xlsx/xlsm files; [accessed 2025 jan 9]. available from: https://openpyxl.readthedocs.io/en/stable/. [7] python software foundation. 2025. glob – unix style pathname pattern expansion; [accessed 2025 jan 9] available from: https://docs.python.org/3/library/glob.html. [5][8] python software foundation. 2025. graphical user interfaces with tk; [accessed 2025 jan 9] available from: https://docs.python.org/3/library/tk.html. [2] pino, d. 2024. archipelago commons intro. [accessed 2025 jan 9] available from: https://docs.archipelago.nyc/1.4.0/. [9] the qt company. 2023. pyqt5 reference guide; [accessed 2025 jan 9] available from: https://www.riverbankcomputing.com/static/docs/pyqt5/introduction.html. [3] twig team. 2025. twig | the flexible, fast, and secure template engine for php; [accessed 2025 jan 9] available from: https://twig.symfony.com/. notes the working code referenced in this article is available at: https://github.com/schaffer-library/qualitycontrolautomation about the authors corinne chatnik is the digital collections and preservation librarian at union college in schenectady, ny. she earned her mlis from the university of alabama. she was previously a professional archivist specializing in digital archiving at the new york state archives. author email: chatnikc@union.edu author url: https://orcid.org/0009-0004-7229-5431 james gaskell is a senior at union college, majoring in computer science and minoring in electrical engineering. his main areas of study are evolutionary algorithms and software verification. he is also currently a work-study student at union college’s schaffer library. author email: gaskellj@union.edu author url: https://orcid.org/0009-0002-9361-6172 subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – using static site generators for scholarly publications and open educational resources mission editorial committee process and structure code4lib issue 42, 2018-11-08 using static site generators for scholarly publications and open educational resources libraries that publish scholarly journals, conference proceedings, or open educational resources can use static site generators in their digital publishing workflows. northwestern university libraries is using jekyll and bookdown, two open source static site generators, for its digital publishing service. this article discusses motivations for experimenting with static site generators and walks through the process for using these technologies for two publications. by chris diaz introduction static site generators build websites from plain-text files. most are free to use and are available under an open source license [1]. they are often described in comparison to content management system (cms) software, like wordpress or drupal. cms websites use database processes on a web server to dynamically create html on demand. static site generators, however, perform all of the plain-text-to-html processing before the files are deployed online. this preprocessing workflow removes the need for high-touch system administration, database installations, server-side processing, and security patching, reducing the need for full-time developers and system administrators for digital publishing services. these advantages make static site hosting, maintenance, and preservation more affordable and sustainable for small teams. northwestern university libraries began using static site generators for our digital publishing service in 2018. we initially licensed the digital commons platform from bepress to support our open access publishing services, but the elsevier acquisition made us question our reliance on proprietary software and motivated us to consider open source alternatives (schonfeld 2018). at the same time, interest in open source software for library publishing was growing (library publishing coalition 2018). this article reflects on our use of two open source static site generators for library publishing, including an overview and evaluation of the technologies while focusing on two popular use cases: scholarly publications and open educational resources. why build static sites? static sites are gaining popularity in the digital library and cultural heritage community [2]. the getty museum is currently developing quire, a new monograph publishing tool that uses the hugo static site generator (the j. paul getty trust 2018). newson’s excellent “tools and workflows for collaborating on static website projects” covers how static website generators work, the benefits they offer, and a case study for a digital collections project (newson 2017). for digital publishing, the benefits of static sites include affordability, sustainability, and preservation. static sites are cheap to host and require very few computing resources. in addition to domain name services, dynamic sites require an operating system to manage the web server, the database, and all server-side scripting, often with significant software dependencies that require regular monitoring and maintenance. modern web browsers are reducing the need for server-side processing for rendering dynamic web content. static sites simply require storage and a content delivery network (cdn) from hosting providers. the requirements for specific operating systems, databases, software dependencies, or server-side scripting for are minimal or nonexistent for hosting static sites. in some cases, static sites can be hosted for free or very cheaply (e.g. $6.00 us – $30.00 us per year) by providers like github pages, netlify, or amazon web services (aws). static sites are generally smaller than dynamic sites. for one of our conference publications with over 30 presentations, the entire working directory is 230 mb (includes the full git version history, markdown, configuration, and sass files) and the output directory is 61 mb (which includes only the html/ css files and pdf full-text content). for comparison, some hosting providers suggest 1 gb of space needed to run one wordpress site, more than 10 times the needed disk space to host a static website (jackson 2018). static websites are easy to maintain because they are composed of asset, html, css, and javascript files. whereas dynamic websites are processed and built on-demand or cached on a server, static site generators process and build the site before the files are deployed online. the website doesn’t change until there’s a new deployment of the site. websites powered by a cms facilitate the exchange of information between the client and the server, such as user login credentials, which is stored in a web accessible database. both the database and the cms require system administrators to install and configure the software on the server, which require monitoring to ensure constant uptime, maintain software dependencies, handle regular security patches, and schedule server migrations. this is not the case for static sites. scholarly publications with jekyll jekyll [3] is an open source, command-line tool that transforms plain-text files into static sites. jekyll is very popular among the open source community. it is often the tool of choice for blogs, project marketing websites, and technical documentation due to its integration with github, which offers github users free static site hosting with its github pages service [4]. gitlab [5] and bitbucket [6] also provide free static site hosting. jekyll’s built-in toolset is well-suited to support scholarly publications, like journals and conference proceedings. both sx archipelagos [7] and the programming historian [8] use jekyll and github pages for hosting and publishing their peer-reviewed publications. we use jekyll to build conference proceedings websites for content we’re storing in our institutional repository. the static site serves as an lightweight presentation layer for a collection we manage in our institutional repository. this provides us a low-cost, highly customizable, simple website for our institutional stakeholders (diaz 2018). figure 1: how static websites enhance institutional repository collections content 3the full-text and metadata are formatted as markdown [9] and yaml [10] respectively. both markdown and yaml are stored in markdown files (.md), which can be opened by any plain-text editor. because static sites do not use content management systems, users interact with the plain-text files and folders directly. if you were to open a basic jekyll project for a scholarly journal, for example, the folder and file structure might look something like this: when you run the build command, the process takes all of the files in the scholarly journal file directory and outputs website files in subdirectory called _site. the _site directory contains everything you would need to deploy the site to the web. jekyll has built-in support for collections, which enables you to define content types, such as pages, articles, and posters (jekyll 2018). each collection will likely have its own designated html layout and default metadata values. for a journal project, you could create a subdirectory called _articles and keep all of the journal articles as markdown files. the markdown files include metadata and full-text content. the metadata appears at the top of the file as a key-value store represented as yaml, a field-driven markup language suitable for metadata. the full-text of the article is formatted as markdown. both yaml and markdown are human-readable, non-proprietary plain-text formats. from a preservation perspective, one could compress the markdown files into a .zip file at the time of publication and store them in a preservation or repository system. the _config.yml file contains all site-level configurations, such as the publication’s title, editor contact information, google analytics tracking code, and the website’s theme. at northwestern, we used jekyll and our institutional repository to create a conference proceedings publication for computational research day [11], an annual conference on data science and high-performance computing topics. we have made the source code [12] available for reference, however the style sheets are restricted to northwestern university websites and are not openly-licensed. layout layouts for jekyll websites are often set by a theme. there are hundreds of jekyll themes available on github [13] or rubygems.org [14] for free. themes allow users to ignore difficult design decisions requiring writing and editing html, css, and javascript. most themes are designed for users to plug into their jekyll projects without needing to edit any html, css, or javascript files, allowing content creators to focus on the full-text and metadata of their sites. however, because jekyll was built with blogs in mind, there may be instances when library publishers would want to edit the layouts of their publications to enhance the display metadata and match the user expectations of scholarly audiences. jekyll includes [15] is a method of modularizing html templates into component parts, sometimes referred to as partials, and can be useful additions to existing themes. these are small html files that are often limited to a specific html element in a page layout, like a search bar or navigation menu. these modules help keep the code better organized. for scholarly publications, jekyll includes are helpful when needing to call html elements on a conditional basis. for example, a scholarly publication might offer two or three creative commons license options for authors. with jekyll includes, you can create a template for displaying each of these licenses with their respective logo and govern their use with conditional logic based on the article’s yaml metadata. for scholarly publications specific to northwestern university, we use the sass [16] files and html pattern library created by the university’s marketing department to defer the layout and styling decisions. for non-northwestern specific scholarly publications, we experiment with open source css frameworks or themes that contain responsive layout elements and usability features that are important to stakeholders. metadata jekyll’s use of the liquid templating language makes reusing yaml data easy. liquid is a system of tags, objects, and filters that jekyll uses during the build process to load yaml and markdown content into specified html elements (shopify 2018). for articles, you can reuse the yaml front-matter for each journal article to create machine-readable metadata in the html header of each article’s web page to assist web search indexing and citation management systems. you could do this by creating an html file called metadata.html and assigning metadata values to designated html <meta> tags according to the scheme’s guidelines. here is an example of a few lines from a metadata.html file following google scholar recommendations [17] that pulls from yaml front-matter for each journal article and places it into the <head> element of the rendered html: the metadata.html could include as many additional metadata schemas as you would like, including dublincore and open graph. each schema has its own <meta> tag attribute but uses the same yaml front-matter from the articles. publishing workflow we partnered with northwestern information technology research computing department to publish the proceedings from computational research day, a conference they organize. research computing announced a call for posters, presentations, and visualizations and marketed them to various departments on campus. the also managed the review and editorial timeline for the publication. we used smartsheets [18] to create a web form and serve as a submission management system for receiving documents, capturing submission metadata, assigning reviewers, and tracking the editorial timeline. the web form including an option for submitters to participate in the proceedings publishing; publication was not required. to build out the website, i made html layouts based on northwestern’s html pattern library and sass files developed by the marketing department. i also set up a collection in our institutional repository to store the final versions of the poster and visualization submissions and mint dois. i used the metadata from the submission form to deposit the files into the repository and to manually create markdown files for each submission. we reused the dois created by the repository for each submission to display on the website in order to facilitate. to deploy the site, i used the s3_website gem [19] to deliver the static site files to a provisioned aws using s3 bucket [20]. while the files are stored in s3, they are delivered using cloudfront [21]. i requested a northwestern subdomain and launched the conference publication [11]. figure 2: screenshot of conference website, available at: http://crd.northwestern.edu/ we expect to maintain the website for the conference for up to three years after the conference has been discontinued. we chose to reuse the dois from the repository under the assumption that the active maintenance of the repository will outlast the availability of the static site. open educational resources bookdown bookdown [22] is an open source r [23] package that transforms plain-text into html, pdf, and epub. bookdown works with text formatted with either markdown or r markdown [24], a version of markdown that allows users to embed executable r code into markdown files. similar to jekyll, bookdown is a static site generator that renders content as online and downloadable e-books. it is popular within the r community of scholars and practitioners working in fields that use statistical computing. we worked with a faculty member in the statistics department to publish a set of modified teaching labs for an introductory statistics course that uses r. we chose bookdown for this project because it was a specifically designed for the content he was creating and the computing environment he was introducing to his students. content the book’s full-text content is stored in either markdown or r markdown files (.rmd). a bookdown project contains a working directory of plain-text files that looks something like this: the index.rmd file is the homepage of the static site and title page for the book. this is where the book’s cover image, preface, biographical information about the author and contributors, rights, and licensing information can be found. the chapters of the book are all r markdown files in the root directory of the project. if they’re not using r code snippets, the chapter files can be standard markdown. the bookdown rendering process reads the chapter files using a specific file naming convention for setting them in order. the first chapter begins with a 01, the second chapter begins with a 02, and so on. the r markdown files use standard markdown formatting with some syntax options that allows the files to reference data, execute r code, and highlight r syntax upon rendering html, epub, and pdf. code snippets in the bookdown package also includes support for python, c++, shell, sql, and a few others [25]. here’s an example of an r snippet in an r markdown file: layout the layout for the book can be specified either in the yaml front-matter of the index.rmd file, or as a separate yaml file called _output.yml stored in the book project’s root directory. this is where users can specify the output directory, such as the folder name set to contain the rendered html files, options for the toolbar or navigation, colors, or typography settings. like jekyll, there are options to take advantage of themes to handle the layout and styles of the rendered html. bookdown has built-in support for three styles: gitbook, bootstrap, and tufte. each of these styles have arguments to specify style options. users can override these styles by replacing css: style.css with css: custom.css in the _output.yml file to reference the custom style sheet. in addition to html, users can create settings for the epub and pdf outputs. these output files, like the site itself, are static files generated during the build process and will be downloadable by end-users through the user interface. the pdf generation is handled with latex commands using the pandoc software [26] and the epub document is created by a css file. metadata bookdown has built-in support for open graph and twitter card metadata for social sharing optimization. additionally, the front-matter in the index.rmd file contains some basic metadata, such as the book’s title, date, citation style, and basic description. it will also include several configuration options you can set with yaml front matter. here is the yaml contained at the top of our index.rmd file for the statistics oer we are building: like jekyll, bookdown has an includes method for adding html components to projects. this can be used add custom metadata to the html <head> element. within the _output.yml file, you could add an instruction to include a metadata.html file. the metadata.html file includes all of the <meta> tags and attributes according to the schema’s guidelines. you could use this same includes method to also add a google analytics script to monitor usage of the book by modifying the includes argument: in_header: [metadata.html, google_analytics.html] workflow we used a git-based workflow to collaboratively set up, edit, and publish the oer with github. i set up a clean working directory with rstudio, installed the bookdown package, which included all of the files needed to complete the plain-text processing, and initialized a new git repository to track changes and commit history. although rstudio supports git, we pushed commits to github with github desktop. we were  unable to figure out how to store ssh keys to circumvent the need for logging into to my github account each time we needed to push a commit in rstudio, otherwise we would have used its built-in git support. the faculty member i worked with had familiarity with markdown and r markdown, so he emailed me all of the r markdown files, which i then edited to fit bookdown styles we set out to follow. this required some basic content organization, such as removing redundant yaml metadata and licensing information. i also provided some basic copy edits for the text. once i was able to render the book as html, i published the git repository to northwestern university libraries’ organizational account on github and added the faculty member as a contributor with write-access to the repository. we were both able to push commits from edits to the book and preview the book in html using the github pages hosting service. in order to take advantage of github pages for hosting the book, this required some edits to the _bookdown.yml file. this file contains some basic settings, such as the ability to automatically label figures, tables, and chapter headings sequentially, and more importantly set the rendered book’s output directory. when you host your source code on github, you have a few options to publish static sites. this is intended to provide provide developers a way to create a static sites for documentation or marketing of the open source package. common methods for doing this includes pushing your static site contents (i.e. html and assets) to a gh-pages branch or a folder called docs. we chose to designate a docs folder in the _bookdown.yml file, instead of the default _book folder so that github knows to create a website with those files. a final step to make this work was to add an empty file called .nojekyll to tell github servers that the files are ready to be served without the need to run any jekyll build processes. figure 3: screenshot of published oer, available at: https://nulib.github.io/kuyper-stat202/ the book is currently a static site hosted by github [27] and allows users to download the epub and pdf versions of the book. future bookdown projects will likely involve more customizations to the book’s style, better integrations with data repositories (i.e. storing datasets in our institutional repository and writing r scripts that interact with those data programmatically), and custom domains. we have also made the book’s source code [28] available. moving from a concept to a service our use of static site generators for digital publishing projects was a success, but scaling this concept into a service that supports distributed editorial teams would present a number of challenges whose solutions might be the topic of a future article. the websites we build serve primarily as public user interfaces for the content we publish; they are not all-in-one editorial and publishing systems. the build processes involved in building the static websites can automate the conversion of markdown to publishing output formats, such as html, pdf, and epub; however, static site generators do not support the initial document preparation, submission management, editorial workflows, or any other backend functionality available in other digital publishing systems. those functions could be handled by software-as-a-service extensions or outside of the website infrastructure entirely. in both jekyll and bookdown, the submission document needs to be converted to markdown in order to be added to the site. this would require some human intervention to ensure that the conversion is successful, and likely the use of a computer program to convert from microsoft word[29] or google docs[30]. in the above case studies, the markdown files for both projects were created by copying and pasting information from spreadsheets or supplied by the author. the planning phase for a scalable workflow would involve decisions around which technologies and services will perform specific functions in the workflow, such as:   publishing function description technologies and services document preparation preparing ms word or google doc submissions to markdown, html, pdf, or jats outputs; copyediting –        pandoc [31] –        context [32] –        freelance copy editors or typesetters –        student workers submission management accepting full-text and metadata submissions from authors and contributors –        smartsheets [18] –        scholastica [33] –        submittable [34] –        box.com [35] –        dropbox [36] –        google drive [37] analytics logging and analyzing page visits, user demographics, and downloads –        google analytics [38] –        matomo [39] archiving depositing content for long-term storage and preservation –        institutional repository –        digital asset management system –        local or cloud-based file storage system registration creating unique identifiers for published content (e.g. doi, issn, isbn) –        cross ref [40] –        datacite [41] –        library of congress [42] –        bowker [43] project management tracking activities, task assignments, and timelines for the editorial team –        smartsheets [18] –        trello [44] –        github project boards [45] –        google drive [46] –        ms excel [47] examples of editorial and publishing functions needed in addition to static site generators  a selection of these editorial and publishing tasks would need to be stitched together with documentation and executed using a several pieces of software, some of which may be difficult for one or more persons to design and a rotating team of editors to manage. this is part of what makes all-in-one publishing systems attractive. and then there’s the issue of deployment. because static sites are generated before they are deployed, all of the content management of the site happens on local copies of the plain-text files on someone’s computer. if an editor needed to update a webpage or publish a new issue, they would either need to understand the static site generator workflow or coordinate with the publisher via email. it would be highly desirable in this case to install a headless cms to push updates to the static site. there are numerous headless cms options [48] that can be added to provide editors a friendly user interface for editing the site. the configuration would involve connecting the cms to the static site’s github repository and using a continuous integration and delivery service to auto-merge and deploy the updates. otherwise, all of content editing and updating would be handled by the people who understand the workflow of the static site generator, have all of the necessary software available to them on their computers, and have the access rights to deploy the changes online. at northwestern, this problem has not yet been a deterrent for us to continue to use static site technologies because we have centralized digital publishing within a single position who is comfortable using git, static site generators, github, and aws resources. conclusion static site generators are great for creating simple, powerful, low-cost websites for scholarly or educational publications, but they cannot alone provide the all-in-one system functionality for scholarly publishing. rather, these open source tools can used as components of the “open scholarly commons” as described in lewis’s “the 2.5% commitment,” which called on the colleges and universities to contribute to a shared infrastructure of open source technologies to facilitate scholarly communications (lewis 2017). the elsevier acquisition of bepress exposed the extent to which the over-reliance on monolithic publishing and repository systems by library-based open access publishers is a problem. using open source static site generators for digital publishing is one option for avoiding futute lock-in scenarios and supporting values of openness to research, education, and technology. at northwestern, using static site generators enabled us to provide nicer online publications to our stakeholders and support new use cases for library-based publishing than we were able to with the bepress platform. for us, the downside of utilizing microservices for digital publishing is managing a complex set of technical components with varying levels of required skills and documentation within a single position; however, we believe that the upside is much greater. we can easily move away from jekyll or bookdown if we need to. this platform-agnostic approach allows us the flexibility to experiment with new publishing technologies and the ability to provide services that prioritize the needs of our stakeholders over the limitations of any particular platform. static sites are a key component to this approach. about the author chris diaz is the digital publishing librarian at northwestern university. he has held positions in academic libraries focusing on scholarly communication, collections management, and outreach since 2013. his previous publications include two edited volumes for the alcts monographs series on college textbook affordability and library collections. he is currently focused on using open source technologies for web publishing and digital libraries. you can find him on twitter @chrisdaaz or reach him by email at chris-diaz@northwestern.edu. notes [1] https://www.staticgen.com/ [2] https://github.com/hardyoyo/awesome-static-digital-libraries [3] https://jekyllrb.com/ [4] https://pages.github.com/ [5] https://about.gitlab.com/features/pages/ [6] https://confluence.atlassian.com/bitbucket/publishing-a-website-on-bitbucket-cloud-221449776.html [7] http://smallaxe.net/sxarchipelagos/index.html [8] https://programminghistorian.org/ [9] https://www.markdownguide.org/ [10] https://learnxinyminutes.com/docs/yaml/ [11] http://crd.northwestern.edu/ [12] https://github.com/nulib/computational-research-day [13] https://github.com/search?q=jekyll+theme&type=repositories [14] https://rubygems.org/search?utf8=%e2%9c%93&query=jekyll-theme [15] https://jekyllrb.com/docs/includes/ [16] https://sass-lang.com/ [17] https://scholar.google.com/intl/en/scholar/inclusion.html#indexing [18] https://www.smartsheet.com/ [19] https://github.com/laurilehmijoki/s3_website [20] https://aws.amazon.com/s3/ [21] https://aws.amazon.com/cloudfront/ [22] https://bookdown.org/ [23] https://www.r-project.org/ [24] https://rmarkdown.rstudio.com/ [25] https://bookdown.org/yihui/rmarkdown/language-engines.html [26] https://bookdown.org/yihui/bookdown/latexpdf.html [27] https://nulib.github.io/kuyper-stat202/ [28] https://github.com/nulib/kuyper-stat202 [29] https://gist.github.com/vzvenyach/7278543 [30] https://word-to-markdown.herokuapp.com/ [31] https://pandoc.org/ [32] http://wiki.contextgarden.net/what_is_context [33] https://scholasticahq.com/ [34] https://www.submittable.com/ [35] http://box.com/ [36] https://www.dropbox.com/ [37] https://www.google.com/drive/ [38] https://analytics.google.com/analytics/web/ [39] https://matomo.org/ [40] https://www.crossref.org/ [41] https://www.datacite.org/ [42] https://www.loc.gov/issn/ [43] https://www.isbn.org/ [44] https://trello.com/ [45] https://help.github.com/articles/about-project-boards/ [46] https://www.google.com/drive/ [47] https://products.office.com/excel [48] https://headlesscms.org/ references staticgen: a list of static site generators for jamstack sites. available from: https://www.staticgen.com/ shopify. (2018). liquid basics. available from: https://help.shopify.com/en/themes/liquid/basics diaz, c. 2018. jekyll and institutional repositories. open repositories 2018. available from: https://doi.org/10.21985/n28x22 jackson, b. [updated 2018 august 16]. how much disk space does your hosting plan really need? [internet] kinsta.com. available from: https://kinsta.com/blog/disk-space-wordpress-hosting/ jekyll. 2018. collections. jekyllrb.com [internet]. available from: https://jekyllrb.com/docs/collections/ lewis, d. 2017. the 2.5% commitment [internet]. available from: http://doi.org/10.7912/c2jd29 library publishing coalition. 2018. owned by the academy: a preconference on open source publishing software [internet]. available from: https://librarypublishing.org/owned-by-the-academy-preconference/ newson, k. 2017. tools and workflows for collaborating on static website projects. code4lib journal [internet]. [cited 2018 september 3]. available from: https://journal.code4lib.org/articles/12779 schonfeld, r.c. 2017. elsevier acquires bepress. the scholarly kitchen [internet]. [cited 2018 september 3]. available from: https://scholarlykitchen.sspnet.org/2017/08/02/elsevier-acquires-bepress/ the j. paul getty trust. 2018. quire: a new publishing tool [internet]. available from: http://www.getty.edu/publications/digital/platforms-tools.html xie, y. 2018. bookdown: authoring books and technical documents with r markdown [internet]. crc press. [cited 2018 september 3]. available from: https://bookdown.org/yihui/bookdown/ subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – closing the gap between fair data repositories and hierarchical data formats mission editorial committee process and structure code4lib issue 52, 2021-09-22 closing the gap between fair data repositories and hierarchical data formats many in the scientific community, particularly in publicly funded research, are pushing to adhere to more accessible data standards to maximize the findability, accessibility, interoperability, and reusability (fair) of scientific data, especially with the growing prevalence of machine learning augmented research. online fair data repositories, such as the open science framework (osf), help facilitate the adoption of these standards by providing frameworks for storage, access, search, apis, and other features that create organized hubs of scientific data. however, the wider acceptance of such repositories is hindered by the lack of support of hierarchical data formats, such as technical data management streaming (tdms) and hierarchical data format 5 (hdf5), that many researchers rely on to organize their datasets. various tools and strategies should be used to allow hierarchical data formats, fair data repositories, and scientific organizations to work more seamlessly together. a pilot project at los alamos national laboratory (lanl) addresses the disconnect between them by integrating the osf fair data repository with hierarchical data renderers, extending support for additional file types in their framework. the multifaceted interactive renderer displays a tree of metadata alongside a table and plot of the data channels in the file. this allows users to quickly and efficiently load large and complex data files directly in the osf webapp. users who are browsing files can quickly and intuitively see the files in the way they or their colleagues structured the hierarchical form and immediately grasp their contents. this solution helps bridge the gap between hierarchical data storage techniques and fair data repositories, making both of them more viable options for scientific institutions like lanl which have been put off by the lack of integration between them. by connor b. bailey, fedor f. balakirev, and lyudmila l. balakireva introduction with the increased prevalence of machine learning technology, the viability of big data projects, and the rapid proliferation of vast quantities of information, data are becoming more usable and useful than ever. however, these datasets must be properly stored and managed to ensure that the greatest possible scientific output can be gained from them. to achieve these goals, there is a push to use more accessible data standards to maximize the findability, accessibility, interoperability, and reusability (fair) of scientific data (wilkinson 2016). open data repositories help with the adoption and usage of the fair principles by providing storage for scientific data repositories and act as hubs for sharing such information. tdms [1] and hdf5 [2] are hierarchical data formats that are commonly used in scientific communities, are well supported by and compatible with popular scientific software, and offer a wide variety of benefits over alternative data storage methods. the hierarchical design allows for many different ways of structuring and organizing data, including large and complex datasets, while also allowing metadata to be included at various levels in the hierarchy, which can help explain the structure, contents, and format of the data. having metadata embedded within the hierarchy of the file can improve its reusability by providing inseparable, detailed information about the various parts and contents of the file. the adoption of fair data practices can be greatly enhanced by hierarchical data storage formats as there are several overlaps between the benefits and use cases of each tool. we found, however, a disconnect between hierarchical data formats and open data repositories and control systems that could be addressed. the advantages of hierarchical data storage formats there are many reasons to use a hierarchical data storage method and many advantages if that storage type is used. these storage types allow for vast quantities of data to be organized efficiently to store arrays of data and the associated metadata that contextualize them. figure 1 tdms sample file structure: tdms files consist of a three level hierarchy: file, group, and channel and can contain two types of data: property metadata and data values. every tdms object can have an unlimited number of properties consisting of a combination of a name (always a string), a type identifier, and a value. channel objects can contain raw data multi-dimensional arrays. hierarchical data storage formats like tdms and hdf5 allow the user to organize data arrays into groups and trees, separating out different trials or conditions, and create any grouping that helps the data make more sense. additionally, important information such as the parameters that correspond to that dataset can be included at the same level as the data embedded within the hierarchy of the storage method. important information such as how the data should be read can also be included at this level; for instance, waveform or time interval data that is integral to the understanding of the data channel. because hierarchical formats work well for time series data, it is a useful format for signal processing, network monitoring, finance, weather forecasting, and largely in any domain of applied science and engineering. the advancement of fair data principles the concept of fair was introduced in “the fair guiding principles for scientific data management and stewardship” by wilkinson et al. in 2016 and is seen as a consolidation of good data stewardship practices (jacobsen et al. 2020). since then these guidelines have seen success in impacting institutions by pushing them towards using better management methods for their data. built into the acronym are the four core goals of fair that institutions should strive for with their data: findability, accessibility, interoperability, and reuse. each has significant importance; however, they all aim to reduce the chance that some valuable data can be lost to future scientific endeavors due to poor data management. each goal is also further detailed in the guidelines and applies to any digital object being generated or stored within a repository. the fair principles do not dictate any kind of specific software or service, however many data repositories exist which can help institutions to adopt the principles (wilkinson et al. 2016). open science framework (osf) is one such data repository that seeks to facilitate open collaboration in scientific research run by the center for open science (cos), a non-profit technology organization. researchers can use osf to research and find data related to their work, and they can use it to host the data they produce themselves. these types of repositories seek to be a hub for data management, helping to facilitate the exchange of valuable scientific data between researchers. an important part of fair is the metadata associated with data sets. metadata gives important information about data; identifying it, contextualizing it, and ensuring that the full meaning of the data can be understood. the globally unique, persistent identifier that is required by the fair guidelines, for example, is an extremely important bit of metadata, and references to other identifiers are useful too. many of the ideas in the fair principles relate to “machine-actionability”, however there is an important human interactivity element to fair which we focus on with our tool here. we found that fair data repositories, even the most well-known, like harvard dataverse [3] or osf [4], lack the support of tdms and hdf file viewing. this deters scientists from engaging with fair practices like publishing datasets for reuse in the open. the repositories store hierarchical format as a binary blob, so users can not see the experiment content of the file or associated metadata. this creates the problem that even if one researcher posts the tdms file to the repository those files are not browsable or findable by others in the scientific community. we created a renderer tool to address the deficiency. many files on osf can be viewed directly in the browser while looking through the repository. however, many types of files may display as plaintext, spreadsheet, or pdf style renders which may not be able to show the full complexity of data. many files such as tdms and hdf5 do not display at all. a rich view of data and metadata would allow a researcher to get a more complete understanding of the data they are viewing directly in the data repository. the hierarchical renderer for fair repositories osf [5] is an open source project and the rendering is done by a tool called the modular file renderer (mfr) [6]. the mfr is made up of “handlers” which handle web requests made to the mfr, “providers” which know how to take a url for a file and retrieve the necessary content for the file, and “extensions” which generate the necessary html for the rendering of the file. osf allows us to integrate new “extensions” into the existing mfr framework which are called to handle the rendering of hierarchical data files. we have consulted with the staff at cos [7] so that our renderer will be added to the main osf library and be usable by all users of osf. using the nptdms [8] and h5py [9] libraries in python, the renderer extracts the data from the hierarchical data files into the python environment and generates the necessary html, javascript, and css content to display the rendering in the browser. in order to create this tool, we created a local development environment and ran a local host of osf in a docker instance. [10] the same steps should be taken to test the tool before it has been implemented in the main osf distribution. using these tools we were able to fork the mfr, develop hierarchical data extensions, and test its functionality locally. the main difficulties arising from a development project like this come from getting the wide variety of disparate software and libraries to properly work together. the renderer tool [11] that we have created allows users to fully render a view of a hierarchical data file directly in a fair data repository. specifically, we can render tdms and hdf5 files in the osf file viewer. this allows researchers and others using osf to store their tdms and hdf5 files in that repository knowing that it will have full support when browsing their files in the future. our renderer builds off of tools such as the ni tdms viewer, a labview program that loads and displays tdms files within a labview program. our renderer gives users the same information directly in the browser while interacting with the files in the osf repository. it is able to display the metadata alongside a table of the data and a plot of the data, visualizing the logical link between elements of data and metadata in a composite view. this allows us to quickly render the necessary information for someone who is browsing through these types of files. figure 2 tdms renderer rendering a sample file: the renderer presents the file content in a split view with the metadata and properties values being shown in tree and table formats on the left and the channel data being displayed in a plot and table format on the right. the metadata tree is expandable, collapsible, and scrollable so that the necessary information is as easy to access as possible. the renderer view is split vertically between the tree style metadata display on the left and the data plot and table on the right (fig. 2). on the left, in an intuitive hierarchical tree structure, the file level metadata is displayed at the top followed by the group and channel metadata and properties table beneath it. each level is expandable, collapsible, and scrollable to allow easy access to whatever level the user is interested in looking at more closely. on the right, an image of the channel data arrays plotted and exported using the matplotlib python library [12] is displayed. this is a quick and efficient way of rendering a plot of the data so that a user can see at a glance what the structure of the data looks like. because the renderer isn’t designed for in depth data analysis, we found this method to be the best way to achieve fast rendering speed while also giving a visual indication of the files contents. a chunk of the data is also included in a data table beneath the plot. the table loads only the first 100 entries in a channel, again to allow for speedy load times, while still giving the user the ability to see what the channel fields of the file are and what the data they contain looks like. figure 3 tdms renderer within the osf file viewer: the view a user gets when accessing a tdms file within the web browser accessing the osf repository. the osf toolbar is displayed at the top of the screen, the file is displayed in a toolbar on the left side of the screen and the main rendering view is shown in the right center of the screen. selecting a tdms file from the file menu on the left calls on our rendering tool to render it in the webpage. the person viewing these files is likely to want to get a general idea of the file they are looking at directly in the osf data repository. they can then identify files they may want to work with further and download those specific files to do more in depth analysis or calculations with. conclusion our rendering tool seeks to target a direct need in the scientific community not only to give researchers better tools for accessing and viewing their data, but to ease the adoption of better methods of data management. our solution aids in bridging the gap between hierarchical data storage techniques and fair data repositories, making both of them more viable options. by bringing this user-friendly tool to a popular fair data repository, we hope that we have helped bridge that gap. scientific institutions like lanl which have been put off by the lack of integration between them now know they can use these tools together and use them more efficiently and effectively. acknowledgments the national high magnetic field laboratory is supported by the national science foundation through nsf/dmr-1644779, the state of florida, and the u.s. department of energy. references wilkinson, m. dumontier, i. aalbersberg, g. appleton, m. axton, a. baak, et al. 2016. the fair guiding principles for scientific data management and stewardship. scientific data 3(mar). jacobsen, r. demiranda, azevedo, n.juty, d.batista, s.coles,r.cornet, et al. 2020. fair principles: interpretations and implementation considerations. data intelligence 2(1-2):10–29. endnotes [1] tdms file format description and documentation from national instruments: https://www.ni.com/en-us/support/documentation/supplemental/06/the-ni-tdms-file-format.html [2] hdf5 file format description and documentation from hdf group: https://portal.hdfgroup.org/display/hdf5/hdf5 [3] harvard dataverse repository: https://dataverse.harvard.edu [4] open science framework repository: https://osf.io [5] open source development of osf.io, the osf repository on github: https://github.com/centerforopenscience/osf.io [6] open source development of the modular file renderer for osf on github: https://github.com/centerforopenscience/modular-file-renderer [7] acknowledgements to fitz elliot at cos for consultation on osf and mfr [8] documentation for the nptdms library that provides interfacing between tdms and python: https://nptdms.readthedocs.io/en/stable/index.html [9] documentation for the h5py library that provides interfacing between hdf5 and python: https://www.h5py.org [10] guide to setting up a local development environment for osf allowing you to run the extended version including our renderer until it is published on the main osf repository: https://github.com/cbb-lanl/modular-file-renderer/blob/develop/configuringdevenv.md [11] fork of the modular file renderer containing our additional extensions in the mfr/extensions folder: https://github.com/cbb-lanl/modular-file-renderer [12] documentation for the matplotlib library which provides plotting functions in python: https://matplotlib.org about the authors connor bailey (cbbcbail@gmail.com) is a graduate student at the pulsed field facility, national high magnetic field laboratory, los alamos national laboratory, usa (maglab). connor works on simulation and visualization strategies for radio transmissions and develops automated data capture, processing, and software programming tools at the maglab user facility. fedor balakirev (fedor@lanl.gov) is a staff scientist at the pulsed field facility, national high magnetic field laboratory, los alamos national laboratory, usa. the focus of his research is studies of materials under extreme conditions of high magnetic fields, including the development of the measurement infrastructure, instrumentation, and computing suites for the maglab. fedor holds a ph.d. in physics from rutgers university, usa. lyudmila balakireva (ludab@lanl.gov) is a research engineer of the prototyping team at the los alamos national laboratory research library. in this role, she focuses on research and development efforts in the realm of web archiving, scholarly communication, digital system interoperability, repositories, and data management. lyudmila holds a ph.d. in mathematics and physics from moscow institute of physics and technology, russia. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – conference report: code4lib 2009 mission editorial committee process and structure code4lib issue 6, 2009-03-30 conference report: code4lib 2009 conference reports from the 4th code4lib conference, held in providence, ri from february 23 to 26, 2009. the code4lib conference is a collective volunteer effort of the informal code4lib community of library technologists. included are four brief reports on the conference from the recipients of conference scholarships. by jie chen, joanna dipasquale, lauren ko, and andreas orphanides. the 4th code4lib conference was held in providence, ri, from february 23 to 26, 2009.  as in years past, the code4lib community was able to offer scholarships focused on gender diversity and minority representation, this year sponsored by oregon state university and brown university.  following are conference reports from the scholarship recipients. for more notes, many informal, by other code4lib community members, find content on the internet tagged “c4l09” or “#c4l09”, a label attendees were encouraged to use to aid collocation. for example: http://delicious.com/tag/c4l09 from jie chen as an ils system administrator, i did not come to this conference with a goal to improve any programming skills. instead, i wanted to have a better understanding of the thriving and energetic code4lib community and find out what’s the latest with open souce library technologies. i was very impressed by what i saw. there’s so much more than mind blowing coding discussions and demos, as code4lib offers valuable dialogues of where the libraries stand today and where we need to go. while i enjoyed all sessions, i found the 3 keynote speeches particularly interesting. a recurrent theme in all three seems to be an emphasis on data openness, which goes beyond simply making use of open source software and tools for libraries. stefano mazzocchi talked about the close to zero marginal cost of communication, and how it’s changing the role of libraries. his demo of freebase showed how collective common knowledge contributed by users can be translated into tremendous value. sebastian hammer pointed out that even though standards suck, they are central in allowing data to flow freely and are essential for collaboration; hence a call for systems and organizations to surrender our data. ian davis talked about the importance of open data. and i thought he put it very well when he said the goal is not to build a web of data, but to enrich people’s lives through access to information. a spirited game of ‘werewolf’ during conference down time. i loved the conference format — 20 minute sessions, single track and the favorite of many people including me: 5 minute lightening talks. jodi schneider and william denton gave a great discussion on the frbr model and demanded strong frbrization in library applications, which made the long-lost cataloger in me want to stand up and applaud. for the same reason, i liked the demo of the open source web based metadata editor ‡biblios.net by liblime. godmar back introduced the libx 2.0 community platform that allows sharing and building upon existing libx services. i was really excited to see that librarians will get to play a role in this platform — non-programmers like us could be adopters who combine modules into libapps, and reuse and share them as packages. attending his pre-conference workshop allowed me to see the editing of libapps in action. in addition to beer and wine, the jokes in the irc backchannel, and the addictive werewolf game, this conference offered me an insight into programmers’ view of the library world, and it has been really fascinating. since coders create applications to handle and process data, i think it gives them a very keen sense of how data could be more efficiently shared, linked and retrieved. just like what ian said at his keynote, because there is more structured data than unstructured data, therefore people who understand structures matter. i walked away from code4lib 2009 with a deeper understanding and appreciation of developers and coders’ contribution to the library community. from joanna dipasquale the themes of interoperability, portability, scalability, cooperation, and, of course, change were on display at the fourth annual code4lib conference in providence, ri. it was an excellent opportunity for library technologists to come together and showcase their innovative software and brainstorm new ideas. but there was much more, for the conference’s messages were both simple and powerful: this community wants to design to interoperate, it wants to share, it wants to create things that make life easier for library users. perhaps sebastian hammer, in his keynote address on wednesday, february 25 said it best: the assumption that the marketplace changes libraries inexorably – where our choices are only to get out of the way or adapt – is coming into serious question as market forces shift.* code4lib provides some of the innovation needed to not just be part of the larger conversation of the “next phase” of libraries, but to lead the conversation. it wants the library to win, and it is doing something about it. the conference provided a wide array of insights into the current and next great library applications. the range of projects was amazing: from geospatial data, to information visualization and dashboard styles, to metadata standards, to more effective searching and indexing, code4lib showed the innovation of library and archive technology (see program, breakout sessions, and lighting talks for more information). through the presentations and breakout sessions, the themes above provided us with two main goals, “what we can do right now” and “what we need to do for the future.” the desire to provide ways to interoperate with already-on-hand systems – from blacklight’s plugin-based customization files for local instances, to the improvements in vendor tools and apis from oclc, ex libris, and serials solutions – gave viable solutions for the present. the push for better standards – from ead to semantic web to sword techniques – provided insight to current and future solutions. from the provocative stefano mazzocchi challenging us to embrace electronic books, to the experimentations with ead, djatoka, and dashboard views, challenges become opportunities. yet, as ian davis stated in his keynote address, code may change, but data remains. our standards and interoperability are key; as we learn more ways to expose our library data to the greater community, we also know that we’ll find better solutions in the future to work with our repositories and catalogs. the conference met these challenges head-on by providing the techniques and the thinking behind them to enable all of us to do more. — * if you attended code4lib 2008, you probably thought i was going to write, “perhaps hammer said it best when he noted: ‘i’m not much of a keynote guy. i always thought that, if you have something to say, you should release it as code.'” but i didn’t (until now). it was an excellent thing to say, but the argument he proceeded to develop was much more powerful than that quip. from lauren ko: bacon, kittens, and brewpubs pre-conference my experience at code4lib 2009 began with the linked data pre-conference. speakers focused on the use of rdf for describing and sharing data to enable the linking of resources via the crawling of uris. the pre-conference was great, not only because my foaf file won me a book, but because its focus on enabling the sharing and connecting of data was a fitting indication of what would come at the main sessions. opening remarks the conference began surreally with roy tennant, whose words i used to read monthly in library journal, showing off his face on a pair of thong underwear. despite this image, his words were significant as he encouraged all attendees to take part in the conference beyond the role of viewer. mark matienzo followed him on stage with a presentation on how to have fun at code4lib 2009. his recommendations included the irc channel, some sort of bacon-donut (plus food in general), and of course, beer. keynotes the idea and implications of opening up data to the web pervaded the three keynote addresses of the conference. first keynote speaker stefano mazzocchi began with a history of information technology from cave paintings to electronic publishing. he spoke about the cost ineffectiveness of libraries keeping physical copies of every book that could be improved by institutions moving to the virtually infinite storage space of servers. he then showed us freebase as an example of marked up data that can be utilized by any number of to-be-developed applications to enrich the lives of users. sebastian hammer, wednesday’s keynote speaker, issued a call to arms. he implored all library coders to become advocates within their organizations and prevent a death of libraries brought about by dependence on the idea of the book. he also spoke of the problem of apis forcing loyalty above interoperability and the need to surrender data freely to help remedy such problems. ian davis, the final keynote speaker, continued the push for libraries to set their data free and contribute to building a more useful semantic web. because data outlives code, it is more important for institutions to push for open data over open source. the rest of it other sessions addressed various areas within the realm of libraries. we heard about indexing collections (solr, lusql), tools for facilitating access to library resources (libx, jangle), and content management systems (drupal, vufind, blacklight). being new to the code4lib community and having limited knowledge of some of its topics, i was most satisfied with speakers who began with a general overview of their topic, followed with further description, shared a demo, and concluded with what it means to the community. particularly fun was richard wallis’s presentation (complete with sound effects) on using the javascript based framework, juice, to extend opacs through embedding related external content (http://www.slideshare.net/rjw/squeezing-more-from-the-opac). the conference facilitated an amazing amount of participation/contribution with its breakout sessions and lightning talks. the breakout sessions allowed attendees to informally discuss current projects, issues, and recommendations. the lightning talks, covering a variety of topics (applications, specifications, protocols, etc.), introduced me to new ideas, resources, and projects for further investigation. my experience at the code4lib conference was overwhelmingly positive. while i don’t pretend to understand the fascination/running joke of bacon and kitten images or the love of brewpubs that permeates, i gained a great respect for the community of programmers and librarians that is working together on ways of enriching the lives of its users. from andreas orphanides: code4lib 2009 with a critical eye one of the things that struck me about my first code4lib conference is the level of bright-eyed idealism that the presenters, and the participants, brought to the conference’s discussions. and certainly, it is a right and proper thing for us code4libbers to keep focused on the horizon as we think about technology, data, and the future of libraries. it’s important to remember, though, that while optimism is all well and good, we must also be mindful of the practical problems we may encounter in implementation. with that in mind, i present a few points of critical analysis of the presentations at code4lib 2009–places where we should pause to think about how the ideal will fit into a real, and very sloppy, world. the linked data preconference was both an exploration of the possibilities of decentralized, interconnected data repositories and a rallying cry for developers to make use of those repositories. linked data as a concept is also one of the areas where reality is likely to end up in conflict with an ideal system. fundamentally, linked data relies on high-quality metadata creation and robust linking technology. the presenters acknowledged the problems inherent in linking, such as expiring uris, and suggested as a solution locally caching linked data. this kind of workaround, however, runs right up against the problem of quality metadata creation. what happens as cached metadata ages, or false or incorrect metadata propagates through the system? a decentralized system will either have to accommodate conflicting metadata, which could have nasty side effects, or it will need a heuristic to resolve the conflict, a notoriously difficult prospect. it seems to me that the only viable alternative would be to cede some of the ad-hoc, democratic nature of this system to establish an authority control system of some kind, sacrificing some of the decentralization that make the linked data concept so appealing. consider also the freebase project, and its genderizer tool. stefano mazzocchi, in his keynote, demonstrated freebase, an open-access database that hopes to catalog, well, everything about everything. he also demonstrated the genderizer tool, a web application where users vote on the gender to objects in the freebase database. stefano indicated that a possible goal of the genderizer would be able to determine the gender of every relevant (i.e., “genderizable”) object for which information exists on the internet, something on the order of four billion objects. this presents an immense problem of scale: the genderizer assigns gender to approximately 150,000 objects per month; at the current rate, it would take over 2,200 years to reach that goal. i won’t even address the problems inherent in assigning genders through voting or the potential for dissonance between someone’s self-identified gender and its majority-assigned one. there’s the potential for similar problems in assigning properties in this way to any large collection of objects. although i’ve pointed out flaws in these two projects, each one has incredible promise, and each will doubtless serve as the foundation for a host of next-generation information tools. and while i don’t have recommendations for how to address the flaws, i think it’s important to acknowledge them, and to think about potential solutions as we explore the possibilities these tools offer. this same earnest, realistic perspective–balancing the ideal of a perfectly crafted design against a reality where implementation is far from perfect–must be maintained if we are to build technologies that are relevant, practical, and practicable, and which serve as the platform for real-world applications that are useful to the information consumers that we serve. about the authors yu-jie chen is the ils librarian at loudoun county public library in leesburg, virginia. joanna dipasquale is a web developer for columbia university libraries’ digital program division. lauren ko has a bachelor’s degree in computer science and a master’s in information sciences. she is currently web archiving programmer for the university of north texas libraries digital projects unit. andreas orphanides has a ba in mathematics from oberlin college, and after a stint as a high school mathematics teacher, he earned his msls from the unc-chapel hill school of information and library science. he is currently a libraries fellow at the north carolina state university libraries, where he works in the libraries’ information technology and research & information services departments. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – editorial introduction: seeking a diversity of voices mission editorial committee process and structure code4lib issue 24, 2014-04-16 editorial introduction: seeking a diversity of voices making the journal the best that it can be. by ron peterson the editorial committee of the code4lib journal is always looking for ways to improve the content of the journal and ensure that it is meeting the needs of our readers. new additions to the committee have brought new ideas about how we can make the journal the best that it can be. when several members of the committee met in march 2014 during the code4lib conference in raleigh, we had the opportunity to discuss these issues in person. inspired by the presentations on data visualization, the editorial committee talked about how we can use the data that we have to help chart the future directions of the journal. we have google analytics data and our own administrative data that we use to track proposals from submission to publication that we could evaluate, but we can go further. we could mine the mailing list or use other kinds of alt-metrics in order to learn more about the impact of the journal, where we are succeeding and where we could do better. some of the questions we would like to explore include the following: what are some of our most popular articles? topics? what is our rejection rate? how does our rate compare with other journals in the field? is there a “season” for proposals? are there regular cycles with peak times for the submission of proposals? how many repeat authors have we had? repeat submitters? one question that i wanted to explore was “how inclusive of different populations have we been?” diversity of voices a theme that emerged at the code4lib conference was the rate of participation for women in technology. valerie aurora, the closing keynote speaker, advocated for a conscious improvement of the diversity of voices that we are listening to.[1] with that in mind, i did a preliminary analysis of gender participation in the code4lib journal. we don’t formally track demographic data for the authors who submit proposals, so those numbers involve a certain amount of guessing about the gender of the author. also, without that demographic data it is difficult to dig deeper to look at other characteristics. diversity of editorial committee members the best data we have is on the editorial committee itself. of the twenty-nine people who have ever served on the committee, only eight of them are women. twenty-one of the editors are men. here is a graph showing the makeup of the committee over time: while we have been mindful of recruiting women to join the editorial committee, this graph demonstrates we haven’t been trying hard enough. diversity of authors out of the 201 articles that the journal has published, 95 articles had female authors and 160 articles with male authors – in some cases the article may have had both male and female authors. in all, the articles published by the journal have been written by 247 male and 142 female authors. women make up less than 40% of the authors published in the journal. we should be able to find more female authors in a profession that is 80% women.[2] for further analysis, i broke the data down by issue. below is a bar graph showing the break down of author gender by issue. the most striking thing about the graph to me is that many of the issues have almost no participation from female authors. lastly, i looked at the 182 articles that were not accepted for publication. participation by female authors remains low but is consistent with the number of female authors of published articles. i think this data raises some interesting questions. does the journal create barriers to publication for women? how do these numbers compare to overall population of people involved in code4lib?[3] are there things we could do to increase participation in the journal? why are the numbers of women on the editorial committee so low? more importantly, how can we correct that? if you have thoughts about improving the diversity of voices that are represented in the code4lib journal, i invite you to post them to the journal’s open discussion list (c4lj-discuss@googlegroups.com). or if you would be interested in helping us make better use of the data we have, drop us a line at journal@code4lib.org. i hope you enjoy issue 24! references [1] aurora, valerie and tennant, roy. (2014). closing keynote – interview with valerie aurora. 2014 code4lib conference. raleigh, nc. https://www.youtube.com/watch?v=b5lfjpersy0&feature=share [2] office of research and statistics, american library association. (2012) diversity counts. http://www.ala.org/offices/diversity/diversitycounts/divcounts [3] metz, rosalyn. (2012). code4lib gender survey summary. https://docs.google.com/document/d/1hbofh63-5f9mwek8y8c83heoknodttaswf5juqglq1e/edit subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – editorial introduction — issue 1 mission editorial committee process and structure code4lib issue 1, 2007-12-17 editorial introduction — issue 1 this mission of the code4lib journal is to cover "the intersection of libraries, technology, and the future." we hope that this journal can be one more contribution to the developing culture of collaboration around library technology, and we welcome you to join in our experiment. by jonathan rochkind, coordinating editor, issue 1 this is a decisive time for libraries. in the changing social and technological environment, libraries must adapt to fulfill their missions and satisfy their users. library technology is acutely involved in this adaptation. digital services, content and tools have become a part of nearly every aspect of library operations. the “digital library” is here–if you work in a library, you probably work in a digital library. this mission of this journal is to cover “the intersection of libraries, technology, and the future.” we plan to provide practical information to help the library community envision and achieve our technological future, to bring libraries’ tradition of collaboration to bear on new challenges. we want the digital libraries of today to be transformed into the digital libraries of tomorrow, providing quality information while meeting new and changing needs. rapid transformation has risks, but maintaining the status quo brings its own, greater, risks. libraries must take a leading role beside their vendors in the technological innovation that must accompany this needed transformation. the code4lib community one locus of pragmatic innovation has been the code4lib community. inspired in part by the social dynamics of distributed open source projects, code4lib is an informal online social and professional network of library technologists, embodying values of transparency, cooperation, and pragmatic problem solving. the code4lib infrastructure includes a listserv, an irc channel, a blog aggregator, and an annual conference [1]. code4lib is a dynamic community which fosters collaboration and encourages the sharing of skills and ideas [2,3,4]. but paradoxically, this amorphous informality can make it hard for someone new to the field—or wanting to take a new look at the field—to find a comfortable entry point to the community and the resources it has to offer. we hope the code4lib journal can manifest the values that have been successful for the code4lib community, while providing increased access to the collective knowledge and experience held throughout our diverse professional networks and local organizations, increasing cross-pollination and collaboration among library technology innovators–and helping more people and organizations become innovators. this journal is an experiment the code4lib journal project aspires to balance a variety of sometimes competing goals. we want to provide quality articles providing useful information and discussion on bringing library technology into the future. we want every article to be a useful intervention into our communities of practice. we value readability over formality, and hope to meet high standards for quality and utility. we’d like articles to have the technical detail for reproducibility, while still being accessible to readers at varying levels of technical expertise. at the same time we want to ensure an easy process for authors, letting authors share their important work and ideas with as few barriers as we can get away with. the journal is intentionally edited rather than refereed, and we try to contribute editing advice to help authors improve their articles without aggravation. we are committed to the journal’s free online availability, to increase its visibility and impact in addition to its accessibility. we want the immediacy of a blog, the usefulness of a professional conference, the reliable quality of a good scholarly journal, and the participatory nature of our online communities, all in one easy to read and easy to produce package. and we are trying to accomplish all of that on a shoestring, with an all-volunteer editorial committee sharing management and editorial responsibilities in an informal, open, and pragmatic way as per the code4lib ethic. our coordinating editor will rotate with every issue; i’ll soon be passing the baton to eric lease morgan. the code4lib journal project is in that sense much like some of the technology projects many of us work on in our daily lives, balancing competing values and priorities with limited resources. and we’ve tackled this project the way we do those projects, with a ‘can do’ spirit and an agile development approach—in other words, we’re making it up as we go along. so how is the experiment working out? we think we’ve got a great first issue. this is due to the great work of our authors, and of the editorial committee. i am not alone among the editorial committee in discovering that inventing a journal—even one solely online which is intended to be relatively informal and agile—is more work than i personally expected. all of our authors and editorial staff deserve to be proud of what we’ve produced together through hard work [5]. but ultimately only the judgments and actions of you, our readers, can measure our success. if you think this first issue is evidence of a worthwhile endeavor, you can contribute to its future success. how can you help? you can read our articles, suggest them to others, and continue the discussions in your blogs, listservs, and in comments attached to the articles themselves. we want every article here to be part of an ongoing conversation towards cooperative innovation among libraries. you can submit articles to us, and when you run into a colleague with an interesting project or idea, you can suggest that they submit articles to us. we’re happy to accept articles and proposals at any time; proposals for our third issue are due by friday march 14th. we welcome anyone interested in participating in the operation of the journal to join our public discussion list for journal business [6]. at some point in the future, we will solicit more official members of the editorial committee, too. we hope that this journal can be one more contribution to the developing culture of collaboration around library technology, and we welcome you to join in our experiment. code4lib journal founding editorial committee carol bean, north county regional library, (palm beach county library system) jonathan brinley, ball state university libraries edward corrado, the college of new jersey, corrado@tcnj.edu tom keays, syracuse university library emily lynema, north carolina state university libraries eric lease morgan, university libraries of notre dame ron peterson, university of delaware library (ronp at udel dot edu) jonathan rochkind, johns hopkins libraries jodi schneider, amherst college library & graduate school of library and information science at uiuc ken varnum, university of michigan notes [1] http://www.code4lib.org [2] barrera, antonio, parmit chilana, kevin clarke and michael giarlo (2007). 2007 code4lib conference report. library hi tech news 24(6). pp. 4-7. http://eprints.rclis.org/archive/00011670/ [3] frumkin, jeremy and dan chudnov (2006). code4lib 2006. ariadne issue 47, april 2006. http://www.ariadne.ac.uk/issue47/code4lib-2006-rpt/ [4] chudnov, daniel (2007). code4libcon shows what a participatory conference looks like. computers in libraries 27(5), may 2007. pp. 37-40. (coins) [5] special thanks to jonathan brinley for providing the nuts-and-bolts web management that many of us wanted to leave at our day jobs. he deserves the credit for the clean look and useful functionality of the site. [6] http://groups.google.com/group/c4lj-discuss/ subscribe to comments: for this article | for all articles one response to "editorial introduction — issue 1" please leave a response below: editors’ choice: on libraries, code, support, inspiration, and collaboration | digital humanities now, 2014-07-22 […] it launched in 2007. at that time, jonathan rochkind (co-ordinating editor of the first issue), wrote “we want the immediacy of a blog, the usefulness of a professional conference, the reliable […] leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – automatic aggregation of faculty publications from personal web pages mission editorial committee process and structure code4lib issue 11, 2010-09-21 automatic aggregation of faculty publications from personal web pages many researchers make their publications available on personal web pages. in this paper, we propose a simple method for the automatic aggregation of these documents. we search faculty web pages for archived publications and present their full text links together with the author’s name and short content excerpts on a comprehensive web page. the excerpts are generated simply by querying a standard web search engine. by najko jahn, mathias lösch, and wolfram horstmann introduction in most academic fields, publication lists on personal web pages are a common practice to promote lasting research visibility. these lists not only function as individual bibliographies, but also as digital archives. often researchers complement these bibliographic references with the corresponding preor postprints in order to allow peers easy access. from a librarian perspective, such personal publication lists provide a rich source of bibliographic data and full texts which would be worthwhile to monitor and grab. in particular, these publication lists might be a good starting point for populating the local institutional repository. if the repository manager could track the deposited documents on the web pages, he could simply ask the researcher to upload these documents to the repository as well. given the large amount of highly distributed personal web pages usually hosted by an academic institution, the question arises of how to keep track of the research output available if no efficient central service is in use for publishing publication lists as part of the personal web page. in this paper, we propose a simple method for the automatic aggregation of the publications on personal web pages. we search faculty web pages for freely available publications and present their urls together with the author’s name on a comprehensive list. moreover, each list entry is being enhanced by a short document excerpt automatically generated using the bing search engine. based on a subsequent text classification, we furthermore distinguish between scholarly documents and other material such as cvs or courseware. background before proposing the method, let us reconsider our motivation. in preparation of the development and implementation of a publication management system at bielefeld university, a middle-sized and research-intensive university in the state of north rhine-westphalia (germany), we wanted to provide evidence that scholars maintain personal publication lists on the web. moreover, we were interested in whether the documents are actually downloadable from these web pages. our challenge is not a new one, but has been approached theoretically and practically by others before. since the advent of the web, scholars from various fields have been discussing scholarly communication patterns on the web and developed methods of how to track and measure these. philpapers.org, [1] an online directory for philosophical publications, shows how fruitful automatic monitoring of personal web pages can actually be. based on crawlers and metadata extraction utilities as part of the open source platform disciplinary virtual research environment (divre) [2], philpapers.org collects and indexes web content coming from the field of philosophy. one of the services philpapers.org provides, once a researcher is registered with the system, is automatically harvesting the researcher’s personal web site for new documents on a regular basis. after indexing the titles, authors, and abstracts of the new documents, the metadata are presented as part of the online bibliography philpapers.org provides. getting scholarly publications in three steps with this example in mind, we similarly look for a simple and easy method to automatically aggregate online publications of bielefeld university faculty. collecting document links in the context of the above-mentioned project concerned with publication management, we needed to gain a closer insight into the habit of maintaining personal publication lists at bielefeld university. however, the content of personal websites is not in central control at large universities [3]. therefore we asked two student assistants to manually collect information on the lists. on the basis of the official directory of staff and departments, they manually searched for the publication lists from the researchers. the search was limited to people holding at least a doctoral degree, and conducted using web search engines such as google. during a period of two months, they identified publication lists from 750 out of 1214 researchers (see figure 1), and compiled the information in a table associating the names and departments of the researchers with the urls of their lists. figure 1. detected publication lists at bielefeld university apart from the rather theoretical interest in the data, we considered to re-use part of it for automatically collecting information on the publications available through those web pages. to this end, we process the table using a python script that downloads the html pages behind the urls, and extracts all hyperlinks pointing to pdf documents, which we assume to be scholarly publications. this is achieved simply by using regular expressions. in total, we extract 3365 pdf links that give us a first idea about the coverage of alleged scholarly publications on personal lists. however, two technical problems occur: one is that some researchers link to sub-pages containing additional information and the full text document. the other is that the extraction of document links coming from script-generated web pages often fails. searching bing next, we query a web search engine to generate a short document excerpt. for this task we could use virtually any search engine that provides an api. however, we pretested a small subset of ten randomly selected entries of our table to determine the best performing engine with regards to the number of found documents and the quality of the results (see below for encountered quality issues). we take into account google, yahoo!, and bing. since bing performed best in this pretest, we also use it in the final program. we query the bing search engine directly from our program via its public web api [4]. as query strings we simply use the document links. the bing api returns a data structure containing the same information that would be presented on a regular result list. a result entry, apart from the hyperlink to the original document, consists of a title (usually the document title) and a description (a short excerpt of the indexed document). for each url we save the title and the description of the top-ranked result entry, if bing has indexed the pdf document behind the link. finally, we combine our initial table with the bing data, and display it all together on an html page that makes up our comprehensive publication list. figures 2 through 5 show excerpts of this list. apart from the titles, the excerpts ideally provide additional information, such as the bibliographic reference, the document type and the beginning of the underlying paper (see figure 2). figure 2. an almost ideal example of how the approach is supposed to work. the title has been correctly identified and the bing excerpt provides useful additional information. however, some problems occur. for example, the search engine apparently has not indexed all the documents, as for some entries there is no title or excerpt (figure 3). closer examination of these cases reveal that most of them come from script-generated web pages. figure 3. a list entry lacking title and excerpt. this may be because the document has not been indexed by bing. another problem that we observe are meaningless excerpts consisting only of junk characters (figure 4). most probably, bing could not read the underlying file and this prevented a proper indexing. figure 4. character junk probably coming from a damaged pdf file. and finally, we see documents whose titles could not be recognized correctly. in those cases, the title mostly is part of or even identical to the document’s url (see figure 5). figure 5. another bad example. apparently the alleged title is part of the document’s url. text classification since the web pages containing the publication lists often include references to other material in pdf format, such as cvs, courseware, guidelines for students, etc., our comprehensive list features some irrelevant entries. to tackle this problem, we explore automatic text classification using a naïve bayes’ classifier. this is a simple machine learning technique widely in use, for example in spam filters [5]. the idea behind this step is to train a classifier that can distinguish a document representing a journal article or thesis from any non-publication material. this approach is also being used by philpapers.org, and is described by w. schwartz in his article on “metaparser” [6]. we systematically download the pdf files and extract their raw texts using the pdftotext program [7]. in total we obtain 1701 files. note that we can only download less than half of the files initially identified. this is because many links point to documents hosted by commercial publishers. we then use langident [8] to automatically sort the plain text files by language; we find 1287 english and 414 german documents. the files are then labeled by our student assistants as publication and non-publication documents. on this corpus of labeled documents, we train a naïve bayes’ classifier using the words of every document as features. note that we eliminate punctuation and function words, and limit the feature space to the remaining 1,000 most frequent words in the corpus to minimize computational complexity. for corpus management and classification we use the natural language toolkit (nltk) python library [9]. in sum, we identify 1493 relevant scholarly publications. in order to evaluate the classifier, we use cross-validation against all documents in our corpus. this means that for each document, a classifier is trained on the whole corpus minus that particular document. the classifier is then evaluated by classifying the left out document. therefore this method is also being referred to as leave-one-out evaluation. all evaluation measures are summed up and divided by the number of documents in the corpus. by this means, we achieve a classification accuracy of 89% for the english documents. that means the classifier is able to predict the correct category (publication or not) for 89% of the documents. one could also say that for a randomly selected document, the probability for the classifier assigning it to the right category is 89%. for the german corpus the accuracy is 82%. we also measure the effectiveness of the classifier in terms of recall and precision [10] for the task of selecting the documents that are truly publications. recall measures the ability of a classifier to select the correct documents from the data set, that is the completeness of the result set. precision measures the ability of a classifier to reject incorrect documents. to compute the recall and precision values for our classifier, we must compare for each document the category assigned by the classifier with its true category, and assign it to one of the four sets (a, b, c, d) outlined in table 1. then the recall is computed by r = a/(a + c) and the precision by p = a/(a + b). reference: publication reference: no publication classifier: publication a b a + b classifier: no publication c d c + d a + c b + d a + b + c + d table 1. contingency table for computing recall and precision. our classifier achieves a recall value of 89% and a precision value of 98% for the english documents. the values for the german documents are a little lower with a recall value of 80% and a precision value of 95%. since we have not yet conducted a qualitative in-depth analysis of these results, we can only suppose that the lower values for the german documents result from the smaller data set (414 documents) as opposed to the english data set (1287 documents). limits and future steps to be taken our approach shows that personal publication lists provide a significant amount of deposited scholarly documents and our search engine-driven excerpt extraction yields promising results in many cases.  the method, however, has its limits. due to various reasons, some list entries are malformed. our method relies on an already existing directory of personal web pages of our university.  furthermore, since our directory is manually compiled, it is hard to keep track of lists that appear on the web after the compilation has been finished.  yet not all libraries may have access to such a directory, if it exists at all. an alternative might be to start with the domain of the academic institution under examination and simply crawl all the documents within the subdomains and file directories. since the number of received documents will then certainly be much higher, text classification will become even more important. in combination with the efficient crawling of websites, text classification and metadata extraction, such an approach would allow the creation of a much more detailed and robust list—certainly at the cost of development effort, since a focused web crawler is a way more complicated piece of software than our python script. considering our text classification experiments, the resulting classifier performs well enough to encourage further investigation in this direction. the high precision values show that we can reliably filter irrelevant material from the publication list. however, the achieved recall values suggest that the classifier would also reject a significant portion of publication documents. therefore we would not recommend using it in a production environment, yet. future work should aim at practical applications including the integration of library services already in use. we think of a web application that would allow repository managers to continuously monitor the faculty publication lists. therefore, a mechanism to refresh the comprehensive list on a regular basis is needed. for convenience, new entries in the list should be highlighted, for example by putting them on top of the list or by making use of color. in the long run, we can even imagine deposit scenarios for the researchers.  for example, a service could be offered to faculty to have their documents, including the extracted metadata, transferred to the institutional repository. this would dramatically lower the barrier for open access self-archiving. in the future, it would also be worth studying how to extract bibliographic metadata directly from the documents in order to overcome the limits of our current search engine-driven approach. g. hatop [11] has recently described a promising method for doing this. the final consideration is of an organizational nature. before presenting such a compiled lists on the web as a library service, methods for involving the researchers have to be developed. this includes workflows for researchers to verify their publications or to refuse to be part of the list. for instance, philppapers.org only harvest publication lists deliberately registered by the authors themselves. conclusion we presented a method for automatically monitoring faculty publication lists and detecting deposited publications in them. our main intention was to keep the implementation as simple as possible. we enhanced the publication list using a standard web search engine. furthermore, we explored text classification to distinguish between scholarly publications and other material. despite the described drawbacks, we found 1493 relevant and even accessible scholarly publications. we therefore think that the automatic aggregation of faculty publications from personal webpages is a complementary approach to digital libraries in general and institutional repositories in particular. if this approach is combined with policies and workflows to form new deposit strategies, publication lists on personal websites could become a valuable source for populating digital libraries in the future. acknowledgements we are indebted to saskia scheibler and agatha walla for their valuable help with collecting data on the publication lists and labelling the corpus data. we also wish to thank torsten wilholt. references 1. philpapers: online research in philosophy. http://philpapers.org/ 2. divre: the disciplinary virtual research environment. http://www.divre.org/ 3. aguillo, isidro f. measuring the institution’s footprint in the web. library hi tech 2009; 27(4): 540-556. available from: emerald; doi 10.1108/073788309. available from: http://biecoll.ub.uni-bielefeld.de/volltexte/2010/5002 4. bing development center. http://www.bing.com/developers 5. sahami, m., dumais, s., heckerman, d., horvitz, e.: a bayesian approach to filtering junk e-mail. in: learning for text categorization: papers from the 1998 workshop, vol. 62, pp. 98–05. aaai technical report ws-98-05, madison, wisconsin (1998) (coins) 6. schwartz, w.: metaparser. http://philpapers.org/metaparser.html 7. xpdf. http://www.foolabs.com/xpdf/ 8. simoes, a., castro, j.: lingua::identify. http://search.cpan.org/~ambs/lingua-identify-0.26/lib/lingua/identify.pm 9. bird, s., loper, e., klein, e.: natural language processing with python. o’reilly (2009) (coins) 10. lewis, d.: evaluating text categorization. in proceedings of speech and natural language workshop, pp. 312–318 (1992); doi: 10.3115/112405.112471 11. hatop, g.: bibliographic metadata extraction from theses. code4lib (7) (2009) http://journal.code4lib.org/articles/1686 code python code about the authors najko jahn (najko.jahn@uni-bielefeld.de) works at bielefeld university library. he is part of the project “publister – personal publication lists as a university-wide service” funded by the “german research foundation” (dfg). mathias lösch (mathias.loesch@uni-bielefeld.de) works at bielefeld university library. he is part of the project “automatic enhancement of oai metadata” funded by the “german research foundation” (dfg). wolfram horstmann (wolfram.horstmann@uni-bielefeld.de) is cio of scholarly information at bielefeld university. subscribe to comments: for this article | for all articles 3 responses to "automatic aggregation of faculty publications from personal web pages" please leave a response below: dori stein, 2010-09-27 i think you should also consider using website scraping tools described in this series of posts: http://www.fornova.net/blog/?p=4 brice stacey, 2010-09-28 line 189 throws an error for me. i’m running ubuntu 9.10, python 2.6.4, and had trouble installing pybing… new department http://www.cs.umb.edu/~cheungr/ http://www.cs.umb.edu/~ding/publications.html http://www.cs.umb.edu/~duc/publications.html traceback (most recent call last): file “facpublister.py”, line 240, in main() file “facpublister.py”, line 189, in main if ‘publication list’ in row[0]: indexerror: list index out of range i’m not sure why the for loop doesn’t stop. i’m not a python programmer, so maybe i didn’t do something right, but before line 189, i added the following to make it work: if len(row) == 0: break anyway, nice job. the result is pretty neat. have you thought of perhaps parsing citations from publication lists and running them through a search engine? that could be pretty powerful. mathias lösch, 2010-10-13 brice, maybe your input csv contains an empty row at the end? i tested that and it throws exactly the error you saw. thanks for catching it! we indeed thought of parsing citations, but gave up on that because we actually found it to be pretty complex. that would be another project. leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – distant listening: using python and apps scripts to text mine and tag oral history collections mission editorial committee process and structure code4lib issue 60, 2025-04-14 distant listening: using python and apps scripts to text mine and tag oral history collections this article presents a case study for creating subject tags utilizing transcription data across entire oral history collections, adapting franco moretti’s distant reading approach to narrative audio material. designed for oral history project managers, the workflow empowers student workers to generate, modify, and expand subject tags during transcription editing, thereby enhancing the overall accuracy and discoverability of the collection. the paper details the workflow, surveys challenges the process addresses, shares experiences of transcribers, and examines the limitations of data-driven, human-edited tagging. by andrew weymouth background the university of idaho’s digital scholarship and open strategies (dsos) department was established in 2008 to digitize the international jazz collection and has since expanded to over 130 digital collections.[1] these are constructed with collectionbuilder, an “open source framework for creating digital collections and exhibit websites that are driven by metadata and modern static web technology”[2]. a companion framework named oral history as data (ohd) was developed in 2016 to visualize encoded transcriptions and allow researchers to explore oral history recordings by keywords and tags. in this paper, “tagging” refers to a custom set of subject designations that can be tailored by the transcriber depending on the recording’s content and themes. figure 1. collectionbuilder browse site and collectionbuilder template interface our physical workspace at the library is the center for digital inquiry and learning (cdil), where our digital labs manager, digital project manager and i support the labor of a small group of student workers and fellowship recipients, generally around 2-5 a semester. both the collectionbuilder and ohd frameworks have been designed to be simple and accessible, only requiring someone with access to google sheets, visual studio code and minimal software installation to create, maintain, and export digital collections. for the process outlined in this paper, student transcribers only need access to google sheets to generate and edit subject tags using google’s apps script extension, while the project manager needs access to adobe premiere and a text editor to run the python script to generate transcriptions, text mine the material, and to create subject tags. the incentive for this project arose from realizing a number of oral history recordings were either untranscribed, partially transcribed or lacking in accuracy following a data migration of our digital collections away from contentdm in the winter of 2023. because of the volume of text that needed updating, it was worthwhile to rethink workflows for overall efficiency and accuracy. distant listening one element i wanted to focus on was the creation of subject tagging to enrich the transcripts. in addition to keyword searching, the ohd interface allows custom subject tags to be highlighted and visualized across the duration of recordings. new york university’s weatherly a. stephan details the importance of subject tagging oral history collections, noting how “transcription alone cannot address the perennial gap of supporting serendipitous discovery through subject-based inquiry rather than simply known-item searching.”[3] previous transcription practices had student workers and cdil fellowship recipients adding subject tags to automated transcripts as they were copy editing dialogue, using the recordings as a reference. transcribers would tag individual recordings of collections that ranged from 20 to over 100 recordings, making their best judgments to what subjects seemed like they might be indicative of the overall collection. this approach, which i am calling linear listening, may mislead transcribers by establishing repeating themes that do not occur across the collection or missing themes that only begin to appear in later recordings. figure 2. challenges of linear listening visualization our physical workspace at the library is the center for digital inquiry and learning (cdil), where our digital labs manager, digital project manager and i support the labor of a small group of student workers and fellowship recipients, generally around 2-5 a semester. both the collectionbuilder and ohd frameworks have been designed to be simple and accessible, only requiring someone with access to google sheets, visual studio code and minimal software installation to create, maintain, and export digital collections. distant listening is an alternate approach that mines combined transcripts and generates tags before the transcriber begins the copy editing process. this moniker is an adaptation of franco moretti’s concept of distant listening, “where distance[…] allows you to focus on units that are much smaller or much larger than the text: devices, themes, tropes—or genres and systems” [4]. by searching across collections for terms and phrases indicating subject matter, oral history project managers can produce richer, more accurate data that increases the discoverability of recordings and makes the transcription process more dynamic and pedagogically rewarding for transcribers. this case study details my experience over the winter and spring of 2024 developing the tools independently and testing the process with two student transcribers working through 30 transcriptions, incorporating their feedback and streamlining processes. i also had a chance to iterate over this process in january 2025, this time working with the cdil digital projects manager to transcribe, tag and copy edit 100 more recordings for digital preservation. challenges the time-intensive nature of transcription, subject tagging and web hosting has made oral history recordings an undervalued format in digital initiatives. as doug boyd, director of the louie b. nunn center for oral history at the university of kentucky recounts in his chapter oral history archives, orality and usability (2015): “from the archival perspective, oral history proved an exciting and enticing resource to acquire. however, the difficulties posed by time-intensive and financially draining realities of processing oral history collections resulted in an analog crisis in the late 1990s. hundreds of oral history archives around the united states claimed large collections, but the overwhelming majority of these collections containing thousands of interviews remained unprocessed, analog, inaccessible, and unused” [5]. in quantifying the need: a survey of existing sound recordings in collections in the united states (2015), created in collaboration with the northeast document conservation center, the authors highlight the scale of archival audio preservation challenges. they conclude that only 17% of audio holdings in u.s. collections have been digitized. the survey estimates that over 250 million preservation-worthy items remain undigitized. of these, more than 80 million (32%) will require a specialized audio preservation workflow. the report goes on to detail how the national recordings preservation plans states that many of these analog recordings must be digitized between 2027 and 2033 before material degradation [6]. in contrast to the digitization of archival photographs or documents, meeting accessibility standards for oral history recordings involves not only transcribing recordings but also presenting them in an intuitive, keyword navigable digital interface. ohd developer devin becker’s solution displays the audio at the top of the page, followed by a visualization of the recording with color valued tags, a key to the tags, a search bar for keyword queries and the transcription below. this allows researchers to follow along with the time stamped transcript as the audio plays. figure 3. demonstration of oral history as data transcription/recording interface, keyword search functionality and tag visualization showing ability to read through audio material both chronologically and vertically despite this advancement in the audio player interface, the initial transcription and tagging remained a significant hurdle in developing these collections. while machine learning speech to text technology has improved considerably since the development of the ohd platform in 2016, early, no cost transcription services were often so poor that they required extensive manual correction. fully human-driven transcription and tagging has its own challenges: it is tedious, slow moving work that, without close supervision, can result in uncontrolled vocabulary, knowledge gaps, and bias from linear listening. process figure 4. visualization of workflow from audio files to csv to python text mining and google sheets overview to summarize the process: transcription: audio is transcribed into comma separated values (csv) files using adobe premiere’s speech to text tool csvs are made into individual google sheets, exported using an google’s apps script extension and added to a folder in the python transcription mining tool on running the python script, these items are combined and searched for all associated terms and phrases built into the different subject tag categories the tool generates a tally of these terms and phrases, which is used to create the primary tag sheet in another google sheet using an apps script function, all individual transcripts are linked to the primary tag sheet so each transcript’s tag column is automatically generated new subject categories or associated terms can be added or removed from the primary tag sheet and these changes can be implemented across all individual transcripts by simply re-running the code transcription moving away from services the department had been working with, i tested adobe premiere’s speech to text tool and found it uniquely well-suited for the ohd framework, with advantages including: powered by adobe sensei, machine learning dramatically increased accuracy in differentiating speakers and transcribing dialogue, even with obscure, regional proper nouns. significantly faster transcription speed, from one 1.5-hour recording every two to three business days up to twenty 1.5-hour recordings in one day. costs covered by our university-wide adobe subscription. direct export to csv utf-8 (avoiding conversion errors necessary for ohd) available non-english language packs, enabling the creation of the department’s first spanish and french language oral history collections. privacy standards with premiere’s general data protection regulation compliance, ensuring all transcription material is stored locally and not uploaded to the cloud.[7]. figure 5. excerpt of transcript with the header names speaker name, start time, end time and text below a portion of sample dialogue that said, the tool is not perfect. while modern recordings in good audio conditions have extremely high transcription accuracy, poor quality recordings and interviews between two similar sounding people can require significant correction. python text mining after using the web-based text mining tool voyant to develop subject tags for previous oral history collections, i wanted to create a text mining tool from scratch using python that would allow the targeting of specific words and phrases and create custom tagging categories. while the natural language toolkit (nltk) has many more complicated modules for text processing, such as tokenization, parts of speech tagging and fuzzy string matching, i found these approaches generated too many false positives when it was attributing areas of the transcripts to subject tags. instead, this text mining approach favors less automation and more transparent and customizable controls. custom subject sections are created which contain around 50 terms or two-word phrases. running the python script concatenates the csv files, minimally processes them and then searches for these terms, then tallies and prints the terms in the terminal according to word frequency. once the csvs of the transcripts are generated in premiere, they are added to a google drive folder that is shared with the student workers who will be copy editing the transcriptions. using the apps script downloadsheetsascsv code (see appendix 2) is run to generate a csv with only the dialogue column. the csvs are then added to the “csv” folder in the python workspace. the code begins by importing the pandas library for data manipulation, string for punctuation removal, natural language toolkit (nltk) stopwords (words removed from text before processing and analysis) for each collection and collection counter to tally identified terms within the dialogue. next, the ‘preprocess_text’ function removes punctuation, converts text to lowercase and handles missing values by replacing them with an empty string. csv file paths are constructed, and the text data is concatenated into a single string corpus. word frequency is tallied and the 20-50 most frequent words and phrases for each subject tag section are generated when the code is run. below this header material in the python file are the three subject tag categories: general: agriculture, animals, clothing, etc. geographic: (based loosely on migration statistics from the 1910 idaho census): basque, britain, canada, china, etc. custom: (example from our rural women’s history project): marriage and divorce, motherhood, reproductive rights, etc. these fifty sections have a list of fifty associated terms and phrases that the script is searching for within the combined transcription corpus. these terms were generated using chatgpt-4 turbo with the following qualifications: the word or phrase is only associated with one section. for example, regarding the sections agriculture and animals, the word “pasture” would be excluded since it could refer to both the land used for grazing animals and also the act of animals grazing. exclude homographs (words that are spelled the same but have different meanings). for example, “sow” refers both to an adult female pig and the agricultural act of planting seeds in the ground. placenames and how certain nationalities would refer to themselves for the geographic sections. for example, “philippines”, “filipino”, “tagalog…”, “norwegian”, “norway”, “oslo…” or “japanese”, “japan”, “tokyo”, etc. terms and phrases favor informal, conversational speech. these text mining categories and sections produce a total of 2,250 associated terms or phrases that are being identified across the combined transcript corpus before the script tallies these words to generate the output shown below: figure 6. sample of identified terms from combined transcripts, tallied in descending order. see appendix 1 for the header script and one subject section or visit the github repository to view code in full. apps script connection and customization once this text mining data is produced, it can be copied and pasted into the primary tag sheet in google sheets, located in the same folder as the transcripts for student workers to access and edit. using the text to columns function, subject tag sections can be split into column a and their associated words into column b using the programmed “##” as the separator. figure 7. example of the formatted primary tags sheet with headers reading tags in column a and associated words in column b. after minor formatting to the individual transcript, student workers access the apps script extension located in the drop-down menu. transcribers then enter the code (see appendix 3), and make two adjustments: change the sheet name of the transcript they are editing on line 6 change the url of their primary tag sheet on line 13. then save and run the code now the individual transcript is connected to the primary tag sheet, which will automatically search the text column for these terms and phrases and fill in the tag column of the transcript with its associated subject tag. it is important to state that this process is not intended to replace human transcribers but shifts the focus from manual tagging to copy editing. if transcribers notice that a tag is either not applicable or missing from the primary tag sheet, they are encouraged to make these additions or subtractions and rerun the apps script on their individual sheets, which will automatically enact these revisions across the entire document. if transcribers notice errors that are more specific to individual transcriptions, they can paste these edits into an additions or subtractions column to the right of the tag column, so the changes aren’t written over by future runs of the apps script code. findings figure 8. example of a pre and post process tagging visualization of a recording, with the post process being dramatically more dense. while initially testing this process, my main concerns were: would transcribers find the apps script coding element confusing and/or anxiety-inducing? due to the complexity of language, would the automated tagging generate so many false positives that correcting these items would become a drag on productivity? working with a student worker and a fellowship recipient, copy editing 30 transcripts over the course of two months in the summer of 2024, these factors were not an issue. possibly helpful in this effort was weekly meetings where we checked in and tested the code, sometimes purposefully breaking it to show how those mistakes can be easily fixed and demonstrate how they can update the primary tags sheet and rerun on their individual transcript sheets. rather than simply asking student workers to transcribe recordings—work that offers little to highlight on a cv and can lead to burnout and high turnover—this process allows transcribers to engage in coding, create and modify tags, and see those changes reflected instantly through the apps script process. i had the opportunity to revisit and iterate on these tools and processes for another oral history digital collection undertaken in january 2025, this time working with the digital projects manager, as opposed to student transcribers. we were able to complete the process lifecycle of transcription, tagging and copy editing for 100 recordings in just over a week, increasing productivity from the summer 2024 initiative by 1233%. regarding the limitations of data-driven, human-edited automated tagging, program managers must communicate that automated tags are only a starting point. tags may be incorrectly applied, missing or need to be applied more broadly to transcripts. even when these measures are taken, the amount of detail this process accrues is easily distinguishable in the before and after ohd tagging visualization shown above (fig. 8). one could argue that the density of the data might now make it difficult for the researcher to navigate, especially on mobile devices. this continues to be a dialogue as we refine this workflow. conclusion while discussing grant funding for digital initiatives, a colleague pointed out that the time-intensive nature of oral history projects often leads to their neglect. as they put it: “would you rather present ten oral history recordings or 500 photographs?” this quantity-focused selection criteria ultimately poses an existential threat, leaving these materials physically vulnerable as they languish in the archives. bicentennial and community oral history initiatives, rich in non-academic perspective, offer a uniquely biographical account of places and provide valuable contrast and context to the accepted historical record. by utilizing machine learning, python, and apps script approaches, this process seeks to make digitizing these resources more efficient and accessible, promoting their preservation and availability to the public. references and notes [1] digital collections, university of idaho. university of idaho library digital initiatives. 2024 [cited 2024 jul 8]. available from: https://www.lib.uidaho.edu/digital/collections.html [2] home. collectionbuilder. [accessed 2025 feb 13]. https://collectionbuilder.github.io/ [3] stephan w. the platinum rule meets the golden minimum: inclusive and efficient archival description of oral histories. journal of contemporary archival studies. 2021;8(1). https://elischolar.library.yale.edu/jcas/vol8/iss1/11 [4] moretti f. conjectures on world literature. new left review. 2000;(1):54–68. [5] boyd da. ‘i just want to click on it to listen’: oral history archives, orality and usability. in: the oral history reader. 3rd ed. routledge; 2015. [6] avp. quantifying the need: a survey of existing sound recordings in collections in the united states. avp. 2014 [accessed 2025 feb 14]. https://www.weareavp.com/quantifying-the-need-a-survey-of-existing-sound-recordings-in-collections-in-the-united-states/ [7] speech to text in premiere pro faq. adobe. [cited 2024 jul 8]. available from: https://helpx.adobe.com/content/help/en/premiere-pro/using/speech-to-text-faq.html about the author andrew weymouth is the digital initiatives librarian at university of idaho, specializing in static web design to curate the institution’s special collections and partner with faculty and students on fellowship projects. his work spans digital scholarship projects at the universities of oregon and washington and the tacoma northwest room archives, including long form audio public history projects, architectural databases, oral history and network visualizations. he writes about labor, architecture, underrepresented communities and using digital methods to survey equity in archival collections. professional site: aweymo.github.io/base appendices appendix 1. excerpt of python text mining tool import pandas as pd import string from nltk.corpus import stopwords from collections import counter import re # download nltk stopwords data import nltk nltk.download('stopwords') # define preprocess_text function def preprocess_text(text): if isinstance(text, str): # check if text is a string text = text.translate(str.maketrans('', '', string.punctuation)) text = text.lower() # convert text to lowercase else: text = '' # replace nans with an empty string return text # load stopwords for both spanish and english stop_words_spanish = set(stopwords.words('spanish')) stop_words_english = set(stopwords.words('english')) # combine both sets of stopwords stop_words = stop_words_spanish.union(stop_words_english) import os # directory containing csv files directory = "/users/andrewweymouth/documents/github/transcript_mining_base/csv" # list of csv file names file_names = [ 'example_01.csv', 'example_02.csv', 'example_03.csv' ] # construct file paths using os.path.join() file_paths = [os.path.join(directory, file_name) for file_name in file_names] # initialize an empty list to hold the dataframes dfs = [] # try reading each csv file and print which file is being processed for file_path in file_paths: try: print(f"processing: {file_path}") # add quotechar and escapechar for handling csvs with quotes dfs.append(pd.read_csv(file_path, encoding='utf-8', quotechar='"', escapechar='\\')) except exception as e: print(f"error with file {file_path}: {e}") # concatenate text data from all dataframes into a single corpus corpus = '' for df in dfs: text_series = df['text'].fillna('').astype(str).str.lower().str.strip() # extract and preprocess text column corpus += ' '.join(text_series) + ' ' # concatenate preprocessed text with space delimiter # preprocess the entire corpus cleaned_corpus = preprocess_text(corpus) # remove stopwords from the corpus filtered_words = [word for word in cleaned_corpus.split() if word not in stop_words and len(word) >= 5] # count the frequency of each word word_freq = counter(filtered_words) # get top 100 most frequent distinctive words with occurrences top_distinctive_words = word_freq.most_common(100) # === general section === def find_agriculture_terms(corpus): # define a list of agriculture-related terms agriculture_terms = [term.lower() for term in ["harvest", "tractor", "acreage", "crop", "livestock", "farm field", "barn building", "ranch", "garden", "orchard", "dairy", "cattle", "poultry", "farming equipment", "fertilizer", "seed", "irrigation", "plow", "farmhand", "hoe", "shovel", "milking", "hay", "silage", "compost", "weeding", "crop rotation", "organic", "gmo", "sustainable", "farming", "rural", "homestead", "grain crop", "wheat", "corn maize", "soybean", "potato", "apple fruit", "berry", "honey", "apiary", "pasture", "combine harvester", "trailer", "baler", "thresher" ]] # initialize a counter to tally occurrences of agriculture-related terms agriculture_word_freq = counter() # tokenize the corpus to handle multi-word expressions tokens = re.findall(r'\b\w+\b', corpus.lower()) # iterate over each token in the corpus for word in tokens: if word in agriculture_terms: agriculture_word_freq[word] += 1 # return the top 20 most common agriculture-related terms return agriculture_word_freq.most_common(20) # call the function to find agriculture-related terms in the corpus top_agriculture_terms = find_agriculture_terms(corpus) # print the top 50 agriculture-related terms print("## agriculture") for word, count in top_agriculture_terms: print(f"{word}: {count}") appendix 2. apps script code for exporting sheets to csv for text mining function downloadsheetsascsv() { // specify the folder id of the folder containing the google sheets var folderid = 'folder-id'; // replace with your folder id var folder = driveapp.getfolderbyid(folderid); var files = folder.getfiles(); // loop through each file in the folder while (files.hasnext()) { var file = files.next(); // check if the file is a google sheet if (file.getmimetype() === mimetype.google_sheets) { var spreadsheet = spreadsheetapp.openbyid(file.getid()); var sheets = spreadsheet.getsheets(); // loop through all sheets and download each as csv for (var i = 0; i < sheets.length; i++) { var sheet = sheets[i]; var csv = convertsheettocsv(sheet); // create a new csv file in the same folder var csvfile = folder.createfile(sheet.getname() + '.csv', csv, mimetype.csv); logger.log('downloaded: ' + csvfile.getname()); } } } } function convertsheettocsv(sheet) { var data = sheet.getdatarange().getvalues(); // find the index of the "words" column and replace it with "text" var headerrow = data[0]; var wordsindex = headerrow.indexof('words'); // locate the "words" column index if (wordsindex !== -1) { headerrow[wordsindex] = 'text'; // change "words" to "text" } // start building the csv with the header row var csv = 'text\n'; // loop through rows and extract the "words" column, removing line breaks for (var i = 1; i < data.length; i++) { // start from 1 to skip the header row var row = data[i]; // extract the "words" column (index of "words" column) var cell = row[wordsindex]; // remove all line breaks (carriage returns, newlines, etc.) within the "words" data if (typeof cell === 'string') { cell = cell.replace(/(\r\n|\n|\r)/gm, ' '); // replace all line breaks with space cell = cell.replace(/[^\w\s,.'"-]/g, ''); // remove punctuation except for some valid ones } // enclose the text in quotes to avoid column splitting due to commas cell = '"' + cell + '"'; // add the cleaned "text" to the csv output csv += cell + '\n'; } return csv; } appendix 3. apps script code for linking transcript to primary tag sheet function filltags() { // get the active spreadsheet var spreadsheet = spreadsheetapp.getactivespreadsheet(); // get the transcript sheet by name var transcriptsheet = spreadsheet.getsheetbyname("your-sheeet-name"); if (!transcriptsheet) { logger.log("transcript sheet not found"); return; } // set the header in cell e1 to "tags" transcriptsheet.getrange("e1").setvalue("tags"); // get the tags spreadsheet by url var tagsspreadsheet = spreadsheetapp.openbyurl("your-spreadsheet-url"); if (!tagsspreadsheet) { logger.log("tags spreadsheet not found"); return; } // get the tags sheet within the tags spreadsheet var tagssheet = tagsspreadsheet.getsheetbyname("tags"); if (!tagssheet) { logger.log("tags sheet not found"); return; } // get the range of the transcript column var transcriptrange = transcriptsheet.getrange("d2:d" + transcriptsheet.getlastrow()); var transcriptvalues = transcriptrange.getvalues(); // get the range of example words and tags in the tags sheet var examplewordsrange = tagssheet.getrange("b2:b" + tagssheet.getlastrow()); var tagsrange = tagssheet.getrange("a2:a" + tagssheet.getlastrow()); var examplewordsvalues = examplewordsrange.getvalues(); var tagsvalues = tagsrange.getvalues(); // create a map of example words to tags var tagsmap = {}; for (var i = 0; i < examplewordsvalues.length; i++) { var word = examplewordsvalues[i][0].tolowercase(); var tag = tagsvalues[i][0]; tagsmap[word] = tag; } // initialize an array to store the tags for each transcript entry var transcripttags = []; // loop through each transcript entry for (var i = 0; i < transcriptvalues.length; i++) { var text = transcriptvalues[i][0]; var uniquetags = []; if (typeof text === 'string') { // use regular expression to extract words and handle punctuation var words = text.match(/\b\w+['-]?\w*|\w+['-]?\w*\b/g); // check each word in the transcript entry against the tags map if (words) { for (var j = 0; j < words.length; j++) { var word = words[j].tolowercase().replace(/[.,!?;:()]/g, ''); var singularword = word.endswith('s') ? word.slice(0, -1) : word; if (tagsmap.hasownproperty(word) && !uniquetags.includes(tagsmap[word])) { uniquetags.push(tagsmap[word]); } else if (tagsmap.hasownproperty(singularword) && !uniquetags.includes(tagsmap[singularword])) { uniquetags.push(tagsmap[singularword]); } } } } // add the determined tags to the array transcripttags.push([uniquetags.join(";")]); } // get the range of the tags column in the transcript sheet, starting from e2 var tagscolumn = transcriptsheet.getrange("e2:e" + (transcripttags.length + 1)); // set the values in the tags column to the determined tags tagscolumn.setvalues(transcripttags); }   subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – a comparison of article search apis via blinded experiment and developer review mission editorial committee process and structure code4lib issue 19, 2013-01-15 a comparison of article search apis via blinded experiment and developer review this study looks at perceived user preference between products that can provide a scholarly article search service via an application programming interface (api). the study set up a blinded review and asked users at johns hopkins to select the service that provided the most useful results. few statistically significant preferences were detected, and some interpretation is provided of what the results might tell us. the specific products evaluated for this study are: serials solutions summon, ex libris primo, ebsco eds, ebscohost ‘traditional’ api, and elsevier scopus. re-usable open source tools for implementing article search were created to support the study and future development, and a developer review of the apis is included based on the developer’s experience in this implementation. *note from the editor: the author has invited representatives from the reviewed products to provide comments on the article.  these comments have been attached as the first comments of the publication. background and goals at johns hopkins libraries, we decided to explore the creation of an improved, more integrated article search experience for our users. the motivation for this work was driven by negative user feedback from our current offerings, staff observation, and reports by other academic libraries investigating patron article search behavior and preferences. we naturally considered the relatively new generation “discovery” products in the library market, such as serialsolutions summon, ex libris primo, ebsco discovery service (eds), and (to some extent) oclc worldcat local. all of these products include their own extensive index of articles and scholarly citations, as well as being marketed as a potential public catalog interface (with integrated article search).  but rather than assume a complete replacement of our catalog discovery with such a product we considered the approach taken by some of our peer institutions using custom designed software that provided catalog search and article search in separate sections of the result page, a layout that’s been called “bento style” (for its visual resemblance to japanese bento lunchboxes) by library technologist tito sierra. examples include: columbia university north carolina state university university of virginia we decided that a bento-style search was worth serious consideration: leaving catalog search powered by our existing infrastructure, but adding an article search section powered by a yet-to-be-determined third party service.  additional details about the reasoning and evidence that led to this direction of pursuit, including local evidence and citations to findings from other peer institutions, can be found in a position paper on the author’s blog [1]. the question arose as to how to evaluate available third-party services that might be used to power an integrated ‘article search’ feature. focusing on article search presented in a bento-style interface, we also realized that some traditional abstracting & indexing services might also be worth considering, in addition to more recently introduced ‘discovery’ products. we wanted to find a way to apply a user-centered empirical investigation to this evaluation. i realized that the same capabilities that would allow one to embed an article search in a local applications bento-style interface (essentially, a sufficiently powerful api), would also allow us to create a ‘blinded’ survey instrument, where we displayed results from two different searches and asked the participating users to tell us which they liked better [2]. we decided to pursue this idea utilizing a blinded survey instrument to evaluate the users’ perceptions of results from different services. an additional benefit of this approach is that the implementation of the blinded survey instrument required a developer to use the same techniques that would be used in implementing the potential final product. therefore, this approach allowed for review of the quality and capabilities of the various product apis, and also served as a ‘proof of concept’ of the ‘bento style’ approach with each product, developing some software building blocks that could be re-used for implementation of a final product. these advantages helped motivate our course of action. others before us have done article search or discovery product comparisons where users were given pre-selected research questions, and/or results were evaluated for quality by librarians [3]. we wanted to try instead letting the users bring their own research questions and queries, and evaluate for themselves which tool they preferred. we thought this might allow us to find out the actual preferences of users for their actual needs (something we were interested in), without having to figure out for ourselves if we had the right ‘typical’ research questions, or if we evaluated results the same way our patrons did. this was our goal and motivation; we had somewhat mixed success.  products included in study in order to be included in our blinded study, a product needed to have a sufficient api for the blinded instrument, which are essentially the same api features required for a ‘bento style’ implementation. while we decided for purposes of the study instrument not to include links from articles to full text, a production implementation would need a way to link to full text and/or to a local link resolver. we decided to only include services in our evaluation that could support such linking. products included in evaluation: ebsco discovery service (evaluation access) ebscohost ‘traditional’ api (http://support.epnet.com/eit/ws.php) with existing licensed ebsco databases (usable with our existing licenses) (set up to search approx 55 of our most used ebsco databases) ex libris primo (evaluation access) serials solutions summon (evaluation access) scopus (elsevier) (usable with our existing licenses) (http://api.elsevier.com/content/search/#d0n17066) securing access to these products and their documentation — both ones we needed trial access to as well as products we already licensed — was a time-consuming process, and comprised a significant portion of the total calendar time to complete our evaluation. products initially considered but not included in the study due to lack of fitness: google scholar: the service has no api, and has terms of service that disallow screen-scraping; it can only be used in google’s own native interface. microsoft academic search: while it does have an api, the terms of service are prohibitive (including a requirement to include all links delivered by the api, in the order they are delivered, even if they lead to paywalls you do not license). additionally, while the ms academic search product itself includes sufficient metadata for an openurl (as revealed, for instance, in the ris export), the api response does not actually include this data with sufficient semantic granularity. thomson reuters web of knowledge: while the product did seem to have a sufficient api (for instance as used by our existing metalib installation), and we believed our existing license ought to give us access to this api — we were unable to secure access/permission/documentation in time to include in the study, despite several months of attempts. our existing metalib-based, xerxes-enhanced product, using its default/general search. we initially hoped to include this as a baseline/control, but writing the evaluation tool that could accommodate metalib’s extreme slowness would have significantly increased the development time, and we ultimately decided that we did not need it.   the survey instrument users would enter a query and be presented with side-by-side ‘blinded’ results from two products chosen at random from the five services being studied. the participating user could choose which results they preferred, or select a neutral choice of “can’t decide/about the same”.   technological implementation the survey instrument was implemented as two ruby on rails projects, both licensed open source. the first, bento_search [4] is a general purpose abstract layer for querying third party search engines. it is designed to: let you quickly implement search functionality in your own custom application, for any of its supported adapters; have a potentially shared implementation of working around the gotchas and undocumented features of certain search services; and let you switch out one service for another in an existing application with as little trouble as possible (to avoid lock-in). bento_search currently supports adapters for the five evaluated services, plus adapters for google books, worldcat and google site search. one of the motivations for undertaking this particular research was that the code required to write the survey instrument could largely be repurposed for an actual production application; the bento_search gem was designed from the start to support both use cases. it supports fielded search entry, pagination, and sorting in an abstract standardized way, but does not yet support multi-field/advanced search or facets/limits. bento_search also supports a standardized rendering/display of search results, across search providers — a necessity for the survey instrument. the survey instrument itself is a rails application called bento_battle, also available as open source. bento_battle, built with bento_search functionality, is only a few hundred lines of generously commented/whitespaced ruby. we believe it’s important that the complete implementation of our experimental instrument is available, both so people can see exactly how the experiment was conducted, and so people can use our implementation to re-run the experiment, either identically or with changes, to investigate reproducibility of our findings. the exact version of the bento_battle application used for the experiment is tagged at jhu-study-10sep2012 [5]. mid-way through the study, a new version was deployed with some bugfixes and enhanced reporting features, at jhu-study-17sep2012 [6]. the nature of a ruby application using bundler means that the exact version of all gem dependencies, including the bento_search gem, is recorded and fixed in these tagged snapshots. per-product configuration and implementation choices there are some inevitable choices to be made in both configuring how to search a given product, and implementing how the product will be searched and how results will be displayed. these choices could affect the experimental results, but need to be made anyway, in some cases to implement anything at all, and in other cases to try and show the products in their best light.  display our standard template shows an abstract or other summary where available. if the product supported showing a query-in-context summary with highlighted search terms, this was used instead of a straight abstract. summon supports a google-style query-in-context snippet from full text, where available — this was used in display in preference over straight abstract. primo and eds both supply only abstracts, but do query term highlighting within the abstract — this capability was used in the survey instrument in these cases. all three products do query-term highlighting in the item title, and this too was used in the survey instrument’s output. ebscohost offered only a straight abstract, so this was used for ebscohost output. in all cases, abstracts/snippets were truncated to just the first few lines. for purposes of the survey instrument and study, we decided not to provide any links from presented search results to full text or additional information, instead only presenting side by side results with no links to click on. search input in order to provide a standard query input syntax across all search products, the bento_search library discussed above supports a normalized standard input format across all products: basically a simple ‘google style’ input, that supports a simple list of terms and phrases (using double-quotes to indicate phrases). bento_search translates this to the most appropriate query the developer could determine for each product, to find results matching that list of terms and phrases with appropriate relevancy ranking. for instance, in the case of ebscohost, bento_search translates the user’s input to an explicit boolean ‘and’ expression; in the case of summon, bento_search passes the user’s input through to summon more or less unmodified. the bento_search gem does not presently support standardized boolean operators or expressions. holdings limiting — not used the three ‘discovery’ tools (primo, summon, eds) all have features were you can limit search results to ‘items held by your institution’ — sometimes this means electronic full text access, sometimes it includes print holdings. in order to do this, you of course need to communicate your holdings to the service in some way. configuring our holdings with each such product would have taken additional setup time, and introduced an additional point for mistakes to be made and inconsistencies introduced. we also weren’t certain whether we (or our users) would want the ultimate production tool we were considering to default to such a holdings limit, or to instead return unrestricted results. considering overall cost vs. benefit to run this study, we chose not to make use of any ‘limit to holdings’ in our tests — all searches were unlimited as to institutional holdings. authorized user some of these products produce different results depending on whether the end-user is currently authenticated as a member of the institution or not. other products only allow results to be shown to authenticated users.  as our test instrument authenticated users before allowing any access, all product api access was configured to assume an authenticated user. content type filtering we noted that many peer institutions implementing a discovery service supply a default pre-set limit to exclude newspaper articles, or to focus exclusively on journal articles — they had found that non-scholarly content in the results was negatively impacting their quality and service. this could have an even more devious effect on a study like this, if left uncontrolled. if users did prefer not to have newspaper articles in their result set, and rated one product poorly for including such results, when that product could have been configured to exclude them, this would be to some extent a misleading result. our choice not to limit to local holdings also resulted in many types of content being included in results that seemed of questionable utility (such as television transcripts and web page text), but which would have ended up excluded by a local holdings limit. i suspect several products have been optimized by their vendors for ‘limit to local holdings’ use. adding some compensatory configuration to exclude these odd content types, where possible, seemed necessary to show each product at its best. we also kept in mind that our original goal was to analyze these products for a basic, simple, article and scholarly citation search — not necessarily for use cases involving other types of content. all these considerations led us to configure the searching for each product to try to exclude non-scholarly citations and focus on articles. where possible, we chose not to limit only to ‘journal articles’, but instead to exclude certain product-specific categories that seemed to create ‘noise’ in the result set, as this generally seemed to produce optimal results. while these configuration choices, specific to each product, were subjective and might influence study results, we still decided they were necessary to present each product at its best to study participants. exact configuration choices made can generally be deduced from the configuration file in the git repository for the test instrument. ebscohost traditional the ebscohost ‘traditional’ api allows you to search from 1 to all of your existing licensed ebsco databases simultaneously, returning a single merged result set. we license nearly 100 ebsco databases (although some of them are strict subsets of others). trying to search all of our databases at once through this api resulted in unacceptable performance. we looked at our usage logs for ebsco, and took the top-most used databases. we then adjusted the list a bit by hand to broaden disciplinary coverage as far as we could within our ebsco licenses, and to remove databases focused on formats we weren’t interested in (image databases, or video databases). the ‘ebscohost traditional’ option was configured to search the resulting list of approximately 40 databases. the exact list of databases used can be seen in the git repository for the bento_battle instrument implementation code [7]. promotion and response our evaluation tool was open to users from september 10th 2012 until october 2nd. login to our enterprise sign-in system was required to access the evaluation tool, in order to comply with all relevant contracts and agreements for included products. promotion was via: links on library home pages inviting participation a link on our metalib-based jhsearch results page, inviting participation — which would execute the user’s already entered query in the search instrument. a similar query-preserving link in the johns hopkins welch medical library multi-search tool (also metalib-based). a library blog post. bulk email to librarians, who then re-publicized the study to their liaison departments. (library staff was also welcome to participate in the study). at the eisenhower library, placards were placed next to reference/information desk stations with the url, for staff to participate in during any free time. other jh libraries may have publicized the tool to their patrons in various ways. it was somewhat more challenging to attract participation than we expected. during the period the study was up, we received 414 total preference responses. this is not as large a sample as it might initially sound, as with 5 total products and 414 total responses, any given pairing of products (say primo vs. summon) were only matched on average 41 times each. we don’t know how many unique people participated; for instance, 40 people doing 10 choices each, or 400 people doing one choice each. we also don’t know which participation events came from which promotion venue. in addition to the 414 registered selections, there were another 214 occasions a user loaded the evaluation tool results page, but abandoned it without making any selection at all. so about 1/3rd of the time an evaluation page was displayed, it was abandoned without a selection. demographics we had an optional area for participants to identify their affiliated hopkins school/institution, and their role/status. as this was optional, many participants chose not to fill it in. in retrospect we still think making demographics optional was the right design choice, to maximize participation. the plurality of responses did not self-report their demographic characteristics, and we decided that we lacked sufficient sample size in any of the demographic groups to justify reporting or comparing results between or within the individual groups. limitations any experimental design has limitations: both things not being studied and potential limitations in the validity of what is being studied. it is important to understand the limitations of our experimental design before we move on to looking at our findings. we only evaluated the article search function. some of these products come with other functions or features that may be of use, but are not being evaluated here. evaluation was limited to a basic, simple style of article search; a usage pattern where a user enters some search terms and looks at the first page of results. our study did not include or evaluate advanced/fielded searching, faceting, looking past the first page of results, etc. some products may do some of these things better than others. our experimental design did not attempt to explore this. our experimental design only reveals relative preferences. if two providers are pitted against each other 100 times, and one tool is selected as better 90 times — it’s quite possible that users considered both providers great, or considered both providers awful. we have no way to distinguish. our experimental design only tells us they liked one better than the other, only a relative preference between the two. our experimental design only measures consistency of relative preference, not strength. if two providers are pitted against each other 100 times, and one tool is selected as better 90 times, this means our participant population very consistently preferred that tool — it does not tell us if they thought it was a lot better, or just a tiny bit better. our design reveals consistency of preference, not strength of preference. we do not know why participants preferred one tool to another, or if different participants had different reasons at different times. although it would be nice to know more details about what users want in a search tool, answering this question was not a goal of this research. a good experimental design tries to control as much as possible, and isolates a very specific phenomenon to be studied. we tried to do this, focusing on the usage pattern/style we were prioritizing: user preferences between products, for a basic, simple, article search functionality. and we wanted to discover which product, of products as they actually exist available to us now, our users would prefer. we tried to design an experiment to efficiently answer this question. however, there are also some limitations to the validity of even what we tried to measure: participants, in the artificial environment of our experiment, may not express preferences that match what their real world preferences would be. when i experimented with the evaluation tool, i found that if i just entered a hypothetical query, i really had no way to evaluate the results. i needed to enter a query that was an actual research question i had, where i actually wanted answers. then i was able to know which set of results was better. however, when observing others using the evaluation tool, i observed many entering just the sort of hypothetical sample queries that i think are hard to actually evaluate realistically. we tried to include links from existing tools to find articles, to capture participants in the middle of the research process with actual queries — but we did not track what portion of participation came via these avenues. this issue does not apply to known-item searches, where either the item you are looking for is there at the top of the list, or it isn’t. looking through the queries entered by participants, there seem to be very few ‘known item’ searches (known title/author), even though we know from user feedback that users want to do such searches. so the study may not adequately cover this use case. findings in summary, the study, largely, did not show significant preferences between products. while this could be due to lack of sufficient sample size, there are reasons to think that is a valid result, at least under this experimental design. the rest of this section and the subsequent interpretation section will explain how we arrived at this conclusion and what we think it means. our complete data set from is publically available at: http://jhir.library.jhu.edu/handle/1774.2/36246 . others may want to analyze that data themselves, or check our findings against the actual dataset. somewhat unruly data set the experimental design resulted in a somewhat unruly data set. 414 total 1-vs-1 selections, with each selection being between two of five possible products. on average, each product was presented in a selection 166 times ( (414*2) / 5 ), but since options to present were chosen randomly, some products were presented slightly more and others less (from 155 times to 171 times). on average, any two products were presented as paired ‘competitors’ 41 times each. (if n=5 is the total number of options, and m=414 is the total number of presentations, this is ( m / (n-1)! )). however, random variation means that actually the number of presentations for each pairing ranged from 35 to 48. you can see that even with 414 total preferences selected, we get relatively few preferences expressed for any given individual pairing: for instance, summon was only matched against primo 48 times, and of those 18 times “about the same/can’t tell” was chosen, so a preference was only expressed 30 times. this ends up being a fairly modest sample size. the one-sample z test to explain the statistical analysis method used, let’s quickly explore an analogy. let’s say you were taste-testing colas. coke or pepsi, we’re not telling you which is which, which do you like better? but let’s say you misled the participants, and it was really coke in both glasses. if you gave 100 people this taste test, you would expect glass on the left (coke) to win about 50 times, and glass on the right (coke) to win about 50 times. because they are the same. of course, it might not be exactly 50/50, just as when you flip a fair coin 100 times you might not get exactly 50 heads, but it will usually be close, maybe 48/52 etc. but let’s go back to an actual taste test, where one glass really is coke and another pepsi, and we don’t tell the participants which is which. let’s say we have 100 participants, and 57 choose coke and 43 choose pepsi. how do we know if this is an actual expressed preference, or is this just random variation approaching 50/50? one way is to use a one sample z-test for proportions, with a null hypothesis of 0.5. if we use that test with an alpha value of .05 (less than 5% chance of happening through random variation), then in the particular example above we find that with a sample size of 100, 57/43 is not in fact a statistically significant difference from the 50/50 null hypothesis. we can apply exactly this test to each individual pairing in our results, say all the times summon was presented against primo and a selection was made, to see if the preferences differ significantly from a 50/50 no preference. but what do we do with the “can’t decide/about the same” choices? i decided that they give us no information about preferences between two products, they are essentially an ‘abstention’ in helping us determine user preferences, and should not be counted. for overall rankings, we can look at all the times a given product, say summon, was included in the choices and a selection was made, across all the different products it could be paired with, and look at overall what summon’s ‘victory rate’ is (proportion of times it was chosen). we can use the one-sample z-test again to determine if its victory rate differs significantly from the 50/50 null hypothesis of no preference. we did not come up with any way to actually test overall comparisons between products — only an overall score for each product differing significantly (or not) from 50%, and each individual pairing. i am fairly inexperienced at statistical quantitative analysis (as are library technology departments in general, i think). although i am happy with our approach here, there may be a better way to analyze these results; if a reader knows of one, please do share. okay, the actual findings overall please note again that few of these numbers were statistically significant. engine num participating wins losses ties victory rate summon 168 76 54 38 58% ebscohost 165 64 51 50 56% eds 155 58 52 45 53% primo 169 62 66 41 48% scopus 171 51 88 32 37%   **(significant)** each row was analyzed with a one-sample z-test, alpha of 0.05, null hypothesis of 0.50 — ties were excluded as giving no information on preferences, the ‘victory’ rate is just wins / (wins + losses). only scopus differs significantly from the 50/50 no preference (with scopus being less preferred compared to alternatives). all the other rows are essentially statistically non-distinguishable from a straight 50/50. (**summon does get close** to being significantly different than 50/50 in the positive direction; if one more summon choice had been received, and no more anti-summon choices, it would have crossed into statistical significance). in order to look more deeply, we have to look at individual pairings: okay, scopus is significantly unliked in general, but in particular is it less preferred than some alternatives, while no less preferred (or even more preferred) than others? individual pairings we can look at a cross-tabulation of each individual possible pairing. summon vs. primo, summon vs. scopus, etc. and we can analyze each pairing for statistical significance using the one-sample z-test again. vs. eds scopus ebscohost eds * total:44tie:10scopus win: 11eds win: 23scopus:0.32 significant total:38tie:20ebscohost win: 13eds win: 5ebscohost:0.72 scopus total:44tie:10eds win: 23scopus win: 11eds:0.68 significant * total:43tie:12ebscohost win: 17scopus win: 14ebscohost:0.55 ebscohost total:38tie:20eds win: 5ebscohost win: 13eds:0.28 total:43tie:12scopus win: 14ebscohost win: 17scopus:0.45 * primo total:38tie:8eds win: 16primo win: 14eds:0.53 total:40tie:6scopus win: 13primo win: 21scopus:0.38 total:43tie:9ebscohost win: 20primo win: 14ebscohost:0.59 summon total:35tie:7eds win: 14summon win: 14eds:0.50 total:44tie:4scopus win: 13summon win: 27scopus:0.33 significant total:41tie:9ebscohost win: 14summon win: 18ebscohost:0.44   vs. primo summon eds total:38tie:8primo win: 14eds win: 16primo:0.47 total:35tie:7summon win: 14eds win: 14summon:0.50 scopus total:40tie:6primo win: 21scopus win: 13primo:0.62 total:44tie:4summon win: 27scopus win: 13summon:0.68 significant ebscohost total:43tie:9primo win: 14ebscohost win: 20primo:0.41 total:41tie:9summon win: 18ebscohost win: 14summon:0.56 primo * total:48tie:18summon win: 17primo win: 13summon:0.57 summon total:48tie:18primo win: 13summon win: 17primo:0.43 * while 414 total responses may seem like a healthy number, it results in a very small sample size for each pairing. for example, summon was matched up against primo only 48 times in which the user made a selection. of those 48 times, summon was chosen 17 times, primo 13 times, and 18 ties (“can’t decide/about the same”). we want to know if choices in an individual pairing represent a significant indication of preference, or just random variation; we again use the one-sample z test with an alpha of 0.05, calculated for each pairing, with .5 as the null-hypothesis (no preference), and considering only active selections.  using this analysis, the only pairings that achieve statistical significance are: a preference for eds over scopus a preference for summon over scopus that is, the only statistically significant preferences we detected both overall and in individual pairings were a negative preference against scopus. interpretation one obvious question is whether we have a finding of no user preference of our users, collectively, thinking all the products are about equal, or if our findings are simply inconclusive.  i suspect the answer is a bit complex. while small sample size made it hard to show statistical significance in some cases, i suspect a larger sample size would still have the products clustering fairly tightly.  as our experiment progressed with more responses submitted, the rankings became closer to each other. the significant number of “ties” in every pair-up (generally from 25%-33% of selections) gives us another reason to suspect that indeed different products perform ‘about the same’. i think there’s a larger issue though with the nature of the experiment. if used over time in production, some products may very well satisfy users better than others — but when asked to express a preference for a small handful of searches in the artificial context of the experiment, users may not have the capability to adequately judge which products may be more helpful in actual use. i think this is quite possible, especially if users were not using their own current real research questions to test. some users, especially beginner/undergraduate users, may simply be unconcerned with relevance of results, being satisfied by nearly any list of results. in a survey-based study receiving 523 responses at a ‘mid-sized midwestern university’, james p. purdy [8] reports: “only six students explicitly identified relevance as a reason a research resource was their favorite. in other words, only six students favored a resource for its ability to return sources relevant for their topic and/or assignment… that students valued quality over relevance indicates students may define research as meeting particular task criteria, rather than generating knowledge. for example, they may see good research as referencing five scholarly sources rather than conversing with topically relevant sources.” if the findings in the purdy study are generalizable, what this means for library practice is still unclear — does that mean that the relevance/topical quality of results does not matter for our users and we need to pay no attention to it either? or does it mean that to maximize quality for our patrons, we need to care about factors that our users themselves don’t, and can not rely on user’s own evaluations to guide us? librarians might possibly be more expert/capable at choosing the right queries to search and judging based on a few sample searches — unfortunately, with only 64 preference selections from self-identified ‘library staff’, we don’t think we have a large enough sample to look at the preferences of that demographic on their own. despite these limitations, we can imagine there could surely be a product so bad that it would have had a significant negative preference. for an extreme example, we can imagine creating a ‘product’ which simply returned random citations from a list, unrelated to the query. surely that would have performed poorly even in this experiment. in retrospect, it would in fact have been useful to include such a ‘service’ in the study as a control. that few significant preferences were demonstrated in this study, can, i think be taken as at least some evidence that all products tested (with the possible exception of scopus) perform ‘well enough’ for supporting basic search usage scenarios across our population served. individual pairings while few of the individual pairing results rose to the level of statistical significance, possibly in part due to small sample size in each individual pairing, it is worth looking at them for trends suggestive of further research and possibly valid tendencies not captured with statistical significance. scopus ‘lost’ to every other product it was paired with — and produces the only statistically significant pairings, losing to both eds and summon with statistical significance. ebsco ‘traditional’ api, the other ‘traditional’ (rather than ‘next generation discovery’) product, does surprisingly well, holding its own — winning (although by a slight and not statistically significant amount) when matched with every other product included! the three ‘discovery’ products are fairly tightly clustered, although there are some trends. primo lost to every product it was paired with, although by fairly small not statistically significant margins. summon won, but by a small and not statistically significant margin, against every product it was matched against — except eds, where it was an exact tie (14 vs. 14 with 7 ties). eds has very mixed success — it beats scopus with statistical significance, but primo only by a very slim non-significant margin (16 to 14 with 8 ties); ties with summon; and is beat by ebsco ‘traditional’. what criteria do participants use to judge? the study gives us no direct evidence of what criteria any individual participant made in selecting a preference. we can categorize hypothetical criteria for preferences into two groups: differences in items in result sets, due to different corpuses or different ‘relevance’ ranking algorithms across products. differences in presentation that could not be normalized out in implementation — primarily presence or absence of abstracts and query-in-context highlighting, and in the case of eds in formatting of citation details. valentina artemieva at the montgomery county campus of sheridan libraries (johns hopkins libraries) did in-person observed run-throughs of the experiment with 5 separate students (4 grad students, one undergrad). she specifically asked them to tell her why they chose the preference they did — all five independently said that they decided based on preferring results with ‘more information’, such as including abstracts. we don’t know for sure if this is representative — but it could indicate that, at least for students, just about any set of reasonable results are ‘good enough’, and they focus on presentational issues. to the extent completeness of presentational context was valued, it could explain why scopus did poorly. scopus is the only product which had no abstracts at all available. (scopus as a service certainly has abstracts, but their api does not provide them, and their terms of service for federated search in fact prohibit presenting them). it’s possible this harmed scopus in participant preferences, although i think it’s probably also true that actual items in result sets were a factor as well: ebsco ‘traditional’ api, while it does provide abstracts, is the only other product in addition to scopus that lacked ‘query in context’ bolded highlighting of search terms. interestingly, this does not seem to have harmed ebsco ‘traditional’ in participant preferences. comparing ebsco ‘traditional’ and eds is particularly interesting, as they are from the same vendor. ebsco traditional is essentially a subset of the eds corpus — eds searches everything our ebsco traditional api setup does, and more. they probably use similar ‘relevance’ rankings. one would expect eds to do substantially better than ebsco traditional, being a newer product, with many enhancements and more coverage, from the same vendor. yet, this did not happen. ebsco ‘traditional’ in fact was preferred substantially more than eds — 13 to 5, not a statistically significant level in part due to small sample size, but striking nonetheless. this pairing of two products from the same vendor, perhaps unsurprisingly, also had a higher number of ‘ties’ (20, 52% of all selected preferences!) than any other pairing. eds does offer a version of query-in-context highlighting, while ebscohost ‘traditional’ offers only abstracts; eds was not preferred over ebscohost despite this. eds is also theonly product that does give us sufficient granularity in the api response allow us to format citation presentation in a standard way; it’s possible users reacted negatively there. it’s also possible the smaller coverage of ebscohost ‘traditional’ was actually to its advantage in this test – it’s a smaller corpus, but also a more focused one, perhaps giving better results for topic searches. if our participants had been searching for known-item titles (they largely were not), the smaller corpus may have been more detrimental. speed of response between products the deployment of the study gave us a rare opportunity to compare the speed of response of the various products, side by side. the bento_battle blinded survey instrument was written to record the duration it took to retrieve and prepare the responses for each query. this measurement is of total end-to-end time, including software parsing and normalization of the response — however, spot checks suggest that the time waiting on http responses from third party service is the great majority of total time elapsed. it should be kept in mind that these results could vary at different network locations, with different software implementations, or as the various products under testing continue to evolve. however, it still seems useful to share our findings with those caveats, as any head-to-head speed of response comparison would be otherwise very difficult to come by for interested potential customers. over the month the study ran, every time a query was made to a third party service, the elapsed time to retrieve and prepare results was recorded. there are more timing results recorded for a given product than actual user preference selections involving that product, because some portion of participants chose to abandon the task after doing a search but before selecting a preference. product count mean median 90th percentile 95th percentile max slowest ebscohost 265 2658 2388 3983 5273 33118 eds 252 3738 2826 5899 8418 27719 primo 277 2356 1611 4545 5878 16280 scopus 261 961 939 1340 1474 3123 summon 261 961 670 1853 2384 7465 response time descriptive statistics, by product, in milliseconds of the ‘discovery’ solutions, summon was definitely the fastest with a median response time under one second — and consistently fast.  even its 95th percentile slowest response was still a fairly reasonable 2.3 seconds. summon, like all the products, did exhibit some extremely slow responses (possibly due to network hiccups, or cold caches on the vendor’s side), but not very many and even its slowest response was faster than other products’ slowest responses. eds was by far the slowest of the ‘discovery’ solutions, and while its median response time was a barely acceptable 2.8s, it had some really pathologically terrible response times in 90th percentile and slower (10% of all responses!). this is likely due to the cumbersome and multi-step authentication process required by the eds api. in even the typical query, two http requests were required to eds (one to get a session token, one to actually execute the query), and in the worst case four http requests are required. eds’s cumbersome and slow authentication process is discussed below in the api review section. primo performed somewhere in between summon and eds. of the non-discovery solutions, scopus’s response time was comparable to summons, while ebscohost traditional api’s performance will vary based on the number of ebsco databases you choose to query. however with the set of around 40 we chose to include, ebscohost traditional was surprisingly slightly faster than eds, perhaps because it did not require the multi-layered authentication process. further research directions? all code used to do this experiment is open source and designed to be easily re-usable. others could do repeat experiment to validate or reproduce; or repeat with changed configuration to see if that changes the results. although i’m interested in how configuration and limiting choices may have changed our results, i suspect a trial done much like this one — with 5 or more products compared — would probably still result in no significant preferences detected. it might be more useful to limit the options considered to only two products, to increase the sample size of that single pairing with the same overall participation. alternately, it might be useful to do a comparison only with librarians participating — it may be that patrons are just not able to predict their actual long-term preferences or benefits from an artificial experiment like this. as an example of a comparison using only subject-specialist librarians, see bietila and olson, “designing an evaluation process for resource discovery tools” in planning and implementing resource discovery tools in academic libraries, igi global 2012. or the slides from a niso presentation by bietila, available open access at: http://www.niso.org/news/events/2012/nisowebinars/discovery_and_delivery/ if there were a way to creatively design the study to be more confident you were eliciting more realistic queries for actual real world research questions and research needs (including known-item searches), it would produce results one could be more confident in. it’s not entirely clear how to do this, although making existing search tool results pages the exclusive entry point to a study instrument might be beneficial. whether focusing on subject experts or not, whether in an experiment similar to this one or not, libraries will continue to have a need to evaluate and compare discovery and article search products. and often, especially if a product’s api’s will be used to create a local ui, it will be beneficial to focus on search results rather than presentation — to compare in a normalized presentation (whether blinded or not), rather than using vendors’ native interfaces with very different ui’s. the bento_search rails gem is intended to be useful in allowing rapid development of testing or prototyping environments for such examinations. another option possible with the bento_search tool would be deploying multiple products in an actual production application, using an “a/b testing” approach to give provide different users with different search products, and tracking analytics to try and determine which product seemed more useful (say, which product resulted in more clicks on results). one could also imagine this preference comparison approach (or an a/b approach) being useful to test multiple configuration options for a single tool one already has licensed access to. one of the biggest barriers to do any kind of comparison between products is the difficulty of getting access to evaluate a product. libraries are increasingly, and rightfully, attempting to gather more evidence before making major decisions (such as purchasing an expensive ‘discovery’ product). ideally, vendors would make it quicker and easier for libraries to get evaluation access in order to do this. conclusion many of us working with library technology would like to incorporate more user-centered assessment into our planning and decision-making process, including quantatitive investigations where appropriate. one barrier that can keep us from undertaking such investigations is how overwhelming it can be to “get it right”, and our worry that it’s not worth doing unless it’s done perfectly with absolutely minimized limitations. i do think it’s important to spend some time understanding the limitations of any research we do (and any research has limitations). but i think it’s better to do more assessment even with limitations, than to do minimal assessment out of desire to only do perfect assessment. if you are aware of the limitations of your conclusions, than almost any assessment can give you some additional information to make better decisions than you would have without it. any time you incorporate assessment into your planning, you gain experience to do it better (and more efficiently) next time. on those grounds, i consider our research here to be a success. we made a foray into incorporating user-centered assessment into our planning, becoming more comfortable with that process and increasing our experience and ability to do it. we believe this research has been valuable despite its limitations, and that additional time spent perfecting the research design would not have had a proportional benefit to our findings. the way this research project allowed us to become familiar with the various projects being investigated–both on a technical and a functional level–was also invaluable in helping us to be more comfortable in our decision-making involving some potentially expensive purchases and projects. the technical implementation aspects of the study were intentionally designed to serve as a proof-of-concept of one possible production implementation path, and an opportunity for technical review of the various products. i consider this aspect of this project a complete success, and will definitely try to find opportunities in the future to efficiently undertake that kind of evaluation as part of planning processes. i think it’s absolutely vital to get full evaluation/trial access to expensive products being considered for licensing, and to find ways to take advantage of that trial access for in-depth technical evaluation, not just a quick surface-level perusal. the actual conclusions we were able to make from the quantitative empirical portion of the study are limited, with few statistically significant differences between products demonstrated. in the absence of such demonstrated preferences, we think there is some reason to believe that end-user’s own preferences with regard to core search functionality truly will not significantly differ between these products, and we can focus mainly on other factors in our decision-making. we are considering moving ahead with an initial deployment based on a product we can use with our already existing licenses, such as the ebscohost ‘traditional’ api. our ability to quickly pivot to another article search results provider increases our confidence in this approach. the ‘bento style’ interface choice is part of what makes this kind of switch-out feasible, especially powered by the ‘bento_search’ ruby gem which aims to support this kind of switch-out. we will try to set expectations at the start of implementation to allow for such a quick switch-out, based on librarian and user response to the initial implementation in real-world practice. our initial implementation plan will include the commitment to do more assessment of how well the service is working within several months of initial roll out.   appendixes developer’s api review notes [1] http://bibwild.wordpress.com/2012/10/02/article-search-improvement-strategy/ [2] only after our study was well underway did we notice that microsoft is promoting a very similarly designed side-by-side comparison for bing marketing purposes: http://www.bingiton.com . [3] for example: asher, duke, and wilson. “paths of discovery: comparing the search effectiveness of ebsco discovery service, summon, google scholar, and conventional library resources”. forthcoming in college & research libraries, anticipated publication date: july 2013, online pre-print available now: http://crl.acrl.org/content/early/2012/05/07/crl-374.short. [4] http://github.com/jrochkind/bento_search [5] https://github.com/jrochkind/bento_battle/tree/jhu-study-10sep2012 [6] https://github.com/jrochkind/bento_battle/tree/jhu-study-17sep2012 [7] https://github.com/jrochkind/bento_battle/blob/jhu-study-10sep2012/config/ebsco_dbs.rb [8] james p. purdy. “why first-year college students select online research resources as their favorite”. first monday 17(9). http://www.firstmonday.org/htbin/cgiwrap/bin/ojs/index.php/fm/article/view/4088/3289) disclosure participating vendors were given the opportunity to review a pre-print draft of this article and provide feedback to the author. additionally, vendors were invited to submit public comments on the article. about the author jonathan rochkind is a software engineer at johns hopkins libraries, focusing on the libraries’ user-facing web presence. he blogs professionally at http://bibwild.wordpress.com. library-focused open source projects in which he is a committer and has made significant contributions include umlaut, blacklight, and xerxes. he can be reached at rochkind at jhu edu.   acknowledgements thanks to the various vendors who gave us trial access where needed, and technical support in all cases, to include their products in our evaluation. thanks to david kennedy, head of systems, and deborah slingluff, associate director for library services and collections, both at johns hopkins milton s. eisenhower library, for recognizing the value of this study to our organization, and supporting its undertaking. thanks to our local article search study committee for advising on study design and helping to market and promote the study. all mistakes are mine, however. developer review of products developers in the library sector know that library market vendor api quality can vary. often an api may look acceptable for desired use cases from the marketing, documentation, or even an initial look at a live demo. but sometimes, when developing for actual use cases, you can discover that an api is buggy, difficult to work with, or functionally incapable of achieving your aims. the implementation of the blinded survey instrument required development of code against the products’ apis, and provided the opportunity to review those apis for developer ease of use and functionality. the ‘proof of concept’ against the actual api’s was perhaps equally as valuable as the study itself. as few developers have the opportunity to compare these apis side by side, sharing what we’ve learned seems a valuable service. however, it should be noted that this review is one developer’s subjective opinion based on his perspectives and concerns and the information available to him. one common thread among all products was the lack of documentation of the quality and completeness a developer would want. most products offered some level of documentation, but most were incomplete, leaving out important details related to error responses, input formats, or content limitations. also, in general the ‘discovery’ products all offer some form of “limit to my libraries holdings” and “limit to items available in online full text”, but this aspect was not explored. the traditional a&i products (scopus and ebscohost ‘traditional’) do not offer such features. the discovery products also, unlike the traditional a&i products, typically offer a did-you-mean/spellcheck feature, but i did not investigate the quality or ease of development use for this feature in any products. ebscohost traditional this is not a discovery layer and lacks ‘discovery’ features, but the api is actually quite reasonable and easy to work with for what it is, as well as reasonably well documented. in using ebsco ‘traditional’ api, you are cross-searching a variety of ebsco databases licensed to you. different ebsco databases can use different controlled vocabularies for such values as format, and metadata completeness may vary between databases. with apis that come bundled with licenses mostly used for native html app access, there can sometimes be a worry that the api is an afterthought not fully supported — especially for ‘last generation’ products like ebscohost (rather than eds). however, the ebsco ‘traditional’ api seems fairly mature, and basic questions on the api through ebsco’s usual support mechanisms were generally answered satisfactorily. notable features for records where it’s available, you can ask for complete fulltext as html in the response. this is a fairly unique feature, although i do not have concrete uses for it in mind. peer-review limit is available, although not all databases may record this info. while extremely limiting ‘facet’-like functionality is available from the “getcluster” api call, when cross-searching multiple databases, the facets are limited, and in all cases they don’t actually have facet counts. the api faceting functionality is not equivalent to that of the native interface. there is sufficiently granular citation metadata to create an openurl or export a citation, and it’s rare that a record is missing proper data for these purposes. api response does clearly indicate if full text is available for a particular record on the ebsco platform for your institution’s licensed entitlements (and whether it is in pdf or just full text), and provides a link to pass the user on to the ebsco platform record-level page. (links directly to pdf take an additional per-record api call to retrieve, which is inconvenient.) abstracts are included. limitations or issues licensed databases are not automatically available for searching through the api. rather, one needs to explicitly configure each database for inclusion in your ebscohost api profile. if a database formerly available becomes unavailable (due to licensing or platform changes), and your api requests still ask to search it — it’s a fatal error and no response is returned, even from remaining available databases. the fact that you are cross-searching multiple databases can lead to some data normalization issues since content types are not particularly consistent across databases. the query syntax is poorly documented. < is not the same as <>. some changes to query parsing may be available by asking ebsco support to enable a different mode on your profile, but not by a self-editable field in the admin interface. some kinds of query syntax errors (such as “a and b”) can result in a zero hit response, without an error message. while there is a language tag in output, it seems to quite frequently be empty or improperly set to “english” even for non-english articles. relevancy ranking is decent, but can sometimes be slightly odd. a single phrase-quoted search on a title can result in the article matching that title exactly being number 2 or 3 on the result list instead of 1. unlike the ‘discovery’ products, there is no “query in context” search term highlighting whatsoever. api authentication account/password passed in ordinary url query parameter. end-user authentication as an ordinarily a&i type database, we assume we are not allowed to show results except to end-users that have been authenticated as affiliated with our institution. documentation there actually is a fair amount of documentation on the public web, although it’s scattered in several places and can be hard to locate. ebsco also maintains an extensive support knowledge base that includes entries on api issues as well. as is typical for all these products, response formats are mostly undocumented and need to be reverse-engineered, but a dtd is provided for the xml response (http://support.ebsco.com/eit/docs/dtd_eit_ws_searchresponse.zip), which is better than nothing, although insufficient at documenting semantics. some notably useful pages: http://support.ebsco.com/eit/ws.php http://support.ebsco.com/eit/ws_faq.php http://support.ebsco.com/eit/ws_howto_queries.php http://support.epnet.com/knowledge_base/search.php?keyword=&interface_id=1082&document_type=&page_function=search http://eit.ebscohost.com/pages/methoddescription.aspx?service=/services/searchservice.asmx&method=search http://support.epnet.com/knowledge_base/detail.php?id=5397 eds the eds api presented a number of challenges, chief of which was dealing with authentication. while eds is provided by the same vendor as the older more basic ebscohost ‘traditional’ api, eds was in many ways more challenging to work with than ebscohost. the eds api does offer some of the ‘discovery’ functions missing from ebscohost traditional api, but it lacks the granular citation metadata found in the ebscohost traditional api product. notable features there is a sophisticated query syntax that should support fielded search and complex boolean expressions. while it’s not clear if it’s suitable for exposing directly to the end-user, it’s reasonably okay to work with in software. there are several different ‘modes’ of query parsing which you can select via api. it’s somewhat confusing what the differences are and which you might want. there is a convenient multi-parameter query style that would be useful for sending multi-field queries entered by users in separate text boxes. a ‘peer reviewed’ limit is available. there is a query-in-context highlighting function, but it’s very cumbersome to use, and only applies to certain metadata fields (google-style fulltext snippets not available). faceting is available, including on content type and langauge, but i did not investigate this extensively. limitations or issues see api authentication section below, there are some significant challenges. the citation metadata (journal name, volume, issue etc) are offered only in a single text field meant to be displayed directly to the user. there are no granular metadata elements allowing openurl generation or citation export, though you can configure the api to provide a pre-configured openurl in the response. oddly, this single text field provided in api response isn’t an end-user presentable literal string, but includes undocumented xml-like tags which must be stripped before presentation to end users. this single text field of citation information is missing from some items, even when the underlying platform is able to provide an accurate openurl for the item. other data elements that are available for facets or limits are not available in the api response for an individual element. a notable example would be the language of the article. the per-record api response is very limited. data tends to be html, with presentational html tags like <b> and <i> as well as html character entitles. but this is somewhat inconsistent, and undocumented. while eds does offer better ‘content type’ limiting and labeling than the ebscohost traditional api, the content type vocabulary used applies to container of the record rather than the record itself — “academic journal” rather than “journal article” — which is a somewhat odd ui. additionally, the terms used for labeling records in results don’t exactly match the terms you need to use for limiting (singular vs plural), which can make coding a flexible ui more challenging than it should be. date sorting reveals metadata inconsistencies limiting the usefulness of this feature. the facetting api does not allow exclusionary facets (eg exclude “newspaper articles”). api authentication the eds api’s authentication process is quite complex, and a significant challenge to writing client software. making an http request to receive an authentication token. this token will need to be repurposed for all subsequent requests to the api. since this token will expire after a period of inactivity, all responses need to be evaluated for token expiration errors. if a token expires, the client must re-authenticate and request a new one. using the authentication token, the api requires a second http request to generate a per-session token. the api requires the session token to be utilized for all future api interactions for a particular session. since it can be difficult to track end-user sessions effectively within a web application, and even if you did so a given session in an app of this type may only make one or two queries, it might make more sense to initiate a new session for every query, despite the significant performance cost of doing so. ebsco’s explanation for this session token is that some of the underlying databases have licensed connection limits, so they need to keep track of how many sessions are currently accessing. however, there’s no self-service way to tell which underlying data sources may have connection limits, or what those connection limits are. it’s unclear what will actually happen if the connection limits are exceeded, and the nature of the error message returned, if any. this cumbersome authentication/setup process significantly adds to the difficulty of writing a client for the eds api. it also degrades the effective response time to deliver results to the end-user, as in some cases several separate http requests have to be made — and each http request can have a one second or more response time. end-user authentication eds does allow use with both authenticated end-users, as well as public/guest end users. if you tell the api you are using public/guest access, it has a rather novel way of accommodating this. other ‘discovery’ products typically will transparently hide some results from the result set for ‘guest’ access. eds also suppresses some results from the result set, but not transparently — instead, in a page of 10 results, several (or more) of them may be replaced by placeholder “can not show this result to guest user” blocks. i am not certain if this results in a better or worse ui than what the other products do, there are some arguments both ways. documentation documentation is quite limited, and behind a customers-only login, which makes it inconveniently unavailable via a google search.http://edswiki.ebscohost.com/eds_api_documentation “console” app at https://eds-api.ebscohost.com/console is quite useful for debugging and exploring the api in a browser, without having to deal with the insane auth/session workflow manually. primo a decent api, which allows you to do what’s needed. getting per-record metadata from the response format can be very confusing and under-documented, although on the plus side extensive metadata is usually present. notable features query syntax does allow fielded searches. it is definitely not suitable for end-user direct entry, and can require some contortions to search multiple terms/phrases. it is also unclear if complex boolean expressions are supported. the query parameter in request can be repeated, which is convenient for multi-field searching. supports facets, including creator, language, format, subject, pub year, db source, journal title. supplies abstracts has a ‘peer reviewed only’ limit. custom relevancy rankings based on discipline and individual status (faculty, student, etc) are available. i did not explore this feature. a ‘highlight’ feature is available, but is poorly documented and difficult to use. this feature seems to only highlight certain metadata fields, and does not provide google-style highlighted full text snippets. limitations or issues individual record response elements have many sections with similar data presented in different ways, without much documentation of the different response elements. this makes it difficult to determine which response element will provide the best, most consistent data for a given use. these varying presentations of certain elements may in fact be powerful support for complex use cases, but lack of documentation is a barrier to doing so. error messages are sometimes xml and other times http status codes with html response bodies. expected error response specifications are not documented. during the survey, we ran into at least one strange edge case bug: searching for a long phrase with a question mark in it resulted in strange error message. this is possibly due to undocumented escape requirements of query input? multiple format/genre vocabularies are used in responses, but there is inadequate documentation defining expected values or semantics. queries with very many terms can have exceptionally long response times. phrase queries with very many terms seem to sometimes return false negatives. while granular citation metadata is provided, it can sometimes be incomplete for certain records and not sufficient to generate a working openurl. there is a management interface to specify which primocentral “collections” you want to include in your search. however, these collections are constantly changing, and i believe one must manually stay on top of changes to update one’s configuration to be searching all that is available. api authentication api is ip-address restricted for authentication, which is certainly easy to develop for, although it can be annoying if your server ip addresses can change. if your api request is not from a registered ip, you improperly get an http 200 response, with an empty body, rather than a proper http error response and/or machine-readable error message. end-user authentication access is allowed by both authenticated end-users and guest users. it is not clearly documented what differences in access levels are. documentation documentation is restricted to customer-access only, making it unavailable for non-customer evaluation. it is on one giant html page with somewhat hard to read formatting. documentation not present for the numerous controlled vocabularies, facets or fields available for limiting, some are mentioned anecdotally like peer-reviewed controls, but there is no complete list. http://www.exlibrisgroup.org/display/primooi/brief+search scopus a fairly reasonably designed, full-featured api for the basic search functions you’d expect from an a&i database. there are some limitations, including restrictive terms of service. notable features uses fielded and boolean search language equivalent to the scopus native interface, making sophisticated searches possible. there is an incredibly full set of search indexes, letting you craft very precise queries. has sufficient granular citation metadata (volume, issue, start page, issn, etc) to build an openurl, or export a citation (except only first author). limitations or issues the documentation says some very limited faceting is available, but i have not tried it. there is no limit to peer review/scholarly content only, although most content in scopus probably is peer reviewed/scholarly. only the first author of a multi-author citation seems to be available from the api, even though documentation suggests otherwise. sorting either doesn’t work at all, or doesn’t work as documented, especially for sort orders than ‘relevance’. no abstracts are provided (see tos issues below). queries that produce zero hits give an error message rather than simply reporting 0 hits. this was undocumented, and discovered through ‘reverse engineering’. unlike the ‘discovery’ products, there is no query-in-context search term highlighting. api authentication scopus requires your api client ip address to be registered, and an api key additionally sent in an http header. the api key sent in the http headers makes debugging or exploration of api in a browser window difficult. other products provide a ‘console’ app of some sort to work around this, but scopus does not. end-user authentication authenticated end-users presumably required to display any results from this licensed a&i database. documentation decent documentation is on the public web, but is occasionally out of date or lacking significant details. http://www.developers.elsevier.com/devcms/content-api-search-request http://api.elsevier.com/content/search/ (scopus section) terms of service limitation the scopus api terms of service (under ‘federated search’ case at http://www.developers.elsevier.com/devcms/content-policies) are unusually limiting. link back to scopus is required with presentation of results. displaying abstracts or citation counts in an interface is prohibited. lack of abstracts may be particularly problematic for many use cases. summon serialsolutions tells us that their default out-of-the-box summon interface uses the exact same api available to customers to power it’s functionality. this kind of “dog fooding” is probably unique among the products reviewed, and it pays off. while not perfect, the summon api is by far the easiest to work with, most consistent, and most flexibly powerful of the products reviewed. notable features sophisticated query syntax, which is actually suitable for exposing directly to end-users if so desired, as it supports google-style list-of-terms-and-phrases with reasonable semantics. it also supports fielded queries, and at least some level of boolean expressions. however, the search syntax isn’t documented. it appears to be based on solr-lucene query syntax, but with enhancements. lack of documentation can make things tricky when you are machine-generating queries; it can be challening to be sure you are escaping what needs to be escaped, and in general that your machine-generated query has the intended semantics. full featured faceting api, including applying ‘and’ (conjunction), ‘or’ (union) and ‘not’ (exclusive) facet limits. large list of search indexes you can target and combine. this is useful for supporting finely targetted queries. however, in some cases, combined/aggregated indexes you might want are missing, such as an “any kind of subject” index as opposed to only “subject geographical”, “subject topical”, etc. also, there is insufficient documentation as to semantics of search indexes, beyond a one phrase label. both a “just scholarly” and a “just peer-reviewed” limit (not sure what the distinction is). a good vocabulary for ‘content type’ that matches my impression of what would be useful to users. however, mis-classification of item format is noticable, for example something classified as a “journal article” that is from a mass market publication nobody would consider a ‘journal’. straightforward response format, with sufficiently granular data elements to generate openurls or export citations, and in general has what you’d expect it to have, in reasonable places. the best query-in-context search term highlighting of any product reviewed, with google-style highlighted snippets from fulltext, that works quite well. an interesting ‘direct link’ feature which aims to provide a url to send the user directly to institutionally-licensed fulltext for an item, avoiding the openurl resolver where possible. i believe this feature supports multiple content providers, not just those owned by serials solutions parent company. i did not investigate this, but if it works well it could be a very useful feature. in limited experimentation, it did sometimes successfully link even to public-access pre-print copies on the web, which is a pretty neat feature. (linking sources can be ranked and turned off in config). a ‘recommend a specific database’ feature, to send users to specific native database interfaces for databases matched to their query. i did not investigate the quality of results, but it is an interesting feature. limits to local holdings and fulltext-available-only are based on the serialsolutions 360 knowledge base, which would be convenient if you are already using this vendor for other products requiring knowledge base configuration. limitations or issues while api returns a ‘language’ code labeling individual responses, it is frequently empty or labeled as english even for non-english articles. there were some problems applying multiple exclusive (“not”) facet limits. it’s not clear if i was encountering a bug in the api or if i was doing something wrong while there are a variety of facet groupings available, there is a lack of clear documentation of what facet groups are available. the developer has to reverse engineer from observed responses. api authentication you need to authenticate your api requests using a fairly complex (but technically secure and competent) algorithm involving cryptographic signatures. the requirements are actually quite similar to aws api (although it is not identical logicimplementing a client complying with summon’s authentication method would be a a bit challenging to implement from scratch, but libraries are provided in several languages to wrap api access, including authentication. i did not want to use the api for all requests; i prefer to stay closer to the http metal and not use an extra layer of abstraction (with fairly spotty documentation) limiting my flexibility. fortunately, the provided ruby implementation of a summon api client was written such that i could easily instantiate and call the right classes to use the request-signing parts of the ruby gem alone in my own code, without using the rest of the gem. end-user authentication as is typical for the ‘discovery’ products, you can provide access both to authenticated local affiliate end-users, and ‘guests’. ‘guest’ access has some restricted results suppressed from the result set. documentation the api is reasonably documented, although it could be better in many places. as with the other products investigated, we could use more documentation of response format semantics and various controlled vocabularies. particularly missed is sufficient documentation of the facet groupings and search field indexes. docs are on the web, with public access, which is great. http://api.summon.serialssolutions.com/help/api/search http://api.summon.serialssolutions.com/help/api/search/fields useful ‘console’ app for debugging/exploring api in a browser window without having to deal with complex auth system manually: http://api.summon.serialssolutions.com/help/api/search/example?s.q=query subscribe to comments: for this article | for all articles 4 responses to "a comparison of article search apis via blinded experiment and developer review" please leave a response below: ebsco, 2013-01-14 jonathan, ebsco agrees that “creating a better article search experience for our users” should be a primary goal for libraries choosing a discovery service. your study’s focus on apis and least common denominator metadata presentation is one data point. other complementary studies have been conducted this year to answer the question you acknowledged wasn’t answered by your study – “we do not know why participants preferred one tool to another”. we wanted to point out one study in particular conducted by librarians at illinois wesleyan (eds) and bucknell (summon) – “paths of discovery: comparing the search effectiveness of ebsco discovery service, summon, google scholar, and conventional library resources” [http://crl.acrl.org/content/early/2012/05/07/crl-374.full.pdf]. this study focused on student search behaviors, speed to success, and quality of research in comparing a variety of services. in addition to ranking eds first, this study came to several interesting conclusions on searching. nearly all of the negative issues you encountered with the technical features of the eds api were either a result of not using best practices (which we would have suggested if your institution were a customer) or are features we have added to the api since your study began in may 2012. details and more complete commentary: http://www.ebscohost.com/uploads/general/johns_hopkins_api_comparison_ebsco_feedback.pdf serials solutions, 2013-01-14 this study does a good job of illustrating the value, performance and ease of use of the summon api, but doesn’t address the fact that library discovery services must do more than simply provide a single search box for article content. with its unified index architecture, powerful relevance-ranking and ability to present precision results in a clean, intuitive and user-friendly interface, the summon service moves users beyond searching siloes of content. the summon service increases usage of more than just popular aggregator packages by mapping content from all providers to a common schema to ensure equal treatment of content and therefore boosting visibility and access to underutilized collections. and, as this study suggests, user experience and results presentation are paramount to users’ perceptions of a discovery tool’s effectiveness. user experience is a critical factor when evaluating discovery solutions. [see external report] the summon service provides layers of contextual research assistance and opportunities for librarians to impact the research process to optimize user experience—all without using the api. this is why the vast majority of customers who do use the summon api for building custom applications also present users with the full-featured, “out-of-the-box” interface to meet a variety of user needs and expectations. emily lynema, 2013-01-16 jonathan, i’m so excited that you did this study. i remember talking about it at code4lib last year. i’ve only skimmed this report and look forward to reading it in more depth. awesome job highlighting the statistical significance (or insignificance) of the results. but i’m not highly surprised about the results. student rankings are often fairly indifferent in terms of relevance. it would indeed be interesting to include some librarian rankings in this set for comparison. it also emphasizes that some of the differences in these tools have little to do with relevance of result sets — things like the usability of the interface and the scope of content included also play a big role. we’ve been doing a periodic re-evaluation of the playing field here, and i’m forwarding this on to our staff for digestion. thanks again! dmitry green, 2014-11-26 hello, thank you for this article. as a physics phd (yale 2001), like many people, i have been frustrated with many of the search interfaces. here’s a tool that i have been building with my team: arximedes.org is a new interface that helps professional researchers separate the wheat from the chaff. at the moment we are focused on physics & technology (it is not a social network and we don’t spam), having integrated nasa ads and arxiv. in addition to getting a laundry list of papers, as in standard search engines, we’ve built another processing layer on top. you can see which authors are driving research within the topic you are searching, visualize publication rates, and track who is citing whom. a newsfeed automatically organizes daily updates to your searches. we’ve also built a simple way for you to rate papers. unless you are an expert in a very narrow field, it is hard to make sense of the literature. at the same time, most professional researchers “know it when they see it”, quickly. you can take register and use it for free. here is a brief video: http://youtu.be/kj_q7iqnq8o and pdf overview: http://arximedes.org/static/project/images/arx_overview.pdf. thank you leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – barriers to initiation of open source software projects in libraries mission editorial committee process and structure code4lib issue 29, 2015-07-15 barriers to initiation of open source software projects in libraries libraries share a number of core values with the open source software (oss) movement, suggesting there should be a natural tendency toward library participation in oss projects. however dale askey’s 2008 code4lib column entitled “we love open source software. no, you can’t have our code,” claims that while libraries are strong proponents of oss, they are unlikely to actually contribute to oss projects. he identifies, but does not empirically substantiate, six barriers that he believes contribute to this apparent inconsistency. in this study we empirically investigate not only askey’s central claim but also the six barriers he proposes. in contrast to askey’s assertion, we find that initiation of and contribution to oss projects are, in fact, common practices in libraries. however, we also find that these practices are far from ubiquitous; as askey suggests, many libraries do have opportunities to initiate oss projects, but choose not to do so. further, we find support for only four of askey’s six oss barriers. thus, our results confirm many, but not all, of askey’s assertions. by curtis thacker and charles knutson motivation the mission statement of the american library association includes the charge to “ensure access to information for all.”[1] this charge comes without restriction, cost or qualification. stated another way, libraries make information freely available to all, regardless of how that information is to be used. similarly, open source software (oss) “licenses must permit non-exclusive commercial exploitation of the licensed work, must make available the work’s source code, and must permit the creation of derivative works from the work itself” (laurent 2004). the core values of libraries and the oss movement are similar, suggesting that libraries should tend to favor the oss model. in particular, they might feel a responsibility to share the code they have developed with other libraries in a spirit of openness and access for all. that libraries are predisposed to oss adoption and contribution is not a new idea. pat eyler, an open source developer for the koha ils project, said “that more librarians aren’t actively using and evangelizing free software is an indictment against us for not letting them in on our secret” (eyler, 2003). nicole engard characterized the issue this way: “it has been suggested that libraries are almost ethically required to use, develop and support open source software” (engard, 2010). richard stallman, the pioneering free software evangelist, stated that “… universities shouldn’t be developing proprietary software. it is better if they develop none at all, because [by doing so] they are betraying their mission to contribute to human knowledge” (anderson, 2008). despite the suggestion that libraries are ethically obligated to use and create oss, it has been observed that libraries seem reluctant to share their code. in 2008 dale askey authored a column in this journal entitled “we love open source software. no, you can’t have our code.” he states that “librarians are among the strongest proponents of open source software. paradoxically, libraries are also among the least likely to actively contribute their code to open source projects” (askey 2008). askey identified a list of six issues he believes contribute to this dichotomy. in his own words: after pondering this issue for some time, i identified the following issues as the driving forces that undermine the sharing of open source software in libraries: perfectionism – unless the code is perfect, we don’t want anyone to see it dependency – if we share this with you, you will never leave us alone quirkiness – we’d gladly share, but we can’t since we’re so weird redundancy – we think your project is neat, but we can do better competitiveness – we want to be the acknowledged leader misunderstanding – a fundamental inability to understand how an open source community works many of these issues operate in combination, but any one of them is sufficient to thwart the development and adoption of open source software in libraries. in this paper, we report on our empirical investigation into askey’s central claim. we examine the six barriers he proposes in light of our empirical results. methods the association of research libraries (arl) “is a nonprofit membership organization of 125 research libraries in north america. the association operates as a forum for the exchange of ideas and as an agent for collective action.” each year arl distributes and publishes a small number of surveys, called spec kits, that are proposed and designed by librarians and other interested parties. in february 2014, arl distributed a 32-question survey authored by curtis thacker, charles knutson, and mark dehmlow, to 125 member libraries. seventy-seven libraries (61%) responded to the survey, the results of which were subsequently published as spec kit 340: open source software (thacker et al. 2014) (hereafter referred to as “the spec survey”). the purpose of the spec survey was to study arl member libraries’ adoption and/or development of oss for the primary functions carried out in libraries. we aimed to understand organizational factors that affect decisions to adopt oss. with regard to development of oss, we studied: 1) research libraries’ policies and practices on open sourcing their code; 2) the frequency of research library contributions to open source projects; 3) the reluctance of research libraries to make their code openly available; and 4) the most common benefits and challenges encountered when research libraries open source their code. questions were reviewed, evaluated and refined by empirical software engineering researchers from the sequoia [2] lab in the brigham young university computer science department. this exercise enabled us to deepen our understanding of issues related to open source software development by applying the growing body of work in the area of empirical software engineering. the creation of the survey instrument followed best practices for empirical software engineering surveys (kitchenham and pfleeger 2008). questions were crafted to empirically test several of the issues laid out in askey’s column. in particular, the following question provided respondents with an opportunity to identify reasons for not openly releasing software they had developed: example spec kit question the table below illustrates the relationship between the options presented in the question and the issues presented by askey. the first column identifies each issue as presented in the survey, while the second column presents the issues as stated by askey. two of the issues offered by askey were not tested because they fell outside the scope of the spec survey. two other issues were added in an attempt to validate additional reasons for which an institution might choose not to open source their code. of these two issues, the second one (“seeking to license or sell the system”) was inspired by a response[3] made to askey’s column. survey results were reviewed and statistically analyzed. free response questions were encoded and qualitatively analyzed for themes and best practices. the executive summary of the spec survey includes an overview of statistical results that spans the entire survey (thacker et al. 2014). a specific set of results relevant to this paper are presented and discussed in the sections below.arl reviewed and administered the survey. participants were given four weeks to respond and arl sent two email reminders as the deadline approached. a spreadsheet of the complete response data was returned to the authors for analysis and preparation for publication. oss adoption askey’s initial premise is that libraries love oss. he cites dan chudnov (chudnov 2007) who asserts that infrastructure software and programming languages are widely adopted by libraries. operating systems such a linux, web servers such as apache, and programming languages such a ruby and java are examples of oss systems commonly adopted by libraries. these applications compete with commercial applications for market share and often hold the largest slice of the pie. askey also pointed out that oss adoption is ubiquitous for other common types of software applications such as web browsers (such as mozilla) and mail clients (such as thunderbird). market share statistics for linux[4], apache[5] and mozilla[6] substantiate these claims. the spec survey found that 74 respondents (97%) had deployed open source software in their libraries, suggesting that, at least for arl libraries, adoption of oss is essentially ubiquitous. this data strongly supports askey’s claim that libraries love oss. we also wanted to understand the specific types of oss that are loved by libraries. askey asserts that libraries have “strongly embraced…object repositories such as dspace and fedora and content management systems such as drupal.” spec survey respondents were invited to provide information about the type of software being used for various purposes. respondents most frequently reported choosing oss solutions for institutional repositories (52 total), blogging (51 total) and digital preservation (50 total). see the table below for more details on how respondents have adopted oss within their institutions. the spec survey revealed compelling evidence for the widespread adoption of library specific software, even beyond askey’s claims.the spec survey confirmed askey’s sense that dspace and fedora were “strongly embraced” by libraries. sixty-six respondents reported the oss projects they had adopted. we found that the most commonly adopted open source systems were dspace (31 respondents, 47%[7]), fedora (21 respondents, 32%), open journal system (19 respondents, 29%), blacklight (14 respondents, 21%), hydra (12 respondents, 18%), vufind (8 respondents, 12%), archivesspace (7 respondents, 11%) and archivists’ toolkit (6 respondents, 9%). respondents were further asked to describe three benefits and three challenges associated with adopting oss. the most commonly reported benefit was the ability to customize the software (50 responses). other common themes included low cost or time to implement (27 responses) and association with an active community (27 responses). the most common challenge was the need for highly skilled staff that could provide support for the oss system (40 responses). other commonly cited challenges included poor documentation (19 responses), a need for additional training or expertise (16 responses), and substandard development practices (12 responses). oss contribution askey shares his perception that libraries are reluctant to initiate and/or contribute to oss projects, despite their nearly universal enthusiasm for adoption. askey’s main claim is: “where we tend to fall flat is in the area of creating, maintaining, and sharing library-specific applications. there are certainly myriad exceptions to this statement, but i would suggest that however large and noteworthy, they remain the exceptions, and not the rule” (askey 2008). while askey’s statement mainly addresses initiation of oss projects, maintaining library-specific applications could be interpreted as contribution to oss projects. askey’s column focused primarily on contributions to oss projects in the form of source code. beyond software, oss projects benefit from many types of contributions including, money, hosting, testing, etc. the table below shows the types of contributions that libraries have made to oss projects.[8] the spec survey found that 56 respondents (78%) had contributed to one or more open source projects; of these, 50 respondents indicated which projects they had contributed to. the most common projects included dspace (12 respondents, 24%[9]), fedora (11 respondents, 22%), hydra (9 respondents, 18%), kuali (6 respondents, 12%), blacklight (5 respondents, 10%) and archivesspace (4 respondents, 8%). the spec survey found that respondents had contributed to an average of 2.6 oss projects and a median of 1 oss project. these findings support askey’s claim that contribution to oss by libraries is common, yet far from universal. oss initiation askey addressed initiation of oss when he claimed that “where we tend to fall flat is in the area of creating, maintaining, and sharing library-specific applications. there are certainly myriad exceptions to this statement, but i would suggest that however large and noteworthy, they remain the exceptions, and not the rule” (askey 2008). thirty-two (42%) respondents identified themselves as the original developer of an open source project. respondents initiated an average of 1.4 oss projects and a median of zero oss projects. thus we see that while a number of institutions have some experience initiating oss projects, initiation is far from the norm. our finding supports askey’s claim. respondents were asked if any of their in-house software could have been, but had not yet been, released under an open source license. fifty-three respondents (69%) answered in the affirmative. additionally, the spec survey revealed libraries that always choose to share their sharable projects, and, conversely, there are libraries that could share but have thus far not chosen to share their code. the table below breaks down these responses in greater detail. perfectionism respondents cited all of askey’s barriers as reasons for not open sourcing a sharable system. we address each of these issues in the sections below. thirty-nine (74%) of those who chose not to open source their code cited “concerns that the code quality is not ready for public adoption.” the perception that the code quality is not acceptable, and therefore cannot be shared, is very common. this particular question in the spec survey was only able to test perceptions of libraries. as pointed out by askey, intrinsic to the open source philosophy is the idea that the community will improve upon an initial system. linus’ law, as described by raymond (1999), describes oss communities this way: “given enough eyeballs, all bugs are shallow”, or more formally: “given a large enough beta-tester and co-developer base, almost every problem will be characterized quickly and the fix will be obvious to someone.” it follows from linus’ law that not sharing code due to quality issues is more a matter of pride than practicality. dependency “nothing is more certain in the world than this: if you share software with someone, you will be asked to support it, even if you make it perfectly clear that you have no ability and no intention to do so” (askey 2008). forty-one respondents (77%) cited “staff time commitment required to support the community” as a reason for not open sourcing a product that could have otherwise been shared. the spec survey offers strong evidence that the perception of dependency is a common barrier among arl members. quirkiness quirkiness is defined by askey as “the sense that one organization’s needs are so locally-tailored that [it] would make no sense to release the software to the broader library community.” later in the same section he cites an example of quirkiness as dependence on “idiosyncratic local metadata scheme.” the spec survey addresses quirkiness in three ways. first, 30 respondents (57%) cited “dependence on internal systems” as a reason for not open sourcing a system that could have otherwise been open sourced. second, 7 respondents (13%) stated “it didn’t occur to us” as a reason for not open sourcing their software. third, the issue of quirkiness was directly addressed by respondents who entered free form responses describing reasons they chose not to open source a system. responses included: “highly customized to address local requirements”; “narrow niche applications where a community is unlikely to develop”; and “often these systems reflect local practices. we’ve not viewed them as useful beyond our local environment.” these data are evidence of quirkiness among arl members and support askey’s claims. redundancy redundancy, as described by askey, “is when there is perfectly acceptable software available and yet is rejected because it’s not quite what one would have done had they created the software.” we found that this issue relates more to adoption than initiation of oss. as a consequence, we did not study this issue in detail. competitiveness askey explains that libraries tend to implement their own systems (e.g., institutional repository, digital libraries, and web services) because they “want to be the acknowledged leader.” while one respondent of the spec survey indicated “a competitive desire to have the best system” as a reason for not open sourcing their software, no other respondent cited such motivation. as a result, while we find some support for askey’s claim, competitiveness does not appear to be widespread. misunderstanding askey describes misunderstanding as “a fundamental inability to understand how an open source community works.” we determined that  “misunderstanding” primarily suggested that respondents did not understand the benefits of involvement with an oss community. this issue represents a catch-all of sorts that encompasses the other issues we’ve discussed. the breadth of “misunderstanding” prevented us from testing this issue in the same manner as the other issues presented above. other questions in the survey do, however, offer insights into the benefits libraries currently enjoy as a result of adoption of and contribution to library-specific oss projects. we highlight some of these insights below. respondents were asked to describe three benefits and three challenges associated with contribution to oss. the benefit most commonly cited was engagement in the open source community (38 responses). other common themes included control of product features and direction (25 responses), and recognition/reputation (14 responses). the most common challenge was allocating sufficient staff time to make meaningful contributions (24 responses). other commonly cited challenges included writing generalized software for use by a larger community (7 responses) and securing the financial resources needed to support the open source project and community (7 responses). control of software emerged as a theme common to both adoption and contribution. those adopting oss products felt that access to source code gave them greater control, allowing them to change the software as needed, rather than being subject to the whims of a proprietary solution. those libraries contributing to oss projects felt that they gained greater opportunity to influence product direction, especially with respect to software features. in both cases, library information technology organizations perceived a sufficient benefit to their overall productivity to justify the expense of their involvement (as adopters, contributors, or both) in oss systems. when asked about reasons for open sourcing their project, spec survey respondents listed the following as being “important” or “very important”: a belief that open sourcing would lead to better software (30 respondents), a desire to contribute to an open source community (29 respondents), and shared effort in development and quality assurance of the project (27 respondents). the experiences shared by respondents who initiated an open source system support the idea that one way to inject quality into a system is to open source it. in contrast to askey’s claim, there were many respondents who demonstrated an understanding of this benefit of open sourcing their code. additionally, of the 54 respondents who have a system they chose not to release as open source, 24 (44%) have initiated at least one open source project. further research is required to understand the motivation of these arls decision to share one system but not another. many respondents expressed a desire on the part of their developers to share with and participate in one or more oss communities. larger lit organizations committed more resources to oss projects than smaller lit organizations, but we found no significant correlations suggesting a disproportionate level of commitment to oss projects as a function of lit staff size. the nearly universal adoption of oss systems and the high level of contribution to oss projects may suggest that adoption of and contribution to oss projects has entered the mainstream for lit organizations. simply stated, lit organizations that develop software have also generally contributed to one or more oss projects. additional insights in the final section of his column askey makes several suggestions on what should be done to overcome the issues he discusses. we address a few of these suggestions in this section. in 2008 askey claimed that there was no standard way of distributing library specific code, suggesting that a single place should be agreed upon as the established method for sharing code. github has emerged as the preferred method for many open source projects (including libraries) to share their code. github accommodates large oss projects such as fedora, dspace, hydra and others as well as supporting what askey calls oss lite[10]. forty-one spec survey respondents indicated that they use a public forge to manage and share their open source projects. thirty-eight of these use github for this purpose. while making use of an open source forge, such as github, to share code is effective, it is unclear whether this tool has impacted the propensity of libraries to initiate an oss project. askey states that “libraries that wish to use open source software need to understand the staffing commitment they are making by going that route. open source software requires programmers, interface designers, and system administrators.” in our review of organizations that contribute to open source projects, software development staff ranged from one or two to as many as fourteen.  while organizations that contribute to large-scale, formal open source projects were clearly investing heavily in programming staff, it was also clear that a few organizations that didn’t have resources for large technology staffs could still contribute to projects with as few as one or two programmers.  the median number of staff reported as working on oss projects was two, with an average of nearly four. the results of the spec survey suggest that we view organizational behaviors surrounding the adoption of open source software separate from contribution to oss projects. for example, while oss adoption is viewed by respondents as a means of saving time and resources, oss contribution is not similarly viewed. rather, contribution to oss projects is viewed as being advantageous for different reasons, namely engagement in an oss community. for developers, the sense of social involvement in a community represented by an oss project can be a positive source of professional satisfaction, ultimately leading to greater productivity and a return on investment for the lit organization. threats to validity care must be taken when generalizing survey findings to a larger population. the spec survey was distributed to all 125 arl member libraries. arl libraries are often considered a model for best practices, but are not a representative set of research libraries or libraries in general. further, the 77 respondents of the survey self-selected, introducing bias toward libraries that are interested or invested in oss. also, survey fatigue is a large concern. the spec survey was relatively long (32 questions), with some questions involving multiple parts and some requiring respondents to look up specific information in order to answer. several instances were found where respondents didn’t answer questions completely, which can be seen in the tables above. future work the spec survey revealed that there are libraries that always choose to share their sharable projects, and, conversely, there are libraries that could share their code but have never chosen to. future work could include looking for correlations between a library’s software engineering, talent management and innovation policies and practices, and its propensity to initiate oss projects. in the years since the publication of askey’s column two significant types of organizations have arisen within the library landscape, exerting considerable influence on open source software projects. governing foundations, such as duraspace, kuali, the islandora foundation, the software conservancy foundation and archivesspace, manage requirements and coordinate resources of member libraries. supporting vendors, such as bywaters and @mire, offer support and hosting services to oss adopters. while outside the scope of the research we performed, the impact of such organizations is highly relevant to the issues posed by askey and warrants further investigation. conclusion we found support for many of the issues presented in askey’s column. the majority of spec survey respondents have adopted and/or contributed to at least one oss project. nearly half of respondents chose to initiate one or more oss project. while most institutions have some experience with oss, most have only made an initial foray into the space. as askey suggests, many libraries do have opportunities to initiate oss projects, but choose not to do so. we found strong evidence supporting the existence of “perfectionism,” “quirkiness,” “dependency” and “misunderstanding,” however, “competitiveness” was extremely rare. thus, we find support for many, but not all of askey’s assertions. the emergence of github as a preferred means of sharing code was highlighted as a development since askey’s 2008 column. we would suggest that library information technology organizations participating in oss projects typically understand that they must dedicate technical personnel and other resources in order to do so. finally, we found that oss comes with a number of financial trade-offs that need to be carefully examined when considering adoption, contribution and initiation of oss projects. acknowledgements one of the authors (curtis thacker) spoke with mr. askey about his column and the work we were doing on this paper at the cni spring 2015 membership meeting in seattle. we appreciated his encouragement and insights in addition to his thought-provoking column which contributed inspiration for both the spec survey and this paper. authors curtis thacker is director of discovery systems in brigham young university’s (byu) harold b. lee library. he is also an m.s. candidate in the department of computer science at brigham young university and member of the byu “software engineering quality: observation, insight, analysis” (sequoia) lab. in addition to bachelor’s degrees in computer science and applied mathematics from brigham young university – idaho. curtis has 10 years industry experience and seven years library specific experience. dr. charles knutson is an emeritus professor of computer science at brigham young university, former director of the byu sequoia (“software engineering quality: observation, insight, analysis”) lab and former director of the byu mobile computing lab. dr. knutson has 27 years of industry experience, including engineering and management positions at hewlett-packard and novell, inc. he was also vice president of r&d at counterpoint systems foundry, inc. (now sybase ianywhere), the world’s leading provider of irda and bluetooth protocol stacks for embedded systems. dr. knutson has more than 120 technical publications in areas including mobile computing, medical informatics, and software engineering. he holds a ph.d. in computer science from oregon state university, and b.s. and m.s. degrees in computer science from byu. notes [1] see http://www.ala.org/aboutala/missionpriorities [2] sequoia = “software engineering quality: observation, insight, analysis” [3] see http://journal.code4lib.org/articles/527#comment-1299 [4] see operating system statistics at http://www.netmarketshare.com/ [5] see http://news.netcraft.com/archives/2015/01/15/january-2015-web-server-survey.html [6] see http://gs.statcounter.com/#all-browser-ww-monthly-201502-201502-bar [7] percentages are based on the 66 respondents who reported the oss projects they had adopted. [8] all percentages are based on the 56 respondents who have contributed to one or more oss project. all 56 of these respondents reported on the types of oss contributions they made. [9] percentages are based on the 50 respondents who indicated which projects they had contributed to. [10] askey defines oss lite as “tiny programs written in various scripting languages that drive all the doodads and widgets on our websites, or extend (or, in some cases, repair) the functionality of our commercial systems.” bibliography anderson p. richard stallman on the road less travelled [internet]. intelligent content. available from: http://oss-watch.ac.uk/resources/stallman askey d. 2008. we love open source software. no, you can’t have our code. code4lib journal(5). ayala cp, cruzes ds, hauge o, conradi r. 2011. five facts on the adoption of open source software. software, ieee 28(2):95-99. chudnov d. 2007. the future of floss in libraries. information tomorrow: reflections on technology and the future of public and academic libraries. medford, nj: information today, inc. p. 19-30. engard nc. 2010. practical open source software for libraries. elsevier. p. 29. eyler p. 2003. koha: a gift to libraries from new zealand. linux journal 2003(106):1. kitchenham ba, pfleeger sl. 2008. personal opinion surveys. guide to advanced empirical software engineering. springer. p. 63-92. laurent ams. 2004. understanding open source and free software licensing. ” o’reilly media, inc.”. raymond e. 1999. the cathedral and the bazaar. knowledge, technology & policy 12(3):23-49. thacker jc, knutson cd, dehmlow m. 2014. open source software. association of research libraries subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – apples to oranges: using python and the pymarc library to match bookstore isbns to locally held ebook isbns mission editorial committee process and structure code4lib issue 56, 2023-04-21 apples to oranges: using python and the pymarc library to match bookstore isbns to locally held ebook isbns to alleviate financial burdens faced by students and to provide additional avenues for the benefits shown to be present when no-cost materials are available to students (equity and access and an increase in student success metrics), more and more libraries are leveraging their collections and acquisition processes to provide no-cost ebook alternatives to students. it is common practice now for academic libraries to have a partnership with their campus bookstore and to receive a list of print and ebook materials required for an upcoming semester. libraries take these lists and use various processes and workflows, some extremely labor intensive and others semi-labor intensive, to identify which of these titles they already own as unlimited access ebooks, and which titles could be purchased as unlimited access ebooks. the most common way to match bookstore titles to already licensed ebooks is by searching the bookstore provided isbn or title in either the library management system (lms), the analytics and reporting layer of the lms, the library discovery layer, or via another homegrown process. while some searching could potentially be automated, depending on the available functionality of the lms or the analytics component of the lms, the difficulty lies in matching the bookstore isbn, often the print isbn, to the library ebook isbn. this article will discuss the use of python, the pymarc library in python, and library ebook marc records to create an automated identification process to accurately match bookstore lists to library ebook holdings. by mitchell scott introduction it is becoming increasingly common for academic libraries to seek out partnerships with campus bookstores [1]. academic libraries are interested in working with the bookstore to receive the title level data that the bookstore collects on what books are being required for an upcoming semester. libraries are seeking these partnerships to leverage the use of their collections, specifically their already licensed unlimited access ebooks, and their ability to purchase unlimited access ebooks, to offer as no-cost alternatives to students. offering library supplied no-cost ebook alternatives to students allows the library to play a role in affordability initiatives and potentially create the no-cost conditions that have been shown with oer (open educational resources) use to increase student success metrics [2]. while some libraries find receptive bookstore partners [3], others do not, and those develop alternate processes to get at the title level data on what books are being required for an upcoming semester [4, 5, 6]. no matter how libraries get their hands on this data, once acquired, they have developed various methods to pursue the daunting task of matching isbns or title level data on what is required to isbns or title level data on what is already licensed, owned, or can be purchased as unlimited access ebooks [7, 8, 9, 10]. ideally, this process could be automated and the hundreds, and potentially thousands, of bookstore isbns could be batch searched against the hundreds of thousands, potentially millions, of ebook isbns that make up the library’s ebook holdings. however, it is not that simple. for one, the isbns represented on the campus bookstore list vary with some book records having only a print isbn, some having only ebook isbns, and some having both a print and ebook isbn. figure 1. examples of a bookstore supplied spreadsheet with one isbn per book. figure 2. examples of a bookstore supplied spreadsheet with multiple isbns per book. another aspect that complicates this process is which isbn is represented in the bookstore list and whether the one selected by the bookstore will correspond to any isbns indexed in the library’s ebook collection, the library management system (lms), or library discovery. unique isbns are granted based on varying formats and editions. this means certain books can have unique isbns for paperback and hardback versions [11]. ebook isbns are also granted by format type and exist for standard ebook versions, epub versions, and pdf versions [12]. unique isbns could also be a part of a record for revised editions of the same book. with varying isbns represented in the bookstore data the library could be attempting to match a print isbn to an ebook isbn (not going to happen) or match a different ebook format to the library held ebook format (might happen). figure 3. example of ebook marc record and the multiple isbns contained within it. another complication to batch searching by isbn is that libraries are dependent on whether their lms has the functionality that allows for the batch searching of isbns and if they are one of the libraries that does have an lms that allows for isbn batch searching, how well it searches indexed isbns. from 2015-2017, i was a librarian at an institution that used ex libris alma, when alma was relatively new, and alma had an analytics component that allowed for the batch input of hundreds, to potentially thousands, of items and searched against alma data made available within its analytics component [13]. although possible, i never used alma analytics for isbn searching and wonder about the ability and accuracy of the system to do this level of searching and matching. it is my understanding that, in most cases, library management systems commonly index one isbn per book, which is usually the primary or preferred isbn for a particular title. this means that not every isbn granted to an ebook is searchable, and a bookstore ebook isbn might not be found in lms or discovery searches [14,15,16]. if print isbns are searched, then they commonly will not result in identifying its ebook alternative within a lms as these systems do not commonly cross search in that way. within the library literature, only one article references a process that potentially does batch searching with isbns and that is at the university of minnesota, where they have worked on similar processes since 2015. eighmy-brown notes that data management staff “match the libraries electronic resource titles and isbns with those in the bookstore“ [3]. i have been working with campus bookstores on this process since 2018 and unfortunately, have never been a user of a lms that allows for any level of batch searching. after having the experience of manually searching bookstore isbns and titles in a library discovery platform to identify assigned materials that have available library ebooks, i was interested in automating the process. having some python programming experience, i turned to python and discovered the pymarc library [17], which was built to work with marc records, and developed a script and process that takes the isbns present in a bookstore list, and matches them against existing library ebook holdings. as a result of this matching, i can then take the isbns that did not match to determine what could be purchased as an unlimited access ebook. this script and process works in accurately identifying matches in this content because the script iterates over all the ebook marc records that i input into it, indexes every isbn that is present in an ebook’s marc record (all print, ebook, and format or version isbns), and then matches the bookstore isbns, whether print or ebook, to the isbns from the ebook marc records. the process this process starts with identifying and harvesting the ebook marc records that are to be a part of the match process. since we only use unlimited access ebooks for this initiative, i only include unlimited access licensed ebook collections and individual purchases, when optional. the outlier here is ebsco ebooks, which does not allow for the separation of unlimited access ebooks from other more limiting user models at the marc level–i still include these but check these matches manually to determine the user model of the ebook. i use the oclc worldshare collection manager [18] to harvest the marc records for the ebook collections and titles that i want to include. figure 4. downloading files from the oclc collection manager. at st. norbert college, where i worked previously, the oclc worldshare management system (wms) was our lms and the collection manager was a required component for e-resource activation, linking, and discovery. despite being on a different lms at indiana university southeast, i still use the oclc collection manager for this process. this is due to complications for extracting institution specific ebook marc records from the shared indiana university lms (largely limited users with these admin privileges and the large amount of e-resource marc content loaded) and because of my familiarity with the worldshare collection manager. once i have all the marc records harvested that i want to include in this process, i run the first python script that kicks off the matching work [19]. figure 5. folder with marc files and python scripts that pull data from these marc files. the first script to run iterates over all the ebook marc records that i have compiled. there are several libraries that i import to use alongside python’s basic library for this process. from pymarc import marcreader import glob import re from collections import defaultdict import pickle this script uses the glob library [20] to iterate over one file type (.mrc) and progress over all marc files until all the marc files have been opened and processed. with the pymarc library and its marcreader function, the script iterates over every marc record within these files and extracts specific data to build a dictionary for later recall. this dictionary is built using the collections library [21] and is compiled with isbns, from the 020 marc field and 020 marc subfields a and z, as the dictionary key. def main(): ebook_dict = {} ###main e-book dictionary that maps vendor, title, and access url to e-book isbn def ebook_get_isbns(): for file in glob.glob('*.mrc'): # glob alows you to open all file types in a folder. will process all .mrc files try: print('getting isbn list...') with open(file, 'rb') as marc: reader = marcreader(marc) for r in reader: for field in r.get_fields('020'): for subfield in field.get_subfields('a', 'z'): re_sub_isbn = re.search(r'/((978[\--])?[0-9][0-9\--]{10}[\---][0-9xx])|((978)?[0-9]{9}[0-9xx])', str(subfield)) # regex to trim isbns this step extracts all isbns contained within the ebook marc record. i use regular expressions in python [22] to trim the isbns before adding them to the dictionary as the key. the dictionary values, also extracted from the marc, are then attached to the isbn keys. these values are the title of the book (245 and 245 marc subfield a) and the url of the ebook (856 marc field). for field in r.get_fields('245'): # gets ebook title from marc ebook_dict[final_sub_isbns].append(field.value()) if field is none: for subfield in field.get_subfields('a'): # gets title in 245 a if subfield is none: pass else: re_title = re.search(r'^(.+?)\/', subfield) # regex for title title = title.strip('/') title = title.strip() ebook_dict[final_sub_isbns].append(title) for field in r.get_fields('856'): # regex for ebook url url = field.value() re_url = re.search(r'http(.+?)$', url) if re_url is none: pass else: url = re_url.group(0) ebook_dict[final_sub_isbns].append(url) one outlier to this is the vendor’s name value that i create from the marc file name (i use a file naming convention that is the vendor_date of extract_number of file (if multiple marc files exist). figure 6. file naming convention for extracting vendor name for dictionary. for this value, i use regular expressions to match to the vendor’s name that prepends the file and use it as a dictionary value. else: final_sub_isbns = re_sub_isbn.group(0) vendor_name = re.match(r'^(.+?)_', file) # regex to get the vendor from the file name vendor_key = vendor_name.group(1) ebook_dict.setdefault(final_sub_isbns, []) #sets the isbn as the dictionary key ebook_dict[final_sub_isbns].append(vendor_key) # adds the vendor name as a value to the isbn key i also use regular expressions in combination with the pymarc library to match to the relevant data, clean the data, and pass the data, when necessary, to basic string functions to clean the data before adding it to the dictionary. a good example of this is the title data from the 245 marc field. in some marc records the title data appears in the 245 field, as seen with here: for field in r.get_fields('245'): # gets ebook title from marc ebook_dict[final_sub_isbns].append(field.value()) # gets title in 245 however, the title data could also be in the 245-subfield a. if no data is present in the 245 marc field, the script looks in subfield a and then uses regular expression to get the title data from the field. the script then passes the title through string functions to remove the ‘/’ from the title and any leftover white spaces. if field is none: for subfield in field.get_subfields('a'): # gets title in 245 a if subfield is none: pass else: re_title = re.search(r'^(.+?)\/', subfield) # regex for title title = title.strip('/') title = title.strip() ebook_dict[final_sub_isbns].append(title) each piece of marc data (vendor, title, url) is added as a value to the ebook dictionary. the isbns are set as the key: ebook_dict.setdefault(final_sub_isbns, []) #sets the isbn as the dictionary key ebook_dict[final_sub_isbns].append(vendor_key) # adds the vendor name as a value to the isbn key the other values are then matched and appended to their corresponding isbn: else: re_title = re.search(r'^(.+?)\/', subfield) # regex for title title = title.strip('/') title = title.strip() ebook_dict[final_sub_isbns].append(title) else: url = re_url.group(0) ebook_dict[final_sub_isbns].append(url) having compiled the ebook dictionary, the script then uses the pickle library [23] to save the dictionary for use in the next script. def pickle_dict(): pickle.dump(ebook_dict, open("ebook_dict.p", 'wb')) # pickle saves the ebooks dictionary so that it can be called in a seperate script. i run this to compile the dictionary and wait till i have bookstore isbns to run second part. pickle_dict() print('all done') i have broken this process into two scripts [26, 27]. the first script that builds the ebook dictionary can take ten to fifteen minutes to finish and in debugging the matching process, i found it easier to break these apart and deal with them separately. pickling or saving the dictionary also allows me to extract marc records and build my ebook dictionary in advance of receiving the isbns from the bookstore. once i have the bookstore isbns, i can input that data into the second script and simply look for matches. the second python script does the matching and outputting of matched titles. the first step is to load the ebook dictionary that was built in the previous script, this is also done with the pickle library. import pickle # imports pickle to load the dictionary def main(): ebook_dict = pickle.load(open('ebook_dict.p', 'rb')) # loads isbn dictionary that was built and saved with 1st script. once i receive the required material list from the campus bookstore, i extract all the isbns and use a text editor, such as notepad ++, to remove duplicates, sort, and delete any isbns that do not begin with the 978 prefix and save as a text file. that file is then opened within the script, and a list is created with its isbns. bookstore_list = [] def get_bookstore(): bookstore_file = 'spring_2023_nov.txt' lines = [line.strip() for line in open(bookstore_file, encoding = 'utf-8')] for line in lines: bookstore_list.append(line) # add all bookstore isbns to list in the find_match function, the script uses two for loops to iterate over the key and values of the ebook dictionary and the isbns within the bookstore list. def find_match(): print('getting ius ebook matches...') for key, value in ebook_dict.items(): for i in bookstore_list: the script then matches any isbn in the bookstore list to the dictionary key (ebook marc record isbns) and if a match is found, it outputs the key (isbn) and associated dictionary values (separated by commas) in the python console window. if i in key: print(key + ',' + str(value)) # prints the matches in the viewing window in python. i copy and paste this into excel. the data is comma delimited. getting ius ebook matches... 9780787980672,['ebraryp', 'because writing matters : improving student writing in our schools / national writing project and carl nagin.', 'http://proxyse.uits.iu.edu/login?url=https://ebookcentral.proquest.com/lib/[pqebk.library.id]]]/detail.action?docid=707687'] 9781474225199,['ebraryp', 'plays one / translated and introduced by don taylor ; with introductions by don taylor and j. michael walton.', 'http://proxyse.uits.iu.edu/login?url=https://ebookcentral.proquest.com/lib/[pqebk.library.id]]]/detail.action?docid=6159981'] 9781474225199,['ebraryp', 'plays one / translated and introduced by don taylor ; with introductions by don taylor and j. michael walton.', 'http://proxyse.uits.iu.edu/login?url=https://ebookcentral.proquest.com/lib/[pqebk.library.id]]]/detail.action?docid=6159981'] 9781474225199,['ebraryp', 'plays one / translated and introduced by don taylor ; with introductions by don taylor and j. michael walton.', 'http://proxyse.uits.iu.edu/login?url=https://ebookcentral.proquest.com/lib/[pqebk.library.id]]]/detail.action?docid=6159981'] 9781462531592,['ebraryp', 'building literacy with english language learners : insights from linguistics / kristin lems, leah d. miller, tenena m. soro.', 'http://proxyse.uits.iu.edu/login?url=https://ebookcentral.proquest.com/lib/[pqebk.library.id]]]/detail.action?docid=4844819'] 9781462531592,['ebraryp', 'building literacy with english language learners : insights from linguistics / kristin lems, leah d. miller, tenena m. soro.', 'http://proxyse.uits.iu.edu/login?url=https://ebookcentral.proquest.com/lib/[pqebk.library.id]]]/detail.action?docid=4844819'] 9781433829789,['ebraryp', 'how to write a lot : a practical guide to productive academic writing / paul j. sivia, phd.', 'http://proxyse.uits.iu.edu/login?url=https://ebookcentral.proquest.com/lib/[pqebk.library.id]]]/detail.action?docid=5525817'] 9780231191630,['ebraryp', 'what is japanese cinema? : a history / yomota inuhiko ; translated by philip kaffen.', 'http://proxyse.uits.iu.edu/login?url=https://ebookcentral.proquest.com/lib/[pqebk.library.id]]]/detail.action?docid=5613954'] 9780674018242,['ebraryp', 'the poems of emily dickinson : reading edition.', 'http://proxyse.uits.iu.edu/login?url=https://ebookcentral.proquest.com/lib/[pqebk.library.id]]]/detail.action?docid=5906829'] 9781788291552,['ebraryp', 'machine learning with r : expert techniques for predictive modeling / brett lantz.', 'http://proxyse.uits.iu.edu/login?url=https://ebookcentral.proquest.com/lib/[pqebk.library.id]]]/detail.action?docid=5752784'] 9781788295864,['ebraryp', 'machine learning with r : expert techniques for predictive modeling / brett lantz.', 'http://proxyse.uits.iu.edu/login?url=https://ebookcentral.proquest.com/lib/[pqebk.library.id]]]/detail.action?docid=5752784', 'ebraryp', i then take the python console output and copy and paste the matches into an excel file. duplicate matches do occur for bookstore list books that have multiple isbns listed for a single title (print and ebook) and these can be easily removed once the data is in excel. i have tried multiple times to create an output of the isbns that did not match as a text file during this process but have failed to write code that works. i use excel and its conditional formatting/highlight cell rules/duplicate values for this process of removing isbns that matched from those that did not match. once i have the isbns that do not match to an existing library ebook, i take those, and batch search them in the gobi platform [24] to identify which required materials could be purchased as unlimited access ebooks. figure 7. gobi platform’s alternate formats search of unmatched isbns. figure 8. gobi platform’s alternate formats search results. conclusions having this python script to automate the identification of already licensed and owned library ebooks has helped immeasurably with this process. as reported in previous studies, existing licensed or subscribed content could make up as much as 27% of the ebooks that a library can provide for programs like this, and up to 17% can come from ebooks that were previously purchased (and reused semester to semester) for these programs [25]. being able to accurately and quickly identify these ebooks increases the turnaround time between receiving the required materials list from the bookstore (which can vary in the timeliness of its delivery) and the notification of faculty and students about these no-cost alternatives prior to the start of the semester. any next steps would be to make this process and these scripts more user friendly. i’d like to output the isbns that do not match to existing library ebooks to a text file so that i could batch search those in gobi without any additional work. i’d also like to create a basic gui interface that could be used by colleagues and/or libraries without any python experience and would only require the marc records to be used in the match process. about the author mitchell scott is the coordinator of collections and online resources at indiana university southeast. bibliography [1] bell, s. (2017). what about the bookstore?: textbook affordability programs and the academic library-bookstore relationship. college & research libraries news, 78(7), 375.https://www.doi.org/10.34944/dspace/98. [2] colvard, nb, watson, ce, park, h. (2018). the impact of open educational resources on various student success metrics. international journal of teaching and learning in higher education, 30(2), 262-276. [3] eighmy-brown, m, mccready, k, riha, e. (2017). textbook access and affordability through academic library services: a department develops strategies to meet the needs of students. journal of access services, 14(3), 93-113. [4] rokusek, s, cooke, r. (2019). will library e-books help solve the textbook affordability issue? using textbook adoption lists to target collection development. the reference librarian, 60(3), 169-181. [5] wimberly, l, cheney, e, ding, y. (2020). equitable student success via library support for textbooks.” reference services review. https://www.doi.org/10.1108/rsr-03-2020-0024. [6] scott, r, jallas, m, murphy ja, park, r, shelly, a. (2022). assessing the value of course-assigned e-books. collection management, 47(4), 253-271. [7] carr, pl, cardin, jd, shouse, dl. (2016). aligning collections with student needs: east carolina university’s project to acquire and promote online access to course-adopted texts.” serials review 42, no. 1: 1-9. https://www.doi.org/10.1080/00987913.2015.1128381. [8] wallace, n, filion, s. (2018). textbook or not: how library ebook purchasing power aligns with curricular content trends.” the evolution of affordable content efforts in the higher education environment: programs, case studies, and examples. [9] hoover, j, shirkey, c, barricella. (2020). exploring sustainability of affordability initiatives: a library case study.” reference services review. https://www.doi.org/10.1108/rsr-03-2020-0016. [10] sotak, d, scott, jg. (2020). affordable education with a little help from the library.” reference services review. https://www.doi.org/10.1108/rsr-03-2020-0012. [11] library of congress isbn faq [12] e-books and isbns: a position paper and action points from the international isbn agency [13] alma analytics [14] ex libris alma [15] folio [16] oclc wms [17] pymarc python library documentation [18] oclc worldshare collection manager [19] my experience with python is limited and i am a self-taught coder. as a result, there are many redundancies and complexities that exist within the code that more experienced and knowledgeable coders will uncover but it works for me. [20] glob python library documentation [21] collections python library documentation [22] regular expression python library documentation [23] pickle python library documentation [24] ebsco gobi [25] scott, m. (2021). shifting priorities: a look at a course adopted text (cats) e-book program and how its success realigned one library’s e-book collection priorities. collection management, 47(4), 238-252. [26] script 1: building the ebook dictionary with marc [27] script 2: matching and outputting matches subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – taming the generative ai wild west: integrating knowledge graphs in digital library systems mission editorial committee process and structure code4lib issue 60, 2025-04-14 taming the generative ai wild west: integrating knowledge graphs in digital library systems since the 17th century, scientific publishing has been document-centric, leaving knowledge—such as methods and best practices—largely unstructured and not easily machine-interpretable, despite digital availability. traditional practices reduce content to keyword indexes, masking richer insights. advances in semantic technologies, like knowledge graphs, can enhance the structure of scientific records, addressing challenges in a research landscape where millions of contributions are published annually, often as pseudo-digitized pdfs. as a case in point, generative ai large language models (llms) like openai’s gpt and meta ai’s llama exemplify rapid innovation, yet critical information about llms remains scattered across articles, blogs, and code repositories. this highlights the need for knowledge-graph-based publishing to make scientific knowledge truly fair (findable, accessible, interoperable, reusable). this article explores semantic publishing workflows, enabling structured descriptions and comparisons of llms that support automated research insights—similar to product descriptions on e-commerce platforms. demonstrated via the open research knowledge graph (orkg) platform, a flagship project of the tib leibniz information centre for science & technology and university library, this approach transforms scientific documentation into machine-actionable knowledge, streamlining research access, update, search, and comparison. by jennifer d’souza introduction there is a growing mountain of research. but there is increased evidence that we are being bogged down today as specialization extends. the investigator is staggered by the findings and conclusions of thousands of other workers—conclusions which he cannot find time to grasp, much less to remember, as they appear. … a record if it is to be useful to science, must be continuously extended, it must be stored, and above all it must be consulted. [1]  — vannevar bush, as we may think (1945) the sheer volume of scientific advancements poses significant challenges for researchers, who must sift through vast amounts of information to derive meaningful insights. this challenge is further compounded by the largely unstructured nature of current scientific communication, which hampers transparency, integration, peer review efficiency, and machine assistance. to overcome these limitations, research records must transition from traditional unstructured formats to machine-actionable, semantically structured representations, such as knowledge graphs (kgs), enabling smarter and more efficient research tools. this shift also aligns with the fair principles—findable, accessible, interoperable, and reusable [2] —ensuring usability for both humans and machines. the open research knowledge graph (orkg), a flagship project of the tib leibniz information centre for science & technology and university library, seeks to address these challenges by enabling the structured representation and comparison of scientific contributions based on their key properties. much like e-commerce platforms that allow side-by-side product comparisons, the orkg facilitates efficient exploration of scientific knowledge, opening new opportunities for researchers and digital libraries[3][4][5]. the orkg is a web-based platform and infrastructure designed to structure and interlink scientific knowledge using semantic technologies. unlike traditional text-based research publishing, the orkg, in close adherence to the fair principles, represents research contributions as interconnected kgs, enabling more efficient comparison, discovery, review, and synthesis of scientific findings. figure 1 depicts the structured machine-actionable scientific information published on the orkg. in the orkg, researchers and librarians can submit structured descriptions of research papers, making scientific knowledge more accessible for both humans and machines. the platform is available as a service with public community access (https://orkg.org/) and is implemented via open-source software (https://gitlab.com/tibhannover/orkg/). it integrates with persistent identifier services such as dois and orcids for improved scientific metadata management and fosters collaboration and open science by enabling interdisciplinary research contributions. additionally, orkg provides an api as a python package (https://orkg.readthedocs.io/) that allows developers to integrate its capabilities and data into external applications, enhancing knowledge discovery and research automation. figure 1. machine-actionable scientific knowledge capture via semantic publishing in the orkg (red) versus non-machine-actionable, article-based publishing (gray). this figure depicts the process by which a user fills out a template of key properties of research that generates structured paper descriptions, used by the orkg to generate comparisons of multiple studies with similar properties. tracking the rapid advancements in generative artificial intelligence (ai) the development of large language models (llms) began in 2018 with openai’s gpt-1 and google’s bert, which pioneered transformer-based architectures for language understanding. since then, the rapid evolution of llms has been characterized by a wave of innovative releases, each pushing the boundaries of performance, efficiency, and accessibility[6]. the year 2023 saw the release of models like mistral 7b which introduced grouped-query attention for enhanced computational efficiency, while mixtral 8x7b leveraged sparse mixture-of-experts architectures to optimize scalability without sacrificing performance. other notable advancements include meta ai’s llama 2, which improved fine-tuning capabilities and context understanding, and openai’s gpt-4, which significantly expanded multimodal capabilities, including image and text input. additionally, anthropic’s claude series focused on safety and interpretability in alignment techniques, and google’s gemini models demonstrated cutting-edge integration of external tools, setting a new benchmark for task-oriented ai. there is a growing focus on open-source development, computational efficiency, extended context handling, and multimodal capabilities. these breakthroughs illustrate the relentless pace of innovation, driven by a combination of architectural refinements, expanded datasets, and collaborative efforts across the ai community. this continuous stream of innovations highlights the transformative potential of llms, as well as the increasing challenge of staying informed in such a rapidly evolving landscape. the continuous influx of new models, techniques, and datasets demands constant vigilance and adaptability. key information about llms is often scattered across scientific articles, organizational blog posts, and source code repositories, making it difficult to track developments efficiently. llms can be compared using key details such as model name, organization, release date, pretraining data, fine-tuning strategies, and carbon footprint. modeling these consistent properties supports better comparison and searchability and is an excellent use case for the open research knowledge graph. background: the open research knowledge graph essential technical considerations when publishing kgs include defining salient entities, attributes, and relationships for generative ai models as standardized resources accessible on the web via uniform resource identifiers (uris). informational statements about llms should also be serialized as structured triples conforming to the resource description framework (rdf) syntax recommended by the world wide web consortium (w3c) [7]. standard vocabularies, metadata, and publishing data on the web using dereferenceable uris are crucial for semantic interoperability and linking to external datasets, enabling a true linked data ecosystem. these functionalities are fully supported by the orkg, a next-generation scholarly publishing platform. in this vein, the orkg is not just software but also a namespace for specifying scholarly vocabularies via uris, intuitive interfaces for defining resources  properties, classes, and templates, and backend technologies for representing the kg in rdf syntax. additionally, its machine-actionable semantic structure supports advanced comparison views, akin to e-commerce product comparisons, and enables sparql queries for customized or aggregated views of structured scholarly knowledge. the orkg adopts a collaborative, dynamic knowledge creation model inspired by wikipedia, where anyone can contribute data using vocabularies of their choice to represent entities interconnected via rdf links, forming a global, discoverable data graph. its open-ended nature allows the content of research artifacts, such as papers and comparisons, to be edited, updated, and republished, with all versions stored for future reference. persistent identifiers, such as dois, enable precise referencing of specific versions, while provenance metadata—capturing details like creator, creation date, and methods—ensures traceability, trustworthiness, and quality assessment. dois assigned to graph components enhance discoverability through global scientific bibliometric infrastructures like datacite and crossref, while resources are also findable via search engines. accessibility is ensured through http protocols, rest apis, and a user interface, with metadata accessible independently of graph data. the orkg achieves interoperability using rdf, the w3c-recommended standard for machine-readable knowledge representation, and fosters reusability by automatically generating provenance metadata and publishing graph data under a cc by-sa license. these features collectively support efficient, trustworthy, and fair-compliant scientific knowledge dissemination. taming the generative ai wild west: a catalog of llms on the orkg publishing information about llms on the orkg begins with defining standard properties for their structured and comparable description. drawing inspiration from hugging face, a central hub for open-source llms and their model cards, the author developed a standardized vocabulary and schema to support knowledge capture and semantic representation. 4.1. the llm specification orkg template standardized nomenclature. to ensure consistency, we established a controlled vocabulary that forms the foundation of the llm specification schema. properties were derived from hugging face model cards, with additional llm-specific predicates added to the orkg web namespace to expand the vocabulary. to align these properties with external ontologies, the rdf same-as relation was employed to establish equivalences across predicates. use of orkg templates. for uniform recording of llm information, we created a form-based template comprising predetermined properties. this template system leverages recurring subgraph property patterns in the orkg, enabling the consistent specification of relevant attributes across multiple contributions. the template includes 27 structured properties that comprehensively describe llms, including: identification: model name, model family, organization, and date created. development details: pretraining architecture, pretraining task, pretraining corpus, size of training corpus (in tokens in billions), knowledge cutoff date and innovation. fine-tuning: fine-tuning task and fine-tuning data for refinement. technical specifications: optimizer, tokenizer, number of parameters, maximum number of parameters (in million), and context length (in tokens). language and usage: supported language, extension, and application. infrastructure and impact: hardware used, hardware description, and carbon emitted (tco2eq). transparency and licensing: availability of the source code, blog post, license, and the research problem addressed. these properties ensure a holistic representation of llms, enabling structured comparisons and detailed analysis. for example, transparency is supported by attributes such as source code availability, while environmental impact is captured through carbon emissions data. the orkg llm specification schema, which encapsulates these 27 properties, is accessible on the orkg platform. this schema empowers researchers to record, compare, and analyze llms with precision and consistency, fostering greater transparency and accessibility in the study of generative ai. 4.2. the orkg catalog of llm-centric papers, comparisons, visualizations, and review different papers on llms are published as structured semantic descriptions in the orkg using the orkg papers module and then aggregated as orkg comparisons of research. comparisons are the core type of orkg content and give a condensed overview on the state-of-the-art for a particular research question. contributions towards the problem are organized in a tabular view and can be compared and filtered along different properties. here is a published comparison of 92 llms with respect to their structured, machine-actionable descriptions published as one research comparison on the orkg. the comparison includes metadata such as a title, creator, and a short description paragraph. the unique aspect of orkg comparisons is that they can be published at any granularity. for instance, if there are structured descriptions of the early openai llms, viz. gpt-1, 2, and 3, i can aggregate them as an orkg comparison. this is depicted in figure 2 below. figure 2. a comparison of early openai large language models (llms), accessible on the orkg platform. additionally, the machine-actionability afforded by this fine-grained semantic information in the orkg supports dynamic interactions, such as computing visualizations from the data of any comparison. for instance, the author created another comparison titled “a catalog of deepmind’s llms including their seminal chinchilla model” ( based on their early models—alphafold, gopher, chinchilla, gophercite, flamingo, gato, and sparrow. figure 3 below depicts a column chart, comparing, for each model, the parameter size of the largest variant (measured in millions). this showcases how machine-actionable scientific knowledge in a structured digital library can provide quick insights into specific research questions. for instance, one could explore questions like “what was the largest model size released by deepmind?” or “what was the maximum variation in llm sizes across deepmind’s models?” figure 3. a column chart comparing the parameter sizes (in millions) of seven early deepmind models—alphafold, gopher, chinchilla, gophercite, flamingo, gato, and sparrow—based on the orkg comparison “a catalog of deepmind’s llms including their seminal chinchilla model” (https://orkg.org/comparison/r1355351). the chart dynamically visualizes machine-actionable data from the orkg platform and hovering over any bar in the chart shows the underlying data stored in the orkg for that specific plot. the resource is published at https://orkg.org/resource/r1355283. finally, the orkg offers reviews, a tool designed for authoring and publishing review articles that facilitates community-based creation of dynamic, living articles by utilizing comparisons from the open research knowledge graph to deliver machine-actionable knowledge. for example, the review available at https://orkg.org/review/r640001 aggregates multiple llm comparisons. it provides granular insights into the features of llms developed by various organizations while also integrating these into a broader comparison of all models. this approach is ideal for review articles, enabling detailed analyses of studies or, in this case, comparisons of the llms reviewed. 4.3. queryability sparql queries can be used to generate smaller, specific views of the data. for example, the query below might be translated into a natural language question as follows: which llm models were created in 2023? this query returns a table (limited to the first 100 results) of paper titles, their accompanying llm, and the llm creation date. natural extensions of this query would be to change the year, and even to be able to specify a range of dates in the form of yyyy-mm-dd within which to search. this is an open query endpoint available to any user of the web. prefix orkgp: <http://orkg.org/orkg/predicate/> prefix orkgc: <http://orkg.org/orkg/class/> prefix orkgr: <http://orkg.org/orkg/resource/> select ?paper_label ?model_label ?date_created where { orkgr:r609337 orkgp:comparecontribution ?contribution . ?contribution orkgp:has_model ?model . ?model rdfs:label ?model_label . ?paper orkgp:p31 ?contribution . ?paper rdfs:label ?paper_label . ?contribution orkgp:p49020 ?date_created . filter(strstarts(str(?date_created), "2023")) } limit 100 while this paper outlines the use case for llm, the possibilities and opportunities for interlinking information on the web are substantial. related work the closest representation of the information captured via the transformer model or llm orkg template are model cards for these models released on huggingface hub [8],. however, the model cards and fine-grained knowledge representation in kgs like orkg differ significantly in terms of discoverability, reproducibility, and sharing. model cards are again human-readable documents offering project explanations, but their discoverability is limited to keyword-based searches in repositories, and they often lack the semantic structure needed for precise, automated discovery. while recommendations for model cards involve filling out relevant values for model properties, this structured representation, while it might improve human readability to quickly gloss over the features of a model, lacks machine-actionability since they remain embedded within a document. in contrast, kgs provide machine-readable, structured, and semantically rich representations that enhance discoverability through advanced querying and metadata tagging. they excel in reproducibility by capturing detailed, standardized metadata, tracking updates, and linking methods and datasets transparently. furthermore, kgs enable seamless sharing and integration within interconnected systems, promoting collaboration and interoperability, unlike the static and standalone nature of any text document regardless of which format it is prescribed to be written in for better human readability. despite its advantages, implementing a structured, kg-based approach for model representation presents challenges, particularly in ensuring the quality of articles authored from structured reviews. while orkg enhances discoverability and reproducibility through machine-readable key aspects of the scientific content, expert oversight remains essential for translating structured data into high-quality narratives. sustainability is another key factor—unlike volunteer-driven efforts, orkg benefits from state funding as the flagship project of the tib, ensuring long-term development. this support enables continuous improvements in automation and usability, reducing the effort required for researchers to contribute structured knowledge. by integrating machine-assisted extraction, incentivizing contributions, and aligning with scholarly publishing workflows, orkg fosters broader adoption. thus, it provides a sustainable and scalable framework for improving model discoverability, reproducibility, and knowledge sharing. conclusion this article highlights the transformative potential of using knowledge graphs, specifically the open research knowledge graph (orkg), to represent and manage information about generative ai technologies such as large language models (llms). by adopting a structured, semantic, and machine-actionable approach, the orkg ensures that the resulting knowledge adheres to fair principles—findable, accessible, interoperable, and reusable. the kg-based representation demonstrated in this article is not only applicable to llms but is easily transferable to other scientific domains. the orkg platform exemplifies the versatility of a modern digital library, offering generic infrastructure that enables users from any field to model their research as structured knowledge using templates, papers, comparisons, visualizations, and reviews. these living records can be edited and curated collaboratively, ensuring that the knowledge remains dynamic, up-to-date, and accessible to both humans and machines. by addressing the challenges of fragmented, unstructured knowledge, this approach mitigates the creation of information silos and fosters greater transparency, reproducibility, and usability. ultimately, leveraging the orkg and similar platforms paves the way for a more interconnected and machine-actionable future in scientific knowledge dissemination and discovery. references [1] bush, v. (1945). as we may think. the atlantic. retrieved from https://www.theatlantic.com/magazine/archive/1945/07/as-we-may-think/303881/ [2] wilkinson md, dumontier m, aalbersberg ij, appleton g, axton m, baak a, blomberg n, boiten jw, da silva santos lb, bourne pe, et al. 2016. the fair guiding principles for scientific data management and stewardship. scientific data. 3(1):1–9. [3] auer s, oelen a, haris m, stocker m, d’souza j, farfar ke, vogt l, prinz m, wiens v, jaradeh my. 2020. improving access to scientific literature with knowledge graphs. bibliothek forschung und praxis. 44(3):516–529. [4] stocker m, oelen a, jaradeh my, haris m, arab oghli o, heidari g, hussein h, lorenz al, kabenamualu s, farfar ke, prinz m, karras o, d’souza j, vogt l, auer s. 2023. fair scientific information with the open research knowledge graph. fair connect. 1(1):19–21. https://doi.org/10.3233/fc-221513. [5] d’souza j, hussein h, evans j, vogt l, karras o, ilangovan v, lorenz al, auer s. 2024. quality assessment of research comparisons in the open research knowledge graph: a case study. jlis.it. 15(1):126–143. [6] fourrier, c. (2023). 2023, year of open llms. hugging face. retrieved from https://huggingface.co/blog/2023-in-llms [7] cyganiak, r., wood, d., & lanthaler, m. (2014). rdf 1.1 concepts and abstract syntax. world wide web consortium (w3c). retrieved from https://www.w3.org/tr/rdf11-concepts/ [8] hugging face. (n.d.). model cards. retrieved from https://huggingface.co/docs/hub/en/model-cards about the author dr. jennifer d`souza, tib leibniz information centre for science and technology, (m.sc. in computer science, university of texas at dallas, 2010 and ph.d. in computer science, university of texas at dallas, 2015). dr. d’souza is a senior postdoctoral researcher at tib leibniz information centre for science and technology, specializing in ai, nlp, and scientific knowledge extraction and organization. she leads the nlp-ai aspect of the open research knowledge graph (orkg) and heads the scinext project, advancing neuro-symbolic ai and nlp for scientific innovation extraction, funded by the german ministry of education and research (bmbf). her projects develop knowledge extraction and organization services, recently using generative ai, aimed at enhancing the strategic use of science and innovation to bolster societal r&d cycles. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – managing an institutional repository workflow with gitlab and a folder-based deposit system mission editorial committee process and structure code4lib issue 50, 2021-02-10 managing an institutional repository workflow with gitlab and a folder-based deposit system institutional repositories (ir) exist in a variety of configurations and in various states of development across the country. each organization with an ir has a workflow that can range from explicitly documented and codified sets of software and human workflows, to ad hoc assortments of methods for working with faculty to acquire, process and load items into a repository. the university of north texas (unt) libraries has managed an ir called unt scholarly works for the past decade but has until recently relied on ad hoc workflows. over the past six months, we have worked to improve our processes in a way that is extensible and flexible while also providing a clear workflow for our staff to process submitted and harvested content. our approach makes use of gitlab and its associated tools to track and communicate priorities for a multi-user team processing resources. we paired this web-based management with a folder-based system for moving the deposited resources through a sequential set of processes that are necessary to describe, upload, and preserve the resource. this strategy can be used in a number of different applications and can serve as a set of building blocks that can be configured in different ways. this article will discuss which components of gitlab are used together as tools for tracking deposits from faculty as they move through different steps in the workflow. likewise, the folder-based workflow queue will be presented and described as implemented at unt, and examples for how we have used it in different situations will be presented. by whitney r. johnson-freeman, mark e. phillips, and kristy k. phillips introduction like many other institutions, the university of north texas (unt) libraries operates an institutional repository to collect, preserve, and make available the scholarly and creative output of its faculty, staff, and students. the institutional repository, unt scholarly works (https://digital.library.unt.edu/scholarlyworks/), has been implemented as a collection in the unt digital library (https://digital.library.unt.edu/) alongside other collections such as the unt electronic theses and dissertations, unt data repository, unt graduate works, and unt undergraduate works. unt scholarly works has been in operation for nearly a decade and has grown to include over 6,100 items. the collection has been managed through a combination of email-based communications and workflows for receiving new materials, and an existing folder-based workflow that is used for digital projects throughout the unt libraries’ digital libraries division. in early 2020, there were two events that caused the team to review our workflows. first, a new repository manager took over the role and needed to learn the workflow. this necessitated the review of existing documentation, recognizing that it was not as thorough as needed, and the consideration of new tools and workflows for managing the repository. the second event that caused a shift in thinking was the covid-19 pandemic, which moved all work with the repository to remote work. this change further demonstrated the need to have documented, repeatable workflows so that multiple people could work on adding new content without getting in each other’s way. as a way of fixing these issues, we turned to a project management tool that was being used successfully by our software development team to manage their projects. this common software suite called gitlab was used to create a method for tracking items and tying our processes together. in addition to gitlab, we further codified our workflows on a departmental wiki that empowers us to keep our documentation more up-to-date as workflows and processes change. finally, we took advantage of the existing folder-based workflows used in other areas of the division and further refined our workflows to take advantage of this simple system for keeping track of files and digital objects on our shared network space. so far, it’s been a successful, simple solution that has been able to evolve as our needs and working arrangements change. workflows: moving from old to new as in most institutions with long-running projects like our unt scholarly works repository collection, the process for collecting, processing, and ingesting content into our digital library system has evolved over the years. since the beginning of the collection, the preferred mechanism for faculty to submit content has been emailing the repository manager at a shared repository email address. the shared nature of the email address allows for multiple members of the team to have access to new submissions but also causes challenges in trying to keep track of these same submissions as they are moved through the workflow. while many messages to the repository email address are content submissions in digital format, others are requests for help in digitizing legacy resources such as publications, datasets, and grey literature. this content is also handled by the repository manager and involves several additional steps that are not present in born-digital workflows. examples of these steps include receipt of the physical material, digitization of the material including quality control, and then finally loading the material into the repository. we’ve tried to keep our workflows simple, so they can apply to most types of submissions. the majority of our submissions are currently received in digital format. the workflow for these items revolves around confirming rights for hosting, preparing the files to be ingested, and creating the metadata record for the item. figure 1 provides an overview of the workflow in place for processing new items into unt scholarly works. figure 1. workflow for processing digital items into unt scholarly works. gitlab overview and our workflow  gitlab is a web-based project planning and source code management application that is built on top of the git distributed version control system [1]. gitlab can be used as a hosted solution or its community edition can be installed on local servers. the unt libraries has been using a locally hosted version of the community edition of gitlab to successfully manage many of its software development projects. additionally, in 2018, the unt libraries’ website was migrated from the drupal content management system to a static site generator using jekyll. to manage the content creation for the website, we used another installation of gitlab and configured it for that type of content. with so much existing expertise with this software platform, it made sense to try using it to manage the content workflows for unt scholarly works. to get started with gitlab, we established a new project called unt-scholarly-works. in gitlab, a project is the primary unit of organization and includes a number of components that are useful for managing a software project. the new unt-scholarly-works project includes features such as a git enabled repository for managing code, an issue tracker for creating issues, discussing specifics of an issue, assigning responsibility, and managing the issues. a project in gitlab also includes a wiki that can be used to develop documentation about the project. finally, a suite of access controls can be used for a project to designate who can perform different tasks such as viewing, changing, and deleting content. once the unt-scholarly-works project was created, the primary component we were interested in using was the issue tracking system. we had seen it used in a number of different configurations in software development projects and had ideas of how it might be useful for managing the unt scholarly works workflow. gitlab issues the component of gitlab that we use for creating tickets is called issues. in a software development project, this is the method for submitting new features, changes, or updates to a project, but we use it as a tracking tool within our workflow. when we receive a content submission at the repository email address, a new issue is opened with details from the submission email. the required parts for an issue are: unique title processing checklist submitter’s email notes labels   a unique title is necessary for tracking items between gitlab and our folder system, especially when multiple items are submitted at once. our titles are structured with underscores separating these parts: submission date, in year-month-day format_ submitter_name, last name followed by first name   for instance: “2020-11-25_last_first”. when multiple items are submitted at once by the same person, identifying details are added to the end of the title, like keywords from an item’s title or the item description: 2020-11-25_last_first_repository_workflow or 2020-11-25_last_first_handout. this unique title is used for the corresponding folder name on our network drive to encapsulate the files for this item and move it through its folder-based workflow. a full description of that process is included later in this article.   a checklist is included in the issue to ensure that all steps have been completed. these checklists are added based on templates we’ve saved in gitlab. the gitlab issues system has a method for creating different templates for issues that get created within a project. these templates can be customized to the workflow and allow for adding placeholders to each issue so that important information for an issue is not missed. we have developed templates for two types of submissions: new item submissions or item replacements. examples of these two templates are provided in figure 2 and figure 3. the templates include the relevant steps for the different processes required for the different submission types. more information on creating issue templates is available from gitlab [2]. figure 2. issue template for new submissions. figure 3. issue template for item replacements. because most of our submissions arrive via email, the submitter’s email address is also included to help connect the issue to the repository account’s inbox. the notes section is used to document any relevant information provided from the submitter, like conference information, and any other information that we might need during processing, like licensing guidelines from a publisher. labels we make extensive use of labels that can be applied to an issue as part of the gitlab issues system. it is possible to add one or more labels to an issue and later these labels can be used to sort and filter issues. for each new issue that is added to the system, we use at least two labels, one based on the type of content of the submission and the other a high-level label designating the type of submission. these labels are helpful for tracking where an item is in the unt scholarly works workflow. we’ve opted for broad labels that can help identify a submission at a glance. two high-level labels are used to label what kind of submission was deposited: submission correction another set of labels reflect the major content types that we receive as submissions. each of these content types have similar workflows but it is helpful to differentiate between them. here is a listing of the major content types we have designated as labels: audio correction data poster presentation text video there are two additional labels we use that aren’t related to item type but that relate to the state of the item in the workflow: checking rights embargo these are steps that tend to hold up processing, so these labels let us see why an issue hasn’t been closed. for embargoed items, we can also add a due date to the issue which will remind us to confirm that the item’s access rights are updated in the repository and notify the submitter that their item is now publicly available. figure 4. example issue for a newly submitted item. in addition to improving issue tracking, another benefit of this label system is that we can go back to see what types of submissions we’ve received. this additional use of the label system provides a data layer that we can reference in the future to see any trends with our submission process. for example, we can select the poster label and see quickly how many poster issues have been closed or that are still open. an example of an issue is provided in figure 4. other benefits of gitlab once an issue has been opened, there are a few different features we use to create an additional layer of documentation and support for our team. the first is that gitlab tracks all changes or updates to an issue with the person, date, and time. the next is the comment feature. comments can be added to issues, and others can respond to that comment or create their own. we can also direct a comment to an individual who is also a member of the unt-scholarly-works project, and they will be notified that there is a comment waiting for them in gitlab. the final feature we’ll mention is the ability to assign individuals to an issue. this adds the issue to the assignee’s list of issues and presents the user with an updated list of activities that need to be completed for items being added to unt scholarly works. in addition to changes being tracked for issues in the system, almost everything is reversible. if an issue is closed but then something comes up where it needs to be reopened, that is a simple task in gitlab and doesn’t require much effort at all. as a team of gitlab users that are also working remotely, the ability to undo changes allowed us to experiment with how to use different features correctly and figure out how they might fit in our new workflows. gitlab has allowed the repository team to monitor the submissions to unt scholarly works through a system that is detached from the primary repository email account, but did not require custom programming or development. we made use of the existing features of gitlab to customize our unt-scholarly-works project to our local needs. finally, it has allowed us to experiment with modifications to our workflows, and the overhead for each modification is very low as we only have to change labels on issues in order to find out if an idea will or won’t work. for example, as we explained earlier, when we initially started with gitlab we modeled our label system too closely to the folder-based workflows we use on the network drives. we were able to experiment with different approaches to labeling issues without affecting other parts of the workflow for unt scholarly works. there are a number of other features of gitlab that we haven’t fully explored, including some related to issues and labels. these include the board and milestones features. milestones, which are a specific tool used to manage timetables, would allow us to indicate the time-sensitivity of a group of issues within a project, such as a deadline for uploading a number of items to the repository. issue boards, which are a tool used for project management, would allow for a more visual way of organizing and managing our submission issues in the unt scholarly works workflow. both of these features are something we plan to explore as we continue with gitlab. gitlab is how we track our ingesting process, but the folder-based workflow is where the processing of deposited files actually happens. as mentioned previously, we’ve created our gitlab checklists and labels to correspond to our folder-based system to help blend the workflows together. folder-based workflow the concept of folder-based workflows has been present in the unt libraries’ digital libraries division since it was formed. while it is a simple concept, it provides a fairly robust process for managing projects on our shared network drive. we have made use of folder-based workflows for both born-digital as well as digitized content workflows and they are flexible enough to allow for modification to suit a given project. what are they? the basic idea of a folder-based workflow is that each project has a folder created for it on a shared network drive. each step in the workflow is used as a folder name. objects within each folder are at that step within the workflow. this ensures that every step in the workflow is known and accounted for. this can be tailored to each project, and we use this workflow for submissions to unt scholarly works. for each project, the workflow will be contained within its own folder. even if the project is relatively small, for example digitizing four maps, it will receive its own workflow folder. this folder may be named something like “scanned_from_ohs” for a project that had digital materials originating at another institution. all of the files that are important to this project are stored in this folder structure. the folder names are based on local naming conventions. the workflow is essentially an ordered sequence of folders that define the steps in a workflow. for example, there is often the process of applying optical character recognition (ocr) to a set of scanned items. this step usually happens after other steps have been completed, like scanning the documents and performing quality control checks on the images. after ocr has been completed, we then need to create descriptive metadata and can stage the content for uploading to the digital library system. in the rough workflow just outlined, there might be one or more people involved in moving digitized items through a workflow. to facilitate this we have outlined a workflow that looks like this. scanned_from_ohs 0.staging/ 1.scanning/ 2.toqc/ 3.issues/ 4.toocr/ 5.tometadata/ 6.toupload/ 7.uploading/ 8.uploaded/ while the steps are somewhat arbitrary and are aligned with workflows at the unt libraries, a description of the steps is helpful to understand how items move through the workflow. before we do that, it is helpful for us to establish a few guiding principles of these workflows. only move content when you are ready for someone else to take it as part of the following step. keep folders as shallow as you can. the location of an object in the folder structure should reflect the work that needs to be done to the object. adjust the folder order and name to reflect the workflow. with these guiding principles in place, we can talk about how content moves through a workflow. first off, in the 0.staging folder, we can begin to sort items that need to be digitized. in many cases, we can create directories that will act as containers for digital objects as they move through the process. we have found that representing a digital object as a folder and naming it with the unique identifier for the item is helpful. then, all files related to that object are placed inside this object’s folder. moving an object between steps in the workflow is now just moving a folder. by keeping the folder structure shallow with ideally just one level of hierarchy, the object folders can easily be moved around within the workflow folders making it easy to see how items are distributed throughout the workflow. the next step in this workflow is called 1.scanning. as an example, we will be digitizing a journal for a historical society. as a technician is digitizing the item, they can place working files into this workflow step’s folder. once they have completed the initial quality control check, they will move the object folder to the next step’s folder in the workflow called 2.toqc. in this step, another round of quality control is performed by another person in the unit. as this step is being completed, if there are any issues, such as missing pages or images that need to be rescanned, the person completing the quality control can move this object folder to the 3.issues folder. if there are no problems, they can move the object folder to the 4.toocr folder in the workflow. at this step in the workflow, the person responsible for ocring items will take items from this workflow folder and process them with an ocr engine. after that has completed, the object folder is moved to the 5.tometadata folder, where they will have descriptive metadata created for them. this step may involve full-record metadata creation or could involve the creation of basic metadata that will be updated and completed later when it is loaded into the system. after this metadata has been generated, the object folder is moved to the 6.toupload folder. the final three steps in this workflow outline the ingest activities of the unt digital library where content is staged for upload and finally ingested into both the access repository, aubrey, and the preservation repository, coda. during this ingest period, the object folder is moved into the 7.uploading folder to indicate that it is being uploaded to the system. once verification of upload has taken place, the object folder is moved to the final folder, 8.uploaded. at some point in the future when the project has completed, the files in the object folder are deleted to recover network space. folder-based workflow discussion with the overview of the folder workflow complete, there are some things to discuss. this is a very simple way of organizing projects. at unt this method allows us to have multiple people working on a single project at different times and it is clear what is happening along the workflow with a given digital object because of its location. it is important for all participants in the project to remember to only move items into a subsequent folder when they are ready for someone else to take that object and process it further. if an object hasn’t finished the step that its location in the workflow folder designates, it shouldn’t be moved. this important rule allows for different team members to work independently and confidently within the workflows. because we try to keep the folder structures as shallow as possible, ideally with only one level of folders representing objects in each of the workflow folders, it is easy to see how much content is waiting in what steps for a project. simply browsing the folder structure gives an overview of where things are in the workflow. we have found that it is important to keep objects at a single level within the workflow folders. as folders become more deeply nested or contain multiple other folders, it becomes harder to understand the progress of the overall workflow or tell when objects are incomplete. because workflows can differ depending on the type of project, type of content, or because of unique requirements, it is important for the folder names to be descriptive about the step in the workflow that it represents. the number preceding the step is important to define the sequence of the steps, but the name of the step in the folder’s name should be clear to make the step understood by others involved in the project. finally, if we start a project and discover that the initial folder structure doesn’t meet the needs of the project, we change the structure to meet the needs. this is helpful because there are often many projects and a variety of types of projects being worked on at any given time. in order to keep track of what is happening within a specific project, it is important to make sure the folder structure matches the workflow. while the folder-based workflow is not a particularly complicated management structure for projects, it has served its purpose well over the years. unt scholarly works folder-based workflows as we have mentioned before, the unt scholarly works workflows are split between high-level tracking in gitlab in combination with object-level tracking in a folder-based workflow. there are different content types that are deposited for inclusion in the repository. we use different folder-based workflows for each of these content types so that their specific requirements are documented as part of the folder structure. each of these folders contains subfolders for each step we have for processing that item type. figure 5. example folder-based workflows for the different content type submissions in unt scholarly works. you will notice that in the seven examples in figure 5 there are many steps or even whole portions of the workflows that are similar to others. this is seen as a feature of the folder-based workflows. in the case of the posters and presentations workflows, it isn’t so much the workflows but the final ingesting steps that are slightly different for oversized posters as compared to the presentations. by splitting them up into two separate workflows, it is easier for the person doing the final ingest to remember the different configurations needed to ingest the items into unt scholarly works. some of the workflows, specifically audio, video, and data do not contain steps such as toocr because that step is not appropriate. this is another example of tailoring the folder names to the needs of the specific workflow. when an item is received for ingestion into unt scholarly works, an object folder is created in the first folder in the appropriate workflow. the object folder’s name is the same as the issue name in the unt-scholarly-works project in gitlab. once an item has been successfully moved through the folder-based workflow on the network drive, the contributor is emailed with information about their submission and the gitlab issue is closed to complete the whole process. conclusion it is important for us to occasionally rethink our workflows and examine the tools we use to manage projects. the occasion of having new staff is often a good time to make sure that documentation is up-to-date and reflects the actual workflows in place at the institution. a situation like the covid-19 pandemic can further identify challenges to workflows when new requirements are placed upon them, such as completely remote workers and different people being involved with the process. the unt libraries took a combination of these two events as an opportunity to do such a thing. by further documenting the use of folder-based workflows that had been used successfully in many other projects, we were able to adjust our processes so that they better reflect the items we were receiving from faculty, staff, and students submitting content to the unt scholarly works repository. likewise, we moved from an ad hoc email-based workflow for tracking submissions to a new process using the web-based tool gitlab. this software allowed us to further codify our workflow using the tools and features present in gitlab projects, such as issues, labels, and a wiki. by focusing our efforts on simple-to-adjust methods for organizing content, like the folder-based workflows, we are able to make changes to workflows without additional overhead. likewise, working with a system that is already being used by several groups within the organization allows us to leverage existing infrastructure for our needs and doesn’t add additional tools to support in the organization. together gitlab and the folder-based workflows have allowed us to improve the productivity of the unit around submissions to the unt scholarly works repository. notes [1] for more information on git and gitlab in a library context see engwall k, roe m. 2020. git and gitlab in library website change management workflows. the code4lib journal [internet]. (48). available from: https://journal.code4lib.org/articles/15250. [2] more information on creating issue templates in gitlab is available at https://docs.gitlab.com/ee/user/project/description_templates.html#description-templates. about the authors whitney r. johnson-freeman is the repository librarian for scholarly works, the institutional repository, at the university of north texas. her primary focus is in digital curation, but she is also interested in data and assessment. mark e. phillips is the associate dean for digital libraries at the university of north texas. he has a wide range of research interests, including web archiving, digital library infrastructure, and metadata quality. kristy k. phillips is a phd student at the university of north texas college of information. her research focuses on computational linguistics, with a particular interest in building tools for low-resource languages. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – purposeful development: being ready when your project moves from ‘hobby’ to mission critical mission editorial committee process and structure code4lib issue 16, 2012-02-03 purposeful development: being ready when your project moves from ‘hobby’ to mission critical throughout the library community examples can be found of development projects evolving into mission critical components within an organization’s workflow. how these projects make that move is unique and varied, but little discussion has been had about how these projects impact their developers and the project community. what responsibilities does a developer have to ensure the long-term viability of their project? does simply freeing the code meet those long-term responsibilities, or is there an implied commitment to provide long-term “care and feeding” to project communities built up over time? code4lib represents a group of developers consistently looking to build the next big thing, i’d like to step back and look at some of my own experiences related to the long-term impacts that come with developing successful projects and communities, and try to provide library developers food for thought as they consider their own ongoing responsibilities to their projects and user communities. by terry reese introduction the role of the developer–and attitude toward open-source in general–has changed a lot within the library community over the last ten years. while libraries have always had individuals writing code and building tools, this work has predominately been relegated to the fringes of library research or hidden away within the technical services and systems communities wrangling with marc and library management systems. and while marc and library management systems haven’t gone away, development activities have become an integral part of the library and have gained acceptance as both fertile ground for library research and a necessary component of a successful library. in many respects, i think that this will be one of the most enduring legacies of the code4lib movement; legitimizing the work developers do to the library community. developers and open-source software no longer flummox the vast majority of library administrators, who now see development opportunities as areas for research, collaboration and innovation. libraries, in large part, have taken on large development projects and encouraged developers to think innovatively and experiment liberally. unshackled, developers have worked to re-engineer the library, and in the process are actively sharing their code, expertise and experiments within the wider library community. what’s more, as libraries continue to evolve, more and more of their critical workflows and systems will be developed utilizing components or tools created in and around the code4lib community. however, in the midst of this sea change within libraries, it is important to consider the long-term ramifications for developers as we move forward. for every library technologist or developer hoping to create the next big thing, there is the underlying question of how a successful development project may impact that developer or their organization. and what’s more, if a project is really successful, how does the developer or host institution determine their long-term responsibilities to a particular community or project? is it enough for a developer to simply open-source his/her work or is there a larger responsibility to provide the necessary “care and feeding” to the communities that they help to establish? these are difficult questions with few “right” answers because they represent large opportunity costs. however, as libraries become more reliant on the larger library development community to provide shared workflow solutions, it becomes even more incumbent on library developers to think critically about the long-term impact a project may have, both for themselves and the larger development community. while every developer hopes to build a successful project, few are prepared for the community entanglements and the long-term hold these projects can have on a developer’s career. the privilege of software development it is sometimes easy to forget that being a developer is a privilege. we spend our days negotiating with unfriendly systems and archaic data models. we find ourselves needing to be licensing experts as we search for components, libraries and building blocks from which to build our work. these every-day issues can make one feel like sisyphus, forever pushing against the mundane tasks that interrupt the development process. but the reality is that as developers, we create things–very cool things that people count on and come to rely on. whether intentional or accidental, people become developers because they have the opportunity to take those things that they imagine, and make them real. it is a privilege because “we collectively are given the opportunity to create things that matter: to individuals, to teams, to organizations, to countries, to our civilization. we have the honor of delivering the stuff of pure intellectual effort that can heal, serve, entertain, connect, and liberate, freeing the human spirit to pursue those activities that are purely and uniquely human” (booch, 2004). we are at our best when the work that we create transparently enables the user to make connections and meet their needs. users become co-developers, not through the writing of code, but through the interactions between the developer and the community. within the code4lib community, we tend to think of our work as being about the code, but in reality, that is just a very small part of a project. users tend to provide the vision, the context that enables developers to find problems and solve them. through our work, the user and developer form a symbiotic relationship allowing the developer and user the ability to imagine new ways to interact with the world. and because of this relationship with the user, the privilege of software development also comes with a responsibility. within the code4lib and library communities, we have done a very good job of thinking creatively and pushing innovative approaches to these larger problems. we continue to encourage library developers to think differently about the library, to reimagine workflows and what it means to be a library in the 21st century. and yet, we tend not to think about the long-term responsibilities developers and organizations will have to the user communities that spring up around these new innovations and research. as the code4lib and library development communities continue to move libraries in new directions, these communities need to engage in a new discussion; one around the idea of more responsible development that requires developers to consider not just the life-cycle of their project, but also the long-term stewardship of their creations and the user communities that spring up around them. developers need to be prepared to fulfill their responsibilities to their work and can do so by planning not just for a successful development, but for accidental success as well. what if you succeed? when starting a “hobby” or experimental project, few people expect for it to fail, but even fewer plan for it to succeed. when one thinks about it, that’s really not all that surprising since most “hobby” development within the library community starts out as a tool to meet a local need. if we looked around our own institutions, nearly everyone could find one project that would meet this description. these projects serve vital needs within our organizations and often times do such a good job, that they become viral within our communities as well. through word of mouth, these tools spread outside of our organizations, taking on a life of their own and, intentionally or not, binding an organization and developer to supporting a particular tool set. i have had some experience with this kind of accidental success. in 1999, i started writing a small marc-based set of code libraries as a way to learn how to understand the marc format. because these tools ended up having a significant impact at oregon state university, i was encouraged and cajoled to release what would become marcedit. when marcedit (http://people.oregonstate.edu/~reeset/marcedit/) was first made available, i had no idea at the time how quickly the tool would become as successful as it has, nor did i understand the size and scope of the community that would develop around it. in the 12 years since i first released marcedit, the application has become a well-used utility for manipulating library metadata. it has been written about and cited in a number of books and articles, and has become in many organizations a critical part of their metadata workflows. for my part, the program has become both a vehicle for experimentation and research; and a never-ending stream of support questions from the user community. as a group, developers tend to underestimate the time necessary to answer questions related to their work. like most, i had made the assumption that the “web” would shoulder the lion’s share of the support requests, while in reality, developers or their organizations need to make some hard decisions related to the amount of support that they are willing to provide. even today, i still receive 20-25 personal requests a week for support or assistance with a metadata project, which has a real opportunity cost in relationship to my time as a developer to do new and creative work for the community. for the first time in 2011, i tried to get a handle on the size and scope of the current marcedit user community. utilizing marcedit’s automatic updating feature, i was able to capture access log statistics representing an active usage of the program. what i found was surprising – taking just the first million executions over the last year, i could see that marcedit had been actively used in over 143 countries/jurisdictions. what’s more, breaking down the active user community, nearly half of all usage from the tool came from outside of the united states. iso code country total uses us united states 619,884 ca canada 103,558 au australia 47,916 gb united kingdom 34,004 es spain 24,402 tw taiwan 18,816 hk hong kong 17,740 sg singapore 14,818 in india 12,022 mx mexico 10,386 it italy 9,344 pk pakistan 9,230 my malaysia 8,842 fr france 8,826 de germany 7,174 table 1. top 15 countries by usage while i have long suspected that marcedit had a substantial user community, i had no idea that it had grown to represent such a diverse set of the library community. without meaning to, i’d managed to create something that a large number of people were finding useful in their daily work. however, this success also raised questions related to my long-term responsibility to the marcedit community. as the sole developer of marcedit, what are the long-term responsibilities to provide support, project stewardship and succession planning of the application – and does a user community have the right to expect this level of support? these are important questions for any member of the code4lib (and larger library development community) as more and more community driven development is becoming critically embedded within the library. how we steward our work now and ensure its long-term success will continue to play a very important role in how the library community views and ultimately relies upon the development community. the responsibilities of software development if you accept that being a software developer is a privilege, then it is equally true that software development comes with a set of responsibilities. the unwritten contract between the user and developer underscores certain expectations related to support, stewardship and longevity – expectations that many developers may not fully consider when building a user community. what’s more, some developers may indeed view the idea of stewardship as out of scope for their particular work or project. within the code4lib community, it’s very common for developers to believe that releasing their tools as open-source fulfills the stewardship contract with their users. from a developer’s perspective, making the source code available ensures that someone, down the road, will have the ability to pick up the project and continue development. for some projects, this type of foster care approach to software development may make sense – especially if the project in question never succeeded in growing its user community. however, projects developed at libraries tend to spread virally. developers or users attend conferences, they write articles–word gets out and very quickly these projects become embedded in best practices and workflows at other organizations. these projects require much more “care and feeding” and for the developer or organization actively seeking to build a community, a great deal more should be expected than simply releasing the source code for a project. stewardship of open-source projects over the years, library administrators have become much more comfortable relying on open-source projects. at the same time, very few within the library community could be considered good open-source citizens. (intridea, 2011) while libraries have generally become much less resistant to using open-source tools and projects, very few actually contribute anything back to those projects. there are obviously exceptions, like the dspace effort, that have had success building robust user and development communities around open-source projects. however, engendering this level of involvement and ownership in a project doesn’t happen by accident. the dspace community, for example, was created through a very intentional development process. securing funding to seed and support a user community, mit worked with a number of early partners to advocate for the long-term health of the project. as the project has grown, mit has been able to step aside and become a participant in the larger dspace community, while handing off the project to a new set of maintainers. however, mit’s experience in growing the dspace community is atypical within the library community. in most cases, developers or organizations may be able to build user communities, but struggle to create the thriving developer community needed to relinquish stewardship of a project. for example, oregon state university libraries (osul) developed an open-source content management system for librarians called library à la carte, to support the development of course guides, subject guides, portal pages and modules. osul sought to develop the project to provide libraries with a more flexible tool kit than found in existing products like libguides, as well as providing a framework for librarians to think innovatively about how they connect to users. by most metrics, the project has been successful building a community of users and pioneering a number of user oriented innovations. likewise, osul has actively pursued a plan to enable libraries to utilize this project, providing support and continued ongoing development. however, while building a user community has proved to be somewhat successful, encouraging the growth of a developer community has proved elusive. while osul has engaged in training, hosted impromptu user groups at national conferences, and even for a brief period offered a hosting service for a limited number of partners, an active developer community never formed around the software. for osul, this means that support for the project continues to be shouldered solely by the libraries, which does incur opportunity costs. the experience osul has had is probably much more typical of development projects within the library world. the issue then becomes what role or responsibility does the developer/organization have for long-term stewardship of a project. in the case of library à la carte, osul encouraged the development of a user community and now feels a deep obligation to that community, making decisions about the library à la carte program much more difficult. so what should developers do? as developers and organizations seek to make their open-source projects available, they should plan not only for success of the projects but include the long-term stewardship of their project as part of the development life-cycle. project developers should consider the following: have clear community goals and a plan of action to reach them many projects flounder because of the belief that a vibrant user community will equate to a vibrant developer community–it doesn’t. just like user communities, projects looking to birth and support a strong development community require purposeful action. it’s not a coincidence that projects that make a priority to support developers and actively court their participation throughout the project lifecycle engender the most community support. at osul, this was probably the biggest single miscalculation that was made when promoting library à la carte, assuming a vibrant development community would spring out of the existing user community. in reality, these two groups have different needs and oftentimes will not be made up of the same principle players. by not having a clear plan of action, osul was able to build user interest in the project, but to this day, has never been able to generate the same interest from within the developer community. what’s more, given the rapid rate of technological change, most projects have a narrow window to make those connections–missing that window will result in a missed opportunity and ultimately will make community building more difficult in the future. determine if you can provide support one of the least transparent processes within the library development community surrounds the topic of support. developers need to be exceedingly clear what type of support will be provided and for how long. invariably, questions related to enhancements, system requirements, component incompatibilities, updates or simple troubleshooting will need to be addressed. when a project or tool is first released, developers will often times go to great lengths to provide support for their work. while this will certainly help to build a user community, it also raises false expectations. users that come to rely on a tool or service may continue to require a high level of hand holding, making long-term support more difficult as a project attracts more users. developers or their institutions need to be fully transparent to potential users regarding the support that they can provide, as well as their own expectations of the user community. while it would be naïve to expect a large number of project users to contribute code back to the project, there are many other tasks in which members of a user community can participate. supporting documentation efforts, creating support groups, financially supporting the development efforts of a developer or host institution – the ability to contribute to a project go far beyond just contributing code back to a project, and all ultimately help determine a projects long-term success. likewise, users themselves will need to understand support expectations so that they can make informed decisions regarding a project’s fit for their institution. provide a development roadmap one of the benefits of an open-source project is that the source code is always transparent to the user, even if the user is unable to do anything with it. just as important to source-code transparency is the accompanying development roadmap. while few users will actually contribute code back to a project, the presence of a clear development road map tends to encourage community members to seek out ways in which they can support a project. users that understand and have confidence in the development process have a much greater incentive to become active project participants as testers, documenters or simply providing valuable feedback relating to functionality. given the rapid rate of change facing libraries, projects actively seeking adoption by other institutions have a responsibility to provide a clear development path and goals. this allows their users the ability to understand and mitigate changes to a tool or project without impacting critical services. have a clear exit strategy for developers projects rarely last forever. at some point, developers and organizations move on. like any project, open-source projects need to have a succession plan in place. for projects that are successful at building strong developer and user communities, the continued stewardship of a project or tool can generally be transferred directly to the community. dspace and koha are great examples of projects started by a set of institutions that successfully transferred stewardship to another entity [1]. given the fluid nature of many open-source projects, developers need to be especially clear when detailing the life-cycle of a tool or project and need to be realistic about their overall interest and capacity to provide long-term support. developers need to engage in responsible development, considering not only their own finite resources, but also the community of users that they have worked to create. have a clear exit strategy for users libraries insist upon exit strategies from vended projects, and should hold open-source developers to the same standard, requiring projects that provide polished exit strategies. while it is true that users of open-source projects or tools always have access to their underlying data, it is unrealistic to expect a project’s users to develop their own methodologies to extract and repack their data when a project is abandoned. developers and their institutions have an ongoing obligation to those user communities through the total life-cycle of their projects. building new tools and making the source code available to the larger library community is an exciting time. for new developers, promoting and seeing your work make a difference is always a thrill. however, taking the time to consider the long-term stewardship of your project will not only make the overall project experience a better one, but builds trust within the larger library community by demonstrating an understanding of the long-term implications of making your work available. stewardship of closed-source projects while the topic of closed-source projects tends to rarely come up for discussion within the code4lib community, a developer or institution may have very good reasons for keeping the source code for a project closed and outside of the public domain. sometimes these reasons are related to remuneration, sometimes due to organizational concerns and, more often i think, because the project represents a hobby or an itch that a particular developer wishes to scratch. and while many in the code4lib community tend to discourage the closed-source model, the library world is full of examples of closed-source projects that have made and continue to make significant contributions to the field. while open-source tools continue to become more ubiquitous, the software that drives the library at the enterprise level still largely remains closed-source. many tools like ezproxy, illiad and contentdm started out as resource projects only to grow into much larger development efforts, while others come from large system vendors or boutique software developers that create tools like bookwhere and marc magician. these tools and systems play an important role in the library software ecosystem, enriching the community and facilitating innovation [2]. however, for developers choosing a closed-source model, considering at the very beginning their responsibilities to their user community becomes even more important. while open source project maintainers need to consider their roles as stewards, the fact remains that the source code is available and open for further development. if a project is dropped, it may languish and fall into disrepair, but if an organization has sufficient need, they can revive the project and continue its development given enough time and resources. closed-source projects are very different, in that the user communities that build up around them are completely dependent upon the project maintainer to continue to support and improve the product. marcedit is another example of a product within the library community that follows a closed-source development model. i’ve been asked a number of times throughout the years why i haven’t open-sourced the application since i participate in and have lead a number of other open-source projects. i think sometimes people may believe that marcedit remains closed due to some bureaucratic wrangling with oregon state university (osu), rather than a choice that i purposely have made regarding the development process. i’m sure that some people within the code4lib community may have experience with this type of bureaucratic nonsense, but this has not been my experience at osu. no, in my case, i have decided to keep marcedit closed because i have a very personal connection to the software–developing it has been a passion of mine for almost 13 years now and one that i selfishly keep to myself. however, in doing so, i understand that i have very specific responsibilities to the user community who have come to view the project as a necessary part of many organization’s technical services workflows. when considering the stewardship of a closed-source application, one should consider all of the questions that relate to open-source development. in addition to those initial questions, maintaining a closed-source application requires careful consideration of some additional issues due to the closed nature of the project: provide a transparent and well-defined enhancement process unlike open-source software which thrives on communities of users that are also good open-source citizens and contribute back to the development, closed-source software generally employs a user feedback model. this doesn’t mean that community members of closed-source projects are passive participants. actually, it can be quite the opposite as communities built around closed-source projects must proactively work with the developers to ensure that community needs remain a development priority. like open source projects, community members can and should play an active role in helping to support the user community. at the same time, developers of closed-source projects and the user communities that build up around them often are insulated from each other. this is a challenge for both users and developers because the developer is for all intents and purposes the bottle-neck within this model. with only so many resources, developers must strive to provide their user communities with a well-defined enhancement process and issue tracking system to engender transparency in the development process. like open-source projects, project roadmaps are vitally important to provide transparency to the development process and provide a way to interact with the user community. one of the dangers for developers of closed-source projects is the potential to become out of touch with the user community. developers of these projects must remain vigilant to maintain the lines of communication and encourage active participation in the enhancement and testing process. determine who will provide support like all projects, one of the primary success drivers for a closed-source project is the level of support provided to the user community. while open-source support tends to be provided through the project community at large, closed-source projects are disadvantaged in that support must come directly from the software developers. for large enterprise development efforts, support is often funded through fees that libraries pay for the software. however, for hobby projects or boutique development efforts, the path to providing support oftentimes isn’t as clear-cut since time spent on support is ultimately time spend away from working on the project. how that support is offered can vary greatly, but users are much more vulnerable to support issues when working with a closed-source project because their ability to troubleshoot issues on their own is often limited by the software. at the same time, an active user community can be a boon for a closed-source project, as it engenders a sense of ownership in the work, as well as providing functional experts that can be utilized for both testing and feedback. in the nearly 13 years of developing marcedit, i had seen the number of support requests being forwarded to me topping over a hundred a day. questions would often be repeated, but answering each question individually eventually begins to take its toll. a developer looking to go the closed-source route need to understand that providing support for a project is the single most time consuming aspect of the development. the project’s user community must become a support resource, through the use of listservs or forums to allow the user community to participate in the support process. this allows users to gain a sense of ownership of the project, while freeing the developer to spend more time developing the project. in my own experience with marcedit, providing the user community with a public listserv to ask support questions has been a boon to the community. what’s more, the listserv has helped to identify who within the user community has specific knowledge of relevant workflow specific tasks. this has given me a set of experts to query when discussing new functionality, and has given the user community more depth and a different type of available support. escrow the code because close-sourced projects don’t make the source-code available to the public, project maintainers must take special precautions to ensure that the source-code remains secure and accessible. this is especially true of single developer projects, where one coder may have the only copy of a project’s source. for developers looking to build a user community around their closed-sourced projects, it is the developer’s responsibility to detail the steps taken to preserve the source-code so that it can survive beyond the project team. again, with marcedit, this scenario nearly played out in the summer of 2010 when i was hit by a car on my bike. since marcedit is a closed-source application, many people began to wonder what would happen to the project if the sole developer suddenly disappeared. this is a very valid question and concern. it’s been my experience that unattended software tends to have a short shelf-life; with its applicability depreciating the longer it remains idle due to software defects, standard changes or changes within the software’s system ecosystem. for developers looking to build closed-source systems, one important decision that needs to be made and documented is how the source-code will be escrowed for future developers. for years, marcedit’s source-code has been escrowed on two independent remote systems, accessible by two or three (it varies) other individuals besides myself. these escrowed copies act as insurance policy of sorts for the user community that the project could survive beyond any individual developer of the project. develop a succession plan in addition to the escrowing of code, closed-source developers must also plan from the very beginning how the project will succeed without them. succession planning is a necessary obligation for a developer to undertake to ensure that the project can continue to be maintained when the developer finally wants to move on to something else. fortunately, there are a number of available options, from deciding to open-source the code and encouraging interested members in the existing user community to take up the development of the project to appointing a new project developer. additionally, closed-source projects featuring a single developer should take additional steps beyond escrowing the project’s code, including considering how to support emergency succession of a project. in my case, this is something that i have long neglected in my own succession planning in regards to marcedit. while those that can access the escrowed code have the ability to continue the project, i’m not sure they would want to take over long-term development of the project. however, closed-source developers have an obligation to their user communities to give a lot of thought to the life-cycle of their projects and work to ensure that their role as the developer is ultimately expendable. as one can see, developing a closed-source project places a number of new and unique obligations on a developer. this is one of the costs for keeping the source-code of an application closed. as i have found working with marcedit, there are tradeoffs going this route, and one of these tradeoffs is the added responsibilities to the user community. whether i want to or not, i’m wedded to the marcedit user community because of the decision to close the project code. developers looking to follow this same path need to realize that this decision comes with many obligations that cannot be easily untangled once a user community develops around a specific project or application. conclusion the negotiation between the developer and their user community can be a very personal thing. however, as the code4lib and library developer communities continue to mature and play a more important role in determining the direction of libraries in the 21st century, we also need to remember what great privilege it is to be a developer. we have the ability to break down barriers, build connections and help individuals make new discoveries. it really is a privilege, but it also comes with an obligation to think beyond our immediate involvement in the work and consider how we will steward a project throughout its life-cycle. it’s an important discussion developers need to have with each other, as well as at their institutions, if libraries are going to continue supporting and encouraging an active development community. about the author terry reese is the gray family chair for innovative library services at oregon state university (osu) overseeing the development and implementation of new strategic initiatives for the libraries. he is the author of a number of metadata related software packages and libraries like marcedit, a marc/xml metadata software suite and the c# oai harvesting package. he is a regular requested speaker and has published a number of works on digital libraries and library metadata issues, including co-authoring a book with kyle banerjee entitled, building digital libraries: a how-to-do-it manual (http://www.amazon.com/building-digital-libraries-how-do/dp/1555706177/). personal homepage: http://people.oregonstate.edu/~reeset/ notes [1] while the current controversy surrounding the koha project would seem to indicate otherwise, i believe that it actually demonstrates how successful this project has been at diversifying its community. koha has been so successful, that the project has been able to survive not only being forked, but has seen both forks flourish. real issues exist between the two communities, but i think that this group still represents a great example of how a project can be transferred to a community (as well demonstrating some of the potential issues that can arise when community members philosophically differ). [2] i know many will argue that many closed-source tools have traditionally impeded libraries from innovating, but i think that attitude is naïve. while many enterprise system in the library community have admittedly provided less than stellar support to extend their systems beyond their scope, it’s unfair to not recognize the work these groups have also done developing tools like electronic resource management systems and supporting openurl development and support. references booch, g. (2004). the privilege and responsibility of software development. retrieved 11 18, 2011, from http://www.ibm.com/developerworks/rational/library/2101.html intridea. (2011). 10 tips for open source citizens. retrieved 11 02, 2011, from http://intridea.com/2011/8/11/10-tips-for-open-source-citizens subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – talking portraits in the library: building interactive exhibits with an augmented reality app mission editorial committee process and structure code4lib issue 46, 2019-11-05 talking portraits in the library: building interactive exhibits with an augmented reality app with funding from multiple sources, an augmented-reality application was developed and tested by researchers to increase interactivity for an online exhibit. the study found that augmented reality integration into a library exhibit resulted in increased engagement and improved levels of self-reported enjoyment. the study details the process of the project including describing the methodology used, creating the application, user experience methods, and future considerations for development. the paper highlights software used to develop 3d objects, how to overlay them onto existing exhibit images and added interactivity through movement and audio/video syncing. by brandon patterson introduction augmented reality (ar) and other display technologies are increasingly being utilized in educational settings to create immersive, interactive, and exciting learning environments[1][2][3]. these technologies are found to promote participation and motivation while creating a novel learning environment that mixes the virtual with the real world[4]. the transition in online learning from conventional multimedia platforms to a more creative and engaging augmented learning environment provides opportunities for educators to engage learners in exciting new ways. according to a study by chang, et al.[5], an ar guide to an in-person museum exhibit enhanced a visitor’s learning effectiveness, their engagement, and extended the amount of time the visitor focused on the paintings. the goal of this project was to create a custom ar software to enhance the experience for viewers of an online exhibit in the same way that previous studies had shown it did with an in-person one. this project was funded in whole or in part with federal funds from the department of health and human services, national institutes of health, and national library of medicine, under cooperative agreement number ug4lm012344 with the university of utah spencer s. eccles health sciences library. with these funds, a small team built a prototype app allowing viewers to see characters from the national library of medicine’s (nlm) online exhibit, “renaissance science, magic, and medicine in harry potter’s world”. using graphic software and local expertise in the library, the online activity, “lumos: renaissance thinkers behind the magic” was transformed to include a virtual environment using ar. the characters interact with viewers as talking portraits did in the harry potter book series[6], providing enhanced engagement and time spent with the exhibit. findings this study found that when someone interacted with the augmented reality portion of the nlm exhibit, it resulted in increased time spent engaging with the material and improved levels of self-reported enjoyment with the experience. the retention of information conveyed by the exhibit was the same as that viewed on the website. this report details the process of the project including methodology used, prototype creation, user experience testing, and future considerations for development. methodology there were six phases of development for this project that included planning, configuration, migration, testing, documentation and operation. phase 1: planning the planning phase consisted of the tasks necessary to successfully setup an augmented reality environment. it allowed us to better acquaint ourselves with the content and what the viewers expect when experiencing the online exhibit. we’ve outlined our roles as follows: a project manager led the week-to-week progress of the study, supported the staff, documented the process, and created the final report. the technical lead came from the it department and built the models for the characters to interact with the viewer. he also led the app integration. for hardware, we used existing pc computers and our personal mobile devices. our software needs consisted of unity, mixamo, blender and custom programs to create and animate 3d elements, described under “configuring the model” heading below. in the planning phase, the first task consisted of creating a character persona for a key figure in the exhibit, nicolas flamel. personas are based off factual information about them and include personal details that will be used to create elements of the characters including script, mannerisms, and animations. experts in the field of augmented reality who work for the university’s electronic arts and entertainment (eae) program were consulted to give insight into the process and provide feedback about the project design. scripts were written by the project manager to align with the character personas created, described under the “persona and script development” heading below. phase 2: configuration the configuration phase was the most time intensive and consisted of the character design and animations. characters were created, wire-framed, animated, and rigged for movement via the blender development platform, which is a free open-source 3d creation suite. the character was exported into unity, a game developer engine, to coordinate when animations play and how it looks, including adding lighting and developing shading. we did a voice recording during this phase to create a working prototype which we tested with patrons of the library. phase 3: migration this phase consisted of moving the animation to an app for deployment. unity creates an app template for developers which we used to connect to the virtual environment app created. the app is currently in prototype form and is not publically available. the authors are working with the funders of the project to build out the prototype and will release the app through the national library of medicine. after the project has ended, the authors would be happy to share the code of the project with others per guidance of the funders. it is restricted to android users at this time with future development to make it available on iphone. content was integrated and transformed to fit the specifications of the mobile devices. the project manager gathered users to test the app. phase 4: testing this phase consisted of user testing which informed the outcomes of the study, which were: increased retention of information conveyed by the exhibit; more time spent engaging with the material; and improved levels of self-reported enjoyment with the experience. we conducted structured interviews before and after the ar-enhanced version of the exhibit. we selected 30 subjects from a pool of patrons from throughout the university, 10 who interacted with the nlm online activity in its existing form (control), another 10 who interacted solely with the ar component of the nlm exhibit and 10 who interacted with both components. identical surveys were given to all groups. some questions for the subjects included: rank your enjoyment of the online activity (1 to 10). what do you enjoy about the online activity? what suggestions do you have to improve the experience? content specific questions. we then analyzed the data based on survey questions and short interviews with subjects. phase 5: documentation throughout the study, the project manager and technical lead documented their progress. a summary report was written and given to nlm to provide suggestions for feasibility and scalability of future virtual environment projects for nlm exhibits. the report included: survey results and analyzed data timeline and key milestones for project creation documentation of project from lead and student developer peer evaluation on technology recommendation and suggestions for future projects guide to support and produce similar content phase 6: operation this prototype was meant to be a test case for the exhibit and additional funds are needed to roll it out into a fully operating application. with additional time and resources, the prototype can be a guide to develop more characters and interactions with guests of the online exhibit. prototype development to create the prototype, we developed a persona and script and configured a model using software and hardware detailed below. persona and script development a learner persona was created to get a sense of the target audience for this application (figure 1). we used this persona as a way to target the app toward a specific audience and got direct feedback from patrons of the library. figure 1: learner persona to use as target audience for app. a character persona and script was then written about nicolas flamel (figure 2). future characters developed for the app and their script would go through a similar process but experts in the field of renaissance science and medicine would be consulted for accurate information. figure 2: nicolas flamel persona and script for app. configuring the model to configure the model for the app, free and for-cost software and assets were used to develop the character of nicolas flamel. instead of hiring an artist, we opted to create our own character via adobe fuse, a character model generator. although less in line with the art of the exhibit, it provided a good prototype for what the app could do. we then used mixamo, blender and lipsync pro as a way to animate the character. to make it ada compliant, we used unity to add closed captioning to match the voice of the character and applied a custom shader to have the character match the style of the exhibit. finally, vuforia was used to create the ar component which took the image and replaced it with the animation on the phone. adobe fuse adobe fuse is a character model generator. because the university of utah has an agreement with adobe, it is no cost to us. one can choose base meshes to work with and modify the model in various ways. for example, selecting the nose allows one to adjust the nose height, width, depth, etc. after the character looked as close to a reference photo as possible, the technical lead exported the model as a 3d model format (obj) with the accompanying material package (see figure 3 & 4). figures 3 & 4: nicolas flamel character created in adobe fuse. mixamo mixamo is a web based character rigger and animator. rigging a model consists of creating a “bone structure” under the model and assigning each vertex to one or more bones; a slow and tedious process to do by hand. fortunately, fuse exports their character models in a standard t-pose position that allows mixamo to rig the character with ease. the technical lead uploaded the object file (obj) generated by fuse and selected the required joints (figure 5). figure 5: rigging of nicolas flamel and selecting joints using software mixamo. after rigging was complete, the character was animated with the many animations available from mixamo (figure 6). the technical lead selected an animation called “sitting talking” and downloaded the pack as an fbx file. this is a file format that includes models, rigging, animations, and materials. figure 6: available preset animations for nicolas flamel on mixamo. blender the final task for getting the model ready for full animation was completed in blender. the technical lead imported the file of the sitting-talking nicholas. he replaced his small beard with one that he modeled to better match the reference photo. then using the shape keys tab, he connected to the mouth of the model and created ten phonemes (the shapes that the mouth makes while making certain sounds while talking) that can be used to visualize english speech (figure 7). he then exported the package as an fbx file. figure 7: facial phonemes created in blender for nicolas flamel character. unity + lipsync pro importing the fbx file and material packages into unity was the next step. once in unity, the technical lead generated a standard shader material for each of the character objects. he linked all of the textures from the material package to the appropriate shaders. he then installed lipsync pro to get started on animating the character’s face (figure 8). figure 8: lipsync pro software interface. this process was quite simple as the blendshapes section in lipsync allowed the technical lead to select the shape keys he created in blender as a simple pulldown menu. once the phonemes were all linked up, he needed to create an audio data file to keyframe the various phonemes with the audio track, a simple yet tedious process (figure 9). figure 9: audio data file of recorded voice to match keyframes for animation. closed caption to address the idea of closed captioning, the technical lead created a simple on-screen text box in unity where he manipulated the text during runtime. single lines of dialogue were typed out and added to a text field along with a time value to sync with the moving lips of the character (figure 10). once populated, a button was added to the start of the app which allows the user to start the video with or without the captions. figure 10: unity function to add caption to application. custom shader the goal of this shader was to generate a “hand drawn” effect to the visuals to match the look of the exhibit. the technical lead wanted the model to look as though it was being drawn or sketched while the video was playing. we purchased a shader pack from the asset store called hand-drawn shader pack which almost created the desired effect, but not quite. fortunately, the author of the hand-drawn asset included access to the code for all of his shaders. the technical lead was able to walk through the code and understand the principle of how it works. using this knowledge, a customized version of the shader was created (figure 11). the differences include: separating the shading texture “levels” into individual images. this gives a little more direct workflow in creating the sketch textures for each shading level. each shading level includes a cutoff threshold to give a hard edge between shading texture levels. this allows adjustment for each shading level individually. the base color for the material can include an alpha channel which will allow the underlying material to “peek” through which generates some interesting effects. figure 11: different shader shapes and subsequent effects. this new shader offers much more flexibility to match the project’s needs. vuforia the final step was to track the main image through a devices camera and display this model and animation within the image. to accomplish this, we made use of the vuforia plugin for unity. after installing the plugin and creating an account, the technical lead uploaded the image to the developers vuforia website where the image was analyzed for trackability. fortunately, our image of nicholas received a 4-star rating, which makes for a good image to track (figure 12)! figure 12: vuforia target manager for nicolas flamel project with four and five star images. the technical lead downloaded the tracking package and activated it with the previously created unity scene. this generates a sprite with the tracking image assigned. at runtime, the image is hidden from view while vuforia searches for that image from the camera input. when found, the sprite within the scene is aligned with the world image and the tracking image on the sprite is then hidden. but, any object attached to the tracking sprite will then stay aligned with the world image (figure 13). figure 13: nicolas flamel sprite as found in nlm exhibit attached to virtual environment. so, with that in place, the technical lead built the background composition with respect to the tracking image, as if it were a window. finally, he masked off the area around the tracking image with a camera mask. this will show whatever the camera is seeing (hiding the background models), and provides the “looking into a window” effect (figure 14). figure 14: desired “looking into a window” effect for app. with everything set to animate on start, the technical lead built the app to be used on an android device and tested the effect. satisfied with the results, he added a simple user interface to delay animation until a button is pressed. this allows for the selection to have captions enabled or disabled. results compiling the series of parts created from developer tools listed above made the desired application – which recognizes the given image of nicolas flamel and replaces that image with a 3d rendered scene – fully animated, with audio (figure 15). this animation tracks the user’s movements and shows the scene in the appropriate orientation. for more about the project, see the following: video of application found at https://youtu.be/yvjp2an0ogw. figure 15: finished prototype app with original image and interactive virtual environment. pricing and costs the following is a list of pricing for the development of the prototype and costs associated to continue the project. unity personal – free to use if revenue or funding raised is less than $100k/year. if revenue exceeds $100k, prices range from $25 – $125/month vuforia – free for development, after deployed on app store, get access to classic ($499 one-time) or cloud-based ($99/month) subscription lipsync pro – available on asset store: $35/ one time fuse – included in adobe creative cloud: $52.99/ month mixamo – free to use blender – free to use user testing a weeklong usability study provided insight into the application and whether or not it met the goals of the study. a testing protocol was developed to remain consistent throughout the process. all users interacted with the same portion of the online exhibit, “renaissance science, magic, and medicine in harry potter’s world.” a pre-survey was given to all participants to get a sense of their familiarity with renaissance science and medicine, harry potter, augmented reality and online learning. we also wanted to get information about their highest level of education and age. we split participants into three groups and timed them while they participated in one of three ways: group a consisted of participants that would look at the online activity and the app. group b would only look at the online activity on the website. group c would only look at the app. we then asked them follow up questions regarding their enjoyment of the app, responses to why they did and did not enjoy it, and a few questions to understand their comprehension concerning nicolas flamel. findings the usability study showed that groups a and c, having interacted with the app, had greater enjoyment interacting with the exhibit and spent more time with the content. interactions with the app (group a and c) were an average of 116.6 seconds and 62.6 seconds compared to no app (group b) at 36.6 seconds. groups a and c rated their enjoyment levels at an average of 4.25 and 4.6 (out of a scale of 5) as compared to 3.05 for group b which didn’t have the augmented reality app. many users commented on how they enjoyed the interactivity, the ability to view the character from multiple perspectives, and the novelty of the experience with added information coming from the application. some pointed to the length of the experience, matched artistic styles and interactions beyond augmented reality as possible improvements to the experience. retention of knowledge about nicolas flamel wasn’t any better than having read the website. researchers in ar development have addressed users paying too much attention to the novelty of the new technology and ignoring the surrounding environment[7]. in fact, one user admitted that they weren’t listening to the content and rather paying attention to the character because they thought it was so cool. in two questions referencing what nicolas flamel was rumored to have created and what the philosopher’s stone could do, those that viewed both the website and app scored similarly to those that had just viewed the website, while those with only the app didn’t do as well on the knowledge questions (see table 1). table 1: usability groups a, b, and c and results of follow-up questions. overall, the demographics of our study participants mirrored that of our learner persona. most participants (63%) were between 19-29 years old and had finished high school (67%). most were also familiar with harry potter and online learning, while mostly unfamiliar with renaissance science and medicine and augmented reality (see table 2). table 2: pre-survey results on familiarity with different aspects of learning experience. future considerations based on the findings from our usability test, conversations with experts in the field, and our own experience, we recommended steps to expand the project. firstly, we would like to create more animated characters and shorten the animations to 30 seconds to stimulate viewers. secondly, adding a digital artist and professional voice actor, while costing additional funds, would raise the profile of the project and better immerse viewers.  finally, we’d like to build features using machine learning software where we’d experiment with interactive features using voice recognition and guided conversation software. conclusion this project helped highlight custom-made ar technology to enhance an online exhibit. the tool helped increased enjoyment of the exhibit and invited viewers to interact with the exhibit for longer periods of time. viewers retained the same amount of information as interacting with the website alone. if this tool were to be developed further, more studies would need to be done to understand how to increase viewers’ retention of information gathered from the interaction. many of the tools used to develop the prototype were free or inexpensive, and library personnel and partnerships proved vital to the project. references [1] sylaiou, s., mania, k., karoulis, a., & white, m. (2010). exploring the relationship between presence and enjoyment in a virtual museum. international journal of human-computer studies, 68(5), 243-253. [2] galani, a. (2003). mixed reality museum visits: using new technologies to support co-visiting for local and remote visitors. museological review, 10. [3] hall, t., ciolfi, l., bannon, l., fraser, m., benford, s., bowers, j., … & flintham, m. (2001, november). the visitor as virtual archaeologist: explorations in mixed reality technology to enhance educational and social interaction in the museum. in proceedings of the 2001 conference on virtual reality, archeology, and cultural heritage (pp. 91-96). acm. [4] damala, a., marchal, i., & houlier, p. (2007). merging augmented reality based features in mobile multimedia museum guides. in proceedings of anticipating the future of the cultural past (pp. 1–6). athens, greece. [5] chang, k. e., chang, c. t., hou, h. t., sung, y. t., chao, h. l., & lee, c. m. (2014). development and behavioral pattern analysis of a mobile guide system with augmented reality for painting appreciation instruction in an art museum. computers & education, 71, 185-197. [6] rowling, j. k. (1997). harry potter and the philosopher’s stone. london: bloomsbury pub. [7] wang, x., & chen, r. (2009). an experimental study on collaborative effectiveness of augmented reality potentials in urban design. codesign, 5(4), 229–244. about the author brandon patterson is the technology engagement librarian at the eccles health sciences library at the university of utah. he connects students, staff, and faculty to digital tools and emerging technologies and creates meaningful experiences using prototyping tools, augmented and virtual reality, and online learning platforms. he is a health sciences education liaison and coordinates with faculty to incorporate information literacy instruction and technology into their classrooms. email: b.patterson@utah.edu contributors chris allen, entertainment arts & engineering, university of utah alex johnstone, entertainment arts & engineering, university of utah subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – how we went from worst practices to good practices, and became happier in the process mission editorial committee process and structure code4lib issue 32, 2016-04-25 how we went from worst practices to good practices, and became happier in the process our application team was struggling. we had good people and the desire to create good software, but the library as an organization did not yet have experience with software development processes. work halted. team members felt unfulfilled. the once moderately competent developer felt frustrated, ashamed, helpless, and incompetent. then, miraculously, a director with experience in software project management and an experienced and talented systems administrator were hired and began to work with the team. people in the group developed a sense of teamwork that they had not experienced in their entire time at the library. now we are happy, excited, and energetic. we hope that you will appreciate our “feel-good” testimony of how excellent people and appropriate processes transformed an unhealthy work environment into a fit and happy team. by amanda french, francis kayiwa, anne lawrence, keith gilbertson and melissa lohrey introduction this paper is unusual in that we have tried to incorporate the unique personal perspectives of the members of our team. throughout the paper, thoughts and opinions of team members appear in blocks of text that address some aspect of our technology or processes. the voices included are those of a systems administrator, a director, a collections specialist, and a software developer. our focus is primarily on how we changed our workflow, and we will also give details about the technologies that support our new process. background we are the team responsible for vtechworks, the institutional repository at virginia tech, hosted by the libraries.  our team is made up of a director, a systems administrator, a metadata specialist, a content specialist, and a software developer, with additional assistance from student employees and occasionally other teams. due to the nature of library work, several of us have many other responsibilities. also, because of this workload, as well as the openness of our new system, the roles are less rigid than our titles imply; for example, we all have access to do software development work, and we all have access to change metadata. vtechworks was brought online using the open source dspace software in the fall of 2011, at a time when only one of the current team members was employed at the library. it was a project of the former digital library and archives department, as a challenge by the new dean that had started in the spring: build an institutional repository by the end of the year. although the library at virginia tech had a history of innovation in information systems, with an early etd system, the vtls ils, and imagebase, the preceding dean had chosen to concentrate efforts in areas other than technology development. therefore, the technology infrastructure wasn’t as robust as one might have expected from an organization in 2011, and some of the culture and processes to support software development weren’t yet in place, even though there were already skilled technologists and managers. the decision was made to launch vtechworks, even though it infrastructure and processes weren’t fully developed. this was a good decision at the time, as we were in need of a repository and couldn’t wait: another technology group on campus, outside of the library, had already started an institutional repository based on fedora 3 technology, but this repository failed to thrive due to problems with content recruitment. today, vtechworks has grown to over 50,000 items, with initial boosts by repackaging previously existing digital collections, such as electronic theses and dissertations, into the repository. equally important to those of us who work on the system, our technology infrastructure and work processes have also grown and improved over the years. we’ll show what’s changed and discuss our unique perspectives. systems when vtechworks was first launched, there was a single development server. due to a lack of hardware resources, the server was actually an old desktop computer that had been reclaimed from an employee that had left several years prior. there was initially no source control, but this was set up soon by a systems administrator after the developer arrived. the developer was the only person on the vtechworks team with access to read or change the source control, and the developer handled all deployments to test and production systems. when francis (our systems administrator) arrived, he made great strides in our systems that ultimately helped with our workflow. our source control was moved to github, the standard site for open source software development, based on the git version control system. all team members now have read and write access to the code. additionally, it is easier for us to collaborate with other organizations using dspace. shortly after the source code was migrated to github, we also began tracking active issues with the github issues tool built into the project. we do have some files stored in a locally hosted instance of gitlab, an open source alternative to github, due to security concerns, including a new system for quickly provisioning test instances of dspace. francis created our local development environment (lde) infrastructure so that all team members have a vtechworks development environment running on their own machines, in a virtualbox vm. the lde system is based on the vagrant +  dspace project, but uses ansible instead of puppet. virtualbox allows us to run virtual machines, which in our case emulate servers, within our standard mac systems that we were already using although some team members with older systems needed to upgrade to at least 8gb of ram.  we use vagrant in our project to configure and deploy the virtual box machine. for our local development environments, the work that vagrant performs includes provisioning a virtual machine with 4gb of ram that runs ubuntu linux, setting up a private network so that we can connect to the virtual machine from the host machine, granting access to the dspace source code folder, and running an ansible playbook. ansible, a deployment and configuration automation tool, allows us to perform further configuration on the virtual machines. first, ansible installs all of the software prerequisites for building and running dspace, including java, apache tomcat, ant, postgresql, and others. ansible then configures postgresql and initializes the dspace database. finally, ansible performs the dspace install process and starts the tomcat application server when the install is completed. the local development environment system has worked well for us. now, every team member has a place to experiment and unfettered access to look at and modify the source code. each person can do this at any time, even when the official development server is in use for testing or demonstrating new features. when we’re finished experimenting, we can easily save the state of our local development environments to disk, or delete it completely using vagrant. at the time of these changes, the old desktop system that we previously used as our development server was retired.  it is now running on server class hardware, and is configured with the same technology as the lde, so that our local development and production systems are very similar. in addition to the development server, we now have a pre-production system to use for user testing while integration work is happening on the development server. the production server was also migrated to a similar setup, though it does have some differences: for example there is a dedicated database server on a separate node. the deployments to our integration, pre-production, and production servers are now handled by systems administrators, typically at the request of amanda, who serves as the product manager.   first try transitional current development server old desktop system old desktop system; developer and student developer also created individual setups on personal computer vagrant, ansible, and virtualbox used to give similar development environments to all team members. source code no source control on private svn server, then private git server. most of the source is on github, with a few things specific to our infrastructure on a private gitlab server. issue tracking no issue tracking issues on redmine. many issues went unentered. issues were not reviewed regularly. issues stored with source code on github and gitlab, or in the backlog spreadsheet. nearly all issues and change requests recorded. issue lists reviewed regularly. documentation initially, there was no official set of documentation. notes, when available, were shared via e-mail from the systems administrator to the developer. eventually, team members began to create documentation and add it to a private redmine wiki, but this was not seen as a priority. when francis created our vagrant and ansible setup for dspace, he created detailed setup documents so that the environment could be recreated. the setup instructions were stored along with the source control project. others improved the documentation as it was used, making edits for errors or clarification. this initial documentation project served as a basis for most of our documentation efforts. any time we have a new system or a new procedure, we document it, and in most cases put the documentation in source control, either in readme files or on the project wiki. this works well for us because everyone our team has a strong urge and motivation to document anyway. each of us is detail-oriented. it’s also reassuring to know that there are others on the team that can handle work in case we need to take time off or devote ourselves to other projects. documentation has become an intentional effort now, and we are considering dedicating entire sprints to documentation improvement efforts. first try transitional current typical documentation practice shared via e-mail on a need to know basis. some documentation in redmine wiki and in source control, created as team members had time, but not as a priority. a big set of documentation shared on public github and private gitlab projects. other documents shared with all team members on google drive. team members improve documentation as they make use of it. we are discussing dedicating entire sprints to improving documentation. openness as our team has evolved, we have become more open. we’re open with each other, in that we share what we are working on and how we do things. we regularly work with others in the library to improve metadata, content, and systems. and we’ve again started to work with others in the dspace community, attending conferences, contributing small changes, and testing fixes. first try transitional current source code source code stored privately and access available only to developer. source code stored privately. write access available to developer, and to student developer after convincing the information technology department that the student needed access. most code stored in public, on github. all team members have read/write access to the source code, and contribute. the team has resumed making small contributions to dspace core. intra-team collaboration it felt like there was no person on the team – developer, manager, content specialists, systems administrators – that had the power to make positive changes to the product. developer hid from other team members. team members are productive during meetings, assist each other as needed, and enjoy the company of each other. team members feel free to share suggestions for improvements. students previously, the student developer position reported directly to the software developer. this worked well when the full-time developer was able to dedicate sufficient time to training the student, but there would be difficulty when students were hired at a time when the full-time developer was not available to train or guide them. additionally, the student developer would sometimes be given independent assignments from multiple team members, without any guidance on how to prioritize those assignments. the student developer position is now fully integrated with the team. she participates in the stand-up meetings when it fits with her work schedule. team members make sure that there is always a list of tasks for her to perform. additionally, in recognition of the value that student developers can bring to our project, our student developer wages have gone from $11.00, to $13.00, and now to $15.00 an hour within the last year. this also helps us to be competitive with other student jobs on campus. first try transitional current team dynamics with student assistant to developer work with other team members as needed.   integrated into team meetings, processes, and priorities. student developer pay unpaid intern $11.00 /hr $15.00 /hr student recruiting process friend of a friend developer lists job, collects resume, performs telephone interview job listed in multiple places. scoring matrix created for candidates. programming knowledge questions asked. entire team plus one outside member evaluates candidates. testing and fixing initially, there was no test process, formal or informal. after the first set of changes, a simple theme change, the site was displayed in a meeting of library directors for feedback and redesign. for later changes, there was no test process, other than testing performed independently by the software developer. bugs and change requests were initially handled as they came in, but this system eventually failed with over 100 open, unaddressed, and unprioritized tickets in the tracker, and a single developer, with other responsibilities, available to handle them. we’ve improved upon these practices: now, all team members have input on what they will work on, but items for a potential sprint are prioritized with the assistance of the team at the sprint planning meeting, and based upon team discussion. with all team members contributing, and a documented deployment process, we are now able to release to the production server approximately once a month. each change is reviewed by someone else on the team. we test before and after each production deployment, and have created a standard test script. first try transitional current frequency problems resolved as they are reported. test on major releases (about once per year) test branches test before all deployments (about once per month) test location test on production server test on development server test on local development environments, on at least one test server, and on production test depth serendipitous, cursory testing test a checklist of items based on previous problems and basic uses. use a written test plan based on a longer list of previous problem reports, add serendipitous testing. experimentation with selenium ide tests. people involved few people, incidental. team, plus a cataloguer and an open access specialist. student developer for minor releases.team for major releases. preparing for additional usability and user certification testing. workflows amanda french, our new director, worked to improve workflows on our team. we went from having no team workflow – essentially chaos when many problem reports and change requests came in – to having a highly organized workflow based on the scrum framework. the elements of scrum that we find particularly helpful include: product backlog – a place to document all potential upcoming changes to a project sprints – time-bounded development units where specific project tasks are accomplished planning meetings – meetings held to set accomplishable goals for each sprint task board – a visual representation of the status of each task in the sprint. ours is made of post-it notes on a whiteboard, with areas for to-do, doing, done, and waiting. stand-ups – frequent, often daily brief meetings where participants discuss what they’ve done since the last meeting, what they’ll work on now, and any obstacles many of our processes and technologies have improved gradually over time, but the best and most revolutionary changes came when two new hires came in spring of 2015 and modernized our team. one of these hires, francis kayiwa,  senior unix administrator, worked on modernizing the systems used for development and production.  his thoughts, along with those of amanda french, and anne lawrence and keith gilbertson, both of whom have been involved with the vtechworks project for several years, are offered below. francis kayiwa’s thoughts: delivery of service “i am not just a proponent of empowering everyone in the project. i would actually rather not be involved in any project that doesn’t aim to do this. my entire career has been around maintaining other people’s work. over time i have worked towards ensuring stakeholders are not only given power to work in a way that suits them, but more importantly are reminded of what part of the project they will be responsible for maintaining. the benefit here is no one person feels the burden of the project, and when done right everyone will have a good understanding of how to avoid disasters. this kind of work now falls under the auspices of the “difficult to define” devops. for this article we will describe it as the ability for developers, application administrators, and operations people to meet periodically to define the way to work most productively. my goal as part of the operations team was to be able to make our institutional repository, vtechworks, be a highly repeatable environment for all involved in the project.  i wanted all involved to be able,  with the minimal amount of documented effort, to stand up and tear down a replica of our production institutional repository. i wanted the entire group to understand that there’s nothing magical about the production server.  in order to do this we needed to define what our preferred development environment would look like. we selected vagrant and ansible tools to do this. we walked through the “dspace way” of setting up a new instance. we worked towards making sure setting up a new instance would require no human action in order to have a replica of the environment. this process was iterative and continues to be improved upon. the most important thing i brought to all of this was my ability to facilitate. it was more important for me to have everyone comfortable working with a repeatable environment than it was for me to use any particular technology.” amanda french’s thoughts: implementing scrum “i began at the virginia tech libraries as director of digital research services in late april, 2015. my director-level position includes responsibility for several areas, the two largest of which are digital humanities and the institutional repository. my background is really in digital humanities, but i’ve picked up some theory and practice of project management along the way, especially but not solely project management of software development projects. i first heard about the agile manifesto and “extreme programming” in 2008 when i was at emory libraries running the reserves service and managing the development of a locally developed reserves system there, reservesdirect. really, it was the developer there, jason white, who taught me about how to manage software projects. at his recommendation i read the classic book the mythical man-month, and i remember nodding vigorously at many of its insights. i also remember beginning to hear the terms “agile” and “iterative” used much more often in the broader culture at that time to indicate desirable qualities. at jason white’s suggestion, we adopted a vaguely agile sprint-based workflow, but honestly it wasn’t strictly necessary, since there were only the two of us involved with developing the software (and both of us had other projects and duties), which made communication and prioritization and evaluation simple. we also weren’t subject to requests for new features or even bug fixes, since the production system was relatively stable and users were used to it. i have, however, spent a lot of time around software developers in the last ten years, first at ncsu in the digital library initiative group, then at emory, then at the roy rosenzweig center for history and new media, and what has become increasingly clear just from listening is that there are an infinite number of possible features and potential bugs in any piece of software, it isn’t possible for most users to judge how much time and effort developing a feature or fixing a bug will require, and it’s often not much easier for a developer to judge how much time and effort developing a feature or fixing a bug will require, either. coming from an english phd program, too, i couldn’t help but liken the difficulties with writing code quickly to the difficulties of writing anything quickly; there are an infinite number of topics you could write about about, an infinite number of books and articles you could read to help you write it, and it’s very difficult to judge how long it will take to answer a research question you pose yourself or how long it will take to produce a passable piece of prose. as a teacher of writing, i had a great deal of success asking my students to write one-page weekly papers instead of one or two longer papers each semester; the papers were better, the students became more used to writing and got much more feedback from me and from each other, and leaving writing until the last minute became at once more difficult and less disastrous. that assignment might have been called “an iterative process” if it were in the tech sphere. similarly, when i learned about the unconference model and later headed up the unconference thatcamp (the humanities and technology camp), i appreciated the innate agility of it: by waiting until the very morning of the event to decide what to do, and to make the decision together, participants would be able to address issues that were on their mind that very day instead of having to read aloud a paper that they had first thought about several months before. therefore, when i came to virginia tech, where i was for the first time in charge of more than one person at a time doing software development, and where (as i soon learned) there was a lengthy backlog of feature requests to address, it seemed to me that i should learn a lot more about methods based on agile principles. i read scrum: a breathtakingly brief and agile introduction by chris sims; i read the elements of scrum by chris sims and hillary louise johnson; and i perused the slides from a  2011 digital library forum presentation by naomi dushay (on behalf of bess sadler), tom cramer, and jessie keck –  agile methodology, innovation, and quality. because i was lucky enough to come in at the same time as a new unix systems administrator, francis kayiwa, who wanted to overhaul the whole technical backend of the boxes our systems resided on, we did that first, which already meant that we would have new github-based workflows. once we were through upgrading dspace, getting set up on github, and getting used to developing on local virtual machines, the time seemed right to introduce a new way of working. it didn’t hurt that just at the same time we were due to move into new offices, as well: why not get all the changes over with at once? before i decided to adopt scrum officially, however, i spoke with a few people beforehand about potential problems, most usefully the head of virginia tech university libraries’ it services, john borwick, who had participated in scrum processes at wake forest. there were three major concerns i had: scrum is really designed for teams who are developing new software, not for teams who are supporting production it systems. i wasn’t sure whether scrum processes would work in that context. the vtechworks team doesn’t only work on the dspace software; we identify and acquire content and check metadata quality and respond to user requests as well. some reports averred that scrum worked well for other work besides software development, but i wasn’t sure it would. much of the seemingly endless debate around scrum online concerned whether it was okay to depart from the established roles and meetings and structures; many people claimed that if scrum failed, it was because that team didn’t do it correctly. i knew, in particular, that my own role mapped most closely onto product owner, but that we didn’t have a scrum master. john’s opinion on these three points was that scrum would probably work for production it (although he’s more a fan of itil, about which he pointed me to some fresh reading), that it would probably work for other types of work as well, but that it was crucial to have a scrum master if at all possible. he also advised that the product owner role might best be filled by a user. ultimately, i decided to forge ahead, and to take the risk of doing without a scrum master. i put together a brief presentation for the four-person team about scrum, identified a place for a physical task board, created a product backlog in a google doc spreadsheet from the 100+ issues in the old issue tracker, and put a bunch of meetings on the calendar: “sprint planning,” “sprint showcase,” “sprint retrospective,” and of course the “daily stand-up,” which in our case is at noon. together we decided on a month-long sprint cycle (sort of: more on that below), and we began sprinting in september 2015. at keith gilbertson’s suggestion, our sprints and sprint releases are named after animals in alphabetical order (aardvark, bichon frise, chipmunk, dogue de bordeaux, emu, frog … ); we plan our work at the beginning of the calendar month and usually (though not always) deploy changes to production at the end of the sprint and calendar month. figure 1. taskboard. i’m definitely happy with the way it has turned out; we’re making significant progress on the backlog of issues, and i think that scrum does work well even in the context of a production it system and for non-technology work such as correcting metadata or acquiring content. i also think that while it would be nice to have a scrum master, someone who would make sure that i’m not asking too much of people in my role as product owner, we’re managing pretty well without one. to make up for the lack, i try to give team members a choice of what to work on and i try to check in often to make sure they’re not overwhelmed. nothing is perfect, of course, and there are a few new problems that have been introduced. the first problem is that running vtechworks is only part of my job, and the scrum process is so powerful (those daily meetings!) that it tends to concentrate my efforts on vtechworks at the expense of other things i should be doing, things that are not part of the scrum process. it’s still not entirely clear if we have a three-week sprint or a four-week sprint: i tried to set it up so that we had a three-week sprint and then a week off so that i could focus on other things and the staff could too (learn new things, write things, clean up old tasks, revise documentation, take a break from each other), but in practice it doesn’t seem to work that way, and we wind up with four-week sprints in which the last week is devoted to testing and bug fixing and code review and deployment and, yes, a few things we didn’t get done. the second problem is that while scrum does function pretty well for non-technology work, there is a sort of emphasis on and privileging of the technology work that is hard to overcome. i’m not sure whether that’s innate to the scrum process or is just a problem with my own prioritizing: some month or months soon, i’d like to plan almost the whole sprint around content acquisition, metadata correction, or learning new skills, or writing and revising and migrating and archiving documentation. but meanwhile, we’re still very focused on learning dspace and recreating or fulfilling old feature requests. the third problem, and perhaps the least important, is that we’re still not very good at judging how long it will take to accomplish particular tasks, nor at how much we’ll be able to get done in the course of a month. we decided not to bother with burndown charts and other tools designed to help us estimate such things, because while we do have a lot of feature requests and projects, so far we haven’t had a firm deadline for any particular feature or project that we’ve been unable to meet. what is clear is that we’re getting a lot done, and even when we don’t get everything done that we thought we might, well, next month is a brand-new month, filled with possibility, and we’ve gotten a lot done in the past month, which is satisfying. and even sometimes fun.”   anne lawrence’s thoughts: the scrum experience “i have degrees in industrial engineering and library science, with an interest in efficiency and the continual improvement of processes and systems. i appreciate that i can contribute to the improvement of our repository in a variety of ways: improving the software, improving workflows, administration, ingesting content, etc. i have practiced getting things done, somewhat loosely, for about 8 years and believe strongly in the utility of getting things out of my head and into an external system. our system allows this principle to extend to our work team. this system not only allows me to free up my brain of things to remember, it facilitates communication of my information and detail to our team. similarly, i am informed of my co-workers’ and supervisor’s projects and priorities. everyone’s ideas and suggestions are welcome; they may not all be implemented but will be considered. we are encouraged to take on projects that are a personal stretch and allow us to build skills and knowledge. our git/vagrant system creates some overhead to working, but provides a good safety net for developing and experimenting. i thought the number of meetings would be overwhelming but it’s ok. i greatly appreciate that meetings are generally held to their allotted time (generally 15 minutes or 1 hour). clear communication is required to do complicated or more involved tasks. high level wishes have to be translated into coding details. github, daily stand-ups, and the planning meetings go a long way to meeting that need. three of us are in the same physical space, which is a help. but some members telework, which can inhibit communication. so far, that has not been a big problem. they communicate via email, video, and the slack-like tool mattermost. we seem to have a good level of communication about issues. our director acts as product manager and scrum leader. it might be better with a larger group, but it works and allows each person time to report on their work and express their ideas. i would prefer if all our git work were in gitlab (limited to the libraries staff) rather than in the public github. we use a large percentage of the functionality in github for code development, the wiki, and issue tracking. github has very nice features which make it easy to add screenshots, create pull requests, relate issues and pull requests, etc. we used redmine previously for issue tracking and to create a wiki. however github/lab integrates better into our code development work, particularly since we have a fork of the dspace github repository. the backlog is well implemented as a spreadsheet on google drive, which makes it easy to edit, while ensuring that it is backed up. separate sheets for the customer help form, content, technology, metadata, and documentation projects aid us in focusing on each category at a time during product backlog meetings. team members are free to add issues to it, recording requests from users and their own ideas or tasks. as i moved a sizable portion of my personal to-do list to the backlog, i appreciated that the list of tasks i had carried personally would now be shared and prioritized as a group. the product backlog allows me to list items without bothering anyone in the moment, knowing that they will be considered in the product backlog meeting. i like the daily stand-up for several reasons: it’s short — i can do anything for 15 minutes! i prefer to stand and stretch after sitting so much. tasks are much easier and much less uncertain if i can check in with others, especially my supervisor, as questions and concerns arise. since we all prefer to work in uninterrupted stretches, i am sometimes hesitant to disturb co-workers, even to negotiate a time to meet. and tracking down people can be even more difficult. rather than not communicate or delay communication too long, it’s much easier and more efficient to plan it into the day, as stand-up meetings. i can save up most issues for the meetings, without having to disturb co-workers at other times. i give the group a summary of my work and bring up issues that would benefit from communication with them. as needed, i provide additional documentation and details in github issues, email, or google docs. and i truly appreciate hearing from my co-workers about their work and news at this meeting. because they are so useful, we have stand ups during our “off sprint” week. our manager tends to concentrate on her other responsibilities during that week, but since we stay on vtechworks during that week, communication is still needed. the monthly policy and procedure meeting isn’t part of the scrum, but it allows us time to improve our procedures and discuss policies and other issues in depth. i find it useful to have this time away from strict task completion in order to handle administrative issues of the repository. so, in short, i am very pleased with our current system and team climate. i feel it fosters communication, a team spirit, and improved results.” keith gilbertson’s thoughts: redemption “for most of the prior three years at virginia tech, i had been the bad egg and the bottleneck on the vtechworks team. i started my career working primarily as a software developer. prior to libraries, my jobs had been in either very small, very focused organizations, where communication was simple, or in very large organizations with well-defined software development processes. my work from 2007-2009 on institutional repositories at ohiolink (the ohio libraries and information network)  served as a bridge into software development for academic libraries. at the time, a significant amount of software development, configuration, deployment, and maintenance happened at ohiolink. the organization was small and flexible, but very productive. for example, when i left ohiolink, a team of one developer, one manager, and a shared systems administrator were providing installation, customization, support, and upgrades for approximately a dozen institutional repositories in the state, with once-a-month guidance from a team of librarians serving at ohio universities. when i began working in academic libraries proper, i experienced somewhat of a culture shock. most libraries aren’t sized as enormous corporations, but they can be ambitious, and often have multiple priorities that compete for staff time. also, while i don’t know if i could say that the same is true today, i will risk stating that 10 years ago, many libraries, even when staffed by talented technologists, had organizational cultures that would have made completion of software development projects difficult. when i arrived at virginia tech, it was clear that there were extremely competent managers with a history of success, and that the organizational competencies in software development didn’t yet match the technology aspirations that came with a new dean. i failed in that environment. i was overwhelmed with all of the responsibilities and challenges. i felt incompetent because i wasn’t able to complete everything that everyone wanted from me. now, things are much better. because of the changes that we have made in our technology and processes, everyone on the team participates in improving vtechworks. i’m not overwhelmed, and i feel confident again that i can help others who will also help me. i’m also available to work on other projects that will benefit the library. the most astonishing thing for me is that i have learned that i can be a “people person”. i used to seek time alone because i feared that every interaction with people would create demands that i wouldn’t be able to fulfill; now i’m a team player and enjoy working with my colleagues again, so much so that when i must miss a meeting, i feel disappointment instead of relief. i’m happy to be here.” conclusion the primary lesson learned from our experience is that having a process for our team, much like we have processes for our own individual work, is helpful. knowing which items we are going to concentrate on at any given time helps us to follow through with our own individual work processes, because we’re less likely to be distracted by incoming work that might seem urgent because it has landed in our email inboxes. instead of distracting us, new feature requests go into our backlog spreadsheet, and newly reported bugs – unless they are showstopping – are placed into our github ticketing system. the most important requests will be selected for attention at our next sprint planning meeting. the process is not perfect. for example, some group members have expressed concern that we under-emphasize task prioritization. members are assigned specific tasks each week, and each task is, in theory, equally important. in reality, this is often not the case and some projects are clearly more time-sensitive or essential than others. due to the challenges of correct time and effort estimation, often not all of the work selected for a sprint can be finished. therefore, we recommend that groups trying scrum also assign priorities to tasks within the sprint planning meeting. and we do recommend that groups try scrum, and adapt it to their own needs. in particular, we recommend documenting all change requests in a product backlog. we recommend sprints, and holding official sprint planning meetings to determine, based on the product backlog, the goals of each sprint. we recommend selecting a primary person of responsibility for each task during the sprint, and posting this information on a task board, whether this is done with post-it notes or virtually. finally, even those of us in our group who initially disliked meetings recommend holding brief, regular stand-ups where each group member quickly explains work done since the last meeting, what will be worked on next, and any obstacles faced. this gives group members an opportunity to share victories, and to ask for help when it is needed. in our case, this regular, structured contact where we report on our successes and help each other achieve goals has allowed some members to have a stronger sense of being a part of a team than they have ever felt before. resources allen d. 2015. getting things done: the art of stress free productivity. 2015 ed. penguin books. brooks fp. 1995. the mythical man month. anniversary ed. addison-wesley professional. dushay n, sadler b, cramer t, keck j. 2011. agile methodology, innovation, and quality. presented at 2011 digital library federation meeting; 2011 november 1; baltimore. slides available at: http://www.slideshare.net/ndushay/agile-for-digital-library-projects. selenium ide. [internet]. seleniumhq; [cited 2016-3-20]. available from: http://www.seleniumhq.org/projects/ide/. sims c, johnson hl. 2012. [internet]. scrum: a breathtakingly brief and agile introduction. available from http://www.agilelearninglabs.com/resources/scrum-introduction/. sims c, johnson hl. 2011. the elements of scrum. dymaxicon. pottinger h, donohue t, prado r. vagrant + dspace. 2016. [internet]. available from: https://github.com/dspace/vagrant-dspace/blob/master/readme.md vtechworks wiki. 2016. [internet]. available from: https://github.com/vtul/vtechworks/wiki. about the authors former clir postdoctoral fellow amanda french is currently director of digital research services and associate professor at virginia tech university libraries, where she is helping to build digital humanities infrastructure and is running the institutional repository, vtechworks. from 2010-2014 she was thatcamp coordinator and research assistant professor at the roy rosenzweig center for history and new media at george mason university. her particular expertise consists of making humanities content (both cultural content and scholarly interpretation of that content) openly available online, as well as introducing scholars to the various methods of and issues with making humanities content openly available online. francis kayiwa is a unix/linux systems administrator with over 15 years of experience, and increasingly serves as a software developer and project manager with expertise in perl and python, and exposure to ruby. his current interest is devops automation of server infrastructure using ansible and salt, while exploring chef/puppet. anne lawrence  is a repository collections specialist at virginia tech. keith gilbertson is a digital technologies development librarian at virginia tech. melissa lohrey is a repository collections specialist at the virginia tech libraries. she assists users from virginia tech (and around the world) to deposit and obtain access to scholarly and cultural heritage resources in vtechworks, virginia tech’s institutional repository. she received her ba in psychology from the university of maryland, baltimore county and her mls from the university of maryland, college park. before arriving at virginia tech, melissa worked as a student assistant in the university of maryland’s special collections department and as a metadata specialist at the national agricultural library.   subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – reaching users through facebook: a guide to implementing facebook athenaeum mission editorial committee process and structure code4lib issue 5, 2008-12-15 reaching users through facebook: a guide to implementing facebook athenaeum facebook athenaeum is an open source application that integrates library resources directly into the facebook website. facebook is one of the single most-visited websites in the world, and its popularity among college-aged students provides a unique opportunity for libraries to redefine how they interact with students. this article walks you through the deployment of facebook athenaeum, and discusses some of the usage trends and pitfalls of deploying applications using the facebook api. by wayne graham introduction in 2007, facebook was positioning itself to overtake myspace as the number one social website. until this point, facebook had only been available to higher-education institutions, but they were in the process of opening the site up to all users and rolling out an api to allow developers to create “useful” applications. seeing an opportunity to provide our own “useful” social application, we set out to integrate an existing tool developed by tom macwright, a student at william and mary, with our library’s rss feeds and catalog/website search. macwright developed a nifty little application that allows users to click on a map of the library and generate an im status link (or anything else that accepted a hyperlink) so your friends could see where you were located  (http://swem.wm.edu/services/swemsignal/). swem signal was used by quite a few students, and we even got the code donated to the library to run on our servers. after the facebook api launched, we thought swem signal would be a really cool social feature to integrate into a library facebook application. the real impetus, however, was to expose our search tools to users who may spend more time socializing than studying. figure 1: facebook athaneum main page [view larger] how do facebook applications work? facebook applications are a little different from other online applications in that they live on two different servers. you host the application on your own server, which is responsible for the business logic of your application, and facebook’s servers work with the user’s profile. when a user interacts with your application, facebook interprets the request and passes it to your server. your pages are constructed with fbml (facebook markup language), a superset of html, to pass back to the facebook server farm. facebook interprets the fbml response from your server and generates the resulting page the user sees. figure 2: facebook page processing server set up so, what do you need to get going? facebook athenaeum uses the php facebook client libraries for interacting with facebook profiles; a database server (for storing cartesian coordinates); smarty for creating the display; and the pear db modules [1] and xml_rss packages for interacting with the database backend and reading any rss feeds you may have. floor maps of your building are really handy and they can be in whatever image format is most convenient for you. the official libraries provided by facebook are in php and java, but there are unofficial libraries for most popular web and application programming languages, so if you are using a different web platform, you can very quickly migrate the code to your local environment. you can find links to all the client libraries on the facebook developer wiki. additionally, there are convenient, unofficial, facebook library implementations in pear (pear install services_facebook) and rubyforge (gem install rfacebook). i have made some assumptions in these instructions, first that you’re running a *nix system, that you have permissions to install software on the server, and you have php, pear, and wget installed. this is not to say you cannot run this on the windows server, but you will need to manually install the dependencies. i will do my best to note the software you will need and where to get further instructions when needed. setting up smarty and pear modules to make things go a little easier, we wrote a bash script for *nix users that will download smarty, extract it, and put it in the php include path and then update pear and install the db and xml_rss packages with their required dependencies. this script is included in the facebook athenaeum source code. #! /bin/bash phpdir=`pear config-get php_dir` smarty_ver="smarty-2.6.20" smartydir="$phpdir/smarty" if [ ! -d $smartydir ] then echo "downloading smarty..." wget http://www.smarty.net/do_download.php?download_file=$smarty_ver.tar.gz echo "extracting smarty" tar -zxf $smarty_ver.tar.gz echo "moving smarty to $smartydir" mkdir $smartydir mv $smarty_ver/libs/* $smartydir echo "cleaning up..." rm $smarty_ver.tar.gz rm -rf $smarty_ver fi ## install pear packages echo "updating pear packages" pear upgrade pear pear install --onlyreqdeps db pear install --onlyreqdeps xml_rss this script was updated in november of 2008. you may need to check to make sure you are getting the latest version by visiting the smarty website (http://www.smarty.net/download.php). if the version changes, just change the smarty_ver line to the correct version and then run the script. note: if you are running windows, you will need to manually install smarty and the pear modules. to install smarty, refer to http://news.php.net/php.smarty.dev/2703 for detailed instructions. assuming you have correctly installed pear on windows (there is a convenient msi installer), you will also need to run the pear lines from the above script (e.g. pear install –onlyreqdeps db xml_rss). get the source code now that you have retrieved the dependencies, it is just a matter of setting up your web environment. you can do this by simply setting up a folder in your web directory (e.g. /var/www/facebook) or creating an alias in your apache configuration. download the files from the google code site (http://code.google.com/p/facebook-athenaeum/downloads/list) and unpack them into the location you want the files. if you are comfortable with working with the development code, you can just check out the trunk. svn checkout http://facebook-athenaeum.googlecode.com/svn/trunk/ facebook-athenaeum-read-only note: if you’re on a *nix web server, you will need to change the permissions on the cache and compile directories to give write access to the web server. chown nobody:nobody /path/to/facebook-athenaeum/cache chown nobody:nobody /path/to/facebook-athenaeum/compile chmod 775 /path/to/facebook-athenaeum/cache chmod 775 /path/to/facebook-athenaeum/cache mysql setup facebook athenaeum includes scripts for creating a mysql database, but since the code uses the pear db library it should work with any database system that pear db supports. you will just need to write the sql to create the locations table. to aid with this, i wrote a basic sql-99 compliant sql script for those wanting to use an alternative relational database management system. create table locations( uid int not null, x int, y int, floor int, updated datetime not null, primary key(uid) ) as you can see, there is not much to the table: it just stores some very basic information about what image was used and the cartesian coordinates set by the user. we did write a helper script for those running mysql on *nix (mysql_helper.sh) to make this as easy as possible. you can also just create the database and table through whatever convenient tool you have available. whatever method you decide to use, just be sure to note your settings for the configuration file. note: if you do write a table definition for an alternative database system, be sure to make a patch and submit it to the project (http://code.google.com/p/facebook-athenaeum/issues/list). facebook setup now that the necessary server libraries and source code are in place, you need to tell facebook about your application. this, unfortunately, is a multi-step process. you first need to add the facebook developer application to your account (accessible at http://www.facebook.com/developers/). once installed, click on the “+set up new application” button to start the process. figure 3: set up new application detail you will be prompted to give your application a name. just remember, you can’t have the word “face” anywhere in the name (not even something like “surface”). after you read the terms for the facebook platform (http://www.facebook.com/developers/tos.php), expand the optional fields. figure 4: facebook optional fields detail [view larger] at this point, you just need to set the callback url (the url of the application you have set up on your server, e.g. http://library.myuniversity.edu/facebook) and the canvas page url (the facebook url for the application, e.g. yourlibraryathenaeum). if you make a mistake or change your mind, you can always go in and change these fields later from the develop application in facebook. once you have finished the wizard, you will be directed to the application manager for the facebook application you just created. on this screen, you will need to note the api key and secret for the configuration file on your server. application setup with all the preliminaries out of the way, it is time to finish setting up your instance of facebook athenaeum. in the configurations folder on your web server, there is a file named config.inc.php that you need to edit. just add your facebook api key, secret, callback url and canvas url. next, edit the dsn settings for your host. you will need to provide the host, user, password, and database for pear db to make the connection. again, this should work for database systems other than mysql, but you will need to reference the pear db (http://pear.php.net/package/db/docs/latest/db/db.html) documentation to make sure the required values are passed to the constructor. the next section lets you set up the application customizations. you can name the application whatever you would like, and if you have an rss feed, you can add that into the feed_url variable. in our configuration, we use this to show the current news feed for our library. the app_dir is also pretty straightforward–it is just the absolute path to where your application lives on the server. now comes the part that can give some headaches. we use a google mini appliance at our library. the google mini is a great appliance and has the ability to produce search results in xml format. [2] this allows us to use a facebook ajax call to pull the search results into the facebook application rather than kick them out to another server. for our database and catalog searches, we actually use a query evaluator to funnel the search into the correct server. you may find yourself needing to refactor this code a bit, but it should be a pretty quick process. and to answer your question, yes, we do plan to make this easier in the future. the rest of the configuration file should be relatively self-explanatory, but i will key you in on one last item. facebook athenaeum also integrates with google analytics (http://analytics.google.com) by using the google_analytics_key. the statistics facebook offers are pretty light and google analytics will give you “better” usage statistics.  you just need the key, which looks like “ua-nnnnnnn-n”. you cannot directly copy from the analytics page as google wants you to copy the entire code block. you can either write the key down, or paste the contents of the code block in a text editor and then copy just the key. the map fortunately for us, we had a set of floor plans from our recent building renovation. figure 5: facebook page with friend locator, including floor plan [view larger] if you’re lucky, you too already have a set of nice floor plans to use for your facebook application. if not, this becomes a slightly bigger project, but basically you need to generate an image format that is easily read by browsers (e.g. gif, png, jpeg, etc.). for ease-of-coding/lazy reasons, we named the images 0-3 of our different floors so they will map easily map to the array. each floor gets a label and a message. the message is just the text that gets displayed when the user wants the application on their profile page. if all went well, everything is configured now and ready for testing. to add the application to your profile, visit your application’s canvas page (e.g. http://apps.facebook.com/your_app_name). you will be prompted to give several permissions to the application from your profile. once you click on “allow,” you can start using the application. publishing when you have worked out any issues you may encounter and have finished adding any additional functionality you may need (and please post any patches to the project page at http://code.google.com/p/facebook-athenaeum/issues/list), you are ready to publish the application in facebook and start letting “friends” know how to use it. publishing an application in facebook’s application directory (http://www.facebook.com/apps/), requires five people to first add the application to their profile. hopefully you have some facebook friends who will help you out. to actually submit the application for listings approval, log on to facebook and go to developer application (should be on the right-hand side of the screen). go into your application, where you will find a tiny link under directory status to submit it to the product directory. figure 6: submit to facebook screen detail finish filling out the form and click save. it can take some time (usually two to three business days) for the approval process to go through because someone actually looks at every application before it gets listed. once added to the directory, facebook users will then be able to discover your application through the search interface. final housekeeping you also will want to develop the content on your application’s about page. this is the page that people can use to become “fans” and allow you to interact with your users (at least those who choose to interact on the page). you can get to the about page from the developer application. just go into your application and click on edit about page. figure 7: facebook atheneum about page [view larger] edit the appropriate sections you want to have appear. this is a convenient page to subsequently interact with your users (or at least the ones that become fans of the application). getting help there are two google groups set up in case you run into a problem with facebook athenaeum. the groups are listed off the main project page (http://code.google.com/p/facebook-athenaeum/). if you need some help in tweaking the application, you can also consult the facebook developer wiki (http://wiki.developers.facebook.com/index.php/main_page). conclusion road map what does the future hold for facebook athenaeum? well, one of the features we want to work on is integrating with vufind (http://www.vufind.org) to pull search results from vufind into the facebook application. as vufind grows to enable libraries to index and search more diverse content types (it currently supports indexing marc content), all of this will also be directly available through facebook. the other big item on the roadmap is to migrate the database storage from a relational database system to facebook’s data store api. this will allow you to keep most of the user interactions directly on facebook’s cloud, decreasing the number of resources this application consumes. usage swem library has  been running its swemtools application for over a year now. we currently have 481 users who have installed the application, with about 40 monthly users (a little less than 10% usage).  doing a little poking around, there seem to be a large percentage of facebook applications that hover around this threshold. we realized when evaluating our application usage that we experienced a phenomenon where there are a lot of initial additions immediately after publicity of the application, and then a sharp drop off in usage. when we set out to develop this project, i will candidly say we did not exactly know what to expect–we were really just testing out this new platform and wrote what we thought would be cool and tried to make it useful. as we progressed, we started to think more about what criteria we might use to evaluate the effectiveness of this project. we realized we are really trying to reach an audience that is more comfortable interacting on facebook than the online catalog; a group that perhaps is not always effectively reached by traditional library outreach. what we have found is that there has been sustained, long-term usage of the application, with little if any prodding from us. the fact that these users are coming and using the application shows that there is at least a perceived usefulness in the application from users. so, should your library have a presence in facebook? i am perhaps biased, but i think the site provides a unique opportunity for libraries to redefine how they interact with students and how libaries can facilitate the interaction between students. i’ll be quite honest though, when the facebook site first launched and i created my account, i really did not see what the point was, and i know i am not the only one who had this reaction. what brought me around was seeing just how many students actually use the site on a daily basis. being able to interact with these students on a platform they are comfortable with seemed like a natural extension of what the library has traditionally done in developing its web content and outreach activities. we have further found that when we advertise an event on facebook, we get far more participation than we do through posters, news feeds, and other outlets. additional resources dia (http://projects.gnome.org/dia/) facebook athenaeum (http://code.google.com/p/facebook-athenaeum) facebook client libraries (http://wiki.developers.facebook.com/index.php/client_libraries) google analytics (http://analytics.google.com) gimp (http://www.gimp.org/) pear db (http://pear.php.net/package/db) pear xml_rss (http://pear.php.net/package/xml_rss) mysql (http://www.mysql.com/) smarty (http://www.smarty.net/) subversion (http://subversion.tigris.org/) notes [1] the pear db package has been deprecated. instead of migrating the code to the mdb2 library, we plan to move the backend to use the facebook data api. [2] for a more in-depth analysis of leveraging the google mini, see: edwin burgess and edward metz, “applying google mini search appliance for document discoverability.” online 32, no. 4 (july 2008): 25-27. (coins) about the author wayne graham is the coordinator of emerging technology at the earl gregg swem library at the college of william and mary. he is the author of facebook api developer’s guide (apress, 2008) and contributes code to the vufind and solrmarc projects. wayne occasionally blogs at http://www.liquidfoot.com, and you can always shoot him a note on facebook. subscribe to comments: for this article | for all articles 5 responses to "reaching users through facebook: a guide to implementing facebook athenaeum" please leave a response below: sheila bryant, 2008-12-18 as an academic librarian in a community college, i do not understand the academic use of facebook. how is facebook applicable to the academic education, informatiion literacy, and critical thinking of students? is there some literature or a work in progress that would detail the academic accomplishments? anna, 2008-12-18 sheila, the way students in today’s institutions of higher learning receive information and communicate with one another is changing every day. collaborative networks, such as facebook, provides us with another avenue to reach our patrons. nancy adams, 2009-10-12 in one respect, facebook is applicable to information literacy and critical thinking of students the same way that e-mail is: it isn’t, except it’s how you get the message about information literacy and critical thinking to the students. in another respect, because facebook allows apps such as the ones described in this article, it allows students to customize their own portals to library content. and that is something we want to encourage. jayasree, 2009-11-04 sir, the article is very much informative. i am doing my research on “role of social networking in the services of university libraries”.kindly send me some information or articles related to this topic. thanking you jayasree s ??????????? ??? ????????? ??????: ????? ??? ????????? « greeklis, 2012-11-08 […] wayne. “reaching users through facebook: a guide to implementing facebook athenaeum“. the code4lib journal 5 […] leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – extending and adapting metadata audit tools for mountain west digital library members mission editorial committee process and structure code4lib issue 41, 2018-08-09 extending and adapting metadata audit tools for mountain west digital library members as a dpla regional service hub, mountain west digital library harvests metadata from 16 member repositories representing over 70 partners throughout the western us and hosts over 950,000 records in its portal. the collections harvested range in size from a handful of records to many thousands, presenting both quality control and efficiency issues. to assist members in auditing records for metadata required by the mwdl metadata application profile before harvesting, mwdl hosts a metadata auditing tool adapted from north carolina digital heritage center’s original dpla oai aggregation tools project, available on github. the tool uses xsl tests of the oai-pmh stream from a repository to check conformance of incoming data with the mwdl metadata application profile. use of the tool enables student workers and non-professionals to perform large-scale metadata auditing even if they have no prior knowledge of application profiles or metadata auditing workflows. in the spring of 2018, we further adapted and extended this tool to audit collections coming from a new member, oregon digital. the oai-pmh provision from oregon digital’s samvera repository is configured differently than that of the contentdm repositories used by existing mwdl members, requiring adaptation of the tool. we also extended the tool by adding the dublin core facet viewer, which gives the ability to view and analyze values used in both required and recommended fields by frequency. use of this tool enhances metadata completeness, correctness, and consistency. this article will discuss the technical challenges of project, offer code samples, and offer ideas for further updates. by teresa k. hebron background originally founded in 2001 as a collaboration by members of the utah academic library consortium (ualc), mountain west digital library (mwdl) has grown to become a digital public library of america (dpla) regional service hub, harvesting digital collections metadata from 16 member repositories representing more than 70 partners in 6 western us states and hosting over 950,000 records in its portal. mwdl was one of dpla’s six inaugural service hubs and, at the time of dpla’s launch in 2013, the largest contributor of records. an in-depth history of mwdl can be found in neatrour, et al., 2016. this article will discuss my adaptation of an existing metadata auditing tool provided by mwdl for its staff and members to extend the tool’s capabilities and offer ideas for future enhancements. focusing on mwdl’s open archives initiative-protocol for metadata harvesting-enabled (oai-pmh) harvesting of digital collections and regional aggregation, this article is a case study for adapting similar tools for metadata auditing against different metadata profiles and standards. mwdl audits incoming metadata to ensure conformance with the mwdl metadata application profile (map). the map currently requires 8 dublin core fields: date, description, format, identifier, rights, subject, title, and type. two further fields are mandatory-if-applicable: conversion specs and creator. language and geospatial fields are recommended, but not required (mountain west digital library metadata taskforce 2011). while the map provides instructions how to apply these metadata standards at the local level, the challenge of checking them for completeness and conformance before harvest at the regional level remains. since the majority of mwdl records are in turn harvested by dpla, this warrants double-checking the records at the point of regional aggregation. dpla periodically audits ingests for missing or non-conforming required fields, which could necessitate a large-scale metadata remediation project before harvesting is possible. these collections range widely in size from a single record to tens of thousands in a single collection. mwdl has a very small staff, consisting of two full-time librarians and several part-time student workers. to extend the ability of the staff to audit collections rapidly and accurately, mwdl launched an adaptation of the north carolina digital heritage center’s dpla oai aggregation tools project available on github in late 2015 (mcintyre 2015). the ncdhc’s original version of this web-based tool offered three features to aid in metadata auditing by dpla hubs: 1) displaying data mapped to simple dublin core fields (mapping checker), 2) displaying the frequency of terms in each of the simple dublin core fields (facet viewer), and 3) displaying records that are missing “required” fields (required data checker). ncdhc staff developed the tool “… to quickly assess (1) the presence of required, recommended and optional fields and (2) the baseline quality of different metadata collections” (gregory and williams 2014). to use each of the features, the user inputs the oai-pmh base url of a repository, chooses a set to test, and the tool issues an oai listrecords command and draws a table with the results. to demonstrate how useful this tool can be, consider the previous auditing workflow. using oai, the metadata librarian can examine the records in a particular set via a browser, checking them visually for conformance with the map. for example, the oai base url for utah state archives’ contentdm repository is: http://images.archives.utah.gov/oai/oai.php oai-pmh uses a standard set of requests, called verbs, to request and provide data in a uniform way. a user can see the group of collections (or sets) available by using the listsets verb: http://images.archives.utah.gov/oai/oai.php?verb=listsets to see the records in a specific set, the user can use the listrecords command (tooele county (utah) county clerk register of deaths): http://images.archives.utah.gov/oai/oai.php?verb=listrecords&metadataprefix=oai_qdc&set=p17010coll40 this collection only contains a single record, making it easy to check. but imagine checking a collection with ten thousand records! the aggregation tool issues the listsets command, allows the user to choose a set from a drop-down, then issues the listrecords command behind the scenes and provides a visual presentation of the entire set for auditing. while this article will not provide an in-depth discussion of oai-pmh, readers unfamiliar with the protocol may consult mwdl’s public overview of oai with examples[1] or other free oai-pmh validators and test tools available on the web, such as the open archives initiative repository explorer (aka “the green tool”)[2] or the oai-pmh validator and extractor tool[3]. use of this tool provides several advantages over traditional metadata auditing workflows. first, as a tool publicly available on the web, it is accessible to mwdl members throughout the collection description process for iterative audits or reviews before they submit a collection for harvest. in using the oai feed from a repository, the tool provides a “what you see is what you get” window into how collections metadata will be used by the mwdl portal aside from a few metadata transforms and enrichments (for example, adding institution or collection-identifying information fields). further, non-professional staff can be easily trained to use the tool without needing in-depth knowledge about oai syntax or learning tools like oxygen, openrefine or r to perform audits. introduction of the tool shifted the focus of mwdl metadata work from metadata auditing based on a small sampling of records in a particular collection’s oai feed to precise, comprehensive auditing able to check all records in a collection before harvest. the original mwdl adaptation of the tool offered the ability to display the mappings of all fields in a set and test a set for required fields in both simple dublin core (oai_dc) and qualified dublin core (oai_qdc) as provided by contentdm. it did not provide the facet viewer that displays frequency of terms. the homepage of the first adaptation is shown in figure 1: figure 1. first mwdl adaptation of metadata tool, launched in 2015 since the original implementation of the tool, the profile of repositories used by mwdl members has shifted. the addition of oregon digital on samvera (formerly hydra), the move of one member to islandora, and the launch of university of utah’s solphal repository (a homegrown solution built on lucene solr, phalcon and nginx) necessitated updating and extending the tool to work with oai-pmh feeds configured differently from those using contentdm. further, the entire staff of mwdl has shifted since the tool was initially launched, leaving a large gap in institutional knowledge about original design decisions. when i began my position as the digital metadata librarian at mwdl in january 2018, i saw this as a natural time to assess and evaluate existing tools and workflows. needs assessment the foremost driver for adapting the audit tool at this time was the ingest of oregon digital’s collections into mwdl. technical difficulties with harvesting and displaying data in the mwdl portal (using exlibris primo) and lack of mwdl staffing contributed to a long onboarding process for oregon digital, but by early 2018 these hurdles had been resolved and we were ready to do a large-scale harvest of collections. this necessitated auditing the collections in a more thorough, comprehensive way than had been previously done. depending on the oai provision of the source repository, other institutions hoping to adapt this tool will want to undertake similar steps as i did when adapting the tool for samvera. comparing the view of records in the oai feed from contentdm against samvera makes it apparent why adaptation was necessary. the following is an example of a single record from weber state university’s contentdm oai stream for their ben lomond high school scrapbooks collection (found using the oai listrecords command on this set, https://cdm.weber.edu/oai/oai.php?verb=listrecords&metadataprefix=oai_qdc&set=blhs): <record> <header> <identifier>oai:cdm.weber.edu:blhs/316</identifier> <datestamp>2017-12-21</datestamp> <setspec>blhs</setspec> </header> <metadata> <oai_qdc:qualifieddc xmlns:oai_qdc="http://worldcat.org/xmlschemas/qdc-1.0/" xmlns:dcterms="http://purl.org/dc/terms/" xmlns:dc="http://purl.org/dc/elements/1.1/"xmlns:xsi="http://www.w3.org/2001/xmlschema-instance" xsi:schemalocation="http://worldcat.org/xmlschemas/qdc-1.0/ http://worldcat.org/xmlschemas/qdc/1.0/qdc-1.0.xsd http://purl.org/net/oclcterms http://worldcat.org/xmlschemas/oclcterms/1.4/oclcterms-1.4.xsd"> <dc:title>1957-1958 ben lomond high school scrapbook</dc:title> <dcterms:abstract> since 1953, students at ben lomond high school have been creating scrapbooks. these books document the memories of the students each year. the scrapbooks hold a snapshot and time capsule of each student body. each one contains photographs, newspaper articles and a written yearly history </dcterms:abstract> <dc:date>1957; 1958</dc:date> <dc:subject> students--1950-1960; education; ogden (utah); ben lomond high school (ogden, utah) </dc:subject> <!--cut for length--> <dc:identifier> http://cdm.weber.edu/cdm/ref/collection/blhs/id/316 </dc:identifier> </oai_qdc:qualifieddc> </metadata> </record> the required dublin core fields are enclosed in tags formatted <dc:elementname> with qualified dublin core fields formatted <dcterms:elementname> . this is consistent across contentdm oai feeds. for comparison, here is an example of a single record from the angelus studio photographs collection in oregon digital’s repository (again found using the oai listrecords command, https://oregondigital.org/oai?verb=listrecords&metadataprefix=oai_qdc&set=oregondigital:angelus-studio): <record> <header> <identifier>oai:oregondigital.org:angelus-studio/df71gd54c</identifier> <datestamp>2018-04-18t22:16:09z</datestamp> <setspec>oregondigital:angelus-studio</setspec> </header> <metadata> <oai_qdc:dcterms xmlns:oai_dc="http://www.openarchives.org/oai/2.0/oai_dc/" xmlns:oai_qdc="http://worldcat.org/xmlschemas/qdc-1.0/" xmlns:dc="http://purl.org/dc/elements/1.1/"xmlns:dcterms="http://purl.org/dc/terms/" xmlns:xsi="http://www.w3.org/2001/xmlschema-instance" xsi:schemalocation="http://dublincore.org/schemas/xmls/qdc/dcterms.xsd"> <dcterms:title> first congregational church at corner park and madison, portland. muddy street in foreground. </dcterms:title> <dcterms:alternative>23712; p2523</dcterms:alternative> <dcterms:description> first congregational church at corner park and madison, portland. muddy street in foreground.; default </dcterms:description> <!--cut for length--> <dcterms:format>image/tiff</dcterms:format> </oai_qdc:dcterms> </metadata> </record> here, all the required dublin core fields are enclosed in tags formatted <dcterms:elementname> . the previous mwdl version of the tool cannot parse this feed, resulting in false positives for all required fields. figure 2 shows a test of the previous required data checker with a collection from oregon digital: figure 2. existing required qualified dublin core data checker fails all records updating and extending the tool the first step of the project was to get and analyze the existing code. a serendipitous query about the tool from the american theological library association in early march 2018 kicked off the process of getting the code downloaded to a university of utah-hosted gitlab, which i then downloaded onto my local machine. having never worked with xsl before, i put myself through a crash course on it using the w3school’s xslt tutorial. i then used atom (v1.24.0 x64) to view the existing project and began to analyze its structure. this is when i learned the facet viewer feature had been excluded from the previous mwdl adaptation of the tool. i left this feature until last in my work, as i wasn’t sure i wanted it to be included. i chose to concentrate first on the required field checker feature, as this gets the most use by mwdl staff when auditing incoming collections. each feature consists of a php page paired with an associated xsl file used to test the data; this pair for the required data checker in qualified dublin core is shown in the red rectangles in figure 3: figure 3. xsl/php file pairs in required data checker my first step was to clone the index_oai_qdc.php file to make a specific landing page for oregon digital, index_oai_qdc_ore.php. i updated that file’s page title and headings to reflect this. then, i cloned the check_req_fields_qdc.xsl file as check_req_fields_qdc_ore.xsl and began updating the xsl variable definitions to work with oregon digital’s feed: <xsl:variable name="title" select="normalize-space(.//dc:title[1])"/> to <xsl:variable name="title" select="normalize-space(.//dcterms:title[1])"/> i applied the same change for all the required fields, resulting in this set of xsl variable definitions: <xsl:for-each select="//oai:record"> <xsl:if test="./oai:header[not(@status)]"> <xsl:variable name="title" select="normalize-space(.//dcterms:title[1])"/> <xsl:variable name="description" select="normalize-space(.//dcterms:description[1])"/> <xsl:variable name="subject" select="normalize-space(.//dcterms:subject[1])"/> <xsl:variable name="date" select="normalize-space(.//dcterms:date[1])"/> <xsl:variable name="type" select="normalize-space(.//dcterms:type[1])"/> <xsl:variable name="format" select="normalize-space(.//dcterms:format[1])"/> <xsl:variable name="rights" select="normalize-space(.//dcterms:rights[1])"/> <xsl:variable name="creator" select="normalize-space(.//dcterms:creator[1])"/> <xsl:variable name="contributor" select="normalize-space(.//dcterms:contributor[1])"/> <xsl:variable name="spatial" select="normalize-space(.//dcterms:spatial[1])"/> <xsl:variable name="id" select="./oai:header/oai:identifier"/> i also added a variable for contributor after analyzing several collections from oregon digital and seeing that some collections use creator and others use contributor. depending on fields required by various metadata profiles or guidelines, other institutions can similarly add or subtract required fields from the variable definitions. i then analyzed the xsl if tests to make sure they were consistent with the variables. because i added a variable for contributor, the tests had to be updated in two places: <xsl:if test="not($title) or not($description) or not($subject) or not($date) or not($type) or not($format) or not($rights) or not($creator) or not($contributor) or not($spatial)"> the column of the table that flags recommended fields is drawn by these tests: <td> <xsl:if test="not($creator)"> <p>creator</p> </xsl:if> <xsl:if test="not($spatial)"> <p>spatial</p> </xsl:if> </td> as oregon digital uses both creator and contributor, i wanted the table to flag a record only if neither was present. after searching stack overflow for possible solutions, i changed the above to: <td> <xsl:if test="(not($creator)) and (not($contributor))"> <p>creator/contributor</p> </xsl:if> <xsl:if test="not($spatial)"> <p>spatial</p> </xsl:if> </td> i then returned to my new landing page for the oregon-specific required metadata checker, index_oai_qdc_ore.php. i searched the document for ‘xsl’ to find and update the call for the xsl file to use my newly revised file (lines 6-7): $xml = new domdocument; if (@$xml->load($feedurl) === false) { echo "<p>please enter a valid feed url.</p>"; } else { $xsl = new domdocument; if ($mp == 'oai_qdc') { $xslpath = 'xsl/check_req_fields_qdc_ore.xsl'; } elseif ($mp == 'mods') { $xslpath = 'xsl/check_req_fields_mods.xsl'; } else { $xslpath = 'xsl/check_req_fields_dc.xsl'; } i moved on to the mapping checker next. it is structured very similarly to the required data checker, with pairs of php and xsl files. i cloned the index_oai_qdc.php and extract_qdc.xsl files in the mapping_checker folder and set about adapting them to work with oregon digital’s oai feed. figure 4 shows the base pair of qualified dublin core files in red rectangles: figure 4. xsl/php file pairs in mapping checker the mapping checker’s xsl (extract_qdc.xsl) uses for-each elements to select dublin core elements from each record and build a table display of required vs. recommended data: <td> <xsl:for-each select="//dc:title"> <div class='value'><a target="_blank" href="{$oaibase}{$recordbase}{./ancestor::oai:record/oai:header/oai:identifier}"><xsl:value-of select="."/></a></div> </xsl:for-each> </td> as with the required data checker, adapting these for-each elements for oregon required updating the select statements: <td> <xsl:for-each select="//dcterms:title"> <div class='value'><a target="_blank" href="{$oaibase}{$recordbase}{./ancestor::oai:record/oai:header/oai:identifier}"><xsl:value-of select="."/></a></div> </xsl:for-each> </td> finally, i had a look at the facet checker. after inspecting it and reading the ncdhc’s documentation, i elected to add the facet checker because it was already coded, didn’t need adaptation, and provides a visual audit tool to help identify inconsistencies in collections. figure 5 shows a view of subjects used in utah state archives’ salt lake city (utah) fire department photographs collection: figure 5. facet viewer demonstrated using subjects in utah state archives’ salt lake city (utah) fire department photographs collection this demonstrates how the same subject headings have been used consistently across the records, but could also reveal variations in spelling or application that needed remediation. when i updated the top-level index.php to include the newly coded features, i uncommented the facet viewer and moved it to the bottom of the list. the pages are called as list items in an unordered list. finally, i updated inc/byline.php; this file is called on each of the tool’s php pages to provide the footer. i added details to reflect the version changes and update the mwdl contact information. technical challenges/testing before making the tool live for member and staff use, i wanted to test it to make sure the updates were working correctly. a colleague suggested using mamp, a free personal web server program (see sources). university of utah library it support built a virtual machine for me with mamp installed that was accessible through vmware fusion (version 8.5.8). the testing process was fairly slow, primarily owing to some issues of copying the project file structure from my local machine to the correct root directory in the virtual machine as per mamp documentation (/applications/mamp/htdocs). there seemed to be a glitch with both the menu-based copy-and-paste and drag-and-drop function that should have worked as per vmware fusion documentation (see references). however, once the project had been copied to the virtual machine, i could test it and this resulted in a few rounds of iterative changes to improve aspects of the tool. for example, i realized i had failed to update the if variable for rights to look for “dcterms:rights”, resulting in false positives for missing rights statements in all oregon records. i then uploaded the project to the gitlab, created a merge request and notified university of utah staff we were ready to make the tool live. at this time, we decided to move the tool to a newer server and update its url from dpla-aggregation.sandbox.lib.utah.edu to mwdlmetadata.tools.lib.utah.edu, a more branded url indicating its publisher and purpose. once the tool was live, further testing revealed a dependency that needed updating for the set-selector to work correctly. a colleague in the utah digital infrastructure development team helped troubleshoot and made this update, and the tool was fully functional. further changes/extensions needed after using the new version of the tool to audit collections, i developed a list of future enhancements and cosmetic issues that need further troubleshooting. first, i realized the recommended field language is missing from the qualified dublin core required data checkers. this was true of the original adaptation, so by copying the files, my updated checkers inherited the same flaw. highlighting of table headers for recommended fields is also inconsistent in the mapping checker. the tool was originally built to display required fields in bright yellow and recommended fields in pale yellow. for example, i forgot to give the contributor column a css class in the mapping checker (extract_qdc_ore.xsl) when i added it, resulting in that column header not having any highlighting at all (line 3): <th class="required">identifier</th> <th class="recommended">creator</th> <th>contributor</th> <th class="recommended">spatial</th> after using the tool on several collections, i also noticed that the oregon required data checker draws the table cells for each record, even if nothing is flagged as missing. this is not the case for contentdm collections; while it isn’t bothersome for small collections, it is cumbersome for ones with large numbers of records. at the time of this writing, i haven’t had a chance to troubleshoot why this is happening. it might also be possible to code a single required data checker for all qualified dublin core feeds by using more sophisticated xsl-if statements and variable definitions. however, i have observed a marked performance issue with the oregon required data checker that i suspect may be due to my combined creator/contributor test; this bears further testing to see if separating them out improves performance and if the tool could be streamlined. i made a subsequent cosmetic update to reflect the current name of oregon digital’s repository (samvera rather than hydra) and updated index.php to indicate the simple dublin core required metadata checker also works with university of utah’s solphal system as well as islandora systems. i anticipate future updates following the forthcoming revision of the mwdl map in 2018. conclusion the project to update the metadata tool was ultimately successful, allowing mwdl staff to rapidly audit oregon digital’s incoming collections. between march and april 2018, we harvested over 50 new collections ranging in size from less than 5 records to more than 20,000, and the bulk of the audit process took roughly 2 days. the work was performed by the metadata librarian and a part-time undergraduate student worker. our student metadata assistant began in february 2018 and, with no prior knowledge of or experience auditing library metadata, was able to immediately add value by efficiently delivering feedback about metadata quality. relying on student workers for first-pass auditing frees the metadata librarian for other tasks. further applications of the tool could include doing internal quality control projects as well as preparing collections for either platform or metadata migrations. the profile of repositories used by mwdl’s members continues to diversify; as of this writing, an additional two mwdl members are planning moves from contentdm to islandora, and i envision continual adaptations and refinements of the tool to meet changing member needs. acknowledgements huge thanks to anna neatrour and brian mcbride at university of utah for advice, technical assistance, and good humor. a special thanks to the north carolina digital heritage center for creating the tool and sharing their work, and to sandra mcintyre for her work on the original mwdl adaptation of the tool. about the author teresa k. hebron is the digital metadata librarian at mountain west digital library, based at the university of utah in salt lake city, ut. footnotes [1] https://mwdl.org/getinvolved/oai_queries.php [back] [2] http://re.cs.uct.ac.za/ [back] [3] http://validator.oaipmh.com [back] references dpla oai aggregation tools project version 1.0 [internet]. [updated 2016 may 25]. north carolina heritage center; [cited 15 march 2018]. available from: https://github.com/ncdhc/dpla-aggregation-tools gregory l, williams s. 2014. on being a hub: some details behind providing metadata for the digital public library of america. d-lib magazine [internet]. [cited may 23, 2018]; 20:7/8. available from: http://www.dlib.org/dlib/july14/gregory/07gregory.html mamp [internet]. [updated 2018]. appsolute; [cited 2018 march 18]. available from: https://www.mamp.info/en/ mamp & mamp pro 4 documentation [internet]. [updated 2018 may 25]. appsolute; [cited 18 march 2018]. available from: http://documentation.mamp.info/ mcintyre s. 2015. new tools for rapid auditing of your collections [internet]. [cited 2018 may 23]. available from: http://mwdlnews.blogspot.com/2015/11/new-tools-for-rapid-auditing-of-your.html mountain west digital library dublin core application profile version 2.0 [internet]. [updated 20 july 2011]. mountain west digital library metadata task force; [cited 2018 may 23]. available from: https://mwdl.org/docs/mwdl_dc_profile_version_2.0.pdf moving and copying files between virtual machines and your mac [internet]. [updated unknown]. vmware; [cited 2018 march 25]. available from: http://pubs.vmware.com/fusion-7/index.jsp?topic=%2fcom.vmware.fusion.help.doc%2fguid-3c0ea5da-98dd-4835-9c84-354472b25303.html neatrour a, cummings r, mcintyre s. 2016. regional aggregation and discovery of digital collections: the mountain west digital library. in: varnum k, ed. exploring discovery: the front door to your library’s licensed and digitized content. ala editions. available from: https://collections.lib.utah.edu/details?id=713372 testing against one of several values in xslt? [internet]. [updated 2017 december 6]. stackoverflow.com; [cited 2018 march 28]. available from: https://stackoverflow.com/questions/47679280/testing-against-one-of-several-values-in-xslt xslt tutorial [internet]. [updated unknown]. w3schools.com; [cited 2018 march 20]. available from: https://www.w3schools.com/xml/xsl_intro.asp subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – breathing life into archon: a case study in working with an unsupported system mission editorial committee process and structure code4lib issue 57, 2023-08-29 breathing life into archon: a case study in working with an unsupported system archival repositories at the university of illinois urbana-champaign library have relied on archon to represent archival description and finding aids to researchers worldwide since its launch in 2006. archon has been officially unsupported software, however, for more than half of this time span. this article will discuss strategies and approaches used to enhance and extend archon’s functionality during this period of little to no support for maintaining the software. whether in enhancing accessibility and visual aesthetics through custom theming, considering how to present data points in new ways to support additional functions, or making modifications so that the database would support utf-8 encoding, a wide variety of opportunities proved possible for enhancing user experience despite the inherent limitations of working with an unsupported system. working primarily from the skill set of an archivist with programming experience, rather than that of a software developer, the author also discusses some of the strengths emerging from this “on the ground” approach to developing enhancements to an archival access and collection management system. by krista l. gray introduction this is a story about breathing life into an allegedly defunct open source platform, archon, to respond to continuing and emerging archival management needs — despite the end of developer support. since its launch in 2006, the university of illinois urbana-champaign has relied on the archon web application software to provide access to archival description, both for researchers and staff. however, following the release of archivesspace in 2013, development on archon has largely lapsed. what is described on github as the “last planned release” occurred in 2014 to provide migration tools for archivesspace. [1] the latest release, funded by institutions in the archon stability update project and completed by a vendor, occurred in 2017 and provided additional security enhancements, as many institutions continued to rely on archon to manage access to their archival metadata. [2] when i came to the university of illinois in early 2015, one of my first projects was to enhance the web presence for the illinois history and lincoln collections (ihlc) unit. this work included learning more about the source code structure of archon, especially in locating the display templates and theme files that could be modified without impacting any core functions. i was fortunate to be able to consult with christopher (chris) prom, one of the creators of archon, and began making changes to improve the appearance and functionality of archon in line with how i saw it being used or not used for different tasks in the ihlc. one of the advantages of this approach of embedded archivist development work, so-to-speak, is being able to see the problems firsthand and to be able to think through how the problems could be solved by adding features through the existing coding structures. over the past eight and a half years, my work with archon has expanded into creating, enhancing, and supporting more elaborate features for each of the units using archon within the special collections division at the university of illinois library, which encompasses four independent instances of archon: illinois history and lincoln collections (ihlc), rare book & manuscript library (rbml), university archives (ua) which includes the student life and culture archives and the sousa center for american music archives, and american library association archives (ala). as ihlc is oftentimes the “middle ground” between the larger rbml and ua, this made my role well-placed to listen and understand the needs of each unit. while some of these approaches and solutions may be particular to archon or the university of illinois, my hope is that others working with legacy or outdated systems can draw inspiration from this case study to determine what might be possible to enhance user experience and functionality now, rather than wait for the next, better system to become possible. one of my early supervisors in the archives field had the motto of “you’re making it better!” and that has been a theme of my work with archon. i can’t fix everything that is holding archon back, but i can make it better. background when archon first launched in 2006, it was groundbreaking. university of illinois archivists and project co-directors chris prom and scott schwartz presented archon, an open-source web-based archival management system built on a mysql or mssql database with a web-based public and admin user interface written in php, to a standing-room-only crowd at the society of american archivists annual meeting in august 2006. [3] designed to facilitate archival management and easy publication of archival descriptive records online, archon increased open access to finding aids at archival repositories across the country, from large universities to small colleges as well as government and nonprofit organizations. archivists could edit records within the web interface and immediately make those changes available to the public, who could either search or browse for descriptions of archival collections and finding aids in the database. by march 2008, about a year and half after its release, archon had been installed hundreds of times and more than thirty repositories had released it into production. [4] in december 2008, the university of illinois library received the mellon award for technology collaboration for “leadership and development work on project archon.” [5] by 2015 archon was showing its age. development resources in the us archives field had largely shifted to a new system, archivesspace, which aimed to combine the strengths of archon and archivists’ toolkit, the two primary archival information management software systems in the us at the time. [6] nearly ten years after its initial release, the default styling in archon no longer matched users’ mental models of how a database would work, or how a modern website might look. for example, on the landing page the search box for the default theme is tucked up at the top corner, and the center of the page presents search tips rather than information about the repository or contents of the database. figure 1. how the main page of archon appears when using the default theme moreover, at the university of illinois, encoding errors arising from using mssql for the backend database (which at the time did not support utf-8) led to collection descriptions inadvertently (and infuriatingly) populated with extraneous characters every time a collection with text that included double spaces, curved quotation marks, or any diacritics was saved to the database. figure 2. excerpt of a collection description with double spaces generating stray characters (from the august 2015 version of the control card for ihlc’s busey-yntema collection) archon also did not integrate with aeon, software that the rbml had adopted in 2014 to manage requests and retrieval of materials for use by researchers in the reading room or by staff. staff and researchers had to create request transactions for archival materials manually in aeon, leading to significant data inconsistencies and a lot of extra work. closer to home at the ihlc, a supplementary word document was employed to track which collection identifiers were already in use in the system (a document which, naturally, was not consistently updated). furthermore, unprocessed collections with no descriptions were displaying publicly with no way for a user to interpret if there was an error in page rendering or if the collection was undescribed, and a new employee like myself had no way of learning about the collection descriptive information in the database as a whole (including which collections remained unpublished) without checking each collection one by one. starting with styling my first steps to enhance user experience with archon were with styling; specifically through archon’s theme and template files. archon’s theme files include the css for the entire public interface, along with the code for generating the headers, footers, and standalone pages (including the main page). templates include the code for generating database record views such as the “control card” (modeled on the old index cards) and the finding aid for archival collections. both theme files and templates were designed to be customizable without touching the core code in archon. styling can seem simple, but it can do a lot to breathe life into an outdated system — whether making the interface more beautiful, adding accessibility features, or making pages responsive. while it might seem superficial, updating the visual elements can enhance how staff and researchers experience the system, making it a more pleasant and informative interaction. [7] styling and modification of templates supports the creation of better visual connections to other institutional pages so that the transition feels less jarring to visitors. i have not attempted to keep up with every style change in the university of illinois library website, but creating a theme that blended elements of the library website and the digital collections site (the library’s access repository for digital and digitized special collections content and other library resources) was a huge step toward a more integrated visual experience. modifying the theme or template can also provide additional context, such as a physical address and contact information (both prominently missing from the default templates in archon). figure 3. at left, ihlc’s archon as it appeared in february 2015. at right, as it appears in 2023. beyond aesthetic considerations of styling, something as simple as converting a link that is easy to miss into a larger button can improve user experience greatly by providing them a clear course of action. in the case of archon, adding code to the theme files also supported adding a large search box to the landing page that better meets current user expectations for interacting with databases. figure 4. at left, ua’s digital object record display as of 2022 that relies on users to recognize the url to click within the metadata. at right, how the digital object record appears in 2023 with the added “access files” button. simple changes, such as adding focus styling to support keyboard navigation and increasing text color contrast, can enhance accessibility. and while rebuilding the interface in a responsive framework like bootstrap was beyond my skill set and time commitment, adding media queries for different screen sizes created a “good enough” solution for access on different devices. [8] reusing data in new ways working more with the templates also led to learning more about how the functions for retrieving data for display from the database worked. when faced with new challenges and workflow needs, this background supported developing ideas of how to get the data out of archon without having to gather it manually page-by-page. one such challenge was how to gain an understanding of all the collections held at ihlc and how they were described (or not) in archon. archon lacked a native reporting function, so to fill this gap i wrote the code to generate data tables, building from the functions i had seen used in other pages and templates. lacking the skills to create a csv file to download, i generated an html table and then copied it into excel to further analyze the data. figure 5. a portion of the custom page that generates a table with basic information for all collections in archon for ihlc. this approach has proved valuable time and time again over the years. most recently, i wrote php code to generate a list of links to the ead versions of collections or records series with science, technology, and medicine-related subject headings to create a page for our partner at the consortium for the history of science, technology, and medicine (chstm) to crawl and ingest into their access system. using a list of relevant subject headings, the code iterates through an array of ids to generate a table for library staff to review with the results for each science-related collection or record series. code for a second page generates a simplified list of ead links with no styling or duplicates for our partner at chstm to crawl for ingest. the appropriate theme directory holds the files for both pages as well as the index.php file to route to them. php code excerpt showing iterating over an array of subject term ids to generate the review table: foreach ($arrtermschstm as $chstmtermid) { $objsubject = new subject($chstmtermid); $objsubject->dbload(); echo("<tr><td>"); if ($objsubject->id) { echo("<a href='index.php?p=subjects/subjects&id=".$objsubject->id."' target='_blank'>".$objsubject->id."</a>"); } echo("</td><td>"); if ($objsubject->subject) { echo($objsubject->tostring(link_none, true)); } echo("</td><td>"); if ($objsubject->id) { $subjectresults = array(); $subjectresults = $_archon->searchcollections(null, search_enabled_collections, $objsubject->id); echo(count($subjectresults)); echo("</td><td>"); foreach ($subjectresults as $subjresult) { array_push($arrresultschstm, $subjresult->id); echo("<a href='index.php?p=collections/ead&id={$subjresult->id}&templateset=ead&disabletheme=1&output={$subjresult->id}'>"); echo($subjresult->title); echo("</a><br />"); } } echo("</td></tr>\n"); } figure 6. review table listing chstm-relevant collections by subject heading in ua’s archon. understanding how to pull data from archon has also enabled the development of a solution for the lack of integration between archon and our special collections request management software, aeon. one of the methods for submitting requests for materials through aeon involves using a url with parameters to generate a get request and to autofill visible and hidden fields in the request form. by understanding how the control card template displays the contents of database fields to the user, i was able to write php code to auto-generate a link with the necessary parameters to autofill in the form. php code excerpt showing retrieving metadata from the collection object: $requesttitle = ($objcollection->title) ? $objcollection->title : ""; $requestdates = ($objcollection->inclusivedates) ? ", " . $objcollection->inclusivedates : ""; $requesttitle = urlencode($requesttitle . $requestdates); $requestidentifier = ($objcollection->classification) ? $objcollection->classification->tostring(link_none, true, false, true, false) . "/" . $objcollection->getstring('collectionidentifier') : $objcollection->getstring('collectionidentifier'); php code showing the process of building a portion of the get request link: $requestbaselink = $_archon->config->requesturl; // defined in config file // concatenate the field names (url parameters) and metadata from the collection to form the request link $requestlink = $requestbaselink; $requestlink .= "&itemtitle=" . $requesttitle; $requestlink .= "&callnumber=" . $requestidentifier; this approach works best for collection-level metadata (it does not allow one to autofill a particular folder title with the request link, for instance). while a shortcoming in some ways, this approach is well-suited to ihlc and university archives finding aids (where individual folder titles would be listed), as these are primarily separate pdf documents rather than encoded in the database itself. responding to emerging needs this increased facility with the archon code base and how data could be reused has also helped with imagining how the database could meet emerging needs not foreseen at the time of its original creation. in preparation for an upcoming move, by summer of 2021 the rbml, ihlc, and ua had all begun projects to barcode archival boxes so they could be uniquely identified when transferred to the new vault. in order to centralize this tracking information, i wanted to find a way to record the barcode assignments in archon. archon already has a location table built into its database for tracking the locations of boxes (by location, range, section, and shelf) that lent itself well to consideration for this purpose. not having the capacity to significantly rewrite the source code, i ultimately decided the best option would be to use the shelf field in the location table to record the barcode and move the shelf information (if it existed) for any barcoded box to the section field, combining the two data points there. while not ideal, this meant avoiding having to change any database tables, which was not within my existing skill set. i consulted with fellow archivists to check if any important functionality would be lost. i already knew that many of the university archives locations did not use the shelf field at all, and for those that did, i confirmed that combining the data with the section information would still serve their needs and received permission to proceed. i also wanted to think ahead to how we would update the locations for hundreds if not thousands of boxes after the initial move. storing barcodes in archon would allow the locations of boxes to be updated more quickly by using the barcodes to retrieve the matching line in the location table to update it through an import file. this information could also be used to uniquely represent boxes in aeon, which had the potential to make it easier for researchers to select the requested box and have it be auto-filled via a link. staff could also employ it to aid in the offsite retrieval process from the library’s high density storage facility. i had previously adapted the request function to create separate links for each line of the location table in order to support the future adaption of aeon by ua and ala, both of which store boxes in multiple buildings. drawing on this previous work, i adapted code to also provide users with the ability to request individual boxes and to send the corresponding barcode to aeon along with the box information for the request. staff could then use the barcode to retrieve the location information without having to look up the collection or record series and then having to skim through the list to find the location of the specific box. figure 7. search box and results page for retrieving location information by barcode (location redacted). implementing these new features involved several components: admin and public user interface changes, as well as an import script. each of these built on archon customization work completed earlier. within the admin interface i increased the input field width for the shelf field to hold a fourteen-digit barcode. this drew on work done years earlier when i had made a similar change to accommodate a 5-digit range value for ihlc locations. as shown in figure 7, i also created a barcode-specific search box that allows staff to retrieve a box’s collection and location by scanning the barcode. this work drew on past experience creating a search box to retrieve collections by their identifier for ihlc. archon’s built-in search is a general keyword search and did not serve our needs for precise retrieval in either use case. the request function in the public interface drew on an approach initially developed to handle ala and ua collections that were stored at different service locations. this version uses a pop-up modal box to display request links for individual lines of the location table so that the associated service location can be displayed and encoded properly when submitting an aeon request. in its most recent iteration, the request modal box checks to see if the shelf field is populated with a barcode. if it is, the code will display the box information to the user and send it to aeon. if not, the line in the location table is hidden and a generic “other” option is generated – a necessary adjustment since ihlc and rbml staff did not originally enter location table descriptions to be displayed publicly and only the descriptions of barcoded boxes can be assumed to be ready for display to the public. figure 8. example of a request modal box (with an example filter of “2”) for an ihlc collection with an embedded pdf finding aid, showing also the “other” option for unbarcoded boxes. the import script drew on an earlier project to survey the archival collections in ihlc and record where they were located. the code includes logic that checks that the data appears to be correctly structured and catches errors. it also allows for variations based on whether the import file is adding location information for a given box or barcode for the first time or is merely updating it. in order to create a means for staff to search and update by barcode without having to specify the collection or box number associated with that barcode, the code also establishes a new getlocationentryidfromshelf($string) function. i based this off the similar get___idfrom____($string) functions, such as getextentunitidfromstring($string). [9] while this did not have a direct correlation from past work, it grew out of many, many readings of the source code to understand the functions underlying how something worked (or didn’t work) in archon. while hacking a database field for a different purpose and collapsing data from two fields into one is never ideal, leveraging my understanding of the inner workings of archon, along with my on-the-ground experience managing an archival repository, allowed me to create a feature that extends the use of the barcoding project. anticipating maintenance as the complexity of the features that i developed grew, i adjusted my approach to make the source code easier to maintain and easier to roll out and reuse in multiple instances. since this was essentially a side project, being able to make changes in a central location or to a file that could be copied whole rather than to code segments that had to be copied and pasted individually was a significant improvement to the workflow. most importantly, the adaptability of the code supported the sustainability of archon. the first improvement came when i began defining variables in archon’s config file rather than burying them in the template or theme files. by defining these variables in a central location in the top level directory for archon, making edits or changes became much easier. for example, defining a variable in the config file for text to create an alert banner on ihlc’s archon to notify users of unexpected closures from the header file in the theme made it easier to change this text quickly. defining a series of variables for the strings and arrays needed to build a request link from the control card and finding aid templates made it easier to change both request links at once as well as to transfer code between different archon instances. this approach also supported easily turning features on or off, such as the ability to offer expanded descriptive information about collections in the browse lists. while this is helpful when working with template or theme files, it is especially valuable when modifying the core code to do something that changes archon’s functionality. in late 2016, a contributor introduced a new feature (or bug, depending on your perspective) to use an inexact search rather than an exact search in archon’s identifier search function. the inexact identifier search did not always return the searched-for identifier at the university of illinois, so the change had to be manually reversed. to provide better control and clarity, i rewrote the php code to allow institutions to choose which type of identifier search they wanted from the config file rather than having to dig through the source code. php code excerpt showing the use of a variable ($_archon->config->searchexactidentifier) from the config file for the identifier search: if ($_archon->config->searchexactidentifier) { $minlengthquery = " or collectionidentifier = ?"; $minlengthvars = array(str_pad($collectionidentifier, config_collections_collection_identifier_minimum_length, "0", str_pad_left)); } else { // replace = with like $minlengthquery = " or collectionidentifier like ?"; // added wildcards with $collectionidentifier for partial search $minlengthvars = array(str_pad("%$collectionidentifier%", config_collections_collection_identifier_minimum_length, "0", str_pad_left)); } all the unit-specific variables being defined for the aeon request link, however, soon created an unwieldy config file. in this case using an include statement to separate out those variables into a request-specific config file made the code easier to manage. expanding the use of include statements to the implementation of the aeon request link made the code easier to manage and modify as well. admittedly, the organic way the request link functionality has developed – from initially providing only basic collection-level information to now supplying information about the individual boxes, barcodes, and service locations – has made even this approach a bit unwieldy. the template directory now holds two main include files for the request link itself, requestprep.inc.php and requestlink.inc.php, but also requestlinkforboxes.inc.php, and three more relating to the location table displays (one for staff, one for the public, and one for a summary). however, it is much more manageable than it would be with everything buried in the existing template files. it also provides for more seamless updating across instances, as each unit’s control card template is unique, but the include files for the request link setup can be identical across units. exploring new possibilities for solving old problems in spring 2022, one of my fellow archivists at the library, bethany anderson, wanted to know how diacritics could be represented in archon. as the project director for the doris duke native oral history revitalization project, she needed diacritics to accurately represent the names of many native tribes and nations (for example, the stó:lō nation). however, the university of illinois’ instance of archon had been unable to properly save and display diacritics or even simple accents. this issue had yet to be fully diagnosed and resolved. an email exchange with chris prom, associate university librarian for digital strategies, opened a new line of inquiry to this problem. my work years earlier investigating a related issue had focused on tracing the issue through the php source code, but prom suggested looking at the underlying database’s character set. he recommended seeking support from library it to create a separate test mssql database for this work to be able to test and debug without concern of the impact on the live, public database. this new line of inquiry soon revealed that the patterns of characters being garbled indicated an issue of double encoding into utf-8 for the browser (for example, ñ would display as ã±). i also learned that in 2019 mssql finally added support for utf-8 encoding, and having the underlying database support utf-8 encoded text began to seem like more of a possibility. [10] working extensively with jon gorman from library it’s infrastructure management and support (ims) group for the database setup, modification, and conversion, i was able to test modifying the php source code to support additional characters, first in the original encoding in a test copy of an existing database and then with the full breadth of utf-8 with the data migrated to a utf-8 mssql database. the process did not go smoothly but drew on a lot of the experience and problem-solving skills i had built with archon for the previous seven years. in contrast to the features and solutions described earlier in this article, these issues could not be resolved through the php source code alone. after tracing through the source code for the display, i discovered that commenting out a line of code that was converting the string from iso-8859-1 (latin-1) to utf-8 led to archon displaying standard latin diacritics and symbols without garbling them. logically this didn’t make sense. our database could not be using utf-8 encoding (since it was a pre-2019 mssql database), but the function for the user interface display was clearly being passed text that had already been converted to utf-8 and was garbling it by converting it a second time. investigating the code, one theory that emerged was that freetds (which, as a non-developer, i had not realized was a part of the process until then) could be converting it as an intermediary between the mssql database and the php code. this understanding soon proved valuable. when i tried to connect archon to use the migrated utf-8 test mssql 2019 database, i got a blank white screen, which turned out to be due to having to set freetds to a higher version to work with mssql 2019. following the pattern outlined in the previous section, i coded the mssql and freetds modifications (which appeared multiple times in different files throughout the source code) so that they could be turned off and on via variables in the config file. my tests of copying utf-8 characters from other languages and the native nations and tribes names eventually worked and saved correctly. [11] we rolled out the changes to the ihlc’s archon in may 2023. migrating the three other instances of archon revealed new complexities in how the code interacted with server settings as well as the impact of previous attempts to fix the double encoding issue. for instance, the earlier strategy of simply commenting out a line of code converting the string from iso-8859-1 (latin-1) to utf-8 worked on the development server but not the production server for the three other archon instances. on this production server it led to text fields with diacritics or double spaces failing to display (and therefore being overwritten with an empty string when saving). tracing through the code, i found the error arising from functions like htmlspecialcharacters and preg_replace returning null if the string passed to them is not in utf-8. i added some code to catch this error and convert the string to utf-8 if needed so that these functions would not delete data if the encoding was not in utf-8 as expected. analyzing the text strings on each server as hexcodes, i was able to find that the php received text in utf-8 on one server and in a different encoding (possibly latin-1) on the other. this pointed to the freetds settings as the likely culprit. to resolve the issue, gorman wrote a stanza to establish the appropriate freetds settings to support utf-8 versions of archon on this other server as well. after resolving these issues, we successfully migrated both the rbml and ala instances of archon to utf-8 versions and began adding in diacritics for names that had been inaccurately represented for years. in migrating the ua instance of archon, we encountered one final issue. in the test copy of their database, with all the same code and server settings that had worked for rbml and ala, sample text in the control card with diacritics would save correctly but display in the same garbled, double-encoded pattern from earlier. figure 9. the saved version of the sample text in the admin editor in the ua utf-8 test database. [12] figure 10. the public display of the same text with the characters in the scope and content note (at right, labeled “description”) displaying with the double encoding error pattern and the sample text in the abstract (at left) displaying correctly. given that the abstract displayed the sample text correctly but the scope and content note did not, this pointed to an issue in the display of that particular field in the control card template. applying a different template set (the one in use for ala’s instance of archon) led to the text displaying correctly. comparing the two templates revealed a modification to the ua template that added another layer of encoding the scope and content note into utf-8, leading to the persistent double-encoding. in the ua template they had: <?php echo (utf8_encode($objcollection->getstring('scope'))); ?> as opposed to in the ala template: <?php echo($objcollection->getstring('scope')); ?> once corrected, the ua instance worked as expected. this issue, while ultimately simple to resolve, also points to the challenging opacity of one-off changes in legacy codebases, and to the value of documentation. in this case, the source code did not even include any comments around this line about why this modification was added, or that it was an attempted fix for a specific issue. in working with legacy codebases, there is a balance between band aid-type solutions vs. more systemic investigations. however, a focus on increased transparency regardless provides value for sustainability as well as future directions. throughout this project, i drew heavily on what i had learned in navigating, reading, and modifying archon’s source code over the previous seven and a half years. i am grateful to jon gorman for all his help in the areas where i did not have experience, and to bethany anderson and chris prom for bringing up the issue and pointing me in the right direction. while it was more involved than anticipated, after more than seven years of struggling with double spaces and having to tell my student employees that, no, we can’t spell names or words in other languages correctly, it was wonderful to finally have archon support utf-8. figure 11. excerpt of the control card description for the doris duke indian oral history program archives, 1908-1995, showing the names of native nations and tribes successfully displaying with diacritics. conclusion: is there really hope for archon? to some extent there is an elephant in the room. after all this, archon is still running on a maximum of php 5.6 and cannot be updated further due to dependencies on a pear library that has not been updated since 2012, with the last stable release from 2007. maintaining archon into the future will eventually require dedicated development support for rewriting the entire interface and database communication layer, likely from the ground up. so, was all this a waste? i would argue it’s not a waste. all this work has provided us with a more functional system that can accommodate changing needs and support emerging workflows. it has also provided proof-of-concept for certain features not originally included in archon. and it has provided years of more accessible information about our holdings. while archon’s limitations have held us back in some ways, being an archivist on the ground who could see new solutions and implement them has allowed us to make far better use of the system rather than staying in a continual holding pattern. beyond improving our current use of archon, the investment in gaining in-depth understanding of archon, its data model, and how it functions can be leveraged also toward the assessment of future systems, as well as eventual plans to transition into a new system. developing new ideas and imagining how the current system could function more effectively lends itself well to envisioning what could and should be improved in a new system as well as to analyzing what else might be possible. not only did this work breathe new life into archon for the present, but it also positions us better in planning archival management systems for the future. notes [1] chris prom, “release archon 3.21-rev. 2,” archonproject/archon on github, released january 17, 2014, https://github.com/archonproject/archon/releases/tag/v3.21.2 [2] anne salsich, caitlin nelson, and nat wilson, “should i stay or should i go: what to do when your beloved software is no longer supported? update it yourself!” (conference presentation slides, library technology conference, 2017, https://digitalcommons.macalester.edu/libtech_conf/2017/sessions/18/). [3] scott w. schwartz, christopher j. prom, christopher a. rishel, kyle fox, “archon: a unified information storage and retrieval system for lone archivists, special collections librarians and curators” partnership: the canadian journal of library and information practice and research 2, no. 2 (2007): 2. while the early documentation emphasized that archon could run on any web server, by 2013 archon’s readme file specifically recommended against using a windows server and noted that archon “was developed and tested on a lamp (linux, apache, mysql, php) server.” https://github.com/archonproject/archon/blob/master/readme.md [4] scott w. schwartz, christopher prom, kyle fox, and paul sorensen, “archon: facilitating global access to collections in small archives” (74th ifla general conference and council, quebec, canada, august 2008), 3. [5] “two university library archivists honored with 2008 andrew mellon foundation award,” university of illinois library, published december 16, 2008, accessed from the december 28, 2015 crawl of the page saved in the internet archive, available at https://web.archive.org/web/20151228234745/http://www.library.illinois.edu/news/archon_award.html. [6] assessing whether archivesspace has achieved this goal is out of scope of this article. while a number of former archon institutions have migrated to archivesspace, others have shared the perspective that “the result of that merger, archivesspace, left many archon users feeling excluded. the technical complexities of hosting a local instance, the exhaustive descriptive possibilities, and the steep learning curve all presented significant barriers to repositories accustomed to archon’s ease of use.” jeremy brett, colleen mcfarland rademaker, doris cardenas, anne thomason, and nancy webster, “light from the north: reviving the spirit of archon through atom” (presentation abstract, midwest archives conference annual meeting, detroit, mi, april 6, 2019, https://www.iastatedigitalpress.com/macmeetings/article/id/294/). [7] research has suggested that higher website aesthetics positively influences the user’s opinion of the corporate body represented by the website as well. see zhenhui (jack) jiang, weiquan wang, bernard c.y. tan, and jie yu, “the determinants and impacts of aesthetics in users’ first interaction with websites” journal of management information systems 33, no. 1 (2016): 229–259. doi:10.1080/07421222.2016.1172443. aesthetics may also have an impact on usability, but research outcomes in this area of study vary. a 2019 meta-analysis of studies examining the impact of aesthetics on user performance found a small positive impact of aesthetics on performance. see meinald t. thielsch, jana scharfen, ehsan masoudi, and meike reuter, “visual aesthetics and performance: a first meta-analysis,” in proceedings of mensch und computer 2019 (new york, ny: association for computing machinery), 199–210. https://doi.org/10.1145/3340764.3340794. in a work-in-progress presented in spring 2023, researchers in germany argued that the aesthetic-usability effect may be strongly impacted by levels of “processing fluency,” an impact they summarize as “mind at ease makes usable and beautiful,” though this research measured only perceived usability (as well as perceived aesthetics and processing fluency) rather than performance. see jan preßler, lukas schmid, and jörn hurtienne, “statistically controlling for processing fluency reduces the aesthetic-usability effect” in extended abstracts of the 2023 chi conference on human factors in computing systems (new york, ny: association for computing machinery), article 261, 1–7. https://doi.org/10.1145/3544549.3585739. [8] there are many guidelines and recommendations available for improving accessibility and responsiveness of websites. “designing for web accessibility” from the w3c web accessibility initiative (wai) is one possible place to start. https://www.w3.org/wai/tips/designing/ [9] one shortcoming of this approach is that it assumes only one location entry is associated with a particular barcode and cases may exist where components from multiple collections are stored in the same box. [10] pedro lopes, “introducing utf-8 support for sql server,” sql server blog, microsoft, july 2, 2019, https://techcommunity.microsoft.com/t5/sql-server-blog/introducing-utf-8-support-for-sql-server/ba-p/734928. [11] i am grateful for frank da cruz’s utf-8 sampler for testing, available at https://www.kermitproject.org/utf8.html. [12] the sample text used came from frank da cruz’s utf-8 sampler, referenced above. about the author krista l. gray is the archives program officer for illinois history and lincoln collections at the university of illinois urbana-champaign library. subscribe to comments: for this article | for all articles one response to "breathing life into archon: a case study in working with an unsupported system" please leave a response below: charles riley, 2023-10-19 great to hear about this successful use case in updating archon to accept utf-8! leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – core concepts and techniques for library metadata analysis mission editorial committee process and structure code4lib issue 52, 2021-09-22 core concepts and techniques for library metadata analysis metadata analysis is a growing need in libraries of all types and sizes, as demonstrated in many recent job postings. data migration, transformation, enhancement, and remediation all require strong metadata analysis skills. but there is no well-defined body of knowledge or competencies list for library metadata analysis, leaving library staff with analysis-related responsibilities largely on their own to learn how to do the work effectively. in this paper, two experienced metadata analysts will share what they see as core knowledge areas and problem solving techniques for successful library metadata analysis. the paper will also discuss suggested tools, though the emphasis is intentionally not to prescribe specific tools, software, or programming languages, but rather to help readers recognize tools that will meet their analysis needs. the goal of the paper is to help library staff and their managers develop a shared understanding of the skill sets required to meet their library’s metadata analysis needs. it will also be useful to individuals interested in pursuing a career in library metadata analysis and wondering how to enhance their existing knowledge and skills for success in analysis work. by stacie traill and martin patrick introduction metadata analysis, or the application of data manipulation and analysis tools and skills to library metadata, has become an increasingly important task in libraries. library job postings and job descriptions show a growing need for metadata analysis skills. metadata analysis, even though it is rarely labeled as such, is a necessary skill for staff in libraries of all types and sizes, and across a variety of technical functions, including cataloging and metadata, acquisitions, electronic resources management, collection management, and discovery, as well as traditionally information technology-heavy functions such as library systems management (binici 2021; gonzales 2019; nasig 2019; ratledge and sproles 2017; hall-ellis 2015; mathews and pardue 2009). metadata analysis can also be an important skillset for library staff in more recently emerging specialties, such as research data services and digital scholarship (hannah et al. 2020; thielen and neeser 2020; skene 2018; xia and wang 2014). despite the need for metadata analysis skills, few jobs use the phrase “metadata analyst” in their titles. but many library staff are doing this work, no matter what their official job titles might be. this is abundantly clear from even a cursory sampling of conference presentations and publications, including many published in this journal. but perhaps because metadata analysis work is distributed across so many library functions and job titles, there is little clear guidance for learning these skills, or even understanding what they are. as people who came to the work of metadata analysis because of some combination of experience, aptitude, and interest, but without the base of skills and knowledge that we have today, the authors know very well how challenging it can be to piece together both the technical and library-specific knowledge required to do metadata analysis well. to put it another way, there are a huge, almost overwhelming, number of case studies and project descriptions from which the aspiring analyst can glean useful information about what to learn and (sometimes) how to learn it, but there are few resources that synthesize and generalize that information into something digestible and actionable. in this article, we will describe metadata analysis and its typical tasks, outline the skills and knowledge required to execute those tasks successfully, and discuss several key analysis and problem solving approaches that a good metadata analyst relies upon. metadata analysis scope and tasks before we can meaningfully explore the skills and knowledge required for good metadata analysis, we will attempt to clarify what we mean by metadata analysis in the library context. data analysis broadly defined includes processes such as data cleaning, exploration, transformation, and statistical analysis. library metadata analysis may include all of these activities, embedded in the understanding of metadata in context as a central, living component of an ecosystem enabling many core library functions. here, “ecosystem” means a library’s larger environment of internal and external repositories, services, and tools, in which planning, analysis, design, deployment, and maintenance of metadata management processes are frequent and ongoing needs. a large number of library business processes — in areas ranging from discovery to access to collection management — depend on accurate, high-quality metadata, meaning that metadata analysis is essential work in most libraries. working from that definition, we can generalize about the tasks involved in completing typical metadata analysis projects as a helpful first step toward identifying the specific skills and knowledge areas an analyst needs. first, the analyst needs to know (or be able to find out) what data is available, and the sources from which it is available. second, the analyst needs to know how the data can be accessed programmatically or in batch with automation tools. third, the analyst needs to know how the data are used, and how they should be used, if that differs from current usage. fourth, the analyst needs to determine what data changes, merges, or enhancements are needed to produce the desired outcomes. finally, if the same process needs to be repeated, the analyst must be able to determine how to do it efficiently, sustainably, and at whatever scale the library requires. with these tasks as a framework, we can start to see what the analyst needs to know in order to work through the tasks successfully. core knowledge areas based on the generalized tasks, we developed a list of core knowledge areas for metadata analysis. the list of seven core knowledge areas presented here is extensive, but not comprehensive; it would be difficult to include every knowledge area that library metadata analysis jobs might require, in part because library staff frequently have multiple intersecting areas of responsibility. the authors have chosen to focus on these seven areas based on their own experience and informal observation of the field. a future project to survey library staff with metadata analysis responsibilities to learn what knowledge areas they find essential would be valuable. seven core knowledge areas for metadata analysis library metadata standards, applications, and systems data cleaning and text normalization data serializations interoperability database design and querying web technologies and services workflow analysis and documentation practices because there are many different educational and experience backgrounds that might make someone interested in metadata analysis work, it should be expected that an analyst will come to the job with more expertise in some areas than others. the authors have relatively similar backgrounds: both had deep experience in library technical services, especially cataloging and electronic resources management, along with technical aptitude and interest, but little formal education in programming, computer science, or business analysis. as a result, some of these knowledge areas were more challenging than others for us to learn, and we both continue to learn every day on the job. most metadata analysts will not need deep expertise in all of these areas, and some may need skills in areas not among these seven, such as data visualization and project management. the key is to know enough to get the job done, and to know how and where to learn more when the need arises. in the next section of the article, we will go into some detail about each area of knowledge. library metadata standards, applications, and systems the first core knowledge area represents the domain-specific knowledge that is a requirement for almost all metadata analysis: library metadata standards, how those standards are applied, and library systems. the analyst should have a broad familiarity with library metadata standards, including content standards, and encoding and transmission standards. additionally, the analyst should have expertise in widely-used standards, which may vary from library to library. some fluency with very widely-used standards, such as marc, is likely to be necessary for most library metadata analysts. the analyst needs a good understanding of the library management system(s), discovery system(s), and other repositories in use at their library, comprising both a high-level understanding of the systems’ capabilities and limitations, and a detailed understanding of their metadata management processes. because metadata is the foundation for many core library business processes, the analyst should be able to identify and articulate the cross-functional impacts of data changes, and how internal data changes may affect external services and tools that rely on metadata. finally, the analyst needs to be competent in using whatever built-in reporting and analysis capabilities are offered by the library’s systems. key domain-specific knowledge that is sometimes overlooked includes legacy standards and practices, and local practices and schema. it is not always easy to gather such knowledge, because documentation of past decisions and practices may be incomplete or inaccessible. depending on how thoroughly a library recorded such information in the past, documenting past practice can be an important part of an analyst’s work. data cleaning and text normalization the second core knowledge area is data cleaning and text normalization, also known as data wrangling. this is the practice of evaluating and transforming data to ensure its validity, accuracy, completeness, and consistency. along with library metadata standards and systems, this is the most important knowledge area for metadata analysis work. it is critical for data remediation and enhancement projects, but also for devising mappings and crosswalks between metadata standards, and for creating any kind of analysis that combines data from varying sources, standards, or application profiles. data wrangling encompasses many specific skills. most important for metadata analysis work are programmatic file and text manipulation, recognizing and expressing patterns (often through tools such as regular expressions), using conditional statements and actions, and isolating elements for evaluation, modification, or both. to successfully wrangle textual metadata, the analyst also needs to understand how to use and convert between character encodings, know where to find documentation of any standardized values, and be able to identify where the library’s metadata departs from those standards. without dwelling on specific tools, the ability to use programmatic manipulations is undeniably important. although learning a scripting or programming language is a likely path for any metadata analyst who does not come to the job with that proficiency, there are many tools that do not require programming ability that can be used to achieve the same goals, and which can also serve as a gentle introduction to programming concepts and ways of thinking. specific tool choices are less important than conceptual understanding. data serializations the third core knowledge area is data serializations. data serializations are ways to package data structures or data objects for storage or transmission. the most common serializations for library metadata (like much other data available on the internet) are xml, json, and csv, but there are many others. a metadata analyst should be well acquainted with those three serializations, and also able to work with others as needed. the keys to working with any serialization are to understand whether and how the data can be processed with familiar toolsets, and to be able to output data, once processed, in other serializations. when working with less common serializations, it may be necessary to explore new or unfamiliar tools to assist in transforming the data to a more familiar format. interoperability the fourth core knowledge area is interoperability. there are two types of interoperability with which a metadata analyst should be familiar: metadata interoperability and systems interoperability. the dublin core metadata initiative glossary describes metadata interoperability as “the ability of … systems or components to exchange descriptive data about things, and to interpret the data … in a way that is consistent with the interpretation of the creator of the data.”[1] metadata interoperability is about semantic compatibility: do these two elements really have the same meaning? do these two records really represent the same resource? the ability to formulate and answer questions like these is critical to the work of a metadata analyst when it involves mapping across metadata schemas, incorporating enrichments from one metadata source into another, or creating data mashups to address core business needs, such as collection analysis. another key concept of metadata interoperability is identifiers, which are essential to almost all automated metadata processing. reliable identifiers are important for the matching, reconciliation, and enhancement tasks that comprise much of the work of metadata analysis. reliable identifiers are also critical for functional linked data. a metadata analyst should understand what identifiers are available in the metadata they work with, which entities and levels of granularity each type of identifier signifies, and how the identifiers are maintained. systems interoperability can be understood as the context surrounding metadata: how is it stored, accessed, communicated between systems? how can the analyst fetch the data when needed, or use it to power external systems or tools? a metadata analyst may not need deep technical expertise in this area, but rather needs to have a clear understanding of how metadata will be used by and communicated between systems, especially if and when full automation of a workflow or integration is the goal. even if an analyst is not writing the code for their own extract-transform-load (etl) pipelines, they will likely need to be able to conceptualize the process at a high level and devise requirements to communicate with a developer. database design and querying the fifth core knowledge area is database design and querying. again, deep knowledge here may not be necessary, but a basic understanding of how relational databases are structured is crucial. an analyst should have at least a high-level understanding of table relationships, logical structure, and data modeling. knowing specific query languages such as sql, xquery, xpath, or sparql may be important depending on the systems and data an analyst works with. but more than fluency in any particular query language, it is important for an analyst to understand the core concepts of relational algebra that those languages implement, such as set operations, unions, differences, joins, and so on. those concepts underlie many of the fundamental tasks in data analysis beyond database querying. web technologies and services web technologies and services comprise the sixth core knowledge area. much of what a metadata analyst needs to know in this area overlaps with interoperability, but beyond that, an analyst needs to understand some specifics of how to work with data on the internet. a high-level understanding of how web applications are created and managed, web technology stacks, html, css, and javascript are useful. in addition, an analyst may need a deeper understanding of web apis, the request-response cycle, and library-specific internet standards and protocols such as openurl, oai-pmh, and z39.50/sru. a working knowledge of semantic web and linked data concepts, standards, and technologies such as rdf, bibframe, json-ld, and sparql are also essential; even if these are not currently required for a metadata analyst’s work, they may well be in the near future. workflow analysis and documentation practices the seventh and final core knowledge area is workflow analysis and documentation practices. important skills in this area include business and requirements analysis, process improvement analysis, and flowcharting. understanding how to devise and communicate workflows and dataflows is especially important for an analyst’s work to be shareable and replicable, and also important when a metadata analyst collaborates with a developer. even when, as is common, an analyst acts as both analyst and developer on small projects, these skills help an analyst understand exactly what they are implementing in code, and how to communicate that to others. finally, because documentation is often a major part of an analyst’s work, they must also understand documentation best practices, such as plain language and documentation accessibility. practical analysis techniques and problem solving approaches libraries create, acquire, maintain, and share significant quantities of metadata, in a wide variety of types and formats, for various functions, including bibliographic, authorities, acquisitions, administrative, holdings, and others. in most libraries the amount of metadata constantly increases, and each year brings new pieces of metadata that are not present in older records, or new standards that create records substantially different from older records. the increasing reliance on externally-sourced metadata and growing demand to share metadata with external partners creates in turn an increasing reliance on varying quality and approaches to the metadata the library manages. both human and computer generated metadata are subject to errors that need to be addressed. it is within this context that the need for metadata analysis, with a goal to solve metadata problems, arises. we recognize at least 6 areas where metadata analysis is frequently needed: workflow and automation, configuration, batch creation and updates, import and export, troubleshooting, and documentation creation. these areas are not necessarily independent of each other, and often overlap or form a sequence within a larger project. the knowledge areas and skills described above, implemented through tools and techniques with which the analyst is comfortable, inform the final shape of the approach taken for any particular analysis. techniques in order to address a metadata problem, the metadata analyst must begin by gathering information about the problem. we will first discuss the basic techniques a metadata analyst can use, and then we will turn to an overview of the kinds of information that are useful for completing a metadata analysis. while not exhaustive, common techniques include observation, reading documentation, advanced queries and searches, conversation, and process mapping and workflow diagrams. sometimes one of these techniques will be enough to resolve the problem; other times the analyst may need to rely on all of them, along with others. as an example, martin was tasked with updating cataloging macros for both a migration to sierra from millennium, and for rda, a workflow and automation problem. he chose to use observation in order to understand how catalogers were using macros, as well as to see where things were different enough in sierra that the macros broke. then, he turned to the autoit – a scripting language for automating windows tasks and macros – documentation in order to understand what was possible. finally, he used a kind of process mapping to map the logic that the macros used in order to get them working again. approaching a metadata problem one high-level way to organize an approach to a metadata problem is by thinking through four aspects of the problem: context, extent, urgency, and importance. in the macro cataloging example above, the context and extent was determined through observation, while the urgency and importance were related to both the move to a new ils, and the recently released rda standards for cataloging. context includes such things as local practices and policies, national and international standards, and the constraints and possibilities of local systems and the tools the analyst is comfortable with. context also includes other, pending job duties and projects on your plate. other important contexts can include whether there is direct, local control over the metadata at issue or not. for instance, as customers of ex libris, we rely on ex libris’s central discovery index for a significant amount of discovery metadata. this means that article level metadata, for example, is not something we can directly edit. before an analyst can formulate a plan to address the metadata problem, it is important to understand if, for example, there is a need to deal with one subfield in one record, or multiple subfields across 25,000 records. this is the extent of the problem. in the modern library, it is likely that one record with an issue means there are more with the same issue. this work involves using preliminary analysis tools at your immediate disposal, such as using a global search in the library catalog, or doing queries through other means such as sql or another tool that the catalog might interact with. if the problem was raised by someone else, the analyst can follow up with that person to find out if they have useful knowledge of the problem’s extent. understanding the extent helps the analyst to determine if the problem would be best addressed programmatically, for example, through the use of a scripting language such as python. depending on the extent of a problem, a one-time solution with many manual steps may be sufficient; for larger or ongoing problems, partially or fully automated solutions may be required. another important aspect to evaluate is the urgency of the problem. urgency could be determined by a number of factors: library administrators, whether access to content is broken, project deadlines, and more. this is subjective, but there could be a vast difference between needing to fix a typo in note fields vs. a typo in title fields. one record with a typo in the title could be fixed immediately, or it could be considered low urgency and put off to a later time. this is why it is so important to put the context, extent, and urgency together with importance. only then does it become clear when, and how, the analyst needs to address the issue. importance is also a subjective determination. that typo in the notes field might seem urgent to a doctoral student who could not find a source while working on their dissertation, but it may not seem urgent to the metadata librarian who already has a backlog of projects. this is where the question of importance comes in. in a large research university, a doctoral student missing out on a key source is pretty important, so while the analyst may disagree on the urgency, there may be agreement on the importance. practically speaking, once all of these aspects are taken into consideration, it becomes clear that there is just not enough time to tackle every problem under the sun. this is an important step in the process because none of these four areas are binary choices, and they all also influence one another. it is also important to remember that the person who reports a metadata issue may believe that the problem is important, urgent, and extensive; it is up to the analyst to see the problem in the context of other pending analysis work. the authors have never yet encountered a life-or-death situation involving metadata. tools solving a metadata problem may sometimes be as simple as adjusting a journal coverage date in the knowledge base, but more often it will involve using a software tool of some kind. the big three tools (external to library system tools) familiar to most librarians are marcedit, python, especially with pymarc and pandas libraries, and openrefine. all can be run on mac, windows, and linux systems. python also benefits from its widespread use in other domains. advanced metadata analyses and transformations can be performed in all of them, and their ability to do so is often only limited by the analyst’s creativity. marcedit is perhaps the best known tool for metadata transformations. it is a powerful tool with an easy to use interface, and can handle simple find and replace tasks as well as extremely complicated transformations. marcedit’s developer, terry reese, continues to actively expand the tool’s functionality to include working with linked data, library catalog apis, oclc apis, authorizing bibliographic headings, processing and outputting metadata in various serializations, among many other helpful functions. users can also develop “tasks” that meet their specific needs that work like scripts and can include many steps to transform marc fields and subfields, including using regular expressions. there are many, many examples of projects completed with marcedit, but one excellent example of using marcedit with xslt to perform advanced transformations is described in marielle veve’s article “from digital commons to oclc: a tailored approach for harvesting and transforming etd metadata into high-quality records.” (veve, 2016). openrefine presents metadata in an interface reminiscent of a spreadsheet, and offers many useful functions for metadata analysis and transformation. one of its most useful out-of-the-box functions for the metadata analyst is clustering, which is especially useful when “columns that have a reasonable amount of overlap but sometimes suffer from transcription errors, such as names” are at issue in a metadata analysis project.[2] python is excellent for efficiency’s sake, and for its ability to handle different kinds of data at the same time. with enough experience, an analyst using python can replicate functions from openrefine and marcedit while also bringing in additional functionality from python’s extensive, modular ecosystem. for instance, an analyst can work with a spreadsheet and marc records in the same script, and can extend that efficiency further if their library systems support apis by using those apis to fetch and update batches of records. another frequently-used python tool is jupyter notebooks. a notebook is a fully-functional python editor that can also record explanatory notes, instructions, and commentary as well as include visualizations. jupyter notebooks are extremely useful when learning python, and also for times when the analyst wants to see the results of each piece of code, and either review it or act on it before moving on. the authors have several processes we use regularly that are executed through jupyter notebooks rather than as standalone scripts, including processes to produce holdings reports for ingest by hathitrust, and to derive lists of print and electronic library holdings from vendor offer lists.[3][4] solving the problem when a metadata problem comes up, regardless of its nature, it is important to gather information to determine its context, extent, urgency, and importance using the techniques discussed above. when the analyst combines that information with knowledge of relevant tools, the solution often becomes clear. the important thing is to understand and use the tools at hand, while building knowledge and skill with other tools. here is a helpful analogy: there are multiple ways to dig a tunnel. you can dig a tunnel with a boring machine, or a spoon. one option might take longer than another, but you can usually get to the same place eventually. while we are capable of using python and other sophisticated tools to analyze and solve problems, if a tool like marcedit can already do what needs to be done, there is no need to write new code. the only time being unable to write code will prevent an analyst from successfully solving a metadata problem is when there is a need to change thousands of unique things across thousands of records, or when the task at hand is labor intensive and needs to be done more than once. in those situations, it is important to eventually find a programmatic solution. but being able to effectively use the tools with which the analyst is already familiar is more important than trying to learn how to use every tool, even if it means needing to run a long sequence of find and replace commands in marcedit. that will likely still be faster than learning python. conclusion the knowledge areas, techniques, and tools of metadata analysis discussed in this paper will no doubt sound familiar to many of the readers of this journal. besides providing a high-level overview of the knowledge, skills, and practices of metadata analysis for those who are already engaged in it, our hope is that this exploration will benefit those newer to the field who seek to learn metadata analysis skills. there is much overlap between metadata analysis and more discrete library functional areas of cataloging, e-resources management, and library systems management, among others. because the need for metadata analysis is not limited to one specialty within libraries, it has been difficult to pin down precisely what a metadata analyst does, and consequently for the library community to provide a concrete action plan for learning to do this kind of work. our own experiences confirm this. through our discussion of metadata analysis and its common tasks, approaches, and tools, we have sought to articulate a baseline of knowledge that can be used by library staff at all levels of the organization to build coherent development plans, write accurate job descriptions, and develop a common understanding of metadata analysis work. notes [1] dublin core metadata initiative. “metadata interoperability,” https://www.dublincore.org/resources/glossary/metadata_interoperability/ [2] phillips m, tarver h, frakes s. 2014. “implementing a collaborative workflow for metadata analysis, quality improvement, and mapping” the code4lib journal (23). https://journal.code4lib.org/articles/9199 [3] umnlibraries/hathitrust-inventories: https://github.com/umnlibraries/hathitrust-inventories [4] umnlibraries/holdings-from-isbns: https://github.com/umnlibraries/holdings-from-isbns references binici k. 2021. what are the information technology skills needed in information institutions? the case of “code4lib” job listings. the journal of academic librarianship, 47(3):102360. doi:10.1016/j.acalib.2021.102360 gonzales bm. 2019. computer programming for librarians: a study of job postings for library technologists. journal of web librarianship, 13(1), 20-36. doi:10.1080/19322909.2018.1534635 hall-ellis sd. 2015. metadata competencies for entry-level positions: what employers expect as reflected in position descriptions, 2000-2013. journal of library metadata, 15(2):102-134. doi:10.1080/19386389.2015.1050317 hannah m, heyns ep, mulligan r. 2020. inclusive infrastructure: digital scholarship centers and the academic library liaison. portal 20(4):693-714. doi:10.1353/pla.2020.0033 mathews jm, pardue h. 2009. the presence of it skill sets in librarian position announcements. college & research libraries, 70(3):250-257. doi:10.5860/0700250 nasig. 2019. nasig core competencies for electronic resources librarians. https://www.nasig.org/competencies-eresources ratledge d, sproles c. 2017. an analysis of the changing role of systems librarians. library hi tech, 35(2):303-311. doi:10.1108/lht-08-2016-0092 skene e. 2018. shooting for the moon: an analysis of digital initiatives librarian job advertisements. digital library perspectives, 34(2):84-90. doi:10.1108/dlp-06-2017-0019 thielen j, neeser a. 2020. making job postings more equitable: evidence based recommendations from an analysis of data professionals job postings between 2013-2018. evidence based library and information practice, 15(3):103-156. doi:10.18438/eblip29674 veve m. 2016. from digital commons to oclc: a tailored approach for harvesting and transforming etd metadata into high-quality records. the code4lib journal (33), 2016-07-01. https://journal.code4lib.org/articles/11676 xia j, wang m. 2014. competencies and responsibilities of social science data librarians: an analysis of job descriptions. college & research libraries, 75(3):362-388. doi:10.5860/crl13-435 about the author stacie traill (trail001@umn.edu) is metadata and discovery analyst at the university of minnesota libraries. martin patrick (patri299@umn.edu) is metadata analyst at the university of minnesota libraries. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – creating a library database search using drupal mission editorial committee process and structure code4lib issue 10, 2010-06-22 creating a library database search using drupal when florida gulf coast university library was faced with having to replace its database locator, they needed to find a low-cost, non-staff intensive replacement for their 350 plus databases search tool. this article details the development of a library database locator, based on the methods described in leo klein’s “creating a library database page using drupal” online presentation. the article describes how the library used drupal along with several modules, such as cck, views, and fckeditor. it also discusses various drupal search modules that were evaluated during the process. by danielle rosenthal and mario bernardo introduction the florida gulf coast university library is a mid-sized academic library serving a four-year public institution. we opened our doors and virtual presence in 1997 with roughly six database subscriptions, since then the collection has grown to over 350. the library also provides access to over 30,000 e-journals, maintains a 115-seat open computer lab, a 10-seat reference lab, and employs twenty staff members and twelve librarians. the interface for accessing databases has changed only a couple of times, with the last iteration in use for over eleven years. when the library was informed that the database locator service was being discontinued, we had to come up with a replacement in a relatively short time. with a small technology staff and  no funds set aside for this project, our solution had to be fairly easy to implement and, ideally, free. our search brought us to explore open source, something we yet hadn’t used. figure 1. database page, 1997 while we initially implemented a “dbl taskforce” consisting of various library faculty and staff to complete this project, ultimately it was just the assistant director of library computing and technology systems (cts) and the web librarian who were able to implement the solution. we have no programmers on staff. while the library can and does seek help from university technology on occasion, for the most part cts operates independently from the rest of the university’s it. the database locator that florida gulf coast university library relied on was a tool created by and hosted at the florida center for library automation (fcla).  fcla is a consortia library technology center which: “…provides state-of-the-art, cost-effective information technology to assist the libraries of the public universities of florida in their support of teaching, learning, research and public service. more specifically, implement and centrally support high quality computer systems that help the libraries acquire, manage and provide access to information resources…”.[1] the shared database locator was used by all state university system (sus) libraries at one point in time. the sus libraries can purchase and license databases through the fcla consortia if they wish and the fgcu library licenses many of our databases though fcla, but not all. figure 2. database locator pre-drupal instance in september 2007, the fgcu library was notified that fcla needed to free up the hardware and software licenses residing on the server hosting the database locator service.  we had to quickly find an alternative solution because we were the last university still using the outdated database locator software and there were no plans to keep it running for one university. all the sus libraries had created new alternatives themselves or had outsourced the work. while fcla was willing to help with a replacement dbl, we weren’t sure what we really wanted. the need to find a new dbl solution presented an opportunity to create something new, as well as deal with known issues, in hopes of creating a more usable system.  incorporation of additional features or fixes of known issues was secondary. in short the new dbl must: be searchable by title as well as keyword:  the outdated system was also only searchable by title, which worked well for experienced users; however our web usability study illustrated the need for a more context-driven interface. during the web study, students unfamiliar with the dbl were observed typing keywords in the search box instead of titles. [2] contain an alphabetical subject list: all 350+ databases must fit into an appropriate subject category or multiple categories for users who don’t know which database to use. users need to be able to browse a list of art databases or biology databases, etc. contain an a-z list: for users who don’t know the correct spelling of a particular database or for users who prefer to click instead of typing in a database name. be reliable and fast: database use at fgcu is very high and stability is a high priority. the locator page is consistently in the top 3 most accessed pages on the web site. users want to get into their database of choice quickly, without long search times. be easy to update: records need to be easy to add, delete, or edit with preferably no lengthy re-indexing time. our old system re-indexed every 12 hours. getting started we started by creating a “dbl taskforce.” the taskforce consisted of the assistant director of cts, the cts staff, the web librarian, head of technical services, our cataloging and metadata librarian, and the e-resource librarian. our initial work consisted of researching and contacting the other sus libraries and comparable institutions to see what they were using. we found, however, that many of them had programmers on staff or the funds to outsource the service. we had no programmers on staff and no funds available to outsource the service. a couple of institutions could not tell us how to re-create their systems because they did not know themselves. multiple authors and programmers had worked on their systems over the years, many of whom had since left.   budget considerations, in particular, led us to consider open source solutions. the development process our initial step was to examine the most common open-source cms products available. we then installed test instances on local hardware and begin evaluating how they would fit the needs of the dbl objectives already drafted.  among the cms’s installed and tested were joomla, drupal, and wordpress. while looking at drupal, we discovered a 20-minute online presentation by another librarian, leo klein, who used drupal to create a searchable system.[3] drupal, as described on drupal.org, is “open-source software distributed under the gpl (“gnu general public license”) and is maintained and developed by a community of thousands of users and developers.” from the presentation we saw how we could leverage the basic framework demonstrated and customize it to fit our needs.  we chose our existing microsoft iis server platforms rather than the more common windows apache msql php (wamp) or the linux based alternative (lamp) often used to host drupal. this choice was based on several factors: availability, established support, familiarity, and necessity to bring the product quickly to production. after deciding on the platform, we downloaded and installed drupal version 5.7. it should be noted that we chose drupal version 5.7 over version 6 (the latest version available at the time) because the marc import module (http://drupal.org/project/marc) was not available in version 6. we then proceeded to import our data and investigate and install various modules.  one of the most challenging aspects of this project turned out to be the preparation and importing of our existing 300+ database records. the records were stored in the older system housed at fcla called citation server, in a customized marc record format. the marc records had multiple fields with repetitive information, often with odd graphic symbols used as field fillers and delimiters. the unusual formatting of the records prevented them from being cleanly imported into drupal. it turned out to be easier to perform a partial import of some elements of our dataset, specifically the titles and description because those fields were clean. we then cut and pasted or manually entered the remaining data into the drupal content forms.  we can assume, as demonstrated with the clean import of the title and description fields, that the data import feature works well and other libraries may not encounter problems unless they are importing data from legacy systems such as the old citation server.  once the data is in the native drupal environment, it is easy to enter and update record data, but getting it there was not as easy as we were hoping.  in hindsight, we should have scrapped version 5.7 and installed version 6.0 since we weren’t using the marc module anyway. when creating web pages using drupal you must pick a theme or use the default. themes allow you to customize the look and feel of your pages and/or site. different themes have different functions and not all themes can be used with all versions. there are many themes (http://drupal.org/project/themes) to pick from and they are easily applied. we chose a basic garland default no frills theme and customized it with our university’s css, header, and footer. drupal organizes content by using content types and comes with only three standard types:  blog, page, and story. we needed to add a module that would enable us to create another content type. modules in drupal are plugins that extend its core functionality and are available at the drupal site. when you install or activate a module, the new functionality is automatically integrated into drupal, appearing as new menu items, tabs, or fields ready for you to use. leo klein’s presentation demonstrated how to use the content construction kit (cck) to create content types. cck allows you to add fields to an existing drupal content type and/or create new content types. we created a custom content type called eresource. the eresource content type needed fields that would be searchable and display all the information related to each database.  for consistency, the fields we chose were the same as our old system: title, url, description, coverage dates, and alternate title. figure 3. the cck module allows the creation of new "content types" we then enabled the optional core module taxonomy. taxonomy allows you to classify content into categories and subcategories, multiple lists of categories for classification (controlled vocabularies) and offers the possibility of creating thesauri (controlled vocabularies that indicate the relationship of terms), taxonomies (controlled vocabularies where relationships are indicated hierarchically), and free vocabularies where terms, or tags, are defined during content creation. we created a vocabulary called “subject” and added a list of terms. the terms are the names of each of our subjects (art, biology, etc.). figure 4. taxonomy screen: where categories are created and edited we then used the views module to display the front page list of subjects in alphabetical order since drupal’s default view is not alphabetical. we also had to create a view for the a-z list of databases by title. the combination of cck and views also enabled us to create a flexible and varied eresource content format to accommodate multiple descriptions and multiple urls per resource if necessary, as well as the assignment of multiple subject heading categories to each resource. there are many search modules available for drupal and we tried quite a few before settling on drupal’s own built-in search module.  some of the search modules we tested included: dynosearcho, faceted search, fuzzy search, live search, trip search, and a few others. many of these search modules were excellent for specific types of searches and content, but because our drupal dbl consists primarily of database names and descriptions, a customized search feature was not necessary. the built-in search module in drupal consistently achieved the best results display for our purposes.  we did enhance the built-in search module by activating the optional core feature called “search config” which allowed us to configure the display of the advanced search form. also, one of the release patches we installed improved partial word searching and provided better context delivery. we also tried the field indexer module, which provides a configuration page where the site administrator may select which fields to index. we decided not to use this module because we wanted all of our fields to be indexed for search. lastly, porter-stemmer was installed in an effort to improve search results for some librarians that were used to typing in shortened titles or acronyms. for example, typing in “pro” for “proquest”. since this module uses a stemming algorithm to reduce each word in the search index to its basic root or stem it is supposed to retrieve more relevant search results. however, we found that it generated too many results and that the database they were looking for, proquest, was getting lost in the result list. with the partial word patch applied, entering “proq” works just fine. acronyms that did not appear in the title or descriptions were added to the alt title field so that they could be indexed and searchable – the alt title search had already been a feature of our previous dbl; for example, “acs” for american chemical society. we also installed the fckeditor module, which is a visual html editor. it works by allowing drupal to replace text area fields with a wysiwyg editor. with the fckeditor, formatting your text is simple, whether you know html or not.  we also used the autologout and poormanscron modules. the autologout module does what its name implies. it allows the site administrator to set the time that will automatically logout users after a certain amount of inactivity. poormanscron module is used to run basic cron jobs using normal browser page requests (as opposed to having to set up a crontab), which are self-maintenance tasks like removing expired session information, and updating indexes and tables. progress around the time the alpha drupal instance was created, we were notified that the old dbl was no longer in jeopardy which resulted in the project being put on hold and the dbl taskforce being disbanded.  we continued working on the drupal solution, however, because we knew we were onto something.  if we could get it to work as envisioned, then at least we would have a backup system in the short term, as well as a long-term opportunity to promote a more up-to-date context-based search system. as it turned out, a few months after we completed this project, we learned that the fcla dbl tool was really going away this time. that same week, fcla had a very serious hardware-driven outage and the existing fcla hosted dbl crashed, something that rarely (if ever) happened. fortunately, our own drupal dbl solution was ready and just needed to be made live. so we presented the option to the head of collections and technical services and were given the go ahead to make the drupal dbl live. the data was still fairly up-to-date because not many records had changed since the last import. we came to the realization that, in all those years, there was no backup to the dbl system in place.  if it weren’t for the drupal-based dbl, we would not have had copies of the data anywhere. fcla probably had a copy, but they had other priorities, like bringing the extensive florida sus systems back online.   having a local solution gave us the ability to bring an important library service back on line quickly. figure 5. new database page using drupal, 2010 looking at our original objectives: objective 1: be searchable by title as well as keyword using the search box that comes with drupal search to search by database title was working just fine. it was the partial word search that was giving us some relevancy problems. we found a partial word search patch on the drupal site and it was applied to the drupal search module. applying the patch produces better context results to title and keyword searches.  other patches and the application of the search-config were also applied to customize search box labels to be more descriptive and similar to previous dbl interface. objective 2: contain a subject list we met this objective by creating categories as our subjects:  biology, health professions, art, etc.  and then assigning  the databases to the appropriate categories. databases can be assigned to multiple categories, which is essential since so many databases cover more than one subject. from a maintenance point of view we don’t edit this list very often. when a new database gets added to drupal it is entered manually and then assigned to a category or categories. objective 3: contain an a-z list it is easy to create an a-z list in drupal using views. objective 4: be reliable & fast this solution has turned out to be reliable. we have had no failures or slow performance since the day we went beta. objective 5: easy to update – records need to be easy to add, delete, or edit with preferably no lengthy re-indexing time. drupal is very fast and easy to update. data updates are performed through to the drupal server by logging into administration, choosing the database to edit, making the necessary changes, and hitting “publish”.  re-indexing is configured to occur within minutes.  we backup the drupal instance, but also created a simple flat html file that consists of the database name and the connection link. the html file resides on our staging web server, and we keep another copy on a share drive. both get updated manually whenever the drupal dbl records are changed in any way. in order to upgrade to a newer version of drupal, we need to determine whether all our current modules and configurations are supported in the new drupal version.  we are currently evaluating and anticipate upgrading sometime in 2010. for consideration if you can live with not knowing exactly how items are ranked by drupal’s search feature, then use the core drupal search.  while records display appropriately, some items appear farther down the list than would be expected. we would like to know how to get certain records to display higher or lower in the results list, but the mechanism of this is still a bit of a mystery.  challenges include learning how to tweak descriptions so they appear higher on the results list or finding a different or customizable search module.  this is a weakness of all the search modules we tried. there appears to be no good way to completely control the granularity of relevancy results or to produce search results based on a mixture of hybrid factors, such as a combination of relevancy and preset manual index indicators.  this is one of the reasons we are very curious to experiment with drupal 6 and the beta of version 7 to determine if search results can be better managed. notes florida center for library automation. (2009). http://fclaweb.fcla.edu/ rosenthal, d. (2009). fgcu library website usability study. unpublished manuscript. http://library.fgcu.edu/webanalysis/reports/usabilitystudyreport.pdf klein, l. (05/15/2010). screencast: creating a library database page with drupal. http://chicagolibrarian.com/node/262 about the authors danielle rosenthal is the library web development and science liaison librarian and mario bernardo is the assitant director of library computing and technology systems at the florida gulf coast university. subscribe to comments: for this article | for all articles 3 responses to "creating a library database search using drupal" please leave a response below: andrea carter, 2010-06-24 congrats! what an accomplishment. leo robert klein, 2010-09-13 hey, great article! thanks for the mention! screencast on using drupal mentioned in code4lib journal | chicago librarian, 2013-06-02 […] "creating a library database search using drupal" […] leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – beyond opac 2.0: library catalog as versatile discovery platform mission editorial committee process and structure code4lib issue 1, 2007-12-17 beyond opac 2.0: library catalog as versatile discovery platform north carolina state university has developed an application programming interface (api) “platform”, called catalogws, to provide web service access to catalog search and availability services. this project was motivated by the realization that the discovery of library collections should not be limited to a single catalog application, and such a platform could support the efficient creation of novel interfaces based on consistent services. some technical discussion of the catalogws architecture is provided, including a technical description of web service protocols implemented. several applications providing discovery in novel contexts have already been developed based on catalogws, and are described in some depth. catalogws has helped create a culture of experimentation and enabled a larger group of staff to work with library catalog data and services in new and interesting ways. by tito sierra, joseph ryan, and markus wust introduction many libraries, with the goal of modernizing their web presence, are racing to deploy a “next generation catalog.” next generation catalog applications typically offer a mix of these features: faceted navigation, keyword searching, relevance-ranked search results, “did you mean?”-style search revisions, item recommendations, rss feeds, and mechanisms to collect and display user feedback [1(coins)]. these “opac 2.0" efforts to replace or upgrade legacy opacs with more powerful alternatives will no doubt improve the overall catalog experience for many library users. unfortunately, the gains from these efforts are limited because a single catalog application cannot be optimized for all library users and uses. consider, for example, the two basic types of searches a user might perform in a library catalog: known-item and exploratory. in a known-item search the user typically has a specific item in mind; the goal is to acquire information about this item as quickly as possible. in an exploratory search the user has a topic in mind and the goal is to acquire a list of items related to the topic. an interface optimized for a known-item search would likely take advantage of the information the user knows about the item, thus emphasizing bibliographic metadata such as the item’s title or author. optimizing for an exploratory search would likely emphasize descriptive metadata that items share in common, such as subject headings or user-defined tags. it would be challenging, if not impossible, to optimize a single library catalog application for both of these common use cases. this conflict begs the question: why should the discovery of library collections be limited to a single catalog application? given the resources required to keep library catalog data accurate and up-to-date, libraries ought to explore methods for integrating this data into relevant external applications. lorcan dempsey, of oclc, refers to this process as “lifting out the catalog discovery experience.” he writes: as we work to aggregate supply… so we must work to place these resources where they will best meet user needs. in this process, discovery of the catalogued collection will be increasingly disembedded, or lifted out, from the ils system, and re-embedded in a variety of other contexts — and potentially changed in the process. [2] realizing dempsey’s vision requires increased flexibility in how we provide programming interfaces to our catalog data. to enable creative use of catalog data and improve interoperability with external systems, we need to think beyond the application-specific enhancements that characterize current opac 2.0 efforts. one strategy is to think about our catalogs as a platform that can support many discovery applications, not just the opac. we believe this approach shows great promise for libraries looking to enhance long-term end-user discovery and use of library collections. catalogws netscape co-founder marc andreessen’s definition of “platform” neatly describes the fundamental difference between platforms and applications: … a “platform” is a system that can be reprogrammed and therefore customized by outside developers — users — and in that way, adapted to countless needs and niches that the platform’s original developers could not have possibly contemplated, much less had time to accommodate. in contrast, an “application” is a system that cannot be reprogrammed by outside developers. it is a closed environment that does whatever its original developers intended it to do, and nothing more. [3] in early 2007 the ncsu libraries began to experiment with the “catalog as platform” approach through an initiative called catalogws. the original designers of catalogws, tito sierra and emily lynema, were members of the team that implemented the first endeca-powered opac [4] in january 2006. following the release of the new opac, the implementation team faced a list of post-launch feature enhancements. two of these enhancements were rss feeds for catalog searches, and the integration of catalog search results into quick search [5], our library website search tool. rather than modify the opac application, the designers decided it would be better to create a separate application programming interface (api) to the search indices used by the opac. once in place, the api would make it easier to build the desired feature enhancements, as well as enable more versatile catalog data use in the future. after a few weeks of development, the catalogws api was born. catalogws currently provides two functions: the “search” service and the “availability” service. the search service returns item and facet values for a given search query, whereas the availability service returns item availability information for a given isbn. documentation for both services is available on the catalogws project homepage [6]. to date, most of our catalogws-powered applications have used the search service. the scope of the api is necessarily limited because it uses search indices, rather than querying the full set of data stored in the ils. for example, the api does not expose all the data in the marc record for our catalog items, since not all marc fields are indexed by our search server. the designers decided to use search indices as a data source because they provide easy access to normalized data. using search indices also made the data modeling process much easier because the indices already captured and consolidated the most useful catalog data for end-user application development. this convenience comes at the price of limiting the type of applications that can be built with the api. since the api is read-only, we could not build tools for modifying bibliographic data or updating the inventory status of items in the catalog. these ils-oriented functions were beyond the scope of what the designers intended to accomplish with the api. figure 1: ncsu libraries catalog architecture [view full-size image] despite the scope limitations, we have realized several benefits by using our existing endeca-generated indices as a data source. architecturally, search indices are designed to accommodate a high volume of low-latency requests. building on search indices has also enabled us to include search-specific features in our applications, such as faceted data and spelling suggestions. the earlier endeca implementation decision to include holdings information in the search indices proved to be very useful for the api. it means we are able to include inventory data in our applications, such as the name of the library that holds the item, the call number of the item, and even the current availability status of the item since the last index update. the technical implementation of the api was modeled after the rest web apis implemented by companies such as yahoo! [7], facebook [8], and amazon.com [9]. requests to the api are made through the http get protocol. here is an example of a basic search request: http://www.lib.ncsu.edu/catalogws/?service=search&query=usability the search service supports many features one might expect from a search api, such as the ability to limit the number of results returned, and ability to change the sort order of returned results. by default, the service queries our catalog’s “keyword anywhere” index, which looks for keyword matches across many fields in the catalog record. one can also search specific fields such as title, author, or isbn/issn. it is even possible to supply a null keyword to browse the entire catalog. including facet results in the response makes it possible to query specialized slices of the catalog using shared facet attributes, such as commonly shared subject headings, material formats, or author names. catalogws responses are available in a variety of output formats, with a custom xml format as the default. the decision to define a custom base xml format for the search and availability services was motivated by several factors. first, the api designers believed it would be useful to include as much of the data from the search indices as possible into a single search request, rather than requiring multiple requests to the same data source. for example, the search service returns several categories of response data including search metadata (e.g. total results, spelling suggestions), bibliographic metadata (e.g. item title, author name), holdings data (e.g. library location, call number), and facet results. although there are many existing standards such as marc-xml, mods, and dublin core that could describe some of this data in varying levels of granularity, no existing standard seemed suitable for such a heterogeneous collection of data. second, the api was intended for use by developers at ncsu, rather than for programmatic external application consumption, so cost-benefit analysis of distilling the base api profile into one or more specific standard formats weighed in favor of a custom approach. below is an abbreviated xml response for the earlier example. a cached version of the full response is available here. <?xml version="1.0" encoding="utf-8"?> <searchresponse xmlns="http://www.lib.ncsu.edu/catalogws/1.0"> <requesturi>http://www2.lib.ncsu.edu:9921/catalogws/?service=search&query=usability</requesturi> <cataloglink>http://www2.lib.ncsu.edu/catalog/?ntt=usability&ntk=keyword&n=0&nty=1</cataloglink> <searchinfo> <query> <terms>usability</terms> <key>keyword</key> </query> <totalresults>477</totalresults> <offset>0</offset> <itemsperpage>30</itemsperpage> </searchinfo> <item id="1791483"> <cataloglink>http://catalog.lib.ncsu.edu/web2/tramp2.exe/do_ccl_search/guest?setting_key=files&record_screen=record_brief.html&*search_button=keyword&servers=1home&index=ckey&query=1791483</cataloglink> <title>cost-justifying usability : an update for an internet age</title> <pubdate>c2005.</pubdate> <format>book</format> <isbn>0120958112</isbn> <holdings institution="ncsu"> <library name="d.h. hill library"> <holdingsitem> <callnumber>qa76.9 .u83 c67 2005</callnumber> <location>stacks (6th floor)</location> <status>available</status> </holdingsitem> </library> </holdings> </item> <!-repeating <item> elements--> <facet id="format"> <title>format</title> <cataloglink>http://www2.lib.ncsu.edu/catalog/?ntt=usability&ntk=keyword&n=0&nty=1&ne=200043#200043</cataloglink> <value> <requesturi>http://www2.lib.ncsu.edu:9921/catalogws/?service=search&query=usability&n=206437</requesturi> <cataloglink>http://www2.lib.ncsu.edu/catalog/?ntt=usability&ntk=keyword&n=206437</cataloglink> <title>book</title> <count>461</count> </value> <!-repeating <value> elements --> </facet> <!-repeating <facet> elements --> </searchresponse> although the base xml format is custom, the catalogws search service has built-in support for returning results in several standard output formats. search results can be returned in rss 2.0, opensearch, and json formats. additionally, the api supports an optional “style” parameter that enables requests to specify a path to an xsl stylesheet. when the style parameter is passed, catalogws provides server-side xsl transformation services for the request. the style parameter makes it possible for developers to create interactive catalogws-powered web applications using only xsl stylesheets. current applications since going into production in january 2007, catalogws has provided the infrastructure for a variety of applications at ncsu libraries. some of the applications developed to date include specialized or experimental catalog interfaces, while others are innovative collection promotion display tools. below, we provide a few examples of how we have used catalogws thus far. to begin, we will discuss the mobilib catalog, an application designed for known-item searches in a mobile context [10]. some example use cases for this application are looking up call numbers and checking the availability of items in the library bookstacks. unlike our information-rich opac, the mobilib catalog is a barebones search tool that allows users to look up an item by keyword, title, author, or isbn. given the importance of item location and item availability implied by the mobile use context, the application emphasizes the display of inventory data, such as call numbers, and enables users to limit their search to items that are not checked out. figure 2: mobilib catalog [view full-size image] developing mobilib presented several challenges. the first was getting the application to function and display properly on a variety of mobile device platforms including handheld pcs, web-enabled cell phones, and pdas. the smaller displays on these devices limit the amount of information that can be displayed in each view. mobile devices also have speed and cost-related bandwidth issues. these unique constraints forced us to rethink the catalog search experience from the ground up. while the mobilib catalog is optimized for known-item searches, facetbrowser is an experimental catalog interface that was designed to emphasize facets for use in exploratory searches [11]. facetbrowser places facets persistently at the center of the application’s interface. this design encourages users to browse the entire catalog at once, promoting use of facets as a way to explore or navigate the results set at every level. we believe this approach has the potential of exposing users to items in the catalog that they otherwise might not have discovered in a more conventional catalog search application. figure 3: facetbrowser [view full-size image] besides serving as an experimental catalog interface, facetbrowser has also provided us with an opportunity to promote library collections in new ways. as part of our recently opened ncsu libraries learning commons, several large screen displays were installed in the new space to promote a variety of library services. some of these displays are dedicated to showing “bookwalls,” which are displays of book cover images for a thematic collection of books from the catalog. facetbrowser provides the means for library staff to quickly generate these bookwalls. the process involves applying one of several built-in output styles to an editorially selected list of items in facetbrowser. the application automatically generates the html used to display the bookwalls on our large screen displays, enabling library staff with limited technical skills to promote specialized collections of books in a visually appealing way. figure 4: facetbrowser bookwall output display [view full-size image] in addition to bookwalls created by facetbrowser, we use catalogws to promote library collections in other ways. the new books bookwall is similar to the staff created thematic bookwalls, except that the focus is on featuring books recently added to the collection [12]. this bookwall is different because the items it displays are not hand selected. instead, a scheduled catalogws request retrieves the 300 newest items, with the goal of providing a peek at what’s new at the library. the new faculty books display is a variant on the new books bookwall [13]. this application focuses on featuring new books published by ncsu faculty. although the user interface is slightly different, the underlying technical implementation is the same. the new faculty books display adds an additional filter constraint to items authored by ncsu faculty. figure 5: new faculty books list [view full-size image] closing thoughts returning to andreessen’s definition of a “platform,” we see that the final part of his definition is that a platform enables outside developers to adapt the system “to countless needs and niches that the platform’s original developers could not have possibly contemplated, much less had time to accommodate.” this observation mirrors the ncsu libraries’ experience perfectly. although the original motivation for the api was to enable development of some opac feature enhancements, we have been able to use the api to create completely new applications that exist independently of the opac. the platform approach has provided us with two distinct benefits. first, the api has reduced catalog application development time because it exposes data in an easy-to-use format. a second benefit is that our api has enabled versatile use of catalog data, as illustrated by the applications we have built so far. as we reflect on our experience to date, we find that the collection of data exposed by our api could be more comprehensive. we encourage those who are developing a similar platform to cast a broad net when deciding what content to include in their api. although our lightweight approach has worked well for us, we acknowledge the limitations of relying on search indices as a catalog data source. those interested in learning from a more ambitious “catalog as platform” approach should look at the talis platform [14], which supports a wide range of interfaces and technologies. overall, we believe the time invested in developing our catalog platform was worth the effort and should continue to pay dividends in the near future. we never anticipated that catalogws would serve as a basis for such a broad range of applications. in our organization, it has fostered a culture of experimentation and enabled a larger group of staff to work with library catalog data in new and interesting ways. although we are in the early stages of working with our platform, we recognize there is a large amount of untapped potential. we would like to see other institutions experiment with the platform approach and we welcome the opportunity to learn from other libraries about how to fully exploit the value of catalog data beyond the opac. notes marshall breeding. next-generation library catalogs. library technology reports 43(4). july/august 2007.(coins) lorcan dempsey. the library catalogue in the new discovery environment: some thoughts. ariadne 48. july 2006. http://www.ariadne.ac.uk/issue48/dempsey/ http://blog.pmarca.com/2007/06/analyzing_the_f.html http://www.lib.ncsu.edu/catalog/ http://www.lib.ncsu.edu/dli/projects/quicksearch/ http://www.lib.ncsu.edu/catalog/ws/ http://developer.yahoo.com/search/ http://wiki.developers.facebook.com/index.php/api http://aws.amazon.com http://www.lib.ncsu.edu/m/catalog/ http://www.lib.ncsu.edu/facetbrowser/ http://www.lib.ncsu.edu/display/bookwall/eboard.php http://www.lib.ncsu.edu/display/facultybooks/list http://www.talis.com/tdn/platform/ about the authors tito sierra is assistant head for digital library development at ncsu libraries where he leads a small research and development team that builds new digital library applications. joseph ryan is an ncsu libraries fellow in the digital library initiatives and administration departments. markus wust is an ncsu libraries fellow in the digital library initiatives and special collections departments. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – visualizing library statistics using open flash chart 2 and drupal mission editorial committee process and structure code4lib issue 19, 2013-01-15 visualizing library statistics using open flash chart 2 and drupal libraries continue to need to demonstrate their value to stakeholders, and while statistics alone do not represent value, they are an important element. we found ourselves, and our stakeholders, uninspired by our infrequently updated bulleted list of statistics on our website and so set out to create a more dynamic and visually appealing look at our statistics. this article outlines how we used our content management system, drupal, open flash chart and custom programming to convert library statistics into flash charts, including how to populate the graphs with dynamic data from external sources. the end result is our library statistics dashboard (http://library.uncw.edu/facts_planning/dashboard) that visually demonstrates the use, activity and resources in the library via interactive and visually interesting graphs. by laura k. wiegand and bob humphrey libraries are experts at collecting statistics. we have complex systems that collect data on transactions, we have analytics that track online visits, we have gate counters that record visits, we have forms to keep track of our interactions with users, we get reports on pdf downloads and how many times our computers get logged into. but what to do with all this data? we often use it internally for decision-making or reporting, but what about communicating these often staggering numbers to our stakeholders and clientele? like many libraries, uncw randall library had a page on our website under “about us” that listed out, in bulleted sentences, some of our more compelling numbers (e.g., “9,185 people received instruction”). but in today’s world of instant metrics and focus on return on investment (roi), these annually updated lists of numbers were uninspiring. in addition, they didn’t show context over time, neither reflecting the academic year cycle nor the steady increase year to year. in response to this less than helpful list of statistics, our university librarian asked that we create a “statistics dashboard” to visually demonstrate activity in the library. other libraries have built dashboards, both external [1] [2][3] and internal[4], used data visualization for art and physical displays displays (such as the well-known ticker display of check out data at the seattle public library) [5], produced kicked up marketing materials [6], and employed it for decision-making [7]. in fact, in an era of increasing availability and flexibility of massive amounts of data, employing effective data visualization has become increasingly important [8]. on the spectrum of data visualization, our project was relatively simple. our goal was to present traditionally collected statistical data about the library in a graphical and dynamic manner. this “dashboard” was to be a piece of the newly created “facts and planning” (http://library.uncw.edu/facts_planning ) section of our website designed to provide the university community with information about our resources and their stewardship. in their article “statistical graphics: making information clear – and beautiful” niemi and gelman outline the qualities of a “well-designed graph” [9]. they write when planning visual representation of statistics or data, the first questions to ask are “who is your target audience? what are you trying to show?” in our case, our target audience was university administration, our users and possibly librarians from other libraries. we were trying to show the large volume of transactions, visits, instruction and usage increasing over time and cycling through the academic year. next they recommend “avoid distracting elements. use informative colour. keep the figure simple.” we attempted to do this by styling the graphs to match our website, creating simple graphs using proven graphing techniques (line, bar, area) and having exact numbers remain hidden until the user hovered over the graph. in terms of infrastructure, we wanted the dashboard to be fully integrated with our drupal 6 website in a way that both allowed for the reuse of content and harnessed existing technology. additionally, we are a medium-sized academic library with limited staff resources, meaning that building a complex, custom system was not an option. since many other libraries utilize drupal, and likewise have limited resources, we thought we would share the steps for setting up a simple dashboard in drupal. tools for building the dashboard to create the dashboard, we chose open flash chart, a free, open source charting package available at http://teethgrinder.co.uk/open-flash-chart-2/. this software creates beautiful animated charts in a variety of styles, including line charts, bar charts, stacked bar charts, and pie charts. the project has adequate online documentation and the charts can be built and customized using php. there are several drupal modules that can be used to integrate charts. we chose the open flash chart 2 api module (http://drupal.org/project/ofc_api) based on its ease of use. the procedure for installing this module was similar to that for any other drupal module except for the addition of a couple of extra steps. once the module had been saved in the drupal installation sites/all/modules folder, then the open flash chart (ofc) files needed to be added to the ofc_api folder. we downloaded ofc from http://sourceforge.net/projects/openflashchart/files/open-flash-chart/version%202%20lug%20wyrm%20charmer/ and extracted the contents of the .zip file. then we moved the ofc files into the module as follows: the open-flash-chart.swf file was copied  from the ofc package to the ofc_api module the php-ofc-library folder was copied from the ofc pagkage to the ofc_api module the swfobject.js file from the js folder within the ofc package was copied to the ofc_api module when we were finished moving files, the dirctory of the ofc_api module looked like this: ofc_api |-php-ofc-library [folder] |-ofc_api.info |-ofc_api.install |-ofc_api.module |-open-flash-chart.swf |-swfobject.js the next step was to configure the module at [website]/admin/settings/ofc_api. we checked the options to use swfobject to embed charts and to use an external library and set the path to the external library file as ‘php-ofc-library/open-flash-chart.php’. figure 1: open flash chart api configuration page and block display setup we chose to put our graphs inside of individual blocks to make it easy to reorder, update, add and remove the various charts that we would be creating over time. since the randall library website already had more than 200 blocks defined (future clean-up project!), there was a real possibility that our new dashboard blocks were going to be difficult to manage. to deal with this, we took some extra steps at the beginning of the project to help organize the new blocks being created. drupal makes it easy to define new display regions that can be incorporated into the theme being used, so we decided to add a new ‘dashboard’ region to our theme. this was done by including a new element in the regions array defined in our theme .info file, as shown below: regions[left]          = left sidebar regions[right]          = right sidebar regions[navbar] = navigation bar regions[content_top] = content top regions[content_bottom] = content bottom regions[dashboard]      = dashboard regions[header]         = header regions[footer]         = footer regions[closure_region] = closure then we had to specify exactly where the new region should appear on a web page. to do this, we modified our theme page template page.tpl.php to display the content in the dashboard region just above any content that would normally be placed in the content-area of a page. <?php if ($dashboard): ?> <div id="dashboard" class="region region-dashboard"> <?php print $dashboard; ?> </div> <!-/#dashboard --> <?php endif; ?> <div id="content-area"> ... </div> all blocks created for the dashboard page were then assigned to the new dashboard region. figure 2: dashboard blocks next we created a page node consisting only of a title and url alias. this provided us with an empty container that we could then fill up with drupal blocks, each one holding a different chart or graph to be displayed in our dashboard. creating a dashboard chart in a block we were then ready to create our first chart, a line graph showing the weekly number of logins to the computers in our learning commons. the process for creating a graph consisted of the following steps: creating and configuring a new block writing a short php script to specify all the parameters for the chart. we set the new block to display on the container page created earlier, and changed the input format to php code. if this input format is not an option to you, it will need to be enabled as an available text format at [website]/admin/settings/filters/list. also, the user creating the block needs to be assigned to a role that has permissions to use the php code input format (grant with care!). we then proceeded to write the code that would display the desired chart. the resulting script consisted of two parts. the first part specified the data that was going to be displayed in the graph, i.e. the number of logins and the week ending date. the second part described the appearance of the chart, including each of its various elements, such as title, data points, lines, and axis. to accomplish the first part of the task, a query was made against the database that contained the login data, returning the following information: week ending number of logins 1-7-12 180 1-14-12 201 1-21-12 3318 1-28-12 4284 2-4-12 5365 2-11-12 6281 2-18-12 6528 2-25-12 6460 this data was then placed into two arrays at the beginning of the script: <?php // data and labels $data = array(180,201,3318,4284,5635,6281,6528,6460); $labels = array('1-7','1-14','1-21','1-28','2-4','2-11','2-18', '2-25'); we then proceeded to the second part of the task, defining all the various elements of the chart. the first item specified was the chart title: // chart title $title = new title( '2012 learning commons computer usage, by week' ); the next step was to describe the appearance of the data points. there are a couple of different approaches to figuring out how to create data points or any other ofc chart element. the open flash chart website contains dozens of code examples for building every kind of chart. if you are familiar with php, it’s not overly difficult to find a chart you like and use the code as a guide for creating your own graph. another method that might be useful, especially for those who are comfortable with object oriented programming, is to make use of ofc data structures class documentation at http://teethgrinder.co.uk/open-flash-chart-2/doxygen/html/annotated.html. studying the class hierarchy, we were able to see that there were 9 different classes for representing data points, each of them a subclass of the dot_base class. for this particular chart, we chose the hollow_dot class, which had no particular properties of its own, but inherited all of the properties of its base class, including size, halo_size, colour, and tooltip. specifying the appearance of our data points was simply a matter of creating a new hollow_dot object and setting its various classes. // define data point appearance $d = new hollow_dot(); $d->size(4)->halo_size(1)->colour('#006666')->tooltip("week ending #x_label# #val# logins" ); the lines were defined in a similar way: a line object was created. the data property of the line was set using the $data array defined at the beginning of the script. the dot style property was set using the data point style defined above. the width and colour properties were set. // define line appearance $line = new line(); $line->set_values( $data ); $line->set_default_dot_style($d); $line->set_width( 4 ); $line->set_colour( '#006666' ); the next task was to define the horizontal x-axis. we followed the same steps as we did in creating the data points and lines. referring to the open flash chart documentation for guidance, we created an x_axis object and then specified its properties. note that the $labels array defined at the beginning of the script was used to set the labels for this axis using the set_labels_from_array() method. // define x-axis $x = new x_axis(); $x->colour('#dddddd') ->grid_colour('#dddddd') ->tick_height(5) ->stroke(3); $x->set_labels_from_array($labels); before creating the vertical y-axis, we needed to determine the scale of the values being represented. according to the data table shown above, the greatest number of logins (6528) occurred during the week ending february 18. therefore, 7000 was selected as the maximum value for the y-axis. this would be large enough to accommodate the full range of values that need to be displayed, and small enough so that the line graph created would expand to use up most of the available chart space. it was also decided that the graph would have horizontal gridlines representing 1000 logins, 2000 logins, and so on, up through the maximum value of the axis. to accomplish this, three range variables were set as follows: // set the range of values for the y-axis $low_value = 0; $high_value = 7000; $step_value = 1000; with the range of values accounted for, the rest of the y-axis could then be defined: // define y-axis $y = new y_axis(); $y->set_stroke( 3 ); $y->set_colour( '#dddddd' ); $y->set_tick_length( 5 ); $y->set_grid_colour( '#dddddd' ); $y->set_range( $low_value, $high_value, $step_value); at this point, all the different elements of the chart had been specified.  the last remaining task was to create a chart object and add to it all the various objects representing the chart elements. // create chart and add all elements to it $chart = new open_flash_chart(); $chart->set_title( $title ); $chart->add_element( $line ); $chart->set_bg_colour('#eeeeee'); $chart->set_x_axis( $x ); $chart->set_y_axis( $y ) we finished the block off with a little bit of text providing some additional information about the chart. // add text below the chart print '<div class="dash-chart-text">'; print ofc_api_render($chart, 400, 250, array('wmode' => 'opaque')); print 'the approximately 100 computers in the learning commons are logged onto several thousand times a week; many students use them daily.'; print '</div>'; ?> all of this code was entered into the block body field of the block input form.  the block was then saved and enabled for the dashboard region of the site display. figure 3: library statistics dashboard with open flash charts graph automatic data updates at first we kept the chart current by manually updating the $data and $labels arrays on a weekly basis with new information. but as soon as time permitted, code was added to obtain this information directly from the database where it was stored, thus eliminating the need for this weekly chore. // data and labels $data_r = array(); $labels_r = array(); // connect to the database and query it $conn=mysql_connect('[server]',[user name],[password]); @mysql_select_db([database name]); $sql =  [sql code to retrieve number of logins and date from the database for the previous 16 weeks]; // populate the data and label arrays with information from the database; // keep track of the largest value being used $high_value = 0; $result = @mysql_query($sql); if ($result) { $x = 2; while ($row = mysql_fetch_array($result, mysql_assoc)) { $value = intval($row['logins']); $data_r[] = $value; $high_value = ($value > $high_value) ? $value : $high_value; $rem = $x % 2; if (!$rem) { $labels_r[] = substr($row['date'], 5); } else { $labels_r[] = ''; } $x++; } } // reverse the arrays so that they display in the correct order on the chart $data = array_reverse($data_r); $labels = array_reverse($labels_r); the new code created a connection to the database and retrieved the number of logins summarized on a weekly basis for the previous 16 weeks. as each record was returned, the arrays $data_r and $labels_r were updated with the information. since the records were retrieved in the opposite sequence in which they needed to be displayed, the arrays were reversed before building the chart. an additional consideration in automating the data retrieval was to programmatically determine the correct scale for the y-axis of the graph. as each record was retrieved from the database, the number of logins was compared to the value of the previous records, and $high_value was updated if the number of logins was the largest value returned up to that point. then, when all the records had been read, $high_value was adjusted up to the next hundreds value. $high_value was then divided by 5 to determine the other amounts to be displayed on the y-axis. // set the range of values for the y-axis $low_value = 0; $places = strlen($high_value) 2; if ($places > 0) { $high_value = ceil($high_value / pow(10, $places)) * pow(10, $places); } $step_value = $high_value / 5; discussion our goal for this project was to replace our traditional statistics page with a more compelling display, and to demonstrate the scope of activity in the library to our users and stakeholders. we feel that we are on our way towards meeting those goals with the current implementation. our previous list of statistics has indeed been replaced by attractive, informative graphics. our university librarian has included demonstration of the dashboard in presentations to administration. the charts are easy to update, harness existing technologies, are integrated with our website and can easily connect to external data sources for hands-off updating. however, we have fallen short in some ways. for example, the number of charts we have set up is limited, and only a handful of them are dynamically updated. we are planning to continue to increase the number of graphs, specifically increasing the number of graphs that utilize dynamic or automatically updated data. second, the metrics we are currently displaying are relatively simple and consist of statistics traditionally reported by libraries (i.e, number of information literacy sessions per month). these traditional counts over time may not be the most interesting way to show the impact of our activity. a better demonstration might be a chart that compared information literacy sessions to the number of librarians and the number of number of students at uncw. enhancing the way we think about data and statistics, and then turning those into charts, would be another way to meet our goals. finally, the statistics dashboard is relatively buried on our website; this is somewhat intentional because it is auxiliary to prime functions on our website. however, as we update our website we will be considering ways to further integrate or display the dashes so that our users can get a glimpse of the activity in the library in relevant contexts. references [1] dashboard_beta :: dashboard information [internet]. providence(ri): brown university library; [cited 2012 oct 31]. available from: http://library.brown.edu/dashboard/info/ [2] dashboard [internet]. [updated 2010 sept 16]. indianapolis (in): the trustees of indiana university; [cited 2012 oct 21]. available from: http://www.ulib.iupui.edu/dashboard [3] library dashboard [internet]. [updated 2012 june].  new york(ny): libraries of the metropolitan museum of art [cited 2012 oct 31]. available from: http://www.libmma.org/watstat/ [4] morton-owens, e, hanson, k. 2012. trends at a glance: a management dashboard of library statistics. information technology and libraries [internet]. [cited 2012 oct 31]. 31(3): 36-51. available from: http://ejournals.bc.edu/ojs/index.php/ital/article/view/1919/pdf [5] legrady, g.  2005. “making visible the invisible” seattle library data flow visualization.  in: digital culture and heritage.  proceedings of ichim05 sept. 21-23, 2005.  paris(france): archives & museum informatics europe (amie).  available from:  http://www.archimuse.com/publishing/ichim05/legrady.pdf [6] south dakota public libraries data digest 2011. pierre(sd): south dakota state library. [cited 2012 oct 31]. available from: http://library.sd.gov/sdsl/publications/doc/rpt-datadigestpublib2011.pdf [7] visualizing library data [internet]. raleigh(nc): ncsu libraries. [cited 2012 oct 31]. available from: http://www.lib.ncsu.edu/dli/projects/dataviz [8] davis, hilary. 2009. not just another pretty picture. in the library with the leadpipe [internet]. [cited 2012 oct 31]. available from: http://www.inthelibrarywiththeleadpipe.org/2009/not-just-another-pretty-picture/ [9] neimi, j, gelman, a.  [2011] statistical graphics: making information clear — and beautiful. significance. 8(3):135-137. about the authors laura k. wiegand (wiegandl@uncw.edu) is an information systems librarian at the university of north carolina wilmington. bob humphrey (humphreyr@uncw.edu) is the library web & applications developer at the university of north carolina wilmington. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – static web methodology as a sustainable approach to digital humanities projects mission editorial committee process and structure code4lib issue 60, 2025-04-14 static web methodology as a sustainable approach to digital humanities projects the web platforms adopted for digital humanities (dh) projects come with significant shortand long-term costs—selecting a platform will impact how resources are invested in a project and organization. as dh practitioners, the time (or money paid to contractors) we must invest in managing servers, maintaining platform updates, and learning idiosyncratic administrative systems ultimately limits our ability to create and sustain unique, innovative projects. reexamining dh platforms through a minimal computing lens has led university of idaho librarians to pursue new project-development methods that minimize digital infrastructure as a means to maximize investment in people, growing agency, agility, and long-term sustainability in both the organization and digital outputs. u of i librarians’ development approach centered around static web-based templates aims to develop transferable technical skills that all digital projects require, while also matching the structure of academic work cycles and fulfilling dh project needs. in particular, a static web approach encourages the creation of preservation-ready project data, enables periods of iterative development, and capitalizes on the low-cost/low-maintenance characteristics of statically-generated sites to optimize limited economic resources and personnel time. this short paper introduces static web development methodology (titled “lib-static”) as a provocation to rethink dh infrastructure choices, asking how our frameworks can build internal skills, collaboration, and empowerment to generate more sustainable digital projects. by olivia wikle and evan peter williamson static web methodology as a sustainable approach to digital humanities projects the shortand long-term costs of the web platforms adopted for digital humanities (dh) projects extend far beyond the financial: when we choose a digital platform we are also making decisions about how to invest and deploy our time as well as technical and social resources in the digital projects we create and maintain (barats et al., 2020). the friction between the institutional, technical, and social infrastructures in which we make choices about our digital projects has too often led to unsustainable projects and practices in dh work. many advocates of a more sustainable approach to dh (including the participants in the dh 2022 panel on “sustainability + the politics of dh infrastructure” which inspired this short paper [footnote 1]) have noted the opportunity cost of investing in systems over people: as dh practitioners, the time (or money paid to contractors) we must invest in managing servers, maintaining platform updates, and learning idiosyncratic administrative systems ultimately limits our ability to create and sustain unique, innovative projects. in response, many librarians and dh practitioners have been reexamining platforms through a minimal computing lens, asking, as risam and gil have summarized, “what is, in fact, necessary and sufficient when developing a digital humanities project under constraint”, a methodological inquiry that leads “a decision-making process driven by the local contexts in which scholarship is being created” [1] (risam and gil, 2022). this reflection in our own context has led librarians at the university of idaho to pursue new project-development methods that minimize digital infrastructure as a means to maximize investment in people, while growing agency, agility, and long-term sustainability in both the institution and our digital outputs. for nearly a decade, we have been developing digital collections, scholarship projects, and instructional content using static web tools, leading to our development and support of the digital collection framework collectionbuilder. building on collectionbuilder, we have remixed template components and techniques to create custom projects such as oral history exhibits, deep maps, and digital editions, which we present below. this development approach, centered around minimal static web templates driven by tabular data, which we call lib-static, seeks to increase the return on learning new technical skills that all digital projects require, while also establishing technical solutions and social workflows that more closely match the structure of academic work cycles and dh project needs. in particular, the static web approach encourages the creation of preservation-ready project data, enables periods of iterative development, and capitalizes on the low-cost/low-maintenance characteristics of statically-generated sites to optimize limited economic resources and personnel time. this short paper will introduce the lib-static development methodology as a provocation to rethink dh infrastructure choices, asking how our frameworks can build internal skills, collaboration, and empowerment to generate more sustainable digital projects. the challenges of dh infrastructure working within the realities of academic funding and project planning, dh practitioners often still assume that traditional large-scale infrastructure is a necessity, which can result in huge sums sunk into outsourced development, contract work, and 3rd party subscriptions. this reflects an economic model that prioritizes purchasing systems over internal development of people and capacity. however, we (dh researchers and publishers) need to work with and maintain these systems, which can drain our time and reduce our agility and creativity, as well as lock us into unsustainable long-term commitments. beyond the missed opportunities to focus our time more productively, on the practical side the lack of human resources to continuously maintain needy digital infrastructure can result in dead links, half functioning websites, and lost data–the strikingly rapid loss of investment poured into the project. for example, we often want to teach creating a digital archive in the classroom, giving students the full experience of publishing their curation, description, and interpretation of archival objects on the web. however, even in setting up a fairly low barrier platform for this use case such as omeka, we are adopting (as they say) a puppy that requires constant ongoing maintenance and attention [2] (dombrowski, 2019). this requires us to either invest significant time in being system administrators, pay a subscription to a 3rd party to do it, or have a clear sunset date to close down the site–each of which requires the correct resources and can have serious drawbacks. will the students be able to reference the outputs of their project? will their work remain an accessible resource? how will the librarians facilitating the project have to invest their time? all dh platforms demand investment in learning (no matter how slick the system)–how can we make the most of that investment to build resilient capacity in our organizations and the dh community? at university of idaho library, balancing these questions has led to the development of a minimal, static web-based approach to creating digital scholarship projects and teaching dh in the classroom, that enables librarians to efficiently collaborate with faculty and students while centering sustainable systems, workflows, and data. static web opportunities most of the web today is built using dynamic web applications, such as content management systems like wordpress or drupal, which use server-side processing and databases to respond to user requests and manage data. likewise these types of cms or repository systems are chosen to build most dh projects because of the powerful web-based features they provide such as administrative interfaces, user management, and theming–think scalar, omeka, storymaps, or samvera. these platforms enact a distinct division between it maintainers/administrators, system designers, ux designers, and project/content creators, allowing any part of the equation to be outsourced. these applications come with heavy infrastructure requirements and ongoing maintenance costs–they require well resourced servers to provide adequate performance and continuous vigilant updating to avoid serious security risks. any customizations to the platform add complexity to the maintenance. in contrast to dynamic platforms, static websites are composed of plain html, css, js, and media files ready to be delivered to users without any server-side processing or database. this simplicity provides high performance for users, minimal infrastructure requirements for it, and lower barriers for developers. it uses less bandwidth, less energy, and fewer server resources–which ultimately means less cost for creators, maintainers, and end users. modern approaches are powered by static site generators that utilize simplified markup, plain text data files, and web apis to simplify the creation of site content and automate building complete websites ready to deploy. static web approaches have some unique benefits that have the potential to make dh projects more sustainable. first, because static assets do not require any server-side processing, they can be deployed on minimal servers (or free hosting services), significantly lowering cost and ongoing maintenance, while providing higher performance and scalability without further investment. this matches the reality of academic dh projects, which often involve lots of work during an initial active period and then are usually left with few if any resources for ongoing maintenance, so project websites need to be okay being left alone for long periods of time. static sites are suited to this type of neglect, avoiding the security risks and increasing breakage associated with insufficiently maintained dynamic platforms. with no additional software to administer on the server, static sites’ comparatively simple workflows and security reduce the need for system administrators, making it possible for dh practitioners to take complete control of the development and hosting process. this gives us a concrete opportunity to learn and build on basic web development and programming skills as we collaborate on a project, which can lead to more creative solutions and use of the platforms we are working with–and build additional capacity to power future projects. this helps acknowledge and remove the division of technical labor and “scholarly” contributions, as chan and sayers argue, “from the labor perspective, a minimal computing project positions technical work as creative and critical work beyond familiarity with management systems” [3] (chan and sayers, 2022). the low cost of this approach also makes the development process more agile and inclusive: we can iterate through project stages with less up front investment, quickly prototyping ideas (or changing course without too much loss if an idea doesn’t pan out) and building out smaller projects. this opens up opportunities to collaborate with more people, contexts, and use cases, including those without significant funding support. in practice, this can mean more people involved directly in development and the creation of projects that were previously impossible–student projects, classroom experiences, one off exhibits, or community led collaborations–while empowering more people with the skills necessary to sustain these resources. finally, an essential benefit is the creation of preservation-ready data as a byproduct of project creation. static web projects are built on structured plain text data and code in open standard formats–the content and data are not locked into a platform, but are prepared for preservation, migration, and reuse in the inevitable future with a new platform or new project. lib-static in practice at the university of idaho library we have been building static web-based projects as the core of our digital scholarship practice, many of which are rooted in the collectionbuilder code framework. below we present a few examples that we hope help illuminate why we have made this choice, highlighting some of the things we think the static approach has enabled while supporting more sustainable collaborations and digital outputs. most of this work takes place in our dh center, the center for digital inquiry and learning (cdil), led by librarians with faculty and student collaborators. the library and cdil do not have dedicated it personnel. prototype project sites are typically hosted on github pages or our self-managed reclaim hosting. final published versions are manually built and deployed to basic static site hosting provided by our main university it. the chart below demonstrates our typical workflow, technical infrastructure, and collaborator involvement for a static web project: table 1. typical workflow, infrastructure and collaborator involvement. project workflow description of work people involved curation, digitization, and metadata ↓ scholarship work focuses on curating data, digitizing objects, and creating quality descriptive metadata. librarians consult with collaborators (students and/or faculty) to envision site features and iteratively develop metadata templates that will support the site’s design. collaborators focus on creating metadata in spreadsheets and curating their items. collectionbuilder project ↓ prepared metadata csv and digital media files are added to a collectionbuilder code template. the template defaults are replaced by project configurations. project code is edited in a text editor such as visual studio code, and stored in github. librarians set up the project and work on code customizations to support new feature ideas. collaborators focus on contributing the site’s content using markdown and metadata spreadsheets. static site generator ↓ the static site generator jekyll knits together the source code in the collectionbuilder template to output a static site. there is no admin interface or log in! jekyll is run on a local computer or via a hosting service such as github pages. during iterative prototyping phases, the site is generally built on github pages to enable collaborators to see their changes to metadata or site content quickly. final deployment is handled by librarians, building the site with jekyll and moving it to the library’s server. static website final site is html, css, js, and media files ready to be hosted on any basic server or hosting service. librarians maintain the static sites for the long term. librarians package the complete data and media files for archival storage preservation. digital collections → research, analysis, access we first started working on a static web approach in pursuit of a more sustainable method to create and maintain our library’s digital collections. at first we redesigned and migrated a handful of sites that had aging / broken features or deserved more specialized presentation. before too long, we were rebuilding all our digital collections using a shared template, enabling us to balance customized browsing features and interpretive content with a sustainable workflow. the work on this template eventually grew into the collectionbuilder project, an open source framework for creating digital exhibits powered by static web methodology. our skills also grew with the project, enabling us to create more features and unique collections along the way, opening up access to more materials and content–but also enabling us to include more collaborators, opening up our collections to more people creating unique interpretive writing and exhibits. digital collections are made up of a few distinct chunks of data: the digital media files, metadata describing the items, and interpretive content. once separated out from a specific platform, this data is easier to work with and update using standard tools, and becomes a preservation-ready package, a digital archive ready to be plugged into any access framework. this separation also makes convenient divisions for collaboration–curating and digitizing objects, describing items in spreadsheets, writing about the context in markdown files, and developing website features. in practice, faculty and graduate students often collect and curate archival content during their research process (often in collaboration with librarians and archivists). with just a folder of files and a spreadsheet, we are able to generate digital collection websites to explore and publish these materials. for students, seeing their items as data and as a digital archive can become an iterative aspect of the research and analysis. their work will prepare access and publishing, while also creating new means of exploring and answering research questions. these types of iterative and exploratory collections were not possible when our infrastructure was tied to a more rigid traditional asset management system. for example, one of our earliest static web collaborations was on the historical japanese ceramic comparative collection with archeology graduate student renae campbell. together, we were able to transform her thesis research into an interactive website with new ways to visualize the information, increasing the impact and access to her research. the work was completed during a summer fellowship, with campbell’s focus on spreadsheet data and ideas for presentation, and librarians’ focus on creating the site. campbell is able to contribute periodic updates with new items and metadata corrections (which only require working with a spreadsheet), but no ongoing maintenance is required. likewise, ctrl+shift started as a collection of interviews of contemporary american poets conducted by librarian devin becker. in building a website for public access, becker’s graduate research assistants created new metadata (tagging the interview transcripts) and imagined new features for the website, which in turn became a means of interrogating the collection, exploring new ways to understand and discover. the flexibility of the platform enabled publishing these evolving interpretive threads by remixing the data and the recipes of display, without needing to rebuild the infrastructure. creative nonfiction writing student isabel marlens was exploring archival materials about historical wildfires in idaho, curating items that informed the project fire lines. during her research, she encountered records of the timber protective associations which contain fascinating historic data and perspectives that informed fire policy in america. as a byproduct of her research process, others are also now able to explore these resources alongside her published interpretive essay. these small collections, responsive to researchers interests and needs, are uniquely enabled by the minimal digital collection template approach. static web for digital scholarship applying the static web approach to our digital scholarship projects has allowed for sustainable publishing: we’re able to develop unique projects and keep them up for the long-term without major maintenance responsibilities. preservation-ready data is prepared as part of the workflow, integrating stewardship of unique materials into our practice. it has also increased options for us to be creative, as we learn more about the technology we’re using and remix our template recipes to create unique features and explore new ideas. and it is continuously expanding the ways we collaborate, from embarking on initiatives to migrate old projects to teaching collaborators how to contribute content to spreadsheets or github repositories. one example of this is storying extinction, a multidisciplinary digital project which maps and documents the extinction of caribou from idaho. graduate students on this project collected interviews and videos, created spreadsheets of data, and sketched ideas for web functionality. librarian devin becker, building from a collectionbuilder template, remixed map features to develop a highly customized site enacting the collaborators’ concept of a “deep map” presenting multiple paths to explore and understand the collected videos, interviews, and interpretive content. this involved combining a typical collectionbuilder item page with the leaflet.js powered map layout, using the “flyto” function to animate movement across the landscape. the same collaborators built on this experience and template to develop keeping watch, a geospatial narrative mapping project documenting the fire towers of idaho. another example is the letters of marie mancini project, which provides access to long hidden letters of 17th-century noblewoman marie mancini. faculty and student collaborators transcribed, translated, annotated, and encoded the letters in tei, which freed librarian olivia wikle to develop a custom site to transform and visualize that data for engaging public access. starting from a base collectionbuilder template, adding approaches from the wax static framework, and remixing a variety of other code recipes, the project represents the flexibility a static approach enables. wikle wrote rake tasks (ruby scripts integrated into the project) that are able to transform the project’s plaintext data into website features presenting the letters’ transcriptions and translations in multiple languages, and to view patterns across them via maps and timelines. teaching with static web when it comes to teaching, we’ve been able to use static web development to create lasting digital artifacts, without adding long-term system maintenance to classroom prep. this approach frees us up to teach students basic digital and data literacy skills (using formulas in spreadsheets, writing for the web using markdown, creating a commit on github) that they’ll likely put to use in their future careers, while we remix templates to create oer resources that fit the unique classroom context. librarian evan williamson even created a digital collection with a 2nd grade class in this way, using paper forms to collect metadata about students’ favorite classroom books. in 2022, we worked with faculty and librarian collaborators on an neh-funded initiative called learn-static to explore and create resources for teaching with static web tools. these include documentation for getting started with github, managing data in spreadsheets, and writing in markdown and html, as well as example lesson plans that demonstrate how these various skills can be incorporated into the classroom as part of a larger dh project (such as building a digital collection or coding oral histories). librarians and instructors worked together to develop project templates, assessments, and resources that were then used and refined in the classroom. in this context, the static web approach facilitated collaboration between instructors, the creation of reusable open educational resources, and the teaching of more fundamental digital literacies integrated into domain-appropriate projects. for example, digital exhibit lab started as a customized version of collectionbuilder designed for a classroom project creating a shared digital collection. to enable students to iteratively test and debug their own metadata spreadsheets (without breaking the live site), the code uses papaparse to load a published google sheet on the fly. students get instant feedback on their work, allowing active prototyping where they can see the impacts of metadata refinements in real time. this functionality was further developed into the collectionbuilder-sheets template. conclusion overall, our experience has been that a static web methodology has opened up a more agile and sustainable approach to our digital initiatives, while growing internal skills, collaboration, and empowerment. the creation of well-structured and well-described data as part of the project process ensures our role in stewardship and preservation of unique materials and research, while the flexibility of static templates has increased our creativity. likewise, the ease of iterative development and prototyping has enabled new opportunities that were previously prohibitively costly. by minimizing infrastructure costs and commitments, we have been able to invest time in growing our internal capacity for deep collaboration, creative thinking, and sustainable project management. while a minimal computing static web methodology is not optimal for all situations, recent work informed by this approach demonstrates that it is a viable and powerful option when holistically balancing the costs of the institutional, technical, and social infrastructures that can sustain innovative dh scholarship. footnotes [1] “panel 1-01: sustainability + the politics of dh infrastructure”, adho dh2022, july 2022, https://dh2022.dhii.asia/dh2022bookofabsts.pdf references barats, c., schafer, v., and fickers, a. (2020). fading away… the challenge of sustainability in digital studies. digital humanities quarterly, 14(3). https://www.digitalhumanities.org/dhq/vol/14/3/000484/000484.html. [3] chan, t., and sayers, j. (2022). minimal computing from the labor perspective. digital humanities quarterly, 16(2). http://digitalhumanities.org/dhq/vol/16/2/000600/000600.html. [2] dombrowski, q. (2019). sorry for all the drupal: reflections on the 3rd anniversary of ‘drupal for humanists.’ quinn dombrowski, published november 8 2019, https://quinndombrowski.com/blog/2019/11/08/sorry-all-drupal-reflections-3rd-anniversary-drupal-humanists/. drucker, j. (2021). sustainability and complexity: knowledge and authority in the digital humanities. digital scholarship in the humanities, 36 supp_2: ii86–ii94. https://doi.org/10.1093/llc/fqab025. gil, a. (2018). design for diversity: the case of ed. the design for diversity learning toolkit. https://des4div.library.northeastern.edu/design-for-diversity-the-case-of-ed-alex-gil/. gil, a. (2015). the user, the learner and the machines we make. minimal computing. https://go-dh.github.io/mincomp/thoughts/2015/05/21/user-vs-learner/. [1] risam, r., and gil, a. (2022). introduction: the questions of minimal computing. digital humanities quarterly 16(2). http://digitalhumanities.org/dhq/vol/16/2/000646/000646.html. about the authors olivia wikle is the head of digital scholarship and initiatives at iowa state university, where she collaborates on projects involving digital scholarship, digital collections, and the institutional repository. she is a co-developer of the collectionbuilder static web framework, and her research interests include sustainability in digital libraries and digital literacy instruction. evan peter williamson is the head of digital scholarship and open strategies (and digital infrastructure librarian) at the university of idaho library, working with the center for digital inquiry and learning to bring cool projects, enlightening workshops, and innovative services to life. despite a background in art history, classical studies, and archives, he always manages to get involved in all things digital–especially websites, digitization, and digital preservation. his recent focus has been on data driven, minimal infrastructure web development, currently embodied in the collectionbuilder project. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – enhancing descriptive metadata records with freely-available apis mission editorial committee process and structure code4lib issue 24, 2014-04-16 enhancing descriptive metadata records with freely-available apis this article describes how the university of north texas libraries’ digital projects unit used simple, freely-available apis to add place names to metadata records for over 8,000 maps in two digital collections. these textual place names enable users to easily find maps by place name and to find other maps that feature the same place, thus increasing the accessibility and usage of the collections. this project demonstrates how targeted large-scale, automated metadata enhancement can have a significant impact with a relatively small commitment of time and staff resources. by mark phillips and hannah tarver introduction the university of north texas libraries’ digital projects unit (dpu) worked on two projects for the portal to texas history [1] that make use of publicly-available application programming interfaces (apis) to enrich and extend metadata records so that more users are able to find the resources which the records describe. the two collections that the dpu used for this work contain two series of maps, both issued by federal agencies. usgs topographic maps the first set of maps processed with the workflow described in this article was the texas 7.5 minute topographic map collection, containing over 8,000 maps acquired by the unt libraries as a “born-digital” collection [2]. the united states geological survey (usgs) [3] created these maps through its ongoing project to chart the geographic detail of the united states at a uniform size and scale (1:24,000) [4]. the collection includes an older set of maps scanned as digital raster graphic maps, and a second set of maps, for the same quadrangles, that were digitally-generated from geospatial data [5]. the united states geological survey makes these maps freely available in a number of ways via its various websites including the usgs store [6], the national atlas [7], and a historical topographic map site. u.s. census reference maps the second set of maps processed with the workflow was a collection of maps produced by the united states census bureau [8]. the maps in this case denoted the boundaries used for the collection and aggregation of census statistics and are part of the census bureau’s “reference map series” which the bureau makes available via its website for download [9]. the united states census bureau produces several kinds of reference maps that denote census blocks, census tracts, and other areas for which it compiles data. the maps include some roadways, railways, geography, hydrography, and other features to aid in identifying a particular location. users can consult the reference maps to determine which geographic areas correspond to census data and statistics. the maps are generated every ten years, corresponding with census data collection. the collection in the portal [10] contains the county block maps from 1990, 2000, and 2010, as well as the block maps generated under public law p.l.94-171 for 2010 that include legislative and voting districts. this paper discusses the workflow for acquiring, parsing, and preparing the map data for use in publicly-available apis and then discusses the apis used to enrich the map metadata before uploading the records and digital objects into the portal to texas history. getting the maps usgs 7.5 minute topographic maps currently, the easiest way to access topographic mapping data in bulk for the kind of work described in this article is from the usgs website, “the national map: historical topographic map collection [11].” the site allows users to specify a state and specific scale of maps to return a listing of search results that matches their query. this list of search results can be downloaded as a comma separated value (csv) file, which is the preferred format for this workflow. once the map metadata is downloaded, it can be opened with standard spreadsheet software such as openoffice or microsoft excel. each map is listed in a separate row and the column with the heading of “download geopdf” contains the download url for each map. this column can be copied into a text file with one url per line; the wget [12] command will download each map using the urls in the text file, designated by the --i argument (see example 1). example 1. wget command to retrieve topographic maps. wget --i topos_urls.txt u.s. census bureau reference maps the u.s. census bureau reference maps can be downloaded from the bureau’s reference maps portal [9], which provides access to a number of map series. for each of the census block maps the number of maps can be narrowed by decade, then state, with options to limit to county, subdivision, or city. after the final selection, the portal displays the relevant maps in a directory structure which contains a number of pdf map files. for example, maps of armstrong county in texas for the 2010 census are available at the following url: http://www2.census.gov/geo/maps/dc10map/gublock/st48_tx/county/c48011_armstrong/ wget can download all of the files in the armstrong county folder by referencing the url (see example 2). example 2. wget command to retrieve census reference maps. wget -r -np -nc --accept *.pdf http://www2.census.gov/geo/maps/dc10map/gublock/st48_tx/county/c48011_armstrong/ the command in example 2 executes the wget program with the following arguments: -r recursively download links found at the designated url. -np no parent, do not move higher in the directory structure. -nc no clobber, if the file has already been downloaded, do not download again, even if the process is restarted. --accept only allow content matching the following pattern, in this case only .pdf documents. there are a number of options for wget that can be tuned to your specific needs. the above example will get you started with downloading the census reference maps you are interested in. two application programming interfaces what is an api? an application programming interface, or api, is a way of programmatically interacting with a component or service within a programming language, within a framework for building applications, or as a way of connecting to and programming against various services offered by builders of applications. builders of web applications often offer apis to their applications and frameworks that allow third-party developers to create new applications and tools. this article uses the term api and web service interchangeably as is often the case when discussing apis and the web. the workflow used for these two collections makes use of two kinds of apis. the first extracts all of the place names that are found within the boundaries created by the four sides of an individual map; the second returns the county name for a given geographic latitude and longitude. these two lookups are used together to generate the place names and county names relevant for a given map and add the values to the map’s metadata record. the addition of these values will improve discoverability and access to the maps by allowing users to search for place names in the records or find other items about the same place(s). place name api the extraction api, citiesjson, is made available by geonames [13]. geonames is an organization that provides access to high-quality name and coordinate data which it aggregates from a number of organizations from around the world. geonames provides a large number of freely-available apis for developers to utilize in building systems which rely on geographic name and location data. in addition to the freely-available tools and data, a commercial support option allows for a higher rate of api usage and access to premium data. however, this workflow made use of the freely-available apis. geonames makes the underlying data, which their apis are built upon, available as bulk downloads so that developers can implement similar systems locally. geonames requires that a user register with the system to receive a free api key (the registered username) and then regulates api requests by the use of tokens. a user can make up to 30,000 daily api “credits,” but different api calls use different numbers of credits; thus the number of requests that a user or program can make in a day will depend on the kind of information requested. this workflow employs the citiesjson api that uses four credits per request, so the maximum number of requests that our application could make in one day from one username is 7,500. in addition, geonames limits api requests to 2,000 credits per hour; for the citiesjson api, we had to ensure that we made only 500 requests per hour to stay under the limit defined by the service. web services typically put these limits in place to reduce abuse of a system; it is important to understand the limits and penalties associated with exceeding them before using apis. in many cases, the web service will return an exception until sufficient time has elapsed. however, if consistent abuse is detected by the system it may permanently throttle or ban the user or ip address. the citiesjson api takes as input the following pieces of information: the north, south, east, and west coordinates of the bounding box, the preferred language in which to return the place names, the maximum number of names to return within a bounding box, and an optional callback function name. finally the api requires a username as the user’s api key. a complete request will look like this: example 3. geonames citiesjson request url. http://api.geonames.org/citiesjson?north=32.028676&south=31.905515&east=-95.371331&west=-95.516337&lang=en&maxrows=100&username=demo example 4. geonames citiesjson api response. { "geonames": [ { "countrycode": "us", "fcl": "p", "fclname": "city, village,...", "fcode": "ppl", "fcodename": "populated place", "geonameid": 4722123, "lat": 32.0232193, "lng": -95.3921752, "name": "reese", "population": 0, "toponymname": "reese", "wikipedia": "en.wikipedia.org/wiki/reese%2c_texas" }, { "countrycode": "us", "fcl": "p", "fclname": "city, village,...", "fcode": "ppl", "fcodename": "populated place", "geonameid": 4700164, "lat": 31.9190566, "lng": -95.3735624, "name": "ironton", "population": 0, "toponymname": "ironton", "wikipedia": "" }, { "countrycode": "us", "fcl": "p", "fclname": "city, village,...", "fcode": "ppl", "fcodename": "populated place", "geonameid": 8479911, "lat": 31.979577, "lng": -95.398622, "name": "corine", "population": 0, "toponymname": "corine", "wikipedia": "" }, { "countrycode": "us", "fcl": "p", "fclname": "city, village,...", "fcode": "ppl", "fcodename": "populated place", "geonameid": 4737012, "lat": 31.9190559, "lng": -95.4777322, "name": "todd city", "population": 0, "toponymname": "todd city", "wikipedia": "" } ] } the result from the api call is a list of place names that fall within the designated bounding box. geonames has many kinds of registered names; each of them has a unique identifier, called a geonameid.   table 1. example values returned by the citiesjson api for a place name. field example value fcodename populated place toponymname reese countrycode us fcl p fclname city, village,… name reese wikipedia en.wikipedia.org/wiki/reese%2c_texas lng -95.3921752 fcode ppl geonameid 4722123 lat 32.0232193 population 0 we limited the number of place names to 100 for each map, in order to get a reasonable number of place names, especially for maps covering a large area. it turned out that for all of the bounding boxes in this project where place names were requested, the request with the most results contained 94 names [14]. because of the nature of these maps and the geography of the state of texas, which was the primary area covered, there were many maps that had no relevant place names. one thing to note about the citiesjson api is that it will only return a result for a city or place name if the latitude and longitude in the geonames database is contained within the bounding box. this can be problematic for large cities or cities that cover an oddly-shaped area when a map shows a portion or edge of a city, but the recorded latitude and longitude for that city are not within the bounding box for the particular map. even with this limitation, the geonames citiesjson api makes it possible to insert a large number of place names into the metadata records that would not be included otherwise. this workflow takes a given geonameid and converts it into the string value that the unt libraries use to designate a place in their metadata system, listing place names hierarchically from the largest (country) entity to the smallest relevant entity. in this case a city such as denton, texas, which has a geonameid of 4685907, is represented as the string “united states – texas – denton county – denton” and follows the pattern of <country> – <state/province> – <county/parish> – <city/named place> used for locations in north america. the geonames hierarchy web service [15] allows for the conversion from a geonameid to the hierarchical string representing each place name. an example of the response for the hierarchyjson api for this service can be seen at the following url: example 5. geonames hierarchyjson request url. http://api.geonames.org/hierarchyjson?geonameid=4685907&username=demo example 6. geonames hierarchyjson api response. { "geonames": [ { "adminname1": "", "countryname": "", "fcl": "l", "fclname": "parks,area, ...", "fcode": "area", "fcodename": "area", "geonameid": 6295630, "lat": "0", "lng": "0", "name": "earth", "population": 6814400000, "toponymname": "earth" }, { "adminname1": "", "countryname": "", "fcl": "l", "fclname": "parks,area, ...", "fcode": "cont", "fcodename": "continent", "geonameid": 6255149, "lat": "46.07323", "lng": "-100.54688", "name": "north america", "population": 0, "toponymname": "north america" }, { "admincode1": "00", "adminname1": "", "countrycode": "us", "countryid": "6252001", "countryname": "united states", "fcl": "a", "fclname": "country, state, region,...", "fcode": "pcli", "fcodename": "independent political entity", "geonameid": 6252001, "lat": "39.76", "lng": "-98.5", "name": "united states", "population": 310232863, "toponymname": "united states" }, { "admincode1": "tx", "adminname1": "texas", "countrycode": "us", "countryid": "6252001", "countryname": "united states", "fcl": "a", "fclname": "country, state, region,...", "fcode": "adm1", "fcodename": "first-order administrative division", "geonameid": 4736286, "lat": "31.25044", "lng": "-99.25061", "name": "texas", "population": 22875689, "toponymname": "texas" }, { "admincode1": "tx", "adminname1": "texas", "countrycode": "us", "countryid": "6252001", "countryname": "united states", "fcl": "a", "fclname": "country, state, region,...", "fcode": "adm2", "fcodename": "second-order administrative division", "geonameid": 4685912, "lat": "33.20526", "lng": "-97.11697", "name": "denton county", "population": 662614, "toponymname": "denton county" }, { "admincode1": "tx", "adminname1": "texas", "countrycode": "us", "countryid": "6252001", "countryname": "united states", "fcl": "p", "fclname": "city, village,...", "fcode": "ppla2", "fcodename": "seat of a second-order administrative division", "geonameid": 4685907, "lat": "33.21484", "lng": "-97.13307", "name": "denton", "population": 113383, "toponymname": "denton" } ] } county name api for this workflow, we also needed to be able to extract the county name for a given latitude and longitude in the united states. because of the way that the information is stored in the geonames services, there is no direct access to an api that will supply this information. this project used an api available through http://broadband.gov–another free web service–that takes a specific place point and returns the associated county name and state federal information processing standards (fips) code. broadband.gov is maintained by the national telecommunications and information administration and provides a number of apis for developers [16] interested in leveraging broadband.gov’s data in local applications. this workflow needed a form of reverse geocoding, which turns a known point location into a readable value such as an address, city, county, or country. the broadband.gov web service does not indicate that it has a rate limit on the number of requests that can be made in a given amount of time. even without a given rate limit, it is good practice to throttle requests to avoid swamping a service; in our case we decided to wait for two seconds after each request as a way of being polite. example 7. broadband.gov county api request url. http://www.broadbandmap.gov/broadbandmap/census/county?latitude=33.827395&longitude=-99.67&format=json example 8. broadband.gov county api response. { "results": { "county": [ { "envelope": { "maxx": -99.47449799999997, "maxy": 34.242023999999994, "minx": -100.04847999999998, "miny": 33.73384899999999 }, "fips": "48155", "geographytype": "county2010", "name": "foard", "statefips": "48" } ] }, "message": [], "responsetime": 7, "status": "ok" } the response for this request includes the following fields:   table 2. example values returned by the broadband.gov county api. field example value envelope “maxx”: -99.47449799999997, “maxy”: 34.242023999999994, “minx”: -100.04847999999998, “miny”: 33.73384899999999 geographytype county2010 statefips 48 fips 48155 name foard for this workflow, we are specifically interested in the statefips value and the name value. we use these two values to construct a string for the location, for example “united states – texas – foard county” or “united states – louisiana – caddo parish.” generating api request urls each api has specific coordinate value needs. the geonames citiesjson web service requires the north, south, east, and west coordinate values for given maps, while the counties api from broadband.gov requires just the latitude and longitude for the point you are interested in reverse geocoding. to accommodate this, the authors created a python module [17] that will parse a simple text format containing bounding box information for each map. in this case, the input of the script is in the format described on the page “map sheet corner point coordinate files” from the u.s. census bureau [18]; the format contains the information for generating the coordinate values, as well as a unique identifier in the “sheet code” field that allows us to uniquely identify a map. corner point coordinate files are available for the p.l. 94-171 county block, and p.l 94-171 voting district/state legislative district outline and census tract outline maps. for the topographic maps, it is relatively straightforward to generate the same kind of corner point file from the usgs topo map csv file, as it contains the needed information. the python script in example 9. uses python’s standard csv module to convert a file downloaded from the national map site into the corner point file format. the script adds the unique identifier for the map as well as the other fields. example 9. python code for extracting corner point coordinate file from usgs csv. import sys import csv import math if len(sys.argv) != 2: print "usage: python extract_corner_points.py <usgs topo csv file>" exit(-1) filename = sys.argv[1] topos = csv.dictreader(open(filename)) def norm_cord(cord): cord = cord.strip() final_cord = math.fabs(float(cord)) * 1000000 return str(int(final_cord)) for topo in topos: row = [] row.append(topo["gda item id"]) row.append(norm_cord(topo["e long"])) row.append(norm_cord(topo["w long"])) row.append(norm_cord(topo["s lat"])) row.append(norm_cord(topo["n lat"])) row.append(norm_cord(topo["scale"])) print "\t".join(row) once the data is in the same format for both the census maps and the usgs topographic maps, subsequent code can be used for both series of maps, producing the same kind of place names for all of the map records. the python module written for this project is capable of constructing the request urls for both the geonames and broadband.gov apis. these urls are paired with a wget command and a sleep command, which when executed will download the response from the api, save it to a unique file, and then wait for a specified amount of time. the authors of the module set conservative default values for sleep values based on the web service being used. the resulting output can be redirected into a new file, which will contain the scripting needed to make requests from the specific web services. example 10. example broadband.gov output. wget "http://www.broadbandmap.gov/broadbandmap/census/county?latitude=32.173850&longitude=-95.215715&format=json" -o 210048001000_ne.json ; sleep 1 wget "http://www.broadbandmap.gov/broadbandmap/census/county?latitude=31.418890&longitude=-95.215715&format=json" -o 210048001000_se.json ; sleep 1 wget "http://www.broadbandmap.gov/broadbandmap/census/county?latitude=32.173850&longitude=-96.072291&format=json" -o 210048001000_nw.json ; sleep 1 wget "http://www.broadbandmap.gov/broadbandmap/census/county?latitude=31.418890&longitude=-96.072291&format=json" -o 210048001000_sw.json ; sleep 1 wget "http://www.broadbandmap.gov/broadbandmap/census/county?latitude=31.796370&longitude=-95.644003&format=json" -o 210048001000_ct.json ; sleep 1 wget "http://www.broadbandmap.gov/broadbandmap/census/county?latitude=32.156354&longitude=-95.649899&format=json" -o 210048001001_ne.json ; sleep 1 wget "http://www.broadbandmap.gov/broadbandmap/census/county?latitude=32.033471&longitude=-95.649899&format=json" -o 210048001001_se.json ; sleep 1 wget "http://www.broadbandmap.gov/broadbandmap/census/county?latitude=32.156354&longitude=-95.789205&format=json" -o 210048001001_nw.json ; sleep 1 wget "http://www.broadbandmap.gov/broadbandmap/census/county?latitude=32.033471&longitude=-95.789205&format=json" -o 210048001001_sw.json ; sleep 1 wget "http://www.broadbandmap.gov/broadbandmap/census/county?latitude=32.094912&longitude=-95.719552&format=json" -o 210048001001_ct.json ; sleep 1 example 11. example geonames output. wget "http://api.geonames.org/citiesjson?north=32.173850&south=31.418890&east=-95.215715&west=-96.072291⟨=en&maxrows=100&username=demo" -o 210048001000.json ; sleep 10 wget "http://api.geonames.org/citiesjson?north=32.156354&south=32.033471&east=-95.649899&west=-95.789205⟨=en&maxrows=100&username=demo" -o 210048001001.json ; sleep 10 wget "http://api.geonames.org/citiesjson?north=32.151701&south=32.028676&east=-95.504864&west=-95.649899⟨=en&maxrows=100&username=demo" -o 210048001002.json ; sleep 10 wget "http://api.geonames.org/citiesjson?north=32.146900&south=32.023732&east=-95.365412&west=-95.510608⟨=en&maxrows=100&username=demo" -o 210048001003.json ; sleep 10 wget "http://api.geonames.org/citiesjson?north=32.046965&south=31.924375&east=-95.928524&west=-96.072887⟨=en&maxrows=100&username=demo" -o 210048001004.json ; sleep 10 wget "http://api.geonames.org/citiesjson?north=32.042616&south=31.919882&east=-95.789205&west=-95.928524⟨=en&maxrows=100&username=demo" -o 210048001005.json ; sleep 10 wget "http://api.geonames.org/citiesjson?north=32.038117&south=31.915241&east=-95.649899&west=-95.794585⟨=en&maxrows=100&username=demo" -o 210048001006.json ; sleep 10 wget "http://api.geonames.org/citiesjson?north=32.033471&south=31.910452&east=-95.510608&west=-95.655454⟨=en&maxrows=100&username=demo" -o 210048001007.json ; sleep 10 wget "http://api.geonames.org/citiesjson?north=32.028676&south=31.905515&east=-95.371331&west=-95.516337⟨=en&maxrows=100&username=demo" -o 210048001008.json ; sleep 10 wget "http://api.geonames.org/citiesjson?north=31.928719&south=31.806134&east=-95.938923&west=-96.077906⟨=en&maxrows=100&username=demo" -o 210048001009.json ; sleep 10 for the state of texas there are generally 4,500 – 7,800 maps per series. each map requires one citiesjson request, and at least five county api requests per map. for these projects a county request is made for each corner of the map and the center point. generating metadata records once we collected all of the cities and counties from the two apis, the next step was to generate the needed metadata records for import into the repository system. in the case of these items, the metadata process followed a workflow similar to the workflow discussed for other collections going into the unt digital collections system [19]. the city and county names for each map were de-duplicated and inserted into the metadata records (see figure 1). in the case of the untl metadata specification [20], this information is added by the record creation scripts to the coverage element with a qualifier of placename. we decided to include all cities and place names in the records, and to include a county only if it is not represented in the hierarchy of the city or place name values. the resulting coverage section can look like this: example 11. place names as they appear in metadata record. <coverage qualifier="placename"> united states arkansas miller county texarkana </coverage> <coverage qualifier="placename"> united states texas bowie county nash </coverage> <coverage qualifier="placename"> united states texas bowie county wake village </coverage> <coverage qualifier="placename"> united states texas bowie county wamba </coverage> <coverage qualifier="placename"> united states texas bowie county whatley </coverage> <coverage qualifier="placename"> united states arkansas miller county </coverage> we also included information in each record about the outside boundaries of the map as well as the center of the map to display the location for online users (see figure 2). these values were encoded using the dublin core metadata initiative’s (dcmi’s) box encoding scheme [21] and point encoding scheme [22] respectively. these are described in full detail on the dcmi’s website and look like this when generated for the untl metadata format (which has coverage qualifiers for placepoint and placebox): example 12. place point and place box values. <coverage qualifier="placepoint"> north=33.476531; east=-94.092814; </coverage> <coverage qualifier="placebox"> northlimit=33.526615; eastlimit=-94.032387; southlimit=33.426447; westlimit=-94.153241; </coverage> the resulting metadata file once constructed and added to the portal to texas history is as follows. example 13. complete census map metadata record. <?xml version="1.0" encoding="utf-8"?> <metadata> <title qualifier="officialtitle"> 1990 census county block map (recreated): bowie county, block 15 </title> <title qualifier="seriestitle">1990 census county block map</title> <creator qualifier="ctg"> <type>org</type> <name>united states. bureau of the census.</name> </creator> <publisher> <location>washington d.c.</location> <name>united states. bureau of the census.</name> </publisher> <date qualifier="creation">1990</date> <date qualifier="harvested">2013</date> <language>eng</language> <description qualifier="content"> parent map for bowie county, texas showing the area of one geographic block for which the u.s. census bureau collected data. the plotted map scale is 1:15,000. </description> <description qualifier="physical">1 map ; 84 x 92 cm.</description> <subject qualifier="lcsh">census -maps.</subject> <subject qualifier="lcsh">bowie county (tex.) -maps.</subject> <subject qualifier="untl-bs">places united states texas bowie county</subject> <subject qualifier="untl-bs">landscape and nature geography and maps</subject> <subject qualifier="kwd">census blocks</subject> <subject qualifier="kwd">statistical areas</subject> <primarysource>1</primarysource> <coverage qualifier="placename"> united states arkansas miller county texarkana </coverage> <coverage qualifier="placename"> united states texas bowie county nash </coverage> <coverage qualifier="placename"> united states texas bowie county wake village </coverage> <coverage qualifier="placename"> united states texas bowie county wamba </coverage> <coverage qualifier="placename"> united states texas bowie county whatley </coverage> <coverage qualifier="placename"> united states arkansas miller county </coverage> <coverage qualifier="date">1990</coverage> <coverage qualifier="timeperiod">mod-tim</coverage> <coverage qualifier="placepoint">north=33.476531; east=-94.092814;</coverage> <coverage qualifier="placebox"> northlimit=33.526615; eastlimit=-94.032387; southlimit=33.426447; westlimit=-94.153241; </coverage> <collection>uscmc</collection> <institution>untgd</institution> <resourcetype>image_map</resourcetype> <format>image</format> <identifier qualifier="local-cont-no">90b48037_015</identifier> <identifier qualifier="local-cont-no">(90rblk) 19348037015000000000</identifier> <note qualifier="display">total sheets: 41 (index 1; parent 27; inset 13)</note> <note qualifier="display">dimensions: 33 x 36 in.</note> <note qualifier="display"> "all maps display the 1990 geography; however, the features displayed on these maps are those shown on the census 2000 maps. these maps show the boundaries and numbers of the 1990 census blocks as well as the named features underlying the boundaries." </note> <note qualifier="nondisplay"> additional information taken from census bureau website: "these large scale, large format (36" x 33") maps depict the smallest geographic entities for which the census bureau presents data, census blocks. the recreated 1990 census block maps were produced for counties. all maps display the 1990 geography; however, the features displayed on these maps are those shown on the census 2000 maps. these maps show the boundaries and numbers of the 1990 census blocks as well as the named features underlying the boundaries. they also show the boundaries, names and codes for 1990 american indian/alaska native areas, counties, county subdivisions, and places. the scale of the maps will be optimized to keep the number of map sheets for each area to a minimum, but the scale and number of sheets will vary by the area size of the county and the complexity of the census blocks." http://www.census.gov/geo/maps-data/maps/1990coublock.html </note> <meta qualifier="metadatacreator">mphillips</meta> <meta qualifier="system">pth</meta> <meta qualifier="ark">ark:/67531/metapth363004</meta> <meta qualifier="metadatacreationdate">2014-01-05, 17:46:10</meta> <meta qualifier="hidden">false</meta> <meta qualifier="metadatamodifier">mphillips</meta> <meta qualifier="metadatamodificationdate">2014-01-10, 08:14:01</meta> </metadata> after the authors’ metadata creation scripts were finished, the resulting metadata files were paired with their corresponding map pdf files and loaded into the portal to texas history. they are presented both as high-resolution map images (see figure 3) which can be navigated using the full screen zoom interface (see figures 4 & 5), and as the original pdf format as it was downloaded from the agencies’ websites (see figure 6). figure 1: landing page for an individual census reference map.   figure 2: bounding box and point information display in a census map record.   figure 3: image view of census map.   figure 4: map zoom interface.   figure 5: detail view of map.   figure 6: user access to view the image file or download the original pdf.   closing the workflow outlined in this article can be used by others who are interested in integrating maps for their states into their local repositories. the apis and methods are easily adaptable and can be used in a number of applications. the ability to provide textual searching of the place names and county names for these maps increases the usability and discoverability of these maps. the enhanced metadata allows users to find all maps that show a particular county within the portal collection without having to consult a paper index map. likewise, a genealogist or historian can locate a map containing a specific place name or city name, which may not be indexed in any other way. the unt libraries placed the first 8,556 texas topographic maps online over the past three years; to date they have received over 61,700 uses [23]. the u.s. census maps were placed online in january of 2014 and at the time of this publication had already received 1,692 uses [24] by the public. the relatively high usage of these two collections illustrates the importance of providing high-quality metadata, hosting, and access for born-digital map collections on the web. the collections also represent examples of ways that metadata records can be augmented and enhanced by simple, freely-available apis. this allows organizations to potentially add large, high-demand collections to existing digital collections with a relatively small commitment of time and staff resources. references [1] the portal to texas history; http://texashistory.unt.edu/ [2] usgs topographic map collection; http://texashistory.unt.edu/explore/collections/ustopo/ [3] united states geological survey; http://www.usgs.gov/ [4] usgs topographic maps; http://topomaps.usgs.gov/ [5] us topo quadrangles, united states geological survey; http://nationalmap.gov/ustopo/index.html [6] the usgs store; http://store.usgs.gov/ [7] the national map, united states geological survey; http://nationalmap.gov/index.html [8] united states census bureau; https://www.census.gov/ [9] u.s. census reference maps; https://www.census.gov/geo/maps-data/maps/reference.html [10] united states census map collection; http://texashistory.unt.edu/explore/collections/uscmc [11] the national map: historical topographic map collection; http://nationalmap.gov/historical/ [12] wget; https://www.gnu.org/software/wget/ [13] geonames; http://www.geonames.org/ [14] 1990 census county block map (recreated): cherokee county, index; http://texashistory.unt.edu/ark:/67531/metapth358801/ [15] geonames hierarchy webservice; http://www.geonames.org/export/place-hierarchy.html [16] broadband.gov developer; http://www.broadbandmap.gov/developer [17] corner_points_parser; http://github.com/vphill/corner_points_parser [18] map sheet corner point coordinate files, u.s. census bureau; http://www2.census.gov/geo/tiger/rd_2ktiger/pl_maps/cornerpt/cornerpt.html [19] phillips, mark, hannah tarver, and stacy frakes. “implementing a collaborative workflow for metadata analysis, quality improvement, and mapping.” code4lib journal. n. page. web. 17 jan. 2014. http://journal.code4lib.org/articles/9199 [20] unt libraries metadata documentation; http://www.library.unt.edu/digital-projects-unit/metadata/ [21] dcmi box encoding scheme; http://dublincore.org/documents/dcmi-box/ [22] dcmi point encoding scheme; http://dublincore.org/documents/dcmi-point/ [23] statistics for usgs topographic map collection; http://texashistory.unt.edu/explore/collections/ustopo/stats/ [24] statistics for united states census map collection; http://texashistory.unt.edu/explore/collections/uscmc/stats about the authors mark phillips is assistant dean for digital libraries at the university of north texas libraries. hannah tarver is head of the digital projects unit at the university of north texas libraries. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – customizing open-source digital collections: what we need, what we want, and what we can afford mission editorial committee process and structure code4lib issue 59, 2024-10-07 customizing open-source digital collections: what we need, what we want, and what we can afford after 15 years of providing access to our digital collections through contentdm, the university of louisville libraries changed direction, and migrated to hyku, a self-hosted open-source digital repository. this article details the complexities of customizing an open-source repository, offering lessons on balancing sustainability via standardization with the costs of developing new code to accommodate desired features. the authors explore factors in deciding to create a hyku instance and what we learned in the implementation process. emphasizing the customizations applied, the article illustrates our unexpected detours and necessary considerations to get to “done.” this narrative serves as a resource for institutions considering similar transitions. by emma c. beck, terri l. holtze, rachel i. howard, and randy kuehn system selection our proprietary past in 2005, the university of louisville libraries selected contentdm to host its digital collection of unique and rare primary source materials and used a library services technology act (lsta) grant for funding. limited technology resources meant that the institution needed a “plug and play” system with the possibility of customization. contentdm seemed functional, and allowed self-hosting with support from oclc, an established library vendor. with the creation of two new positions in 2006, the digital initiatives librarian and the web services librarian, along with the assistance of a server administrator who managed the self-hosted server, we established our digital collections program using a distributed approach to creating scans and metadata. we added content from multiple libraries at the university plus a few non-library campus collections; upgraded to a license without limits on the number of collections and items; created interactive maps and custom queries; and added a development server to test upgrades and customizations along the way. by the time oclc announced that it would no longer support self-hosted contentdm instances after 2018, we had experienced increasing dissatisfaction with oclc’s lackluster customer service, slow pace of software development, and recurring issues of server downtime with their other products. a move to a hosted instance would mean a loss of our customizations, an increase in costs, and a deeper dependence on an unreliable vendor. we thought we were outgrowing the “out of the box” approach to repository software and hoped that adopting open-source tools would better serve our needs and lower costs. system requirements as we began the search for a new system, our digital collections consisted of approximately 100,000 digitized and born-digital works, including images, maps, newspapers, and videos, with some “compound objects” (a contentdm term) comprising a parent item with two or more children (for example, a multi-page newspaper issue). the new system would need to accommodate such hierarchies. it would also need to provide a more user-friendly image viewer and be able to handle full text provided as either metadata or through optical character recognition (ocr). we also wanted a system that provided rights protection, controlled vocabulary, batch import/export, advanced search, facets, and reports. in addition to the built-in capabilities, we sought the ability to edit the look of the pages, including item records, headers and footers, and the ability to create additional informational pages. we preferred, but did not require, that the new system allow for the scalability and interoperability that would allow us to move our electronic theses and dissertations (etds) and oral histories into the same software stack, saving more money on vendor licenses and removing silos. as well as the system requirements and functionality, we also needed to keep in mind our lean staffing. digital initiatives, which oversees digital collections and the institutional repository, has three full-time equivalent (fte) employees, up from 2.5 at the outset of the migration. web services, responsible for multiple sites and systems, has just 1.7 ftes, and the server administrator oversees linux and windows servers across five libraries. we needed a system that could be implemented and maintained with these staffing levels. open-source solutions given our dissatisfaction with stagnant proprietary software, we found ourselves drawn to open-source software (oss) as a compelling alternative. it held the promise of access to a community of users in an environment of knowledge sharing, innovation, and transparency. its inherent flexibility would enable us to customize to suit our specific needs, empowering us to tailor the software to our requirements. we had successfully adopted oss for specific, limited uses prior to this exploration. for example, our archives catalog runs on archivesspace, and the oral history audio files and transcriptions now live in a locally developed application employing the oral history metadata synchronizer (ohms) oss. we use lockss (lots of copies keeps stuff safe) for digital preservation. other applications posed implementation or maintenance challenges, often due to underdeveloped documentation and the absence of dedicated software support, but vended solutions had also lacked those resources. in 2018, the two newest oss systems in the digital asset management space were islandora and hydra/sufia (now hyrax, created and maintained by the samvera community). both could handle audiovisual materials and texts, including scholarly publications such as etds. at the time, islandora offered a hosted solution through lyrasis, although that would limit design and customization options. hydra users tended to be at institutions with teams of in-house developers. however, they were working on a more turnkey approach called “hydra-in-a-box” (now hyku), which sounded like a good fit for institutions like ours who needed an entry-level oss application. it was still in development, but the university of houston had recently received a grant from the institute of museum and library services (imls) grant to build a bridge2hyku toolkit which sounded like exactly what we needed to move forward with our migration. the toolkit wasn’t as far along as the software development, but we did not feel hurried. we thought we could keep the public contentdm instance up (albeit without vendor support) while we took our time setting up the new system (which we thought would require our time but not any new expenses). implementation unboxing hyku when we began working with hyku, we undertook basic customizations to incorporate our preferred work types and metadata schema, essential for ensuring a seamless data migration process. this initial phase involved extensive planning and proved time-consuming. the functionality enhancements benefited from the available hyrax documentation, but we found ourselves needing to fill in the gaps. these initial customizations to improve functionality and usability provided the foundation for our initial instance of hyku. this version operated without any significant issues and was put into production. however, only a fraction of our collection was represented. importing the remainder of the collection required further system enhancements. outsourcing development we had not yet focused on more complex customizations of the hyku oss. hyku did not include some necessary features, and the imls grant had not produced them as we had hoped. we also sought to improve the user experience. the most critical requirement for us, the newspaperworks gem, was not part of the core code. we had previously completed a community transcription project with text transcribed and shared in page-level metadata. the system needed to be capable of housing this metadata and making it searchable. an entire issue of the newspaper should be treated as a parent, with each page (and its accompanying searchable metadata) treated as a child. a search for any term in the child should resolve to the parent record, so as not to lose context, but have a way within the viewer to navigate to the appropriate page (this was possible, albeit clunky, in contentdm). parent/child records made up the bulk of our collections and help would be needed to incorporate this functionality. we realized that outsourcing development would be necessary to complete our migration. incorporating the newspaperworks gem continued to be our primary concern. aside from that, our list of customizations needing to be outsourced started out small. however, due to the task complexity, the list grew significantly when broken down into individual steps. we worked closely with the software development company contracted to improve our instance of hyku, including meeting monthly, and testing code changes. troubleshooting issues occupied most of our sole in-house programmer’s time over the duration of the project. the digital initiatives team needed to restructure metadata to match the new code’s handling of compound objects, adding and populating a “parent” field to every parent and child and to import or reimport all metadata and associated files through the system’s bulkrax tool – another time-consuming process. even with a team of contracted developers handling the code changes and moving the project forward, we underestimated the time and involvement necessary to complete this phase of the project. additionally, while they did their own testing, it did not always represent how we used the system. for example, load time of a single newspaper issue on the developer’s test server appeared to be speedier than that same newspaper loaded on to our server with 100,000 items and heavy server traffic. usability we took advantage of the ability to customize the homepage and other page layouts. the custom homepage allowed us to highlight document types and themes like louisville maps and materials about the university. users could also use filters from the homepage to browse the collection rather than being required to complete a search before seeing the filters, as was the case in contentdm. we were also able to customize the thumbnail images throughout the system to be larger to help users more quickly identify items they wanted to investigate. once they did access an item record, the image could be zoomed in with exceptional clarity. on the negative side, we found that compound objects loaded so slowly that they were essentially inaccessible. to make matters worse, there was no indication to tell users that the files were loading – only a blank space in the viewer where the items would eventually appear. while importing our parent/child records, we contracted for a second round of development, focusing on additional customizations involving lower-priority features such as friendly urls, more options for sorting search results, and the ability to limit the size of the files downloaded by users. this round was a bit more difficult to navigate, as we discovered the customizations interfered with one another. at the same time, hyku’s core code was evolving faster than expected. we had to halt all planned local work on the system to avoid any further conflicts with the developers’ work. the project dragged on as we blew past our initial estimate for completion, which was not critical as long as the contentdm site could continue to provide access. however, campus information technology professionals had become increasingly concerned about the security risks posed by the aging server and unsupported software and gave us a hard deadline for its end of life. with the development contract ended and file importing nearing completion, we noticed that the navigation of full text within the image viewer no longer functioned, which, in conjunction with the image load time issues, critically affected the usability of the system. furthermore, the code did not meet ada standards. accessibility to begin with, hyku lacked adequate development in terms of accessibility. however, it is particularly important. firstly, it provides the best possible experience for all users. secondly, for public institutions – as many of hyku’s potential users are – accessibility is legally required under the americans with disabilities act. the samvera community has discussed the accessibility issues in working group meetings, but open-source communities have complex and differing needs, and development still focuses on creating additional functionality within the system rather than on correcting known accessibility issues. whether stemming from the programmers or the clients, many updates focus on improving functionality rather than improving accessibility even though the accessibility issues would be easier to fix. we began testing for and correcting accessibility issues early in our project; however, this was not an efficient use of our time because further development overwrote large sections of the code and the work had to be repeated after the customizations and fixes from the outsourced development were implemented. knowing that the accessibility issues existed, however, did help in planning what steps needed to be taken to eradicate the problems. the main accessibility issues we noted include: headings nested incorrectly and/or out of semantic order. pages without unique titles. settings on buttons and links need to show an outline on focus. various spots where <div> was a child element of <u> which is semantically incorrect. license and rights links opened in new windows. search form elements needed ids and labels. in the image viewer: needed to remove “maximum-scale=1.0” in meta tag and set scalable as “yes” so zoom functionality would work. no title attribute. throughout the customization process the in-house programmer and the web services librarian tracked changes to the code in a spreadsheet. this served multiple purposes: first, with multiple people working on the files it ensured communication about edits between collaborators; second, as different versions were implemented it allowed us to review the changes we had made and determine if they needed to be recreated in the newest version, and third, we shared the items in the spreadsheet that pertained to accessibility with the samvera metadata interest group (smig) and with our developer contractors in the hope that accessibility issues we had found would assist with future development. evaluating open source our hyku implementation took longer than anticipated, involved out-of-pocket costs to outside developers as well as extensive time on the part of the in-house team, and did not produce all of the desired results. when we finally launched our site in august 2023, full-text search and video streaming were not working within the image viewer, and image loading could be so slow that larger records were not useable, so we held off on adding yearbooks and a journal which had been available in contentdm, angering users who relied on free online access to those resources. users did appreciate the improvements to the design, site navigation, and show pages. the university of louisville project team participated in samvera community work groups and enjoyed the collaborative nature of this open-source community. we watched and listened as other desired improvements were developed for other projects, and eagerly anticipated the day when we could add them to our code. however, by trying to push the boundaries early on with new functionality, we now struggle to maintain the system with these customizations. where do we go from here? at the university of louisville, our extensive customizations, though tracked in github, have led to challenges as we try to upgrade our system. we are currently on hyku 3 and the release we would like to be on, hyku 6, is tentatively scheduled for this summer. to upgrade and take advantage of improvements made by others in the community, we need to separate our customizations from the core code, since some (not all) of our custom features have been refined and added to newer versions, but there is no simple way to do this. we are either stuck in our current system with its problematic features, or we could start from scratch. some of the work we outsourced to external developers and already paid for would likely be lost. we would also have to re-import all our collections and reconfigure all the settings. as an early implementer, the work we did or contracted, and the bugs we helped find, assisted in the overall development of the system, but have left us in a tricky spot. operating without full functionality left us in a challenging place. development slowdowns resulted in fewer upgrades, which could cause security problems. it became hard to delineate between necessary changes and the point of no return. which customizations will create update issues? would it be better for us to stay with the core code? at that point is there a difference in open-source software and proprietary except the ethos? the system’s complexity forces institutions staffed like ours to outsource for maintenance and upgrades of the system, requiring funding for an unknown and unplannable amount of work on an ad hoc basis. these unexpected requests put stress on an already tight budget, leading us to wonder if other proprietary systems, with known costs, would be a better option. we have begun to explore the digital collection landscape and wonder about other possibilities that would not strain our budget or personnel. digital collection options are scarce. there are a few open-source options still in early development and a few proprietary systems currently being built out with a variety of capabilities and cost structures. so where does that leave us? between a rock and a hard place. could we hire a new developer or two, whose time is dedicated to oss development? we wish, but with tight budgets, it’s difficult to get approval for a new employee line and our administration would rather pay for a temporary outsourcing expenditure than take on a continued annual expense. we are at the mercy of a stark budget for the coming year. these issues will plague us until we find a sustainable way forward. resources bridge 2 hyku – https://bridge2hyku.github.io/ digital collections relaunch – https://uofllibraries.wordpress.com/2023/08/04/digital-collections-relaunch/ guidance on web accessibility and the ada – https://www.ada.gov/resources/web-guidance/ hyku documentation – https://samvera.atlassian.net/wiki/spaces/hyku/overview hyku source code – https://github.com/samvera/hyku hyrax source code – https://github.com/samvera/hyrax samvera documentation – https://samvera.github.io/ samvera project – https://samvera.org/ university of louisville digital collections – https://digital.library.louisville.edu/ about the authors emma c. beck is metadata librarian at the university of louisville libraries. terri l. holtze is web services librarian at the university of louisville libraries. rachel i. howard is digital initiatives librarian at the university of louisville libraries. randy kuehn is digital technologies systems librarian at the university of louisville libraries. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – editorial mission editorial committee process and structure code4lib issue 59, 2024-10-07 editorial welcome to a new issue of code4lib journal! we hope you like the new articles. we are happy with issue 59, although putting it together was a challenge for the editorial board. this was in no small part because issue 58 was so tumultuous, including a crisis over our unintentional publication of personally identifiable information, a subsequent internal review by the editorial board, an extra editorial, and much self-reflection. all of this (quite rightly) slowed down our work. several editorial board members resigned, which left us with a much smaller team to handle a larger workload. as a volunteer-run organization without a revenue stream, code4lib journal is a labor of love that we all complete off the side of our overfilled desks. it was demoralizing to feel that we had lost the support of many in our community. a lot of us were tempted to quit rather than try to pick up and carry on. so, although we have published issue 59 later than planned, and with a different coordinating editor, we made it. this issue is testament to the perseverance of my colleagues on the editorial board, and to the wonderful articles contributed by our community. in this issue, you will find: jack o’sullivan, sarah romkey and karin bredenberg, in their article response to premis events through an event-sourced lens, continue a conversation on the premis data model begun by ross spencer in issue 56. they provide an interesting rejoinder to spencer, and we welcome – and value – these dialogues which enrichen the conversation on data preservation. this work demonstrates that the journal can be a useful home to conversations impacting our profession. sometimes the challenges of developing with and for open source projects are substantial. emma c. beck, terri l. holtze, rachel i. howard and randy kuehn share their travails working with early versions of hyku in customizing open-source digital collections: what we need, what we want, and what we can afford. despite spending substantial resources to contribute to an open source software project, and in building their own in-house customizations, they found that the solution they ended up with was far from ideal. their article describes some of the potential pitfalls when implementing an open source solution in a glam institution. the cost of electronic subscriptions is an evergreen issue for many acquisitions and e-resources librarians. having accurate data to inform these costs is essential for effective decision making about collections. lydia harlan, kristin buxton, and gabriele hayden describe their iterative process for harvesting counter data in their article cost per use in power bi using alma analytics and a dash of python. these journeys into harvesting useful data will be relatable – and useful – to many librarians in many collections-facing roles. wilhelmina randtke discusses the implementation of an intranet at georgia southern university libraries. in launching an intranet in libguides cms at the georgia southern university libraries, randtke describes some of the reasons for – and challenges inhering from – using technologies such as libguides and google docs as the building blocks for a library intranet. randtke focuses on some of the organizational considerations that bear upon rolling out a successful intranet project. in the dangers of building your own python applications: false-positives, unknown publishers, and code licensing, corey schmidt describes an exploratory process of running into some of the sharp edges of developing desktop applications in python. the article describes how the author met and overcame these hurdles, and provides the beginnings of a roadmap for anyone looking to build their own python desktop applications. finally, harry bartholomew describes a conversion of bliss classification to rdf, in converting the bliss bibliographic classification to skos rdf using python rdflib. bartholomew’s work is being done at queen’s college, cambridge, but offers important new developments for other collections who may also be using bliss. offering a representation of bliss data in rdf opens exciting possibilities. hannes lowagie and julie van woensel describe the creation of subject classification tools for the national library of belgium. their python-based approach forgoes two common ai strategies: large language models and self-trained models. helpfully in their article, simplifying subject indexing: a python-powered approach in kbr, the national library of belgium, they lay out an alternate third way forward, one that is customized to their specific requirements and community needs. our thanks to past serving editors, and to everybody who helped pull this issue together, including our generous authors. my hope is that the editorial board can continue to aspire to the journal’s mission of ‘fostering community and sharing information among those interested in the intersection of libraries, technology, and the future’. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – using amazon mechanical turk to transcribe historical handwritten documents mission editorial committee process and structure code4lib issue 15, 2011-10-31 using amazon mechanical turk to transcribe historical handwritten documents the developing “information age” is continually unraveling new ways of discovering, presenting and sharing information. most new academic material is digitally formatted upon its creation and is thus easy to find and query. however, there remains a good deal of material from times prior to the “information age” that has yet to be converted to digital form. much of this material can be found in library collections—whether academic, public or private—and thus remains available only to a limited number of locals or willing-and-able sojourners. using ocr technology, most typeset documents can be digitized and made available online; and there are several projects underway to do exactly this. however, there remains little to be done for handwritten materials. those who own collections of handwritten documents are increasingly wanting to make the content thereof available to the general public. unfortunately, traditional transcription models typically prove to be expensive or inefficient and pdf snapshots are not searchable. we have developed a model for digital transcription using google docs and amazon’s mechanical turk. using this model, one can use an online workforce to efficiently transcribe handwritten texts and perform quality control at a cost much lower than professional transcription services. to illustrate the model we used amazon’s mechanical turk to transcribe and then proofread the frederick douglass diary which we have made available on a public searchable wiki. the total cost of transcription and proofreading for the 72 page diary was less than $25.00 with some pages being transcribed and proofread for as little as $0.04. our results show that using amazon’s mechanical turk holds great promise for providing an affordable transcription method for hand-written historical documents making them easily sharable and fully searchable. by andrew s.i.d lang and joshua rio-ross introduction transcribing often hard-to-read historical handwritten documents is a painstakingly slow and costly undertaking. this has meant that the majority of important writings are not available for public study and those that are available are often only viewable as digitally scanned images. after more than 100 years of effort, only slightly more than fifty percent of james madison’s papers have been transcribed and published, [1] while the transcription of thomas jefferson’s papers, begun in 1943, will take another 15 years to complete. [2] [3] the remaining untranscribed portion of madison’s and jefferson’s writings are only available as “digitally scanned images of microfilmed copies of handwritten documents.” [2] transcription can be outsourced to a professional transcription service, but this is very costly, usually costing somewhere between $7.00 and $15.00 per hour depending on the level of service. so it is not surprising that many archivists are turning to crowdsourcing for transcription. most current attempts at crowdsourcing are run in-house, where they provide and preserve software for end users, upkeep servers, build and maintain web sites, and have editors proofread results. all of this requires a certain level of technical knowledge and support staff, meaning the projects can become time-consuming and costly. when done correctly, such crowdsourcing efforts have been remarkably successful, but they do not come without obstacles. daniel stowell, director and editor of the papers of abraham lincoln, is quoted in a recent new york times article on crowdsourcing transcription, saying that volunteers produced so many errors and gaps that “we were spending more time and money correcting them as creating them from scratch.” [3] our transcription project, the written rummage project, has itself been a rummage for the most efficient method of using crowdsourcing and various other internet resources to digitally transcribe handwritten documents. our primary objective was the successful and accurate transcription of the selected frederick douglass diary documents to searchable, digital text. however, the project would be of little general interest if it could not eliminate the cost barrier that usually stands between editors and accurate transcription. we therefore resolved to develop a procedure that is cheaper to implement than other transcription services. while crowdsourcing is inherently crowded with workers, managing the workers should require relatively few people so long as transcriptions can be completed efficiently. thus, the last objective was that written rummage be expedient. academic crowdsourcing crowdsourcing is the process by which a task is outsourced to an undefined group of people (the crowd) rather than contracting professionals to accomplish that task. crowdsourcing is used in both business and academic settings and the crowd can either be unpaid volunteers or be incentivized with micro-payments or other benefits. crowdsourcing is therefore especially useful for tasks that need to be repeated numerous times but are beyond the skills of computers–such as classifying images, gathering data, and transcribing handwritten text. a good example of a successful academic crowdsourcing project is zooniverse, [4] which now consists of several separate crowdsourcing projects but originally began with just the galaxy zoo project. zooniverse now has close to half a million volunteers and has led to several academic papers and scientific discoveries. [5] [6] one recent addition to the set of zooniverse projects is old weather, where volunteers transcribe the location and weather from british royal navy ship log books from around the time of world war i. [7] to encourage participation, old weather leverages google maps technology to allow volunteers to follow the course of a ship over time as they enter data. old weather also provides an experience where volunteers can learn about the ships, the log books and the people who kept them. as volunteers contribute they also move up in rank, with the top contributor for each ship being designated captain. another project of transcriptional interest is the open dinosaur project, where volunteers are asked to transcribe dinosaur limb bone measurements from academic papers. citizen scientists usually need little impetus to get them to contribute to a “dinosaur project”, because “dinosaurs are cool”, but the organizers have offered an additional incentive for participation stating that “all participants will be included as junior authors on the resulting scientific paper.” [8] both old weather and the open dinosaur project have a large user base of volunteers because participation is fun, interesting, and incentivized; and in the case for the old weather project liberally funded. [9] an up-to-date list crowdsourcing projects of general interest can be found on wikipedia. [10] crowdsourcing digitization projects while the old weather project and the open dinosaur project both use crowdsourcing for transcription, the resulting digitization is not the primary focus of these projects. the former focusing on climate change and the latter on evolution. both projects do show that to be successful in the increasingly crowded crowdsourcing arena it helps for a project to be either of great interest, well funded, or fun. if your transcription project is none of the above, then it will take greater effort to be successful using volunteers for transcription. one technique to increase participation in more esoteric transcription projects is “community engagement”, where organizers use various enticements to attract volunteers. recent success stories “have been particularly adept at using social media, developing refined mechanisms for ensuring that contributions are quality assured, working with large data sets, and creating interfaces that interact in a way that reduces complexity and confusion.” [11] a good example of using social media and other incentives for the volunteer transcription of handwritten documents is the transcribe bentham project, which is a “participatory project” transcribing the manuscript papers of jeremy bentham based at university college london. [12] the transcribe bentham project has had great media exposure and also a lot of success with “encouraging undergraduates and school pupils studying bentham’s ideas… to use the site to enhance their learning experience.” [13] the project is also being aided by the arts and humanities research council who awarded the project a grant of £262,673.00. [14] there are many other noble efforts in using crowdsourcing for the transcription of historical handwritten documents, with new ones seemingly appearing every other day. [15] one of the more recent projects aims to help transcribe the soon to be released us census data by using an ingenious hybrid of automation and crowdsourcing, automatically recognizing handwritten text using word spotting. the current government proposed system plans to pay for “transcribing the handwritten content of the images, a task that will take thousands of trained laborers anywhere between 6 and 12 months.” [16] how successful this and other projects will be remains to be seen, though newcomers are being aided by open source transcription tools such as scripto. [17] using mechanical turk to transcribe handwritten manuscripts several methodologies were tried, scrapped, and/or modified in the nascent stages of the written rummage project. the primary hurdles to overcome were developing a way to ensure transcription accuracy—a problem encountered by other transcription projects—and tweaking various aspects of management to expedite the project. what emerged is a technique that uses two separate transcriptions for each document, such that the first transcription is proofread in the process of the second transcription. the procedure is as follows. assuming that some storage of manuscripts to be transcribed exists in pdf format—in the present case, the frederick douglass papers—the first step is to prepare the mechanical turk human intelligence task (hit), a simple task performed by an anonymous workforce for micropayments, in this case, transcribing a page of handwritten text. amazon’s mechanical turk, often referred to as mturk, is among the premier crowdsourcing resources on the internet. mturk’s framework can be used to submit various tasks to a crowd for a prescribed amount of compensation, which amazon also takes a percentage of. one must have an account with mturk as a “requester” to do this. [18] once we do, we can click “design” on the toolbar. from here, a hit can be designed from a number of templates, depending upon what sort of task is to be accomplished by the crowd, see figure 1. for transcription purposes, text specifying the requirements of the task, a link to the manuscript image, and a text/comment box for the worker to type the transcription are all the necessary components. the “design” stage is also when the requester designates how much a worker is compensated for each completed hit. the lower the compensation, the slower the transcription process takes, since workers have a market of tasks to select from. written rummage has found that $0.08 per hit is enough to ensure quick acceptance and completion of hits at this stage—usually a set of seven (7) hits can be transcribed in three or four days. figure 1.template for hit design in amazon mechanical turk. in order to implement the hit, a comma-separated values (csv) file has to be loaded containing the links to each of the respective pdf files the requester wants to have transcribed. one of the convenient features of mturk is the ability to publish multiple hits at once using a .csv file. a list of any order x will in turn instruct the server to produce x hits corresponding to each linked item, see figure 2. typically a batch of seven hits is most manageable. figure 2. one of our csv files used to submit multiple transcription hits. once the desired csv file is made, we click “publish” on the toolbar, select the desired template to implement—“first transcription,” or the like—upload the csv file, confirm, name and “publish” the batch. mturk will subtract the appropriate amount of pending funds from the requester’s account, and all that is left to do is wait. we thereafter visit the “manage” screen to check on the hits progress, see figure 3. figure 3. managing batches in amazon mechanical turk. as aforementioned, the batch will typically be transcribed in three or four days. to access the submissions, requesters need only click the “manage” link on the toolbar, then select the batch we want to view. the next step is to copy-and-paste the submissions to google docs, a free cloud-based document service, where they are saved and stored. each transcription is saved as an independent document, resulting in a unique url for each document. since google docs has privacy settings, these must be set to “public” so that anyone with the url can view them—they will be used again in the second transcription process. the second stage of transcription has proven the most troublesome. the purpose of having a second transcription at all is quality control. rather than proofreading the original submissions ourselves, which would be tantamount to undertaking the transcriptions ourselves, we decided to crowdsource this responsibility as well. providing the original manuscript image alongside the first transcription, however, did not guarantee that any proofreading would be done—users could simply copy and paste the first transcription’s content and submit it as their own. yet again, having to manually read through each second submission and compare it to the first submission is little improvement upon transcribing the documents ourselves. we therefore considered another means of ensuring that the documents were at least being thoroughly scanned a second time for errors. once a first transcription is submitted and saved in google docs, a second, corresponding document is made from that transcription with errors inserted throughout the document. some errors are unusual sequences of letters, such as “vxz,” interpolated into words—for instance: “interpvxzolated.” to avoid mere “spellcheck” proofreading submissions, we also insert entire words, such as “bark,” into the pages—for instance: “we also insert bark entire words.” the insertions of each page are logged and the edited manuscripts are saved with their own urls. once second transcriptions are submitted, we simply search the submissions for the key words or sequences of letters. if they have been edited out by the worker, then the submission is assumed to have been edited and proofread thoroughly, and therefore accepted. if they have not been edited, the page is assumed to have not been proofread, the worker is not compensated, and the hit is published again. the process for publishing the proofreading hit is similar to that of the first transcription. the primary difference is that two documents—the modified first transcription and the original manuscript image—are uploaded as links in the hit rather than one. using two columns rather than one in a csv file allows us to enter the corresponding manuscript and image for each hit. workers are asked to open both the image and the transcription side-by-side, then to check the work for congruity. compensation for the proofread transcription is a dime—slightly more than for the first transcription due to the request to scan for congruity between two documents rather than simply type what one reads in one document. as with the first transcription, the chosen compensation usually yields a fully transcribed batch of seven manuscripts in three to four days. if multiple batches of seven manuscripts are published simultaneously, this projected completion time usually holds constant. results the written rummage project has shown promise for providing a viable alternative for private manuscript collectors to transcribe their documents to digital, searchable text. our test collection of documents, the aforementioned diary from the frederick douglass papers, is completely transcribed and proofread. to make the 72-page transcription viewable, we created a free wiki for anyone interested to visit: frederickdouglassdiary.wikispaces.com. each manuscript page has its own web page on the wiki, and that page contains the transcribed text as well as a link to the original source manuscript corresponding to that text. as an example, using the search box on the wiki, we find five pages (10, 29, 34, 42, 58) in the diary where the word “slave” or “slaves” appears. (figure 4) figure 4. searching for the word “slave” in frederick douglass’ diary. by clicking on a page, for example page 42, we can see the entire transcribed text of that page. (figure 5) figure 5. fully transcribed and proofread text from page 42 of frederick douglass’diary. then clicking the link provided retrieves an image of the original handwriting, as shown in figure 6. figure 6. original image of page 42 of frederick douglass’ diary. copyright library of congress. time has been the most fickle variable in the nascent stages of written rummage. we have sought to develop an efficient model for receiving, transcribing, storing, proofreading, and re-storing documents. most pressing has been the need to ensure that the hits through mechanical turk were completed as hastily as possible. we began with low compensation per hit. while documents were eventually transcribed for transcription rates of $0.01 per first transcription and $0.03 per proofread, they typically took two to four weeks to finish a batch of five to seven pages. after several infrastructural changes and changes in transcription rates—now about $0.08 per first transcription and $0.10 per proofread—most batches of six to eight pages are completed in less than or equal to a week. if multiple batches are published simultaneously, this number tends to hold constant rather than proportionally extend; that is, several batches of six to eight hits can typically be transcribed and proofread in as much time as one batch. with rates as they are presently set, our method offers transcription and proofreading at a rate of $0.18 per page, plus the 10% service charge to mechanical turk. thus, our services can be implemented by private collectors for roughly $0.20 per page. should one choose to add another stage of proofreading, this might increase to $0.30 per page. the exact cost will depend on exactly how long you are willing to wait for the transcription and proofreading process. even then better price and speed performance can be attained. [19] our research has found that this is a remarkable improvement upon other professional transcription rates, which range from $2.00 per page to $8.00 per page depending upon the service and service provider. further, unlike volunteer transcription efforts, which require it support and have server maintenance issues, all our money went directly to the transcription effort. to provide some perspective on the difference these rates make, we can take, as an instance, the same 72-page diary used for this pilot run of written rummage. with rates held constant where they are now, the projected cost of transcribing the whole diary with written rummage is between $14.26 and $21.60. due to complications and test runs and other restructuring expenses, we actually spent $22.86 to complete the diary. in contrast, using the range of rates offered by other services, the same transcription project would cost somewhere between $144.00 and $576.00. our rates were so low with mechanical turk that at one point we had serious ethical discussions as to its use. certainly the economy of mechanical turk has given critics and users pause—and rightfully so: is it ethical to request tasks for such low compensation? [20] is work simply being outsourced to people who benefit from the exchange rates, but at a pay rate otherwise unacceptable—for instance, unacceptable in the u.s.? we decided that using mturk to transcribe handwritten documents was not unethical but we realize that these questions may be potential barriers to some researchers who are considering using our methods. we in the end felt comfortable using mturk because the character of crowdsourcing in general has typically been avocational for the crowd rather than vocational; that is, the workers are typically not performing hits for living wage, but rather either as a hobby or for pocket cash, though this seems to be changing. [21] many crowdsourcing projects offer no compensation at all but instead only call for volunteers. we use mturk under the assumption that workers enter the mechanical turk marketplace aware of its supply and demand economy and choose their hits accordingly—no hits are compulsory. we therefore determined that, so long as workers choose to accept the tasks for the prescribed compensations, the mutual agreement establishes that the hit’s value is acceptable. conclusion the “information age” is voracious; libraries and private collectors are looking for means of transcribing their handwritten manuscripts to make them available to the academic community and broader public. as the void they have yet to fill becomes more apparent, collectors are seeking out convenient, inexpensive ways of digitizing their documents. digitization is the focus of a number of major projects going on in the academic and internet community; likewise, crowdsourcing is increasingly being used to accomplish otherwise tedious or impossible tasks. our project utilized avant-garde technology to transcribe handwritten historical documents, and it did so affordably. the crowdsourcing model that we have presented uses no funds for server maintenance or it support, nor for any other support personnel. because of this, given a collection of handwritten documents from a library or other manuscript collector, we can have that collection transcribed and proofread at rates near $0.30 a page–approximately 15% the nearest competition’s rates. a 200-page collection that would cost at least $400 to transcribe elsewhere would cost only around $60 to be transcribed with us. further, due to significant milestones in overcoming past quality control challenges, we can expediently offer accurate transcriptions. entire collections, depending upon their size, can be returned digital and searchable in 2-4 weeks, allowing collectors to contribute them to shared library resources or other academic spheres. the written rummage project set out to develop a means of expediently and accurately transcribing handwritten documents to digital, searchable text; it did so–and thereby broadens the possibilities for sharing knowledge in the digital realm. references [1] the james madison papers -american memory from the library of congress [internet] [cited 2011 october 13] available from: http://memory.loc.gov/ammem/collections/madison_papers/mjmabout.html [2] the thomas jefferson papers -american memory from the library of congress [internet] [cited 2011 october 13] available from: http://memory.loc.gov/ammem/collections/jefferson_papers/mtjabout.html [3] cohen p. 2010. scholars recruit public for project. new york times. [internet] [cited 2011 october 13] available from: http://www.nytimes.com/2010/12/28/books/28transcribe.html?nl=books&pagewanted=all [4] zooniverse – real science online. [internet] [cited 2011 october 13] available from: http://www.zooniverse.org/home [5] cardamone c. et. al. 2009. galaxy zoo green peas: discovery of a class of compact extremely star-forming galaxies. monthly notices of the royal astronomical society, 399: 1191–1205. doi: 10.1111/j.1365-2966.2009.15383.x [6] lintott c. et. al. 2009. galaxy zoo: ‘hanny’s voorwerp’, a quasar light echo?. monthly notices of the royal astronomical society, 399: 129–140. doi: 10.1111/j.1365-2966.2009.15299.x [7] old weather – our weather’s past, the climate’s future. [internet] [cited 2011 october 13] available from: http://www.oldweather.org/ [8] the open dinosaur project [internet] [cited 2011 october 13] available from: http://opendino.wordpress.com/ [9] old weather sails on. [internet] [cited 2011 october 13] available from: http://blogs.zooniverse.org/oldweather/2011/02/old-weather-sails-on/ [10] list of crowdsourcing projects – wikipedia, the free encyclopedia. [internet] [cited 2011 october 13] available from: http://en.wikipedia.org/wiki/list_of_crowdsourcing_projects [11] dunning a. 2011. innovative use of crowdsourcing technology presents novel prospects for research to interact with much larger audiences, and much more effectively than ever before. [internet] [cited 2011 october 13] available from: http://blogs.lse.ac.uk/impactofsocialsciences/2011/08/25/innovative-use-of-crowdsourcing/ [12] moyle m, tonra j and wallace v. 2010. manuscript transcription by crowdsourcing: transcribe bentham. liber quarterly, 20 (3-4) [13] transcribe bentham. [internet] [cited 2011 october 13] available from: http://www.ucl.ac.uk/transcribe-bentham/ [14] winners of the ahrc’s £4m digital programme announced.[internet] [cited 2011 october 13] available from: http://www.ahrc.ac.uk/news/latest/pages/winnersdigitalprogramme.aspx [15] brumfield b. 2011. 2010: the year of crowdsourcing transcription. [internet] [cited 2011 october 13] available from: http://manuscripttranscription.blogspot.com/2011/02/2010-year-of-crowdsourcing.html [16] mchenry k. et al. 2011. toward free and searchable historical census images. electronic image and signal proscessing. 22 september 2011. http://dx.doi.org/10.1117/2.1201109.003833 [17] scripto | crowdsourcing documentary transcription. [internet] [cited 2011 october 13] available from: http://scripto.org/ [18] amazon mechanical turk. [internet] [cited 2011 october 13] available from: https://www.mturk.com/mturk/welcome [19] faridani s, hartmann b and ipeirotis p. 2011. what’s the right price? pricing tasks for finishing on time. [internet] [cited 2011 october 13] available from: http://husk.eecs.berkeley.edu/courses/cs298-52-sp11/images/c/c7/faridani-hcomp11.pdf [20] fort k, adda g and cohen k. 2011. amazon mechanical turk: gold mine or coal mine? computational linguistics. vol. 37, no. 2, pages 413-420 doi:10.1162/coli_a_00057 [21] ipeirotis p. the new demographics of mechanical turk. [internet] [cited 2011 october 13] available from: http://www.behind-the-enemy-lines.com/2010/03/new-demographics-of-mechanical-turk.html about the authors andrew stuart ian donald lang, is a mathematical physicist and professor of mathematics at oral roberts university. he has received a number of awards, including being named a 2010 davinci institute fellow for his groundbreaking work in virtual worlds. he is an open science advocate, publishing in open access journals whenever possible. he has published in many academic fields, including mathematics, physics, and chemistry. andrew can be contacted at alang@oru.edu. joshua rio-ross graduated from oral roberts university with a b.s. in mathematics and a b.a. in english literature. he is currently working toward his master’s in mathematics at university of missouri, after which he intends to study christian theology and literary criticism. beyond the academic, joshua enjoys reading dostoevsky, writing poetry, swing dancing, and, as of late, crafting his bike-riding skills on the way to school. joshua can be contacted at another.leaf.in.the.current@gmail.com. subscribe to comments: for this article | for all articles 4 responses to "using amazon mechanical turk to transcribe historical handwritten documents" please leave a response below: greg lindahl, 2011-12-16 this brings to mind the project gutenberg distributed proofreaders website (http://pgdp.net), which i’ve used for handwritten documents such as an early 17th century italian dance manual. it provides a lot of support for proofreading, such as a dictionary, division of labor (multiple rounds of proofreading and formatting), and a ready population of volunteer proofreaders, including enough italian speakers to ensure that my project was done very accurately. session proposal: crowdsourcing thatcamp american historical association 2012, 2011-12-30 […] mechanical turk or various freelancing sites?  if so, how do you ensure accuracy?  (one recent project introduced known bad data to transcripts before paying users to proofread and transcribed a […] practical example on #crowdsourcing using amazon mechanical turk | antoni barniol, 2013-09-17 […] original page: http://journal.code4lib.org/articles/6004 […] brian tkatch, 2013-10-14 instead of proofreading, the same jobs can be done a second time, (by different people, if possible). this was save $.02. a comparison of the two (diff) would point out and differences. where there is no difference, it is better than a proofread, as the same thing was types in twice. where it is different: if it is a small amount, just fix them manually. where there are many, the transcription itself is problematic, redo the job. (and the person with the bad transcription should be banned from future hits.) leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – using xml schema with embedded schematron rules for mods quality control in a digital repository mission editorial committee process and structure code4lib issue 41, 2018-08-09 using xml schema with embedded schematron rules for mods quality control in a digital repository the michigan state university libraries digital repository relies primarily on mods descriptive metadata to convey meaning to users and to improve discoverability and access to the libraries’ unique information resources. because the repository relies on this metadata for so much of its functionality, it’s important that records are of consistently high quality. while creating a metadata guidelines document was an important step in assuring higher-quality metadata, the volume of mods records made it impossible to evaluate metadata quality without some form of automated quality assessment. after considering several possible tools, an xml schema with embedded schematron rules was ultimately chosen for its customizability and capabilities. the two tools complement each other well: xml schemas provide a concise method of dictating the structure of xml documents and schematron adds more robust capabilities for writing detailed rules and checking the content of xml elements and attributes. by adding the use of this schema to our metadata creation workflow, we’re able to catch and correct errors before metadata is entered into the repository. by lisa lorenzo introduction and context galleries, libraries, archives, and museums (glam) rely on high quality metadata to make their collections discoverable and understandable by their users. metadata quality is also a crucial precursor to emerging tools and technologies for search and discovery, such as linked data and machine learning. a future where users can leverage linked data to discover resources with a common attribute across institutions (“how many university libraries have oral histories about world war ii available online?”) requires consistent description. emerging efforts to automate subject assignment for electronic resources relies on record sets with accurate subject analysis to serve as teaching sets for algorithms. further, organizations such as the digital public library of america (dpla) also rely on high quality and consistent metadata in order to develop their aggregation platform. metadata standards and best practices abound, and a great deal of work has been done in the glam community to assess how well those standards, whether international, national, or institution-specific, are met. the digital library federation (dlf) metadata assessment working group [1] and the dpla’s metadata analysis tools [2] are but two examples. while more tools are becoming available for assessing metadata, it is still a challenge to choose metrics that are appropriate and meaningful to the particular institution and a method of assessment that is sustainable and integrable into existing workflows. what follows is a description of how the michigan state university libraries (msul) digital repository team approached this challenge and how it uses the chosen method, xml schema and schematron, to ensure metadata quality. msul digital repository the msul digital repository is a freely available collection of digitized and born-digital content covering a variety of topics and media. the repository is built on fedora with an islandora front-end and uses mods as its descriptive metadata. indexing is done with apache solr. collections come to the repository from a variety of different locations, including msu graduate students and faculty, library grant projects, and donations. unsurprisingly, metadata is in a range of formats and of varying quality when it reaches the repository team. from there, the metadata librarians on the team convert the metadata into xml if necessary, enhance and correct information as needed, and create an xslt to transform the source xml metadata into mods records. in order to normalize metadata over the entire collection, the repository team put together a set of metadata guidelines in 2016. the creation of this document was an important first step in a collection-wide metadata normalization effort that took place over the course of 2017 and guided metadata creation for new collections ingested into the repository. however, the guidelines alone weren’t enough to ensure consistent metadata quality in the entire collection. while xslts were updated to reflect the new guidelines, due to the volume of records in the digital repository, it was still impossible to ensure that every individual record conformed to the guidelines. errors resulting from mistakes in source metadata, such as incorrect date formats or improperly formatted uris, would often go unnoticed unless they caused a solr error during indexing or happened to be seen when browsing repository content. the repository team’s metadata librarians evaluated a few tools that might address these issues, namely openrefine [3] and metadata breakers [4]. openrefine, an open source tool for cleaning up messy data, is a powerful tool with many functionalities. it boasts subject and name reconciliation capabilities, support for regular expressions through general refine expression language (grel), and ability to edit large numbers of records at once. its widespread use in the glam community was also attractive due to the large user base that could potentially be tapped for guidance. however, importing a hierarchical metadata scheme such as mods into openrefine proved to be overly cumbersome. the python-based metadata breakers set of tools was helpful for getting a sense of certain problems within the metadata. it worked well in areas such as showing field usage within a collection, but wasn’t designed for identifying problems in each individual record. this is an important functionality when working with, for instance, electronic theses and dissertations metadata. the original metadata for this collection comes from individual authors and can vary widely, and may require remediation of individual records. additionally, the repository team’s metadata librarians’ relatively low comfort level with python meant customizing this tool would require too much of a time investment for this project. glam institutions often use xml schemas as validation tools for xml-based metadata and definitions for metadata standards. while validation documents of course don’t offer the editing ability that openrefine does, their customizability and extensive capabilities, when extended with additional languages, made them an attractive solution for msul. oxygen xml editor [5], the software used by the repository team for creating and transforming metadata, has a very simple process for running xml schemas, making it easy to add one extra step to the existing metadata workflow. while learning a new language was time-consuming at first, the metadata librarians’ strong familiarity with xml, xpath, and xslt helped a great deal, making the process of creating a custom xml schema relatively intuitive. the next sections will provide an overview of what the msul repository team considers the most helpful features of xml schema definition (xsd) and schematron. xml schema definition xsd is a humanand machine-readable language maintained by the world wide web consortium (w3c) for expressing constraints on xml documents.  an xml schema specifies how and in what order data appears in an xml document. it can provide a list of allowed elements and attributes, specify how many times each can appear within the document and whether they are required or optional, and define the datatype of each element and attribute. while these capabilities make xsd a flexible and powerful tool for validating xml documents, it fell short of expressing constraints on several xml features that were necessary for determining the quality of mods records. for example, while xsd can declare that an element or attribute is required, it cannot express a more complex requirement like a dependency, such as “<mods:url> may contain either a usage or an access attribute, but not both.”  xsd also cannot express co-occurrence constraints, or rules about which elements may occur together. this means that a rule such as “if <mods:name> contains an authority attribute, then it must also contain an authorityuri attribute” cannot be defined in xsd. in fact, xsd is not designed to satisfy every possible validation scenario. as stated in its purpose, “the language defined by this specification does not attempt to provide all the facilities that might be needed by any application. some applications may require constraint capabilities not expressible in this language, and so may need to perform their own additional validations” (w3c 2004). to this end, xsd is easily extensible. embedding rules from other validation languages, such as schematron, is as simple as adding them to an <xs:annotation> element, as shown in the “validating content with schematron” section. schematron schematron is a rules-based xml validation language and an open iso standard [6]. it uses xpath to locate nodes within xml documents and then displays a natural language error message written by the creator of the schematron document. it is quite flexible and simple in structure, and there are a number of ways to accomplish any validation task. while schematron provides a concise way to test the content of elements and attributes, it is less straightforward than xsd for defining the structure of an xml document (obasanjo 2004). implementation given the strength of xsd in concisely defining the structure of an xml document and schematron’s ability to handle conditional requirements and co-occurrence constraints, the team decided to create an xsd with embedded schematron rules. the following section describes how each tool was used within the context of a mods validation. some examples are given throughout, and the entire schema can be found on github [7]. defining structure with xsd xsd is an extremely complex validation language with many capabilities that are out of scope for this article. what follows is a very brief overview of the portions of the xsd language that have been most useful at msul. xsd defines two categories of elements: simple types and complex types. simple types include elements that contain only text, attributes, and restrictions (a list of allowed values for an element). complex types include elements that contain other elements or attributes. the author of the schema defines the constraints on each individual complex type that may exist in a valid xml document. the <xs:complextype> element can contain an <xs:choice> element that gives a list of possible child elements for an element or an <xs:sequence> element that gives a list of child elements in the order in which they must occur in the xml document. both <xs:choice> or <xs:sequence> will contain a list of references to elements that are defined elsewhere in the schema. each <xs:element> can contain optional maxoccurs and minoccurs attributes which define the maximum and minimum number of times the element can occur. if not explicitly defined, the default value for maxoccurs is “unbounded” and for minoccurs is “1”.  each <xs:attribute> element may contain a use attribute that defines whether the attribute is optional or required. xsd further defines a list of data types that restrict the type of content allowed in xml elements or attributes. some examples are <xs:string> which can contain characters, spaces, returns, and so on, <xs:date> which can contain a date in yyyy-mm-dd format, and <xs:decimal> which can contain numeric values. the w3c defines a complete list of xsd data types [8]. in the following partial example from the msul digital repository mods schema, the <mods:name> element is defined. it contains a sequence of three child elements, <mods:namepart> (which must occur at least once, but can occur any number of times), <mods:titleinfo> (which can occur one time or not at all), and <mods:role> (which also must occur at least once, but can occur any number of times). it also contains a required attribute, type, which has the type xs:string. finally, it contains an optional usage attribute, which may only contain the xs:string value, “primary”. <xs:element name="name"> <xs:complextype> <xs:sequence> <xs:element ref="mods:namepart" minoccurs="1" maxoccurs="unbounded"/> <xs:element ref="mods:titleinfo" minoccurs="0" maxoccurs="1"/> <xs:element ref="mods:role" minoccurs="1" maxoccurs="unbounded"/> </xs:sequence> <xs:attribute name="type" use="required" type="xs:string"/> <xs:attribute name="usage"> <xs:simpletype> <xs:restriction base="xs:string"> <xs:enumeration value="primary"/> </xs:restriction> </xs:simpletype> </xs:attribute> <xs:attributegroup ref="mods:authority-attributes-optional"/> </xs:complextype> </xs:element> validating content with schematron six elements make up the basic structure of a schematron document: <schema>, <ns>, <pattern>, <rule>, <assert>, and <report>. the <schema> element is the root of the document, and <ns> and <pattern> are its child elements. the <ns> element defines any namespaces used in the document. a <pattern> element contains one or more <rule> elements. the <pattern> element works like an if/then statement: only the first <rule> that matches a node in the xml being tested will execute (if rule 1 matches a node, execute its child elements, if not, try to match rule 2, and so on). the <rule> element contains a context attribute that specifies an xpath. this defines where in the xml document to search for the xpath statements in its child <assert> and <rule> elements. the <assert> element specifies an xpath in its test attribute and plain text within the element that will be displayed to the user if the test assertion is false. for example, if the test attribute contains a simple xpath, the presence of that node is what’s being tested. so, the following: <sch:rule context="mods:mods"> <sch:assert test="mods:titleinfo">there is no title.</sch:assert> </sch:rule> checks for a <mods:titleinfo> element as a child of <mods:mods> in the xml document being validated. if the element is not present, the error message “there is no title” will display in the error report. any valid xslt functions and expressions may be used in a test attribute. the <report> element functions in the same way as assert, but is its inverse. where <assert> will return its error message if its test is false, <report> will return its statement if its test is true. for example: <sch:rule context="mods:dateissued"> <sch:report test="contains(text(),'u'">date contains "u". change to "x".</sch:report> </sch:rule> this rule checks <mods:dateissued> for the “u” character, and if it is present gives the user the message “date contains ‘u’. change to ‘x’.” rules may also be written outside of a specific context, known as abstract rules. the advantage of abstract rules is that the author of the schema only needs to write them once and then may call them in any context, eliminating the need to rewrite the same rule multiple times. this partial example from the msul mods schema shows an abstract rule that tests whether an element contains a trailing period, and then calls that rule on the <mods:namepart> element with <sch:extends>: <sch:pattern> <sch:rule abstract="true" id="trailing-punct"> <sch:assert test="not(ends-with(.,'.'))"> check <sch:name/> for trailing punctuation. </sch:assert> </sch:rule> </sch:pattern> <sch:pattern> <sch:rule context="mods:namepart"> <sch:extends rule="trailing-punct"/> </sch:rule> </sch:pattern> one other important aspect of this example is the <sch:name/> element in the content of <sch:assert>. this will print the name of the context node in the error report if the assertion comes back as true. so, in this example when the rule is called on <mods:namepart>, the error would read “check mods:namepart for trailing punctuation.” schematron can be used on its own or embedded into an xsd, as msul has done. adding schematron rules to an xsd is as simple as entering the rule within <xs:annotation> and <xs:appinfo> in the element declaration where the rule will take effect. the above example for <mods:namepart> would be called in an xsd like so: <xs:element name="namepart"> <xs:annotation> <xs:appinfo> <sch:pattern> <sch:rule abstract="true" id="trailing-punct"> <sch:assert test="not(ends-with(.,'.'))"> check <sch:name/> for trailing punctuation. </sch:assert> </sch:rule> </sch:pattern> <sch:pattern> <sch:rule context="mods:namepart"> <sch:extends rule="trailing-punct"/> </sch:rule> </sch:pattern> </xs:appinfo> </xs:annotation> </xs:element> workflow a variety of software, both open source [9] and proprietary, is available for validating xml with xsd and schematron. since the msul digital repository team already uses oxygen xml editor to create and transform metadata, it made the most sense to validate metadata with this software as well. running xsd validations with embedded schematron rules in oxygen is very simple [10] and can be done by right clicking a file or a directory in the project pane and clicking “validate with schema.” from there, the user simply has to navigate to the correct xsd file and select the option to include embedded schematron rules. oxygen generates a list of errors and displays it in a new pane, or indicates that validation was successful. if the user ran the validation on multiple files, the list of errors is divided by file name. for the msul digital repository, a metadata librarian will run the schema against every new collection before it’s ingested. after reviewing the list of errors, the reviewer will decide how to address them. typically, if an error is widespread, it is solved by adjusting the xslt that transformed the original source metadata into mods. for instance, referring to an earlier example, if the validation found a series of records where unknown or uncertain dates were systematically entered with ‘u’ instead of ‘x’, a simple replace() function could be added to the xslt and solve the issue each time it appears.  isolated or minor errors, such as a missing author name, are usually fixed in the source metadata. in order to address errors in legacy collections, the schema was run against all collections in the digital repository over the course of several months. the resulting error reports were used to improve the xslt documents for each collection and create new metadata records, which were then ingested into the digital repository.  with the changes that were made, metadata records now conform with local standards, allowing more consistent display between collections. this also facilitated sharing repository metadata with the dpla and the msu libraries’ discovery layer, ultimately making repository resources discoverable in more environments. future work while xsd and schematron are effective tools for validating document structure, validation of content contained in elements and attributes is mainly limited to structural checks. spell checking and name and subject reconciliation are two areas in particular where the metadata quality control workflow could be improved. future areas of work for the repository team include taking a closer look at tools such as openrefine and metadata breakers. for example, openrefine’s reconciliation tools may be useful in converting local subject headings to fast headings, the preferred subject vocabulary for digital repository collections. bibliography obasanjo d. 2004. improving xml document validation with schematron [internet]. [cited 2018 june 4]. available from:https://msdn.microsoft.com/en-us/library/aa468554.aspx. piez w, lapeyre d. 2008. introduction to schematron [internet]. [cited 2018 june 6]. available from: http://www.mulberrytech.com/papers/schematron-philly.pdf. robertsson e. 2010. combining schematron with other xml schema languages [internet]. [cited 2018 june 4]. available from: http://www.topologi.com/resources/schtrn_xsd_paper.html. w3c (world wide web consortium). 2004. xml schema part 1: structures second edition [internet]. [cited 2018 june 5]. available from: https://www.w3.org/tr/xmlschema-1/#intro-purpose. xsd – quick guide [internet]. c2018. [cited 2018 june 4]. available from: https://www.tutorialspoint.com/xsd/xsd_quick_guide.htm. notes [1] for more information, see: http://dlfmetadataassessment.github.io/ [2] available here: http://openrefine.org [3] available here: https://github.com/dpla/metadata-analysis-workshop [4] available here: https://github.com/cmh2166/metadataqa [5] available here: https://www.oxygenxml.com [6] for more information, see: http://schematron.com/2016/11/iso-schematron-2016-released/ [7] available here: https://github.com/lmlorenzo/mods-schema [8] available here: https://www.w3schools.com/xml/schema_dtypes_string.asp [9] one open source tool available for running xsd and schematron validations is schemanon, available here: https://github.com/thelanguagearchive/schemanon. this tool was not evaluated for this project. [10] video tutorial available here: https://www.oxygenxml.com/demo/schematron_validation.html about the author lisa lorenzo is a metadata librarian at michigan state university libraries working primarily with the library’s digital repository metadata. she is a member of an agile development team working to improve and expand the msul digital repository and is responsible for transforming metadata from various sources and formats into standards-compliant mods and dublin core to facilitate searching within the repository and metadata reuse in other systems, such as the digital public library of america (dpla). lisa’s recent projects undertaken with her colleagues on the development team include normalizing mods records across all collections in the repository, updating and implementing local metadata guidelines documentation, and dynamically generating json-ld metadata as part of an experimental search engine optimization initiative. she also has a quarter-time appointment as a reference librarian. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – connecting historical and digital frontiers: enhancing access to the latah county oral history collection utilizing ohms (oral history metadata synchronizer) and isotope mission editorial committee process and structure code4lib issue 29, 2015-07-15 connecting historical and digital frontiers: enhancing access to the latah county oral history collection utilizing ohms (oral history metadata synchronizer) and isotope the university of idaho library received a donation of oral histories in 1987 that were conducted and collected by a local county historical society in the 1970s. the audio cassettes and transcriptions were digitized in 2013 and 2014, producing one of the largest digital collections of oral histories – over 300 interviews and over 569 hours – in the pacific northwest. to provide enhanced access to the collection, the digital initiatives department employed an open-source plug-in called the oral history metadata synchronizer (ohms) – an xml and php driven system that was created at the louie b. nunn center for oral history at the university of kentucky libraries – to deliver the audio mp3 files together with their  indexes and transcripts. ohms synchronizes the transcribed text with timestamps in the audio and provides a viewer that connects search results of a transcript to the corresponding moment in the audio file. this article will discuss how we created the infrastructure by importing existing metadata, customized the interface and visual presentation by creating additional levels of access using complex xml files, enhanced descriptions using the getty art and architecture thesaurus for keywords and subjects, and tagged locations discussed in the interviews that were later connected to google maps via latitude and longitude coordinates. we will also discuss the implementation of and philosophy behind our use of the layout library isotope as the primary point of access to the collection. the latah county oral history collection is one of the first successful digital collections created using the ohms system outside of the university of kentucky. by devin becker and erin passehl-stoddart introduction, or the difficulty of oral history in the digital age in today’s libraries, museums, and cultural institutions, oral histories are viewed as important primary sources that are utilized by everyone from students to researchers to genealogists to family members. historian donald a. ritchie acknowledges that historians have generally conducted interviews to acquire information that would otherwise not have existed, including the small but telling details that previously escaped notice and were overlooked in the historical narrative (ritchie 2011) [1]. new technologies are helping to advance the preservation, access, and use of oral histories. the medium, however, is in a transitional moment. some oral histories are now born digital, but many still exist in previous formats of audio cassette tapes, minidisc, mini-dv, etc. moreover, while oral histories are gaining in popularity as historical resources, they remain underutilized resources due partly to the difficulty researchers have had in gaining access to oral history collections (boyd 2011) [2]. this is especially true of oral histories stored in analog forms, such as the latah county oral history collection, where the expense and staffing needed to migrate audio cassette tapes over to a digital format is often prohibitive. many times oral histories contain multiple components, including audio, transcripts, indexes, and sometimes additional materials such as photographs or handouts. it is therefore necessary for information providers to, as doug boyd argues, “make the individual components of the collections all work together in a user-friendly, efficient, useful, and intuitive manner; be engaged and discovered more easily, more widely and effectively distributed, and more responsibly preserved” (boyd 2011) [3]. all of these components are addressed in the new open-source plugin developed at the university of kentucky named ohms (oral history metadata synchronizer). this article will address the steps taken by the digital initiatives department at the university of idaho library to preserve and make accessible the latah county oral history collection. the latah county oral history collection the latah county oral history collection consists of audio interview recordings and transcriptions for over 300 interviews that were collected in the 1970s. the interviews feature the recollections of long time residents of latah county who lived in the area as early as the 1890s. latah county is located in the northern idaho panhandle, which is a rural area of the state that produced much of the wheat, lentils, peas, oats, barley, and timber in the united states. it is the only county in the united states to be created by an act of congress in 1888. early settlers came seeking wealth through prospecting, mining, lumbering, and farming (originally through grazing, later as soil for agriculture) (hubbard 1957) [4]. it is also the home of the state’s land-grant university, the university of idaho. in the early 1970s, the latah county museum society board of trustees selected an oral history project committee to facilitate oral recordings of citizens of latah county, idaho. one of the main reasons behind conducting these oral histories in latah county was the idea that “by hearing people tell of the range of circumstances they faced, the choices they made, and the traditions that have given them strength, [we] can discover the dominant concerns of american working people during the last century” (schrager 1978) [5]. the funding originally came from a grant by the idaho bicentennial commission to design questions for oral history interviews, conduct the oral interviews, and present an oral history tape and slide program throughout the county during the bicentennial celebration. the project later received additional funding from the idaho commission on the arts and humanities to extend the scope of the project to include interviews from 1976-1978, as well as the creation of transcripts, photography of narrators, and the creation of a series of oral history booklets for elementary school social studies (schrager 1978) [6]. by the end of the project, the collection included approximately 570 hours of recorded audio interviews on cassette tapes featuring over 200 narrators from 1973-1978. by cooperative agreement, the collection was transferred to the university of idaho library special collections for public reading, listening, and preservation. initial digitization, ingest, and metadata the digitization of this collection began with a request from a patron who desired to use the audio from the cassettes for a documentary film. we did not at that time have the expertise or capacity to digitize the audio cassettes, but we were able to quickly digitize and publish the contents of all the transcripts in the collection. to do this, we used a guillotine to cut the bindings to the bound transcripts and then used an auto feed scanner to scan each transcript. we produced high-resolution tiff files for each page and large ocr generated pdf files for each transcript. we deposited the tiff files into our digital archive, then extracted the individual interview transcripts from the larger pdfs into smaller pdfs and named the files according to the interviewee’s name and the date/number of the interview (some participants were interviewed numerous times). digitization of this collection was prioritized for many reasons, not least among them was the thorough descriptions collected about each interview and interviewee during the initial recording of the interviews. this metadata was only available via the typed version of the guide to the latah county, idaho oral history collection, which listed each interview participant along with his/her biographical information and a brief synopses of each interview in which he/she participated. each synopses also included one uniformly formatted line that listed the date of the conversation, the length of the recording of the conversation, the length (in pages) of the transcript of the interview, and the initials of the interviewer [7]. we extracted all this information by copying the ocred text from our pdf, after which we worked to convert this text into an xml file. to transform the text document into a valid xml document we used regular-expression-enhanced find-and-replace strategies to tag the individual name, birthplace, origin, birth year, and occupation of each individual, as well as a nested listing of each interview he/she participated in, which included tags for the date, audio length, page length, interviewer, and synopsis. this initial xml file used common html tags such as <p> and <a> so that we could then create a listing of these interviews and people online. eventually, the file was transformed using xsl into a tab-delimited spreadsheet that we used to upload the corresponding interview transcripts into our contentdm instance. we accomplished the digitization, clean up, description, and upload of transcripts into contentdm  that users could publicly search, read, and browse in approximately one month. figure 1. initial xml file generated from the guide to the original collection. digitizing the audio files the stories contained in these oral histories were consistently compelling, depicting as they did the exhilarating stories of the hardships and trials many early homesteaders, loggers, and miners endured while living here, but the digitization of the audio recordings and building of the collection website took some time. after considering sending the tapes out to vendors, we finally decided to use some older hardware and software in the library to digitize the audio cassettes. since audio cassette digitization occurs at a 1:1 time ratio (one hour of effort to digitize one hour of recording), this conversion of 569 hours of interviews required several staff over the course of 18 months. the procedure for digitizing these cassettes involved multiple steps: the tapes were recorded onto a macintosh desktop computer using peak audio. each side was saved individually as a sound document 2 (.sd2) file (a proprietary format developed by avid technology). once all the tapes in an archival box were recorded, the .sd2 files were converted to audio interchange file format (.aif) using peak audio and the files were copied to a windows 7 machine for editing and preservation. using wavepad sound editor, the individual side recordings were trimmed for dead air and then combined into full interviews. interviews were converted from .aif format to .mp3 format using a batch process in wavepad. a common problem we encountered was broken and/or deteriorated tape cassettes, which required additional staff effort. over the course of a year and a half, however, we were able to digitize the vast majority of the tapes and ingest them into contentdm as mp3 files. discovering the oral history metadata synchronizer (ohms) as the digitization process came to a close, we saw a presentation by doug boyd at the 2014 northwest archivists conference about the development of an open-source software application for oral histories called the oral history metadata synchronizer (ohms) (boyd 2014) [8]. ohms is a web-based system that provides users with word-level search capability and a time-correlated transcript or index, connecting the textual search term to the corresponding moment in the recorded interview online (boyd 2013) [9]. ohms originally began in 2008 as a way to work with oral history transcripts for the kentucky digital library; it was later redesigned with an indexing component and was distributed as an open-source product with funding from the institute of museum and library services (imls). in its current form, ohms exists as a plugin that can work with content management systems such as contentdm. the ohms system uses xml files containing an interview’s metadata together with a php viewer to create the interface. several javascript files are used for audio/video players to connect the recording with that interview’s metadata, transcript, and index, which are all full-text searchable. this is the most important aspect of the ohms system, namely that it presents and connects the text and recording of the oral history interview on the same web page, which in turn allows the user to stay on one page while exploring both the audio and textual recording of the history. to help administrators create the xml document that drives the system, ohms operates an online tool that uses forms and tagging features to help users properly describe and timestamp an interview [10]. the ohms tool allows an interview’s metadata to be uploaded as an xml or csv file. it can then be enhanced by adding minute by minute timestamps to the transcript. administrators can also define and describe sections of the interview to create a linked index, which can also be enhanced with links, maps, and other features. after enhancement and conversion with the tool, these correctly tagged xml files will then run on one’s own system after being downloaded from the ohms account. figure 2. forms for filling out an index section via ohms online application figure 3. the tool that helps users add timestamps to the transcript. ohms implementation after downloading and installing ohms onto our web server, we customized the system to fit with the design of our other digital collections. both the installation and customization of the ohms system proved to be exceptionally easy to perform. we did not end up using the ohms online tool because we discovered that it was easier for our purposes to edit the xml files directly. we created our xml files, however, from an initial model file that was uploaded into the ohms tool, enhanced via its features, then downloaded back into our system. we decided to go this route because we already had indexes for most interviews and initially felt that adding minute timestamps to the audio would be too time consuming. modifying the ohms system the ohms system allows for the incorporation of a number of different playback systems, from youtube to brightcove. since this collection was solely audio, the main playback mechanism we were concerned with was jplayer, a common javascript application built for the playback of mp3, mp4s, and other audio files online (that is included in the distribution).[11] appearance/client-side modifications files modified: ohms-viewer/css/viewer.css ohms-viewer/tmpl/viewer.tmpl.php ohms-viewer/js/viewer_other.js ohms-viewer/tmpl/player_other.tmpl.php we had two goals when designing the appearance of these interviews: 1) to fit the display into our digital collections template and 2) to create interactive controls and web site architecture that promoted easy discovery of and access to the interviews themselves. to accomplish the first we revised the viewer.css file and the viewer.tmpl.php to accommodate our web site backgrounds, layouts, and colors. within viewer.tmpl.php, we simply erased the boilerplate html containers included and replaced these with the html of our customized header and container for the collection; we also included links to our custom css and javascript files to better fit with our design. after doing this, we adjusted the viewer.css file so that the html would display at the proper size in our container. we then made a small change to the player_other.tmpl.ph to properly layout the jplayer skin. figure 4. viewer.tmpl.php file (gist) back-end/server-side modification files modified: ohms-viewer/lib/version3cachefile.class.php ohms-viewer/lib/transcript.class.php ohms-viewer/config/config.ini the customizations we made to the back-end php and config files were relatively small in nature; they were based on both the design of website we planned for the collection and the collection’s metadata itself. the first file to set up is the config.ini file. there was ample documentation on how to do this via the ohms site (weig 2014) [12]. the other back end customizations we performed were based on our own metadata needs for the collection and the needs of those who would be enhancing our metadata, namely our student workers. for the most part, we were able to fit the fields of the metadata in our collection to the template established by the ohms system. the ohms metadata template includes options for basic fields such as title, date, interviewer, interviewee, length (in time), file name, rights, keywords, subjects, and other fields.  however, our collection did contain several idiosyncratic fields that were original to the project that we wanted displayed, namely birth year, family origin, residence location, occupation, and interview number. [13] to enable these fields to then display via our viewer, we added several classes to the array defined in version3cachefile.class.php file. figure 5. adjustment of version3cachefile.class.php file (gist) we also needed to address the metadata for the index of each interview. out of the box, ohms and jplayer use seconds to determine its timestamps, so for each point listed in the index portion of the xml file, a corresponding second is required. given that our original metadata for the indexes was recorded in minutes, we decided to adjust the transcript.class.php file to multiply the “point” field by 60 to accommodate the input of minutes in the xml file rather than seconds. this eventually allowed our student workers to expedite their work with the indexes. figure 6. adjustment of transcript.class.php file using student workers to create complex metadata the digital initiatives lab at the university of idaho library employs between four to eight student workers and interns. these students assist with scanning, file management, and the creation of metadata for many digital collections, gaining technical skills along the way. for this particular project, students participated in active learning tasks through the preparation and processing of oral history interviews, including the editing of complex xml files (wilton 2011) [14]. in order to bring student workers into the metadata creation and wrangling of xml files workflow, we first tested out specific tasks and goals on staff and created a guide to assist with students who were not familiar with xml editing. eventually, students were assigned specific interviews in batches of approximately ten xml files. students worked in oxygen with surrogate files, for which they received brief introductions to the software before starting work. students were assigned four goals for each xml document: create a searchable index with corresponding timestamps, create a searchable transcript, and assign keywords and locations for each individual interview using getty art and architecture thesaurus. creating a searchable index with corresponding timestamps many indexes found in the transcript pdf file listed minutes and brief descriptions on each side of a cassette tape; students copied and pasted the time segments and synopses into “points” in the xml document. a point consisted of slots for time, title, and synopsis. students counted how many points were in each index and created the correct number of points in the xml file. figure 7. screenshot of uncustomized xml file viewed in the test server. figure 8. screenshot of index from pdf transcript. the timestamps needed to be manually adjusted to correlate with the digital length of time versus the 30 minute increments found on the original cassette tapes. we created a timestamp cheat sheet to help students adjust correctly depending on the side of the cassette tape they were working on. figure 9. cheat sheet for adjusting time based on cassette side length. students were then asked to create a title for each synopsis, since that wasn’t part of the original metadata. we asked them to create a phrase or a few words that describes the corresponding segment within a minute or two. students were asked to skip this process and make a note in their progress spreadsheet in cases of garbled ocr text from the transcript upon copying and pasting into the xml file, as well as in cases where there was no transcript at all or a transcript without an index. figure 10. screenshot from oxygen editor of xml file looking at a point (timestamp, title, synopsis), created by students. next, while the interview was still fresh, students assigned keywords and locations for each interview. students entered this information in their progress spreadsheets and used a controlled vocabulary text file that contained entries from getty art and architecture thesaurus. location keywords were generated from the original metadata but then added to by students to include not only where the narrators were born or lived, but what areas the interviews discussed at length. creating a searchable transcript students then created searchable transcripts within the xml file when they were available for each interview. if there was no transcript, students were asked to copy in the phrase “transcript not available” in the transcript tag; they then skipped to indexing. if a transcript existed, students copied the ocr text from the pdf and dumped it into a text file using notepad++. upon the dump, the student workers deleted all text that occurred before the actual transcript began. students then cleaned up the ocr text by getting rid of a number of common symbols by using find/replace. common symbols included < , > , & , • , £ , © , and ® . students also did a quick scan through the text file to delete any large text strings errors. we did not have students check the text against the transcript, as that was too time consuming. these files were saved as ‘scrubbed’ text files. figure 11. text file filled with ocr errors. students tracked how much time it took per interview so we could get an estimate for the project, as well as which students worked faster and could take on the harder files. times ranged from five minutes (interviews with no transcript or index) to 80 minutes (large, complex interviews with multiple issues), with the average time of approximately 32 minutes per interview for student workers. once students completed these steps, the digital projects manager completed quality control checks on the xml files. this included adding breaks between speakers listed by their initials in the transcripts to make for a more readable document online; copying the final scrubbed transcript into the students’ interview xml file; performing checks on the titles, keywords, and locations; and finally grabbing the code and moving it into the final xml files located on the server, which students do not have access to. students then clicked on the synopses to confirm that the xml file was indeed working correctly. figure 12. final view of transcript in viewer with ocr corrections and breaks. issues with the process/metadata staff provided quality control checking on keywords and locations generated by the student workers, as consistency proved to be an issue when considering the subjectivity of classification coming from six different people. as more interviews were completed, additional keywords were added to speak for the multitude of topics found in each interview. figure 13. excel spreadsheet of all interviews and corresponding keywords and locations. another issue that occurred through student generated metadata was that certain audio files did not match up with the corresponding transcript and/or index. we left these files towards the end, and the project manager decided it was worthwhile to listen to certain interviews and figure out where the errors occurred, or in some cases, to listen to the audio in real time to generate points (minute, title, and synopsis) for some interviews. these files took significantly longer to adjust and were done on a case by case basis. building out the website using xsl templates once the student work and quality control was completed, we were ready to design the discovery interface and information architecture for the collection. we are committed to providing several different modes of access—timelines, subject clouds, maps, and other collection-determined modes—to each of our digital collections.  these interfaces are always driven by the metadata provided in the collection, and the latah county oral history had metadata both related and unique to our other digital collections from which to determine our means of discovery. for instance, we knew we would want to provide a map for users to browse the locations covered by the interviews, as well as subject clouds from which to discover the breadth and extent of the locations and keywords present in the collection. before these visualizations were produced [15], we wanted to address the more idiosyncratic metadata we had for the collection, which included the biographical information and list of interviews for each interviewee. as noted above, the original metadata for this collection included information on each interviewee’s family origins, residence, occupation, and year of birth. we wanted users to be able to quickly search and access individuals and their interviews by these attributes and by their name. we also wanted to provide similar means of browsing sequentially between interviewees as we did for the interviews themselves. so we first constructed a page that listed each interviewee, along with their biographical information, and linked to a main page for each person. next, we needed to create each person’s page, so we built a basic template page in xsl for any interviewee, listing, much like the original guide did, his/her name along with biographical information and a list of each interview he/she participated in  together with its synopsis. figure 14. xsl file for producing a page for an interviewee this structure of information and the subsequent sites created via the template established in xsl was based on the structure of our updated, master  xml file, which nested all the individual xml interview files within the information we obtained from the initial processing of the ocred guide to the collection. to build this master xml file, we first joined all the interview xml files together, then deleted the transcript field entirely to limit the size of the file. this very large xml file sometimes caused oxygen to crash, but we were able to use it carefully along with the customized xsl sheets to create pages for each individual with the information listed above.  the xsl file and operation extracted the relevant information for each individual and interview and displayed that according to our layout. figure 15. typical page for an interviewee. we also wanted to have some piece of visual interest on each page, so we used the location data we had to create automatically generated google map images of latah county with markers based on the latitude and longitude of each location referenced in the interview. the “src” for each image looked like the following, with each latitude and longitude coordinate creating the red marker you can see in the image above: we were also able to use some images from the latah county historical society of the interviewees to add to select pages. to do this, we simply added a field to the large xml file that listed the location for the image on our web server. we then added an xsl:choose command in the xsl sheet to say that if an image was present, then put the html code to display the image on the page. figure 16. xsl choose code for adding a picture to a page. figure 17. page for an interviewee with a photo. building the front page and central feature using isotope having built the majority of pages for the collection, including pages for each interviewee and interview, we finally needed to address the central issue of this project, namely how to structure the site as a whole so as to use our extensive metadata to drive discovery of the interviews. essentially, we had created the wide foundation of our collection and now needed to organize the site and design our interfaces to best point users to individual interviews of interest, at which point they could further search and browse the interviews’ content themselves using the metadata created by the original directors of the projects and enhanced by our students. in order to spark curiosity and encourage use of the collection, we also needed to provide a user interface that was engaging visually and interactively while at the same time intriguing a user via the interviews’ metadata. unlike most of our collections, the latah county oral history collection had no inherent visual objects to design around, so this was a particularly challenging aspect of the design process.  ultimately, we decided to use isotope, which is a javascript application created by metafizzy, a small web development shop led by david desandro that dynamically arranges and filters html objects. isotope adjusts the layout of objects on a web page based on the width of the browser being used to view the page. the application is not simply for images, however, as it also offers means to filter the html objects appearing on a screen based on classes and other variables added to the connected html objects. we have used isotope in the past to create image galleries for our digital collections [16], but this collection represented the first time we had attempted to implement isotope’s filtering capabilities. for the latah county oral history collection, we wanted  a user to be able to search and browse the interviews using keywords or locations in order to quickly discover those in which he/she was interested. to create these filterable objects, we again extracted the requisite information from our large xml file using a customized xsl file,  structuring the information in each container to contain the interviewee, interview number, date of the interview, short synopsis, and a list of the assigned keywords and locations. we tried several different means of filtering before we decided that a search box best fit our purposes. this would allow a user to input any word they might think of to limit the selection of interviews presented. we also wanted to include suggested keywords and locations to guide the user’s browsing, as well as connect the keywords and locations listed for each interview to encourage discovery. to do this, we added additional jquery code to enable the value of any “vocab” link to replace the value represented in the search box. this enabled us to utilize the search box function and use our keywords and locations to limit the interviews listed. figure 18. jquery for filtering objects via replacing the value of the text box with the value of the “data-filter” attribute within a link (gist) once we had this function built, we added lists of suggested keywords to the side of the isotope container, as well as a short introductory paragraph. at this point, we also wanted to enable a dynamically filtered list of interviews based on the incoming url. to address this issue, we added additional jquery commands to extract any text after a hash symbol in the url. upon the loading of the page, this extracted text was then inputted into the search box itself, filtering interviews based on the text entered. for instance, the url http://www.lib.uidaho.edu/digital/lcoh/index.html#railroads will filter the interview records displayed on the main page so that only those records that include the word “railroads” in their html will display. this development also enabled us to link all of our additional visualizations, including our locations map and subject tag clouds back to filtered front pages as well. figure 19. jquery code allowing filtering based on hash substring. figure 20. example of a filtered front page for interviews related to “murder” in the end, this feature enabled a circular architecture for our web site, one in which almost all links in the collection lead back to a filtered homepage. we went with this feature because we wanted our users to be focused immediately and continuously upon discovering the interviews themselves, especially since the metadata available via ohms on the interview pages themselves would further allow for deep investigation and speedy discovery of pertinent portions of the interview.[17] site release and next steps the latah county oral history collection website officially launched in april 2015 with a public program open to the community. over 80 people attended the informational event that featured the history of the collection, its new digital presence, and how students have already made use of the website through research. latah county historical society announced that it received a grant to continue collecting oral histories from the county and encouraged people to sign up or refer family and friends. this partnership between the latah county historical society and the university of idaho will continue to ensure these future interviews will be preserved through the digital collection for future citizens and students to enjoy. notes [1] ritchie, d. 2011. introduction: the evolution of oral history. in: ritchie, d, editor. the oxford handbook of oral history. oxford:  oxford university press. p. 12. [2] boyd, d. 2011. achieving the promise of oral history in a digital age. in: ritchie, d, editor. the oxford handbook of oral history. oxford:  oxford university press. p. 291. [3] boyd, d. 2011. achieving the promise of oral history in a digital age. in: ritchie, d, editor. the oxford handbook of oral history. oxford:  oxford university press. p. 295, 286. [4]  hubbard, cr. 1957. mineral resources of latah county. moscow (id): idaho bureau of mines and geology. http://www.idahogeology.org/pdf/county_reports_%28c%29/c-2.pdf, p. 1-2. [5] schrager, s. 1978. guide to the latah county, idaho oral history collection. moscow (id). p. x. [6] schrager, s. 1978. guide to the latah county, idaho oral history collection. moscow (id). p. xiii. [7] this points to a national trend in digitization here that is often little acknowledged. so much of the metadata used in many digital collections across the us and abroad was gathered by the excellent librarians and volunteers that worked in the library and special collections and archives departments of their respective libraries and archives years before any digital collections were born. [8] boyd, d. 2014. one bourbon, one wine, one beer: academic alcohol archives that document the cultural history of a community. northwest archivists annual conference; 2014 may 30; spokane, washington. http://northwestarchivistsinc.wildapricot.org/conferenceschedule2014. [9] boyd, d. 2013. ohms: enhancing access to oral history for free. oral history review; 40(1): 95–106. available from: doi:10.1093/ohr/oht031. [10] to use the tool, users first need to create an ohms account through kentucky by signing up for access. [11] one thing to note about jplayer: we initially were serving our mp3 files off of our contentdm server, which was an iis windows server. however, jplayer had difficulty reading the length of the mp3s coming from this server, so we moved all the mp3s to our production web server. this cleared up the problem. [12] weig, e. 2014. installing & customizing the ohms viewer. http://www.oralhistoryonline.org/wp-content/uploads/2013/06/ohms-viewer-installation-v3-2-13.pdf [13] when we originally developed the metadata schema for this collection, we based it off of a basic dublin core schema we often used in our contentdm system, then added these original fields to the collection. [14] wilton, j. 2011. oral history in universities: from margins to mainstream. in: ritchie, d, editor. the oxford handbook of oral history. oxford: oxford university press. p. 477. [15] we use google fusion tables to create our maps and tagcrowd.com to create our subject clouds, and we did the same for this collection, collecting latitude and longitude coordinates to add to our google fusion table for each of the locations pertinent to latah county and then cleaning and processing the lists of keywords and locations via tagcrowd.com’s interactive interface. for more information on this process, see previous work from the department, particularly, devin becker’s article “each item its own time and place: using google fusion tables and simile timeline to map the ott historical photograph digital collection” (coins) [16] see the university of idaho’s northwest historical postcards collection: http://www.lib.uidaho.edu/digital/postcards/. [17] as many who work with digital collections can guess, the development process for this website and digital collection was not the smooth, sequential process we present above. we present the narrative here as such, however, to expedite and assist others who might like to envision or apply the procedure. in reality, the metadata was being created at the same time as the interface, and so the complete website had to be overhauled several times throughout the process to include new metadata and additional information several times. thankfully, having our xsl style sheets expedited the process of updating the collection each time new additions or revisions were added to the general xml files. about the authors devin becker is the digital initiatives and web services librarian at the university of idaho. his first book of poetry, shame | shame, won the a. poulin jr. poetry prize and was recently released by boa editions, llc. erin passehl-stoddart is the head of special collections and archives at the university of idaho. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – book review: the success of open source by steven weber mission editorial committee process and structure code4lib issue 1, 2007-12-17 book review: the success of open source by steven weber the success of open source by steven weber details the history, process, motivations, and possible long-term effects of open source software (oss). weber’s book can be used as a set of guidelines – a description of a framework – for building software solutions for the computing problems facing libraries. weber, s. (2004).the success of open source. harvard university press. isbn: 0674012925 (coins) by eric lease morgan introduction the success of open source by steven weber details the history, process, motivations, and possible long-term effects of open source software (oss). this scholarly yet easy-to-read, well-written, and provocative book is worth the time of anyone who wants to understand how open source software is affecting information technology. using linux as its primary example, the book describes how the process of open source software may affect business & economics, methods of governance, and concepts of intellectual property. it is also a great read for those of us librarians who desire to play a role in the building of “next generation” library catalogs and other library-related information systems.my acquisition of the book was rather embarrassing, and at the same time typical. as the leader of an open source software project called mylibrary, i asked some of my fellow hackers and open source software aficionados for advice on how to promote mylibrary and build a larger community around it. one of the suggestions was to read weber’s book. like most people, i searched google for the title, and google returned it as the first hit. i was able to read the entire text online, if i desired. i didn’t. the librarian in me then went to worldcat where i learned the book was located down the street in another academic library. i could borrow the book for free. all i had to do was visit the library and check it out. i didn’t. instead, i looked the book up on amazon.com and found a “new” copy from an amazon.com associate. twelve dollars and four days later my book arrived. easy. convenient. cheap. food for thought.the book can be divided into four overarching topics: the history of open source, the process of open source software development business models and open source software’s relationship to the idea of a “commons”, and finally, a summary as well as a look to the future describing how the process of open source software might affect other human endeavors. history of oss weber traces the history of open source software from its roots in at&t unix and the berkeley software distribution (bsd) to the present day linux operating systems. weaved throughout is the development of networking technologies, specifically the internet. the history brings to light two very influential computing philosophies: the “unix way” and software as something to be bought and sold.the first computing philosophy is the “unix way”, an outline of three engineering principles for making good software: write programs that do one thing and do it well, write programs that work well together, and write programs that handle text streams because that is a universal interface. software that adheres to these principles is typically considered more useful than software trying to be all things to all people. such software is modular, portable, easy to create and maintain, and can be applied in any number of settings.the second philosophy revolves around ideas of intellectual property. suppose i spend time and creative energy writing a piece of software. through this expenditure i have the right to sell the software to other people in order to gain compensation for my efforts. software, like other goods, can be exchanged for other things of value, namely money. moreover, if people copy my software and give it other others, then such a process is just like stealing from me since i am not being compensated for my efforts. this was the attitude of bill gates as he stated it in his “open letter to hobbyists” as they distributed his implementation of the basic programming language in order to run programs they had written against it. this perspective regarding software as something to be bought and sold ultimately led to the creation of microsoft. at the time of gates’ writing of the 1976 “letter” computers were bought and sold. software sort of just came along for the ride.while it is important to note that i, the author of this review, am certainly an advocate for open source software, the book takes no sides one way or another. instead, throughout the book, steven weber tows a middle ground by simply asking questions and then does his best to answer them as objectively as possible. the book neither advocates nor condemns open source software. it simply observes the environment and makes generalizations accordingly. oss as a process not a thing the chapters i found most interesting described the open source software process, how it works, and the motivations of its participants. i learned from the book that open source software is more about a particular process and less about a thing or product. consider two statements a priori: software can be copied an infinite number of times and not denigrate the original version, and globally networked computers allow the tiniest numbers of like-minded individuals to find each other easily. given such an environment the open source software process flourishes, and that process is outlined here: someone has a computing problem they want to solve — an itch to scratch. the person builds on the good work of others and writes a computer program. the person “freely” shares their software with others under some sort of license agreement. a community forms along with norms of behavior and guidelines (governance) for contributing back to the solution. the software grows and matures, hopefully go to step #1 until the software is “done” or until someone else wants to take on the leadership role. the resulting open source software application, the book points out, is not necessarily better (or worse) than “closed source” software. instead, it is simply different. it is more easily modified. it is vetted through the eyes of end-users — a set of self-described pragmatists wanting to scratch their own itch. furthermore, since the process easily accommodates the philosophy of letting a thousand flowers bloom, the resulting software is not necessarily designed for the mass market: the software is not aimed at a lowest common denominator.what motivates open source software participants? weber shies away from the altruistic motives espoused in eric raymond’s the cathedral and the bazaar. instead, weber imagines that the motivations stem from a desire for artistry & craftsmanship, the desire to create better software, and the desire for recognition from peers (“ego-boosting”). many of the people who participate in open source software seem to enjoy puzzle solving. some of them like reading and writing beautiful code — software as poetry. they are engineers looking to build better solutions to known problems, to “build a better mouse trap”. like scholars and researchers in academia, peer-review is an important aspect of the work, and if you write something that is used (cited) by many people, then in the eyes of others your value is increased. oss and business the sections and chapters regarding open source software and business are probably the most significant aspects of the book. they cover issues that are the least understood by the wider community including the definition of “free”, the concept of property and the “commons”, the differences between the bsd and gpl licenses, how these differences effect business opportunities, and business models in an environment where the primary thing to be exchanged for value is bound by rights of distribution as opposed to exclusion.in our increasingly commercial society, the concept of “free” software is something most people find confusing. weber compares and contrasts “free” from the point of view of the free software foundation (led by richard stallman) and the point of view of dot-com companies such as red hat (a commercial distributor of open source software). in both cases the concept of “free” should be equated with the word “liberty” as opposed to “gratis”. however, the fsf approach has a moral slant while the red hat approach emphasizes practicality and the ability to easily improve software solutions.the concept of property plays a big role in business models. how can you sell something that is “free”? how can you earn money on a thing that is a part of the community commons? who owns this intellectual property? these questions can be answered, according to weber, by interpreting the bsd and gnu (or general) public license. in both cased the software is give away gratis. the differences lie in redistribution. under the gpl license, new software based on gpl-licensed software must be re-distributed under the gpl or a less restrictive license. the bsd license does not have this stipulation; new software based on bsd licensed software does not have to be redistributed for “free”. mac osx is an example of bsd-based software which has been commercialized: apple , inc. started with bsd unix, enhanced it, then redistributed their modified version for a fee. weber outlines a number of possible business models for open source software: support sellers, loss leaders, “sell it, free it”, accessorizing, service enablers, and branding. for each of these models weber points to a number of examples. the book’s summary the last two pages of the book summarize much of weber’s observations, and listing them here feels a bit like spoiling the ending of a mystery novel. the effective open source software process/project needs to take into account and support: disaggregated contributions, the need for a critical mass of users, peer review, the positive effects of the internet, belief that a small group can generate something truly useful, and a voluntary community. weber also outlines the characteristics of effective agents (open source software participants). they: can judge the viability of the product, can make an informed bet their contributions will be used, are driven by things beyond simple economic gain, gain personal knowledge through the process, and hold a positive ethical valence towards the process. oss and libraries naturally, i read the book through my rose-colored glasses of librarianship. after reading it and combining it with more recent personal experiences, i am now less of a “believer” in open source software. i take away a more realistic perspective on the definition of open source, its process, and what motivates its participants. this does not in any way diminish my belief that the open source software process can benefit the library community and therefore library users.throughout the book i kept comparing the kernel of the linux operating system to the integrated library system (ils) of libraries. both provide fundamental interface functions between two entities. in the case of an operating system kernel, the interface is between hardware and people. in the case of an ils the interface is between a library and patrons.many library open source projects already exist. the extent they meet with continued success and wider adoption, i predict, will be measured by the extent that they can accomplish the following things. first, an easy-to-understand vision statement needs to be outlined by one or more people who possess leadership qualities. second, those leaders need to amass the resources required to make their vision a reality. third, they need to put their vision into practice allowing as many people to participate as possible. fourth, start small and work up. encourage the building and re-use of existing core applications, such as databases, indexers, editors, and server platforms. make sure the applications are modular and standards-compliant. practice the unix way. make it work first, then improve things. don’t even attempt to create the perfect system the first time. the process won’t be quick. the process won’t be easy. the process won’t be “free”. on the other hand, the process will empower and enable the profession. it will give it increased choice and opportunity. weber’s book, the success of open source, can be used as a set of guidelines – a description of a framework – for building software solutions for the computing problems facing libraries. about the author eric lease morgan <emorgan@nd.edu> is the head of the digital access and information architecture department at the university libraries of notre dame. he considers himself to be a librarian first and a computer user second. he and his fellow teammates have primary responsibilities for the university library’s website, the campus-wide search engine, and a number of digital library projects. he is also on the editorial committee of the code4lib journal. in his spare time he can be seen folding defective floppy disks into intricate origami flora and fauna. tags: book review subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – machine learning based chat analysis mission editorial committee process and structure code4lib issue 50, 2021-02-10 machine learning based chat analysis the byu library implemented a machine learning-based tool to perform various text analysis tasks on transcripts of chat-based interactions between patrons and librarians. these text analysis tasks included estimating patron satisfaction and classifying queries into various categories such as research/reference, directional, tech/troubleshooting, policy/procedure, and others. an accuracy of 78% or better was achieved for each category. this paper details the implementation details and explores potential applications for the text analysis tool. by christopher brousseau, justin johnson, curtis thacker introduction in 2019 whitchurch and merrill published a qualitative analysis of chat transcripts between librarians and patrons of brigham young university’s harold b. lee library (whitchurch and merrill, 2019). their dataset was created by coding 4,475 chat transcripts recorded during 2016. they coded into 25 categories – described below. using this dataset, the byu library attempted to create a machine learning model that will programmatically code a chat transcript. using the resulting model, we hope to programmatically code about 9 years or 70k additional transcripts. this larger dataset would allow for analysis of trends over time. background background on machine learning “machine learning is the study of computer algorithms that improve automatically through experience” (mitchell, 1997). a machine learning model is an algorithm that uses patterns in data and previous examples to learn to perform a task. detecting a cat in an image and predicting how much a 3,000 square-foot house in boston will be worth in 2025 are examples of tasks that can be performed by using machine learning. figure 1. supervised learning (classification/regression) | unsupervised learning (clustering) there are two basic paradigms for machine learning models (hereafter called models), supervised and unsupervised learning. a supervised learning model uses patterns found in example data to find relationships between inputs and outputs such as whether a cat appears in a given picture. unsupervised learning algorithms are used in pattern detection and descriptive modeling tasks. background on text classification for our experiment, there are two tasks to accomplish, sentiment analysis and classification. sentiment analysis is a classic natural language processing (nlp) task which tries to predict the overall positivity or negativity of a statement or utterance. our experiment required us to use text to provide a variety of labels for a block of natural language text. nlp is a broad field in machine learning, spanning from text processing to speech recognition. one of the main focuses and most common tasks of nlp is creating what are called language models. language models are essentially large records of computer-generated rules pertaining to a language, and there are many different types of language models. in order to minimize the amount of time it takes to train but maximize the accuracy and learning rate, we’ve opted to use a pre-trained language model for our task, and fine-tune or adapt it to our specific dataset. bert (devlin, 2019) is a sophisticated unsupervised machine-learned language model created by google. its purpose is to take natural language in plain text and represent it as a sequence of numbers that capture the nuance and meaning of the language. doing this enables the creation of other models that perform tasks like sentiment analysis and translation. for all the tasks described in this paper, we used dilbert (sanh, 2020), a smaller version of bert that doesn’t sacrifice accuracy or efficiency. other very common tasks in nlp are text processing, part-of-speech tagging, tokenization, and sentiment analysis; our model uses all of them except for part-of-speech tagging. text processing or pre-processing involves cleaning data so that it can be used without introducing any unnecessary bias into the results. for example, in our dataset all of the customer interactions originally contain ip addresses and other personal data that isn’t relevant to the task we’re trying to complete. we don’t want to introduce bias into the results that we aren’t prepared to deal with like the idea that only a certain ip address ever asks questions about certain topics. tokenization is the act of splitting up a text by tokens, in our case individual words. in other cases, those tokens could be morphemes or prefixes/infixes/suffixes, etc. some language models are based on tokenization, such as the bag of words statistical model, which counts each unique token in order to draw important conclusions from the text. others, while likewise based on tokenization, do not use only tokenization to draw conclusions, like the term frequency-inverse document frequency model, which deems words that appear fewer times as being more important, but it’s safe to assume that where a language model exists, some form of tokenization is done. we did not use part-of-speech tagging in our model but it is often used to introduce bias for grammatical structure into language models, usually for models where grammar and syntax are important for accurate results, such as those models used for translation or summarization of text. back to the first task we completed, sentiment analysis combines several of the nlp methods above in order to get results. it’s a corpus-based method, meaning that it’s either needed to make our own corpus of “known” words or to piggy-back off of an already created corpus. in our case, we used both methods. we utilized bert’s pre-trained word embeddings as the base of our corpus, and during fine-tuning whenever we came across a new word that wasn’t in the current corpus, we added a new entry to represent that word. these entries in the corpus are represented by vectors. the vectors, or word embeddings, take into account how frequently the word shows up, but are also based the word’s placement in a sentence along with its perceived semantic meaning. all of these are taken into account for determining the overall positivity/negativity of a given word. once we have those vectors, to analyze sentiment, we add up each individual word’s “score.” from there, we determine how positive/negative the whole utterance is. we aren’t analyzing straight sentiment, although our model has learned to consider the overall sentiment of a chat as a statistically significant factor in determining the satisfaction of the patron. our second task is classification which is predicting which of two or more categories (or classes) a given input fits into. for the related task of binary classification, prediction is limited to just two classes such as true or false, cat or dog, black or white, dead or alive, etc. we’ve opted for much more of a black-box model, as opposed to other more easily understandable models for classification. instead of manually injecting bias by predetermining rules for the classification to follow, we allowed the language model bert to follow its algorithm to “learn” how it should classify each input. this approach leaves us in a position where we will never be one hundred percent certain about everything that the model learned or which features it deemed especially important for determining the classifications it gives. the justification for this approach, even with what could be taken as a huge drawback is twofold: 1. as we previously stated, we do not want to inject any bias into the model that we aren’t prepared to deal with. that includes bias that we as researchers have about whether we ourselves would be satisfied with a particular interaction with librarians. 2. we have a dataset of human-labeled-and-verified examples, meaning that if our model gets a good score generalizing on that dataset, we are guaranteed a certain amount of confidence in its ability to generalize on data within this very limited space effectively. an example: say we were to analyze a chat starting with this question from a patron, “patron: when does the library close on friday?” this question within our model will receive a neutral score, as it’s neither positively nor negatively worded. the remainder of the chat will determine the overall sentiment of said chat. if the librarian gives a polite and accurate answer, and the patron then says thank you at the end of the chat, the overall chat will be classified as weakly positive. if the librarian takes a long time to respond and ultimately gives inaccurate information, the model will still classify the chat as weakly positive if the patron at the end of the chat indicates that they are happy with the service. if they are unhappy with the service, and they indicate that at any point in the chat, it will be classified as negative. background on library chat a review of the literature shows that libraries primarily improve their chat services in two ways: first, by making technological improvements to the service and second, by evaluating the service and improving it through training and support to library employees. a sentiment analysis tool would help in regard to the second method. evaluation of chat services is typically done using a qualitative analysis or evaluation of the chat service. this is done via a review of chat transcripts using an established rubric or dataset to evaluate all chats in the same manner. there are many publications that demonstrate the usefulness of this approach and describe the method of coding typically used for these evaluations. of the studies reviewed, the following examples are included in more detail to show the methods used to evaluate chat. these evaluations are one review of the interactions based on established reference frameworks such as the acrl framework (hunter, 2019), and one study identifying answers through the evaluation of the questions being asked (moore, 2017). a 2009 study from the library at the university of guelph did an assessment of their virtual reference and instant messaging services by reviewing all transcripts from the past two years and categorizing them into five broad categories of the type of questions asked. the analysis was done by a team coding the interactions into categories that were directional, policy, ready reference, specific search, and research questions. in comparison, a study published in 2019 by a task force from berkeley college reviewed 369 chats. they focused on coding chat interactions that were identified as needing research or writing help. the researchers analyzed each of the interactions according to the acrl’s framework for information literacy in higher education. the examples included above show how libraries have typically conducted qualitative analysis of chat interactions by using teams or task forces of researchers using established rubrics and conducting coding of each transcript. these demonstrate the potential benefits that are available in creating a machine learning or sentiment analysis model to assist in this type of analysis. there has been some use of sentiment analysis within libraries already such as a study that evaluated libqual+ surveys’ open-ended comments (moore, 2017). in this libqual+ study, the author created a sentiment model using manually-coded data for its dataset to identify positive and negative comments within the survey. he took 514 coded entries from five separate surveys to create a set of positive and negative word vectors for the open comment sections of the libqual+ survey. he then ran the comments from those five surveys through his model to review the results. expansion of work like this can be applied to other word-based interactions or transcripts such as chat. these are examples of how libraries can use an established dataset to create a qualitative analysis of chats through sentiment analysis. project description we performed supervised learning on 21 different tasks, one of which was fine-grained sentiment classification, the other 20 being binary classifications. we have synthesized, or used our model to create, some analyses similar in accuracy to michael whitchurch’s projects, but without needing human researchers. we analyzed and classified 77,000 chats in 21 hours. whitchurch’s project analyzed 4475 chats with 3 people over the course of 1 year. this means that we analyzed 1720% of the project in 0.02% of the time, resulting in an almost 2000% increase in time efficiency. we specifically checked with michael whitchurch about presenting this number, and he agreed it was accurate considering the project and its circumstances. we expect this vast increase in speed to be valuable to anyone who would like to implement this project for themselves. our experiment makes use of three python code libraries: pytorch (https://pytorch.org/), transformers (https://huggingface.co/transformers/), and fastai (https://docs.fast.ai/). we used the transformers library to access dilbert from huggingface. using transformers version 2.5.0, we have easy access to the three portions of the model we need most: the pre-trained model, tokenizer, and configuration. the dilbert model has a maximum input sequence length of 512, which isn’t immediately compatible with fastai’s architecture, which is built more for working with a recurrent neural network (rnn) than a transformer model. beyond that, it’s much shorter than many of the chats we were attempting to analyze. we had to implement a custom wrapper for the tokenizer to normalize the input and the max sequence length. this allowed us to make our model as flexible as possible. in order to get our model to correctly process our data, we needed to first preprocess the text, then create a way to load it into the model. to properly load our data into the model, we rely on the fastai databunch api, which gives us an easy way to not only load the data but also shuffle and test it without requiring too much configuration. you’ll remember from the brief explanation of apis that they define standards for input and output for different types of data, and one of the reasons why this databunch api is helpful for us, is that it accepts so many different types of data, and presents the same types of output, meaning that our model is very flexible as to what types of data and what formats it can work with. based on this excerpt, https://github.com/huggingface/transformers#models-always-output-tuples (huggingface 2020), the creators of the models state that all of their pytorch-based transformer models output tuples with some variation between all models. because a goal of ours is flexibility between models, we don’t consider those variations especially helpful, and adjusting our whole architecture for each and every possible model would not be useful for our purposes. with this in mind, we created another custom wrapper that funnels the output to only the most important probabilities for our results (logits), no matter which pretrained transformer model we choose to put through it. in the last part of the setup, you’ll remember that we are working with the 21 data categories defined in michael whitchurch’s experiment. some of these are binary classification, while others are multiclass, meaning that for testing, we need to be able to seamlessly transition between the two tasks. thanks to transformers, we have easy access to these features in the form of the config for each model. the model is now set up correctly. because of all of the work we put into custom wrappers, our program works not only with dilbert, but also bert, roberta, and xlnet (all large pre-trained language models). we are training on the data collected and labeled by michael whitchurch and predicting all of the same categories. we are using adamw as an optimizer, and again fastai allows us to bundle our model up nicely with the learner api. fastai allows us to visualize our data immediately and easily after every epoch. to analyze the results, classificationinterpretation is a useful api to implement, allowing you to generate confusion matrices quickly and effectively. after that, we are able to predict any of the classes we trained on in our model, along with saving our model to use within larger systems and output our predictions as files. saving the model is especially important, as it significantly reduces prediction time, as we don’t need to retrain anything. dataset our dataset consists of 4476 chats collected in 2016. this data was collected through the harold b. lee library’s chat system and analyzed by michael whitchurch’s team and detailed in his publication. it contains 21 categories of questions that whitchurch found to be useful in determining a couple of things: are patrons generally being satisfied by the chat experience, and how do trained library professionals compare to students when interacting meaningfully through the chat. we extended the first of these questions to our project, utilizing the same categories present in the original. the data originally came in an unusable format for our model. in order to clean it up, we first replaced all personal information that came with the chats to protect our users’ privacy. now, instead of displaying studentid@ipaddress: and libraryid@ipaddress:, it displays patron and library. the next step in cleaning was replacing all of the text answers for the categories with numbers, so instead of true or false, it was 1 or 0. once our data was cleaned, we were ready to begin training and testing. something worth noting is that despite all of the effort put into cleaning the dataset, there were still several biases that inhibited the model from performing at what we would call a “perfect level,” all of which have already been discussed earlier. here’s a breakdown of the data on question types and how they are portioned throughout the entire experiment: table 1. question type breakdown question type total percentage research/reference 2936 65.6% policy/procedure 124 20.4% tech/troubles 397 8.9% directional 314 2.8%   lastly, here’s a breakdown of each of the 21 categories tested in detail: table 2. category breakdown label scale meaning 1 patron satisfaction 1-5 overall satisfaction of the patron 2 unanswered true/false whether or not a librarian or student answered the chat 3 check original chat true/false whether a given chat was the original contact, or if another chat needed to be referenced 4 premature exit true/false whether the patron left before the question was answered 5 cite source(s) true/false whether a patron was citing a source from the library 6 guided to source true/false whether the patron was successfully guided to the source 7 no source true/false whether the source actually existed at the library 8 unnecessary true/false whether the source citation was actually necessary 9 research/reference true/false question type – research question 10 directional true/false question type – directions to something on campus/in the library 11 tech/troubleshooting true/false question type – help with using or debugging technology 12 policy/procedure true/false question type – help understanding what the rules are and why they’re in place 13 inappropriate true/false whether the patron behaved inappropriately or used the chat for an unintended purpose 14 student-to-student true/false whether the patron (a student) was connected to a student employee through the chat 15 greeting true/false whether a greeting was expressed at the beginning of the chat 16 follow-up true/false whether any follow-up is required based on the chat 17 closing true/false whether a closing statement (e.g. bye) was expressed at the end of the chat. 18 campus question true/false whether a question is about a campus schedule or activity 19 perceived inaccurate true/false whether the answer given by the librarian is completely accurate 20 perceived incomplete true/false whether the question needed more information than answer given by the librarian contained 21 employee inappropriate true/false whether at any time the employee behaved inappropriately during the chat, or if they used the chat outside of its intended purpose   results our project was largely a success, and we expect that anyone who implements this model will experience the same significant margin of improvement over more traditional methods of analysis. this project was very useful for our purpose, which in this case was providing a means for quick and accurate analysis of customer interactions. this analysis provides a bunch of meaningful data for a variety of purposes including employee training and hr management, customer satisfaction and quality assurance, and overall efficiency. this data can be used as the basis for a custom chatbot specific to a company’s data and market, or as a helpful start to building better-customized training for employees. its main usage, however, is beginning a library’s journey towards analyzing their patron chat interactions. in building a custom dataset for fine-tuning on your own data, at least a couple thousand examples of labeled data are needed, be those chats, reviews, or just general email interactions. our model accepts csv, xlsx, or tsv formats. you should only have to run through training one time to get accuracy in the high 80’s or low 90’s with a good dataset. table 3. satisfaction confusion matrix x – predicted y – actual dissatisfied/frustrated neither satisfied above and beyond dissatisfied ~2% <1% <1% 0% neither 0% 8% 4% 0% satisfied 0% ~2% 82% 0% a&b 0% 0% <1% ~2%   table 4. employee appropriateness confusion matrix x – predicted y – actual appropriate inappropriate appropriate 100% 0% inappropriate 0% 0%   table 5. training data for satisfaction epoch training loss validation loss accuracy error rate time elapsed 0 0.357155 0.375009 0.891374 0.108626 00:39 1 0.320672 0.391226 0.891374 0.108626 00:41 2 0.232104 0.412163 0.900958 0.099042 00:37   table 6. category breakdowntraining data for employee appropriateness epoch training loss validation loss accuracy error rate time elapsed 0 0.000000 0.000000 1.0 0.000000 00:25 1 0.000000 0.000000 1.0 0.000000 00:29 2 0.000000 0.000000 1.0 0.000000 00:23   with the results, we can look at trends over the last few years based on the criteria we used. we can identify potential issues based on the results and use these for the purpose of evaluation or improved training. being able to fine tune the results can also help in identifying some of the underlying issues that might be present in the service being provided. fixing these issues is an opportunity to improve a service by understanding better how the interactions have progressed. an example of this is using the satisfaction rating to see if there have been any trends in our ratings in chat interactions and identify potential issues that might be present within the service being provided. looking at graph 1, you can see that our number of satisfied chats have been decreasing the last three years. while our total number of chats have decreased as well, this alone does not account for the decrease. comparing ‘neither’ and ‘dissatisfied’, we can see that these two categories have stayed roughly the same across the three years. in comparison, the ‘satisfied’ category is seeing a marked decrease. identifying this trend can help in finding areas for improvement in our chat services. this can be done through improvements to the service and to training of the employees who handle all the chat interactions. with the results available we can spot the trends and do more in-depth analysis to improve our service. two other categories we can look at for potential value are the unanswered group and the wait time. using these results, we can track how long it is taking for chats to be answered and identify the number that go unanswered. these are areas we can achieve focused improvement through training by working to answer chats and make sure that they do not go unanswered more quickly. for example, in 2018 we had 1422 chats have a wait time of 1 minute or longer and had 739 go unanswered. these are areas we need to focus on in conducting training with employees to address these issues. we can provide better customer service and these results give us the opportunity to establish reliable benchmarks that we can track and use in annual evaluations and in developing focused training. these are just examples of some of the benefits available within the results we were able to get from the text classification model we developed for our chat interactions.   conclusion having started out with an initial dataset and a goal to programmatically implement an analysis based on the dataset, the byu library was able to create a sentiment analysis chat tool. the tool gave us the ability to quickly and accurately analyze all saved library chat transcripts based on the initial dataset created by michael whitchurch. the data obtained was highly accurate and useful for the purpose of analyzing the chat interactions. so, while we accomplished our stated goal, further development and growth is needed. first, expansion of the dataset to eliminate or correct any bias is needed. this will help to refine the results and improve accuracy. expansion of the dataset will also help in better tuning the chat analysis tool to meet the needs of supervisors in evaluating chat interactions. second, working with library staff to identify how best to use the tool and what role it will play in chat analysis and evaluation in the future. this is an equally important step, and it will help provide direction for future development and show how best to use the tool. for example, while our current dataset can give some good baselines on evaluating chat interactions between staff and patrons, it might be better refined to be used in future training and evaluations. with these future modifications implemented to correct bias and bring the tool more into line with user needs, we feel the product will be a viable chat evaluation tool for libraries to analyze chat interactions. about the authors christopher brousseau is an nlp research scientist at lifepod, where he creates, analyzes, and manages automatic speech recognition and text processing functionalities for their chatbot system. chrisbrousseau304@gmail.com justin johnson is the collections access librarian at the byu lee library, which includes managing the main help desk and chat. justin_johnson@byu.edu curtis thacker the former director of the research & development, and discovery systems teams at brigham young university’s (byu) harold b. lee library. his research interests include machine learning, open source software, empirical software engineering. in addition to a m.s. in computer science from byu, curtis has worked in libraries for 13 years. curtis.thacker@byu.edu bibliography devlin j, chang mw, lee k, toutanova, k. 2019. bert: pre-training of deep bidirectional transformers for language understanding. arxiv.org[internet]. [cited 2020 june 27]. available from https://arxiv.org/pdf/1810.04805.pdf. howard j, ruder s. 2018. universal language model fine-tuning for text classification. association for computational linguistics [internet]. [cited 2020 jun 27]; 29(5):328-339. available from: https://www.aclweb.org/anthology/p18-1031/ hunter j, kannegiser s, kiebler j, meky d. 2019. chat reference: evaluating customer service and il instruction. reference services review. 47(2):134-150. available at http://jlisnet.com/vol-7-no-2-december-2019-abstract-2-jlis. mitchell t. machine learning [internet].  mcgraw hill. 1997[cited 2020 june 27]. available from http://www.cs.cmu.edu/afs/cs.cmu.edu/user/mitchell/ftp/mlbook.html. moore m. 2017. constructing a sentiment analysis model for libqual+ comments. performance measurement and metrics 18(1):78-87. rourke l, lupien p. 2009. learning from chatting: how our virtual reference questions are giving us answers. evidence based library and information practices 5(2):63-74. sanh v, debut l, chaumond j, wolf t. 2020. distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arxiv.org[internet]. [cited 2020 june 27]. available from https://arxiv.org/pdf/1910.01108.pdf. whitchurch mj, merrill, e. 2019. chat response competency: library professionals vs. undergraduate student employees. journal of library and information sciences dec 2019 7(2):10-23.   appendix i: custom transformer process custom transformer tokenizer class: class transformersbasetokenizer(basetokenizer): """wrapper around pretrainedtokenizer to be compatible with fast.ai""" def __init__(self, pretrained_tokenizer: pretrainedtokenizer, model_type = 'bert', **kwargs): self._pretrained_tokenizer = pretrained_tokenizer self.max_seq_len = pretrained_tokenizer.max_len self.model_type = model_type def __call__(self, *args, **kwargs): return self def tokenizer(self, t:str) -> list[str]: """limits the maximum sequence length and add the special tokens""" cls = self._pretrained_tokenizer.cls_token sep = self._pretrained_tokenizer.sep_token if self.model_type in ['roberta']: tokens = self._pretrained_tokenizer.tokenize(t, add_prefix_space=true)[:self.max_seq_len 2] else: tokens = self._pretrained_tokenizer.tokenize(t)[:self.max_seq_len 2] return [cls] + tokens + [sep] the basic databunch api from fastai, this is the portion that we did: databunch = (textlist.from_df(train, cols='phrase', processor=transformer_processor) .split_by_rand_pct(0.1,seed=seed) .label_from_df(cols= 'sentiment') .add_test(test) .databunch(bs=bs, pad_first=pad_first, pad_idx=pad_idx)) custom transformer wrapper class: # defining our model architecture class customtransformermodel(nn.module): def __init__(self, transformer_model: pretrainedmodel): super(customtransformermodel,self).__init__() self.transformer = transformer_model def forward(self, input_ids, attention_mask=none): logits = self.transformer(input_ids, attention_mask = attention_mask)[0] return logits transformer config for our model: config = config_class.from_pretrained(pretrained_model_name) config.num_labels = 5 the fastai learner api that combines everything we’ve created together: learner = learner(databunch, custom_transformer_model, opt_func = customadamw, metrics=[accuracy, error_rate]) training loop: learner.save('untrain') learner.load('untrain'); learner.freeze_to(-1) learner.lr_find() learner.fit_one_cycle(1,max_lr=2e-03,moms=(0.8,0.7)) learner.save('first_cycle') learner.load('first_cycle'); learner.freeze_to(-2) lr = 1e-5 learner.fit_one_cycle(1, max_lr=slice(lr*0.95**num_groups, lr), moms=(0.8, 0.9)) learner.save('second_cycle') learner.load('second_cycle'); learner.freeze_to(-3) learner.fit_one_cycle(1, max_lr=slice(lr*0.95**num_groups, lr), moms=(0.8, 0.9)) learner.save('third_cycle') learner.load('third_cycle'); learner.unfreeze() learner.fit_one_cycle(2, max_lr=slice(lr*0.95**num_groups, lr), moms=(0.8, 0.9)) appendix ii: custom transformer model architecture customtransformermodel( (transformer): distilbertforsequenceclassification( (distilbert): distilbertmodel( (embeddings): embeddings( (word_embeddings): embedding(30522, 768, padding_idx=0) (position_embeddings): embedding(512, 768) (layernorm): layernorm((768,), eps=1e-12, elementwise_affine=true) (dropout): dropout(p=0.1, inplace=false) ) (transformer): transformer( (layer): modulelist( (0): transformerblock( (dropout): dropout(p=0.1, inplace=false) (attention): multiheadselfattention( (dropout): dropout(p=0.1, inplace=false) ) (sa_layer_norm): layernorm((768,), eps=1e-12, elementwise_affine=true) (ffn): ffn( (dropout): dropout(p=0.1, inplace=false) ) (output_layer_norm): layernorm((768,), eps=1e-12, elementwise_affine=true) ) (1): transformerblock( (dropout): dropout(p=0.1, inplace=false) (attention): multiheadselfattention( (dropout): dropout(p=0.1, inplace=false) ) (sa_layer_norm): layernorm((768,), eps=1e-12, elementwise_affine=true) (ffn): ffn( (dropout): dropout(p=0.1, inplace=false) ) (output_layer_norm): layernorm((768,), eps=1e-12, elementwise_affine=true) ) (2): transformerblock( (dropout): dropout(p=0.1, inplace=false) (attention): multiheadselfattention( (dropout): dropout(p=0.1, inplace=false) ) (sa_layer_norm): layernorm((768,), eps=1e-12, elementwise_affine=true) (ffn): ffn( (dropout): dropout(p=0.1, inplace=false) ) (output_layer_norm): layernorm((768,), eps=1e-12, elementwise_affine=true) ) (3): transformerblock( (dropout): dropout(p=0.1, inplace=false) (attention): multiheadselfattention( (dropout): dropout(p=0.1, inplace=false) ) (sa_layer_norm): layernorm((768,), eps=1e-12, elementwise_affine=true) (ffn): ffn( (dropout): dropout(p=0.1, inplace=false) ) (output_layer_norm): layernorm((768,), eps=1e-12, elementwise_affine=true) ) (4): transformerblock( (dropout): dropout(p=0.1, inplace=false) (attention): multiheadselfattention( (dropout): dropout(p=0.1, inplace=false) ) (sa_layer_norm): layernorm((768,), eps=1e-12, elementwise_affine=true) (ffn): ffn( (dropout): dropout(p=0.1, inplace=false) ) (output_layer_norm): layernorm((768,), eps=1e-12, elementwise_affine=true) ) (5): transformerblock( (dropout): dropout(p=0.1, inplace=false) (attention): multiheadselfattention( ) (sa_layer_norm): layernorm((768,), eps=1e-12, elementwise_affine=true) (ffn): ffn( (dropout): dropout(p=0.1, inplace=false) ) (output_layer_norm): layernorm((768,), eps=1e-12, elementwise_affine=true) ) ) ) ) (pre_classifier): linear(in_features=768, out_features=768, bias=true) (classifier): linear(in_features=768, out_features=5, bias=true) (dropout): dropout(p=0.2, inplace=false) ) ) appendix iii: methodology based on the universal language model fine-tuning for text classification (ulmfit) method of transfer learning in nlp, we have achieved a state-of-the-art system that can still be visibly improved. ulmfit was developed by jeremy howard, a co-founder of fast.ai, which he uses to demonstrate state-of-the-art results using models trained on 100x less data than the originals, making it a very flexible and powerful algorithm. some of the key techniques involved in ulmfit namely discriminative learning rates, gradual unfreezing, and slanted triangular learning rates have been essential to the reported level of results, and explained below. the network architecture utilized is a standard distilbert layered neural network as pioneered by huggingface nlp. we wrote a custom transformer wrapper to make it more flexible for testing and development, allowing us to substitute different base transformer language models (such as xlnet and roberta) in the easiest way possible.throughout this testing we learned that in order to optimize our network for the shortest amount of time, distilbert was the best choice, because it is the smallest. the network is initialized in layers in the following sequence: 1 – pre-trained embedding layer 2 – distilbert transformer layer 1 3 – distilbert transformer layer 2 4 – distilbert transformer layer 3 5 – distilbert transformer layer 4 6 – distilbert transformer layer 5 7 – distilbert transformer layer 6 8 – pre-classifier layer 9 – classifier so on to what we’ve done differently. discriminative learning rates propose a theory that to improve a model, a different learning rate should be used for different layers, with the highest learning rate at the very end of the model (pre-classifier layer). when learning with stochastic gradient descent, the equation changes slightly: figure 2. images credited to howard and ruder (howard et al. 2018). slanted triangular learning rates build off of the first idea by increasing the dynamic learning rate at the beginning of training, then decaying it linearly to form a sort of triangle graphically: figure 3. image credited to howard and ruder (howard et al. 2018). gradual unfreezing bring the first two ideas home, by starting the training with the entire model frozen except for the very last layer (thus having the highest learning rate) during the first epoch of training, then drop the learning rate slightly and do the second round of training with everything frozen but the last 2 layers, then the last 3 layers, then with the lowest learning rate, we unfreeze the entire model and train for 2-5 epochs, with the learning rate decreasing each time. this basic methodology has allowed us to achieve state-of-the-art results on 21 categories, with only 4475 labelled examples in our training set. this dataset is relatively small, making this achievement that much more impressive. with that in mind, our results could be much more useful for the purposes of byu library if we put some more effort into cleaning our current dataset, augmenting outlier examples, and add more data into it. subscribe to comments: for this article | for all articles one response to "machine learning based chat analysis" please leave a response below: daniel van strien, 2021-02-12 this was a lovely paper. i really liked that you avoided using a ‘black box’ commercial solution for a domain that is probably quite different from commercial chat services. i had whether you have (or considered) using any additional metatada related to the chat message. for example, do you have information on time, day of the week of the chat? i haven’t worked with library chat systems but wonder if they follow some patterns i.e. more questions about when the library closes come later in the day and some times of year may have more questions related to dissertations. if this information is given to the model it may be able to use it to help make predictions without having to hand craft these rules. there are some example of doing this with the ulmfit approach https://www.novetta.com/2019/03/introducing_me_ulmfit/ i’d be curious to hear about any thoughts on this and whether you think it might be a possible approach. thanks again for the wonderful paper :) leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – developing sinopia’s linked-data editor with react and redux mission editorial committee process and structure code4lib issue 45, 2019-08-09 developing sinopia’s linked-data editor with react and redux an important software product for the linked-data for production phase 2 grant from the mellon foundation was the creation of a linked-data editor that professional cataloging staff would use to create original rdf descriptions of their collections. using the bibframe editor from the library of congress as inspiration, the stanford university library-based software development team are actively building a react/redux linked-data editor for use by a cohort of national, academic, and special libraries. a very popular combination for front-end javascript applications, this article will explain how react and redux are used with great success in the editor’s implementation of a domain-specific-language (dsl) called profiles containing one or more resource templates that specify an html form-based user interface for cataloging using rdf. by jeremy nelson background of sinopia funded by the andrew w. mellon foundation as part of the ld4p (linked data for production) grant, sinopia, available at sinopia.io, is an open-source, cloud-based collaborative linked-data cataloging environment that could be used in a production environment. the sinopia project’s primary software development team are members of the stanford university libraries with michelle futornick the project owner, prioritizing user needs and requirements in an iterative manner following agile software practices. the agile development methodology provides an approach and processes for responding to changes in requirements or environment by emphasizing flexibility and adapting to those changes. specific agile practices of the stanford development team include pair programming, testing, stand-up meetings, planning sessions, and sprints. in pair programming two or more developers collaborate together to code or solve specific problems by having one programmer type while both developers talk through the problem together, thereby increasing code comprehension and knowledge transfer within the team. another critical agile practice used by the sinopia development team is creating unit tests for specific javascript modules in addition to integration tests that test interactions between multiple components and modules in the code base through scripting actions in a headless web browser. the sinopia development team also has daily stand-up meetings where the developers check-in with each other and the product owner on progress and challenges encountered in the previous day, as well as plan for the upcoming day. finally, the sinopia team is organized in one to two week sprints that attempt to accomplish specific goals and tasks within the sprint. each sprint also includes a planning meeting at the beginning of the sprint and weekly storytime meeting where challenges and issues that often required the project owner’s feedback and decisions that are then incorporated into the sprint development. at the start of the sinopia project, the decision made by the team was to build the linked data editor based on the previous development work done by the library of congress in the creation of their bibframe editor or bfe along with supporting projects like the bibframe profile editor and verso. the library of congress catalogers have been using the bfe for cataloging books and other materials in specific workflows related to the library of congress infrastructure. christina harlow, now at temple university, created sinopia’s architecture starting with a forked version of the library of congress bibframe profile in amazon web services (aws) with the linked data editor to follow. because the initial bfe codebase was a monolithic javascript module and the bfe lacked any unit or integration tests, it was decided that a hard fork of the bfe code base for sinopa linked data editor was necessary in order to address design and implementation shortcomings present in the bfe. in addition, a new backend architecture was necessary that leveraged aws services like cognito, a user authentication service, and to host an instance of trellis, a linked-data platform that allows the use of a postgres relational database to store the output of the linked data editor. two partners in the sinopia project provide an important service called questions authorities (qa), a joint project of cornell university and the university of iowa. qa is an api service that queries lucene indexes of both rdf and non-rdf datastores and returns json data of the results. this service provides an intermediary source of lookup data for the library of congress subject and name authorities, the getty linked data vocabularies and sharevde. functional javascript within the programming community, the ubiquity of javascript as the default programming language for scripting websites and applications means most developers have at least a passing knowledge of the language. while client-side manipulation of the html dom (document object model) has changed over the years with the introduction of javascript libraries like jquery and the continued addition of different supported technologies, javascript has morphed into a server-side language with the emergence of the node.js ecosystem. sinopia uses ecmascript features and conventions that are then converted into javascript using the babel compiler. ecmascript 6 features used in sinopia include class declarations such class input {...}, importing of javascript modules like import input from './input' and to support the import, using the export feature like export const input. because sinopia’s linked data editor communicates in an asynchronous fashion with the sinopia server back-end, the ecmascript 7 and 8 features like promises with async and await keywords are used in the code. finally, sinopia extensively uses the ecmascript spread operator … for expanding passed arguments or elements to simplify object cloning like this expression const newstate = { ...state } where newstate is a clone of the existing state.  sinopia also adopted the arrow function syntax () => {} throughout the code base. javascript supports multiple programming paradigms like imperative, object-oriented, and in the past ten years, the functional programming approach. the popularity of building functional react and redux code for web browser and native user interfaces has encouraged the use of javascript built-in map and reduce operators along with support for currying and other functional-friendly constructs like the => arrow functional form. the key insight in writing functional javascript is focusing on functions with minimal side-effects and deterministic expectations that given a set of inputs, the function will always return the same set of outputs. for example, a traditional javascript function is declared using the function keyword: function addtwo (x,y) { return x+y; } a more functional ecmascript using the const keyword and arrow functional form looks like: const addtwo = (x,y) => { return x+y } both of these addtwo functions are equivalent in that they take two input parameters and return a single value. the const keyword for the second is a critical difference in that it creates an immutable function that cannot be reassigned to another function or variable in subsequent code, while defining the addtwo function in the first form can be reassigned without the babel compiler complaining. the const keyword reduces the opportunity for accidentally introducing bugs by preventing the developer from reassigning the function with potentially harmful or unintentional effects. a javascript array is a list-like object that contains two methods map and reduce that helps in developing functional code for react components and redux reducers. the map method returns a new javascript array with the results of executing a function on every member in the original array. using the arrow functional form with the map method results in simpler implementation code and is fully functional by taking two inputs, the array and an anonymous function defined with the arrow functional form, and returns a new array. in the example below, starting with an integer array, performing a map call returns a new array firstfivesquared with each value of the original array raised to the second power: const firstfive = [1,2,3,4,5] const firstfivesquared = firstfive.map(row => row**2) console.log(firstfivesquared) array(5) [ 1, 4, 9, 16, 25 ] the reduce method with the arrow functional form applies an anonymous function that reduces the members of the array to a single member object. in sinopia, the reduce function is used in the inputpropertyselector function to go through a complex json hierarchy representing the current values of the react components in the user interface and either return the component’s items or an empty array based on the values of the component’s reduxpath. const inputpropertyselector = (state, props) => { const reduxpath = props.reduxpath let items = reduxpath.reduce((obj, key) => (obj && obj[key] !== 'undefined' ? obj[key] : undefined), state.selectorreducer) if (items === undefined) { items = [] } return items } resource templates and profiles the library of congress’s bibframe editor is a bibframe-focused linked-data editor that was an important source of inspiration for sinopia. to support a wide range of cataloging use cases, the bfe followed a two-part strategy. first, a user creates a json profile containing one or more resource templates in the bibframe profile editor. these templates can vary by the material being cataloged and are saved in the backend verso middleware server from the library of congress. this template is then loaded into the bfe to generate different types of user interfaces depending on the requirements for the material. so, for example, a monograph profile contains resources templates for a bibframe work and instance along with any supporting resource templates. a cataloger could then load into the bfe to catalog a book instance or work. the profile json file containing one or more resource templates is in effect a domain specific language (dsl) for generating the user interface needed by the user to catalog a specific type of bibframe entity. the profile dsl, while developed for the bibframe ontology, is generic enough that other linked-data vocabularies can be used in addition to bibframe, though they must be specified within a resource template. a resource template contains a number of properties that define properties such as the label to display in the user interface, a uri to use as the rdf type, and one or more properties contained in the propertytemplate list. sinopia, following the bfe’s example, only displays one top-level resource template at a time in the editor user interface. a source of confusion early on in the project arose when analyzing the library of congress set of profiles, as the id of a profile was often duplicated as one of the ids of the contained resource templates. this was fixed in later iterations of these library of congress profiles. here is an example of a resource template for a bibframe note that could then be referenced by other resource templates: { "id": "resourcetemplate:bf2:note", "resourceuri": "http://id.loc.gov/ontologies/bibframe/note", "resourcelabel": "note", "propertytemplates": [ { "propertyuri": "http://www.w3.org/2000/01/rdf-schema#label", "propertylabel": "note", "mandatory": "false", "repeatable": "false", "type": "literal", "resourcetemplates": [], "valueconstraint": { "valuetemplaterefs": [], "usevaluesfrom": [], "valuedatatype": {}, "editable": "true", "repeatable": "false", "defaults": [] } } ] } the source for constructing the actual react components containing the html inputs used by the sinopia linked data editor are the properties in the propertytemplates list. each property contains a propertyuri attribute that is used as the rdf predicate in the constructed rdf while the propertylabel is displayed in the user interface either as an html label or as a placeholder in the html input. with the resourcetemplate:bf2:note resource template above, sinopia’s editor generates the following display: figure 1. title note property panel. after entering the text, “a great title note”, this resource template generates a rdf graph using a relative uri as seen here (when the user clicks the preview rdf button): figure 2. rdf preview modal other attributes for the property include mandatory set to true if the property is required, and repeatable if the property can be duplicated. early in the development of sinopia, the decision was to limit and simplify what values are supported in the type property, with the simplest being literal, and the others being lookup, list, and resource. the literal property is for string values that are entered by the user, while the list and lookup types references typeahead components for searching and linking external entities and values in the user interface. the valueconstraint attribute contains a number of sub-attributes like defaults, as a list of one or more values that are used to pre-populate the values in the input fields. the resource type is more complicated in that it references another resource template through the valueconstraint‘s valuetemplaterefs attribute. the valuetemplaterefs attribute is a list of one or more resource template ids that are then used to embedded a resource template into the user interface so that the values entered act as a separate entity that is then linked to the calling property. when generating the rdf, a uri or blank-node of the embedded resource templates is positioned in the rdf object role with the original resource template as the rdf subject and the propertyuri property functioning as the rdf predicate. below is an example of a propertytemplate with a type of resource that references the id of the resource template above, resourcetemplate:bf2:note in the valueconstraint.valuetemplaterefs attribute: { "propertyuri": "http://id.loc.gov/ontologies/bibframe/note", "propertylabel": "notes about the work", "mandatory": "false", "repeatable": "true", "type": "resource", "resourcetemplates": [], "valueconstraint": { "valuetemplaterefs": [ "resourcetemplate:bf2:note" ], "usevaluesfrom": [], "valuedatatype": {}, "defaults": [] } } react components an open-source project sponsored by facebook, react is a javascript library that wraps html elements in defined classes and functions for building user interfaces. react classes and functions are often defined in an extension of javascript called jsx that is used to build components that can be assembled into larger, more complex user interfaces. for example, a relatively simple jsx component could represent a title on a page with an html h1 tag with a valid javascript expression embedded between curly brackets: const title = <h1>book title: {title}</h1> react components also have two important javascript arrays called props and state. the props array contains read-only properties that can be referenced within the component itself using the curly braces syntax and are set when the component is constructed. because the component’s props are read-only, this enforces the constraint that the props are immutable and follow a pure functional form. creating react components in pure functional form is possible and often desired, using the es6 class syntax allows for more comprehensible code that extends the base react.component class. a react component class must implement a render method that returns the desired javascript. the title javascript expression above could be refactored as a simple es6 class by extending the react.component and returning the html snippet with the title prop referenced using the this keyword indicating a class instance variable: class title extends react.component { render() { return ( <h1>book title: {this.props.title}</h1> ) } } if coming from object-oriented languages, the temptation might be to create a hierarchy of react components but this pattern is discouraged by the designers of react because react components are intended more for composition, where more complex react components are made-up of simpler components where the enclosing components pass properties down through the child’s initial props. to illustrate, here is a header react component that contains other react components. class header extends react.component { render() { return( <header> <title title={"pride and prejudice"} /> <author givenname={"jane"} familyname={"austen"} /> </header> ) } } the header component sets the title prop for the included <title> and the <author>‘s givenname and familyname props.  this could render html output to the web browser’s dom like this: <header> <h1>book title: pride and prejudice</h1> by jane austin </header> sinopia’s react components sinopia’s react components are built as pure functions or as jsx class components. to build out the user interface for the linked data editor, sinopia uses a combination of third party react components along with custom react components in a hierarchy of components with  <rootcontainer> being the top-level react component. the <rootcontainer>imports the <offcanvas> that is composed of two children, the <offcanvasbody> and the <offcanvasmen> components both from the react-offcanvas module. the <offcanvasmenu> presents a list of links to help and third-party resources displayed in a pane which is displayed when the help and resources link in the navigation bar is clicked. the other function of the <rootcontainer> component is to connect the react user interface to the redux state store. the <offcanvasbody> contains another react component from the react-router-dom third-party module that allows for the easy generation of a single-page application <browserrouter> react component that matches specific url patterns into multiple routes for base route / to the homepage, the /editor route to editor forms, a /templates route that displays a list of available templates and to upload a new resource template. other supporting routes include the /login to allow the user to authenticate using aws cognito, the /menu for the off-canvas help and resources page, and a 404 route for unmatched routes entered by the user. <rootcontainer> <offcanvasbody> <browserrouter> <app> <offcanvasmenu> <canvasmenu> on the sinopia’s homepage, the top level react components are outlined in the following image when the help and resources is clicked and the offcanvasmenu is displayed: figure 3. sinopia rootcontainer and offcanvasbody. homepage and user authentication when accessing sinopia’s homepage at sinopia.io, the <app /> react component, using the <switch /> and <route /> react components from the react-router-dom package, displays the <homepage /> and the standard sinopia <footer /> react components. the <homepage /> component contains three children, the <header />, <newspanel />, and the <descpanel /> react components. the homepage’s <header /> contains links to the /templates route, a link to sinopia’s profile editor, and finally a link that activates the <offcanavsmenu />‘s child’s <canvasmenu />. the <newspanel /> has a <newsitem /> that allows the product owner to make announcements by editing and then creating a pull request that can then be reviewed and then pushed to the aws environments by the developers and system administrators. the <descpanel /> contains more general description of the sinopia project and what the project hopes to accomplish during the grant period and after. amazon’s amplify sdk is used to authenticate the user to the aws cognito service that then generates a valid json web token for user authentication in sinopia and in sinopia’s backend trellis linked data platform. for the initial release of sinopia, all catalogers are authorized to add, edit, or delete any resources stored in trellis. this may change in future releases with more restrictive user rights for resources created in specific group containers within trellis. the react component hierarchy for the <app />‘s <homepage />, <loginpanel />, and <footer /> is: <app> <loginpanel> <switch /> <route /> => <homepage /> <header /> <newspanel /> <newsitem /> <descpanel /> <footer /> below are these react components diagrammed in a screen shot of sinopia’s homepage at sinopia.io: figure 4. sinopia homepage components. <homepage /> child components <header> the <header> component for sinopia’s homepage has a different navigation bar with links to the <importresourcetemplate >, sinopia’s profile editor, and a link to activate the <offcanvasmenu> to display the help and resources links. <newspanel> and <newsitem> the <newspanel> component is composed of one <newsitem> component that is a list of news items maintained by sinopia’s product owner and usually includes recent items like new sinopia releases, conferences or presentations about sinopia, and other items of interest to the sinopia’s community. <descpanel> the <descpanel> component is a short description of the goals for sinopia and the partner institutions in this project. resource templates upload and listing from sinopia’s <homepage />‘s <header />, an authenticated cataloger is taken to the /templates route using the <importresourcetemplate /> component containing a <header /> component, an <importfilezone .> component used to display and handle a button for catalogers to upload a json profile file containing one or more resource templates, and a <sinopiaresourcetemplates /> component displaying an html table populated by resource templates contained either in a running instance of the sinopia server or the sample resource templates if running the editor in use_fixures mode. <app> <loginpanel> <switch /> <route /> => <importresourcetemplate /> <header /> <importfilezone /> <sinopiaresourcetemplates /> <footer /> figure 5. linked data editor resource templates tab with components. <importresourcetemplate /> child components <header> for both the /templates and /editor routes and components, the same <header> component is used for displaying the navigation bar and the three tabs for the browser, editor, and resource templates. the <header> component also contains three <navlink> react components from the react-router-dom package that highlights the correct tab depending on the route being displayed in the editor. <importfilezone /> the <importfilezone /> component wraps a third-party react component that allows the cataloger to use their computer’s drag-and-drop feature to place a json profile file containing one or more resource templates into the application. the editor validates the profile and resource templates first before uploading the resource templates to the sinopia server with the results appearing in the <sinopiaresourcetemplates /> component. <sinopiaresourcetemplates /> the <sinopiaresourcetemplates /> component is initialized with a call out to sinopia server’s trellis linked data platform and loads all of the available resource templates for selection by the cataloger. linked data editor tab when a user selects a resource template from the <sinopiaresourcetemplates /> table, sinopia loads the json resource template into a new instance of the <editor />‘s <resourcetemplate /> component. in the <resourcetemplate /> component functioning as a container and connector to the redux state, a <resourcetemplateform /> creates a <propertypanel /> for each property. the react component hierarchy is outlined below: <app> <loginpanel> <switch /> <route /> => <editor /> <header /> <rdfmodal /> <groupchoicemodal /> <resourcetemplate /> <resourcetemplateform> <propertypanel> <inputliteral> <inputlistloc> <inputlookupqa> <inputlookupsinopia> <propertyresourcetemplate> <propertytemplateoutline> <outlineheader> <propertytyperow> <propertycomponent> <inputliteral> <inputlistloc> <inputlookupqa> <inputlookupsinopia> <resourceproperty> <propertytemplateoutline> <footer /> figure 6. linked data editor components with the work title resource template. <editor /> child components <rdfmodal /> part of the <editor/> is an html button labeled preview rdf that when clicked shows a bootstrap modal containing the generated rdf based on the redux state of the application. <groupchoicemodal /> at any point when adding or editing the rdf for the entity, the cataloger clicks the save & publish button that brings up a pop-up modal that displays a drop-down list of institutions and organizations <resourcetemplate /> and <resourcetemplateform /> the <resourcetemplate /> component includes the label of the loaded resource template, that preview rdf and save & publish buttons, and the <resourcetemplateform /> that wraps one or more <propertypanel />. the <propertylabel/> component provides the text in the panel’s header with one or more optional styling using such components as the <requiredsuperscript /> to display a red asterisk for a required <propertypanel />. <propertypanel /> component for every property template in the loaded resource template, an instance of the <propertypanel /> is rendered. understanding how the <propertypanel /> react component as a pure function is constructed illustrates how sinopia is able to build an editing environment for rdf triples. the first line in the <propertypanel /> source code imports two javascript classes from the react node.js module. the proptypes import provides a means to check if the <propertypanel />‘s variable props are of a certain type. the third line imports the <propertypanel /> component. import react from 'react' import proptypes from 'prop-types' import propertylabel from './propertylabel' the <propertypanel /> is a pure function with two constant variables, a floatclass that sets the panel div’s bootstrap floating class to create two columns of <propertypanel />s, the cssclasses variable sets all of the css and includes the floatclass variable. const propertypanel = (props) => { const floatclass = props.float > 0 && props.float % 0 > 0 ? 'pull-right' : 'pull-left' const cssclasses = `panel panel-property ${floatclass}` } finally, the <propertypanel /> renders a combination of html elements and other react components, including any children stored in the props for the component instance. the <propertypanel /> doesn’t need to know or care what the children components are, just that the child react component render some content. return ( <div classname={ cssclasses } data-label={ props.pt.propertylabel }> <div classname="panel-heading prop-heading"> <propertylabel pt={ props.pt } /> </div> <div classname="panel-body"> { props.children } </div> </div> ) <inputliteral /> component the most basic html input in sinopia is part of the <inputliteral /> react component. depending on if the property template’s repeatable is true, the cataloger can enter multiple values that are displayed immediately below the html input element or only enter a single value. each of those values can have a language attribute set and the value can be edited by clicking on the edit button. below is an example of a property template literal with multiple items: figure 7. input literal component. the challenge of representing a resource’s properties that include referencing a target resource template needs to be solved in the user interface when using these profiles. in the library of congress bfe, the ui solution was to open a modal that would render the target resource templates propertytemplates. those propertytemplates often reference further resource resources that require a new pop-up modal until the user loses the context with a whole series of modals layered on top of one  another. in an early prototype, astrid usong, sinopia’s user interface designer at stanford libraries, came up with different approach. instead of using modals, her design represented these relationships as an outline. when demonstrated during a sinopia-focused pre-conference at the 2019 code4lib conference in san jose, ca., the participants preferred the outline view over the pop-up modal because it was easier to keep the context of the target resource template in relation to the original resource template when editing a resource’s metadata. the outline view also provide a visual hierarchy built on familiar user interface tree where parent nodes can be expanded to reveal one or more component children. to support this tree hierarchy of propertytemplates one or more layers deep, the <propertytemplateoutline> react component is used and is composed of an <outlineheader> made-up of a collapsed plus-sign icon that when expanded shows any child components. the <propertytemplateoutline> contains either <propertycomponent> made up of either <inputliteral>, <inputlookupqa>, or <inputlistloc> , or a <resourcecomponent> that has <propertyactionbuttons> (containing an <addbutton> if the property template’s repeatable property is true) and one or more <propertytemplateoutline> components. when the <resourcecomponent> is expanded, a network call with the resource template id is made to the sinopia server and the resource template json is retrieved and displayed in the expanded view of the resource, with each of the target resource template’s property templates having their own <propertytemplateoutline>. linking to existing sinopia entities and other sources sinopia is able to link to other sources through one or more custom react components that provide a typeahead input using a third party node.js module called react bootstrap typeahead. <inputlistloc> library of congress id.loc.gov component the library of congress’s linked data service at http://id.loc.gov provides a number of subjects, thesauri, classifications, and other vocabularies. for large linked data services sinopia uses questioning authorities service, but for smaller vocabularies the inputlistloc react component directly connects and retrieves a json list that is presented to the end user as a lookahead provided by the ‘react-bootstrap-typeahead'(#rbt) module. these linked data vocabularies are pulled from a json configuration object that is shared with the <inputlookupqa> and soon the <inputlookupsinopia /> react components. here is a screen shot of the inputlistloc component within sinopia: figure 8. inputlistloc component with drop-down. the uri and label is saved in the <inputlistloc> props with the uri becoming an rdf object of the entity as the rdf subject and the propertytemplate’s propertyuri as the rdf predicate. <inputlookupqa> questioning authorities component from the beginning, the sinopia project team included close collaboration with huda khan of cornell university who is working the react components to support searching the questioning authorities (qa) service. qa is run as a collaborative effort with lynette rayle, and david echimann at the university of iowa school of information science. the qa service has a cache built with solr and the fusuki triplestore, with a swagger api integrated using with sinopia’s <inputlookupqa> component. figure 9. inputlookupqa component. <inputlookupsinopia /> component the <inputlookupsinopia /> component allows catalogers to reference existing entities created within sinopia and will look and act in a similar fashion as the <inputlistloc> and <inputlookupqa>. editor’s redux state initially, sinopia’s react components were structured with extensive props and state changes to represent and respond to actions and user expectations to accepting values both from any parent information and also push state information to any composited child components. as the team became more conversant with redux, the refactoring implementation of the react components simplified both the component creation as well as what props and state variables are needed at this level of components. the redux state in sinopia is a snapshot in time of what data is in the active react components. for example, loading the following resource template, a bibframe work title, into the editor and entering a couple of values so the current state of the user interface looks like: figure 10. populated work title resource template. is represented in this global redux state: selectorreducer: { 'resourcetemplate:bf2:worktitle': { 'http://id.loc.gov/ontologies/bibframe/maintitle': { items: [ { content: 'pride and prejudice', id: 'xsgpxn4su' } ] }, 'http://id.loc.gov/ontologies/bibframe/partname': {}, 'http://id.loc.gov/ontologies/bibframe/partnumber': {}, 'http://id.loc.gov/ontologies/bibframe/note': { buolqnhya: { 'resourcetemplate:bf2:title:note': { 'http://www.w3.org/2000/01/rdf-schema#label': { items: [ { content: 'many editions, adaptations, and films', id: 's7h8lzdj4' } ] } } } }, rdfclass: 'http://id.loc.gov/ontologies/bibframe/title' } } for the <inputliteral > component instance for the propertytemplate maintitle, the values are extracted by using a redux reducer inputpropertyreducer function that takes the <inputliteral>‘s reduxpath as a javascript array and using the redux state as the starting point, reduces the redux state to the javascript object containing an items javascript array. the reduxpath for the bibframe main title <inputliteral> is in this case: ['resourcetemplate:bf2:worktitle','http://id.loc.gov/ontologies/bibframe/maintitle']. react components in sinopia, they reflect or mutate the global redux state through two methods, mapstatetoprops and mapdispatchtoprops as recommended in the official react-redux documentation. rdf generation and the sinopia server as the redux state captures the event of a user entering or linking data to when it happens in the react components, sinopia can generate rdf at any particular point-in-time on demand. from the previous redux example, clicking on the preview rdf button uses a redux reducer to create an instance of the rdf graphbuilder class and generate the following rdf graph that can then be saved and published through the backend sinopia server: <> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://id.loc.gov/ontologies/bibframe/title> . <> <http://id.loc.gov/ontologies/bibframe/maintitle> "pride and prejudice" . <> <http://id.loc.gov/ontologies/bibframe/note> _:b1 . _:b1 <http://www.w3.org/2000/01/rdf-schema#label> "many editions, adaptations, and films" . next steps the sinopia linked data editor targeted minimal viable product release is at the end of july 2019. this release will allow the sinopia cohorts to start cataloging using the linked data editor and provide valuable sources of requirements, pulled from their usage and experiences, to help the sinopia team learn and plan for the next work-cycle. as the sinopia user and developer community expands beyond the stanford and cornell development teams, sinopia as a cataloging tool built as a generic cataloging editor for linked data, is a strong base to extend and expand into the future. currently sinopia has three linking sources, the library of congress, the questioning authorities, and internally created entities. an early requirement of sinopia is to provide the ability of catalogers to do two different, but related, workflows. for pre-existing graphs of rdf entities either available from third party authorities like library of congress, viaf, or sharevde or from internally created entities stored in sinopia’s server, catalogers will need to be able to either edit, add, or delete triples about these entities or derive a new rdf entity by copying those triples, much like the copy-cataloging current marc based work-flows in libraries. finally, to encourage and broaden adoption of sinopia beyond aws specific services, a new sinopia server infrastructure based on kubernetes or docker swarm would allow other organizations to run their own version of the sinopia stack. in addition, integration with existing open-source library systems like folio will be explored in the future. references agile alliance agilealliance.org react bootstrap typeahead node.js package at http://ericgio.github.io/react-bootstrap-typeahead react redux usage guidelineshttps://react-redux.js.org/using-react-redux/connect-mapstate#usage-guidelines acknowledgments sinopia would not have been possible without the talents, hard-work, and experience of the current and former members of the development team including joshua greben, naomi dushay, sarav shah, johnathan martin, joseph atzberger, michael giarlo, justin coyne, peter mangiafico, justin littman, and aaron collier. about the author jeremy nelson is a software engineer with the stanford university libraries and technical lead on the sinopia project. he is also co-founder and cto of knowledgelinks.io.   subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – to everything there is a session: a time to listen, a time to read multi-session cds mission editorial committee process and structure code4lib issue 56, 2023-04-21 to everything there is a session: a time to listen, a time to read multi-session cds when the cost of cd burners dropped precipitously in the late 1990s, consumers had access to the cd-r, a format with far greater storage capacity than floppy disks. multiple session standards allowed users the flexibility to add subsequent content to an already-burned cd-r, which made them an attractive option for personal backups. in a digital preservation context, cds with multiple sessions can pose significant challenges to workflows and can lead to data errantly not being acquired or reviewed if users are using a workflow designed for single-session, single-track cds. in workflows that include cds as software installation or transmission media, extra-session behavior can have an impact on software supply chain review. this article provides an overview of the structure of a multi-session cd and outlines tool behavior of disk images generated from multi-session cds. to support testing in specific contexts, we provide a guide to creating a multi-session cd that can be used when developing workflows. finally, we provide techniques for extracting content from physical media as well as existing disk images generated from multi-session cds. by dianne dietrich and alex nelson disclaimer ​​the views and opinions expressed in this article are those of the authors and do not necessarily reflect the official policy or position of any agency of the u.s. government or cornell university. any mention of a vendor or product is not an endorsement or recommendation. logos and trademarks are copyright of their respective owners. 1. introduction many digital preservation organizations have developed workflows to handle optical media that consist of single-session, single-track data cds, especially at scale (rothrock et al, 2021). workflows for single-session cds consisting of one data track followed by one or more audio tracks also exist, as demonstrated by geoffrey brown’s work handling mixed-mode [1] cd-roms from the voyager collection (brown, 2012). cds with multiple sessions can pose significant challenges to workflows and can lead to data errantly not being acquired or reviewed if the user is not aware, especially since guidance written for single-session media may fail in interesting and subtle ways when applied to multi-session discs. the goal of this paper is to provide practitioners a technical framework by which they can develop appropriate analysis and preservation workflows for these types of media. this work improves acquisition workflows, understanding of the varied storage models in optical media schemes at layers before file systems, and, in the case of software installation media, software supply chain review. reflecting on our own experience with cd-rom corpora, we saw an unfulfilled need for guidance on preservation techniques for multi-session cds, since a digital preservation practitioner might encounter this as personal backups on cd-rs and cd-rws. 1a. scope within this document, the discussion around optical media is limited to the compact disc digital form factor and approximate quantity of storage capacity, that is, under 800 megabytes. while “optical media” as a general term encompasses other media classes, such as dvds and the analog laserdisc format, the technical details of arranging storage reviewed in this article on those media differ from compact discs, and are left out of scope. 2. background cd standards are detailed in technical standards known as the rainbow books, which started in 1980 with the red book, the standard for encoding audio on a cd [2]. in 1983, the yellow book established the cd-rom standard [3] for the storage of data. related, iso 9660 defines the file system standard for a cd-rom. the orange book (1990) [4] defined the cd-r standard, which included writing multiple sessions of data to a cd; related information can also be found in ecma-394 (2010) [5]. iso 13490 (1995) and ecma-168 (1994) defined how operating systems read cd-rs with multiple sessions [6]. in 1995, the blue book [7] provided the technical specifications of a multi-session disc with both audio and data components, to address the limitations of single-session, mixed-mode cd-roms. a blue book cd contains exactly two sessions: the first containing audio tracks, and the second containing one data track. disc level session level session 1 session 2 … track level lead in (with toc) track 1 … lead out lead in (with toc) track 1 … lead out … figure 1. structure of a general multi-session cd. here toc refers to table of contents. (adapted from wikipedia [8]). individual tracks may be audio or data. (see also ecma-394, figures 12–18.) in 2017, johan van der knijff of the open preservation foundation shared findings from the dutch national library showing that disk images created from blue book cds failed their validation step, even though nothing in the imaging process indicated that the capture had failed in any way (van der kniff, 2017). van der knijff explained that this was happening because the second data track required a physical offset value in order to be correctly parsed. with a disc in hand, it is possible to determine the location of the data track and the appropriate offset value. van de knijff described how to extract content from a disk image by generating a derivative image file left-padded with zeros to represent the physical offset. in our 2018 talk (dietrich and nelson, 2019), we built on the foundation of van de knijff’s work, providing examples of the user’s perspective in imaging a blue book cd and attempts to read the resulting disk images, showing how each tool we tried (and, at times, different options within the same tool) yielded different output formats. we also showed that each kind of output required different post-processing actions in order to extract content. we concluded by offering one suggestion for determining the correct offset value of a blue book cd-rom if the original disc was no longer available that appeared to successfully identify the correct offset value for the set of testing discs we sourced from our collections. in reflecting on that work, we noted the similarities between blue book cds and multi-session cd-r/ws: understanding that the offset value was necessary for a data track located in any session subsequent to the first was undoubtedly crucial. one important question remained for us, however. what findings from working with blue book cds did not hold when applied to multi-session cds? the goal of this paper is to provide additional detail on multi-session cds to aid digital preservation practitioners in accurate and complete captures of the data they contain that can be applied in optical media workflows. 3. generating a multi-session cd on the command line this section is intended to illuminate the inner workings of a multi-session cd by guiding the reader through the creation of one using ubuntu 20.04 lts. the steps illustrated below will generate a “data disc” as defined by ecma-394: “a disc on which every session contains one or more data tracks.” the test cd that we use throughout this paper will be a data disc constrained to one track per session. in the test cd, each data track will contain only iso 9660 filesystem data. the standards define abilities to include multiple data tracks per session, but our experience with disc authoring software frequently showed incomplete or inconsistent support for the capabilities provisioned by the standards. see appendix b for a discussion on creating discs with multiple data tracks in a single session. the purpose of this section is not to attempt to re-create the functionality of cd authoring tools, nor replicate all possibilities for multi-session cds users may have in their collections. these steps make use of command line tools installed on an ubuntu 20.04 lts workstation. there are many options within the tools used here – xorrisofs [9] and mkisofs [10] – that can be used to generate a testing cd. the parameters specified in the following example detail the steps necessary to generate a basic disc that can be used for testing tools and workflows. the following examples specify xorrisofs, but mkisofs has equivalent parameters and can be used to generate a test cd using the same technique. (the documentation for xorrisofs also includes documentation for generating multiple-session cds.) to understand the sample commands below, the directory first_session contains files selected for the first session of the disc. in this sample multi-session cd, each session contains a single text file named after the session it is in: session1.txt is in the directory first_session. the contents of session1.txt are “this is track 1 in session 1.” we will use this convention for subsequent sessions (i.e., files for the second session will be indicated by a directory named second_session, and it will contain one file called session2.txt containing text “this is track 2 in session 2.” and so on). the first step to making a multi-session cd is to generate an iso file with the data for the first session by running the command indicated below: xorrisofs –o first_session01.iso first_session/ the next step is to burn first_session01.iso to blank cd-r or cd-rw media using the following command [11]: cdrecord –multi –v –eject –speed=4 first_session01.iso next, determine what the offset for the second session must be and use that information to generate an iso file with the data for the second session. this information is provided by the cdrecord command, when the freshly-burned disc is inserted back into the closed tray. this command records the necessary data: cdrecord –msinfo > msinfo_after_session_1.txt this command will not produce the correct result if the cd (that was just burned to) is not currently loaded into the drive. in the event of error, msinfo_after_session_1.txt will be empty. the following command generates the iso file with the data for the second session: xorrisofs –o second_session02.iso –c $(head -n1 msinfo_after_session_1.txt) \ -m /dev/cdrw second_session/ next, burn second_session02.iso to the media with the data from the first session: cdrecord –multi –v –eject –speed=4 second_session02.iso keep repeating the iso generating step that calculates the next offset from the physical media and burning steps until the last session, and then run cdrecord without the -multi flag. forgetting to calculate the session offsets with each iso generating step will result in an unusable disc. utilities that list out the contents of each generated iso file show how each session’s file listing display includes the files from the prior session. for example, a view of the first session of the test cd described above would include session1.txt, a view of the second session of the test cd would include session1.txt and session2.txt, a view of the third session of the text cd would include session1.txt, session2.txt, and session3.txt, and a view of the fourth session would include session1.txt, session2.txt, session3.txt, and session4.txt. 4. behavior patterns in disk imaging outputs of multi-session discs it can be difficult to know when a multi-session cd is present in a workflow. this section elaborates on tool behavior of multi-session cds created by disk imaging processes. if the provenance of a disk image is unknown, there are a few possibilities for how it might be structured. the following list details categories of outputs that are possible given ecma-394. raw disk image of an entire disc, including data tracks from all sessions disk image of the user data portion of an entire disc, including data tracks from all sessions (referred to after this as a user data disk image) raw session image of an individual session within a disc user data session image of an individual session within a disc raw track image of an individual data track within a session user data track image of an individual data track within a session using the imaging tools listed in appendix a, we were able to generate disk images in all of the above categories except for category 4. we repeat, as we believe it deserves clarification: if we consider the general, abstract class of “disk images,” we are considering two disjoint subclasses of disk images, the raw disk image and the user data disk image. we also define three disjoint subclasses of a more-general “image”, the disk image, session image, and track image. a raw image will include error correction and detection data from the physical disc and its size will be a multiple of 2,352 bytes. a user data disk image does not include this information, and only includes the “user data” [12] portion of each sector. its size will be a multiple of 2,048 bytes. detecting that one has a disk image of a multi-session cd using various parsing tools can be subtle. the lack of an explicit error message does not necessarily mean the tool successfully displayed the contents of that disk image file. the table below summarizes behaviors of tools when presented with a disk image file that matches the description in the middle column. the table specifically does not indicate how the disk images were created. it only provides detail on the behavior of the tools we tested to parse our test cd generated in section 3 (i.e., multiple sessions, each session containing only one data track with iso 9660 filesystem data). tool input image format tool behavior mount -t iso9660 -o loop cd.iso mount_point image of any level (full disc, session, or track), raw non-applicable. the mechanism to create a loopback device (/sbin/losetup) does not support conversion from raw sectors to userdata sectors (i.e. flag-value --sector-size 2352 is rejected). mount -t iso9660 -o loop cd.iso mount_point [13] image of the full disc, user data file listing at the mount point only includes the first track of the first session; listed file contains expected data. mount -t iso9660 -o loop,sbsector=[offset to last session on disc] cd.iso mount_point [14] (see also section 5 for description of the sbsector option.) image of the full disc, user data file listing at the mount point includes all files from all tracks; all files contain expected data. mount -t iso9660 -o loop cd.iso mount_point track image from any of the sessions but the first, user data error message: can’t read superblock on /dev/loopx mount -t iso9660 -o loop,sbsector=[offset to session of track on disc] cd.iso mount_point track image from any of the sessions but the first, user data error message: wrong fs type, bad option, bad superblock on /dev/loopx, missing codepage or helper program, or other error. mount -t iso9660 -o loop,sbsector=[offset] cd.iso mount_point track image from any of the sessions but the first, user data, prepended with zeros corresponding to the session offset file listing at the mount point includes files from all previous tracks and sessions back to the first; however, all files not originally included on the track/session specified by the offset contain no data. see section 7 for further detail. isoinfo [15] image of the full disc, user data with no offset provided, display metadata for first session data only with -l; -p only reports one path table.when using -t with offset to session: display metadata for that session with -l; display start of path table for that session with -p. isoinfo image of the full disc, raw non-applicable – isoinfo does not support raw images. isoinfo session image of any of the sessions but the first, raw non-applicable – isoinfo does not support raw images. isoinfo track image from any of the sessions but the first, user data with no offset provided, error message: “short read on old image.” when using -n with offset to specific session: display metadata for that session with -l; display start of path table for that session with -p fiwalk [16] image of the full disc, raw or user data reports out file listing for first session only. fiwalk session image of any of the sessions but the first, raw tsk_error ‘cannot determine file system type’ using a variety of sector offset values and sector sizes. fiwalk track image from any of the sessions but the first, raw or user data tsk_error ‘cannot determine file system type’ … using a variety of sector offset values and the automatically-used sector size [17]. fls [18] image of the full disc, raw or user data reports out file listing for first session only. options for specifying an image offset (-o) yield the following error: “cannot determine file system type.” fls session image of any of the sessions but the first, raw “cannot determine file system type.” fls track image from any of the sessions but the first, raw or user data with no extra parameters, error message: “cannot determine file system type.” with -b 2048 (defining sector size) and -o (with offset value), “sector offset supplied is larger than disk image.” tsk_recover -a [19] image of the full disc, raw or user data files recovered: 1 file corresponding to contents in the first track of the first session only. tsk_recover -a -o [sector offset corresponding to session] image of the full disc, raw or user data cannot determine file system type (sector offset: 11702)files recovered: 0 tsk_recover -a session image of any of the sessions but the first, raw cannot determine file system type (sector offset: 0)files recovered: 0 tsk_recover -a -o [sector offset corresponding to session] session image of any of the sessions but the first, raw cannot determine file system type (sector offset: 11702)files recovered: 0 tsk_recover -a track image from any of the sessions but the first, raw or user data cannot determine file system type (sector offset: 0)files recovered: 0 tsk_recover -a -o [sector offset corresponding to session] track image from any of the sessions but the first, raw or user data cannot determine file system type (sector offset: 11702)files recovered: 0 5. accessing content from images of the full disc this section will illustrate how to use isoinfo to extract content from an image of a full disc, whether the image is raw or user data. the technique for extracting content in this and the next sections 5 through 7 focuses only on images that contain an iso 9660 file system. first, obtain the appropriate offsets; either with the physical media in hand, described in 5a; or from imaging metadata, described in 5b; or from the image itself, described in 5c.if you have a raw disk image, follow the directions for converting to a user data disk image as described in 5d. the process of using bchunk [20] will generate track-level images; follow the directions for extracting content in 7. if you have a user data disk image, use the technique described in 5e. it is also possible to extract content from an image of the full disc by using mount and the value of the session offset (in 2kib sectors) for the last session on the disc. the following command worked using a full disk image generated from our test cd: mount -t iso9660 -o loop,sbsector=[offset to last session] \ cd.iso mount_point 5a. obtaining the disc offset(s) from the physical media if the original physical disc is still accessible, the command line utility cd-info [21] will display the offset(s) present on that media. sample output from the multi-session cd created in section 3 is below: disc mode is listed as: cd data (mode 2) cd-rom track list (1 4) #: msf lsn type green? copy? 1: 00:02:00 000000 data false no 2: 02:38:02 011702 data false no 3: 04:14:04 018904 data false no 4: 05:50:06 026106 data false no 170: 05:54:08 026408 leadout (59 mb raw, 58 mb formatted) media catalog number (mcn): 0000000000000 last cd session lsn: 26106 __________________________________ cd analysis report cd-rom with iso 9660 filesystem iso 9660: 184 blocks, label `isoimage ' application: preparer : xorriso-1.5.2 2019.10.26.180001, libisoburn-1.5.2, libisofs-1.5.2, libburn-1.5.2 publisher : system : volume : isoimage volume set : session #2 starts at track 2, lsn: 11702, iso 9660 blocks: 174 iso 9660: 174 blocks, label `isoimage ' session #3 starts at track 3, lsn: 18904, iso 9660 blocks: 174 iso 9660: 174 blocks, label `isoimage ' session #4 starts at track 4, lsn: 26106, iso 9660 blocks: 174 iso 9660: 174 blocks, label `isoimage from this output, we see the logical sector numbers (lsns) corresponding to our sessions are 0, 11702, 18904, and 26106. these are the numbers we would use, e.g., in the sbsector option to mount. 5b. obtaining the disc offset(s) from imaging metadata some imaging utilities automatically generate metadata that includes offset information for sessions present on a cd. while technically, the cue sheet syntax [22] does not include fields that would supply that value, some programs supply this information in the unstructured-text comments of their respective cue sheets. lines starting with rem are comments that some utilities use to store offset values. 5c. obtaining offset values from full user data disk images for user data images of full multi-session discs, obtaining the offset of any session after the first may be possible using isoinfo through testing every possible option. to find all of the possibilities, divide the disk image file size by the sector size (2,048). for each possible offset, run isoinfo -i cd.iso -t [sector number] any sector value that does not yield the error message, “cd-rom is not in iso 9660 format” or “short read on old image” is a usable offset value. beware that using -n with an incorrect offset value may not result in an error message, but instead yield an incomplete listing of files. 5d. conversion to user data from raw using bchunk one method of converting a raw image to a user data image is by post-processing it using the command bchunk. this utility accepts in a raw image and a cue sheet and uses these to generate user data images for data tracks and “native cd audio” or wav files for audio tracks (quoted directly from bchunk man page). a sample bchunk command follows: bchunk raw_disc_image.bin cd.cue derivative_iso_basename not all disk imaging utilities supply a cue sheet as output by default, and in some cases, creation of a cue sheet must be handled separately from the image creation process. 5e. extracting files from images of the full disc in our testing, we found the command line tool isoinfo is able to extract files from an image of the full disc using the corresponding session offset(s). consider the test multi-session cd created in section 3. if the file session1.txt was added in the first session as the first track, session2.txt was included in the second session as the second track of the disc, and session3.txt was included in the third session as the third track, (and so on), in order to extract session2.txt from an image of the full disc, one would need the offset for the second session. the offset for the third or fourth session would not work, even though when listing files from a specific track using a session offset, that file would appear to be listed (because all of the files from each of the previous sessions’ tracks are listed). we used the following command to get a complete listing of all of the files in each track from all of the sessions of our test cd. isoinfo -i cd.iso -t 26106 -l when working with an image of the full disc, isoinfo requires the -t flag (instead of the -n flag for the single-track image). we used the following command to extract an individual file from the image and redirect it to a file named session4.txt: isoinfo -i cd.iso -t 26106 -x '/session4.txt;1' > session4.txt the ;1 in the filename display is for the iso file version and may not be present on all multi-session cds. if it is present in the isoinfo listing, it should be used when invoking the specific commands to extract files, as shown in our example below. ensure that the full absolute pathname within the disc’s last session’s file system is accurate (the -f flag will provide the full path to be used when invoking the -x flag). with the commands exercised so far, it is possible to see how in some cd authoring programs, it was possible to “delete” a file in a subsequent session. when working with a multi-session disc where the creator did this, the file listing of the track in the last session will not include those so-called deleted files. they can still be located in an image of the full disc, and within the metadata, by using the offset to the session where that track was included. do carefully consider how to handle this “deleted, but recoverable” (casey et al., 2019) data when working with real-world collection material (lassere and whyte, 2021). 6. accessing content from single-session images in our testing, we were only able to generate single-session raw images. since isoinfo only handles user data images, it cannot be used directly with a raw image. use the technique for converting the single-session raw image into single-track user data images described in 7b first, obtain session offsets (either by following 5a, 5b, 5c, or 7a), and then proceed to the techniques outlined in 7c. 7. accessing content from single-track images in order to extract content from single-track images, you must have the image of the track where the file was originally included on the disc in order to extract content, as well as the session offset for that particular track. depending on your starting point, it may be possible to obtain session offsets using the techniques available with the disc in hand (5a), or using imaging metadata (5b), or from an image of the full disc (5c). if these techniques aren’t possible, try the technique in 7a. if the imaging tool that you used generated track-level user data images, or you generated such images using bchunk (described in 5d) you can use the technique described in 7c to extract content from those images. if you have single-track raw disk images, convert them to user data using the technique in 7b. it is also possible to extract files from single track-images using the technique referenced in (van der kniff, 2017 and [23] ) of prepending an image with sectors of zero-bytes corresponding to the session offset and mounting the resulting image. we confirmed this in our review of mount in the table in section 4. the same caveat applies here too: not every file that will be listed under the resulting mount point will contain data captured from the physical disc. in our testing, only the files originally added in that track will have the expected data. 7a. offsets from single-track images for single-track images, isoinfo combined with the -p flag will report the start block of the type-l path table [24]. noting the iso 9660 spec [25], the path table immediately follows the volume descriptor set, which consists of at least two volume descriptors – a primary volume descriptor and a volume descriptor set terminator, which is itself preceded by 16 sectors designated as a system area. from this information, it should be possible to backtrack and determine the value of the original offset for that session. in testing the single-track disk images created from our sample multi-session disc, subtracting the path table location from the number of volume descriptors and the 16 sectors of the system area consistently yielded an offset value that was one sector greater than the value reported by utilities that read the physical disc. thus, our formula for determining the offset of the session was calculated by the following formula: type-l path table location – number of volume descriptors – 17 7b. conversion to user data from raw one method of creating a user data disk image from a raw image is described in 5d. it assumes an image of the full disc and the existence of a cue sheet generated from the original physical media. if all you currently have is a track image and no cue sheet information, it might be possible to generate a cue sheet that represents an abstract single data track in order to use as input to bchunk. save the following text as file.cue (with each indentation level being two single-whitespace characters): file "data_track.bin" binary      track 01 mode1/2352           index 01 00:00:00 once this is generated, it is possible to use the following bchunk command to convert a raw image to user data: bchunk data_track.bin file.cue derivative_iso_basename under the scope of images we inspect in this paper—one data track per session—this “default” supplement should enable conversion to user data. the “mode1” string may need to be changed to “mode2”, because user data within a mode-1 sector appears at a different location than a mode-2 sector. if you use the wrong mode, the resulting image will not appear to have an iso9660 file system, so there is effectively no chance of getting incorrect file extractions with the incorrect mode – the file system will not function. 7c. extracting files from single-track images the process of extracting files from single-track images is similar to what is described in 5e. use isoinfo with the offset to the session that has the track where the file is originally added to the disc and the pathname. unlike in 5e, use the -n flag (instead of the -t flag) with the session offset. working with an image of a track from the last session of our test multi-session cd, we used the following command to get a listing of all of the files listed in the fourth file system. (this will not necessarily be the listing of every file in every track, but for our test cd which did not exercise “deletion,” this will be a complete listing.) isoinfo -i session4.iso -n 26106 -l we have provided sample output of this command below: [ 24 00] session1.txt;1 [ 11726 00] session2.txt;1 [ 18928 00] session3.txt;1 [ 26130 00] session4.txt;1 we used the following command to extract a specific file: isoinfo -i session4.iso -n 26106 -x '/session4.txt;1' > session4.txt this example works because the file session4.txt was added in the fourth track (within the fourth session), but generally having only an image of the last track does not mean you will be able to extract every file referenced in the file listing from that image. one way to check whether a file is extractable from an image is to observe the corresponding session offset value and compare it against the values in the column to the left of the file name in the isoinfo file listing output. if the leftmost number in the bracketed area is less than the session offset that will be supplied to isoinfo, then the extraction will fail. in the above example, 26106 is the offset value for the fourth session, and the number next to the file name listing for session2.txt is 11726. that means that any isoinfo command that supplies an offset value for the fourth session (26106) will not be able to successfully extract session2.txt. 8. extracting files from physical media with the media in hand, it is possible to directly access files from a multi-session cd. on the graphical interfaces for windows 10, ubuntu 20.04 lts, and macos big sur, when presented with our test data-only multi-session cd, all of the files from every track in each session of the disc were displayed and accessible. when copying files directly from the media, if a multi-session cd includes file systems other than iso 9660, copying directly from the media may not include the originating file system features a current workstation’s file system does not support, (e.g., resource forks from the source file system not being supported in a destination file system in a windows environment). 9. non iso 9660 filesystem data we attempted to generate a test multi-session cd that included hfs file systems using mkisofs and the technique outlined in section 3, but were unsuccessful. (xorrisofs only supports embedding hfs+ partitions, and could not be used for this test.) according to the documentation for mkisofs, “only files from the last session will be in the hfs volume…mkisofs can not add existing files from previous sessions. however, if each session is created with the -part option, then each session will appear as separate volumes when mounted on a mac.”. we did find one resource that suggested hfs discs cannot be “multi-session”, but instead must be “multivolume.” [26] without confirmation through tests, this information suggests that images of individual sessions with hfs data can be accessed through specific file system tools such as hfsexplorer [27] or the hfsutils hmount [28] command; however it is unclear how those same tools might work on an image of the full disc. conclusion when working with optical media, workflows that are unaware of, or do not account for, formats such as multi-session discs may risk losing data. multi-session cd-r/ws may pose significant challenges if treated in the same manner as single-session, single-track media. acknowledgements the authors gratefully acknowledge bez thomas for supplying the inspiration for the title of this article. about the authors dianne dietrich is the digital assets librarian at cornell university library. (orcid: 0000-0002-0009-8736) alex nelson is a computer scientist at the us national institute of standards and technology. (orcid: 000-0002-3771-570x) disclaimer​​ the views and opinions expressed in this article are those of the authors and do not necessarily reflect the official policy or position of any agency of the u.s. government or cornell university. any mention of a vendor or product is not an endorsement or recommendation. logos and trademarks are copyright their respective owners. appendix a: imaging pathways this section describes some of the pathways by which one can create an image of a multi-session cd. this might be helpful in understanding the provenance of images if the physical media is no longer available. guymager when imaging our sample multi-session cd, guymager [29] created a user data disk image of the entire disc. while conducting testing, guymager logged a significant number of bad sectors, even though the resulting image was understandable (using the techniques described in this paper) with isoinfo. dd when imaging our sample multi-session cd, dd [30] gave the following error: dd: error reading ‘dev/cdrom’: input/output error the resulting file was understandable (through file [31] and disktype [32] ) as a disk image containing iso 9660 filesystem data, but it contained only the data from the track of the first session; thus, it appeared to generate a single-track user data disk image. this contrasts with the behavior exhibited by the gui interface for accessing the mounted disk. ddrescue we issued the following command to test ddrescue [33] (installed from the gddrescue package) on our sample cd: ddrescue –n –b 2048 /dev/cdrom [iso file] [log file] the utility reported 7 bad areas and 864 read errors for one of the authors, and 6 bad areas and 861 errors when tried by the other; but in both cases produced a disk image that contains iso 9660 filesystem data, as indicated by running both disktype and file. this image contained data from all tracks from all sessions’ data; thus, it appeared to generate a full disc user data image. to confirm, when we supplied the appropriate offsets we could obtain file listings for all four sessions (using isoinfo), as well as read the contents of all of the files. cdrdao this utility will create a raw image and toc-file for a single session only (see: –session flag ); thus, it is generating a single-session raw disk image. the default filename if none is given for the output is data.bin. a sample command is given below: cdrdao read-cd –-read-raw --session 2 --datafile cd.bin cd.toc the toc-file output, which is specific to the cdrdao [34] utility, includes the same technical data as a cue sheet (tracks, pre-gaps, etc.) in a different structure and layout. the command line utility called toc2cue [35] can convert this toc-file into a cue sheet. appendix b: multiple data tracks ecma-394 defines a “data disc” as “a disc on which every session contains one or more data tracks” suggesting that an individual cd may contain multiple data tracks. however, it was difficult to find examples to support that these kinds of disc were common, or that their creation was supported by cd burning software. to give an example, in the dialogue box for cd burning software (pcmag, n.d.), the option to leave a session open to add new tracks is not available if the user had previously selected to generate a data cd. while there were examples for generating mixed-mode cds in the man pages for cdrecord and cdrdao, there were no examples given in either man page to generate a disc with multiple data tracks within the same session. further, the man page for cdrecord notes that, “many operating systems are not able to read more than a single data track, or need special software to do so.” we attempted to generate two testing discs: one cd that consisted of a single session and nine data tracks and another cd that consisted of three sessions, each with three data tracks per session. in order to generate the first test cd, we used the following command: cdrecord –v –eject –speed=4 track01.iso track02.iso track03.iso \ track04.iso track05.iso track06.iso track07.iso track08.iso \ track09.iso in order to generate a testing disc with three sessions and three data tracks per session we used the methodology outlined in section 3, with each cdrecord step including three data tracks that were each generated (when applicable, using the offset of the previous session) with xorrisofs. we noticed the output of cd-info with both testing discs in the drive reported nine “sessions”, with one data track per “session”. this discrepancy between the layout we intended and the tool output confounded our ability to establish ground truth, and without understanding our initial inputs into the workflow, we chose to not pursue this line of inquiry further. references brown, geoffrey. developing virtual cd-rom collections: the voyager company publications. international journal of digital curation (7): 2 (2012). https://doi.org/10.2218/ijdc.v7i2.226 casey, eoghan, alex nelson, and jessica hyde. standardization of file recovery classification and authentication, digital investigation (31) (2019). https://doi.org/10.1016/j.diin.2019.06.004 dietrich, dianne and alex nelson. the emperor’s new grooves: recognizing multisession cd-rom tracks not captured in disk images. talk at bitcurator user forum 2019. https://bitcuratorconsortium.org/panel-media/ lassere, monique and jess m. whyte. balancing care and authenticity in digital collections. journal of critical library and information studies (3): 2 (2021). https://doi.org/10.24242/jclis.v3i2.125 pcmag. definition of track-at-once. pcmag encyclopedia. accessed 2022-12-02. https://web.archive.org/web/20220127030639/https://www.pcmag.com/encyclopedia/term/track-at-once rothrock, michelle, alison rhonemus, and nick krabbenhoeft. assessing high-volume transfers from optical media at nypl. code{4}lib journal (51): 2021-06-14. https://journal.code4lib.org/articles/15908 van der knijff, johan. imaging cd-extra / blue book discs. open preservation foundation. 25 april 2017. https://openpreservation.org/blogs/imaging-cd-extra-blue-book-discs/ endnotes [1] mixed mode cd – wikipedia [2] compact disc digital audio – wikipedia [3] yellow book (cd-rom standards) – wikipedia [4] orange book (cd standard) – wikipedia [5] ecma-394 [6] iso 13490 – wikipedia, ecma-168 [7] blue book (cd standard) – wikipedia [8] track (optical disc) – wikipedia [9] xorrisofs(1) – linux man page [10] mkisofs(8) – linux man page [11] cdrecord(1) – linux man page [12] ecma-130 (see figure 11) [13] mount(8): mount filesystem – linux man page [14] attempts to use the session=x option in mount with an image of the full disc were unsuccessful; only the file from the first track of the first session was listed at the mount point. [15] isoinfo(1) – linux man page [16] fiwalk – forensics wiki [17] our attempts to force tsk tools to use a sector size, any power of 2, were disregarded by the tsk library on encountering the iso9660 file image. raw data was also translated into user data for analysis. hence, we did not experiment further with attempts to force sector sizes in tsk. [18] fls(1) manual page [19] tsk_recover(1) manual page [20] bchunk(1) – linux man page [21] cd-info(1) – linux man page [22] cue sheet (computing) – wikipedia [23] [libcdio-devel] re: retrieving data session from multisession audio disc [24] https://en.wikipedia.org/wiki/iso_9660#path_tables [25] iso 9660 – osdev wiki [26] information on cd formats [27] catacombae – hfsexplorer [28] hfs utilities home page [29] guymager [30] dd(1): convert/copy file – linux man page [31] file(1): determine file type – linux man page [32] disktype documentation [33] gnu ddrescue manual [34] cdrdao(1) – linux man page [35] toc2cue(1) – linux man page subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – implementing a real-time suggestion service in a library discovery layer mission editorial committee process and structure code4lib issue 10, 2010-06-22 implementing a real-time suggestion service in a library discovery layer as part of an effort to improve user interactions with authority data in its online catalog, the unc chapel hill libraries have developed and implemented a system for providing real-time query suggestions from records found within its catalog. the system takes user input as it is typed to predict likely title, author, or subject matches in a manner functionally similar to the systems found on commercial websites such as google.com or amazon.com. this paper discusses the technologies, decisions and methodologies that went into the implementation of this feature, as well as analysis of its impact on user search behaviors. by benjamin pennell and jill sexton introduction in the summer of 2007, the library at the university of north carolina at chapel hill joined with our partners in the triangle research libraries network (trln) to create a shared catalog using endeca software [1]. the search trln consortial catalog [2] was released in march of 2008, and the unc chapel hill libraries released a locally-scoped public catalog [3] using the shared endeca index in july of 2008. at the time of implementation, trln opted not to include authority records in the shared index. the endeca index is optimized for keyword searching, and does not include the necessary data structure to support “begins with” searching. furthermore, we were not trying to replicate the functionality of a traditional opac and felt that we didn’t need to reinvent a true authority browse functionality; our traditional catalogs are optimized for that particular use. while overall feedback on the new interface was positive, one of the most common criticisms unc’s catalog implementation team received from both patrons and staff concerned the interface’s lack of support for authority searching and browsing. during the summer of 2009, unc chapel hill libraries began to investigate ways we could enhance our users’ search experience by reintroducing some limited authority data into the catalog. at the same time, we had been considering introducing an auto-suggest feature for the catalog interface that would provide real-time suggestions based on what the user types in the search box. users are accustomed to auto-suggest features in most commercial search interfaces and we wanted to see if we could replicate this service using the data in our catalog. populating the auto-suggest feature with authority data from our collection addressed both of these concerns while providing a balance between traditional library use of authority and the expectations and habits that users have formed in using the web. traditional authority browse functionality accepts a user’s query and returns an alphabetical list of nearby authority headings based on a “begins with” match. most opacs force users to adhere to certain conventions when entering query terms: e.g., the requirement to enter author searches in “last name, first name” format; the need to leave leading articles out of title searches; and the need for users to know something about lcsh structure in order to perform successful subject searches. real-world experience, as well as a scan of user search terms gleaned from log data, revealed that many of our users do not follow these conventions when searching the catalog. we hoped to develop a solution that would be easier for typical library patrons to use than a traditional authority search, but at the same time would provide useful suggestions for our more sophisticated users. the result is an auto-suggest feature that presents relevant authority-controlled author, subject, and title suggestions to the user as they type, without users having to understand traditional library opac syntax. we hope that this “do you mean?” strategy of providing real-time suggestions is more useful than a similar service would be if the suggestions were presented after the initial search. methodology request process from the perspective of a catalog user, the process of submitting and receiving suggestions should be a seamless extension of the individual’s standard search interactions with the interface.  the interface presents users with a typical search form, including a query input box, a drop down menu for indicating which type of metadata they wish to search, and a submit button.  as the user begins to type, the interface displays a panel populated by suggested known entities below the query box. a system involving javascript, servlets and solr is used to provide these suggestions in real-time, following the request process outlined below: user begins to type a query into the search box. every 150 milliseconds while the user’s query is changing the client sends an ajax request to the suggestion servlet, containing: query as typed search index presently selected (ie, title, author, subject, keyword, journal_title) the suggestion servlet validates the user’s input and submits it to the suggestion datahandler. the datahandler transforms the user’s input into a solr query, and then submits it to the index as an http request. see section “suggestion query construction” for more details. when solr has completed the requested search, it returns json results to the datahandler. the datahandler then processes these results and returns them to the suggestion servlet. the suggestion servlet writes a json response containing the results restructured as a dictionary containing a list of suggestion/index pairs and the time at which the servlet originally received this request, in milliseconds. user client receives the response. it compares the timestamp of the most recently displayed suggestion results versus that in the json response, and discards the json results if they are older. javascript formats the results into a list of suggestions. repeat steps 2-8 until one of the following occurs: user stops typing. user continues typing, but the new results are contained within a cache of previous results in the auto-suggest user interface. user submits the search, either by pressing enter or clicking the search button. user clicks on a suggestion, causing the client to submit the selected suggestion/index pair. user highlights a suggestion with the up or down arrows keys, causing the client to place the highlighted suggestion in the search box. at this point the user may highlight another suggestion, begin typing again, or submit the suggestion. if the user submits a query, a search behavior parameter is added to the catalog search url, indicating if the user searched for a suggestion, and if the user manually selected a search index. the catalog performs a search as it would normally, and returns results. display results. the user’s search is displayed in the search box, and the index of the last search is selected, whether it was automatically set, manually chosen, or not selected. when the user clicks in the search box, if the search index of the last search was automatically set by a suggestion (if it was an author or subject search), the search index is reset to keyword. figure 1: request process flow diagram. (view larger) user interface we implemented the user interface using a modified version of the jquery autocomplete plug-in [4], as well as css and javascript as necessary to style and configure the plug-in. while it would have been preferable to retain the autocomplete plug-in as is, there were a few behaviors that we could not modify via the provided configuration options. one such change is adding verification that incoming suggestions are more recent than those currently being displayed, since network traffic or slow queries can result in responses appearing out of order. as is mentioned in step 2 of the overview, the interface includes a drop down menu that allows users to specify whether to retrieve all available matching suggestions or to only return suggestions of a particular type. the latter option allows users to separate out suggestions categorized as “titles,” “authors” or “subjects” in order to reduce the number of unrelated matches. for example, a user may be seeking all records authored by john c. adams, but cannot remember the author’s full middle name. to further complicate matters, the suggestion list for an “anywhere” search for “john adams” would be filled with works about presidents john adams and john quincy adams. by selecting “author” from the drop down menu, the user limits the suggestion results to only author entities, and can quickly locate “adams, john crawford” on the ensuing suggestion list. figure 2: the suggestion interface providing feedback for the query “john adams c” even if the user had not decided to preselect “author” from the search type drop down, selecting a suggestion labeled as an “author” (or “subject”) would have still resulted in the interface issuing an author or subject search rather than a keyword search. while this behavior helps to keep suggestion searches as relevant as possible, our test groups did find it introduced some confusion on the subsequent results page. by default, the catalog interface presets the index type on its results display to that of the last search performed so that users can easily modify their query. in the case of automatic pre-selection of the search type, however, users may have forgotten about or not been aware of the change in search type, and would therefore wonder why they were now only receiving author or subject suggestions when they were previously receiving more suggestions. our test groups found the system much more intuitive when we altered the behavior to revert back to a generic “keyword” search type when users began editing queries that had automatically selected a search type. step 9 mentions one other important detail about the jquery autocomplete plug-in; it has a built-in result caching mechanism which avoids making additional requests for suggestions when the user’s input is an extended version of a suggestion query that was already processed and returned fewer than the maximum number of displayable results. in these cases, it instead reuses the results from its cache and limits them by the new portions of the query. web service layer & query construction the web service layer is comprised of two components, the first of which is a java servlet that directly handles interactions with user requests for suggestions. the servlet extracts and validates user input before forwarding it to the solr datahandler, and retransmits results to the user. the second component acts as a datahandler for interacting directly with solr. primarily it handles the generation of solr queries and processing of results. the auto-suggest index is queried via a combination of begins-with, keyword, and partial keyword matching along with result filtering by search index type and collection. additional weighting mechanisms are applied to each search, in general to weight word order matches and popularity more highly. see table 1 for definitions of all the involved fields. while many aspects of the suggestion query are consistent for all cases, such as the maximum number of results (m, where m=15 in the unc implementation) or the result format, there are three attributes that allow us to adjust the suggestion query to provide more meaningful results for specific subclasses of queries: number of search terms (n) if the last search term was a stop word (i.e., “the last of” returns true, “the last” returns false) if the previous search pass had fewer than m results, where there can be up to 3 search passes below are explanations of how these attributes interact to form actual queries, where “terms” is a list of words from the user’s query after it has been tokenized, terms[x]* indicates a partial term wildcard search, keyword indicates a match-anywhere search, and begins-with indicates a word order specific search: case 1: n = 1. only a single search term provided: selected by: keyword(terms[1]*) weighted by: occurs + beginswith(terms[1]) + keyword(terms[1]) in most cases while a single search term is present, that search term is an incomplete word. however, clause 3 of the weighting formula gives full word matches additional weight. case 2: n > 1, last search term was not a stop word: selected by: keyword(terms[1..n-1]) and keyword(terms[n]*) weighted by: occurs + beginswith(terms[1..n]) all terms prior to the last word are considered to be “finished” by the user, while the last term is the active word and is most likely not complete. case 3: n > 1, last search term is a stop word: first pass: selected by: beginswith(terms[1..n]) weighted by: occurs perform the narrowest search, a begins-with. second pass: selected by: keyword(terms[1..n-1]) and keyword(terms[n]*) weighted by: occurs + beginswith(terms[1..n]) the first pass did not return the maximum number of results (m), so broaden the search by treating the last term (a stop word) as a partial term. search behavior is identical to case 2. third pass: selected by: keyword(terms[1..n]) weighted by: occurs + beginswith(terms[1]) passes 1 and 2 did not get full results, so perform the broadest search possible by treating all terms as keywords. case 3 provides a far better match rate for queries ending in stop words by progressively expanding the suggestion selection, but only doing so by as much as is required to get a full set of m results. in our testing, multiple passes were not a common occurrence, but when employed were sufficiently helpful to warrant the extra resources. in addition, two categories of filters are applied on top of the previous three query methods: search index type, where the type can be “title,” “author” or “subject.” source/collection identifiers, where each suggestion retrieved must match all the provided collections. these filters offer a method of limiting the suggestion results based on predetermined criteria. the first filter type enforces matching based on the type field described in the data dictionary in table 1, and provides the behavior described in the user interface portion of this paper for limiting suggestions to particular types of entities. similarly, filtering based on source identifiers limits by the source field, and allows for the pre-scoping of suggestion results to subsets of the data or for other applications. both filters are applied as facet filters in order to improve result caching in solr. dataset for the implementation of this feature, the online catalog served as both the data source and the main platform into which it was integrated. the opac at unc chapel hill is a two tiered system, where an iii integrated library system provides the originating data source while an endeca-based web application acts as the public interface. details of the consortial endeca-based indexing system used by the triangle research library network, of which unc is a part, have been discussed previously [5]. while the auto-suggest enhancement was developed as a component of the unc discovery layer, in reality it operates in parallel to the two layers of our opac and is not directly dependent on the current public interface. when designing this system we were interested in maintaining a platform independent infrastructure that would continue to function even if the library migrated to different indexing software. in order to satisfy the technical requirements necessitated by user keystroke triggered suggestion requests from a large text dataset while maintaining platform independence, we selected apache solr [6] instead of endeca to provide the core indexing and searching functionality, populating it with marc data extracted from the iii ils. we designed the dataset schema to store the minimum level of data required to make suggestions relevant to a user’s query. the data dictionary in table 1 outlines the schema, as well as how values for each field are derived. ac main suggestion field. stores the actual suggestion value that is displayed to the user. internally, values are tokenized around whitespace and non-alphanumeric characters. stop words are not indexed on this field in order to improve performance, but stemming is disabled. all suggestions are stored in lowercase. type: text searchable: true displayable: true required: true type enumerated field used to indicate which of the three index types this data represents. possible values are: author, title, subject. these values are pulled from the following marc fields: author – 100 title – 245 subject – 6** type: string searchable: true displayable: true required: true enumerated: true occurs weighting mechanism similar to a popularity metric. this value is derived in two different ways: authors and subjects: occurs = count(ac,type) titles: occurs = ceiling(sqrt(count(ac,type) + sum(circulation(ac,type)))) where count(ac,type) returns the count of the number of duplicates for all entries matching a suggestion/index pair, while circulation(ac,type) returns a list of circulation values for each instance of a suggestion/index pair. ceiling(x) rounds numbers up to the next whole number. the divergence here is due to the nature of each index type, where authors and subjects are far more likely to repeat than titles. meanwhile, circulation levels (taken from the cumulative circulation statistics in our ils) are not available in all cases, but are useful in judging popularity when present. as a result, they are included in the occurs value for titles, but the value is normalized, using the square root, so as not to dramatically overemphasize items with circulations present. type: sint searchable: true displayable: false default: 0 source originating sources for this suggestion entry, indicating what institutions and/or collections this entry is relevant for. there can be any number of sources per entry. sources are represented by a limited set of textual identifiers, such as “unc”, “filmfinder”, etc. type: string searchable: true displayable: true required: false multi-valued: true enumerated: true ackey this field serves a double purpose. its primary role is to act as a unique identifier for the entry, as there can be duplicate “ac” values with different “type” values that must be tracked separately. solr does not allow multiple fields for an index’s unique key. its secondary purpose is to enable begins-with phrase searching in the collection, as this is not possible with a tokenized field such as a text (in other words, the ac field). solr/lucene allows for full begins-with wildcard searching as long as there are not spaces in the given search phrase, so we replaced spaces with underscores. therefore, this field has the form: replace_spaces_with_underscores(ac)|type for example, “shakespeare,_william,_1564-1616|author” type: string searchable: true displayable: false required: true unique key: true table 1: data dictionary for auto-suggest solr schema data loading there are two variations of the solr index loading process: full loads and partial loads.  in both cases, the process is performed using python scripts that extract the data from the marc extract files used to populate the production endeca index. for both load methods, the selected fields are: 100 for authors, 245 for titles, and 6** for subjects.  subject headings are only loaded in their complete forms, rather than as individual subcomponents.  internally, each type of data is stored and de-duplicated in an index specific dictionary, which also contains the occurs field as described in the data dictionary.  de-duplication is performed as data are retrieved by temporarily storing the selected fields as the keys of a python dictionary data structure, meanwhile tracking the number of collisions in the occurs field.  in the case of titles, circulation data are extracted and added to the occurs value in the partial extract script, only records newly cataloged since the previous update are included. deletions and modifications to existing suggestions do not factor into partial loads; these changes can only take place in full reloads.  to enable either of these functions would have required a large expansion of the data set stored in solr in order to recognize if a record edit should result in the removal of a suggestion from the index.  in our implementation, we perform partial updates twice daily in accordance with the update schedule for the catalog.  full reloads are performed when catalog reloads occur, but this is not a scheduled event. in order to export the data from a python dictionary to a format readable by solr, the loading script generates pipe delimited files containing the full set of properties contained in the data dictionary: ac, type, occurs, source, and ackey.  it should also be mentioned that for partial loads, if a suggestion previously existed, the occurs value is retrieved from solr and combined with the new value.  as the pipe delimited data are generated they are segmented into x number of files, where x is the total number of suggestions by index type divided by the maximum number of suggestions per file (300,000 in the unc implementation).  in our testing, attempts to load more than a gigabyte of data at once slowed down the insert process considerably.  as a result we limited the file length to 300,000 lines to keep the file size below 100 megabytes each.   these pipe delimited files can then be loaded directly into solr via a post request to a built-in servlet that handles delimited inputs rather than xml. development process early in our development cycle, we attempted to use mysql for storing and searching the suggestion collection. we started with this approach because mysql is the standard package used for custom database applications at our institution. while the system did work under this arrangement in a development environment, we worried that a 7 million item database performing multiple text searches per catalog query would exceed the scope of mysql. at that point, we began to investigate the use of solr. this decision carried with it some additional benefits, namely the speed with which solr allows for mixing text search methods and other weighted ranking metrics. we did briefly experiment with alternative search methods, such as double metaphone phonetic matching [7] to compensate for spelling errors, but concluded that this introduced too much noise into the results. after considerable tweaking and research into the behavior of other similar suggestion features in commercial websites, we settled on the mixed begins-with and keyword model described in this paper. the adoption of this model allowed for easier suggestion of unknown or partially known items than a begins-with only model, particularly for lcsh subjects where users often do not know the proper syntax and formatting. aside from these concerns, refinement of the solr query structure to accommodate more difficult cases, such as stop word heavy inputs or author names that commonly appear as titles, required a significant amount of development time. ultimately, there were additional refinements that, while beneficial to the quality of the suggestions, resulted in a performance hit. if the system were not required to provide users with immediate feedback, as in a more traditional synchronous query and response model, these enhancements likely would have been reasonable inclusions. it should be mentioned that some tweaking of rank weightings would likely need to be done at any institution implementing this feature to best match their own datasets, even if the algorithm were consistent. additionally, in our original design we had planned to include user queries as well as suggestions extracted directly from our catalog data, merging the two sets in a method similar to that described by yang et al. [8]. however, concerns over privacy, index maintenance, and development time eventually led us to include only bibliographic and circulation data. analysis to determine the effectiveness of the auto-suggest feature we have performed periodic access log analysis since the feature’s launch in january 2010. based on this information, we found that users began using the feature immediately upon its implementation, with 18.99% of queries issued in the first week using suggestions from our index. since then, the usage rate has averaged 18.06% of searches from forms that offered the service. some pages, such as our advanced search pages, do not at present employ the suggestion feature, but 88.68% of all searches are made from forms that do offer it. users perform on average 7682 new searches in the catalog per day, which generate an average of 38426 requests to the auto-suggest service, an average of about 5 requests per search performed. one of our primary interests in implementing this feature was to determine what effect it would have on users’ utilization of authority data. as a result, we focused much of our attention in statistics gathering on tracking the changes in usage of index specific searching and how these related to which suggestions users selected. figure 3: change in the average number of search requests per day by index after implementing auto-suggest. (view larger) in doing so, we discovered that the largest change in search behaviors was in the rate of subject index searching. although subject searches are a relatively minor portion of the overall queries performed in the catalog each day, just 3.92%, they have seen the largest percentage increase in use. during the baseline period before enabling real-time suggestions, there were on average 115 subject searches in the catalog per day. after implementation, the average number increased to 343.8, a 198.6% increase, of which 222.7 were from the suggestion service. similarly, author searches increased 69.64% over the same period. overall, the total number of searches in this period increased 42%, so the changes for subject and authors are significantly higher than that of the overall set. it is not clear how much of the total search increase is related to the time of the academic year and how much is related to changes in the catalog, but both are likely factors. we attempted to minimize the impact of the former by choosing a baseline during moderate to heavy traffic periods during the 2009 fall academic semester, but we did not have log information available far enough back to compare year over year changes. figure 4: percentage change in the average number of searches issued by index per sampling period versus the baseline period before the implementation of the auto-suggest feature. the all indexes series represents the change in the average number of searches across all indexes. (view larger) users choosing to preselect a search index made up 14.52% of suggestion searches. the increase in subject and author suggestion searches are mainly due to the automatic index selection feature described in the user interface portion of this paper. on the other hand, generic keyword searching, which makes up 56.89% of all searches, saw a 3.01% decrease in usage during this period. the impact of the service on system resources has so far been minor. as of the launch of the service at unc there were 7244000 suggestion entries, including 1131000 authors, 3933000 titles, and 2179000 subjects. these suggestions were derived from around 4.95 million individual records in the unc catalog. the actual memory usage of the index was roughly 115mb, excluding caching, and suggestion queries average around 200ms execution times. it should be noted, however, that the implementation of this feature resulted in an increase in the average number of terms per catalog query, which can have an impact on performance of the catalog itself. conclusion providing users with real-time feedback in the form of query suggestions provides unique opportunities for presenting users with catalog authority data in a familiar and helpful manner. since doing so, the unc libraries’ discovery layer has seen a significant increase in the rate of subject and author searches, where almost 2/3 of subject searches are selected directly from the suggestion service. the use of apache solr has offered a scalable and flexible back-end for implementing a feature that has been adopted by users for 18% of all searches in a major research library’s online catalog. jquery’s autocomplete library allowed for a heavily configurable user interface experience, while only requiring a minor time investment to establish the basic functionality needed to enable the user’s interaction with a query suggestion web service. the success of this project has led us to consider other potential applications for the auto-suggest service. we are currently investigating the possibility of integrating the service into tools used for local non-marc metadata creation, such as for digital collections materials, or items in the institutional repository. in these collections, which often depend on student labor for metadata creation, we face challenges in normalizing descriptive metadata within and across collections. a tool that would allow users to enter a descriptive term and receive a list of possible authorized forms of the term would both improve the accuracy of descriptive metadata and speed the description process. we invite your comments and questions about this service, and would welcome your ideas for other useful applications of this tool. references [1] endeca information access platform: http://www.endeca.com/products-information-access-platform-mdex-engine.htm [2] search trln: http://search.trln.org/ [3] unc chapel hill library’s online catalog: http://search.lib.unc.edu/ [4] jquery autocomplete plugin: http://docs.jquery.com/plugins/autocomplete [5] apache solr: http://lucene.apache.org/solr/ [6] hemminger, b. & lown, c. (2009). extracting user interaction information from the transaction logs of a faceted navigation opac. code4lib journal, (7).  http://journal.code4lib.org/articles/1633 [7] wikipedia: http://en.wikipedia.org/wiki/double_metaphone [8] yang, j., et al. (2008).  search-based query suggestion.  proceeding of the 17th acm conference on information and knowledge management, 1439-1440. (coins) about the authors benjamin pennell (bbpennel@email.unc.edu) is applications analyst and jill sexton (jill@email.unc.edu) is information infrastructure architect at the university library at the university of north carolina at chapel hill. together they are unc’s catalog implementation team. subscribe to comments: for this article | for all articles one response to "implementing a real-time suggestion service in a library discovery layer" please leave a response below: technical skills for lis professionals « librar*, 2011-06-25 […] on june 25th, 2011 i am pretty fascinated by some of the really great developments to library web interfaces i’ve seen recently. i am also a bit disillusioned at times by the lack of it flexibility […] leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – creation of a library tour application for mobile equipment using ibeacon technology mission editorial committee process and structure code4lib issue 32, 2016-04-25 creation of a library tour application for mobile equipment using ibeacon technology we describe the design, development, and deployment of a library tour application utilizing bluetooth low energy devices know as ibeacons. the tour application will serve as library orientation for incoming students. the students visit stations in the library with mobile equipment running a special tour app. when the app detects a beacon nearby, it automatically plays a video that describes the current location. after the tour, students are assessed according to the defined learning objectives. special attention is given to issues encountered during development, deployment, content creation, and testing of this application that depend on functioning hardware, and the necessity of appointing a project manager to limit scope, define priorities, and create an actionable plan for the experiment. by jonathan bradley, neal henshaw, liz mcvoy, amanda french, keith gilbertson, lisa becksford, and elisabeth givens. background qr code audio tour at virginia tech, guided tours of newman library, the main branch, were provided for incoming freshmen. the tours were approximately an hour long, and were led by librarians. as the size of incoming classes increased – now at over 6500 incoming students for the class of 2019 (freshman snapshot) – it was recognized that self-guided tours would allow more effective use of librarian time, as well as give each student more flexibility as to when to visit the library for her orientation. the classroom manager and technology liaison for the library, neal henshaw, used instructional design techniques to create a self-guided tour for new students, based on qr code technology and ipad 2 tablets. the original self-guided tour was designed primarily as an audio tour. students began at the circulation desk, and were shown by circulation staff how to scan a qr code which plays a video that gives an introduction to the tour. the introductory video described the tour, explained procedures for ending the video at each stop and moving to the next stop, and detailed a workaround for problems with unreliable wi-fi. though the original tour was primarily designed for audio, most of the scan targets on the tour were actually video files. the video was used only occasionally, to help the student identify a tour stop by showing an image of the area, or to demonstrate computer search techniques. figure 1. a qr code placard from the current library audio tour in the original, qr code based self-guided tour, information about the stops was carefully written into a script for each of nine stops. the audio on the tour was narrated directly by neal, who has a perfect radio voice. the tour introduces the students to quiet and group study areas, circulation and reserves, reference, and collaborative projects with other university groups, including a writing center and a communications lab. students are also shown basic instructions for accessing the library web site and performing searches using summon. this self-guided library tour has been required for the standard english course for the freshman class, introduction to writing. students taking the tour for the purposes of this course are required to take a quiz after completing the tour. technology update: mobile app with ibeacons what’s an ibeacon? an ibeacon is a bluetooth low energy (ble) device that broadcasts a unique identifier, repeatedly. mobile applications can be created to listen for these identifiers, and perform location or context aware activities when the identifier is detected. (ibeacon 2016) the ibeacon identifiers are set up in three parts: a uuid, a major number, and a minor number. the beacon identifiers are configurable by the beacon owners; the method for doing this is configurable according to the brand. the most common example given for the use of ibeacons is in retail. the ibeacons can be placed near retail store locations, and the retail store brand’s app running on a user device receives a notification when a user is near one of the stores. the app can alert the user to sales or special events at the store, and attempt to entice the user to enter. the ibeacon aware apps can also detect specific locations within a store, with the use of major and minor numbers. the apple developer documentation for ibeacon cites an example where all beacons belonging to a retail store with multiple locations are assigned one uuid, while a differing major number is assigned to each location of the store, and a different minor number to each department in the store. (getting started … 2014) the app for the retail store can now perform context-sensitive actions when a person enters each department in each specific store. figure 2. estimote beacons. photo courtesy of estimote, inc. updating the tour we chose to update the tour primarily because this saves students the step of having to fumble with qr codes at each stop. converting the tour to a native app will also give us the opportunity to improve the experience in the future, through better presentation and assessment of learning content. this has been an especially challenging project, primarily because it requires time and coordination from several teams, all of whom are busy on several other projects. over the last few years technology units within the library have been better on completing technology projects within units specifically created for that task. this project has been different because graphics designers, video editors, marketers, script authors, education and learning specialists, and developers have all been needed. during planning and implementation of this project we have faced library reorganizations, the loss of the initial designer to other employment opportunities, a deficiency of test hardware capable of working with the technology, and the loss of a talented project manager because the project was not within her primary area of responsibility. we’re excited to finally be implementing and deploying this application, and to tell you how we have been able to scrounge resources in order to pull this off. user roles during the design of a technology application, it is sometimes useful to think about the roles of each user. so, let’s begin with the primary role, that of the student. student liqin is a first year student that has just moved to blacksburg to study computer engineering. liqin was assigned to take the self-guided tour of the library by an english instructor, but does not understand why this is necessary. during the tour given to high school students and parents wishing to choose a university, it was clearly stated by the tour guide that there’s no need to actually go to the library; everything is online. however, the student tour guide could be wrong; he also stated that the library was open 24/7, when in fact it is actually open 24/5, which in real life means that there are only 4 days in the week when the library is truly open 24 hours. whatever the hours of the library are, liqin has only one and a half hours between the end of quidditch and the beginning of aerial robotics club to eat and take the self-guided tour of the library. in addition, there are some other use cases for minor roles that we had in mind when we designed the app. instructor dr. jones is a professor associated with the english department, and teaches four sections of the introductory writing course. dr. jones needs to make sure the students know how to use the library for research, and wants to be able to follow up with each student that might be having difficulties understanding how to use library resources. learning services and education technologies devon is a librarian responsible for helping students learn how to use the library, in both face-to-face and technology based scenarios. as the library undergoes incremental remodeling and reorganization, new spaces and services are introduced each year. this year, a studio of 3d printers has been added to the library. devon needs a way to update the tour that incoming students are shown so that it explains the 3d printing studio, preferably without waiting for a software developer to become available. requirements thinking about the three roles in the system helped to determine requirements for the application. requirement supports student supports instructor supports library not in this version load specific video file when near a beacon x x x ask questions at every stop x x display important information on screen x give students the correct answer when they miss a question x subtitles for the deaf and hard of hearing x x content needs to be easy to update x includes maps x runs on library-owned equipment x records pass/fail grade for each student x x records name of student and class, and stops visited x introductory screen x offer help for wi-fi problems x students visit stops in order x additional languages x run on student-owned equipment x use indoor mapping x using the roles to determine the system requirements was a useful exercise. as you can see from the chart, some requirements support multiple roles. some requirements were determined not to be core requirements at all, but have been deferred instead as wishes for future versions. this type of chart doesn’t reflect that some requirements are basic capabilities for this type of app, regardless of the roles that we defined. interface design a basic understanding of the requirements for the application gave liz mcvoy, the digital media specialist, the ability to design the interface using a largely visual approach. liz designed screens that reflect the requirements. introductory screen figure 3. the introductory screen for the new tour app click to enlarge the introductory screen is designed to entice the student to explore the library. there is a single button, labeled “begin”, to start the tour. the color scheme, referenced throughout the interface, is that of a fantastic sunset behind hokie stone. main interface figure 4. the main user interface for the tour app click to enlarge the main interface is the primary screen that students will see while taking the tour. this main screen has several elements taken from the requirements document: a large window that displays videos for each tour stop a “maps” button that displays maps of the library a “wifi help” button to help the student with connectivity issues a display area to show important library facts there is also a progress bar to indicate the remaining stops on the tour. explore prompt figure 5. an example of a prompt in the tour interface click to enlarge this prompt appears during the tour when students are asked to look at specific details in the environment. maps figure 6. an in-app map of the second floor of newman library, with tour locations marked click to enlarge the app has maps of each library floor on the tour, with paths and numbered markers showing how to get to each tour stop. maps are potentially important, because even people who work in the library occasionally get lost. the ibeacon technology allows for the possibility of indoor mapping using signal triangulation to detect location more precisely, and to give directions. the current version of the app doesn’t use this ability; instead, the beacons associated with each tour stop location are stored in a list. the correct video plays when the student is near the correct beacon for the next stop. wi-fi help screen figure 7. the wi-fi help screen this screen was included in design sessions because in the past, the library has often had problems with wi-fi connectivity, especially when moving throughout the library. the qr code version of the tour has a video that shows students the steps to perform on the device when there are wi-fi problems. it might be possible to eliminate this screen in the deployed implementation, because the videos are now stored on the device instead of streamed via wi-fi. an internet connection is still required in order for a student to sign-in and take the quiz, but this is done at the end of the tour in an area with reliable wi-fi coverage. sign-in screen figure 8. student sign-in is required for students to get credit for the quiz this screen is used by students to “sign-in” to the tour. after the tour is complete, the tour will record the email address of each student so that instructors can keep track. after signing in, students will take the quiz. quiz interface figure 9. the multiple-choice quiz interface click to enlarge the quiz interface presents students with a series of multiple choice questions. the questions that are asked were covered in the tour stop videos, and in many cases, the answers to these questions were shown in the display area of the main interface. the buttons to answer the questions are very large, to accommodate people with big fingers, while still allowing for use by people with small fingers. completing the quiz leads to the final screen. final screen there has been intense debate about whether balloons should be used as the final image, or fireworks. currently, the balloon proponents are winning the debate. figure 10. the final screen, shown after the tour is completed click to enlarge figure 11. an alternate final screen,with balloons instead of fireworks click to enlarge reboot and mini-hackathon after the project had languished for some time due to lack of resources, the project was rebooted. jonathan bradley, the web learning environments application developer, became the main developer on the project. after observing amanda’s success in managing and completing other projects with limited resources, keith gilbertson, another developer on the project, attempted to get amanda french involved in the project. amanda held a reboot meeting and clarified requirements on the app. from the meeting, each person was finally assigned a set of action items. she created a project in virginia tech’s gitlab environment, documented the project, created an issue tracker, and organized a mini-hackathon and proof-of-concept demo. during the hackathon, jonathan and keith worked on coding what we perceived as the most important pieces of the app: the code to detect the beacons, and the code to detect the appropriate video. shortly after the hackathon, the developers were able to present a demonstration to the rest of the team. beacons were placed in several locations throughout the library. the team walked from beacon to beacon, and observed as the correct video played in each location. during the hackathon and the resulting demonstration, we noticed three interesting issues. first, in the time it takes to walk from station to station, especially when the stops are between floors, the ipad screen would go blank. when the screen went blank, the app would stop searching for beacons. our current workaround for this is a simple one. the ipads are library owned, and we have chosen to disable automatic screen blanking on the ipads. second, the beacons had been placed approximately one half hour before we started our demonstration. as we approached a stop on the first floor, where a beacon had been placed in an out of the way area, but not secured, we observed a student handling and examining the beacon. in recognition that brightly colored technology devices might be interesting to visitors at a university with a large number of technology students, secure, 3d printed mounting devices have been proposed. the mounts will also help with another issue – sometimes the adhesive isn’t strong enough to keep the units attached directly to certain surfaces. third, we learned that the ipad units that were allocated for library circulation at the time would not work with our app. we initially tested the estimote sdk on a newer, personal ipad air 2 model successfully. the same test failed when used with the older ipad 2 models owned by the library. after researching the issue, we found that the ipad2 model is not compatible with the bluetooth low energy specification, meaning that it will not work with the ibeacon technology. however, new ipad air 2 units were being ordered for the library, and they arrived before we would finish work on the app. after the hackathon, amanda was no longer available to work on the project due to the workload in her primary areas of responsibility. however, her work was useful in organizing and defining the project, and helping us to make progress. implementation choices jonathan bradley became the lead developer on the project around the time of the hackathon, and as evidenced by e-mail timestamps, at times had to work in the wee hours. swift was chosen over objective c as the programming language for the ios app. this is primarily because ios development in general tends to be shifting over to swift, which has been open-sourced. swift also has a reputation for being easier to read and easier to maintain. (solt 2015). the estimote sdk required an objective c bridging header in order to be used with swift. the procedure is documented at http://developer.estimote.com/ibeacon/tutorial/part-1-setting-up/#swift-users-add-an-objective-c-bridging-header. we chose to use the latest, non-beta version of xcode as our development tool. xcode is the standard development tool from apple for creating ios applications, and requires a system with os x to run. by choosing the latest release version, but avoiding beta we are able to use new features without sacrificing stability, and can test on devices with release versions of ios. however, due to the unexpected timeline of the project, the beta versions of xcode eventually made it to release status, and this required changes to our code as the relatively new swift language has evolved. source code for the project is stored in our gitlab repository, and will be open-sourced and placed on github when proven. we are using estimote beacons, primarily because they were the first available when the project was envisioned, but we have decided to stick with them for the initial implementation because of a strong sdk and a large developer community. this version of the app is loaded manually onto library-owned ipads, instead of being distributed through the app store. learning objectives while designing the new version of the app, learning services came up with a set of eight core learning objectives. by the end of this tour, the student will identify the ways in which they can receive reference help (ref desk, college librarians, ask a librarian chat). by the end of this tour, the student will differentiate how special collections is different from the rest of the library, by identifying its unique mission and collection specifics. by the end of this tour, the student will list the type of items that can be checked out through circulation services. by the end of this tour, the student will compare and contrast between different study spaces and the benefits of each. by the end of this tour, the student will be able to find library materials using summon. by the end of this tour, the student will differentiate the services offered by the comm lab, writing center, and 3d printing studio. by the end of this tour, the student will explain the basics of using the library print collection (reading call nos., library terminology, returning unshelved books, classifications). by the end of this tour, the student will describe how to print from their devices and locate the printers in the library. the learning objectives, along with the tour stops, were used to create a script for the new tour videos. videos as part of the overall renewal of the self-guided library tour, new videos were desired. there has been some difficulty in making this happen. the library has video equipment and an event capture team that, upon request, records and edits events around the university. they’ve been booked solid. additionally, creating new videos requires collaboration between educators, who determine learning objectives, scriptwriters, video recording staff, and editors. as mentioned earlier, collaboration across functional groups within the library is often difficult. in order to make progress on the project, an outside consultant was contracted. the outside consultant is a neighbor of one of the project members, and was hired from money out of his own pocket. while this was useful in making progress on the project, we are not recommending it as a general practice for developing this type of application! use library funds when possible. fortunately, the consultant that was hired is an experienced videographer and has won competitions in creating short videos. armed with the learning objectives, a list of tour stops, and scripts from the old videos, the videographer consultant has been recording new videos. while the original videos were meant to be primarily auditory in nature, the new videos are meant to have some visual appeal and humor. here’s a draft of one of the new videos. figure 12. old-timey lettering in a screenshot from one of the new tour videos note the young sounding voice in the video. hypothetically, a younger voice will appeal to our “typical” college audience, but this is an accidental experiment. our videographer is 15 years old! she has offered to have the videos narrated by her father, if the young voice doesn’t work out, or by neal, the previous narrator, who has a professional-sounding, radio-quality voice. operational challenges we’ve struggled with many of the same issues in deploying a beacon app as the brooklyn museum – including lost beacons, and beacon confusion. (bernstein 2015) while the availability of different beacon colors is not as much of an issue in our deployment as in that of an art museum, where bright colors may act as visual distractions from exhibits, we wished that the beacons would either blend into our walls, or come in our school colors of maroon and orange. battery replacement we have two different revision numbers of the estimote beacon hardware, and the differing revisions use different battery types. the initial beacons, which were considered beta hardware, but will be used in our production application, use cr2477 coin batteries. the newer estimote beacons use cr2450 batteries. we need to keep a supply of both types of batteries on hand. the older, beta beacons require a minor surgery, and must be sliced open in order to replace the batteries. in the newer beacons, the adhesive backing can be separated from the main unit, and the battery is easily replaced. figure 13. a cross-section of the newer revision of estimote beacons, showing the battery location. photo courtesy of estimote, inc. we don’t yet have statistics for how often the batteries will need to be changed, but an initial estimate is every four months. the devices broadcast the id repeatedly after a specified advertising interval. estimote initially claimed a battery life of two years, but this was based on an advertising interval of 900ms, while apple recommends a much shorter, and energy expensive, advertising interval of 100ms. however, we can control power expenditure somewhat by adjusting the signal strength of the beacons, instead of just the advertising interval. we’re optimistic because of the current design of our application. we’re not using indoor location services. each stop is far away from other beacons, and will be visited in a specific order. this combination of characteristics means that we are able to set the broadcast power quite low on the beacons, which will extend battery life. updating videos at existing stops in order to support the requirement that video content can be updated easily by the library, the app has some built-in functionality to check for new videos when it starts. videos are stored in google drive. when the library needs to update content for a specific stop, we will place a new video with the same name into the google drive folder. the library will then restart the app on the library-owned ipads, and the new video will be detected and downloaded. as the videos are now pre-loaded, this reduces the need for dependable wi-fi at all stops. videos are large, but the current version of the application runs only on library owned devices, so we don’t yet need to worry about taking too much storage space on the device. adding new stops adding a new stop to the tour is currently a heavyweight operation that requires many hands and many steps: write a narrative script for the new stop produce a new video for the stop deploy a new ibeacon device at the stop location add the stop information to the app code and recompile redeploy the app to the library ipads all of the stops on the tour, along with information for the associated beacons and videos are currently hard-coded into the app. // this represents a stop on the tour struct tourstop { let major: clbeaconmajorvalue let minor: clbeaconminorvalue let identifier: string let videoname: string // filename of video, without .mp4 extension } the stops are ordered according to their position in an array: let tourbeacons = [ tourstop(major: 10, minor: 5, identifier: "reading_room", videoname: "04_readingroom"), tourstop(major: 10, minor: 1, identifier: "special_collections", videoname: "05_spec"), tourstop(major: 20, minor: 5, identifier: "reference_desk", videoname: "09_refdesk"), ] we anticipate that we will be able to eliminate the recompilation and redeployment steps by writing code that retrieves stop information from a network location at the start of the tour. collecting student data the initial version of the app uses the the same method for collecting student data as the original self-guided tour. students log into a web page and take a quiz. this is where we hope that future versions of the app will have improvements that help the students, teacher, and library. the goal is to automatically record grades into canvas, our new university-wide learning management system. tracking beacon id numbers while coding for the post-hackathon demonstration, one morning the app stopped working. after troubleshooting, jonathan discovered that the id numbers on the beacons were different from the id numbers that had been coded into the app. either the id numbers had been reset when the batteries were replaced, or we accidentally mixed up our beacons. we decided that, especially for deployment and operational purposes, we would need a way to track the id that the app expects for each beacon location. we have a google drive document for this purpose, which looks like the following: location color real color uuid major minor reference desk mint cocktail light green library default 20 1 special collections icy marshmallow light blue library default 10 1 4th floor blueberry pie purple library default 40 1 learning commons mint cocktail light green library default 20 5 reading room icy marshmallow light blue library default 10 5 the major number corresponds to a particular floor in the library. major number 10 corresponds to the first floor, 20 corresponds to the second floor, 30 to the third floor, and so on. the minor number, (when taken along with the major number), corresponds to a specific stop on a specific floor. the beacon with major number 20 and minor number 1 corresponds to the reference desk on the second floor; the beacon with major number 20 and minor number 5 corresponds to the learning commons on the second floor. for the purposes of our current app, it isn’t necessary for all of the beacons on the same floor to share the same major number, but the organization helps our mental model and will make things easier if we ever decide to create an app that uses the beacons for indoor mapping. we’ve also left large gaps in the sequence for minor numbers on each floor, in case we add more beacons later. an oddity of the chart is the existence of both “color” and “real color” columns. the “color’ column displays the estimote name for the beacon color. the estimote names, such as “icy marshmallow”, were confusing to us, so we added a “real color” column. the “real color” is the basic color of the beacon, to the eyes of a person that does not have color blindness. an icy marshmallow beacon is actually light blue. if we ever paint the beacons, we will update the “real color” column. we kept the “color” column, because the beacon reports this color information in estimote apps that are used for beacon configuration. this document is useful in case a beacon needs to be swapped in temporarily for another that is having its battery replaced; it makes it easier to set the id of the replacement beacon to the correct settings. unfortunately, the app doesn’t read directly from the document at this time. it serves only as an aid for humans to maintain the beacons. beacons can be easy to lose. one hectic morning, one of the developers who had a slight problem with insomnia had taken home a set of beacons and was working on the project at odd hours. while packing up to go to the office, he noticed that one of the beacons was missing. the “icy marshmallow” beacon was eventually found with the aid of a bluetooth scanning application in a box of cereal on the pantry shelf. beacon security we are having some early problems with beacons staying on the wall, possibly due to the adhesive on the backs of the beacons. two beacons went completely missing during testing. we have proposed, but not yet designed or printed custom mounts for the beacons using our 3d print lab. future opportunities additional languages our assistant director for international outreach has proposed an idea to increase the usability of the tour for new international students. some international students arrive at virginia tech fluent in english; others have difficulty listening in real time to fast conversations that have domain specific terminology, such as conversations about libraries. for example, think about the terms “stacks”, “interlibrary loan”, “course reserves”, “circulation”, and “journal database.” in order to facilitate the introduction of the library to these students, the director has suggested making the tour available in other languages, in particular arabic, spanish, and chinese. this could be done incrementally. first, the subtitles for the deaf and hard of hearing could be translated into written representations of these other languages. students would hear the videos in english while seeing subtitles in a preferred language. as a next step, the audio for each stop could be recorded again in each language, and students would be asked at the beginning of the tour to select a preferred language so that the appropriate audio track could be used. the international outreach director has offered to recruit students to translate and record the videos. student-owned devices this version of the tour was developed for the ipad air 2 devices owned by the library. having a single known device eased development, because we didn’t need to be concerned with multiple screen sizes, multiple operating systems, or with differing video playback capabilities for each type of device. for future updates, we have in mind that students would be able to bring their own devices into the library and take the self-guided tour. the first update would allow the tour to be taken on all ios devices that students might bring, including iphones. the next update would allow the app to run on android devices, including tablets and phones. the android update would require significantly more developer time, as the android apis are based on an implementation of the java language, and all of the code would be new. making the tour available to student owned devices would also mean that the app would be distributed by the app stores for each platform. currently, the tour app is loaded manually onto the ipad air 2 tablets that belong to the library. references bernstein, shelly. the realities of installing ibeacon to scale. [internet]. 2015 feb 04. available from https://www.brooklynmuseum.org/community/blogosphere/2015/02/04/the-realities-of-installing-ibeacon-to-scale/ freshman snapshot [internet]. [updated 2015 may 15]. virginia tech. available from: http://www.admiss.vt.edu/apply/freshman-snapshot/ getting started with ibeacon [internet]. 2014 june 02. apple, inc. available from: https://developer.apple.com/ibeacon/getting-started-with-ibeacon.pdf ibeacon [internet]. [updated 21 feb 2016]. wikipedia. available from: https://en.wikipedia.org/wiki/ibeacon solt p. 2015 may 11. [internet]. it’s high time to make the switch to the more approachable, full-featured swift for ios and os x app dev. infoworld. available from: http://www.infoworld.com/article/2920333/mobile-development/swift-vs-objective-c-10-reasons-the-future-favors-swift.html about the authors dr. jonathan bradley is the web learning environments application developer at virginia tech’s newman library. jonathan creates educational technologies and trains faculty and staff in their use, both inside and outside of the library. he is the technical admin for virginia tech’s libguides instance, and most of his previous research has surrounded pedagogy at the university level. neal henshaw is the educational technology and instructional designer at virginia tech’s newman library. neal designed and maintains the self-guided tour that freshman take, and also creates tutorials for use in online instruction. neal has a background in education in both the public and private sectors, as well as experience as a server/network administrator. liz mcvoy lends her keen design and film eye to virginia tech’s university libraries as the digital media specialist. she has a strong background in video production and graphic design, and combines these skills to create impactful, visually exciting film and design pieces that break down the invisible “fourth wall” to reach students, faculty, staff, and other users around virginia tech and beyond. part of her role is to manage the event capture service, which offers the recording and storing of scholarly lectures in the university repository, vtechworks. this service invites people all around the university into a new relationship with the libraries, highlighting just one of the ways we can partner with them to learn and be more successful. former clir postdoctoral fellow amanda french is currently director of digital research services and associate professor at virginia tech university libraries, where she is helping to build digital humanities infrastructure and is running the institutional repository, vtechworks. from 2010-2014 she was thatcamp coordinator and research assistant professor at the roy rosenzweig center for history and new media at george mason university. her particular expertise consists of making humanities content (both cultural content and scholarly interpretation of that content) openly available online, as well as introducing scholars to the various methods of and issues with making humanities content openly available online. keith gilbertson serves as a digital technologies development librarian at virginia tech. lisa becksford is the learning services and educational technologies librarian at virginia tech university libraries, where she provides face-to-face and online learning experiences that help students across the university grow as researchers and scholars. elisabeth givens is the director of digital strategy and outreach at virginia tech’s university libraries, where she oversees the development and implementation of marketing initiatives and outreach campaigns that serve to connect the virginia tech community with the services and resources provided by the libraries. prior to virginia tech, elisabeth was the senior social catalyst at 919 marketing, a national content marketing agency based in raleigh, nc, where she worked with clients of all sizes, from startup companies to fortune 500 corporations. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – determining usability of vufind for users in the united arab emirates mission editorial committee process and structure code4lib issue 19, 2013-01-15 determining usability of vufind for users in the united arab emirates in late 2011, the higher colleges of technology, a higher education institution in the united arab emirates, implemented vufind as the search interface for the libraries’ resources. before launching vufind in the 2012 academic year, usability testing occurred across three campuses to test the functionality of the search interface features. twenty-one participants, including emirati students and expatriate faculty, were tested using a performance based assessment along with think-aloud protocol, which was recorded using camtasia screen capture software. as a result of the testing several features of vufind were customized including language, layout and prioritization of results. the current study builds on the limited existing body of literature on vufind, which has previously indicated a number of design elements and practices which should optimize user experience. several key findings are consistent with and confirm results from prior studies with findings from this study adding to the literature by observing how or why linguistic orientation affects user behavior in search systems. by nicole johnston, alicia salaz, and rob o’connell introduction in 2011 the higher colleges of technology (hct) formed a usability committee to look at usability throughout the library system. the hct library system consists of seventeen campuses with a central department of library technical services (lts) overseeing electronic resources and systems for the whole organization. the library, as of 2011, had been using two search interfaces for its resources; innovative interfaces’ integrated library system, millennium, for physical library holdings, and serials solutions’ product, summon, for electronic resources. the supervisor of lts initiated development and testing of the open source software vufind as a possible means of unifying the interface for these two search systems and creating a more seamless experience for hct library users. vufind was conceptualized in this circumstance to sit on top of both search systems and mediate queries between them and users. there is limited literature outlining the use and testing of vufind as a search interface, especially outside of america. western michigan university (wmu), york university, the university of illnois and yale university have all published articles on the testing and usability of vufind (bauer, 2008; denton & coysh, 2011; emmanuel, 2011; ho, kelley, & garrison, 2009). this literature was used as a basis for the initial design of the interface, but the usability committee at hct felt it important to pursue a study of our particular users in the context of a public institution of higher education in the united arab emirates.  this user group is comprised of primarily emirati students who are non-native speakers of english, as well as a group of globally diverse faculty members. vufind is highly customizable and each implementation is quite different. this article will describe core features of the design adopted by the hct along with functionally important snippets of coding for any developer attempting to replicate the design.  additionally, details of usability testing conducted with hct users and a discussion of findings and design changes that were made as a result will inform developers in making decisions about vufind implementation. design concept the higher colleges of technology (hct) created its vufind interface to cater to both first-time patrons and advanced users.  we initially designed the interface to be a wrapper for the summon electronic resources search service with the intention of continuing to use innovative interfaces’ millennium opac for physical holdings material (“local” material).  as we began our testing, we quickly realized that we needed a simpler search mechanism for first-time patrons.  after several design changes, we settled on a layout which would divide the results of an initial search into two columns, with electronic resources and local material divided respectively to the right and left.  an “everything” button was included to shift patrons to a single-column vufind-summon interface which also offered more advanced search options. designing the split column layout the split-column layout provided us with the opportunity to ease new patrons into the library catalog.  patrons searching for light reading material such as graphic novels, graded readers or fiction can quickly locate their resource and not be bogged down by facets and limiters.  exact title searches for books and electronic resources can be quickly located.  patrons have the ability to compare our local content with the electronic resource collection.  this works particularly well for ebooks as patrons may select an ebook if their local copy is checked out. the split screen requires the addition of a resource folder in web/services and three new files: ajax.php, home.php, and results.php.  additionally, interface adjustments must be made in the blueprint theme to include two separate searches – one for solr and one for summon.  to facilitate fluid interaction between solr and summon, a toolbar was added to each page that would switch according to the module selected or the template used.  the code is located in blueprint theme under layout.tpl: {if $module == "summon" && $pagetemplate != 'advanced.tpl' && $pagetemplate != 'record.tpl'} {include file="$module/searchsum.tpl"} {/if} {if $module == "summon" && $pagetemplate == 'record.tpl'} {include file="$module/searchsumrec.tpl"} {/if} {if $pagetemplate == 'advanced.tpl'} {include file="summon/searchsumadv.tpl"} {/if} {if $module != "summon" && $pagetemplate == 'record.tpl' } {include file="search/searchnav.tpl"} {/if} {if $pagetemplate == 'view.tpl' } {include file="search/searchnav.tpl"} {/if} {if $module=="resource"} {include file="$module/searchall.tpl"} {/if} {if $module == "search" && $pagetemplate != 'history.tpl' && $pagetemplate != 'advanced.tpl'} {include file="search/searchbrowse.tpl"} {/if} {if $module == "myresearch"} {include file="myresearch/myresearchbar.tpl"} {/if} {if $module == "author"} {include file="author/searchauthor.tpl"} {/if} vufind as a wrapper for summon advanced patrons require more search options than our first-time users.  those that had been using our summon interface had been extremely happy.  in our design, we wanted to take the good functionality of summon (eg facets, limiters) and integrate it into vufind. this can easily be done in the vufind configuration file.  the difficulty we ran into was that we wanted to keep our local materials searchable in summon so we could provide a unified search, however information about the availability of material from millennium would not display correctly.  fixing this required that we copy the availability code over from the result.tpl file, located in record drivers/index, and place it in the summon/list-list.tpl.  however, it is not enough to just copy over the code, as summon adds additional information in the url for local material.  this required us to strip out the additional information so that we could grab the record id: <span class="ajax_availability hide" id="callnumber{$record.id.0|substr:18:8|escape}">{translate text='loading'}...</span><span class="ajax_availability hide" id="location{$record.id.0|substr:18:8|escape}">{translate text='loading'}...</span><span class="ajax_availability hide" id="status{$record.id.0|substr:18:8|escape}">{translate text='loading'}...</span> we also needed to stop summon from pulling availability information on electronic materials out of millennium, which wouldn’t display correctly. availability information should be displayed for hard copy materials only, since electronic resources are always available. millennium can only display availability information correctly for material that has an item record. because most of our electronic material has only a bibliographic record and not an item record in millennium, we were able to use an ‘if’ statement based on the existence of an item record to either display availability information (for hard copy material) or a link to full text (in the case of electronic material). title</pre> <div> <div>{* <a href="{$url}/record/{$record.id.0|substr:18:8|escape:"> {if $record.id.0|substr:6:13 == "hct_catalog_b" } </a><a href="{$url}/record/{$record.id.0|substr:18:8|escape:"> {else} </a><a class="title" href="{$url}/summon/record?id={$record.id.0|escape:">{if !$record.title.0}{translate text='title not available'}{else}{$record.title.0|highlight}{/if}</a></div> </div> <h4>availability</h4> <pre>{if $record.id.0|substr:6:13 == "hct_catalog_b" && $record.contenttype.0|getsummonformatclass|escape != "electronic" && $record.contenttype.0|getsummonformatclass|escape != "government document"} {else} {if $record.link} <span><a class="fulltext" href="{$record.link|escape}">{translate text='get full text'}</a></span> {elseif $record.uri && (!$openurlbase || !$record.hasfulltext)} <a class="fulltext" href="{$record.uri.0|escape}">{translate text='get full text'}</a> {elseif $openurlbase} <span>{include file="search/openurlresults.tpl" openurl=$record.openurl}</span> {/if} once these additional bits of codes were added, the vufind summon interface was able to correctly display status information for hct material that had item records and move everything else to a status of get full text. usability testing method usability testing of vufind was conducted at three campus libraries across the united arab emirates. each campus tested the vufind search interface with five undergraduate students and two faculty members for a total of 21 participants. the usability testing was conducted by three librarians who formed part of a four person usability committee set up to test and investigate new library technologies. one set of tests was completed with male students at a men’s campus and the other two sets of testing were done with female students. faculty members were also tested to see if they faced similar issues as students in using vufind. nielsen (2000) argues that five participants in a usability study is an optimal number; beyond five, the rate of new user issue discovery drops significantly and most data is redundant. therefore the most useful information and highest return on research investment comes in the first five users. usability testing of search interfaces conducted at other higher education institutions libraries have tested between five to fifteen users. (bauer, 2008; denton & coysh, 2011; emanuel, 2011) in this case, the total of 21 users allowed us to include people from a range of distinct groups of male, female, faculty, and student users. the participants were given eight tasks to complete such as finding books, journal articles and narrowing searches. (see appendix a) the participants were also asked a series of questions after they had completed the tasks on the appearance and usability of the search interface to help determine what features were easy or difficult to use. (see appendix b) participants were recorded navigating through vufind using the screen capture software camtasia. recording allowed us to record the time required for each task, review the same performance multiple times and to adjust specific areas (for example, originally the tabs were on the right, then they were changed to the left after watching the recordings. this study used the “think aloud protocol”. the think aloud protocol is a testing method where the user is presented with vufind and given a set of tasks to perform. the user speaks his/her thought processes. the benefits of using think aloud protocol include being able to redesign elements of your user interface based on users’ misconceptions, or things they did wrong as well as discovering which parts of the user interface users found easy to use. (nielson, 2012) findings of the usability testing our inquiry resulted in several changes and enhancements to the original draft interface. users found the interface intuitive and were able to perform most of the test tasks independently without advice from the accompanying investigator. overall, users said they preferred even the draft interface to the previous version of the catalog.  it was also observed that users progressed rapidly in their proficiency from the first task to the eighth, taking less time to navigate the interface and narrow searches after each subsequent task.  some of the intuitiveness might be attributed to the carryover of verbiage from the prior catalog.  where possible, the draft interface employed similar language as the catalog – for instance, using “request” vs. “hold”, “cite” vs. “reference”, etc. figure 1 : previous version of summon search interface used before implementation of vufind figure 2: previous version of library catalogue used before implementation of vufind social features social features commonly found in library search systems include functions such as the ability to tag content, comment on content, share content via twitter, facebook or other social media sites, and make use of social bookmarking. we found a disparity between students and faculty users that we tested in terms of personalization and social features.  similar to what ho, kelley, & garrison (2009) found, neither our student nor faculty users made use of these features, although the study tasks didn’t ask them to. the students we tested generally ignored the presence of these features. faculty in our tests, in contrast, expressed a dislike for the way these features are presented in catalog records – deeming them unimportant and secondary to the primary goals of a library search system.  faculty users were also unsure of where to find what they considered more critical information, such as the location and call number of a book. this information was further down the page and required scrolling to view. multiple faculty members suggested removing personalization and social features from item record pages altogether, or moving them down the page or into other, less obvious places in favor of prioritizing “more important” information. this recommendation was taken and the final version featured a less cluttered item record page with personalization and social features tucked into labeled menus on the left side of the page [figures 3 and 4].  these changes also resulted in other information about the items moving up from the bottom of the page, becoming partially visible without scrolling. figure 3: draft interface shows social, sharing and personalization features such as an item qr code, comments, tags and favorites, located prominently in the center of the page. similar items are displayed on the left. item location information is not visible anywhere on the screen and the user must scroll down to see it. figure 4: revised interface shows certain features categorized and moved to a less prominent location on the left and bottom of the page, including the qr code, comments, and favorites. similar items are now hidden behind a clickable button, rather than displayed by default. users are invited to scroll down the page by visible tabs indicating that more information is available. linguistic orientation because the draft vufind layout separated print and electronic resources into two vertical columns [figure 5], one of our key questions prior to the study was whether native arabic-language users would begin their interaction with the page from left or the right.  if our students were to begin looking for important features on the right side of the page, this would have design implications for us. the arabic language is written and read right to left and many arabic websites place their menus and navigation on the right side.  (for example, the abu dhabi government website ) somewhat surprisingly, we found that these users did tend to interact with the search interface and navigation menus from left-to-right; perhaps because they are accustomed to doing so when content is presented in the english language.  most users intuitively looked for books and significant headers on the left (as indicated by mouse movement), and when asked explicitly, agreed that the layout was logical. figure 5: draft interface initial search results through vufind are separated into print results (left) and electronic results (right). accessing full text our test subjects did not intuitively understand how to access the full-text of e-books and articles. users frequently attempted to access the full-text of a retrieved item by clicking its hyperlinked title; which retrieves a bibliographic record for the item but does not immediately present the full-text [figure 6]. figure 6: draft interface users tended to click the hyperlinked title of an item when searching for full-text, overlooking the “get full text” link just below. the committee considered linking titles to full-text, but decided against it. although many users do not care to examine an item’s bibliographic record, there are many others that do. the bibliographic record is the key route for more sophisticated researchers to select a database source of their choice when the item is available in multiple databases. we also recognize that many search engines and electronic interfaces (eg databases such proquest and ebsco) link the title to the bibliographic records, and that our user group may reap the benefit of greater transferability if they learn to look for a ‘full-text’ button or option in our search interface. therefore, we have addressed this usability issue with a combination of some minor changes to the interface [figures 7, 8, 9], along with a user education initiative which focuses on accessing full-text. figure 7: revised interface the item type and full-text link has been emphasized with space and content divider bars figure 8: draft interface after a user has clicked on an item’s title seeking full-text, the bibliographic record provides no clear avenue to continue. most users did not recognize that the holdings link at the top would retrieve full-text, and instead pressed the back button. figure 9: revised interface the “get full text” link has been added to the item’s bibliographic record as well, to enable users who do come here by mistake to proceed. new and multiple searches another finding was that users in our test who performed multiple searches would re-search from the same page where they finished their prior search, using a search box at the top. this resulted in new search listing results in the same format as the last page, rather than the original results format of books on the left and electronic resources on the right. given the positive user feedback about this division, we reset the search box to always default to this format on a new search, regardless of where the search was performed from. emotionally intelligent wording additionally, wording was changed to be more emotionally intelligent. we tried to anticipate the emotional reactions of our users and promote more positive reactions. the users we tested, particularly faculty, expressed feelings of annoyance and anger through the think-aloud protocol when they encountered wording on an initial results list about the location of book copies.  in order to save space, the first draft interface included a results page which listed the physical location of a hard copy in the format location : [campus – collection]. however, if multiple copies of the item existed in multiple campuses, the results would read location : multiple locations [figure 10].  users could see a list of locations by accessing the item record after clicking its title. users, however, felt that the multiple locations indicator was useless, redundant, and unhelpful, resulting in negative emotional reactions. users did not understand where they could see a list of locations for the item. this wording was changed to an imperative construction providing a better indication of how users could get more information [figure 11]. figure 10: draft interface users reacted negatively to the “multiple locations” indicator, exhibiting frustration. figure 11: revised interface the “check record for locations” indicator provides a useful instruction for users about where to get the information they are seeking. all fields search user search behavior tended towards the default all fields search regardless of what they were looking for; be it author, title, topic, or other. author searches tended to come in first name, last name order rather than the reverse. this information was used in customizing data input and search parameters to make results more relevant. conclusion our users are largely consistent in their behavior with other groups in prior studies. (denton & coysh, 2011; ho, kelley, & garrison, 2009) based on the results of this study, several changes were made to the vufind interface in order to facilitate a more intuitive search experience for our users, including the rearrangement of features on item record pages, the nature of wording and the size and placement of text and images. based on these results, we also made some decisions about what usability issues should be treated through interface adaptations, and which should be treated through user education. further study into whether user behavior changes along with the language of use (right-to-left or left-to-right) and whether native language orientation has any impact, would be helpful to confirm and support our initial observations in this study. future directions the revised vufind interface was launched in september 2012 incorporating the findings and observations from the usability study.  feedback from faculty and students on the new interface was positive. the usability committee plans to keep adding features to the vufind interface and another usability test or user satisfaction survey will be conducted in order to ensure that the search interface continues to be user friendly. references bauer, k. (2008). yale university library vufind test—undergraduates.   retrieved 28 aug, 2012, from https://collaborate.library.yale.edu/…/summary_undergraduate.doc denton, w., & coysh, s. j. (2011). usability testing of vufind at an academic library. [doi: 10.1108/07378831111138189]. library hi tech, 29(2), 301-319. emanuel, j. (2011). usability of the vufind next-generation online catalog. information technology and libraries, 30(1), 44-52. ho, b., kelley, k., & garrison, s. (2009). implementing vufind as an alternative to voyager’s webvoyage interface: one library’s experience. library hi tech, 27(1), 82-92. nielsen, j. (2000). why you only need to test with 5 users.   retrieved 28 aug, 2012, from http://www.useit.com/alertbox/20000319.html appendix ausability study task list usability study task list determine if we have the book “twilight” at any college. is it available at this college? find books by the author j. k. rowling. how many copies of “harry potter and the sorcerer’s stone” are there? can you find similar items to this book? what are they? cite “harry potter and the sorcerer’s stone” in apa. find the following book: introducing public relations : theory and practice who published this book? was that information easy/difficult to find? find a book about drawing comics or manga characters at any college, and make a request for it. find and open an article (newspaper/magazine/journal) from 2005 or 2006 about diabetes in the uae. find and open an e-book which will help you learn about adobe photoshop. find information about ‘human resource management’. find a newspaper article on the topic. open the full text. find a scholarly journal article about emiratization. open the full text of the article. appendix b – task debrief [to be discussed verbally with, and recorded by, interviewer]. overall, what do you like most about this search interface? what do you not like? how did the search interface help/not help with your searches? what do you think of the arrangement of what is on the left and what is on the right side of the screen? how would you change it, if you could? about the authors nicole johnston is a faculty librarian at the higher colleges of technology. she has previously worked as a librarian and english teacher in australia, ireland and japan. she has a bachelor of arts degree, masters of library and information science degreeand is currently a phd candidate at qut in brisbane, australia. nicole.johnston@hct.ac.ae alicia salaz is a faculty librarian at the higher colleges of technology and doctor of higher education student at the university of liverpool. she earned her masters of library and information science from the university of washington in seattle in 2006. alicia.salaz@hct.ac.ae rob o’connell is the manager of library technical services at higher colleges of technology with eight years of library experience. rob specializes in managing integrated library systems and has overseen the millennium library system at the higher colleges of technology and liwa library consortium for four years. rob has a bachelors degree in graphic design and received his masters in library science from clarion university in pennsylvania. roconnell@hct.ac.ae subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – click tracking with google tag manager for the primo discovery service mission editorial committee process and structure code4lib issue 55, 2023-1-20 click tracking with google tag manager for the primo discovery service this article introduces practices at the library of oregon state university aiming to track the usage of unpaywall links with google tag manager for the primo discovery interface. unpaywall is an open database of links to full-text scholarly articles from open access sources[1]. the university library adds unpaywall links to primo that will provide free and legal full-text access to journal articles to the patrons to promote more usage of open-access content. however, the usage of the unpaywall links is unavailable because primo does not track the customized unpaywall links. this article will detail how to set up google tag manager for tracking the usage of unpaywall links and creating reports in google analytics. it provides step-by-step instructions, screenshots, and code snippets so the readers can customize the solution for their integrated library systems. by hui zhang introduction in 2020, staff at oregon state university library started a project to provide single-click links to open access content in 1search[2], the university’s library discovery interface built on the primo service platform[3]. the goal of the project is to provide free and legitimate access to full-text scholarly resources for the patrons. although primo already includes open access content in the search results, there are studies [1] that show the primo solution has significant flaws in indexing and providing open access resources. ultimately, we decide to use unpaywall, an open database that harvests and indexes tens of millions of open access scholarly articles, as the source of open access content in addition to primo. by customizing the user interface (ui) of primo, we added unpaywall links of open access content to the search result and the individual item view. however, it has a problem in that we cannot get the link usage statistics from primo’s analytics tool because primo does not track the unpaywall links. this article will detail tracking customized links in primo using google tag manager [4] including testing, troubleshooting, and creating usage reports with google analytics. although the case study is specific to tracking unpaywall links, the workflow and configuration of google tag manager are general. users may also adapt the included snippets and tags to tracking activities for websites beyond the types used in library systems. adding open access links with primo customization finding open access resources in primo primo users can find open access content in two ways. the first way is to filter the search results by selecting “open access” in the availability facet. figure 1. open access facet in primo. the second way of finding open access content is to look for the open access icon that will appear for an item identified as open access both in the search results and in the full item view. figure 2. open access indicator in primo. primo provides open access content in many resource types, such as journal articles, books, and theses. why adding unpaywall links to primo the significant flaw in primo’s solution of providing open access is the preference for subscribed content over open access. one study [1] finds search results in primo will provide article links to subscribed journals even when the articles are open access, undermining the visibility of open access content to the readers. with the growing demand to make more open access content available to the patrons, developers at primo added a feature to integrate the unpaywall api so users can find open access articles that may not appear or be available to them initially [2]. however, librarians at oregon state university (osu) finally decided to provide unpaywall links to primo by adopting a solution called oadoi link [3] developed by the primo customization standing group. the standing group is formed by the orbis cascade alliance, which oregon state university is a member of. one advantage of oadoi link is that the osu library has better access to technical support as it is locally developed. but more importantly, a recent study [4] found that oadoi links can provide an estimated 30% more open access articles compared to primo’s unpaywall feature. we extend the oadoi links in our solution [5] to provide unpaywall links to open access items in the brief display view next to the availability status. figure 3. customized unpaywall link shown in the brief display of an open access item. offering unpaywall as single-click links is a significant improvement to usability as library patrons can access the full-text contents without authentication with their university credentials. the challenge of tracking unpaywall links usage in primo because the unpaywall links are added to primo by a ui customization, we cannot get usage statistics of these links from primo because they are not tracked. it is a major problem as usage data is crucial evidence to assess the impact and success of the unpaywall project. to overcome the problem, we investigated the possibility of using google tag manager to track the customized unpaywall links. as primo continues to provide new features including link tracking, it is worth updating the latest situation so that the readers will have a better understanding of the motivation and contribution of our approach. ex libris, the company that develops primo, added the capacity to track and report the usage of unpayall links in august 2021 [6]. however, that feature is only available to primo ve [7], a newer and different cloud computing platform to primo. the ex libris solution was unavailable when we investigated the potential of google tag manager in 2020, and we then used primo, not primo ve, as the discovery interface. for full disclosure, oregon state university library migrated its discovery interface to primo ve in the summer of 2022. however, many libraries worldwide are still using primo, and our work on google tag manager will help them to track customized links like unpaywall in their discovery interfaces. tracking unpaywall links with google tag manager how google tag manager works perhaps many readers, like us, are confused when they try to understand what google tag manager is at the beginning. we will explain how google tag manager works by answering two questions: what is a tag and what is the difference between google tag manager and google analytics? according to google, a tag is a code snippet deployed to measure website user activity [8]. these tags, or tracking codes, were usually created and deployed by developers before tools like google tag manager were available. however, with google tag manager, people can create, test, and deploy a tag without programming skills. creating triggers that will tell the manager when, where, and how to operate the tag is required to set up a tag. google tag manager provides two types of triggers: all elements and just links. the all elements trigger can track clicks on any element on a page, e.g. links, images, and buttons. the just links trigger can track clicks on html links that use the element. google tag manager and google analytics are two different tools, but you should always use them together. google tag manager can add google analytics tracking code (i.e., tag) to the website but can not create reports. instead, it will send activity data of the website to google analytics for analyzing and reporting. adding google tag manager to primo your first step to adding google tag manager to primo will be creating an account. go to the google tag manager website to create an account or log in using your google account. then you need to create and name a container with “web” as the target platform, and you will be given an id in the format of “gtm-xxxxxxx” after the container has been created. take note of the container identifier because you will need it in the next step. we suggest creating a container for every website, then defining tags for activities you want to track in the newly created container. you will add a snippet to primo to allow google tag manager to track the web activity. the technical detail of managing and customizing the primo ui package is beyond the scope of this article. however, the primo administrators should have the knowledge and privilege to add the sample javascript snippet below to primo. make sure you use the correct container id in the snippet. /* google tag manager */ const gtmid = 'gtm-xxxxxxx' function addgtm(doc) { const newscript = doc.createelement('script') const scripttext = `(function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start': new date().gettime(),event:'gtm.js'});var f=d.getelementsbytagname(s)[0], j=d.createelement(s),dl=l!='datalayer'?'&amp;l='+l:'';j.async=true;j.src= '//www.googletagmanager.com/gtm.js?id='+i+dl;f.parentnode.insertbefore(j,f); })(window,document,'script','datalayer','${gtmid}');` newscript.innerhtml = scripttext doc.head.append(newscript) const noscript = doc.createelement('noscript') const noscripttext = `&lt;iframe src="//www.googletagmanager.com/ns.html?id=${gtmid}" height="0" width="0" style="display:none;visibility:hidden"&gt;&lt;/iframe&gt;` noscript.innerhtml = noscripttext doc.body.insertbefore(noscript, doc.body.firstchild) } addgtm(document) then you need to save and deploy the change so it will take effect in primo. congratulations! you have done all the required configuration for primo, and all the rest will happen in google tag manager. creating tag and trigger for unpaywall link clicks you will create a google tag manager tag and trigger in the newly created container. for example, you can create and name a tag called “unpaywall” with the type of google analytics and associate the tag with an existing google analytics account. in our case, we associate the tag with the google analytics account of primo. figure 4. google tag manager tag with the type of google analytics inside the new tag, you need to create a trigger that fires the tag when the unpaywall link is clicked. figure 5. google tag manager trigger for tracking unpaywall link clicks for our purpose of tracking the link click events, make sure you will select “all elements” for the trigger type. the next step is to attach a condition or rule to the created trigger. in our example, we create a condition like this: the tag is activated when links with the text “open access(via unpaywall)” are clicked. the link text used in the trigger is the label of the unpaywall link we added to primo ui by our customization code. if you want to customize the method, you can define a trigger with different conditions that are appropriate for your need. trigger testing and troubleshooting people can test newly created tags and triggers using the preview feature in google tag manager. figure 6. google tag manager preview for testing tags and triggers. the preview feature is available at the container level. a pop-up window appears after clicking the “preview” button, where you can enter the url of the primo instance for testing. after the connection is established, you can go to the primo website, click the unpaywall link, and check whether the tag is triggered as expected in google tag manager. the browser’s developer console will be your best tool for troubleshooting. for example, we use chrome’s console to confirm the label text for the unpaywall link is the same as entered in the trigger condition. after finalizing the configuration with the preview, you must verify the changes by clicking the “submit” button next to the right of the “preview.” you will be asked to create a version of your container and finally publish it. the deployment is reasonably quick, and in our example, we can see click data shown in google analytics a few minutes after we published the change in google tag manager. generating usage report with google analytics you can use the many report functions in google analytics to review and analyze the data collected by google tag manager. in our example, we can find statistics of unpaywall link usage in the event under behavior, then in the “click” event category. figure 7. click event report in google analytics. the result is promising as it shows patrons are attracted to open access content, and there is a clear trend that more patrons are using the unpaywall links. for instance, the total number of clicks for the unpaywall link is 53,361 during the calendar year of 2021. that number jumped to 60,534 for the first six months of 2022 until osu migrated its discovery interface to primo ve. conclusion in this article, we describe the work of tracking customized unpaywall links with google tag manager in primo. we outline the motivation of our project, introduce google tag manager, and provide details on how to define tags for tracking unpaywall link clicks. we have used data collected by google tag manager for decision-making. for example, we have decided to continue to provide open access and unpaywall links in primo ve with the usage statistics of unpaywall links collected in primo. by integrating google tag manager and google analytics, we can also get more insights into patrons’ activities, such as which open access databases are popular and which subjects patrons are most interested in. we hope the code and screenshots are helpful and the readers can refer to them in their works. notes [1] unpaywall: https://unpaywall.org/ [2] 1search: https://search.library.oregonstate.edu/ [3] primo service platform: https://exlibrisgroup.com/products/primo-discovery-service/ [4] google tag manager: https://tagmanager.google.com/ references [1] bulock, c. (2021). finding open content in the library is surprisingly hard, serials review, 47:2, 68-70, doi: 10.1080/00987913.2021.1936416 [2] how to utilize the unpaywall api for open access content and resources in discovery. (2021). ex libris knowledge center. https://knowledge.exlibrisgroup.com/alma/knowledge_articles/how_to_utilize_the_unpaywall_api_for_open_access_content_and_resources_in_discovery [3] oadoi link. (2022). orbis cascade alliance. https://www.orbiscascade.org/programs/systems/pcsg/primo-ve-toolkit/oadoi-link/ [4] veldhuisen, k. (2020). unpaywall in alma, oadoi customization in primo (and other open access). retrieved from https://docs.google.com/document/d/1rbz7l4_ktra7psxfxatpyjev-og3sizw1qjrpxc5ebk/edit [5] osulp/1search-ui-package. (2021). github. https://github.com/osulp/1search-ui-package [6] primo ve 2021 release notes. (2022, may 6). ex libris knowledge center. https://knowledge.exlibrisgroup.com/primo/release_notes/002primo_ve/2021/010primo_ve_2021_release_notes [7] primo ve overview. (2022, september 18). ex libris knowledge center. https://knowledge.exlibrisgroup.com/primo/product_documentation/020primo_ve/primo_ve_(english)/010getting_started_with_primo_ve/005primo_ve_overview [8] overview. (2022). google developers. https://developers.google.com/tag-platform/devguides about the author hui zhang (hui.zhang@oregonstate.edu) is the digital services librarian at the oregon state university. subscribe to comments: for this article | for all articles 5 responses to "click tracking with google tag manager for the primo discovery service" please leave a response below: jonathan rochkind, 2023-01-23 nice article! it’s great to see the implementation — but i’d also be super interested in hearing about what you learned from the tracking, what the numbers and your interpretation of them were! perhaps a future article? hui zhang, 2023-01-25 hi, jonathan: thanks for the comment. the tracking numbers are reported at the bottom of the article (admitted that it is provided like a footnote): for instance, the total number of clicks for the unpaywall link is 53,361 during the calendar year of 2021. that number jumped to 60,534 for the first six months of 2022 until osu migrated its discovery interface to primo ve. we also get some feedback from the patrons that they like the unpaywall link because it is easy and fast to use. author, 2023-01-25 thanks for the comment, jonathan. the tracking numbers are at the bottom of the “generating usage report with google analytics.” osu migrated to primo ve in the summer of 2022, and ex libris now provide a very similar implementation to oadoi in ve. it will be interesting to run a study on that implementation. anna couthures-idrizi, 2023-02-01 dear hui, thank so much for this article ! i would like to know if the script you propose can also be integrated into a primove custom.js file best regards, anna scott k, 2023-05-02 thanks for sharing this–i added it as-is (with our own container id) and it worked to get ga4 collecting data in primo ve for me. i noticed that unless i put it at the very bottom of my customization package’s js file, none of the scripts that came after it would run. i’m not good enough with javascript to understand why, but i wondered if the author (or anyone else) might be able to tell me why that happened. leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – response to premis events through an event-sourced lens mission editorial committee process and structure code4lib issue 59, 2024-10-07 response to premis events through an event-sourced lens the premis editorial committee (ec) read ross spencer’s recent article “premis events through an event-sourced lens” with interest. the article was a useful primer to the idea of event sourcing and in particular was an interesting introduction to a conversation about whether and how such a model could be applied to digital preservation systems. however, the article makes a number of specific assertions and suggestions about premis, with which we on the premis ec disagree. we believe these are founded on an incorrect or incomplete understanding of what premis actually is, and as significantly, what it is not. the aim of this article is to address those specific points. by jack o’sullivan, sarah romkey, and karin bredenberg introduction the premis editorial committee (ec) read ross spencer’s recent article “premis events through an event-sourced lens” with interest.[1] the article was a useful primer to the idea of event sourcing and in particular was an interesting introduction to a conversation about whether and how such a model could be applied to digital preservation systems. however, the article makes a number of specific assertions and suggestions about premis, with which we on the premis ec disagree. we believe these are founded on an incorrect or incomplete understanding of what premis actually is, and as significantly, what it is not. the aim of this article is to address those specific points. a brief introduction to premis data model premis (preservation metadata: implementation strategies) is a data model describing “the information a repository uses to support the digital preservation process”.[2] the data model is comprised of four top-level entities: objects, the subjects of preservation; events, the actions performed on the objects in the course of preservation; rights, the bases under which a repository is permitted to perform preservation actions; and agents, the people, organizations and/or software who perform preservation actions or confer rights to the repository. each entity has a set of “semantic units” (properties) associated with it. some of these semantic units are mandatory whilst some are optional. some are intended to be single value property fields, whilst others have further substructure. some can be repeated multiple times for the same entity whilst others are only expected to occur once. some semantic units are expected to be populated with values from a controlled vocabulary. the data model itself is defined in the data dictionary, which is the primary artifact maintained by the premis ec. premis is technology and implementation agnostic. it defines the set of information that a repository is likely to need to know to support preservation activities. it does not mandate how that information should be stored, or how systems using premis should be architectured. whilst the data model is technology independent, the premis ec do maintain two additional resources to support implementation, an xml schema and an rdf owl ontology, both of which are technology specific encodings of the data model. similarly, for semantic units where the value is expected to be populated with values from a controlled vocabulary, this is a guideline, expressed as a “should” rather than a “must” requirement. the premis ec maintains a list of controlled vocabularies that are applicable to these semantic units, which have been developed through community collaboration over an extended period of time, and which we believe to be broadly applicable. individual repositories may have other constraints and policies regarding these values, or different scopes of what they determine or require “preservation” to mean, and so the actual vocabularies used are an implementation specific decision. conformance since premis does not mandate any specific implementation details, it provides a means for implementers to assert the conformance of their specific implementation. this conformance statement provides three levels that implementers can assert. for simplicity, we are omitting the a and b sub-clauses at each level, which cover the scope of the premis data that is conformant, rather than the manner in which it is conformant. level 1 (mapping) – implementers assert that the data model in use by their system can be mapped to the premis data model, including assertions that all mandatory semantic units can be supplied or derived. level 2 (export) – implementers assert that their repository provides functionality or processes to export internally held metadata as valid premis metadata. level 3 (internal implementation) – implementers assert that their repository directly implements the premis data model as the internal data model of the system. even at level 3, this statement is agnostic as to the technology, details and strategy for implementation. what premis is not as stated above, premis is deliberately intended to be agnostic of any particular implementation or technology. as a consequence, it is not the xml schema, the rdf owl ontology, or any of the controlled vocabularies, all of which the ec maintain as references and supporting materials for implementers. specific suggested premis changes given what we have said about premis, we can now look in some detail at some of the specific suggestions and assertions made about premis in the original paper. nouns become actions in the past tense the assertion that premis events are often “nounified” seems to us to imply one (or both) of two things. the first is that the event types in the premis maintained controlled vocabularies are nouns. the second is that systems that implement premis tend to use nouns as event labels. the first of these is undoubtedly true. this can be understood as the controlled vocabulary describing the types of preservation actions that preservation systems are likely to enact, rather than describing the performance of the act itself. the second of these is also, in our experience, true. we believe that this can be understood as a symptom of the fact that, as the author points out, preservation systems today do not tend to be architectured as event sourced systems, but instead as records of how things are, and how they came to be. that being said, we don’t believe that it follows that the author’s assertion that “titular premis nouns become a barrier to engage with, use and extend without committee” is also true. the event type controlled vocabulary is intended to cover as broad a range of use cases as possible, but is not intended to be prescriptive or proscriptive. implementers are free to use their own vocabularies where they wish or need. the premis data dictionary strongly suggests only that these vocabularies should be controlled so as to facilitate exchange of information and description. less importance placed on the data dictionary as discussed above, the data dictionary is the primary artifact of premis. it is the definition of the data model, so it’s hard to see how less importance could be placed on it. given what the author actually says under this heading, our reading is that the author actually means that less importance be placed on the set event types covered by the controlled vocabulary. as discussed, this is already a guideline for implementers, who can choose the degree of importance they attach to adhering to the list. closer to a technical implementation it is not clear to us that the author is actually suggesting any change here. we would like to reiterate however that premis is explicitly and very deliberately implementation agnostic. premis conformance restrictions around internal schemas are made more flexible the author is really covering three distinct issues here, firstly around the potentially hierarchical nature of the levels of conformance, secondly as to whether the distinctions between the levels are still valid, and finally whether level 3 is “reasonable” today. are the levels hierarchical? the author asks, “while described as levels, should they be viewed hierarchically”, without seeming to posit a reply. whilst it is not explicit what is meant by this, we can at least offer the beginnings of an answer. logically, the three levels do indeed build hierarchically. level 1 states that the metadata that the repository stores or knows can be mapped to premis. that is, for each premis entity and semantic unit, the repository stores or knows an equivalent metadata value. level 2 takes this a step further, stating that not only does the repository store or know that metadata, but that, on request, it can provide a user with that metadata using premis terminology directly. level 3 then further states that the repository actually stores that metadata using premis terminology directly. if you can assert level 2 conformance, then by definition you must also meet the criteria for level 1. similarly, if you can assert level 3. it is theoretically possible that you could assert level 3 by directly implementing premis, but not be able to assert level 2 because all export mechanisms transformed the metadata away from premis (or there are no export mechanisms), but it seems so unlikely that any implementation would actually do this that for all practical purposes we can think of the levels as a hierarchy. this is not to say that there is an implied hierarchy of merit in approach. it is not inherently “better” to have level 3 conformance than level 2, or to have level 2 rather than level 1. the levels simply represent different choices made by each implementer. are all three levels still required? the author asserts that: the first level of conformance clearly makes interpretation of an internal schema next to premis more difficult, and vice versa. one may remove this level of conformance completely. conceptually, however, two and three, have the same outcome, functionally, they take different paths toward getting there. can one approach realistically be viewed as better than the other? there are several threads to pick at in this statement. the first is contained within the first sentence. it is not “clear” to the ec that providing an option to allow implementers to map their internal schemas to a common standard makes anything more difficult. in practice, this means that implementers may have to include elements in their data models that they would not have included in the absence of premis or a requirement or desire to attain conformance with premis. it does not constrain their freedom to include other elements that are beyond the scope of premis, or indeed to choose how to interpret the premis model. there follows an assertion that level 1 could be removed completely, and that levels 2 and 3 “have the same outcome”. like everything in premis, the levels of conformance are subject to periodic review based on user and implementer feedback. that said, the ec believes that the current levels and definitions are each useful and sufficiently distinct from each other that they are not in need of such drastic overhaul as suggested here. for any system implementing with level 1 or 2 conformance, the implication is that there is not a single unique 1:1 correspondence between the internal data model and premis. in most cases that will mean that there is some ambiguity in exactly how the internal data model should be mapped, with multiple valid choices that could be made. conformance level 2 imposes functional requirements that level 1 does not. it requires that a canonical mapping between the system’s own internal model and premis be established; and that this canonical mapping be formalized through explicit api contracts or functionality. for providers of systems that are used by multiple institutions, level 2 could be seen as restricting the freedom of those individual institutions to choose a mapping that best suits their needs. further, the additional implication of the second part is that the repository has to commit to maintaining functionality. we believe therefore that there are, and will always be repositories for whom level 1 is a valid end state to aim to achieve, and that there will be others willing to, or being required to, take the additional steps required to reach level 2. to believe that level 2 and level 3 have the same outcome, is to consider export of premis metadata as the only, or the main function of a digital preservation system. for level 2, an implementer can choose to have data in the system that does not map to premis at all, or to have a working model that differs substantially in form from premis in order to facilitate other functionality, accepting that exporting to premis is a transformation performed for a specific use case. this is a fundamentally different proposition from a system that directly implements premis. level 3 essentially requires that an implementer is constrained by decisions made by an external body, i.e. the premis ec, at least in regards to preservation metadata. this may be a drawback for level 3 implementation, but there are also benefits in choosing to take an internationally defined and agreed data model and use that as the basis of your system. the benefits of not implementing an external data model are broadly around increased control and flexibility, however the trade-off to consider is the likely loss of easy interoperability and exchange with other systems. the benefits and considerations of conforming to any level are outlined above; ultimately the systems designer can take all benefits and drawbacks into consideration and still be in alignment with premis. premis can only provide this flexibility by continuing to define the current levels of conformance. is level three reasonable in today’s software development world? again here, the author posits a question, which actually raises multiple issues. initially they state that “premis can be trivially demonstrated to record information that extends beyond that which is useful for preservation”, citing access as one of those things. there are two responses to this, the first is to note that access has always been considered a part of digital preservation, to the point that one of the functional areas of the oais model is access. the second is to reiterate the point made above, that premis as a model specifies that you should know information about the objects being preserved, and the actions performed, without explicitly mandating which actions a repository should concern itself with. the second issue is whether premis is the most efficient format to be used as a model for digital preservation metadata. this is not really expanded on within this section, but does seem to be raised again within the conclusion. here the author notes that there have been efforts to reduce the size of premis metadata. again, what we believe is at issue is not premis as a data model per se, but specific technological implementations. premis in xml format can indeed be verbose; however this is not unique to premis, one of the more common complaints of xml is its verbosity. we can reduce the footprint of premis metadata in xml by using compression tools, such as zip, gzip, exi encoding etc. obviously this comes at the additional preservation risk of having an extra layer of technology in our solution, and at the expense of some cpu overhead for each read and write operation. alternatively, a different technological encoding altogether of the model could be used. the author notes that the rdf owl ontology is an alternative that might reduce size, but equally the premis metadata could be expressed in json or yaml, or one of any number of data serialization schemes, which in their uncompressed forms would likely be smaller than xml. indeed, the premis metadata could be directly encoded in a database, which for a large collection could also reduce size. the core point is that the premis data model specifies the metadata that most repositories are likely to need to know in order to perform preservation; how efficiently this is stored is an issue for each implementation. conclusion the first version of the premis data dictionary was published nearly 20 years ago, and in that time the practices of digital preservation have benefited from numerous advances in technology, while simultaneously being challenged by technical and societal factors. the aim of the premis data dictionary continues to be to provide a technology and implementation agnostic way for digital preservationists to capture the necessary metadata for continued preservation and access to digital objects. we’d like to thank ross spencer for contributing his ideas to the community and hope that this response provides clarification or illumination for anyone interested in implementing premis in their own systems or practices. we would also welcome the opportunity to continue a dialog around how an event sourced system could express the data it preserves in premis form. references [1] ross spencer, “premis events through an event-sourced lens,” code4lib journal, no. 56 (april 2023). https://journal.code4lib.org/articles/17264. [2] premis editorial committee. data dictionary for preservation metadata: premis version 3.0 (june 2015, revised november 2015). https://www.loc.gov/standards/premis/v3/premis-3-0-final.pdf. about the authors jack o’sullivan, sarah romkey, and karin bredenberg (chair) are members of the premis editorial committee. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – auto-populating an ill form with the serial solutions link resolver api mission editorial committee process and structure code4lib issue 4, 2008-09-22 auto-populating an ill form with the serial solutions link resolver api in this article we’ll take a tour of the openurl protocol; discover how to use it to get an xml api response from the serial solutions link resolver; and see how to receive and process that xml data using php to create an interlibrary loan webform. finally, we’ll see a few examples of how to handle form processing. this article will be of interest to beginner programmers interested in examples of programming with openurl and xml in php, and to more experienced programmers interested in taking a look at the serial solutions 360 link api. by daniel talsky introduction in this article we’ll take a tour of the openurl protocol; discover how to use it to get an xml api response from the serial solutions link resolver; and see how to receive and process that xml data using php to create an interlibrary loan webform. finally, we’ll see a few examples of how to handle form processing. this article will be of interest to beginner programmers interested in examples of programming with openurl and xml in php, and to more experienced programmers interested in taking a look at the serial solutions 360 link api. what is an openurl link resolver? an openurl is a formatted citation embedded in a url.[1] link resolver products receive openurl requests identifying a citation, and provide services to the user for the given citation. the service which generally motivates an institution to set up a link resolver is providing online full text links for a citation, generally based on a ‘knowledge base’ of licensed full text maintained in the link resolver product. most openurl link resolvers return results as an html page. this is great for a human, who can read the results and manually link to the full text. but sometimes a library might want to set up more advanced features than the link resolver product supports, still based on underlying link resolver functionality. even significant customization of the html output may be limited by the interface of the link resolver. you may want to use or develop other tools which provide more functionality, while still using the full text resolution services of the link resolver. an xml api allows you to receive results as a package of data, and use a program to pull the data you want and display or use it in any format. want to populate a spreadsheet with link resolver data? a pdf file? generate an rss feed? put the information in an email? all of these are possible using an xml api for link resolver data. some openurl link resolvers provide xml apis. what is an xml api? an xml api is a web service where either the request or the response are in xml. in the case of the 360 link xml api, the request is in the form of a url, and the response is in xml. read more about web services at wikipedia. 360 link vs. other openurl link resolvers 360 link is a commercial link resolver product of serials solutions. there are other link resolver products available that accept openurl requests, some of which also provide xml apis. i will use 360 link as an example because i work for serials solutions (full disclosure). other openurl services include ex libris sfx, and ebsco link source. the 360 link xml api official documentation is not public, so this will be the first public look at how our api works. serials solutions includes the api with the purchase of 360 link. an interlibrary loan form as practical example when a library doesn’t have a full text resource, it’s common to direct patrons to an ill form, so that they can request the resource from another library. then the user needs to manually enter the citation data into the form; this is an error-prone and time-consuming process. our example is of a hypothetical “citation finder” service, which will direct the user to a pre-populated ill form only when the local library does not have a licensed electronic copy of the article citation requested. the patron could get to this ill form from either of two starting points. either the patron would fill out citation information in a search form on a library’s website, or the original data would come via an openurl string passed from another database’s search results. [2] the xml api client in php writing a client our client will be a computer program that sends an openurl request and receives the xml package. then we can use the data in the xml package to construct our ill form. the specific tasks our client software will engage in are: collect the citation data based on a user search for an article assemble the data for an openurl query generate a valid url from the data send it across the web to the 360 link api and recieve the xmlresponse read and extract variables from the xml display the variables as an html form many programming languages have all the tools you need to complete these tasks. i chose php, a scripting language designed for the web, for a few reasons: it's widely available, can be easily deployed in a wide variety of environments, is considered very easy to work with even for beginners, and has very good documentation at php.net. the request: an openurl query the first step in talking to the api is to generate an openurl request. constructing the url once we have a set of key-value pairs, we need to express them in the language of a url. the baseurl first we need to point to a certain location on the internet. this is done with the baseurl. each openurl provider has their own baseurl. the baseurl for serial solution 360 link is: http://client identifier.openurl.xml.serialssolutions.com/openurlxml the client identifier refers to a code provided by serials solutions for your organization. if your client identifier is 000-00, then your url would be: http://000-00.openurl.xml.serialssolutions.com/openurlxml if this url is opened in a browser, you will get an xml package back, but it will just be a diagnostic message in xml telling us that the server needs more information. next we will explain how to provide that information. openurl key-value pairs if we send only the baseurl, we’ll get a diagnostic message saying you’re missing a required field. first we’ll describe the information to provide, and then show how to encode this information in a url. openurl 1.0 has a certain minimum set of key-value pairs you need to send with each query, just to meet the openurl standard. in addition, the serials solutions 360 link xml api requires an additional version key-value pair that is not a part of the openurl standard. the version allows us to make changes to our api interface over time. add to these the fields for citation data, and we have a basic set of key-value pairs that allow us to interact with the server. if we miss one of these values, the server will return an error message in xml. required values friendly name notes key = value example value range 360 link api version included to allow for backward compatibility when newer versions of the api are released. this is not part of the openurl standard, but 360link xml api queries won’t work without it. version = 1.0 currently the 1.0 version is the only 360 link xml api version openurl version this refers to the ansi/niso standard for openurl itself. url_ver = z39.88-2004 z39.88-2004 refers to the current standard (1.0), and the only currently supported by the 360 link xml api. metadata format specifies what resource we’re referring to. openurl offers several metadata formats for different types of items. for the the purposes of this article we will use the journal format which is for journal articles among other things. rft_val_fmt = info:ofi/fmt:kev:mtx:journal read the openurl specification for other kinds of queries. in general, use the example value for 360 link xml api requests. in addition to including these basic minimum values, you need values that refer to a specific resource. the full openurl specification is fairly complex, and is the only place where all possible fields are documented. however, here is a list of the most common key-value pairs for citation data: friendly name key article title rft.atitle journal title rft.title genre rft.genre issn rft.issn journal volume rft.volume journal issue rft.issue page number or range rft.spage issue date rft.date author last name rft.aulast author first name rft.aufirst author full name rft.au author corporation rft.aucorp when you send this citation data for an article to the server, 360 link attempts to figure out what journal resource you’re looking for, and return full text results if they are available. to refer to a specific article, we can instead use a doi number or pubmed id by using the rft_id key with a special prefix for the value: friendly name key = value prefix doi number rft_id = info:doi/[doi number] pubmed number rft_id = info:pmid/[pubmed id] what data to send the api is expecting at least enough data to identify a journal, and will respond in one of three different ways: give an error message because of lack of data return only the most basic journal level information return full citation information for an article usually, the api will not return the article data (like the page number or article name) unless they’re passed in in the request. in many cases, however, a pubmed or doi id will fetch full article citation data automatically by contacting either a pubmed or doi service. our first example: a doi number query the only real required value is the version, which refers to the version of the serials solutions xml api we’re using. at the time of this writing, the 360 link api has only one version, 1.0, so this is the value we’ll use. then we need some way of referring to an actual resource, and the simplest way is to refer to an article by a unique article-level link. as an example, let’s use the doi number for the article how the mind hurts and heals the body. the doi id is 10.1037/0003-066x.59.1.29. here’s the key-value pair for our doi number: friendly name key = value doi number rft_id = info:doi/10.1037/0003-066x.59.1.29 notice that a doi number is specified by adding info:doi/ at the beginning of the rft_id value to tell it what kind of id we’re using. key-value pairs in a url query string this is where we send the actual information. url’s have a format for sending key-value pairs called the query string. the query string always starts with a question mark (?) that separates it from the baseurl: http://000-00.openurl.xml.serialssolutions.com/openurlxml? then we add a key-value pair, separated by an equals sign (=) version=1.0 then we add our required key-value pairs, separated by ampersands (&) version=1.0&url_ver=z39.88-2004&rft_val_fmt=info:ofi/fmt:kev:mtx:journal then we add our doi number so we can get some citation data: version=1.0&url_ver=z39.88-2004&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft_id=info:doi/10.1037/0003-066x.59.1.29 now we add the query string to the baseurl: http://000-00.openurl.xml.serialssolutions.com/openurlxml?version=1.0&url_ver=z39.88-2004&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft_id=info:doi/10.1037/0003-066x.59.1.29 this is close, but won’t quite work. url’s have rules about what characters are allowed. for example, question marks (?), equals signs (=), and ampersands (&) all have special meanings in a url, so if we need one of these characters in a url key or value, we need a different way to represent it. the only characters that are allowed in the key or value of a url are: upper case letters (a-z) lower case letters (a-z) a few symbols: the period (.), the dash (-), the underscore (_) and the tilde (~) all other characters, including the space character, need to be percent encoded. percent encoding url encoding is sometimes called percent encoding, because url codes start with the percent sign (%) and are followed by a 2-digit number. the percent code for a space is %20, but it can also be represented by a plus sign (+). you can view a chart of the percent codes at the wikipedia page on percent encoding. keep in mind that you don’t encode every character in a url, just the ones in the keys and values. when we convert these keys and values, we get the following url: http://000-00.openurl.xml.serialssolutions.com/openurlxml?version=1.0&rft_val_fmt=info%3aofi%2ffmt%3akev%3amtx%3ajournal&url_ver=z39.88-2004&rft_id=info%3adoi%2f10.1037%2f0003-066x.59.1.29 there’s no reason to do this process manually, because this is a great job for a computer. encoding the key-value pairs with php so, the first part of writing our client is getting your program to encode the values for you, and making a url the program can send. php has a function: urlencode() that takes an unencoded string, and returns a string that’s all ready for the key or value part of the url. here is a sample piece of php code that goes through a list of url keys and values like this: // required url values $required_url_elements = array(); $required_url_elements['version'] = '1.0'; $required_url_elements['rft_val_fmt'] = 'info:ofi/fmt:kev:mtx:journal'; $required_url_elements['url_ver'] = 'z39.88-2004'; // citation information $citation_url_elements = array(); $citation_url_elements['rft_id'] = 'info:doi/10.1037/0003-066x.59.1.29'; $url_elements = array_merge($required_url_elements, $citation_url_elements); and encodes each key and value, assembling our completed url: // url setup $client_identifier = '000-00'; $base_url = 'http://' . $client_identifier . '.xml.search.serialssolutions.com/openurlxml'; // query string values assembler $first_value = true; $query_string = ''; foreach ($url_elements as $key => $value) { // if it's the first value, use a ? to start the query string // otherwise, use the &amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;. $seperator = $first_value ? "?" : "&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;"; $first_value = false; // add this key-value pair to the query string // using php's urlencode() function $query_string .= $seperator . urlencode($key) . "=" . urlencode($value); } // add the base url to the query string $api_url = $base_url . $query_string; once you have the query string, you can use php to send it to the serials solutions server. now we have a valid openurl request submittable to 360 link’s xml api. trying it out in a browser if you are not a 360 link customer, with a valid client identifier and authorized ip address for that client identifier, you’re not going to be able to issue this request. if you can, go ahead and put that url right in a browser and see what you get. get a login popup? or a 501 error? then you’re not authorized. get a hunk of xml? perfect! sending the query with curl via php we’ve constructed our url, and we could easily print it out, and then cut and paste it into a browser. the whole point of using an xml api is being able to automate this kind of process, so that we can build our ill form automatically based on any resource our user wants. so we need a way to be able to send our key-value pairs to the serials solutions server and get the xml in response. next we will demonstrate sending a url and getting a response in php using the curl library[3]. curl in php since curl is an external library, we need to make sure it’s enabled in php. php has a handy way to look at its settings. just create a file that looks like this, and open in a web browser: <?php> phpinfo(); ?> php will generate a page showing all of its settings, and external modules. if curl is installed, you should see something like this included on the page: if you don’t see that, then you may need to do some additional steps to set up curl on php (or see curl php docs). alternately, you could use fopen()[4]. assuming you have php with curl working, here is a simple function that uses curl to send the url and get the resulting xml: /* * a simple example function using curl to send an openurl request and * receive an xml string for processing. * * @param string $url an openurl to attempt to retrieve * @return mixed $result an xml string, or false if an error */ function get_xml_result($url) { // initialize our curl handle $ch = curl_init(); // set the url curl_setopt($ch, curlopt_url, $url); // set the user agent $user_agent = $_server['http_user_agent']; curl_setopt($ch, curlopt_useragent, $user_agent); // we don't want the http header, we just want the xml curl_setopt($ch, curlopt_header, 0); // this makes the curl_exec call return a string curl_setopt($ch, curlopt_returntransfer, 1); // follow any location headers curl_setopt($ch, curlopt_followlocation, 1); // set some sane timeouts curl_setopt($ch, curlopt_connecttimeout, 30); curl_setopt($ch, curlopt_timeout, 120); // attempt to fetch the xml $result = curl_exec($ch); // make sure we don't have whitespace at the beginning $result = trim($result); // check to see if we were successful if (curl_errno($ch) != 0) { $result = false; // we're not using this, but this is how to get the error message: $error_message = curl_error($ch); } // close our curl handle curl_close($ch); return $result; } and what is $result? hopefully, it’s a piece of data from the serials solutions servers that we can read and parse. here’s an example of using the function properly: // url input for testing if ($api_url) { $xml = trim(get_xml_result($url)); } // check to see if we successfully got xml if ($xml !== false){ print("<h2>we got a successful response:</h2>"); print(htmlspecialchars($xml)); } else { print("<h2>curl error</h2>"); } let’s go over the basics of xml and what we need to know to parse our response. understanding the 360 link xml response the diagnostic message first of all, here’s the url we’d use to get this diagnostic message. it’s just the raw baseurl with no key-value pairs at all: http://000-00.openurl.xml.serialssolutions.com/openurlxml and here’s the resulting xml we’d get: <?xml version="1.0" encoding="utf-8"?> <ssopenurl:openurlresponse xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:ssdiag="http://xml.serialssolutions.com/ns/diagnostics/v1.0" xmlns:ssopenurl="http://xml.serialssolutions.com/ns/openurl/v1.0" xmlns:xsi="http://www.w3.org/2001/xmlschema-instance" xsi:schemalocation="http://xml.serialssolutions.com/ns/openurl/v1.0 http://xml.serialssolutions.com/ns/openurl/v1.0/ssopenurl.xsd http://xml.serialssolutions.com/ns/diagnostics/v1.0 http://xml.serialssolutions.com/ns/diagnostics/v1.0/diagnostics.xsd"> <ssopenurl:version>1.0</ssopenurl:version> <ssdiag:diagnostics> <ssdiag:diagnostic> <ssdiag:uri>sersol/diagnostics/7</ssdiag:uri> <ssdiag:details>version not specified</ssdiag:details> <ssdiag:message>mandatory parameter not supplied</ssdiag:message> </ssdiag:diagnostic> </ssdiag:diagnostics> <ssopenurl:echoedquery timestamp="2008-07-18t14:57:21"/> </ssopenurl:openurlresponse> here is a visual diagram of the elements we’d see in a diagnostic message. compare the elements in the xml to the diagram and it should give a pretty good idea of the shape of this data. let’s go over a few of these containers and their meanings: openurlresponse this element will always be present, whether we get a successful response or an error message. version this is the api version being echoed back to us, a kind of sanity check, if you will. we can always expect this container. echoedquery this is another sanity check, and should always be returned. it’s a copy of the url query we sent so we can make sure the api server got the same information we sent. diagnostics the container for one or more diagnostic elements. in fact, there’s only going to be one diagnostic element at a time. if we’re missing multiple required parameters, for instance, the api will just pick the first one we’re missing and report on that. diagnostic this element is a container for the actual error. theoretically there could be multiple diagnostic elements in the diagnostics element, but we will only ever get one from this version of the api. uri uri means uniform resource indicator. a url is one kind of uri. this refers to a code for the error message that was returned that refers to the 360 link and openurl documentation. message this is the error type that the uri refers to. this can be only one of a few messages and won’t give a lot of detail about what the problem was. details this will be more specific text about what went wrong. used in combination with the message you should be able to decipher what went wrong. taking a look at our earlier message, and putting together the message and details, we should be able to figure out what went wrong. message: mandatory parameter not supplied details: version not specified so, the message is not very specific. it just gives us the message that we missed a mandatory parameter. it is the details section which tells us which parameter was missing. other diagnostic messages the full list of diagnostic information is detailed in the 360 link xml api documentation, but some other kinds of diagnostic message types are: system temporarily unavailable authentication error unsupported version identifier with no data not enough metadata supplied a successful query now let’s look at an example of a successful query that returns results. here’s the example for our earlier doi example query: http://000-00.openurl.xml.serialssolutions.com/openurlxml?version=1.0&rft_val_fmt=info%3aofi%2ffmt%3akev%3amtx%3ajournal&rfr_id=info%3asid%2fsersol%3arefinerquery&url_ver=z39.88-2004&rft_id=info%3adoi%2f10.1037%2f0003-066x.59.1.29 and the xml result: <ssopenurl:openurlresponse xmlns:dc="http://purl.org/dc/elements/1.1/"> xmlns:ssdiag="http://xml.serialssolutions.com/ns/diagnostics/v1.0" xmlns:ssopenurl="http://xml.serialssolutions.com/ns/openurl/v1.0" xmlns:xsi="http://www.w3.org/2001/xmlschema-instance" xsi:schemalocation="http://xml.serialssolutions.com/ns/openurl/v1.0 http://xml.serialssolutions.com/ns/openurl/v1.0/ssopenurl.xsd http://xml.serialssolutions.com/ns/diagnostics/v1.0 http://xml.serialssolutions.com/ns/diagnostics/v1.0/diagnostics.xsd"> <ssopenurl:version>1.0</ssopenurl:version> <ssopenurl:results dbdate="2008-07-18"> <ssopenurl:result format="journal"> <ssopenurl:citation> <dc:title>how the mind hurts and heals the body.</dc:title> <dc:source>the american psychologist</dc:source> <dc:date>2004</dc:date> <ssopenurl:issn type="print">0003-066x </ssopenurl:issn> <ssopenurl:issn type="electronic">1935-990x </ssopenurl:issn> <ssopenurl:volume>59</ssopenurl:volume> <ssopenurl:issue>1</ssopenurl:issue> <ssopenurl:spage>29</ssopenurl:spage> <ssopenurl:doi>10.1037/0003-066x.59.1.29</ssopenurl:doi> </ssopenurl:citation> <ssopenurl:linkgroups> <ssopenurl:linkgroup type="holding"> <ssopenurl:holdingdata> <ssopenurl:startdate>01/01/1946</ssopenurl:startdate> <ssopenurl:providerid>prvcsa</ssopenurl:providerid> <ssopenurl:providername>csa</ssopenurl:providername> <ssopenurl:databaseid>cpb</ssopenurl:databaseid> <ssopenurl:databasename>apa psycarticles</ssopenurl:databasename> <ssopenurl:normalizeddata> <ssopenurl:startdate>1946-01-01</ssopenurl:startdate> </ssopenurl:normalizeddata> </ssopenurl:holdingdata> <ssopenurl:url type="source">http://www.csa.com/htbin/dbrng.cgi?username=xxxxx&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;access=xxxxx&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;db=psycarticles-set-c&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;mode=browse </ssopenurl:url> <ssopenurl:url type="journal">http://www.csa.com/htbin/dbrng.cgi?username=##########&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;access=$$$$$$$$$$&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;db=psycarticles-set-c&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;issn=0003-066x&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;mode=all </ssopenurl:url> </ssopenurl:linkgroup> <ssopenurl:linkgroup type="holding"> <ssopenurl:holdingdata> <ssopenurl:startdate>01/01/1946</ssopenurl:startdate> <ssopenurl:providerid>prvovd</ssopenurl:providerid> <ssopenurl:providername>ovid</ssopenurl:providername> <ssopenurl:databaseid>opa</ssopenurl:databaseid> <ssopenurl:databasename>journals@ovid psycarticles</ssopenurl:databasename> <ssopenurl:normalizeddata> <ssopenurl:startdate>1946-01-01</ssopenurl:startdate> </ssopenurl:normalizeddata> </ssopenurl:holdingdata> <ssopenurl:url type="source">http://www.ovid.com/site/catalog/group/886.jsp?top=2&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;mid=3&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;bottom=7&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;subsection=12 </ssopenurl:url> <ssopenurl:url type="article">http://ovidsp.ovid.com/ovidweb.cgi?t=js&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;page=fulltext&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;news=n&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;d=ovft&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;search=0003-066x.is+and+59.vo+and+1.ip+and+29.pg </ssopenurl:url> <ssopenurl:url type="journal">http://ovidsp.ovid.com/ovidweb.cgi?t=js&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;page=toc&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;d=ovft&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;mode=ovid&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;news=n&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;an=00000487-000000000-00000 </ssopenurl:url> <ssopenurl:url type="volume">http://ovidsp.ovid.com/ovidweb.cgi?t=js&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;page=titles&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;news=n&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;d=ovft&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;search=0003-066x.is+and+59.vo </ssopenurl:url> <ssopenurl:url type="issue">http://ovidsp.ovid.com/ovidweb.cgi?t=js&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;page=titles&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;news=n&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;d=ovft&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;search=0003-066x.is+and+59.vo+and+1.ip </ssopenurl:url> </ssopenurl:linkgroup> </ssopenurl:linkgroups> </ssopenurl:result> </ssopenurl:results> <ssopenurl:echoedquery timestamp="2008-07-18t15:02:32"> <ssopenurl:library id="cs3rs7ts5u"> <ssopenurl:name>julia's test library 3 (linkers as of 2.03)</ssopenurl:name> </ssopenurl:library> <ssopenurl:querystring>rft_id=info%3adoi%2f10.1037%2f0003-066x.59.1.29&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;version=1.0&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;rft_val_fmt=info%3aofi%2ffmt%3akev%3amtx%3ajournal&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;rfr_id=info%3asid%2fsersol%3arefinerquery&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;url_ver=z39.88-2004</ssopenurl:querystring> </ssopenurl:echoedquery> </ssopenurl:openurlresponse> even though this example is a little more complicated, you should still be able to make out the structure of these responses. and a diagram of the successful result: now, let’s sort out the important information from things you can safely ignore. namespaces take a look at this piece of xml: <ssopenurl:results dbdate="2008-07-18"> <ssopenurl:result format="journal"> <ssopenurl:citation> <dc:title>some article not at the library</dc:title> <dc:creator>some guy</dc:creator> <dc:source>some journal not at the library</dc:source> <ssopenurl:volume>5</ssopenurl:volume> <ssopenurl:issue>32</ssopenurl:issue> <ssopenurl:spage>12</ssopenurl:spage> </ssopenurl:citation> </ssopenurl:result> </ssopenurl:results> the element names are prefixed by some characters seperated by a colon (:). these represent groups of elements. the dc: prefix, for instance, refers to meta-information specified by the dublin core standard for library information. for the purposes of interpreting our xml results, they can mostly be ignored, and you can use only the element name (the part after the colon) to refer to them. you’ll notice that the first few lines of the xml mostly serve to set up these namespaces. these can be safely ignored as well. it’s beyond the scope of this tutorial to explain every part of the xml response, since we’re mostly focusing on what we’d get in the citationdata container, but let’s go over a few of the containers we’ll see: results if you got this container, you know some real data was retrieved, and not just a diagnostic message. a results element can contain one or more result elements and nothing else. result a result element represents a single journal-level resource. it’s possible to get more than one result, but that means that the 360 link server wasn’t sure which resource you mean. when you get more that one result element, you usually need some kind of disambiguation to help the user decide what resource they meant. linkgroups / linkgroup this is usually what the 360 link web-based tool and api are both primarily for: to indicate and link to full-text resources for the requested article. the details of a linkgroup element are beyond the scope of this article, since we want to fill out an ill form only when the library does not have a full text resource available and it needs to be requested from another library. citation this is the focus of most of our attention. when we get back citation data, but no full text link, that means we can use the citation data to populate our ill form. usually, we will get journal level citation data back from the server. we will not usually, however, get article level links, unless we passed them into the query. for example, if we pass in a page range in the url, we will see that page range echoed here. an exception is when we pass in a doi or similar article level identifier. in this case, the api will sometimes fetch more complete data about the article from a similar service. see the later code examples, and the sample ill form for examples of the kind of citation data we’ll be looking for to populate our ill form. parse the xml response with php now that we can send a query and get the results back as xml, we need to parsing a piece of xml means to use a program to break it apart into its component parts, so any piece can be accessed. there are many methods for parsing xml files, and i’m choosing the dom method for this tutorial. there is an overview on the php site that shows all the different methods. parsing a piece of xml means to use a program to break it apart into its component parts, so any piece can be accessed. there are many methods in php for parsing xml files, and i’m choosing the php4 domxml method for this tutorial. there is an overview on the php site that shows all the different methods. php processing changed a bit in php5. i use a php4 example because php4 is still the most widely installed. if you read and understand these examples, it shouldn’t be too hard to read the php5 dom documentation and adapt the code here for php5. turn an xml string into a php object the first thing we need to do is take our xml string and turn it into something useful in php. the way to do this is to use the domxml_open_mem() method. this method takes an xml string and attempts to turn it into something called a domdocument object. this is a php object that acts like an xml container with containers inside it, but allows you to ask the container for information. this is how you use it: if ($xmldomdoc = domxml_open_mem($xml)) { // successfully created a domdocument object. } else { // failed to create a domdocument object } get the ill data out of the php object now that we have an object in php that represents our xml, the xml is in a much more useful form. now we can use some php tools to look for specific containers and get their contents. looking for elements with get_elements_by_tagname() now that we have a php object, we have a handful of things we can do with it, but the most useful thing is to ask the object for its elements. that is done using the get_elements_by_tagname() method. get_elements_by_tagname() takes a name of an element, and always returns an array whether it finds any elements by that name or not. getting the contents of elements with get_content() the next important method of the domdocument object is the get_content() method. if you’re in a container that has the actual data you need, you can call get_content() to get the actual text inside. these two methods are just about all we need to get the data for our ill form. here’s some examples of how to use them: // getting an array of containers with get_elements_by_tagname() $array_of_containers = $xmldomdoc->get_elements_by_tagname('version'); // working our way through each each element in the collection foreach ($array_of_containers as $container) { $version_number = $container->get_content(); } // getting the only element if we know there's only one $the_only_version_container = array_pop($xmldomdoc->get_elements_by_tagname('version')); // if we do it this way, we need to check to make sure // we got an actual container object, otherwise calling // get_content() on null will crash our program if (is_callable(array($the_only_version_container,'get_content'))) { $version_number = $the_only_version_container->get_content(); } using array_pop() and is_callable() to confirm the presence of data some containers may or may not appear in the xml result. issn, for instance, we would want to display in our ill form if available, but our xml might not have an issn container. if we call get_elements_by_tagname() on the issn container, it will return an empty array. we’re calling array_pop(), which returns the first element in an array. if the array is empty, it returns null, meaning no value. if you try to call get_content() on null, php will return a fatal error: call to a member function on a non-object. one way of avoiding this in php is to use is_callable() to make sure get_content() is allowed for the object you’re about to call. you can see an example of use in the example above. now we have the tools we need to handle our ill form. when to display the ill form we don’t automatically want to display an ill form. the only case where we want to is when a user requests a valid resource we don’t have access to at the library. here’s a chart that shows the different cases and how to handle them: situation meaning how to handle it we get a diagnostic message somehow we didn’t construct a valid request. recover gracefully and display a useful error message to the user. log the error and troubleshoot the problem. we get multiple results. there was more than one matching possibility for the information sent. display a disambiguation page to let the user choose which resource they meant. we get a result with full text links the resource is available to the organization or library. display links to the full text of the article. we get a result with no full text links the resource isn’t available. use the citation information to populate an ill form. attached are some helper methods and code that will let you determine which of these decisions to make. these methods work by looking for containers by a certain name, and then using the count() method to see how many containers were found. if there is no diagnostics container, then we didn’t get an error message. if there are more than one result container then that means we got more than one article result. if there is a linkgroup container, that means we got link resolver results. once we have these methods, we can use them to make a decision about what to display: // ill display decision tree $do_ill_form = false; if (docxml_has_diagnostic($xmldomdoc)) { // display error message } elseif (docxml_has_multiple_article_results($xmldomdoc)) { // display disambiguation page } elseif (docxml_has_link_groups($xmldomdoc)) { // display links to fulltext article } else { // display the ill form $do_ill_form = true; } getting the variables we need let’s assume we know we want to display the ill form now. here’s a query example of a fake article, where we pass in citation data we made up. of course, the link resolver won’t find any full text results, and we will want to construct the ill form. http://000-00.openurl.xml.serialssolutions.com/openurlxml?rft.title=some+journal+not+at+the+library&rft.atitle=some+article+not+at+the+library&rft.au=some+guy&rft.volume=5&rft.issue=32&rft.pages=12-14&version=1.0&rft_val_fmt=info%3aofi%2ffmt%3akev%3amtx%3ajournal&rfr_id=info%3asid%2fsersol%3arefinerquery&url_ver=z39.88-2004 attached is an example of an xml response we’d get, with the citation data included and no full text links. here’s the code we’d use to get the variables we need. note that the xml element names are not always the same as the openurl variables! // first, get the citation container, it has everything we need $citation_container = array_pop( $xmldomdoc->get_elements_by_tagname('citation')); // then get the container and content for each variable to display: // author $creator_container = array_pop( $citation_container->get_elements_by_tagname('creator')); if (is_callable(array($creator_container,'get_content'))) { $au = $creator_container->get_content(); } # more ( more complete example attached here. ) constructing the ill form in html now that we have all the php variables we need, we need only the smallest effort to turn an html form into one that auto-populates with the citation data. using an html form as an ill form ill form example we will start with an example of a simple static ill form that will always have it's values start out blank, then we'll explain how to make one that's populated by our php script with the citation values returned in xml by 360 link. attached is some sample html for a simplified version of an ill form. include some css to make it pretty: form .formfield { margin-bottom: .3em; clear: left; } form .formfield label { width: 12em; float: left; } form .formfield input { width: 20em; } adding php to populate the form we take that static html from before, and embed it in a php file, adding code to populate the form the variables we've grabbed from the xml response. here’s one example: <input type="text" id="au" name="au" value="<?php print($au); ?>" /> attached here is the whole form retooled to show our variables. what’s missing this form doesn’t have any code included to make it validate. that is to say, it doesn’t do anything to ensure that the values people enter are valid, or that they have filled out the required fields. but form validation using php or javascript is well documented on the web and beyond the scope of this article. here’s a few good articles on form validation: simon willison’s weblog: easier form validation with php form validation using php to highlight non valid fields sensible forms: a form usability checklist a guide to unobtrusive javascript validation for the purposes of this article, we’re only populating the citation data, but if you have a logged in user, it’s possble to populate the name and email address of the requester as well. handling the submitted ill form let’s take a look at the form tag in the email form: <form action="incoming_ill_form_handler.php" method="post"> there’s two useful pieces of information here. one is the action and the other is the method. this sends the information in the form to a php script called incoming_ill_form_handler.php via the http post method. reading the form data php fills an array called $_post with incoming form data. here’s a simple way to debug and look at the information: <span class="html-tag">print</span>(<span class="quoted-strings">'<h2>post values</h2>'</span>); <span class="html-tag">print</span>(<span class="quoted-strings">"<pre>"</span>); var_dump(<span class="php-variables">$_post</span>); <span class="html-tag">print</span>(<span class="quoted-strings">"</pre>"</span>); if we submitted the form as filled, we’d see this: array(14) { ["name"]=> string(0) "" ["division_dept"]=> string(0) "" ["email"]=> string(0) "" ["priority_needby"]=> string(0) "" ["au"]=> string(8) "some guy" ["atitle"]=> string(31) "some article not at the library" ["title"]=> string(31) "some journal not at the library" ["date"]=> string(0) "" ["volume"]=> string(1) "5" ["issue"]=> string(2) "32" ["pages"]=> string(2) "12" ["issn"]=> string(0) "" ["pmid"]=> string(0) "" ["s_request"]=> string(14) "submit request" } previewing the data in a web page the simplest use of the form data is to display it on a web page as a preview. attached is a quick code sample for doing this safely. see the security section for more details on why to use htmlspecialchars(). sending an email with the ill form sending an email is similar. we just need to construct a string for the body of the email and use php’s mail() function. here is a basic example: $body = "ill form data \n\n"; $body .= "name: " . $_post['name'] . "\n"; $body .= "division:" . $_post['division_dept'] . "\n"; $body .= "email:" . $_post['email'] . "\n"; $body .= "need date:" . $_post['priority_needby'] . "\n"; $body .= "author:" . $_post['au'] . "\n"; $body .= "article:" . $_post['atitle'] . "\n"; $body .= "journal:" . $_post['title'] . "\n"; $body .= "issue date:" . $_post['date'] . "\n"; $body .= "pages:" . $_post['pages'] . "\n"; $body .= "issn:" . $_post['issn'] . "\n"; $body .= "pubmed id:" . $_post['pmid'] . "\n\n"; /** * returns false if text contains newline character */ function has_no_newlines($text) { return preg_match("/(%0a|%0d|\\n+|\\r+)/i", $text) == 0; } $sanitized_name = 'default name'; if (has_no_newlines($_post['name'])) { $sanitized_name = $_post['name']; } $subject = "an ill request form has been filled out by " . $sanitized_name; $to = 'danieltalsky@gmail.com'; mail($to, $subject, $body); storing the ill form in mysql covering all aspects of storing data using php and mysql is beyond the scope of this article, and is well covered in the excellent mysql documentation, mysql documentation about php and php documentation about mysql. still, a few useful examples are provided below. first of all, here’s a simplified example of a table structure that would store our ill form: create table `ill_form` ( `id` serial, `name` varchar(100) not null, `division_dept` varchar(50), `email` varchar(255) not null, `need_date` varchar(50), `author` varchar(100), `article` varchar(255), `journal` varchar(255), `issue_date` varchar(50), `pages` varchar(20), `issn` varchar(10), `pmid` varchar(20), `time_submitted` timestamp not null default now() ); and here’s an example of php code that would construct a safe sql query: $sql_insert = 'insert into ill_form '; $sql_insert .= ' (name, division_dept, email, need_date, author, '; $sql_insert .= ' article, journal, issue_date, pages, issn, pmid) '; $sql_insert .= ' values '; $sql_insert .= ' ('; $sql_insert .= "'" . mysql_real_escape_string($_post['name']) . "',"; $sql_insert .= "'" . mysql_real_escape_string($_post['division_dept']) . "',"; $sql_insert .= "'" . mysql_real_escape_string($_post['email']) . ', '; $sql_insert .= "'" . mysql_real_escape_string($_post['priority_needby']) . "',"; $sql_insert .= "'" . mysql_real_escape_string($_post['au']) . "',"; $sql_insert .= "'" . mysql_real_escape_string($_post['atitle']) . "',"; $sql_insert .= "'" . mysql_real_escape_string($_post['title']) . "',"; $sql_insert .= "'" . mysql_real_escape_string($_post['date']) . "',"; $sql_insert .= "'" . mysql_real_escape_string($_post['pages']) . "',"; $sql_insert .= "'" . mysql_real_escape_string($_post['issn']) . "',"; $sql_insert .= "'" . mysql_real_escape_string($_post['pmid']) . "',"; $sql_insert .= ' );'; using mysql_real_escape_string() allows mysql to handle most tricks people would use to try to get mysql to execute sql code you didn’t intend. finally, a couple of samples of retrieving the data using raw sql (mysql specific): /* gets all entries */ select * from ill_form; /* gets all entries in the last month */ select * from ill_form where date_sub(curdate(),interval 1 month) unpleasant but necessary security addendum always remember, do not trust data supplied by a user-submitted form. people or automated bots can place code in webforms that’s meant to hack the browser through javascript, php, or sql. these kinds of attacks are common and easy to do, so it’s vital to be responsible with your organization’s data. some security can be handled by good validation of the form when it’s submitted, but that is not a substitute for handling user-submitted data responsibly. it is easy for a hacker to send data to your processing page in a way that bypasses your form validaton. it can seem like there’s so much you can’t know about security that it’s almost not worth bothering, but the following examples should provide coverage from the most obvious kinds of attacks. for display on an html page people can enter both html and javascript in form fields. if you’re displaying things people entered in a web form back to them, it’s vital that you use a function like php’s htmlspecialchars(). this converts any special characters to a safe character set for display and derails the potential for javascript execution. just imagine if someone entered the following into a form field and you just printed it right to an html page! <script type="text/javascript">alert('bad security!');</script> for emailing the essential thing to remember is if you are inserting something from user input into the headers of an email (the subject, the from, to or cc lines), you need to make sure there aren’t any non-printing characters or linebreaks. usually a regular expression that makes sure it’s a valid email address or at least doesn’t contain the linebreak in the \r\n format is sufficient to prevent email injection. this is covered in detail in this excellent tutorial on preventing php email injection. for insertion into a database this is possibly the most dangerous and easily avoided problem. this kind of attack is called sql injection, and is well detailed on the wikipedia page on sql injection. all sql api’s will have an appropriate quoting mechanism or prepare statement. the important thing to remember is to never put raw data from a user into a sql query like this: $sql = "select * from users where name = '" . $user_submitted_name . "';"; this sample doesn’t make sure special characters are escaped, and allows room for sql injection. see our mysql example earlier for one example of doing it properly using mysql_real_escape_string() anytime you use data from a user form, just think about how the data is being used, and consider the possibility of abuse. in conclusion there are many possible uses for the 360 link xml api, and this article was meant to explain one of the simplest ones: using the returned citation data to construct an ill form. we covered the basics of querying the api server, interpreting the response, parsing the xml and displaying the citation data in an html form, then storing the results of that form in a mysql database or sending them in an email. these skills, especially the security notes, are useful in almost any kind of web application. if you’re considering using the xml api in any way, i’d love to hear about it. please contact me at danieltalsky@serialssolutions.com and tell me what you’re working on. i’ll be able to answer a few simple questions and point you in the direction of further resources based on your goals. notes [1] note from the editors: niso standard z39.88 formally defines openurl, but it can be a bit confusing for some. more information is also available at http://openurl.code4lib.org and from jeff young’s q6 blog: http://q6.oclc.org/. [2] many ill forms are behind a login, but the new york academy of medicine has been kind enough to allow us to link to the nyam ill form as an example for this article, and use their form as an inspiration for the basic html example used in the article. [3] curl is a php library for accessing files over the internet. php has another way of accessing files over the internet: its native fopen() function. fopen() is a lot slower and in some ways is less reliable. curl is more reliable, flexible, and over ten times faster. [4] if there’s some problem with curl on your server, it shouldn’t be too hard to use fopen(). here’s the php fopen() documentation, and information about using it to open files across the internet. acknowledgements kat ortland for copy editing, shawn kilburn for the article concept and consultation on 360 link, brenna flood for extensive consultation on the 360 link xml api, chris phillipe, for 360 link consultation, ross bleakney for getting me into xml api documentation at serials solutions, mark foong & harry kaplanian for providing the time and operational support for me to write this article and lisa genoese & steve chiaffoni at the new york academy of medicine for supplying code and permission to use their ill form as an example. about the author daniel talsky is a programmer and tech educator at serials solutions. he is beginning a tutorial and outreach program to educate librarians and web developers about serials solutions xml api’s. he can be contacted at daniel.talsky@serialssolutions.com . subscribe to comments: for this article | for all articles 5 responses to "auto-populating an ill form with the serial solutions link resolver api" please leave a response below: emily lynema, 2009-03-25 nice article, daniel! we’re considering the possibility of switching to 360 link from sfx, and this article helps make clear the kind of functionality available via the api. daniel talsky, 2009-03-26 thanks emily, that’s really the purpose of the article, to give an example of use and processing. however, it’s worth saying, you don’t need the api to be able to generate an ill form. even using 360 link, you can create simple links back to your ill form that contain all the information you’ll need. patsy, 2009-09-10 thanks, david! we’re in the process of creating an ill form and were stuck at this point in the process. much appreciate! mark, 2012-02-14 thanks, our library just got 360link so this is very helpful to getting started with using the api. is there any other public documentation available? one note for new readers: the api base url is currently’http://000-00.xml.open.serialssolutions.com/openurlxml?’, not ‘http://000-00.openurl.xml.serialssolutions.com/openurlxml?’ as described in the code in this article. benjamin, 2015-03-03 since this article was written in 2009 and the last post was in 2012, has there been any more development? we are currently looking at a process to do this with ill also and any information or current work would be great. thanks leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – editorial: beyond posters: on hospitality in libtech mission editorial committee process and structure code4lib issue 40, 2018-05-04 editorial: beyond posters: on hospitality in libtech in this editorial, i will be using the word hospitality to mean the intentional welcome of others into a space which one currently occupies, possibly as a member of a dominant group. i do not wish to encourage the idea that one should cultivate or maintain a role of benevolent host in a way that forces others to remain forever guest or outsider, although there will always be newcomers. hospitality may be a first step to ceding one’s position as host in a space. it may be expanding that space to become a place with many potential hosts, each respected for their varied contributions and skillsets. it may also be supporting those in a different space or a different role, such as those who use the technologies we build and support (both colleagues and patrons), and respecting them in that space. i’d like to thank both becky yoose and kim phipps (messiah college president) for fixing the term hospitality in my mind, chris bourg for the challenge she extended and the phrase “for the love of baby unicorns,” linda ballinger for asking questions, julie hardesty for a year of collaboration toward a useful metadata documentation plan, the samvera documentation working group for their work and for helping the metadataists formulate a plan, and the c4lj editorial committee and becky for providing feedback on this draft i once owned a star trek uniform. it was deep space nine[1] style, blue for science and for jadzia dax, and an awkward, terrible fit on me. the second website i ever built was a ds9 geocities fan site. i was the kind of teenage girl who spent hours downloading miniscule 30-second trailers of episodes on startrek.com, just to get her fix, and i don’t regret that… for all that i still can be a walking memory alpha[2]. what i’m writing about today is hospitality in libtech, and my starting point is chris bourg’s 2018 code4lib keynote (keynote video). i understand, appreciate, and support the message of the study chris cited which mentioned star trek posters. i’m writing on the side of trekkies for inclusion. i shouldn’t assume, for example, you know that memory alpha is the trek wiki, named for the united federation of planets’ library/data archive.[3] this is the last trek reference, i promise. i’m going to walk through some of my own reflections on hospitality in library technology, building from chris’s talk to encounters with the idea i’ve had over the last year in particular. my undergraduate institution pushed “hospitality” hard (i was given a foot-washing towel at graduation) and it’s taken a long time for me to move beyond feelings about its pitfalls and a buzzword feel to appreciate its real worth as a concept. listening to becky yoose describe documentation as hospitality in an interview on the ( open paren podcast began the word’s rehabilitation for me. chris challenged me again to think intentionally about spaces i occupy and not just how i speak to but about (vouching for) others. in this editorial, i will be using the word hospitality to mean the intentional welcome of others into a space which one currently occupies, possibly as a member of a dominant group. i do not wish to encourage the idea that one should cultivate or maintain a role of benevolent host in a way that forces others to remain forever guest or outsider, although there will always be newcomers. hospitality may be a first step to ceding one’s position as host in a space. it may be expanding that space to become a place with many potential hosts, each respected for their varied contributions and skillsets. it may also be supporting those in a different space or a different role, such as those who use the technologies we build and support (both colleagues and patrons), and respecting them in that space. intentionality as hospitality when it comes to behavior in the physical and virtual spaces we create for our community, a thoughtful and enforceable code of conduct is only the bare minimum for hospitality. actually following up and enforcing such codes takes time and labor, but that work, followed by reports back to the whole community are the only way to build a deserved trust. beyond the language, behavior, and imagery covered by codes of conduct, however, we should intentionally consider the less overt ways in which we may exclude. do we consider whether everyone in a group knows what we mean by ils, git branching, 245$a, or other terms which arise in our daily work? at the 2017 samvera connect, my colleague linda ballinger encouraged her fellow metadataists on proactive responses to feeling overwhelmed by a barrage of new-to-you terms. such terms are much like trek posters, inoffensive in themselves, but exclusionary in certain contexts. the onus is on all of us to create environments where it’s ok not to know a thing and safe to ask. the recurse center social rules, which are becoming more common at libtech events, provide guidance for creating such an environment. they read as follows: no feigning surprise no well-actually’s no back-seat driving no subtle -isms these are an excellent starting point. if we’re trying to extend boundaries of the spaces we occupy, i would challenge us all to try the following as well: recognize that you can learn from those just starting out. embrace familiar questions as an opportunity to reflect on why they recur. consider your usage of acronyms and jargon. value both technical (code, qa, testing, ux, and accessibility) and interpersonal contributions to projects. regarding acronyms and jargon, this challenge is not to stop using them entirely but to proactively ensure everyone involved has sufficient context to understand what they mean. documentation as hospitality beyond physical spaces and gatherings for sharing and learning, we need to be intentional about the kinds of materials we build for those learning on the job through self-teaching. i would venture that anyone, no matter their area of expertise, has encountered some kind of documentation which fell two or three steps short of where they needed it to be. perhaps it was a primer with assumptions. perhaps they could find beginner and advanced materials but nothing to guide folks who’ve got the basics down but want to move to an intermediate level. and, of course, sometimes there’s no documentation at all. as mentioned in the introduction, becky yoose spoke on the ( open paren podcast[4] of the concept of documentation as hospitality. she then expands on her definition of hospitality as: “making sure that folks can enter in a particular community with the lowest bar possibly that i can do personally or i can build into the community systematically (or persuade others or bribe others) to make a more inclusive community” (emphasis added) this one-on-one work is a critical aspect of building community, but relying solely on it will leave out those uncertain who or where to ask questions and burn out those offering such assistance. as communities, we would do well to determine systematic approaches to easing community entry, both through concerted efforts at building regular, inclusive learning spaces and through our documentation.[5] a project which i’ve been watching and admiring is the samvera documentation working group’s a guide for the perplexed samvera developer. while still under development, it is an excellent example of documentation which tries to avoid presumption. early on in the “new? start here” section, the page “how to ask for help” includes directions for joining the samvera slack. most real-time conversation happens on the slack, particularly on its dev channel. the page then points readers to slack’s own documentation on getting started as a slack user. will many readers already know what slack is and how to use it? sure. will all? nope, and the documentation working group made the welcoming choice. meanwhile, while terry reese wrote his previous editorial on learning to be a selfish librarian, he puts a great deal of work into supporting marcedit. while many may be familiar with his tutorial videos, i only recently encountered the in-progress learning marcedit, which includes full chapters to spell out things like what every choice in preferences means, with context. as with samvera, this documentation does not require one to start out at some intermediate stage while still describing how to do complex tasks. from my own experience, i’m aware how difficult it is to block out time to write up documentation, even if you consider it a critical component of a project. besides time, however, i see a couple other impediments which documenters face. one is skill—knowing a lot about a thing and knowing how to write do not necessarily mean you’ll be good at writing its documentation. the other is that one may feel uncertain or even presumptuous in attempting documentation alone. does one know everything which should be documented? can one assess where others will need to start? this is one case where what i’m grateful for overlaps with the write the docs community, which provides principles and hosts events where one may practice them. working on the practice in a group, as above, also means that one person need not feel the full weight of creating welcoming documentation. one has partners, gets feedback, and benefits from multiple perspectives and backgrounds. the samvera metadata interest group has had false starts in figuring out how such a group should work, although we are now trying to align our work more with the developer-focused documentation wg, an effort which i hope will be productive.[6] design as hospitality as we work to create more welcoming environments and the means for everyone in them to participate in the work we’re doing, we may also turn our attention to what we build (or implement) together. a fundamental level of how we design hospitality in our systems is accessibility. rather than give a meager overview in this short space, i’ll refer readers to ng and schofield’s article in issue 37 of this journal. beyond their accessibility and usability, however, there are other aspects of our systems which i would like to frame through the lens of hospitality. specifically, i want to address system design, record-keeping, and hospitality. in my january 2017 editorial, i dedicated a section to “the records we keep. here i am concerned with a false hospitality which alleges we can become more hospitable through aggressive data collection. jones and salo have written an overview of learning analytics systems in academia and how these intersect with libraries and library ethics. besides the continual pressure libraries feel to justify themselves through learning outcome assessment, these programs are pitched as helping students do better in college. if such programs actually improve student success (according to some definitions of success) should we change our principles around privacy? beyond the challenges to this assertion outlined by jones & salo, i think we must also ask ourselves what kind of a world it is we want to be creating for our students. is it one in which every institution surveils on them and reports back to others, ostensibly with their own good in mind? is it one in which yet another dataset about them could be given or sold to researchers, demanded by governments, or compromised by hackers? hardly a hospitable choice. nor is the problem limited to academic libraries. public libraries already collect and share certain portions of patron data with a variety of content and service providers, including amazon. through its new product, “wise,” oclc now proposes to partner with libraries to collect all that data in one place, managed in the cloud by oclc. some of the marketing around this product struck me as connected the editorial i was already drafting. in particular, their blog post includes the following testimonial, that wise “removes subjectivity—instead of an interpretation of customer wants or what customers say they want, the system uses data to determine need.” i read this statement (from a customer, but promoted by oclc) as a fundamentally false, hollow hospitality. algorithms developed by human beings will reflect their assumptions and biases and impart their own oppressions. since the product is only just being marketed, i contacted oclc with some of my questions and concerns.[7] the responses i received from their new data protection officer were primarily marketing language, uncertainty about most technical specifics (it appears they still need to learn what’s in the wise system, which they purchased from its developer, and build from there), and assurances that they take privacy seriously without any firm commitments to minimum standards. i believe the following excerpt may be an accurate assessment of what wise attends to accomplish: “…it takes the very best of disparate systems and combines them into a unified, powerful solution; the collection and use of data, however, is not materially different from today’s practices.”[8] i was reminded of facebook’s recent statement to reuters, that “data collection is fundamental to how the internet works.” data collection certainly is the way things are right now. but rather than designing systems to collect, aggregate, and analyze that data more smoothly, perhaps we should assess where we are and reconsider. humans will remain subjective, frustrating, and beautiful, no matter what kind of technology we throw at them. i hope we’ll remain aware enough about that to meet them as fellow humans whose inquiry, escapism, and entertainment we facilitate and whose privacy and safety in doing so we respect. the labor of hospitality if any of the above were easy, we wouldn’t have to talk about it. intentionality about our space and words takes work. for most of us, it’s a practice that requires mindful attention. i certainly have a way to go when it comes to jargon. writing the documentation may take as much time as creating the code did, with far less administrative support or reward in our careers. and trying to understand or negotiate data in external systems or advocate for better choices in what we’re making may require a good deal of time and emotional labor. pushing back may even be dangerous to one’s employment. i’m aware of and grateful for the time another colleague recently put into phone calls and emails with a vendor after an internal group expressed concerns regarding privacy. neither getting the answers nor documenting them in a way that the whole group could understand were easy. if code4lib is good for one thing, though, it’s community. as we continue to recognize where we fall short, personally and collectively, in welcoming others, let us support each other in lowering barriers to entry and in building systems which extend that same hospitality and respect to the communities we serve. end notes [1] deep space nine was the third live-action star trek series, running in the mid-to-late 1990s. over its 7 years, two styles of starfleet uniform were used. mine was, regrettably, the former. [2] memory alpha is a wiki-based project reference for the star trek universe. [3] i’m now overcome by the desire to write a comparison of the star trek united federation of planets’ library/archive setup and that of the galactic empire as shown in star wars: rogue one. [4] ( open paren. [ for the writer’s piece of mind…let’s close these ) ) ) ] [5] speaking of systematic approaches, tools which serve as programmatic intermediaries, such as marcedit, the c# marc editor, and openrefine and the accompanying documentation and tutorials are invaluable in lowering bars to entry and expanding communities. their support of multiple skill levels also provides an opportunity for users to gradually build advanced skills (and confidence in their tooling expertise) which they may bring to other contexts. [6] in my own experience of trying to lower barriers for entry, i’ve also realized some of the pitfalls when documentation may become the primary way one learns about a thing. in creating the eadiva project, my goal was simple—rewrite the ead tag library in a way my classmates, who weren’t familiar with the language of document type declarations, could use it. and interlink it for usability. however, simply learning what elements exist in ead and how to encode data in them is not a good way to learn the practice of archival description. beyond linking out to the html form of describing archives: a content standard and pointing out on the about page that it is primarily a technical reference, i sometimes struggle with how i can shape the site to make it that the site really is about the technical aspects of encoded archival description and not about its practices. [7] in the interest of disclosure on my end, i asked the following questions: is the entire wise program opt-in for the patrons themselves? (e.g. if my public library adopts it, will it opt me in to some aspects and allow me to opt into others or does it give me a full opt-in option on my account page/elsewhere? does the library decide this?) can a patron continue to use a public library which has adopted it without their data being gathered in some way by wise? (this question includes shadow profiles, similar to those facebook created for people without accounts.) is the data stored entirely within an ils system or centrally on oclc’s servers/in the cloud? answer: cloud, like oclc’s ilses. will the data from multiple libraries be aggregated for overall analysis by oclc? while oclc has made clear it won’t sell patron data, will it be sharing that data with researchers or other library partners? which other oclc products might be informed by that data? answer: reiteration that they won’t sell patron data. if patrons opt in, are they told explicitly how their data is being aggregated and used? (i’d appreciate a copy of whatever disclosure is given, but also how it’s being conveyed in a non wall-of-text format since it’s a known issue that end-users don’t often read such agreements in detail) if data is stored centrally, would this be the largest dataset of library patrons’ borrowing and taste information ever compiled? what plans does oclc have to keep an unprecedented and tempting dataset safe from hackers? does oclc have a plan for what will happen if wise data (centralized or in one library) gets compromised? this question includes everything from possible punitive fines to the damage to libraries’ reputations as a whole. [8] ranjan, brandy. (2018). re: questions re: oclc wise (context for code4lib journal editorial). [email]. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – cost per use in power bi using alma analytics and a dash of python mission editorial committee process and structure code4lib issue 59, 2024-10-07 cost per use in power bi using alma analytics and a dash of python a trio of personnel at university of oregon libraries explored options for automating a pathway to ingest, store, and visualize cost per use data for continuing resources. this paper presents a pipeline for using alma, sushi, counter5, python, and power bi to create a tool for data-driven decision making. by establishing this pipeline, we shift the time investment from manually harvesting usage statistics to interpreting the data and sharing it with stakeholders. the resulting visualizations and collected data will assist in making informed, collaborative decisions. by lydia harlan, kristin buxton and gabriele hayden introduction/ background at university of oregon (uo) libraries high level collections decisions are made by a collection managers group consisting of representatives from acquisitions, electronic resources, subject librarians, and access services. facing a possible large cut to the collections budget in fiscal year 2022, this group started talking about how to decide which serials to cut. cost per use (cpu) was raised as one starting point for making data driven decisions. historically at uo, usage data has been stored in manually created spreadsheets with data compiled from all vendors, while cost data has been stored in alma. when small package cancellations were on the table in previous years, cpu was hand generated for those titles. the employees who manipulated the spreadsheets were understandably reluctant to hand generate cpu for the over 78,000 rows of usage numbers for combined vendor statistics at that time. two of the authors, members of the collection managers group, began exploring alternative options. we knew that there were commercial products that would provide this data but didn’t have buy-in from the necessary stakeholders to purchase a tool. instead, we pulled in a third colleague with data experience to help create our own tool. we wanted to provide cpu for as much of the continuing resources collection as we could in ways that would be easily understandable for the subject librarians who would need to use the data. literature review as harker ( [1] 2022, 355) has observed, the cost per use metric “has become ubiquitous and nearly universal for evaluating resources, especially renewable resources such as journals and databases,” because it is straightforward and easy to grasp. most articles discussing cost per use address controversies regarding whether cost per use should be used, the specific metrics and processes that should be included in a cost per use analysis, or limitations to the specific metrics (such as counter 5) currently in use. less common are articles describing the technical details of reproducible pipelines for gathering and deploying cost per use analyses. this may reflect the challenges of using proprietary software (such as power bi) to work with data protected by license agreements. the few articles that did address technical access questions were very helpful to our project. pesch ( [2] 2015) offers a basic primer for using sushi and counter, while boardman and thompson ( [3] 2022) describe using alma analytics to programmatically query sushi apis and visualize counter statistics, documenting that it could be done even if they did not share the technical details needed to execute the project. this paper aims to fill a gap in the scholarship by offering a clear, step-by-step description of what we did that may be of use to others. approaches to the pipeline in the first iteration of this project, we created a cpu table in excel by merging spreadsheets of usage and cost data. (figure 1) figure 1. we performed lookups on title and issn for journals where we had both usage numbers and payment history. we were able to calculate the cpu for many of our journals and created graphs to make the data more visual. we ended up not needing to make major cuts to our journals budget at that time, so this tool wasn’t needed immediately. we decided there was still value in improving our report. this excel-only version had two major limitations: 1) it was challenging to update with new usage numbers and 2) sharing the underlying usage along with the visual meant sharing a very large excel file vulnerable to user error, easily unintentionally manipulated to show false results. we next decided to import the excel files into microsoft power bi. (figure 2) figure 2. power bi allows easier update of datasets, provides additional visualization options, and offers a way to share visuals and tables of data without needing to share the underlying data files. we successfully translated the data transformations we had conducted in excel into power bi. however, this version still relied on the laboriously hand-created excel usage data file. the next step was to gather usage statistics dynamically by querying the sushi apis of our vendors. we experimented with making sushi calls directly from within power bi and were successful in getting the process to work with a static table, but we couldn’t find a way to continually add the next month’s usage in a way that would store the data properly. (figure 3) figure 3. we learned that this is a deliberate limitation of the power bi software, which is not designed to function as a data source but instead to draw on more stable data sources such as sql databases. we would have needed to store the data outside of power bi, which negated the elegance of establishing sushi calls from within power bi. we finally settled on using the sushi harvester in alma to gather and store our data. (figure 4) figure 4. alma is designed to both harvest and store usage data, and we knew we could get the information out of alma using alma analytics’ sql database. however, if we wanted a fully dynamic report, we would then need to query alma to retrieve cost and usage reports on a monthly basis. we explored querying the alma analytics api using power query (m) within power bi but found it very difficult to troubleshoot errors. (figure 5) figure 5. we decided instead to use a python script embedded in power bi to query the analytics api to retrieve both cost and use data. we then used power bi to transform the data, calculate and visualize cost per use, and distribute our dashboard to internal stakeholders. in this final model sushi api accounts are set up for as many vendors as possible in alma, which harvests the tr_j1 reports monthly. alma analytics reports are established to capture cost and usage statistics. python scripts embedded in a power bi m query retrieve each report via an api and loop through the data using a resumption token (required for data of more than 1,000 rows). the python code pulls in the complete dataset each time we want to update it, which we can do by simply pressing the refresh button in power bi. more m code simplifies the data. dax code then merges the data from the two datasets to calculate cost per use. power bi’s visualization tools are used to create a dashboard that can be shared with stakeholders. if you want to implement a model like this one in your system, full step-by-step details can be found in appendix a. the dashboard visualized our current dashboard includes four views. the first uses a bar chart to display journal titles and their cost per use with the ability to filter by fund and fiscal year. (figure 6) the second provides a table of usage data across multiple fiscal years, filterable by fund and searchable by platform, publisher or title. (figure 7) our third display is a table of titles that might be cheaper to obtain by ill than subscription, based on the usage patterns and average ill cost. (figure 8) finally, we have an about page that provides information about the dashboard. figure 6. figure 7. figure 8. cost per use defined because this is a proof of concept for our technical pipeline, it uses a very simple definition of cost per use, as the number of uses as defined by counter tr_j1 reports divided by the annual amount we paid for the product — counting only journal use and excluding databases and ebooks. we do not, for example, address any of the issues of open access articles discussed by jabaily ( [4] 2020) or addressed by the unsub [5] product. however, the nature of power bi as a tool that can aggregate and visualize data from a variety of sources means that we hope that this pipeline will be the basis for creating more robust analyses. in the future, for example, we might adjust for open-source access, include ebooks and databases, or incorporate elements such as tracking the journals our faculty and graduate students cite or publish in. assumptions and limitations in the process of writing up this project we had to distill it down to steps that are communicable and helpful to others. in reality, this was an iterative experience with several dead ends. we frequently tested elements as proof of concept before investing time in developing those same elements on the larger scale. it’s possible there are more efficient approaches, especially related to the normalization of titles and issns. we decided to stick with what was working and not get bogged down in perfection, in the spirit of getting it done. we know there are steps we’d like to improve, some quirks that need troubleshooting, and plenty of opportunity for additional report types and other upgrades, as well as regular ongoing maintenance. we welcome feedback and inquiries via email or via issues on our github repository. there are gaps in the data where publishers don’t provide sushi access, or counter 5 formatted reports, or don’t do so in a timely fashion. additionally, we have not yet systematically filled in gaps between the beginning of the existence of counter 5 and the initiation of sushi harvesting within our instance of alma. at times the titles or issns don’t line up perfectly, or we have usage reported from multiple platforms. we removed the gale and jstor aggregator platforms from our data, and don’t currently have a way of evaluating this usage. we do not include print usage of our collection at this time. in our initial excel model print usage was very low and usage was not of recent issues, so we determined that including print usage would not add value to our model. we made several assumptions along the way that led to the data being useful but not always completely precise. for example, if we don’t have a payment for the current year, we base our calculation on the previous year’s payment, even though it is likely not the same. we also have instances where the calculated cpu is zero. this includes cases where the title and/or issn aren’t a perfect match between our usage and cost data, journals that we pay for as part of a package but receive usage on a per-title basis, and journals only available on platforms we removed from consideration because they’re primarily backfiles, like jstor. in short, the data is still messy, and while we have plans for how to fix some of these issues, it will never be perfect. power bi’s visualization can be deceptive, even seductive. we think our model looks pretty, but we still must put our thinking caps on to be sure that those attractive charts mean what we think they mean. each of our displays is meant to be a starting point for cancellation decisions rather than a rigid list of things that must be cancelled. discussion and future work we set up this model using only the tr_j1 report, and once we gain some experience consulting the model for assistance with deselection decisions, we plan to add more report types. we’re interested in adding historical usage data by uploading counter 4 stats to alma in order to add more years into the model. we may also consider additional calculations beyond cpu that might be useful for our collection assessment needs. the way these reports are written, we shouldn’t have to make major changes until there are significant changes with either counter or alma analytics. this is intentional, so that we can keep the pipeline running with as little maintenance as possible. power bi lets us share out a dashboard that any of our colleagues can use to see the visualizations and tables of data. we have yet to use this model for significant cuts, and surely that experience will teach us a lot about what else we need to do to improve this tool. in the meantime, we have sent the model out to our collection managers and subject librarians for review, feedback, and to begin using it to help inform decisions. we encourage our colleagues to use the data as a starting point for collections decisions, rather than as the single tool to make the decisions. along with the data we’ve shared information about the limitations of that data. so far, we have used this model to extract statistics for administrative reports, and as a starting point for database assessment. visualization can help us analyze the data to uncover patterns, trends, and correlations that might not be apparent from a spreadsheet alone. these deeper insights can help us make better decisions. by automating the process, it makes year-over-year comparison more achievable. it also sets us up for expanding the types of reports to intake and types of content to analyze in a reproducible way. with regular updates, the tool will grow more useful over time as gaps are filled in. we expect this tool, and permutations of it, will help our collections management team make data-driven decisions to meet the collections and access needs of our students and faculty. notes [5] unsub: https://unsub.org/ references [3] boardman, erika, and jennifer france thompson. 2022. “visualizing counter metrics with sushi: exploring alma analytics for e-resource evaluation.” serials review, september, 1–3. https://doi.org/10.1080/00987913.2022.2120328. [1] harker, karen r. 2022. “the depths of cost-per-use: historical context and applications.” library trends 70 (3): 355–86. https://doi.org/10.1353/lib.2022.0000. [4] jabaily, matthew j. 2020. “recalibrating cost-per-use: implications of counter release 5 and unsub.” serials review 46 (4): 292–99. https://doi.org/10.1080/00987913.2020.1850040. [2] pesch, oliver. 2015. “implementing sushi and counter: a primer for librarians: edited by oliver pesch.” the serials librarian 69 (2): 107–25. https://doi.org/10.1080/0361526x.2015.1063029. about the authors lydia harlan is the budget analyst for collections, discovery, and digital strategy at university of oregon libraries. a librarian by training and budget analyst by circumstance, she holds an mlis from san jose state university and a ba in film from emerson college. kristin buxton is head of science liaisons and librarian for computer science, mathematics and physics at university of oregon. gabriele hayden is head of data services and librarian for research data management and reproducibility at the university of oregon. she holds a phd in english from yale and a ba from reed college. appendix a a step-by-step guide to building an alma to power bi pipeline for cost per use establish prerequisites to replicate our approach, you will need access to, knowledge of, and some skills in alma, alma analytics, counter 5, apis, sushi, python, and power bi. power bi desktop requires a windows pc. it is helpful to have some elementary familiarity with programming with sql in alma analytics and m and dax languages in power bi, in order to make adjustments that work better for your needs and workflow. set up vendors with sushi api in alma with tr_j1 reports types enabled follow the “managing sushi accounts” section of ex libris documentation “managing counter-compliant usage data”. [1] set up vendors manually in alma. identify administrative usernames and passwords for the content providers and the next layer of credentials for sushi: the customer id, requestor id, and occasionally, api key. please note: some sushi credentials can be found on the content provider’s website, but others cannot. it may be necessary to contact customer service to obtain them. many credentials changed during the switch from counter 4 to 5. credentials can change at any time without notice. establish the base url for the sushi api string. please note: commonly used vendors are already established in our consortial instance of alma, and most of the time the base urls were correct. if the “vendor url” doesn’t work in alma, experiment with the “override url” field. one may need to add “/sushi”, “/sushi/”, “/reports”, or “/reports/” to the end of the base url to pull a report successfully. “requester” id is spelled with an “e” in alma and is spelled “requestor”, with an “o” in counter api strings. this is an important detail if you test the api strings in a browser. under “usage report types” in alma, click “add report type” to see the list of report types. select tr_j1 to follow our example. select “test with response” to test whether the connection is working and see an example of the harvest that alma returns. set up auto-harvest of sushi in alma follow the “managing sushi harvesting” section of ex libris documentation “managing counter-compliant usage data”. [1] once the vendors are set up, tested, and verified, enable the auto-harvest. select a harvest date. after experimenting with different alma preset dates, we settled on the 18th of the month to collect new usage reports. enable the function to receive the monthly sushi harvest report by email once the harvest has been completed. the monthly sushi harvest report will list completed and failed attempts, with errors. handling errors is described in step 12.a. this email signals that the power bi model can be refreshed to intake updated usage statistics, described in step 12.c. create an alma analytics report on costs experiment with ex libris’ out-of-the-box reports or create your own. we did not find an out-of-the-box report that met our specific needs. to follow our example, use library name, fiscal period description, transaction expenditure amount, fund code, title (normalized), issn, and po line reference details including item description and filter on “continuous” continuity and an “active” status. item description provided the best results for us to be able to match titles with normalized title in the usage data subject area of analytics, but this may vary with your cataloging workflow. edit the column formula of item description to turn all item descriptions into lower case, like so – “lower("po line"."item description“) – to begin normalizing the data. removing proceeding articles occurs in a later step, in power bi. it is also an option to lower the case in power bi to do all the normalization steps in one environment. this may be the preferred method if analytics formulas aren’t familiar. write both analytic reports to display the fiscal year as “fy-xxxx” to enable matching between tables. test the report and adjust as necessary for your data and institution. create an alma analytics report on counter 5 tr_j1 use we did not find an out-of-the-box report that met our specific needs. to follow our example, use tr_j1 – unique item requests, tr_j1 – total item requests, usage date fiscal year, usage date year, usage date month key, usage date month, usage date quarter, normalized title, normalized issn, normalized eissn, origin issn, origin eissn, normalized publisher, publisher, normalized platform, and load file counter report type. this report uses the normalized title, so no additional transformations are necessary. write both analytic reports to display the fiscal year as “fy-xxxx” to enable matching between tables. test the report and adjust as necessary for your data and institution. obtain permissions to use the alma analytics api key for acquisitions work with your institution’s alma developer to get permissions and the api key to use the “retrieve analytics report” api, with secrets held in the alma developer network. follow gandalf’s advice to “keep it secret, keep it safe.” [2] build the path for each analytics report with the root url, analytics report path, and acquisitions api key. our path begins with https://api-na.hosted.exlibrisgroup.com/almaws/v1/analytics/reports?path= but it may vary for your alma instance. open either the cost or use report in analytics. in the browser, copy the portion of the link for the analytics report that begins after the equal sign. add the copied text after the equal sign of the path you are building. notice that when you copy and paste, blank spaces are replaced by “%20”. (figure 1) figure 1. add “&apikey=” and then the api key for acquisitions to the end of the path. repeat the steps in 6.b to build the second report url. test each report. there are many options for testing apis, including putting the whole string in the browser. another way to test this is to set the api as the data source in power bi. review the data. if your data is more than 1,000 rows, you will need to use a resumption token to retrieve the full set in power bi. we were unable to get the resumption token to work using power query (m) directly, so we used a python script to retrieve the report and pass in the resumption token, as described in the next step. write a python script to query the alma analytics api, including resumption tokens to retrieve more than 1,000 results we based our python query on a well-documented set of instructions for how to query the open alex api created by eric schares and sandra mierz. [3] feel free to copy our code, which is available in several different versions. [4] because the alma analytics api has access to patron information, we maintained a high level of security. we initially did this by creating a python script tracked in github that used the configparser library to pull in our secrets (api key and report paths) from a separate config.ini file not tracked in github. we recommend testing your python code outside of power bi using your preferred python interpreter with debugger since power bi doesn’t have a python debugger built in. however, because power bi does not allow the python code to consult other files (such as a config.ini file) we manually created a version that no longer used configparser to reference a separate file with secrets, but instead included all secrets hard coded into the python script. in the next section, we discuss how we added this hard-coded script to power bi, and how we modified it to allow each secret to be a power bi parameter. establish power bi connection cost and use reports using python script to add a python script to power bi, first follow the instructions in microsoft documentation to install python on the desktop (a virtual environment is recommended), import any necessary python libraries (we used pandas, xml.etree.elementtree, requests, and time), configure python access in power bi, and enable permissions. as of this writing, all python data sources need to be set to “public” to work with the power bi service. once permissions are configured, in power bi desktop, select “get data” and scroll down to select “more” to open up a new box. scroll down to select “python script” and then click “connect”. paste the python script in the text box, then click “ok”. if the script runs correctly, you will be prompted to load data into a power query table. the first applied step for this data will be “source,” and a gear icon to the right of the step allows you to open and view the python script as originally entered. to see the python script as it appears loaded into a power query (m) script, open “advanced editor” in the power query ribbon. note that power bi has translated the script by wrapping it with a command to run in python and adding #(lf) where each line ending was in the original python. we recommend turning the hard coded secrets (api key and report path) into power bi “parameters” that can be more easily updated by those less proficient with python or m. to do this, first follow microsoft’s instructions to create your parameters. for our code we created four: url, api_key, use_data_path and cost_data_path. be sure to specify data type “text” and enter each parameter with quotation marks around it. for example, the api_key parameter will look something like “k3lk2ke4536kjklk4” (quotes included). now return to the “advanced editor” window in power query and replace the hard coded api key, which will be something like “k3lk2ke4536kjklk4”, with “& api_key &”, where api_key is the name of the parameter, and the replacement code begins and ends with double quotes. to see our python code in the power bi context, open the power bi template (.pbit) file available in our github repository, https://github.com/uodataservices/cost_per_use/. [5] if no appropriate alma analytics reports are configured, enter the real url, api_key, use_data_path and cost_data_path values. if not, enter any placeholder value into the fields (like “test”) to access our power bi model. either way, click “load”. you may need to give permission to power bi to run the script. there will be a delay as power bi tries to make the api calls. if parameters have “test” entries, you will see a popup window warning of python script errors. go ahead and close that error box. despite more error messages, you are now in our power bi file and can explore every step of our data transformations in power query (m) and dax. find the python code by choosing “transform data” and then clicking on the “advanced editor” for either the usage_data or cost_data tables. the following steps will explain how we worked with our data once it was imported into power bi. you can look for these steps in the power bi environment to go with the steps described. match cost and use data on issns and titles using m and dax languages once a connection is established, transformations will need to be done in power query using the m language to open the tables, normalize data, and prepare it for matching. starting with the cost data, open the data tables by clicking on the arrows that point simultaneously left and right until all the column carrots point down and you have drilled down to the columns so that they match the alma analytic on costs. (figure 2) figure 2. you may need to change data types (e.g. convert numbers to currency), remove unnecessary columns, rename columns for clarity, etc. so that your data makes sense and functions well in power bi. this is an interactive process that you will need to customize for your data and needs. recall that we lowered the case of the item description in alma analytics. at this point use m code in power query to remove the beginning “the” and “a” to further normalize and match these titles with the titles that will appear in the use table. code for removing “the”: = table.replacevalue(#"renamed columns", each [title normalized], each if text.startswith([title normalized],"the ") then text.removerange([title normalized],0,4) else [title normalized], replacer.replacetext, {"title normalized"}) there may still be differences in the cost table versus usage table between “and” and “&”, punctuation in the middle of a title, and other unknown differences. iv. the field issn contains multiple issns. there were as many as six issns, but we decided that two would be enough to match on. we split the column by a delimiter to separate the first two issns into two columns. code for splitting by delimeter: = table.splitcolumn(#"replaced value", "issn", splitter.splittextbydelimiter("; ", quotestyle.csv), {"issn.1", "issn.2"}) optionally, add a conditional column for fund code display and group fund codes by broader category to make this information more digestible to subject librarians. anecdotally, our funding schema is more complicated than most libraries, so you may not need this step. move to the usage data, and open up the data tables until you have drilled down to the columns so that they match the alma analytic on use, like in step 8.a.1. normalize titles here as well, removing the beginning “the” and “a” so that they will match with the titles in the cost table. again, you may need to change data types, remove columns, rename columns, etc. so that your data makes sense and functions well in power bi. this is an interactive process that you will need to customize for your data and needs. navigating out of power query, and into the tables, at this point you should have one table for cost and one table for use. our table for cost has all the columns we built in the alma analytic on cost as well as the split issn columns and the fund code display column. create a use summary table. cost data is available per fiscal year, while usage data is available aggregated by month. to simplify joining information, we summarized usage data in a new table and matched the summaries with cost data on issns or normalized titles. here is the dax code for the usage summary table including the sum of total requests: usage_summary = summarize('usage_data', usage_data[usage date fiscal year], usage_data[normalized title], usage_data[normalized platform], usage_data[normalized publisher], usage_data[normalized eissn], usage_data[normalized issn], "total requests", sum(usage_data[total requests])) with the usage summary table established, match on issn and title. we started with issn1 of the cost table and matched it against the normalized eissn of the use table. here is the dax code: issn1 by normalized eissn = calculate (sum ( 'usage_summary'[total requests] ), filter ( 'usage_summary', and(and('usage_summary'[normalized eissn] = 'cost_data'[issn1], 'usage_summary'[usage date fiscal year]='cost_data'[fiscal period description]), if(and('cost_data'[issn1] = "", usage_summary[normalized eissn] = ""), false, true))) we made a new column for issn2 by normalized eissn, and so on, as well as title by title. because of data irregularities, we chose to calculate cost per use in a fuzzy way using the most expensive year and the highest use year. it will not be exact for data in a specific year. since this information is meant to discover outliers, this level of detail is sufficient for our needs. to calculate cost per use we added columns for maximum use and maximum cost per use: electronic use max: electronic use max = round(max('cost_data'[title by title],max(max('cost_data'[issn1 by normalized eissn],'cost_data'[issn1 by normalized issn]),max('cost_data'[issn2 by normalized eissn], 'cost_data'[issn2 by normalized issn]))),0) the employment of the round function is to resolve troubles with a fraction of a use. electronic max cpu: electronic max cpu = if([electronic use max]=0,0,'cost_data'[transaction expenditure amount]/[electronic use max]) recognizing that borrowing material is also an option in libraries, we included a formula for the cost beyond ill threshold, and a column for how that formula translated into whether to consider ill instead of subscription. this is our formula for cost beyond ill threshold, assuming documents through reprints desk cost $35 and we can request an item 20 times before having to pay for it [5]: cost beyond ill threshold = if('cost_data'[electronic use max]-20 the consider ill column is a simple true / false statement: considerill = if(and('cost_data'[cost beyond ill threshold] < 'cost_data'[transaction expenditure amount],cost_data[electronic use max] > 0 ), true, false) create a table for ill vs subscriptions so that it can be shown as a visualization. here is the code: ill_vs_subscription = summarize('usage_summary', usage_summary[normalized title], usage_summary[normalized platform], usage_summary[normalized publisher], usage_summary[normalized eissn], usage_summary[normalized issn], "max yearly requests", max(usage_summary[total requests])) create visualizations based on cost and use the point of all of that data wrangling was to be able put together a cost per use visualization, and now that all that hard work is done, the visualization is relatively straightforward. for cost per use, we chose a stacked bar chart, used title normalized for the y-axis and the sum of electronic use max cpu (step 7.d.ii) for the x-axis, and added the fund code display (which we shortened to fund) as the legend. we added slicers for fund and fiscal year. the display can be filtered by these slicers, or it can be searched by title. (figure 6 of main paper) for a closer look at the usage data, on the second tab, we created a table drawn from the usage summary table’s normalized title, total requests, and usage data fiscal year, and added in a slicer for fund. this table can be filtered by fund or searched by platform, publisher, or title. (figure 7 of main paper) the third display is a table of titles that might be cheaper to obtain by ill than subscription, based on the usage patterns and average ill cost. (figure 8 of main paper) the last tab is an about page, covering the highlights of what is and isn’t included in the data, known limitations, and the process overview. publish the power bi web dashboard/report so you can share it with colleagues in the power bi cloud app, create a workspace where you will share the model. name the workspace, describe it, and add the people who you want to have access to it. in power bi desktop, click “publish” and select the workspace where you want to upload it. navigate to the report in the workspace and then click the “share” button to share the report colleagues. perform ongoing maintenance continually refresh vendor usernames and passwords, harvest credentials, api keys, changes to platforms and base urls as needed. there are a variety of errors to work through on the monthly sushi harvest report, ranging from the vendor’s usage dates not being available yet, to failure to connect, to retrieving unwanted reports. it is helpful to have alma’s monthly sushi harvest report emailed the person managing the accounts to identify and troubleshoot errors. sushi errors are described by project counter. [6] alma / sushi errors are well described by carli. [7] add new vendors as they enable counter 5 reporting. monthly, after alma performs the sushi harvest, refresh the data in power bi. open the power bi model in power bi desktop click “refresh” to bring in both the updated cost data and usage statistics. click “publish” to overlay the currently published model with the updated one. click “save” to save the updated workbook. a message will pop up about overlaying the currently published data set. agree, and click “replace”. appendix notes [1a][1b] “managing counter-compliant usage data”: https://knowledge.exlibrisgroup.com/alma/product_documentation/010alma_online_help_(english)/020acquisitions/090acquisitions_infrastructure/010managing_vendors/managing_counter-compliant_usage_data [2] see the movie “the lord of the rings: the fellowship of the ring” [3] openalex-citedreferences: https://github.com/eschares/openalex-citedreferences [4] available at our project github repository (https://github.com/uodataservices/cost_per_use) and archived in zenodo at https://zenodo.org/records/10426231. [5] this formula is based on a formula created by our colleague david ketchum, director of access services at uo libraries. [6] appendix f: handling errors and exceptions https://www.projectcounter.org/appendix-f-handling-errors-exceptions/ [7] alma sushi harvesting status examples https://www.carli.illinois.edu/products-services/i-share/electronic-res-man/sushierror subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – using low code to automate public service workflows: three cases mission editorial committee process and structure code4lib issue 52, 2021-09-22 using low code to automate public service workflows: three cases public service librarians without coding experience or technical education may not always be aware of or consider automation to be an option to streamline their regular work tasks, but the new prevalence of enterprise-level low code solutions allows novices to take advantage of technology to make their work more efficient and effective. low code applications apply a graphic user interface on top of a coding platform to make it easy for novices to leverage automation at work. this paper presents three cases of using low code solutions for automating public service problems using the prevalent microsoft power automate application, available in many library workplaces that use the microsoft office ecosystem. from simplifying the communication and scheduling process for instruction classes to connecting our student workers’ hourly floor counts to our administrators’ dashboard of building occupancy, we’ve leveraged simple low code automation in a scalable and replicable manner. pseudo-code examples provided. by dianna morganti and jess williams introduction what is low code? ifttt, short for “if this, then that”, launched in 2011, and became one of the first major forms of massively accessible low code or automation. they created and crowd-sourced “recipes” (now called “applets”) that allowed users to connect multiple interfaces to create a trigger and a result. for example, to get an email that tells you to bring your plants inside when it’s forecasted to freeze, you would just choose the icon for this applet, and sign into weather underground and tell it your location. the applets have matured now to performing multiple actions and integrating more deeply into apps, but ease of use has remained their priority. since 2011, a number of similar consumer-level low code apps have emerged, the most famous being google home, apple homekit, zapier, and microsoft’s power automate. these platforms have created replicable code with graphical user interfaces to make it easy to point, click, and leverage automation to make life or work easier. programmers work behind the scenes creating these low code solutions for consumer use. someone has already written an applet so that consumers can use their phone’s gps to trigger their home’s porch lights and air conditioners to all turn on when they pull into their driveway–users just need to click a few buttons and give permission to get it working across all their devices. low code is the use of a graphical user interface to take pre-written code and make it easy to use for just about anyone to leverage automation. the smart home market isn’t the only place low code has transformed our lives; the business world may have seen even more benefits than the home market. the low code solutions presented in the three cases below all leverage microsoft power automate. power automate (formerly known as “flow” and found at flow.microsoft.com) is a low code way to link microsoft products and other products and apps. it is intended for business solutions such as sending customized emails to customers based on their responses to a survey, or going through an approval workflow when a new document is uploaded to a specific folder on sharepoint. the example cases given below demonstrate how the team’s knowledge of the platform grew and we iterated over time. problem: mask mandate enforcement although the texas state university’s main library building never closed during the covid-19 pandemic, the students had been sent home from their campus residences in spring 2020 and all classes moved online, which meant that the library had a very low floor count for the remainder of that semester. the university allowed students to return to campus in fall 2020, and the library anticipated a great increase in foot traffic as a result. we had state-wide, local, and campus mask mandates as well as occupancy requirements for social distancing, and as stewards of a public building we needed a way to enforce those for our seven-story heart of campus. we explored some turn key solutions such as body heat occupancy monitors or all seats and spaces requiring student reservation, but ultimately the infrastructure (needed outlets and data drops for monitors or for reservation kiosks) didn’t exist yet and could not be implemented in a timely enough manner to make these a good solution. we also anticipated that mask compliance would be a greater issue than occupancy, and no turn key solution existed for that. peers who had also remained open shared some good practices such as constantly surveilling for mask compliance on each floor, but this solution wouldn’t work for us. we needed another solution. constraints the top constraint surrounding a solution to this problem was time. we couldn’t take the time to find the best solution, because the students would be returning in fall. however, we began this work during the summer and so we had a little time to experiment.` we started with an assessment of the resources we had at hand. we had the microsoft environment, student workers, and staff with safety at top of mind. we also had our own time (that is, the authors’) – we aren’t afraid to tinker with technology, but we also aren’t “coders”. we knew we could start with student workers doing hourly floor counts and checks for mask compliance, but we didn’t want to put the onus of enforcement for either concern onto student workers. so we needed some mechanism to communicate their information to managers who would do that work. student workers used a paper data collection form, a clipboard, and a handheld tally counter while walking the floors then transposed the data into a microsoft form when they returned to their work station. our first deployment of a low-code solution used microsoft power bi, which can automatically retrieve the information from the excel spreadsheet generated by the microsoft form. this initial solution involved teaching all the managers how to sign in to power bi, set up alerts for when they needed to respond, and teach them how to get those alerts on their mobile devices. there were many opportunities for user error in this system. one of our administrators suggested creating a team in microsoft teams for the enforcing staff to all receive the same alert from power bi using a flow in power automate, and from there we began exploring microsoft power automate. in this version 1.1 of our workflow, we only used power automate to take the alerts from power bi and send them to the team. but in exploring the template flows we were quickly able to envision a whole new automation that was much more efficient. the newly discovered template flows in power automate allowed us to remove power bi from the workflow, which was a positive thing as power bi created a delay in response. we retained the power bi dashboard to view the data, but it was separate from the response automations so that they could be done swiftly. this brought us to the 2.0 version of our workflow which we present here. solution (in workflow chronology) while we worked to iteratively improve the process over time, we can present the final product here. we ceased use of this process in may 2021, when the mask mandates were lifted by our state governor. find the workflow below listed roughly, with more detailed pseudocode demonstrating the low code process in power automate below. a student worker conducts an hourly floor walk and gathers data on occupancy, mask noncompliance, and any additional notes. a chart assists them with determining whether the floor is in “red”, “yellow”, or “green” based on predetermined cautionary levels of both mask compliance and occupancy (a proxy for social distancing) they update the alkek library’s entry-way floor status chart that assists customers in making a decision about the safest place to study they enter the data into microsoft forms. [student worker is done] power bi pulls the microsoft forms data to update a live dataset and dashboard of floor counts for library decision-makers to study trends and respond to concerns power bi also publishes to a web page summarizing the previous 7 days’ statuses to allow customers to make informed study space decisions microsoft power automate has a flow that detects every form entry and compares against criteria for concern if a floor meets a threshold for concern, an alert is sent to the “covid-19 response team” in microsoft teams. response team members communicate amongst each other in the discussion thread to determine who will respond. if response team members expel a customer for mask noncompliance, they fill out another microsoft form. the powerbi dashboard that tracks data and trends also pulls data from this form to track expulsions. libstaffer (a scheduling tool from the springshare company) creates a transparent schedule of who is responding to alerts from the floor counts. libstaffer updates a live schedule on an internal webpage so that student workers can see and contact responders about any concerns they observe this webpage also has complementary information such as links to the counting form, suggested scripts for giving warnings (optional for student workers), phone numbers for supervisors, and more. libchat has an sms text phone number for customer complaints about mask violation or other safety concerns, which come in almost daily. a few of us monitor this and alert enforcement staff via teams to respond. for the microsoft power automate section of the workflow, we started with a blank flow and used the “automated cloud flow” option. this means we wanted an automation triggered by an event (as opposed to an automation where we pushed a button to trigger it). for this automation the tasks are triggered by the submission of new form entries (the floor counts students entered). we used a template for sending a teams message when a new form response comes in, but made it so that it only takes action when certain conditions are met (that is, the occupancy or mask compliance are at cautionary levels). “when a new response is submitted” is the template for triggering a flow from a form entry. the creator will then choose the form from the drop-down list of all their office 365 forms. “get response details” is a necessary template for gathering the data from the form entry. “condition”: if the “floor” was “1st”, and: if the “occupancy” was more than a or if the “mask noncompliance” was more than b or if the “floor was “2nd”, and: if the “occupancy” was more than c or if the “mask noncompliance” was more than d or if the “floor was “3rd”, and: if the “occupancy” was more than e or if the “mask noncompliance” was more than f or if the “floor was “4th”, and: if the “occupancy” was more than g or if the “mask noncompliance” was more than h or if the “floor was “5th”, and: if the “occupancy” was more than i or if the “mask noncompliance” was more than j or if the “floor was “6th”, and: if the “occupancy” was more than k or if the “mask noncompliance” was more than l or if the “floor was “7th”, and: if the “occupancy” was more than m or if the “mask noncompliance” was more than n if yes: “post a message in teams” is the template to do so, and since we got all the response details in step 2, we can include all the form data. the message template supports basic html so i was also able to add a link to report any expulsions to another form. if no: do nothing. problem: instruction scheduling, first year writing classes in pre-covid times instruction for the university’s first year writing classes, particularly english 1320, was executed through a high number (around 125 ) of one-shot sessions. in academic year 2020-2021, the instructional content was moved to a self-paced tutorial accessible through the campus lms, canvas. however, many of the english instructors still requested synchronous library instruction though zoom. around the same time, the department primarily responsible for teaching these lower-division undergraduate courses lost team members and was left with one librarian and a handful of volunteer librarians from another department. to meet the instructional need, the library created a pseudo-flipped classroom model: instructors were given the option to schedule a question and answer session with a librarian on zoom after their students had completed the online tutorial content. for assessment purposes, we needed to know two sets of information: 1) which instructors were assigning the online tutorial only and 2) which instructors also wanted a live q&a session. the latter set required more details for coordination and scheduling; we also wanted a system to help ensure that the students were completing the online tutorial before the live session. this automation was the solution to these needs. constraints the group of volunteer librarians came from three different departments in the library, though they were also all part of a functional library instruction team. this team had been working collaboratively using an existing microsoft sharepoint site and teams since spring 2020 and the start of the covid-19 pandemic. continuing use of microsoft products made the most sense organizationally; it also limited some functionality, particularly with the ms planner components. further automation could have been added to overcome these limitations, however we also knew that this entire process itself was temporary and would be replaced by a new instruction model by the next semester. while ms forms, power automate, and planner were used in this scenario, other options are also viable. google forms, zapier, and trello is another powerful combination, in particular because trello now has utilizes built-in automation through butler. an additional constraint of this solution is that it existed outside of the library’s system for collecting instruction statistics, springshare’s libinsight, which is not interoperable with microsoft. logging the data was a manual process but also the inspiration for the following use case and iteration. solution (in workflow chronology) in fall 2021, the instruction model for first year writing classes will change and no longer require instructors to request or schedule a session. a more sophisticated iteration of this use case is presented below. the temporary nature of this workflow meant that we made the conscious decision not to tweak or improve certain aspects; suggestions for improvement are included at the end of this use case for the benefit of the reader. many ms planner features like progress status, priority, and labels were not used here but have the potential to greatly improve the user experience for the librarians. data collection using branching in ms forms the foundation for this automation is a microsoft form that utilizes branching. every instructor filled out sections 1 and 2 but only instructors requesting a live session were moved to section 3. instructors were asked to fill out one form per class section. section 1 questions course section number (text) name (text) email (text) number of students (number) due date for library tutorial (date) do you need help embedding the library guide (with tutorial) into your canvas course? (y/n) would you like to receive assessment results for your students? (y/n) section 2 instruction options (this was included as a text block before the next set of questions) the interactive tutorial provides students instruction on the following: how to use the library website, databases, catalog, and research guides how to chat with a librarian & schedule a research consultation an overview of the research process choosing a topic, core concepts, and keywords background research + library vs. google search strategies (creating queries, boolean operators, troubleshooting) source types interpreting search results page and refining the search query citations evaluating sources questions would you like to schedule a live zoom session for your class? (y/n) this session can be used to address student questions and research topics, with the librarian acting as a coach and guide; this time may also be used for the librarian to introduce content not covered in the tutorial. please make sure your students come prepared to this session for best results. branching note: if no, instructors were taken directly to section 4. if yes, the instructor was taken to section 3. section 3 schedule a live zoom session class time (text) first choice date (date) second choice date (date) how will your students prepare for the live session? (short answer) ex: by adding questions for the librarian on a discussion board beforehand additional comments (long answer) include any special topics you’d like covered or assignment details that would help your librarian prepare. section 4 complete! any questions? submit using ms power automate to create cards on ms planner board all librarians available to teach the live zoom sessions were already members of the library instruction group sharepoint site/teams group. to integrate this workflow into the existing structure, we created a ms planner board with the teams app. the primary purpose of the power automate flow was to create one card from each ms form submission/class section and then to display the data from the form in an easily available and readable format on that card. workflow overview ms form: submission ? ms planner: create a task ? delay (5 sec) ? ms planner: update task details ? send success email ?? send failure email workflow details trigger: when a new response is submitted to ms forms action: create a task in ms planner plan id selected from library instruction planner board title generated from dynamic content from two submission fields: [section number] [name] display example: 1320.255 juan doe five second delay action: update task details in ms planner pulling dynamic content from the form submission into the description field on the card: figure 1. dynamic content from the form submission added to the description field. action: using notification operation, send email send failure email (ensures the user doesn’t miss a request if the automation fails for any reason) send success email with dynamic content from the newly created planner task: figure 2. dynamic email in planner. completing workflow from ms planner board once cards were created in the incoming requests bucket, they moved from left to right across the board, creating a highly visual workflow. the buckets were: table 1. incoming requests bucket workflow. incoming requests scheduled & assigned needs assessment results sent needs added to libinsight closed a few weeks into the semester, the group of librarians met to learn the workflow and divvy up class sessions. one librarian was assigned to each card representing a session, and then that card was moved to the schedules & assigned bucket. due dates were set in order to trigger email reminders, which ms planner does by default. a checklist was added to each card to include: 1) email course instructor 2) confirm live session & zoom link 3) do the class. this workflow transparency helped the home department librarian keep track of each session and to ensure that requests were met. for courses not requesting a live session, those cards were moved directly to the needs assessment results sent bucket; live session course cards were also moved into this bucket by the assigned librarian at the conclusion of that session. due dates were added to each card, and either the home department librarian or an admin assistant would download and email assessment results to the instructor. these same individuals would move the cards though the next bucket and to completion. suggestions for improvement again, because of the temporary nature of this particular workflow, the author noted several areas for automation and improvement but opted to not to employ them for the sake of time. if using ms planner again, the author recommends: utilizing ms planner features like progress status, priority, and labels. these have the potential to greatly improve the user experience for the librarians. use ms power automate to populate a predetermined checklist, on the “update task details” action since ms planner does not have the option to copy and paste a checklist. automate the process of statistics and record keeping (see below for how we did that in the second iteration of this workflow). problem: instruction scheduling scaling up we faced a problem many of you faced most likely in 2020: we continued losing staff at at least the normal pace (if not a higher pace due to stress/anxiety around the pandemic), but we were in a long-term hiring freeze that prevented us from replacing staff we lost. specifically we lost all the staff who had provided the administrative and logistical support behind our instruction scheduling workflow. in the past, when a course instructor wanted a librarian visit, they went to our website and filled out a libinsight form. this form created an initial ‘shell’ record in libinsight and emailed an alert to our admin staff. the admin staff compared it to the subject librarian list and sent it out to the right librarian. that librarian, after they taught the class, would complete that shell record in libinsight with the statistics of duration, attendees, etc, and they would also send the faculty member a survey asking for feedback. we lost all the admin staff who had supported this process, and although other library staff could have stepped in to replace them, we were also doing that for many other jobs and responsibilities in the library. in order to show support for the staff workload and attempt to reduce some of the tasks that could contribute to the feeling of always needing to ‘do more with less’, we wanted to use some automation to reduce the administrative support necessary for this process. the scheduling described in the previous case gave us the idea of how we could scale up, and our model of starting the initial shell of a statistical record at the point when a faculty member submitted a request was a great one. that model allowed for a little bit of accountability in ensuring all our classes were recorded and accounted for at the end of each semester. a bit of reading and tinkering in power automate showed us that we could use sharepoint lists to replicate that part of the process, and we could use a series of if/then conditions to help send requests to the right librarians. we didn’t want to totally replicate the data collection form we had in the past – it was an onerous form that had been added onto over time to the point of being something everyone dreaded about teaching. so we started from scratch in determining what to ask faculty and what to ask librarians about the library instruction classes, tours, and workshops. we began by gathering every data point we needed to report out on to our various stakeholders and reporting agencies and classifying them according to whether they are necessary, are optional but useful, and which are optional and not used. with this study we were able to cut a significant amount of fields from the form, streamlining the process before automation. table 2. fields evaluated for need. necessary optional and useful optional and not used date and time room number was librarian embedded in lms? number of students anticipated desc of assignment month of instruction time (for synchronous classes) 2nd choice date department of class synchronous/asynchronous delivery number of students anticipated librarian requested (not used since we send request to the assigned librarian no matter what they chose) number of students attended assessment score (compiled elsewhere) prep time course name and number course libguide campus course level (duplicated in course number) we went through two full testing environments consisting of a unique instructor form for requesting a class, a sharepoint site for managing/recording class information, and flows to connect those and alert librarians. a small group of librarians helped test the automation over the course of about 6 months before we presented a fresh test environment to the entire group of 20+ librarians for full testing. constraints in the eng 1320 example above, the librarians were a small controlled set of people. it is easier to train 5 people to use a new platform (like teams and planner, utilized in that system) than the 20+ we’d need to train for this scaled up approach, so we prioritized using tools that the staff already knew well (email and forms and to a lesser extent sharepoint). when i had to convert some tasks to sharepoint, i focused on ease of use, since we weren’t a heavy sharepoint-using group. i moved any fields they had to interact with in sharepoint to the center of the page to catch their eye, and i hid all the fields they didn’t need to interact with. i used the sharepoint powerapps forms service to simplify the form in case any librarians preferred entering their data later. again, i hid the fields they didn’t need to input – for example, we ask faculty instructors to give us a 2nd choice date, but the librarians entering data later won’t even see that field. resources and time were a constraint in that we wanted to use only pre-approved software and environments for interacting with class information to limit the need for going to it security to approve any new tools. using google or qualtrics, for example, would have given a lot more customization options for a form, but they aren’t pre-approved for this type of use on our campus. while i’m familiar with the process of getting software approved, i also knew my timeline was a constraint as well. while time was a flexible constraint, since this was an improvement project where the existing solution worked just fine, i wanted to work on a deployment of a new solution by the next academic year. going through it security processes to approve new software is a seemingly unpredictable timeline, so i chose to be limited, then, to the options available in our pre-approved tools for this type of data on campus. lastly, we constrained the project to those apps for which microsoft power automate has pre-built flow templates (that is, it needed to be “low code”). while we may be confident enough to tinker around with simple code, this process needs to be sustainable. building it entirely using pre-built templates in a low code environment means there are faqs and tutorials online for how to use each component, and the entry level for the technology is lower. solution (in workflow chronology) the process starts with the template automation trigger for when a new response is made in a microsoft form. i then used the “conditions” to compare form entry data about what class it is to the librarians’ responsibilities to figure out who it should go to. then i used templates for “send an email” and “add to sharepoint list” to get the data to the right people and record that request in the statistical dataset. the microsoft form that serves as the trigger uses branching to ask questions based on previous answers. the first relevant question for the logic is asking what campus, then college, the faculty requesting instruction is in. from there, if needed, it will ask the department. if the faculty is from the round rock campus, though, the questions are moot since all those requests go to the same place; we don’t make the faculty fill out extraneous information. similarly once we get to date/time/location, if they’ve chosen an asynchronous format, we only ask what date they’d like it available – not time and location. when the triggering event occurs (someone fills out the form), the flow built in microsoft power automate will: check if subject equals a or b or c if yes: then fill out the stats instance – add librarian abc to field “librarian” and: email librarian and faculty with details and: stop. if no: nothing. it will move to the next condition. checks if subject equals x or y or z copy of above, with next subject librarian…. copy as needed for each librarian (we have 16 conditions – 15 individuals and 1 campus that didn’t want their requests sent to individuals) with the final librarian, it starts the same. check if subject equals aa, bb, or cc if yes: then fill out the stats instance – add librarian aabbcc to field “librarian” and: email librarian and faculty with details and: stop. if no: still fill out the stats instance – leave librarian blank and: send an email to a manager that says “class not assigned!!” (human intervention required to get this to a librarian to teach it) and: stop. we have thought of a few enhancements, and we’ll think of a few more, i’m sure. two likely enhancements are: that the process launch a calendar invitation for the preferred class date, booking the librarian and the classroom to have microsoft power automate send a follow-up satisfaction survey to the instructor after the class date. we will likely implement this before the first semester. problem: consultation request automation our final case is one that is not yet solved. we believe it is important and valuable to share the entire learning process, even those cases that are failures (or possibly just stuck in a failure stage on the way to success). while it was easy to envision a solution for faculty class instruction requests, it was harder to do so for consultation requests which can come from faculty, students, and even sometimes other staff. students, in particular, usually don’t know the colleges and departments their classes belong to, and the automated assignment to the right librarian depends on faculty knowing that information for their classes. an envisioned solution that we haven’t yet implemented might be to assign based on course prefixes instead, but that becomes more difficult when one also wants to accommodate faculty, staff, public, and non-academic student requests for research consultations also with the same form. conclusion automation inspiration knowing where to start is often the most difficult part of any endeavor, and the same can be true for workflow automation in service roles. here are some questions to spark inspiration: is there a repetitive task that you or your team dislikes doing? explore if it can be automated. do you have a workflow with a human bottleneck? investigate how you can employ conditions or branching to improve efficiency. do you have a process that consumes a lot of staff time? find a tool that manages it and then use low code automation to connect it to your existing tools or structures. this could look like: use a scheduling tool to automate research consultation booking and syncs with ms outlook or google calendar. automate checklists or daily tasks for student workers using a recurring duplication recipe. refresh your knowledge of the current tools and products you use for ways to connect them. browse your automation tool of choice for suggested workflows based on the tool. how do you know if your process can be automatable? in each of the cases presented here, we are providing examples where information is moved from one place to another. in the case of the floor counts, student workers count the number of occupants and mask noncompliance on each floor hourly. for both of our class instruction requests, as well as the envisioned consultation requests, the information consists of customer data. in all of those cases, a judgment needs to be made as to where to send the data, but that judgment isn’t reliant on human intuition or creativity – it’s enough to check pieces of the data against some criteria to determine where to send it. so what we have are the input, the automation which may have some criteria and conditions, and an output. our primary constraint of sticking with the environment and technologies that were already approved to handle our data was what determined that we use the microsoft ecosystem. if your ecosystem is google or is more open for your use case, then there is a wide variety of tools out there you can explore. any time you are plugging in a new app or tool, though, be sure to check that it meets your information security requirements for the type of data you’ll be using. for even the use cases we’ve presented, we can envision other scenarios involving: input sources ms forms google forms qualtrics survey monkey ms bookings automation tools ms power automate ifttt zapier means of output email/calendar event row in excel spreadsheet or ms list power bi dashboard or data visualization/report low code automation is often overlooked in public services, but taking the time to set up a system once saves an exponential amount of time in the long-term. in times of staffing shortages, when remaining staff are at their wits’ end of ‘doing more with less’, some automation inserted into the workflows of everyday life can help them feel supported and relieved. the possibilities for automating tasks and procedures continues to increase as the interoperability of software and applications becomes more common; public services departments in academic libraries should continue to explore these solutions in order to free up more time for their employees to provide services and produce scholarship. about the authors dianna morganti (she/her) is the interim head of research, instruction, and outreach for the texas state university libraries. she enjoys tinkering with technology and finding solutions to make work easier for the team. she’s also a project manager, a stem librarian, and a collector of hobbies. jess williams (she/her/ella) is the head of information & undergraduate services at texas state university libraries. she has loved crafting efficient systems for her team ever since her teenage gig managing a snow cone stand. she is passionate about student success, organizational culture, and creating spaces that facilitate authenticity and self-expression. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – epub as publication format in open access journals: tools and workflow mission editorial committee process and structure code4lib issue 24, 2014-04-16 epub as publication format in open access journals: tools and workflow in this article, we present a case study of how the main publishing format of an open access journal was changed from pdf to epub by designing a new workflow using jats as the basic xml source format. we state the reasons and discuss advantages for doing this, how we did it, and the costs of changing an established microsoft word workflow. as an example, we use one typical sociology article with tables, illustrations and references. we then follow the article from jats markup through different transformations resulting in xhtml, epub and mobi versions. in the end, we put everything together in an automated xproc pipeline. the process has been developed on free and open source tools, and we describe and evaluate these tools in the article. the workflow is suitable for non-professional publishers, and all code is attached and free for reuse by others. by trude eikebrokk, tor arne dahl and siri kessel, oslo and akershus university college of applied sciences; with thanks to: eirik hanssen introduction oslo and akershus university college of applied sciences (hioa) offers a publishing platform based on the open source software open journal systems (ojs).  the learning centre and library maintains the platform. after a couple of years working with open access publishing it has become apparent that many online journals are still based on a print-centric publication model. it is easy to establish an e-journal using a traditional print workflow in an academic environment, because most researchers use a word processor as their major work tool. many know a bit about copy-editing from their contact with scholarly journals, and current word processors can save documents as pdf files. that is why pdf is the standard format used in the open access journals appearing outside the professional publishing industry. we decided to establish a project with the goal of creating a new workflow for the journals using epub as the main publication format. the project succeeded, and our aim is to share  experiences and guide others planning to do the same. why not pdf? there are two important reasons why we wanted to replace pdf as our primary e-journal format. device independency we started considering alternatives to pdf after trying to read some of the journal articles on e-book readers (a sony reader touch prs-650 and a papercaster boox). pdf files do not work well on these e ink devices. there are font problems and it is hard to scale the text size. there are solutions to some of these problems, but pdf is a print format. it will never be the best choice for reading on tablets (e.g. ipad) or smartphones, and it is challenging to read pdf files on e-book readers with e ink displays (like most amazon kindle models and the sony reader). we wanted to replace or supplement the pdf format with epub to better support digital reading. this means that we needed to change the entire workflow in the journal in order to take full advantage of the opportunities provided by digital publishing. universal design and accessibility our second reason for replacing pdf with epub was to alleviate accessibility challenges. pdf is a format that can cause many barriers, especially for users of screen readers (synthetic speech or braille). for example, excel tables are converted into images, which makes it impossible for screen readers to access the table content. pdf documents might also lack search and navigation support, due to either security restrictions, a lack of coded structure in text formats, or the use of pdf image formats. this can make it difficult for any reader to use the document effectively and impossible for screen reader users. on the other hand, correct use of xhtml markup and css style sheets in an epub file will result in search and navigation functionalities, support for text-to-speech/braille and speech recognition technologies. in the last decade, accessibility and universal design have become a legal issue, both nationally and internationally. access to web content is required through article 9 in the un convention on the rights of persons with disabilities (crpd) [1] and some national anti-discrimination legislation and procurement regulations. the norwegian discrimination and accessibility act (daa) [2] authorizes universal design of ict in articles 13 and 14, and associated regulations for ict will come into force july 1, 2014.  daa defines universal design as “can be used by as many people as possible”. accessibility is therefore an essential aspect of publishing e-journals: we must consider diverse user perspectives and make universal design a part of the publishing process. as early as in 1996, the world wide web consortium (w3c) established the web accessibility initiative (wai).  wai has made technical guidelines for accessibility for all, e.g. to web content [3][4] and authoring tools [5][6]. the norwegian standardization organization (standards norway) released a standard for electronic documents in 2013.  these guidelines and standards are useful tools when establishing a new publication workflow and format for e-journals. it is difficult to apply the wai guidelines and make the e-journals fully accessible within the current workflow of e-journal publishing at hioa. some of the main requirements for universally designed e-journals that can be satisfied by the epub format are: text format compatibility with different devices, software and older versions of devices and software structured markup of titles, headings, links, notes, tables, etc. informative alternative text on graphs and essential images layout with style sheets liquid layout of font size and line space, enlarging that adjust to the screen size high contrast between text and background highlighting of text or items in at least two ways why epub? epub solves both of the problems mentioned above. epub is a reflowable format. this means that the text will always fit the screen without the need for horizontal scrolling. the user can increase and decrease the font size without any changes in page width. this accessibility quality has always been a design goal of the w3c web standards. epub is based on web standards and inherits these qualities. additionally, epub is an open standard maintained by the international digital publishing forum (idpf). on the other hand, amazon’s file format azw/mobi is a proprietary format. amazon kindle devices cannot read epub files directly, but free tools like calibre can easily convert an epub file to the native amazon kindle format. epub has other advantages as well. the current version, epub 3 from october 2011, is based on a subset of html5 and css3, making it more suitable for multimedia content than earlier versions. at present, few reading devices and applications support epub 3. epub 2 from 2007 is still the most widely used and supported version of the format and works well for most academic journals. that is why we will concentrate on epub 2 in this article. the case our goal was to change the publication format in professions & professionalism from pdf to epub. professions & professionalism is an open access journal that “invites research-based empirical, theoretical or synoptic articles focusing on traditional professions as well as other knowledge-based occupational groups approached from any perspective or discipline” [7]. the first issue was published in november 2011. the articles have cc-by licences and are typically text-based with references, footnotes and tables. some contain images, too. the current workflow follows these steps: the authors upload their articles to the ojs software. most articles are uploaded as microsoft word documents (meaning the docx format). the editors distribute these microsoft word documents for peer-review and send the comments to the authors. the authors update their microsoft word documents according to the peer reviewers’ comments and re-upload them into ojs. the copy-editor does the finishing work on layout, proofreading and reference-checking. when finished, a document is saved as a pdf file. the editor sends the pdf file by e-mail to the author, who will then check the final layout. if there are changes, the copy-editor will apply the changes and save the final pdf document that is the published online version of the article. first attempt: conversion tools at first, we thought this was going to be a very simple project requiring little effort. we would apply some of the easy-to-use doc to epub converters freely available and end up with well-formatted and functional epubs ready for publishing. our prediction was that we maybe would have to ask the journal managers to tweak a thing or two in their original microsoft word files, but under no circumstances did we expect to have to make any big changes in the journal’s existing method of editorial work. we researched what kinds of tools were available for epub conversion. the following table describes the tools we tested and some of the issues we encountered. to validate the results, we used epubcheck. software description conversion main issues calibre an open source e-book library management application that can convert e-books to different formats. calibre is unable to convert directly from docx (microsoft word), but can convert the open document format (odt). there are tools for docs to odt conversion, for example by way of openoffice or libreoffice. broken tables, separated image and captions, footnotes at end of article counting only to 9. the remaining footnotes had number 1. missing headers and footers. epub file not valid. sigil a multi-platform epub ebook editor and open source software. sigil converts the docx file to filtered xhtml and packs it as an epub file. epub file not valid. writer2epub an openoffice.org extension that creates an epub files from the word processor. install the open office extension. open the document; choose writer2epub to generate the epub file. mainly a voluntary effort. images missing. epub file not valid. adobe indesign adobe indesign has native support for epub export. create or open a document in adobe indesign. export the document to epub. epub file not valid. none of our journals actually uses indesign in their publishing process, so that would mean another tool for them to learn. also licensed, so they would have to pay for it. none of the resulting epub files were valid. because non-valid epub files give unpredictable results on different reading devices (with some e-readers even rejecting such files), none of the tools we tested seemed conducive to accessibility in our academic e-journals. new workflow since none of the existing conversion tools were sufficient, we decided to change direction and start looking into developing our own conversion workflow. we considered marking up the journal articles directly as epub, but discarded this approach because epub, like xhtml, is more of a presentation format than a rich semantic publishing format. for instance, footnotes and references are typical parts of a journal article, but there are no such elements in xhtml. because an epub file is a collection of xml files, creating our own xml-based process to generate epub files seemed possible. we also saw xml as an ideal format for long-term preservation of the articles: it is open (not proprietary), can be read by humans and machines and is software independent. input: the journal article tag suite (jats) public library of science (plos) is the flagship of open access publishing, currently publishing seven peer-reviewed open access journals. the main publishing format in the plos journals is the native web format (in this case xhtml 1.0 transitional) with lots of hypertext links, for instance in the references section (for an example article, see michel and knouft [8]). pdf and—most interesting for us—xml are alternative formats. plos uses journal publishing tag set version 3.0 as its xml source format. the national center for biotechnology information (ncbi), which is part of the u.s. national library of medicine (nlm), develops and maintains this xml application. the journal publishing tag set is part of a larger family of xml applications called nlm journal archiving and interchange tag suite. pubmed central, the free archive of biomedical and life sciences journal literature at the u.s. library of medicine, requires use of the tag set in their file submission specification [9]. when we started working on our project, we found out that the tag suite was being prepared for a national information standards organization (niso) standardization process. niso formally approved the standard on 9 august 2012, and it is referred to as journal article tag suite (jats), version 1.0 (ansi/niso z39.96-2012). this is the version we have used in markup and will refer to from now on in the text. output: epub epub 2 consists of three sub specifications: the open publication structure (ops), the open packaging format (opf) and the open container format (ocf) [10]. the resulting epub 2 file is a zipped archive of a simple file structure including some required files with metadata information about the files in the archive. a typical and simple epub file has this content: a root folder with one file called mimetype. this file always has this content: application/epub+zip. a subfolder called oebps (an acronym for open ebook publication structure, which is inherited from epub’s predecessor file format). the subfolder has the following content: all the files with the main content of the epub 2 file, i.e. xhtml for structure and content, css for presentation, and any pertaining image files. these files can be organized in subfolders or put directly into the oebps folder. the epub “root file”, which is an xml file with the suffix .opf. the file has a metadata part (described in dublin core), a manifest part (a list of all the files in the epub zip file), and a spine part (defining the linear reading order of the files in the manifest part). the complete structure of this file is described in the opf specification. a navigation control xml file (with the suffix .ncx) that works like a table of contents and will help the user navigate through the content. a subfolder called meta-inf with only one file called container.xml (this is the only required file in the folder according to the epub 2 specification). the xml file has a reference to the name of a required opf file (the epub “root file”) located in the oebps folder. overview the figure below illustrates the old and the new workflow. the lower part of the figure shows the old workflow, where we simply save the microsoft word articles as pdf and publish them online. the upper part of the figure illustrates the new workflow. we mark up the content of the microsoft word document from the old workflow as a jats xml document, and that is the only manual work to do. we generate the different publishing formats—xhtml, epub and mobi—by running a script which is attached to this article. the national center for biotechnology information (ncbi) has created a toolset of xslt style sheets for transforming jats documents to (x)html and xsl-fo. we have only implemented the epub workflow so far but will create pdf versions by way of the xsl-fo style sheet in the future. building the epub file is not part of the ncbi toolset, so that is our contribution to the production pipeline. we did this by adding xslt style sheets transforming the jats source file into the mandatory files in the epub file structure explained above. our goal was to automate the epub production process completely. the ncbi toolset provides two shells or pipelines, putting all the steps together. one of them uses proprietary saxon extensions and the other uses xproc, which is a w3c recommendation. in the following sections, we go through the steps in the new workflow used in professions & professionalism to replace the pdf format with three digitally native formats: xhtml, epub and mobi. to illustrate, we use the article medical management in norwegian hospitals by ivan spehar and lars erik kjekshus [11] from professions & professionalism. the article contains 4 tables, 2 illustrations and 56 references of different kinds (books, chapters in anthologies, journal articles, reports and phd theses). we chose the example article because of its complexity, which should provide others with multiple templates for xml markup. jats markup as mentioned, we chose jats 1.0 as the native xml application in the new workflow. all xml files, including the example article, conform to this standard (or more specifically the journal publishing tag set that is a part of jats). when we start jats markup, we have the final microsoft word version of the article. we consider it unrealistic and unnecessary to use xml throughout the work cycle. microsoft word is the daily writing tool for the article authors and the copy-editor, and we do not want to change their already efficient collaboration and communication. we copy and paste the text from the microsoft word document to the content of the root element of a jats document in an xml editor. there are many xml editors to choose from. some are parts of integrated and commercial software packages, like liquid xml studio, oxygen xml editor and xmlspy, but we chose the simpler xml copy editor, which is free software released under the gnu general public license (gpl) version 2. xml copy editor has the basic features that we needed to do most of the markup and transformations: xml editor, validation against dtds and xml schemas, and xslt transformations. it only supports xslt and xpath version 1.0, but that works in most cases. the example jats document has three parts, which are child elements to the root element: head: metadata about the article, including title, authors and their affiliations, classification and subject headings, license and the abstract. body: the main content of the article, divided into sections and paragraphs. back: this is the reference section. the markup of the head part is not very challenging, but the journal has to have some clear principles about classification and subject headings to create a taxonomy that clusters articles about the same topics together. this will help navigation on the website as the number of archived issues increases. we use a standard cc-by license on all our articles. for this, jats has a permissions element, which is a child to the head element: <permissions>  <license license-type="open-access" xlink:href="http://creativecommons.org/licenses/by/3.0/">      <license-p>this is an open-access article distributed under the terms of the creative commons attribution license, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.</license-p>  </license> </permissions> the body part is straightforward markup for anyone familiar with html. some elements have different names in jats (for example, italic for i, bold for b), but the semantics are the same. the hierarchical structure of the article is marked up using the sec element (which uses semantics identical to section in html5). sections can be nested into any number of subsection levels. each (sub)section has a title, which is the heading. paragraphs are marked up using the p element, just as in html. manual markup of tables is time-consuming, but not difficult. however, we emphasize the need for correct markup of row and column headings for accessibility reasons. the most complex markup is the reference section in the back part of the jats document. the references can either be marked up using the element-citation element, or the mixed-citation element. the difference is that element-citation is a strictly structural markup of the bibliographic description of the work, while mixed-citation is a simpler, more presentational markup of it. using mixed-citation, spacing and punctuation are preserved during transformation to other formats like html or pdf. with element-citation, the xslt style sheet is responsible for spacing, punctuation and italicization. we chose to use element-citation because of its rich semantics. the structural markup takes more effort than mixed-citation, where just some parts of the description are marked up, but the richer semantics provide more flexibility. the examples below show the differences between the two markup styles on a bibliographic description of an online journal article. first is the element-citation version that we have used: <ref id="aw2008">   <element-citation publication-type="journal" publication-format="online">      <person-group person-group-type="author">         <name>            <surname>album</surname>            <given-names>d.</given-names>         </name>         <name>            <surname>westin</surname>            <given-names>s.</given-names>         </name>      </person-group>      <year>2008</year>      <article-title>do diseases have a prestige hierarchy? a survey among physicians and medical students</article-title>      <source>social science &amp; medicine</source>      <volume>66</volume>      <issue>1</issue>      <fpage>182</fpage>      <lpage>188</lpage>      <uri>http://dx.doi.org/10.1016/j.socscimed.2007.07.003</uri>   </element-citation> </ref> as can be seen, every part of the bibliographic description is marked up with elements. on the other hand, a mixed-citation has mixed content with only some parts marked up. the dots, commas, brackets, and other punctuation are preserved through transformations. this can be an advantage when dealing with uncommon publication types. <ref id="aw2008">   <mixed-citation publication-type="journal">album, d., &amp; westin, s. (2008). do diseases have a prestige hierarchy? a survey among physicians and medical students. <source>social science &amp; medicine</source>, <volume>66</volume>(1), 182-188.   <uri>http://dx.doi.org/10.1016/j.socscimed.2007.07.003</uri>   </mixed-citation> <ref> in both cases, the citation has the parent element ref, which has the mandatory attribute id. the value of the id attribute must be a unique identifier for the reference in the xml document. this identifier is used as a cross-reference when citing the work in the article. jats has the xref element for this. the xref element can take several attributes, but we only use two of them: ref-type: the type of reference. for bibliographic references, bibr is the correct value. rid: the cross-reference to the unique identifier in the reference list. if the value of the attribute is not in the reference list, the xml document will not be valid. when citing the reference in the example, the xml markup looks like this: <xref ref-type="bibr" rid="aw2008">album &amp; westin, 2008</xref> the unique identifiers are very important, but the person doing the markup must make them up. we found a system for creating them that simplifies markup and ensures the quality of the reference list. we combine the first letters of the authors’ surnames with the publishing year, and then, if the combination is not unique, we add a letter (a, b, c etc.). in the example, the identifier is aw2008 from album, d., & westin, s. (2008). as mentioned, the jats document will not be valid if the rid attribute value of an xref does not match the identifier in the reference list. we have found several errors in the published articles this way. transformation to xhtml the process of transforming a jats document to an xhtml document using the ncbi toolset has three steps: pre-process the reference list. the toolset has xslt style sheets for two citation styles, american psychological association (apa) and nlm/pubmed central guidelines. transform the result document from step 1 to html. post-process the html document from step 2 to xhtml. in our case, we have to go through all these steps, since our main goal is to produce epub files, and epub requires xhtml 1.1 for the main content file. the journals use apa as citation style, so we chose apa transformation as the base xslt style sheet for references in the pipeline. the main xslt style sheet is jats-html.xsl, the web preview style sheet. there is also a css style sheet for the web presentation, jats-preview.css. none of them fit all our needs, so we had to customize them, just as we had to do with the apa pre-processing style sheet. we had to recreate the look-and-feel of the journal on the web and in the epub version. we have used a trial-and-error approach in the project. we succeeded in the end, eventually realizing that merging changes directly into the ncbi style sheets is not the proper way of doing it. piez calls this the ‘monolithic’ customization method [12]. all custom changes will be lost when upgrading the toolset. in his article, piez recommends ‘vertical customization’, which means creating separate xslt and css style sheets with custom templates and styles and then importing the ncbi master style sheets into them. the custom templates and styles will then override the master when needed, otherwise falling back on the master style sheet. in xslt, the xsl:import instruction does the trick; in css the @import rule achieves the same. our custom style sheets—hioa-citations-prep-apacit.xsl, hioa-xhtml.xsl and hioa-web.css—are provided with the article. the particular adjustments we have made will probably be of no use to others, but we would like to stress the importance of creating a vertical customization from the start. it will save you hours of work compared to separating them from the master style sheets afterwards. it is also worth mentioning that the reference pre-processing style sheets from ncbi are using xslt 2.0, which xml copy editor does not support. to run these transformations in tests before the final xproc pipeline, we had to download an xslt 2.0 transformation engine. we chose saxon-he. packaging the epub file the file structure of an epub file is described in the previous section titled output: epub. all the mandatory files in the file structure are either static files or results of xslt transformations of the jats source file. the xhtml version of the article is the output from the transformation described in the previous section. we use our customized css style sheet for presentation. these and any image files (gif, jpeg, png) must be copied to the oebps folder or subfolders. mimetype and container.xml are static files. only two files, content.opf (the epub root file) and toc.ncx (the table of contents) depend on the content of the particular epub file. we can generate both of these files from the source jats file using xslt. we provide two style sheets: epub-content.opf.xsl for the epub root file and epub-toc.ncx.xsl for the table of contents file. they should be applicable for any other journal using the same process as we are describing here. the xslt style sheet fetches the variable file content (title, author, metadata etc.) from the jats source file. you should, however, rename the css style sheet from hioa-epub.css in epub-content.opf.xsl to the proper name used in the actual journal. we tried using different zip tools to package the epub file but had problems with adding files in the correct order (the mimetype file must be first in the archive). however, the open source windows software info-zip worked well. conversion to mobi amazon kindle is the market-leading e-reader device but has no support for epub. the kindle’s native format is an amazon proprietary format, but it also supports mobi, which was developed by the french company mobipocket that amazon bought in 2005. during the testing phase, we used calibre to convert from epub to mobi. the result is lossless and close to impeccable in all our test cases. putting it all together: the xproc pipeline as mentioned above, the ncbi toolset includes a default xproc pipline, but we had to make several changes to make it suit our needs. we provide the resulting file, process-jats-xml.xpl, as part of the source code. the pipeline uses an input folder (for jats source files), a working directory for temporary files, and an output folder for the final result files. here is an overview of the pipeline: pre-process the jats references to apa citation style with hioa-citations-prep-apacit.xsl. we have made some slight adjustments to the xslt style sheet jats-apacit.xsl. fix some empty id attribute problems on graphic elements in the result file from the previous step. (these problems can probably be solved in a better way, but we have not yet managed to find a better way.) run the main transformation of the result file from step 2 with hioa-xhtml.xsl. this is our vertical customization of the master style sheet jats-html.xsl. in this case, we made major revisions to the master style sheet and chose to cast the result directly into xhtml rather than run the previously mentioned post-processing style sheet. fix empty namespaces in the result document from step 3. just like step 2, this should ideally be a non-issue, but we have not solved it completely. namespaces in xml generally and jats specifically are challenging [13]. most web browsers ignore empty namespaces, but in the strict xhtml 1.1 required by the epub standard, they cause crucial problems. transform the jats source file into content.opf using epub-content.opf.xsl. transform the jats source file into toc.ncx using epub-toc.ncx.xsl. the xproc file has to run on an xproc engine. we used xml calabash, developed by norman walsh. xml calabash includes saxon-he for xslt 2.0 transformations. xproc is a vital part of the epub production but does not make up the full workflow. for example, an epub file is a zip-archive of the epub file structure, but we also need a mobi version of this file. therefore, we have created two scripts (a windows batch file and a linux bash shell script) that include the xproc pipeline and also automate the rest of the process. the scripts take a jats input file and generate three versions of the article: xhtml (for the web), epub and mobi. see the source code section of this article for detailed information about downloading and running either script. costs the main issue with our new workflow is that it is time consuming. the time we spent on each article depends on the length and complexity of the article, and it is hard to predict how long it will take to mark up each issue of the journal. as a proof-of-concept we marked up all five articles in volume 2, issue 1 of professions & professionalism and measured the time spent on each article. we used xml copy editor for the manual markup and we received the articles in microsoft word format from the copy-editor of the journal. the articles had been through the editorial process and adjusted to a print layout. much of this work, like hyphenation and apa formatting of the reference lists, will be unnecessary in the future, when we will use the author’s postprints as the basis for markup. one of the xml coders has a great deal of experience with xml; the other is an expert. the table below shows the time used on markup, and the articles are linked so you can see the level of complexity of each article. tor arne marked up abrahamsen, holte, & laine, 2012, and nygaard, 2012; while trude marked up saks, 2012, kallberg, 2012, and spehar & kjekshus, 2012. article time (in hours) saks, 2012 3 ½ nygaard, 2012 4 ¾ kallberg, 2012 2 ½ spehar and kjekshus, 2012 2 ¼ abrahamsen, holte, and laine, 2012 8 ¾ sum 21 ¾ to compare our efforts, we asked the copy-editor to estimate the time he had spent on the copy-editing work. as mentioned in the section the case, microsoft word is used in the process. the copy-editor told us that it was hard to separate the different parts of his work but estimated that he had used “a full workday” on the issue, i.e. 7 ½ hours. he needed additional help from the journal editor on tables. the tables were exported graphic files from microsoft excel. this work took about 1 ½ hours, which means that the editorial work for the issue in the current production takes 9 hours. the xml markup takes a bit more than twice the time, but this additional work enables publishing in four additional formats: xml, xhtml, epub and mobi. conclusion in this project, we have established a new workflow for an e-journal to replace pdf with epub as the main publication format. by doing this we we have made the journal more accessible for everyone, both in terms of reading devices and user diversity. we have focused on accessibility, universal design, device and software independency, and preservability. marking up an article in xml is not very complicated and is something anyone should be able to do. as long as a document is correctly marked up as jats xml, a single command is enough to run the entire xproc pipeline, which applies all the transformations in the correct order and creates output files in xhtml, epub and mobi. the source code provided should make it easy for others to do the same. an important outcome has been to convince the journal editors of the benefits of the changed workflow. an unforeseen aspect of this project has been the automated crosschecking of the references in the article. any citation in the text that does not have a corresponding part in the reference list will stop the validation process. in addition, if any parts of the reference are missing (for example the value for issue) it will become very apparent in the markup process, as it will be an empty field in the markup. the editors consider this important and believe it adds value to the editorial process. our method will no doubt cost more in effort than the traditional microsoft word to pdf method that the journals so far have employed. however, we believe the value of the output we create will make up for the added costs. we plan to implement the same workflow in our other journals in the future. source code readers can download a zip file from github, which includes everything needed to run the pipeline with the example article as input. the file is located at https://github.com/eirikhanssen/jats2epub and is licensed under a gplv3 license. save the zip file to a folder and unpack the archive. the distribution requires java runtime environment but includes all the software needed with the exception of tools for epub to mobi conversion. the script needs amazon’s kindlegen to do this. you can freely download the software from amazon, but it is illegal to distribute it as part of the zip file. the readme files in the zip archive document the distributed files, folders and usage. eirik hanssen, learning centre and library, oslo and akershus university college of applied sciences has written the scripts, the xproc file and the documentation. references [1] united nations. 2006. convention on the rights of persons with disabilities. available from: http://www.un.org/disabilities/convention/conventionfull.shtml [2] barne-, likestillingsog inkluderingsdepartementet. 2008. diskrimineringsog tilgjengelighetsloven. available from: http://lovdata.no/dokument/nl/lov/2013-06-21-61 [3] w3c. 1999. web content accessibility guidelines 1.0. available from: http://www.w3.org/tr/wcag10/ [4] w3c. 2008. web content accessibility guidelines (wcag) 2.0. available from: http://www.w3.org/tr/wcag20/ [5] w3c. 2000. authoring tool accessibility guidelines 1.0. available from: http://www.w3.org/tr/atag10/ [6] w3c. 2013. authoring tool accessibility guidelines (atag) 2.0. available from: http://www.w3.org/tr/atag20/ [7] professions & professionalism. 2014. editorial guidelines. available from https://journals.hioa.no/index.php/pp/about/editorialpolicies#focusandscope [8] michel mj, knouft jh. 2012. niche variability and its consequences for species distribution modeling. plos one 7:e44932. available from: http://dx.doi.org/10.1371/journal.pone.0044932 [9] pubmed central. 2011. file submission specifications. available from: http://www.ncbi.nlm.nih.gov/pmc/pub/filespec/ [10] international digital publishing forum. 2010. epub 2.0.1. available from: http://idpf.org/epub/201 [11] spehar i, kjekshus le. 2012. medical management in norwegian hospitals. professions & professionalism 2:42–59. available from: http://dx.doi.org/10.7577/pp.v2i1.178 [12] piez w. 2010. fitting the journal publishing 3.0 preview stylesheets to your needs: capabilities and customizations. in: journal article tag suite conference (jats-con) proceedings 2010 [internet]. bethesda, md: national center for biotechnology information (us). available from: http://www.ncbi.nlm.nih.gov/books/nbk47104/ [13] piez w. 2011. taming the beast: jats data, non-jats data, and xml namespaces. in: journal article tag suite conference (jats-con) proceedings 2011 [internet]. bethesda, md: national center for biotechnology information (us). available from: http://www.ncbi.nlm.nih.gov/books/nbk62086/ about the authors trude eikebrokk is a senior librarian and leader of the digital services group in the learning centre and library. she is the technical manager of the publishing platform open access journals at the oslo and akershus university college of applied sciences. tor arne dahl teaches web technologies in the department of archivistics, library and information science at the oslo and akershus university college of applied sciences. he has previously worked as a software developer and is working on a phd in the study of professions concerning information architecture. siri kessel is an assistant professor at the department of computer science at oslo and akershus university college. she is one of the coordinators of the master’s programme in universal design of ict, teaches “user diversity and ict barriers”, and is doing research in this field. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – building an institutional author search tool mission editorial committee process and structure code4lib issue 45, 2019-08-09 building an institutional author search tool ability to collect time-specific lists of faculty publications has become increasingly important for academic departments. at ohsu publication lists had been retrieved manually by a librarian who conducted literature searches in bibliographic databases. these searches were complicated and time consuming, and the results were large and difficult to assess for accuracy. the ohsu library has built an open web page that allows novices to make very sophisticated institution-specific queries. the tool frees up library staff, provides users with an easy way of retrieving reliable local publication information from pubmed, and gives an opportunity for more sophisticated users to modify the algorithm or dive into the data to better understand nuances from a strong jumping off point. by david forero, nick peterson, andrew hamilton frequently academic departments want to know how many publications their faculty and researchers have published over a particular period of time. at the oregon health science university (ohsu) these questions were traditionally handled by having a librarian manually  search various publication databases. these searches were complicated, and the outputs of these searches were often cumbersome and unwieldy. this meant that the answers to questions were often limited by how much effort could be put forth by the librarian whom may spend many hours managing these searches and the resulting data. the development of search strategies to capture accurate and relevant data brings with it many challenges. this is a problem of trying to generate an accurate and comprehensive list based on incomplete and sometimes inaccurate data (databases of publications) with almost always incomplete and contradicting information about authors (lists of university faculty or lack thereof). in this article, we will discuss the overall problem and what efforts we previously made to address it. then we will discuss how we approached the problem differently. we developed an augmented search tool that lightens the load and normalizes the process on our campus. what’s more, the tool is open and could easily be customized for other institutions. the challenge here is a brief list of overarching problems when conducting a search of publications for an academic department: who: what authors are considered part of the academic department. what: what source is being used to find all the publications. when: which of the many dates associated with the publication is being searched. the problem of “who” the most difficult challenge is figuring out which papers in indexes of publications should be counted. many publications include the organizational affiliation of the authors, though, not all. those that include the information tend to solicit the information from the paper’s authors and present the information as free-text. additionally, authors can have multiple organizations to which they belong to and may list some or all of these affiliations. this is further confounded by temporality, as people often change organizations and there is rarely any kind of tracking of who was part of what organization when. rather than looking at the problem top down and searching for organization data, one could do searches based on individual names instead. the typical problems with people with the same name crop up. if authors all used orcid ids (https://orcid.org/), that could be a very effective strategy, though, our experience has been that too few authors utilize them to make this identifier useful. also, getting definitive lists of people in the organization can be difficult. this fact might seem surprising but our experiences in higher education lead us to believe this is more often the case than not. the problem of “what” this issue is determining an authoritative source of what has been published. there is no one source to refer to for all publications for an institution. this is one of many compromises that must be made to approximate the full answer. for an institution like ohsu, scopus and pubmed represent a fairly comprehensive publication source as both offer a wide selection of publications but neither encompass all possible publications formats; for example, smaller journals, thesis, technical reports and other publications which are not monitored by these two services. also, there are the rare publications outside the main domains of faculty at our institution. for example, this article is extremely unlikely to turn up as a publication in either publication index and therefore won’t be counted in any tallies of ohsu publications. the problem of “when” as the previous point alludes, the exact definition of what counts as published might be hard to pin down. this may be a little in the weeds but it underscores the depth of problems facing a searcher. also, the date it is published turns out to be often squishy. publishers often “pre-publish” an article, and then later officially publish the article. what this means is the data in indexes like scopus and pubmed are dynamic, in other words, the data can change and be updated and often have different object identifiers. so, a diligent person who monthly tallies the publications of a particular organization might count an article in two different months. avoiding this requires tracking individual articles. our previous process prior to the development of our automated search tool, a list of ohsu authored publications was manually created by the ohsu library by running a monthly update for ohsu affiliations in both pubmed and scopus.  the results of these updates were then imported into refworks, and later endnote, for processing which consisted of identifying and eliminating duplications between the two databases and moving newer versions of an article over to replace any existing versions in refworks.  this monthly process was incredibly labor intensive and unsustainable for library staff.  it was decided that the monthly update strategy for ohsu affiliations within pubmed could serve as the foundation of the new tool. as discussed, we had a very resource intensive process. initially attempts to address this problem involved essentially building a local database of publications. this involved constantly resynchronizing the local list of publications with an index like scopus and required a considerable amount of curation. in particular, the de-duplication and curation of members of organizational units was still considerable. essentially this was a fancier version of the existing manual process. instead, the approach we took was to augment people’s searches to approximate some of the normalization done in the manual process. part of this decision was based on the fact that, we knew we couldn’t get a set of perfect results and needed to focus on good enough. the new attempt would acknowledge the potential for incompleteness but instead focus on making the efforts less. essentially, if we’re not going to get a perfect answer, can we put in less effort to get an equally complete answer. for the “what” problem, the tool would leverage an existing publication index. we chose pubmed because though it was a smaller index, it does cover the main focus of our institution, and we could freely share the results without violating any contractual restrictions. we can easily capture the kinds of searches we were already doing. for the “when” problem, we were essentially leaving that up the individual searcher. if the searcher cares deeply, they could track their list of publications. however, we were no longer going to try to deal with this problem at the institutional level. the big problem was how to address the “who” problem. searching for specific people is fraught with difficulty as people change names and often have identical names with others. our approach is only to search for people after we have a search limited by the scope by institution. since ohsu does not have an identity management system, there are many collections of people in different departments for different purposes. we worked with the registrars and human resources offices to get a list of people and their primary organizational unit within the institution. the idea was to essentially extract a list of most likely ohsu publications from pubmed and then try doing name matches with the lists from the registrars and human resources. this isn’t a perfect solution, however; we have greatly reduced the likelihood of name collision by restricting the publication list to likely ohsu articles. also, we have a problem with older articles, where it is more likely that an author is no longer associated with ohsu and therefore not in the “master” list provided by the registrars and human resources. fortunately, publication requests were most often for recent publications. automated method the service <http://library.ohsu.edu/oast/> we created uses a fairly standard lamp (linux, apache, mysql, php) stack. we will highlight two main high-level components of the search tool: php which is used to query the pubmed api for citation information and mysql to query a local db that is culled from our institution’s available registrar and human resource data. in practice, any available web programming language will suffice, but we choose php since it is utilized heavily in-house. for the people data, we are still in the process of finding the most optimized way of matching authors from pubmed queries but for now mysql was chosen since we already had the data on-hand in this form. an important design choice here was to use fields that users already commonly use to search and mapping our fields directly to the fields used in pubmed. the basic gist of the php form takes a series of user provided inputs, constructs and submits a pubmed query, compares those results with available people data and returns the final result set to the user as an ajax datatable for viewing or export to common tabular data formats. user inputs on the form range from journal title [ta], date range of publication [dp], author name [au], grant number [gr], etc. fig. 1: ohsu author search tool additional fields are available under the advanced options button for fine tuning the search. under the advanced options is the original base query which is fully customizable by the end user. all user inputted fields are anded to this base query and submitted to the pubmed esearch api when hitting ‘search’ on the form. fig 2: ohsu search tool advanced options the api returns our result set and makes it viewable as a paginated table to the right of the search form using the ajax datatables library. this allows for responsive sorting/filtering as well as exporting to common tabular data formats including csv and excel. if the user selects the ‘match ohsu departments’ checkbox then an attempt to match author names from the pubmed result set against our mysql db of ohsu people data is made and those names are bolded in the report and their associated campus departments are returned. fig. 3: ohsu search tool results set example the form is ‘self-posting’ in the sense that all modified search fields (including the base query) are constructed as a uri query string. this means that links to specific result sets are entirely portable or even entirely customizable outside of using the form itself for input. this makes canned result sets easy to construct against the tool. our current plans are to expand this functionality to allow for proper api calls or direct csv/excel dumps for specific need cases. code is available on github here: https://github.com/ohsu-library/pubmed-query-util lessons learned our goal was to provide a service of equivalent quality with less resources dedicated to the effort. in this we were successful. as we did user testing, we learned more about what the users of this service needed. one particular lesson involved grant numbers. essentially, ohsu staff used grant numbers to search for publications. although most of the grants at ohsu are federal grant numbers which have a very specific format, we quickly discovered that these grant numbers are not consistently entered in different publications. the biggest success in our minds is this tool augments the capabilities of people using it. since everything in it is completely transparent. users can bookmark the search and since the search query is embedded in the url, it provides users with quick jumping off points from previous efforts. also, since the tool provides an easy way to modify the core query for institution, individuals can tailor the query to accommodate their needs. for example, as other organizational units are formed, the user does not have to wait to request a change to the base search from the library. instead they can modify the search for themselves and that change is also embedded in the url. this tool enables individuals to start a search with a fairly intelligent starting place. these individuals can still get help from an experienced reference librarian but now they come to the conversation with a reasonable start and the librarian can augment and tweak the search. this is educational and empowering for the user and allows our librarians to focus efforts on more complicated requests. as discussed, more consistent entry across all publications will make this process easier. perhaps one day ohsu will have a central registry of authors’ orcid ids that is always up to date with human resource data and organizational affiliations while all publications have consistent orcid id usage and some kind of usage of controlled organizational identity. until then, at ohsu we have found a way to empower our staff to make fairly quick power searches that provides solid results. about the authors david is the technology director for the ohsu library in portland oregon. he is active with the equity, inclusion, and diversity work both at ohsu and acrl. he also co-manages the game lending collection at ohsu. andrew hamilton, ms/mls is an assistant professor employed as a health science education & research librarian at the ohsu library. prior to joining ohsu, he worked for the nn/lm national online training center from 1996-2002 where he travelled the united states teaching thousands of librarians how to search pubmed. mr. hamilton earned his master of library science degree as part of a double master’s degree for drug information specialists (m.s.-pharmaceutical science/m.l.s.) from st. john’s university in 1993. nick is a systems analyst at the ohsu library whose primary responsibilities include management, maintenance and support of the library’s servers, software, desktop computers, printers and av equipment. his specialties include database development, systems integration and custom web development. outside of work he is an avid photographer, filmmaker and budding woodworker.   subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – feminism and the future of library discovery mission editorial committee process and structure code4lib issue 28, 2015-04-15 feminism and the future of library discovery this paper discusses the various ways in which the practices of libraries and librarians influence the diversity (or lack thereof) of scholarship and information access. we examine some of the cultural biases inherent in both library classification systems and newer forms of information access like google search algorithms, and propose ways of recognizing bias and applying feminist principles in the design of information services for scholars, particularly as libraries re-invent themselves to grapple with digital collections. by bess sadler and chris bourg libraries are not neutral in spite of the pride many libraries take in their neutrality, libraries have never been neutral repositories of knowledge. research libraries in particular have always reflected the inequalities, biases, ethnocentrism, and power imbalances that exist throughout the academic enterprise through collection policies and hiring practices that reflect the biases of those in power at a given institution. in addition, theoretically neutral library activities like cataloging have often re-created societal patterns of exclusion and inequality. for example, in the power to name, hope olson documents the ways the dewey decimal system has historically reflected patterns of knowledge organization that now seem archaic, such as classifying the subject of pregnancy under the heading of disease, and the subject of lynching under the heading of law enforcement (olson 2002). while historic distance makes it easier to recognize these examples as obvious instances of societal power imbalance appearing in library practice, it might not be so easy to recognize current examples of supposedly neutral practice in libraries that are actually perpetuating similar power imbalances. those of us creating library software and building digital libraries must also address this issue. just as the classification systems created by libraries carry the aura of “neutrality” and mask the bias reflected within them, the digital systems that libraries provide to our users are also assumed, by virtue of existing within a library, to be a “neutral” reflection of subject knowledge. however, just as with classification systems, this neutrality is an illusion. in this paper, we argue that without an explicit feminist agenda, the same processes of exclusion and marginalization that have always influenced libraries — and therefore scholarship — will continue to play out in our digital library and online discovery environments. we define library discovery as the set of affordances through which users search, explore, find, and interact with the information resources they need, particularly collections held by a library. for the purposes of this paper, we will focus on digital systems for library discovery, such as search algorithms, library software, and online collections. for our analysis, we borrow from the field of human computer interaction (hci) and shaowen bardzell’s paper “feminist hci: taking stock and outlining an agenda for design” (bardzell 2010). bardzell outlines some of the characteristics of feminist software interaction, which she defines as plurality, self-disclosure, participation, ecology, advocacy, and embodiment. we hope that by examining these qualities in the context of library discovery we can encourage the development of library discovery systems that resist patterns of erasure, exclusion, and marginalization. pluralism and self-disclosure in search in “feminist hci,” bardzell discusses how feminist theory can inform the field of human computer interaction. certain constellations of qualities characterize feminist human computer interaction, she argues. so what would library discovery that possessed these qualities of feminist interaction look like? and how might we explicitly design for that? we might, for example, invoke feminist rhetoric in our critique of search technologies and demand that the search solutions we employ value pluralism and self-disclosure. according to bardzell, “the quality of pluralism refers to design artifacts that resist any single, totalizing, or universal point of view” (p. 1305) and the quality of self-disclosure refers to the extent to which the software renders visible the assumptions it is making about its users (p. 1307). search software, in order to provide feminist interaction, needs to be particularly concerned with pluralism and self-disclosure because it deals with questions of relevancy and significance. a single, totalizing, or universal point of view to the questions “what is relevant?” and “what is significant?” is likely to re-create existing societal patterns of exclusion and marginalization. unfortunately, we have something very similar to a single point of view on the question of relevance and significance in the form of google’s search algorithms. although google’s exact search algorithms are trade secrets and shift over time, we do know that they are based on a patented and published algorithm called pagerank, and that they work by defining relevance and significance by looking at what pages on the internet are linked to most often on a given subject (https://en.wikipedia.org/wiki/pagerank). this creates a majority-rules definition of relevance that masquerades as neutrality. the concept of neutral relevance is an oxymoron, and yet neutrality is often what is sought in the design of search algorithms. in a recent antitrust lawsuit brought against google, rival search engines demanded that google use “neutral” search algorithms and display search results in a “neutral” manner. one need not invoke feminist theory to see how meaningless such a demand is; forbes, hardly a bastion of feminist thought, called the concept of a neutral search engine “incoherent” (ammori and pelican, 2012). in “missed connections: what search engines say about women,” safiya umoja noble illustrates some of the problems with google’s consensus-based relevancy. she describes an exercise in which she asks her university students to imagine themselves an african-american aunt, mother, mentor or friend who is trying to help young multicultural women learn to use the internet. she asks what they would search for to find content young black girls might be interested in, and asks what search strategies they might employ to learn about black accomplishments, identities, and intellectual traditions: someone inevitably volunteers to come forward and open a blank google search page–a portal to the seemingly bottomless array of information online–intending to find accurate and timely information that can’t easily be found without a library card or a thoughtful and well informed teacher. last semester, sugaryblackpussy.com was the top hit. no matter which year or class the students are in, they always look at me in disbelief when their search yields the result. they wonder if they did something wrong. they double-check. they try using quotation marks around the search terms. they make sure the computer isn’t logged into gmail, as if past searchers for pornography might be affecting the results. they don’t understand. noble summarizes: “try google searches on every variation you can think of for women’s and girls’ identities and you will see many of the ways in which commercial interests have subverted a diverse (or realistic) range of representations” (noble 2013). noble’s classroom exercise demonstrates how google search algorithms fail both our criteria for pluralism — because they represent a single majority-rules point-of-view masquerading as neutrality — and our criteria for self-disclosure — because the system does not render visible the extent to which is has been designed for an “ideal user.” libraries would do well to keep these lessons in mind as we adopt new technologies like linked data, which holds much promise for breaking the stranglehold of rigid categorization systems, but makes many of the same assumptions that the google algorithms make — namely that the most commonly asserted statements must be the most true. libraries should also be asking these questions of our journal article search systems. until we and our users are allowed to data mine and index scholarly journal content ourselves, we rely on commercial publishers to define significance and relevance for us. participation and ecology in the production of technology bardzell’s quality of ecology in feminist interaction design calls for an awareness of the broadest contexts of the effects and of the widest range of stakeholders in considering design (p. 1307). the quality of participation refers to valuing participatory processes in the creative endeavor (p. 1306). in the creation of digital library systems, one might ask about the people building these systems and the environments in which the software is produced, as part of the software’s ecology. similarly, one might ask about the methods used in the software’s creation and how participatory or inclusive they were. more bluntly, when software is created in environments hostile against women, people of color, and other underrepresented groups, many kinds of inequity result. currently, the majority of digital library software is built with open source software tools; even commercial or hosted digital library systems are built with open source components, requiring anyone who wants to help build them to participate in the world of open source software. this has many positive implications for the production of software capable of feminist interaction, as the open source software movement is built around the idea that software code should be freely shared and that everyone has the right to understand how software works. however, the open source software community is also a notoriously sexist space, as documented in the twitter feeds of many women software engineers and in academic papers such as “free as in sexist: free culture and the gender gap” by joseph reagle. reagle analyzed six years of discourse about gender and sexism within free culture communities. in spite of many attempts by women in those communities to raise awareness of the misogyny they encountered, reagle still found that many communities rationalized low female participation as a matter of women’s choice, in part the result of a pernicious narrative of “techno-eager men and techno-phobic women” well documented in books like making technology masculine by ruth oldenziel. the perpetuation of this narrative makes women’s exclusion seem natural and expected, and therefore a problem unworthy of addressing (oldenziel, 2004). this climate is part of the reason that only about 2 percent of open source software developers are women (nafus, krieger and leach, 2006) and is incompatible with the idea of a feminist future for library discovery software, since it either prevents women from taking software development jobs, or forces them into hostile work environments. unlike other harassment battles, however, the internet has no human resources department to intervene. this is an ongoing ethical challenge to libraries. as we shift our practice to digital collections, more library jobs — and many of the highest paying library jobs — are in the area of library software, requiring engagement with open source communities. some strategies to make these jobs more accessible include efforts to foster involvement from underrepresented groups, such as internships targeted at women and gender minorities. however, efforts that focus exclusively on recruiting underrepresented groups into technology work reinforce the narrative that they are not naturally interested in technology and must be persuaded to participate. crucially, these efforts fail to address harassment, and talented community members who are driven out of technology work every year by hostile environments (melymuka, 2008; wu, 2014). some of the best ideas on challenging sexism in open source software communities come from the ada initiative, a feminist organization dedicated to supporting women in open technology and culture communities. one of the initiative’s recommendations has been to encourage software communities and conferences to adopt codes of conduct (ada initiative, 2015). what was once a controversial effort has become a widely accepted best practice within the realm of open source software, and, increasingly, among library conferences. in 2013, stanford university library became the first employer to encourage staff to attend only conferences with a code of conduct, and to lobby conferences without a code of conduct to adopt one (keller, 2013). the authors hope this will be an increasing trend, as libraries and other employers grapple with their responsibility to create a safe work environment for their staff, even when that work environment extends beyond the walls of the institution. advocacy and embodiment in library interactions bardzell’s quality of embodiment means focusing attention on the bodies, emotions, motivating drives, and primordial urges of users. the quality of advocacy asks how we can design systems that improve users’ lives without imposing the designer’s view of what might constitute an improvement. focusing attention on the bodies of our users might mean prioritizing accessibility in our software, and thinking about the many ways we can make library resources available to users with a wide variety of impairments. conducting usability tests with patrons who use assistive technologies can be a particularly enlightening technique, and one that allows them to tell us what would constitute an improvement. embodiment might also mean valuing emotional responses to our collections. for example, historian natalie zemon davis discusses the specific emotional connection she feels when touching physical artifacts, like books. among other subjects, she studied the lives of 16th-century printers to see what she could glean about the everyday experience of people who are (typically) undocumented in archives. one of the ways she did that was to visit libraries that held the books these 16th-century subjects had made. she argues that actually holding the books connected her to the people she was studying in a meaningful way (cbc radio, 2011). in our rush to digitize the kinds of rare books davis loves to touch, have we asked ourselves what might be lost? will high-resolution scans produce the same feeling of emotional connection? if there is a specific emotional response achieved by touching something that the person you are studying also touched, is that a valid reason for allowing physical access to a collection? we know this emotional response exists because people like davis tell us so all the time, but we often dismiss it as unimportant. should we dismiss it because it is emotional in nature, or should we make some effort to understand it, in hopes of recapturing that feeling in the digital realm? from listening to historians like davis, and from our understanding of psychology, it is clear that design decisions which divorce scholarship from emotional response are not in the best interests of our users. we make choices with our emotional responses, and then we come up with justifications after the fact. we know this from studies of people who have been truly divorced of their emotions on a neurological level and then become unable to make even simple decisions (damasio, 1994). we also know that emotional states affect problem solving, and are tied to many unconscious problem solving centers in the brain (kahneman, 2013). the necessity of design that engages our emotions is well documented in the hci literature, for example in don norman’s emotional design (norman, 2005). insisting upon the principles of feminist hci, including valuing users’ emotional responses, as we envision the future of the archive and our digital transition, thus becomes an urgency, particularly for those who are often erased from official archives. so what would it take for us to seriously focus on it? archival theorists like jack halberstam, in the queer art of failure, and ann cvetkovitch, in an archive of feelings, both discuss the unrepresentability of trauma and the nature of the archive of trauma. both engage with erasure of homosexuality in the third reich, recent history, and our present, as a trauma that must be grappled with in the archive. meanwhile, we who build digital libraries (which we hope future researchers will utilize) are designing new trauma archives. the open source software we build enables discovery for the collections of the united states holocaust memorial museum, and archival collections at stanford like the records of the stop aids project. one should not attempt to engage with these collections without a sense of embodiment and advocacy. we need to push back against the notion of the dispassionate researcher, the dispassionate archivist. libraries should not try to be neutral the work of libraries and librarians can do more than just support feminist research and agendas. we can play a critical role in supporting the causes of inclusion, plurality, participation and transparency. building collections and developing the tools to access them are inherently political acts; we are creating the future library, the tools and collections that will be used to create new knowledge. the means of production for the archives of humanity are up for grabs, and within our reach is the possibility of new production methods that resist the recreation of existing patterns of exclusion and marginalization. however, this will be possible only if we approach this work consciously and with an understanding of the impact of our decisions. about the authors bess sadler has been building open source software for libraries for over a decade, and is a co-founder of several widely-used open source software projects including project blacklight and project hydra. she is currently employed as the manager for application development in the digital library systems and services group at stanford university library. her opinions are her own, and do not necessarily reflect the views of her employer or any software projects with which she is associated. chris bourg is the director of libraries at massachusetts institute of technology (mit), where she also has oversight of the mit press. prior to assuming her role at mit, chris was associate university librarian for public services at stanford university. chris has a phd in sociology from stanford university, and spent 3 years on the faculty at the united states military academy at west point. she is keenly interested in issues of diversity and inclusion in higher education; and in the role libraries play in advancing social justice and democracy. works cited anti-harassment work: conference anti-harassment policy [internet]. ada initiative. [cited 2015 feb 17]. available from: https://adainitiative.org/what-we-do/conference-policies/ ammori, m, and pelican, l. 2012. why search bias claims against google don’t hold up. forbes. [cited 2015 february 17] available from: http://www.forbes.com/sites/ciocentral/2012/06/07/why-search-bias-claims-against-google-dont-hold-up/ bardzell, shaowen. 2010. feminist hci: taking stock and outlining an agenda for design. in: chi 2010: hci for all. april 10–15, 2010, atlanta, ga, usa cbc radio. the best of the sunday edition july 17, 2011. [internet]. [cited 2015 february 17]. available from: http://castroller.com/podcasts/thesundayedition/2429203 cvetkovich, ann. 2003. an archive of feelings: trauma, sexuality, and lesbian public cultures. durham, nc: duke university press. damasio, antonio. 1994. descartes’ error : emotion, reason, and the human brain. new york: putnam. halberstam, judith. 2011. the queer art of failure. durham, nc: duke university press. kahneman, daniel. 2013. thinking, fast and slow. new york: farrar, straus and giroux. keller, michael. 2013. sul supports conference anti-harassment policies. [internet.] [cited 2015 feb 17]. available from: http://library.stanford.edu/news/2013/07/sul-supports-conference-anti-harassment-policies melymuka, kathleen. 2008. why women quit technology. computerworld. [internet]. [cited 2015 feb 17]. available from: http://www.computerworld.com/article/2551969/it-careers/why-women-quit-technology.html nafus, dawn; kreiger, bernhard; and leach, james. 2006. gender: integrated report of findings. free/libre and open source software: policy support. [internet] [cited 2015 feb 17] available from: http://www.flosspols.org/deliverables/flosspols-d16-gender_integrated_report_of_findings.pdf noble, safiya umoja. 2012. missed connections: what search engines say about women. bitch magazine, 54. [cited 2015 feb 17]. available from: https://safiyaunoble.files.wordpress.com/2012/03/54_search_engines.pdf norman, donald. 2005. emotional design. new york: basic books. oldenziel, ruth. 2004. making technology masculine: men, women, and modern machines in america, 1870-1945. amsterdam, netherlands: amsterdam university press. olson, hope. 2002. the power to name: locating the limits of subject representation in libraries. dordrecht, netherlands: kluwer academic publishers. 1 reagle, joseph. 2013. “free as in sexist?” free culture and the gender gap. first monday [internet]. [cited 2015 feb 17] available from: http://firstmonday.org/ojs/index.php/fm/article/view/4291/3381 wu, brianna. 2014. no skin thick enough: the daily harassment of women in the game industry. polygon. [internet]. [cited 2015 feb 17]. available from: http://www.polygon.com/2014/7/22/5926193/women-gaming-harassment subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – searching for meaning rather than keywords and returning answers rather than links mission editorial committee process and structure code4lib issue 57, 2023-08-29 searching for meaning rather than keywords and returning answers rather than links large language models (llms) have transformed the largest web search engines: for over ten years, public expectations of being able to search on meaning rather than just keywords have become increasingly realised. expectations are now moving further: from a search query generating a list of “ten blue links” to producing an answer to a question, complete with citations. this article describes a proof-of-concept that applies the latest search technology to library collections by implementing a semantic search across a collection of 45,000 newspaper articles from the national library of australia’s trove repository, and using openai’s chatgpt4 api to generate answers to questions on that collection that include source article citations. it also describes some techniques used to scale semantic search to a collection of 220 million articles. by kent fitch motivation and goals our expectation of how search works changed ten years ago when google introduced their “hummingbird” search algorithm [1].  our experience until then was that you needed to formulate your search query using some exact words that appear in the document you were seeking. ten years later, we’ve been further trained by “ok google” and continuous improvement in search algorithms to find what we need by issuing a query defining our intent rather than exactly matching words, and increasingly, by using natural language to simply ask a question. google’s knowledge graph [2] introduced in 2012 taught us to expect direct, summarised answers to queries, and now, bing and google are both anticipating a near future where the traditional “ten blue links” of a search result are superseded by text built by extracting information from the documents found by the search, enriched by the context provided by a knowledge graph and presented as an answer to the question inferred from the search query. the gap in capability between the almost universally used search engines that set community expectations and libraries and related repositories has continued to grow. that gap, opened by pagerank [3] 25 years ago, is now a chasm. to provide modern and efficient discovery services for their communities rather than risking mutual abandonment by becoming perceived as baffling backwaters of byzantine business-rules redolent of a bygone era, libraries must adopt the same core search technologies used by the commercial services: semantic search, knowledge graphs and generative ai to support summarisation and chat. ten years ago, even semantic search would have been an impractical goal: the technology was specialised and expensive to implement and wasn’t well advanced. just two years ago, attempting to implement much of this capability was more like a “science project” than a routine information technology undertaking. but as quickly and surely as public expectations have changed, so has the accessibility of the technology:  the use of ai has spread from university labs and tech leviathans and is now becoming commonplace, further increasing the pressure on the library community to embrace its effectiveness for the benefit of the public. this article does not provide definitive answers to the many questions raised when introducing semantic search and chat-based question answering across a digital repository. rather, it presents findings from a preliminary exploration in the hope of encouraging widespread experimentation with this rapidly evolving technology so that the benefits it will deliver to library communities can be expedited and shared. the context for the technologies investigated and described in this article is a repository of newspaper articles. the main investigations were performed on a very small subset of the national library of australia (nla) digitised newspaper repository. this study examined the feasibility and effectiveness of: semantic search named entity identification and disambiguation of entities described by wikipedia question answering, summarisation and a “chat” interface a further investigation of the feasibility of scaling semantic search to the full nla digitised newspaper repository (about 220 million articles) was also performed. context the national library of australia (nla) digitised newspaper repository forms the most accessed component of their trove service [4]. it contains over 220 million newspaper articles published since 1804 in over 1,800 separate newspaper and gazette titles. during may 2023, google analytics recorded about 1.2m searches on trove, and 3.8m pageviews of newspaper articles. the repository is popular with family researchers looking for information using personal names and place names (particularly in birth/death/marriage notices) and is also widely used by academics and general researchers. an indicator of trove’s popularity is that when government budget measures threatened the ongoing sustainability of trove in 2022, a petition calling for adequate funding received almost 34,000 signatures [5]. from the trove newspaper corpus, 45,494 news articles from the title the canberra times from the year 1994 were extracted. articles of type advertising (display and classifieds) were excluded as the intent of this prototype was to focus on the benefits of semantic searching on news rather than on “births, deaths and marriages” searches which tend to be more “known item” searches on people and places. there were many uncorrected ocr errors in the extracted text. to evaluate the practicality of name disambiguation using wikipedia articles, 38,359 articles about organisations and 155,781 articles about people were extracted from an august 2022 wikipedia dump, limiting the extract to articles that contained the word “australia” or contained at least 3,000 words. the goal was to attempt to include all australian people and organisations, and all “prominent” (worthy of 3,000 words) people and organisations even if non-australian. heuristics were used to classify articles as being of interest to this trial (ie, about people or organisations, not place names). approach and results named entity identification the news articles were processed using the stanford nlp library [6] to identify entities of type person, location, organization and misc. common non-specific entities such as “he”, “his”, “she”, “her”, “hers” were dropped. minor processing to encourage better uniformity (particularly in the face of ocr artefacts) was performed, eg, “nsw” was changed to “new south wales”, “mel bourne cup” was changed to “melbourne cup”. embedding creation the text from each news article was used to create an “embedding”, which is a vector characterising the text as a point in a high-dimensional space in such a way that texts which have similar meanings have vector representations which are “close” in that multi-dimensional space. embeddings can be created in many ways, but those created by transformer-based large language models (llms) have been successfully used as a foundation for semantic search. the better the embedding represents the meaning of text, the better a semantic search using a set of those embeddings is likely to be. as well as characterising text, many models can also create embeddings for images, hence mapping images and text to a common high-dimensional space and enabling searches across both images and text. although not the immediate focus of this experiment, the trove newspaper repository contains millions of newspaper images (such as photos published in newspapers) and other sub-repositories in trove contain tens of millions of other images, so an embedding model able to characterise images and text is of interest. although it is impossible to visualise points in a high-dimensional space, this simplification may help:   figure 1. simplified representation of embeddings. here, four input documents (three of text, one image) are each separately processed by an embedding model to produce four embedding vectors. if we just look at the first three values and treat them as coordinates in three-dimensional space, we can imagine how the four points represented by those coordinates could be compared for proximity, and how, if the embedding is successful in representing texts or images as vectors, it would be possible to find things similar to some other thing. figure 2. searching data flow. a semantic search engine accepts a query (a text string, or possibly an image) and creates an embedding for that query, just as it had previously created an embedding for the documents it stores. it then finds the closest document embeddings to that query embedding in the high-dimensional space of the embedding model, and returns the closest documents, ranked by the proximity of their embeddings to the query embedding. figure 3. simplified representation of embeddings. it must be noted that the language model used to create the document embeddings must be also used to create the query embeddings. for example, if openai’s ada-002 was used to create the document embeddings, ada-002 must also be available and used when creating the embedding of the query to be compared to those document embeddings. for large or long-lived repositories, the ongoing availability of the language model and associated software required to create embeddings is a key driver of the choice of the embedding to be used. embeddings map text to a vector space of hundreds or even thousands of dimensions. a higher-dimensional space derived from a very large language model, well-trained and well-aligned with human understanding, can typically represent more subtly of meaning than a lower-dimensional space. however, more dimensions are more expensive to store, index and compare. for the proof-of-concept, three embeddings were tried, derived from: the bert language model – a very early transformer llm. clip vit l-14 [7]: maps image and text to a single vector space. although not the “best in class” sentence transformer, clip vit l-14 performs well, is quick, and importantly for this proof-of-concept was very easy to set-up using “clip as a service” [8]. ada-002:  a recent and highly-regarded embedding provided as a commercial cloud service by openai. evaluation of bert was discontinued because qualitative performance was inferior to clip. clip produces embeddings of 768 dimensions, and  ada-002 produces 1,536 dimensions. qualitatively, ada-002 embeddings are more useful for semantic search, but its extra resource requirements, cost and non-open nature raise concerns for a repository with lots of data and a long life. each language model has a maximum size of text it is able to process when creating an embedding. text beyond that maximum is ignored and does not contribute to that embedding. the  ada-002 embedding can accept about 6,000 words which is more than almost all newspaper articles, and even for longer articles, probably perfectly adequate (it is hard to imagine an article that buries the lede that deep). however, clip, like many other embedding-generating sentence transformers, is limited to about 70 words. it is not obvious how best to process longer text. one approach is to generate and index multiple embeddings per article. for example, for an average-length news article of, say, 500 words, generate 8 embeddings and index all of them with the article and somehow at query time, combine the similarity scores for each embedding to generate an article match score. however, it is important to “break” the embedding text on sentence boundaries, as sentence transformers are trained on sentences, not arbitrary extents of text. also, it isn’t obvious that splitting “meaning” across separate embeddings will allow that meaning to be represented by summing or otherwise combining the match scores of the component embeddings. and finally, such splitting greatly increases the size of the embedding index and will negatively affect performance. another approach is to somehow merge the multiple embeddings: that is, in the above example, generate perhaps 10 sentence embeddings then add the vectors together (and then “normalise” this summed vector to have a unit length required for best performance by embedding index engines). this proof-of-concept uses a “hybrid” approach for clip: it generates an embedding for the first (up to) 70 words honouring sentence boundaries which is indexed separately, and also produces a summed embedding from multiple sentence embeddings from the entire text. qualitatively, the use of the first 70 word embedding was unnecessary – summing the embeddings did not seem to “blur” or “smear” the meaning as much as was feared. the choice of embedding model is an important decision with significant consequences discussed below. embedding indexing embeddings can be indexed and searched using one of several specialist “vector” databases (such as pinecone, milvus and weaviate) or by using vector extensions to existing relational databases (such as the pgvector extension for postgres) and document stores (such as elastic and lucene/solr). as lucene/solr is an existing key component of nla’s technology stack, it was chosen for this proof-of-concept and performed well and “out of the box” with the clip embeddings. lucene implements “approximate nearest neighbour” (ann) search using the hierarchical navigable small worlds (hnsw) search graph [9]. the ada-002 embeddings however are longer than lucene’s current vector limit of 1,024 dimensions, and so required a trivial source-code change to enable. semantic searching the proof-of-concept lucene/solr index of 45,494 newspaper articles contains two separate types of indices: “traditional” free text index able to support keyword and phrase searching and ranking based on relative occurrences of search terms in the index, the document and the document’s length vector index able to match a query embedding against all document embeddings to generate a similarity score used for ranking each type of index has strengths and weaknesses. free text indices are great for exact matches on the words in the supplied query, and for “boosting” the ranking of documents where those words are found in more important fields (perhaps a title or abstract), or are found together as a phrase. vector indices are great for finding documents that are “like” the query – not necessarily due to word matches, but by matching the meaning of the query. an example of a good candidate for a text search is a name search – if you search for john smith, you are almost certainly searching for a document containing those words very close together. it is unlikely there are useful “semantics” beyond that – that is, there are probably not “john smith”-like documents that don’t contain those words close together. another possible example is a single-word search, such as kamikaze: it is likely that the searcher is looking for that exact word rather than content that does not contain that word but somehow represents the “spirit” or meaning of that word. however, train crash is less clear cut, as the searcher is possibly interested in railway accidents or derailment or level-crossing smash or train smashes into bus or any other number of other slightly broader, narrower or related concepts. as queries get longer, they tend to convey even more “semantic intent” that is impossible to honour with simple keywords, even when attempting to automatically expand the search with keywords. for example, a search for articles using the fall of john major (a prominent conservative british prime minister during the 1990s) conveys a clear semantic intent. if a patron walked up to a librarian and asked for content using that phrase, the librarian would have a good idea about what they were after. performing this search on trove (limited to 1993 canberra times articles to enable a comparison) produces results a user familiar with google does not expect: many results have nothing to do with “john major” – they just happen to have each of those 5 words somewhere in the article. perhaps worse, many directly relevant articles are not found: perhaps they use ousting or downfall or undoing or unravelling or humiliation or collapse of support rather than fall, perhaps they describe the political pressure he was under in other ways. the proof-of-concept explores a blended keyword and semantic search. it does this by: issuing a “standard” keyword and phrase text search with standard keyword ranking. for each of the top-10 keyword-ranked results, fetch that document’s embedding and use this to issue a semantic search to find other documents with embeddings most similar to it. the intent is to “enrich” the result candidates by including documents very similar to those best keyword results but which may not contain all the keywords. creating an embedding of the original search query and issuing a semantic search to find documents with embedding most similar to it. each of these searches produces a ranked list of documents with a search score (in this case, generated by lucene). the score for the first search (the “standard” keyword and phrase search) is calculated by lucene based on its default bm25 [10] ranking using keyword repository and document frequencies with a boost applied if all keywords were found nearby (ie, a phrase-like boost). the scores for the second and third searches are also calculated by lucene based on the distance in vector-space between the embeddings of the search embeddings and the document embeddings. the proof-of-concept then applies a separate weight to the scores produced by the three types of searches and adds the results across all three searches to generate a document result set for ranking. the three weights are referred to by the proof-of-concept user interface as: keyword boost keyword-found doc similarity boost query similarity boost hence to perform a ranking purely on keyword score, set the second and third weights to zero, and to perform a ranking purely on semantic similarity to the query, set the first and second weights to zero.   figure 4. search results for the fall of john major: pure keyword results on the left, blended keyword and semantic results on the right.   it is not obvious how to set the three weights. empirically, a search using the clip embeddings benefits more from a lower relative query similarity boost than a search using the ada-002 embeddings, probably because the “semantics” captured in the ada-002 embedding of both query and documents are much better. also empirically, a search on just one word, or on a “known item” such as a person’s name often gives better results when the scoring is dominated by keyword boost (but only when that word exists in the repository – see cybersecurity counter-example below) ranking of results is subjective and no formal recall/precision analysis has yet been performed. it must be noted that the vector search algorithm used by lucene, hnsw, does not guarantee to find the “most relevant” results, that is, the documents whose embedding vectors are closest to the query vector: hnsw is an “approximate nearest neighbour” algorithm that trades accuracy against resource consumption (cpu, storage, i/o). however, if reasonable care is taken with hnsw index construction parameters and query parameters, hnsw is highly accurate. a version of the proof-of-concept is available for demonstration here: http://nla-overproof.projectcomputing.com/knnblend a typical search demonstrating the comparison of keyword and blended results is this search using the query the fall of john major: http://nla-overproof.projectcomputing.com/knnblend?set=1994&embedding=ada-002&stxt=the%20fall%20of%20john%20major the following table is a summary of the first 50 results of each search using the ada-002 embeddings, showing how keyword results were “demoted” (rendered as with pink background in the blended results), “promoted” (rendered with light green background), and shows results only found by the semantic search (dark green background, designated as “knn”). for example, the first keyword result (about the war in bosnia containing comments by john major) was demoted to rank 48; the second keyword result (about a fall in regional bank values that includes many occurrences of the query keywords but is not at all about the intent of the query) is demoted to rank 50; the fourth keyword result is promoted to rank 1; the 6th through 17th blended results were only found by a semantic search.   table 1. showing how keyword results where re-ranked after blending with semantic results keyword search blended search rank article re-ranked to rank article re-ranked from 1 international bihac still under siege by kurt schork sarajevo, friday: bosnian s 48 1 jig is up for uk conservatives bill mandle the british local-govern ment electio 4 2 business and investment regional banks shed value after rates increase by michae 50 2 major problems for unpopular pm ‘there is not much point in spending money on a 5 3 international major warns tories: eu yes or a poll – by richard meares london, f 18 3 momentum against major the administration of britain’s prime minister, john majo 9 4 jig is up for uk conservatives bill mandle the british local-government electio 1 4 the undoing of john major malcolm booker the conservative lead ers in britain, a 13 5 major problems for unpopular pm ‘there is not much point in spending money on a 2 5 major’s fortunes turn around john major must be almost unable, to believe his lu 23 6 brambles 313m loss due to big abnormal sydney: a huge abnormal loss and poor per 72 6 [foreign major’s moral crusade nauseating: lamont london: a bitter attack by nor knn 7 hopes now ride on future of falcon comment by peter brewer by axing the capri co 73 7 seeds sown for conservative uprising major facing mutiny after eu climb-down lon knn 8 community services the key to more jobs john quiggin says only a major policy re 74 8 international conservative crisis after budget defeat by alan wheatley of reuter knn 9 momentum against major the administration of britain’s prime minister, john majo 3 9 major crushes eurorebels for now london, saturday: the brit ish prime minister, knn 10 market hits 14-month low as base metals bear brunt melbourne: the australian sha 75 10 international mps expelled in row over eu key win saves maior bv alan whfatlfy l knn 11 business and investment steel leads bhp to record profit melbourne: steel, backe discard 11 international major humiliated in local elections london: in a humiliating setba knn 12 business and investment coal ‘rabble’ irks union australia’s coal industry was b discard 12 international major may face leadership test within weeks by don woolford of aap knn 13 the undoing of john major malcolm booker the conservative lead ers in britain, a 4 13 by-election disaster kicks off major’s dice with voters london: prime minister j knn 14 state bank sale to be finalised soon sydney: the sale of the state bank of nsw h discard 14 international tories sink into a moral stew three mps exposed . more rattling in knn 15 fears for peace push dublin, sunday: a dispute in the irish republic’s coalition discard 15 unravelling not due to major alone from hugo young in london the slow disintegra knn 16 in brief qdl expands into victoria brisbane: pharmaceutical whole– salcr and d discard 16 british tories rally behind their leader london: john major is expected to secur knn 17 elated rsl wins battle for wake’s war medals by marion frith a jubilant returned discard 17 blair rides high on eve of conference as major founders in sea of ‘sleaze’ by al knn 18 irc decision gives all workers on federal awards 11 public holidays a year by mi discard 18 international major warns tories: eu yes or a poll – by richard meares london, f 3 19 history highlights in history on may 8: 1704: british forces under duke of marlb discard 19 international major on the ropes as bribery claim minister resigns by michael wh 22 20 aids patients ‘incorrectly’ diagnosed by catriona bonfiglioli, aap medical corre discard 20 international major stakes govt on high-risk bill by donald macintyre london, th knn 21 business and investment ; i, colonial mutual rejects state bank sell-off claims discard 21 international eu vote revolt poses poll threat to tories by richard meares londo knn 22 international major on the ropes as bribery claim minister resigns by michael wh 19 22 mps cold on heat tax london, tuesday: the brit ish prime minister, john ma jor, knn 23 major’s fortunes turn around john major must be almost unable, to believe his lu 5 23 fresh row adds to tory woes bournemouth, england, wednesday: britain’s ruling co knn 24 collectables dion skinner logos of old still capture the mind in the world of ad discard 24 international pm fancies a return to banking the british prime minister, john ma knn 25 pm backs rate rise as recovery tool by, ian henderson prime minister paul keatin discard 25 elections speaking in the house of commons, labour leader john smith described t knn 26 us economy like banker’s dream the united states economy is showing signs of tur discard 26 heseltine front-runner to take over as leader the president of the british board knn 27 norman struggles, six shots off pace pittsburgh, pennsylvania: a disappointed gr discard 27 cloud is cast over major’s morals drive london: two forced resignations from t knn 28 saints fall victim to resurgent hawks melbourne: pundits proclaiming an end to h discard 28 heseltine rises from the political grave london: michael heseltine, once the gol knn47 29 , . , business and investment sale of state bank still not a sure thing sydney: discard 29 avoid a security over-reaction the attack on prince charles has set off the pred knn 30 b u sin e ssl an d investment bargain hunters help market rally shares recover a discard 30 world briefs major hangs on as party leader london: prime minister john major es knn 31 international villagers flee enclave, rebel onslaught serbs move on un safe have 76 31 they said it it could only have been expected … it’s the fault of the governme knn 32 fawcett quits in rift over fal plans perth: david fawcett, the man viewed by man discard 32 racist row over speech london: a high-flying minister has forced prime minister knn 33 adsteam shares take a dive sydney: shares in the debt laden adsteam group plunge discard 33 richard farmer liberals cannot merely sit back the liberal party’s election stra knn 34 power plays of men and women oleanna, a piece about sexual harassment, is meant discard 34 richard farmer economic news of no help to libs at first blush an oppo sition ma knn 35 us turnaround leads australian market recovery sydney: the australian stock mark discard 35 all hope is not gone for liberals peter cole-adams i.f the liberal party was in knn 36 bannister and friends remember a record oxford, england: after 40 years, about t discard 36 beggars ‘offend’ british leader london: the british prime minis ter, john major, knn 37 books discouraging view of the journalism scene in us who stole the news? why we discard 37 domestic issues dominate the elections for the euro pean parliament now under wa knn 38 1 head of gatt to quit this year . j from john zarocostas in geneva peter suther discard 38 uk budget freezes spending . london, wednesday: brit ain’s unpopular conservativ knn 39 book two decaying belief in the dentist j ust what i need: another reason not to discard 39 nz teeters towards poll resignation hits shaky majority wellington: new zea land knn 40 rates rise ‘not justified’ by keith scott the managing director of bhp, john pre discard 40 saturday forum the conjurers around the cabinet table federal politics ross peak knn 41 tjdget ’94 strategy depends on business investment recovery in 1994-95 budget cl discard 41 pm told to spend on the jobless by tom connors, economics writer the prime minis knn 42 accused mp vows to fight to e nd sydney: the embattled state liberal mp barry mo discard 42 a good test of downer’s mettle alexander downer had another bad day yesterday. h knn 43 test tickets now prized possessions ! roo tour i [notebook! by bevan hannan seat discard 43 pm keeps budget tax rise on cards by ross peake, political correspondent prime m knn 44 forestry protesters chip directly at pm australian federal police officers remov discard 44 crystal ball starts to get a bit hazy around 1999 political punditry is a risky knn 45 brave-faced lions view the future melbourne: the victim of the afl’s five-year p discard 45 crushing swing rocks british conservatives brierley hill, england, friday: the b knn 46 southern cross woden’s talent exodus leaves stephens with shaky foundation on th discard 46 kohl to call major’s bluff bonn: chancellor helmut kohl and his european allies knn 47 magic bullets’ take their aim on disease possible breakthrough in treating rheu discard 47 a state loss is a federal gain john black analyses the figures and concludes tha knn 48 loggers ready to blockade woodchip warning to govt by paul chamberlin the loggin discard 48 international bihac still under siege by kurt schork sarajevo, friday: bosnian s 1 49 small business confidence slips back a notch by ian henderson, economics writer discard 49 downer faces possible revolt on two fronts by peter cole-adams, political editor knn 50 senior libs consulted on hewson’s fall by jack waterford the four most senior of discard 50 business and investment regional banks shed value after rates increase by michae 2   the blended result is a great improvement, but it is far from perfect: blended results 33 to 44 are almost all about australian or new zealand rather than british politics, and results 63 and 67 are quite relevant to the search. result 63 (“death is another major setback london: the death of a conservative legislator in what police call suspicious circumstances adds to the troubles of prime minister john major”) ranks highly (at 21) on a “pure” semantic search, so it is being pushed down the rankings by keyword and similar-to-keyword results. result 67 (“harold macmillan’s fate haunts modern tories”) is not improved by a pure semantic search using the ada-002 embeddings, but is ranked at 34 using the clip embeddings. this result demands further investigation. the value of a semantic search is particularly evident when performing a search most naturally expressed as a question or phrase rather than a set of keywords, and it does indeed seem that the ada-002 embedding (and to a lesser extent, the clip embedding) manages to encode a commonly understood intent behind the phrase and match it successfully with articles, for example, this search on infiltration of asio by communist spies which returns zero keyword matches yet many relevant semantic matches:   figure 5. search results for infiltration of asio by communist spies: there are no pure keyword results on the left, but many relevant semantic results on the right.   another interesting example is the ability of a semantic search to find articles using current-day terminology that was not used at the time the article was written. cybersecurity was not a term much used in 1994, and a keyword search finds zero matches, but the semantic search finds many relevant articles:   figure 6. search results for cybersecurity: there are no pure keyword results on the left, but many relevant semantic results on the right.   named entity resolution when navigating very large repositories and trying to understand the context of their contents and how they relate to other resources, being able to see the people, organisations, places and events mentioned in articles and result summaries can be very useful. people especially can be referred to in many ways, often with name variations that sometimes are contextual (eg, joe biden, joseph biden, president biden, the president, j.r. biden jr (1942-), joe, mr biden, senator biden, ..), and identifying what is a name and the “real-world” entity it refers to is both useful and non-trivial. the proof-of-concept used the stanford natural language library [6] to identify named entities which were then indexed with their containing articles. this allowed displaying names in article and result summaries (the latter as facets which can be used as search filters). however, this is just the first step in providing a useful entity identification and context capability. it is valuable for an entity’s names (including their many variants) to be linked to a real-world entity. in the above example, it is probable, dependent on context, that joe biden, senator biden, president biden, etc. should all be linked to the same “real world” person represented by the wikipedia page https://en.wikipedia.org/wiki/joe_biden. this linking has not been attempted in the proof-of-concept, but an experiment in trying to resolve people and organisation names to wikipedia entities has been implemented which uses a combination of name matching and semantic similarity between the article containing the entity and wikipedia text describing the entity to generate a similarity score between people and organisation entities identified by the stanford natural language library and wikipedia articles. some examples from the first semantic match on the fall of john major search:   figure 7. entities found in a newspaper article about the fall of john major.   the first entity listed is mr major.   figure 8. wikipedia articles matching the entity mr major ranked by textual and semantic similarity.   although not the closest name match amongst at least 10 name matches on mr major, the wikipedia article on john major is the closest semantic match to the article, and reasonably confidently disambiguates the name. this article also contains a named entity for john major:   figure 9. wikipedia articles matching the entity john major ranked by textual and semantic similarity.   unsurprisingly, the better name match and same article/wikipedia similarity match very confidently disambiguates this name. the next entity is kenneth clarke (in this context, another prominent conservative british politician):   figure 10. wikipedia articles matching the entity kenneth clarke ranked by textual and semantic similarity.   semantic similarity confidently disambiguates this entity reference, as it also does for michael portillo,  michael heseltine (both conservative british politicians) and paul keating (australian prime minister, 1991-1996). however, it fails with john smith:   figure 11. wikipedia articles matching the entity john smith ranked by textual and semantic similarity.   the english politician is not even in the top 10, which is perhaps not surprising given the dozens of john smiths on wikipedia. however, this is a good illustration of the difficulties in linking named entities, particularly of common names, to wikipedia entities. these attempts at resolving named entities are encouraging but further work is needed to refine matching using names, contextual dates, semantic similarity and common co-appearing entities. scaling experiments although performance on this small repository of 45,494 documents is very good, the complete trove newspaper repository contains 220 million articles. a notable characteristic of the hnsw algorithm used by many vector search engines (including lucene/solr) is that it is a hierarchical graph search in which most nodes (representing the embedding vectors of documents) are highly connected, typically to 16 – 128 other nodes, and the algorithm proceeds by following many promising parallel paths of increasing similarity through the graph. this causes effectively random access to the data in the graph, and as a consequence, very high io rates unless the graph is held in memory. even when in memory, the number of “probes” performed navigating the graph requires very high bandwidth between memory and the processor.  for example, finding the “top 10” semantically-near results to a query with a hnsw graph storing about 200 million vectors, each linked with 60 nearest neighbours, requires about 45,000 “probes” distributed across the graph. hence, the size of the graph is of utmost importance. the maximum number of other nodes each node can be connected to has a moderate influence on the graph size, but the biggest factor is likely to be the size of the embedding vector. the ada-002 vector contains 1,536 floating point values, which equates to 6,144 bytes per vector. this uses less than 300mb for the small proof-of-concept repository, but for 220 million articles, requires over 1,300gb of memory. even on a machine with such a large memory, the need to randomly access this memory into the processor’s memory caches suggests undesirable processing bottlenecks which will reduce query performance. if the ada-002 vector contained vector positions (“dimensions”) which were highly correlated with each other or had near-constant values, then it would be possible to drop some dimensions whilst leaving search results unaffected. however, an analysis of the ada-002 vectors created for the newspaper and wikipedia articles showed there was no correlation between vector positions and no near-constant values: that is, all the dimensions seem to add discriminatory value to the embedding. however, this analysis did note that all but five of the 1,536 dimensions have almost all of their values in a narrow range between -0.1 to +0.1, whereas the five “outlier” dimensions had characteristic different narrow ranges (for example, dimension 194’s range is almost entirely -0.7 to -0.6). this suggested a low-error way to quantise the 4 byte float into a 1 byte value using six quantisations: one for each of the five outliers and one for the central “clump”, by choosing quantisation values to minimise total error. with this approach, the 1,536 float values are quantized and stored as 1,536 bytes, and at “query time”, when calculating the vector similarity, the quantisation tables are used to “expand” each byte to a float having a value close to the original float value. the proof-of-concept implements this approach as the “ada-002quant” embedding, and qualitatively, the results are equivalent to the unquantized values. this approach reduces storage/memory/memory-cache-shuffling by 75%, but still requires 340mb of memory for 220m records just to represent the embeddings. further compression can be achieved using product quantization (pq) coding [11]. with this approach, some number of vector positions, typically between 2 and 10, is represented by a single quantised value stored as typically between 4 and 16 bits. experimentation using the ada-002 embeddings gave good results using pq coding on 3 vector positions and representing them with 1 byte quantisation. this required 1,536 / 3 = 512 separate pq tables, each with 256 values, which represented 3 floats as 1 byte. each of the 512 pq tables were built using k-means clustering to minimise total error. pq coding hence reduces memory requirements by a further two-thirds, storing 512 bytes per embedding (down from the original 6,144 bytes), and allows the embeddings for 220m records to be stored in just 113gb. the proof-of-concept implements this approach as the “ada-002pqcode” embedding, and qualitatively, the results are similar to the unquantized values. chatting with a newspaper repository after the openai chat api was released in march 2023, a simple chat proof-of-concept was implemented as follows: an embedding is created for the initial user chat input (exactly as it would be for a search). a semantic search on the newspaper repository (canberra times news articles from 1994) is issued to find the 8 documents with embeddings closest to that of the user input. up to ~3,000 words are extracted in total from the 8 documents, in proportion to their relative similarity score and are used as context for the openai chat request. a system prompt and the user input are added to the article context and passed to openai’s chat api endpoint. the response is received, article references changed to hyperlinks (to the original article on trove) and shown to the user. any followup response from the user is appended to the previous inputs and sent again to openai’s chat api. figure 12. chatting data flow. as an example, a user input of “what problems did john major have in 1994?” generates the following request to openai’s chat api: "messages":[ {"role":"system","content":"you answer questions factually based on the context provided. the context consists of newspaper articles, each within their own article tag (<article>) which starts with their article number and date of issue formatted as yyyy-mm-dd. reference every article you used to construct your answer by starting every use of text derived from an article or articles by the source articles' sequence in the context, for example  [from article 2] answer text derived from article 2... [from article 8, 3] answer text derived from articles 8 and 3"},; {"role":"system","name":"context","content": "<article> article 1. date of issue: 1994-05-07. content: international major humiliated in local elections london: in a humiliating setback for prime minister john major, britain's ruling party yesterday suffered its worst defeat in local elections as traditional conservative voters deserted his crisis plagued government. diehard conservatives throughout britain switched allegiance or stayed [content removed for brevity …] norman fowler</article> <article> article 2. date of issue: 1994-02-16. content: momentum against major the administration of britain's prime minister, john major, has been deeply hurt by the continuing sex scandals which have seen [content removed for brevity …] too long in the political game-to,fall into that trap</article> <article> article 3. date of issue: 1994-04-01. content: seeds sown for conservative uprising major facing mutiny after eu climb-down london: british prime minister john major faced the growing threat of a mutiny against his leadership yesterday.. and so on for 8 articles, concluding with: .. david ashby, had shared a hotel bedroom in france with a male friend</article> "}, {"role":"user","content":"what problems did john major have in 1994?"} ] chatgptv3 frequently failed to cite the source article as requested, but chatgptv4 routinely cites correctly, resulting in output such as: figure 13. response from openai’s chatgptv4 to the query what problems did john major have in 1994? given a context of 8 semantically-searched documents from the canberra times, 1994.   followup questions are kept within the provided context:   figure 14. response from a followup question, showing effective preservation of the original question’s context.   chat provides an interesting way to summarise a large amount of content from different articles quickly and will be attractive for many people wanting to get an overview whilst still being able to “dig in” to the source material via the citation links to the original articles. further work the proof-of-concept has merely scratched the surface of possibilities and briefly explored just some of the approaches for improving discoverability of resources held in large digital repositories. some of the areas identified for further investigation are listed below. named entity identification and linking large language models have been successfully used to identify named entities and informal experiments with some newspaper articles and appropriate prompts using gpt3.5 and gpt4 are extremely encouraging, although the cost currently greatly exceeds running traditional named entity recognition libraries. it is possible that large language models, perhaps fine-tuned with content from wikipedia, will be able to accurately identify and disambiguate people, organisations and places. many of the people and organisations described in newspaper articles will not be prominent enough to appear in wikipedia. they will, however, be of great interest to some communities. how to identify and represent these, and for example, not link an unrelated “john major” to the wikipedia entry of the former british prime minister is unresolved. assuming reasonably accurate identification of named entities and linking to wikipedia/wikibase entities can be achieved, there exist great opportunities for representing content containing those entities in a broader context. for example, it would be possible to search for articles about  british conservative leaders visiting ireland without needing to specify names of people or places in ireland, by using an equivalent of google’s knowledgebase (possibly based on wikidata [12]) to find the appropriate people and places, and then using named entity indices on articles to find the intersections and finally using a semantic filter to identify those describing an appropriate named entity (british conservative leader) visiting another appropriate named entity (place in ireland). appropriate embeddings this proof-of-concept compares the clip vit-l-14 and ada-002 embeddings, but there are many other options for embeddings. clip vit is no longer considered state of the art, but was used because it was very easy to try and because the ability to map text and images to a single high-dimensional space is very attractive, providing as it does a simple, unified way to retrieve images as well as text. although ada-002 qualitatively gave better results, it has several downsides. the cost is non-trivial. nla’s newspaper archive contains around 140 billion words, which equates to around 190 billion tokens. openai has reduced the cost of embeddings dramatically, and at the time of writing (june 2023), ada-002 embeddings cost $us0.0001 per 1k tokens, giving an embedding cost of $us19,000 for the nla newspaper repository. however, the nla’s web archive repository contains about two magnitudes more text. another concern with a proprietary embedding approach is that should the vendor ever deprecate then discontinue support, the existing embeddings are unusable: incoming queries need to be converted to the same embedding model as the documents they are going to search. similarly, a future decision to increase the price of obtaining an embedding could greatly increase search costs. the hugging face massive text embedding benchmark leaderboard [13] compares the attributes of embedding models, providing a starting point to investigate a balance between attributes such as cost, openness, speed, ability to characterise long runs of text, ability to characterise various languages, ability to characterise images, effectiveness, vector length and compressibility. vector search this proof-of-concept only examined the performance of lucene/solr’s hnsw approximate nearest neighbour search engine. this is a good “fit” for nla’s technology stack, and it performed well and was demonstrated to scale to the size of the entire current newspaper repository. however, there may be better options, and vector search is a rapidly developing field. the time taken and space used to construct the hnsw graph is affected by two tunable parameters: the number of “nearest” neighbours each node should be connected to (although the hnsw algorithm seeks to have somewhat diverse rather than strictly nearest but extremely similar neighbours). how exhaustively the algorithm should search for nearest neighbours. searching the hnsw graph is also affected by a parameter that specifies how exhaustively the algorithm should search for best matches. optimising these hnsw parameters is likely to be rewarding. compressing the embedding representation (to smaller storage values such as a byte rather than a float, and by reducing the number of values by techniques such as pq coding) are likely to be necessary for large repositories. for a repository the size of nla’s web archive (15 billion documents and counting), tradeoffs between embedding accuracy and performance seem inevitably required, at least with current hardware. user interface and search logic many searches are intended to be exact keyword searches. a searcher issuing paul keating bankstown is very likely only interested in results about the named entity paul keating (the former australian prime minister) and the named entity bankstown (the sydney suburb and his birthplace), and “blending in” semantic results may result in more annoyance than joy. preprocessing the query to identify those entities will be valuable (for example, paul keating may be referenced in a sought article as prime minister keating, pj keating, mr keating, or depending on the article’s context, just prime minister). however, a search for paul keating immigration policy will benefit from semantic searching, at least on the immigration policy part. perhaps a starting approach would be to attempt to identify named entities in the query and treat them as requiring a keyword match, or, if a knowledge graph is available, a known-entity match. a last resort (admitting failure) would be to rely on a mechanism such as “advanced search” in many library systems, which basically puts the onus on the searcher to devise and communicate the search strategy. but for those trained by google, such an approach is likely to be as mystifying as it is inadequate. where to terminate a semantic result set is another interesting problem. unlike keyword searches which always have a hard cutoff (e.g., for an “anded” search, only documents with all the keywords can be returned), similarity has no obvious hard boundary: all documents are similar to all others to some degree. previously, “documents like this” capabilities operated on a “bag of words” approach, typically favouring rarer words. semantic similarity offers the potential to make “documents like this” more useful. named entities combined with a knowledge graph will allow the search interface to suggest more “abstract” facets that are not directly contained in the text of the search result. for example, a search for train crashes may usefully present facets of country or state as higher-level groupings of place-names mentioned in the found articles. a search for paul keating immigration policy may usefully present facets such as political parties or factions as higher-level groupings of people mentioned in the found articles. ocr text correction semantic search is more robust than keyword search in the presence of ocr errors because it relies on more than specific occurrences of keywords to create an embedding representing the meaning of text. however, as described above, many searches will still rely on exact keywords (e.g., searching for a particular person and place), and removing as many ocr errors as possible prior to creating embeddings and keyword indices, whether by crowd-sourcing (as successfully implemented by trove [14]), by automated ocr correction (such as overproof [15], also used by trove) or using generative ai, may allow materially more accurate searching and summarisation. summarisation and “chat” interface moving away from the traditional search result of hyperlinks, each with a document title and brief context of found keywords, will be a big change for library systems, but following the design lead of large public search engines may minimise surprise. at the time of writing (june 2023), microsoft’s bing search results show a feature search result (with section headings and summaries taken from that feature web page) followed by traditional-looking links in a left hand column, and a ai generated summary response that starts filling the right hand column and “learn more” citations used by the summary, common explanatory follow-up questions, and an invitation to “let’s chat”:   figure 15. bing’s ai-augmented search results as of june 2023.   (note that the actual search results returned will be informed by many factors such as the searcher’s location, ip address, search history, cookies associated with them and their browser and its settings.) the citations can be followed to show the “source” web page. alternatively, “let’s chat” or one of the “pre-canned” follow-up questions can be clicked to continue the chat:   figure 16. follow-up chat on bing’s ai augmented search portal.   the summarisation/chat is very effective at presenting an overview derived from multiple source documents with mechanisms to dive deeper, and this hybrid of traditional search results shown alongside summarisation/chat allows the introduction of this new capability in a gentle way that’s unlikely to disorientate people seeing it for the first time. while the choice of which model to use for creating embeddings has long-term implications, the choice of which model to use for chat/summarisation is “tactical”, and can be relatively easily changed as technology develops. although at the time of writing (june 2023) openai’s gpt4 chat api has by far the best capabilities, it is expensive to scale (a single chat interaction with a provided context of 4,000 words generating a 500 word response costs about $us0.20) and some institutions may not be happy with resources and patron questions being supplied to a commercial service. highly capable open-source models trained for chat are appearing, such as the  falcon-40b [16] and mpt-30b  [17] models. mpt-30b appears particularly attractive because its size allows it to run comfortably on a single (very) high-end graphics processor and because it offers an 8k maximum token context, the same as the base gpt4 api. a larger context means more “source” text can be provided for the chat engine to summarise or use as a base for question-answering. the proof-of-concept always selects chat context from the 8 best articles, and always selects text from the start of the articles. it is unlikely this naive strategy is optimal. instead, it is probable that sometimes more, sometimes fewer articles should be selected, and that the most relevant context won’t always be confined to some fixed amount of text at the start of the article. for web pages (rather than newspaper articles) in particular, text at the start of a page is often boilerplate and hence normally irrelevant to the chat. the proof-of-concept persists with the same context as the chat progresses, simply adding questions and the chat-engine’s response to that context. this is unlikely to be optimal: followup questions in the chat probably change the original best-choice of articles selected to provide the context. finally, a production service may need to ensure that the chat remains within the context provided, regardless of follow-up questions which may, adversarially, attempt to elicit responses that the hosting institution would rather not appear under its banner. references [1] google hummingbird. wikipedia [accessed 2023 june 28] https://en.wikipedia.org/wiki/google_hummingbird [2] introducing the knowledge graph: things, not strings, amit singhal, google [accessed 2023 june 28] https://blog.google/products/search/introducing-knowledge-graph-things-not/ [3] pagerank. wikipedia [accessed 2023 june 28] https://en.wikipedia.org/wiki/pagerank [4] trove. national library of australia [accessed 2023 june 28] https://trove.nla.gov.au [5] #fullyfundtrove. change.org [accessed 2023 june 28] https://www.change.org/p/fully-fund-trove [6] stanford natural language processing core [accessed 2023 june 28] https://stanfordnlp.github.io/corenlp/ [7] clip vit-l-14. hugging face [accessed 2023 june 28] https://huggingface.co/sentence-transformers/clip-vit-l-14 [8] clip as a service. jina [accessed 2023 june 28] https://clip-as-service.jina.ai/ [9]  hierarchical navigable small worlds (hnsw) search graph. pinecone [accessed 2023 june 28] https://www.pinecone.io/learn/hnsw/ [10] okapi bm25. wikipedia [accessed 2023 june 28] https://en.wikipedia.org/wiki/okapi_bm25 [11] product quantization. pinecone [accessed 2023 june 28] https://www.pinecone.io/learn/product-quantization/ [12] wikidata [accessed 2023 june 28] https://www.wikidata.org/wiki/wikidata:main_page [13] massive text embedding benchmark (mteb) leaderboard. hugging face [accessed 2023 june 28] https://huggingface.co/spaces/mteb/leaderboard [14] ‘singing for their supper’: trove, australian newspapers, and the crowd. marie-louise ayres  [accessed 2023 june 28] https://library.ifla.org/id/eprint/245/1/153-ayres-en.pdf [15] report on comparative search results following overproof correction of 10 million nla newspaper articles. project computing [accessed 2023 june 28] http://nla-overproof.projectcomputing.com/ [16] falcon-40b. hugging face [accessed 2023 june 28] https://huggingface.co/tiiuae/falcon-40b [17] mpt-30b: raising the bar for open-source foundation models. mosaicml  [accessed 2023 june 28] https://www.mosaicml.com/blog/mpt-30b about the author kent fitch has worked as a computer programmer since 1980 and been a partner in project computing pty ltd since 1982. he was the system architect and lead programmer of the national library of australia’s newspaper digitisation system, then of trove, and recently added pagerank to trove’s huge web archive full text index. he also develop systems at university of new south wales and for project computing. he does not speak for the nla and the work described in this article was performed independently of the nla. this work is licensed under a creative commons attribution 3.0 united states license. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – open access publishing with drupal mission editorial committee process and structure code4lib issue 15, 2011-10-31 open access publishing with drupal in january 2009, the colorado association of libraries (cal) suspended publication of its print quarterly journal, colorado libraries, as a cost-saving measure in a time of fiscal uncertainty. printing and mailing the journal to its 1300 members cost cal more than $26,000 per year. publication of the journal was placed on an indefinite hiatus until the editorial staff proposed an online, open access format a year later. the benefits to migrating to open access included: significantly lower costs; a green platform; instant availability of content; a greater level of access to users with disabilities; and a higher level of visibility of the journal and the association. the editorial staff chose drupal, including the e-journal module, and while drupal is notorious for its steep learning curve—which exacerbated delays to content that had been created before the publishing hiatus—the fourth electronic issue was published recently at coloradolibrariesjournal.org. this article will discuss both the benefits and challenges of transitioning to an open access model and the choice drupal as a platform over other more established journal software options. by nina mchale background since 1975, the colorado association of libraries (cal) has published a quarterly journal, colorado libraries. the print journal was a benefit for members of the association. in recent years, issues have consisted of six to eight articles focused on a theme of current interest to librarians and library workers in colorado and the rocky mountain region, as well as several regular columns. a content editor and a layout editor were in charge of the workflow process that culminated in the final physical product. the content editor worked with guest editors—each issue had a guest editor who developed the theme and recruited authors—and the layout editor prepared the copy for a commercial printer. at its january 2009 meeting, the cal executive board temporarily suspended publication of colorado libraries due to the high cost of printing and the fragile state of its budget. content was complete for issues 35.1 and 35.2, and content had been planned and work begun on issues 35.3 and 35.4. members of the publications committee, which oversaw publication of the journal, suggested at the time that an online, open access solution could allow continuation of the journal with much less stress on the budget. the board was initially concerned that transitioning the journal to an open access format would result in a decrease in cal membership, as it would be viewed as a loss of a membership benefit. a method for providing online access to members only was briefly considered, but the lack of a platform and the urgency of publishing the content finally made the case for open access successful. ultimately, however, the board requested that a final decision on the fate of the future of publishing colorado libraries be made after the association’s annual conference in october. at the january 2010 board meeting—the first following the annual conference—the publications committee presented a proposal to migrate colorado libraries into an online, open access format, supported by an open source software solution. the benefits that the committee presented to the board included: significantly reduced publication costs; a “greener,” more environmentally friendly format; and greater accessibility to users with disabilities. (additional benefits realized after the fact included the ability to include color photos and graphics, which had been prohibitively expensive in print, and the ability to make corrections post publication.) the annual budget for the print journal had been $6,600 per issue, for a total annual cost of $26,400. in contrast, the cost of publishing online was that of hosting—the association did not have a suitable server space for a drupal installation at the time—and registration of the domain coloradolibrariesjournal.org, which amounted to approximately $325, or slightly more than one percent of the annual print cost. the board requested that the committee return to the march meeting with a plan for migrating, and in march, the motion to migrate to an online, open access format passed unanimously. platform options: ojs, wordpress, and the e-journal drupal module even as recently as five years ago, migration of a print journal to an open source software solution would not have seemed as achievable as it is now, with the wealth of open source web software available today—not to mention the robust development communities behind them. the three platforms considered by the publications committee were open journal systems (ojs), wordpress, and drupal—more specifically, the drupal e-journal module. in the end, drupal plus the e-journal module was selected as the best fit for the journal’s needs. open journal systems, commonly known as ojs, was created by the public knowledge project (http://pkp.sfu.ca/?q=ojs) with an aim to provide a robust, open-source solution for peer-reviewed publishing. with over 7500 journals (as of december 2010) using ojs, it was a well-established and well-supported option. however, only the academic corner section of colorado libraries is peer reviewed. the rest of the publication consists of shorter, trade-related articles about 2,000 words in length, with the columns in the second half of the journal about 750 words each. also, the ojs platform contains ten roles to which editors, authors, managers, etc., may be added. these roles are not customizable, so combined with the rigorous workflow required by the peer review process, ojs ultimately seemed like overkill. (note: drupal’s e-journal module contains seven similar roles; however, these can be overridden to some extent with drupal’s flexible permissions system. more on this follows in the “challenges publishing with drupal” section.) finally, there was a desire to publish solely in html and not pdfs; ojs requires a non-web document submission process. after ojs was ruled out, wordpress was considered briefly due to its widespread use and support in online publishing, particularly blogs. as previously mentioned, a hosted solution was required as there was no web environment available to support a robust cms, and there were several “one click” installation and supported hosting options for wordpress. while drupal is generally agreed to be more difficult to learn from scratch, the content editor had been learning it for her regular job. ultimately, the existence of the e-journal module, with its journal-specific functionality, combined with the content editor’s prior experience with drupal, led to the final decision to use drupal. drupal itself is much more customizable than ojs thanks to its highly flexible and extensible theming system as well as its modularity. when the publications committee was making its software selection, drupal 6 was coming into its own, and there was a rapid expansion of the drupal community among libraries. support venues included the drupal4lib list (http://listserv.uic.edu/archives/drupal4lib.html) and the drupal interest group in ala’s library and information technology association (lita) (http://connect.ala.org/node/71787). the drupal e-journal module was developed by czech-born roman chyla, a librarian, developer, and cern research fellow. chyla released the contributed module to the drupal community in early 2006 for version 5. unlike ojs, which is a single-purpose software system developed specifically for online publishing, the e-journal module adds online publishing functionality to drupal’s multi-purpose web framework. however, chyla actually had ojs in mind when he created e-journal. in fact, he notes in the documentation for the module that he created it “to prove that it is possible to build a professional publishing system on the drupal platform;” chyla’s “proof of concept” “provides the basic functionality, organizes the published nodes around issues.”[1] the module can support multiple journals with a single installation, supports online workflow, and records an eissn, if desired. benefits to using drupal/e-journal drupal’s inherent flexibility and extensibility drupal has become such a popular cms due to its flexibility and extensibility, both in terms of its content development/management features and design, or “theming,” in drupal terms. combined with the e-journal module and the contributed content construction kit module, the core drupal taxonomy system makes creating and assembling journal issues a snap. a full list of the contributed modules and the functionality that they provide to the journal site is available in table 1. table 1. contributed modules used and functionality provided module functionality provided e-journal arranges drupal content (“nodes”) into issues administrative menu places most administrative tasks in a convenient dropdown menu available on all pages of the site, visible to site administrators only cck (content creation kit) allows creation of custom content types, including articles, columns, blog posts, etc., and the online forms for content managers to add content ck editor adds a wysiwyg toolbar for site content creators who do not know html contemplate allows for customization of templates for individual content types filefield, imagefield used together, allow for upload of images into content types (drupal 6) google analytics tracks use of the journal worldwide with web analytics mollom security and spam prevention/reporting, including captchas for forms and comments pathauto creates custom “blog-like” urls for more user-friendly and seo-friendly urls security review checks site setup for common security holes that could pose threats to the site upgrade status provides a report of which modules are ready for the next version of drupal views arranges content (“nodes”) into lists based on filters for custom displays after installing and enabling e-journal—which gives editorial staff the ability to create journals, volumes and issues (see figure 1)—volumes and issues may be created, and then a standard drupal taxonomy must be created to provide the structure of individual issues. e-journal administrative options are also added to the administrative menu, a contributed drupal module that provides site administrators easy access to the parts of the site that need administrative attention. in the taxonomy, editors create all of the journal’s parts, i.e., editor’s introduction, guest editor’s introduction, articles, and columns. this must be done prior to creating those individual content types because the taxonomy must be in place prior to the creation of individual pieces of content so that those pieces may be assigned to the correct part of the journal. figure 2 shows the full taxonomy of colorado libraries. figure 1: e-journal functionality for creating volumes and issues figure 2: taxonomy of journal parts next, content construction kit (cck) is used to create “content types” (drupal jargon) for each type of content in a journal. the 20 content types in colorado libraries are shown in figure 3. each content type developed has an associated web form that allows non-web experts to quickly create and add content to the site. the “create article” form is shown in figure 4. (incidentally, no changes were made to word counts for each type of content.) the ckeditor library and module add the wysiwyg toolbar shown in figure 4. also note the “available issues” and “journal structure” dropdowns; the former assigns the content being created by this form to the appropriate journal issue, and the latter assigns the article being created to the appropriate part of the journal as defined by the taxonomy—“articles,” in this case. the beauty of this structure is that guest editors and authors may be provided access to the site, with permissions to create specific content types, and upload content themselves, creating the journal issues almost automatically. colorado libraries’ workflow has not reached this point yet, but the editors hope to advance it to author submissions in the coming months. another benefit to allowing direct author submission in this way is that it allows authors to create and submit directly the type of publishing metadata that can slip through the cracks in the print workflow process, such as author biographical information, images and image captions. figure 3: all colorado libraries content types figure 4: article creation form once all of the content for an issue has been created, the journal editor(s) can manage the content for that issue at the “manage articles” screen, which is shown in figure 5. here, editors can change articles from unpublished to published, reassign journal content to different issues, if necessary, and also reassign the content to a different section of a journal—again, if necessary. most importantly, editors can use the dropdowns in the “weight” column to change the order of items in the issue. the end result from the reader’s point of view is shown in figure 6; this is the table of contents for issue 35.1, the first issue of colorado libraries to be published online. figure 5: manage article screen figure 6: reader’s view of the table of contents for a complete issue as shown above in table 1, eight additional contributed modules have improved the out-of-the-box drupal installation for the journal publication process; they are contemplate, filefield/imagefield, google analytics, mollom, pathauto, security review, and upgrade status. contemplate provides a means to customize individual content types if the default theme does not work as well as it could for those content types. combined, filefield and imagefield allow upload of photos and images with content types via a widget (not unlike attaching a file to an email). google analytics provides the editorial staff with rich data about site usage worldwide, and mollom provides captchas on the site feedback form as well as the individual comments on articles. pathauto automatically provides a blog-like url, which makes an article or column more easily identifiable than drupal’s standard node numbering system. for example, http://coloradolibrariesjournal.org/?q=node/83 is changed to http://coloradolibrariesjournal.org/?q=content/beyond-echo-chamber-library-messaging-social-networks. an added benefit to pathauto is that it improves search engine optimization (seo) for individual articles. security review conducts tests for loopholes that sometimes are left open in drupal installations, and upgrade status notifies site administrators of the status of all modules currently used by a site. this is particularly helpful when the time comes for a major drupal version upgrade. in terms of site design and theming, the quick turnaround necessary to publish the journal content that had been on hold for over a year due to the mandated hiatus dictated that a well-supported contributed theme be selected for the first issues, as there was no time to develop a custom theme. the site uses the giordani subtheme of the popular marinelli theme (http://drupal.org/project/marinelli). the only major design changes made to the subtheme were to replace the rotating stock banner photos of mountain ranges—which were not entirely inappropriate for the publication, save for the fact that they were mostly european ranges—with pictures of colorado library facilities and events taking place at them. an email to the statewide library email list yielded several stunning photos that were quickly and easily resized and added to the theme folder. all other design elements—colors, fonts, and layout—use the default giordani/marinelli settings. managing additional content in the cms another benefit to using drupal as a platform is that additional information about the journal workflow and processes can be included in the same “place” as the journal. for example, when the journal was published in print, there were a number of documents hosted on the cal website, including author guidelines, guest editor guidelines, editor job descriptions, a schedule of upcoming issue themes, and more. this documentation was posted in word and/or pdf documents on the cal website, was difficult to find, and quickly became outdated. the colorado libraries editorial staff did not have access to the association web site and could not easily maintain this meta-content, which exacerbated the problem of it becoming outdated. now, in drupal, all of this information—which has been expanded to include sample language for calls for contributors and sample letters to authors for guest editors, and an online style guide—is available in the online workspace shared by all contributors to the journal. the information that is presented to each user is dependent upon which of the five roles—author, column editor, editor, guest editor, and web admin—that the user plays in the journal’s management and production. for example, authors see links to “author guidelines,” whereas guest editors see both the author menu and the guest editor menu, which contains links to “guest editor guidelines” (see figure 7). the information that users are able to see and the actions that members of each role are allowed to perform are controlled in this way as well, thanks to drupal’s highly granular permissions, menu and block systems. since the information is written in html (in the basic drupal “page” content type), it is much easier to edit and maintain collaboratively among the regular editorial staff. figure 7: support documentation for guest editors. added social elements finally, the journal takes advantage of drupal’s social aspects in the core comments and blog features to encourage communication with the colorado libraries readership. the blog entries—which are displayed on the colorado libraries home page—contain news and announcements, such as the release of new issues and calls for themes, editors, authors, and articles. users can subscribe via an rss to keep up-to-date with the journal. enabling comments allows the readership to provide feedback on all parts of the journal website—articles, columns, and blog entries—and, as previously mentioned, is protected from comment spam by moderation as well as the mollom module. to date, only three total comments have been received, but one column author was able to add more information to what she had originally written, noting the success of the program that she was describing in her column, and another comment was a suggestion from a reader about the possibility of including short videos and webinars providing updates from around the state. while there are no immediate plans to post video on the site, the online platform brings suggestions like this into the realm of possibility. challenges to using drupal and e-journal in spite of these successes, the path from print to online was by no means strewn with rose petals. three significant challenges faced by the editorial team included delays to publication due to drupal’s learning curve; lack of community surrounding the e-journal module; and the way the module determines managerial and editorial roles. the most significant challenge faced during the transition from print to online was the publication delay due to drupal’s steep learning curve. the publishing hiatus requested by cal lasted fourteen months, and even after the editorial team got the go-ahead to publish online in march 2010, it was six months until content for 35.1 was published in september. in some cases, authors needed to revisit and update content that had been previously submitted and accepted. due to the technical editor’s inexperience with drupal at the time, certain aspects of the site were not constructed properly—or could have been built better—and have required or will require revisiting. one example of this is the display of author and editor headshots. shortly before the move online, the editorials staff had begun requesting and including author headshots on the title pages of articles. this practice continued in the online environment as well, improved, as noted above, by the ability to use color photos rather than black-and-white. rather than placing the photos into a template for the articles—which would have inserted the author photos into the same node with the text content—individual “blocks” (drupal term) were added for each author picture. blocks can be thought of as smaller building blocks for a site’s page layout. the drupal block system is flexible, allowing one to place a block only on one page or on as many pages as the site admin or content creator desires. one positive aspect of adding author photos in this way is that they are reusable, which is handy because there are often repeat authors in colorado libraries. however, it has quickly become clear after three issues that this is unsustainable because it generates too many blocks—20 blocks after three issues—to manage, and these are mixed in with other, structurally more important blocks. figure 8 shows the abuse of the block system. figure 8: misuse of drupal’s block system to display author photos. additionally, from a workflow perspective, anyone wishing to submit an author photo in this way would need access to drupal’s block system on the site, which is generally a level of permissions best kept to web administrators or at least more advanced content creators/developers. it is also much more labor and time-intensive to create the block, insert the content, position the block in the left sidebar, and configure the block to only show on that page, than to have it upload as part of an article, column, or editorial introduction. thankfully, e-journal comes with a better way to promote authors who contribute to the journal. a sub-module called “e-journal authors” allows journal administrators to create a taxonomy of author names in order to link content written by an author to their user profile—effectively, these would be individual “about the author” pages for all authors, that can then be maintained by the authors themselves. currently, this is not possible with colorado libraries, as authors are not granted user accounts at this time. the plan to expand access to authors to improve online workflow will make this possible in future issues. lack of “module community” another of drupal’s greatest strengths is its developer community. the most popular contributed modules have dedicated maintainers who review and release code regularly. the most popular contributed module, views, is in use on over 300,000 sites and is actively maintained. by comparison, the e-journal module not as actively maintained, and as of this writing, it is in use on only 44 sites. the last stable release of the code is dated november 2009, and the latest development release is dated february 2011. creator chyla, who notes that he developed e-journal as a proof of concept on his own time, is not sure if a drupal 7 release is in the cards for the module (email communication, 2011 aug 29). the documentation on the module is also, as chyla openly admits, meager.[2] experienced drupal users will be able to implement e-journal relatively quickly and easily, but without previous experience with and knowledge of cck and taxonomy, new users may find it difficult to implement, as this author did at the time. duplicate e-journal user roles one final issue that initially caused confusion in the colorado libraries project was an additional set of roles provided in the e-journal module. normally, drupal developers take advantage of the system’s robust user, roles, and permissions to allow users access to the parts of the site that they require to do their work. users can easily be assigned to roles, which determine a user’s permissions. in the case of e-journal, permissions are used primarily to determine who can view content at the different stages of review and publication. the roles created for colorado libraries, as previously mentioned, are web admin, editor, guest editor, and author. the preset e-journal roles, which are a slightly more flexible approximation of the prescribed and inflexible roles in ojs, are administrator, chief editor, editor, reviewer, proofreader, staff, and others. access rights for each of these roles is customizable on the “e-journal settings” page, as shown in figure 9. this secondary set of roles and permissions can be overridden by matching the predefined site roles to the roles created locally by the site administrator in drupal’s main permissions page, as shown in figure 10. however, it requires some extra work. further, if the site administrator is initially unaware of these roles, it can create headaches when content does not display as expected to users assuming locally created roles. figure 9: “e-journal settings” administrative page figure 10: overriding the default e-journal roles with locally created roles the cal communications committee—which has absorbed the publications committee—has big plans for future issues of colorado libraries, both in terms of content development and platform improvements. as for content, prior to 2006, book reviews had been published in the hard copy issues. in 2006, a decision to publish book reviews online on the cal website was made in order to get the content to the readership sooner, without the requisite delays caused by the layout and printing process. plans are underway to reintroduce book reviews in issue 36.1, now that physical layout and printing are no longer necessary. other new types of content will be introduced as the desire or demand arises, supported by the flexibility of the drupal/e-journal platform. as for the technology platform, one of the first improvements will be to provide readers with a means to download and/or print the articles as pdfs; this was a desire expressed via feedback about the online platform. unrelated to the pdf issue, while the giordani/marinelli theme has served the journal as an attractive and accessible (section 508/wcag) theme, as more content is added, it will become desirable and perhaps necessary to develop a theme from scratch that matches cal branding as well. as previously mentioned, a more complete online workflow that includes author submissions will streamline the production process even more. drupal’s mobile modules continue to improve, so a mobile site might eventually make a debut as well. finally, the editors wish to make more of the social aspects of drupal; one recent idea in this vein was to present a list on the journal home page of the most-read content with a link labeled “what’s popular?” options for discovery: repositories and databases as of the time of writing, the communications committee is appealing a rejected request to have colorado libraries indexed in the directory of open access journals (doaj). additionally, we are finishing requirements to be included in google scholar. the committee is also exploring other options for exporting data from drupal into other formats for ingest into appropriate repositories. colorado libraries has been indexed in wilsonweb’s library literature & information science full text back to its first issue, and full text is available from 1984-2008 and recently resumed with issue 35.3. the communications committee is working actively with ebsco during its acquisition of wilsonweb products to ensure inclusion in important scholarly resources for library literature. conclusion as of the time of writing, colorado libraries online has received 4,981 visits from 3,600 absolute unique visitors since issue 35.1 came online in september 2010. the visitors have come from 84 countries. readership has expanded not only in number—the 3,600 absolute unique visitors are nearly triple the cal membership—but in scope as well, from the state and the rocky mountain west to a global presence. some referring sites include course management systems from library schools. this has elevated the status of both the publication and the colorado association of libraries. despite the risk, using a drupal module that had low use and maintenance has proved successful and paid off in the short term, satisfying the immediate need to publish the final content developed for print, and migrating to online. however, if the module is no longer maintained and a version for drupal 7 is not forthcoming, an alternative will need to be found. when contacted about whether the e-journal would see a drupal 7 release creator chyla noted via email message, “if there is substantial interest and i see many people calling for it, i can do it” (email communication, 2011 aug 29). the drupal e-journal module, in spite of the shortcomings described above, has given the colorado association of libraries the ability to continue publication of colorado libraries in a much less expensive and more environmentally friendly format. the journal will continue to showcase the great work of librarians and library workers around the state and region and has helped the communications committee adjust to a modern mode of publication to better serve as a voice of the association. notes [1] e-journal : publishing system [internet]. roman chyla. [cited 2011 aug 29]. available from: http://drupal.org/node/187987. [2] e-journal : publishing system [internet]. roman chyla. [cited 2011 aug 29]. available from: http://drupal.org/node/187987 about the author nina mchale is assistant systems administrator at the arapahoe library district (colorado). her professional interests include the use of open source web content management systems in library environments, usability and user experience, and web accessibility. as a colorado resident, she is required by law to love the outdoors, where she spends much time hiking, biking, and skiing with her three kids, two dogs, and one husband. see what she’s up to professionally at milehighbrarian.net. subscribe to comments: for this article | for all articles 6 responses to "open access publishing with drupal" please leave a response below: open access publishing with drupal in january… « infodoc microveille, 2011-10-31 […] : http://journal.code4lib.org/articles/5913 […] cary gordon, 2011-11-01 you have pointed out a key issue in the adoption of a complex, modular free content management system. drupal 7, at its release, had 469 core contributors, a bit less than one tenth of one percent of its user base. obviously, that percentage of users for the e-jounal module — used on about 40 sites — would be more or less nobody. the e-jounal module was created and is occasionally maintained by a cern fellow who is also teaching and pursuing his phd. i would venture to say that this represented a huge effort on his part. he has no obligation to maintain it forever. as it has had no updates in almost a year, it is, for practical purposes, abandoned. this means that users of the module have three choices: * they can use the module in its current release as long as they like. * they can find a way of assembling the modules functionality using other modules — presumably more generic modules. * they can provide resources to keep the module up-to-date and to port it to drupal 7. if they chose the latter, and i hope that at least one of them does, it could be done several ways: – provide a coder to contribute and possibly take over the module. – offer resources to the original contributors to resume work on the module. – hire a drupal shop to update and/or maintain the module. it might be advantageous to start a drupal group (on http://groups.drupal.org/) so that you can collaborate with other module users. you can also get an idea of who might be using the module by noting who has posted issues in the issue queue. once you have a group, you can see if any of them are willing to commit resources, as well. one alternative might be to investigate the openpublish profile, which i believe contains most of the features of ejournal, with the possible exception of peer review. openpublish is a collection of core, contrib and custom modules that is sponsored by and supported (commercially) by a major drupal shop. openpublish is supported for drupal 7. nina, 2011-11-01 hey, cary! thanks for your comments. my co-editor tabby farney and i just discovered openpublish-we worked it into our lita forum presentation about this project :)-and it’s something that we will seriously consider. roman chyla was very kind when i emailed him out of the blue about the future of the e-journal module, and it was my hope that through writing this, we could either convince him to go to 7 or that we might possibly find a group of interested folks who might be willing to help maintain it, with his help and/or blessing. ilibrarian » open access publishing with drupal, 2011-11-06 […] by nina mchale, assistant systems administrator at the arapahoe library district, on the topic of open access publishing with drupal. if you’re considering moving to an open access model for your journal, you’ll want to […] thomas dodson, 2011-11-07 great article. i’ve found drupal to be a great platform for literary journals as well. i started my journal, printer’s devil review (http://pdrjournal.org) using ojs but found it’s workflow and submission process much too cumbersome for a non-peer-reviewed publication. building a submission form with drupal using the webform module, however, was a snap. i also have a lot more control over the look and feel of the journal through customizing the “journalcrunch” theme. readers also may be interested in yana, an open source project we’re working on at harvard designed to provide oa journals with a template for mobile apps using existing content: http://osc.hul.harvard.edu/yana. we’ve tested it with ojs and drupal and it’s very easy on both platforms to simply give yana the url of the rss feed for your current issue, for example, and instantly have it work as a web app. ilibrarian » 16 new library tech stories you may have missed, 2011-11-22 […] open access publishing with drupal – code4lib […] leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – automated collections workflows in gobi: using python to scrape for purchase options mission editorial committee process and structure code4lib issue 49, 2020-08-10 automated collections workflows in gobi: using python to scrape for purchase options the nc state university libraries has developed a tool for querying gobi, our print and ebook ordering vendor platform, to automate monthly collections reports. these reports detail purchase options for missing or long-overdue items, as well as popular items with multiple holds. gobi does not offer an api, forcing staff to conduct manual title-by-title searches that previously took up to 15 hours per month. to make this process more efficient, we wrote a python script that automates title searches and the extraction of key data (price, date of publication, binding type) from gobi. this tool can gather data for hundreds of titles in half an hour or less, freeing up time for other projects. this article will describe the process of creating this script, as well as how it finds and selects data in gobi. it will also discuss how these results are paired with nc state’s holdings data to create reports for collection managers. lastly, the article will examine obstacles that were experienced in the creation of the tool and offer recommendations for other organizations seeking to automate collections workflows. by katharine frazier introduction the ability to quickly identify purchase options for in-demand books can help libraries better meet and anticipate their users’ needs. at the nc state university libraries, the collections & research strategy department conducts a monthly audit of items with multiple hold requests. we also keep track of items that are lost, missing, or have been checked out for more than one calendar year. these reports are created from a combination of our local holdings data and data from our print and ebook vendor gobi (formerly yankee book peddler): each item in the report is paired with purchase information, such as price and format. collection managers use these reports to find instances where purchasing a new copy, whether electronic or print, can alleviate high demand and broaden access to our users. while reports are an important part of making sure users can access the resources they need, compiling them is a manual, time-consuming task. this article discusses a new approach to creating reports through web scraping with python, which expedites the collection of data from gobi and greatly reduces the hands-on time required to create each report. background staff in collections & research strategy are responsible for compiling two monthly reports known as the “lost, missing, checked out report” and the “multiple holds report.” these reports are created by downloading lists of lost, missing, checked out, and held items from our ils, sirsi. an average report contains up to several hundred items. to identify suitable print or ebook copies available to buy, each item is searched in gobi. the final report includes holdings data (publication date, number of circulations, home location, etc.) for each item and, where available, purchase data (format, publication date, price). whenever possible, this report includes purchase information for the newest available edition of an item: for example, if we have lost the 2014 edition of an item, the report will prefer data for a 2019 edition. unfortunately, gobi does not offer an api to make searching an easy, automatic process. despite expressing an interest in an api to our gobi representatives, they have not shared any plans to develop one. because of this, library staff have done all of this work by hand for years: searching, finding the best match, and manually inputting purchase data into a spreadsheet. this process often took anywhere from 10 to 15 hours a month per report. most of this time was spent searching for items in gobi, which can be a lengthy and imperfect process. a laundry list of obstacles gobi has many qualities that make it a great resource for collection managers, but the search experience can sometimes be difficult. one significant cause of this is the default gobi search bar, which appears at the top of each gobi webpage. this bar only allows users to search by any one of the following: keyword, title, isbn, author/editor, subject, or keyword and table of contents. for some items, searching by title or keyword is fine because the title is unique enough to retrieve only results for that particular item. for items with more general titles, this is a very inefficient way to search. regardless of a title’s uniqueness, searching by title or keyword often returns an array of results that vary in relevancy. while gobi allows users to sort (but not filter) these results by author name, this requires a user to spend time clicking through result pages until the desired author is found. to more quickly find accurate results, we set out to find a better way of searching. initial attempts to identify a better search experience in gobi included using an item’s isbn as a search term rather than title. while this did a better job of producing exact matches for a given item, we realized that gobi does not provide links between different editions of items. since we prefer the newest available edition for all items, searching by isbn did not work. we settled on a title and author search to access the most accurate results. to search by title and author, a user must navigate to the “standard” gobi search, which is located in the website’s search-related drop-down menu. while this produces higher-quality results, it causes unnecessary delays and frustration. when searching for hundreds of items, taking time to navigate to a separate search page can add up. figure 1. gobi’s “standard” search is only accessible through the search dropdown menu. even though the standard search improved result quality, it did not remove all issues with the accuracy of results. gobi orders results by “relevancy,” which means results are not sorted by date or edition number – the method that we would prefer to quickly gather information for our reports. a user must manually sort the results by publication date through another dropdown menu of options. just like the separate “standard” search page, this process eats up valuable seconds and greatly increases the amount of time it takes to collect data for our reports. with all of these issues creating delays in the search process, we decided to try a new approach. what if we didn’t have to do any of this searching by hand anymore – could this be automated? developing a scraper before beginning to design our scraper, we conducted research to see if any other libraries had run into and solved a similar problem. however, we found no scripts or tools to fit our needs. because the nc state university libraries is home to a community of python users, we decided to explore python as a potential avenue for automating these gobi searches. our process for developing the scraper consisted of identifying what the scraper would need to accomplish, bridging the gap between our ils (sirsidynix) and gobi, and troubleshooting the scraper’s access to gobi’s internal data. an important note is that using web automation to scrape gobi is not in violation of our agreement with ebsco, gobi’s parent company. another consideration is that only existing gobi customers will be able to use or adapt the following script. all information extracted from gobi using our script is accessible only through logging in with an authenticated username and password. bridging the gap between ils and gobi before beginning to write our script, we determined that data from our reports of in-demand items would have to be cleaned before it could be searched in gobi. reports are drawn directly from our ils, which outputs title information in a way not compatible with gobi’s search functions. without cleaning, a title from one of our reports might read: database systems : the complete book / hector garcia-molina, jeffrey d. ullman, jennifer widom. titles in sirsi are paired with author(s), separated by a forward slash. when an unformatted title is passed into a gobi search, gobi will return either zero results or only results with the author’s name in the title, such as studyguide for first course in database systems by ullman, jeffrey d. fortunately, cleaning data from our ils involved only a few steps. in excel, we replaced occurrences of “/*” in the title column with a blank space. the wildcard removes all author information following the forward slash. then, we used excel’s in-built =trim(cell) formula to remove blank spaces in the title string. while this quick process is currently done by hand, it is a candidate for future automation. writing the script the most important function our scraper needed to have was the ability to access and navigate web pages. we identified selenium, a python package that automates web actions, as the best module to use in developing our scraper thanks to its ease of use and robust documentation. selenium was chosen over other modules for its focus on automation capabilities; other modules, like beautifulsoup and scrapy, are more focused on data extraction, and we needed to quickly log into, navigate, and engage with the gobi website. selenium, which interacts with web browsers like a human would, relies on webpages’ internal html structure (css, xpath) to navigate pages and engage with interactive elements (links, search boxes, radio buttons). we also installed chromedriver, a tool that allowed us to automate chrome rather than selenium’s default browser, firefox. while this choice was made because we prefer to work with google chrome at nc state, the scraper would be equally operable in other browsers. the following code demonstrates how the script establishes a web browser window, navigates to gobi’s homepage, and locates the input boxes to which it needs to send username and password information. #establish webdriver (ex: chromedriver) browser = webdriver.chrome(r'path-to-webdriver') #tell browser to fetch website browser.get('http://www.gobi3.com') #find username field, send username userelem = browser.find_element_by_id('guser') userelem.send_keys('username') #find password field, send password passwordelem = browser.find_element_by_id('gpword') passwordelem.send_keys(‘password’) passwordelem.submit() after generating its own instance of a browser window, the scraper logs into gobi and begins searching for paired titles and authors. this information is supplied from our monthly reports of in-demand items. to pair titles and authors, we drew title and author information from the report into two lists and created a dictionary from these lists. #create lists for title, author, and year from original file titlelist = [] authorlist = [] datelist = [] #read in original file of report output wb = pd.read_excel(r'path-to-report.xlsx') #append column values from original file to lists titlelist = wb['title'].values authorlist = wb['author'].values datelist = wb['year'].values #create dictionary to match titles with authors zipobj = zip(titlelist, authorlist) titlesandauthors = dict(zipobj) to find the most accurate results, our scraper searches via gobi’s standard search page. the script works by looping through the dictionary of paired titles and authors and completing the following actions for each item in the dictionary. a try and except clause is used to account for items that did not include author information in the original report. #begin iterating through dictionary of title/author in gobi for k, v in titlesandauthors.items(): #find search dropdown, click it searchelem = browser.find_element_by_id('menu_li2') searchelem.click() #find "standard" option in search dropdown standardelem = browser.find_element_by_xpath('/html[1]/body[1]/div[1]/div[2]/div[1]/div[12] /a[1]') standardelem.click() #find title search bar on the standard search page: account for slow load time with wait try: titleelem = webdriverwait(browser, 10).until(ec.presence_of_element_located((by.xpath, '//*[@id="txttitle"]'))) except: continue #send title from dictionary titleelem.send_keys(k) #find author search bar on the standard search page authorelem = browser.find_element_by_xpath('//*[@id="author"]') #send author from dictionary; account for error if no author present try: authorelem.send_keys(v) except typeerror: pass we initially struggled with identifying the correct web element to use while scraping data from result pages. each result in gobi is housed in a separate html cell, making it difficult to scrape a page of results for all titles, all publication years, and all prices. after a few failed attempts to grab these individual data elements and merge them into a dictionary, we decided to instead scrape the entire page of results in one block. this action builds a web element containing a list of all results. the script then iterates through this list of items and turns each resulting web element into a text element. #find all results on results page itemelem = browser.find_elements_by_xpath('//div[@id="containeritems"]/div') #begin iterating through the results, and transform each web element to text element for item in itemelem: individualitem = item.text next, the script extracts title, publication date, format (print/ebook), and price strings from each text element and determines the “best” fit out of all captured results. to do this, we created a series of regular expressions that search for patterns matching how gobi formats each of these data points. data extracted from these regular expressions are assigned the variables of “title,” “binding,” “date,” and “price.” once clean data points are extracted, we narrow the list of results to only those matching the original item from our report. because our script has only gathered results with the correct author, we narrow by matching on title strings. for this, we decided to use a python module named fuzzywuzzy, which calculates the difference between two given strings with the levenshtein difference. because of slight formatting differences between titles from our ils and titles in gobi, the threshold for a match was set at 70%. we found that stricter thresholds caused legitimate matches to be excluded. when the script finds a title match, the title and its accompanying data (binding, date, and price) are sent to lists. lists are then combined into a dictionary called “choices.” #apply fuzzy string matching to find accurate matches (not 100 to account for punctuation) ratio = fuzz.partial_token_set_ratio(k,title) #send matches to lists; match set at 70% for best results if ratio > 70: namelist.append(title) bindinglist.append(binding) yearlist.append(date) pricelist.append(price) #add lists to results dictionary choices.update({'title':namelist}) choices.update({'binding':bindinglist}) choices.update({'date':yearlist}) choices.update({'price':pricelist}) after trying to proceed using this dictionary, we realized a need to have a more flexible format that would allow us to sort and split results. we identified pandas, a package that creates dataframes (flexible sets of data presented in rows and columns) as the ideal tool to help us with our next steps. from the “choices” dictionary, we create a dataframe representing all of the gobi results. then, we split this into two sub-dataframes representing print-only and electronic-only titles. #create initial dataframe gobidf = pd.dataframe(choices) #create sub-dataframe for ebook results ebookoption = gobidf[gobidf['binding'] == 'binding:ebook'] #create sub-dataframe for print results bestprint = gobidf[(gobidf['binding'] == 'binding:cloth') | (gobidf['binding'] == 'binding:paper')] because titles from gobi are often formatted differently than titles from our ils, the script runs another fuzzy string match to swap out the gobi titles with the properly-formatted titles from our ils. this process makes the eventual merging of the print and ebook dataframes with our original report easier by ensuring a merge on the title column will not result in mismatches. #create list to contain confirmed correct print titles correcttitles=[] #compare each paper title to original title, keep only best match for x in printtitles: correct = process.extractone(x,sirsititles) correcttitles.append(correct[0]) #swap out list of print titles for list of confirmed correct print titles bestprint['title'] = pd.series(correcttitles) in some cases, multiple matching results for an item are included in each dataframe. for any items with multiple results, we wanted to extract the “best fit” result. we defined “best fit” as the item with the most recent publication date. using pandas, this turned out to be a simple task. we applied a descending sort to the date column and removed all duplicate titles, leaving us with only the most recently published item. this step is repeated for both print and electronic dataframes. #sort dataframe by date of publication, with newest of each title at top bestprint.sort_values(by=['title','publication year in gobi (print)'], ascending=false, inplace=true) bestprint.drop_duplicates(subset='title', keep='first', inplace=true) finally, we merge our deduplicated dataframes with our report, which has been read into a dataframe named “original.” we use an outer sort, meaning that information from both dataframes will be present in the new, merged dataframe. by merging on title, we ensure that the information gathered in the two gobi dataframes will align with the correct rows in our report dataframe. once merging is complete, the dataframe is sent to excel using pandas’ to_excel function. dfmerge = original.merge(bestprint,on='title',sort=false,left_index=false,right_index=true,copy=false, how= 'outer') #merge ebook dataframe with other merged dataframe (print & original combined) newmerge = dfmerge.merge(ebookoption, on='title', sort=false, left_index=false, right_index=true, copy=false, how='outer') #send to excel file newmerge.to_excel(r'path-to-output.xlsx') this leaves us with a document that includes original holdings data from our ils, paired with information about new copies in gobi (price/publication date). figure 2. a sample of the script’s output displaying holdings information and purchase information. not pictured are several columns to the right including total # of circulations, # of holds, home location, and current status for each item. troubleshooting one significant issue we ran into while developing the script was the length of time certain searches in gobi took to complete. selenium is willing to wait for pages to load, but eventually it reaches a limit and times out, causing the script to return an error. to work around this, we included several time-out exceptions that, whenever a timeout exception occurs, override the exception and keep the script running. except selenium.common.exceptions.timeoutexception: pass including these exceptions allowed gobi enough time to fully load its results pages. the timeout exceptions most often occurred while searching for items with generic titles, such as microbiology or even michelle obama’s biography becoming. another issue we experienced was selenium working much more quickly than gobi could load. sometimes, gobi would load a page (such as the standard search page) but not display all page elements immediately. selenium, thinking the page had loaded, searched for a given element and produced an error when it couldn’t find it. we solved this by using an explicit wait, which told selenium to pause until an expected condition was met. titleelem = webdriverwait(browser, 10).until(ec.presence_of_element_located((by.xpath, '//*[@id="txttitle"]'))) in this example, the expected condition is the title element’s xpath being present on the page. once this element loads on the page, selenium can continue. until then, it is forced to wait. impact of the script since developing this scraper, the time spent per month compiling reports has been drastically reduced. a task that used to take 10 to 15 hours a month now takes only 30 minutes of manual time. this manual time includes data cleaning and writing up an email summary of the reports, which is shared out to the libraries’ collections interest group (cig). when the reports were compiled manually, they would be sent out to the libraries’ cig mid-month. now, the group receives access to them in the early days of the month. this gives collection managers more time each month to identify what needs to be purchased to fill user needs. because selenium operates in its own browser window and can run silently in the background, the implementation of this script has also greatly increased the capacity of staff to spend time on other projects. it has even allowed for the inclusion of “creative time” to imagine new projects and applications for python. conclusion since gobi does not offer an api, we used python’s powerful automation and data modules to design our own querying tool. while this scraping tool was created to address the unique needs of the nc state university libraries, this tool could be adapted to other libraries’ specifications. the scraper currently searches for both print and electronic copies; depending on a library’s collection development plan, this could be reduced to just print or just electronic copies. for libraries interested in providing maximum access to e-resources, the script’s method for choosing electronic copies could be altered. gobi displays these options in alphabetical order of provider: for instance, ebsco will always come before proquest. within each provider, choices are arranged in ascending order by the access model: 1 user copies are displayed before 3-user and unlimited-user copies. figure 3. example of electronic purchase options for an item in gobi. when considering electronic purchase options, our scraper defaults to the first choice available for each “best fit.” if a library was interested in only choosing unlimited user copies, one could refine the script to select those copies where available instead of the first option. areas for future exploration include automating the data cleaning process and adding a function to automatically select items and send them to a gobi folder. this step would expedite the ordering process for libraries not wanting to make title-by-title purchase decisions. our script produces a report that is still reviewed by a team of collection managers, but libraries with smaller collections departments or fewer staff could potentially skip this step. a complete script for automating searches in gobi and extracting purchase data can be found on the author’s github page. about the author katharine frazier is an incoming library fellow at the nc state university libraries, where she has previously served as a university library technician in the collections & research strategy department. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – editorial introduction – issue 3 mission editorial committee process and structure code4lib issue 3, 2008-06-23 editorial introduction – issue 3 code4lib journal seeks to share ideas for solving issues and, hopefully, inspire new ideas. by ron peterson the code4lib journal reaches out the code4lib journal is a unique avenue of communication within the code4lib community. unlike the other avenues which invite people in, the journal seeks to reach out to people in all areas of library work. most of the communication channels for code4lib are directed internally to the community. the mailing list provides a venue for people interested in libraries and technology to come together and talk about issues that interest them. the irc channel provides the same people a way to communicate with each other in real time. and the code4libcon gives them an opportunity to meet face to face to present and discuss ideas. but the code4lib journal attempts to disseminate these ideas and others, in order to engage people wherever they are, whether they are technologists or not. the code4lib journal aims to reach beyond the email list, the irc channel, and the conference to find people looking for technological solutions to the problems facing libraries or proposing solutions for those problems. the journal invites people to participate by contributing their successes (and failures), ideas, and thoughts to the journal by submitting articles. but it also invites people to participate as readers of articles. these readers can then contribute to the discussion by commenting on an article or implementing an idea in their library that they read about in the journal. or maybe they will take something they read about in the journal, build on it, and make it better. hopefully, issue 3 (as with issues 1 and 2) will provide that spark for someone and they will take one of the ideas presented and make it their own and take it to the next level. and once they have done that, they will come back and tell us how they accomplished their goals so that someone else can learn from their experiences. to that end, issue 3 of the code4lib journal continues the breadth and depth of articles that were established by the first two issues of the journal. what’s inside articles range from mason hall’s solution for collecting and analyzing statistics on virtual reference transactions using im to an article on using marc records to build an archival collections portal, a process that terry catapano, joanna dipasquale, and stuart marquis describe. there are articles that address solutions to everyday problems, such as how rebekah kilzer and beth black improved off-campus access to library resources at the ohio state university. and there are articles that take on broader issues like creating a controlled environment where the tools used for the preservation of digital objects can be tested and evaluated, as brian aitken, petra helwig, andrew jackson, andrew lindley, eleonora nicchiarelli, seamus ross, and jacqueline slats detail in “the planets testbed: science for digital preservation“. issue 3 also includes practical solutions for problems, like jeremy mcwilliams‘ use of the flickr api to create machine tags for a ceramics image collection. at the same time, this issue looks at the future of cataloging, with galen charlton discussing the possibility of distributed cataloging using git and bazaar to create, enhance, and exchange library metadata. if cataloging doesn’t interest you, then you can read how andrew bullen used ocr software to bring historic sheet music to life and tell the story of the iroquois hotel fire of 1903. for web developers, joshua dodson describes how he used wordpress to create subject guides. and finally, jason clark’s article, “making patron data work harder“, describes how to improve search results by tracking patron searches. the code4lib journal gets its license in order to facilitate the ability of our readers to build upon the ideas presented in the journal, beginning with issue 3 all articles are licensed under the creative commons attribution (cc-by) license. the cc-by license lets you reuse, share, and build upon the work presented in the article, as long as you credit the author for the original creation. this licensing is required for inclusion in the directory of open access journals (doaj) and to receive a sparc europe seal. code snippets included in the text are included under the cc-by license. for other code included with an article, we recommend, but don’t require, an open source license. we are contacting all authors with articles published in previous issues to request they license their previously published code4lib journal articles under the cc–by license. as mentioned above, code4lib is more than a journal. so you don’t need to wait for the next issue to learn more. even if, like myself, you weren’t able to make to the 2008 code4libcon, you can catch up with the presentations on google video or at internet archive. or you can join the discussion on the irc channel or the email list. you can learn more about code4lib at http://www.code4lib.org. code4lib is about building a community to address the problems facing libraries using technology. if you aren’t already a part of that community, i hope that the articles in this issue will reach out to you, inspire your own ideas, and draw you into our community. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – building a scalable and flexible library data dashboard mission editorial committee process and structure code4lib issue 35, 2017-01-30 building a scalable and flexible library data dashboard data dashboards provide libraries with the means to demonstrate their ongoing activities and usage in an engaging and communicative fashion. yet, due to the number of service platforms used by libraries, and the wide-ranging technical specifications they entail, bringing all of this content together in a sustainable way is a significant challenge. this article describes portland state university’s project to design and build a data dashboard based on a scalable and flexible infrastructure that would enable them to present data in a visually compelling and dynamic interface. introduction data dashboards are an excellent way for libraries to provide an at-a-glance view of their many ongoing activities. a well-organized, well-built data dashboard is easily understood, narrates a coherent story about the library, and provides readily identifiable metrics and data points that can inform evaluation and decision-making activities. in addition, because they can include a broad cross-spectrum of data sources, from circulation data, to collection counts, to server logs, dashboards make it possible to accomplish two things that are of particular value for libraries: illustrate the full extent of a library’s activities: visually representing these disparate data points alongside one another helps to thoroughly illustrate the wide-range of activities that libraries support. this can be very helpful in demonstrating the library’s impact and value to campus and community stakeholders. uncover interconnections between data: dashboards offer an opportunity to directly relate different data points to one another, and in doing so uncover previously unseen relationships between them. for example, by overlaying gate count with website visits, it’s possible to illustrate what relationship exists between the two different types of traffic. yet, taking advantage of these potential benefits is challenging. while the library generates a huge amount of data, it is far from consistent in its form or the systems that house it. as a result, there are three principal problems that libraries face in building a dashboard: disparate data sources: each library uses a number of different systems for providing services. examples include the integrated library system, interlibrary loan software, gate count trackers, web server logs, etc. each of these systems requires a unique approach for accessing and merging the data programmatically. heterogeneous data: the data structure is unique among for each of the source systems that house it. representing this data in a holistic fashion requires normalizing it across systems. accessing the data: each system provides different mechanisms for accessing the data. while some systems provide apis for retrieving data, or the ability to export reports, in some cases you may need to screen-scrape web pages to extract the data. in spring 2016, the portland state university library undertook a project to engage with these challenges and build a public-facing data dashboard and the technical infrastructure to support its ongoing maintenance. the data dashboard is part of the library website, and is aimed at illustrating the library’s broad range of activities and the relationships between them. most importantly, it is a first step towards enabling the library to more effectively analyze the vast range of data available to it. portland state university library’s dashboard project as a first step in tackling this project, the library included the following goal in its 2015-2016 annual plan: “develop the technical infrastructure and workflows for collecting and reporting on library metrics across systems and units.” the library technologies unit was tasked with leading work on this goal, and began work on it mid-way through the year. given the broad nature of the goal and the sheer size of the data challenge that the library faced, the scope for the project was quickly narrowed down to the creation of a data dashboard for the library website. the presentation of the library’s annual data on the website had always been rather simplistic, with tables and bulleted lists including a limited number of facts and figures from the previous fiscal year: figure 1.screenshot of previous library statistics page. therefore, focusing on this specific goal enabled the project to achieve two good outcomes in one step: 1) replacing the current content with a more compelling and engaging dashboard, and 2) developing the infrastructure for managing and reporting on library data in a more extensive way in the future. the project got underway with the formation of the project team, that included library technologies’ two developers (mike flakus and chris geib), our content strategist (sherry buchanan), and myself, the manager of library technologies. at the outset, several decisions were made to focus the project’s scope in order to meet our brief six month timeline. specific decisions included: the audience for the dashboard was defined as campus stakeholders (administrators, students, staff) and library administrators. the dashboard would be focused on including data that could be harvested on a monthly basis and that was of enduring value to the library. the process for harvesting and storing data had to be sustainable in terms of ongoing staff time to maintain the system, and so needed to be automated and reuse application modules wherever possible. the infrastructure for storing, making available, and presenting the data had to be as scalable as possible, and so it needed to be possible to add new volumes of data and new data sources with minimal effort. the interface needed to be visually compelling and as interactive as possible. data reflecting all of the library’s principal public-facing services needed to be included, such as circulation and electronic resource usage, reference and instruction, and both online and in-person traffic. the dashboard would be focused on ongoing, repeated reporting, as opposed to ad-hoc reporting. with the scope defined, the project team broke the overall project down into discrete phases, to best structure our work and map out our timeline: identifying data sources, evaluating available data, and selecting target data sources selecting the data store, schema, and harvesting data building the user interface extracting and caching the data for repeated use in production step 1: data sources our first step was to identify the data sources that we would harvest information from. as with most libraries, the portland state university library makes use of a number of different systems to support all of our services, each of which is managed differently. some are vendor-hosted, some locally-hosted, some have apis while others don’t, etc. the sources we chose fit each of these models, giving us an opportunity to develop tools for working with most any data source. the specific data sources were: alma integrated library system and primo discovery system: the alma/primo system is vendor-hosted, and includes an analytics engine that has a dedicated api for extracting reports google analytics: vendor-hosted, but offers an api for reports illiad interlibrary loan: this application is locally-hosted, and while it does not offer an api, we do have the ability to query the sql server database directly gate count: this is a locally-developed application for storing gate count statistics, and whose database we can query directly study rooms booking system: this is a locally-developed application for reserving and checking out study rooms, and whose database we can query directly libstats: this is a locally-hosted open source application that we use for tracking reference statistics, and whose database we can query directly instruction stats: this is a locally-developed application for tracking instruction session statistics, and whose database we can query directly pdxscholar: this is the library’s institutional repository, which is vendor-hosted, and doesn’t offer an api or extractable reports. instead, we needed to screen-scrape to get the data that we wanted. each of these systems houses a wide-range of data. but for the purposes of the dashboard, we focused on data that would clearly demonstrate high-level trends over time, and that would paint a broad picture of the library’s ongoing activities. so after discussion with a range of stakeholders, we settled on including monthly data for each of the following data sets: website visits for all of our sites (e.g. main website and all subsites, libguides, pdxscholar, primo discovery system, ) (google analytics) walk-in (gate count) traffic (gate count app) study room checkouts (study room app) overall item checkouts (alma) item checkouts by specific item types (alma) consortial borrowing counts (alma) interlibrary loan borrowing counts (illiad) reference question counts by question type (libstats) student counts for instruction sessions (instruction stats app) we also determined that the dashboard would include a table of annual usage numbers, which would be derived from the harvested data, and a chart illustrating the library’s collections. this last one would be hard-coded, since the data was collected annually. we also wanted to include information reflecting the usage of our electronic resources, but at the time of implementation that data was not available programmatically (though we were able to add it by hard-coding it into the system). step 2: data storage and harvesting with the data sources selected, we turned our attention to how best to store the data itself. our initial schema for the data, based on our vision for how we would use it, looked like this: name: the descriptive name of the data being collected, which would be used for display purposes on the dashboard counts: the harvested count(s) data month: the month the data was accrued year: the year that the data was accrued tags: descriptive term(s) for categorizing the data, which would be used for grouping related data sets together the counts and tags fields were especially challenging to handle when thinking about our storage mechanism. while some data points would include a single number for counts, such as “website visits”, others would include more granular data and multiple numbers for the counts, such as circulation data including “book checkouts, dvd checkouts, laptop checkouts, …” as a result, the storage mechanism needed to be flexible enough to accommodate a variable number of granular counts, with no upper limit. the same was true for the tags field, which would need to accommodate a variable number of values for any given data point. tags would be one of the critical aspects of the system, as we planned to use these to dynamically build groups of data sets and related visualizations. up until this point, we had principally used mysql for all of our data storage, while recently adding memcached and redis for specific applications. but, none of these three solutions were optimal for this use case due to the variable nature of the data described above. so, for maximum flexibility we opted to use mongodb (https://www.mongodb.com/), one of the many “nosql” databases that have risen to prominence in the last several years. mongodb allows for data to be added to the system without a predefined schema, and has a robust query interface for retrieving data. and it can scale to support as much data as we could possibly throw at it. with mongodb selected, we designed our data model for the initial implementation. mongodb stores data sets as “documents” that are part of a “collection” (akin to mysql’s rows and tables), and so the document for a set of harvested data would use the following schema: document name: (string) raw_data: (hash) -[data point name (string)]: [data point value: (integer)] -… month: (integer) year: (integer) tags: (array) -[tag name (string)] -… (this excludes the automatically included fields, all of which are denoted with an underscore in the name (e.g. _id) in the examples below) here are a couple examples of how this schema would play out for specific data points: *data collected with a single raw_data count { "_created": "fri, 02 dec 2016 18:08:52 gmt", "_etag": "ea2e8a6078cf32a4ebedaadf7b4a43908a5d40e5", "_id": "5841b8b43a3d5f05b79c71aa", "_links": { "self": { "href": "library_stats/5841b8b43a3d5f05b79c71aa", "title": "name" } }, "_updated": "fri, 02 dec 2016 18:08:52 gmt", "month": 7, "name": "website visits", "raw_data": [ { "website visits": "63817" } ], "tags": [ "website", "traffic" ], "year": 2014 } *data collected with a multiple raw_data counts { "_created": "fri, 02 dec 2016 18:09:20 gmt", "_etag": "fc55dca90348b7fe20751ec4be676b734da61d04", "_id": "5841b8d03a3d5f05b79c7257", "_links": { "self": { "href": "library_stats/5841b8d03a3d5f05b79c7257", "title": "name" } }, "_updated": "fri, 02 dec 2016 18:09:20 gmt", "month": 11, "name": "reference questions answered", "raw_data": [ { "email": 164 }, { "chat/text": 344 }, { "in person": 2066 }, { "phone": 139 } ], "tags": [ "reference" ], "year": 2016 }, while mongodb doesn’t enforce a schema on its own, we did need to ensure consistency in the centrally stored data, and so we implemented a python library named eve (http://python-eve.org/) that would function as middleware for enforcing consistency. eve enables you to define a schema for each collection in your mongodb database, and adds an api for the mongodb database that validates data as it’s saved, before pushing it to the database. using eve, data is saved to the database via the eve api, instead of being passed directly to the database, and data that does not meet the eve-defined schema will be rejected. eve enables us to be sure that data that is saved will have a consistent schema, while at the same time still benefiting from the flexibility in data structures that mongodb offers. in other words, while the schema sets a structure for the data, within that structure we can proliferate data as needed. for example, the raw_data section of each document stores a hash of key-value pairs describing the data from the source system. in practice, this can store any number of hashes, describing sets and subsets of data as needed. eve allows us to ensure that the data being saved to raw_data will be a hash and that it is present, but within that relatively broad definition, we have the flexibility to add any number of hashes as are appropriate for the data set. schema definition in eve: schema = { 'name': { 'maxlength': 128, 'minlength': 1, 'required': true, 'type': 'string', 'unique': false, }, 'tags': { 'maxlength': 64, 'minlength': 1, 'required': true, 'schema': { 'maxlength': 64, 'minlength': 1, 'type': 'string', }, 'type': 'list', 'unique': false, }, 'raw_data': { 'minlength': 1, 'required': true, 'schema': { 'type': 'dict', 'required': true, }, 'type': 'list', }, 'month': { 'max': 12, 'min': 1, 'required': false, 'type': 'integer', }, 'year': { 'min': 2000, 'required': false, 'type': 'integer', }, } with the database and data validator in place, we wrote mechanisms that would pull data from each of the systems listed above, and then pass it to the eve api to be stored in the database. as noted previously, each system had different mechanisms for accessing the data, most using apis or queries directly to the database. but the common element in each process was formatting the data for the api. here’s an example of what that looks like when using python to format the data from alma, with the key section highlighted in bold: def reformat_to_json(report_data, name, month_filter, key='key', tags=[]): soup = beautifulsoup(report_data, features='xml') this_year = date.today().year this_month = date.today().month 1 group_name = "circulation" type_name = "circulation_all" library_stats = [] for row in soup.find_all('row'): material_type = str(row.column1.string) if material_type != 'other': # we don't want type other month = str(row.column3.string) year = str(row.column4.string) key = str(row.column5.string) # 'key' argument is ignored value = str(row.column6.string) if month_filter: if month == str(this_month) and year == str(this_year): library_stats.append({ 'name': name, 'tags': tags, 'raw_data': [ {key: value} ], 'month': month, 'year': year, }) return library_stats in this example, the data stored in the library_stats object is passed through the eve api and stored in mongodb. we opted to write individual connectors for each data source, with each connector capable of pulling multiple data sets as needed. for example, if we queried the alma api for multiple data sets (e.g. checkouts by type, and resource sharing requests), the same connector would be used for collecting both and pushing the data to the database, and the above code block could be used for processing the data from each individual query to alma. step 3: building the dashboard principal goals for the dashboard were for it to be both visually compelling and able to present data in an analytically interesting way. we had used the google charts api in the past, which offers a nice set of tools for formatting and displaying data. but after doing some research into other options, we quickly settled on the c3 javascript library (http://c3js.org/). c3 is built on top of the d3 javascript library, which is great for creating rich, dynamic charts. yet, d3 can be challenging to leverage on its own, which is where c3 comes in. c3 makes it simpler to build charts with d3, by abstracting out the javascript code needed to build the chart itself, enabling you to simply pass your data and the chart type (you can see examples on the c3 site) to c3, which will then build the chart based on this. with the chart library selected, we set about embedding the dashboard in our website. to do this, we developed shortcodes for each chart. each shortcode includes all of the variables needed to construct and display the chart, and can be used to dynamically build and place it in the appropriate place on the page. for example, this is a shortcode for including the circulation count data: [chart name="materialscirculated" type="bar" title="materials circulated" caption="users borrow more than 28,000 psu-owned items in certain months."] in this example, the variables will be parsed as: chart name=“materialscirculated” is the name of the tag that will be used to wrap the chart type=“bar” indicates that the type of chart that will be used will be a bar chart title=”materials circulated” is the text that will be used as the label for the chart caption=”users borrow…” is the text that will be used as the caption for the chart in turn, a php script parses the variables and builds the block of html and javascript that will embed the chart on the page: function chart($attr){ $result = ""; if(isset($attr['title'])) $attr['title'] = str_replace("&amp;","&",$attr['title']); if(isset($attr['name'])) { switch($attr['name']) { case 'walkin_v_website': $result = '<div id="'.urlencode($attr['name'])."_".urlencode($attr['type']).'"></div><div class="chart-caption">'.$attr['caption'].'</div><script src="//yourcontentserver.edu/static/data_dashboard/walkin_v_website_bar_and_spline.js" charset="utf-8"></script>'; break; ... default: $result = '<div id="'.urlencode($attr['name'])."_".urlencode($attr['type']).'"></div><div class="chart-caption">'.$attr['caption'].'</div><script src="//yourcontentserver.edu/static/data_dashboard/'.urlencode($attr['name'])."_".urlencode($attr['type']).'.js" charset="utf-8"></script>'; } } return($result); } in this code example, specific charts that are more complex (such as the website versus walkin chart that overlays a spline chart on top of a bar chart) are called out and handled specifically, whereas the majority of charts are handled by the default case. the resulting block of html/javascript that is added to the page looks like this: <div> <div id="materialscirculated_bar"></div> <div class="chart-caption">in one month, users may borrow more than 28,000 psu-owned items.</div> <script src="//yourcontentserver.edu/static/data_dashboard/materialscirculated_bar.js" charset="utf-8"></script> </div> the javascript file referenced is then called when the page is loaded, and in turn constructs the c3 chart: var full_column_data_materialscirculated_bar = new array(); full_column_data_materialscirculated_bar.push(['x','dec','jan','feb','mar','apr','may','jun','jul','aug','sep','oct','nov']); var mobile_column_data_materialscirculated_bar = new array(); mobile_column_data_materialscirculated_bar.push(['x','aug','sep','oct','nov']); var colors_materialscirculated_bar = new array(); colors_materialscirculated_bar['previous 12 months'] = '#00759a'; colors_materialscirculated_bar['previous 4 months'] = '#00759a'; colors_materialscirculated_bar['current 12 months'] = '#a8b400'; colors_materialscirculated_bar['current 4 months'] = '#a8b400'; full_column_data_materialscirculated_bar.push(['previous 12 months','15378','23576','25526','25222','28040','26295','18107','10178','8286','9681','27936','25797']); full_column_data_materialscirculated_bar.push(['current 12 months','15513','23394','23890','20997','23058','22155','14359','8017','7749','9410','24133','20151']); mobile_column_data_materialscirculated_bar.push(['previous 4 months','8286','9681','27936','25797']); mobile_column_data_materialscirculated_bar.push(['current 4 months','7749','9410','24133','20151']); var chart_materialscirculated_bar = { bindto: '#materialscirculated_bar', title: {text: 'materials circulated'}, data: { x : 'x', colors: colors_materialscirculated_bar, columns: {}, type: 'bar', groups: [ [''] ], order: 'asc', }, tooltip: { format: { value: function (value, ratio, id) { var format = d3.format(',') return format(value); } } }, color: { pattern: ['#00759a', '#a8b400', '#dc9b32', '#a33f1f', '#60351d', '#474334', '#e6dc8f', '#650360', '#6a7f10', '#dff2f5'] }, axis: { x: { type: 'category' }, y : { tick: { format: d3.format(",") } } } }; var materialscirculated_bar_chart = c3.generate(chart_materialscirculated_bar); by default, we opted to include twelve months of data for each chart, with most presented as year-to-year comparisons by month. for example: figure 2. example 12 month chart. our website is responsive, and we needed the dashboard to be responsive as well. so we configured the dashboard to present an appropriate amount of data for each chart depending on the screen size, in this case reducing the number of columns when the screen size became too small: figure 3. same chart as above image, dynamically modified based on screen size. step 4: putting the dashboard into production the last step in building and launching the dashboard involved moving away from the need to query the database each time the page was loaded. to do this, we built an automated process to extract the needed data from the database on a scheduled, monthly basis and store it in json format on a local web server. this would ultimately simplify the process (and the overhead associated with it) for retrieving the data, since the json formatted data could be easily parsed using javascript, and would result in much improved performance relative to having to query the database multiple times whenever the dashboard was loaded. for the initial launch of the dashboard, all of the data was extracted to support the included charts. the compiled json data was formatted similar to this: {"in person": {"201412": 1256, "201501": 2366, "201502": 2043}, "chat/text": {"201412": 217, "201501": 423, "201502": 319}, "email": {"201412": 249, "201501": 280, "201502": 296}, "phone": {"201412": 62, "201501": 99, "201502": 112}} in turn, when the dashboard is loaded and the charts are generated, they pull the data from these static files on the web server. next steps & future development as as a result of this project, the library has a dashboard that is dynamic and engaging for users, that requires a sustainable level of staff time to maintain, and that can scalably incorporate additional data sources and data points over time. but at the outset of the project we’d set a number of additional goals that we wanted to move forwards on, and shortly after having completed this initial project, we’ve already begun working on these. dynamic dashboards chief among these, is the ability to dynamically create additional dashboards, by leveraging the tag structure of the stored data. as i mentioned earlier, each stored document includes one or more tags, such as “circulation” or “study rooms”. the tags have a flat structure, meaning that there is no hierarchy among them. this enables us to make them as granular or broad as we’d like, and to easily add them to documents in the future. once the tags are associated with a document, querying the database by them is very simple. for example, to retrieve all documents in the database that include circulation data, you would just add the following elements to your base mongodb url: [base url]/v1.0/library_stats?where={“tags”:”circulation”} this would be the data returned for this query: { "_items": [ { "_created": "fri, 02 dec 2016 18:08:43 gmt", "_etag": "31e1693b93850f5b0149b0ca2f4d3ff900025a75", "_id": "5841b8ab3a3d5f05b79c71a8", "_links": { "self": { "href": "library_stats/5841b8ab3a3d5f05b79c71a8", "title": "name" } }, "_updated": "fri, 02 dec 2016 18:08:43 gmt", "month": 11, "name": "summit regional borrowing", "raw_data": [ { "books, journals, and audiovisual": "2801" } ], "tags": [ "circulation", "resource sharing" ], "year": 2016 { "_created": "fri, 02 dec 2016 18:08:51 gmt", "_etag": "592029b2113c4c5d7d1a842273ef96e79f6e9ee7", "_id": "5841b8b33a3d5f05b79c71a9", "_links": { "self": { "href": "library_stats/5841b8b33a3d5f05b79c71a9", "title": "name" } }, "_updated": "fri, 02 dec 2016 18:08:51 gmt", "month": 11, "name": "summit regional lending", "raw_data": [ { "books, journals, and audiovisual": "289" } ], "tags": [ "circulation", "resource sharing" ], "year": 2016 }, …] to limit this to a specific timeframe, such as a month, you add that element to the url query string: [base url]/v1.0/library_stats?where={“tags”:”circulation”},{“month”:9},{“year”:2016}]} this ability to dynamically retrieve all data for a given tag (or tags), is then coupled with standard definitions for how to present the data that is retrieved from the database. for this, we’ve set the following rules: retrieved data is presented in charts that are grouped by the documents’ “name” field, and document groups are sorted by “name”. data that is multifaceted (i.e. has multiple individual counts within the raw_data field) will be presented with three charts: bar chart with summary data by month, for 12 months, presented as month-to-month, year-to-year comparisons (see the study rooms usage chart above for an example) stacked bar chart presenting each granular count relative to one another, for 12 months. pie chart, presenting the data broken down by the granular counts, summarized for 12 months. in comparison, data that is not multifaceted will be presented with just a month-to-month, year-to-year bar chart, as in the study rooms example above. users will be given a choice of time periods to choose from, such as 6 months, 12 months, etc. figure 4. example of how the dynamic dashboard would render charts for multi-faceted data, while offering a drop-down for additional time-range options. setting out these rules enables us to programmatically build these tag-based dashboards dynamically. in turn, in our curated dashboard where we’ve laid out which charts appear, how they are grouped, and in which order, we can link out to the dynamic dashboards to offer users access to more granular data. for example, we can display specific charts in the circulation & resource sharing section of the dashboard, and then link out to a dynamic dashboard for the tags “circulation” and “resource sharing” that will include a range of charts that present this area of data in greater detail. ultimately, we will be able to take advantage of these tags and the programmatic logic behind the dynamic dashboards to scalably tap into the stored data to give users (whether internal or external) insight into the data in ways that are most useful and relevant to them. annual data our initial project was focused on collecting monthly data. but historically our library’s “data” page included annual data too, and we want to include this in the dashboard as well. data such as the library’s collection profile, and the annual data summary for total circulation and other data, are both key elements for the finished dashboard. during the initial project, these charts were simply hard-coded, but we want to develop a long-term solution for them. the chart for the annual data summary can reliably include information that is summarized from the data stored in the database. and so this chart will be relatively straightforward to include. the biggest challenge will be determining the optimal means of extracting and summarizing that data, and then storing it and presenting it in the dashboard. in contrast, the annual collection profile data is very different from the data collected thus far. that data reflects our collection counts by material type at a specific point in time, and not how the counts are accumulating or decreasing month by month, the way something such as monthly circulation data does. as a result, how we derive and store the data needs to be carefully considered, so that it can comfortably be stored in the database alongside the monthly data, without causing issues in data retrieval and presentation. at the moment, the collection profile chart only displays on the curated dashboard, and so we can easily control how the data is retrieved and presented. if we automate collecting the data on a monthly basis, it will need to be incorporated into our current structure for harvesting and storing data. long-term, by collecting this data on a monthly basis and tagging it appropriately, we will also be able to leverage the tag-based workflow for the dynamic dashboards to illustrate trends in our collection profile over time. additional data sources we have initially included the principal data sources that were already being used for reporting in the library. but this is a subset of all of the library’s relevant data sources, and there are additional ones that we would like to include in the long-run. for example, ezproxy data would be very useful to include for giving us a picture of patrons’ use of our electronic resources, particularly in terms of usage trends over time, and illustrating onvs. off-campus use. and ezproxy data is sufficiently well-structure that it would flow very well into the architecture that we have developed already. in addition, our university uses a program called labstats for tracking computer usage across campus labs. tapping into this data would enable us to illustrate the use of library labs and technology in other spaces in the building, and to juxtapose this with usage info from other labs on campus. these are just a couple of examples of other data sources that might prove valuable to incorporate as additional phases down the road. as i mentioned at the outset of this article, libraries have many data sources available to them, and mountains of data within them to creatively tap into. our overall goal is to continue to take advantage of the infrastructure that we’ve developed to make creative, effective use of this data. open source lastly, we plan on developing and sharing open source versions of the components of the system that will enable other libraries to take advantage of the work that we’ve done. particularly since so many libraries use similar software for systems such as their integrated library system, interlibrary loan services, and proxy server, it will be possible to share source code that would enable libraries to tap into these common systems to create dashboards. in addition, the structure for how the application stores and retrieves data is essentially source-system agnostic. the only part of the system that is thoroughly wedded to the source system for any given data set is the harvesting process. but even here, as long as data can be harvested from the source system and processed through the eve interface for the mongodb database, it would be possible for libraries to build additional system connectors with relative ease. the portland state university library’s github page (https://github.com/pdxlibrary) already includes a number of other projects that we’ve created open-source versions of. when we release the code for this project in spring 2017, this code will be included there as well. conclusion tapping into the library’s wealth of data to fuel ongoing evaluation and decision-making is a challenging project to undertake. it is particularly complicated by the broad array of systems that each library uses, and the need to arrive at a sustainable means of collecting and reporting on data in an ongoing fashion. but, arriving at a solution for this is a project that is ultimately of great benefit to libraries. such a solution represents an opportunity to overcome data silos that prevent us from effectively leveraging data to support our ongoing work and strategic planning. harvesting data centrally enables us to directly compare it with like data, offering insights that would have been difficult to obtain otherwise. what is more, because there are a number of commonly used library systems in place today, the development of such a solution would potentially benefit a many libraries. the portland state university library’s project has provided an initial glimpse of what is possible with such a solution. we have a much greater ability to take advantage of data today than we did a year ago, and as we learn more and add new functionalities, our capabilities in this area will continue to improve. our project also illustrates the type of steps that other libraries will need to consider if/when embarking on their own solutions, providing a roadmap of sorts for libraries who are also seeking to better leverage the data available to them. about the author nathan mealey (mealey@pdx.edu) is the manager of library technologies at portland state university library, where he oversees the team of technologists who develop and maintain the library’s web-based and desktop technologies. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – indexing linked bibliographic data with json-ld, bibjson and elasticsearch mission editorial committee process and structure code4lib issue 19, 2013-01-15 indexing linked bibliographic data with json-ld, bibjson and elasticsearch linked data is a powerful tool for sharing bibliographic metadata. by combining the decentralization of the web with the use of globally defined metadata vocabularies, data from many sources can be treated as a single, aggregated graph. supporting search across these distributed data sources within the same application, however, requires considerable work in vocabulary alignment and data transformation. aggregate systems must convert data into a unified model which must (almost inevitably) be generic at the expense of the structure and granularity of the original data. this paper presents a novel solution for representing and indexing bibliographic resources that retains the data integrity and extensibility of linked data while supporting fast, customizable indexes in an application-friendly data format. the methodology makes use of json-ld to represent rdf graphs in json suitable for indexing with elasticsearch. bibjson is used as a common index format capable of handling a wide range of library resources. since all three technologies (rdf/json-ld, bibjson and elasticsearch) share an emphasis on extensibility, it is possible to create an index of bibliographic data that is both generalized and flexible enough to handle linked data from multiple sources. by thomas johnson introduction linked data is a powerful tool for “sharable, extensible, and easily re-usable” bibliographic metadata (baker, 2012). by combining the decentralization of the web with the use of globally defined metadata vocabularies, data from many sources can be treated as a single, aggregated graph. working with this chaotic mass of data, however, can be daunting. each major dump of bibliographic data comes with its own quirks in terms of vocabulary choice, scope, and data model. combining multiple heterogeneous data sources for use in the same application typically requires considerable work on data transformation. even within the context of a single domain, there is little realistic possibility for a universal schema. aggregate systems must convert data into a unified model which must (almost inevitably) be generic at the expense of the structure and granularity of the original data. one place where this is a clear problem is in search. we want linked data search engines to expose data from across the web, but with a degree of integration that insulates users from the specifics of the models. how can we bring data from distributed sources together onto a single search platform? this paper presents a novel solution for bibliographic resources which retains the data integrity and extensibility of linked data while supporting fast, customizable indexes in an application-friendly data format. the methodology makes use of json-ld to represent rdf graphs in json suitable for indexing with elasticsearch. bibjson serves as a common index format capable of handling a wide range of library resources. since all three technologies (rdf/json-ld, bibjson and elasticsearch) share an emphasis on extensibility, it’s possible to create an index [1] of bibliographic data that is both generalized and flexible enough to handle linked data from multiple sources. the method demonstrated here was developed at oregon state university as a part of an ongoing project to build search services atop an rdf dataset for our theses and dissertations. additional information about this project can be found in johnson and boock (2012). json-ld json-ld aims to express linked data in otherwise normal looking json documents. by design, it is “as simple as possible, very terse, and human readable” (lanthaler and gütl, 2012). though there had been previous attempts to express linked data in json (see rdf json and jron), this specification emphasizes application and developer friendly json (lanthaler and gütl, 2012). it forgoes the rigid, unnatural structures seen in past serializations in favor of compact representations and supplementary documents which hold the details of the graph. while the specification is a work in progress, json-ld largely succeeds. it makes it possible to introduce linked data principles and vocabularies into existing json data without changing the data structure or application code. by the same stroke, the simple, readable structure of its json makes it easy to use existing rdf graphs with typical json programming practices and with systems like elasticsearch. contexts and framing working with json-ld requires some understanding of linked data—especially the practice of using internationalized resource identifiers (iris) to refer to terms and concepts—and several concepts used to apply its principles in json. chiefly, contexts and frames. a context is a mapping between json properties and equivalent iris. contexts are themselves expressed as json objects, using the “@context” keyword, which express equivalencies between iris and more readable json keys. given an applicable context, a json-ld document can be compacted into a simple, usable form or expanded from “normal” json to its full linked data representation. for example, a context might specify: "name": "http://xmlns.com/foaf/0.1/name" an expanded document would use the longer foaf:name iri in its key-value pairs, while a compacted one would simply use “name”. a context document consists of a number of statements of this form, designating the relationship between json keys and rdf nodes. the concept behind framing is somewhat more complex. frames complete the mapping from a particular rdf graph to a corresponding json tree. this is important, since most graphs can be represented as many distinct trees (e.g. by selecting a different node as the root). specifying a single structure allows a one-to-one relationship between the source rdf and the json-ld representation. the resulting json tree is predictable enough for applications to rely on. a frame can be helpfully thought of as a template for a generated json document. the simplest framing documents might contain a single line telling the algorithm to treat the data as a representation of a number of books (contrasted with a number of authors, containing lists of their books): "@type": "book" further key-value pairs can tell the framing algorithm to expect other data to appear within the root object and specify embedding behavior to ensure that child nodes are described fully each time they appear [2]. the most useful way to learn these concepts and their various quirks is to try them out. the json-ld playground is an easy way to do this. the examples there are instructive; more importantly, it allows you to interactively construct your own json-ld and view it in various stages of compression and normalization. bibjson though json-ld represents a flexible way to convert rdf to json without degradation, a general purpose bibliographic index will need to share a common json format. our target format needs to be simple and extensible; it must be able to accommodate the needs of various record types and models, and capable of representing commonly indexed fields in a predictable way. bibjson comes close to being ideal for our purposes. at its core, it is little more than a set of conventions for using bibtex fields as json keys, made extensible by supporting arbitrary additional keys as needed. those fields cover the most important metadata fields associated with bibliographic records; indeed, bibjson’s primary existing use case is the creation of a distributed and portable collection of bibliographic data (jones, 2011). though it won’t work as a universal format—it is distinctly record-centric and won’t fit complex entity-relationship models, as we’ll see later—it does surprisingly well. in particular, it offers clarity surrounding most important search fields and will extend to fit json-ld, so long as some bibligraphic entity is used as the root node. { "title": "indexing linked bibliographic data with json-ld, bibjson & elasticsearch", "author":[ {"name": "thomas johnson"} ], "type": "article", "year": "2013", "journal": {"name": "code4lib journal"}, "issue": "19", "link": [{"url":"http://example.org/tjohnson-2013"}], "identifier": [ {"type":"doi", "id":"10.1000/182", "url":"http://dx.doi.org/10.1000/182"} ] } figure 1. an example bibjson record. a basic bibjson record for this article is given in figure 1. as a rule, simple data points that can be represented with a string (e.g. title) are given as key-value pairs. for more complex fields, bibjson allows representation as an object or list of objects. this convention is explicitly invoked for several common fields [3] and can be used for others where needed. a simple example for a minimal demonstration of the entire process, consider the rdf in figure 2. this expresses, using common vocabulary, a graph similar in scope and structure to the bibjson above. @prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>. @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>. @prefix dc: <http://purl.org/dc/elements/1.1/>. @prefix bibo: <http://purl.org/ontology/bibo/>. @prefix foaf: <http://xmlns.com/foaf/0.1/>. <http://id.example.org/tjohnson-2013> dc:title "indexing linked bibliographic data with json-ld, bibjson & elasticsearch" ; dc:creator <http://id.achelo.us/tjohnson> ; dc:date "2013" ; dc:link <http://example.org/tjohnson-2013> ; rdf:type bibo:article ; dc:ispartof <urn:issn:19405758> ; bibo:issue "19" ; bibo:doi "10.1000/182" . <http://id.achelo.us/tjohnson> foaf:person ; rdfs:label "thomas johnson" ; foaf:lastname "johnson" . <urn:issn:19405758> a bibo:journal ; rdfs:label "code4lib journal" ; rdfs:seealso <http://journal.code4lib.org/> . figure 2. example.ttl — an example bibliographic ‘record’ in rdf. the combined context and framing document in figure 3 will support the conversion of this data into a json document similar to the initial bibjson in figure 1. the constructions “‘type’: ‘@type'” and “‘id’: ‘@id'” create aliases for json-ld keywords which would otherwise appear in our final document, making them more human readable and closer to the default bibjson terms. where bibjson calls for a “list of objects”, the context specifies @set as the container to ensure compliance. the second part of the document (following the @context object) is the frame. it can be read as declaring that the “article” is the root node, and that the other objects should be embedded in its tree. figure 4 shows the bibjson-like results; the python code used to generate the json-ld, and an un-framed example of the same graph are included as an appendix. { "@context": { "type": "@type", "id": "@id", "article": "http://purl.org/ontology/bibo/article", "journal": "http://purl.org/ontology/bibo/journal", "journal": "http://purl.org/dc/elements/1.1/ispartof", "title": "http://purl.org/dc/elements/1.1/title", "issue": "http://purl.org/ontology/bibo/issue", "author": { "@id":"http://purl.org/dc/elements/1.1/creator", "@container": "@set" }, "person": "http://xmlns.com/foaf/0.1/person", "link": { "@id": "http://purl.org/dc/elements/1.1/link", "@container": "@set" }, "doi": { "@id":"http://purl.org/ontology/bibo/doi", "@container": "@set" }, "name": "http://www.w3.org/2000/01/rdf-schema#label", "year": "http://purl.org/dc/elements/1.1/date" }, "@type": "article", "author": { "@type": "person", "@embed": "true" } , "journal": { "@type":"journal", "@embed": "true" }, "link": { "@type":"id", "@embed": "true" } } figure 3. example_frame.jsonld — context and framing document. { "@context": { ... }, "@graph": [ { "title": "indexing linked bibliographic data with json-ld, bibjson & elasticsearch", "author": [ { "foaf:lastname": "johnson", "type": "person", "name": "thomas johnson", "id": "http://id.achelo.us/tjohnson" } ], "type": "article", "journal": { "http://www.w3.org/2000/01/rdf-schema#seealso": { "id": "http://journal.code4lib.org/" }, "type": "journal", "name": "code4lib journal", "id": "urn:issn:19405758" }, "link": [ {"id": "http://example.org/tjohnson-2013"} ], "year": "2013", "issue": "19", "id": "http://id.example.org/tjohnson-2013", "doi": "10.1000/182" } ] } figure 4. example.json — framed json-ld. comparing the output with the initial bibjson example reveals a few interesting differences. firstly, unexpected fields (foaf:lastname, rdfs:seealso, and various iris) are handled gracefully and without data loss. this will continue to work even if the unexpected nodes give rise to complex graph structures. unexpected data can simply be ignored by any software using the index, while remaining available on an ad-hoc basis and subject to elasticsearch’s default analysis. it can also, of course, be converted back into the original rdf along with the rest of the data. secondly, there are some minor differences in the tree structure. the doi in particular was represented in bibjson by an “identifier” object with a type, url, and id. in our generated json-ld, it is simplified to a key-value pair. this is mainly due to the nature of the source data. bibo’s doi property expects us to infer information that bibjson makes explicit. json-ld doesn’t offer a mechanism for forcing simple data types into more complex structures, so there’s not a natural way to resolve this issue. some attention will later be given to workarounds for eventualities like these. for now, it is enough to know that not all graphs can be represented with the same json tree structure; json-ld documents are, first and foremost, representations of rdf graphs. indexing the graph indexing items in elasticsearch is as easy as sending an http put request containing the appropriate json. the example data in figure 4 could be indexed with a request like curl -xput http://localhost:9200/bibjson/articles/1 -d '{...}' where ‘…’ is replaced by the record itself (here, the contents of @graph in figure 4). the url parts ‘bibjson’ and ‘articles’ are, in order, the index name and the type of the document. by default, elasticsearch creates new indexes automatically and creates a default mapping for new types. the final part of the url (‘1’) will be elasticsearch’s internal identifier; indexing with http post enables automatic id generation. the simplest method for indexing multiple records, therefore, is to iterate through the list in @graph, posting each record in turn. once indexed, documents can be retrieved via http get and indexes can be searched via elasticsearch’s query api. elasticsearch is intended to have “sensible defaults” for types with no explicitly defined search mappings (elasticsearch, n.d.). mappings configure the searchability, tokenization and faceting of fields, as well as data types, boosting, and inclusion of fields in ‘_all’ searches. the defaults are often sufficient and, when they aren’t, still manage to usefully handle unexpected data. custom mappings can be applied on a per-index and per-type basis. [4] creating aggregate indexes adding data from a second source can be accomplished by creating a new context and frame, then adding the resulting json-ld in elasticsearch. to the extent that the two datasets share bibjson as a common format, no new configuration is needed. however, when adding data generated from more than one context, experience at oregon state has suggested that using separate indexes is good practice. the context associated with a given index can be added alongside its data as type “context”, and searches configured to ignore these documents. keeping the context alongside the data gives applications a convention for extracting semantics back out of the indexed documents. organizing indexes this way also allows different search mappings to be applied on a by-index basis, helping to address discrepancies between data sources. queries can be easily run across both types and indexes and the segregation doesn’t affect the performance of searches. support for (unlinked) bibjson in addition to supporting a wide variety of linked data models, this index could also accommodate original bibjson data. to enhance its interoperability with json-ld indexes, a generic bibjson context document could be applied. frbr and other limitations data modeling perhaps the strongest limitation faced is the inconsistency between different data models. while our chosen common format prefers simple key-value pairs where possible, rdf data models often use more explicit and complex structure. a core example in the library domain is the entity-relationship model specified by frbr (ifla, 1998). expressing frbr’s multi-tiered approach to bibliographic data in json will lead to a fundamentally different data structure than the bibjson used by the rest of the index. any simplification for compatibility would be lossy, flattening the graph (and can’t be done using json-ld’s algorithms). it’s worth noting that these conceptual incompatibilities originate with the data models themselves and are not an artifact of the index process. datasets using differing models could still be expressed in json-ld and indexed in elasticsearch, but querying them would require a substantial amount of work on the application side to adjust for their differences. models with major incompatibilities in the index could instead be transformed into a compatible, bibjson-friendly, graph prior to generating json-ld. if needed, elasticsearch could still be used as a datastore for the original graph (in json-ld) in a separate index, not included in default searches. in this case, some internal convention would be needed to maintain an association between the indexed record and its original rdf. for smaller model incompatibilities like the doi issue encountered above, the best option may be to add duplicate data after the initial json-ld conversion. in the doi example, we would add the “identifier” object alongside the existing “doi” term. so long as the data produced by the json-ld algorithm is untouched the result will be a more consistent (and still linked) dataset. name collisions in contexts a smaller, but significant, limitation is the potential for name collision in context documents. since json-ld won’t allow multiple iri’s to be mapped to the same json key (this would prevent re-expansion), terms can’t always be represented using bibjson’s default key. the most common problem in our experience is due to the use of “name” as a key for both people (often expressed with foaf:name) and journals (dc:title). mapping both terms to “name” in json-ld would destroy the distinction between the two. the solution here is simple, though it does require an additional step: find a term (e.g. rdfs:label) that applies to both needs and add a triple to the incoming rdf. elasticsearch will index both the shared term (as “name”) and the original (with its full uri as the key). follow your nose one final weakness is that many rdf datasets will only contain a subset of the graph relevant to the creation of a full index. for example, a bibliographic dataset might hold comprehensive data about a book, but make reference to the author only by link to an external source—the ability to make use of related external data being a defining feature of linked data. json-ld generated from such a source would retain the author iri used in the original, but may be missing data points as crucial as the author’s name. the solution, here as elsewhere, is to follow your nose [5], dereferencing the iris to build a more complete graph. summary json-ld can serve as a useful tool for making linked data more accessible to applications. using it in conjunction with bibjson and elasticsearch provides a low barrier method for indexing a wide variety of bibliographic data. though the index has some limitations, it succeeds at joining heterogeneous linked data in a generic form without compromising the full graph structure and semantics of the original data source. notes [1] note that this doesn’t yield “semantic search” in the usual sense of inference, fuzzy logics and concept mapping, but rather a traditional analyzed index. because both semantics and graph structure are retained, clever use of the index might emulate semantic search features like concept-based facets. [2] the behavior of ‘@embed’ is currently in flux. the treatment we rely on here has clear support in current discussion: https://github.com/json-ld/json-ld.org/issues/119. [3] author, editor, license, identifier, link, and journal. [4] a good introduction to mapping is available in a pair of blog posts at http://euphonious-intuition.com/2012/07/an-introduction-to-mapping-in-elasticsearch and http://euphonious-intuition.com/2012/08/more-complicated-mapping-in-elasticsearch. [5] see http://patterns.dataincubator.org/book/follow-your-nose.html. references baker, t., bermès, e., coyle, k., dunsire, g., isaac, a., murray, p., …zeng, m. (2011). library linked data incubator group final report. available http://www.w3.org/2005/incubator/lld/xgr-lld-20111025/ ifla study group on the function requirements for bibliographic records. (1998). functional requirements for bibliographic records final report. available http://archive.ifla.org/vii/s13/frbr/frbr_current_toc.htm johnson, t. and boock, m. (2012). linked data services for theses and dissertations, proceedings of the 15th international symposium on electronic theses and dissertations, lima, peru. available http://hdl.handle.net/1957/32977 lanthaler, m., gütl, c. (2012). on using json-ld to create evolvable restful services, proceedings of the third international workshop on restful design. available from: http://dx.doi.org/10.1145/2307819.2307827. (coins) rdf json. [internet]. talis systems. available from: http://docs.api.talis.com/platform-api/output-types/rdf-json hawke, s. from json to rdf in six easy steps with jron. [internet]. [updated: june 4, 2010]. available from: http://decentralyze.com/2010/06/04/from-json-to-rdf-in-six-easy-steps-with-jron/ macgillivray, m. and pitman, j. how to do bibjson. [internet]. available from: http://www.bibjson.org/ jones, r., macgillivray, m., murray-rust, p., pitman, j., sefton, p., o’steen, b., and waites, w. (2011). open bibliography for science, technology, and medicine, journal of cheminformatics, 3:47. http://dx.doi.org/10.1186/1758-2946-3-47 elasticsearch guide: mapping. [internet]. available from: http://www.elasticsearch.org/guide/reference/mapping/ appendix #!/usr/bin/python import json, rdflib from pyld import jsonld g = rdflib.conjunctivegraph() g.parse(example.ttl', format='n3') # pyld likes nquads, by default expand = jsonld.from_rdf(g.serialize(format="nquads")) framed = jsonld.frame(j, json.load(open('example_frame.jsonld', 'r'))) print json.dumps(framed, indent=1) python framing code. { "@context": { ... }, "@graph": [ { "http://xmlns.com/foaf/0.1/lastname": "johnson", "@id": "http://id.achelo.us/tjohnson", "name": "thomas johnson", "@type": "person" }, { "title": "indexing linked bibliographic data with json-ld, bibjson & elasticsearch", "@id": "http://id.example.org/tjohnson-2013", "journal": { "@id": "urn:issn:19405758" }, "author": [ { "@id": "http://id.achelo.us/tjohnson" } ], "link": { "@id": "http://example.org/tjohnson-2013" }, "year": "2013", "doi": "10.1000/182", "issue": "19", "@type": "article" }, { "http://www.w3.org/2000/01/rdf-schema#seealso": { "@id": "http://journal.code4lib.org/" }, "@id": "urn:issn:19405758", "name": "code4lib journal", "@type": "journal" } ] } un-framed (compacted) json-ld document. about the author thomas johnson is digital applications librarian at oregon state university libraries, where he works on digital curation, scholarly publication, and related metadata and software issues. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – beyond open source: evaluating the community availability of software mission editorial committee process and structure code4lib issue 31, 2016-01-28 beyond open source: evaluating the community availability of software the code4lib community has produced an increasingly impressive collection of open source software over the last decade, but much of this creative work remains out of reach for large portions of the library community. do the relatively privileged institutions represented by a majority of code4lib participants have a professional responsibility to support the adoption of their innovations? drawing from old and new software packaging and distribution approaches (from freeware to docker), we propose extending the open source software values of collaboration and transparency to include the wide and affordable distribution of software. we believe this will not only simplify the process of sharing our applications within the library community, but also make it possible for less well-resourced institutions to actually use our software. we identify areas of need, present our experiences with the users of our own open source projects, discuss our attempts to go beyond open source, propose a preliminary set of technology availability performance indicators for evaluating software availability, and make an argument for the internal value of supporting and encouraging a vibrant library software ecosystem. by bret davidson and jason casden ”in  the open source community there exists a tremendous need for exactly the skills librarians have always used in making information resources truly useful. in particular, systems testing, evaluation, and feedback to open source designers is welcome and even sought after; documentation for open source systems is always needing improvement; instructional materials for open source products are often lacking. these are all areas in which librarians excel.” from “open source library systems: getting started.” (chudnov, 1999) the code4lib community has produced an increasingly impressive collection of open source software over the last decade, but much of this creative work remains out of reach for large portions of the library community. to what extent has our community fulfilled the promise of free and open source software to improve service to our users, particularly as it concerns the vast swath of users served primarily by smaller institutions? do relatively privileged institutions have a professional responsibility to support the adoption of their innovations? “we hope the code4lib journal can manifest the values that have been successful for the code4lib community, while providing increased access to the collective knowledge and experience held throughout our diverse professional networks and local organizations, increasing cross-pollination and collaboration among library technology innovators–and helping more people and organizations become innovators.” from “editorial introduction — issue 1” in the code4lib journal. (rochkind, 2007). despite this stated goal, there are large segments of our broader community of cultural heritage institutions that are unable to implement even the most well documented open source software. these organizations, that represent unique and often marginalized user communities, often have extremely limited access to it resources. we’ve seen firsthand, through our own experiences in supporting open source communities, the impact this has on the ability of organizations to adopt open source library software. in this article, we will present an initial set of performance indicators and technical guidelines to support those developers interested in moving beyond open source to genuinely available software. background early free software “in the early days of computing, the need to generate massive adoption was strong and the compatibility threat was weak—open-source software reigned supreme.” (campbell-kelly & garcia-swartz, 2009). in the 1950s and 1960s, software was often free, source code was available and modifiable, and major development projects were undertaken as a collaboration between vendors, user groups, and academics. one of the earliest examples is often cited as one of the first open source systems and also the first source code compiler. the prevailing idea at the time was that software sold systems, and that the computing industry was primarily a hardware business. software was often collaboratively developed and bundled for free with modifiable source code. the distinction between source code access and technology availability, however, is evident in major software packages such as the ibm airline control program (acp), which was developed by ibm between 1965 and 1979: “what makes acp interesting to open source supporters is that, while technically i suppose ibm ‘owned’ it, the source code was completely available for any developer to change, fix, and enhance. you needed an ibm mainframe on which to run the code, which even the geekiest did not consider (those people were drooling over the dec vax back then), so availability is only a relative term. i don’t remember exactly how the code was contributed back to ibm, which served largely as a code repository (and also sold high-priced technical consultants to help these large enterprises install, support, and deploy the software on ibm hardware — some things don’t change). but i do know that it was done.” (schindler, 2009). it’s important to note that the computing landscape differed in profound ways from our current environment. systems were huge and extremely expensive, and only a small number of extremely wealthy organizations could participate in these communities. software generally was only compatible with one vendor’s platform, and huge technological challenges faced any independent vendor seeking to port an application to another vendor’s environment. this assumes, of course, that these vendors existed, although the independent software vendor (isv) industry would not take off until the 1970s. (philipson, 2004). this approach of bundling “free” source code with hardware was restrained by a long, but ultimately unsuccessful, federal antitrust lawsuit against ibm that was later echoed by the microsoft antitrust case in the 1990s. “such practices allegedly included anticompetitive price discrimination such as giving away software services for ‘the purpose or with the effect of… enabling ibm to maintain or increase its market share… ‘ (id. at 9.) the government also alleged that ibm’s bundling of software with ‘related computer hardware equipment’ for a single price was anticompetitive. (id. at 10.)” (brown, et al., 1995). the combination of more affordable computing hardware, legal challenges, and the emergence of independent software vendors led us into the emergence of a computing industry in which the notions of free and commercial software may be more recognizable to the reader. hacker culture and the opposition to free software the hobbyist computer culture of the 1970s developed alongside the much wider availability of academic computing and the development of minicomputers and diy kits. the increasing prominence of academic and hobbyist computing communities led to the integration of these communities’ ideals, such as intellectually, rather than commercially, driven code sharing, “hacking,” and open collaboration, into the computing field. at the same time, a nascent software industry was developing independently of the major hardware vendors. these vendors, heavily relying on software sales and licensing for revenue, began to resist the open and collaborative approaches to software development that academics and hobbyists value intellectually and hardware vendors and users value commercially. more closed development models allowed these vendors to compete not only based on advertised functionality, but also on the ability of these vendors to stitch together a wider base of platform support from a fragmented hardware industry. among these newer independent vendors to emerge from hobbyist and academic communities is microsoft. “one thing you do do is prevent good software from being written. who can afford to do professional work for nothing? what hobbyist can put 3-man years into programming, finding all bugs, documenting his product and distribute for free? the fact is, no one besides us has invested a lot of money in hobby software. we have written 6800 basic, and are writing 8080 apl and 6800 apl, but there is very little incentive to make this software available to hobbyists. most directly, the thing you do is theft.” (gates, 1976). this quote, taken from bill gates’ legendary “open letter to hobbyists,” aptly captures the moment in the history of software when independent software creators diverged into more formally managed free and commercial communities. emergence of free and open source software in this environment of increasingly powerful independent software vendors, partnerships between hardware and software (such as the critical pc partnership between ibm and microsoft), and complex inter-vendor software compatibility needs, ideological and academic free software movements and licenses began to formalize their organizations and policies. examples of these included the free software foundation and its gnu license, the “bsd” license, and the “mit” license. “by working on and using gnu rather than proprietary programs, we can be hospitable to everyone and obey the law. in addition, gnu serves as an example to inspire and a banner to rally others to join us in sharing. this can give us a feeling of harmony which is impossible if we use software that is not free. for about half the programmers i talk to, this is an important happiness that money cannot replace.” (stallman, 1985). these licenses helped to establish a software community that supported the revival and expansion of collaborative and distributed development for common systems handling functions such as email (sendmail), web service (apache), and server operating systems (linux). however, many of these licenses were designed either to foster scholarship or to pursue ideological ends. in 1998, partially in response to commercial experimentation with free distribution of application source code, the open source initiative, and the descriptor “open source,” were born. “the conferees believed the pragmatic, business-case grounds that had motivated netscape to release their code illustrated a valuable way to engage with potential software users and developers, and convince them to create and improve source code by participating in an engaged community. the conferees also believed that it would be useful to have a single label that identified this approach and distinguished it from the philosophicallyand politically-focused label ‘free software.’ brainstorming for this new label eventually converged on the term ‘open source,’ originally suggested by christine peterson.” (history of the osi…[updated 2015]). this newer free software movement, which has helped to drive the commercial success of free software that is widely apparent today, has not proceeded without criticism (stallman, [updated 2015]) for its business-friendly approach that explicitly distances itself from earlier “politically-focused” free software organizations. the “open source” moniker has since come to dominate discussions of pragmatic and even commercial free software efforts. the reemergence of the single deployment platform one thing that movements ranging from the free software foundation to the open source initiative seem to agree on, however, is the need to work collaboratively to develop a new common application infrastructure to recapture the benefits of the earlier period of vendor behemoths. whether this is seen to encourage more productive competition or higher availability to independent developers is beyond the scope of this paper, but suffice it to say that both commercial and ideological motives were often aligned in this work. “the more recent switch to hybrid strategies reveals (a) an attempt to recreate, via open-source software, the single-platform scenario that ibm achieved with system/360 (and that ibm failed to recreate, in a proprietary context, via saa), and (b) an attempt to boost revenues from middleware and services while allowing the old cash cows (e.g. proprietary operating-system software) to peacefully die an unavoidable death.” (campbell-kelly & garcia-swartz, 2009). ”unix is not my ideal system, but it is not too bad. the essential features of unix seem to be good ones, and i think i can fill in what unix lacks without spoiling them. and a system compatible with unix would be convenient for many other people to adopt.” (stallman, 1985). the most critical factor within the context of this article is that a large and influential group of academics, hobbyists, and commercial organizations have moved beyond the dichotomy of developer freedom and revenue towards seeking a common infrastructure to reach more users with less work. unix, and particularly linux, began to be seen as a strong candidate for providing a layer of abstraction over extremely fragmented hardware and systems layers. our own experience unfortunately, we’ve found that library open source software has tended to favor the support for narrow (well-resourced) institutional and developer communities over this goal of widespread support for diverse user communities. we’ve seen this most clearly with our own open source releases. while we’ve been reasonably successful when it comes to promoting projects at conferences and journals, as well as supporting software adoption at similar (research) libraries, we’ve failed at enabling the adoption of our approaches at other cultural heritage institutions. suma suma (suma … [updated 2015]) is an open-source mobile web-based assessment toolkit developed by ncsu libraries for collecting and analyzing observational data about the usage of physical spaces and services. this software suite, used by dozens of libraries, is designed to provide easy and rapid analytics capabilities for spaces and service similar to that which is available for web services. the following quote is from an email from a librarian at a research university seeking to install suma (used with permission). “who would be the best person on your team to talk with about the technical requirements and skills needed for us to install suma and get it up and running. for example, do we need a computer programmer with such and such skills. i’m sure you have good documentation available, however, we would like to talk with someone. we are also curious as to how much time it would take to get suma working for us. i am not sure we have the expertise in-house.” and from a librarian at a research university (also used with permission) on canceling a pilot suma evaluation: “ultimately, i didn’t want to get on our it support’s bad side and because i’m not allowed to play with our development servers i can’t exactly go at the installation process alone.” these librarians (and others who already have the fortune to work at relatively large research institutions) were struggling to find the it resources necessary to deploy what we humbly feel is a fairly well-tested and documented web-based application. of course, much of the community who could benefit from space and service usage data analytics has access to far fewer resources. no amount of testing and documentation will solve this problem. social media archives in order to develop accurate historical assessments, researchers must have access to primary materials that represent a large and diverse set of participants. social media platforms have become a venue where serious discourse and creation take place, but much of this critical and ephemeral content is lost to researchers as few institutions collect and preserve this content. in 2014, the ncsu libraries was awarded an ez innovation grant (2014-2015 lsta grant programs … [updated 2014]) from the state library of north carolina (state library of north carolina … [updated 2015]) to develop a freely available web-based social media archives toolkit (social media archives toolkit … [updated 2015]) that publicly documents our own effort to develop a sophisticated social media archival program in a way that may help guide cultural heritage organizations that are interested in collecting and curating social media content. as a part of this grant, ncsu libraries conducted a survey of north carolina cultural heritage organizations about the promise of social media data inclusion in archival collections. among the responses was this discussion of the tension between interest and ability: “along with email, social media will probably provide the main source of information for researchers studying our current time. however, our institution just does not have the resources right now to collect and store the social media of other people or organizations.” (social media archives toolkit … [updated 2015]) this confirms our own anecdotal evidence from presentation questions and conference presentations: most institutions simply don’t have access to the resources necessary to pilot new initiatives with technical prerequisites. this is especially true for those cultural heritage organizations who could contribute the most to conversations addressing archival practices for social media community documentation, such as public libraries, historical societies, and community colleges. the future of libraries “i believe that building and maintaining library software is vitally important work and it’s too big a job to leave to a small group of people. we are creating the future of libraries here.” from “creating a commons.” (sadler, 2013) library open source software has improved technical collaboration and sharing among large institutions, but it retains problems of obligatory collaboration by requiring a relatively costly common environment and skill set. the goals of library open source, if viewed through the lens of journal publications, conference presentations, project users, and project contributors, must be seen as more closely aligned to collaborative software development rather than representative project development. unfortunately, since most developers are employed by institutions with the resources to foster specialized human resources, when developers vote with their code many categories of users may not be reflected in the development of the project. community members who would help address many other types of users (serving many other types of institutional users) never have the opportunity to participate. library technology should be able to thrive in diverse organizational and technological environments. within the library community, technical settings vary wildly. for example, the 2011-2012 ala public library funding & technology access study showed that 88 percent of public libraries depend on “non-it specialists” for their it support services, with 59.5% of rural public libraries depending on their library director for it support (american library association, 2012). we will never reach the bulk of our colleagues and users without dramatically reducing the cost of supporting individual software projects. in order to build more representative software communities, we need to ask more of the software that we produce. we need software that is conscientious. “our software is like children. … we expect that after a time the child will mature, will grow up, will be able to take care of itself, to solve problems, to cope, and perhaps to contribute something new. initially selfish—for what other options are there?—the child becomes responsible. with luck or persistence or as the result of good upbringing, the child may become conscientious. shall we hope similarly for our software?” (gabriel and goldman, 2006) in the context of this challenging view of software as an entity that exists on some sort of maturation curve, open source software is not grown-up software. this software has left the house and begun to develop its own life outside of our influence, but it has not yet learned how to seek and maintain an environment in which it can thrive. it has a narrow, provincial view of the world which still must be expanded. this software is not yet fully realized in its role of providing utility to a large and diverse community. it is open but not available. software availability our goal now is to find methods for measuring the maturity, or availability, of our software as well as techniques for improving those measures. gabriel and goldman provide some suggestions, some of which fall into the realm of technical speculation, and some of which are difficult to evaluate. we need a way to quickly and consistently identify gaps in the availability of our software and to measure improvement after changes are made. most importantly, we need an evaluation tool that exclusively measures the ability to implement and use software. in other words, software does not need to be open source, easy to modify, or even free to be considered highly available to targeted user communities. these performance indicators must specifically target the goal of wide software adoption across all potential user groups. stopwatch technology availability performance indicators we have developed an initial set of task performance indicators to measure software availability for a given user group, derived from the common practice in user experience design of developing key performance indicators for tracking and evaluating progress toward strategic organizational goals (5 ux kpis [accessed 2016]). by focusing on task performance rather than specific techniques such as documentation, we are attempting to provide indicators that are easy to measure, compare, and communicate for a wide variety of projects. these indicators are: time to pilot on a laptop. defined as the time needed to install and configure, at minimum, a demonstration instance of the application, particularly for use in organizational evaluation of software. time to export data. defined as the time needed to export unique data captured in the application, e.g. social media metadata and space assessment data. time to update dependencies. defined as the time needed to update the dependencies on which an application is built, e.g. ruby on rails gems. time to upgrade application. defined as the time needed to update the application itself, e.g. version updates, bug fixes, and database migrations. time to migrate application. defined as the time needed to transfer application to a new server environment, i.e. application portability. time to new production deployment. defined as the time needed to create a robust production instance of the application. time to reasonable security. defined as the time needed to secure the application relative to the sensitivity of the data being stored. this should consider the security policies of the adopting institution. guidelines it’s important to follow a small set of guidelines when using these performance indicators in order to ensure that these measures are as consistent and reliable as possible. for each set of performance indicators, a targeted user group and rate of success should be defined before measurement. this will allow us to easily compare results across projects and to chart these results to measure the continual maturation of a particular project. use the same user group within each set of performance indicators use the same success rate within each set of performance indicators assess all performance indicators in the set examples of measure definitions: time to pilot for subject liaison librarians with 90% success, i.e. 9 out of 10 subject liaison librarians were able to pilot the application within the time listed. time to pilot for assessment librarians with 50% success time to pilot for systems administrators with 80% success we can now not only identify gaps in the availability of our software to specific user communities, but begin to search for tools and techniques that can be applied to mitigate these issues. applying stopwatch technology availability performance indicators several approaches exist for improving the ease with which software projects can be adopted by users including usable documentation, packaged installers like those offered by wordpress (installing wordpress … [accessed 2015]), vendor specific deployment scripts from companies like google (google cloud computing [accessed 2015]), microsoft (microsoft azure [accessed 2015]), amazon (aws [accessed 2015]), and heroku (creating … [accessed 2015]), and hosted and managed services like duracloud (duracloud [accessed 2015]) and dspace direct (dspace direct [accessed 2015]). another approach, and the one we have been exploring with our own software projects, is to take advantage of virtualization tools like vagrant and docker. vagrant (vagrant [accessed 2015]) is a tool for building development environments by provisioning systems inside a variety of virtualization platforms like virtualbox, vmware, and others, providing developers with isolated, disposable, and consistent environments. vagrant is easily installed on various operating systems including linux, windows, and os x using a packaged installer. applications are then provisioned using vagrantfiles provided by the application maintainers. as of version 1.8, vagrant will even install virtualbox automatically to create a simpler initial provisioning process. with vagrant, developers can be more confident that their code is running in the same environment as their team members, improving the reliability of code and easing the transition from development environments to staging and production environments. vagrant also simplifies operations by providing a disposable environment and consistent workflow for developing and testing infrastructure management scripts. docker (docker [accessed 2015]) is an open platform for building, shipping, and running distributed applications. it provides a common toolbox to take advantage of the distributed and networked nature of modern applications and enables the packaging of an application with all of its dependencies into a standardized unit for software development called a container. containers have similar resource isolation and allocation benefits as virtual machines but a different architecture. containers share the kernel with other containers, resulting in improved performance over virtual machines. unlike virtual machines, which include the application plus the necessary binaries, libraries and the entire guest operating system, containers run as an isolated process in user space on the host operating system. they’re also not tied to any specific infrastructure; docker containers can run on any computer, on any infrastructure, on any cloud. vagrant and docker may be used together and often are. vagrant supports docker both as a provisioner and as a provider, meaning that docker containers may be mounted to a guest virtual environment or directly to the host environment(lowe 2015.) what this means is that end users of applications don’t need to be familiar with docker at all in order to take advantage of it’s features since vagrant can be configured by the application maintainer to automatically manage the provisioning of docker containers. at the same time, these docker containers can also be used for production deployments to hosting providers and in-house servers. vagrant and docker have become popular tools for simplifying the process of creating and maintaining software environments. john fink wrote in 2014 that “for software development, when programmers check their code into git, a dockerfile could be included in the source code, allowing for quick testing of code on remote servers or as a demonstration tool to let other quickly bring up their own version of an application without having to worry about specific building instructions or dependency management” (fink 2014). furthermore, these tools can also help reduce the costs associated with software adoption and create mutual benefit for both project maintainers and adopters. as we have continued to develop and support open source software projects and communities, we have found that many cultural heritage organizations that might benefit from adopting our software were simply unable to do so based on budgetary constraints, staffing, or other institutional hurdles. we believe that integrating tools like vagrant and docker into our projects will not only improve internal development practices, but help lower the barrier to adopting our software and increase our score against our proposed stopwatch technology availability performance indicators. what follows are two case studies of our own open source software projects that demonstrate how these performance indicators might be used to evaluate the availability of library software projects. note:we are using estimates based on prior system support and user discussions and not on formal user testing. these performance indicators are most helpful as tools for identifying gaps in the software adoption process and are not meant as performance profiling tools. case study: suma-vagrant suma-vagrant (suma-vagrant [accessed 2015]) is an experimental demonstration and development environment that utilizes vagrant, virtualbox and ansible (ansible [accessed 2015]) to create a reproducible, isolated virtual environment for suma. developers working with suma-vagrant can be more certain that code they develop will be compatible with the large collection of dependencies and libraries used in the suma application. the suma-vagrant environment has allowed us to more easily test against consistent test data, explore integration with third party tools like tableau desktop, and on-board new developers to the project. suma-vagrant has also simplified the installation and configuration of suma for less-technical users. some of the installation issues (figure 1) users have encountered when trying to install suma at their home institutions include configuration file errors, environment issues, and database access. these issues are not only costly for us to support, but also impede the ability of potential users to evaluate suma and to argue for its adoption at their institutions. using suma-vagrant, users can quickly create a working, fully functional suma environment in only four steps (figure 2). evaluating suma (figure 3) and suma-vagrant (figure 4) against our stopwatch technology availability performance indicators shows that suma-vagrant greatly reduces the time and expertise required to install and configure a working demonstration and development environment for suma, making the benefits and use cases of suma easier to explore. mod_rewrite disabled curl missing configuration errors symlink problems server hardening software db access installation method confusion figure 1. selected list of suma install issues install vagrant clone or download the suma-vagrant repository on github. execute “vagrant up” from within the project directory using a terminal visit http://localhost:19679 in a web browser on the host machine. figure 2. suma-vagrant installation steps time to pilot on a laptop: days time to export data: 4 hours time to update dependencies: 4 hours time to upgrade application: 1 hour time to migrate application: unknown time to new production deployment: days, if at all time to reasonable security: unknown figure 3. suma evaluation: assessment librarians at 80% success time to pilot on a laptop: 40 minutes time to export data: 10 minutes time to update dependencies: 2 minutes time to upgrade application: 2 minutes time to migrate application: 20 minutes time to new production deployment: under development time to reasonable security: 10 minutes (pilot), under development (production) figure 4. suma-vagrant evaluation: assessment librarians at 80% success case study: social media combine as part of the ez innovation grant from the state library of north carolina, the ncsu libraries developed the social media combine (social media combine … [updated 2015]), a pre-configured collection of tools including social feed manager (social feed manager … [updated 2015]) and lentil (lentil .. [updated 2015]) for easily building twitter and instagram social media archives on your own computer. it pre-assembles lentil for instagram data harvesting and social feed manager for twitter data harvesting, along with the web servers and databases needed for their use into a single package that can be deployed to desktop and laptop computers using windows, os x, or linux. applying our “stopwatch technology performance indicators” to lentil before (figure 5) and after (figure 6) integration shows tasks that might take days for a standard installation take only minutes as part of the social media combine. potential users are not required to be familiar with application frameworks, server environments, or even the different programming languages used for the project (figure 7). once the social media combine environment is installed, users are presented with an intuitive web configuration tool for managing the configuration of the system (figure 8). the integration of lentil and social feed manager into a single application that can be run on everyday hardware with usable web-based configuration makes the adoption of these archival tools much more viable for under resourced institutions. time to pilot on a laptop: days time to export data: automatic time to update dependencies: 4 hours time to upgrade application: 1 hour time to migrate application: unknown time to new production deployment: days, if at all time to reasonable security: unknown figure 5. lentil evaluation: archivists at 80% success time to pilot on a laptop: 30 minutes time to export data: automatic (lentil), variable (sfm) time to update dependencies: 2 mins time to upgrade application: 5 mins time to migrate application: 20 minutes (under documented) time to new production deployment: under development time to reasonable security: 10 minutes (pilot), under development (production) figure 6. social media combine (lentil + social feed manager + configuration) evaluation: archivists at 80% success install git and vagrant clone or download the social media combine repository on github. execute “vagrant up” from within the project directory using a terminal enter configuration parameters in the web configuration form and click ok visit lentil: http://localhost:3001, social feed manager: http://localhost:8001, or the configuration tool: http://localhost:8081 figure 7. social media combine installation steps figure 8. screenshot of web configuration tool (attached) conclusion by applying our stopwatch technology availability performance indicators to our own software projects, we have shown how approaches that help local practices and deployments, like consistent, repeatable virtual environments, can also be used to improve the availability of library technology. holding ourselves accountable in this way is like testing; it’s good for you, it’s something you should be doing anyway, and will have a positive and significant impact on the total quality and usefulness of library software. these performance indicators might help us think more holistically about our software projects, focusing less on simply releasing code and more on what is required for users to actually adopt and use our software, especially when those users are under resourced. open source itself is not sufficient for distributing innovative library software. but, by taking the best parts of emerging devops tools and integrating them with the natural library mission of supporting the diffusion of innovation, perhaps we can ensure that institutions who might benefit from our software can install and apply it to their own needs. we believe privileged institutions have a professional responsibility to support the adoption of their innovations. our community has fulfilled the promise of free and open source software inasmuch as we have made our code freely available, created vibrant cross-institutional developer communities, and improved the level of service to our direct constituencies through truly innovative software. but, we have failed to fulfill the promise of promoting the diffusion of innovation in library technology to those smaller, less resourced institutions and their users. genuine software availability should not be defined only as open source releases of freely available software in a code repository, but as a fully considered commitment to ensuring that software can be adopted by those institutions that stand to benefit the most from our collaborative efforts and, in turn, that our projects benefit from the participation of more representative communities across the spectrum of cultural heritage institutions. references 5 ux kpis you need to track. [accessed 2016 jan 4]. http://designmodo.com/ux-kpi/ 2014-2015 lsta grant programs. [accessed 2015 dec 6]. http://statelibrary.ncdcr.gov/ld/grants/lsta/2014-2015grants.html ansible. [accessed 2015 dec 6]. http://www.ansible.com/ aws marketplace applications now available with 1-click deployment in sydney. [accessed 2015 dec 6]. //http://aws.amazon.com/about-aws/whats-new/2013/04/24/aws-marketplace-applications-now-available-with-1-click-deployment-in-sydney/ brown k, adler sm, irvine rl, resnikoff da, simmons i, tierney jj. united states’ memorandum on the 1969 case. washington, dc, usa; 1995. campbell-kelly m, garcia-swartz dd. pragmatism, not ideology: historical perspectives on ibm’s adoption of open-source software. 2009;21(3):229–244. chudnov d. open source library systems: getting started | oss4lib. 1999 feb 1 [accessed 2015 jan 20]. http://www.oss4lib.org/readings/oss4lib-getting-started.php creating a “deploy to heroku” button. [accessed 2015 dec 6]. https://devcenter.heroku.com/articles/heroku-button docker. [accessed 2015 dec 6]. http://www.docker.com/ dspacedirect. [accessed 2015 dec 6]. http://dspacedirect.org/ duracloud. [accessed 2015 dec 6]. http://www.duracloud.org/ fink j. docker: a software as a service, operating system-level virtualization framework. 2014 [accessed 2015 dec 6];(25). http://journal.code4lib.org/articles/9669 gabriel rp, goldman r. conscientious software. in: proceedings of the 21st annual acm sigplan conference on object-oriented programming systems, languages, and applications. new york, ny, usa: acm; 2006 [accessed 2015 jan 4]. p. 433–450. (oopsla ’06). gates b. an open letter to hobbyists. 1976 [accessed 2015 dec 6];2(1). https://commons.wikimedia.org/wiki/file:bill_gates_letter_to_hobbyists.jpg google cloud computing. [accessed 2015 dec 6]. https://cloud.google.com/ history of the osi | open source initiative. [accessed 2015 dec 6]. http://opensource.org/history hoffman j, bertot jc, davis dm. libraries connect communities: public library funding & technology access study 2011-2012. digital supplement of american libraries magazine. american library association; 2012 [accessed 2015 dec 6]. http://www.ala.org/research/plftas/2011_2012#final report installing wordpress. [accessed 2015 dec 6]. https://codex.wordpress.org/installing_wordpress#famous_5-minute_install lentil. [accessed 2015 dec 6]. https://github.com/ncsu-libraries/lentil lowe s. using docker with vagrant. [accessed 2015 dec 6]. http://blog.scottlowe.org/2015/02/10/using-docker-with-vagrant/ microsoft azure. [accessed 2015 dec 6]. https://azure.microsoft.com/en-us/ philipson g. a short history of software. in: barrett r, editor. management, labour process and software development: reality bites. 2004. p. 13. rochkind j, editor c, editorial introduction — issue 1. 2007 [accessed 2015 dec 6];(1). http://journal.code4lib.org/articles/39 sadler b. creating a commons. 2013 [accessed 2015 dec 6]. http://www.ibiblio.org/bess/?p=302 schindler e. an abbreviated history of acp, one of the oldest open source applications | itworld. 2009 aug 20 [accessed 2015 jan 12]. http://www.itworld.com/article/2767585/open-source-tools/an-abbreviated-history-of-acp–one-of-the-oldest-open-source-applications.html social feed manager. [accessed 2015 dec 6]. https://github.com/gwu-libraries/social-feed-manager social media archives toolkit. [accessed 2015 dec 6]. http://www.lib.ncsu.edu/social-media-archives-toolkit social media combine. [accessed 2015 dec 6]. https://github.com/ncsu-libraries/social-media-combine stallman r. the gnu manifesto. 1985 [accessed 2015 dec 6]. http://www.gnu.org/gnu/manifesto.en.html stallman r. why open source misses the point of free software. 2015 [accessed 2015 dec 6]. http://www.gnu.org/philosophy/open-source-misses-the-point.en.html state library of north carolina. [accessed 2015 dec 6]. http://statelibrary.ncdcr.gov/index.html suma. [accessed 2015 dec 6]. https://www.lib.ncsu.edu/reports/suma suma-vagrant. [accessed 2015 dec 6]. https://github.com/ncsu-libraries/suma-vagrant vagrant. [accessed 2015 dec 6]. https://www.vagrantup.com/about.html appendix a: social media combine sample code social media combine vagrant file: https://github.com/ncsu-libraries/social-media-combine/blob/master/vagrantfile social media combine docker file: https://github.com/ncsu-libraries/social-media-combine/blob/master/docker-compose.yml appendix b: suma-vagrant sample code suma-vagrant vagrantfile: https://github.com/ncsu-libraries/suma-vagrant/blob/master/vagrantfile suma-vagrant ansible:https://github.com/ncsu-libraries/suma-vagrant/blob/master/ansible_tasks/roles/demo/tasks/suma.yml about the authors bret davidson is a digital technologies development librarian at north carolina state university where he works to advance library services through applied research and application development. he provides technical leadership for the open-source space and service assessment toolkit “suma” and contributes to a broad portfolio of library applications. previously, he was an ncsu libraries fellow, a public school music educator, and a performing musician with the river city brass band in pittsburgh, pa. jason casden is the interim associate head for the digital library initiatives department at the north carolina state university libraries. jason has served as a project or technical lead for several projects designed to help the ncsu and wider library communities interact with library resources, services, and spaces in new ways. these include the “my #huntlibrary” community-driven photographic student documentation project and the supporting open source “lentil” instagram harvesting platform, the “suma” space and services assessment system, and the “social media combine” social media harvesting environment that pre-assembles social media data harvesting software into a single installable package. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – better together: improving the lives of metadata creators with natural language processing mission editorial committee process and structure code4lib issue 51, 2021-06-14 better together: improving the lives of metadata creators with natural language processing dc public library has long held digital copies of the full run of local alternative weekly, washington city paper, but had no official status as a rights grantor to enable use. that recently changed due to a full agreement being reached with the publisher. one condition of that agreement, however, was that issues become available with usable descriptive metadata and subject access in time to celebrate the upcoming 40th anniversary of the publication, which at that time was in six months. one of the most time intensive tasks our metadata specialists work on is assigning description to digital objects. this paper details how we applied python’s natural language toolkit and openrefine’s reconciliation functions to the collection’s ocr text to simplify subject selection for staff with no background in programming. introduction in 2001, yale university scanned the full run of washington, d.c. alternative weekly newspaper washington city paper (wcp) to microfilm, and provided digital surrogates to dc public library (dcpl) as a professional courtesy. at that time, dcpl had neither dedicated staff nor infrastructure to manage digital materials in a meaningful way, thus the collection remained largely dormant until 2014, when the special collections division (now named people’s archive) both hired its first digital curation librarian and launched its digital collections platform, dig dc. even then, aside from occasional reference requests, use of the digital wcp was limited due to there being no formal agreement between dcpl and the original publisher. two key shifts occurred in swift succession that led to movement on the wcp project. firstly, dcpl reached an agreement with publishers of the historic gay newspaper washington blade to digitize from microfilm, describe, and make available all issues of the paper from 1969 to the present day. the project is still ongoing, but is considered a success by both parties. thus, when the opportunity to have formal discussions with representatives of wcp presented itself, dcpl had in place not only a model for how the work might be completed, but also a legal framework to allow the publisher to retain copyright, and also confer generous rights grantor privileges to the library. dcpl reached a similar agreement with the publishers of wcp in 2020, and planned for the collection’s debut to coincide with the 40th anniversary of the paper in 2021. secondly, the covid-19 pandemic, while catastrophic, forced dcpl to reevaluate its telework policies and revise them for the better, at least temporarily. library staff required projects that could be completed remotely, and descriptive metadata work was deemed appropriate considering the situation. it should be noted that staff experienced significant anxiety about both their health and employment status at that time due to the impending recession and worsening coronavirus spread, and that this project, as conceived, was designed partly to remove one element of friction from the workdays of people who were under unusual and understandable levels of stress. that element of friction was, in short, the selection of subject headings and name authorities. dcpl’s previous work in the authoring of descriptive metadata for digital objects was not necessarily seen to be lacking, merely time-consuming. to illustrate, consider the workflow for the washington blade project: the project manager provides a batch of pdf issues and mods template to another staff member; many fields are pre-filled, but that staff member creates extensive entries for the item description field, and browses the library of congress (lc) linked data service to select a set of relevant subject headings for each issue in the batch. each issue must be read in detail, and a great deal of time must be spent exploring the lc site. with the 40th anniversary of wcp approaching quickly, the question arose: what could be done to speed up this process? could “subject” become one of those pre-populated fields? workflow dcpl’s people’s archive digital unit had long been interested in harnessing, in some way, the ocr text that is automatically generated when an appropriate item is ingested into dig dc (an islandora 7 repository). the period of confusion between the beginning of the pandemic and the now-familiar rhythm of telework created an opportunity to step back from normal responsibilities and, with distance, conceive an approach that could generate library of congress subject heading (lcsh) and library of congress name authority file (lcnaf) matches from the ocr text of a newspaper issue. the workflow that was developed follows. since dcpl does not usually perform ocr until an object, or set of objects, is ingested into islandora (typically the final stage of a digital collections project), and our goal was to assist with descriptive metadata creation (one of the initial stages), we opted to run tesseract on the collection in bulk before any other work was done. we decided to focus only on the cover pages of each issue for two reasons: one, it is much simpler and computationally less demanding to run tesseract on a single folder containing one tiff per issue than it is to do the same for a series of nested folders containing complete editions (which are around 80 pages each); and two, the covers are in general more succinct and focus largely on the unique content for that issue, which we hoped would result in more accurate subject headings. since the digital issues utilized a nested folders-within-a-folder structure, we wrote a python script to extract files with names that matched a pattern shared by all issue covers and place them into a single directory. this resulted in a folder containing 950 images, and consequently 950 files of ocr text. figure 1. the cover for the first issue of the publication, which was simply titled “1981” before adopting its current title of “washington city paper”. the next step was to extract commonly-used words in those files, and we identified python’s natural language toolkit (nltk) as the most appropriate tool for the job. we created another python script to iterate through the text files, produce a cleaned version of each, and apply nltk to the cleaned versions to output a list of ten most frequency encountered terms to an additional set of text files. the resulting top ten rankings were then combined into a single csv featuring two columns (“filename” and “content”) with a final python script. while the process technically worked from the beginning, results were not initially usable because nltk’s default list of 127 english stopwords used in the text cleaning process is simply not extensive enough. ultimately, we employed an expanded list of 1,160 stopwords compiled from multiple sources. lastly, the csv was brought into openrefine and cleaned up so that each term to be matched against lc occupied a single cell. we then used christina harlow’s lc openrefine reconciliation endpoint to reconcile the text in each column, and imported high percentage lcsh and lcnaf matches into the mods templates used by our metadata workers. results and analysis we were not able to fully pre-populate the subject field. that said, this process identified 1,972 lcsh and lcnaf matches for 950 issues, and 525 of those matches (roughly one quarter) were selected as relevant to the content of the issues being described. we found that terms with a certain level of specificity resulted in more useful matches (consider the following examples that were included in the final collection metadata: “vodka“, “nicaragua“, “rent“, “preservation“, “gambling“, “radio“, “squirrels“, “poetry“, “christmas“, “assassination“, “interferon“, and – the author’s personal favorite – “leprechauns“; versus “american” and “home“, which were not). one element of this workflow that was not particularly successful and that could be iterated upon in future implementations is the focus on identifying single words instead of terms for reconciliation against lc. to illustrate, this solution in no way accounts for terms like “american ambassador” (which in one instance was split into “american” and “ambassador,” both reconciled against lc separately), “infant formula” (which was treated as “infant” and “formula.”), or “enola gay” (which was treated as “enola” and “gay”). while terms were technically selected, the original meanings were altered due to this limited approach. unfortunately, we have little to no data to determine how this work impacted the quality of life of our metadata creators. what is obvious, however, is that their work cannot be easily replicated, merely supplemented; nltk, for all of its strengths, cannot match the creativity, cognition, and intuition of library workers and their abilities to read contextually and make judgement calls. the process of computationally identifying subjects and name authorities certainly adds value to the collection, and may save workers small amounts of time, but whether that time saved is statistically significant is debatable. the author believes, however, that there is further work to be done here and that there is great potential to improve upon these results in future projects. the washington city paper digital collection is a work in progress, and is available on dig dc. about the author paul kelly (paul.kelly2@dc.gov) is digital initiatives coordinator for the people’s archive at dc public library. he holds his m.s. in library and information science from the catholic university of america and m.a. in english literature/film and television studies from the university of glasgow. his professional interests include collections as data, web archiving, and digital preservation. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – enhancing serials holdings data: a pymarc-powered clean-up project mission editorial committee process and structure code4lib issue 58, 2023-12-04 enhancing serials holdings data: a pymarc-powered clean-up project following the recent transition from inmagic to ex libris alma, the technical services department at the university of southern california (usc) in los angeles undertook a post-migration cleanup initiative. this article introduces methodologies aimed at improving irregular summary holdings data within serials records using pymarc, regular expressions, and the alma api in marcedit. the challenge identified was the confinement of serials’ holdings information exclusively to the 866 marc tag for textual holdings. to address this challenge, pymarc and regular expressions were leveraged to parse and identify various patterns within the holdings data, offering a nuanced understanding of the intricacies embedded in the 866 field. subsequently, the script generated a new 853 field for captions and patterns, along with multiple instances of the 863 field for coded enumeration and chronology data, derived from the existing data in the 866 field. the final step involved utilizing the alma api via marcedit, streamlining the restructuring of holdings data and updating nearly 5,000 records for serials. this article illustrates the application of pymarc for both data analysis and creation, emphasizing its utility in generating data in the marc format. furthermore, it posits the potential application of pymarc to enhance data within library and archive contexts. by minyoung chung and phani chaitanya pendyala introduction in may 2023, one archives [1] successfully completed the data migration from inmagic to ex libris’ alma. it’s noteworthy that the metadata schema utilized in inmagic differed from marc 21, and the holding information was encoded as “checkin.” exemplified in figure 1 are instances of the checkin values, integral components of the migrated metadata from inmagic. figure 1. examples of checkin values throughout the data migration, the “checkin” data was systematically mapped into the 949 $c field within bibliographic records. additionally, this mapped information from the 949 $c field was further transposed into the 866 $a field within the holding records. this configuration not only allowed for the batch import of bibliographic records into the alma system but also facilitated the generation of holdings information in the alma import profile. context the usc libraries’ one archives stands as the preeminent repository of lesbian, gay, bisexual, transgender, queer (lgbtq) materials globally. boasting an extensive collection, one archives houses millions of archival items, encompassing periodicals, books, films, videos, audio recordings, photographs, artworks, organizational records, and personal papers. since 2010, one archives has been integrated into usc libraries. with usc libraries’ adoption of the exlibris alma integrated library system (ils) in 2017, the migration process also included the transfer of records from one archives. however, due to the complex nature of holdings information, particularly in specific serials and audio records, special attention was warranted. consequently, certain records remained pending ingestion into alma. the conclusive migration of data for these serials and audio materials was successfully executed between march and may 2023. this process necessitated extensive communication and collaboration among one archives, the technical services & collection development department, and the integrated library systems department within usc libraries. this paper discusses our follow-up cleanup initiative conducted between may and july 2023, right after the final data migration. our primary focus was directed towards a thorough examination of one archives serials holding records within alma. a noteworthy issue that surfaced pertained to the presentation of summary holdings in primo, prompting concerns about the clarity of the holding information. the challenge centered around the difficulty in understanding the holding details within primo, which proved to be time-consuming for myself and my fellow serials cataloger. from a technical perspective, the cleanup of holdings information to enhance clarity and comprehensibility was entirely feasible. consequently, embarking on this cleanup project was an evident and straightforward decision. this challenge is further compounded by the inherent non-browsability of these materials, given that the serials are securely housed in the restricted and closed stacks in one archives. therefore, ensuring the accurate display of holding information through the primo interface became imperative. the primary objective of the project was to enhance the user-friendliness of the primo display by presenting detailed information in a clear and understandable manner. usc libraries employ the ansi/niso z39.71 standard in the cataloging of serials and their holdings records. notably, primo does not utilize the 853 (captions and pattern) and multiple 863 (enumeration and chronology) fields. instead, primo employs the equivalent free-text fields, specifically the 866 field. consequently, ensuring the accuracy and well-organization of data in the 866 field is crucial for serials’ holdings records. as depicted in the figure 2 below, for example, the input of numbers for each month and season in the 863 field initiates the automatic generation of paired coded text data in the 866 field. figure 2. example of marc codes for months/seasons and the ansi/niso abbreviations in holdings record approach our methodology involved augmenting holding records with 853 (captions and pattern) and multiple 863 (enumeration and chronology) fields. in alma, the automatic creation of the summary field (866) was based on the information in the 853 and 863 fields. the addition of 853 and 863 fields was intended to update the 866 fields, ultimately reflecting the revised holdings information in primo. however, we encountered a challenge due to the variability in the provided holding information for each serial, along with distinct patterns. also, all holding information was consolidated in the 866 field (summary field) as presented in primo (figure 3). the incorporation of 853 and 863 fields across over 5,000 records proved to be a time-intensive process. consequently, we chose to explore the application of regular expressions to identify the issuing pattern for 853 and the enumeration/chronology for 863. figure 3. holdings information in primo before the update the data this study utilized a dataset comprising 5,254 serials holding records subjected to ingestion. to improve accessibility during the migration process, a free-text note labeled “oneinmagicserials” was appended to the local note 910. this deliberate step enhanced retrieval from alma, allowing for precise searches within both alma and primo ve. importantly, this approach facilitated the generation of a comprehensive excel-based list (see figure 4), encompassing key bibliographic information, including titles (245$a), mms id (001$a) [2], oclc number (035$a), and library of congress subject headings (650$a). figure 4. examples of exported bibliographic data data retrieval and update via z39.50 in marcedit although alma offers the capability to export bibliographic, inventory, and authority records, it is important to note that alma does not provide specific functionality for exporting holdings records in marc21 xml or binary format. consequently, we opted to utilize the marcedit-alma integration to obtain the holdings records. leveraging the z39.50/sru client-server, we conducted batch searches for mms ids and subsequently downloaded the corresponding holdings records in bulk. this task takes a considerable amount of time in marcedit, so we have divided the files into six separate parts to manage it more efficiently. figure 5. batch marc record retrieval using z39.50/sru in marcedit marcedit was utilized for the retrieval and updating of holding records in alma. upon extracting all holding records, the files were saved in both binary marc file (mrc) and mnemonic marc text file (mrk) formats [3]. the .mrc file is employed during program execution, while the .mrk file serves as a tool for reviewing current records and extracting insights for constructing regular expressions. moreover, the .mrk file serves as a backup to enable the restoration of records in case of errors or inadvertent mistakes. serials cataloging demands advanced skills and knowledge. while reviewing the holdings records, we collaborated with our senior serials cataloger to deliberate on the optimal approach for generating 853 and 863 fields. concurrently, a meticulously coded holdings record is conducive to machine readability, affording us the opportunity to explore a semi-automated enhancement process. =ldr 00366nx a22000851n 4500 =008 1011252u\\\\8\\\4001uueng0000000 =005 20230424100908.0 =852 8\$bonearchive$cone-mbl$xmagazine =866 \\$a2014: (31 [jun])2015: (1 [dec])2016: (56 [oct])2017: (66-67 [sep-oct])2018: (73-74 [apr-may],79-80 [oct-nov]) =999 \\$aalma_bib_number $b991043829289303731 $calma_holdings_number $d221271756360003731 =ldr 00291nx a22000851n 4500 =008 1011252u\\\\8\\\4001uueng0000000 =005 20230424100909.0 =852 \\$bonearchive$cone-nwp$xnewspaper =866 \\$a1991: 2 (9[may/jun], 10[jul/aug]) =999 \\$aalma_bib_number $b991043829287003731 $calma_holdings_number $d221271756320003731 =ldr 01274nx a22000851n 4500 =008 1011252u\\\\8\\\4001uueng0000000 =005 20230424100910.0 =852 \\$bonearchive$cone-nwp$xnewspaper =866 \\$a1979: 1 (6-8 [sep-dec])1980: 2 (1-3 [jan-mar], 5 [may], 10-11 [oct-nov], [dec])1981: 3 (1-11 [jan-nov], 1{lcub}sic{rcub} s/b 12[dec])1982: 4 (1-12 [jan-dec])1983: 5 (1-12 [jan-dec]) 1984: 6 (1-12 [jan-dec)]1985: 7 (1-2 [jan-feb]); v.8 ({lcub}sic s/b v.7{rcub} (3-12 [mar-dec])1986: 8 (1-12 [jan-dec]) 1987: 9 (1-12 [jan-dec])1988: 10 (1-5 [jan-may], 6 [jun 3], 8 [jul 1], 10-17 [aug 5-dec 2])1989: 11 (1 [jan6], v.10 {lcub}sic s/b v.11{rcub} (2 [feb 3], 4-7 [mar 3-apr 21], 9 [may 19], 11 [jun 23], 13-14 [jul 21-aug 4]); v.11 {lcub}sic s/b v.12{rcub} (1 [sep 1], 2 [sep 15], 4 [oct 20], 5 [nov 3], 6 [nov 17], 7 [dec 1], 8 [dec 15])1990: 11 {lcub}sic s/b v.12{rcub} (10 [feb 2], 11 [feb 16], 12 [mar 2], 13 [mar 16], 14 [apr 6], 15 [apr 20], 16 [may 4], 17 [may 18], 18 [jun 1], 19 [jun 15], 20 [jul 6], 22 [aug 3])1991: 12 [sic s/b v.13{rcub} (12 [mar 1], 23 [aug 16])1992: 14 (2 [oct 2])1993: 15 (1 [sep 3]) 1995: 16 (11 [feb 17], 12 [mar 3])1996: 17 (8 [jan 19], 23 [aug 16]); v.18 (2 [oct 4])1997: 18 (13-14 [mar 21-apr 4], 16 [may 2], 24 [sep 5])1998: 19 (23 [aug 7]) =999 \\$aalma_bib_number $b991043829282703731 $calma_holdings_number $d221271756190003731 figure 6. examples of retrieved holdings records in marcedit (in mrk format) implementation in this section, we will provide a detailed explanation of the implementation of the python-based code for automating marc record management. this code leverages the pymarc library and other python modules to efficiently process marc records and extract issue-specific information. importing necessary libraries the code begins by importing the required python libraries. these libraries provide essential functionalities for marc record processing, file selection, regular expressions, and data manipulation. # import necessary libraries from pymarc import marcreader, field, subfield from tkinter.filedialog import askopenfilename import re from copy import deepcopy ‘pymarc’: this library is used for working with marc records, providing functions to read, manipulate, and create marc records. ‘tkinter’: this library is employed to create a file dialog for the user to select the input marc file. ‘re’: the re library is used for regular expressions, which play a crucial role in extracting issue-related information from marc records. ‘deepcopy’: the deepcopy function is utilized to create a deep copy of marc records to prevent modifying the original records. loading marc records: the code proceeds by prompting the user to select a marc file using the ‘askopenfilename’ function from the ‘tkinter’ library. this selected file will serve as the input for further processing. # load marc records from a file filename = askopenfilename() records = readfile(filename) processing and modifying marc records this section focuses on the parsefile function, which iterates through the marc records, examines their content, and extracts issue-related details. let’s break down the steps in more detail: deep copying records before processing any marc records, the code creates a deep copy of each record. this is a crucial step to ensure that the original marc records remain unaltered. deep copying prevents any unintended modifications to the source data, preserving its integrity. new_record = deepcopy(record) parsing ‘866’ field the code checks if the current marc record contains an ‘866’ field. in marc records, the ‘866’ field is typically used to store information about issues and volumes of a publication. if '866' in new_record: regular expression matching the code uses regular expressions (regex) to match and extract relevant information from the ‘866’ field. in particular, it targets patterns that represent years, volumes, and issue information. here’s a breakdown of the regex pattern: year_pattern = r’(\d{4}):\s?(\d+)?\s*\((.*?)\)’ (\d{4}): this group captures a four-digit year. the \d{4} pattern matches four consecutive digits, which represent the publication year of the issue. :\s?: this part matches a colon followed by an optional space. the colon is used to separate the year from the rest of the content. (\d+)?: this group captures an optional set of digits for volume information. the \d+ pattern matches one or more consecutive digits, representing the volume number. \s*: this part matches any additional spaces. it allows for flexibility in handling different formatting styles in the ’866’ field. \((.*?)\): this group captures text enclosed in parentheses, which typically represents issue information, such as issue numbers and issue months. the \((.*?)\) pattern matches any text enclosed in parentheses. by using these groups, the regular expression effectively extracts the year, volume (if present), and issue-related information from the ‘866’ field. formatting data once the regex matches are found, the code proceeds to format and organize the extracted data. it splits the issues into separate components, including year, volume, issue number, and issue month. issues = breakintoissues(content) issues_text = [] for issue in issues: ret = extractissuenumberandmonth(issue) if not ret: continue issue_number, issue_month = ret if issue_number != "": contain_issue_number = true if issue_month != "": contain_issue_month = true issues_text.append([year, volume, issue_number, issue_month]) splitting issues the ‘breakintoissues’ function splits the issues into a list of individual issues, typically separated by commas. extracting issue details within the section of code responsible for processing and formatting issue-related information, the extractissuenumberandmonth function is utilized to extract issue-specific details from each issue string. here’s an in-depth look at how this function works: the extractissuenumberandmonth function takes an issue string as input, which represents one of the issues extracted from the ‘866’ field of a marc record. it initializes issuenumber and issuemonth as empty strings to store the extracted issue number and issue month, respectively. the function examines the first character of the issue string: if it starts with a digit (0-9), it assumes that the issue string contains both an issue number and an issue month/year. it splits the string into two parts using the first space as a delimiter. if there’s no space, it assigns the entire string to issuenumber. if the first character is “[“, it assumes that the entire issue string is an issue month/year in square brackets and assigns it to issuemonth. if neither of these conditions is met, it means that the issue string doesn’t contain relevant information, and the function returns an empty list. if issuemonth has been identified, it removes the square brackets (if any) and performs additional cleaning by calling the cleanissuemonth function. this function standardizes issue months to a numerical format, such as converting “spring” to “21.” finally, the function returns a list containing [issuenumber, issuemonth] as extracted from the issue string. if either issuenumber or issuemonth is not present, the respective field in the list will be an empty string. the extracted issue number and issue month (if available) are then used to populate the issues_text list for further organization and are eventually added to the ‘863’ fields within the marc record. this function plays a pivotal role in accurately extracting and formatting issue-related details, making them suitable for integration into marc records. let’s consider an example issue string: 1983: 5 (7-30 [jan 28-dec 29])1984: 6 (1-4 [jan 5-jan 26], 6 [feb 9], 11-12 [mar 15-mar 22]) the extractissuenumberandmonth function processes each issue string individually. first issue (1983: 5): the function encounters the first issue in the example, which is “1983: 5.” in the example, this issue string starts with a digit (1983), indicating the year. it then looks for a space to split the issue string into two parts. in this case, it finds a space between “1983:” and “5.” the first part, “1983,” is interpreted as the year, and the second part, “5,” is the issue number. these details are extracted and stored. second issue (1984: 6): the function proceeds to the next issue, which is “1984: 6.” similar to the first issue, it recognizes the year (1984) and issue number (6). again, these details are extracted and stored. issue months and additional details: in the example, there are additional details enclosed in square brackets, such as “[jan 28-dec 29],” “[jan 5-jan 26],” “[feb 9],” and “[mar 15-mar 22].” the extracted month and date details enclosed in square brackets are now sent to the cleanissuemonth function for processing. the cleanissuemonth function removes the square brackets and further breaks down the month and date information. additionally, if the extracted issue month is a season, it is replaced by a number as explained in a subsequent section. resulting data structure the ‘issues_text’ list is structured to contain lists for each issue, each containing the year, volume, issue number, and issue month. this structured data is used to create ‘863’ fields in the marc record. creating new marc fields for each set of extracted issue details (year, volume, issue number, and issue month), the code creates new ‘863’ fields in the marc record. these fields are added to the record to store the extracted information. new_863_field = field ( tag = '863', indicators = [’40’ if '-' in issue_month or '-' in issue_number else '41'], subfields = [ subfield(code='8', value = f' 1.{i} '), subfield(code='a', value = f' {year} ') ] ) the ‘863’ field is created with appropriate indicators based on whether issue numbers and months contain hyphens, which is used to signify ranges. subfields are added to store year information. if available, volume, issue number, and issue month subfields are added. these ‘863’ fields effectively capture and organize the extracted issue-related data within the marc record. final organization after processing all the matches and creating ‘863’ fields, the code also adds a ‘853’ field to the marc record. this field is used to signify the organization of the record, including information about years, volumes, issue numbers, and issue months. the code also appends an ‘007’ field to represent the content type of the record. these additional fields enhance the cataloging and organization of marc records. new_record.add_ordered_field(new_853_field) new_record.add_ordered_field(new_007_field) by following these steps, the code efficiently processes and modifies marc records, ensuring that issue-specific information is accurately captured and organized within the records. this automation simplifies the cataloging process and enhances the consistency and accuracy of catalog records in libraries. results upon executing the .mrc file in the code program, it appended 853 and 863 fields in .txt format. subsequently, we accessed the .txt file using marc editor in marcedit. in the last phase of the cleanup, just before the update, it was imperative to ensure the removal of existing 866 fields. this step was necessary because new 866 fields were generated in alma, reflecting the updated information from the newly added 853 and 863 fields. alma facilitates the automatic generation of 866 fields for summary holding information based on the content of the linked 853 and 863 fields. =ldr 00336cy a22000851n 4500 =007 ta =008 1011252u\\\\8\\\4001uueng0000000 =005 20230424100048.0 =852 \\$bonearchive$cone-mag$xmagazine =853 20$8 $a (year) $b v. $c (season) =863 41$8 1.1 $a 1972 $b 1 $c 22 =863 41$8 1.2 $a 1973 $b 2 $c 24/21 =863 41$8 1.3 $a 1975 $b 3 $c 24 =863 41$8 1.4 $a 1976 $b 5 $c 22 =999 \\$aalma_bib_number $b991003920649703731 $calma_holdings_number $d221271644230003731 =ldr 00288cy a22000851n 4500 =007 ta =008 1011252u\\\\8\\\4001uueng0000000 =005 20230424095827.0 =852 \\$bonearchive$cone-mag$xmagazine =853 20$8 $a (year) $b (month) =863 41$8 1.1 $a 1975 $b jan =863 41$8 1.2 $a 1975 $b mar =863 41$8 1.3 $a 1975 $b apr/may =999 \\$aalma_bib_number $b991003916459703731 $calma_holdings_number $d221271636040003731 =ldr 00279cy a22000851n 4500 =007 ta =008 1011252u\\\\8\\\4001uueng0000000 =005 20230424095846.0 =852 \\$bonearchive$cone-mag$xmagazine =853 20$8 $a (year) $b v. $c no. =863 41$8 1.1 $a 1986 $b 1 $c 3 =863 41$8 1.2 $a 1988 $b 1 $c 8 =999 \\$aalma_bib_number $b991003915539703731 $calma_holdings_number $d221271633800003731 figure 7. examples of updated holding records in marcedit we identified instances where records were omitted due to incomplete information in the 866 field, specifically when the 863 fields were absent. approximately 600 records were found to lack the necessary 863 fields. consequently, we undertook the task of extracting these records for further attention. samples of these instances are provided below: =866 \\$a?: 0 (1 [?]) =866 \\$a?: (17 [?]) =866 \\$a?: 1 (6 [?], 7 [?], 12 [?]) =866 \\$a?: (3 [?]) after running the .mrc file through the code program, which added 853 and 863 fields in .txt format, the .txt file was manually inspected using marceditor in marcedit to check for any errors. in the conclusive cleanup phase, conducted just before the update, it was crucial to ensure the elimination of the existing 866 fields. this step was taken because new 866 fields were created in alma to align with the updated 853 and 863 fields. consequently, the omitted records did not generate 863 fields and lacked the corresponding information. this prompted the extraction of records lacking 863 fields, totaling approximately 600 entries. recognizing the presence of skipped holding records emphasized the necessity of collaboration with onearchives to validate actual holding information, acknowledging the limitations observed in the migrated serials holding data. in summary, we successfully updated 4,629 holding records out of a total of 5,254. throughout the updating process, alma mms id and holding id were utilized to search for and update records via the z39.50/sru client-server and marcedit. both identifiers are embedded in the 999 field, which is automatically created when downloading records through the marcedit-alma integration. figure 8. holdings information in primo after the update [4] discussion and next steps through this project, we have validated the capacity to identify patterns within semi-structured holding data using regular expressions, thereby elevating the quality of holding records. in our approach, we abstained from the conventional practice of cleaning up raw data, instead choosing to employ regular expressions to directly derive organized data from the unrefined dataset. this decision served a dual purpose, acting as both an experimental venture and, in hindsight, an unfortunate one. opting for the initial cleanup of raw data within the 866 field through openrefine, followed by the execution of the .mrk file in the code program, might have facilitated the formulation of simpler regular expressions and potentially yielded a more refined outcome. the paramount goal of this project was to enhance the presentation of serial holdings, aiming for increased user efficiency in verifying holding information. we anticipate that users engaging with information in the reopened archive will find it more user-friendly and informative. in the aftermath of this project, our future endeavors include collaborating with onearchives to address the records that were inadvertently skipped. references/notes our codebase: https://github.com/chaitupendyala/inmagic-project alma documentation: export and publishing. [accessed 2023 sep 21] https://knowledge.exlibrisgroup.com/alma/product_documentation/010alma_online_help_(english)/040resource_management/075publishing_profiles/030export_and_publishing#:~:text=alma%20supports%20the%20ability%20to,binary%20and%20dublin%20core%20xml. alma documentation: holding records. [accessed 2023 sep 21] https://knowledge.exlibrisgroup.com/alma/product_materials/050alma_faqs/print_resource_management/holding_records abrahamse, benjamin. (2010). batch marc record retrieval using z39.50. new england technical services librarians. (netsl) / 2010 massachusett. https://netsl.files.wordpress.com/2013/01/abrahamsegofishnetsl2010.pdf ansi/niso z39.71-2006 (r2011) holdings statements for bibliographic items. (2011). [accessed 2023 sept 22]. https://www.niso.org/publications/z3971-2006-r2011 marc 21 format for holdings data: table of contents (network development and marc standards office, library of congress). [accessed 2023 sep 26]. https://www.loc.gov/marc/holdings/ pymarc codebase: https://gitlab.com/pymarc/pymarc pymarc documentation: https://pymarc.readthedocs.io/en/latest/ pymarc pdf: https://pymarc.readthedocs.io/_/downloads/en/stable/pdf/ python regular expression: https://docs.python.org/3/library/re.html reese, terry. (2016). marcedit alma integration. https://blog.reeset.net/archives/1950 notes [1] one archives at the usc libraries is the world’s largest repository of lesbian, gay, bisexual, transgender, and queer (lgbtq) materials. since 2010, one archives has been an integral part of the university of southern california libraries. it is located at 909 west adams boulevard, los angeles, ca 90007. [accessed 2023 oct. 11]. https://one.usc.edu/about [2] mms id = metadata management system id. in alma, mms id can be 8 to 19 digits long (with the first two digits referring to the record type and the last four digits referring to a unique identifier for the institution). [accessed 2023 oct. 17] https://knowledge.exlibrisgroup.com/alma/product_documentation/010alma_online_help_(english)/010getting_started/085_alma_glossary [3] binary is a format consisting of a series of sequential bytes, each of which is eight bits in length. the mnemonic marc text file is a format used by marcedit that is a human-readable version of the binary file. [4] primo permalink for the example record: https://uosc.primo.exlibrisgroup.com/permalink/01usc_inst/qk93vi/alma991003915619703731 about the authors minyoung chung is a monographs and special projects cataloging librarian at the university of southern california. phani chaitanya pendyala is a computer science graduate student at the university of southern california. alongside his studies, he serves as a student assistant at the usc grand library. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – converting the bliss bibliographic classification to skos rdf using python rdflib mission editorial committee process and structure code4lib issue 59, 2024-10-07 converting the bliss bibliographic classification to skos rdf using python rdflib this article discusses the project undertaken by the library of queens’ college, cambridge, to migrate its classification system to rdf applying the skos data model using python. queens’ uses the bliss bibliographic classification alongside 18 other uk libraries, most of which are small libraries of the colleges at the universities of oxford and cambridge. though a flexible and universal faceted classification system, bliss faces challenges due to its unfinished state, leading to the evolution in many bliss libraries of divergent, in-house adaptations of the system to fill in its gaps. for most of the official, published parts of bliss, a uniquely formatted source code used to generate a typeset version is available online. this project focused on converting this source code into a skos rdf linked-data format using python: first by parsing the source code, then using rdflib to write the concepts, notation, relationships, and notes in rdf. this article suggests that the rdf version has the potential to prevent further divergence and unify the various bliss adaptations and reflects on the limitations of skos when applied to complex, faceted systems. by harry bartholomew background of the bliss bibliographic classification the second edition of the bliss bibliographic classification (bc2) is a universal, fully faceted, highly synthetic classification system used in 19 english libraries, the first schedules of which were published in 1977. though microscopic when viewed on the scale of dewey decimal (ddc) and library of congress (lcc) classification systems in terms of the number of libraries using the scheme, 11 of the bliss-classified libraries are found within the collegiate universities of oxford and cambridge, making bc2 a competitive alternative to ddc and lcc in these academic library communities. often chosen to replace simple enumerative in-house classification systems at oxbridge college libraries in the 1980s-90s, bliss’s synthesisability and granularity responded to the obstacles to browsing these imposed in the pre-computerised and early computerised periods of library catalogues, and it served a double purpose as a source for subject terminology in bibliographic records ( [1] sargent 1990) ( [2] watson 1997). bliss allows for the synthesis of classmarks from multiple through a process it terms retroactive notation, explained thus on the official website: the classifier must first analyse the subject of the work, then arrange the components, or facets, of the subject into the reverse order of the bc2 schedule, going from the most specific to the most general facet. the classmarks for each facet are then combined by dropping the repeated initial class letter from all but the first. this technique is called retroactive notation. as bc2’s citation order is inverted, the facet cited first actually comes last in the schedules—( [3] bliss classification association : bc2 : using the scheme) for example, in the philosophy & logic schedule, aco b is the class for subjectivism and ahk for ethics, so ethical subjectivism would have the synthesised classmark ahk cob. the idea here is that the schedule need not repeat itself by enumerating every possible classmark; instead, a more specific class can always be subdivided by the more general aspects preceding it, thus allowing an incomprehensibly vast number of potential notation and subject combinations from a reasonably sized list of concepts. 1977 saw the first volumes of bc2 published, however work on the system continues as many schedules are still only available as unfinished drafts. while efforts of the bliss classification association (bca)—the editorial body comprising bliss users and supporters—focus on finalising the unpublished schedules, those finished decades ago have a growing need for maintenance. its incompleteness and partial outdatedness have led to divergence in the application of the classification scheme between libraries, with significant in-house patches having been developed by individual libraries inconsistent with their fellow bliss-users. the first published bliss schedules were manually typed and photographed for publication, though a machine-readable encoded form was later developed alongside software to generate a formatted schedule and index from the encoded data ( [4] the way we were: development of the printed schedules in bc2 2007). figure 1 shows an extract from the final published version of the class a – al: philosophy & logic schedule and figure 2 shows the source code from which it was generated. the two digits preceding the node label indicate the indentation on the published page used to show the hierarchy of concepts. the indentation bullets in figure 1 do not equal the number in the source code; each column must have indentation relative within itself, and a header at the top of each column shows the current location in the concept hierarchy. figure 1. section from published schedule class a – al: philosophy & logic figure 2. extract of source file used to generate the formatted version shown in figure 1. the bc2 source code adheres to an idiosyncratic schema which allows for the generation of properly typeset and indexed schedule to be used by classifiers. the structure of the source code can be summarised in the following way: the file must be written in the order of the notation. each concept begins a new line with its notation as the first character, or with @ if there is no notation for the concept. the notation is followed by one or more spaces. the first 2 digits following the notation and spaces denote the length of the indentation on the page. the indentation numbers are followed by the labels used for the concept, each separated by a comma. the labels can be enclosed in a pair of matching brackets, indicating the node’s category: )inverted parentheses( show a “brought-down” class, meaning that the concept is first defined earlier in the schedule, but reappears at this point subsumed under a specific parent class. (normal parentheses) indicate that this node represents a facet ((double parentheses)) indicate a node introducing an array of classes the visibility of a concept in the schedule, thesaurus or index is controlled by a singular closing square bracket followed by the initial of the output in which the label is not to be visible. e.g. in the following snippet, the labels for the alphabetical ranges for classifying named but unlisted 17th-century british philosophers will not appear in the index or the thesaurus: ads 0917th century ]it adse 10)british philosophy( adsecy 11((schools & doctrines)) adsed 12cambridge platonists @ 11((individual philosophers)) adsef 12a bac ]it adseg 12bacon f adseh 12bac hob ]it adsej 12hobbes t adsek 12hob loc ]it adsel 12locke j adsem 12loc z ]it new lines beginning with spaces followed by an asterisk indicate a note; *sn indicates a scope note; ** precedes comments. labels and notes can continue multiple lines. one can easily draw parallels between this source code format for classification data and marc for cataloguing. roy tennant, in his influential and well-cited library journal denunciation, “marc must die,” explains that “marc and aacr2 are largely focused on capturing the paper catalogue card in computer form” and bases his critique on the format’s inconsistent granularity, limited applicability, and relative obsolescence in the face of xml ( [5] tennant 2002). in like fashion, the bc2 source code’s aim to digitally reproduce the printed schedule limits its extensibility, and its flat unitary structure impedes the systematic expression of relationships between concepts. the software developed and used by the bca to parse the source code is not openly available, and so the hitherto focus on the end product (the published schedule as a physical volume or an inflexible pdf) has withheld useful editorial capabilities from the classifiers at bliss-using libraries, who, anecdotally, often rely on handwritten annotations to the volumes to record their local applications of and updates to the system. aim and objectives the library of queens’ college, cambridge, needed a more robust system than handwritten annotations and printed addenda to manage its classification. queens’ converted to bliss in 1988, relying in large part on draft schedules to cover the unpublished subject areas ( [6] sargent 1990: 11). subsequent librarians have updated and revised the system and logged these changes with varying diligence. schedule annotations indicated, where bliss offers alternatives in notation and structure, which variant is locally used. annotations also revised offensive and prejudicial terminology and hierarchies ( [7] bartholomew 2023). where sections of the system were deemed too granular for our purposes, ticks and crosses showed which classes were to be used. for subjects still without an official bc2 schedule, printed in-house classification systems were kept in ring binders, and these too were more often revised by hand rather than by editing and reprinting the documents. schedule indexes were not consistently updated with revisions, leading classifiers looking up terms in the index to obsolete parts of the schedule. when changes were made to the system, reclassification projects were limited to the open-access collection only, meaning books in closed-access stores are arranged according to an older version of the system. as books are routinely relegated to storage, the closed-access shelves follow a disorganised shelving order according to multiple versions of the system. further to this, queens’ had no consistent system to record the components of its synthesised classes, having ceased to use bliss class labels as subject vocabulary in bibliographic records, and so a revision to a class’s notation was difficult to implement as there was no complete index to show which books’ classmarks used the old notation in its synthesised form. lastly, and importantly, queens’ classifiers cannot be sure whether the classmark assigned at a fellow bliss library for a particular book is valid for the same book in the local system, as system revisions in one or both libraries could have led to a divergence; therefore, copy-classifying is severely hindered. the objectives of an encoded classification system for queens’ college thus were: replicate the structure of the source code, avoiding the loss of any information encoded in the original file. allow for the generation of a formatted and indexed schedule, both in print and electronic form, from the encoded form. use persistent identifiers for concepts in the scheme, so that links can be expressed between: (1) different classification systems and variants; (2) bibliographic metadata and the classification system; (3) a synthesised classmark and its components. enable a form of version control logging changes to notations and labels so that the origin of a particular classmark can always be traced. skos-rdf structure the simple knowledge organisation system (skos) is a model for use in rdf that standardly encodes a traditional thesaurus, taxonomy, or classification hierarchical structure. a skos concept can have a preferred and alternative label, a notation, various types of notes, and expressions of broader, narrower and related concepts. development of skos began in 2006, and soon its potential use for bliss was discussed in the bliss classification bulletin. alistair miles ( [8] 2006), whilst making the case for skos-encoding, pointed out that “currently there is no built-in support for the synthesis of conceptual units to represent compound meanings” (p.11). leonard will ( [9] 2008) also observes the limitation of skos for encoding bliss, whose schedules “show examples of pre-coordination” but “users are expected to create others as required. skos can not yet encode this type of structure”. this remains true of skos in 2024. despite this limitation, skos was still chosen as the data model for this project. an owl ontology categorising concepts more granularly and defining more particular potential relationships between concepts could be developed for bliss, which could then define the node categories indicated by the enclosing parentheses as well as encode visibility of labels in the index and thesauri. however, as the published version of bliss still adheres to a traditional hierarchical classification structure, it is entirely possible to replicate the hierarchy in skos; further to this, polyhierarchical relationships are expressible in skos, and so a compound class could be a narrower concept of multiple concepts from different branches of the schedule. skos, owing its eponymous simple nature, has a small vocabulary, and it was applied to queens’ classification as follows: skos: conceptscheme each bliss schedule is considered a separate skos:conceptscheme, owing to the fact that the principle of retroactive notation applies only within a schedule, and compounding between schedules is only sometimes permitted. also, queens’ in-house schedules adhere to separate compounding rules. skos:concept every node in a schedule is a skos:concept, regardless of whether it has a notation; notationless nodes are necessary to describe the structure of the hierarchy and separate arrays for different facets. skos:inscheme every concept is linked to its conceptscheme. skos:preflabel the first label in a list of comma-separated values for each node is treated as the preferred label. skos:altlabel all subsequent labels are included as alternative labels for the concept. skos:broader every child concept has its parent concept as a broader term. skos:topconceptof concepts without a parent concept are linked to the concept scheme as a top concept. skos:note basic notes following a label. skos:editorialnote used for source code comments. skos:scopenote used for notes defined as scope notes in the source file. these do not represent the full range of the skos vocabulary; some classes and properties were deemed superfluous to requirements; others could not be encoded automatically based on the source file data. implicit inversion pairs of skos uris are inversions of one another, e.g. if a is broader than b, then it follows that b is narrower than a; if x is the top concept of the y concept scheme, then y has x as a top concept. to avoid redundancy and for consistency, the hierarchy is expressed in only one direction throughout, narrowest to broadest. transitivity the skos broader/narrower relationships are neither inherently transitive nor intransitive; they merely express immediate hierarchical relationships. while transitivity can be explicitly expressed with skos:broader/narrowertransitive properties, the potential benefits of this are unclear and it could be that an explicit declaration of transitivity may turn out to be problematic in places. collections skos concepts can be collected into arrays using the skos:collection and skos:orderedcollection classes, in which the concepts would be members. in some ways, this would usefully reflect the nature of the array nodes enclosed in double parentheses in the source file. however, array nodes are treated similarly to ordinary classes in the source file structure, where they are the narrower term of the parent concept and the broader term of the array of child concepts. as skos:collection is disjoint with skos:concept, the children would need to be narrower terms of the concept above the array node, thus disrupting the established hierarchical structure indicated in the source file. some array and facet nodes also have a notation and could therefore be used as a concept, or in a synthesised class. therefore, collections were not used in this automatic conversion, but could be useful in future revisions if it is decided that the hierarchical structure ought to be altered. notes the source file distinguishes only three types of note which can be systematically converted, though other subproperties of skos:note would be useful. a skos:historynote would serve when revising the system, and notes acting as definitions for the class ideally would be separated from notes advising on its usage within the system. parsing the source file with python and rdflib imports and constants from rdflib import graph, rdf, literal, namespace from rdflib.namespace import skos ex = namespace("http://example.org") g = graph() the only imported library is rdflib, from which five objects used: rdflib.graph is the web of nodes and relationships which is eventually serialised into turtle format in this project rdflib.rdf is the rdf vocabulary rdflib.literal is the rdfs:literal property (i.e. strings and integers) for use when the datatype must be specified rdflib.namespace allows for generation of uris with a common prefix rdflib.namespace.skos is a readily available skos namespace the queens’ classification is not yet available on the web and so a placeholder namespace is used in this project temporarily. this still allows the resulting turtle file to be visualised and aid local classification. creating the concept scheme def parse_source_file(file_path): with open(file_path, ’r’) as file: g.add((ex[""], rdf.type, skos.conceptscheme)) the first class added is the concept scheme, which currently creates a bare node to which top-level concepts can be linked: <http://example.org> a skos:conceptscheme . this will need to be named and described before the classification is hosted online. parsing the file the source file is read line-by-line, however the challenge lay in the fact that node labels and notes can, and often do, span multiple lines. def identify_line(line): if line[0] != " ": return "new node" if line.strip()[0] == "*": return "new note" else: return "continued" if a new line begins with a non-space character (either the notation or an “@” to indicate no notation), it is a new node, and each note always begins with an “*”. if neither of these conditions are true, it is assumed that the line continues its predecessor. a new node is assigned a uri suffix consisting of a 4 digit suffix to the namespace indicating its order in the source file. unlike the integrated levels classification (ilc) which was also migrated to skos ( [10] binding et al. 2021), bliss notations are not unique identifiers: the same notation, or no notation at all, can apply to multiple classes, hence the need for arbitrary generation of identifiers. structuring the hierarchy another way in which bc2 differs from ilc is bliss’ dissonance between its hierarchy and its notation. where in ilc, “crucially, the hierarchical structure of the classification was inferred from the letter sequence of the notational codes” ( [11] binding et al. 2021: 933), a bliss child concept could have a briefer notation than its parent class, e.g. bgy is the notation for electricity & magnetism, yet bh is the notation for its child concept electromagnetism. the hierarchy of concepts must therefore be inferred from the numerical indentation indicator from the source file, and is tracked using a python dictionary, starting as: hierarchy = {0: "root"} then, narrower concepts are added to this dictionary as the parsing progresses, with their indentation number as the key and their uri as the term. def create_hierarchy(hierarchy, uri, line): def extract_indentation(line): return int(line.split()[1][:2])</p> indentation = extract_indentation(line) hierarchy[indentation] = uri if indentation == 1: g.add((uri, skos.topconceptof, ex[""])) else: broader_uri = hierarchy[indentation 1] g.add((uri, skos.broader, broader_uri)) return hierarchy anything with an indentation of 1 is a skos:topconceptof the concept scheme, and those further down the hierarchy have their parent concept’s uri added with a skos:broader relationship. adding rdfs:literal properties to nodes 3 types of literal properties can be added to each node: labels, notation, and notes. the notation is easily extracted, with “@” interpreted as an absence of notation. the string of characters between the indentation indicator and the subsequent note or node is the label string, consisting of a list of comma-separated labels that can have enclosing brackets, denoting its “category”, or a final closing square bracket, providing instructions for the visualisation of the classification schedule and index or inform on the origin or type of the concept. the information these brackets convey are not encodable in skos alone, but the bracket type is kept in a skos:editorialnote so that any future interpreter can make use of it. the labels, extracted from any brackets are added to each node, with the first in the list being the preferred label, and following labels as alternative labels. notes are then added with their preceding asterisks removed. output the output of this conversion process is a turtle file. the output of the sections shown in figures 1-2 is: @prefix skos: <http://www.w3.org/2004/02/skos/core#> . <http://example.org/0045> a skos:concept ; skos:broader <http://example.org/0022> ; skos:editorialnote "("@en, "facet"@en ; skos:inscheme <http://example.org> ; skos:note "these are defined by period & place and are therefore only found within a particular broad tradition. see ad western philosophy & ai oriental philosophy."@en ; skos:preflabel "historical schools of philosophy"@en . <http://example.org/0047> a skos:concept ; skos:broader <http://example.org/0046> ; skos:inscheme <http://example.org> ; skos:notation "aafb" ; skos:note "see explanatory notes & examples at ag."@en, "very little, if anything, will locate here."@en ; skos:preflabel "comprehensive works on branches of philosophy"@en . <http://example.org/0048> a skos:concept ; skos:broader <http://example.org/0046> ; skos:inscheme <http://example.org> ; skos:notation "aafd" ; skos:note "general studies only; see notes at aaf."@en ; skos:preflabel "metaphilosophy"@en . <http://example.org/0049> a skos:concept ; skos:broader <http://example.org/0046> ; skos:inscheme <http://example.org> ; skos:notation "aagg" ; skos:preflabel "metaphysics"@en . <http://example.org/0050> a skos:concept ; skos:broader <http://example.org/0049> ; skos:inscheme <http://example.org> ; skos:notation "aagh" ; skos:preflabel "ontology"@en . <http://example.org/0051> a skos:concept ; skos:broader <http://example.org/0049> ; skos:inscheme <http://example.org> ; skos:notation "aagm" ; skos:preflabel "cosmology"@en . <http://example.org/0052> a skos:concept ; skos:broader <http://example.org/0049> ; skos:inscheme <http://example.org> ; skos:notation "aagq" ; skos:preflabel "special topics in metaphysics"@en . <http://example.org/0053> a skos:concept ; skos:broader <http://example.org/0046> ; skos:inscheme <http://example.org> ; skos:notation "aagr" ; skos:preflabel "epistemology"@en . <http://example.org/0054> a skos:concept ; skos:broader <http://example.org/0046> ; skos:inscheme <http://example.org> ; skos:notation "aagw" ; skos:preflabel "philosophy of language & logic"@en . <http://example.org/0055> a skos:concept ; skos:broader <http://example.org/0046> ; skos:inscheme <http://example.org> ; skos:notation "aahd" ; skos:preflabel "philosophy of mind &amp; action"@en . <http://example.org/0056> a skos:concept ; skos:broader <http://example.org/0046> ; skos:inscheme <http://example.org> ; skos:notation "aahj" ; skos:preflabel "axiology"@en . <http://example.org/0057> a skos:concept ; skos:broader <http://example.org/0056> ; skos:inscheme <http://example.org> ; skos:notation "aahk" ; skos:preflabel "ethics"@en . <http://example.org/0058> a skos:concept ; skos:broader <http://example.org/0056> ; skos:inscheme <http://example.org> ; skos:notation "aahp" ; skos:preflabel "aesthetics"@en . <http://example.org/0059> a skos:concept ; skos:broader <http://example.org/0046> ; skos:inscheme <http://example.org> ; skos:notation "aahr" ; skos:note "alternative (not recommended) to locating with subject."@en, "not restricted to one broad tradition. even if this option is used most of the literature will go at ahr (where instructions for use are given)."@en . skos:preflabel "philosophy of particular subjects (general)"@en ; achievements, limitations, and future directions this project provides a method for converting bliss source files into rdf using the skos data model. this has so far only been used internally at queens’ college library, and while other bliss-using libraries are welcome to download the rdf version or even convert the files themselves using this code (or an adapted version of it), the full benefits of an rdf-encoded linked classification system can only be realised when the uris become urls, i.e. accessible via the internet, and ideally with an api endpoint. skos alone is arguably too simple for bliss, and a more complex ontology is needed to extend the basic hierarchical classification structure. many properties encoded as rdfs:literal in a skos:note inform the classifier of how the class can be used to build compound classmarks; defining bliss-specific properties and relationships between concepts could make this standardised and machine-readable. a further limitation is that the documentation for the bliss source file format provides more flexibility in formatting than is outlined here. the python code used for conversion assumes that a particular standard has been followed, though does not account for valid variants that may be used in other schedules’ source files. the code will need to be edited to accommodate conversions of such files. consistent with the source files, but a hindrance for linked data, is the lack of relationship between the nodes of “brought down” classes and their origin. given the polyhierarchy permitted in skos, an altered structure could see the children of brought down classes as narrower concepts of both their parent class, but from earlier in the schedule, and their grandparent class. for example, in the source file snippet featuring 17th-century british philosophers above, adsed: cambridge platonists could be restructured as a narrower term of both ads: 17th century western philosophy and adq e: british philosophy, though this might entail label changes (figure 3). figure 3. current simple hierarchical structure compared with a proposed polyhierarchical structure it is hoped that queens’ rdf classification will eventually incorporate links to an official bliss rdf version. where there has been a schism from bliss orthodoxy, the skos mapping properties can be used to show the matches and variations between the standard and local versions. bibliography [7] bartholomew h. 2023. obsolete orders: the need to reclassify queens’ war memorial library. queens’ college old library blog. https://queenslib.wordpress.com/2023/05/15/obsolete-orders-the-need-to-reclassify-queens-war-memorial-library/. [10][11] binding c, gnoli c, tudhope d. 2021. migrating a complex classification scheme to the semantic web: expressing the integrative levels classification using skos rdf. jd. 77(4):926–945. doi:10.1108/jd-10-2020-0167. [3] bliss classification association : bc2 : using the scheme. [accessed 2024 may 17]. https://www.blissclassification.org.uk/bcclass.shtml. [8] miles a. 2006. simple knowledge organisation and the semantic web. the bliss classification bulletin.(48):10–13. https://www.blissclassification.org.uk/b48.pdf#page=10 [1][6] sargent c. 1990. classifying the undergraduate collection at queens’ college, cambridge. the bliss classification bulletin. (32):10–12. https://blissclassification.org.uk/b32-1990.pdf#page=10 [5] tennant r. 2002. marc must die. library journal. 127(17):26–27. https://www.libraryjournal.com/story/marc-must-die [4] the way we were: development of the printed schedules in bc2. the bliss classification bulletin. 2007. (49):17–22. https://www.blissclassification.org.uk/b49.pdf#page=18 [2] watson r. 1997. quincentenary library, jesus college, cambridge. the bliss classification bulletin.(39):14–16. https://www.blissclassification.org.uk/b39.pdf#page=14 [9] will l. 2008. bliss on the web. [accessed 2024 feb 25]. https://www.blissclassification.org.uk/blissontheweb.pdf. about the author harry bartholomew (harrybartholomew.github.io) is a librarian with systems responsibilities at the royal institute of international affairs, london, united kingdom. until june 2024 he was the reader services librarian at queens’ college, university of cambridge. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – bc digitized collections: towards a microservices-based solution to an intractable repository problem mission editorial committee process and structure code4lib issue 44, 2019-05-06 bc digitized collections: towards a microservices-based solution to an intractable repository problem our digital repository services department faced a crisis point in late 2017. our vendor discontinued support for our digital repository software, and an intensive, multi-department, six-month field survey had not turned up any potential replacements that fully met our needs. we began to experiment with a model that, rather than migrating to a new monolithic system, would more closely integrate multiple systems that we had already implemented—archivesspace, alma, primo, and metaarchive—and introduce only one new component, namely mirador. we determined that this was the quickest way to meet our needs, and began a full migration in spring of 2018. the primary benefit of a microservices-based solution for our collections was the potential for customization; we therefore present our experiences in building and migrating to this system not as a blueprint but as a case study with lessons learned. our hope is that in sharing our experience, we can help institutions in similar situations determine 1) whether a microservices-based solution is a feasible approach to their problem, 2) which services could and should be integrated and how, and 3) whether the trade-offs inherent in this architectural approach are worth the flexibility it offers. by chris mayo, adam jazairi, paige walker, luke gaudreau introduction in the summer of 2017, the digital repository services department at boston college libraries initiated a project to replace our digital special collections repository platform. we first implemented digitool, an ex libris product, in 2003. while it served our needs for several years, the product eventually reached end of life, and ex libris discontinued active development. we had configured an interim discovery platform based on blacklight in 2015, but this interface relied on the digitool backend and image viewer, and we knew all of these components would eventually need to be replaced. one strength of digitool was its ability to interpret and render sophisticated mets structmaps. our implementation made extensive use of the structmap data structure to order and show the relationship among multiple constituent files in a single digital object. maintaining this functionality was a requirement; in order to mirror the reading room experience for archival materials, we have always matched our digitization workflows and the presentation of digitized materials to the levels of description and the physical arrangement of the materials. we used mets to represent order within items and files, and in some cases the differences between the physical and intellectual order of materials. we were committed to maintaining this robust level of structural description, but we also understood that this functionality was uncommon in repository software. with our custom mets implementations in mind, we initiated an environmental scan to determine if there were any viable repository candidates that could fit our needs. after identifying our functional requirements, we set off on a six-month odyssey to research and test eight options in consultation with a diverse group of stakeholders from archives, systems, and digital scholarship. although some of these candidates seemed promising at the outset, they each presented challenges. some were new open-source solutions that did not yet include the functionality we required, while others were proprietary systems that lacked sufficient transparency and customizability. others were more mature open-source solutions that raised concerns about long-term support. our assessment process ultimately failed to identify a monolithic solution. conception by this point, it had become clear to us that we needed to think creatively. in a final attempt to meet our needs with existing software, we sent an email to the samvera community listserv, requesting responses from institutions using mets in a samvera context. we received a reply from a developer who informed us that mets was part of his institution’s workflow, but that it was used primarily to generate iiif manifests, and that these manifests, not the mets itself, were used to render structured digital objects. around this time, we received some alarming news from our university’s information technology services department. they had been scanning local servers for struts, a development framework with critical vulnerabilities that were exploited during the equifax hack. as it happens, digitool was built with struts, and because the application was end-of-life, it was using a particularly outdated version. because digitool is closed-source software, we had little agency to address the root cause of this problem. instead, we worked with its to isolate the server from our campus network to minimize the impact of any potential attacks, then rapidly accelerated our migration timeline. we had previously reacted to the samvera community response by surmising whether we could make do by spinning up an iiif image server to host our digital collections. now with the report from its, it started to seem like a more viable solution. we began discussing what services we would actually need to support search, display, and data backup and integrity in a more distributed ecosystem, and began seriously considering a solution based on the microservices architectural model, where multiple single-purpose systems are integrated to provide the functionality of a larger piece of multi-purpose software. in brainstorming the services provided by repository software that we’d need to replicate, we discovered that systems we already had in place could cover a surprising amount of ground. our archivists create and maintain canonical descriptive metadata for archival materials in a locally hosted instance of archivesspace. we already had workflows in place for automatically generating description of digital archival objects (daos) in batches, and for exporting mods and mets to ingest into our repository systems along with digitized files. in addition to our custom blacklight discovery interface, we also harvested our digitized collections into primo, our library discovery platform. while this allowed for serendipitous discovery alongside other library resources, it did not provide a facility for browsing collections. conveniently, as we were researching solutions, ex libris announced a new collection discovery feature. with this feature, we could build collections in alma and publish them to primo alongside other records. primo then presents these as browsable collection galleries. alma also includes an oai-pmh provider, which we could leverage to serve content to external entities like digital commonwealth. from a digital preservation standpoint, digitool had conducted automated fixity checking of collections and stored the checksums in its database. however, we also stored a second set of our collections outside the digitool environment. these copies lived on an in-house preservation server, which was configured to run automated checksum audits using audit control environment. additionally, copies of these collections were transferred from this server to our lockss-based distributed digital preservation solution, metaarchive. even if our new solution lacked preservation functionality, we could make up for that with already extant systems. we also discovered, as anticipated, two major areas where we would lack coverage after moving out of a traditional repository system. the first was an image viewer. we saw the migration from digitool as an opportunity to adopt the international image interoperability framework (iiif), an exciting initiative that we were eager to apply to our own digital library work. while alma offers partial iiif support with its implementation of universal viewer, the viewer services module was not customizable enough for our needs. digitool had also provided access to technical metadata through digital entity files stored in a database. although this option was clunky and often riddled with imperfections, it had fulfilled our need to search for and store premis within our repository. we recognized that we would also need to devise a way to store technical metadata in an archival management system. feasibility as we explored the feasibility of our proposed digital repository infrastructure, one of our most pressing concerns was how to render structural metadata effectively. we used osullivan, a ruby implementation of the iiif presentation api, to write a ruby gem that would convert our mets files into iiif manifests. fortunately, our digital scholarship group had already developed something similar for a separate iiif project, which we used as a starting point. because that code was written for a specific context, we had to refactor it significantly to make the gem compatible with all of our mets. this proved time-consuming but well worth the effort, as it provided the foundation for another gem, aspace-to-iiif, that generates a iiif manifest from an archivesspace digital object record. selecting the right image viewer also took some trial and error. we demoed a few iiif viewers to our stakeholders, but the one that stood out was mirador. mirador had been on our radar as a potential viewer for digital scholarship projects, but we had not yet implemented it at boston college. for this use case, it proved to be a perfect match. our team appreciated the clean, modern interface, as well as the ability to display structural information in the side panel. for our iiif image server, we wanted something that would be relatively easy to install and maintain. we decided on loris, which was already in use for a different project at the libraries. while we were pleased with the user experience presented in primo, we found ourselves with several new challenges. first, we needed to devise a workflow for importing our dao records into alma. since archivesspace serves as our source of canonical metadata, we only needed the alma records for arranging into collections and publishing to primo. with some trial and error we were able to generate exports of object records in marc format that captured critical descriptive metadata and distinguished the material from other alma records. we worked to arrange the alma records into collections by building sets of records matching by collection name. we also needed to address the need for persistent links to dao records in primo. while primo does include a permalink mechanism out of the box, it is not customizable and the links cannot be predictably controlled. we worked around this by creating a primo customization that replaced the default permalink with a new one based on the legacy digitool pid and our internal handle server. we also added this pid as a searchable pnx field and added it to our existing primo search middleware, primo-services. our handle server was then configured to point to primo-services to perform a search for the given pid. finally, we needed to determine whether it was possible for us to generate and store technical metadata in archivesspace for our files, to have this data available outside of our preservation environment. as we already had a process in place to automatically build digital archival objects with bitstream-level data, this was primarily a question of finding an appropriate file characterization tool and mapping data to appropriate fields in the database. we began experimenting with exif, but eventually settled on fits for its ability to characterize a wider range of filetypes. some, but not all of the data we were interested in storing already has controlled fields in the archivesspace data model, and so in many cases we have been forced to store data in labeled general notes fields. managing these fields will be an ongoing challenge. when we had outlined all of our needs and the ways we expected to fill them, we created the system model depicted in figure 1. figure 1. diagram of system model. outcomes as noted, the purpose of this article is to outline our experiences with implementation of a microservices-based repository solution. given that any institution implementing a similar solution will ideally be selecting individual systems that best meet their own needs or are already implemented locally, we have chosen not to describe the technical specifics of our migration and integration workflows. we also omit our efforts to implement an agile-style project management mode for the process of building and migrating to this system. we intend instead to present our outcomes as a series of challenges and positive outcomes, with an analysis of how each outcome relates to the microservices model: are there problems we could have avoided, or desired outcomes we would have lost, if we had not pursued a microservices-based solution? one aspect of our new architecture that seemed beneficial at first was use of the description in archivesspace as the canonical source of descriptive metadata. it had to be verified against our legacy data, and we would need to devise a way to publish it in primo, but with the description already present in archivesspace, no migration was required. however, this assumption was refuted by two of our major digitized collections. one, an archival photographic collection consisting of tens of thousands of images across hundreds of files, was not described in archivesspace because we had not yet implemented an archival management system when digitization on this collection began over a decade ago. a similarly large collection of early new england financial and legal documents had not been described in archivesspace because it is owned by our law library, not our archives. in both cases, we used an export of legacy metadata to make these collections available in bc digitized collections. still, they would require extensive backend work to be maintainable. in the six months since the system launched, we have partially imported the data for the photographic collection into archivesspace, but much work remains to be done, and work on the legal documents collection is only just beginning. if we had been migrating to a new repository, we would have these collections under better intellectual control in the digital repository, but we would not have had the excuse we currently have to put the effort into migrating them into archivesspace. the microservices model has unarguably given us more work, but it is work that needed to be done. using alma and primo to make our digitized collections discoverable presented its own challenges. significant resources were dedicated to trial and error configuration of primo normalization rules for display and search. in many cases the rules are duplicates of existing rules targeted specifically to digital archival objects. the large number of rules and their reliance on precise combinations of metadata values has translated to significant maintenance costs. we also learned that primo collection discovery relies on additional api calls to build the browseable collection hierarchy. that dependency has proven to be problematic, with performance, caching, and security problems abounding. over time many of these problems have been fixed, but in the meantime it has required a large number of manual workarounds, communication with public services staff, and persistent troubleshooting with ex libris. our integration of alma in the bc digitized collections architecture is a weak point, particularly with regard to archivesspace. we currently export archivesspace digital object records as mets with embedded mods, then transform them to marc before ingesting to alma. this workflow is effective, but it requires a fair amount of manual work that could be automated. we plan to explore the archivesspace oai provider as a potential solution, and have also discussed the possibility of writing a plugin to export digital objects as marc. there is interest within archivesspace community to pursue alma integration (kevin clair at the university of denver has already written a plugin to that effect), so this could be an opportunity to collaborate with other institutions that have similar needs. many of our preservation-related workflows remain manual rather than automated. after its initial ingest into archivesspace, technical metadata is updated infrequently. our fixity checking program, audit control environment, does not integrate with archivesspace, so related premis events are not documented alongside other metadata. we also lack a solution for batch normalization of select file formats. to account for these needs, our team is investigating archivematica as a potential addition to our suite of microservices. in spite of these challenges, we’ve also experienced an equal or greater number of positive outcomes. similarly to the challenges, many of these beneficial outcomes are related to the choice of a microservices model. of the eight potential repository solutions that we investigated before undertaking this project, five were open-source and three were vended solutions. we have the available technical resources that if we had chosen an open-source solution, we would have been hosting it locally, and our experiences with digitool had strongly predisposed us toward open-source software. due to the proprietary formats used in the database, although we were able to get our data out of digitool, we did not reuse it extensively in the migration process other than as an audit checker. some of the components of our solution are vended—primo and alma—and others are locally controlled and installed instances of open source projects—archivesspace, mirador, loris, and audit control environment. critically for us, our canonical metadata lives in an open-source management system, which helps us feel more comfortable about our ability to migrate forward in the future. although there were additional considerations, one of the reasons an open-source repository was infeasible for us to implement was the amount of time our current staff of systems and support librarians had available. as noted, we already host several open-source systems locally, but we haven’t customized any of those systems to the extent that the available open-source repositories would have required. we felt that we couldn’t do justice to our needs without committing 1 fte of developer time, which we simply didn’t have. had we chosen a vended solution, upfront and ongoing monetary costs would have been significant. we were able to implement bc digitized collections without purchasing additional software or hiring new staff, clearly making it our most financially responsible choice. while we relied on dedicated project time from a systems librarian during the implementation and migration, ongoing maintenance and upgrades have been folded into existing primo support mechanisms. we also gained a seat on our library’s primo working group, where we can advocate for our needs alongside issues from other stakeholders. it’s difficult to say whether we’d have been able to implement an existing repository solution, open-source or vended, in such a budget-neutral way. however, this budget-neutrality was achieved partly through a combination of extant technical skills and our own interest and ability to gain new skills. in particular, our digital repository applications developer gained more expertise with the iiif presentation and technical apis in his support of mirador and loris and manifest generation. our digital collections and preservation librarian explored and implemented fits in her file characterization workflows to enable us to generate and store technical metadata. our digital production librarian expanded their work with the archivesspace api to support import and linking of new metadata fields, both technical and descriptive. the microservices model allowed us to select and implement services that matched with and enhanced our staff’s current technical capabilities, benefiting the project and the library. we committed a significant amount of staff time to develop migration tooling for this project. ordinarily, such scripts and transforms would hold little value once the migration ends, but we were thrilled to find the opposite with bc digitized collections. some of these tools were brand new, such as our aspace-to-iiif gem. others were heavily revised and refactored, such as the script that we use to batch generate digital objects. regardless, many of them still find regular use in our daily workflows. the microservices approach to our repository architecture has allowed us to develop greater flexibility and transparency in our workflows. because these services are interoperable, we can apply them beyond our repository to improve our digital library program as a whole. for instance, we are now exploring integrations between archivesspace, bitcurator, and archivematica to create a more robust digital preservation campaign. due to the open-source nature of many of these programs, we can also analyze their code to better understand their technical foundations. this allows us to assess how programs may impact the integrity of our collections and the privacy of our patrons. we believe that these flexible integrations will lead to fewer silos, more efficient workflows, and a stronger digital library overall. finally, one of our enduring goals as a department is to connect with and contribute to the larger community of practice. by taking an active role in multiple open-source communities, this project has positioned us to do that. our interactions with the individuals and institutions involved in archivesspace and iiif have been deeply rewarding, and we look forward to working more with these communities. next steps as bc digitized collections was a significant departure from our previous repository platform, we were eager to see how our end users responded to it. we conducted usability testing in fall 2018, which predictably revealed some design flaws. for example, we currently link to an external instance of the mirador viewer from catalog records, which caused confusion for some test participants. we made this design choice because it was the fastest way to ship the product, but we would like to explore embedding mirador in primo to reduce friction. we plan to test this with the upcoming mirador 3. similarly, we would like to expand our iiif implementation, which at this point is fairly lightweight. we plan to add support for annotations, and are beginning to consider iiif as a delivery mechanism for audiovisual content, as will be supported by version 3 of the presentation api. we have also discussed extending our iiif image server to other uses, such as a web app that allows staff to resize and crop images. usability testing also revealed biases in our design process that had gone unnoticed. a particularly salient example of this is the name ‘bc digitized collections’. as archivists and librarians, we took for granted that our users would be able to decipher this and understand what they would be able to find within the system. in fact, our test results suggested the opposite. a reevaluation of how we are naming and describing the system is forthcoming, with a greater sensitivity to our end users’ level of familiarity with library jargon and information architecture conventions. we must also work to streamline how some of our systems are integrated. this includes development of additional archivesspace export functionality, and perhaps use of the archivesspace oai provider to pipe records to alma in a more automated manner. this is relatively low priority technical debt, but it is a notable downside of microservices: loose coupling can mean additional overhead to make cross-component workflows run efficiently. we are also investigating integration of archivematica with archivesspace, with the hope that technical metadata generated through archivematica’s fits may be ingested into archivesspace and associated with the appropriate descriptive components. although we are unsure of archivematica’s capacity to accomplish this goal, we will continue to investigate integrations for our manual and siloed technical metadata workflows. conclusion we decided to share our experiences with implementing a microservices-based solution because it’s an architectural approach that more libraries are becoming interested in exploring. overall, our experiences have been highly positive: we gained significant technical expertise and broadened our horizons for giving back to the open-source community. at the same time, we implemented a new system that has garnered positive feedback from internal stakeholders in a largely budget-neutral fashion. on the other hand, this was possible because a large number of the services we intended to combine were already implemented in our library, and our staff had the interest, ability and bandwidth to take on new technical challenges. any library considering implementing a microservices architecture should make sure to include stock-taking and feasibility steps in their project planning to minimize nasty surprises during the implementation phase. will we continue to use a microservices architecture for our digital repository in the future? we aren’t certain. given that lack of maturity was a major roadblock in some of the repository solutions we considered, we may eventually find them maturing to suit our needs without requiring extensive local customization. when initially implementing bc digitized collections, we viewed it as a stopgap measure. we conceived of it to be as flexible and migration-friendly as possible, expecting to migrate one or more components forward eventually. it will be interesting to look back in five years and see whether we are still employing the same architecture, still using microservices with a different suite of interlinked services, or have migrated to a more monolithic repository system. we hope that we have provided some idea of the benefits and tradeoffs inherent in this microservices-based model. we strongly believe that this approach can be beneficial to libraries that already implement a wide variety of systems, provided that they have the resources and technical skills to support tighter integration between those systems. we hope to see more mid-sized institutions like our own implementing similar solutions and contributing to the open-source ecosystem in the near future. resources and github repositories primo/alma general documentation boston college implementation primo services: an application for building persistent links to primo records and searches primo explore search collections: a primo customization that adds a search button to each collection/sub-collection page primo explore galley item: a primo customization that adds an item count to multi-file digital objects alma integrations, a university of denver archivesspace plugin allowing communication between archivesspace and alma international image interoperability framework (iiif) iiif consortium website o’sullivan, a ruby api for working with iiif presentation manifests burns antiphoner, a boston college digital scholarship project utilizing iiif aspace-to-iiif: a boston college ruby gem converting archivesspace digital objects to iiif manifests mirador: a iiif-based image viewing platform loris: a iiif image server implementation other resources batch dao workflow, a boston college workflow for batch generation of digital archival objects in archivesspace audit control environment about the authors chris mayo is the digital production librarian at boston college. they are a project manager, metadata wrangler, and general-purpose script writer and cat herder. adam jazairi is the digital repository applications developer at boston college. he manages bc’s digital library infrastructure. paige walker is the digital collections & preservation librarian at boston college. she works with the preservation, privacy, and security of the libraries’ born-digital and digitized content. luke gaudreau is the discovery systems librarian at boston college. he focuses on library discovery, user experience, project management, and design thinking. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – the brooklyn health map: reflections on a health data dashboard for brooklyn, ny mission editorial committee process and structure code4lib issue 56, 2023-04-21 the brooklyn health map: reflections on a health data dashboard for brooklyn, ny recent years have put a spotlight on the importance of searchers of all kinds being able to quickly and easily find relevant, timely, and useful health information. this article provides a general overview of the process used when creating the brooklyn health map, an interactive brooklyn-based health data dashboard that visualizes community health information at the census tract, zip code, and neighborhood levels. built using html, css, bootstrap, and javascript, the brooklyn health map presents information in the form of interactive web maps, customizable graphs, and local level data summaries. this article also highlights the tools used to simplify the creation of various dynamic features of the dashboard. by sheena philogene introduction as a librarian at brooklyn college, i have found that patrons often want to do local research focused on brooklyn. however, it can be difficult for both students and faculty to find the right sources for the local level information they need quickly or easily. with that user need in mind, the idea of the brooklyn health map (bhm) was born. this article provides an overview of the tools used when creating the brooklyn health map, an interactive brooklyn-based health data dashboard that visualizes community health information at the census tract, zip code, and neighborhood level, in the form of maps, graphs, and data summaries.  background  recent years have put a spotlight on the importance of searchers of all kinds to quickly and easily find relevant, timely, and useful health information. although there are already many sources of high quality health data, reporting on the city, state, and national level, many of these sources can be hard to find and navigate. because health information is often scattered across various specialized organizations, users may be required to visit multiple sites to collect all the information they want to find. in many cases, data will be initially reported on the federal level, and if users are interested in finding local data, they will be required to narrow down to their area of interest using filters or other built in tools. this can often be a difficult or confusing process for people with less experience or facility navigating these web applications, because certain features will only be available after using a filter or other tool. recognizing these limitations, the brooklyn health map (bhm) compiles data from several sources of health and population information (e.g., nys cancer registry, us census, and centers for disease control and prevention [cdc]), limits the coverage area to one relatively small geographic area, and provides simple navigation for users to quickly and easily collect the information they need.  the bhm focuses on presenting data about the health characteristics of adults living in brooklyn, new york. the dashboard is intended to simplify access to relevant and comprehensive health and demographic information about brooklyn. this article seeks to provide an overview of the technical aspects involved in the development of the bhm. for information about the goals, development, and pedagogical applications, see philogene (2022).  as the bhm is a web based dashboard, the entirety of the page is built using html, css, bootstrap, and javascript. the webpage is divided into three areas (figure 1), which correspond to (1) an interactive web map, (2) zip code selector summary panel, and (3) interactive neighborhood level charts. figure 1. the brooklyn health map. the necessary data the bhm includes health and population information from three sources: the places: local data for better health project (2020) new york state cancer registry (2020) u.s. census bureau (2015; 2019).  in addition to the population information, the dashboard also uses shapefiles of the census tracts, zip codes, and other areas of brooklyn collected from the new york city open data portal.  the dashboard incorporates 20 types of health and population metrics, including languages spoken in the area and health characteristics such as high cholesterol and cancer screening rate on the census tract, zip code, and neighborhood level.  the data were initially compiled into .csv files (one for each geographic level) and converted to geojson files using arcgis. geojson is a geographic file type based on the javascript object notation (json) file type, which stores both geographic feature information (i.e., coordinates) and non-geographic data. using geojson files makes web mapping easier, as this file type helps to consolidate several types of data into one file, making file management and coding easier. the dashboard panel 1. web mapping with leaflet the first section of the dashboard is an interactive web map, built using the leaflet. leaflet.js is an open source javascript library used to create interactive, mobile friendly web maps. leaflet provides many short cut methods for adding dynamic functionality to web maps. the leaflet website provides full tutorials to get started with using the library. to prepare the page for a web map, a leaflet css and javascript file needs to be added to the html <head> element, and a <div> element needs to be added in the html where the map is intended to be placed.  the css file added to the <head>: <link rel="stylesheet" href="https://unpkg.com/leaflet@1.7.1/dist/leaflet.css" integrity="sha512-xodzbntc5n17xt2attpue1hxjvmsvlvw9ocquklscc5cxdbqcmblashomas6/ keqq/smzmz19scr4pszchsr7a=="crossorigin=""/> the leaflet javascript file (must be after the leaflet css file): <script src="https://unpkg.com/leaflet@1.7.1/dist/leaflet.js" integrity="sha512xqoymqmtk8lvdxxyg3nz448hoeqiglfqkjs1noqv44cwnurbc8pkaocxy20w0vlaxavueariobhixz5v3y nxwa=="crossorigin=""></script> the <div> element added to the <body>: <div id="map" class="row" style="margin-right: 0px;margin-left: 0px;"> ... <div id="mapid" style="width:100%; height:700px"></div> <script src="script.js"></script> </div> the javascript for the dashboard was written as a separate file (script.js), and also added into the same <div> element.  jquery was used to read the geojson files that store the coordinate data and to populate the maps. as mentioned above, the dashboard includes 20 different types of health and population information that fall into four categories: brooklyn demographics, unhealthy behaviors, health outcomes, and screening rates. separate geojson files were created based on each data source. the majority of the data included on the map is at the census tract level, however, there are also outlines of zip codes, neighborhood tabulation areas (ntas) and public use microdata areas (pumas) which users can overlay on any map layer.  here is a snippet using jquery to read the geojson file storing both language and population information:  $.getjson("brooklyn_language.geojson", function(data) { languagelayer.adddata(data); }); languagegroup.addto(mymap) the jquery .getjson() function is used to read the geojson file. from there, the data is added to the languagelayer object using .adddata(data). the data file stored information for the most commonly spoken language, the percentage of people who speak a language other than english, and the distribution of speakers of over 15 languages per census tract. this data was then used to produce a collection of 20 distinct map layers: the initial “brooklyn languages” map shown when landing on the page and an additional 19 maps showing the distribution of specific languages spoken across brooklyn. this added detail is accessed using a dropdown visible only on this map layer (figure 2). the collection of maps is added to a featuregroup called languagegroup. by adding all the layers to a single featuregroup, all added functionality (e.g., dynamic pop up boxes) are added to all the layers included in the group. finally, the .addto(mymap) method is used to populate the map with the stored data and geometry.  figure 2. the brooklyn languages layer showing default view (right) and layer selected using drop down menu (left). map category navigation as the map included 20 different types of health and population information, it was necessary to divide the content into thematic categories to help with navigation. so similar types of information can be easily identified and explored, the data were placed into four groupings (i.e., “brooklyn demographics,” “unhealthy behaviors,” “health outcomes,” and “screening rates”).  in order to create navigable categories, bootstrap buttons were added above the script.js file included in the <div> above: <div class="col-sm-8"> <div style="width: 100%; height:70%" > <div class="btn-group"> <button type="button" class="btn w3-white"><b>census tract level</b></button> <button type="button" id="demographics" class="btn w3-cyan">brooklyn demographics</button> <button type="button" id="unhealthy" class="btn w3-blue-grey">unhealthy behaviors</button> <button type="button" id="outcomes" class="btn w3-pale-red">health outcomes</ button> <button type="button" id="screening" class="btn w3-purple">screening rates</ button> </div> <div id="mapid" style="width:100%; height:700px"></div> <script src="script.js"></script> </div> the resulting button controls: figure 3. category buttons linked to map layers. the button ‘id’ for each category corresponds to a jquery .click() method, which simultaneously removes the previous layer, legend, and any other available controls, then replaces them with a new group of layers that correspond to the button selected. this script shows what the “unhealthy behaviors” button does:  $("#unhealthy").click(function() { mymap.eachlayer(function (layer) { mymap.removelayer(layer); // clear map on click }); baselayer.addto(mymap); // re-add basemap currentlayer = uninsuredlayer; uninsuredlayer.addto(mymap); mymap.removecontrol(currentlayercontrol); // remove old layer controls mymap.removecontrol(languagedrill); // remove language layer dropdown mymap.removecontrol(cancerdrill); // remove cancer layer dropdown currentlayercontrol = behaviors; // add new layer controls behaviors.addto(mymap); // add layer group to match category mymap.removecontrol(currentlegend); // remove old legend currentlegend = uninsuredlegend; // update current legend uninsuredlegend.addto(mymap); }); panel 2. exploring zip codes the second section of the dashboard, located on the right-hand panel, allows users to investigate data at the zip code level (figure 4). this zip code level panel actually has two purposes: (1) to introduce users to the dashboard and provide information about navigating the page and (2) to summarize health and language information available across the 20 map layers.  figure 4. zip code panel, showing welcome message (right) and zip code summary (left). the zip code information is stored on a separate geojson file, primarily because it uses a different geography than the mapped data. the same .getjson() method is used to access the data, and a similar <div> element is used to control what information populates the panel.  from the zip code selector panel, there are three possibilities of messages that can be returned:  introductory welcome text (only visible when the page loads or refreshes)  summary for selected zip code (changes to reflect selected zip code)  text to suggest other options for exploring  jquery is used to handle the call, when a zip code is selected from the drop down. the selected zip code is converted to a string, so it can be used as a selector in the function used in the .on(‘change’) method. the string is then used to check for the matching zip code ‘id’ in the geojson file and return the summary data for the correct zip code. if a user selects the text “select a new zip code” instead of an actual zip code, they will be given the third text option for other ways to explore.  handling changes in the zip code selector panel: // on selecting zip code on drop down $(document).ready(function(){ $('#zipcodes').on('change',function(){ zipselected = $("#zipcodes option:selected").text(); // match zip code from drop down and to data variable; return appropriate text for( var i = 0; i < 99; i++ ) { if( zipdata[i][0] == zipselected ) { document.getelementbyid("zipcontainer1").innerhtml = zipdata[i][1]; // get zip code highlight(zipselected) // highlight selected zip code break; } else { document.getelementbyid("zipcontainer1").innerhtml = '<br><br><h3 style="text-align:center">select a new zip code</h3><br><h3 style= "text align:center"><i>or</i></h3><br><h5 style="text-align:center">check out the charts below to see an overview of the health statistics on a neighborhood level.</h5><br><h4 style= "text-align:center"><i class="fa fa-arrow-down" aria-hidden="true"></i><h4><br><h5 style= "text-align:center"><i class="fa fa arrow-down" aria-hidden="true"></i></h5><br><br><h6 style= "text align:center"><i class="fa fa-arrow-down" aria-hidden="true"></i></h6>'; } } }); }); in addition to updating the panel to display a summary of zip code level data corresponding to the selected zip code, there is also the highlight(zipselected) function, which results in a yellow highlighted boundary of the selected zip code, which appears on the currently visible map layer (figure 5).  function controlling zip code boundaries: function highlight(zipselected) { mymap.removelayer(zipoutlines); zipoutlines = l.geojson(null, { pane: "pane660", style: stylezip, interactive: false }).addto(mymap); full view of the result of selecting a zip code: figure 5. view of dashboard with zip code boundary highlighted on map and summary of data for corresponding zip code. panel 3: looking at neighborhoods figure 6 shows the final section of the dashboard, which provides a view of the health specific data, summarized at  the neighborhood level. these summaries exclude the first category seen on the map (i.e., language and population statistics), as these could not be consolidated onto the neighborhood level, while continuing to be meaningful.  figure 6. neighborhood summary charts. the layout of the neighborhood summary section follows a similar bootstrap button selection method, as described above, when selecting each information category. the bar charts are created using the highcharts.js library. in addition to the javascript file, three additional modules were added to expand the end user options: the exporting.js and export-data.js modules, which provide options to export a snapshot of a chart or a table of the raw data as a .csv or .xls file; and the accessibility.js module, which makes the charts compatible with assistive technologies.  the highcharts javascript file must be added to the html <head> element: <script src="http://code.highcharts.com/highcharts.js"></script> <script src="https://code.highcharts.com/modules/exporting.js"></script> <script src="https://code.highcharts.com/modules/export-data.js"></script> <script src="https://code.highcharts.com/modules/accessibility.js"></script> in addition to simplifying the creation of interactive mobile friendly web charts, the other main benefit to using the highcharts library is that it allows you to create fully dynamic charts that can be customized both when writing the code and when being used by the end user. figure 7 shows an example of a chart with three health metrics excluded, and the export menu options displayed. this means each individual user can customize the data included in any of the tables to reflect their specific research interest, and export the relevant data as either a figure or raw data to use as they need.  figure 7. example of customized map, with export options visible. looking ahead: next steps the brooklyn health map has been a tool in development since it was created. in the time that it has been available to the public, it has been very well received and has been shared as both a research and educational tool. since sharing this tool with faculty, i have become a regular guest lecturer in a community health program planning course, where i have introduced students to the bhm as an example of a community needs-driven project. considering that the course is targeted towards students studying health and nutrition sciences, many have expressed excitement and interest in the possibility of developing similar tools that focus on their own communities using data they generate in their own practice-based research (e.g., community surveys).  currently, the immediate next steps for the dashboard involve updating it to reflect the new releases of data from the sources mentioned above. for the most part, new data is released yearly, so i am investigating how best to streamline updates.  with this in mind, one consideration is the use of jquery in the code. while javascript coding conventions are moving away from the use of jquery, my main purpose for using the library was to simplify the coding process and develop the brooklyn health map more quickly, since jquery is just a library of pre-built javascript code. additionally, jquery helps to ensure compatibility across different (primarily older) web browsers. however, as browsers become generally more cross-compatible and jquery less popular, i plan to restructure parts of the code without the use of libraries. in the longer term, i plan to expand the bhm into a new york city (nyc) health map, which includes separate dashboards for each of the five boroughs and a citywide summary. since much of the groundwork has already been done, with this work, the hope is that most of the labor involved will be in collecting and compiling the data for the rest of nyc. from there, the resulting geojson files should make the rest of the process simple for the county-level additions. however, since the citywide summary will be a completely new addition, it will likely present new design opportunities and potential obstacles in the next steps. about the author sheena philogene is the science librarian at brooklyn college (cuny) and the librarian at the brooklyn college cancer center (bccc-cure). bibliography philogene s. 2022. the brooklyn health map: reflections on a health dashboard visualizing connections between social factors and health outcomes in brooklyn, ny. journal of map & geography libraries [internet]. [cited 2023 mar 6];18(1-2):22-40. available from: https://doi.org/10.1080/15420353.2022.2155752 subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – user experience is a social justice issue mission editorial committee process and structure code4lib issue 28, 2015-04-15 user experience is a social justice issue when we’re building services for people, we often have a lot more practice seeing from the computer’s point of view than seeing from another person’s point of view. the author asks the library technology community to consider several case studies in this problem, including their root causes, and the negative impact of this problem on achieving our mission as library technologists. the author then recommends specific actions that we, as individual contributors and organizations, can take to increase our empathy and improve the user experience we provide to patrons. by sumana harihareswara editors’ note: the content of this article was originally presented as a keynote address at the code4lib 2014 annual conference under the title, “ux is a social justice issue.” the editors have adapted the speech for print, and the author has provided a new introduction and conclusion reflecting on the state of user experience design one year later. introduction the code4lib community asked me to give one of the keynote speeches at code4lib 2014 while i was the community manager for the wikimedia foundation’s open source projects and an advisor to the ada initiative. my wikimedia expertise intersects with the interests of the code4lib community, as wikimedia — like many libraries and other cultural institutions — aims to empower patrons with access to information. before i worked in open source, i worked in customer service. i saw first-hand how design flaws (in architecture, signage, and websites) could frustrate and drive away customers and make more work for me. every time i participated in an open source project — altlaw, gnome, mediawiki, and more — i’ve brought that experience with me. i found it particularly striking that small changes on wikipedia could cause large changes in user behavior, as i discuss in this essay, which is adapted from my keynote speech. this issue goes beyond software, as i explain with the healthcare and banking examples. the spark that caused me to write the speech was reading professor lisa j. servon’s piece in the atlantic about the usability of storefront check cashing services; i saw a pattern where poor user experience repels people from crucial and empowering services, and decided, in a flash of anger and inspiration, to write “user experience is a human rights issue.” i am following in the steps of jeremy prevost and bess sadler and mark matienzo, among others, in their previous code4lib presentations and writings about design and emotion [1]. in adapting my speech for this article, i toned down the title to “user experience is a social justice issue,” but the narrative is largely the same. i hope you find it thought-provoking. the last mile problem the largest hurdles we as technologists face are choosing to make the right things in the first place and choosing to make them usable. in the 1990’s, telecommunications companies laid down a lot of fiber to connect big hubs to one another, but often it took years to connect those hubs to the actual houses and schools and shops and offices, because it was expensive, or because companies were not creative enough to do it well. this is called the “last mile problem,” and i think usability has a similar problem. we have to be creative and disciplined enough to actually provide services in a way that people can use them. when we’re building services for people, we often have a lot more practice seeing things from the computer’s point of view or from the data’s point of view than from another person’s point of view. in tech, we understand how to build arteries better than we understand how to build capillaries. personally, i think capillaries are more interesting than arteries. maybe it’s just personal temperament, but i like all the little surprising details of how people end up experiencing the ripple effects of big new systems, and how users actually interact with the user interface of a service, especially ones that we don’t really think of as having a user interface. like taxes, or healthcare, or hotels. all these big systems end in little capillaries, where people exchange information or get healed or get whatever they need. and when those capillaries aren’t working correctly, then those people just don’t get what they need. the hubs are connected to each other, but people aren’t connected to the hubs. over and over, in lots of different fields, we see that bad usability makes a huge difference. when choosing between two services, people will make very different choices, depending on which service actually seems designed around the user’s needs. in march 2014, librarian jessamyn west was in a conversation on metafilter about coffee machines, in particular, keurig pod machines. these machines will only make coffee from pods manufactured by the company that makes the machines. one person said, “this convenience thing is a bit overstated.” jessamyn’s response was: not that i’m not more in your camp taste-wise than the keurig camp but i see this as a great exercise for people generally in the “other people have different priorities in life and aren’t just bad versions of you” direction. not you [personally], just the general you. i know a lot of people for whom keurigs solve a problem and they don’t mind the downsides. i respect that. i also know a lot of people who wouldn’t be caught dead with one and i respect that as well. i know it’s tough to get your head around but the goal state for people with keurigs is generally not to just have the best cup of coffee. it’s to have a coffee solution that is easy to clean up after, or that turns itself off, or that has pre-measured sizes, or that has all the brands that people like, or that makes cocoa, or that offers holiday flavors, or that you can buy at the department store, or that is easy to clean, or that can be modified to accept change, or that you can put in a place with no running water, or that descales itself, or can be put in a place without a kitchen, or that has funky modern lines, or that can make ten cups of passable coffee in five flavors (caf and decaf) in ten minutes. these things do not solve problems for me, but they solve a problem for an awful lot of people which is why these things are so popular (west 2014). so, yes, usability makes a difference in what coffee people drink. and if you care about health, and education, and the working class, then this is actually scary, because differences in user experiences are driving people to make bad choices. i want to give you a few examples. banking: payday lending and check-cashing services have sprouted up in many working-class communities. these services are often predatory or incredibly expensive, compared to traditional banking services. they do not provide a way to save and earn interest instead of paying interest. but lisa j. servon, an urban policy professor at the new school, worked in a ritecheck for four months and found that one reason people chose ritecheck over a traditional bank was the user experience. it was the hospitality people were getting. she wrote in the atlantic: the glue at ritecheck is the customer/teller relationship. i interviewed 50 ritecheck customers after my stint as a teller and, when i asked them why they brought their business to ritecheck instead of the major well-known bank three blocks away, they often told me stories about the things the ritecheck tellers did for them… at ritecheck, the tellers treated the customers as individuals and went the extra mile to assist them, perhaps in the same way that a neighborhood grocer might allow a trusted customer to run a monthly tab. on busy days, tellers regularly skipped lunch and coffee breaks in order to keep the wait times down. ana paula, our manager, often joined us at the window. the customer always came first and knew it (servon 2013). servon believes that to attract these depositors, banks would need a better product — fee and service structures that work for them and are designed around user experience. ebook lending: the new york public library (nypl) has been lending out e-books for years. many people in new york city have smartphones or dedicated e-book reading devices. i think we could expect that people who are already buying e-books would be interested in borrowing them for free. but e-book borrowing rates at nypl are abysmal. for context, across the united states, for every one hundred print books sold, somewhere from 30 to 80 e-books are sold. this calculation is from some rough numbers that some friends and i put together. it might be off, but it’s not much beyond, let’s say, 3-to-1. given that new york public library lent out about 250,000 print books every week last year, you might hope that nypl would be lending out, let’s say, maybe 150,000 e-books a week, or lower than that, maybe 100,000. let’s say it’s a 5-to-1 ratio – 50,000 e-books a week. but instead, it was actually much lower than that. in 2013, nypl lent out 19,000 e-books a week. so instead of a 3-to-2, 3-to-1 ratio, it’s more like a 13-to-1 ratio. new york public library has this department, nypl labs, which is researching this problem. nypl labs has broken down the e-book borrowing experience and found that it currently takes 18 separate steps for an nypl user to borrow an e-book. i saw this diagram that was called “19 steps of hell.” they have been working on a project to take that down to three steps. and i would predict that this would raise e-book borrowing rates beyond this 13-to-1 print-to-e-book ratio. wikipedia: i have first-hand experience of this from my work at the wikimedia foundation, the organization behind wikipedia. wikimedia’s number one concern is gaining and retaining wikipedia editors, as well as contributors to wikidata, wikivoyage, and so on. if we want a world in which every single human being can freely share in the sum of all human knowledge, then we need diverse perspectives editing and providing that knowledge. we’ve found that sometimes little things in usability make a really big difference. for a subsection of a wikipedia article, the “edit” link used to be all the way at the right side of the screen. it wasn’t obvious what that link was for, and some users thought it was the link to edit the previous section. so we moved that link to be right next to the subsection’s header. the click-through rate more than doubled. it went up by 117 percent, and that higher volume of clicks led to 8.6 percent more edits as well (wikimedia contributors 2012). we see the bounce-off effect where people hit the edit button, see a form full of wikitext, the markup language that evolved with mediawiki, and just hit the back button. generally when newcomers see that, they worry that something’s broken, because when else would you see that kind of gibberish on a website? to improve this user experience, we have invested in making a visual editor appropriately called the visualeditor. you can try it right now on english wikipedia, if you log in and turn on beta features in your preferences. our hypothesis, which we’re working to prove, is that if we make the user experience better, more people, and more different kinds of people, will edit. one last note about wikipedia and usability: you don’t have to register to read wikipedia. it’s just there, no paywall, no registration wall. and there are a lot of really great other open educational resources out there, oers, that have content that readers would really find useful, maybe just as useful as wikipedia – except they’re behind a registration wall. that just massively reduces how many people are going to see it. healthcare: i think healthcare in the united states is kind of fractally broken: it’s broken on the macro and the micro, you just zoom in and you keep finding new kinds of breakage. if you look at the user experience for normal use cases: general practitioner acquisition, appointment booking and service delivery, usability is pretty poor. a lot of offices aren’t open at night or on weekends, you have to call them during business hours to make an appointment, you have to book a whole appointment just to get a flu shot, and there’s a long wait once you get there. so in recent years we’ve seen some new startups that are usability hacks. zocdoc, a website that lets you see doctors in your area who have open appointment slots this week, lets you filter out the ones who don’t take your insurance and book the appointment online. minuteclinics and other quick clinics at drugstores and in big box stores are also walk-in, low-cost ways to get quick diagnoses or shots. this convenience, however, does come at a cost to the user. patients on zocdoc are often just going to book with the next available doctor. people who use minuteclinics are skipping a longer doctor visit, so these patients might not get long-term preventive care or form long-term relationships with their doctors (harihareswara 2007). in the library world, researchgate and academia.edu act similarly to zocdoc. they’re hacks that patrons use to self-serve, which is great except that they’re routing around the kind of deep and long-term research help your institutions could give them. encryption: let’s talk about privacy tools. pretty good privacy (pgp) is basically the standard, the common standard for email encryption. but it’s incredibly hard to wrap your head around. encryption tech in general has terrible usability. it has terrible usability, terrible customer support, terrible localisation to different languages and locales, and thus, abysmal adoption rates. as a result, journalists and dissidents and activists generally don’t use it, even when the stakes are really high. they’re instead using twitter and facebook and email – unencrypted – because they are much easier to use. the open internet tools project’s james vasile said at the keynote at open source bridge 2013 that the privacy tech community would be better off spending the next year of our time, spending all of that time, on user experience design, localisation, and end user customer support, rather than writing any new tools or features (vasile 2013). why the bottleneck? these examples – banking, lending, wikipedia, healthcare, encryption – show you how bad usability can really change what choices people make. we in the open source community, including code4lib, put a lot of energy into lowering visible barriers to entry, like licensing and cost, that stop people from getting information. but bad usability is a barrier to entry, too–less visible but just as real. free access to education, free expression of political opinions, and privacy from unwarranted surveillance are human rights in the un declaration of human rights [8]. even if you don’t want to go that far, they are social justice goals that i think we can all agree on here. our community shares a vision of less waste and more empowerment and getting knowledge into people’s hands. but transferring these benefits out of our libraries and into the hands of our users, the new last mile problem, requires that the users actually be able to use our software. in getting rid of user experience obstacles, we are working to achieve social justice goals and implement human rights. it’s so clear why we ought to make our products usable , but why do we have this bottleneck? why do we have this barrier? it’s not just that the problem is hard. we’ve had other hard problems in the tech world – from going from mainframe to web, distributed computing, localisation — that we have done better on than usability: so why? this isn’t just about coffee. let’s look at what it takes to do user experience work. you have to look at your service from the point of view of someone who knows a lot less than you and see where they’re coming from. you have to imagine the reasons why they want what they want. seeing that causation, seeing the connection between what someone’s doing now and all the causation that went before it, that’s empathy. it’s a little like reverse engineering. you’re trying to break the drm (digital rights management) that’s stopping them from getting what they need. we need to exercise a disciplined empathy. it’s an empathy that includes qualitative thinking, like interviews and watching people use stuff to see where the snags are, and quantitative thinking, like a/b testing and heat maps. the tech industry is not very good at empathy. i’m speaking from my own experience here–i know library tech is its own field–but in my experience, we just drop the ball on empathy and hospitality a lot. one reason is that our industry systematically undervalues the jobs and roles that require empathy and has deeply gendered associations with hospitality and empathy is that they’re not seen as masculine traits. this isn’t simply a women-versus-men issue; it affects everyone. the tech industry values masculinity over femininity, meaning traits like hospitality are devalued. those who perform or reinforce masculinity — whether they’re male or female, are privileged. everyone, women and men, become trapped in this cycle of usually unknowingly reinforcing hospitality, empathy, and so on as stuff that can lead to demotion on the respect ladder and the pay scale ladder. naturally, all this stuff is then smushed out of our software, because it’s just not incentivized, it’s actually penalized, and when the group making the software isn’t very diverse, the cycle just repeats, and becomes even worse. this is one reason that diversity in a group, especially a group making software, is useful. it includes people with different perspectives and is more likely to include more people with the ability to see issues from multiple perspectives–including the users’ perspectives. in general, marginalized people develop more empathy than the dominant group because we have to. we have to be able to see from other people’s points of view, including the dominant point of view, as a matter of survival, and we need to be able to see from many different users’ points of view, even when it’s uncomfortable or shows us that we have failed [2]. what you can do a disciplined empathy means that we need to treat customer support, those front-line desk and phone tasks, as a first-class responsibility and a source of important data. these are your bug reports. this is how you know if you aren’t being as hospitable as you want to be. i’ve heard lore about a library where they log, in a shared document, every time they have to tell a patron “no,” and then they try to fix that. disciplined empathy means coders listening to designers, designers and coders learning to speak each other’s languages, and designers learning how to integrate their work into the development workflow. and this is happening. in her blog post, “code talks and designers don’t speak the language”, crystal beasley says that in general, coders right now in the open-source world don’t know what to expect from user experience designers, and developers don’t have good judgment procedures to know whether a proposed design is correct and that “representing the work of a designer requires a shift in culture,” with which i agree. beasley also says, “the solutions to all these problems lie in communication and building a trusted relationship. it’s a higher barrier for designers that takes time to overcome. i’ve found all of my team to be receptive when i’ve taken the time to explain the principles that guide my decisions [10].” this is a good point. i think in any negotiation, it’s good to help people see the principles you’re reasoning from, and the experiences you’ve had, that led you to where you are now. if you don’t think empathy is one of your strong suits, you can change that. we all came into this community with some skills and without others. you can learn and exercise your empathy skills as well. if you have a sequential learning style where you prefer a structured approach, you could take a course in conflict resolution. if you prefer self-study, you could read novels and blogs written by people with lives very much unlike yours. at your own institution, observe real users using your library and your library’s digital services, including patrons and colleagues. let’s go as bess sadler once said to me, “at this intersection of technology and librarianship, we need to not only bring technology skills and values to library services; we need to bring library skills and values to technology services.” and i’d say that includes hospitality and access. here are some of the resources that you can access right now. university of michigan’s library information technology group has a user experience department [3], and suzanne chapman, interface and user testing specialist and head of the university of michigan library’s user experience department, also has a really interesting blog [4]. university of illinois’s library has a usability testing lab with two workstations and a room set up to conduct usability studies [5]. matthew reidsma at grand valley state university libraries has this awesome “work notes” blog so you can see, for instance, how the gvsu web team responded to data and anecdotes to make incremental improvements to their site. i also suggest you read his pieces “the library with a thousand databases” [6] and “how we do usability testing” [7]. usertesting.com is a quick and easy-to-use site that lets you commission a test. testers perform tasks and record and narrate on video what they did. it’s cheap; someone i know commissioned a test for $89 and got results in less than an hour. if you just want to start with usability testing and maybe just find some glitches in what you just rolled out, this is an option. influx, a consultancy started by aaron schmidt, amanda etches, and nate hill, focuses on library user experience [8]. within lita, some folks have proposed a user experience interest group [9], that’s also something to look into. libraries are doing this work. sometimes we call it ui, sometimes we call it human-computer interaction, sometimes we call it user-centered design or interaction design, and it intersects with product management. several people on the code4lib email list in october talked about what a huge difference a dedicated ux team or person makes. for example, tom cramer said, “we have been lucky to have a full time interaction designer within our library it group for about 6 years. it makes a world of difference in the quality of our products (cramer 2013).” better user experience is the best force multiplier we have at our command, so it is vital that we make it a first-class priority, throughout the development process. and with disciplined empathy we can do that. here at the intersection of libraries and tech, we can figure out how to scale hospitality, fix the new last mile problem, and actually achieve the social justice goals that so many of us got into this for. conclusion: post-speech reflections after my keynote speech, attendees responded with approbation. one attendee tweeted, “the last time i heard the word “hospitality” used this often was at the catholic worker. #c4l14 #justsaying” [10]. another said, “i want to take yesterday’s keynote, bess’ last 2 years of talks, matienzo’s lightning talk from 2013: burn to a dvd, send as a mass mail” [11]. an archivist used the talk as a jumping-off point to ask, “what kind of user experience are we providing to our researchers from start to finish?” [12]. outside the immediate code4lib community, a specialist in infotech for international development also found my talk applicable to her work: “…if we want to use technology–any kind of technology, from radio to broadband–to give people more options and choices in their lives, we have to get imagining. we don’t really have a choice” [13]. as i look back a year later, i see related discussions (including andromeda yelton’s keynote from code4lib 2015) continuing in our communities, and i am particularly happy to see research and development continuing along the lines i described. for instance, library simplified [14], from nypl labs, is going to launch an e-book lending product for new yorkers in the middle of 2015, and will be working to help other libraries use the software. work on usable security by adrienne porter felt [15] and others has improved the ux of privacy for millions of web users, and points to opinionated design principles that libraries can use to empower patrons while keeping them safe. perhaps the best response i received was during the code4lib conference last year, in raleigh. i was looking for an acquaintance in the hotel lobby, and poked my head into the bar. i saw a stranger wearing the conference badge, so i greeted him, and he said that he’d liked my speech. i asked whether there was anything in particular he’d liked. he quietly told me that he’d been working in library technology a long time, and had just been through a long stretch of feeling ground down. but my speech had reminded him of why he’d gotten into this particular business in the first place – why he was here. that touched me. and i hope it continues to help him, and anyone else who needs it. acknowledgements i want to thank all the people who helped me work out the ideas in my original speech: coral sheldon-hess, mel chua, andromeda yelton, bess sadler, emma molls, leonard richardson, jared zimmerman, and sky croeser. and i want to thank all of you for the work you’re going to do, hacking yourselves and your institutions to serve users better. notes [1] see bess sadler’s “creating a commons” (http://www.ibiblio.org/bess/?p=302) and mark matienzo’s “emotion, archives, interactive fiction, and linked data” (http://matienzo.org/blog/2013/emotion-archives-interactive-fiction-linked-data/). [2] for further reading see mel chua’s notes (http://blog.melchua.com/2013/08/20/engineering-education-discourses-on-representation-why-problematization-matters-beddoes-2011/)on kacey beddoes’ “engineering education discourses on representation: why problematization matters.” [3] http://www.lib.umich.edu/library-information-technology/user-experience-department [4] http://userslib.com [5] http://www.library.illinois.edu/sc/services/usability_testing/usability_testing.html [6] http://matthew.reidsrow.com/articles/58 [7] http://matthew.reidsrow.com/articles/13 [8] http://weareinflux.com/ [9] http://www.ala.org/lita/about/igs/user/lit-igue [10] https://twitter.com/helrond/status/448455510971252736 [11] https://twitter.com/chrpr/status/448879826620145664 [12] http://rockarch.org/programs/digital/bitsandbytes/?p=1036 [13] http://thehillarylp.com/blog/2014/4/14/me-and-you-and-everyone-we-know [14] http://www.librarysimplified.org/ [15] for further reading, see http://adrienneporterfelt.com/ references beasley, c. 2012, october 12. code talks and designers don’t speak the language. retrieved from http://skinnywhitegirl.com/blog/code-talks-and-designers-dont-speak-the-language/930/ cramer, t. 2013, october 30. usability person? [online forum comment]. retrieved from https://listserv.nd.edu/cgi-bin/wa?a2=ind1310&l=code4lib&f=&s=&p=243880 harihareswara, s. 2007, january 16. bad relationship retrieved from http://www.harihareswara.net/sumana/2007/01/16/0 servon, l. 2013, september 11. the real reason the poor go without bank accounts. citylab. retrieved from http://www.theatlanticcities.com/jobs-and-economy/2013/09/why-poor-choose-go-without-bank-accounts/6783/ united nations. the universal declaration of human rights. retrived from http://www.un.org/en/documents/udhr/ vasile, j. 2013, june 18. keynote address presented at the 2013 open source bridge conference, portland, or. retrived from https://www.youtube.com/watch?v=xihsrpd_u-0 west, j. 2014, march 10. your new coffee overlord [online forum comment]. retrieved from http://www.metafilter.com/137379/your-new-coffee-overlord#5456577 wikimedia contributors. 2012, may 16. research:section edit modification. retrieved from https://meta.wikimedia.org/wiki/research:section_edit_modification#overview_analysis_from_previous_testhttps://meta.wikimedia.org/wiki/change_to_section_edit_links about the author sumana harihareswara is a programmer and open source community leader living in new york city. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – challenges in sustainable open source: a case study mission editorial committee process and structure code4lib issue 9, 2010-03-22 challenges in sustainable open source: a case study the archivists’ toolkit is a successful open source software package for archivists, originally developed with grant funding. the author, who formerly worked on the project at a participating institution, examines some of the challenges in making an open source project self-sustaining past grant funding. a consulting group hired by the project recommended that — like many successful open source projects — they rely on a collaborative volunteer community of users and developers. however, the project has had limited success fostering such a community. the author offers specific recommendations for the project going forward to gain market share and develop a collaborative user and development community, with more open governance. by sibyl schaefer introduction from july 2007 to september 2009 i served as an archives analyst for the archivists’ toolkit (at).[1] the project was co-sponsored by the university of california, san diego, and new york university, where i was based. the archivists’ toolkit has been an extremely successful and groundbreaking open source software program for archives. the program has revolutionized archival description and data management by enabling easy authority control processes, linking accessions with descriptions, and allowing multiple standard data outputs from the same data set. the at project was funded by the andrew w. mellon foundation but was not self-sustaining at the end of the grant phase (september 2009), six years after the inception of the project. instead of continuing to fund the product, the mellon foundation has suggested it merge with archon, another open source archival data management program. while this move will unite two user communities in support of one product, it remains unclear whether the product is tenable outside of grant funding. during my experience with the at project it became evident that: 1) the product needed more users to become sustainable and 2) governance of the project needed to be more open, delegating tasks to users whenever possible in order to minimize overhead costs and essentially becoming a true collaborative and community-based open source venture. the merger of the at with archon provides an excellent opportunity for the new product team to increase market share and establish a system to harness user contributions. by doing so, the end product will be in a much stronger and more sustainable position. background on the archivists’ toolkit the archivists’ toolkit claims to be the “first open source archival data management system”[2] although archon was also released around the same time.[3] both projects originated out of the need for a tool to support the management and automation of archival data in a manner reflecting archival practices and to output this data in professional standards, such as encoded archival description (ead). this need was realized first in 2002, when the digital library federation and california digital library co-sponsored a couple of meetings to discuss what was then called the archivists’ workbench. two years later, in june 2004, new york university and the university of california at san diego, working in conjunction with the five colleges, were awarded a grant from the andrew w. mellon foundation to develop this tool. the project commenced and by december 2006 the first version of the software was released with the “archivists’ toolkit” moniker. the project team at that point was mainly divided by task: project management was handled at ucsd and development at nyu. analysts wrote specifications from ucsd, nyu, and the five colleges area. the archivists’ toolkit was groundbreaking for its successful integration of previously separated data stores. in the pre-at world, an archival repository may have kept accessions information in an access database, an excel spreadsheet or maybe even in a hand-written log. finding aids, the predominant access tool for archival materials were kept in paper inventories, word documents, html files, or discrete ead files (if the archives had been able to adopt the technology). authority control may have been accomplished through the creation of collection-level marc records uploaded into the ils of the repository’s parent organization or may not have existed at all. the at changed these idiosyncratic practices. accessions information, descriptive data, donor information, location information, and authority records are all kept within one searchable space. reports and standardized outputs can also be easily generated. the product was similar to an integrated library system (ils) but specifically designed for archival materials. although ilss have existed since the rise of computing, the at was the first software system to tackle the unique needs of archival management and description. the first release of at version 1.0 prompted a lot of investigatory testing, but few institutions adopted it on a full-scale production basis. this was to be expected. the program still had some kinks, and most repositories remained wary of a tool that was not fully tested. in 2007, at version 1.1, a much more mature and stable release, was offered. it allowed for more flexibility in importing legacy data: multiple ead files could be imported at once, an xml import schema was developed for accessions data, and user-defined fields were allowed within the accessions record. this release prompted many institutions that had previously been working on development instances to move fully into production, although full-scale adoption of the program was still low. after the release of 1.1, the at project team hired ithaka, a non-profit consulting group, to develop a sustainable business model. ithaka scoped out the archival landscape, divided it into market segments, and conducted interviews with at users within those segments. from early on the outlook was glum. first, the at needed to increase its user base. the small size of the archival market meant that the at needed to become a dominating force within that market. second, the at needed to either find a parent organization that would be willing to incubate the project, sharing resources until the project became financially stable– or the project would have to develop its own means of generating income. in the case of the latter, serious budget cuts would need to be made. in order to make up for these cuts in operating expenses, ithaka suggested the project do what most open source projects do: rely on their users. working towards sustainability: increasing the user base the at offered archivists new ways to capture, control, and manipulate archival data, and thus can be considered discontinuous innovation; users are required to modify their behavior in order to take advantage of its benefits.[4] discontinuous innovations are in danger of falling into the “chasm,” or the space that lies between the early adopters and early majority in george moore’s revised technology adoption life cycle model.[5] moore warns that early adopters are distinctly different from the early majority. unlike early adopters, who are eager to implement the latest technology, the early majority tend to be pragmatists when making their decisions. they won’t accept a buggy, unpredictable product, and they want the benefits from the product to be demonstrable, especially in increasing productivity. as pragmatists, the early majority are also looking for the whole product — not just the software, but support, training, and other services and technologies that surround the program to integrate it fully into their existing framework.[6] to cross the chasm effectively, the at product needed to start gaining support from the early majority, particularly by appealing to its pragmatist qualities. taking heed of ithaka’s advice to focus on increasing the user base, at senior administrators changed my analyst position into a user services support position. i led organized regional user group meetings, held mainly at professional conferences. the meetings generally turned out to be a mix of formal communication from the team and informal communication between users. these meetings were key in informing potential users about the software and its capabilities. the presence of the software at conferences showed we were a viable player and provided a venue for current users to share their successful implementations with potential users. in addition, the at team focused on providing the whole product. extensive documentation was developed and maintained. user support, always a strong point for the team, became a priority. we also began working in conjunction with the society of american archivists (saa) to provide training on how to use the program in adherence to national and international standards for archival description. this last move was especially ingenious — working with saa gave the program an informal endorsement from the nation’s leading professional archival organization.[7] combining the training with a focus on standards was also key; it sent the message that an additional benefit of adopting the program was making standards compliance easier to implement. the project was fairly successful in using marketing methods that appealed to the pragmatist nature of the early majority, but the team neglected to review aspects of the application that could be modified in order to broaden its appeal. one major oversight was a lack of focus on increasing productivity in every applicable area in the application. although the at saves a considerable amount of time by providing automatic ead markup, the main data entry mechanism in the program is unwieldy, especially when it comes to creating the hierarchy that is inherent in most archival arrangement and description. archivists accustomed to entering information in excel or access are often frustrated by the additional keystrokes, mouse clicks, and navigation required by the at. other editing functions typically found in desktop applications are missing in the at, notably a find and replace feature and an undo mechanism. because the at is database-driven, these features are more difficult to implement than in other types of programs, such as word processing software. yet this difference is not obvious to the typical at user who generally doesn’t distinguish between one desktop application and another.[8] since the early majority seeks gains in productivity, the lack of these basic and very noticeable features make it easy for them to overlook the other time-saving mechanisms provided by the at. the at also missed another essential part of the whole product — ead search and delivery mechanisms. the inability to serve up ead from the at is probably the predominant reason some institutions chose archon, which does offer immediate web-publishing functionality. the at does provide basic html and pdf transforms, but repositories usually find these outputs insufficient because they don’t allow for complex searching, and because customization requires additional technical skills. most repositories don’t have the access to technical resources that the at’s parent universities do; creating a homegrown ead publishing system is out of their financial and technical reach.[9] the at did not provide the whole product and missed out on a considerable amount of possible market share.[10] i believe the lack of web delivery, as well as the web access archon offers, figured predominantly in the decision to merge the two applications. lastly, the at also missed the opportunity to develop relationships with third-party partners. such relationships can help a software project deliver aspects of the whole product that they can’t deliver themselves. they also serve to make the product appear stable and well supported to potential adopters. these qualities are especially important for an open source product looking to cross the chasm and which has to face questions not only about its sustainability, but also about the effectiveness of open source software. these partnerships do not just magically appear; they need to be actively sought out.[11] the at, for instance, could have partnered with a company willing to host the at database backend and/or provide an ead publishing system. such partnerships could have provided more options for users who otherwise were not able to adopt the at. reducing overhead: providing a framework for user contributions in the spring of 2008, ithaka informed the at project team that it was highly unlikely the at software could expect volunteer development contributions. the archival community is small, and the number of archivists with enough programming prowess to contribute to the at code base is even smaller. since we couldn’t expect code contributions from our users, the project was always going have to cover development costs. ithaka advised the team to find a parent organization that could take the project under its wing. ideally this would be an organization whose ideology mirrored that of the at project and who would be willing to incubate it by sharing resources until the at became financially stable. should the project not find a suitable parent, ithaka suggested other business models which could generate income to cover some development costs. they strongly recommended that costs not associated with actual programming – testing, documentation, specification etc. – could all be achieved in the traditional open source manner: using volunteer contributions from the user community. relying on contributions for these tasks would decrease overhead enough to make the project’s sustainability viable. by the spring of 2009, talks with potential parent organizations had failed to reach fruition. around that same time a few prominent archives decided they wanted to extend and improve on the at base by contracting a programmer (a former lead developer on the at project) to do so. during this time, six months before the end of the grant period and without any enacted sustainability plan, the at should have done as much as possible to open up governance of the project to users by delegating tasks and responsibilities whenever feasible. yet this did not happen, even with the promise of outside development contributions. by all appearances, the project had started off with intentions of encouraging a “bazaar” atmosphere, one in which development is marked by contributions and involvement of numerous people outside the original development team. in order to ensure bugs and other problems were detected, the program was beta tested by 20 different institutions prior to the first release. the original lead developer also embraced the mantra “release early, release often.” although there were some initial attempts to open up development of the program, it was largely done in a more closed, “cathedral” fashion.[12] for example, although jar files are made available on the at website for each release, a way to access the latest code, or guidelines on how to be an official “committer” have never been officially provided. testing eventually became a task completed solely by the project team, due to time considerations and the difficulty in setting up appropriate test conditions.[13] in his work of managing open source software projects, karl fogel provides guidelines on how to incorporate and encourage volunteers to contribute back to an oss project. one way of doing so, he states, is to “treat every user as a potential volunteer.”[14] the at project team was congenial and helpful to its users, but it did not recruit users to the extent that it could have. instead of operating from a level on par with the users, the adopted tone was more one of an expert. this tone was mirrored by a lack of delegation of tasks to the user community. part of this had to do with the fact that team members were all in salaried positions in which it was their responsibility to work with the software. when it is your business to be an expert on something, it is hard not to assume that stance. it is tempting take on tasks yourself rather than teaching volunteers how to do them, and it can be difficult to enforce tight deadlines when managing volunteers who have full-time positions outside of their volunteer contributions. however, as fogel states, “the goal is to make every user realize that there is no innate difference between herself and the people who work on the project.”[15] this realization, he believes, is what prompts users to take the initiative to do such tasks in the first place; everyone can become an expert if they are willing to put in the time. in the case of the at project, we were experts on the program, and could be relied on to provide documentation, run beta testing, answer support questions, etc. there was little incentive to contribute when everything seemed already under control.[16] because we were experts the team could complete tasks in less time than it would take to train volunteer collaborators . the first sketches of a framework in which user contributions could be accepted gradually did emerge. the proliferation of new development initiatives forced the at project to finally start coming to terms with the open source nature of the software. the lead programmer created a plugin framework to allow for new functionality to be added without changing the core code. this isolated code reduced testing time and provided basic means for code contribution without forking the code. in a parallel development, active leaders in the at user group formed the archivists’ toolkit roundtable, an official saa interest group, which first met and elected new officials in august of 2009. however, by this time the merger with archon had been announced, and the role of the roundtable remains undetermined. likewise, it is uncertain whether the plugins currently in development will be compatible with the merged at/archon system. conclusion the archivists’ toolkit/archon merger team has an important task in front of it. both products were revolutionary in changing the way archivists manage their collections data and greatly eased the production of access instruments. but even though the two products will no longer be competing, the merged product will still need to gain market share to be sustainable. in order to do so, the merger team should: enable increased productivity by making data entry tasks as easy as possible, provide more visualization for users as they work in the hierarchical description section, include a find and replace function, and add shortcut key functionality. prioritize usability from the start of application development. review plugins that are in development as a means of assessing what features are truly important to users. provide the whole product package. the addition of web services (publishing and editing) will help, but there are other areas that could use revision, notably the reports. work with other people and companies to provide whole product services the core at/archon team can’t provide, such as training, set-up, data migration, plugins, etc. the merger team should also address sustainability issues early in the project. delegating tasks to dedicated users who are willing to volunteer their time can help cut costs substantially. in order to create a more open and participatory environment, the merger team should: provide access to the most updated code. provide guidelines on how code contributions are handled, and who gets “committer” status. set up an infrastructure that eases the burden of testing on volunteers. delegate everything that can be possibly delegated to users outside of the development team. create experts in documentation, testing, specification, etc., in the user community, rather than on the project team. this structure would ideally be organized before the product goes live, so potential new users would not be deterred by the lack of the whole product. use the roundtable as a form for governance/decision making and task delegation, and keep the number of salaried staff that are not programmers to the absolute minimum it’s easy to reflect back and discuss what should have been done, but it’s much more difficult to implement systemic changes in the middle of a project with finite resources and tight deadlines. the end product of the at/archon merge will likely confront the same problem the at did: improbable sustainability without a greater number of users, and a coordinated volunteer support system. the new at/archon merger team has the unique opportunity to focus on making users part of the infrastructure and fabric of the new program, paving the way for a truly sustainable open source archival data management system. notes [1] during the two years i spent with the project i worked on various different aspects of it: namely marketing, user support, documentation, specification, testing and training. grant funding for my position expired in july of 2009. nyu generously offered to keep me on for at least one additional year, but several months into that year i left for another position. [2] archivists’ toolkit, “introduction to the archivists’ toolkit,” http://archiviststoolkit.org. [3] university library, “library archivists receive award,” university of illinois at urbanachampaign, http://www.library.illinois.edu/news/archon_award.html. [4] geoffrey a. moore, crossing the chasm: marketing and selling disruptive products to mainstream customers (new york: harpercollins, 2006), 10. (coins) [5] moore, crossing the chasm, 16-20. [6] moore, crossing the chasm, 112-113. [7] saa never officially endorsed the at. archon also later began offering training sessions through saa. [8] the at can be run in different configurations — entirely local with the client and database run on the same machine, or with a networked database. the program is written in java and supports mysql, ms sql, or oracle backends. hibernate is used as a communication layer between java and the database. [9] ucsd contributes to the online archives of california (oac), which serves as an ead repository for institutions across the state of california. nyu has had its own ead publishing system in place for several years. [10] this is based on numbers provided in an interim report to the at project team from ithaka. roughly 6,000 archivists work in repositories that lack technical support for all but the most basic needs. in contrast, approximately 850-1200 archival institutions do have the technical resources to create and support their own ead publishing system. [11]moore, crossing the chasm, 117. [12] eric s. raymond, “the cathedral and the bazaar,” http://catb.org/~esr/writings/homesteading/cathedral-bazaar/ [13] many archivists who expressed interest in testing were unable to replicate their data and upgrade a different database to the test version, either because they personally did not have the technological know-how to complete such tasks, or because they had technical support that were not amenable to performing them. [14] karl fogel, producing open source software: how to run a successful free software project (2005/2009), 145. available on the web at http://producingoss.com/ (coins) [15] fogel, producing open source software, 149. [16] ironically, many of these tasks, especially documentation and support, are appealing to potential users and offer more of the whole product. i think the project was right in taking on those tasks but should have incorporated users more in their production. about the author sibyl schaefer is the metadata librarian for the university of vermont’s center for digital initiatives, where she provides metadata expertise for digital resources and manages the center’s digitization and metadata projects. she currently is a member of the society of american archivists’ standards committee and a certified archivist. sibyl previously served as the user services liaison on the archivists’ toolkit project, a role that included providing customer support and training on the application, conducting usability testing, developing the user group, and specifying new or improved features. subscribe to comments: for this article | for all articles 2 responses to "challenges in sustainable open source: a case study" please leave a response below: oss eval methods-lit review « practical e-records, 2011-06-02 […] as i noted a few weeks ago, emily brock and i are reviewing formal evaluation methods for open source software (oss). we’re doing this because i would like to get a handle on what worked or didn’t work with the archon project.  having an objective understanding of that project’s strengths and weaknesses will be critical as the archivesspace project moves forward.  the article that emily and i hope to write will complement sybil shaefer’s excellent code4lib piece. […] a case study of supply-driven product development hanging together, 2020-09-03 […] reflection on the development of the archivist’s toolkit in the most recent code4lib journal. challenges in sustainable open source: a case study was written by sibyl schaefer who worked on the project. she does the kind of brave, objective […] leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – using imagemagick to automatically increase legibility of scanned text documents mission editorial committee process and structure code4lib issue 14, 2011-07-25 using imagemagick to automatically increase legibility of scanned text documents the law library digitization project of the rutgers university school of law in camden, new jersey, developed a perl script to use the open-source module perlmagick to automatically adjust the brightness levels of digitized images from scanned microfiche. this script can be adapted by novice perl programmers to manipulate large numbers of text and image files using commands available in perlmagick and imagemagick. by doreva belfiore project background since 1997, the law library of the rutgers university camden campus has engaged in a project to provide open access to digital legal materials from the state of new jersey, other u.s. states, and the united states federal government. some of the documents are harvested from publically-accessible open data websites, and some of the documents come from the digitization of paper and microfiche volumes. problem for the digitization of state session laws and state constitutional documents, the library uses a mekel m565-200 microfiche scanner to scan individual images from microfiche frames and generate jpeg image files of each document page. the mekel filmscan scanning software allows for the manipulation of image density and brightness on the fiche level, but not on the frame level for individual page images. this creates a problem of potentially wide variations in brightness among images scanned from the same fiche. the library holds a collection of state constitutional law documents on microfiche which were produced by the greenwood press. greenwood press selectively inserted watermarks on random images, presumably by super-imposing a plastic or glass film sheet over the printed page prior to filming. these large watermarks (see figure 1) often render the resulting microfiche image dark and, frequently, illegible. in digitizing the microfiche images, there is no way to control for the brightness of the darkened watermarked files while maintaining the brightness and legibility of the non-watermarked ones. due to the scale of the state constitutions digitization project (estimated 350,000 image files), there would be no cost-effective way to individually brighten each watermarked file by hand using photo manipulation software such as photoshop or gimp. instead, we needed to find a way to automatically brighten and correct image density for large numbers of scanned text images. figure 1 – example of a greenwood press watermark. solution imagemagick is a free image manipulation program that can be run on various platforms (windows, macintosh, unix) and includes a perl module, perlmagick, for use in scripting. these features make it an excellent choice for high volume image editing. imagemagick, from the command line, can generate a large amount of image metadata, from exif values to size, filetype, brightness levels, histograms and more. it can also be scripted to perform a wide variety of image manipulations automatically. testing imagemagick alone on the command line, we found that the modulate command could increase the brightness of files. once we determined that imagemagick could manipulate individual files successfully to increase legibility, we created a script that would work efficiently on a large number of files at the same time. after the microfiche are scanned, the resulting jpg files are stored in one master folder and retained in their original format until all processing and quality control is complete and the images are uploaded to the rutgers camden law library website for public access. copies of each fiche set are made specifically for manipulation via perl scripts that assign pagination, embed descriptive dublin core and exif metadata into each image, and take preservation checksums of each image file. before pagination and metadata assignment, this copy set is used for any manual or automatic image correction procedures. for this image brightening project, we started with a clean reference file that we chose on the basis of legibility, and used the brightness level of that file as the example from which to evaluate other files in the directory. because greenwood press microfiche do not have a standardized image grid, the person scanning the microfiche has to use his or her best judgment in centering the fiche for the camera. as the camera scans each frame of the fiche sequentially, occasionally black edges occur around the document pages when the frames shift during the scan process. in order to prevent the black edges from skewing the brightness values of our reference image or the other images, we decided to select a sample block for testing from the center of the image. each original image (frame) on the microfiche is scanned at 400 dpi and is approximately 3804 pixels wide by 3193 pixels long, although the specific dimensions may vary among fiche sets. through trial and error, we found that the imagemagick shave command could crop a specific number of pixels from all edges of the image (500 pixels from each side formed an ideal ‘slice’), leaving us with a rectangular block in the center from which to evaluate each file. the “slice” or “snapshot” is loaded temporarily into memory for evaluation, leaving the original file unchanged. in addition to testing the center portion of each image, we also set the image to monochrome in order to increase text contrast and limit the mathematical comparisons to a smaller number of values (0 to 256). this was a useful choice for our state constitution text files, but might not be appropriate for photographic or other image files. using perl and perlmagick, we gather and count the number of files in a directory specified by the user, to stdin. we then open the selected reference file clean.jpg and take the value of the mean of image levels (see shave script below). as this file is monochrome, color values run from 0 to 256. for the purposes of our project, a precision level of 2 decimal points was sufficient to demonstrate the brightening effects that we sought. the script can be adjusted to use a different reference file for each fiche set, allowing the user to customize the baseline standard to the needs of each particular set of images. im_autocompare.pl script – shave function #instantiates an image object in imagemagick. $clean = new image::magick(); $rg = $clean->read('clean.jpg'); die "$rg" if $rg; die "$rg" if $rg; $rg=~ /(\d+)/; print $1; # print the error number print logfile $1; # log the error number #changes grayscale to monochrome to limit to shades of black $rg = $clean->set(monochrome=>'true'); #this command shaves 500 pixels off the top and sides so that it evaluates the middle #of the image and not the edges to get an accurate brightness level. $rg = $clean->shave(geometry=>'500x500'); $rg = $clean->set(page=>'0x0+0+0'); die "$rg" if $rg; $rg=~ /(\d+)/; print $1; # print the error number print logfile $1; # log the error number #gathers statistics about the clean file @rstats = $clean->statistics(); $rmean = $rstats[3]; print "the mean of the good file is: $rmean \n"; print logfile "mean of clean file is : $rmean \n"; we then open the images to be processed using perlmagick and read each into memory. the shave command crops out a “snapshot” of each image. the statistics command then reads the snapshot and calculates its mean color levels, which are stored in a variable. finally, the script compares the mean value of the image file against the mean value of the “clean.jpg” reference file. using a graduated percentage scale, if the current file is brighter than the reference file or within 25 points of the reference file value (i.e., a difference in means of less than 25), the script reports that the file is acceptable and takes no action. if the current file is darker than the reference file by a difference of 25 to 49 points, the script issues a modulate command of 250, or approximately 25% brightening (see modulate command below). if the file is darker by a difference of 50 to 74 points, the script issues a stronger modulate command of 500, or approximately 50% brightening. if the difference between the reference file and the clean file is 75 to 90 points, the subsequent modulate command is issued for 750 or approximately 75% brightening. for any difference greater than 91 points, the maximum modulate level of 999 is issued. this split scale of brightness can be adjusted for the needs of the specific files to be corrected. in all cases the script reports to the user what action is being taken and this report can be logged for troubleshooting and documentation. im_autocompare.pl script – modulate command #5 adjust files for brightness if needed # performing level adjustments based on scale of difference # between the mean level of the current file and the mean level of the # difference file. # modulate command adjusts brightness from 1 999 (highest). if (($diff > 25) && ($diff < 49)) { print "adjusting brightness by 25% of file $document now ....\n"; print logfile "adjusting brightness by 25% of file $document now ....\n"; #instantiates new imagemagick object. $fixer = new image::magick(); $ff = $fixer->read("$input/$document"); die "$ff" if $ff; $ff=~ /(\d+)/; print $1; # print the error number print logfile $1; # log the error number #adjusts brightness by 25% and writes over the file. $ff = $fixer->modulate(brightness=>'250'); $ff = $fixer->write("$input/$document"); } as all manipulations take place on the fly in system memory, the brightening process is efficient and takes approximately 1-2 seconds per brightened file. at the end of the brightening step, the original scanned file is overwritten with the corrected image. we maintain our original scans on a separate server as a backup in case of problems or errors. this overwrite step is completely optional, and can be changed by adjusting the perlmagick write command to write to a temporary file that can be reviewed, edited or replaced at a later time, as in: $x = $y->write(“$tempfile”); in addition, for error handling, the program is set to quit, print, and log any error messages generated by perlmagick. below are before and after examples to demonstrate the results of this script: figure 2 – “clean” reference file the “clean” reference value or mean brightness of the center of this “clean.jpg” is 222.43. example #1: figure 3a – original example file the mean of the brightness value of the center of this image = 147.017 (difference = 75.413) figure 3b – example file brightened by 75 percent example #2: figure 4a – original example file the mean of the brightness value of the center of this image = 170.566 (difference = 51.864) figure 4b – example file brightened by 50 percent examples of documents that have been improved by this script can be seen in the rutgers law library’s nebraska constitutional documents online collection: http://lawlibrary.rutgers.edu/stateconst/neconst/index.shtml known issues and limitations at the present time, this script has problems evaluating an image that is composed of only one side of a double-page spread. taking a center “snapshot” by using the shave command will necessarily include a partially black side and will skew the mean value of the image. for this reason, we did not choose to loop this script to automatically repeat the correction process. looping would brighten the printed page side to the point of washed-out illegibility. instead, we run the script sequentially, printing results and errors to a logfile and evaluating the output file for legibility after each pass. as a future enhancement to this script, we plan to edit the main script to recognize a highly skewed value and fire off a subroutine that will test the image a second time. by taking two vertical slices as “snapshots”, one on each side of the horizontal page, the script will be able to detect an empty page, which shows as a pure black frame. once detected, a flag can be set for the file to be marked as a single page and exempt it from further brightness evaluation. another limitation of the script as currently written is that the gradients for image brightening are deliberately broad. we sought to simplify the brightening process into 4 “strengths” at 25%, 50%, 75%, and 99%, respectively. the script could be improved to utilize better mathematical algorithms to measure the image brightness more tightly, and run multiple passes and checks on the images as they reach a desired brightness level. in a related project using imagemagick, we are testing the combination of perl scripts running perlmagick with cgi scripts presenting images to a non-expert user for evaluation. a negative user response to a prompted question sends the presented image file into a subroutine for further testing and checking, while a positive user response proceeds to the next image. in the future, combining these types of testing methods with an enhanced version of this image brightening script from perlmagick will allow for faster turnaround time in quality control for the state constitutions digitization project. future projects and other ideas as the primary goal of the rutgers camden law library digitization project is to digitize and make available as many public domain legal documents as possible, our overriding interest is the provision of public access. in order to achieve the scale of production needed to meet our goals, an automated processing method was absolutely necessary. for our purposes, the percentage scale of image brightening using our method was found to be sufficient to produce legible text from scanned greenwood press microfiche when viewed via a standard web browser. this script can be enhanced to increase brightness, sharpness, contrast, or other image levels on a more fine-grained scale, depending upon the needs and goals of the user and programmer. this script could also be extended to take advantage of other file manipulation commands offered by perlmagick. this use case is an example of how free and openly available tools can be used to enable the creation of large scale digital library collections at an affordable cost per image that is accessible to libraries and archives with very modest digitization budgets. at the same time, the code involved in this script can be understood and maintained by someone with a fairly modest level of technical programming knowledge, which also makes it an accessible tool for institutions that do not have dedicated programming staff. we have found it to be highly useful, and look forward to improving it over time and applying it to more digitization projects. references imagemagick studio llc. [internet]. [updated 2011]. imagemagick: convert, edit and compose images; [cited 2011 april 20]. available from: http://www.imagemagick.org/script/index.php. imagemagick studio llc. [internet]. [updated 2011]. perlmagick api; [cited 2011 april 20]. available from: http://www.imagemagick.org/script/perl-magick.php. joergensen, j.p. 2002. the new jersey courts publishing project of the rutgers–camden law library. law library journal 94(4):673-689. thyssen, a. [internet]. [updated 2011 march 15]. examples of imagemagick usage; [cited 2011 april 20]. available from: http://www.imagemagick.org/usage/. acknowledgements the author wishes to thank john joergensen of the rutgers university camden school of law for his mentoring and guidance. about the author doreva belfiore (dorevabelfiore@gmail.com) received her masters of library and information science in 2011 from the drexel university ischool in philadelphia, pa. she works as a digital library and circulation intern at the law library of the rutgers university school of law in camden, new jersey. appendix script: im_autocompare.pl #!/usr/bin/perl # # # imautocompare.pl # # # comparing brightness of a clean reference file # to then adjust levels of all files in a given directory # using perlmagick. # # doreva belfiore # rutgers university school of law camden, nj # law library #1 use imagemagick perl module use image::magick; #2 set variables here my($document, $clean); my($rg, $average); my($rf, $fixer, $ff); open (logfile, ">>im_autocompare.log"); #3 load the information about what a "clean", "good" or at least #"repaired" file looks like. #here we are using clean.jpg , a very clean document file as a reference image. $gdir = ("./reference"); chdir ("$gdir"); print "loading reference information now. please wait... \n"; #instantiates an image object in imagemagick. $clean = new image::magick(); $rg = $clean->read('clean.jpg'); die "$rg" if $rg; die "$rg" if $rg; $rg=~ /(\d+)/; print $1; # print the error number print logfile $1; # log the error number #changes grayscale to monochrome to limit to shades of black $rg = $clean->set(monochrome=>'true'); #this command shaves 500 pixels off the top and sides so that it evaluates the #middle of the image and not the edges to get an accurate brightness level. $rg = $clean->shave(geometry=>'500x500'); $rg = $clean->set(page=>'0x0+0+0'); die "$rg" if $rg; $rg=~ /(\d+)/; print $1; # print the error number print logfile $1; # log the error number #gathers statistics about the clean file @rstats = $clean->statistics(); $rmean = $rstats[3]; print "the mean of the good file is: $rmean \n"; print logfile "mean of clean file is : $rmean \n"; #4 open the user-specified folder and get information about the files $input = $argv[0]; chomp ($input); #$rdir = ("../scratch/alconst"); $rdir = ("../test"); chdir ("$rdir"); print "processing folder $rdir \n"; opendir(dir, "$input"); @files = grep /\.jpg/i, readdir dir; closedir dir; @files = sort @files; $number = @files; #checks files found in the directory print "found $number files in the directory $rdir \n"; foreach $document (@files) { #instantiates an image object with imagemagick. print "reading $document...\n"; $average = new image::magick(); $rf = $average->read("$input/$document"); die "$rf" if $rf; $rf=~ /(\d+)/; print $1; # print the error number print logfile $1; # log the error number print "transforming image...\n"; #this command shaves 500 pixels off the top and sides so that it evaluates #the middle of the image and not the edges to get an accurate brightness level. $rf = $average->shave(geometry=>'500x500'); $rf = $average->set(page=>'0x0+0+0'); die "$rf" if $rf; $rf=~ /(\d+)/; print $1; # print the error number print logfile $1; # log the error number #changes greyscale to monochrome to limit to shades of black $rf = $average->set(monochrome=>'true'); #gathers statistics about the current file @avstats = $average->statistics(); $amean = $avstats[3]; $diff = ($rmean $amean); print "the difference between $document and the reference file is $diff \n"; print logfile "the difference between $document and the reference file is $diff \n"; #5 adjust files for brightness if needed # performing level adjustments based on scale of difference # between the mean level of the current file and the mean level of the # difference file. # modulate command adjusts brightness from 1 999 (highest). if (($diff > 25) && ($diff < 49)) { print "adjusting brightness by 25% of file $document now ....\n"; print logfile "adjusting brightness by 25% of file $document now ....\n"; #instantiates new imagemagick object. $fixer = new image::magick(); $ff = $fixer->read("$input/$document"); die "$ff" if $ff; $ff=~ /(\d+)/; print $1; # print the error number print logfile $1; # log the error number #adjusts brightness by 25% and writes over the file. $ff = $fixer->modulate(brightness=>'250'); $ff = $fixer->write("$input/$document"); } die "$ff" if $ff; $ff=~ /(\d+)/; print $1; # print the error number print logfile $1; # log the error number elsif (($diff <= 74 ) && ($diff >= 49)) { print "adjusting brightness by 50% of file $document now .... \n"; print logfile "adjusting brightness by 50% of file $document now ....\n"; $fixer = new image::magick(); $ff = $fixer->read("$input/$document"); die "$ff" if $ff; $ff=~ /(\d+)/; print $1; # print the error number print logfile $1; # log the error number #adjusts brightness by 50% and writes over the file. $ff = $fixer->modulate(brightness=>'500'); $ff = $fixer->write("$input/$document"); } $ff=~ /(\d+)/; print $1; # print the error number print logfile $1; # log the error number elsif (($diff <= 90) && ($diff >= 74)) { print "adjusting brightness by 75% of file $document now .... \n"; print logfile "adjusting brightness by 75% of file $document now ....\n"; $fixer = new image::magick(); $ff = $fixer->read("$input/$document"); die "$ff" if $ff; $ff=~ /(\d+)/; print $1; # print the error number print logfile $1; # log the error number #adjusts brightness by 75% and writes over the file. $ff = $fixer->modulate(brightness=>'750'); $ff = $fixer->write("$input/$document"); } $ff=~ /(\d+)/; print $1; # print the error number print logfile $1; # log the error number elsif ($diff > 91) { print "adjusting brightness by 99% of file $document now .... \n"; print logfile "adjusting brightness by 99% of file $document now ....\n"; $fixer = new image::magick(); $ff = $fixer->read("$input/$document"); die "$ff" if $ff; $ff=~ /(\d+)/; print $1; # print the error number print logfile $1; # log the error number #adjusts brightness by 99% and writes over the file. $ff = $fixer->modulate(brightness=>'999'); $ff = $fixer->write("$input/$document"); } $ff=~ /(\d+)/; print $1; # print the error number print logfile $1; #logs the error number else { print "image fine. skipping to next file... \n"; print logfile "image fine. skipping to next file... \n"; } #6 undefine variables undef $average; undef $rf; undef $rg; undef $fixer; undef $ff; undef $document; undef $clean; } #end foreach loop close logfile; subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – an introduction to optical media preservation mission editorial committee process and structure code4lib issue 24, 2014-04-16 an introduction to optical media preservation as the archival horizon moves forward, optical media will become increasingly significant and prevalent in collections. this paper sets out to provide a broad overview of optical media in the context of archival migration. we begin by introducing the logical structure of compact discs, providing the context and language necessary to discuss the medium. the article then explores the most common data formats for optical media: compact disc digital audio, iso 9660, the joliet and hfs extensions, and the universal data format (with an eye towards dvd-video). each format is viewed in the context of preservation needs and what archivists need to be aware of when handling said formats. following this, we discuss preservation workflows and concerns for successfully migrating data away from optical media, as well as directions for future research. by alexander duryee introduction the role of optical media in archives has shifted in the past decade from preservation medium to at-risk format. while longevity models estimated a lifespan of 25-200 years for recordable media[1], recent testing has found this range to be optimistic by orders of magnitude. one collection of data cd-rs from the 1990s yielded a 92% failure rate after approximately twenty years of storage.[2] given how recently archives have approached optical media as an object of preservation, there is little literature and research regarding the format. while a comprehensive account of all migration and preservation issues of optically stored data is impossible, there is a need for a broad overview of optical media, which this document hopes to provide. this document will cover five of the most common data storage standards for optical media. in doing so, it hopes to inform institutions working with such media as to the nature and challenge of optical media with regards to preservation. the author also hopes that, by providing a general overview of optical media preservation, more advanced conversations and explorations of the medium can take place. note that this article deliberately limits its scope to the most common formats and uses of optical media. the total forms and functions of optical media are many and varied – from analog video to hidden data to graphics stored in control codes – and are beyond the scope of this document. archivists dealing with such discs are recommended to explore more specialized resources, such as technical communities and standards documentation. this document, while offering recommendations, is also not a prescriptive cookbook of workflows and tools for dealing with optical media, as the necessary research for such recommendations has yet to be performed. logical structure it is crucial to understand the logical layout of optical media before attempting any preservation activities. as the first optical media standard was iec 60908 (1982) for audio storage and playback, future standards reflected a media-centric approach in how they structure the disc. thus, despite standards such as iso/iec 10149 (available as ecma-130[3]) providing for filesystem-based storage, the language and structure of red book persists[4]. hence, while it seems strange to discuss file-based data in terms of logical tracks, this is precisely how data is stored on cd-rom. a simplified model of a compact disc’s logical structure is as such: a series of sessions, each containing a series of tracks. a track is, as its name implies, designed to be one discrete track of audio. for non-audio data (e.g. cd-rom), any number of filesystems may be contained within a single track. these tracks are arranged in a linear series and bounded by a lead-in (which contains the table of contents – locational and descriptive metadata – for the following tracks) and a lead-out. this collection comprises one session. early cds were designed as if only one session would be on a disc; this was expanded in 1990 to provide for multiple sessions. figure 1. a diagram of the logical structure of a multi-session cd. due to the flexibility of how data can be arranged on sessions and tracks, the following example configurations are possible: -one session, many audio tracks (typical audio cd) -one session, one track, many filesystems (typical data cd) -two sessions, one with many audio tracks, one with one track and many filesystems (enhanced cd) -one session, one data track, many audio tracks (video game) it merits noting that, on cd-r and cd-rw, a session can be open or closed. a session can have new tracks written to it so long as it is marked as open; by closing it, it prevents further data from being added and creates the final table of contents. methods of handling discs with open sessions are documented by the digital forensic community, due to their importance in evidence collection.[5] for purposes of this article, the structure of dvd is sufficiently similar to cd-rom that it does not merit in-depth analysis. iso9660 iso9660 is the most common format for carrying data on cd-roms, often appearing as the baseline filesystem on cross-platform discs. it is also common for it to serve as the foundational filesystem for more sophisticated systems, such as joliet and hfs – these often (but not always) serve as a layer on top of iso9660 data to provide additional functionality. its limitations (such as 8.3-style non-unicode filenames) make it compatible with all common operating systems, thus allowing for a fallback in case more advanced filesystems are not supported. as such, it is extremely common to find compact discs with one iso9660 filesystem alongside a mix of hfs and joliet – such a disc would provide advanced features for windows and macos, but the data would be accessible on virtually any operating system. from the perspective of preservation, iso9660 can be treated similarly to filesystems on magnetic media. as each sector must return data consistently on each read (consider the consequences of a dropped sector when executing compiled code!), and each sector must be easily found on the disc, these discs use the mode 1 sector structure. in contrast to cd-da’s use of every byte in a sector for data, mode 1 dedicates 2048 bytes to data, 16 bytes to sector sync and identification, and 288 to error detection and correction. these non-data bytes reduce the amount of user data on the disc but crucially allow for it to behave as a filesystem by providing for rapid accurate sector seeking and consistent data reading. as such, for purposes of migration, long-term preservation, and access, an iso9660-based disc can be treated similarly to a magnetic disk. while there is no need for the specialized hardware that magnetic media requires (such as write blockers), iso9660 tracks can be imaged at the byte level[6] via the standard variety of data transfer tools (dd, guymager, ftk, isobuster, etc.)[7]. the subsequent image can then be mounted as any other filesystem for archival analysis and user access. various access models and post-migration workflows, such as remote image mounting and automated filesystem analysis, have previously been explored similarly and parallel to research in magnetic media.[8] joliet/hfs while iso9660 is the most common data format for cd-rom, it is rarely seen by itself. due to the restrictions on file structures, there was a demand for some mechanism to expand the capabilities for cd-rom filesystems. to address this, microsoft established the joliet specification as an extension to iso9660 in 1995. joliet’s primary improvements over iso9660 are long filenames, unicode filenames, and deeper directory trees. as joliet does not manage the data on the disc – it only provides for enhanced filesystem metadata – it is typically packaged as a metadata layer next to an iso9660 filesystem. an exploration of a disc’s logical structure will reveal that an iso9660 file and its joliet counterpart both point to the same sector of the disc – while the filenames differ, one copy of the data is used. the hierarchical file system (hfs) was employed to allow for macintosh-specific file behaviors on cd-roms and to work around the limitations of iso9660. hfs, while specific to apple machines, is a more powerful filesystem than iso9660 – notably, it allows for longer filenames and metadata necessary to integrate more naturally with the operating system[9]. as such, it was heavily used to provide compatibility with macos; for example, a document without the necessary hfs-specific metadata will lack the format/software signifiers to open correctly. thus, even in environments where the hfs filesystem is no more than metadata pointing at underlying iso9660 data, the hfs filesystem is critical in retaining the compatibility of the disc. typically, hfs filesystems will contain data unique to the disc on macos, such as compiled bytecode and documentation specific to the operating system, which should be migrated as part of any archival workflow. figure 2. a disc containing an hfs filesystem, viewed in an emulator. note the floating text in the shell – this was accomplished by creating empty files with a blank icon and placing them within the finder window. [17] udf since udf was designed as a universal standard for data storage, it has few surprises for preservation. as opposed to a compact disc, a dvd will almost certainly (barring unusual circumstances) behave in a standard fashion across any number of devices. despite the apparent variations between dvd-video, dvd-audio, and data dvds, the underlying disc format is identical. while there are a variety of quirks between versions and implementations of udf and highly specific cases where its structural features are critical to a successful migration, these are beyond the scope of day-to-day preservation. note that, due to varying hardware support for udf versions, it is often paired with an iso9660 filesystem. due to iso9660’s near-universal support, it can be used as a “bridge” to allow older hardware to read the disc. as such, it is up to the individual archive to decide if one or both of the filesystems should be preserved. one important subclass of dvds is dvd-video, which is the predominant format for consumer video. using a screen essentialist approach, dvd-video is seemingly equivalent to audiovisual tape formats: the user inserts the disc into a player, which provides an interactive menu and a collection of linear audiovisual streams. however, this masks the underlying structure of the disc. in actuality, the dvd contains a udf filesystem with a standardized directory and file structure (which a dvd player recognizes and parses in a seemingly single stream). put generally, the filesystem contains a video_ts directory, which contains the mpeg streams (vob) and playback metadata (ifo/bup).[10] dvd-video can therefore be treated similarly to any data dvd for purposes of migration. the major issues that archives will face with dvd-video are overcoming the content scramble system (a copy protection scheme that prevents disc migration and hence preservation) and providing access to the raw audiovisual files post-migration. cd-da the data structure of a cd-da is more akin to tape than a traditional data disk. instead of dividing the storage area into discrete files, cd-da data is written as a linear pulse-code modulation (pcm) stream, divided into separate tracks. as the data is read by the disc drive (at 44,100 16-bit samples per second, chosen specifically to be above the necessary rate for perfect human reconstruction), the playback device interprets the pcm stream and generates the corresponding waveform. in this regard, cd-da is more closely allied to tape-based formats than it is to traditional magnetic disk, as it represents not a structured filesystem but a linear stream of media. figure 3. the pulse-code modulation stream to digital signal to analog signal chain as cd-da is fundamentally different from data formats like iso9660, there are a number of factors to consider. since there are no ‘files’ on a cd-da session, but streams of raw pcm data, the file listing provided by an operating system’s shell does not reflect the contents of the disc. windows, for example, will display .cda files that will play via media software. these files, however, contain no data – they are merely pointers to locations on the cd where their corresponding tracks begin. the data itself is something that cannot be parsed by the shell and thus requires special software to migrate. cd-da was also designed to maximize space at the cost of accuracy; thus, it sacrifices a third level of error correction in exchange for more data per sector.[11] the advantage of this approach was a marked increase in capacity and a level of error tolerance, as misread sectors will not interrupt smooth playback in all but the most egregious cases. thus, errors introduced during the read process can go undetected by the drive. typically, read errors will manifest as traditional sonic problems, such as clicks and pops. as a result, reading an audio track in a single pass will provide unreliable results, with consumer hardware being roughly 95% accurate at the track level.[12] this is unacceptable for preservation – consider the consequences of introducing errors to five to ten percent of tapes and records during transfer! various techniques have been developed to account for cd-da’s inherent lossiness. software designed for reliable cd-da extraction combine a variety of methods to ensure consistent reads across a given disc. for example, an extraction tool may read each sector of the disc multiple times in order to find the correct value, or it may compare the data against online databases containing other bitstreams of the same disc. while these methods are powerful in ensuring consistently good reads of cd-da, they are specific to this data structure, and thus they are not portable to iso9660 or other cd formats. this presents issues during the migration of so-called enhanced cds (ecd, also known as cd extra) and mixed mode cds (common with video games), which contain both audio and filesystem data. as the specific workflow for such discs is reliant on the structure of a particular disc, it is beyond the scope of this document. as it is unlikely that these will be found outside of specific subject collections, archives handling such media would do well to conduct research and contact specialists with regard to migration. applications in preservation workflows the most important analysis an archivist can perform on a disc occurs before it ever enters a workstation. recordable optical discs are frequently used by individuals as simple portable file carriers – for example, a donor may burn a dvd with material to transport as part of their collection, or a disc may be created to share documents around an office. in these cases, the disc serves as a simple carrier, analogous to an envelope or package – in other words, something that never merits preservation. as such, the archivist may decide to copy the files over via their shell (finder, windows explorer, et cetera) instead of imaging a disc, as there is no value in preserving the low-level structure. dependent on the nature of the discs in a given collection, this may be the preferred method of migration. the physical aspects of optical media, while typically not important to the data on the disc, do merit discussion with regard to migration processes. the read speed of a disc should be set as low as possible (via tools such as cd-rom tool spti or hdparm), as lower speeds can provide more accurate results. the quality of drive can also make a dramatic difference – empirical data has found high-quality consumer drives to be approximately five percent more accurate at the track level than poor ones[13] – and thus should be taken into account when designing an archival workstation. traditionally, plextor drives have served various preservation-minded communities well and therefore often recommended for optical migration.[14] given that an optical disc can have any number of filesystem and sector structures – some of which are invisible via an operating system’s shell – it is necessary to use dedicated tools for analysis. by reading a disc’s table of contents directly (instead of relying on what the operating system recognizes), it becomes possible to view a complete list of tracks on a disc (and their corresponding contents in case of data tracks). a handful of tools exist for exploring the structure and contents of a disc, such as cd/dvd inspector and isobuster (both windows-only). these will allow for an archivist to understand the broader structure of a disc (sessions/tracks, filesystems, files, etc.), as well as catching any discs containing cd-da before migration. figure 4. isobuster with a disc containing an iso9660 filesystem, extended by joliet and hfs layers due to cd-da’s lack of error correction in comparison to other cd/dvd formats, it requires a special workflow to migrate properly from cd. a linear read of a cd-da track will not give an accurate transfer of data, due to reasons outlined above; additionally, audio-specific issues (such as placement of silence between tracks) are issues not addressed in more general software. to solve these problems, specialized software was developed to address problems in cd-da extraction. the author recommends tools such as exact audio copy (windows), dbpoweramp (windows) and cdparanoia (linux/osx/windows), which are specifically designed to overcome migration barriers via methods to account for silent read errors, pregap detection, et al. the combination of empirical data and technical analysis of these tools make them the gold standard for audio cd migration. the target formats for optical migration are dependent on a variety of factors. a complete image containing every byte on a disc will not necessarily be the optimal format for long-term preservation, as it will include error correction and sync data. this will increase the size of an image by 305 bytes per sector (96mb across the entire disc), provide minimal utility, and create potential access issues. however, by using a binary image (typically a .bin file) and a cue sheet (.cue file, which provides the metadata necessary to divide the binary stream into sessions/tracks and interpret it), a disc can be replicated perfectly. in certain cases, particularly that of complex discs, this may be necessary for creation preservation masters. an alternative when handling single-track discs is a single track image, which will capture the user data within the track and provide a mountable and usable image. general workflows for the creation of preservation masters can be found in the appendix. for cd-da, the target migration format is typically 16-bit/44.1khz wave. as the raw pcm data is equivalent to a headerless wave file, the header can be appended with no change to the audio data. this file can then be manipulated to an archive’s specifications for transcoding, broadcast wave metadata, et cetera. if the structure of the disc is important, the bin/cue format is an alternative to discrete wave files that retains the timing and layout of the disc[15]. a suggested workflow using exact audio copy can be found in the appendix. as dvd-video is structured no differently than any other data dvd, a binary image of the disc is sufficient for preservation. given that there is no real need for transforming the mpeg streams within the vob files for preservation, the image can be stored as-is and mounted upon request for access. however, due to the variety of needs and methods for preserving, describing, and accessing digital video, it is impossible to suggest a single target format – for example, an asset management system may require a specific codec for playback. while the vob files are stable and generally playable, there may be a need to transform the files for broader/easier access (e.g. concatenating split files into one object). an archive will need to set policy for handling copy protection, transcoding, storing, and providing access to the media. future research the current state of preservation research with regard to optical media is rather lacking, as it limits its scope to the physical artifact. while relevant in the short term, the inevitability of disc rot, coupled with the decrease in the manufacture of quality drives, places an urgent imperative on migration research and practice. as there has been little research performed in allied fields such as law enforcement and computer science, there is a greater need for the establishment of practices and knowledge relating to optical media by archivists. areas of future research may include: documentation of best practices with regard to known and unknown types of optical carriers establishment of techniques and workflows for damaged media scaling optical media migration and balancing best practices with efficiency outlining of significant properties of classes of optical media metadata standards for describing optical media and data stored as such exploration of methods for preserving more exotic optical formats, such as cd+g and laserdisc-based formats the author anticipates that, as the archival scope begins to encompass the era of optical storage, a greater need for established workflows and advanced research will arise. the use of optical media as objects of archival preservation has been limited to specific projects (namely government documents and digital art) and external communities (particularly music and video game collectors). by applying and adapting existing knowledge to generalizing optical preservation, archivists can prepare for the next generation of digital preservation challenges. appendix cd-rom suggested workflow analyze disc with isobuster and determine workflow (with a particular focus on cd-da, number of tracks/sessions, and initial errors) if the disc is cd-rom only, analyze for structure (iso-based, hfs, hybrid, etc) and describe as per metadata standard extract cd <image> -> user data, or if non-data bytes are necessary (for byte-level alignment), raw this will capture all cd-rom sessions/tracks on a disc in a binary image. if user data was selected, this can be mounted as a disc within one’s operating system; if raw was selected, one may need to extract the user data before mounting the image. raw images can be translated into mountable user data via the bchunk tool on linux/osx. cd-da suggested workflow due to the complexity – and necessity – of properly configuring exact audio copy (eac), it is beyond the scope of a general document. a guide to the varying software and drive options can be found at the hydrogen audio knowledge base.[16] note that, while eac’s software options can be generalized for preservation, every drive is unique and thus requires individual configuration. it is also prudent to align one’s drive with the accuraterip database in order to compare individual drive accuracy against others of the same model. analyze disc with isobuster and determine workflow if the disc is entirely cd-da tracks, open eac and read the disc into it detect pre-gaps and check for unusual deviations (gaps are typically approximately 2 seconds each) test & copy selected tracks -> uncompressed this will produce wave files as per specifications set in eac’s preferences. due to eac’s emphasis on reliably reading cd-da (via methods such as comparing multiple reads against each other), this will be a very accurate migration of the data on the disc. if there is a need to maintain the structure of the disc via a cue sheet, this can be created within eac using the current gap settings. glossary cd: compact disc. a format designed to hold audio data (and later expanded to general data) on a 12cm plastic disc, using a laser to read a series of pits and lands as binary data. cd-da: compact disc – digital audio. the standard used to define the logical, physical, and data structures of audio discs. cd-r: compact disc – recordable. a worm (write once – read many) format allowing for compact discs to be created using consumer hardware. cd-rom: compact disc – read only memory. the standard used to define the logical, physical, and data structures of data discs. cd-rw: compact disc – rewritable. a format, similar to cd-r, that allows for data on the disc to be written over. dvd: digital versatile disc. a format designed to store cinematic-length motion pictures on a 12cm disc. this led to the creation of a disc that offered 6-10 times the capacity of cd-rom in the same form factor. dvd-video: the standard filesystem and file formats for distributing motion pictures on dvd. enhanced cd: a format for storing cd-da and cd-rom data on a single disc. this was originally impossible, but was permitted with the blue book standard in 1995. hfs: hierarchical file system. the standard filesystem used on macos, which was used on cd-rom to provide compatability and enhanced features. iso9660 (“iso”): the standard used to define the filesystem and file structure in place on compact discs. despite being highly limited, it is near-ubiquitous on cd-rom. joliet: an extension to iso9660, designed by microsoft, that provides a metadata layer to overcome the limitations of iso9660. mode 1: a substandard of the cd-rom data structure. mode 1 defines 2048 bytes of user data per sector, with the rest of the sector used for error correction and sync bytes. mode 1 is much more commonly used than mode 2, which was designed for use in more exotic formats such as cd-i and video cd. pcm: pulse-code modulation. a method for storing analog audio data as a series of binary values. udf: universal disk format. a standard designed to supersede iso9660 by lifting some of its restrictions. udf set out with the goal of replacing the myriad cd data formats with a single one. correction when this article was first published, a citation to figure 2 was missing and has since been added. we apologize for this oversight. references [1] range commanders council, optical systems group. multimedia archiving: videotape, compact disc (cd), digital versatile disc (dvd), and blu-ray disc (bd) media [internet]. white sands missile range, new mexico: range commanders council [updated 2010 february; cited 2014 february 14]. available from http://www.wsmr.army.mil/rccsite/documents/462-10_multimedia%20archiving%20-%20cd,%20dvd,%20and%20blu-ray/462-10_multimedia%20archiving%20-%20cd,%20dvd,%20and%20blu-ray.pdf [2] wilsey l, skirvin r, chan p, edwards g. capturing and processing born-digital files in the stop aids project records: a case study. journal of western archives [internet]. [cited 2014 february 14]; 4:1. available from http://digitalcommons.usu.edu/cgi/viewcontent.cgi?article=1026&context=westernarchives [3] ecma international. data interchange on read-only 120 mm optical data disks (cd-rom) [internet]. second edition. geneva, switzerland: ecma international. [updated 1996 june; cited 2014 february 14]. available from https://web.archive.org/web/20130116084229/http://ecma-international.org/publications/files/ecma-st/ecma-130.pdf [4] the red book is the colloquial name for iec 60908, due to the color of its cover. the various standards that define optical media are collectively known as the rainbow books, as each has a differently colored cover (e.g. iso/iec10149 being the yellow book). [5] more information on the theory and method of retrieving data from discs with open sessions can be found in crowley and kleiman’s cd and dvd forensics [6] note that the term “bit level” is meaningless with regard to optical media. instead of using eight physical bits per byte, all compact discs use eight-to-fourteen modulation (efm): the 256 combinations of fourteen bits most favorable to accurate reading are mapped to corresponding eight-bit bytes, and the fourteen bit words are written to disc. as such, a true “bit level” rendition of a compact disc would require an analysis of the physical artifact. for more information, see the ecma-130 standard linked above. [7] note that the colloquialism “iso image” is frequently misleading. despite implying that a disc image file ending in .iso captures only iso9660 data, it is instead used to describe a general disc image. as such, unless the provenance of a .iso file is documented, it cannot be used when determining the scope and content of migration. [8] woods, kam a. (2010). preserving long-term access to united states government documents in legacy digital formats [dissertation]. bloomington (in): indiana university. available from https://web.archive.org/web/20140217161345/http://www.digpres.com/publications/kam-woods-dissertation-pre.pdf [9] for example, the creator code and type code allow for macos to open a file automatically using the intended software. [10] the dvd video specification also defines an audio_ts directory. this is only used on the exotic dvd-audio format and will likely not be encountered in most archival scenarios. for the purposes of dvd-video, it will most always be empty. [11] the specifics of error correction of cd-da data is complex enough to fall beyond the scope of this document. more information can be found at: https://web.archive.org/web/19970616191108/http://www.ee.washington.edu/conselec/ce/kuhn/cdmulti/95×7/iec908.htm [12] cd/dvd drive accuracy list [internet]. [updated 2013 may 14]. [cited 2014 february 14]. available from https://web.archive.org/web/20131203200732/http://forum.dbpoweramp.com/showthread.php?30430-cd-dvd-drive-accuracy-list-2013 [13] ibid. [14] note that plextor ceased production of its own hardware in the mid-2000s; any drive with the plextor name made after that point was manufactured by a different company. [15] the canonical example of bin/cue being necessary for cd-da is music designed for dance, which often uses no gaps between tracks. as bin/cue allows for direct manipulation of gaps, this allows for the session to be played back as intended. [16] eac options [internet]. [updated 2013 august 3]. [cited 2014 february 14]. available from http://wiki.hydrogenaudio.org/index.php?title=eac_options [17] norie neumark, shock in the ear, 1998. rose goldsen archive of new media art, #8200. division of rare and manuscript collections, cornell university library. screenshot by dianne dietrich, mac os 8 in basilisk ii. about the author alexander duryee is a digital preservationist with avpreserve experienced in web archiving, optical and magnetic storage migration, and data preservation. holding an mlis from rutgers university, his background includes archival tool development, data access, preservation infrastructure analysis, and metadata management. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – git and gitlab in library website change management workflows mission editorial committee process and structure code4lib issue 48, 2020-05-11 git and gitlab in library website change management workflows library websites can benefit from a separate development environment and a robust change management workflow, especially when there are multiple authors. this article details how the oakland university william beaumont school of medicine library use git and gitlab in a change management workflow with a serverless development environment for their website development team. git tracks changes to the code, allowing changes to be made and tested in a separate branch before being merged back into the website. gitlab adds features such as issue tracking and discussion threads to git to facilitate communication and planning. adoption of these tools and this workflow have dramatically improved the organization and efficiency of the ouwb medical library web development team, and it is the hope of the authors that by sharing our experience with them others may benefit as well. by keith engwall and mitchell roe a library website is a complex entity that consists not only of web pages containing static content, but a variety of dynamic content, integrated access to other information systems, and other features that rely on customized code. as websites become more reliant on code, changes to the site run greater risk of errors that may lead to downtime. therefore, it is good practice to design, implement, and test changes within a development environment prior to releasing them to the production site. setting up a development environment may, however, be a challenge to some libraries, especially if there are multiple personnel working on the site. coordinating changes, backing up files, and ensuring that everyone is working on the most recent version adds a layer of complexity that needs to be addressed. this article details how a small medical library designed and implemented a development environment and workflow for a multi-developer team without the need of a development server, and discusses the challenges and benefits of setting up and using this environment for the management of a production library website. overview of library’s web environment in 2011, the oakland university william beaumont (ouwb) school of medicine opened its doors to its first class of students, and the medical library began development of its library website. initially, the medical library piggy-backed off the university library’s website as a subfolder within that site. necessitated by specific needs of the various authors of the university library’s site, the library website was managed through dreamweaver, a web integrated development environment (ide), and consisted of html, css and javascript. the site was configured to use dreamweaver’s template system to cordon off and synchronize sections of html designated to common elements (e.g. header, footer, navigation, etc.). eventually, a separate virtual host was set up for the medical library on the university library’s web server. the medical library’s web librarian and technology specialist started building a new site from scratch, using editors of their choosing and replacing dreamweaver’s template system with php modularization (shared elements would be placed within their own files and pulled into pages through include statements). php also allowed for the development of dynamically generated content from data files. for example, the medical school’s required textbook spreadsheet was converted into a set of dropdowns, providing a list of required textbooks for each course with links to electronic versions. the medical library’s team was uncomfortable with the idea of performing development directly on the production site, but no development site was immediately available. their request for one was delayed by review and approval processes of the university’s central it service. to move forward, the medical library designed a development infrastructure and a workflow that would effectively accommodate their collaboration on site development. they decided to use git, an open-source distributed version control system, to track changes during the development process. git builds a self-contained repository around a set of folders and files, storing metadata about the files and changes made to them in a hidden folder within the root folder. each of the web authors would have their own copy of the website repository from which to make changes. to coordinate their changes, the official repository would be stored on the university’s implementation of gitlab, an open-source web-based devops platform built on top of git. each team member would be able to pull the most recent copy of the repository from the gitlab server, make changes, and push them back to the server. git would track these changes and maintain a history of them on both the local computer and the gitlab server, interleaving the changes made by each person and detecting conflicts. in addition to providing centralized storage for the repository, gitlab had project management features that allowed them to better track and plan their work. they developed a six-step workflow around these tools, and found that they had everything they needed for a development environment without the need of a centralized development server. the following section will provide more detail about the development infrastructure and the features that informed the design of their workflow. the role of git and gitlab in website development workflow the core of this development environment is git [https://git-scm.com/], an open-source, distributed version control system. git is very flexible and can be set up in a variety of configurations to fit several different development workflows. git commands can be run from the command line, from a dedicated graphical user interface (gui) client, or as a set of functions built into an integrated development environment (ide) or development tool. git tracks changes made to files and stores them along with metadata as “commits” within a repository. git only stores changes made within files, and not full copies of the files themselves (except for binary files, such as images), making git’s use of disk space extremely efficient. a “branch” is like a “version” of the repository. a branch is actually stored as a set of commits tracked by git until the branch is “checked out”, at which point git applies the commits to the files, bringing them up to date with the most recent commit in the branch. in most configurations, each repository has a “master” branch representing the core version of the files. development branches may be created from this “master” branch to serve as an independent copy of the master branch, providing a safe place to make changes to files without risk to those within the master branch (figure 1). figure 1. development branches in git. once the changes are complete and have been tested, a development branch can be merged into the master branch to apply the changes to the files within. git can track multiple branches, and will gracefully interleave changes so long as they don’t conflict within an individual file. in such situations, git will identify the conflict, present the alternative edits, and provide the opportunity to resolve the conflict prior to merging. after the file has been edited to retain the intended changes, the merge may be attempted again and should be successful. git thus isolates the master branch from these risks, and provides a means to roll back changes from a commit or an entire branch. this allows for relatively quick recovery as well as the information necessary to identify and analyze the cause of the problem. git is distributed by design, which means that repositories can be copied (a.k.a. “cloned”) between computers, including the metadata for changes made thus far in the master branch, as well as additional branch metadata, as needed. git is kept up to date by “pulling” changes from a remote repository. the remote repository is updated by “pushing” changes in a branch, along with the corresponding metadata, to the remote repository. this may be done from development branches, allowing a gatekeeper to oversee the merging of development branches into the master branch. this allows developers to work independently without cluttering remote repositories with branches containing trivial changes, depending on the situation. git’s feature set makes it extremely suitable for website change management, especially in situations where multiple individuals are collaborating on a site. although it is fairly technical and may be difficult to master, this may be attenuated by the use of an ide such as jetbrains phpstorm [https://www.jetbrains.com/phpstorm/] or adobe dreamweaver [https://www.adobe.com/products/dreamweaver.html], which incorporate git functions within the interface in a way that may be more user-friendly. as a front-end to git, gitlab provides a robust interface for git repositories stored on its server. users create accounts on gitlab and may create or be invited to participate in git repositories that are stored on the gitlab server. access to the gitlab server directly from git can be facilitated by setting up an ssh authentication key in the user’s profile. beyond providing functions to work with the repository itself, gitlab extends this functionality through additional integrated tools and features. one key feature that gitlab provides is the issue tracker. this acts as a ticketing layer that sits above the repository, facilitating higher-level project management through the creation and organization of task items (“issues”). issues contain several features often found in task management systems: they can be assigned or claimed, given due dates, and/or attached to project milestones. they contain a title, description, and a comment thread for high-level discussions about the issue (the nature of the problem or desired change, expectations, time frames, etc.). stakeholders outside of the development team can be given gitlab accounts with limited access to the repository, which will allow them to review, submit, and participate in issues. this ticketing layer integrates with the repository through the creation of branches. branches created from the issue are linked to the issue. this integration reduces the overhead of project management, resulting in a more efficient work process. gitlab also provides a set of tools and an additional comment thread from within its merge request interface. once a branch is ready to be merged, a merge request can be created from the issue. at this point, members of the web team may open the merge request to review the changes made in the branch and discuss the changes if necessary. the merge request interface contains a list of commits made to the branch and a comparison view, which shows the differences in each file between the development branch and the master branch (by default). the comparison view may also be configured to show changes between any two commits from within the branch. once satisfied, a member of the dev team can merge the branch into the master branch from the merge request interface. gitlab will use git to handle the merge, and any conflicts detected by git will be noted in the merge request, where they can be resolved prior to merging. a six-step workflow for web development after much discussion and some trial and error, the web team settled on a six-step workflow that would be applied for each change to be made to the website (figure 2): create an issue for a proposed change create a branch to isolate the change until it has been tested make, test, and commit edits to the code until the change is complete review the change merge the branch and the change within back into master push the change to production figure 2. workflow for implementing changes to the medical library website. this workflow operates across three layers: the ticketing layer outlines the use of gitlab’s issue tracker feature as a ticketing system, allowing for a proposed change to be tracked as an issue with documented discussion available at every stage in the workflow, until the change is pushed to production and the issue is closed. the change management layer uses git and gitlab to create a branch for the overall change from the corresponding issue, track the edits that are made to the code by storing them within commits, and then, once the change is complete and tested, merge the branch back into master and push the change to production. the development layer is where the change is made via edits to the code and testing of those edits. this workflow is flexible and allows developers to use the tools of their choice in conjunction with gitlab, whether that be git, a text editor, and an amp stack, or an ide that pulls everything into a single interface. each of the steps in the workflow are detailed below. step 1: create issue when a change needs to be made to the website, the first step is to create an issue. gitlab’s issue tracking feature serves as a ticket layer in the workflow, allowing any stakeholder, whether it be a librarian, a staff member, or one of the web developers; to document an observed need for the website (e.g. problem, desired feature, or content change) by submitting an issue. as with a ticket in a helpdesk system, the person submitting the issue is automatically added to it as a participant, allowing them to receive emails when the issue is updated. each issue has a comments section that can be used as a discussion thread for the proposed change. this can be useful both in planning and implementing changes. in figure 2, this is represented by extending the discussion action across the entire workflow diagram. developers may request more information from stakeholders, discuss potential solutions with other members of the development team, or link to related issues, whether open or closed. posts to the thread are emailed to participants, and individuals may be invited to participate from within the text of a post, using social media conventions, by referencing their gitlab username prefaced by ‘@’ (e.g. @engwall). the issue will appear in the issue list for the website repository in gitlab. members of the development team can monitor the issue list, coordinating and prioritizing issues as necessary. gitlab will display the user who last acted on an issue along with the date of the update, and so working on an issue is typically a sufficient way to automatically “claim” it. however, if desired, issues may be formally assigned (or claimed) by selecting someone in the optional “assignee” field. step 2: create branch in order to make changes to the website code for a proposed change, a development branch is first created off the master branch in the repository from within the corresponding issue in gitlab. gitlab uses git to create the branch and then automatically links it to the issue within gitlab. the branch title becomes the id number of the issue followed by the issue title. for example, a branch created from the 23rd issue created for the repository in gitlab, that has the title “add chat feature”, would be given the title “23-add-chat-feature”. step 3: make changes and test thus far, the branch exists only in the repository on the gitlab server. all of the actual work on a change to the website is done on the web developer’s local repository on their computer. this step has several substeps: pull an update from the repository on gitlab to the local repository. this will update the master branch as well as git metadata about the issue branch and any other branches that may have been created or updated since the last time they pulled from gitlab. check out the development branch to be worked on. as changes are committed to the checked-out branch, git will store and track them within the branch. this isolates changes and protects the master branch from them until they have been tested and the development branch is ready to be merged back into master. edit files and test changes. the developer has complete freedom over how they choose to edit files and test the changes they make. the medical library team uses text editors (e.g. vim [https://www.vim.org] or emacs [http://www.gnu.org/software/emacs/]) to edit files and mamp to test changes. an ide (e.g. phpstorm, dreamweaver) may not only provide a means to edit files and test changes, but also provide an interface for and/or automate other git actions in these substeps. commit changes made to files. changes to files are not stored by git until they have been committed. technically, each commit is preceded by an “add” action in git, which stages the added files for the commit and allows for more precise control over which edits are contained within each commit. however, typically, these two actions are done in immediate sequence, and are thus presented as a single substep. as part of the commit, git expects a brief text summary that summarizes the content of the commit. this can prove to be very useful during review and troubleshooting. push branch back up to the gitlab repository. although, technically, this can be done at any time (which can be useful if you want someone else to look at the code in their own repository part-way through working on an issue), typically the branch is pushed to gitlab after all of the edits have been made and all testing is complete. at this point, the code is ready for a final review before being merged and pushed to production. step 4: review changes this is an optional but recommended step in the change workflow. a second member of the development team reviews the change before merging and pushing it to production. this may either be done by pulling and checking out the branch on their own computer for testing or, for simple changes, by reviewing the changes to the code directly from gitlab. gitlab has several tools within its merge request feature (like issues, merge requests are another gitlab extension to git functionality) that facilitate code review, including a list of commits with the summary provided for each commit and a compare feature that will display all edits made, either to all files in the branch over all, to all files between two commits, or to an individual file. each merge request also has its own comments section, where technical code-level discussions can take place without cluttering the higher-level discussions in the issue. a merge request may be created at the time the branch is created or at any time afterwards. some prefer to wait until the branch is ready to be merged to create a merge request, as this is fairly intuitive. others prefer to create a merge request up front in order to have access to the additional functionality. gitlab links merge requests to the issue and its corresponding branch, using a similar naming convention (since a single issue’s branch may have multiple merge requests, the id used in the merge request title is the merge request number instead of the issue number). gitlab provides a way to tell whether a merge request is ready to be merged by automatically applying a “work in progress” (wip) status to the merge request when it is created (this is also reflected in the title of the merge request, which is prefixed by “wip”). the wip status must be removed before the merge action becomes available. step 5: merge branch once the change has been reviewed, and the wip status removed, the branch can be merged from the merge request interface in gitlab. this migrates the commits from the development branch to the latest version of the master branch, which may contain other changes from other branches that were merged after this branch was created. prior to the merge, git will automatically check for conflicts (changes made to the same lines within the same file), display any that it finds, and prepare them for resolution. each file with a conflict will have a delimited section for each conflicting set of edits. once all conflicts have been resolved by removing the unwanted edits and the surrounding delimiters, another merge may be attempted. there are options to automatically delete the branch and/or close the issue. typically, these are selected, since this and the next step are usually performed together. due to the distributed nature of git, each developer will need to remove any branches from their own repository. all participants will automatically be notified when an issue is closed. an issue may always be re-opened if follow-up discussion is needed. if further changes are necessary, it is best to create a new issue and corresponding branch, and reference the previous issue in the discussion thread. step 6: push to production while this step is almost always performed immediately after the merge, it is presented as a separate step because it is distinct from the merge, and the methods used to carry out this step depend on access to the production web server, and configurations of the production environment may vary widely from one institution to another. at ou, the gitlab server and production web server are separated by a firewall, and only specific individuals have access to the web server from their computers. thus, in order to push a change to production, a member of the team with such access must pull the latest version of the repository to the local repository on their computer and push the master branch from there to the production server. one of the best features of this workflow is that it will accommodate multiple simultaneous changes in active development by one or more individuals. this allows each issue to be worked on without worrying about what others are working on, which provides a great deal of flexibility in prioritizing and planning and improves efficiency. limitations & challenges this workflow does have one major limitation. it is designed for an environment where the production web server is not directly accessible from the gitlab server. the developers’ local computers serve as an intermediary between the gitlab server and the production web server. failure to perform the final step and push a change to the production server means that the issue may be fixed in the code base and closed in the tracking system, but not be represented on the production web site. therefore it is recommended that the master branch be pushed to the web server immediately after a merge. there are other challenges that may arise. like most software, flexibility often comes at the cost of complexity, and git is no exception. it is elegant, but can be a challenge for new users to get their head wrapped around. this article does not provide the specific git commands to perform the tasks described therein. rather, it attempts to describe the tasks performed with git such that the specific commands can be easily referenced, and provides links to resources that describe the execution of these tasks. the command line version of git may be suitable for web developers, but may be difficult for those with limited experience with this kind of interface. content creators may benefit from the use of an ide that interfaces with git (many do). alternatively, they may submit content changes through the gitlab interface or, if the website uses a content management system (cms), through the website’s interface. this raises another challenge in that git is designed to track changes to plain text files. although binary files (e.g. database files, images, pdfs, etc.) can technically be tracked, even minor changes to a database (e.g. oracle, mysql, mariadb) or to an image file will cause a new version to be stored in its entirety, which can cause a repository to grow prohibitively in size. this also means that individual changes made within a database cannot be tracked. thus, git may not be an effective tool for websites using a database-backed cms, unless the cms can be configured to use a flat-file database, which is stored as a plain text file. some cms solutions (e.g. grav [http://getgrav.org], kirby [https://getkirby.com/], etc.) are specifically designed to use flat-file databases. other binary files, such as images and pdfs, require separate consideration. git may be configured, through use of a .gitignore file, to ignore these types of files. they would not be tracked by git and would need to be uploaded to the webserver separately. if space is not a concern or if these files will not be updated frequently, it is ok to allow them to be tracked by git. if so, it is recommended to perform image work within its own commit, and only perform the commit once the final version of the image has been selected. this was not a significant issue for the medical library, since most of the images used on the medical library website do not change often. it is, however, something to keep in mind. gitlab itself also provides its own set of challenges. gitlab is a shared web-based tool and thus must be hosted on a site that can be accessed by everyone who would use it. if your institution does not have its own instance of gitlab, your choices are to install it yourself (which may be non-trivial, depending on your comfort level and access to resources) or to set up accounts on gitlab.com. there are free options for both gitlab software to be installed locally and for hosted accounts on gitlab.com that would provide all of the features described in this article, but there may be factors at your institution that could preclude storing and/or accessing your website files on an external third-party server. in addition, the merge request feature of gitlab can be somewhat counterintuitive. the tools it provides are absolutely beneficial, particularly if an issue is complex or difficult. however, the notion of creating a merge request before a single line of code is edited does not sit well with some. it’s an oddity that can be safely ignored from a practical sense, since the merge request can be marked as a “work in progress”, which serves as both an indicator of whether or not the branch is ready to be merged, as well as a mechanism to block merging the branch before it is ready. it can be enough of a conceptual disconnect that some may be reluctant to use it at first. on the plus side, git can absolutely be used without gitlab. a centralized repository can be set up on any networked computer on which git is installed. if all else fails, if git can be used on the production web server, it can provide a measure of protection for the production version of the site on its own, as changes can be made independently of the production web server and pushed there when ready. the changes can also be rolled back via git if needed, providing a mechanism for elimination of regressions or mistakes. an alternative solution would be required for issue management, but there are project management tools available, such as trello, that could fill this role reasonably well, though without the benefit of git integration. also, there are several git clients [https://git-scm.com/downloads/guis] that have robust features both for interfacing with git as well as code review. considerations for testing this workflow is flexible enough to allow developers to choose how they make and test changes. the medical library made use of two solutions for testing changes made to the website code. initially, the technology specialist set up a virtual machine (vm) using virtualbox [https://www.virtualbox.org/], a free, open-source virtualization application owned by oracle. this vm was installed and run on each team member’s computer. a virtualization app (also known as a “hypervisor”) allows for the installation of an operating system which runs on top of the computer’s main operating system. this “guest” operating system is stored in a file on the computer, large enough to serve as a hard drive for the virtual “machine.” the app acts as a bridge between the virtual computer and the actual computer’s hardware, and provides a virtual network interface between the vm and the computer. the virtual machine can be booted up and run in the background as though it were an independent system connected via network. virtualbox sets up a virtual network adapter on the vm and assigns it a localized ip address. by pointing the browser to the ip address of the vm, the developer can view the site just as it would appear on the production server. the local repository can either be stored within the vm and edited by logging in to the vmor on the local computer and mounted in the vm (e.g. via ssh, smb, nfs, etc.). the test vm was set up using a freely-available version of the production system’s red hat linux operating system (i.e. centos), replicating the production environment as closely as possible. a second, more lightweight option the medical library tried was to install and run mamp [https://www.mamp.info/], an application for macos. an amp stack is a bundle of software containing apache, mysql or mariadb, and php, perl, and/or python. mamp (the “m” in which refers to “macos”) bundles these tools together and provides a management interface for starting, stopping, and changing the services as needed. other, similar tools include xampp [https://www.apachefriends.org/] and ampps [http://ampps.com/], both of which can be run on windows, mac, or linux. depending on the tool used, server applications may run directly on the computer itself or on a built-in vm running a lightweight os strictly for the server apps. the amp manager application can be configured to select the folder to be used as document root and to specify software versions to be used. mamp was configured to use the repository directory as its document root and to use the same version of php as the production server. using a custom url specified in the mamp documentation, the web developer was able to view the local copy of the website in a browser. this alternative traded the fidelity of the cloned environment for efficiency and ease of use, under the assumption that so long as the php version was consistent, the results would be sufficiently accurate for testing purposes. with their local repositories and development environments set up, the web team collaborated to design a workflow that would meet their needs. regardless of what method of testing you choose, it is important to know what versions of software are running on your production server, such as apache and, especially, the code interpreter (such as php, perl and/or python, and to configure your test server environment to use the same versions. while this is not straightforward in many amp stack solutions, including xampp and mamp, it is nonetheless easier than setting up a vm, and although setting up a vm for a test environment may provide the opportunity to set up the most accurate replica of your production environment, the overhead involved in both setup and use make this difficult to recommend. looking to the future in addition to the protection that git provides, the ability to retrieve previous versions of the site has promising implications for archival purposes. in most cases, aside from restoring an old backup, web content tends to be ephemeral and requires proactive steps to be taken in order to preserve the style and design of the website at various points in time. however, in many cases, changes to the website can be gradual enough that it is difficult to determine in advance when to preserve the site. however, by default, git keeps all changes made, indefinitely. git also can be used to “tag” specific commits as significant. this can be a version number tagged to the final commit after a significant set of changes, or it can be any text that may prove useful in identifying commits that you want to return to later, if only temporarily, by checking out that commit by its tag name. some potential next steps for the medical library include retroactive use of tags at strategic points in the history of the website, expanded use of gitlab as a ticketing system for other types of library issues within a separate “helpdesk” repository, and investigation into additional features of git, gitlab, and/or other git-related applications that may further enhance the website development process. like any production resource, library websites should have a development environment in which to make changes. although this may seem out of reach, especially in multi-developer environments, it can be achieved using freely-available software and tools. there are free open-source tools such as git, gitlab, and amp stacks, that can be used to create isolated development environments without the need of a development server, as well as collaborative environments for sharing information and resources. the workflow is as important to an effective development process as the tools used. changes should be tracked, coordinated, and documented, and should follow a process that is efficient, open to communication, and flexible enough for the needs of various stakeholders. through use of these tools and this workflow, the productivity of the web development team increased greatly. capturing ideas for the website as well as problems became much easier. whenever the issue list became too long, priorities could be shifted away from other tasks to concentrate on the website. since every change was documented, reporting on website updates became much easier, and going back through the closed issue list provided an easy way to figure out what changes were made and when. the medical library website has undergone several large scale improvement projects quickly and with great success. one of the best things about using these tools is that they are flexible enough to adjust to a variety of workflows, so even if the workflow described in this article would not fit your library, git and gitlab should be able to work with whatever workflow meets your library’s needs. about the authors keith engwall is the web and technologies librarian for the ouwb medical library and is the webmaster for the library website. mitchell roe served as a technology specialist for the ouwb medical library and was instrumental in the development of the environment and workflow detailed in this article. he currently serves as a systems administrator for oakland university technology services. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – oprm: challenges to including open peer review in open access repositories mission editorial committee process and structure code4lib issue 35, 2017-01-30 oprm: challenges to including open peer review in open access repositories the peer review system is the norm for many publications. it involves an editor and several experts in the field providing comments for a submitted article. the reviewer remains anonymous to the author, with only the editor knowing the reviewer´s identity. this model is now being challenged and open peer review (opr) models are viewed as the new frontier of the review process. opr is a term that encompasses diverse variations in the traditional review process. examples of this are modifications in the way in which authors and reviewers are aware of each other’s identity (open identities), the visibility of the reviews carried out (open reviews) or the opening up of the review to the academic community (open participation). we present the project for the implementation of an open peer review module in two major spanish repositories, digital.csic and e-ieo, together with some promising initial results and challenges in the take-up process. the opr module, designed for integration with dspace repositories, enables any scholar to provide a qualitative and quantitative evaluation of any research object hosted in these repositories. by pandelis perakakis, agnes ponsati, isabel bernal, carles sierra, nardine osman, concha mosquera-de-arancibia, and emilio lorenzo background the peer review system is the norm for many publications as it is an essential element in the avoidance of errors in scientific literature and therefore in the improvement in its quality. it involves an editor and several experts in the field providing comments for a submitted article. one of its characteristics is that the reviewer remains anonymous to the author, with only the editor knowing the reviewer´s identity. this model is now being challenged and open peer review (opr) models are viewed as necessary in improving some of the problems in the existing system. in the opr model, openness and transparency are two aspects that are considered necessary to address the issue of biased or non-expert opinions, which is inherent in the anonymous peer review model, characterized by the anonymity of reviews and the unaccountability of reviewers. while there are differences in the definition of the opr concept, “open” here means that the public has free online access to the full text of the reviews, reviews can be submitted by an unlimited number of peers over the lifetime of the content, the identity of authors and peers is disclosed during the entire peer review process. the project to build an opr module for open repositories is based on a series of premises: the need to incorporate quantitative assessment of the hosted research items that will facilitate the process of selecting the most relevant and distinguished content of a repository. common currently available metrics, such as number of visits and downloads, do not necessarily reflect the quality of a research work the need to assess and review content (grey literature, research results in earlier stages, etc.) related to research but removed from the customary flows of scientific publication, as well as enabling post-publication peer-review the opportunity to connect the review model (linked to content ratings) to an author reputation model (sabater-mir and sierra, 2005) with the aim of offering an incentive to researchers to undertake high-quality assessments. importantly, our open peer review module includes an authors and reviewers reputation system based on the assessment of reviews themselves by other peer reviewers. this allows a sophisticated scaling of the importance of each review on the overall assessment of a research work, based on the reputation of the reviewer to address these issues, we developed an open peer review module (oprm) to be installed on existing open access repositories, digital.csic and e-ieo [1], and offered as an overlay service. any digital research work hosted in a compliant repository can then be evaluated by an unlimited number of peers who offer not only a qualitative assessment in the form of text, but also quantitative measures that are used to build the reputation of the research work and its authors. like many other innovations or emerging processes, and opr should be described as such, the development of the module faced many challenges with regard to its inception, development, implementation, and adoption. these included complex decisions in the aspects relating to the opening of the model to researchers from other institutions and the consequent complexity in their identification, the transparency of the ratings issued, the configuration of the invitations model, attempting to strike a balance between the opening of the model and the restriction of the activity of trolls, the role of administrators of the repositories, etc. always important, the technical challenges involved introducing new workflows into functionally complex repositories designed to be minimally intrusive and to take maximum advantage of the already existing code of the dspace software. the code developed [2] is based on standard functionalities of dspace, aligning it therefore with the software’s constant evolution. equally challenging was the translation of the reputation model to the calculation algorithms and corresponding data structures. the convergence of the algorithms (reputations are calculated cyclically and reiteratively) and its impact on the calculation times and, consequently, on the performance of repositories, required constant adjustments and measurement for its control. finally, of no less importance, the adoption of the new opr paradigm by users is complex. we believe that the future frontiers of opr will be centred on how to attract authors and reviewers while confidence is being built in the new reputation system, as well as the search for incentives that encourage researchers to undertake public and transparent reviews. characteristics of the reputation model the reputation assessment model is based on peers evaluating (quantitatively, in addition to qualitatively) each other’s research works as well as each other’s reviews. the latter allows for a sophisticated scaling of the importance of each review on the overall assessment of a research work, based on the reputation of the reviewer. we note that our model assumes that evaluations may be done on a number of dimensions (e.g. originality, technical soundness, predicted impact, etc.), however, an ‘overall quality’ dimension is used for computing the general reputation of the research work. in brief, the model quantifies a reputation (osman, provetti, riggi, and sierra, 2014) for works (which can be any research object hosted by the repository), authors, reviewers, and reviews. the reputation of an article is the weighted aggregation of the reviews it receives, where the weight depends on the reputation of the reviewer (discussed below). a single metric is provided for each evaluation dimension: overall quality, expected impact in the field, expected impact for society, etc. a scholar’s reputation as an author is an aggregation of the reputation of their papers. again, this reputation is computed for each dimension separately. note that the impact of the reputation of a particular work in the general reputation of an author is inversely proportional to the number of authors of the work. the reputation of a reviewer is essentially a weighted aggregation of the comments about his or her reviews by other reviewers who evaluated the same research works. the weight in this case is the reputation of reviewers who offer an opinion. finally, the reputation of a review is similar to the one for articles. it is a weighted aggregation of the ratings received in comments, where the weight is the reputation as reviewer of the researcher who wrote the comment. the module allows an unlimited number of expert reviewers to provide an evaluation for any research work, either preprint or already published. reviewers can either be invited through the system (for example following a request by an author or editor) or can volunteer to review any object of the repository. in both cases, reviewers receive the review request details by email and are asked to offer their review reports within a specified deadline. the review and reviewer credentials are submitted to the system administrator for inspection and verification. after this process is completed, the review is linked to the original research object and becomes openly accessible. implementation details compared to other design options, we should note that the development was selected on the basis of the existence of advanced author models in both repositories. these models, for the two repositories under consideration, are based on the authority-control functionality [3] of dspace. the e-ieo repository, with dspace version 5 and xmlui interface, incorporates an authors-extension [4] based on code from atmire. the repository digital.cisc, with dspace version 4 and jspui interface, has dspacecris [5] module installed. in our view, basing the developments performed on one of the existing models in dspace for extending author-related functionality, it is necessary to: disambiguate and identify the authors of articles and reviews unequivocally so that it is possible to calculate the reputations of articles by taking into account the reputations of reviewers and authors, show the reputations of authors and reviewers in their personal pages. this is necessary for credit and recognition. the open peer review module is built around a number of components described below: the invitations component has been developed as an extension of the workflow and submission capabilities of dspace. it allows the author to send review requests to selected peers. alternatively, any user can request a token to make a review. the submission-item-interface has been extended to specify the email addresses of the reviewer. the system sends a customized email including a token that grants the reviewer access to the research object and to the reviews’ subsystem, bypassing the login and authorization checking of dspace the reviews component. this is also an extension of the workflow and submission capabilities of dspace. the reviewer accesses the reviews’ subsystem acting with sufficient privileges granted by the token, and the evaluation forms are then presented to him/her, together with relevant terms and conditions regarding the whole review process. the proposed forms can be configured using standard data types when applicable, although an additional schema has been added to accommodate a specific model’s metadata. when the review form is completed, a new object is created in the submission workflow. the submission workflow can assign the review object to the repository administrators, with a single accept/reject/edit metadata step or just deposit the review in a specific collection. the administrator can complete the submission process with any necessary metadata enrichment. following this step, specific background tasks are attached to the process, via consumer events, to perform automatic validation of the metadata, linking reviews and reviewed objects, and calling the reputation submodules to calculate new numeric values (for authors and research objects) and automatically incorporating them into the reviewed object and into the review. the comments component complements the reviews component, allowing comments (also called annotations) to be attached to the reviews. reviewers use it to comment or annotate other reviews of the same research object. comments/annotations are automatically deposited into a specific collection, which contains the submit group anonymous group. this allows annotations by those who do not have a user account (the same principle followed with reviews). annotations are restricted in that they only can be made by those who have created a review on the item, or by any of the authors. the component could be limited, eventually, like the reviews component, to one or more email subdomains, in order to avoid uncontrolled use of the system. the dspace data model has been extended to incorporate relevant metrics as well as the back-and-forth relationships between research objects and their reviews and judgments. specific metadata schemas have been incorporated into both repositories, although an extension of the qualified dublin core metadata scheme could be used, possibly leading to simpler implementations. in order to process information about the reputation of the authors, and make this information persistent, the system uses extensions to the author´s metadata. it is important to note that the module can be used without these extensions, although in this case the consolidation and visualization of the reputation of authors and reviewers is not available. the reputation engine was developed as a separate plugin, allowing easy adaptation to other reputation algorithms and making feasible its implementation and functional adaptation to other repositories. the reputation engine meets the following principles: the reputation of a research object is calculated by aggregating the weighted ratings by reviewers the reputation of research objects will then be used to calculate the reputation of individual authors the reputation of reviews will be calculated by aggregating the evaluations (called judgments) of other reviewers. they are subsequently used to calculate the reputation of individual reviewers. if a person plays more than one role (author, reviewer and/or single user) in the system, the module will estimate a global reputation for this person, combining her/his reputation as an author and reviewer together with the above components, a group of modifications to the repositories were made. these include item view and author’s pages customization with the aim of presenting the different reputation values of objects and agents involved; adjustments in the searching, indexing and filtering subsystems to enable the search for the new types of objects, reviews and comments; and adjustments in the filter system of oai_pmh to avoid exposing the reviews and comments in normal harvesting interfaces. figure 1. item´s view showing reputation values project data the project was funded by the european union, grant id 643410, in the context of the openaire horizon 2020 initiatives [6] that aim at promoting opr and studying its effects in the context of digital infrastructures for open scholarship. its management, coordination and implementation took place between june, 2015, and march, 2016, through the formation of a consortium of six partners, including: the open scholar cic organisation, which acted as the project promotor and coordinator the institutional repository of the spanish national research council (digital.csic) the repository of the spanish oceanographic institute (e-ieo) the artificial intelligence research institute of the spanish national research council (iiia-csic) in catalonia the multidisciplinary laboratory of library and computer sciences (secaba) in granada a dspace registered service provider (arvo consultores) conclusions and future directions open access repositories can play a far more significant role in scholarly communication by integrating an open and transparent evaluation system. this additional functionality can help to address many of the issues related to the peer review system, the current journal-based research reputation system. the reputation model produces novel metrics that directly reflect the perceived quality of a research work by expert peers, as opposed to current available metrics in repositories that only indirectly account for quality through usage statistics. likewise, the reputation model of the agents participating in the process (authors and reviewers) is an element that is different to other review models as it lays the foundations of new author assessment metrics, which also promote, as opposed to the non-disclosed peer model, transparent reviews. despite the short length of time in which this module has been operational, it may be seen that the integration of peer review into repositories promotes open scholarship by enabling a direct, transparent collaboration between authors and reviewers. a simple survey among the participating authors in both pilot projects has produced responses that are consistent with the general view of researchers about open peer-review (callaway, 2016), with support and misgivings expressed with regard to the new processes. open peer review is regarded as a long-awaited repository service, whose primary use is for preprints and other unpublished works. however, it is viewed as limited in terms of its applicability for works that have been already evaluated and published. we would indicate that generally even those researchers who support this new repository service reported issues that can slow down the uptake of this feature. these include finding the time to select works to be reviewed, inviting peers, and commenting on the reviews received. furthermore, it appears that inviting peers to an open evaluation may place authors and reviewers in an uncomfortable situation. we surmise that this aspect may be associated with the still small circle of experts available for invitation and the inevitable issues with regard to co-authorship and competition. finally, researchers indicate that current systems of merit and professional evaluation do not take into consideration (and it is not anticipated that they will do so in the short term) authors’ performance (either as a reputation index or any other process indicator), which is regarded as a major obstacle to the paradigm deployment of open peer review. reviewers feel that the system implemented is a great idea that merits success as currently peer review is not credited in the cvs of researchers at all due to its anonymity. but, in keeping with the above paragraph, researchers will not have time to review and comment on the work of other peers as long as this activity remains beyond the scope of cvs and lacks strong support from the research institutions. naturally, authors and reviewers would like the model, which in its current implementation is limited in those who can review and those who can comment, to be endowed with a greater openness, saying that the service should promote spontaneous discussion between those who wish to send comments. they say that it is demanding, delicate and difficult to make public reviews of sufficient quality and to continue with the dialogue that the review and comments system may require. this leads reviewers to request that the works associated with the review process be introduced into the system that assesses research activity. the two participating institutions, csic and ieo, have encountered difficulties in providing authors and reviewers with explanations (we acknowledge that it is hard to operate) about the invitations process together with the concepts that underpin the reputations and scoring model. similarly, improvements have been suggested. these include the need to connect or link the objects to be reviewed to potential reviewers on the basis of similarities between the subject matter of the pre-print or article and the reviewers’ expertise. a large part of the system’s viability appears to involve making this connection between objects and reviewers in an automatic manner, striving not to exclusively base it on the peer-review invitation model. in terms of the functionality provided by the module, it should be noted that the improvement of the user interface is seen as a priority in order to make it more comprehensible and easy to handle and even to automate the invitation-review-comment process. the two participating repositories already had advanced author functionalities incorporated (authorities models, researchers’ personal pages, etc.) and consequently they viewed the addition of the oprm module as an extra step in the offer of advanced services to researchers and general users. questions may be raised about the model’s viability or usefulness for repositories lacking this functional maturity. both the csic and the ieo feel that it is necessary to design an effective and attractive campaign that reaches out to the wider institutional community in order to consolidate the service as an active one for the repositories in the coming months. without such a campaign, misgivings about the lack of linkage with institutional assessment exercises and a rewards system, limitations associated with an invitation-based module and misunderstandings about the oprm reputation sub-module and what type of open peer review it supports are expected to be potential stumbling blocks. in light of the above, the immediate directions for development of the open peer review module in the two participating repositories are oriented towards facilitating the creation of reviews by authors from connected scientific disciplines, connecting the invitations’ system with the repository subscription service, opening up the invitations system to enable reviews to be made by a wider audience, improving the user interface, explaining and refining the algorithms that calculate reputations, and improving the understanding of reputation ratings. notes [1] csic repository, found at digital.csic.es and spanish oceanographic institute repository, found at http://www.repositorio.ieo.es/e-ieo/ [2] the code is currently publicly available. to download it or see the detailed module specification, visit the project wiki at https://github.com/arvoconsultores/open-peer-review-module/wiki [3] the dspace authority control system is documented in https://wiki.duraspace.org/display/dspace/authority+control+of+metadata+values [4] documented in https://wiki.duraspace.org/display/dspace/author+profiles+xmlui and https://github.com/dspace/dspace/pull/668 [5] originally developed by cineca (http://www.caspur.it/) is an open source extension of dspace that includes cris functionality. documented and available for download at https://wiki.duraspace.org/display/dspacecris/dspace-cris+home [6] https://www.openaire.eu/h2020openaccess/ references sabater-mir, j. and sierra, c. (2005) review on computational trust and reputation models. artificial intelligence review, 24:33–60. osman, n., provetti, a., riggi, v., and sierra, c. (2014) more: merged opinions reputation model. in proceedings of the 12th european workshop on multi-agent systems (eumas 2014). springer. callaway, e. (2016) open peer review finds more takers. nature 539, 343. doi:10.1038/nature.2016.20969 about the authors pandelis perakakis is director of open scholar cic, uk, and researcher at mind, brain and behaviour research centre (cimcyc), universidad de granada, spain agnes ponsati works at the spanish national research council as director of the unit of information resources for research, and manages the library and scientific information resources as a supportive infrastructure for research, dealing with the open access policy and the institutional repository digital.csic. isabel bernal manages digital.csic, institutional repository of the spanish national research council. carles sierra is chancellor of the artificial intelligence research institute, iiia-csic, campus uab, bellaterra, catalonia, spain. nardine osman, is a reseacher at artificial intelligence research institute, iiia-csic, campus uab, bellaterra, catalonia, spain. concha mosquera-de-arancibia, is researcher and editor of the spanish oceanography institute and coordinator of e-ieo repository. emilio lorenzo, is director at arvo consultores, a spanish company of digital repositories solutions. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – islandora for archival access and discovery mission editorial committee process and structure code4lib issue 58, 2023-12-04 islandora for archival access and discovery this article is a case study describing the implementation of islandora 2 to create a public online portal for the discovery, access, and use of archives and special collections materials at the university of nevada, las vegas. the authors will explain how the goal of providing users with a unified point of access across diverse data (including finding aids, digital objects, and agents) led to the selection of islandora 2 and they will discuss the benefits and challenges of using this open source software. they will describe the various steps of implementation, including custom development, migration from contentdm, integration with archivesspace, and developing new skills and workflows to use islandora most effectively. as hindsight always provides additional perspective, the case study will also offer reflection on lessons learned since the launch, insights on open-source repository sustainability, and priorities for future development. by sarah jones, cory lampert, emily lapworth, and seth shaw introduction this article is a case study describing the implementation of islandora 2 [1] to create the university of nevada, las vegas special collections and archives portal, a digital asset management system (dams) that provides a unified public interface for the discovery, access, and use of archives and special collections materials. founded in 1957, the university of las vegas, nevada (unlv) is a public land-grant university, and unlv special collections and archives (sca) is a division within the unlv university libraries. sca documents university history, as well as the history, culture, and environment of las vegas, the southern nevada region, and the global gaming industry. as of 2023, sca is composed of four units: digital collections, public services, technical services, and the oral history research center. prior to the public launch of the sca portal in 2021, users accessed sca material through three separate access points: the unlv libraries catalog (monographs, maps, periodicals), an online database of archival collections (manuscripts, photographs, oral histories, and university archives), and digital collections websites (digitized collection material and digital exhibits). technical services (ts) processes and manages all of sca’s archival holdings, physical and digital. archivesspace is the primary collection management tool, and is used to create finding aids for oral history interviews, and manuscript, photograph, and university archives collections. these finding aids are available for browsing and downloading via the sca portal. digital collections (dc) digitizes and provides online access to select materials via the sca portal, including archival photographs, manuscript materials, newspapers, oral history interview transcripts, and born-digital records. these two units have developed a deep collaboration that came about through selecting, implementing, and using this islandora-based joint digital asset management system that brings several ts and dc workflows together in a centralized repository. these two sca departments work closely with two departments in the library technologies division in order to maintain our local installations of islandora and archivesspace. the library technologies division of unlv libraries includes the web and application development services (wads) department and the library systems department. wads is responsible for application development, front-end web development/design, and user experience. this department includes the lead developer responsible for islandora and archivesspace. the systems department is responsible for back-end server maintenance and administration as well as backup and storage workflows and amazon web services (aws) administration. choosing islandora unlv libraries began digitizing materials on a small scale in the web department, with a larger and more coordinated effort beginning in 2006. over the years, the digitization program grew through grant funding, outsourcing of collection digitization with vendor partners, and with the creation of a department (dc) dedicated to the reformatting of materials in a digitization lab, creation of digital object metadata, and management of a digital asset management system. the first system in use was the oclc-created contentdm system which provided for ingest of images and associated metadata, public display of the objects in templates, and tools for metadata creation and controlled vocabulary management. in 2014, unlv began to focus on large-scale digitization and linked data and began to face issues with workflows, system scalability, and the absence of integrated digital preservation solutions.  unlv libraries leadership team charged a digital asset management system (dams) task force in 2014 to “provide recommendations on the best technology solutions to support the acquisition, preservation, and management of born digital archival content and the continued development and delivery of robust, unique library digital collections.” at the time, unlv libraries was using a locally hosted implementation of contentdm to provide access to digitized items from unlv special collections and archives (sca) –  mainly images, but also pdf documents, audio clips, and video files. in 2014, sca implemented archivesspace for archival collection management and finding aid creation. at this time, researchers were faced with navigating three different websites and four different search boxes in order to find and access unlv sca materials, so a more unified user interface was one of sca’s goals. figure 1. digital collections and archival description system diagram pre-islandora. the three sca websites users encountered were: contentdm (d.library.unlv.edu), the digital collections drupal-based exhibits site (digital.library.unlv.edu), and the main unlv libraries website (library.unlv.edu). the four different search boxes that could be used to find sca materials queried the unlv libraries catalog, a database of collection-level descriptions, the full text of collection finding aids, and contentdm. digital collections used local networked server locations (workspace and read-only vault) to store master copies of files, which were also backed up using amazon web services. sca was also focused on improving its stewardship of born-digital archives, therefore improving unlv libraries’ digital preservation practices was another important goal. the dams task force priorities led to adjacent work that resulted in the creation of a formal digital preservation policy in 2017. in summary, the task force was looking for a dams that would meet all three of its main goals: replace contentdm’s flat file data structure with a linked data model that could provide public access to digital objects and other structured data provide a unified search interface for digital objects, archival description (finding aids), and agent records (people and organizations connected to our collections) provide functionality for born-digital archives and digital preservation in 2015, the dams task force tested two hosted solutions, preservica and archives direct (a service combining duracloud and archivematica). while these options offered robust digital preservation and born-digital processing capabilities, they fell far short of matching the user interface and metadata management capabilities that contentdm provided, which sca did not want to lose. at the time of testing these options also did not seem to scale well in handling the large amount of digital material that sca needed to ingest and manage. the vendor costs of hosting a large (and growing) amount of data was another concern. in 2017, a dedicated application developer was hired to focus on sca technology, which allowed the task force to more seriously explore open source systems. four different potential directions for development were identified: a samvera repository next to an archivesspace public user interface unified by common theming, linking, and a common search interface an islandora repository similarly integrated with archivesspace as described above a full drupal integration using islandora and pulling in archivesspace data via the api an in-house built interface harvesting from both fedora and archivesspace apis. the decision was based on the dams task force’s answers to three questions: should the interfaces be tightly or loosely coupled? should we focus development on an in-house solution or contributing to and benefiting from a community-based solution? how important was leveraging existing technical expertise (php+drupal) or were we willing to adopt new ones (ruby)? the task force members choose a tightly integrated interface (instead of linking between distinct systems), a community-based project, and leveraging the developers’ existing php and drupal experiences. islandora is an open-source eco-system for the display and management of digital cultural heritage materials, with an active community of supporters and users, with drupal as its primary interface. figure 2. planned architecture where drupal now serves as the unified interface for special collections web content. dashed lines indicate planned future projects. source: seth shaw, “islandora @ the university of nevada, las vegas“, islandoracon, prince edward island, can, 2022. according to the product website, “islandora is an extensible, modular, open source digital repository ecosystem focused on collaborative authorship, management, display, and preservation of digital content at scale. islandora adheres to widely adopted best practices and open standards and frameworks used in information practice.” the product is augmented by community-developed modules that add functionality and are shared back into the community. islandora users are supported by the islandora foundation, which acts as a governance structure setting priorities, and building consensus on issues of architecture and feature development.  when unlv made this decision in 2017, the islandora community was working on a major change from islandora 7 to the next version (confusingly) islandora 2.0, which was then code-named islandora claw. [1] islandora 7 was based on drupal 7 and fedora 3, both of which were rapidly approaching the end of their official support windows, and which used drastically different internal structures than their new versions requiring an updated conceptualization of how islandora works and a complete rewrite of the software. in light of these circumstances, unlv libraries decided to help the islandora claw project complete the new version rather than initially invest in islandora 7, only to migrate away shortly thereafter. the drastic change in architecture is one of the reasons the new version was named islandora 2 or modern islandora. unlv’s contract with oclc to host contentdm had an annual renewal date, which worked as a convenient deadline for development. based on the answers to the three guiding questions above, and with insight from the development community’s timelines on product releases, unlv made the decision to select islandora and aim for a launch timed with the end of the oclc contract. this signaled a new phase of work implementing islandora, followed by migration of all unlv special collections data into the new system. implementing islandora implementing this new architecture consisted of five major tasks: getting islandora claw to an initial release, integrating the new islandora into our existing drupal-based online exhibits site, the archivesspace integration, designing the user interface, and migrating existing and new collections. much of this work was done in parallel with changes in one aspect impacting the others. getting an initial islandora claw release unlv began working with claw in december of 2017. initial work focused on the community’s deployment tooling so we would have a stable development platform for both contributing code back to the community and testing our local code. as commonly happens with pre-release software, there were multiple large design shifts that occurred over the course of 2018 which changed how derivatives and indexing actions are triggered (january) [2], inverting the relational direction of metadata records and their associated files (june) [3],  and how binary files flow between drupal and fedora (august) [4].  the community continued active development of new features and remediating bugs as they were discovered. then, in early 2019 the islandora community performed a series of targeted two-week efforts, called sprints, to document and test the new system before making its initial public release in june 2019 [5]. while this was a significant milestone, developers continued to add new features and fix bugs that were discovered as we continued building our new repository site and the number of other institutions using it increased. integrating islandora with the existing exhibits site unlv digital collections had an existing drupal-based exhibits site where content specialists could describe the digitized content and give historical context. this site included a locally developed integration with a self-hosted contentdm instance, which is now unavailable due to contentdm’s discontinued option to locally host instances. these interfaces were popular with researchers and users beginning their search process and the team asked the developer to investigate methods to preserve the user experience from the former repository using the new drupal structures leveraged by the islandora community. integrating digital objects directly into the existing drupal exhibits site allowed us to bring these components together again. unfortunately, automated islandora deployments provided by the community assume a brand new drupal instance which prevents us from using them as part of our workflows [6]. these automated deployments were useful to the community because islandora adopted a microservices approach to indexing and derivatives which meant there were several distinct applications that needed to be installed and properly configured to work together [7]. eventually the developer resorted to manually installing and updating the necessary components. theming proved to be another challenge for integrating with the existing site as several structure and styling decisions had been made with the existing content in mind which did not display well with the new data structures and content. the design and styling of the new digital object, agent, subject, and search pages went through several iterations with the stakeholders to find an acceptable solution (see the “designing the user interface” section below). archivesspace integration when we began looking at integrating archivesspace with our new islandora repository, we discovered there was an existing drupal 7 integration in use by the american academy in rome [8]]. we reached out to the developer and began an early collaboration to re-implement an integration for drupal 8+ [9]. the new integration would need to first define the content models in drupal to receive the archivesspace data, establish the synchronization mechanism, and establish basic data views [10]. while most archivesspace fields are relatively simple text fields, some were more complex than the field types provided by drupal: dates, extents, and physical instances [11]. for example, the drupal provided date fields presumed iso dates, however archivesspace dates included properties for certainty, types (e.g. bulk or inclusive), calendar, and a string-based expression.  archivesspace also included a concept of relationship types, where an agent can be linked to a record using a list of possible relationships. replicating this in drupal natively would require creating a separate field for every relationship type, which seemed untenable. we decided to create a new field type to support this type of linked agent field. it was added to a separate drupal module, controlled access terms, as it seemed like it might be useful to the broader islandora community outside of the archivesspace module [12]. the controlled access terms module also included a new drupal field type and associated plugins for representing dates using the library of congress proposed extension to iso 8601, extended date time format (edtf) [13] which we used for digital object dates instead of the archivesspace-specific date field type. controlled access terms was later transferred to the islandora foundation’s official github organization for community management. early on we decided to leverage drupal’s new migrate api which was intended for ingesting and updating content and provided a plugin framework for extension. we expected this would limit technical debt by leveraging existing apis rather than coding the migration code wholly from scratch. two critical pieces we needed were an authentication mechanism to login and maintain a session with archivesspace and a migrate api “source” plugin which could use the session to iterate over the archivesspace api endpoints and expose the api fields drupal would pull from. further, because of the more complex nature of the archivesspace data model, we needed additional plugins to assist in transforming the source data into structures drupal could use to populate its own records. the final step of this process was to create some default migration configurations which would map the archivesspace data to the target models. these consist of several configuration files which can be adapted by local site implementers. finally, after modeling the data and using the drupal migrate api, we also set up some default data views. drupal has a powerful query and display builder called views. these allow us to select which data related to the current item in question should be displayed and how. this allows the site builders a great deal of flexibility in deciding how to display their data. for example, when a user views a drupal page corresponding to an archivesspace resource record a view can display all the child archival objects with whichever fields are desired and provide links to archival objects’ pages. figure 3. screenshot of an archivesspace resource record after being loaded into drupal figure 4. a view of a collection’s series powered by a drupal view. once we were able to load our archivesspace data into a drupal site we then considered how best to link archival objects with their digital representations. archivesspace can create digital object records, but does not have tooling around loading and transferring the files themselves. we decided that the simplest workflow would be to load new digital objects directly into the new islandora and maintain the links to archival objects in islandora instead of creating archivesspace digital object records. all we needed was to include the archival object reference id in our loading spreadsheets and have drupal perform a lookup on that field to create the link. once again, the drupal views functionality allowed us to control how we wanted related items to appear on each item’s page with a great deal of flexibility and control. figure 5. screenshot of an archival object record with a thumbnail of the digital object record in a sidebar. content migration once our local instance of islandora was stable, the digital collections department worked with the application developer to migrate the digital objects in contentdm. the metadata was exported from contentdm and remediated by the metadata librarian to conform to a new, consistent application profile. since only access copies of digital images and other files were stored in contentdm, the digital collections librarian created spreadsheets to match the metadata with the master files that were saved on local file servers. the metadata and file spreadsheets were sent to the developer, who used the migrate_islandora_csv module combined with custom code to ingest the content onto the development server for testing [14]. after any issues were resolved, the content was ingested onto the production servers. figure 6.  migration from contentdm to islandora. source: seth shaw, “islandora @ the university of nevada, las vegas“, islandoracon, prince edward island, ca, 2022. each digital collection in contentdm had its own set of metadata fields; for example, there was a field for “cuisine” in the menus digital collection. in islandora, all digital objects would have the same metadata fields, so the metadata librarian had to review all of the existing metadata in contentdm and create a new unified metadata application profile, then remediate the metadata from each contentdm collection to conform to the new profile. he also converted geospatial, genre, subject, and agent terms to uris. figure 7. digital object metadata remediation process. source: seth shaw, “islandora @ the university of nevada, las vegas“, islandoracon, prince edward island, ca, 2022. with the intention that born-digital items would eventually also be added to the portal, and that digital preservation functionality would be built out, sca added several new metadata fields for digital objects, and also at the file/media level. the “digital provenance” field uses a controlled set of statement to distinguish between different types of digital objects, including:  “original archival records created digitally” (born-digital archives) “digitized materials: physical originals can be viewed in special collections and archives reading room” “digitized materials: physical originals are not available for viewing because of fragility or obsolescence” (such as digital video files created from vhs tapes) “digitized materials: physical originals are not held by unlv special collections and archives” (“scan-and-return” items, i.e. digital surrogates are part of unlv’s collections but physical originals were retained by the donor)  a “digital processing note” field uses another controlled set of statements to document if the digital surrogate was edited or redacted, and if it was transcribed manually or using ocr. at the file level, portland common data model use extension uris are used to identify the different types/versions of files, such as original, service, thumbnail, etc. sca created a matrix that cross-references the use extension with the information in the digital provenance and digital processing note fields to assign different preservation levels at the file level, for example:  original born-digital files are “high priority”  “priority” is assigned to the original files of digital surrogates where the original physical item is the ultimate “preservation copy” “low priority” is assigned to automatically generated derivatives like thumbnails sca also decided to assign archival resource keys (arks) to all digital objects to be used as persistent identifiers and urls. at first, arks were not assigned to the children of complex/compound digital objects, they were only assigned to the parent object. however, child digital objects are their own drupal nodes and show up independently in the search results, so sca decided to also assign arks to these child objects for consistency. unlv uses california digital library’s ezid service to create and manage arks. the developer wrote a script to assign arks to digital objects and update information in ezid in bulk. later on the developer further automated ark creation by using drupal insert “hooks” to trigger ark minting on item ingest. designing the user interface one of the key successes of this project was the centralization of search across a wide variety of content types with one search box. with so many different types of content in unlv’s islandora site (digital exhibit pages, archival description, agent records, and digital objects), the team spent over a year discussing the best way to display everything to users. stakeholders from sca, including public services and curatorial staff, worked with wads to create personas, inventory all content, do card sorting exercises, create mockups, conduct internal testing, and gather feedback from internal stakeholders. the personas sca developed for users included an amateur historian, phd candidate, undergraduate student, filmmaker, media relations consultant, art historian, and video game designer. two sca librarians also conducted in depth interviews with nine different types of users of unlv’s digital collections, which also helped to inform the ui design. the team aimed to make the user interface (ui) and labels intuitive enough for first time users, but also wanted to include features and information that would be useful to expert researchers. there is one main search bar for the site, and a set of filters to refine search results. the filter at the top of the sidebar allows users to refine by content type: digital objects: digital files and their metadata. archival collections: collection-level descriptions archival components: descriptions of series, sub-series, files, and items within archival collections people: descriptions of people who are creators, contributors, or subjects of archival collections or digital objects organizations: descriptions of organizations that are creators, contributors, or subjects of archival collections or digital objects families: descriptions of families that are creators, contributors, or subjects of archival collections or digital objects subjects: subject terms related to archival collections or digital objects geographic locations: location terms related to archival collections or digital objects webpages: webpages of digital exhibits blog entries: blog entries related to digital exhibits each individual instance of each content type is its own webpage; with over 740,000 individual pages indexed, the team created a total of eleven different filters to help users refine their search results. the team also deliberated carefully over which metadata fields and thumbnails to display in the search results for each different content type. figure 8. search results page. the team designed a webpage template for each content type. archival collection pages show collection-level description, a search box to search all the digital objects and archival components within that specific collection, and an inventory of the archival components that are in the topmost level of the archival hierarchy. figure 9. archival collection page. archival component pages show component-level description, as well as the basic information needed to request the materials in the sca reading room. the component’s hierarchical organization within the collection is shown in the breadcrumbs at the top of the page. the search box to search within the collection is also present. if the component is digitized, a link to the digital object is displayed as a thumbnail. if the component has child components or multiple child digital objects, those are displayed in an inventory under the “search within this collection” box.  figure 10. archival component page. unlv’s digital objects are mainly images and text documents, but also some audio and video files. there are many complex digital objects in the sca portal, meaning multiple digital items grouped together and described in the aggregate. a complex object could be two images of the front and back of a photograph or letter, or it could be an entire folder of photos or documents that has been digitized. there are links between parent and child records in breadcrumbs at the top of the page, in the metadata, and also below images. digital objects are also linked to archival components when possible, and this hierarchy is part of the breadcrumbs. the most important metadata fields appear next to the digital file, with other fields placed below into different tabs. figure 11. user interface for the parent record of a complex digital object consisting of images. adjusting to islandora unlv’s islandora dams was named the unlv special collections and archives portal and publicly launched in august 2021. sca had achieved its goals of replacing contentdm and creating a unified search interface to improve the online user experience. motivated by the end of unlv’s contentdm contract with oclc, and a development timeline that was stretching longer than anticipated, the portal was launched while it was still missing some of sca’s desired features, including those related to digital preservation. below, the authors discuss the reactions of users and staff to the new system, as well as challenges faced, adjustments made, and lessons learned. labor disruptions less than a year after the public launch of the portal, unlv experienced a common challenge for libraries working with in-house technical development projects. they lost their lead developer to a better employment offer. with the covid pandemic greatly disrupting labor across the country and the advent of more fully remote positions for developers, this was a challenging time for many organizations. unlv strove to fill the gaps by asking existing staff to learn new skills, relying more on the community for troubleshooting, and through the significant work of a contract developer hired to partially fill the gap (with 20 hours/week devoted to helping maintain the system). it is hoped that with the hire of a permanent developer, additional progress can be made on portal priorities, but for the most part the portal has been in a holding pattern/maintenance-only mode for over a year.  documentation and extensive cross-training were a lower priority than launching a functional product, resulting in only minimal documentation being created. additionally, the system itself changed frequently during its development (as detailed above) making it impractical to create extensive documentation and invest in cross-training before the portal was stable. while the original developer created basic documentation for the most important aspects of the portal, a top priority for the new developer will be to update and enhance local documentation alongside conducting an audit of the portal to identify and prioritize upgrades, fixes, and improvements necessary for the security, stability, and sustainability of the system. since the original developer left, sca and technologies staff have learned more technical details by necessity during troubleshooting. a new lead developer will fill in the rest of the picture where the gaps in knowledge currently exist. unlv’s approach to adopting islandora 2 was driven by local necessity in terms of timing and also by ambitious goals for features. reflecting on this undertaking, the benefit it offered the developer was the chance to contribute to the community codebase and participate in the decision-making involved with the redesign of islandora. now, institutions that are interested in adopting islandora 2 have the benefit of a stable version to build from, or working with a vendor experienced in islandora 2, which reduces the amount of in-house development and support needed, compared to the scope of the project that unlv set for itself. user feedback the web content and user experience specialist led virtual usability testing of the new portal with nine participants and wrote a report of the findings in spring 2022. the team also set up an online form for user feedback that is linked to from the top menu of the sca portal. the most often cited dislikes of the portal focused on difficulty understanding labels (definitions of what the filters would return) and difficulty understanding oral history content (how to access items by a narrator, access a transcript, or find out if an oral history had restrictions). there was feedback that one search box was preferable to the previous system and that the hierarchies were particularly useful as context. overall, the user feedback and usability test results confirmed that unlv had successfully launched a repository that not only met our initial launch goal of a minimally functional project, but exceeded users expectations in ease of use, speed, and the availability of content filters to find items. while the internal team was very much aware of a long list of features that were still pending development, they were encouraged by these strong results and decided to go forward with dismantling the previous repository and making a full switch to the new portal. metadata management tools digital collections (dc) metadata workflows were altered significantly by the migration from contentdm to islandora. dc initially planned to develop an application that replicated the functionality of the contentdm project client for the sca portal. the project client software had a user-friendly gui that made it possible for student workers to create metadata in a spreadsheet or item-level view, choose terms from locally managed lists of controlled vocabulary terms, and ingest digital objects in bulk. it also allowed staff to review metadata quality and make edits in bulk before the digital objects were published publicly online. however, due to lack of resources, this project kept getting pushed back and was eventually put on hold indefinitely. in the meantime, the team experimented with drupal views for bulk metadata editing. mostly this has also been put on hold as the department has adjusted workflows to take advantage of the community supported tool islandora workbench. islandora workbench is a “command-line tool that allows creation, updating, and deletion of islandora content from csv data” [15]. digital collections staff learned to use this tool to ingest new digital objects, edit metadata, and delete and replace files (media) in bulk. there was a slight learning curve setting it up and learning how to edit the configuration files and format the csv files, but within a couple months of using it regularly, dc librarians felt comfortable with this tool. however, dc decided that workbench is a bit too complicated to train student workers to use. instead, the department adjusted its workflows so that librarians are creating most of the metadata in bulk in csv files and ingesting via workbench, and students enhance metadata at the item level in the web interface (for example, adding subject terms). this process is rather slow and has made metadata enhancement project timelines longer than in the previous system as students cannot work on a batch of items at one time. some of the strengths of workbench are that it is widely used, actively maintained, and updated often; however, that can turn into a challenge when a new update causes a new error. dc librarians rely on the islandora community and wads staff to help troubleshoot and solve these errors, which occur every few months. archivesspace integration (part two) users and staff are generally happy with the unified interface that includes archivesspace data in the sca portal, but several adjustments needed to be made after launch to both the user experience and staff workflows. once a week, an automated script updates the sca portal with any additions, revisions, and deletions made in archivesspace. updates can also be refreshed “on demand” if there is an urgent need. initially, the sync was only adding new archival resources and components, but any edits to existing components or removal of components were not reflected. this led to multiple versions of the same components displaying at once and changes not being reflected to the general public until the problem was resolved by library technologies. each archival component is its own webpage in the portal, which can require many clicks to navigate the finding aid for a single collection. sca decided to also provide access to finding aids in pdf format, which is how they were previously available to users. the pdfs are generated in archivesspace and available for users to download in the portal on the archival collection page. however, technical services (ts) must manually create or refresh the pdfs using the portal staff interface.  dc and ts also created workflows to link digital objects and the archival components they are the same as or part of. dc uses a view to export a csv file of the archival components of a collection from the portal. this spreadsheet includes description fields such as title, date, drupal node id, and archivesspace reference id. dc reuses the archival description as the basis for the metadata for digitized materials, and adds the archival component node id to the metadata to create links between the two in the portal. ts must separately add digital object arks into archivesspace so that digital object links are also available in the pdf version of the finding aid. after digital objects are ingested into the sca portal, the digital collections librarian sends a spreadsheet to technical services of the digital objects ids, their corresponding arks, and the archivesspace reference ids of the archival components that the digital objects are the same as or part of. technical services uses the “digital object bulk import” spreadsheet feature in archivesspace to simultaneously create and link digital objects to existing archival objects. once this task is completed, the pdf is regenerated and links appear for users to immediately go from pdf archival inventory to the corresponding digital object. subject headings and agent records are also synced from archivesspace to the portal. merging the controlled vocabularies from archivesspace and contentdm has been a challenge for sca. different controlled vocabularies, subject application styles, and different teams creating subject terms led to significant duplication. while some work to merge duplicate terms has been completed, there are over 20,000 terms in sca’s local controlled vocabularies, so further work has been postponed until a new full-time lead developer is hired and can help automate and complete these edits in bulk. in the meantime, sca has adjusted workflows to avoid duplication so that all agent records are created only in archivesspace and synced to the portal every hour. missing functionality there are some development priorities that were on the initial functional requirements list for the system that were developed prior to launch. while the metadata functionality evolved with the community and these tasks have been partially addressed with workbench, some needed modules are not yet prioritized in the community or unlv has lacked the development resources to implement community work. some of the main areas that require significant local development work include: advanced search functionality automating text extraction from pdfs to an indexed metadata field to improve full text search for researchers for textual digital objects and newspapers [16] a method to restrict content (this would enable the portal to be used for born-digital content and embargoed or sensitive collections) [17] better quality control functionality using drupal views (to aid in metadata enhancement) integrated digital preservation workflows (such as fixity checking) ultimately, the dams replacement project was a success. we have a new linked data repository with a unified interface that simplifies the access to multiple types of special collections content. most users find it easy and satisfying to use and approve of the design philosophy. at this point, a new phase of refining and enhancing the portal would provide an opportunity to address some of the missing functionality and to meet the third goal of addressing born-digital access and preservation. reflection at various points in this repository development work, there have been points where the team has expressed uncertainty about the stability and security of selecting an open-source, community-led software product. following the resignation of the lead developer, there was fear about support needs and conversations about rolling back the project, looking for hosted support, or returning to a vendor product. but, with time and more experience, many of these anxieties have transformed into actionable projects and manageable strategic priorities. the advantage of encountering challenges is that there is a rich environment of learning and growth. some of the key lessons learned in the project include how expectations have been modified, and how to manage them going forward as the team looks towards further development. most technical projects are managed with a goal, allocated resources, and a deadline. but working with open-source software, especially in a solo developer environment, meant that project managers, supervisors, and colleagues had to adapt to the realities of working in sometimes ambiguous or uncertain circumstances. for instance, a module may exist in the community, but when explored further it may be found that documentation is minimal, that the module requires customization to work in the local context, or that it simply is not mature enough to work reliably. in other cases, a simple upgrade or security issue that seemed straightforward ended up involving more staff members and time than initially planned. a key lesson was to understand that timeline forecasting is difficult and that using techniques like agile project management are often more suitable than strict deliverables and hard deadlines. the team also had to learn new ways of working and adapt expectations of what “technical support” looks like. there is a significant adjustment for staff when moving from a vendor supported product to an open-source community. no longer can staff expect that if they uncover an error that a service call will be made and the issue resolved in an expected time-frame. rather, the team had to build a new workflow that included how to report a portal issue, who the issues should go to, what roles each member of the team played in troubleshooting, how to document errors and troubleshooting work, and how to engage in the community if outside support was required. after several months of refinement, this system is now in place and has become more comfortable, but there can still be discomfort or a feeling of insecurity for staff wondering, “what if the portal goes down” when they are working at a public service point or in critical moments with patrons or donors.  expectations have also changed in areas of decision-making. philosophically, unlv supports the open-source model; but there are times when it can be too slow or the community priorities are out of alignment with the organizational needs. we have learned that we need to find a middle road between completely vendor-provided products and services, and completely custom, built-to-order open source software development. we take the approach to try and reuse community work when possible. we see the value in working in uniform ways with the islandora core code and modules. but some workflows may need the boost of dedicated development resources to complete development of a new function or service and the team is looking for ways to supplement our internal resources with project work and vendor solutions (such as a site audit to improve our documentation, or a vendor contract to help implement restricted materials functionality).  in the current environment, the team also noticed a shift in the power dynamic from an administrator or director setting the work plan and then the project manager setting milestones and the functional experts reporting the progress. instead, we now work in a complex environment with shifting power dynamics encompassing the open source community, workflow owners who may or may not be able to optimally complete their work in the portal, our users/stakeholders, staff who prioritize technical development work (but cannot do it themselves), the technology experts who have the technical expertise to solve problems, the managers responsible for evaluating job performance, and administrators – who may or may not understand the true measures of the system’s success or value. this all means that more communication is required and every person on the team needs to advocate for the system. future goals the unlv team has had a chance over the last twelve months to look back and reflect on the selection, migration, and use of the sca portal and despite the aforementioned challenges, the future looks bright. currently our future tasks include: on-boarding a new lead developer and delving more deeply into how islandora skills are acquired and sustained designating a product owner to liaison between the stakeholders and the developer and who will manage communications and priorities building relationships in the community and possibly hosting an event to bring potential collaborators to us locally leveraging vendor relationships to fill gaps in large development projects continuing to nurture collaborative relationships in the organization making our digital library more inclusive of content and merging the provision of agent information and archival information into one unified interface for our users has been a challenging but rewarding project. it is certain that new technical issues and challenges will arise as the team enters the second phase of repository development, but with the experience gained since launch, the support of the community, and the dedication to the principles that drove the initial project, the team can continue to build an innovation solution to special collections discovery and ensure a sustainable path forward. references and notes [1] islandora 2 has also been known as islandora 2.x, islandora 8, and modern islandora. it was also known by its pre-release codename, islandora claw. for more information, see islandora’s versioning policy at https://islandora.github.io/documentation/technical-documentation/versioning/“ [2] islandora commit: using context instead of rules https://github.com/islandora/islandora/commit/c807bab123421b70c84d4d6127e923b8018820e9 [3] islandora commit: content modeling overhaul https://github.com/islandora/islandora/commit/a1987aecb747b0b297176fc4d34afdc22cbddb4d [4] islandora commit: flysystem https://github.com/islandora/islandora/commit/cb8bb07238d1822370aca73e7011fdea9235c501 [5] islandora release 1.0.0 https://github.com/islandora/islandora/releases/tag/1.0.0 [6] at the time it was limited to an ansible playbook although the community has since added docker-based deployments. [7] see “architecture diagram” in the islandora documentation: https://islandora.github.io/documentation/technical-documentation/diagram/ [8] lavinia ciuffa, “from processing to public service: the digital humanities center at the american academy in rome”, visual resources association bulletin, vol 44, no. 1 (2017). https://online.vraweb.org/index.php/vrab/article/view/40 [9] unfortunately, the collaboration only lasted a few months before our partner turned their attention to other things. [10] some components of this integration are discussed in more detail in an archivesspace hosted webinar available online: seth shaw, “integrating archivesspace with drupal and islandora at university of nevada, las vegas libraries”, archivesspace webinar series, july 15, 2020. https://youtu.be/qic3svcbauc [11] archivesspace complex fields https://git.drupalcode.org/project/archivesspace/-/tree/8.x-1.x/src/plugin/field/fieldtype [12] islandora controlled access terms module https://github.com/islandora/controlled_access_terms [13] the library of congress proposed the extended date/time format (edtf) specification in 2012 on which the unlv module was based. edtf received a new version on february 4th, 2019 and the unlv module, now transferred to the islandora organization, was updated accordingly. see https://www.loc.gov/standards/datetime/. [14] islandora migration documentation https://islandora.github.io/documentation/technical-documentation/migrate-csv/ [15] islandora workbench repository https://github.com/mjordan/islandora_workbench  [16] newspapers are currently not indexed and only available as a list of titles and issues: https://special.library.unlv.edu/collections/newspapers  [17] the early version of the repository did have support for restricted content based on a module the islandora community presumed would work. however, as the size of the repository grew we discovered performance degraded significantly, so the stakeholders decided to table that requirement until a more performant solution for restricted content could be found. see seth shaw, “islandora: performance testing & content access control”, islandora open meeting, 2021-02-23. recording available at https://youtu.be/tkqidyjsvdo with slides available at https://seth-shaw-unlv.github.io/files/2021-02-23_islandora_open_meeting_performance_testing_and_content_access_control.pdf about the authors sarah jones is head of special collections & archives technical services at the university of nevada, las vegas where she leads a team in the preservation, description and access, and overall management of archival holdings in both physical and digital formats. cory lampert is professor and head of digital collections at the university of nevada, las vegas where she is responsible for leading a team responsible for digitization, metadata and linked data creation, and digital asset management.  emily lapworth is associate professor and digital collections librarian at the university of nevada, las vegas. her work focuses on digitization, digital asset management and preservation, and increasing online access to archives and special collections materials. seth shaw was the software engineer for special collections and archives at the university of nevada, las vegas where he was responsible for the maintenance and development of software applications used in the special collections and archives; primarily islandora and archivesspace. he is now a digital library software engineer at arizona state university libraries.   subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – fair principles for library, archive and museum collections: a proposal for standards for reusable collections mission editorial committee process and structure code4lib issue 40, 2018-05-04 fair principles for library, archive and museum collections: a proposal for standards for reusable collections many heritage institutions would like their collections to be open and reusable but fail to achieve that situation because of organizational, legal and technological barriers. a set of guidelines and best practices is proposed to facilitate the process of making heritage collections reusable. these guidelines are based on the fair principles for scholarly output (fair data principles [2014]), taking into account a number of other recent initiatives for making data findable, accessible, interoperable and reusable. the resulting fair principles for heritage library, archive and museum collections focus on three levels: objects, metadata and metadata records. clarifications and examples of these proposed principles are presented, as well as recommendations for the assessment of current situations and implementations of the principles. by lukas koster, saskia woutersen-windhouwer introduction most libraries, archives, museums (lam) and other heritage institutions manage a variety of collections. these collections can be physical and digital book and journal collections, repositories of publications, physical artifacts collections and digital representations of these, archives of all kinds of physical and digital material, etc. the collections and objects are described in a large variety of information systems and databases, using all kinds of digital object and metadata formats, access protocols and storage facilities. more and more people and organizations want to be able to access and reuse both digital objects and the (meta)data describing the physical and digital objects, not only individually but also in bulk. some lam institutions already comply with these wishes. examples are: the british library free data services (british library free data services [2018]), the ghent university library open data services (universiteitsbibliotheek gent open data [updated 2018]) and the rijksmuseum api (rijksmuseum api [2018]). in other cases, there may be technical or legal barriers or data services lack clear documentation making discovery and access ambiguous. not all collections apply or provide the universal metadata standards and protocols that are necessary for exchanging or reusing information in a simple way. furthermore, some collections lack essential metadata (like global persistent identifiers), making implementation of universal standards and protocols impossible. and reuse can be complicated, as data or data services may lack information related to data reuse or license restrictions. not only are these circumstances an obstacle for reuse of collection objects and metadata by external parties, but also for internal and external interoperability between various information systems used within lam institutions. to summarize: it is not enough to make collections available through web based end user interfaces and provide download options for individual objects and metadata records. on the contrary: collection data and objects must be findable, accessible, interoperable and reusable for people and software in their entirety and in specific parts.[1] in this article[2] we propose a body of requirements for making lam collections findable, accessible, interoperable, and reusable (fair), as well as a set of recommendations for assessing existing institutional situations and for implementation of the requirements. these requirements and recommendations are based on the fair principles for scholarly output (fair data principles [2014]) and adapted for cultural heritage collections. fair collections are not necessarily open collections. the open knowledge foundation applies a definition of “openness” that adds modification permission and universal participation to the accessibility and reusability criteria, “at no more than a reasonable reproduction cost” (what is open? [2018]), referring to the open definition. the short form of the open definition is “open data and content can be freely used, modified, and shared by anyone for any purpose“. the full definition stresses openness in license (public domain or open license), in access (on the web and free of charge) and formats (without restrictions on its use and processable with at least one free/libre/open-source software tool) (open definition 2.1 2018). accessibility and reusability in the fair sense do not require collections and objects to be freely available, modifiable and shareable with free tools as such. that is why we have chosen to use the term ‘fair collections’ instead of ‘open collections’. some metadata or objects will be copyright protected, have privacy issues or local law issues. local law issues are for instance database rights in the european union[3] or portrait right as a personality right in the netherlands[4]. it can be discussed if open can be used for the metadata and fair is only necessary for the objects. unfortunately, there are still vendors that protect data that is (or should be) in the public domain by a contract. for that reason, metadata cannot always be open access or be freely reused. the same applies to objects that should be, according to the copyright law, in the public domain. even in those cases an owner can, as the owner of the object or metadata, lay down the conditions and determine what a user may or may not do with the data or objects. current initiatives in recent years there have been various local and global collaborative initiatives to develop sets of guidelines and best practices for making objects and associated metadata open and reusable. many of these initiatives focus on open and reusable research datasets. but there are also initiatives that relate to cultural heritage objects and collections. a number of these initiatives are described below. first, we will look at the fair data principles, then focus on a number of other initiatives in comparison to fair and finally we will discuss the respective relevance of all initiatives for reusability of lam collections. fair in 2014 the fair data principles were proposed as “a concise and measureable set of principles” to support “the reuse of scholarly data” (wilkinson et al. 2016): to be findable: f1. (meta)data are assigned a globally unique and eternally persistent identifier. f2. data are described with rich metadata. f3. (meta)data are registered or indexed in a searchable resource. f4. metadata specify the data identifier. to be accessible: a1 (meta)data are retrievable by their identifier using a standardized communications protocol. a1.1 the protocol is open, free, and universally implementable. a1.2 the protocol allows for an authentication and authorization procedure, where necessary. a2 metadata are accessible, even when the data are no longer available. to be interoperable: i1. (meta)data use a formal, accessible, shared, and broadly applicable language for knowledge representation. i2. (meta)data use vocabularies that follow fair principles. i3. (meta)data include qualified references to other (meta)data. to be re-usable: r1. meta(data) have a plurality of accurate and relevant attributes. r1.1. (meta)data are released with a clear and accessible data usage license. r1.2. (meta)data are associated with their provenance. r1.3. (meta)data meet domain-relevant community standards. these principles describe a set of guidelines to make data (meaning ‘scholarly output’, more specifically ‘research datasets’) and metadata about these datasets findable, accessible, interoperable and reusable (fair). the authors stated that “… the fair principles put specific emphasis on enhancing the ability of machines to automatically find and use the data, in addition to supporting its reuse by individuals” (wilkinson et al. 2016). the fair principles do not require long term sustainable storage and access of datasets, but only long term sustainable access of metadata: “a2 metadata are accessible, even when the data are no longer available.“. the phrasing of the fair principles is somewhat confusing, very probably because of the wish to be concise. some principles (not all) refer to both “data” and “metadata”, which is formulated as “(meta)data”, for example in “f1. (meta)data are assigned a globally unique and persistent identifier”. one principle is self-referencing (“i2. (meta)data use vocabularies that follow fair principles“). also the enumeration of the four sections (f, a, i, r), each containing 3 or 4 principles, is rather uncommon, for example “(f)air” has four individual principles f1, f2, f3, f4, but “(r)eusable” has one main principle r1 and three sub-principles r1.1, r1.2, r1.3. the logic of this subdivision is unclear, because all four reusable principles are guidelines on the same level in their own right. in fact, the authors say so themselves: “the elements of the fair principles are related, but independent and separable.” (wilkinson et al. 2016, p.4). fair annotations the fair principles can be applied to other object types. collocated with the force2017 conference in berlin, a meeting was organized by the annotating all knowledge coalition dedicated to fair annotations (martone 2017). it is an initiative to make scholarly annotations on the web findable, accessible, interoperable and reusable. a list of issues (under construction) is available (at the time of writing) in the form of discussions of all fair principles applied to annotations (annotations and fair [2018]). the list raises several questions, for instance how to link annotations to specific fragments of a text, what a retrieval protocol for annotations should look like, etc. parthenos parthenos is a consortium of national and international research and research infrastructure organizations, such as inria, knaw, clarin, dariah. “parthenos aims at strengthening the cohesion of research in the broad sector of linguistic studies, humanities, cultural heritage, history, archaeology and related fields through a thematic cluster of european research infrastructures, integrating initiatives, e-infrastructures and other world-class infrastructures, and building bridges between different, although tightly, interrelated fields.” (parthenos/about the project/the goal [2018]). parthenos aims to “provide common solutions to the definition and implementation of joint policies and solutions for the humanities and linguistic data lifecycle“. for this end parthenos “will deliver guidelines, standards, methods, services and tools to be used by its partners and by all the research community”. in their “report on guidelines for common policies implementation (1)” (parthenos 2017), parthenos presented a set of high-level principles as common guidelines to their community.  these guidelines are based on the fair principles, but also extend these. for instance, fair does not directly address long term preservation of scholarly output as such, but only long-term preservation of the metadata of those objects. parthenos explicitly formulates policies, strategies and guidelines for long term digital preservation of objects (parthenos 2017, par. 3.3.2). it refers to the much used oais (open archival information system) standard (iso [updated 2012]). metadata2020 metadata2020 (metadata2020.org) is a collaboration of stakeholders who create or use scholarly metadata. these stakeholders are service providers of platforms and tools, publishers, librarians, researchers, data publishers, data repositories and funders. with metadata2020 they advocate richer, connected, and reusable, open metadata for research output with the aim to improve the quality of metadata for research. they make users and creators aware that metadata should be richer to fuel discoverability and innovation, to bridge the gaps between systems and communities, and reusable to be able to eliminate duplication of effort. richer metadata of the metadata2020 initiative fits all aspects of the fair principles ‘findable’ and ‘reusable’ (especially r1). improving the connection between systems and communities is related to fair principle ‘interoperable’ (i1, i2 and i3) and ‘reusable’ (r1.3). the fair principle ‘reusability’ is also one of the three pillars of metadata2020 (most relevant are the fair principles r1.1. and r1.2). some special challenges between researchers and libraries are mentioned by metadata2020, such as that libraries have invested in siloed metadata standards, seem hesitant to develop new systems that support new models and prefer to fit new models into legacy systems (metadata2020 librarians [2018]). this has led to interoperability problems, and duplication of data entry for the use with different systems. another challenge is that librarians use metadata in a different way compared to researchers, and that even different research fields use different metadata within their own systems. libraries can play an important role if they work with the other stakeholders to build a common metadata vocabulary, including the nuances per research field. libraries can also help with the adoption of persistent identifiers to prevent disambiguation of names and institutions. to conclude, libraries, together with the other stakeholders, should work towards open, de-siloed metadata that promotes cross-platform interoperability and sharing for all stakeholders. cultural heritage reuse charter parthenos, together with a number of other european institutions (clarin and dariah, both partners in parthenos; apef, europeana, e-rihs) participates in the initiative to set up a ‘cultural heritage reuse charter’. a draft mission statement was published in 2017 (apef et al. [2018]) and a survey was held to collect feedback on the mission statement (clariah [updated 2017]). the mission statement formulates six generic principles that both heritage institutions and researchers should subscribe to: reciprocity, interoperability, citability, openness, stewardship, trustworthiness. the subject of these principles is a bit unclear. throughout the statement the phrase “cultural heritage data” is used without explaining the meaning of “data“. sometimes “content and knowledge” is used, “digital and non-digital assets“, “research data“, “output“, “data, digital collections or data descriptors“. of these six principles reciprocity seems somewhat out of scope. it states that “both cultural heritage institutions and researchers agree to share content and knowledge equally with each other, making use of data centres and research infrastructures” and as such it merely serves as a recommendation of signing a mutual contract. in fact, all principles are two-sided and describe obligations for institutions as providers and researchers as consumers of heritage data. interoperability is a combination of part of the fair principles ‘interoperability’ and ‘accessibility’: “international standards, frameworks and interoperability protocols“. the citation principle states that heritage institutions should apply human and machine readable “relevant data citation standards“. it can be questioned that this really something institutions should provide. it seems that citation standards can be derived from provided standard metadata formats as facilitated by tools like refworks, mendeley and zotero. the openness principle only addresses the use of licenses, which is represented by the fair reusable 1.1 principle. openness usually also comprises findability and accessibility principles, such as the use of open standards and protocols, persistent identifiers and online availability, for instance as formulated in tim berners-lee five stars of open data (berners-lee [updated 2015]). the stewardship principle focuses on long term preservation and accessibility. trustworthiness refers to provenance (fair principle reusable 1.1), including descriptions of “any relevant materials, equipment, techniques, procedures and protocols used“. the final trustworthiness requirement “alert to any defamatory use of cultural heritage data” seems rather out of scope and hardly feasible. it does not describe guidelines for presentation of content and data, but for monitoring usage of these. den – de basis den, the dutch knowledge centre for digital heritage and culture, has published a set of minimal requirements for digitization of cultural heritage, in dutch “de basis” (den [updated 2016]).[5] it consists of six areas of attention: rights management, findability, creation, presentation, digital sustainability/preservation, description. de basis is a mixture of guidelines, principles, policies, references and roadmaps. as such it serves in fact as a set of principles combined with detailed roadmaps how to implement these principles. rights management extensively covers fair 1. (meta)data are released with a clear and accessible data usage license. findability actually focuses on four issues: identification (data are unique and persistently identified: fair f1, f4, a1), accessible (data are accessible via the internet: fair a1), search machine searchable (more or less fair f, a, plus html search machine recommendations), reuse (fair a1, a1.1, i1, i3, explicitly not the same as fair reusability). presentation covers the separation of content and presentation, look and feel, multilinguality, web accessibility digital sustainability/preservation focuses on the long term preservation and accessibility of digital collections (fair does not really cover this as stated before) description covers all types of metadata about digital objects (fair f1, f2, f4, i1, i3, r1, r1.3). discussion in their brief existence, the fair data principles have experienced widespread publicity, adoption and adaptation in the scholarly information world and beyond (mons et al. 2017). some of the initiatives discussed here explicitly mention the fair principles and their adoption and adaptation of them. in other, older initiatives fair principles can be partly identified “after the fact”. whereas fair and most of its emulators are targeting the scholarly output environment, some initiatives relate to heritage collections, which is the focus of this article. can the fair principles be transposed and applied to heritage collection objects? not as such, but they are a good starting point. the two main limitations are the lack of explicit attention for long term preservation of digital objects, besides their metadata, and the excessive interwovenness of “data” (or objects) and “metadata”. the fair-like initiatives focusing on heritage collections differ substantially, which is caused by differences in target audience and objectives. the cultural heritage reuse charter is in fact a mutual contract between institutions and researchers, and for that reason cannot be implemented unilaterally by institutions. the den de basis set of best practices, although it is very broad and complete, is not compact enough for quick adoption. we deem it necessary for collections to be fair. when collections do not meet the fair criteria, these collections will be harder to use by teachers, researchers and enterprises. collections that will not be fair will run the risk of being out of the picture, as will in the long term their institution. the objective of this article is to provide a compact, and practical list of guidelines for achieving reusability of lam collections that can be fairly easily applied by lam institutions according to their own roadmaps and resources. minimum requirements for objects and metadata the fair principles for scholarly output can be applied to lam collection objects with a small number of adjustments and extensions, including some principles from the other initiatives mentioned here. the fair principles for lam collections are applied to three levels instead of two: objects (such as books, journals, artifacts, videos, datasets, etc.) metadata about the objects on elementary level (such as title, creator, identifier, date, etc.) metadata records (body of metadata elements about an object in a specific database) fair principles for objects findable objects (physical and digital) have a globally unique persistent identifier objects are described with metadata accessible digital objects are permanently accessible by: sustainable storage (hardware, storage medium) open universal access protocols version management backups interoperable digital objects are stored in preferred or acceptable formats reusable digital objects have a date-timestamp objects have a license for reuse, which is also available in a machine readable form fair principles for metadata findable metadata specify the global persistent identifier (pid) of the object. this pid is used in all systems/databases that contain a description of the object in question metadata about specific objects are available via one or more searchable online repositories, catalogs, online databases, etcetera accessible metadata specify information about an object’s availability, obtainability and/or access options interoperable metadata are available at least in one metadata schema appropriate for the specific type of object metadata are available in various additional generic standard data formats for other contexts metadata contain links/references to other objects/authority files, by using other global persistent identifiers reusable metadata specify the object’s rights holder metadata contain license information referring to the object metadata specify the object’s provenance fair principles for metadata records findable metadata records have their own global persistent identifier accessible metadata records are machine readable metadata records are accessible using open universal protocols interoperable metadata records must be of sufficient quality reusable metadata specify the metadata record’s provenance metadata specify the entity responsible for the metadata record metadata records have their own licence for reuse, which is also available in a machine-readable form clarification of the fair principles for lam collections persistent identifiers by “persistent identifier” (pid) we mean a “globally unique persistent identifier”. an “identifier” is a code which can be used to refer to an object. “unique” implies that the identifier is used exclusively for one particular object. “global” means that the identifier is valid throughout the entire world wide web. “persistent” entails the permanent availability of the identifier, independently of any individual organizations, systems or systems implementations. several systems are available for assigning global persistent identifiers to objects, such as handle (handle.net), doi (doi.org), urn-nbn (hakala and network working group [updated 2001]). for references to people (as author, subject etc.) a number of identifier systems can be applied, such as orcid (orcid.org), isni (isni.org), viaf (viaf.org). for indexing terms (subject headings, thesaurus terms, classification codes etc.) many identifier systems are available too, such as library of congress subject headings (library of congress subject headings [2018]), aat (getty research institute [updated 2017]), mesh (national center for biotechnology information, [2018]), etc. besides persistent identifiers for objects it is also recommended to assign identifiers to metadata records describing the objects. the metadata records in question must contain the identifier of the described objects as one of its metadata elements. both types of persistent identifier should be findable and usable via the online user interfaces of all systems and repositories that the metadata records are published in. it is still widespread common practice to use internal dedicated system identifiers which refer to both the metadata record and the object described. this type of internal identifier can be published on the web using a url based on the particular system or domain, but it is not a global and persistent identifier, because it depends on the specific system or domain. sustainable storage it is vital that digital objects are stored in stable server environments in order to assure their long-term availability. servers have to be managed, monitored and secured in a professional manner. whenever necessary the objects have to be migrated to new environments. a professional backup system must be in place to avoid the loss of objects. a versioning system must be applied in order to maintain previous versions of newer replacements. these versions must be assigned date-timestamps in order to distinguish between different versions. objects must be stored in preferred or acceptable formats (recommended formats statement [2016]), such as pdf (text), jpeg, png (image), wave, flac (audio). access protocols for access to both digital objects and digital metadata, universal access protocols must be made available on several levels. the first level comprises server level access protocols for copying files between servers and workstations (ftp/sftp), direct server command line access (ssh) and downloading files and data through web browsers (http). the second level consists of application level access protocols for accessing metadata, such as application programming interfaces (apis), harvesting (oai-pmh) and retrieving metadata (z39.50, sru) and linked data (sparql). finally, there are the general formats and frameworks, other than the domain specific data formats (see standard metadata formats below), such as xml, rdf, json, json-ld, which are used to represent the domain specific formats. for instance, marc can be represented as marc-xml, dublin core in xml, rdf, etc. in order to support reuse of metadata and digital objects as much as possible, options for selecting and downloading in bulk must be presented as well as options for retrieving and downloading records and objects individually. standard metadata formats it must be possible to provide metadata records in various standard formats, other than the format the metadata is stored in. examples are marc, dublin core, mods, mets. the stored metadata must be sufficient to describe the specific object type and support the intended reuse. in the linked data context vocabularies or ontologies in rdf are used. for this various standards are available, such as edm (europeana data model), cidoc-crm (for cultural heritage), bibframe (the prospective linked data successor to marc), as well as well-known formats like dublin core and schema.org. copyrights and licenses in order to avoid the recurring necessity of requesting permission for reuse in every individual case, it should be instantly clear what actions are permitted with the objects and metadata. this is possible by including a license provided by the rights holder. the most used open content license is the creative commons (cc) license.[6] the cc license is a worldwide, irrevocable, non-exclusive license for the duration of copyright and similar rights, and sui generis database rights. the most used version is the ‘cc-by’ (creative commons. attribution 4.0 international [2018]). with a cc-by, the rights holder gives permission that metadata or an object may be distributed, copied and adapted free of royalties under the condition of attribution. in addition there are 3 other elements that can be added: nd (the metadata or object may not be adapted); sa (must be distributed under the same license); and nc (may not be used for commercial purposes). if copyright no longer applies, or if the copyright is waived by the rights holder, a cc-0 statement can be used to explicitly state that the object or metadata is a part of the public domain or that all copyright and related rights to the fullest extent allowed by law are waived and can be used by anyone without restrictions (cc0 [2018]). if a license cannot be granted for any reason, consideration can also be given to providing a standardized statement, such as ‘unknown rightsholder’. these standards can be found on the site rightsstatements.org that has been set up by europeana and the digital public library of america (dpla). organizations will have to determine which license for reuse will be granted to which objects and metadata. in addition, organizations must take into account other rights such as contract, portrait, personality and privacy rights in relation to objects and metadata. external links for the description of entities related to the objects (people/creators, subjects, organizations, places etc.) the accepted commonly used persistent identifiers, available in accepted external authority files, must be recorded in the metadata whenever possible. for people for instance orcid, isni, viaf etc. for subjects many sources are available, such as library of congress subject headings, getty art & architecture thesaurus aat, medical subject headings mesh, etc. provenance provenance originally stands for “the chronology of the ownership, custody or location of a historical object … the primary purpose of tracing the provenance of an object or entity is normally to provide contextual and circumstantial evidence for its original production or discovery, by establishing, as far as practicable, its later history, especially the sequences of its formal ownership, custody and places of storage. the practice has a particular value in helping authenticate objects.” (provenance [updated 2018]) in digital contexts it can also be applied to digital objects, digital representations of objects, data and metadata. in this case provenance can be recorded for physical objects, digital objects and data in the context of all kinds of collections and archives. quality assurance in order to promote reuse of metadata and objects the metadata must be correct, complete and up-to-date. to this end a practice of quality assurance for metadata must be adopted. this implies the establishment of requirements the metadata must comply to, monitoring these requirements, verification of the actual metadata, assigning sufficiently qualified staff. the quality requirements at least include the minimum requirements proposed here, with the addition of criteria for the specific domain. roadmap to fair collections in order to make collections fair, a lam institution should draw up a roadmap, consisting of a number of steps. some of these steps can be taken simultaneously, some are preconditions for other steps, some are temporary, others will be permanent. we recommend the following steps: set up a working group the first step should be setting up a working group, taskforce, or similar, to coordinate and carry out the activities needed for making the institutions collections fair. the working group should consist of representatives from all internal stakeholders and specialists, such as metadata experts, developers, systems coordinators, legal copyright experts, etc. the intention is to cover all perspectives of fairness: description, hardware, software and legal issues. the working group could be established as a temporary task force until all collections are fair. but it is recommended to have some kind of permanent body to monitor and adjust the situation as needed. compile an overview of collections one of the first tasks of the working group is to compile an overview of all of the institution’s collections, and all the databases, repositories and systems these collections are described and presented in. compile an overview of existing agreements an overview of agreements and collaborations of the institution with national and international standards organizations will be helpful in assessing what standards have already been adopted, implemented or agreed upon. compile an overview of existing fair principles per collection the overview of collections and the overview of agreements can serve as input for compiling an overview of already implemented fair principles per collection. besides these overviews of course also thorough examination of the collection infrastructure is indispensable. a useful instrument would be a kind of scorecard containing all fair principles on which one can mark the score for each principle per collection, including comments. obviously, the scores can only be established by communicating with all stakeholders. a completed scorecard for each collection will automatically reveal the issues that have to be addressed. involve stakeholders in order to identify the fair issues that your users experience, it is recommended to involve all stakeholder groups: end users, researchers, teachers, etc. an advisory group can be established with representatives of all stakeholder groups that are relevant to the institution. communication between working group and advisory group can produce an overview of usage requirements, use cases, etc. talk to others if possible, talk to other institutions who already managed to make collections fair and learn from their experiences. there is no need to invent the wheel again. execute a pilot if appropriate, carrying out a pilot project with a sample collection or sub-collection can be useful to identify the main issues. the pilot can for instance focus on providing bulk downloads of collection data, connecting the collection to other platforms using linked open data, or publishing digitized objects. the pilot will reveal all obstacles that are preventing findability, accessibility, interoperability and reusability of the specific collection’s data and objects. establish a fair policy an institutional policy should be established for achieving fair collections, preferably based on a pilot, feedback from stakeholders and the compiled overviews. write an implementation plan once an institutional fair policy is in place and solid overviews of collections, agreements and issues are available, it is time for writing an implementation plan. in this plan it is essential to set priorities and establish implementation decisions for those principles that are fundamental across all collections. for instance, in order to make anything fair, you need globally unique persistent identifiers. so before you do anything else, it is necessary to decide on the specific pid-schemes available (doi, handle, uri, etc.) that are suitable, identify the practical implementation issues involved and configure a pid infrastructure. other fundamental principles are sustainable storage, license management, links/reference management, provenance information and metadata quality. the information system/database/software infrastructure is also important. an institution has to be able to implement fair principles in the available technical infrastructure. if the institution is dependent on external system and database providers, there have to be plans for negotiating with these providers if necessary. also, when procuring new database/system environments, it is advisable to incorporate the fair principles into the requirements. when these organizational and technical foundations are adequate, it is time to focus on the systems infrastructure. this depends highly on the degree of distribution and fragmentation of the systems infrastructure. a museum with just one information system for both cataloguing and discovery can focus on that one system only for implementing most of the technical requirements. a university library with physical text material, online digital resources, scholarly output and special collections will have a number of systems and databases for cataloguing, handling and presenting their collections. in this case focus on source systems/databases first, so all improvements can be propagated to dependent target systems. for instance, objects can be described in a backoffice library management system, the metadata can be copied to an end user discovery tool and a repository, the digital objects can be made available in an institutional repository and copied to an online end user tool. finally, it is essential that existing workflows and responsibilities are adapted to the new fair principles, so that new objects and collections are fair from the start. conclusion making lam collections fair (findable, accessible, interoperable and reusable) is a complex endeavor. it is important that all involved organizational units of an institution work closely together in order to break down the organizational, legal and technological barriers that prevent it. besides that, external stakeholders, users of the collections and the collection data, have to be involved. as noted, this process requires addressing a wide range of facets when adopting fair principles. therefore, the strategy must be methodical and systematic. guidelines must be determined, and a policy must be established for the three collection levels “objects”, “metadata” and “metadata records” based on a pilot. it is important to observe, evaluate and, if necessary, adapt the guidelines with every new implementation. this approach should lead step by step to make an institution’s collections fair. references annotations and fair [internet]. [2018]; [cited 2018 feb 24]. available from:https://docs.google.com/document/d/1uobmtncl_dw5_tqljkwgfkbsm_4ypcxxiu3zj2fsuv8/edit apef, clarin, dariah, europeana, e-rihs, iperion-ch, parthenos [internet]. [2018] cultural heritage data reuse charter: the mission statement; [cited 2018 feb 24]. available from: https://sondages.inria.fr/index.php/593568/lang-en berners-lee, t. [internet]. [updated 2015 aug 31]. 5-star open data; [cited 2018 feb 24]. available from: http://5stardata.info british library free data services [internet]. [2018]; [cited 2018 feb 24]. available from: http://www.bl.uk/bibliographic/datafree.html cc0 [internet]. [2018]; [cited 2018 may 1]. available from: https://wiki.creativecommons.org/wiki/cc0 clariah [internet]. [updated 2017 aug 9]. cultural heritage data reuse charter : mission statement. amsterdam: clariah; [cited 2018 feb 24]. available from: https://www.clariah.nl/en/new/news/cultural-heritage-data-reuse-charter-mission-statement creative commons. attribution 4.0 international [internet]. [2018]; [cited 2018 feb 24]. available from: https://creativecommons.org/licenses/by/4.0/ den [internet]. [updated 2016 nov 23]. de basis. den; [cited 2018 feb 24]. available from: http://www.den.nl/debasis the fair data principles [internet]. [2014] la jolla, ca: force11; [cited 2018 feb 24]. available from: https://www.force11.org/group/fairgroup/fairprinciples getty research institute [internet]. [updated 2017 mar 7]. art & architecture thesaurus® online. getty research institute; [cited 2018 feb 24]. available from: http://www.getty.edu/research/tools/vocabularies/aat/ hakala j, network working group [internet]. [updated 2001 oct]. using national bibliography numbers as uniform resource names. request for comments. the internet society; [cited 2018 feb 24]. available from: http://www.ietf.org/rfc/rfc3188.txt iso [internet]. [updated 2012 sep] iso 14721:2012 (ccsdss 650.0-p-1.1) space data and information transfer systems — open archival information system (oais) — reference model. geneva: iso; [cited 2018 feb 24]. available from: https://www.iso.org/standard/57284.html library of congress. subject headings [internet]. [2018]. library of congress; [cited 2018 feb 24]. available from: http://id.loc.gov/authorities/subjects.html metadata2020 librarians [internet]. [2018]; [cited 2018 feb 24]. available from: http://www.metadata2020.org/communities/librarians/ mons b, neylon b, velterop j, dumontier m, da silva santos, lob, wilkinson, md. 2017. cloudy, increasingly fair; revisiting the fair data guiding principles for the european open science cloud. information services & use [internet]. [cited 2018 feb 24]; 37(1):49-56. available from: http://doi.org/10.3233/isu-170824 martone m. 2017. annotating all knowledge, fairly. hypothes.is [internet]; [cited 2018 feb 24]. available from: https://web.hypothes.is/blog/annotating-all-knowledge-fairly/ national center for biotechnology information. mesh [internet]. [2018]. bethesda md: national center for biotechnology information; [cited 2018 feb 24]. available from: https://www.ncbi.nlm.nih.gov/mesh open definition 2.1 [internet]. [2018]. open knowledge international;[cited 2018 feb 24]. available from: http://opendefinition.org/od/2.1/en/ parthenos/about the project/the goal [internet]. [2018]; [cited 2018 feb 24]. available from: http://www.parthenos-project.eu/about-the-project-2/ parthenos. 2017. report on guidelines for common policies implementation (1). d3.1. [internet]; [cited 2018 feb 24]. available from: http://www.parthenos-project.eu/download/deliverables/d3.1_guidelines_for_common_policies_implementation.pdf provenance [internet]. [updated 2018 jan 19]; [cited 2018 feb 24]. available from: https://en.wikipedia.org/wiki/provenance recommended formats statement [internet]. [2016] library of congress; [cited 2018 feb 24]. available from: https://www.loc.gov/preservation/resources/rfs/ rijksmuseum api [internet]. [2018]; [cited 2018 feb 24]. available from: https://www.rijksmuseum.nl/en/api universiteitsbibliotheek gent open data [internet]. [updated 2018]; [cited 2018 feb 24]. available from: https://lib.ugent.be/en/info/open what is open? [internet]. [2018] open knowledge international; [cited 2018 feb 24]. available from: https://okfn.org/opendata/ wilkinson md, dumontier m, aalbersberg ij, appleton g, axton m, baak a, blomberg n, boiten j-w, da silva santos lb, bourne pe, bouwman j, brookes aj, clark t, crosas m, dillo i, dumon o, edmunds s, evelo ct, finkers r, gonzalez-beltran a, gray ajg, groth p, goble c, grethe js, heringa j, ‘t hoen pac, hooft r, kuhn t, kok r, kok j, lusher sj, martone me, mons a, packer al, persson b, rocca-serra p, roos m, van schaik r, sansone s-a, schultes e, sengstag t, slater t, strawn g, swertz ma, thompson m, van der lei j, van mulligen e, velterop j, waagmeester a, wittenburg p, wolstencrof k, zhao j, mons, b. 2016. the fair guiding principles for scientific data management and stewardship. scientific data [internet]. [cited 2018 feb 24]; 3:160018 available from: http://doi.org/10.1038/sdata.2016.18 notes [1] see also “the library as an open global platform”, https://future-of-libraries.mit.edu/ [2] the article is based on an internal advice for the library of the university of amsterdam by the authors (2018) [3] the directive 96/9/ec of the european parliament and of the council of 11 march 1996 on the legal protection of databases [4] articles 19-21 dutch copyright act [5] in english “digital heritage – building a successful ict strategy.” [6] patent and trademark rights are not licensed under a creative commons license. about the authors lukas koster msc (l.koster@uva.nl) is library systems coordinator at the library of the university of amsterdam and the amsterdam university of applied sciences (hva), focusing on data infrastructure and discovery. his current activities are project manager improvement of the functionality and user experience of the primo discovery tool, co-lead of the working group fair collections and project team member of the amsterdam cultural heritage linked open data network project. orcid: 0000-0003-0214-4721. saskia woutersen-windhouwer llm (s.windhouwer@uva.nl) is specialist electronic publishing/repository manager and open access service manager at the library of the university of amsterdam (uva) and the amsterdam university of applied sciences (hva). she is also involved in open science at the uva and in the horizon2020 project openup, and co-lead of the working group fair collections at the uva. orcid: 0000-0003-0120-266x. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – designing digital discovery and access systems for archival description mission editorial committee process and structure code4lib issue 55, 2023-1-20 designing digital discovery and access systems for archival description archival description is often misunderstood by librarians, administrators, and technologists in ways that have seriously hindered the development of access and discovery systems. it is not widely understood that there is currently no off-the-shelf system that provides discovery and access to digital materials using archival methods. this article is an overview of the core differences between archival and bibliographic description, and discusses how to design access systems for born-digital and digitized materials using the affordances of archival metadata. it offers a custom indexer as a working example that adds the full text of digital content to an arclight instance and argues that the extensibility of archival description makes it a perfect match for automated description. finally, it argues that building archives-first discovery systems allows us to use our descriptive labor more thoughtfully, better enable digitization on demand, and overall make a larger volume of cultural heritage materials available online. by gregory wiedeman introduction archives are weird. or at least that seems to be the perception of many library technologists. while archives are often part of larger research libraries, archival methodologies are often misunderstood by our administrator, technologist, and librarian peers. this confusion has become more problematic as archives continue to need and develop more complex access systems to make description, digitized materials, and born-digital objects available over the web. implementing these systems requires cross-domain partnerships, and the misunderstandings and miscommunications around archival description in particular have severely hindered the development of discovery and access systems for archives. archives access systems do not work like library catalogs or really anything else on the web and currently have major usability barriers. to those who work mostly with the bibliographic description used by libraries and most of the web, it can be unclear why archives cannot just use the same systems, or why archives systems and practices just seem so limiting for users. archival methodology and its reasoning can be easily obscured among the more esoteric traditions of archives, like the celebration of famous men to demonstrate value to donors, hollinger boxes, or finding aids. it is often hard to differentiate between the value and the dogma. archivists themselves often find it hard to articulate why their needs are just different than their librarian peers. it can be challenging even for many archival practitioners to acquire strong expertise in archival description. in the united states, archival training is a concentration within a library credential, which can mean merely one or two archives-specific courses. you might only get one single class that discusses archival description, and even that is often taught by a faculty member with a research focus rather than extensive practitioner experience. archival description skills often need to be learned on-the-job and seem to be mostly effectively passed on through peer groups, mentorship, or other types of informal professional development that not everyone has access to. even archivists that do have strong knowledge of archival description may not have a detailed understanding of how web applications or other technologies are designed or work in practice. while many archivists see firsthand the constant friction in current access systems, they often struggle to articulate how they can be designed better as web applications. the divide in domain knowledge between discovery systems and archival description is a challenging one to bridge. i hope to clarify the core differences between archival and bibliographic description and outline a path towards more effective discovery systems. while bibliographic description is much more intuitive and commonplace in our web applications, archival methods free us to apply the valuable descriptive labor that is the main bottleneck in our digitization and born-digital acquisitions programs more thoughtfully and appropriately.[1] if used properly, archival description could enable us to better provide digitization services on user request at scale and make these materials available online for future users. the extensibility of archival metadata also makes it a perfect fit for using automated description, such as optical character recognition (ocr), entity extraction, or automated transcription to enhance discovery, as it combines imprecise output with human-created records. i try to make it much clearer why archival metadata makes discovery so peculiar, highlight the cases where it can be advantageous, outline a path forward to increase the usability of archives access systems, and make the case for privileging archival description when planning and designing discovery systems. the misunderstandings around archival description have hidden an enormous problem: there are no available off-the-shelf systems that provide access to digital materials using archival description. every digital repository, digital asset management system (dams) or institutional repository (ir) uses bibliographic description as an unrecognized design assumption. to illustrate this, i provide a case study of ualbany’s existing hyrax and arclight implementations which use archival description for discovery by linking data from these systems over apis. this approach works functionally but has substantial usability and maintenance issues. in working to combine these systems into a single archives discovery system, i wrote a custom indexer that adds digital materials, full-text ocr and extracted text content to arclight as a proof-of-concept example that i hope can illustrate a path forward towards designing access systems that work directly with archival methods. finally, i will point to some ways we can experiment with how archival inheritance is indexed to potentially mimic bibliographic usability. archival vs. bibliographic description by bibliographic description, i mean the creation of individual metadata records for each object with a set of descriptive fields. this has been the intuitive method of managing information going back beyond our relevant professional history. i’m sure you could go back thousands of years and find library workers creating some kind of discrete bibliographic record describing an individual item. library catalog cards and online public access catalogs (opacs) are canonical examples of bibliographic description. each record has a set of descriptive fields and is self-contained – all of the available information is contained within the record. dublin core states this explicitly in its “one-to-one principle,” where it declares that each discrete entity “should be described by conceptually distinct descriptions.”[2] while linked data adds some complexity by potentially breaking up records into statements, data structures and descriptive practices usually remain the same. most of the information on the web is displayed to users in a way that looks like bibliographic description. a search engine, a major e-commerce site, or wikipedia will display records of objects to users that contain all the available information. these records often link to other records, but each record still describes an isolated object and is fully comprehensible by itself. the ubiquity of this format proves its intuitiveness and usability. i am sure that this is to some degree an oversimplified caricature of bibliographic practices, but it is a useful contrast to help us to better understand the impact of archival description. while archives may appear to be just a specialized type of library, they have a fundamentally different methodology for managing and providing access to materials. why did early archivists reinvent the wheel and develop incompatible practices that are less intuitive for both professionals and users? the answer is very practical: they simply had too much stuff. the early development of archival description in the united states illustrates how usability was a conscious and necessary tradeoff to be able to adequately manage the scale of records that were working with. the american national archives was first created in the 1930s and, since the government had been functioning and creating records for over 150 years, records had been previously managed by individual departments and offices, often with a variety of different methods and techniques. by 1941 archivists had accessioned 302,114 cubic feet of records from seventy-two different agencies.[3] these early american archivists actually wanted to use bibliographic methods to make all these records easily accessible in familiar ways. they made multiple attempts to use various forms of card catalogs to describe materials and established a classification division devoted to somehow providing subject-based discovery. however, “…given the diverse mass of materials in the national archives, classification demanded vastly more time and expense than the agency could afford,” and the division was disbanded in 1941.[4] with truck after truck moving more and more records to the archives, all archivists were able to feasibly do was document the source of records and their existing arrangement. the provenance of each set of records was important because each source had a different arrangement system and discovery process. a user would have to use the “preliminary inventories” created by the archivists to find what office created the records they were seeking, and how that office arranged or maintained them to navigate that file series or records component.[5] these “preliminary inventories” evolved into paper and online finding aids over time.[6] of course, it would be simpler for users if all the records had a single discovery process, but to early american archivists, that was obviously (if regretfully) infeasible. usability was a conscious trade off to make the enormous volume of materials even somewhat accessible. as a rule of thumb, the approaches used by archivists are useful primarily because of the scale of the materials they manage. got a large but manageable amount of stuff? use bibliographic description. got a seemingly never-ending vast mountain of materials? use archival description. this is an oversimplification, as archival methods are also very good at retaining context of materials, but scale alone is a useful distinction to show how archival systems are meaningfully different.[7] the reality is that in our current unlimited information environment, archives and libraries have larger collecting scopes and volumes of materials than they have descriptive resources, much like the early national archives. even with the additional catalogers and archivists that should be hired to address this, archival methods should be reassessed in order to make the line between the available and the inaccessible more gradual, and to make a larger body of materials open for use. archival description in practice most of our librarian and technologist peers understand that archival data is structured differently, as archival data is hierarchical, with a tree structure of “components,” such as collections, file series, folders, and perhaps items. however, the way archival description inherits is not widely understood and has really important implications for system design. even archivists do not often articulate how the relationships between components of description work. for example, a repository might hold a folder called “meeting minutes, 1989 july 26.” this component only has a title and a date, which alone are not very helpful to users. who was meeting? what were they discussing? unlike a bibliographic record, not all of the available information is contained within the record and the relationships to other records are very meaningful. in this example, the file is part of a series titled “new jersey proportionality review project records,” which is part of a collection titled the “leigh b. bienen papers.” both higher order components have fields where a user might learn the purpose of the meeting and its potential participants and outcomes. image 1. the file is part of a series titled “new jersey proportionality review project records,” which is part of a collection titled the “leigh b. bienen papers.” https://archives.albany.edu/description/catalog/apap312aspace_c264f5e1f93f9d58e5b60483c32d76e9 here is where we have to get into the weeds a bit. at all levels, components may use twenty-five elements that are described in the archives content standard, describing archives: a content standard (dacs). eleven of these elements are required fields. the standard also outlines a set of requirements for multilevel descriptions that articulates rules for the relationships between multilevel archival components like the above example. this section of dacs is particularly impactful, but it is challenging for non-experts to fully appreciate its meaning. what is not often understood here is that, while most of the eleven required elements are often only used at the collection level in practice, each component is required to be described by every one of these fields. even the above “meeting minutes, 1989 july 26” example needs to have a name of creator(s), a scope and content note, an access conditions note, etc. this example is actually described by those elements, they are just stored outside of the record in higher order components. lower-level components only use dacs elements if they supersede or are more granular than the higher order component. if this is not the case, the element from the higher-level component applies. thus, the scope and content element from the new jersey proportionality review project records series component and several elements from the leigh b. bienen papers collection component also describe the “meeting minutes” file component. when archival repositories used paper finding aids, inherited elements were implicitly displayed using front matter, indents, and other design features that conveyed this relationship, but our current discovery systems do not account for this. archival description also provides us with a tremendous amount of flexibility, allowing for the discovery of full text, bibliographic records, and description automatically derived from digital materials within a single descriptive schema and discovery system. dacs allows archivists to use bibliographic metadata, such as dublin core fields, to further describe materials when there is a user-driven reason to do so. it just requires a clear and explicit relationship between these records and the archival component that describes them. this allows archivists to create high-quality descriptive records when appropriate. an archival collection can easily contain one series of lower-value or rarely used materials that are only generally described by the series description, and another series of high valued items containing high quality detailed metadata for each item that you would expect in a library catalog. instead of allocating a similar amount of descriptive labor to all materials as bibliographic description often does, archival description empowers archivists to use their appraisal skills and spend their valuable time in proportion to the value of materials they are working with. for rarely used items with less value, materials can still be accessible, just with a higher usability cost.[8] because archival methodology accommodates lower quality descriptive records, this also makes it a perfect fit for automated approaches that derive description from digital materials. this includes full text, technical metadata, or the output of computational techniques such as entities extracted using natural language processing (nlp). archival description is also a perfect fit for using emerging machine learning techniques for extracting meaningful information from digital images and documents for discovery if these tools can be used without causing harm. there have been some experiments that have used automated approaches to describe special collections materials. however, no matter how sophisticated, automated methods alone produce lower-quality records that limit discoverability and usability in bibliographic systems.[9] in archival description, these records would always be linked with higher-level metadata created by a human professional. the flexibility of archival description also makes it easy to manually enhance automated description when needed. for lower value materials that would not receive detailed description, automated description can also be better than nothing. archivists are also welcome to use automated description at first while assessing its use and potentially enhancing the description later as appropriate. yet, as i’ll discuss later, while archival description encourages these practices, the current systems available for managing digital materials are designed only to work with bibliographic description, thus they are blocking the use of automated approaches in practice. archival methods do have significant drawbacks. this is an idealized vision of archival description. systems that support the creation of quality archival description are a relatively new phenomenon and a lack of training and support can mean that archival methods are sometimes inconsistently mixed with bibliographic approaches, or just poorly applied. additionally, even if we design discovery systems that make use of archival description, there is a usability cost that may be unavoidable when we compare it to the simplicity of bibliographic description. when you compare catalog cards to finding aids or opacs to archivesspace, bibliographic description is often more familiar and comfortable for most users. the usability problems of online finding aids and archives access systems are very well-documented.[10] the more complex relationships in archival data are simply just more challenging to navigate and display intuitively. yet, there are paths forward if we design digital repositories to match the affordances of archival description, then we may be able to improve the usability of discovery systems to where the advantages are well worth the costs to users. the current landscape of digital repositories when we apply a strong understanding of archival description to the current landscape of digital repositories, we see that there are several digital repositories available, but no system allows for the discovery of digital material using archival description. this is true across both open source and proprietary systems. using archival methods for discovery is simply not currently possible without substantial customization. most repositories are designed as digital asset management systems (dams) like contentdm for the upload and discovery of digital objects or designed as institutional repositories (irs) like islandora, samvera-based applications, or bepress digital commons that have built-in multi-user submission workflows. every single one of these systems is designed with bibliographic description in mind. each assumes that librarians or archivists will enter a set of descriptive metadata fields when uploading digital objects. each tool also envisions itself as a self-contained system for this description. no complete off-the-shelf system expects description to be managed and made discoverable outside of its interface. remember that if an item is described by an archival component, dacs requires a clear and explicit relationship between that item and its higher-level components so that users can use those inherited descriptive fields, and it is reasonable for a user to expect at least a navigable link here. a common workflow is for archivists to digitize an item that is already described by an archival component, but since all dams and irs assume they are self-contained, the archivist then has to spend additional time and labor to create a separate set of dublin core or other bibliographic elements for a digital repository. this both duplicates effort and creates an obvious usability barrier. users often must navigate both a system for archival description and a separate system for digital content. this problem is particularly acute for small repositories, as to make digital content available, they are incentivised to change their local descriptive practices to match the system used by whatever consortial repository is available to them. it is probably correct to say that none of the current tools, including  contentdm, islandora, dspace, bepress digital commons, or samvera-based systems like hyrax or hyku, are compliant with dacs. archivists have no options. this is a major use case that is simply not being met with available tools, likely because of the divide in domain knowledge between archivists and administrators, librarians, and technologists. there is no off-the-shelf product that provides access and discovery for digital materials using archival repositories’ existing description methods and systems. over the last decade or so, there has been a lot of progress in designing and developing systems to manage archival description, with the development of archivesspace being a major success. however, archivesspace, access to memory (atom), and arclight all only manage and/or provide access to description, not digital content. while these tools all help provide us with an important piece to the access puzzle, users want to access materials, not just descriptive records. in-person research will always be a key part of archival repositories, but more and more archival research is being done primarily or solely online, with the covid-19 pandemic possibly being a major turning point. the closure of reading rooms finally forced many archives to regularly accommodate digitization requests on-demand. this is a major advancement in user services, yet many of these materials are often sent directly to users and not uploaded into digital repositories for future use. this is because these systems are not able to accommodate items without additional descriptive labor, despite them already having archival description and the fact that they were already discovered by a user.[11] archival repositories need systems that manage digital content to do less – focus on asset management, file serving, and interoperability. archivists are already able to create and manage complex archival description in tools like archivesspace or atom. archives need digital repositories to manage digital content but be interoperable with and rely on their existing description systems. the international image interoperability framework (iiif) is a great way to make these connections. there are some important roles that repositories should take on, such as processing or ingest workflows and technical metadata, but digital repositories as currently constituted cannot serve as the primary end-user discovery system for archival materials. it also could be advantageous to designate digital repositories and discovery systems as separate concerns, as repositories can better serve as a “back-end” systems that may better provide or are more interoperable with preservation functions. in the future this may help us avoid design problems like the samvera architecture, which too tightly coupled preservation and access functionality though activefedora.[12] this separation may also make it easier for systems to manage access restrictions, as archivists need to manage and preserve digital materials that cannot currently be made publicly available, as “virtual reading rooms” or limited or controlled access systems are another important piece of the access puzzle.[13] but most importantly, separating discovery from asset management may also provide us with the space and flexibility to design access systems that allow end-users to discover and navigate that content using archival description. ualbany case study a case study of the espy papers from ualbany illustrates both the potential for using archival description to manage digital objects, particularly by enabling digitization on demand, as well as the practical challenges that arose attempting this with current systems. m. watt espy spent most of his life documenting capital punishment in the united states. he dug up information for every death row inmate he could find from corrections records, county histories, court proceedings, and popular publications, and summarized each case on index cards – colorfully documenting victims, alleged perpetrators, and circumstances. at his height he had a large network of collaborators that sent him documentation sourced from all over the country. this collection represented the most complete documentation of executions in what is now the united states dating back to european colonization. in 1984 the national science foundation (nsf) awarded a grant to the university of alabama to create a computational dataset based on the materials, which was first released as executions in the united states, 1608-1987: the espy file.  on espy’s death, the original source materials along with other papers were donated to ualbany’s national death penalty archive and in 2010 it received detailed folder-level processing with funding from the national historical publications and records commission (nhprc).[15] while the espy file dataset became a canonical source for criminal justice researchers, abstracting the stories of these thousands of individuals onto a spreadsheet took away a lot of meaning and serviced only certain types of research. some researchers had found issues with the dataset and reference staff had heard a number of anecdotes from users about discrepancies they found between the index cards and the espy file data. seeing so many users willing to travel to see the index cards, along with the potential of leveraging the existing metadata from the dataset made it a strong candidate for digitization and in 2016, ualbany was awarded a council on library and information resources (clir) hidden collections grant to digitize two file series and make them openly available online.[16] since the collection had previously received detailed folder-level processing and the materials were the source for an existing dataset, it seemed wasteful and duplicative to create additional item-level records with bibliographic metadata for what would be about 125,000 digital objects. the espy file dataset was not created as descriptive metadata to our current standards and did not map to the paper materials in a machine-actionable way, so it was not useful as a drop-in replacement for bibliographic metadata in a dams. thus, the collection seemed like an excellent candidate for using existing archival description to provide access to the digital scans, as it could make practical use of the problematic espy file data. our existing systems provided no way to use the existing description to provide discovery and access to digital scans. we had recently completed migrating our archival description to archivesspace and were using extensible text framework (xtf) and the luna dams for access, but neither xtf or luna were interoperable or sustainably customizable and no digital repository was available that used archival description for discovery out-of-the-box. the archivesspace rest api provided the potential to use archival description in new ways, and we were eager to fully leverage our descriptive labor already dedicated to the collection to benefit users and make our work more impactful. we decided to implement an open-source digital repository that would be more customizable to use folder-level description from archivesspace along with the espy file dataset. for much of the source materials series, we thought that the quality folder-level description already existed should be sufficient to provide access. also, if we could implement a successful process for using existing archival description for digitization, we hoped that we could do the same for other collections, and potentially even provide digitization services on request for single folders without having to create detailed bibliographic metadata. we decided to implement a lightly customized hyrax repository which uses the samvera framework. hyrax is not a “turnkey” system, but a fully featured set of open components that can be implemented into a digital repository. we hoped the openness of hyrax would make it easily adaptable to our existing archival description. over the course of the project, the arclight mvp project made arclight into a viable option for providing access to archival description. because it uses a similar ruby on rails stack as hyrax, it became easier to implement arclight and integrate it with hyrax than doing a similar level of customization with the archivesspace public user interface (pui). we needed data to be passed both ways, from hyrax to arclight and from arclight to hyrax, and both systems exposed json metadata with rest apis, which was an invaluable feature we could not have done without. since both systems used the same technology, much of what we learned customizing one system could also be applied towards the other. we did not quite know what we were getting into. the project was significantly under-resourced in both outside funding and internal expertise. however, despite some delays, data problems, and the challenges of learning new technologies, the systems we implemented were a major success. the espy project execution records website provides open access and discovery to the espy papers. our university libraries also gained a lot of skills and capacity to implement and host open-source applications that would be applicable to other projects, we developed a more productive relationship with the university-level information technology services division, and we are better able to utilize our on-campus virtualized data center. the need to support these systems was successfully used in 2019 to justify filling a vacant technologist position that otherwise was not likely to have received university-level approval. the project enabled us to use existing archival description for digitized and born-digital items and allowed us to provide online access to a much greater volume of materials. on the hyrax side, we had to develop multiple custom data models to handle both legacy materials from our existing dams as well as objects that would rely only on a link to a component of archival description. it was relatively straightforward to create image and av models to handle the schemas used in our existing dams, but hyrax’s use of linked data uris was a barrier to creating a sensible digital archival object (dao) model for archival description.[17] to make connections between digital objects and components of archival description, we used the 32-character ref_id generated by archivesspace and indexed into arclight. each folder-level component would have a ref_id for itself, could have multiple ref_ids for higher level series and subseries components, and always had a collection identifier for the top-level collection. we thus needed three identifier fields, one containing multiple ids where the order mattered, and each having a separate meaning. it also made sense to store the name of each component in the model as a string. this was challenging to model using linked data uris since hyrax requires a unique uri for each field. once we got a set of uris that hyrax accepted, we essentially ignored the uris downstream and relied on local meanings for the fields. i am skeptical that even a perfectly designed or customized ontology would have provided any value to this project, and trying to use any form of the records in context (ric-o) ontology currently being designed by the international council on archives experts group on archival description (ica egad) would have been a nightmare.[18] once the dao model was complete, we customized the workflow page where an archivist would upload and describe a digital object. this worker would enter the ref_id for the component of archival description, the collection identifier, and click a “load record” button. this button would make a javascript ajax call to the arclight json api and automatically fill most of the descriptive fields. the worker would then only be required to add a resource type and a license or rights statement before uploading the object. image 2. dao model. we also customized the display page for each object to pull relevant archival description from arclight also using client-side javascript calls. when an object page loads, it uses the ref_id and collection identifier to query the archival description component and all of its parent components. the page then displays the names and links for all higher-level components as any scope and content notes. the use of client-side ajax calls is imperfect but allowed us to integrate the two systems without much more complex customization within the rails applications. if a worker was digitizing an item, they would just have to find the ref_id and collection number for the folder in archivesspace or arclight, and enter those fields in hyrax with a resource type and rights statement. for descriptive metadata hyrax would then only contain a title (example: skandalon, vol. 3, no. 9) and date (example: 1965 march 10), which by itself would not be very helpful to users. when a user accesses the item, hyrax will query and display scope and content notes for the skandalon and the university publications collection. a user could then read that this is a single issue from a bi-weekly journal of news and opinion published by campus christian council, which was part of an artificial collection of student publications.  this minimal descriptive workflow, along with rapid lower-quality scanning allowed for digitization on user request. we later implemented a new digital reproduction fee schedule that charged by the time required for digitization rather than page counts.[19] since we were using existing archival description for metadata and avoiding page count estimates with back-and-forth emails, in many cases we were able to digitize an item in about the same time as a traditional reference request and make requests that take under 30 minutes free to users. this practice improved user experience, allowed us to digitize a much larger volume of materials and make them accessible online, and has the added benefit of making our digitization labor more transparent to users. in this example, i received a request for one issue and digitized the whole run of 42 issues in an afternoon merely because i had some extra time and thought the materials were interesting and worth digitizing. in addition digitizing individual items on request, we also developed a batch upload workflow for large sets of items sent to an outside vendor for digitization. the process relied mostly on spreadsheets. here we also used existing archival description so the materials did not require item-level bibliographic metadata. this proved to be really useful for university publications, for example, where we had existing volume and issue lists. we had an existing tool for exporting this metadata from the archivesspace api, so we added on a process where an archivist could paste in the corresponding access file for each issue and a script would generate another spreadsheet that could be uploaded into hyrax using a rake task. this workflow enabled us to rapidly digitize large collections or file series that were really valuable for reference use, such as student newspapers, university publications, commencement programs, university organizational charts, press releases, and university senate legislation. while additional descriptive care would have improved discoverability as always, making these materials discoverable using existing archival description plus full text ocr and extracted text was a major advancement. while our arclight and hyrax implementations were very successful in providing access to digital materials using archival description, they also have a number of practical limitations. the most obvious problem is that users still must navigate two separate systems, one for archival description and another for digital materials. we implemented a “bento” style discovery layer based on quicksearch to make search results from both arclight and hyrax available from a single search box but found that users still had trouble navigating back and forth between the two systems.[20] a redesign in early 2022 based on the duke university arclight implementation addressed some minor issues with this integration, but the core problem remains.[21] additionally, getting data from hyrax back into arclight is challenging. it was easy to modify arclight templates to point to hyrax for digital materials, but once an archivist uploaded a new object into hyrax, that uri had to be added to a new archivesspace digital object record. we were also storing separate preservation copies for each object outside of hyrax so we needed to download the object, store it as a local archival information package (aip), and add an identifier that references the aip into hyrax. since hyrax does not provide an api for this, we were only able to automate this using a very wonky script that queries the hyrax solr index, adds a new digital object in archivesspace, schedules it to be indexed to arclight, downloads and stores the object as an aip and adds the identifier to hyrax by literally scraping the hyrax login and edit pages and posting data to the edit form using the python requests module. it worked, but it was a hack. this process along with overall support for hyrax creates major sustainability risks. our library systems department has struggled to maintain hyrax without anyone with a strong ruby or rails background on staff. major cuts to library staff in 2020-2022 only minimally impacted applications support, but with overall library staff reduced by about 30% due to unfilled retirements, our long-term support for customized applications should be questioned, particularly when we are adapting systems like hyrax and not using them quite as they are intended. overall, there is a need for this setup to be simplified. a discovery system designed for archival description archivists need a discovery system for digital materials that uses archival description. a true archival discovery system would query archival description along with item-level bibliographic metadata and automated description derived from digital materials, such as extracted text, ocr text, and a/v transcripts in a single search interface. arclight has the potential to be this system. currently arclight is an access system for archival description based on blacklight. it does not manage digital assets but returns individual components of archival description and lets users navigate through connected records. since arclight merely displays data indexed in solr just like blacklight, it also has the potential to display and return search results for digital objects, including full text. description_indexer is an experimental tool that overrides the default arclight indexing pipeline. out-of-the-box, arclight uses traject to index archival description from ead-xml files, often exported from archivesspace. while traject is set up to be easily configurable to select which xpath to use for each solr field, it is not easily customizable to add the significant logic needed to index archival description or data from other sources. instead, description_indexer is a python library that uses archivessnake and pysolr to index archival description directly from the archivesspace api. this approach is potentially very useful for individual repository instances but may be less so for consortial aggregators because of the high permissions levels currently needed to access the archivesspace api. description_indexer contains two very basic json data models, one for archival description and another for the arclight solr index. this extra layer of abstraction is useful, as any data source that can map to the archival description model would then be automatically indexable into arclight. the archival description model is very much a draft and is likely too simple to be comprehensive, but community consensus around a model like this is key to consistently representing digital materials in the arclight index. the description_indexer main branch is set up to be a “drop-in” replacement for the current traject indexer. the dao-indexing branch is designed to be a more experimental branch that flexibly indexes from digital repositories or other systems that manage digital assets. it is designed to be extensible, since individual implementations will likely need to index asset data from a number of different sources, you can write your own plugin-in to index digital assets from your local system. once description_indexer is installed, you can add a custom class in a .py file in your home directory or using an environment variable that will allow for local logic to override how digital objects are indexed. the ualbany example that is included queries json from our hyrax instance to index links to content and other item-level data not managed in archivesspace. description_indexer also contains multiple “extractors” for pulling content from digital files using apache tika and/or tesseract, however running these during indexing is a challenge and a better design would be to extract and store this data while processing digital files and make it available to the indexer via a file system or a rest service. here is also where there is the potential to experiment with new tools for extracting useful information from documents for discovery using nlp or models generated with machine learning. the data pipeline to the indexer needs further consensus and standardization. in writing description_indexer, i discovered that digital objects, files, and file versions are under-theorized in archival description and archivists need to better define these objects and their relationships. the portland common data model (pcdm) provides helpful definitions of objects and files, and should be incorporated as much as possible, but the relationships between objects and archival components in lieu of pcdm collections is ill-defined and current practice is inconsistent.[22] archivesspace attaches digital objects to archival components, but allows component attributes such as subjects and note fields to be attached to digital objects as well. digital objects also do not have href or url attributes but contain file versions which have file uri attributes. both digital objects and file versions also have is_representative boolean attributes that are likely useful for digital objects. overall, it should be clearer that digital objects are an abstraction that do not necessarily correspond to a file, and digital objects should probably have a field for an international image interoperability framework (iiif) manifest, as that also can be an abstraction and should be the preferred method of linking archival description to digital materials. attributes for how files and versions are displayed in the absence of a iiif manifests are also likely necessary, and overall, it was challenging to model this and broader and more complex community use cases are needed.[23] the biggest barriers to enabling the discovery of digital materials in arclight are establishing consensus data models and data pipelines. once content from digital materials is indexed into an arclight solr index, we can display those objects in arclight with only some minor customizations and a iiif-compliant image server. i implemented a simple demonstration application that illustrates what this could look like in practice. this system returns results based both on archival description and full-text content extracted from digital objects. this implementation has data and design limitations, but i hope that this can be a useful model that shows the potential for what arclight can be going forward. privileging archival description in discovery systems academic libraries and other cultural heritage institutions also manage digital objects using bibliographic description. to avoid implementing and maintaining multiple discovery systems, archival materials are often forced into off-the-shelf irs and dams designed for bibliographic description. a better understanding of archival description shows that it is actually more appropriate to do the reverse, and index bibliographic records into systems designed for archival materials. here, it might be helpful to see archival description as an organizational schema for managing materials which have many different organizational schemas. in the same way that the early national archives used archival description to manage different descriptive methods used by different government agencies, archival systems can also accommodate bibliographic metadata that provides more usable and familiar access. this provides the best of both worlds. we can have one discovery system that provides both a strong user experience for higher value materials while still providing some level of access for materials that do not receive wide interest and otherwise would not receive detailed descriptive care. this also works from a purely technical perspective. while it is possible to model archival description in digital repositories like hyrax, the more complex structure of archival data makes this very challenging. it is comparably much simpler to model bibliographic metadata into archival systems than the reverse. with well-defined data models, we can easily add bibliographic metadata to an arclight index, just like with a blacklight instance. these records could stand alone or also be linked to archival description. this provides arclight with the potential to unify bibliographic and archival metadata in a single user environment, offering the usability of detailed records with the extensibility of archival hierarchy. this would provide us with the full potential of archival description to flexibly allocate our descriptive labor based on the value of materials and user needs. navigating complex archival data structures for items with lesser value may still be challenging for users. if we can make decisions based on the value of materials, rather than systems limitations, this should actually be an effective allocation of our limited descriptive resources. there are also additional opportunities to improve the usability of archival description. since arclight is just an extension of blacklight, it presents description to users in search results as discrete units much like bibliographic metadata. what we can do is experiment with how archival tree structures are indexed to better match how dacs envisions inheritance. since dacs expects notes that are usually only applied at collection or series levels—like scope and content or historical notes—to apply to lower-level components as well, we can experiment with indexing these notes as part of lower-level components too and just return them with lower relevancy scores. arclight currently indexes parent access and use notes like this but does not use them to return search results. this has the potential to return better results for minimally-described materials, but would need to be part of an iterative usability testing process so that results are weighted appropriately. these are exciting possibilities, but we cannot do usability testing on archival discovery systems until they exist. conclusion archival description takes a very different approach to description than what is commonly used elsewhere – whether that be in library catalogs, digital repositories, or on the web. archival methodology has key strengths that make it very useful for managing the vast quantity of digital materials held by libraries and avoiding a digital divide in an era where pandemics and the emissions costs of travel may limit in-person research. our descriptive labor, no matter how extensive it is or should be, has limits. if academic libraries continue to prioritize bibliographic approaches to metadata and apply the same level of descriptive care to objects one by one regardless of value, there will always be a hard line between what is accessible and what is not. archival description provides flexibility that empowers us to apply that valuable descriptive care based on the needs of users and prepares us to experiment with automated metadata approaches and iterative workflows. archival methods simply more accurately and appropriately model our descriptive resources to our materials. unfortunately, it is currently very challenging to use archival description to manage and provide access to digital materials, as current digital repositories are not designed to work with archival description. archivists manage description for materials in systems like archivesspace that are designed for archival description, but dams and similar digital repository systems expect them to create additional bibliographic metadata for any digital material they manage, whether that is an appropriate use of resources or not. there is usually no easy way to link that metadata from two different systems together. in practice, this means that lower-valued items or the increasing number of items digitized by archives on user request are not made available or discoverable for future users because they do not have the value needed to receive detailed bibliographic description. this is silly considering archival description already exists for them. since archives data structures can accommodate bibliographic metadata, but the reverse is very challenging, discovery systems design must privilege archival description. currently, there is no easy way to integrate archival description from systems like archivesspace with digital materials managed in digital repositories into a single discovery point for users. ualbany’s approach of using a “bento” style discovery layer on top of these two systems works functionally but has substantial usability limitations and sustainability concerns. the misunderstandings around archival description have marginalized archival systems in academic libraries. because our digital access systems never have worked for archival methods, libraries long took shortcuts by establishing whole separate programs to manage unique digital materials and limiting archives and special collections to a very traditional understanding of their collecting scopes. instead of working with archives, libraries often worked around them – often causing needless duplication in metadata work, digitization, asset management, and digital preservation across different reporting structures.  arclight has the potential to unify discovery of archival and bibliographic description and provide a single discovery point for physical and digital materials that allows archivists to fully leverage the affordances of archival description. we need further community consensus on a data model for archival description – most notably for digital objects, files, and file versions. i hope description_indexer can be a helpful example that can be iterated upon, that further work can be done to index digital materials in the arclight index, and that we can experiment more with indexing archival description in general. while not really discussed here, archival description’s focus on agents and functions behind the creation of records has the potential for opening new patterns for discovery.[24] overall, we need examples of digital materials in arclight alongside archival and bibliographic description for iterative usability testing. about the author gregory wiedeman is the university archivist at the university at albany, suny where he helps ensure long-term access to the school’s public records. he manages the university archives and supports born-digital collecting, web archives, and systems implementation for the department’s outside collecting areas. he currently serves as co-chair of the technical subcommittee for describing archives: a content standard (ts-dacs). endnotes [1] joyce chapman, kinza masood, chrissy rissmeyer, dan zelner, “digitization cost calculator raw data,” digital library federation (dlf) assessment interest group (2015). https://dashboard.diglib.org/data/. amanda j. wilson, “toward releasing the metadata bottleneck: a baseline evaluation of contributor-supplied metadata,” library resources & technical services vol. 51, no. 1 (2007). https://journals.ala.org/index.php/lrts/article/view/5384/6604. [2] “dcmi: one-to-one principle,” dublin core metadata innovation. https://web.archive.org/web/20220627093857/https://www.dublincore.org/resources/glossary/one-to-one_principle/ [3] mccoy states that “…the national archives had to deal with the greatest volume of records in the world; the unparalleled diversity of their origins, arrangement, and types; and their widely scattered locations in 1935.” donald r. mccoy, the national archives: america’s ministry of documents 1934-1968 (chapel hill, nc: the university of north carolina press, 1978), 45, 69. [4] mccoy, 78-80. philip m. hamer, “finding mediums in the national archives: an appraisal of six years’ experience,” the american archivist, vol. 5, no. 2 (1942): 86-87. [5] the national archives, guide to the material in the national archives (washington, dc: united states government printing office, 1940), ix. [6] this process is discussed in more depth in gregory wiedeman, “the historical hazards of finding aids,” the american archivist, vol. 82, no. 2 (2019): 381-420. https://doi.org/10.17723/aarc-82-02-20. [7] in addition to working well at scale, archival description is also more effective at maintaining contextual relationships between records, their creators, and the activities that created them. this is further discussed in jodi allison-bunnell, maureen cresci callahan, gretchen gueguen, john kunze, krystyna k. matusiak, and gregory wiedeman, “lost without context: representing relationships between archival materials in the digital environment,” the lighting the way handbook: case studies, guidelines, and emergent futures for archival discovery and delivery, eds. m.a. matienzo and dinah handel (stanford, ca: stanford university libraries, 2021). https://doi.org/10.25740/gg453cv6438. [8] this practice is best described in daniel a. santamaria, extensible processing for archives and special collections: reducing processing backlogs (chicago: neal-schuman, 2015). shan c. sutton also discusses the further extension of this to digitization in shan c. sutton, “balancing boutique-level quality and large-scale production: the impact of “more product, less process” on digitization in archives and special collections,” rbm: a journal of rare books, manuscripts, and cultural heritage vol. 13, no. 1 (2012). https://doi.org/10.5860/rbm.13.1.369. [9] paul kelly, “better together: improving the lives of metadata creators with natural language processing,” in code4lib journal issue 51 (june 14, 2021), https://journal.code4lib.org/articles/15946. kaldeli, eirini, orfeas menis-mastromichalakis, spyros bekiaris, maria ralli, vassilis tzouvaras, giorgos stamou, and evaggelos spyrou, “crowdheritage: crowdsourcing for improving the quality of cultural heritage metadata,” information vol. 12, no. 2 (february 2021). [10] christopher j. prom, “user interactions with electronic finding aids in a controlled setting,” american archivist 67, no. 2 (2004): 234–68, https://doi.org/10.17723/aarc.67.2.7317671548328620. anne j. gilliland-swetland, “popularizing the finding aid: exploiting ead to enhance online discovery and retrieval in archival information systems by diverse user groups,” journal of internet cataloging 4, nos. 3–4 (2001): 199–225, https://doi.org/10.1300/j141v04n03_12. luanne freund and elaine g. toms, “interacting with archival finding aids,” journal of the association for information science and technology 67, no. 4 (2016): 1007, https://doi.org/10.1002/asi.23436. wendy scheir, “first entry: report on a qualitative exploratory study of notice user experience with online finding aids,” journal of archival organization 3, no. 4 (2005): 49–85, https://doi.org/10.1300/j201v03n04_04. joyce celeste chapman, “observing users: an empirical analysis of user interaction with online finding aids,” journal of archival organization 8 (2010): 4–30, https://doi.org/10.1080/15332748.2010.484361. [11] james e. murphy, carla j. lewis, christena a. mckillop, and marc stoeckle, “expanding digital academic library and archive services at the university of calgary in response to the covid-19 pandemic,” ifla journal vol. 48, no. 1 (2021). https://doi.org/10.1177/03400352211023067. florence sloan, “special collections practice in response to the challenges of covid-19: problems, opportunities, and future implications for digital collections at the louis round wilson library at the university of north carolina at chapel hill,” masters thesis, university of north carolina at chapel hill school of information and library science (april 30, 2021). https://cdr.lib.unc.edu/concern/masters_papers/1z40m3313. the infeasibility of creating item level records is also discussed in stephanie becker, anne kumer, and naomi langer, “access is people: how investing in digital collections labor improves archival discovery & delivery,” the lighting the way handbook: case studies, guidelines, and emergent futures for archival discovery and delivery, eds. m.a. matienzo and dinah handel (stanford, ca: stanford university libraries, 2021), 33. https://doi.org/10.25740/gg453cv6438. [12] esmé cowles, “valkyrie, reimagining the samvera community,” https://library.princeton.edu/news/digital-collections/2018-06-05/valkyrie-reimagining-samvera-community. [13] elvia arroyo-ramírez, annalise berdini, shelly black, greg cram, kathryn gronsbell, nick krabbenhoeft, kate lynch, genevieve preston, and heather smedberg, “speeding towards remote access: developing shared recommendations for virtual reading rooms,” the lighting the way handbook: case studies, guidelines, and emergent futures for archival discovery and delivery, eds. m.a. matienzo and dinah handel (stanford, ca: stanford university libraries, 2021). https://doi.org/10.25740/gg453cv6438. [14]  m. watt espy, john ortiz smykla, executions in the united states, 1608-2002: the espy file (icpsr 8451), (ann arbor, mi: inter-university consortium for political and social research (distributor), 2016-07-20). https://doi.org/10.3886/icpsr08451.v5. [15] m. watt espy papers, 1730-2008. m.e. grenander department of special collections and archives, university libraries, university at albany, state university of new york. https://archives.albany.edu/description/catalog/apap301. “commission recommends $7 million in grants,” the u.s. national archives and records administration, 2010 june 1. https://web.archive.org/web/20220307211150/https://www.archives.gov/press/press-releases/2010/nr10-107.html. [16] blackman and mclaughlin summarize the widespread praise for espy’s work, while also highlighting some of the espy file’s limitations and criticizing its use for quantitative analysis. blackman and mclaughlin, “the espy file on american executions: user beware,” homicide studies vol. 15, no. 3 (2011): 209-227. [17] models for the ualbany hyrax instance. https://github.com/ualbanyarchives/hyrax-ualbany/tree/main/app/models. [18] egad – expert group on archival description, “records in contexts – ontology,” july 22, 2021. https://www.ica.org/en/records-in-contexts-ontology. [19] “request items for digitization,” m.e. grenander department of special collections & archives, university at albany, suny. https://archives.albany.edu/web/reproductions/. [20] “quicksearch,” north carolina state university libraries. https://www.lib.ncsu.edu/projects/quicksearch. [21] sean aery, “arclight at the end of the tunnel,” november 15th, 2019. https://blogs.library.duke.edu/bitstreams/2019/11/15/arclight-at-the-end-of-the-tunnel/. [22] portland common data model (april 18, 2016), https://web.archive.org/web/20220912065008/https://pcdm.org/2016/04/18/models. [23] description_indexer experimental archival description model, https://github.com/ualbanyarchives/description_indexer/blob/dao-indexing/description_indexer/models/description.py. [24] the rockefeller archive center’s dimes access system is a really interesting step in this direction by emphasizing agent records and requiring users to click through archival components to convey description inheritance. renee pappous, hannah sistrunk, and darren young, “connecting on principles: building and uncovering relationships through a new archival discovery system,” the lighting the way handbook: case studies, guidelines, and emergent futures for archival discovery and delivery, eds. m.a. matienzo and dinah handel (stanford, ca: stanford university libraries, 2021). the records in contexts – conceptual model (ric-cm) also has a very intriguing focus on agents and functions for discovery that deserves further practical exploration. “records in contexts – conceptual model.” expert group on archival description (egad), https://web.archive.org/web/20221007020234/https://www.ica.org/en/records-in-contexts-conceptual-model. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – editorial introduction: it’s all about data, except when it’s not. mission editorial committee process and structure code4lib issue 30, 2015-10-15 editorial introduction: it’s all about data, except when it’s not. data capture and use is not new to libraries. we know data isn’t everything, but it is ubiquitous in our work, enabling myriads of new ideas and projects. articles in this issue reflect the expansion of data creation, capture, use, and analysis in library systems and services. it’s all about data. as we all know, libraries have been generating, capturing, using, analyzing, and manipulating data for a long time. “data” is a popular buzzword now, especially when preceded by the word “big” or followed by the word “management”. the code4lib journal has been publishing articles about data since its first issue, with andrew bullen’s piece about wrangling historical demographic data,[1] among others. this issue’s lineup has an unusually heavy emphasis on data, perhaps a reflection of the urgency among government and publishing stakeholders regarding data generation and preservation, perhaps because for libraries it is all about data. except when it’s not. we also know, because libraries have a long relationship with data, it’s not all about data. it’s also about conventions that become standards in working with data. it’s about things like data security, tools to manipulate data, and careful planning for curation because of factors like hardware and software obsolescence. and it’s about what data can do for us, like improve discovery, also a major topic in this issue. while there are several data-centric articles here, there are also cool tools created by library coders sharing their work with the community, such as, in this issue: integration of library services with internet of things technologies, where kyriakos stefanidis and giannis tsakonas present selida, a module developed for koha. topicspace: rapid prototyping a mobile augmented reality recommendation app, in which jim hahn, ben ryckman, and maria lux describe their work using ux methods to drive development. streamlining book requests with chrome, which describes the development and use of a handy chrome extension developed by dr. rachel schulkins and joseph schulkins sierradna – demonstrating the usefulness of direct ils database access, by james padgett and jonathan hooper, which describes three use cases with innovative interface’s database navigator application. in the data-related cool tools department, we have: manifold: a custom analytics platform to visualize research impact, in which steven braun describes his work developing a research assessment tool for the university of minnesota school of medicine. data munging tools in preparation for rdf: catmandu and lodrefine, where christina harlow explains the handy tools she set up for catalogers at the university of tennessee knoxville. generating standardized audio technical metadata: aes57, by jody ridder, describing the aes57 standard and the fits2aes tool developed at the university of alabama. collecting and describing university-generated patents in an institutional repository: a case study from rice university, in which scott carlson and linda spiro describe the fondren library’s project to populate a local repository with university owned patents open journal systems and dataverse integration– helping journals to upgrade data publication for reusable research is about an open source tool for submitting data for publication, described here by a group of authors representing the institutions responsible for its development (micah altman, eleni castro, mercè crosas, philip durbin, alex garnett, and jen whitney) and saving the best for last, collected work clustering in worldcat, where janifer gatenby, gail thornburg and jay weitz describe recent progress implementing clustering in a frbr world. we hope you enjoy this data-centric issue, remembering that it’s all about libraries and data. notes [1] connecting the real to the representational: historical demographic data in the town of pullman, 1880-1940 subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – spruce mashup london mission editorial committee process and structure code4lib issue 19, 2013-01-15 spruce mashup london spruce digital preservation mashups are a series of unique events that are being organized in the united kingdom to bring together digital preservation practitioners and developers to work on real-world digital preservation challenges. during the 3-day event the digital preservation developers work to create practical solutions to real-world challenges the practitioners are having related to digital preservation. meanwhile, the practitioners work to create compelling business cases for digital preservation at their institution. this article describes the spruce mashup london event held in september 2012. by edward m. corrado introduction the spruce project is a jisc funded partnership led by leeds university library. other partners include the british library, digital preservation coalition, london school of economics, and open planets foundation. the spruce project is a partnership designed to “collaborate to develop a strategy for engaging the academic community around digital preservation and will also contribute technical expertise and solutions.” [1] besides organizing digital preservation mashups and other events, spruce has a number of other initiatives. two other major initiatives are providing funding awards of up to £5,000 for developing “practical digital preservation outcomes and/or development of digital preservation business cases” [2] (available to institutions in the uk-only) and a community project know as crowd sourced representation information for supporting preservation (crisp) that is aiming to combat the challenges of digital preservation representative information (ri) using crowd-sourcing techniques and the collective wisdom and knowledge of digital preservation experts. [3] spruce digital preservation mashups are a series of events that are being organized in the united kingdom that are intended to bring digital preservation practitioners and developers “together to discuss, test, code […], plan, and share challenges related to the new types of content entrusted to libraries, archives, and museums to preserve and manage.” [4] practitioners bring real world digital preservation challenges and are then paired with a developer, who works in a sprint-like software development fashion to come up with a working solution before the end of the three-day event. it is hoped that the solution will not only help the practitioner with their specific problem, but also be reusable by other practitioners who experience similar issues in the future. the solutions and other outcomes of the event are documented on the spruce project wiki. while the developers work to create practical solutions based on real-world problems, the practitioners work to create compelling business cases for their digital preservation work with the goal of helping them request and receive funding for their work and digital collection management duties. the structure of the spruce mashups are rather unique in how they are structured and how they bring both developers and practitioners together at the event. therefore, the structure of spruce mashup london will be described in some detail below. spruce mashup london the spruce mashup london was held at the hubworking centre in london from september 18-20, 2012. the event was free to attend and attendees were provided food and lodging throughout the event free of charge. the event started with an introduction to the spruce digital preservation mashup and why the organizers felt they are necessary. they said they felt that, in some cases, there was a mismatch of solutions to digital preservation problems with software solutions and that better coordination between practitioners and developers was needed. they believed that there were possibly some open source solutions available to practitioners but that they might not be aware of them or they might not have enough software knowledge to implement them. the organizers also identified a need for more education about digital preservation sustainability and how to create a business case that could lead to additional institutional support and funding for digital preservation projects. there were two main goals for spruce mashup london to help address these problems. the first goal was to solve some concrete digital preservation challenges. this was to be done by “capturing” some real-world preservation challenges and then solving the problems using an agile software development process. the second goal was to prepare a generic business case for digital preservation. digital preservation is not often seen as “exciting” to upper-level administrators. therefore, practitioners need to build a business case to present to administration that outlines the reasons why they should support digital preservation. the purpose of the generic business cases that the practitioners created were to help identify the benefits and stakeholders of digital preservation services at the practitioner’s institution. during the mashup practitioners would also work on an organizational digital preservation skill gaps analysis and on a digital preservation elevator speech that they could present to administrators at their organization. after the organizers described the purpose, goals and agenda of the mashup, the participants introduced themselves and discussed their experiences with digital preservation. the practitioners also described the particular problem that they brought with them to the event in hopes of having a developer provide a working, practical solution. the issues presented ranged from organizational and where to start to highly technical issues. for example, rachel macgregor from the birmingham archives and heritage service stood up and said “help – i’ve got digital content and i don’t know how to manage it” [5] while maurice de rooj of the national archives of the netherlands (naneth) had an issue where tiff files would not render correctly (or at all) even though jhove marked them as valid and well-formed. [6] other issues presented at the london mashup can be viewed on the open planets knowledge base wiki. indeed, the wiki [7] was used throughout the event to document what was being discussed and created during the mashup. after the practitioners gave their introductions, the developers followed with their own introductions and responded to some of the practitioner’s issues. in a few cases attendees were both developers and practitioners, but for the purposes of the mashup they choose one hat or the other (normally developer, but not always). in total there were approximately twenty attendees, not counting the organizers. most of the participants were from the united kingdom although some came from other european countries and one (the author of this article) came from the united states. once introductions were complete the organizers broke the participants into small groups that each contained a couple of practitioners and developers. in these small groups the practitioners and developers discussed the issues presented earlier so that the developers could get an understanding of what type of solution was needed. a summary was then presented to everyone and after a short break, the developers went off to begin working on the solutions while the practitioners got together and described their issues on the mashup wiki, based on their conversations with the developers. while the developers were developing solutions to the practitioners’ problems, the practitioners began to build their business cases for digital preservation. the first step of building the business case was to identify the benefits of digital preservation. some of the benefits identified included 1) increased access to digital objects (allow for multiple people accessing documents, fulfilling legal and other institutional obligations, and making objects more discoverable through enhanced metadata); 2) developing expertise and knowledge that increases the status and creditability of the organization and enables the organization to provide digital preservation services and accept digital collections; 3) financial benefits that could be gained from providing digital preservation services to other organizations and the possibility of receiving grants; 4) automation of preservation tasks (thus saving valuable staff time); 5) fulfilling the institutional mission which in many libraries and archives can not be done without being able to handle digital material; and 6) preserving institutional memory and local/regional history created utilizing digital media. the second day began with a brief meeting between the developers and the practitioners that they were creating tools for. this was followed by a brief update from the developers on where the stood with the solutions they were working on to all of the participants. after this, once again the developers broke off and started developing while the practitioners went back to work independently on their business plans – this time focusing on stakeholder analysis. the day continued with developers working on projects while practitioners worked on their business cases with time provided in between the sessions for the developers to consult with the practitioners. during the day, practitioners also worked on a digital preservation skills gap analysis for their organization. during the third and final day of the mashup, the main agenda item was to finish what everyone had started, commit code, and finish the descriptions on the wiki of the developers’ solutions as well as the business cases that were created by the practitioners. the practitioners also created, and presented to everyone in attendance, short elevator pitches that they could give to administrators at their organizations. some of the solutions developed include: peter may wrote a java program that makes “use of a custom apache tika wrapper to extract file format identification and metadata from a directory of files and present aggregated data for identifying which files have full descriptive metadata and which don’t.” [8] it was agreed that this program worked well at summarizing information about various files within a directory. this can be used for identifying possible duplicate files and could also be used to determine the amount of descriptive metadata that is embedded in a set of files. rob talbot and peter cliff developed a solution for archivist practitioners thom carter and rebecca webster that built on previous work from peter may and carl wilson which extracted metadata from files utilizing python and apache tika to produce output in json and a simple xml format. talbot and cliff felt that “tika was also a good choice as the archivists indicated that these collections were predominantly text-based documents in relatively recent formats – mostly ms office and pdf – both formats that tika handles very well for both metadata and text extraction.” [9] cliff also participated in this solution by creating three further scripts that used this output to create n-gram word clouds. a related project at the mashup utilized perl scripts “that used the metadata […] extracted using apache tika to help locate duplicates and different versions of the same document” [10] (see correction) dominic ivaldi worked on a solution that used ffmpeg as a video transcoder. the practitioner had numerous extremely large video files that need to be stored and preserved. for example, he has a 28gb file that contained a 18 minute long promotional movie created for ford motor comapny in the 1950’s. this appears to be excessive “considering entire movies are sold commercially on 9.4gb dvds.” [11] maurice de rooj worked on a php script that reads exif data from a image file file using exiftool and then normalizing the output into a dublin core compatible xml file. it also adds specific metadata to the dublin core xml which is contained in an .ini file. [12] maurice de rooj also took the lead on solving an issue about how to move records from microsoft sharepoint to eprints for digital preservation. it was not feasible to come up with a direct solution during the mashup but they were able to create a new sharepoint view that contained all of the fields and could be exported as a microsoft excel file. another issue identified was that future content should be properly formatted which means “users need to be educated and made aware” [13] that if future documents are not properly formatted thet may not be able to be preserved. bram lohman, dirk von suchodoletz, and tom woolley investigated how to preserve video games and how to provide public access to the preserved games. instead of developing code, they created a list of issues involved and available emulators.[14] maurice de rooj added nexus file recognition to fido. [15] fido (format identification for digital objects) is a simple, command line tool written in python that can be used “to identify the file formats of digital objects. it is designed for simple integration into automated work-flows.” [16] a team of developers worked on a problem the national archives of the netherlands had with corrupt tiff images. the tiff images were unusable although tools such as jhove marked them as well-formed and valid. the team discovered that the problem was the images claimed to be 16-bit greyscale files however they were really 8-bit greyscale files. the team used exiftool to detect and correct the problem files. this situation leads to important questions and takeaways included what does it truly mean for a file to be valid and well-formed. [17] mashup participants voted on awards for the best developer and the best practitioner during the event. the winner of the best developer award was maurice de rooij from the national archives of the netherlands while rachel macgregor won the award for best practitioner. all of the materials from the event, including the business plans, the digital preservation issues the practitioners had, and the solutions the developers came up with are available on the event wiki. impressions although i occasionally write shell scripts and other small computer programs, i participated in this event as a practitioner. overall i found this to be an excellent event and i would encourage people that have real-world problems to solve relating to digital preservation or are developers that can solve some of these types of problems to participate in future spruce digital preservation mashups or similar events. as rachel macgregor reported in a blog post, “i suspect that any of the developers could have offered solutions for my datasets” and at the end of the event “i already had an answer to my question!” [18] another practitioner mentioned how useful the mashup was to them and i imagine most of the practitioners would agree. there were multiple benefits to attending this spruce mashup london as a practitioner. the first, and probably the most obvious, benefit was that i came away with working code created by an expert developer that solved a real problem i was having with creating and maintaining a list of metadata mappings for a large photograph collection. i also was able to work on a digital preservation business plan with the help and feedback of other people involved in digital preservation. lastly, the networking opportunities were tremendous. i met a number of digital presentation developers and practitioners that i can consult with and ask questions of. while i wasn’t participating as a developer i imagine that they equally benefited. not only did they get to solve real-world problems, they got to collaborate with other, highly skilled developers. in some cases, they also brought digital preservation issues with them and were able to get other developers to help them figure out solutions to difficulties they were having. in many cases developers working on digital preservation projects do not have other developers at their workplace that they can go to when they are trying to discover the best way to solve a problem. by getting together at events like this mashup they can get other sets of eyes to look at their problems and also will know who they can consult when they have additional questions in the future. i believe this cooperation and the networking that occurred during the event will help build a strong uk community of developers and practitioners interested in digital preservation. the funding for this event was covered by jisc. i believe this was a key factor to the success as the only expenses to participants and their employers were related to traveling to the mashup. in many organizations, especially for developers, it may have been difficult to get funding to attend the event if there were food and lodging expenses associated with the event that the organization had to cover. in order for this type of event to be replicated elsewhere, i think receiving some level of outside funding for the event will be crucial to having a critical mass at the event. in the united states, because of the geographic size, the travel costs may be a hindrance to attendance although that could be mitigated somewhat by having regional mashups and choosing locations that are easy and inexpensive for enough people to attend. event calendar september 2012: spruce mashup (london, uk) april 2012: spruce mashup (glasgow, scotland) april 30-may 2, 2013: spruce mashup (leeds, uk) july 2013: spruce mashup (london, uk) endnotes [1] http://www.dpconline.org/advocacy/spruce/786-spruce [2] http://wiki.opf-labs.org/display/spr/spruce+awards+-+funding+opportunity+for+digital+preservation [3] http://wiki.opf-labs.org/display/spr/crowd+sourced+representation+information+for+supporting+preservation+%28crisp%29 [4] http://www.dpconline.org/advocacy/spruce/878-2nd-spruce-mashup-london-tuesday-18-20-september-2012-?format=pdf [5] http://openplanetsfoundation.org/blogs/2012-10-01-spruce-mashup-london-2012-practitioners-tale [6] http://wiki.opf-labs.org/display/spr/valid+and+well-formed+tiff%27s+with+scanline+corruption [7] the main page for the spruce mashup london event is available at http://wiki.opf-labs.org/display/spr/spruce+mashup+london. readers of this article are encouraged to review the week for more details about the event. [8] http://wiki.opf-labs.org/display/spr/distinguishing+files+with+descriptive+metadata [9] http://wiki.opf-labs.org/display/spr/extracting+and+aggregating+metadata+with+apache+tika [10] http://wiki.opf-labs.org/display/spr/using+perl+to+write+scripts+for+reporting+on+the+content+of+the+collection [11] http://wiki.opf-labs.org/display/spr/ffmpeg+as+video+transcoder [12] http://wiki.opf-labs.org/display/spr/maintain+a+list+of+metadata+mappings+outside+of+the+script [13] http://wiki.opf-labs.org/display/spr/moving+records+from+sharepoint+to+eprints+for+preservation+solution [14] http://wiki.opf-labs.org/pages/viewpage.action?pageid=16713667 [15] http://wiki.opf-labs.org/display/spr/nexus+data+collection+isis+-+stfc+-+solution [16] http://www.openplanetsfoundation.org/software/fido [17] http://wiki.opf-labs.org/display/spr/solving+tiff+malformation+using+exiftool [18] http://openplanetsfoundation.org/blogs/2012-10-01-spruce-mashup-london-2012-practitioners-tale correction 16 january 2013: the original version of this article misidentified some of the people and/or their roles on solution #2. the author apologizes for this error. about the author edward m. corrado is director of library technology at binghamton university located in binghamton, ny (usa). at binghamton, he provides leadership for information technologies and digital initiatives and overall direction, administration and management of computer resources, systems, and networking in the libraries. corrado also supervises the systems department; oversee the libraries’ technology infrastructure, web services, and other information access and production technologies; responsible for the libraries’ ils (ex libris aleph); work with the library faculty and staff to research and develop new and innovative technologies and services; recommend policies; plan upgrades; maintain current awareness of digital library technologies; work with university information technology services; and represent the libraries’ information technology interests within the university and in suny-wide initiatives. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – automated processing of massive audio/video content using ffmpeg mission editorial committee process and structure code4lib issue 23, 2014-01-17 automated processing of massive audio/video content using ffmpeg audio and video content forms an integral, important and expanding part of the digital collections in libraries and archives world-wide. while these memory institutions are familiar and well-versed in the management of more conventional materials such as books, periodicals, ephemera and images, the handling of audio (e.g., oral history recordings) and video content (e.g., audio-visual recordings, broadcast content) requires additional toolkits. in particular, a robust and comprehensive tool that provides a programmable interface is indispensable when dealing with tens of thousands of hours of audio and video content. ffmpeg is comprehensive and well-established open source software that is capable of the full-range of audio/video processing tasks (such as encode, decode, transcode, mux, demux, stream and filter). it is also capable of handling a wide-range of audio and video formats, a unique challenge in memory institutions. it comes with a command line interface, as well as a set of developer libraries that can be incorporated into applications. by kia siang hock, li lingxia introduction audio and video content forms an integral, important and expanding part of the digital collections in libraries and archives world-wide. while these memory institutions are familiar and well versed in the management of more conventional materials such as books, periodicals, ephemera and images, the handling of audio (e.g., oral history recordings) and video content (e.g., audio-visual recordings, broadcast content) requires additional toolkits. in particular, a robust and comprehensive tool that provides a programmable interface is indispensable when dealing with tens of thousands of hours of audio and video content. ffmpeg is comprehensive and well-established open source software that is capable of the full range of audio/video processing tasks (such as encode, decode, transcode, mux, demux, stream and filter). it is also capable of handling a wide range of audio and video formats, a unique challenge in memory institutions. it comes with a command line interface, as well as a set of developer libraries that can be incorporated into applications. uniquely audio/video singapore content excellence in singapore content has been one of the core pillars of the national library board (nlb) of singapore. when the national archives of singapore (nas) joined the nlb family in november 2012, it brought with it a huge and highly valuable collection of audio/video resources, including oral history recordings, sounds, audio-visual recordings and broadcast content. nlb has been progressively digitising the audio/video content for preservation and access purposes and managing the digitised content with its robust and scalable content management service. the nlb content management service is based upon the alfresco community edition (kia & wang, 2013). the challenges of audio-visual content complexity of audio-visual content compared with most other formats, audio-visual content are relatively more complex. figure 1 shows the steps typically required to process such content [1]. figure 1: processing of audio-visual content the content is kept in a ‘container’, more commonly known as a format. the container is used to identify and package the various data components that make up the audio-visual content. some of the most popular containers are avi (audio video interleaved), flv (flash video), mpeg (moving picture experts group), mp4 (mpeg-4) and mov (quicktime). a video container will typically contain one or more data streams for video and audio. each of these data streams will need to be managed by a codec (coder-decoder) program that is capable of encoding and decoding the data stream based on a specific codec standard. generally, the codec encodes an input video data into storage, and decodes video content for playback. there are literally hundreds of codecs; the popular ones are h.264 and mpeg4. digital preservation requires high resolution as a memory institution, it is a key role of nlb to preserve singapore content that are of historical, cultural and heritage value for future generations to come. for this purpose, the audio-visual content will need to be digitised at the highest possible resolution. the following are some examples of the standards adopted by nlb for the preservation, working and access copies of the content (not exhaustive): -standard definition & high definition videos o mp4 – h.264 o wmv standard definition broadcast quality videos o apple pro res hq o imx 30 high definition broadcast quality videos o apple pro res hq o xdcam hd 2k film resolution broadcast quality o apple pro res hq o hdcam sr multi-screen access to audio-visual content with the proliferation of mobile devices in singapore (and in many other countries world-wide), it is critical for nlb to be able to deliver its content on all devices (including phones, tablets, notebook and desktop computers). these devices provide varying support for the types of container and codec standards. for optimal delivery, it is necessary to transcode the audio-visual content to cater to the devices to be supported. a media player that is able to detect the device capabilities so as to select html5 or flash to stream the appropriate audio/video format will also be required. nlb uses jw player for this purpose [2]. about ffmpeg ffmpeg (fast forward moving picture experts group) is a comprehensive, cross-platform solution to record, convert and stream audio and video [3]. it is free software licensed under the lgpl or gpl, and supported on linux/unix, windows and mac os x operating systems. ffmpeg is made up of 5 key components, namely: 1. ffmpeg: the command line interface 2. ffserver: the media streaming server 3. ffplay: the media player 4. ffprobe: the media data analyzer 5. developer libraries while the project started in 2000, active development appears to happen from 2010. more recently, a new major release takes place quarterly (version 2.1 was released on 28 october 2013, while version 2.0 was released on 10 july 2013). who is using ffmpeg? the projects page on the ffmpeg official website lists some of the users of ffmpeg [4]. for example, the google chrome browser uses the ffmpeg libraries to support html5 audio and video playback. the popular vlc media player is another project that leverages the cross-platform ffmpeg software [5]. nlb has recently implemented an audio/video streaming infrastructure based on the wowza solution [6]. ffmpeg has been integrated with that solution. beyond these, there have been speculations on the internet about youtube and facebook using ffmpeg in their video upload mechanism. however, these have not been confirmed by google and facebook. basic audio/video content processing as mentioned earlier, ffmpeg comes with the streaming server and media player components. we will, however, not cover them in this article since the focus is on programmatic processing of audio-visual content. let’s start our tour of the ffmpeg functionalities with some basic tasks. for ease of illustration, we will perform the tasks using the command line interface. metadata we will start by extracting some basic metadata about the audio-visual content. this is done simply by issuing the following command: ffmpeg –i audio.mp3 the ffmpeg command reads from one or more ‘input’ sources, as specified by the -i option. the input source can be a regular file, a pipe, a network stream, or a grabbing device. the output on the console is shown in figure 2. figure 2: metadata of mp3 audio file nlb was interested specifically in the duration information of the audio or video content. in fact, we have successfully integrated the ffmpeg software with our content management service to extract the duration of ingested audio-visual content automatically. resize another common task with regards to video content is the changing of the width and height of the content, which is needed to create versions of the original video to cater to the different form factors of the playback device. the following command takes in a video (video.wmv), and creates a new video (video_o.wmv) with width of 320 pixels and height of 240 pixels as denoted by the -s flag: ffmpeg –i video.wmv –s 320x240 video_o.wmv note that the format is normally detected automatically. there are a number of convenient ways to specify the width and height of the output file required. for example, the parameter of -vf scale=iw/2:ih/2 will result in halving the input width (iw) and input height (ih). iw and ih are predefined variables. the -vf flag applies the video filter. ffmpeg –i video.wmv –vf scale=iw/2:ih/2 video_o.wmv another useful filter is -vf super2xsai. it enlarges the source frame size by 2 times using a scaling algorithm that minimises the reduction of sharpness. crop cropping is done through the crop filter. the following commands crop the left, middle and right segments of the input video respectively: 1. ffmpeg –i video.wmv –vf crop=iw/3:ih:0:0 video_left.wmv 2. ffmpeg –i video.wmv –vf crop=iw/3:ih:iw/3:0 video_middle.wmv 3. ffmpeg –i video.wmv –vf crop=iw/3:ih:iw/3*2:0 video_right.wmv the syntax for the crop filter parameter is as follows: crop=ow[:oh[:x[:y]]] where ow – output width oh – output height x,y – the co-ordinates of the top left corner of the cropped area convert before we move on to the more advanced features of ffmpeg, we will cover the important topic of conversion. the conversion can take place from one container format to another (e.g., from wmv to mp4): ffmpeg –i video.wmv video_o.mp4 while nlb was exploring the use of voice-to-text automatic transcription technologies, we needed to extract the audio component of our video content. this was done simply with the following command: ffmpeg –i video.wmv audio_o.mp3 advanced audio/video content processing the ffmpeg software is extensive and comprehensive. a thorough discussion of the major features would be beyond the scope of this article. for this rest of the article, we will cover three more features that we find useful in nlb: overlay, extract and join. overlay many a times, it is necessary to ‘watermark’ the video content, just like how we would watermark an image. we will show the commands to overlay a company logo and a copyright statement onto a video. to overlay the nlb logo, we will first need the image file (nlb.png in the illustration below). we can then use the following command to achieve the result: ffmpeg –i video.wmv –i nlb.png –filter_complex overlay video_o.wmv as shown in figure 3, the nlb logo is placed by default at the top-left corner. figure 3: logo on top left corner you can determine the exact positioning of the logo by providing an x:y position that represents the top left corner of the logo within the video. ffmpeg defines 4 variables that can be used in the expression for x and y: w – width of main video h – height of main video w – width of overlay h – height of overlay with these predefined variables, you can easily place the nlb logo at the bottom right corner using the following command (see figure 4): ffmpeg –i video.wmv –i nlb.png –filter_complex overlay=w-w:h-h video_o.wmv figure 4: logo at bottom right corner it is common to add a copyright statement on our digital content. to do just that, we can use the following command: ffmpeg –i video.wmv –vf drawtext=”fontfile=/windows/font/arialbd.ttf:\ text=’all rights reserved.national library board singapore 2013’\ :fontsize=14:fontcolor=white:x=220:y=450” video_o.wmv by now, the above ffmpeg command should be self-explanatory. figure 5 shows the result of the above command, with the logo placed on the bottom left. figure 5: watermarks extract most libraries and archives provide the service for organisations and individuals to purchase content from their digital collections (where copyright permits), and this service may include audio-visual content. in some cases, the user is able to purchase specific segments of the content, and the libraries/archives will deliver only the segments requested. many times, such requests will need to be manually fulfilled. the next 2 commands from ffmpeg can be used to automate the fulfillment process, thereby enabling a fully automated end-to-end, straight-through order processing system. assuming that the user-facing ordering system gathers the information about the segment(s) the user would like to purchase, we can then program the system to extract each segment from the video using the following command: ffmpeg –ss 00:00:10 –t 00:00:20 –i video.wmv video_1.wmv the segment starting from the 10th second for the duration of 20 seconds will be extracted into video_1.wmv. this can be repeated if the user has selected multiple segments. join if the user selected multiple segments, these segments will need to be joined together. the command to join multiple audio-visual content is as follows: ffmpeg –i video_1.wmv –i video_2.wmv –filter_complex concat video_o.wmv the –filter_complex option is typically used when there are more than one input and/or output, such as the joining of the 2 video input streams in this example and the overlay examples described earlier. the output file can then be electronically delivered to the user. conclusion we just went on a whirlwind tour of ffmpeg, an excellent open source software for audio-visual content. it will do most of the tasks that you will ever need to perform. while we used audio-visual content to illustrate the features covered in this article, you will be happy to note that quite a number of the ffmpeg options work with images too. the online official documentation is adequate. however, given the many formats, codecs and numerous parameters and options that ffmpeg supports, it is not easy software to master. the book ffmpeg basics written by frantisek korbel is a good way to start, and the authors of this article have hugely benefited from it. moreover, while the illustrations are based on the command line interface, you can easily wrap the command line interface into web services, or embed the developer libraries into your applications. in fact, there are already many ffmpeg wrappers available for popular programming languages, such as the media handler pro by mediasoft [7]. we now have a powerful tool that we can leverage to handle the complex nature of the tens of thousands of hours of valuable audio-visual resources, and deliver an experience that is engaging anytime, anywhere, on any device. notes [1] taken from ffmpeg basics by frantisek korbel [2] http://www.jwplayer.com/ [3] http://ffmpeg.org/ [4] http://ffmpeg.org/projects.html [5] http://www.videolan.org/vlc/ [6] http://www.wowza.com/ [7] http://www.mediasoftpro.com/media-handler-pro.html references kia, siang hock and wang, zhi liang (2013) content-as-a-service platform with the alfresco open-source enterprise content management system. paper presented at: ifla world library and information congress, 17 – 23 august 2013, singapore. about the authors kia siang hock (siang_hock_kia@nlb.gov.sg) is currently the deputy director overseeing the it architecture and innovation at the national library board, singapore (nlb). in this role, he and his teams are heavily involved in the conceptualisation, proof of concepts (pocs), design and development of various innovative services at nlb. li lingxia (lingxia_li@nlb.gov.sg) is a senior systems analyst at the national library board, singapore. she provides it solutions and technical supports for internal and public-facing services at nlb. lingxia is responsible for the developments involving the ffmpeg software. subscribe to comments: for this article | for all articles one response to "automated processing of massive audio/video content using ffmpeg" please leave a response below: nilesh, 2014-03-03 hi, how join work with multiple audio files ? leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – mapfast: a fast geographic authorities mashup with google maps mission editorial committee process and structure code4lib issue 14, 2011-07-25 mapfast: a fast geographic authorities mashup with google maps when looking for information about a particular place, it is often useful to check surrounding locations as well. fast geographic subjects provide clean access points to this material, and a google maps mashup allows users to see surrounding locations that are also fast subjects. moreover, the web service to the underlying data is also open and available for use. the map interface allows for simple selection of a location, with links to enter it directly as a search into either worldcat.org or google books. by rick bennett, edward t. o’neill, kerre kammerer, and jd shipengrover background most search engines are keyword driven.  if you want to know something about a place, using the name of the place as a search term is usually sufficient.  however, not all place names appear in a controlled vocabulary such as the library of congress (lc) subject headings and name authorities [1].  even if the location you are interested in has a controlled subject heading, some material of interest may be cataloged using a nearby town or lake name. the mapfast prototype interface was developed to address this problem [2].  mapfast is a mashup that uses google maps to present fast (faceted application of subject terminology) [3] geographic authority records.  the prototype presents a different way to look at subject access to bibliographic records. the map interface allows for simple discovery of a geographic location, selection of a location via a map pin or a search result list, and display of location information including links that initiate either a worldcat.org or google books search for the selected entry. mapfast uses records from the fast database.  the fast schema reworks library of congress subject headings (lcsh) rules so that they are easier to use, understand, and apply. fast subject headings are facetted into their component parts such as topics, geographics, and events. the result is a more machine-friendly schema designed to handle a large volume of materials with less effort and cost. fast geographic data currently the fast authority file contains over 1.6 million authority records. the authority file is extensively indexed to support a variety of search options. documentation on searching the fast authority file is available on the fast web site [4]. fast is also available through the oclc terminologies service prototype [5]. the full fast authority file can be licensed for non-commercial use. there are about 160,000 geographic headings in fast.  the geographic coordinates for these headings were extracted from the corresponding lc records.  the lc coordinate is stored in a free-text field, usually in one of a few fairly consistent patterns, so the coordinate could be read and put into a machine-consistent format in the fast version.  when a coordinate was not available in the original lc record, the geonames [6] database was used as an additional source of coordinates. geonames is a geographical database combining several geographic databases. a matching algorithm was used to match fast headings to the geoname headings, and the coordinates and place types were added from these records. in all, over 62.5% of fast geographic headings have coordinates. in addition to geographic headings, events often occur at specific places.  when a battle or other event could be placed in a particular location, that coordinate was also added to the fast record.  this was done automatically by using the information in the records.  this is often approximate.  for example, the battle of gettysburg was assigned the coordinate for the nearby town of gettysburg rather than the exact place of the battle.  it is hoped that manual review or the availability of other databases for matching will provide more accurate coordinate information in the future. figure 1mapfast display of the battle of gettysburg corporate bodies are another facet of fast headings that may have coordinates assigned. these are often buildings, parks, or other facilities that have a particular location.  as of this writing, coordinates have not been assigned for any corporate headings but it is being considered by the fast team. mapfast interface in late 2009, google opened its map and data apis to include attribute searching [7].  the example in the google geo developers blog post was a college-finder application where the result could be refined by type of college, size, and distance from a location.  with these new features, we used this example as a basis to develop an interface where the fast geographic headings could be displayed, and allow them to be refined by the geographic type of the heading. in a google maps mashup, the google maps api [8] provides basic map functionality though javascript classes, and the developer implements a map interface using that functionality in combination with local data or some specialized functionality.  for mapfast, we modified the layout, provided fast data to be included as markers on the map, and customized some of the user-map interactions. mapfast has a simplified single-page user interface (figure 2) designed with the end-user in mind.  the interface consists of five sections: search entry, search results display, map display, subject heading detailed display, and permalink display. search entry is the first section and its main component is a search entry box. this section also has two search limiters presented to users as standard drop-down select boxes. the limiters are: radius (5, 10, 20, 50, 100 km) and type (all; populated places; regions or government districts; lakes, rivers, streams; other; events; or undefined). limiters can be placed on a search before or after the search has been executed. the second section is the search results display. this section is found below the search entry area. it is a simple list of fast subject headings displayed in order of distance from the center of the map.  the result list displays up to 20 subject headings.  each subject heading’s type is indicated via a simple icon to the left of the entry and each entry on the list is also a link that can be used to display the subject heading detailed display. . figure 2 – example of mapfast user interface the entries on the search result list are displayed on the map in the third section, the map display area.  this area is to the right of the search and results sections.  it is the display area for the google map and each entry on the search result list is displayed on the map as a google maps marker or pin. the google map can be controlled using standard google maps controls. either the search results list entries or the map display pins can be selected to bring up the fourth user interface section, the subject heading detailed display.  this section is a google maps overlay bubble that displays further information about the selected subject heading.  additional information includes: the full heading, the heading type, alternative subject headings (if available), and search links into worldcat.org and google books. the final user interface section is the permalink display. this is simply a unique link for a given location.  it allows anyone wanting to share or save a location to do so.  the permalink is presented in the “share this location” text box provided below the search results and map displays areas. mapfast interface mechanics search and display in the mapfast interface is a multiple step process.  this process is outlined in figure 3. first, the search terms are submitted to the google map engine.  the first result from this search is examined, and a latitude/longitude coordinate pair is extracted.  for example, a search for amsterdam, netherlands returns the results in an array, and the first element contains: result[0]. formatted_address =  amsterdam, the netherlands result[0].geometry.location.lat()  = 52.3730556 , and result[0].geometry.location.lng()  = 4.892222199999992 the geographic coordinates are then submitted along with the desired radius to the mapfast web service described below. a jquery.getjson request is used to make this request.  this returns a list of fast geographic headings and associated data.  this first-cut result is accurate, but it may not be pleasing.  the results may be too concentrated within the map display, or skewed to one side.  therefore, the results are examined, and a new longitude/latitude box is defined that contains the initial set of results, but which is likely to change the center of the result. this adjusted box is now submitted to the mapfast web service to obtain the final result set.  the google maps display is also adjusted based on this box.  the fast result list is processed to calculate markers to display on the map. map movements are handled in a similar fashion.   the google maps engine takes care of moving the map.  the new bounds of the google map are used for a new longitude/latitude box search to the mapfast web service, and the results are displayed. figure 3 –  mapfast interface flow as is standard with the google maps display, map markers may be expanded to provide information about specific map locations. in the marker expansion bubble, url links are provided to search worldcat and google books.  fast is not implemented in worldcat, but the keywords are typically found in worldcat subjects anyway, so using the fast heading as a keyword search to the subject index works well.  a normalizedname parameter is provided by the mapfast web service to avoid doing the complicated normalization in javascript.  an example worldcat search for netherlands— amsterdam is: http://www.worldcat.org/search?q=su: netherlands%20amsterdam&qt=advanced google books collects bibliographic data for many of its entries too, so a subject search also works well there: http://books.google.com/books?q=+subject:netherlands%20amsterdam&lr=&as_drrb_is=q&as_minm_is=0&as_miny_is=&as_maxm_is=0&as_maxy_is=&as_brr=0&source=gbs_metadata_r&c . mapfast web service restful query the web service is a simple restful [9] service, returning json objects.  this means that a simple url can access the information, and the lightweight json object returns all the data requested.  a sample url is: http://experimental.worldcat.org/mapfast/services?geo=-33.863,151.208;crs=wgs84&mq=&sortby=distance&radius=30000&max-results=3&callback=? the key parameters here are: geo = the latitude & longitude radius = search radius in km max-results = maximum number of records requested this query is processed by a perl script and mysql database as described below. building the sql query library search engines tend to be keyword oriented.  finding records with coordinates “near” an input coordinate is not a typical feature.  therefore, the more general mysql database was chosen for its ability to handle a complex sql statement that could calculate the distance from a point and rank the result by that distance. the critical database tables for this query are longitude and latitude.  in sql, it is quite easy to select all the records from a box of a latitude range and a longitude range, and a perl script is used to build and process the query. however, on a globe, that box is not square.  as the latitude increases, the width (longitude) of the box narrows, to zero at the poles.  to adjust for that, the longitude is increased by the dividing it by the cosine of the latitude.  at the equator, the divisor is one, so there is no adjustment.  at 60 degrees latitude, the longitude range is doubled.  of course the box is not truly square at this point, but it is close enough for retrieval.  google maps doesn’t support locations above 85 degrees latitude, so dividing by zero is not an issue. now that a result set within a longitude/latitude box can be obtained, it is only necessary to rank the result by the distance from the center.  the center coordinate is, of course, the midpoint of the query range.  the quadratic equation is used to calculate the distance from the center coordinate for ranking.  the sql query is then: $query="select [table list] from fast where latitude  >='$latlow' and latitude <='$lathi' and longitude >'$lonlow'  and longitude <'$lonhi' order by sqrt( pow((latitude-'$latcen'),2)+pow((longitude-'$loncen')/$longcorr,2)) limit $maxresults";&#91;/sourcecode&#93; where <ul> <li>latitude and longitude are tables in the mysqldatabase</li> <li>$lonlow, $lonhi, $latlow, $lathi are the longitude and latitude ranges.  the longitude has already been widened  corrected for latitude to approximate a square box.</li> <li>$latcen= ($lathi+$latlow)/2;</li> <li>$loncen= ($lonhi+$lonlow)/2;</li> </ul> $longcorr = 1/( cos 2*3.14*$latcen/360 ); $maxresults = maximum desired results to be returned. when the url request has only the center point and radius, the script constructs an approximate box around the requested circle, and treats it as a box type request. <h3>json return</h3> the following is an example search return in json format.  only a single record is shown. [sourcecode language='js'] { "name": "fast authority records", "status": { "code": 200,      "request": "geocode"   }, "placemark": [ { "id": "fst01320412", "name": "new south wales -sydney -australia square", "description": "", "extendeddata": [ {  "name": "normalizedname",  "value": "new south wales sydney australia square" },                             {  "name": "feature",           "value": "unknown"        },                             { "name": "fcode",          "value": "u"        }                                              ],</li>                 "point": { "coordinates": "-33.8650,151.2070" }                   },                ] …placemark entries repeat } each placemark parameter contains information from a single fast record, and the parameters are described in the web service documentation [4].  since the return is in json, the javascript can use it directly without further processing. when the url parameter “callback=[functionname]?” is added, this makes the request jsonp, where a javascript function call is returned around the json data. the “?” is required, and a random function name is returned when no parameter is given to process anonymous callback function. otherwise the functionname parameter is used as the callback function name. summary by combining the fast geographic and event subject headings, the geographic coordinates associated with the subject headings, and google maps, a map mashup was developed. this map interface is a simple, easy to use way to access materials about an area that were difficult to get at by other means. users can search for a place, then use features of the mashup to identify, locate, and access resources related to that place, or nearby places or geographic features. restricting the result to a feature type, such as events or lakes, rivers, and streams, allow the user to focus on the information they are most interested in. from a library developer aspect, mapfast shows how to develop a google maps mashup to make map based interfaces using local data. the restful web service that underlies this interface provides a json list of fast geographic headings that are near an input coordinate.  the web service results are compatible with both fast and lc subject headings, and is available so that others may develop additional applications. references [1] library of congress authorities. available from:  http://authorities.loc.gov/ [2] the mapfast interface is at http://experimental.worldcat.org/mapfast/ . the project is described at http://www.oclc.org/research/activities/mapfast/default.htm. the web service is documented at http://www.oclc.org/developer/services/mapfast. [3] fast: faceted application of subject terminology, by lois mai chan and edward t. o’neill, 2010. coins oclc research activities associated with fast are summarized at:  http://www.oclc.org/research/activities/fast/ [4] the fast database is available at: http://fast.oclc.org/ [5] oclc research terminology service pilot for fas:t http://tspilot.oclc.org/fast/?operation=explain&version=1.1 [6] the geonames geographical database: http://www.geonames.org/. [7] the maps data api: bringing geospatial search to google’s cloud (holden, 2009). available from: http://googlegeodevelopers.blogspot.com/2009/12/maps-data-api-bringing-geospatial.html [8] google map api: http://code.google.com/apis/maps/documentation/javascript/ [9] restful, representational state transfer. available from:  http://en.wikipedia.org/wiki/representational_state_transfer about the authors rick bennett (rick_bennett@oclc.org)  is a consulting software engineer in oclc research, where he works on processing and manipulating bibliographic and authority data.  currently he has been focusing on developing, maintaining, and displaying authority data for the fast project. rick was an undergraduate at penn state and graduate at georgia tech, where he completed a program in computer engineering. edward t. o’neill (oneill@oclc.org) is a senior research scientist at oclc research and project manager for the fast project. ed did his undergraduate work at albion college and his doctorial work at purdue university in industrial engineering. his research interests include authority control, subject analysis, database quality, collection management, and bibliographic relationships. he is active in ifla (international federation of library associations and institutions) and is a member of the ifla standing committee of the classification and indexing section. kerre kammerer (kammerer@oclc.org) is a consulting software engineer in oclc research.  kerre’s research interests include database quality control and authority data.  her current research activities involve the creation and maintenance of fast authority records and the conversion of lc subject headings to fast headings in bibliographic records.  kerre holds a ba in economics from depauw university. jd shipengrover(shipengj@oclc.org) is a senior web and user interface designer for oclc research.  she works to bring user-centered design principles to the web applications created by oclc research.  her research interests include web usability, web standards and information visualization.  she holds bas in journalism and the history of art from the ohio state university and has a mlis from kent state university. subscribe to comments: for this article | for all articles 2 responses to "mapfast: a fast geographic authorities mashup with google maps" please leave a response below: mashups bibliotecarios: mapfast | blog eco escuela 2.0, 2011-11-21 […] qué es mapfast, cómo ha sido construido y cómo funciona: code4lib journal {lang: 'es'} etiquetas: biblioteca, google maps, mashups en herramientas web 2.0, recursos, […] worldcat's mapfast mobile service | reading, writing, research, 2013-07-31 […] geographic place names can offer special difficulties in searching. a system faceted application of subject terminology (fast) takes library of congress subject headings and presents them in a more machine-friendly way to search for them. a mashup of fast with google maps makes mapfast […] leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – dimensions & vosviewer bibliometrics in the reference interview mission editorial committee process and structure code4lib issue 47, 2020-02-17 dimensions & vosviewer bibliometrics in the reference interview the vosviewer software provides easy access to bibliometric mapping using data from dimensions, scopus and web of science. the properly formatted and structured citation data, and the ease in which it can be exported open up new avenues for use during citation searches and reference interviews. this paper details specific techniques for using advanced searches in dimensions, exporting the citation data, and drawing insights from the maps produced in vos viewer. these search techniques and data export practices are fast and accurate enough to build into reference interviews for graduate students, faculty, and post-phd researchers. the search results derived from them are accurate and allow a more comprehensive view of citation networks embedded in ordinary complex boolean searches. by brett williams introduction in an early library position, i was working with second-language learners aiming for higher education in the united states, canada and the uk. in an attempt to explain the difference between what they needed to think through and what a computer search could accomplish, i coined this aphorism “computers are dumb, people are smart. but computers are good at doing the same thing over and over again.” this aphorism is a constant in my information literacy instruction and it informs how i teach search strategies and research. using vosviewer to map search results throws the power of a computer doing the ‘same dumb thing over and over again’ at a series of common search techniques of keyword, author and citation searching. the resulting maps are impressive and the interactive mapping allows a careful researcher to gain a comprehensive view of an author, a keyword search or a set of articles. literature review vosviewer has been in active development since 2009 [1]. vosviewer uses van eck and waltman’s vos (visualization of similarities) algorithm to display the relationship between entities in a way in which both direct and indirect connections between entities result in placing those entities closer together on a map [2]. in the library science literature, bibliometrics is presented in the context of research data management, often as a service provided by research data management departments in large research universities [3]. bibliometrics is also used as a tool for after-the-fact analysis of data gathered through other means, such as the content management system used for virtual reference transactions [4]. zahedi et al. applied the text mining tools inside vosviewer to identify common topics of interest to users of mendely [5]. however, up to this point there does not appear to be any application of bibliometrics or citation mapping to searching during the reference interview. vosviewer supports mapping citation data extracted from web of science, scopus, dimensions and pubmed. the demonstrations and live links in this paper will use data solely from dimensions, but the techniques and tools rely on generally applicable boolean search strategies. dimensions is a relatively new discovery layer developed by digital science, the developers of the readcube citation management software, integrating scholarly articles, book chapters, grant and funding material and allowing the export of bibleometric data. dimensions can be found at app.dimensions.ai. dimensions coverage and citation accuracy compares favorably to google scholar, scopus and web of science [6][7]. search techniques the 5w’s mnemonic is a helpful structure to use during the reference interview. it is used extensively in the reference interview techniques in this paper to organize and identify search strategies suitable for bibliometric mapping. table 1. who identify and record key researchers, including links to their profile in dimensions if available when narrow the date range what identify and record ‘fields of research’ inside dimensions that correspond with the research request where identify and record key institutions and research groups with the patron why identify the broader purpose of the research request and the intended outcome for the paper export figure 1. dimensions supports exporting the first 2500 results. for early stages of exploring a topic, simply using the relevance sort and mapping the first 2500 results can be extremely effective for identifying major topics, refining a search strategy and identifying major authors. a basic export is initiated by clicking “save/export” and then clicking “export for bibliometric mapping.” an export of 2500 records will take between 30 seconds and 3 minutes to export depending on the load on the dimensions servers.the export is a zipped csv file with each publication listed along with its citations. figure 2. while waiting for this export, continue the reference interview by identifying key authors under the “researcher” limit, narrowing the date range under the ‘publication year’ limit and identifying the categories used by dimensions under the “fields of research” limit. when the export is complete, you will get an email notification and the export will appear in the dimensions export center. the downloaded file is a zipped csv file. mapping unzip the file and open vosviewer you can launch vosviewer from the website at http://www.vosviewer.com/ under file > create > “create a map based on bibliographic data” figure 3. then “read data from bibliographic database files” and choose the dimensions tab, navigating to the location of the unzipped csv file figure 4. the type of analysis is ‘citation’ and the unit of analysis is ‘documents’ figure 5. depending on the search, modify the following: minimum number of citations figure 6. number of documents figure 7. you can verify and exclude any of the documents figure 8. if there are disconnected items, vosviewer will alert you before mapping and give you the option of only displaying the largest cluster 1. advanced boolean search – sorted by relevance tools dimensions vosviewer example search string (burnout and (suicide or “suicidal ideation” or (suicid* or parasuicid* or self-harm*))) and (surgeon* or doctor* or physician* or clinician* or anesthesiologists or anesthetist* or psychiatrist* or oncologist* or internist* or geriatrician* or cardiologist* or dermatologist* or rheumatologist* or nephrologist* or neurologist* or radiologist* or respirologist* or rheumatologist* or paediatrician* or endocrinologist* or pathologist* or neurosurgeon* or gynecologist* or obstetrician* or ophthalmologist* or urologist* or neuropathologist*) map figure 9. access to the original csv file: https://drive.google.com/drive/folders/1pphjiqj4htaekgnz0e6ecvga6hl4pqw6?usp=sharing techniques identify citation clusters vosviewer colors clusters of citations using a set of colors selected using the menu on the right hand side. the largest clusters are colored with a dark red for the largest, green for the second largest, a dark blue for the third largest and yellow for the fourth largest cluster. scan for these four largest clusters immediately and pass your mouse over the largest bubbles to identify the key topics. in the example map, the green cluster is largely about training therapists. the large red cluster is largely about medical students. figure 10. figure 11. you can click on any bubble to go directly to the article referenced. identifying these key clusters, and the most highly cited articles within them will allow you to fill out your 5w’s chart with the patron in a more comprehensive way. pay close attention to finding highly cited articles, systematic reviews, and authors. you may also wish to adjust the search based on the search strategy and parameters you set earlier. identify highly cited articles highly cited articles are indicated by the larger size of the bubble. in the green cluster at the bottom, the beidas 2010 article in figure 10 has 488 citations that match the criteria. the dyrbye 2006b article in figure 11 has 777 citations that match the criteria. look at the citations for these two articles, when looking at the search indicated in the example: beidas, rinad s., and philip c. kendall. “training therapists in evidence-based practice: a critical review of studies from a systems-contextual perspective.” clinical psychology: science and practice, vol. 17, no. 1, 2010, pp. 1–30. wiley online library, doi:10.1111/j.1468-2850.2009.01187.x. dyrbye, liselotte, et al. “systematic review of depression, anxiety, and other indicators of psychological distress among u.s. and canadian medical students.” academic medicine, vol. 81, no. 4, apr. 2006, pp. 354–73. both are relevant, but clearly the focus on medical students is a strength of this search and the cluster around the dyrbye article is something to look into. identifying connectors the tyssen 2001 article is closely connected to both the red (medical students), orange (medical students) and yellow clusters (first responders). it is also highly cited with 169 citations. tyssen, reidar, et al. “suicidal ideation among medical students and young physicians: a nationwide and prospective study of prevalence and predictors.” journal of affective disorders, vol. 64, no. 1, apr. 2001, pp. 69–79. sciencedirect, doi:10.1016/s0165-0327(00)00205-6. figure 12. 2. key papers & citations tools dimensions vosviewer dimension vosviewer tools “citation link” for creating larger data sets: https://docs.google.com/spreadsheets/d/1f73emj-fhrnrdthncpl04tvbtlh_i5cgqp70qkqwm_w/edit?usp=sharing example search exported and mapped bibliographies and citations for: donald, i., cartwright, s., taylor, p., millet, c., cooper, c., & johnson, s. (2005). the experience of work?related stress across occupations. journal of managerial psychology, 20(2), 178–187. https://doi.org/10.1108/02683940510579803. dyrbye, l. n., massie, f. s., eacker, a., harper, w., power, d., durning, s. j., … shanafelt, t. d. (2010). relationship between burnout and professional conduct and attitudes among us medical students. jama, 304(11), 1173. https://doi.org/10.1001/jama.2010.1318. dyrbye, l. n., west, c. p., satele, d., boone, s., tan, l., sloan, j., & shanafelt, t. d. (2014). burnout among u.s. medical students, residents, and early career physicians relative to the general u.s. population: academic medicine, 89(3), 443–451. https://doi.org/10.1097/acm.0000000000000134. dyrbye, l., thomas, m., & shanafelt, t. (2006). systematic review of depression, anxiety, and other indicators of psychological distress among u.s. and canadian medical students. academic medicine, 81(4), 354–373. retrieved from insights.ovid.com. map – including key articles – keeping citations – keeping cited by 1 citation, top 1000 documents, resolution 1: figure 13. map – excluding key articles – keeping citations – keeping cited by 1 citation, top 1000 documents, resolution 1: figure 14. techniques one of the most effective search techniques when talking to graduate students and faculty has been building off of articles that they already know about. inside the five w’s model, this helps fill out many aspects including when (publication dates), where (institutions where the authors have published, where studies were done), who (author and co-author searches), what (specific terminology) and why (schools of thought, methodological tools, validated measurement surveys). using dimensions you can export both the citations from a paper, and all of the papers that cite the paper. this technique works best with 3-5 ‘key articles.’ search for a key article by doi, using the doi search in dimension. with a single entry in the search results, “export for bibliometric mapping.” then open the full records and scroll down to ‘publication citations.’ click ‘see all,’ then export this view. repeat this for each key article. this creates a set of files that can be easily identified by size. figure 15. what exporting both the article with its references and the publication citations creates is a 360 degree view of the networks associated with each individual article, allowing you to map connections between them. as the process of extracting the individual articles can get a little confusing, i’ve developed a tool to help keep track of larger groups of articles. to use, find the individual article pages and paste them under “article url.” individual article url’s look like this, but may contain a longer search string: https://app.dimensions.ai/details/publication/pub.1083855537 the page to export the citations will appear on the right and will have a url like this: https://app.dimensions.ai/discover/publication?subset_publication_citations=pub.1083855537 https://app.dimensions.ai/discover/publication?subset_publication_citations=pub.1083855537 figure 16. adjusting resolution one of the quick adjustments you can easily make to simplify a map is to adjust the resolution. under the analysis tab in the clustering section you can adjust how sensitive the clustering algorithm is. lower numbers (0.1, 0.25, etc.) are less sensitive and will result in fewer, larger clusters. higher numbers (1, 2, etc.) are more sensitive and will result in many, smaller clusters. click ‘update clustering’ to apply the new resolution. adjusting resolution is particularly effective when you’ve already identified the importance of individual articles. mapping excluding only the ‘key articles’ you can also exclude the key articles you’ve identified already in the last selection before launching the map. this can be particularly effective if you know already that most of your articles will cite your key article. this emphasises the citations between the publication citations and publication references without the extra, known noise of the citation of the known key article. figure 17. 3. key authors tools dimensions vosviewer example search burnout and physician against all publications by liselotte n dyrbye burnout and physician against all publications by tait d shanafeldt map figure 18. techniques reviewing clusters under the ‘items’ tab in the sidebar you can review each cluster with an alphabetical view of each paper included in the cluster. cluster 1 is the red cluster. you can right click to jump directly to each individual bubble in the cluster. figure 19. as you have already identified these scholar’s works as important and you have already filtered by keywords for the specific type of journal article you are looking for, it is often effective to review each item in the cluster for relevance. filtering by first author built into the items tab is an option to filter. searching for a first author will show in which clusters that author’s work shows up. figure 20. 4. list of papers by doi (google sheets tool) tools zotero zotero doi manager plugin dimensionvosviewer tools spreadsheet under “doi” https://docs.google.com/spreadsheets/d/1f73emj-fhrnrdthncpl04tvbtlh_i5cgqp70qkqwm_w/edit?usp=sharing dimensions vosviewer example search 10.1001/archinternmed.2010.350 or 10.1001/jama.2012.6183 or 10.1001/jama.2019.4168 or 10.1001/jamainternmed.2013.8519 or 10.1001/jamainternmed.2016.3284 or 10.1001/jamanetworkopen.2019.0932 or 10.1001/jamaophthalmol.2016.5392 or 10.1001/jamaophthalmol.2016.5399 or 10.1002/hec.1344 or 10.1002/hec.1378 or 10.1002/hec.1663 or 10.1002/hec.3572 or 10.1002/jhm.2400 or 10.1002/lary.27311 or 10.1007/s10198-014-0582-8 or 10.1007/s10198-016-0861-7 or 10.1007/s11606-018-4462-2 or 10.1007/s11606-018-4579-3 or 10.1007/s11606-018-4632-2 or 10.1007/s11606-019-04940-9 or 10.1007/s12630-019-01321-y or 10.1007/s13524-014-0320-y or 10.1016/j.acra.2006.06.010 or 10.1016/j.adaj.2017.01.005 or 10.1016/j.ajo.2018.09.003 or 10.1016/j.amjms.2017.12.001 or 10.1016/j.fertnstert.2019.02.004 or 10.1016/j.genm.2010.07.003 or 10.1016/j.healthpol.2010.06.019 or 10.1016/j.healthpol.2015.01.005 or 10.1016/j.healthpol.2019.05.002 or 10.1016/j.ijrobp.2018.09.042 or 10.1016/j.jacc.2015.10.038 or 10.1016/j.jamcollsurg.2017.03.018 or 10.1016/j.jcjo.2014.02.007 or 10.1016/j.jcjo.2014.03.005 or 10.1016/j.jcjo.2016.04.013 or 10.1016/j.jebo.2005.06.007 or 10.1016/j.jhealeco.2011.05.005 or 10.1016/j.joms.2018.07.010 10.1016/j.joms.2018.07.010 or 10.1016/j.juro.2017.11.039 or 10.1016/j.sapharm.2011.06.003 or 10.1016/j.sapharm.2013.01.004 or 10.1016/j.socscimed.2011.03.047 or 10.1016/j.socscimed.2012.07.028 or 10.1016/j.socscimed.2013.06.018 or 10.1016/j.socscimed.2015.08.001 or 10.1080/03630242.2012.674092 or 10.1080/03630242.2012.674092 or 10.1097/01.anes.0000264766.41297.ce or 10.1097/01.aog.0000231720.64403.6f or 10.1097/acm.0000000000001250 or 10.1097/acm.0000000000001283 or 10.1097/acm.0b013e3182a71519 or 10.1097/aln.0000000000000834 or 10.1097/aog.0000000000002420 or 10.1097/hmr.0000000000000069 or 10.1097/mlr.0000000000000310 or 10.1097/mlr.0b013e318268ac0c or 10.1097/mpg.0000000000000637 or 10.1097/sla.0000000000002920 or 10.1097/sla.0000000000002928 or 10.1111/1475-6773.13120 or 10.1111/acem.13694 or 10.1111/caje.12334 or 10.1111/ceo.13523 or 10.1111/j.1442-9071.2007.01480.x or 10.1111/j.1572-0241.2008.01976.x or 10.1111/j.1572-0241.2008.01988.x or 10.1136/bmj.i2923 or 10.1136/bmj.l1510 or 10.1136/bmj.l2089 or 10.1136/bmjopen-2018-023811 or 10.1136/postgradmedj-2016-134094 or 10.1177/0046958017709688 or 10.1177/0950017015590760 or 10.1177/1077558715589705 or 10.1177/1077558718754573 or 10.1177/1203475418762719 10.1186/1471-2296-13-94 or 10.1186/1478-4491-12-32 or 10.1186/s12913-017-2343-8 or 10.1186/s12960-016-0162-3 or 10.1186/s12960-017-0258-4 or 10.1186/s13054-018-2139-1 or 10.1186/s40697-016-0117-6 or 10.1245/s10434-018-6450-5 or 10.1371/journal.pone.0129197 or 10.1377/hlthaff.2010.0597 or 10.1377/hlthaff.2017.0014 or 10.1513/annalsats.201804-252ar or 10.2106/jbjs.17.00532 or 10.2106/jbjs.17.00532 or 10.2106/jbjs.17.01501 or 10.2214/ajr.17.18256 map figure 21. techniques collecting dois and mapping while collecting articles to meet a research request, zotero offers several powerful tools for capturing accurate metadata. however, the most effective tool for managing dois is the zotero doi manager. after the plugin is installed, make sure to change the setting for dois under tools > get dois for new items > long dois. to easily extract dois ingested into zotero, select all of the items you wish to export, and choose the csv format. this will create a file you can open in excel or other spreadsheet software with a column of all of the dois from your selection. figure 22. dimensions allows you to search for articles by doi and supports boolean operators. however, the search starts to return inconsistent results after a search for about 45 dois. the search is also extremely sensitive to extra spaces or any small errors in forming the boolean search string of doi’s. to ease the creation of these long doi search strings, i created a tool to create 40 doi long perfect boolean search strings for dimensions. you can create a copy of this tool here: https://docs.google.com/spreadsheets/d/1f73emj-fhrnrdthncpl04tvbtlh_i5cgqp70qkqwm_w/edit?usp=sharing. a map based on doi’s is a pre-selected group of ‘key papers” exported along with their citations. if you have both the time and the justification, you can also apply the technique found in ‘key papers & citations.” for a collection of 90 papers as in the example map, this is rarely justified. jump to clusters as the majority of the articles have already been collected as relevant articles, the most effective use of these maps is to very carefully look at newer and less cited articles in the clusters. in the ‘items’ tab, right click on the cluster and choose “show cluster in visualization.” this will jump you directly to the cluster. figure 23. using the mouse tooltip to highlight the jagsi(2012) article results in five articles not included in the initial list of doi’s already identified as key articles, including an update of the jagsi(2012) article in 2013 and a specialized focus article on cardiologists, jagsi(2016). figure 26. 5. list of authors (google sheets tool) tools zotero dimensionvosviewer tools spreadsheet under “authorlist” <https://docs.google.com/spreadsheets/d/1f73emj-fhrnrdthncpl04tvbtlh_i5cgqp70qkqwm_w/edit?usp=sharing dimensions vosviewer example search https://app.dimensions.ai/discover/publication?or_facet_researcher=ur.01136211600.84&or_facet_researcher=ur.01233634501.14&or_facet_researcher=ur.01211404006.02&or_facet_researcher=ur.010153355772.07 map figure 24. techniques clusters & networks of citations in the general set of topics that have been used as examples in this paper, most of them have come from authors that publish in multiple areas. sometimes it is useful to get a comprehensive view of what an author or cluster of authors have published in a selected area. the example map above is a raw export of four authors who have published in the general area of burnout among physicians. however, most of these authors are also specialist physicians. the crossover between the clinical work of a specialist and their work on physician issues is not always obvious, and mapping their work provides an easy way to get a general overview of both aspects of their work. the largest section of the above map that mostly refers to physician burnout is the large right hand cluster. figure 25. the tait shanafelt (stanford) and lisolette dyrbye (mayo clinic) papers in the (roughly) green/light blue and dark blue clusters above are all generally about burnout among physicians and are all generally based on samples in the united states. search against selected authors https://app.dimensions.ai/discover/publication?or_facet_researcher=ur.01136211600.84&or_facet_researcher=ur.01233634501.14&or_facet_researcher=ur.01211404006.02&or_facet_researcher=ur.010153355772.07&search_text=physician%20burnout&search_type=kws&search_field=full_search the search link above is a search created by taking the author search created with the authorlist tool and then doing a keyword search against it. this can be a particularly powerful tool when a reference interview identifies a large number of important authors. this technique is similar to the key authors search detailed above, but allows a much larger number of authors to be searched. the pre-filtering involved in identifying important authors creates dense, highly connected maps and works best in getting a comprehensive view of an author’s complete publication history on a specific topic. figure 26. practical integration in the reference interview when working directly with patrons during a reference interview i have found that the most effective maps are the advanced boolean search and key papers and citations maps. graduate students and phd’s seem to immediately grasp the idea of mapping a search (advanced boolean search) and mapping references and citations. the diagram of the two point-to-point triangles (see figure 16) is particularly effective in illustrating how the vosviewer map works. the time necessary to create the key papers and citations map makes it less effective than the speed in which a properly set up advanced boolean search map can be spun up. list of papers by doi, list of authors and key authors searches work best on followup. bibliography [1] van eck n, waltman l. 2009. software survey: vosviewer, a computer program for bibliometric mapping. scientometrics. 84(2):523–538. doi:10/cx2w6z. [accessed 2019 may 13]. https://akademiai.com/doi/abs/10.1007/s11192-009-0146-3. [2] van eck nj, waltman l. 2007. vos: a new method for visualizing similarities between objects. in: decker r, lenz h-j, editors. advances in data analysis. berlin, heidelberg: springer berlin heidelberg. p. 299–306. [accessed 2019 may 13]. http://link.springer.com/10.1007/978-3-540-70981-7_34. [3] corrall s, kennan ma, afzal w. 2013. bibliometrics and research data management services: emerging trends in library support for research. libr trends. 61(3):636–674. doi:10/f4zzh4. [accessed 2019 may 13]. https://muse.jhu.edu/article/508619. [4] shachaf p, shaw d. 2008. bibliometric analysis to identify core reference sources of virtual reference transactions. libr inf sci res. 30(4):291–297. doi:10/frdmmf. [accessed 2019 may 13]. https://linkinghub.elsevier.com/retrieve/pii/s0740818808000741. [5] zahedi z, van enjp, cwts, cwts. 2015. identifying topics of interest of mendeley users using the text mining and overlay visualization functionality of vos viewer. 20th international conference in science & technology indicators, 2-4, september, 2015, lugano, switzerland. httpwwwsti2015usichposter-sess. [accessed 2019 may 13]. https://openaccess.leidenuniv.nl/handle/1887/48264. [6] thelwall m. 2018. dimensions: a competitor to scopus and the web of science? j informetr. 12(2):430–435. doi:10/gdqwvj. [accessed 2019 may 13]. http://www.sciencedirect.com/science/article/pii/s175115771830066x. [7] harzing a-w. 2019 may 8. two new kids on the block: how do crossref and dimensions compare with google scholar, microsoft academic, scopus and the web of science? scientometrics. doi:10/gf24xf. [accessed 2019 may 13]. http://link.springer.com/10.1007/s11192-019-03114-y. about the author brett williams (@brettlwilliams, brett.l.williams+c4l@gmail.com) is a senior research analyst at the ontario medical association in toronto, ontario where he acts as the systems librarian and a reference and instruction librarian. he has been been munging library metadata since he first learned how to do an =importxml() in google sheets shortly after graduating from library school at mcgill university in 2008. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – the fachref-assistant: personalised, subject specific, and transparent stock management mission editorial committee process and structure code4lib issue 37, 2017-07-18 the fachref-assistant: personalised, subject specific, and transparent stock management we present in this paper a personalized web application for the weeding of printed resources: the fachref-assistant. it offers an extensive range of tools for evidence based stock management, based on the thorough analysis of usage statistics. special attention is paid to the criteria individualization, transparency of the parameters used, and generic functions. currently, it is designed to work with the aleph-system from exlibris, but efforts were spent to keep the application as generic as possible. for example, all procedures specific to the local library system have been collected in one java package. the inclusion of library specific properties such as collections and systematics has been designed to be highly generic as well by mapping the individual entries onto an in-memory database. hence simple adaption of the package and the mappings would render the fachref-assistant compatible to other library systems. the personalization of the application allows for the inclusion of subject specific usage properties as well as of variations between different collections within one subject area. the parameter sets used to analyse the stock and to prepare weeding and purchase proposal lists are included in the output xml-files to facilitate a high degree of transparency, objectivity and reproducibility. by eike t. spielberg, frank lützenkirchen introduction place gets rare. more and more libraries are confronted with this problem. in their desire to offer more and better equipped working space to students and scientists, they need to evaluate their stock and make room for high level desk space. weeding is a magic key to achieve this goal (martin et al. 2013). due to their size, textbook collections are highly promising candidates for gaining space. at a large university as essen-duisburg with more than 40,000 students, some of the titles are 100 or even more than 200 items strong. but how to proceed and how to quickly identify those items which give rise to the most extensive gain in room while resulting in the smallest disturbance and problems possible for the users? only by a thorough evaluation of the usage in combination with the subject-specific judgement of the subject librarian on which titles are useful (even if 15 years old) and which are not, can this question be answered (cottrell 2013; raphael 2013). while the second part of the question is difficult to replace by automated routines, the first one can be. the main concern is basic data of sufficient quality. these data can be extracted from the library system (aleph (exlibris [accessed 2017]) from exlibris (exlibris [accessed 2017]) in our case). but even if the data are of sufficient quality and are evaluated by automated routines, an interpretation of the results is often difficult for table based information. a clear and understandable visualization allows the user to grab the necessary information quickly (finch and flenner 2016). hence the representation of loans, requests, and stock plays a key role. for several years, a tool called “ausleihprotokoll” (loan protocol) has been used at the university duisburg-essen, written by one of the authors. this web application gives a general overview of the editions and the items present at the library and visualises the temporal evolution of loans and requests in a graphical chart. however, the temporal evolution of the stock was not included. some time had to be spent to acquire this part of the information from tables and to compare it to the loans depicted in the chart. two types of information representation (graphical and tabular) had to be combined, resulting in additional time for information collection and making the decision, if and how many items can be withdrawn. even if it takes only a few seconds for scrolling and gathering the stock information, in the run of analysing hundreds of titles this can add up to a considerable amount of time. in addition, to apply automated analysis these data have to be harmonized, so that the routines can relate the loans to the actual stock at a given time. a project at the university of applied sciences cologne extracted the stock data from the item data and included them in the graphical representation. in addition, a first routine to analyse a single title had been developed (eike spielberg [accessed 2017]). in this paper, we show how this procedure can be extended to whole stock ranges and how personalized parameters are used to calculate subject-specific weeding proposals [1]. our approach usage statistics for one title before expanding to larger stock areas, we explain how the usage statistics for a single title are calculated. given the shelfmark of a title, the corresponding editions are collected from the library system. for each edition, the available items are collected as well as the events (loans and request) for each item. as stated above, these have been displayed in the older version of our tool. a subsequent step was added, calculating similar events from the item information: the date of inventory is used to define an accession event, the date of removing the item as a withdrawal event. we obtain three series of events, in which each series comprises of a start and an end event: loan/return, request/hold and accession/withdrawal. if we look at the temporal evolution of one of these series (for example the loans), a starting event (loan) increases the corresponding counter (number of items simultaneously loaned) while an end event (return) decreases the counter. the three series provide all data necessary for an in-depth analysis. screenshot 1. the upper part of the loan protocol (protokoll) for an item from a textbook collection. the query-form (“abfrage”) is followed by the chart depicting the loans (red), the requests (green) and the stock (blue) for a single shelfmark (= edition, “auflage”). the inclusion of the stock data into the chart of loan/request data already provides a significantly easier and accurate overview over the principal usage characteristics (see screenshot 1)[2]. from this diagram alone, the trends in usage, the relation of loan and stock items and the adequacy of the current stock is easily obtained. as the estimation of items for weeding is usually governed by rather clear and obvious guidelines – how many books have been maximally loaned at the same time, how many should we keep as reserve? – it can be calculated automatically. but how exactly can this be done? relating usage and stock – the weeding potential first, we need to define a time range to be looked at, for example, the last five years. in this time range the absolute maximum number of simultaneous loans is extracted and compared to the stock present today. the difference between these two numbers represents something, which might be called a “weeding potential”. it represents the number of unused items, resting on the shelf when the maximum number of items has been loaned. in a rather aggressive approach, all these items could be selected for withdrawal. however, one should take precautions for the case that items might get damaged or lost. to account for this, every librarian would keep some items as a kind of security reserve. in the fachref-assistant a two-step approach is applied. firstly, the weeding potential is corrected for a static reserve (for example three items or 20% of the stock). secondly, a variable reserve is applied to account for the usage of items (items that are used more regularly have a higher probability to get damaged or lost). for this reason, the ratio between the mean and maximum relative loan is calculated in the given time range. the mean relative loan is calculated by summing up all days the items were loaned and dividing this value by the sum of days the items were in stock. the maximum relative loan is calculated as the maximum fraction of loaned items over items in stock within the time range. a high ratio of these two values indicates a high usage of the individual items, thus making a future replacement more likely. screenshot 2. analysis page showing statistical metrics (“analyse”) and a chart depicting the loans of a single edition with respect to different user groups (“ausleihe nach gruppen”). the table depicts the mean relative loan (“mittlere ausleihe”), the maximal relative loan (“maximale ausleihe”), the maximal number of items simultaneously loaned (“maximale ausleihe absolut”), the actual stock (“aktueller bestand”), the calculated weeding proposal (“vorschlag zur aussonderung”) and the shelfmarks of the items found (“signaturen”). the time range for the analysis can be adjusted via the input field below (“zeitraum ändern”). the chart depicts the distribution of the relative loan with respect to four library user groups: university staff (“interne ausleihen”), students (“studentische ausleihen”), people who are not member of the university (“externe ausleihen”) and other (“andere ausleihen”). the size of the variable reserve is calculated by multiplying a user-defined maximum reserve (e.g. 40 % or 10 items) with the ratio of the mean and maximum relative loan. therefore, a higher ratio results in a larger reserve of items kept to avoid unnecessary costs. using only three parameters – time range, static reserve and variable reserve – weeding proposals can be calculated, which account for many effects in a way very similar to the decisions a librarian makes subconsciously. the results of such an analysis is displayed together with further information about the composition (user groups) in a separate analysis page, accessible from the loan protocol (see screenshot 2). however, the actual values of these parameters may vary strongly from subject to subject. for titles with high, but short-lived relevance (for example genetics or computer science), short time periods might be sufficient to estimate the use in the future. for other subjects such as humanities, the time range to be looked at needs to be much longer. in the case of natural sciences with a very rigid course of lectures, the general textbooks are often required every year at the same time. for lecture courses in german literature, the subject of a seminar might vary from time to time, giving rise to high loans in a series of titles during one semester and with longer periods of neglectable loans. to account for these subject specific problems, the fachref-assistant allows the definition of profiles, a set of parameters used for individual stock ranges. following this approach, the stock in humanities can be analysed with a much longer time range than the stock in chemistry or computer science. for the sake of clarity and to not overload the list of profiles, a personalization option is included, thus allowing each subject librarian to define their own set of parameters for their subjects. from one to many – the weeding and purchase recommendation lists the analysis presented up to now can be performed only for a single title (but with several editions). for several years, lists were prepared at our library by querying the aleph system for some criteria and then thoroughly testing each title. so we query the system with one program, prepare a list and manually test each title by querying the system again with another application. but wouldn’t it be much easier to avoid the intermediate list? to query the system, perform the described analysis right away, and prepare a list only of those titles which have a significant weeding potential? to this end, we implemented a section-based analysis within the personalized part of our application. after login, users with the role subject librarian can switch to the profiles module. here, an individual set of parameters can be defined for each subject area (see screenshot 3). the definition page consists of three major parts: range parameters, loan analysis parameters, and request analysis parameters. screenshot 3. definition of profile for a certain subject. in the first block (“angaben zum bereich”) the subject, collections and media are defined. the second block (“parameter zur ausleihanalyse”) allows for the input of the parameters used to calculate the weeding potential. the last block (“parameter zur vormerk-analyse”) offers parameters used to calculate purchase recommendations based on the requests put on the individual items. range parameters: at the university duisburg-essen, we use a three letter code in the shelfmark as subject categorization. each subject has a certain range (e.g. biology: vna-vtz). in a dropdown menu, the main subject categories can be selected, setting the ranges automatically. in addition, one can define certain ranges or lists of shelfmarks or combinations of both to restrict the analysis to a certain range. afterwards, two fields allow for a restriction to certain collections or media. loan analysis parameters: in the next part, the time range and the two reserves mentioned above are defined as well as a minimum time range; media have to be in stock before they might be selected for weeding. for the display of lists, the minimum value of the weeding proposal can be defined as well. in addition, an optional email address can be entered, to which a bcc of a weeding decision is sent for additional archiving purposes (a copy of the final list is always stored as xml file on the server). request analysis parameters: the last part defines two parameters used to analyse the requests and to make recommendations for extending the stock. again, a time range can be defined and a minimum number of days that people had to wait for their desired item. this choice of the number of days corresponds to the idea of ascertaining a certain service level (“our user does not have to wait longer than 10 days to get his book”). screenshot 4. list of profiles (“profile“) with parameter sets to analyse specific parts of the stock. the two green buttons in each row lead to the weeding (“aussonderung”) and purchase proposal list (“erwerbung”), the blue button with the star starts the analysis, the yellow allows changing the parameters of a profile and the red one deletes the corresponding profile. after a profile has been defined, the analysis can be started from the blue button in the profile overview page (see screenshot 4). after completion, two lists can be accessed: a list of weeding proposals and one of purchase recommendation. an example of such a list is depicted in screenshot 5. using javascript, these lists can quickly be filtered, searched and ordered. the list includes the shelfmark (showing bibliographic details upon hover-over), the collection information, a link to the loan protocol (see above) the trend of the maximum loan over the last ten years, the actual stock, the absolute maximum loan and the weeding recommendation. it also contains a column for comments, which can be used later to give additional information to the give commentaries for the colleagues who perform the actual withdrawal. screenshot 5. list of weeding proposals (“aussonderungsvorschläge”). the columns depict the shelfmark (“signatur”), the collection information (“standort”), a link to the loan protocol (“ausleihprotokoll”), the trend of the maximum loan over the last ten years (“trend/jahr”), the actual stock (“bestand”), the absolute maximum loan (“maximale ausleihe”), a comment (“kommentar”) and the weeding proposal (“vorschlag”). the last column contains buttons to put these titles on a so-called blacklist. titles that shall be protected from weeding (even if hardly used) can be put on this list in order to prevent any further analysis. this might be the case for special collections, for books written by university staff, media used in the library only and so on. the two buttons offer two ways to protect a title: the green one uses a standard entry with a time range set in the personal settings and the blue one allows the staff to set the time period manually and include comments. after working through the list of proposals, saving preservable items to the blacklist and adjusting the numbers of items to be withdrawn, a click on the “aussondern” (withdraw) button at the very bottom of the page generates an email in the user’s local mail program, which can be sent to the members of staff responsible for performing the withdrawal process itself. in this email the shelfmarks, the number of items to be withdrawn and the comments added are listed in plain text. <analysis key="123456789" trend="-0,350" shelfmark="abc123(2)"> <mab> weeding in libraries ... </mab> <meanrelativeloan> 0.349613 </meanrelativeloan> <maxrelativeloan> 0.966667 </maxrelativeloan> <maxloansabs> 31 </maxloansabs> <laststock> 43 </laststock> <proposeddeletion> 12</proposeddeletion> <finaldeletion>12</finaldeletion> <comment /> <totaldaysrequest>0</totaldaysrequest> <numberrequests>0</numberrequests> <maxnumberrequest>0</maxnumberrequest> <proposedpurchase>0</proposedpurchase> </analysis> ... <stockcontrolproperties> <stockcontrol>43_1484895218825</stockcontrol> <collections>e33 e43</collections> <materials>book</materials> <subjectid>43</subjectid> <systemcode /> <minimumyears>5</minimumyears> <yearstoaverage>10</yearstoaverage> <groupedanalysis>false</groupedanalysis> <staticbuffer>0.0</staticbuffer> <variablebuffer>0.0</variablebuffer> <threshold>0</threshold> <yearsofrequests>2</yearsofrequests> <minimumdaysofrequest>5</minimumdaysofrequest> </stockcontrolproperties> listing 1. example – part of the resulting xml-file depicting an imaginary title (analysis-node) and the parameters used to analyse this book (stockcontrolproperties-node). transparency and persistence – using xml-files. with the generation of this email, the list is archived on the server as well. to be as transparent as possible and to allow for a future check-up of performed weeding, the lists are stored as blank xml-files in a separate archive-folder on the server used to run the application disk. not only the items to be withdrawn (finaldeletion-node) are included, but also the proposed number of items (proposeddeletion-node) and the parameters set in the profile (stockcontrolproperties-node, see xml 1). this way, the calculated proposals stay linked to the parameters used to determine the proposed actions. comparing the differences between the proposed number and the final number of items might be used to automatically optimize the chosen parameters. this way a kind of learning weeding agent can be thought of, using machine learning techniques to be as close as possible to the user generated weeding lists. adoption by others – the use of generic mechanisms currently, the application works only with the exlibris (exlibris [accessed 2017]) integrated library system aleph (exlibris [accessed 2017]). in addition, for the use of the profiles a lot of library specific data have to be entered, such as collections, the classification used and so on. for these reasons, a transfer of the application to other libraries might look kind of difficult. therefore, from the very beginning, modules and methods were designed to be as generic as possible. for example, the list of shelfmarks is generated from an xml-file of the used classification and is stored within an in-memory database. in principle, this could easily be adopted to allow for the use of other systematics or to include other types of sources (for example comma-separated values). only the import of the systematics needs to be changed; the internal use of these criteria depends only on the entries stored in the in-memory database and would therefore work also with other data. similar arguments hold true for the use of collections. to manage these a web-frontend is included, allowing the user to define certain ranges. at this point, some options are still hard-coded to the xsl-files which render the web-frontend page, but this could be easily replaced by different mechanisms or just by editing the web-page. the connection to the local database is done via a set of java classes (so-called getters), all located within one package. by replacing the access information, the expressions and mechanisms within these very few classes, an adjustment to other systems could be easily achieved. we therefore believe, that adaptation of this application to other systems need several things to be done, but compared to the overall size of the project, this should be possible and relatively easy. we also started to restructure the application in order to build an independent microservice to access the library system. this approach shall result in a kind of abstract data provider layer. this layer can be adapted for each system without changing the rest of the application. outlook and future work the fachref-assistant has proven its value in our daily life at the university library duisburg-essen. the “ausleihprotokoll” is a common tool to evaluate the usage of single titles and the preparation of lists with the help of the fachref-assistant has been done in many cases. however, we still see a lot of work to be done. one of the main tasks for the immediate future is the adaptation to other library systems, especially to next-generation systems like alma (exlibris [accessed 2017]). three major challenges exist: adopt the getter-classes to other databases / apis include other classifications and collections adopt the mechanisms defining the local settings to allow for an easier setup and transfer. in addition, we would like to separate the application into a set of smaller modules in order to allow for a more agile development. this mainly includes the separation into restful backend services and a web-frontend. this would also make the localization much easier. conclusions we present in this paper the fachref-assistant, a personalized web application for the weeding of printed (and online) resources. it offers an extensive range of tools for evidence based stock management, based on the analysis of usage statistics. the personalization allows for the inclusion of subject specific usage properties as well as the variations between different collections within one subject area. the parameter sets used to analyse the stock and to prepare weeding and purchase proposal lists are included in the output xml-files to facilitate a high degree of transparency, objectivity and reproducibility. the application has been designed to work with the aleph-system from exlibris, but efforts were spent to keep the application as generic as possible. thus, the fachref-assistant supports the library staff by aggregation of objective criteria to achieve higher work efficiency, setting free additional financial and time resources that may be used for other tasks and challenges. acknowledgement the authors thank the university library duisburg-essen for the support for the project. special thanks goes to jutta kleinfeld, nils verheyen and paul rochowski for helpful discussions about technical details. for testing and extensive feedback, we would like to thank christina kläre, rosemarie kosche and felix schmidt. technical details the web application is java/xml based. the mycore (mycore-community [accessed 2017]) framework was used to design the main servlet handling the requests. connection to the library system is established using oracle java database connectivity (jdbc) (oracle [accessed 2017]). upon calling java web servlets with specific url-parameters the data are collected and evaluated according to the user-specified parameters. the results are collected in an xml-document, which is rendered into an html-page via xsl-transformation (w3c [accessed 2017]). to allow user role specific views and option, authentication and authorization was performed using the apache shiro framework (the apache software foundation [accessed 2017]). the general design of the web pages is bootstrap based (@mdo and @fat [accessed 2017]), the charts are generated by the highcharts-package (highcharts [accessed 2017]) and d3.js (bostock [accessed 2017]). the source code is available via github (https://github.com/etspielberg/ub-statistics), documentation (javadoc) under https://etspielberg.github.io/ub-statistics/. about the authors eike t. spielberg (orcid: 0000-0002-3333-5814) studied chemistry at the friedrich-schiller-unviersity in jena and now works as subject librarian and scientific co-worker at the university library duisburg-essen in germany since 2015. he develops tools for automation of library workflows and for new services within the context of bibliometrics. frank lützenkirchen (orcid: 0000-0001-5065-6970) studied business informatics at the university of essen. he is active member and co-founder of the mycore-community and head of the department “digital library” at the university library duisburg-essen. notes [1] an extensive description (in german) of this application has also been submitted as master thesis at the university of applied sciences cologne and will soon be published on the local document repository (https://epb.bibl.th-koeln.de/home). ets acknowledges the support by his supervisors dr. peter kostädt and prof. dr. achim oßwald. [2] as the application was written at a german university the web pages depicted in the screenshots are all in german. a localization has not been performed yet. in captions and references throughout the text, the german expressions found in the screenshots are highlighted in quotation marks. references @mdo, @fat. [accessed 2017 jan 28]. bootstrap: the world’s most popular mobile-first and responsive front-end framework. http://getbootstrap.com/. bostock m. [accessed 2017 jan 28]. d3.js: data driven documents. https://d3js.org/. cottrell tl. 2013. weeding worries, part 1: books. bottom line. 26(3):98–102. doi:10.1108/bl-06-2013-0015. eike spielberg. [accessed 2017 may 17]. aussonderungsassistent für fachreferenten an wissenschaftlichen bibliotheken. malis projekte-blog. http://malisprojekte.web.th-koeln.de/wordpress/aussonderungsassistent-fuer-fachreferenten-an-wissenschaftlichen-bibliotheken/. exlibris. [accessed 2017 jan 28]. aleph integrated library system. http://www.exlibrisgroup.com/de/category/aleph. exlibris. [accessed 2017 jan 28]. exlibris alma. http://www.exlibrispublications.com/alma/. exlibris. [accessed 2017 jan 28]. the bridge to knowledge. http://www.exlibrisgroup.com/de. finch jl, flenner ar. 2016. using data visualization to examine an academic library collection. college & research libraries. 77(6):765–778. en. doi:10.5860/crl.77.6.765. highcharts. [accessed 2017 jan 28]. interactive javascript charts for your webpage. [place unknown]: [publisher unknown]. http://www.highcharts.com/. martin j, kamada h, feeney m. 2013. a systematic plan for managing physical collections at the university of arizona libraries. collection management. 38(3):226–242. doi:10.1080/01462679.2013.797376. mycore-community. [accessed 2017 jan 19]. mycore: das framework zur präsentation und verwaltung digitaler inhalte. de. http://mycore.de/. oracle. [accessed 2017 jan 19]. database jdbc developer’s guide. https://docs.oracle.com/cd/e11882_01/java.112/e16548/toc.htm. raphael l. 2013. killing sir walter scott: a philosophical exploration of weeding. [accessed 2017 jan 23]. in the library with the lead pipe. http://www.inthelibrarywiththeleadpipe.org/2013/killing-sir-walter-scott-a-philosophical-exploration-of-weeding/. the apache software foundation. [accessed 2017 jan 19]. apache shiro: simple. java. security. https://shiro.apache.org/. w3c. [accessed 2017 may 22]. xsl transformation (xslt). https://www.w3.org/tr/xslt. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – editorial introduction: join us at the table mission editorial committee process and structure code4lib issue 22, 2013-10-14 editorial introduction: join us at the table introducing issue 22 by sara amato this issue, born during the long days of summer in the northern hemisphere, focuses on what libraries can bring to the digital table both in terms of special collections and metadata expertise. articles range from an analysis of a large cross institutional collection of ead finding aids, to mixing it up with wikipedia and authority records, to using apache hadoop, apache mahout and html5 to further institutional collections storage and discovery. many thanks to the authors for sharing their work and expertise. if you enjoy reading this journal and would like to contribute to the the code4lib community, please considering applying for a seat on the code{4}lib journal editorial board. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – digital forensics on a shoestring: a case study from the university of victoria mission editorial committee process and structure code4lib issue 27, 2015-01-21 digital forensics on a shoestring: a case study from the university of victoria while much has been written on the increasing importance of digital forensics in archival workflows, most of the literature focuses on theoretical issues or establishing best practices in the abstract. where case studies exist, most have been written from the perspective of larger organizations with well-resourced digital forensics facilities. however organizations of any size are increasingly likely to receive donations of born-digital material on outdated media, and a need exists for more modest solutions to the problem of acquiring and preserving their contents. this case study outlines the development of a small-scale digital forensics program at the university of victoria using inexpensive components and open source software, funded by a $2000 research grant from the canadian association of research libraries (carl). by john durno and jerry trofimchuk introduction back in 2012, marshall breeding wrote: it’s [..]interesting to consider the ways that technology has changed the challenges that libraries face as they accept the collections from retiring scholars or literary figures. in times past, such a collection sold or donated to a library or archive would consist of books and boxes of notes, manuscripts, correspondence, and other physical artifacts. today, retiring scholars might also include their word processing files, research data sets, or even the computers themselves. the files might reside on media long obsolete. how many libraries, for example, still have equipment that can read 5.25″ floppy diskettes that were standard in the early days of computers and word processors? or magnetic tapes from a 1970s vintage mainframe? processing such collections may involve increasing measures of digital archaeology. breeding’s observation had particular resonance for us because shortly prior to its publication our university archives had received a donation of physical materials with inventory lists on 5.25″ floppies. this was not the first time donations had contained such materials, but so far all we had been able to do with them was to store them along with the other physical materials in the collection. of course, this strategy was not optimal, both for the reason mentioned by breeding and also because such media degrade over time. the longer we waited to attempt to extract their contents the less likely we would be able to do so. we decided the time had come to try. as an obvious prerequisite, we would need to obtain equipment that could do the job. at one time, of course, our library had equipment capable of reading old media. but that was back when the media were not yet obsolete. storage space being at a premium, over the years we conscientiously discarded older computers to make room for their replacements, and the last 5.25″ floppy drive had long since left the building. one possibility was of course to buy a complete older system from a vendor specializing in such things, but we were reluctant to go that route for three reasons: vintage equipment tends to be expensive, particularly if it has been refurbished. even with refurbished equipment there would be no guarantee how long it would continue to function. mechanical parts wear out, capacitors burst, and rust never sleeps. while there are off-the-shelf digital forensics solutions, such as the forensic recovery of evidence device (fred)[1] from digital intelligence, these solutions appeared to be overbuilt for our requirements, and with a correspondingly high price tag. our hardware budget for this project was $2000, and we did not exceed it. as a corollary to #2, it did occur to us that we will probably not require this kind of equipment to last forever. eventually there will be no more donations of 5.25″ or even 3.5″ floppies, as the members of the generations that used such things join eliot’s dancers under the hill. and of course it is questionable whether such media will still be readable 20 years further down the line, even if we still have the equipment to do it. bearing that in mind, we decided to aim for a 20 to 25 year operational window. what could we do now, if anything, to ensure we maintained the capacity to read old floppy disks for the next quarter century? it was at this point we decided to look into how possible it was to build an older computer out of new parts. we did some research, concluded it was feasible, and obtained a grant from the canadian association of research libraries (carl) to underwrite the cost of hardware purchase. the rest of this paper describes how we approached the problem, the design considerations as applied to hardware, software and the places where they intersect, and some of the preliminary outcomes. hardware of course, anyone at all conversant with the field of digital forensics could legitimately ask why we did not simply do what everyone else does: use a modern computer equipped with a usb floppy controller (for 5.25″ floppies, the “go-to” solution appears to be the fc5025 floppy controller from device side data, or at the higher end the kryoflux). (reside, 2012) it’s a good question, and before enumerating our reasons i will note that in fact we did build a second workstation with the fc5025, so we have a benchmark for comparison. for the rest of this paper, we will refer to these two computers as the “new/old” computer (old computer built with new parts) and the “usb floppy” computer (workstation with usb floppy controller). reasons why one might want to consider replicating an older hardware platform include the following: even if the fc5025 controller solves the problem of reading 5.25” disks very well (and early indications are that it does), it does not address the problem of reading 3.5” disks. external usb-attached 3.5” floppy drives are still commercially available, but there are reports these have more problems reading older media than do direct attached devices. many can only read high density floppies, not the earlier double density variety (reside, 2012). creating disk images is of course only the first step along the way toward preserving content, which is of course what we care about. you need to be able to mount the disk image or otherwise access the files it contains, and run a variety of older software and operating systems in order to be able to view documents or run older executables. while emulation and format conversion are both possible and necessary (as it would be unrealistic to attempt to duplicate every possible hardware configuration you might encounter), that approach does have its limits. having the ability to hardware-boot older operating systems provides additional flexibility for extracting content from old media. that, at least, was the rationale we started with. we should stress at the outset that we are relative neophytes when it comes to recovering data from old computer media, and were even more so when this project got underway in 2012. jerry is a library systems technician and hardware hacker with many years of experience; john is a librarian and self-taught jack of all trades who manages the library systems department. our goal here is to describe how we approached the problem in the hopes that it will have some value to others involved in the field, or anyone wanting to replicate the functionality of older hardware using new components, possibly for reasons other than simply reading data off of older media. figure 1. ibase mb820 motherboard, cpu and fan. the core component of our hardware strategy was an ibase mb820 industrial motherboard, readily available from multiple suppliers. industrial motherboards differ from their consumer counterparts in that they are designed to support operational continuity for specialized software frequently found in industrial applications. typically, upgrading industrial software simply to keep it compatible with newer computer hardware is not worth the expense. it is more economical to keep the hardware environment constant for as long as possible. thus the mb820 has some interesting and valuable characteristics from our perspective. [2] the mb820 motherboard supports an intel pentium 4 processor, giving it the equivalent processing capacity of a computer from the early 2000s. in terms of supported components, it straddles the space that lies between older legacy devices such as 5.25″ floppy drives, and ps/2 keyboard and mouse, as well as more recent devices such as sata hard drives, an important consideration as ide hard drives are becoming difficult to purchase. [3] with the mb820 as our foundation, we were able to construct a system with the following components: intel pentium 4, 2.8 ghz processor with 512k l2 cache 4gb ddr memory 500gb sata seagate hard drive ati radeon 9200 agp graphics card creative labs soundblaster awe 64 sound card (useful for audio in dos) 5.25″ floppy drive 3.5″ floppy drive cdr/dvd ide optical drive logitech ps/2 keyboard hewlett packard ps/2 mouse these components together allow us to run a rich cross-section of legacy software and operating systems, including ms-dos and windows 3.11, as well as more modern operating systems such as windows xp pro and centos linux 5.10, to take advantage of the different capabilities of each. for example, the motherboard supports both usb and ps/2 input devices, which is useful given that dos does not readily support usb. and of course, on board support for floppy drives obviates the need to utilize usb controllers, making it feasible to directly access the disk drives from within dos, should we ever want to. as mentioned above, the goal of the project was to build an older system out of new parts. we were mostly but not completely successful. the floppy drives themselves (both 3.5” and 5.25”) were purchased secondhand, as were the sound and graphics cards. we were not successful in locating suppliers for new iterations of these older components, and secondhand components were readily and cheaply available. buying spares of key components was already central to achieving our goal of a 25 year lifespan. given that the used parts are less likely to survive for an extended period than were the newer components, we bought additional spares for these in case it proved difficult or expensive to procure replacements in the future. figure 2. bios settings showing floppy drive capacities. software environment as noted above, our “new/old” computer can boot multiple operating systems. however, for day to day use we find the centos 5.x (currently 5.11) partition to be the most flexible. we chose centos 5.11 because it supports both our 32-bit hardware and the cert forensics tools package, which contains most of the utilities we need to acquire and examine disk images.[4] centos 6.x also comes in a 32-bit flavor but the cert package does not appear to be available in a 32-bit version past centos 6.4, which meant we would not have been able to run updates going forward had we opted for the newer version of centos. of course, centos 5.x updates will cease in 2017, at which point we will have to decide whether to simply freeze the machine (and block external network access for security reasons) or upgrade the os. it is worth noting that centos 5.x automatically saw and configured support for both floppy drives during the installation process. this was both unexpected and highly gratifying, as we had anticipated a certain amount of effort would be required to compile the appropriate drivers into the kernel. our “usb floppy” computer runs windows 7 enterprise, with bitcurator running in virtual box.[5] however, as noted below, bitcurator is not currently operational in our environment. disk imaging imaging floppies is an important first step toward their preservation. floppy disks were never the most durable of media, and older floppies in particular need to be handled extremely carefully. at the same time, it is not enough simply to extract their contents, since the goal is to preserve not only the individual files but also how they were organized, timestamps, and any additional metadata that might have been part of the original file system. ideally as exact a copy as possible should be created with the least amount of handling of the original disk. disk imaging addresses both those requirements: first a bit-level copy of the disk is created (hopefully in a single read) and any subsequent content extraction is done from a copy of the image, not the original disk (woods & lee, 2012) however, it turns out acquiring disk images is not altogether straightforward, even after you have the necessary hardware. a variety of imaging software is available, and images can be saved in multiple formats. what’s the best approach? when we consulted the literature it appeared that the best practices for archival disk imaging had been derived from the field of digital forensics, the reason being that both archives and law enforcement have to credibly establish that their images are accurate and complete copies of the original (duranti, 2009, and fox, 2011). however, it is still possible to question how far best practices developed in one field should extend to the other, particularly with regard to file formats for long term preservation. the simplest image format is a raw, bit-level copy of the original source media. this format is output by utilities like the venerable dd command, which has existed in one form or another since the 1970s and is included on every posix-compliant unix system. however there seems to be some consensus in the literature that the advanced forensics format (aff) is preferred, largely due to its ability to store user-created metadata in the same file as the disk image, along with other features like digital signatures and more efficient storage capabilities. (garfinkel et. al., 2006) but in the context of long-term preservation, and particularly in the context of the minimal storage requirements presented by floppy disk images, the use of a more complex and evolving format like aff seems to us to present at least as many drawbacks as advantages. our confidence was not increased when we determined that earlier versions of aff have since been deprecated for future use (forensics wiki, n. d.). we believe raw images are more likely to be readable over the long term than are more specialized formats. in addition to dd, we are exploring the use of other tools for imaging disks. one of these is a proprietary program that ships with device side data’s fc50525 usb floppy controller. for imaging 5.25” floppy disks this is turning out to be our best option, both for its overall ease of use and for its ability to read disks that dd cannot. its biggest weakness is that it does not log damaged sectors, even though it briefly displays this information on screen while the reads are in progress. it outputs raw disk images. figure 3. disk type options for the fc5025 usb floppy controller. bitcurator is another, more complex, suite of tools that is gaining significant traction in the digital preservation space, and we are only just beginning to explore its capabilities. a custom distribution of ubuntu, bitcurator features the open source guymager as its primary imaging tool. guymager is flexible and relatively user-friendly imaging software that can output raw, aff and ewf images along with checksums and metadata [6]. however, as of this writing we have had no success using bitcurator for imaging floppies of either the 3.5” or 5.25” variety. it has a 64-bit requirement and so does not run on our “new/old” 32-bit machine. and while it will run under virtual box on our “usb floppy” workstation and does see the usb floppy drive, it freezes when attempting to read from it. we have not yet determined what is causing this problem. however, it should be noted that while bitcurator is a very convenient way to obtain and use a wide range of forensics tools, most of its software components are available in other ways (such as via the cert forensics tools package mentioned above) and they can compile and run quite readily in our 32-bit centos 5.x environment. finally, one of the bigger imaging challenges we have encountered to date has been simply determining what combination of technologies wrote data to the disks in the first place. even taking a bit-level copy of a disk, it can be critical to know, for example, whether the data it contains was saved by an apple ii or a dec rainbow 100, and which version of operating system it was running at the time. it is perhaps surprising how infrequently this kind of metadata accompanies the older floppy disks in our archives. content retrieval to illustrate the kinds of challenges posed by older media we will describe one of our earliest projects, imaging and extracting content from a box of twelve 5.25” disks in the fonds of a local (vancouver island) poet, rona murray. at the outset we knew that the disks were approximately 30 years old, from a variety of manufacturers, all single-sided double density. a label pasted on the side of the box indicated they contained files created in wordstar 2.2 and in addition, each disk was labelled with a list of the files it contained. however, we did not know what kind of computer the author had used. our first attempts to image the disks failed. we used ‘dd’ with some standard configuration options but it was unable to pull an image from any of the disks we tried. thinking the disks may have been corrupted we then employed a program called ‘ddrescue’ to see if we could recover data that way, as ddrescue reads at a lower level than dd. using ddrescue we did manage to extract a considerable amount of data but the logs indicated significant error rates. we were not able to mount the ddrescue images so we were not able to view their contents using standard file system tools. grasping at straws, we then dumped the binary contents of the images to screen using the unix ‘more’ command. and this is what displayed: figure 4. binary data dump to terminal. it is interesting to note that, even if we had not been able to further refine our process much of the content would still have been readable, thanks to the fact that wordstar saved files primarily in plain text (albeit 7-bit ascii). but even better, wordstar also indicated which version of itself created the file, in this case the “kaypro ii 64k cp/m” version. armed with this information, we were able to go back and pull some very good images off ten of the twelve disks using the usb floppy drive, this time using device side data’s proprietary imaging tool (which has a built-in setting for kaypro ii cp/m disks). unfortunately two of the twelve disks were visibly corroded and many sectors were unreadable, but overall this was a much higher success rate than we anticipated. in this case, of course, our ability to run and emulate a range of historic computing environments was not helpful for accessing the contents of the disk images, given that cp/m is not yet one of the environments we can emulate. fortunately we were able to use michael haardt’s cpmtools package [7] on a modern ubuntu linux workstation to list the contents and extract individual files. although we currently lack a copy of wordstar, we were able to open the files easily in a variety of backwards-compatible applications, such as wordperfect 12. our experience so far has been software like dd and guymager have a much higher success rate for imaging floppies dating from the late 1980s through the 1990s, when disk geometries had become more standardized and the biggest compatibility issues were between microsoft and apple systems. imaging and retrieving content from disks dating from the early 1980s requires higher levels of effort and expertise. not all of our recovery projects have been successful. our attempt to recover files from a set of 5.25” floppies that accompanied a 1984 masters’ thesis did not result in anything useful. in this case, the disks had been stored in the back of a hardbound thesis on the public shelves in one of our branch libraries, not in our climate-controlled archives. they contained executable programs written on an apple ii in an early educational software called pilot, and data had been written on both sides, (so-called ‘flippy disks,’ a fairly common practice back in the day, but not encouraged by the disk manufacturers, who controlled for quality only on one side). given our minimal success in reading any data at all from these disks, we suspect they may have passed through a library desensitizer at least once in the previous 30 years. an object lesson, if one is needed, that floppy disks are fragile media and need to be treated as such. we are pleased to report that the hardbound thesis is still quite legible, however. long term storage of course, simply having the technological capability to image old media is only a first step [8]. we are only beginning to work out how we want to operationalize the storage of images going forward. we will need to work through a number of issues, such as: metadata: what metadata do we want to capture for each image? in addition to the usual subject classification, it would be very useful to note any additional information we can gather about the original software environment, as well as what tools and settings we used to capture the images. it would also be useful to capture any sector read errors reported during the disk creation process. then there is the question of metadata formats. guymager provides some useful functionality in this area, but may not be the best choice for imaging all of the disks in our collection, particularly the early ones. format: what image formats do we want to preserve? as noted above, the raw file format seems to us like a reasonable choice. others may disagree. given the small size of these images it would be possible to store them in multiple formats if necessary. content: how much effort do we want to put into extracting and forward-migrating the files contained on the disk images? creating images is fairly trivial if you know what was used to write to the disk in the first place. extracting content from the images is a variable process: it can be relatively time consuming and technical, as illustrated in our example above; or it can be easy if the disk images can be mounted and explored using standard file system utilities.given that there is a pressing need to image many of these disks before they become unreadable, it may be better to focus our efforts on imaging and only do minimal content sampling to ensure the imaging process was successful. organizing files: what is the best way to organize the files so that their conceptual and physical relationships are preserved in a digital storage system? storage system: our library recently acquired an instance of archivematica, an open source, oais compliant digital preservation system from artefactual systems. this would appear to be our best long-term storage solution, particularly given that it recently (version 1.2, september 2014) acquired the ability to ingest forensic disk images [9]. we have not yet tested this capability, however. conclusions although we are in the early stages, we can provisionally conclude that having a variety of hardware and software options is a good thing when it comes to working with early media and their contents. in general we have so far found that our “new/old” workstation provides more flexibility when it comes to working with older media than does our “usb floppy” workstation, and consequently it has been the one we use first when confronting a set of archival floppies for the first time. yet as noted in the example above, the “usb floppy” workstation is better for certain specialized applications, notably imaging 5.25” disks from the early 1980s. so far the ability to boot into multiple operating systems has not been all that useful for retrieving older content. most of the early documents we have so far encountered have been in some version of ascii with minimal proprietary formatting, and conversion tools for many early document types are still readily available. this is not to say that we won’t ever encounter cases where having these capabilities will be of help to us, only that we haven’t yet. however, we are beginning to give some thought to contexts in which the multiple boot and emulation options could be very useful indeed. it seems probable to us that systems constructed around industrial motherboards could have a wider application than disk imaging. as legacy systems become increasingly difficult to maintain and expensive to purchase, systems like our “new/old” machine could bridge the gap for researchers wishing to study old software running in its natural habitat. douglas reside (2010) has observed that “… some of the most useful tools for migrating data from an obsolete to a modern (or at least slightly less obsolete) format are those computers that were manufactured at a moment when a popular new media format or transfer protocol had just emerged. such computers often have ports or drives, along with associated drivers, capable of using older, and in their time more common, technologies as well as new ones.” he calls these kinds of computers “rosetta machines”. in reside’s view the rosetta machine “par excellence” is the macintosh wallstreet powerbook, because having been manufactured in mid-1998 it came with swappable cd, dvd, floppy drives capable of reading 400k and 800k disks, and could accommodate a swappable zip drive. it had an ethernet port and usb capability (via onboard pcmcia slots), and it is capable of running older versions of linux. its’ one main drawback is the lack of a 5.25” floppy drive. although we did not set out to build a rosetta machine according to reside’s specifications, we were pleased to note that our “new/old” workstation meets or exceeds every one of them. this, coupled with the fact that the supply of wallstreet powerbooks is finite (and ever more so as time takes its toll), suggests to us that rather than focusing on keeping old computers alive, building them out of a combination of new and secondhand parts may represent a more viable strategy for reading older media over the medium to long term. acknowledgments the authors would like to gratefully acknowledge the canadian association of research libraries (carl) for providing a research grant to support this project. notes [1] see http://www.digitalintelligence.com/products/fred/ [2] for complete specs, see http://www.ibase-i.com.tw/mb820.htm [3] whereas in the fall of 2013 it was still possible to purchase ide drives through ncix, as of a year later ide drives were not available from the same supplier. [4] for more information on the cert linux forensics tools repository, see https://forensics.cert.org/ [5] bitcurator is available at http://www.bitcurator.net/ [6] see http://guymager.sourceforge.net/ [7] michael haardt’s cpmtools package: http://www.moria.de/~michael/cpmtools/ [8] in fact, developing the technology before developing a preservation plan is probably not the optimal order of precedence. see ricky erway (2012), you’ve got to walk before you can run: first steps for managing born-digital content received on physical media. dublin, ohio: oclc research. retrieved from http://www.oclc.org/research/publications/library/2012/2012-06.pdf [9] see https://www.archivematica.org/wiki/archivematica_release_notes references breeding, m. (2012). from disaster recovery to digital preservation. computers in libraries, 32(4). retrieved from http://librarytechnology.org/ltg-displaytext.pl?rc=16821 duranti, l. (2009). from digital diplomatics to digital records forensics. archivaria 68. retrieved from http://journals.sfu.ca/archivar/index.php/archivaria/article/view/13229/14548 forensics wiki.(n. d.). aff. retrieved from http://www.forensicswiki.org/wiki/aff fox, r. (2011). forensics of digital librarianship. oclc systems & services: international digital library perspectives, 27(4). retrieved from http://dx.doi.org/10.1108/10650751111182560 garfinkel, s. l., malan, d. j., dubec, k., stevens, & c., pham, c. (2006). advanced forensic format: an open, extensible format for disk imaging. retrieved from http://nrs.harvard.edu/urn-3:hul.instrepos:2829932 kirshenbaum, m., ovenden, r., & redwine, g. (2010) digital forensics and born-digital content in cultural heritage collections. washington, d.c.: council on library and information resources. retrieved from http://www.clir.org/pubs/reports/pub149 reside, d. (2010). rosetta computers. in m. kirshenbaum, r. ovenden, & g. redwine, digital forensics and born-digital content in cultural heritage collections. washington, d.c.: council on library and information resources. retrieved from http://www.clir.org/pubs/reports/pub149 reside, d. (2012). digital archaeology: recovering your digital history. retrieved from http://www.nypl.org/blog/2012/07/23/digital-archaeology-recovering-your-digital-history woods, k. & lee, c. (2012). acquisition and processing of disk images to further archival goals. archiving 2012, copenhagen, denmark, june 12-15, 2012.retrieved from http://ils.unc.edu/callee/archiving-2012-woods-lee.pdf about the authors john durno (jdurno@uvic.ca) has been head of the library systems unit at the university of victoria since 2006. jerry trofimchuk (jtrofimc@uvic.ca) has been a technician in the library systems unit at the university of victoria since 1998. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – enhancing print journal analysis for shared print collections mission editorial committee process and structure code4lib issue 51, 2021-06-14 enhancing print journal analysis for shared print collections the western regional storage trust (west), is a distributed shared print journal repository program serving research libraries, college and university libraries, and library consortia in the western region of the united states. west solicits serial bibliographic records and related holdings biennially, which are evaluated and identified as candidates for shared print archiving using a complex collection analysis process. california digital library’s discovery & delivery west operations team (west-ops) supports the functionality behind this collection analysis process used by west program staff (west-staff) and members. for west, proposals for shared print archiving have been historically predicated on what is known as an ulrich’s journal family, which pulls together related serial titles, for example, succeeding and preceding serial titles, their supplements, and foreign language parallel titles. ulrich’s, while it has been invaluable, proves problematic in several ways, resulting in the approximate omission of half of the journal titles submitted for collection analysis. part of west’s effectiveness in archiving hinges upon its ability to analyze local serials data across its membership as holistically as possible. the process that enables this analysis, and subsequent archiving proposals, is dependent on ulrich’s journal family, for which issn has been traditionally used to match and cluster all related titles within a particular family. as such, the process is limited in that many journals have never been assigned issns, especially older publications, or member bibliographic records may lack an issn(s), though the issn may exist in an oclc primary record. building a mechanism for matching on issns that goes beyond the base set of primary, former, and succeeding titles, expands the number of eligible issns that facilitate ulrich’s journal family matching. furthermore, when no matches in ulrich’s can be made based on issn, other types of control numbers within a bibliographic record may be used to match with records that have been previously matched with an ulrich’s journal family via issn, resulting in a significant increase in the number of titles eligible for collection analysis. this paper will discuss problems in ulrich’s journal family matching, improved functional methodologies developed to address those problems, and potential strategies to improve in serial title clustering in the future. by dana jemison, lucy liu, anna striker, alison wohlers, jing jiang, and judy dobry background west, western regional storage trust, founded in 2010, is a distributed shared print journal repository program comprised of 68 members, plus five past members who continue to retain journals on behalf of west [1]. member institutions participate in a number of ways depending on local resources and capacity, for example by: committing to retain titles on behalf of west on site (i.e., in campus libraries) or in dedicated storage facilities; physically validating retained titles for completeness and/or condition; actively seeking holdings from other members to fill gaps (i.e. missing volumes) in retained titles; offering volumes to fill gaps in retained titles; and supporting west’s objectives by taking part in the program’s governance and strategic decision-making. west members, on agreeing to commit to archiving proposals made by the west operations and collections council (west-occ), record information about the proposals in their library and consortial catalogs in the form of marc holdings records following program guidelines. these holdings are submitted to west-ops once a year for inclusion in a local database managed by the california digital library (cdl), which supports ongoing collection analysis and local comparison functionality. west archival commitment files in the form of marc holdings records are supplied by west-ops once a year in the form of marc holdings records for inclusion in the print archives and preservation registry (papr) of the center for research libraries (crl). the papr system supports the archiving and management of serials collections by providing comprehensive information about titles, holdings, and the terms and conditions of archiving for major print archiving and shared print programs. in addition to submitting to papr, member institutions also supply these same archival commitments to oclc as marc holdings records in order to also surface these holdings in worldcat. shared print programs strive to ensure ongoing, long-term access to the scholarly print record, while also allowing member institutions to optimize campus library space. it is of the utmost importance that the functionality of the process includes as many member held titles for consideration as possible, while still retaining accuracy in the process of pulling together related titles to build journal families. collection analysis west collection analysis is predicated on titles being organized into clusters, based upon the ulrich’s journal family id. ulrich’s assigns a unique journal family id to related titles such as preceding, foreign language parallel, supplemental, and other interconnected titles. titles that are successfully matched with journal families are candidates for collection analysis at both the family and the individual title level. titles submitted by member institutions are matched in ulrich’s using issn, and assigned a journal family id. titles which do not match are considered to be “orphan” records, and are removed from the process stream at that point. after the matching process, titles having matched with a journal family go through a number of exclusionary filters to identify only those titles which have an optimal value-add to shared print archiving. records that do not successfully pass through the exclusion filters are dropped from the process stream. exclusion filters include: west previously archived titles non-print formats government documents lc classification k (law) title keywords “online” and “monograph” location exclusions as directed by the owning institution. for example, exclude any titles with location “reference desk, current periodicals, rare materials.” the remaining titles are organized and clustered into their relevant journal families and are output with their summary holdings and other various data points as reports. west staff use the reports to further refine for title review and proposal by west-occ to west members for shared print archiving. internal processing, pre-enhancements before internal functionality enhancements were made, titles were matched on limited, selected data fields found in marc bibliographic serial records furnished by west members. west-ops loaded the bibliographic records and local holdings to a database and the following input fields were isolated as match candidates with ulrich’s journal family issns: primary issn (022$a) linking issn (022$l) former title issn (780$x) succeeding title issn (785$x) west-ops further enhanced the data by using the oclc search api to harvest 022$a, 022$l, 780$x, 785$x fields from primary oclc records (master records) and loading them to a dedicated table (table 1). this data, in most cases, was a supplement to the supplied data in west member records and, in some cases, was a correction to one or more submitted values. it should be noted that data harvested from the oclc primary record was always assumed to be most current and most correct, and so took precedence over input values in ulrich’s match search order. table 1. oclc harvested supplementary issn values in the issn table. issn values are harvested using the oclc search api. the oclc_primary number connects data from this table to matching records in the source data table (by way of the oclc_xref table) which stores full bibliographic details. oclc_primary primary issn linking issn former issn succeeding issn 1102013 1001-3456 1001-3456 3344-9930 9131027 3498-1209 3399-2020 1443-9008 6775-6678 in addition to issns, the oclc primary number and any cross reference oclc numbers [2] harvested were stored locally as an oclc number “set” to prevent the searching of a title more than once (table 2). during processing, the primary oclc number from each contributor record was first searched in a dedicated table containing harvested primary and cross-reference numbers (table 2.) if a match was found, the oclc number set would then be associated with that contributor record in the source data table. if no match was found, then a new search would be done using the oclc api. any issns and oclc numbers gleaned from the matching oclc record would then be harvested to further populate and enhance the issn and oclc_xref tables for further searching. this strategy specifically addresses the ubiquitous issue of multiple oclc numbers (primary and xrefs) associated with the same oclc record, but which may show up in local catalogs as primary values. multiple oclc numbers associated with a single oclc record occur when oclc detects duplicate records in their union catalog. duplicate records are weighed to determine which is of better quality. holdings are then moved over to the better record, the “winner”, while the “loser’s” oclc number is inserted into 019$a [3] as an oclc cross-reference number (xref.) since a bibliographic record in a local catalog is, in essence, a snapshot of the primary record in oclc, its primary oclc number may deprecate to a cross-reference in future. it is therefore important to have a complete set of primary + cross-references when searching and matching on oclc number. table 2. oclc harvested supplementary oclc values in the oclc_xref table. oclc number sets are stored so that any unique title isn’t searched more than once by the oclc search api. the oclc primary or x-ref number connects this table’s applicable oclc number “set” to the source data table which stores full bibliographic details. oclc primary number oclc x-refs 1102013 23456, 89076, 10003322 9131027 334467 following the oclc api harvesting process, the system then attempted to match records to ulrich’s journal families, using issn values. for each source table title, issns from both the source data table and the issn table were searched in ulrich’s, in the order listed below, until a match was found [4]: oclc record harvested primary 022$a oclc record harvested primary 022$l oclc record harvested preceding title 780$x oclc record harvested succeeding title 785$x source table record primary 022$a source table record primary 022$l source table record proceeding title 780$x source table record succeeding title 785$x following ulrich’s journal family matching, records went through the title exclusionary filters (as described in the previous section), eliminating government documents, non-print formats, etc. slightly less than half of member-contributed records (48%) in the pre-enhancement process matched with an ulrich’s journal family, leaving approximately half (52%) that did not (orphan records). there were several reasons why records did not match: no issn, primary or otherwise, was assigned to that record. this is especially common in older titles, which unfortunately, are often those most at risk. low-level records [5] sometimes lacked an oclc number, nullifying the oclc search api harvesting process, consequently omitting potential, additional issn match candidates for ulrich’s. titles were not found in ulrich’s despite oclc number and issns in source and harvested data. based on ulrich’s internal criteria, not every title is eligible for inclusion, or it may be that while the title is marked for inclusion, ulrich’s hasn’t yet captured that title as part of their latest database update. figure 1. overview of internal process, pre-enhancements. internal processing post-enhancements to address the approximately 900,000 orphan records comprising a little more than half of member contributed records (52%), the west-ops team devised a more complex match strategy using additional control number values, resulting in a significant improvement in ulrich’s journal family matching. the processing steps are identical, with the addition of several enhancements (figure 2). additional issn values from both the source table data and the oclc harvested values, 770$x, 772-777$x, 786-787$x, are searched in ulrich’s as part of the match journal family step. this captures additional matches on the following ulrich’s title categories: abridged edition abridged edition of alternative frequency edition alternative media edition cumulative edition cumulative edition of issued with title parallel language edition partial translation edition partial translation edition of regional edition seasonal edition seasonal edition of special edition special edition of supplement supplement to translation translation of additional control numbers (lccn, coden, oclc) are harvested from both source data and oclc to form an expanded set of match values. these values are used for a more robust issn search in ulrich’s, as well as an additional matching strategy which matches any remaining orphan records with records that have already been assigned a journal family id; orphan records that match are assigned that same journal family id. this expanded match set includes: oclc numbers [6],[7]: primary, 019$a, 035$a$z, 079$a, 770$w(ocolc), 772-777$w(ocolc), 780$w(ocolc), 785-787$w(ocolc) lccn numbers: 010$a, 770$w(dlc), 772-777$w(dlc), 780$w(dlc), 785-787$w(dlc) issn numbers [8]: 022$a$l, 770$x, 772-777$x, 780$x, 785-787$x coden numbers: 030$a, 770$y, 772-777$y, 780$y, 785-787$y as orphan records are searched and matched, their control numbers are pooled and deduplicated with the target record’s control numbers. this provides an expanding pool of control number values from which subsequent orphan records can match. if an orphan record does not match, it is shuttled to the “end of the line,” in hopes that as additional orphans match and expand the pool of match targets, these remaining orphans will eventually find matches with target records. when all orphan records in line have failed to match with a target record pool, this match processing step ends. the unmatched orphan “line” may cycle through numerous times until matches are no longer found. added control number values, both primary (019$a, 035$a$z, 079$a, 010$a, 022$a$l, 030$a), and from relevant linking data fields (770$wxy, 772-777$wxy, 780$wxy, 785-787$wxy), allow for multiple, highly reliable matching options, barring member submitted record content errors [9]. matching on records that previously successfully matched with ulrich’s provides a back-door method to garner new matches for records which otherwise would have been dropped from collection analysis. the oclc search api is employed to make a final pass over orphan records to harvest any additional oclc numbers and issns that may have otherwise been missed. a last attempt is then made to match any of these remaining orphan records with records previously matched in ulrich’s. orphan records that match with a record with a journal family id are assigned that same journal family id. in this final step, oclc-harvested oclc numbers and issns from remaining orphans are searched in the following order: oclc numbers in the following order: primary, 019$a. if no match, then issn numbers in the following order: 022$a, 022$l, 780$x, 785$x. it should be noted here that the oclc search api was set up in previous development cycles to harvest a limited set of additional, or corrected, issn values 022$a$l, 780$x, 785$x. ideally, this would have been refactored to include the full, expanded control number set of 010$a, 019$a, 022$a$l, 030$a, 035$a(ocolc)$z(ocolc), 079$a, 77[02-7]$w(dlc)$w(ocolc)$x$y, 78[05-7]$w(dlc)$w(ocolc)$x$y. however, because of the significant development time this expansion would require, we were unable to fold this particular enhancement into this project. we hope to add this expanded set into post-enhancement step 4 (see following section) during a future development cycle, anticipating that it will further improve our match rates. figure 2. overview of internal process, with enhancements.. results post-enhancements table 3. processing post-enhancements results. steps 3 and 4 were done as part of the processing enhancement project only, illustrating the improvement in journal family matching using the expanded set of control numbers for ulrich’s matching, and matching orphans to records with journal families as an added strategy using an expanded control number set.   baseline(a) step 1(b) step 2(c) step 3(d) step 4(e) total records 1,721,416 1,721,416 1,721,416 1,721,416 1,721,416 matched journal family 0 783,983 825,419 920,619 921,127 did not match journal family 1,721,416 937,433 895,997 800,797 800,289           matched journal family           … and issn(s) in source data 0 783,983 789,214 815,273 815,409 … and no issn(s) in source data 0 0 36,205 105,346 105,718           matched journal family           … and source data lccn(s) only 0 0 0 681 683 … and source data coden(s) only 0 0 0 3 3 … and source data oclc#(s) only 0 0 0 39,599 39,833             baseline preprocessing source data table captures counts of primary control number values 022$a$l, 780$x, 785$x in input marc record. match ulrich’s journal family using source data issns. marc fields 022$a$l, 780$x, 785$x. match ulrich’s journal family using additional oclc api harvested issns. marc fields 022$a$l, 780$x, 785$x. match ulrich’s journal family using expanded control numbers from source data, then match remaining orphan records to records with journal families using expanded control numbers. marc fields 010$a, 019$a, 022$a$l, 030$a, 035$a(ocolc)$z(ocolc), 079$a, 77[02-7]$w(dlc)$w(ocolc)$x$y, 78[05-7]$w(dlc)$w(ocolc)$x$y. harvest any remaining control numbers from oclc to match remaining orphans to records with a journal family. marc fields: 019$a, 022$a$l, 035$a(ocolc), 780$x, 785$x. a set of 1,721,416 contributed bibliographic serial records was used as a baseline (table 3) pre-enhancement processing steps 1 and 2 were run, resulting in an ulrich’s match rate of 48% (825,419). 895,997 records (52%) failed to match with ulrich’s data. pre-enhancement processing step 1 used source data issns from 022$a$l, 780$x, 785$x, while step 2 used additional, or corrected, issns where available, from oclc (022$a$l, 780$x, 785$x) were used to match with ulrich’s issns. following the enhancement of the process, and using the same baseline set of bibliographic source data, steps 3 and 4 were run following steps 1 and 2. fifty-four percent (54%) of records (921,127) matched with a journal family, a positive increase of +6%, leaving 46% of records (800,289) still orphaned. if we review improvements based on a percentage of increased records ((enhancement total – pre-enhancement total)/pre-enhancement total), we gained +11.6 percent (95,708 records.) step 3, matching records to ulrich’s journal family using an expanded set of control numbers from the source data and, following this, attempting to match any remaining orphan records using an expanded control number set to records with an ulrich’s journal family, garnered the most significant gain in journal families, +95,200 (~10%.) step 4, a final search of oclc for a limited set of issns and oclc numbers (019$a, 022$a$l, 035$a(ocolc), 780$x, 785$x), gained very few additional titles, +508 (~.06%), but in light of the total number of journal titles proposed, (table 4), this may have had some significance.[10] in examining matching and prevalence of controls numbers at each step in the updated processing routines, we found that expanding our control number match set from the source data had a significant effect where the records had no, or limited, issn values in the source data: 105,718 records lacked base set issns (022$a$l, 780$x, 785$x) in the source data. 36,205 of these found matches following the first oclc api harvest of base set issns. an additional 69,513 records were added when their additional linking data field issns (77[02-7]$x, 78[67]$x) were used to match  with  ulrich’s journal families or match with those records  previously assigned a journal family id.. 40,519 titles were gained where there was no available issn for matching in ulrich’s. these records had one or more lccn or, one or more codens or, one or more oclc numbers only. these control numbers were used to match with a record having already been assigned a journal family id. conclusion following enhancements, a total of 921,127 records (54%) were assigned a journal family id. this was an improvement of 6% over the previous collection analysis cycle, pre-enhancement (table 4) we theorize that this gain is primarily comprised of older titles lacking issns and/or more obscure titles which have a tendency to be represented by brief cataloging records in local catalogs,[11] titles that are, furthermore, more likely at risk due to age and rarity. marc records for older and more obscure titles can be lacking the expansive metadata of more recently published titles, as they may lack access points that were not implemented and used until decades following the development of marc standards; older title metadata, for example, may be limited to data transcribed from catalog cards to marc when a library first moved to an automated library catalog. where a retrospective catalog conversion [12] would otherwise upgrade these types of records, they still lurk in many library catalogs. performing oclc api searches on these records can be thought of as an abbreviated retrospective conversion. this search fleshes out control numbers from primary and linking data fields garnered from the oclc primary record, the “record of record”, which generally speaking, is the fullest and most accurate version. these additional control numbers then allow for enhanced matching capabilities with ulrich’s, and on records having already matched with ulrich’s, using the full array of, and most correct values known to an individual title as expressed in the oclc “record of record.” table 4. record total overview comparison, preand post-enhancement, journal families. pre-enhancement number of records percent of total post-enhancement number of records percent of total total contributor records loaded 1,922,108 100% 1,721,416 100% records loaded with issn 1,068,059 56% 977,937 57% records matching ulrich’s journal family 917,032 48% 921,127 54% the collection analysis report, seeded by journal titles which successfully matched with an ulrich’s journal family, and subsequently passed exclusion criteria, was likely improved. after correcting for duplicate titles shared across contributors, 191,750 unique titles were included in the 2020 collection analysis report. of these, 8,051 titles were proposed for archiving in 2020 (table 5) table 5. record overview, post-enhancement, collection analysis report and titles proposed. pre-enhancement number of records, 2018 % of total post-enhancement number of records, 2020 %of total total contributor records loaded 1,922,108 100% 1,721,416 100% number of titles in collection analysis report [13] 197,956 10% 191,750 11% number of individual titles proposed 8,074 .42% 8,051 .47% of significance is the relatively small number of records which are ultimately included in the collection analysis report, and of these, the even smaller number of titles that are proposed for archiving. this would indicate that while overall improvement was only a positive gain of 11.6%, these 95,708 records comprise a significant pool from which titles may be gleaned for a relatively smaller set of archiving proposals. the number of individual titles included in the collection analysis report, and proposed after enhancement, improved proportionally between 2018 and 2020 by a small margin (table 5.) however, due to changes in exclusion criteria and other considerations between collection analysis cycles, it’s impossible to draw any sort of concrete conclusions at such a granular level when comparing across cycles. unfortunately, due to the complexity of programming efforts it would take to track individual field and process step matching, we were unable to determine how many additional titles were captured and added to the final set of archive proposals. so, while the overall number of records with matches to journal families may have increased with enhanced matching routines by 11.6%, we do not know how many of these specific additions may have dropped out during the exclusion process, or, having passed, subsequently failed to be proposed for archiving. ideally, we’d be able to identify which titles were added post-enhancement, if any, to determine how successful this project was. to improve tracking and matching, further enhancements might include: expanding on control numbers harvested from oclc in step 4 for inter-record matching, including lccn and coden numbers, both base and linking data values. using tightly controlled and normalized author/title string matching between records using primary values from 1xx/2xx fields with appropriate linking data field contents 77x-78x. adding tracking fields to database records to trace marc field and subfield match values in steps 3 and 4 for further analysis as to which fields produce the highest match rate. adding tracking fields to database records see which records having gone through step 3 and step 4 matching processes were included in the collection analysis report, and were subsequently nominated for archiving. move from the ulrich’s model to an entirely in-house model, where control number and author/title string matching are used outside of ulrich’s to cluster titles together for “west journal families.” bibliography california digital library. 2019. discovery & delivery. california digital library, services and projects. [online] may 2019. https://cdlib.org/services/d2d/ california digital library. 2020. west membership. [online] june 2020. https://cdlib.org/west/about-west/west-membership/ california digital library. 2020. west: western regional storage trust. [online] may 2020. https://cdlib.org/west/ center for research libraries. 2020. center for research libraries. global resources network. center for research libraries. global resources network. [online] 2020. https://crl.edu center for research libraries. 2020. papr – print archive and preservation registry. [online] 2020. http://papr.crl.edu/ library of congress. network development and marc standards office. 2020. marc code list for organizations. marc standards. [online] may 2020. https://www.loc.gov/marc/organizations/ library of congress. network development and marc standards office. 2020. marc standards. [online] march 2020. https://www.loc.gov/marc/ library of congress. network development and marc standards office. 2003. the lccn namespace. marc standards. [online] november 2003. https://www.loc.gov/marc/lccn-namespace.html oclc. 2007. technical bulletin 254, oclc-marc format update 2007 and institution records to accommodate the rlg union catalog. s.l. : oclc, 2007. 1097-9654. proquest. 2020. ulrichsweb.com(tm) — frequently asked questions. ulrichsweb global serials directory. [online] 2020. https://www.ulrichsweb.com/ulrichsweb/faqs.asp notes [1] as of 08/2020. https://cdlib.org/west/. [2] found in 019$a and 035$z. [3] 019$a contents in oclc are duplicated in 035$z in exported records. therefore, these are found in either, or both 019$a and 035$z in records evaluated during these processes. [4] oclc primary record issns were searched first, as these were deemed to be most current and therefore most correct. [5] a low-level record is considered to be one that is brief in nature, often input “on-the-fly” in the local catalog, and will often lack an oclc number and other metadata descriptive elements. [6] note that for matching purposes, the organizational code as supplied in parentheses, is used in tandem with the control number, to identify control number type. https://www.loc.gov/marc/ [7] 079$a is the location of the oclc primary number in oclc institution records. institution records (irs) were created to accommodate rlg clustering practices when rlg merged with oclc in 2006. irs are no longer in use, but 079$a’s containing an oclc master number may still exist in legacy records residing in local catalogs. (oclc, 2007) [8] lccns were normalized for optimal matching using the lccn namespace documentation. https://www.loc.gov/marc/lccn-namespace.html [9] garbage in/garbage out rule. [10] unfortunately, due to the complexity and programming efforts it would take to track field and process step matching, we are unable to determine how many additional titles captured with updated processing were added to the final set of archive proposals. [11] because, at this time, we are unable to track titles matching under the new processing routines, we have no way of knowing the composition of the gained set of title matches. [12] a process used to upgrade a library’s catalog of marc records, generally done with oclc by matching and supplying up-to-date cataloging records from oclc to replace low-level records in the local catalog. [13] note that the number of titles in the collection analysis report will include duplicate entries across owning institutions as these are submitted as individual marc records. in other words, if institution a and institution b both own that title, that title will show up twice in the report. about the authors dana jemison (dana.jemison@ucop.edu) is the principal metadata analyst for the discovery and delivery group at california digital library. she works primarily with the western regional storage trust (west), the cdl, crl, and the hathitrust shared print collaboration (cch collaboration.) she holds her m.l.i.s. from university of texas, austin and a b.s. in botany from california state university, long beach. lucy liu (lucy.liu@ucop.edu) is the application programmer for the discovery and delivery group at california digital library. she works primarily on the agua and papr project to support the collection analysis for western regional storage trust (west), the comparison tool for cdl, crl, the hathitrust shared print collaboration (cch collaboration). jing jiang (jing.jiang@ucop.edu) is a programmer analyst for the discovery and delivery group at california digital library. she works primarily on library metadata management projects including the hathitrust metadata management system (zephir) and the western regional storage trust (west) shared print programs. anna striker (anna.striker@ucop.edu) is the shared print operations and collections analyst at the california digital library. she works primarily with the western regional storage trust (west) and ucl shared print programs. she also works with the rosemont shared print alliance and contributes to the cdl, crl, and hathitrust shared print collaboration (cch collaboration). she holds her m.l.i.s. from san josé state university and a b.a. in english from the university of virginia. judy dobry (judy.dobry@ucop.edu) is technical team manager for the discovery and delivery group at california digital library. she works on a variety of shared print projects including the western regional storage trust (west), the cdl, crl, and hathitrust shared print collaboration (cch collaboration), and the partnership for shared book collections. she holds her b.a. from university of north carolina at chapel hill. alison wohlers (alison.wohlers@ucop.edu) is the shared print program manager at california digital library. as such, she supports and coordinates collaborative print management efforts in the uc libraries system, the western regional storage trust, and connects that work to national and north american collaborations. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – creating an institutional repository for state government digital publications mission editorial committee process and structure code4lib issue 9, 2010-03-22 creating an institutional repository for state government digital publications in 2008, the library of virginia (lva) selected the digital asset management system digitool to host a centralized collection of digital state government publications. the virginia state digital repository targets three primary user groups: state agencies, depository libraries and the general public. digitool’s ability to create depositor profiles for individual agencies to submit their publications, its integration with the aleph ils, and product support by exlibris were primary factors in its selection. as a smaller institution, however, lva lacked the internal resources to take full advantage of digitool’s full set of features. the process of cataloging a heterogenous collection of state documents also proved to be a challenge within digitool. this article takes a retrospective look at what worked, what did not, and what could have been done to improve the experience. by meikiu lo and leah m. thomas introduction the library of virginia (lva) is a state library and is responsible for collecting and distributing state publications to designated virginia depository libraries.[1] due to the increasing number of born digital publications, lva began developing the digital virginia state publications collection in 2006 to accommodate the digital publications of over 140 state agencies, including lva. two full-time professional library staff are devoted to the development and maintenance of this collection. one staff member trains agency contributors, or endusers,[2] while the other is responsible for the metadata, cataloging, and management of the digital publications. digitool, an exlibris product, was selected to manage the collection. the library of virginia’s depository program although lva was established as virginia’s state library in 1823 and has been collecting government materials since that time, it was not designated as a depository library until 1981. the state publications depository program’s purpose is to ensure that the public has access to government information as mandated in §42.1-92 of the code of virginia.[3] because state publications were increasingly being born digital, lva needed a way to provide the public with access to these publications and to maintain the collection in conformance with its responsibility as a government publications depository. lva began working with selected agencies in december, 2007, to test the digital program. the first general request for volunteer agencies was february-march 2008. in july, 2008, the program was switched to the production server. the second and third requests for volunteer agencies occurred in august and october, 2008. the general training sessions were done largely in-house due to the proximity of many state agencies to lva and the availability of an onsite computer lab with staff to assist with training. additionally, due to budget constraints that restricted travel during the latter half of 2008, lva implemented alternate methods of offering training for agencies, colleges, and universities not located in the local area, which included offering one-on-one telephone training sessions. these sessions usually took about 10-20 minutes and have been as successful as the in-house training sessions, as determined by the feedback and depository submissions from these agencies. a step-by-step workflow with screenshots was sent to all trainees. approximately ninety agencies received training. why digitool? prior to 2006, when lva was considering a digital asset management system, two primary needs were taken into consideration: 1) a system’s flexibility to be used across collections and 2) system support, maintenance, and development. the foremost reason that digitool was selected was because of the reputed administrative and technical support that exlibris was providing to the user community[4] and the company’s efforts toward digital preservation. exlibris assured lva that it would provide lva staff assistance with any maintenance, development, and support issues for digitool’s implementation, operation and functionality, which it has done. lva also considered contentdm; however, at the time it was determined that this management system could not handle the governor’s records according to the needs of lva. lva wanted one system that would provide the functionality for maintaining every type of digital content. lva is a small institution without the staff infrastructure, including the assistance of graduate students, for onsite development. consequently, open source products that lva considered were not an option because they required onsite development. open source systems did not have the same level of technical and development support that proprietary products had. [5] open source products today, however, may no longer require the same level of onsite support. digitool was selected to enable the virginia state publications depository, as well as collection management, access, and preservation of varied digital collections. for the purposes of the depository, digitool allowed lva to create depositor profiles with email notification for off-site users, and agency login accounts so that endusers at the agencies could deposit agency publications. an enduser repository system was preferred because agency employees would know better than lva would, when their electronic publications were available. the agencies would be able to deposit their digital publications themselves without lva having to search for them, with the possibility of some publications being missed. other reasons for selecting digitool were customization, functionality, and synchronization with aleph, lva’s ils, which is also an exlibris product. many of the digital publications managed by digitool are also cataloged in both oclc connexion and aleph. instead of cataloging these publications twice, the metadata could be transferred to aleph through the synchronization of digitool and aleph. automated synchronization could be done selectively by transferring specific data from aleph to digitool and vice versa, which was preferable to not having control over what data gets transferred to and from either program. selective synchronization would be beneficial when modifications were made to records in either system. both aleph and digitool use z39.50 protocol,[6] enabling the retrieval of records from different databases, such as the library of congress, or reciprocation between aleph and digitool. the z39.50 feature in digitool is essential for locating and attaching pre-existing machine-readable cataloging (marc) records from aleph. although lva chose digitool for the aforementioned benefits, some drawbacks have been discovered. while digitool can locate an aleph record, once it is located, the record cannot be opened. it can only be linked, and is identifiable by displayed information, such as the title, call number, and year. when creating “objects,” as they are called in digitool, and linking to metadata and other objects, it has to be done by the persistent digitool internal id (pid) numbers, rather than by title, author, or another access point. in order to locate a record in meditor, the metadata editor utility, the pid has to be known in order to retrieve a digital object or its metadata that have been imported or created. meditor is not a user-friendly interface for catalogers. it is not intuitive and changes to records are not made easily. an entire record may have to be deleted in order to make changes. fields have to be deleted rather than changed. only limited reports can be generated. for example, instead of being able to generate a report to indicate the number of electronic publications ingested every month, a monthly collection had to be established in digitool in order to record the number of these publications, as elaborated below. another drawback is only one mapping schema can exist for all collections in digitool, and once a schema is set, it is not easily changed, so there is little flexibility, if any, for switching dublin core (dc) elements. also, no authority control exists in digitool’s infrastructure. standards for metadata and access points have to be determined locally and documented for purposes of consistency and having an institutional record of what was done and the rationale for it. metadata issues and best practices lva houses both library and archival collections and is required to archive and manage the governor’s communications as well, from the more traditionally scanned objects, such as photographs, maps, and manuscripts, to digitally born publications, including email and video. both acquisitions and access management (aam)[7] and archives are further broken down into collections: archives includes private papers and local and state records; aam includes special collections, the map collection, and federal and state documents. initially, three catalogers worked on three projects simultaneously: state publications, special collections, and the map collection. each was described differently, using inconsistent dc elements and labels. state publications that were born digital were described in dc and were referred to as “native” digitool objects. however, digital resources that had already been cataloged in oclc and lva’s ils were for the original copy, e.g. paper, not the digital reproduction. this did make it easier for catalogers because the marc record could be imported into digitool and linked to the image, but the distinction between description of the original and description of the digital object needed to be made. also, because of workload and time constraints, differences in description and punctuation in the marc records were not modified, so the catalogers had to allow for some flexibility in standards, and thus in presentation, of metadata. as a result of these metadata issues, especially the inconsistency in standards, an ad hoc metadata standards working group was formed from those who would be creating the metadata. because there were different collections being entered without any centralization, they were being cataloged using different practices for each collection along with different mappings. the goal of the ad hoc metadata standards working group was to develop a mapping schema and best practices for the collections which were already in digitool and for future collections. the group met to determine the mapping for the marc fields and labeling for the individual collections and for the generic overall digitool display, based upon the collections that had already been entered. because it was not known what to anticipate for each collection, it was determined the best way to proceed was to be as specific as possible in the mapping and the application of dc elements, while allowing for flexibility. once the application of dc elements and mapping was accomplished, each cataloger devised a data dictionary for each collection. additionally, the group had to resolve discrepancies in controlled vocabulary and abbreviations as they arose. it helped to review the best practices of other institutions. these, including crosswalks and data dictionaries,[8] were developed based upon those that already existed within the metadata community, such as those of colorado state university,[9] boston college,[10] cornell university,[11] and virginia commonwealth university,[12] in addition to the standards established by the open archives initiative (oai),[13] and following the dublin core metadata initiative (dcmi).[14] digitool’s feature for electronic submission, which was developed primarily for theses and dissertations, was utilized for the digital state documents depository. this feature allowed us to create a form which requires the submitting agency to complete the six required dc elements, including language, on the deposit form interface. the optional dc elements, in addition to the required ones, are title, creator(s), summary, keyword(s), and subject(s) as shown in figure 1. figure 1. lva deposit form click here for larger image two sets of labels were necessary: one for digitool overall that applied globally to all collections and another, collection-specific set. a data dictionary was developed to define the usage of the labels so that they would be used consistently throughout all of the collections. an unanticipated mapping issue that arose was duplicate fields, a result of mapping with marc where these mapped fields were also applied to native digitool records originally described in dc only. (see figure 2 for an example). because digitool can handle only one mapping schema across collections, it imposes this mapping onto all collections, as shown in figure 2a, below. this issue of duplicate fields has not yet been resolved. figure 2. issues in mappings: duplicate fields click here for larger image figure 2a. issues in mappings: duplicate fields click here for larger image as mentioned previously, digitool required one mapping schema for all lva projects. but having only one schema caused inconsistencies in the presentation of metadata, (see figure 3), due to the marc mapping overriding the dc description. for sorting purposes, the agency name is used instead of division and unit as publisher. figure 3. issues in mappings: one mapping schema for all collections click here for larger image records have to be reviewed for various unexpected problems, and documentation has to be updated as fields become obsolete. figure 4 shows mappings, including both the dc elements and labels, for the state documents collection. the example identified in figure 4 is the 440 field, which is now obsolete. the 490 field will need to be added, as well as a footnote documenting the reason for the change so that a historical record of the changes is maintained. in addition to having changes in bibliographic standards, there are issues related to the system itself over which the cataloger has no control, as noted above in figures 2 and 3. figure 4. mappings click here for larger image like the university of minnesota, lva discovered that well-prepared mapping and a well-established taxonomy in the beginning may have decreased the amount of work that had to be done later. in this case, where only one mapping schema could be used for all collections, and the same practices across collections, necessary for access purposes, needed to be followed for each collection as well. developing standards would have gone more smoothly had it been centralized with someone designated in charge. semantic differences arose on how to use various dc elements. it was even suggested that we did not need best practices for description of resources across collections because dc allows for more flexibility than marc with aacr2. the aforementioned documentation from the digital library community helped resolve many of these issues. yet, there is still a tendency among different departments to apply dc elements and labels however they want due to the lack of official centralization of metadata standards. we had thought developing the taxonomy retrospectively might actually save work because the necessary categories would not be determined until lva started receiving the digital publications. but because the taxonomy for the state documents collection was not developed until later, thousands of records had to be re-linked. although these records did not require re-cataloging or additional fixes, the re-linking was time-consuming. re-linking such a considerable number of records was not anticipated, and it was not realized that digitool would be as difficult to manipulate as it was. although no authority control exists in digitool, the name authority used in the ils could not be used because it would have caused problems with sorting within the state documents collection. if the name authority that is used in the ils were used for the state documents collection, for example, every one of them would begin with “virginia.” an agency directory, as a drop-down list, was used on the depository form instead. the public is generally more familiar with the agency name and can use it for searching. however, the established authority is used in other collections because it does not cause the same types of sorting problems. collection management issues when endusers submit digital publications, the objects are ingested into digitool, after the approval of the state publications librarian. digitool extracts the administrative metadata, and the state publications cataloger modifies metadata provided by the depositor, adds descriptive metadata, and organizes the digital content. the sample below shows administrative metadata from the ingest process: the pid (35387) for the object; the issue (no. 51, november 2009); the deposit number (441453); the name of the ingested item (“virginia water central, v. 51”); the partition, or collection, (“statedoc”); as well as the date, time, and name of the person who ingested the deposited publication. figure 5 shows the meditor view of this metadata. <xb:digital_entity xmlns:xb="http://com/exlibris/digitool/repository/api/xmlbeans"> <pid>35387</pid> <control> <label>no. 51, november 2009</label> <note xsi:nil="true" xmlns:xsi="http://www.w3.org/2001/xmlschema-instance" /> <ingest_id>dep441453_ing3537</ingest_id> <ingest_name>virginia water central, v. 51</ingest_name> <entity_type xsi:nil="true" xmlns:xsi="http://www.w3.org/2001/xmlschema-instance" /> <entity_group xsi:nil="true" xmlns:xsi="http://www.w3.org/2001/xmlschema-instance" /> <usage_type>view</usage_type> <preservation_level>any</preservation_level> <partition_a>statedoc</partition_a> <partition_b xsi:nil="true" xmlns:xsi="http://www.w3.org/2001/xmlschema-instance" /> <partition_c xsi:nil="true" xmlns:xsi="http://www.w3.org/2001/xmlschema-instance" /> <status xsi:nil="true" xmlns:xsi="http://www.w3.org/2001/xmlschema-instance" /> <creation_date>2010-01-06 10:51:06</creation_date> <creator>creator:nverilla</creator> figure 5. meditor view: administrative metadata click here for larger image the cataloger reviews the data and makes modifications, such as formalizing the title to make it consistent with the one in aleph and adding the issue date and subjects (descriptive metadata) to the ingested object. additionally, because the metadata submitted by the enduser is not always consistent or correct, the cataloger has to review and modify that metadata as well. the sample data below indicates changes applied (that the metadata is descriptive and in dc and when and by whom it was modified). “water-supply – virginia – periodicals,” for example, is a library of congress subject headings (lcsh). the meditor view of the added descriptive metadata is shown in figure 5a. <modification_date>2010-01-06 14:29:09</modification_date> <modified_by>super:mlo</modified_by> <admin_unit>lva01</admin_unit> </control> <mds> <md shared="false"> <mid>58476</mid> <description /> <name>descriptive</name> <type>dc</type> <value> <?xml version="1.0" encoding="utf-8"?> <record xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:dcterms="http://purl.org/dc/terms/" xmlns:xsi="http://www.w3.org/2001/xmlschema-instance"> <dc:title>virginia water central (no. 51, november 2009)</dc:title> <dc:stateemployee>nathan verilla</dc:stateemployee> <dc:publisher>virginia water resources research center, virginia</dc:publisher> <dc:date>2009</dc:date> <dc:format>pdf</dc:format> <dc:language>english</dc:language> <dc:subject>water-supply -virginia -periodicals</dc:subject> <dc:subject>water resources development -virginia -periodicals</dc:subject> <dc:subject>water -pollution -law and legislation -virginia -periodicals</dc:subject></record> ]]> </value> initially, the dc element “contributor” was used to identify the state employee who deposited the digital content. however, because every digital collection in digitool had to have the same mapping, the cataloger would have to delete that field for each record, since the depositor was not to be displayed to the public. the “contributor” element, however, is publicly displayed. therefore, the field “state employee,” which was unique to the state documents collection, was created, as shown in figure 5a. this field does not display to the public and, thus, does not have to be deleted. figure 5a. meditor view: descriptive metadata click here for larger image the parent-child relationship, or complex object, is used to manage continuing assets, such as periodicals. figures 6 and 6a illustrate an example of a complex object in resource discovery (rd), the user interface that displays searchable digital content and metadata, while figure 6b shows the meditor view. figure 6. parent-child relationship, or complex object click here for larger image figure 6a. parent-child relationship, or complex object click here for larger image figure 6b. meditor view: parent-child relationship, or complex object click here for larger image the administrative metadata for a complex object shows the object is “part of” another object and provides the pid for the child. the child is given a separate label, “virginia water central,” and shows both the creation and modification dates and times. the entity type is identified as “complex,” meaning “complex object.” lva uses either the complex object feature in digitool or a staff member converts a multi-file pdf into a single file using acrobat adobe professional. <relation> <type>part_of</type> <pid>8703</pid> <label>virginia water central</label> <note xsi:nil="true" xmlns:xsi="http://www.w3.org/2001/xmlschema-instance" /> <usage_type>view</usage_type> <creation_date>2008-08-27 13:55:59</creation_date> <modification_date>2010-01-06 14:31:38</modification_date> <file_extension xsi:nil="true" xmlns:xsi="http://www.w3.org/2001/xmlschema-instance" /> <mime_type xsi:nil="true" xmlns:xsi="http://www.w3.org/2001/xmlschema-instance" /> <external_type>0</external_type> <directory_path xsi:nil="true" xmlns:xsi="http://www.w3.org/2001/xmlschema-instance" /> <entity_type>complex</entity_type> <file_id xsi:nil="true" xmlns:xsi="http://www.w3.org/2001/xmlschema-instance" /> </relation> to set up a periodical, a parent object is created and four or five dc elements, e.g. title, publisher, date, language and/or subject(s), are added. the child objects are then linked to the parent object, and the parent object is linked to the periodicals collection. finally, the url is added to the marc records in both oclc and aleph, which could be done using a separate program. currently it is being added manually. after the object has been modified with additional descriptive metadata, it is incorporated into the appropriate subcollection or collections. the main collection of depository documents, state government publications, is subdivided into three categories: 1) browse by topics, 2) depository submissions by date, and 3) periodicals, as shown below in figures 7 and 7a. the topics were devised taxonomically, by the state and federal publications cataloger, because this was the only collection requiring this level of sub-categorization at the time. figure 8 displays the general collections as seen in rd. figure 7. collection in resource discovery (rd) click here for larger image there are two collection levels with ten main categories and subcategories. the two collection types are nodes and itemized. each object is assigned one subcategory. the subcategories can be seen in the state government publications view in figure 7a. figure 7a. subdivisions of main collection click here for larger image the statistics that are kept track the publications and the number received. the total number of publications displayed in the rd of digitool is misleading, however, because it combines all subcollections within the state government publications collection. these subcollections contain items that overlap other subcollections in the state government publications collection in digitool, thus inadvertently inflating the number of records in the collection. in order to retrieve the accurate number of digital publications that have been deposited in digitool, a collection of depository submissions, entitled “depository submissions by date,” was created (see figure 8) showing the number of items deposited by year on the first level then further subdivided by month so that the number of items can be identified and calculated monthly. to retrieve the number of deposited digital state documents in aleph, the local note marc field 590 is used to indicate the submission date of the digital object, which can be used to create a report in aleph to retrieve the number of items deposited by date. figure 8. depository submissions by date click here for larger image another aspect of collection management outside of digitool, yet based on the collection of depository submissions by date in digitool, is the online shipping lists, created by the state documents cataloger. the lists are derived from oclc connexion export reports that are entered into microsoft word, which is then translated to a pdf. the shipping lists identify items cataloged monthly. they are listed numerically with different lists for monographs and serials as shown in figure 9. in this way lva can easily notify depository libraries of newly available titles. lva creates monthly online shipping lists to display on its website so that other state agencies can get access this way as well. figure 9. electronic shipping lists click here for larger image conclusion digitool has provided the functions for a successful digital repository program for the virginia state publications depository through its facility of digital publication deposit, email notification, and agency login accounts. yet, the success of the program has been limited due to digitool’s requiring management by a systems person or department and inability to be manipulated by catalogers. fields cannot be easily changed across the system, much less for individual collections. this lack of support infrastructure has prevented lva from taking advantage of digitool’s full functionality. while support from exlibris has been excellent, response time to questions from exlibris can take anywhere from thirty-six and forty-eight hours or longer. communicating in real-time has not been possible due to their being headquartered in israel. there also has not been the level of synchronization between aleph and digitool as we had hoped and anticipated. we have also realized that digitool cannot handle newspapers very well and offers a limited text searching capability. for this reason, lva will have to use another system for its extensive digital virginia newspaper collection. while digitool was a good choice at the time for what was needed, unless it can be manipulated and better utilized by catalogers, the system will continue to be a laborious and time-consuming product with much potential but limited functionality for smaller institutions. the most challenging aspects of developing the program were the mapping, and metadata standards, both developing these locally as well as enabling them in digitool. the mapping, metadata standards, and taxonomy would have been more successful had they been developed at the beginning. metadata in libraries needs to be understood as cataloging requiring standards, and digital asset management systems like digitool need to be intuitive and practical for purposes of functionality, display, and most important, efficiency. notes [1] see library of virginia state publications depository program, accessed 1/7/10, http://www.lva.virginia.gov/agencies/statedocs/. [2] for the purposes of this article, enduser will refer to the agency contributor, the person who manages the agency’s publications and has the responsibility to submit electronic publications to lva. [3] “faq about the state publications depository program,” library of virginia website, accessed 1/12/ 2010, http://www.lva.virginia.gov/agencies/statedocs/faqstate.asp. code of virginia, legislative information system website, accessed 2/22/10, http://leg1.state.va.us/000/src.htm. [4] see valerie stevenson and sue hodges’s article (2008) “setting up a university digital repository: experience with digitool” in oclc systems & services: international digital library perspectives, 24:1, p. 48-50 for liverpool john moores university (ljmu) case study on digitool. ljmu also developed a “process for self-deposit” (p. 49) and determined that the “main advantages in choosing digitool . . . have been the guaranteed level and availability of support available from exlibris and the purchase of a single product . . . to create and manage the full range of digital collections at ljmu” (p. 50). see also “digitool implementation and documentation” on boston college’s website accessed via internet archive, accessed 2/22/2010, http://web.archive.org/web/20070809040644/http://www.bc.edu/bc_org/avp/ulib/staff/digilib/digitool/digitool.html and este paskausky’s (2004) “digitool at boston college” in north american aleph user group fifth annual meeting, june 13-16, 2004, massachusetts institute of technology, cambridge, ma, accessed 1/7/10, http://documents.el-una.org/77/1/naaug04-present_digitool-bc.pdf. [5] see valerie stevenson and sue hodges’s article (2008) “setting up a university digital repository: experience with digitool” in oclc systems & services: international digital library perspectives, 24:1, p. 48-50. [6] “’z39.50’ refers to the international standard, iso 23950: ‘information retrieval (z39.50): application service definition and protocol specification,’ and to ansi/niso z39.50. the library of congress is the maintenance agency and registration authority for both standards, which are technically identical (though with minor editorial differences),” z39.50 international standard maintenance agency, the library of congress network development & marc standards office website, accessed 1/31/2010, http://www.loc.gov/z3950/agency/. [7] acquisitions and access management (aam) is a department at lva that resulted from the merging of technical services, which included acquisitions, and government documents. [8] network development and marc standards office, library of congress, “dublin core to marc crosswalk,” library of congress website, accessed 1/31/2010; see carol jean godby, jeffrey a. young, and eric childress’s article (2004) “a repository of metadata crosswalks” d-lib magazine, 10:12, accessed 1/7/10, http://www.dlib.org/dlib/december04/godby/12godby.html. [9] hunter, nancy chaffin, et al., “phase one report: core metadata elements for csu electronic theses and dissertations, faculty papers, and university historic photographs collection (glass plate negatives),” accessed via csu’s digital repository, 1/31/2010, http://hdl.handle.net/10217/3146; cdp metadata working group, “dublin core best practices, version 2.1.1, september 2006, accessed 1/31/2010, http://www.bcr.org/dps/cdp/best/dublin-core-bp.pdf; rettig, patricia j., et al., “csu core data dictionary: version 1.1 (july 2008),” accessed via csu’s digital repository 1/31/2010, http://hdl.handle.net/10217/3147; rettig, patricia j., et al., “csu core data dictionary: version 1.0 (october 2007),” accessed via csu’s digital repository 1/31/2010, http://hdl.handle.net/10217/3150; rettig, patricia j., et al., “developing a metadata best practices model: the experience of the colorado state university libraries,” journal of library metadata, vol. 8, no. 4 (2008): 315-339, accessed via csu’s digital repository 1/31/2010, http://hdl.handle.net/10217/9025. [10] paskausky, este, “digitool at boston college,” 2004, accessed 2/1/2010, http://documents.el-una.org/77/1/naaug04-present_digitool-bc.pdf. [11] cornell university library metadata standards & tools, accessed 2/1/2010, http://lts.library.cornell.edu/lts/dm/ms/mst; see also metadata services : resources from cornell university, accessed 2/1/2010, http://metadata.library.cornell.edu/resources.html. [12] mary anne dyer, metadata librarian at vcu, provided authors with examples of vcu’s crosswalk and data dictionary for one of vcu’s online repository collections via email, 3/9/2009. [13] “implementation guidelines for the open archives initiative protocol for metadata harvesting,” open archives initiative website, accessed 1/31/10, http://www.openarchives.org/oai/2.0/guidelines.htm. [14] “library application profile,” dublin core metadata initiative website, accessed 1/31/10, http://dublincore.org/documents/library-application-profile/index.shtml. about the authors mei kiu lo (meikiu.lo@lva.virginia.gov) is state and federal publications cataloger and leah m. thomas (leah.thomas@lva.virginia.gov) is cataloging coordinator at the library of virginia. mei kiu developed the taxonomy for the state publications collection. in tandem, they started the ad hoc metadata standards committee and devised the crosswalk for digitool. subscribe to comments: for this article | for all articles 3 responses to "creating an institutional repository for state government digital publications" please leave a response below: beanworks » comparing digital library systems, 2011-04-11 […] expertise in varying levels in order to realize full functionality of their features (see, e.g., creating an institutional repository for state government digital publications).  for another, as the number of libraries implementing digital libraries with resource discovery […] manoj, 2012-05-16 i wish to undertake project of digital library of state governemnt publications for government of maharashtra. how to go abut it manoj, 2012-05-16 it is aggod article leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – recognizing cultural diversity in library interface development mission editorial committee process and structure code4lib issue 28, 2015-04-15 recognizing cultural diversity in library interface development the rapid increase in complex library digital infrastructures has enabled a more full-featured set of resources to become accessible by autonomous users, whether onsite or remote. however, this trend also necessitates careful consideration of the usability of new interfaces for populations with increasing cultural, geographic, and socioeconomic diversity. researcher aron marcus has become an authority on how cultural principles affect interface perceptions and inform their development. this article will explore marcus’ work to contextualize diversity issues within usability before exploring the redevelopment strategy for the new york university libraries’ web presence, which serves a broad and global set of users. by nik dragovic with the increasing predominance of remote and digital access to academic library resources, the design of user interfaces and online access is a correspondingly more visible concern. highly networked online environments draw a more varied audience to these web resources. the division of libraries at new york university (nyu) is responsible for serving a diverse group of students, faculty, staff, and other stakeholders at multiple global sites. accordingly, the usability of online resources for the wide range of users and uses in this complex organization has become a point in focus. this article will discuss the ways in which current initiatives seek to enhance wayfinding for information seekers across academic discipline, university role, and cultural background. conceptual frameworks in universal access will frame the discussion of user needs in the context of the user community and institutional strategies and methodologies. a growing corpus of research exploring the influence of cultural background on user experience (ux) has emerged in the past two decades, much of which can inform the efforts currently underway at nyu. one of the major frameworks applied to this topic is that of aron marcus. marcus’ thinking on cross-cultural ux design is predicated on five dimensions of culture proposed by cultural anthropologist geert hofstede. power-distance – the extent to which less powerful members expect and accept unequal power distribution within a culture collectivism vs. individualism – strong group cohesion and mutual dependence versus loose social ties femininity vs. masculinity – traditional gender roles linked to domesticity versus competition and toughness uncertainty avoidance – the degree to which a culture tolerates ambiguity and resulting anxiety, expressed in varying rituals and customs longvs. short-term orientation – confucian-inspired dedication to the search for virtuous behavior versus the western belief in, and search for, truth (marcus and gould, 2012) marcus has examined the ways in which these considerations affect digital interface expectations, preferences, and interactions for users in varied cultural contexts, where culture is defined as “the set of shared attitudes, values, goals, and practices…” (marcus and baumgartner, 2004). his framework has in part been derived from contrasting the interfaces of multinational company websites and university websites. resulting implications tied to the five dimensions extend to content, visual design, navigation, and all other elements of digital interfaces, and manifest in ways that range from strikingly clear to incredibly subtle (marcus and gould, 2012). working under marcus, valentina-johanna baumgartner derived 29 detailed dimensions of culture from the work of nine theorists (including hofstede) and surveyed 57 multinational user interface designers and analysts to rank the importance of the various dimensions within the professional community. many respondents found the context of the user interface to be a major influence on which elements were most essential (marcus and baumgartner, 2004). respondents also asserted that the degree of compliance with the dimensions of culture is a matter of resources, since the cost of development and testing of localization features for culturally responsive interfaces is high even when only a few elements are undertaken (marcus and baumgartner, 2004). in the academic library environment, the question raised is how the relatively new implementation of dedicated user experience staff can accommodate these challenges and provide equitable access to resources for each of its users. for digital interfaces in particular, user diversity can be difficult to identify or even perceive. nyu is approaching this challenge with a variety of user research methods enacted through project work by dedicated ux staff as well as newly-established interdepartmental committees. contextualizing the information needs of the global network university accommodating the diverse backgrounds and information tasks of users is a complex endeavor which the division of libraries at new york university (nyu) is currently addressing. as one of the largest private universities in the united states, the school has a remarkably diverse population of students, faculty, and staff, in addition to a growing international presence. a 2014 report by the institute of international education found that nyu enrolled the highest number of international students of any postsecondary institution in the nation (iie, 2014). the school claims 90 countries of origin for its new york student population, and 77 for its group of international faculty and research scholars. in addition to 11 global academic centers that are analogous to traditional study abroad sites, the university has also established two global campuses that exist as autonomous degree-granting sites and recruit students worldwide for a us education. the abu dhabi campus, established in 2008, currently hosts students from over 100 distinct countries and claims 98 languages are spoken onsite. nyu shanghai, established in 2012, hosts students from 49 countries. furthermore, the range of services extends to the general public as well. considering the robust set of archives and special collections in new york and abroad, as well as mandates to provide access to documents deposited from the us federal government as well as the united nations, the school must also consider independent researchers in its efforts to improve interfaces for access and discovery. with one website currently in place to provide online access to the catalog and electronic resources of the shared library system, there exists a prominent need to provide an experience sensitive to this culturally and geographically diverse user base. index scores for hofstede’s five dimensions of culture at nyu’s global campuses (marcus and gould, 2012) power distance individualism/collectivism masculinity/ femininity uncertainty avoidance long/short term time orientation china 80 20 66 40 118 united arab emirates 80 38 52 68 – united states 40 91 62 46 29 the chart above lists the index scores of hofstede’s five dimensions of culture for the countries in which each of nyu’s degree-granting campuses are situated. though these figures are not fully representative of respective student populations, since each campus draws a widely international audience, the dramatic range in these measurements illustrates a point of contention for interface design. focusing on power distance, in which a higher index suggests rigid hierarchical relationships in a society and a lower number indicates more equality and status, has a corollary to library services at nyu. students coming to the university from high power distance countries are often challenged to acclimate to an american educational environment that is more democratic and participatory than their places of origin. to support this transition, programs are offered in library services to help explain the differences in both information retrieval and academic culture germane to an institution that operates on an american model despite a global presence. this highlights a conflict for interface design as well: should the web presence defer to the understandings of the user, or seek to support them in a transition to an environment with a different value system? having identified user-centered practices as a foundational design principle, nyu’s objective will be to achieve the former. strategizing the development of our online research hub nyu libraries’ last major website redesign occurred in 2008. while the staff has continued to incorporate new back-end technologies to improve access to online reference, electronic resources, and surrogate records, the need to revise the user interface, which brings together a wide variety of discoverability engines and content platforms, is acknowledged. in accordance with the current strategic plan, which names user experience as a focus area, the libraries have originated an eponymous department for that purpose. the confluence of these factors, in addition to the growing global audience for the web platform, has presented an interesting set of challenges to strategize around. with hundreds of full-time library staff members as stakeholders in addition to the broader university community, a web presence group, consisting of members from multiple library departments, was convened. this committee is led by the head of user experience and ensures that representatives from these separate divisions share their perspectives and advocate for their staff and users. while directly addressing diversity as defined by professional roles, this structure also includes permanent representation for staff of global sites and a global services librarian. these geographically defined roles, by extension, implicate culture. this team-based approach to the development and maintenance of the library site is one of the first implicit strategies for addressing cultural diversity. with more equitable representation of different stakeholders by committee members, those more intimately familiar with the needs of those from different backgrounds have a direct voice in the strategy and implementation of new web elements. the approach is enhanced with a dedication to regular communication within the group and to outside stakeholders. the group often meets face-to-face, with synchronous virtual participation from members overseas. communication is supplemented with institutional google apps including gmail and drive, as well as the libraries’ shared wiki and workspace. the group itself is divided into subsections with specific responsibilities such as user testing, which helps direct the workload and leverage the expedience of smaller teams. this more egalitarian form of web development is augmented by liaison work by the committee in the form of periodic updates and outreach to explain progress and solicit feedback. monthly emails charting group progress are distributed to the entire library staff, and the group also visits library departments to share goals and obtain needs and feedback from constituents. some of the essential service developments currently planned or underway include updates to research guide platforms and the testing and selection of a new site-wide content management system, both subordinate to the overall reimagining of the libraries’ web presence. harnessing the power of agile scrum the first strategic goal emerging from this group was the re-envisioning of the library web presence, to be interpreted as a re-engineered information architecture, user interface, and digital strategy. the last of these elements represents a particular departure from prior conceptualizations of how to manage the website. the recognition of complex and multifaceted user needs and the advent of agile development methodologies meant that web initiatives need no longer be confined by monolithic, complete redesigns. instead, iterative processes mean that specific elements of the platform can be taken under consideration as the need arises, then swiftly developed and tested. this has enabled the popular scrum methodology to be newly applied to this academic environment, functioning as a way to expedite work within the user experience department and enact the findings of the web presence group. several new functions have already been deployed under this workflow, and assessment has shown that even minor tweaks have paid dividends in the usability of the site. the broader implications of harnessing this responsive framework and resulting ability to reformulate elements of the user interface with ease draw back to practical concerns in marcus’ work. even resource-heavy corporations struggle with the cost and complexity of directly addressing the cultural dimensions and framing of their user interface. agile development enables incremental changes to be tested and adopted quickly, meeting these needs with an indirect and less resource-intensive approach. the deployment of global and local experiences the other main thrust of the new digital strategy is that of localization. a new conceptual model is in the pipeline to implementation, in which users will be able to customize the library site to their given location for easier access to relevant information. the prior web redesign centered on the bobst library, while current initiatives must strive to incorporate the rapidly complexifying library ecosystem growing in tandem with global sites and initiatives. the libraries strategic plan 2013-2017 makes specific reference to “a user experience that is high quality, consistent, and robust regardless of the user’s location, access method, or objective,” and the first initiative subsumed under this goal targets “sensitivity to its global context.” marcus’ work goes into depth about the consequences of localization of interface for users, namely that such efforts can help accommodate a more geographically and culturally diverse audience by customizing the experience with respect to relevant interface elements. the practical adoption at nyu will involve the user’s selection of their local or global site in order to streamline their interface. providing information customized to the user and their intended use of physical sites will reduce the complexity of the current site, which must dictate policy and procedure for many user types and institutional sites within one space. this has been a challenge for users, as demonstrated by user testing. the aforementioned agile development plays a significant role in enabling these customized experiences tailored to users with different locations, motivations, and cognitive needs. the fragmenting of different user flows into global and local contexts and the iterative design process set the foundation for a more holistic development process that accommodates cultural diversity in addition to a host of other priorities in one workflow. instead of attempting to meet the needs and contexts of all users with a single interface, variable user approaches can now be divided into more cohesive and workable sections. given this structure, user testing can be tied to a more specific site function, and findings can be more seamlessly implemented under an agile methodology. with the conceptual framework in place for this initiative, the choice of infrastructure is now being made. testing is now underway for a content management system that most effectively meets the needs identified to make global and local site experiences function as envisioned. the value of a user-centered design process in turn, this reinvigorates the commitment to a user-centered process. with a user experience department now available to perform the labor-intensive research this entails, the library now employs several methods to glean insight into the development of improved interfaces and workflows for users. one of the earliest initiatives involved personas, which are commonly defined as generalized personifications of user data that emerge from user research. these are typically assigned fictitious names and images in order to provide a concrete framework to contextualize and speculate on user behavior. in the case of nyu, personas originally emerged from careful study and coding of virtual user reference transactions occurring over a chat service. (tempelman-kluit & pearce, 2014) after a qualitative interpretation of the data, investigators were able to identify information needs and motivations, which acted as a springboard for the development of four evidence-based personas, as suggested by the research outputs. with the increasing need to incorporate global workflows into these resources, two global personas were also developed, which focus on the abu dhabi and shanghai sites. the user experience department developed these with attention to the differing library policies and services that apply to users at these sites, since resource sharing among the libraries has resulted in complex request and delivery mechanisms that are generally managed through user interaction with the web interface. all six personas are now commonly employed in interface development projects. user stories provide another tool for development with diverse users in mind. based on personas, the stories address a particular information need and compel the researcher to remain cognizant of the user perspective throughout the complex process of walking through an information seeking task through a third party perspective. a number of these guiding narratives have been created to help add a practical perspective on how best to address user needs, from basic access to resources and services, to more complex engagements requiring work across platforms. direct work with real users is generally the most resource-intensive but rewarding form of user research, and this is currently underway. focus groups are being conducted with students and staff on a voluntary basis, and the scope includes diverse users and perspectives from international centers. user testing, in which users are observed during the course of performing interface interactions, has also become a staple in the user-centered environment. by embracing fast and flexible agile development principles at the core of the website, usability testing can be more effectively deployed for a broader swath of users, and its results can be more easily implemented for incremental but telling improvements in user experience. guerrilla testing, which in this context means open recruitment of study participants from central library locations, has been deployed to useful effect. by meeting the user in physical library space as well as collecting online data to drive ux research, the libraries are making a concerted effort to gain the perspectives from across the community to represent modes of access to information and the approaches taken by those of different academic disciplines and cognitive frameworks. conclusion overall, new approaches to website design and feature implementation at nyu libraries put the focus more squarely on the needs of the user. though dedicated user experience initiatives are still in their infancy at academic libraries, progress so far has already paid dividends by improving the experience for a diverse pool of constituents, and more sophisticated means of assessment will help demonstrate the value of such efforts. though development of a renewed web presence is still in the early stages at the institution, the charted path shows promise for the kind of all-embracing development advocated by leading researchers in the cross-cultural user experience realm. as a distinctly american university operating in many different cultural contexts, the ultimate iterations of the nyu libraries web presence are as yet to be determined.   the members of the web presence group at nyu are katherine boss, zachary coble, michael haag, laura henze, gerald heverly, sarah jones, jessica mcgivney, dan perkins, raymond pun, charlotte priddle, andrew rarig, beth russell, nadaleen tempelman-kluit, and kara whatley. references institute of international education (iie). 2014. open doors report on international educational exchange. available from http://www.iie.org/research-and-publications/open-doors new york university division of libraries. strategic plan 2013-2017: mapping the library for the global network university. available from https://library.nyu.edu/about/strategic_plan.pdf marcus a, baumgartner vj. 2004. a practical set of cultural dimensions for global user interface development. in: masoodian m, jones s, rogers b, editors. proceedings of the computer human interaction: 6th asia pacific conference, apchi 2004; 2004 june 29-july 2; rotorua. berlin (de): springer-verlag berlin heidelberg. p. 252-261. marcus a, gould ew. 2012. globalization, localization, and cross-cultural user-interface design. in: jacko ja, editor. the human-computer interaction handbook. 3rd ed. boca raton (fl): crc press. p. 341-366. tempelman-kluit, n, pearce a. 2014. invoking the user from data to design. college & research libraries 75(3):616-640. about the author nik dragovic is a collections assistant at new york university and reference librarian at st. francis college. he was recently granted a master of library and information science degree from pratt institute, and his continuing research interests include usability, digital projects, and diversity in the library profession. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – creating a low-cost, diy multimedia studio in the library mission editorial committee process and structure code4lib issue 44, 2019-05-06 creating a low-cost, diy multimedia studio in the library this case study will explain steps in creating a multimedia studio inside a health sciences library with existing software and a minimal budget. from ideation to creation to assessment, the process will be outlined in development phases and include examples of documentation, user feedback, lessons learned, and future considerations. we’ll explore multimedia software like one button studio, gamecapture, kaltura, adobe creative cloud, garage band, and others and compare their effectiveness when working on audio and visual projects in the library. by bryan elias hull and brandon patterson introduction as audio and video projects increase in demand to meet growing multimedia-centered teaching and learning, libraries adapt to create spaces where these projects can be facilitated. the spencer s. eccles health sciences library (eccles library) at the university of utah has seen an increase in demand for multimedia projects because of a push for faculty to create flipped classroom videos, staff to create videos for their workplace, and students making introductory videos for medical school applications (patterson, et al, 2018) [1]. a technology team at the eccles library was tasked with creating a low-cost, diy multimedia studio space to meet the demand of our patrons. the following article describes the four phases for developing a multimedia space from scratch and include a narrative that may be helpful when considering developing a similar multimedia studio space in your library. phase 1 in late 2016, the library began investigating multimedia options for a growing number of faculty requesting audio and video equipment to record lectures for use in their classrooms. the initial scope of the project was to create a mobile audio/visual station that could be checked out as a set for students, faculty, and staff to use at their own workstations or other remote locations. the idea was to allow patrons to record higher quality video for lectures and other video projects with different microphones, headphones, an audio interface, ipad/apple pencil, and an imac computer. originally, the project was called the multimedia design & innovation lab or “mdil” for short (figure 1). due to space limitations within the eccles library, the initial request was to make as mobile friendly of an equipment setup as possible. as mdil started forming and feedback from patrons signified a preference toward a dedicated studio, it became evident that a space would need to be acquired that could serve as the central location to house the equipment. to obtain additional equipment and software for a dedicated studio space, the technology team sought funding from the campus teaching and learning portfolio (tlp); a university information technology committee that distributes funds collected through a student computing fee (table 1; patterson, et al, 2018). table 1.items, quantity and prices of original equipment and software purchase. items quantity price imac 27 inch retina 5k display fusion 8gb 1 $1,899.00 blue nessie – adaptive usb microphone 1 $80.00 sony mdrv6 studio monitor headphones with ccaw voice coil 1 $110.00 focusrite scarlett 2i2 audio interface 2 $149.00 ea. sennheiser ew 112-p g3 camera-mount wireless microphone system with me2 lavalier mic – a (516-558 mhz) 1 $630.00 dracast led500 pro bi-color led light 1 $500.00 apple ipad pro with pencil 1 $880.00 cables & adapters multiple $200.00 total $4,597.00 the library’s technology team also provided equipment already in the library including: a sennheiser shotgun microphone, gopro cameras, and adobe creative cloud software that exists through software agreements with the university. figure 1. initial equipment for the multimedia design and innovation lab items for the mdil were chosen for various reasons based on price, quality, and how user friendly they were (table 2). the selection of equipment and software was specifically chosen with a few different scenarios in mind: higher quality lecture captures podcast recording mobile video recording student video projects educational support staff projects table 2. mdil items and why they were chosen. items rationale imac 27 inch retina 5k display fusion 8gb the imac was chosen for a few specific reasons: having the computer and monitor all in one sleek package. no need for a separate tower and monitor but still having enough computing power for video, and graphic projects. the free software options bundled with osx including imovie, garageband, keynote, and other productivity software. the large, 5k display for better viewing & editing of video & graphics. blue nessie – adaptive usb microphone the blue nessie connects directly to any computer via usb providing a simple option for recording audio, allowing better sounding recordings than typical webcam or internal computer microphones. sony mdrv6 studio monitor headphones with ccaw voice coil having a good pair of headphones is important for editing, previewing audio while recording, and being able to critically listen to recordings. the sony mdrv6 are highly rated on most web shops and reasonably priced. focusrite scarlett 2i2 audio interface the focusrite scarlett 2i2 allows for professional microphones to be used with a computer seamlessly. the device acts as an audio interface where microphones can be plugged in with either xlr or ¼” inch cables, phantom power provided to microphones that require 48v of power, and gain adjustment on the microphones. there is also a port for headphones that allows you to preview the audio that is coming through the microphones and the ability to connect studio monitors (speakers) to the device. there are several focusrite scarlett options with more inputs and other features available. sennheiser ew 112-p g3 camera-mount wireless microphone system with me2 lavalier mic – a (516-558 mhz) having a lapel microphone is helpful for mobile recordings, lectures where the presenter is the primary focus, and other creative video projects. this sennheiser lapel microphone pack includes the microphone & transmitter as well as a smaller, more streamlined receiver unit that is easier for mobile applications than a traditional receiver unit. the downside is that all of the equipment utilizes aa batteries that are an additional cost and concern when checking out equipment. dracast led500 pro bi-color led light good lighting is sometimes the difference between a poor quality video and a more professional product. having a flexible, adjustable, and mobile led studio light allows for patrons to add an extra level of quality to their recordings. this panel includes brightness and color temperature controls to allow for dialing in the warmth or coolness of the light and the brightness. apple ipad pro with pencil the ipad contains a decent camera for video recording and can be paired with video editing apps. the device can also be used as a teleprompter through apps. the ipad and the apple pencil further expands the capabilities for demonstration, display, and markup depending on the project. cables & adapters the request included a general figure for cables & adapters for the inevitable forgotten cable, adapter, or small item that would be necessary for everything to function properly. instead of itemizing based on every cable, the team was able to request a general figure for flexibility and the ability to purchase throughout the fiscal year. after equipment was purchased in summer 2017, the library created a workstation within an existing macintosh computer lab that would accommodate the mdil. equipment was checked out and used in the space through the imac computer purchased for the mdil. many of the projects for the initial lab setup revolved around audio recording and podcasting. several student video projects used the space for curriculum and non-curriculum applications. phase two the second phase of the mdil saw two primary developments. first, the macintosh lab was moved to the public computing area of the library in order to dedicate the space entirely to the mdil. second, additional equipment was repurposed by the library in order to facilitate more elaborate audio/visual projects (table 3). computer lab to multimedia studio a 16 x 12 foot room that was originally a dedicated macintosh computer lab was repurposed to be a dedicated studio space for the mdil equipment. this allowed for a better controlled working environment for those requesting audio and video support. after approval was obtained by library leadership, the macintosh computer lab was moved to a public computing space in a different area of the library. after the migration, the team began setting up the space to be a dedicated, self-service multimedia studio (figure 2). round two of equipment acquisitions before making a request for more student computing fees, the library’s technology team examined existing library equipment and items that could be purchased through university surplus. this was done to bolster the multimedia lab without going through a lengthy budget request process and to avoid relying on external funding entirely to obtain the dedicated studio. table 3. repurposed or existing library items and why they were chosen. canon vixia hrf200 hd video camera the library had a pool of camcorders available for checkout. one camera was taken out of circulation to act as the dedicated camcorder in the new studio space. magnus vt-4000 tripod similarly, tripods were also available for checkout. one was taken out of circulation to be dedicated to the new studio. elgato hdmi capture device in order to enable recording directly to the desktop, the library repurposed an hdmi capture device originally used for lecture captures to instead act as the recorder for the mdil. this peripheral device came with its own software “game capture hd” that allowed for recording of video and live streaming directly from the camera. rode shotgun microphone another shotgun microphone from the library’s equipment pool was included in order to have two shotgun microphones available for simultaneous recording. especially useful for interviews and podcasts. sennheiser ew 100 g3 wireless lapel microphones (x2) to bring the total number of available microphones to four, two wireless lapel microphones, transmitters, and receivers were added to the space in order to allow greater flexibility for recording audio. lapel microphones are wireless and easy to hide, making them a better choice for recording video than the shotgun microphones. various cables & adapters as necessary in order to make sure all the equipment worked, the following cables were also repurposed to the space: mini hdmi to hdmi cable for the camera to hdmi capture device connection 4 xlr cables in order to connect the microphones to the focusrite scarlett 2i2 power cables and a power strip halogen studio lights the library acquired two halogen studio lights from the university of utah surplus store that were left by another department to bolster the studio space at minimal cost. initial room setup all but one desk and two chairs were cleared from the old macintosh lab to create the new space. a cabinet with locking doors houses equipment that was not in use. the door to the room was kept locked providing additional security to the camera, computer, and lights within the space. figure 2. initial room setup of the mdil. the room was set up with a large recording area with the intention of making it flexible for different scenarios (note the light blue area in figure 2). the “recording space” gave patrons the option to bring in various pieces of furniture from the library depending on the project. this proved to be especially useful as users brought in more comfortable seating for longer on-microphone discussions. the space did not include a backdrop, initially, and used the wall in the space instead which sufficed for most projects. initial equipment setup to continue providing maximum flexibility for patrons, the equipment was configured to facilitate audio, video, and other tasks to be completed simultaneously or individually. the two peripheral devices—the focusrite scarlett 2i2 and the elgato hdmi capture—served as the inputs for the audio and video feeds respectively. the camera’s hdmi output would be connected to the elgato device via an hdmi mini to hdmi cable. the shotgun (see figure 3) or the wireless lapel microphones (see figure 4) could then be connected to the focusrite scarlett 2i2 via xlr cables. this configuration became a limiting factor because the focusrite scarlett 2i2 has two inputs allowing for only two microphones to be used at any given time. as we’ll discuss later, a peripheral audio device such as a focusrite scarlett 2i4, that has at minimum 4 inputs, is recommended to accommodate multiple microphone configurations. the blue nessie usb microphone and the sony mdrv6 headphones were available in the equipment cabinet. we discovered that the option of letting users configure and plug in microphones themselves was more confusing than it was helpful. this resulted in a less hands-on audio configuration in a future phase saving library staff time spent with patrons troubleshooting audio. figure 3. equipment setup with camera and shotgun microphones. figure 4. equipment setup with camera and wireless lapel microphones. initial software setup the imac was updated to the latest version of mac osx and the library’s it department set up the computer to be network-login enabled. this allowed users with a valid university id and password to login to the computer with their credentials, instead of using a single user account on the computer. network login also gave patrons greater privacy for their projects, a necessity in the academic health sciences setting because of laws protecting patient and student information (hipaa, ferpa, etc.). however, the network login option presented a challenge because user accounts are wiped from the computer after several days. to address this, additional documentation and warnings were placed throughout the space to inform users that they must save their work to a cloud storage service, usb thumb drive, or an external hard drive. the following software was then installed on the imac (descriptions provided under “software used” heading below): adobe creative cloud suite including: premiere pro after effects photoshop illustrator audition adobe captivate (separate from creative cloud) elgato game capture hd kaltura capture space garageband imovie quicktime to ensure that every user had properly installed equipment drivers, the library’s it group configured network login accounts to have the following basics prepared to minimize any errors or troubleshooting: making the focusrite scarlett 2i2 the default audio input device on the imac. in the system preferences in the mac os, “sound” is selected with the input device switched to “scarlett 2i2” as the default. the output remained as the internal imac speakers, in anticipation that most users would want to hear audio through the computer speakers and would naturally gravitate towards the imac’s volume controls on the keyboard instead of the focusrite scarlett. shortcuts to the libguide for the mdil, the university’s cloud storage system “ubox,” and the university’s video hosting platform “mediaspace” were added to the desktop. program defaults were made to always utilize the focusrite scarlett 2i2 as the audio driver, and the elgato game capture device as the video source. the software elgato game capture hd was used for video recording because it was the proprietary software of the elgato hdmi capture device. the primary function of the software is to record any video source that had an hdmi output such as a camcorder, video game console, or other source. additionally, the software had a “live stream” option allowing users to livestream their video on various social platforms, live stream platforms, or anything that accepted the real-time messaging protocol (rtmp). figure 5. screenshot of the elgato game capture software interface. one advantage of the elgato software was that it separated the audio from the focusrite scarlett 2i2 (shown as “live commentary” in figure 5) and the audio coming from the hdmi capture device (shown as “game audio” in figure 5.) this meant that microphones plugged into the focusrite scarlett could be recorded instead of the audio coming from the internal camera microphone that is fed into the hdmi capture device. this is achieved by muting the “game audio” and selecting the “commentary” button (shown in blue in figure 5), allowing for better sound quality than the internal camera microphone. audio recording was achieved primarily through garageband or adobe audition depending on the user, their skill, and experience levels. with the audio inputs already set up in the system preferences, the studio was largely “plug and play” where users select which microphones they wished to use and plugged the xlr cables into the focusrite scarlett 2i2 and started recording. instructional how-to’s a series of instructional how-to documents and supplemental videos were developed to help users utilize the space without having a dedicated staff person present at all times. the documents & videos outlined the step-by-step procedure for the most popular uses of the space including: video recording with game capture hd software, audio & podcast recording with adobe audition or garageband, lecture & screen capture using kaltura capture space, and how to livestream with game capture hd. each document was printed and placed next to the imac computer for easy access to users. the documents and videos were also available in a dedicated libguide that was linked to on the desktop. finally, a poster with the general guidelines and rules of using the space was displayed prominently behind the imac and desk. policy & procedure the space was made available for reservation by either emailing the library’s reference email list or speaking to the library desk staff in-person. if the space was not in use or reserved, patrons had the option to “drop-in” and use the space by speaking to the library desk staff. once a reservation or request was made, patrons would be let into the space by the desk staff. the patron would be required to sign an agreement prior to using the space stipulating that they held responsibility for not stealing, losing, breaking, or otherwise misusing the space. once a patron was finished with their allotted time in the space, they would be required to “check-out” with the library desk staff to ensure that all equipment was present and undamaged. the amount of equipment in the space required that the door to the studio space remained locked at all times. the equipment cabinet in the space had the ability to lock, but the decision was made to leave it unlocked with the assumption that the door into the space was locked at all times. posters were put up in the room with the guidelines and rules, including a large poster with images and labels listing the equipment for easy reference for both staff checking the space for accuracy and library patrons not familiar with the devices or cables. figure 6. the media design & innovation lab pictured in december 2017. phase three the latest development phase focused on converting the space to a “one button studio” (obs) and further expanding the capabilities of the equipment while trying to simplify the recording process for all users. to kick off this transition, the library’s technology team rebranded the space from the “media design & innovation lab” to the “tree of hippocrates education studio,” or “the studio” for short (see figure 7). the name was inspired by an offspring of the original “tree of hippocrates” that is planted outside of the library and the memorable “the” acronym. figure 7. logo for the tree of hippocrates education studio. equipment requests similarly to the first phase of development, the library’s technology team sought equipment funding for multimedia equipment and software from the campus teaching and learning portfolio (table 4). the primary focus of this equipment request was to: obtain equipment necessary to utilize penn state’s one button studio software allow for the use of a teleprompter slim down the lighting set-up with compact led light panels provide some sound proofing of the space with sound treatment foam bolster the audio recording capabilities with a mix board add a backdrop to make video recordings more professional and visually appealing table 4. equipment and cost requested for the studio. items quantity cost auralex studiofoam wedgies (charcoal gray, 24-pack) 1 $103.00 rode wsvm shotgun windscreen 1 $9.00 rode dead cat wind muff 1 $30.00 sony fdr-ax100 4k ultra hd camcorder 1 $1,400.00 magicue mobile teleprompter kit with hard case 1 $360.00 ikan elite-remote bluetooth ipad teleprompter remote 1 $73.00 behringer xenyx 802 8-channel compact audio mixer 1 $60.00 sd cards 4 $160.00 impact crushed muslin background gray 1 $53.00 impact background system kit with 10’x12′ black, white, chroma green muslins 1 $219.00 blackmagic design h.264 pro recorder 1 $428.00 dracast led500 s-series bi-color led 3-light kit with v-mount battery plates 1 $479.00 button – griffin technology powermate assignable usb multimedia controller 1 $45.00 anker 4-port usb 3.0 hub 1 $20.00 cables & adapters several $250.00 total $3,689.00 once the approval was granted to the library’s technology team, the equipment was purchased and delivered to the library. in anticipation of the transition to the new “one button studio,” the space was closed for a two-week period during the summer semester. initial room setup the desk, equipment cabinet, and the camera remained in the same location as the previous development phase. the older halogen studio lights were removed and replaced with newer led lights. a backdrop was also added to the space as shown in figure 8. figure 8. initial room setup of the studio. to better soundproof the room, auralex studio foam wedges were purchased and installed in the room by a library colleague with an audio production background. the box of 24 wedges was enough to create two 12-wedge blocks located in line with the camera as pictured in figure 9. while a good start for dampening some of the sound reflections in the room, at least 4 times as many wedges are needed to create soundproofing blocks on all four of the walls. the purchase of bass traps that fit in the corner between walls and the ceiling or floor are also desired for additional treatment of the room. figure 9. teleprompter, camera, led light panels, and backdrop are pictured in the studio, december 2018. initial equipment setup to transform the room into a one-button studio, notable adjustments were made. first, all four of the microphones in the studio were routed through a mix board so that they could be recorded simultaneously through the focusrite scarlett 2i2 (see figure 10). the original idea was to run the microphone audio from the mix board into the camera for the one button studio. we soon abandoned this idea because this approach resulted in the multiple microphones being sent to a single stereo output, making multitrack recording in adobe audition or garageband impossible. figure 10. initial equipment setup of the tree of hippocrates education studio. another adjustment was investing in a new camcorder, the sony fdr-ax100 4k ultra, although there was one major hiccup. the sony camcorder featured only a powered 1/8” microphone input jack. this was incompatible with the “unbalanced” audio output from the mix board and caused significant audio distortion in video recordings. we realized that an ideal camcorder would be one that has a “line in” input in addition to a microphone input jack so that the unbalanced audio from our mix board could be used with the camera (see starter pack for studio equipment below). to gain greater flexibility in the space, we would recommend purchasing one of the focusrite scarlett models that have more than two xlr inputs. this allows for multitrack recording with more than two microphones with no need for a mix board to be purchased. additionally, audio from the focusrite scarlett could be run directly into the camera from the headphone jack or the stereo output on the device utilizing either a ¼” to ?” phone cable or a 3.5 mm trs to dual 1/4″ ts stereo breakout cable. equipment set-up for the one button studio software included swapping out the elgato game capture hd device with the blackmagic h.264 pro recorder and the older canon camcorder with the sony fdr-ax100 camcorder. the anker 4-port usb hub and the griffin technology powermate assignable usb multimedia controller a.k.a. “the one button,” were also installed and plugged in. finally, the magicue mobile teleprompter kit was installed on the tripod holding the camera. the decision was made to leave the teleprompter equipment up at all times to make it easier to use the teleprompter and remind users of its availability. the magicue mobile teleprompter has a glass reflector like most teleprompter systems and a platform that holds an ipad or other mobile tablet. in order to best utilize the ipad as a teleprompter, an app described in the next section, allows for mirrored text to be read on the screen. initial software setup despite the extensive equipment changes most of the software setup that we accomplished in the second development phase transferred over with only a few minor tweaks and additions. first, a new user account was created that would be explicitly used for the penn state one button studio software with the recommended system and software settings. this includes opening the obs and button controller software upon login, suppressing on screen prompts and messages, and allowing administrative permissions for disk writing, among other settings. this meant that users that wanted to use the obs software would be able to do so by default, without logging into the computer with their university id or password. we chose not to purchase the equipment necessary for allowing the studio lights to be turned on automatically by the obs software. this was decided because the lights were easily accessible by hand and assumed users would not find too much trouble switching on the lights by themselves. users would have the ability to change the brightness and light temperature or move the lights around in the space freely. finally, a teleprompter app called “promptster” was downloaded for the apple ipad pro. the app was one of the only free app options that provided a text editor, scroll speed setting, text size, and mirrored text options. to make text readable on a teleprompter screen, mirrored text is required as the teleprompter screen reflection reverses what it is being presented to it. instructional how-tos with the changes to the equipment and the addition of the one button studio software, all of the instructional handouts and videos had to be edited or redone. while not a difficult task, it proved to be time consuming. editing the documentation should be taken into consideration when making videos and written handouts. if the interest is flexibility for updates, it is easier to edit textual content over video. policies and procedures with the obs addition, we saw an increase in patrons using the space. they continue to reserve the space whenever possible as with the previous phase. in order to receive feedback from patrons, we’ve relied on a short survey that is sent after they’ve reserved and used the space. the survey consists of questions related to overall satisfaction, productivity, knowledge of how to use equipment, where to go for assistance, and their likelihood to use the space again. we also collect user demographic information and provide space for added comments or concerns. while response is relatively low (31% response rate of total users), information collected has been valuable in constituting the need for the space to library administration, as well as instigating improvements for the space. users have been satisfied with all aspects of the space. on average, users have consistently marked as being satisfied with the space and feeling productive. the indicate understanding how to use the space, where to get assistance and are likely to use the space again. we have been able to make improvements based on comments received like improved lighting, sound proofing, software updates, and guidance via written instructions in the space (survey and results in appendix 1). software used the following list breaks down the pros and cons of the software used in the studio: one button studio by penn state (with updates) pros: the biggest pro for this software is that it is one of the only free software options available for simple video recording at the press of a single button. this is a great option for those seeking to make a diy recording studio that is simple and easy to use for patrons and “non-technical” users. the version 1.2.5 update released in november 2018 greatly improved performance of the video recordings and made the file saving process clearer at the end of video recording. ample documentation is provided for setting up the obs software and equipment with several recommendations. cons: updates are seemingly rare based on the version history in the app store. the software is only compatible on macintosh os x 10.10 or later with a 64-bit processor and will not work on a pc. you are required to use the equipment specified in the setup documentation provided by penn state. the audio options are limited to the sound feed from the camera and hdmi output. thus we recommend selecting a camera that has a line-in input instead of just a microphone input. there are no editing options so other software must will be required for users which will likely require consultation from experienced staff members. elgato game capture pros: has a plethora of features including: video recording, separate audio source recording, live streaming, basic editing, and more. the program is included with the purchase of an elgato hdmi capture device and works with any camera with an hdmi output and most streaming platforms. cons: the software only works with a compatible elgato hdmi capture device which must be purchased. the software is mildly complicated, especially for a user that does not have technical experience or video know-how and will likely require instruction to be utilized by library users. in our experience, the live stream function worked well so long as you had an rtmp address and key. the embedded login functions for live streaming didn’t appear to work as well. kaltura capture space (personal capture) pros: the software easily records audio, screen capture, and webcam in any combination of the three. for the university of utah, kaltura is used as the sole institutional video hosting and display solution which offers integration across many platforms including the university’s learning management system (instructure canvas) and the university’s video hosting platform (mediaspace.utah.edu) cons: the update from “capture space” to “personal capture” removed the functionality to record a powerpoint presentation on its own, without the use of the “screen capture” which is more complicated for presentation recording. the software only uploads to the kaltura video hosting platform, meaning finding the original recording or downloading a copy is difficult and requires downloads to be turned on in settings. adobe creative cloud pros: the standard for which most creative software is measured against. the suite includes a plethora of tools such as photoshop, premiere pro, illustrator, after effects, and many more. if included in an institution’s licensing agreement with adobe, it may be available at little or no cost at the departmental or individual level. the individual software are powerful and generally as capable as the user using them with a plethora of instructional information on the web. cons: if the institution does not have a licensing agreement with adobe, it can be very expensive to pay for. most library users are going to be intimidated by the idea of the software and will require either extensive help documentation or staff/faculty help to get going on their projects. similarly, the interfaces are dense and complex, and require a certain level of familiarity with adobe products to truly get a hang of outside of some basic functions. garageband pros: garageband has a much simpler interface compared to adobe audition or other similar “digital audio workstations.” it also comes free with apple computers. the ability to record multitrack, especially when paired with an audio interface such as the focusrite scarlett series makes for easy, professional sounding audio recording without much experience. many audio plugins such as equalizer, compression, reverb, etc. are included in the software at no additional cost. cons: several functions that are available in apple logic pro, garageband’s sister software, such as razor tool, waveform editing, and simple track controls are not available or as prominent in garageband. compared to adobe audition, garageband is less robust and doesn’t offer as many features for higher production value such as track automation. imovie pros: many users that have a macintosh may already have some experience using imovie. the interface and editing functions are fairly rudimentary and straightforward but are lacking in comparison to something more robust like premiere pro. cons: many simple functions that are obvious on premiere pro are not always so obvious on imovie. in many cases the project requires more robust editing than what imovie provides. libguides pros: as a solution for hosting library information, springshare’s libguides is a helpful tool for creating, editing, and storing information on a subject or topic basis. part of the libapps suite, libguides offers a “wysiwyg” editor that makes it simple to make decent looking web pages for your projects. it also allows tie-in with other libapps products such as libcal which can handle scheduling for your multimedia studio. cons: in some cases the libguides can be cumbersome to use, especially when a particular libguide has a lot of information to display. the handling of widgets and boxes in libguides is not very intuitive and adds a level of complication when laying out the look and feel of your page. unless otherwise dressed up through web design and images, the default look and feel of a libguide can be lacking. libwizard pros: part of springshare’s libapps suite, libwizard is a tool that helps make forms, surveys, quizzes, tutorials and assessments easily. data is easy to collect and analyze with libwizard’s built in data, field, and cross tab analysis tools. it is helpful for creating a “one-stop shop” solution instead of utilizing a hodgepodge of free services such as survey monkey or google forms. cons: similarly to libguides, an institution must have a license to the libapps product through springshare which comes at a cost. most of the functions are the same as free services like survey monkey or google forms. phase four to support the tree of hippocrates education studio, a student intern with prior multimedia experience from a public library makerspace was hired in january 2018. they helped create closed captions for the instructional videos, worked closely to consult on specific student and faculty projects, and created guides for podcasting. the additional student support was invaluable and avenues for internships are being sought to help provide additional dedicated support to the studio. having interns with video and audio production knowledge will provide one-on-one guidance on projects while also providing the interns an opportunity to expand their skills in multimedia design. after the first year of operation, more and more clinicians from the university of utah hospital were using the studio. patient education videos are in high demand and the hospital lacks the equipment, expertise, and coordination to help with such projects. the eccles library has partnered with imagine perfect care, a hospital grant-giving entity, so the library can provide further assistance with projects specific to patient education. while only in the pilot stage of this partnership, clinicians have found many added benefits of having a nearby multimedia space to create videos for their patients. in reflection, we found specific equipment and software as essential to the success of the studio. while not an exhaustive nor necessarily an “ideal” list of equipment, the following pieces can help kick start a successful multimedia studio that works with penn state’s one button studio software and allows for the creation of other multimedia projects with different software. starter pack for studio equipment: computer imac 27 inch retina 5k display fusion 8gb audio/video devices: canon vixia hf g21 full hd camcorder focusrite scarlett 18i8 or similar audio interface (the more xlr inputs the better) blackmagic design h.264 pro recorder (required for use with obs software) elgato hdmi capture device (for use with elgato game capture software; not compatible with obs) microphones: sennheiser ew 112-p g3 camera-mount wireless microphone system with me2 lavalier mic – a (516-558 mhz) røde ntg2 shotgun microphones w/ windscreens and desk stands headphones: sony mdrv6 studio monitor headphones with ccaw voice coil teleprompter: magicue mobile teleprompter kit with hard case lighting: dracast led500 pro bi-color led light dracast led500 s-series bi-color led 3-light kit with v-mount battery plates backdrop: impact background system kit with 10×12′ with a variety of backdrops (white, gray, black, green, etc) usb devices: button – griffin technology powermate assignable usb multimedia controller (for obs) anker 4-port usb 3.0 hub (for obs) cables & adapters: a generous slush fund to purchase those pesky forgotten cables and adapters. approximate total price: $5,850 in looking to the future, new equipment and software will be added to complement those already existing in the space. an upgraded camera with a line-in audio port, more robust teleprompter applications, and additional sound treatment foam are set to be purchased. camtasia, a screen capture and video editing software package is frequently requested and is slated to be purchased. another future development will be a user-centered reservation system where users can book the studio space based on their availability without emailing or visiting the library staff. this will ideally be done through a calendar on the web and perhaps on a tablet or kiosk outside of the studio space. we look forward to continually evolving and building on the success of the space and continually inquire feedback to support future developments from our students, staff, and faculty. references [1] brandon patterson, tallie casucci, bryan elias hull & nancy t. lombardo (2018) library as the technology hub for the health sciences, medical reference services quarterly, 37:4, 341-356, doi:10.1080/02763869.2018.1514899 appendix 1 follow-up survey with responses overall how satisfied were you with the studio satisfied somewhat satisfied neutral somewhat dissatisfied dissatisfied 13 1 – – – overall how productive did you feel you were with the tree of hippocrates education studio? productive somewhat productive neutral somewhat productive unproductive 12 2 – – – i was able to understand how to use the space or equipment. agree somewhat agree neutral somewhat disagree disagree 9 3 2 1 – i knew where to go to get assistance with the space or equipment. agree somewhat agree neutral somewhat disagree disagree 13 2 – – – how likely are you to use the space or equipment again. likely somewhat likely neutral somewhat unlikely unlikely 13 2 – – – what is your role at the university? student staff faculty other 3 5 4 – what is your university affiliation? school of medicine college of nursing college of pharmacy school of dentistry college of health hospitals/clinics other 6 1 – – – 4 1 did you experience any problems with the space or technology? the lighting was hard to adjust without getting a huge glare on the tv screen. it would be great if there was more guidance or information on the website for tips with this. it was also not set up for the one-touch. we had to plug some things in and restart the program to get the recording to start. it was a bit tough to figure out which cables i was supposed to plug in where. no the button didn’t work to start the recording. but the space bar did. the first time i used it, i had to take the flash drive out and re-start the program each time i wanted to record (new recording). this did not occur though the second time i used the studio. microphone wasn’t connected. one button record didn’t work. yes. the equipment worked just fine but we heard a lot of noise/conversations from the office next door. we had to pause our recording quite a bit because the recording are going to go to quite a large audience. i think the room is great and i am very happy to have access to it. however, it needs soundproofing. when i was recording, the helicopter landed or took off and it made it on the tape. i also had to pause a few times because of conversations in the office next door. no problems whatsoever. great experience do you have any recommendations to better the space or technology for yourself or future users? more information and troubleshooting information in the room, or at least on the website. the podcasting platform i use works only on the phone and not on the computer. thus, it would be nice if we had a microphone i could plug into my phone. have a table in the taping area have a projection ability for a script. i use the chair from the desk to sit in front of the camera. i am reading off of a script, which i have to hold because there is no where to put. i think this makes it more obvious that i’m reading. ideally, a teleprompter would be fantastic. but having something that allows you to see and read your script that is situated behind the camera would be really very helpful. this is such a great space to assist with education not only academic but professional education as well!! i had a great experience using the studio twice–once with a student and once by myself. i hope to use it again in the future. it’s a great resource to have! provide better soundproofing to the room and if possible move the office that is next to it. however, i loved the how to and instruction documents, i can see them being really helpful for someone that doesn’t have a audio engineering background. it was a little tricky recording good sound because the room isn’t sound proof. none, except spread the word i do not. i’ve already recommended the studio to several colleagues. wonderful resource! about the authors bryan elias hull is the digital publishing program manager at the eccles health sciences library at the university of utah. bryan’s work focuses on enhancing scholarly communication through the library’s digital publishing and educational technology platforms and programs. email: bryan.hull@utah.edu brandon patterson is the technology engagement librarian at the eccles health sciences library at the university of utah. he connects students, staff, and faculty to digital tools and emerging technologies and creates meaningful experiences using prototyping tools, virtual reality, and online learning platforms. he is a health sciences education liaison and coordinates with faculty to incorporate information literacy instruction and technology into their classrooms. email: b.patterson@utah.edu subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – egosystem: where are our alumni? mission editorial committee process and structure code4lib issue 24, 2014-04-16 egosystem: where are our alumni? comprehensive social search on the internet remains an unsolved problem. social networking sites tend to be isolated from each other, and the information they contain is often not fully searchable outside the confines of the site. egosystem, developed at los alamos national laboratories (lanl), explores the problems associated with automated discovery of public online identities for people, and the aggregation of the social, institution, conceptual, and artifact data connected to these identities. egosystem starts with basic demographic information about former employees and uses that information to locate person identities in various popular online systems. once identified, their respective social networks, institutional affiliations, artifacts, and associated concepts are retrieved and linked into a graph containing other found identities. this graph is stored in a titan graph database and can be explored using the gremlin graph query/traversal language and with the egosystem web interface. by james powell, harihar shankar, marko rodriguez, and herbert van de sompel introduction egosystem was developed to support outreach to former los alamos national laboratory (lanl) employees. as a publicly funded research organization, there are  many reasons why lanl would want to maintain contact with its “alumni.” scientific research is often collaborative. former employees know the lab and its work, and often have colleagues who remain employed at lanl.  these relationships fuel intraand interdisciplinary projects. government research agencies are also encouraged to partner with the private sector. productizing lanl output becomes an opportunity for a public-private or commercial entity via a technology transfer process. some small businesses (and jobs) owe their existence to ideas that were first developed at lanl. public support for the ongoing research at lanl plays a role in ensuring support for  adequate funding levels for that work. outreach to alumni can encourage them to serve as ambassadors and advocates for lanl and its mission. cross searching of online social networks remains an unsolved problem. these networks are, to some extent, a reflection of the ties between people in the real world. people  use online social networks to establish connections with one another and to share information. they may also explore the social networks of their connections to make new connections of their own to friends, family members, co-workers, experts in a particular field, or popular celebrities such as writers, film or tv stars, politicians, etc. users are generally limited to exploring each online social networks to which they belong within the confines of that site. for example, the social network within twitter consists of people who follow or are followed by other users. twitter users can explore these relationships when they are logged into twitter to find new connections. facebook users can use facebook graph search (https://www.facebook.com/about/graphsearch) to locate other users. but aggregate comprehensive search is generally not possible. online social networking sites remain walled gardens [1]. egosystem is intended to address social searching across identities for a defined population (postdoctoral students). postdocs spend a fixed amount of time at lanl and many then move on to private sector employment. exit interviews ensure that lanl retains some basic demographic information about the students. egosystem uses this demographic information to discover postdoc identities on the web. a discovery process uses some simple, iterative heuristics to find public online network identities and the explicit networks surrounding them. it also retrieves public online information about the individual that might lead to additional connections. person identities can be connected through co-authorship relationships, co-employment relationships, or information that reveals that two individuals work, study or perform research in similar fields. in  egosystem’s graph, discovered identities are mapped to primitive vertex types  which we refer to as platonic vertices. egosystem provides a web interface with social search exploration and analysis capabilities for this graph. platonic vertices that correspond to a person, concept, artifact, or institution each have their own profile page. select end users can use the web interface to search or traverse the connections among these vertices. figure 1: egosystem displaying profile page for marko rodriguez discovery the discovery process for egosystem is designed to find online identities of lanl’s  postdoc alumni community. lanl conducts a basic exit interview of postdocs before leaving the lab, and the discovery process uses this as seed data. it constitutes  minimal data necessary to locate and validate an online identity for a person. an example seed entry for a postdoc is shown below: name: rodriguez, marko phd university: university of california, santa cruz field of study: computer science, computer networks joining company: at&t title: scientist field of work: graph theory, graph databases the discovery process searches for identities in a predefined set of social and academic web portals. it approximates the steps a human would use to find online identities. discovery starts with a web search, which is based on some basic demographic information about a person. when a person identity is found, identities are also often found for related institutions and artifacts. this information is used to augment the initial seed data, and the process is repeated. during subsequent iterations of the discovery process, other person identities, affiliations, artifacts and concepts may be discovered. concepts are discovered using the augmented data since they are based on keywords associated with affiliations and artifacts. the same is true for connections, which are discovered once person identities, institutions, and artifacts have been found. egosystem starts to create connections between person vertices as more concepts, artifacts, identities and institution affiliations are found. figure 2: discovering identities at this point, egosystem’s discovery process targets identities within microsoft academic, linkedin, twitter, and slideshare. it also searches for public homepages and wikipedia pages. institutional identities are retrieved from wikipedia, linkedin, or the institutional home pages (like http://lanl.gov/).  geocoordinates for institutional locations are discovered using the google maps api. concepts are retrieved from linkedin, microsoft academic and slideshare. artifact metadata is retrieved from microsoft academic, slideshare and twitter. some identities are discovered through yahoo search using the yahoo boss api, which is its programmatic interface. the yahoo search strategy employs the same techniques a person might employ, using various combinations of seed data in conjunction with the name of a site of interest. for example, here are some variations for a query on marko: marko+rodriguez+linkedin marko+rodriguez+los+alamos+national+labs+linkedin marko+rodriguez+graph+database+slideshare the results of these various search strategies are analyzed, duplicates are removed, and the html page for each resulting url is retrieved. this html page is parsed and identity information is passed to the validator for verification.  this works well for locating content-rich pages such as those within wikipedia, but less so for social networking sites that lack an api, such as linkedin. screen-scraping is brittle, as even minimal changes to the underlying html can cause identity discovery to fail. therefore, whenever possible, site specific programmatic apis are used. microsoft academic, twitter and google maps provide apis that support searching and retrieval of data in machine readable formats such as xml and json. native, service-specific apis like microsoft academic provide information about the person directly, and this data can be passed on to the validation module. web homepages also pose a challenge. hence, other techniques as discussed in [4] and [5] are used to retrieve information and validate this content. to determine if a webpage is indeed a homepage of a person, egosystem checks if the domain name in the url is the domain name of an institution the person is associated with. then, it checks to see if the title of the page contains the full name of the person. it also checks for the person’s name in the page’s url (like http://public.lanl.gov/mrodriguez). seed data first discovery iteration adds second discovery iteration adds person: name: rodriguez, marko affiliation (phd university): university of california, santa cruzfield of study: computer science, computer networks institution (left to work at): at&t title: scientist concepts (field of work): graph theory, graph databases new identity: www.linkedin.com/in/markorodriguez affiliation: worked at graph systems architect institution: aurelius, llc affiliation: visiting researcher at institution: vrije universiteit brussel concepts: machine learning, distributed systems, hadoop, graph theory, algorithms, semantic web, natural language, rdf, etc. new identity: twitter.com/twarko artifact: (recent tweets) connections: identities of people that  follow, are followers of this identity new identity: http://academic.research.microsoft.com/author/3534935 artifacts: a path algebra for multi-relational graphs the rdf virtual machine constructions from dots and lines the graph traversal pattern concepts: artificial intelligent, computers and society, data model, data structure, decision making, digital library, etc. connections (co-authors): johan bollen, herbert van de sompel, jennifer h. wakins, alberto pepe, etc. new identity: http://www.slideshare.net/slidarko artifact: faunus graph analytics engine solving problems with graphs titan: the rise of big graph data the pathology of graph databases connections: identities of people following this identity the validator component is responsible for verifying that the profile obtained by the discovery process is a correct identity for that person. it compares the seed information to the information retrieved from searches, and awards points for every correct match. if the profile scores more points than a set threshold, then it is considered a correct identity for the person. an exact match for the last name of a person is necessary to validate a profile, otherwise the validation fails at this step. for first names, if an exact match is not found, fuzzy pattern matching is used to determine if the first names are close. for the name “james powell”, “jim powell” would be accepted by the validator while “james powel” will not be. after the name is validated, at least one of the retrieved institution names must match the institution listed for that person in the seed data. again fuzzy pattern matching is used to match the name of each institution. if all of the institutions in the seed information are determined to be valid, then the profile is accepted as a valid identity. if only one institution is deemed valid, then at least one area of expertise or the job title must match in order for the profile to be validated. the scoring system to validate an online profile was introduced by northern and nelson [2]. twitter profiles generally do not contain professional information, so egossytem uses a validation technique comparing institutional twitter identities to their immediate social neighborhood. if a candidate twitter identity is following an institution listed in that person’s seed data, and the name for this identity also matches, then the twitter identity is accepted as valid. pilot project seed information for 3005 postdocs were uploaded to egosystem and the discovery algorithms were executed for these individuals. the discovery algorithms for linkedin, twitter, and microsoft academic only require the original seed data and can run independently of one another. these algorithms supply additional information required by the other discovery algorithms which locate homepages, wikipedia pages, mendeley,  slideshare, and other identities.  linkedin and microsoft academic were the sites that provided the most information about the lab’s postdoc community. among the 3005 postdocs, the discovery modules found 1963 microsoft academic identities, 833 linkedin identities and 176 twitter profiles. at the end of the pilot project, we conducted a cursory review of the discovery algorithm performance. this consisted of a manual search of a subset of postdocs which was then compared with the results of the discovery algorithms. among the 100 postdocs that were manually searched, we found that the discovery algorithm results were correct for 86 identity searches in microsoft academic. more specifically the algorithm found 71 correct identities, and did not find identities for 15 other people who in fact did not have profiles.  among the 14 incorrect identities, 2 were wrongly assigned and 12 identities that existed were not found. for linkedin, the discovery algorithm results were correct for 88 identity searches in linkedin. in this case, it found 34 identities which were correct, and did not find identities for 54 other individuals who did not have linkedin profiles. the remaining 12 identity search results were incorrect, of which one profile was incorrectly assigned to a user, and 11 existing profiles were not discovered even though they existed. for twitter identities, 4 of the identities found by the algorithms were incorrect and 2 existing profiles were not found. the remaining 94 people did not have twitter identities, and the discovery algorithm correctly determined this to be the case. in this limited sample, it appears that the discovery algorithms achieve both high precision and high recall for this set of online identities in the targeted web sites. the graph the data accumulated via the discovery process is mapped into a property graph model [4] defined for egosystem (see figure 4). a property graph contains vertices and edges which can have arbitrary key/value pairs associated with them. it is the specific key/value pairs chosen for a given property graph that refine the vertex and edge types for a given application. in the case of egosystem, vertices fall into two categories, platonic vertices and affiliation vertices. a platonic vertex  is one of the four primitive types: person, institution, artifact, and concept. these four types share many of the same properties, as illustrated in figure 3. affiliation vertices bind platonic nodes together in flexible ways. for example, a postdoctoral student might have both worked for and studied at a given institution for different time periods, and two affiliation vertices are used to express these distinct relationships. edges represent the direction of a relationship and their properties are concerned with the type of relationship, including workedfor, studiedat, authored, hasconcept, and hasaffiliation. these vertices and edges allow all the various relationships and objects discovered to be represented within egosystem’s graph. figure 3: example of a property graph the property graph is stored in aurelius’ titan, a distributed graph database capable of representing and processing graphs on the order of hundreds of billion edges and sustaining tens of thousands of transactions a second over a multi-machine compute cluster. this technology enables organizations to build massive domain models that include not only the direct information of their objects of inquiry (for example, particular people), but also the periphery of information that can be leveraged to understand how these objects are embedded within a larger context (for example, other objects of interest such as other people, artifacts, organizations, and concepts). titan enables egosystem to encompass not only the data in professional graphs (for example, lanl employee database, linkedin), but also knowledge graphs (for example, wikipedia), artifact graphs (for example, microsoft academic, arxiv, github), and personal graphs (for example, facebook, twitter). adding machine nodes to the cluster enables more storage and compute resources to be allocated to storing and processing the graph. problem-solving is accomplished via the gremlin graph traversal language. gremlin allows graph traversals to be represented as path expressions. for example, a request to list the names of people who marko doesn’t know, but who are known by marko’s friends: g.v('name','marko').out('knows').name to expand upon that query, a request to list the names of the people that marko’s friends know that marko doesn’t already know is expressed as: g.v('name','marko').out('knows').aggregate(x).out('knows').except(x).name gremlin supports memory and branch structures which allow it to recognize or traverse any arbitrary path through a graph. when gremlin is used with titan, sub-second local neighborhood traversals are enacted (similar in form to the traversals above). in order to do long-running, global analytics of the graph, aurelius provides faunus. faunus executes parallel sequential scans of titan graphs in order to support bulk loading, global graph analysis, bulk mutations, etc. as is the case with titan, gremlin is also the graph traversal language leveraged by faunus. for example, to determine the distribution of age within the graph, the following global scan can be enacted: g.v('type','person').age.groupcount we chose titan over other currently available property graph databases (such as orientdb, neo4j, dex, and infinitegraph) because of its apache2 license, horizontal-scalability, global analytics package (via faunus), and direct support of the standard blueprints graph api (analogous to jdbc but for the graph database community). a property graph is known in academic circles as a directed, attributed, multi-relational, binary graph: directed: an edge has a tail and a head vertex. attributed: vertices and edges can have an arbitrary number of key/values pairs. multi-relational: edges are types to support multiple types of relationships binary: an edge connects only two vertices another popular graph representation is the rdf graph data model of the semantic web effort. there are two primary distinctions between these two models: in rdf, vertices and edge labels are identified by uris; and property graphs support properties on edges. a major modeling benefit of the property graph model is that edges can support properties and thus, edges can have, for example, timestamps, weights, provenance information, etc. in rdf such edge reification is solved using “blank nodes” and typically leads to a graph structure that is difficult to query. another major distinction between rdf and property graphs is the means by which they are queried. sparql is a prolog-like language for rdf where patterns are specified and those resources that bind to the pattern are returned. conceptually, sparql queries a graph using pattern matching and gremlin queries using traversal. for example, “who are the friends of marko that have also been employed by the los alamos national laboratory?” in sparql: select ?y where { lanl:marko foaf:knows ?x ?x foaf:workedfor lanl:lanl ?x foaf:name ?y } in gremlin, the above is written as: g.v('name','marko').out('knows').as('x') .out('workedfor').has('name','lanl').back('x').name gremlin supports complex path traversals as well as property value matching and filtering on both vertex and edge properties. free-form text searches against property values use elasticsearch, which is bundled with titan. elasticsearch uses lucene to build a full text index for the name property value for each platonic vertex. this enables users to search for person and institution names, artifact titles, and keywords. elasticsearch is also used to build a geospatial index of points representing locations for institutions. geospatial searches against this index return items that fall within a circle that represents the user query. elasticsearch makes it possible to search for person vertices that have institutional affiliations within a certain distance to a given location.  gremlin works transparently with elasticsearch. here is a  traversal which expands on a previous example to determine the distribution of the ages of people in the santa fe area: g.v('type','person').has('location',geo.within,geoshape.circle(35.6843,-105.961,20)).age.groupcount egosystem was initially populated with data about 3005 postdoctoral students, which were represented as platonic person vertices. the discovery process found 116,466 non-lanl people which were connected to these postdocs in some way, of which 38,111 represented twitter identities and 78,355 were author identities from microsoft academic. it also found 1,395 institutions through linkedin. the total number of vertices in the resulting property graph for egosystem was 9,015,844, and the total number of edges was 19,399,683. figure 4: egosystem property graph model the user interface the property graph can be explored using egosystem’s web interface. the default starting point is a simple, google-like search page with a single input field and a search button. this search executes four searches against properties for all four platonic vertex types. results are organized by platonic vertex type: people, institutions, artifacts and concepts. egosystem’s web interface is implemented in javascript and runs in the user’s browser. it makes extensive use of ajax technologies to communicate with the server hosting the property graph, and to dynamically generate web content. the interface makes calls to a server side rest api, implemented in groovy, which uses the blueprint api [5]. the backend accepts json requests and returns vertex and edge information as json objects. when a user searches for a person by name, the following steps occur: their query is converted to a json object the json object is submitted to a rest request called getthingbyname the service responds with a json object that contains all of the information for matching platonic vertices here is an example of a json client request, and a json server response: request: {"name": "marko"} response: { "properties": { "platonic": "person", "name": "rodriguez, marko", "starttime": null, "endtime": null, "frame": "gov.lanl.egosystem.frames.platonic.person", "locked": 1374698679437, "uri": "urn:uuid:e92925c5-91d1-4b26-a67b-48c561643edf", "alias": null }, "methods": [ { "method": "discovertwitter", "parametertypes": "[egosystemgraph]", "returntype": "boolean", "canexecute": true, "description": "execute the twitter discovery algorithm for this platonic" } ... ], "identities": [ { "properties": { "location": null, "handle": "twarko", "service": "twitter", "logo": "http://a0.twimg.com/profile_images/395313322/exterminator_normal.png", "name": "marko a. rodriguez", "starttime": null, "endtime": null, "frame": "gov.lanl.egosystem.frames.identities.twitter.twitterperson", "locked": null, "uri": "http://twitter.com/twarko" }, "methods": [ { "method": "getfollowers", "parametertypes": "[]", "returntype": "iterable", "canexecute": true, "description": "no description is available" }, ... the javascript interface is implemented as a set of objects representing platonic types, the results of a query, or the results of a path traversal. these objects are implemented using prototype functions [7].  for example, instances of the egoplatonic object encapsulate  platonic vertex properties, regardless of the platonic type, as well as set and get methods. when an instance of an  egoplatonic object is created, values from properties of a vertex returned by a getthingbyuri request are mapped to fields in the object. the object can then be inspected via its get methods. function egoplatonic(jsonplatonic) { this.uuid = jsonplatonic.properties.uri; this.name = jsonplatonic.properties.name; this.hash = jsonplatonic.properties.hash; this.kind = jsonplatonic.properties.platonic; ... egoplatonic.prototype.getname = function() { return this.name; } egoplatonic.prototype.getkind = function() { return this.kind; } egoplatonic.prototype.getservices = function() { return this.services; } the web interface dynamically constructs a profile page using vertex data contained in an instance of an egoplatonic object, in combination with a set of simple rules represented as a json object. a platonic person profile is a dynamically generated web page that incorporates all of the affiliations and identities in the property graph for that individual. a profile page is permanently addressable via a persistent and shareable url. much of the profile page information depends on the identities associated with the particular platonic vertex. each identity vertex includes properties which define the kinds of services it supports. a rule set matches service types with graphical components that can render them. { "antecedent": { "methodname": "getplatonic", "returntype": "platonic", "parametertypes": "[]", "canexecute": true, "uri": "http://academic.research", "service": [ "msacademic" ], "renderas": "*", }, "consequent": { "component": "author.html" } }, all platonic types have their own distinct representation in the web client. an institution page includes institutional identities such as the organization’s homepage and wikipedia page, if found, as well as clickable lists of concepts and people associated with that organization. concept pages (shown in figure 5) and artifact pages are built in a similar fashion.  egosystem constructs a concept or profile page by first retrieving information about the concept or artifact’s platonic vertex, and then traversing outgoing edges to gather information about identity vertices to which the platonic vertex is connected. when a user clicks on concepts, people or artifacts, they are actually navigating the underlying graph via identity and platonic vertices. if a user clicks on a link for an identity, egosystem will attempt to resolve that identity to a platonic vertex and display a profile page. if no platonic vertex exists  (for example, a non-lanl person identity), egosystem will redirect the user to an external web page for that identity. figure 5: egosystem concept profile page the user interface provides two other types of search. a user can search for individuals who are or were located within a certain distance of a particular geographic location. this interface uses an ajax-based location lookup service to suggest and resolve place names to latitude and longitude coordinates, which are then searched against location information associated with identities in the graph. results are overlaid onto google maps.  users may also search for individuals who were associated with an institution during a certain time range. the results are presented as a list of person names which point to the profile page for their platonic vertex. graph visualizations and textual list-based representations of first degree social networks are available from a person’s profile page. the graph visualization is generated using the javascript d3 libraries [8]. users may interact with the graph, rearrange vertices, or click through to profile pages just as they can from the text view of the social network. both lanl and non-lanl identities are included in these social networks. path discovery is also supported through the interface. a user may “save” a person as they search or browse egosystem. from another person profile, the user can locate identity-specific paths that might exist between these two people, including institution, twitter, co-authorship, etc. if a path exists between these two people, the traversal can be viewed as a graph visualization or a list of links for the vertices that connect them. the “save” function also supports a rules-based comparison tool. a user may compare the person they are currently viewing with a saved person. during the compare process, egosystem’s interface iterates through the identities for each person. when both individuals possess a given identity, rules determine which services associated with those identities should be called. if both people have a given identity, say a twitter identity, then service requests (such as getfollowers) are made for each twitter identity, and the results are merged and displayed. the comparison view shows shared social connections, concepts linking two people, and a combined timeline of employment history. figure 6 shows the results of an egosystem compare request for two lanl alumni. starting at the top left, egosystem presented their combined publication metrics as a textual summary and as a bar chart. since both people have a twitter account, egosystem called the getfollows and getfollowers methods to construct a pair of graph visualizations to show their combined follows and followers networks respectively. next, egosystem displayed a combined employment and education timeline. since both people have linkedin identities, egosystem called the getconcepts method for each to construct a combined concept graph. both people also have microsoft academic identities, so egosystem called the gettagcloud method for each to generate a combined tag graph from that data. finally, egosystem called two other methods associated with microsoft academic identities, getcoauthorpaths and getcoauthors, to generate a shared co-authorship path visualization and a combined co-authorship graph. figure 6: egosystem comparison interface the following example shows the json request and response for a getfollowers service request: request: {"uri": "http://twitter.com/twarko" "method": "getfollowers"} response: { "result": [ { "properties": { "outdegree": null, "indegree": null, "location": null, "handle": "drmichaelham", "service": "twitter", "logo": "http://a0.twimg.com/profile_images/2927745307/5e05b6cc43e0e10bb1dee2f95c5af5e7_normal.jpeg", "name": "drmichaelham", "starttime": null, "endtime": null, "frame": "gov.lanl.egosystem.frames.identities.twitter.twitterperson", "locked": null, "uri": "http://twitter.com/drmichaelham", "alias": null } }, ... egosystem’s web interface also allows adding new people to the graph via the “man on the street” form. a user can supply some basic demographic metadata about a person to which they seek a connection through a lanl alumni. this information, like the seed data used to initially populate the system, includes name, a place they worked, job title, a university they attended, and some keywords related to their area(s) of expertise. as with the initial bootstrap process, a platonic vertex is added for this person, and then each discovery method is executed. any identities are added to the graph. the user can then immediately view the new person’s profile, look at their identities, concepts, institutions, and artifacts, find paths to them through lanl people, or compare them to other people in egosystem. profiles provide access to a person’s immediate neighborhood. the profile comparison tool and the path queries show a portion of the graph that links two people. there are also lower level tools for cross-graph analysis. the system can generate reports on graph-wide characteristics such as total vertex and edge counts, or counts by vertex or for all vertices which are of a particular type, or which have a property that has a particular value. demographic reports such as how many postdocs came from or went to a given institution are possible because the underlying graph records the direction of an affiliation and the temporal data associated with that relationship. there are also some more advanced, focused reports for analysis such as by-institution flows. data for these reports is output in standard graph markup languages because the resulting subgraph can be quite large. this enables users to download the graph representation and then load, visualize and manipulate the data using more robust desktop graph visualization tools such as gephi. conclusion egosystem’s discovery module, property graph, and web interface work together to provide what we believe is a novel and useful aggregated social search capability for a defined community. it automates the otherwise time-consuming and laborious task of finding public online identities with a reasonably high degree of accuracy. it aggregates and stores this information, many aspects of which are inherently network oriented, as a richly descriptive property graph in a graph database. the results describe a community of former lab-affiliates augmented with social network information, geographic data, institutional affiliations, temporal data, and data about their intellectual pursuits. as originally envisioned, select lanl staff can use this system to locate, re-establish contact, and establish closer ties between lanl and its alumni. perhaps just as  importantly, the connections that accumulate in this graph can reveal additional information that is not readily apparent. this could prove beneficial for activities such as recruiting and identifying partners for collaborations in the future. endnotes [1] hinchcliffe, dion, 2014. where is interoperability for social media? zd net, enterprise web 2.0, february 28, 2014. http://www.zdnet.com/where-is-interoperability-for-social-media-7000026894/ [2] northern, carlton t., and nelson, michael l. 2011. unsupervised approach to discovering and disambiguating social media profiles, proceedings of mining data semantics (mds 2011). [3] yi fang, luo si and mathur, aditya p. 2010. discriminative graphical models for faculty homepage discovery. inf. retr. 13, 6 (december 2010), 618-635. http://dx.doi.org/10.1007/s10791-010-9127-7 [4] qi, x. and davison, b. d. 2009. web page classification: features and algorithms. acm comput. surv. 41, 2, article 12 (february 2009), 31 pages http://doi.acm.org/10.1145/1459352.1459357 [5] rodriguez, marko a. and neubauer, peter. 2010. constructions from dots and lines. bulletin of the american society for information science and technology, volume 36, issue 6, pages 35–41, august/september 2010. http://dx.dio.org/10.1002/bult.2010.1720360610 [6] blueprints. https://github.com/tinkerpop/blueprints/wiki [7] port, sebastian. 2013. a plain english guide to javascript prototypes. http://sporto.github.io/blog/2013/02/22/a-plain-english-guide-to-javascript-prototypes/ [8] d3 data-driven documents. http://d3js.org about the authors james powell is a research technologist in the research library at los alamos national laboratory, where he is currently a member of the digital library research & prototyping team. since joining lanl, he’s been involved in a number of information retrieval projects that incorporate semantic web and graph analysis technologies. his current interests include graph theory and complex systems. harihar shankar is a research and development engineer in the research library at los alamos national laboratory. he holds a master’s degree in computer engineering from the university of new mexico and his interests include digital preservation and retrieval, information infrastructure, data mining and scientific communication. dr. marko a. rodriguez is the founder and ceo of the graph computing company aurelius, where the team focuses on the development of the open source graph computing technology. marko is the lead developer of the gremlin graph traversal language and the faunus graph analytics engine. previous to aurelius, marko was a graph architect at at&t and a postdoc at the center for non-linear studies at the los alamos national laboratory, where he focused on the development of a multi-relational graph algebra. marko received his masters and ph.d. from the university of california at santa cruz in computer science and his bachelors in cognitive science from the university of california at san diego. herbert van de sompel graduated in mathematics and computer science at ghent university (belgium), and in 2000 obtained a ph.d. in communication science there. for many years, he headed library automation at ghent university. after leaving ghent in 2000, he was visiting professor in computer science at cornell university, and director of e-strategy and programmes at the british library. currently, he is the team leader of the prototyping team at the research library of the los alamos national laboratory. the team does research regarding various aspects of scholarly communication in the digital age, including information infrastructure, interoperability, digital preservation and indicators for the assessment of the quality of units of scholarly communication. herbert has played a major role in creating the open archives initiative protocol for metadata harvesting (oai-pmh), the open archives initiative object reuse & exchange specifications (oai-ore), the openurl framework for context-sensitive services, the sfx linking server, the bx scholarly recommender service, and info uri. currently, he works with his team on the open annotation, memento (time travel for the web), resourcesync, and hiberlink projects. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – building a library app portfolio with redis and django mission editorial committee process and structure code4lib issue 19, 2013-01-15 building a library app portfolio with redis and django the tutt library at colorado college is developing a portfolio of library applications for use by patrons and library staff. developed under an iterative and incremental agile model, these single-use html5 applications target multiple devices while using bootstrap and django to deliver fast and responsive interfaces to underlying frbr datastores running on redis, an advanced nosql database server. two types are delineated: applications for access and discovery, which are available to everyone; and productivity applications, which are primarily for library staff to administer and manage the frbr-rda records. the access portfolio includes book search, article search, call number, and library hours applications. the productivity side includes an orders app and a marc batch application for ingesting marc records as frbr entities using rda core attributes. when a critical threshold is reached, the tutt library intends to replace its legacy ils with this library application portfolio. by jeremy nelson background colorado college’s tutt library is a small, urban, academic library serving the needs of over 2,000 students along with faculty and staff in colorado springs, colorado. as a member of the colorado alliance of research libraries, a consortium of academic and public libraries in colorado and wyoming, we participate in a union catalog comprising the collections of our member institutions. our islandora institutional repository is also hosted through the colorado alliance’s repository service. we operate our own instance of iii’s millennium ils, however. like most academic libraries, our material budgets have shifted from physical to electronic resources and, as a consequence, we are doing more batch loading of marc records for these electronic resources and less original or copy-cataloging of print material. with the quality of vendor-supplied marc records varying considerably, the old workflows to manipulate and load records into our legacy ils were often long and laborious processes. by scripting the manipulation of these marc records with python and the pymarc python module [1], and with a web front-end built in django, we brought considerable time savings to our cataloging staff. this led to a second django project that enabled senior students to self-submit their thesis or final essay, along with any supporting datasets, to our digital repository. a parallel effort was started as we looked at various commercial and open source options for a new discovery layer. given monetary and resource constraints, along with the worry of maintaining multiple codebases in different programming languages and environments, the library decided to fork kochief, a django-based discovery project. this decision led to the release of aristotle, the tutt library discovery-layer project available on github under the apache 2 open source license [2]. in 2011, we started to explore using redis, a popular nosql technology, to represent bibliographic and operational information. this research led to the frbr-redis datastore project that was the topic of a 2012 code4lib presentation [3]. using the library of congress bibliographic framework initiative’s “a bibliographic framework for a digital age” [4] as starting, high-level requirements for the frbr-redis datastore, we tested the suitability of using redis with over 850 unit tests representing marc21, mods, and vra metadata structured as frbr work-expression-manifestation-item entities with a variety of redis data primitives. one of the trends highlighted at the 2012 code4lib conference was the growing importance of mobile devices, both in usage by library patrons, and in the limitations and opportunities presented by developing applications. this caused us to rethink how we offered access and management to bibliographic information from a desktop/rich web “discover” model to a simplified mobile app user interface used in mobile and tablet native applications such as apple’s ios and google’s android operating system. we started experimenting with twitter’s bootstrap support for responsive web design to build simplified html5 apps that were much closer to the design aesthetic of a mobile and tablet application. the first app we released was a call number app that provides an embedded widget for our discovery layer as well as a standalone app targeted for mobile and tablet devices but fully usable by modern desktop web browsers. this app was the beginning of the aristotle library apps project [5] and can viewed online at http://discovery.coloradocollege.edu/apps/call_number/. since 2011, we continued to monitor the modeling work done by the bibliographic framework transition initiative. as we became more experienced with redis, the initial implementation of a redis frbr datastore, with a single redis instance using multiple databases, wasn’t as flexible as having each each of the first group of frbr entities (work, expression, manifestation, and items) run as separate redis instances on different ports on the same server. in september 2012, sally mccallum, in an igelu presentation [6], offered the first glimpse of a new bibliographic model she referred to as marcr, for marc resource. this new model supports rda and frbr but the core entities have been whittled down to four: work, instance, annotation, and authority. because of redis’s flexibility, we were quickly able to modify our existing frbr redis datastore key structure to follow this new model using rda. a primer on this new model was published in november of 2012 [7], with the name changed to bibframe and minor changes made to mccallum’s introduction from september. namely, the work entity became creative work and the namespace for the model was formally set as bibframe. the bibframe redis datastore is now a separate open source project licensed under the gnu general public license 2 and includes documentation, redis server configuration files for each of the bibframe entities, and lua server-sides scripts [8]. more about redis redis, a key-value datastore, is one of the many new nosql data technologies that offer alternative models for data representation and use. redis supports data persistence in two ways: an rdb mode that saves the dataset at periodic intervals, and an aof mode that saves the dataset with every write operation. while there are advantages and disadvantages to each approach [9], most libraries could employ a combination of rdb mode for bibliographic records and aof mode for library transactional data like circulation statistics. redis is fundamentally different from the flat-file structure of a marc record and the relational databases of more traditional library systems. the flexibility of redis allows for the rapid development of multiple apps by supporting different information schemes and structures within a single datastore. the manner in which a redis key is constructed allows for the embedding of semantic or heuristic information about the data structure in its naming structure. redis assumes that related data use a key naming pattern, and even provides a global function to increment the key id. another important design consideration when using redis is the type of data primitive to associate with a key. the simplest value is an atomic string. the redis list is a collection primitive which stores unordered and duplicate string values. the redis set stores unique string values, while the sorted set assigns a sort weight to each value. if a weight of 0 is used in a sorted set, redis does a lexical sort based on the string values in the set. the last redis data primitive is a hash. a redis hash associates multiple sub-keys with a single redis key and with the hget command returns the value associated with that sub-key. redis is not a relational database and it would be suboptimal to attempt to replicate an rdbms. in redis, the key is the fundamental structure, not a table-row as in an rdbms. redis keys can also serve as a string value for other keys in the datastore, providing a sort of crude sql join, but offering more flexibility in representing relationships between keys in a manner that would be difficult or impossible to replicate in an rdbms. the downside is that referential integrity between different tables is not built into redis. eventual consistency can be achieved either through application logic or through strategies involving a combination of redis server commands. the redis string, set, sorted set, list and hash data primitives all offer different ways to represent library information in the redis server. redis also provides a number of server and primitive-specific commands that ease application development, including exist and type. for the exist command, a string is passed in as a parameter and a boolean is returned confirming whether the string is a key in the datastore. the type command, when passed in a key string, returns the type of redis data structure that is represented by the key or a null value if it doesn’t exist. for large datasets that may not fit into ram, the lead developer on the redis project, salvatore sanfilippo, recommends presharding [10] to break up the datastore among different redis instances. the naming schema used by the aristotle library apps and bibframe datastore project can easily support presharding for use in large collections, though for the time being, a single virtual machine with 4g of ram is more than sufficient to run all of colorado college’s bibframe datastore redis instances. the following graphic shows our colorado college bibframe datastore set-up. figure 1. colorado college’s bibframe datastore setup the evolution of a native bibliographic redis datastore the first iteration of a redis bibliographic datastore used a naming schema based on frbr and rda, using a notation similar to xml namespaces to create a collection of keys to represent values and relationships with other entities in the datastore. for example, in the aristotle library apps project, rda core frbr entities and attributes are extracted from marc records following the marc-to-rda mappings provided in the ala’s rda toolkit [11] that are then organized into redis key collections following a pattern that maps to bibframe linkable information resources like creative work, instance, authority, and annotation. the figure below shows one such bibframe redis key collection for a creativework’s rda:title with supporting redis keys for that entity. the phonetic, rda:varianttitleforthework:sort subkeys and subvalues in the bibframe:creativework’s rda:title support title searches by various apps and colorado college’s discovery layer. figure 2. example of bibliographic redis keys and data primitive values for a bibframe creative work’s title the bibframe datastore can support other metadata formats and values as well. continuing the previous example, if we wanted to use mods or dublin core instead of rda, the bibframe datastore key could look like bibframe:creativework:10089:mods:titleinfo with each of the mods titleinfo sub-elements as hash key-value pairs for an individual creativework. likewise, a redis key like bibframe:creativework:10089:dc:title could be either a simple static string or a hash like the current rda:title key used in the current iteration of the bibframe datastore project. we could even have all three (or as many different keys as desired) exist in the same redis datastore instance. the design of the redis key’s structure is through convention and convenience with an effort to be explicitly descriptive of the data that is returned by redis without being too verbose. meanwhile, financial information, such as material orders and invoice information stored in our ils, is extracted and added to the redis datastore for reporting and budget forecasts. we even store the library hours as a redis string for use in a standalone app and as a json data feed for our discovery layer. redis datastore interactions are abstracted via python classes, which are built with the redis-py module [6]. if the app developer needs custom data storage or extended redis functionality, python custom classes can extend existing classes through direct manipulation of the datastore. html5, responsive web design, and twitter bootstrap while native apps generally run faster and more closely follow the recommended user interface guidelines for their respective platforms, the tutt library does not have the resources to maintain multiple app development environments. thankfully, with evolving web standards and protocols, coupled with the development of css and javascript libraries, fast and easy-to-use html5 apps can be created that are more universal and can run on multiple mobile and tablet platforms as well as on personal computers running more modern web browsers that support html5. this is the goal of responsive web design, as expressed in the original article on a list apart: rather than tailoring disconnected designs to each of an ever-increasing number of web devices, we can treat them as facets of the same experience. we can design for an optimal viewing experience, but embed standards-based technologies into our designs to make them not only more flexible, but more adaptive to the media that renders them. in short, we need to practice responsive web design. [7] the aristotle library app project uses the popular web framework bootstrap as the basis for the user interfaces that respond and adjust for different client devices and displays. it is prohibitively expensive and impossible for a small library with limited staff and resources to test out apps on all of the different platforms, web browsers, and devices used by our users. by focusing on the most popular and available devices in the library (windows 7, macintosh, ios, and some android phones and tablets), the tutt library targets specific functionality needed by its patrons and staff. the design intention of this html5-based app development environment is that creating a new app should be roughly equivalent in difficulty to building a simple website leveraging librarian and staff’s pre-existing competencies with such tools as dreamweaver and content management systems. while there is training involved in educating staff about bootstrap and html5, the training burden and requirements for app development is considerably less than if the library tried to develop native apps for the ios and android environments. access and discovery apps the majority of apps in the initial aristotle library app project are categorized as access and discovery apps, which allow users to find and access the library resources and more general information about the library. access and discovery apps broadly address the generic tasks by users to find, identify, select, and obtain resources as expressed in the frbr specification [14]. also included in this category, the tutt library’s hours app informs a patron if the library is open or closed. while not bibliographic in nature, the hours app addresses one of the top questions the tutt library receives from patrons. the call number app figure 3. screenshot of the call number app the call number app was the first app released, and served as the catalyst for the entire aristotle project. as we worked on the discovery layer, another librarian was inspired by a feature in stanford university’s searchworks (built with blacklight and hydra) that allowed a patron to see which call numbers were near each other in the library’s stacks. while investigating stanford’s implementation based on solr, we realized a simplified data model could be used with redis. to create the type of sorted indexes needed for this app, normalized library of congress, sudoc, and local call numbers were added as weights to redis sorted sets. once we had embedded the call number app into the discovery layer, we explored the further development of dedicated, simplified apps for common searches. these independent apps could be used in larger systems, like the discovery layer or the library’s website, through the use of json apis and raw html. library hours app figure 4. screenshot of library hours app when the college adopted a cms incapable of building a dynamic feed of the library’s hours of operations for the library’s homepage, we felt that a dedicated app with a json feed and hours data stored in redis would work instead. the library hours app stores dates and hours in a string using redis bit operations commands. each day has a unique key with the following pattern: “library-hours:yyyy-mm-dd”, with the value being a 96-bit string with each bit representing a quarter hour with bit offset 1 representing 00:00 to 00:14, bit offset 2, 00:15-00:29, etc. for each quarter hour of a twenty-four hour day. bits set to zero (the default) means the library is closed, the bit set to 1 means the library is open for that quarter hour. to see if the library is open, the hours app queries the library’s operational redis datastore by using a redis getbit command as demonstrated in this code snippet from the hours app’s redis_helpers.py [15]: def is_library_open(question_date=datetime.datetime.today(), redis_ds=redis_ds): """ function checks datastore for open and closing times, returns true if library is open. <br/> :param question_date: datetime object to check datastore, default is the current datetime stamp. :param redis_ds: redis datastore, defaults to module redis_ds :rtype: boolean true or false """ offset = calculate_offset(question_date) status_key = question_date.strftime(library_key_format) return bool(int(redis_ds.getbit(status_key,offset))) the patron user interface for the hours app displays a simple message with the library’s current hours. if the library is closed, the app displays the next available date and time when the library is open. with 365 keys per year, one for each day, this redis structure easily supports the requirements of the hours app administrative user interface, allowing authenticated library staff to add or modify the posted hours. productivity apps the second category of apps is for productivity, developed to either manage or report on resources in the collections. these apps require the user to first authenticate, then, depending on the app and the user’s authorizations, allow for the manipulation or reporting of library information, which includes the native bibframe entities in the datastore. in the orders app, order records were imported from tutt library’s legacy ils into the frbr redis datastore. by doing so, we freed this information from the proprietary ils vendor that tightly binds order information to the marc bibliographic record (even going so far as to create custom 9xx fields for order information). by separating the order information into redis sets with each invoice and order as distinct redis keys, visualizations and budget reporting became much simpler. before we had this tool we would have to export this data from the ils and extract the data from the marc21 record, then clean it up before importing it into microsoft excel for analysis of this critical aspect of library operational information. we have also developed a productivity app for a collection of fedora commons utilities used by library staff to move objects around in the digital repository, ingest batches of objects, and apply a metadata batch update to one or more fedora objects. while this app does not directly use the bibframe entities, it has streamlined the workflow of staff in the library who work with the digital repository. the aristotle library apps project is flexible enough to accommodate workflows with other library systems, like our fedora commons digital repository. roadmap for the aristotle library app project the tutt library uses its call number, hours, and discovery apps to augment the library’s website and discovery layer. these are publicly available, along with the article and book search apps, at http://discovery.coloradocollege.edu/apps/. the same json interface that the call number app uses to populate a shelf-browser is also used in the record view in the discovery layer. the hours app provides an embedded html snippet for inclusion in various locations in the library’s website. the next wave of app development will focus both on improving local workflows for such library services as material check-out, course reserves, and collection inventories, as well as improved discoverability of resources from other institutions through shared bibframe redis datastores. in keeping with an agile software development philosophy, each app should be simple enough to design, implement, and start testing within a three-to-four week sprint, which nicely coincides with the current academic block calendar at colorado college. peer-to-peer and consortium union catalogs concern over interoperability was brought up as a challenge to any radical technology change as the tutt library moved to an app model. the regional colorado alliance of research library’s prospector union catalog, in which colorado college is both an active lender and borrower, supports the college’s block plan, where students take intense, 3 1/2 week courses for college credit. with students needing research material promptly, tutt library strives to deliver to students, faculty, and staff as promptly as possible, preferably under 72 hours. the prospector-based ill service is critical to meet this tight deadline for materials. any replacement or legacy ils cannot diminish that service. our plan to tackle these challenges involves an incremental approach, starting off with bibframe datastores shared by just two institutions before increasing the scale to a consortium or regional scale. figure 5. multiple institutions peer-to-peer bibframe datastores we are currently in the early stages of prototyping a peer-to-peer bibframe datastore with the university of denver’s penrose library. as the above diagram illustrates, in this early exploratory work colorado college and the university of denver each have their own redis master running in their respective, local virtual machine environments. each institution also has local redis slaves of the other’s creative work, authority, and annotations that sync up with their corresponding master. our intention is to test the usability of a shared bibframe redis-based union catalog with two institutions to prepare for scaling up to a consortium-level service. figure 6. design of a consortium bibframe datastores maintaining record-level interoperability should be relatively easy in the library app portfolio as long as the marc utilities and productivity apps are in active development. the challenge is in integrating material requests and real-time circulation status into the proprietary system the alliance currently uses for the prospector. a strength of redis is its ability to store and serve large volumes of data, as it has done for sites such as github, engine yard, craigslist, disqus, and stack overflow [16]. the library is in early discussions with the alliance about expanding the aristotle library apps project to scale for millions of records. some interesting network topologies for bibliographic information may be possible when using redis as the underlying datastore and frbr/rda as the organizing principle. for example, the alliance may host the shared work and expression datastores, then subscribe to manifestation and item datastores managed and hosted locally at each institution, at the alliance, or at a commercial cloud provider. each institution could also use the alliance’s hosted work and expression datastores in their own access and discovery apps. to support the increased scale of a consortium or regional bibframe redis ecosystem, we are closely following redis cluster, a sub-project of the redis open source project. redis cluster offers a distributed and fault tolerant [17] environment for very large and distributed data nodes that matches well with our peer-to-peer and consortium-level bibframe datastores. redis custer is under active development, with the developers planning a stable beta release in 2013. the library and the alliance are also exploring grant opportunities to fund the development and support for a new bibliographic datastore that will scale to hundreds of millions of frbr entities. notes [1] pymarc. available from: https://github.com/edsu/pymarc [2] aristotle discovery layer project. available from: https://github.com/jermnelson/discover-aristotle [3] nelson j. nosql bibliographic records: implementing a native frbr datastore with redis. code4lib 2012, seattle, washington. available from: http://discovery.coloradocollege.edu/code4lib [4] a bibliographic framework for the digital age. october 31, 2011. available from: http://www.loc.gov/marc/transition/news/framework-103111.html [5] aristotle library apps project. available from: https://github.com/jermnelson/aristotle-library-apps [6] mccallum, s. bibliographic framework initiative approach for marc data as linked data, september 13, 2012. igelu conference presentation. powerpoint available at: http://igelu.org/wp-content/uploads/2012/09/igelu-sally-mccallum.pptx [7] miller, e., ogbuji, u. and k. macdougall bibliographic framework as a web of data: linked data model and supporting services. november 11, 2012. available from: http://www.loc.gov/marc/transition/pdf/marcld-report-11-21-2012.pdf [8] bibframe-datastore project. available at: https://github.com/jermnelson/bibframe-datastore [9] redis persistence. available from: http://redis.io/topics/persistence [10] redis presharding. available from http://antirez.com/post/redis-presharding.html [11] ala’s rda toolkit mappings. available with subscription at: http://access.rdatoolkit.org/document.php?id=jscmap1] [12] redis-py. available from: https://github.com/andymccurdy/redis-py/ [13] marcotte e. responsive web design. a list apart. may 25, 2010. available from: http://www.alistapart.com/articles/responsive-web-design/ [14] functional requirements for bibliographic records. international federation of library associations and institutions. december 26, 2007. available from: http://archive.ifla.org/vii/s13/frbr/frbr_current2.htm [15] from the aristotle library apps’s project https://github.com/jermnelson/aristotle-library-apps/blob/master/hours/redis_helpers.py [16] who’s using redis? available from: http://redis.io/topics/whos-using-redis [17] redis cluster specification (work in progress). available from http://redis.io/topics/clster-spec about the author jeremy nelson (jeremy.nelson@coloradocollege.edu) is the metadata/systems librarian at colorado college. he is responsible for ensuring that the tutt library technology that students, staff, and faculty at colorado college depend on is available when they need it both on and off campus. he is also responsible for the cataloging department, ensuring that electronic and physical material acquired by the library is cataloged correctly and is positioned for future further use. subscribe to comments: for this article | for all articles 2 responses to "building a library app portfolio with redis and django" please leave a response below: new issue alert: the code4lib journal (issue 19) is now online | lj infodocket, 2013-01-15 […] building a library app portfolio with redis and django […] jurnal media syariah – building a library app portfolio with redis and django, 2013-03-09 […] by scripting the manipulation of these marc records with python and the pymarc python module [1], and with a web front-end built in django, we brought considerable time savings to our cataloging […] leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – extracting, augmenting, and updating metadata in fedora 3 and 4 using a local openrefine reconciliation service mission editorial committee process and structure code4lib issue 31, 2016-01-28 extracting, augmenting, and updating metadata in fedora 3 and 4 using a local openrefine reconciliation service when developing local collections, librarians and archivists often create detailed metadata which then gets stored in collection-specific silos. at times, the metadata could be used to augment other collections but the software does not provide native support for object relationship update and augmentation. this article describes a project updating author metadata in one collection using a local reconciliation service generated from another collection’s authority records. because the goddard library is on the cusp of a migration from fedora 3 to fedora 4, this article addresses the challenges in updating fedora 3 and ways fedora 4’s architecture will allow for easier updates. by ruth kitchin tillman introduction this article documents a project undertaken to augment metadata within the nasa goddard library repository.[1] the repository runs on a custom fedora 3.3/drupal 7 setup. it contains five collections spotlighting aspects of work produced by the goddard space flight center—authors & publications, case studies, colloquia, balloon technology documents, and the goddard news. the fedora objects for each collection consist of metadata recorded in an appropriate language indexed in fedora’s xml-based gsearch; a simple dublin core file indexed by both gsearch and fedora’s risearch [2]; a rels-ext file (fedora’s rdf/xml language, used to create links between fedora objects) indexed in risearch; and pdfs or externally-hosted video files, depending on the collection. fedora versions 1 to 3 use permanent identifiers (pids) to identify objects. pids consist of a colon-separated prefix and suffix. the prefix may be used across a repository or vary based on collection within the repository. when combined with the suffix, it forms a unique object identifier. this article focuses on two of the collections, authors & publications and colloquia. the first documents the publication output of authors affiliated with nasa goddard. it consists of two types of fedora object, author objects and publication objects. author objects include local mads [3] authority records for each goddard author. publication objects use niso jats [4] to record article, book chapter, technical report, or book metadata. the second collection, colloquia, documents the colloquia presented at nasa goddard by scientists from around the world as well as local goddard speakers. its metadata is recorded in the goddard encoding metadata schema (gems), an extended version of dublin core. the goddard repository uses different pid prefixes for each type of object, which will be referenced in code samples as authorpid, publicationpid, and colloquiapid. historically, the two collections have remained separate. because authors & publications collects publications by authors who have an affiliation at the time of publication, each publication must connect to at least one author object. this relationship is a core component of the collection. the collection is in the process of being back-dated to 2000, but will not extend earlier because of the challenges in gathering reliable author metadata from over 15 years ago. in the colloquia collection, on the other hand, only 710 of the over-5000 records (around 14%) have a goddard presenter. moreover, the collection includes metadata records for library-held colloquia recordings as far back as the 1970s, although digitized copies of older recordings are not yet available on the website. because of the timespan of the collection and the high percentage of non-goddard presenters, connecting presenters in the colloquia collection to the author records from the authors & publications collection was not considered suitable at the time. instead, the colloquia gems records include a speaker’s name and institutional affiliation. if the author is from goddard, their organizational code is recorded as well. these fields may be used in searching the colloquia or in cross-site search, but there is no formal relation to an author’s fedora object record or publications. author names are taken from colloquia materials and not transformed into controlled forms (i.e. “dave” vs. “david” or “chip” vs. “charles”). when considering augmentations to the repository in preparation for its eventual upgrade to fedora 4, i began exploring the possibility of reconciling these two collections for a more unified repository experience. this article outlines my step-by-step process, with scripts and screenshots, for extracting metadata from the authors & publications collection, transforming it into an rdf reconciliation service for openrefine, extracting appropriate colloquia metadata from fedora, using that openrefine service to augment the colloquia metadata, and re-ingesting the augmented data back into fedora. because of the project’s timing, i explored methods for re-ingesting the augmented metadata in both fedora 3 and fedora 4 for comparative purposes. project goal the author and publication records are connected using fedora 3’s built-in rdf/xml language, rels-ext. a publication would include the statement <rel:ispartof rdf:resource="authorpid"/> for each goddard author. in the fedora 4 model for this collection, the relationship will be documented in the publication object itself (vs. an attached file) with the rdf property dc:creator linking to the author’s fedora object. at the end of this reconciliation project, i needed to have related colloquiapid and authorpids in a format which could be used to update colloquia records with either an appropriate rels-ext statement or a dc:creator statement. i also needed to solve any complications in re-ingesting the augmented metadata. phase 1. creating a local reconciliation service for the metadata reconciliation process, i chose to work in openrefine[5][6], a popular data manipulation and transformation tool. i added the rdf refine plugin from deri[7] (these services have also been combined to form lodrefine, but as i worked from openrefine with the plugin installation, that’s how i’ll discuss it), which allows one to both export openrefine projects as linked data and to use linked data services alongside other reconciliation services. unlike cases where one might be drawing from established name authority services, i had to create my own service based on the goddard authors and their uris (pids). one can do this with the rdf refine plugin, adding a reconciliation service from a file, whether locally-generated or downloaded from a major authorities site. i just needed to generate my own rdf file with author information. while many rdf vocabularies could have been used to describe authors, i chose the friend of a friend[8], or “foaf” schema because it’s one of the defaults in rdf refine and all i needed was something to handle names. my choice of vocabulary would affect the process but not the end result. first, i needed the names and object pids for all goddard authors. i exported these through the risearch interface by running a query against the brief dublin core files for author records. the resulting data set was a csv. a sample excerpt: "atlas, david",authorpid:823462024 "tilton, james c.",authorpid:411735411 "esaias, wayne e.",authorpid:868841903 "lyon, richard g.",authorpid:982601626 "schnase, john l.",authorpid:580647414 "esper, jaime",authorpid:564921526 i considered importing the csv into openrefine and using the rdf refine plugin’s option to export data as an rdf skeleton[9], but soon realized i could create an rdf file more quickly using regular expressions. additionally, i wanted to offer alternative names depending on whether or not the author used a middle initial, since colloquia metadata was never standardized to use the official form of the author’s name. manipulating names could be done in open refine, but could also easily be done with regular expressions as a part of creating the rdf file. in sublime text 2[10], the program i used for all non-openrefine data manipulation during this process, i searched for the following regular expression: ("(.+?),(\sdr.\s|\s{2}|\s))(.+?)(\s(\w\.)"|"|"),authorpid:(\d{9}) and replaced it with: <http://gsfcir.gsfc.nasa.gov/authors/id/\7> a foaf:person;\n foaf:name "\2, \4", "\2, \4 \6".\n\n the initial search captures a surname, matches a comma, captures any initial “dr.,” captures the first name that follows and any middle initial if it exists, and captures the unique suffix of the pid. the second query returns a url to the author’s page (urls derive from pid suffixes[11]), an rdf type statement that the entity belongs to the class foaf:person, a line break, two forms of the author’s name, and two more line breaks to keep objects neatly separated. i then ran a second find for foaf:name "(.+?)", "\1 ". and replacement foaf:name "\1". to account for cases in which the author’s middle initial wasn’t recorded, leading to duplicate names. a snippet from the result: <http://gsfcir.gsfc.nasa.gov/authors/id/823462024> a foaf:person; foaf:name "atlas, david". <http://gsfcir.gsfc.nasa.gov/authors/id/411735411> a foaf:person; foaf:name "tilton, james", "tilton, james c.". <http://gsfcir.gsfc.nasa.gov/authors/id/868841903> a foaf:person; foaf:name "esaias, wayne", "esaias, wayne e.". i then added the appropriate prefix declarations: @prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> . @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> . @prefix foaf: <http://xmlns.com/foaf/0.1/> . to the top of the file and saved it as an rdf/turtle .ttl file. the author urls were used instead of pids because, when reconciling, one may click on the link for a suggested match and preview the page. this functionality proved useful in cases where author names closely matched speaker names but there was some uncertainty. one could infer, for example, that a speaker primarily published on flight dynamics and trajectories would likely be giving a presentation about the interplanetary transport network. to add the file-based reconciliation service, i selected rdf in the upper right hand corner of openrefine, chose to add a reconciliation service, and selected “based on an rdf file.” figure 1. adding a file-based reconciliation service the process i used to upload the file and add the service is quite simple, as shown below. note that rdf refine includes built-in support for rdfs:label, skos:preflabel, dcterms:title, dc:title, and foaf:name. choosing the “other” box allows one to add other vocabularies if desired. figure 2. defining a file-based reconciliation service (enlarge) the service is now installed and ready to reconcile against any data which includes a column of goddard author names. phase 2. extracting metadata to reconcile after the reconciliation service had been created and installed, the next step in the process was to extract appropriate metadata from the colloquia collection. first, i had to determine what metadata would be needed to complete the reconciliation process. i settled on the following fields: colloquiapid, speaker name, date, title the only two fields actually needed to update records would be the colloquiapid and the speaker’s authorpid, which i would be extracting based on the speaker’s name during the reconciliation process. however, the date and title seemed useful for settling edge cases without having to look up the individual colloquia. a colloquia’s pid gives no indication of the actual topic and no date context. as mentioned above, the author reconciliation service would link to a page with the author’s primary topical field. on a low-confidence match that seemed entirely outside the author’s field, it would be safer to discard the data. similarly, if the colloquia were delivered in the 1980s and had a low-confidence match to an author who had only joined nasa in the 2000s, that possible match should be discarded. to get this data, i would need to process information based on each speaker in a colloquia, not each colloquia record. i began by extracting colloquia records from fedora using its gsearch output. since the gems field is indexed in gsearch, i ran a search to return raw xml output for each object in which affiliation contained the words “goddard” or “gsfc.” because i was searching contents and not exact matches, this result included affiliations described as “goddard space flight center,” “nasa goddard,” etc. i set the search to return 100 items per page, manually offsetting the page start until i reached the end of the search. i saved each page individually as an xml file then copied the object-level results into a single xml file with gsearch’s headers and footers. a single-object example with two authors can be seen below. <?xml version="1.0" encoding="utf-8"?> <resultpage xmlns:zs="http://www.loc.gov/zing/srw/" xmlns:foxml="info:fedora/fedora-system:def/foxml#" xmlns:dc="http://purl.org/dc/elements/1.1/"> <gfindobjects hitpagelast="6" hitpageend="200" hitpagestart="101" hitpagesize="100" hittotal="591"> <objects> <object score="73.88713" no="11"> <field name="pid">colloquiapid:3827</field> <field name="abstract">the solar dynamics observatory (sdo) sdo launched on february 11, 2010, 10:23 am est on an atlas v from slc 41 at cape canaveral. sdo's main goal is to understand the solar variations that influence life on earth and humanity's technological systems. these variations are caused by the solar magnetic field. the sdo science investigations will determine how the sun's magnetic field is generated and structured, how this stored magnetic energy is released into the heliosphere as the solar wind, ... </field> <field name="accessrights">public</field> <field name="affiliation">nasa/|!gsfc!|</field> <field name="affiliation">nasa/|!gsfc!|</field> <field name="browse">s</field> <field name="callnumber">2011-0404 (eng)</field> <field name="code">671.0</field> <field name="code">671.0</field> <field name="date">2011</field> <field name="dc.creator">pesnell, w. dean</field> <field name="dc.creator">chamberlin, phillip</field> <field name="dc.date">2011-04-04</field> <field name="dc.format">dvd</field> <field name="dc.format">webcast</field> <field name="dc.identifier">2011-0404 (eng)</field> <field name="dc.identifier">colloquiapid:3827</field> <field name="dc.title">the solar dynamics observatory: your eye on the sun</field> <field name="dc.type">video</field> <field name="fgs.createddate">2011-05-13t12:14:09.102z</field> <field name="fgs.label">colloquiapid:3827</field> <field name="fgs.lastmodifieddate">2012-11-01t14:47:05.745z</field> <field name="fgs.ownerid">fedoraadmin</field> <field name="fgs.state">active</field> <field name="ismemberofcollection">|!!|collection:3</field> <field name="mediachk">true</field> <field name="presentationdate">2011-04-04</field> <field name="resourcetype">video</field> <field name="series">engineering colloquium</field> </object> </objects> </gfindobjects> </resultpage> using the following xslt, i cross-walked the object results into a csv in which each row contained the fields listed above. the xslt matches an object’s dc:creator field, steps back one level in the xml tree to the object, and extracts the pid, dc:title, and dc:date to go along with that dc:creator. it applies to each dc:creator within an object, creating multiple rows as needed so each speaker will be in an individual row. <?xml version="1.0" encoding="utf-8"?> <xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/xsl/transform" xmlns="http://www.openarchives.org/oai/2.0/" xmlns:xs="http://www.w3.org/2001/xmlschema" xmlns:xsi="http://www.w3.org/2001/xmlschema-instance" xmlns:zs="http://www.loc.gov/zing/srw/" xmlns:foxml="info:fedora/fedora-system:def/foxml#" xmlns:dc="http://purl.org/dc/elements/1.1/" exclude-result-prefixes="xs foxml zs xsi" version="2.0"> <xsl:output method="text" indent="no" media-type="string" /> <xsl:strip-space elements="*"/> <xsl:template match="/resultpage/gfindobjects/objects"> <xsl:apply-templates select="object"/> </xsl:template> <xsl:template match="object"> <xsl:apply-templates select="field[@name='dc.creator']"/> </xsl:template> <xsl:template match="field[@name='dc.creator']"> <xsl:text>"</xsl:text><xsl:value-of select="../field[@name='pid']"/><xsl:text>","</xsl:text><xsl:value-of select="."/><xsl:text>","</xsl:text><xsl:value-of select="../field[@name='dc.date']"/><xsl:text>","</xsl:text><xsl:value-of select="../field[@name='dc.title']"/><xsl:text>"</xsl:text><xsl:call-template name="newline"/> </xsl:template> <xsl:template match="field"> <xsl:apply-templates/> </xsl:template> <xsl:template match="text()" /> <xsl:template name="newline"> <xsl:text>&#xa;</xsl:text> </xsl:template> </xsl:stylesheet> a line from the resulting csv: "colloquiapid:5088","mcelwain, michael","10/8/2014","see the ball, be the ball: how to find and characterize planets beyond our solar system" i then ingested the resulting csv into openrefine. phase 3. reconciling metadata in openrefine once i’d created and installed the local reconciliation service, derived the csv of pertinent data, and ingested it into openrefine, i could begin the real work of reconciliation. starting a reconciliation process in openrefine is fairly straightforward. from the column i wished to reconcile, author, i selected the drop-down “reconcile” and “start reconciling.” this allowed me to choose from any services i’d already installed, but did not give me a chance to install a new service. figure 3. selecting reconciliation from the column’s drop-down menu once i selected the service, openrefine spent several minutes parsing the service for the different types of data it contains so that it can offer options for reconciliation. from doing this several times, i discovered that, even if a file-based service has been installed and used already, it will need to re-parse each time. figure 4. openrefine must re-parse the reconciliation service once openrefine has figured out what options are available, it will offer to match against the different rdf classes found in the service. if i had made a file with both foaf:person and foaf:organization, for example, i could have chosen to reconcile only against foaf:person for this project. or i could have opted to “reconcile against no particular type” if i wanted to match against both (this is especially useful when a reconciliation service combines data from multiple sources which use different vocabularies to describe the same type of thing, e.g. foaf:person and schema:person). at this point one also chooses whether or not to automatically match when a high-confidence match exists. i experimented both ways and found that automatic matches required such a high degree of confidence that i felt i could trust them for this project. it should be noted that the right-hand column with header “also use relevant details from other columns” is not supported by the rdf refine plugin and only works for other types of reconciliation services. my first version of the reconciliation service had included the organization code where a person worked in an attempt to improve matches, but without this working for rdf reconciliation, i was unable to add that level of confidence. figure 5. select class (or no particular type) against which to reconcile and whether to auto match (enlarge) the following is an example of matches after i ran basic reconciliation. the green line at the top shows percentage of matches found. each match immediately links the author’s name to their uri (in this case, their author page), see figure 7. clicking this link opened the author page as a pop-under within the service where i could confirm a match or evaluate several possible matches. if i wanted to see other possibilities for an automatic match or create a new topic, i could “choose a new match.” in cases where the service did not find an automatic match, it lists several possible options with the match confidence, the closest form of the name, and the same link to the uri. figure 6. an example of initial match results figure 7. clicking on an author link allows one to view the uri’s webpage and then opt whether to match this cell, all identical cells, or simply close the view (enlarge) initial reconciliation with auto-matching found 175 matches on the 710 names extracted from the collection. because many authors were not caught by auto-match (see figure 7 above for multiple authors who didn’t match as well as several for whom there were no matches), i performed individual review of the author column. in most cases where a match existed, it was easy for me to confirm through personal experience with the author’s work or by seeing it was simply a different form of the name, e.g. gail skofronick-jackson vs. gail skofronick jackson. for around a dozen names, i had to open the author’s page and sometimes even perform brief local research on their publications. in total, i found another 110 matches for a total of 285 author matches. as for the rest, some authors had simply left goddard, some had never published and were thus never in the authors & publications collection, some were mis-identified as affiliated with goddard at all (the oldest metadata came from transcribed textual records which were occasionally too brief to ensure accuracy and i recognized certain errors from my institutional knowledge alone), and some were non-persons such as the “office of human capital management” visible in figure 7. once the automatic and manual reconciliation was complete, i extracted the data from the names column into a new column. the reconciliation data is only available in openrefine as a part of the service and exporting the project into excel or another other format would not include the data from the reconciliation process. first, i chose to “edit column” and “add column based on this column…” figure 8. to extract data, start by adding a column based on the reconciled column i then used the google/general refine expression language (grel) expression cell.recon.match.id to select the uri/id of the rdf subject to which the person was matched. the preview pane gives examples of what would be extracted for each row. i chose to leave the cell in the new column blank “on error” (if no reconciliation data existed). figure 9. the object uri may be extracted with grel expression cell.recon.match.id (enlarge) figure 10. an example of extracted data (enlarge) after extracting the data into a new column, i exported the project into excel for simple manipulation and extraction. since the authorpid (vs. author page url) is necessary to create relationships in fedora 3, i replaced the first part of the author url with the prefix authorpid:. i then removed all columns but those containing colloquiapid and authorpid and sorted the sheet based on values for authorpid so rows without a pid could be grouped and deleted. this file then became my master copy of the relationship data from which i would create update solutions for fedora 3 and fedora 4. phase 4. updating fedora 3 and 4 at this point in the process, i had a simple two-column sheet outlining relationships between colloquia objects and author objects. what remained was to transform these into a format which could be used to update fedora. if a colloquia had multiple presenters who also had author objects, i would have to take that into account when performing updates. i also had to handle major structural differences between fedora 3 and fedora 4. the processes turned out to be significantly different, primarily due to fedora 3’s requirements for updates. updating fedora 3 fedora 3 supports an xml language, fedora batch modify,[12] which can be used to create or purge fedora objects or to modify datastreams (xml files, images, pdfs, etc.) attached to a fedora object. the difficulty? one must completely replace the datastream one is modifying. in this case, to add an author relationship to the rels-ext rdf/xml datastream for the colloquia object, one must recreate the entire datastream for that object. at first, this challenge seemed insurmountable. recreating the entire rels-ext added a degree of risk to the project that simply inserting an author relationship wouldn’t. but i decided to see just what recreating the information would entail. i began addressing the complication by asking three questions about the colloquia collection’s rels-ext files: what is the same across all rels-ext files? what is different? what does the new information look like? in the case of colloquia files, the answers were: same: each has the relationship <collection:3> indicating that they’re colloquia collections and <fedora-model:hasmodel> <cmodel:2>, <cmodel:4>., which, respectively, designate the set of expected fedora datastreams and indicate the object may have children. different: each colloquia is related to a particular subcollection using <rel:ismemberof> <subcollection:n>. the subcollection indicates which division (science, engineering, etc.) sponsored the colloquia, therefore a colloquia may belong to multiple subcollections if it had multiple sponsors. new: each colloquia being updated will have one or more relationships added to author records, the point of the project. to fit the fedora 3 models currently in use for publications in authors & publications, this relationship will be defined as <rel:ispartof> <authorpid:nnnnnnnnn>. based on this assessment, creating a replacement rels-ext would involve some standard relationships for every colloquia, the author reconciliation data, and the subcollection information for each colloquia that was being updated. it was at this point that openrefine became very useful again. first, i queried fedora’s resource index (again via risearch) to extract subcollection data for all colloquia. i considered attempting only to extract data for the colloquia i was going to be updating, but soon realized that my process would allow me to easily sort out colloquia which did not need updating. i combined the subcollection export with the author reconciliation data in the same sheet, ending with columns: colloquiapid subcollectionpid authorpid and reordered based on the colloquiapid column. the file included one line per relationship. a colloquia with one subcollection and three authors, for example, would have 4 separate lines. i imported this spreadsheet into openrefine where i began work on it. openrefine offers options for working on data on different lines that belongs to the same overall record. after import, an example colloquia looked like this: figure 11. an example of colloquia information before blanking down to have openrefine see all of these as one record, i had to remove the duplicate entries for the colloquiapid. one does this by selecting the “blank down” option under “edit cells” in the column’s drop-down menu. functionally, blank down removes all duplicate data in the column that is both subsequent and adjacent. imagined vertically, a a a will become a – – but a b a will remain the same. blank down works on any row, but for it to make openrefine view the objects as one record, the modified column must be the farthest left and it must be ordered so that all rows for a record are adjacent. figure 12. the blank down function this is the result, once blank down is applied: figure 13. the same information, after openrefine’s blank down has been applied note the numbering change on the left-most column when compared to figure 11 (openrefine’s row/record number). openrefine sees the group of rows as a single record and applies a single number. now i could use openrefine’s functions for processing multi-valued cells. in this case, i need to join multi-valued cells. (see figure 12 for where this can be found in the menu.) i selected a comma separator for the data which had previously been in separate cells. the operation applies to one column at a time, so for this project i needed to apply it to both subcollections and authors.   figure 14. results from joining multi-valued cells. (enlarge) at this point in the process, the data consisted of a single row for each colloquia with its pid in the first column, the pids of all subcollections in the second column, and the authorpids of all speakers in the third column. the data was sufficient to recreate a rels-ext but now had to be transformed into fedora batch modify statements. while i initially considered parsing it with python or even xslt, considering phase 1 of the process reminded me that it could be done much faster with a simple regular expression find and replace. first, i had to isolate the data that actually needed to be replaced. as mentioned earlier in this section, when extracting subcollection relationships, i had done a collection-wide extract. however, i would not need to update the thousands of records which would be unchanged. i exported the openrefine project as a tab-separated value (tsv) sheet, opened the sheet in excel, and used row sorting to group rows which had nothing in the authorpid column. i then deleted rows without authorpids and resaved the project before opening it in sublime text 2 for actual manipulation. in sublime text 2, i used a series of regex searches and substitutions to search for a variety of possibilities, taking into account that each pid followed a known pattern and that a brief test indicated that there were no more than 3 subcollections and 4 authors for any one record. what follows is a sample line from the sheet, the regular expression used to match it, and the entire datastream as a replacement value for that line with backreferences to pids captured in the regular expression search. colloquiapid:4873 subcollection:3,subcollection:5 authorpid:837483920,authorpid:892493054,authorpid:294850493 (colloquiapid:\d{1,4})\t(subcollection:\d{1,2}),(subcollection:\d{1,2})\t(authorpid:\d{9}),(authorpid:\d{9}),(authorpid:\d{9}) <fbm:modifydatastream pid="\1" dsid="rels-ext" dscontrolgrouptype="x" dslabel="colloquia_rels-ext_ds" logmessage="batchmodify modifydatastream"> <fbm:xmldata> <rdf:rdf xmlns:rel="info:fedora/fedora-system:def/relations-external#" xmlns:fedora-model="info:fedora/fedora-system:def/model#" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"> <rdf:description rdf:about="\1"> <rel:ismemberofcollection xmlns:rel="info:fedora/fedora-system:def/relations-external#" xmlns:rel="info:fedora/fedora-system:def/relations-external#" rdf:resource="info:fedora/collection:3"/> <rel:ismemberof xmlns:rel="info:fedora/fedora-system:def/relations-external#" rdf:resource="\2"/> <rel:ismemberof xmlns:rel="info:fedora/fedora-system:def/relations-external#" rdf:resource="\3"/> <fedora-model:hasmodel xmlns:fedora-model="info:fedora/fedora-system:def/model#" rdf:resource="info:fedora/cmodel:2"/> <fedora-model:hasmodel xmlns:fedora-model="info:fedora/fedora-system:def/model#" rdf:resource="info:fedora/cmodel:4"/> <rel:ispartof xmlns:rel="info:fedora/fedora-system:def/relations-external#" rdf:resource="\4"/> <rel:ispartof xmlns:rel="info:fedora/fedora-system:def/relations-external#" rdf:resource="\5"/> <rel:ispartof xmlns:rel="info:fedora/fedora-system:def/relations-external#" rdf:resource="\6"/> </rdf:description> </rdf:rdf> </fbm:xmldata> </fbm:modifydatastream> this may sound time-consuming, but it was simply a matter of creating a regular expression search and a template for a single colloquia, subcollection, and author, and adding/removing possibilities. once the template had been created, the find and replace process took no more than 5 minutes. to ensure none of the lines had been missed and the file created was properly valid, i added the header: <?xml version="1.0" encoding="utf-8"?> <fbm:batchmodify xmlns:fbm="http://www.fedora.info/definitions/" xmlns:xsi="http://www.w3.org/2001/xmlschema-instance" xsi:schemalocation="http://www.fedora.info/definitions/ http://www.fedora.info/definitions/1/0/api/batchmodify.xsd"> and closing </fbm:batchmodify> tag and validated the xml file. i then uploaded the modification file to one the of repository’s testing sites and used sparql and risearch queries to see if calls for all items which considered themselves ispartof authors whom i’d identified in the process would return colloquia as well as publications. the raw results were a complete success, so it was time to move on to experiments with fedora 4. updating fedora 4 after the challenge of updating fedora 3, fedora 4 turned out to be a much simpler situation. all rdf data in fedora 4 is handled directly on the object rather than in an attached bitstream (fedora 4’s term for what fedora 3 called a “datastream”). these can be sent as a curl patch call to the fedora server, using sparql-update to insert a single rdf relationship to the author’s object. not only did this negate the need to extract and group subcollection information with the author information, patch calls can be made over and over to the same object, so there was no need to group multiple colloquia speakers into a single patch. i started with a copy of the master file i’d created at the end of phase 3 of the process, which i saved as a tsv sheet and opened in sublime text 2. i decided to use a similar regular expression method to create the patch statements. the following are sample data, query, replacement, and result. data: colloquiapid:1375 authorpid:160862354 colloquiapid:4524 authorpid:160862325 query: colloquiapid:(\d{1,4})\tauthorpid:(\d{9}) replacement: curl -x patch -h "content-type: application/sparql-update" -h "cache-control: no-cache" -d 'prefix dc: <http://purl.org/dc/elements/1.1/> prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> delete { } insert { <> dc:creator <http://example.com:8080/fedora/rest/authors/\2>. } where { }' 'http://example.com:8080/fedora/rest/colloquia/\1'\n result: curl -x patch -h "content-type: application/sparql-update" -h "cache-control: no-cache" -d 'prefix dc: <http://purl.org/dc/elements/1.1/> prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> insert { <> dc:creator <http://example.com:8080/fedora/rest/author/160862354>. } where { }' 'http://example.com:8080/fedora/rest/colloquia/1375' curl -x patch -h "content-type: application/sparql-update" -h "cache-control: no-cache" -d 'prefix dc: <http://purl.org/dc/elements/1.1/> prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> delete { } insert { <> dc:creator <http://example.com:8080/fedora/rest/author/160862325>. } where { }' 'http://example.com:8080/fedora/rest/colloquia/4524' when saved as a curl file and run from the command line, this immediately inserted the correct relationships into our fedora 4 test instance (note, no passwords have been included in the patch statement, but if your fedora 4 repository requires passwords for curl calls, you would include them as well). similar tests against the triplestore revealed that dc:creator searches would now retrieve colloquia uris as well as publication uris. while this method seems simple, some idiosyncrasies recognized during the test process require a brief explanation of why it might require a few more steps to do to a live fedora 4 repository. the testing version of the repository i updated uses pid prefixes and suffixes to create object uuids, splitting authorpid: 948347594 into /author/948347594/, etc., rather than using fedora 4 default uuids, which are generated using a non-semantic pairtree process[13]. however, because of some of the advantages of complex, pairtree-based uuids and the need to create new objects in these collections, the final fedora migration process used on the goddard repository may abandon pids for fedora 4-generated uuids. in that case, a second step of reconciliation would need to be done before these curl patches could be generated. pids will migrate into fedora 4 as triples using the predicate fedora3model:pid. as i envision it, one would use a triplestore to extract uris with full uuids and pids for each set of pids. one could then store these in separate sheets in the same excel workbook and use excel’s match and index functions to recreate the pairings of colloquiapid and authorpid in a third and fourth column of colloquiauuid and authoruuid. a similar find/replace search based on matching and capturing each uuid would allow one to generate curls in the same fashion as above. alternatively, one might simply wait until the migration was complete and recreate the project from phase 1 onward, using slightly different methods to extract data in phase 2 since gsearch will be deprecated, so one is starting with the new uuids and not relying on pids. pids were used in this case because no permanent fedora 4 version of this collection exists. comparing methods for updating fedora 3 and fedora 4 while this paper is meant to be a case study of the overall process of using one’s own data to create a reconciliation service and enhancing one’s own collections, the process of updating fedora and the major differences between fedora 3 and 4 prompted comparative reflection. when updating relationships between objects, fedora 3’s methods are clearly more onerous and heighten the risk of data being lost through negligence or accident. if this project had also required updating the author objects from authors & publications, it would have required extraction of every single relationship the repository records between two authors (hascoauthor) and every relationship an author has to their publications (haspart). for prolific authors, this would have involved hundreds of relationships and while one could have followed similar methods as for subcollection, the risk of data loss would have caused greater anxiety. in future, when working with any data that gets handled through rels-ext or could be handled as rdf in fedora 4 (especially dublincore), i would err on the side of waiting until the data was in fedora 4 to perform the updates. conclusion the process of reconciling and updating relationships between objects in a repository doesn’t end with performing the updates. while both the fedora batch modify file and the curl file stand ready to be deployed to the repository’s staging server, neither has been implemented at this point. the project was primarily an exploration of “could we do this?” and “how?” but more questions need to be answered before it can be deployed on the repository website. first, how should the display be changed to reflect an author’s colloquia? while updating the colloquia module to include a link to the speakers author page instead of a textual search for their name would be simple, the question of the author page proved worthy of deeper consideration. as author pages currently only record publications, they may be regarded as incomplete curriculum vitae (incomplete because the repository’s mission statement does not include publications prior to or after their period of affiliation with goddard). should colloquia simply be intermixed with publications? should they be two sections of the page or have tabs that allow users to switch between the two? these questions and more need to be asked before the relationships can be used and this part of the process will likely be folded into the overall redesign planned for the site’s migration to fedora 4. second, what should be done about colloquia presenters who a) still work at goddard and b) do not have author records? is it inconsistent to only offer links to author pages if the speaker is also a published author? or is it overkill to create author pages for those who don’t publish? creating author records and pages for those who don’t publish requires us to reimagine “author” pages. it would also take significant time, at least in the beginning, to research the 400+ cases of a colloquia indicating a goddard affiliation and whether or not that speaker was still at goddard (if they have left, we cannot create a retrospective record as some of the data necessary to do so gets removed from our internal source) and then to create the record. but once it was done, the ongoing process would not be too time-consuming to maintain. at this point, we’re faced with a lot of possibilities. this project and process will allow us to provide integrated access to resources in our repository. what remains is to decide how best to implement the other changes and to remain aware of possibilities for future enhancements. acknowledgements this work was done under nasa/gsfc contract nng13az16c. the author wishes to thank christina harlow, kate deibel, and benjamin armintor for resources or conversations which helped her solve several technical problems during the process. about the author ruth kitchin tillman worked as the goddard library’s metadata librarian through january of 2016. she starts in february 2016 as the digital collections librarian for notre dame’s hesburgh libraries. endnotes [1] the nasa goddard library repository. http://gsfcir.gsfc.nasa.gov [2] fedora 3’s resource index search (risearch) allows access to the resource index which indexes relationships between fedora objects using and basic metadata about the objects (dublincore, dates created/modified, etc.). this search merely exposes data from the index and cannot be used to perform updates to the objects or index. for further details, see: https://wiki.duraspace.org/display/fedora34/resource+index+search [3] metadata authority description schema. used to encode authority records. the goddard library uses a mildly adapted version which takes into account additional aspects of employment at goddard. http://www.loc.gov/standards/mads/ [4] niso journal authoring tag suite. http://dtd.nlm.nih.gov/publishing/ [5] http://openrefine.org/ [6] the first of three useful tutorials explaining openrefine. https://www.youtube.com/watch?v=b70j_h_zawm [7] an rdf plugin for openrefine. http://refine.deri.ie/ [8] friend of a friend rdf vocabulary specification. http://www.foaf-project.org/ [9] this method of importing data into openrefine and extracting as an rdf skeleton may be used as an alternative, particularly to handle more detailed data or data that’s going to be used for more than a basic reconciliation: http://refine.deri.ie/rdfexport [10] sublime text 2 supports use of regular expressions. http://www.sublimetext.com/2 [11] e.g. http://gsfcir.gsfc.nasa.gov/authors/id/857747614 [12] documentation for the fedora batch modify process in fedora 3. https://wiki.duraspace.org/display/fedora34/batch+processing#batchprocessing-batchmodify [13] in a pairtree filesystem, objects nest under hierarchical two-character directory names, e.g. http://fedorarepository:8080/fedora/rest/39/3b/b2/64/object-uuid. these directories are generally non-semantic. unlike complex, proprietary structures intended to hide or obscure, the pairtree hierarchy can easily be parsed by basic system tools, easily backed up, retrieved by any system which could parse the identifier, and migrated to or reindexed by any system which understands pairtree. pairtree isn’t the only such hierarchical structure, but it was the one chosen by fedora’s developers. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – premis events through an event-sourced lens mission editorial committee process and structure code4lib issue 56, 2023-04-21 premis events through an event-sourced lens the premis metadata standard is widely adopted in the digital preservation community. repository software often include fully compliant implementations or assert some level of conformance. within premis we have four semantic units, but “events”, the topic of this paper, are particularly interesting as they describe “actions performed within or outside the repository that affects its capability to preserve objects over the long term.” events can help us to observe interactions with digital objects and understand where and when something may have gone wrong with them. events in premis, however, are slightly different to events in software development paradigms, specifically event driven software development – though similar, the design of premis event logs does not promote their “being complete” nor their consumption and reuse; and so, learning from logs in event driven software development, may help us to simplify the premis data model; plug identified gaps in implementations; and improve the ability to migrate digital content in future repositories. abstract the premis metadata standard is widely adopted in the digital preservation community. repository software such as archivematica includes a full representation of premis and other software like preservica assert premis conformance [1][2]. for the purpose of this paper, the implementation of premis is focused on digital objects and the different events around them that impact long-term preservation [3]. event sourcing is a software development methodology that provides a mechanism of persisting data that focuses on recording what has happened rather than how things are; rather than storing current state, event sourcing maintains “state mutations” as separate records called “events” [4]. event-sourced events and premis events sound similar to one another, but they look different and are implemented differently. in premis, compliance is often achieved through strict structural properties of the model and recording information in xml. there is less rigidity around the technical implementation of premis. as premis is difficult to engage with, i.e. write and consume, with xml a defacto encoding, we often see ad-hoc decisions made about when information is recorded in a system because not all events fit neatly inside the published data dictionary and not all events look like preservation events – even though they may have an impact on the digital object in the future. event sourcing on the other hand requires that all events are recorded for all aggregates (objects) in that model. event sourcing has a place in long-running processes, e.g. stock-processing, but this paper suggests it may also play a role in helping to record preservation metadata. linguistically we see a difference in how events are recorded in premis (as nouns) versus in the event-sourced model (as action verbs). we assert in this paper that this also makes it difficult to engage with, and may hold back data modeling. it is through an event-sourced lens that this author suggests that we may be able to advance premis further: better align it with software development methodologies that could potentially implement the standard; simplify the data model; plug gaps identified in the current premis 3.0 data dictionary; and improve the ability to migrate digital content in future repositories. introduction premis is a digital preservation data dictionary and schema. it stands for “preservation metadata implementation standard [5]. the standard: supports the viability, renderability, understandability, authenticity, and identity of digital objects in a preservation context; represents the information most preservation repositories need to know to preserve digital materials over the long term; emphasizes “implementable metadata”: rigorously defined, supported by guidelines for creation, management, and use, and oriented toward automated workflows; and, embodies technical neutrality: no assumptions made about preservation technologies, strategies, metadata storage and management, etc. there are four main elements of the data model described as semantic units: objects, agents, rights, and events[6]. objects – named and ordered sequence of bytes that is known to an operating system; units of information in digital form rights – assertion of rights and permissions associated with a premis object. agents – actors (human, machine, or software) associated with one or more event and/or rights statements associated with a digital object. events – actions performed within or outside the repository that affects its capability to preserve objects over the long term. events and how they interact with objects are the primary focus of this paper. physical objects, as described in premis 3.0, are outside of the scope of this work. agents are important inasmuch as an event does not occur without an agent associated with it. rights are only touched on incidentally in the remaining text [7]. event sourcing is a software development methodology that provides a mechanism of persisting data that focuses on recording what has happened rather than how things are; rather than storing current state, event sourcing maintains a “log” of “state mutations” as separate records called “events”. event sourcing is usually contrasted with a “crud” style of writing software that requires some form of long-term persistence. in some ways it is a parallel style of software development. crud stands for “create”. “read”, “update” and “delete” and those are usually implemented as functions on top of a database; functions to read, to delete, and so on. the database in this approach only records current “state”, i.e. records about how something is “at this moment.” optionally, if such a capability has been implemented, an audit trail may be persisted in a crud database that may, conversely, describe snapshots of a previous state and potentially what was done to change the state at the time. event sourcing does not assert a single use in a specific domain, and so its few domain objects are “primitives” upon which we can build. artifacts that we may be interested in are: aggregate – an aggregate is a unique object, artifact, or item that exists. event – is a record describing something that has happened to an aggregate; events tell us how we got to a current state. projection – provides a “view” of an aggregate, e.g. a representation of its current state or states. a projection is built from the underlying event-source model. an example of what an event log for a digital file may look like: aggregate img_001.jpg events {“event”: “checksum recorded”, “datetime”: “2022-01-01t00:01:00z”, “outcome”: ”ba5eba11”, “previous_id”: “e7e4de42”, “id”: ”8e66dad9”}, {“event”: “checksum recorded”, “datetime”: “2022-06-01t00:01:00z”, “outcome”: ”ba5eba11”, “previous_id”: “8e66dad9”, “id”: ”b5ef0830”}, {“event”: “checksum recorded”, “datetime”: “2022-12-01t00:01:00z”, “outcome”: ”badf00d”, “previous_id”: “b5ef0830”, “id”: ”db39686b”}, projection img_001.jpg → warning: the checksum has been modified, please investigate further figure 1. we may consider that aggregates in an event-sourced digital preservation model may be an intellectual entity in premis terms, or digital file or files. we simply have to define what the aggregate looks like and what events are associated with it. premis in more detail premis describes events that are performed inside or outside of a repository that affect the ability to preserve “objects” over time. there are 52 events listed on the library of congress website [8], including: accession, appraisal, compression, creation, encryption, ingestion start, ingestion end, message digest calculation, printing, rendering, replication, recovery, redaction, replication, unpacking, validation, virus check. in premis terms, an event theoretically affects the ability to preserve objects over time. in reality, this is not always the case. printing is extremely unlikely to affect our ability to preserve an object and is in contrast with compression which rewrites an object’s entire byte stream. events, as mentioned, are associated with objects, and those include the following as described in the premis 3.0 data dictionary: intellectual entity – a set of content that is considered a single intellectual unit for purposes of management and description: representation – a digital representation is the set of stored digital files and structural metadata needed to provide a complete and reasonable rendition of the intellectual entity. bitstream – contiguous or non-contiguous data within a file that has meaningful properties for preservation purposes. file – named and ordered sequence of bytes that is known to an operating system. an event may look as follows (this happens to be associated with a single file and is extracted from a sample ingest in the archivematica system): <premis:event xmlns:premis="http://www.loc.gov/premis/v3" xsi:schemalocation="http://www.loc.gov/premis/v3 http://www.loc.gov/standards/premis/v3/premis.xsd" version="3.0"> <premis:eventidentifier> <premis:eventidentifiertype>uuid</premis:eventidentifiertype> <premis:eventidentifiervalue> 46efc236-fc87-46d8-8f79-99778496bc05 </premis:eventidentifiervalue> </premis:eventidentifier> <premis:eventtype>virus check</premis:eventtype> <premis:eventdatetime>2022-10-04t20:24:15+00:00</premis:eventdatetime> <premis:eventdetailinformation> <premis:eventdetail> program="clamav (clamd)"; version="clamav 0.103.6"; virusdefinitions="26679/tue oct 4 07:56:50 2022" </premis:eventdetail> </premis:eventdetailinformation> <premis:eventoutcomeinformation> <premis:eventoutcome>pass</premis:eventoutcome> <premis:eventoutcomedetail> <premis:eventoutcomedetailnote></premis:eventoutcomedetailnote> </premis:eventoutcomedetail> </premis:eventoutcomeinformation> <premis:linkingagentidentifier> <premis:linkingagentidentifiertype> preservation system </premis:linkingagentidentifiertype> <premis:linkingagentidentifiervalue> archivematica-1.13.2 </premis:linkingagentidentifiervalue> </premis:linkingagentidentifier> <premis:linkingagentidentifier> <premis:linkingagentidentifiertype> repository code </premis:linkingagentidentifiertype> <premis:linkingagentidentifiervalue> artefactual </premis:linkingagentidentifiervalue> </premis:linkingagentidentifier> <premis:linkingagentidentifier> <premis:linkingagentidentifiertype> archivematica user pk </premis:linkingagentidentifiertype> <premis:linkingagentidentifiervalue> 2 </premis:linkingagentidentifiervalue> </premis:linkingagentidentifier> </premis:event> event sourcing in more detail a useful summary of what an event means is written by jon vines from ao.com [9]. beginning with the dictionary definition of an event: “an event is something that happens, especially when it is unusual or important.” they go further: when thinking about this in the context of our application estate, we can further define an event as something that has happened, that the organization cares about. further, we should remember that an event reflects the past nature of the occurrence. events are also immutable in their nature and are unable to be changed once they have occurred. there are examples that we can draw on: an action that a user has taken e.g. an order complete event in a retail scenario originating from a machine e.g. a delivery van, tracking its location as it moves a system event e.g. a logging event, showing duration of a transaction (think observability) fowler [10] describes a key principle of event sourcing; the core idea of event sourcing is that whenever we make a change to the state of a system, we record that state change as an event, and we can confidently rebuild the system state by reprocessing the events at any time in the future. the event store becomes the principal source of truth, and the system state is purely derived from it. for programmers, the best example of this is a version-control system. the log of all the commits is the event store and the working copy of the source tree is the system state. “when we change the state of a system, we record that state change as an event”. if we think in terms of the objects in our system, and think about the events associated with those, we can see a parallel with premis. having previously introduced the concept of these objects as aggregates, we can take the sum-total of all events across all our aggregates to tell us something more about our “system” as a whole. revisiting the checksum example in figure 1 an event that is important in digital preservation is fixity checking. the name itself implies three things: fixity is calculated, e.g. using a sha256 algorithm. fixity is compared with a previous value. fixity is determined to be good, i.e. unchanged; or bad, i.e. changed. fixity is an important “event” but if we inspect the process above, then only one event is described: “fixity is calculated.” in this process, the previous state is read to check against the current value and a new state is potentially recorded. discovering something has changed may be “eventful”, but it is more likely to be the trigger for a set of corrective processes than an event that is recorded. in figure 1 we have an aggregate “img_001.jpg”, and for that aggregate we record the event: “checksum calculated” three times, in january, june, and december [11]. our events simply record state changes in our system. in a digital preservation system we can think about the changes to the metadata for the objects in our repository. as humans we can see there is a problem in december when we observe a change to the outcome of recording fixity. our “projection,” one of many potential views of our data, discussed shortly, can be regenerated, and in this case, our view is specifically designed to trigger a warning that the checksum has changed. this warning can be forwarded to the repository manager and remedial actions can begin. remedial activities are likely to create new events: aggregate img_001.jpg events {“event”: “checksum recorded”, “date”: “2022-01-01t00:01:01z”, “outcome”: ”ba5eba11”, “previous_id”: “329623b1”, “id”: ”7f914e0e”}, {“event”: “checksum recorded”, “date”: “2022-06-01t00:01:01z”, “outcome”: ”ba5eba11”, “previous_id”: “7f914e0e”, “id”: ”0e9b3a89”}, {“event”: “checksum recorded”, “date”: “2022-12-01t00:01:01z”, “outcome”: ”badf00d”, “previous_id”: “0e9b3a89”, “id”: ”54089c0a”}, {“event”: “object recovered”, “date”: “2022-12-01t00:02:01z”, “outcome”: ”img_001.jpg retrieved from lockss server 002_de”, “previous_id”: “54089c0a”, “id”: ”df63ed10”}, {“event”: “checksum recorded”, “date”: “2022-12-01t00:03:01z”, “outcome”: ”ba5eba11”, “previous_id”: “df63ed10”, “id”: ”a1c82d8a”}, projection img_001.jpg → info: file integrity remains good. figure 2. the projection created from the new event log in figure 2 is a heuristic that compares recent checksum values with the first checksum of the aggregate as it was ingested. in this case the result for our aggregate is a positive one and recovery has been successful. the projection may be created as soon as a “checksum recorded” event happens, but the projection here is produced by just one consumer of the event feed, the event source, and other consumers may have different desires for this information. take, for example, a view of the file’s characteristics based on looking at the object’s characteristics: an example with metadata extraction aggregate img_001.jpg events {“event”: “format identified”, “agent”: “siegfried”, “date”: “2022-02-01t00:01:00z”, “outcome”: ”unknown”, “previous_id”: “005f52d2”, “id”: ”9eeda82f”}, {“event”: “metadata extracted”, “agent”: “mediainfo”, “date”: “2022-03-01t00:01:00z”, “outcome”: ”bit-depth: 24-bit”, “previous_id”: “9eeda82f”, “id”: ”70eaa5cc”}, {“event”: “format identified”, “agent”: “siegfried”, “date”: “2022-04-01t00:01:00z”, “outcome”: ”unknown”, “previous_id”: “70eaa5cc”, “id”: ”d7e644b4”}, {“event”: “metadata extracted”, “agent”: “gimp”, “date”: “2022-05-01t00:01:00z”, “outcome”: “resolution: 1000x1000; layers: 1;”, “previous_id”: “d7e644b4”, “id”: ”56e8d2f3”}, {“event”: “metadata extracted”, “agent”: “exiftool”, “date”: “2022-05-01t00:02:00z”, “outcome”: ”created-by: windows paint 128-bit”, “previous_id”: “56e8d2f3”, “id”: ”3bfbf661”}, {“event”: “format identified”, “agent”: “siegfried”, “date”: “2022-12-01t00:01:00z”, “outcome”: ”fmt/30001; jpeg-3000-super-hd;”, “previous_id”: “3bfbf661”, “id”: ”488a41b9”}, projection img_001.jpg → “metadata”: { “projection date”: “2023-01-01”, “resolution”: “1000x1000”, “bit-depth”: “24-bit”, “layers”: 1, “created-by”: “windows paint 128-bit”, “puid”: “fmt/30001”, “format name”: “jpeg 3000 super hd”, } figure 3. in our second example, we see additional events for the same aggregate over the course of a year. the events tell a story: a format could not initially be assigned to the file, and it took a few attempts, eventually resulting in the new format “fmt/30001; jpeg 3000 super hd” by the end of 2022. different metadata extraction tools were used over the course of the year, “mediainfo”; “gimp”; and “exiftool”, and at each separate attempt new information was extracted. in the projection we simply return summary information about the file encoded as json. an alternative projection may encode this as xml. another may take some of our other events associated with the aggregate and also display the checksum so that it can be downloaded safely by those accessing it. notably, the projection has a date as well – perhaps this was configured as an argument to the function creating it, and in another rendering of this information with the given date february 2022, we would only receive a block of json with “format name” and “puid” as “unknown” – importantly: the information we had about the object at an earlier point of time in the system; a view of its “historic” state. other important features of event sourcing append only event-sourced logs are “append only.” data is added at the end of the log and earlier data should (may) never be deleted. data can be write-once, read-many and over the long-term, the changes to the record are all accessible and auditable; this is in line with other modern approaches to recording strongly provenanced information in technologies such as web 3.0, e.g. blockchains. append-only logs are less susceptible to inadvertent changes – while not beyond the capability of any developer, modifying data structures such as xml or json in memory is inherently risky, especially a structure such as xml that requires the document tree to be read into memory correctly, appended, and then written back to disk. data for append-only logs need only be added at the end of a process. objects (aggregates) can be continuously monitored and updated with an event-sourced approach. if one decides to run a process such as a “reingest,” then the events occuring during the re-processing of an intellectual entity, or entire accession, are simply added to the end of a log, removing the need for reversioning, or modifying anything that already exists – the log tells the story. different consumers with different needs consumers of event sources have different needs. we have seen two examples in this text: creation of “object characteristics” like structures. checksum monitoring. but other projections are needed: outputs in different metadata formats. access logs. derivation logs, i.e. listing derivatives of files and their locations. system-wide: collection-wide views of digital objects. events are the simple but powerful building blocks that contribute to creating complex views of our digital preservation systems. we do not need to understand all the views we need today, we can simply delve into the event logs at a point in the future and pull out the information we need to understand past and present state. no need for migrations, or, at least, migrations may not be such a burden in a structural metadata model, usually stored on disk, or in a database, such as mets/xml with premis, if the standard or representation changes in any way, then a migration (from one structure or schema to another) may be required to make the old version of the model compliant to the updated standard. using events, the generation of a new structural model using a different or newer standard, e.g. portland commons data model, or premis 4.0, can be done by versioning and incrementing the code used to generate the previous model. the foundational events and data they contain does not change. users and systems, then, have easy access to both versions, as they do multiple representations, using projections. functions used to create projections today when a new projection is created. while any structural model can then be maintained on disk, the creation of a projection is a more distributed [12] approach to doing this, and can be performed at run-time, as opposed to complete versions of structural models being maintained alongside archival packages [13]. premis changes we cannot simply apply an event-sourced lens to premis in its current form. if premis were to become compatible at all with this approach some changes are required. nouns become actions in the past-tense premis events are often nounified. in some cases the event verb is simply used as a noun through a “functional shift” and remains the same read both ways – as noun and a verb. as nouns, events lose their temporal energy and become static, instead of simply being “something that happened” they are the titles of big, capital “e” “events,” choreographed into rigid digital preservation processes as opposed to something dynamic and changing, especially actions that may be unseen. it is this author’s assertion that titular premis nouns become a barrier to engage with, use, and extend without committee. if we maintain the focus that events describe something that has happened and hold onto the original verb, then it may become easier to work with. event sourcing uses action verbs, but also uses the past tense “something has happened,” and so mapping premis 1:1 today into an event log is an unnatural fit. instead we need to convert as follows: noun verb action accession accessioning …was accessioned creation creating …was created replication replication …was replicated unpacking unpacking …was unpacked validation validating …was validated figure 4. less importance placed on the data dictionary a data dictionary is a valuable contribution to digital preservation literature, however, rather than trying to capture all possible events, we can potentially find more flexibility in an event-sourced approach where any activity is captured, not just prescribed actions. events that do not fit easily into a “nounified” category, e.g. “checking for zero byte files” can be logged as an event simply labeled with the activity itself: “checked for zero byte files”. this may be associated with an intellectual entity aggregate, or the file “checked size on disk is greater than zero;” the latter could also be rendered as a projection at the end of a process, where the projection performs some form of validation function. closer to a technical implementation the only conscious decision of an event-sourced approach to event logging is that all events are captured for an aggregate. a system capturing all events need only consider the design of the event log itself and ensure that it can capture enough information. more elaborate crud style data models are not needed, and can indeed become emergent as projections sitting on top of the logs being created. “design”: schema, or monolithic system’s design, are not front-loaded, meaning that systems can be developed quicker, while the use of the data from these future systems can be designed simply knowing the data is coming in and that there will be access to an event log. replace object characteristics and object-characteristic extensions with projections capturing an object’s characteristics is done in premis using the object-characteristics section of the data model. characteristics unique to a particular representation of a file can only be captured using an “extensibility mechanism” described as “objectcharacteristicsextension”. this is itself somewhat of a data model within a data model, and, crucially, more difficult to work with in that there are no restrictions on the data captured here or the model used. if xml is captured from jhove and also tika, the xml can be stored here verbatim and requires further parsing to split up. with an event-based approach, there is potential to capture unique object characteristics through events and then render information about characteristics through a new projection, as illustrated in figure 3. premis conformance restrictions around internal schemas are made more flexible a transformation on any set of data creates an abstraction. premis itself is an abstraction of the data accumulated to build our digital preservation knowledge base. what event sourcing does differently is ask that we engage with a lower-level abstraction of some set of source data (our encoded events) to enable us to generate many more higher level abstractions. as we become more proficient with data-like events [14], one thing we may take away from looking at premis through an event sourced lens is that lower-level abstractions like events are not “less” than their higher level structural metadata counterparts, they’re just the building blocks. as the premis committee considers their next revision, then taking another look at what it means to be conformant, not from a data modeler’s perspective, but from a software development perspective, may be beneficial. the premis conformance statement suggests three levels of conformance while making clear that: adherence to the conformance principles is not a formal requirement for implementing the premis data dictionary (although the editorial committee does believe that following these principles would be good practice in nearly all implementation contexts). the conformance statement goes on to describe the following: the levels are built around three ways of implementing premis in any repository system: being able to map preservation metadata to premis, being able to export preservation metadata as premis, using premis as an internal schema in a way that does not require any further mapping or conversion. two challenges to the levels of conformance are: while described as levels, should they be viewed hierarchically? the first level of conformance clearly makes interpretation of an internal schema next to premis more difficult, and vice versa. one may remove this level of conformance completely. conceptually, however, two and three, have the same outcome, functionally, they take different paths toward getting there. can one approach realistically be viewed as better than the other? is level three reasonable in today’s software development world, is it reasonable in today’s environmental climate? given any well-formed dataset we have the potential to convert it into numerous representations – abstractions at the front and center of premis’ understanding of digital records. despite the conformance statement’s secondary description of events “sufficient event metadata to document actions the repository has taken to preserve the digital objects.” that seems to be at odds with its formal description, then premis can be trivially demonstrated to record information that extends beyond that which is useful for preservation. it is useful for presentation, access, other maintenance activities, and so on. is premis reasonably expected to be the source dataset for new representations of this data, beyond those that support digital preservation? is it anticipated that premis becomes a companion dataset alongside another representation?whatever the cost, storage costs “something”, fiscally, environmentally, and intellectually – data asks a lot of us and our resources. environmentally we should be reducing our data footprint, we do have to ask if premis is the most efficient format to be used as a sole representation of digital preservation metadata; and if the answer is no, level three conformance may be re-considered, or nuanced somewhat, as optional, but no more importance placed on it than being able to extract premis as an abstraction of another format. premis conformance should be separate from representation. if we acknowledge premis is at least one important representation of preservation metadata, i.e. for its ability to act as an interface to those looking to interpret preservation metadata, then whether it exists logically on disk, or is generated through an event sourced projection, is irrelevant. how a representation complies with the premis data model remains of greater importance, but this is measured from the same eventual view, whatever intermediate abstraction it sits within. analysis compared to premis potential improvements there is an opportunity when looking at premis though an event-sourced lens to simplify both processes outputting event information and the structural data model, even potentially alleviating the need for parts of premis that do not sit easily in the implementation, such as unique object characteristics (objectcharacteristicextension), because they can exist naturally in an event-sourced approach and used only as the consumer requires. event-sourced logs are simple, so they are potentially easier to write into any style of archival package. there is no encoding implied by event-logging, only that events need to be able to be read and processed and other encodings can be generated from event logs as projections, satisfying many of the different uses of the data in an archival environment. events are simple and complex database schema are not required for their logging. they can even be stored as plain-text, and/or distributed alongside archival packages. an event-sourced approach leads with the premise that “not everything has to be known up-front,”, i.e. fit into a neat model. some information is simply not available early on in a preservation workflow, e.g. for new file formats, other times information needs to fall out of other research processes and becomes available over time. potential drawbacks in a truly event-sourced system, logs can grow large because of the number of processes occuring or objects processed. there is always a potential they will grow large for the aggregates in an archival workflow as well, resulting in the need for greater computing resources and subsequent greater impact on the environment. additionally, time is required to process lots of events, and so an event-sourced view of an archival package may mean slower access times. time can be saved by caching projections. the amount of data for aggregations of archival objects may grow quite large, but those for individual aggregates may be somewhat less. thinking about events for individual objects is a different proposition than thinking about the logs of long-running systems and processes. conclusion premis remains a hugely important tool in digital preservation. its implementation has worked well for many institutions to this day. the data model is, however, verbose, and for the number of elements required to encode data, there is not a lot of information encoded – its entropy is high and still requires a lot of processing to develop views on top of what is there. there is no easy way to read the data as a human without additional viewers such as metsflask. projects have been undertaken to understand how to optimize premis output, looking at reducing files with lines numbering in their millions into thousands because of how much space premis requires [15]. if one looks at the essence of premis then arguably, its single greatest drawback is the verbosity of the structural data model, not the concept of events as “actions performed within or outside the repository that affects its capability to preserve objects over the long term.” indeed, others have looked at making the encoding of premis more light-weight [16]. current premis models also tend to show us just the current state of an object in time. we do not necessarily see how we got there. we know systems can generate new snapshots through processes such as “reingest”, modifying the existing output, or adding a new version of the metadata to a package, but these are only snapshots, not a complete story. perhaps, then, it doesn’t track, that this paper will shortly conclude by suggesting that using events differently, it may be possible to keep more data than premis currently supports in its model, as well as keep metadata sizes low – but this is the prospect of an event-sourced lens on the topic. if one takes compression as their analogy, then an event-log is like a losslessly compressed package [17]; concentrated information distilled down to its essence. we can store more and when we “uncompress” or expand this data we can generate a number of different representations of this data that satisfy many more of our needs – these are our projections. adopting a truly event-sourced approach for premis creates an important separation between “data that is recorded” and “data that is used”. event sourcing activates stored data by its very nature – the event-log requires projections to live beyond events – as data, triggers, or components of user interfaces, e.g. charts and graphs. as we become more proficient at processing event log data, we can find emergent representations we may not yet be considering. alexandra chassanof asked back in 2020: what i’m really wondering is about whether institutions are using premis elements to analyze & manage ingested preservation objects and whether this happens periodically or regularly – if at all. probably a longer conversation would be helpful [18]. further back the premis health-check (phc) project in 2013 looked at aligning premis with the spot threat model for actively monitoring digital preservation repositories [19]. in an event-sourced premis model, events record everything that can affect the numerous different states of the information around the digital objects (aggregates) in our repositories. the projections, of which, we require at least a minimum number, even to simply return sensible user-facing information about object metadata; these become the foundation for other projections that consume the events needed to actively manage objects in our repository, from the phc’s perspective, multiple consumers can be configured to generate different projections revealing different levels of threat. additionally, event streams can be combined to create projections for any entity the project may have desired, e.g. digital object, collection, or repository. we can create as many dashboards as we desire with warnings galore informing us that our content is potentially at risk [20]. if we have been writing projections for consumers to see object metadata, alone, then modifying these consumers of events becomes trivial to start exploring the other possibilities such as those in the phc. readers may also see that premis is evolving too – premis rdf [21] may achieve some of what is described here, the extensibility discussed, a slightly smaller footprint [22] for example. a graph-based approach to recording metadata still has its drawbacks. to start, it still tries to (needs to) assert a structure through an ontology that codifies its purpose and use. this is how we have tended to approach our work in digital preservation; perhaps coming from fields so closely linked to knowledge organization, we strive for structure which may in turn come from the field’s home in archives and libraries. an event-based approach still complements this, but asks that structure to become emergent [23]. a graph of digital preservation information (premis/rdf) can be yet another output. it is this paper’s assertion that we can store more, and “do more” by taking an event-sourced approach to storing events associated with the “objects” described in the premis data dictionary. as of yet, this paper is just one projection. the next step is to develop some more models, describe some principles of event sourcing for digital preservation, and generate some real world data, and to try and prove it. about the author ross spencer (orcid: 0000-0002-5144-9794) is a digital preservation specialist for the international games and book publisher, ravensburger ag in germany. ross has been part of the digital preservation community for over a decade and has formerly been a digital preservation analyst and programmer in the teams at the national archives uk, archives new zealand, and artefactual systems inc. acknowledgements this author would like to thank peter van garderen for his review and support around the direction of this article. they would also like to thank evelyn mclellan at artefactual systems for her generosity in sharing her knowledge around premis while i had the opportunity to work alongside her. endnotes [1] level b is cited in (o’sullivan et al., 2019) [2] premis conformance statement and levels, (premis editorial committee, 2015) [3] the paper has yet to find a way to incorporate analog “versions” as introduced in premis 3.0. [4] (zimarev, 2020) [5] (premis editorial committee, 2015) [6] this information is most easily accessed through the premis owl ontology, (premis 3 ontology, n.d.) [7] while rights are interesting, the scope of rights statements in premis is limited to documenting whether a repository has the right to perform a certain action in an automated way, and documenting that assertion. we will assume this right for this discussion although it is also possible to envisage events not happening due to permissions being unavailable. [8] (event type, n.d.) [9] (vines, 2020) [10] (fowler, 2017) [11] in december the checksum has changed, which is problematic – but also, let us assume that we normally sample at greater frequencies than this in reality, and this is the first time the change has occurred. [12] i.e. bringing data, from heterogeneous “distributed” sources, not just the local disk, to generate one concrete projection [13] the university of north texas (unt) adopts a hybrid approach, using a distributed events model, i.e. premis events are separated from their archival packages, but they are stored as a structural model and delivered wrapped in the atom publishing format and protocol which provides additional contextual metadata. (phillips et al., 2011) [14] or in general in our different information management disciplines. [15] evelyn mclellan investigated how bloat in archivematica’s xml representation of premis could be reduced, (mclellan, 2018) [16] the bodleian library, for example, began a mets/premis analysis that sought to reduce the size of mets/premis in archivematica. one of the published artifacts around this work includes a survey seeking to understand the community’s appetite for rdf-turtle as a way of improving throughput performance, (bodleian libraries, n.d.) [17] claude shannon’s theories showed us that messages could be losslessly reduced, by removing redundancies, but then still reproduced, i.e. source coding. similarly, data can be encoded like this from the outset, and still be used to construct familiar (and predictable) structures, (aftab et al., 2001) [18] (chassanoff, 2020) [19] (van der werf, 2013) [20] the corollary of which are the green lights giving us much comfort. [21] premis-rdf examples, (caron et al., 2019) [22] there are different tradeoffs with rdf representations. rdf/xml still contains a lot of excess characters, where turtle (ttl) removes a lot of that but maintains a lot of whitespace. n-tuples, are “maybe” the closest to an “event log”, however, “subject” uris are often repeated many times over, also taking space. [23] young (2014), on events, alludes to the question “do i store a structural model” or events from which a structural model can then be generated, i.e. “give you back” a structural model, (young, 2021) references aftab, o., cheung, p., kim, a., thakkar, s., & yeddanapudi, n. (2001, december 16). information theory information theory and the digital age [6.933 final paper]. retrieved february 02, 2023, from https://web.mit.edu/6.933/www/fall2001/shannon2.pdf bodleian libraries. (n.d.). archivematica metadata survey. webarchive.org. retrieved 02, february, from https://web.archive.org/web/20210201133052/https://docs.google.com/forms/d/e/1faipqlseahxmtkyk7yz2uzmd86z-ams0ccofme6uzpu-07mqdny1rlq/viewform caron, b., mclellan, e., duval, m., & cowles, e. (2019, october 25). premis ontology examples. github. retrieved february 17, 2023, from https://github.com/premis-owl-revision-team/premis-owl/tree/master/examples chassanoff, a. (2020, february 19). twitter. retrieved february 17, 2023, from https://perma.cc/sh8b-nd5q event type. (n.d.). premis 3 ontology: event type. retrieved february 16, 2023, from https://id.loc.gov/vocabulary/preservation/eventtype.html fowler, m. (2017, february 7). what do you mean by “event-driven”? martin fowler. retrieved february 16, 2023, from https://martinfowler.com/articles/201701-event-driven.html mclellan, e. (2018). premis/mets for scalability. premis/mets for scalability: wiki.archivematica.org. retrieved february 02, 2023, from https://wiki.archivematica.org/index.php?title=premis/mets_for_scalability&direction=prev&oldid=13322 o’sullivan, j., smith, r., gairey, a., & o’farrelly, k. (2019). a pragmatic application of premis: mapping the key concepts to a real-world system. proceedings: 16th international conference on digital preservation, 1-10. retrieved february 16, 2023, from https://ipres2019.org/static/pdf/ipres2019_paper_49.pdf phillips, m. e., schultz, m., & nordstrom, k. (2011). premis event service. retrieved february 02, 2023, from https://digital.library.unt.edu/ark:/67531/metadc40413/m1/1/ premis 3 ontology. (n.d.). premis 3 ontology. retrieved february 16, 2023, from https://id.loc.gov/ontologies/premis-3-0-0.html premis editorial committee. (2015, april). conformant implementation of the premis data dictionary. retrieved february 17, 2023, from https://www.loc.gov/standards/premis/premis-conformance-20150429.pdf premis editorial committee. (2015, june). premis data dictionary for preservation metadata (3.0). premis data dictionary for preservation metadata. retrieved february 16, 2023, from https://www.loc.gov/standards/premis/v3/premis-3-0-final.pdf van der werf, t. (2013, september 5). preservation health check: work in progress. retrieved february 17, 2023, from https://www.loc.gov/standards/premis/pif-presentations-2013/05premis-vdw-phc.pdf vines, j. (2020, july 2). what is an event, anyway?. by software development team lead, jon… | by ao.com | ao’s engineering blog. medium. retrieved february 16, 2023, from https://medium.com/aos-engineering-blog/what-is-an-event-anyway-651122f4f3e6 young, g. (2021, march 22). transcript of greg young’s talk at code on the beach 2014: cqrs and event sourcing. event store. retrieved february 17, 2023, from https://www.eventstore.com/blog/transcript-of-greg-youngs-talk-at-code-on-the-beach-2014-cqrs-and-event-sourcing zimarev, a. (2020, june 3). what is event sourcing? event store. retrieved february 17, 2023, from https://www.eventstore.com/blog/what-is-event-sourcing subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – shiny fabric: a lightweight, open-source tool for visualizing and reporting library relationships mission editorial committee process and structure code4lib issue 47, 2020-02-17 shiny fabric: a lightweight, open-source tool for visualizing and reporting library relationships this article details the development and functionalities of an open-source application called fabric. fabric is a simple to use application that renders library data in the form of network graphs (sociograms). fabric is built in r using the shiny package and is meant to offer an easy-to-use alternative to other software, such as gephi and ucinet. in addition to being user friendly, fabric can run locally as well as on a hosted server. this article discusses the development process and functionality of fabric, use cases at the new college of florida’s jane bancroft cook library, as well as plans for future development. by atalay kutlay, cal murgu and tammera race introduction at the new college of florida, we sought to develop a tool that allowed us quickly to visualize the relationships of our instructional librarians and teaching faculty using internal data. our main objectives with this project were twofold: 1) develop an open-source application that can easily render network graphs depicting instructional activities at our library using our libinsight datasets; 2) include export/reporting functions for straightforward sharing. over the course of development, we included several stretch-goal features, including relationship weighing and additional reporting features. this article documents our experience creating a shiny app, fabric, that quickly and efficiently renders interactive network visualizations. fabric renders an illustration of the recent and current relationships between librarians and faculty members, and offers fast exporting functions for planning purposes. ultimately, fabric is a lightweight, open-source alternative to other social network analysis (sna) applications, like gephi or ucinet, that supplements the visualization features currently available in libinsight. background like many other academic libraries, the new college of florida jane bancroft cook library began collecting various data to document the work of librarians in different service areas, including instructional data as well as research consultations. we started using libinsight in 2016. we created a series of standardized forms that collect a variety of information, such as front-desk analytics, library instruction, and research support interactions with students and faculty. while we have been analyzing this data in different ways since then, either through libinsight’s built-in visualization tools or through local analysis using excel or r, we sought to create an application that allowed us to easily render network graphs depicting instructional activities and enabled us to create reports for internal and external stakeholders. while libinsight provides basic reporting functionalities, such as line, column, spline, and pie charts, we wanted to represent the relationships of our librarians with faculty and students in a way that is more akin to how we see ourselves: as part of an academic network. visualizing library data has been in vogue recently. others have written about their experiences creating applications centered on the application of library data. costello and martin, for example, detailed stony brook university libraries’ efforts to consolidate their data streams into a tableau dashboard that enabled them to “tell stories with data” (costello and martin, 2018). their objective is a familiar one: they required “a way to quickly share information among different [stakeholders within] and outside the library” (costello and martin, 2018). similarly, horne-popp, tessone and welker detailed the implementation of a local data dashboard, built on rails, concluding that, among other things, “the ability to visualize data by setting a few parameters greatly increased the flexibility of data sharing with library administration and other internal and external stakeholders” (horne-popp et al., 2017). other studies focus on how we can leverage data to support library strategic-decision making (murphy, 2013), scholarly communications (smith, 2013), library assessment (murphy, 2015), and collection development (lewellen and plum, 2016). it is not at all surprising that tableau features in much of this scholarship, although not exclusively, given the software’s relatively low barrier of entry yet excellent querying and visualization options. this speaks to an important element of any data centered visualization strategy: it needs to be accessible, both for a user as well as the administrator. the aforementioned literature informed our own approach to visualizing library data. in particular, we noticed that the type of graph that made most sense to us given our focus on relationships was missing from many of these dashboards: network graphs. after considering several options, including an internal pipeline that pushed libinsight data to proprietary software such as ucinet and open software such as gephi, we decided to explore the r package, shiny (https://shiny.rstudio.com/). shiny is an open source package which makes it easy to build interactive web applications or data dashboards. shiny leverages r’s abilities to enable you to create interactive web applications that focus on data (beeley and sukhdeve, 2018). most shiny applications revolve around displaying data in interactive, manipulable views. in addition to shiny’s relatively simple development process, we used shiny because of its limited footprint. essentially, shiny applications are r projects that can be run locally and do not require a server or an intricate environment to run. the application can also be hosted, either on a web server or on shinyapps.io, a freemium hosting solution provided by rstudio that enables low-volume users to host shiny applications at no charge (limited to 25 hours of active hosting per month). as a result of this flexibility, and a desire to learn more about how we could leverage r in the library, we decided to build a shiny web application. networks before we explore our application, fabric, a quick primer on network graphs would be useful in situating our project. social network analysis emerged as a methodology in the field of social psychology and sociology in the early 20th-century (scott, 1988). it promoted the empirical and theoretical analysis of relationships between agents (people, institutions), taking methodological direction from graph theory, the mathematical study of graphs, as well as the qualitative study of social relationships. depicting these relationships as a series of visual models was a natural extension of social network analysis. taking cues from graph theory, scholars interested in social network analysis developed a variety of measures to determine the centrality of certain nodes in a network (a proxy for importance), the density of a graph (number of relationships), and node clustering, among other methods. however, a basic network graph is actually quite simple. it includes nodes (people, objects, things) connected by a series of lines, known as edges or relationships. a simple network graph can be seen in fig 1. figure 1. a basic network graph for our use case, the nodes represent instructors and librarians, and the edges represent instructional activities. the first iteration of our results can be seen in fig 2. while this basic data is enough to create a network graph, it limits its potential for identifying patterns among relationships in the graph because it does not include other variables that introduce additional nuance. for example, our first result did not delineate between type of nodes (librarians or teaching faculty), departments or schools (arts, sciences, social sciences), and the number of interactions and intensity of the relationship between nodes. the graph required a significant amount of institutional knowledge to be able to be read and applied in a strategic way. figure 2. network graph of librarians and teaching faculty additionally, we realized that beyond visualizing the nodes and relationships that are evident within the dataset, we needed to include those individuals (in our case, instructors) who were not included in the dataset. in other words, we needed to include nodes that, to date, did not have a relationship with a librarian. after an analysis of our libinsight data, we recognized that what we collected enabled us to render more complex network visualizations. an example of the final iteration of our network graph rendering can be seen in fig 3. figure 3. complex network graph showing additional variables functionality our application allows users to render graphs varying in complexity based on the type of data that is available. while we happen to collect data using springshare’s libinsight, fabric simply needs a properly formatted csv to function. to render a basic network graph, a dataset need only include two columns: a column listing librarians, and a column listing instructors. this basic information is enough to create a graph that shows the relationships with faculty, with dynamically weighted edges. should a user want to delineate between departments or schools, an additional dataset which includes two columns is required: a column listing faculty members, and a column listing their departments or schools. developing the shiny application there are 2 main components in a shiny app: ui and server. for the ui component, developers can either use shiny’s own functions to create a fluid ui or use their own custom html templates with correct div classes and ids. for fabric, we used a custom html template for more flexibility and better user experience, such as mobile compatibility. the data manipulation, analysis, and rendering occurs in the server component. we used the visnetwork package to visualize the social network, the dt package for the tables, and the tidyverse package collection to handle most of the data manipulation and analysis. fig. 4 shows the structure of a shiny application, as well as the libraries we used in fabric. figure 4. the structure of fabric shiny application using the application fabric offers a simple user interface. there are 4 main steps to generate and export a social network, which can be seen in fig. 5. figure 5. featured steps in fabric 1. file upload: the first step requires users to upload their interaction data and/or an optional complete list of faculty. users may choose either one of the existing datasets for testing, or upload their own datasets for analysis. users can also choose to anonymize the data that is visualized. figure 6. screenshot of the file upload 2. data verification: like with other network graph visualization software, users need to describe certain features of the dataset. the second step requires users to explain which column lists librarians and instructors. if users choose to include an option complete faculty list, they will need to clarify which columns have the faculty members and their division/school/department in the second optional dataset. figure 7. screenshot of the data verification 3. results: the third step renders the interactive network graph. users can click on each node for additional information, such as the distribution of their relationships among divisions, top 5 interactions, and average number of interactions with the same person. users can also facet by particular groups of nodes, or by individual nodes. figure 8.1 screenshot of results figure 8.2 screenshot of results 4. export: finally, the fourth step offers an export function that creates an interactive html file, which can be opened and manipulated locally. we find exporting helpful to use the data in reports or presentations. 5. understanding the results understanding a complex network graph may be confusing at first. in the following section, we briefly explain the information you can gain from the network graph. figure 9 different weights of edges first, we will look at the types of information that can be gained from the edges. 3 different types of edges can be seen in fig. 10. edges are weighted with the number of interactions between nodes. looking at fig. 9, we can deduce that the number of interactions between person 2 and person 26 is greater than the number of interactions between person 2 and person 39. figure 10 pie chart of the division/school distributions second, we will consider the charts and tables that our tool generates. fig. 11 indicates the distribution of a librarian’s instructional relationships across different schools or divisions. na may represent possible errors in the dataset (disambiguation, missing fields, spelling mistakes). this information may be particularly useful in settings where instructional librarians are not linked to specific departments or schools. finally, fig. 12 shows a table summary of the top interactions a librarian has with instructors. figure 11 table of top interaction list for a librarian discussion we have developed this tool for our own particular needs but with the larger library community in mind. visualizing library data is an important practice given the ability of visuals to communicate information, and for visuals to enable us to determine insights and patterns that are sometimes obscured by the spreadsheet. the following provides you with a notion of how we have used this tool at the jane bancroft cook library: to assess whether the shift from the liaison to team-oriented instruction model has changed the way we interact with instructors across disciplines. a year-to-year comparison made clear that despite moving away from our liaison model, our interactions with faculty continue to be informed by previous relationships between subject librarians and their “home” disciplines; to identify strong faculty partnerships that consistently ask for and are active supporters of library instruction; to identify gaps in our instruction services with respect to who and what departments we collaborate with more often than others. this allowed us to explore why that gap exists in the visualization, and prioritize actions to reconcile these gaps; allows us to onboard new librarians in a way that is much more intuitive, by making it possible to visualize institutional memory. we are working to embed this workflow as a part of our onboarding process. this is critical given that according to several studies, the cost of recruiting a new employee “approaches 30 percent of the new hire’s annual salary and the total cost of replacing an employee can be as high as 150 per cent of the annual salary after calculating lost productivity, recruiting expenses, retraining and time-to-productivity.” moreover, research indicates that the return on investment for a comprehensive onboarding program can reduce employee turnover from an average of 44 to 14 percent while improving new employees’ time-to-productivity by 33 per cent (hall-ellis, 2014). visualize instruction activities in promotion material, strategic planning, and stakeholder engagement meetings. conclusion and future development in conclusion, while much work is being done to operationalize library data for a variety of institutional and professional reasons, we found that there was a lack of focus on network graphs. beyond simply quantifying the number of instruction sessions, we consider network graphs to be the most appropriate way to visualize instruction data given the importance of relationship building to successful library services, including instruction. using shiny, an r package for web application development, we created fabric to enable library professionals to quickly visualize relationships in the form of network graphs. the resulting visualization is interactive, and can be exported and easily shared with internal and external stakeholders. we have found that this approach has helped us confirm or reconsider certain ideas that we hold about our contributions to the institution. we hope our examples will give you a sense of how this tool could be modified/applied in your specific context. we plan to continue developing certain aspects of the application. for example, we would like to include temporal delimiting in the application to allow for effortless year-to-year analysis. try it out the complete source code for this project can be found at: https://github.com/ncflib/fabric if you’d like to try the application, check out the most current version at the following link: http://dss.ncf.edu/fabric. notes beeley, c., & sukhdeve, s. r. (2018). web application development with r using shiny: build stunning graphics and interactive data visualizations to deliver cutting-edge analytics. packt publishing ltd. costello, l & heath martin (2018). creating the library data dashboard. in using digital analytics for smart assessment (pp. 107-116). american library association. hall-ellis, s. (2014). onboarding to improve library retention and productivity. the bottom line, 27(4), 138-141. horne-popp, l. m., tessone, e. b., & welker, j. (2018). if you build it, they will come: creating a library statistics dashboard for decision-making. in developing in-house digital tools in library spaces (pp. 177-203). igi global. lewellen, rachel, and terry plum.“assessment of e-resource usage at university of massachusetts amherst: a mines for libraries study using tableau for visualization and analysis.” research library issues: a report from arl, cni, and sparc, no. 288 (2016): 5–20. murphy, s. a. (2013). data visualization and rapid analytics: applying tableau desktop to support library decision-making. journal of web librarianship, 7(4), 465-476. murphy, s. a. (2015). how data visualization supports academic library assessment: three examples from the ohio state university libraries using tableau. college & research libraries news, 76(9), 482-486. scott, j. (1988). social network analysis. sociology, 22(1), 109-127. smith, v. s. (2013). data dashboard as evaluation and research communication tool. new directions for evaluation, 2013(140), 21-45. about the authors atalay kutlay is the digital scholarship library fellow at the new college of florida jane bancroft cook library. cal murgu is the digital humanities librarian at the new college of florida jane bancroft cook library. tammera race is the systems, metadata & assessment librarian at new college of florida subscribe to comments: for this article | for all articles one response to "shiny fabric: a lightweight, open-source tool for visualizing and reporting library relationships" please leave a response below: samato, 2020-03-31 updated to include third author. leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – getting more out of marc with primo: strategies for display, search and faceting mission editorial committee process and structure code4lib issue 41, 2018-08-09 getting more out of marc with primo: strategies for display, search and faceting going beyond author, title, subject and notes, there are many new (or newly-revitalized) fields and subfields in the marc 21 format that support more structured data and could be beneficial to users if exposed in a discovery interface. in this article, we describe how the orbis cascade alliance has implemented display, search and faceting for several of these fields and subfields in our primo discovery interface. we discuss problems and challenges we encountered, both primo-specific and those that would apply in any search interface. by kelley mcgrath and lesley lowery introduction we’ve been hearing for years, even decades, that the marc format is dying. nevertheless, the marc advisory group continues to meet twice a year and continues to approve new fields and subfields. in recent years, many new fields and subfields have been introduced into marc 21 that could help users if the data they contain were exposed by library discovery interfaces. in this article, we will look at several of these, describing how we incorporated them into our consortial primo discovery layer and discussing the challenges we encountered. primo norm rules primo is a patron-facing discovery interface for library resources created by ex libris that can ingest and combine data from multiple sources. primo translates incoming data into standardized primo normalized xml (pnx) records. the pnx record is divided into sections, such as display, search, facets and links. each section includes multiple fields, both out-of-the-box fields, such as “display:title,” and locally-customizable fields such as “display:lds12” [1]. incoming data is transformed into the pnx format by primo normalization rules (or norm rules), which are maintained in the web-based primo back office [2]. unlike traditional library catalogs, which generally only allow libraries to turn specific marc fields and subfields on and off, and to have them displayed and indexed as-is, primo norm rules enable significant customization of incoming data. each primo norm rule consists of four main parts: source, conditions, transformations and action (figure 1). source defines what the rule takes as input. this could be a marc or dublin core field, a constant or another pnx field. a rule is only triggered if the input it is looking for exists in the source record. the second section contains conditions that have to be met before the rule will write anything to the pnx record. the third section contains transformations, or changes that are made to the source data before writing it to the pnx record. the options for conditions and transformations are hard-coded and must be selected from a dropdown list in the primo back office. the final section is labeled action and specifies how the output of the rule relates to the output of the other rules for a given pnx field. the output may be added as a new instance of the current pnx field or appended to an instance containing data written by previous rules. the action section also determines whether there can be only a single value for the pnx field or if multiple instances of the field are allowed. we will now provide some examples of how we used the primo norm rules to provide access to some marc fields that are not commonly used in library discovery interfaces. the fields and facets described in this article may all be seen in the sandbox interface at http://alliance-primo-sb.hosted.exlibrisgroup.com/primo_library/libweb/action/search.do?vid=uo_nrwg. figure 1. a primo norm rule in the primo back office interface showing the source, conditions, transformations and action. creator and audience demographics several new marc fields were created as a consequence of the library of congress (lc) developing a new vocabulary for genre and form terms. although library of congress subject headings (lcsh) were primarily designed to encompass topical subjects (what a resource is about), they have also traditionally been used to describe genre or form (what a resource is) in certain circumstances. in most cases, the same term is recorded in the same field (650: subject added entry-topical term) for both topical use and genre/form use. for the end user, it is usually not possible to distinguish between these two functions when searching. even when a different term is used (e.g., symphony for topic and symphonies for genre), the distinction is unlikely to be intuitive for a naïve user. in 2007 the lc began a project to create a new separate vocabulary for genre and form with the intention of phasing out the recording of genre/form information in 650 (subject added entry-topical term) and moving it to 655 (index term – genre/form), a field that is intended specifically for genre and form information.[3] in addition to enabling more precise searching, this change was meant to lead to data that is more suitable for use in faceted interfaces and for algorithmic processing. however, as the sample lcsh strings below show, the goal of implementing clean faceting faced a significant challenge. although the underlined terms below could be incorporated into this new vocabulary, something had to be done with the related information that was also part of the existing library of congress subject headings, but is neither genre/form information nor topical information. french poetry $y 20th century english drama $x women authors children’s films american literature $z new york (state) songs (high voice) with piano to account for this information, lc created some additional new vocabularies, including library of congress demographic group terms (lcdgt) to describe characteristics of creators of and audiences for resources. in conjunction with the development of lcdgt, two new fields were added to marc 21 to accommodate this information: 385 (audience characteristics) and 386 (creator/contributor characteristics). we implemented display, search and faceting of 386 (creator/contributor characteristics). figure 2 shows the display for a book of plays by english women that included an existing lcsh of “english drama–women authors.” figure 2. the lcsh “english drama–women authors” is split out to become genre/form “drama” + creator demographic group “women; english; britons” + original language “english” (the meaning of the term english in lcsh’s “english drama” is ambiguous; in this particular case it signifies both the original language of the dramas and the nationality of the playwrights. note the variety of constructions found in lcsh for this type of information, such as nigerian drama (english), english drama–irish authors, american drama, and hispanic american drama (spanish), which makes it quite complex for the average person to search by characteristics of resource creators). figure 3 shows an example facet list from a search for “autobiographies.” figure 3. creator demographic group (386) facets from a search for autobiographies. 386 (creator/contributor characteristics) is straightforward to display and facet, but there are many questions about best practices for choosing what data to put in this field. it’s also likely that there is a better way to label this field for the public. we also implemented display, search and faceting for 385 (audience characteristics), but this was more complex. audience information appears in several places in marc bibliographic records so we incorporated data from three different sources into a single primo display field: the audience code in the 008 fixed field (008/22), the recently-created 385 field (audience characteristics) and the audience note field (521). display of 385 (audience characteristics) is straightforward and the rule follows the pattern used in primo for displaying topical subject fields (figure 4). figure 4. primo norm rules that displays all the 385 (audience characteristics) subfield a’s (audience term) separated by “semicolon space” punctuation. (note that ^ signifies a space in primo’s norm rule expressions.) figure 5. audience facet results and display in primo. the 008 (fixed field data) is a little trickier. the meaning of bytes in 008 depends on a combination of position in the 008 and record type in the ldr. 008/22 is not used for target audience in all record formats. in the map format, this byte has a completely different meaning. if maps are not excluded from this primo norm rule, suddenly all the quadrangle maps will appear to be intended for an audience of preschool children! we therefore add conditions to exclude the record formats (maps, serials and mixed materials) that do not include positions for audience in 008 (figure 6). figure 6. first part of 008/22 (target audience) norm rule showing the condition that checks to make sure the ldr/06 (type of record) format does not equal maps using ex libris’ internal abbreviations for record types where mp = ldr/06 (type of record) e or f. we also added a condition to ignore the fixed field if the 385 field (audience characteristics) also exists. this is done to reduce redundancy in display and would be unnecessary if primo de-duped pnx display fields as it does facets. we preferred 385 (audience characteristics) because it supports more granular information, as well as multiple values. we then isolate the value of the audience fixed field, but, of course, we don’t want to display the raw value. the primo back office provides the ability to create what it calls mapping tables that enable a list of values to be transformed into a second list of values. we send the audience fixed field code to one of these mapping tables to turn it into a human-friendly value for display (figure 7). we use a single mapping table for display, search and faceting. it is important to not only map the coded values from the fixed field to textual terms for display purposes, but also for inclusion in the search index so that users who see a term in a record can search for others like it. because we also use the textual terms from the lcdgt vocabulary for faceting, we mapped the fixed field codes to that vocabulary to improve collocation. for example, the label for the value “c” in the marc format is “pre-adolescent.” however, “preadolescents” is a cross-reference for “preteens” in lcdgt so we used “preteens” as our label. figure 7. mapping table that changes 008/22 (target audience) fixed field values into human-readable strings. the sourcecode1 column contains the input and the targetcode column contains the output of the mapping table. commas are used to separate multiple output values, which are later separated by the norm rule using the “split field” transformation. finally, we display the 521 field (target audience note). this field includes indicator values that mark narrower meanings. we add prefixes to the note display based on these indicator values where applicable, as shown in figure 8 with 1st indicator 1 for interest age level. figure 8. primo norm rule that takes 521 (target audience note) with 1st indicator 1 as its source and prefixes the content of the note with the associated more-specific label “interest age level:” we add both the terms from the 385 (audience characteristics) and the terms mapped from the audience fixed field to an audience facet. however, because of the way primo weights facet values, the facet results are dominated by the fixed field terms, which far outnumber the uses of lcdgt, and it is hard to find examples with lcdgt. it’s also not clear how useful the “general,” “adult” and “specialized” audience fixed field values are as facets since they are much less consistently used than the ones for juvenile materials. medium of performance medium of performance is another type of information that is commonly combined with genre in lcsh and now needs to be accommodated somewhere else. lc and the music library association have developed a new vocabulary called library of congress medium of performance thesaurus for music or lcmpt and the 382 (medium of performance) field was added to marc 21 as a place to record this information. 382 (medium of performance) is designed to support more complex details about medium of performance than lcsh and includes a large number of subfields. it was also intended to be useful in a faceted environment. looking at the raw 382 (medium of performance) field below, it is clear that some intervention is needed to make this new data usefully displayable and searchable in primo. 382 01 $a viola $n 1 $p clarinet $n 1 $a piano $n 1 $s 2 $2 lcmpt although the primo norm rules provide powerful tools for manipulating marc data, there are some limitations on what is possible, especially for complex fields with many subfields. there are two approaches that can be used to deal with fields that include more than one type of subfield. the first approach is to process each type of subfield separately in its own rule and then combine the results. this allows you to treat specific subfields differently, but it doesn’t preserve the order of the subfields. that is, you could get all the subfield a’s followed by all the subfield n’s. the second approach is to process the different types of subfields with a single rule. in this case, you preserve the order of the data, but you can’t do something different to subfield a from what you’re doing to subfield n. we used a combination of both approaches to turn 382 (medium of performance) into something eye-readable. we will now walk through how we got from the raw marc field shown above to the following human-readable display. duet: viola (1); piano (1) (alternate instrumentation: clarinet) the first rule that applies takes subfield s (total number of performers), a non-repeatable subfield, as input. subfield s (total number of performers) should contain a number representing the total number of performers. we use a mapping table to change numbers into words for display (figure 9). from our example field, the rule takes the input of 2 and maps it to the word duet. figure 9. primo norm rule that sends the contents of 382 subfield s (total number of performers) to the mapping table “no_musical_parts” and outputs the results of the mapping. the second rule takes its input from multiple subfields: a (medium of performance), b (soloist), e (number of ensembles of the same type) and n (number of performers of the same medium) (figure 10). the starting input for this rule in our example is the string “viola 1 1 piano 1”, which retains the order of the data, but does not identify the source subfield for any of the data. figure 10. first part of a rule that takes subfields a (medium of performance), b (soloist), e (number of ensembles of the same type) and n (number of performers of the same medium) of a single instance of 382 (medium of performance) as input. we then use a transformation to define a subfield delimiter (first line in figure 11). we use a semicolon and space (denoted by ^ in primo expressions), which results in each subfield being separated by a semicolon and space. in our example, this gives us “viola; 1; 1; piano; 1”. we then take advantage of the fact that we can identify numbers separately from instrument names to remove the semicolon preceding any number and instead put the number in parentheses. this is a situation that is crying out for a regular expression. there is a transformation in primo’s documentation that looks like it would be helpful here, but unfortunately, that function is broken. so we explicitly wrote a replace transformation for the numbers 1-9 (figure 11). this will fail if there are ten or more performers coded as playing the same instrument, but that is an uncommon situation. this gives us “viola (1) (1); piano (1)” figure 11. transformations that separate each subfield with a semicolon followed by a space and then find semicolon, space, number and replace it with space (number). in primo norm rule transformations, the string to find and the string to replace are separated by “@@” the “^” is used to represent a space in the replacement string. there is another problem, though, which is that subfield n (number of performers of the same medium) is used both for the number of performers in the main medium of performance statement and also after doubling and alternate instrumentation as shown in our example with the alternate clarinet part in subfield p (alternative medium of performance). 382 01 $a viola $n 1 $p clarinet $n 1 $a piano $n 1 $s 2 $2 lcmpt we don’t want to mix up the main medium of performance with the alternate or doubling instrumentation. for this rule, we only want the subfield n’s (number of performers of the same medium) that correspond to the default instrumentation. however, we have no way to distinguish these different meanings of subfield n (number of performers of the same medium), which is causing the multiple appearance of the number 1 after viola. we undertake another series of find and replace actions to remove extra numbers (figure 12). because we cannot do this conditionally and each transformation only removes a single instance of an extra number, we tried to include enough transformations to account for the vast majority of situations. however, it is still possible to find medium of performance statements in our catalog with extra numbers if you know where to look. figure 12. transformations that find ) space (number) and replace it with just ). this removes the extra numbers. finally, the merge action for this rule prepends a colon and space to the instrumentation if there is preceding text, such as the “duet” written by the previous rule in our example (figure 13). figure 13. data is merged onto an existing instance of a pnx field and is preceded by a colon delimiter, with a space after the delimiter. the last thing to deal with in this example is the alternative instrumentation in subfield p (alternative medium of performance). given our choice between preserving the order of the data and thus associating clarinet with the instrument that it is an alternative for and being able to process the subfield separately, we have chosen the latter. this enables us to separately label the alternative instrumentation. in 2015 the music oclc users group (moug) commissioned a report that found that music librarians preferred a display that looks like viola or clarinet (1); piano (1) where subfield p (alternative medium of performance) is preceded by the word “or” and directly follows the instrument for which it is an alternative [4]. our display does not preserve the relationship between the viola and clarinet. viola (1); piano (1) (alternate instrumentation: clarinet) in order to preface the clarinet in subfield p (alternative medium of performance) with some indication that it is an alternative, we have to process it in a separate rule (figure 14). we therefore lose the ability to retain the original subfield order. figure 14. rule adds “(alternate instrumentation: “ label to the beginning of 382 (medium of performance) subfield p (alternative medium of performance) data, and closes the data with a “)”. multiple instance of subfield p (alternative medium of performance) are separated by a semicolon and space. for the most part, we have just added the 382 (medium of performance) subfields to the general search index without any intervention. however, we have included rules to index the words, such as duet and alternate instrumentation, that we generated for display. it turns out that at least some users do want to search by words like “solo.” currently, we have implemented three facets based on medium of performance in the 382 field. the first one is a facet for instrument, voice or ensemble type. this includes everything from subfield a (medium of performance) and subfield b (soloist). the terms from subfield b (soloist) are double-posted with the qualifier solo to help users find pieces that have solo parts for their instruments (figure 15). this seems pretty straightforward, but there are a couple ways this facet behaves that are likely to clash with user expectations or preferences. figure 15. rules that (1) add instrument/voice/ensemble type and (2) double-post soloist data with (solo) qualifier. first of all, users probably want to find pieces that include all the instruments or voices that they select. that is, they want to do a boolean and search. if users select terms consecutively from the facet list in primo, they will get an and search. however, if they select multiple terms at once using primo’s checkboxes and the “apply filter” button (figure 16), primo will perform an or search. this makes sense from the standpoint that for most long lists, an or search would be preferred, and it’s desirable for all the facet lists to predictably exhibit the same behavior. but it probably isn’t what users want from this particular facet most of the time. figure 16. primo check list of facet values. another important thing to notice is that the facets are working against bibliographic records for the whole resource being described so violin and piano just have to be somewhere in the record. the terms don’t have to be referring to the same piece as in the example in figure 17, which includes a piano concerto and a violin concerto. figure 17. facet values are related to different pieces that are part of a compilation. in addition to medium of performance, we also implemented a facet for total number of performers if that is given (figure 18). it doesn’t attempt to count ensembles, but just lists the presence of an ensemble. some of the larger numbers of individual performers should possibly be double-posted under ensemble, but we’re not doing that right now. again, the facets are working against the bibliographic record so if a user picks 1 performer and saxophone, there’s no guarantee that all the results will include a piece for solo sax. however, the results are not as misleading as they might be thanks to relevancy ranking and the fact that users can only ever see the top twenty values in a given facet. figure 18. facet results for number of performers. in order to compensate somewhat for that weakness, we have also created a medium of performance statement facet (figure 19), which allows users to specify exactly what combination of instruments or voices they are looking for. figure 19. medium of performance statement facet values. there are a couple places where this facet doesn’t provide the collocation that users will expect. one is due to the fact that the marc format allows you to omit the 1 for number of performers for any instrument or voice that has only a single performer. since the facets are looking at literal text strings, they see piano and piano followed by 1 as two different things. (starred lines in figure 19) this is an old screenshot and we have since been able to resolve this issue by supplying a 1 following any medium of performance that doesn’t have a number of performers specified. this improves collocation in most cases, but does not account for situations where counting performers is not straightforward and conflates some cases where the number of performers is in fact unknown or unspecified. for instance, when a score includes multiple percussion instruments, in many cases, it is possible for a single performer to play more than one of the instruments [5]. a thornier issue is that the instruments or voices are not listed in a standard order, which again results in different strings that don’t collocate. the first and third lines in figure 20 both contain the same instrumentation, just in a different order. this is unlike lcsh headings for medium of performance, where there are rules for ordering the instruments so that correctly-constructed headings do collocate. as far as we can tell, reordering the instruments is beyond the capability of primo’s data manipulation tools so the only solution would be to do some massaging of the data before sending it to primo. figure 20. lack of collocation for medium of performance statement facet values. country of production one of the challenges of working with marc data is dealing with inconsistencies. definitions of fields, as well as practices and rules for entering data have changed over time. incorrect information will exist in any large dataset, but lack of effective validation features in most software for editing marc records leads to a higher error rate than would otherwise exist. the rules we wrote for dealing with the 257 field (country of producing entity soon to be renamed area of producing entity) [6] are a good example of compensating for this problem. the 257 field was originally defined as country of producing entity for archival films. it was designed to support the data element for country of production in archival moving image materials: a cataloging manual (amim) [7]. amim defines country of production as the country or countries where the headquarters of a film’s production company or companies are located, or the location of an individual producer if there is no production company. this was a purely descriptive element and contained values such as 257 $au.s. 257 $au.s. ; france ; west germany. 257 $a[s.l.]. prior to 2009, a controlled vocabulary was not used and the united states was commonly identified by the abbreviated form “u.s.”. subfield a (country of producing entity) was not repeatable and multiple countries were separated by the prescribed punctuation “space-semicolon-space.” the latin abbreviation s.l. (sine loco) was used if the location of the producing entity was unknown. in 2009, the field was renamed country of producing entity and the limitations to archival cataloging and to moving images were both dropped, although the field has not seen much uptake outside of moving image cataloging. it was also revised to support controlled vocabulary. both the field as a whole and subfield a (country of producing entity) were made repeatable and subfield 2 (source) was added as a place to record a code for the vocabulary from which the country name(s) were taken. catalogers primarily use the lc name authority file (naf) as a source for country names in 257 (country of producing entity) so the first two examples given above would appear as follows in current, non-archival cataloging. 257 $aunited states$2naf 257 $aunited states$afrance$agermany (west)$2naf we are not aware of any method being used to record an unknown location with a controlled vocabulary. the marc format still supports the older amim-style usage so our norm rules have to be prepared to deal with both forms of data. in addition, in the wild there exist examples that incorrectly mix both of these forms of recording data. using the punctuation from amim and the repeated subfield and source vocabulary in $2 from the controlled vocabulary method: 257 $aitaly ;$afrance$2naf using an indication of controlled vocabulary in $2, but combining multiple locations in a single $a and separating them with punctuation: 257 $aitaly ; france$2naf identifying the variations that exist in any real dataset and figuring out how to deal with them is an iterative process. we eventually settled on the sequence of transformations shown in figure 21 to achieve a consistent display. figure 21. transformations for 257 country of production. table 1 walks through the effects of the transformations on each of the variants described above. the fourth variant is omitted as its $a (country of producing entity) differs from the first variant only because it ends in a period, which is removed in the sixth transformation. yellow highlighting is used to show the effect of each transformation. in the table “_” indicates the addition of a space and “ ” indicates that a character was removed and replaced by nothing. in step 3, the transformation “replace spaces by string” is a necessary intermediate step because the general transformation “replace string by string” will not take a space as part of its input to find. note that in primo transformations “^” denotes a space and “@@” is used to separate the string to find and the string to replace it with. table 1. transformation impact on source data variants source = 257$a $afrance ; italy. $afrance$aitaly $afrance ;$aitaly. 1 $a ? “; “ france ; italy. france;_italy france ;;_italy. 2 “;;” ? “;” france ; italy. france; italy france ; italy. 3 “ “ ? “++” france++;++italy. france;++italy france++;++italy. 4 “++;” ? “;” france;++italy. france;++italy france;++italy. 5 “++” ? “ “ france;_italy. france;_italy france;_italy. 6 remove “.” from end france; italy  france; italy france; italy  it is also necessary to think about consistency for facets. because primo only displays a limited number of values for dynamic facets (primo facets where the range of possible values is not limited to a predetermined list), we wanted to minimize redundancy in the results. we therefore only draw the facet values from 257 (country of production) fields that include a subfield 2 (source) with “naf” indicating that the country names are in the form used in the lc name authority file (figure 22). primo provides transformations that allow us to split each subfield a into its own pnx facet field and to remove trailing semicolons and periods. trailing characters in all but the final subfield have to be removed before using the “remove characters from the end” transformation as this transformation only acts on the final characters of the field as a whole. although we could additionally split subfield a on internal semicolons, we have currently chosen not to do that. instead we treat multiple places recorded in a single subfield a (country of producing entity) as metadata errors to be cleaned up as they are identified. figure 22. primo norm rules for the 257 (country of producing entity) facet. note the condition that checks to make sure the 257 field currently being processed includes subfield 2 with a source code for the lc name authority file and the transformations to remove ending punctuation and separate each instance of the repeatable subfield a (country of producing entity) into its own pnx facet field. composers, directors and performers the marc format has long had subfields that can be used to record the relationship between access points for names of agents and the resource being described. roles such as editor and composer can be recorded either as relator terms (usually in subfield e) or as relator codes (in subfield 4). the emphasis on relationship designators in the new cataloging standard resource description and access (rda), which are recorded in subfield e as relator terms in marc, has driven a resurgence of interest in these relationships. the group that maintains the primo norm rules for the orbis cascade alliance has been mostly reactive, focusing on requests from member libraries. one of the requests we received was to make use of relationship designators, which indicate the relationship between a person or other agent and a resource, for search and faceting. initially, it was not clear how we might effectively provide access to relationship designators via facets. it is possible to just add the relationship designators to the end of names in the creator and contributor facet. however, this has several significant drawbacks, as shown in the hypothetical example below. clint eastwood, 1930(116) clint eastwood, 1930actor (45) clint eastwood, 1930director (23) clint eastwood, 1930director, actor (26) first of all, many names in our dataset are not associated with relationship designators. many of the 116 unqualified instances of clint eastwood probably should be marked as actors or directors, leading to a lack of both precision and recall. secondly, some names are associated with multiple relationship designators. we do not think it is possible to separate these with primo norm rules. in the example above, there are actually 71 results for eastwood as actor, but they are split between the ones marked only as actor and those marked as both actor and director. in order to get all the resources for which eastwood is an actor, a user would have to select both facets, which is not intuitive. finally, because primo only displays a limited number of values for this facet, splitting names by role can lead to a significant reduction in the variety of names the user can pick from. this problem is exacerbated by the fact that records can potentially contain role information in varying degrees of specificity (actor vs. performer, director vs. film director), from multiple vocabularies (rda’s “director” vs. amim’s “direction”) and either as terms or as codes ($e director vs. $4 drt). it is possible, but complex, to reduce some of this variation. instead of adding relationship designators to names in the creator and contributor facet, it is also possible to create separate facets for specific roles (see example at http://sinenomine.co.uk/curwen/?ix0=name&id0[]=119&roleback=&letter=s) [8]. this is the approach we took. it is not practical to make facets for all possible roles. the marc code list for relators alone includes 268 valid terms [9]. even if facets were made for all of those, usability would remain a big challenge and standards such as rda in its instruction 18.15.1.3 (recording relationship designator) allow the use of “another concise term” if none of the relationship designators in the rda vocabulary is “appropriate or sufficiently specific.” [10] we selected three roles for experimentation: composer, director and performer. we selected these for two reasons. first, we believe there are use cases where patrons would benefit from being able to search and browse for names in these categories separately. secondly, role information is more likely to be present in records for music and moving images than in book records. although $e (relator term) was commonly used to identify non-author roles, such as editor or translator, in older pre-aacr2 records, it was rarely used during the period when aacr2 was the predominant cataloging standard. most libraries followed the library of congress rule interpretation that said not to record functions except for illustrators of children’s books. with the implementation of the rda cataloging standard in 2013, the use of relationship information in name access points became far more widespread and consistent, but the legacy of decades of cost saving by not adding this information remains. for composers and directors, we currently look only at personal names in 100 (main entry-personal name) and 700 (added entry – personal name). for performers, we look at both personal names and at corporate names in 110 (main entry – corporate name) and 710 (added entry – corporate name) to pick up performing groups. the rules take the name subfields that we will display in the facet as input. we then use a series of conditions about data in the same instance of the marc field to flag the name if it is associated with the role that we’re looking for. for example, for the composer facet we look for the string “composer” in $e (relator term) or “cmp” in $4 (relator code) (figure 23). figure 23. primo norm rule for composer facet writes name from 100 field (main entry-personal name) if that 100 field includes a relevant relator code or term. when working with marc data, it is sometimes possible to compensate for missing data by making inferences from other data in the record. in the case of the “composer” relationship, we are able to increase recall in this way. for scores, we assume that all 100 fields (main entry-personal name) and names from analytic name-title access points in 700 (added entry – personal name) are composers. there are occasional older records for scores where the 100 field (main entry-personal name) contains an editor or compiler, but the increase in recall from assuming that 100 (main entry-personal name) contains a composer far outweighs the loss of accuracy. for musical sound recordings, we only inferred that the 100 (main entry-personal name) represents the composer if a uniform title was also present in 240 (figure 24). aacr2 cataloging permits performers or conductors to be recorded in 100 (main entry-personal name) for musical recordings in some cases. however, if 240 (uniform title) is present, it almost certainly means that the name in 100 (main entry-personal name) is the composer of that title. for recordings, we again included all the names from name-title entries in 700 (added entry – personal name) with 2nd indicator 2 (analytic added entries). figure 24. primo norm rule that takes the subfields from 100 (main entry-personal name) that we intend to display in the composer facet. features two conditions that must both be true: (1) 240 (uniform title) exists and (2) ldr/06 (type of record) equals “j”(musical sound recording). unlike composer, which can be accurately identified by the presence of the word composer in the relator term in subfield e, the “director” relationship presented a challenge due to the multiple contexts in which the word “director” appears. we both had to account for the various strings that identify directors (“director,” “direction,” “film director,”) and exclude strings that include the word director, but specify a different role (“director of photography,” “art director”). to identify potential strings for matching, we reviewed the marc relator code list, the list of rda relationship designators and the terms used in amim. marc relator codes are all three-letter strings and are recorded in subfield 4 (relator code). there may be multiple instances of subfield 4 (relator code) in a single 100 (main entry-personal name) or 700 (added entry – personal name) field as a person may have performed multiple functions. the primo rule takes all the subfield 4’s (relator code) in a single field as input as a single string with a space separating each subfield. so “$4drt$4pro$4act” is turned into “drt pro act” when subfield 4 (relator code) is the source for a rule or condition. the marc relator code strings are unique so it is sufficient to test for the presence of “drt” anywhere in the input from subfield 4 (relator code) in order to decide to write the associated name in that field to the director facet. this may become more complicated in the future as the definition of subfield 4 (relator code) has recently been expanded to allow the use of uris [11]. this greatly expands the number of potential values for subfield 4 (relator code), which means that a search for a marc relator code such as “act” (actor) could pick up false drops where, for example, the string “act” is part of the domain name of a uri. relator terms are usually recorded in subfield e. 111 and 711 (conference names) are exceptions. here relator terms are recorded in subfield j because a subfield for relator terms for conference names was only defined recently in response to rda and subfield e was already in use. for the most part, our primo norm rules for relator terms, like the rules for relator codes, merely check for the presence of a string, such as “television director.” like subfield 4 (relator code), subfield e (relator terms) can be repeated. this creates a difficulty for identifying the plain string “director.” we want to pick up director when it is one of a number of relator terms as in “$edirector,$eproducer,$eactor,” but we don’t want to get “$edirector of photography.” to identify “director” when it is the only term in a specific subfield e, we have to isolate the strings in each individual subfield. to do this, after removing any periods or commas at the end of a subfield, we change the subfield delimiters to “+++” which gives “director+++producer+++actor”. we then add “+++” to the beginning and end of the whole string and look for “+++director+++” (figure 25). if no matching terms or codes are present, we did not attempt to infer a directorial relationship. figure 25. condition that identifies plain string “director.” we tried to include as many types of performers as possible in the “performer” facet. rda helpfully subarranges many types of performers under the performer relationship designator. we browsed the list of marc relator codes for equivalents to the rda terms and anything else that implied performance. this results in a rule with a long list of “or” conditions that test for the presence of terms and codes for roles such as “performer”, “actor”, and “musician” (figure 26). if no matching terms or codes were present, we did not attempt to infer performer relationships. figure 26. norm rule for performer facet, showing conditions for “actor” and “dancer” relator codes. challenges as with any innovation that attempts to capitalize on the granularity of marc coding, bringing these new display fields and facets to our users via the discovery interface provided challenges large and small. some of the problems we encountered are areas of concern in any discovery tool. others were specific to our use of ex libris’s primo discovery interface. one general challenge is something we’ve dubbed the “chicken-or-egg” question. although no large-scale dataset is likely to support perfect recall, new fields that aren’t widely used will suffer from significantly incomplete results when used as faceting values. this causes users to see them as inaccurate, and degrades trust in the facet’s accuracy, leading to disuse. this situation makes institutions hesitant to add facets or develop other functionality for new fields in their public search interfaces – which makes their cataloging shops hesitant to spend time encoding them. this results in the new fields not being widely used, and the cycle continues. to address this problem, it’s possible to engage in retrospective conversion projects. the american library association’s subject access committee is optimistic about the potential for “retrospective implementation of faceted vocabulary terms using algorithms developed, vetted, and tested by expert communities.” [12]. in fact, the music library association has partnered with gary strawn to develop a tool that automatically adds marc fields for non-topical facets, such as medium of performance, based on lcsh subject headings in individual bibliographic records for music in oclc’s connexion metadata editor. [13][14], (files available at http://files.library.northwestern.edu/public/music382/). the tool is currently intended to be paired with manual review by an informed user before updating the record. caution should be exercised here, because such projects sometimes use incomplete or inaccurate algorithms to batch process records, which can result in badly-converted data. retrospective conversion is limited to information that was previously recorded in an identifiable way and will also carry over inaccuracies from the existing data. retrospectively-converted data will generally be less accurate, less complete and less granular than data added with human review. our consortial catalog subscribes to updates to oclc worldcat master records so we hope that the combined effort of retrospective metadata enhancement and the wider community of catalogers will help us get beyond the “chicken-or-egg” dilemma. another problem in most discovery systems is search specificity in records for aggregates like music compilations on cd and literary anthologies. for instance, in the case of our “medium of performance” facet, users limiting by multiple instruments will get results with those instruments present in the record – but that doesn’t mean the instruments all appear in the same piece. we’ve tried to improve this situation by providing the medium of performance statement facet, which groups the instrumentation for each piece. however, if the user is looking for a specific medium of performance statement in a piece by a specific composer or in a specific key, they’re back to the aggregate problem (figure 27). figure 27. aggregate records can defy user expectations when faceting. another challenge is the question of discovery “real estate”. one reason that many libraries in our consortium haven’t implemented new facets is that there are usability challenges with a long list of facets. many of these fall “below the fold,” making them easy for users to miss. amazon and other retails sites often compensate for this problem by showing a shorter list of facets that are contextually linked to the user’s search. one way to achieve this is to build a material-specific sub-interface in your library’s discovery interface. this raises its own crop of new challenges, however. how will you market the sub-interface to bring it to users’ attention? how easy is it to switch between the general search interface and the material-specific one? will users get confused about which interface they’re using? [15] other challenges we came up against were specific to our discovery layer, ex libris’s primo. in theory, facets could be used not only for narrowing searches, but also for open-ended browsing and exploring of a library’s collection [16]. however, primo does not support true exploratory search [17] due to processing limitations. users must enter a search term before they are shown any facets. primo also limits the number of facet values that are displayed to the end user in most cases. primo supports two types of facets. static facets are based on enumerated lists where all the possible values are determined in advance and have complete recall. in the case of static facets, the users are presented with all the values found in their complete result set and selecting a facet value will retrieve all the records in the database with that value. however, static facets are limited to relatively short lists of values. we use static facets for things like resource types and the marc language codes. the marc language code list contains around 500 values and it’s not clear how many more values could be supported by a static facet. primo also supports dynamic facets, which use whatever terms occur in the data. in the out-of-the-box primo configuration, dynamic facets select the top 20 terms that occur in that pnx facet field in the top 200 ranked records in the user’s result set and display only those twenty terms as options. each facet value is qualified by the number of records that it will retrieve if selected. this total matches against only the first 50,000 records in the result set so it will be incomplete if the user has done a broad search. the number of terms shown and the number of records from which those terms are drawn can be adjusted, but there is a trade-off between improved recall and system performance. [18] in primo, customers can activate a database of article-level discovery data called the primo central index (pci). while access to this data greatly improves discovery for specific scholarly resources, the pci metadata does not go through the same normalization process as the metadata in our local bibliographic database (ex libris’s alma). the result is that the customized facets and display fields we’ve built for primo cannot incorporate the data from the pci. only “out of the box” facets and fields can make use of the pci metadata. we also have no control over the values that are written to the pci pnx records. we would like to experiment with building material-specific sub-interfaces for music or videos, but we are unable to find a way to incorporate the pci resources. in fact, we cannot even create an accurate resource type facet for online audio and online video because the pci records contain a value of “media” for both audio and video. the two types of resources are identified separately for display so the necessary metadata seems to be there, but we are forced to create a single “eaudio & evideo” resource type facet to align our local materials with the records coming from the pci. we’ve also been frustrated in our work by primo’s interface for building customized fields and facets. as shown in this article, primo uses normalization rules to translate marc metadata into xml that can be used by the primo front end. like most saas application configurations, these rules are managed via a browser-based gui with hard-coded elements. that gives us a fixed set of logical conditions and data transformations that we can use to find specific marc data and turn it into user-friendly elements in primo. apart from this inflexibility, some specific interface problems we’ve encountered are broken transformations that require multi-step workarounds, long page load times, and limitations on copying single rules from one rule set to another. one final problem of note in our specific case is that we would like to create marc field-specific search and browse indexes for primo. however, this is not currently supported. we can build “good enough” functionality in this regard by customizing “lateral linking” (hyperlinked) fields in primo, but we very quickly come up against some limitations here as well. first, the limit in primo is 9 locally-defined linking fields. recent developments have seen an increase in the number of available “out of the box” linking fields, but not for locally-defined fields. furthermore, these linking fields provide functionality that’s similar to faceting, but not equivalent. a user clicking on the linking field executes a new search rather than limiting their current search results. whether this behavior is acceptable depends on local user expectations and the search context. conclusion and looking forward we’ve been able to use many of the new fields and subfields introduced into marc 21 to add user-friendly features to our consortium’s faceted discovery interface. while some of those developments were straightforward, others proved more challenging to achieve. some of those challenges were due to the nuances of marc and the inconsistencies of legacy data, while others were due to the nuances of our discovery tool (and of discovery systems in general). despite these challenges, we feel strongly that our work has helped users across our consortium find the resources they need to advance their scholarly work. as time goes on, and the use of fuller and more structured metadata increases, it is critical for libraries to explore new ways of making that metadata available for their users in human-readable displays and user-friendly elements like facets and search indexes. while these developments can present many challenges, we also feel confident that discovery tools will continue to evolve, making the process of customization easier and more sustainable as well. references [1] ex libris. 2018. the pnx record. in: ex libris knowledge center. [internet][jerusalem]: ex libris; [cited 2018 june 5]. available from: https://knowledge.exlibrisgroup.com/primo/product_documentation/technical_guide/010the_pnx_record [2] ex libris. 2018. working with normalization rules. in: ex libris knowledge center. [internet][jerusalem]: ex libris; [cited 2018 june 5]. available from: https://knowledge.exlibrisgroup.com/primo/product_documentation/technical_guide/020working_with_normalization_rules [3] library of congress, policy and standards division. 2017. introduction. in library of congress genre/form terms for library and archival materials (lcgft) [internet]. washington, d.c.: the library of congress. p. 1-6. [cited 2018 june 5]. available from: https://www.loc.gov/aba/publications/freelcgft/2017%20lcgft%20intro.pdf [4] belford r. 2015. worldcat discovery display preferences for medium of performance. columbus, oh: music oclc users group. available from: http://musicoclcusers.org/wp-content/uploads/wcd_medium_report_201504291.pdf [5] lee d. 2017. numbers, instruments and hands: the impact of faceted analytical theory on classifying music ensembles. knowledge organization [internet]. [cited 2018 june 5]; 44(6):405-415. available from: http://openaccess.city.ac.uk/18645/ [6] library of congress network development and marc standards office. 2017. marc proposal no. 2017-10, rename and broaden definition of field 257 in the marc 21 bibliographic format. marc 21 proposals [internet]. [cited 2018 june 5]. available from http://www.loc.gov/marc/mac/2017/2017-10.html. [7] library of congress. archival moving image materials: a cataloging manual (2nd edition) [internet]. washington, d.c.: the library of congress; 2000 [cited 2018 june 7]. available from: https://archive.org/details/amim2 or via cataloger’s desktop. [8] camden b. 2013. relationship designators and facets: vcat@penn in: program for cooperative cataloging participants meeting; 2013 june 27; orlando. available from: https://www.slideshare.net/bethcamden/relationship-designators-and-facets-vcatpenn [9] library of congress network development and marc standards office. 2018. relator code and term list — term sequence: marc 21 source codes. marc 21 proposals [internet]. [cited 2018 june 5]. available from http://www.loc.gov/marc/relators/relaterm.html. [10] rda toolkit [internet]. [updated 2017 april]. american library association, canadian federation of library associations, and cilip: chartered institute of library and information professionals; [cited 2018 june 7]. available from: http://access.rdatoolkit.org/ [11] library of congress network development and marc standards office. 2017. marc proposal no. 2017-010, redefining subfield $4 to encompass uris for relationships in the marc 21 authority and bibliographic formats. marc 21 proposals [internet]. [cited 2018 june 5]. available from https://www.loc.gov/marc/mac/2017/2017-01.html. [12] working group on full implementation of library of congress faceted vocabularies. 2017. a brave new (faceted) world: towards full implementation of library of congress faceted vocabularies. chicago (il): american library association, association for library collections and technical services, cataloging and metadata management section, subject analysis committee, subcommittee on genre/form implementation. available from: http://www.loc.gov/aba/pcc/documents/poco-2017/bravenewfacetedworld-170713.pdf [13] mullin c. 2018. retrospective implementation of faceted vocabularies for music. middleton (wi): music library association, vocabularies subcommittee. available from: http://works.bepress.com/casey-mullin/12/ [14] mullin c, strawn g. 2018. deriving faceted terms from library of congress subject headings for music: challenges and possibilities. in: music library association annual meeting; 2018 jan 31 – feb 4; portland, or. available from: https://hcommons.org/deposits/item/hc:19023/ [15] music library association. music discovery requirements [internet]. middleton (wi): music library association; 2017 [cited 2018 june 5]. available from: https://www.musiclibraryassoc.org/mpage/mdr_ivf [16] mcgrath k, kules b, fitzpatrick c. 2011. frbr and facets provide flexible, work-centric access to items in library collections. proceedings of the 11th annual international acm/ieee joint conference on digital libraries, p.49-52. available from: https://doi.org/10.1145/1998076.1998085 [17] exploratory search. 2018. in wikipedia. [internet]: wikimedia foundation, inc.; [cited 2018 june 5]. available from: https://en.wikipedia.org/wiki/exploratory_search [18] ex libris. 2018. facets. in: ex libris knowledge center. [internet][jerusalem]: ex libris; [cited 2018 june 5]. available from: https://knowledge.exlibrisgroup.com/primo/product_documentation/060back_office_guide/100facets suggested reading library of congress information page for lcdgt http://www.loc.gov/aba/publications/freelcdgt/freelcdgt.html mcgrath, kelley. getting more out of marc for music and movies with primo: strategies for display, search and faceting (2017 olac conference poster) http://pages.uoregon.edu/kelleym/publications/primo&marcforvideo&musicposter.pdf mcgrath, kelley. using marc facets for music with primo: strategies and challenges (ala midwinter 2018 presentation) http://pages.uoregon.edu/kelleym/publications/mcgrath_using_marc_facets_for_music_with_primo.pptx mcgrath, kelley and lesley lowery. getting more out of marc with primo: strategies for display, search and faceting (2018 eluna presentation) http://pages.uoregon.edu/kelleym/publications/getting_more_out_of_marc_with_primo-eluna2018.pptx music library association music discovery requirements (2017) http://www.musiclibraryassoc.org/mpage/mdr_ia music library association lcmpt best practices (version 1.3, january 2018) https://c.ymcdn.com/sites/www.musiclibraryassoc.org/resource/resmgr/bcc_resources/bpsforusinglcmpt.pdf music library association lcgft for music best practices (version 1.1, january 2018) https://c.ymcdn.com/sites/www.musiclibraryassoc.org/resource/resmgr/bcc_resources/bpsforusinglcgft_music.pdf oclc’s longitudinal statistics on field usage http://experimental.worldcat.org/marcusage/ the university of oregon’s primo sandbox for norm rules testing at http://alliance-primo-sb.hosted.exlibrisgroup.com/primo_library/libweb/action/search.do?vid=uo_nrwg about the authors kelley mcgrath (kelleym@uoregon.edu) is metadata management librarian at the university of oregon libraries. she is an experienced media cataloger and has been active in olac (online audiovisual catalogers) for many years. she is a member of the orbis cascade alliance primo norm rules working group/standing group. lesley lowery (llowery@orbiscascade.org) is the network zone manager at the orbis cascade alliance. until this year she worked at western washington university, where she served as the shared ils administrator and prior to that worked as a cataloger. she is a former member and chair of the orbis cascade alliance primo norm rules working group. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – how to build a computer availability map mission editorial committee process and structure code4lib issue 12, 2010-12-21 how to build a computer availability map most libraries house one or more computer labs. wouldn’t it be nice to be able to let your patrons view how many and what type of computers are available at any given time? well, now you can. follow along in this tutorial that takes you through the stages of implementing a real-time computer availability map that works for a mobile and full website. the complete code package is provided under the gpl v3 license, and is available at: http://github.com/griggsk/availability-map. by kim griggs computer availability maps are popping up all over university and library websites. some libraries have created the maps in-house (see appendix a), while others have used a vendor like labstats (see appendix b). i became interested in implementing a computer availability map when i attended a presentation by ncsu libraries [1] where they listed their computer availability map as a top-viewed page on their mobile site. since a real-time availability map supported the mobile context of a user on the go, i proposed to the oregon state university libraries (osul) mobile team that we add an online availability map of our computer lab to the mobile site. the result was a mobile computer availability map (http://m.library.oregonstate.edu/computers/) implemented with ruby on rails and scripts embedded in the computer’s build to track and display the availability status of the computers in the library’s learning commons. the map was released first to the mobile site and then was ported to osul’s drupal website (http://osulibrary.oregonstate.edu/computers/). both versions get high use, especially around mid-terms, finals and the beginning of term. over the last year, the map has been the second-most-viewed page on the mobile site and in the top 20 most-viewed pages on the main site. since releasing the map i have received many compliments and requests from the library community to share how i implemented it. computer availability map requirements the following is a list of requirements the osul mobile team developed for our computer availability map. your own requirements may vary. requirement 1: the map shall work on multiple screen sizes to enable us to offer the computer availability map to our patrons on multiple devices, the map needs to be easily extendable to work on both the mobile site (300 px) and the full website (600 px) as well as the large lcd (900px) in the computer lab. this requires 3 maps with dimensions that can easily be scaled and code that can handle the multiple dimensions. requirement 2: the map shall be accessible the map needs to be accessible to all our users. this requires paying attention to color use in icons, providing a text equivalent, standards compliant code and accessible mark-up. requirement 3: the map shall display a realistic view of the computer lab to enable patrons to easily understand the map and be able to locate available computers in the lab, the map needs to be a realistic view of the lab with location-based pointers. this requires a digital image of the computer lab. requirement 4: the map shall display computers in use and not in use the map needs to clearly display the status (available or unavailable) of the computers in the lab. this requires a database to store the location, an identifier for every computer, and scripts that are triggered when a user logs on or off. this also requires icons to identify the status. requirement 5: the map shall display which type of computer is available we also want to capture what type of computers are in the lab, so the map needs to clearly display the type (pc or mac) visually. this requires a database to store the type of every computer. this also requires icons to identify the types. requirement 6: the information on the page and in the map shall be updated in real-time to keep the availability status up-to-date the map needs to be updated in real-time with an acceptable lag time of 5 min. this requires the database to be queried automatically as well as manually. system design & architecture the computer availability map application i am going to show you how to build is a centralized logging system with a standard lamp stack. a mysql database is the backbone of the app and contains data about the computers in the lab. each computer stored in the database contains a unique identifier, such as a name or ip address; an x, y location that relates them to their physical space in the lab; and the type of computer (win/mac). embedded in each computer’s login and logout scripts are perl files that send http requests to a php script to update the individual computer status when a user logs in or out in real-time. a standard lamp application and hand-drawn images of the lab provides the user interface of the computer availability map customized for different screen sizes. figure 1. system design & architecture note: osul mobile team programmers decided to use this architecture because of our current development environment and expertise. implement the system building the availability map involves 6 steps. first you need to create the framework of the system and get the computers and the database talking to each other. then you are going to create a map and icon images. finally, you are going to build the user interface and finish with scaling the map for multiple screen sizes. step 1: create the database note: i am assuming you have access to a mysql database and a basic understanding of database creation and management. the computer availability database schema is pretty simple, all you need is one table. compstatus id (int) (primary key) computer_name(string) top_pos(int) left_pos(int) status(int) computer_type(string) updated_at(date) if you have multiple labs, then you would need a field indicating which lab the computer is in. you could also extend the schema to include a listing of installed software or anything else you want to capture and display about the computers. create database 'computer_availability' create table `compstatus` ( `computer_name` varchar(250) not null default "", `status` int(11) default null, `computer_type` varchar(250) default null, `left_pos` int(11) default null, `top_pos` int(11) default null, `updated_at` timestamp not null default current_timestamp on update current_timestamp, primary key (`computer_name`) ) at this point you may not have the data for your computer lab; if you want to play along you can just use your own computer as a test case. step 2: embed the login/logout scripts note: i am assuming you have access to the computer’s imaging process and a basic understanding of perl scripts. you need to track when a user logs on and when they log off; to do so you need 2 scripts that can pass along the computer’s status and name to a database. #!/usr/bin/perl -w use lwp; use sys::hostname; my $host = hostname(); #the hostname after which you can find out the ip address my $ipaddr = inet_ntoa(scalar gethostbyname($host || 'localhost')); $ua = lwp::useragent->new; #timeout in 60 seconds if we can't make a connection $ua->timeout(60); #send the http request with the status of 1 and computer name $request = $ua->post('http://yourdomain.edu/statuschange.php', ["status" => "1", "workstation" => $ipaddr, "host" => $host]); #if successful log 200 status else quit and log error if ($request->is_success) { $content = $request->content; print "content-type: text/html\\n\\n"; print $content; } else { die "can't get to url", $request->status_line; } exit; script 1. login.pl #!/usr/bin/perl -w use lwp; use sys::hostname; my $host = hostname(); #the hostname after which you can find out the ip address my $ipaddr = inet_ntoa(scalar gethostbyname($host || 'localhost')); $ua = lwp::useragent->new; #timeout in 60 seconds if we can't make a connection $ua->timeout(60); #send the http request with the status of 0 and computer name and the host $request = $ua->post('http://yourdomain.edu/statuschange.php', ["status" => "0", "workstation" => $ipaddr, "host" => $host]); #if successful log 200 status else quit and log error if ($request->is_success) { $content = $request->content; print "content-type: text/html\\n\\n"; print $content; } else { die "can't get to url", $request->status_line; } exit; script 2. logout.pl now you need to embed these scripts into the computer’s login and logout scripts. first, for this to work each computer is going to need a dns entry. the process of adding the scripts will differ depending upon the operating system. see appendix c for resources. step 3: update the computer’s status note: i am assuming you have a basic understanding of php. now that you have the perl scripts installed on the computers and triggering the http request you need a script to capture that data and update the computer’s status in the database. this script needs to be stored at the location you pointed to in the http request. #add your database username and password $user="username"; $password="password"; $database="computer_availability"; #unless the computers name was empty if($_post['workstation'] != ""){ $workstation = strtoupper($_post['workstation']); } else{ #build the computer's name from the host $host_domain = strstr($_post['host'], '.'); $workstation = strtoupper(str_replace($host_domain, '', $_post['host'])); } #connect to the database $db = mysql_connect('mysqlcluster.adm.yourmysqlserver.edu', $user, $password); @mysql_select_db($database) or die("unable to select database"); #get the computer's row based on it's name $checkquery = "select computer_name from compstatus where computer_name = '".$workstation."'"; $result = mysql_query($checkquery); #if we find a computer update it's status if(mysql_numrows($result)>0){ $query="update `compstatus` set status = '".$_post['status']."' where computer_name = '".$workstation."'"; mysql_query($query) or die(mysql_error()); } mysql_close($db); script 3. statuschange.php sanity check: let’s test it out at this point you can run some tests and make sure the computers are talking to the database. since you don’t have the web service built yet, you’re going to log on and off then query the database to see if it recorded the status changes. note: you can use your own computer as a test case. check that the perl scripts are installed into the computers build check that your computer name is stored in the database and the status is 0 select 'status' from 'compstatus' where name = 'your computer name' login to your computer check the database to see if the status was recorded select 'status' from 'compstatus' where name = "your computer name" if you see that the status is now 1 then it worked! now test log off. you should see it set back to 0 step 4: create the map osul mobile team had a student designer create the computer lab image and the computer icons based on our design requirements. you will need to define your own requirements and create your own images. design requirements the map shall be a simple but realistic view of the lab the map shall include physical markers such as the reference desk, printers or walls the map dimensions shall be 300px, 600px and 900px the icon dimensions shall be 10px, 20px and 30px the icons shall use both color and shape to identify its status and type the map shall have an empty spot for every computer (x,y location) the map is comprised of two layers: the lab image, and then the computer icons placed on top at an x,y location with css. figure 2. computer lab image lessons learned requiring that we support multiple screen sizes greatly increased the complexity of creating the maps. we used whole multipliers (2x) to scale the image and icons up and down. this allowed us to use simple math to change the computer’s x,y location when it was displayed on the various sizes of maps. since we wanted a map that could be usable and attractive on a mobile phone and a desktop, we decided to use a grayscale color scheme and simple line drawings to indicate where the computers are in regards to physical landmarks, such as the printers. we wanted the map to be realistic so we also tried to capture the shapes of the computer pods and clusters. there were a couple little nuisances when we where creating the images. one thing was drawing the map so that the computer’s space in the pod or cluster was big enough to fit the icons. choosing the orientation of the map also posed a problem. we choose to orient the map based on major reference points like the printers and help desk. to support our accessibility requirements we decided to use both color and shapes to represent the status and type. we chose squares for pcs (pun intended) and triangles for macs; red and an x for unavailable and green and a plus sign for available. we used both the color and the shapes to address color accessibility. we also provided a legend and text equivalents of the map’s information. we used just the text-equivalent for non-smart phone mobile users that cannot display the 300px image. figure 3. computer icons finally, we chose to implement a css layout. step 5: build the user interface (ui) note: i am assuming you have a basic understanding of php, sql and css. this code is intended to be used in a standard lamp stack. now let’s put that final tier on and build the ui to display the map with the real-time computer availability status and computer types indicated by the correct computer icon. <?php #add your database username and password $user="username"; $password="password"; $database="computer_availability"; #connect to the database $db = mysql_connect('mysqlcluster.adm.yourmysqlserver.edu', $user, $password); @mysql_select_db($database) or die("unable to select database"); $total_pc_results = mysql_query("select * from compstatus where computer_type='pc'"); $avail_pc_results = mysql_query("select * from compstatus where status='0' and computer_type='pc'"); $pcs = mysql_num_rows($avail_pc_results) . '/' .mysql_num_rows($total_pc_results); #get the textual data total numbers and available numbers of macs $total_mac_results = mysql_query("select * from compstatus where computer_type='mac'"); $avail_mac_results = mysql_query("select * from compstatus where status='0' and computer_type='mac'"); $macs = mysql_num_rows($avail_mac_results) . '/' . mysql_num_rows($total_mac_results); #get all the computer's row of data $result = mysql_query("select * from compstatus"); mysql_close($db); ?> <div id="computer_map"> #text equivalant <p>pc's available: <?php echo $pcs; ?> mac's available: <?php echo $macs; ?></p> # the map <div id="computer_map_600"> <dl> #loop through the rows and display the correct icon at the computer's location #the computer's location is multiplied by 2 because this is the large map #<dt class="avail_pc_600 icon" style="left:200px;top:100px>icp10</dt> #<dt class="avail_pc_600 icon" style="left:200px;top:100px>icp10</dt> <?php while($row = mysql_fetch_assoc($result)){?> <dt class="<?=($row&#91;'status'&#93;==0 ? 'avail' : 'busy');?>_<?=(strcmp($row&#91;'computer_type'&#93;,'pc')==0 ? 'pc' : 'mac');?>_600 icon" style="left: <?=($row&#91;'left_pos'&#93;*2);?>px; top:<?=($row&#91;'top_pos'&#93;*2)+30;?>px" ><?=$row&#91;'computer_name'&#93;?></dt> <?php }?> </dl> #last update <p>map is updated every 5 minutes. last updated: <?php echo date("m j, y \&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;\m\d\a\s\h\; h:i:s"); ?></p> #end map </div> </div> script 4. computers.php figure 4. osul computer availability map note: we first implemented this in ruby on rails (ror) for our mobile site and then ported it to php so that it could be used on osul main site which is in drupal. i have made the ruby on rails mobile version and the drupal version available at: http://github.com/griggsk/availability-map/ sanity check: let’s test it out now that you have all the pieces put together you can see it in action. check that you have added your computer’s location to the database and it corresponds to a place on the map login to your computer navigate to the computers.php page check to see that your computer is now displayed as unavailable test logout. you should see that your computer is available optional step 6: support multiple sizes if you want to support multiple sizes then you will need maps and icons in those sizes. osul choose 300px (mobile), 600px (full site), and 900px (lcd). you can use the same web app you built above to have a mobile version of the application. to alter the map you just built to work on smart phones you need to show the smaller images and reduce the multiplier of the location. <div id="computer_map_300"> <dl> #loop through the rows and display the correct icon at the computer's location #the computer's location is not multiplied by 2 because this is the mobile site #<dt class="avail_pc_300 icon" style="left:100px;top:50px>icp10</dt> #<dt class="busy_mac_300 icon" style="left:120px;top:70px>icp20</dt> <?php while($row = mysql_fetch_assoc($result)){?> <dt class="<?=($row&#91;'status'&#93;==0 ? 'avail' : 'busy');?>_<?=(strcmp($row&#91;'computer_type'&#93;,'pc')==0 ? 'pc' : 'mac');?>_300 icon" style="left: <?=($row&#91;'left_pos'&#93;);?>px; top:<?=($row&#91;'top_pos'&#93;)+30;?>px" ><?=$row&#91;'computer_name'&#93;?></dt> <?php }?> </dl> </div> figure 5. osul mobile computer availability map conclusion there are some obvious problems with this solution. there’s the false negative/positive problem, such as students not logging off or computers needing service. there is also upkeep of the image to reflect changes to the layout of the lab. then there is also one strict requirement: you must have access to the computer’s imaging process. if you don’t have that you can’t use this method. i hope that this tutorial has been helpful; feel free to use this code in part or whole. the code can be easily integrated into drupal sites or as a standalone lamp or ror application. like this code, the tools used to build the map are free and open-source. the complete computer availability map application from this tutorial is available at http://github.com/griggsk/availability-map. reference 1 ncsu’s computer availability map system overview (pdf) greatly influenced how we implemented osul’s computer availability application. appendix a: in-house availability maps university of technology sydney library ncsu libraries asu libraries smith college libraries boston college georgia institute of technology libraries rice university library uw-madison college learning support services unc charlotte uncg jackson library appendix b: labstats availability maps wilfrid laurier university library florida gulf coast university appendix c: computer imaging resources windows: frequently asked questions about logon scripts http://www.rlmueller.net/logonscriptfaq.htm option 1: use window’s group policy create system startup / shutdown and user logon / logoff scripts http://technet.microsoft.com/en-us/magazine/dd630947.aspx setting up a logon script through gpo in windows server 2008 http://www.petri.co.il/setting-up-logon-script-through-gpo-windows-server-2008.htm option 2: use active directory setting up a logon script through active directory users and computers in windows server 2008 http://www.petri.co.il/setting-up-logon-script-through-active-directory-users-computers-windows-server-2008.htm macintosh: creating a login hook http://support.apple.com/kb/ht2420?viewlocale=en_us logout hooks for dummies: http://discussions.apple.com/thread.jspa?messageid=6891519 note: many of us may not have access to the computer’s imaging process. osul mobile team worked with the computer lab’s itc (information technology consultant) to embedd the scripts. our itc provided me with the links above and said he uses option 1 for windows and hooks for macs. if your labs are under management by someone else, you may need to form a similar relationship. about the author kim griggs is a programmer/analyst at oregon state university libraries. she has a b.s in computer science and a m.s in human computer interaction. she is the developer of a number of projects including: library a la carte, an open-source cms, osul mobile website & catalog, and beavertracks, a location-based campus history tour mobile application. subscribe to comments: for this article | for all articles 2 responses to "how to build a computer availability map" please leave a response below: coding for librarians : applied knowledge is the best kind « mmit blog, 2012-01-10 […] osu libraries’ computer availability map […] mike tishman, 2014-01-08 hello kim! i am a first year student here over at the florida institute of technology and was trying to put together a system like this; however i didnt want to have to re-image the computers. i have been doing research for the past couple of days and have been trying alot of scripts i’ve written to obtain information that would indicate an active computer. however i have had no luck. do you have any suggestions for me that wouldnt require re-imaging of the computers. any help would be greatly appreciated! thank you! ~mike leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – query translation in europeana mission editorial committee process and structure code4lib issue 27, 2015-01-21 query translation in europeana europeana – a database containing european digital cultural heritage objects – recently introduced query translation in order to aid users in searching the collections regardless of language. the user enters query terms, and the portal searches for those terms in multiple languages. this article discusses the technical details of query translation with the aim of assisting similar projects to implement similar features. by péter király introduction europeana [1] is a multilingual database that contains cultural heritage metadata from all european countries in many languages. the service provides different tools to help users run automatic machine translations, one of which is a new tool for query translation. the usage scenario is simple: the user sets the languages he or she wants to use in the process (up to six languages in total), then when a search expression is entered, the europeana portal extends the query by injecting translations of the terms or phrases found in it, and runs the modified version against the search index. this paper highlights the details of this process in the hope that it can be reused in other projects. europeana makes use of apache solr for searching. solr reads and parses the search expressions, runs a couple of transformations, analyses and filters, and creates its own internal representation. the process is highly configurable, and solr provides tools to modify the query by injecting synonyms or by skipping stop words. query translation is a similar modification; however, while the synonyms come from a static source (usually a file), translations come dynamically from external sources. this would normally not be a problem because solr provides methods which enable the developer to write a custom plugin, but there is a price for that: the developer has to separate this particular part of his/her project into a different codebase because its product will be a distinct artifact – a solr plugin. it should have its own development cycle with deployment and maintenance issues – partly-independent from the main project. this plugin should be inline with the solr version and can become outdated if the solr api changes. at europeana, we chose a different route. we made this injection code part of the main europeana project and the system performs a query modification before the query reaches solr. this code takes a query string and produces another query string. the string should conform to the underlying lucene query syntax [2], which has not changed much in the past ten years and is less dependent on solr api changes. in order to do this, the process somewhat mimics what solr itself does: normalize the boolean operators of the query extract the terms from the query call translation service on the extracted terms inject the translations back into the appropriate position of the query string below we will discuss these steps in detail. boolean normalization normalizing the boolean operators of the query is the easiest step. solr lets us set a default boolean operator. solr’s default operator is or, but europeana uses and. this means that when the user enters mona lisa, solr understands that the query has two atomic terms and it sets the boolean and operator between them. so the resulting query will be mona and lisa. but what happens if the user enters mona lisa or la joconda? you would expect that solr understands that it is two expressions: mona lisa, and la joconda, however what solr really understands is three expressions: the first is mona, the second is either lisa or la, and the third one is joconda. fortunately the lucene query language allows us to use parentheses for grouping terms together forming an expression in an unambiguous way. the query (mona lisa) or (la joconda) does what the we originally expected: it finds the records either having both mona and lisa or both la and joconda. the query normalization uses some text analysis to add parentheses to the user query and groups together the common parts. it does not use any solr-related library. it is purely text manipulation based on guessing. extraction of terms now we have a normalized query from which we have to extract terms and phrases. for instance, in dog or cat we have two terms; in (mona lisa) or (la joconda) we have two phrases and each contains multiple terms. solr supports a number of syntactical patterns to distinguish between query types (descendant of lucene’s basic query object). some examples include: spinoza – a term query (lucene’s termquery object) den haag – a boolean query (booleanquery) composed of two term queries, using the default boolean operator “den haag” – a phrase query (phrasequery) spinoza* – a prefix query: terms starting with spinoza (prefixquery) spinoza~ – a fuzzy query: terms similar to spinoza [3] (fuzzyquery) [alpha to beta] – a term range query: terms in lexicographical order standing between alpha and beta (termrangequery) timestamp_created:[2013-03-15t19:58:36.43z to 2013-04-15t19:58:36.43z] – a special term range query: because the type of this particular field is a date (represented in iso 8601 format for utc standard), the query using date comparisions. this example matches records created between 15th march and 15th april, 2013. *:* – matches all documents (matchalldocsquery) any query could be a component in a bigger boolean query. the process should take into account that we do not have to inject translations into certain types of queries. to be precise, we only need to inject them into term queries and phrase queries. sometimes we should handle two terms as a compound phase (for example mona lisa or den haag), and we have to subject them to query translation. the lucene java library provides a way to deconstruct queries into atomic components (via the queryparser class) by extracting terms and query types, but the query object and descendants do not give information about the position, e.g. where the term or phrase takes place in the user-entered query. moreover, usually the extracted term has undergone some character transformations such as white space removal, making terms lower case, and truncation. let us imagine a fictitious query for ‘“prince” is the prince of music’. the first prince is a phrase query, so we do not extract it. the second prince is a term query, so we would like to get synonyms for that. now we have translations for only one prince, but how do we inject it? we do not know which prince should be modified in the string. in the time we built a mapping of the normalized query terms and their translated versions, we already lost the information where the term occurred in the original query string. lucene provides tokenization as another tool for solving this problem. the same analyzer – which the process uses in creating the queryparser – can extract the stream of tokens, which are terms in the query string with additional metadata, such as position (start and end offset). when both tokenization and query parsing end, the next process merges the two kinds of information and tries to figure out which token belongs to which query component. by the end of this process our aforementioned mapping will contain the position information as well. thus it has now been enabled for use in injecting translations into the right position. calling the translation service currently, europeana uses wikipedia’s api to run translations via the following url pattern: http://[language].wikipedia.org/w/api.php?action=query&prop=langlinks&lllimit=200&format=json&titles=[query term] there are two variables in this pattern. the first is the [language] variable which is the iso 639-1 language code [4] and in europeana’s case comes from the user’s settings. the second variable is the [query term], which is a title of a wikipedia entry and comes from the term extraction. we know the user’s language preferences, but we do not know which language was used in entering the query string, so the process runs the same api call in all languages in order to be certain. it then extracts all possible language variations (so it calls en.wikipedia.org, de.wikipedia.org, nl.wikipedia.org, etc., with the same query). the api returns the response in json format and the software ingests it using google’s gson library. an example for the returning json (querying for the hungarian version of den haag http://hu.wikipedia.org/w/api.php?action=query&prop=langlinks&lllimit=200&format=json&titles=h%c3%a1ga): { ... "query": { "normalized": [ { "from": "hága", "to": "hága" } ], "pages": { "123336": { "pageid": 123336, "ns": 0, "title": "hága", "langlinks": [ ... {"lang": "de", "*": "den haag"}, {"lang": "en","*": "the hague"}, ... ] } } } } the wikipedia api does some normalization and corrects lower case issues (line 4-9). the current language’s main translation in the title property (line 14), and all other versions are available in the langlinks array (we removed the bulk of the items in the array, keeping only two language versions). post-translation positioning now that we have both the positions and the translations, all we have left to do is modify the query in order to inject the translations back into the appropriate position within the query string. the basic pattern looks like this: original query: terma and termb modified query: (terma or translationa1 or translationa2) and (termb or translationb1 translationb2) for example: original query: den haag and warsaw modified query: (“den haag” or “the hague” or hága) and (warsaw or varsó or warschau) the injection is usually rather straightforward, but there are some cases where it is a bit complicated.  for example: if the translations are the same for multiple languages, the second instance of the term is filtered out. if the term or translation is a phrase or expression, then we have to handle quotation marks correctly (lucene queryparser does not accept them as part of the phrase). lucene handles phrases containing only one element such as place:”paris” as a termquery and not a phrasequery, so place:”paris” and place:paris are equivalent. user interfaces the query translation feature is available in the europeana api and europeana portal. in the europeana api the end point is http://europeana.eu/api/v2/translatequery.json. on top of parameters common to all europeana apis (namely wskey and callback, see http://labs.europeana.eu/api) it accepts the following two parameters: term, the query string to be translated languagecodes, space or comma separated list of iso 639-1 language codes denoting the language(s) of the translations it returns the list of the translations and a well-formed query string which can be used in a search api call. the example: http://europeana.eu/api/v2/translatequery.json?wskey=xxxxxxxx&term=%28den%20haag%29%20or%20warsaw&languagecodes=de,nl,hu,en returns   { "apikey": "xxxxxxxx", "action": "translatequery.json", "success": true, "requestnumber": 13811, "translations": [ {"text": "den haag",  "languagecode": "de"}, {"text": "warsaw",    "languagecode": "de"}, {"text": "warschau",  "languagecode": "de"}, {"text": "the hague", "languagecode": "en"}, {"text": "warsaw",    "languagecode": "en"}, {"text": "hága",      "languagecode": "hu"}, {"text": "varsó",     "languagecode": "hu"}, {"text": "den haag",  "languagecode": "nl"}, {"text": "warsaw",    "languagecode": "nl"}, {"text": "warschau",  "languagecode": "nl"} ], "translatedquery": "(\"den haag\" or \"the hague\" or \"hága\")) or (warsaw or \"varsó\" or \"warschau\")" } the api key used in this example (line 2) is fictitious – please use your own api key. the translations key (lines 6-17) contains the list of translations, each having two properties: text is the translation and languagecode is the iso 639-1 code of the language. the translatedquery key (lines 18-19) contains the solr query the api user can run in a distinct search call. in the portal the user has to set up language preferences in order to utilize the query translation feature. it can be done in language settings within my europeana (available for anonymous users as well).  below the search input box the user can see the individual translations (with language information), and he or she can remove them one by one or all of them with one click. the language versions are saved in all the links of the search result page, and if the user follows a link, the portal extracts the translation from the url instead of trying to call the translation api again.   conclusion this model can be followed in other projects as it is flexible enough to use a different translation service or any other service to help users enhance their query by other means than multilinguality. one can imagine a classification system, a thesaurus, or an ontology instead of a translation service – each would serve different purposes and target different user experiences. the hardest part of implementation is term extraction. europeana’s solution is closely coupled with the lucene library and java language. if another project would like to follow this approach they should find out which techniques support the parsing of the query language used by their favorite search engine. if they do not use java but still use solr (or any other lucene-based search engine), lucene query parsers are available in almost all programming languages (such as python, php, c#, c++, perl, ruby, javascript). appendix in this section you will find some code snippets from the europeana’s queryextractor class. i removed some of the details such as parameter checking and try-catch blocks etc. to concentrate instead on the lucene library usage. if you are interested please consult the code at europeana’s github repository. [5] a) by initializing the queryparser we made use of simpleanalyzer, and “text” as the default query field [6]: analyzer analyzer = new simpleanalyzer(); queryparser queryparser = new queryparser(version.lucene_40, "text", analyzer); queryparser.setdefaultoperator(operator.and); b) getting the topmost query object: query query = queryparser.parse(rawquerystring); c) iterating over the query components: public void deconstructquery(query query, stack<querytype> querytypestack) { if (query == null) { return; } if (query instanceof termquery) { querytypestack.add(querytype.term); deconstructtermquery((termquery)query, querytypestack); } else if (query instanceof phrasequery) { group++; querytypestack.add(querytype.phrase); deconstructphrasequery((phrasequery)query, querytypestack); } else if (query instanceof booleanquery) { group++; querytypestack.add(querytype.boolean); deconstructbooleanquery((booleanquery)query, querytypestack); } else if (query instanceof prefixquery) { group++; querytypestack.add(querytype.prefix); deconstructprefixquery((prefixquery)query, querytypestack); } else if (query instanceof fuzzyquery) { group++; querytypestack.add(querytype.fuzzy); deconstructfuzzyquery((fuzzyquery)query, querytypestack); } else if (query instanceof termrangequery) { group++; querytypestack.add(querytype.termrange); deconstructtermrangequery((termrangequery)query, querytypestack); } else if (query instanceof matchalldocsquery) { group++; querytypestack.add(querytype.matchalldocs); deconstructmatchalldocsquery((matchalldocsquery)query, querytypestack); } else { log.warning("unhandled query class: " + query.getclass()); } if (querytypestack.size() > 0) { querytypestack.pop(); } } in this method querytype is an enumeration of the lucene querytypes and denotes the name of the actual query class. the object collects them into a stack, for later usage (in some cases we have to investigate the parent query component’s type to determine what to do with the terms of the actual component). the deconstructxxxquery() methods usually extract and save the terms. i mentioned previously that booleanquery connects two or more components together, so the deconstructbooleanquery() method recursively calls this deconstructquery() method to find those components. the role of the group variable is simple. it helps to find implicit phrases such as den haag without the quotes. d) lastly, extracting tokens from the user-entered search query. tokenstream ts = analyzer.tokenstream("text", new stringreader(query)); offsetattribute offsetattribute = ts.addattribute(offsetattribute.class); chartermattribute chartermattribute = ts.addattribute(chartermattribute.class); ts.reset(); while (ts.incrementtoken()) { int start = offsetattribute.startoffset(); int end = offsetattribute.endoffset(); string term = chartermattribute.tostring(); string terminquery = query.substring(start, end); // save the above information } we use the same analyzer as in the first snippet. we explicitly have to ask the tokenstream object to provide information about the position (offsetattribute), and the term (chartermattribute). in line 10 we extract the original term, since the token may be in an already transformed state, which could be different than the user entered version. notes [1] http://europeana.eu [2] apache lucene is a java library created for supporting indexing and searching. solr is built on top of this library, and makes use lucene’s query language. the europeana query translation also depends on some functionalities of this library. [3] the similarity between terms are measured with the edit distance: how many atomic letter changes (replacement, deletion, addition) should be done to get from terma to termb. see http://lucene.apache.org/core/4_0_0/queryparser/org/apache/lucene/queryparser/classic/package-summary.html#fuzzy_searches and http://en.wikipedia.org/wiki/damerau%e2%80%93levenshtein_distance. [4] http://en.wikipedia.org/wiki/list_of_iso_639-1_codes [5] https://github.com/europeana/api2/, and https://github.com/europeana/corelib/. you can use the api’s querytranslationcontroller class (https://github.com/europeana/api2/blob/master/api2-war/src/main/java/eu/europeana/api2/v2/web/controller/querytranslationcontroller.java) as entry point. [6] text is a kind of super field in europeana’s index. the indexing process copies almost every other field’s content into this field. when the user does not specify a field, the search runs this field again. about the author péter király is a software developer with a humanities background (history and philology). he works at gesellschaft für wissenschaftliche datenverarbeitung göttingen (germany). his main interests are publishing and searching large textual corpora, digital libraries, digital humanities, and new possibilities of digital cultural heritage. he has participated in projects like project gutenberg, extensible catalog and europeana. you can reach péter at peter.kiraly[at]gwdg.de. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – openumisma: a software platform managing numismatic collections with a particular focus on reflectance transformation imaging mission editorial committee process and structure code4lib issue 37, 2017-07-18 openumisma: a software platform managing numismatic collections with a particular focus on reflectance transformation imaging this paper describes openumisma; a reusable web-based platform focused on digital numismatic collections. the platform provides an innovative merge of digital imaging and data management systems that offer great new opportunities for research and the dissemination of numismatic knowledge online. a unique feature of the platform is the application of reflectance transformation imaging (rti), a computational photographic method that offers tremendous image analysis and possibilities for numismatic research. this computational photography technique allows the user to observe on browser minor details, unseen with the naked eye just by holding the computer mouse rather than the actual object. the first successful implementation of openumisma has been the creation of a digital library for the medieval coins from the collection of the bank of cyprus cultural foundation. by avgoustinos avgousti, andriana nikolaidou, ropertos georgiou introduction as the implementation of software application grew in the creation of numismatic digital collections, the need for better supporting applications grew correspondingly with a better provision of the digital libraries’ requirements. for the study on a range of objects such as numismatics, with delicate materiality and rich historical background, an application that can digitally reflect their illustrious nature is necessary. in response to this growing need a web-based platform “openumisma” is being developed by the cyprus institute. the platform supports an advanced imaging technology, reflectance transformation imaging (rti) [1] that offers powerful new methods of documenting and studying numismatic cultural heritage information. rti is a computational photography technique that enables the virtual examination of objects in front of a computer monitor and assists for visual analysis, conservation and documentation. reflectance transformation imaging is shown to capture more complete documentation than traditional photographic methods and broadens the scientific research of a coin. we intend to demonstrate how the addition of openumisma can bring a feasible step forward to general-purpose service system numismatic digital libraries and overview its current features and future additions. the significant system performance formation and software architecture is analyzed by giving a coherent view of components, where the architectural elements meet all of the technical and operational requirements of the software. the presentation of the platform is based on the first successful implementation of openumisma in the online coinage collection of the bank of cyprus cultural foundation, showing how the software successfully managed to accomplish its significant system performance. numismatic software’s applications a tremendous amount of digital projects are available to the public that provides online access to coins collection from different cultural institutions [2]. many of the numismatic collections that are available online today are using tailor-made software developed for their specific needs or open source software applications. particular examples using tailor-made applications are the roman provincial coinage online [3] ,the sylloge nummorum graecorum [4] and vatican library: catalogues-coins and medals [5]. on the other hand, the custom coded software gives endless possibilities to a project to create anything it can “imagine” but at a significant time and cost, without any open collaboration and more diverse scope of development perspectives. ocre: online coins of the roman empire [6] is using the open source of numishare [7] along with coinage of the roman republic online [8] and dar al-kutub collection of the egyptian national library [9]. openumisma background and development working on the creation of the medieval numismatic digital library of bank of cyprus cultural foundation, the main aim was to develop a web-based platform. this platform had to be usable particularly for digital numismatic collections, in order to improve the context of the library. working on different advanced technologies that could enrich the medieval numismatic digital library of bank of cyprus cultural foundation we developed the platform “openumisma”. a few of the advanced technologies that openumisma provides are reflectance transformation imaging, embedded interactive maps, bibliographical reference, metadata, taxonomical terms, advanced search, facets, images search, xml export and more. for the development of openumisma we selected drupal 7, an open source content management framework (cmf) with a large community of developers and users. at that time, in late 2013, drupal 8 was not quite ready for the development of such a project. drupal 8 is newer and most probably will be supported longer and possibly with more features than drupal 7. we are looking forward to releasing openumisma in a drupal 8 version when the framework is more stable and ready for supporting the complexity of a project like openumisma. study of coins through reflectance transformation imaging (rti) the examination of artwork’s surfaces under tangential light was one of the main study techniques in the past. the illumination technique, called raking light, casts a strong light at an oblique angle, almost parallel to the surface of the object, the light intensifies in order to reveal and highlight information clearly. the created light and shadows give a greater understanding of the relief of the object by emphasizing any surface irregularities and deformations such as cracks, distortions, and material condition. however, nowadays reflectance transformation imaging (rti) is a unique computational photography method for examination, conservation documentation, and identification for coins. the advanced computational photography technique simulates all possible light angles within a virtual hemisphere atop the surface of the coin for effective study and analysis (figure 1). this powerful imaging technique proved to be extremely important for archaeologists, conservators, and historians for the examination of a variety of artifacts. these imaging techniques can “re-animate” and penetrate surfaces in a manner that conventional photography and digital methods of acquisition such as 3d laser scanning cannot fully convey but instead complement [10]. in addition, these methods feature creative and reliable ways for material examination reproducing reality with minor computation demands for the end-user and it aims to provide easily accessible and straightforward practical methodologies for experts in the archaeological field as well the museum and the web [11]. by moving a mouse or any other pointing device, the user can control the direction of the light, zoom in and out and observe minor details (figure 2). figure 1. medieval coins from the bank of cyprus cultural foundation collection figure 2. medieval coins from the bank of cyprus cultural foundation collection http://numismatics-medieval.dioptra.cyi.ac.cy rti data acquisition a custom-build rti dome that produces polynomial texture map (ptm) files consists of a hemispherical dome with a hole at the apex, and thirty-six halogen lights embedded at randomly fixed intervals around the dome and controlled by a pre-scripted programmable microcontroller (figure 3). an artifact is placed at the base of the dome, while a camera is positioned looking downward, focusing through the hole at the top, on the aforementioned object. thirty-six photographic images are sequentially taken, each with a single light shining on the artifact, thus creating thirty-six images with different light angles. then, the ptm algorithm synthesizes the data from these images to create a single image that can be examined on a ptm viewer with a “virtual spotlight”. the viewer allows the user to move the light angle intuitively in real time so that the combination of light and shadow representing the relief features of the object’s surface can be freely altered. rti also permits the enhancement of the subject’s surface shape, color and luminance attributes, which extracts detail out of the surface that cannot be otherwise derived. rti viewers are open source[12], however, rti datasets can be accessed just through desktop applications. consequently, the requirement to create an open-source drupal rti web platform that encapsulates coins and can be assessed and visualized through web applications has been achieved through the development of the openumisma platform. figure 3. the rti dome of the cyprus institute reflectance transformation imaging custom drupal 7.x module the functionality of rti on the platform of openumisma is developed on a custom drupal module. the module is based on webrtiviewer from the visual computing laboratoryisti-cnr [http://vcg.isti.cnr.it/rti/webviewer.php]. webrtiviewer uses html5-webgl, can display high-resolution rti images (ptm and hsh), and is available under the gnu general public license version 3. openumisma is already preconfigured with the rti drupal module, however having in mind organizations that have an existing drupal 7 installation and wish to have only the rti functionality, the rti module can be downloaded and installed separately with only a couple of clicks as all drupal contributed modules (figure 4). the rti drupal 7 module is already helping some organizations like “the cultural informatics research group” at the university of brighton. the module can be downloaded from github: https://github.com/avgoustinos/drupal-7-webgl_viewer future plans are to have the module on drupal.org /** * implements hook_field_formatter() */ function webgl_viewer_field_formatter_info() { $formatters = array(); $webgl_viewer_settings = webgl_viewer_settings_info('default_value'); $default_settings = array(); // return a single depth array with the given key as value. foreach ($webgl_viewer_settings as $key => $setting) { if (isset($setting['fieldset'])) { $default_settings[$setting['fieldset']][$key] = $setting['default_value']; } else { $default_settings[$key] = $setting['default_value']; } } $formatters['webgl_viewer'] = array( 'label' => t('web viewer'), 'field types' => array('image', 'file'), 'settings' => $default_settings, ); return $formatters; } listing 1. code example from the custom rti drupal module openumisma data modeling and content types for the profile of each coin that can be added to any digital library, openumisma created a default data modelling and content type design. in collaboration with numismatics experts, we developed a list of controlled vocabularies. for each coin addition there is a complex set of fields with metadata elements related to numismatics such as the material of the coin, the description of the obverse and reverse surface of the coin, the location and many more (figure 5). for example, a coin was minted in a specific location. the name of the place along with its geographical location is listed in a vocabulary called mint. by assigning the term mint with the name of the place to the image or rti, the content is classified according to geo-location minted. with this process openumisma can be used to implement this concept and create a retrieval tool. using the example above the user is able to view all coins minted in a specific location. moreover, the information can be viewed in a different medium, such as an interactive map where you can drill down to a specific location. the set of fields for a coin covers most of the standard needs and requirements for numismatics collections, and the administrator of the platform can easily extend the functionality by adding new fields based on their specific data and metadata requirements. all taxonomical terms and related data can be exported in xml format. figure 4. openumisma coin content type information retrieval for easy retrieval of information, openumisma comes by default with some preconfigured views that provide a listing of data and nodes. for the display of the coins collection in a digital library we used the module drupal view (figure 6) that displays all the coins in a single table list. users can easily browse the collection and filter their search by using the fields on the top of the page. by clicking on the image of each coin, the user is redirected to the rti web viewer and has the opportunity to use the rti on the coin in a separate window. another way to search the database is by the interactive maps that display content based on a geographical location (figure 7). the interactive map helps the user to discover coins by their location and also by the date they were minted. moreover, clustering and drill down capabilities of the map make it user-friendly when it comes to numerous geo-points. to enhance performance and readability of data-heavy maps, the system clusters geo-points and allows displaying a large amount of data in a preformatted way. figure 5. the visual collection of medieval coins from the cypriot medieval coins: history and culture library figure 6. the visual interactive map from the cypriot medieval coins: history and culture library linked open data: semantic web the semantic web is a set of standards and best practices for sharing data over the web that make it more machine-friendly, which makes computers more intelligent. the web pages are structured data in a way that computers can understand. according to the w3c, “the semantic web provides a common framework that allows data to be shared and reused across application, enterprise, and community boundaries”. openumisma is based on cidoc conceptual reference model (cidoc-crm) ontology, which is designed to facilitate the exchange of heterogeneous cultural heritage information among cultural heritage institutions [12]. cidoc-crm appeared to be the perfect match for openumisma; a selection of standard openumisma fields was easily mapped to the most relevant cidoc fields. however, in order to adapt and extend cidoc-crm for numismatic collections more work needs to be done. extensible markup language xml the system is able to generate and export xml data files for all the data and metadata stored in the system. currently, systems administrators can export data in xml format; some of the advantages of using xml-encoded documents for the long-term archive are that xml is an open and well-established notation. xml is both human and machine-readable (listing 2). text format and internationalized characters are supported. with these characteristics, we can facilitate data migration and data transparency. <?xml version="1.0" encoding="utf-8"?> <coins> <coin> <title>ar gros. john ii, 1432-1458</title> <coin-image>/sites/default/files/1995-07-05-ob.png</coin-image> <mint>nicosia</mint> <material>ar</material> <axis>07:00</axis> <diameter>25 mm</diameter> <description>cross of jerusalem. </description> <weight>4.18 gm</weight> <denomination>gros</denomination> <inscription>ioan rex.d</inscription> <bibliography>clc 3,70-73,pl.10.8-13.</bibliography> <location>latitude:35.185451000000 longitude:33.382411000000</location> </coin> <coin> <title>billon sizin. janus, 1398-1432</title> <coin-image>/sites/default/files/1995-02-03-ob.png</coin-image> <mint>nicosia</mint> <material>billon</material> <axis>12:00</axis> <diameter>19 mm</diameter> <description>cross of jerusalem.</description> <weight>1.57 gm</weight> <denomination>sizin</denomination> <inscription>ianvs roi dei ieru</inscription> <bibliography> *bmk1997, 92, no.40; *clc3, 124-127, pl.27.2 (not correct weight). </bibliography> <location>latitude:35.185451000000 longitude:33.382411000000</location> </coin> </coins> listing 2. example of xml export filemedieval coins from the collection of the bank of cyprus cultural foundation, cypriot medieval coins: history and culture. appearance openumisma offers a preconfigured functionality and a custom drupal sub-theme with css and javascript files related to the display of each coin. the theme is responsive and flexible enough to adapt to any mobile devices, as well on desktop screen resolution.. openumisma allows the user to choose and configure the look and feel of the platform, however, the rti viewer is not responsive, an additional effort will need to be done in order to achieve this goal. applicability the first successful implementation of openumisma was the creation of a digital library for the medieval coins from the collection of the bank of cyprus cultural foundation (figure 7). cypriot medieval coins: history and culture is a novel digital library dedicated to the study, promotion, and dissemination of the history of medieval cypriot coinage (12th-16th centuries). the project is the development of the ongoing collaboration between the bank of cyprus cultural foundation and the cyprus institute, which originally began in february 2012 with the pilot implementation of the advanced reflectance transformation imaging (rti) photography on a large part of the collection of the museum of the history of cypriot coinage. the results of the above work have been the basis for the development of the digital platform for the documentation and dissemination of medieval coins from the museum of the history of cypriot coinage. the digital platform offers an interactive exploration of the medieval collection with the innovative implementation of reflectance transformation imaging, complemented with text descriptions, essays, links to other collections and repositories that provide context as well as alternative ways to study and access the material. figure 7. cypriot medieval coins: history and culture http://numismatics-medieval.dioptra.cyi.ac.cy further development future development of openumisma will be focussed on the implementation of nomisma.org, which has been hosted by the american numismatic society since 2010. nomisma.org is a trustable resource in numismatics that provides uris to many numismatic concepts and terms. moreover, the ontology was created to integrate the openly available databases. the ontology is limited to numismatics and provides an easily understandable way to describe numismatic datasets. our main goal is to have openumisma as a downloadable application package from our website by the end of 2017. moreover, our plans are to have on on github and drupal.org. conclusion the creation of openumisma aims to significantly enhance the documentation and preservation of cultural heritage objects and to improve digital online collections with the use of the reflectance transformation imaging. the innovative web-based platform can be extended in order to support more advanced functionalities as well as to exploit the ever-increasing technological opportunities to enhance the interactive experience between the user and online databases. about the authors avgoustinos avgousti is a research technical specialist at the cyprus institute at the science and technology in archaeology research center (starc). he is the web architect and lead developer of dioptra: the digital library for cypriot culture. he received his bachelor of science (bsc) from st. francis college in new york city in information technology/computer science and master of science (msc) in medical informatics from the state university of new york downstate medical center in new york city. he has more than 12 years of experience in information technology and 3 years of teaching computer science at a college level. furthermore, he participated in a number of eu and national funded projects. avgoustinos’ research activity has been focused in the last five years on digital humanities and digital cultural heritage. andriana nikolaidou joined the science and technology in archaeology research center (starc) of the cyprus institute in september 2015 as an intern in advanced imaging for art and cultural heritage. she contributes to the digital libraries of the cyprus institute as a research assistant. andriana received a bachelor’s degree in fine arts with first-class honors from de montfort university (uk) in 2013. adriana is interested in the aspect of archeology, researching the theme of legend as it relates to cultural narratives by questioning the notions between visual and scientific perspectives. her aim is to combine new and porous ways of digital technologies and specifically advanced imaging into digital art scenarios. ropertos georgiou received an associate degree of applied science in electronics engineering from tci institute in new york city, a bsc in computer engineering from the university of technology in budapest, hungary and an msc in multimedia applications and virtual environments from sussex university, uk. currently, he holds a position at the cyprus institute at the science and technology research centre (starc) as a research new media specialist. his research interests span through innovative methods of electronic and image-based acquisition, documentation, and visualization for the development and support of research infrastructures. acknowledgments the bank of cyprus cultural foundation for funding openumisma: https://www.boccf.org the cyprus institute: http://www.cyi.ac.cy reflectance transformation imaging webrtiviewer: http://vcg.isti.cnr.it/rti/webviewer.php imaging cluster for archaeology and cultural heritage: http://imlab.cyi.ac.cy notes [1] cultural heritage imaging,reflectance transformation imaging (rti) [internet]. available from: http://culturalheritageimaging.org/technologies/rti [2] the 2014 arl spec kit on open source software notes that 53% of 66 academic and public libraries responding to its survey use a locally hosted and supported oss solution for digital preservation. thacker, j., knutson, c., dehmlow, m. (2014) spec kit 340: open source software. united states of america, washington, d.c. : association of research libraries. chapter 1, pg. 12. [3] the roman provincial coinage online is developed by oxford university. the present archive is grounded on ten collections and includes information on more than 13,000 coin types based on 46,725 specimens and with more than 9,000 images. some of the functionalities of the system are that the user can search by city, inscription, mental, diameter and much more. the roman provincial coinage online [internet]. [updated 2017] university of oxford. available from: http://rpc.ashmus.ox.ac.uk/project [4] the sylloge nummorum graecorum is one of the larger numismatic databases dedicated to greek numismatics. more than 25,000 coins are available online, and the database can be searched by mint, material, ruler, period, denomination, coin descriptions and many others fields. all records include images of the coin and descriptive material.the sylloge nummorum graecorum [internet]. [updated 2017] the fitzwilliam museum: department of coins and medals. available from: http://www.sylloge-nummorum-graecorum.org [5] the catalogue of the vatican library’s numismatic cabinet was created in 2001 and includes the collection of parthian coins the collection of imperial roman coins is still underway. vatican library: catalogues-coins and medals. for more information see: http://opac.vatlib.it/iguana/www.main.cls?surl=homemed&language=eng [accessed 07/04/2017] [6] online coins of the roman empire (ocre), a joint project of the american numismatic society and the institute for the study of the ancient world at new york university, is a revolutionary new tool designed to help in the identification, cataloging, and research of the rich and varied coinage of the roman empire. the project records every published type of roman imperial coinage from augustus in 31 bc, until the death of zeno in ad 491. ocre: online coins of the roman empire. for more information see: http://numismatics.org/ocre [accessed 15/01/2017] [7] numishare is an open source suite application developed and maintained by the american numismatic society for managing digital heritage artifacts, with a particular focus on coins and medals, built on open source applications: solr, exist xml data store, and others. some of the functionalities of numishare are browsable facets, handling images, search and more. for more information see: https://github.com/ewg118/numishare [accessed 14/01/2017] [8] coinage of the roman republic online (crro) aims to provide in effect an online version of michael crawford’s 1974 publication roman republican coinage (rrc), which is still the primary typology used for the identification of roman republican coin types. coinage of the roman republic online for more information see: http://numismatics.org/crro [accessed 16/01/2017] [9] our catalog of 6,500 numismatic pieces – coins, glass weights, dies, medals, etc. – is the third major catalog of islamic numismatic material held in the egyptian national library, formerly the khedivial library, egypt’s most important library. dar al-kutub collection of the egyptian national library. for more information see: http://enl.numismatics.org [accessed 16/01/2017] [10] mudge, m. et al. 2010. principles and practices of robust, photography-based digital imaging techniques for museums. in: 1th international symposium on virtual reality, archaeology and cultural heritage vast, edited by a. artusi, m. joly-parvex, g. lucet, a. ribes, and d. pitzalis, paris. available from: https://www.si.edu/mci/earlyphotography/references/vast2010.pdf [11] malzbender, t., gelb, d., and wolters, h., 2001. “polynomial texture maps.” in siggraph ’01 proc. of the 28th annual conference on computing graphics and interactive techniques, (august 2001), 519-528. los angeles, california [12] for more information see: http://www.cidoc-crm.org [accessed 28/02/2017] references velios a. 2011. the john latham archive: an online implementation using drupal. art documentation volume 30, number 2 wiltshire n.g. 2013. the use of sahris as a state-sponsored digital heritage repository and management system in south africa. cipa symposium 2-6 september doi: 10.5194/isprsannals-ii-5-w1-325-2013 available from: researchgate.net berry a, byron a and de bondt b. 2012. using drupal using drupal, 2nd edition kawana h. 2009. towards digital archive systems: architecture and design of digital museum archive: the eighth international symposium on operations research and its applications (isora’09) zhangjiajie, china, september 20–22, 2009 copyright © 2009 orsc & aporc, pp. 15–21 h. kawano. 2012. digital archive system using cms and gallery tools –implementation of anthropological museum. wooster g.e. 2006. recent advances in roman numismatics: ba, the pennsylvania state university. muller eva, klosa uwe hansson peter, andersson, stefan siira erik. using xml for long-term preservation. reflectance transformation imaging webrtiviewer: http://vcg.isti.cnr.it/rti/webviewer.php vandyk j.,tomlinson t, 2010. pro drupal 7 development (expert’s voice in open source) 3rd ed. united states of america, new york: apress. chapter: 1. pg. creating custom modules: https://www.drupal.org/docs/7/creating-custom-modules earl, g.p., martinez, k. and malzbender, t. 2010. archaeological applications of polynomial texture mapping: analysis, conservation, and representation. in journal of archaeological science, 37, pg 40-50. dutr’e, p., bekaert, p., and bala, k. 2003. advanced global illumination 2nd ed. united states of america, natick: a k peters/crc press bakirtzis, n., georgiou, r. 2014, light on el greco from a technological perspective: the baptism of christ and the view of mount sinai at the historical museum of crete in heraklion. in international conference proceedings “el greco the cretan years”, 21-23 june, heraklion earl, g.p., martinez, k. and malzbender, t. 2010. archaeological applications of polynomial texture mapping: analysis, conservation, and representation. in journal of archaeological science, 37, pg 40-50. cultural heritage imaging,reflectance transformation imaging (rti) [internet]. available from: http://culturalheritageimaging.org/technologies/rti for more information see: https://www.drupal.org/project/views [accessed 28/01/2017] thompson k, richard j. 2013. moving our data to the semantic web: leveraging a content management system to create the linked open library. journal of library metadata, 13:290–309, 2013. doi: 10.1080/19386389.2013.828551. available from: https://repository.si.edu/bitstream/handle/10088/22149/wjlm_a_828551.pdf gerth, p and dai (2016) ariadne d14.2: pilot deployment experiments. version: 1st. available from: http://www.ariadne-infrastructure.eu/resources/d14.2-pilot-deployment-experiments [accessed 14/03/2017] subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – revamping metadata maker for ‘linked data editor’: thinking out loud mission editorial committee process and structure code4lib issue 55, 2023-1-20 revamping metadata maker for ‘linked data editor’: thinking out loud with the development of linked data technologies and launch of the bibliographic framework initiative (bibframe), the library community has conducted several experiments to design and build linked data editors. while efforts have been made to create original linked data ‘records’ from scratch, less attention has been given to copy cataloging workflows in a linked data environment. developed and released as an open-source application in 2015, metadata maker is a cataloging creation tool that allows users to create bibliographic metadata without previous knowledge in cataloging. metadata maker might have the potential to be adopted by paraprofessional catalogers in practice with new linked data sources added, including auto suggestion of virtual international authority file (viaf) personal name and library of congress subject heading (lcsh) recommendations based on the users’ text input. this article introduces those new features, shares the user testing results, and discusses the possible future steps. by greta heng, myung-ja han introduction libraries have been using machine readable cataloging (marc) as a tool to create bibliographic and authority data since the 1960s. while marc brought libraries a new way to organize information in the past, the evolving information landscape asks for libraries to explore other means of information organization that can connect library collections with resources on the web. as a successor to marc, bibliographic framework (bibframe) initiative was launched by the library of congress (lc) in 2012.[1] it is expressed in the resource description framework (rdf, a data model for structured data)[2] and based on three categories of abstraction (work, instance, item). as the library’s new entity relation data model, bibframe is grounded in linked data techniques, which allows metadata creators to build relationships with web resources by facilitating shared structured data and uniform resource identifiers (uris). many national and research libraries have been exploring the possibility of converting the marc format metadata to bibframe and, even further, creating metadata as linked data using a linked data/bibframe editor. libraries such as the swedish national library,[3] the french national library,[4] the german national library (dnb),[5] and the library of congress[6] have been involved in the marc to linked data conversion, linked data based new discovery services, and linked data editor experiments. in addition, some external linked data management platforms are gaining popularity among glam (galleries, libraries, archives, and museums) institutions. wikidata,[7] an open, collaborative, and multilingual global linked data repository, is being used by libraries as an alternate source of name and subject data for bibliographic description. however, since wikidata is designed to represent all domains of knowledge and not specific for library use, concerns about its capacity and suitability for describing library resources were raised by the wikidata community.[8] while there has been much discussion on and development of tools for creating full original linked data, less attention has been given to copy cataloging workflows (creating new short records by deriving from other records or creating minimum records) in linked data environments. developed and released as an open-source application in 2015, metadata maker[9] is a metadata creation tool that can be used by anyone regardless of their cataloging experience and knowledge, allowing them to create a minimum level catalog record. metadata maker has been updated in several areas since then, including supporting different formats of resources (currently in ten modules) and bibframe output service for monographs.[10] as more and more cataloging and metadata creation work relies on paraprofessional catalogers or non-catalogers[11] with language or subject expertise, the authors tried to revamp metadata maker with linked data authority services to test whether this tool and the updated functions can facilitate the minimum record creation in a linked data cataloging environment. this paper shares the revamping process and issues found in linked data sources and their service, and discusses the user testing results of metadata maker and a bibframe editor. changing landscape the development of linked data technologies brought out a systematic change in libraries’ cataloging production practice. as van der werf said, “libraries used to be knowledge organizations and library professionals were trained in bibliographic description and authority control. now, authorities are called entities and the new description logic is about creating a ‘knowledge graph of entities.’”[12] it is noticeable that the focus of metadata creation has gradually shifted from the curation of text strings to the management of entities (work, persons, corporate bodies, places, events, etc), i.e., linking resources using uris and managing uris instead of name strings.[13] this revolution has triggered a discussion on linked data cataloging models, standards, and tools in the library. changing library cataloging production practice libraries have carried out several initiatives to re-design cataloging workflows and devise the transition plan from traditional cataloging to linked data cataloging, for example, the development of marc to bibframe conversion tools and bibframe editors. notably, the linked data for libraries (ld4l)[14] community made a series of significant efforts on linked data cataloging from 2014 to 2022, including linked data for libraries labs (ld4l labs),[15] linked data for production (ld4p),[16] linked data for production: pathway to implementation (ld4p2),[17] and linked data for production: closing the loop (ld4p3).[18] albeit those new linked data cataloging tools, catalogers need to be versed in new linked data related knowledge and exercise new skills, such as rdf, sparql, bibframe ontology, and more, to create library data as linked data. in addition, as linked data implementations in libraries are still under development, it is hard to keep up to date with the most current linked data application developments, e.g., bibframe editors. it is challenging to identify the type of skills that catalogers need to be developing. as a result, catalogers may feel overwhelmed by the new linked data technology, and administrators are experiencing challenges in designing and providing training for the ever-growing skill set and emerging linked data tools for catalogers.[19] the shifting roles of librarians and staff in technical services are an additional challenge in linked data training and planning. libraries used to depend on professional cataloging librarians to do original cataloging. copy cataloging was usually performed by paraprofessional catalogers. however, this is no longer true. with shrinking budgets, organizational restructuring, and changes in cataloging software and workflows, more paraprofessional staff are responsible for both original and copy cataloging tasks (el-sherbini & klim, 1997; zhu, 2012).[20] as van der werf articulated, the number of professional librarians is decreasing while paraprofessional staff are increasing in cataloging departments.[21] in fact, not only are professional librarians decreasing, but the whole cataloging team is also shrinking. while there are several options that can ease the shortage of manpower, such as outsourcing to vendors, cooperative cataloging programs, and more productive cataloging workflows, libraries still lack staff with expertise to catalog special collections and/or foreign language materials. the need for foreign language and special collection cataloging will not go away in a linked data environment as libraries keep purchasing resources from foreign countries and work with perpetual backlogs. bibframe editors and copy cataloging currently, there are three bibframe editors that are widely known and used: lc’s bibframe editor,[22] marva,[23] and ld4p’s sinopia.[24] all three editors seem to target experienced catalogers as their user group, not paraprofessional catalogers or non-catalogers. for one, they use the resource description and access (rda) terms[25] as field names and bibframe’s three categories of abstraction, work, instance, item, as record/data types. those cataloging terms, though commonly used by professional catalogers, may result in a learning curve for paraprofessional catalogers. for example, “parallel title” is not a common phrase and the differences between work and instance are not self-explanatory for many. for another, some abbreviations that appear in the user interface as controlled vocabularies, including getty_aat, lcgft, and gac, are not familiar to paraprofessional catalogers. in order to use the editor and add appropriate values to those data fields, it requires training on rda, bibframe ontology, authority, and the editor itself at the very least. another challenge is a lack of clear definition as to what makes full level and brief bibframe data. the core bibframe data fields are still under discussion by the program for cooperative cataloging (pcc) bibframe interoperability group (big).[26] as there are no clear guidelines, some bibframe editors mark required fields while some do not. for catalogers or users of bibframe editors, it seems that one needs to fill out all fields to create full-level bibframe data and provide values for those required fields, if applied, to generate brief bibframe data. as there is no quick way of filling out the minimum data fields to produce brief bibframe data, the cataloging workflow used in the current bibframe editors might not meet libraries’ needs for cataloging large volumes of perpetual backlogs with a shrinking cataloging team. lorimer (2022) stated that the notion of copy cataloging has broadened and expanded in a linked data environment,[27] which emphasizes reusing metadata rather than creating completely new metadata from scratch. some bibframe editors like sinopia indeed allow catalogers to search, load, and copy or clone existing bibframe data to revise and reuse those descriptions by sharing uris. this workflow would help reduce duplicate work-level bibliographic records and increase cataloging efficiency. yet, considering the reality and looking into the future, libraries, with professional catalogers and language/subject experts shortage, will have to resort to non-catalogers and paraprofessional catalogers with limited linked data and cataloging knowledge to create records in bibframe editors. shall users adapt to the bibframe editors or shall the editors be designed to be more friendly to their users? this dilemma raises a question: is it possible to build a linked data editor without cataloging jargon in the application interface? given the above mentioned issues, this project is an attempt to build a straightforward linked data editor that does not use rda terms for non-catalogers for the purpose of copy cataloging. libraries may benefit from adopting metadata maker as it does not require new hiring or training for catalogers and allows non-catalogers with needed language/subject knowledge to create minimum level cataloging records. the authors also conducted a small-scale survey to learn catalogers’ opinions about metadata maker and a linked data editor. revamping metadata maker metadata maker enables any user to create catalog records that are “good enough” (provide sufficient information to identify a bibliographic item and generate a basic bibliographic description)[28] in various formats, including marc, regardless of one’s knowledge of or experience with cataloging standards, integrated library systems, or oclc. it now has ten different modules or templates (datasets[29], monographs[30], monographs (ld)[31], ebooks[32], government documents[33], maps[34], microfilms[35], scores[36], serials[37], theses and dissertations[38]). users can select a module based on the resource type, fill out basic information about the resource, and choose the download format, including marc binary, marcxml, metadata object description schema (mods), html, and bibframe.[39] for this phase, two new linked data features, virtual international authority file (viaf) personal name suggestions and library of congress subject heading (lcsh) suggestions were added in the monographs (ld) module in metadata maker. the new functions support search and auto completion of personal names in viaf, and lcsh (keywords) generation based on the user provided text. uris of the controlled terms are added in the output metadata. figure 1. metadata maker interface screenshot. linked data input viaf name search the viaf personal name autocomplete dropdown list in fig. 2 uses viaf auto suggest api[40] to retrieve the personal name’s label, viaf uri, and library of congress name authority file (lcnaf) uri. when a name is selected, the links to both uris, if they are available in viaf, will be presented on metadata maker. users have the option to verify the name entity’s information on either the viaf or lcnaf page if so desired. the application then retrieves values of the 100 field subfields a to d from lcnaf whenever they are available. if no lcnaf uri is provided in viaf, the preferred label from dnb[41] is the alternative option if that can be found in viaf. the lcnaf and viaf uris are added to subfield 0 and 1 respectively in the marc and marcxml 100 field or 700 field based on their role. for other supported output formats, the uris and the label/preferred name are also inserted into the appropriate elements. if there is no satisfactory result in the autocomplete dropdown list, it also allows users to manually input the name strings. the code is available online.[42] figure 2. viaf auto suggest dropdown list. // using viaf auto suggest api fetch personal names (function($) { $.widget("oclc.viafauto", $.ui.autocomplete, { options: { select: function(event, ui) { alert("selected!"); return this._super(event, ui); }, source: function(request, response) { var term = $.trim(request.term); var url = "https://viaf.org/viaf/autosuggest?query=" + term; var me = this; $.ajax({ url: url, datatype: "jsonp", success: function(data) { if (data.result) { response( $.map( data.result, function(item) { if (item.nametype == "personal"){ var retlbl = item.term + " [" + item.nametype + "]"; var uri = "http://viaf.org/viaf/" + item.viafid; if (item.lc){ return { label: retlbl, value: item.term, id: item.viafid, viafuri: uri, lcuri: "http://id.loc.gov/authorities/names/" + item.lc, nametype: item.nametype } }else{ return { label: retlbl, value: item.term, id: item.viafid, viafuri: uri, lcuri: "nolc", nametype: item.nametype } } } })); } else { me._trigger('nomatch', null, {term: term}); } }, }); } }, _create: function() { return this._super(); }, _setoption: function( key, value ) { this._super( key, value ); }, _setoptions: function( options ) { this._super( options ); } }); })(jquery); // get information for user selected name in the author input field $(function() { $(".author").viafautox( { select: function(event, ui){ var item = ui.item;} } }); }); lcsh and fast suggest the second function that was added to metadata maker is the lcsh suggestion using annif api.[43] annif (http://annif.org/) is a subject suggest tool for documents, originally developed by the national library of finland.[44] according to its webpage, annif can be trained through natural language processing and machine learning algorithms to support any kind of subject headings. to make annif support lcsh, the ld4p group used annif’s built-in algorithms and training corpus from the ivyplus platform for open data (pod)[45] and share-vde (virtual discovery environment)[46] to train annif (hahn, 2022;[47] khan, 2020[48]).[49] upon request, annif lcsh api returns a list of suggested lcshs labels, uris, and predicted scores. the list is sorted by the predicted score from high to low: the higher the score, the more relevant the subject heading is. // annif lcsh api response [ { "label": "clothing and dress--china--history", "notation": null, "score": 0.06058865785598755, "uri": "http://id.loc.gov/authorities/subjects/sh2003012066" }, { "label": "costume--china", "notation": null, "score": 0.014286939986050129, "uri": "http://id.loc.gov/authorities/subjects/sh85033251" }, { "label": "costume--china--history", "notation": null, "score": 0.014127381145954132, "uri": "http://id.loc.gov/authorities/subjects/sh85033252" }, { "label": "clothing and dress--history", "notation": null, "score": 0.011828765273094177, "uri": "http://id.loc.gov/authorities/subjects/sh2003012061" }, { "label": "clothing and dress--social aspects", "notation": null, "score": 0.008354970254004002, "uri": "http://id.loc.gov/authorities/subjects/sh85027167" }, { "label": "fashion--history", "notation": null, "score": 0.008040583692491055, "uri": "http://id.loc.gov/authorities/subjects/sh2008103592" }, { "label": "fashion--history--20th century", "notation": null, "score": 0.007795797660946846, "uri": "http://id.loc.gov/authorities/subjects/sh2008103594" }, { "label": "chinese poetry--translations into english", "notation": null, "score": 0.007471516728401184, "uri": "http://id.loc.gov/authorities/subjects/sh2008100615" }, { "label": "medicine, chinese", "notation": null, "score": 0.0065437802113592625, "uri": "http://id.loc.gov/authorities/subjects/sh85083125" }, { "label": "clothing and dress in literature", "notation": null, "score": 0.005863940808922052, "uri": "http://id.loc.gov/authorities/subjects/sh85033275" } ] using annif lcsh api, metadata maker can recommend ten lcsh terms given a book summary in any romance languages. users can select zero to ten lcsh terms by checking the provided checkbox. it is also possible to re-run the suggest function by updating the summary in the input box and clicking the suggest button. if one is not satisfied with the recommended keywords or uncomfortable using lcsh, users can still use an autocomplete faceted application of subject terminology (fast) heading search box to add keywords. figure 3. keyword (summary suggest and keyword search box) screenshot. // if a user clicks the #lcshsuggest button, based on the user’s // text input in the #summary box, lcsh will generate in the #lcshresponse div container $(function() { document.getelementbyid('lcshsuggest').onclick = function(){ document.getelementbyid("lcshresponse").innerhtml = ""; var summary = document.getelementbyid('summary').value; if (summary!=null){ var requests = "text=" + summary; var url = "http://annif.info/v1/projects/upenn-omikuji-bonsai-en-gen/suggest"; var xhr = new xmlhttprequest(); xhr.open("post", url, false); xhr.setrequestheader("content-type", "application/x-www-form-urlencoded"); xhr.setrequestheader("accept", "application/json"); xhr.onreadystatechange = function () { if (xhr.readystate === 4) { var data = xhr.responsetext; var jsonresponse = json.parse(data); console.log(jsonresponse); if (jsonresponse["results"] && jsonresponse["results"].length){ for (var i = 0; i < jsonresponse["results"].length; i++){ var lcshabel = jsonresponse["results"][i]["label"]; var lcshurl = jsonresponse["results"][i]["uri"]; document.getelementbyid("lcshresponse").innerhtml += '<input type="checkbox" name="lcsh" class= "lcshcheckbox" uri="'+lcshurl +'" value="'+lcshabel+'">'+lcshabel+'<br>';} } } }; xhr.send(requests); } }; }); bibframe output with recent updates, the bibframe output data now includes uris of personal names, lcsh, and fast headings in the monographs (ld) module. below is an example of a <bf:contribution>. the lcnaf uri of “shakespeare” is added to the agent node. both viaf and lcnaf uris of “shakespeare” are added as the value of identifiers. <!-example of a <bf:contribution> --> <bf:contribution> <bf:contribution> <bf:role> <bf:role rdf:about="http://id.loc.gov/vocabulary/relators/aut"/> </bf:role> <bf:agent> <bf:agent rdf:about="http://id.loc.gov/authorities/names/n78095332"> <rdf:type rdf:resource="http://id.loc.gov/ontologies/bibframe/person"/> <rdfs:label>shakespeare, william, 1564-1616</rdfs:label> <bf:identifiedby> <bf:identifier> <rdf:value rdf:resource="http://viaf.org/viaf/96994048"/> </bf:identifier> </bf:identifiedby> <bf:identifiedby> <bf:identifier> <rdf:value rdf:resource="http://id.loc.gov/authorities/names/n78095332"/> </bf:identifier> </bf:identifiedby> </bf:agent> </bf:agent> </bf:contribution> </bf:contribution> the second is an example of a <bf:subject>. the fast heading uri is added to the topic node. these are represented in bibframe metadata as below. <!-example of a <bf:subject> --> <bf:subject> <bf:topic rdf:about="http://id.worldcat.org/fast/1921567"> <rdfs:label>comedy plays [form/genre]</rdfs:label> <rdf:type rdf:resource="http://www.loc.gov/mads/rdf/v1#topic"/> <bf:source> <bf:source rdf:about="http://id.loc.gov/vocabulary/identifiers/fast"/> </bf:source> </bf:topic> </bf:subject> some consideration while developing new features for metadata maker, the authors found some issues with the apis and linked data sources. encoding viaf provides a single name authority file that combines name authority files from more than 40 organizations,[50] making it convenient for libraries to take advantage of linked data and obtain information about name entities from one source. yet, the aggregation process might cause some encoding issues in viaf records. for example, when one searches for “greta reyghere,”[51] the name includes empty boxes in the dropdown list returned by the api. the same issue also appeared in the viaf json record: the source of the name with empty boxes was dnb according to the viaf json record (see fig. 5).[52] however, the dnb record did not have anything anomalous.[53] it seems that the empty boxes in the name label only exist in the viaf record; aggregation setting in viaf might be the reason why. figure 4. empty boxes in viaf. figure 5. empty boxes in viaf json. name entities search scope when describing resources in bibframe editors, cataloging experts tend to use name authority files like lcnaf. however, non-catalogers or paraprofessional catalogers may not be aware of those sources and are more likely to rely on the linked data editor itself. it is expected that bibframe editors understand the different name entity search behaviors between experienced and nascent catalogers. specifically, there are two expectations for the name entity search function in linked data editors: (1) no restraints on the order of a name; and (2) supporting variant name searching. as many non-professional catalogers may not receive identity management (authority) training, it is not intuitive for them to search names following the marc 100 field format: “last name, first name.” it is also important to make bibframe editors connected to various linked data sources for name entities on the web and collect the name variances from as many sources as possible. to meet the two expectations, metadata maker adopts viaf auto suggest api for personal name searching. the viaf auto suggest api supports both preferred name and variant name searching without any name format or name order constraints. this flexibility allows non-catalogers to find the desired personal name in different ways. one bibframe editor that was tested for this project supports only preferred name label search. a korean author, han, shin-kap,[54] has name variances: “한신갑” and “shin-kap han” in his authority record. the bibframe editor only brought a result when the term “han, shin-kap” was searched, as it matched with the existing lcnaf record 100 field. the other two variant names did not bring any results as the selected editor does not support variant name search. the failed search may drive non-catalogers to create duplicate name entity records or use strings instead of uris to represent the person. figure 6. search han, shin-kap in a linked data editor. figure 7. search 한신갑 in a linked data editor. figure 8. search shin-kap han in a linked data editor. quality of authority data viaf authority data provided via json-linked data (json-ld) format does not always have detailed and granular information. viaf authority cluster endpoint allows catalogers to retrieve authority data in various formats.[55] the name-related elements in the json-ld representation of viaf authority records include family name, given name, alternative name, and name (full name). more complicated names may contain title, numeration, and other information about the entity. take “john paul ii, pope” as an example.[56] “pope” is the title of “john paul ii.” “john paul” is the papal name and “ii” is the numeration. however, in his viaf json-ld record (see below), “john paul ii” is treated as the family name and “pope” is treated as the given name, which is not correct. while this would not be a problem when using data models that do not require name parts information like bibframe, it could be a problem for schemas that have fields or attributes specifically designated for name part, e.g., first name and last name. // json-ld description of john paul ii, pope in viaf { ... "familyname" : [ "janis", "john paul ii", "juan pablo ii.", "jawién", "joannes paulus ii.", "ioannis pauli ii", "yūhạnnā būlus at-tanī", "ioann pavel ii", "wojytla", "jean paul ii.", "ויטילה", "vajtyla", "wojtila", "ii", "jean paul ii", "jean-paul ii.", "voitilah", "ján pavol ii.", "jános pál ii.", "ivan pavao ii.", "yuhạnnā-būlus at-tanī", "jawieň", "ṿoiṭilah", "juan pablo ii", "vojtyla", "ivan pavlo ii.", "ян павел ii", "johannes paulus ii.", "giovanni paolo‏ ii", "voityla", "jasien", "jasień", "yoḥanan paʾulus ha-sheni", "voitila", "xoán paulo ii", "ṿoiṭilah", "jasień", "ואיטילה", "gruda", "giovanni paolos ii.", "wojtyla", "johano paŭlo la dua", "войтыла", "jawień", "wojtiła", "johannes paul ii", "paulus", "yuḥannā-būlus at-tānī", "johannes paul ii.", "john paul ii.", "wojtyła", "보이티야", "アンジェイ", "jan paweł ii", "jean-paul ii", "yuhạnnā-būlus at-tanī", "보이티와", "janez", "jan paweł ii.", "jawien", "jan paweł", "jawień", "yūḥannā būlus at-tānī", "giovanni paolo ii", "janez pavel ii.", "ioannis paulus ii.", "yūḥannā būlus at-tānī", "vojtila", "iohannes paulus pp. ii", "yūhạnnā būlus at-tanī", "jan pavel druhý", "ioannes paulus ii.", "jānis pāvils ii.", "yuḥannā-būlus at-tānī", "joannes paulus ii" ], "gender" : "http://www.wikidata.org/entity/q6581097", "givenname" : [ "karal'", "pape", "papież", "karol józef", "al-bābā", "stanislaw andrzej", "carlo", "karols", "karol'", "stanisław a.", "ḳarol", "pope", "папа рымскі", "кароль", "קארול", "‏ papa", "karol joźef", "johannes", "andrzej", "pāvests", "papież", "ḳarol", "carol", "ヤヴィエニ", "karol j.", "카롤", "piotr", "saint", "lolek", "stanisław", "k.", "stanisław andrzej", "papa", "heiliger", "santo", "karolis", "karol jozef", "pavils", "pápa", "papa", "카롤 유제프", "karol józef", "karolʹ", "papst, heiliger", "papst", "al-bābā", "ii", "karol", "karel", "pavel", "pape", "john paul", "paus", "קרול" ], ... } testing after adding the viaf api into metadata maker, the authors did a very small scale unofficial usability testing in university of illinois with eleven participants: five paraprofessional catalogers who create original cataloging records as part of their responsibilities; two hourly catalogers who did not have cataloging knowledge but with language and subject knowledge; two graduate assistants; and two cataloging and metadata librarians. they were asked to create a record for a monograph book in sinopia and metadata maker and share their thoughts on two things: ease of use and knowledge/skills required to use each tool. the survey also had a section where testers could add their thoughts.[57] ease of use for the first question, testers could choose one answer from the following options: extremely hard hard, but can follow through it easy very easy figure 9. survey result: ease of use. eight participants said that metadata maker is easy to use (five chose “very easy” and three chose “easy”) while ten people said that sinopia is hard to use (five chose “extremely hard” and another five chose “hard, but can follow through it”). the survey reveals that the majority of participants prefer the simple interface of metadata maker to the relatively complex and verbose interface of sinopia. there is one person who chose that metadata maker is “extremely hard to use” and two people chose “hard, but can follow through it”. those who answered that metadata maker is hard to use are paraprofessional catalogers who create original records in oclc. during the follow-up interview, they expressed that they do not like the simple interface of metadata maker and the notion of creating short/minimum records. they want the bibframe editors to be similar to the oclc connexion, the tool that they are familiar with and allows them to create full level cataloging records. an undergraduate student with language skills answered that sinopia is easy to use. the student added that while there is a lot to learn and it takes time, they can follow through the sinopia by reading the information provided for each element. while sinopia allows users to view the output data in json-ld, turtle, n-triples, rdf table, and interface view formats, three participants commented that it is hard to check the outcome of their work in sinopia. it might be because those participants have not learned rdf data models and linked data serialization formats. metadata maker, however, allows records to be downloaded and viewed locally. those participants also added that it would be helpful to know the dataflow once the record is created in both editors. knowledge and skills required to use the editors the second multiple-choice question was to ask participants what kind of skills they thought were needed for the two bibframe editors, such as functional requirements for bibliographic records (frbr),[58] rda, bibframe, lcsh, and other controlled vocabularies, name authority, linked data, and marc. however, the authors quickly realized that the jargon and acronyms in this question caused misunderstandings for many participants as they did not know some or all options, especially the two non-catalogers who do not have cataloging knowledge/education. those staff members who routinely create original records also are not familiar with frbr, bibframe, and linked data. as a result, the answers to this question are all over the place as below: table 1. answers from 11 participants: knowledge and skills required to use the editors. sinopia metadata maker unsure none bibframe, marc, lcsh and other controlled vocabularies, name authority, linked data marc, none rda, bibframe, frbr, lcsh and other controlled vocabularies, linked data, need an extreme understanding of frbr terms and rda standards just to read/understand the interface i feel like you don’t actually need to know anything about cataloging standards to use this interface marc, lcsh and other controlled vocabularies, name authority, linked data marc marc, lcsh and other controlled vocabularies, name authority, linked data none bibframe none rda, bibframe, frbr, linked data, i did not use it enough to know all that one needs to know, but this is meant for experienced (and very technically savvy) catalogers none, if applicable, an non-english language. marc lcsh and other controlled vocabularies i do not know? rda basic book information everything basic book information none none however, one thing that is clear is that while many participants said there are things that are necessary to learn in order to use the bibframe editor, the majority of participants said no knowledge is needed to use the metadata maker. discussion and next steps the process of revamping metadata maker with linked data sources and bibframe output presented a possibility for building a linked data editor without any cataloging terminologies that can be used by anyone. the intuitive design, self-explanatory wording, and one-page web form break the learning barriers of bibframe cataloging and allow non-professional catalogers and language/subject experts to get involved in linked data metadata creation. as metadata maker is designed for generating “good enough” records, it can also serve as a quick bibframe generation tool for paraprofessional catalogers. however, the authors have learned some concerns from catalogers with regard to using this tool in practice, such as an oversimplified interface and unclear dataflow. the authors were perplexed by the variant degree of acceptance for metadata maker among survey participants. paraprofessional catalogers are inclined to use quasi-connexion editors with the option to describe detailed information about resources; whereas nascent catalogers might be more comfortable using linked data editors that do not require such prerequisite knowledge. the developers of linked data editors will need to balance those two needs. while the library domain has made significant progress in the development of and experimentation with linked data and bibframe production, there are still many things that the library community has to think further about and work together on to find a solution. first, a clear dataflow needs to be established. as of now, bibframe linked data created from the current bibframe editors are not automatically ingested into any integrated library system.[59] this was brought up by several staff members who tested sinopia. in addition, most vendors do not support bibframe import as of this writing. the authors acknowledge that the dataflow requires a possible new integrated library system that can work with metadata in different formats and with a different ontology. second, libraries may have a completely different data sharing method in the linked data environment compared with the current centralized shared database.[60] if that is the case, what would a data sharing model be like? if it is still possible to have a centralized linked data database, then who is going to manage it, and how is it going to be managed? third, a discussion of work distribution between human catalogers and machines needs to start. as machines can do marc to bibframe conversion and authority reconciliation work rather effectively, libraries might want to think about what machines can do and what cataloging and metadata professionals should do. if there are tasks that machines can do better, then it would be better to leave those to the machines, and identify what cataloging and metadata professionals should focus on, in terms of linked data creation and workflows. fourth, according to fortier, pretty, and scott (2022),[61] the understanding and knowledge of bibframe among canadian libraries is still low after close to two decades of ongoing discussion and development efforts. while it is important to understand the underlying structure of bibframe and linked data, it would be worthwhile to think about how much training is adequate for cataloging professionals and how much integration of rda terms into the bibframe editors is necessary for the transition to linked data creation. or, maybe what libraries really need is a linked data editor rather than a bibframe editor. if there are problems in understanding bibframe and rda among ourselves, it would be much more difficult for users on the web to understand what kind of data we are sharing. about the author greta heng (orcid: 0000-0002-3606-6357) is cataloging and metadata strategies librarian at san diego state university. myung-ja (mj) k. han (orcid: 0000-0001-5891-6466) is a professor and metadata librarian at the university of illinois at urbana-champaign. bibliography [1] library of congress. bibliographic framework initiative. https://www.loc.gov/bibframe/. [2] world wide web consortium (w3c). rdf. https://www.w3.org/rdf/. [3] wennerlund, b., & berggren, a. (2017). leaving comfort behind: a national union catalogue transition to linked data. paper presented at: ifla wlic 2019 – athens, greece – libraries: dialogue for change in session s15 – big data. in: data intelligence in libraries: the actual and artificial perspectives, 22-23 august 2019, frankfurt, germany. [4] french national library. semantic web and data model. https://data.bnf.fr/en/semanticweb. [5] german national library. linked data service. https://www.dnb.de/en/professionell/metadatendienste/datenbezug/lds/lds_node.html. [6] library of congress. marva editor. https://bibframe.org/marva/editor/. [7] wikidata. wikidata main page. https://www.wikidata.org/wiki/wikidata:main_page. [8] godby, j., smith-yoshimura, k., washburn, b., davis, k., detling, k., eslao, c., folsom, s., li, x., mcgee, m., miller, k., moody, h., thomas, c., & tomren, h. (2019). creating library linked data with wikibase: lessons learned from project passage (pp.70). oclc research. https://doi.org/10.25333/faq3-ax08. [9] han, m. k., ream-sotomayor, n. e., lampron, p., & kudeki, d. (2016). making metadata maker: a web application for metadata production, library resources & technical services, 60(2), 89–98.; all the source codes are available in github: https://github.com/dkudeki/metadata-maker; metadata maker is still in the exploratory phase and currently only supports linked data cataloging for monographs. [10] michael, b., & han, m. j. k. (2019). assessing bibframe 2.0: exploratory implementation in metadata maker. proceedings of the international conference on dublin core and metadata applications, 26-31. [11] non-catalogers refer to people who do cataloging work but do not have adequate cataloging experience or may not need it as they do not pursue a career in cataloging. [12] van der werf, t. (2021, march 4). next generation metadata… it’s getting real! hanging together, oclc research blog. https://hangingtogether.org/next-generation-metadata-it-is-getting-real/. [13] dalgord,c. shared entity management infrastructure project update. oclc. https://www.loc.gov/bibframe/news/source/bibframe-from-home-oclc-update.pptx. [14] linked data for libraries. https://wiki.lyrasis.org/pages/viewpage.action?pageid=41354028. [15] linked data for libraries labs. https://wiki.lyrasis.org/pages/viewpage.action?pageid=77447730. [16] linked data for production. https://wiki.lyrasis.org/pages/viewpage.action?pageid=74515029. [17] linked data for production: pathway to implementation. https://wiki.lyrasis.org/display/ld4p2. [18] linked data for production: closing the loop. https://wiki.lyrasis.org/display/ld4p3. [19] lnenicka, m., kopackova, h., machova, r., & komarkova, j. (2020). big and open linked data analytics: a study on changing roles and skills in the higher educational process. international journal of educational technology in higher education, 17(1), 1-30. [20] el-sherbini, m. & klim, g. (1997). changes in technical services and their effect on the role of catalogers and staff education: an overview. cataloging & classification quarterly, 24(1-2), 23-33; zhu, l. (2012). the role of paraprofessionals in technical services in academic libraries. library resources & technical services, 56(3), 127-154. [21] van der werf, next generation metadata… it’s getting real! [22] library of congress, bibframe editor. https://bibframe.org/bfe/index.html. [23] library of congress, marva. https://bibframe.org/marva/editor/. [24] linked data for production: pathway to implementation. sinopia. https://sinopia.io/. [25] rda toolkit: https://www.rdatoolkit.org/. [26] bibframe interoperability group. (2022. april 15). terms of reference. https://www.loc.gov/aba/pcc/bibframe/taskgroups/big/big-tor.pdf. [27] lorimer, n.(2022, march 8). re-use or copy? redefining copy cataloging in a linked data environment. ala copy cataloging ig, online. https://docs.google.com/presentation/d/1ukxcdjea-cwmxnfixfibdpbcvn_jxvmymoojgzibi9o/edit?usp=sharing. [28] library of congress. appendix c – minimal level record examples. https://www.loc.gov/marc/bibliographic/bdapndxc.html. [29] http://quest.library.illinois.edu/marcmaker/dataset/. [30] http://quest.library.illinois.edu/marcmaker/. [31] aka, monograph (linked data), http://quest.library.illinois.edu/marcmaker/monoviaf/. [32] http://quest.library.illinois.edu/marcmaker/ebooks/. [33] http://quest.library.illinois.edu/marcmaker/govdocs/. [34] http://quest.library.illinois.edu/marcmaker/maps/. [35] http://quest.library.illinois.edu/marcmaker/microfilms/. [36] http://quest.library.illinois.edu/marcmaker/scores/. [37] http://quest.library.illinois.edu/marcmaker/serials/. [38] http://quest.library.illinois.edu/marcmaker/theses/. [39] bibframe is only added to two monograph modules for now. [40] oclc developer network. authority cluster resource. https://www.oclc.org/developer/api/oclc-apis/viaf/authority-cluster.en.html. [41] dnb was selected as an alternative name label source because (1) it provides linked data service; and (2) it is a national library for a non-native english speaking countries which may compensate for lcnaf. [42] https://github.com/dkudeki/metadata-maker/blob/monoviaf/lcsh/lcshsearch.js. [43] suominen, o., inkinen, j., virolainen, t., fürneisen, m., kinoshita, b. p., veldhoen, s., sjöberg, m., zumstein, p., neatherway, r., & lehtinen, m. (2022). annif (version 0.60.0-dev) [computer software]. https://doi.org/10.5281/zenodo.2578948; https://api.annif.org/v1/ui/. [44] annif github repository. https://github.com/natlibfi/annif. [45] ivyplus platform for open data. https://pod.stanford.edu/. [46] share-vde (virtual discovery environment). https://www.svde.org/. [47] hahn, j. (2022, june 20). cataloger acceptance and use of semiautomated subject recommendations for web scale linked data systems. 87th ifla world library and information congress (wlic) / 2022 in dublin, ireland. https://repository.ifla.org/handle/123456789/1955. [48] khan ,h. (2020, march 10). annif use and explanation. linked data for production: pathway to implementation. https://wiki.lyrasis.org/display/ld4p2/annif+use+and+explanation. [49] when accessed http://lcsh.annif.info/ in october 2022, annif lcsh api project updated its vocabulary sources: “ivyplus-tfidf” was changed to “penn-fasttext-en” (penn (lcsh english) conference papers and proceedings), “upenn-omikuji-bonsai-en-gen” (upenn (lcsh english) all genres), and “upenn-omikuji-bonsai-spa-gen” (upenn (lcsh spanish) all genres). [50] virtual international authority file. https://www.oclc.org/en/viaf.html. [51] viaf authority record for de reyghère, greta. retrieved on september 11, 2022, from http://viaf.org/viaf/69118441. [52] viaf authority record in json for de reyghère, greta. retrieved on september 11, 2022, from https://viaf.org/viaf/69118441/viaf.json. [53] dnb authority record for de reyghère, greta. retrieved on september 11, 2022, from https://hub.culturegraph.org/entityfacts/134496175, and https://d-nb.info/gnd/134496175. [54] viaf authority record for han, shin-kap. retrieved on september 11, 2022, from http://viaf.org/viaf/198153409742041581752. [55] oclc authority cluster resource. https://www.oclc.org/developer/api/oclc-apis/viaf/authority-cluster.en.html. retrieved on october 5, 2022. [56] viaf authority record in json-ld for john paul ii, pope. retrieved on september 20, 2022, from https://viaf.org/viaf/35605/viaf.jsonld. [57] we chose sinopia over other bibframe editors because it is created for the community and has pcc templates that have been tested out by many catalogers. we also understand that the purpose of the bibframe editor and metadata maker are different. [58] the international federation of library associations and institutions. functional requirements for bibliographic records (frbr). https://www.loc.gov/marc/bibliographic/bdapndxc.html. [59] there are some unofficial statements that folio and ex libris have been working on bibframe data import. but as of october 6, 2022, there has not been a bibframe data import function released by them. [60] library of congress. bibframe and the pcc. https://www.loc.gov/aba/pcc/bibframe/bibframe-and-pcc.html. [61] fortier, a., pretty, h., & scott, d. (2022): assessing the readiness for and knowledge of bibframe in canadian libraries, cataloging & classification quarterly. https://doi.org/10.1080/01639374.2022.2119456. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – editorial introduction: how things change mission editorial committee process and structure code4lib issue 21, 2013-07-15 editorial introduction: how things change introducing issue 21 by terry reese one of the things that i’ve always appreciated about this community has been its ability to accept changes.  maybe this is because of the type of work that we do, technology, it’s rarely stable, but i think that one of the things that has made the code4lib community so successful and vibrant has been its ability to be malleable, to find common ground and move on.  a great example of this process happens every year in the run up to the code4lib conference.  as each new host works through their own unique set of challenges, the conference changes, it evolves and moves on.  as someone that has been fortunate enough to attend every code4lib conference, starting in 2006 when 80 library technologies descended on corvallis, oregon, it’s been easy to track the changes in both the community and our seminal event. so it should come as no surprise that this same flare for change, this same need to continue to evolve and improve processes exists with the journal.  as editors cycle on and off, processes are looked at, evaluated, maybe changed.  likewise, as new personalities are added to the editorial committee, new feature ideas may be discussed.  in the short time that i’ve been a part of the editorial committee, a number of such discussions have been had, and i imagine, will continue to be had.  it’s how we keep the journal fresh, and how we continue to work together to produce a product that represents not only the broad interests of our community, but does so in a way that honors a community that values openness and transparency. one of the benefits of being the coordinating editor is the ability to try something new, and in this issue we experimented with how articles were selected for this issue.  the editorial committee is always looking at the process that we use to evaluate proposals.  each issue regularly receives a significant number of proposals each issue, and it’s up to the editorial committee to select the best proposals for inclusion into the journal.  in the past, proposals were evaluated as they were submitted, a process that allowed the committee to provide very quick feedback to potential authors.  the problem for the editorial committee (myself included), is that as the issue fills up, it can be difficult to judge each new proposal without considering those that came before it. when faced with this type of problem, the obvious answer is to hack a solution.  rather than provide a final evaluation of each proposal as it came in, we waited until the proposal solicitation period had passed, and then looked at what we got.  did this process work better?  i don’t know.  on the one hand, i think it may have been harder for authors because there was more downtime between submitting a proposal and a response of acceptance – but as an editorial member, i personally felt like i had a much easier time evaluating each proposal on equal footing.  as i look back and assess the process, i definitely think that there are things that we learned and can apply to future journal issues.  for example, if we were to continue to tweak the evaluation process, the next coordinating editor should probably try to do a better job keeping potential authors abreast of the process. as we put this issue to bed and start focusing on issue 22, i expect that the editorial committee will decompress and continue to come up with a better process.   and you know what, that’s ok…because that’s what code4lib is all about. summary of issue 21 it’s been my great pleasure to be the coordinating editor for issue 21.  as one of the newer members of the editorial committee, it’s been interesting seeing how each new issue of the journal takes on its own personality and tells its own story.  while the editorial committee doesn’t solicit or select articles based on a specific set of agreed upon themes, each issue that i’ve been a part of seems to organically take shape around a few common topics – and it appears that this issue is no different.  issue 21 is a substantial body of work, consisting of 10 articles coalescing around linked data, api usage, and the broad topic of finding new and novel ways to repurpose tools and data available in the library.  what’s more, this issue has a distinctly international flavor, with articles being provided by authors from the united states, canada, germany, and the united arab emirates; issue 21 highlights both the universality of many of the challenges currently facing the library community, as well as the success the code4lib community has had in bridging culture and language through a shared interest in technology. one of the new themes that pop up in this issue are around the practical implementation of linked data.  i expect that future issues and authors will continue to wrestle with and discuss new and exciting ways in which linked data will allow libraries to become more interconnected with the world outside of libraries – and i expect that this issue will signal the start of that trend.  m. cristina pattuelli, matt miller, leanora lange, sean fitzell, and carolyn li-madeo’s paper, “crafting linked open data for cultural heritage: mapping and curation tools for the linked jazz project” and götz hatop’s paper entitled: “integrating linked data into discovery” provide an interesting look at two real life projects seeking to integrate linked data and linked data concepts into their design.  these articles talk about some current tools, as well as some of the successes and challenges found in this approach. in the proud code4lib community tradition, this issue also provides a number of articles for readers interested in learning how to do things better.  ted diamond, susan price, and raman chandrasekar’s article “actions speak loaders than words: analyzing large-scale query logs to improve the research experience” offers a timely look at how libraries can take an active role in improving the academic research experience for their users.  while diamond, et al’s article specifically looks at summon query logs, the authors offer up a novel approach for any library currently utilizing a unified discovery service.  kyle banerjee and maija anderson take a fresh look at utilizing existing tools and utilities to leverage facial recognition software to streamline the creation of image collection metadata in “batch metadata assignment to archival photograph collections using facial recognition software,”  while donald moses and kirsta stapelfeld provide a closer look at some of the new features that are a part university of prince edward island’s institutional repository utilizing the islandora repository system in “renewing upei’s institutional repository:  new features for an islandora environment.”  richard anderson writes about stanford’s decision to utilize the “moab” design for versioned archiving of digital objects in “the moab design for digital object versioning” and edward iglesias and arianna schlegel explore how libraries can leverage raspberry pi devices to create inexpensive electronic signage in “using a raspberry pi as a versatile and inexpensive display device.”  lisa gayhart, in “out from behind the firewall: towards better library it communications” provides a thought provoking piece on the need to improve it-specific communication within the library. finally, libraries continue to utilize a wide variety of services to improve discovery, enhance displays, and improve user experiences.  services are wide and varied, but what each have in common are application programming interfaces (api) that facilitate this sharing of information.  issue 21 features two articles specifically looking at a number of widely used library services.  the first, by thomas hodge and james macdonald entitled “relevance and phrase searching in summon:  looking under the hood” provides an excellent analysis of some of the black magic behind the summon api search and some ways to ensure that queries made using the summon api return only relevant results.  the second takes a closer look at three different bibliographic services, comparing developer ease of use, license restrictions and data provided in “comparing the librarything, oclc, and open library isbn apis.” as one can see, issue 21 covers a wide range of topics with something for just about everyone.   whether you are a data wonk, an administrator, or just someone interested in library technology and the challenges therein, the journal and its authors continue to illustrate the wide and ever expanding interests and talent found within the library community. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – developing weeding protocols for born digital collections mission editorial committee process and structure code4lib issue 43, 2019-02-14 developing weeding protocols for born digital collections as collections continue to be digitized and even be born digital, the way we handle collection development needs to also shift towards a digital mindset. digital collections development are not so much concerned about shelf or storage space, as expansion can be as simple as procuring a new hard drive. digital collections, when not archival, need to focus on issues of access and accessibility. for a born digital library, quality and usefulness must be the primary factors in the collection development policy. this article will walk through the steps taken by one digital library (pbslearningmedia.org) to assess their collections with an eye to quality and user experience as well as a multi-phase deaccessioning project that occurred and is ongoing. the process, including the multi-iteration drafting of subject specific rubrics, targeted to the needs of the site’s core audience. it also included the quantitative assessment of thousands of items in the collection and the distribution of qualitative and quantitative data to stakeholders across the country. special attention to the setting of minimal required standards and the communication of those standards was paid. finally, as this process is now an ongoing review schema for learningmedia, the article will discuss the issues faced in this project, recommendations for other organizations attempting their own digital weeding/deaccessioning projects, and the plans for the future of the project. by athina livanos-propst introduction in 2018, pbs learningmedia (an online destination offering free access to thousands of classroom-ready resources, made possible through a partnership between  pbs and wgbh) took on a project to review and refresh its library of teacher-focused, born digital materials. what follows is an overview of the process that was undertaken by pbs learningmedia. it is possible for organizations of any size to adapt the process to their own non-archival digital collections. the software utilized are generally accessible and open to librarians at many different levels of technical expertise. pbs learningmedia was launched in may of 2011. at launch, the site’s goal was to make pbs’s media and teacher materials accessible in one digital destination, as well as to distribute high-quality educational resources from pbs member stations and other organizations. these materials included full length videos, clips, documents, still images, games, and lesson plan. the editorial standards and policies when the site was founded were carefully crafted to ensure that the collection would be robust and beneficial to patrons. since then, editorial standards have shifted to include a requirement for contextualizing video clips, a desired reduction in linking to external sites, and combining small pieces into larger, more robust teacher materials. in the summer of 2017, the collection reached a critical mass, surpassing 100,000 available resources. the content team realized that due to the acquisition policy shifts, the full collection was no longer up to the current editorial standard. a full-scale audit of the content collection began to weed out materials that were no longer up to current editorial specifications and to inform stakeholders of the removal of those materials and the reasons for removal. the collection had also never been previously been reviewed with deaccessioning in mind. to that end, a process was developed to review and assess the materials and begin this process for the first time. considerations & rationale at 100,000 resources, the collection was too large for users to easily navigate and find the high-quality resources needed for their classrooms. a few choices were made at the beginning of the process. the nearly 45,000 resources that were single images would not be assessed. these pieces were used minimally. therefore, it was decided that select image collections would be removed from search, but still accessible via direct links. it was also decided that the weeding review would only look at materials that had been added to the collection prior to 2015. because the acquisition and editorial standards became better enforced around that time, that would be the area of the collection that would likely experience the largest loss during the weeding process. with that narrowed scope, that left 7,457 resources to be reviewed in the humanities and 6,827 in stem fields (science, technology, engineering, and math). the process that will be outlined here is focused exclusively on what was done to review the humanities materials. the stem materials were reviewed by a partner organization and followed a different process. organizations wishing to perform their own assessment of born digital content should note that the humanities process is not exclusive to the humanities and could be readily adapted to the sciences. the process occured in two primary phases. the first would be the development of a rubric tool with which we could assess the content on the site. the second phase would be the actual utilization of those rubrics by members of the content team to perform the assessment itself. the results from phase two would then be shared with the appropriate contributor as a tool for requesting and making changes to bring the resource materials up to the editorial standard. design & iterations it is vital when undergoing any weeding process to consider your patron community. in this case, our community was educators, which have different user needs than other patron groups. our first step was to develop an impartial tool to grade each resource that kept the needs of our patron community at the forefront. multiple weeding guidelines for physical collections were reviewed (see further reading at end).  we selected six primary areas to focus on for the structure of the resource review: accuracy, currency, appearance, relevance, contextualization, and usage. for the purposes of this review and assessment, each of the terms was defined in the following manner: accuracy is the information presented in the resource technically accurate and factually correct? example: shakespeare was a famous writer in his time. example: jackson pollock did large scale, semi-performative abstract paintings. example: an explanation of the sounds letters make to teach chunking. currency is the resource balanced? are there missing perspectives due to when the materials was made? was this accurate at the time, but no longer reflect current understanding? example: does the resource state that lincoln freed slaves, or is a more nuanced approach taken? example: does the lesson on uncle tom’s cabin include both the inspiration it gave abolitionists as well as the backlash from the african american community? relevancy is this easy for teachers to use? how well is the resource aligned to standards? does the wording of the intro paragraph aid in the usage of the materials? example: the standards listed under the resource all make sense. example: resource approaches a standard topic, but via a pop-culture lens to encourage interest in students. appearance does the video quality meet current standards? is the audio quality intelligible? example: is the video quality distractingly granulated? example: are there images of technology or fashion that are so outdated it could cause a distraction from the learning material? contextualization does the resource have support materials attached? are there contextualizing questions and/or activities ‘baked in’ to the primary resource? what is the overall quality of the contextualization efforts? example: the resource has support materials for handouts, quizzes, and discussion questions. example: the resource incorporates a quiz within the media itself. example: do the support materials call for simple recall tasks or for a larger understanding and interpretation? usage how often has the resource been viewed? how often has the resource been favorited? raw numerical counts based on the information listed on the resource page. after primary areas of focus had been established, the next phase of the review process was able to begin. this stage involved hiring a series of subject matter experts (smes) to make specific rubrics, one for each subject area that pbs learningmedia covers. these experts were teachers from across the country, who could speak directly to the content needs of our primary patron base. the smes were contracted to deliver three iterations of rubrics during the development process. for each iteration, the subject specific rubrics would be tested by a small group of internal team members. the internal team would grade two resources per subject in accordance with the rubric draft provided. the smes were then able to take the grades in front of them and see how they, as teachers, would have assessed the same materials. in this manner, we could test to see that the language for the rubrics both meant something concrete to teachers and was understandable for those outside the education field. we could also be assured that our standards for retaining content were in line with usefulness to our core user base. all smes provided first drafts of rubrics that covered every point of which they could think, both about the quality of the materials and the quality of the site itself. the smes threw all potential points on their grading forms. this was both beneficial and problematic. it was beneficial in that it allowed the smes to ask all their questions and get a better sense of the limitations of the system. smes asked questions about accessibility features that were already integrated into the site, colors, text size and formatting, and similar issues. these systematic issues for the site, while important, were not a part of the resource quality review. it was problematic in that the smes had to be strongly course-corrected away from being bogged down by formatting of the site itself instead of classroom focused needs. it was also useful, in that it allowed for detailed notes on site design that were passed along to the usability design team, working on a different, yet parallel, project. the second iteration of the rubrics was more focused on a resource’s viability in the classroom. the questions brought forward on this rubric draft from the smes focused on areas of content and technical specifications that were able to be edited by either the pbs learningmedia content team or the contributors themselves. in retrospect, it would have been helpful to have the smes more fully prepared about the site’s existing specifications and adaptability. that would have allowed for the first draft to be more focused. the second iteration also saw the largest change in approach to the project overall. at the suggestion of one of the internal team members, we opted to transition from a grading system that allowed for huge amounts of variation to a stripped down three tier option. each question would be assessed on the simple scale of “pass,” “needs work,” or “fail.” resources would either be good enough, fixable, or below our standards. the third and final iteration of the rubric process focused predominantly on getting our smes to write out what “pass,” “needs work,” or “fail” would look like for each question in their subject area. the descriptions for what was good enough, what a teacher could somewhat work with, and what a failure looked like were highly valuable in communicating educator needs to non-educator content producers. it was this final step that allowed our production teams to know what they were looking for and to more accurately assess if the materials on the site were meeting teachers’ needs. once the rubrics were developed, the first major hurdle of the weeding project was overcome. we now had a tool with which to assess the born digital content that would be understandable by any user, while ensuring value to our core patron base. the rubrics were loaded into a single, combined google form. the questions that applied to all pieces, regardless of subject matter, were asked first, then a pointer question that led to the the subject-specific questions, with a final section of data metrics and ‘nice to have’ features completing the form. the combined form came in at just under 37 pages. that was spread out over seven subject areas, so each individual resource went through only a small portion of the complete rubric. each subject was assigned to an individual with some level of subject expertise in the given area. library staff with the right backgrounds were pulled in from other projects as well as retaining three of the smes from the rubric writing. each individual was given a list of resources to review that had been published on the pbs learningmedia platform prior to 2015 and tagged to their designated subject area. reviewers were tasked with reviewing what pbs learningmedia defines as a “single resource page,” that is watching the video(s) that were on each given page as well as any accompanying documentation and teacher support materials. each review took, on average, 10-15 minutes to complete and enter results into the google form. the phrase two review process took four months. during that time, small corrections were made to the form, mostly clarifications of terms and combining and moving of questions. during the first two months of the review, weekly check-in meetings were held. these meetings allowed for consistent interpretation of the rubrics across all team members. after the first two months, the team’s work was consistent and of high enough quality that it was determined that the hour-long meeting time each week could be better spent doing the review work itself. assessment once the review phase was completed, the task of assembling and assessing the data began. we used the fact that google forms automatically outputs to google sheets to our advantage. the raw data was downloaded from google to compile individual sheets for each organization that had contributed to the database. scores for resources were calculated into the following fields: rubric score a simple percentage score based on how the resource performed in every pass/needs work/fail question answered on the rubric three points were awarded for a pass, two for a needs work, and one for a fail formula to assess graded areas for each resource line =countif(g2:ei2, “pass*”) formula to calculate overall percentage score =(average((ej2*3)+(ek2*2)+(el2*1))/sum(ej2:el2)/3) factual a check on a question in the universal section of the rubric, designed to ensure that only pass level factual materials will be approved formula for results check on factual question =if(countif(p2, “pass*”)+countif(ay1, “pass*”)+countif(bc1, “pass*”)+countif(cc1, “pass*”), “good”, “review”) support materials present a check on a several cells that checks if the resource has been given support materials in any format formula for check of presence of support materials, multiple questions =if(sum(countif(aa2, {“pass*”;”needs work*”}),countif(ab2, {“pass*”;”needs work*”}),countif(ak2, {“pass*”;”needs work*”}), countif(aw2, {“pass*”;”needs work*”}),countif(bd2, {“pass*”;”needs work*”}),countif(bg2, {“pass*”;”needs work*”}),countif(bj2, {“pass*”;”needs work*”}),countif(bp2, {“pass*”;”needs work*”}),countif(cf2, {“pass*”;”needs work*”}),countif(ec2, {“pass*”;”needs work*”})), “present”, “missing”) overall score pass: rubric score, 90% or above factual, positive result support materials, positive result needs work rubric score, 80% or above factual, positive result fail rubric score, below 80% factual, negative result formula to assess final score, dependent on above scoring results =if(and(em2>=0.89,en2=”present”,eo2=”good”),”pass”,if(and(em2>=0.89,en2=”missing”,eo2=”good”),”needs support materials only”,(if(and(em2>=0.8,en2=”missing”,eo2=”good”),”needs work”,(if(and(em2>=0.8,en2=”present”,eo2=”good”),”needs work”,if(and(em2<0.8,en2=”missing”,eo2=”review”),”fail”,”fail”))))))) additionally, all results were color coded, as well as all notes fields in the results panel itself. figure 1. a more condensed summary tab was also provided for a quick assessment of the materials. figure 2. the data for each score section was also depicted in a series of charts. figure 3. sample view of a the executive summary tab, intended to convey key information to report recipients. figure 4. graphic breakdown of full data, showing the scores in key areas as well as the overall score. the prepared sheets were sent to each contributor who had added material to the site prior to 2015. even though much of the feedback was that many materials would have to be pulled due to the elevation in standards, contributors were mostly positive in their response. they readily accepted that their materials would not be hosted indefinitely, and they were open to making now required changes on existing materials. many contributors were also very thankful to have a list of questions they could ask of their own materials before adding new pieces to the collection. there were some contributors who took the feedback as an opportunity to reorganize, restructure, or rearrange their existing materials to allow for educational goals to be met. in this way, many resources that originally scored poorly could be spared by combining them together to create a larger narrative that was able to achieve a passing score. another benefit of sending out the result forms was that it encouraged several lapsed contributors to become interested in adding to the platform once again. we have seen a spike in potential users signing up for training session about how to utilize the database for uploading new content. notes one of the issues that we planned around was the awareness that our collection numbers would drop significantly. all resources that received a grade of “fail” would be removed from the website, and all that received a grade of “needs work” would be un-indexed in search, but still available to users with a direct link. we tied the launch of the weeding project results to align with an overall site redesign. the thought was that the materials on the refreshed site would all align to the new acquisition standards. timed with the new look and feel, both changes could be positioned as the new and improved pbs learningmedia. this choice proved to be a wise one, as we had minimal reports from users asking about the reduced resource count. moving forward, we are continuing to review content that was added to the site after our initial cut off point of january 1, 2015. the current plan is to be able to review materials as they hit three years of age in the system. we have reduced the review team to 1.5 full time staffers, who are able to review an average of 500 resources per month. we also hope to be able to convert the forms being sent to contributors to an automated form. about the author athina livanos-propst is a digital librarian for pbs education, overseeing the metadata implementation and strategy for pbs learningmedia. she earned her mlis from the catholic university of america, where she specialized in cataloging and special collections. her primary focus is born digital collections and supporting linked data. further reading larson, jeanette. 2008. crew: a weeding manual for modern libraries. texas state library and archives commission [internet]. [cited 2019 jan 23]; available from: https://www.tsl.texas.gov/sites/default/files/public/tslac/ld/pubs/crew/crewmethod08.pdf vnuk, rebeca. 2016. weeding without worry. american libraries [internet]. [cited 2019 jan 23]; available from: https://americanlibrariesmagazine.org/2016/05/02/library-weeding-without-worry/ subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – editorial introduction – issue 5 mission editorial committee process and structure code4lib issue 5, 2008-12-15 editorial introduction – issue 5 welcome to the 5th issue of the code4lib journal. we’ve come a long way in just over a year! we hope you take a few minutes to celebrate those accomplishments with us as you explore issue 5 and read about the innovations, ideas, and experiences shared there. let’s learn from each other. by emily lynema coordinating editor, issue 5 celebrating a year this 5th issue of the code4lib journal celebrates our first full year of publication. i think i can speak for the entire editorial committee when i say we are very excited to see that the journal is still going strong a year later. what have we accomplished in a year? it has been a busy year for the code4lib journal. for starters, we’ve done a lot of talking. google groups tells me we’ve sent 3,758 message covering 865 topics on the private c4lj-articles email list alone [1]. while some are the inevitable email spam, it’s still a lot of words. but we’ve done more than just talk; all this discussion has led to the publication of 44 articles, editorials, and columns authored by about 75 different community members (give or take a few) across 5 issues. overall, we’ve responded to about twice that many proposals. we’re happy to see so much community interest in publishing in our venue. in conjunction with the fantastic work of our authors, all this has been accomplished by a fairly small group of people. we currently have an editorial committee of 12, having gained 3 additional hands while only saying farewell to 1 in the past year. many thanks go out to eric lease morgan for his year of service with the journal, especially his masterful coordination of issue 2. and we have a few readers, as well! google analytics claims that journal.code4lib.org has seen 58,316 visits in the past year, unsurprisingly spiking with the release of each new issue. the most visited article so far? free and open source options for creating database-driven subject guides, co-authored by the editorial committee’s very own edward corrado. the most common referrers pointing folks into the pages of the code4lib journal are code4lib.org, google.com, and bloglines.com. however, we still see about 40% of direct incoming traffic and another 25% coming from search engines. in fact, the code4lib journal content is starting to pop up in more and more places. the journal has been added to the directory of open access journals, where we upload article metadata for each issue. in fact, the doaj is actually the 4th most frequent referrer to the journal site. we have established cc-by licensing for all article content to make it more freely re-usable. representatives from the editorial committee are currently working with ebsco to determine if our full-text content can be included in the library, information science, and technology abstracts database. and if you use sfx as a link resolver, you’ll find the code4lib journal has been added to its knowledgebase. we’re doing whatever needs to happen to get the work done. 9 wiki pages, 8 wordpress pages, 4 wordpress custom plugins, 3 google groups pages, 3 google docs spreadsheets, 2 wordpress templates, and 2 email lists later we’re still having fun. what more do we want? from the beginning, we have maintained that the code4lib journal is all about you, the readers. we envisioned a journal that lowered barriers to publication, focused on practical, useful, applicable content, and encouraged an open conversation with readers. this desire for an open conversation is one of the major reasons we chose wordpress as a publishing platform for the journal. 5 issues later, we hope that we’ve succeeded in lowering barriers to publication and we’re pretty sure we’ve provided some interesting and practical articles to the community. but we’re still hoping to see these accomplishments provide a platform for an ongoing and open conversation about problems and solutions, questions and answers, ideas and projects. out of 44 published articles, we’ve only seen 37 (non-spam) comments. i encourage all of you, our readers, to consider joining the dialogue by posting comments on the articles you read with questions, opinions, and ideas. oh, and you should feel encouraged to submit proposals, too! libraries, technology, and everything else as saddened as i am by the possibility that i won’t be able to attend this year’s annual code4lib conference, it makes me more appreciative of the fact that i can still contribute to this vibrant community through the journal, the email list, and the irc channel. time and again, i’ve turned to the helping hands there to discuss ideas and to point me towards technologies and tools that i can employ in my workplace. as libraries around the country face budget cuts and freezes, it’s increasingly important to maintain a cohesive, helpful virtual community. sharing ideas and expertise can help us solve problems cheaper and faster; an excellent option as many face a short-term future with limited staff and resources. in the midst of an economic crisis, the show must go on for libraries. innovation remains important as an increasingly diverse array of resources becomes available and our patrons attempt to navigate a sea of confusing options among which the library probably doesn’t even appear on the first page. libraries must effectively choose and use technology to help reach these patrons. this means we have to find new ways to do many things; new ways to reach out into our patrons’ spaces, new ways to support their preferred style of doing business, new ways to make our systems and metadata flex for better finding and discovering, new ways to send our data out into the web and bring the web back into our local systems, new ways to be more intuitive and more responsive. it’s a lot of new. fortunately, it’s not the first time that libraries have faced change. librarians working with technology are not working alone. and we continue to see growth and innovation in the tools available, including a surge in free and open source technologies [2]. issue 5 the 5th issue of the code4lib journal continues to contribute to this community of sharing and helping. it features 10 articles focusing on new ideas and tools, how to information, and sharing of experience gained over time. new ideas and tools ilana kingsley and mark morlino describe the process of creating a dvd browser for patrons by mashing up data from the library catalog, imdb, rotten tomatoes, and freecovers.net and integrating it into a customized drupal module. jill strass shares how the folks at st. olaf college used an excel workbook to wrangle together scanned newspaper images and individual file names to create a browsable, searchable collection in contentdm without requiring manual cataloging. wayne graham discusses creating a library facebook application using facebook athenaeum to reach out to patrons who use this social networking tool frequently. cody hanson, shane nackerud, and kristi jensen demonstrate how the university of minnesota libraries have taken advantage of enterprise “affinity strings” generated by peoplesoft to provide customized library resource pages within the campus portal and to gather more targeted usage statistics for electronic resources. chris catalfo of liblime introduces the code4lib community to ‡biblios, an open source cataloging editor that is currently integrated with the koha ils. how to noel peden describes the process perfected over the past 2 code4lib conferences to capture presentations and make them available online for those unable to attend. sharing experience henrik lindström and martin malmsten from the national library of sweden share how their team integrated user-centered design processes and agile development methodologies to create a new, adaptable, patron-focused search for the swedish national union catalogue. dianne dietrich, jennifer doty, jen green and nicole scholtz suggest a set of guidelines to use when thinking about terminating vs. reviving digital projects created by your predecessors. kelley mcgrath and lynne bisko describe in detail the outcome of their attempts to automatically extract work-level data from marc bibliographic records for moving images. dale askey provides a rousing editorial on why libraries struggle to effectively share the light-weight “doodads and widgets” we use to glue our web sites together and how we might overcome these tendencies. we hope that you enjoy issue 5! notes [1] note that all numbers in this introduction are slightly estimated so as to provide the spirit of the past year without worrying too greatly about 100% accuracy. [2] i’ve provided a link to the project page of oss4lib as a place to get started. i’m sure there are many other relevant open source tools available. please feel free to add links to other individual projects or project aggregator pages as comments. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – on dentographs, a new method of visualizing library collections mission editorial committee process and structure code4lib issue 16, 2012-02-03 on dentographs, a new method of visualizing library collections a dentograph is a visualization of a library’s collection built on the idea that a classification scheme is a mathematical function mapping one set of things (books or the universe of knowledge) onto another (a set of numbers and letters). dentographs can visualize aspects of just one collection or can be used to compare two or more collections. this article describes how to build them, with examples and code using ruby and r, and discusses some problems and future directions. by william denton introduction these checkerboard dentographs compare the holdings of the toronto and san francisco public libraries. without knowing anything more about dentographs, it is clear at a glance that whatever it is san francisco has, toronto has more. figure 1. checkerboard dentographs of the toronto and san francisco public libraries when you know that both libraries use the dewey decimal classification, that the hundreds digit is shown along the x-axis and the tens along the y-axis, and that the colour of the square at (8,1) tells how many items are the 810s (“american literature in english”), you can see that toronto and san francisco collect the same kind of material, but toronto’s collection is much deeper. mountain dentographs are better for the library of congress classification. they are so called because they look like mountain ranges, with one line of mountains for each lcc class. here are mountain dentographs that compare two branches of the university of toronto. the split between the arts, humanities and social sciences at robarts and science at gerstein is clear. figure 2. mountain dentographs of two university of toronto branches: robarts (arts, humanities and social sciences) and gerstein (science) in this article i will show in detail how to generate both checkerboard and mountain dentographs to visualize and compare the holdings of different libraries. this is one of many possible uses for dentographs. others include: collection usage: dividing circulation by holdings gives usage ratios. in a checkerboard dentograph the colours could show the usage; in a mountain dentograph, the height. interlibrary loan lending and borrowing: this is another kind of usage. the number of items borrowed or lent could be visualized: borrowing could be subtracted from lending to show net intake or outtake in a given call number range, and this could be shown either by colour or positive and negative mountains. collection overlap or distinctness: the examples in this article are all based on call numbers, but other standard identifiers such as isbn or oclc numbers may also be used to calculate the overlap or distinctness between collections. dentographs could show what percentages of their holdings libraries have in common in different subjects, or how much of one library’s collection is unique and not held by other libraries. this approach could be particularly useful in consortia, such as for “last copy” holding agreements, or, at the other end of the scale, for small specialized collections to show that in their specific area they have better holdings than large libraries or consortia. with regard to ebooks, a library could compare its print collection to an ebook vendor’s offerings to see where it would benefit most. university ranking reports: these usually have small sections about the libraries, giving some numbers about study space or student satisfaction. dentographs could be included to give the reader a quick impression of collection size and strength. theoretical digression on mathematics: classification schemes are functions dentographs are a practical implementation of the idea that classification schemes are functions. it may be years since the reader last thought of mathematical functions, so i will only briefly describe my thinking here. a full explanation will come in a subsequent paper that goes into the theory. in grossly simplified terms, a function is a formula that, given some input, will generate some output. functions such as f(x) = 3x + 4 take a number and turn it into another number by applying a simple rule: “multiply by three and add four.” wolfram alpha defines a function as a “relation that uniquely associates members of one set with members of another set. more formally, a function from a to b is an object f such that every a ∈ a is uniquely associated with an object f(a) ∈ b.” the library of congress and dewey decimal classifications are functions that map the universe of knowledge onto combinations of numbers and letters. it’s possible to quibble with the “uniquely associated” part of the definition, but for our purposes, we can think of classification schemes as functions. and when we have a function, a natural next step is to ask: can we graph it? f(x) = 3x + 4 is easily graphed on the x-y plane as a straight line. dewey numbers are in the range (0 < n < 1000): how can they be graphed? lcc call numbers are made up of letters from ac to za and numbers from 1 to 9999: how can they be graphed? dentographs combine call numbers with holdings counts and graph the result to visually represent a library’s collection. getting ready to make dentographs all of the examples in this paper are fully reproducible. by downloading the code and data sets you will be able to follow along, line by line, and generate all of the graphics. if you have access to call numbers from your own library system you can easily adapt what is here to make dentographs for that data. installing r all of the graphics will be generated with r, described on its site as “a language and environment for statistical computing and graphics.” it’s a powerful tool for advanced statistics, but it’s also used for other purposes such as data mining and, as we’ll be doing here, visualization. r on its own has a fairly simple interface, so i recommend also installing rstudio, a gui that provides a powerful and friendlier interface. install r first, then rstudio. download r at your nearest cran mirror download rstudio desktop getting the code all of the scripts used in this paper are available at http://github.com/wdenton/c4lj-dentographs. every shell or r command is fully reproducible. there are two kinds of snippets of code below: $ is at the command line and > is in r. to get the code you need git, and to run the scripts you need ruby. download git download ruby once git is installed, to make a local copy of the repository run this at the command line: $ git clone http://github.com/wdenton/c4lj-dentographs $ cd c4lj-dentographs the last step is to set your r working directory to this same c4lj-dentographs directory. either run r at the command line in that directory, or in rstudio use tools > set working directory in the menu bar. (if you use rstudio, use its export button to save images instead of the saveplot command in the code below.) getting data files of call numbers to generate dentographs we need call numbers. luckily there is a good source: marc records some libraries have uploaded to the internet archive to help the open library. a number of libraries have made their data available, and i use three here: san francisco public library marc records (1 of 16; search for the others) (dewey decimal classification) toronto public library marc records (ddc) university of toronto marc records (library of congress classification) two libraries supplied data to me on request: university of prince edward island provided a list of call numbers (lcc) and locations. my thanks to melissa belvadi for this. york university libraries, where i work, does not give open access to its marc records, but i obtained a dump for this research. we use lcc. to save time, i processed the records and made available data files to replicate the examples. download the five files there to the c4lj-dentographs directory: sfpl-092.txt.gz tpl-090.txt.gz upei-call-number.txt.gz utoronto-949.txt.gz york-call-number.txt.gz they are all compressed with gzip so you will need to uncompress each before it can be used. for example: $ gunzip utoronto-949.txt.gz how to extract data from marc records dealing with a large set of marc records can be painful. there are so many ways that a library can customize its data for its individual needs that writing one script to extract call numbers from any of the open library dumps became tedious and complicated. in the end i found it was much easier and faster to run yaz-marcdump on all the files, pick out the one marc field i needed, and then process those lines to pick out the call numbers and store them in a text file. i’ll show how i did this with the toronto public library (tpl) data. the goal of operating on the tpl catalogue records was to extract every numerical call number in the range (0 < number < 1000). this leaves us with all nonfiction material and any fiction (or drama, poetry, etc.) that was classified with a number. anything without a number will be ignored. this is a problem in fairly assessing public library collections, where fiction is often classified as fic or something similar. the dentograph will only accurately represent the nonfiction collection. visual inspection of the tpl marc records is easily done with yaz-marcdump. the dewey number is stored in the 090 field (probably for historical reasons, because it is now obsolete: see marc bibliographic definition of 09x), and it was easy to extract all 2,210,126 to a file. (to save you the trouble of doing all the downloading, tpl-090.txt is one of the data files available, but to recreate it yourself you would get the files and run yaz-marcdump ol.20100104.* | grep ^090 > tpl-090.txt.) $ wc -l tpl-090.txt 2210126 tpl-090.txt $ head -5 tpl-090.txt 090 $a fiction rob 090 $a feature aik 090 $a 614.59939 rep 090 $a 614.59939 r25 090 $a 598.29729 ffr extract-tpl-ddc-from-090.rb pulls out the numerical dewey call numbers, ignoring everything else, such as fiction aik. 645,244 090 fields are thrown out.. $ ruby extract-tpl-ddc-from-090.rb tpl-090.txt > tpl-ddc-numbers.txt $ wc -l tpl-ddc-numbers.txt 1564882 tpl-ddc-numbers.txt $ head -3 tpl-ddc-numbers.txt 614.59939 614.59939 598.29729 extracting lcc call numbers from the university of toronto records is much the same, as we will see below, but with the advantage that fiction is also classified, so the call numbers cover everything in the collection. everything, that is, with a proper lcc call number: special schemes for government documents, audio, video, maps and so on are left out. checkerboard dentographs the dewey decimal classification is nicely suited to visualization because of its rigidity. the hundreds define the ten top-level classes: computer science, information and general works (0xx), philosophy and psychology (1xx), religion (2xx), social sciences (3xx), language (4xx), science (5xx), technology (6xx), arts and recreation (7xx), literature (8xx), and history and geography (9xx). each hundred is divided into ten tens and each ten into ten ones, within which the decimal expansions can go much farther. we can divide up a dewey collection into these hundreds, tens, ones, and even decimals, in various ways, and each leads to a dentograph of different granularity, complexity, and visual informativeness. to do these dewey dentographs we will use the levelplot command in r. one-by-one the most basic dewey dentograph shows the collection broken down to the tens: the ten hundreds are broken down into ten tens each, making a 10×10 grid with 100 squares. i call this a one-by-one dewey checkerboard dentograph, because it uses one number of importance on each side of the grid. to build the one-by-one we need to pick out the hundreds and tens from our list of call numbers. make-one-by-one-data.rb in the repository does this. run the script on the file of all call numbers and generate a text file of pairs of numbers (notice how 614.59939 becomes “6 1”): $ ruby make-one-by-one-data.rb tpl-ddc-numbers.txt > tpl-one-by-one.txt $ head -3 tpl-one-by-one.txt 6 1 6 1 5 9 finally we are ready to begin work in r. this is often how work with r goes: another language is used to clean the data first. r has text manipulation tools, but that is not its main strength, and if you are comfortable in a scripting language then you will probably find it easier to massage your data there. it only takes four commands in r to generate a raw, unadorned checkerboard dentograph. one: load the lattice library, which provides the levelplot command. two: load the data. three: turn that data into a table. four: generate a levelplot from the table. > library(lattice) > tpl.one.by.one <read.table("tpl-one-by-one.txt") > tpl.one.by.one.table <table(tpl.one.by.one) > levelplot(tpl.one.by.one.table) figure 3. raw, unadorned toronto public library one-by-one checkerboard dentograph we’ll make that look nicer, but first let’s look into the data structures. the head command in r is much as in unix, but instead of showing the first few lines in a file it shows the first few elements in a data structure. the numbers here are the same as above, but r has converted them into two columns and many rows. > head(tpl.one.by.one) v1 v2 1 6 1 2 6 1 3 5 9 4 3 6 5 3 6 6 9 7 > ncol(tpl.one.by.one) [1] 2 > nrow(tpl.one.by.one) [1] 1564666 the table command “uses the cross-classifying factors to build a contingency table of the counts at each combination of factor levels”, according to the ?table help file. in other words, with our dewey data, it will build a 10×10 table that counts how many times each pair of numbers appears in the tpl.one.by.one data frame. > tpl.one.by.one.table v2 v1 0 1 2 3 4 5 6 7 8 9 0 11648 19793 10232 1453 21 456 1298 4547 589 737 1 738 915 1661 6566 815 15094 398 4877 2100 4193 2 2654 878 5245 4987 3526 1523 5457 4636 11107 15440 3 59634 1518 34402 72476 25494 32259 50131 24536 20430 20600 4 1471 2694 13608 1943 3786 1044 1755 383 516 12257 5 8261 5968 5262 5499 3740 11867 1792 8387 3994 17269 6 2412 43022 40281 19564 28335 35299 5786 3149 6261 6193 7 18948 10984 13396 9673 27130 18819 4962 8883 30110 62074 8 18840 93554 80340 13161 17987 5302 8613 1421 3155 40156 9 7625 72508 41884 7230 63727 19756 5688 67239 2652 1987 for example, the value of the (3, 5) entry in this table is 32,259. this means that “3 5” appeared 32,259 times in the data file. we can confirm this at the command line: $ grep -c "3 5" tpl-one-by-one.txt 32259 the toronto public library has 32,259 items classified in the 350s (“public administration and military science”). now we can make a prettier dentograph. there are a vast number of ways to customize graphs and charts in r. i won’t go into many details here, because most of the commands will be self-explanatory when you see them and then look at the generated image. two things about this next snippet: i create a function palette to change the colours used, and the scales parameter lets me customize what appears on the axes, defining some new labels and rotating them where necessary. > palette <colorramppalette(c("#eeeeee", "purple")) # prepare a better colour palette > levelplot(tpl.one.by.one.table, col.regions = palette, xlab = "hundreds", ylab="tens", main = "tpl one-by-one dentograph", scales=(x=list(rot=90, at=seq(1, 10), labels=c("general 0xx", "philosophy, psychology 1xx", "religion 2xx", "social sciences 3xx", "language 4xx", "science 5xx", "technology 6xx", "arts 7xx", "literature 8xx", "history, geography 9xx"), y=list(rot=0, at=seq(1, 10), labels=10*seq(0, 9)))) ) figure 4. toronto public library one-by-one checkerboard dentograph note the depth of the collection in the 300s and the relative paucity of the 200s 400s, and watch for how the representation changes in the next two dentographs. one-by-two the next step is to go further into the numbers. let’s make a one-by-two checkerboard dentograph, again with the hundreds on the x-axis but now tens and ones on the y-axis. this will be a 10×100 matrix. the process is the same as above, but make-one-by-two-data.rb prepares the data: $ ruby make-one-by-two-data.rb tpl-ddc-numbers.txt > tpl-one-by-two.txt then in r: > tpl.one.by.two <read.table("tpl-one-by-two.txt") > tpl.one.by.two.table <table(tpl.one.by.two) > levelplot(tpl.one.by.two.table, col.regions = palette, xlab = "hundreds", ylab="tens and ones", main = "tpl one-by-two dentograph", scales=(x=list(at=seq(1, 10, by = 2), labels=seq(0, 9, by =2), y=list(rot = 0, at=seq(11, 100, by=10), labels=paste (seq(1, 9), "0", sep=""))))) figure 5. toronto public library one-by-two checkerboard dentograph it’s interesting how the hundreds form columns that run up the image (the 300s stand out again, for example), but perhaps there is both too little and too much here to be very useful. two-by-two going one more level into the dewey numbers, to make a two-by-two dentograph of a 100×100 matrix, is far more interesting. make-two-by-two-data.rb will generate the file of pairs of numbers we need: $ ruby make-two-by-two-data.rb tpl-ddc-numbers.txt > tpl-two-by-two.txt then in r, again the data is loaded in and levelplot run. here, to make it a little clearer about where the numbers fall, a grid of dashed lines is added (the way that this is done with lattice graphics, with a function declared and separate commands run within it, is a little confusing): > tpl.two.by.two <read.table("tpl-two-by-two.txt") > tpl.two.by.two.table <table(tpl.two.by.two) > levelplot(tpl.two.by.two.table, col.regions = palette, main="tpl two-by-two dentograph", xlab="hundreds and tens", ylab="ones and decimals", scales=(x=list(at=seq(1,100, by=10), labels=paste(seq(0, 9), "0", sep=""))), panel=function(...){ panel.levelplot(...); panel.abline(h=seq(11,99, by=10), lty="dashed", col="light grey"); panel.abline(v=seq(11,99, by=10), lty="dashed", col="light grey") } ) figure 6. toronto public library two-by-two checkerboard dentograph here again we see the 300s as much stronger than the 200s and 400s. the darkest colours, representing the deepest parts of the collections, are even more visible now in the 800s. three strong lines in the 900s have emerged: the 910s (geography and travel), 940s (history of europe) and 970s (history of north america), which matches the deep colouration those squares have in the one-by-one dentograph. where are the most items, and how many are there? two commands tell us: > which(tpl.two.by.two.table == max(tpl.two.by.two.table), arr.ind=true) row col 82 83 40 > max(tpl.two.by.two.table) [1] 29366 there are 29,366 items at (83, 40) in the table, but the way r counts rows and columns does not equal how we are putting dewey numbers into the table: row 1 of the table is 00, row 2 is 01, and so on; column 1 is 00, column 2 is 01, etc. row 83 in the table is for dewey 82x, and column 40 is 39 within that, giving up the call number 823.9 (english fiction, 1900-). sure enough, if you look in the graph, count two lines over from 80 on the x-axis, and go up to one line below 40 on the y-axis, there it is, the darkest square. comparing two dewey collections comparing two dewey collections is easily done by putting two one-by-one checkerboard dentographs beside each other. next we will create the toronto and san francisco public libraries comparison shown in the introduction. we already have tpl.one.by.one.table in memory, so we begin by generating the data from the sfpl-092.txt data file. $ ruby make-one-by-one-data.rb sfpl-092.txt > sfpl-one-by-one.txt and then in r: > sfpl.one.by.one <read.table("sfpl-one-by-one.txt") > sfpl.one.by.one.table <table(sfpl.one.by.one) > max(tpl.one.by.one.table) [1] 93554 > which(tpl.one.by.one.table == max(tpl.one.by.one.table), arr.ind=true) row col 8 9 2 > max(sfpl.one.by.one.table) [1] 11417 > which(sfpl.one.by.one.table == max(sfpl.one.by.one.table), arr.ind=true) row col 9 10 2 when doing a comparison like this we must make sure the same z-axis scale is used for both collections. the deepest part of the tpl collection at the tens level is the 810s (row 9 is the 800s, column 2 is the 10s), with 94,201 items. the deepest part of the sfpl collection is in the 910s (row 10, column 2) with 11,417 items. the at parameter to levelplot sets out where the cuts on the z-axis will happen. this is not necessary for a one-collection checkerboard dentograph, but when comparing two collections the colour schemes must match up and show collection depth levels in absolute and not relative terms. here we force r to use a scale from 0 to 100,000, with 50 colour gradations (49 cuts) along the way. > levelplot(tpl.one.by.one.table, col.regions = palette(50), cuts = 49, main = "toronto public library", xlab = "hundreds", ylab = "tens", at = 2000*seq(1:50)) > saveplot(filename="comparison-tpl.png", type="png") > levelplot(sfpl.one.by.one.table, col.regions = palette(50), cuts = 49, main = "san francisco public library", xlab = "hundreds", ylab = "tens", at = 2000*seq(1:50)) > saveplot(filename="comparison-sfpl.png", type="png") back at the command line, convert from imagemagick turns the two images into one: $ convert +append comparison-tpl.png comparison-sfpl.png comparison-tpl-to-sfpl-large.png $ convert -resize 800 comparison-tpl-to-sfpl-large.png comparison-tpl-to-sfpl.png figure 7. comparison of toronto public library and san francisco public library one-by-one checkerboard dentographs (note that holdings counts are ignored in both the toronto and san francisco public libraries data files. tpl’s call numbers are taken from the 090 field in the marc records, but special codes in the 906 show how many copies are at different branches. sfpl’s call numbers were taken from the 092, but multiple 945s are used for the holdings.) mountain dentographs the library of congress classification doesn’t have dewey’s methodically rigid structure. lcc call numbers can begin with one, two or three letters, which is manageable, but instead of being laid out neatly from 0 to 999 the numbers can range from a maximum of 9 (in lh, college and school magazines and papers) to 9999 (six classes outside of law, the first being bx, christian denominations). instead of trying to fit lcc call numbers to some procrustean bed to make a checkerboard dentograph, we can leave them as they are in a mountain dentograph. mountain dentographs are three-dimensional, with the lcc class letters on the x-axis, the numbers on the y-axis, and the item counts on the z-axis. they look like very orderly mountain ranges. to keep things simple i am going to ignore everything in k (“law”), which has 156 subdivisions, ending at kzd (space law, law of outer space). my apologies to any law librarians reading this. processing call numbers for the first examples we’ll get call numbers from the university of toronto marc records in the internet archive. i want to keep the branch information to generate branch-specific dentographs, so the call number extraction will be a little different, and here we will use holdings counts. the first step is to extract the 949s with yaz-marcdump as above (yaz-marcdump utoronto.mrc | grep ^949 > utoronto-949.txt) but to save time i’ve done this and put the results in the utoronto-949.txt data file. we’ll run 949-extractifier.rb to pull out the branch and call number of each item. there are 6,787,653 949s in the marc file, and after processing 5,414,215 proper lc call numbers are left in a very simplified listing. $ wc -l utoronto-949.txt 6787653 utoronto-949.txt $ head -2 utoronto-949.txt 949 $a ac1 .h32 n4 $w lc $c 1 $i 31761016601411 $d 17/4/2003 $e 17/4/2003 $l stacks $m robarts $n 2 $r y $s y $t book $u 26/8/1992 949 $a ac1 [online resource 47903] $w lc $c 1 $i 2-2001 $l online $m e_resour ce $r y $s y $t e_resource $u 7/2/2008 $ ruby 949-extractifier.rb utoronto-949.txt > utoronto-branch-call-number.txt $ wc -l utoronto-branch-call-number.txt 5414215 utoronto-branch-call-number.txt $ head -2 utoronto-branch-call-number.txt robarts:ac 1 e_resource:ac 1 let’s look first at university of toronto’s entire collection. for this, we want all the call numbers regardless of branch, and it’s easy to pull that out with cut. then we need to prepare the data for r. we want to make a 3d graph where the class letters (ac, ae, ag, …, za) run along the x-axis, the numbers run along the y-axis, and the z-axis shows the number of items at each call number. to do this the class letters need to be turned into numbers. convert-lc-to-numbers.rb does this by mapping ac -> 1, ae -> 2, …, za -> 212. this forces the x-axis to always run from 1 to 212, so it will be the same width for all libraries. the script also adds points along the y-axis from (0,0) to (0,10000) to ensure that the graph has the same depth for all collections. if the width and depth were not forced it would be impossible to compare collections reliably: f or q could end up in different places, or one chart might only go to 500 while another goes to 9,000. $ cut -d ":" -f 2 utoronto-branch-call-number.txt > utoronto-call-number.txt $ ruby convert-lc-to-numbers.rb utoronto-call-number.txt > utoronto-mountain-data.txt now we can visualize this with persp in r (theta and phi set the angles the graph is seen at, adjust them to move your point of view left/right and up/down): > utoronto <read.table("utoronto-mountain-data.txt") > utoronto.table <table(utoronto) > persp(utoronto.table, theta = -5, phi = 20, scale = true, border = na, axes = f, box = f, col = "cyan", shade = 0.5, main = "university of toronto") > max(utoronto.table) [1] 19748 > which(utoronto.table == max(utoronto.table), arr.ind=true) row col 140 141 77 figure 8. unlabelled mountain dentograph of the university of toronto you’ll see one standout high peak. as shown with max, we can locate it at (141, 77). 141 on the x-axis is qa, and 77 on the y-axis is 76 in call numbers (the y-axis starts at 0), so that peak is at qa 76: a number so familiar to readers i need hardly mention it is where computer science is found. lcc is incredibly limited in how it can accommodate books on that subject. this is the highest peak in every library i’ve graphed. a script for mountain dentographs the next two examples will be easier with an r script we can run at the command line: dentograph.r is in the repository. it takes arguments on the command line that set the data file it’s reading, the filename of the image it will output, and the title to use on that image. the dimensions of the output image are set with png. persp is run again, but with zlim to force the z-axis to the same scale across collections (just as we did with the dewey checkerboard dentographs). #!/usr/bin/env rscript # usage: dentograph.r mountain-data.txt filename.png "library name" args <commandargs(true) datafile <args[1] output <args[2] library_name <args[3] png(filename=output, height=1600, width=1600, units="px") d <read.table(datafile) table <table(d) x <1:nrow(table) y <1:ncol(table) res <persp(x, y, table, zlim = c(0,10000), # change as necessary, or comment out theta = -5, phi = 20, scale = true, border = na, axes = f, box = f, col = "cyan", shade = 0.5, main = library_name) # label x-axis with class letters xpoints = read.csv("x-axis-labels.csv") for (i in 1:nrow(xpoints)) { points(trans3d(xpoints$point[i], 5, 0, pmat = res), col = "#000000", pch = xpoints$label[i], cex = 1) } axes = f means no axes are drawn, which means that there is no scale on the z-axis to show how many items are in the collection. it’s easy enough to enable axes if needed, but it doesn’t seem necessary to have them to simply give a quick impression of how two collections compare. the persp object is stored in res so that we have access to its layout later in order to label the x-axis. the script reads from a file a list of x-axis positions and ascii codes such as (1, 65) and (11, 66). this puts “a” (65) at 1 on the x-axis and “b” (66) and 11 on the x-axis, skipping over “ac” at 2, “ae” at 3, and so on. it makes the graph slightly easier to read. comparing branches university of toronto libraries is a large system, with over fifty branches. the biggest are robarts (holding arts, humanities and social sciences) and gerstein (science). comparing those two shows how different their holdings are. first we’ll grep the robarts and gerstein holdings from the full list. then we need to make sure dentograph.r knows the proper scale of the z-axis to suit these collections. the classic sort | uniq -c | sort -rn pipeline on robarts and gerstein holdings shows us that the highest number of holdings at one call number is just under 10,000 in both branches. the script in the repository is already set to use that number so it will work here without any edits. $ grep ^robarts utoronto-branch-call-number.txt | cut -d":" -f 2 > utoronto-robarts-call-number.txt $ grep ^gerstein utoronto-branch-call-number.txt | cut -d":" -f 2 > utoronto-gerstein-call-number.txt $ sort utoronto-robarts-call-number.txt | uniq -c | sort -rn | head -1 9574 pg 3476 $ sort utoronto-gerstein-call-number.txt | uniq -c | sort -rn | head -1 9482 qa 76 $ ruby convert-lc-to-numbers.rb utoronto-robarts-call-number.txt > utoronto-robarts-mountain-data.txt $ ruby convert-lc-to-numbers.rb utoronto-gerstein-call-number.txt > utoronto-gerstein-mountain-data.txt $ ./dentograph.r utoronto-robarts-mountain-data.txt utoronto-robarts-mountain.png "robarts (arts/hum/soc sci)" $ ./dentograph.r utoronto-gerstein-mountain-data.txt utoronto-gerstein-mountain.png "gerstein (science)" $ convert +append utoronto-robarts-mountain.png utoronto-gerstein-mountain.png utoronto-branches.png figure 9. university of toronto’s robarts and gerstein branches compared the distinctness of the two collections is clear. gerstein is almost entirely concentrated in q (science) and r (medicine) with some in s (agriculture) and t (technology). robarts sprawls heavily throughout a-p, especially p (linguistics and literature). because of how lcc works, the relatively small range of numbers used in q and r is also easy to see. seven of the nineteen letters in p go into the 9,000s, but the maximum number possible for any of the letters in the qs is under 1,000 (for example q stops at 510 and qa (mathematics) at 939). comparing libraries finally, let’s compare the university of toronto collection to the libraries of two other canadian universities, york university and the university of prince edward island. these dentographs tell us at a glance how the collections compare, even without numbers to show how large they are. first, as a bit of background, some basic facts about the universities and their libraries. (enrolment numbers are total students as of fall 2010, taken from the association of universities and colleges of canada’s enrolment by university page): university of toronto is the biggest university in canada with 78,900 students, and its library system is also the biggest, with about 11,350,000 “bookform” items in its collection as of april 2010. york university (also in toronto) has 54,600 students and its library reports almost 2,500,000 print volumes in its collection as of april 2010. university of prince edward island has 4,590 students, and its 2007-2008 library annual report says it had about 370,000 books and ebooks (print books are not separated out). the following commands will generate the dentographs. before running them, note that the maximum value in university of toronto’s holdings is 19,748 (this can be found with a sort | uniq | sort as above), so you will need to edit dentograph.r to change the zlim value to 20,000 to force the z-axis to be the same in all graphs. if you don’t edit it, some spikes will run out the top of the dentographs. $ ruby convert-lc-to-numbers.rb york-call-number.txt > york-mountain-data.txt $ ruby convert-lc-to-numbers.rb upei-call-number.txt > upei-mountain-data.txt $ ./dentograph.r utoronto-mountain-data.txt utoronto-mountain.png "university of toronto" $ ./dentograph.r york-mountain-data.txt york-mountain.png "york university" $ ./dentograph.r upei-mountain-data.txt upei-mountain.png "university of pei" $ convert +append utoronto-mountain.png york-mountain.png upei-mountain.png mountain-comparison.png $ convert -resize 800 mountain-comparison.png mountain-comparison-smaller.png figure 10. university of toronto, york university and university of pei compared pei’s collection is sparse and shallow compared to the others, which is no reflection on anything other than its size. it’s unfair to compare it to much larger libraries except to serve some kind of illustration like this. on the other hand, comparing toronto and york, two large universities in the same city, is quite interesting. toronto is clearly broader and deeper than york: its collection is larger and covers more subjects, apparently across the board. in b (philosophy, psychology, religion) toronto has more (both close to the x-axis and stretching out to the far side), probably because it has divinity programs. m (music) and n (fine arts) are both denser. p is much richer than at york, with far more high spikes. the science cluster in q is also much denser. future directions i hope readers find the ideas here interesting and will extend them beyond what i’ve described. aside from the possible uses described at the beginning of the paper, there are two lines of future work that i see, but i hope readers will find more. first, there are more ways to use the three dimensions of mountain dentographs. perhaps it would be possible to fly around inside the mountain dentograph, exploring the collection and seeing flags or labels on the mountains to identify what lc number or subject they represent. in r some interactivity is possible with the persp3d command, which makes it possible to rotate and zoom the image. the arguments are the same but the experience is very different from persp. run this to try it: > library(rgl) > persp3d(utoronto.table, theta = -5, phi = 20, scale = true, border = na, axes = f, box = f, col = "cyan", shade = 0.5, main = "university of toronto") second, perhaps three dimensions for lcc isn’t best, and a two-dimensional representation would work better. lcc is so sprawling and varied a classification that it would probably work best not to map it directly but to make clusters. for example, the university of toronto dentographs show a strong line along bx (christian denominations), which has these subsections, as listed in b – philosophy. psychology. religion in the library of congress classification outline: 1-9.5: church unity. ecumenical movement. interdenominational cooperation 100-189: eastern churches. oriental churches 200-756: orthodox eastern church 800-4795: catholic church 4800-9999: protestantism five data points are much easier to understand than 10,000. if the rest of lcc was similarly clustered and mapped it would be easy to generate checkerboard lcc dentographs. they would be ragged because there would be different numbers of clusters per letter, so instead of a neat 10×10 or 100×100 visualization it would be 212 wide (more if law is included) by varying depths, but that doesn’t matter. a mapping somewhat like this is in fact already available: the oclc conspectus. this is old work, now abandoned, but perhaps still useful here. it has 29 top-level subjects, such as art and architecture, medicine, and philosophy and religion. there are 378 narrower second-level subjects. philosophy and religion has 18, such as “philosophy – ancient, medieval, renaissance,” “philosophy – modern (1450/1600),” and “logic.” each subject is associated with lcc and ddc call number ranges (so it is possible to assess collections regardless of classification scheme), and here bx is boiled down to three headings: “eastern christian churches & ecumenicism” from bx 0-765, “roman catholic church” from bx 800-4795, and “protestantism” from bx 4800-9999. it would be possible to use this to generate either checkerboard or mountain dentographs by mapping the top-level subjects on the x-axis and the second-level subjects on the y-axis. finally, there are undoubtedly other, and i hope better, forms of dentographs than checkerboards and mountains. three problems the biggest problem with dentographs of holdings is that they show quantity but not quality. there is no easy way, automated or manual, to assess the quality of a collection. we can only visualize available data, so dentographs are limited. however, as mentioned above, dentographs can also show usage, overlap or uniqueness, which in their own ways tell us something about a collection’s quality. the second main problem is that dentographs depend entirely on call numbers from a standard classification (or, with the oclc conspectus, subject assignments to a controlled vocabulary). for most print material, that is fine. everything on a library shelf will have some kind of call number. if the call number is not lcc or dewey, however, that is a problem. collections that are special for their format or location will be overlooked, as may huge collections of fiction or children’s books in public libraries. electronic resources are especially susceptible to lacking call numbers. at my library, very few electronic books or journals have valid lcc call numbers: they are all assigned electronic. however, restricting to print or other physical resources may actually be useful. among academic libraries, if all libraries of a similar size subscribe to the same electronic resources then comparing that part of their collections is pointless. the comparison of university of toronto to york university shows how much better university of toronto is with its print collection, but when it comes to electronic resources, the two are more or less the same (except for subjects university of toronto teaches that york doesn’t, such as medicine and architecture). more and more, it is the local print material that makes collections special, and dentographs are good tools for assessing that. a side effect of using standard classifications such as dewey and lcc is that the dentographs reveal the limitations of the schemes. in the checkerboards for the toronto public library, dewey’s 290s are coloured darker than any other ten in the 200s (“religion”) because it is everything that is not christianity (“other religions”). the toronto public library’s expansive, multicultural religion collection is shoehorned into dewey’s nineteenth century organization. similarly, lcc was not built to handle computer science, and now qa 76 is so overcrowded that it cannot be fairly compared to other numbers. thirdly, quality and access are always problems with cataloguing records. bad data can be worked around, but not every library makes regular catalogue dumps available. they should. that data, and union and consortial catalogues, should be available under open licenses. tools used r rstudio git convert and resize from imagemagick ruby yaz-marcdump from the yaz toolkit further reading about r an introduction to r by w.n. venables, d.m. smith, and the r development core team. of the many books about r, i found two from o’reilly particularly useful: r cookbook by paul teetor and r in a nutshell by joseph adler. r programming wikibook is under development. the r journal. questions tagged with r at stackoverflow are a great place to look for answers to problems. blogs: revolutions has great coverage, and is part of r-bloggers, which collects many r-related blogs into one place, in a more readable way than planet r. about the author william denton <wdenton@yorku.ca> is web librarian at york university in toronto, canada. he has a b.sc in mathematics and an mist from the university of toronto. his web site is miskatonic university press. subscribe to comments: for this article | for all articles 4 responses to "on dentographs, a new method of visualizing library collections" please leave a response below: ingrid mason, 2012-02-04 william, this is awesome work you’ve done here. i love how you’re looking for potential real library use cases for using collection data visualisations. also i really like the way you’re teasing out the limitations of using classification schemes as the means to base the calculations and create meaningful dentographs. have you thought about using other types of collection data? the powerhouse museum in australia has an api of its collection data and they use a thesaurus that perhaps you could plot at bt level? anyway – keep up the great work! ingrid william denton, 2012-02-05 thanks for the kind words, ingrid. i didn’t know of the powerhouse museum but its api and data access (http://www.powerhousemuseum.com/collection/database/download.php) look great—just the kind of thing we should all be doing. i’ll definitely look into what i could do with their data. elizabeth, 2012-06-08 thank you very much for the detailed instructions–if i ever feel ambitious i might try this. it could be a good tool for libraries. great idea and very nice dentographs! 필리핀 여행, 2025-07-05 it’s a potentially useful innovation. it’s a creative approach to information visualization. leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – “what if i break it?”: project management for intergenerational library teams creating non-marc metadata mission editorial committee process and structure code4lib issue 28, 2015-04-15 “what if i break it?”: project management for intergenerational library teams creating non-marc metadata libraries are constantly challenged to meet new user needs and to provide access to new types of materials. we are in the process of launching many new technology-rich initiatives and projects which require investments of staff time, a resource which is at a premium for most new library hires. we simultaneously have people on staff in our libraries with more traditional skill sets who may be able to contribute time and theoretical expertise to these projects, but require training. incorporating these “seasoned” employees into new initiatives can be a daunting task. in this article, i will share some of the strategies i have used as a metadata project manager for bridging diverse generations of library staff who have various levels of comfort and expertise with technology, and strategies that i have used to reduce the barriers to participation for staff with diverse perspectives and skill sets. these strategies can also be helpful in assisting a new librarian with technology-rich skill sets to more successfully orient themselves when embedded in a “traditional” library setting. by kelly j. thompson introduction i am an early-career metadata librarian embedded in a traditional cataloging department in a large academic library. part of my job is to train the marc cataloging (and copy-cataloging) staff in my department in non-marc metadata creation for our digital collections and institutional repository. this staff is most diverse in age ranges and technological ability—i am several decades junior to the majority of my colleagues and the first new hire the department has made in a long time. most of the library literature to date which addresses the transition of traditional marc cataloging librarians and support staff to non-marc metadata creators focuses on applicable skill sets which can be applied in both types of work, ways to build on those cataloging-specific skills to foster skills in metadata creation, and the types of projects which may be appropriate to assign to these staff, both as they are being trained and as they gradually build up competencies. (boydston and leysen 2008; keenan and sipe 2013; o’bryan and palmer 2007; rodgers and sugarman 2012; veve and feltner-reichert 2010) i want to speak to a gap in this literature related to generational diversity: the inter-generational differences in work styles, theoretical understandings, and comfort with technology which manifest in a collegial environment. i will speak to a number of barriers to participation in new metadata projects for age-diverse staff, and identify a few strategies that have worked to build inclusivity in my experience managing these projects. i write from the perspective of a new metadata librarian without formal supervisory responsibilities, who is also engaged in the process of acclimating to a very traditional library environment. i will share some of the communication, training, and project management strategies i have utilized while leading metadata projects that have eased some of the tensions experienced in this process. from technology anxiety (what if i break it? what if i accidentally delete everything?) to sitting with uncertainty (is there a rule i can read about how to format this title?) and navigating spreadsheet basics, i will talk about some of the on-the-ground challenges and opportunities which exist for the new metadata professional tasked with midwifing the training of a multi-generational cataloging staff with a diverse array of comfort and familiarity with technology in general. i am strongly convinced that what has made our projects successful are mutual reciprocity, active listening, and frequent and open communication. i will talk about ways we can build co-working communities to foster growth beyond barriers to participation in projects that require some members to develop new technology skills with assistance from others. strategies used to bridge diverse generations/classifications of staff communication, documentation, and checking your own assumptions in my work as a metadata project manager, i have observed a number of indicators which alerted me to gaps in my understanding of what i was communicating and what was being understood by project staff. invisible gaps in expertise and shared language can easily lead to frustration and tension. as a new librarian, i had a lot of assumptions about what i knew, and a different set of assumptions about what i thought my coworkers knew. the fact that i was coming into the department as a new person who was supposed to guide them in doing new work was anxiety-provoking for all parties. i observed anxieties among some of the more senior staff in my department regarding their lack of the newest skill sets, or their understanding of what those skill sets were. i dealt with this by increasing the amount of communication in every area of work possible. i continue to ask a lot of questions and strive to practice active listening. it is important for anyone creating new workflows to inquire about procedures already in place, how similar types of work have been done in the past, and how new initiatives may impact other workflows in the organization. if there is existing documentation available about workflows, this is a good place to start, but often you may need to create this documentation as you do your homework. i find it important to consult with my department head before implementing any major steps forward, so that we can go over the plan i’ve developed and the proof of concept. she will often share insight about how a similar situation might be handled in a traditional marc cataloging situation, which i believe improves our metadata quality measures and makes our non-marc metadata more interoperable with our marc catalog data once in a discovery system. i like to discuss any preliminary plans or roadblocks i encounter with my colleagues, as they will often have insight about a way to improve the process, or refer me to another person in the library who has relevant expertise or has done something similar in the past. because i am curious about the work that my organization is already doing, it is easier for those i am training to develop an interest in what i am asking them to do. it has helped that i was brand new to traditional marc cataloging when i arrived in the department, as this provided a mutual reciprocity in knowledge transfer that bridged some of these tensions. on a regular basis i will ask my senior cataloging colleagues questions about complex original cataloging, and they will ask me questions about non-marc metadata they are working on for a digital collection. by recognizing that everyone is bringing some kind of expertise to the table and asking for help when you need it, you set a tone of equality from the get-go. one challenge of working with staff at different classifications (we have unionized merit staff, professional and scientific staff, and librarians at either faculty or professional and scientific rank) is that non-professional staff sometimes have completely different understandings of the department’s work. often the same level of communication is not present from administration to non-professional staff as it is to professional staff or librarians. different types of staff have different amounts of specialized education as well – for example, typically the professional librarians in our organization have completed an mlis degree, and thus have a background in library theory and a shared vocabulary that accompanies that experience. these differences often lead to non-professional staff being less likely to go about their day with the big-picture view of a problem or project in mind, as they are responsible primarily for on-the-ground details. one strategy to bridge this is to clearly communicate what the project goals and objectives are, providing context for the work within the library enterprise, and to explain why you are doing this project (who will this benefit?). i have found that it is helpful to succinctly and non-technically share what all of the necessary steps to a project are, even if staff will not be participating, so that all staff have an awareness of the whole workflow. this makes it easier if there is a snafu in one part of the production workflow that impacts other steps down the line. people find it easier to be flexible in their timelines and work if they understand the driving forces causing disturbances to the routine. it was difficult for me to develop a sense for how much and at what points i needed to be communicating with project staff. the answer to this is usually more often than you think, and i find that i tend to assume people have more understanding of what i want them to do than they actually do. by communicating the big-picture process as clearly as possible, and clearly stating your expectations at each step of the process, you can help project participants both move their skills forward and learn more about the new work you are asking them to do. we communicate in-person and via email, one-on-one and in group meetings, depending on what needs to be communicated, decided, or taught. i have found that i need to document nearly everything. whether this is a data dictionary for a digital project with examples given for each field value, or a simple procedures document for navigating a document in the contentdm project client, everyone is more comfortable when all staff have a common starting point and clear guidelines with step-by-step references they can return to later on their own. it sounds simple, but i received feedback from one of the catalogers that even simply numbering the steps in a written procedure makes it easier to check in with colleagues when questions arise. regardless of how good [you think] your documentation is, as a project manager, staff will still be coming to you with questions, with the expectation that you will have answers. in surveying traditional cataloging staff, veve and feltner-reichert found that 68% of non-mls catalogers they surveyed said that they will seek assistance first by approaching someone at their institution to answer questions or for other assistance. while 43% of mls catalogers in their study reported that they would first try to figure out the answers themselves, asking someone at their institution was still the first choice for 31% of the mls catalogers. (veve and feltner-reichert 2010) this data reassures me that i am most likely not the only metadata librarian who has ever felt flooded with questions from trainees. the deluge of questions was a bit surprising and “imposter syndrome”-provoking for me at first, but i’ve found a few strategies which helped me become more comfortable with this situation. first, don’t be afraid to admit when you don’t know the answer to a question, to explain where you’ll find this information, and to ask people if you can do some digging and get back to them in a timely manner. by demonstrating that you will follow up and that you are not the sole source of these answers, you can help your staff become more comfortable themselves with not knowing, which can make question-asking less stressful. i also tend to show people how i found the answers to their questions if i sense that they will be open to this. in addition, workflows aren’t always going to function as well as you imagined them. i’ve found that i need to change or update some part of my carefully produced procedures and documentation in almost every project as staff find and report problems. own your mistakes! honesty may feel like displaying a weakness, but ideally it will build trust and a culture which supports well-researched risks and innovation. good documentation not only makes your life easier today, but it is also good succession planning. by setting clear parameters for the work and documenting much of the project planning, teams have a shared understanding and a good starting point for training. good documentation makes it is easier to return to the project or replicate it after time away. it enables others to replicate the process, understand the context of the metadata, and add more items to an existing collection while ensuring consistency. organizational culture my department’s organizational culture was one area where i needed to bridge some gaps in understanding while implementing some of the newer technology-rich projects i’ve worked on. i am on a number of cross-functional teams at my institution and am frequently outside of my cubicle at meetings or co-working with collaborators in other departments. for technical services departments such as mine, where the number of support staff is much greater than the number of librarians, and where support staff are a bit more rooted to their desks during the day, this was a strange transition for everyone. this was especially hard for staff in the department who would prefer to stop by for a quick chat instead of sending an email when they run into a problem with metadata work they’ve been assigned. even though my calendar was always up-to-date in our enterprise email/scheduling software, most people did not know how to effectively use this tool. at my department head’s suggestion, i began posting a copy of my weekly schedule outside my office door. this made it less frustrating for staff to know when they could find me in person, or to know that i was in a meeting and was simply delayed in answering their email until i was back to my office. by knowing that i was working somewhere else and not just absent, this created more camaraderie between myself and staff who do not have as hectic of a meeting schedule. as we’ve been engaged in library-wide strategic planning efforts recently, more staff have experienced what it is like to have more meeting commitments, which has also built empathy in my department for differences in schedules. by communicating and demonstrating new modes of work, we’ve been able to come to more mutual understandings of what a 21st century library’s work can look like. strategies used to avoid barriers to participation organizational culture, continued… a major barrier to the catalogers participating in this new work in my institution was the long-standing internal perceptions that our digital team had about catalogers. “people think of us as dinosaurs,” one of the catalogers said to me during my first weeks on the job. part of my work was to be an ambassador for the cataloging staff when i went out to meet with other groups in the library to provide metadata expertise and report on projects. i strive to give credit to the cataloger librarians and staff contributing to the metadata work these groups are asking for, and i make sure to give updates on the training and “next steps” i see the catalogers make. i try to emphasize the substantial amounts of theory that these professionals carry, and that while they may not be technologically adept, they are certainly adaptable. for example, by sharing that one of our monographs catalogers had recently independently completed metadata (including customizing a controlled vocabulary) for a new digital collection and was now going to complete the documentation for the project on her own for the first time, some members of the digital team expressed surprise that catalogers were trainable in this area. to anyone who has been a cataloger, you understand how ridiculous this sentiment is, but it continues to be a real, prevailing bias that i encountered among several people in our library, and i share it simply to encourage those working with traditional technical services staff to speak up about the good work people are learning to do. by genuinely acknowledging their hard-earned growth and hard work, you boost morale and excitement among your trainees as well. i also make it a point to suggest that colleagues in my library who are working on digital projects contact cataloging librarians who i know have expertise in areas they are struggling with, such as metadata for library-published journals (serials) in our institutional repository. technical services librarian christine dezelar-tiedman writes, “do not assume that your more “seasoned” staff cannot get involved in digital projects or would not be interested. invite everyone to the table. experienced librarians have seen radical changes over the years and are used to changing along with them.” (dezelar-tiedman 2004) it is vital to select tasks of appropriate scope and complexity for beginning metadata creators. you may have to start with what seems to you like the ground floor, but consider it an investment in the sustainability of your metadata program long-term, as well as the sustainability of your own time. scaffold on skill sets most of the library literature focusing on what skill sets catalogers can bring to the metadata table imply that metadata creation is such a natural fit for these groups, they tend to gloss over the sticky practical implementation snafus inevitably present. while catalogers bring skills, expertise, and loads of relevant theory, they still need training and lack many of the technology skills that newly minted librarians may take for granted. (boydston and leysen 2008) for example, even though catalogers are trained to intuitively understand data models and the need for standardized element schemas, many of them have not been trained to read, produce, or navigate xml, which is how most of the widely adopted schema standards have been documented and serialized. by acknowledging these differences, we can begin to identify where to begin training. in library technology culture, i perceive that there is often an assumption that people should be able to teach themselves anything, to pick up new technologies on their own. i think this is often not the case for many library staff because they do not have a number of ‘preliteracies’ required to begin thinking about library data like metadata librarians or digital library developers.  before elementary school students can learn to read, they need to develop an understanding of how to hold a book, that letters are put together to form words, and that words represent meanings – these skills are referred to as preliteracies.  (stahl et al. 1990) there are a number of preliteracies which technical services staff need to develop before they can gain metadata literacies. background knowledge in how to read and write mark-up languages, and understanding how resource discovery works, how databases work, and the ways that data can exist in aggregate (and outside of one individual record) are some pieces of this. preliteracies for metadata training are an area of study warranting further research. i agree with dezelar-tiedman’s suggestion that “perhaps the best way to get involved is incrementally. not slowly or timidly, but in manageable steps.” (dezelar-tiedman 2004) i consider participation in projects to be the biggest source of training we do in my department. it is critically important to select appropriate projects and tasks for beginners, and build up progressively more complex skills from there. for example, we’ve included a number of staff in the metadata creation for retrospectively digitized theses and dissertations to be loaded into our institutional repository. this work includes assigning standardized department headings from a local controlled vocabulary managed in complex spreadsheets. although the staff have experience working with controlled vocabularies in marc records, they had very little experience with metadata in excel spreadsheets in general, a competency i took for granted. the staff had widely varying levels of comfort with spreadsheets. as one of them said, “there is a big difference between typing in a basic word document versus formatting tables!” by participating in this project, these staff have all developed basic to intermediate excel skills. i have also been able to begin to identify the individual talents and aptitudes that these staff bring to the department, which will help in deciding who to assign future metadata work to, observing who staff go to as peer mentors, and understanding who is most ready to develop more complex skills which go far beyond spreadsheet work. by starting with spreadsheets, we kept the work at a manageable scale, and we also avoided the need to train the staff to navigate the sometimes complex content management system we use until they have developed some more metadata preliteracies. technology anxiety from the beginning, the catalogers i work with exhibited a fair amount of technology anxiety. this manifests in questions such as, “what if i break it?” or “what if i accidentally delete everything?” this fear is a barrier to learning, because in order to learn a new skill, you need the freedom to experiment and make mistakes with a safety net. we worked around these fears (and potential real issues) by discussing the larger scale systems-view of the work we were doing. by explaining that they were each granted specific permissions in software such as the contentdm project client which did not allow them to delete all of the data, or by explaining that if a program froze up it was not their fault but a system bug, they felt freer to navigate and explore on their own. we work on a lot of spreadsheets, so i keep back-ups of any files that they are working on. because of this, i was able to demonstrate when one of them accidentally deleted a column of urls that it was easy to restore that information. you can also think strategically about how to provide low-risk access to shared files necessary for their work, for example, placing read-only permissions on the folders with digital objects in them to be described, or putting files on a server so they can be accessed through the web. by keeping versioned back-ups of all important documents, files or datasets, you can sooth some of the “what if i accidentally delete everything” anxiety for all parties, and free people up to concentrate on the actual intellectual work required of the metadata description process. it is nice to separate the training aspects of metadata content creation from that of software navigation, although this is not always possible. technology anxiety can also be due to unfamiliarity with the vocabulary being used. i think that vocabulary is a huge barrier to participation in technology-rich work in libraries. technical terms, jargon, and acronyms are present in most library work, and it is a worthwhile investment to learn these from each other, as it pays dividends in the efficiency of your communication and the level of understanding you can build in common. at the beginning, it was great to have people remind me when they had no idea what i was talking about, so i could back up and clear up any confusion before proceeding. change anxiety not all of the anxieties that are barriers to participation are related to technology. change anxiety is also often present. people in technical services are especially aware of the fact that the information landscape is moving underneath them, but don’t always have the necessary information to know what to do about it personally. i have observed that many cataloging staff members have a real discomfort for sitting with uncertainty. i have noticed that this tends to be stronger among those who have worked with monographs, as those who work with serials tend to have an easier time, perhaps because they are accustomed to the constant flow of title changes and irregularities encountered in the work. as technical services librarian christine dezelar-tiedman writes, “there is a codified set of standards and years of tradition behind the way that we operate. ingrained in at least some of us is a discomfort with chaos and uncertainty. cataloging is our own way to create control over a small corner of the universe, which can be very satisfying. in contrast, the digital realm is nebulous and constantly changing.” (dezelar-tiedman 2004) cataloging staff have been socialized over decades to refer to the relatively small handful of standards used in traditional cataloging, including marc, lc headings, aacr2, and now rda. all of these standards are meticulously documented and detailed, and are thoroughly rule-based to avoid as much ambiguity as possible. a metadata project manager should not be surprised when presenting catalogers with a data dictionary with few content rules explicitly stated, to receive a follow-up email asking, “is there a rule i can read about how to format this title?” one of the training exercises we have done in my department is a comparative standards reading, where we looked at content standards for titles in aacr2, rda, dacs, and dcrm-g. by taking a look at standards from different kinds of cultural heritage institutions and discussing what we liked and disliked about each, we were able to agree on a local policy that worked for us and our users’ needs. i also remedied this by providing example values in all data dictionaries, as well as going through a few practice records together as a group during our training sessions. during projects, i send out regular faq emails (the frequency depends on the pace of the project — and number of questions!) recapping all of the questions i’ve gotten since the last update which may be of value to others beyond the original question-asker and which don’t “out” the question-asker. reinventing the wheel is not the best use of your time one of the barriers that i personally had to overcome was that i had very little experience with teaching, project management, or training before stepping into my current position. for me, it was important to learn about each of these in order to help our projects stay organized and run smoothly. before setting out to reinvent any of these essential skill sets solely through my own experience, i sought out information in the library literature as well as education, agile development, and project management literature. i also partook of services on my university’s campus to assist new professionals in developing these skills. i experimented with some existing workflow management tools (including trello, which we use to track in-process digital team projects as the work progresses through various library departments), and i’ve developed my own spreadsheet-based workflow tracking tools for individual projects. for individual projects, it is essential to have good file identification systems. to avoid confusion for the cataloging staff, i “assign” each project participant a folder on our shared network drive, where i put spreadsheets or other documents for them to work on. this space also serves as a repository for their own copies of project documentation, so they may edit or notate as needed. by relieving them of having to navigate through dispersed file hierarchies and needing to know the file ids they were responsible for, we were able to create a much friendlier workflow. i am always able to know who is working on a particular file or who completed which work by referring back to my managing spreadsheet. this is also useful as staff email me questions, because i can use this system to quickly know which batch of data they are referring to. many campuses have dedicated staff and centers for helping staff and faculty develop their pedagogical skills. i participated in several workshops through our campus’s center for excellence in learning and teaching, including a teaching and learning circle which aimed to help participants develop methods for incorporating critical thinking into their pedagogy. this group helped me to begin developing training activities which incorporate active learning, reflection, and problem-solving. i look at example documents from instructional designers at my institution to draw inspiration for my documentation and training handouts. i try to consider a diversity of learning styles and incorporate active learning into our training sessions by having demonstrations, software “tours,” and group practice activities. i’ve also learned from our training sessions that when doing small group instruction with new software, it is nice to have a “helper” in the room. this person can assist those who get behind or need help finding a button so you don’t have to be in all places at once. this can help you to avoid leaving anyone behind during training sessions. i think it will help overall project management stress to remember that learning is really inefficient and that planning how you are going to use work time also requires an investment of time. by budgeting time for this in your planning, you can save a lot of stress during the project implementation stages. it was hard for me at first to accept that i needed to train my colleagues to take on this new work when it felt like it took three times as long to train them as it would for me to just do the work myself. to move past these thoughts, i needed to think of the long-term sustainability of my time, and to think of these initial projects as investments of time. each subsequent project has required less start-up time and generally become easier for everyone involved, and i’m now moving toward including certain staff in more complex projects (such as developing local metadata application profiles from scratch, following shawn averkamp’s metadata design and development course project model). (averkamp 2013) i try to see that with each project, everyone gets a manageable amount of new challenges or experiences. i’m now also trying to incorporate evaluation at the end of each project, so that we can share feedback with each other on how the project went, and incorporate that into the next phase of our work. it is important to budget time for assessment as it is always possible to improve communication or documentation, or the ways that you demonstrate the value of the team’s work in a new way or to a new audience. achieve common goals with a diverse group i have found that by feeling included and listened to, people are more motivated to do the hard work of learning and trying new things. in my institution, metadata was still just a buzzword when i arrived. for technical services staff who have long been overlooked when the library is considering doing something new, it feels energizing to be included in something new, mysterious, and exciting, and this has been a great help in generating the energy necessary to overcome the many barriers and gaps we’ve needed to bridge. another way to generate energy for this work is to orient your project people to the big-picture goal of the project work. communication about the library’s broader mission and goals aren’t always emphasized in the same way for support staff, who often don’t have access to the same meetings or emails where these are reiterated. by sharing understanding of the “why” of the work they are doing, as well as the “who” it will benefit, you can help people choose to align themselves with common goals of a diverse group. conclusion i think that by intentionally including efforts to be inclusive in staff training, we can increase the diversity of participants in technology-rich library work. an important step in this process is acknowledging barriers to participation faced by diverse work groups, and identifying solutions for moving past those barriers. it is equally important to acknowledge that it takes extra effort and communication to successfully bridge the perspectives and experiences of a diverse work group. i would love to hear more about the processes that others have used to incorporate “traditional” library staff in their technology-rich workflows, as well as what the best practices for on-boarding processes look like for new hires in emerging or innovative positions. i hope that by sharing my experiences, we can start a conversation around the human resources that drive our development as a community. the author sincerely thanks joan leysen for her valuable insight related to the topics in this paper. she also wishes to thank lori kappmeyer and the entire staff of the metadata and cataloging department at iowa state university for their patience and understanding while she learned how to train staff and manage projects on the job. references averkamp, s. 2013. course project. course materials for slis 021:239 topics: conceptual structures/systems: metadata design and development. available from: https://github.com/saverkamp/metadata-design boydston jmk, leysen jm. 2006. observations on the catalogers’ role in descriptive metadata creation in academic libraries. cataloging & classification quarterly 43(2):2-17. doi: 10.1300\j104v43n02_02 (coins) dezelar-tiedman c. 2004. crashing the party: catalogers as digital librarians. oclc systems & services: international digital library perspectives 20(4):145-147. available from: http://dx.doi.org/10.1108/10650750410564600 keenan t, sipe v. 2013. transitioning from cataloging to creating metadata. alcts webinar presented february 27, 2013. available from: http://www.ala.org/alcts/confevents/upcoming/webinar/022713 o’bryan a, palmer kl. 2007. the evolving cataloging department. idea: iupui’s digital archive. available from: http://hdl.handle.net/1805/705 rodgers jr, sugarman t. 2012. library technical services: key ingredients in the recipe for a successful institutional repository. the serials librarian 65:80-86. doi: 10.1080/0361526x.2013.800632 (coins) stahl sa, osborn j, lehr f. 1990. “beginning to read: thinking and learning about print” by marilyn jager adams: a summary. urbana-champaign (il): center for the study of reading, the reading research and education center, university of illinois at urbana-champaign. (coins) valentino l. 2010. integrating metadata creation into catalog workflow. cataloging & classification quarterly 48(6-7):541-550. doi: 10.1080/01639374.2010.496304 (coins) veve m, feltner-reichert m. 2010. integrating non-marc metadata duties into the workflow of traditional catalogers: a survey of trends and perceptions among catalogers in four discussion lists. technical services quarterly 27(2):194-213. doi: 10.1080/07317130903585477 (coins) about the author kelly thompson is the metadata management and cataloging librarian at iowa state university library. her duties include managing metadata projects, wrangling descriptive metadata for isu’s digital repository and digital collections, and cataloging scientific, agricultural, and technical resources. she enjoys thinking critically about discoverability, metadata for open access/library-published resources, gender, and critical thinking pedagogy. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – providing information about reading lists via a dashboard interface mission editorial committee process and structure code4lib issue 19, 2013-01-15 providing information about reading lists via a dashboard interface as developers of the open source lorls resource/reading list management system we have developed a dashboard to better support academic staffs’ understanding of how their students use reading lists. this dashboard provides both graphical and tabulated information drawn from lorls and the aleph integrated library system. development of the dashboard required changes to back-end functionality of lorls such as logging views of reading lists and caching of loan data. changes to the front end included the use of html5 canvas elements to generate pie charts and line graphs. recently launched to academic staff at loughborough university, the dashboard has already garnered much praise. it is hoped that further development of the dashboard will provide even more support for academics in the compilation of their reading lists. by dr jason cooper, dr jon knight and gary brewerton introduction in the uk academics typically produce an annotated list of material to support students in their course of study. traditionally these lists would contain references to books, book chapters and journal articles, and thus are known as reading lists. as the variety of material available to academics and students has evolved so too has the list, which now covers other media types including audio visual material and websites/webpages, and are therefore now sometimes called resource lists. a reading/resource list management systems (rlms) provides academics an easy way to create and maintain their reading lists; it also provides the institution’s libraries with easy access to all the academics’ reading lists as the content of these can directly impact the library’s stock management decisions. the provision of statistical feedback on reading lists to their academic owners has historically been seen as a weak spot for rlms. this issue is one of the areas being tracked in the latest version of the loughborough online reading list system (lorls) (brewerton et al. 2003, knight et al. 2012) by the inclusion of an academic dashboard. the idea of providing a dashboard for the academic owner(s) of lists was strengthened by discussions in the first meeting the reading list challenge workshop (meeting the reading list challenge [updated 2012]) in july 2011. being able to feed back list related information to academics was seen as a good incentive for them to make better use of the system. the current version of lorls is separated into two halves, the back end and the front end. the back end handles the data storage and security checks and is known as the loughborough universal metadata platform (lump). lump is written in perl and uses a mysql databases for storage. the front end is a client for lump (named clump) which handles the user interface. this is an ajax style interface, written in javascript and using the jquery javascript library (jquery … [updated 2012]). the front and back ends of lorls communicate via a number of restful apis which can return data in either xml, json or jsonp. implementing the dashboard in lorls required a number of changes to the existing apis as well as the creation of a number of new apis. the front-end interface also needed to be extended to make use of the new features added to existing apis. a new dashboard interface had to be created to present the information provided by the new apis to the academic owners of each reading list. the dashboard the academic dashboard in lorls is available on a per reading list basis, to any users with edit permission for that list. the dashboard option has been included in the toolbar alongside other options like search and logout. clicking on the dashboard option brings up a lightbox effect window within which the dashboard contents are displayed. it was decided early on in that the information presented by the dashboard should, where possible, be presented in a visual format. for example, rather than just reporting the total number of views for a reading list a graph should be used to show those views over time, thus presenting more information without overloading the academic with it. figure 1. screenshot of the academic dashboard the list composition information gives a summary count of the types of items on the list, how many items have a url as part of their metadata and if any of those urls had been marked as not working by the system. if there are any urls that have been marked as not working an option is provided that returns the owner to their list and highlight those items with nonworking urls. to the right of the list composition information are two graphs, one above the other. the top graph shows the number of users who have viewed the reading list over the last year. library staff are not reported in the number of views to give a more accurate representation of student usage. the bottom graph shows the number of items on the reading list that have been loaned out from the library within the last year. the remainder of the academic dashboard consists of three questions. clicking on a question will cause it to expand out with the answer in tabular form. when viewing an item on a reading list the student is presented with two icons for rating the item: a thumbs up and thumbs down. the thumbs up icon can be used to mark that they liked the item and the thumbs down icon can be used to mark that they disliked the item. the students’ votes against items are used to provide the answers to the first two expandable questions on the academic dashboard. figure 2. screenshot showing ratings icon the “which items do my students like?” question shows a table displaying the highest user ranked items, how many people had marked them in a positive way, how many had marked them in negative way and the net score for the item. only items with a positive net score are considered when deciding which items to display. an option to return to the reading list and highlight the items shown is also provided. figure 3. screenshot showing expanded answer the “which items do my students dislike?” question shows the same information as the previous question except for the lowest user ranked items. this time only items with a negative net score are considered for displaying. again an option is provided to return to the reading list and highlight the items shown. the final question, “which items aren’t being borrowed?”, presents a list of items that haven’t been borrowed in the last year, yet are available in the integrated library system (ils). once again there is an option to return to the reading list and highlight these items. logging views the initial change required for lorls was to set up logging views of reading lists. this required changes to both the front and back end. to access information on a reading list the front end calls the fastgetstructuralunit api. due to the threaded nature of clump the fastgetstructuralunit api is called multiple times when displaying lists; this in turn required an additional option to be added to the fastgetstructuralunit api so that the front end could specify that a call shouldn’t be logged as a view. when logging a view, the source ip address of the request, the name of the client software in use (usually clump in our case), the id numbers for the particular reading list, the username of the person and the time of the request are recorded in a row in a new “view_log” table in the lump database. a new viewusage api was added to allow the front end to access information stored in the view_log table. it takes the id of the reading list and returns a list of views including a timestamp for each view. this information can then be processed by the front end and used to generate a graph of views over a period of time. the viewusage api allows requests to discard views made from machines in given ip address ranges (details of which are held in a separate table). this allows accesses by library staff or during development/debugging of the code to be ignored. the remaining ip addresses are anonymised using the ip::anonymous perl module from cpan (ip::anonymous … [updated 2012]). the viewusage api can also restrict returned viewing matches based on any combination of the client software name, the username of the person viewing the sus, the source ip address (prior to anonymising) and a start/end timestamp. item loans initially the local ils was queried dynamically to obtain the loan information. as lorls attempts to be ils agnostic the code to query it was kept separate. this turned out to be beneficial as due to performance issues actively querying the ils was soon demonstrated to be an unworkable solution. a local loans cache database was created that was specifically designed to provide the required information more efficiently. the loans cache table uses a script that runs each day and gets any loans that were either issued or returned for the previous day. these details are then used to update the local loans cache. there are currently two api calls available for the local loans cache: ‘loanhistory2’ and ‘lastloaned’. the first api call takes a list of isbns and returns, for each day in the last year, the total number of items on loan which match any of those isbns. the second api also takes a list of isbns but returns for each isbn if and when it was last issued. the front end uses the data returned by the first api to provide the graph showing items loaned and the data from the second api to produce the list of items not borrowed in the last year. item rating the introduction of a feature for users to rate individual items on a reading list required the addition of a new table in the database and a number of new apis for the front end to interact with it. the new apis are ‘editsurating’ and ‘getsurating’. editsurating will set a logged in users rating as either positive (thumbs up) or negative (thumbs down). an individual user can only have one rating attached to any structural unit; any future calls of editsurating for that su will alter the existing rating for that user. the getsurating api retrieves for each su specified the total number of positive ratings and the total number of negative ratings. graphing technology traditionally graphs shown in web sites are either static images generated and stored on the server or images dynamically generated by cgi scripts running on the web server. both methods increase the load on the web server and are impractical when the graphs are expected to make full use of the available space on a page, even after the browser has been resized. the solution to this issue used in lorls’s dashboard is to have the back end supply the data to the browser and for the browser to dynamically generate the graphs itself. this became possible with the introduction of html5’s canvas element and flot (flot … [updated 2012]), a jquery plug-in for drawing graphs. flot supports a number of different graph styles (line, bar, pie, etc.) and also supports a number of advanced features including dynamically resizing the graph when its containing element is resized. this method is not without its own issues though. for example, version 8 and lower of internet explorer (ie) does not support the canvas element. this requires lorls v7 to add a canvas element to these versions of ie through the use of the explorercanvas javascript library (explorercanvas [updated 2012]) which emulates the canvas element through the use of ie’s vector markup language (vml). launch and future development the dashboard was launched to academics at loughborough in august 2012. feedback has been very positive with many academics praising the graphical design of the dashboard and appreciating the insight the dashboard gives them into the use of their reading lists. enhancing the dashboard will be part of future versions of lorls as there is still a lot of information that the academic owners may find useful when creating their lists. areas that are currently under investigation to be added or enhanced in future versions of the dashboard include loan information, pricing and download counts for online items. the loan information can be extended with more details such as which books are the most loaned, what are the average number of loans and what the standard deviation is. given the recent increase in tuition fees for students, academics may feel the need to be more aware about the cost of resources they are expecting their students to own/access. information relating to the total and average cost of books on the list may be of benefit to the academic owners as well as librarians when updating their lists. as more items on reading lists are available electronically via special copyright cleared licenses, being able to view the number of downloads of these items alongside the loan information will help present a fuller picture of how students are making use of the reading list. conclusion the inclusion of an academic dashboard for reading list owners increases the information they have available for managing their reading lists. they can use it to gain an understanding of how their students are using reading lists: if and when they are checking them or borrowing items from them. they can also see which items their students have reported as being particularly useful and which items were not. finally they can see which books on their reading list haven’t been loaned out by the library and use this information when revising their reading list and lecture notes. the dashboard in lorls will continue to be enhanced in future versions with an emphasis on providing information to the academic owners that helps them make the most of their reading lists. references brewerton g, knight j. 2003. from local project to open source: a brief history of the loughborough online reading list system (lorls). vine, 33(4):189-195. available from https://dspace.lboro.ac.uk/handle/2134/441 knight j, cooper j, brewerton g. 2012. redeveloping the loughborough online reading list system. ariadne, (69). available from http://www.ariadne.ac.uk/issue69/knight-et-al meeting the reading list challenge [internet]. [cited 2012 october 08]. available from http://blog.lboro.ac.uk/mtrlc/ jquery: the write less, do more, javascript library [internet]. [cited 2012 october 08]. available from http://jquery.com/ ip::anonymous : perl port of crypto-pan to provide anonymous ip addresses [internet]. [cited 2012 october 08]. available from http://search.cpan.org/dist/ip-anonymous/lib/ip/anonymous.pm flot: attractive javascript plotting for jquery [internet]. [cited 2012 october 08]. available from https://code.google.com/p/flot/ explorercanvas [internet]. [cited 2012 october 22]. available from http://excanvas.sourceforge.net/ about the authors dr. jason cooper <j.l.cooper@lboro.ac.uk> is the library systems analyst/programmer at loughborough university. he is responsible for developing and maintaining a number of key systems for the library, including the front end of the lorls (an open source reading list management system). dr. jon knight <j.p.knight@lboro.ac.uk> is the library systems developer at loughborough university. he is responsible for developing and maintaining the back end of lorls (an open source reading list management system). gary brewerton <g.p.brewerton@lboro.ac.uk> is the library systems manager at loughborough university and is interested in library automation, information and communication technology, and open source development (e.g. lorls). subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – institutional data repository development, a moving target mission editorial committee process and structure code4lib issue 51, 2021-06-14 institutional data repository development, a moving target at the end of 2019, the research data service (rds) at the university of illinois at urbana-champaign (uiuc) completed its fifth year as a campus-wide service. in order to gauge the effectiveness of the rds in meeting the needs of illinois researchers, rds staff developed a five-year review consisting of a survey and a series of in-depth focus group interviews. as a result, our institutional data repository developed in-house by university library it staff, illinois data bank, was recognized as the most useful service offering by our unit. when launched in 2016, storage resources and web servers for illinois data bank and supporting systems were hosted on-premises at uiuc. as anticipated, researchers increasingly need to share large, and complex datasets. in a responsive effort to leverage the potentially more reliable, highly available, cost-effective, and scalable storage accessible to computation resources, we migrated our item bitstreams and web services to the cloud. our efforts have met with success, but also with painful bumps along the way. this article describes how we supported data curation workflows through transitioning from on-premises to cloud resource hosting. it details our approaches to ingesting, curating, and offering access to dataset files up to 2tb in size–which may be archive type files (e.g., .zip or .tar) containing complex directory structures. by colleen fallaw, genevieve schmitt, hoa luong, jason colwell, and jason strutz introduction in 2016, the university of illinois at urbana-champaign (uiuc) launched illinois data bank, a public access repository for publishing research data from illinois researchers. illinois data bank is part of a larger digital library system at uiuc called medusa, which is composed of a set of internal digital preservation and management tools, along with several patron-facing access systems (robbins, 2018). all deposited datasets receive professional curation services to ensure their completeness, understandability, and access for use in the future. in the strategic plan for 2018-2023, uiuc continues emphasizing the importance of “responsible data sharing practices throughout the institutional and constituent lifecycles” (strategic plan, 2018). the year it launched, publication of the article “overly honest data repository development” provided a holistic view of the development process for the illinois data bank (fallaw et al., 2016). the initial technical and service design choices were detailed in the article. going into illinois data bank’s fifth year of operation, researchers increasingly express a need to share large, complex datasets. this calls for reliable, highly available, cost-effective, scalable storage accessible to computation resources. the promise of these features in cloud services appeared well-suited to meet those needs, and implementation met with some meaningful success, but also with painful bumps along the way. challenges of large, complex datasets for ingestion, curation, and access all faculty, graduate students, and staff conducting research affiliated with uiuc can deposit their research data into the illinois data bank. published datasets are then available to everyone around the world for viewing and downloading. to align with the spirit of open data for data sharing, all deposited datasets are curated to ensure completeness, understandability, and accessibility in the future. curatorial review is a “human layer” that brings the disciplinary knowledge and software expertise necessary for reviewing incoming submissions to ensure that the data meet the fair principle: findability, accessibility, interoperability, and reusability (johnston, 2018). the depth and length of curation review depends heavily on the size and structure of the dataset and how much detail is in the documentation. based on our internal data, approximately two hours is an average amount of time spent to curate a deposited dataset in the illinois data bank. the curation services that we offer include, but are not limited to: • review files and metadata for completeness. • check documentation thoroughly for understandability. • validate files, including running any included computer code to the extent availability, license terms, and resources permit. • link to related publications and other materials. while researchers surveyed by springer nature identified non-technical issues, such as “organizing data in a presentable and useful way,” as the main challenges to data sharing, file size was also found to impact data sharing. “respondents that generate the smallest data files (<20mb; n = 2,036) have the highest proportion of data that are neither shared as supplementary information nor deposited in a repository (42%). in contrast, 70% of those with data files greater than 50gb (n = 700) share their data, with a strong preference for sharing through repositories (59%)” (stuart et al., 2018). large datasets also create challenges for the curation process. in order to curate a dataset, our curator downloads as many files in the dataset as possible to check for completeness and possible errors. however, this is not always a case, especially when the dataset contains multiple large files. for example, research data from data-intensive disciplines, such as atmospheric sciences, electrical, and computer engineering, are normally deposited in large, zipped files (some above 50gb). to curate those, the curator needs technical assistance from development and infrastructure professionals in library it. in most cases, physically sharing and transporting an external, portable hard drive which contains files that curator needs, would work. however, this consumes cumbersome time and resources even when participants are not navigating a pandemic, so we were motivated to find an alternative solution to not only support depositors and users of datasets, but also curators. for this paper and in planning discussions among the staff supporting the illinois data bank, file size in datasets starts being “large” at the size it is not trivial to ingest and access it in a standard web form. because our policies allow for multi-tb deposits, we have pragmatically found the need for additional solutions at approximately 4gb, 15gb, and 50gb. as shown in figure 1, almost all the files deposited are 4gb or smaller. however, as shown in figure 2, datasets with larger files are being published in the illinois data bank with increasing frequency. figure 1. file counts by size category in illinois data bank 2016-04-01 to 2021-04-01. figure 2. file sizes over time in illinois data bank 2016-04-01 to 2021-04-01. in the context of technically supporting data sharing in illinois data bank, a dataset is “complex” when there is more than about one page worth of files to list in a dataset, or if the data is organized hierarchically in directories. curators have identified archive formats such as .zip, .7z, and .tar to be preservation challenges, in part because of a lack of transparency as to the format of the files inside (braxton, 2018). also, in informal feedback, depositors and users of datasets in illinois data bank have identified being able to see the structure of the datasets in the web application as important to data sharing. while the datafiles in illinois data bank are in a large variety of file formats, 19.5% (n = 859) are in some kind of archive format, requiring additional processing to reveal the internal structure and file types for use and curation (fallaw, 2021). figure 3. file type categories. on-premises solutions when illinois data bank launched, all storage and web service resources were hosted and administered on-campus. at launch, illinois data bank supported ingestion of files up to 15 gb directly from box through the web interface. larger files were transferred with portable hard drives and ingested manually by technical support staff. we soon added a data file ingest api and sample python client, which worked by incrementally adding chunks to a file on a filesystem, which became the go-to solution for command-line transfer of files from research compute clusters. box offers api access, and illinois data bank uses a file import box widget with features that include single sign-on and integrated file selection. box has a per-file limit of 15gb. the data file ingest api was used to transfer data exclusively with command line access, which is the case for some of the compute clusters where researchers store and process data. it also supported files up to the policy limit of 2tb. on-premises to cloud migration the file storage element of medusa at the time of illinois data bank’s launch had been designed to be integrated with a compute cluster for active research projects, not to support highly-available web applications with uncertain and variable storage needs. we wanted storage designed to be more highly available, and to scale better with storage needs, which we expected to grow, though it was not clear to us how much or when. an additional element of the environment building momentum toward migration to cloud architecture was that our university’s centralized technical services group was brokering a deal with amazon web services (aws) and encouraged us to take advantage of the potential benefits they were evaluating. this aligned with trends in the library community (goldner, 2010). in approaching our initial migration to cloud architecture, we understood that we had a lot to learn and that every change would have potential to uncover complications, but we judged that it was worth at least a pilot program to test the concept. as is common in large innovative technical projects, it was easy to underestimate the time, effort, and complications involved. on february 18, 2019 we switched the production instances of our digital library storage and web applications from on-campus platforms to amazon web services (aws). file system storage systems are the kind of system with folders containing files like you would find on a personal computer. in contrast, aws s3 and google drive are referred to as object stores. aws does offer a flexible file system storage called elastic file system (efs), but it has a per-gb per-month price of ~30 cents for efs vs ~3 cents for s3 for our initial ~150tb. we opted to adapt to s3 object storage buckets (robbins et al., 2019). a fully on-premises implementation was the most familiar to us. we wanted a minimum of three replicated copies in two physically separate locations in order to durably protect data, and in order to maintain a secondary read-only option during scheduled and unscheduled maintenance at the primary location. the physical, logistical, and technical challenges were daunting. appliances that managed replicated copies were readily available on the market, but were expensive compared to bulk storage. creating and maintaining a data center within library it facilities would require a large initial investment with repeating refresh cycles, which could not be accommodated at the scale we needed on a consistent timeline given the fluidity of higher education budgets. at launch, illinois data bank and the supporting infrastructure was built on virtual hardware. different functional components or software stacks were built on separate virtual servers, with cpu, ram, disk, and network bandwidth allocated according to each function. the production environment had eight virtual servers, all of which had to be managed for operation system and software updates and monitored for security and performance limitations. the virtual cluster disk was configured for daily backup via volume snapshot, which was great for the operating system volumes but was not feasible in a realistic backup window for the amount of raw data in the repository. for that purpose, we turned to network attached storage, which replicated via rsync to a secondary block storage location in a different building. we were able to fail over by changing the mount point configurations on each application server and restarting them in the appropriate sequence. although the on-premises setup was cost effective, there were problems with reliability and scale. the primary and secondary storage were in different buildings from the virtual server cluster and frequently experienced network issues. an outage to any of the locations had the potential to partially or completely disable the repository. we were also required to buy additional storage in 30tb chunks with up to eight weeks of lead time due to procurement and installation delays. in order to accommodate growth and to improve stability, we decided to move all the primary repository storage to amazon s3. for both campus-based and cloud-based approaches, we decided to use aws s3 glacier as a completely separate disaster recovery location for all data. with the application and primary data on premises, we periodically uploaded an archive file (tar.gz format) with the content to be preserved to glacier, and maintained an index of what content was in each archive. with the cloud implementation, we have cross region replication setup to copy data unidirectionally to a separate aws region when any object is uploaded to medusa’s s3 bucket in the primary region. this ensures strong durability at extremely low cost, with no chance of automated processes propagating accidental deletions or changes to the preservation copy. it is very slow and expensive to retrieve the entire data set, and complicated to identify a smaller part of the data set to restore in the event of disaster, but this cost was accepted as a last resort storage location for the digital objects. setting up mount points to s3 buckets is comparable in complexity to other types of network attached storage. however, because s3 is an object storage platform, we could take advantage of apis to reduce or eliminate file system calls. combined with services offered natively in aws, such as the relational database service and elasticsearch, we were able to completely remove several virtual machines from the system. this created a shift in how infrastructure maintenance was done—instead of keeping a list of all servers that needed to be patched, monitored, and controlled for changes, we had services with monitors and alerts maintained within the aws console. because everything was defined in our infrastructure provisioning tool (terraform), we could predict when changes would have an effect on other parts of the system so they could be tested, and any mistakes could be rolled back by reverting to the previous known-good configuration. we went from managing many parts of a single system separately to managing the entire system holistically. cloud solutions some of the medusa servers involved with supporting the illinois data bank were hosted on virtual machines (vms) administered by centralized campus it services, while others were entirely maintained by library it staff. while we used the infrastructure automation framework puppet to some extent, we hoped to administer our infrastructure more efficiently and effectively—focusing on innovation rather than routine maintenance. cloud solutions include globus-based solutions to handle large files. the illinois data bank manages the complexity of a large number of files or hierarchically structured files by organizing them into archive files types such as zip or tar. a custom service developed in-house examined and reported on the contents of archive file types such as zip and tar to support use and curation of complex dataset directory and file structures. this service relied on filesystem tools. our initial adaptation to cloud infrastructure involved migrating the vm to an ec2 and temporarily copying binary objects from s3s buckets to ecs. to further adapt this service to cloud infrastructure, we dove into aws services for scalable microservices. the differences between how file storage and object storage work required extensive adaptation of our systems and approaches to content transfer. and even file system storage provided by cloud services over the internet has connection interruptions and delays that require more flexible and robust solutions than an all on-campus infrastructure. we created custom workflows that involve the transfer of files between campus storage and cloud storage: dataset files of up to 2tb into and out of illinois data bank by campus researchers. the challenge of needing increased availability, scalability, and flexibility has been effectively addressed with our migration to cloud infrastructure. the biggest change for system administration is that library it has full control over infrastructure. using on campus resources relied on system administrators from other groups for provisioning and responding to incidents for storage and compute resources. overall, library it has a much easier time managing resources now since s3 is very reliable, and if more compute resources are needed, that can be adjusted directly. one side-effect to having full control over infrastructure is the need to better keep track of expenses. the library it infrastructure team was used to using monitoring tools to keep track of servers and services, but we needed a similar vision into our daily and weekly aws infrastructure costs. after tracking expense trends in the first few months operating in aws, we created aws budgets to notify us when expenses are beyond what we were expecting. with the added responsibility of keeping costs under control, the change has been a net positive for the infrastructure team. it has turned many of our less fun infrastructure management tasks like provisioning and deprovisioning infrastructure, user and access management, security, and cost management into challenging puzzles using aws’s automation tools and third-party tools like terraform and ansible. while system administration became more interesting and effective, many elements of the application needed to be adapted with migration to the cloud. one of the adaptations from an all on-campus infrastructure vs. cloud object storage and web servers involved modifying the api for data transfer. there is support in the aws s3 api for chunked file upload, although the minimum chunk size was larger than we had used previously. to adapt to cloud infrastructure and improve upload performance, library it staff implemented tus resumable file upload protocol (tus) with s3 backend. this produced noticeably faster uploads in the web interface compared to http without the tus layer, and initial testing of the api indicated this could be a workable solution. however, “the [tus] protocol uses a data model which is mostly incompatible with aws s3” and integrating “requires some workarounds” (kleidl, 2016). in field conditions, our researchers started encountering frustrating reliability and transparency issues with files larger than ~100 gb using the updated api and sample client adapted to our cloud architecture. on top of the aggravation of failed transfer attempts, this led to rushed, disruptive portable hard drive transfers. enter globus. as described on their website at the time of this writing, “globus is a non-profit service for secure, reliable research data management. with globus, subscribers can move, share, & discover data via a single interface – whether your files live on a supercomputer, lab cluster, tape archive, public cloud or your laptop, you can manage this data from anywhere, using your existing identities, via just a web browser. developers can also use globus to build applications and gateways leveraging our advanced identity management, single sign-on, search, authorization, and automation capabilities” (globus, 2021). because globus is a transfer service, not a separate kind of file system, it does not maintain exclusive control of the binary objects it has access to through a storage gateway. files transferred between endpoints using globus can be manipulated using other tools in the same location. this was key to its utility for our purpose. globus offers a subscription service. the university of illinois has a campus-wide non-profit research and education subscription for globus online. our subscription includes unlimited managed endpoints plus premium storage connectors for aws s3 and google drive. a managed endpoint means the endpoint can use the subscription features, such as hosting shared endpoints and any premium storage connectors. shared endpoints (a.k.a guest collections) are required to share data using globus connect server v5. posix storage systems include the kind of system with folders containing files like you would find on a personal computer. aws s3 and google drive are non-posix object stores. premium storage connectors are required to use non-posix storage with an endpoint. setting up globus for use by illinois data bank required significant organizational and technical investment initially, but it not only supported considerably smoother ingestion of data files into datasets, but library it staff now use it for other backend services as well as additional file transfer tasks that have arisen. globus ingest steps figure 4. globus ingest steps. while the pain that prompted us to integrate globus into illinois data bank related to transfer in, once it was set up, we also offered transfer out, which is increasingly important as illinois data bank hosts larger datasets. it also supports networked curation of large complex datasets within the data curation network. at launch, illinois data bank adapted a system for supporting large downloads that was already in place for the medusa preservation system. one of the shared backend services of medusa is a downloader service: a ruby on rails application combined with an nginx server running mod_zip and another nginx server to allow streamable zip downloads of medusa content. downloads with a large number of files are served directly through rails using a helper program clojure-zipper. illinois data bank interacts with the medusa downloader service using an api—sending a list of files and getting back a download link, which it in turn displays to the end user. while the medusa downloader system was updated effectively to adapt to cloud infrastructure, larger datasets can be transferred more robustly and flexibly using globus. also, for institutions and individuals already using globus, some may prefer to use it when available even for smaller files. we set up a shared globus endpoint on an s3 bucket with read permission granted to all logged-in users. as each dataset is published, a background job copies the files to that endpoint using a naming convention. when a dataset landing page is served, illinois data bank checks if the files are available in globus. if available, the interface offers a link to the globus file manager page for that dataset. if problems arise from illinois data bank users attempting to download very large datasets using other methods, we may disable non-globus downloads for those datasets, where “very large” means whatever size causes problems. curator preservation challenges for archive file types many of the largest and most complex datasets include archive type files, such as .zip, .tar, or .7zip. in order to offer a listing of the contents of the archive files in the user interface and for curation analysis, we extract the files to filesystem storage and traverse the resulting tree. other times it is possible to extract the needed information from metadata in the archive file, but even that requires tools that require the object to be stored on a filesystem. we use s3 for long term storage instead of efs because of the order of magnitude difference of expense mentioned above in our description of migrating from on-premises to cloud infrastructure. however, the flexibility of cloud infrastructure means that we can provision efs on a temporary basis cost effectively when processing archive files. the process of extracting content metadata from archive files is resource intensive, but infrequent, making it a good candidate for something more flexible than a set-size vm on all the time. to take advantage of cloud infrastructure support for flexible resource use, the archive extractor process utilizes an aws fargate elastic container service task running a custom docker image stored in the aws elastic container repository (ecr). the image consists of a ruby script utilizing efs for transient structured storage and aws simple queue service (sqs) for communication back to the illinois data bank with the information extracted from the archive files. this serverless architecture is a simplification from a persistent aws elastic compute cloud (ec2) instance running a ruby on rails api to process the archives. the ec2 solution followed existing development paradigms during the transition from campus-hosted infrastructure to the aws cloud. as development has settled into the aws ecosystem it became apparent the archive extraction process could be transitioned into a serverless solution. research archive deposits into the illinois data bank are large, infrequent, and require increased compute power for short bursts of time making the extraction process ideal to utilize a serverless solution. the most notable serverless architecture provided by aws is lambda. it allows developers to simply upload code as a .zip file in a variety of supported languages, seamlessly integrates with aws elastic file system, and is triggered through customizable json events. naturally, lambda was the initial architecture investigated for the archive extractor and after nearly a month of implementation, the limitations of this solution were brought to our attention. while extremely powerful and user-friendly for novice cloud developers, lambda functions can only run for a maximum of 15 minutes before they timeout. illinois data bank can handle files up to 2 tb which can take hours to process even under ideal computing conditions. with this constraint in mind, we worked with an aws technical account manager to brainstorm another aws serverless solution and we decided to pursue aws fargate elastic container service. fargate is a serverless solution similar to lambda with more customization and fewer restrictions which can lead to development complications. whereas lambda works with .zip program files, fargate utilizes container images thus working with fargate requires prior knowledge of docker containers or rapid familiarization. once the ruby script was migrated to a docker container and uploaded as an image, the next hurdle was connecting the fargate task to the efs. ideally, the process to set up an efs connection with fargate is straightforward, the file system is added to the fargate task as a mount point and connected as an external volume. however, this relies on the aws virtual private cloud (vpc) utilizing the aws domain name system (dns), and due to university constraints, the existing vpc uses a custom dns. in an attempt to mitigate this issue, we tried to attach the efs volume through the docker container itself outside of the cloud, however, this is not supported in aws. instead, we were left with two solutions. first, we could add dns forwarding and tackle the setup, and maintenance while risking the stability of our existing infrastructure on the vpc or we could create a new separate vpc to use aws dns for the task. we chose to create a new vpc, accepting the added risk for the increased stability and reduced risk to our existing production environments on the vpc. with this new infrastructure in place, we were able to connect the efs to the fargate task and begin fully testing the archive extractor process. initial testing proved successful as smaller files were able to be uploaded to the efs processed in the ruby script and the output was written to the sqs to be queried and handled by the illinois data bank. subsequent testing of larger files demonstrated fargate’s ability to compute for extended periods of time, however, it also highlighted a key limitation in the sqs communication between the archive extractor service and illinois data bank. sqs queues have a size limit of 256 kb per message. this can be extended up to 2 gb through the sqs extended client library which is unfortunately only available for java at the moment. this new constraint left us pondering a few different solutions including splitting json results into smaller messages, creating a wrapper for the java library to be used in ruby, or uploading the json result to an s3 bucket and sending the bucket key to the illinois data bank as the sqs message. with a similar thought process to creating a new vpn, to minimize future maintenance and increase stability we chose to write the json to an s3 bucket to be read from the illinois data bank backend server. this solution utilized existing aws products without the need to maintain any new technologies ourselves. as with any technology there will always be more bugs, however, for now it appears we have worked through the major challenges of transforming an ec2 solution into a serverless architecture with minimal overhead and increased technological support via aws. conclusion and beyond the initial technical and service design choices made at illinois data bank’s launch addressed the institutional data repository challenges of depositing, using, and curating large, complex datasets with tools designed to work with file systems rather than object stores and internal networked connections between storage systems and web servers rather than cloud-based connections. during the five years since launch, library it at uiuc has pursued improvements in reliability, performance, and flexibility through a migration of repository infrastructure to a cloud platform. we discovered and developed tools, such as globus and illinois data bank archive extractor. in working through what sometimes seemed like intensely nested layers of complexity and surprises in the migration, we developed new patterns of infrastructure management that afford higher levels of flexibility and position us for further innovation. our anticipated next targets for institutional data repository development and migration efforts are using sqs to replace our rabbitmq server, and more automation integration with globus. we currently maintain a rabbitmq server on an ec2 to handle communication between repository service components. we are analyzing a migration to sqs to potentially further reduce server administration and infrastructure costs. serving as a pilot use case, the recently migrated, custom developed archive extractor described above uses sqs to communicate with illinois data bank. while migration to sqs would be invisible to users of the systems, improved integration with globus could improve user experience. each step in the globus ingest workflow represented in figure 4 is vulnerable to staff availability, and many steps require coordination between the depositor and repository staff. emerging features of the globus and aws apis may support smoother automation of more steps. on the dataset curation and use side, very large datasets could overtax html-based download options, so limiting download options in the interface to only globus for those datasets may be necessary for reliability and performance–for the users downloading the files as well as other users accessing repository web services at the same time. information sharing needs of publishers, curators, and users of datasets in institutional data repositories continue to be moving targets in an ever-shifting technology landscape. research data service and library it staff expect to continue to use a mix of on-premises resources, cloud platforms, third party tools, custom code, and whatever comes next to meet those needs. acknowledgements we thank heidi imker, director of the research data service and associate professor, university library at uiuc, for feedback and editing. references [1] braxton, s., fallaw, c., luong, h., orlowska, d., hetrick, a., rimkus, k., anderson, b., & imker, h. (2018). should we keep everything forever? determining long-term value of research data. poster presented at ipres2018, boston, usa. http://hdl.handle.net/2142/91659 [2] fallaw, c. (2021). datafile features in illinois data bank datasets 2016-04-01 to 2021-04-01. university of illinois at urbana-champaign. https://doi.org/10.13012/b2idb-7291801_v1 [3] fallaw, c., dunham, e., wickes, e., strong, d., stein, a., zhang, q., rimkus, k., ingram, b., & imker, h.j. (2016). overly honest data repository development. the code4lib journal, no. 34. https://journal.code4lib.org/articles/11980 [4] gewin, v. (2016). data sharing: an open mind on open data. nature 529, 117–119. https://doi.org/10.1038/nj7584-117a [5] globus. (2021). what we do. globus. www.globus.org/what-we-do [6] goldner, m. (2010). winds of change: libraries and cloud computing. oclc online computer library center, inc. https://www.oclc.org/content/dam/oclc/events/2011/files/ifla-winds-of-change-paper.pdf [7] johnston, l.r., carlson, j., hudson-vitale, c., imker, h., kozlowski, w., olendorf, r., stewart, c., blake, m., herndon, j., mcgeary, t.m., hull, e., & coburn, e. (2018). data curation network: a cross-institutional staffing model for curating research data. international journal of digital curation, 13(1), 125-140. https://doi.org/10.2218/ijdc.v13i1.616 [8] johnston, l. (2020). how a network of data curators can unlock the tremendous reuse value of research data. oclc. https://blog.oclc.org/next/data-curators-network/ [9] kim, y., and burns, c.s. (2016). norms of data sharing in biological sciences: the roles of metadata, data repository, and journal and funding requirements. journal of information science, 42(2), 230–245. https://doi.org/10.1177/0165551515592098 [10] kleidl, m. (2016). tus.io (blog). https://tus.io/blog/2016/03/07/tus-s3-backend.html [11] kowalczyk, s., & and shankar, k. (2011). data sharing in the sciences. annual review of information science and technology journal, 45, 247-294. https://doi.org/10.1002/aris.2011.1440450113 [12] national science board. (2011). digital research data sharing and management. https://www.nsf.gov/nsb/publications/2011/nsb1124.pdf [13] office of science and technology policy. https://web.archive.org/web/20160304043850/https:/www.whitehouse.gov/sites/default/files/microsites/ostp/ostp_public_access_memo_2013.pdf [14] office of the provost. (n.d.). the illinois strategic plan, the university of illinois at urbana-champaign. https://strategicplan.illinois.edu/2013-2016/goals.html [15] office of the provost. (n.d.). the next 150: strategic plan for 2018-2023. the university of illinois at urbana-champaign. https://strategicplan.illinois.edu/ [16]robbins, s., troy, j., rimkus, k., strutz, w.j., & colwell, j. (2019). the challenges and charms of a cloud-based repository infrastructure transition: lifting and shifting the library’s medusa repository. presented at open repositories 2019 conference, hamburg, june 13, 2019. http://hdl.handle.net/2142/104029 [17] robbins, s. (2018). medusa: service-oriented repository architecture at the university of illinois. presented at open repositories 2018 conference, bozeman, mt, june 6, 2018. http://hdl.handle.net/2142/100077 [18] strutz, j. (2017). the medusa repository: turning data into stone. presented at the university of illinois at urbana-champaign it pro forum november 1, 2017 [19] stuart, d., baynes, g., hrynaszkiewicz, i., allin, k., penny, d., lucraft, m., & astell, m. (2018). whitepaper: practical challenges for researchers in data sharing. https://doi.org/10.6084/m9.figshare.5975011.v1 [20] tenopir, c., dalton, e.d., allard, s., frame, m., pjesivac, i., birch, b., pollock, d. & dorsett, k. (2015). changes in data sharing and data reuse practices and perceptions among scientists worldwide. plos one, 10(8): e0134826. https://doi.org/10.1371/journal.pone.0134826 [21] tenopir, c., allard, s., douglass k., aydinoglu, a.u., wu, l., read, e., manoff, m., & frame, m. (2011). “data sharing by scientists: practices and perceptions.” plos one 6,(6): e21101. https://doi.org/10.1371/journal.pone.0021101 about the authors colleen fallaw research programmer at the university of illinois at urbana-champaign library http://orcid.org/0000-0002-0339-9809 mfall3@illinois.edu genevieve schmitt software engineer at the university of illinois at urbana-champaign library https://orcid.org/0000-0002-4974-9501 gschmitt@illinois.edu hoa luong associate director, research data service at the university of illinois at urbana-champaign library https://orcid.org/0000-0001-6758-5419 hluong2@illinois.edu jason colwell systems administrator at the university of illinois at urbana-champaign library https://orcid.org/0000-0003-4150-9172 colwell3@illinois.edu w. jason strutz manager, it infrastructure at the university of illinois at urbana-champaign library https://orcid.org/0000-0002-0284-3517 strutz@illinois.edu subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – easing gently into opensrf, part 1 mission editorial committee process and structure code4lib issue 10, 2010-06-22 easing gently into opensrf, part 1 the open service request framework (or opensrf, pronounced “open surf”) is an inter-application message passing architecture built on xmpp (aka “jabber”). the evergreen open source library system is built on an opensrf architecture to support loosely coupled individual components communicating over an opensrf messaging bus. this article introduces opensrf, demonstrates how to build opensrf services through simple code examples, explains the technical foundations on which opensrf is built, and evaluates opensrf’s value in the context of evergreen. part 1 of a 2 part article in this issue. by dan scott an article in two parts. introducing opensrf opensrf is a message routing network that offers scalability and failover support for individual services and entire servers with minimal development and deployment overhead. you can use opensrf to build loosely-coupled applications that can be deployed on a single server or on clusters of geographically distributed servers using the same code and minimal configuration changes. although copyright statements on some of the opensrf code date back to mike rylander’s original explorations in 2000, evergreen was the first major application to be developed with, and to take full advantage of, the opensrf architecture starting in 2004. the first official release of opensrf was 0.1 in february 2005 (http://evergreen-ils.org/blog/?p=21), but opensrf’s development continues a steady pace of enhancement and refinement, with the release of 1.0.0 in october 2008 and the most recent release of 1.2.2 in february 2010. opensrf is a distinct break from the architectural approach used by previous library systems and has more in common with modern web applications. the traditional “scale-up” approach to serve more transactions is to purchase a server with more cpus and more ram, possibly splitting the load between a web server, a database server, and a business logic server. evergreen, however, is built on the open service request framework (opensrf) architecture, which firmly embraces the “scale-out” approach of spreading transaction load over cheap commodity servers. the initial gpls pines hardware cluster, while certainly impressive, may have offered the misleading impression that evergreen requires a lot of hardware to run. however, evergreen and opensrf easily scale down to a single server; many evergreen libraries run their entire library system on a single server, and most opensrf and evergreen development occurs on a virtual machine running on a single laptop or desktop image. another common concern is that the flexibility of opensrf’s distributed architecture makes it complex to configure and to write new applications. this article demonstrates that opensrf itself is an extremely simple architecture on which one can easily build applications of many kinds – not just library applications – and that you can use a number of different languages to call and implement opensrf methods with a minimal learning curve. with an application built on opensrf, when you identify a bottleneck in your application’s business logic layer, you can adjust the number of the processes serving that particular bottleneck on each of your servers; or if the problem is that your service is resource-hungry, you could add an inexpensive server to your cluster and dedicate it to running that resource-hungry service. programming language support if you need to develop an entirely new opensrf service, you can choose from a number of different languages in which to implement that service. opensrf client language bindings have been written for c, java, javascript, perl, and python, and service language bindings have been written for c, perl, and python. this article uses perl examples as a lowest common denominator programming language. writing an opensrf binding for another language is a relatively small task if that language offers libraries that support the core technologies on which opensrf depends: extensible messaging and presence protocol (xmpp, sometimes referred to as jabber) – provides the base messaging infrastructure between opensrf clients and services javascript object notation (json) – serializes the content of each xmpp message in a standardized and concise format memcached – provides the caching service syslog – the standard unix logging service unfortunately, the opensrf reference documentation, although augmented by the opensrf glossary, blog posts like the description of opensrf and jabber, and even this article, is not a sufficient substitute for a complete specification on which one could implement a language binding. the recommended option for would-be developers of another language binding is to use the python implementation as the cleanest basis for a port to another language. enough jibber-jabber: writing an opensrf service imagine an application architecture in which 10 lines of perl or python, using the data types native to each language, are enough to implement a method that can then be deployed and invoked seamlessly across hundreds of servers. you have just imagined developing with opensrf – it is truly that simple. under the covers, of course, the opensrf language bindings do an incredible amount of work on behalf of the developer. an opensrf application consists of one or more opensrf services that expose methods: for example, the opensrf.simple-text demonstration service exposes the opensrf.simple-text.split() and opensrf.simple-text.reverse() methods. each method accepts zero or more arguments and returns zero or one results. the data types supported by opensrf arguments and results are typical core language data types: strings, numbers, booleans, arrays, and hashes. to implement a new opensrf service, perform the following steps: include the base opensrf support libraries write the code for each of your opensrf methods as separate procedures register each method add the service definition to the opensrf configuration files for example, the following code implements an opensrf service. the service includes one method named opensrf.simple-text.reverse() that accepts one string as input and returns the reversed version of that string: #!/usr/bin/perl package opensrf::application::demo::simpletext; use strict; use opensrf::application; use parent qw/opensrf::application/; sub text_reverse { my ($self , $conn, $text) = @_; my $reversed_text = scalar reverse($text); return $reversed_text; } __package__->register_method( method => 'text_reverse', api_name => 'opensrf.simple-text.reverse' ); ten lines of code, and we have a complete opensrf service that exposes a single method and could be deployed quickly on a cluster of servers to meet your application’s ravenous demand for reversed strings! if you’re unfamiliar with perl, the use opensrf::application; use parent qw/opensrf::application/; lines tell this package to inherit methods and properties from the opensrf::application module. for example, the call to __package__->register_method() is defined in opensrf::application but due to inheritance is available in this package (named by the special perl symbol __package__ that contains the current package name). the register_method() procedure is how we introduce a method to the rest of the opensrf world. registering a service with the opensrf configuration files two files control most of the configuration for opensrf: opensrf.xml contains the configuration for the service itself, as well as a list of which application servers in your opensrf cluster should start the service. opensrf_core.xml (often referred to as the “bootstrap configuration” file) contains the opensrf networking information, including the xmpp server connection credentials for the public and private routers. you only need to touch this for a new service if the new service needs to be accessible via the public router. begin by defining the service itself in opensrf.xml. to register the opensrf.simple-text service, add the following section to the <apps> element (corresponding to the xpath /opensrf/default/apps/): <apps> <opensrf.simple-text> <!-<1> --> <keepalive>3</keepalive> <!-<2> --> <stateless>1</stateless> <!-<3> --> <language>perl</language> <!-<4> --> <implementation>opensrf::application::demo::simpletext</implementation> <!-<5> --> <max_requests>100</max_requests> <!-<6> --> <unix_config> <max_requests>1000</max_requests> <!-<7> --> <unix_log>opensrf.simple-text_unix.log</unix_log> <!-<8> --> <unix_sock>opensrf.simple-text_unix.sock</unix_sock> <!-<9> --> <unix_pid>opensrf.simple-text_unix.pid</unix_pid> <!-<10> --> <min_children>5</min_children> <!-<11> --> <max_children>15</max_children> <!-<12> --> <min_spare_children>2</min_spare_children> <!-<13> --> <max_spare_children>5</max_spare_children> <!-<14> --> </unix_config> </opensrf.simple-text> <!-other opensrf services registered here... --> </apps> the element name is the name that the opensrf control scripts use to refer to the service. the <keepalive> element specifies the interval (in seconds) between checks to determine if the service is still running. the <stateless> element specifies whether opensrf clients can call methods from this service without first having to create a connection to a specific service backend process for that service. if the value is 1, then the client can simply issue a request and the router will forward the request to an available service and the result will be returned directly to the client. the <language> element specifies the programming language in which the service is implemented. the <implementation> element pecifies the name of the library or module in which the service is implemented. (c implementations only): the <max_requests> element, as a direct child of the service element name, specifies the maximum number of requests a process serves before it is killed and replaced by a new process. (perl implementations only): the <max_requests> element, as a direct child of the <unix_config> element, specifies the maximum number of requests a process serves before it is killed and replaced by a new process. the <unix_log> element specifies the name of the log file for language-specific log messages such as syntax warnings. the <unix_sock> element specifies the name of the unix socket used for inter-process communications. the <unix_pid> element specifies the name of the pid file for the master process for the service. the <min_children> element specifies the minimum number of child processes that should be running at any given time. the <max_children> element specifies the maximum number of child processes that should be running at any given time. the <min_spare_children> element specifies the minimum number of idle child processes that should be available to handle incoming requests. if there are fewer than this number of spare child processes, new processes will be spawned. the`<max_spare_children>` element specifies the maximum number of idle child processes that should be available to handle incoming requests. if there are more than this number of spare child processes, the extra processes will be killed. to make the service accessible via the public router, you must also edit the opensrf_core.xml configuration file to add the service to the list of publicly accessible services: making a service publicly accessible in opensrf_core.xml <router> <!-<1> --> <!-this is the public router. on this router, we only register applications which should be accessible to everyone on the opensrf network --> <name>router</name> <domain>public.localhost</domain> <!-<2> --> <services> <service>opensrf.math</service> <service>opensrf.simple-text</service> <!-<3> --> </services> </router> this section of the opensrf_core.xml file is located at xpath /config/opensrf/routers/. public.localhost is the canonical public router domain in the opensrf installation instructions. each <service> element contained in the <services> element offers their services via the public router as well as the private router. once you have defined the new service, you must restart the opensrf router to retrieve the new configuration and start or restart the service itself. complete working examples of the opensrf_core.xml and opensrf.xml configuration files are included with this article for your reference. calling an opensrf method opensrf clients in any supported language can invoke opensrf services in any supported language. so let’s see a few examples of how we can call our fancy new opensrf.simple-text.reverse() method: calling opensrf methods from the srfsh client srfsh is a command-line tool installed with opensrf that you can use to call opensrf methods. to call an opensrf method, issue the request command and pass the opensrf service and method name as the first two arguments; then pass one or more json objects delimited by commas as the arguments to the method being invoked. the following example calls the opensrf.simple-text.reverse method of the opensrf.simple-text opensrf service, passing the string "foobar" as the only method argument: $ srfsh srfsh # request opensrf.simple-text opensrf.simple-text.reverse "foobar" received data: "raboof" =-----------------------------------request completed successfully request time in seconds: 0.016718 =-----------------------------------getting documentation for opensrf methods from the srfsh client the srfsh client also gives you command-line access to retrieving metadata about opensrf services and methods. for a given opensrf method, for example, you can retrieve information such as the minimum number of required arguments, the data type and a description of each argument, the package or library in which the method is implemented, and a description of the method. to retrieve the documentation for an opensrf method from srfsh, issue the introspect command, followed by the name of the opensrf service and (optionally) the name of the opensrf method. if you do not pass a method name to the introspect command, srfsh lists all of the methods offered by the service. if you pass a partial method name, srfsh lists all of the methods that match that portion of the method name. note the quality and availability of the descriptive information for each method depends on the developer to register the method with complete and accurate information. the quality varies across the set of opensrf and evergreen apis, although some effort is being put towards improving the state of the internal documentation. srfsh# introspect opensrf.simple-text "opensrf.simple-text.reverse" --> opensrf.simple-text received data: { "__c":"opensrf.simple-text", "__p":{ "api_level":1, "stream":0, \ # <1> "object_hint":"opensrf_application_demo_simpletext", "remote":0, "package":"opensrf::application::demo::simpletext", \ # <2> "api_name":"opensrf.simple-text.reverse", \ # <3> "server_class":"opensrf.simple-text", "signature":{ \ # <4> "params":[ \ # <5> { "desc":"the string to reverse", "name":"text", "type":"string" } ], "desc":"returns the input string in reverse order\n", \ # <6> "return":{ \ # <7> "desc":"returns the input string in reverse order", "type":"string" } }, "method":"text_reverse", \ # <8> "argc":1 \ # <9> } } stream denotes whether the method supports streaming responses or not. package identifies which package or library implements the method. api_name identifies the name of the opensrf method. signature is a hash that describes the parameters for the method. params is an array of hashes describing each parameter in the method; each parameter has a description (desc), name (name), and type (type). desc is a string that describes the method itself. return is a hash that describes the return value for the method; it contains a description of the return value (desc) and the type of the returned value (type). method identifies the name of the function or method in the source implementation. argc is an integer describing the minimum number of arguments that must be passed to this method. calling opensrf methods from perl applications to call an opensrf method from perl, you must connect to the opensrf service, issue the request to the method, and then retrieve the results. #/usr/bin/perl use strict; use opensrf::appsession; use opensrf::system; opensrf::system->bootstrap_client(config_file => '/openils/conf/opensrf_core.xml'); # <1> my $session = opensrf::appsession->create("opensrf.simple-text"); # <2> print "substring: accepts a string and a number as input, returns a string\n"; my $result = $session->request("opensrf.simple-text.substring", "foobar", 3); # <3> my $request = $result->gather(); # <4> print "substring: $request\n\n"; print "split: accepts two strings as input, returns an array of strings\n"; $request = $session->request("opensrf.simple-text.split", "this is a test", " "); # <5> my $output = "split: ["; my $element; while ($element = $request->recv()) { # <6> $output .= $element->content . ", "; # <7> } $output =~ s/, $/]/; print $output . "\n\n"; print "statistics: accepts an array of strings as input, returns a hash\n"; my @many_strings = [ "first i think i'll have breakfast", "then i think that lunch would be nice", "and then seventy desserts to finish off the day" ]; $result = $session->request("opensrf.simple-text.statistics", @many_strings); # <8> $request = $result->gather(); # <9> print "length: " . $result->{'length'} . "\n"; print "word count: " . $result->{'word_count'} . "\n"; $session->disconnect(); # <10> the opensrf::system->bootstrap_client() method reads the opensrf configuration information from the indicated file and creates an xmpp client connection based on that information. the opensrf::appsession->create() method accepts one argument – the name of the opensrf service to which you want to want to make one or more requests – and returns an object prepared to use the client connection to make those requests. the opensrf::appsession->request() method accepts a minimum of one argument – the name of the opensrf method to which you want to make a request – followed by zero or more arguments to pass to the opensrf method as input values. this example passes a string and an integer to the opensrf.simple-text.substring method defined by the opensrf.simple-text opensrf service. the gather() method, called on the result object returned by the request() method, iterates over all of the possible results from the result object and returns a single variable. this request() call passes two strings to the opensrf.simple-text.split method defined by the opensrf.simple-text opensrf service and returns (via gather()) a reference to an array of results. the opensrf.simple-text.split() method is a streaming method that returns an array of results with one element per recv() call on the result object. we could use the gather() method to retrieve all of the results in a single array reference, but instead we simply iterate over the result variable until there are no more results to retrieve. while the gather() convenience method returns only the content of the complete set of results for a given request, the recv() method returns an opensrf result object with status, statuscode, and content fields as we saw in the http results example. this request() call passes an array to the opensrf.simple-text.statistics method defined by the opensrf.simple-text opensrf service. the result object returns a hash reference via gather(). the hash contains the length and word_count keys we defined in the method. the opensrf::appsession->disconnect() method closes the xmpp client connection and cleans up resources associated with the session. accepting and returning more interesting data types of course, the example of accepting a single string and returning a single string is not very interesting. in real life, our applications tend to pass around multiple arguments, including arrays and hashes. fortunately, opensrf makes that easy to deal with; in perl, for example, returning a reference to the data type does the right thing. in the following example of a method that returns a list, we accept two arguments of type string: the string to be split, and the delimiter that should be used to split the string. basic text splitting method sub text_split { my $self = shift; my $conn = shift; my $text = shift; my $delimiter = shift || ' '; my @split_text = split $delimiter, $text; return \@split_text; } __package__->register_method( method => 'text_split', api_name => 'opensrf.simple-text.split' ); we simply return a reference to the list, and opensrf does the rest of the work for us to convert the data into the language-independent format that is then returned to the caller. as a caller of a given method, you must rely on the documentation used to register to determine the data structures – if the developer has added the appropriate documentation. accepting and returning evergreen objects opensrf is agnostic about objects; its role is to pass json back and forth between opensrf clients and services, and it allows the specific clients and services to define their own semantics for the json structures. on top of that infrastructure, evergreen offers the fieldmapper: an object-relational mapper that provides a complete definition of all objects, their properties, their relationships to other objects, the permissions required to create, read, update, or delete objects of that type, and the database table or view on which they are based. the evergreen fieldmapper offers a great deal of convenience for working with complex system objects beyond the basic mapping of classes to database schemas. although the result is passed over the wire as a json object containing the indicated fields, fieldmapper-aware clients then turn those json objects into native objects with setter / getter methods for each field. all of this metadata about evergreen objects is defined in the fieldmapper configuration file (/openils/conf/fm_idl.xml), and access to these classes is provided by the open-ils.cstore, open-ils.pcrud, and open-ils.reporter-store opensrf services which parse the fieldmapper configuration file and dynamically register opensrf methods for creating, reading, updating, and deleting all of the defined classes. example fieldmapper class definition for “open user summary” <class id="mous" controller="open-ils.cstore open-ils.pcrud" oils_obj:fieldmapper="money::open_user_summary" oils_persist:tablename="money.open_usr_summary" reporter:label="open user summary"> <!-<1> --> <fields oils_persist:primary="usr" oils_persist:sequence=""> <!-<2> --> <field name="balance_owed" reporter:datatype="money" /> <!-<3> --> <field name="total_owed" reporter:datatype="money" /> <field name="total_paid" reporter:datatype="money" /> <field name="usr" reporter:datatype="link"/> </fields> <links> <link field="usr" reltype="has_a" key="id" map="" class="au"/> <!-<4> --> </links> <permacrud xmlns="http://open-ils.org/spec/opensrf/idl/permacrud/v1"> <!-<5> --> <actions> <retrieve permission="view_user"> <!-<6> --> <context link="usr" field="home_ou"/> <!-<7> --> </retrieve> </actions> </permacrud> </class> the <class> element defines the class: the id attribute defines the class hint that identifies the class both elsewhere in the fieldmapper configuration file, such as in the value of the field attribute of the <link> element, and in the json object itself when it is instantiated. for example, an “open user summary” json object would have the top level property of "__c":"mous". the controller attribute identifies the services that have direct access to this class. if open-ils.pcrud is not listed, for example, then there is no means to directly access members of this class through a public service. the oils_obj:fieldmapper attribute defines the name of the perl fieldmapper class that will be dynamically generated to provide setter and getter methods for instances of the class. the oils_persist:tablename attribute identifies the schema name and table name of the database table that stores the data that represents the instances of this class. in this case, the schema is money and the table is open_usr_summary. the reporter:label attribute defines a human-readable name for the class used in the reporting interface to identify the class. these names are defined in english in the fieldmapper configuration file; however, they are extracted so that they can be translated and served in the user’s language of choice. the <fields> element lists all of the fields that belong to the object. the oils_persist:primary attribute identifies the field that acts as the primary key for the object; in this case, the field with the name usr. the oils_persist:sequence attribute identifies the sequence object (if any) in this database and provides values for new instances of this class. in this case, the primary key is defined by a field that is linked to a different table, so no sequence is used to populate these instances. each <field> element defines a single field with the following attributes: the name attribute identifies the column name of the field in the underlying database table as well as providing a name for the setter / getter method that can be invoked in the json or native version of the object. the reporter:datatype attribute defines how the reporter should treat the contents of the field for the purposes of querying and display. the reporter:label attribute can be used to provide a human-readable name for each field; without it, the reporter falls back to the value of the nameattribute. the <links> element contains a set of zero or more <link> elements, each of which defines a relationship between the class being described and another class. the field attribute identifies the field named in this class that links to the external class. the reltype attribute identifies the kind of relationship between the classes; in the case of has_a, each value in the usr field is guaranteed to have a corresponding value in the external class. the key attribute identifies the name of the field in the external class to which this field links. the rarely-used map attribute identifies a second class to which the external class links; it enables this field to define a direct relationship to an external class with one degree of separation, to avoid having to retrieve all of the linked members of an intermediate class just to retrieve the instances from the actual desired target class. the class attribute identifies the external class to which this field links. the <permacrud> element defines the permissions that must have been granted to a user to operate on instances of this class. the <retrieve> element is one of four possible children of the <actions> element that define the permissions required for each action: create, retrieve, update, and delete. the permission attribute identifies the name of the permission that must have been granted to the user to perform the action. the contextfield attribute, if it exists, defines the field in this class that identifies the library within the system for which the user must have prvileges to work. if a user has been granted a given permission, but has not been granted privileges to work at a given library, they can not perform the action at that library. the rarely-used <context> element identifies a linked field (link attribute) in this class which links to an external class that holds the field (field attribute) that identifies the library within the system for which the user must have privileges to work. when you retrieve an instance of a class, you can ask for the result to flesh some or all of the linked fields of that class, so that the linked instances are returned embedded directly in your requested instance. in that same request you can ask for the fleshed instances to in turn have their linked fields fleshed. by bundling all of this into a single request and result sequence, you can avoid the network overhead of requiring the client to request the base object, then request each linked object in turn. you can also iterate over a collection of instances and set the automatically generated isdeleted, isupdated, or isnew properties to indicate that the given instance has been deleted, updated, or created respectively. evergreen can then act in batch mode over the collection to perform the requested actions on any of the instances that have been flagged for action. article continues, see part two at http://journal.code4lib.org/articles/3365. about the author dan scott is the systems librarian for laurentian university and a committer for the the evergreen and opensrf projects. you can reach him by email at dan@coffeecode.net or follow his occasional ramblings at http://coffeecode.net. subscribe to comments: for this article | for all articles one response to "easing gently into opensrf, part 1" please leave a response below: evergreen, software libre para informatizar bibliotecas | tramullas.com, 2011-03-29 […] web de la aplicación usando xulrunner. el servidor descansa sobre una implementación básica de opensrf, y, para seguir con las diferencias frente a otros sistemas, la base de datos no descansa sobre […] leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – editorial: journal updates and a call for editors mission editorial committee process and structure code4lib issue 55, 2023-1-20 editorial: journal updates and a call for editors journal updates, recent policies, and a call for editors. by junior tidal this is my second and last time as coordinating editor for code4lib journal. after serving on the editorial committee for 7 years, i am rotating off of the committee to focus on other research projects. code4lib has played a big part in my career. in 2012, i published my first article for the journal. after attending my first code4lib conference at north carolina state university in 2014, funded by a code4lib diversity scholarship, i really wanted to get more involved with this wonderful and supportive community. i was co-convener for the local new york city chapter of code4lib, presented at two national pre-conferences, and served on a couple of code4lib national conference committees since then. out of participating in various code4lib related activities, i have to say that working with the editorial committee (ec) has been the most rewarding experience. i have learned quite a deal from my fellow editorial committee members, and for that i am immensely grateful. this includes everything from copy editing, the article review process, communicating and collaborating with authors, and most especially, managing a journal. i would like to share two recent developments with the journal: a guest editorial policy and a retraction policy. the ec has implemented a guest editor policy. editorial members have a wide skill set reflective of library coders and technologists, however, some of the articles that we review are beyond our scope of expertise. in those situations, we feel it necessary to consult with experts outside of the ec. the guest editor policy is in place to make it clear to the author, guest editor, and readers, their role in the review process. a retraction policy has also been implemented. this retraction policy was developed so the ec could withdraw articles that may include work that violate ethical standards or may be unreliable. retractions are not to be taken lightly, and as such, the journal will inform readers why the article was retracted. this will be another part of the article’s lifecycle post-publication. since there is now an opening on the editorial committee of code4lib journal, please respond to this call for editors. if you are interested in reading and learning about library information technology, as well as being part of a great team of editors, then this is an excellent opportunity. applicants from diverse communities are highly encouraged to apply. i believe that every issue of code4lib journal has practical applications for almost any library, archive, museum, and other related spaces, and this issue is no exception. this issue includes: a fast and full-text search engine for educational lecture archives which outlines the development of a search engine for educational videos using python in india. click tracking with google tag manager for the primo discovery service explores how to track open access content through unpaywall links. creating a custom queueing system for a makerspace using web technologies is a case study on streamlining the queue process of a makerspace. data preparation for fairseq and machine-learning using a neural network details the use of sequence-to-sequence models and how it can be applied for a variety application with the appropriate formatting of datasets. designing digital discovery and access systems for archival description compares the differences between archival and bibliographic description and the challenges of utilizing discovery based systems for digital born materials. drying our library’s libguides-based webpage by introducing vue.js investigates how to better streamline redundant html code from the popular libguides web content management system. revamping metadata maker for ‘linked data editor’: thinking out loud looks at using and evaluating the catalog record creation tool using linked data sources. using python scripts to compare records from vendors with those from ils examines the use of python to identity and synchronize out-of-sync vendor and ils catalog records.   subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – scaling iiif image tiling in the cloud mission editorial committee process and structure code4lib issue 47, 2020-02-17 scaling iiif image tiling in the cloud the international archive of women in architecture, established at virginia tech in 1985, collects books, biographical information, and published materials from nearly 40 countries that are divided into around 450 collections. in order to provide public access to these collections, we built an application using the iiif apis to pre-generate image tiles and manifests which are statically served in the aws cloud. we established an automatic image processing pipeline using a suite of aws services to implement microservices in lambda and docker. by doing so, we reduced the processing time for terabytes of images from weeks to days. in this article, we describe our serverless architecture design and implementations, elaborate the technical solution on integrating multiple aws services with other techniques into the application, and describe our streamlined and scalable approach to handle extremely large image datasets. finally, we show the significantly improved performance compared to traditional processing architectures along with a cost evaluation. by yinlin chen, soumik ghosh, tingting jiang, james tuttle introduction for over 30 years, virginia tech university libraries’ special collections and archives has amassed a hidden treasure of architectural drawings, design sketches, personal and professional correspondence, project files, business records, photographs by more than 400 women who practiced architecture and design around the world from the 1800s through the present day. the collections, including many oversized materials, were only available on-premises until 2019. in 2016, virginia tech university libraries was awarded a digitizing hidden collections grant from the council on library resources (clir). the grant, women of design: revealing women’s hidden contributions to the built environment [1], partially funded the scanning, description, and provision of access to 800 cubic feet of materials from the iawa collections. through the generous support of clir, virginia tech university libraries’ special collections and archives digitized and described a significant part of the collection. the grant also supported collaboration between special collections and archives and the libraries’ information technology services to design and build a web application to make the work of 30 women architects freely accessible to the world through an online image repository: international archive of women in architecture (iawa) [2]. to enhance access to virginia tech’s cultural heritage collections, iawa supports the international image interoperability framework (iiif) [3], providing a world-class image viewing and sharing experience by exposing high-quality digital images along with rich metadata. iiif is a set of shared application programming interface (api) specifications built to support image interoperability across digital image repositories. enriched by a community of software tools and technology systems, iiif enables: a uniform and rich access to image-based resources hosted around the globe by repositories promoting the common technical specifications better, faster and cheaper image delivery with iiif compatible image servers and web clients advanced features for viewing and interacting with structured collections of images by iiif api-enabled image viewers. due to limitations in our on-premise infrastructure and rapidly increasing computing and storage needs during this project, we suspected a cloud solution might best address our needs. relatively early in the project, libraries’ network attached storage was nearing capacity at over 90 percent utilization. budget and time constraints conspired against expanding on-premise storage within the project timeline, which led us to use aws simple storage service (s3) [4], amazon’s cloud object storage, which was used for other purposes on campus. numerous efficiencies were easily realized after moving the data to the aws cloud. hosting the data in aws cloud facilitated leveraging other aws services. to improve performance and lower latency to and from our application in addition to reducing data transfer costs, we leverage amazon cloudfront [5], a content delivery network (cdn) service, to deliver static content. to perform a long-running and compute-intensive task, we use aws batch [6], a service that manages batch computing workloads. finally, to receive system event and response action notifications, process intermediate data, and trigger other aws services, we use aws lambda [7] to implement our customized microservices. we describe in detail our implemented aws solutions in the following sections. problem description iawa is a hyrax/savmvera application [8] supported by fedora [9]. specifically, it is based on hyrax 2.1.0 and fedora 4. initially, we took the native, out-of-the-box approach, and ingested high-resolution tiff images and stored them directly in fedora. we integrated an iiif-api enabled viewer – mirador [10] – with frontend hyrax to display our image collections. the viewer supports zoom, pan, and other features, as well as selection among related images. the underlying image tile and manifest generation are processed in real-time. upon receiving a user’s request, the application first retrieves all the collection image files from fedora, stores them into the server’s temporary location, subsequently triggers the iiif image service using the riiif gem [11] to start creating tiles and manifests. after the generation process is completed, the mirador viewer can then use the generated iiif url to display the images to the end-user. this is the program logic as it was implemented in hyrax [12]. however, this implementation is only suitable for a small amount of image data and the application performance drops significantly when both the size of image files and the number of images in the collection are large. in the iawa project, for example, a collection might contain 20 pages (image files) at 100mb each. under this circumstance, even with 32gb ram and 50gb storage space allocated, the image viewing process became unbearably slow, often causing the application to halt and crash. there was an alternative implementation of caching the manifests to avoid frequent fedora access; however, due to the large number of our image collections, the response time to newly requested images was still too long for a production service. moreover, this approach added the overhead of periodically running a rake task to refresh and rebuild the cache. thus, we sought an approach to display images from pre-generated image tiles and manifests instead of generating them on the fly in hopes of eliminating the need to make exhaustive search requests in fedora. our solution was to build an iiif level 0 compatible image server with all the pre-generated files in place. since all the tiles and manifests are ready to serve the requests, the website performance increased immediately. our challenge then became how to efficiently generate terabytes of iawa image collection metadata and tiles. we performed a local test that used a batch script to generate a set of image collection of around 20gb, which took about a week to finish. extrapolating out the time it would take to process the expected size of the entire digitization process, several months or possibly a year, it was clear that this was not a viable approach. finally, we designed and implemented our cloud-based aws solution and successfully generated all our iawa image collections in 3 days. we describe our architecture design, software implementation, deployment, and performance in the next sections. system architecture this system architecture was designed with several goals in mind. the system provides a single entrypoint to assign tasks. it can run multiple tasks in parallel and adjust the resource needs for each task automatically without human intervention. the system can monitor task status and produce detailed task and cost reports. it reduces or eliminates maintenance needs by using managed services in a serverless architecture. figure 1 shows an overview of our system architecture design. we use amazon s3 and aws batch management services and implement our business logic as microservices using aws lambda. after a user uploads a task file to a designated folder in s3, an aws lambda is triggered automatically and parses the content in the task file and then submits multiple jobs to aws batch. these batch jobs run in parallel and each job processes a set of image collections and uploads generated tiles and manifests into s3 buckets. when batch jobs are finished, aws batch disposes of resources that are no longer needed. given this design, resources are provisioned only when needed to process tasks. an added benefit to aws managed services is that there is no longer a need to perform system maintenance such as installing os updates, security patches, etc. figure 1. an overview of our aws cloud-based image tiling system architecture the iiif tile and manifest generation relies on several aws components. there is an s3 bucket for task files and another bucket for tile and manifest files. our image collections reside in another s3 bucket. an aws batch is instantiated by creating a job definition, compute environment, and a job queue. a job definition specifies the type of job to be run and the amount of resources to be allocated. for example, it may specify a docker image, the number of vcpus, amount of memory, what command to be executed, etc. a compute environment defines what kind of ec2 instance is to be provisioned and used to run the aws batch jobs. a job queue is mapped to one or more compute environments and a job is submitted to the job queue. once there are jobs in the job queue, aws batch will acquire the necessary resources, prepare the compute environment with amazon ec2 instances, and schedule these jobs to be run within the instances. then, our microservice implementation is deployed in the aws lambda. the entire system configuration with instructions is published in our project github page. [13] software implementation our software implementations include an aws lambda function, an iiif ruby gem implementation, and an executable script. we package the ruby gem and executable script into a docker image, which is the core business logic component to be run in the aws batch job. figure 2 illustrates the overarching workflow inside the aws batch job. figure 2. the overarching workflow inside our aws batch job the iiif ruby gem generates two iiif items. one is iiif image api compatible image tiles that can be requested and delivered using iiif specific uris, and the other is image metadata and the structure/layout of the image objects can be displayed through iiif presentation api compatible image manifests. with these two iiif items, we can support advanced viewing capabilities, including deep zoom for individual images, metadata information sharing as well as structured/ordered view of images in a collection. currently, there are several libraries capable of generating iiif level 0-compatible image tiles and manifests for the image files that can be statically served in local or cloud image server, such as iiif_s3 [14]. however, the implementation of an iiif image viewer, such as mirador, which we are using, and universal viewer require specific image metadata information in order to properly display the images with the above viewing capabilities. although iiif specifications support bare thumbnails as string objects, mirador only accepts thumbnails as image api service objects. thus, we extended the iiif_s3 work and implemented our own iiif ruby gem giiif [15] to fulfill this specific requirement. furthermore, we fixed a bug in iiif_s3 when using imagemagic to transform high-resolution image files to default (e.g., jpeg) format: if the original image is in the default format, the transformation step should be skipped in order to avoid a code break. the executable script is a set of bash commands to execute the workflow. it first fetches the images to be processed into a local disk inside the docker container. it then executes a ruby script [16] which gathers the required information for generating tiles and manifests, including metadata csv file for image collections, the path to the folder containing image files in the collections, and root path information for the generated image stack. finally, it uploads all the generated files (tiles and manifests) to the designated location in s3. to automate the iiif tiling process, we implemented an aws lambda function that processes task definition files uploaded to an s3 folder and submits aws batch jobs. when a task file is added to a specific folder in s3, this lambda function automatically triggered. the lambda function reads the content in the task file, including parameters such as job definition, job queue, command to execute, environment parameters list, etc and then submits an aws batch job respectively. figure 3 shows the python code snippet representing this procedure. figure 3. aws lambda function in python: receives task and submits aws batch job finally, we update our iawa hyrax application by storing an item’s iiif manifest url in a new metadata field called import_url. this iiif manifest url contains links to the pre-generated tiles for that item. when a user views the images in an item using the mirador viewer, the application does not query fedora for image data; it instead retrieves the information from s3. more specifically, the program now reads a static string instead of calling a complex and time-consuming function to retrieve the image metadata and create the tiles on the fly. with this change, users can view an image in milliseconds to seconds, compared to minutes in the original implementation. the performance improvement is approximately 1,000x faster. figure 4 is the code snippet shows the difference between our implementation (left) and hyrax implementation (right) figure 4. iawa (left) vs. hyrax (right) implementations for image manifest deployments and tasks executions as mentioned briefly in the system architecture section, aws batch organizes its work into four components: jobs – the unit of work submitted to aws batch. job definition – defines how jobs are to be run, memory and cpu requirements, docker container properties, environment variables, and mount points for persistent storage job queues – submitted jobs go into queues with priorities. queues can be associated with one or more compute environments. compute environment – the compute resources that are used to run jobs. when jobs are scheduled to run, they are placed into a single queue associated with a compute environment. at a single point in time, there can be multiple active queues trickling jobs into their respective compute environments. we can see all job statuses, number of jobs in each queue, and compute environment information through aws batch dashboard, which is shown in figure 5. figure 5. aws batch dashboard we can see more granular information, such as the logs from running docker containers through cloudwatch as shown in figure 6. cloudwatch logs are updated in real-time as well as retained after job completion. figure 6. cloudwatch log showing the output of a running job performance and cost there are 31 iawa image collections and sizes ranged from tens of gigabytes to hundreds. we selected four collections to conduct the performance and cost for this article. the smallest one we tagged for calculating costs was around 23 gb while the largest one was upwards of 230 gb. table 1 shows a breakdown of the execution time for these collections. table 1. execution time of the selected collections collection name size (gb) # of tasks time ms1995_007_piomelli 229.80 gb 62 12 hours ms2007_007_feuerstein 149.43 gb 44 5 hours ms2008_089_alexander 35.14 gb 19 4 hours ms2016_012_womens_development_corp 22.99 gb 16 3 hours the total cost of generating tiles for these collections was $36.54. the smallest collection cost $1.71 while the largest one cost $20.08. as we can see, this method of generating tiles by leveraging the concurrency of cloud services is fast and extremely cost efficient. table 2.cost to process selected collection collection name size (gb) cost ms1995_007_piomelli 229.80 gb $20.08 ms2007_007_feuerstein 149.43 gb $12.93 ms2008_089_alexander 35.14 gb $1.82 ms2016_012_womens_development_corp 22.99 gb $1.71 total 437.36 gb $36.54 iawa application the iiif tile generation system is deployed in aws and we currently use this system to process all iawa image collections. whenever a new iawa collection with raw image files along with the metadata csv files are ready to be processed, sysops need only create a task file for the new collection and upload that task file to s3 at which point the automated process will initiate. figure 7 shows one page of image files in an item (a group of image files). with our new design, no matter how large each image size or how many images that item contains, our application can display it with a fast and stable response time. figure 7. item page with multiple images conclusion in this article, we describe how we establish an auto-scaling system to generate iiif tiles using aws services. our proposed serverless architecture design eliminates the need to manage underlying servers and delegate all the heavy lifting, e.g. auto-scaling, instance provisioning, scheduling, resource deprovisioning, etc to aws. we focus on implementing microservices and meshing aws to accomplish our tasks. this system can process any amount of image collections automatically and efficiently. we reduce the time to process a collection from weeks to hours and simplify the human process from several manual operations to a simple file upload step. we also share our aws experience and implementations details about the entire system building and deployment process. finally, we publish our work in the authors’ institution github page [13]. future work this proposed system is deployed in the aws and becomes one of our cloud-based image processing services. while we manually deployed this system into aws currently, we start working on developing an aws cloudformation and/or terraform template in order to have the ability to deploy the entire system at one click of a button. furthermore, our proposed serverless architecture can be extended to perform other tasks by simply replacing the software in the docker image and changing the aws batch configuration files. for example, we can package jhove software [17] into a docker image and configure the aws batch to use this new docker image. a new service is then established and ready to perform format identification, validation, and characterization of digital objects. we will generalize this procedure and use this approach to create other library services. references [1] women of design [internet]. [updated 2019]. available from: http://registry.clir.org/projects/23081249 [2] international archive of women in architecture (iawa) [internet]. [updated 2019 august 1]. blacksburg (va): university libraries special collections and information technology department, virginia tech. available from: https://iawa.lib.vt.edu/ [3] international image interoperability framework (iiif) [internet]. [updated 2019 july 1]. available from: https://iiif.io/about/ [4] amazon simple storage service (amazon s3) [internet]. [updated 2019]. available from: https://aws.amazon.com/s3/ [5] amazon cloudfront [internet]. [updated 2019]. available from: https://aws.amazon.com/cloudfront/ [6] aws batch [internet]. [updated 2019]. available from: https://aws.amazon.com/batch/ [7] aws lambda [internet]. [updated 2019]. available from: https://aws.amazon.com/lambda/ [8] hyrax/samvera [internet]. [updated 2019]. available from: https://hyrax.samvera.org/ [9] fedora [internet]. [updated 2019]. available from: https://duraspace.org/fedora/ [10] mirador viewer: [internet]. [updated 2019 august 10]. available from: https://projectmirador.org/ [11] riiif ruby gem: [internet]. [updated 2018 april 30]. available from: https://github.com/curationexperts/riiif [12] hyrax github: [internet]. [updated 2018 march 30]. available from: https://github.com/samvera/hyrax/blob/v2.1.0/app/presenters/hyrax/displays_image.rb#l15 [13] aws batch iiif generator: [internet] [updated 2019]. available from: https://github.com/vtul/aws-batch-iiif-generator [14] iiif_s3: a generator for iiif level 0 compatible static server on amazon s3. [internet] [updated 2017 november 26]. available from: https://rubygems.org/gems/iiif_s3/versions/0.1.0 [15] giif: iiif compatible image stack and metadata generator. [internet] [updated 2019 august 12]. available from: https://github.com/vtul/giiif [16] a ruby script to gather image-related information, generate tiles and upload to amazon s3. [internet] [updated 2019 august 12]. available from: https://github.com/vtul/image-iiif-s3/blob/master/create_iiif_s3.rb [17] jhove: a file format identification, validation and characterisation tool. [internet] [updated 2019 april 18]. available from: https://jhove.openpreservation.org/ about the authors yinlin chen (ylchen@vt.edu) is a digital library architect at the virginia tech libraries, blacksburg, va. he holds a ph.d. in computer science and applications from virginia tech and a m.s. and b.s. in computer science at national tsing hua university, taiwan. his professional interests include digital libraries, cloud computing, microservices and serverless architecture. soumik ghosh (soumikgh@vt.edu) is a systems engineer at virginia tech libraries since 2017. he received his masters in computer science and applications from virginia tech in 2017 and his bachelors in computer science from west bengal university of technology. apart from containerisation and cloud native architectures, he likes to dabble in casual reading, writing, astronomy and microbiology. tingting jiang (virjtt03@vt.edu) has been a software engineer at virginia tech libraries since 2014. she received the b.s. (summa cum laude) and m.s. degrees in computer science from virginia tech, blacksburg, va. from 2007 to 2009, she was a software engineer with intrexon corporation, blacksburg, va. she was a recipient of an nsf graduate research fellowship (2011–2014) and a microsoft research graduate women’s scholarship (2011). james tuttle (james.tuttle@vt.edu) is associate director for digital libraries at the virginia tech libraries, blacksburg, va. he received the mslis and ba, anthropology from the university of illinois at urbana-champaign. he specializes in digital curation and preservation of complex materials, systems design, workflow analysis and efficiency design, and general systems planning with experience in higher education and industry. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – editorial edit mission editorial committee process and structure code4lib issue 42, 2018-11-08 editorial edit a few words about our editors. a farewell to one editor. a solicitation for new editors. this is the editorial, so let’s have a quick hurrah for the editors. we’re all volunteers, and i would hazard a guess that we’re all quite busy in our day jobs. work for the code4lib journal comes in pretty regular waves: there is a wave of proposals to consider and vote on; a wave of first drafts, second drafts, and all the edits in between; next entering the article into wordpress; and a final push to read as many completed articles as possible and give them a final vote of approval. the coordinating editor—for this issue, me—stands in the surf and throws stones at the editors, trying to distract them from their day jobs and make sure the process keeps flowing. why do we do it, then? i can’t speak for everyone, but there are perks: for instance, we get a first look at some really incredible ideas and projects. i must admit that more than once i have chosen to edit an article because the author was describing something my department was contemplating, and this seemed like an excellent way to jump in and learn more. we also get to work with some pretty interesting people and help them shape their ideas and text. sometimes our contributions are modest, sometimes they are significant. every article is different, every author is different. each issue is something new–jam packed with newness if we’re lucky. finally, we get to work with great people, our colleagues on the editorial committee. there are currently fourteen of us listed on the editorial committee page, which sounds like a lot, but when you have eleven articles as we do in issue 42, and each article needs an assigned editor and a “second reader,” we can get stretched thin. if you scroll down the editorial committee page, you will find a further twenty editors who have passed into emeritus status. that number is about to grow by one. carol bean is one of three remaining founding members of the journal. she has been here for all 42 issues, but has decided to step back, as she has moved on from libraries and put together an interesting mix of new pursuits: “data munging,” stocks and options, grandkids. in fact, carol tried to step back last issue, but when she saw that we could use another hand, she stepped forward again. the journal is part of the code4lib solar system and shares many of the same traits. it’s still very much diy, and when something needs doing there is the assumption that someone will make it happen (ideally without even being asked). carol embodied this trait—she took care of things that needed doing. she was generous in other ways. when fellow editors might be ready to throw in the towel on an article, she would always give the author one more chance to get it into a publishable state. she also wanted to make sure that public and special libraries were represented on the editorial committee. finally, she was a master of the sometimes enigmatic signature line, which was a tiny window into her life tucked beneath her name. a recent sampling: carol --who will now hunt down all the crumbs and demand someone pick them up --who thinks the maple tree outside is being overly melodramatic --who issues disclaimers when the wine is good --who is waiting for the other shoe to fall --who watches with curiosity as the squirrels try to hop through foot-deep snow call for editors you probably saw where this editorial was leading. we’re looking for more editors to step into carol’s big shoes. we need more carol: people who are engaged, smart, generous. you can find out more about this on our call for editors page. eric hanson and junior tidal from the editorial committee are leading this effort, and they ask that you submit your letter of interest by friday, december 7. issue 42 we have a great issue. take some time and read it all. consider helping us put together issue 43. andrew –who is ready for a nap subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – from finding aids to wiki pages: remixing archival metadata with ramp mission editorial committee process and structure code4lib issue 22, 2013-10-14 from finding aids to wiki pages: remixing archival metadata with ramp the remixing archival metadata project (ramp) is a lightweight web-based editing tool that is intended to let users do two things: (1) generate enhanced authority records for creators of archival collections and (2) publish the content of those records as wikipedia pages. the ramp editor can extract biographical and historical data from ead finding aids to create new authority records for persons, corporate bodies, and families associated with archival and special collections (using the eac-cpf format). it can then let users enhance those records with additional data from sources like viaf and worldcat identities. finally, it can transform those records into wiki markup so that users can edit them directly, merge them with any existing wikipedia pages, and publish them to wikipedia through its api. by timothy a. thompson, james little, david gonzález, andrew darby, and matt carruthers introduction in recent years, with the emergence of the glam-wiki initiative (galleries, libraries, archives, and museums with wikipedia) [1], cultural heritage institutions have begun to share their resources and expertise with the global audience of wikipedia. data are already beginning to show that glam-wiki collaboration is making information more accessible to researchers (szajewski 2013). one particular focus of glam-wiki collaboration has developed around authority control, an area that has been an important part of the traditional role of libraries. authority control is essentially the process of establishing and maintaining lists of official (“authorized”) and alternative names for different entities in order to uniquely identify them. in recent years, this process has been greatly enhanced by the pairing of name strings with uris. tools like oclc’s viafbot [2] have helped integrate library authority data into wikipedia (at web scale) by automatically adding over 250,000 reciprocal links from the virtual international authority file (viaf) [3]. as these initiatives progress, however, there is also a need for glam-wiki projects that balance robust data-sharing with the curatorial attention to detail that has been another traditional strength of the cultural heritage sector. whereas libraries have typically focused on maintaining name authority files, the archives community has produced a body of detailed biographical descriptions that help provide access to the broader archival context. in an effort to help archivists disseminate the information they have compiled about the people and organizations whose documents they preserve, a team of librarians and programmers at the university of miami libraries has developed the remixing archival metadata project (ramp). ramp is a lightweight web-based metadata tool that lets users generate enhanced archival authority records and gives them the ability to publish the content of those records as wikipedia pages. the ramp editor first extracts biographical and historical data from archival finding aids to create new authority records for persons, corporate bodies, and families associated with archival and special collections. it then lets users enhance those records with additional data from sources like viaf and worldcat identities [4]. finally, it transforms those records into wiki markup for direct publication to wikipedia through its api. background for decades, archivists have devoted considerable effort to describing the collections they curate and creating finding aids to help users navigate through their content. the encoded archival description (ead) standard was created to store and present this information online in machine-readable format. ead, in combination with the more product, less process (mplp) methodology (greene and meissner 2005), has enabled many more finding aids to be made accessible online, greatly increasing the availability of collections and information for researchers around the world. these archival collections, however, do not exist in a vacuum. understanding the circumstances surrounding the creation of a collection and its contents can help researchers gain important knowledge about the primary sources they are studying. the eac-cpf (encoded archival context–corporate bodies, persons, and families) metadata standard was designed to aggregate contextual information about collection creators and provide a unified access point for associated archival materials (wisser 2011). eac-cpf is a relatively new standard, endorsed by the society of american archivists in 2011. the eac-cpf user community is growing steadily, spurred on by projects such as social networks and archival context (snac), an initiative of the national archival authorities cooperative (naac) [5], and “connecting the dots: using eac-cpf to reunite samuel johnson and his circle,” a collaboration between yale university libraries and harvard university libraries [6]. <?xml version="1.0" encoding="utf-8"?> <eac-cpf xmlns="urn:isbn:1-931666-33-4" xmlns:ead="urn:isbn:1-931666-22-9" xmlns:xlink="http://www.w3.org/1999/xlink"> <control> <recordid>ramp-27-48-102</recordid> <otherrecordid localtype="uml">chc0339.48.r27</otherrecordid> <otherrecordid localtype="wci:dbpedia">http://dbpedia.org/resource/lydia_cabrera</otherrecordid> <otherrecordid localtype="wci:lccn">lccn-n80-98243</otherrecordid> <otherrecordid localtype="wci:viaf">79081464</otherrecordid> <maintenancestatus>derived</maintenancestatus> <publicationstatus>inprocess</publicationstatus> <maintenanceagency> <agencycode>fmu-n</agencycode> <otheragencycode localtype="oclc">fqc</otheragencycode> <agencyname>university of miami libraries</agencyname> </maintenanceagency> <languagedeclaration> <language languagecode="eng">english</language> <script scriptcode="latn">latin</script> </languagedeclaration> <maintenancehistory> <maintenanceevent> <eventtype>derived</eventtype> <eventdatetime standarddatetime="2013-10-11"/> <agenttype>machine</agenttype> <agent>xslt ead2eac.xsl/libxslt</agent> <eventdescription>new eac-cpf record derived from ead instance and existing eac-cpf record.</eventdescription> </maintenanceevent> </maintenancehistory> <sources> <source xlink:type="simple" xlink:href="http://proust.library.miami.edu/findingaids/index.php?p=collections/findingaid&amp;id=48"> <sourceentry>guide to the lydia cabrera papers</sourceentry> </source> <source xlink:href="http://worldcat.org/identities/lccn-n80-98243/identity.xml" xlink:type="simple"/> <source xlink:href="http://viaf.org/viaf/79081464" xlink:type="simple"/> </sources> </control> <cpfdescription> <identity> <entitytype>person</entitytype> <nameentry> <part>cabrera, lydia</part> <authorizedform>lcnaf</authorizedform> </nameentry> </identity> <description> <languagesused> <languageused> <language languagecode="spa">spanish;castilian</language> <script scriptcode="latn">latin</script> </languageused> </languagesused> <localdescription localtype="650"> <term xml:id="fst00821914" xmlns:xml="http://www.w3.org/xml/1998/namespace">authors, cuban</term> </localdescription> <bioghist> <abstract>the lydia cabrera papers contain the personal and research papers of 20th century cuban anthropologist, writer, and artist, lydia cabrera.</abstract> </bioghist> </description> <relations> <cpfrelation xlink:arcrole="associatedwith" xlink:href="http://id.loc.gov/authorities/names/n78095648" xlink:role="http://rdvocab.info/uri/schema/frbrentitiesrda/person" xlink:type="simple"> <relationentry localtype="editor">hiriart, rosario</relationentry> </cpfrelation> <resourcerelation resourcerelationtype="creatorof" xlink:href="http://proust.library.miami.edu/findingaids/index.php?p=collections/findingaid&amp;amp;id=48" xlink:role="archivalrecords" xlink:type="simple"> <relationentry>lydia cabrera papers, 1910-1991</relationentry> </resourcerelation> <resourcerelation resourcerelationtype="creatorof" xlink:href="http://merrick.library.miami.edu/cubanheritage/chc0339/" xlink:role="archivalrecords" xlink:type="simple"> <relationentry>lydia cabrera papers (digital collection)</relationentry> </resourcerelation> </relations> </cpfdescription> </eac-cpf> figure 1. sample xml illustrating the basic structure of an eac-cpf record. eac-cpf contains both administrative and descriptive information, intended to allow for the management of the record as well as the description of the entity the record is about. the administrative portion of the record is located within the element and is designed to capture information regarding the identification of the file, its status and history, and declarations of conventions and languages used in the description. the descriptive portion of eac-cpf is broken down into three main components: <identity>, <description>, and <relations>. the <identity> portion is concerned with uniquely identifying the entity being described. the portion deals largely with biographical or historical information about the entity. the real power of eac-cpf comes in its <relations> section, which provides the ability to associate the entity being described with other entities or resources. these links help create a web of contextual information that could potentially have a substantial benefit for researchers. currently, it can be very difficult to locate all archival resources related to an individual or entity, especially if collections are geographically dispersed or held by different institutions. there is not yet a central authority file or database for the creators of archival collections that could link out to all of these related resources. with eac-cpf and national initiatives like naac, the archival community is now beginning to encode this authoritative information. as with any new metadata standard, there are barriers to implementing eac-cpf within the archival community. full eac-cpf records can potentially contain hundreds or thousands of lines of description, making the manual creation of records very time consuming. often, research must be performed to collect enough information about the entity being described to create a full eac-cpf record. another issue arises with the dissemination of the information contained within the records. there are not currently many avenues for publishing this information or making it available to users, outside of prototype systems such as snac’s customized xtf platform. development process and technology ramp brought together curators, programmers, and cataloging and metadata librarians, each of whom had a unique perspective on the project and its goals. finding aid data was initially drawn from two repositories: the university of miami libraries’ special collections [7] and its cuban heritage collection [8]. curators from these collections were the project’s primary stakeholders. the development process itself was guided by the agile scrum methodology [9]. over the course of three two-week sprints, team members adopted the various scrum roles: the “product owner” (played by matt carruthers, metadata librarian) gathered user stories about the desired functionality and clarified these stories for the developers; the “scrum master” (played by andrew darby, head of web & emerging technologies) facilitated the process; and the three developers (played by david gonzález, digital programmer; jamie little, digital programmer; and tim thompson, metadata librarian) broke the user stories down into programming tasks and got to work. the development team began each sprint day with a 10–15 minute stand-up meeting to outline the previous day’s accomplishments and set goals for the coming day. at the end of each two-week sprint, colleagues and stakeholders were invited to see a demo of the functional (but incomplete) application. one of the goals of ramp was to make the software accessible to other institutions. with this in mind, the project utilizes the common lamp stack. on the back end, php is used to facilitate xslt transformations, xml validation, and database access. much of this functionality is exposed as lightweight services that can be accessed with javascript. client-side, jquery is used to communicate with those services. project workflow in the ramp workflow, data passes through three stages: first, ead-encoded archival finding aids are ingested into the system and biographical data is extracted to produce authority records for collection creators. this initial step utilizes two xml metadata formats: ead for finding aids and eac-cpf for authority records. second, users select individual records for editing. at this stage, the ramp tool can do basic named entity recognition and pull in data from external sources like viaf and worldcat identities. the user is presented with a list of potential matches and can select which ones to add to the eac-cpf record to create relation elements. third, a relevant subset of the enhanced eac record is transformed into wiki markup. the tool can then check to see if there is an existing wikipedia page for a given person or organization. if there is no entry, a stub page is created containing custom wikipedia metadata, the entity’s viaf id (if available), a biography, and sections for references, works by, and external links back to the local finding aid and any associated digital collections. if there is already an entry, that wiki markup is pulled in for potential integration and presented alongside the locally created markup. finally, ramp allows users to publish this new or updated entry directly to wikipedia with the push of a button. figure 2. process diagram of ramp workflow. (click to enlarge) data transformations ead to eac-cpf the availability of archival management systems like archon, archivists’ toolkit, and now archivesspace has facilitated the processing and description of archival and special collections. our current local system, archon, contains separate modules for creating both finding aids for archival collections and authority records for creators. archon also includes a batch export utility that outputs ead/xml records for finding aids and basic eac-cpf/xml records for creators. ramp uses xslt 1.0 stylesheets to transform ead records into eac-cpf records and then eac-cpf records into wikipedia markup. preliminary stylesheets created in the early stages of the project had been written in xslt 2.0, but the continued absence of any xslt 2.0 transformation engine that does not require java meant that these stylesheets needed to be rewritten. one of the benefits of using external xslt 1.0 stylesheets is that a large portion of the core functionality of ramp can be repurposed using other programming languages that support xslt, or through workflows based on desktop xml editors. an effort was made to create stylesheets that were as generalizable as possible. for example, several global parameters were created to allow users to specify local data in an external configurations file (conf/inst_info.php). when the transformation is called, the values for these parameters are passed into the stylesheet from php. notwithstanding, the potential for interoperability is somewhat limited by the idiosyncrasies of individual archival management systems like archon and the way they format ead and eac-cpf records on export. for example, although the ead metadata standard includes elements for encoding chronologies and references to related material, archon does not support this level of structured markup in its data entry interface. when finding aids did include chronologies or related reference sections, this data was often marked up in simple <p> or <emph> tags. markup limitations were compounded by inconsistencies in the way data had been entered over time. the ead-to-eac stylesheet tries to parse out as much relevant information as possible, but this code would most likely need to be adapted to work with markup from other institutions. the need to account for one-to-many relationships between creators and collections, and vice versa, also presents a major challenge for data transformation. at present, ramp’s file import routine generates a single eac-cpf record for each ead finding aid. in the future, this process will need to be improved to enable information from related finding aids and any existing creator records to be merged into a single eac-cpf instance. much of the work for merging multiple eads into eac-cpf records has already been accomplished by the snac project, and its code, soon to be publically released, may help resolve this issue. as a temporary workaround, a shell script and xslt 2.0 stylesheet were developed to merge our local ead and eac-cpf files as part of a separate preprocessing stage prior to import into ramp. eac-cpf to wiki markup by comparison, the transformation from eac-cpf to wikipedia markup (an intermixture of a unique plain text format and html tags) presented fewer challenges. wiki markup is meant to be an abstraction from html that allows for easier editing of wikipedia articles. at first glance, wikipedia’s markup has a simple syntax, but in practice wikipedia articles rely on a multitude of templates that can be quite complex. one of the goals of ramp was to include an infobox for each person or organization (at present, ramp’s eac-to-wikipedia stylesheet does not transform records for families). infoboxes are essentially metadata about an entity that is displayed as a sidebar on the public page, and they contain dozens of associated properties [10]. the stylesheet outputs a subset of infobox properties that a user can fill in manually, and it also attempts to extract basic information from the eac record to pre-populate certain infobox fields such as those for birth and death dates. whereas infoboxes are intended for display, wikipedia pages contain an additional metadata template called persondata, which includes a limited set of properties and is used only for machine processing [11]. ramp’s eac-to-wikipedia stylesheet employs the same methods used in creating infoboxes to output and attempt to pre-populate persondata templates. for other data, top-level sections are created according to the following basic crosswalk: eac-cpf tag name / xpath wiki markup section / template bioghist/p biography (persons) / history (organizations) chronlist/chronitem chronology localdescription/term[@localtype=’6xx’] category resourcerelation[@resourcerelationtype=’creatorof’ and @xlink:role=’resource’]/relationentry works or publications cpfrelation/relationentry see also sources/source[1]/sourceentry notes and references resourcerelation[@resourcerelationtype=’subjectof’ and @xlink:role=’resource’]/relationentry further reading resourcerelation[@xlink:role=’archivalrecords’]/relationentry external links sources/source/@xlink:href[contains(.,’viaf’)] otherrecordid[@localtype=’wci:lccn’] authority control figure 3. basic eac-cpf-to-wikipedia crosswalk. inline xml editor ramp utilizes the open source ace editor [12] for editing xml records. the cloud9 online ide [13], github [14], and the unc libraries [15] are currently using this editor. the editor is written in javascript and includes many features, such as syntax highlighting, that users of desktop xml editors would expect. in addition to the editor, ramp includes a validation service that checks to see whether the xml being edited is well formed and valid as the user edits the xml. if the xml is valid and well formed, a green icon is displayed. if the xml contains errors, the validation service returns specific error information. the xml records edited in ramp are not stored on the file system, but in a database. although ramp may seem like the ideal project to utilize an xml database, it uses mysql because of its ubiquity and xml capabilities. individual ead and eac-cpf records are stored in the database along with basic metadata about the file. mysql versions beginning with 5.1 allow queries to contain xpath statements, and this functionality is utilized by ramp to extract names for display. in future iterations of ramp, it may be possible to extract more information directly from xml stored in the database. diff functionality ramp incorporates php diff [16], a library that displays differences between text, and jquery merge for php diff [17], a javascript-based library that lets users perform a merge based on the differences generated by php diff. this functionality was included to cover situations where users may need to incorporate changes or updates made to local ead records. during ramp’s import process a user will be presented with a merging interface for files that have changed: figure 4. diff functionality in the ramp import interface. (click to enlarge) api lookups ramp currently uses the viaf [18] and worldcat identities [19] apis to ingest new information into the eac-cpf record. ramp’s api functionality was designed to be extensible, and other data sources, like geonames.org, may be incorporated in the future. ramp uses the php curl [20] library to connect and communicate with these apis, and xml dom [21] to manipulate all xml it encounters. data from viaf lets ramp create new <nameentry> (with authorized and alternative forms of names) and relation elements, and data from worldcat identities provides associated subject headings, works by and about an entity, and data about relations to other entities. with these data sources, ramp can ingest a considerable amount of information about an entity to enhance the eac-cpf record. most importantly, it provides an easy method for harvesting uris, which can be leveraged in future linked data applications. although the task of assigning viaf ids to wikipedia articles is already being ably handled by the viafbot, ramp provides a mechanism for ensuring that this data is stored and associated with records for locally held resources. ramp automates and centralizes the process of integrating data from a viaf or worldcat identities record. it first uses the apis’ search query to provide the user with a list of search results, with links to the record for each result. from this list, the user can choose the appropriate record to ingest. after the user chooses the desired result, ramp queries the api again to retrieve the actual record and then parses the response to collect the desired data (for example, the viaf id from a viaf record and associated subjects from worldcat identities). once this data has been collected, it needs to be converted into objects that can be easily mapped to dom elements in order to facilitate ingest into xml. the following snippet of data from worldcat identities illustrates how these objects are structured (json formatted) and represented in xml: { "cpfrelation": [ { "attributes": { "xlink:arcrole": "associatedwith", "xlink:href": "http://id.loc.gov/authorities/names/n78095648", "xlink:role": "http://rdvocab.info/uri/schema/frbrentitiesrda/person", "xlink:type": "simple" }, "elements": { "relationentry": { "attributes": { }, "elements": "hiriart, rosario" } } } ] } figure 5. worldcat identities sample object as json. <cpfrelation xlink:arcrole="associatedwith" xlink:href="http://id.loc.gov/authorities/names/n78095648" xlink:role="http://rdvocab.info/uri/schema/frbrentitiesrda/person" xlink:type="simple"> <relationentry>hiriart, rosario</relationentry> </cpfrelation> figure 6. worldcat identities sample object as xml. this data mapping allows for an easy transition from objects (either php or javascript) to dom elements. after conversion into objects occurs, ramp then prompts the user with a list of elements so that he or she can choose which ones should actually be inserted into the eac-cpf record. before generating <nameentry> elements using the viaf api, ramp also attempts to extract other named entities from the local file, prompting the user with a list of possible matches. currently, this extraction is performed by applying a regular expression to the text of selected fields in the eac-cpf record (<bioghist> and <resourcerelation>) and in the original ead record (<unittitle>), which is now stored in the database. matching strings are output and presented to the user for selection (see figure 7). figure 7. partial results of ramp named entity recognition for the lydia cabrera papers. (click to enlarge) once appropriate matches have been chosen, a subsequent lookup is triggered and, if there are potential matches in viaf, new results are presented for final selection (figure 8). to help users determine whether a result from viaf is an appropriate match, links to the corresponding viaf pages are provided, preceded by the original matching string. selected results are then inserted into the eac-cpf record as <cpfrelation> elements that store the value of the viaf id in an @xlink:href attribute. figure 8. results of viaf lookup of named entities in the lydia cabrera papers. (click to enlarge) this method does have its shortcomings: it retrieves a relatively high number of false hits and does not catch name variations that do not match the pattern, which is based on sequences of capital letters (so “lydia cabrera” would count, but “lydia cabrera” would not). it also requires users to verify the accuracy of the matches pulled in from viaf. future integration of a named entity recognition program or api would greatly improve this functionality. notwithstanding, the ability to extract relationships from unstructured text and match names to uris is already a significant step toward tying local resources into the emerging web of data (van hooland et al. 2013). in-tool wiki editing ramp’s wiki markup editing interface is shown below (figure 9): figure 9. ramp wiki editing interface. (click to enlarge) the text area on the left contains locally stored wiki markup, and the one on the right contains wiki markup that a user wants to post to wikipedia, including any content from an existing wikipedia page. the tool makes it easy to move content between text areas. every time the user selects text from either text area, ramp stores that text in memory; when the user wants to move the text to the other side, he or she need only actuate one of the arrows: “<” will move text to the focus of the left text area and “>” will do the opposite; the focus of each text area is stored by the browser, and ramp uses that to move text between areas. interacting with the wikipedia api the wikipedia api is both useful and versatile, allowing access to wiki features, data, and metadata. ramp uses this api to log in, search pages, get pages, and edit pages on wikipedia. to begin editing pages on wikipedia, ramp searches wikipedia for existing pages that are relevant to the entity being described. after searching, it displays a list of results, with links, so the user can choose the desired page to edit. one problem encountered while implementing this portion of ramp was that newly created pages are not indexed until the following day and do not appear in search results from wikipedia’s api until then. therefore, if a user creates a page for “lydia cabrera” and comes back to it later on the same day, the search results will not display the new page. a solution to this problem was to add, on top of wikipedia’s search results, an exact match result. in the previous example, if the user searches for “lydia cabrera,” ramp does a separate http request to wikipedia, constructing a wikipedia url with the search term as a page title. if the page exists, ramp places that result above the search results from wikipedia’s search api. once a page has been selected for editing, ramp queries wikipedia again to get the wiki markup of the requested page and then displays it in the wiki editing interface shown in figure 9. ramp also provides an option to create a new wiki page if none of the search results reflects the entity a user is writing about; in this case, an empty text area is displayed on the right. after the user enters or edits content in the right-hand text area, changes are then ready to be posted to wikipedia. first, however, the user is prompted to log into his or her personal wikipedia account from within the ramp editing interface. having users log into wikipedia with a personal account helps ensure compliance with wikipedia’s username policy, which prohibits institutional or shared accounts [22]. after a successful login, ramp stores a cookie file with the user’s wiki session information for later requests, and the user can now post a new page or changes to an existing page to wikipedia, along with comments. some wikipedia edits also require a captcha upon submission. if a captcha is needed, ramp prompts the editor with the captcha image and a text box in order to get the solved captcha. it then tries the edit request again with new parameters, including the captcha id and answer. finally, ramp facilitates the editing process by letting users save wikipedia-hosted drafts of their articles. for example, rather than editing and posting new content directly to http://en.wikipedia.org/wiki/lydia_cabrera, ramp users can check a “draft” box to save their work-in-progress to a subpage of their wikipedia user homepage (like http://en.wikipedia.org/wiki/user:timathom/lydia_cabrera). in the wikipedia community, saving draft articles to user subpages is standard practice. preliminary results it is still too early to point to concrete evidence, but our preliminary experience using ramp to create and edit wikipedia pages has been very positive. to date, only a few live pages have been created using the tool. minor edits or additions have also been made to a handful of existing pages. for each of us, ramp represented our first foray into the world of wikipedia editing. it has been especially interesting to note the speed with which wikipedia users begin contributing to and improving newly created pages. for example, one of our first test pages was for a composer named rodrigo prats llorens (see figure 10). as the edit history for the page reveals, contributors began to edit this new entry quite quickly [23]. in the span of five days, three different wikipedia users provided edits or enhancements, and the first edits came within six minutes of the page’s creation. these contributions suggest that users are viewing and adding to wikipedia pages at an impressive rate, and they reaffirm that wikipedia is a robust platform for information-sharing and collaboration. in future versions of ramp, we hope to explore automated methods for incorporating these “crowd-sourced” enhancements back into local records. figure 10. wikipedia revision history for the ramp-created page about the composer rodrigo prats llorens. (click to enlarge) lessons learned as with any collaborative project that involves multiple stakeholders, there were challenges that arose during the ramp development process. like chaucer’s pilgrims [24], each participant involved in the project represented a socio-professional type, as it were: curators care deeply about the information they have worked to structure and provide; programmers are focused on problem solving and achieving specific results; and cataloging and metadata librarians are concerned with data conversion and the integrity of formats and standards. these different frames of reference made constant communication a necessity. the agile scrum process provided a structured framework for facilitating communication on the development side, but it may also have made communication with stakeholders more difficult, since each sprint tended to create a kind of isolation zone around the project. our experience reminded us that the demands of development should be balanced with the need to stay engaged, early and often, with stakeholders and interested parties, who may run the risk of feeling left out of the project management process. this balance was particularly hard to achieve during the initial sprint, before we had a working product to demonstrate. the end-of-sprint demos, however, gave stakeholders an opportunity to comment on a concrete thing rather than an idea of a thing. as a result, we received concrete and practical feedback, which we worked to incorporate into subsequent sprints. there were also technical challenges during the development process. the original concept for manually editing eac-cpf files in ramp was to utilize a web form, mirroring the functionality of archival information management systems such as archon, archivist’s toolkit, and archivesspace. the advantage to this approach is that it limits the possibilities for user error, which may result in malformed xml or invalid eac-cpf documents. the developers quickly encountered a number of challenges when trying to build the web form, however. eac-cpf is designed to be a fairly flexible schema, allowing for things such as the description of multiple identities for a single entity (for example, barack obama the president and barack obama the law school lecturer). building a web form to accommodate this would have consumed time and resources that needed to be devoted to other core functionality. utilizing a web form for data entry in eac-cpf files also effectively limits the user to a single application profile (which is a problem alluded to in the discussion of archon’s xml export format, above). we decided instead to use the ace editor as the interface for editing eac-cpf files. it proved to be a lightweight but powerful solution, allowing users full control over the structure of their eac-cpf files. although working directly in the xml can potentially place a greater burden on users, the validation service will alert them to any errors in the encoding. conclusion now that the core functionality of the ramp editor is in place, we have begun to explore the possibilities for integrating it into local workflows. because of the nature of the tool, an important part of adoption will involve establishing local guidelines and best practices for wikipedia editing in general. one important issue to address before beginning to utilize the tool is that of copyright and permissions. wikipedia provides two basic options for “donating” text for republication on its pages [25]. the first option is to include a permissions statement on the webpage of the local source of information, licensing it under a creative commons attribution-sharealike 3.0 unported license (cc-by-sa) and gnu free documentation license (gfdl). as an alternative, content contributors can also send an email to wikipedia with a formal declaration of consent for each text to be republished [26]. although there is a clear analogy between the structured content of an eac-cpf record and that of a biographical wikipedia entry, it is important to recognize that two different sets of conventions and expectations are at play. whereas wikipedia places a premium on rigorous citation of sources, this is not a practice that archival metadata standards have traditionally emphasized. when contributing biographical pages to wikipedia through ramp, it may be necessary to add references or check the sources of the local description (especially if it involves a living person) in order to bring it into line with the expectations of the wikipedia community. as a ubiquitous, international platform, wikipedia can provide users with new access points to libraries, archives, and special collections; at the same time, publishing archival metadata on wikipedia can challenge librarians and archivists to evaluate the quality and accessibility of their own descriptive practice. moving forward, we hope that future development of the ramp editor will be driven by feedback from users. we have carried out a round of usability testing with colleagues who have volunteered to test the tool, and based on initial usability feedback, we have been iteratively enhancing the ramp user interface to make its editing workflow more intuitive. usability testing has drawn our attention to the challenge, generally speaking, of working with data across domains and platforms. working with each of the three data formats involved in ramp places a different burden on the user: there is a significant amount of background knowledge involved in reaching a certain comfort level with the data itself, whether it be ead, eac-cpf, or wiki markup. some of the functionality that did not make it into the initial stage of development includes workflow management features like user accounts, notifications, and approval queues for newly created content, as well as integration with additional apis. we also plan to explore ways to track usage statistics and measure the impact of ramp-created articles on access to local resources and materials. although we do not have the capacity to provide technical support for the tool, we are eager to explore possibilities for collaboration with other institutions and to find ways to plug ramp into the broader glam-wiki movement. ramp is licensed under an educational community license, version 2.0 (ecl-2.0), and its source code is available at https://github.com/umiamilibraries/ramp. ultimately, we hope that the ramp editor can serve as a gateway for more librarians and archivists to become active on wikipedia. the ramp platform could be used as a starting point for wikipedia edit-a-thons and could lend support to initiatives like the wikipedia library [27], an initiative focused on putting active wikipedia editors in touch with library resources, or the rewriting wikipedia project, which aims to bring increased diversity to wikipedia by creating content about “marginalized groups and their histories” [28]. in the end, this project should serve to remind us that archives and special collections are not only repositories of unique content, they are also repositories of information about unique people and organizations. the university of miami libraries, for example, are home to collections that reflect the history and culture of south florida, the caribbean, cuba, and the cuban diaspora. many of the people and organizations present in our collections, including women and minorities, have not yet been adequately represented in wikipedia. the overall goal of ramp is not simply to provide additional access points to local content—although we certainly see that as important. above all, its objective is to further the altruistic mission of wikipedia by enriching the global cultural context with information that for too long has been removed from the broader conversation. notes [1] http://en.wikipedia.org/wiki/wikipedia:glam [2] http://en.wikipedia.org/wiki/user:viafbot [3] http://www.oclc.org/research/news/2012/12-07a.html and http://viaf.org [4] http://worldcat.org/identities/ [5] http://socialarchive.iath.virginia.edu/prototype.html [6] https://osc.hul.harvard.edu/liblab/proj/connecting-dots-using-eac-cpf-reunite-samuel-johnson-and-his-circle [7] http://library.miami.edu/specialcollections/ [8] http://library.miami.edu/chc/ [9] http://en.wikipedia.org/wiki/scrum_(software_development) [10] https://en.wikipedia.org/wiki/template:infobox_person [11] https://en.wikipedia.org/wiki/wikipedia:persondata [12] http://ace.c9.io/#nav=about [13] https://c9.io/ [14] https://gist.github.com/ [15] https://github.com/unc-libraries/jquery.xmleditor [16] https://github.com/chrisboulton/php-diff [17] https://github.com/xiphe/jquery-merge-for-php-diff [18] http://www.oclc.org/developer/documentation/virtual-international-authority-file-viaf/using-api [19] http://oclc.org/developer/documentation/worldcat-identities/using-api [20] http://www.php.net/manual/en/intro.curl.php [21] http://www.w3schools.com/dom/ [22] http://en.wikipedia.org/wiki/wikipedia:username_policy [23] we eventually discovered that this page was a duplicate of another page, titled “rodrigo prats,” and we used ramp to create a redirect and merge the new page with the existing one: http://en.wikipedia.org/w/index.php?title=rodrigo_prats. for the revision history of the original page, see http://en.wikipedia.org/w/index.php?title=rodrigo_prats_llorens&action=history. [24] http://en.wikipedia.org/wiki/the_canterbury_tales [25] http://en.wikipedia.org/wiki/wikipedia:donating_copyrighted_materials [26] http://en.wikipedia.org/wiki/wikipedia:declaration_of_consent_for_all_enquiries [27] https://en.wikipedia.org/wiki/wikipedia:the_wikipedia_library [28] http://dhpoco.org/rewriting-wikipedia/ references greene ma, meissner d. 2005. more product, less process: revamping traditional archival processing. the american archivist, 68:208-263. available from http://archivists.metapress.com/content/c741823776k65863/fulltext.pdf szajewski m. 2013. using wikipedia to enhance the visibility of digitized archival assets. d-lib magazine [internet], 19(3):9-. available from http://www.dlib.org/dlib/march13/szajewski/03szajewski.html van hooland s, de wilde m, verborgh r, steiner t, and van de walle r. 2013. named-entity recognition: a gateway drug for cultural heritage collections to the linked data cloud? available from http://freeyourmetadata.org/publications/named-entity-recognition.pdf wisser km. 2011. describing entities and identities: the development and structure of encoded archival context–corporate bodies, persons, and families. journal of library metadata [internet], 11(3):166-75. available from http://dx.doi.org/10.1080/19386389.2011.629960 about the authors tim thompson (t.thompson5@miami.edu) is a metadata librarian at the university of miami libraries, where his specific responsibilities include metadata creation for collections in spanish. james little (j.little@miami.edu) is a digital programmer at the university of miami libraries. he holds an mslis from the university of illinois urbana-champaign. david gonzalez (d.gonzalez26@umiami.edu) is a digital programmer at the university of miami libraries. he holds a bs degree in computer science from the university of miami. andrew darby (agdarby@miami.edu) is the head of web & emerging technologies at the university of miami libraries. matt carruthers (m.carruthers@miami.edu) is a metadata librarian at the university of miami libraries, where he supports and enhances discovery of and access to the libraries’ digital content. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – openwemi: a minimally constrained vocabulary for work, expression, manifestation, and item mission editorial committee process and structure code4lib issue 60, 2025-04-14 openwemi: a minimally constrained vocabulary for work, expression, manifestation, and item the dublin core metadata initiative has published a minimally constrainted vocabulary for the concepts of work, expression, manifestation and item (wemi) that can support the use of these concepts in metadata describing any type of created resources. these concepts originally were defined for library catalog metadata and did not anticipate uses outside of that application. employment of the concepts in non-library applications is evidence that the concepts are useful for a wider variety of metadata users, once freed from the constraints necessitated for the library-specific use. by karen coyle, https://kcoyle.net introduction a previous code4lib journal article (coyle, 2022) and a book chapter before that (coyle, 2016) described some ways that non-library metadata creators have incorporated the concepts of work, expression, manifestation, and item (wemi), first described in functional requirements for bibliographic records (frbr). the article described examples of communities using the wemi concepts that they had discovered in frbr. however, these communities were using wemi for data unrelated to libraries with the result that some of their uses did not adhere to the model expounded in the frbr report. the code4lib article proposed that a version of wemi could be developed that was more appropriate to non-library data by abstracting the base concepts from the library-specific design. a vocabulary defined using the resource description framework schema (rdf/s) in the dublin core metadata initiative (dcmi) namespace is now available that implements this simple idea. part i: openwemi vocabulary key documents specification html vocabulary display vocabulary in turtle (normative) the openwemi cookbook an ongoing gathering of use cases and examples. additional documents, including examples, are at the openwemi github site. the motivation the library community produced a multi-layered conceptual model of its catalog metadata in 1998, in a report titled functional requirements for bibliographic records (frbr) (ifla 2009). this was later developed into the library reference model (lrm) (riva 2017). the frbr report introduced a view of library catalog resources with a four-part structure at its base: work, expression, manifestation, and item (wemi), and these were incorporated into the lrm. although the frbr and lrm models were defined only in relation to library catalog data, communities unrelated to libraries found utility in the wemi concepts for the description of a variety of created resources. (see the openwemi bibliography for examples.) in many cases these reuses varied significantly from the library application model, and in ways that, at least in the rdf world, could result in inconsistencies in the semantics of the vocabulary terms. openwemi frees the wemi concepts from the original library application to allow these to be adapted by other metadata communities where their use would contradict the semantics of the library-defined model. the changes to the wemi model in openwemi are subtle yet are designed to overcome the restrictions included in the library model. openwemi can be used by communities adjacent to libraries, such as non-library archives, but can also be expanded to anyone using metadata to describe created resources in any domain: architecture, manufacturing, entertainment, and more, including less formal data practices. the philosophy it is to be expected that those creating metadata in the course of their work will be thinking only of their own application needs. it is less common that models are created with no specific application in mind, or with a goal of serving a wide variety of needs.(gruber, 1993) a prime example of the latter is the dublin core metadata terms (dcmt). these minimally constrained terms have gained traction on the web and elsewhere precisely because they have few constraints on their definitions. each term in dcmt describes an aspect of a resource, any resource, and aside from term definitions and some suggestions for value types, the rest is unconstrained. similarly, openwemi is also not limited to a particular application, but where dcmt is primarily made up of terms to be used in descriptive metadata, openwemi consists mainly of concepts — in the form of rdf classes — that can provide the scaffolding for metadata and metadata vocabularies. a goal of openwemi is to offer a possible way for metadata developers to think about their data with the least possible constraint on their modeling choice. the openwemi rdf vocabulary the openwemi vocabulary is defined in rdf/s with use of web ontology language (owl) functions when a union of classes is needed. classes the openwemi classes are: openwemi:endeavor: “a creation” openwemi:work: “an abstract notion of an endeavor.” openwemi:expression: “a perceivable form of an endeavor.” openwemi:manifestation: “a realization of an endeavor in physical, digital, or experiential form.” openwemi:item: “an instantiation of an endeavor.” the class openwemi:endeavor originated in the frbr-based vocabulary created by ian davis, richard newman, leigh dodds, and bruce d’arcus.(davis, 2005) this vocabulary was not sanctioned by ifla, yet it resulted as the only rdf-defined vocabulary for frbr and thus was included in projects wishing to use frbr in their rdf instance. figure 1. openwemi class diagram. openwemi includes openwemi:endeavor as an umbrella class to the wemi classes. each of the wemi classes is subclassed to openwemi:endeavor, and each class is defined in its relation to openwemi:endeavor. openwemi:endeavor can allow reference and searching in situations where the precise usage of the openwemi classes is unknown. it also will support the addition of other classes at the level of the openwemi “stack” if needed. like other high-level classes such as owl:thing, openwemi:endeavor is in the background of openwemi and is not likely to be used explicitly in metadata, although there is no prohibition against this. properties the properties of openwemi define only the essential relationships between the classes. there are no properties for descriptive metadata. that latter is left entirely to the metadata efforts that make use of the openwemi vocabulary. the properties are designed to reflect the general semantics of wemi, maintaining the logical direction from the most concrete (item) to the most abstract (work). within that one constraint, however, the ranges and domains of the properties allow the properties to be used with any classes of a more abstract definition. thus, openwemi:instantiates has a domain of openwemi:item and a range of openwemi:manifestation, openwemi:expression, or openwemi:work, or any combination of those. the primary properties of the model are: openwemi:expresses (domain: expression | range: work) openwemi:expressedby (domain: work | range: expression) openwemi:manifests (domain: manifestation | range: work or expression) openwemi:manifestedby (domain: work or expression | range: manifestation) openwemi:instantiates (domain: item | range: work or expression or manifestation) openwemi:instantiatedby (domain: work or expression or manifestation | range: item) note that each property has two options, one that takes a bottom-up view (openwemi:manifests, openwemi:instantiates) and a parallel property that takes a top-down approach (openwemi:manifestedby, openwemi:instantiatedby). figure 2. openwemi class relationships. “related” properties another set of properties supports statements of relationships between like classes: openwemi:relatedwork (domain:work | range: work) openwemi:relatedexpression</code (domain:expressio>n | range: expression) openwemi:relatedmanifestation (domain:manifestation | range: manifestation) openwemi:relateditem (domain:item | range: item) a property such as openwemi:relatedexpression could be the superclass to a property expressing the relationship between an original text and a translation, both of which are defined as expressions in this : mymd:translationof rdfs:subpropertyof openwemi:relatedexpression . after which all uses of mymd:translationof will be inferred to be between two openwemi:expressions. “common” properties most communities work with data that are not defined in terms of the wemi concepts. data managers may still wish to make use of some of the wemi concepts to express relationships between two resources that may or may not be modeled as wemi. with these properties, originally defined by ross singer of talis at open.vocab.org, it is possible to state that any two entities share a common work, common expression, common manifestation and/or common item. for example, one can make a statement that two metadata instances, one in a library catalog in marc format and one in amazon in their proprietary format, do indeed represent the same work, the same expression, and/or the same manifestation. these properties differ from the openwemi “related” in that there is no definition of domain or range. while not interrelated with the openwemi-defined properties we included these as in the spirit of openwemi and of rdf, allowing anyone to make connections between metadata instances with these useful concepts. openwemi:commonendeavor openwemi:commonwork openwemi:commonexpression openwemi:commonmanifestation openwemi:commonitem how openwemi might be used you can use openwemi directly without creating subclasses or properties, but keep in mind that openwemi as defined has very loose semantics. if that fits your use case, then by all means use the defined openwemi vocabulary. if you wish to “make it your own” then you will want to create a resource-specific version that is more expressive of your metadata by subclassing your model components to openwemi. this provides a wemi-based vocabulary that is compatible with your semantics. for example, if you are creating metadata for music materials (prefixed “mu:” here), you can create music-defined subclasses to openwemi classes: mu:musicwork rdfs:subclassof openwemi:work . mu:musicexpression rdfs:subclassof openwemi:expression. etc. and you would give your classes definitions that make sense in your data environment: mu:musicwork “a distinct musical creation” mu:musicexpression “the work as expression in musical notation” mu:musicmanifestation “a produced or published realization of the music.” mu:musicitem – “a single exemplar of the realized music.” additional levels of subclasses can be used to describe types of resources in your collection, such as: class subclass mu:musicwork mu:symphony mu:contata mu:serenade mu:musicexpression mu:score mu:libretto mu:performedmusic mu:musicmanifestation mu:cd mu:streaming</code mu:musicitem mu:archivalcopy mu:lendingcopy because openwemi has few constraints of its own, additional constraints can be added to the locally subclassed elements. for example, the classes subclassed to openwemi classes can be defined as disjoint, and domains and ranges can be defined as more limited than in openwemi proper. mu:item rdfs:subclassof openwemi:item owl:disjointwith mu:musicmanifestation ; owl:disjointwith mu:musicexpression ; owl:disjointwith mu:musicwork . mu:item rdfs:range mu:musicmanifestation . if you already have a vocabulary that you are using but you wish to include openwemi concepts in your instance data you can use the the openwemi class directly in the instance data you are creating. using the music classes above and dublin core terms, in rdf this could look like: ml:z1234 a mu:serenade ; dct:creator "dvořák, antonín" ; dct:title "serenade in e major" . ml:z345 a mu:performedmusic ; openwemi:expresses ml:z1234 ; dct:type "serenade" . ml:z897 a mu:cd ; openwemi:manifests ml:z345 ; dct:title "serenade in e major for string orchestra" ; dct:description "program notes by yoshiyuki fujita in english, french and german translation" ; dct:identifier "co-78919 denon" . ml:abcd a mu:lendingcopy ; openwemi:instantiates ml:z897 ; dct:identifier "cd 12757" . this example uses the language of wemi (mu:musicwork) but there are no restrictions on the names of classes or properties derived from openwemi. part ii: openwemi vs frbr/lrm openwemi is purposely not the same as the wemi concepts in frbr and the lrm. this section describes the primary differences. openwemi is not specific to any application or community frbr and lrm are explicitly in support of the library catalog application. as the lrm document states: “ifla lrm, as its name indicates, remains a model issuing from the library community for library data.” (riva 2017 p. 12) both base their analysis on a set of user tasks that mirror the actions of library users who approach the catalog for the purpose of finding library materials to use: find, identify, select, obtain, explore. openwemi aspires to be useful in the description of anything that can be considered “created.” (while “created” can be contrasted to “natural” it is not forbidden for anyone to make use of openwemi to describe something in the natural realm.) our goal in developing openwemi was to create a vocabulary that makes use of the general concepts in frbr and lrm’s wemi but to remove any aspects that would favor any one application. this meant broadening the definitions of the wemi classes beyond those in the library models and eliminating any elements of the model that would define the classes and properties as being limited to a specific application or community of practice. openwemi is an rdf vocabulary ifla has designated both frbr and lrm as “conceptual models.” both are defined using terms from an entity-relation model. no coded vocabulary was provided for frbr; lrm has an rdf vocabulary defined in the ifla standards namespace which, however, is not mentioned in the lrm documentation and does not appear to be in use. openwemi is first and foremost an rdf vocabulary. frbr and lrm can in theory be implemented in any data model, while openwemi serves best those that support the rdf framework, such as json linked data (json-ld). openwemi does not enforce interoperability… but neither do frbr/lrm openwemi classes and properties are semantically broad. any metadata models based on openwemi are not guaranteed to be interoperable. that metadata functionality must be instilled at a more detailed level than openwemi and must be built into community agreement on the metadata model being used. the lrm model itself is at least one level more abstract than the cataloging rules that libraries follow and is not itself designed for interoperability. the lrm documentation explains that “[a] wide range of decisions made in cataloging rules can be accommodated by the model” (riva 9999, p.9) and that neither frbr nor lrm include decisions regarding the nature of works, expressions, etc. those decisions are defined (to the extent that they are) in the cataloging rules. different communities could implement these differently in their metadata practices. in theory there could be a multitude of different bibliographic metadata practices based on frbr or lrm; in practice, the library community adheres to one metadata model through the cataloging rules which are specifically designed to enforce interoperability of library metadata. openwemi does not include descriptive properties both frbr and lrm are vocabularies for bibliographic description. although having many fewer data elements than the bibliographic vocabularies like the machine readable cataloging format (marc) and the more recent bibliographic framework (bibframe), the properties (called “attributes” in the lrm entity-relation model) in these vocabularies are specific to the library catalog application. openwemi is a kind of empty vessel. it does not specify any properties beyond those that are necessary to create fundamental semantic relationships between the openwemi classes. this means that there are no assumptions about the context within which openwemi will be used, or what type of resource can be described. openwemi classes are not disjoint the library model defined in frbr and lrm treats the entities of wemi as disjoint. this means that the attributes defined for each entity are distinct. this creates a solid model but it also limits the possibility of variants and of interoperating with variant models such as with the bibframe work, which combines the wemi concepts of work and expression in a single class.(baker, 2014) a consequence of defining the classes as disjoint is that no attribute can be can be defined for use with more than one class. it is for this reason that lrm needs three different data elements for “language”: lrm-e3-a6 language (expression) lrm-e6-a3 language (agent) lrm-e9-a7 language (nomen) and six for “category.” rda, which follows the lrm, must have separate elements for each entity, resulting in properties such as “has title of expression”, “has title of manifestation”. this results in a large vocabulary with little reuse of data elements, which may not be suitable for some communities. openwemi does not define the wemi classes as disjoint which is reflected in the openwemi property definitions. this particular openness means that classes may be defined as disjoint in vocabularies whose classes are subordinate to openwemi. the general rule for subclassing is that you can further constrain the class definition but you cannot contradict the constraints that are there. if openwemi classes were disjoint there would be no possibility of using them as non-disjoint in consequent metadata schemas. openwemi defines multiple relationships between its classes the lrm wemi has a single relationship between each class that defines a linear model from work to expression to manifestation to item. this requires that all entities exist in the metadata model based on frbr and lrm, and only certain patterns of relationships are valid in the instance data. for example, an lrm work can only be “realized” by an lrm expression, but not an lrm manifestation. figure 3. lrm class relationships. openwemi defines multiple relationships between its classes that however retain the semantics of abstract-to-specific that is fundamental to openwemi. it does not limit those relationships to a linear connection between classes. each openwemi class can have properties that link directly or from any of the other classes as long as the general semantics of “abstract-to-concrete” are maintained. in openwemi an openwemi:item can instantiate an openwemi:manifestation or an openwemi:expression or an openwemi:work. an openwemi:manifestation can manifest an openwemi:expression or an openwemi:work. there are two important reasons why openwemi defines multiple possible relationships between classes. the open world assumption in rdf allows for the case in which not all data is present at a particular moment in time. it is easy to imagine various situations in which a “full” description does not (yet) exist, or the information is not available when the data is first created. the other reason is that some communities may determine that their metadata model does not require all of the wemi classes. (others, of course, may find they need to add more classes and relationships to meet their requirements.) if a metadata model does not make use of, say, the expression class, creating a local subproperty to openwemi:manifests with a domain of openwemi:manifestation and a range of openwemi:work creates no inconsistencies with the openwemi vocabulary definition. properties that are subproperties of openwemi properties can be defined such that their ranges are any or all of the classes defined as ranges in the openwemi vocabulary. because the ranges in openwemi are a union of the openwemi classes, any or all of those classes may be used without contradicting the openwemi vocabulary definition. figure 4. openwemi class relationships. openwemi:endeavor is not lrm-e1 res the lrm defines a high-level class, res, which is defined as “everything considered relevant to the bibliographic universe, which is the universe of discourse in this case…. “(riva, 2017, p.19 ) openwemi has also defined a class, endeavor, that is a superclass to the four openwemi classes and that provides a unifying class relationship for those four classes. the primary difference between lrm:res and openwemi:endeavor is that openwemi:endeavor is open-ended, where lrm:res is in the context of the bibliographic universe that the lrm is defining. openwemi is only wemi the original frbr model was made up of three different “groups”: group 1 had entities work, expression, manifestation, and item and their attributes group 2 had entities person and corporate body and their attributes group 3 had entities concept, object, event, place, which are to be used only as subjects of a work. the lrm combined the conceptual models that defined groups 1 and 2. it also added concepts for time and place. openwemi is only wemi. other elements like agents, subjects, etc., are left to the implementations based on openwemi that communities choose to develop. as stated above, this makes openwemi non-aligned with any particular application or community use case, providing an unrestricted, but supported, structure for describing creative endeavors. conclusion there is a need in the area of metadata development for general vocabularies that adhere to the maxim that “less is more” and that facilitate the creation of application-specific models. as gruber states, sharing is best facilitated by the creation of vocabularies that follow the philosophy of “minimum ontological commitment.” “an ontology should make as few claims as possible about the world being modeled allowing the parties committed to the ontology freedom to specialize and instantiate the ontology as needed.” (gruber 1993, p910) with openwemi we have aimed to provide a model that is flexible and extensible. acknowledgments the dublin core metadata initiative provided the technical framework for the development of openwemi and is committed to maintaining the vocabulary in the namespace “https://ns.dublincore.org/openwemi/“. the following members of the openwemi community group were key to the development of this vocabulary: phil barker, cetis llp sean petiya, adjunct professor, kent state university ross singer, talis graeme williams and thank you to all who commented on our work throughout the months of development. about the author karen has decades of experience with metadata use and development. she has published articles and books on the topic, and has participated in a number of standards efforts with libraries, niso, w3c, and dublin core. she has been a speaker at many events and conferences. her writings and presentations are available on her web site: https://kcoyle.net email: kcoyle@kcoyle.net references baker, t., coyle, k. and petiya, s. (2014), “multi-entity models of resource description in the semantic web: a comparison of frbr, rda and bibframe”, library hi tech, vol. 32 no. 4, pp. 562-582. https://doi.org/10.1108/lht-08-2014-0081 coyle, karen. (2022) works, expressions, manifestations, items: an ontology. code4lib journal, issue 53. https://journal.code4lib.org/articles/16491 coyle, karen. (2016) “chapter 10: bibliographic description and the semantic web.” in:frbr, before and after: a look at our bibliographic models. chicago : ala editions, an imprint of the american library association. davis i, newman, r. (2005) expression of core frbr concepts in rdf. http://vocab.org/frbr/core.html. gruber thomas r. (1993). toward principles for the design of ontologies used for knowledge sharing. in: international journal human-computer studies 43. p.907-928. ifla study group on the functional requirements for bibliographic records. (2009). functional requirements for bibliographic records. den haag. http://archive.ifla.org/vii/s13/frbr/frbr_2008.pdf. library of congress. bibframe 2 ontology. version 2.4.0. https://id.loc.gov/ontologies/bibframe-2-4-0.html riva p, le boeuf p, žumer m. (2017). ifla library reference model: a conceptual model for bibliographic information. den haag, ifla. https://repository.ifla.org/handle/20.500.14598/40 subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – are we still working on this? a meta-retrospective of a digital repository migration in the form of a classic greek tragedy (in extreme violation of aristotelian unity of time) mission editorial committee process and structure code4lib issue 41, 2018-08-09 are we still working on this? a meta-retrospective of a digital repository migration in the form of a classic greek tragedy (in extreme violation of aristotelian unity of time) in this paper we present a retrospective of a 2.5 year project to migrate a major digital repository system from one open source software platform to another. after more than a decade on dspace, oregon state university’s institutional repository was in dire need of a variety of new functionalities. for reasons described in the paper, we deemed it appropriate to migrate our repository to a samvera platform. the project faced many of the challenges one would expect (slipping deadlines, messy metadata) and many that one might hope never to experience (exceptional amounts of turnover and uncertainty in personnel, software, and community). we talk through our experiences working through the three major phases of this project, using the structure of the greek tragedy as a way to reflect (with stasimon) on these three phases (episode). we then conclude the paper with the exodus, wherein we speak at a high level of the lessons learned in the project including patience, process, and perseverance, and why these are key to technical projects broadly. we hope our migration story will be helpful to developers and repository managers as a map of development hurdles and an aspiration of success. by steve van tuyl, josh gum, margaret mellinger, gregorio luis ramirez, brandon straley, ryan wick, hui zhang prologue project overview scholarsarchive@osu is oregon state university’s institutional repository. started in 2004 as an open repository for faculty articles and other scholarly content, scholarsarchive@osu quickly became integrated into multiple workflows for production of scholarly content across the university, including the deposit and distribution of graduate theses and dissertations, undergraduate theses, and historical publications from osu’s extension and experiment station communications office. more recently, oregon state university libraries and press (osulp), the organization managing scholarsarchive@osu, has been tasked with implementing the osu faculty senate’s open access policy. osulp has also started accepting research datasets in scholarsarchive@osu in order to help researchers comply with journal and funding agency mandates for data sharing. as of this writing, scholarsarchive@osu holds approximately 60,000 scholarly works across a variety of resource types and scholarly domains. since its inception, scholarsarchive@osu has been built on the dspace repository platform. dspace is an open source repository solution that was developed over 15 years ago to meet the needs of institutions seeking long term preservation and access of digital works. dspace now boasts thousands of adopters, primarily in academic and government user spaces. in 2015, after over a decade on dspace, osulp administration tasked scholarsarchive@osu managers with evaluating the repository (at that time on dspace 3.x), determining whether an upgrade or migration was necessary, and if so, to recommend a target platform to which to upgrade/migrate. scholarsarchive@osu managers evaluated the existing dspace repository, collected requirements for the repository from stakeholder groups, and evaluated major repository platforms for their ability to fulfill our requirements (van tuyl et al. 2015). as a result of this evaluation, we determined that migrating our content to a samvera repository solution was in the best interest of our organization in part because it was clear that major changes were needed to meet our use cases for the repository and that, as an organization heavily invested in samvera through another project (oregon digital), it would make sense to unify our resources around samvera repository solutions. why a tragedy? at its most basic, the greek tragedy carries a structure that is familiar to software development teams who have used the retrospective as a way to reflect on a development effort. in the case of an entire project, though, there are often cycles of work or phases of a project that could be evaluated. these phases are similar, in many ways, to the episode of the greek tragedy – chapters of an overall story. similarly, these phases of the project each have their own lessons from which we can learn, similar to the stasimon, or commentary in the tragedy on the events of each episode. in essence, the greek tragedy offers us a structure to create a retrospective of an entire project, including all the periods of activity in the project, and the stasimon and exodus give us opportunity to reflect on lessons learned at the periodic and project-wide time scales. this tragedy for this paper, we will not focus, overly, on the technical details of the project or software tools created and used. this is really a social story, rather than a technological one, and we feel that too many technical details will get in the way of the social, organizational, and community aspects of our project. this will also help those reading this, but working in other contexts, impute our experiences onto their own – something we feel techno-jargon would hinder. we hope our migration story will be helpful to developers and repository managers as a map of development hurdles and an aspiration of success parados the resourcing (in terms of personnel) varied a great deal across this project. that said, when sharing technical project experiences with others, one of the first questions that arises is “but how many people did you have working on the project?” at peak staffing, we had five software developers, a product manager, a product owner, a metadata librarian, and 2-3 additional staff and librarians contributing to the project. on the other hand, there were times on this project when we were working with 2-3 software developers working part time on the project along with one person acting as project manager and product owner. the challenges of this variability in personnel resourcing will become apparent as we examine the timeline and major events of the project. beyond personnel, the project had full support from osulp administration to use time and resources as needed to complete the project. this speaks to a general atmosphere of support for technology projects at our organization and recognition that these projects are part and parcel of our core library services, and deserve this level of support. in addition to personnel on our project in our organization, it is important to note that personnel from the samvera community at large played an important role on this project. as the reader will see, one of the important outcomes of this project is the value that we found, and believe to be universal, of meaningfully engaging with the open source community we are part of. it is challenging, if not impossible, to usefully quantify the return on investment of involvement in a community like samvera, but the time spent by other community members providing technical support (solicited and unsolicited), conceptual support, and emotional support have provided us with benefits beyond what one could expect from contracts or subscriptions. we will save further discussion of the value of participating in open source communities for another place and another time. episode 1 – optimism the aforementioned evaluation and reporting on the future of scholarsarchive@osu took place from fall 2014 through spring 2015, with the report published in february 2015. the publishing and administrative approval of the report, and its recommendations, initiated the planning phase of our project. at this point, the team determined that leveraging an existing repository solution bundle in the samvera community, sufia, was the best starting point for our future system. sufia is primarily built as a self-deposit style repository and lacked many of the features we require for our institutional repository application, such as deposit management workflows, collections, and robust analytics. it was clear to us that we had work to do to bring sufia to a place where we would be able to apply it to our institutional repository use case. at that time, the samvera community was engaged in a number of initiatives that we found compelling for our use cases, primarily among these was implementation of the portland common data model, which showed promise for arbitrarily complex compound objects in our repository. as a first step, we initiated a number of development sprints to stand up a generic sufia repository as a starting point for both digging into the technology stack for sufia and for helping the repository management team get a sense for the look and feel of the new repository and how it, generally, would function. by summer 2015, we had a number of development sprints under our belts and had started to make local changes to sufia to meet our requirements. it was clear, however, that the team still had a great deal to learn about the technology stack, communication within the team, and the processes we needed to have in place to function effectively. the amount of learning we needed to do was not entirely surprising – we had three developers who were relatively new to the samvera technology stack and a product owner who had never formally played that role before. making our way up the technological learning curve wasn’t a bad thing, in fact it is safe to say that diving into the technology was probably the best way for our team to come to a common and in depth understanding of how the technology worked. the product owner and project manager also had a lot of learning to do, both about the technology (at their respective levels of granularity and need) and about communications and expectations for the project. concepts like sprint planning, standup meetings, retrospectives, and demos, while conceptually easy, were challenging for the product owner to implement, given his lack of experience and background in this area. the product owner came from a background of project management and implementation, but had never done this kind of work in a software development environment and never as an officially titled product owner or manager. this required the product owner to parse the concepts of the development framework the team was using (a loose scrum model), crosswalk them to terms or project components he was familiar with, and then try to determine how to apply lessons learned in previous project environments to these concepts. doing this while engaging in a development project as a product owner was challenging, and the resulting loss of confidence by the product owner likely resulted in a loss of efficiency on the project. we also struggled through understanding how best to manage the infrastructure of our project such as github repositories, project boards, documentation spaces, metadata application profiles, and ancillary data sources required for the project. suffice to say, we progressed in fits and starts during all of this learning, though it is important to note that we progressed, nonetheless. summer and fall of 2015 were also a time of intense personnel churn in our organization. in late summer, the head of our technology department left the organization, followed shortly by our lead developer. through this time, we maintained monthly, two week development sprints on the project, but the rate of progress in these sprints slowed dramatically as we worked through rehiring for these positions and moving forward on development while still on the learning curve for the project. suffice to say the departure of these two individuals created an enormous delay in the overall project. though support for the project remained in place, this disruption left a lot of questions in our minds about the future of the project. some conversations among team members at this time hinted at withdrawing from the migration project – we’d had some successes, but the project was new enough that we could have called it off without too much loss. fortunately, we chose to keep moving forward – our need for a new repository system was too great, the remaining team was motivated to continue, and we had full support of our administration, even in the face of these challenges. stasimon 1 – acquired knowledge and lost naivete the learning curve this is something we expect on every new project, a period of exploration and learning in order to understand enough about the technology, data, requirements, and team to allow the project to progress. we must say, though, that the learning curve for our team, on this project, was steep and long – not only were we working with new technology, but the trappings of a repository migration project were largely new to the team. it is challenging to say how much time must be allotted for scaling the learning curve for a project of this sort. in fact, one might even suggest that such an approach is futile, that the learning must happen either independent of the project, or while the project progresses – otherwise the team will not know enough to effectively do their work. in our case, this looked like a lot of time spent. in our initial months of development sprints, exploring code, and data, and metadata to understand what we were even looking at and how it all fit together. this is an intangible that can be uncomfortable in the planning stages of a project, due to uncertainties around how much time and resources it will require, but it is a necessary discomfort that allows for growth, and building a baseline of experience and understanding. managing personnel churn this is going to happen eventually. there are better and worse times for it to happen, but regardless, it will be disruptive. the primary mistake we made was allowing ourselves to have a single point of failure with respect to succession planning for personnel turnover. this would have been helpful both in the case of management turnover (department head leaving) and in developer turnover (lead developer leaving), though one can imagine that succession planning for all personnel would be useful. the primary issues in management turnover were uncertainty about whether the initial amount of resourcing committed to the project would continue under new leadership. process could have helped alleviate this problem – having a clear project plan in place that defines the resourcing can allow incoming management understand the expectations for ongoing projects. the issues raised with the departure of our lead developer were that he held all (or at least most) of the connection to the community we worked with, and that much of our work was benchmarked against his skill set. after the lead developer’s departure, we made a conscious effort to engage with the samvera community across our team and to set production benchmarks based on individual developers capacity and/or to benchmark based on what the team as a whole could accomplish. it is so critical to understand where the entire team is with respect to ability in order to set realistic milestones and to buffer against losing a team member. episode 2 – clarity we started winter and spring 2016 with an introspective period kicked off by re-evaluating project timelines, expectations, and roles in the face of the issues we’d faced at this point in the project – namely unclear process and significant personnel changes. this re-evaluation gave us an opportunity to assess our previous assumptions about requirements, minimum viable product, and stakeholder engagement in the project. out of this re-evaluation we realized that we needed more realistic milestones, redefined what we meant by minimum viable product for the repository, and determined that periodic re-evaluations of our milestones and progress should become part of our regular work schedule. during this time period, we also hired a new lead developer into our group. this hire was indispensable for completing our team, but also resurfaced challenges experienced earlier in the project related to the learning curve for the technology we were using and how we functioned as a team. at the same time, the samvera community was focusing resources around an architectural realignment of some of its repository solution bundles, the result of which would be hyrax, which targets a wide range of types of digital repositories. as we tracked this community work, we identified three areas where we thought our team could provide assistance. one area was creating a feature that had not yet been spoken for by another institution or individual in the community, implementing nested works. the second effort was initiating a community effort to build flexible workflows into the system. lastly, we helped initiate an interest group in the community focused on migrating content from dspace to samvera. this interest group provided a number of opportunities for collaboration (or at the very least, idea sharing) and, maybe more importantly, peer-to-peer counseling on issues arising from this type of migration. stasimon 2 – patience and engagement iterative evaluation of process this period of the project started with a re-evaluation of all of our expectations for the project. initially, this felt like a step backwards – like we had lost traction and were moving in the wrong direction. in retrospect, though, this was an opportunity for our team to learn about iterative evaluation of project expectations, timelines, and roles. architectural churn the architectural churn we experienced in the samvera community during this time period, was not unique to our institution – we are but one adopter of the software in question. we believed, though, that our project encountered this churn at just the wrong time – that we were particularly caught up in the difficulty of these changes and uncertainty in the code we were using. one approach to managing this type of churn is to stop and wait until it calms down, evaluate where things stand, and then make a decision about what software (version etc.) to use. another approach, the one we chose, was to move our development effort onto the master branch of the new system (hyrax) and “ride the wave” of change and enhancement. this approach requires a special amount of attention and acceptance of change and difficulty, but we found that it paid off. the enhancements being made to the new system, especially as the initial year of work on hyrax progressed, were enough to justify the risk of riding the wave. community engagement it turns out that this is super valuable. by being closely involved with the samvera community, not only were we better able to see what directions the community was moving in, as a team committing resources to doing work in the community we were able to have a hand on the steering wheel. this isn’t to say that we swayed the community to do things that were not in its interest, but that our commitment of resources offered a nucleation site for community development around the features we sought to build. this resourcing and the effort that we, and other institutions, put into the samvera community resulted in features that both met our local needs and the needs of many other community members. we’re learning we’ve learned something since episode 1: changes will happen, we need to be flexible, but keep moving forward. the introduction of a new department head and lead developer, and the loss of our project manager were challenging, but we had experienced personnel challenges before during this project (see above), and we knew how to better work with modifications to our team – namely, expect a learning curve. we also worked hard to accept more frequent reevaluation of project milestones in the face of the skills and personnel at hand, which came in form of regular, approximately quarterly, evaluations of the status of the project and whether our existing milestones were realistic given how things were going. episode 3 – closure by winter 2017, we had determined that it was necessary to re-evaluate and reset the project (again) in the face of the aforementioned architectural changes to the software we intended to use. it was clear to us that we wanted to target our repository migration at the emerging hyrax 2.0, given the set of features that would be included in that release, compared with the hyrax 1.0 release. but the release of hyrax 2.0 was only on the horizon when we made this decision, and we needed to develop a plan for tracking the work happening on the hyrax master branch in order to be ready to release our repository close on the heels of the 2.0 release. this was tricky in some ways, but the extent to which we were engaging closely with the samvera community helped guide us through these challenges – we knew what was happening, when it was expected to happen, and how we might maneuver our work around any obstacles. parallel to developing against an impending release of hyrax, we were developing tooling for migrating our repository contents from dspace to hyrax 2.0. this development was closely tied to metadata remediation, controlled vocabulary development, and initial user experience testing and conversations with stakeholders. ultimately, and somewhat surprisingly, this tooling and metadata work proved to be the primary time sink for the project from this point until project completion. by this time, we had refined our planning and scheduling processes in a way that worked well for our team. we engaged in re-evaluation of project milestones about every 2.5 months, settled on longer work cycles for the project, and worked to define roles and expectations for all team members. in addition, we set a series of meetings to discuss the major aspects of the project, coupled with brief daily check in meetings (stand ups) and longer weekly meetings to discuss ongoing or larger issues. the repository migration project officially finished in january 2018 at which point we switched from migration to maintenance, cleanup, and enhancement. stasimon 3 – parts and enhancements this is where we should have been the hardest part of looking back on this project is seeing how, in the last year of the project, we managed to do most of the heavy lifting, while prior to this year the project felt untethered. as discussed, there were a lot of reasons for the differences in our ability to gain traction on this project in its first years, but it is clear that experience, personnel stability, and code stability played a huge role in our ability to move the project forward in its final year. all we can do, at this point, is learn from our experience, and we have been – over the past six months since our repository migration concluded, our technology department has been working to revise a number of procedures for selecting projects, setting expectations for project structure and process, and defining roles and responsibilities for project participants in order to help future projects gain traction earlier. this project has multiple tracks explicitly defining and splitting the project into multiple tracks (building the repository on hyrax vs. migrating the content) was critical to breaking this project into manageable (if large) pieces. it also allowed us to better assign resources to these tracks, to define when work needed to happen on these tracks, and to see where the tracks were and were not dependent on one another. this breakdown of the project helped us move past a period of paralysis wherein it felt like we couldn’t make progress because the entire team was trying to solve all of the problems at once. risk can be rewarding ultimately, our choice to pin our repository work to the impending release of hyrax 2.0 was a good one. at times, though, it felt risky to place so much control over our project into the hands of the community at large. we feel, though, that our direct and frequent engagement with the samvera community throughout this phase of our project alleviated much of the mystery around when we would get this new release and what it would look like. we also found that by working so closely with the community during this time period, we knew so much more about hyrax when we finally launched our repository. we had spent so much time both in the code and working in various iterations of the software as users and repository managers that we had a very good sense for the ups and downs of the final product. exodus in the end, this project was a success, but suffice to say there were a number of times in the past 2.5 years when our grasp on the work felt tenuous at best. our prior experiences suggested that this would be the case, to an extent, as it is for all projects – nothing ever goes smoothly or well all the way through. but the issues we encountered in this case were manifold, and in this way it feels like we hit all of the bumps one could. so what did we even learn? we’ve discussed each of the three major phases of the project, and lessons learned along the way. here we distill those lessons. so what does it take to migrate a digital repository from one platform to the next? patience unexpected things will occur during the repository migration – in our case these primarily came in the form of organizational change, personnel change, disruptions in our repository community. when these changes come, it is necessary to be patient while they happen, and this patience can express itself in a few ways. first, one can stop and wait for whatever disruption is occurring to pass, then pick the project back up. second, one can keep moving forward on the project, working through, past, and with whatever challenges come with the disruption. the second approach, the approach we took, certainly resulted in slowdowns to our project. at first, these slowdowns were hard because they were pushing against our timelines and delivery dates. eventually, though, as we gained patience with the onslaught of disruptions, we learned to take the disruptions in stride and use them as opportunities to evaluate project expectations and goals. it is hard to know which of these two approaches results in a longer timeline for any given project, but we do know that choosing the second approach resulted in a long project timeline. we also know, though, that it was impossible to predict the disruptions we faced, and that commiting to the project was right for us. process if at all possible, have good processes in place before starting a major project. of course, this isn’t always possible and in many cases, like ours, one believes the processes in place to run a project are sufficient, or even exemplary. what are the key elements of process that we can look back and point to as critical? project planning and decomposition – at an abstract level, breaking the project into its component pieces and then, where possible, recomposing these pieces into logical project stages or milestones iterative project evaluation – periodically evaluating progress on the project, roles and expectations, requirements, and milestones well defined roles and expectations – who is responsible for the decomposed elements of the project and what does it mean to be responsible for those elements? the first of these, project planning and decomposition, is an area that we felt we had a pretty good handle on at almost every phase of the project. in some ways we did, but our ability to understand the project in its entirety was limited in the beginning, as it always is. that said, we believe it wasn’t really until fairly late in the project that we took the time to fully break the project down into its elemental parts and discuss at length the requirement each part fulfilled, the approach to resolving each part, and who was responsible for each part. which brings us to the second element, iterative project evaluation – this allows team members to feed their experiences with implementation of requirements and unexpected disruptions to the project back into the project planning, roles, and responsibilities. in the end, we found that evaluation on a number of time scales was helpful, with weekly and bi-monthly evaluations of progress on meeting requirements, roles, and expectations. last, defining these roles and expectations clearly among team members helps everyone understand to whom they can turn with questions or for help on any element of the project. like other elements of process, these roles and expectations can change over time, and should be evaluated frequently to ensure every team member is comfortable and clear on their role and the expectations that come along with it. perseverance we had originally named this lesson “peril,” and indeed peril lurked around many corners throughout. but what gets one past the peril is perseverance – moving forward on the project regardless of the dark places we can all find ourselves when working hard on something challenging. part of what allows perseverance to flourish is celebrating when things go well, focusing both on the good and the bad in the project. this is something we could have done more of on our project and this is a lesson we’re still trying to learn on our team. we should celebrate our perseverance as much, if not more, than we celebrate an incremental win or the completion of the project. perseverance is what gets us through those increments and through to the end. conclusion we’ve learned a lot of lessons from this project and expect to learn more as we move on to the next projects. but, here, months after the completion of our repository migration, these three ps – patience, process, and perseverance – have been a focus for our team and our department as a whole. as we move to other projects, some larger even than this one, we are actively working to learn our lessons and to move into the future on surer footing than we were on before we started. acknowledgements we would like to acknowledge the support of our colleagues at oregon state university libraries and press and in the samvera community, who provided insight, feedback, and encouragement throughout this project. resources city dionysia – explore the tragic structure. http://artsedge.kennedy-center.org/interactives/greece/theater/playstragicstructure.html (accessed 31 may 2018). samvera – an open source solution for digital content. http://samvera.org/ (accessed 31 may 2018) van tuyl, s, zhang, h, boock, m. 2015. analysis of challenges and opportunities for migrating scholarsarchive@osu to a new technical platform: requirements analysis, environmental scan, and recommended next steps. http://ir.library.oregonstate.edu/concern/technical_reports/k06989027 (accessed 1 may 2018). about the authors steve van tuyl is the digital repository librarian at oregon state university libraries and press. he currently acts as a project manager for the emerging technologies and services department, as well as product owner for a number of repositories. he also has worked for many years building and supporting research data services programs and as an all-around data flunky. josh gum is a analyst programmer at oregon state university libraries and press. he has worked in technology related fields for more than 20 years with an emphasis on web technologies. gregorio luis ramirez is an analyst programmer at oregon state university libraries and press. his work focuses on a broad spectrum of digital projects in the emerging technologies and services department. margaret mellinger is the director of emerging technologies and services at oregon state university libraries and press and has worked with open source projects for several years. brandon straley is an analyst programmer at oregon state university libraries and press in the emerging technologies and services department. his work primarily focuses on both on emergent and legacy web technologies and agile development practices. he aspires to continue refining his development practices and learning more about how to better work on large, long term projects. ryan wick is an analyst programmer at oregon state university libraries and press, with emerging technologies and services as well as the special collections and archives research center. he works primarily with repositories, metadata and digital collections. hui zhang is a digital applications librarian at oregon state university libraries and press where he focuses on repositories and special projects. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – integrating linked data into discovery mission editorial committee process and structure code4lib issue 21, 2013-07-15 integrating linked data into discovery although the linked data paradigm has evolved from a research idea to a practical approach for publishing structured data on the web, the performance gap between currently available rdf data stores and the somewhat older search technologies could not be closed. the combination of linked data with a search engine can help to improve ad-hoc retrieval. this article presents and documents the process of building a search index for the solr search engine from bibliographic records published as linked open data. by götz hatop introduction many bibliographic datasets are already published as linked data and it is now crucial to see how this data can fit into existing information management workflows.  the exercise is useful not only to see how novel improvements from semantic web development can influence data management in libraries, but also because library data is highly structured and may as well serve as an example to demonstrate the benefits of semantic web technologies and thus can help to bootstrap the semantic web. in the library community, the vufind discovery system, published by villanova [1] is known to be a flexible open source resource discovery system. the software has developed an active user community, with a number of libraries currently utilizing it as their discovery platform. vufind itself is built on top of the solr search software, which is developed and distributed by the apache lucene project. building a search index for the vufind solr search engine from a corpus of linked data is an interesting task which can be accomplished through the use of a number of open source software components in conjunction with the development of some programs to embed the required functionality.  the result is a flexible system that works with the definition of sparql queries and a xslt transformation to finally perform the indexing. this way, only standardized techniques from the semantic web stack are required to index a linked data source. the programs used to achieve the indexing are published open source and may be of interest to the semantic library community [2]. libraries and linked data linked data represents a major opportunity for libraries to integrate their information resources within the semantic web.  in addition, the semantic web stack itself provides a robust and highly standardized framework offering libraries the ability to provide better data management and data re-use. another area where linked data may play an important role for libraries is the field of semantic publishing, the enrichment of scientific articles with machine readable rdf statements as described by peroni and shotton [3]. tim berners-lee first outlined the basic idea of linked data in 2006. in his seminal design note [4] he described four principles to be followed for linked data: all items should be identified using uris. all uris should be dereferenceable, that is, using http, uris enable anyone (machine or human) to look up an item identified through the uri. looking up a uri leads to more data, also known as the follow-your-nose principle. links to uris in other data sets should be included to enable further data discovery. a more detailed analysis about how bibliographic data can be published as linked data is described in bizer, et al’s., how to publish linked data on the web [5]. linked data published on the web are usually represented using the resource description framework (rdf). the framework provides a simple data model, but is a key component tailored towards information exchange within the semantic web. according to the w3c, “rdf provides interoperability between applications that exchange machine-understandable information on the web”. even if the semantic web as a whole may still be years away from reaching its goal of being society’s chief information platform, the rdf framework has already reached much attention and provides useful and accepted standards for data and information management. at the time of this writing, the data hub [6] contains more than 300 datasets published as linked open data by many different organizations. included in this set are more than 50 datasets obtained from libraries, most of them with an open data license and formatted according to the rdf framework. however, although publishing rdf datasets as a dump file with an open data license is important, it is only one more step towards the vision of the “web of data”, which is just another name for the semantic web. with sparql, the w3c has introduced a rdf query language that supports querying of multiple rdf graphs. sparql is not only a query language for rdf, the standard specifies the execution of distributed queries over different sparql endpoints. sparql can therefore be called a federated query language. also, the protocol defines how to send a query and the results of a query across the web. in this way, sparql makes it possible to publish rdf data through standard interfaces.  some libraries and journal publishers already provide their metadata via sparql endpoints. however, the operation of a sparql service endpoint is not an easy task, and the number of public sparql endpoints is growing only moderately according to the statistics made available by vandenbussche [7]. in terms of the fast evolving technologies in the web age, the semantic web can already be called an old stack. for example, rdf was originally recommended by the w3c on the february 22, 1999. greenberg [8] points out many similarities between libraries and the semantic web: both have been invented as a response to information abundance, their mission is grounded in service and information access, and libraries and the semantic web both benefit from national and international standards. nevertheless, the technologies defined within the semantic web stack are not well established in libraries today, and the semantic web community is not fully aware of the skills, talent, and knowledge that catalogers have and which may be of help to advance the semantic web. on the other hand, the apache solr [9] search system has taken the library world by storm. from hathi trust to small collections, solr has become the search engine of choice for libraries. it is therefore not surprising, that the vufind discovery system uses solr for its purpose, and is not built upon a rdf triple store. fortunately, the software does not make strong assumptions about the underlying index structure and can coexist with non-marc data as soon as these data are indexed conforming to the scheme provided by vufind. implementation strategies in general, web crawling of linked data can be a complex task, since rdf data may be distributed among several data access points administered by different organizations. that may require fail-over mechanisms as well as some kind of agreement if data providers have established a fair use policy. bibliographic datasets are often available as linked open data in the form of a rdf data dump file. in the library world, authority data and title records are often separated into distinct chunks of data. for the purpose of index building, all data relevant must be gathered together so that a solr index record can be built. building a solr index out of rdf data can be done in different ways: rdf data have a xml serialization and therefore can be transformed to a solr document as needed by the index engine. some triple stores use a full-text index internally. since most of these indexes are based on lucene, it is possible to use this index from the outside of the systems. the rdf data can be loaded into a triple store and then the programming api from the triple store can be used to access the data. this way makes it possible to also index some inferenced results. while the first option is simple and straightforward, its main drawback is that only relative small datasets can be tackled. the resulting filesystem operations over large datasets become difficult to handle and harder to scale. the second option requires some investigation of the underlying software architecture and is bound to a specific triple store. only the last option allows the use of standard apis and has chances to scale up to large datasets and remain free of dependencies.  as a result, the third approach was chosen and a prototypical implementation in java with the jena framework [10] developed. after loading all data required for indexing into a jena triple store, three steps can be identified for building a solr search index. the software written in the context of this article supports these three steps: record enumeration: all resources that should be indexed need to be identified in this step. resource dump: a query that shows what the underlying triple store knows about a resource. result transformation: the mapping and filtering of the results such that data can be loaded into the index engine. whereas steps 1 and 2 work with sparql, the last step works with xslt. the main advantage of this is that the definition of an indexing procedure depends only on well-defined semantic web technologies. record enumeration the formulation of a sparql query to enumerate all resources that should be indexed is crucial to the success of the overall operation. the following sparql query enumerates the complete data store and was used during the development of the index routines. prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> prefix dct: <http://purl.org/dc/terms/> prefix xsd: <http://www.w3.org/2001/xmlschema#> # general resource enumeration : every subject, but no blank nodes select distinct ?subject where { ?subject ?p ?o . filter (!isblank(?subject)) . } order by desc(?subject) the query will deliver all subjects available from a sparql service, but without blank nodes. however, in practice it is often necessary and easily possible to restrict the record enumeration by using date properties or some other known statements about the resources in question. resource dump data modeling with rdf is based on statements with subject, predicate and object. therefore, the original document-orientated structure of the bibliographic data is destroyed. in this phase of indexing, all information available about a resource is queried from the store. a common solution to the problem of querying a graph for all relevant information is described in the literature [11] as “symmetric consise bounded description”, which informally is the extraction of a sub-graph with all nodes associated with the resource. the sparql standard uses the describe keyword for this purpose. some, but not all triple stores currently used in practice do implement the sparql describe part of the specification, but the implementation is often limited to simple consise bounded descriptions, that is in effect a description without those nodes, where the resource is the object. to the best of our knowledge, symmetric consise bounded descriptions are not built into triple stores right now and have to be formulated as sparql construct-queries. since the sparql query language does not support recursion, the required query can not be given in general, but depends on the specific data model. typically, author information and partof-relationships need to be considered here. a typical query may look like the one shown below. prefix dct: <http://purl.org/dc/terms/> prefix dcmitype: <http://purl.org/dc/dcmitype/> prefix dcq: <http://purl.org/dc/qualifier/1.0/> prefix gnd: <http://d-nb.info/gnd/> prefix foaf: <http://xmlns.com/foaf/0.1/> prefix fabio: <http://purl.org/spar/fabio/> prefix urn: <http://www.d-nb.de/standards/urn/> prefix shg: <http://localhost/view/> ## resource dump. construct { ?p ?o <subject> . ?o ?x ?y . } where { ?p ?o . optional { ?o ?x ?y filter (isblank(?o) && !isblank(?y)) } optional { dct:ispartof ?o . ?o ?x ?y . } optional { dct:creator ?o . ?o ?x ?y . } } in the query shown, the <subject> parameter is resolved step by step by the record enumeration of the previous step. the query is intended to filter out blank nodes and to fetch information that is hidden behind uri nodes like the author name and the information about records related by a partof-relationship. the sparql standard is prepared for federated queries, and the jena framework does implement the service keyword to support this behavior. accordingly, this phase of the indexing process is the place, where the separation of authority and title data into distinct data sets can be treated.  the approach taken here, to first load all required data into a triple store, makes the overall setup relative simple.  however, an approach using federated sparql queries for resource dump could take the distributed nature of linked data into account. buil-aranda et.al. [12]  formalize such a setup, and also provide some background information on the problem of federated query evaluation. result transformation since sparql query results can be returned in a standardized xml format, the process of transforming the result to the solr index format used by vufind can be done with a xslt transformation. an example of a xslt stylesheet to transform the rdf/xml response format to the solr index scheme in question is contained within the sources of this project [2]. data written in rdf may be considered to be on a higher level of abstraction than those in xml. for that reason, the translation from xml to rdf is often called lifting, while the opposite direction is called lowering. apart from its syntactic ambiguities, processing rdf/xml via xslt loses another feature of rdf, namely its interplay with ontological information. the “lowering” of the rdf/xml data in our use case is the mapping to the solr index scheme used by vufind. this final processing step provides all the flexibility that is available with xslt and benefits from the standardized xml formats that come with rdf. xslt can be problematic with respect to performance, and input size is a determining factor. the processing described here deals largely with only small data blocks, but every record has to be transformed from its rdf/xml representation to the xml variant required by solr before it can be loaded to the index. although detailed performance studies have not been undertaken in this project, some very simple tests gave hints that performance is more likely to be determined by the solr engine rather than xslt, at least as long as the standard http based solr update processing is used. proof of concepts to test the concepts in a more general setting, two datasets published by the dnb in 2012 were indexed. the datasets are published as two big sets of rdf data, known as gnd (german norm data) and dnb (deutsche nationalbibliographie). both data sets could be loaded into a jena triple store and indexed with the developed programs within three weeks on a standard desktop pc. the problems found in the datasets tested are most often due to the existence of space characters in uris, which is not allowed by specification. the long running indexing process could handle these problems through the use of careful java exception handling and ignoring invalid records. the jena framework was found to be flexible and easy to work with, and the adjustments could be made without the need to study the jena code in greater detail. in a somewhat simpler environment the developed indexing program is in production use since some month. metadata from an institutional repository and from journals hosted with the open journal system are collected via their oai interfaces, lifted to rdf and stored in a jena tdb triple store. the indexing of about 5400 bibliographic records from the rdf data store to the solr index engine can be done within a few minutes, and different mappings from rdf to the index scheme used by vufind are tested out easily. conclusion the predicated nature of the resource description framework has emerged as a widespread formalism for information and data exchange. this report describes a method of building a search index from linked data sources and shows that the combination of semantic web technologies and already established search technology is possible without profound changes to existing systems. the approach taken here to solve the problem of indexing linked data works with a simplified setup by providing a single point of data access. areas for future investigation include optimising data processing by adjusting the indexing algorithms to better utilize the federated nature of the underlying sparql services. references [1] villanova university: vufind. http://vufind.org (2012-04-22). [2] hatop, g.: the shanghai linked data indexer. 2013. https://github.com/cloud8/shanghai (2013-05-04). [3] silvio peroni and david shotton: fabio and cito: ontologies for describing bibliographic resources and citations. 2012-08-13. web semantics: science, services and agents on the world wide web. [4] berners-lee, t.: linked data – design issues. 2006-07-27. http://www.w3.org/designissues/linkeddata.html (2012-11-22). [5] bizer, c., cyganiak, r., heath, t.: how to publish linked data on the web. 2007. http://www4.wiwiss.fu-berlin.de/bizer/pub/linkeddatatutorial (2012-12-01). [6] open knowledge foundation: the data hub. 2013. http://datahub.io/group/lld (2013-01-03). [7] pierre-yves vandenbussche: public sparql endpoints stats. 2013. http://labs.mondeca.com/sparqlendpointsstatus (2013-06-13) [8] jane greenberg: advancing the semantic web via library functions. chapter 11, knitting the semantic web. cataloging & classification quarterly 43(3-4). 2007. [9] apache software foundation: apache solr. 2012. http://lucene.apache.org/solr (2012-08-30). [10] apache software foundation: apache jena. 2012. http://jena.apache.org (2012-09-02). [11] stickler, p.: cbd – concise bounded description. 2005. http://www.w3.org/submission/cbd (2013-05-04). [12] c. buil-aranda et. al.: federating queries in sparql 1.1: syntax, semantics and evaluation. web semantics: science, services and agents on the world wide web 18 (2013) 1–17 about götz hatop is an information scientist and works at the it department of the university library of marburg/lahn.  he is interested in semantic web technologies and information architecture. subscribe to comments: for this article | for all articles 2 responses to "integrating linked data into discovery" please leave a response below: paul hermans, 2013-07-30 the example construct query dor dumping the resource seems to contain errors (invalid triple patterns). götz hatop, 2013-08-07 yes, there seems to be some rendering problem with the correct triple pattern used in the described case, since some < and > are included. you may want to look at the sources to see the plain sparql code, it is not much more than to have subject predicate object in the right order. leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – using xslt’s sql extension with encyclopedia virginia mission editorial committee process and structure code4lib issue 16, 2012-02-03 using xslt’s sql extension with encyclopedia virginia this paper explores how to integrate data across a hybrid relational database and xml-based management system. it examines specifically how xslt’s sql extension can be used to communicate information between sql tables and tei-conformant xml documents to make data-centric content more manageable and flexible and thereby leverage the strengths of both systems. in what follows, one will learn about some of the methods, benefits, and shortcomings of xslt’s sql extension in the context of encyclopedia virginia, an open access publication of the virginia foundation for the humanities that utilizes a suite of digital humanities and digital library xml vocabularies such as tei and mets. by matthew gibson background and context ­­­­­­­some questions so why and when would one want to mix xslt—arguably one of the most powerful languages for performing xml transformations—with sql expressions, a language for talking to, managing, and extracting information from a relational database? having cut my own xml teeth beginning in 1998 marking up and managing the markup of electronic texts at the university of virginia library, i embraced a poor assumption that mixing xml and relational databases didn’t make much sense. why, i thought at the time, would you want to store richly structured information—information that, in some ways, was its own self-described and self-contained relational database—inside of a relational database management system? it just seemed redundant and not really keeping with the spirit of what xml was about: platform independent content that could be read and generally understood by humans and easily manipulated by machines. my approach and view of xml during this time was less that it was an efficient and consistent way to transport data and more that it provided the rules by which i could describe complexly structured literary texts with accurate structural and semantically rich markup. i was an xml purist: disdainful of textual projects that did not employ xml applications in their production and reluctant to see and imagine the strengths and improved flexibility that mixing xml and xml tools with other environments and systems—such as a relational database management system (rdbms)—could yield for content and data. a hybrid publication system in 2005, i started developing and building encyclopedia virginia (ev), a digitally-born, open access publication of the virginia foundation for the humanities that explores the history and culture of the state of virginia. bringing to this project all of my xml biases and experiences, i decided that all encyclopedia entries—i.e. the textual essays—in ev would conform to the tei (text encoding initiative) p5 guidelines[1] and that all media—still and moving images, and audio files—would be described using different metadata standards inside of a mets (metadata encoding and transmission standard) wrapper. thus the content structure of an entry and that entry’s relationship to other entries and other objects (e.g. media, external urls, etc.) would be defined and “preserved” in the deeper hierarchies of the xml to allow for the content’s portability to different platforms and contexts. while much of the semantic and mixed content markup that xml affords might get lost, or might be more difficult to reconstruct from a pure rdbms environment, there are other data and workflow requirements for encyclopedia virginia that make an xml/mysql hybrid publication system a better solution than what a pure xml database option might provide. the strengths that an rdbms provides for ev are: version control over every piece of content that goes into the encyclopedia; one-to-many relationship management of, for instance, one author and/or editor to many articles, one chronological event referenced by many articles, and one media object shared by many articles; and, most importantly, more efficient and scalable performance in looking up and retrieving data. a very generalized view of encyclopedia virginia‘s publication workflow for an entry looks something like this: xslt transforms word processed and edited entry to tei p5-compliant xml php shreds tei into appropriate mysql tables (e.g. authors, editors, events, and geolocations) but retains the canonical and intact xml document for future editing and versioning a sql challenge for xslt although all of ev‘s content is stored in and delivered from a lamp (linux, apache, mysql, and php) platform, the original tei documents that are ingested into the mysql tables are created and normalized with a series of xsl transformations before the php/mysql processing takes place. over time we have realized that in the tei there are several structures that are better stored and managed through mysql and that, instead of describing those in the xml, it is more efficient and consistent to point to unique keys in the database that represent that content. one challenge, in particular, was how tei might reference authors. because ev publishes the affiliation of each author and because we want that information to be consistent across, say, multiple entries that a single author might write, marking up that text across multiple tei documents can lead to variability. what is more, if a given author contributes a large number of articles and her affiliation needs to change, it is difficult to manage that change over multiple files. in the past, marking up an author across tei documents might look something like this (figures 1 & 2): figure 1. tei snippet: “divorce in early virginia indian society” <tei xmlns="http://www.tei-c.org/ns/1.0" xml:id="divorcepc"> <teiheader> <filedesc> <titlestmt> <title>divorce in early virginia indian society</title> <respstmt> <resp>contributor</resp> <persname> <forename>john</forename> <surname>doe</surname> <affiliation>professor of history at collegiate university</affiliation> </persname> </respstmt> </tei> figure 2. tei snippet: “pocahontas” <tei xmlns="http://www.tei-c.org/ns/1.0" xml:id="pocahontas"> <teiheader> <filedesc> <titlestmt> <title>pocahontas (d. 1617)</title> <respstmt> <resp>contributor</resp> <persname> <forename>john</forename> <surname>doe</surname> <affiliation>professor emeritus of history at collegiate university and author of <hi rend="italic">love and death in the virginia colony</hi> (1990) and <hi rend="italic">hard times for early virginia coloniasts</hi> (2005) </affiliation> </persname> </respstmt> </tei> between these two entries by john doe, the author’s affiliation has changed. while one could certainly reflect these changes consistently in the xml with a little searching, cutting and pasting, we decided it would be much easier and more modular to control the author’s relationship to his biography and to the entries he contributed by inputting and storing that information directly in the sql database. thus if we had to change any of the fields related to that author, such as his affiliation, we would do it once, and those changes would be reflected in all of the entries that he wrote. thus each of the entry snippets above would, before being published, look like this (figure 3): figure 3. tei snippet: “pocahontas” with pointer to primary index in author table <tei xmlns="http://www.tei-c.org/ns/1.0" xml:id="pocahontas"> <teiheader> <filedesc> <titlestmt> <title>pocahontas (d. 1617)</title> <respstmt> <resp>contributor</resp> <name key="195"/> </respstmt> where the result in the author table with primary index matching “195” contains the following (figure 4): figure 4. result from sql table of author with with a primary index of 195 *************************** 1. row *************************** contributor_id: 195 salutation: dr. first_name: john last_name: doe middle_name: d. bio: professor emeritus of history at collegiate university and author of <em>love and death in the virginia colony</em> (1990) and <em>hard times for early virginia coloniasts</em> (2005) with the reference now made consistently across all entries written by john doe, we can pick and choose which values we want to display and how we want to display them. however, the missing piece here was how we would get that reference to the author into the xml itself. if we still valued keeping the xml intact for future portability, we had to figure out a way to retain the relationship between the tei file and the author’s key in the database. and the challenge, of course, was to get those keys into the xml in an automated way. because we use xslt to transform, test, and normalize all of our xml before we publish it, we looked at saxon’s xslt sql extension as a logical solution to create this relationship. the saxon sql extension while xalan—the apache project’s xslt processor—has an sql extension library, because our production process uses features of xslt 2.0 we turned to the saxon processor’s sql extension library. (nb: while you can run saxon’s sql extension with command-line parameters, all work below is done in the environment of oxygen’s xml editor.) prerequisites to get the saxon sql extension up and running and talking to your database you need several things: the java database connectivity (jdbc) driver for whatever database you are running—in our case it is mysql. the saxon-sql.jar (recent releases of oxygen come prebundled with this) a configuration file that allows oxygen to see the saxon-sql.jar for validation and processing. appropriate credentials (i.e. username and password) to access the database with which you want to communicate and that your database server is configured to be accessed by remote machines. xslt sql elements saxon’s sql extension defines five new xsl elements for interacting with a database that are all bound to the “sql” namespace: sql:connect, sql:query, sql:insert, sql:column, and sql:close. i actually don’t recommend using sql:insert and sql:column very much since injecting xml data into what, typically, is the more controlled environment of a relational database introduces too much variability and presents possibilities to really cause some chaos. the other three sql elements work as such: <sql:connect> establishes, as you might have guessed, the connection to the database and the attributes that it takes (driver, database, user, and password) provide the information that will validate the connection. <sql:query> does the real work for us here. it performs the query on the database and writes the results of the query to the result tree. while there are six attributes for sql:query, the main ones we use are: table—which defines the name of the table to be queried. this attribute is mandatory. column—defines the name of the column or columns to be retrieved. using “*” as the value of the column attribute selects all columns. this attribute is also mandatory. where—defines the conditions to be applied in a given selection. this is optional. <sql:close> closes the sql connection. an xslt example: authors2sql.xsl given these elements—and the snippets from figures 1 and 2 above as our source tree—let’s look at a sample of xslt code that places these elements in context: <?xml version="1.0" encoding="utf-8"?> <xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/xsl/transform" xmlns:xs="http://www.w3.org/2001/xmlschema" xmlns:tei="http://www.tei-c.org/ns/1.0" exclude-result-prefixes="xs" version="2.0" xmlns:sql="http://saxon.sf.net/sql" extension-element-prefixes="sql"> <xsl:output method="xml" encoding="utf-8" indent="yes"/> <!-the following parameters define the driver, the database that the connection will need to open, and the credentials to pass to the database to verify permissions --> <xsl:param name="driver" select="'org.gjt.mm.mysql.driver'"/> <xsl:param name="database" select="'jdbc:mysql://databaseurl/databasename'"/> <xsl:param name="user" select="'username'"/> <xsl:param name="password" select="'password'"/> <!-the template match selects the nodeset that will need to be analyzed for the author's name --> <xsl:template match="//tei:titlestmt/tei:respstmt"> <xsl:message>connecting to database...</xsl:message> <!-the "connection" variable establishes jdbc connection by selecting as its value the sql connection to the database. the use of the java namespace types the variable as a java language object in order to validate the connection type. the value of the connection variable is determined by the values of the parameters created above --> <xsl:variable name="connection" as="java:java.lang.object" xmlns:java="http://saxon.sf.net/java-type"> <sql:connect driver="{$driver}" database="{$database}" user="{$user}" password="{$password}"> <!-used primarily for debugging, if, for whatever reason, the credentials or something incorrect is passed in the connect statement, the process will terminate with the following message --> <xsl:fallback> <xsl:message terminate="yes">connection to mysql failed.</xsl:message> </xsl:fallback> </sql:connect> </xsl:variable> <!-if the variable is successfully established, the following message will be output --> <xsl:message>connected...</xsl:message> <!-the following variables select the context values from the source tree to test for values in the database to retrieve the correct key --> <xsl:variable name="fname" select="normalize-space(tei:persname/tei:forename)"/> <xsl:variable name="lname" select="normalize-space(tei:persname/tei:surname)"/> <!-the contrib-table and contribid variables below select an sql query statement as their values. components of the queries include: the value of the connection variable created above, the name of the table inside of the database upon which you want to perform the query, the conditions you are establishing for your selection, and the columns that you want to retrieve if those conditions are true. --> <xsl:variable name="dbase_lname"> <sql:query connection="$connection" table="contributors" where="last_name='{$lname}' and first_name='{$fname}'" column="last_name"/> </xsl:variable> <xsl:variable name="dbase_fname"> <sql:query connection="$connection" table="contributors" where="last_name='{$lname}' and first_name='{$fname}'" column="first_name"/> </xsl:variable> <xsl:variable name="contrib-table"> <sql:query connection="$connection" table="contributors" where="last_name='{$lname}' and first_name='{$fname}'" column="*"/> </xsl:variable> <xsl:variable name="contribid"> <sql:query connection="$connection" table="contributors" where="last_name='{$lname}' and first_name='{$fname}'" column="contributor_id"/> </xsl:variable> <!-the following messages can be used as debugging statements to see what type of data the query has retrieved. if the conditions you establish are true, for example, the value of the $dbase_lname variable below will be a copied xml nodeset from the mysql database if the $lname and $fname variables are equal to values in the table. the result returned would look like this in the message: <row><col>last_name</col></row> --> <xsl:message>first name: <xsl:copy-of select="$fname"/></xsl:message> <xsl:message>last name: <xsl:copy-of select="$lname"/></xsl:message> <xsl:message>table: <xsl:copy-of select="$dbase_lname"/></xsl:message> <xsl:message>contrib id: <xsl:copy-of select="$contribid"/></xsl:message> <!-the choose sequence below evaluates the tei:respstmt nodeset with the following conditionals: 1) if tei:name exists (i.e. if you have already run this stylesheet on this document and have generated the contributor's id), then you will just want to copy the tei:respstmt source tree into the result tree 2) if, on the other hand, any of the values of contrib-table match the last name and first name values extracted from the xml, then the stylesheet will copy the contributor_id from that row where those conditions are true 3) otherwise, all children of the tei:respstmt are copied into the result tree. --> <xsl:choose> <xsl:when test="tei:name"> <xsl:copy-of select="."/> </xsl:when> <xsl:when test="matches($contrib-table, $lname) and matches($contrib-table, $fname)"> <respstmt xmlns="http://www.tei-c.org/ns/1.0"> <resp xmlns="http://www.tei-c.org/ns/1.0"> <xsl:value-of select="tei:resp"/> </resp> <name xmlns="http://www.tei-c.org/ns/1.0"> <xsl:attribute name="key"> <xsl:copy-of select="$contribid"/> </xsl:attribute> </name> </respstmt> </xsl:when> <xsl:otherwise> <respstmt xmlns="http://www.tei-c.org/ns/1.0"><xsl:copy-of select="*"/></respstmt> </xsl:otherwise> </xsl:choose> <!-finally, before the xsl:template closes, you utilize the sql:close element and identify the value of the connection you want to close (which is the connection variable defined at the beginning of the template). --> <sql:close connection="$connection"/> </xsl:template> </xsl:stylesheet> summary despite these drawbacks, if a situation arises that requires information interchange between an xml-formatted document and a relational database, xslt’s sql extension can bridge these two platforms that describe information and information relationships in very different ways. what is more, when a project’s needs require leveraging the strengths of xml and a relational database—when managing and delivering information necessitates a hybrid solution between complex xml description and structure and a relating data efficiently—xslt can remain a powerful tool in your suite of options for working with xml content. [1] the p5 specification was in the process of being created during this period. the official release came about in 2007. about the author matthew gibson is director of digital programs at the virginia foundation for the humanities (vfh). he holds a ph.d. in english from the university of virginia. prior to joining the vfh in 2005, matthew served as associate director of the university of virginia library’s electronic text center. at the vfh, he oversees encyclopedia virginia, a digital publication about virginia history and culture; provides supervision and support for documents compass, a mellonand nhprc-funded initiative to facilitate digitization and interoperability between documentary editions; and guides the shortand long-term planning for the organization’s digital efforts and scholarly communications. matthew has a multitude of conference presentations and several publications to his credit, mostly in the field of digital library standards. for the past nine years he has taught week-long xml application building workshops as an independent consultant and for the association of research libraries. urls: http://virginiahumanities.org and http://encyclopediavirginia.org. subscribe to comments: for this article | for all articles one response to "using xslt’s sql extension with encyclopedia virginia" please leave a response below: bernadette, 2012-02-10 i see where the variable ‘contribid’ gets assigned a value in lines 50-53, but i can’t figure out where the variable ‘contrib-table’ gets a value to use in line 68? is it picked up from the ‘table’ attribute value in the sql:query somehow? otherwise, thanks so much for this! it’s a straight-forward example! leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – practical digital forensics at accession for born-digital institutional records mission editorial committee process and structure code4lib issue 31, 2016-01-28 practical digital forensics at accession for born-digital institutional records archivists have developed a consensus that forensic disk imaging is the easiest and most effective way to preserve the authenticity and integrity of born-digital materials. yet, disk imaging also has the potential to conflict with the needs of institutional archives – particularly those governed by public records laws. an alternative possibility is to systematically employ digital forensics tools during accession to acquire a limited amount of contextual metadata from filesystems. this paper will discuss the development of a desktop application that enables records creators to transfer digital records while employing basic digital forensics tools records’ native computing environment to gather record-events from ntfs filesystems. by gregory wiedeman the acquisition of born-digital records for institutional archives is nothing new. over the past five to ten years a consensus has developed within the profession that the best practice for the long-term preservation of digital records is the use of forensic disk imaging to preserve the records’ original bitstreams in their original context. yet, despite the recent growth and success of these techniques, disk imaging may provide some problems for institutional archives that may be subject to records retention schedules and public records laws. this article examines a different and more challenging approach to managing permanent digital records in institutional archives: the use of digital forensics tools in records’ original creation environment to preserve contextual metadata at accession, and the preservation of records at the file layer. forensic tools have developed to a point where they can be employed as part of an automated workflow – limiting practical usability barriers. even gathering a small portion of valuable contextual metadata in the form of filesystem timestamps has the potential to make records more usable and discoverable. the key to employing digital forensics at accession is developing a user-friendly transfer application that runs basic forensic tools in records’ native environments prior to transfer. at the university at albany, suny, we are developing ants – a basic desktop application that will enable records creators to transfer born-digital records to an institutional archive over shared network drives or file transfer protocol (ftp). written in python and limited to ntfs filesystems, this experimental tool contains a basic gui that allows records creators to describe the accession – either as a whole or by adding descriptions or access concerns to individual directories and files. ants then attempts to run mftrcrd, plaso, and python’s os.stat functions to gather as many record-events as possible before packaging records with bagit and transferring them to the archives. ants should serve as a useful proof of concept and a tool for future experimentation. evolution of digital forensics in archive digital records first became widely discussed by archivists in the 1990s, and the challenges of these new formats appeared formidable. some of the first major questions concerned the stability of records in volatile digital environments where they could be easily altered or manipulated. in order to grapple with the complexity and volatility of digital records, archivists turned to the field of diplomatics to help identify the general requirements to manage digital records – namely the establishment and maintenance of their authenticity, reliability, and integrity. [1] still, modern computers were not designed for the permanent maintenance of electronic records and the technical challenge of preserving this information was still daunting. the application of forensic disk imaging to archives gained widespread traction after 2010. matthew g. kirschenbaum, richard ovenden, and gabriela redwine made the case that the work of digital forensic practitioners overlapped with archivists’ requirements for managing digital records. digital forensics methodologies, they offered, can provide a way to stabilize digital information and preserve it in a state that includes the contextual information necessary for demonstrating records’ authenticity. [2] soon after, kam woods and christopher a. lee began an effort to directly apply digital forensics tools to archival workflows – work that evolved into the bitcurator project. they emphasized how, “disk images that contain complete operating systems capture significant information about the ‘digital ecosystem’ in which documents and media were created.” [3] the bitcurator project itself has developed effective and accessible forensics tools for archivists – enabling them to use, and see value from, disk imaging. [4] generally, the stated focus of these efforts has been on “cultural heritage collections” and “collecting institutions” generally, but how does digital forensics apply to institutional archives? in conjunction with the developing digital archives theory and the evolution of forensics tools, over the past five years there have been a number of practically-focused publications on managing born-digital archives. [5] these case studies and best practices have correctly prioritized getting file contents onto stable media, establishing intellectual control, and providing access. the available literature seems to show that many institutional repositories are accepting digital records as files rather than disk images. only more recently have practical case studies been published that showcase the preservation of contextual metadata outside of files. [6] part of this is likely due to lag time in the publishing process, but it might just be that some archivists are still making an educated decision to preserve digital records as files. this could be because the technical barriers to disk imaging are still too high for some repositories, or perhaps some repositories are finding that disk imaging—despite its applicability to digital archives as a whole—is not a perfect fit for many types of materials. institutional archives and public records while disk imaging offers the easiest way to ensure that digital materials are authentic and complete, the consensus on disk imaging for digital records also poses a few major challenges for institutional archives: imaging all of the physical disks which hold institutional records can be very impractical. disk imaging retains additional information that should have been disposed of in accordance with a traditional records retention schedule. archives managing records that are subject to public records laws may not be able to limit or restrict access to information that was unwittingly preserved in disk images. even with these concerns, institutional archivists should not easily abandon digital forensics or even disk imaging. recordkeeping practices are always changing and archivists may be able to update their records management strategies to meet these new realities. [7] organizations could also adopt an enterprise document or records management system to manage the entire records process from creation to final disposition. still, these methods may not help archivists who are legally obligated to make all information obtained from disk imaging available for public use. the university at albany, suny is a good example of where a large disk imaging program is unworkable for an institutional archives. first, we manage the permanent records of a large public university with over 17,000 students and more than 3,400 employees on three campuses. since the university archives has only one full-time professional staff member, imaging innumerous hard disks and managing the data footprint that would be created is more than impractical without a large organization-wide commitment driven from the university administration. secondly, as a public university, we are subject to new york state’s foil law which mandates that all records must be made available to the public, with only minor exceptions. [8] copying all the bits from a disk inevitably retains additional information outside of what was selected to be retained, and—by law—we are not able to enforce access policies to restrict the availability of deleted files, data found in slack space, or other records unwittingly transferred. outside of select cases, the risk of keeping all the bits is simply too great. managing records as files without disk imaging, we must manage records using the same abstractions used by operating systems: files, stored on filesystems, which likely contain the primary content of records. documenting the authenticity and integrity of the file itself is easy with the use of cryptographic hashes, commonly called checksums. this ensures the preservation of content as well as whatever contextual metadata that is included within the file. we can handle file format identification, extract embedded metadata like exif data, and document file size after records are transferred to the archives if we rely on checksums. so what are we missing if we preserve only files? computers also maintain a variety of other information on how a record was created, used, and maintained outside of a file, providing valuable context. as the bitcurator project reported, “extraction of basic technical metadata (such as timestamps) from file systems can provide a foundation on which one can establish a ‘ground-truth’ for content and structure on a device. they can provide significant support for assessment and preservation activities.” [9] additionally, dates have traditionally supported description in archives and worked to make records more accessible. the most useful and pragmatic approach is to gather as many forensic artifacts that can be attached to records as possible. these “record-events” would be any point of activity relating to a record that can be gathered from the original computing environment, including timestamps found in a filesystem, as well as actions entered into an event log or journal. these events could signify record creation, alteration, or use, and helps to contextualize the record itself. to gather this information, we need to take action in records’ native environment, which means we need to run forensics tools on each individual’s desktop before records are transferred to the archives. by combining an effective transfer application with basic forensic tools, we can make it easier for creators to transfer records while establishing a consistent submission package that includes at least some contextual metadata in addition to the files themselves. building a transfer tool at the university at albany, suny, we are building a basic gui desktop application that can be run on a records creators’ computer that will easily perform these tasks while transferring records to servers accessible to the university archives. ants, for archives network transfer system, is written in python and uses wxpython, a cross-platform gui wrapper that allows us to easily use and manipulate wxwidgets in python. application data is stored and managed in xml using the lxml library, and the program is frozen as an executable using pyinstaller, so it should run on any modern computer without any external dependencies. [10] while the technology used to develop ants is not very fast nor resource-efficient, the application manages and stores only a small amount of data and only performs a few simple tasks. these tools also make ants very open and easy to develop, and since it runs on the desktop, efficiency and scalability are much smaller concerns. while it might not be the most popular gui framework, wxpython is mature, easy to use, and is open and well-documented. the code below places a text label, a one-line text box where users can enter a path, and a button which simply checks if that path is a valid location. this is a simplified example of some of the code in the ants options panel and serves to show how easy desktop development can be. #sizer that positions widgets gridsizer = wx.flexgridsizer( 4, 3, 0, 0 ) gridsizer.setflexibledirection( wx.both ) gridsizer.setnonflexiblegrowmode( wx.flex_growmode_specified ) #label for textctrl self.tranferlabel = wx.statictext( self.transfertab, wx.id_any, u"transfer location (path or ftp url)", wx.defaultposition, wx.defaultsize, 0 ) self.tranferlabel.wrap( -1 ) gridsizer.add( self.tranferlabel, 0, wx.all, 5 ) #textctrl where user enters transfer path self.transferlocinput = wx.textctrl( self.transfertab, wx.id_any, configdata["transferlocation"], wx.defaultposition, wx.size( 300,-1 ), 0 ) gridsizer.add( self.transferlocinput, 0, wx.all, 5 ) #button to test if transfer path is valid self.checklocation = wx.button( self.transfertab, wx.id_any, u"test transfer location", wx.defaultposition, wx.defaultsize, 0 ) gridsizer.add( self.checklocation, 0, wx.all, 5 ) #binds button to function self.checklocation.bind( wx.evt_button, self.testtransferlocation ) #function to test path on button click def testtransferlocation(self, event): #gets content of textctrl location = self.transferlocinput.getvalue() #check to see if textctrl contains value if not len(location) > 0: #shows popup dialog noloc = wx.messagedialog(none, 'you must enter a local or network path.', 'location test error', wx.ok | wx.icon_error) noloc.showmodal() else: #example of testing a path if os.path.isdir(location): #shows popup dialog gooddir = wx.messagedialog(none, 'the directory you entered is correct.', 'found directory', wx.ok | wx.icon_information) gooddir.showmodal() else: #shows popup dialog baddir = wx.messagedialog(none, 'invalid location, did not find the directory you entered.', 'incorrect directory', wx.ok | wx.icon_exclamation) baddir.showmodal() complex positioning using sizers can be unintuitive, but wxformbuilder is an open-source application that can easily position widgets with a gui and export the code. there are some limitations to wxformbuilder, as it does not allow you to directly bind functions to widgets, but the code is easy to modify after it is exported. figure 1. ants splash page when users run ants, they are shown an entry window that looks like a splash screen. this asks them to browse and select a folder to transfer while letting them directly access ants’s other functions. using an entry window like this is not ideal, but in this case the user needs a “browse” button, otherwise dropping them directly into a directory select dialog is disorienting. functionally, ants performs three primary tasks. most importantly, it allows users to select, describe, package, and transfer directories and files. ants also maintains a running receipt of transferred files that will let users easily request files they transferred in the past. clicking this button creates a bootstrap html report with links that lets donors email the archivist with a specific file identifier, accession number, and transfer information. for ualbany, this is configured to send email requests directly into our ticket system. finally, ants allows users to view and download files through the same method they transferred them. [11] when users click browse, ants asks to elevate the user account control (uac) privileges that will help it run forensics tools later. windows does not easily allow programs to elevate privileges within a single process, so after the elevation request ants actually runs a separate executable, antsfromboot.exe, which contains the major portion of the tool, and can test for elevated privileges. this enables ants to run different tools depending on the privileges that it has been granted. figure 2. ants file browsing after users browse and select a directory, the main panel opens with an expanded directory tree that displays every file and folder. checkboxes let users select which files and folders to transfer, and which they would like to omit. this encourages records creators to select files in their native environment without unnecessarily transferring them to avoid neighboring files. the most difficult part of the gui development was that wxpython’s default treectrl widget does not contain checkboxes. instead, ants uses a customtreectrl widget which is part of an extension library written by andrea gavana. the issue with this extension is that the checkbox rules that automatically toggle parents and children did not work as stated in the documentation. the solution was to write the directory information into an xml file at the same time the widget is displayed. each file or folder is bound to an event that enters checkbox information into the xml file and reloads the widget. the rules for checking parents and children had to be written manually. although tedious to develop, this works rather well and allows ants to add the description and access information to the xml file during the same event, but may cause scalability issues for transfers with hundreds of subdirectories and/or thousands of files. the right side of the panel invites users to add descriptions and access concerns at any level of the directory. while it is unlikely that users will describe their files manually, ants does provide this functionality. archivists might ask donors to provide a little information about files that may not be effectively named, and in a few cases they could provide some useful information. the bottom-right of the panel is a submit button and an option to compress the data before transferring. once ants is configured, users can transfer records simply by selecting a directory and clicking “submit.” additional notebook tabs enable users to configure ants through the gui. the “creator profile” tab sets the donor metadata that will be included with the packaged files. this includes the creator, a creator id, the donor and the donor’s contact information such as an email address that will let archivists notify donors of a successful transfer. the “options” tab contains the transfer and receive locations, credentials for ftp transfers, and timestamp, compression, and checksum options. here users can also export the receipt data in bootstrap html, csv, or xml. all of this information is stored in the user’s appdata directory in a config.xml file. ants is also designed to be configured remotely, so that archivists can send a customized config.xml file that can be installed with the application. figure 3. ants options tab when ants is set up using the installer that is provided, there is an option to place a shortcut to ants in the windows context menu – the “right-click” menu. this allows records creators to simply right click any file or folder in windows explorer, omit the splash screen, and transfer records in two clicks. figure 4. ants context menu packaging files with digital forensic tools a number of free and open-source forensic tools have emerged in recent years and most of them are distributed with permissible licenses. this permits us to use python’s subprocess library to run these tools from within ants and parse their output. right now, there are two tools that we can utilize to perform digital forensics at accession: joakim schicht’s mftrcrd and the plaso timeline engine developed by kristinn gudjonsson and joachim metz. there is also hope that these and other tools will continue to evolve where they can further support archivists and gather more record-events at accession. forensics tools are designed for digital forensics investigations not archives. in most cases, tools choose to extract large bodies of contextual metadata from disk images, not—as we require—to gather metadata on individual files. this means that most forensic tools available today often take too long to gather record-events for individual files. a second problem is that we need records creators to run these programs on their own computers, so they need to be able to access live filesystems instead of disk images. this is possible on modern windows operating systems, by accessing a disk as a device rather than through the filesystem. to do this we just need to use the device namespace in the command line with the \\.\ prefix. so the c drive would be accessed as \\.\c: instead of c:\. [12] we can use this prefix as part of a command wrapped in a python application using the subprocess module. the only problem with this is that windows requires elevated privileges to access disks as devices, privileges that many records creators may not have. this is the reason why ants requests elevated privileges before browsing for a directory, though even privileged users will not have privileges for shared network drives, likely leaving us with only a portion of our records creators that can use this method. with these restrictions in mind, ants must be able to run different tools based on the privileges it has been granted. the most common open-source forensic tools currently used by archivists is the sleuth kit (tsk), written by brian carrier. tsk is a set of over 20 command line tools used to analyze disk images. tsk can access live systems, but only with administrator privileges. tsk metadata layer tools (iprefix) at first seem to be particularly useful to us. ifind.exe produces the master file table identifier for a file and istat.exe displays that master file table (mft) record. the mft is where most filesystem metadata is found in ntfs systems, and contains a file’s mac times or mace times—for modified, accessed, and created—these primary timestamps are called the $standard_information attribute on ntfs systems. in addition to the primary $standard_information timestamps, the mft record also contains at least one other additional set of mac times, the $file_name attribute, that are typically hidden from the common user and are not as volatile. according to corey altheide and harlam carvey, the $file_name timestamps, “…are not affected by normal system activity or malicious tampering…” and are generally more stable than the $standard_information mac times. [13] brian carrier goes even further to claim that the $file_name timestamps, “…frequently correspond to when the file was created, moved, or renamed.” [14] in addition, the $file_name attribute also identifies its parent directory, and that directory will also have a set of mac timestamps for its child files. if nothing else, these different sets of timestamps can support or complicate the standard mac times, and together they can help future archivists and researchers better understand how records were used. tsk’s ifind.exe’s speed was a concern for this project. in basic, unscientific tests, ifind.exe took about 13 seconds per file to obtain the mft identification number. running as a subprocess within a python script, looping through 25 files took about 334 seconds, or just over five and a half minutes. for comparison’s sake, using bagit to run md5 checksums on those files—taking up a fairly small 178 megabytes—takes less than a third of a second. users will not expect ants to take over five minutes to package this small number of files, and that time does not include transferring that data. fundamentally, tsk’s ifind takes too long to identify mft identifiers for individual files, so we sought a better option. at first glance, joakim schicht’s mft2csv tool appeared to meet our needs, but instead we encountered similar issues – namely that forensic tools are often focused on the disk level. as the name states, mft2csv decodes the mft of a ntfs disk image or live filesystem to a csv file. users have a wealth of options and may employ mft2csv with a gui or via the command line. the problem with this tool is that we would have to dump a disk’s entire mft to a csv or sql file before parsing it for individual records. this is the same for similar tools such as pymftgrabber, analyzemft.py, and indxparse. while this method would not be difficult to write, for large disks it would take up additional resources and time while records creators are making the transfer. schicht has developed another tool that better fits our needs. mftrcrd can extract an individual file’s mft entry from a live system by providing both that file’s path and the device namespace method discussed previously. using the same methods we used to test tsk tools, mftrcrd ran at about 0.7-0.8 seconds per file to total just over 19 seconds to loop though the entire directory of 25 files. starting mftrcrd by joakim schicht version 1.0.0.37 target is a file filesystem on c: is ntfs file indexnumber: 418561 bytespersector: 512 sectorspercluster: 8 reservedsectors: 0 sectorspertrack: 63 numberofheads: 255 hiddensectors: 206848 totalsectors: 976564223 logicalclusternumberforthefilemft: 786432 logicalclusternumberforthefilemftmirr: 16 mft record size: 1024 record number: 418561 found at disk offset: 0x0000001bc8e89400 $logfile sequence number (lsn): 13711955478 ... $standard_information 1: file create time (ctime): 2015-10-20 13:40:47:010:6144 file modified time (atime): 2015-10-20 13:57:12:762:9689 mft entry modified time (mtime): 2015-10-20 13:57:12:762:9689 file last access time (rtime): 2015-10-20 13:40:47:027:6178 dos file permissions: archive max versions: 0 version number: 0 class id: 0 owner id: 0 security id: 2866 quota charged: 0 usn: 3219407008 $file_name 1: parent mftreference: 397315 parentsequenceno: 7 file create time (ctime): 2015-10-20 13:40:47:010:6144 file modified time (atime): 2015-10-20 13:40:47:027:6178 mft entry modified time (mtime): 2015-10-20 13:40:47:027:6178 file last access time (rtime): 2015-10-20 13:40:47:027:6178 allocsize: 0 realsize: 0 easize: 0 flags: archive namelength: 8 nametype: dos+win32 namespace: 3 filename: test2.py condensed example of mftrcrd’s output the output includes some filesystem information such as the type of the filesystem and the sizes and number of sectors and clusters. more importantly, it includes the four primary $standard_information timestamps and the four additional $file_name timestamps. also included is the file’s $logfile sequence number and the update sequence number (usn) journal number – both of which may be useful in the future to gather record-events from files that have been more recently accessed. while this output is unstructured text, it was fairly easy to run it as a subprocess like the example below, parse it with python, and add the timestamps to ants’s xml directory file. readmft = subprocess.popen("mftrcrd.exe " + path + "-d indxdump=off 1024 -s",\ shell=false, stdin=subprocess.pipe, stderr=subprocess.pipe, stdout=subprocess.pipe) out, err = readmft.communicate() using the mftrcrd tool is currently the most effective method to gather contextual metadata at accession, but the plaso timeline engine may be a more intriguing tool for the future. unlike mftrcrd, it can be run without the elevated privileges that many records creators may not have. here the plaso engine may offer a helpful backup plan by enabling us to gather at least the primary mac timestamps. plaso is different than the forensic tools discussed previously as it is more user-focused – designed more for ease of use and at-a-glance information. forensics tools need to be effective and comprehensive, yet plaso is not designed to gather all forensic artifacts, but to gather the most readily available evidence of activities and display them in a useful and intuitive timeline. plaso has evolved from a single perl tool, called log2timeline, to an “engine” that amalgamates a number of useful forensic processes to produce “super timelines.” this is particularly intriguing for archivists who would benefit from using a number of smaller and more obscure tools and producing a simple and standardized output format. plaso’s functionality matches the needs of institutional archivists by gathering a variety of record-events from different sources in a simple and user-friendly manner. however, for archivists plaso offers more hope for future gains than current productivity. it does not access live disks as devices, so the only way it can run on live systems is to read a directory and gather the $standard_information mac times that are available. while it still gathers artifacts from sources other than the filesystem, in practice this means that in rare cases it may be able to gather record-events embedded within a file itself – most often with microsoft’s office open xml files (.docx, xlsx, etc.). additionally, plaso does take a significant amount of time to run through a directory at just over 23 seconds for 25 files. this is not much more than mftrcrd, yet since it is run with a single process it was a challenge showing progress to the user. running plaso, ants’s progress bar dialog freezes for that 23 seconds, giving users the appearance that ants itself is frozen. it is possible, although challenging, to show progress as plaso updates the console, but plaso does not provide feedback to the console during most of its run, so updates to the progress bar dialog would likely be intermittent at best. for uses without elevated privileges, ants can employ both plaso and python’s os.stat library. unlike plaso, os.stat runs nearly instantaneously and can show progress for each file. it gathers the basic $standard_information mac times that are available to the operating system. this makes it a realistic minimum for performing digital forensics at accession. plaso is still included with ants as a proof of concept and a tool for future experimentation. it runs effectively, yet for now there seems to be little tangible benefit to using plaso over os.stat. transferring records to the archives after employing basic forensic tools to read filesystem metadata, ants runs the python version of bagit to package files with checksums. an unexpected problem was that using the existing bagit tool required ants to move and hash the files twice – a redundant and time-consuming step for large transfers. in order to prevent the unwitting modification of the original files, ants makes a copy of the entire transfer directory after running the forensics tools. ants then runs bagit, which again moves the directory to a “data” folder according to the bagit standard. ants later adds an xml metadata file to the bag, which requires updating the manifest to make it valid again – running checksums on the data for a second time. this redundancy could be eliminated by creating the bagit package before transferring the files and running the checksums, but creating an empty bag raises an error. fixing this issue would require reworking the python bagit tool or building the bag framework from scratch, and the latter option will probably be implemented in future versions. another improvement that may be made in the future is the use of robocopy. currently, ants uses python’s native shutil library to move files, which is significantly slower than native windows utilities. ants could instead run robocopy as a subprocess, which would greatly improve performance, but in practice, this caused similar issues as running plaso. robocopy provides a large amount of feedback in unstructured text though both the console and a log file. this would have to be parsed to show progress to the user and raise an exception if the process failed. after the records are packaged, ants creates an xml directory of the folder and file hierarchy that includes identifiers, paths, and any descriptions and access concerns that were entered by the user. here ants adds all of the record-events it was able to gather from the files, and logs all of the actions it took on the files as curatorial events. if a user opted to compress the accession, ants then compresses the directory either in zip or gzip formats. ants does not use major metadata standards for its output. after initial plans to incorporate premis or mets, we found that the information provided by ants was different enough that mapping to these standards would be very imprecise and serve only to add unnecessary complexity. the xml provided by ants also lacks namespaces or a stated schema. namespaces can be difficult to deal with in lxml, and since ants creates standardized xml, a schema seemed unnecessary. archivists are welcome to map the data provided by ants to their favorite standards upon receipt, and omitting namespaces and schemas makes this easier to do. here is an excerpt from ants’s output that depicts a single file: <file name="dutchesscountyre00cook.pdf"> <id>418f3945-d66d-494a-a352-146ba45e0053</id> <path>f:\libraries\documents\dchs\dutchesscountyre00cook.pdf</path> <description>listing from promotional event.</description> <access/> <curatorialevents> <event timestamp="2015-12-19 15:17:16">ran mftrcrd to gather ntfs timestamps</event> <event timestamp="2015-12-19 15:17:20">ran bagit-python to package accession</event> <event timestamp="2015-12-19 15:17:20">ftp transfer</event> </curatorialevents> <recordevents> <timestamp source="ntfs" timetype="local+4:00" parser="mftrcrd" type="standard_information" label="file_create_time">2015-11-07 17:54:09</timestamp> <timestamp source="ntfs" timetype="local+4:00" parser="mftrcrd" type="standard_information" label="file_modified_time">2013-07-13 11:50:48</timestamp> <timestamp source="ntfs" timetype="local+4:00" parser="mftrcrd" type="standard_information" label="mft_entry_modified_time">2015-11-07 17:54:10</timestamp> <timestamp source="ntfs" timetype="local+4:00" parser="mftrcrd" type="standard_information" label="file_last_access_time">2015-11-07 17:54:09</timestamp> <timestamp source="ntfs" timetype="local+4:00" parser="mftrcrd" type="file_name" label="file_create_time">2015-11-07 17:54:09</timestamp> <timestamp source="ntfs" timetype="local+4:00" parser="mftrcrd" type="file_name" label="file_modified_time">2015-11-07 17:54:09</timestamp> <timestamp source="ntfs" timetype="local+4:00" parser="mftrcrd" type="file_name" label="mft_entry_modified_time">2015-11-07 17:54:09</timestamp> <timestamp source="ntfs" timetype="local+4:00" parser="mftrcrd" type="file_name" label="file_last_access_time">2015-11-07 17:54:09</timestamp> </recordevents> </file> ideally, ants is designed to transfer files over shared network storage. the transfer directories do not need to be mapped to a drive letter, and users may not even be aware that the directory exists – ants just requires a local or unc path. it is expected that archivists will remove accessions from the transfer directory and not store them there. this can be automated with a simple script that can also send a notification email drawn from the metadata included with the files. by transferring records over network shares, users are authenticated by their windows login. the advantage of this method is that it leverages network protocols which may already be in use by an organization, such as window’s active directory or other forms of lightweight directory access protocol (ldap). ants can also serve as an ftp client using individual user credentials. the problem here is coordinating credentials between ants and the ftp server. credentials are stored in the user’s appdata directory, encrypted with cryptprotectdata from the windows api using python’s win32 library. while this is similar to how many applications store user credentials, it is not very secure, and it still requires manual entry of credentials into the ftp server. conclusion employing digital forensics at accession is and will always be an imperfect and challenging method of managing digital records. unlike disk imaging, employing these tools at accession requires proactive planning and the preemptive use of resources. if we do not use disk imaging, all contextual metadata outside of file content must be gathered in advance – actions that must be performed by the records creators themselves in records’ native environment. yet, institutional archivists may choose not to accept disk images for the problems they pose to records retention schedules and public records laws. here, gathering record-events at accession may be the most effective way to maintain some basic context without retaining additional information that may conflict with retention scheduling. while we have to employ digital forensic at accession under a number of constraints, forensic tools have developed to a point where it is now a feasible option. ants should act as proof of this viability and serve as a tool for future experimentation. right now we are able to only gather some of the many possible record-events that may exist in the form of filesystem timestamps. still, each timestamp documents a point of action that may help tell a more complete story. every little bit helps, and if we can gather even this basic information systematically with limited barriers for records creators, the cost of digital forensics at accession can be small enough that archives can significantly benefit. there is always the potential for future growth that may enable us to gather more information at accession. digital forensic tools are evolving – getting more effective and easier to use. there is a strong potential for performing more sophisticated forensic analysis at accession, particularly for records creators who have administrative privileges. digital forensics at accession will never be able to gather all of the contextual information that disk imaging can, yet it can preserve the record-events that are most likely to be useful to future archivists and researchers. with reasonable future advances in tools, gathering record-events at accession can empower archivists to appraise and preserve the contextual information with the greatest value and minimum risk. for many repositories, particularly institutional archives which manage public records in a relatively controlled environment, employing digital forensics at accession can be viable and effective approach to managing digital records. about the author gregory wiedeman is the university archivist in the m.e. grenander department of special collections & archives at the university at albany, suny where he is charged with developing a digital records program. notes [1] interpares project authenticity task force, “requirements for assessing and maintaining the authenticity of electronic records,” march 2002. [2] matthew g. kirschenbaum, richard ovenden, and gabriela redwine, digital forensics and born-digital content in cultural heritage collections, (washington, d.c.: council on library and information resources, 2010): 8, 21, 32. [3] kam woods and christopher a. lee, “acquisition and processing of disk images to further archival goals,” in proceedings of archiving 2012 (springfield, va: society for imaging science and technology, 2012), 148. & kam woods, christopher a. lee, and simson garfinkel, “extending digital repository architectures to support disk image preservation and access,” in jcdl ’11: proceeding of the 11th annual international acm/ieee joint conference on digital libraries, (new york, ny: acm press, 2011). [4] christopher a. lee, kam woods, matthew kirschenbaum, and alexandria chassanoff, “from bitstreams to heritage: putting digital forensics into practice in collecting institutions,” september 30, 2013. [5] ben goldman, “bridging the gap: taking practical steps toward managing born-digital collections in manuscript repositories,” rbm: a journal of rare books, manuscripts, and cultural heritage 12:1 (2011) & aims work group, aims born-digital collections: an inter-institutional model for stewardship, january 2012 & ricky erway, you’ve got to walk before you can run: first steps for managing born-digital content received on physical media, (dublin, ohio: oclc research, august 2012) & cyndi shein, “from accession to access: a born-digital materials case study,” journal of western archives 5:1 (2014) & joseph a. williams and elizabeth m. berilla, “minutes, migration, and migraines: establishing a digital archives at a small institution,” the american archivist 78:1 (2015). [6] julianna barrera-gomez and ricky erway, walk this way: detailed steps for transferring born-digital content from media you can read in-house, (dublin, ohio: oclc research, june 2013) & sam meister and alexandria chassanoff, “integrating digital forensics techniques into curatorial tasks: a case study,” the international journal of digital curation 9:2 (2014) & john durno and jerry trofimchuk, “digital forensics on a shoestring: a case study from the university of victoria,” code4lib journal 27 (january 21, 2015). [7] the national archives’ capstone email model comes to mind as an example. national archives and records administration, “email management,” retrieved from https://www.archives.gov/records-mgmt/email-mgmt.html. [8] new york state’s freedom of information law states definitively that, “…government is the public’s business and that the public, individually and collectively and represented by a free press, should have access to the records of government…” new york (state) legislature, “freedom of information law,” public officers law article 6 (s 84-90), 2008. exceptions to foil requests include records that, if made available, “…would constitute an unwarranted invasion of personal privacy…,” impede collective bargaining processes, contain certain trade secrets, or “…interfere with law enforcement investigations or judicial proceedings.” all other records must be made “available for public inspection and copying.” [9] lee, woods, kirschenbaum, and chassanoff, 5-6. [10] ants was developed primarily for windows and has only been tested on windows 7 and later. the pyinstaller executable should run on mac osx and linux machines, yet ants is only written to read ntfs filesystems and non-windows users are warned of this when running ants. [11] ants envisions that records creators will have both a transfer and a request directory. it is designed that after a request is make, through the receipt or otherwise, an archivist can place the requested materials in the request directory temporarily and creators can download them through ants using either shared network folders or ftp. the transfer process is detailed further in the article. [12] pär österberg medina, “how to acquire ‘locked’ files from a running windows system,” open security research blog, october 25, 2011, retrieved from http://blog.opensecurityresearch.com/2011/10/how-to-acquire-locked-files-from.html. [13] cory altheide and harlan carvey, digital forensics with open source tools (walthan, ma: syngress, 2011), 73-74. [14] brian carrier, file system forensic analysis (addison-wesley: upper saddle river, nj, 2005), 318. bibliography aims work group. aims born-digital collections: an inter-institutional model for stewardship. january 2012. retrieved from http://dcs.library.virginia.edu/aims/white-paper/. altheide, cory and harlan carvey. digital forensics with open source tools. walthan, ma: syngress, 2011. barrera-gomez, julianna, and ricky erway. walk this way: detailed steps for transferring born-digital content from media you can read in-house. dublin, ohio: oclc research, june 2013. retrieved from http://www.oclc.org/content/dam/research/publications/library/2013/2013-02.pdf. carrier, brian. file system forensic analysis. addison-wesley: upper saddle river, nj, 2005. durno, john, and jerry trofimchuk. “digital forensics on a shoestring: a case study from the university of victoria.” code4lib journal 27 (january 21, 2015). retrieved from http://journal.code4lib.org/articles/10279. erway, ricky. you’ve got to walk before you can run: first steps for managing born-digital content received on physical media. dublin, ohio: oclc research, august 2012. retrieved from http://www.oclc.org/content/dam/research/publications/library/2012/2012-06.pdf. gengenbach, martin, alexandria chassanoff, and porter olsen. “integrating digital forensics into born-digital workflows: the bitcurator project.” proceedings of the american society for information science and technology 49:1 (2012). retrieved from https://www.asis.org/asist2012/proceedings/submissions/343.pdf. goldman, ben. “bridging the gap: taking practical steps toward managing born-digital collections in manuscript repositories.” rbm: a journal of rare books, manuscripts, and cultural heritage 12:1 (2011). gudjonsson, kristinn. “mastering the super timeline with log2timeline.” sans institute information security reading room. june 29, 2010. retrieved from https://www.sans.org/reading-room/whitepapers/logging/mastering-super-timeline-log2timeline-33438. interpares project authenticity task force. “requirements for assessing and maintaining the authenticity of electronic records.” interpares project. march 2002. retrieved from http://www.interpares.org/book/interpares_book_k_app02.pdf. kirschenbaum, matthew g., richard ovenden, and gabriela redwine. digital forensics and born-digital content in cultural heritage collections. washington, d.c.: council on library and information resources, 2010. retrieved from http://www.clir.org/pubs/reports/reports/pub149/pub149.pdf. lee, christopher a. “archival application of digital forensics methods for authenticity, description and access provision.” comma 2:14 (2012). retrieved from http://ils.unc.edu/callee/p133-lee.pdf. lee, christopher a. “digital forensics meets the archivist (and they seem to like each other).” provenance 30 (2012). retrieved from http://digitalcommons.kennesaw.edu/cgi/viewcontent.cgi?article=1023&context=provenance. lee, christopher a., kam woods, matthew kirschenbaum, and alexandria chassanoff. “from bitstreams to heritage: putting digital forensics into practice in collecting institutions.” september 30, 2013. retrieved from http://www.bitcurator.net/docs/bitstreams-to-heritage.pdf. meister, sam and alexandria chassanoff. “integrating digital forensics techniques into curatorial tasks: a case study.” the international journal of digital curation 9:2 (2014). retrieved from http://www.ijdc.net/index.php/ijdc/article/view/325. national archives and records administration. “email management.” retrieved from https://www.archives.gov/records-mgmt/email-mgmt.html. new york (state) legislature. “freedom of information law.” public officers law article 6 (s 84-90), 2008. retrieved from http://www.dos.ny.gov/coog/foil2.html. medina, pär österberg . “how to acquire ‘locked’ files from a running windows system.” open security research blog october 25, 2011. retrieved from http://blog.opensecurityresearch.com/2011/10/how-to-acquire-locked-files-from.html. shein, cyndi. “from accession to access: a born-digital materials case study.” journal of western archives 5:1 (2014). retrieved from http://digitalcommons.usu.edu/westernarchives/vol5/iss1/1/. williams, joseph a., and elizabeth m. berilla. “minutes, migration, and migraines: establishing a digital archives at a small institution.” the american archivist 78:1 (2015). woods, kam and christopher a. lee. “acquisition and processing of disk images to further archival goals.” proceedings of archiving 2012. springfield, va: society for imaging science and technology, 2012. retrieved from http://ils.unc.edu/callee/p147-woods.pdf. woods, kam, christopher a. lee, and simson garfinkel. “extending digital repository architectures to support disk image preservation and access.” jcdl ’11: proceeding of the 11th annual international acm/ieee joint conference on digital libraries. new york, ny: acm press, 2011. retrieved from http://ils.unc.edu/callee/p57-woods.pdf. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – new metadata recipes for old cookbooks: creating and analyzing a digital collection using the hathitrust research center portal mission editorial committee process and structure code4lib issue 37, 2017-07-18 new metadata recipes for old cookbooks: creating and analyzing a digital collection using the hathitrust research center portal the early american cookbooks digital project is a case study in analyzing collections as data using hathitrust and the hathitrust research center (htrc) portal. the purposes of the project are to create a freely available, searchable collection of full-text early american cookbooks within the hathitrust digital library, to offer an overview of the scope and contents of the collection, and to analyze trends and patterns in the metadata and the full text of the collection. the digital project has two basic components: a collection of 1450 full-text cookbooks published in the united states between 1800 and 1920 and a website to present a guide to the collection and the results of the analysis. this article will focus on the workflow for analyzing the metadata and the full-text of the collection. the workflow will cover: 1) creating a searchable public collection of full-text titles within the hathitrust digital library and uploading it to the htrc portal, 2) analyzing and visualizing legacy marc data for the collection using marcedit, openrefine and tableau, and 3) using the text analysis tools in the htrc portal to look for trends and patterns in the full text of the collection. by gioia stevens introduction the early american cookbooks project is a case study in analyzing digital special collections as data by using new tools to explore trends and patterns in the metadata and the full text of a collection. the project is a useful example for librarians, technologists and researchers interested in exploring the wealth of data in the public domain corpus of hathitrust and the computational tools available in the hathitrust research center (htrc) portal. it is also a useful example of how analyze and visualize sets of legacy marc catalog records and offer users new insights into the depth and breadth of library collections. the project workflow described in this article includes: (1) creating a searchable public collection of full-text titles within the hathitrust digital library and uploading it to the htrc portal, (2) analyzing and visualizing legacy marc data for the collection using marcedit, openrefine and tableau, and (3) using the text analysis tools in the htrc portal to look for trends and patterns in the full text of the collection. building a collection within hathitrust is a way to create a searchable digital collection for a topic of particular research interest or a specialized subject area with few digital resources (such as early american cookbooks). it can also be used to create a digital collection to mirror the print holdings available in a particular library. the full text of collections is keyword searchable independently of the full hathitrust library and basic metadata for search results can be downloaded to create new subgroups of titles. analyzing collections as data reveals valuable new uses for old catalog records. however, working with legacy marc records from many different libraries over time presents complex challenges. this project developed a workflow using marcedit, openrefine and tableau to transform, clean, and visualize marc records. when this legacy data is cleaned up and presented as a dataset rather than as individual records, it can yield new information about a collection in aggregate. metadata about authors, publishers, places of publication, dates, and subject content can offer insights into a collection and its significance within a broader historical scope. this information can be visualized and presented to users as a new way to gain understanding of the depth and breadth of a collection. the htrc portal offers a convenient platform for importing a hathitrust collection and then analyzing that collection using text analysis algorithms. these algorithms are built into the htrc portal as ready-made tools that can be run on a collection within the portal. no coding skills are required and the portal is a useful way to begin experimenting with computational methods. topic modeling using the meandre topic modeling algorithm is “a method for finding and tracing clusters of words (called “topics” in shorthand) in large bodies of texts.” [1] these topics show trends and patterns in the contents of a collection and can serve as an overview of its thematic content. comparing two sets of text using the meandre dunning log-likelihood to tag cloud algorithm is a useful way to trace differences in subject matter by sorting the under-represented terms and over-represented terms in each set. these results can be used to highlight how a particular subgroup of titles differs from the collection as a whole. early american cookbooks project overview the idea for this project grew out of cataloging hundreds of early print cookbooks for the marion nestle food studies collection at the fales library & special collections at new york university libraries. most early cookbooks in printed form are accessible only in special collections libraries or private collections. these books are an incredibly important resource for food historians or anyone with a passion for old cookbooks, but there are very few searchable full text resources available online. the feeding america digital archive created by michigan state university libraries is the largest digital collection, but it contains only seventy-six full-text titles. the early american cookbooks project was designed to meet this need. the purposes of the project are to create a freely available, searchable online collection of early american cookbooks, to offer an overview of the scope and contents of the collection, and to analyze trends and patterns in the metadata and the full text of the collection. the project has two basic components: a collection of full-text titles within the hathitrust digital library and a website to present a guide to the collection and the results of the analysis. early american cookbooks collection the early american cookbooks collection is a separate public collection within the hathitrust digital library which was created specifically for this project. it contains 1450 full text cookbooks published in the united states between 1800 and 1920. users can browse the titles, read them online, and search the contents of each book. the collection as a whole can be searched independently of the rest of the hathitrust. keyword searching across all 1450 titles allows the user to find particular recipes, ingredient names, or anything else and go immediately to the results in the full text. this type of search would be very valuable for a food historian tracing something such as the history of the hamburger or when americans first started eating spaghetti. early american cookbooks website the early american cookbooks website serves as a gateway and guide to the collection. the site gives a general introduction to the history of american cookbooks, an overview of the scope and contents of the collection, a discussion of certain interesting books or groups of titles within the collection, and links to other online resources and library collections specializing in early american cookbooks. the site presents and interprets the results of explorations in the metadata and the full text of the collection. data visualization of the results shows trends and patterns in the collection that may add to our understanding of early cookbooks and the history of american food. building the collection on hathitrust and uploading worksets to htrc workflow overview: create public or private collection in hathitrust from full-text search results evaluate search results and edit collection contents check and de-dupe individual records (multiple printings, editions, and scans) download basic collection metadata and upload volume id numbers to create a workset in htrc hathitrust collections are a way to group and save a selection of titles for public or private use. collections are searchable independently of the full hathitrust library and are a convenient way to assemble and research a subgroup of hathitrust materials. collections may be created by any member of a hathitrust partner institution or by anyone who chooses to sign up for a university of michigan “friend account” login. some examples of recent public collections include the english short title catalog, islamic manuscripts at the university of michigan, patent indexes, and action and adventure fiction. items can be added to a collection from the full-text search results (not the catalog search results) or from the page-turner interface when looking at an individual full-text title. members of hathitrust partner institutions also have the option to limit the full text search results to titles held at their own libraries. this feature could be useful for a library wishing to create a digital collection to mirror specific print titles in their holdings. creating a collection involves evaluating and manually de-duplicating the record set. using precise search terms and carefully evaluating the results are important to avoid adding irrelevant titles to a collection (such as books on captain cook or cook county, illinois, in a collection of cookbooks). search results frequently contain duplicate records from different libraries and many records offer multiple scans from different libraries. it can be quite complex to compare many similar records and determine which are duplicates and which represent different editions or printings of a book. for the early american cookbooks collection, evaluating the initial search results and distinguishing and de-duping multiple printings, editions, and scans reduced the collection from over 2000 records to a final set of 1450. the next step is to download the metadata for the collection. this metadata is includes the following items in a tab-delimited text file: htitem_id – the hathitrust item identifier which is used to uniquely identify every hathitrust digital item title author date rights – the copyright status for this item as determined by hathitrust a series of identifiers commonly used by libraries: oclc, lccn, isbn catalog_url – the url for the catalog record with which the item in question is associated handle_url – the permanent url for the hathitrust digital item the full marc records for the collection are available from the htrc, the research arm of hathitrust. the htrc portal “provides an infrastructure to search, collect, analyze, and visualize the full text of nearly 3 million public domain works and is intended for nonprofit and educational researchers.” [2] anyone possessing an email address from a nonprofit institution of higher education is allowed to register for a user account, including those whose institutions are not hathitrust members. the htitem_id identifiers (also called volume id numbers) downloaded from the hathitrust collection can be uploaded to the htrc portal to create a new workset. the htrc portal also provides a workset builder tool for users who do not have a collection within hathitrust and wish to create and analyze a workset within the portal. figure 1. hathitrust research center home page. analyzing collection metadata with marcedit, openrefine and tableau workflow overview: download records in marcxml format using htrc marc downloader tool convert marcxml to marc21 and join records into one file using marcedit export master metadata spreadsheet as csv file using marcedit clean up metadata using openrefine create topical subsets of records using openrefine upload csv files to tableau to explore data produce visualizations in tableau it is important to note that this workflow could be done with any set of marc records, whether downloaded from hathitrust or from another source. for hathitrust records, the htrc portal offers the marc downloader tool to provide individual catalog records in marcxml format. legacy marc data for early books held in special collections presents particular challenges. for many 19th century books, such as the ones in the early american cookbooks collection, these records are idiosyncratic, legacy data, much of it drawn from old catalog cards. many of these records contain archaic information such as “brooklyn, opposite new york” rather than “brooklyn, n.y.” as the place of publication. these records also contain many different forms of abbreviation, punctuation, and terminology, resulting from different cataloging standards over time and also the local practices at individual libraries. cleaning and standardizing this legacy data is an essential step in analyzing special collections metadata as a dataset rather than as individual records. marcedit workflow marcedit is an open source gui-based application for editing and manipulating marc data. it was developed by terry reese beginning in 1999 and has become “one of the more complete metadata edit suites available to librarians.” [3] this workflow uses three different utilities in marcedit which are all available in the same utilities dialog window. the first step in the workflow is to convert the individual marcxml records from hathitrust to marc records using the batch process marc records utility. the next step is to join the individual records into one record using marcjoin. the final step is to create a master spreadsheet for the metadata using the export tab delimited records utility. this utility will export selected marc fields and subfields in .csv format. figure 2. marcedit’s utilities dialog window showing the batch process marc records utility openrefine workflow the next step is to upload the master metadata spreadsheet into openrefine. openrefine, described as a “power tool for working with messy metadata” [4] is a very useful open source program for working with legacy marc records. the program functions to explore data, clean and transform data, and reconcile and match data. for example, it was easy to find all the different forms of an author’s name (example: fanny farmer, fannie farmer, fannie merritt farmer), group them together and use one transformation to change all of them to the authorized version. dates and many different ways of recording dates (example: 1882, [1882], [1882?], ©1882) posed particular challenges. openrefine made it possible to group and reconcile these dates and then transform the data from number format into date format. the screenshot below shows the original date in the 260$c column and the transformed date in the edited date column. this date cleanup work was necessary to be able to sort the titles by date and then visualize the collection metadata along a timeline. figure 3. collection data in openrefine openrefine is also useful for the creation of additional metadata and sorting records into subgroups. this process was useful for preparing the data for visualization in tableau and also for creating topical subsets of titles (examples: all books by a particular author, southern cookbooks). in the screenshot above, the state name column was created based on the place of publication in the 260$a field. this involved grouping and transforming numerous different forms of state abbreviations (example: mass., ma, m.a.). it was also necessary to add the state when only the city name appeared in 260$a. sometimes this was obvious (example: boston) and sometimes it required more examination of the marc record or the full text of the book (example: springfield). the full state name, rather than an abbreviation, proved to be an essential piece of metadata for creating map visualizations in tableau. the region column was created based off of the state name and the united states census regions. dividing the collection into different worksets for the northeast, midwest, west, and southeast regions allowed for regional map visualizations and text analysis explorations to look for regional variations in recipes. tableau visualizations the next step in the workflow is to upload the metadata spreadsheet into tableau, a data visualization software program which is available in both licensed versions and in a free online version called tableau public. tableau offers “drag and drop visual analytics” [5] designed for users without prior experience in data analysis. using tableau to experiment with graphing metadata is an interesting way to explore a collection and think about marc records in aggregate as a dataset. tableau offers a wide range of visualizations and there are numerous different ways to display information, such as the number of titles per author, publisher, state, or year. figure 4. number of cookbooks in collection published each year (1800-1920) the tableau bar chart in figure 4 shows the number of books published each year for the full collection of 1450 books. creating this chart required cleaning and transforming the dates in the 260$c field in openrefine prior to loading the spreadsheet into tableau. the chart gives a useful overview of the collection because it shows that the number of cookbooks published per year grew steadily during the 19th century and bounded upwards in the early 20th century. in the early 19th century, most families used collections of handwritten recipes, often handed down through generations and shared with neighbors and friends. the publishing industry in the united states expanded rapidly during the late 19th century and commercially produced cookbooks became widely available. figure 5. number of cookbooks in collection published per state the tableau map in figure 5 shows the number of books published per state for the full collection of 1450 cookbooks. the map is a filled map with the darker shades of color representing the states with the greatest numbers. tableau has a user-friendly mapping function and is able to plot latitude and longitude coordinates based on certain common sets of geographic identifiers such as countries, states, and zip codes. this automated feature is helpful because it eliminates the need to geocode data in order to create a map. adding a column with full state names to the spreadsheet in openrefine made it possible to create a basic map in tableau without any prior experience in visualizing geographic data. this map visualization of the metadata is useful because it shows how the collection reflects the history of cookbook publishing in the united states. new york has the greatest number of books published, followed by massachusetts, illinois, pennsylvania, and california. these numbers align with the growth of the book publishing industry in the united states. new york city, traditionally the publishing center of the united states, published the greatest number over time, followed by other publishing centers in boston, philadelphia, chicago, san francisco, and los angeles. the trend in the numbers also shows the history of westward expansion from 1800 to 1920, with the greatest total numbers in the east and much lower numbers in the west. analyzing full text using htrc portal tools workflow overview: search selected terms as keywords in full text and download metadata for results upload new worksets in htrc for these keyword search sets upload new worksets in htrc for topical subgroups of titles created using openrefine run htrc algorithm on selected worksets evaluate results, look for errors in data, and re-run as needed export results from htrc this workflow moves beyond analyzing metadata to look at the full text of the books as a corpus. analyzing the full text can be done using the full collection of titles or it can be done by creating subsets of titles and uploading them as separate worksets in htrc. openrefine can be used to create subgroups based on the metadata (examples: all books by a particular author, southern cookbooks) and then the hathitrust volume id numbers can be uploaded to create worksets for each of these subgroups. another way to create a subgroup is to use the full text keyword search feature on the collection in the public facing hathitrust digital library to look for works containing a particular term. the basic metadata with the volume id numbers for titles in the search results can be downloaded and then uploaded to htrc as a new workset. metadata analysis of the worksets created by keyword search can offer another window into the contents of the collection. the timeline below was created by searching for the word “vegetarian” in the collection and then visualizing the number of titles per year in tableau. the number of books which contain the word “vegetarian” in the text is near zero until 1850, increases slowly in the late 19th century, and then grows substantially in the years from 1900 to 1920. the american vegetarian society was founded in new york in 1850 and the vegetarian movement in the united states grew over the same time span. [6] figure 6. “vegetarian” keyword over time (1850 to 1920) htrc algorithms the htrc also offers an array of text analysis tools to explore trends and patterns in the full text of a collection. the algorithms offer a range of functions as described in the documentation. [7] meandre classification naive bayes: classify the volumes in a workset into categories of your choosing meandre dunning log-likelihood to tag cloud: compare and contrast two worksets by identifying words that are more and less common in one workset meandre opennlp date entities to simile: visualize the dates in a workset on a timeline meandre opennlp entities list: generate a list of people and places in a workset meandre spellcheck report per volume: find misspelled words that are the result of ocr errors meandre tag cloud: create a tag cloud visualization of the most frequently occurring words meandre tag cloud with cleaning: performs cleaning of the text before it allows you to create a tag cloud visualization of the most frequently occurring words meandre topic modeling: identify “topics” in a workset based on words that have a high probability of occurring close together in the text simple deployable word count: identify the words that occur most often in a workset and the number of times they occur all of these are built into the htrc portal as ready-made tools that can be run on a workset within the portal. no coding skills are required and the portal serves as a single platform for managing worksets, analyzing them, and exporting the results. the htrc algorithms that proved most fruitful for the early american cookbooks project were the meandre topic modeling algorithm and the meandre dunning log-likelihood to tag cloud algorithm. meandre topic modeling algorithm topic modeling is an automated text mining technique that offers a “suite of algorithms to discover hidden thematic structure in large collections of texts.” [8] topic modeling is a methodology developed in computer science, machine learning, and natural language processing that has recently become very popular in the digital humanities and related fields. [9] tools such as mallet [10] generate “topics” or groups of related words through statistical analysis of word occurrences in a corpus. the meandre topic modeling algorithm in the htrc portal (created by loretta auvil at the university of illinois) creates a topic model using mallet and displays the top 200 tokens in a tag cloud as well as exporting the topics in an xml file. the meandre topic modeling algorithm shows some interesting trends and patterns in the text for the early american cookbooks collection. the selected tag clouds below show different topics or clusters of words that recur across all of the texts. the names of the topics were not generated by the algorithm, but rather devised by the researcher as a way to label and interpret the clusters. while it is impossible to draw definitive analytical conclusions, the topics do provide an interesting snapshot of the subject matter. early american cookbooks had many common themes, largely because the diet and cookery techniques in the 1800 to 1920 period were far more homogeneous than they are today. nearly every cook used salt, pepper, and butter as the primary methods of seasoning (figure 7), prepared meat most frequently with gravy or sauce (figure 8), and baked cake (figure 9). the “food and family” tag cloud (figure 10) reaches beyond the ingredients and instructions into the how and why of cooking and homemaking. words such as food, time, good, made, great, people, work, body, give, family and years are commonly present in the forewords and introductions to cookbooks which sought to provide inspiration for readers. figure 7. seasoning tag cloud figure 8. meat tag cloud figure 9. cake tag cloud figure 10. food and family tag cloud meandre dunning log-likelihood to tagcloud algorithm the meandre dunning log-likelihood to tagcloud algorithm was useful for comparing and contrasting two worksets. the dunning log-likelihood statistic was developed by ted dunning at the university of new mexico. it employs a statistical measure based on likelihood ratios that can be applied to the analysis of text. [11] the statistic has been employed by digital humanities researchers as a way to compare corpuses of text and discover “subtle differences between closely related sets.” [12] the meandre dunning log-likelihood to tagcloud algorithm (also created by loretta auvil) calculates dunning log-likelihood based on two worksets provided as inputs: an “analysis workset” and a “reference workset.” the “overused” words and the “underused” words in the analysis workset (relative to the reference workset) are displayed in a tag cloud and made available via a csv file. this tool has been very useful in analyzing different subsets of the early american cookbooks collection. one example shows how the content of books by fannie merritt farmer (1857-1915) differs from the collection as a whole. farmer was a major figure in american cooking in the late 19th and early 20th centuries. her most successful cookbook, the boston cooking-school cook book, was first published in 1896 and sold millions of copies in many subsequent printings and editions. the book was the first to introduce precise measurement and farmer later became known as “the mother of level measurements.” [13] the dunning log-likelihood to tagcloud algorithm results clearly illustrate her emphasis on precise measurement. in the tag clouds below, the over-represented terms (figure 11) are the more precise terms tablespoons, teaspoons, and cup. the under-represented terms (figure 12) are the more vague terms teaspoonful, tablespoonful and cupful which were frequently used in cookbooks of the era. figure 11. fannie farmer over represented terms figure 12. fannie farmer under represented terms challenges and solutions this project encountered several problems in the course of its development. the htrc portal algorithms sometimes failed to work or created poor results. tracing the cause of these problems meant questioning the validity of upload data, the suitability of the tool, and/or the composition of the dataset. sometimes the answer was simple, such as when an algorithm repeatedly failed to process a workset because a library call number had been substituted for a hathitrust volume id number in the upload metadata. other problems seemed to stem from a mismatch between the chosen algorithm and the dataset. for example, the opennlp entities algorithm is designed to generate lists of people and place names, but the results included terms such as “brown sauce” and “butter taffy” as if these were names of individuals. more complex problems involved the composition of the dataset and how decisions made in subdividing the data had influenced the results. for example, the word “feces” was a valuable clue in interpreting and correcting a problem with one of the datasets. the word first appeared in a meandre dunning log-likelihood tag cloud of over-represented terms for books published in the southern census region of the united states. it seemed hard to believe that cookbooks on southern cuisine featured feces, so a re-examination of the dataset was in order. washington, d.c. is part of the southern census region, but it is also the place of publication for large numbers of government documents. separating out the books published by government agencies from the larger southern set proved to be the answer to the problem. running the meandre dunning log-likelihood algorithm and the meandre topic modeling algorithm on the government documents alone (see figure 13 below) showed that these cooking publications were concerned with scientific approaches to human nutrition. feces along with other terms describing digestion were prominent in this dataset. rerunning the algorithms on the set of southern books without the government publications created results that did not contain the word feces and that were more consistent with the subject matter of the titles in the set. figure 13. government documents over represented terms conclusion the early american cookbooks project workflows may prove useful for librarians, technologists and researchers interested in exploring small scale approaches to digital special collections and digital humanities methods. institutions with limited budgets and staff time may not have resources to devote to large digital projects. the early american cookbooks project was planned and completed by one person working several hours per week over the course of six months. there was no need to find funding for the project because access to the hathitrust digital library, the htrc portal, and all of the tools discussed in this article are available at no cost. the tools and the workflows can be adapted to suit many different types of projects, including those not using hathitrust. for example, visualizing sets of marc records offers a useful overview of any library collection. the workflow using marcedit, openrefine, and tableau could be used to visualize all of a library’s holdings or to highlight the contents of a particular collection. visualizing catalog records in aggregate can offer users a valuable new way to understand the depth and breadth of a collection. references [1.] posner m. very basic strategies for interpreting results from the topic modeling tool. miriam posner’s blog. 2012. [internet]. [cited 2017 may 15]. available from: http://miriamposner.com/blog/very-basic-strategies-for-interpreting-results-from-the-topic-modeling-tool/ [2.] htrc home. hathitrust research center. [internet]. [cited 2017 may 15]. available from: https://analytics.hathitrust.org/ [3.] reese t. about marcedit. marcedit development. 2013. [internet]. [cited 2017 may 15]. available from: http://marcedit.reeset.net/about-marcedit [4.] openrefine. [internet]. [accessed 2017 may 15]. available from: http://openrefine.org/ [5.] tableau for teaching. tableau software. [internet]. [cited 2017 may 15]. available from: https://www.tableau.com/academic/teaching [6.] shprintzen ad. the vegetarian crusade: the rise of an american reform movement, 1817-1921. chapel hill: university of north carolina press, 2013. [7.] description of the htrc portal algorithms – documentation – htrc docs. [internet]. [accessed 2017 may 15]. available from: https://wiki.htrc.illinois.edu/display/com/description+of+the+htrc+portal+algorithms [8.] blei dm. topic modeling and digital humanities. journal of digital humanities. 2012. [internet]. [cited 2017 may 15]. available from: http://journalofdigitalhumanities.org/2-1/topic-modeling-and-digital-humanities-by-david-m-blei/ [9.] weingart sb, meeks e. the digital humanities contribution to topic modeling. journal of digital humanities. 2012. [internet]. [cited 2017 may 15]. available from: http://journalofdigitalhumanities.org/2-1/dh-contribution-to-topic-modeling/ [10.] mccallum ak. mallet: a machine learning for language toolkit. mallet: a machine learning for language toolkit. 2002. [internet]. [cited 2017 may 15]. available from: http://mallet.cs.umass.edu [11.] dunning t. accurate methods for the statistics of surprise and coincidence. computational linguistics. 1993:19(1):61–74. [12.] schmidt b. sapping attention: comparing corpuses by word use. sapping attention. 2011 [internet]. [cited 2017 may 15]. available from: http://sappingattention.blogspot.com/2011/10/comparing-corpuses-by-word-use.html [13.] longone jb. feeding america: the historic american cookbook project. feeding america: the historic american cookbook project. [internet]. [cited 2017 may 15]. available from: http://digital.lib.msu.edu/projects/cookbooks/html/authors/author_farmer.html about the author gioia stevens is special collections cataloger at new york university libraries. she holds an mlis from pratt institute and recently completed an ma in the digital humanities track of the master of liberal studies program at city university of new york graduate center. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – using google tag manager and google analytics to track dspace metadata fields as custom dimensions mission editorial committee process and structure code4lib issue 27, 2015-01-21 using google tag manager and google analytics to track dspace metadata fields as custom dimensions dspace can be problematic for those interested in tracking download and pageview statistics granularly. some libraries have implemented code to track events on websites and some have experimented with using google tag manager to automate event tagging in dspace. while these approaches make it possible to track download statistics, granular details such as authors, content types, titles, advisors, and other fields for which metadata exist are generally not tracked in dspace or google analytics without coding. moreover, it can be time consuming to track and assess pageview data and relate that data back to particular metadata fields. this article will detail the learning process of incorporating custom dimensions for tracking these detailed fields including trial and error attempts to use the data import function manually in google analytics, to automate the data import using google apis, and finally to automate the collection of dimension data in google tag manager by mimicking seo practices for capturing meta tags. this specific case study refers to using google tag manager and google analytics with dspace; however, this method may also be applied to other types of websites or systems. by suzanna conrad, digital initiatives librarian and head of digital services & technology, cal poly pomona introduction cal poly pomona launched an institutional repository, dubbed “bronco scholar,” in february 2014 to support various types of scholarship from the campus community. as one of twenty-three california state university (csu) campuses, cal poly pomona was able to use a multi-tenant, shared instance of the open-source repository software dspace hosted by the csu chancellor’s office. while we are provided with a number of helpful statistics about bitstream downloads, searches, and other statistics within this multi-tenant instance, accessing granular data about specific downloads is a very time-consuming and manual process. it was important to us to be able to track specific details about individual downloads and pageviews including author names, advisor names for student research projects, content types, and titles of items accessed. and, because our dspace instance is part of a multi-tenant instance and we do not have full administrator privileges to edit code, we needed a solution that would not require much custom code. this article will detail the learning process of incorporating custom dimensions for tracking these detailed fields including trial and error attempts to use the data import function manually in google analytics, our attempts to automate the data import using google apis, and finally how we successfully automated collection of dimension data in google tag manager by mimicking seo practices for capturing meta tags. this specific case study demonstrates using google tag manager and google analytics with dspace; however, this method may also be applied to other types of websites or systems. google analytics & events tracking google analytics is often implemented by libraries of all types to track website usage. a simple snippet of javascript code on a website linked to a google analytics account enables google to intercept page requests and capture standard data that is sent by a user agent along with the request — such as information about the requested page and the user agent that’s making the request. with vanilla google analytics, website owners can view a standard set of metrics (such as the number of pageviews or unique pageviews) grouped by a standard set of dimensions (such as page url, page title, user location, etc.). using custom javascript, one can capture additional data about page activity by using event tracking. with event tracking, one can set up javascript event handlers to submit data to google analytics every time a particular event is triggered (such as when a page element is clicked or a key is pressed). for instance, if one specifically wanted to track downloads of pdfs on a website, one could set a particular google analytics event to be recorded each time a relevant download link is clicked. other authors have discussed technical implementations and benefits of event tracking using javascript edits and custom variables.[1] once event tracking is implemented in google analytics, it is possible, for example, to track downloads by file types, file names, and other dimensions. furthermore, it is possible to set up goals to better track conversions, such as tracking the percentage of users visiting the site who actually download repository content. google analytics custom dimensions & metrics if website owners want to track certain metrics or group data based on dimensions that are not included in default google analytics, they can set up custom metrics or dimensions as discussed in the google analytics platform principles course. [2] within the standard google analytics reporting tools it is possible to use custom dimensions as secondary dimensions when viewing reports. in our case, we were specifically interested in grouping data based on a key set of metadata tracked in dspace: content type, author name, advisor name, and title. adding these metadata fields as custom dimensions was a logical way to accomplish this goal. [3] google tag manager google tag manager is a product that provides a simplified means of updating website “tags.” by google’s definition, “a tag is a snippet of javascript that sends information to a third party,” such as a piece of custom google analytics event tracking code. [4] coates and durrant discussed the usage of google tag manager to track events specifically within dspace. [5] however, these principles can apply to all types of websites that need to track events. as long as a code snippet for google tag manager can be implemented on the website, it is possible to automate tag creation without editing the code on each and every page. for example, many event tracking implementations include sample code such as: <a href="/mydownload.pdf" onclick="_gaq.push(['_trackevent', 'downloads', 'download: pdf', 'mydownload']);">download pdf</a> in google tag manager this function can be handled by a tag. the tag fires on a page according to the rules established for that tag. in order for a google tag manager tag to work, it relies on rules and macros. a tag in google tag manager is implemented for whatever is being measured such as pageviews or downloads. a rule defines in what instance the tag should be fired. a macro, whether pre-defined or custom, is a name-value pair that the tag references to define values in the tag. tracking events in google tag manager the first goal of our project was to track downloads of dspace bitstreams within google analytics so that we could filter out administrator traffic and glean more information about the users of our repository. specifically, we wanted to utilize built-in demographic data about our audiences and also track and analyze their behavior in a more granular fashion than is possible in dspace. normally we would not have the ability to maintain a set of custom events within the code of our consortial dspace instance, but google tag manager requires only the implementation of one code snippet and was easy for our administrator to implement across our entire dspace instance. referencing coates and durrant’s solution and rachel sweeney’s blog on setting up download tracking in google tag manager, [6] we set up three tags in google tag manager: a link click listener to listen for clicks on bitstream files, a total downloads tag, and an outbound links tag to listen for clicks to external websites. two rules supported these tags: a rule that fired the tag on all pages and a rule that fired the download tag when a bitstream was clicked on. macros were also configured based on sweeney’s recommendations to strip out any information other than the file type and to return the file format. using these recommendations, our tags were configured as follows: figure 1: google tag manager configuration at cpp troubleshooting custom dimensions in google analytics we still had the larger problem of not being able to use specific metadata fields as dimensions in our analytics reports. we wanted to be able to provide authors with reports of their downloads or to group traffic by certain types of content such as student research. custom dimensions were a good solution to track these additional metadata fields. as outlined in a blog post by justin cutroni, it is possible to use the data import function to “widen dimensions” with a csv file. [7] essentially, adding custom dimensions from an external file would allow one to map external data (not available in the standard google dimensions) to existing google analytics data structures using a unique key. since the google academy and bloggers have published a number of step-by-step guides for importing custom dimensions using the data import function, this was a starting point for attempting to implement custom dimensions. to test this data import function, we first set up four custom dimensions in google analytics using custom definitions in the property admin section: figure 2: custom dimensions admin setup in google analytics then we set up these dimensions in the data import section (also located under property admin in google analytics), which provided us with the keys to map each of the metadata fields to a dimension, e.g., ga:dimension3 maps to the “advisor” custom dimension. figure 3: mapped metadata fields in google analytics data import dspace does offer a metadata export function, which we used to get a metadata dump for the csv file. this file was unfortunately very messy; often the metadata fields would span two columns so we had to combine those fields manually. also quotations within titles were corrupted during the export and some fields had to be normalized before they could be imported successfully to google analytics. we were only interested in tracking four metadata fields in custom dimensions, so we also had to reduce the csv to include only those four columns and the unique key that google analytics would reference to assign dimensions to pages. in our case, the unique key was the pagepath, which corresponded to the handle url in dspace. our final csv included columns for the pagepath and four columns for the custom dimensions as defined in google analytics. figure 4: sample csv file for google analytics data import once we had cleaned up the csv file and set the paths and dimensions to correspond to the custom dimensions in google analytics, we were able to upload the csv under the “data import” section under property admin. this process was successful and we were able to see the custom dimensions in our reports. figure 5: custom dimensions displaying in google analytics (data import) there were a number of disadvantages to this method. dimension widening is not retroactive, so the dimensions applied only to downloads and pageviews occurring after the data import had happened. the manual process of cleaning the csv was also extremely time-consuming, and a more automated process would ensure that all downloads and pageviews could be captured. within the platform principles course, the google analytics academy also mentions apis for dimension widening, so we began to investigate how we could automate this process using the apis instead. we tested scraping the site and dumping the correct metadata fields into a csv that could be pulled into google analytics using the apis. we were able to get a cleaner data pull of the metadata fields than with the dspace metadata export. however, two problems still existed. first, a process still had to run to pull the data into google analytics using the apis, and second, the dimensions would only be recorded as of the time that process ran, which meant that usage statistics of new uploads would not be tracked immediately after upload. since google tag manager was already firing on our dspace instance pages, we began investigating the possibility of using existing tags to track metadata fields and include those fields as custom dimensions. one seo blogger discussed the possibility of tracking meta tag values using google tag manager to determine what optimized keywords were performing best on individual webpages. [8] dspace includes metadata fields as meta tags in the page source, so this solution offered automated tracking of dspace item-level metadata using custom dimensions. our solution using google tag manager first, as with our earlier “data widening” solution, in google analytics we added a new custom dimension each for content type, author, advisor, and title. [9] figure 6: additional custom dimensions in google analytics we set the scope to “hit.” other options include user, session, and product. the first two scopes only record the first item that a user clicks on or downloads, and if the user subsequently downloads content while on the website, those downloads will inherit the first item’s metadata records. the scope “product” is an e-commerce specific scope that applies to a product for purchase. the index number was noted so that it could be mapped in google tag manager. in google tag manager, we had already configured tags and rules to collect download statistics per examples above (see the section, tracking events in google tag manager). we set up two additional types of macros including a data layer macro to hold the information for that dimension and a macro for tracking the meta tag element we wanted to track. the second macro consisted of custom javascript that collected the information from the page. for example, this is the macro we set up to pull author information: function getdata() { var x = document.getelementsbytagname("meta"); var txt = []; for (var i=0;i<x.length;i++) { if (x[i].getattribute('name') == "dc.creator") { var s = x[i].getattribute('content') + " | "; txt.push(s); } } return txt; } this cycles through the page when the tag is fired and looks for multiple dc.creator meta tag elements on the page (to account for multiple authors). all custom javascript in google tag manager macros must include a return statement, so the return statement here returns the text of the meta tag elements. once the additional macros were enabled, it was simple to add the custom dimensions to the existing tags we had created in google tag manager using the index number we set up in google analytics custom dimensions. figure 7: mapping custom dimensions in google tag manager after this was set up, we were able to add secondary dimensions to reports using the custom dimensions in google analytics. figure 8: custom dimensions as secondary dimensions in google analytics (downloads) in order to track conversions, we also added a pageview tag that referenced the same two macros. again, we designated the custom dimensions with the correct index number from google analytics. figure 9: custom dimensions as secondary dimensions in google analytics (downloads) after this was implemented, we were also able to track dimensions for pageviews in google analytics. figure 10: custom dimensions as secondary dimensions in google analytics (pageviews) our final implementation to track custom dimensions on downloads and pageviews included the following tags: figure 11: complete setup in google tag manager meta tag challenges with dspace while setting up the custom macro to find the meta tag elements, we noticed some inconsistencies in the way that dspace produces meta tags. specifically, many of the qualified dublin core metadata fields we entered displayed as other meta tag fields. for instance, dc.contributor.advisor and any other dc.contributor fields besides dc.contributor.author would appear as dc.contributor metatags without the additional qualifier. the dc.contributor.author fields appeared as dc.creator meta tags. figures 12 and 13 illustrate these discrepancies. figure 12: metadata fields in qualified dublin core from dspace full record view figure 13: meta tag fields in the dspace page source view while this was not ideal, especially in instances where we were using metadata fields such as dc.contributor.department and dc.contributor.other, we were still able to return the text of the fields we were interested in displaying in google analytics using the google tag manager custom javascript macro. when multiple values are found for these metadata fields in our current configuration, all values found populate the google analytics reports with pipes (“|”) between the text from each meta tag. as a result, because multiple different meta tag fields might get collapsed into the same custom dimension, we have to make sure we account for these special cases (such as our dc.contributor.department fields appearing along with advisor names) when looking at reports. tips for working with google analytics and google tag manager setting up a “catch-all” profile view with no filters or restrictions in google analytics is important for making sure settings on subsequently created views are working properly. for instance, if one sets up a view with a filter to exclude traffic to subdirectories, having a catch-all view with no filters assigned will help one to troubleshoot any configuration issues with the new view. this is especially important when one is setting up filters, as these cannot be reversed. also, setting up a test view is helpful for testing all conversions and customizations without excluding or filtering any administrator traffic. events appear to take some time to start showing up in google analytics; it can take up to twenty minutes to see if google tag manager changes are reflected in google analytics. custom dimensions will take effect and be accessible across all views in a property, so one should choose wisely and carefully. in google tag manager, tags, rules, and macros are published in containers. these have to be re-published every time a change is made and each previously published version is archived under “versions” in google tag manager. we found that it was helpful to name the containers with whatever was changed. for instance, when adding a pageview tag to track custom dimensions in an existing container, we labeled the title of the container with this change. then, if the individual tag was not functional, we could revert to a previous functional container. site performance is also something to consider in google tag manager. having too many tags on a site can degrade site performance, so we chose to be selective about how many tags we were firing on a page. a disappointing revelation about events tracking in google analytics unfortunately, because google analytics relies on javascript to track activity, hits on files such as pdf downloads cannot be tracked directly via any method in google analytics. in dspace, bitstream statistics reports will still track the additional pdf downloads. if a “catch-all” view is in place in google analytics, statistics from that view can be compared to dspace administrator statistics to determine what was not captured using the event tracking in google analytics and google tag manager. files retrieved directly from search engine results are the most affected by this issue; if someone searching in google clicks directly on the pdf link in google search results, it will not be tracked. results provided by google scholar, however, link to the repository item record page, allowing those downloads to be tracked. conclusion google tag manager not only serves as a more automated solution to updating tracking code on websites, it also can be an effective way to automate the addition of custom dimensions to google analytics reports. using a combination of methods presented by others at conferences, researched online, and a bit of simple javascript to create custom macros, we were able to begin tracking custom metadata fields from dspace as secondary dimensions in google analytics. this has allowed us to provide granular reports on author, title, advisor, and content type statistics from our repository without requiring any data normalization. acknowledgments thanks to aaron collier from the csu chancellor’s office for brainstorming with us as we sorted through different options to track these statistics in google analytics. thanks to cpp library’s web developer alfredo lafarga for testing and trying various solutions. thanks to summer durrant for her advice on getting download tags to work and for alerting us of google tag manager through her 2013 lita forum presentation. also, thanks to the step program grant at cal poly pomona for funding the launch of the repository and the office of undergraduate research for their efforts in its continued success. endnotes [1] wilson, j. (2014). personalize your google analytics data with custom events and variables. code4lib. http://code4lib.org/files/wilson-c4l14.pdf; farney, t., & mchale, n. (2013). event tracking: getting all your data. library technology reports, 49(4), 27. [2] google analytics academy: google analytics platform principles. (n.d.) https://analyticsacademy.withgoogle.com/course02 [3] note that you must use universal analytics in order to create custom dimensions and metrics. classic google analytics uses custom variables, instead. [4] tag manager help: about google tag manager. (n.d.) https://support.google.com/tagmanager/answer/2574305?hl=en [5] coates, h. & durrant, s. (2013). improving user engagement in a data repository with web analytics. lita annual forum. http://connect.ala.org/files/durrant%26coates_2013_litaforum_datarepowebanalytics_0.pdf [6] coates and durrant, 2013; sweeny, r. (2013). google tag manager series 4: tips and tricks when setting up a google analytics implementation in google tag manager. http://www.fourthsource.com/web-development/google-tag-manager-series-4-tips-tricks-setting-google-analytics-implementation-google-tag-manager-16156 [7] cutroni, j. (2013). dimension widening: import data directly into google analytics. http://cutroni.com/blog/2013/10/30/dimension-widening-import-data-directly-into-google-analytics [8] moore, a. (2013). seo reporting with google tag manager. http://www.lunametrics.com/blog/2013/12/30/seo-reporting-google-tag-manager/#sr=g&m=o&cp=or&ct=-tmc&st=%28opu%20qspwjefe%29&ts=1415479900 [9] note that up to 20 custom dimensions can be added with the free version of google analytics. about the author suzanna conrad is digital initiatives librarian and head of the digital services & technology department at cal poly pomona. subscribe to comments: for this article | for all articles one response to "using google tag manager and google analytics to track dspace metadata fields as custom dimensions" please leave a response below: peter g, 2015-03-26 thank you for your article suzanna. this was a very detailed and thorough walkthrough of your solution. leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – library widget for moodle mission editorial committee process and structure code4lib issue 19, 2013-01-15 library widget for moodle any course within a course management system is generally considered the intellectual space of the professor teaching it. research tools and guides, such as search boxes for discovery services or links to course-specific and subject-specific guides, are created and maintained by librarians. in trying to get our tools and services closer to where students spend their time devoted to coursework, oakland university libraries have developed a library widget – a self-serve code generator that allows professors to select what tools and services they want to bring into their course space. this approach has proven to be flexible, because it does not depend on a library presence within the course management system. it also offers persistent presence within courses since professors can archive courses, including the library widget, at the end of a semester and restore them in the system in future semesters. we are using the library widget as a pilot to inform decisions on future full integration of such functionality into moodle. by mariela hristova introduction any content within a course management system (cms) is generally considered the intellectual space of the faculty member teaching it. thus, many libraries focus on integrating into the top level of the campus cmss by creating a library presence in the system and not necessarily in each course. such an approach has the benefit of serving all students (lawrence 2006). in fact, a study of faculty and students illustrates that while both groups support a “library tab”, i.e. a generic top level link to the library website, students specifically are more supportive of “librarian-vetted web site lists” (mclure & munro 2010, p. 38). students often need help with the use of library tools and resources as those relate to their coursework. hence, the goal of integrating library tools and resources in campus cmss is ideally to bring those resources as close to students and course spaces as possible, increasing their ease of use, as well as making their presence immediate and their relevance obvious. having a search box in the cms course space to the library’s discovery service, for instance, can offer immediate access to books and articles, rather than expecting students to go to the library website or, more likely, to google. at the university of minnesota – twin cities, for instance, librarians have recently developed automatically generated library course pages that are linked in each course within the learning management system (jeffryes et al. 2011), allowing for consistent access to library tools and resources for students from all classes, with individual guide customization still being an option. they achieve broad relevance of the library course pages by going beyond coverage of research tools and resources to include “other existing structures (reserves catalog, e-reserves) into a new, more holistic, system” (jeffryes et al. 2011, p. 26). this type of integration combines the pervasiveness of system-level library presence with the immediacy of having course-specific resources embedded within the learning space at the course level. it is, however, made possible by a “robust database-driven system of resources” (jeffryes et al. 2011, p. 26) – a key factor in their ability to automatically generate guides for all courses that include core resources relevant to each subject or discipline. at oakland university, we use an open source application for creating and managing our guides, called subjectsplus (www.subjectsplus.com). we see it as a self-hosted alternative to libguides that allows us to centrally manage records for our databases and other resources and to create guides at the subject and course level. in adopting subjectsplus, we wanted to allow for future integration into the course management system on campus, moodle. thus, we decided to create subject guides that correspond to the subject listings in the course catalog and have the url of each guide explicitly state which course rubrics it corresponds to. for example, we used to have one subject guide for management and marketing, which turned into two separate guides: the management guide now corresponds to any courses with rubrics of mgt and org, while the marketing guide corresponds to courses with a rubric of mkt. the close correspondence of subject-level guides to course rubrics enables course-level integration into moodle, because all classes that do not have a custom course-level guide can be easily matched with the corresponding subject-level guide. our new approach to subject guides not only helps with integration into moodle, but has also helped solve common nomenclature dilemmas, e.g. whether a guide should be called english or literature was easily solved by deferring to the course catalog that has helped establish student habits and expectations. in fall 2011, we recognized that system-level integration into the campus cms was likely to be a long process. the technical team administering moodle was spending the academic year preparing for an upgrade to the next version of moodle; thus, it was out of the question for them to develop custom functionality for a version that was being phased out. we decided to create a library widget for moodle that can serve as a pilot for the more complete future integration and can help us evaluate the need for, and potential use of, library tools and resources directly from within the cms course spaces. widget features & content we designed the library widget as a php-based, self-serve code-generator. faculty would use the widget interface to select what links to library tools and resources they want to bring into their course space and then paste the resulting html code into moodle. the availability of customizable menu blocks in moodle provides a convenient place for the html code, which does not include any presentational elements and integrates well with any of the available moodle themes. by definition, a code-generator would expose users to some code they need to manipulate. one of the main concerns was the ease of use of the interface, because the tool is intended for faculty of any comfort level with technology. the interface builds expectations about the process by having steps listed as tabs and indicating where users are in the process. we also found it important to label each step in a clear and action-oriented manner. as a result the interface includes three sections: step 1: select content, step 2: preview & revise, and step 3: paste in moodle (see figures 1-3). additionally, the language on the first page of the interface had to be clear and inviting, so we chose to include brief explanations and to make the multiple choice options as conversational in style as possible. on the last page, we include not just the html code for pasting, but also explicit directions on how and where to paste it in moodle. figure 1: library widget interface – step 1 allows for initial content selection. http://library.oakland.edu/information/services/web_services/moodle_widget.php figure 2: library widget interface – step 2 allows for a functional, live preview of the resulting widget and for a revision in content selections. figure 3: library widget interface – step 3 offers directions on how and where to paste the resulting html code in moodle. in selecting what content options to offer for the library widget, we prioritized content that will be relevant across the disciplines. library onesearch – our summon discovery search – had just been introduced widely on campus. we were de facto starting to use library onesearch as the main search tool for most undergraduates by incorporating it in our information literacy instruction and in our interactions at the reference desk. marketing efforts for library onesearch on campus had reached the faculty in multiple ways and the search box was also incorporated prominently both on the front page and in every content page of the library website. thus, library onesearch made a logical first choice for the widget interface, but it still needed a description because the label referred to a branding choice and would not have been sufficient if someone was unfamiliar with the tool. the library research guides both at the course level and at the subject level provided the impetus for the widget in the first place. our course-level research guides, known on campus as library course pages, were traditionally getting high web traffic because they were used as presentational tools in our information literacy instruction sessions – librarians would prepare a course page for each class they visited, work from it during the class session, and show students how they can get back to it throughout the semester. about a hundred courses, as well as all sections of freshman composition classes (ranging from about 60 to 100 sections each semester), receive information literacy sessions each semester and have corresponding course pages. the widget searches through the mysql database of subjectsplus for any guide that has the specified course identifier in its url, even if the course is cross-listed and has more than one identifier in its url (e.g. a course called language and society is cross-listed with three identifiers: http://research.library.oakland.edu/sp/subjects/guide.php?subject=als376/als576/soc376). if a course has multiple sections that are sufficiently different and necessitate different course pages, the interface shows the multiple course pages as choices for the faculty member to select from, i.e. these urls contain not just the course identifier, but additional section-related information as well. this is a rare situation, but happens occasionally with course rubrics that allow for special topics to be covered in different sections. for example, we have two sections of hst 300: seminar in historical research, so the urls for each course guide contain faculty names in addition to course identifiers as in http://research.library.oakland.edu/sp/subjects/guide.php?subject=hst300miller. if a course page cannot be located, then the letter portion of the course rubric is used to identify the corresponding discipline and offer a subject-level research guide instead, i.e. the widget delivers the marketing subject guide (http://research.library.oakland.edu/sp/subjects/guide.php?subject=mkt) for any mkt course that does not have a custom course guide. finally, electronic course reserves constitute some of our most visited content online based on web traffic data. they consistently rank on par with visits to the list of library databases. moreover, they represent library-based content that is relevant to many courses that otherwise do not involve conducting research for assignments; the inclusion of course reserves as an option extends the potential usefulness of the widget on campus. the course management system has been found as a natural and effective home for disseminating course reserves information by a number of universities (black 2008, p. 497). at this time, our course reserves link leads to a standard landing page rather than a course-specific link. a course-specific link would be ideal, but is not currently supported by the reserves workflow or systems, i.e. the print reserves are in the library catalog, while the electronic reserves are kept in a password-protected web area. we do recognize the reserves landing page is a more reliable url that will work even before faculty have actually placed materials on reserve for their course. code-generator vs. widget the development of the library widget as a code-generator rather than a traditional widget has largely determined both its strengths and its weaknesses. the choice between a code-generator interface and a traditional widget influences key features of the tool and should be taken into careful consideration. table 1 below summarizes the main differences. if we had conceived our tool as a traditional widget, we would have had to offer code for embedding based on javascript, for instance, and store the unique content choices for each widget instance to deliver dynamically into moodle. the biggest benefit of creating a traditional widget would have been the centralization of its content management, i.e. we would retain control over the content and could change the link urls or the code for the search box globally for all widget instances in use. the downside of that approach for us related to the need to use even more dynamic content and external resources in moodle, which would have made the loading of pages even slower than it is currently. moreover, the content options in the library widget are considerably stable, so losing control over them did not represent a huge risk – it replicated the pre-widget status quo of emailing links to faculty who then would place them somewhere in their moodle courses. we are hoping to eventually have the library widget integrated into moodle as custom functionality of the system, thus the current one has a somewhat temporary role to play, which further lowers the risk of decentralizing its content management. the choice might be different if one envisions long term use of the library widget. as a result of these considerations, we created the library widget as a code-generator. the main benefit of that choice consists in the ease of modifying content selections. selections are saved within the structure of the code-generator url through the use of the get method. this approach allows librarians to go through the code-generator interface and make the appropriate content choices for a course they are visiting, for instance. then they can email the step 3 url to faculty of different levels of comfort with technology, who can either proceed to insert the code in moodle or make modifications in the content selections without the need for user authentication and the risk that they might be affecting a widget instance that is already in use. this added flexibility has allowed librarians to offer the library widget even to faculty who would have been intimidated by the tool or discouraged by the need to go through several steps. it has also allowed faculty to cycle through the code-generator interface multiple times and make revisions that match each of their courses without creating administrative overhead for the library or contacting us for technical support. additionally, we have discovered an unanticipated benefit stemming from the treatment of the library widget as course content in moodle rather than a moodle feature to be turned on or off. when faculty teach the same course in subsequent semesters, they usually archive the course content and restore it in the new course space for the current semester. we have encountered numerous faculty informing us they do not need to add the library widget in their current courses, because it is already there from the previous semester. full moodle integration in the future might require the activation of the library widget every semester and thus necessitate recurring marketing efforts by the library. in any integration efforts, we believe that it will be imperative to work towards a feature that is activated by default, so faculty are aware of it but have the option to deactivate it. the main drawback of the widget as a code-generator relates to its limited tracking of usage. currently, we record each instance of someone getting to the final step of the code-generator interface, where they are presented with the html code to insert into moodle. the data we record includes their content selections and the rubric of their course. we cannot, however, capture directly if they actually have the library widget inserted in moodle and how much it is being used. alternate sources of that data are available for the web pages that we host: google analytics referral traffic sources capture the number of visits to specific pages that come from the moodle domain. having the library widget fully integrated into moodle might ultimately offer the best tracking opportunity. code-generator widget moodle integration content management decentralized centralized centralized authentication to configure content selections no yes built-in tracking of usage interface usage tracking is separate from content tracking ability to track content usage directly ability to track both feature activation and content usage, but conducted by moodle administrators faculty awareness of tool high due to need for action high due to need for action high only if activated by default persistence across semesters yes yes depends on integration table 1: key considerations in developing a library widget marketing strategies creating awareness of the tool on campus has been done through email and through our instruction sessions. faculty received direct email twice when it was first created and at the beginning of each semester afterwards. additionally, subject librarians have emailed faculty in their areas offering various library tools and services, including the widget. the most effective method of raising awareness, however, has anecdotally been the direct communication librarians would have with the faculty when visiting their course to conduct an information literacy session. the end of a class session led by a librarian has proven to be a great time to show the faculty member how the widget can bring the course page that was just used for their class into the moodle course space. that has also been an opportunity for the librarian to actually help the faculty member paste the code into moodle if necessary. apart from the planned efforts by librarians, a campus committee on teaching and learning invited us to conduct a workshop for faculty on library-related technologies for instruction. the librarians offering the workshop decided to devote half of the workshop time to the widget and received positive comments from the attending faculty. the attendees also had an opportunity for hands-on practice and a number of them used that time to add the widget into their courses. adoption rates during the first full year since the creation of the library widget, october 2011 – october 2012, it has been used 185 times. based on our data on content selections, all of the widget instances include library onesearch. the widgets also contain 137 links to course-level guides and 48 links to discipline-level guides. the widgets linking to course-level guides span the disciplines, representing 36 unique academic programs on campus. finally, course reserves are included in 47 of the widgets. figure 4 illustrates the distribution of new widget creation over the months of this first year of use. the initial high usage correlates temporally with our marketing efforts. the addition of new widgets over time shows continued interest in the widget, since the figure does not capture all the widgets that faculty carry over through the semesters without having to come back to the interface and make new content selections. we see the steady creation of new widgets as data indicating an increase in adoption either by new faculty or for new courses. figure 4: widget usage for oct 2011 – oct 2012. lessons learned one of our biggest concerns in launching the library widget as a code-generator rather than waiting until we can have an integrated moodle feature was the need for faculty action in order to place the widget in their courses. this less automated approach, however, has focused the launch process on raising awareness across campus to encourage faculty action. furthermore, faculty seem to have a concrete understanding of what tools they are bringing into their course space due to the content selections they have to make. as a result, the library has been able (1) to illustrate to moodle administrators what type of a feature we are hoping to integrate into moodle one day; (2) to illustrate to faculty what value the library tools and content can offer directly within their courses; and (3) to gauge faculty interest in the availability of an integrated moodle feature. the library has not yet surveyed the campus to have a formal assessment of faculty needs and interest in having this tool integrated into moodle, but when that survey is conducted, we believe more of the faculty will know from personal experience what value the library widget offers for their courses. inadvertently, we are learning that an approach involving human action might actually be beneficial in the case of a pilot project like our library widget. our experience of developing the widget has informed our understanding of what challenges the future integration into moodle might present. for example, we had originally assumed that an integrated feature will offer pre-determined content selections, i.e. a menu block with the correct links automatically included for each course. our development experience and the adoption data, however, support the need for configuration options built into the future moodle feature. the ability to choose content options might increase the relevance of a library feature to each course, i.e. some courses might need a guide, while others might need a link to course reserves. the occasional, but still very real, existence of multiple courses with the same rubric and number, yet very different thematic content, further indicates that explicit choices by the faculty might prove more useful than a link to a guide that cannot be customized. the lower than expected inclusion of course reserves in library widgets has surprised us, because we saw course reserves as a means to make the tool relevant to more courses. at the same time, our marketing efforts were focused on the research-related widget content more than reserves. this indicates that perhaps marketing efforts specific to course reserves might be useful, since that content is rather different from the rest of the content options in the widget and might be relevant to a different group of faculty. it is also possible to explore ways to raise awareness of the library widget from within the workflow of putting materials on reserve. we have not yet attempted to integrate the library widget within the reserves clerk’s communications with faculty who are using that service. if sharing the widget at the end of instruction sessions is any indication, we can expect that integration in the reserves workflow might increase adoptions in a similar manner. we plan on analyzing long term trends in our use of reserves first, because the high web traffic might indicate intensive use by a small number of courses. finally, tracking of widget usage can provide valuable data in the decision-making process for full integration of library tools into moodle courses. we found that tracking content selection has been useful and informative, but the overall usage of the widget cannot be verified. thus, it is critical to base observations of campus support for future integration efforts on a survey or another means of gathering input. the existence of the widget, however, is unique and valuable in that process, because it can clearly illustrate what the library is interested in bringing into the campus cms, rather than relying on respondents’ prior knowledge of library tools and resources. next steps we plan on continuing to promote the use of the library widget through our ongoing interactions with faculty. the widget, while not completely automated and integrated in moodle, does allow faculty to make useful content selections and easily bring library tools and resources into their course spaces. the need for action on the part of faculty imposes a burden on librarians to keep raising awareness of the tool, but it also respects the intellectual control that faculty have over their virtual course spaces. as the use of the widget grows, we plan on surveying faculty to determine their needs for library tools and resources in the cms. we hope to pursue a full integration with moodle that is implemented in a manner respectful of faculty ownership of their course spaces. meanwhile, we are happy to share the library widget php code with anyone interested in using it or customizing it for different contexts. the specifics of each context are likely to be so different as to necessitate significant revisions based both on course management systems and on library guide management systems. [zipped code files.] references black el. 2008. toolkit approach to integrating library resources into the learning management system. the journal of academic librarianship 34(6):496-501. (coins) jeffryes j, peterson k, crowe s, fine e, carrillo e. 2011. integration innovation: launching the library into a course management system. journal of library innovation 2(1):20-34. (coins) lawrence dh. 2006. blackboard on a shoe string: tying courses to sources. journal of library administration 45(1/2):245-265. (coins) about the author mariela hristova is an assistant professor and coordinator of web services at oakland university. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – frbrizing an e-library : migrating from dublin core to frbr and mods mission editorial committee process and structure code4lib issue 12, 2010-12-21 frbrizing an e-library : migrating from dublin core to frbr and mods western state college in gunnison, colorado developed an open-source ecataloger framework, based on dublin core metadata, on google’s app engine to manage and serve electronic resources to the library’s patrons. pressed to find new solutions for failing manual workflows for serials and government document resource management, the ecataloger framework was extended to frbr to automate and enhance serials management and government documents receiving. based on successfully frbrizing the ecataloger, western state college converted their e-library management from dublin core to frbr and mods. this paper examines the processes of each of these implementations using python, ajax, and jquery, the details of the frbr data model, including using frbroo, and the successful user interface supported by a frbrized catalog. by jeremy nelson and alan cleary introduction: for a small academic library, the dual challenges of meeting patron demands for accessing information electronically coupled with static or shrinking budgets requires creative technology solutions. at the leslie j. savage library at western state college, we developed an e-library, running on google app engine, to manage the presentation of purchased and free electronic resources. ebsco’s academic search premier, lexisnexis, jstor, and others, along with vetted research-oriented websites from government, corporations, and non-profits, are presented to our primary patrons, liberal arts undergraduate students and faculty. the development of the ecataloger framework began with a class project through an ala electronic collection online course which encouraged the curating and the development of a web interface to the library’s collection of purchased resources and vetted research websites. almost concurrently with this class project, google corporation announced the release of a development software development kit (sdk) for a new “cloud” application hosting environment called google app engine or gae (google inc.)  google app engine does not use a traditional relational database but is built upon a python interface to google’s bigtable distributed storage systems for managing structured data at petabytes scale (chang, et al., 2010). google app engine also uses the open-source django web framework (django software foundation, 2010) for its web interface. this presented an opportunity for the savage library to develop and deploy an e-library to our patrons without worrying about the logistical and budget requirements of hosting this application internally through the campus information technology department. to represent a web-accessible resource, dublin core based data models were created as gae native data classes with supporting data classes built using the controlled vocabularies of the metadata object description schema (mods) language from the library of congress. the first version of western state college’s e-library was released in the summer of 2008 and served 85 resources organized by title, subject, and genre facets. with the successful deployment of the e-library using the ecataloger application framework, we turned our attention to a couple of manual workflows in the library that used two legacy microsoft access databases for government publication processing and for tracking budget and usage for print serials. both of these databases suffered from poor design, and usability was further limited because they were not accessible to front-line student workers at the library. using ecataloger looked like a possible solution; however, the dublin core data classes were not semantically descriptive enough and lacked the type of relationship modeling to describe federal documents and print serials without extensive modifications. the current state of bibliographic research, the functional requirements of bibliographic records (frbr), provided the necessary descriptive and modeling framework to create records for government publications and printed serials. western state college deployed a serials management gae application on february 2009 that contains 1,135 frbr-based works tracking 5,374 total individual items. a federal and state government publications application deployed on april 2009 contains over 8,500 frbr-based works, expressions, manifestations, and items and over 3,000 federal and colorado state shipping lists. in january 2010, the design of the original e-library for western state college was starting to show its age and the library administration suggested that an update would be desired for the new year. coupled with a new developer joining the ecataloger project, we decided to migrate the original e-library classes and data to the latest ecataloger version along with an updated user interface using css and jquery. the remainder of this paper will describe the original e-library data classes and structure of the gae application, the iterative development of the frbr-based data models in developing two internal applications, and the challenges of migrating the original dublin core e-library to ecataloger version 1.2. dublin core e-library the dublin core data classes supporting the gae e-library are just part of the model-view-controller structure of the python-based django web application framework. the basic design strategy for implementing the e-library was to map some dublin core elements, like title, directly to a standalone data class (titles) and to map other dublin core elements to properties of a larger data class, like resource, in the application.  additional properties were associated with resource for management purposes that are not included in the dublin core specification. this version did not implement the full dublin core element set but just enough to support the requirements for the e-library. for the dublin core data model, all base entities were represented as individual instances in the resources google app engine model. the information represents metadata “for simple and generic resources” (dcmi metadata basics) and also includes additional properties used in the public display and in the management of these entities. some of these dublin core based properties such as creators, dates, uids (for unique identifiers), and editors, were placeholders but not actually used in the implementation of the e-library. the subjects, publisher, and description properties were directly mapped to their corresponding dublin core definitions. management properties, such as true/false boolean properties like is_subscription, were used to differentiate between paid subscription databases and freely available resources. the is_trial boolean property was used for trial period database subscriptions that were presented separately to the end user of the e-library. the rating property provided a way to rank the resources so that a small sub-set could be highlighted as a “popular resources” facet in the patron web user interface. in the initial design, a collections property was added as a way to host multiple e-library’s on the same google app engine application instance. the ease of creating additional applications on google app engine allowed us to eliminate the collections management layer in the data model design for subsequent iterations of ecataloger. when an additional e-library or other ecataloger application is needed, a new application instance is created on google app engine thereby shifting many of these collection management functions to the hosting framework. the full python code for the resources model is presented below. the resources class was derived from a basemodel class (basemodel was further derived from the google app engine model class) that included common properties used through-out the data classes in the e-library. class resources(basemodel): titles = db.listproperty(db.key,default=none) creators = db.listproperty(db.key,default=none) editors = db.listproperty(db.key,default=none) uids = db.listproperty(db.key,default=none) subjects = db.listproperty(db.key,default=none) publisher = db.referenceproperty(organizations) base_url = db.linkproperty() description = db.textproperty() genres = db.listproperty(db.key,default=none) collections = db.listproperty(db.key,default=none) rating = db.ratingproperty() dates = db.listproperty(db.key,default=none) languages = db.listproperty(db.key,default=none) website_check_frequency = db.stringproperty(required=false, choices=(["never", "daily", "weekly", "monthly", "quarterly", "yearly"])) is_subscription = db.booleanproperty() is_trial = db.booleanproperty() following the best practices suggested by the dublin core metadata initiatives (dcmi metadata terms), a number of controlled vocabularies were used to restrict the range of data that would be accepted in the genre data model class.  the library of congress’s mods genre element was used as the controlled vocabulary. genres, while not present in the dublin core metadata, were incorporated as a property to further differentiate resources. the first public web user interface presented a genre facet of the e-library’s resources. in further refinements of the user interface, the design was simplified by removing the genre facet. frbr implementation in ecataloger following the successful deployment of the e-library, attention turned to possibly using gae to improve the operation of other areas within the responsibilities of the electronic resources, serials, and government publications librarian. for a small academic library, the print serials costs consume a large portion of the library’s material budget so tracking usage of these materials is an important task for selection and retention decisions. the workflow for generating print serials usage statistics was an extremely manual process involving student workers and a poorly-designed microsoft access database.  a similar, highly manual workflow was also in place for processing federal and state publications when these materials entered the library that required multiple approval steps by staff and another manual input workflow into microsoft access.  due to staff reductions and subsequent consolidation of job responsibilities, a single librarian with two or three part-time student workers were responsible for these workflows. the amount of time these workflows consumed, the number of errors introduced in the manual data entry steps, and the increasing demands in other areas of the library on staff time meant that the current manual workflows were unsustainable.  by creating web applications for both workflows, the use of paper tracking and management of these essential library resources could be eliminated. double or triple re-entry of the data into a database would be reduced to a single point-of-entry into a web datastore.  using a web datastore allows for all of the data to be managed by a third party, an advantage of using cloud-based services, instead of the ad-hoc management of microsoft access databases on a shared network drive as had been the previous practice.  finally, historical and current data could be merged on a web datastore for better and more comprehensive reports. by necessity, the manual workflows for these serials and government publications did not track all of the information for individual items such as title, volume, and issue number. including more identifiable item-level information would allow better and more granular reporting on the library’s resources.  this requirement is better met with web applications. while the dublin core metadata models can be easily re-purposed to track all this information for individual items, it lacked the type of relationship modeling of printed periodicals and government publications that would be necessary to replace these old workflows. for example, a serial has information associated both at an abstract collective level (title of the serial, issn number) and at the individual item level (volume, issue, barcode). the government publications needed to track individual items as well as its associations with one or more shipping lists. with google app engine as the hosting and application framework, research was done to see if a single data modeling strategy could be adopted that was flexible enough for meeting all of the requirements for replacing the current serials and government publications workflows. very soon, the conceptual models and underlying entity-relationship theory behind the frbr effort (frbr final report, 2009 pp. 9-10) became a leading candidate to use as the underlying datastore classes for these new web applications. frbr explicitly attempts to model different levels of abstraction of bibliographic records that better fit both the serials application and government publication requirements. for the serials application, frbr’s work, expression, and manifestations provide the levels of abstraction separate from the individual items needed in the application. in the case of the government publications, specific types of items, print, microfiche, and media, could also be modeled as frbr entities. by using frbr for the datastore classes, common crud (create, read, update, and delete) operations would be the same but with different user interface views and flow-control code for meeting the different requirements of the serials and government publications workflows. the library decided to first start with implementing the serials management web application with the intention that the same datastore models could be used for later development in other contexts like the government publications workflow. serialsmanager web application as the first ecataloger application to implement frbr as gae datastore classes, the serials management application needed additional descriptive information to fully support an implementation-ready application. for example, the frbr specification states that frbr works, expressions, and manifestations (frbr final report, 2009 pp. 33,36,40) have a title attribute with descriptions that closely match the library of congress mods titleinfo element. by implementing the full mods metadata elements as native gae data model classes, these classes provided a rich enough base descriptive set to be used as frbr works, expressions, manifestations, and items attributes without the need to create yet new classes to represent common metadata. while the frbr specification does not explicitly define type 1 entity sub-classes, it is clear from their organization of entity attributes that they implicitly associate types of entities with certain attributes. for example their title for the expression attribute 4.3.14 is ‘expected regularity of issue (serial)’ and in the description, the expression type is referenced as serial (frbr final report, 2009 p. 38). implementing frbr as an object hierarchy, we decided to create subclasses organized around these sets of attributes from base work, expression, and manifestations classes. however, because two classes cannot have the same name in a module’s namespace, we could not use the same class name of serial for both an expression and manifestation as mentioned in the frbr specification. to differ between a serial expression and a serial manifestation, a new manifestation subclass of periodical was used in the serialsmanager application. as with the dublin core e-library data classes’ additional properties, management properties like is_current_subscription, were added to the expression class to meet management and reporting requirements of the application. below are pseudo-python code snippets showing the ecataloger mods and frbr classes with selected class properties and relationship linkage used in the serial management application. class modsmodels.titleinfo(basemodel): type = db.stringproperty(required=true, choices=(['abbreviated', 'translated', 'alternative', 'uniform'])) displaylabel = db.stringproperty(required=true) subtitle = db.stringproperty() title = db.stringproperty() class frbrmodels.work(basemodel): titles = db.listproperty(db.key,default=none) class frbrmodels.serial(expression): frequency = db.stringproperty(required=true, choices=(['annual', 'bimonthly', 'semiweekly', '...'])) is_current_subscription = db.booleanproperty() issn = modsproperties.identifierproperty() issuance = db.stringproperty(default='continuing') title = db.referenceproperty(modsmodel.titleinfo, collection_name='expression_title') work = db.referenceproperty(work, collection_name='realized_work') class frbrmodels.periodical(manifestation): issue = modsproperties.defaultproperty(default=none) expressions = db.listproperty(db.key,default=none) title = db.referenceproperty(modsmodel.titleinfo, collection_name='manifest_title') volume = modsproperty.defaultproperty(default=none) class item(basemodel): barcode = modsproperties.identifierproperty() manifestation = db.referenceproperty(manifestation, collection_name='exemplified') usage = db.listproperty(datetime.datetime,default=none) a nice feature of using gae data models is that creating associations between data store objects is not as restrictive as using tables and foreign keys in a traditional relational database. for example, the item object’s manifestation property can reference a specific child class instance of the more general parent manifestation class without the overhead of joining multiple tables in a foreign-key relationship as in a traditional database. gae uses keys-values to create one-to-one, many-to-one, and many-to-many relationships which follows closely to the frbr entity-relationship model. some unexpected benefits occurred when using frbr as the organizing framework for data modeling in the serials management application. while the application was developed primarily to track usage of individual issues, by abstracting issue information from specific item data, we could now note damaged and replacement items and still track usage at the manifestation level for a better snapshot of the serials inventory. missing items could now be better identified and tracked using the same application. the serialsmanager application also provided near real-time usage data, unlike the previous manual workflow that produced reports at semester intervals. when the serials management application went live, student workers were able to enter an issue’s barcode or search for an issue and have the usage updated automatically using any web browser. librarians and library administrators were able to run multiple types of reports from the same interface used by the students. to explore the additional functionality provided by google app engine, django, and python, a dynamic mashup was created that used oclc and gold rush web services to augment the display for a single work report. gold rush provided openurl linkage to any existing electronic full-text and abstract versions of the serial in the library’s databases while the oclc web services provided oclc identification and subject associations, all merged into a single view with the latest usage and items statistics for an individual work. govdocsmanager web application building the second web application to replace the government documents manual workflow required some slight refactoring of the core frbr and mods classes in ecataloger. the manifestation was first sub-classed to a federaldocument child class that associated the manifestation with one or more shipping lists. the federaldocument class was further sub-classed to support specific media like cds, dvds and microfiche. for application control and user interface convenience, an endeavor class was added as a way to link a single work-expression-manifestation-item grouping for security and searching functions. this class’s name comes from the frbr specification that briefly mentions grouping specific “products of intellectual or artistic endeavours”  (frbr final report, 2009 p. 13) into work, expression, manifestation, and item entities.  the following code snippet shows how the endeavor groups a single work, expression, manifestation, and item along with a title into a single object. class endeavor(basemodel): work = db.referenceproperty(work, collection_name='endeavor_work') expression = db.referenceproperty(expression, collection_name='endeavor_expression') manifestation = db.referenceproperty(manifestation, collection_name='endeavor_manifestation') item = db.referenceproperty(item, collection_name='endeavor_item') title = db.referenceproperty(modsmodel.titleinfo, collection_name='endeavor_title') the time savings and operational efficiencies of the government documents application over the manual workflow came about through additional ecataloger functionality at the application controller and user interface levels. manual entry of many shipping lists was eliminated entirely by utilizing a united states federal depository service that hosts its entire shipping lists as electronic database files in dbf format on their servers, which is downloaded automatically by inputting the received shipping list number.  this single-step approach has produced lower error rates and significant time savings.  the government documents web application also allows for manual entry of items and shipping lists since the state depository library did not provide their shipping lists in any accessible electronic form. e-library migration to ecataloger 1.2 with frbroo as the ecataloger framework matured through the release of two successful internal applications, implementation and design compromises that were initially made to facilitate deployment became more apparent. this, coupled with the library administration’s desire to update the now one-and-a-half year old e-library’s user interface and to standardize all of the library’s web applications, presented an opportunity for a major refactoring of the ecataloger code base. one of the compromises that had been made with the government publication application was the creation of shippinglist and its child classes. the shippinglist base and child classes were not frbr entity classes but they did contain metadata and were a collection of frbr federaldocument frbr entity classes. while this type of additional object-oriented class structure fit well with the frbr classes, it felt arbitrary and frbr should be able to represent the fact that such a work that was also a collection of other work objects. fortunately, a draft definition of an object-oriented version of frbr, called frbroo, had been released that is “a formal ontology intended to capture and represent the underlying semantics of bibliographic information and to facilitate the integration, mediation, and interchange of bibliographic and museum information.” (frbroo, 2007 p. 10). frbroo provides an object hierarchy starting from base work, expression, and manifestation classes with derived child classes that could replace the ad-hoc frbr and supporting classes used in ecataloger. for example, the frbroo work subclass, containerwork, would be a good parent for the ecataloger shippinglist class. a full refactoring of the core ecataloger frbr and mods libraries using frbroo as an implementation guide resulted in a number of new additional classes and relationships.  unit tests were added to ecataloger to ensure the new frbroo data classes would still support the existing serials and government documents applications while adding new e-library classes.  unit testing improves the robustness and extensibility of the entire ecataloger framework by creating stub frbroo instances to test for consistency and usefulness when implemented in the controller and view aspects of the ecataloger framework. unit testing is also a formalized way of ensuring units of code perform in an expected manner and allows for better change management by assuring that any new changes to the data models would not break existing functionality as well as testing if any new changes could accomplish the expected design outcomes. over 250 unit tests were created to test the data models representation and linking characteristics of the datastore and the new frbroo classes. if the modeling of these entities in our unit tests resulted in expected behavior, we would feel confident in adding the changes to ecataloger. to migrate the data from the original e-library hosted at http://ecataloger.appspot.com/ to a new instance of the western’s e-library hosted at http://wscecataloger.appspot.com/, client-side python scripts were developed to transform the dublin core original data instances into the latest version of the frbroo-based ecataloger data instances. a sqlite database was used as transitional data storage and to manage the linked nature of the data models in both the dublin core and frbr models. after the supporting classes were migrated to the ecataloger 1.2 data class entities with the resulting google app engine unique keys saved back to the sqlite database (please see appendix 1 for a complete mapping), a resource entity was deconstructed into the ecataloger frbr work, expression, manifestation, and items classes or sub-classes. the actual resources entity is represented in ecataloger 1.2 data models as an individualwork, selfcontainedexpression, remoteaccesselectronicresource manifestation, an item, and finally either of these two convenience collection endeavor classes: purchasedelectronicresourceendeavor or websiteendeavor. the remoteaccesselectronicresource class comes from an implicit sub-class in frbr (frbr final draft, 2009 p. 40). for a complete mapping between the original dublin core e-library resource and the ecataloger frbr and mods data classes, please see appendix 2. e-library jquery user interface, ajax, and proxy cache an essential requirement of moving to version 1.2 of ecataloger for western’s e-library involved an update to the web user interface. the javascript and cascading style-sheet open-source framework jquery was selected for building a more responsive and richer functional interface for the students, faculty, staff, and the local community of western state college. the new interface is driven primarily by jquery base library functions and the jquery user interfaces plugins. sizing is one of the jquery functions used to fit the interface into the frame dynamically. although somewhat of a hack, the sizing function was selected as an alternative to moving towards a more complex vector-based interface. a second useful function is the jquery dialog function that created quick notifications to the end user in the administration interface and was also very useful for debugging the application. three jquery plugins, cookie, highlight and treeview, add functionality to different interactive aspects of the application the cookie plugin is imported in multiple templates and used by other jquery plugins. the highlight plugin is used in the public interface to highlight search terms found within search results. finally, the treeview plugin proved to be the most versatile and is used in both the public and staff interfaces as a way to organize and present different free and purchased endeavors as seen here: figure 1. e-library use of jquery treeview plugin the jquery ui accordion, animated, tabbed, and vertical menu plugins are used within the staff interface. both the accordion and the tab jquery ui plugins organize the various tools and functions of the administration interface into a single web-page as seen below: figure 2. e-library use of accordian plugin the quick manage tab’s simplified user interface allows administrative users an easy way to create, edit, and remove purchasedelectronicresourceendeavor or websiteendeavor objects in the e-library. the advanced cataloger tools displays a full treeview of all of the frbr and mods objects in the datastore and allows the user to create, edit, or remove any of these objects. the set-up & help tabs assist in configuring the e-library and adding help topics for the public e-library. the reports tab provides usage information on endeavor and help topics by users in the e-library. here is a screen shot of the edit view for the lexisnexis individualwork instance. all data objects in ecataloger can be managed in the same interface: figure 3. edit view for the lexisnexis individualwork instance basic usage of these jquery plugins is fairly simple with some javascript customization that allows for reliable and rapid development of the user interface. unfortunately there were a few issues with cross-browser compatibility. though jquery works in browsers such as firefox, chrome, opera, and safari, microsoft’s internet explorer was consistently problematic. although the majority of the issues were fixed, a bug in the treeview persisted. the focus property found in the styling was being overridden by some preset in internet explorer causing a hidden gif used for the selection area to be highlighted. despite attempts at numerous fixes applied to both the treeview and jquery stylesheets the bug remained. in the first version of the e-library, all of the public interface’s facets were generated in a django template and then stored as a string cache for faster serving to the end user. in ecataloger 1.2, we implemented dynamic loading of the different facet tree-views jquery plugins using ajax calls that run in the background until the trees are fully populated from the datastore. this shortens the overall load time for the application and provides a more responsive user interface. full-text searching of the titles, subjects, and descriptions of the resources is implemented using ajax in the 1.2 version of ecataloger. using the jquery framework allows us to highlight the text the user searched in a single area within the user interface. supporting the full-text searching in the datastore is done with google app engine’s searchable data-model on the server to create crude title, description, and subject associations with an endeavor. a rough weighting scheme for the results is also implemented on the server side through the use of native python data structures that weights duplicate results higher for the end user’s search query. under the current design, the remoteaccesselectronicresource’s proxy_address property contains a url for off-campus access to the resource through an authentication proxy service hosted by the library’s catalog vendor. this proxy address can differ from the remoteaccesselectronicresourcer’s access address and ecataloger 1.2 serves the proxy address automatically to any patron attempting to access the e-library from off-campus. in the ecataloger 1.2 management dashboard, the ecataloger administrator sets the ip range for the institution. when a resource is requested from a user ip that is outside that range, the proxy address is delivered to the patron instead of the access address associated with the frbr remoteaccesselectronicresource manifestation. since deploying the new e-library, the application averages over 1,100 visits a week during the school. conclusion and future directions from its beginnings as a dublin core-based e-library, the ecataloger framework has evolved into a rich native frbr-based data and application framework running on google’s cloud services. with frbr becoming more established as a bibliographic standard for catalogers, the focus and debate on its usefulness has been on how well it meets patron needs and tasks. this paper provides another reason for using frbr. the ecataloger project demonstrates the advantages and value of using frbr as an operational tool for improving library workflows as well as fulfilling the more traditional conception of frbr as an access tool to library resources in the latest version of e-library. while google app engine platform and application stack has been very useful in developing and deploying the e-library and internal applications, there have been some scalability and processing limitations with google’s free quota.  some of these limitations have been addressed with later improvements to google app engine. however, for system librarians, library students, library software developers, and others interested in how a cloud computing environment can help them offer more services and resources to their patrons and staff, ecataloger running on google app engine offers a lot of functionality for very little financial cost. the direction and development road-map of the ecataloger project has shifted because the librarian and project lead left western state college for a new position at a different institution. the ecataloger framework, having already demonstrated its utility in the daily operations of small academic library, is now being used as an informatics research platform. as a research tool, two ecataloger import modules were developed to investigate the challenges of importing external mods xml and marc files into native ecataloger frbr data classes. unit tests were built to test the import parser and it was a surprisingly easy task due to the use of mods as the descriptive data schema in the design for the ecataloger data classes. to further test the interoperability of the ecataloger framework, a suite of unit tests were created to test a marc binary record import into native ecataloger frbr data classes, by-passing any need to do an xml-crosswalk between marcxml and mods xml records. while this first iteration of the marc import module successfully imported a couple of sample marc binary records into ecataloger data objects, more work and testing is needed. the ecataloger project is looking to offer a complete open-source stack that leverages the current python code-base but is not limited by the constraints of google’s app engine service. the apache software project hadoop offers an open-source implementation of google’s bigtable infrastructure and is being investigated as a potential datastore for the next version of ecataloger. the distributed, entity-relationship nature of hadoop would allow ecataloger to scale up from its current maximum of around twenty thousand entities under the free quota of gae to a frbr-based datastore managing hundreds of thousands to millions of frbr entity objects. a full open-source stack would also allow ecataloger to leverage the work being done on open-source next generation catalog user interfaces like the ruby-on-rails blacklight project as an ecataloger alternative user interface.  future ecataloger applications could be run either as a google app engine application or as a self-hosted “cloud” application. a more robust test of ecataloger’s scalability and usability would be to import an entire library’s collection of marc records into ecataloger frbr entities. over the next year, the ecataloger project lead will begin work on the next iteration of the ecataloger with the intention of importing an entire academic library collection into a single datastore. why do this? because it is exciting to think of the potential operational efficiencies and improved user experience frbrizing an entire library would bring using ecataloger on a complete open-source stack. appendix 1 – supporting classes migration mapping original ecataloger model mapping strategy ecataloger 1.2 model countries manually associated the contained_in property to a  pre-existing continent type hierarchicalgeographic entity mods hierarchicalgeographic entity with type equal to “country” states all of the states were only associated with a single hierarchicalgeographic entity for the united states. mods hierarchicalgeographic entity with type equal to “state” cities cities contained in the united states had both state and country links, the contained_in property was set to the hierarchicalgeographic entity for that state. for cities in other countries, the contained_in property was set to the hierarchicalgeographic entity for that country. mods hierarchicalgeographic entity with type equal to “city” dcsubjects (dublin core subjects) each dcsubjects entity created two 1.2 entities, a mods subject entity and a frbr concept entity, with the the subject entity created first and its key associated with a concept entity for a 1:1 relationship. mods subject entity with authority set to “local” frbr concept entity with identification_status set to “provisional”, subject references mods subject entity genres maps to mods genre class, all genres values initially taken from the controlled marc genre terms vocabulary mods genre entity with authority set to “marcgt” organizations organization is directly mapped to a frbr corporatebody entity, the mods hierarchicalgeographic entity for the city is referenced through the corporatebody place property, the organizations entity description is first created as a mods note entity and then associated with the corporatebody entity mods note entity with type equal to “description” frbr corporatebody entity, identification_status set to “provisional”, place refers to mods hierarchicalgeographic entity, description refers to mods note titles all titles were mapped 1:1 to mods titleinfo entities, the titleinfo display_form property was set to the raw value of the titles while the titleinfo title property value was set to the all lower case of titles value. mods titleinfo with type set to “uniform” appendix 2 – resources to frbr entities mapping resource property mapping strategy ecataloger 1.2 entities base_url create mods url entity with value set to base_url mods url entity frbr remoteaccesselectronicresource entity access_address property references mods url entity creators placeholder, no data in implementation, would map to work creators property dates placeholder, no data in implementation, could map to frbr work properties, frbr events, frbr expression, and frbr manifestation depending on the type of date description create mods note entity with type set to description mods note entity with type set to “description” frbr individualwork entity notes property includes mods note entity frbr remoteaccesselectronicresource entity notes property includes mods note entity editors placeholder, no data in implementation, would map to work creators property with an frbr creator entity associated with a mods role with “editor” type genre mods genre entities created in previous step mods genre entities associated with the frbr remoteaccesselectronicresource entity genres property is_subscription manifestation property frbr remoteaccesselectronicresource entity is_subscription property is_trial manifestation property frbr remoteaccesselectronicresource entity is_trial property publisher frbr corporatebody entity already created in previous step frbr remoteaccesselectronicresource entity publisher property references frbr corporatebody rating maps to frbr remoteaccesselectronicresource popularity property frbr remoteaccesselectronicresource popularity property subjects mods subject entities and frbr concept entities created in previous step frbr concept entities associated with frbr individualwork entity subjects property titles mods titleinfo entity already created in previous step mods titleinfo entity part of frbr individualwork entity titles property mods titleinfo entity referenced by frbr selfcontainedexpression title property mods titleinfo entity referenced by frbr remoteaccesselectronicresource  manifestation title property mods titleinfo entity referenced by either frbr purchasedelectronicresourceendeavor or frbr websiteendeavor title property uids placeholder, no data in implementation, would map to mods identifier entity with the type (issn, barcode, etc.) of the uid. mods identifier entity would be associated with frbr expression, frbr manifestation, or frbr item classes or sub-classes depending on the type of uid references “appscale open source google app engine.” appscale open source google app engine. uc santa barbara racelab, dec 2009. web. 7 jun 2010. http://appscale.cs.ucsb.edu/ chang, fay, jeffrey dean, and sanjay ghemawat, et al. “bigtable: a distributed storage system for structured data.” osdi’06: seventh symposium on operating system design and implementation. seattle, wa, 2006. web. 7 jun 2010. doi:10.1145/1365815.1365816 “dcmi metadata basics.”dublin core metadata initiative. dublin core metadata initiative limited, 2010. web. 7 jun 2010. http://dublincore.org/metadata-basics/index.shtml “django – the web framework for perfectionists with deadlines.”django. django software foundation, 2010. web. 7 jun 2010. http://www.djangoproject.com/ “google app engine – google code.” google code. google inc., 2010. web. 7 jun 2010. http://code.google.com/appengine/ “functional requirements for bibliographic records.” ifla website. international federation of library associations and institutions, feb 2009. web. 7 jun 2010. http://www.ifla.org/en/publications/functional-requirements-for-bibliographic-records “frbr object-oriented definition and mapping to frbr-er.” cidoc – crm. cidoc crm special interest group, may 2007. web. 7 jun 2010. http://cidoc.ics.forth.gr/docs/frbr_oo/frbr_docs/frbr_oo_v0.8.1c.pdf “jquery : the write less, do more, javascript library.” jquery : write less, do more. the jquery project, 2010. web. 7 jun 2010. http://jquery.com/ lagoze, carl, herbert van de sompel, michael nelson, and simeon warner. “open archives initiative protocol for metadata harvesting.” open archives initiative. open archives initiative, 14 jun 2002. web. 7 jun 2010. http://www.openarchives.org/oai/2.0/openarchivesprotocol.htm metadata object description schema: mods (library of congress). washington dc: library of congress, 2010. web. 7 jun 2010. http://www.loc.gov/standards/mods/. project blacklight. web. 2 sept 2010. http://projectblacklight.org about the authors jeremy nelson is the metadata and systems librarian at colorado college. prior to working as a librarian at western state college and other libraries, he was a programmer and project manager at three different software companies. he is a graduate of knox college and the graduate school of library and information science at university of illinois. jeremy can be contacted at jeremy dot nelson at coloradocollege dot edu. alan cleary is an undergraduate student pursing computer information science and mathematics at western state college. his work primarily consists of web based applications and database management software. alan can be contacted at the dot alancleary at gmail dot com. subscribe to comments: for this article | for all articles 2 responses to "frbrizing an e-library : migrating from dublin core to frbr and mods" please leave a response below: aida, 2011-10-07 i really like the article..i just wondering how to convince all d librarians in malaysia to migrate to thinking to frbr..tq record creation for graceling by kristin cashore | sderleth, 2014-01-05 […] http://journal.code4lib.org/articles/4357 […] leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – preservation and visualization of the rural route nomad photo and video collection mission editorial committee process and structure code4lib issue 54, 2022-08-29 preservation and visualization of the rural route nomad photo and video collection this article documents the steps taken in the preservation of a personal photo and video project, “rural route nomad,” consisting of 14,058 born-digital objects from over a dozen different digital cameras used on world travels throughout all seven continents from the end of 2008 through 2009. work was done independently, “diy” if you will, with professional standards implemented in a manageable way sans the more extensive resources of a larger institution. efforts were undertaken in three main stages: preservation, dataset generation, and visualization. by alan webber this project encompasses the preservation and visualization of the rural route nomad photo and video collection, consisting of 14,058 born-digital objects from over a dozen different digital cameras during my world travels throughout all seven continents from the end of 2008 through 2009. preservation work stemmed from appraisal of my digital audiovisual archive for at-risk media, planning for immediate and future preservation, and ingesting a submission information package (sip) into a consolidated archival information package (aip). the next step was creating a dataset that inventories the contents of the newly-created aip, utilizing exiftool to generate audiovisual metadata, and then cleaning and editing the comma-separated values (csv) output using openrefine software, and finally, working with the information in microsoft excel. the end goal was to create attractive visualizations of this material using tableau software, thus i spent time utilizing tableau online tutorials to learn the basic structure and functionality of the tableau interface before sketching out the information i wanted to portray in charts, graphs and maps. i then went about designing individual visualization worksheets, going back to manipulate the dataset as needed, and compiled my visualizations into several dashboards, focused on geography and date, camera totals, and camera specs. preservation dataset creation visualization survey collection for at-risk digital photo and video formulate sip from old drives and optical discs create consolidated aip on new hdd generate csv with audiovisual metadata using exiftool through terminal clean and edit with openrefine and excel plug dataset into tableau to create map, chart and graph worksheets combine into dashboards, highlighting geography, totals, camera specs, etc. export to tableau public figure 1. workflow overview through the three stages of preservation, dataset creation and visualization [1]. preservation the genesis of this project comes from a much-needed survey of digital materials from my previous film, video and photo projects [2], along with my work as a programmer and film festival curator [3], that have been kept in near line storage (electronic records management guidelines [date unknown])  at my home studio. while there are physical master copies on tapes and celluloid prints for many of these projects, i realized in most cases there is only one digital copy stored on an external hard disk drive (hdd) or optical disc. a number of these hdds and discs are fairly old for digital media, dating back to the early 2000s in some cases [4]. i decided that all of the hdds which were older than five years old should eventually be backed-up on new multiple-terabyte drives. as optical discs have proven to be unreliable long-term media due to scratches or environmental degradation due to data layer dye, i decided that all of them needed to be preserved onto new hdds as well. since preserving all of this media would take some time, i zeroed in on the rural route nomad collection materials as the initial priority. this collection consists of photos and videos i recorded while backpacking around the world from late 2008 through 2009. my travels corresponded with my role as director of the rural route film festival for a tour of intermittent screenings on all seven continents. photos and videos were shot on over a dozen different cameras on 33 countries, capturing a multi-media portrait of the world at a particular moment in time. the project is of high importance to me, and it also lends itself to preservation, as i have archived many physical objects from that timespan (e.g., maps, travel knickknacks, my passport and vaccination card, and my lucky t-shirt). a lossless copy of all of the nomad photos, along with some video content, was kept in my home studio over the span of dozens of cd, dvd and bd optical discs. the majority of the video was stored on four hdds, along with some original minidv cassettes and memory cards, in my  offsite climate controlled storage space. in addition to these primary digital materials, i also have nine portfolio books of printed 8×10” select photographic prints and an assortment of larger-scale prints. since 2009, lossy photo copies have also been housed in four different google photos accounts [5]. my goal was to consolidate all of the lossless nomad media onto a two terabyte hdd, creating an additional copy of the files and consolidating the media in one place for easier access and analysis. i purchased a western digital drive and retrieved the video hdds from my offsite storage space. in consideration of the 3-2-1 backup rule and lots of copies keep stuff safe (lockss) (3-2-1 backup rule 2018), i would continue to keep different lossless digital copies between my home studio and the offsite space, along with the lossy copies on google photos. this preservation plan is not industry-standard perfect, as the offsite storage space is close to my home and not located in a geographically diverse enough location to prevent protection against major environmental or manmade disasters, but i am signing off on it as ‘good enough’ for now. once i had compiled all of the optical discs and hdds together as a sip [6], i began ingesting material onto my imac. i started with the optical media, utilizing a removable samsung blu-ray drive. next, i transferred the contents from each of the four hdds from offsite storage, which required appropriate firewire cables and adaptors to accommodate drives from different eras [7]. at the end of each transfer session, i copied the media on my hard drive to the new external hdd where the aip would live, and deleted the files on my computer to keep local storage space available. i performed basic fixity checks through comparison of the file size and number of items across folders through their relative paths, using cmd+i to access finder windows. there wound up being one corrupt hdd. i was able to get some files off before the drive became unreadable, but after troubleshooting through disk utility, trying different usb and firewire cable combinations, researching online and going through chat and phone support with seagate, the drive’s manufacturer, i had to send it in for further analysis. seagate was unable to retrieve the data in a usable form, thus i had to go back to the minidv production tapes and re-digitize five tapes/71.25 gigabytes of footage from china, india, nepal, and the u.s. dataset creation once i had assembled my aip, i needed to create an inventory, and i also wanted to generate descriptive information pertinent to the photographic and moving image mediums represented. using terminal, i tried employing mediainfo, but found it difficult to obtain data in spreadsheet format. instead, i went to exiftool, which returned a plethora of metadata on the photos and videos, much more than mediainfo, and i had no trouble dumping the information into a csv file. without filtering the results, i received a whopping 873 columns of data categories. going back to the command line, i chose 30 values from the exiftool documentation that i thought could be useful, as shown in figure 2. exiftool returned the narrowed down results, with 124 directories scanned and 14,668 image files read. figure 2. exiftool command in terminal. next, i took the csv file into openrefine, where i performed initial clean up of columns and rows with excess characters and integers that ran over two places beyond the decimal point, exemplified in figure 2.  after that i brought the csv data into excel. in the end, i wound up with 19 working categories/columns. overall, the dataset proved to be an ongoing, essential component of this project, as i have continued going back to add, delete and modify records in conjunction with tableau visualization software, sometimes manually, out of necessity, or via excel formulas. the version of the dataset used for the majority of published visualization work is available on zenodo for public use: https://doi.org/10.5281/zenodo.6818292. figure 3. dataset cleaning: rounding down the decimal for file size with an edit cells/transform general refine expression language (grel) command in openrefine. visualization i knew tableau was a powerful tool to create dynamic, aesthetically pleasing visuals. however, i had encountered frustrations using the software before on a tight deadline for another project [8], so i wanted to take the time to get to know the fundamentals and different possibilities. tableau seems to be considerate of technical access, providing a series of training videos specifically for the older version of the software i had to use with my mac os sierra system [9]. i spent my free research time over the course of several weeks watching the supplied videos and following along with the exercises, learning about: connecting to data, pill types, discrete dimensions, continuous measures, marks, filters, hierarchies, tooltips, calculations, design formatting and geographic information systems (gis) (free training videos [date unknown]). i had a general idea of the sort of visualizations i wanted to make, but through my research i found two useful examples of other video and photo visualization projects. ashley blewer’s media collection viewer (2021) is a custom-made programming tool that uses mediainfo json files to create bar and pie charts which depict technical statistics for audiovisual elements in a collection. randy runtsch works specifically with photography, analyzing and mapping locations using python and tableau (2021). both of these resources addressed specific aspects that i wanted to visualize within the nomad collection and encouraged me to continue through my own methods. i started by outlining worksheets on paper before jumping into tableau. i wanted to highlight specific combinations of information from the dataset in order to imply elements of narrative storytelling. the country in which the media was recorded should be seen in accordance with total files over time to show the progression of my journey, and i definitely wanted a map of geo-specific totals. pairing camera models with countries would also spur personal memories, such as which cameras got stolen in argentina, which one was the underwater camera for great white shark territory in south africa, and which one broke in india. it was also important to detail technical information from the 12+ individual cameras, as they represent a short but significant phase in the early digital/pre-smartphone era. i wanted to showcase average megapixels, lens length, aperture, iso, shutter speed, and frames per second. this information would detail particular cameras, and, ideally, could also be used for image analysis to study how to get the ‘look’ of particular shots. i encountered plenty of obstacles in realizing my ideas, which required research, trial-and-error, and more dataset tweaking. for instance, while exiftool produced a date for most digital objects, almost all of them were wrong due to faulty metadata from many of the cameras. since this footage was predominately shot on handheld digital cameras from the ‘00s, the date would have had to have been manually set in order to record properly; therefore, i had to go through manually and assign a month to each country’s photos. also, most cameras, aside from the iphone, which was only purchased in the u.s. at the end of the project, did not record gis information, requiring the manual entry of a country name for each group of files. further complicating the geographic identity was the fact that some media contained multiple intellectual objects within the same folder, or even the same file in the case of longer minidv content, leaving me to make the call in assigning a single location to that which occupied the majority of space on the file. i did a conversion of the bytes exiftool had supplied, to megabytes with excel calculations, and also manually converted video and cumulative totals to gigabytes for specific visualizations. once i had a draft of each individual worksheet in tableau, i was able to utilize individual visualizations across the workbook, making them interactively appear within appropriate tooltips (e.g., highlighted map sections as you scroll over specific timeline dates and camera makes). in the end, i wound up adding finishing touches to the project, such as representative photo ‘shapes’ for each country represented on the map by pairing them individually within a separate worksheet. figure 4.tableau map worksheet with interactive tooltip, showing country totals and representative photo. i began combining worksheets into dashboards, and finished with three main types: a high level with monthly totals and gis, one with camera totals, and one with photographic specs. in doing so, i took taras bakusevych’s 10 rules for better dashboard design (2019) into account, with tips on defining the purpose before design, and personalization over customization, leaving the potential for future dashboard edits for new users. the dashboards work well as jumping off points for deeper dives into the nomad collection data, as they link out to individual worksheets and the dataset itself, within tableau. figure 5.tableau dashboard for camera totals: make and model/country used, file type. finally, i did a data export from my tableau workbook, and saved the three dashboards to tableau public, where the information can easily be shared and accessed online: https://public.tableau.com/app/profile/alan.webber5364. while the visualizations maintain their aesthetic integrity and additional sheets can be navigated to within tableau public, i tested out working and editing within the interface and did find it considerably slower and more awkward than the desktop version. luckily, i had completed the project several weeks before my free copy expired. figure 6.dashboard for photographic specifications (aperture, focal length, shutter speed, resolution, iso) on tableau public. evaluation minidv footage was a problem at large, as exiftool did not report results for a sizeable amount of the files that are present, likely because of the way they were originally exported from final cut pro 7. a possible solution could be to try to run mediainfo on this type of footage specifically, and manually incorporate it into the dataset. the number of minidv files constitutes a very small percentage of the overall collection, as they were captured in large tape-sized chunks, however, the size of these files is substantial, and as they are a part of the collection, it is important that they are represented. as recommended by dundon et al. (2020), in the future, i aim to install bagger software to create and validate checksums, and data accessioner to manage archival content within more of a repository set-up, including the ability to generate and/or store sidecars of metadata and preservation description information (pdi). i would ideally like to crosswalk a mixture of metadata schemas, such as encoded archival description (ead), metadata and encoding transmission standard (mets), and preservation metadata: implementation strategies (premis) recommended for archives and preservation by schafer and bunde (2013), along with european broadcasting union (ebucore) which i have used in the past for moving image data. although an additional monthly bill is not appealing, i would also prospectively consider utilizing deep cloud storage, e.g. aws glacier, to provide more geographic diversity with multiple archival copies.  i recently used conifer to web archive the vlog [10] i kept in accordance with this photo and video collection, and would also like to experiment with incorporating it into a tableau visualization. currently, the dataset and visualizations are designed to serve my own needs and standards. potentially these components could be a finding aid or access point for other users to access the collection and request dissemination information packages (dips). i would, in turn, want to comb through the media first for personally identifiable information (pii) that i did not want made available. conclusion i believe that the preservation efforts, dataset and visualizations are an effective beginning to build upon over time. i identified a specific collection of at-risk media and took steps to increase its expected lifespan. the csv dataset and tableau dashboards set a solid template, holding all pertinent information, which can be used for further enhancements of this project, or can be modified accordingly for similar photo and video visualizations. the tableau worksheets present statistics in a manner which implies narrative storytelling through geography and time, specifications for a variety of early-2000s cameras, and individual photographic settings. the statistics and visualizations will assist with the use of materials from the nomad collection for new projects and exhibitions, and aid planning for future preservation projects. overall, i believe this template is an effective system with which to see the scope of technical and content aspects of media, and potentially serve as an access point for dip requests from other designated communities. notes [1] icons in diagram all from https://thenounproject.com, created by (from left to right): creative stall, garrett knoll, ted grajeda, and vectors point. [2] includes short films, music videos, featurettes, industrials, promos, travel and street photography, and screenplay materials. samples can be found at https://www.youtube.com/user/ruralroutefilms/playlists and https://vimeo.com/alanwebber. [3] primarily for the rural route film festival, dedicated to unique people and cultures overlooked by the mainstream media, https://ruralroutefilms.com/, along with myriapod productions and work done at large. [4] one example being a large lacie drive, which only holds 80 gigabytes, and requires a hefty external power plug-in, with fan which spins loudly when powered on. [5] originally picassa. [6] this report utilizes terms from the open archival information system (oais) reference model to describe preservation elements at different stages (i.e., sip, aip and dip). archaeology data service and digital antiquity provide a brief summary of oais within their guides to good practice: https://guides.archaeologydataservice.ac.uk/g2gp/app_oais. [7] ideally, a write blocker would be used in this process to assure data is not corrupted during transfer, and, especially in this case, that original timestamps are preserved, retaining information on when photos and videos were shot or edited. [8] i stopped upgrading at sierra 10.12.6 to preserve functionality of final cut pro 7, my preferred video editing software, which the majority of my older projects are in. i am, however, looking into emulation options, and at purchasing a second system to allow for better overall functionality through new updates. [9] short film online exhibition analytics research/visualization for myriapod productions. [10] vlog, spanning october 2008 to january 2010, https://ruralroutenomad.tumblr.com/page/70. references 3-2-1 backup rule [internet]. 2018. austin (tx): the state and local records management division of the texas state library and archives commission, the texas record; [cited 2022 april 6]. available from https://www.tsl.texas.gov/slrm/blog/2018/11/3-2-1-backup-rule/ bakusevych, t. 2019. 10 rules for specifications better dashboard design. ux planet [internet]. [cited 2022 april 6]. available from https://uxplanet.org/10-rules-for-better-dashboard-design-ef68189d734c blewer, a. 2021. media collection viewer. blog progress process [internet]. [cited 2022 march 25]. available from https://bits.ashleyblewer.com/blog/2021/04/03/media-collection-viewer/ dundon, k, duckworth, s, kutay, s, & vigor, e. 2020. anything is better than nothing: minimum viable actions for accessioning born-digital. [internet][santa cruz (ca)]: uc santa cruz library. pdf slides. [cited 2022 april 10]. available from https://escholarship.org/uc/item/7p0276q8 electronic records management guidelines [internet]. [date unknown]. minnesota historical society: minnesota state archives; [cited 2022 july 28]. available from https://www.mnhs.org/preserve/records/electronicrecords/erdigital.php free training videos—2019.3 [internet]. [date unknown]. tableau, learning; [cited 2022 april 11]. available from https://www.tableau.com/learn/training/20193 runtsch, r. 2021. analyze and map photo locations with python and tableau. towards data science [internet]. [cited 2022 march 28]. available from https://towardsdatascience.com/analyze-and-map-photo-locations-with-python-and-tableau-7b2a4af971eb schafer, s, bunde, jm. 2013. standards for archival description. in: prom c & frusciano t, editors. archival arrangement and description. chicago (il): society of american archivists. p. 9-85. about the author alan webber is a digital archivist at the new school, and creative director/co-founder of the rural route film festival. his background is in filmmaking and writing, along with an assortment of other experience across the fine arts and performing arts worlds. alan has an mslis with archives certificate from the pratt institute and an ma in media studies with film production certificate from the new school. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – a principled approach to online publication listings and scientific resource sharing mission editorial committee process and structure code4lib issue 9, 2010-03-22 a principled approach to online publication listings and scientific resource sharing the max planck institute (mpi) for psycholinguistics has developed a service to manage and present the scholarly output of their researchers. the pubman database manages publication metadata and full-texts of publications published by their scholars. all relevant information regarding a researcher’s work is brought together in this database, including supplementary materials and links to the mpi database for primary research data. the pubman metadata is harvested into the mpi website cms (plone). the system developed for the creation of the publication lists, allows the researcher to create a selection of the harvested data in a variety of formats. by jacquelijn ringersma, karin kastens, ulla tschida and jos van berkum introduction scholars report and document their work in the form of peeror publisher-reviewed publications. this research output, listed on websites, has become one of the most important sources for information exchange. individual scholars, research groups and entire institutes are interested in presenting all relevant data regarding their research output in an accessible and efficient manner. in the context of a new website designed for the max planck institute for psycholinguistics (mpi-pl), we developed a principled solution to dynamically render all relevant data related to a publication into publication lists on different levels of the website. we addressed the need for open access as well as the requirement to make primary data and other supplementary information simultaneously available. the publication metadata is harvested from an institutional repository, called pubman. the data is then presented on publication lists, which can be customized according to the researcher’s or the institute’s preferences. these lists contain links to the full text resources which are stored in the institutional repository, links to supplementary data which are stored in the institute’s imdi research data archive, and links to other resources. in this paper, we will first describe the two databases involved, the pubman publication repository and the mpi-pl primary data archive. we will then elaborate on the data exchange requirements for the website and the information flows in the institute. finally, the mechanisms and functionality for the data exchange between the databases and website are explained and illustrated. this paper might be of interest for librarians and information officers who are seeking a solution to help researchers present their research output elegantly bundled together. 1. publication repository and primary research data archive two databases play an essential role in our principled approach to online publication listings and scientific resource sharing: pubman from the max planck digital library and the mpi-pl primary research data archive. 1.1 institutional background the german max planck society (mps) is a large, non-profit research organization, with 76 research institutes, conducting basic research in physical science, arts, and humanities [1]. the service unit max planck digital library (mpdl) was established in 2007, with the aim to provide information management services to the mps researchers and the institute’s libraries [2]. one of the primary mpdl projects has been the development of an online publication management system (pubman), which is based on escidoc [3], an eresearch infrastructure. pubman stores, manages and delivers the mps publication data, publications and corresponding supplementary materials. the mpi-pl is one of the mps institutes [4], with some 200 researchers in 9 research groups. the technical support unit (tg) has built up extensive knowledge of technical infrastructures and archive federations. the tg maintains a digital archive which stores primary research data collected by the researchers of the institute. the data is available in the online imdi archive [5]. the scientific library of the institute has a collection of 30,000 books, 20,000 bound journal volumes, and provides online access to 30,000 scientific journals. the publication management of the institute and its individual researchers is supported by the library. 1.2 pubman pubman supports scientists and institutes of the mps in the sustainable management, re-use and digital curation of their publications. pubman is based on a service-oriented architecture, the escidoc infrastructure [3], which uses fedora as its underlying repository. objects and their relations are stored in xml [s-1]; the software supports open formats and interfaces such as rest, sru/srw and oai [9]. the publications are described with a metadata set, which is based on an extended dublin core standard [6]. the metadata set is very rich and includes elements to specify the publication details (title, abstract, date(s), doi, publication type), author details (name, affiliations), accessibility, source, and other relevant data. interoperability is guaranteed with other publication metadata standards, like mods [7]. in pubman, various document types can be described, with their corresponding bibliographic metadata. to describe the content type of the referenced text documents and to enable concise search and retrieval, the uploaded or externally referenced documents are categorized as “preprints”, “postprints”, “publisher pdf” or “supplementary material” [14]. the vocabulary used is based on the recommendations by the sherpa project [15]. as soon as the object is released, the metadata is open for search. however, the access to the publications themselves can be set to public, private or can be made available only to defined user groups. persistent identifiers (handle system [8]) are being attributed to both the metadata object and the file(s) attached. the handle system is a technology specification for assigning, managing, and resolving identifiers for digital objects and other resources on the internet. each object is assigned a (numerical) name, which is unique for that object. it allows users to bookmark the resources or create persistent links to the resources. persons are managed with a unique id in a separate service, the control of named entities service, which allows the tracking of name variants and (historic) relations of a person to organizational units. organizational units themselves are managed as objects, with unique ids, via the object handler. this allows queries for specific organizational units, such as projects, departments or institutes. various functionalities have been developed for scientists to submit their publications, in order to make imports and submissions easy. references can be imported from external databases (e.g. arxiv) or local reference management software (e.g. endnote); manual submissions can be done in a short, 3-step submission. 1.3 mpi-pl research data archive the mpi-pl research data archive contains some 250,000 objects with a total size of around 40 terabytes. the archive mostly holds linguistic data, bundled in ‘sessions’. a session may include video files, audio files as well as transcriptions or annotations (the resources). the resources all refer to the same linguistic event which is described by a compulsory metadata description, using the imdi metadata set [5]. the archive is organized according to the imdi catalogue principles, in that resource bundles pertaining to the same linguistic event are described with a set of metadata and organized in a tree structure of corpus nodes [5, 16, 17].  in the mpi archive for research data, we follow the principles of oai by having the metadata files online available (open) for browsing and searching on the web [9]. an access management system (ams) has been developed to regulate access to the resources. using ams the researcher can grant (or deny) access to individual users or groups of users, set ‘codes of conducts’ on the data to be signed by visitors, or can allow access to the data for everybody (open resources). each object (metadata file or resource) in the archive has a persistent identifier. 2. website requirements on the new mpi-pl website each researcher is assigned a set of folders and pages (the full set is named ‘person pages’) of which one folder is designed to display publication lists and one folder to display presentation lists. the same is true for research groups and projects: each of these organizational units is assigned a set of folders and pages (‘group pages’ and ‘project pages’ respectively), again two of which are designed to display publication lists and presentation lists. in the following sections of this paper we discuss the requirements and functionality designed for the publication lists. because the requirements and functionality are the same for presentations lists as they are for publication lists, we will not be discussing the presentation lists in this paper. for the website to be successful, it had to support the researcher’s normal workflow for managing publications.  this meant that the following requirements had to be fulfilled: researchers should only have to submit their publication data once and in a trusted repository; the output from the databases had to be formatted according to currently defined standards in scientific publication listing. in our institute we use the rules of the publication manual of the american psychological association[13]. it had to be possible to supply more details, like abstracts, links to full texts and/or doi’s without disturbing the initial apa styled publication list; the difference between the primary data in the mpi-pl imdi archive and the publication metadata in pubman had to be invisible for the website visitor. links to primary data and links to publication data belonging to one publication had to be listed together; the labeling of ‘default’ publication types, like, e.g., journal article, book, book chapter, proceedings papers, etc., had to conform to the standards used in our institute. the sorting of the publications on the page had to conform to standards used in a specific scientific domain. it had to be possible to sort by publication year (mostly used by psychologists) or by publication type (used by linguists). the initial information flow from the databases into the website is given in figure 1. the mpi website (www.mpi.nl) is right of the center in the blue box. in the information flow, the publication lists of the person, project and group pages are filled from the pubman database. data is entered into the pubman database by scientists, the secretaries of the research groups, and the library staff. the latter also moderates the submissions, based on a quality assurance workflow. figure1 information flow during the design phase of the website, it became clear that the pubman database also required access to primary research data. however, pubman is not designed to store larger amounts of primary data (e.g., video or audio files). the metadata set is specific for publications and the access management system is not (yet) refined enough to easily define access rights and user roles for individual users and groups of users outside the mps community. we therefore had the pubman metadata set extended by a field (locator) which is specifically designed to link to external research data. these data can originate from the mpi-pl data archive, or from other external resources. thus, pubman allows both the storage of supplementary material within the repository as well as referencing of externally located material. the third database mentioned in figure 1 is the mpi-pl people database. in this paper we do not further discuss this database, but its role is to feed the website with up to date information on the mpi-pl staff. the ‘person pages’ are created from this database. figure 1 shows that pubman may also output to other platforms, such as the annual report or publication lists for research evaluations. these publication lists may also need to appear on the website. we have described the general requirements for data submission, retrieval and interaction between the two databases. during the design phase of the website it also became clear that researchers have individual requirements and requests on the lay-out of their publication lists, such as whether or not to show certain publications on their web pages. we list below a few of these special requirements. during the project we have also learned that the number of variations on these special requirements can even be larger than the number of researchers in the institute: some researchers do not just want to display their publications lists either by year or by publication type, but by a combination of the two; the default publication types which were defined by the research community during the initial design phase could not meet the needs of everyone, therefore we had to create the functionality for the creation of publication lists that are flexible, allowing researchers to define their own set of publication types; with respect to multi-authored publications, one researcher might want it to be displayed on his web page, while the co-authors don’t; on certain pages researchers only want to display a list of their very special (golden) publications, or publications that are theme specific. 3. website implementations the website was implemented using the open source content management system (cms) plone. a plone cms is built in a zope environment. zope is an open source application server for building content management systems, intranets, portals, and custom applications. zope is written in python, a highly-productive, object-oriented scripting language [10, 11]. in plone, a website consists of a file system containing style sheets (css), content type definitions (e.g. for folders, pages, news items) and database communication scripts. each content type has a specific aim, structure and lay-out. folders are simply containers which can hold other content types; pages can contain content, news items are pages with a special format with regard to textual and image content. content types are defined in a python module containing the content type class declaration. the plone file system further contains images used in the default design of the website (e.g. banners) and the website configuration. the content of the website is stored in one data file (data.fs) in the content zope object database (zodb). the file system and zodb are joined by the zope server application creating a website. an external company (zest software, hoogvliet) was hired to implement the plone file system, special content types and scripts, and to set up the zope application server. nightly, our website cms harvests the metadata for the publications of the institute from the pubman database through a rest interface [12]. a search query is sent to the pubman server (see figure 2).  the query consists of the following elements. first the pubman rest interface is specified, next the escidoc contexts in which the mpi-pl data are stored are specified, then the export format and outformat are listed. for the needs of the cms of the mpi-pl, the output format ’snippet’ is selected, which provides the complete xml, including the bibliographic citation in the desired export format (apa). the full texts and supplementary material are not imported into the cms; however, the persistent urls to the resources are included in the metadata. an xml file containing the publication metadata is returned to the web server. http://pubman.mpdl.mpg.de/search/searchandexport?cqlquery=escidoc.content-model.objid=escidoc:persistent4 and escidoc.context.objid any “escidoc:54203 escidoc:61348 escidoc:57277″&exportformat=apa&outputformat=snippet&language=all&sortkeys=&sortorder=ascending&startrecord=&maximumrecords= figure 2 – website query to pubman database a special plone content type ‘publication-data’ was created to store the data for each individual publication. the content type ‘publication-data’ has a set of attributes which can have a value: (1) the pubman id, (2) the apa citation of the publication, (3) the abstract, (4) the publication type, (5) publication key words and (6) local tags. for each publication, the information from the harvested xml file is stored in one ‘publication-data’ item. the name of each instance of a ‘publication-data’ is equal to the file id in the pubman database. the complete set of ‘publication-data’ items are stored in a ‘publication’ folder in the plone cms. the folder is not visible within the navigation system or site set-up. its main purpose is to store the ‘publication-data’ items. the process of import and conversion from the xml file to individual ‘publication-data’ items for each publication is controlled by a python script on the server [s-2]. the next step is to show a selection of the harvested publications on the publication list of the researcher’s ‘person pages’ (or likewise on the project or group publication lists). for this we have created a special plone content type, built out of a standard plone ‘collection’. a ‘collection’ in plone works like a query does in a database. in the ‘collection,’ users can specify a set of criteria to search the website. in the regular content type ‘collection’ these criteria contain items such as ‘type’ or ‘publishing data’. for the ‘publication list collection’ in the website, the researcher performs a query on the hidden ‘publication’ folder holding all the institute’s publications. the set of search criteria is based on the requirements for publication lists, namely: author name, organizational unit, publication year, publication type, etc. researchers can create a list containing only their own publications, simply by selecting their name from the value list in the ‘author’ field. once a researcher has set his criteria, all publications subsequently entered into pubman and then harvested by the cms that match these criteria will be automatically displayed on his publication list. lists can be displayed either by ‘year’ or by ‘publication type’ (see figure 3). figure 3 publication list (by type) on the publication pages, the references are formatted in apa citation style (see figure 3), which displays the author names, publication year, title, source, and, if needed, a reference pointing to a doi or url where an online resource can be retrieved. a bibliographic reference does not give specifications on content, like an abstract. nor does it always specify a link to a publisher doi, a version of the full text, or a link to primary data that supports findings in the paper. for the researchers in the mpi-pl, it was essential that this kind of information would be available on their publications lists. when additional information is available in pubman a ‘more >’ after the apa citation provides a link to a pop-up window. in this pop-up window, the apa citation is followed by the abstract and link(s) to the persistent pubman url of the full text if the researcher has set the full text as publicly available. if the full text allows only restricted access, a request form for the material is offered. in addition, links to supplementary material are either stored in pubman or elsewhere are displayed (see figure 4). figure 4 supplementary material link to mpi-pl imdi archive in the system we developed, researchers are offered a default set of publication types. this set conforms to the standards used in our institute. these publication types are also used as headings for the publication lists by type. however, some of our researchers wished to deviate from this standard set and use publication types chosen by themselves. we made this possible through a mechanism that creates sub-folders in combination with a new metadata element. on our request the pubman mpdl development team included in the administrative metadata for publications, an extra field called ‘local-tags’. local-tags are used to group publications, such as ‘my special publications,’ and also to collect certain publications which couldn’t be selected with the standard genre types, such as ‘book review’ or ‘invited talks’. local-tags can also be used to exclude specific publications from being displayed even though they match the selection criteria. for specific list displays, local-tags are used to display a certain group of publications under the desired heading, for instance, ‘book review’. using the plone subfolder system, lists can be headed and titled by the required publication type naming matching the search criteria from the local-tag. the end result, consisting of a set of sub-folders, has exactly the same look as the default list (as shown in figure 3), but with adjusted headings. finally, the ‘local-tag’ element in the pubman database is used to create publication lists for special purposes, such as annual reports, special internal publication lists or lists for evaluations. the library staff adds a ‘local-tag’ value to the metadata of those publications which need to be on (one of) these specific lists. the lists can then be generated using the same collection mechanism as the website lists. the collection can be stored in the cms of the website and either be made public and linked in or have a restricted access for specific reviewers only. 4. conclusion researchers are assessed by their output which, in most cases, is publications. therefore views on publications, aggregated by different criteria (e.g. persons, projects, topics, “hot” papers etc.) are of major importance. the presentation of these lists, both in format and layout, depends on the carrier media (online or print) and the audience, for which they have been compiled (researcher colleagues, advisory boards, funding boards, reviewers). besides the provision of bibliographic citations, it is beneficial to provide access to the actual publication, and in most cases, the access to the corresponding supplementary material is of benefit to the audience. however, the core task of researchers is to do research and write publications. the management of the research output should be easy, flexible, and time efficient. the functionality we developed involves the storage of the publication data in one single repository: the pubman database. this database not only contains the bibliographic metadata of the publications, it also stores the full texts of the publications with the possibility to set access levels, thus providing an instrument for open access. it also provides links to the full texts through their native interfaces, like publisher repositories, and links to supplementary data stored in archives, for instance the mpi-pl primary data archive. this results in a complex item in pubman that combines all relevant data on a publication. the principled approach to publication listings is complex in information structure, but our researchers find it easy to use. both pubman and our website became functional in the spring of 2009. after some introductory sessions on the cms, the data entry in pubman and the pubman-website interface, the standard listings on the website are mostly created by the researchers themselves. for the creation of listings with special headings, or for output for other platforms, the library staff of the institute still intervenes. about 80% of our research staff, and 30% of our support staff, have pubman-based publication listings on the website. the system we have developed was designed especially for the plone/zope cms. since we are committed to the open source principle, the python script, special content types and selection mechanisms can be re-used in any plone/zope based website and can be obtained from the mpi-pl information management unit or one of the authors of this paper. we cannot judge whether the script and functionality are also suitable for other cmss; however, the principle behind it will be of interest to other information management units in research institutes. the functionality is not specific to the linguistic scientific domain in which the mpi-pl is active. by accident, we found out that the python script which harvests the mpi-pl data from pubman, also works for other scientific domains. during test phases we made a typographical error in the harvest query, which resulted in the harvest of the publication data of another max planck institute (in the physics domain). the ‘publication-data’ that were stored in our cms were correct and conform to the scientific requirements. in principle, this means that the functionality we have created has (at least) a user potential of 12,000 researchers of the mps, but might also be of interest for other scientific libraries or information management units. links [1] max planck society – http://www.mpg.de [2] max planck digital library – http://www.mpdl.mpg.de/ [3] escidoc – http://www.escidoc.org [4] max planck institute for psycholinguistics – http://www.mpi.nl [5] imdi data archive – http://corpus1.mpi.nl [6] dublin core, application profile – http://colab.mpdl.mpg.de/mediawiki/escidoc_application_profile_publication [7] mods metadata object description schema – http://www.loc.gov/standards/mods/ [8] handle system – http://www.handle.net/ [9] open archives initiative (oai) – http://www.openarchives.org [10] plone/zope – http://plone.org/ [11] python – http://www.python.org/ [12] pubman rest interface –  http://pubman.mpdl.mpg.de/search/searchandexport_rest_sample.jsp [13] american psychological association (2010) publication manual of the american psychological association. 6th ed. washington, dc: apa. http://www.apa.org/ [14] metadata categories – http://purl.org/escidoc/metadata/terms/0.1/content-category [15] sherpa project – http://www.sherpa.ac.uk/romeoinfo.html [16] the imdi metadata framework, its current application and future direction, by daan broeder, peter wittenburg international journal of metadata, semantics and ontologies (ijmso), vol. 1, no. 2, 2006 [17] broeder, d., brugman, h., & senft, g. (2005). documentation of languages and archiving of language data at the max planck institute for psycholinguistics in nijmegen. linguistische berichte, no. 201, 89-103. supplementary material [s-1] xml output of pubman query for journal paper:  van berkum, j. j. a., brown, c. m., zwitserlood, p., kooijman, v., & hagoort, p. (2005).  anticipating upcoming words in discourse: evidence from erps and reading times. journal of experimental psychology: learning, memory, and cognition, 31(3), 443-467. [s-2] python script: ‘pubman.py‘ list of figures figure 1: information flow figure 2: website query to the pubman database figure 3: publication list (by type) figure 4: supplementary material link to mpi-pl imdi archive about the authors jacquelijn ringersma (jacquelijn dot ringersma at mpi dot nl), karin kastens (karin dot kastens at mpi dot nl) and jos van berkum (jos dot vanberkum at mpi dot nl) work at the max planck institute for psycholinguistics in nijmegen, the netherlands and ulla tschida (tschida at mpdl dot mpg dot de) works at the max planck digital library in münchen, germany. the max planck institute (mpi) for psycholinguistics has developed a service to manage and present the scholarly output of their researchers. the pubman database manages publication metadata and full-texts of publications published by their scholars. all relevant information regarding a researcher’s work is brought together in this database, including supplementary materials and links to the mpi data base for primary research data. the pubman metadata is harvested into the mpi website cms and allows selected data to be presented in a variety of formats. subscribe to comments: for this article | for all articles one response to "a principled approach to online publication listings and scientific resource sharing" please leave a response below: jason p, 2010-09-12 i think the pubman project is a wonderful concept. i’ve supported it since i found out about it. leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – user-centred design and agile development: rebuilding the swedish national union catalogue mission editorial committee process and structure code4lib issue 5, 2008-12-15 user-centred design and agile development: rebuilding the swedish national union catalogue with a new generation of opacs emerging that attempt to address longstanding shortcomings, how do we make sure that we do not lose ground again in the future? this article suggests a combination of iterative development and user-centred design as a way to develop systems that will meet the constantly changing expectations of users by providing both functionality and usability. it gives a short introduction to iterative software development and user-centred design. a case study of the development of the new version of libris (http://libris.kb.se), the swedish national union catalogue, is used as an example of how these methodologies can benefit from each other in practice. by henrik lindström and martin malmsten introduction many of the present opacs have been in use for years, during which time little development and improvement has taken place. these systems have not only fallen behind in terms of functionality, but also suffer from overly complex and library specific solutions that are unfamiliar to most non-expert users. today, one expects user friendly, intuitive and aesthetically pleasing interfaces. users have also adopted new search behaviors from other services, such as google, that highly influence how they expect a library system to work. these well known shortcomings, as well as suggestions for improvements, have been discussed for several years in conferences, papers, and email lists (schneider 2006, bates 2003, calhoun 2006, numerous threads on ngc4lib [1], etc.). we know what we would like to see in the next generation of opacs, but how do we turn this into a usable system, and how do we make sure that in five years we are not back in the same position of outdated systems? on december 19, 2007, a new version of the search service for libris (http://libris.kb.se), the swedish national union catalogue, was released after just over a year of development. the system was completely rebuilt from top to bottom, including the search engine. why did we not choose an existing solution? the answer is simply that there was no system that could meet our requirements. for example, we wanted to experiment with relevancy ranking and had specific requirements regarding providing faceted browsing without compromising performance. we also wanted to avoid vendor lock-in and external dependencies. it was important for us to be in control over the whole design and development process. this also makes for the ability to continuously develop the system, even after the release. the release of libris was preceded by half a year with a public beta version, which was used for continuous development and testing. not only did we develop the application itself, we also tested and adopted new methodologies for software development. during the project an iterative agile development methodology evolved in combination with a user-centred design (ucd) process. we argue that these two methodologies allow you to meet the demands and expectations of today’s users, as well as keep up with and adapt to the constantly changing world of web applications. this article will briefly present these two methodologies and describe how they can benefit from each other. we are not the first ones to point to the benefits of combining agile development and ucd. for example, patton (2002) describes how interaction design was successfully added to an agile software development project and gulliksen et al. (2003) discuss how agile methods could be adapted to fit a user-centred design process. usability in the introduction we mentioned the importance of designing for usability. but what do we mean by a “usable” system? iso 9241-11 defines usability as: the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use (iso 1997). in the case of an opac, this implies that the system can be usable for a librarian, who is familiar with library systems, at the same time as the usability is quite poor for a first-time, non-librarian user. expert users (e.g. a librarian) and non-expert users (e.g. an undergraduate student) may differ in goals and satisfaction (the librarian wants an exact and exhaustive answer, whereas the student is content by finding the course book), the degree to which they can perform the task efficiently (the librarian is familiar with the system and knows how to perform a search, whereas the student may struggle finding what he or she is looking for, perhaps because the system differs in design and functionality from other systems he or she is used to) and so forth. therefore, in many respects, designing a usable opac is a task of designing a tool for at least two quite different user groups. for example, the non-expert users may need a simpler and more supportive system that guides them to functionality they do not know of, or are uncertain of how it works. however, this does not necessarily imply a conflict in design as long as the transparency and supportiveness of the system is provided for in a manner that does not obstruct the workflow of an expert user. generally speaking, designing for usability is a matter of finding compromises. and if you take a closer look at both the “expert” and the “non-expert” user groups you probably find them to be quite heterogeneous within themselves. however, it is important to understand who the users are and in which contexts they are going to use the system. user-centred design user-centred design (ucd) can be defined in many ways, but all definitions are characterized by a focus on the user, and on incorporating the user’s perspective in all stages of the design process. donald norman describes ucd as “a philosophy based on the needs and interests of the user, with an emphasis on making products usable and understandable” (norman 2002, p.188). by this definition, actual user involvement is not a part of ucd by necessity. however, involving the users in the user-centred design process is a common way of ensuring that their needs and interests are being met. the user-centred design process has also been formalized in the iso-standard 13407 human–centred design processes for interactive systems (iso 1999). the standard describes ucd as an iterative process consisting of five steps, depicted in figure 1, and also states the following key principles: the active involvement of users and clear understanding of user and task requirements. an appropriate allocation of function between user and system. iteration of design solutions. multi-disciplinary design teams (iso 1999). much could be said about each of these principles, but we will only briefly comment on the second and the fourth, since the other two should be quite self explanatory. “an appropriate allocation of function between user and system” means that it is important to determine which aspects of a task should be carried out by the system and which should be carried out by the user (maguire 2001). it is, for example, desirable to limit the amount of tedious time-consuming routine work for the user and instead let him or her focus on the aspects that require the users’ expertise. ucd is collaborative in its nature and implies the involvement and exchange of different competences, which leads to the last principle of multi-disciplinary teams (maguire 2001). we will return to the benefits of working across disciplines later in this article. figure 1. the human-centred design process, iso-13407 (iso 1999) in figure 1 we see a schema of the iterative user-centred design process as defined in iso-13407. in order to make the users an active part of each step in this process there are a number of empirical methods, such as: interviews surveys workshops focus groups field studies usability testing the different methods each have their strengths and weaknesses. for example, interviews can give in-depth, qualitative data, but are resource-consuming to carry out on a large number of users. surveys, on the other hand, can be distributed to a large number of subjects, but with more quantitative data as result. however, if we carry out a survey and conduct interviews to study the same thing, we can eliminate some of the weaknesses and at the same take advantage of the strengths of each method. this is known in social science as triangulation, a concept based on the idea that if you use more than one method to study one phenomenon you increase the validity of the study (jick 1979). choosing the best methods depends on what we are studying and which stage in the design process we are working on. generally speaking, interviews, surveys and field studies are well suited for the first two steps (see fig. 1), i.e. when specifying and understanding who the users are, in what context they are going to use the system, and for what purposes. it is perhaps needless to say that interviews and surveys are not very well suited for producing actual design solutions. hence for the last two steps, when designing and evaluating, we need different methods such as focus groups, workshops and prototyping. agile and iterative development iterative approaches to development have been around since the early days of software development but have not been formalized until the last few decades. iterative development is a central part of models such as rational unified process, the dynamic systems development method and agile methods, such as extreme programming and scrum (basili & larman 2003). there are many reasons why the classic single-pass waterfall model [2] is still popular, especially amongst managers; for instance, it gives you a feeling of knowing exactly what the end result will be, how much it is going to cost, etc. however, at the same time as the advantages of iterative development are becoming better documented and more widely known, the shortcomings of the single-pass, document driven waterfall model are becoming clearer. many of the iterative methodologies have, in fact, evolved from bad experiences with waterfall projects (basili & larman 2003). the biggest problem of the waterfall model is failure to adapt to discoveries made during the design and implementation phase. there is little or no room for adjustments once the analysis phase is over. also, the demand for constantly evolving products and services can be hard to meet with such a rigid model. if the duration of a project runs over a significant amount of time, chances are that the desired end-result changes during the project. simply put: the waterfall model does not take into account the fact that conditions and goals change over time. agile software development is a generic term for a number of methods using iterations to avoid the problems described above. one iteration usually lasts from one to four weeks and optimally comprises planning, analysis, design, coding and testing. one of the main points is that evaluation is done many times during the project, which enables change in direction and/or redesign at an early stage, saving time and resources (schwaber & beele 2002). the key principles of agile software development are found in the so called agile manifesto. it states that priority should be given to: individuals and interactions over processes and tools working software over comprehensive documentation customer collaboration over contract negotiation responding to change over following a plan (agile alliance 2001) since iteration in design is a key principle of ucd, there is much to gain by employing an iterative approach for implementation as well. it makes the developers more aware of the changing nature of the project, but also lets them give feedback to the design process. a focus on in-person communication (for example, sitting together in one room or not communicating via paper) also lowers the participation threshold for all team members. case study: libris – the swedish national union catalogue in this section we will describe the development of a new search service for the swedish national union catalogue, libris (http://libris.kb.se). the different methods we have used for the user-centred design process are described and exemplified, as well as the model for agile development. the focus is on these two aspects; therefore, this section is not a report on the entire libris project. it should also be noted that the user-centred design process as described here leaves out many important aspects concerning interaction design and user experience. for example, we do not touch upon visual design and communication. in the project the methodology was not formalized from the start, but rather it evolved with the project. in particular, agile development is something that we have adopted as much by experience as by actually studying documented methodologies. how we worked with user-centred design even though we have not followed the iso-13407 standard to the letter, we have worked according to many of its concepts and principles (see section on user-centred design). firstly, the design process has been iterative, as described in more detail under the section about iterative development; secondly, the project group has been multi-disciplinary, including experts on: design (visual and interaction) engineering (system architecture, databases, interface programming etc.) human computer interaction (hci) format (metadata) as much as possible, we have also included the users in every step of the design process, even though the degree and methods of participation have varied during the project. this reflects that the design process is an ongoing and changing process. the main components of our user-centred design process consisted of a survey, a focus group/workshop, and usability testing. we also conducted interviews with researchers and library colleagues and worked in close collaboration with a reference group. when talking about user-centred design we have sometimes come across the misconception that it would imply some sort of “least common denominator solution” or “design by majority vote.” however, this is not the case. the role of the project group is by no means diminished when users are involved. on the contrary, the work of conducting and analysing usability tests, focus groups, workshops, surveys, etc. makes great demands on the development team. and in the end it is up to the team to make informed decisions on how the system best can meet user preferences and demands. survey the iso-13407 standard states that the first step in the design process is to understand and specify the context of use. in order to get this background information we conducted a user survey and a focus group/workshop. the user survey was publicly accessible from the start page of the old version of libris, and during seventeen days in november 2006 we got 873 replies. the survey consisted of multiple choice questions in combination with open questions. the selection was uncontrolled and therefore the results cannot be seen as representative for libris’ users. rather, the survey aimed at gaining general background information and finding out what some of our users expect from a new version of libris. one of the most valuable outcomes of the survey was the suggestions for improvements and new features that showed up in answers to the open questions. focus group the focus group/workshop was conducted in november 2006 and aimed at gaining more qualitative data. the group consisted of 7 users: 1 pensioner, 3 doctoral students and 3 undergraduate students. the participants were recruited by postings at libraries, universities and email lists. the workshop consisted of two parts; the first part concerned search behavior and was conducted with a scenario-based method. the scenario was about an undergraduate student who had to find literature for a paper. the participants were divided into two groups with the task of coming up with a strategy for the student. the second part of the workshop consisted of a design task where the participants were asked to come up with features they would like to see in a personal login area and to describe how this area should integrate with the rest of the system. the groups presented their suggestions to each other and we discussed pros and cons of the different designs. even though the login area did not become part of the release, the workshop was valuable for getting an understanding of the users and will also provide background material for an upcoming project to design and implement a personal login area. in general, the focus group was a good method for getting qualitative data and for going into detail with some key concepts. however, the composition of the group is very important, both when it comes to profile and personality of the participants. it is also important to have a good moderator who is able to make sure that the discussion does not go off topic nor get stuck on one subject. usability testing during the project we had two extensive test periods. the first was conducted on a so called vertical prototype, where the functions we wanted to test were almost fully implemented, but the rest of the interface was inaccessible. the other test was conducted on the aforementioned public beta version. these tests were controlled and took place in the office premises of the national library. the subject was seated by a computer and the test leader orally presented the tasks. we used a “think aloud” methodology where the subject is instructed to talk out loud about how he or she perceives the system and how he or she reasons when performing the tasks. the tasks alternated between open ended and more specific. the screen was filmed with a dv-camera which also recorded the sound. in connection with the test we also conducted an interview. the interviews had two purposes: firstly to get some background information about occupation, age, and comfort with new web services, etc., and secondly to discuss in more detail how the subject experienced the system. this also gave the opportunity to ask about desirable functions and other things that were not part of the test itself. there are many methodological problems with controlled tests such as those we conducted: the formulation of the tasks strongly influences the subjects, there is a lack of real motivation to perform tasks, the environment is unfamiliar, subjects experience additional stress from being observed, etc. on the other hand, you have the possibility to pinpoint certain parts of the system and functions that you want to test. in order to get some real use data we also conducted a more naturalistic test in a library environment. the test took place at a university library and we asked visitors to conduct their searches in the beta version of libris instead of the local opac or the old version of libris. the task was therefore determined by the subjects and there was a real motivation for searching. this test mainly aimed at studying how users approached the system when seeing it for the first time and how the basic search functionality worked. we also conducted tests on a couple of librarians to see how the system was used by expert users. one important question regarding usability testing is how many test subjects you need. this depends on what you are testing and how heterogeneous the users are. however, nielsen (2000) argues that when it comes to discovering usability problems a few test subjects (he mentions 5) are sufficient in order to discover most of the problems. he also advocates running many small tests throughout the design process, rather than having one big test period. we used 4 subjects for the prototype tests, 7 for the controlled beta tests and 8 for the naturalistic tests conducted in the library. we feel that these subjects were sufficient to test the basic functionality and work flow of the system, though we could have used a couple of more subjects for the prototype tests. what we feel is lacking, in terms of testing, is a longitudinal study that examines how the system is being used by experienced users and how it functions as a working tool. how we worked with iterative development our focus on flexibility extended both to the physical environment and to our development environment. turnaround time in any part of the project needed to be short, whether making changes to the server configuration, fixing a bug, or redesigning part of the interface. it was vital that the different competences needed for these tasks were part of the project team. flexible development environment in order to facilitate rapid communication and knowledge sharing between project members, a separate project room was acquired. this room was big enough to hold all project members, including any consultants, and was also used for project meetings and reference group meetings. it was, in essence, the heart of the project. however, the room was not available to us exclusively. this meant that we had to find a way to set up and take down the project room, sometimes on a daily basis. increased knowledge sharing also led to increased security due to the fact the project success or failure did not depend on one single project member. it was also important to get the right tools for the job when it came to development. we needed to be flexible, both in terms of software and hardware. we used open source tools almost exclusively and applications were deployed on apache tomcat servers. the project team included members with experience in unix maintenance, which meant short turnaround time for server related bug fixing and reconfiguration. when needed, changes could often be made on-the-fly. since almost all development was done in java, we could choose any platform for developers. however, since the application was being deployed on hardware from sun microsystems, a unix-based development environment was considered optimal. we chose imacs from apple inc. running mac os x. the imac has the advantage of being in one piece (computer + display) and thus can be effortlessly moved, meaning that we could assemble or disassemble the project room in a matter of minutes. by design, our day-to-day communication had almost no external dependencies on people or hardware outside the project group. this led to low information latency, allowing us to make most decisions quickly and easily. an external consultant was used for one task: building browser independent css. since it was vital for us to understand and learn about the work of the consultant, he was integrated into the project rather than being an external partner. the consultant was in essence a part of the development team, albeit for a shorter time. iterative development as mentioned before, we did not have a clear view of any particular iterative development method before starting the project. we did have enough experience to understand that in order to realize our goal of continuous development, we would have to adopt some kind of iterative approach. however, in retrospect, we realize that we have fulfilled most of the individual parts of the agile manifesto (see section on agile and iterative development): individual interactions over processes and tools – the shared project room made communicating easy. our development process was deliberately malleable from the start. working software over comprehensive documentation. customer collaboration over contract negotiations – meetings with the reference group throughout the project. responding to change over following a plan – we revised our plan continually to meet changing requirements during the project. the first version that could be used to test some functionality was produced two months after the project started. after seven months of development, about half-way through the project, we released the first version of the public beta version of the service. this gave us, in addition to user input, invaluable data concerning performance and statistics. since the iterative approach encourages continuous change, there is a risk that the codebase will become unstable or that the same problem will be solved in different ways depending on when the solution was implemented. also, there is a tendency when working against a deadline to favor quick fixes in one place and to not propagate these changes throughout the code. this is natural since the “gold standard” changes during the progress of the project. to normalize the codebase we planned a 3 week long code review phase into the project. in the code review the whole codebase was put under scrutiny and discussions were made about the best way to solve a certain problem. parts of the code were then refactored [3] to meet the requirements. steps were also taken to improve the overall readability of the code. all this previously theoretical flexibility and security was put to the ultimate test one month before launch of the beta version. the computers used for development were stolen while two developers, including the technical lead, were at a conference in another country. it took the remaining project members less than two days to have everything up and running again (mostly time waiting for new computers to arrive). this proves that enough knowledge was spread among the project members. even though strictly this has nothing to do with iterative development, we are convinced that the constant cycle of plan/design/implement/deploy helped iron out any hidden problems. discussion the methodology outlined in this article requires engagement from everyone in the project group; there is no room for hiding or not doing one’s share. this can be strenuous at times, but in general we believe that it leads to devotion and a greater sense of satisfaction. furthermore the highly collaborative way of working ensures that any problems that may arise are noticed at an early stage. by focusing on user-centred design, you develop an understanding of the user, and thereby an understanding of why and for whom you are developing the system. this, we have found, is highly motivating, whether you are doing visual design, writing code or working with metadata specifications. agile methods have been criticized for not taking usability into consideration, with a too narrow focus on deliverable code (gulliksen et al. 2003). however, even though the agile scrum methodology does not say anything about ucd, we do not see anything inherent in the methodology that prevents a focus on usability (schwaber & beele 2002). at the same time that the ucd process ensures an understanding of the users, the agile development model ensures that you can work iteratively. the two methods reinforce each other; the latter makes for faster development of functional prototypes, which are more easily communicated and tested, thus giving you better input for the next iteration, and so forth. an interdisciplinary project group also gives an understanding of one’s colleagues and their different competences. this is, in our opinion, almost a prerequisite for working iteratively. our experience from other projects is that external dependencies slow down the process. in the libris project, we could implement new features in very small and effective iterations within the project group, doing everything from specifying and implementing metadata transformations to designing and implementing the interaction in the user interface. furthermore, this could be done without having communication delays sending specifications and bug-reports back and forth with external consultants or development teams. and by communicating directly with each other in person, the amount of intermediary paperwork and lengthy email conversations is reduced. it is important to stress the need for management support and trust when working with agile development. since there is no final specification of requirements to show up front, before the project starts, you have to convince management that the methodology and process themselves will lead to a good product. what could we have done differently? of course, there is no such thing as a perfect methodology; especially when trying new approaches you are bound to do some things backwards and run into difficulties now and then. however, during the libris project problems and drawbacks have been surprisingly absent. that is not to say that the methodology could not be improved. sometimes the work has been a bit unstructured and perhaps ad hoc. this is not necessarily a problem with the methodology itself, but rather a lack of planning and of making sure that each and every design decision is well founded. now, at the end of the project, we also feel that there is a lack of documentation. while there has been quite a lot of paperwork, some parts of the system are quite obscure for anyone other than the person who wrote the code. this is also a result of lack of knowledge transfer between the programmers. agile methods such as extreme programming stress the importance of pair programming, and this is something that we have to work more with in coming projects. the user-centred design process could be improved in many aspects, not least in making the users even more involved in the design process. during the project, a few team members had most of the contact with the users, and it would be interesting to conduct more collaborative design workshops with the entire project group and the users working together. the methodology of making the users an active part of the design process is called participatory or cooperative design, and is also known as the “scandinavian model” (bødker et al. 2000). even though we had from previous experience a good knowledge of the context of use of the system, we could also have done more toward specifying and understanding this context, e.g. by doing field studies. we also feel we could have devoted even more time for usability testing, especially by doing more naturalistic tests with real tasks in real contexts. conclusion the fact that we had a version of the system with all essential parts integrated at an early stage in the project, in combination with the six months of beta-testing, made us confident that the system would work. the continuous user tests and user-centred design process also gave us confidence that the system would be well received by the users. furthermore, the iterative development model in combination with an efficient release process put us in a position were we can easily update and add new features to the live version. this allows us to have a service that can meet the continuously changing demands on web services in general, and on library systems in particular. finally, we would like to conclude that working with user-centred design in combination with iterative development is a better, faster and cheaper way of software development, compared to traditional models. better – the product being released at the end is a more up-to-date and bug-free version than had we worked with a more traditional approach. faster – it is our conviction that with traditional methodology we would not have finished on time, or at least not with the same amount of features implemented. cheaper – if the same number of people are able to do a better job in a shorter amount of time, it is a more cost-effective way of getting the job done. references agile alliance (2001). manifesto for agile software development. webpage available at: http://agilemanifesto.org/ (2008-03-05) basili, victor r. & larman, craig (2003). iterative and incremental development: a brief history. ieee computer. 36(6), p.47-56. doi:10.1109/mc.2003.1204375. available online at http://www2.umassd.edu/swpi/xp/articles/r6047.pdf (2008-02-28) bates, marcia j. (2003). improving user access to library catalog and portal information. final report (version 3). available at: http://www.loc.gov/catdir/bibcontrol/2.3batesreport6-03.doc.pdf (2008-03-31) bødker, ehn & sjögren, sundblad (2000). co-operative design – perspectives on 20 years with ‘the scandinavian it design model’. proceedings of nordichi. available at: http://cid.nada.kth.se/pdf/cid_104.pdf (2008-03-28) calhoun, karen (2006). the changing nature of the catalog and its integration with other discovery tools. prepared for the library of congress. unpublished. available at: http://www.loc.gov/catdir/calhoun-report-final.pdf (2008-03-31). gulliksen, göransson, bovie, blomkvist, persson & cajander (2003). key principles for user-centred systems design. behaviour & information technology. 22(6), p. 397-409 (coins) iso (1997). iso 9241-11: ergonomic requirements for office work with visual display terminals (vdts). part 11 – guidelines for specifying and measuring usability. geneva: international standards organisation. also available from the british standards institute, london. (coins) iso (1999). iso 13407: human–centred design processes for interactive systems. geneva: international standards organisation. also available from the british standards institute, london. (coins) jick, todd d. (1979). mixing qualitative and quantitative methods: triangulation in action. administrative science quarterly, 24(4), p.602-611. doi:10.2307/2392366 (coins) maguire, martin (2001). methods to support human-centred design. int. j. human-computer studies. 55, p. 587-643. doi:10.1006/ijhc.2001.0503. (coins) nielsen, jakob (2000). why you only need to test with 5 users. webpage available at: http://www.useit.com/alertbox/20000319.html (2008-03-18) norman, donald a. (2002). the design of everyday things. new york: basic books. isbn: 0-465-06710-7 (coins) patton, jeff (2002). hitting the target: adding interaction design to agile software development. oopsla 2002 practitioners reports, p. 1-ff. doi:10.1145/604251.604255. isbn: 1-58113-471-1 (coins) schneider, karen g. (2006). how opacs suck, part 2: the checklist of shame. blog entry available at: http://www.techsource.ala.org/blog/2006/04/how-opacs-suck-part-2-the-checklist-of-shame.html (2008-03-31) schwaber, ken & beedle, mike (2002). agile software development with scrum. upper saddle river, nj: prentice hall. isbn: 0-13-067634-9 (coins) notes [1] ngc4lib is a mailing list for discussions about the next generation catalogs for libraries. see http://dewey.library.nd.edu/mailing-lists/ngc4lib/ (2008-03-31). [2] the waterfall method generally involves a single iteration where all requirements are defined and system design is completed before any development occurs. [3] code refactoring is revision and restructuration of source code intended to make it more streamlined and to increase its readability. about the authors henrik lindström is a developer at the national library of sweden. he holds a m.sc. in media technology from the royal institute of technology, stockholm, sweden. henrik works with programming and interaction design, and has a special interest in language technology and machine learning. email: henrik.lindstrom@kb.se martin malmsten is a senior developer at the national libary of sweden. he is currently working on the linked data implementation for libris, the swedish union catalogue, and has a special interest in semantic web technology and search engines. email: martin.malmsten@kb.se tags: agile software development, iterative development, usability, user-centred design subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – fedora commons with apache hadoop: a research study mission editorial committee process and structure code4lib issue 22, 2013-10-14 fedora commons with apache hadoop: a research study the digital collections digital repository at the university of maryland libraries is growing and in need of a new backend storage system to replace the current filesystem storage. though not a traditional storage management system, we chose to evaluate apache hadoop because of its large and growing community and software ecosystem. additionally, hadoop’s capabilities for distributed computation could prove useful in providing new kinds of digital object services and maintenance for ever increasing amounts of data. we tested storage of fedora commons data in the hadoop distributed file system (hdfs) using an early development version of akubra-hdfs interface created by frank asseg. this article examines the findings of our research study, which evaluated fedora-hadoop integration in the areas of performance, ease of access, security, disaster recovery, and costs. by mohamed mohideen abdul rasheed introduction digital content managed by libraries is growing rapidly. libraries are already moving from terabyte to petabyte scale storage platforms (altman et al. 2013). this huge volume of digital content makes its storage and maintenance a challenging task. native filesystems do not scale to the demanding requirements of today’s digital libraries. libraries, in order to meet their terabyte scale storage requirements, commonly use networked storage solutions such as san and nas. these storage platforms isolate the storage from the application server so that the high volume of access requests is not choked by the application server’s storage bottlenecks. the university of maryland libraries is currently reevaluating its digital library infrastructure to meet its growing needs. in this research study we investigate a scalable storage platform for the libraries’ digital collections repository. apache hadoop, an open source big data platform that provides a scalable, robust, and distributed storage and processing framework, is our primary contender for numerous reasons. this article summarizes our evaluation of hadoop’s ability to serve as a backend-storage for fedora commons repository. what is hadoop? hadoop is a distributed computing framework. hadoop distributed file system (hdfs) and mapreduce are the two key components of hadoop. hdfs is a solid storage platform that can easily handle petabyte scale data. major features of hdfs include data integrity, availability, and fault-tolerance. mapreduce is a distributed processing framework that can distribute tasks to the nodes in the cluster, and aggregate the results efficiently. for any application to utilize hadoop’s distributed processing framework it has to be designed conforming to the mapreduce application architecture. why hadoop? hadoop is highly scalable. the ability to scale horizontally without compromising performance is an important trait of hadoop. clusters as large as 4500 nodes exist at yahoo. hundred node hadoop clusters are a commonplace (poweredby – hadoop wiki … [updated 2013]). fault-tolerance is another noteworthy feature of hadoop. it can handle disk, node, or network failures elegantly. in case of a node failure, hdfs automatically copies data that fall below the configured replication factor to other nodes to ensure data availability. failed nodes can be replaced without downtimes or manual restoration operations. hdfs can also be configured to automatically balance data across nodes. in addition to the hadoop hdfs data storage, hadoop’s mapreduce serves as a powerful data processing engine. digital library processes involving computationally intensive tasks such as image format migration, document recognition, or pattern identification could benefit from mapreduce. for example, mapreduce applications can be designed with facial recognition tools to generate complementary metadata for digital image collections. hadoop has an active group of dedicated contributors and it is supported by a host of successful startups. hadoop’s ability to run on commodity hardware, efficient large-file handling, and ever-growing capabilities are other factors that made hadoop a top candidate for our investigation. architecture akubra, a pluggable storage module, was developed under a collaborative effort between fedora commons and topaz, an open source content modeling and storage software framework project. it was designed to provide a consistent low-level storage interface across fedora commons and other related projects. akubra was first incorporated into fedora commons in version 3.2 and has been the default storage interface since version 3.4. it allows fedora to connect to any storage platform that has a compatible interface implementation. popular platforms such as irods and dell dx object storage have akubra compatible interfaces available [1]. akubra-hdfs (asseg et al. 2012), an experimental project by frank asseg, provides a basic akubra interface implementation for hdfs [2]. we used akubra-hdfs augmented with some customizations for this experiment. akubra-hdfs is available as an open source project. although the akubra-hdfs interface is functional, it needed a few customizations to successfully integrate fedora with hadoop. we enhanced akubra-hdfs by adding a cache feature, making it more configurable, and enabling support for secure hadoop clusters. the cache feature can be used to selectively cache files from hdfs to the local filesystem based on the path suffix. we adapted android image cache [3], a library that caches images to an android device’s local storage for conserving network bandwidth, to serve as a simple linux file cache [4]. next, the enhancements allow us to utilize the standard hadoop configuration files for configuring the akubra-hdfs’s hadoop client. all our customizations are available on github [5]. we intend to contribute the enhancements back to the original project. hdfs is optimized for large objects. however, for small files the performance efficiency could be undermined by the network overhead of hdfs (white 2009). the cache module can overcome this problem to a certain extent. we have configured the akubra-hdfs to cache image thumbnail files to the fedora server’s local disk. the cache module serves hdfs requests for configured files from the cache on local disk instead of retrieving them from hdfs. on a request for a file that is not yet cached, the requested file is retrieved from hdfs and added to the cache. fedora is very flexible in configuring the storage interface. the objects and datastreams can be configured to use different storage platforms. in this case, we configured fedora to use akubra-hdfs for the datastreams, and the local filesystem for the objects. this again limits the number of small files that need to go into hadoop. figure 1. high level architecture of the fedora-hadoop integration. hadoop architecture typically, a hadoop cluster consists of a single namenode and multiple datanodes. hadoop can handle datanode failures very well, but if the namenode fails the entire cluster goes down. to overcome such a scenario a hot-standby namenode can be added to the cluster making the namenode highly available. the standby node will always be in sync with the active namenode and it can automatically take over the cluster when the active namenode fails. the namenode manages the metadata of the hdfs, whereas the datanodes host the actual content. all client requests to hdfs are first processed by the namenode, and it redirects clients to appropriate datanodes. hadoop uses redundancy to achieve fault tolerance, storing copies of the same data on different nodes so that when one node fails the data can still be retrieved from other locations. the number of copies hadoop stores is configurable, with a default setting of three copies. the effective storage capacity of the cluster becomes one-third of the actual capacity under this configuration. hadoop uses an xml based configuration framework. every node in hadoop needs matching, correctly configured xml files for proper functioning of the cluster. we used cloudera standard [6], an easy to use tool to deploy and manage hadoop clusters, from cloudera [7], to administer the test cluster for this research. performance to understand the performance of hadoop storage, it needs to be contrasted with the regular storage. we benchmarked the performance of hdfs backed fedora against fedora with the default storage platform. the fedora server application is run on a linux virtual machine with san provisioned storage. see appendix i for specification of the systems used in this research. we used apache jmeter [8], a performance-testing tool, to conduct the tests. jmeter can simulate high workload conditions by generating large numbers of parallel requests to the server. jmeter captures different aspects of the response from the server to provide valuable performance information. create object, delete object, create datastream, read datastream, and delete datastream operations were benchmarked. as expected the create object and delete object operations had similar performance on both hdfs and local filesystem based servers, since the fedora objects were configured to be stored in the local storage in both cases. the delete datastream operation also had a close performance on both servers, as it only involved metadata operations. the create datastream and read datastream operations showed significant performance improvement on the hdfs backed server. figures 2 and 3. throughput comparisons of the fedora server with local filesystem and hdfs. the charts above depict the throughput of the fedora server with local filesystem and hdfs for 200 parallel requests generated by jmeter. multiple jmeter clients were used to maximize the parallelism of the test requests. the results are dependent on factors such as network bandwidth, disk performance, and san performance, so it cannot be generalized that hdfs storage will always provide better performance. rather, to obtain a more accurate comparison the above-mentioned factors pertaining to the specific environment should be studied. we have hosted our jmeter test plans on github [9]. the test plans can be used to rerun the performance tests on any fedora server. ease of access easy access to the storage is an important requirement for any filesystem. digital collections staff, and other potential users would need direct hdfs access to perform day-to-day data handling operations. the functionality to natively mount hdfs to users’ workstations is currently limited. however, additional services can be installed on the server side that allows easier access to the filesystem. the available options include hue file browser fuse-hdfs httpfs nfs and webdav hue file browser hue [10], hadoop user experience, is an open source graphical user interface for hadoop developed by cloudera. the hue file browser exposes a web-accessible interface to hdfs. it is rich in functionality and very intuitive. the interface can be used to upload, download, or delete files and folders in hdfs. the file upload is simplified by the drag and drop interface. a folder should be zip archived before upload, and hue will automatically extract the archive once uploaded. the file browser’s features like permissions management and text editors are very handy [11]. fuse-hdfs fuse [12], filesystem in userspace, allows external filesystems to be mounted to unix machines. it provides users the convenience of accessing remote filesystems similarly to local filesystems. the fuse-hdfs module can be used to mount hdfs to linux machines. an easy guide to setup fuse-hdfs is available in the cloudera website [13]. fuse-hdfs is not recommended as a production ready utility, as there are several disadvantages. the performance concerns of fuse mounted filesystem and the lack of support for windows os users are notable. httpfs httpfs is a proxy service that allows clients to access hdfs via a http rest api. file transfer operations in httpfs can be performed using a command line tool like curl. a custom graphical user interface can be built on top of the rest api to provide users with a remote file client. alternatively, httpfs can also be accessed using the hadoop filesystem api for easier integration with other applications. httpfs service has built-in security to authenticate clients on secure clusters. also, as a proxy server it improves security by eliminating datanode visibility to clients outside the firewall (hadoop hdfs over … [updated 2013]). nfs and webdav nfs and webdav plugins for hdfs are in very early stages with nfs support targeted for hadoop 3.0. the hadoop project tracker shows an active development effort in progress for the nfs module [14]. the webdav module looks like an orphaned project with little or no attention in the past few years [15]. security authentication hadoop clusters can be secured using kerberos authentication. by default, the security option is disabled to facilitate easy deployment and testing, but this leaves the cluster vulnerable to intruders. after successfully deploying the cluster, kerberos security can be enabled by configuring the necessary properties in the hadoop xml configuration files. also, the kerberos client needs to be installed and configured in all the nodes. once security is enabled, all hadoop services will need kerberos security credentials to communicate with each other. the cloudera standard cluster management tool makes this process straightforward [16]. the hdfs clients need to have valid tokens from the kerberos server to be able to access the secure cluster. many libraries might already be using kerberos for authenticating their users in other applications. the same kerberos server can be used to authenticate the hadoop users. however, for authenticating hadoop processes and services, a dedicated kerberos server is recommended. this will prevent any accidental overload caused by faulty mapreduce user applications from affecting non-hadoop services that depend on the same kerberos server. when two separate kerberos servers are used, a trust relationship needs to be established between them [17]. though kerberos adds reliable security to a hadoop cluster, hadoop is still evolving its security infrastructure to make it more flexible, fine-grained, and interoperable. intel’s project rhino [18], cloudera’s sentry [19], and hortonworks knox [20] are major security improvisation efforts undertaken by a few vendors. access control once kerberos establishes the identity of users, hadoop allows users to access the filesystem in accordance with their permissions. hdfs uses an authorization model similar to posix. file and folder permissions can be set using the read, write, and execute flags (permissions guide … [updated 2013]). users & groups by default, hadoop uses unix users and groups information for managing access to hdfs. hadoop expects all user accounts to exist in every node. creating and managing user accounts in each node of the cluster will not be practical. instead, hadoop can be configured to use ldap for user information. as most institutions already use ldap or compatible services to manage their users, this will help simplify and centralize hadoop user management. hadoop can either use ldap or unix user management, but not both simultaneously. quotas hdfs provides capabilities to limit the storage usage on a per directory basis. space quotas limit the maximum number of bytes a directory and its subdirectories can hold and name quotas limit the maximum number of files a directory and its subdirectories can have. unlike other storage platforms, quotas in hdfs are not user or group specific. the dfsadmin tool can be used to set, clear, and monitor quotas in hdfs. more information on quotas can be found at the hadoop documentation website [21]. disaster recovery hadoop does not have a solid disaster recovery option yet. currently, a secondary cluster needs to be used for disaster recovery, which is very inefficient. in using the secondary cluster, hadoop users can choose between a mirrored cluster and a passive backup cluster. in the mirrored cluster approach, every transaction is duplicated to two clusters, and all the processing happens twice. this option is suitable for highly critical systems where data loss is completely unacceptable. in the backup cluster approach, the data from the main cluster is periodically backed up to the second cluster using distcp, a mapreduce based parallel copy tool. systems that are less stringent with data recovery requirements can use the second option, as there is a risk of losing data changes after the last backup (ranganathan 2013). efficient disaster recovery options need to be built into hadoop. meanwhile, regular backup options such as offsite tape backups can be used. hdfs can be mounted to linux using fuse-hdfs, and then backed up using regular tape backup tools. similarly, other disaster recovery tools can be used with the mounted hdfs. however, this approach might not be performance efficient. snapshots snapshot is an essential feature for the digital library storage. it helps recover data lost in file corruptions, accidental user errors, and other such scenarios. hadoop does not have the snapshot feature in the current version. the snapshot feature is being actively developed, and it will be available in the near future (sze 2012). the initial versions will support read-only snapshots that are sufficient for the restore functionality. the read-write support will be added consecutively. the status of the feature can be followed at its project tracker page [22]. fedora’s built-in content versioning functionality should be relied upon until the hdfs snapshot feature is ready. costs cost is one of the deciding factors for adoption of any technology. this is especially true with budget cautious libraries. running on low-cost hardware is one of the features hadoop touts. hadoop is designed to handle failures, and it does not require high-cost fault tolerant hardware. a well-planned cluster with well chosen hardware can balance between cost and performance. the primary objective of the hadoop cluster here is to provide a scalable and robust storage service. for storage-optimized clusters, each node should have multiple high volume storage disks with sufficient memory and processing resources. though some servers can support a large number of storage disks, there are other factors that limit the number of disks per node. data distribution will be impaired by too much storage per node. cloudera recommends staying within twelve 1tb or 2tb disks per node (loddengaard 2010). data locality is an important factor for mapreduce applications to succeed, and it can be compromised on a under-distributed cluster (lin and dyer 2010). a well-designed cluster would have maximized resource usage, and reduced investment on under utilized resources. the chart below represents the cost per gigabyte and the number of nodes of a hadoop cluster proposed for university of maryland libraries. the cost for the initial 8tb is very high as it includes the cost for two namenodes and three 8tb datanodes. for the next 16tb, there is no new node added. the existing datanodes would be upgraded to 24tb each. after that, single 24tb datanodes will be used to add additional capacity to the cluster. please refer to appendix ii for the costs of the individual nodes. figure 4. hadoop storage cost per gigabyte for the university of maryland libraries. next, we investigated the costs involved in setting up an amazon aws [23] cloud based hadoop cluster. amazon aws has a complex pricing structure that depends on multiple factors including network bandwidth, storage, and cpu. this makes it difficult to estimate the overall expense. we calculated the aggregate costs of the services for different usage levels and system specifications. in almost every scenario, the costs were very high. the large amount of persistent storage required for the digital collection was one of the main factors that drove the cost up. please see appendix iii for the aws cost tables. amazon provides elastic mapreduce (emr) [24] instances that are optimized for mapreduce processing services, but they are not well suited for storage-centered clusters. therefore, we used the standard aws cloud instances to build and evaluate the hadoop. although it would require additional management overhead, running the cluster in-house is the cost-efficient and propitious option for this scenario. conclusion this research study explains the real value hadoop can bring to libraries. the first hand experience integrating and testing fedora with hadoop shows where hadoop does well, and where it doesn’t. though hadoop performed well in our experiments, there are a few major areas where it falls short. the lack of a straightforward disaster recovery option is the biggest drawback. there seems to be alternative means to implement disaster recovery using fuse-hdfs, but it’s not proven. next, the performance tests conducted were on a very small scale – a few gigabytes. hadoop scales well, but the performance of akubra-hdfs for large-scale deployments cannot be extrapolated. akubra-hdfs might need more tests and optimizations before it can be production ready. future research we are upgrading our test cluster with more nodes and additional resources. this will allow us to run more realistic tests on the cluster, and analyze other critical requirements for hadoop backed digital libraries. the viability of tape backups with hadoop needs to be examined. this will help identify the challenges and potential solutions in hadoop disaster recovery. next, the potential for hadoop hbase [25] to serve as the fedora object store needs investigation [26]. it would be desirable to use a single storage for all the data so that the overall maintenance and backup processes are simplified. currently, the fedora objects are configured to use the local filesystem to mitigate the hdfs small files problem. hbase, the distributed database that runs on top of hdfs, handles small files very well. this will consolidate all data to be stored within hadoop. hbase can also be tested with other data-driven applications that can profit from its strengths. hadoop solr is another interesting tool to try. it is an implementation of apache solr, the popular enterprise search platform, built to work directly on hdfs. hadoop solr uses the cluster to provide a scalable and highly available search service. lastly, the complementary benefits hadoop can bring to digital libraries needs to be explored. the workflows in the digital object lifecycle that can potentially benefit from hadoop’s distributed processing should be identified and assessed. the workflows that involve time-consuming processes could be good candidates to start with. notes [1] akubra project. available from: https://wiki.duraspace.org/display/akubra/akubra+project. [2] fasseg/akubra-hdfs. available from: https://github.com/fasseg/akubra-hdfs [3] mimtel/android-image-cache. available from: https://github.com/mitmel/android-image-cache [4] umd-lib/filecache. available from: https://github.com/umd-lib/filecache [5] umd-lib/akubra-hdfs. available from: https://github.com/umd-lib/akubra-hdfs [6] cloudera standard. available from: http://www.cloudera.com/content/cloudera/en/products/cloudera-standard.html [7] cloudera, inc. available from: http://www.cloudera.com/content/cloudera/en/home.html [8] apache jmeter. available from: http://jmeter.apache.org/ [9] jmeter test plans. available from: https://github.com/umd-lib/jmeter-testplans [10] the ui for apache hadoop. available from: http://cloudera.github.io/hue/ [11] demo: hdfs file operations made easy with hue. available from: http://blog.cloudera.com/blog/2013/04/demo-hdfs-file-operations-made-easy-with-hue/ [12] filesystem in userspace. available from: http://fuse.sourceforge.net/ [13] mountable hdfs. available from: http://www.cloudera.com/content/cloudera-content/cloudera-docs/cdh4/latest/cdh4-installation-guide/cdh4ig_topic_28.html [14] support nfsv3 interface to hdfs. available from: https://issues.apache.org/jira/browse/hdfs-4750 [15] expose hdfs as a webdav store. available from: https://issues.apache.org/jira/browse/hdfs-225 [16] configuring hadoop security with cloudera manager. available from: http://www.cloudera.com/content/cloudera-content/cloudera-docs/cm4ent/latest/configuring-hadoop-security-with-cloudera-manager/configuring-hadoop-security-with-cloudera-manager.html [17] set up a local kdc and default domain for the hadoop cluster. available from: http://www.cloudera.com/content/cloudera-content/cloudera-docs/cm4ent/4.5.3/configuring-hadoop-security-with-cloudera-manager/cmeechs_topic_4_3.html [18] project rhino. available from: https://github.com/intel-hadoop/project-rhino/ [19] sentry. available from: http://www.cloudera.com/content/cloudera/en/products/cdh/sentry.html [20] apache knox gateway. available from: http://hortonworks.com/hadoop/knox-gateway/ [21] quotas guide. available from: http://hadoop.apache.org/docs/stable/hdfs_quota_admin_guide.html [22] support for rw/ro snapshots in hdfs. available from: https://issues.apache.org/jira/browse/hdfs-2802 [23] amazon web services. available from: https://aws.amazon.com/ [24] amazon electronic mapreduce. available from: http://aws.amazon.com/elasticmapreduce/ [25] apache hbase. available from: http://hbase.apache.org/ [26] fasseg/akubra-hbase. available from: https://github.com/fasseg/akubra-hbase references altman m, bailey j, cariani k, gallinger m, mandelbaum j, owens t. 2013. ndsa storage report: reflections on national digital stewardship alliance member approaches to preservation storage technologies. d-lib magazine [internet]. [cited 2013 may]; 19:5/6. available from: http://www.dlib.org/dlib/may13/altman/05altman.html poweredby – hadoop wiki [internet]. [updated 2013 jun 19] forest hill (md): apache software foundation; [cited 2013 aug 13]. available from: http://wiki.apache.org/hadoop/poweredby asseg f, razum m, hahn m. 2012. apache hadoop as a storage backend for fedora commons. in: or2012. the 7th international conference on open repositories; 2012 jul 9-12. available from: http://www.scape-project.eu/publication/apache-hadoop white t. 2009. the small files problem [internet]. palo alto (ca): cloudera, inc; [cited 2013 aug 13]. available from: http://blog.cloudera.com/blog/2009/02/the-small-files-problem/ hadoop hdfs over http [internet]. [updated 2013 jun 8] forest hill (md): apache software foundation; [cited 2013 aug 13]. available from: http://hadoop.apache.org/docs/current/hadoop-hdfs-httpfs/ permissions guide [internet]. [updated 2013 aug 4] forest hill (md): apache software foundation; [cited 2013 aug 13]. available from: http://hadoop.apache.org/docs/stable/hdfs_permissions_guide.html ranganathan j. 2013. hadoop backup and disaster recovery. big data techcon boston; 2013 april 8-10. available from: http://www.slideshare.net/cloudera/hadoop-backup-and-disaster-recovery sze n. 2012. community-driven snapshot for hdfs – part two [internet]. palo alto (ca): hortonworks; [cited 2013 aug 13]. available from: http://hortonworks.com/enterprise-ready-community-driven-apache-hadoop/ loddengaard a. 2010. cloudera’s support team shares some basic hardware recommendations [internet]. palo alto (ca): cloudera, inc; [cited 2013 aug 13]. available from: http://blog.cloudera.com/blog/2010/03/clouderas-support-team-shares-some-basic-hardware-recommendations/ lin j, dyer c. 2010. data-intensive text processing with mapreduce [internet]. san rafael (ca): morgan & claypool publishers. p. 26-28. available from: http://lintool.github.io/mapreducealgorithms/ about the author mohamed is a software developer at the university of maryland (umd) libraries. he is a recent alumnus of umd’s ischool, where he earned his masters in information management. author’s email: mohideen@umd.edu linkedin: http://www.linkedin.com/in/mohideenrasheed appendix i. system specifications servers fedora fedora server runs on a linux virtual machine. cpu: intel xeon x5670 2.93ghz (quad core) memory: 4gb storage: 80gb (san storage) os: centos 6.4 test hadoop cluster dell optiplex 755 desktop machines are used for hadoop.. name nodes: 2 (1 active, 1 failover) data nodes: 3 dell optiplex 755 cpu: intel e2200 2.20ghz (dual core) memory: 4gb storage: 80gb hdd os: centos 6.4 kerberos kerberos server runs on a linux virtual machine. cpu: intel xeon x5570 2.93ghz (dual core) memory: 4gb storage: 26gb (san storage) os: centos 6.4 software versions fedora version 3.4.2 hadoop cloudera standard 4.6.0. (earlier known as: cloudera manager) cloudera distributed hadoop cdh 4.3. kerberos version 5 ii. hadoop cluster costs cost of dell poweredge rack servers: namenode server – $5299usd cpu: 2 intel xeon 2.53ghz 6 core memory: 64gb hdd: 500gb * 3 8tb datanode server – $3368usd cpu: 2 amd opteron 3.1ghz 8 core memory: 16gb hdd: 2tb * 4 24tb datanode server – $5987usd cpu: 2 amd opteron 3.1ghz 8 core memory: 16gb hdd: 2tb * 12 cost of 2tb hdd – $325usd the dell website is used to calculate the prices (calculated on april 2013). iii. aws pricing the costs below were calculated on may, 2013 from amazon aws website and represented in us dollars. figure 5. instance cost (replication factor of two is used. the effective storage is half the actual storage). figure 6. instance data transfer cost figure 7. s3 storage cost figure 8. s3 data transfer cost figure 9. s3 request cost subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – ganch: using linked open data for georgia’s natural, cultural and historic organizations’ disaster response mission editorial committee process and structure code4lib issue 50, 2021-02-10 ganch: using linked open data for georgia’s natural, cultural and historic organizations’ disaster response in june 2019, the atlanta university center robert w. woodruff library received a lyrasis catalyst fund grant to support the creation of a publicly editable directory of georgia’s natural, cultural and historical organizations (nchs), allowing for quick retrieval of location and contact information for disaster response. by the end of the project, over 1,900 entries for nch organizations in georgia were compiled, updated, and uploaded to wikidata, the linked open data database from the wikimedia foundation. these entries included directory contact information and gis coordinates that appear on a map presented on the ganch project website (https://ganch.auctr.edu/), allowing emergency responders to quickly search for nchs by region and county in the event of a disaster. in this article we discuss the design principles, methods, and challenges encountered in building and implementing this tool, including the impact the tool has had on statewide disaster response after implementation. by cliff landis, christine wiseman, allyson f. smith, matthew stephens cultural heritage emergency response in georgia as the largest u.s. state east of the mississippi river, with 159 counties, georgia has extensive and diverse natural, cultural and historic (nch) sites that preserve and document the unique history and culture of the state. these nch sites are critical to cultural tourism, and include libraries, museums, archives, historical societies, historic sites, state parks, national parks and performing arts organizations. in recent years a spate of natural disasters — including hurricanes matthew, irma, michael, sally and laura, as well as regional flooding, and a very active tornado season in southwest and northwest georgia — has resulted in damage to historic structures, important documentary and artifact collections and valued cultural resources throughout the state. the 2020 hurricane season was the most active atlantic season ever with a record 30 named storms, 13 hurricanes and 6 major hurricanes [1]. furthermore, responses to these events were hampered due to the covid-19 pandemic. these natural disasters have also significantly impacted, and sometimes closed, community organizations important to citizens coping with disasters, such as public libraries and local archives. a rapid response is critical to successfully salvaging nch organizations and collections that are important to our citizens and economy. georgia, however, did not have a means for quickly identifying nch sites in areas impacted by disasters, thus impeding response. having an up-to-date and comprehensive inventory of existing nch sites is essential to preventing loss of important nch resources and limiting the financial impact of a disaster. only with accurate geographic locations of these resources can emergency responders and cultural heritage managers work together to expedite the recovery process. today, georgia has a robust community of organizations that cooperate and collaborate to mitigate damage to cultural heritage collections in the event of a disaster, but this was not always the case. the destruction of cultural heritage collections and organizations inflicted by hurricanes katrina and rita in 2005 served as a wakeup call in the cultural heritage community across the southeast and the nation. in the same year, a groundbreaking nationwide survey, heritage health index, was published by heritage preservation and the institute for museum and library services. the heritage health index was the first survey to assess the condition and needs of u.s. collections held in the public trust. responses related to disaster planning found that an alarming “80% of collecting institutions did not have an emergency plan or staff trained to carry it out.” [2] although cultural organizations within georgia were not severely impacted by hurricane katrina, the destructive nature of the storm coupled with the heritage health index galvanized the community to begin collaborative efforts to increase awareness about the importance of disaster planning in the cultural heritage ecosystem. modeling the federal planning structure, georgia’s emergency planning is organized into 14 emergency support functions (esf). nch organizations are included in the state of georgia emergency operations plan esf-11 annex [3] which is coordinated by the department of agriculture and the department of natural resources. under these primary agencies, support agencies and organizations are included in the plan, including the georgia archives, georgia historical society, heritage emergency response alliance (hera), and savannah heritage emergency response (sher). these nch organizations were added during the 2015 update which was the result of years of building relationships with state agencies and first responders.  since then, the nch community continues to cultivate relationships within the primary and support agencies of esf-11 to ensure that cultural collections are considered in disaster planning and recovery, in addition to the goals of saving lives, protecting property and restoring essential services. pilot and project design linked open data is “a group of technologies and standards that enables 1) writing factual statements in a machine-readable format, 2) linking those factual statements to one another, and 3) publishing those statements on the web with an open license for anyone to access.”[4] wikidata (https://www.wikidata.org/) is the linked open data knowledge graph (database of facts) produced by the wikimedia foundation, alongside sister projects like wikipedia. in 2018, the atlanta university center robert w. woodruff library conducted a pilot project to test the feasibility of updating an existing but out-of-date nch data set. the georgia historical records advisory council’s historical and cultural organizations directory [5] was web-scraped and placed into a table. the data was enhanced using free web tools like geocod.io (https://www.geocod.io/) and maplarge’s geocoder (https://geocoder.maplarge.com/geocoder) to include geospatial coordinates and county information. a subset of ten records was reconciled against wikidata using openrefine, a free data cleaning tool. this subset was then uploaded with openrefine to wikidata [6], enabling us to query the data and display the results using the wikidata query service. two example displays were created and published online [7]. these examples showed 1) an auto-generated list of contact information for nchs in an area (city/county/state), and 2) an auto-generated map of all nchs in a geographic area. figure 1. wikidata in 2018: galleries, libraries, archives, and museums (glam) in georgia with coordinate information: 40 institutions. building upon the growing strength of georgia’s nch emergency response community network, we used this pilot example to present the idea of an openly editable directory to state-level nch organizations. the idea was well received, with most organizations expressing a willingness to serve as partner organizations to provide source datasets and meet bimonthly to provide feedback [8]. some organizations committed additional resources for hosting the website and performing annual updates. these additional resource commitments ensured that the project would be sustainable and well-maintained over time, rather than a one-off project. after gathering support for the idea, we designed the project to be as open as possible. our design principles included: flexible structure – wikidata is able to represent many relationships in repeatable fields; records are easily enhanced with additional data decentralized updates – simple edits can be done manually by anyone; no need for an institutional or individual “owner” to grant access to the data for regular updates, thus removing bottlenecks open license – facts represented in wikidata are public domain sourced information – reference links would be included for each statement, for verification and to provide direction for future updates broad access and free implementation – the project will be completed using free software model for other states – by documenting and presenting on the processes used, the project would provide other states with a model that can be easily reproduced, adapted, and improved graduate student paid internship – the project would provide a graduate student in library and information science, archival studies or a related field with practical experience working with linked open data. in june 2019, the atlanta university center robert w. woodruff library received a lyrasis catalyst fund grant to support the project.   data workflow after the successful pilot workflow test, we scaled-up the workflow into a seven-step process: receive dataset – use python scripts to scrape websites, download available datasets, or request datasets from project partners format to spreadsheet template – copy relevant data over to the dataset template in csv (comma separated value) format index and remove duplicate entries – search each organization name in wikidata to see if a record exists and to see if we’ve already created/edited the record in a previous upload. if so, delete the duplicate entry to prevent performing duplicate enrichment work gather, verify, enrich, and create references for data – perform web, phone, and email research to verify the existence and directory information for the organization. enrich with county and geocoordinate location information. use the internet archive wayback machine to create snapshots for references reconcile in openrefine – create a data dictionary to identify and prioritize which data points are most valuable for disaster recovery, and to match them to wikidata’s data model [9] upload dataset to wikidata – using openrefine, upload the revised and referenced records in the dataset to wikidata perform quality control on ingests – review each record in wikidata and make any post-upload edits necessary. examples include verifying geocoordinate locations, removing duplicate field entries, selecting a “preferred rank” value for multi-value fields, and adding parent/subsidiary organization relationships. to help other organizations who may want to adapt or reuse this workflow, we’ve developed a workflow manual that describes this step-by-step process in greater depth [10].   figure 2. mapping nch organizations’ directory information to wikidata’s schema.   website application while datasets were being processed through the data workflow, we developed a mobile-friendly website application that uses prepared linked data queries to display the results of all our data.  at its heart the ganch application is a web proxy, fetching and re-presenting information from wikidata, caching it locally. wikidata provides the wikidata query service, which lets users construct requests in the sparql query language. our site facilitates the housing of queries for nchs at various levels in georgia and presents the resulting datasets in both tabular and cartographic formats. figure 3. desktop and mobile views of ganch search results, showing both cartographic and tabular displays. sustaining a website like this requires user accounts and related authentication, but at a very simple level. authenticated accounts are restricted to project staff, who can create, read, update, and delete the sparql queries and view the responses in their raw form.  this also provides us with an audit trail for query edits.  however, unauthenticated public users are able to view a wikidata result set for a given georgia county or gema region, or the state as a whole, as both a table and an interactive map. public users can also follow instructions on the site to export the tabular data directly from wikidata. secondary requirements included some instructional pages for public visitors to the site, responsive design to facilitate use on mobile devices, and some scripted tasks to update the 159 georgia county queries in a single batch, to relieve project members of tedious data entry.   figure 4. the “dekalb county” query in the wikidata query service, available at https://w.wiki/tzi . figure 5. the “gema region 7” query in the wikidata query service, showing the combination of several counties into a single result set, available at https://w.wiki/tzk. figure 6. editing the “dekalb county” query on the back-end of the ganch website, showing both the full sparql query and the wikidata response.   the site interfaces with wikidata’s sparql api and stores the query response as text/json, parsing and transforming the responses as needed for presentation to the user. one obvious need was to combine the organization identifier and its human-readable label into a single column (for creating a hyperlink.) thus, json from wikidata in the shape of { "organization": { "type":"uri", "value":"http://www.wikidata.org/entity/q69766025" }, "organizationlabel": { "type":"literal", "xml:lang":"en", "value":"alma-bacon county public library" } } is transformed into an object like this for use in a responsive bootstrap table: { "type"=>"uri", "value"=>"http://www.wikidata.org/entity/q69766025", "as_link"=>"<a data-value=\"alma-bacon county public library\" target=\"_new\" href=\"http://www.wikidata.org/entity/q69766025\">alma-bacon county public library</a>", "search_data"=>"alma-bacon county public library" } in the database, both query and response are stored as text, and when read, the response json is parsed into an object that can be traversed. the wikidata response includes the response parameters as a header row, which will be familiar to most spreadsheet users. the header serves as the source of the table structure, though as mentioned we did find a need to group related columns such as ‘organization’ and ‘organizationlabel’ into one to present a readable table. keeping presentational logic out of the data storage layer of the application was a standard separation of concerns, the idea being that what is stored as “response” actually matches the response from the data source. a further goal was to allow admin users to edit their queries (checking their work against the wikidata query service, which offers an online query editor), and this arrangement allows for easy troubleshooting of purely data errors, as opposed to errors in the applications rendering of the page. we did similar on-the-fly cleanup of some fields to adjust wikidata-formatted values to work in the map, as latitude and longitude needed to be split into their own parameters. in addition, if wikidata records included an email for a given institution, the site must be capable of sending occasional emails to that organization, to encourage them to keep their public records up-to-date. to that end, the response json was scanned for such emails, which are compiled into a separate list in the database, along with their related institutional identifiers. some other requirements were shaped by our partnership with galileo, georgia’s virtual library. this consortium generously offered to host our project. their expertise with ruby on rails made that framework a logical choice in which to write our website code. in addition, galileo makes use of the gitlab devops suite of tools, and this was an integral part of our development and release process. rails made sense to us for a number of reasons, most notably because ruby offers a reliable sparql library, which meant that queries could be easily managed within rails. initial creation and testing of those queries, we found, was best done on wikidata’s own query editor. gitlab also offered many desirable features, such as continuous integration testing, automated deployment, and issue tracking. with a limited budget for developer time, it was essential to make good use of open source tools and community knowledge. we chose not to use wikidata’s own maps as an iframe as the wikimedia foundation maps terms of use suggests reliance on them might be counterproductive (“please respect our limited services and resources. we reserve the right to discontinue.… if you use the service too heavily, it could affect the service’s stability for others or degrade our quality of service.”).[11]  we instead chose openstreetmap as the source of our geographic data, and together with the leaflet javascript library for interactive maps, it offered us the ability to plot nchs on a map with minimal effort, while also ensuring that the maps would work if wikidata was unavailable. likewise, the use of bootstrap and the bootstrap-table ui frameworks made the creation of sortable, searchable tables straightforward. it is our expectation that these will also be easy to maintain in the long term, due to their widespread use. the site includes a job scheduler (we used rufus-scheduler, a popular ruby library), which runs a nightly update, querying wikidata. if no response is received, the site does not update its local data, which means that, for these items at least, our site provides a backup in case wikidata is itself disrupted or unavailable. points of unexpected effort included: rails 6 setup, as this new version of rails does significantly change project setup and structure (for the better.) we also found a need to create numerous functions to transform/clean up various data items from within the large and heterogeneous json objects returned by wikidata. these are available on the ganch website’s github repository [12]. several aspects of system administration/environment setup (email handling, security/server configurations) required significant time. while deployment went smoothly, there was the usual time and effort getting team members up to speed on the desired processes, as well as remotely troubleshooting issues as they arose. in contrast, routine preparation in beginning a rails 6 project for the first time led to many useful time savers. much of the navigation ui and authentication work was expedited by implementing ideas found in chris slade’s excellent youtube tutorial on rails 6 development [13]. this allowed us to overcome a growing challenge in rails development, the reliance on outdated information and design patterns. we anticipate that this will improve the site’s sustainability. gitlab’s stringent deployment criteria required regular upgrading of components as vulnerabilities/exploits were discovered by the community. this means that the site has seen regular maintenance and is staying up-to-date with minimal effort. gitlab’s issue tracking made work that was, by necessity, 100% remote much smoother than it might have been. to ensure that the final website would meet cultural heritage emergency responders’ needs, we facilitated a focus group with emergency response coordinators at gema, hera, and sher in april 2020. the feedback from this focus group session was instrumental, as it resulted in the inclusion of the eight gema regions, which are used to coordinate emergency response at the state level. we also adjusted the website to include instructions on how to export search results via the wikidata query service; this enables cultural heritage emergency responders to integrate the results into the state-level webeoc system. addressing challenges overall the project worked well. the most frequent concern that people have raised about using wikidata for this project is the risk of vandalism, but over the last two years we’ve only seen a handful of minor instances of unhelpful edits, which were easy to revert. this is possible because records that have been created or edited by a project staff member are automatically added to that member’s wikidata watchlist, allowing them to monitor ongoing changes.  weekly review of the watchlist allows project staff members to quickly identify any unhelpful edits and revert them. it was a pleasant surprise to see that the most commonly raised concern did not present a challenge for the project. however, we did experience a few unexpected challenges. one problem we encountered was outdated information persisting in the initial datasets we received; in some cases nch organizations had been dissolved for over a decade, yet still persisted in government and professional organization directories. the process to verify dissolved dates included reaching out directly to the last available contact, reviewing newspaper articles, and utilizing the trip advisor and the georgia corporations division business search websites. whenever possible, we included these dissolved organizations in our uploads, and included the dissolution date with a reference to show evidence that the organization no longer exists. this not only provides public documentation of the organization’s closure, but it also allows us to exclude these closed organizations from our search results to prevent cultural heritage emergency responders from wasting time during a disaster. another issue we discovered was the challenge of trying to map real-world messiness to the much cleaner data model of wikidata. the problem was not only trying to map it, but trying to reach consensus with the wikidata community on how it should be handled. in wikidata, the p131 property is the metadata field for administrative territorial entities in wikidata [14]. the scope of this p131 property is broad and can cover municipalities, counties, and states. in the wikidata documentation for p131, it recommends that you only list the single most local administrative territorial entity, since the field is supposed to be both transitive and hierarchical, and cascade upwards as seen in figure 7 on the left. figure 7. diagram showing the conflict of the transitive property of p131 between wikidata and georgia. but in georgia, the borders of municipalities and counties were drawn independently, and about 10% of the municipalities in georgia exist in more than one county; as such, they are not transitive, as seen in figure 7 on the right. this means that our project cannot rely on wikidata’s cascading hierarchy to be correct when it comes to searching for organizations by county. we reached out to the wikidata community to try and find a solution to this challenge. after several proposals and over a month of discussion on four separate wikidata pages, no consensus was reached. as a result, we decided to explicitly declare municipality, county, and state, all in the p131 field, with the hope that consensus on a solution can be reached in the future. in the meantime, our queries are functioning well. we reason it is better to have too much well-sourced information, rather than too little. we also encountered a workflow challenge with nch organizations’ name variations. we knew in advance that many of the organizations would appear in multiple source datasets, therefore we created an index file to assign accession numbers for each organization. as we prepared new datasets, we compared the organization names against the index to prevent duplication of our work. we also knew that organizations’ names would vary between datasets, so we planned to check each dataset against the index using the microsoft excel fuzzy lookup add-on to identify similar/identical organization names. unfortunately, this method was insufficient at identifying duplicate records; therefore we changed our workflow to check the names of organizations manually in wikidata before enriching the data. as we discovered new name variations, we added them to wikidata as aliases (alternative names). this improved our results by reducing duplication of work, however, a few duplicate records of organizations were present on each dataset; these duplicates had to be merged manually during quality control in wikidata. a bonus to this process is that we generated a clear list of how many organizational records were edited (via the index), and that wikidata now includes a comprehensive list of all the name variations we encountered for each organization. figure 8. recording labels (name variations) in wikidata. originally, we planned to use free, online geocoding tools to quickly obtain geocoordinate locations for the nch organizations. unfortunately, since these tools rely solely on the address of the organization, they can often yield incorrect results. after a disaster, wayfinding markers, or even whole buildings can be destroyed; it was important for us to identify the physical locations of these nch organizations, so the mailing address or nearest intersection was often insufficient. as a result, we shifted to manually locating each nch organization’s geolocation coordinates via google maps, which in some cases required a fair amount of research. for example, most of the technical colleges in georgia have multiple campuses, each with their own libraries. many of these libraries do not have their own webpages to reference, they often exist in shared facilities (e.g. academic building c), and they are not distinctly identified in google maps. by looking for online campus maps and cross-referencing with google maps, we were able to identify the physical location of each library. lastly, an unfortunate component of this project was the emotional toll it took on us. georgia has a diverse and challenging history, reflected in its organizations and their records: the confederacy, american slavery, jim crow, civil rights, indigenous people and sacred land, bipoc, and the lgbtqia+ community are all part of the state’s history. there are organizations in georgia who celebrate those that defended the practice of slavery, and there are also organizations who celebrate those that sought to abolish it. this project was completed at the same time as international protests were taking place against systemic and police violence committed against black americans. seeing this same battle for justice play out on american streets while we worked on these records was uncomfortable, to say the least. yet, no matter how we may personally feel about parts of georgia’s history, we endeavored to record it all fairly and equally. no record should be erased because we are uncomfortable. our discomfort should push us into discourse about what each record means to the state of georgia then and now. to help other organizations who may adapt or replicate this project, we have documented the challenges that we encountered and how we addressed them [15]. sustainability we recognized early on that this project would need a state-level institutional home, therefore we created a sustainability plan with our partners for inclusion into the project. galileo, georgia’s virtual library, graciously provided website hosting for the ganch website. the georgia public library service’s archival services and digital initiatives office and galileo both volunteered to provide ongoing staff support for dataset maintenance via annual updates. to help ensure long-term sustainability of the project, we built a semi-automated emailer on the back-end of the ganch website to send out reminder emails to nch organizations. since we gathered many organizations’ contact email addresses during the enrichment stage of the data workflow, update reminder emails can be sent annually to confirm accuracy of, or obtain updates to, organizational information. if inaccurate, the nch organizations can reply to provide updates. if the email bounces, staff can check to see if the organization still exists. if we were to repeat this project, we recommend asking for additional funding in two areas: to pay for promotional activities and to pay for an additional graduate student assistant. we did not originally allocate any funds for travel costs to spread the word about the project, therefore our opportunities in this area were limited. however, in the wake of covid-19, many conferences transitioned to a completely virtual format and waived their registration fees for presenters. this provided us with additional opportunities to present on ganch at the state and national level. although we were able to accomplish a great deal, we had to pull in additional staff members jessica leming and alex dade from the woodruff library’s digital services department to assist with record enhancements and quality control reviews to meet deadlines, for which we are forever grateful. however, with additional funding for a second graduate student assistant, we could have expanded the project to include additional nch organizations (such as county clerk of court offices). we also could have further enriched the records for the organizations that were included; with limited time, we had to cut back on recording social media accounts, inception dates, and parent organization/subsidiary organization relationships part-way through the project to ensure we were able to capture core data for more organizations. impact & future possibilities at the beginning of this project, there was no easy way for cultural heritage emergency responders to reach out to nch organizations before or after a disaster. the data available was scattered across the internet in disparate databases and tables, and was rarely up-to-date. what started in 2018 as 40 glam institutions mapped in wikidata has now grown to 1,916 nch organizations indexed as of august 2020. table 1. ganch project impact overview. source datasets 13 properties 19 total nch organizations (de-duplicated) 1,916 total wikidata edits (all project staff) 15,501 we retrieved, verified, updated, enriched, and uploaded thirteen datasets gathered from our partner organizations, each with as few as 15 to as many as 556 records. additionally, we have broadened the scope of our search queries to meet the needs of cultural heritage emergency responders. for example, our search queries now include historic districts which, although they don’t have contact information, are important for gema fly-overs to assess damage of historic properties not associated with cultural heritage organizations. as a result, our largest search query now includes 1,960 results. figure 9. wikidata in 2020: nch organizations in georgia with coordinate information: 1,960 institutions. however, the first real test of the web application’s effectiveness came in september 2020 as hurricane sally approached the southeastern u.s. in anticipation of heavy rainfall and flooding, members of hera sent out emails to 330 nch organizations in gema regions 1, 3, 4 and 7.  thankfully, none of the emailed organizations requested emergency assistance following the storms, but hera did receive gratitude from organizations for reaching out. from the email blast, we experienced a 9% bounce rate where emails could not be delivered. not all organizations publicly provide email addresses, so we knew that we were not reaching everyone in these regions, but broadcast emails are a good way to get the word out quickly that emergency assistance is available. for more acute disasters such as a tornado, emergency responders can reach out to organizations on a county-by-county basis using the table of all available contact methods, including email, phone, facebook, twitter, and website forms. we can now follow up with the organizations whose emails bounced to see if the organizations are still active, and if so, which email address they can provide (if any), for emergency contacts. a pleasant surprise in working on the project was seeing how wikidata was being used by others to store and discover additional information about the nch organizations. during the project, a wikidata user uploaded visitor counts for all georgia public library system libraries from 2015-2017. since wikidata is an open system, additional information can be added to these records, such as budget changes, collection statistics, branch openings and closings, and other facts, all stored as linked open data. likewise, the data that we entered for disaster preparedness and recovery is now freely available to be repurposed for tourism, planning infrastructure, performing historical analysis, or other topics we have yet to imagine. as such, we hope our website will not only serve as a one-stop location for georgia’s cultural heritage emergency responders to identify nch organizations at risk but will also serve as a tangible introduction to linked open data for cultural heritage workers of all kinds. in order to make this project a model for other states, we have made all project documentation and data freely available via github at https://github.com/auc-woodruff-library/ganch-data and https://github.com/auc-woodruff-library/ganch-website. the ganch website is available at: https://ganch.auctr.edu/.   notes [1] national oceanic and atmospheric administration. november 24, 2020.  record-breaking atlantic hurricane season draws to an end. [internet]. available from: https://web.archive.org/web/20201201212728/https://www.noaa.gov/media-release/record-breaking-atlantic-hurricane-season-draws-to-end [2] heritage preservation.  2005. a public trust at risk: the heritage health index report on the state of america’s collections. [internet]. washington (dc): heritage preservation. available from: https://web.archive.org/web/20200820135617/https://www.imls.gov/sites/default/files/publications/documents/hhifull_0.pdf [3] planning [internet] [accessed 12/2/2020] atlanta (ga): georgia emergency management and homeland security agency.  available from: https://web.archive.org/web/20201023073629/https://gema.georgia.gov/what-we-do/planning [4] landis, cliff. 2019.  “linked open data in libraries” in k. j. varnum (ed.), new top technologies every librarian needs to know (pp. 3-15). chicago: ala neal-schuman. http://hdl.handle.net/20.500.12322/auc.rwwlpub:0029 [5] historical and cultural organizations directory. georgia historical records and advisory council.  https://web.archive.org/web/20200804170808/https://georgiaarchives.org/ghrac/directory [6] edit group by clifflandis: united states (96014bf). editgroups. https://web.archive.org/web/20200312135440/https://tools.wmflabs.org/editgroups/b/or/96014bf/ [7] wikidata iframe test. clifflandis.net. https://web.archive.org/web/20200312135539/http://clifflandis.net/wikidata_ga-cho-dr_test.html [8] ganch project partners list. github. https://web.archive.org/web/20201208160419/https://github.com/auc-woodruff-library/ganch-data/blob/master/docs/project_partners.md [9] ganch data dictionary. github. https://web.archive.org/web/20201208160608/https://github.com/auc-woodruff-library/ganch-data/blob/master/data/data_dictionary.md [10] ganch workflow manual. github. https://web.archive.org/web/20201208160839/https://github.com/auc-woodruff-library/ganch-data/blob/master/docs/workflow.md [11] wikimedia foundation – maps terms of use. https://foundation.wikimedia.org/wiki/maps_terms_of_use [12] ganch website. github. https://web.archive.org/web/20201208181502/https://github.com/auc-woodruff-library/ganch-website [13] beginner rails 6 tutorial series. youtube. https://www.youtube.com/playlist?list=plmz7pzjxns1vla-k5qpji6aydyys0mbia [14] located in the administrative territorial entity (p131). wikidata. https://web.archive.org/web/20201206193449/https://www.wikidata.org/wiki/property:p131 [15] ganch addressing challenges. https://web.archive.org/web/20201208161050/https://github.com/auc-woodruff-library/ganch-data/blob/master/docs/challenges.md about the author cliff landis is digital initiatives librarian at the atlanta university center robert w. woodruff library. his research interests include linked open data, archival technologies, digitization, metadata, and the coevolution of humanity and technology. christine wiseman is assistant director of digital services at the atlanta university center robert w. woodruff library. her research interests include cultural heritage emergency preparedness and archival and digital preservation. allyson f. smith is graduate assistant at the atlanta university center robert w. woodruff library. matthew stephens is a web developer who has collaborated with the atlanta university center robert w. woodruff library since 2015. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – a novel method for creating a distributed, collaborative commenting environment for bibliographic items mission editorial committee process and structure code4lib issue 14, 2011-07-25 a novel method for creating a distributed, collaborative commenting environment for bibliographic items this paper discusses a novel approach to adding user comments to existing platforms for bibliographic information, such as library catalogs. the application is built using simple and free services that support advanced functionality at a low price without requiring high-level technical skills. the strength of the approach described here is that it increases the number of comments available for display in any local catalog by consolidating comments from multiple sites and by clustering comments at the frbr work level. to do this, a central store of comments from multiple sites is created. in addition, the application uses isbns and oclc’s work ids to consolidate comments from different publications (frbr manifestations) for the same work. by rurik thomas greenall introduction this paper introduces an ultralightweight “add-on” system for commenting on things in library catalogues, although in reality, the system can be used to comment on any web page containing a minimum of bibliographic data. additionally, it improves the commenting experience for users by presenting comments on related bibliographic items together and increasing the volume of comments by providing access to the same comment system from multiple sites. commenting commenting is the provision of a space for users to give their opinions, information and feedback on/about the item described on a page. in many contexts, comments are personal opinions of the work in question and can vary from longer book-review type comments to shorter “awesome!”-type comments. in the case of non-fiction literature, the reviews often provide meta-information about the up-to-date-ness of an item as well as alternative resources. there are a number of reasons to provide this kind of service: adds value to pages about content enriches user-participatory experience provides clues to the reception of collection building strategies provides another point of contact between an institution and its users of course, it must be assumed that the likelihood of users commenting is relatively low; it has been pointed out that users do not necessarily expect commenting facilities in online library resources, and additionally, commenters are generally assumed to be a small subgroup of the whole user base [1]. nevertheless, provision of this feature can prove successful, as attested by the numerous sites that provide this kind of service. because the volume of comments is expected to be relatively low, it is necessary to improve the way in which user comments are distributed online — if a comment on an item exists, then it should also be viewable in other contexts where it is relevant. the best example of this is two editions of the same book existing in the same catalogue: comments on one edition should appear in the comment space of the other edition. the system developed here uses the frbr entities work and manifestation, where a work is the top-level concept of the creative work, and a manifestation is a published version of this work [2]. this factor and the further distribution of comments to/from other sites increases the “comment ecology”, or volume of comments and users who comment. existing approaches the existing approaches to commenting in library catalogues fall into one of two categories: local and distributed, which in turn can be modular or built-in. in the case of “local”, a comment system is implemented for a single site; this may be as part of a configurable part of the system (as is the case for koha [3]) or as an external module. in either case, if the catalogue is frbr-aware, it can support work-level commenting, but the extent of the comment ecology will be small and these systems typically only work well for libraries with very large user bases. bibliocommons provides a distributed, built-in approach whereby the opac is provided as software as a service. in fact, each library has a dns subdomain of bibliocommons.com (for example http://wilmette.bibliocommons.com and http://christchurch.bibliocommons.com), and there is a shared, work-related comment ecology that functions for all media types. because this functionality is bound to the bibliocommons platform, it cannot be implemented in other existing systems; the bibliocommons sites do however show that commenting systems engage users and the feedback provided is often useful. the distributed, modular approach is adopted by the swedish project öppna bibliotek [4] (translation: open libraries), which provides an api for submission and retrieval of comments. this approach creates a large-scale comment ecology, but it does not support work-level commenting. additionally, it has a technical overhead that is potentially insurmountable for many libraries as it is an api, not a complete, easily installable system. it is apparent that the bibliocommons approach is both elegant and creates a good level of comment frequency on items in the catalogue. it could be said that this paper attempts to provide bibliocommons-style commenting without using that system. note, however, that using an integrated system obviously provides benefits regarding knowledge of and access to the structure of the content (allowing more precise facet appraisal to identify works). system design the system needs to generate a work id and link to a comment space associated with this id. in this scenario, two external systems are assumed to exist: one that provides the work id and one that provides the comment space for the work id that is passed to it. to find a work id, the system must pass a variable that allows the work-id service to identify the work. thus three further system components are necessary: extraction of criteria for work identification designation of work ids linking to comment space figure 1 shows how these components interact with each other. figure 1. system components functional requirements the functional requirements for the system are: the system must automatically find criteria on the local page which allow a work to be identified the system must provide a comment space which is explicitly connected to the work that has been identified so that users can view or add comments the system must, upon encountering errors, provide an intelligent failure method additional requirements for the software: the system must require a minimum of programming for installation the system must use cross-platform html and ajax on the client side the system must cost as little as possible, and preferably be free further, functional requirements for the work-id service: upon receipt of a manifestation identifier the system must return a unique id that can be used to identify works, or, in the case of no work identifier being found, fail gracefully the system must cost as little as possible, and preferably be free and the comment service: the system must provide a comment space that can be distributed across multiple web domains and sites based on shared ids the system must be simple to install and not require modification of the code to work for each site the system must provide robust and reliable methods for authentication, including single sign-on for libraries that wish to implement this the system must allow export of data to ensure the future stability of the services built upon it the system must cost as little as possible, and preferably be free implementation initially, the system needs to be broken into its components: the comment system, the work-id system and the core system that is used to link these together. these components will be treated in turn. comment system the comment system needs to be stable and distributable; there are many examples of commercial services that can be used for this purpose. in the current case, disqus [5] was chosen for a number of reasons: widely-used, familiar system free good level of support, even for free service well-documented api [6] supports multi-domain and multi-site, distributed commenting options for login, including single sign-on (for paid plan) small code footprint the standard code to include disqus commenting was adapted to receive a work id, but otherwise remains largely unchanged. this code can be found in https://github.com/brinxmat/global-book-commenting/blob/master/globalcomments.js. work-id system work ids are typically considered difficult to generate; this is because it is not easy to identify which expressions, manifestations or items are to be linked to a work. typically, various facets of the bibliographic record are used, including standard titles, responsible authorities and so forth. because the current implementation assumes nothing about the systems into which it is to be integrated, it cannot be assumed that there is access to any of these facets. rather, the lowest level identifier that is available is taken: isbns. at this point, it should be noted that isbns are not unique, they are not available for every item, nor are they consistent in identifying the same kind of thing; however they are largely unique and largely represent manifestations. in the absence of true manifestation ids, isbns are the best and most realistic option. it is possible to combine an isbn search with the other facets mentioned above. however, for the sake of simplicity, this approach has not been used here; indeed, it is not clear that this approach would make enough difference for it to be worthwhile. the current approach simply identifies one of six possible isbn types, whereas identifying titles and author names on pages is far from simple as these are largely represented as text strings that are not semantically marked in a way that makes it possible to identify them programmatically. isbns largely identify books, and this means that much content on the target site will not be identified as a work, but rather as a manifestation — consequently, the system fail-safes to providing a unique id where none is provided. this issue can easily be addressed by querying other services that provide identifiers for other item types (for example, ismns and eans). for the current implementation, a service was built around oclc’s xisbn [7] and xoclcnum [8] services. these were chosen because they are relatively stable, have straightforward restful apis and encompass a large volume of content (albeit scantier for works in languages not widely covered in worldcat). the librarything api [9] was also considered, but it provided no transparent method of assigning a unique id for works as it just returns a list of isbns. these services have restricted free use (one thousand requests per day) and do not permit the caching of data retrieved from them [10] . the service was designed as described in figure 2. figure 2. work ids the system was implemented in php, the code: https://github.com/brinxmat/global-book-commenting/blob/master/oclc.php. core system the core system has three major processes: finding isbns passing found isbns to the work-id service initializing the comment service with an id the initial step is to identify the isbns on the page where the code is embedded. this was done using four regular expressions to identify isbn10/isbn13 (with and without separators) on the page. the flow is described in figure 3: figure 3. core system flow the software was implemented in javascript, using the jquery [11] framework and a jquery plugin for cross-domain ajax [12] ; the latter is necessary because the work id service is queried using this, although it is not necessary to do this in this way — alternatives include jsonp [13] and cors [14] . using javascript in this way means that after the jquery and the plugin are loaded, those wishing to use the comment system need only include a single <div> tag and a <script> tag. the javascript code: https://github.com/brinxmat/global-book-commenting/blob/master/globalcomments.js. a live example can be found at http://folk.ntnu.no/greenall/comments. installation the system is simple to install. load jquery and the plugin, then put <div id="disqus_thread"></div> in the body of the page and <script type="text/javascript" src="path/to/globalcomments.js"></script> just before closing the closing </body> tag (as it is important to load the script after the page has loaded). the road ahead the system has not been widely tested, so it is difficult to know whether it would take the strain of real use. more testing across more sites is the first task that needs to be performed, and the author welcomes interested parties to get in touch about this. improvements can be made to every part of the system, including removing reliance on jquery and the cross-site ajax plugin (these potentially conflict with systems already using jquery) and improving the isbn matching regular expressions. it is worth looking for other systems that can return work ids for other content types so that this feature can be incorporated. additionally, the benefits of improved work identification balanced against the cost of more complex installation should be investigated; this will solve many of the issues that arise from using isbns as the sole identifiers of manifestations. while the system is a very early phase in its development, it has many of the “must-have” features that are seen in higher-end systems, without the costs. it also shows that lightweight solutions that are implementable in very many contexts can be used to solve relatively complex problems in an elegant way. source code disqus integration: https://github.com/brinxmat/global-book-commenting/blob/master/globalcomments.js. oclc integration: https://github.com/brinxmat/global-book-commenting/blob/master/oclc.php global commenting: https://github.com/brinxmat/global-book-commenting/blob/master/globalcomments.js references 1. tam w, cox a m, bussey a. student user preferences for features of next-generation opacs: a case study of university of sheffield international students. program: electronic library and information systems. 2009;43(4):349–374. 2. ifla study group on the functional requirements for bibliographic records. functional requirements for bibliographic records: final report [internet]. münchen: k.g. saur; 1998 [cited 2011 may 23]. available from: http://www.ifla.org/files/cataloguing/frbr/frbr_2008.pdf 3. koha – open source ils – integrated library system [internet]. [cited 2011 may 23]; available from: http://www.koha.org/ 4. biblioteket.se. öppna bibliotek [internet]. öppna bibliotek. [cited 2011 may 22]; available from: http://biblioteket.se/default.asp?id=14392 5. discover your community – disqus [internet]. [cited 2011 may 22]; available from: http://disqus.com/ 6 disqus docs | developers [internet]. [cited 2011 may 22];available from: http://docs.disqus.com/developers/ 7. worldcat web service: xisbn [oclc – worldcat affiliate tools]: xoclcnum api [internet]. [cited 2011 may 22]; available from: http://xisbn.worldcat.org/xisbnadmin/xoclcnum/api.htm 8. xisbn [worldcat affiliate program] [internet]. [cited 2011 may 22]; available from: http://www.worldcat.org/affiliate/webservices/xisbn/app.jsp 9. apis and easy linking | librarything [internet]. [cited 2011 jun 26]; available from: http://www.librarything.com/api.php 10. worldcat: oclc affiliate services terms and conditions [internet]. [cited 2011 may 22]; available from: http://www.worldcat.org/ldap/terms.jsp 11. jquery: the write less, do more, javascript library [internet]. [cited 2011 may 22]; available from: http://jquery.com/ 12. cross-domain-ajax at master from jamespadolsey/jquery-plugins – github [internet]. [cited 2011 may 22]; available from: https://github.com/jamespadolsey/jquery-plugins/tree/master/cross-domain-ajax/ 13. json-p: safer cross-domain ajax with json-p/jsonp [internet]. [cited 2011 may 22]; available from: http://www.json-p.org/ 14. cross-origin resource sharing [internet]. [cited 2011 may 22]; available from: http://www.w3.org/tr/cors/ about the author rurik thomas greenall is a research librarian at the section for information resources of the library of the norwegian university of science and technology. he is a programmer and data modeller, working mostly with rdf and linked data. acknowledgements the idea for this project originally arose in discussions at the norwegian library laboratory (www.biblab.no), where a similar system was outlined for public libraries in norway. the author would like to take this opportunity to thank david massey of oslo university college for help testing the initial implementations of the software, as well as the editors of the code4lib journal for their help in shaping this text. all errors, of course, remain the sole responsibility of the author subscribe to comments: for this article | for all articles 2 responses to "a novel method for creating a distributed, collaborative commenting environment for bibliographic items" please leave a response below: code4lib journal – n° 14 « pintiniblog, 2011-07-27 […] a novel method for creating a distributed, collaborative commenting environment for bibliographic it… […] swib13: semantic web in bibliotheken 2013 | brinxmat's blog, 2013-11-28 […] as the system uses isbns, the traditional issues are noted (isbn is not an id, linking editions, etc.) might be worth looking at the methods in http://journal.code4lib.org/articles/5339 […] leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – editorial introduction — openness mission editorial committee process and structure code4lib issue 8, 2009-11-23 editorial introduction — openness on openness and the code4lib journal. by andrew darby coordinating editor, issue 8 openness is one of the defining concepts at the intersection of librarianship and programming, a neat ontological overlap that cries out for a venn diagram. libraries strive for openness in terms of physical space, operating hours, and access to information; many programmers seek to freely share their ideas, expertise, and code. openness is an idea and an attitude that you can feel good about, and i think this good feeling is one reason why the code4lib community in its various incarnations (irc, listserv, conference, journal) continues to grow. openness means several things for the code4lib journal. first, we are an open access publication, listed in the directory of open access journals, and we require that our authors publish their articles under a creative commons attribution (cc-by) license. the cc-by license “lets others distribute, remix, tweak, and build upon your work, even commercially, as long as they credit you for the original creation” [1], which sounds like a scary proposition if you imagine all the terrible frankenstein’s monsters people might generate from your work, but it also allows for the greatest utility, exposure, and value [2]. open access, in turn, means that there are fewer barriers to people reading the journal. it exists natively in the “light” web, happily mingling with all the other content—and being indexed by the same spiders and bots. you don’t have to be a “researcher,” with access to a university’s subscription resources, to find our articles. this allows for engagement with the world outside of our very specific domain—mighty google acting as matchmaker—and hopefully a fruitful cross-pollination of ideas. the journal also has a toehold in the “dark” web—we are now indexed by ebsco, so we are not invisible to the traditional academic researcher. from a corporate point of view, openness can seem like a lack of center. when an ebsco representative wanted us to sign their “standard non-exclusive licensing agreement”—well, who should do that? the same person who will cash our (entirely hypothetical, i fear) royalty checks? and then, what’s the postal address of an entity made up of emails, spreadsheets, and web pages? finally, considering our license, couldn’t ebsco have just taken the contents of the journal themselves, without asking? (thanks for asking, though.) there are numerous other ways the journal is “open”: our publication platform (wordpress) is free and open source software; we encourage an immediate and open dialogue with readers through comments; our “closed” internal deliberations are nonetheless “open” (any editor can deliver their two cents on any article); and importantly, many of our articles include code with an open source license. openness can also mean transparency. at the journal, only our editorial discussions are closed. on the code4lib wiki, we post documents that expose our procedures for selecting articles, our internal deadlines, even our basic “thank you for your submission” email templates. it’s a practical matter: we need to store this information somewhere, and we could put it under virtual lock and key; but why? there is also a google group, c4lj-discuss, to hash out non-editorial issues with anyone who cares to join; and even this editorial is a stab at transparency. one downside of openness is noise. we have all searched the internet for a solution to a problem, found publicly-posted code, and come to the frustrating realization (sooner or later) that it was wrong—or maybe just wrong for our purposes. but at least something was there, and this wrong answer hopefully moved us closer to a solution by closing that particular door. imagine instead that your search turned up only an abstract with the promise of an answer, perhaps a solicitation to pay just 99 cents for the full text (which might still be wrong)? we’d all be so much further behind if that were the winning paradigm. better to accept some noise. so, hurrah for openness, for open access journals, for people who index content, and for authors (of all sorts) who are willing to make their work freely available. notes [1] http://creativecommons.org/about/licenses/. [2] cc-by is a stricter standard than required by the doaj, but it is required for the sparc europe seal, and we agree with the contention that “[i]n order for open access journals to be even more useful and thus receive more exposure and provide more value to the research community it is very important that open access journals offer standardized, easily retrievable information about what kinds of reuse are allowed” (http://www.doaj.org/doaj?func=loadtempl&templ=080423). subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – “with one heart”: agile approaches for developing concordia and crowdsourcing at the library of congress mission editorial committee process and structure code4lib issue 46, 2019-11-05 “with one heart”: agile approaches for developing concordia and crowdsourcing at the library of congress in october 2018, the library of congress launched its crowdsourcing program by the people. the program is built on concordia, a transcription and tagging tool developed to power crowdsourced transcription projects. concordia is open source software designed and developed iteratively at the library of congress using agile methodology and user-centered design. applying agile principles allowed us to create a viable product while simultaneously pushing at the boundaries of capability, capacity, and customer satisfaction. in this article, we share more about the process of designing and developing concordia, including our goals, constraints, successes, and next steps. by meghan ferriter, kate zwaard, elaine kamlley, rosie storey, chris adams, lauren algee, victoria van hyning, jamie bresner, abigail potter, eileen jakeway, and david brunton concordia and by the people are enactments of the user-centered focus of the library of congress strategy and digital strategy. in “enriching the user experience,” the library of congress 2019-2023 strategic plan articulates four essential goals: expand access, enhance services, optimize resources, and measure results. released in october 2018, the library of congress’ first digital strategy complements the library of congress 2019-2023 plan by emphasizing approaches to meet those goals digitally. specifically, the digital strategy promises that we will throw open our treasure chest, we will connect, and that we will invest in our future. we were fortunate to build concordia and by the people at a time when the library of congress strategic plan and digital strategy were being articulated. these two documents signal directions that you can see in both by the people’s community management and concordia’s design and development. underscoring the objectives of the digital strategy by providing interactive engagement with the library’s collections, by the people connects to audiences across levels of learning and geographic location. concordia’s development is ongoing and sustained in such a way as to build a road toward future, as-yet-undefined capabilities. furthermore, by the people and concordia expand access to the library of congress through enhanced technical services and a robust program of engagement. the design principles for this project emphasize trust and approachability as a complement to the authoritative role the library of congress plays in american memory. both by the people and concordia offer opportunities to deeply engage our audiences with our collections and interactive interfaces. they are also aligned with a manageable guiding practice within the digital strategy office and library of congress labs team: “do something, gather evidence, and then refine the approach.” in the sections below, we briefly describe the by the people program, how we adopted an agile approach, and the process of developing concordia. we hope the discussion demonstrates the ways we designed opportunities for public audiences to engage with library resources. becoming by the people by the people aims to engage and inspire participants by inviting them to transcribe and tag digitized images from the vast collections of the library of congress. our program has a user-centered approach that supports exchange between many communities and the library of congress. the by the people program launched on october 24, 2018, featuring five digital collections already available on loc.gov from the manuscript division at the library of congress: selections from the papers of mary church terrell, letters written to abraham lincoln, clara barton’s diaries, branch rickey’s baseball scouting reports, and memoirs of disabled civil war veterans in the william oland bourne collection. the transcriptions created by volunteers improve access to and enhance the discoverability of handwritten and typed documents that computers cannot accurately translate without human intervention. transcriptions also increase access through greater compatibility with accessibility technologies such as screen readers used by people with low vision. they also provide legibility as an aid for people who have difficulty reading the source materials. the transcriptions are also the basis for keyword searching and, when served alongside their corresponding digitized images in loc.gov, support pathfinding in the library of congress digital collections for lifelong learners, students, teachers, and researchers. furthermore, we aim to share the data at various levels–page level, item level, and campaign level–and in amenable formats for those seeking to perform large scale textual analysis and explore library of congress collections with other methods. by the people is deeply informed by the ways the library of congress and cultural heritage organizations have invited the public into participatory projects. we have benefitted from the great design and experimentation that preceded our entry into this space. by the people is also a program guided by carefully articulated user-centered design principles. the design principles offer a framework for the path of development and a touchpoint for our decisions. in the design principles, we address privacy, ethics, broad participation, lowering the barrier to entry, and meeting people where they are. we also set expectations for clear communication, use and availability of data, and continuing to improve through user feedback. in sum, the design principles set a vision for the future of concordia and strong start for a program like by the people. our design principles can also be viewed as sequential phases crafted to inform one another: engage, understand, connect, and grow. working with these principles and goals as foundations for by the people, we have been able to better frame the software, collections selection, and ways we strive to meaningfully recognize the efforts of volunteers. we value volunteers as active participants and co-creators who help define new paths to our collections and data to facilitate access. an agile mindset the user-centered, participatory mission of by the people is reflected in our choice to implement an agile methodology for development of concordia. we needed to have an approach that allowed us to map and manage our work as we continue to learn more from our audiences and about the scope of collections that will be presented. we knew from the beginning that agile methods would be a core requirement for whatever software was to be developed. this approach suited what was to be both a new software implemented at the library of congress as well as a new program of engagement. agile allows for research-based hypothesis testing, releasing early and often, risk management, integrating user feedback, a primacy on close communication between motivated individuals across roles, and reflection on how to be more effective and adjust accordingly. [1] the project kicked off with a technical lead, a product owner-program lead, a stakeholder group with 6 representatives from across the library of congress, and a vendor team. six weeks prior to its public unveiling and formal launch, our core project team consisted of the technical lead and product owner-program lead, product manager/community manager, senior it specialist, two additional community managers, and 20% part-time designer/ux expert. this group participated directly in the agile process. the stakeholder group remained involved in the process through weekly report outs and monthly meetings. at the beginning of each sprint cycle, we started with an overarching idea of what we wanted to achieve and, through the agile process, added in details and made refinements as we went along. using this process allowed us to stay focused on the most meaningful aspects of the project and to continue iterating while avoiding unnecessary complexity and feature creep. we used agile ceremonies to manage three parallel tracks of software development, collections management, and communication. we joined these efforts together in our daily check-ins. these daily check-ins complemented weekly sprint grooming, as well as sprint closeouts that included demonstration and retrospectives (“retros”) every two weeks. even though we knew an agile approach would lead us through product development, we also knew it would take more than a good piece of software to make the program of engagement a success. identifying collection materials to feed into the software, and figuring out the manner of getting transcription data out of it, were projects unto themselves. similarly, the task of managing outreach and user engagement around the crowdsourcing program were significant enough to necessitate a dedicated effort. to that end, three community managers were hired to support the concordia development process and concurrently build out what would be by the people outreach internally and externally, identify launch collections needing transcriptions, and collaborate on data migration and design. as we jumped into an agile methodology, we realized its suitability for developing concordia was extended to managing the program of engagement for by the people. an agile approach also has allowed for responsive, flexible community engagement and collections and content selection. as a way of thinking and doing, agile suited the concordia project team and showed strongly when the team was called to adapt along the way. one thing we observed about our product team: in each phase described below the responsibilities and time commitment of team members shifted to some extent. in each sprint, we refined the responsibilities, process, and level of involvement in specific tasks based on the outcomes of retros and the context of the project. those conditions included roadblocks in development, assessing content for launch, and balancing wider organizational priorities. this process of adjustment allowed us to focus on responding to the state of the product and its needs – before launch, as we established the public program, and now as we move forward with improving and implementing a user-centered product. concordia development phase in this section, we define the product development phases and key decisions, as well as the roles and activities of the product team, in those phases: setting the stage, into development and launch, and maintaining and improving. setting the stage as we kicked off planning and development, the team identified some technical considerations and constraints. first, this would be a containerized python-django-postgres-etc web application. we intentionally selected mature technologies for the stack so we avoided pitfalls of managing rapid change and experimentation in two areas; therefore we could move forward with trying experimental work in a familiar stack. additionally, this stack aligns to those that support other services at play at the library of congress. the decision allowed us to build a tool aligned with existing practices and services at the library of congress. it also means that it is more likely that concordia and by the people could be supported by existing staff in the long term and grow according to user needs and institutional goals. furthermore, these decisions resulted in a tool that was developed openly to be possibly useful for other organizations using similar stacks. secondly, we knew we wanted to host concordia and by the people in the cloud with no on-premises dependency to benefit from serverless technologies, auto-scaling, and fault recovery without having to implement those ourselves; we also sought to avoid any performance limitations or strain on our coverage and on-premises data center. using our existing content delivery network or image servers in real time to serve materials on the site wouldn’t be sufficient, since we would need to provide scalable access to many distributed users at the same time. we piloted a partial solution in a crowdsourcing experiment launched by library of congress labs in 2017. this pilot project, called beyond words, allowed us to pinpoint challenges with serving high resolution images—necessary for volunteers to zoom in on detail as they transcribe and review—via our content delivery network. for beyond words, we experimented with requesting images and metadata via the chronicling america application programming interface (api), conversion of images in the cloud, and delivery from aws s3 buckets to the crowdsourcing interface. we were able to bring some of these approaches forward into concordia. a third technical consideration: we wanted to use the publicly-available loc.gov api to call the collection metadata and images in jpeg format and save a copy for use in concordia. this would leverage our existing technology and build upon the work already undertaken to present our digital collections through a public interaction layer in https://www.loc.gov. another technical consideration and decision was to use openseadragon as an image viewer. library of congress staff contributed to openseadragon and it is used in library of congress public interfaces. with this experience, we can be prepared to update or adopt a different viewer should any need arise. finally, we wanted to use the bagit package format for completed transcription data to be joined with item records. the library of congress helped co-author the specification for bagit, which celebrates its 10th anniversary this year. bagit workflows are used daily at the library of congress to manage many collections. matching concordia’s capabilities and functionality to existing library of congress technical workflows and collections management practices would allow by the people to align with staff efforts. we believed that alignment would, in turn, support lightweight processes to reunite volunteer-created text with the source digital assets for access and preservation. we also needed to blend technical considerations with project design considerations. at the center of managing concordia was a practical reality: the project had a fixed launch date. as a direct result, we had to be flexible on functionality requirements, including narrowing scope based on setbacks during development. at the heart of concordia’s beginnings were 4 design principles: engage, understand, connect, and grow. these principles provided foundations to pursue the project aims of engaging and inspiring audiences through trust and approachability. key stakeholders from the office of the chief information officer, library services, and national and international outreach had a brainstorming session prior to development kickoff, during which we created 55 user stories. the product owner and technical lead mapped these user stories to ranked priorities and technical needs, and further categorized them in relation to the design principles. we then identified an essential subset of the user stories in the highest priority category. we used these user stories to scope the development and focus on building a web application that would allow us to best support volunteers, get the best quality transcriptions, and provide the least friction in task workflows. the functionality must-haves we knew we needed from the beginning were: full-text transcription in a plain text editor (no rich text editing) allow anonymous transcriptions ethical design for user registration for contribution tracking transcription mapped to unique identifiers at the asset level some kind of tagging at the image level we further articulated functionality as we better understood user needs through testing: peer review to achieve completed transcriptions that supported transparency in the transcription workflow asset reservation system to prevent volunteers from clobbering each others’ work structured data model to set context for the materials (grouping items into projects and campaigns) these features and consideration were, we decided, the bare minimum needed to meet the design principles, create an engaging task workflow to inspire volunteers and encourage high quality transcriptions by volunteers and to get a start on collecting volunteer-contributed tag data. for transcription, we pursued a peer review workflow that allowed volunteers to see and edit each others’ work; we believed this would further deepen engagement with collections and shape community activity. tagging was a feature that was impacted by development setbacks. we decided we would take a baselining approach at initial launch; gathering and then evaluating tagging data. therefore, we skipped implementing any tag functionality other than adding and removing tags to images. it was more important to make sure the transcription experience was as smooth as possible while managing our constraints on time and staffing. figure 1. an early wireframe representing features determined for minimum viable product launch. into development and launch a note on the development environment: we use docker to run the database and message broker services supporting the application. using docker allows developers to get up and running quickly without having to install, configure, and run a list of prerequisites. the main concordia application itself can run in development as a docker container or using the django “runserver” command. in production, the application is deployed as a docker container in amazon web services elastic container service. developers have the ability to run the exact same docker image on their workstations as that which runs in the elastic container service in production. we kicked off development by creating tickets for the user stories that we considered the core essential functionality needed for a minimum viable product. as we worked sprint by sprint through those tickets, we identified discrete tasks that were of significant enough scope to warrant being documented in their own tickets and estimated level of effort using story points. from there, we broke user stories down into tasks which were small enough to be handled by a single developer in no more than a few days. this allowed us to see tangible movement on the application and measure progress one sprint at a time. the team worked intensively up to and through launch in late october 2018. in the sprint following launch, we continued to refine basic functionality and improve upon the core product. this process saw us address existing workflow and add significant refinement in 10 releases in the month and a half after by the people’s public unveiling. along the way, we balanced the minimum viable product approach across our team and with stakeholders; it was often challenging to keep quality high and focus clear while keeping expectations for the near and medium future in check. figure 2. the transcription interface at launch, featuring transcribing in progress. additionally, we grappled with the realities of intensive launch and stabilization activity, along with post-launch uncertainty. as our pre-launch roles and responsibilities across the product and program team shifted–and as we made steps toward building upon the basic functionality at launch–we identified some ways our process could be improved using agile methods. as we moved into active and then intensive development, agile’s promises showed strongly. it was essential across this period for the technical lead and product owner to work closely with the product manager to remain flexible, deliver and assess the product regularly, and continuously evaluate the needs of the project team with the circumstances of the product, as well as present risk and wider institutional context from many service units across the library of congress. maintaining and improving since launch, we have consistently improved concordia through 53 releases. the next functionality we needed to address after launch was the review workflow. we took time to consider the experience of review and the ways a volunteer might connect to that task. we gathered feedback via email, social media, surveys, and our discussion forum history hub. we also continued to explore the best way to design a flexible experience for volunteers, as their motivations may change through activity and over time. we landed on the idea of tracks as a means of continuing support for volunteers to engage more deeply, but also including choice to perform other activities including transcription, review, and tagging. this approach would allow them to explore by the people and contexts of items on https://loc.gov; and still ensure a workflow for high quality transcription and review. we implemented the functionality to support that experience in early 2019. in recent months, we have been focusing some of our next steps on prototyping a task-based approach, refining and expanding our onboarding, and data-driven improvement of the tagging feature. we continue to use agile principles as we layout our roadmap and determining priorities for these plans. figure 3. presenting the opportunity to volunteers to stay within the review experience–or to select a new task–after accepting a page during review. what worked for the team in this section, we would like to share some of the successful elements of our process in each sprint and across the phases of development. agile ceremonies were key in providing a framework for the team to reach iterative goals. we paid particular care to the rhythm of agile ceremonies, hosting design studios, using our design principles, and balancing priorities with consultation leading into sprint grooming. during development we moved between two week sprints and three week sprints. our regular agile ceremonies include sprint planning, sprint grooming, and a sprint close out that consists of demo and retro, followed by release. these ceremonies also provided structure for the product team, which made it possible for us to adapt to the conditions and needs of the product and the program. within the sprint planning process and during each sprint, developers were assigned tickets by the project manager. once the ticket is assigned, the next step is for the assigned developer to create a branch, do work on that branch, create a pull request, and assign another developer as the reviewer. the reviewer and the developer may go back and forth a few times before a pull request is approved. after that, the branch is merged into master, deployed to the continuous integration environment, and checked by a non-developer member of the product team. this tester may go back and forth with the developer (and possibly the reviewer as well) until all outstanding issues have been either resolved or documented as issues in the backlog. once the tester is satisfied, the issue can be closed and is considered done. we closed out sprints with retrospectives. these ceremonies provided the space for the team to make the necessary adjustments to the process. we approached retrospectives with an improvement mindset; in each retro we would identify what went well, what “was what it was,” and what could be improved. we dot voted to surface shared views in each category and discussed these topics in more detail. from there, the project manager defined action items, which we discussed in further detail with the technical lead and product owner, if the retro identified process needing adjustment. even beyond the formal agile ceremonies, this project was a true collaboration across service unts at the library of congress. we managed different expectations, forms of feedback, and the process of development with input from many different stakeholders who were not familiar with agile and represented different views and goals. our agile ceremonies allowed us to more effectively bring stakeholder input to our decision-making and prioritization. as we moved from implementation to improving the product, we used two related tactics to help move the product forward: design studios and prototyping passes. design studios are facilitated workshops that aim to bring many ideas to the table. while our design principles and our user stories provide a framework on identifying the goals we want to meet, how we execute meeting those goals can be interpreted in many different ways. design studios not only provide an opportunity to rapidly produce ideas and have them heard by the product team, they also serve to align the team by matching proposed features to our design principles. for example: volunteers were experiencing friction in the review process and finding an item to which they could contribute. the team reviewed the current flow of the site to identify where volunteers were having blockers. then the team was broken up into two groups and were asked to collaboratively define ways to remove those blockers. at hand, the teams had paper print outs of the site, markers, sticky notes, and blank paper to build a paper prototype. after individual brainstorming and team discussion, each team presented their idea to the other. it turned out the two teams had not only similar core ideas, but also complementary features and user flows in mind. we viewed this alignment as representative of our ongoing process to maintain shared vision. following that meeting, the product owner, technical lead, and project manager consolidated ideas and decided to build out a code prototype of those ideas. through three sprints, the development team intensively built versions of the prototype, which places primacy on selecting assets at the task level. we tested and released this new feature in the spring; along with user-testing and a survey, the prototype has allowed the by the people team to present collections to volunteers in new ways. registered volunteers could try the prototype when logged into by the people. figure 4. a view of the adjustable activity prototype interface featuring images waiting to be reviewed. a remote-friendly environment has made a difference for this team. this type of working condition is modern, flexible, accessible from anywhere, and boosts morale. the concordia team has incorporated this into how we work. working in this way, we reduce the stress of commuting, give more time to staff to dive deep into a technical problem, and have a streamlined way to connect with one another. while in-person meeting time is still very important, providing a remote option opens up the accessibility of participation and does not alienate opportunity. to complement this flexibility, we try to commit to holding backlog grooming and retrospectives and any design-related workshops in person. by normalizing video conferencing, online chat platforms, and discussion in github, collaboration can happen in many formats. we also consistently returned to the design principles to help refine expectations, determine product feature prioritization, thoughtfully respond to functionality (or efficiency or productivity) requests, and track the progress of the product. assessing approaches with agile we also wish to share details of the approaches that, once applied, were not quite adequate for this project. an agile methodology allowed us to identify, isolate, and address what didn’t work and create contextual, specific solutions. undertaking deep and robust user research practice remains a challenge with limited expertise available; however, in consultation with the library of congress design & development team, the concordia product team has identified lightweight research mechanisms. some of the ways we are exploring user research approaches include: refining understanding through user surveys brief in-person interviews on public engagement days creating a feature branch capability so that we can invite remote participants to provide feedback on functionality, requirements, and task barriers prototyping prior to full product releases deep listening to inform product roadmap and prioritization we also explored the idea of using the publicly-available iiif presentation metadata and extending an iiif viewer such as mirador. not all of the library of congress digital collections on www.loc.gov are available in iiif yet, and we did not want to become responsible for a major open-source project fork. the data model for iiif is focused on the item level, where one item may have many images representing many pages. we wanted to be able to break up all the pages in an item so that transcription activity could be isolated at the image level rather than the item level. although we have a great interest in making this tool compatible with other data source sets, we determined that functionality was not a must-have at launch and it would be expensive to implement up front. however, it may be a worthwhile trade-off to do so in the near future. an agile approach allows us to explore and adjust to that possibility should it arise. piloting workflows and approaches at the library of congress via concordia this crowdsourcing tool and program have cleared paths for new technical approaches and project management workflows. creating this tool at the library of congress allowed the product team to articulate standards in coding and continually improve communication. central to this discussion are the a) design considerations to align the tool with the technical approaches and requirements of the library of congress and b) allowing concordia to scale and adapt to new collections and user needs. we connected a series of existing workflows to accommodate user-generated content; so, new content delivered into existing workflows. for this project, collection materials are copied from their publicly-available hosted locations on www.loc.gov (which is on-premises) to cloud-based infrastructure, using our api. concordia makes them available on crowd.loc.gov and collects transcriptions provided by volunteers. then, bagit format is used to package up completed transcription data in a directory structure that mirrors the source content. from there, we send it through our accessioning workflow where it is preserved on long-term storage and lands back on www.loc.gov backend storage. finally, the bag is picked up by the www.loc.gov extract-transform-load (etl) process and newly mapped content made available on our main website. concordia continues to evolve through iterative development. at the time of writing, we have created and evaluated an activity-based prototype. as we described above, this approach allows volunteers to take action more quickly – such as those who are motivated by transcribing or review or want to contribute across collections or filter, rather than drilling down through specific content. in the coming months, we plan to apply a concentrated approach to refine to tagging. we expect to begin by evaluating data generated in our tagging pilot phase, engaging stakeholders around their ideal uses of tagging, through design studios and prototyping, and through interviews with users about the ways in which they respond to the opportunity to tag. conclusion we believe concordia advances the three core objectives of the digital strategy: to throw open the treasure chest, to connect, and to invest in our future. concordia was created as a user-centered project developed using an agile methodology. approaching concordia’s development in this way was an opportunity to identify and test key software development and project management practices that might be extended to other work across the library of congress. with this approach, we are integrating a view to invest in the future. we have been taking small steps and high touch on a product, guided by research-based and specific hypotheses but with leeway to evaluate and adjust as we learned. unpacking our process for concordia allows us to consider larger questions of services at the library of congress. as we look ahead, stakeholders will be called to consider how we can build scalable platforms that allow us to work with and use the collections instead of merely view them. as we continually improve concordia, we are extending opportunities to showcase and invite deeper involvement with our digital collections. we also seize opportunities to connect with a range of audiences from lifelong learners, educators, students, and researchers with by the people. we also want to share transparently with our peers in other organizations about the thinking and practices we used to create concordia. since our public launch of by the people in october 2018, more than 11,000 volunteers have registered and provided 232,000 contributions. over 30,000 pages have been completely transcribed and reviewed (with 58,000 in progress). we continue to return transcriptions for keyword searching with 6,700 transcribed pages now searchable on https://www.loc.gov. if you fancy it, try a search for lincoln and “live like princes”. figure 5. completed transcriptions presented alongside the image in https://loc.gov and attributed to by the people volunteers. we want to end this discussion by sharing that concordia is free to reuse and the development process is documented in our concordia github repository. we invite you to explore whether it can be of use to you. concordia improves as people participate, use the tool, and provide feedback across our repositories and forums. we hope that this article shares our process of creating concordia and demonstrates the ways the tool and an agile approach may be of use to other organizations. endnotes [1] agile at 18f. accessed 20 march 2019. available at https://agile.18f.gov/agile-is-something-you-are/ subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – assessing the potential use of high efficiency video coding (hevc) and high efficiency image file format (heif) in archival still images mission editorial committee process and structure code4lib issue 41, 2018-08-09 assessing the potential use of high efficiency video coding (hevc) and high efficiency image file format (heif) in archival still images both hevc (iso/iec 23008–2) video compression and the heif (iso/iec 23008-12) wrapper format are relatively new and evolving standards. though attention has been given to their recent adoption as a jpeg replacement for more efficient local still image use on consumer electronic devices, the standards are written to encompass far broader potential application. this study examines current hevc and heif tools, and the standards’ possible value in the context of digital still image archiving in cultural heritage repositories. by michael j. bennett introduction high efficiency video coding (hevc), also known as h.265, is a video compression standard that has recently received attention for its use with still image data. with apple’s 2017 announcement of their planned deployment of hevc-encoded high efficiency image file format (heif) files throughout the company’s software and hardware ecosystems, came the news of a possible jpeg replacement that would provide more efficient image storage and transfer in consumer electronic devices such as smartphones and tablets (concion, 2017; shah & thomson, 2017; “high efficiency video coding,” 2018; “high efficiency image file format,” 2018). in this context and in much of the literature that surrounds hevc and heif’s potential as highly efficient still image data standards, the focus has mostly centered on mathematically lossy but visually lossless use (lainema, hannuksela, vadakital, & aksu, 2016; nguyen & marpe, 2015; hannuksela, lainema, & vadakital, 2015; hanhart, rerabek, korshunov, & ebrahimi, 2013). the aims of compression in archival still imaging within the cultural heritage domain, however, are more particular and tend to focus on retaining all rendered bits in a mathematically lossless manner. when lossy compression is considered, it is generally done on the more conservative end of the visually lossless acceptance scale (rieger, 2016). in turn, this study evaluates hevc’s attributes as both a mathematically lossless and as a visually lossless standard with these particular aims in mind and compares hevc to the jpeg 2000 specification which has gained steady adoption among cultural heritage institutions over time. basic techniques for successfully encapsulating hevc bitstreams in the new heif wrapper format are also examined in detail. methods and results figure 1. reference 8-bit, srgb, tiff images (left-right, top-bottom) “born_digital,” “medium_format_color_01,” “medium_format_color_02,” “map,” “card” (click image to see full detail version) test 1: lossless hevc/h.265 (bitstream) vs. lossless jpeg 2000 (rendered 8 bit srgb) step 1: reference 8 bit, srgb tiff > lossless hevc bitstream reference tiff files (bennett, 2018) were converted through ffmpeg v3.4.1’s use of the open source x265 hevc encoder (multicoreware inc., 2018) to a mathematically lossless bitstream with the following command: ffmpeg -i reference.tif -preset veryslow -c:v libx265 \ -x265-params lossless=1 lossless_bitstream.265 the x265 encoder’s lossless option disables rate control along with all quality metrics and bypasses both discrete cosine transforms (dct) and quantization (multicoreware inc., 2014). additionally, it employs hevc’s rext main 4:4:4 profile, level-8.5 (main tier). to determine whether the hevc-encoded bitstreams were truly mathematically lossless and reversible, the bitstreams were first converted to uncompressed output tiffs: ffmpeg -i lossless_bitstream.265 -compression_algo 1 \ -preset veryslow -pix_fmt rgb24 output.tif these output tiffs were then compared to the original reference tiffs through peak signal-to-noise ratio (psnr) analysis: ffmpeg -i reference.tif -i output.tif \ -filter_complex psnr output_psnr.tif subsequent psnr results for all rgb channels confirmed successful mathematically lossless hevc encoding (psnr values of infinity indicate that there is no difference between two given input signals): psnr r:inf g:inf b:inf average:inf min:inf max:inf step 2: reference 8 bit, srgb tiff > lossless jpeg 2000 reference tiff files were converted to mathematically lossless jpeg 2000 files through the following openjpeg v2.3.0 command: opj_compress -i input.tif -o output.jp2 note: openjpeg uses the following default conversion settings when only the input and output files are specified in a given command: lossless compression 1 layer of quality 1 tile size of precinct 215 x 215 (means 1 precinct) size of code-block 64 x 64 number of resolution (i.e. dwt decomposition level): 6 no sop marker in the codestream no eph marker in the codestream no sub-sampling in x and y direction no mode switch activated progression order: lrcp no index file no roi upshifted no offset of the origin of the image no offset of the origin of the tiles reversible dwt 5-3 (openjpeg, 2015/2018) the resulting output jpeg 2000s were then similarly compared to the original reference tiffs through psnr analysis to confirm lossless encoding: ffmpeg -i reference.tif -i output.jp2 \ -filter_complex psnr output_psnr.tif reference tiff file width (px) height (px) tiff file size (bytes) mathematically lossless hevc bitstream file size (bytes) mathematically lossless jpeg 2000 file size (bytes) percent difference filesizes: lossless hevc/heic vs lossless jpeg 2000 born_digital_color.tif 3,840 5,760 66,389,704 34,137,689 22,011,085 27 medium_format_color_01.tif 6,090 7,920 144,724,980 50,610,168 43,842,691 9 medium_format_color_02.tif 6,730 5,206 105,134,840 46,383,234 42,883,096 5 map.tif 14,540 3,104 135,450,536 79,008,269 54,362,217 23 card.tif 1,850 1,302 7,265,360 4,303,967 2,686,857 29 table 1. comparative file sizes, mathematically lossless hevc bitstream vs. mathematically lossless jpeg 2000 figure 2. graph of mathematically lossless compression comparison by filesize (click image to see full detail version) test 2: visually lossless heif (~42db ave rgb psnr) vs. visually lossless jpeg 2000 (~42db ave rgb psnr rendered 8 bit srgb) step 1: reference 8 bit, srgb tiff > hevc yuv420p bitstream reference tiff files were converted to hevc-encoded bitstreams through the following ffmpeg command: ffmpeg -i reference.tif -preset veryslow -c:v libx265 -crf 17 \ -pix_fmt yuv420p crf17_yuv420p_bitstream.265 note: given this ffmpeg command syntax, the x265 encoder employs hevc’s main 4:2:0 profile, level-5 (main tier). also, constant rate factor (crf) settings were used in an iterative manner on an image-by-image basis to create hevc-encoded bitstreams and subsequent rgb heif files with targeted psnr values. the 42db psnr figure represents averaged rgb and falls in the middle of the range of commonly understood psnr values that suggest a visually lossless comparison between two color still image inputs (lainema et al., 2016; hannuksela et al., 2015; comstock, 2011). in addition, the yuv420p pixel format (yuv planar color space with 4:2:0 chroma subsampling) was used in order to ensure subsequent best operation with the nokia heif writer library and example application described below. at the time that these tests were commenced in december 2017, the heif writer v2.0 was unable to produce renderable files from anything other than yuv420p color or 8 bit source data, though the heif format specification itself outlines support for alternative color spaces and bit depths (lainema et al., 2016; hannuksela et al., 2015). next, the resulting bitstreams were converted to the heif wrapper file format through nokia’s heif writer application v2.0 (nokia technologies, 2015/2017): ./bins/writerapp ./config.json finally heif files were then converted in ffmpeg to uncompressed rgb tiff derivatives for objective psnr evaluation against original reference tiffs and to iteratively fine-tune crf settings in original hevc bitstream creation: ffmpeg -i reference.tif -i 42db_hevc_heif.tif \ -filter_complex psnr output_psnr.tif step 2: reference 8 bit, srgb tiff > visually lossless jpeg 2000 (~42db ave rgb psnr rendered 8 bit srgb): openjpeg allows users to specify target psnr values when encoding jpeg 2000 files through the program’s -q switch command line control. in practice, it was discovered that setting the -q switch to a value slightly above the targeted psnr figure produced jpeg 2000 files that best matched the intended psnr value. the -i command toggles irreversible color transformation (ict) in place of the reversible color transformation (rct), and the irreversible discrete wavelet transform (dwt) 9-7 filter in place of the 5-3 filter. for visually lossless situations, the 9-7 transform is more suited than the 5-3 transform in conditioning image data for superior compression results. this comes at the cost of being unable to recover the original data due to round-off errors in its calculations. because of these round-off errors, the 9-7 transform is irreversible (buckley & tanner, 2009). in turn, reference tiffs were converted to visually lossless jpeg 2000s using the following openjpeg command: opj_compress -i reference.tif -i -q 43 -o lossy_q43_output.jp2 resulting jpeg 2000 files had the following profile, as verified by jpylyzer v1.18.0 (“jpylyzer,” 2017): lossy compression 1 layer of quality 1 tile no precincts size of code-block 64 x 64 number of resolution (i.e. dwt decomposition level): 5 no sop marker in the codestream no eph marker in the codestream no sub-sampling in x and y direction no mode switch activated progression order: lrcp no index file no roi upshifted no offset of the origin of the image no offset of the origin of the tiles irreversible dwt 9-7 reference tiff file width (px) height (px) tiff file size (bytes) visually lossless hevc/heic yuv420p crf value visually lossless hevc/heic file size (bytes) visually lossless jpeg 2000 -q 43 file size (bytes) percent difference filesizes: visually lossless hevc/heic vs visually lossless jpeg 2000 visually lossless hevc/heic psnr (db) visually lossless jpeg 2000 -q 43 psnr (db) born_digital_color.tif 3,840 5,760 66,389,704 11 4,246,957 3,876,338 6 41.721753 42.388142 medium_format_color_01.tif 6,090 7,920 144,724,980 17 1,228,082 699,375 34 42.475259 42.303822 medium_format_color_02.tif 6,730 5,206 105,134,840 7 7,949,377 4,319,621 36 42.369392 42.619542 map.tif 14,540 3,104 135,450,536 3 20,062,510 11,500,289 33 41.730134 42.182009 card.tif 1,850 1,302 7,265,360 7 776,759 684,338 8 41.777099 42.399001 table 2. comparative metrics, visually lossless hevc-encoded heic vs. visually lossless jpeg 2000 figure 3. graph of visually lossless compression comparison by filesize (click image to see full detail version) discussion in examining the results of mathematically lossless compression (table 1 and figure 2) it is interesting to consider that across the five test images jpeg 2000 encoding provided an average of 19% file size reduction compared to hevc. this may be due in part to the limitations of hevc’s main profile lossless options. it should be noted that this preliminary study’s scope did not venture beyond the testing of hevc’s main profile and jpeg 2000, part 1. hevc is an intricate specification, and its growing ecosystem of format range extensions to the main profile are designed to support 16 bit data and yuv444 chroma sampling among other features (“hevc screen content coding extension (scc) | jct-vc,” 2016; “hevc format range extension (rext) | jct-vc,” 2014). however the specification’s mathematically lossless still image encoding option, which may leverage various features of the range extensions, remains mostly absent from the literature and was conspicuously missing from the 2016 icip compression challenge (alexiou et al., 2016). in turn, future testing and investigation is warranted in this regard as the extensions and the tools that support their implementation further develop. visually lossless testing attempted to closely follow elements of real world application of compression on source rgb still image files. as a result, the nokia heif writer application, though limited in the type of data that it can currently address, was an important tool. essentially it imports an hevc-compressed still image codestream and wraps it in a file format that allows the image data to be decoded as an rgb image. these heif images can then be opened and comparatively analyzed in common viewers and editing tools that the reference tiff and compressed jpeg 2000 files share. for this study, visual comparisons were subsequently made on an apple macbook pro (os x 10.13.3, 16gb ram, 3.1 ghz intel core i7 processor, 1tb ssd) with 13 inch 2,560 x 1,600 pixels color-calibrated display. with its october 2017 release of adobe creative cloud 2018 (19.0.0), photoshop first acquired the ability to view (but not save out) heif files. in turn, the resulting ~42db psnr heif files (with .heic extensions) could be subjectively assessed against their original reference tiff files and similarly derived jpeg 2000s through comparative side-by-side zooming, panning, and pixel-level difference visualizations in photoshop. heif files created by the nokia application render as 8 bit, srgb files when natively opened in photoshop (i.e. no default photoshop color profile adjustments applied). visually lossless jpeg 2000 compression provided an average of 23% file size reduction compared to visually lossless hevc across the five test images (table 2 and figure 3). however, results varied among images based most likely upon content and overall size. this variance was most pronounced in the “medium_format_color_02” image which jpeg 2000 was able to deliver a 36% file size reduction compared to hevc/heif. the smallest percent file size savings that jpeg 2000 afforded was 6% and 8% in the “born_digital_color” and “card” files respectively. a closer examination of the “map” image illustrates the different techniques that hevc and jpeg 2000 encoding take towards image data. utilizing the image’s hevc/heif and jpeg 2000 versions as individual layers over the reference tiff background layer allowed for the use of photoshop’s difference blending mode and the addition of a threshold adjustment layer to visualize pixels changed due to each compression method. through this visualization technique, pixels that changed from the background reference image layer to the compressed upper layer appeared white. those unaltered appeared black. each compressed layer’s visibility could easily be toggled on or off for individual comparison against the reference tiff. figure 4. photoshop difference blending mode and threshold adjustment layers figure 5. “map.tif” reference image (click image to see full detail version) figure 6. “map.tif” reference image with visually lossless (42db psnr) hevc/heif difference blending mode layer. changed pixels appear white. (click image to see full detail version) figure 7. “map.tif” reference image with visually lossless (42db psnr) jpeg 2000 difference blending mode layer. changed pixels appear white. (click image to see full detail version) as can be seen in figures 6 and 7, hevc and jpeg 2000 work in somewhat different ways. both, however, leverage traits of the human visual system model (hvs) in order to perform adaptive encoding based upon image content. though the normative portion of jpeg 2000 part 1 does not explicitly mention the hvs, it nonetheless uses wavelet transform filters that assign various transform levels to high and low frequency image areas selectively through sub-band coding. various quantizer step sizes for different sub-bands can also be adaptively employed based upon the dynamic range of the sub-band (schelkens, skodras, & ebrahimi, 2009; allen & triantaphillidou, 2011). this results in subtle shifts in compression levels throughout an image based upon such variances (figure 7). one similar technique through which hevc performs its own adaptive encoding is through smart quantization which increases compression levels in areas of an image where the hvs is less perceptive to compression artifacts, such as high frequency image content. additionally, the standard employs an in-loop deblocking filter to high contrast edges. blocking artifacts are visible discontinuities that occur at coded block boundaries and are particularly symptomatic of jpeg and mpeg-based discrete cosine transform (dct) coders like hevc. the hevc deblocking filter performs detection of such artifacts and attenuates them by applying a selected filter. through this process, block boundary transitions can be smoothed which can result in better qualitative visual results. in addition, hevc also supports a second in-loop filter termed, sample adaptive offset (sao). sao is a local filter that is adaptively applied to the output of the initial deblocking filter and can further improve subjective image quality (concion, 2017; sze & budagavi, 2014; norkin et al., 2012). note: due to its discrete wavelet transform (dwt) architecture, jpeg 2000 normally avoids classic dct blocking artifacts, unless tiling is heavily used (schelkens et al., 2009). zooming into a particular area of adjacent high and low frequency content in the “map” image starkly reveals these hevc coding features selectively at work and allows for comparison to the more uniform compression of jpeg 2000. figure 8. “map.tif” reference image 100% zoom. (click image to see full detail version) figure 9. “map.tif” reference image with visually lossless (42db psnr) hevc/heif difference blending mode layer 100% zoom. changed pixels appear white. (click image to see full detail version) figure 10. “map.tif” reference image with visually lossless (42db psnr) jpeg 2000 difference blending mode layer 100% zoom. changed pixels appear white. (click image to see full detail version) it bears noting that subjective visual assessment of all individual versions of the “map” image without difference blending applied revealed no perceptible difference among versions at 100% – 200% zoom. so, despite hevc’s adaptive encoding techniques, jpeg 2000 still produced a 33% smaller file in this example. future testing of hevc’s screen content coding extension, which is specifically optimized for similarly mixed text and graphics content as maps, is warranted for this format type (“hevc screen content coding extension (scc) | jct-vc,” 2016). an examination of the “medium_format_color_01” image using difference blending produced similar visualization results as the “map” image. once again, high frequency image areas received a noticeably higher proportion of overall hevc compression relative to areas with less detail. jpeg 2000, on the other hand, again applied its compression more evenly across the entire image. figure 11. “medium_format_color.tif” reference image. (click image to see full detail version) figure 12. “medium_format_color_01.tif” reference image with visually lossless (42db psnr) hevc/heif difference blending mode layer. changed pixels appear white. (click image to see full detail version) figure 13. “medium_format_color_01.tif” reference image with visually lossless (42db psnr) jpeg 2000 difference blending mode layer. changed pixels appear white. (click image to see full detail version) though 34% larger in resulting file size vs. jpeg 2000, here hevc’s adaptive encoding appeared to have paid discernable dividends. based upon subjective visual inspection, general skin texture and the repeating patterns of the shortest boy’s shirt and suspenders are more clearly rendered in the hevc/heic file than in the jpeg 2000 file where the patterns’ contrast borders appear somewhat blurred at 200% zoom (figure 14). blurring is a known jpeg 2000 compression artifact that appears at higher levels of compression (schelkens et al., 2009; jakulin, 2002; allen & triantaphillidou, 2011). figure 14. reference tiff, visually lossless (42db psnr) hevc/heif, and visually lossless (42db psnr) jpeg 2000 versions of “medium_format_color_01” image, 200% zoom. (click image to see full detail version) it should be stressed that these local artifacts were very slight and only began to be noticeable at 200% zoom. however, their presence corroborates buckley’s observation on the shortcomings of using psnr as a quality metric for compressed images… this behavior [visible local artifacts] also highlights the limitations of a global metric such as psnr-a single number to describe the effect of compression over the entire image, although compression does not affect all parts of an image equally. larger distortions and very visible compression artifacts in some parts of the image can be offset by smaller distortions in the rest of the image to give an average distortion for the entire image that would be acceptable if it occurred uniformly over the image (buckley, 2013). conclusion this study represents an introductory evaluation of hevc/heif in the context of potential real-world application within cultural heritage imaging using currently available hevc encoding and heif rendering tools. to date, hevc/heif support remains either in its infancy or is non-existent among most still image editing software packages. with this in mind, it bears noting that comparative file size results between hevc/heif and jpeg 2000 were mostly dissimilar to those found in the existing literature (alexiou et al., 2016; lainema et al., 2016; hannuksela et al., 2015). this may be explained in part due to the fact that research to date has focused more strictly on hevc and jpeg 2000 encoder comparisons using y luminance channel psnr analysis of yuv bitstreams, which primarily gauges spatial image quality, not color quality. current real-world archival adoption of file formats such as tiff and jpeg 2000, however, centers on rgb color. therefore, this investigation has made end-to-end comparisons not of hevc and jpeg 2000 encoders exclusively, but of fully rendered rgb formats based upon these encodings instead. as a result, yuv to rgb color conversion steps, often excluded from pure codestream comparisons, were necessarily introduced as part of the final comparative renderings (and averaged rgb psnr tests). such transforms can lead to varying degrees of fidelity loss depending upon the conversion software used. in the case of heif, it should also be noted that the nokia freeware writer is sparsely documented, proprietary, and does not purport to be a reference implementation of heif. it is instead offered as a demonstration tool for early testing of hevc/heif 8 bit srgb renderings, which was how it was utilized in this inquiry. in addition, its constraints to yuv420p source data cited herein may be more a result of the limits of current heif viewers rather than the writer itself. nonetheless, at present there is no other comparable heif tool available that can similarly encapsulate high-quality hevc encodings into visually lossless heif files. it is hoped that adobe, for one, will someday add flexible quality-controlled photoshop hevc/heif export support to its current read-only capabilities. despite these restrictions, hevc/heif vs. jpeg 2000 comparisons remain visually intriguing, for they offer glimpses into the unique underlying compression techniques that each specification employs. hevc as a next generation dct-based technology, for example, uses sophisticated filtering to address compression artifacts that have traditionally manifested themselves in previous block-based specifications like jpeg and mpeg. jpeg 2000, on the other hand, employs discrete wavelet transforms that mostly avoid such blocking but can create other visual artifacts at higher compression ratios. in summary, this initial study’s findings are inconclusive with respect to the hevc/heic’s tangible advantages as an archival still image format. additionally, hevc’s evolving patent landscape (ozer, 2018) which was outside the scope of this examination, continues to cloud the specification’s prospects within the cultural heritage domain which closely considers how patents can infringe on digital formats’ long-term sustainability (u.s. library of congress, 2017). however, with its asserted file size savings and extensible architecture that is designed for future computational photography data management, the specification bears continued assessment as the software that supports it broadens and matures. about the author michael j. bennett is head of digital imaging and conservation at the university of connecticut. there he oversees the digital capture and conservation operations for the university’s archives and special collections. his research interests include technologies and techniques that focus on digitization, post-processing, and 2d and 3d data formats. he holds a ba from connecticut college and an mlis from the university of rhode island. bibliography alexiou, e., viola, i., krasula, l., richter, t., bruylants, t., pinheiro, a., … ebrahimi, t. (2016, september). overview and benchmarking summary for the icip 2016 compression challenge. presented at the 23rd international conference on image processing, phoenix, az. retrieved from https://jpeg.org/downloads/aic/wg1n73041_icip_2016_grand_challenge.pdf allen, e., & triantaphillidou, s. (2011). the manual of photography (10th ed.). oxford; burlington, ma: elsevier/focal press. retrieved from http://uconn.worldcat.org/title/manual-of-photography/oclc/280389938?referer=di&ht=edition bennett, m. j. (2018). dataset to assessing the potential use of high efficiency video coding (hevc) and high efficiency image file format (heif) in archival still images. retrieved from https://opencommons.uconn.edu/libr_pubs/62 buckley, r. (2013). using lossy jpeg 2000 compression for archival master files. retrieved from http://www.digitizationguidelines.gov/still-image/documents/jp2lossycompression.pdf buckley, r., & tanner, s. (2009). jpeg 2000 as a preservation and access format for the wellcome trust digital library, 17. retrieved from https://wellcomelibrary.org/content/documents/22082/jpeg2000-preservation-format.pdf comstock, w. (2011, may). using psnr thresholds to modulate the degree of lossy compression in jpeg2000 files. presented at the jpeg 2000 summit workshop, library of congress. retrieved from http://www.digitizationguidelines.gov/still-image/documents/comstock.pdf concion, d. (2017). high efficiency image file format – wwdc 2017 – videos – apple developer. retrieved december 3, 2017, from https://developer.apple.com/videos/play/wwdc2017/513/ hanhart, p., rerabek, m., korshunov, p., & ebrahimi, t. (2013). subjective evaluation of hevc intra coding for still image compression. presented at the seventh international workshop on video processing and quality metrics for consumer electronics – vpqm 2013, scottsdale, az. retrieved from https://infoscience.epfl.ch/record/182874/files/vpqm2013.pdf hannuksela, m. m., lainema, j., & vadakital, v. k. m. (2015). the high efficiency image file format standard [standards in a nutshell]. ieee signal processing magazine, 32(3), 150-156. https://doi.org/10.1109/msp.2015.2419292 hevc format range extension (rext) | jct-vc. (2014). retrieved march 10, 2018, from https://hevc.hhi.fraunhofer.de/rext hevc screen content coding extension (scc) | jct-vc. (2016). retrieved march 10, 2018, from https://hevc.hhi.fraunhofer.de/scc high efficiency image file format. (2018, january 6). in wikipedia. retrieved from https://en.wikipedia.org/w/index.php?title=high_efficiency_image_file_format&oldid=819009945 high efficiency video coding. (2018, january 4). in wikipedia. retrieved from https://en.wikipedia.org/w/index.php?title=high_efficiency_video_coding&oldid=818680480 jakulin, a. (2002, 2004). baseline jpeg and jpeg2000 artifacts illustrated. retrieved march 24, 2018, from http://www.stat.columbia.edu/~jakulin/jpeg/artifacts.htm jpylyzer. (2017). retrieved march 27, 2018, from http://jpylyzer.openpreservation.org/ lainema, j., hannuksela, m. m., vadakital, v. k. m., & aksu, e. b. (2016). hevc still image coding and high efficiency image file format. in 2016 ieee international conference on image processing (icip) (pp. 71-75). phoenix, az. https://doi.org/10.1109/icip.2016.7532321 multicoreware inc. (2014). lossless – x265 documentation. retrieved february 9, 2018, from http://x265.readthedocs.io/en/default/lossless.html multicoreware inc. (2018). x265 hevc encoder / h.265 video codec. retrieved march 29, 2018, from http://x265.org/ nguyen, t., & marpe, d. (2015). objective performance evaluation of the hevc main still picture profile. ieee transactions on circuits and systems for video technology, 25(5), 790-797. nokia technologies. (2017). heif: high efficiency image file format. c++. retrieved from https://github.com/nokiatech/heif (original work published 2015) norkin, a., bjontegaard, g., fuldseth, a., narroschke, m., ikeda, m., andersson, k., … van der auwera, g. (2012). hevc deblocking filter. ieee transactions on circuits and systems for video technology, 22(12), 1746-1754. https://doi.org/10.1109/tcsvt.2012.2223053 openjpeg: official repository of the openjpeg project. (2018). c, université catholique de louvain. retrieved from https://github.com/uclouvain/openjpeg (original work published 2015) ozer, j. (2018, march 13). hevc advance cuts content fees on streaming – streaming media magazine. retrieved april 28, 2018, from http://www.streamingmedia.com/articles/news/online-video-news/hevc-advance-cuts-content-fees-on-streaming-123828.aspx rieger, t. (2016, september). fadgi technical guidelines for digitizing cultural heritage materials, september 2016. retrieved from http://www.digitizationguidelines.gov/guidelines/fadgi%20federal%20%20agencies%20digital%20guidelines%20initiative-2016%20final_rev1.pdf schelkens, p., skodras, a., & ebrahimi, t. (2009). the jpeg 2000 suite. hoboken, nj: wiley. shah, a., & thomson, g. (2017). introducing heif and hevc – wwdc 2017 – videos – apple developer. retrieved december 3, 2017, from https://developer.apple.com/videos/play/wwdc2017/503/ sze, v., & budagavi, m. (2014). design and implementation of next generation video coding systems (h.265/hevc tutorial). presented at the ieee iscas 2014, research laboratory of electronics at mit. retrieved from http://www.rle.mit.edu/eems/wp-content/uploads/2014/06/h.265-hevc-tutorial-2014-iscas.pdf u.s. library of congress. (2017, march 9). sustainability factors. retrieved april 28, 2018, from https://www.loc.gov/preservation/digital/formats/sustain/sustain.shtml subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – improving enterprise content findability through strategic intervention mission editorial committee process and structure code4lib issue 42, 2018-11-08 improving enterprise content findability through strategic intervention this paper highlights work that information specialists within the jet propulsion laboratory have done to strategically intervene in the creation and maintenance of jpl’s intranet. three key interventions are discussed which best highlight how work in enterprise “knowledge curation” fits into emergent knowledge management roles for institutional librarians (lustigman, 2015). these three interventions are: 1) guided document creation, which includes the development of wiki portals and standard editing processes for consistent knowledge capture, 2) search curation, which includes manual and organic enterprise search relevancy improvements, and 3) index as intervention, which describes how metadata mapping and information modeling are used to improve access to content for both local and enterprise-wide applications. by rebecca townsend and camille mathieu introduction for any organization with significant digital content, the ability to search across this content has become an operational necessity. despite this, unified enterprise search and retrieval of digital content remains an elusive goal for many organizations (miles, 2014). the large amount of time that knowledge workers spend trying to find relevant information when they need it, in addition to the time they spend re-creating information when it cannot be found, represents a significant cost to any knowledge-producing organization (feldman and sherman, 2001). in the two decades since this “high cost of information” was uncovered, many commercial enterprise content management solutions and consulting agencies have cropped up which strive to diminish this cost. one approach in solving findability issues is to rely on technology investment and the implementation of artificial intelligence, machine learning, natural language processing, and similar cutting-edge tools (findwise, 2016; miles, 2014). additional and very different solutions crop up when the focus shifts to enterprise content itself. rather than focusing efforts and investments on new technologies, it is possible and potentially more valuable to address issues of findability and aggregability at the source, from the moment knowledge is captured in an enterprise information ecosystem. one such enterprise information ecosystem underlies the daily operations of the nasa/caltech jet propulsion laboratory in pasadena, ca. the jet propulsion laboratory (jpl) – a world leader in robotic deep space planetary missions, earth monitoring systems, and associated science and technology – has an intranet with digital content housed in over 100 different repositories, applications, and other storage locations. the storage locations contain documents and other items in a variety of formats, created by thousands of different knowledge workers (employees whose daily work products rely on interactions with enterprise information) over the course of the ecosystem’s life. while some of this information is structured, the majority is unstructured. because of this proliferation of repositories, as well as the inconsistent description of content within them, critical information is frequently siloed and difficult to access. as a result, it has been challenging for jpl to implement findability tools which can effectively search across siloes or manage content in a unified way. while jpl has invested in a number of technology solutions designed to allow for better content findability, the software approach has not been the only approach taken by the institution in addressing this challenge. jpl is a matrix organization; one of the results of this structure is that, historically, formal knowledge management has been mandated for some work (i.e. flight projects), while significantly less knowledge management has been mandated for information produced as a result of daily business operations. this paper will review some of the work jpl has recently engaged in to fill this gap and better maintain the knowledge it produces. specifically, it will review a recent “knowledge curator” pilot role, and the projects and outcomes of this role pilot. this paper will argue that, regardless of the specific content management system (cms) used by an organization, effective enterprise content findability requires some manual review and editing in order to produce results that meet users’ expectations. additionally, it will make the case for an expansion of traditional library and information science (lis) roles to include tasks associated with “knowledge curation,” specifically for enterprise intranets and other finite information ecosystems. findability and the enterprise studies conducted since the establishment of the high cost of information in 2001 have consistently found that information findability remains a central issue for many knowledge workers and their employers. an industry study from 2013 found, for example, that “60% of information workers say it is difficult to find the right information,” and that, by spending “an average of 8.8 hours every week searching for information,” workers were costing their organizations “$14,000 per employee each year” in lost productivity (leher, 2013). another study, conducted three years later, found that “around 45% of respondents” said that “they are dissatisfied with the enterprise search applications,” while a third of respondents say it is difficult to find information in their organization in general (findwise, 2016). this continuing dissatisfaction has many root causes, one of the biggest drivers of this ongoing issue is the fact that user expectations are increasing rapidly. search and information retrieval tools outside of the enterprise are growing ever more sophisticated, while internally, the same tools seem cumbersome and inefficient. one of the biggest drivers of dissatisfaction with enterprise content findability is the consistently excellent experience of findability that users experience outside of the enterprise. while significant cost associated with poor content findability has been well-established, there are also large costs incurred in implementing solutions to this content findability issue. one paper addressing this issue notes that, despite the “substantial” benefits associated with descriptive business metadata, such individual solutions can also have “an inherent saliency problem” such that “the cost associated with the time it takes to pre-categorize documents… is a major perceptual impediment to the implementation of a navigable document hierarchy” (schmyik et al., 2015). though it may be hard to prove a return on investment given the “soft costs” (athey, 2009) of knowledge worker time and reuse benefit, there is certainly a financial incentive over time for organizations that address findability issues at the root of the problem, rather than via technology applications later down the line (wu et al., 2009). the jet propulsion laboratory has likewise been considering the question of more effective knowledge management since the early 2000s. one document written during that time addresses some possible strategies for encouraging knowledge capture within the organization, such as “recognize and reward people for sharing knowledge,” “encourage and support communities of practice,” and “strike a balance between long-term corporate needs (capturing knowledge) with short-term local needs (completing a task quickly)” (holm, 2001). all of these strategies implicitly assume the support of the institution for knowledge management practices, and the significant benefits that result from strategically intervening in employee knowledge capture processes. a later revival of these efforts lead to the determination of similar best practices which currently inform jpl knowledge curation, namely: “historical technical records of jpl work should be available to all relevant employees who meet itar requirements,” “metadata for search purposes can be easily added,” and “refining access groups and authorizing access are straightforward processes” (devereuax et al, 2015). in a 2018 interview, jpl deputy chief knowledge officer charles white advocates that enterprises take up the mantle of “knowledge husbandry,” since he foresees that the “next frontier” of enterprise knowledge management will not just be getting people to share their information with repositories, but getting repositories to share information with other repositories. “we haven’t been good at networking information systems,” says white, “and moving forward, we need to talk about a system of systems” (white, 2018). the future of findability in the enterprise relies on the ability to break down silos and provide a unified view of enterprise content. the ‘knowledge curator’ role the pilot of a “knowledge curator” role emerged as the result of a collaborative partnership between two organizations at jpl: the mechanical systems engineering, fabrication, and test division (35x) and the jpl beacon library group (319x). the jpl library had long supported efforts by the 35x engineering division to capture and effectively store information, specifically using the institutional wiki. however, in order to scale up knowledge capture and organization efforts, a new, more formal role was required. since it was clear to both parties that a library and information science (lis) skill set would be ideal for implementing knowledge capture solutions, an mlis student intern was hired within the 35x engineering division to do embedded lis work for that division. this student intern was hired to work part-time for just over 6 months, from february 2018 to august 2018. the work performed by this student intern can be broken down into thematic projects as follows: guided document creation, i.e. the development of wiki portals and standard editing processes for consistent knowledge capture  search curation, i.e. manual and organic enterprise search relevancy improvements  index as intervention, i.e. metadata mapping and information modeling to improve access to content for both local and enterprise-wide applications these project tasks, as performed by the student intern, began to form the basis for a new role at jpl: the role of the “knowledge curator.” this new emergent role is characterized by an ability to perform tasks which are both manual in that they allow for rapid changes to increase content findability, and organic in that they allow for more long-term, systemic changes to that same end. additionally, this role encourages proactive curation activities using implicit user feedback in search logs and other metrics, as well as the ability to be reactive to evolving user needs, expectations, and explicit feedback. the role of the knowledge curator is to intervene strategically at each stage in the knowledge management process, from creation and storage to search and retrieval for reuse. in action, this translates to facilitation of content creation and knowledge capture, improving the user experience of information storage and retrieval, and designing architectures that both make users’ jobs easier and align with institutional goals. finally, this role uses traditional lis skills in an innovative new challenge: understanding the dual function of the enterprise knowledge worker as both content consumer, and content creator. to develop this understanding, the knowledge curator role seeks to improve organizational information literacy and to foster a shared culture of knowledge capture. strategic interventions in order to support better knowledge management at jpl, the knowledge curator makes small, calculated adjustments to existing employee workflows and/or existing findability tools. these adjustments are designed to be as non-invasive as possible while at the same time having a discernible effect on the findability and usability of enterprise content. the current success of the solutions employed by knowledge curators is in part a result of the high-impact, low-cost design. the following three interventions were initiated by the knowledge curator intern during her half-year tenure. 1. guided document creation figure 1. jpl wired logo and landing page. the primary project for the knowledge curator was guided document creation within the context of jpl’s internal wikipedia, jpl wired. enterprise wikis are valuable knowledge management tools in use within many organizations (seibert, 2010). jpl’s wired wiki is built on the collaboration platform confluence, and is open to everyone in the jpl community, with the notable exception of foreign nationals (jpl employees who are not us citizens, who make up less than 5% of the workforce.) jpl wired began in 2009 as a way to help newly-employed engineers find content that was critical to their work. over time, the wired “intrapedia” spread beyond its initial audience and user group, as the wiki filled a niche knowledge management need by allowing any jpl employee to directly contribute to jpl’s collective knowledge (rober and cooper, 2011). in 2016, after several years of undirected growth, the wired wiki got a boost when the wired steering committee was initiated. the committee began to design tools for wiki content creators and users, such as indicators of page validity (wiki confidence level indicators) and wired portals (structured areas to support collaborative content creation from subject matter experts). the wired steering committee then brought on the knowledge curation intern in early 2018 to further the adoption of wired portals among various subject matter expert groups (or “communities of practice”) at jpl. wired portals are modeled on wikipedia portals and wiki projects, which are the landing pages for communities of editors. each community of editors is focused on a specific topic or domain within wikipedia. wikipedia’s portals and projects help to create structure and consistency within the wiki site. portals and projects also help the communities of editors to assure that articles within their domain are systematically monitored for accuracy. in the same way, portals in jpl’s wired wiki were developed with the intention of establishing a functional information architecture and a community dedicated to the upkeep of content around a given area of expertise. this need to efficiently facilitate knowledge sharing within a community of practice arose because, due to the matrix structure of jpl as an institution, individual employees working on different projects previously did not have an easy mechanism for sharing knowledge, even when the nature of their work is similar. wired portals, created around jpl communities of practice, provide a centralized location for knowledge capture and sharing of lessons learned that circumvents the matrix structure and crosses project lines. figure 2. the landing page of the wired wiki portal for the extraterrestrial sampling community of practice at jpl. the role of the knowledge curator in establishing wired portals was to assist these communities of practice by helping to identify categories of articles within that community’s scope. these “categories of articles” have typically followed two veins: 1) article type, and 2) article topic. that is, when allowed the opportunity to self-organize, subject matter experts tend to tangibly organize their tacit knowledge first by document type (i.e., “training resource,” “reference material,” or “sample product”) and then by subject (i.e., “entry stage,” “parachute descent,” “landing stage”). by sorting existing wired content into categories such as these, communities are given an opportunity to identify gaps in content and suggest standard information required for new articles within a given category. after two or three meetings with a few key stakeholders within a community of practice, a larger group comes together for an “edit-a-thon” to begin labelling, editing, and creating new content. wired edit-a-thon participants will typically add labels to content, and paste into each article “navigation boxes” and other standard information that provide links to related content and to the portal homepage. this inclusion of set labels and standardized information makes existing content more findable by providing multiple points of entry to articles. in addition to providing technical support and information architecture set up for the wired wiki portals, knowledge curators facilitate strategic thinking within the communities of practice by asking members to consider how future users of their articles might best find their content. questions like “if you were new to jpl, what would you want to know about this topic?” and “what would you as a new hire expect to find in this type of article?” lead to additional structure recommendations for individual articles and the wiki portal as a whole, making the community’s content more findable and usable over the long term. this additional structure is not only valuable to the communities of practice experts as they fulfill their roles as content creators, but also helps to align their content with institutional priorities for content usability. the structure added to wired portals and portal articles helps to inform enterprise-wide taxonomy and facilitates better integration with jpl’s internal intranet search. 2. search curation “successful enterprise search,” according to information architect marianne sweeny, “is not achieved through spontaneous assembly of features and programming” but rather through “a clear understanding of search behavior inside the fire wall so that the right search features will be mapped to specific needs and behaviors” (sweeny, 2012). the aspect of “mapping” search to meet enterprise user needs is one of the functions of the knowledge curator role. the jpl library, in conjunction with the knowledge curator intern, engages in targeted enterprise search improvement called “search curation.” jpl’s intranet has an internal unified search, known as jsearch, which is built on the open-source search engine elasticsearch (as of this writing, elasticsearch version 2.3.5 is in use). up until 2014, jpl used a google search appliance to run its unified search. however, the black box nature of the google appliance, combined with higher expenses that made scaling the search impractical, drove the lab’s office of the chief information officer (ocio) and the jpl search team to evaluate alternatives. elasticsearch was selected and has been in use since that time. jsearch currently indexes hundreds of internal intranet pages, and documents from cloud-based document stores. though this is only a fraction of the repositories currently in use on lab, the scope of the search index is constantly expanding. users of jsearch consistently rate the elasticsearch systems as better than the google appliance; however, the majority of searchers still experience poor results relevancy, especially for uncommon search queries. a lack of linkages between internal web pages, minimal metadata attached to sites, and varying levels of access for different employees all contribute to the challenges of effective organic enterprise search. while more automated solutions are developed behind the scenes, a manual fix is in place. since 2014, the jpl library and the ocio have collaborated on the search curation project, using a software tool built in-house to perform targeted, manual adjustments to jsearch. the purator tool is an internally-developed tool which can be used to create search cards, map results to keyword queries, boost or downgrade organic search results, and index sites. the tool was developed by the jpl search team. information managers use the purator tool to make edits directly to the search index after edits are either 1) requested by knowledge workers who report that they cannot find what they are looking for, or 2) sourced from search logs and metrics. the purator tool’s code repository is located in jpl’s github enterprise instance. while the code is considered proprietary and not cleared for inclusion in this paper, interested parties should contact the authors (library@jpl.caltech.edu) for more information and specifics on the purator tool code. for search logs and metrics, matomo (previously piwik) is used. matomo is an open-source web analytics application used to determine what jpl knowledge workers are searching for so that the appropriate results can be mapped to user needs. while there are a variety of functions enabled by the purator tool, the most commonly used for search curation are keymatch, boost, seed list, manual crawl, clean-up, deadlinks, blacklist, and knowledge cards. keymatch the most often utilized tool of the purator suite is keymatch. this function allows curators to boost certain links whenever a series of associated keywords are queried in the search system. for example, the terms “calendar” or “fiscal year” may both retrieve as the top result a link to information about the fiscal year end if that information has been keymatched to those terms. this function is designed to be similar to what in search is sometimes called ‘best bets,’ or to sponsored search results in google, which are featured at the top of search results and more prominently for certain keywords. a single keymatch record may have many different query terms associated with it, but each record may only point to a single resource. typically, curators use the keymatch tool to boost certain known-items to the top of search results pages. these known items are frequently guidance, how-to, or reference documents and web pages, while boosted links to applications are typically developed using the knowledge card tool. figure 3. the keymatch landing page, showing keymatch records. knowledge card the knowledge card tool is the most unique in the purator’s suite of tools. this tool was added in after all the others, in an update about two years after the purator was first being used. it retains its original name, it services, as it was originally intended to be a card catalog of it tools only. however, the usefulness of this tool impressed curators and patrons alike, and the it services tool quickly turned into a general knowledge card tool, used to describe any entities at jpl, not just it tools and services. future updates to the purator tool will reflect this more general usage of the knowledge card function. knowledge cards are created manually by curators, and automatically appear at the top of search results pages for given queries. as alluded to, knowledge cards tend to be created for jpl “entities,” such as specific online resources, cafeterias and other service centers, missions and projects, or other named items on lab. this focus on creating cards for entities comes from an attempt to mimic google search results, which often include knowledge cards for entity searches. in addition to fields where the curator may specify the title, url, and associated keywords for a knowledge card, curators also include logos for each of the cards, either sourcing one (via screenshot/download) from the resource being described or generating a logo as needed. figure 4. the knowledge card record for jpl’s ebis tool, shown in edit mode. boost similar to the keymatch and knowledge card functions, the boost function allows for the improvement of search results on a query by query basis. however, unlike these other tools, boosting can be considered a more organic form of search curation. both keymatch and knowledge card functions involved creating a distinct record, which is then inserted into the search results. boosting, however, allows curators to directly increase or decrease the weight of a certain result in the search index. weights are calculated for each search result retrieved by the system for a given query; the boost function allows curators to tip the scales and artificially reorder the index weighting for certain query term searches. figure 5. the boost function for query ‘ebis,’ showing a boosted result (‘about ebis’) that has been heavily weighted in the search index. manual crawl and seed list jsearch’s search crawlers index contents from jpl’s intranet according to a seed list of sites to crawl. the purator tool allows curators to edit the seed list by adding or removing urls which should or should not be indexed by search. this function is especially helpful when sites are commissioned or decommissioned. typically, this function is used when a jpl employee creates a new internal web site, and want to ensure it is made available via the enterprise search system. similar to the seed list is the manual crawl function. this tool allows curators to push a one-time, “manual” crawl of a site for instantaneous inclusion in the search index. because jsearch reindexing occurs at variable frequencies depending on the resource (certain resources may be recrawled daily, while others are recrawled weekly or monthly), this tool is an effective way to provide an immediately improved user experience of search by instantly including sites in the search index. after a site is crawled manually, it is then added to the seed list so it will be recrawled again at the scheduled time. figure 6. manual crawl dashboard showing several successful crawl jobs. clean-up, deadlinks, and blacklist the clean-up, deadlinks, and blacklist trifecta of tools is used by the knowledge curator to remove bad results from the search index. the blacklist function is effectively the opposite of the seed list; urls added to the blacklist are excluded from the regular recrawling and reindexing of jpl intranet content. while the lab recommends the use of a robots.txt file on sites that do not wish to be crawled by the index, this practice is not completely adapted, and so the curator takes on the task of manually excluding sites which should not be indexed. the deadlinks and clean-up functions are similar, but use differing methods to identify and remove bad links from the search index. the deadlinks tool attempts to access sites, and then lists those which return a 404 code so that these ‘dead links’ can be reviewed and either ignored or deleted by curators. this human mediation of the dead links is intended to ensure that all deleted links are truly degraded, not just temporarily inaccessible. finally, the clean-up function allows curators to remove links at the individual url level. the tool searches the index for a single url specified by the curator. if the url links to a single page, then only that single result is made available for deletion. if the url is more general, pointing to a site rather than to a single page, then all of the child pages under that url are available for deletion. this allows for the instant removal of jsearch results which are not necessarily ‘dead links’ but which may, for various reasons, need to be removed from the search index. accidental indexing of sensitive materials is one of those reasons. figure 7. the blacklist tool. over the course of the knowledge curator pilot, the student intern completed a “top 100 search terms” project which involved manually improving results for the top 100 search queries according to search log data. most of these top search terms could be considered “known-item” queries, since workers typically seemed to be searching for a tool or service that they know exists. often, these searches are related to lab technology (software, repositories, and other digital objects) or human resources and similar institutional policy. to improve results for these searches, the intern went through a multi-step process for each search concept. first, a knowledge card was created with a title, description, logo, and direct link to the resource. then, one or two more results were keymatched; selected results were mapped to the keywords associated with a search concept. typically, these keymatched results were mapped to pages about the known-item resource, providing contextual or how-to information “about” a given known-item. this method is meant to imitate the typical google search results where the primary result is a card (which google calls a “knowledge panel”) and the secondary result is a wikipedia entry. an understanding of page metadata and knowledge worker search habits was a key asset for completing this task. figure 8. image on the left is of organic search results retrieved by jsearch prior to any curation. image on the right shows the same search results after curation, where the top two results are 1) a knowledge card, and 2) a link to a page “about” the known item. just as creating portal categories in wired identifies gaps in wired content, looking to identify “about” pages for known-items within jpl’s intranet frequently reveals that no such page exists, or that information is scattered among many sites, some of which may be outdated. in this case, it is sometimes the search curator’s turn to become a content creator, by creating a wired page to relate contextual and how-to information about a given resource. having already gone through the work of locating relevant resources, search curators are well-poised to summarize and point to this information from a single resource. furthermore, wired integrates smoothly with the organic search, reducing the need for future manual keymatches. while most search curation tasks are manual, an understanding of the information ecosystem also provides mechanisms for interventions that will contribute to organic improvements. for example, adding more structured data to wired pages will further enhance integration with jpl’s current and future search systems, improving both short-term and long-term findability. 3. index as intervention another responsibility explored under the knowledge curator role pilot, and the final part of the knowledge curator internship, is content management. while wired serves as one platform for knowledge capture at jpl, it does not stand alone. there is a wide range of documentation produced by knowledge workers in the process of building flight hardware, which includes everything from test data to software code to drawings to review presentations. these information products are stored in a variety of in-house repositories, document stores, desktops, and shared drives. much of this documentation is formally captured at the project level in an internal document library called docushare, a document store product from the xerox corporation. however, this means that these documents are typically only available to individuals currently working on a particular project. an individual may even lose access to documentation he or she produced when transferred to a new project. as a result, it is not always easy for a team designing flight hardware for the mars 2020 rover to view design documentation or test data for the same piece of hardware for the curiosity rover, even though it is likely that reviewing such documentation would provide valuable insights. this access rights issue impinges on jpl’s ability to effectively re-use enterprise content, even when it can be discovered via search and other means. access restriction is not the only barrier to effective re-use. documentation in the earliest stages of project development, as well as in most stages of development for smaller projects, is less regulated by the institution. interviews with knowledge workers reveal that teams and individuals may devise shared drive file structures, use other institutional tools such as microsoft sharepoint, or simply store files on individual desktops and share by email. this ad hoc file management only serves to create further data silos in the long term, and limits the ability of search applications to utilize this content in a unified manner. furthermore, this content is frequently lacking metadata and any other document-level contextualization. for example, though all mechanical parts at jpl are given a part identification number, many documents about that part do not include the part’s number, but rather include the descriptive name of the part. while knowledge workers may find a part name more user-friendly than an arbitrary number in the short term, but the name will very likely may have many variations, which can cause difficulty in searching across multiple platforms to find documentation related to a single part. this area represents the final strategic intervention in which knowledge curators may engage to increase enterprise content findability. information modeling serves as a starting point for identifying inconsistencies in metadata and storage practices. facilitating the addition of a minimal standard metadata to documents as they are produced, as well as recommending where to store different types of files, for example, would greatly improve the eventual discovery and retrieval of these documents. though the work done in this area over the course of the internship pilot was preliminary, knowledge curators at jpl have successfully created several proof-of-concept indexes which map siloed data to a single schema for easier, more unified content search and retrieval. this targeted content indexing, combined with the development and mapping of minimal standard metadata, is one way in which knowledge curators can help to break down silos as they occur in the enterprise. user as creator, librarian as curator information management at jpl has always put employees at the center of knowledge capture. as a 1999 jpl knowledge management study team document states, while technology is an important ingredient to effective knowledge management, humans are the essential ingredient, both for knowledge production and for knowledge use. any effective knowledge management process or system must add value that, in the minds of its users, exceeds the cost of its use. (doane et al, 1999) viewed from the perspective of the user, knowledge management tasks must provide some tangible benefit in order to be adapted into workflows. as discussed above, there is a distinct role in “knowledge curation” that ought to be supported institutionally. however, that role does not stop at a single dedicated employee or even with a dedicated team – rather, every jpl employee that creates or uses digital content has some role to play related to institutional knowledge management. when asked about this role that all jpl employees play in regards to content and knowledge management, jpl deputy chief knowledge officer charles white noted that “everyone is a knowledge worker, they just don’t know it.” while knowledge management is valuable in any enterprise, the unique and pioneering nature of jpl’s work makes information capture and knowledge transfer essential to jpl’s continued success. as alluded to in previous sections, jpl employees frequently find themselves in the position of both information creator and information seeker, sometimes with regard to the same information object. however, without an adequate culture of knowledge capture, the content creation aspect is neglected and further complicates findability issues later down the line. indeed, in order to perform knowledge management effectively, the institution ought to provide “practical guidelines to support all editors that create content to be published and found on the intranet” (findwise, 2016). the siloization of content in a large enterprise is fairly inevitable, especially when an enterprise supports many different functions.the skillset of lis professionals is uniquely poised to help break down those siloes by cultivating a culture of knowledge sharing and conveying information literacy principles to enterprise workers. significant benefits are incurred when a workforce is instructed in best practices for interacting with an organization’s digital information ecosystem (lucic and blake, 2016). with an understanding of information-seeking behavior, metadata best practices, information architecture, and information literacy, lis professionals can identify strategic interventions at every stage of the information lifecycle, and especially at the moment of content creation, that facilitate knowledge capture. conclusion as organizations and their workers become increasingly reliant on shared digital resources to complete work tasks, librarians and other information professionals must decide what our roles will be in facilitating information capture and findability in an enterprise information ecosystem. while technology solutions must be part of any findability improvement plan, there is significant value that can be added at the content level, with interventions into existing workflows. these interventions ought to be strategic; that is, they must strive to align knowledge worker practices with organizational goals while minimizing impact on daily workflows. the strategy behind these interventions is part of what helps to ensure that any attempts to improve content findability will work not only in the short term, but in the long term as well. jpl information managers have found that intervening strategically in the areas of document creation, enterprise search relevancy ranking, and document indexing are effective ways to improve enterprise content findability while simultaneously increasing knowledge workers’ understanding of their dual role as both content consumers and content producers. indeed, part of the long-term solution for content findability issues comes from the knowledge workers’ improved understanding of their own role in the content lifecycle. this ownership of content creation, description, and storage is an important paradigm shift for knowledge workers whose daily work includes producing increasingly more valuable content for their organization’s digital ecosystem. librarians and other information specialists have a growing role in knowledge management, especially in the special library setting. this role is partially one of education; that is, information specialists within the enterprise have an expanding mandate to help create a culture of knowledge capture, and to demonstrate to knowledge workers that standardization and other principles of information management have a direct impact on the later findability and reusability of enterprise content. as the lis field broadens and evolves, information specialists should continue to evaluate and expand upon the emergent role we have in improving institutional digital asset management, content findability, and knowledge management for the organization. bibliography athey, jake. getting to a digital asset management roi. widen. 2009. bryan, l. making a market in knowledge. mckinsey quarterly. 2004. https://www.mckinsey.com/business-functions/strategy-and-corporate-finance/our-insights/making-a-market-in-knowledge devereaux, a., goldstein, b., hanna, r., martinez, e., matthews, v., powers, r., seixas, b. tompson, s. strategic outbrief: functional enterprise documentation working group. 2014. 16 p. doane, j., hess, s., cooper, l., holm, j., fuhrman, d., u’ren, j. a knowledge management architecture for jpl. d-16577. 1999. 210 p. feldman, s, sherman, c. the high cost of not finding information: an idc white paper. international data corporation. 2001. 10 p. findwise. enterprise search and findability survey 2016. 2016. seibert, m. 111 reasons why you need an enterprise wiki. 2010. https://www.atlassian.com/blog/archives/111_reasons_why_enterprise_wiki holm, jeanne. architecting an approach to knowledge management. 2001. 25 p. leher, mark. the value of descriptive metadata to improve enterprise search. a wand, inc. white paper. 2013. lucic, a., blake, c. preparing a workforce to effectively reuse data. in proceedings of the 79th asis&t annual meeting: creating knowledge, enhancing lives through information & technology (asist ’16). american society for information science, silver springs, md, usa, article 75. 2016. 10 p. lustigman, a. your intranet needs you: how information professionals can add value to intranets and portals. legal information management. 2015 mar 19;15(1):57-60. http://dx.doi.org/10.1017/s1472669615000171 miles, d. aiim industry watch search and discovery – exploiting knowledge, minimizing risk. silver spring (md): aiim; 2014. 34 p. schymik, gregory, karen corral, david schuff, and robert st. louis. “the benefits and costs of using metadata to improve enterprise document search.” decision sciences 46, no. 6: 1049-75. 2015. sweeny, marianne. delivering successful search within the enterprise. informer. 2012. white, c. personal interview. 2018. wikipedia:portal guidelines. retrieved 09-21-2018. https://en.wikipedia.org/wiki/wikipedia:portal_guidelines wikipedia:wikiproject council/guide. retrieved 09-21-2018. https://en.wikipedia.org/wiki/wikipedia:wikiproject_council/guide#what_is_a_wikiproject? wu, mingfang, james a. thom, andrew turpin, and ross wilkinson. “cost and benefit analysis of mediated enterprise search.” in proceedings of the 9th acm/ieee-cs joint conference on digital libraries, 267-76. austin, tx, usa: acm. 2009. acknowledgements this work was carried out at the jet propulsion laboratory, california institute of technology, under a contract with the national aeronautics and space administration. an additional special thanks to deputy chief knowledge officer charles white, ocio software engineer jeff liu, and the jpl search team for their support in this work. about the authors rebecca townsend (rebecca.m.townsend@jpl.caltech.edu) is an information management engineer who began her career at jpl as the “knowledge curator” intern. she received her mlis from ucla in 2018, and has a background in romance languages and anthropology. her primary research interests include wiki maintenance and information literacy. camille mathieu (camille.e.mathieu@jpl.caltech.edu) is in information science specialist with the jpl beacon library. she received her mlis from ucla in 2015, and has a background in english literature. her primary research interests include enterprise search improvement and metadata for content management. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – mdmap: a tool for metadata collection and matching mission editorial committee process and structure code4lib issue 26, 2014-10-21 mdmap: a tool for metadata collection and matching this paper describes a front-end for the semi-automatic collection, matching, and generation of bibliographic metadata obtained from different sources for use within a digitization architecture. the library of a billion words project is building an infrastructure for digitizing text that requires high-quality bibliographic metadata, but currently only sparse metadata from digitized editions is available. the project’s approach is to collect metadata for each digitized item from as many sources as possible. an expert user can then use an intuitive front-end tool to choose matching metadata. the collected metadata are centrally displayed in an interactive grid view. the user can choose which metadata they want to assign to a certain edition, and export these data as marcxml. this paper presents a new approach to bibliographic work and metadata correction. we try to achieve a high quality of the metadata by generating a large amount of metadata to choose from, as well as by giving librarians an intuitive tool to manage their data. by rico simke background the more libraries become interconnected the more important the question of data integrity becomes. correcting metadata and improving their quality may facilitate identification of bibliographic records and will increase data integrity. applications that handle large amounts of data, e.g., bibliographic metadata, are in need of automation. however, automation can be difficult to achieve and may even make results more complicated, because automatic collection and aggregation of metadata can produce a data set more faulty than the original one that then needs correction by an expert user, e.g., a librarian. another possibility, however, is to leave the responsibility of creating metadata for a record to the librarians but to let them be guided by the existing metadata. bibliographic work normally is a highly manual task: a librarian holds a book in their hand and either creates a new dataset by inserting metadata information into a database or tries to match it to pre-existing data in some catalog. newer approaches try to incorporate the frbr [1] schema into the organization of bibliographic datasets. however, oclc research shows that automatic organization of metadata with respect to the frbr schema is not possible and that automatic matching – especially for records without author and/or uniform title – is a difficult task [2]. the need for a tool that can overcome these difficulties and assist its users arose within a concrete scenario of research work. the esf-funded project library of a billion words [3] is a digitization infrastructure project for primarily greek and latin editions from the 19th and 20th century. in order to store the documents and make them usable for researchers in the humanities, the project will provide a repository with high-quality scans. ultimately, the project aims to create a digital citation infrastructure. the library of a billion words project depends on high-quality metadata in order to be able to build a persistent, searchable catalog and infrastructure for digitized books. the project’s primary source of scanned documents is the internet archive [4] (ia), which provides basic metadata information for every document. that metadata is, however, mostly insufficient for building such an infrastructure. a typical metadata file from the ia catalogs author and title, along with technical information about the scanning process. to improve the data it is possible to search catalogs all over the web for the concerned record. however, the problem of keeping track of and choosing the correct metadata for a digitized object is a recurring problem in the project. there is no place where users of the infrastructure can collect metadata from different sources and view them in direct comparison with each other. an application is needed where the potentially large amount of metadata can be joined and displayed in a comprehensive and user-friendly way. furthermore, users should be able to create a new record by combining the displayed metadata. the intention of the project is to combine metadata from sources like the perseus catalog [5] and the catalog of the leipzig university library (lul). it appears that it is not possible to match records from these sources automatically, using identifiers from the internet archive record files. similar records can have different identifiers, such as oclc numbers, when they belong to different items or manifestations. this is why in the majority of cases a record in the lul catalog can not be identified by the oclc number provided in the ia metadata. hence, further oclc numbers are generated that are likely to be related to the record in question. additionally, the fact that only about 40% of the archive.org metadata used in the project contain oclc identifiers from which additional metadata for a record can be derived makes it impossible for these records to have any kind of automatic metadata retrieval. thus, the key components of our solution are a web-based, user-friendly front-end with import and export options and a sleek interface for displaying and combining large amounts of metadata for a single record in a table view. besides scans from the ia, the project intends to include editions that are already available in and indexed by the leipzig university library catalog in its repository. in the event that here too the metadata is insufficient, the tool can be used to improve the data with the help of other sources such as the worldcat [6] or the perseus catalog. approach starting from the question as to how metadata from different sources can be combined in such a way that its high quality is ensured, our basic approach is to make a front-end available that has an intuitive user interface and visual guidance combined with a central database for all metadata connected to a certain edition. the editions are derived from, e.g., a scanned document or book. the idea behind this tool is to enable the user to put an adjusted metadata record together in a convenient way. our approach is new in the way it uses metadata as a starting point for a cataloguing task. the process can be started with only one identifier, such as an oclc number or title-author combination. the tool can be used to collect as much metadata as possible from identifiers or from other search activities. this makes sense in light of the fact that, in a digital library, one often does not have the physical item at hand but only has a combination of metadata available. the tool is designed to enable the user to search through and store metadata from different sources. the stored metadata can be analyzed and compared with other metadata, and a new record can be built by combining entries and can subsequently be exported. marcxml and metadata when collecting and comparing metadata, all of it has to be available in the same format. among all available bibliographic formats, marc21 proves to be best for collecting and creating metadata. it is the most-used standard worldwide and can be generated from dublin core [7]. to simplify parsing and handling the data, the format of choice is marcxml[8], which is marc21 in xml markup. most sources natively export metadata as marcxml and there is no loss of information when transforming metadata from e.g., mods to marcxml. the hierarchical structure of marcxml is suited for displaying in a table, as will be shown when we discuss the tool’s grid view. the following is a short overview of the metadata types used as sources for the project. the tool itself is organized in a modular fashion, which means that it takes few programming skills to include additional sources. the user only needs a web service that provides metadata when given an identifier – e.g., marcxml for a given library identifier. archive.org the majority of scanned documents are obtained from the internet archive. these documents come with very sparse metadata and need to be improved before they are of use to the project. a study of metadata files from the internet archive displays the following distribution of identifiers. total number of files: 2,194,251 oclc-id: 800,870 / 2,194,251 : 36% identifier-bib: 382,132 / 2,194,251 : 17% google-id: 162,809 / 2,194,251 : 7% out of a total of about two million metadata files, only 36% have oclc identifiers. in total, about 55% of the metadata files contain at least one of the three identifiers. oclc control numbers oclc control numbers are worldwide record identification numbers, hosted centrally in the worldcat catalog, which is administrated by oclc. worldcat is the world’s largest online catalog and therefore is a good source of metadata. if an oclc number is available, one can easily get the metadata provided as marcxml for that particular record via worldcat. xoclc service xoclc is a web service that delivers a list of similar oclc numbers, when given an oclc number as input. the xoclc service is supposed to deliver records that are related to the item identified by the given oclc number. this goal was formulated by thom hickey in 2005: xoclc: take in an oclc number and return the oclc numbers in its frbr work-set [9] currently, the service does not provide any information on the type of relation between two oclc numbers in an xoclc request. the tool uses the geteditions [10] method to obtain a list of related items, and stores them as a list of numbers affiliated with an edition. perseus catalog some of the research in the library of a billion words project is in collaboration with the perseus catalog project at tufts university. the perseus catalog is part of the perseus digital library and aims to “create a catalogue that would provide coverage of greek, latin, and ultimately other literatures in a way that was suitable to a digital age.” [11] this makes it an ideal source for enriching metadata in the library of a billion words project. the metadata in the perseus catalog is organized to try to reproduce the frbr schema for its records. this means that metadata is enriched with information about related items, such as host editions of works. this information is preserved in the metadata tool, since at a later stage it will be important for the project’s catalog infrastructure. since metadata from the perseus catalog, provided in the mods format, can be transformed into marcxml, it can be imported to this tool. finc – leipzig university library catalog since the library of a billion words project is based at leipzig university, the metadata tool integrates an interface to the leipzig university library catalog, finc [12]. in the tool, all oclc control numbers that have been retrieved via the xoclc service can be automatically queried in the finc catalog. if an oclc number is found in the finc catalog, the associated marcxml record is stored in the database and connected to the corresponding edition. the application the application is written in ruby, using the ruby on rails framework with a mysql database for metadata storage. in the database every edition is connected with a number of metadata entities. the user can take different approaches to importing metadata. the first is a simple upload of a marcxml record to the metadata tool from the user’s machine. the second is to provide an internet archive identifier, for which the corresponding metadata is retrieved automatically by the tool. the third is to provide an oclc number for which a worldcat record and its metadata can be obtained. the following subsection outlines a typical workflow when adding a new document. workflow the normal workflow is as follows: the user starts by creating a new empty edition in the metadata tool. the new edition is identified by a name or any other kind of identifier. as a start, that edition exists only in the tool and is not yet assigned to any edition outside of it, e.g., in the library of a billion words infrastructure. the user can then choose the metadata sources that they want to be included in the new edition. if a record is not in the marcxml format originally, it will be transformed, e.g., records from the perseus catalog, which are supplied in an embedded mods-format, will automatically be transformed to marcxml. if the user chooses to add an oclc number, the metadata for that number will be fetched and the user can then submit an xoclc request to add further metadata to the edition. all marc records are stored in the database, and different types of associations define which metadata records are assigned to an edition. once the user has finished collecting metadata, they can enter the grid view to compare and choose an edition’s metadata. the grid view in the grid view all source records assigned to a certain edition are displayed in a table, together with their metadata. the leftmost column contains the new record that is built. that column remains empty until the user selects metadata from other source records or types in adjusted metadata. the second-left column labels the marc21 field-subfield combination. the third column provides statistical information on the metadata contained in each row (see screenshots 1 and 2). the remaining columns represent the different source records, with the metadata to be compared – one column for every source record. the rows in the table represent the marcxml fields. each row is again divided into subfields. screenshot 1: grid view each field-subfield combination is displayed at least once. if two or more sources share a field-subfield combination, the data is displayed in the same row. metadata that share the same field but differ in their subfields are displayed in different rows, e.g., two sources sharing the field 245 1 0 $a share the same row for that combination of fields, whereas any source with the fields 245 1 0 $a $b would have its metadata displayed in a different row. screenshot 2: grid view with selected values metadata fields within sources that do not share a certain field-subfield combination are left empty. for every field, the user can select values from a certain metadata record or enter their own values, e.g., if the values are incorrect or if the required values are not available (screenshot 2). the tool guides the user in finding the correct metadata by displaying statistical information about all collected metadata. this is achieved with two different types of statistical information: first, for every marc field the five most prominent values and their quantity, i.e., number of appearances of the same value, are displayed. the idea is that more frequently occurring content is more likely to be correct for the corresponding field. the user can hover over the bars to view their values as a tooltip, and to highlight source records that contain these values (screenshot 3). screenshot 3: grid view with statistics and activated hover feature second, all fields are compared with regard to their relative frequency in all source records (screenshot 4). the underlying idea is that a more frequently occurring field is more important to a cataloguer than a less frequently occurring one. this helps the cataloguer to instantly see the distribution of marc fields in all source records. the frequency of the metadata is displayed as a bar chart: screenshot 4: total share in all data sets for every marc21 field other metadata standards this tool was primarily designed for working with marcxml and its particular way of structuring metadata. in its current version, the tool cannot yet handle metadata standards with a different structure. however, as shown above, it is straightforward to, e.g., transform marcxml to mods or vice versa. thus, it is possible to integrate more classes of bibliographic metadata, as long as they can be transformed to marcxml. future work a question that arises with regard to future work is the handling of identifiers. each source delivers a different type of identifier. records that derive from worldcat are typically identified by an oclc number in field 001. marcxml records extracted from the internet archive have a different type of identifier (e.g., operarecensuitri10libauoft) in that field. since each operation in the tool can lead to a new dataset, it would be inconsistent to simply apply one of these identifiers to the newly created record. therefore, we need a solution as to how to create new identifiers for new records. however, taking care of these difficulties and creating rules for processing identifiers has to mainly be the responsibility of the end-users, within their concrete use cases. to make mdmap more convenient, better integration of catalog search functionality into the tool is needed – e.g., to provide search directly in the finc catalog or worldcat, from within the application. additionally, performance of the grid view needs to be improved. scalability of the tool presents another issue. in the library of a billion words project, the tool is intended to store and connect millions of metadata records. however, we have not yet tested the tool under such realistic circumstances. given the fact that in a typical use case the user needs to find a certain record in a collection of millions of editions, a logical first step is to integrate more effective search technologies, such as solr, into the tool. in our project, the library of a billion words, we intend to make the tool available to librarians with practical experience working with marc records, to help them clean up metadata for editions incorporated into the library of a billion words text repository that are believed to have insufficient metadata available. at this point, the tool is still in an experimental stage of development and has not yet been tested by expert users. it has, however, been presented to librarians at the leipzig university library, whose feedback has been taken into consideration during further development of the metadata tool. conclusion we proposed a new approach to bibliographic work, and presented a tool that can handle metadata from a large amount of sources. anyone interested in mdmap is strongly encouraged to download and test the tool [13]. acknowledgements parts of the work presented in this paper are the result of the project die bibliothek der milliarden wörter. this project is funded by the european social fund. die bibliothek der milliarden wörter is a cooperation between the leipzig university library, the natural language processing group at the institute of computer science at leipzig university, and the image and signal processing group at the institute of computer science at leipzig university. notes [1] http://www.ifla.org/publications/functional-requirements-for-bibliographic-records [2] http://www.oclc.org/research/activities/frbralgorithm.html [3] http://asv.informatik.uni-leipzig.de/en/projects/30 [4] https://archive.org/ [5] http://catalog.perseus.org/ [6] http://www.worldcat.org/ [7] http://www.loc.gov/marc/dccross.html [8] http://www.loc.gov/standards/marcxml/marcxml-design.html [9] http://outgoing.typepad.com/outgoing/2005/05/xisbn_extension.html [10] http://xisbn.worldcat.org/xisbnadmin/xoclcnum/api.htm#geteditions [11] http://sites.tufts.edu/perseuscatalog/documentation/history-and-purpose/purpose/ [12] https://katalog.ub.uni-leipzig.de/ [13] https://github.com/ubleipzig/mdmap references hickey, thomas b., edward t. o’neill, & jenny toves. (2002). experiments with the ifla functional requirements for bibliographic records (frbr). dlib magazine 8, 9 (september). zeng, m. l., & qin, j. 1. (2008). metadata. london: facet. about the author rico simke (simke@ub.uni-leipzig.de) is a research associate in the project library of a billion words at leipzig university library and a phd student in the natural language processing group at the department of computer science (leipzig university). subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – electronic resources security: a look at unauthorized users mission editorial committee process and structure code4lib issue 12, 2010-12-21 electronic resources security: a look at unauthorized users much of the literature written on electronic resources security focuses on systematic downloading.  however, when the unauthorized use from two cases of stolen identities at the university of saskatchewan was studied in more depth, a different pattern emerged.  by analyzing proxy server data, we found that the unauthorized use was coming from all over the world, was focused on science, technology and medical resources, and included both small-scale and excessive downloading.  this article outlines some steps that libraries can take to detect and prevent small-scale unauthorized use and implications as libraries move towards shibboleth authentication. by heather tones white introduction quietly and regularly, international electronic journal thieves are downloading your electronic resources with stolen logins and passwords.  sound far-fetched?  not really.  in late 2008, the university of saskatchewan was refused access to a major publisher because a user tried to download more than 150 articles in a session.   when the incident was investigated, the library did not find an overzealous patron in need of a stern warning.  instead, the library found a stolen login id and password that had been used undetected 15 other times that day from various international locations.   while incidents like this one are not uncommon, the topic of electronic journal theft is not often considered by librarians and administrators.  additionally, the topic is not often delved into more deeply to determine the breadth of electronic journal misuse.  so, how are electronic resources being misused?  how do you stop it? types of e-resource misuse the license agreements for electronic resources typically limit access to authorized users, as defined in the agreement, and also often explicitly prohibit downloading in a systematic way.  when either of these conditions is not met, the library is in violation of its legal contracts and can face serious consequences including partial or full revocation of access.  baker and tenopir identify three examples of misuse1: excessive downloading with a stolen login and password; excessive downloading from an authorized user; and downloading by an unauthorized person from an open proxy server that did not require authorization. this case at the university of saskatchewan appeared, at first, to be similar to baker and tenopir’s second example. the first sign of a problem was the e-resource vendor revoking access to the library’s proxy server (ezproxy), which allows remote access to resources. instead of access to journal articles, library users saw a webpage stating that access had been denied because of excessive downloading. examination of the ezproxy files identified the library user that had triggered this situation and the user was contacted immediately. however, when contacted, the user stated that they had no part in this incident. at this point, we dug a little further into the log and audit files and found that the user’s id was being used from 16 different locations around the world that day. the client’s id and password were stolen. at this point, the example looked like baker and tenopir’s first example, excessive downloading with a stolen login and password but while taking a closer look at the log and audit files, another pattern of misuse emerged.  the other 15 times that day the stolen password was used was for limited downloading.  these other incidents went undetected because the usage was not excessive, and the login appeared to be authentic.  this type of misuse may be more widespread than libraries realize, because it appears to be legitimate use.   in fact, after a detection script was implemented to prevent this type of misuse, a second stolen login was immediately identified. the second stolen id had been used undetected for some time because it was not being used for excessive downloading. this type of misuse fits into a fourth pattern: limited downloading with a stolen login and password. how the logins were stolen in the first place is not known. the library user may have simply shared their password with a friend overseas. once contacted, the owners of the library accounts were not able to, or perhaps not willing to, shed any light on how it might have happened. there are several methods by which passwords can be stolen, but with it security in place at most organizations, one of the simplest ways is to take advantage of human gullibility through phishing, “the impersonation of reputable companies in order to induce individuals to reveal personal information, such as passwords and credit card numbers, online”.2 for example, simply by posing as an employee of the university it department in an email, the original attacker could have convinced the user to share personal information.  the ids are then sold or shared on the internet through online forums or web pages.  it is unknown if phishing was employed in these cases of stolen identity, but it does illustrate the point that no system is bulletproof, at least not yet.  stolen identities are not going away anytime soon. in both cases, the library clients immediately changed their passwords when notified, which stopped the unauthorized access and the vendor also quickly reinstated our access.   it was a fairly standard resolution.  however, curiosity remained about the identity of the unauthorized users and what they were doing.  the ezproxy log and audit files from these two stolen identities provide an interesting case study. who are our unauthorized users? once it was determined that there were many unauthorized users, questions arose about who they were and what they were doing.  one could romantically envision these people as the robin hoods of the e-journal world, stealing from the information rich to give to the information poor.   or as thieves driven only by profit, repackaging and reselling resources.  perhaps they are simply genuine scholars and medical professionals looking for the access that they need.  are they open access vigilantes, taking access into their own hands?  while we cannot tell their motives from the log files, this case study does shed some light on the question of identity and download behavior in the areas of location, subject matter and quantity. first, i created a database and imported the ezproxy audit files and log files into separate tables.   i was able to join the audit data, which contains user ids, and the log data, which contains download information, by joining the two tables on the session field.   i then limited the data to only the compromised ids.  from this data i could identify ip addresses, download urls and download sizes.  another table was created with location information for each ip gathered from http://ip-lookup.net/ . for privacy and storage reasons, we delete files after a set period of time.  this meant that i could only obtain four days worth of data, in other words, only a snapshot in time. location during those four days, the two stolen ids were accessed from a total of 17 different countries from all over the world (figure 1).  some of the ip addresses could not be identified, but most could be traced to a suspected country of origin.  it is possible that some of the ips were intermediate hosts or proxies, and the user actually resides elsewhere, but that is not possible to detect from the data available.  interesting to note is the wide variety of countries from many continents. figure 1.number of ip addresses by country. host information was also looked up and investigated further with google searching.  hosts fell into seven categories (figure 2).  the biggest category was legitimate commercial and/or residential internet service providers (isps).  the second biggest category was unknown.  if the host of an ip is unknown, this may indicate that it is not reputable, but not necessarily.  perhaps the most interesting finding, is that some use was coming from known university or healthcare organization ip ranges. figure 2.organization or isp types of ip addresses. subject matter although one could investigate the subject matter of each article downloaded, for the sake of speed and simplicity, i instead looked at the domain names and determined the vendors or publishers.  not surprisingly, the majority of the visited domains were vendors that specialize in science, technology, medicine (stm) or have strong stm collections within their large multidisciplinary holdings.  the complete list of vendors/publishers accessed is in appendix a. quantity there is no question that some excessive use was occurring that was likely systematic downloading. after all, that was the behavior that triggered this investigation. however, if you look at all the sessions over that four day period, there are only a few spikes in download quantity, with many sessions showing moderate quantities. please note that the bytes represented in figure 3 are total bytes downloaded in the session including not just articles, but also information like search results and webpage components. figure 3.download quantity for each session. limitations this data is only from two ids over a short period of time and cannot be extrapolated to represent the nature of all unauthorized use.  this case study is not meant to prove that there exists any particular percentages, rather it attempts to shed light on the fact that these types of unauthorized use do exist.  for example, not all unauthorized use is coming from china.  not all unauthorized use is excessive downloading or spidering activity.  unauthorized use from hospitals and universities does exist. prevention regardless of the type of unauthorized use, it is still important to try reasonable prevention measures to uphold our licenses, and also to prevent downtime when vendors shut off access.  the systems in place for security must be reasonable for libraries to implement.  for example, it is not reasonable to think that libraries are going to eliminate stolen passwords completely.   in addition to general good practices for it security, there are a couple of proxy-specific prevention methods that are also reasonable to implement. one relatively simple method to detect stolen passwords, even if the usage is not abnormal, is to have a script run nightly on the ezproxy audit files and scan for logins to a single id that are coming from multiple locations.  when multiple locations are detected, the script outputs a warning message that is put into an email message to alert staff so they can ensure the library user changes their password.  if local ip ranges are excluded, and a minimum threshold set, any usernames caught will likely be stolen.   so far in our experience, this script has not falsely fingered a library user on a legitimate whirlwind international tour.   also, in our experience, simply having the library user change their password is enough to stop the misuse. #!/usr/bin/perl use strict; use socket; my $today = `date +"%y%m%d"`; $today =~ s/\s*$//; $today = $argv[0] if ($argv[0]); my $file = "/usr/local/ezproxy/audit/" . $today . ".txt"; my $min_threshold = 3; my %logins = (); &dofile($file); # multiple (min_threshold) successful login locations outside local city for a user # should be an indication of a stolen password # foreach my $user (keys %logins) { my @ips = split /::/, $logins{$user}; if ($#ips >= $min_threshold) { print "\nalert: multiple login locations for $user:\n"; foreach my $ip (@ips) { print "\t",$ip,"\n"; } } } exit; sub dofile { my ($filename) = @_; open(in,"<$filename") || die; while() { chop; my ($xime,$event,$ip,$username,$session,$other) = split /\t/; if ($event eq "login.success") { $logins{$username} = listbuild($logins{$username},$ip); } } close in; } sub listbuild { my ($list,$add) = @_; # our own ips return $list if ($add =~ m/^10\./); return $list if ($add =~ m/^128\.233\./); # other local isps return $list if ($add =~ m/^71\.17\./); return $list if ($add =~ m/^216\.197\./); if ($list eq "") { return $add; } my @items = split /::/, $list; foreach my $item (@items) { if ($item eq $add) { return $list; } } return $list . "::" . $add; } another relatively simple method of finding stolen passwords is to occasionally google your proxy server domain name.  the logins and passwords are sometimes shared or sold on websites and discussion boards in a fairly open way.   this method would only catch a small number, but it is worth a look. future of access and unauthorized use the move from ip and proxy based authentication towards shibboleth authentication has begun and is growing in popularity.  there is a hope that shibboleth may reduce the workload of libraries in maintaining proxy configurations and submitting ip ranges.  at the same time, it protects patron privacy by continuing to authenticate users at their home institution and is based on trust in the local authentication.3 a stolen password will appear to be legitimate and will be authenticated against the local organization in the same way it is now.  shibboleth will not protect against stolen passwords any more than our current system does.  the difference is that when a vendor detects excessive use, they will only need to deny access to that single user, rather than blocking an entire proxy server and effectively blocking all remote access.  limited unauthorized use may go undetected unless the vendors themselves look for multiple locations or other markers of suspicious activity.  with shibboleth, libraries will often rely on their parent organizations to be the local authentication (identity provider).  if libraries want to learn more about who is accessing resources, they will need to communicate with the identity provider and/or their vendors (service providers), rather than simply look at their own proxy server activity. in the print-based world, security and access were not so diametrically opposed.  stopping theft from the library was necessary to ensure continued access for the rest of the patrons.  limiting use of rare or fragile books was necessary to preserve them for use by future generations.  of course, access was also restricted due to the physical nature of the resources and costs/budgets.  those things were not under the libraries control and so we could focus on providing access.  in the electronic world, things changed.  people far away seem a lot closer and walk-in use, although important, now seems paltry.  resources were freed from their more cumbersome paper bodies, but not from copyright.  e-resource license agreements and authentication systems have become as much about keeping people out as they are about letting people in and libraries now have an active role in keeping people out.  it is a policing role that many libraries will be happy to give up to their parent organization and the access federations under shibboleth. but if libraries do less of the policing and gate keeping, will the awareness of the issues surrounding unauthorized access also be reduced?  the notification email that is generated with the data from the detection script in this article has become not just a practical tool for managing access, it is also a regular and uncomfortable reminder of the issues of unauthorized use: the difficulty in restricting information in a digital age, pricing barriers and access inequalities.  when i no longer receive them, will unauthorized use be out of sight, out of mind? conclusion many resources, such as time, hardware, and software are spent on protecting our electronic resources and, indeed, this is the agreement that we have made with many of our content providers and we must uphold our end of the bargain.  but when we see some of the people that we are keeping out, they start to look an awful lot like the people we are usually trying to serve.  it is not a coincidence that some of the resources being accessed by the unauthorized users are among the most prohibitively expensive in the world.  we can’t let the unauthorized users in now, but we can promote and support open access policies, negotiate for more consistent interlibrary loan rights in our licenses and continue to be vocal about excessive pricing.  if libraries do realize efficiencies in moving to shibboleth, perhaps this time and energy needs to be spent on opening up access.  out of sight, but not out of mind. references 1. baker g and tenopir c. 2006. managing the unmanageable: systematic downloading of electronic resources by library users. journal of library administration 44(3/4):11-24. (coins) 2. phishing [internet]. 2008. in: oxford english dictionary online. oxford university press; [cited 2010 10/13]. available from: http://www.oed.com 3. shibboleth [internet]. [copyright 2010]. internet2; [cited 2010 10/13]. available from: http://shibboleth.internet2.edu/ about the author heather tones white is an information technology librarian at the university of saskatchewan in saskatoon, saskatchewan.  she has been with the u of s since 2005 after completing her m.l.i.s. at the university of western ontario.  heather can be reached at heather.white@usask.ca acknowledgements special thanks to doug macdonald for the script and also for his patience in explaining the ins and outs of library authentication. appendix a: vendors/publishers accessed by unauthorized users   vendor or publisher subject category acls humanities e-book humanities american association for cancer research medical american association for clinical chemistry science american chemical society science american physiology society medical american society for biochemistry and molecular biology science american society for microbiology science american society for pharmacology and experimental therapeutics medical american society of hematology medical association for computing machinery technology/engineering blackwell synergy multidisciplinary book 24×7 multidisciplinary british medical journal medical canadian foreign relations index social science ebsco multidisciplinary elsevier multidisciplinary elsevier md consult medical elsevier science direct multidisciplinary elsevier scopus multidisciplinary gale multidisciplinary heinonline law ieee technology/engineering informaworld multidisciplinary informs social science journal of the american society of nephrology medical karger medical lexi-comp medical metapress multidisciplinary mit press journals multidisciplinary myilibrary multidisciplinary national academy of sciences science national center for biotechnology information science/medical nature science new science press science ovid multidisciplinary ovid silverplatter multidisciplinary oxford journals multidisciplinary proquest multidisciplinary proquest umi multidisciplinary pub med central medical refworks other royal college of psychiatrists medical sage publications multidisciplinary shattauer publishers medical society for endocrinology medical society for general microbiology science springer link multidisciplinary springer protocols science stat!ref medical statesman yearbook reference swetswise multidisciplinary thieme publishing group medical university of michigan digital collections multidisciplinary wiley multidisciplinary world book reference subscribe to comments: for this article | for all articles 8 responses to "electronic resources security: a look at unauthorized users" please leave a response below: jodi schneider, 2010-12-25 “e-resource license agreements and authentication systems have become as much about keeping people out as they are about letting people in and libraries now have an active role in keeping people out.” – a sad truth about libraries today–and one of the main things that turns patrons against libraries. tim ribaric, 2011-01-06 just wondering what the underlying password/username authentication was? something homegrown or ldap based? our ldap passwords expire after 120 days so any stolen/phished password only has 4 month self life. not the best thing to rely on but i try to tell myself that it is. heather tones white, 2011-01-19 hello tim, our university it manages the underlying password/username authentication. the library can use either cas or ldap to authenticate our patrons against their university password/username. currently there is no policy to require password changes periodically – this probably would be a good thing to do, although a bit annoying from a user perspective. thanks for the comments. jonathan rochkind, 2011-07-21 interseting related anecdote from a blog post by karen coyle: “this person told me that all access to one of the uc campuses had been cut off recently for a few days because it was discovered that someone was systematically downloading entire journal runs. when they found the student it turned out that it was a foreign graduate student who would soon be returning home. knowing that leaving the uc system would mean losing access to the journals he would need to continue his research, he was making himself a copy to take home.” http://kcoyle.blogspot.com/2011/07/unequal-access.html library values and the growing scholarly digital divide: in memoriam aaron swartz | bibliographic wilderness, 2013-01-13 […] by drip and drab as well as in bulk,  if they care to look for it.   (see for example, heather tones white, “electronic resources security: a look at unauthorized users” in the code4lib journal.) we know that people regularly take advantage of our networks to […] is china that scary? | bibliographic wilderness, 2013-01-22 […] anecdotal observations are that indeed efforts to pirate scholarly articles often originate in china, russia, and other […] patrick berry, 2013-04-22 in the sample code, you might want to make a change to the while loop so that it doesn’t run for infinity. – while() { + while() { patrick berry, 2013-04-22 that is of course: while(<in>) leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – editorial: a modest proposal for the mitigation of impostor syndrome mission editorial committee process and structure code4lib issue 45, 2019-08-09 editorial: a modest proposal for the mitigation of impostor syndrome thoughts on impostor syndrome and participation in the code4lib community by eric hanson “… a psychological term referring to a pattern of behavior where people doubt their accomplishments and have a persistent, often internalized fear of being exposed as a fraud.” [1] a frequent topic of discussion amongst my closest friends in the profession is impostor syndrome. every time i or one of my friends is up for a promotion, assigned new responsibilities, or interviews for a new position; the same things are said: i’ve never done anything like that, i don’t even know where to start. i probably won’t apply, i don’t think i could learn how to … i can’t ask anyone about that because i’ll feel incompetent. how long before they figure out i don’t know what i’m doing? i am no stranger to these feelings as i’ve transitioned from an acquisitions assistant to a metadata specialist to my current position as a software engineer. however, my status as a straight, white, cis male puts me in a privileged position that insulates me from many of the issues that contribute to the impostor syndrome affecting my friends. there are a multitude of factors rooted in sexism, racism, homophobia, transphobia, ageism, ableism, and other prejudices that create a more insidious strain of impostor syndrome than what i’ve experienced. with so many factors influencing these feelings of inadequacy, there is no one-size-fits-all cure. however, i believe that the code4lib community is an excellent resource for building knowledge and confidence through the professional connections it offers. in addition to simply gaining more experience as i spent time in each new role, the code4lib community was a major influence that helped me tame the impostor syndrome. unlike much of the professional literature i encountered, this community was focused on practical solutions and there was a lack of formality that i found inviting. it seemed to be a far more approachable community than many other professional organizations. these were not people constantly theorizing and developing complicated universal standards. there are certainly elements of that higher-level work in the code4lib community but i quickly saw that most participants, like me, were just looking to fix everyday problems. i’ve never counted myself as a very active participant but i am also not a passive consumer. i asked questions. i contacted people off-list when i thought we were working on similar problems. i emailed the authors of articles in the code4lib journal to express my thanks for the lessons i learned from their work. i made a number of connections when i attended my first code4lib conference in san jose earlier this year. i also answer questions when i feel qualified, which has become a more frequent occurrence. at first, i worried that my questions were too basic and would just be ignored. i quickly discovered these worries were misplaced. even when someone did not have “the solution,” they would often point me to documentation or other people that might have the answer. everyone i contacted in the community responded humbly and without condescension. this made me far more comfortable seeking advice, which led to a number of deeper professional connections. i not only gained knowledge from these individuals but they also modeled how to effectively provide information so that i understood both how to fix a problem and why that fix worked. as a result of this, i’m also more eager to offer assistance to others, from answering one-off questions to more extended collaborations and mentoring opportunities. when i consider my own positive feelings and gratitude towards those who assisted me, i want to pay that forward to others seeking solutions and greater knowledge. while i can only speak to my own experience in the community, i hope it is a common experience and not an exceptional one. if other members have different experiences, i hope that the community can find a way for them to safely share their stories so that we all have an opportunity to learn. we should always be asking ourselves tough questions in the service of making our community more welcoming and doing it thoughtfully so that the burden does not fall on those who are marginalized. the recent discussions about what topics are appropriate for the listserv revealed a community that is very much a work-in-progress with regards to inclusivity but willing to engage in these types of necessary, and often uncomfortable, conversations. i encourage anyone who has just been lurking to reach out to others who are working on similar problems and to let people know when their work has helped you. similarly, i encourage all members, as they are able, to be as receptive and welcoming to inquiries as those i’ve contacted. if you have knowledge, share it openly and without the harmful attitudes that are commonly associated with seeking and giving technical advice. library technology is a wonderful community in that we are encouraged to share our solutions with each other and extend them. unlike the private sector, which generally treats solutions as closely guarded secrets to be kept proprietary and hidden. let us take advantage of that distinction. *thank you to maggie dull and lora woodford for their editing help with this article.* references [1] dalla-camina, megan. “the reality of imposter syndrome.” psychology today, https://www.psychologytoday.com/us/blog/real-women/201809/the-reality-imposter-syndrome subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – standardization of journal title information from interlibrary loan data: a customized python code approach mission editorial committee process and structure code4lib issue 57, 2023-08-29 standardization of journal title information from interlibrary loan data: a customized python code approach interlibrary loan (ill) data plays a crucial role in making informed journal subscription decisions. however, inconsistent or incomplete data associated with journal titles and international standard serial numbers (issns) as data points often entered inaccurately by requestors, presents challenges when attempting to make use of the ill data. this article introduces a solution utilizing customized python code to standardize journal titles obtained from user-entered data. the solution incorporates a preprocessing workflow that filters out irrelevant information and employs application programming interfaces (apis) to replace inaccurate titles with precise ones based on retrieved issns, ensuring data accuracy. the solution then presents the processed data in a dashboard format, highlighting the most requested journals and enabling librarians to interactively explore the data. by adopting this approach, librarians can make well-informed decisions and conduct thorough analysis, resulting in more efficient and effective management of library resources. by jennifer ye moon-chung 1. introduction the gumberg library at duquesne university has been utilizing the illiad statistics tool from atlas systems with support from oclc. this software not only facilitates efficient interlibrary loan (ill) transactions but also provides valuable statistical data and web-based reporting systems (atlas systems 2019). through the analysis of the data gathered via illiad, gumberg library has been able to review monthly trends in request rates, fulfillment rates (the percentage of successfully filled requests), and reasons for request cancellations, considering both lending and borrowing activities involving returnable items, such as physical books, and non-returnable items like electronic journal articles and book chapters. these metrics have proven instrumental in understanding the demand for different document types, analyzing usage patterns by date, and identifying the specific school or program within our institution making the most requests. moreover, the statistics derived from illiad have played a vital role in completing the library’s annual report, offering a comprehensive overview of ill transactions (acrl 2006). this data can be accessed through a monitoring tool managed by gumberg library known as the “illiad activity” dashboard. [1] however, the gumberg library sought to delve deeper and leverage this data to make informed decisions regarding journal subscriptions, as these subscriptions represent a significant portion of the library’s budget (petrick 2002). one noteworthy observation from the analysis of the “reasons for cancellations” data, accessible through the “illiad activity” dashboard, was the significant number of cancellations resulting from fulfilling requests through existing locally-held journal subscriptions. this observation prompted the library to critically evaluate its subscription decisions, considering whether to add new journals or retain existing ones, with a focus on data-driven analysis. developing a new reporting system to address this question posed certain difficulties. the data required for analysis was primarily created by patrons who requested interlibrary loan materials. while the library provides a form for patrons to fill out, there are no additional steps or verification process involved between the form and the data entered by patrons is stored directly in the illiad system. consequently, the library faced numerous typos, inconsistencies, and errors in the data when downloading reports from the illiad reporting system. additionally, the aggregated data offered by illiad did not fully align with the library’s specific reporting needs, necessitating customization. for instance, although we could obtain information about the most requested loans (limited to title information) within a selected date range throughout the illiad web reporting system, we desired additional insights such as whether those specific title requests were filled, cancelled, and the reasons for request cancellations if applicable. furthermore, due to concerns regarding personally identifiable information, we remove entire transaction records from the system after a specified period of time. to address these challenges, we utilized custom python code. while python provides a wide range of open-source libraries, we opted for simplicity in this specific task and relied on a minimal set of libraries, including pandas [2] for data manipulation and cleaning tasks, as well as requests [3] and json [4] for efficient communication through api calls via http, ensuring effective data retrieval for analysis purposes. the code presented in this paper has been specifically developed to address the unique requirements of processing collected journal request data. while the term “customized” implies that its direct applicability may vary in other scenarios, sharing this information aims to offer real-world use cases and assist others in overcoming similar challenges. by adapting the code to their specific needs, making necessary adjustments, and exploring innovative solutions, practitioners can collectively make progress in effectively handling issues associated with messy data. 2. methods 2-1. importing libraries and accessing dataset to begin the customized code implementation, the first step involves importing the required libraries or modules. in the presented code snippet, the following libraries are imported: pandas, datetime, json, and requests. import pandas as pd from datetime import datetime import json import requests next, the code accesses the library staff’s designated box.com folder, which serves as the storage location for the monthly harvest data from the illiad system. [5] to ensure the security of duquesne online services, which are protected by single sign-on (sso) authentication, we utilized box.com. if library staff have their box folder, or any shared drive (e.g., onedrive or google drive) synced to their desktops, the default file paths for windows are as follows: c:\users\{username}\box> c:\users\{username}\onedrive> c:\users\{username}\google drive> the term {multipass_id} is used interchangeably with {username}, as in the duquesne environment they are identical. by prompting the library staff to enter their multipass id, the input file path is programmatically constructed, preventing accidental modifications to the code and ensuring exclusive access to the box folder by authorized staff. # note: please ensure that the inputted data is in excel format(.xlsx). multipass_id = input("enter your multipass id: ") input_name = input("enter illiad monthly file name: `do not enter .xlsx part`" ) input_file_name = input_name+".xlsx" input_file_path = (f"c:\\users\\{multipass_id}\\box\\\annual report procedures\\illiad statistics\\ill_titles_data\\{input_file_name}") excel_object = pd.excelfile(input_file_path) df_all = pd.read_excel(input_file_path, sheet_name = excel_object.sheet_names[0]) df_all.info() please note that this code is specifically designed to work with excel files (.xlsx) since we export illiad data in this format on a monthly basis. library staff can modify the line input_file_name = input_name+".xlsx" to input_file_name = input_name+".csv" if they wish to use csv files by utilizing with the pd.read_csv [6] method instead of pd.read_excel [7]. we have tested this code on windows 10 with python version 3.8.13 and higher, but library staff should ensure compatibility with their python environment for optimal results. furthermore, to use this code, library staff need to modify the input_file_path variable with the file directory where their data is stored. for mac os users, the file path can be adjusted as follows: input_file_path =(f"/users/{multipass_id}/box/annual report procedures/illiad statistics/ill_titles_data/{input_file_name}") this structure points directly to the file directory. therefore, if the file is stored in the downloads folder, the path would be users/{user_name}/downloads/{input_file_name}. alternatively, instead of pointing to a local file, the input_file_path can also be set to open a url, such as input_file_path = "https://github.com/imyem7/duq_open/raw/main/sample_data.xlsx".[8] enter your multipass id: my_username enter illiad monthly file name: `do not enter .xlsx part`: illiad-export-april-2020 <class 'pandas.core.frame.dataframe'> rangeindex: 1674 entries, 0 to 1673 columns: 219 entries, transaction number to courier info dtypes: bool(1), datetime64[ns](4), float64(90), int64(2), object(122) memory usage: 2.8+ mb figure 1. code execution result − summary of raw data, “illiad-export-april-2020” the test dataset used as a sample is the april 2020 illiad usage data, deliberately selected due to its relevance during the covid-19 pandemic when interlibrary loan transactions were at their highest, effectively demonstrating the impact of this tool. this process can be applied to any current dataset and is currently employed as part of the dashboard update procedure. the dataset originally contained 1674 rows and 219 columns, including sensitive patron information that necessitates careful handling, as is common with many usage datasets (figure 1). however, since patron information is not required for analyzing the illiad journal request report, we sliced the dataset to remove unnecessary and potentially sensitive data by selecting only the columns necessary effectively anonymizing the data. the following code block demonstrates how to select the necessary data when defining a data frame: ill_df = pd.read_excel(input_file_path, usecols=["transaction number",\ "request type", "process type", "photo journal title",\ "photo journal year", "issn", "creation date", "status",\ "transaction status", "reason for cancellation", "document type",\ "department"]) # filter for only article requests filter = ill_df["request type"].isin(["article"]) article = ill_df[filter] article.info() required columns are selected using the usecols parameter, which includes “transaction number”, “request type”,”process type”, “photo journal title”, “photo journal year”, “issn”, “creation date”,”status”, “transaction status”, “reason for cancellation”, “document type”, “department”. the dataset is further filtered to include only the rows where the request type is “article” using the pandas dataframe.isin() method. <class 'pandas.core.frame.dataframe'> int64index: 1629 entries, 1 to 1673 data columns (total 12 columns): #   column                   non-null count  dtype --------                   --------------  ----0   transaction number       1629 non-null   int64 1   request type             1629 non-null   object 2   photo journal title      1625 non-null   object 3   photo journal year       1589 non-null   object 4   transaction status       1629 non-null   object 5   issn                     1597 non-null   object 6   reason for cancellation  767 non-null    object 7   process type             1629 non-null   object 8   document type            1172 non-null   object 9   creation date            1629 non-null   datetime64[ns] 10  status                   1629 non-null   object 11  department               1626 non-null   object dtypes: datetime64[ns](1), int64(1), object(10) memory usage: 165.4+ kb figure 2. code execution result − summary of sliced data, a subset of “illiad-export-april-2020” as a result, the dataset is reduced from 1674 rows and 219 columns (figure 1) to 1629 rows and 12 columns (figure 2). 2.2. string manipulation string manipulation techniques played a crucial role in this project. they typically involve a variety of methods, including removing white spaces, finding and replacing specific characters, changing case, splitting strings into substrings, and joining multiple strings together. our primary focus in this project was on cleaning the issns and international standard book numbers (isbns) within the illiad system which had been directly input by users. although we didn’t require it for this particular project, in more complex cases, regular expressions using the re module would be worth considering. [9] the issn typically consists of two groups of four digits, separated by a hyphen [-] following the format “issn 0123-4567”. however, users often input issns with variations, such as removing hyphens, leaving spaces, or omitting spaces altogether, which complicates the automation process for practitioners. similarly, isbns present additional complexities. with the transition of the isbn system from ten-digit to thirteen-digit isbns since 2007, and the inclusion of hyphens in both formats, the range of variations in isbn entry becomes even wider. the provided code result includes a randomly selected dataset from rows 330 to 350, showing the issn and isbn variations along with their respective frequencies (figure 3). 1538-9235; 1040-5488 1 0015-5357 1 0361-5227 1 9780761922605 1 9781555811044 1 9780890891278 1 9789004122581 1 0250 6262 1 1041-6080 1 0022-2224 1 1439-9598 1 1066-5684 1 978-0-7456-6130-8 1 0147-9563 1 1387-6740 1 0039-7911 1 2397-334x 1 0079-6123 1 0882-8245 1 0074-7742 1 name: issn, dtype: int64 figure 3. variations in issn and isbn entries in this study, our approach involved removing all white spaces and hyphens from the issn numbers using the strip() and replace() methods. for instance, df.replace("-", "") replaced hyphens with an empty string, and replace(" ", "") removed the white spaces between numbers. the strip() method was used to remove leading and trailing characters, including empty spaces. the code below provides an example of the approach we implemented. issn = '1234-5678' issn_clean = issn.replace("-", "").replace(" ","").strip() print(issn_clean) issn = '1234 5678' issn_clean = issn.replace("-", "").replace(" ","").strip() print(issn_clean) issn = ' 1234-5678' issn_clean = issn.replace("-", "").replace(" ","").strip() print(issn_clean) while issn numbers may vary in format, the resulting cleaned issn (“issn_clean”) is consistently represented as “12345678” (figure 4). 12345678 12345678 12345678 figure 4. code execution result − the numeric issn once we obtained the numeric issn digits, which have been cleaned of symbols or spaces (regardless of whether they were initially properly placed), we reinserted the hyphen (-) in the appropriate place. this step was necessary because most issn databases utilize hyphenated issns and expect the correct formatting. the following code block demonstrates how to format an issn number appropriately. issn=str("12345678") x_issn= issn[0:4] + "-" + issn[4:8] as a result, the output value “12345678” (figure 5) becomes “1234-5678” (figure 5). '1234-5678' figure 5. code execution result − formatted issn 2.3. api requests unlike cleaning issn or isbn entries, cleaning the title information entered by users can be a more complex task. we have observed a variety of variations, including mixed use of upper and lower cases, incorrect spacing, typos, and incomplete titles. while advanced string manipulation techniques, such as using regular expression, can enhance the accuracy, limitations still exist, particularly when title entry itself is incomplete. instead of directly cleaning the title entries, we opted to verify title information based on the issn or isbn data. our approach involved retrieving the journal or book title from the selected databases using api requests. in the context of this project, while we recognized that there are other potential data sources available, we chose crossref and google books as our data sources based on several factors. these include the robust documentation provided by these platforms, the availability of accurate and extensive data, and the convenience of free access. crossref is a widely recognized database in the academic community, primarily focused on scholarly content, including journal articles and conference papers. by leveraging the crossref rest api, we can access a vast collection of bibliographic data, such as publication titles, authors, abstracts, and even full-text articles in some cases. [10] to retrieve metadata from crossref, we utilized the simple requests library, which allows sending http/1.1 requests. the data received from the api is in json format, which can be parsed using the json method. python users also have the option to use third-party libraries for interacting with crossref, such as crossref-commons [11], habanero [12], crossrefapi [13]. once we chose the library to be utilized, the initial step was to identify the suitable endpoints that expose the specific information required. this information can be found on the crossref unified resource api page (https://api.crossref.org/swagger-ui/index.html). in our case, we utilized the "/journals/{issn}" endpoint to retrieve journal information based on an issn. by constructing the proper url, we could make a request and receive a response containing the desired data. for example, to retrieve journal information for the issn “1940-5758”, we defined the “issn” variable as “1940-5758” and the “api_url” variable as “f’https://api.crossref.org/journals/{issn}'”. we then initiated a get request using the requests library and stored the response in the “response” variable. finally, we were able to access and parse the data in json format using the response.json() method. here is the corresponding code we tested: issn = "1940-5758" api_url = f"https://api.crossref.org/journals/{issn}" headers = { 'user-agent': 'issn-info/1.0 (mailto:myemail@email.com)'} # insert your project name and contact info response = requests.get(api_url, headers = headers) data = response.json() data by executing the code, metadata for the journal with the issn “1940-5758” was retrieved (figure 6). the parsed data included various details, such as the journal’s title, which in this case was “code4lib”. {'status': 'ok', 'message-type': 'journal', 'message-version': '1.0.0', 'message': {'last-status-check-time': 1687724240530, 'counts': {'current-dois': 1, 'backfile-dois': 3, 'total-dois': 4}, 'breakdowns': {'dois-by-issued-year': [[2018, 2], [2021, 1], [2019, 1]]}, 'publisher': 'crossref test', 'coverage': {'affiliations-current': 0.0, … some results omitted for space limitations. … 'title': 'code4lib', …. some results omitted for space limitations. 'issn': ['1940-5758'], 'issn-type': [{'value': '1940-5758', 'type': 'print'}]}} figure 6. code execution result − journal information associated with the issn “1940-5758” similarly, the google books api provides access to google books’ extensive database, allowing us to retrieve comprehensive book information such as titles, authors, publishers, descriptions, and more [14]. to initiate an api call, we followed a similar process as with the crossref api, but with one major difference: authentication is required for google api calls, as specified in the google api terms of use. to test this process, users are required to obtain an api key from the google cloud console (https://console.cloud.google.com/). once authentication is completed, we constructed the api url by combining the base url for google books api (https://www.googleapis.com/books/v1/volumes), the query parameter specifying the isbn to search for (q=isbn:{isbn}), and the api key (key={api_key}). for example, if we wanted to retrieve metadata associated with the isbn “9781530051120”, the code would be as follows: isbn= "9781530051120" api_key = "-----------------" # insert google api key url = "https://www.googleapis.com/books/v1/volumes?q=isbn:" + isbn\ + "&key=" + api_key response = requests.get(url) book_data = response.json() book_data as a result, we were able to confirm that the book associated with the isbn “9781530051120” is titled “python for everybody,” along with other relevant information (figure 7). {'kind': 'books#volumes', 'totalitems': 1, 'items': [{'kind': 'books#volume', 'id': 'zjqzdaeacaaj', 'etag': 'o2x+zybqctm', 'selflink': 'https://www.googleapis.com/books/v1/volumes/zjqzdaeacaaj', 'volumeinfo': {'title': 'python for everybody', 'subtitle': 'exploring data in python 3', 'authors': ['charles r. severance'], 'publisheddate': '2016-04-09', 'description': 'python for everybody is designed to introduce 'industryidentifiers': [{'type': 'isbn_10', 'identifier': '1530051126'}, {'type': 'isbn_13', 'identifier': '9781530051120'}], 'readingmodes': {'text': false, 'image': false}, 'pagecount': 242, 'printtype': 'book', …. some results omitted for space limitations. … figure 7. code execution result − metadata associated with the isbn “9781530051120” furthermore, by understanding the json structure of the response, we were able to extract the specific information we needed. in this case, we observed that the result data is structured within nested curly braces,{}. the “title” information is located under “volumeinfo”, which, in turn, is nested under “items”. the following code demonstrates how we extracted the title information. items = book_data.get('items', []) volume_info = items[0].get('volumeinfo', {}) title = volume_info.get('title') print("title:", title) the result of our extraction process was as follows: title: python for everybody figure 8. code execution result − extracted book title from nested json structure 2.4. automating the process with a for loop in order to automate the process, considering the limitations in library data work such as limited staff and time constraints, we utilized a for loop in python. the for loop operates by iterating through a provided list of items and executing the defined steps sequentially for each item. for instance, we conducted a test using a list of issn numbers, such as "194057-58", "1940.5758", and "        19405758". the for loop iterated through this list, performing tasks such as cleaning the issn numbers and retrieving the corresponding titles from the crossref database. the following example code illustrates this test: import time issn_list = ["194057-58", "1940.5758" , " 19405758"] headers = { 'user-agent': 'issn-info/1.0 (mailto:myemail@email.com)'} for issn in issn_list: cleaned_issn = issn.replace("-", "").replace(" ", "").strip() time.sleep(2) crossref_url = f"https://api.crossref.org/journals/{cleaned_issn}" crossref_response = requests.get(crossref_url, headers = headers) crossref_data = crossref_response.json() if 'message'in crossref_data: journal_info = crossref_data['message'] print("journal title:", journal_info.get('title')) else: print("not available") as a result, regardless of the different representations of the issn numbers in the list, we obtained the same title information, “code4lib,” indicating the successful iteration and retrieval of corresponding titles from the crossref database using the for loop (figure 9). journal title: code4lib journal title: code4lib journal title: code4lib figure 9. code execution result – processed titles by iterating with a for loop building upon this foundation, we extended the functionality to incorporate data requiring isbn numbers. by implementing an if statement, we systematically assessed each item in the “issn_list” to determine whether it represented an issn or an isbn. [15] if the value corresponded to an eight-digit issn, the code executed the necessary steps to request title information from the crossref database. conversely, if the value represented a ten-digit or thirteen-digit isbn, the code performed the relevant tasks to retrieve the title from the google books database. the final version of the code we deployed is presented below: [16] """ issn look up""" issn_list = list(article["issn"]) transaction_list = list(article['transaction number']) # initialize a new dataframe new_df = pd.dataframe(columns=['transaction number', 'issn', 'title', 'original']) # update the transaction number for idx, v in enumerate(transaction_list): new_df.loc[idx, 'transaction number'] = v # iterate through the transaction list and search for corresponding issns and titles from crossref for idx, x in enumerate(issn_list): x = str(x) new_df.loc[idx, 'original'] = x x_short = x.replace("-", "").replace(" ", "").strip() if len(x_short) == 8: x_issn = x_short[0:4] + "-" + x_short[4:8] api_url = f"https://api.crossref.org/journals/{x_issn}" try: headers = {'user-agent': 'issn-info/1.0 (mailto:myexample@email.com)'} response = requests.get(api_url, headers=headers) data = response.json() # retrieve the issn and title from the response if "message" in data: journal_info = data["message"] if "issn" in journal_info: new_df.loc[idx, "issn"] = journal_info["issn"][0] if "title" in journal_info: new_df.loc[idx, "title"] = journal_info["title"] except: pass # iterate through the transaction list and search for corresponding isbns and titles from google books elif len(x_short) == 13 or len(x_short) == 10: try: api_key = "-------------" # your key here. url = "https://www.googleapis.com/books/v1/volumes?q=isbn:" + x_short + "&key=" + api_key response = requests.get(url) book_data = response.json() if 'items' in book_data: items = book_data['items'] if len(items) > 0: volume_info = items[0]['volumeinfo'] new_df.loc[idx, 'issn'] = volume_info['industryidentifiers'][0]['identifier'] new_df.loc[idx, 'title'] = volume_info['title'] except: pass in summary, this final code encompasses the following steps. firstly, it defines the “issn_list”, which is the list of issn numbers to be parsed alongside the transaction numbers. subsequently, a new dataframe is created to store the parsed issn numbers and their corresponding titles. using a combination of a for loop and if/elif statements, the code evaluates each issn number to determine whether it represents an issn or an isbn. based on the identification, the code retrieves the title information from either the crossref or google books database. the resulting titles and their corresponding issn numbers are then stored in the newly created dataframe. 3. results and discussion by implementing our customized python code, we achieved a substantial improvement in the accuracy of our data project. to showcase this improvement, we conducted a comparison between the original dataset and the processed dataset, specifically focusing on the top 10 most requested journal titles. the comparison revealed both minor and significant differences in the counts and specific journals between the two datasets. the table below presents the top 10 records. original data order title num of requests 1 biochimica et biophysica acta. international journal of biochemistry and biophysics 0006-3002 20 2 the journal of speech and hearing disorders : jshd 15 3 clinical pharmacy 14 4 archives internationales de pharmacodynamie et de therapie 9 5 american journal of hospital pharmacy 8 6 journal of the american dietetic association 8 7 psychosomatic medicine 8 8 the journal of laboratory and clinical medicine 8 9 journal of reproductive medicine 7 10 zhurnal obshcheĭ khimii 7 processed data order title num of requests 1 biochimica et biophysica acta 22 2 journal of speech and hearing disorders 15 3 clinical pharmacy 14 4 gestalt review 11 5 jama 10 6 journal of laboratory and clinical medicine 9 7 american journal of hospital pharmacy 9 8 psychosomatic medicine 9 9 archives internationales de pharmacodynamie et de therapie 9 10 журнал общей химии 8 table 1. top 10 journal title comparison (original vs. processed dataset) the top three records showed minimal variation. for instance, the most requested journal titled “biochimica et biophysica acta. international journal of biochemistry and biophysics 0006-3002” had a count of 20 requests in the original data, whereas in the processed dataset, it appeared as “biochimica et biophysica acta” with a count of 22 requests. however, notable disparities were observed in the fourth and fifth records. in the processed data, “gestalt review” emerged as the fourth most requested journal with 14 requests, whereas it did not appear in the top requested list of the original dataset. upon investigating the original records using the transaction number, we discovered that variations in how users entered the journal name resulted in it being listed as several different journals in the data (figure 10). examples of these variations include entries with different capitalizations, variations in subtitle abbreviation, and the presence of punctuation. 58 gestalt review 61 gestalt review : a publication of the gestalt international study center. 323 gestalt review. 324 gestalt review. 697 gestalt review 698 gestalt review 699 gestalt review 701 gestalt review 776 gestalt review 990 gestalt review 1136 gestalt review : a publ. of the gestalt international study center. figure 10. variations in title entries for "gestalt review" similarly, while “jama” did not appear in the top 10 list of the original dataset; it ranked fifth in the processed dataset with 10 requests. upon examining the original records, we identified similar issues in how users entered the journal name, such as inconsistencies in the use of punctuation, and the absence of words like “the” in front of the subtitle (figure 11). 291 jama : the journal of the american medical association. 500 jama : the journal of the american medical association. 1131 jama : the journal of the american medical association. 1248 jama: journal of the american medical association 1456 jama : the journal of the american medical association 1497 jama : the journal of the american medical association 1523 jama : the journal of the american medical association 1547 jama : the journal of the american medical association 1593 jama : the journal of the american medical association 1653 jama : the journal of the american medical association figure 11. variations in title entries for “jama” additionally, when considering the dataset for april 2020 alone, we found that 992 out of 1629 records had modified titles after applying our customized code (figure 12). this corresponds to approximately 60% of the titles being changed through our data cleaning and standardization process. the number of unique journal titles decreased from 1,151 to 1,097, as well. these findings demonstrate the effectiveness of our data processing approach in cleaning and standardizing journal title data. figure 12. bar graph showing the impact of data cleaning once the data is clean and standardized, the value of the data increases, enabling the data’s use in various applications. in our case, we harnessed the power of the clean and processed data to integrate it into the powerbi visualization tool, facilitating streamlined data visualization and enabling prompt identification of patterns and trends (figure 13). notably, we observed significant fluctuations in the request volume for certain journals over time. for instance, there was a substantial increase in requests for medical journals like the lancet during the covid-19 outbreak in 2020, followed by subsequent decline in request volume. these insights provided a comprehensive understanding of journal request dynamics and facilitated data-driven decision-making [17]. figure 13. illiad journal request dashboard. notably, the size of the blue bubble is larger compared to other bubbles on the dashboard in fy2020. additionally, the size of this blue bubble appears to decrease in consecutive years, indicating a trend of decreasing journal requests for “the lancet” over time. in terms of maintaining this dashboard, our customized code for data cleaning and standardization ensures consistent data formatting and sustainability, eliminating the need for additional manual labor for regular updates. while our data processing approach has achieved significant accomplishments and improvements, it is important to acknowledge certain limitations and consider future enhancements. although our python code has effectively cleaned and standardized the data, there is still potential for further modifications to enhance its annotation capabilities and facilitate collaborative work. additionally, the development of graphical user interface (gui) applications can provide added convenience and accessibility. considering the constraints of time and resources, we have prioritized simplicity and efficiency in our current code implementation, which has proven valuable for our workflow. however, moving forward, we will explore opportunities for code optimization and the development of user-friendly interfaces to enhance the efficiency and accessibility of our data processing and analysis tasks. 4. conclusions this study demonstrated the importance of clean and standardized data in library data projects. clean and standardized data is invaluable in any data analysis project, including library data projects. it not only lays the groundwork for analysis but also enhances the quality of insights and facilitates more effective data utilization. by implementing the customized python code, we successfully processed and cleaned journal title data, resulting in improved accuracy and efficiency. the combination of string manipulation techniques, api requests, and automation through for loops allowed us to handle issn and isbn variations effectively and retrieve the corresponding titles. future enhancements should focus on maximizing code annotation capabilities and developing user-friendly interfaces to further enhance efficiency and collaboration. references acrl. 2006. standards for libraries in higher education. association of college & research libraries (acrl). [accessed 2023 jun 26]. https://www.ala.org/acrl/standards/standardslibraries. atlas systems. 2019. reports and statistics. atlas systems. [accessed 2023 jun 26]. https://support.atlas-sys.com/hc/en-us/articles/360011809534-reports-and-statistics. batalha f. 2023. a python library that implements the crossref api. [accessed 2023 jun 26]. https://github.com/fabiobatalha/crossrefapi. chamberlain s. 2023. client for crossref search api. [accessed 2023 jun 26]. https://github.com/sckott/habanero. crossref python commons library. 2022 aug 8. gitlab. [accessed 2023 jun 26]. https://gitlab.com/crossref/crossref_commons_py. crossref unified resource api 0.1. [accessed 2023 jun 26]. https://api.crossref.org/swagger-ui/index.html. imyem7. 2023. duq_open. [accessed 2023 jun 26]. https://github.com/imyem7/duq_open. json encoder and decoder. python documentation. [accessed 2023 jun 26]. https://docs.python.org/3/library/json.html. moon-chung jy, behary r, slimon l. libguides: gumberg library analytics: gumberg library. [accessed 2023 jun 26]. https://guides.library.duq.edu/dashboards. overview | google books apis. google for developers. [accessed 2023 jun 26]. https://developers.google.com/books/docs/overview. pandas – python data analysis library. [accessed 2023 jun 26]. https://pandas.pydata.org/. pandas.read_csv — pandas 2.0.3 documentation. [accessed 2023 jul 16]. https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html. pandas.read_excel — pandas 2.0.2 documentation. [accessed 2023 jun 26]. https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_excel.html. petrick j. 2002. electronic resources and acquisitions budgets: suny statistics, 1994‐2000. collection building. 21(3):123–133. doi.org/10.1108/01604950210434560. python documentation: compound statements. python documentation. [accessed 2023 jun 26]. https://docs.python.org/3/reference/compound_stmts.html. regular expression operations. python documentation. [accessed 2023 jun 26]. https://docs.python.org/3/library/re.html. reitz k. requests: python http for humans. [accessed 2023 jun 26]. https://requests.readthedocs.io. notes [1] “illiad activity” dashboard, which is available at https://guides.library.duq.edu/dashboards, is updated monthly and contains data from january 2017 to the present. (moon-chung et al.). [2] pandas is a fast, powerful, and user-friendly open-source data analysis and manipulation tool for python. pandas documentation is available at https://pandas.pydata.org/docs/ (pandas – python data analysis library). [3] the requests library is a widely-used python module for making http requests and interacting with web apis. the documentation is available at https://pypi.org/project/requests/ (reitz). [4] the json library is a built-in module in python and it allows for easy conversation between json strings and python objects. the official python documentation is available at https://docs.python.org/3/library/json.html (json encoder and decoder). [5] box refers to a cloud-based file sharing and collaboration platform. https://www.box.com/ . [6] for further information and details, refer to the documentation available at https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html (pandas.read_csv — pandas 2.0.3 documentation) [7] the pandas.read_excel function offers support for reading file extensions such as xls, xlsx, xlsm, xlsb, odf, ods, and odt from both local filesystems and urls. for further information and details, refer to the documentation available at https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_excel.html. (pandas.read_excel — pandas 2.0.2 documentation). [8] the complete code with sample data is available on the project’s github page, https://imyem7.github.io/duq_open/ (imyem7 2023) [9] the re module in python enables pattern matching, searching, and manipulation of strings in a flexible and efficient manner using regular expressions. documentation is available at https://docs.python.org/3/library/re.html (regular expression operations) [10] please make sure to refer to the full crossref rest api documentation for detailed instructions on its usage. the documentation is available at https://api.crossref.org/swagger-ui/index.html (crossref unified resource api 0.1). [11] the documentation for crossref-commons is available at https://gitlab.com/crossref/crossref_commons_py (crossref python commons library 2022 aug 8). [12] habanero also offers features such as citation counts and doi resolution. the documentation is available at https://github.com/sckott/habanero (chamberlain 2023). [13] the documentation for crossrefapi is available at https://github.com/fabiobatalha/crossrefapi (batalha 2023). [14] for further information about the google books api and its capabilities, please visit their official documentation at https://developers.google.com/books/docs/overview (overview | google books apis). [15] if you would like to learn more about compound statements in python, please visit https://docs.python.org/3/reference/compound_stmts.html for further information (python documentation: compound statements). [16] the complete code with sample data is available on the project’s github page, https://imyem7.github.io/duq_open/ (imyem7 2023) [17] the illiad journal request dashboard is available at the gumberg library analytics page, https://guides.library.duq.edu/dashboards (moon-chung et al.) about the author jennifer ye moon-chung is a digital projects and analytics manager at duquesne university, and an mlis candidate focusing on applied data driven methods (addm) at the university of pittsburgh. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – video playback modifications for a dspace repository mission editorial committee process and structure code4lib issue 31, 2016-01-28 video playback modifications for a dspace repository this paper focuses on modifications to an institutional repository system using the open source dspace software to support playback of digital videos embedded within item pages. the changes were made in response to the formation and quick startup of an event capture group within the library that was charged with creating and editing video recordings of library events and speakers. this paper specifically discusses the selection of video formats, changes to the visual theme of the repository to allow embedded playback and captioning support, and modifications and bug fixes to the file downloading subsystem to enable skip-ahead playback of videos via byte-range requests. this paper also describes workflows for transcoding videos in the required formats, creating captions, and depositing videos into the repository. by keith gilbertson and liz mcvoy motivation vtechworks is the institutional repository of the university libraries of virginia tech, which launched in late 2011 using the dspace software. at launch, the plan for vtechworks was for it to act as a general-purpose repository for digital collections at the library, including theses and dissertations, scholarly articles, historical images and digital scans of cookbooks from the special collections department, and digitized materials from the archives. during the mass import of legacy content, it was noticed that several digital video files had entered into the collection, including promotional videos for the cooperative extension office and tribute videos that were sent to virginia tech after the campus mass shooting tragedy of april 16, 2007. we had interesting video content in the repository, but the initial use cases for dspace institutional repositories focused primarily on textual content: for example, collections of journal article pre-prints in pdf format.  dspace has been described as largely format agnostic, in that it allows users to deposit any type of file and to make these various file types available for download, if not in-browser display. however, the repository group at virginia tech thought that it would be fun to allow playback of videos from within the item page. also, popular video streaming sites such as youtube have conditioned users to expect embedded video playback in the web browser, instead of clicking on a video filename to download it first. thus, video playback in dspace was quickly hacked together, and the initial release of the video playback project in vtechworks happened in august of 2012.  closed caption and subtitle support was added in november of 2012, in anticipation of future needs. the service was advertised in internal (but public) blog postings (video playback …, 2012; vtechworks closed caption …, 2012), with a few sample videos converted to the required formats. by advertising the capability on a blog posting and listing the format requirements, the hope was that others would continue to submit video content to vtechworks. there was collaboration with computer science students for a class project to record computer science seminars, but outside of this successful project, few new videos were submitted to vtechworks. then, in 2013, the library event capture group was formed. the event capture group provides, free of charge, video recording services for scholarly events up to two hours in length (event capture, 2015). from the time of its formation, this group has deposited videos of over one hundred events to vtechworks, with more pending. functionality the project allows for playback of digital videos within a viewer embedded in the item pages of vtechworks.  the customized software detects whether an item has videos encoded in the necessary formats, and if so, renders an html5 video player within the web browser. if the web browser does not support html5 video, then a fallback option using the adobe flash player is presented. our implementation of this project uses the video.js (video.js player, 2015) player to customize the embedded html5 video player and to present the flash fallback option in the theme. with theme changes, the project is also compatible with alternative players, such as flowplayer, or can be made to rely entirely on the built-in controls for video playback in html5 capable web browsers. the system supports the use of a “movie poster”, an image file that is displayed in the embedded viewer before playback is requested. this can be a still from an interesting frame of the video, or a title frame for the video, and is meant to avoid display of a blank canvas in the video playback area. playback support is intended to work on popular browsers on popular operating systems, including firefox, internet explorer, chrome, and opera on os x, windows, and linux, and on mobile devices based on android and ios operating systems. some basic requirements of video playback are supported; for example, the video playback can begin even as just the beginning of the file has been downloaded, and the viewer can skip ahead to view parts of the video that have not yet been downloaded. these features are especially important for mobile devices, which may have limited storage space. if a webvtt file is present in the item, the file is used to provide text indexing of the video for the search discovery system, and to present subtitles on screen. selection of video formats after the event capture group became involved and began producing videos frequently, more thought was put into standardizing the video formats for ease of processing. in this project, the video formats can be conceived of as three types. the original format, as captured by the camera the preservation format, an archival copy that can used to create other formats presentation formats, used for display in the web browser the preservation format was selected after research by the video event capture group at the time of the initial implementation. members were liz mcvoy, scott pennington, and therese walters. the event capture group produces the preservation copy after the video has been edited. the specification is as follows: preservation format: multimedia container file: .mov video codec: prores 422 resolution: 1920×1080 the presentation formats were selected based on the capability of web browsers at the time of the original implementation, and based on experimentation. browser support for html5 video has improved greatly, but there are still differences between browsers and platforms. an updated chart of html5 compatible multimedia formats on both mobile and desktop is on the mozilla developer network. some web browsers, such as safari on os x and ios, natively supported mp4 files with h264 encoding, while other browsers, such as chrome, supported webm files with vp8 encoding. therefore, each video has two presentation files stored in vtechworks, an mp4 file and a webm file. presentation format, .mp4: multimedia container file: .mp4 video codec: h264 video, two-pass encoding resolution: 854×480 target ( do not scale up if the original is smaller) audio: aac pixel format: yuv420p moov atom: located at front of file bitrate: 1200 kbps variable presentation format, .webm: multimedia container file: .webm video codec: vp8 resolution: 854×480 target ( do not scale up if the original is smaller) audio: ogg vorbis pixel format: yuv420p bitrate: 1200 kbps variable there are some peculiarities to these specifications. the pixel format of “yuv420p” was selected for compatibility with google chrome for windows. when the videos were encoded with other pixel formats, they did not play back on chrome in windows. our specification for the .mp4 file states that the moov atom should be located at the beginning of the file. the moov atom serves as a time-based index to the rest of the file. if the moov atom is not located at the beginning of the file, then skip-ahead functionality is not enabled, and in some cases the video will not play at all. (placing moov atom … , 2010) if an mp4 file and a webm file are both present in an item, the video playback features will be enabled in the theme. in addition to the video files, it is possible to include other files to support optional features in the playback functionality. image files support thumbnails and movie posters; webvtt files support subtitles. thumbnail: file format: .jpg file naming convention: video_filename.mp4.jpg, video_filename.webm.jpg size: 100 pixels, maximum width location: thumbnail bundle in vtechworks movie poster: file format: .jpg file naming convention: video_filename.jpg size: same dimensions as 1 video frame (target 854×480) location: movieposter bundle in vtechworks dspace organizes bitstreams, or files, into logical groups called bundles. typical bundles in a dspace repository include original, thumbnail, and license. the movieposter bundle is a special bundle that was added for this project. subtitles: file format: .webvtt, specification at http://dev.w3.org/html5/webvtt/ location: original bundle in vtechworks with the current implementation, the video playback functionality will only detect and use one webvtt file for each item. due to this limitation, it is not currently possible to display subtitles in an alternate language, for example. code changes overview the code changes that were necessary to support this feature modified several subsystems of dspace, including the file download system, web theming, and configuration of indexing and file types. code additions are stored in the vtul (virginia tech university libraries) group github repository. file type registry dspace utilizes a file format registry that lists file types known to dspace. registry entries for the mp4, webm, and webvtt file formats were added to the registry, bitstream-formats.xml. <bitstream-type> <mimetype>video/mp4</mimetype> <short_description>mp4 container</short_description> <description>mp4 container format for video files</description> <support_level>0</support_level> <internal>false</internal> <extension>m4v</extension> <extension>mp4</extension> </bitstream-type> <bitstream-type> <mimetype>video/webm</mimetype> <short_description>webm video</short_description> <description>the webm video container format</description> <support_level>0</support_level> <internal>false</internal> <extension>webm</extension> </bitstream-type> <bitstream-type> <mimetype>text/vtt</mimetype> <short_description>webvtt caption file</short_description> <description>closed caption or subtitle file for html5 video</description> <support_level>1</support_level> <internal>false</internal> <extension>vtt</extension> </bitstream-type> these entries in the file type registry allow the theme to detect the presence of these files and to customize its view accordingly. the xml file shown here will configure file entries for new dspace installations. for previously installed repositories, there is a web interface to add new format entries. theming vtechworks uses the xmlui user interface for dspace 5.4. this interface uses xslt to transform xml representations of dspace objects and metadata into html for display in a web browser. the xmlui allows themes to customize views for the entire repository, or for individual collections. the current theme for vtechworks is a lightly customized version of a theme created with responsive design principles, called “mirage2”, from the @mire company. the main customization to the mirage2 theme for this project is a modification to the item view page that displays an embedded video playback area when the user is looking at video items. there are also changes to include and style the video.js plugin. first, the page-structure.xsl file was modified to include the necessary css and javascript files for the video.js tool. <!-include css and javascript for video playback --> <link type="text/css" rel="stylesheet"> <xsl:attribute name="href">http://vjs.zencdn.net/c/video-js.css</xsl:attribute> </link> <script src="http://vjs.zencdn.net/c/video.js">&#160;</script> the page links to the files on the video.js project server, instead of linking to copies that could be stored on the vtechworks server. this potentially introduces performance issues, or instability when the files are modified. however, it also allows vtechworks to automatically and effortlessly benefit from the latest improvements to the video.js project. the item-view.xsl file checks for the existence of an mp4 and a webm file, and if and only if both exist, displays the video playback area in the item view page. if a webvtt file is provided for subtitles, or an image file for use as a movie poster, these are also detected and handled. the code uses the standard html5 video, source, and track elements. video.js detects these elements and customizes them accordingly. <!-v.t. add html for the video in a videojs frame --> <xsl:if test="(./mets:filesec/mets:filegrp[@use='content']/mets:file[@mimetype='video/mp4']) and (./mets:filesec/mets:filegrp[@use='content']/mets:file[@mimetype='video/webm'])"> <!-v.t. best guess at aspect ratio of most of our videos --> <video controls="controls" preload="none" width="853" height="480" class="video-js vjs-default-skin" data-setup=""> <xsl:if test="./mets:filesec/mets:filegrp[@use='movieposter']"> <xsl:attribute name="poster"> <xsl:value-of select="./mets:filesec/mets:filegrp[@use='movieposter']/mets:file/mets:flocat[@loctype='url']/@xlink:href" /> </xsl:attribute> </xsl:if> <source type="video/webm" > <xsl:attribute name="src"> <xsl:value-of select="./mets:filesec/mets:filegrp[@use='content']/mets:file[@mimetype='video/webm']/mets:flocat[@loctype='url']/@xlink:href" /> </xsl:attribute> </source> <source type="video/mp4"> <xsl:attribute name="src"> <xsl:value-of select="./mets:filesec/mets:filegrp[@use='content']/mets:file[@mimetype='video/mp4']/mets:flocat[@loctype='url']/@xlink:href" /> </xsl:attribute> </source> <!-v.t. display captions --> <xsl:if test="./mets:filesec/mets:filegrp[@use='content']/mets:file[@mimetype='text/vtt']"> <track kind="captions" srclang="en" label="english" default="default"> <xsl:attribute name="src"> <xsl:value-of select="./mets:filesec/mets:filegrp[@use='content']/mets:file[@mimetype='text/vtt']/mets:flocat[@loctype='url']/@xlink:href" /> </xsl:attribute> </track> </xsl:if> </video> <hr /> </xsl:if> the theme is not able to detect if the moov atom is located at the front of the mp4 file, as is required per proper operation. the site superuser.com has suggestions about how to detect the location of the moov atom from scripts. also, all videos are given a viewport of the same dimensions, regardless of their aspect ratio. in our current revision,  this size is set to 854×480, which matches the dimensions of our current presentation formats in vtechworks, and is a 16/9 aspect ratio. bitstreamreader in dspace repositories using the xmlui interface, there is a java class called the bitstreamreader that is responsible for reading deposited files from storage and sending the files back to the requesting web browser. the bitstreamreader class at one time contained byte range support that enabled downloading of arbitrary portions of files. in the stock dspace distribution, this code had been commented out because it was reported to cause problems with some downloads, in particular with some external pdf viewers. changes to the bitstreamreader.java file include enabling these byte range downloads again, and fixing bugs related to arithmetic and http headers. http byte range offsets start with 0; this led to an off-by-one error that was corrected: /* entityrange = byterange.intersection( new byterange(0, this.bitstreamsize)).tostring(); */ // this code fixes off by 1 error in commented line above requestedrange = byterange.intersection( new byterange(0, this.bitstreamsize 1)); entityrange = requestedrange.tostring(); it was also discovered that the content-length http header was missing, and that the word  “bytes” was absent from the content-range header. //response.setheader("content-range", entityrange + "/" + entitylength); // v.t. fix for headers response.setheader("content-length","" + requestedrange.length()); response.setheader("content-range", ”bytes " + entityrange + "/" + entitylength); properly functioning byte range downloads are necessary to enable viewers to skip ahead to a portion of the video that has not yet been downloaded, and for allowing playback on mobile devices, which typically have little storage space. after these problems were corrected, mobile device playback and the skip-ahead feature functioned properly. at the time of publication of this article, another problem was discovered with byte range downloads.  see the “problems” section for more details. transcoding scripts during the earliest stages of experimentation, videos were encoded on an individual basis as the embedded playback feature was tested on different file types. after the event capture group was started, videos were added to the repository more often. the preservation video format had been standardized, and it was an appropriate time to attempt automation for transcoding to the presentation formats. the event capture group was created at a time when system administrator availability was difficult to come by in the library. a mac mini and several external drives were purchased to act as a makeshift file server. a perl script was hacked together (garbage scripts, 2015) to create the mp4 and webm presentation files from the preservation copy, along with the thumbnails and movie poster file. the “-movflags faststart” option is used to make sure that the moov atom is placed at the beginning of the file. this script was retired at the same time as the makeshift file server was retired and moved to a production appropriate system, and video transcoding is now handled with iffmpeg, a closed-source, commercial gui front-end for the ffmpeg encoder that runs on os x. the iffmpeg program meets all of our technical needs and is more flexible, but due to a shortage of available staff time, there is now a demand for the script to be returned to service. this is more challenging than expected because of the need to find an available server node with large storage capabilities and high compute capability. configuration the dspace configuration file, dspace.cfg, has an entry that lists bundle names that appear in the web interface for item uploads. xmlui.bundle.upload = original, metadata, thumbnail, movieposter, license, cc-license the modified line shown here adds the movieposter bundle so that the movie poster files can be uploaded from the web interface. dspace has a set of media filters that process common file types such as images, to build thumbnails, and pdf files, for full text indexing. even though webvtt is not an html file, we can configure the htmlfilter to provide full text indexing of the subtitles: filter.org.dspace.app.mediafilter.htmlfilter.inputformats = html, text, text/vtt testing the very first item to serve as a test for video in vtechworks is a tribute video that was presented to virginia tech. figure 1. screenshot of embedded tribute video the video playback functionality was tested on the most popular platforms, including android and ios, and safari, chrome, firefox, and opera on os x, and firefox, chrome, and internet explorer 6, 8, and 9 on windows. multiple versions of internet explorer were tested, because the html5 <video> tag was not supported until internet explorer 9, but internet explorer 6 was still in wide use, even in computers within the library.  for browsers that do not support html5, such as ie6, video.js provides a fallback mode that uses adobe flash. html5 video is now widely supported among commonly used browsers, and flash appears to be on the decline. during debugging, curl, as well as the live http headers plugin for firefox were both used extensively. deposit workflow after editing and exporting the preservation copies of the videos and saving them on a file share with nightly backups, the event capture group places a copy of the preservation file into a special file share folder (ir_prepare/incoming_mov) for transcoding into presentation formats, as described into the “transcoding scripts” section. the outputs from the transcoding section, a webm file, an mp4 file, and jpeg thumbnails and movie posters, are placed into a different folder (ir_prepare/ready_for_deposit) on the file share. previously, perl scripts were used to run the ffmpeg tools. currently, iffmpeg, an os x gui front-end for ffmpeg is utilized by the event capture group. all videos are deposited to the repository by either members of the event capture group or members of the vtechworks group, using the web upload interface built into dspace. figure 1. video workflow. subtitle workflow at present, only three videos produced by the library have been captioned.  captions are created in the webvtt format. webvtt is a text-based format with time codes and corresponding words to be displayed on the screen (webvtt, 2015) during each of these time segments. 01:49.025 --> 01:55.039 we will prevail we are virginia tech! the webvtt file for the initial captioning experiment was created with the microsoft caption maker web tool. in this tool, the user provides a link to a video file. the user can see the video in the web browser while typing in captions, and can pause, rewind, and play the video. after the video has been viewed and captioned, the user can download a webvtt file containing the time codes and titles. captioning is a time-consuming process, and could easily take half of a working day or more to caption a one hour video. when the second and third videos were captioned, the microsoft caption maker tool was temporarily unavailable. instead, a similar tool called amara was used. the amara tool is operated much like microsoft caption maker, and also has the very useful feature to save work still in progress to a user account.  there’s also an additional step in the amara process that allows the user to adjust the synchronization of the titles on screen by dragging bars in the web interface. after the webvtt file is created, it is uploaded to the dspace item via the web interface. the theme detects the existence of the webvtt file and displays captions, which can be disabled via a button in the playback interface. problems before this project was implemented, some videos had already been deposited into vtechworks. the embedded playback feature doesn’t work with these videos, because the theme customization requires that both an mp4 and a webm file are present to display the embedded player. the video files are still available for download, even though they can’t be played back directly on the item page. there is currently no plan to correct these pre-existing files. automating the process is not a very easy matter because the files are not in standard, identical formats – and preservation or original copies often are not locatable. pre-existing video items are instead occasionally encoded on an item-by-item basis into the required formats as desired. some of the problems that we have experienced are related to the byte range request feature which had been disabled in dspace, that we had enabled specifically for this project. recently, after our video encoding and deposit workflows had changed, and we experimented with encoding at a higher resolution, two larger video files were deposited into vtechworks where playback was broken. if videos aren’t encoded into the specific formats, with the moov atom at the front of the h264 file, this can cause problems with embedded playback. these particular files, though, not only had problems with embedded playback, they were also unable to be downloaded to client machines. the files were both larger than two gigabytes. the byte range request feature that was enabled in the bitstreamreader class of the xmlui in dspace depends on byte range code in the cocoon project, which uses a java int to store byte offsets (dspace chokes … 2015) another problem related to byte range requests involves the dspace statistics features. as part of the statistics subsystem of dspace, file downloads of each bitstream (content file) are recorded and tallied. with byte range requests enabled, each request – even if it is only for part of the file – is counted as a request. thus, streaming a large video file on a mobile device can result in many, many file visits, even if the particular file was viewed only once. this has not been a major issue for us, because at this time we aren’t concerned with the specific number of views on each file. we use the statistics for viewing general trends, such as the total change in repository-wide usage on a month-to-month basis. next steps a first priority is integrating video playback capability into the main dspace codebase.  video playback from dspace is a desired feature, but the functionality that we have built has thus far been limited to virginia tech’s customized version of dspace. this has two unwanted consequences. first, organizations that wish to stream video from their dspace repositories, but can’t dedicate resources to developing a solution are unable to try this method. second, each time a major upgrade of dspace is released, virginia tech must spend time integrating the solution and testing it again to make sure that it works as expected, which tends to take at least one or two days of development time. things may break, because testing isn’t as thorough as it was during initial development. we are using maven war overlays in dspace to lessen the problem during upgrades. with the maven war overlay, our custom code is stored in a separate directory path and used in place of the standard dspace code for customization. this makes it easier to incorporate our code without breakage during upgrades; however, we may miss new features or bug fixes that have been added to the standard dspace code. one possible complication of contributing this code to the dspace project is that the bitstreamreader component of this experiment is part of a specific dspace interface, the xmlui, aka manakin, which is based on cocoon, while a substantial number of organizations use an interface that is based upon the jspui, and other interface technologies are currently under evaluation. however, the vtechworks group has been restructured to follow agile planning processes, and is now committed to giving back to open source communities. our process for encoding the videos has gone from ad-hoc, automated scripts on an unofficial file server set up by the event capture group, to a people-driven process utilizing iffmpeg. it may be desirable for us to reach some sort of compromise between automation and control. dspace curation tasks may be one way of accomplishing this goal. a dspace curation task is a module of code that can be run upon request by staff on a specific item or collection to accomplish a well-defined goal. curation tasks often make use of external software services. examples of current curation tasks are virus scan, using clam av, and a metadata translator, using microsoft translate. a dspace curation task for video encoding could make use of ffmpeg and reasonable defaults to allow other institutions to encode videos in the proper format. ffmpeg, which is cpu intensive, would be required on the dspace server. subtitles for the deaf and hard-of-hearing are currently supported using webvtt files, but at the present moment, only two of the videos in the institutional repository have had subtitles created. library staff are currently working on a project to add closed captions to all library produced videos as part of a campus-wide effort. figure 2. caption initiative.   additionally, an exploratory app for the apple tv,  known as vtechworks videos, or vt tv, has been released (vtechworks video …, 2015). the app presents a selection of videos from the vtechworks institutional repository in a simple menu. the videos are streamed directly from vtechworks, using the bitstreamreader portion of the project described in this paper. the videos in the repository were encoded with settings considered appropriate a few years ago for display in a web browser or in mobile devices. seeing these videos on a large screen with 1920×1080 resolution, which has become relatively common, has spurred a desire to re-encode the archival videos into a higher resolution, higher quality setting. for the present time, dspace will serve as the main video presentation system for academic works created at virginia tech, with the exception of some video exhibits put together by the special collections department that are stored in omeka, while library marketing videos are placed on youtube and vimeo. the experimental vt tv app may evolve as an aggregation point to make all of these videos accessible from a single location. preservation copies of videos are currently on a large storage system awaiting deposit into an upcoming dark archive that is likely to be based on archivematica. while nearly all of these videos in the vt tv app are streamed from the repository,  one title is currently being streamed from amazon s3 to the apple tv. this was done to allow fast experimentation with playlists and content delivery networks without placing additional experimental code into the institutional repository codebase. there are a few items in the repository that have multiple videos; in the current implementation, only the first video is played back in the embedded window. therefore it makes sense to also add playlist support to the repository implementation. in addition to playlist support, the vt tv application utilizes adaptive bitrate streaming, so that an appropriate video is selected according to user bandwidth. if the bandwidth is not available for quality playback of the 1920×1080 video, the apple tv selects a video at a lower resolution. this feature is also implemented through m3u8 playlists, and is a likely candidate for future inclusion in the vtechworks institutional repository video playback features. links dspace [internet]. [retrieved 2015 dec 02]. duraspace. available from: http://www.dspace.org dspace chokes on large uploads (over 2gb) [internet]. [retrieved 2015 dec 02]. mello99(github username), virginia tech university libraries. available from: https://github.com/vtul/vtechworks/issues/91 event capture [internet]. [retrieved 2015 dec 02]. center for digital research and scholarship, virginia tech university libraries. available from: http://www.cdrs.lib.vt.edu/services/event-capture.html garbage scripts [internet]. [retrieved 2015 dec 02]. available from: https://github.com/keithgee/garbage_scripts/blob/master/event_capture/1process_mov_for_ir.pl placing “moov atom” at the beginning of an mpeg-4 video with ffmpeg [internet]. [updated 2011 april 28]. cut from the north. available from: http://cutfromthenorth.com/placing-moov-atom-at-the-beginning-of-an-mpeg-4-video-with-ffmpeg/ video.js player [internet]. [retrieved 2015 dec 02]. available from: http://videojs.com video playback in vtechworks [internet]. [updated 2012 sep 07]. digital library and archives, virginia tech university libraries. available from: https://blogs.lt.vt.edu/dlablog/2012/09/07/video-playback-in-vtechworks/ vtechworks – closed caption support [internet]. [updated 2012 nov 07]. digital library and archives, virginia tech university libraries. available from: https://blogs.lt.vt.edu/dlablog/2012/11/07/vtechworks-closed-caption-support/ vtechworks video: an apple tv app for our institutional repository [internet]. [updated 2015 nov 02]. marooned librarian. available from: https://maroonedlibrarian.wordpress.com/2015/11/02/vtechworks-video-an-apple-tv-app-for-our-institutional-repository/ webvtt [internet]. [retrieved 2015 dec 02]. wikipedia. available from: https://en.wikipedia.org/wiki/webvtt about the authors keith gilbertson (keith.gilbertson@vt.edu) is a digital technologies development librarian at virginia tech. liz mcvoy (lizmcvoy@vt.edu) is the digital media specialist on the creative services team at virginia tech’s university libraries. a combination of education, internships, and previous work endeavors shaped her love of video production, specifically video editing and graphic design. she uses her videography and design skills to promote the libraries’ spaces, services, and resources. a digital media specialist in the public relations and marketing department of the university libraries at virginia tech. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – linked data is people: building a knowledge graph to reshape the library staff directory mission editorial committee process and structure code4lib issue 36, 2017-04-20 linked data is people: building a knowledge graph to reshape the library staff directory one of our greatest library resources is people. most libraries have staff directory information published on the web, yet most of this data is trapped in local silos, pdfs, or unstructured html markup. with this in mind, the library informatics team at montana state university (msu) library set a goal of remaking our people pages by connecting the local staff database to the linked open data (lod) cloud. in pursuing linked data integration for library staff profiles, we have realized two primary use cases: improving the search engine optimization (seo) for people pages and creating network graph visualizations. in this article, we will focus on the code to build this library graph model as well as the linked data workflows and ontology expressions developed to support it. existing linked data work has largely centered around machine-actionable data and improvements for bots or intelligent software agents. our work demonstrates that connecting your staff directory to the lod cloud can reveal relationships among people in dynamic ways, thereby raising staff visibility and bringing an increased level of understanding and collaboration potential for one of our primary assets: the people that make the library happen. by jason a. clark and scott w. h. young introduction libraries and librarians exist in a space where vocabularies are structured and defined. rubrics, like the library of congress subject headings or the dewey decimal classification system, shape how we can talk about the people, places, and things in our libraries. yet, meanings and definitions are in flux and there are times when the language of the web moves faster than our classification systems. it was in this transitional space that our project began. at our library, we had conducted research on semantic web and structured data that focused on description and discovery of the things – the books, the digital images, the journals and articles – that make up the collections within our library, but we recognized a different opportunity. we asked, “what if we turned this practice of linked data into an application that helped in the discovery and description of the people in our library?” the “linked people” project was our answer to this question [1]. at its core, the project is about remaking our people pages by connecting the local staff database to the linked open data (lod) cloud. with this proof-of-concept project, we were looking to find another use-case for a graph-inspired data model using web-ready open source tools. the project led to a new data model, a new user interface (ui), a network graph visualization, search engine optimization (seo) improvements for our people pages, and our people pages encoded as five-star linked open data (lod). in this article, we focus on the code to build this library knowledge graph as well as the linked data workflows and ontology expressions developed to support it. this work includes: explaining how to encode and publish web-ready people profiles as rdfa html, json-ld, and rdf. discussing leading schemas for person and identifier entities, such as schema.org, foaf, and eduperson, and how to apply them. detailing and sharing the ontology and data structures used for expressing relationships among people, their organizational roles, and research topics. demonstrating how this structured data work impacts search engine optimization (seo) and discoverability for people profiles within commercial search engines. visualizing research networks based on a graph data model. we wanted to see what our library knowledge graph might look like and how it could generate new insights into how our people collaborate and how we present our skills and services to our users. we ended up learning even more about how data needs context and meaning and how people want to be understood and valued. literature review our literature review is focused on several broad themes that we weaved together in developing this practical implementation. there really wasn’t a direct corollary article that encapsulated this idea of applying linked data to a library staff directory. more specifically, we focused on the linked open data landscape in libraries, research into the patterns of html5 and structured data markup, use cases and implementations of knowledge graph data models, and the development of metadata standards for describing people. each of these components was essential in providing the frame for how we would link and construct our idiosyncratic library linked data application centered around library staff data. linked open data landscape in libraries libraries and cultural heritage institutions have been working with and around linked open data (lod) for a number of years. a distillation of that work is beyond our scope here, but two studies did help ground our understanding of the progress in our field. in their introductory study, goddard and byrne outline the benefits and obstacles to bringing library data into the linked open data field. they also put forward several ways for libraries to get involved in the practice of linked data and offering an encouraging call to experiment linking data in “small standalone collections” to develop “expertise and technologies within libraries” which we saw as a call to action (goddard et al. 2010). another primary resource to understand the lod landscape is smith-yoshimura’s international linked data survey for implementers in 2014 and 2015 which includes a rich dataset featuring which institutions have implemented or are implementing linked data; what linked data sources institutions are consuming, and why; what institutions are publishing, and why; and finally, barriers and advice from the implementers (smith-yoshimura, 2016). there are efforts within libraries and cultural heritage institutions to connect linked data descriptions of people to their activity. vivo is the project that immediately comes to mind and its original implementation was focused on mapping the networks and activities of life sciences researchers (devare et al. 2007). in vivo’s focus on people, their networks, and their activities, we took our inspiration for a library knowledge graph. patterns of html5 and structured data markup when the html5 specification was introduced, new markup elements and attributes were included that opened up new possibilities for how we might make the web machine-readable. in his study and tutorial of these possibilities, ronallo notes that “html5 includes a syntax called microdata, that allows web publishers to layer richly structured metadata directly into their web presentations.” (ronallo 2012). rdfa and json-ld are two other syntax encodings that enable structured data in html pages. in their analysis of structured data in the common crawl dataset, bizer et al. (2013) note that the growth of machine-readable descriptions of web content continues to grow. additional efforts like the web data commons provide a continually updated picture of the most recent usage and implementation of microdata, json-ld, rdfa, and other microformats. each of these structured data formats enables incredible granularity of description when paired with a vocabulary like schema.org and its full range of types that includes everything from creativeworks to places to organizations to persons and their actions. google even has an “introduction to structured data” that explains the hows and the whys of structured data noting that when “…information is highly structured and predictable, search engines can more easily organize and display it in creative ways.” our interest in description of the entities on our staff pages and the benefits of marking up our people, their networks, and their activities for better searching experiences convinced us that structured data was necessary work. use cases and implementations of graph models graph visualization and its technique, application, and evaluation have been present in library and computer science literature since at least the early 1990s, and in the last five years has especially risen in prominence with the availability of new technologies (beck et al. 2016). the application of graph modeling is widespread, covering ground ranging from the humanities to social science to computer science. notable examples are numerous. linked jazz, for instance, represents a high-water mark of humanities-based graph visualization (pattuelli et al. 2013; pattuelli et al. 2015). the relational data of social networks is a natural point of application for graph modeling, as shulman et al. (2015) show in visualizing library twitter networks. anthropological pursuits have similarly intersected with graph visualization (pasquaretta et al. 2014). research communities and the connections between authors have also been studied and visualized through graph modeling (nart et al. 2015), as has the world wide web itself (meusel et al. 2014). graph visualization is a powerful and flexible interdisciplinary tool for making visible the relationships among entities, especially when those entities are individual persons. for this reason, building a graph model was well suited for this project. metadata standards for describing people some of the earliest uses cases and vocabularies for the semantic web focused on people and their networks. in many teaching settings, the “hello world” of an rdf xml expression is an encoding of people and their social networks. the friend of a friend (foaf) ontology was one of the first means by which person entities could be defined and people could describe their connections. developers also realized it could be used to solve pressing problems like disambiguation of individuals and modelling organizational structures (graves et al. 2007). other person description schemas have been developed for implementation in software protocols like ldap (lightweight directory access protocol). the eduperson standard is one of these schemas designed to include widely-used person and organizational attributes in describing higher education. it provided us with another model for description of people and with required properties like edupersonorgdn (a distinguished, unique name representing the institution) reinforced our need to build organizational roles and descriptions into our application. while foaf and eduperson are useful in filling out models for descriptions of people and organizations, they tend to be more focused on internal or personal use of staff data. schema.org, an initiative for creating and supporting a common set of schemas for structured data markup on web pages has a more external focus on discovery in commercial search engines. in tracing the history of schema.org’s development, guha et al. (2016) note that the “goal was to provide a single schema across a wide range of topics that included people, places, events, products, offers, and so on… the idea was to present webmasters with a single vocabulary.” within this vocabulary, the person type remains one of the most implemented (guha et al. 2016). given its popularity, simple key/value structure, and impact on findability, the schema.org person type became the keystone component in our library knowledge graph. building a staff directory application with html5, rdfa, and linked data practices with our research complete and broad project goals defined around discovery in search engines, improving the browsing and searching of our staff data, publication of linked open data, and visualizing organizational and topic networks, we turned toward the work of building a new staff directory application. a linked data model one of our first tasks was to construct a data model for the reworked application. the data existed in a relational database and we thought we could remodel those relationships using a new data model based on typical linked data structures that connect multiple entity topics through lod vocabularies. this was a significant decision as it meant we would not be storing the data as a linked data triple store, but within a relational data model that would output linked open data. it allowed us to focus on how our existing technologies (sql) could accommodate linked data practices. in thinking through the model, we needed a means to structure and encode a link from a library person entity to a library activity or topic entity. this decision was largely driven by our goal of following a graph data model and the resource description framework (rdf) triple pattern. an rdf triple is a means of describing a relationship of things as three constituent parts: the subject, predicate, and object. in our library case, we were looking to describe two primary triple patterns: how a person (subject) is related to (predicate) other colleagues (object) and how a person (subject) has (predicate) a certain topic of expertise (object). a draft of how this ontology would work is below (figure 1). in this model, a library staff member is given a unique person id stored locally in our database. each expertise topic is also given a unique topic id stored locally. through rdfa markup, we can then articulate the relationship between person id and topic id. we provide context and meaning to each person id by connecting the id to that person’s name via schema.org markup and a google scholar profile url and an orcid url using schema.org’s “sameas” property. likewise with topic ids, we provide context and meaning by connecting the topic id to a name via schema.org and a dbpedia url using schema.org’s “sameas” property. through this process, each staff member is situated within a linked open data graph that expresses identity and expertise. figure 1. a visual representation of our ontology and where external vocabularies might appear (enlarge) structured data markup with html5 and rdfa after the data model for staff profiles was finalized, we turned to markup that was “of the web”, namely semantic html and rdfa. we focused on the staff profile pages that would display the richest expression of staff data that would be accessible to machines (figure 2). figure 2. web browser view of a staff profile page (enlarge) to the user, these pages perform as a typical web page allowing browsing and linking to related pages and people. but, in the html source, we encoded rdfa so that each staff profile is an expression of linked open data (figure 3). <main id="page" vocab="https://schema.org/" typeof="profilepage" resource="https://www.lib.montana.edu/people/about/139"> <meta property="about" content="http://www.lib.montana.edu/people/about/139#person"/> <meta property="thumbnailurl primaryimageofpage" content="https://www.lib.montana.edu/people/meta/img/photos/sara_mannheimer.jpg"/> <h2 class="mainheading" property="name">sara mannheimer people at msu library</h2> <div id="person" typeof="person" property="mainentity" resource="#person"> <ul class="aboutlist"> <li class="photo"><img property="image" src="https://www.lib.montana.edu/people/meta/img/photos/sara_mannheimer.jpg" alt="photo of sara mannheimer"/></li> <li class="infocontact"> <h3 property="name">sara mannheimer</h3> <link rel="sameas" href="http://scholar.google.com/citations?user=ytlns9saaaaj"/> <link rel="sameas" href="http://orcid.org/0000-0002-1433-6782"/> <p><span class="about-heading">title </span><span property="jobtitle">data management librarian [assistant professor]</span></p> <p><span class="about-heading">department </span><a href="/people/search.php?team=library+informatics+and+computing">library informatics and computing</a>, <span property="affiliation worksfor">montana state university (msu)</span></p> <span id="org" typeof="organization" about="http://www.lib.montana.edu" resource="#organization"> <meta property="name" content="montana state university (msu) library"/> <meta property="department" content="library informatics and computing"/> <link rel="sameas" href="https://www.wikidata.org/entity/q15255419"/> <link rel="sameas" href="https://www.worldcat.org/wcr/organization/data/56245"/> <span id="role" rel="employee" typeof="organizationrole" resource="#role"> <meta property="url" content="http://www.lib.montana.edu/people/about/139#person"/> <meta property="jobtitle" content="data management librarian [assistant professor]"/> </span> </span> </li> ... <li class="infocode"> <dl class="liaison" property="makesoffer" typeof="offer" resource="#offer"> <dt class="about-heading">liaison role(s)</dt> <dd><a property="category" typeof="url" resource="https://dbpedia.org/resource/us_government_documents" href="/people/search.php?liaison=government+documents">government documents</a></dd> </dl> </li> </ul> </div> </main> figure 3. machine-readable view of html rdfa for a staff profile (json-ld version also available for reference) (view as large image) the data appears on the page and in the source via our database, but we tagged the data with rich structured data. a couple of highlights in the rdfa include: the definition of the page type as a profilepage, a disambiguation of the person using a sameas equivalency and the orcid.org or viaf identifier, addressable nodes for machines using a resource attribute and a uri #hash, and a mapping of liaison expertise using the “offer” type. all of these encodings lead to a rich machine interpretation of the page where linked data entity expressions can be pulled and reused for linked data settings. and that might be the most compelling feature of this data model/encoding practice: the data exists in an agnostic format that can be transformed into alternative linked data expressions. for example, each staff profile page is also available as rdf/xml and json-ld. below is the same profile page as json-ld interpreted by the google structured data testing tool (figure 4). figure 4. structure data testing tool view of a staff profile page expressed as json-ld (enlarge) again, we worked this graph model into a relational database setting with an emphasis on encoding the relationships that we saw as necessary to show our research networks and linked expertise topics. a snippet of the data model is available below (figure 5). figure 5. snippet of person entity connected to an activity entity. the full data model is available in the corresponding linked-people github repo (enlarge) in the snippet, you can see the first few fields of the person entity and its identifier (person_id), but the more interesting work here was finding a way to express the action entity associated with the person. the “interact_action” is borrowed from the schema.org action type and in this case the table stores the data and uris to describe discipline and liaison relationships of our staff. you can also see how we created space for external vocabularies in the system using uris in the “object_uri” field that are mapped to the equivalent resource uri in external lod sources like dbpedia. we followed a similar pattern in a “create_action” table (not pictured here) to describe topical expertise relationships. impact we expected to see certain results from our linked data work and this project produced impact in the following four aspects of our staff directory: web discoverability, user interface design and local search functionality, linked data implementation, and graph visualization. web discoverability structured data markup allows commercial search engines to parse and understand web content, and to deliver that web content accurately to users. by structuring our metadata with schema.org tags, our individual staff profile pages experienced a visible improvement in the search engine results page of today’s leading search engine, google. we have included two examples below. in the first example, a search for our staff member michelle gollehon shows that her staff directory page appears in the third position, following a row of images and links to a facebook page and a linkedin page (figure 6). following the implementation of structured data, a similar search places our staff directory page in the first position, followed by facebook and linkedin (figure 7). figure 6. seo before linked people (enlarge) figure 7. seo after linked people (enlarge) likewise in our second example, a search for msu library faculty member sara mannheimer demonstrates a name ambiguity with sara mannheimer of sweden, an award winning fiction writer (figure 8). sara mannheimer of sweden is a more prominent figure, so google ranked the swedish sara mannheimer ahead of the montana sara mannheimer, even for searches originating from montana. following the implementation of structured data markup, the google search engine better understood sara mannheimer and her role as a faculty member at a montana university, and consequently ranked our staff directory page higher than the wikipedia page for sara mannheimer of sweden (figure 9). our structured data also produced additional metadata about sara’s role that now appears on google search results pages, visible in grey text. figure 8. seo before linked people (enlarge) figure 9. seo after linked people (enlarge) google analytics reporting shows an increase in key metrics for our staff directory when comparing the fall academic term of 2013 (august 26-december 13) with the fall academic term of 2014 (august 25-december 12), representing pre and post periods for this project. overall sessions increased by 280% (601 in fall 2013 to 2,288 in fall 2014) and overall visitors increased by 240% (530 to 1,801); inbound sessions from organic search increased by 436% (221 to 1,131), and inbound sessions specifically from google search increased by 441% (203 to 1,099). google analytics reporting from the fall academic terms in 2015 (august 24-december 11) and 2016 (august 29-december 16) show that the gains in sessions, visitors, and search traffic have been sustained (see table 1). fall term 2013 fall term 2014 fall term 2015 fall term 2016 sessions 601 2,288 3,518 3,531 visitors 530 1,801 3,178 3,146 sessions via organic search 221 1,131 985 1,176 sessions via google search 203 1,099 913 1,118 table 1. analytics reporting for montana state university library staff directory, showing increased visits overall and increased visits from web search following the implementation of linked and structured data on staff directory web pages. user interface design and local search functionality this project allowed us to link individual library staff with specific liaison and expertise areas. to provide end-user access to these connections, we developed a new user interface that now allows users to search and browse our staff directory via liaison area and expertise area. our staff directory homepage previously lacked these points of discoverability (figure 10), where now users are invited to “browse by liaison area” and “browse by expertise topic.” (figure 11). figure 10. staff directory homepage, before linked people (enlarge) figure 11. staff directory homepage, after linked people (enlarge) individual profile pages have also been enhanced to highlight the liaison and expertise areas of our staff (figure 12; figure 13). our new staff directory pages allow users to find and contact staff members in ways that are relevant for their own work. figure 12. profile page before linked people (enlarge) figure 13. profile page after linked people, showing connected liaison roles, expertise, and rdf machine-readable link (enlarge) linked open data implementation (five-star lod) a primary goal of the project was the release of our staff data as five-star linked open data. this star rating is a formulation by tim berners-lee from 2006 that outlines the steps to release open data. we had a goal of reaching the 3, 4, and 5 star ratings which state: 3 star – make data available in a non-proprietary open format (e.g., csv as well as of excel). 4 star – use uris to denote things, so that people can point at your stuff. 5 star – link your data to other data to provide context. the various expressions – rdfa, json-ld, and rdf/xml – of our staff profile pages allowed us to achieve 5-star linked open data. figure 14. rdf/xml of a staff profile page as an example of 5 star open data (enlarge) when we generated and finalized our rdf/xml expression for each staff profile and linked to external uris for context and description, our project became a linked open data resource at the 5 star level (figure 14). and at this point, we are in a position to move beyond being a linked data publisher to a linked data user that can query and reuse the triples for further enhancements and analysis. graph visualization the fourth and final impact we saw from the project was around visualization. more specifically, we were able to produce a new method for visually browsing our staff members and their relationships. the d3.js library provides numerous options for visualizing a graph network, and we selected the “d3 process map”, a web app that integrated with our existing php scripting and uses a multi-field json object to generate data that also fit well into our rdf arrays. using this d3 web app, we visualized the liaison and expertise areas for each of our faculty members (figure 15). this graph allows us to illustrate relationships with faculty members who share liaison and expertise areas. [2] in our example in figure 15, our faculty member jan zauha shares expertise in “information literacy” with five other librarians, and shared a liaison role with one other librarian. figure 15. graph visualization for a faculty member at the msu library (enlarge) by clicking or tapping on a node, additional information connects users with the dbpedia record for a topic or the staff directory profile for a faculty member (figure 16). figure 16. graph visualization for a faculty member at the msu library, showing node information (enlarge) this graph visualization serves as an extension of our standard browse interface. users can navigate through the network to discover new connections and relationships. within the context of the project itself, the graph represents a working proof-of-concept for the application of linked data visualization for a library staff directory. conclusion and lessons learned the work of this project was instructive and helped to build some new technical competencies for our library. in working through a linked data application and release into production, we became conversant in semantic web models and description practices, structured data markup, and network graph visualization. the smaller scope of a single dataset – our library people – gave us the chance to work in detail on a linked data model and build solutions towards specific goals. beyond these new skills, we recognized key lessons that any organization might see as takeaways from a similar project. first, think broadly about how these smaller projects can lead to future research. from this initial project, one of the writers was able to fashion a sabbatical project and a productive research thread. second, there are advantages to building and prototyping in a simple format, like json-ld. moving from the simple key/value format of json-ld into rdfa and rdf/xml is a much easier path because you can focus on sketching out the form of your expression without the complications of html nesting and or xml nodes. third, don’t underestimate staff interest in describing themselves. a secondary goal of this project was to build interest and generate participation from our faculty and staff in owning their descriptions and definitions. in our initial proof-of-concept application, we worked from an abridged vocabulary based on our subject liaison topics and expertise. as the project evolved, we moved to a library-wide invitation where all interested parties could assign self-defined expertise topics to enrich the data model. the invitation was highly successful and left us wishing we had involved staff earlier in the process. fourth, find standards that work with the grain of the web. our decision to use schema.org was based on our interest in seeing libraries as publishers in the web of data, a recognition that machine-understanding of our data could lead to more relevance for library data and improved discoverability through search engines. moreover, schema.org’s backing by commercial search engines and wide implementation made for an easier learning curve due to ample documentation and examples. even our choices to apply dbpedia, orcid, and viaf were in recognition of the weight of the one of the most populated and popular lod cloud data sources (dbpedia) and leading choices for researcher identification and disambiguation (orcid). and finally, we note that linked data practices and graph models can lead to better data models overall. the attention to detail needed to build and scope for disambiguation and connecting your data to external vocabularies enforces an intimacy with the data that isn’t always present in standard application development. while linked open data’s “killer app” may not yet have emerged, this project represents a contribution to the overall conversation around the practical application of linked data concepts and technologies. most importantly with this project, we were able to realize a technical implementation of linked data that underscores what our library is about–the people who make the library happen–and conveys that “aboutness” to machines and humans. notes [1] code for the “linked people” project is available under an open mit license at https://github.com/msulibrary/linked-people [2] demo code for the visualization is available in our project github repo at https://github.com/msulibrary/linked-people/tree/master/network references allemang, dean, and james a. hendler. semantic web for the working ontologist: effective modeling in rdfs and owl. waltham, ma: morgan kaufmann. 2011. beck, fabian, michael burch, stephan diehl, and daniel weiskopf. “a taxonomy and survey of dynamic graph visualization.” in computer graphics forum. 2016. bizer, christian, kai eckert, robert meusel, hannes mühleisen, michael schuhmacher, and johanna völker. “deployment of rdfa, microdata, and microformats on the web–a quantitative analysis.” in international semantic web conference. springer berlin heidelberg, 2013: pp. 17-32. devare, medha, jon corson-rikert, brian caruso, brian lowe, kathy chiang, and janet mccue. “connecting people, creating a virtual life sciences community.” d-lib magazine 13, no. 7/8 (2007). http://www.dlib.org/dlib/july07/devare/07devare.html graves, mike, adam constabaris, and dan brickley. “foaf: connecting people on the semantic web.” cataloging & classification quarterly 43, no. 3-4 (2007): pp. 191-202. goddard, lisa, and gillian byrne. “the strongest link: libraries and linked data.” d-lib magazine 16, no. 11/12 (2010). http://www.dlib.org/dlib/november10/byrne/11byrne.html guha, ramanathan v., dan brickley, and steve macbeth. “schema. org: evolution of structured data on the web.” communications of the acm 59, no. 2 (2016): pp. 44-51. http://queue.acm.org/detail.cfm?id=2857276 meusel, robert, sebastiano vigna, oliver lehmberg, and christian bizer. “graph structure in the web—revisited: a trick of the heavy tail.” in proceedings of the 23rd international conference on world wide web. acm, 2014: pp. 427-432. de nart, dario, dante degl’innocenti, marco basaldella, maristella agosti, and carlo tasso. “a content-based approach to social network analysis: a case study on research communities.” in italian research conference on digital libraries, pp. 142-154. springer international publishing, 2015. pasquaretta, cristian, marine levé, nicolas claidiere, erica van de waal, andrew whiten, andrew jj macintosh, marie pelé et al. “social networks in primates: smart and tolerant species have more efficient networks.” scientific reports 4 (2014). pattuelli, m. cristina, matt miller, leanora lange, sean fitzell, and carolyn li-madeo. “crafting linked open data for cultural heritage: mapping and curation tools for the linked jazz project.” code4lib journal 21 (2013). http://journal.code4lib.org/articles/8670 pattuelli, m. cristina, alexandra provo, and hilary thorsen. “ontology building for linked open data: a pragmatic perspective.” journal of library metadata 15, no. 3-4 (2015): pp. 265-294. ronallo, jason. “html5 microdata and schema. org.” code4lib journal 16 (2012). http://journal.code4lib.org/articles/6400 shulman, jason, jewelry yep, and daniel tomé. “leveraging the power of a twitter network for library promotion.” the journal of academic librarianship 41, no. 2 (2015): pp. 178-185. yoshimura, karen smith. “analysis of international linked data survey for implementers.” d-lib magazine 22, no. 7 (2016). http://www.dlib.org/dlib/july16/smith-yoshimura/07smith-yoshimura.html about the authors jason a. clark (jaclark@montana.edu) is an associate professor and head of library informatics and computing at montana state university, specializing in web development, metadata and data modeling, web services and apis, search engine optimization, and interface design. you can find him online at http://www.jasonclark.info and as @jaclark on twitter. scott w. h. young (swyoung@montana.edu) is an assistant professor and digital initiatives librarian at montana state university, specializing in user-centered design, web development, and community building with social media. read more on his website, http://scottwhyoung.com. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – libalerts: an author-level subscription system mission editorial committee process and structure code4lib issue 18, 2012-10-03 libalerts: an author-level subscription system patron requests for the ability to subscribe to their favorite authors so they could receive notifications when new titles are released, presented an opportunity for westlake porter public library to learn, to build, and to engage with patrons on the development of a new service. the library’s libalerts service, which launched in june 2012, was the culmination of a process that involved the development of a drupal-based website augmented with a hand-coded preprocess interface that addressed critical concerns for the effectiveness of the service. by matt weaver introduction: a “roll your own” approach this paper discusses the development process and launch of westlake porter public library’s libalerts service, an author-subscription service. in 2011, a couple of patrons at westlake porter public library had asked about such a service. apparently one of our vendors, dear reader, had been working on subject-level notifications. i am strident in my beliefs that libraries need to solve as many of these problems on our own as we can, so i investigated what was possible. during the process, i did find one vendor, engagedpatrons.org, which provides an alert service at a reasonable price. westlake porter public library, as a sirsi customer, used to have the “tell me when” functionality in the opac that would send notifications to patrons when new items came out based on their past checkouts. since the list could not be refined to eliminate categories that patrons did not want notifications for, and some other issues, our systems administrator turned off the service, an action that was hardly noticed by our customers. having been given the freedom to explore what was possible with the resources at my disposal, the savings of a few hundred dollars per year in fees to a vendor might not be seen as worth the investment of time and effort; but the experience that the library has in launching and sustaining a service on its own, from the coding experience, to the practice of involving the patrons who will use the service in its development are powerful motivators. also, such a service would provide test data that we could use to better understand our patrons’ reading choices. excited at the possibility of further in-house development of readers’ advisory solutions, i decided to proceed with a drupal-based solution after initial testing proved the viability of the project. the development process took around a year, with the project getting put on the back burner for months at a time. if ongoing efforts to improve the technology, normalize code and bring the modules into a neat package are successful, libalerts could be shared easily with other libraries. minimum viable product the goal for the initial development was to understand what tools i could find, and what skills i needed to learn, in order to get a pilot project up and running. this minimum viable product [1] would be used for initial testing to determine whether an actual service could be developed. if the basic service functioned in initial testing, a fuller version would be developed for a testing program using a group of patrons. accepting constraints: a development team of 1, a budget of 0 normally it would be customary to seek input from patrons about the value of such a service; but the most important question for me was whether it was possible. i had no budget for the project. if, in the process of seeking patron interest in such a project, we engendered an expectation that it would be forthcoming, that could be problematic. in order to provide such a service, i would need publishing data. running reports from the ils could prove time consuming on an ongoing basis. also, due to budget cuts in recent years, all departments have been running lean. if possible, i wanted to do everything i could to make the service fit in our operations as neatly as possible, so i asked the library’s cataloger and the acquisitions specialist about the procedures for ordering. the latter explained that she downloads shell marc records from publishers that are loaded into the ils after an order is completed, and then those marc records were simply deleted from her computer. the cataloger arranged for her to download the marc files into a shared folder using a desktop shortcut, and instead of deleting the files, she would leave them there for me to process for libalerts. she also offered to add suffixes to file names to indicate what dewey category, format, and age group the files were for. this process didn’t change her download process at all. based on the expectation that the service would appeal to fans of prolific, mainstream authors, the library’s systems administrator ran a set of reports of bestselling titles in marc format, which i uploaded in order to populate the site with a good representation of popular authors. also, for a time, i uploaded a large number of marc records without considering whether patrons would want to subscribe the authors in those records in order to receive the notifications when their new titles were published.. a maintenance task in the future will be to delete content/authors who have been in the system for a certain amount of time and have no subscriptions attached to them. drupal the library’s public website and intranet are built on the popular content management system drupal (version 6) in multi-site setups, and both sites feature subsites. libalerts functions as a subsite of the main website i explored available modules and quickly found that i could piece together suites of contributed modules into a mechanism for patrons to subscribe to authors and receive email notifications when new content by those authors was added to the site. libalerts are content types built with the powerful content creation kit (cck) module, and feature custom fields for a book’s isbn, which is applied as a variable in a template to complete an img element’s src attribute that displays syndetics cover art for each title (see example 1). i created a custom template to “theme”—drupalspeak for changing the layout—the libalert content type. figure 1. node-libalert.tpl.php and sample node. critical modules notifications — provides subscription and notification functionality. the taxonomy notifications module, which is part of the notifications group of modules, allows users to subscribe to taxonomy terms, which is how i stored author names. sms email gateway/sms framework — enables the transmission of text messages using email gateways. sms email gateway used to be part of the sms framework but is now its own project.[2] however, the sms framework module is better supported and is still under development. marc module — allows drupal nodes to be created by importing marc record data. upon finding another suite of modules that would provide the ability to send text messages, and discovering that the ohio public library information network provides libraries with sms capability for free, i was eager to include text messages as a possible mode for alert delivery. to get text messaging to work, however, required hacking the sms email gateway module. initially, this was for diagnostic purposes, in hopes it would give me insight to a fix for the module. essentially, i hacked the module to force the gateway address for the carrier. this hack appears in a thread that i started on the drupal forum [3]: in sms_email_gateway.module, line 194, i changed <?php $to = $number .'@'. $options&#91;'carrier'&#93;; ?> to <?php $to = $number .'@'. 'sms.yourcarrier.com'; ?> i have not tested the patches in the thread noted above, nor explored the latest version of the sms framework to see if it will handle text messaging. this hack remains a part of the service, but redressing this is a goal of future improvements to the site. simulating taxonomy search drupal’s search system does not search taxonomy terms. given that these authors’ names are stored as a taxonomy terms, i needed a mechanism for search. the best that i could do was to use a setting in the views module. i exposed the taxonomy term filter, in effect creating a search box. then, i set “exposed form in block” to “yes” creating the search block. figure 2. exposed taxonomy term filter.. proof of concept: staff testing staff testing began with an initial group of subjects from the library’s technology team, which i lead. i had them sign up, add authors and subscribe to them. those with cell phones who were willing to receive texts as part of the testing process were enlisted. i found as many types of cell phones and smart phone operating systems as possible. after this initial testing of the core alert-transmission system, i expanded staff testing to any employee willing to participate. testing focused on the basic functionality of the site: subscribing to authors, and transmission of alerts triggered by the uploading of data from marc records into drupal nodes. core functionality, for both email and text message recipients, proved successful. post proof of concept: the survey the library was planning to run a series of short surveys in the near future: out of fear of survey fatigue, i was permitted to run a short survey on the website and as a handout, for up to two weeks. the survey aimed to capture interest among readers in such a service and which authors respondents would like to subscribe to. while the number of recipients was only 50, interest in the service was very high (see below). the authors that recipients would subscribe to read like a bestsellers standing order. not every reader follows authors so closely, but if a core of voracious readers would be satisfied with this service, libalerts would certainly hit that sweet spot of mainstream fiction. forty out of fifty survey recipients added a total of 234 authors for an average of 5.85 per respondent. but several respondents also added phrases like “plus many more,” or “hundreds of others” in their list of authors. questions responses how likely would you be to use this service? somewhat likely 6 very likely 8 would definitely use it 36 how many books do you read a year? 3 or fewer books per year 1 4 to 6 books per year 1 7-9 books per year 2 10-12 books per year 2 more than 12 books per year 43 who are your favorite authors? left blank 10 user entered value 40 average submission length in words (excluding blanks) 13.68 table 1. survey responses. dealing with the data both from the staff testing phase and the survey revealed some major issues that i would have to address before turning this into a real service. a pre-process interface after the concept of the site had been proven, the initial development process revealed a couple of major concerns that had to be addressed for the actual service to work. first, for an alert to be triggered, the author name that appears in a marc record must match the author name in the libalerts site exactly. if a patron would fail to receive an alert because of a dropped middle initial, then the service fails. the marc records that we receive from publishers can include variations in a name form, or misspellings. also, authors may have very similar names: david a. adler and david d. adler both write children’s books, but are different authors. i found nothing in drupal that would allow comparison of marc records to the contents of the database. the second concern was the size of the site. libalerts was never intended to duplicate all content in the catalog, but all marc records that the library receives from publishers would have to be processed in order to catch every possible subscription. surplus content would make it harder for users to find desired authors. so, i only wanted to upload marc records for authors to whom at least one patron had subscribed. as libalerts is designed to be as automated as possible, i needed to solve these problems programmatically. file_marc the preprocess interface reads and writes marc records using the file_marc library, which is installed via the pear package manager as part of a server’s php installation. this library can be used with drupal’s marc module, which is discussed later in detail. selectively adding content because libalerts has the means for users to add authors not included in the site, and the site was pre-populated with hundreds of authors, i decided that the marc records should be analyzed so that only records with authors for which a subscription was attached should be included. the example below shows author values gathered by a sql query of the database and placed in an array. another array is created from authors in the marc record, and each item in that array is compared against the array of existing authors: /* *get array of authors in libalerts that have subscriptions associated with them */ //get term names for authors that have subscriptions attached. $subauth = mysql_query("select distinct tid, name from term_data inner join notifications_fields on notifications_fields.value = term_data.tid where notifications_fields.field = 'tid';"); /* * displays all authors in a marc file -needs form input to choose file */ $newbooks = new file_marc(‘[path to directory]' . $filename); while ($record = $newbooks->next()) { $objkey++; $newauthors = $record->getfields('100'); if ($newauthors) { foreach ($newauthors as $field) { $author = substr($field,9); mysql_data_seek($subauth,0); while($rows = mysql_fetch_array($subauth, mysql_assoc)){ $pairs[] = implode('|',$rows); foreach($pairs as $pair){ $final = explode('|',$pair); } if(in_array($author,$rows)){ $newrecords[] = $objarray[$objkey]; echo '<span style="color:red;font-size:20px;border: dashed 5px red;">match found for ' . $author . '</span>'; echo '</div>'; } the levenshtein distance the process of comparing two strings– the author names as they appear in the marc record, and the existing name in the libalerts database– involved querying the author names in the marc records and creating an array of author names in the database to which a site member had subscribed. in php, the levenshtein function calculates the “the minimal number of characters you have to replace, insert or delete to transform” the first string into the second string. this measurement is the levenshtein distance, or the edit distance. [4] in the following example, author names from marc records for which there is not an exact match in the database are compared against the database with the levenshtein function to see if there is anything close. the code below successfully detects name forms in marc records that are off by as few as one character when compared to the form of the name in the database. /* * calculate the edit distance with levenshtein() * example from http://us2.php.net/manual/en/function.levenshtein.php */ //evaluate edit distance // no shortest distance found, yet $shortest = -1; // calculate the distance between the input word, // and the current word $lev = levenshtein($author, $row); // check for an exact match if ($lev == 0) { // closest word is this one (exact match) $closest = $row; $shortest = 0; $newrecords[] = $objarray[$objkey]; break; } // if this distance is less than the next found shortest // distance, or if a next shortest word has not yet been found if ($lev <= $shortest || $shortest < 0) { // set the closest match, and shortest distance $closest = $row; $shortest = $lev; } if ($shortest > 0 && $shortest < 5) { echo '<fieldset>'; echo '<p>name as it appears in marc record: <strong>' . $author . '</strong></p>'; echo '<p>'; echo '245 field from marc record: <strong>' . substr($record->getfield('245'), 0) . '</strong>'; echo '</p>'; echo 'no match in the libalerts database, but <strong>' . $author . '</strong> is close to: <a onclick="return popup(this,\'notes\')" title="click to check the author in the catalog" href="http://catalog.westlakelibrary.org/uhtbin/cgisirsi/x/0/0/5?search_type=browse&srchfield1=au^author^authors^author processing^author&searchdata1=' . str_replace(',',' ',$final&#91;1&#93;) . '">' . $closest . '</a><br />'; potential intended authors from within libalerts are presented and the code below shows a javascript function that changes the value of the author name in the marc record from the publisher to the version of the author name that exists in the site’s database. function setauthor(el) { var author = el.innerhtml, id = el.id.replace(/^closest_/, ""); if (console) { console.log(author); } // window.alert(author+id); document.forms["myform"].elements["auth_"+id].value = author; return false; } echo "click name to add <span style='background:gray;border:outset 3px black;' id='closest_{$fieldkey}' onclick='setauthor(this)'>{$closest}</span>"; for the development of the full libalerts service, another consideration was age groups. if possible, i would like to separate authors by the age group they write for, but without pegging a writer who writes across age groups into one category. even though the records that the library’s acquisitions specialist downloaded were shell records, they contained enough information to sort authors into adult, teen and juvenile categories via drupal’s taxonomy system. i created hierarchical taxonomies using the fixed values of the 947f and 947h fields, which wppl uses for local holding information. i placed these values within hierarchical taxonomies beneath top-level terms of adult, kids and teen, which were used to create author-browse pages for each. for instance, “amys”, a local holding value for our adult mystery collection, is a child of the “adult” term. however, capturing this information required another hack of a contributed module. the drupal marc module was created based on a local institution which uses the 852 field in marc records for local holding information. the marc records that we download from publishers use the 947 field, which was not able to be captured by this module. i had to add the 947f and 947h fields to the marcbib and marcnode modules, which are part of the marc module package. testing: a focus group from the wild for live testing i had six patron volunteers (a pair of staff members joined in as well). testing consisted of having participants find and subscribe to a specific author (although most subscribed to many others, too). when i saw that everyone had subscribed, i would create a libalert and email them to tell them that it had been sent. in all cases, alerts were received; no patrons opted to receive messages via text messaging. one test subject became incredibly engaged, sharing her experience in detail, even catching typos in my “how it works” page. of the six, four completed all tests and a final survey. either in direct email communication or in the survey, my test group revealed usability issues, and other problems that i cleared up before launch. limitations due to the popularity of ebooks, i wanted to include overdrive data. we do not receive marc records in a timely manner; sometimes it is weeks or months after the titles are available. while overdrive is promising new marc services[5], the one that appears to be sufficient has a cost associated with it. yet reports generated within overdrive’s content reserve module often include insufficient metadata to use in the libalerts processes. as of 7/30/12, overdrive has launched its developer portal, which includes a metadata ap[6] that may allow me to include ebooks and digital audiobooks. the public’s response at the time of writing, the library has promoted the service with signs, small bookmarks placed in books by popular authors, and a press release; but the edition of the library newsletter notes that features libalerts had not gone out. the press release was not picked up by the largest paper in the area, but it did appear on that publishing company’s websites which combine local content from the plain dealer and the smaller sun news, which did pick up the story. tutorials and an introductory video have been shared on youtube, twitter and facebook. response has been steady but slow, gaining a new user or two a day. new users who have received alerts are returning to add more author subscriptions. current users are receiving their alerts and returning to add more subscriptions. the average is around 10 subscriptions per user, with one user having subscribed to nearly 75 authors. some people have created accounts but added no subscriptions. through drupal, i contact these users to direct them to tutorials or invite them to contact me directly for help. slowly, those users add a subscription or two. mobile issues i felt it was important to have a mobile site for libalerts so that patrons who received alerts as text messages would have a dedicated interface that would allow them to place holds. also, patrons could easily add a subscription wherever they are learning about new books/authors that they might like to follow, but the mobile interface in a multisite setup posed problems. ultimately, i used the themekey module for theme switching. the themekey properties module allows theme switching to be triggered on operating system or browser type. this method does not allow for tablet users to choose the desktop version of the website, which is a usability issue. figure 3.mobile view. figure 4.theme switching rule. future improvements finished products are for decadent minds. –isaac asimov normalized structure, no more hacks the current system, while functioning as designed, can be improved. one problem is that some of the modules that the site relies on have not been written for drupal 7. so i am exploring alternatives that will be likely to ensure the future growth of this system. these alternatives can clear up hacks of two modules that i had to put into place in order to get the functionality that i wanted. these two hacks are, personally, major maintenance nuisances, which is reason enough for solving them. they also pose a major obstacle to sharing the site with other libraries. moving forward, i will have to ditch the marc module altogether. it does not include fields for marc 700 fields, the 508 and 511 fields, etc. as the marc module was created with a local institution’s usage in mind[7], and only reflects books, it is not sufficient for movies or music, which i plan to add. a better option will be to output xml records and import them via drupal’s feeds and feeds xpath parser modules. using the file_marc library, to effect this change, i will have to change two lines in my pre-processed interface code from: $newrecord->toraw()); to $newrecord->toxml()); the sms_email_gateway hack will be harder to deal with. at this point, continued development of the sms framework module may be the best solution to having text messaging in the system. if text messages prove popular, there are limitations on the gateway provided by the ohio public library information network: each user can only receive 5 texts per month. for the site’s heavy subscribers of mass fiction, they could easily hit that limit; as i plan to add the ability to subscribe to actors/directors of movies in the next phase of the site’s development, that limit could be insufficient for a large number of users. i might have to see if the library can come up with money so that we can access a commercial gateway for text messaging while there is a downside of not having it integrated into the ils, i’ve never found any ils, even the newer sexier versions, to be great sources for readers’ advisory help. readers’ advisory is the direction i intend to take the site. drupal offers modules for rating content, and i can allow people to write reviews simply by enabling comments on the libalert content type. i will work with active users to see what they want from the site. sharing drupal features: i am trying to see if the alert functionality in the current site can be performed by the rules module and associated modules. rules is compatible with the features module. if the core functionality of my libalerts site can get saved as features, it will be much easier to share it with other libraries. another approach, especially if a large number of libraries would be interested in this, would be to create a distribution[8], which is a specialized drupal package, complete with all of the necessary modules, themes, libraries, etc., for a given site. sirsi web services i am planning to take training in sirsi web services(sws). at the very least, using sws and the ils authentication module[9], i would be able to add user authentication based on the ils user library card number and pin. pros and cons of roll-your-own while i am not anti-vendor, in my work i am driven by a belief that i am not doing my job if i don’t try to solve problems myself. this opinion scales, when considering my industry. the benefits of roll-your-own for the library include control control over the system, which means as long as we keep the server running, the service will be there. because we control access to the system, we can learn from it and build on the existing service. we can make improvements based on our local usage: this responsiveness is something of great value that is lost when dealing with vendors. the library can engage with the community of users. i collaborated with a small test group of patrons, but plan to open future development to active users. in a small way, this is building community, which i believe is essential to the future of libraries. control also means taking responsibility for patron privacy. if there is a data breach, for instance, we can’t grouse about the vendor. this is not a “con” of roll-your-own, by the way; it is a reality, one that is in line with our traditional library values, and that–in my opinion–is quickly forgotten when signing with vendors. the stress that accompanies control is substantial: this is my baby, and i have no vendor to blame if something fails. but accountability should never be considered a “con.” cost by using open source software, there are no licensing costs. the investment of time in this development will be paid off over time. we are not paying for access to a text messaging gateway at this time. if the current free gateway proves insufficient for actual usage, i will explore other options, including commercial gateways. if future development involves sirsi web services, the library already has the required subscriptions in place: libalerts will not add to those costs. collaboration within the library, this project has been a collaboration. the acquisitions specialist who has helped with the workflow for the marc records has taken an active interest in the site, and has been sharing more information that she finds relevant to the service. because we built this system on open source software, i can share it with other libraries. an opportunity for this level of collaboration is an exciting prospect. the disadvantages of roll-your-own cost there is cost advantage to roll your own, but, if all you want is the functionality described above, there are vendors who will provide it. that cost may be less than the staff cost of building the service from scratch. however, by sharing this project in the means described earlier the investment of staff time will be reduced. time a major problem with going roll-your-own for most projects is: where do you find the time to do it? the development process for libalerts was interrupted frequently. as the only person doing web work at the library, if my skills were needed elsewhere, libalerts got put on the back burner. because of my involvement with ebooks, libalerts development completely stopped from october 2011 until march 2012. any particular diversion from the development team’s project can kill momentum, interest and support. stress libalerts is “my baby.” throughout development, testing and especially at launch, the specter of “did i think of everything” was constant and weighed heavily, and has led to some rather obsessive thinking. insufficient skills i am self-taught in all technological aspects of my job. perhaps there are solutions to problems that i faced that are outside my skill level. as a broader discussion for any project, it may be difficult, expensive and, and time-intensive to add necessary skills to bring any project to fruition. conclusion libalerts began with requests from patrons. the work needed to understand the motives behind those requests, and how to satisfy them, not only was an invaluable learning process, but also resulted in a viable product. the roll-your-own approach to its development presented many problems whose solutions will serve me and my institution in future development efforts. i believe that libraries should hire vendors to do things we cannot do; but we should try to do things ourselves first. we talk the talk of promoting our institutions as places of personal transformation through lifelong learning. when it comes to developing new services to meet expressed demands within the community, we need to walk the walk, partnering with our patrons to transform our own institutions. about the author: matt weaver is the web librarian at westlake porter public library (westlake, oh). his email address is matt.weaver@westlakelibrary.org. you can connect with him at facebook.com/mattrweaver and twitter.com/mattrweaver. notes [1]. http://en.wikipedia.org/wiki/minimum_viable_product [2]. http://drupal.org/project/sms_email_gateway [3]. for a full discussion of the issue and some possible patches/solutions: http://drupal.org/node/1222038 [4]. http://php.net/manual/en/function.levenshtein.php [5]. http://overdriveblogs.com/library/2012/06/21/marc-records-options-expand-with-ebibliofile/ [6].https://developer.overdrive.com/ [7]. email communication: “all the school oriented automation systems that i work with keep holding code info in the 852 field.” send by andy austin to matt weaver on 10/31/11. [8]. http://drupal.org/documentation/build/distributions [9]. http://drupalib.interoperating.info/ilsauthen subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – advancing arks in the historical ontology space mission editorial committee process and structure code4lib issue 50, 2021-02-10 advancing arks in the historical ontology space this paper presents the application of archival resource keys (arks) for persistent identification and resolution of concepts in historical ontologies. our use case is the 1910 library of congress subject headings (lcsh), which we have converted to the simple knowledge organization system (skos) format and will use for representing a corpus of historical encyclopedia britannica articles. we report on the steps taken to assign arks in support of the nineteenth-century knowledge project, where we are using the hive vocabulary tool to automatically assign subject metadata from both the 1910 lcsh and the contemporary lcsh faceted, topical vocabulary to enable the study of the evolution of knowledge. by mat kelly, christopher b. rauch, jane greenberg, sam grabus, joan boone, john kunze and peter m. logan introduction the 19th-century knowledge project is building an extensive, open, digital archival collection to support the study of knowledge and its transformation. the project uses both the contemporary library of congress subject headings and the 1910 lcsh for topical representation. the use of both ontologies provides more comprehensive access to the material and facilitates the study of knowledge over time [1]. one of the most important early development decisions relating to the publication of the project was the application of ontologies with persistent identifiers (pids) for subject representation [2]. this was necessary even in the very first stages of the project because we planned to use identifiers as the functional reference for both users and in coding, templating, and implementation. the challenge is that many ontologies do not have pids, particularly historical terminologies. additionally, not all approaches for persistent identifiers are suited for ontologies. many pids have heavy requirements and costs associated with their creation, formatting, and resolution. handles and dois impose annual and/or per-identifier fees, and both require use of their centralized registration and resolution mechanisms. dois have strict metadata requirements that are hard to meet during development and pre-publication stages.we chose the ark (archival resource key) as our identifier scheme for historical ontologies because the requirements and costs are relatively low [3]. any institution can create an unlimited number of arks without paying for the right to do so. institutions can also assign arks to any research objects they wish. this can safely be done in the planning stages well before the object is created or published since planners can keep their arks as private as their plans, and resolution does not need to be enabled until the objects are unveiled for wider use [4]. archival resource keys (arks) have been used as persistent identifiers (pids) for vocabulary projects, such as the periodo, which has curated a canonical dataset of time periods [5], and the yamz.net crowdsourced metadata dictionary [6]. these exemplary projects, however, do not investigate the suitability of arks for historical ontologies. arks and the 19th-century knowledge project this project began with the attempt to expand the utility of the 1910 vocabulary by building on an existing vocabulary server, helping interdisciplinary vocabulary engineering (hive) [7], and assigning unique identifiers to both the ontology itself, and its terms. we recognized that to refer to terms uniquely would require a unique identifier. the problem that we encountered was twofold. first, the electronic structure of the repository we were trying to describe (individual entries across four historical editions of the encyclopedia britannica) was itself under development, and second, the full transformation of the 1910 lcsh to the skos format was a multi-step process, given historical conventions of terminology design and those that align more with computational approaches guiding today’s work. the 19th-century knowledge project uses historic editions of the encyclopedia britannica to build an open digital collection of the encyclopedia’s contents. automatic metadata generation is being used for assigning topical metadata. a set of 130,000 entries are being processed first through the stanford name entity recognizer [8], followed by the application of skos enabled versions of both the 1910 lcsh vocabulary and the lcsh fast topical vocabulary. because multiple passes were required to prepare each encyclopedia entry with metadata for discovery and access, we needed a flexible pid that we could assign to terms at the beginning of a multi-step workflow during which the terms would evolve considerably. while we completed this work, we also wanted to resolve those pids for ourselves before going public with them. these requirements steered our choice of pid to the ark scheme [9], and the remainder of this article describes how we implemented arks for the 1910 lcsh, from minting, to resolution, to metadata. minting arks there is no prescribed way to create and maintain arks, and the method we chose is based on the popular open source noid (nice opaque identifiers) [10] utility, which includes a command line script and a lower-level module that runs on linux, mac, and windows systems. this utility provides an easy way to mint (generate) unique, opaque strings ending in a check digit that can be used to detect the most common transcription errors. the outputs of the noid minting function are used during ark creation. noid also offers optional resolver and metadata management functions, but they are independent of minting – one can use the latter without the former or vice versa. we chose to use our own resolver and metadata database. uniqueness is a concern for any pid scheme. relational databases make local uniqueness easy to ensure when database tables are first set up. that provides a failsafe behind the noid minting function’s own assurance of uniqueness. global uniqueness is then ensured by prepending a globally unique “prefix” to the locally minted string. in the usual ark scenario that “prefix” is formed by appending a naan (name assigning authority number) to the “ark:/” label. a naan uniquely identifies a managed assignment workflow, typically that of a cultural heritage organization. large organizations often have multiple workflows, for which they delegate assignment via a prefix formed by extending their naan with a short, unique string assigned to each separate workflow. that extended prefix is known as a “shoulder” [11]. just like the naan-based prefix, the shoulder-based prefix is globally unique and so any ark starting with such a prefix will be globally unique, provided the workflow keeps the assignments locally unique. for all its virtues in the face of change, opacity takes a toll on usability. there are a few cases where slightly reducing opacity is considered safe and useful, and these are signaled by arks bearing one of four well-known “shared naans” that aren’t associated with any particular organization. two of those naans are for test (99999) or example (12345) arks that aren’t “real” enough for link-checkers to bother with. the other two are for identifying things that are either vocabulary terms (99152) or “agents” (99166), both of which are considered immutable properties of such things. while 99152 doesn’t exactly scream “vocabulary term” to the average person, receiving software can usefully recognize these four naan and leverage the semantics without having to fetch any ark metadata. both a per-organization naan and a per-organization shoulder under one of the four shared naans are globally unique prefixes that must be requested [12]. noid is an open source perl package available via cpan (comprehensive perl archive network). after installing the cpan package manager [13], we installed the noid package and its associated dependencies. for our debian linux environment, this required the installation of the perl berkeley database wrapper (libberkeleydb-perl) [14] to support the berkeleydb module dependency [15]. we executed the following command to create the identifier database, which creates a directory called noid and places the ark minter database within. $ noid dbcreate b4.reedeedk long 99152 cci.drexel.edu mrc/lcsh1910 the parameters of this command have the following significance [16]: b4 is the shoulder (prefix) for the lcsh1910 vocabulary and reedeedk is a template for the ark, in which r requests quasi-randomly generated identifiers, e requests an “extended digit”, one of { 0123456789bcdfghjkmnpqrstvwxz } (“betanumeric”, which is the set of alphanumeric characters minus vowels and the letter “l”), d requests a pure digit, one of { 0123456789 }, k requests a check character to detect transcription errors, long is a declaration of the intended longevity of an identifier and establishes extra restrictions to ensure no identifier will be generated twice. other options are short (for generating identifier strings that may be re-used) and medium, the 99152 is the naan associated with the minter, which requests that it generate strings prefixed with 99152/b4), and the final two arguments, cci.drexel.edu and mrc/lcsh1910, are the name mapping authority and a string chosen by the person setting up the minter to identify the project that will be operating it, respectively; these effectively become minimal minter provenance metadata [17]. thus established, the minter database represents that initial “state” of the minter. that state changes after each minting operation so that the minter does not lose track of what it has minted already. next, we issued the command to generate our first 30,000 ark strings, $ noid mint 30000 there are approximately 29,000 terms in the 1910 ontology, so this became our pool strings with which to create arks. note that noid distinguishes the act of minting an identifier (generating a candidate string) from the act of creating an identifier (associating that string with a thing). from the pool of strings “waiting” to be assigned, some will become identifiers and some will be discarded, which is by design. moreover, among those that become identifiers, some might never be shared beyond the confines of the database, some might never be shared beyond immediate team members, and some might be published (whether deliberately or accidentally) widely on the internet. the last of these steps — publication, not creation — is the least revocable and thus the most consequential. therefore, we create arks in a less committed manner than we publish them. when the minting process is complete, the minter state, stored as a berkeley db database in the noid directory, reflects that 30,000 strings were minted. the database can be manipulated using any berkeley db client api (perl, c, python, java, etc.) or the noid utility, and additional arks can be generated against the database in the event they are needed. the database is filesystem-based (rather than server-based), so it does not require an external server and can be ported to any machine that has the appropriate libraries installed. the noid directory itself can be renamed and moved aside so that a new list of arks can be generated according to different initial input parameters (shoulder and naan). to expedite the assignment of arks to the related subject terms, rather than manipulating the minter database, we used a shortcut. when the mint command is given, the arks are also written to standard output. to capture them, we used the “tee” command [18], $ noid mint 30000 | tee arks.txt in the future, we also plan to integrate the functionality of the noid utility or the associated library into our record creation procedure for vocabulary terms so that the arks are generated when the term entry is created. figure 1. minting arks using the noid utility. ark resolution the global ark resolver, n2t.net, contains redirection rules based on naans or on shoulders for institutions that publish their arks using n2t.net in the hostname. this mechanism relies on rules that recognize incoming ark prefixes (naans or shoulders) and sends the arks to the target institutional resolvers associated with those prefixes. institutions generally register a naan for this purpose, which we have obtained, but in our case, we saw an advantage to also registering a shoulder on a particular shared naan [19], 99152, because it signals to specialists (e.g., software) that such arks are for “terms”. while this choice to disclose semantics makes for a less opaque identifier, it does not threaten longevity since the property of being a “term” is immutable and can be expected not to change. in order to associate the 1910 ontology with our institutional server, we registered a shoulder “b4,” which we designated to indicate ontologies under our control. the “b4” string is short and opaque: short to help reduce overall identifier length and opaque to shore up overall longevity. this string also conforms to the primordinal (“first digit betanumeric”) shoulder convention, which calls for a sequence of one or more betanumeric characters ending in a digit. advantages of primordinal shoulders include that there is an infinite number of them possible under any naan [20], and the character repertoire restriction preserves error detection via the check digit. persistence is served by limiting the potential for semantic degradation that will naturally occur as institutions are reorganized or commonly used names change. when in the context of a uri, the ark will be sent to the host through an http (or similar protocol) request. if the ark is received by the webserver at n2t.net [21] it will redirect based on the b4 shoulder of the object name of the term to an authoritative server for b4. this might take the form https://historical-ontologies.info/ark:/99152/b4cc2g3s [22]. to design our resolver, we used nginx, python-based django, and django’s rest frameworks with mysql and reactjs for the interface. the project is currently hosted on resources allocated from the extreme science and engineering discovery environment (xsede) project. xsede allocates compute resources to (us-based) applicants in pursuit of reproducible science [23]. the nginx webserver at https://historical-ontologies.info acts as a secure reverse proxy [24]. it receives the http request and parses it according to the pattern described in the nginx configuration file. this, in turn, calls the corresponding matching function in the react node.js application running as a background using pm2 (process management model) [25]). a react component then passes the ark id internally to the api endpoint exposed by the django rest framework to retrieve and display the associated record contents. in future work, the ark will be checked here for transcription errors and validated against the existing database of arks created by the minting process. the noid utility provides hooks so that it can be called on to do this work [26]. after all tests have passed, the part of the function that renders the view proceeds, and the term represented by the ark and its associated metadata is returned according to the corresponding view template. the term itself is displayed in the context of the ontology and resembles a typical term entry for an ontology. figure 2. the term “abbeys” displayed with associated ark and links to hive data. ark metadata and transformation part of the usefulness of the ark ecosystem is that it encourages stewardship of digital objects by encouraging metadata that speaks to institutional commitment as well as to object description. it is possible to add inflections to arks to query the metadata associated with it. (under development are extensions that would allow inflection arguments to request metadata in arbitrary format [27].) this metadata might be in any format or of any type but adding the ‘?info’ inflection is expected specifically to include an assertion of the organizational commitment to persistence. our implementation generally follows the dcmi (dublin core metadata initiative) draft specification for kernel metadata [28]. for example, http://historical-ontologies.info/ark:/99152/b4cc2g3s/?info might return the kernel metadata “about” statement along with the support statement related to commitment [29]. erc: who: nineteenth century knowledge project what: library of congress subject heading term: “abbeys” when: 1910 where: library of congress. subject headings used in the dictionary catalogues of the library of congress, vols. 1-5. washington: g.p.o., library branch, 1910-1914. erc-support: support-who: metadata research center at drexel university support-what: permanent: stable content. support-when: 2020.12.03 support-where: https://historical-ontologies.info/ark:/99152/b4cc2g3s/?commitment figure 3. collection related kernel metadata returned using the ?info inflection as depicted in figure 3, using the ark naming scheme permits the use of inflections to qualify a request to retrieve different forms of the data. in principle, content negotiation achieves the same end, but it requires software to use special protocol headers. inflections make the entire request transparent when submitting a modified ark to any web browser. by publishing an endpoint that will always return the latest version of the ontology and exposing the services it can provide, data can be returned in a variety of formats. the current resolver we are constructing is intended to resolve arks based on the uri of a get request. while the ark specification standard does not preclude incorporating content negotiation into the services provided, we have not yet identified a use case specific to the lcsh1910 ontology for this retrieval method. the service at historical-ongologies.info is intended as an intermediary between an ark resolver and the underlying vocabulary server, hive [30]. further integration may occur, but this separation of concerns is intended to make the ark resolver useful to additional projects. we recognize there is room for expanding in this approach. for example, additional transformations might be performed on requests that can be made using variations on these inflections. directives inferred from these inflections would pass through to the responsible data source (in this case, the hive vocabulary server) to return metadata in various formats or even include the retrieval of associated scanned records or representations of digital objects in various stages of processing from a third system. in this way the resolver would act as an aggregator of services related to each term entry. conclusion the preparation of a large corpus of text as represented by the 19th century knowledge project’s electronic publication of historical volumes of the encyclopedia britannica is made more useful when described by appropriate metadata, including topical terms drawn from a standard ontology. automatic metadata generation helps to address the daunting nature of this task. additionally, standardized terminology enables collocation during information retrieval, aids a researcher in discovering relevant information, and provides insight into the topicality (aboutness) of a resource. unique identifiers are as much a part of the software development process of a publication framework as they are of the ultimate publication of the underlying collection itself. using the ark persistent identifier allowed the early incorporation of unambiguous references to objects that enhanced the workflow of deriving metadata from multi-faceted sources. we have outlined the steps undertaken to assign these identifiers for our project in the hope that it may guide other researchers who wish to employ a similar process for assigning unique identifiers to terms. notes [1] nineteenth-century knowledge project. https://tu-plogan.github.io/ [2] for a general discussion, see koster l. persistent identifiers for heritage objects. the code4lib journal. issue 47, 2020 feb 17 https://journal.code4lib.org/articles/14978 and kelly m, greenberg j, rauch cb, grabus s, boone jp, kunze ja, logan pm. a computational approach to historical ontologies. arxiv:2011.13114 [cs]. 2020 nov 25. http://arxiv.org/abs/2011.13114. [3] kunze j. ten persistent myths about persistent identifiers. 2018 aug 24. https://escholarship.org/uc/item/73m910w8 [4] ark identifier faqarks in the open. https://wiki.lyrasis.org/display/arks/ark+identifiers+faq [5] periodo – periods, organized. https://perio.do/en/ [6] the yamz metadictionary: https://yamz.herokuapp.com/ [7] the current iteration of hive prior to the assignment of ark persistent identifiers: https://hive2.cci.drexel.edu/ [8] the stanford named entity recognizer (ner): https://nlp.stanford.edu/software/crf-ner.shtml [9] for a comprehensive overview archival resource keys (arks), see the project wiki: https://wiki.lyrasis.org/display/arks/arks+in+the+open+project [10] more detailed instructions on using the noid module are available on its cpan homepage: https://metacpan.org/pod/distribution/noid/noid [11] ark shoulder faq – arks in the open. https://wiki.lyrasis.org/display/arks/ark+shoulders+faq [12] naan request form: https://n2t.net/e/naan_request [13] installing cpan modules: https://www.cpan.org/modules/install.html [14] the debian distribution of the libberkeleydb-perl package: https://packages.debian.org/sid/libberkeleydb-perl [15] the dependency can also be satisfied by installing the berkeleydb perl module: https://metacpan.org/pod/berkeleydb [16] a more complete explanation of how templates work in noid: https://metacpan.org/pod/distribution/noid/noid#templates [17] ezid: identifier concepts and practices at the california digital library. https://ezid.cdlib.org/learn/id_concepts [18] tee invocation: https://www.gnu.org/software/coreutils/manual/html_node/tee-invocation.html [19] information about shared naans and shoulders: https://wiki.lyrasis.org/display/arks/ark+identifiers+faq#arkidentifiersfaq-shoulders [20] for example, ark:/99152/b41910/cc2g3s has the shoulder b4. in this case, each shoulder is a string of one or more letters ending in a digit (inclusive). there could be an unlimited number of additional shoulders added to the general naan including b4, bb4, bbc4, etc. following this convention. [21] the registry for naans maintained by n2t.net: https://n2t.net/e/pub/naan_registry.txt [22] this uri points to a demonstration of how a resolver might integrate with the hive vocabulary server at https://hive2.cci.drexel.edu/ [23] for us based researchers, it is possible to request compute resources from the nsf (national science foundation) funded xsede (extreme science and engineering discovery environment): https://www.xsede.org/ecosystem/resources. see also: john towns, timothy cockerill, maytal dahan, ian foster, kelly gaither, andrew grimshaw, victor hazlewood, scott lathrop, dave lifka, gregory d. peterson, ralph roskies, j. ray scott, nancy wilkins-diehr, xsede: accelerating scientific discovery, computing in science & engineering, vol.16, no. 5, pp. 62-74, sept.-oct. 2014, doi:10.1109/mcse.2014.80 [24] see nginx reverse proxy. nginx documentation. https://docs.nginx.com/nginx/admin-guide/web-server/reverse-proxy/ [25] how to setup pm2 and run node.js applications. fit-devops. https://fitdevops.in/how-to-setup-pm2-and-run-node-js-applications/ [26] noid – nice opaque identifier generator commands – metacpan.org. name resolution and redirection interface. https://metacpan.org/pod/distribution/noid/noid#name-resolution-and-redirection-interface [27] kunze j, calvert s, debarry j, hanlon m, janée g, sweat s. persistence statements: describing digital stickiness. 2016 nov 10. https://escholarship.org/uc/item/2zm9x47c [28] kernel metadata is a small prescriptive vocabulary designed to support highly uniform but minimal object descriptions for the purpose of orderly collection management. dcmi kernel metadata community: kernel metadata and electronic resource citations (ercs). https://dublincore.org/groups/kernel/spec/ [29] traditionally, arks had the single and double question mark (’??’) inflections to request a short or long (full) record. the ’?info’ inflection is designed to be easier to recognize, and a number of parameters are being considered by the arksintheopen tech wg. [30] helping interdisciplinary vocabulary (hive). https://hive2.cci.drexel.edu/ about the authors mat kelly (mkelly@drexel.edu) is an assistant professor in the department of information science at drexel university’s college of computing and informatics. he holds a ph.d. in computer science from old dominion university. his research focuses on digital preservation with a specialization in archiving and accessing private web content. christopher rauch is currently a phd student at drexel university’s college of computing and informatics. he holds master of science degrees in library and information science and information systems from drexel and a jd from rutgers law school. he is a recent collaborator with the metadata research center at drexel and a former army national guard signal officer. jane greenberg is the alice b. kroeger professor and director of the metadata research center at the college of computing & informatics, drexel university. her research activities focus on metadata, knowledge organization/semantics, linked data, data science, and information economics. her research has been funded by the nsf, nih, imls, neh, microsoft research, glaxosmithkline, santander bank, library of congress, as well as other agencies and organizations. sam grabus is an information science phd candidate at drexel university’s college of computing & informatics, where she received her mslis (2016). she is also a research assistant for drexel’s metadata research center, and project manager for the leading fellowship program. her primary research is centered on knowledge organization systems, relevance, and automated subject representation for digital humanities resources. joan boone (jpboone@email.unc.edu) is an adjunct instructor at the university of north carolina at chapel hill and drexel university. she holds an ms in applied science from the college of william and mary, and an ms in computer science from the university of north carolina at chapel hill. prior to teaching, she spent most of her career as a senior software engineer at ibm where she worked in the software labs developing networking middleware and mobile applications. john kunze is an identifier systems architect at the california digital library. with a background in computer science and mathematics, he wrote bsd unix software that comes pre-installed with mac and linux systems. he created the ark identifier scheme, the n2t.net scheme-agnostic resolver (which redirects over 600 kinds of “compact identifier”), and contributed heavily to internet standards for urls (rfc1736, rfc1625, rfc2056), archiving (bagit – rfc8493), web archiving (warc), and dublin core metadata (rfc2413, rfc2731). peter m. logan leads the “nineteenth-century knowledge project,” an effort to identify how knowledge changes over time by analyzing historical editions of encyclopedia britannica. this project is funded by the national endowment for the humanities. he co-leads the “online diaries of ‘michael field,’” a tei-xml edition of writing by two fin-de-siècle women poets. he is emeritus professor of english at temple university and past academic director of the digital scholars studio in temple university libraries. he was also instrumental in creating temple libraries’ new cultural analytics certificate program. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – wms, apis and libguides: building a better database a-z list mission editorial committee process and structure code4lib issue 41, 2018-08-09 wms, apis and libguides: building a better database a-z list at the american university of sharjah, our databases by title and by subject pages are the 3rd and 4th most visited pages on our website. when we changed our ils from millennium to oclc’s worldshare management services (wms), our previous automations which kept our databases a-z pages up-to-date were no longer usable and needed to be replaced. using apis, a perl script, and libguides’ database management interface, we developed a workflow that pulls database metadata from wms collection manager into a clean public-facing a-z list. this article will discuss the details of how this process works, the advantages it provides, and the continuing issues we are facing. by veronica ramshaw, véronique lecat and thomas hodge the database a-z list is a staple of many academic libraries’ online presences, allowing students and faculty to quickly see which familiar databases are available and to directly access specific resources. traditionally, our own database a-z list has existed as two pages fed by the same xml document: databases by title and databases by subject. according to our site’s google analytics, these database pages are very important to our users as they represent the third and fourth most visited pages on our website. after the american university of sharjah (aus) library migrated its integrated library system (ils) and discovery interface to oclc worldshare management system (wms) in 2016, we needed to find a new way to synchronize database data. this article will outline how we automated this process using wms apis and libguides. old procedure prior to 2016, the flow of information from our millennium system to the two database pages was automated. millennium resource and license records contained the necessary metadata such as database name, description, url, and subject keywords. the data was harvested from a list of all resource and license record numbers via /xrecord, a facility of millennium’s public access catalog that displayed the records in xml format. a script harvested these records into a dublin core xml serialized file. this was done whenever changes were made to the resource or license records. the resulting file, titled databases.xml, was then placed on our website’s server where the two public database pages, databases by title and by subject, would be dynamically generated from that data. in this way, we were able to synchronize the database information stored in millennium to the pubic database pages, ensuring information was kept up-to-date and consistent. after the wms migration in 2015, the aus library began the process of migrating from millennium to wms and in 2016 had stopped using millennium completely. these changes meant it was no longer possible to generate our public database pages using the old procedure. as a result, any changes to the details of a database, such as a new name, modified url, or updated description, had to be changed manually in wms collection manager, and in the databases.xml file. in addition to the amount of time this took, the duplication of work and dependence on manual changes created the potential for human error to introduce inconsistencies between the systems. our new discovery interface, worldcat discovery, does provide a list of active collections through the a-z titles interface, so we considered it as potential replacement for our database a-z. unfortunately this tool has a number of problems. the largest problem is the sheer volume of duplication due to the fact that it lists every collection your library has activated in wms collection manager. for example, we have almost 200 individual collections of springer ebooks for each year and each subject we’ve purchased (springer behavioral science and psychology ebooks 2016, springer behavioral science and psychology ebooks 2017, springer economics and finance ebooks 2017, and so on) which makes for a cumbersome list to browse through. another problem is that we have a number of collections active in collection manager that we wouldn’t consider databases. for example, we have access to a handful of journals from the university of chicago press from individual subscriptions, but this is far to small to be considered a database. these problems made worldcat discovery’s journal finder interface too cluttered and unwieldy to a useful replacement for our database a-z list. finding a new way our goals for a new database a-z list were that it should: be automated: to reduce the duplication of work and possibility of human error. be clean: human readable with no duplicate content cluttering out the useful information. be consistent: every platform (website, libguides, discovery interface) should have the same metadata. users should see the same names for the same databases across every library platform to minimize confusion. maintain continuity: similarities between the old a-z database pages and the new one should help avoid user disorientation in the transition. to begin we needed to get a thorough understanding of now wms handles database data. it does this through two different modules: collection manager and license manager. collection manager contains the metadata about the database (e.g. description, url) as well as a list of every ejournal or ebook contained in the database and the date range covered. license manager handles the renewal dates, prices, terms of use, and allows us to upload renewal documentation. next, we looked at using the libguides database a-z tool. for those unfamiliar with the product, libguides is used to make subject and course specific guides to library resources. since our initial purchase of the libguides system, the vendor springshare added a public-facing database a-z, however we did not have any use for it up to this point because we were already generating our own database a-z from data harvested from millennium. with this project, we decided to use the libguides database a-z page to replace our existing database lists. for the sake of clarity, in this article we will refer to it as the libguides database a-z. new procedure the following is an outline of how our new procedure works. it is divided into three sections: (1) the preparation work that needs to be done in wms and libguides before the script is run ; (2) running the script which exports database metadata from wms license manager and collection manager ; and (3) loading the resulting file to libguides. preparation wms license manager license statuses must be current (as opposed to expired or pending) licenses must contain the correct linked collections [see figure 1] wms collection manager select a representative collection for each database. for example, we have 3 separate collections for proquest ebook central: our main subscription, our dda subscription, and our perpetually owned titles. we selected one of these, our main collection, as the representative collection. this prevents unnecessary duplication in the database list. ensure the collection title is appropriate. this can be locally customized by the library. for example, we renamed ebook central academic complete as proquest ebook central ensure the collection has description text verify the collection url is correct include the libguides content_id in the staff note field include the subject heading(s) in the staff note field [see figure 2] libguides clean up the existing database list in libguides. in our case, we had a lot of links that were saved as databases in libguides that we either changed to link assets, or deleted completely. ensure each database is saved in libguides and is assigned a content_id. the script only updates existing database assets, it does not create new ones. classify the database type as abstract-only or full-text, as needed. running the script [see figure 3] using the oclc license manager api, the script runs through all our licenses. for every license, it gets the license status (current, expired, or pending) and the list of linked collections. using the oclc collection manager api, the script provides the details of the collections linked to the licenses, such as name, description, url, proxy yes/no, content_id, and subject headings. the script searches for the linked collections via the collection manager api. it creates a brief delimited record for each license or collection which should be included in the a-z list. it outputs a tab-delimited .txt file [see figure 4] loading to libguides check the tab-delmited .txt file to ensure data integrity. go to libguides > content > a-z database list click “import / update databases” and upload the file. all changes made in collection manager are now synced to the libguides database a-z [see figure 5] figure 1. license of proquest ebook central contains three linked collection. the script will look at the collections linked in every current license. figure 2. collection for proquest ebook central where we added a description paragraph. in the staff note field, we added a content_id that links it to it’s corresponding libguides database asset. we also added subject headings. figure 3. the internal steps of the script which exports database metadata using the license manager api and collection manager api, and outputs it as a tab-delimited file. figure 4. script output (.txt file imported into excel) figure 5. our new database a-z page. users can filter by subject or database type (abstract-only or full-text). pseudo script this perl code snippet represents the core function of the script. the full script is available at: https://github.com/hodgenet/oclc-libguides-integration access to the perl authenication modules is available by contacting the author at thom@hodgenet.com. ## set these to defaults, modify as needed from results: ## when total results falls below itemsperpage, we have everything while ($startindex < $totalresults) { ## get the parameters for license manager api my %queryparams = ( 'itemsperpage' => $itemsperpage, 'startindex' => $startindex, 'q' => "licensestatus:$licensestatusfilter" ); ## call makeoclcrequesturl function to make the url correctly my $requesturl = make_hru(\$apiurl, \%queryparams); my $response = $ua->get( $requesturl, 'authorization' => $atauthheader ); ## get the first bit of the license listing my $apirecord = $response->content; ## parselicenses gets some metadata for the license ## and each of the collections linked to it ($totalresults) = parselicenses(\$apirecord); ... foreach my $collection ($entry->child('content') ->child('license') ->child('collections') ->child('collection')) { my ($thiscollectionname, $thiscollectionprovider); my $thiscollectionid = $collection->child('id')->value; if (defined $collection->child('name')) { $thiscollectionname = $collection->child('name')->value; } else { $thiscollectionname = 'no name'; } if (defined $collection->child('provider')) { $thiscollectionprovider = $collection->child('provider')->value; } ## call getcollectionrecord my $thisxmlcollectionrecord = getcollectionrecord( \$thiscollectionid, \$thiscollectionname ); } ## increment the start index for next batch of licenses $startindex = $startindex + $itemsperpage; } ongoing issues a major limitation of the current iteration of the script is that it doesn’t alert us to the cause of errors. if the script encounters an error, we must hunt for the cause. the common problem we encounter if a collection was deleted in collection manager but was not also removed in license manager, it will cause the current script to fail. other errors arise from incorrect formatting of the content_id or subject headings, invalid characters, or a license status is not set to “current” in license manager. we plan to improve the script so it can return human-readable error reports. one feature of the libguides database a-z tool is the ability to label certain databases as “popular.” we identified a few databases as popular after looking as usage numbers. unfortunately the libguides import process overwrites whether a database is labelled “popular” in libguides, meaning we have to re-label them manually every time we run an update. conclusion after a few months spent developing this procedure, writing the script, and preparing our data, we finally launched our new libguides database a-z in late may 2018. based on our experiences so far, we plan to further refine the script and procedure to improve error reporting, troubleshooting and further reduce the amount of manual work involved (such as re-labelling databases as “popular”). this procedure and script were the products of necessity. after migrating to a new ils, our old workflow was no longer viable. we were able to exploit the tools we had available — oclc’s apis (which are free of cost), and the libguides database a-z — to devise a new procedure. in doing so, we were able to achieve the goals we set forth at the start: an automated procedure that produces a clean output, creates consistency, and maintains continuity. other libraries using wms and libguides should consider using a similar procedure to produce easily automated database a-z pages that work well for their patrons. about the authors veronica ramshaw web services librarian american university of sharjah vramshaw@aus.edu https://orcid.org/0000-0002-4086-9419 véronique lecat cataloging and metadata librarian american university of sharjah. vlecat@aus.edu https://orcid.org/0000-0002-1373-8384 thomas hodge associate university librarian for technology and technical services american university of sharjah. thodge@aus.edu https://orcid.org/0000-0002-0026-2532 subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – column: we love open source software. no, you can’t have our code mission editorial committee process and structure code4lib issue 5, 2008-12-15 column: we love open source software. no, you can’t have our code librarians are among the strongest proponents of open source software. paradoxically, libraries are also among the least likely to actively contribute their code to open source projects. this article identifies and discusses six main reasons this dichotomy exists and offers ways to get around them. by dale askey introduction not long ago, a public university library created citation builder, where users enter the pieces of a citation, click a button, and have them magically formatted as apa, mla, etc. slick, but not really revolutionary since others have done this before. somewhat later, another university in the same state got their code, enhanced it, and rolled out their own edition, explicitly acknowledging the other university’s work on the original product. still later, my library hired a librarian who recently earned her mls at the university that had created the second version of the service. the new librarian offered to contact some people she knew at her degree school and see if we could get their code to use as a basis for our own tool, provided we promised to thank them and share any enhancements we developed. sounds a bit like ad hoc open-sourcedness in the making. except that it wasn’t. said university responded by asking us to sign a software licensing agreement, which we refused to do because we don’t like to sign licenses for such trivial things, and we certainly didn’t want to be bound by any limitations on sharing the code further, since we intended to build significantly on their core. in the end we opted to develop our own citation generator. this is inefficiency piled on top of inefficiency. citations and style manuals are constants; there is neither reason for nor benefit to having everyone create their own citation tool for their users. this kind of library-to-library software sharing saga is sadly not all that uncommon. (see correction.) before offering any further observations on the ways libraries share and fail to share open source software, i should clarify my perspective on the topic. first and foremost, i am not a programmer. what i am is one of those librarians who is otherwise technically semi-competent, i.e., i can install software on solaris/linux, know what a shell is, sometimes find myself hitting :wq in word, and can do something i’d refer to as sys-admin lite. i’ve also spent many productive and not-so-productive hours wrangling both open source and commercial library software, giving me a head full of lessons and ideas. this is not unique—there are many librarians who exist in this netherworld between it work and “librarian” work. the general topic of open source software is simply too broad to cover here. as dan chudnov points out in his chapter on open source software in the book information tomorrow, open source software has, at least on one level, successfully integrated itself into libraries, particularly when it comes to system infrastructure and programming languages (chudnov 22). additionally, libraries have strongly embraced certain consumer open source applications, such as firefox and thunderbird, as well as object repositories such as dspace and fedora and content management systems such as drupal. that’s all fine, but where we tend to fall flat is in the area of creating, maintaining, and sharing library-specific applications. there are certainly myriad exceptions to this statement, but i would suggest that however large and noteworthy, they remain the exceptions, and not the rule. we also miss the boat on sharing what i’ll call open source lite: tiny programs written in various scripting languages that drive all the doodads and widgets on our websites, or extend (or, in some cases, repair) the functionality of our commercial systems. these are typically less than a couple of hundred lines and are often deemed lightweight and insignificant to those who write them, but are godsends for the many librarians who lack access to programmers. collectively, they represent thousands of hours of valuable work time. even those libraries rich in programming resources can benefit from not having to create everything on their own. after pondering this issue for some time, i identified the following issues as the driving forces that undermine the sharing of open source software in libraries: perfectionism – unless the code is perfect, we don’t want anyone to see it dependency – if we share this with you, you will never leave us alone quirkiness – we’d gladly share, but we can’t since we’re so weird redundancy – we think your project is neat, but we can do better competitiveness – we want to be the acknowledged leader misunderstanding – a fundamental inability to understand how an open source community works many of these issues operate in combination, but any one of them is sufficient to thwart the development and adoption of open source software in libraries. perfectionism perfectionism is perhaps the most obvious issue that arises with open source software, and plagues mainly smaller applications that were developed by one or two people at a given institution. we have all encountered this type of programmer, where untidy code, lack of code commenting, and–sin of sins–inefficient or redundant code makes them unwilling to show it to anyone else who might be able to point out the flaws and correct them. it may seem trivial to peg some of our woes with open source software on the programmers’ desire to hide their imperfections, but the issue repeatedly arises. one hears it mentioned in conversations with programmers and reads it in the messages one receives in reply to a request to see this or that snippet of code that they have written. even at library conferences where many successful programmers congregate, i’ve heard people at the podium talk down their code and make self-deprecating jokes about it, even though it’s perfectly functional and useful. there is always that fear that someone else in the room knows it better and is silently smirking at the fact that the person at the podium didn’t know this or that trick. expand that audience to the wider library world, and it’s a wonder that any less-than-perfect code ever gets shared. dependency somewhat related to perfectionism is dependency, which has the programmer and his or her time at its core. nothing is more certain in the world than this: if you share software with someone, you will be asked to support it, even if you make it perfectly clear that you have no ability and no intention to do so. this can be traced to how open source software moves through libraries. it begins with personal contact: librarian x asks programmer y for some code. flattered and temporarily overcoming their innate perfectionist side, programmer y happily complies. when something goes awry — inevitable due to the myriad system variables in play — librarian x writes a “sorry to bother you, but” message. being a nice person in a service-oriented organization, programmer y assists them, but may end up somewhat cranky about it since it detracts from their own work and yet does not add to their list of accomplishments. there are at least two ways out of this conundrum. one is to accept one’s fate and provide support more or less explicitly. virginia tech’s libx toolbar falls into this category. as an adopter, i’m certainly grateful to have a functional toolbar to offer users and an edition builder in place to create and maintain it, but often wonder about the sustainability on virginia tech’s end. the other path, the one that i wish more libraries would follow, would be to post the code in some public repository, along with at least rudimentary documentation (perhaps the work of one or two hours). that way, the code can be picked up anonymously, breaking that person-to-person chain. a public repository makes it clearer that the code is being offered as a service, but support may not be forthcoming. needless to say, it should be posted with an unambiguous open source license. this sounds easy enough to do, but it’s not a reach to say that it rarely happens. libraries typically do not make it a priority to share their successful tools with other libraries. new projects become the focus, staff changes, priorities shift; this renders this last, critical step expendable all too often. quirkiness perhaps most maddening among the various reasons for not sharing software is quirkiness, the sense that one organization’s needs are so locally-tailored that would make no sense to release the software to the broader library community. one would think that for institutions with a high awareness of standards this would not be a major issue – but it is. at a coalition for networked information meeting in 2007, the university of maryland presented the digital library that they had built on a fedora platform. the features and tools they had created were impressive, and i naively assumed that the question i posed was an obvious one, namely, when did they intend to release the code for others. what they offered was a reason why they could not: they had used an idiosyncratic local metadata scheme. if they had to do it over, one of the presenters offered, they might reconsider that choice. this was an unwise decision on their part. they now have applications that they must support on their own, entirely at their own cost. should the developers leave, they may struggle to find others to maintain it. even if they do, will they find someone content to babysit someone else’s application, or who wants to reshape it according to the context and frameworks they know best? they also lose any potential benefits that could accrue from having others extend or refine the functionality. i have some sympathy for choosing this path. programming is neither a trade nor a rote process, it’s a creative art. there are myriad ways to solve problems, and a bevy of languages and toolkits that can be learned and applied. babysitting someone else’s code, or worse, being asked to clean up someone else’s code written in a language one doesn’t know well or doesn’t like, is thankless and uninspiring work. it is much more interesting and enriching to create applications and build something that is one’s own. redundancy the flip side of quirkiness is redundancy, which is when there is perfectly acceptable software available and yet is rejected because it’s not quite what one would have done had they created the software. in simple terms: reinventing the wheel. this could also be called the “big challenge” dilemma. librarians who are not programmers or system administrators want services, and want them delivered on time via the path of least resistance. programmers, on the other hand, are typically systems thinkers, with strong opinions about how best to build and maintain a system. perhaps a bit of arrogance is also in play. it’s tempting to look at software, identify its flaws, and declare that one could do better. however, the reality is that libraries and universities have limited resources, and often have to make compromises to be able to sustain all of their services. it is more than likely that the software that results from this line of thinking will evince different, but equally obvious, flaws. how many of us work at universities where some it entity or authority has declared that the software available for a given task, whether it be commercial or open source, is unsuited to the unique situation at hand. what can ensue is a multi-year exercise in wheel reinvention, which often does not end happily. my university has fallen into this trap. it refuses to sign on with itunes u as a podcasting platform, choosing instead to write its own platform to compete with apple. a press release issued in 2006 was subsequently picked up by the chronicle of higher education, which declared that kansas state was about to “become the educational-podcasting capital of the world” (read). not long after that, someone from the office tasked with developing this wunder-platform remarked sarcastically in a meeting that “we don’t need no stinkin’ itunes.” this was over two years ago, kansas state isn’t even the podcasting capital of kansas, and faculty still clamor for itunes u. moving back to libraries and to open source software, i would suggest that one concrete manifestation of the “big challenge” dilemma is the emergence of the open library environment project. its goal is to define and develop an open source integrated library system geared toward academic libraries. there are, however, academic libraries who have either implemented or are about to implement an open source ils, as well as a healthy number of active developers for this particular ils. while it would be mean-spirited to wish the ole ill, it seems unwise to fragment what fragile momentum there is toward the open source ils in academic libraries by creating a new fork. competitiveness one could also view the ole as an example of a bit of competitiveness. competition can serve a purpose, driving innovation and development. it can also sap resources, and in the context of libraries, where we have no access to venture capital, clearly collaboration is to be preferred to competition. most libraries do not have a budget line item for research and development, and so we scratch together bits of money to drive our development projects. while grant-funding, such as the mellon money driving the ole, is a wonderful thing, it comes with its own problem, namely, how to sustain the project beyond the scope of the grant. having a limited set of partners is acceptable while the money flows, but you need a mass community to keep things going. at least within the united states, much of the competitive spirit between universities arises, sadly, from our sense of athletic competition. practically every institution is interested in building its brand and creating its community, all in the name of generating revenue for the institution. as such, it is more the exception than the rule to see substantial collaboration between universities as part of their normal way of doing business. this certainly extends to libraries, where we all dutifully build our own institutional repository, our own digital libraries, and develop our own web services for our users, typically with little regard for what’s going on down the road. how many libraries have invested how much labor in getting an institutional repository off the ground? what results has this intense investment brought most institutions? these are questions we do not ask for the most part. we need to move past the notion that partnering and using the tools at hand is tantamount to capitulation. misunderstanding perhaps the most vexing issue that plagues open source software in libraries is a simple misunderstanding on the part of many librarians and library administrators about the nature of open source. many librarians have acculturated to an environment where our software needs are met by using a product from an established vendor that has a large booth at our major meetings. any new library technology that arises predictably leads to the development and marketing of multiple commercial ‘solutions.’ on the one hand, this is understandable. licensing a product from a firm removes many of the variables, provides stability, and does not require hiring too many people into libraries whose skill sets the typical librarian does not understand and cannot assess. on the other hand, we have a culture of complaint where we expect vendor software to react immediately to our every whim and fancy. this is, of course, unrealistic, and leads to myriad enduring versions of the “our opac sucks” meme. open source clearly represents the remedy to this issue. if you need functionality, find software that comes close, hire some programmers, and get to work. find partners, share code, collaborate. in the end, you significantly shorten the feedback loop between users (whether they be staff or the public) and developers, and can exercise far more control over the software than with a vendor. sounds simple, but it often fails because of the innate culture of many libraries. having become mired in the vendor-driven rut, we cannot find our way out. in that world, we are simply consumers of software. we pay our money and get a product. we provide feedback, ask for enhancements, but that is more or less the extent of our involvement in the development of the software. with open source, one can choose simply to download the code, install it, and start using the software. that’s not likely to get one very far. there are versioning issues, customizations, security concerns, backup plans, bugs, and so on. what often happens with open source software in libraries is that non-technical librarians and staff begin to view their colleagues down the hall in the it shop as the vendor, and simply continue reporting issues and asking for enhancements. what they fail to understand is that with open source, they have the ability and, i would argue, the responsibility to seek out their own solutions and participate in their implementation. they need not know all the technical details, but they should be able to articulate what they need, seek solutions from the community, and suggest how these would fit within their institution’s suite of services. they should assist with usability and documentation issues, which do not require programming skills. essentially, it’s at the point of shifting from being a consumer of software to being an active participant in the open source community that the misunderstanding hits its zenith. most librarians do not understand the open source software model and fail to recognize that by using open source software, they are tacitly declaring their membership in the community. an example may help illustrate this point. as reported in technical services quarterly, karen calhoun presented itso cul, the integrated tool for selection and ordering for cornell university libraries, at the ala conference in 2004. not surprisingly, she was “asked if cornell would share the program with other libraries.” she responded by asking if the audience would prefer an open source or a shareware approach. the audience favored shareware, “not only for the benefits to those in the audience who wished to use the program, but to cornell by reducing their administrative costs and responsibility for the program” (coulter 82). this shows a clear misunderstanding of both open source and shareware. the shareware paradigm squarely puts the onus on the developer to support and develop the software, since the idea behind shareware is to download and use the software as is, either as a trial version or with a request to pay a small amount to the developer. it would be the open source path that would reduce cornell’s burden, since some adopters would become partners in the ongoing development. not surprisingly, neither of those options ended up being the path taken. instead, itso cul morphed into worldcat selection, a fee-based service of oclc. while one could correctly argue that this makes it more immediately accessible to more libraries since we’re quite accomplished at writing checks to oclc, in the long term it just adds another maintenance bill to the library’s budget, and consumes resources that could be used for other things, such as a programmer, who could work on many projects. perhaps more importantly, it also puts the ongoing development in the hands of a single vendor. in libraries, we certainly have extensive experience with the outcome of that scenario. what can be done? while it’s good sport describing what is wrong with our use of open source software, one should also suggest ways out of the jam. a simple way would be for us to agree on how best to share software between libraries. in this case, i would distinguish again between “macro” open source and open source lite. the former concerns large and robust enterprise software, such as evergreen, koha, ojs, dspace, etc. those clearly have their own user communities and distribution channels. with open source lite, it’s much more wild west. some of it can be found on google code, some in sourceforge, but most of it lives in little pockets at hundreds of libraries, and we pass it around as email attachments. that’s not terribly effective. one promising development in this regard, which also happens to be an example of open source overlapping with commercial code, is the recently announced el commons, which appears to be intended to be a clearing house for user-created code that enhances ex libris products such as sfx. having such product-centered spaces makes a lot of sense on many levels. one concern with such vendor-supported spaces is that they are rarely open to non-customers, out of fear that the tools might reveal proprietary information. at the very least, libraries who deposit code in such places should offer it elsewhere to the broader community, whether the vendor approves or not. for the rest of it, should we just avoid reinventing the wheel and use google code? one could do worse, but that doesn’t make it easy for the non-technical to access. google code is certainly findable, but sites such as sourceforge and google code use a language and presentation style that tends to repel the non-technically inclined librarian, i.e., most librarians. one solution would be a library specific code repository. that seems an anathema even to me, since it’s potentially redundant given the options already available. if we want our not-so-technically bold colleagues to participate, however, perhaps it’s a necessary evil to lower the threshold for them. even if we lack such central points, libraries need to develop a far more open-minded attitude toward sharing their open source lite. going back to my opening vignette, i would ask what the purpose is of asking a library to sign a licensing agreement for a couple hundred lines of php, other than to demonstrate how beholden our thinking is to our generally risk-averse library culture and our vendor-oriented business model. if someone asks for such code, shouldn’t the response be, sure, here you go, give back any improvements. let’s be nice about these things, slap some sort of standard open source license on our software, and pass it on without the pointless negotiations or stern warnings. this step is nearly always skipped at a project’s end. more fundamentally, libraries that wish to use open source software need to understand the staffing commitment they are making by going that route. open source software requires programmers, interface designers, and system administrators. these people are often not librarians, although one frequently sees job ads asking for librarians who have such skills, as if library schools were actually cranking out such people in any quantity (they should be, and this failure will negatively impact libraries for decades). as such, we struggle as organizations to define the needs, properly prioritize them, and make progress on finding the right people with the right skills. we don’t commit to paying the salaries necessary to attract and retain enough experienced and talented people. a related issue is the question of recognizing and rewarding staff contributions to the open source community. libraries typically have no reward systems for putting something out there that helps others. these are largely questions of leadership in libraries and the necessity of moving past the traditional roles and obligations of libraries and embracing radically different organizations than we currently have. leaders must have a new vision of using large software packages, where the question isn’t simply open source vs. commercial, but, if one chooses open source, how to provide the resources necessary to make it successful. as dan chudnov pointed out in the conclusion to his chapter on open source software, in 2007, it is “impossible to run a library without software, but you can have a library without a building” (29). this insight should force us to significantly revise how we set priorities as organizations, but that shift has been slow to develop. why is all of this so important? the central concern i have is for the financial future of libraries. surely all libraries face chronic budgetary concerns. every time serials go up or our funding agencies issue cuts, our ability to do the things we lay out in our strategic plans contracts. on the it side of the organization, we often do not get approval to recruit for the positions we need, or even if we do we can’t offer sufficient salary to fill them, at least not for very long. and trading the acquisitions budget for software purchases and maintenance contracts just seems like recreating the problem. it’s often said that we need to become more efficient organizations. the it side of libraries is one of the more underexplored areas for efficiency, given its relative youth on the budget sheet and the fact that some, shall we say, legacy administrators have difficulty understanding the it puzzle, let alone how to create efficiencies. open source is clearly part of the answer to this dilemma. while in the short term it requires new investment in staff—although one could point out that it’s really a zero sum process of converting vacant legacy positions into it or programmer positions—offering better services free from the morass of maintenance and purchase contracts that accompany traditional vended software puts libraries in a much better position moving forward. notes chudnov, dan. “the future of floss in libraries.” information tomorrow: reflections on technology and the future of public and academic libraries. medford, n.j: information today, inc, 2007. 19-30. (coins) coulter, cynthia m. “acquisitions technology trends: changing the way we do business. a report on the alcts acquisitions section technology committee program. american library association annual conference, orlando, june 2004.” technical services quarterly 22.4 (2005): 77-82. (coins) read, brock. “the wired campus: a harvest of podcasts.” chronicle.com 27 sep 2006. 6 oct 2008 http://chronicle.com/wiredcampus/article/?id=1599. correction 7 january 2009: actually, the colleague mentioned here pointed out to me that after working with both the second-named institution and the originators, we eventually did sign a license, albeit with the institution that originally created the software. we have been working with and adapting the original code, which is a good thing, but the fact that a license intervened means that this is still not open source when it could/should have been. many apologies to my colleague and for any confusion my telling of the story caused. acknowledgement this article is based on a talk given at access 2008 in hamilton, ontario, on october 4, 2008. about the author after stints at washington university, the university of utah, and yale university, dale askey currently fills the role of web development librarian at kansas state university. on the information technology side of the library, he specializes in interface design and project management. he also works actively on scholarly communication issues, serving as publisher for the open-access journal imprint new prairie press. tags: column, open source subscribe to comments: for this article | for all articles 14 responses to "column: we love open source software. no, you can’t have our code" please leave a response below: jon gorman, 2008-12-21 i’d add one more: lawyers and university control. at least for some, various ip offices on campus seem to be treating any software developed to be a possible revenue stream. the odds of actually getting permission to release software as open source if it has any commercial benefit is small. this leaves the libraries who have developers stuck with trying to create something that’s not successful enough to be noticed but still share or to run the risk of being noticed by people who probably have more pull in the institution than them. (if it’s a big research university). it leads to a type of chilling effect. meg, 2008-12-22 there are a lot of types of libraries out there. you seem to be focusing almost exclusively on academic libraries. public libraries often have whole other issues. it might be a good idea to specify. shaun ellis, 2008-12-24 kudos and thank you, dale. i have been voicing similar concerns, though not nearly as eloquently. i also second jon gorman’s addition. i should mention that contributions don’t need to come in the form of actual code. testing, ui, design, and simple feedback are just as critical, but the infrastructure has to be there to be able to allow these contributions to make it into the “eco-system,” so to speak. at rutgers, we’re just starting to use the drupal project module to supply the infrastructure. it seems very promising. i also agree that schools offering mlis degrees need to include oss development courses in their curriculum. it’s frustrating that they have not jumped at the opportunity to fill the it gap. christopher harris, 2009-01-05 great overview! as the director of a school library system, i have been able to establish an environment where using and contributing back to open source software is the expectation. i did this by meeting with the superintendent and the board of education and explaining that without the open source drupal framework, we would not be able to build and grow. i also took the superintendent and our cfo through the general public license (gpl) under which drupal is released line by line to explain it to them. the cfo and board were especially happy to learn that they retained the copyright on the code and could still sell the code/program/service for a profit while still meeting the requirements of the gpl by releasing the code. i think administrative misconceptions are another area that need to be addressed so they can help champion the cause for open source (and the sharing therein) if others are against it. joann ransom, 2009-01-24 great story – thanks for raising the issues. i work for horowhenua library trust, a small public library system servicing a district of 30,000 in new zealand. we developed koha (library management system) back in 2000 and more recently kete in 2007 (a community built digital library of documents, video, audio, articles, weblinks, comments). both of these projects have been released as open source and are doing very well. we are not a wealthy organisation, and we must spend our it budget prudently. open source has been fabulous for us. we have been happy and proud to release fully functioning starting points for both projects. we know what problems we need to solve but we need and want the brains and skills and vision of many librarians and developers and programmers to help us take the projects to the next step. foss seems to me to be a no-brainer for public libraries; it fits perfectly with our ethos of sharing knowledge. thanks. jo. wow, someone wants to publish me! « the bibliobrary, 2011-08-02 […] one, the piece she mentions is an article in the open access code4lib journal. it would make a mighty thin book. beyond that, there’s the sheer inanity of asking–in […] we love open source software. no, you can’t have our code | my blog, 2012-01-21 […] we love open source software. no, you can’t have our code/ dale askey […] jenny reiswig, 2012-02-03 echoing jon gorman 100%. anything i produce in the course of my job is owned by the regents, not me or my department. and it can be a byzantine process to figure out how and whom to ask for permission to open source something developed for the library and there is no guarantee it would be granted. regeniaec, 2012-05-22 they should be more careful while creating a opensource software. many will try to attempt to steal their codes to use it for other purpose. digital tools, trends, debates – reading list compiled by dorothea salo | lilabrary, 2013-03-19 […] […] briefly noted for february 10, 2009 archaeoinaction.info, 2013-06-21 […] back a bit to the december issue of code4lib journal, dale askey considers why librarians are reluctant to release their code and suggests some strategies for stemming their reluctance. i have to say i sympathize completely […] digital humanities 2013: session notes | the bibliobrary, 2013-07-19 […] thing he discussed rings true for me based on my observations on the failure of library coders to make their code available, namely, that many people are afraid to put less than perfect materials out there for people to […] in the library with the lead pipe » open ethos publishing at code4lib journal and in the library with the lead pipe, 2015-08-03 […] love open source software. no, you can’t have our code.” code4lib journal, 5. retrieved from: http://journal.code4lib.org/articles/527 bornmann, l., marx, w., schier, h., thor, a., & daniel, h.-d. (2010). from black box to white […] access 2015 toronto thursday and friday notes | the bibliobrary, 2015-09-22 […] said some nice things about my access 2008 talk in hamilton. i never knew that that was her first access, nor that my talk had resonated with her that way. […] leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – “you could use the api!”: a crash course in working with the alma apis using postman mission editorial committee process and structure code4lib issue 54, 2022-08-29 “you could use the api!”: a crash course in working with the alma apis using postman while there are those within libraries that are able to take vendor apis and use them to power applications and innovative workflows in their respective systems, there are those of us that may have heard of apis but have only the slightest idea of how to actually make use of them. often colleagues in various forums will mention that a task could be “just done with the api” but provide little information to take us from “this is what an api is” or “here’s the api documentation” to actually putting them to use. looking for a way to automate tasks in alma, the authors of this article both found themselves in such a position and then discovered postman, an api platform with a user-friendly interface that simplifies sending api calls as well as using bulk and chained requests. this article gives a basic primer in how to set up postman, how to use it to work with exlibris’ alma apis, as well as the authors’ use cases in working with electronic inventory and course reserves. by rebecca hyams and tamara pilko introduction library workers come into their daily work from a variety of different backgrounds, skills, and expertise. advanced users may already have the coding skills to streamline or automate workflows, but many more of us are left unsure how to get from step one to step three. we may see there are powerful tools that can improve our work (or at least get us through a particular task or project) but may lack the necessary skills and information to actually make use of them as-is. a prime example of one of these tools are the robust apis provided with ex libris’ alma library services platform. these application programming interfaces allow users (as well as other applications and systems) to interact with alma’s data directly, and can power innovative workflows and even just ways to get around what is sometimes a lot of clicking and repetition in the interface. while there are a handful of basic guides explaining alma’s apis or what apis even are, there seemed to be little to bridge the gap between “introduction to apis” and “how to use them to make better workflows” unless you already knew how to script or code. the authors of this article both found ourselves at this crossroads. we could see that we had work that needed doing that would have required lots of repeated actions in the interface, but not enough background in programming to know how to go about harnessing the power of the alma apis without a user interface. both of us eventually found our way to postman, an api platform that supports a wide array of workflows. built with api developers in mind, postman provides a straight-forward interface for plugging in parameters and data to use apis in a way that’s flexible and relatively easy to understand with minimal coding knowledge required. in this article, we will outline how to set up postman and how we were able to use it to work directly with the alma apis to manage data-intensive projects. of course, the same general process for configuration will be similar for working with other apis from other vendors and platforms. we will also discuss our individual use cases: cleaning up electronic inventory and loading course reserves data with reading lists and citations. while this is not meant to be an extensive treatise on how to use postman or work with apis, we do feel this will help bridge the gap for those out there that find themselves in situations like we did. what is postman? postman is a downloadable client and web application that was created as a tool to help with the api testing process. it is now a robust platform for api development, with features to support both the building and use of apis. postman has tools for documentation, collaboration with teammates or the larger community, and makes it easy to iterate projects and share them. the postman website (https://www.postman.com/) is the place to start to create a free account and download the app or utilize the web version. for the purpose of this article, and likely many library-based api projects, postman is a helpful interface that lets you view, send, interact with and use api requests. you can easily see if your request worked and what response was returned. while this article will not go into depth about api development, there are many great resources out there and we have included some in our citations. for an introduction to apis in alma, please see chad kluck’s presentation from eluna 2019 [1] and the exlibris developer network “api tips for beginners.” [2] postman also offers a learning center [3] that has extensive documentation and videos. setting up postman basic configuration and prerequisites regardless of exactly what api or data you’ll be working with, there are a few things that you will need to do to get started. as of the time of writing, the first step is to create an account with postman and then download and install either the postman client itself or the postman agent that allows you to use the browser-based version. in addition to getting set up with postman, you also need access to the api(s) you wish to use. as previously mentioned, for the purposes of our discussion here we will be using the apis for ex libris’ alma platform. in order to use the apis for your alma environment you will need access to the developer network and will need to build an api to generate a key to authorize the connection. ex libris has extensive documentation for getting started with the developer network and building your own api keys. alma’s apis are divided into various functional areas of the platform (electronic resources, bibliographic information/inventory, users/fulfillment, etc.) and can be configured to be read-only (this is fine for projects where you just want to retrieve data) or read/write. the api documentation gives a breakdown of what data can be accessed via each of the api endpoints and by what methods, as well as sample output that can be useful in working with the data. lastly, while not required, it is helpful to have at least enough familiarity with javascript to be able to edit small snippets of code. we will provide some examples of code via the example environments we’ve created (linked below), but you may need to adjust it to better suit your own needs. javascript is mostly used to pull in or retrieve variables, which is particularly useful if you’re running requests in a chain. it’s also helpful to be familiar with the syntax of the data you’re working with (most likely json or xml). one last thing to keep in mind as you follow along is that postman is a very flexible platform. you may find that different documentation or tutorials will describe how to do the same thing any one of several ways, all of which may work. we found that we had both taken slightly different approaches ourselves, and while we have done our best to ensure consistency in this article there may still be some minor discrepancies. for purposes of demonstration (and to easily share our code and other materials in context), we have set up two read-only public workspaces in postman: deleting electronic collections and setting up course reserves. both contain documentation and annotated code that should prove useful if embarking on a similar project for the first time and to reference while reading this article. lay of the land: workspaces, collections, folders, requests postman allows you to organize api requests hierarchically. a workspace contains collections, which can contain folders, which contain requests. figure 1 shows the basic hierarchy within postman. each user has a default “my workspace,” but it is possible to create new workspaces, including public ones, where multiple users can work on (or simply share) a project. collections and folders are useful in keeping requests organized [4] and are a good place to add documentation. both are also important when you have a file of variables to be used in the request(s), as they allow you to “run” the requests together (a process we will describe below) and make it easy to keep certain requests that are to be run together in order, relative to one another. however, folders aren’t required for using postman, particularly if only using one or two api requests that don’t rely on others. lastly, requests are the api calls themselves that can retrieve, add, delete and update data. requests can send parameters, authorizations or body details. figure 1. the basic hierarchy in postman. environments and variables one of the streamlined parts of using postman is the ability to use variables. variables can be used in multiple places and save a lot of extra typing. not to mention, using variables can help avoid potential mistakes during data entry! in the example request in figure 2, all three of the highlighted fields are different types of variables: figure 2. the api call with different types of variables. the {{base_url}} is a global variable, meaning that it is used for the entire workspace. one approach is to set a variable at the highest level, so that if it changes, it can be edited in the least number of places. in that case, you could set the alma api base url as a global variable. environments are where you can store variables to be used in collections. environments save you time because you enter information in one place that can be reused in all of your requests. for example, considering alma, you might have an environment for production as well as one for your sandbox, since they each have different api keys. the environment will also temporarily store any variables that are used by requests, so that as a collection is run, postman can reference the environment and use those values. it is easy to switch between environments, so once you have tested in the sandbox, it is only a couple clicks to move your work into production. setting up a request once all of the basic configurations have been put into place, it is relatively simple to create a request and make sure that everything is working as expected. there are many ways to create a new request, starting with the “+” found in the main pane of the interface. (you can also hit the “new” button in the sidebar and select “http request”). in the example in figure 3, the request contains two environment variables, one for the base alma url and the other for the api key which were set up as part of this “alma” environment. the drop down ahead of the path allows you to select the method for the request (get by default, which retrieves data). if everything is working properly, hitting “send” will bring up the response body which you can then view to make sure the data returned is what you expected. figure 3. the basic request screen. parameters parameters have values that are used in the api request url. often, an api request will require authentication in the url itself as well as other ids. in the case of the alma/primo apis, the api key is sent as a parameter, rather than in a header or elsewhere in the request, and the url will often also require some kind of identifier like an mms id, which is the “metadata management system id” and unique number for the bibliographic record in alma. postman contains a tab labeled as params where you can enter the key (parameter name like apikey) and value, which can be either a variable in curly brackets like {{apikey}} or a static entry like limit=10. postman divides the params tab into query params and path variables. the main difference between the two is that query params are appended to the end of the url following a question mark (?apikey=12345), while path variables are part of the url and use a placeholder preceded by a colon (:collection_id). the params tab displays the data in a table so it is easy to enter and view parameters. other parts of the request depending on the task at hand or api you’re working with, there are more options that can be set up as part of the request. these additional tabs following the params tab are for the remaining parts of the request: authorization, headers, body, pre-request script, tests and settings. the authorization tab is where you can enter authentication details and postman will automatically place that information into the request. there are several types of authorization to choose from, including api key and oauth. our examples do not make use of this tab, but it can be a helpful way to enter data in one place, similar to variables. collections and folders also contain an authorization tab where you can set the details to be inherited by all of the requests contained in it, if you would prefer to set these details more globally. the headers tab includes auto-generated headers from postman, and is where you may enter your email address, if working with an api that offers a polite option with additional features, meaning that the user identifies themself and in exchange may get a faster response. this is also a place where you can specify what format you want the response returned in, which is useful in working with the alma apis as the default response is returned in xml, but json is the preferred format for working with postman, particularly for parsing the data and using it in scripts, which is discussed shortly. the body tab is where you can enter information that needs to be sent as part of the request to add or update structured data. this often takes the form of json or xml, and can accommodate variables from your workspace. you will likely need to include body data when using a put or post request. the pre-request script tab is where you can enter code snippets written in javascript that are run before the request is sent. these are essential for doing tasks like retrieving a variable that was obtained from a previous request and stored in your active environment. it’s not necessary to know javascript, since postman provides some examples in the right sidebar on this tab that are easily adapted. one example of a pre-request script is this code, which gets a variable from the environment. pm.environment.get("collection_id"); the tests tab contains code written in javascript that is run after the request response is received. like with pre-request scripts, tests can use variables and postman provides code snippets that can be easily adapted. you can carry out tests on response data as well as pass data between requests using this tab. for example, the following code was edited from the existing snippet “status code: code is 200” available in postman. however, the exlibris documentation mentions that a code of 204 indicates that the alma delete electronic collections request was successful, so the snippet was edited to read. pm.test("status code is 204", function() { pm.response.to.have.status(204); }); this will create a green (pass) or red (fail) indicator when the api request is sent in the postman collection runner, discussed below, and makes it easier to identify the api requests that fail and require attention. tests are also where you can record response data to the console, for human viewing or saving as the request is run. the console is accessed at the bottom of the screen in postman and provides details about a request, including the results of tests, for you to review. for example, this code logs the “collection_id” variable in the console: console.log(pm.environment.get("collection_id")); figure 4 shows the result of the console.log code to display the service_id that was retrieved for this request. figure 4. request test and its subsequent display in the console. running collections and folders using csv with all of the basic pieces in place, you could very well go ahead and run individual api calls, one at a time. for some purposes this might be totally fine, but the real power in using a tool like postman is that you can run a series of api calls using variables imported from a csv file. postman has a tool called runner that can work on either the collection or folder level and will run a series of requests iteratively for each line of the csv. this is useful when you have a series of requests to run that only differ in specific variables that can be set in the url path, query parameters or request body. just note, if you are loading a csv, it is important to ensure that the column headers match any variable names you’re using exactly. when you open a collection or folder, a series of actions will be shown near the title that include run (depending on your screen, you may see a play button in a square instead). this will launch the runner within the current level. from there you can select which of the api calls in the collection or folder you wish to run or adjust their order. figure 5 shows the runner screen, set up to run a series of requests. figure 5. the postman runner. in the runner there is an option for loading a data file. loading a file will update the number of iterations to match how many lines are in the file, and a delay can be set in milliseconds if desired. you can also preview the input file to make sure it looks right before running the calls. a note about number strings and csvs one important thing to note is that when data is loaded in the runner, postman makes its best guess as to the format of the data in each of the columns. usually this is fine, but we have found that long strings of digits such as alma’s mms ids, or 10-digit isbns that have vital leading zeros may get reformatted. for postman to see these values as a string and not truncate them, they must be enclosed in quotation marks. to ensure the values are enclosed in single quotation marks ends up being a bit of a convoluted process, but is very important. the value can either be enclosed in quotation marks in excel (or comparable software) or added directly to the column formula if using alma analytics. however, when the data is saved as a csv file, you may then find that your values now have three sets of quotation marks instead of just one. you’ll have to take the intermediary step of removing them in your text editor of choice before using them with the runner in postman. chaining requests the real power from a tool like postman is that not only can you run a series of requests easily, but you can use data returned from one request to serve as variables in a subsequent one, a process known as chaining. this is particularly useful when the first response returns a value that is directly related to a subsequent request in the series. one example, discussed in more detail below, involves the creation of a reading list that requires the course id for the associated course as part of the request’s url. having a chain set up means that the course id created in the first process and then stored as a variable can then be used for the next request’s path to create the associated reading list. as mentioned above, this is achieved through the use of tests and pre-request scripts. to set up chained requests the test on the first request needs to include javascript to set environment variables based on the (in this case) json response body data. then, the next request has pre-request javascript that pulls those same environment variables in to be used again. the postman learning center has a good video explaining this process, that can be found here: https://youtu.be/4fulcou_7wc [5]. use cases cleaning up electronic resources at the university of california, santa cruz, we encountered two scenarios that necessitated a bulk delete of electronic inventory. unfortunately the alma interface does not include an automated or bulk process to delete electronic collections, however, the alma api includes a request to perform the delete. we discovered that by using postman it was possible to perform a bulk delete of collections without building or coding a separate tool to send the needed api requests. our library migrated from millennium to alma in 2018. during migration about 1000 titles were erroneously configured as local electronic collections (databases) instead of local portfolios (ebooks). this led to confusion for users in search results. similarly, in july 2021, the library migrated from a standalone instance of alma to a consortial alma environment with the other university of california campuses, which included a change in our discovery index. this led to many unnecessary electronic collections being activated in our local instance. both of these projects remained on our to-do list because of the large amount of staff time and repetitive clicking required to correct them without using the apis. for the purposes of this discussion, a public version of the setup is available as a postman workspace here: deleting electronic collections. included are detailed descriptions of each part of the request and any code that was repurposed in the pre-request script and tests sections. additionally, these scenarios require an alma api key with read/write permissions for the bibs, electronic, and configuration areas. as a reminder, you should consider setting up an api key for both the sandbox and production environments of alma and using the sandbox key first to test the outcome of your postman projects. deleting database electronic collections in bulk the first scenario required the deletion of about 1000 electronic collections of the “database” type, meaning the collection only has one component, the collection itself, and no service that could contain portfolios. additionally, we wanted to retain the bibliographic record, which was then reused to attach to the proper inventory (a local portfolio) for an electronic book. to complete the task, we conducted a bulk delete of electronic collections using a csv file of ids and the delete electronic collection api request. since this scenario only uses one api request, the setup is fairly simple, and has only two parameters, a pre-request script, and a test to indicate if the request was successful or failed. as described above, once the ids are compiled into a file, you must ensure that each value is enclosed in single quotation marks with the correct label to match the variable. here we used a simple label of “id” to use as the variable in postman. in the pre-request script, which runs before the api request, the javascript snippet (pm.iterationdata.get("id");) “gets” the numbers from the csv file, identified as “id” iteratively and makes them available as a variable {{id}} for the request. the variable value {{id}} is then used as a path variable in params for the collection_id key. this will result in the collection id number from the csv file being entered into the api request url. the additional query param of apikey has a variable {{apikey}} that is set in the environment – either sandbox or production, depending on which environment is currently selected. also, the first part of the url was converted to the environment variable {{base_url}} to shorten the length of the request in this view and make it more readable. once the api request is configured in postman, you can perform the bulk delete by clicking the ellipsis next to the collection name, selecting “run collection”, then “select file”, opening the csv file, and clicking the blue run button. postman will iterate through the ids and return a pass/fail for each. once the run is complete, it is possible to scroll through the results, sort by type or export the results. deleting electronic collections with multiple parts the second scenario required the deletion of all parts of an electronic collection, including the collection’s service (which in this case had no portfolios (ebooks or ejournals)), the collection itself, and the bibliographic record for the collection. to use api requests to do all these deletes, five requests were configured which passed or reused ids that were obtained and used by other requests. starting with the mms id for the bibliographic record, the steps include two get requests, two del (delete) requests and one post (update) request. before starting, an “all titles” itemized set needs to be created in alma and you need to save the set id, which will be used in request five. since there are five requests that will be done in succession, they will need to be connected or chained together using variables and some easily adjusted javascript code. also, because the exlibris apis default to an xml response but postman works best with json, we chose to specify “&format=json” in the query params for the first two api get requests to ensure information will be returned as json. as mentioned above, it is also possible to set this in the header instead. as with scenario one, we created a csv file of ids, this time with the mms id of the bibliographic record and the label “mms” to use as the variable in postman. in the first of the five requests for this scenario, the pre-request script gets the “mms” variable and uses it as the path variable value for the key “mms_id” in params. the tests contain more code than scenario one since the request now returns a response in json and we want to extract some information (the electronic collection id and service id) from it. the first get request contains the following javascript in its test with markup explaining the function of each line of code. // shows if the request was successful "pass" or not "fail" pm.test("status code is 200", function() { pm.response.to.have.status(200); }); // parses the json from the api response const jsondata = pm.response.json(); // logs the jsondata in the console for human viewing or saving console.log(jsondata); // sets the environment variable called "collection_id" // from the json response property called "id" pm.environment.set( "collection_id", jsondata.electronic_collection[0]["id"] ); // logs the "collection_id" variable, the electronic collection id, // in the console for human viewing or saving console.log(pm.environment.get("collection_id")); the subsequent get and del requests make use of pre-request scripts and tests to both retrieve and set variables in the environment and perform the needed deletion of the electronic service and electronic collection inventory in alma. the fifth and final request is a post request that adds the remaining bibliographic records to the previously set up itemized set in alma using the mms id and set id. this request sends data in the body tab in json and includes the mms id as the variable {{mms}} that will be added to the set. while these cleanup scenarios were largely one-time projects, the knowledge gained during the process has already been useful in performing one-off tasks. additionally, the library is considering the use of the alma apis for other types of cleanup, including editing vendor and license information. loading course reserve information at the borough of manhattan community college (bmcc), we maintained a reserves collection in the library, but never managed the collection using aleph (through early 2020) or alma. this meant that information about the courses that textbooks were associated with was nowhere to be found in-system, which led to difficulties when students were looking for course materials. we wanted to both improve the experience for our users and make it easier for library faculty and staff to find the correct information, but had roadblocks in setting things up in a more orthodox way. most importantly, we lacked the available staff time to enter the data manually, and configuring the alma course loader (that could have pulled directly from the college’s student information system) would have involved coordination with multiple other units outside of our library that wouldn’t be able to work on our tight timetable. additionally, it was clear that most of the information that needed to be entered was identical from course to course and list to list. instead, it was decided that we would attempt to load the data from various spreadsheets via the api to circumvent the need for the course loader and work around the staffing limitations in the circulation department. the data on courses and associated books and instructors came primarily from two different sources: from faculty (or their departments) in the college’s student information system (sis) or from the information provided to the college’s bookstore. while neither data source was complete and both had various errors and other issues, it was more than sufficient to get us started as we prepared for the next semester. the initial setup course reserves in alma are comprised of three separate parts: courses, reading lists, and citations. courses can have multiple reading lists, but reading lists cannot be reused between courses without exlibris’ leganto product. citations, however, can appear on multiple reading lists. in an attempt to simplify the inevitable upkeep and maintenance, we decided that courses that used common book(s) would get a single course shell (ex: mat 301), and courses where the texts differed would be split by instructor (instead of section) as the same instructor may teach several sections with the same book (ex: psy 101duncan). we also decided, for the sake of maintenance, that each course we set up would only have a single reading list. since using the api to load data was a totally new thing for us, there was some initial trial and error to get things going, but once that was worked out the project went smoothly. again, for the purposes of this discussion, a public version of the setup described below is here: setting up course reserves, and includes more robust documentation as well as example code. workflow while the bulk of the work for this project was actually looking up and correlating data before it could be loaded (a phenomenon noted here), once postman was configured the data load process itself was somewhat simple and is shown in figure 6. courses would be loaded (with or without instructor data) from csv, and the api request that created the course was chained to one that would generate a reading list based on the variables in the course request and subsequent response. then, with a course id and reading list id in hand, those could be plugged into a separate csv file with corresponding citation information, and then citations would be loaded in a separate process. figure 6. the workflow for loading course reserves via api. the main csv file used in the first part of the process has eight columns (with an additional three if we needed to include instructor information), each corresponding to a variable that was then set up in the request body. as everything else was the same from request to request, the remaining required fields could be fixed in the request body information. it should be noted that the academic department, processing department, and term values and descriptions must match existing values in alma, taken from the corresponding tables in the alma fulfillment configuration. likewise, any added instructor information must match exactly to the instructor information already in alma (capitalization and all). the request body was structured in json with anything coming from the csv input as a variable in curly brackets. in order to get the structure for the request body, an initial get request was run that was then modified with placeholders for the variables. this was done to ensure that everything was structured in a way alma expected. { "code": "{{code}}", "name": "{{name}}", "section": "", "academic_department": { "value": "{{deptcode}}", "desc": "{{deptname}}" }, "processing_department": { "value": "course_unit", "desc": "course_unit" }, "term": [ { "value": "{{termcode}}", "desc": "{{termname}}" } ], "status": "active", "start_date": "{{startdate}}", "end_date": "{{enddate}}", "weekly_hours": 0, "participants": 0, "year": "", "instructor": [], "campus": [], "note": [] } then, regardless of which version of the initial course creation request was run (with or without instructor data), a test was set to grab variables for use in the second request that would generate the associated reading list: var jsondata = json.parse(responsebody); postman.setenvironmentvariable("courseid", jsondata.id); postman.setenvironmentvariable("coursecode", jsondata.code); postman.setenvironmentvariable("coursetitle", jsondata.name); let codestring = json.stringify(pm.environment.get("coursecode")); let namestring = json.stringify(pm.environment.get("coursetitle")); to make use of these variables, the follow-up request to generate the reading list then had pre-request code that more or less replicated the end of the previous test. the reading list code and name were derived from information provided in the first request (already present as variables), and the course id generated in the initial request was then fed into the request url path of the second (reading lists must be linked to a specific course id, which is part of the request url for all reading list-associated requests). with the courses and reading lists created, it was then time to add citations. there are many ways to add citations to reading lists in the alma interface depending on current workflow needs, but for this project many of the citations were attached using the api. this ended up being especially useful in cases where we had multiple instructor-based courses/reading lists that used the same textbook, as the csv file could just have duplicate rows for each of those books with the correct corresponding ids for the courses and lists the books belonged to. this part of the process required retrieving two sets of data from analytics: the existing course and reading list id numbers (which can only be accessed by api or analytics) and the bibliographic data to use for the citation so it would match existing records. the course and reading list id information report used here is shared in the alma analytics community folder under shared folders/reports/constoria/cuny/course reserve and reading list information, and the citation data report can be found under shared folders/reports/constoria/cuny/book data for citation loading via api. by far, the bulk of the work for this phase of the project was ensuring that the texts requested by faculty were actually in our collection and that we had the correct isbn for them. then, once we had the requested titles on the analytics report it was only a matter of putting the correct course and reading list ids in columns alongside the appropriate citations for loading so they could be used as path variables in the request url. the only other major issue at this stage was ensuring that the mms ids of the citations were read as a string (see above) so it wouldn’t be parsed incorrectly. end result in the end we loaded data to create just under 500 courses and related reading lists and a comparable number of citations. while the initial setup process took a fair amount of time (both in learning how to use postman and untangle data), it is expected that future upkeep will be very manageable. now that everything is configured, any new courses or updates to reading lists will be very straightforward to add by using all of the processes that are already in place. further uses while we have outlined two specific use cases for using postman, the tool is flexible and can be used with a wide array of other apis in addition to the alma apis. while it is easy enough to send single one-off get requests for many apis through a browser, when additional parameters are needed or multiple requests need to be made, using a tool like postman makes it much easier for a new or intermediate user to construct and send requests, view responses and make use of the data retrieved. several organizations and vendors also suggest the use of postman to interact with their data and apis, including niso [6], crossref and web of science. as an example, we have also been able to do batch lookups for holdings in the internet archive using their public api and postman, which has saved significant staff time for that particular project. conclusion postman has proven itself an invaluable tool for those of us without a strong coding background who still want to make use of some of the more advanced features that interacting via api would allow. while it does take some effort to get things set up initially (especially if you’re starting completely from scratch), once postman is set up the payoff is great. we came up with much more efficient workflows that were easy to reuse and repurpose for other projects and have already begun to apply what we’ve learned to other tasks like cleaning up old patron fines and working with patron-facing collections. basically, if there’s a project full of repetitive work that could take forever to do one at a time in something like the alma interface, postman is a handy tool to assist in working with apis to leverage the mighty power they hold. additional resources for purposes of demonstration (and to easily share our code and other materials in context), we have set up two read-only public workspaces in postman: deleting electronic collections and setting up course reserves. both contain documentation and annotated code that should prove useful if embarking on a similar project for the first time and to reference while reading this article. citations [1] kluck c. 2019. alma elsewhere: using apis to sprinkle data around your organization. atlanta, ga. http://documents.el-una.org/1913/. [2] api tips for beginners. 2018 feb 27. ex libris developer network. https://developers.exlibrisgroup.com/blog/api-tips-for-beginners/. [3] grouping requests in collections. postman learning center. https://learning.postman.com/docs/sending-requests/intro-to-collections/. [4] postman documentationintroduction. postman learning center. https://learning.postman.com/docs/getting-started/introduction/. [5] postman. 2021. intro to postman | part 5: chain requests. https://www.youtube.com/watch?v=4fulcou_7wc. [6] working with scholarly apis: a niso training series. https://www.niso.org/events/working-scholarly-apis-niso-training-series. about the authors rebecca hyams (rhyams@bmcc.cuny.edu) is the web and systems librarian at the borough of manhattan community college, cuny. she is an avid reader of manga and graphic novels and can easily overwhelm you with title recommendations if given the chance. tamara pilko (tpilko@ucsc.edu) is the electronic resources librarian at the university of california, santa cruz. when done bending metadata to her will, she enjoys hiking and skiing in the mountains or traveling and exploring. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – the moab design for digital object versioning mission editorial committee process and structure code4lib issue 21, 2013-07-15 the moab design for digital object versioning the stanford digital repository has adopted the “moab” design for versioned archiving of digital objects–a locally developed approach that optimizes data transfer, storage, and replication while providing efficient single file retrieval or full object reconstruction for any version of an object. this paper includes a review of various versioning strategies including forward-delta, reverse-delta and content-addressable mechanisms, the pro’s and cons of each, and highlights the relative advantages of the moab design. in our approach, the fixity information of a file manifestation is used as its primary identifier and the filename is treated as metadata. storage and retrieval of an object’s files is faciliated by mapping between a virtual version inventory and the physical location via a file signature catalog. by richard anderson table of contents 1 introduction  3.2 delta versioning  4.1.4 manifests folder  1.1 abstract  3.3 forward-delta versioning  4.2 file manifestations  1.2 the stanford digital repository  3.4 reverse-delta versioning  4.3 a versioning example  1.3 digital object diversity  3.4.1 cdl redd design 5 the moab-versioning gem  1.4 the need for versioning  3.5 content-addressable storage (cas)  5.1 moab manifests 2 versioning requirements  3.5.1 git  5.1.1 the version inventory  2.1 accessioning requirements  3.5.2 boar  5.1.2 the signature catalog  2.2 retrieval requirements  3.6 summary of versioning approaches  5.1.3 version additions report  2.3 fidelity requirements  3.7 tape archive efficiency vs. versioning design  5.1.4 file inventory differences  2.4 efficiency requirements 4 the moab design  5.1.5 algorithm used to compare file inventories  2.5 replication requirements  4.1 online storage folder structure  5.1.6 version metadata  2.6 technology requirements  4.1.1 object folder  5.2 moab storage objects 3 versioning approaches  4.1.2 version folder 6 the sdr preservation workflow  3.1 whole object versioning  4.1.3 data folder 7 conclusion 1 introduction 1.1 abstract the stanford digital repository has adopted the “moab” design for versioned archiving of digital objects–a locally developed approach that optimizes data transfer, storage, and replication while providing efficient single file retrieval or full object reconstruction for any version of an object. this paper includes a review of various versioning strategies including forward-delta, reverse-delta and content-addressable mechanisms, the pro’s and cons of each, and highlights the relative advantages of the moab design. in our approach, the fixity information of a file manifestation is used as its primary identifier and the filename is treated as metadata. storage and retrieval of an object’s files is facilitated by mapping between a virtual version inventory and the physical location via a file signature catalog. 1.2 the stanford digital repository the stanford digital repository (sdr) utilizes a set of interrelated accessioning workflows to assemble digital objects in a temporary workspace, augment them with metadata, make them publicly accessible, and preserve them in long-term storage. sdr uses fedora[1] in the course of this process to hold active copies of metadata in the form of xml datastreams and to specify the relationships between objects. we do not, however, use fedora for the preservation storage component of our repository. instead, we serialize the metadata as xml files and store them along with the original content files in a normal filesystem directory structure, where the object’s home directory is named using the object id, and the metadata and content files are located in subdirectories of that parent. 1.3 digital object diversity our repository aims to support the preservation of a wide variety of digital information being created and used by stanford communities engaged in learning, scholarship, and research. the types of digital objects being preserved include electronic theses and dissertations, images, scanned books and manuscripts, audio files, and video files. some of these objects are quite large, both in number and size of the component files. for example, a manuscript collection already ingested contains objects ranging in size from 1 to 600 gigabytes. objects containing video files may be in the terabyte range. 1.4 the need for versioning some of our collections have proved to have volatile content. for example, a significant number of manuscript pages have needed to be rescanned (in high-resolution mode) to correct problems that were found after initial ingest had occurred. updating of metadata also occurs over time. in either case, the modifications of a digital object that result in a new version can be viewed as a mixture of add, delete, modify, or rename file operations. the repository storage system can most efficiently accommodate digital object versioning by archiving only the changed files and metadata in a way that preserves the history of changes to an object, retains the ability to retrieve earlier versions, and minimizes resource consumption. 2 versioning requirements the versioning approach selected for the sdr had to meet the following set of basic requirements. 2.1 accessioning requirements the pipeline between a depositing agent and the accessioning system should be flexible enough to accommodate various modes of new version submission. a depositor should be able to provide a full set of files comprising a new version (which may include files carried over from the previous version), or they should be able to just send only the files that have been added or modified and/or directives about which files should be deleted or renamed. 2.2 retrieval requirements the mechanism for retrieval of a digital object should have a similar flexibility. one should be able to retrieve all or only a subset of the files from any version. it must be possible to retrieve, reconstruct, and deliver an exact copy of a digital object as it was originally submitted. the original filesystem properties for a file, such as relative pathname and last modification date should be faithfully preserved as metadata along with the file. 2.3 fidelity requirements the preservation storage system should avoid introducing file content changes that cannot be reversed or recovered from. any form of lossy compression is of course to be avoided, as is line-ending normalization of text files. and even lossless compression has been shown to exacerbate the preservation risk associated with bit rot. according to (wright, miller, & addis, 2009) an uncompressed .wav file with .4% errors will exhibit barely noticeable differences from the undamaged original whereas a similarly damaged mp3 file will not even open. in order to confirm the fidelity of storage, checksums (message digests) are used to verify that file corruption has not occurred. the checksum for a file at the time of submission should be a constant that carries through to the ultimate storage location. for extra insurance the system should allow more than one checksum to be tracked for each file. 2.4 efficiency requirements the storage system must accommodate all sizes of digital objects in a way that minimizes disk space consumption as well as the cpu and network resources required for such operations as file transfer, fixity checking, replication, and tape archiving. greater efficiency of processing correlates with increased throughput in the ingest workflow. unintentional redundant storage of content files should be avoided. this is especially true for large binary files, such as video files or high-resolution tiffs. if a new version of an object has only minor changes, then it should not be necessary to re-archive all the files in the object. if only a filename has changed, but not the internal content, then it should be possible to record that fact as metadata without creating a duplicate copy of the file. 2.5 replication requirements a standard feature of any preservation system is the creation of extra copies of each digital object, either at other disk locations or on geographically dispersed tape media. the replication operation should consume a minimum of processing, i/o, and storage resource needed for making file copies or performing media migration. sdr currently uses the rsync[2] file transfer utility to copy files from one location to another. it has the benefit that it can synchronize data in two locations by detecting and copying only the file differences from one location to another. it uses stat to figure out which files have changed and compare-by-hash to determine which data blocks are different between two locations so that it can send only the blocks with different hashes. but even rsync’s speed can suffer when one is replicating a digital object in its entirety. at a minimum, rsync needs to stat all the files in the specified directory structure and examine the file properties (size and date), which can take an appreciable time for an object containing a large number of files. replication of a new version’s files can therefore be made more efficient by using a directory structure that isolates a new version’s changes to a known sub-location, reducing the number of files whose properties need to be compared. archiving of versioned objects onto tape is also impacted by the way in which digital object storage is structured. archiving of the initial version of an object is straightforward, but archiving of a subsequent version requires a design that facilitates the copying of only the new or changed files. 2.6 technology requirements we are currently minimizing costs through use of open source software and commodity hardware components as much as possible for our system. we have in the past experimented with a variety of vendor-supplied products, which have been quite promising. but we had difficulty addressing the long-term finances for scaling those systems to the size needed for our anticipated volume of content. we also need to minimize our system’s dependency on vendor-supplied solutions. we should not need to recover from the business failure of a vendor, or from the obsolescence of a technology. 3 versioning approaches let us now turn our attention to a review of versioning approaches and then examine some implementations that illustrate those options. this list is by no means exhaustive, but summarizes many of the technologies that have been discussed by digital curators. 3.1 whole object versioning a foolproof mechanism for storing multiple versions of an object would be to ingest each new version as if it were a completely new set of files, and keep each version’s files in its own separate bucket. figure 1: whole object versioning this is a simple design, which should be easy to implement and would support fast reconstruction of any version. however, it entails a high level of file duplication and resource consumption. 3.2 delta versioning one way to reduce duplication of file storage is for each new version to store only the changes that have occurred between versions. this mechanism has been used in virtually all software revision control systems. figure 2: delta versioning this design has a much lower level of file duplication than the whole version option, but does require more complex algorithms for storage and retrieval. a file rename operation can be treated as a combination of a file delete and a file add. there are 2 varieties of delta versioning: forward-delta and reverse-delta. 3.3 forward-delta versioning in forward-delta versioning, you start by creating a storage bucket containing all of the initially ingested files. when later versions are added, each new version’s bucket contains only the files that have been added or modified since the previous version (plus a list of which files have been deleted). figure 3: forward-delta versioning this approach requires less work to add a new version and has less impact on replication and tape archive, but requires effort to reconstruct later versions, especially if the number of versions is large. to illustrate an insert file composite change we start with a version 1 container holding all the files comprising the original submission. we need to insert a new file as page-3.tif in place of the existing file of that name, and move the remaining pages into new positions in the sequence. broken down to smaller steps, this operation is a combination of a file addition and a couple of file renames. the simplest way to implement this would be to delete page-3.tif and the files that follow it in sequence, and then add in the new file and the renamed files. figure 4: forward-delta insert file to retrieve a copy of the complete version 2 object, one would copy the files from version 1, then delete the version 2 deletes (pages 3 and 4), and finally add pages 3 to 5. if more versions exist, then the process must be repeated for each of the new versions. 3.4 reverse-delta versioning reverse-delta versioning is the inverse of forward-delta versioning. in this approach, the storage bucket for the most recent version of an object will contain all the files comprising that version, and the storage buckets for previous versions will contain only the files that were “left behind” when a new version was created. figure 5: reverse-delta versioning this approach requires less work to reconstruct the latest version, but more work to add a new version. and there is a greater impact on replication and tape archiving. to demonstrate this, let’s again begin with the same version 1 object, and then see what the result is when we add a new version using the insert file scenario. figure 6: reverse-delta insert file version 2’s container now holds copies of all version 2 files, while version 1’s storage area has been revised to contain only the information required to restore version 1 using version 2 as a starting point. 3.4.1 cdl redd design the california digital library has implemented a very rigorous and well-documented storage mechanism (storage micro-service[3]) based in part on a framework called dflat[4] and a reverse-delta implementation called redd (reverse directory deltas)[5]. below is a skeleton example of a dflat/redd/dnatural directory structure that is holding 2 versions of a digital object. v002 is the latest directory and contains a full complete version. v001 is a reverse-delta version. <object-identifier>/ # dflat home directory [ current.txt ] # pointer to current version (e.g. ‘v002’) [ dflat-info.txt ] # dflat properties file v001/ # reverse-delta version [ manifest.txt ] # version manifest delta/ # redd directory [ add/ ] # files to be added relative to subsequent [ delete.txt ] # files to be deleted relative to subsequent v002/ # current version [ manifest.txt ] # version manifest full/ # dnatural directory [ consumer/ ] # consumer-supplied files directory [ producer/ ] # producer-supplied files directory [ system ] # system-generated files directory the cdl redd implementation works by first adding a new complete full version. at this point the latest version and the previous version are both full versions. analysis is then performed to determine how one would generate the previous version from the new latest version, and a new reverse-delta version is created to replace the previous full version. the sdr has imitated aspects this directory hierarchy in our own storage layer. we did, however, have concerns about the redd mechanism: • each time a new version is added there must be a subsequent backup of the new (full) version’s files as well as the now scaled-back previous version. this requires a complete archive of both directories to tape, and therefore a much higher amount of network traffic and tape i/o than would be required by a forward-delta, or content-addressable mechanism. • there does not seem to be a provision for renaming files across versions other than to delete the original file and then add in the file with the new name, resulting in duplicate storage of the content under two different names. 3.5 content-addressable storage (cas) the phrase “content-addressable storage”[6] covers a broad range of meaning, but in the case of a version control system it refers to a practice of using a file’s checksum value as an identifier and locator for the file in the storage system. this allows any given file instance to be stored only once using the checksum value as the key to its location, and enables multiple version manifests to reference that file location regardless of the filename used in a given version manifest. this usage is also known by the phrase “compare-by-hash”[7] (black, 2006), since the usual method for creating a suitable checksum (aka “message digest”) is to use a cryptographic hash function.[8][9] figure 7: content-addressable storage the cas approach has a simple conceptual design that eliminates file storage duplication and facilitates easy reconstruction of any object version. it introduces, however, a dependency on the version manifests, since that file inventory becomes the only authoritative source for keeping track of which files are part of each version and what are their “real” filenames. another factor to consider has to do with statistical probabilities. what is the potential for 2 files having differing contents to hash to the same checksum value? [10] if using sha1 digests, the odds are 1 in 2^160 (~ 10^48). for sha256 the odds are 1 in 2^256 (~10^77), which is comparable to the number of atoms in the entire universe.[11] sha2 is also much more immune to brute-force attempts to ‘crack’ the hash algorithm. the concern about algorithm cracking is mainly focused on the risk of someone figuring out how to create a bogus password or digital signature, but could also be raised in the context of a digital archive if there is a worry about fraudulent alteration of preserved digital assets. the probability that a collection of unique files may contain a pair of files with the same checksum is actually larger than the above number, but would only be worrisome if you had an extremely large set of files. if using sha256 and your repository’s size were to approach 2^128 (~10^38) files, then duplicate checksums would begin to be observed. this is often called the “birthday paradox” because in a group of 57 people, there is a 99% probability that 2 of those people will have the same birthday. i will next examine 2 existing open source implementations of content-addressable storage that have been suggested as viable candidates for digital object versioning (git and boar), before finally describing the moab design. 3.5.1 git git[12] is a popular distributed version control system that was initially created by linus torvalds for tracking changes to the linux operating system. due in part to its operating system heritage, it has an object model that looks very similar to a file directory hierarchy. in this design blobs are used to store the content of files, trees are used to represent directories, and commits are used to tag versions.[13] figure 8: git object model git does not store files as they are received, but tacks on a header containing the string blob followed by a space, the length of the data in bytes, and a null byte. a tree functions similarly to a directory. it contains a list of all filenames (and subdirectories) contained in a directory at a given point in time along with the file permissions for those items. a commit is a special artifact that contains metadata about a revision (version) of the project. sha1 checksums are calculated for each for each object stored in git. these checksum values are used to express relationships as well as to locate the listed files and sub-directories in the storage area over time as revisions are made to a software project (and committed to the repository) a network structure evolves such that any given version of the project consists of a combination of old and new blob and tree references.[14] figure 9: git revisions note that content data stored in git is usually also compressed, initially into separate loose files. figure 10: git loose file and eventually multiple loose items are combined into larger packfiles. figure 11: git packfile in spite of its sophisticated design, git seems to be an undesirable choice for preservation storage of digital objects. not only does it alter the contents of files in several ways, it also has repeatedly been observed to suffer from poor performance when dealing with large binary files. and because git is a decentralized version control system, one needs to have a local copy of a full object and all its versions before commits of new versions can be made. we need to be able to transmit only a version’s changed files to the storage layer. 3.5.2 boar boar[15] is being developed by an author who felt that none of the existing open source backup and version control systems were a good fit for the management of collections of binary files. this software has a simple, but effective, storage design. in the boar design, each repository (storage node) has a top-level blobs directory that contains the entire repository’s content files, each of which has been renamed using the value of the file’s md5 checksum. a simple hierarchy is used to subdivide the blobs folders in a manner similar to a git repository. repository blobs bc bc7b0fb8c2e096693acacbd6cb070f16 sessions <snapshot name> bloblist.json (map between filename paths and checksums) session.json (metadata about the snapshot) <snapshot name> bloblist.json session.json the application uses a forward-delta design with a linked-list aspect, wherein a series of session snapshots are stored in a flat structure under a top-level sessions directory. each snapshot contains only the most recent changes to a filesystem object being backed up, plus a pointer to the previous snapshot of the given session. it was somewhat challenging for me to figure out how this application could be applied to the preservation of digital objects. boar’s vocabulary of session, blob, and snapshot can be correlated to our terminology of digital object, data file, and version, but the need to create mental mappings between these concepts makes discussion of operations and storage somewhat confusing. the main point to observe is that all of an object’s files are stored in a pooled location that is content-addressable via file checksum values, and that the history of changes to the object are stored in a sequence of version deltas (the chain of bloblist.json files) although closer in functionality to sdr’s needs than git, and a source of design insights, boar in its present incarnation does not seem to be a good fit for our repository. the software is designed for incremental backup of a working directory to a repository location on the same machine, which is a different concept from explicit archiving of a digital object version to a secure location. the pooling of all files in a single area is appealing on one level, but is not compatible with our requirement to segregate a new version’s files in a way that simplifies replication. 3.6 summary of versioning approaches here are the pros and cons of each of the versioning approaches we have examined so far. versioning mechanism file fixity no file duplicates replication efficiency version reconstruction forward-delta good fair (file renames) good good for first poor for latest reverse-delta (cdl redd) good fair (file renames) poor (latest version is always full) good for latest poor for earlier cas (git) poor (headers & pack files) good poor (pooled data) good cas (boar) fair (md5) good poor (pooled data) good for first poor for latest as you can see, there is no clear winner, but each of these system shines in at least one aspect. note that a row in the table is included for forward-delta, but i have not examined any implementations that utilize that strategy. 3.7 tape archive efficiency vs. versioning design in many of the versioning approaches we have discussed, addition of a new object version could trigger duplicate tape storage of many or most of an object’s files, requiring a proportionately higher consumption of system and network i/o resources. this inference is based on the assumption that tape archiving will use replication at the directory level. in the cdl reverse-delta (redd) design, the directory for an object’s latest version always contains copies of all files comprising that new version. since this is in essence a new full object, the tape archive mechanism will copy all those files, even if they were previously archived from another location. also, since the previous version has been converted from a full version to a reverse-delta version, that previous version must be re-archived as well. in the git and boar designs, all of an object’s files are pooled in a common data store. changing the content of this pool by addition of new files would likely require that a new tape archive be made of the common pool. the forward-delta approach minimizes the volume of files that would need to be archived to tape. the new folders that are created only contain new files, and the previously existing folders of the object are not altered by versioning changes. so it looks like the ideal would be to come up with a design that combines the replication efficiency of the forward-delta approach with the reconstruction efficiency of the content-addressable approach. 4 the moab[16] design stanford has developed a storage architecture, dubbed the moab design, which aims to meet the sdr’s functional requirements by incorporating what we feel are the best aspects of the versioning mechanisms previously discussed. in this design: • a version inventory manifest specifies a version’s contents each object version has a full version inventory manifest that is used to specify the complete set of files that comprise a digital object version. additional manifests are derived from the object’s complete set of version inventories: a signature catalog provides a union list of all file manifestations that have been preserved from all versions of the digital object. a version additions report lists which files were added to the object’s catalog in a given version. and a file inventory differences report shows the details of the changes that occurred between any two given versions. • fixity data is used for file deduplication at least two of md5, sha1, or sha256 hash values are generated for each file ingested. these checksums are used along with the file size to determine whether a newly submitted file is a duplicate of another file in the same version or is an exact copy of a file that was previously ingested. this deduplication is facilitated by the use of a master file signature catalog that collects this data across versions. this file also contains the sdr storage location of all the object’s files making it possible to quickly reconstruct any version of an object using that version’s full inventory together with the catalog. • files are treated as immutable the application layer does not modify the internal bytestreams of files in the preservation system. whatever benefit might be obtained performing any sort of file compression or other alteration of curated content files would be outweighed by the cost of processing cycles and the risk of data loss. • files are stored using original filenames a preliminary design mimicked git and boar by storing data files in a common pool after renaming them using the checksum value. this raised concerns about the resulting dependency on manifest files to preserve the filename information. the latest iteration of the design retains an object version’s original filenames during ingest. • each version’s new files are stored in a separate data bucket the cdl dflat structure provides a good model for storage of a digital object and its versions, such as the use of v001, v002, etc. subdirectories within an object’s directory. the moab design imitates this framework to keep each version’s new data files separate from those of previous versions. note that a “new file” is defined in this context as a file manifestation whose checksum did not occur in any previous version of the digital object. a file instance having a filename that was present in a previous version, but which resulted from a modification of that old file will hash to a new digest value and thus be considered a new file. • file replication is simplified the segregated storage of version data facilitates disk and tape replication with a minimum of processing and content duplication. • hash collision risk is minimized because each digital object has its own separate file storage structure (rather than a common file pool), the risk of checksum collisions is very low. 4.1 online storage folder structure the file folder structure used for a digital object is similar in appearance to the cdl dflat design. ab123cd5678 v0001 data content title.jpg intro.jpg page1.jpg page2.jpg page3.jpg metadata versionmetadata.xml descmetadata.xml identitymetadata.xml manifests versioninventory.xml signaturecatalog.xml versionadditions.xml fileinventorydifference.xml manifestinventory.xml v0002 data content page2.jpg metadata versionmetadata.xml technicalmetadata.xml manifests versioninventory.xml signaturecatalog.xml versionadditions.xml fileinventorydifference.xml manifestinventory.xml 4.1.1 object folder each digital object is stored in a separate filesystem directory named using the object id. within the repository as a whole, the path to an object is a tree structure derived by segmenting the object id, such as /ab/123/cd/4567/ab123cd4567. 4.1.2 version folder each object-level folder is subdivided into version folders, named v0001, v0002, etc. the v0001 folder holds the initial version of the digital object. additional sub-directories of the form “v” hold the files of subsequent versions of the object. the directory names are left padded with zeros so that the directory pathname is 5 characters long and will sort into the correct chronological sequence. each version folder contains a data folder and a manifests folder. 4.1.3 data folder the first version’s data folder will usually contain the majority of the object’s files. the data folder of subsequent versions will contain only those new files that were added in that version or which resulted from modifications of an older version’s files. in the sdr this data folder is subdivided into content and metadata subdirectories. 4.1.4 manifests folder each version has a manifests folder containing file inventories, a location index, and version differences reports. • the versioninventory.xml manifest is used to record the complete set of files that comprise a digital object version. this information is required to reconstruct the full version of the object. • the signaturecatalog.xml manifest provides a union list of files from the current and all previous object versions. it serves as a lookup table mapping file signature to storage location. • the versionaddtions.xml manifest lists the subset of files that were added to the object’s data folder in the current version. • the fileinventorydifferences.xml report documents the details of all changes that occurred between the current and previous version. • the manifestinventory.xml file holds fixity information about the other manifest files. the moab manifests all contain size and checksums values for each of the files that are listed in them. the manifests therefore serve a fixity verification role similar to the manifest files that are a part of the bagit specification.[17] the versionadditions manifest is most similar to bagit’s manifest-xxx.txt “tag” files used for payload verification, and the manifestinventory is similar in function to the tagmanifest-xxx.txt files that list checksums for the other tag files. but this is where the similarity ends. the moab manifests have a structure in which the file fixity or signature is the key to a file’s identity, instead of the filename. 4.2 file manifestations the moab design is based in large part on the concept of a file manifestation (in the frbr[18] sense of manifestation). in this view, a file’s uniqueness is defined by its bytestream content, which will have a unique signature (= file fixity). multiple file instances might share the same content (and thus signature) even if they do not share the same name (filesystem path) and timestamp. in the version inventory files used in the moab design, a file manifestation consists of a combination of a signature and at least one file instance. figure 12: file manisfestation above is a visual depiction of a file manifestation having one instance. below is an example of a manifestation having 2 file instances. figure 13: file with 2 instances this latter file manifestation would be represented in a version inventory by the following xml stanza: <file> <filesignature size="32915" md5="c1c34634e2f18a354cd3e3e1574c3194" sha1="8a354cd3e3e17328c364b2ea0b4a79c507ce915ed"/> <fileinstance path="resume.txt" datetime="2012-01-01"/> <fileinstance path="resume-copy" datetime="2012-03-04"/> </file> 4.3 a versioning example the figures below illustrate the basic logic and relationships of core components of the moab versioning design. in the first diagram, version 1 of an object has been ingested into the repository storage area. the v0001/data/content folder contains the physical files. the v0001/manifests/versioninventory.xml file contains a list of all the files in this version together with the signature of each of those files. the v0001/manifests/signaturecatalog.xml file contains a mapping from each file signature to the physical location in which the file is stored. figure 14: version 1 in version 2 of the object, one file (intro) is removed, and the content of page1 is modified. there is only one new physical file (page1) which is placed into /v0002/data/content. the v0002/manifests/versioninventory.xml file is written to record the new composition of the object, and the v0002/manifests/signaturecatalog.xml file is extended to document the storage location of the new page1 file. the arrows show the pathway used to reconstruct the full version 2 object. figure 15: version 2 finally, a version 3 is added in which a new page3 is inserted in the file sequence, with a domino effect that the old page3 must be renamed to page4. the version inventory of the new version doesn’t need to know any of the previous history. it simply documents which file name/signature combinations define the new version. as before, the signature catalog only needs to be updated with the signature/location data for the newly added page3 file. reconstruction of version 3 remains a simple task of looking up storage locations via the signature catalog. figure 16: version 3 5 the moab-versioning gem a source code library has been written which implements a set of tools which can be used to create, parse, and work with moab manifest files; and facilitates the storage of digital object versions in a filesystem structure. this library has been written in the ruby programming language, and is used internally by the stanford digital repository. the code project has been published on github at https://github.com/sul-dlss/moab-versioning api documentation can be viewed at http://rubydoc.info/github/sul-dlss/moab-versioning/master/frames the remainder of this section reviews the classes provided by this code library, providing more detailed descriptions and examples of the various moab manifest files. the manifest field names are mostly self-explanatory, but you can look up additional definitions via the above links. for examples of code that interacts with manifest files and storage structures, see the rspec files at: https://github.com/sul-dlss/moab-versioning/tree/master/spec/unit_tests if you download the project and run the rspec test suite, then the fixtures folder of the spec directory will be completely filled out with data files that can be examined for more insight. this gem is still being packaged and hosted internally at stanford university libraries, but if a community of users were to coalesce, then we could change that to use rubygems.org. 5.1 moab manifests 5.1.1 the version inventory each digital object version has a version inventory (serialized as versioninventory.xml) that is an instance of the fileinventory schema wherein the type attribute is set to “version” (this schema is shared by several document types). every version inventory contains a complete list of all files comprising the specified version of a digital object. in the sdr examples below, the data storage area is divided into content and metadata subdirectories, which is mirrored by the content and a metadata filegroups in the inventory. within each filegroup is an array of file manifestation (<file>) elements, each of which represents a point-in-time snapshot of a given file’s properties. the fixity data for the file is stored in a filesignature element, while the filename and last modification date are stored in one or more fileinstance elements (since copies of a given file may be present in multiple locations within an object). <?xml version="1.0" encoding="utf-8"?> <fileinventory type="version" objectid="druid:jq937jp0017" versionid="3" inventorydatetime="2012-04-19t12:12:48z" filecount="7" bytecount="187851" blockcount="191"> <filegroup groupid="content" datasource="/users/rnanders/code/ruby/moab-versioning/spec/fixtures/data/jq937jp0017/v3/content" filecount="5" bytecount="171902" blockcount="170"> <file> <filesignature size="32915" md5="c1c34634e2f18a354cd3e3e1574c3194" sha1="0616a0bd7927328c364b2ea0b4a79c507ce915ed"/> <fileinstance path="page-1.jpg" datetime="2012-03-26t15:35:15z"/> </file> <file> <filesignature size="39539" md5="fe6e3ffa1b02ced189db640f68da0cc2" sha1="43ced73681687bc8e6f483618f0dcff7665e0ba7"/> <fileinstance path="page-2.jpg" datetime="2012-03-26t15:58:53z"/> </file> <file> <filesignature size="39450" md5="82fc107c88446a3119a51a8663d1e955" sha1="d0857baa307a2e9efff42467b5abd4e1cf40fcd5"/> <fileinstance path="page-3.jpg" datetime="2012-03-26t15:23:36z"/> </file> <file> <filesignature size="19125" md5="a5099878de7e2e064432d6df44ca8827" sha1="c0ccac433cf02a6cee89c14f9ba6072a184447a2"/> <fileinstance path="page-4.jpg" datetime="2012-03-26t15:24:39z"/> </file> <file> <filesignature size="40873" md5="1a726cd7963bd6d3ceb10a8c353ec166" sha1="583220e0572640abcd3ddd97393d224e8053a6ad"/> <fileinstance path="title.jpg" datetime="2012-03-26t14:15:11z"/> </file> </filegroup> <filegroup groupid="metadata" datasource="/users/rnanders/code/ruby/moab-versioning/spec/fixtures/data/jq937jp0017/v3/metadata" filecount="2" bytecount="15949" blockcount="21"> <file> <filesignature size="3046" md5="a60bb487db6a1ceb5e0b5bb3cae2dfa2" sha1="edefc0e1d7cffd5bd3c7db6a393ab7632b70dc2d"/> <fileinstance path="descmetadata.xml" datetime="2012-03-26t17:43:49z"/> </file> <file> <filesignature size="12903" md5="ccb5bf2ae0c2c6ad0b89692fa1e10145" sha1="3badb0d06aef40f14e4664d2594c6060b9e9716b"/> <fileinstance path="identitymetadata.xml" datetime="2012-03-26t17:44:48z"/> </file> </filegroup> </fileinventory> a filegroup to be added to a version inventory can be generated by traversing a directory that contains the files being inventoried. this capability is provided by the filegroup#group_from_directory method, which is also utilized by the fileinventory#inventory_from_directory method. a special case of the latter method will create both a content and a metadata filegroup from a directory that contains two subdirectories having those names. a filegroup can also be generated by transformation of a local file inventory in another format. at stanford a contentmetadata datastream contains all the information needed to create the content filegroup. a method to perform this transformation is provided in the stanford:: contentinventory#group_from_cm method. 5.1.2 the signature catalog a digital object’s signature catalog (serialized as signaturecatalog.xml) is derived from a filtered aggregation of all the version inventories of a digital object. it has a file signature entry for every file manifestation ever ingested for the object, along with a record of the sdr storage location that was used to preserve a copy of the manifestation. once this catalog has been populated, it has multiple uses: • an index of file signatures is used to determine which files of a newly submitted object version are duplicates of files previously ingested. (when a new version contains a mixture of new files and files carried over from the previous version we only need to store the files from the new version that have unique file signatures.) • reconstruction of an object version requires a combination of the version’s file inventory and signature catalog. • the catalog can also be used for performing consistency checks of repository storage files. <signaturecatalog objectid="druid:jq937jp0017" versionid="3" catalogdatetime="2012-04-19t12:12:48z" filecount="11" bytecount="295970" blockcount="309"> <entry originalversion="1" groupid="content" storagepath="intro-1.jpg"> <filesignature size="41981" md5="915c0305bf50c55143f1506295dc122c" sha1="60448956fbe069979fce6a6e55dba4ce1f915178"/> </entry> <entry originalversion="1" groupid="content" storagepath="intro-2.jpg"> <filesignature size="39850" md5="77f1a4efdcea6a476505df9b9fba82a7" sha1="a49ae3f3771d99ceea13ec825c9c2b73fc1a9915"/> </entry> <entry originalversion="1" groupid="content" storagepath="page-1.jpg"> <filesignature size="25153" md5="3dee12fb4f1c28351c7482b76ff76ae4" sha1="906c1314f3ab344563acbbbe2c7930f08429e35b"/> </entry> <entry originalversion="1" groupid="content" storagepath="page-2.jpg"> <filesignature size="39450" md5="82fc107c88446a3119a51a8663d1e955" sha1="d0857baa307a2e9efff42467b5abd4e1cf40fcd5"/> </entry> <entry originalversion="1" groupid="content" storagepath="page-3.jpg"> <filesignature size="19125" md5="a5099878de7e2e064432d6df44ca8827" sha1="c0ccac433cf02a6cee89c14f9ba6072a184447a2"/> </entry> <entry originalversion="1" groupid="content" storagepath="title.jpg"> <filesignature size="40873" md5="1a726cd7963bd6d3ceb10a8c353ec166" sha1="583220e0572640abcd3ddd97393d224e8053a6ad"/> </entry> <entry originalversion="1" groupid="metadata" storagepath="descmetadata.xml"> <filesignature size="3046" md5="a60bb487db6a1ceb5e0b5bb3cae2dfa2" sha1="edefc0e1d7cffd5bd3c7db6a393ab7632b70dc2d"/> </entry> <entry originalversion="1" groupid="metadata" storagepath="identitymetadata.xml"> <filesignature size="12903" md5="ccb5bf2ae0c2c6ad0b89692fa1e10145" sha1="3badb0d06aef40f14e4664d2594c6060b9e9716b"/> </entry> <entry originalversion="2" groupid="content" storagepath="page-1.jpg"> <filesignature size="32915" md5="c1c34634e2f18a354cd3e3e1574c3194" sha1="0616a0bd7927328c364b2ea0b4a79c507ce915ed"/> </entry> <entry originalversion="2" groupid="metadata" storagepath="technicalmetadata.xml"> <filesignature size="1135" md5="d74bfa778653b6c1b285b2d0c2f07c5b" sha1="0ee15e133c17ae3312b87247adb310b0327ca3df"/> </entry> <entry originalversion="3" groupid="content" storagepath="page-3.jpg"> <filesignature size="39539" md5="fe6e3ffa1b02ced189db640f68da0cc2" sha1="43ced73681687bc8e6f483618f0dcff7665e0ba7"/> </entry> </signaturecatalog> a signature catalog is created by using the signaturecatalog.new method and assigning the digital object id. the signaturecatalog#update method is run against the first version’s file inventory, which generates an entry in the catalog for each unique file. when a subsequent version of the digital object is deposited in the repository, the #update command is again run, but this time it only creates new entries for files whose signatures were not already present in the catalog. each new version’s manifests folder holds an updated copy of the signaturecatalog.xml file that was created when that new version was ingested. 5.1.3 version additions report a version additions report is an instance of the fileinventory schema (with type=”additions”) that lists which file manifestations were newly added to a given version. because it is generated by applying a new version’s file inventory against the previous version’s signature catalog, it is of secondary importance as a preservation artifact. it serves the following purposes: • provides a human-readable summary of new files • provides a file manifest for use in validating checksums of a given version’s storage files. <fileinventory type="additions" objectid="druid:jq937jp0017" versionid="2" inventorydatetime="2012-04-19t12:12:48z" filecount="2" bytecount="34050" blockcount="37"> <filegroup groupid="content" datasource="" filecount="1" bytecount="32915" blockcount="33"> <file> <filesignature size="32915" md5="c1c34634e2f18a354cd3e3e1574c3194" sha1="0616a0bd7927328c364b2ea0b4a79c507ce915ed"/> <fileinstance path="page-1.jpg" datetime="2012-03-26t15:35:15z"/> </file> </filegroup> <filegroup groupid="metadata" datasource="" filecount="3" bytecount="2669" blockcount="4"> <file> <filesignature size="1135" md5="d74bfa778653b6c1b285b2d0c2f07c5b" sha1="0ee15e133c17ae3312b87247adb310b0327ca3df"/> <fileinstance path="technicalmetadata.xml" datetime="2012-03-28t14:04:24z"/> </file> </filegroup> </fileinventory> a version additions report is created by using the signaturecatalog#additions method of the previous version’s catalog with the new version’s file inventory as an input parameter. 5.1.4 file inventory differences a file inventory differences report compares two fileinventory instances based primarily on file signatures and secondarily on file pathnames. although the usual use will be to compare the content of 2 different temporal versions of the same object, it can also be used to verify a manifest document against an inventory harvested from a directory. the report is subdivided into sections for each of the file groups that compose the inventories being compared. <fileinventorydifference objectid="druid:jq937jp0017" differencecount="5" basis="v1" other="v3" reportdatetime="2012-04-19t12:12:48z"> <filegroupdifference groupid="content" differencecount="4" identical="2" renamed="1" modified="1" deleted="1" added="1"> <subset change="identical" count="2"> <file change="identical" basispath="title.jpg" otherpath="same"> <filesignature size="40873" md5="1a726cd7963bd6d3ceb10a8c353ec166" sha1="583220e0572640abcd3ddd97393d224e8053a6ad"/> </file> <file change=" identical " basispath="page-2.jpg" otherpath="same"> <filesignature size="19125" md5="a5099878de7e2e064432d6df44ca8827" sha1="c0ccac433cf02a6cee89c14f9ba6072a184447a2"/> </file> </subset> <subset change="renamed" count="1"> <file change="renamed" basispath="page-3.jpg" otherpath="page-4.jpg"> <filesignature size="39450" md5="82fc107c88446a3119a51a8663d1e955" sha1="d0857baa307a2e9efff42467b5abd4e1cf40fcd5"/> </file> </subset> <subset change="modified" count="1"> <file change="modified" basispath="page-1.jpg" otherpath="same"> <filesignature size="25153" md5="3dee12fb4f1c28351c7482b76ff76ae4" sha1="906c1314f3ab344563acbbbe2c7930f08429e35b"/> <filesignature size="32915" md5="c1c34634e2f18a354cd3e3e1574c3194" sha1="0616a0bd7927328c364b2ea0b4a79c507ce915ed"/> </file> </subset> <subset change="deleted" count="1"> <file change="deleted" basispath="intro.jpg" otherpath=""> <filesignature size="41981" md5="915c0305bf50c55143f1506295dc122c" sha1="60448956fbe069979fce6a6e55dba4ce1f915178"/> </file> </subset> <subset change="added" count="1"> <file change="added" basispath="" otherpath="page-3.jpg"> <filesignature size="39539" md5="fe6e3ffa1b02ced189db640f68da0cc2" sha1="43ced73681687bc8e6f483618f0dcff7665e0ba7"/> </file> </subset> </filegroupdifference> <filegroupdifference groupid="metadata" differencecount="1" identical="2" renamed="0" modified="1" deleted="0" added="0"> <subset change="identical" count="2"> <file change="identical" basispath="descmetadata.xml" otherpath="same"> <filesignature size="3046" md5="a60bb487db6a1ceb5e0b5bb3cae2dfa2" sha1="edefc0e1d7cffd5bd3c7db6a393ab7632b70dc2d"/> </file> <file change="identical" basispath="identitymetadata.xml" otherpath="same"> <filesignature size="12903" md5="ccb5bf2ae0c2c6ad0b89692fa1e10145" sha1="3badb0d06aef40f14e4664d2594c6060b9e9716b"/> </file> </subset> <subset change="renamed" count="0"/> <subset change="modified" count="1"> <file change="modified" basispath="technicalmetadata.xml" otherpath="same"> <filesignature size="1619" md5="b886db0d14508884150a916089da840f" sha1="b2328faaf25caf037cfc0263896ad707fc3a47a7"/> <filesignature size="1376" md5="44caefdbaf92f808c3fd27d693338c40" sha1="eb6f9c6539c90412bb4994d13311cc8dfe37c334"/> </file> </subset> <subset change="deleted" count="0"/> <subset change="added" count="0"/> </filegroupdifference> </fileinventorydifference> to generate a file inventory differences report, use the fileinventorydifference#compare method, supplying it with two file inventories as parameters. a convenience method, fileinventorydifference#compare_with_directory, provides a simple interface for comparing an existing inventory against a directory containing physical files. 5.1.5 algorithm used to compare file inventories in the diagram below, the word “manifest” is used as a synonym for file inventory. figure 17: compare manifests in the moab design, the inventories for two different versions contain no indication of the history that occurred between those two versions. the differences must therefore be inferred by performing a comparison analysis between two version inventories. this comparison is a 2-step process. we first compare the sets of signatures present in the two inventories, determining which signatures are present in both versions and which others are unique to either the old version or the new version. finally, the lists of signatures that are the output of the first step of the comparison are further analyzed to report the detailed nature of the differences between the two versions of an object: a.  if two files have the same signature in both manifests, and the same filename, then we consider the files to be identical. b.  if two files have the same signature in both manifests, but the filenames differ, then we conclude that a file rename has occurred. c.  if a signature only occurs in the newer manifest, and the filename is unique to that version, then we conclude that this is a newly added file. d.  if a signature only occurs in the newer manifest, but the filename was associated with a different signature in the older manifest, then we conclude that the file was edited or otherwise modified. e.  if a signature only occurs in the older manifest, and the filename only exists in that manifest, then we conclude that the file was deleted. note that this logic gets more elaborate if there are multiple file instances within an object version that share the same checksum value. 5.1.6 version metadata at the sdr a versionmetadata.xml file for each version is stored as part of the digital object metadata (and not in the manifests folder). it records simple descriptive information about a digital object’s collection of versions and is updated for each new version. the example presented here shows the simple schema adopted by stanford. it is not an essential part of the moab design, but is used at sdr as a place to record an optional tag value and a reason for the creation of the version. <versionmetadata objectid="druid:ab123cd4567"> <version versionid="1" tag="1.0"> <description>initial version</description> </version> <version versionid="2" tag="2.0"> <description>replacing main pdf</description> <note>replaced at student request to fix citation errors</note> </version> <version versionid="3" tag="2.0.1"> <description>fixed title typo</description> </version> </versionmetadata> 5.2 moab storage objects the moab-versioning gem also provides classes that can be used to represent the levels of an online folder hierarchy that can be used for storage and retrieval of digital object files and manifests. the class types that are provided include: • storagerepository, which represents a digital object repository storage node • storageobject, which represents a digital object’s repository storage location. • storageobjectversion, which represents a version subdirectory within an object’s home directory. • storageservices, which supports application layer access to the repository’s objects, data, and metadata. • a bagger utility for creating bagit packages for ingest or dissemination. see http://rubydoc.info/github/sul-dlss/moab-versioning/master/moab/storagerepository for more details. 6 the sdr preservation workflow we’ll end this tour of the moab versioning design by describing the steps of the sdr accessioning and ingest workflow to illustrate an example of how a real-world implementation of this design is being utilized at stanford for the long-term storage of digital objects accessionwf workflow steps (dor) start-accession initiate the workflow content-metadata verify presence of, or create, contentmetadata datastream (this becomes the content filegroup in the version inventory) descriptive-metadata verify presence of, or create, descmetadata datastream rights-metadata verify presence of, or create, rightsmetadata datastream technical-metadata verify presence of, or create, technicalmetadata datastream (uses a version comparison to determine what files need characterization) provenance-metadata verify presence of, or create, provenancemetadata datastream shelve transfer content to be held online to the digital stacks (uses a version comparison to determine what files are changed) publish publish shared, public metadata to the digital library sdr-ingest-transfer bag content and metadata file for export to preservation core ingest (heavy use of moab versioning toolkit to record the version inventory, figure out which files have new signatures, then package only those files for export) sdr-ingest-received callback from preservation core ingest indicating ingest completed end-accession complete the accession workflow sdringestwf workflow steps (preservation core) start-ingest initiate the workflow register-sdr ensure that there is a fedora entry for the object transfer-object copy the object package from the export location validate-bag verify package structure and checksums populate-metadata update fedora datastreams as needed verify-agreement ensure that the object’s governing policy object has already been ingested complete-deposit create the online storage folders for the object and/or version. verify file checksums and the ability to reconstruct the full version. the package that is exported from the dor system is a normal bagit bag containing content, metadata, and manifest files. the bag is transferred to the preservation core system using rsync, then validated a first time, before being used as the source for creating either a new digital object location in the storage system, or a new version location within a previously created digital object store. the steps are: • create the new version’s storage folder • copy the version’s data files into storage • update the object’s signature catalog • generate a version differences report • generate an inventory of all the manifest files 7 conclusion the moab design for digital object versioning was born out of an urgent need to figure out an efficient, practical, and robust mechanism for making changes to digital objects that had already been ingested. the need for a versioning capability was holding us back from fixing some problems that had already occurred, or opening up the repository door to new sources of documents that we anticipated might get modified in a relatively short time. our archival philosophy restrains us from simply overwriting an old version of an object with a new one. we wish to retain all old copies of files in case a replacement proves worse than the original, or so that we can look back an object’s history for whatever reason might arise. the nature of a preservation repository turns out to be different enough from that of a software change management system or file backup system that some technologies which initially appeared promising (such as git and boar) turned out to have design aspects or behaviors that ruled them out as solutions that we could adopt off of the shelf. the cdl micro-services approach had the best promise and well-written specifications, but we were disappointed when we realized that the redd reverse-delta mechanism would likely prove too resource consumptive for us. inspiration was derived, however, from all of those designs and the result of assembling pieces from each approach into a new framework has yielded rewarding results. recapping the criteria used in section 3.6 to evaluate those other technologies, here is how the moab design measures up: • file fixity: good moab does not alter any content of object files so that the fixity remains consistent throughout the system. manifests record file sizes and checksums for use in verification. • no file duplicates: good by using file signatures as the key to file identity, moab guarantees that only one copy of any given file manifestation is stored at any given storage or replication location. • replication efficiency: good for the same reason, the transmission, processing, and storage of a new version’s files only needs to involve the files that were newly introduced in that version. • version reconstruction: good any version of a digital object can be reconstructed and disseminated with equal ease to any other version. sdr welcomes feedback and collaboration so that we can identify any deficiencies in our design or move the development forward with new useful enhancements. works cited black, h. (2006). compare-by-hash: a reasoned analysis. proceedings of the annual conference on usenix ’06 annual technical conference , 85–90. http://www.cs.colorado.edu/~jrblack/papers/cbh.html wright, r., miller, a., & addis, m. (2009). the significance of storage in the “cost of risk” of digital preservation. the international journal of digital curation , 4 (3), 104-122. also presented at ipres,2008: http://www.ijdc.net/index.php/ijdc/article/viewfile/138/160 notes [1] http://fedora-commons.org/ [2] http://rsync.samba.org/ [3] https://confluence.ucop.edu/display/curation/storage [4] https://confluence.ucop.edu/display/curation/d-flat [5] https://confluence.ucop.edu/display/curation/redd [6] http://en.wikipedia.org/wiki/content-addressable_storage [7] http://valerieaurora.org/monkey.html http://valerieaurora.org/review/hash2.pdf http://www.nmt.edu/~val/review/hash2.pdf [8] http://en.wikipedia.org/wiki/cryptographic_hash_function [9] http://csrc.nist.gov/groups/st/toolkit/secure_hashing.html [10] i have not found a single authoritative reference that covers the issue of hash algorithm security and collision probabilities, but there are many well written commentaries to be found on the stackoverflow web site: http://stackoverflow.com/questions/800685/which-cryptographic-hash-function-should-i-choose http://stackoverflow.com/questions/2479348/is-it-possible-to-get-identical-sha1-hash http://security.stackexchange.com/questions/19705/is-sha1-better-than-md5-only-because-it-generates-a-hash-of-160-bits http://stackoverflow.com/questions/6776050/how-long-to-brute-force-a-salted-sha-512-hash-salt-provided [11] http://www.universetoday.com/36302/atoms-in-the-universe/ [12] http://git-scm.com/ [13] http://schacon.github.io/gitbook/1_the_git_object_model.html illustrations from the “git community book” used by permission of scott chacon [14] http://www.slideshare.net/pbhogan/power-your-workflow-with-git illustrations from “power your workflow with git” used by permission of patrick hogan [15] http://code.google.com/p/boar/ [16] the “moab” name is not really an acronym, but is the name of the town in which this author is living. i have come up with some acronym expansions, however, for the fun of it: multi-version object archive bags; make objects act better; my own and best [17] http://tools.ietf.org/html/draft-kunze-bagit-09 [18] http://en.wikipedia.org/wiki/functional_requirements_for_bibliographic_records about the author richard anderson (rnanders at stanford.edu) works in the digital library systems and services department of the stanford university libraries. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – python, google sheets, and the thesaurus for graphic materials for efficient metadata project workflows mission editorial committee process and structure code4lib issue 35, 2017-01-30 python, google sheets, and the thesaurus for graphic materials for efficient metadata project workflows in 2017, the university of virginia (u.va.) will launch a two year initiative to celebrate the bicentennial anniversary of the university’s founding in 1819. the u.va. library is participating in this event by digitizing some 20,000 photographs and negatives that document student life on the u.va. grounds in the 1960s and 1970s. metadata librarians and archivists are well-versed in the challenges associated with generating digital content and accompanying description within the context of limited resources. this paper describes how technology and new approaches to metadata design have enabled the university of virginia’s metadata analysis and design department to rapidly and successfully generate accurate description for these digital objects. python’s pandas module improves efficiency by cleaning and repurposing data recorded at digitization, while the lxml module builds mods xml programmatically from csv tables. a simplified technique for subject heading selection and assignment in google sheets provides a collaborative environment for streamlined metadata creation and data quality control. by jeremy bartczak and ivey glendon project overview in 2017, the university of virginia (u.va.) will launch a two year initiative to celebrate the bicentennial anniversary of the university’s founding in 1819. the u.va. library is participating in this event by digitizing some 20,000 photographs and negatives that document student life on the u.va. grounds in the 1960s and 1970s. the collection shows sporting events, teaching and research, and the residential aspects of life at the university. we anticipate that this collection will be very popular with researchers, students, and alumni. the idea for collection digitization began in summer 2015 through a partnership between special collections curators and the library’s digital production group (dpg) as plans for university-wide recognition for the bicentennial anniversary came into focus. the project evaluation process involved the following stakeholder groups: table 1. stakeholder groups team role special collections curators photograph selection digital production group (dpg) digitization metadata analysis and design (mad) metadata schema/content standard selection and evaluation and workflow management digital content management and dissemination (dcmd) collection publication locally and to aggregators through the fall of 2015 and winter of 2016, the curatorial staff worked with the digitization group to select photographs, test the conversion, and establish a digitization schedule to meet the desired volume within the given time period. in spring 2016, the dpg turned to the mad team to evaluate metadata design and creation possibilities for this collection within the library’s standard mods schema for digital objects. metadata design: approaches to description various criteria impacted the mad group’s decision making processes for metadata design, including limited staffing resources, metadata creation efforts on other digital conversion projects, and feedback from students, researchers, and colleagues. given these circumstances, the mad group aligned around three guiding principles for metadata design for this collection: a strong preference for manipulating existing metadata rather than creating original description selection of a vocabulary for subject access that is easily understood and applied by student assistants but also meaningful to a wide variety of users development of a metadata creation framework easily understood by undergraduate student assistants completing the metadata creation work past experience with a variety of digital conversion efforts has led mad staff to favor metadata manipulation over fresh metadata creation whenever possible. in the past, confusion around content standards and supplying titles for unpublished material has incurred significant metadata rework costs and necessitated a burdensome quality review process. the photographs identified for the bicentennial celebration come with folder-level metadata that the dpg student assistants transcribe as part of the digitization process; the existence of this metadata, however scant, provides an anchoring point to create a metadata manipulation process rather than relying on new metadata creation. consequently, the mad team designed an approach to titling wherein digitized images are examined and given more specific titles based on actual subject matter and the data on the folder. for example, a folder labeled “candid shots geology department” would inform an item-level photograph caption: “two students observing exhibit in geology department.” the selection of a controlled vocabulary for subject access for this collection was also a significant decision point for the mad team, and one that represents an evolution in approaches to subject access for locally unique collections. anecdotal evidence from students, researchers, and library colleagues has led the mad team to evaluate the merit of compound library of congress subject heading (lcsh) strings against simple headings from other sources, such as the faceted application of subject terminology (fast) authority file. students, researchers, and colleagues have found the complex lcsh headings “too specific to be useful,” “difficult to navigate,” and “impossible to use for finding like materials.” reflecting on this feedback, the mad team aligned around a simplified subject application approach for several reasons, some related to metadata philosophy and some for workflow expediency: complex, compound lcsh headings describe single items well but to the detriment of an aggregation of objects simplified headings map to permanent uris where complex strings sometimes do not simplified headings will be more accessible to students as users but also much more accessible to the students who will be doing the metadata creation work in a production environment agreed in this approach, the mad team further considered the vocabulary possibilities to meet the desired specifications for creation by students and engagement by researchers. the library of congress thesaurus for graphic materials (lctgm) was identified as a desirable vocabulary for this collection, as it provides a list of targeted headings ideal for describing visual materials, a persistent uri for every term, and accessibility to students such that they can easily view an image and apply an appropriate heading. since the scope of the collection is limited to a very predictive range of topics, an additional step was taken to simplify the workflow: a metadata librarian compiled and made available to students a list of around 40 pre-selected lctgm terms that potentially apply to photograph subject matter across the collection for the project for selection during processing. additionally, the mad team limited assignment of subject terms to not more than three per photograph. deliberation on title and topical subject access for these materials represented the bulk of the metadata design effort in advance of moving into a production workflow for digitization and metadata manipulation and creation. other metadata, including dates (captured at digitization), the corporate heading “university of virginia,” and the geographic heading “charlottesville, va.,” are available for all objects. the creation of a five-part metadata design framework – title, date, topical subject, corporate heading, and geographic access point – represents a radically lighter metadata approach for this collection compared to past projects. by eschewing elaborate, narrative description, we have developed a process that is easily understood by undergraduate students doing the metadata creation work, and that reflects acceptance of a “good enough” standard to support rapid cataloging of objects. metadata creation: workflows, pandas, and google sheets implementation of these strategies begins upstream with the library’s digitization workflow which is managed in an in-house system where images and metadata are staged for repository ingest. two database fields (title and description) are available for dpg student workers to record information gathered from physical items at the point of digitization. for this particular project, dpg student workers proceed through the collection box-by-box in archival context. as the items are digitized, students manually record baseline descriptive metadata in the system, which is transcribed either from the physical item itself or from the original photographer’s folder that it is housed in. digitized master files from each box are grouped as “units” within the system, complete with this first-pass at descriptive metadata plus additional automatically generated administrative metadata. some examples of data created or recorded at this stage includes fields like identifier, tiff filename, repository pid, and the aforementioned photographer supplied job title. for this particular project, both date and folder data are collapsed into the description field and pipe-delimited. the workflow system provides an interface to navigate this information within units, at the item-level, with both digital object and metadata displayed. notably, the metadata initially associated with these objects is designed to accommodate digitization and ingest workflows. descriptive mods xml is uploaded at a later stage for final online display. certain fields, such as photographer supplied title data, folder numbers, dates, and administrative identifiers, are fully incorporated or re-purposed into mods fields downstream. based on this workflow, the mad team had to implement an efficient strategy for creating mods records. one option considered was to design a mods template and have students manually combine data from the digitization workflow with assigned subject headings and reformatted titles (e.g., “candid shots geology department”–as noted earlier in this paper, the students replace the photographer supplied title with a more descriptive caption for the object) in an xml editor. a different option involved downloading digitization metadata to a csv and munging that dataset. cataloging work could take place within the csv, with a student worker simply adding titles and assigning subject headings to each record in the table. mods records could then be built by iterating over the table using lxml. the latter approach was selected because it avoided the additional steps of training students in xml and getting them oriented with the editor. it also was faster because administrative data was automatically added to each record as opposed to manual re-entry. with the decision to work with csv data in place, the next step was to clean the data up and get it into a presentable display for further enhancement. metadata entry would happen in the mad unit and be completed by a second group of student workers. a script was designed to clean up the data. this included standardizing date formatting and reducing extraneous columns to prevent a wide, scrolling display for a more straightforward data entry interface. creating a script, similar to automating the mods record build, would save significant time by preventing the need to document and implement steps for manual clean up. python’s pandas module proved to be an ideal tool for cleaning up the datasets per these requirements. pandas is a python package developed for statistical analysis and data manipulation. the csv is read in as a dataframe—a tabular, two dimensional spreadsheet-like object with columns that can accommodate different datatypes (mckinney 2013). columns and rows can be easily selected using various pandas methods (10 minutes to pandas 2016). there is also built in functionality for working with textual data and empty cells. the pandas dataframe was inspired by the r programming language’s dataframe, thus providing similar baseline functionality with the r data analysis programming environment (comparison with r/r libraries 2016). an obvious alternative to this approach would have been to use openrefine, the popular data cleaning tool. the goal was to be able to save a series of scripted commands that could be run for a large set of similar spreadsheets in the workflow. this could easily be accomplished in pandas or openrefine (or r for that matter). openrefine also provides a jython language interface for those who prefer python. this paper doesn’t advocate for one approach over the other. pandas was a logical decision in this context for multiple reasons: the project provided an opportunity to apply a new skill-set gained by a team member who had recently taken related coursework with u.va.’s research data services and sciences program. this was an opportunity to develop that skill-set and diversify expertise within the department. the learning curve was reasonable. given prior experience, pandas was the best bet for quickly getting a script up and running to perform what was required. the department prides itself on curiosity and exploration. while pandas is immensely popular in the scientific data analysis community, there hasn’t been as much discussion in the library metadata literature. here was an opportunity to test out the tool on a large metadata project and share the results. even though openrefine offers a templated export option, the team was committed to using python’s lxml module for building the mods records in python. it made sense to keep related code streamlined as data was pushed and pulled through the workflow. the following code snippets demonstrate the data cleanup workflow and some very basic pandas functionality. the complete script is available at: https://github.com/jtbmadva/bicentennial-project-scripts import the modules used in the script and read the csv in as a dataframe. import pandas as pd import re from dateutil.parser import parse df = pd.read_csv('sample_data.csv') columns that aren’t important for the eventual mods output are removed using the drop function. note that the axis has to be specified. the dataframe is indexed as a two-dimensional array, with axis 1 corresponding to data along the columns. here is one of several columns dropped from the downloaded csv. to reiterate, this is data that helps with ingest workflows but is not required for final mods output, so it is removed. df = df.drop('filesize', axis=1) a column with multiple values is split using pandas string functionality. recall that at the point of digitization, students entered date and job number data into a single, generic “description” column with that information pipe delimited. the values from the description column are split at the delimiter into a list, then indexed. values are then assigned to two new columns, appropriately labelled date and job in square brackets. this example shows how easy it is to process each value in the column through vectorized string methods. df['date'] = df.description.str.split('|').str[0] df['job'] = df.description.str.split('|').str[1] an existing title field is renamed ‘info from job’ to more accurately describe data there. dpg students used this column to transcribe notes directly from photographer job folders. this data informs actual title creation downstream in the workflow. the former column name is replaced using a dictionary in the columns parameter. df = df.rename(columns={'title': 'info from job'}) columns are then added for the fields required for the next step in the description: a new title field, and three subject fields. subject assignment is intentionally limited to three terms, hence three subject columns. to insert the columns to a specific location in the dataframe, the insert function is used. column location in the frame is specified by index and assigned a column header. the value parameter includes empty quotes to ensure that the cells are empty. df.insert(2, 'title', value='') df.insert(4, 'subject 1', value='') df.insert(5, 'subject 2', value='') df.insert(6, 'subject 3', value='') a final step involves cleaning up inconsistent date data previously entered by students with a group of conditional statements. here data from the date column is thrown into a list and iterated over by conditional statements in order to catch and remediate some of the various ways this information was recorded by students. this block leverages both the regular expressions and dateutil python modules which are imported at the top of the script. fortunately the various conditions for entering date information for this project have been fairly consistent and alleviated as digitization work progressed. some issues do slip through though (seasons of the year or multiple dates entered) and cause the code to fail, requiring a quick scan of the csv and manual correction. dates = df['date'].tolist() for i in dates: if len(i) == 4: i = "'" + i datecol.append(i) elif re.match('jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec', i): i = "'" + parse(i).strftime('%y-%m') datecol.append(i) elif list(i).count('-') == 1: i = "'" + parse(i).strftime('%y-%m') datecol.append(i) else: i = "'" + parse(i).strftime('%y-%m-%d') datecol.append(i) after the above steps are implemented, the dataframe is output to a newly titled “clean” csv. df.to_csv('clean_sample_data.csv', encoding='utf-8') the new csv is now prepped and ready for data entry by student workers. the script described above outputs a much more streamlined view of the data originally downloaded, plus columns have been renamed or added to be as clear as possible about the data students are working with. to further facilitate our goals, the csv is uploaded as a google sheet with settings adjusted to allow editing privileges for those who have access to the url. the list of lctgm terms mentioned earlier is manually added as a drop-down menu in the subject columns of the sheet using the google sheets data validation feature. in addition to the data validation term, the unique id.loc.gov uri suffix is included along with a pipe delimiter for any potential future linked open data initiatives. students consult basic guidelines written in-house and modelled on the excellent indexing tutorials made available online by the library of congress prints and photographs division (cataloging & digitizing toolbox 2010). the actual workflow takes place at the computer with a given google sheet open and the corresponding digital images accessed via the library’s digitization workflow system. description happens by unit (recall that digital object units are organized to reflect the arrangement of the physical collection in boxes) with metadata assigned in the sheet based on matching tif filenames. students peer review each data set for basic errors by checking for typos and suggesting improved titles or alternative subject terms. this process helped significantly to standardize approaches within the project as they looked at each other’s work after it was completed and proceeded through the collection. a metadata librarian performed a final quality control check before outputting to mods. building the mods records with lxml once the csv data is entered and reviewed, a python script using the lxml module iterates over the data and builds mods xml records. these final records are then validated and uploaded for object display in u.va.’s digital library. lxml is a python module for working with xml data. a blog post highlighting the power of this module for remediating archival metadata features some excellent examples and use cases (boyle 2015). two detailed tutorials are also available that provide all of the necessary information to get up and running: http://infohost.nmt.edu/~shipman/soft/pylxml/web/index.html http://lxml.de/tutorial.html this section will step through some of the methods used for this project to output metadata from a pandas dataframe to a valid mods record (the entire script is available in the github link shown above). the xml processes described here use the lxml elementtree api, datetime, and pandas modules. datetime and elementtree are imported as follows: from lxml import etree import datetime as dt after reading the csv in as a dataframe, the first step is to iterate over the frame using the pandas iterrows() function. for index, series in df[:].iterrows(): the function iterates over the frame in index (row) and series (columns) pairs, so variables are created for both “index” and “series”. now data can be easily referenced by column name to output cell values from each line of the frame. at the beginning of this process the root element for the mods record must be created. three things happen in this process: attributes are included for both the mods schemalocation and version namespaces are declared for both xsi and mods namespaces are mapped to namespace prefix strings for the final xml output to display correctly the tricky part is that lxml doesn’t use namespace prefixes. instead it uses clark notation which places the entire namespace uri inside curly braces in front of the element name (clark 1999). so, for example: <mods:mods xmlns:http://www.loc.gov/mods/v3/> maps to: <{http://www.loc.gov/mods/v3}mods/> variables are set up by convention for both namespace uri’s, the mods schema location, and the namespace mapping. these variables can be referenced in the code to keep things cleaner and more readable. mods_ns='http://www.loc.gov/mods/v3' xsi_ns='http://www.w3.org/2001/xmlschema-instance' mods_schema_location='http://www.loc.gov/standards/mods/v3/mods-3-6.xsd' ns_map = {none: mods_ns, 'xsi': xsi_ns} the name of the mods root element is set up using clark notation. note the string formatting where curly braces are concatenated with the mods_ns variable alongside the element name. '{'+ mods_ns +'}mods' an attribute is created in a dictionary for xsi:schemalocation, again using clark notation, the same string formatting as above, plus the mods schema value assigned to mods_schema_location attrib={'{'+ xsi_ns +'}schemalocation' : mods_schema_location} now the namespaces prefix strings must be mapped to their uris via a dictionary and using the tree.element() nsmap parameter and the ns_map variable. the mods namespace is mapped to none, since it’s already been declared as the root element’s default namespace. nsmap=ns_map the etree.element class allows for an extra attribute to be added. here a version attribute is added to document the mods version number. version = '3.6' all of the sub-elements moving forward will now inherit the mods namespace declared in the root element. the script can be designed to either read directly from the dataframe or supply templated information as the mods records are built. building out the tree structure is simple: create a variable, use the etree.subelement() builder, specify the parent element in the parent parameter, and add the element’s name and any attributes. add text for elements using the .text method. here’s how data for the compiler of the collection is added under mods:name: name_a = etree.subelement(mods, 'name', type='personal') namepart_a = etree.subelement(name_a, 'namepart') namepart_a.text = 'skinner, david m., 1921-2009' similar operations are performed to supply templated data for mods:typeofresource, mods: physicaldescription, mods:note, mods:subject (geographic terms), mods:physicallocation and mods:recordinfo. incorporating data from the dataframe is just as easy. again use the subelement() builder and .text method as shown above, then reference the dataframe column name with the series iteration variable. identifier_d = etree.subelement(mods, 'identifier', type='local', displaylabel='file name') identifier_d.text = series['filename'] pandas provides many different options for slicing and selecting data (indexing and selecting data 2016). one basic approach for this is shown below. for dealing with subjects in the data set, a subset of three columns is sliced using the .loc method. the three values are then iterated through for insertion into the record. cell values are selected using the series variable used throughout the script. for i in df.loc[:, 'subject 1':'subject 3']: topical_term = series[i] an if clause is set up to do nothing if the cell is blank. otherwise subjects and identifiers are split at the pipe delimiter. identifiers are concatenated with the appropriate uri prefix to create the valueuri value. note that a special scenario (not shown here) had to be handled separately since one term from the pre-selected list of subject headings came from the fast vocabulary instead of the lctgm. if topical_term == 'nan': topical_term = '' else: authority_uri = 'http://id.loc.gov/vocabulary/graphicmaterials/' valueuriprefix ='http://id.loc.gov/vocabulary/graphicmaterials/' subject = etree.subelement(mods, 'subject', authority = 'lctgm', authorityuri=authority_uri) subject_data = topical_term.split('|') subject_data = [i.strip(' ') for i in subject_data] topic = etree.subelement(subject, 'topic', valueuri=authority_uri + subject_data[1]) topic.text = subject_data[0] the script imports python’s datetime module to include today’s date in the mods:recordcreation subelement. the date.today() function returns today’s date, then it’s formatted as a string according to the extended date time format. today = dt.datetime.today().strftime('%y-%m-%d') recordcreationdate.text = today once the entire tree is built, the script creates a file object and writes the record out using the tostring() function. with open(filename +'.xml', 'wb') as f: f.write(etree.tostring(mods, xml_declaration=true, encoding='utf-8', pretty_print=true)) assessment and conclusion the project surpassed original expectations for fast and efficient cataloging of the photographs in the collection. at the project’s outset, 4000 objects were described by two non full-time metadata assistants in just over a month. it was difficult to keep up with quality control, and the students, who had been hired for an entire semester, had to be given additional tasks since they caught up with the digitization workflow. at first pass, the lctgm subset appears to facilitate this rapid description process while providing facets that are useful for winnowing down the collection into manageable groupings. the consistent themes shown throughout the collection (e.g. sports, classrooms, students, etc.) certainly lend themselves to this approach. the mad team plans to solicit testing and perspective from the library’s ux team in this area to determine what users think of the indexing strategy in comparison to more precise compound headings. future digital conversion indexing projects can be calibrated based on this feedback. in the thick of the actual work, it is worth noting how willing students are to access the lctgm themselves and navigate its straightforward display to suggest additional categories to be incorporated into the sub-set. students feel comfortable with the terminology and are engaged in the development of the mini-vocabulary. google sheets was an excellent option for sharing and enhancing metadata sets. this was especially true in a university setting where collaborators were dispersed across physical space. editable links could be sent by email and sheets were easily viewed and edited by users simultaneously. this was very helpful for the peer-review and qc processes. the workflow wasn’t perfect though. one significant pain point were the challenges recognized early on of pushing and pulling csv data to the cloud and back. google sheets and excel try very hard to do users a favor by formatting and remediating values. leading zeros in folder names or any kind of date data are prime examples. readers might have noticed that single quotes are added to the front of date strings in python to force these tools to leave this data as it is. leading zeros in folder names proved even more challenging and required the addition of a textual string and further indexing when being read to xml. future efforts will focus on the google sheets api to avoid such issues and eliminate the step of downloading data before it’s transformed. pandas can be used as an effective tool for baseline tabular library metadata clean-up. many resources are available for support in the stackoverflow community, the pandas module documentation, online tutorials, and published books. the first script covered here automates the consistent manipulation of like datasets including adding, removing, renaming and splitting columns, and remediating date information. this approach saved significant time compared to documenting and executing a series of steps for cleaning up this data manually. pandas also paired up well with lxml in providing a straightforward way of reading data from a dataframe. in comparison to pandas, openrefine stands out in terms of its cluster functionality for correcting typos and its straightforward rdf extension for reconciling terms with published lod vocabularies (although python has robust tools for working with both rdf and web data in general). a nice template is available for exporting data to mods from openrefine (allain 2014). perhaps python users will find the code shared in this article of similar value. lxml is a tool that metadata librarians and archivists should have on their radar screen. the power is in the elementtree api’s almost too-easy syntax for building and transforming xml data. it would be interesting to read feedback about using a tool like this as a full blown replacement for xslt for tasks such as legacy metadata remediation projects or other scenarios for transforming xml. if nothing else, the pros and cons of xslt vs. a pythonic (or any other high-level language) approach for standard library metadata-related xml tasks would be an interesting discussion. looking forward within u.va.’s mad team, the plan is to build on experiences from this project and push on. there is potential, for example, for using pandas to merge metadata sets for analysis and visualization along the lines demonstrated in a recent code4lib article (harper 2016). for streamlined description, the use of fast headings shows great potential similar to the workflow covered here as has been demonstrated in comments during a recent alcts webinar (pastva and odell 2016). while keeping an eye on such trends team members will focus on cross-departmental collaboration and look for creative opportunities to develop and apply technical skills where appropriate. metadata project design and management will emphasize sustainable description strategies and the necessary tools and skills to meet or even outpace future library digitization workflows. references 10 minutes to pandas. pandas 0.19.1 documentation [internet]. [updated 2016 nov 3]. [cited 2016 nov 29]. available from: http://pandas.pydata.org/pandas-docs/stable/10min.html allain, s. 2014. converting spreadsheets into modsxml using open refine. utsc library digital scholarship unit admin’s blog [internet]. [cited 2016 nov 29]. available from: https://www.utsc.utoronto.ca/digitalscholarship/content/blogs/converting-spreadsheets-modsxml-using-open-refine boyle, w. 2015. tools for the programming archivist: ead manipulation with python and lxml [internet]. [cited 2016 dec 14]. available from: http://archival-integration.blogspot.com/2015/10/tools-for-programming-archivist-ead.html cataloging & digitizing toolbox [internet]. [updated 2010 oct 22]. [cited 2016 nov 29]. available from: http://www.loc.gov/rr/print/cataloging.html#tips clark, j. xml namespaces [internet]. [updated 1999 feb 4]. [cited 2016 nov 29]. available from: http://www.jclark.com/xml/xmlns.htm comparison with r/r libraries. pandas 0.19.1 documentation [internet]. [updated 2016 nov 3]. [cited 2016 nov 29]. available from: http://pandas.pydata.org/pandas-docs/stable/comparison_with_r.html harper, ca. 2016. metadata analytics,visualization, and optimization: experiments in statistical analysis of the digital public library of america (dpla). code4lib [internet]. [cited 2016 nov 29]; issue 33. available from: http://journal.code4lib.org/articles/11752 indexing and selecting data. pandas 0.19.1 documentation [internet]. [updated 2016 nov 3]. [cited 2016 nov 29]. available from: http://pandas.pydata.org/pandas-docs/stable/indexing.html mckinney, w. python for data analysis [internet]. beijing: o’reilly, 2013 [cited 2016 nov 29]. available from http://proquest.safaribooksonline.com/book/programming/python/9781449323592/5dot-getting-started-with-pandas/id2828378 pastva, j, o’dell, aj. 2016. using fast for faster workflows and discovery. association for library collections & technical services webinar. [cited 2016 nov 29]. available from: http://www.ala.org/alcts/confevents/upcoming/webinar/092816 about the authors jeremy bartczak (jtb4t@virginia.edu) is a metadata librarian at the university of virginia library. ivey glendon (img7u@virginia.edu) is the manager for the metadata analysis and design team at the university of virginia library. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – automated playlist continuation with apache predictionio mission editorial committee process and structure code4lib issue 42, 2018-11-08 automated playlist continuation with apache predictionio the minrva project team, a software development research group based at the university of illinois library, developed a data-focused recommender system to participate in the creative track of the 2018 acm recsys challenge, which focused on music recommendation. we describe here the large-scale data processing the minrva team researched and developed for foundational reconciliation of the million playlist dataset using external authority data on the web (e.g. viaf, wikidata). the secondary focus of the research was evaluating and adapting the processing tools that support data reconciliation. this paper reports on the playlist enrichment process, indexing, and subsequent recommendation model developed for the music recommendation challenge. by jim hahn 1. introduction the purpose of this paper is to report the results of the minrva team’s (https://minrvaproject.org) development of a data-focused recommender system for the 2018 acm recsys challenge [1]. the guiding contention of our team’s work was that while algorithm development, ensemble construction [2], and selection [3] are vital elements of playlist completion, we viewed the primary problem of playlist completion as one of data reconciliation. in this paper, we describe the data processing that the minrva team researched and developed for foundational reconciliation of the million playlist dataset (mpd) [4] using external data sources on the web. the secondary focus of the research was the processing tools that support data reconciliation. machine learning is of interest to many data-intensive fields and is a growing topic in multiple applied computing domains in addition to powering innovations in music recommendation systems (mrs) [5]. as an example, digital library environments are no exception to the machine learning trend for recommender systems [6]. digital libraries traditionally have focused on data preservation and access. digital preservation seeks first to manage and describe data, but more recently, the field has sought to add value to data, much like the minrva team’s approach to enriching the spotify mpd. the minrva team, a software development research group based at the university of illinois library sought to reconcile a portion of the spotify mpd with the virtual international authority file or viaf, a large dataset available openly that library professionals have curated for many years. the viaf provides an authoritative source of linked data for named entities, such as the names of artists from the spotify mpd [7]. the field of digital librarianship’s focus on data curation and e-science [8] has generated an underlying premise for data intensive research of the re-use of datasets over time and adding value to data while also supporting their preservation. overall, the acm recsys challenge offered a rich area in which to put data curation and accompanying reconciliation techniques from digital librarianship into practice, including the enrichment and transformation of the mpd’s large corpus (over 2 million unique tracks). together with machine learning’s noted popularity in many different applied computing domains, the parallel development of machine learning tools, such as common algorithms, is now available within software frameworks that are implemented easily. with this growing integration of machine learning software into big data tools, such as mllib in apache spark in general, and the alternating least squares algorithm for recommendations in particular, the minrva project team was able to reuse existing machine learning algorithms, frameworks, and data models for the playlist continuation task. specifically, the alternating least squares implicit feedback approach [9] incorporated into apache spark was used to generate minrva’s recommendation model for the acm recsys challenge 2018. before moving on to challenge metrics and tasks, we explore the anonymized playlist metadata provided by spotify within the mpd and the accompanying song metadata. 1.1 million playlist dataset (mpd) metadata elements the playlist metadata that were made available by spotify were encoded in json and used associative arrays to convey higher level details about the playlist. as shown below, the playlist incorporates song data in a hierarchical structure. track data are the second level of data with the playlist data inherited by each song in the list. metadata in the playlist element field included: the name of the playlist (free text); whether it was collaboratively developed (true/false); a playlist id (numeric), the last time it was modified (numeric), the number of tracks in the playlist (numeric), the number of albums in the playlist (numeric), and the number of followers (numeric). "playlists": [ { "name": "example", "collaborative": "", "pid": , "modified_at": , "num_tracks": , "num_albums": , "num_followers": , "tracks": [ { "pos": 0, "artist_name": "", "track_uri": "spotify:track:" , "artist_uri": "spotify:artist:" , "track_name": "", "album_uri": "spotify:album:", "duration_ms": , "album_name": "" } ... ] } ... ] metadata elements which spotify provided as track data included: the position of the track in the playlist; the name of the artist (free text); a track uri (spotify controlled identifier); a track name (free text); an album uri (spotify controlled identifier); duration of the track (milliseconds); the name of the album (free text). it should be noted that these are the elements that were provided to our team after registering for the challenge, we did not see nor did the team consult documentation regarding spotify data storage, therefore, it could be an approximation or abstraction as to how spotify stores metadata. 1.2 baseline properties of the million playlist dataset baseline statistics of the mpd were provided in [1]. the values underscored several properties of the mpd reporting the average playlist length (or tracks) as 66.35; 17,381 unique normalized playlist titles; 92,944 unique playlist titles; 295,860 unique artists; 734,684 unique albums; 2,262,292 unique tracks; 66,346,428 tracks; which made up the 1,000,000 playlists [1]. 2. background music recommendation in the recsys challenge focuses on the task of playlist continuation when given a set of seed tracks with a playlist name [1]. for the spotify mpd recsys challenge, the task included the call to “…develop a system for the task of automatic playlist continuation. given a set of playlist features, participants’ systems shall generate a list of recommended tracks that can be added to that playlist, thereby ‘continuing’ the playlist” [4]. metrics for the challenge included traditional information retrieval metrics, such as a precision metric, together with two additional metrics that were unique to the spotify recommendation challenge. these are delineated in table 1. metric definition r-precision r-precision is the number of relevant tracks retrieved divided by the number of relevant tracks known (i.e., the number of tracks withheld). normalized discounted cumulative gain (ndcg) discounted cumulative gain (dcg) measures the ranking quality of the tracks recommended, and increases when relevant tracks are placed higher in the list. normalized dcg (ndcg) is determined by calculating the dcg and dividing it by the ideal dcg in which the recommended tracks are ranked perfectly. recommended songs clicks recommended songs clicks is the number of refreshes needed before a relevant track is encountered. table 1: metrics of the recsys challenge 2018 [10]. the challenge area is described more precisely as a challenge to complete ten different types of playlists using the mpd as training data. in the ten focus areas, the recommender system must generate 500 recommended tracks in decreasing order of confidence when provided a variation in either a) seed tracks and b) playlist name or c) just a playlist name [1]. one of the most challenging aspects of this challenge for our team was to improve our baseline metrics for playlists when only a playlist name is provided with no seed tracks. the challenge allowed for both non-academia (industry researchers) and academia researchers to participate. only the research teams from academia (verified by our @illinois.edu email addresses, presumably) such as the minrva team, were able to access the spotify mpd in its entirety [10]. non-academia or industry researchers were able to join a majority academic research team and have access to a subset of the spotify mpd. these steps were taken to better ensure user privacy [10]. 3. methods as our overall goal was to enrich the mpd, we sought to use a baseline entry in the main track as a comparison to the reconciled mpd entry in the creative track. the creative track is the portion of the recsys challenge that allows integration of external datasets to improve recommendations. the schema used in our model are described in section 3.4, model training, but we review first several machine learning methods of our recommendation engine (section 3.1) together with data transformation and data enrichment processing and tools (sections 3.2 and 3.3 respectively). 3.1 recommendation engine: apache predictionio the recommendation engine used the similar product template [11] in apache predictionio. the similar product template was designed to perform machine learning with the alternating least squares (als) algorithm, as implemented in spark mllib. matrix factorization techniques, the basis for the minrva team’s algorithm, were proven successful as a recommender algorithm for the netflix prize [12]. the data load scripts used to import items, users, and views are available from the minrva team’s public github repository [13]. the predictionio system provides several bootstrapping advantages for participating in the recsys 2018 challenge. the apache project webpage lists the advantages of the predictionio framework on the predictionio website as an “…open source machine learning server built on top of a state-of-the-art open source stack for developers and data scientists to create predictive engines for any machine learning task. it lets you quickly build and deploy an engine as a web service on production with customizable templates” [14]. the open source stack is a “full machine learning stack,” which includes the following interlocking set of tools as a framework for recommendation processing: apache spark, mllib, hbase, spray, and elasticsearch [14]. 3.2 data transform tools: openrefine openrefine is a fundamental tool for data transformation and reconciliation tasks in targeted areas of enrichment [15]. openrefine generally is marketed as a data cleaning tool to use with “messy data”—however, we found that its transformation and enrichment functionality were critical to our approach. as an example of its transformation affordances, the openrefine project helped us to format the mpd corpus from a set of json files into csv files by a simple import/export command from the web app interface. the need for csv became apparent to us, as we intended to store data in a relational model using postgresql and use relational modeling to create associations in the dataset using external data. the relational model has been a profound bedrock of computing since the advent of the modern relational database era. therefore, one of the team’s first processing tasks was loading the mpd into a relational model that could be enriched and transformed further. the postgresql database proved to be suited well to this approach and provided data persistence overall in the recommendation system. the minrva team focused on the artist name with which to reconcile to related artists using viaf. the plugin to openrefine that made this reconciliation possible was the open source project conciliator, which is available on github to use within openrefine [16]. 3.3 data enrichment: conciliator according to the conciliator project webpage, the software is, “…a growing collection of openrefine reconciliation services, as well as a java framework for creating them. a reconciliation service tries to match variant text (usually names of things) to standard ids for the entity represented by that text.” viaf reconciliation was undertaken through conciliator functionality that we used on our server together with openrefine, which also ran on our development server for reconciliation. the conciliator tool is packaged in such a way that it can be run as a standalone spring project on our system. conciliator was used specifically to generate new rows of data for entities that were related to the free text of a named artist. this is a several step process in which the conciliator tool looks for a name match in the viaf dataset first, and then looks for related artists. when a named entity, such as the artist, can be matched in viaf, the best match (a high confidence score) of a related artist is associated in our dataset. one of the motivations for doing this with the free text for name artists stems from the need to create additional connections among tracks. the more connections among the data we could provide, the better our model training process. furthermore, and perhaps more critically for machine learning, it was our team’s observation that controlled identifiers were available for musical tracks, the album, and the artist. like a controlled vocabulary, controlled identifiers are a preferred element in model training, as they will be less noisy for a machine learning process, and in theory, are assigned uniformly while adhering to a standard syntax and semantics. other cultural heritage projects have suggested controlled vocabulary reconciliation to achieve greater value from existing named entities [17]. we note that controlled identifiers and vocabularies stand in stark contrast to the free text elements in the mpd. our team theorized that these data points may produce unintended noise for machine learning model development when used in combination with other free text from the project, including the playlist name. we note also the unique challenges posed by playlist names, which are valuable free text, but contain no corresponding controlled id. the data fields and schema used to store data from the mpd challenge are summarized below, in which table 2 is the main track schema with entities, while table 3 delineates the schema for the creative track entry. any entity without a trailing _uri represents a potential noisy data point for the model during training, as it is an uncontrolled vocabulary and may represent free text or other words that the model will have to interpret during training. note that in the main track, we used most of the relevant entities available, including all the non-controlled vocabulary. in contrast, in the creative track, we attempted to minimize the amount of textual data, while at the same time including the enriched, reconciled related artists vocabulary. track_uri modified_at, playlist_name, album_uri, artist_uri, track_name, album_name, artist_name table 2: main track schema [18]. track_uri playlist_name, album_uri, artist_uri, artist_name, related_artist table 3: creative track schema [18]. 3.4 model training: automating playlist continuation with als algorithms preditictionio’s similar product template implements the alternating least squares algorithm and can use the implicit training set of users who have viewed (included in a playlist) products (playlist tracks). once the datasets have been loaded into a persistent data store, such as postgresql, the engine is trained with a simple train command. spark uses as many cpu cores as the cluster on which it is running, and depending on the size of the dataset, may require large amounts of ram to complete the training. therefore, model training was completed using two different types of system resources with varying cpus, ram, and storage depending on the challenge track. the maximum amount of system ram described below was used for each respective model training process. for the spotify corpus of over two million items, training time was approximately 2-4 hours. for the main track, system resources included: ubuntu 18.04 lts (gnu/linux 4.15.0-20-generic x86_64) java-8-openjdk-amd64 128gb ram, 24 cpu cores, 2.5 tb ssd psql (10.4 (ubuntu 10.4-0ubuntu0.18.04)) the creative track system was comprised of the following resources: ubuntu 18.04 lts (gnu/linux 4.15.0-20-generic x86_64) java-8-openjdk-amd64 300gb ram, 16 cpu cores, 340gb ssd psql (10.4 (ubuntu 10.4-0ubuntu0.18.04)) 3.5 playlists without seed tracks as noted earlier, we employed two different strategies to generate recommendations when a challenge set included a playlist name with no seed tracks. in the main track, we simply used the top 30 tracks from the training data paired with the playlist name to generate recommendations. for the creative track, we used a reverse indexing approach to infer likely tracks from the data available via a playlist name. tracks from the training data that had used the same or a similar playlist name as the challenge set were identified; then, those tracks were used to generate recommendations. the system used the playlist name as a category in the query pattern described above. note that challenge data were used to create approximately half of this reverse index, as the creative track allowed the use of public datasets. 4. results after training the model using the imported data described in tables 2 and 3, the system was then able to make recommendations for playlist completion. throughout the duration of the recsys challenge, submissions were evaluated against half of the challenge set, and for this portion of the challenge, the als-based recommendation system earned the following scores. metric minrva score r-precision 0.056733 ndcg 0.085486 clicks 14.0374 table 4: baseline/main track challenge results [19]. metric minrva score r-precision 0.058689 ndcg 0.090218 clicks 13.263 table 5: focus area/creative track challenge results [20]. figure 1. average aggregate scores for all teams in the main track [19]. (click to enlarge) figure 2. average aggregate scores for all teams in the creative track [20]. (click to enlarge) 5. discussion and conclusion overall, the minrva team’s entry exhibited improvement from the baseline entry in the main track compared to the enriched mpd with the viaf artist name associations in the creative track. this finding indicates that creative approaches to playlist continuation with external datasets hold promise for improved retrieval. the single algorithm employed for this task contributed to the lower ranking in the challenge overall, as the minrva team’s entry did not reach the top 10 team entries and at the end of the challenge was in the lower tier of all entries. the main track had 113 final team submissions, and the creative track had 33 team submissions [1]. revisiting our original hypothesis with respect to machine learning software and tools, it seems that considering algorithm development and data processing equally may have yielded better results overall. enriching data with external data sources can be time consuming if the data endpoints are accessed by web services. this is attributable largely to request limitations. there were additional areas of enrichment work that were identified near the end of the challenge, in that our team became aware of a wealth of resources available as openrefine add-ons for a variety of reconciliation tasks related to linked data [21]. the minrva team is interested in pursuing future data enrichment work to supplement the viaf data that was integrated into our dataset. to this end, the team will explore reconciliation processing with wikidata. wikidata model exploration and experimentation through openrefine is time consuming because it uses web-based end-points for reconciliation. however, there are graph traversing procedures that may prove quite valuable for linked data in wikidata corpus reconciliation [22]. text mining approaches [23] in playlist continuation were underdeveloped in our project, and further development of the reverse index of playlist names is an area for future research. identifying the tools and corresponding data processing steps took precedence over algorithm development for this entry, which resulted in a lower ranking overall. integration of two-stage machine learning and ensemble approaches would be valuable and desirable in the next step of our research for the apache predictionio framework for playlist continuation. a recsys 2018 paper summarized the methodologies from the top 10 scoring teams in the creative and main tracks and found approaches that utilized ensemble entries were markers of high performing teams noting — “.. most approaches ensemble the results obtained by several well-known methods, including matrix factorization models, neighborhood-based collaborative filtering models, basic information retrieval techniques, and learning to rank models. the results show that the models work best when a sufficient number of tracks per playlist is provided and they are randomly selected from the playlist (as opposed to the sequential order from the beginning of the playlist)” [24]. the winning (top scoring) entries generally had in common a strategy that employed two-stages machine learning. we note that several teams in the top 10 entries incorporated two-stage machine learning with matrix factorization of the type used in our system and that collaborative filtering, when used in ensemble approaches, was a viable part of high scoring playlist continuation methodologies. a final unique contribution to the challenge we note from our work is that in an analysis of the top 10 scoring teams, the creative track scores did not outperform the main track scores [25]. our results indicated the opposite — our final creative track scores with viaf reconciliation of related artist names provided an improved score in all metrics as compared to the baseline main track entries. we emphasize this finding to advance our research concern with reconciliation-based recommendation approaches. the method improves baseline scores and add valuable context to recommender systems. data reconciliation processes given equal importance in ensemble machine learning could support contextual richness overall in providing recommendations. therefore, a future step of our recommender system development is to implement two-stage machine learning paired with data reconciliation processes, which our results show is a vital component for information retrieval metrics in general and machine learning based automated playlist continuation specifically. acknowledgements the author sincerely appreciates the research contributions of software developers nathaniel ryckman & benjamin ryckman. further reading proceedings of the acm recommender systems challenge 2018 (recsys ‘18): https://dl.acm.org/citation.cfm?id=3267471 about the author jim hahn is an associate professor in the undergraduate library at the university of illinois (urbana-champaign, urbana, il usa). his research into technology-enhanced learning has led to many software development projects within library settings and provides unique insights into new student’s expectations and needs and helps inform the work that he does as the orientation services and environments librarian for undergraduate students at the university of illinois. he founded and manages the minrva project (https://minrvaproject.org) and currently serves as project pi for a software development grant funded through the ebsco/folio innovation challenge, the aims of which are focused on incorporating portions of the minrva mobile app wayfinding utility functions into the open source folio codebase. email: jimhahn@illinois.edu notes [1] ching-wei chen, paul lamere, markus schedl, and hamed zamani. 2018. recsys challenge 2018: automatic music playlist continuation. in twelfth acm conference on recommender systems (recsys ’18), october 2–7, 2018, vancouver, bc, canada. acm, new york, ny, usa, 2 pages. https://doi.org/10.1145/3240323.3240342 [2] peter romov and evgeny sokolov. 2015. recsys challenge 2015: ensemble learning with categorical features. in proceedings of the 2015 international acm recommender systems challenge (recsys ’15 challenge). doi: 10.1145/2813448.2813510 [3] marco tulio ribeiro, anisio lacerda, adriano veloso, and nivio ziviani. 2012. pareto-efficient hybridization for multi-objective recommender systems. in proceedings of the 6th acm conference on recommender systems (recsys’12), 19-26. doi: 10.1145/2365952.2365962 [4] million playlist dataset, official website hosted at https://recsys-challenge.spotify.com/ [5] markus schedl, hamed zamani, ching-wei chen, yashar deldjoo, and mehdi elahhi. 2018. current challenges and visions in music recommender systems. international journal of multimedia information retrieval, 7, 2, 95-116. doi: 10.1007/s13735-018-0154-2 [6] joeran beel, bela gipp, stefan langer, and corinna breitinger. 2016. research paper recommender systems: a literature survey. international journal on digital libraries, 17, 4, 1–34. doi: 10.1007/s00799-015-0156-0 [7] viaf (virtual international authority file) 2018. https://viaf.org/ [8] samuelle carlson and ben anderson. 2007. what are data? the many kinds of data and their implications for data re-use. journal of computer-mediated communication, 12, 635-651. doi: 10.1111/j.1083-6101.2007.00342.x [9] yifan hu, yehuda koren, and chris volinsky. 2008. collaborative filtering for implicit feedback datasets. in 2008 eighth ieee international conference on data mining, 263-272. doi: 10.1109/icdm.2008.22 [10] spotify recsys challenge. 2018. challenge rules. https://recsys-challenge.spotify.com/rules [11] predictionio similar product engine template (scala-based parallelized engine). https://github.com/apache/predictionio-template-similar-product [12] yehuda koren, robert bell, and chris volinsky. 2009. matrix factorization techniques for recommender systems. computer, 42, 8, 30-37. doi: 10.1109/mc.2009.263 [13] jimfhahn/recsys-challenge-2018-creative-track. 2018. https://github.com/jimfhahn/recsys-challenge-2018-creative-track/tree/master/load-data [14] predictionio. 2018. http://predictionio.apache.org/ [15] openrefine. 2018. http://openrefine.org/ [16] codeforkjeff/conciliator. 2018. https://github.com/codeforkjeff/conciliator [17] seth van hooland, ruben verborgh, max de wilde, johannes hercher, erik mannens, and rik van de walle. 2013. evaluating the success of vocabulary reconciliation for cultural heritage collections. journal of the american society for information science and technology, 64, 3, 464-479. doi: 10.1002/asi.22763 [18] see full data load details for the main track entry here: https://github.com/jimfhahn/recsys-challenge-2018-main-track/blob/master/load-data/import-items.py and the data load details for the creative track entry here: https://github.com/jimfhahn/recsys-challenge-2018-creative-track/blob/master/load-data/import-items.py [19] main track ranking. 2018. https://recsys-challenge.spotify.com/leaderboard [20] creative track ranking. 2018.https://recsys-challenge.spotify.com/leaderboard/creative [21] christina harlow. 2015. data munging tools in preparation for rdf: catmandu and lodrefine. code4lib journal, 30. https://journal.code4lib.org/articles/11013 [22] wikidata:tools/openrefine. 2018. https://www.wikidata.org/wiki/wikidata:tools/openrefine [23] andreu vall, hamid eghbal-zadeh, matthias dorfer, markus schedl, and gerhard widmer. 2017. music playlist continuation by learning from hand-curated examples and song features. in proceedings of dlrs 2017, como, italy. doi: 10.1145/3125486.3125494 [24] hamed zamani, markus schedl, paul lamere, and ching-wei chen. 2018. an analysis of approaches taken in the acm recsys challenge 2018 for automatic music playlist continuation. https://arxiv.org/pdf/1810.01520.pdf subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – the integrated library system’s apis, an open-source web 2.0 catalog, and university computing live happily ever after mission editorial committee process and structure code4lib issue 12, 2010-12-21 the integrated library system’s apis, an open-source web 2.0 catalog, and university computing live happily ever after it is widely accepted that students prefer a library catalog that offers the features that they find using google or amazon. one of these features would be dynamically delivered services. this article describes the obstacles faced trying to integrate traditional integrated library system (ils) architecture with an open source web 2.0 search interface, and outlines the path to a solution for delivering user services such as the hold and recall functions. by birong ho i. background for the western michigan university (wmu) web 2.0 catalog western michigan university (wmu) implemented vufind (hereafter referred as wmu new catalog) as the primary opac interface for its voyager system in september 2009. vufind was originally developed by villanova university in 2008. it is offered for no cost under the gpl open source license. vufind 1.0 release was made available in july, 2010. vufind includes many web 2.0 catalog features such as user tagging, comments, and reviews, as well as “more like this” and “did you mean.” despite the fact that wmu’s license for voyager does not allow the library to write directly to voyager’s underlying database tables, which makes interaction between the ils and vufind or any other stand-alone discovery layer difficult or impossible, wmu chose to implement vufind. vufind now has a community of libraries that have implemented and improved the catalog. a partial list of libraries from around the world that have implemented vufind can be found at: http://vufind.org/wiki/installation_status. vufind runs on top of apache solr. solr is an open source enterprise search platform from the apache lucene project. its major features include powerful full-text search, hit highlighting, faceted search, dynamic clustering, database integration, and rich document (e.g., word, pdf) handling. solr offers amazing performance and scalability to allow for vufind to respond to searching queries in milliseconds time. solr is different from a relational database management system such as oracle which consists of many different index tables. solr consists of a single large table that is optimized for fast lookup. wmu’s new catalog utilizes solr’s basic search functionality, and further extends solr to meet the needs of users. extended features implemented include searching location and format facets, and relaxing the standard porter stemmer included with apache solr. ii. usability issues to overcome librarians at wmu believe that students today prefer a library opac (online public access catalog) that delivers search results ranked by relevance, faceting, “more like this” and “did you mean,” etc. a traditional library opac does not meet these expectations. due to the license restrictions mentioned above, wmu could not directly write user transactions such as recall/holds and account activities directly into voyager’s underlying oracle tables. instead, it was necessary to redirect these functions to voyager’s classic webvoyage interface. (classic webvoyage is the original opac interface with voyager integrated library system (ils) which was first owned by endeavor information systems and later was acquired by ex libris.) redirecting users to the classic webvoyage interface for these functions caused confusion. iii. the solution webvoyage, the voyager opac, has been completely redesigned for voyager 7.0. the new interface made its debut in voyager libraries in june, 2008. as part of voyager 7.0, ex libris implemented webvoyage web services. wmu embedded webvoyage web services into vufind’s driver program. php function codes interact with the university’s lightweight directory access protocol (ldap) authentication system. the two interactions are put together to resolve the problem. the solution of using web services to replace redirecting users to classic webvoyage was implemented in wmu new catalog in january, 2010. voyager offers its web services via the apache tomcat server. apache tomcat is an open source servlet container developed by the apache software foundation (asf). tomcat implements the java servlet and the javaserver pages (jsp) specifications from sun microsystems, and provides a “pure java” http web server environment for running java code. web services are defined as follows in wikipedia: a web api is typically a defined set of http request messages along with a definition of the structure of response messages, typically expressed in json or xml. while “web api” is sometimes considered a synonym for web service, the web 2.0 applications typically have moved away from soap based web services towards more direct rest style communications. (wikipedia contributors 2010) webvoyage web services are provided via the voyager x services web services (vxws) services. there are vxws and vwebv services provided by voyager’s apache tomcat. regular voyager’s webvoyage (version 7.0) are served from the vwebv directory. a definition regarding the scope and functionality of voyager’s web services was detailed in an ex libris’s press release: the set of web services that was introduced previously has been enlarged to increase the openness of voyager, facilitating its interoperability with other systems and enabling customers to develop code extensions that they can share with the voyager community. all web services are available in the developer zone of the el commons collaborative web site. (continued momentum for voyager… 2009) the developer zone of the codeshare (previously known as el commons) website hosts programming-oriented information related to ex libris systems and serves as a channel of communication among community members: customer developers from the user community and ex libris developers. these voyager web services are sets of application programming interfaces (apis) that enable the interaction between voyager and other application programs. there are two types of open interfaces in voyager’s web services: restful apis and xml over http. both types of interface are provided via the vxws services in voyager. wmu uses xml over http web service for the applications deployed at wmu described in this article. these web services are free of charge to licensed customers of voyager. according to duong, the webvoyage web services design is characterized by stateless, atomic transactions and consumer/provider (client/server) architecture [1]. each web service requires an opac instance to access the oracle database. ex libris designed these web services for the following rationales: the interoperability between different systems, the integration between different applications, to enrich customer functionality, and to create a federation of resources and to facilitate faster time for software development cycle. detailed information regarding these services is outlined in the collaborative web site: http://www.exlibrisgroup.org/display/voyageroi/xml+over+http+web+services. web services include search services, media services, patron requests services, etc. duong (2009) further identified the process of setting up the web services. the configuration file on voyager server is vxws.ini. opac/circ server configuration in the server.xml was also verified. and further apache web server, tomcat, and oracle database connection were validated, and that opacsvr/circsvr is operational. according to ex libris, the voyager’s web services are heavily used by ex libris’s discovery tool primo. libraries that choose to integrate voyager with the primo® discovery and delivery solution by ex libris group provide their users with a universal solution for the discovery and delivery of all material types—print, electronic, and digital—regardless of format and location. (voyager integrated library system 2010) it was essential that wmu’s new catalog include functionality that allowed end-users to place holds/recalls, renew items, use account services, add tags, and mark items as favoriates. it also includes options such as utilizing personal bibliographic software such as refworks, and endnote. functionality as such involves operations from both voyager and vufind. see figure 1: web api, and vufind connections and figure 2: wmu’s vufind connections to other local systems for vufind’s interaction and inter-operation with voyager and other local systems such as the university ldap system. figure 1. web api and vufind connections.   figure 2. wmu’s vufind connections to other local systems. the university computing (office of information technology) has strongly “suggested” that a single interface for authentication is to be used to ensure security and uniformity among various campus systems. ldap is the single authentication interface implemented at wmu. in addition to the university’s suggested single interface requirement, it was also desired to simplify the different user ids used among different library systems such as interlibrary loan (illiad), michigan electronic catalog (melcat), etc. please see figure 3 for the various library accounts for wmu users. figure 3. various library accounts for wmu users. there is no software module or library in voyager that works directly with a ldap server. however, there is a standard module in vufind which allowed wmu to use ldap to authenticate users by using the university’s bronco netid (bronco netid is the name given to the university’s single sign-on ldap system.) the decision to use ldap with the new catalog was thus made based on the above rationales. in order to interact with the university computing’s ldap server, it was necessary that a configuration file be set up in wmu’s new library catalog. for wmu’s new catalog to complete the voyager’s web services, the extra step of the interaction with voyager’s oracle tables to get patron’s last name and institution id is also required. placing holds and recalls, renewing loans, looking up account details, and similar functions are important features that are heavily used by patrons at wmu. the implementation of these functions require some pertinent information to be retained from the users’ input and some additional information retrieved by querying voyager’s oracle database. once this information is obtained, then the transactions will be constructed. as mentioned previously, due to the license restrictions, transactions such as renew cannot be written directly to voyager’s oracle tables by using php or other programming languages together with their sql insert/update commands. however, web services provided by voyager can be utilized to work with oracle tables. as a result, php together with its http request is used to implement voyager web services to achieve the above needs and functions. the request process includes the following (see figure 4): the user places a request on a bib record/holding record. the process will check to see if the user has logged in. if the user is not logged in, then the user will be directed to the login/authentication page. if the user is logged in, the user will be directed to the page where other options such as pickup locations are presented. authfactor type i is used for authentication with voyager web services. before placing the request with voyager web services, an oracle oci is placed to query voyager about the user’s last name and institution id. (see appendix a – php pdo’s interact with voyager oracle™ tables). the hold request is placed via voyager web services. see appendix b – code snippets for renewing which require patron id, last name, and institution id. see appendix c – code snippets for hold/recall. after the request is placed, the user is redirected to the account page. the request is verified and listed there. figure 4. request process. the hold/recall php function is very straightforward. see appendix d – the hold/recall php program in wmu new catalog. a special thank you goes to purdue university’s php programmer matthew t. riehle for sharing the original codes with the vufind community. wmu heavily revised the original code created by riehle to accommodate the local needs. ex libris provides technical seminars that librarians can attend that cover “using voyager web services.” in these technical seminars, the perl programming language is typically used for demonstration. iv. conclusion the implementations described in this article detailed the possibility of integrating applications from different parties. such a project creates more opportunities to enrich functionality and ease of use for opac users. providing users the options of placing a hold on items and selecting pickup locations provides unique opportunities for user services. this endeavor further reinforces the university strategic plan to implement a single sign-on system. doing so further helps to serve users on various campuses so they do not have to travel across campuses and thus their travel time is shortened. users’ high volume of utilizing such user services will be anticipated in the coming semesters. similar projects, yet to be developed by the library community, such as code4lib, are likely to further assist users and enrich their use of the library catalog. appendix a: php pdo’s interact with voyager oracle™ tables public function patronlogin($barcode, $lname) { $sql = "select patron.patron_id, patron.institution_id, patron.last_name from . $this->dbname.patron where patron.institution_id = '$barcode'" ; try { $sqlstmt = $this->db->prepare($sql); $sqlstmt->execute(); $row = $sqlstmt->fetch(pdo::fetch_assoc); if (isset($row['patron_id']) && ($row['patron_id'] != '')) { return array('id' => $row['patron_id'], 'institution_id' => $row['institution_id'], 'last_name' => $row['last_name']); } else { return null; } }catch (pdoexception $e) { return new pear_error($e->getmessage()); } } appendix b: code snippets for renewing which require patron id last name and institution id. note: see appendix a for how to get the patron’s id and last name and institution id and see document from voyager’ codeshare site for complete codes and vufind-tech listserv for complete codes. the renew function should include paramterized the ws consumer(i.e. pass itemid and patron information) , such as: public function renewmyitems($itemid, $patron) create a client of http_request which will call renewservice: $client = new http_request('http://voyager.library.wmich.edu:7014/vxws/renewservice'); xml message should include authfactor: <ser:patronidentifier lastname="' . $patron&#91;'last_name'&#93; . '" patronhomeubid="1@wmichdb20020730122958" patronid="' . $patron&#91;'id'&#93; . '"> <ser:authfactor type="i">' . $patron['institution_id'] . '</ser:authfactor> </ser:patronidentifier> should include service names: <ser:definedparameters xsi:type="myac:myaccountserviceparameterstype" &#91;/sourcecode&#93; </pre></li> <li>should include item information: [sourcecode language='xml'] <myac:itemidentifier> <myac:itemid>' . $itemid . '</myac:itemid> <myac:ubid>1@wmichdb20020730122958</myac:ubid> </myac:itemidentifier> appendix c: code snippets for hold/recall. note: see document from voyager’ codeshare site for complete codes and vufind-tech listserv for complete codes. the function should pass the following parameters: $bibid, $itemid, $patron, $pickuplocation. for example: public function submitrecall($bibid, $itemid, $patron, $pickuplocation) ser:parmater should include: <ser:parameters> <ser:parameter key="bibdbname"> <ser:value>western michigan university libraries westcat</ser:value> </ser:parameter> <ser:parameter key="bibdbcode"> <ser:value>wmichdb</ser:value> </ser:parameter> <ser:parameter key="requestcode"> <ser:value>recall</ser:\value> </ser:parameter> <ser:parameter key="reqnna"> <ser:value>90</ser:value> </ser:parameter> <ser:parameter key="cval"> <ser:value>thiscopy</ser:value> </ser:parameter> <ser:parameter key="requestsiteid"> <ser:value>1@1@wmichdb20020730122958</ser:value> </ser:parameter> appendix d: hold/recall php program in wmu new catalog <?php require_once 'catalogconnection.php'; require_once 'action.php'; require_once 'services/myresearch/lib/user.php'; require_once 'services/myresearch/lib/resource.php'; require_once 'services/myresearch/myresearch.php'; require_once 'record.php'; class hold extends record { function launch() { global $interface; global $configarray; global $user; // check if user is logged in if($user){ ob_start(); $catalog = new catalogconnection($configarray&#91;'catalog'&#93;&#91;'driver'&#93;); $patron = $catalog->patronlogin($user->username, $user->password); $interface->settemplate('hold.tpl'); // display page $interface->display('layout.tpl'); if(isset($_post['hold'])){ $catalog->submitrecall($_get['id'], $_request['itemid'], $patron, $_request['pickuplocation']); header('location: ' . $configarray['site']['url'] . '/myresearch/holds'); ob_end_flush(); } } else{ ob_start(); header("location: " . $configarray['site']['url'] . "/myresearch/home?sendurl=https://" .$_server["server_name"] .$_server["request_uri"]); require_once 'services/myresearch/login.php'; login::launch(); exit(); ob_flush(); } } } ?> notes: continued momentum for voyager with the release of version 7.1. 2009 jun 25. [place unknown]: library technology guides; [cited 2010 dec 4]. available from: http://www.librarytechnology.org/ltg-displaytext.pl?rc=14041 duong, c. 2009 mar 26. focus point of open platform, voyager web services. at: developers meet developers march 2009; [cited 2010 nov 17]. note: some presentations from this meeting are available at: http://www.exlibrisgroup.org/display/presentations/developer+meets+developer+march+2009 voyager integrated library system. c2010. [place unknown]: ex libris limited; [cited 2010 dec 4]. available from: http://www.exlibrisgroup.com/category/voyager wikipedia contributors. web api. wikipedia, the free encyclopedia; 2010 nov 16, 8:45 utc [cited 2010 dec 4]. available from: http://en.wikipedia.org/wiki/web_api. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – developing applications in the era of cloud-based saas library systems mission editorial committee process and structure code4lib issue 26, 2014-10-21 developing applications in the era of cloud-based saas library systems as the move to cloud-based saas library systems accelerates, we must consider what it means to develop applications when the core of the system isn’t under the library’s control. the entire application lifecycle is changing, from development to testing to production. developing applications for cloud solutions raises new concerns, such as security, multi-tenancy, latency, and analytics. in this article, we review the landscape and suggest a view of how to be successful for the benefit of library staff and end-users in this new reality. we discuss what kinds of apis and protocols vendors should be supporting, and suggest how best to take advantage of the innovations being introduced. by josh weisman, ex libris introduction no system lives in a vacuum, and no system provides 100% of features required by its users. this maxim is equally true of library systems, and many institutions invest in the development of applications that live along side the primary library management system. additionally, code is often required to integrate the library management system within the larger institutional technology landscape. this integration code is equally critical for the smooth operations of the library. in a local environment, the options for integrations are manyaccess to the server environment, direct sql queries against the database, log file access, manipulation of configuration files. the institution has full control over the application and its infrastructure. as more organizations seek to enjoy the benefits of moving applications and infrastructure to the cloud, many are joining the wave of adoption of cloud-based library management systems. software as a service (saas) offerings in this space are increasing in number and features. developers face new challenges in designing, coding, and testing their applications against these saas systems. this article discusses the considerations for vendors when designing their api and integration offerings for cloud-based systems, and for developers when integrating with those systems. it is based on the experience ex libris gained in building our developer network1, a next-generation developer and api platform for our cloud-based library systems. many of the examples and screen shots in this article are taken from our implementation. development lifecycle there are many models for the software development lifecycle, but most of them can be summarized as design, develop, test, deploy. figure 1. development lifecycle when developing applications against locally-hosted systems, there are often a number of installed environments that support this lifecycle. when working against cloud-based systems, we need support for a similar tiered-development environment. we also need the supporting material and documentation to be thorough and accessible. design: documentation good documentation is important for all development projects. in the cloud, it is essential. developers must be provided with details of the available apis to understand what functionality is possible. details about the input and output parameters are necessary to follow the data flow between the custom application and the library management system. workflow examples should be provided to help analysts orchestrate the apis into full stories. in the cloud, vendors have an opportunity to ease the design phase by making api documentation openly available and searchable by standard search engines. removing barriers such as usernames and passwords allows developers and analysts to more easily discover the information they need and share it with others. automatic generation of documentation cloud systems operate on an accelerated release cycle. keeping documentation up-to-date with changes in the product is even more challenging. documentation which is out-of-date or incorrect erodes the confidence of developers and reduces their interest in using the product’s apis. automating the creation of api documentation is one way to reduce friction. automation means the documentation is always in synch with the api and is always accurate. our process leverages-third party tools in addition to some homegrown code. figure 2. automatic documentation process first, the api developers add documentation directly to the api code. this is done for both the web service description and for the data transfer objects which are passed as a payload in the body of the api request or response. for web service descriptions, we add attributes to the code based on the apache cxf2 library. the cxf library then creates a wadl3 file based on the annotations. an example of the documentation annotations follows: @get @path("funds") @descriptions({ @description(value = "this api returns a list of funds.", title = "retrieve funds", target = doctarget.method), }) public funds getfunds( @description("the code of the library that owns the po line for which the relevant funds should be retrieved. optional.") @queryparam("library") @defaultvalue("") string librarycode) { figure 3. documentation annotation to produce documentation for the body objects, we use xsd schemas. the developers add documentation directly to the schema using the <xs:annotation> tags. <xs:element name="primary_id" type="xs:string" minoccurs="0" maxoccurs="1"> <xs:annotation> <xs:appinfo> <xs:tags>api sis get post put</xs:tags> </xs:appinfo> <xs:documentation>the primary identifier of the user. mandatory. note that the primary_id is case insensitive. </xs:documentation> </xs:annotation> </xs:element> figure 4. sample xsd documentation we further leverage the schema by placing tags on the elements which allow us to filter fields in our documentation based on parameters in the querystring. for example, we can provide an html page which displays fields that are relevant for post requests only. the page filters the fields based on the tags taken from the xsd. the final step in the automatic documentation process is to execute an xsl transform on the wadl and xsd files to produce html documentation4. this assures that the documentation provided to developers is timely and accurate. develop: console in traditional application environments, it can often take a long time for a developer to successfully make and request and get a response to an api call. once the test environment is deployed, the developer still needs to read the documentation, understand how to format a request, create the test script or application, and provide the required server details and credentials. in the world of cloud, we can significantly reduce this “time to first ‘hello world’” (ttfhw5) by providing a way to call the apis directly online, from a web browser, against a guest sandbox environment. an “api console” helps developers be productive quickly by removing barriers and allowing them to learn by experimentation. an api console provides details of the selected api call, allows the developer to enter the parameters, executes the request directly from the browser to a test or demo environment, and provides the response from the server. this instant feedback provides developers with quick successes, and allows experimentation with various parameters and formats. once the developer gets the desired results, the console can also provide a code stub in various languages that can be used inside a real application. this shortcut can help decrease development time and allow programmers to concentrate on the business logic rather than the “plumbing” code. figure 5. api console trial-and-error coding using an api console allows the developer to quickly explore the api and its parameters. for example, we can execute a get request in the console such as: get /myapp/v1/users/test accept: application/json the console immediately returns as response: date: tue, 16 sep 2014 19:07:59 gmt response-status: http/1.1 400 bad request content-type: application/json;charset=utf-8 content-length: 187 { "errorsexist": true, "errorlist": { "error": [ { "errorcode": "401861", "errormessage": "user with identifier j was not found.\n(tracking id: e01-1609190759-cbb7c-awae151072507)" } ] }, } we’re on the right track, but we need a valid user. once we modify the request slightly: get /myapp/v1/users/josh accept: application/json we get a better response: date: tue, 16 sep 2014 19:12:57 gmt response-status: http/1.1 200 ok content-type: application/json;charset=utf-8 content-length: 1438 { "primary_id": "josh", "first_name": "josh", } while this is a trivial example, these advances allow developers to explore the effects of parameters and the response objects provided by the server iteratively in a convenient environment. once satisfied, the console can produce scaffolding javascript code which can be used inside a real application. var xhr = new xmlhttprequest(); var url = '/myapp/v1/users/{user_id}'.replace(/{user_id}/g, encodeuricomponent('josh')); xhr.open('get', url + queryparams); xhr.setrequestheader('accept', 'application/json'); xhr.onreadystatechange = function () { if (this.readystate == 4) { alert('status: '+this.status+'\nheaders: '+json.stringify(this.getallresponseheaders())+'\nbody: '+this.responsetext); } }; xhr.send(''); this type of immediate feedback without the need to set up an entire development environment considerably reduces development time. test: sandbox once the application is developed, it must be tested. in local systems, this is usually done in a test or staging environment. many cloud providers offer a sandbox environment to customers that may include either sample data or a version of the customer’s actual data. developers should have api access to the sandbox environment to test their applications. deploy: production in a local environment, deploying an application in production often requires configuration changes to update server names, ip addresses, ports, credentials, etc. in the cloud world, we can offer a more seamless deployment process. in the example below, we see a developer using a dashboard to change their application from sandbox mode to production mode. without requiring any changes to their own application, it is now pointing to their production instance. figure 6. deploying an application to production the following figure depicts the development lifecycle in the cloud: figure 7. development lifecycle in the cloud modern standards the move to the cloud provides an opportunity to implement apis according to modern standards. any discussion of web api standards begins with the rest architectural style6. but even for those whose apis adhere to the rest style, there exists much room for interpretation. vendors should invest effort in defining a style for their apis and work to ensure consistency. doing so reduces friction for developers. we recommend that the vendor’s api style guide be shared with the developer community7. hard decisions as part of the process of defining an api standard, providers are forced to make some hard decisions. while there are widely-agreed guidelines for building modern apis (rest, json, etc.), there are other questions to which there is no clear answer. if the decision is well-reasoned and consistent, api consumers can accept and adapt to the decided-upon style. response codes for invalid objects one example of this type of question is the proper response code for a request to a uri of an object which does not exist. a client might make a call to /cars/123 while there is no such car with that id number. there are two possible approaches: return a 400 bad request, which indicates that while the service is valid (/cars), the requested car does not exist. provide additional details in the response body indicating why the request is bad. return a 404 not found, which indicates that the requested resource does not exist. while this accurately describes the circumstances, a client might be confused between the case of a valid service with and invalid id, or an invalid service (/carrs/123, for example) we decided on the first option in our implementation. non-crud services another example of a hard decision is how to handle non-crud requests in rest apis. the standard rest methodology provides for the following allocation of http verbs to crud services: crud action http verb create post read get update put delete delete however, there is no natural mapping for non-crud services. for example, in the library world, we have a service such as “renew a loan” or “receive an item”. we want to avoid falling into the trap of defining soap/rpc-like endpoints, and it’s important to stick with the rest standard of nouns as uris. so a service such as post /items/1234/loans/renew or post /items/1234/renewloan is undesirable. given this, our decision was to post to the object with the operation in the query string. a put request updates the object, while post is reserved to other undefined actions. the renew loan service might look like post /items/1234/loans/5678?operation=renew. again, api providers should make their decisions known in these cases so that api consumers can adapt accordingly. rest error handling one of the benefits of using modern standards is that most development environments provide considerable infrastructure which makes coding against such apis easier. library vendors should provide standards-compliant apis so that developers can take advantage of that infrastructure. one important example is how errors are handled. library system vendors who create apis should leverage http status codes to report errors back to client applications. rather than returning 200 ok with a response that looks like this: { “error”: “true”, “description”: “invalid input” } the api should return an http error. in general, an error code in the 400 range means that the client has provided invalid data and should not try the request again. an error in the 500 range means the server had an issue and the client is free to try the request again. this methodology allows the client developer to assume the request is successful and rely on the framework to raise a language-appropriate exception if a non-success http error code is received. this is illustrated with the pseudo-code below: try { make api call; handle successful response; } catch { handle exception; } security one of the main concerns that organizations have in moving to cloud-based systems is that of security. this concern carries over into access to saas systems via apis. apis provide access to sensitive data and must be secured as much as, if not more than, access by a standard user account. https api calls should be restricted to https only. any attempt to access an api via non-secure http should be rejected or redirected. an https only policy ensures the security of authentication credentials and sensitive data. the power of modern hardware reduces the performance impact https encryption requires. authentication in some instances, api calls may be intended to be openly available. in most cases, however, some sort of authentication is required. there are several strategies for authenticating api requests. basic authentication: a style of http authentication where a username and password are base-64 encoded and placed in an http header oauth8: an authentication standard which is best for supplying delegated access from one application to another; for example, allowing a custom application to post to facebook on a user’s behalf api keys: a long random string which identifies the caller of an api request and can include authorization information cloud applications are multi-tenant, which means that several institutions occupy the same server resources and application instance. therefore, authentication may also identify the institution to which the api request is intended. we have adopted the use of api keys, as many experts9 have recommended. this method allows api requests to the server to authenticate the request and route it to its intended institution. it is trivial for a developer to implement api keys for authentication, as they can be sent via an http parameter or header. they are also sufficiently long and random as to not be vulnerable to a brute force attack. finally, api keys can easily be revoked should the need arise. third party access many libraries today no longer have the resources in-house to embark on large development projects. for those institutions, outsourcing development work is an increasingly attractive option. using third parties to do development brings with it security challenges. best practices for third party access to apis include limited access to required apis only, auditing api requests, the ability to control access to test and real data, and the ability to revoke access when needed. cloud library management system vendors should take these requirements into consideration when designing api platform security. the api key authentication methodology described above works well for third party access. api keys can be uniquely assigned to a particular application or contractor, can limit access to the desired apis, and can be configured for access to the desired environment. attack mitigation the move to the cloud allows vendors to deploy edge devices that enable mitigation against common web attacks. such gateways protect customers’ data from intentional or accidental attacks via api calls. these gateways inspect each request and evaluate them against known attack patterns. suspicious requests are rejected before they even reach the application servers and the customers’ data. some common web attacks that api gateway devices can protect against include: cross site request forgery (csrf) xml body attacks, such as long text or attributes, long attribute names, and node nesting depth code injections, including sql injection, shell injection, and xpath injection validating payloads against json or xml schema message size limit to prevent a denial of service attack cross site scripting (xss) figure 8. api gateway cloud compatibility the cloud brings with it challenges for a software vendor. taking a system designed to be used locally and trying to adapt it for cloud deployment is a difficult undertaking. similarly, taking apis from a locally-installed system and making them available without modification in the cloud is not sufficient. there are api requirements unique to a cloud-based system that must be addressed. multi-tenancy governance at the heart of any saas application is multi-tenancy. in order to achieve economies of scale, vendors host many institutions on the same infrastructure and the same software instance. to ensure that a single institution does not negatively impact other institutions most cloud vendors implement a governance policy. such policies limit the amount of transactions institutions can do in a defined time frame in accordance with the size of their licensing agreement. for api platforms, these governance policies require the implementation of two types of limits: throttling and quotas. throttling ensures that no more than a specified number of api requests for a single institution are processed in a short period of time, per second or per minute. this prevents an institution from monopolizing server resources thereby preventing the requests of other institutions from being processed. requests that exceed this limit can be queued and processed subsequently. quotas define the total number of requests that an institution can perform in a longer period of time, generally a day. this ensures that resources are being utilized efficiently and proportionally. self-service the cloud opens opportunities for efficiency in process too. a well-designed api platform enables institutions to manage their own environment. for example, cloud api systems can provide the ability for a library to perform functions such as creating its own api keys, managing access to its environments, and adding developers to their account. self-service improves efficiency and reduces tco of the solution for the library. figure 9. self service dashboard latency cloud systems are hosted remotely from the library’s network. there will naturally be some additional latency for each api request. it’s important that developers take this into consideration when designing their applications by minimizing the number of calls and data requested from the server. in addition, vendors can help limit the affects of latency by supporting features in their rest api implementation that can help reduce the bandwidth required for larger requests. such features include http compression10 and setting caching policies and headers11 that allow clients to cache the retrieved data where appropriate. community the library technical community has always been strong. the move of library systems to the cloud opens up additional opportunities for that community to work together to share knowledge and leverage the work of others. sharing code the explosion of open source projects in recent years provides a strong platform for the sharing of code in library management solutions. sites such as github and sourceforge allow developers to make their code repositories public and to accept fixes and improvements from the community. these advances can shorten the time required to get a resolution for issues and allow interested developers to help move the projects forward with their contributions. cloud library system vendors need to leverage these advancements and encourage the use of open source repositories for applications and integrations that work against their systems. the opportunity also exists for vendors to make parts of their systems or integration bridges with other systems open source, thereby realizing those same benefits. in the spirit of openness, vendors should avoid creating walled gardens of code contributions, documentation, forums, etc. wherever possible these communities should be openly available avoid recreating the functionality of community-accepted repository solutions. sharing experiences developers today work with their code open in one window and google in another. the internet is filled with blogs, q & a sites, and other resources that allow programmers to search for the advice of others who have had the same problem. as library system developers, we have a responsibility to contribute to that huge knowledge base to help others who will encounter the same issues we face. library platform vendors can provide a platform for the community to share success stories, integration recipes, tips and tricks, code samples, and other content that will benefit the community. a blog is one popular way this can be accommodated. vendors can contribute blog entries themselves, but the blog should be open so that others can contribute as well. contributions should be encouraged, curated, and openly discoverable by search engine queries. sharing solutions when all else fails, developers need a way to ask questions. popular question and answer sites such as stackoverflow provide a great opportunity for programmers to get help when trouble strikes. often sections of such sites are created organically for popular development languages, environments, and frameworks. library system vendors can also offer a forum for those who develop on their systems. vendor staff should monitor the forum and answer questions where appropriate. ideally community members will also participate in forum discussions and help colleagues with their challenges. advanced features the move to the cloud provides an opportunity for vendors to add additional features for those who develop against their systems that local topologies made impractical or impossible. analytics one major feature the cloud enables is analytics. since all api requests are routed through the cloud, vendors can offer developers reports that show the number of api requests, reported per application. that allows developers to monitor the usage of their applications against any quotas. reports can also show latency data, which allows developers to understand how long each request takes and monitor their user experience. figure 10. analytics report of api requests conclusion the general it trend of moving to the cloud brings with it new opportunities and challenges, and library systems are no different. in this article, we’ve discussed the benefits to developers that this new era brings, including improved developer experience, more transparency and analytics, lower maintenance cost for applications, and opportunities for social coding and sharing. for vendors, there are challenges, but those who position themselves well in this new world can leverage technology to offer a better experience to their developers. notes 1 the ex libris developer network is openly available at https://developers.exlibrisgroup.com/. 2 http://cxf.apache.org/ 3 web application description language, the wsdl of rest apis. 4 our xsl transform code is available at https://github.com/exlibrisgroup/wadl-doc. 5 see http://sendgrid.com/blog/three-ways-to-decrease-time-to-first-hello-world/ 6 see http://en.wikipedia.org/wiki/representational_state_transfer 7 see https://developers.exlibrisgroup.com/blog/how-we-re-building-apis-at-ex-libris for an example of such a style guide 8 see http://en.wikipedia.org/wiki/oauth 9 see https://stormpath.com/blog/secure-your-rest-api-right-way/ for example. 10 see http://en.wikipedia.org/wiki/http_compression for more information 11 a nice introduction is available at http://www.mobify.com/blog/beginners-guide-to-http-cache-headers/ about the author over an eighteen-year career in the software industry, josh weisman has managed development teams working on a wide range of applications, from enterprise workflow tools to large scale e-commerce websites. he currently is a development director at ex libris, where among other duties, he is responsible for interoperability of their next generation library management system. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – the library search engine: a smart solution for integrating resources beyond library holdings mission editorial committee process and structure code4lib issue 4, 2008-09-22 the library search engine: a smart solution for integrating resources beyond library holdings the cooperative library network berlin-brandenburg (kobv), germany addresses the problem of how to integrate resources found outside the library and library holdings into a single discovery tool. it presents a solution that uses open source technology to develop a next-generation catalog interface called the library search engine. this pilot project was launched in 2007 with the library of albert einstein science park, potsdam. the idea was to design and develop a fast and convenient search tool, integrating local holdings (books, journals, journal articles) as well as relevant scientific subject information such as open access publications and bibliographies. by karin herm and sibylle volz introduction with the following article, we want to share experiences from a pilot project that started in 2007, reached production level that same year, and is still developing further. within this project we used search engine technology to search both catalog data and external subject relevant information and developed the library search engine (lse). the library technology world is changing quickly driven by the fast alterations in the information technology sector. this trend has intensified since the triumph of search engine technology. quite heterogeneous projects implementing these new technologies can be subsumed under the slogan “working on the next-generation catalog.” in the area of non-commercial or open-source software solutions we find well-known projects such as vufind, the danish summa project, and last but not least our new lse. the cooperative library network berlin-brandenburg (kobv) [1] is one of the six existing library networks in germany. the kobv offers it and other services to libraries in the german states of berlin and brandenburg. currently, we have more than 70 member libraries, ranging from academic to special and public libraries. one of our key aspects of activity is the development of new services. two small development teams handle one or more projects. they consist of one or two technical developer(s) and one librarian. the team responsible for driving the library search engine project had a strong background in implementing search engine technology, gained from an earlier project. in order to make this know-how available for our member libraries the library search engine pilot project was launched in april 2007. background and main goal library patrons typically need access to both library holdings and other subject specific information resources, e.g., open access publications. they are usually comfortable using different search tools, but have difficulties in identifying and locating the appropriate resources. furthermore, it is a well-known issue that patrons are neither satisfied with interface and retrieval features of the search component of traditional online catalogs, nor with the presentation of search results. additionally, scientists and academics want fast and easy access to as many relevant full-text publications as possible. we set ourselves the following goal: we didn’t want to reinvent the online public-access catalog (opac) or substitute the integrated library system (ils), since we were quite satisfied with its circulation module, the user sign-in, and the personalization features. instead, we focused on creating a tool to compensate for the unsatisfactory search and retrieval components of the online catalog. that was where we identified the most urgent need for action. our ambition was to expand the search space and to process metadata and – where technically and legally possible – full-text from both the library’s own collections and external resources, indexing them by using search engine technology and presenting them in one search engine application. our pilot library, the library of albert einstein science park, potsdam, is a joint library of three research institutes. it offers services to geoscientists and climate and polar researchers. this library was unsatisfied with the search component of their opac (self-knitted, the ils itself is an oclc sisis-sunrise system), which they wanted to replace, and thus enthusiastically signed up as the pilot library. in our first meetings with the head librarian and one of his staff members, we agreed to design and develop a fast and convenient search tool, integrating local holdings (books and journals) and relevant scientific subject information such as open access publications or metadata from bibliographies that were compiled during research programs and not yet publicly accessible. approach our project team mainly consisted of four people: the head librarian of the pilot library; one of his staff members, who had both a strong it background and experience in the library field, and this article’s two authors from the cooperative library network (kobv). also, the implementation was supported by a student and other library staff tested the application. as mentioned above, we wanted to develop a fast and convenient search tool that integrates data from a variety of sources, but of course none of us had a clear-cut idea of what this should look like. thus, our approach was to work with a rapid prototyping ansatz, i.e., fast development of a prototype with certain functionality and certain “mock-up” functions to quickly get a look and feel of the application and visualize the data at hand. this approach proved even more valuable later on because we were able to respond with flexibility to the needs of our pilot library [2]. the rapid prototyping approach was also useful with regard to the choice of data sources: we started out with six and added more and more sources during development finally reaching ten. we were able to verify the scalability of the design during development as these sources could be added without major increase of implementation time. solution this section describes the data we included, gives an outline of the implementation, and depicts the flow of data from the library to the search engine user. in addition, we explain the user interface of the resulting application. data the lse handles various heterogeneous data sources: records of two library catalogs, journal title records of all licensed journals (electronic and print), data from two institutional publication databases, metadata of open access articles from an open access portal focusing on geology and related sciences called geo-leo [3], a primary data collection, and content of bibliographical information databases. the current article metadata from tocs of the library’s e-journal subscriptions and an e-book collection are preprocessed and provided as xml by the library, which receives the content via rss feeds and email. one common feature of all data is the basic format: everything is provided in some form of xml, but the “dialects” vary: mabxml (definition: http://www.d-nb.de/standardisierung/pdf/mabxml_1_dok.pdf (german)) scientific primary data (http://www.std-doi.de/) journal metadata from a journal database (http://www.allegro-c.de (german)) publication servers: harvested via oai-pmh using metadata format oai_dc many variations of oai_dc (rss-feeds, e-books, geo-leo data, bibliographical information from different sources), and full-text wherever available and accessible. quite a few sources in the above list use the oai_dc metadata schema (http://www.openarchives.org/oai/2.0/oai_dc.xsd), which is a very open definition, i.e., each of the defined fields can occur multiple times or not at all and their content is pretty much arbitrary. therefore, it was necessary to define different parsing structures for each of the supplied xml data sources, but the underlying parser for the oai_dc schema is, of course, the same. implementation we decided to build the project’s application using free and stable open source technology wherever possible in order to reduce implementation cost. in fact, commercial solutions were out of the question due to the non-existing budget. this led to a java application based on apache tomcat and the search engine library apache lucene. other open source “ingredients” are: ant, log4j, apache commons (e.g. digester, utilities), pdfbox, and many more. to present the different data types in a user interface and to make the metadata searchable we had to normalize our data. as we “simply” wanted to provide a search engine, we could limit the data stored to the fields we wanted to search and present. hence, it was possible to work without a database, which also reduced the demands of data quality (e.g., consistency, completeness, etc.). thus, the library could focus more on content than on metadata quality, and we could avoid implementation of components for verifying and correcting data. the normalization of the data was defined according to the search and the user interface, i.e., the librarians on our team defined the mapping for each of the data types to searchable fields, export features, and display fields. these fields and a few additional technical fields (e.g., identifiers for accessing the ils) are stored in the search index. technically, the application can handle any (reasonable) number of lucene indexes (see sample 1 and sample 2). the question of which and how many is just a matter of configuration. this makes it easier to keep the data up-to-date or to add new sources. # listing of all index directories (1-n) that should be accessed by the # search engine # indexdirectory1=pathelement1/index1 indexdirectory2=pathelement2/index2 indexdirectory3=pathelement3/index3 indexdirectory4=pathelement4/index4 sample 1: sample from the configuration file try { indexreader[] ir = new indexreader[count]; indexsearcher[] is = new indexsearcher[count]; int fehler = 0; for (int i=0; i < count; i++){ try{ ir[i-fehler]= indexreader.open(conf.getstring("indexdirectory"+(1+i))); is[i-fehler] = new indexsearcher(ir[i-fehler]); }catch(exception ie){ fehler =+1; log.error("index zugriff fehlgeschlagen (count="+i+" fehlerzahl="+fehler+") "+ ie.getmessage()); } } indexreader[] irbereinigt = new indexreader[count-fehler]; indexsearcher[] isbereinigt = new indexsearcher[count-fehler]; for(int i=0; i < count-fehler; i++){ irbereinigt[i]=ir[i]; isbereinigt[i] = is[i]; } log.debug("erzeuge reader und searcher."); reader = new multireader(irbereinigt); searcher = new multisearcher(isbereinigt); }catch (exception e) { log.error("index zugriff fehlgeschlagen; "+ e.getmessage(), e); } [/sourcecode] sample 2: code sample demonstrating the use of lucene’s multireader and searcher during the development phase the library identified the need to fine tune the search results ranking. as the number of the external sources exceeded the number of library catalog records tremendously, so that the library records were hardly visible in the result lists. but, of course, they contain the most relevant information. we adjusted the ranking using the standard lucene boosting functionalities so that – still depending on their relevance to the user’s search – library records receive a higher ranking than external sources. we end this section with a few numbers about the application: currently 265,000 records, increasing with every update indexed full-text resources: 3,500 and increasing with every update amount of data: 3 gb increasing search time well below a second – not increasing lines of code: 18,500. dataflow how the (relevant) data reaches the user: the library identifies the data collections to be included in the library search engine and adapts or builds xml files according to the specification we defined during development. for instance, they export catalog records from their library catalog and convert these to the appropriate format – mabxml using a tool provided by the german national library [4], or they produce oai_dc records from e-book email alerts they receive from the publisher. most of the data is regularly updated via ftp or harvested online from the publication servers. other data doesn’t change that frequently or not at all, e.g., the publication databases or the contents from geo-leo (approximately twice a year). once the data becomes available to the lse it is normalized and processed resulting in a lucene search index. searches are entered through the web interface, features are processed with lucene search functionalities, and results of the relevant data are retrieved from the index. figure 1: processing of data information search features and graphical user interface after having thrown a short glance at the backend procedures, let’s have a look at the web interface functionalities (see figure 2): the library search engine offers common search engine features such as “did you mean” functionality, relevance ranking (with a boosting of local holdings), fast retrieval, intuitive interface, and stable performance. the simple search is the default search option, but we also offer advanced search and an a-z list of all journal holdings. we tried to simplify navigation, which means for instance that we offer export and mailing of search results directly from the results list so that it is not necessary to save results first to a saved results list, which we nevertheless still offer. figure 2: web interface of the lse another feature is the linking of author names and series titles within the search results list. the activation of these links automatically starts a new search with the selected author name/series title. also, results from records of the two library catalogs link back to the catalogs for status information and user account functionalities. if the indexed metadata contains links to full-text publications, abstracts, or journal websites, we index these full-text resources and link directly to the full-text and/or related websites. when starting the project, we considered offering within the advanced search the possibility of limiting the number of searched data sources before starting a search. in the course of further development, we changed our minds, since search time is fast enough even if we always perform a search over the entire index. therefore, we decided to offer the possibility to partition the results after having executed a search. we assembled our data sources into four different result groups: books (data from the library catalogs), journal titles (journals held by the library), open access (open access publications and primary data), and more (current contents, e-books, publication databases, bibliographies). going live going live with the library search engine was short and sweet. without any announcement, the library’s opac search component was closed and replaced with the lse as the unique access point for searching the library’s media and all additional resources [5]. user feedback was astonishing: either there were no remarks at all or they mentioned that they had long awaited a tool like this! in some ways, users took for granted not only using the new web interface but also finding catalog records and external information together in one application. the library carried out a contest among their users to find their own specific name for the new search tool soon after the implementation. ‘marvelous’ was one of the proposals which was handed in – this tells its own story. however, the library chose the more descriptive name ‘albert – all library books, journals and electronic records telegrafenberg’. (‘telegrafenberg’ is where the library is located.) conclusion patron and staff feedback have shown that we are on the right track. how would our other member libraries react to the new search tool? we conducted a workshop where we explained in detail the functionalities and usage possibilities of the library search engine. afterwards, six libraries stated that they were interested in implementing a local version of the lse for a central or alternative access point to their catalog records and other data collections. but thorough analysis of the responses revealed that the libraries are actually seeking the one access point to their complete collection, including licensed databases. we all know this is something libraries and their patrons want, however, it is something we cannot offer. unlike present commercial products such as primo by ex libris or sirsidynix’s enterprise portal solution, we don’t include a distributed search module. the library search engine needs to obtain the metadata and/or full-text materials to build the index(es), which might be a problem with licensed resources. libraries have already started asking for metadata, and especially with metadata from backfiles of licensed e-journals this has been successful. nevertheless, this is something each library or library consortium has to deal with. kobv is only a service provider and doesn’t hold any licensed material itself. that is why we can’t clarify legal and license questions. the complexity of this implementation presented another challenge. by realizing this project, we demonstrated that it is possible to develop and implement this new search tool for libraries. deployment of open source technology has ensured cost limitation. however, the further procedure is undetermined, because we are unsure that the effort to implement the application for another library is really justifiable, considering our small-sized team of two people. with the pilot project, we wanted to create a standard application that could easily be implemented for other libraries. but we are very aware that these libraries will have other requirements. for instance, an academic library with different branch libraries has different needs than our singular pilot library. this means that the development and implementation of the library search engine for another library is still far from being a standard procedure and is therefore work intensive. furthermore, there are features that have to be improved, for instance update procedures, or to be expanded, for instance reusing options through open-sourcing. we are currently analyzing our situation and seeking both national and international cooperation and exchange. we would be pleased to receive feedback from libraries or institutions who are involved in similar projects. [1] for more information please visit: http://www.kobv.de [2] for example, the implementation of the a-z journal list: a few weeks after the launch of albert (library search engine (lse) during the development phase), many library user complained, because they missed the alphabetical list of all licensed journals, which used to be supplied as a static web-page. we could easily supply the wanted list dynamically employing search technology. [3] http://www.geo-leo.de/geoleo/www-docs/ [4] http://www.d-nb.de/eng/index.htm; mabxml tool: ftp://ftp.ddb.de/pub/tools/mab/ [5] visit http://waesearch.kobv.de to get a firsthand impression! about the authors karin herm (zibdevelopment-searchengine at yahoo dot es) is a mathematician and currently working as research associate at the cooperative library network berlin-brandenburg (kobv), germany. sibylle volz (volz at zib dot de) is also currently working as research associate at the kobv. she received her master’s degree in library and information science in 2004 at humboldt-university, berlin. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – editorial introduction — issue 4 mission editorial committee process and structure code4lib issue 4, 2008-09-22 editorial introduction — issue 4 welcome to issue 4 of the code4lib journal! we are pleased to present articles covering an impressive breadth of topics. the strength of the code4lib journal lies in its readers. you are not only the audience but also the authors for the articles published here. we hope the items you see in this, and future, issues inspire you to innovate and to share your discoveries with the rest of us. welcome to issue 4 of the code4lib journal!. the journal sports an updated look that gives a bit more streamlined and unified appearance to articles. thanks go to sean hannan and editorial committee member jonathan brinley for the initial inspiration and the implementation of the final design, respectively. in keeping with our tradition (if 4 occurrences of something can have such), this issue contains articles on a breadth of topics. we start with a handful of how-to articles: daniel talsky on using the serial solutions link resolver api to auto-populate an interlibrary loan form and alfred kramer on mining citation data from isi web of science® to assist in deciding whether it is more cost effective to subscribe to a journal or purchase articles one at a time based on citations of that journal in your scholars’ work. ross singer and james farrugia provide details about jangle, a specification employing simple, common and well supported web protocols to enable the creation, maintenance and distribution of resources among library-related data sources. pam sessoms and eric sessoms describe libraryh3lp, an integrated im and web chat system designed specifically for virtual reference services in libraries based on the jabber protocol. libraryh3lp offers an open-source alernative to current tools such as meebo. in his article describing the workings of the openbook wordpress plugin, john miedema describes the tool he wrote for book reviewers, book bloggers, library webmasters, and anyone else who wants to put book covers and data on their wordpress blog or website. karin herm and sibylle volz write about the library search engine, a next-generation catalog implemented at the cooperative library network berlin-brandenburg (kobv) in germany. the lse indexes local bibliographic holdings, open source items, licensed journals, institutional publications, primary data, and the content of bibliographical databases. we conclude the issue with a book review by editorial committee member christine schwartz. schwartz compares two books about frbr (functional requirements for bibliographic records): robert maxwell’s frbr: a guide for the perplexed and understanding frbr: what it is and how it will affect our retrieval tool, edited by arlene taylor. the strength of the code4lib journal lies in its readers.  you are not only the audience but also the authors for the articles published here. we hope the items you see in this, and future, issues inspire you to innovate and to share your discoveries with the rest of us. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – marc21 as data: a start mission editorial committee process and structure code4lib issue 14, 2011-07-25 marc21 as data: a start the forty-five-year-old marc format, currently at version marc21, is an obvious barrier to the provision of library services in a web-based environment. there is a growing consensus that the time has come for libraries to move to a new format. we cannot, however, decide on a new data format until we at least have an inventory of the data elements that are carried in our current one. listing those data elements is not simple: over the years this record format has undergone constant change that has pushed the limits of the record structure and introduced inconsistencies in the way that data is coded. this article describes one person’s attempt to decode the content of marc21. by karen coyle in the library community there is a growing recognition that there will need to be a successor to the marc21 bibliographic format. discussions about this tend to focus on structural issues: will the new format be an xml format? will it make use of rdf and linked data standards? what these questions do not address is the much more difficult task of translating the semantics of library data into something new. it takes only a short investigation of the data coded in marc21 to reveal that the tags and subfields themselves are inadequate to define the actual data elements carried in library catalog records, as jason thomale discussed in his recent article [1]. it is therefore logical that a first step in the transformation of marc21 to another format (or formats, for there may be more than one) is to identify the data elements that are contained within the marc21 record. that task, as thomale evidenced in his example, is more difficult than it may seem. a few caveats are in order. the analysis here is an analysis of marc21 format data elements, not of marc21 instance data. this mapping will not be sufficient to translate individual marc21 records to a different format; that would require further information that would need to be provided by one or more applications. this analysis is intended solely to identify the bits of information that library data can contain. to illustrate, the marc21 field definition for an isbn is tag 020; an instance of the isbn would be the isbn number itself tagged in a marc21 record: 020 $a 0877790019.  the description of the marc21 bibliographic format is available at http://www.loc.gov/marc/bibliographic/, while the records in library catalogs that use this format are examples of instance data. this analysis is currently limited to the marc21 bibliographic format. i suspect, however, that the techniques used here and the patterns that are revealed will be the same in the authorities format. none of this is intended to guarantee round-trip mapping between marc21 and any other data format. i decided early on that to insist on round-tripping for all data elements would greatly complicate the task at hand. some round-tripping would depend on instance data (e.g. keeping track of whether the fixed field value came from an 006 or an 008), which is not in scope here. i subtitle this paper with “a start” because my analysis so far has only covered the fixed fields and the number and code fields. these are the easier portions of the marc21 record to tackle, with the fixed fields being the lowest of the “low-hanging fruit.” what i have learned in this process is the focus of this article. the back story my investigation into the underlying data elements of the marc21 record began in 2005 with an unpublished report that i produced for the library of congress that defined some possible steps that could be taken to prepare for “life after marc.” at that time my assumption was that the next library format would be in xml. however, i stated that a necessary step in the process of transforming marc21 would be to identify all of the data elements the format contains. in essence, a distillation of marc21 into data elements would allow the same data to be carried in any future serialization. i wasn’t aware of how difficult a task this would be until i embarked on it later myself. i began my work with a database of tags and subfields that had been developed by tom delsey in his “functional analysis of marc21 bibliographic and authority formats” [2]. i was able to supplement this using the tables of fixed fields and their values from marc.pm modules [3]. if nothing else, this task made me painfully aware that we do not have a sharable machine-readable form of our own data format. the marc21 standard is represented only by an online text, something that is rather astonishing if you think about it. anyone wishing to develop applications for marc21 must create their own usable version. my own database still lacks definitions and descriptions, and those will probably need to be added by screen-scraping hundreds of screens from the lc web site. having the marc21 fixed fields, tags and subfields in a database first allowed me to do some quick statistics based on the names of fields and fixed field values. these results are in a page on the futurelib wiki [4]. i was particularly struck by the large number of different fixed field elements (many of which are probably encoded only rarely in actual instance data), and how many of them have “non-values” like “unknown.” the use of values for “unknown” and “no attempt to code” are directly related to the characteristic of fixed length fields made up of positional data elements, where all positions must be filled in to retain the positioning. it becomes therefore necessary to decide if “unknown” is truly a value or can be treated as the absence of a value. as i once argued at a meeting of the marc21 standards group, marbi, about the use of the value “no attempt to code,” it is highly unlikely that someone who is not going to bother to code a value will carefully encode their intention not to bother. some of these values may exist in instance data only because they have been used as defaults. in the extraction of fixed fields that i have done so far i have ignored the three values “other,” “no attempt to code,” and “unknown” although there may be a case to be made for including “other” in this analysis (and i may decide to add those later). i also did not include elements that have a status of “obsolete” in the marc21 documentation. any of these could be added later if there is a need for them. in addition to the fixed field values like “unknown,” almost 60% of indicator positions (206 out of 350 in my database) have the value “undefined.” these truly represent empty positions in the record format and they can be ignored. the analysis in beginning my analysis of the data elements in marc21, i found it convenient to break the task into three primary sets: control fields (00x) number and code fields (0xx) descriptive fields (xxx) the fixed fields are fixed-length strings with positional data elements that take coded data that is presumably useful for machine processing. these are primarily in the form of controlled term lists with the terms represented by 1-2 character codes, and are tagged 006, 007, and 008. the “00x” area also includes the record number (001), record source (003), and a date/time stamp (005). these data elements are for record management, and do not belong in an analysis of marc21 content. one of the difficult tasks in analyzing marc21 is to separate the record structure from its content, as the two are not distinguished in the format itself. the remaining marc21 fields (tags 010 to 888) are variable-length fields with variable-length subfields. the fields tagged from 010 to 088 contain data-like identifiers, classification numbers and other data that is not generally displayed as part of the bibliographic description. the fields tagged from 100 to 888 are the “meat” of the bibliographic record, with authors, titles, notes, subjects, etc. users are presented with much of this latter data in a textual display. the bibliographic description area contains the most voluminous fields, some with nearly thirty subfields with complex interactions and dependencies between them. all of the data elements in my study must be assigned an identifier, and i use http uris for this under the registered domain name “marc21.info”. it is also convenient if the description of the data elements contains information that would lead back to the original encoding of that data in marc21. i therefore use the marc21 tags and positions to create an identity and a link between the marc21 field origins and the derived data. more detail is given on this below. control fields (00x) the control fields include record numbers and identifiers as well as three fields made up of fixed-length data elements: 006, 007, and 008. the fixed fields that i include in my analysis are the 007 and 008, for reasons that i explain below. these look like: 007    sd fsngnn   ed 008    981123p19981996enkmun   efhi           d the data elements in these fields are defined by their positions in the fixed field, such as: 007 position 03 = speed (of the sound recording) the “f” in position 3 (all numbered positions are counted starting from “0”) in the string below is the value for the speed of the sound recording in this 007: 007 sd fsngnn ed since the element has only one byte in length for its values, the information is recorded using single-letter codes for each desired value: a – 16 rpm (discs) b – 33 1/3 rpm (discs) c – 45 rpm (discs) d – 78 rpm (discs) e – 8 rpm (discs) f – 1.4 m. per second (discs) h – 120 rpm (cylinders) i – 160 rpm (cylinders) k – 15/16 ips (tapes) l – 1 7/8 ips (tapes) m – 3 3/4 ips (tapes) o – 7 1/2 ips (tapes) p – 15 ips (tapes) r – 30 ips (tape) u – unknown tags 007 and 008 are each more than one set of data elements since the values carried by the field vary based on the format of the resource being described. the different formats of the 008 are the ones that are defined in the marc21 leader.  the leader contains a small number of record-level fields, including the primary format of the resource: books, computer files, maps, music, continuing resources, visual materials the formats covered by the 007 are different from those of the 008. they are coded in the first byte of the 007 and are: map, electronic resource, glob, tactile material, projected graphic, microform, nonprojected graphic, motion picture, kit, notated music, remote-sensing image, sound recording text, videorecording, unspecified this gives us six different 008 fields and fourteen different 007 fields to analyze. i do not include the 006 data elements in my analysis since every 006 element is a duplicate of an element in the 008. the 006 exists because the 008 is not repeatable in marc21 and therefore cannot represent different formats for a multi-format resource.  in a semantic sense, there are no new or different data elements in the 006, and repeatability of data elements is not part of the semantic analysis. the elements only need to be defined once even though they could be used more than once in a data record. in essence, i am treating the 006/008 the same way i would treat a repeatable field for a subject heading. in spite of removing the repetition of the 006 data elements, the number of data elements in the fixed fields is very large. the 007 has 118 data elements, although only 55 of those are unique; the others are repeated in two or more of the 007 formats. the 008 has 58 unique data elements.  most of these data elements take as values a controlled list of terms that are represented by alphabetic codes. there are nearly one thousand different terms between these 113 term lists. the fixed fields pose few intellectual problems, and the results of my preliminary analysis are on the futurelib wiki [5] with the resulting lists of elements [6] and an rdf-defined value vocabulary [7]. (with thanks to gordon dunsire for help with the rdf coding.) note that the tag alone is not sufficient to identify the data elements for the fixed fields because either the leader “type of record” (offset 06), in the case of the 008 field, or the first position of the 007 essentially creates a different fixed field for that type of material. i therefore add the marc21 term for the relevant format to the identifier. data elements in the fixed fields get identifiers with the tag, the format and the position or positions. data element marc21 data identifier reduction ratio tag 007 format=microform (007 pos. 0 = “h”) data element at position 05 007microform05 projection tag=008 format=map (leader 06 “type of record” = “e”) data element in offset positions 22-23 008map22-23 the actual coded values in each field consist of oneor two-character codes. identifiers for these coded values are constructed using the identifier for the tag, the format and the position or positions as described above with the addition to the string of the code that represents the value of the data elements. thus each term in the controlled lists that make up the vast majority of the fixed fields has an identifier. data element value marc21 data identifier reduction ratio low reduction tag 007 format=microform (007 pos. 0 = “h”) data element at position 05 value=low reduction (code=”a”) 007microform05a projection sinusoidal tag=008 format=map (leader 06 “type of record” = “e”) data element in offset positions 22-23 value=sinusoidal (code=”bg”) 008map22-23bg these identifiers are currently included in the uris for the fixed field data elements, but i am not wedded to that solution. perhaps these would be better coded as skos:notation elements. <skos:concept rdf:about=”http://marc21.info/vocab/007map03/007map03c“> the value vocabularies in my analysis are those elements whose values are included “in line” in the marc21 standard. there are also external lists, such as the language codes and country codes, that are maintained separately. some of these are appearing on the library of congress authorities site http://id.loc.gov, and others have been defined at http://marccodes.heroku.com/. number and code fields (0xx) while the fixed field data elements are essentially structured as a compact string of key/value pairs, marc’s variable fields have a complex structure. each field has two indicator positions that can modify the field’s semantics in a variety of ways, and the majority have multiple subfields that are subordinate to the field and sometimes to each other. in these variable length fields we encounter some of the tough questions of interpreting marc21 as data elements. the first thing to realize is that a single field may in fact contain more than one distinct data element that describes the primary resource. these data elements can be simple — that is, they can consist of a single string — or complex, requiring values from multiple subfields to complete the data element. in other cases, some subfields in the field may be qualifiers for other subfields in the field, most notably the $2 subfields that carry the code for the source of an otherwise un-credited data element. the marc21 record also carries some administrative data related to the instance record. the aforementioned 001, 003, and 005 are of this type. the 040 field (cataloging source) records the agencies that created and modified the instance record. there are identifier fields, such as the 010 (library of congress catalog number) that are record numbers for the same bibliographic data in another system. this would probably be best thought of as expressing relationships between instances of the data. i include these in my analysis even though these relationships may be handled very differently in a future implementation of bibliographic data. effect of indicators the meaning of the entire field may change on the application of indicator values. the 024 is a good example of this: 024 other standard identifier indicator 1 0 – international standard recording code 1 – universal product code 2 – international standard music number 3 – international article number 4 – serial item and contribution identifier 7 – source specified in subfield $2 8 – unspecified type of standard number or code indicators of this type expand the capabilities of the marc21 format, overcoming the inherent limitations of the structure. if you consider that the 0xx range at best allows only 89 different fields, and that all fields with a “9” in the tag are supposed to be reserved for local use, the actual number of available fields is only 70. without this ability to expand the number of data elements through the use of indicators and through qualifying subfields (like $2) the record would have been unable to incorporate new codes and identifiers. as it is, 49 tags have been assigned in this range. the 024 field is also an interesting example of an open-ended list that uses “source specified in subfield $2” to make the value of the field greatly expandable. this is the most common use of indicators in this tag range, but there are others. some fields have indicators that are data values that could stand on their own, such as “existence in lc collection,” which is semantically binary, “yes/no.” there are indicators that give information about how the data is structured in the field. in the coded cartographic mathematical data field (034) the first indicator states whether the field contains a single scale or a range of scales. this appears to be redundant with the data itself: a range is expressed with two successive $b subfields. as i am not knowledgeable about cartographic data, finalizing this field will await input from someone with that specific expertise who can clarify the meaning of this indicator. in general, though, i am not including record structure in my analysis, only semantics, since other serializations of the data elements may express structure (like ranges) differently. what is a data element? unlike the fixed fields, in working with the variable fields it is necessary to decide what makes up a data element. the nature of the tag-plus-subfields structure often masks the fact that in some cases the subfields in a field are independent data elements that have been placed in a single field because they have something in common. as an example, the field for the isbn, tag 020, has three subfields: 020 $a isbn $c terms of availability $z canceled/invalid isbn these are actually three separate simple elements: isbn, terms of availability, and canceled/invalid isbn. each of these is a statement about the primary resource that is the focus of the bibliographic record, and they are not defined as dependent on each other. their collocation in a single field does not have an effect on their semantics. (note: although the instructions in marc21 do not indicate a dependency between the isbn in the $a subfield and the terms of availability in the $c, it may not be wrong to interpret them in that way given that the field is repeatable – as are most variable fields. one could agonize at length over situations of this type, or simply make a decision and move on. i generally opt for the latter, especially because i consider the work at this stage to be informative, not prescriptive.) using the 024 again as an example, it has two types of data elements. the first are those defined by first indicator codes 0-4, 8. these have the following subfields: $a – standard number or code $c – terms of availability $d – additional codes following the standard number or code $z – canceled/invalid standard number or code (r) 024 fields with a code of “7” in the first indicator also have a subfield $2 that has a standard value for the organization or system providing the identifier; something that can also be called the provenance. based on the indicators it is pretty clear that the 024 breaks out into more than one type of identifier. each of the values coded in 0-4, 8 essentially defines the type of identifier in the field, and value of “7” designates a complex data element that carries within it the designation of the entity providing the identifier. that would give us seven separate data elements based on this marc21 field. there are additional aspects of these subfields that need to be examined, particularly the relationships among them. the subfield $a is the primary identifier string, with the subfield $d being additional characters that belong to that identifier. this indicates that $d is dependent on $a, and those two codes need to be treated as a multi-unit data element. in the case of identifiers whose  source in included in $2, the $2 is also part of this complex. so we now have two patterns: $a $d, and $a, $d, $2. then there is the $z, the canceled/invalid standard number or code. it isn’t at all clear that this has any direct relationship with the data element that begins with the $a. the marc21 documentation says: “if a valid number or code of the same type is not known, subfield $z may appear alone in field 024.” there isn’t a relationship of dependency between the $a and the $z, the $z is a separate data element. oddly, it has no subfield for “additional codes” and it’s probably best to assume that a $z $d combination is not intended. however, presumably a $z $2 combination is possible. the $c subfield for terms of availability has an odd relationship to the primary data element of the field; the marc21 documentation says: “information is only recorded in this subfield when a number is present in subfield $a.” that’s a constraint on input, which is usually to be managed by an application, but it’s less clear how this affects the meaning of the data element represented by the $c. at this point, without having clear use cases to help us understand how this data might be employed in actual applications, it is hard to make a clear decision about the data element, but since marc21 treats it as dependent, i decided to keep it together with the $a. the 024 also has a second indicator called “difference indicator” that states whether the code is available both in eye-readable and machine-readable form. this indicator qualifies the $a $d and therefore belongs with those.  i have coded this by adding “/02” to the tag in the chart below. out of this one marc21 field we therefore derive fourteen separate data elements. i’ve used shortcuts here in the display but the names of the data elements and their parts are spelled out in the pdf version of the 0xx analysis [8]. (i have not yet created an rdf-encoded version of this range. presumably it would make use of owl, and i would welcome collaboration on that aspect of the task since owl is notably complex and has many options.) element marc21 field/subfields isrc 024$a 024$c 024$d 024/02 upc 024$a 024$c 024$d 024/02 ismn 024$a 024$c 024$d 024/02 ean 024$a 024$c 024$d 024/02 sici 024$a 024$c 024$d 024/02 other number 024$a 024$c 024$d 024$2 024/02 unknown number 024$a 024$c 024$d 024/02 isrc cancelled 024$z upc cancelled 024$z ismn cancelled 024$z ean cancelled 024$z sici cancelled 024$z unknown number cancelled 024$z other number cancelled 024$z 024$2 because this is mainly an exercise in identifying the semantics of marc21 it is probably a good idea to hold in check our impulses to normalize the data elements or make them more consistent. consistency is not marc21’s forte, and i am sure that there will be fields in the xxx area that make this one look reasonably well-designed. i could see a later step that attempts to “fix” some of the problems that we encounter in marc21, but i’m not sure that will ever be needed. i say that because i sincerely doubt that we will want to reproduce the entirety of marc21 in another format even though each of the individual data elements may find a future user base. in fact, it makes perfect sense to me for us to rebuild our bibliographic data from the ground up, looking back to marc21 from time to time to remind ourselves of data that we once thought necessary but perhaps no longer do. finding patterns when analyzing marc21 variable fields one has to understand the relationships between the subfields. certain patterns are revealed that can be applied repeatedly to the task. i have so far identified these three patterns: figure 1.types of relationships between marc subfields examples: a. 041 language code $a language of text/sound track or separate title $b language of summary or abstract $d language of sung or spoken text $e language of librettos $f language of table of contents $g language of accompanying material other than librettos $h language of original and/or intermediate translations of text $j language of subtitles or captions the 041 has multiple types of languages, each in a standard iso 3-character coded form. there is no dependency between them and each describes an aspect of the resource being cataloged without a need for the others. b. 100 personal name main entry $a person name $d dates associated with a name the $d holds the dates for the person whose name is in the $a, not dates for the primary resource being described. in this case, the $d is a qualifier on the $a, because the dates are used to qualify the name to make the name heading unique. c. 050 library of congress call number $a classification number $b item number the call number is in two parts: classification and item. both are describing the primary resource, but the item number cannot be used alone, therefore it is a part of the whole that is the call number, with parts classification number and item number. within a single field (and this will happen more frequently in the xxx fields) more than one of these patterns may exist. for another time: bibliographic description fields (xxx) it’s obvious from the analysis of the code and identifier fields (0xx) that the remainder of the bibliographic fields, those in the tag range 100-899, will present some significant and interesting problems.  for example, there are fields that are defined more by their function than by their semantics, as in the uniform title field, marc21 tag 240. in the marc21 documentation, the field is defined as: “used when a work has appeared under varying titles, necessitating that a particular title be chosen to represent the work.” the $a subfield of the 240 is defined as the “uniform title,” with the other subfields having the role of modifiers on that title. although identically coded, the $a subfields in the following 240 fields have totally different meanings: 240 10  $a pendolo di foucault. $l english this uniform title brings together the translations of a work. “pendolo di foucault” is the title of the work (in the frbr sense of work), in the original language. the translation being described is in english. 240 10  $a selections this is used for an item “…consisting of three or more works in various forms…” the title of the work is not “selections.”  this title, and others such as “works” or “complete works” are supplied by the cataloger. 240 10  $a concertos, $m harpsichord, string orchestra, $n bwv 1052, $r d minor this is a practice of music librarianship. it is a coded description of music that may have little or nothing to do with the actual title on the item being described. it is a structured field that contains the distinguishing information about a piece of music in a set of facets: medium of performance, standard number, key, arrangement. all of these uses of the 240 field fulfill the function of providing a single title for the work, but there is no way to give an a priori meaning, such as “title of original work” or “musical form” to subfield $a as a data element because that difference only appears in instance data. (note that in some cases there will be no way to algorithmically distinguish these different semantics even in instance data.) the bibliographic description fields also have a significant amount of redundancy, in the sense that elements like “author” and “title” appear in numerous fields. however, because of the limitations of the marc21 structure, these elements have been coded differently in different fields in order to conserve the use of subfield codes (in particular in the 77x linking fields). 100 field, personal name subfields $a – personal name (nr) $b – numeration (nr) $c – titles and words associated with a name (r) $d – dates associated with a name (nr) $e – relator term (r) $q – fuller form of name (nr) $u – affiliation (nr) 773 field, personal name subfields $a – main entry heading (nr) examples: 100 1_ $a hamilton, milton w. $q (milton wheaton), $d 1901773 0_ $a hamilton, milton w. (milton wheaton), 1901$t sir william johnson and the indians of new york. $d [albany] : university of the state of new york, state education dept., office of state history, 1967 can these be considered semantically equivalent even though they may be composed of different parts? it remains to be seen how effectively the bibliographic description fields can be analyzed into their component semantics. it may be necessary to create elements that are as ambiguous as their marc21 origins, and to use this to illustrate areas of library data that need to be more precisely defined to facilitate use and re-use. comparison to rda there is a comparison [9] between marc21 and rda that was part of the rda development process. this document was produced in 2007 and was not updated as rda evolved beyond that point. however, it is clear from that document that the list of rda elements that was issued as rda element analysis [10] is not at the same level of detail as the marc21 standard, and also that many data elements, in particular those in the marc21 fixed fields and coded data fields, are not addressed in rda. an obvious next step is to attempt the same comparison between the marc21 semantic analysis and rda.  i have no doubt that this will have some similar results to the study produced by the jsc, but cannot speculate at this time on what we should conclude about those differences. conclusion as i said in my introduction, undertaking a study of this depth as a solo project may not have been wise. however, my frustration as a library professional at not having a good grasp on the scope and nature of our own metadata has led me to throw caution to the wind. there is a certain amount of magical thinking on my part: if we could just create a definitive ontology for library data then we could make a rational transformation from marc21 to a more modern metadata format. this doesn’t mean i expect that we would use this analysis to create a map or cross-walk from marc21 to marc++. in fact, armed with this knowledge we might decide to greatly trim or radically transform the data elements that we carry with us into the future. i will continue to make available my results in machine-usable formats so that others can do their own analyses or make use of what i have done. i welcome comments and criticisms, and anything else that this work might inspire. notes [1] thomale, j. (2010). interpreting marc: where’s the bibliographic data? code4lib journal, 11. http://journal.code4lib.org/articles/3832 [2] delsey, t. (2003) functional analysis of the marc 21 bibliographic and holdings formats. http://www.loc.gov/marc/marc-functional-analysis/original_frbr.html [3] http://marcpm.sourceforge.net [4] http://futurelib.pbworks.com/w/page/13686649/data-and-studies [5] http://futurelib.pbworks.com/w/page/35201344/fixed_fields [6] http://kcoyle.net/rda/elementslist.txt [7] http://futurelib.pbworks.com/w/page/36289829/resulting-data [8] http://kcoyle.net/rda/0xx.pdf [9] kiorgaard, d. (2006). rda and marc21. http://www.loc.gov/marc/marbi/2007/5chair12.pdf [10] danskin, a, (2009). rda element analysis. http://www.rda-jsc.org/docs/5rda-elementanalysisrev3.pdf about the author karen coyle is a librarian and consultant with particular interest in metadata. she began encouraging the library community to work on the next generation of library metadata at least ten years ago. as this was not successful, it has become a diy project. subscribe to comments: for this article | for all articles 9 responses to "marc21 as data: a start" please leave a response below: code4lib journal – n° 14 « pintiniblog, 2011-07-27 […] marc21 as data: a start […] formato xml pode substituir o formato marc? | processo técnico bice/ucs, 2011-07-29 […] para acessar o artigo, em inglês, clique aqui […] matthew beacom, 2011-08-04 karen, great article on your marc modernization effort. at the end you say, “… armed with this knowledge we might decide to greatly trim or radically transform the data elements that we carry with us into the future.” we may begin to trim the needed data elements immediately and not wait until we are fully-armed with this knowledge. we can create a simple bibliographic model that can begin to capture or propose those elements (with definitions) that would allow us to create services for users. among them may be identifiers of various sorts, titles, publication as an event, etc. that said, you have given us all a great start. thank you. scott lewis, 2011-08-04 though i have not worked extensively with marc, i’ve done enough with it to notice many of the complexities described in this piece. overall, it is my impression that attempts to describe these data structures in terms of databases and database-type concepts(such as codd’s normalization) may not be the most fruitful path to follow. my impression of marc is that what it maps to most closely would be the way in which we construct and make use of objects in the more pure o-o programming languages, such as smalltalk and self. one way of looking at that shift is that data/database type constructs tend to concentrate on the \nouns\ of the given information. the formation of marc itself is attempting to supply both the \nouns\ and something like a \verb\ form (or at least a gerund) as well. this noun/verb form (data and the integral means of accessing it) is very much what objects offer. i think there may be two consequences to following this path. firstly, if the goal of a user/translator of marc is to do something helpful with it within a program, it might be best to consider moving from a standard rdbms to an object-based rdbms. secondly, if the goal is to serialize the marc data in a different form, such as xml (perhaps in owl), it might be best to first transform the data into an object form, then work out how to serialize that object. using objects as an intermediary in this way may seem to be either a distraction or an additional task. however, the goal of o-o has never been to, say, make computers work faster. it has always been about finding better ways for humans to conceive of complex systems. marc might be a good place to make use of these tools. cheers, scott “marc21??????”??? » ????iii, 2011-08-06 […] 2011?8?6? ? catwizard ?? » marc21 as a data: a start / by karen coyle. code4lib journal, issue 14, 2011-07-25. issn 1940-5758 […] jakob, 2011-08-10 @scott: i would not call it object-oriented and i strongly doubt that using object-based rdbms makes any difference, because this is too much on a technical level. the representation in a (possibly oo-)rdbms is just another (internal) serialization, comparable to xml, or rdf. instead we need a better understanding of the \data elements\ as caren calls it: the task is to determine the basic conceptual entities (you can also name it aspects) of a record and their dependencies. codd’s or other datbase theory maybe helpful in doing so, but i find caren’s diagrams helpful enough. @caren: i found that sometimes order of repeatable elements is relevant and sometimes it is not. do you have more on repeatable fields? eric nord, 2012-03-19 while new to marc i’ve already become accutely aware of the need for a new format. one of the great difficulties with marc21 is identification of records is aproximate not exact. i belive this is due in part to many new formats and item types being forced into a standard that was never built to handle them (describing cd, dvd, blu-ray in the 007 for example) i wonder how the well a modular approach would work where each “item type” had it’s own standard identifiers, rather than trying to cram everything into one rule set (which leads to abstract identification). this would compartmentalize the “standardization” to a specific item type and perhaps allow for more direct and clear identification by smes of that specific item type. this may also lead to simplificaiton where libraries would only need to understand the item types they deal with directly. in the end we’ve all used marc21 differently to best describe our varied holdings – due to marc21s “approximate” nature i really don’t see a universal application that will cover all cases. we’ll all have to dig ourselves out of the corner we’ve painted ourselves into with marc21 and the more we stuff in the worse it gets. re: browse and search bnb open data | first thus, 2014-06-26 […] posting to rda-l concerning karen coyle’s article at http://journal.code4lib.org/articles/5468 […] understanding marc | library metadata, 2014-09-28 […] web sites and other resources available with details about marc. but, karen coyle has written a summary which is a good introduction for people looking for a quick […] leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – tools for reducing and managing link rot in libguides mission editorial committee process and structure code4lib issue 17, 2012-06-01 tools for reducing and managing link rot in libguides while creating content in libguides in quite easy, link maintenance is troublesome, and the built-in link checker offers only a partial solution. the authors describe a method of using purls and a third-party link checker to effectively manage links within libguides. by wilhelmina randtke and matthew d. burrell introduction. this article describes tools for managing links in springshare’s libguides, a popular web platform for libraries. libguides includes a built-in link checker for only some links. instructions are provided on how to run an automated link checker on all links within a single guide. persistent uniform resource locators (purls), hyperlinks which are maintained centrally and rarely break, are recommended for library electronic resources. special consideration is given to the fact that many libraries using the libguides platform may not have easy access to in-house information technology (it) personnel. a method is presented for implementing purls entirely within the libguides platform. the libguides platform. libguides is a popular platform which allows librarians to create simple web pages using standard templates.  the libguides community directory at http://libguides.com/community indicates 2,823 libraries have created 218,355 research guides published in libguides as of january 2012.  libguides is marketed as an online replacement for paper pathfinders, or lists of research sources for a particular topic.[1] in the summer of 2009, florida state university (fsu) libraries purchased libguides.  since its implementation, librarians at fsu have used libguides extensively and now feature 170 guides available on the libraries’ website. libguides allows easy content creation. fsu’s overall experience with libguides has been positive.  librarians appreciate having the ability to manage their own content and be able to simply modify their pages when they see the need.  web development staff appreciate that an account can be quickly created and managed.  libguides provides a way for librarians to identify and fix problems with outdated content instantly by editing content in the guide.  this eliminates the necessary additional time which may occur on the libraries’ websites where only a small group of staff has the ability to update content. ease of use and management is significant in light of the fact that many libraries do not have an in-house information technology (it) staff or full time website staff within the library, and may need to rely non-library staff for it services.  while fsu libraries has a dedicated in-house it and web development team, fsu libraries recognize that libguides is especially valuable for libraries which are not equipped to set up and run a web server. libguides presents universal problems in content maintenance. one of the emerging problems as fsu libraries’ libguides system matures is broken links, or link rot. as online content relocates and uniform resource locators (urls) for electronic resources change, links break.  link rot is a universal problem when dealing with online content. this article seeks to describe ways in which fsu libraries manage and prevent broken links in the libraries’ libguides system, and to give step-by-step instructions on link maintenance which can be implemented by libraries either with or without a dedicated it and web team. pitfalls of relying on libguides built-in link checker web links libguides has a built-in link checker.  to get to this a librarian can log into libguides, click “my admin” in the top right corner, and click “link checker” at the bottom of the column in the center of the screen. libguides’ link checker has some disadvantages.  first, according to the small print explanation at the top of the list of broken links provided by the link checker, “it can only check links that are added within the following box types:  web links, links & lists, dates & events, rss feeds, podcasts, books, user submits, and polls.  links entered through the rich text editor are not checked.”[2] links in bulleted lists will be checked.  however, other links will not. a link inserted in the text of a paragraph cannot be checked using the libguides interface.  examination of libguides included in the directory at http://community.libguides.com demonstrates that it is common for links to be placed in paragraphs of text in the libguides system.  when links are placed in a paragraph, rather than in a bulleted list, libguides provides no link maintenance tools, and libraries need to find other avenues to identify broken links. electronic resource links. the second weakness of libguides’ built-in link checker requires a short explanation of how electronic resources links work.  in this article, an electronic resource refers to a database purchased by fsu libraries.  these databases require an institutional subscription, and require authorization recognizing that a patron of fsu libraries is accessing the resource.  most databases will recognize fsu libraries patrons based on the ip address of the computer.  if the computer is located on the fsu campus, then the ip address of the computer will show the computer’s location and allow access. if a patron accesses a database using a computer that isn’t on the campus, then the database needs to have a way of recognizing that the user is a library patron.  this is done through a proxy login.  a proxy login is a special link, which routes the patron’s computer connection through a server at fsu, and lets the database know that an fsu patron is using the database. for our purposes, a proxy server is seen as a prefix to a url that shows the ip address of the user as if they were on campus.[3] unfortunately, libguides’ built-in link checker does not check links with a proxy or any electronic resources which require off-campus login.[4] how to check web links in a libguide. running an external link checker on the libguides system is complex.  libguides uses a “robots exclusion protocol” (robots.txt)[5] to instruct search engines not to make cached copies of the guides, and tells the internet archive not to spider and store any guides, and which informs some automatic link checkers not to examine the guides.[6] this is why some popular link-checkers, such as the world wide web consortium’s link checker at http://validator.w3.org/checklink , cannot be successfully used with libguides. fsu libraries experimented with link checkers and found that xenu link sleuth will allow a librarian to run an automated link check on his or her guides.  xenu link sleuth is a powerful tool for website maintenance, which is available for download at http://home.snafu.de/tilman/xenulink.html.  in exchange for using the program, the programmer requests donations be made to a list of specified charities, or that a t-shirt, or a thank you letter be sent.[7] in order to check all outgoing links from a single guide: install and open xenu link sleuth.  go to the <options> menu and select <preferences>. set “maximum depth” to exactly 2. “maximum depth” determines how many links the link checker follows away from the page you are on.  so, 0 means just whether the url you check loads.  1 means whether links on the page you check load.  2 means whether links on the page you check load, and whether links on those pages load.  so, 2 will check the links on the tab of the guide for the url you input, and for each other tab and subtab in the guide.  putting a higher number would start to check other guides, as well as yours, which will take more time to check and will result in flagging some urls which you cannot edit because they are not on your guide.  (a higher number would be appropriate if you wish to check all libguides in the system.) under “report” make sure “broken links, ordered by links” or “broken links, ordered by page” is checked. click <ok>. under the <file> menu select <check url…>.  (f)  paste in the url of any one of the tabs in the libguide. make sure the “check external links” is checked.  this will check links that go to anywhere outside of your libguide (ie. to google scholar, a university department’s webpage, etc.) click <ok>. figure 1. in the preferences menu, maximum depth should be set to 2 in order to check links within a single guide using xenu link sleuth. xenu will check all of the links on a page including images, style sheets, scripting, as well as url’s. don’t be put off by so many links – the routine links will work, and you don’t need to worry about links that work. once the link checker has finished running, sort the list of links by “status”.  xenu link sleuth shows problematic links written in red. figure 2. once the link checker finishes, sort links by status, and right-click on links which are “not found” in order to view the libguides tabs within a guide which contain the broken links. right click on a broken link and a menu will appear.  scroll down to <url properties>.  a box will appear and at the bottom of that box will be a list of all the pages that contain the broken link.  all the pages should have urls with the same number after “pid=”.  (if they don’t, then repeat these directions from the top, and especially check that “maximum depth” was set to exactly 2.)  correction is only necessary for the pages which contain “…/content.php?….”.  pages with “…/print_content.php?….” and “…/content_mobile.php?…” in the urls should be ignored, because these are changed automatically when links from “…/content.php?…” pages change. to correct each link go to the page with a url containing “…/content.php?…” which the broken link appears on and make the change in libguides. after you have corrected the links, go back to the beginning of this section, and rerun the link checker.  links should be working now. what a link checker can’t do. a word of warning is appropriate.  a link checker only checks that when a link is clicked, a page will load.  the link checker does not check that the correct page loads.  the link checker cannot check most electronic resource links. the link checker cannot check that a linked page is what a person clicking on the link will expect.  for example, if i promise a link to google scholar, and then link to google, to a shopping site, or to any other page on the web, the link checker will find the link, load the page, and report that it found a working page. now, picture a link to an electronic resource.  usually, when proquest or other vendors change the urls for databases, the old urls result not in an http error status, but in a success status, and load a page saying to “please login” or to pay x amount to view the article or search the database.  while a person clicking that link will find it broken, a link checker will see only that a page loaded.  a page which requests someone to “please login” is still a page.  the link-checker sees a working link no link checker can monitor whether linked electronic resources recognize the user as someone who has or has not  legitimately paid for access to the database. and a high proportion of links on a library website will be to electronic resources.  further, some vendors explicitly request that their links not be checked.  therefore, automated link checkers have limited use for libraries. at this time, the only way to check whether an electronic resource link is working is to click on that link, and retrieve a full text resource from that database.  this can be done “on the cheap” by assigning the task of link checking to a worker with less training.  it is recommended that a student worker or paraprofessional periodically be assigned to retrieve one full-text resource from each linked resource or database on a library’s list of databases.  if a full-text resource cannot be retrieved, then the database may not recognize that the user comes from a library which has paid for access.  in this case, the webpage and link should be brought to the attention of a librarian who can determine why the connection failure has occurred. managing electronic resource links in libguides. libguide has a tool for handling electronic resource links.  as described below, this tool does not adequately provide for long-term link maintenance.  in 2010, libguides partnered with serial solutions®.  the result of the partnership is that libraries with libguides and either a serial solutions 360 account or summon service are able to automatically generate a single libguide containing a list of working links to electronic resources.[8] this libguide can be updated at any later point by running the database import tool.  however, this update and a-z list has some drawbacks.  first, if database descriptions are not included in the library’s 360 or summons plan, then in order to include database descriptions, library staff must manually add each description.  during subsequent updates, the librarian performing a database update must remember to uncheck the box for “import database descriptions” while doing the database update.  otherwise, if this box is checked, then a list of empty descriptions will be imported and will overwrite any descriptions librarians have previously created.[9] the second drawback is that a database update will not remove databases which the library no longer subscribes to.  so, if a subscription is dropped, the link must be removed manually from the a-z list.[10] the third and most significant drawback is that a list of working links in a single place in the libguides system does not mean links from any other guides in the system will work.  in libguides, a librarian has three options when adding a link.  the librarian can create a new link by pasting in a url, copy a link from elsewhere in the system, or reuse a link.  reusing a link means that when the parent link changes the new link will change as well [11] if all librarians reuse links from the a-z list and do not copy links, or cut-and-paste the url, and the administrator for the a-z list runs regular updates and diligently removes databases, which are no longer subscribed to, then links to electronic resources will be relatively stable. however, this requires that all librarians using the libguides system always “reuse” links from the a-z list.  this is administratively difficult to achieve. in many cases, directions were given months or years before the time when they add or change a link.  it is unlikely that all libguide account holders will use links to electronic resources from a central list.  it is very tempting to cut-and-paste links from elsewhere in libguides or from within a database. persistent urls:  an old idea gets new life in libguides. persistent uniform resource locators (purls) refers to a system for creating and maintaining urls which will always render the same resource even if the resource moves to a new address on the web.  this is done by maintaining a list of purls and corresponding list of current urls.  fsu libraries’ purls may be found at http://guides.lib.fsu.edu/databasesaz .  the purls in this list never change.  so the purls may be cut-and-pasted, copied, or reused, and may even be added to a syllabus or course website for use outside of the libguides system. each purl on the databases a-z libguide refers to a database hosted at fsu libraries which contains the list of purls in the a-z databases libguide and connects that purl to the correct location of the resource.  proxy logins are provided by the purl database as well, so librarians do not need to be reminded to include a proxy login in a link to an electronic resource.  while fsu libraries’ purls use a mysql database, it is possible to implement purls entirely within libguides. how to implement purls in libguides. libguides allows librarians and administrators to place a redirect on an entire guide.  so, when someone clicks to that guide, the person will automatically be taken to a destination url specified for that guide, which can be anywhere on the web or in an electronic database.  libguides also allows one to assign a short human-readable url to a guide.  by combining these two features, it is possible to use libguides to create a list of purls which is contained entirely within the libguides platform. to create a purl for a single database:  first, log into libguides and click “my admin” in the top right corner. click the link in the center of the screen to “create new guide”.  for “guide title” type the name of your database.  for “redirect url” type the current url for the database, including your library’s proxy login.  click the button to create this guide. now, edit the guide.  along the top of the screen, click the menu for “guide setting”, and then click “change guide information” in the drop down menu.  change “guide publication status” to private.  that way links to this purl guide will work for patrons, but numerous purl guides for numerous databases will not clutter the library’s libguides homepage.  under “friendly url” assign the purl for this database by giving it a short identifiable name.  now click “save”. links which point to this guide will always redirect to the correct database url.  if the url changes, you can simply go to “change guide information”, “advanced settings” and change the “redirect url”.  because the “friendly url” is something that librarians can recognize as reliable, librarians will constantly use a list of reliable links and know that each is part of the library’s purl system and works. create a purl for each library database.  as you go, create your own databases a-z guide with the database name, and a link to your purl. while purls will take a significant amount of time to set up initially, the time will be saved later by consistently linking to the purls (guides which redirect to individual databases).  so, when a database url changes, you will only have to correct the link in one single location: the redirect on your purl guide. a clear advantage to using purls is that links can be inserted in any way by librarians editing the guides.  so all three options, cutting and pasting in a new url, and reusing or copying a url from elsewhere in libguides, will result in a link that is maintained.  another advantage is that by using a consistent naming system for all assigned friendly urls, for example, by beginning all friendly urls with “purl___” or “data___” and then an abbreviation for the database, a librarian can quickly mouse over links to view the destination url, and verify that the destination is a library purl.  so, a librarian can know that the link is reliable without clicking it and retrieving a document from the electronic resource. purls can also be promoted to professors, student organizations, or tutor organizations who can print the purl in syllabi or handouts and be assured that the link will not break.  so, a reliable system of purls has potential as a marketing tool for promoting library services outside the library. other solutions to identifying broken links there are other solutions for identifying broken electronic resource links in libguides. one is to ask each librarian to manually check his or her own libguides. this takes a considerable amount of time that the librarian may not have, and because the same database may appear on different guides, effort will be duplicated as every occurrence of a link to the database is checked. further, if the librarian checks links from an office on campus, the librarian will not notice links for which a proxy login should be but was not included.   these links will appear to function from an on campus computer, but will not allow a student to retrieve resources from office campus, because the database will not recognize the user as coming from the university which licensed its databases. pros and cons of implementing purls in libguides. implementing purls for electronic resource links will likely save staff time in checking for broken links within the system, nevertheless, initially creating the purls within libguides involves a significant investment of staff time.  this investment of time is a concern, because the purls within libguides cannot be moved to another system.  if the library cancels its libguides account, then that time is lost, and incoming links in other portions of the library website, and in professors’ syllabi will break perhaps causing a lack of confidence in the library.  libguides may also change its interface in the future, leading to loss of the purls.  in short, the problem is:  implementing purls in libguides leads to buy-in. a more technologically complex solution is to implement purls in-house using a database.  fsu libraries has chosen to do this using a mysql database.  mysql is not necessary, and a comparable system can be implemented in oracle, or any other database server.  crucially, fsu libraries has technical staff within the library including multiple staff that can set up a database server and make regular updates to the databases on that server.   continued access and updates are important, because every time an electronic resource link changes, the database must be updated with the new url.  housing purls on a system controlled and owned by the library is recommended for libraries, so long as the library is able to (1) have a database set up by it staff, (2) be able to have the database updated with a fast turnaround time on a regular basis, and (3) keep the database running, up-to-date, and working consistently indefinitely into the future.  advantages to housing the purls in house are significant.  it prevents buy-in to a hosted system which is priced on a yearly basis, and may become increasingly expensive over time.  it prevents buy-in to a hosted system whose functionalities may be altered or discontinued, without the library’s input or consent.  and, finally, for staff with technology training sufficient to work with an in-house database, it is possible make regular backups of purls and then to restore them later in case of a problem with the database.   true system wide back ups are not possible in libguides.  libraries with sufficient in-house access to technology and the skills to make regular updates to a database should explore options for housing purls in-house.  an excellent source to begin exploring in-house options is tonkin’s article persistent identifiers:  considering the options, which gives an overview of different ways of hosting purls and points to more detailed resources on implementing purls using various technologies.[12] despite potential buy-in issues with implementing purls in libguides, the simplicity of the system allows non-technical staff to comfortably and intuitively interact with libguides.  this prevents bottlenecks which might occur, if technical staff were needed to make simple regular revisions to a database, and technical staff is not easily available.  this article has attempted to describe a system of implementing purls directed as these libraries with small technical staff or those with limited access to it.  the it staff and capability of the library to host a database should determine the choice of what tool to use to implement purls. conclusions. the libguides platform is a useful tool for creating librarian-managed subject or class webpages.  as this platform ages the problem of link rot which is universal to online resources is a threat to the accuracy of published library content, as well as the efficacy of a well-maintained library website.  the libguides platform provides minimal tools for checking and managing links to electronic resources.  by using widely available tools any librarian can check web links within his or her own guide.  it is also possible to use libguides to create and maintain a system of purls, or links which refer to a single central location and so will not break as long as that central location is kept current.  it is recommended that libraries consider using purls to manage electronic resource links. references. burke, a.  2011a.  a-z subscription database list management [internet].  [updated 2011 jul 01].  springshare.  accessed at http://help.springshare.com/azlist burke, a.  2011b.  box types [internet].  [updated 2011 jul 01].  springshare.  accessed at http://help.springshare.com/boxtypes hausherr, t.  2011.  find broken links on your site with xenu’s link sleuth [internet].  [cited 2011 jul 07].  available from http://home.snafu.de/tilman/xenulink.html libguides for libraries [internet].  [updated 2011].  springshare; [cited 2011 jul 07].  available from http://springshare.com/libguides/ link checker – subject guides at the florida state university [internet].  [updated 2011 jul 07].  springshare; [cited 2011 jul 07].  on file with the author. serials solutions knowledgeworks powers springshare libguides a-to-z database list [datasheet].  [updated 2010].  serialssolutions: a proquest company.  accessed at http://www.serialssolutions.com/assets/resources/data-sheet-knowledgeworks-springshare-libguides.pdf notes [1] libguides for libraries [internet].  [updated 2012].  springshare; [cited 2012 jan 18].  available from http://springshare.com/libguides/ [2] http://guidefaq.com/a.php?qid=4399 [3] for more information about proxy servers see the oclc article at http://www.oclc.org/us/en/services/brochures/213330usf_ezproxy.pdf. [4] email communication: “we filter out (i.e., don’t check) urls that have multiple “http” strings in them, which means that links using a proxy prefix are not checked.” sent jan 18, 2012 from anna burke to matt burrell [5] see “view source on any libguide page: ‘robots’ content=’noarchive’ [6] http://noarchive.net/refs/search/#google [7] http://home.snafu.de/tilman/xenulink.html#donation [8] http://help.springshare.com/azlist [9] burke, a.  2011a.  a-z subscription database list management [internet].  [updated 2011 jul 01].  springshare.  accessed at http://help.springshare.com/azlist [10] ibid. [11] ibid. [12] tonkin, e.  2008.  persistent identifiers:  considering the options [internet].  [cited jan 30, 2012]  ariadne; 56.  available at http://www.ariadne.ac.uk/issue56/tonkin/ . about the authors wilhelmina randtke florida state university libraries graduate assistant in web development strozier library, room 215e 116 honors way tallahassee, fl 32306 (850) 345-6123 wvr05@fsu.edu matthew d. burrell florida state university libraries web developer strozier library, room 209a lib 116 honors way tallahassee, fl 32306 (850) 645-6986 mdburrell@fsu.edu subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – library/mobile: tips on designing and developing mobile web sites mission editorial committee process and structure code4lib issue 8, 2009-11-23 library/mobile: tips on designing and developing mobile web sites mobile applications can support learning by making library resources more ubiquitous, by bringing new users to the library through increased accessibility to the resources libraries offer, and by creating a new way to enhance connections between patrons and libraries. this increased use of mobile phones provides an untapped resource for delivering library resources to patrons. the mobile web is the next step for libraries in providing universal access to resources and information. this article will share oregon state university (osu) libraries’ experience creating a mobile web presence and will provide key design and development strategies for building mobile web sites. by kim griggs, laurie m. bridges, hannah gascho rempel mobile explosion over the past several years at library conferences and in library blogs there has been considerable chatter about mobile technologies; more specifically about mobile phones. while there has been a lot of talk, little work has been done in libraries to make library web sites easily accessible to the growing number of users who access online via their phones. according to joan lippincott, “as with most technology developments, this one is fast-moving. this is not a time to sit on the sidelines as other campus units are developing services for mobile users…” (2008). a pew studies memo from 2008 notes, “58% of adult americans have used a cell phone or personal digital assistant to do at least one of ten mobile non-voice data activities, such as texting, emailing, taking a picture, looking for maps or directions, or recording a video” (horrigan). a 2009 survey by the pew research center’s internet & american life project found “32% of all americans have gotten online with a mobile device” and 25% of mobile users who are 18-29 years old are accessing the internet on a typical day (horrigan, 2009a, p. 3). in addition, the july 2008 edition of library technology reports cited that young adults do the following with mobile phones and wireless devices: 74% send or receive text messages, 44% send or receive pictures, 24% send or receive emails, and 23% use instant messaging (kroski). the pew research center’s project found that 46% of cell phone and wireless laptop users think mobile access is very important for getting online information on the go (horrigan, 2009b). [updated, see correction note]. “mobile access facilitates full participation in the flow of online information. for many americans, always-on, always-available access is a basic part of their lifestyles. they don’t see home broadband access alone as sufficient for their digital needs,” says john b. horrigan, associate director of the pew internet project (2009c). the mobile experience sites designed specifically for mobile devices increase usability. users performing typical tasks with their phones were successful 64% of the time on mobile sites compared to 53% success rates for desktop users on standard websites (nielsen, 2009).  testers also gave higher subjective satisfaction ratings to mobile sites (nielsen, 2009). nielsen, a leader in human-computer interaction (hci) advises that, “if mobile use is important to your internet strategy, it’s smart to build a dedicated mobile site.” similarly, the world wide web consortium (w3c) technical document, mobile web best practices 1.0 (mwbp), recommends that the mobile experience merits its own design. “mobile users typically have different interests to users of fixed or desktop devices. they are likely to have more immediate and goal-directed intentions than desktop web users. their intentions are often to find out specific pieces of information that are relevant to their context.” (2008). a use-case scenario for mobile use within the library context is that of a patron who is in the stacks and would like to locate a book listed in the bibliography of a book they have in hand. a rich mobile experience leverages the context of mobile visitors. there are three important considerations when developing a mobile application: what is the context of the mobile visitor? what are the goals of the mobile visitor? what tasks are they likely and unlikely to do on a mobile device? even though mobile devices provide access to various applications, they have limitations such as small screen size, limited screen resolution, and awkward input mechanisms. however, some services and information are more suitable for mobile sites. for example, users are less likely to read lengthy documents or browse content on their mobile device, while location-based services primarily have a mobile appeal. as a result of the differences between desktop and mobile uses, mobile developers need to move beyond shrinking content to fit small screens, and instead use the mobile experience as a new way to connect with patrons. the library technology report summarizes the functionality of several major university libraries’ mobile web sites (kroski, 2008). according to the report, the university of richmond library offers a catalog search, real-time laptop and pc availability information, and ask-a-librarian services by email, sms, or im. university of virginia libraries’ mobile web site provides news and events, information about library exhibitions, directions, library hours, and a text-only version of their entire web site. boston university center medical library made mobile versions of their subject guides, and made their e-books, e-journals, databases, and library site searchable. new york university libraries, with the arch mobile portal, allows searching of their electronic resources by title, subject, or format, and includes basic library information. ball state university libraries’ mobile site offers library patrons a catalog and journal search, videos about the libraries, information about collections, services such as interlibrary loan and course reserves, and quick links to mobile reference web sites (west, hafner, and faust, 2006). both innovatice interfaces, inc. (iii) and online computer library center (oclc) have implemented a mobile search application. airpac is a iii module specially designed for wireless mobile devices. users can browse the library catalog, check due dates, request materials, and view their patron records using their mobile devices. examples of libraries that use the airpac mobile search include nashville public library, westerville public, cuyahoga public, multnomah county library, wayne state university, and hennepin county library. airpac also has an iphone application. oclc is currently offering a mobile search application pilot, worldcat mobile, (http://www.worldcat.org/mobile/) and an iphone search app (http://www.oclc.org/news/announcements/announcement353.htm). [updated, see correction note.] the oclc web site states, “the worldcat mobile pilot allows users to search for and find books and other materials available in their local libraries through a web application they can access from a pda or a smartphone.” osu libraries have also recently developed a mobile site (http://mobile.library.oregonstate.edu/), which includes basic information such as hours, ask-a-librarian, call numbers, floor maps, and faq, as well as a full catalog search, and computer availability map. patrons can quickly search the libraries’ catalog and get one click access to availability, call numbers and floor maps. the custom mobile catalog offers searching by keyword, title, subject, isbn and course reserves by instructor or subject. the search can be filtered by location and results can be limited to 5, 10 or the top result. the item records include title, author and description, and a link to the shelf-view of the item. the computer availability map provides real-time information about the number and types of computers available in the libraries’ learning commons. to take advantage of students’ high use of text messaging, patrons can email or text themselves the libraries’ contact information, directions, or call-numbers and floor locations as well as a list of course reserves. in addition, the mobile site makes use of a mobile device’s native capabilities, such as google maps, to provide location based directions to the library and phone numbers that can be auto-dialed with one click. to provide a good mobile experience, offer services that support time critical and location sensitive activities, such as real time information about computer availability and map-based directions; and time saving applications, such as displaying the call number next to a link to the floor map in a catalog search; and use mobile device’s native capabilities, like auto-dialing phone numbers and texting a course reserve title. content adaptation techniques developing mobile web sites that provide an acceptable mobile experience for a wide range of devices can be difficult. in the same way that we work tirelessly to make full web sites work on all browsers and operating systems, we should do the same for mobile sites. one common trend is to develop web applications that are optimized for smart phones – which reads “fat chance it’s going to work on your standard web-enabled cell phone.” while this an attractive solution due to these phones’ increased screen size, browser capabilities and standardization, this is leaving a large portion of mobile users out in the cold. avianresearch, l.l.c, recently released a list of the 10 most popular handsets in the united states (beckman, 2009). the blackberry curve and iphone are at the top of the list, but more importantly, more than half of the top 10 devices are not smart phones. the implication of this list is that there are still quite a large number of people using small-screened devices for web browsing. therefore, if a library’s initiative is to provide a mobile version of their site they should develop applications that work on both smart phones and web-enabled phones equally. however, if the initiative is to build innovative applications that use functionalities that are included on a family of devices, then developing an application that exploits the device’s capabilities to enhance a user’s experience is the better solution. as stated in the mobile web best practices (mwbp) report, “it is likely that application designers and service providers will wish to provide the best possible experience in the context in which their service has the most appeal. however, while services may be most appropriately experienced in one context or another, it is considered best practice to provide as reasonable an experience as is possible given device limitations and not to exclude access from any particular class of device, except where this is necessary because of device limitations” (2008). adaptation is the practice of delivering content per each user device’s capabilities. the mwbp working group has defined the purpose of the default delivery context (ddc), explaining “this allows providers to create appropriate experiences in the absence of adaptation and provides a baseline experience where adaptation is used” (2008). the ddc is defined as follows: usable screen width 120 pixels, minimum. markup language support xhtml basic 1.1 [xhtml-basic] delivered with content type application/xhtml+xml. character encoding utf-8 [utf-8]. image format support jpeg. gif 89a. maximum total page weight 20 kilobytes. colors 256 colors, minimum. style sheet support css level 1 [css]. in addition, css level 2 [css2] @media rule together with the handheld and all media types (see css 2 media types). http http/1.0 [http1.0] or more recent [http1.1]. script no support for client side scripting. there are three adaptation techniques that can enable your mobile site to offer an optimized experience for a wider range of devices based on the device’s capabilities. each technique has its pros and cons and range from simple to advanced. 1. detect the device and link an appropriate css file. media types can be used to determine the device type and link to an appropriate css file. media types have been around since the early web and are used to render content differently for different screens. the most common use of media types is the print type, which is used to display different css styles for printing. the media types applicable for mobile devices are the handheld and screen types. w3c specifications for optimizing mobile sites recommend using the handheld media type to link to mobile stylesheets. the problem with this is that not all mobile browsers adhere to this recommendation. handheld media are supported by opera, openwave, and partially by pocket internet explorer. palm’s blazer and iemobile support both screen and handheld. apple and nokia do not support the handheld media type, rather they use media queries to render the screen type. even though it sounds impossible, media types can still be used to link to css styles that target mobile devices. the following method works correctly on a large number of mobile devices and requires only two stylesheets, the reference design and the adapted smart phone design. some mobile browsers do not support media declaration, so the first step is to link to the reference design without a media type. <link rel="stylesheet" type="text/css" href="mobile.css" /> next, to catch devices that can render the handheld media type, link to the reference design with the handheld media type. <link href="mobile.css" rel="stylesheet" type="text/css" media="handheld" /> to specifically target smart phones like iphones, android phones and others with browsers that can support media queries, link to the adapted stylesheet with the max-device width and only screen media type. <link rel=”stylesheet” href=”iphone.css” media=”only screen and (max-device width:480px)” /> media types are an easy way to give users on both high and low-end mobile devices an optimal experience by linking to device appropriate stylesheets. you can take this idea a step further by detecting the device being used and redirecting to different version of a mobile site depending on device capabilities. 2. detect the device and redirect to multiple versions of pages a detection algorithm can be used to detect whether a user is on a mobile device or not and then redirect users to specific versions of pages. using a server-based scripting language (like php), it is easy to detect device types by reading the useragent string. most device switching algorithms detect if a user is on a mobile device. the biggest problem with this approach is that useragent strings on mobile devices are inconsistent and maintaining a list of possible browsers and operating systems is a constantly moving target. an alternative approach is to turn the problem upside down and detect if a user is not on a mobile device. this method is called the “not-detection” device method. the not-detection algorithm works due to the smaller number of desktop operating systems (os), in addition desktop browsers are better at knowing which os they are running. in december 2008, osu libraries developed a mobile initiative, which required that we not make the user think when it came to finding the mobile site. we aliased our mobile site to the three common mobile urls: 1. ) m.your-domain, 2.) mobile.your-domain and 3.) your-domain/mobile. this allowed users of any type to access and bookmark the mobile site. the initiative also mandated we help mobile users who access the desktop site to find the mobile site. when a user on a mobile device goes to our main web site, he or she will be redirected to our mobile site and given the option to opt-out and be returned to the full site. to achieve these goals, we used the not-device detection method on the home page of our full web site to detect if a mobile user is accessing the site, and if so to redirect them to our mobile site. the algorithm works as follows: it first checks if the noredirect cookie exists, which indicates the user has opted out of being redirected to the mobile site. it then checks for one of the known desktop os: windows, macintosh/mac osx, linux, freebsd, solaris, and vms, and that the user agent does not contain one of the following strings: bot, slurp, spider, or crawl. there are two “gotchas” in this group that need a little more detection. the iphones and mobile windows devices’ os’s will initially identify themselves as not mobile devices. further processing the useragent for the strings, ‘iphone’ and ‘ipod’ will catch the iphone family and checking for ‘mobile’ will catch all mobile window devices. at this point, you probably have a mobile device and can redirect them to a mobile site. <?php if(($_cookie&#91;"noredirect"&#93; != true)){ $mobile_browser = 1; $currua = strtolower($_server&#91;'http_user_agent'&#93;); $agents = array( 'windows' ,'macintosh','mac','linux','freebsd','solaris','bot', 'slurp', 'spider', 'crawl', 'lynx', 'vms','openvms'); foreach($agents as $ua){ if (strpos($currua, $ua)>=0) { $mobile_browser = 0; break; } } if ((strpos($currua, 'iphone')>0)||(strpos($currua,'ipod')>0)||(strpos($currua, 'mobile')>0)) { $mobile_browser = 1; } if($mobile_browser == 1){ header( 'location: http://mobile.library.oregonstate.edu/about.html'); } } ?> algorithm based on a php script written by andy moore [1] and another by mobify [2]. this version of the algorithm is released under the terms of the gnu library general public license. detecting if the user is on a mobile device is only one part of the algorithm. the second part is detecting what class of mobile device the user has and providing an optimal experience for those users. some services can do a better job of detecting devices and can give more granular details about their capabilities. using one of the following tools in conjunction with a detection algorithm can enable data manipulation and other optimizations. developed by dotmobi, deviceatlas is a mobile device database [3]. deviceatlas provides up-to-the-minute device detection data, such as screen dimensions, javascript capabilities, touch screens, or full keyboard access. a similar, open-source tool is the wireless universal resource file [4]. wurfl is an xml configuration file that contains information about capabilities and features of many mobile devices. the information is kept up to date by a world-wide network of contributors. device detection often results in multiple copies of content formatted specifically for a range of devices, and therefore requires constant maintenance as devices come and go. and it’s worth noting; device detection is rarely 100% reliable. a feedback link on your mobile site can help patrons contact you if they experience a false redirection. osu libraries’ mobile developer has received feedback from patrons that prompted the addition of openvms and lynx to the list of desktop browsers. rather than maintaining multiple copies of content, it is better to have content that can dynamically adapt for the type of device accessing it. 3. auto-generate device-specific versions of the same content there are commercial and open-source tools that can auto-generate device-specific versions of your site. instant mobilizer from dotmobi is an automated tool that takes any existing content and transforms it into a mobile site [5]. mymobileweb is a low-cost open-source tool that provides a content adaptation suite of tools for the development of mobile web sites [6]. hawhaw is a toolkit that can “help make websites accessible by a wide range of mobile devices and standard browsers.” hawhaw is free software; released under the terms of the gnu library general public license that uses a simple api to create the markup for a wide range of mobile devices [7]. similarly, the wireless abstraction library (wall) is a free, open source java tag-library that provides a universal mark-up for wireless devices. wall will generate optimized content for a variety of devices [8]. with the rise of smart phones and the movement toward open os devices, content adaptation will become obsolete. however, until we reach saturation, developers should continue designing functional, attractive mobile applications for all devices. designing for the small screen as described in the mobile web best practices report (mwbp), “it is particularly important in the mobile context to help the user create a mental image of the site. this can be assisted by adopting a consistent style and can be considerably diminished by an uneven style” (2008). regardless of the device type your application is targeting, a consistent, well-designed layout can serve all of your users well. before the osu libraries’ mobile team started development of the osu libraries mobile site, we drew up a list of 10 design recommendations based on mwbp and the iphone human interface guidelines [9]. hierarchy a. use a “drill down” method and headers to break information up into tasks. b. put the most used content at the top and less frequented content below. c. don’t make the user click more than 3 times to get to the content. links a. assign and display each link in the navigation to a number on phone pad. do not exceed that limit. b. assign links that point out to non-mobile sites a descriptive icon. c. make phone numbers into links that call automatically. navigation a. minimize scrolling on navigation pages. b. assign each navigation link an icon.use icons that describe the content and provide a preview of the content. c. put most-used navigation sections “above the fold.” d. include basic navigation (home and back) at the top of every page. keep this navigation on one line. e. include secondary navigation (related links) at the bottom of the page. footer information a. include link to full website. b. include link to an about and feedback page. page titles, navigation links, and urls a. keep page title and navigation links to 15 characters. b. keep urls short and easy to type. c. minimize typing of non-alphanumeric characters (-, _, &). page content a. “frontload” content of interest. b. ensure subject matter of the page is near the top. c. balance pagination and scrolling. page layout a. do not use frames or tables. b. do not use pixel measures or absolute units. c. do not rely on support of font-related styling. d. do not rely on stylesheets for content organization. forms a. use drop downs, radio buttons or checkboxes rather than text fields to minimize typing. b. keep the number of key strokes to a minimum. c. provide pre-selected default values where possible. d. label all form controls appropriately and explicitly associate labels with form controls. images and color a. design a logo that can work on multiple screen sizes. b. use a pared down version of your color scheme. c. keep images to a minimum and compress to keep small. d. do not use graphics for spacing. e. content images should be no larger than 80% of the device’s screen width. screen sizes a. create 2 designs, one for web-enabled phones and one for smart phones. one design should target 176x220 and gracefully degrade to fit on 128x160. the other design should target 320x480. figure 1: osu libraries mobile site. 320x480 design. figure 2: osu libraries mobile site. 176x220 design. we started simple and created a default layout that would work on all mobile devices. this layout serves as the reference design. the css mobile profile 2.0 is a valuable resource for creating the stylesheets [10]. the reference design meets w3c’s default delivery context requirements. next we made simple changes to the reference design that adapted the content to be more visually attractive and work better on smart phones with larger screens. the adapted design makes use of the ‘full web’ browser and the css3 properties to emulate the iphone user interface. we used the two standard iphone design patterns, rounded rectangle and the edge to edge design, which follow the iphone human interface best practice recommendation and make full use of supported css3 properties that safari and webkit implement [11]. following iphone design patterns is one way to optimize content for smart phones; the viewport is another important technique. the iphone and other phones running webkit browsers use a metadata tag, the viewport, to scale a site down to fit their smaller screens. when an iphone loads a webpage, it sets the initial content area to fit the width of the iphone screen by setting the viewport’s width property to 980 pixels, the width of most webpages. you can change these widths and other viewport properties using the viewport meta tag in the head of your desktop or mobile web site. for an iphone-specific web application, you should set the width to the width of the device: <meta name = "viewport" content = "width = device-width"> even though devices like the iphone can do a good job of rendering most sites, developers should be aware of their limitations and design accordingly. maximum size for decoded gif, png, and tiff images is 2 megapixels. individual resource files must be less than 10 mb. javascript execution time is limited to 10 seconds. avoid using flash and java. iphone blocks pop-up windows by default (user preference). testing and validating for standard web sites, coding a site that will work in all browsers is as easy as writing xhtml. mobile sites on the other hand must work for thousands of devices and hundreds of browsers. this daunting challenge makes dealing with internet explorer seem like child’s play. the group working toward establishing standards and best practices is w3c’s mobile web best practices 1.0 (mwbp) working group. but there has been little movement from cell phone providers and browsers to agree to and adopt these standards. the mobile web best practices 1.0 working group aims to improve the experience of mobile users who experience barriers due to the device used to access the internet. “people with disabilities using computers have similar interaction limitations as people without disabilities who are using mobile devices. both experience similar barriers when interacting with web sites” (web accessibility initiative, 2009). an alternative to mwbp guidelines is the global authoring practices for the mobile web [12]. just as w3c offers a validation certificate for americans with disabilities act (ada) compliant code, it also rewards developers of compliant mobile sites. the mobileok basic tests are based on mobile web best practices’ 60 guidelines. mobile sites that claim mobileok conformance meet the default delivery context. in addition to advertising conformance with the mobileok icon, content providers can enhance discoverability of their mobile sites by using the protocol for web description resources [powder] to claim mobileok conformance. mobile search engines, like google mobile search, rank mobile sites higher in the results and checks powder as an indicator. to claim conformance, mobile sites must pass the mobileok tests. the mobileok checker is an online tool that automates the tests and provides detailed information about where and how a site fails [13]. the checker tests for the “mobile-friendliness” of a mobile site. the tests consist of several atomic subtests. a subtest can pass, fail, or generate a warning. a warning indicates an area that may need work, while failing indicates that a guideline is being violated. when performing the tests, you can select to group failures by category or tests. grouping by category is useful when beginning the testing process, while grouping tests is useful when working to get a particular area to validate. a page on your mobile site is scored on a scale of 0-100 for the “ok-ness.” this rating system lets you work towards meeting the guidelines and to prioritize the severity of the failures. in addition to testing your own site, the mobileok checker provides w3c with data from all the sites put through the checker. the number one point of failure is code compliance to xhtml basic. in addition to the mobileok checker tool, dotmobi provides a tool, mobiready, that “evaluates the mobile-readiness” of a mobile site [14]. the tool is based on w3c best practices guidelines and others. unlike the w3c tool, you can test your complete site at once. pages are rated on a scale of 1-4 and detailed analysis of the site is provided in an easily digestible format. the report includes page load speed and weight, validation errors and warnings, and can tidy up the offending code. the tool also includes a handful of emulators and browsers for testing. validating your code should be done continuously as you develop your site. the level of conformance you choose to require should be part of your development plan. osu libraries’ mobile initiative requires that the reference version of the site pass the mobileok checker. adapted versions of the site have a sliding scale level of conformance depending on the guideline and if the violation enhances a user’s experience. this practice is supported in the mwbp report, “it is recognized that content providers may need to violate specific best practices in order to support their intentions on devices that exhibit deficiencies in implementation” (2008). mobile application testing is often an issue in mobile application development. developing a test plan is an important step to track and evaluate the readiness of your mobile site. prepare to test by writing a script of tasks to follow, along with the desired outcomes. perform the test with a design reference to compare to and a form to capture results. track the devices you test on and cross-reference with known limitations. knowing a device’s limitation will allow you to throw out faulty tests, as well as decide if you should work to support the device. for example, we were able to rule out a handful of phones due to their inability to render xhtml. the thousands of mobile devices make testing on every one out of reach for most developers. attacking testing with a three-pronged approach can provide adequate testing coverage. the first line of defense for testing mobile sites are desktop browsers. test your mobile application on the desktop to ensure functionality. mozilla’s firefox has add-ons that you can use to manipulate the useragent, such as the user agent switcher, xhtml mobile profile and modify headers. opera mini and mobile was shipped on 7.8 million phones last year, providing reason enough to test on opera. and iphone safari is similar enough to safari for first stage testing. the next approach is testing on browser simulators and device emulators. there are opera mini simulators [15] [16], iphone safari simulators [17] and downloadable iphone emulators through the iphone sdk [18] and iphoney [19]. nokia developer toolkit includes an emulator [20], as does the android sdk [21], openwave tool kit [22], blackberry developer tools [23], mobile windows [24], and palm developer sdk [25]. and dotmobi offers a “universal” emulator [26]. the simulators that come included with a developer tool kit are more true to the mobile device’s actual behavior. some simulators do a poor job at rendering adaptation techniques. more importantly simulators cannot mimic the mobile context and limitations that mobile devices present in real life usage. the final stage is testing on actual devices. even though testing on all devices is out of the question, testing on a subset of phones that target a set of screen sizes and capabilities is achievable. by identifying device families between which the user experience is the same or similar, you can greatly reduce the numbers of devices on which to test. for example, all palm os devices are largely the same, and all pocket pc devices are largely the same. you can also identify families of phones that have the same characteristics. then out of those families of devices, test the slowest, smallest ones with the most limitations. when osu libraries’ mobile development was at this phase in testing, the team drew up a list of devices to test as well as our test script and drove to the nearest t-mobile store. at the store we introduced ourselves to the store manager, and asked if we could test the site on the display phones. the manager agreed and even ended up helping us test. he proved invaluable, as he knew which phones came with what default setting and browsers. while we cannot recommend that everyone inundate their local cell phone store, the story illustrates how imagination can lead to testing opportunities. another option is to rent mobile devices [27] or pay for a service called deviceanywhere [28], which enables you to test a mobile web site across a wide range of mobile devices. to develop an easy-to-use mobile site user interface, effective usability evaluations should be integrated into the development cycle. there are many graphic packages that make mocking up mobile sites easy. paper prototyping a mobile site can address design issues before a line of code is written and can be very successful for testing functionality with users. another usability method is to develop a usability metric and evaluate the mobile site based on the guidelines. metrics should include standard heuristics, such as ease of navigation, consistency, useful feedback, and speaking the user’s language, as well as mobile specific requirements, such as page load times and image sizes. evaluations and user testing can be performed on simulators, but nothing beats testing on mobile devices in the real world. hallway usability testing can easily be performed on mobile devices and provide the on-the-go context of mobile users. go to anywhere that people are congregating, locate someone with a phone and ask them if they can participate in a user study. usually offering an incentive is a motivator, but make sure you tell your volunteers that they may incur charges from their cellphone provider. focus groups and online surveys are also ways to have users evaluate a mobile site and provide feedback even after the mobile site is released. tracking use statistics on page views, as well as information regarding browser types is useful information when deciding on features to add. osu libraries has been tracking our mobile site since we released the first stage of the project in april 2009. the first stage included information about the library, texting the library contact information, and location-based direction. over the first three months the site received an average of 80 unique user sessions per day. the top three viewed pages are the library hours, text the contact, and floormaps tied with call numbers. the top two browser/platform combination used to access the mobile site are; safari / mac and docomo 2.0 / ntt docomo / n905i. the safari / mac platform are obviously iphone users, but docomo is a japanese mobile provider that comes on a variety of web-enabled phones. this indicates our patrons are accessing the site with non-smart phone devices and furthers our position for providing an optimal mobile experience for all. in september we released the catalog search and computer availability map and will continue to track usage and gather user feedback. continued maintenance of the site has been minimal and is comparable to maintaining any web site. to reduce content upkeep issues, we pull constantly changing content, such as the hours, from a database that feeds both the mobile and desktop web sites. osu libraries’ mobile web site future development includes searching mobile versions of subject guides and course pages, library events that can be added to calendars, and a location-based digital collection application that integrates gps and social media. positive feedback, increasing use statistics and media attention are good indications that osu libraries mobile site is a much needed and appreciated service. conclusion “as libraries consider their re-tooling for mobile users and mobile devices, they should examine the consequences of mobility and the opportunities for innovation in the areas of content, systems and tools, services, and environments, both physical and virtual” (lippincott, 2008). evidence and trends point toward increased levels of user dependence on mobile devices now and in the foreseeable future; therefore, the technical challenges that crop up while creating and maintaining a mobile site are minor when compared with the potential benefits for and goodwill from our users when we create and provide user-centered services like mobile applications. osu libraries’ mobile web site proves attractive, feature-rich mobile sites can be built using standards and industry best practices. the techniques we used to provide an optimal experience for all mobile patrons are easy to implement as long as you follow six main guidelines: consider the mobile context, adapt the content per mobile device, design based on tried and true patterns, test on a subset of phones, validate conformance and evaluate with your patrons. corrections correction #1: november 24th, 2009: we have removed a sentence in the original version of this paper which collapsed two pew reports. previous version: “the pew internet project’s typology of information and communication technology (ict) surveyed 2054 mobile users about how they get information or communicate with others while away from home or work: 46% of cell phone and wireless laptop users said mobile access is very important for getting online information on the go (horrigan, 2009b).” the corrected version reads: “the pew research center’s project found that 46% of cell phone and wireless laptop users think mobile access is very important for getting online information on the go (horrigan, 2009b).” correction #2: december 4, 2009: we fixed the link to the oclc worldcat mobile site. previous version: “oclc is currently offering a mobile search application pilot, worldcat mobile, (http://www.worldcat.org/mobile/) and an iphone search app (http://mobileworldcat.org/).” corrected version: “oclc is currently offering a mobile search application pilot, worldcat mobile, (http://www.worldcat.org/mobile/) and an iphone search app (http://www.oclc.org/news/announcements/announcement353.htm).” authors kimberly griggs is oregon state university libraries’ programmer. she is the designer and developer for osul mobile project and can’t live without her iphone. laurie m. bridges is oregon state university libraries’ business and economics librarian. she is osul mobile team project chair and is attached at the hip to her blackberry. hannah gascho rempel is oregon state university libraries’ biosciences librarian and graduate student services coordinator. she is a member of osul mobile team and is zen-like in her refusal to own a cell phone. references beckman, k. (2009, january). by the numbers: top 10 most popular u.s. handsets in november. rsrwirless, retrieved 09/21/09, from http://www.rcrwireless.com/article/20090108/wireless/901079989/1081/by-the-numbers-top-10-most-popular-us-handsets-in-november horrigan, j. (2008, march). mobile access and data information. pew memo, retrieved 09/21/09, from http://www.pewinternet.org/press-releases/2008/mobile-access-to-data-and-information.aspx horrigan, j. (2009a, july). wireless internet use. pew report, retrived 09/21/09, from http://www.pewinternet.org/reports/2009/12-wireless-internet-use.aspx horrigan, j. (2009b, july). mobile internet use increases sharply in 2009 as more than half of all american have gotten online by some wireless means. pew press release, retrieved 09/21/09, from http://www.pewinternet.org/press-releases/2009/mobileinternet-use.aspx horrigan, j. (2009c, march). the mobile difference: wireless connectivity draws many users more deeply into digital life, but most americans still connect to the internet mainly on wireline and rarely use a mobile device to access digital resources. pew press release, retreived 09/21/09, from http://www.pewinternet.org/press-releases/2009/the-mobile-difference.aspx yesilada, y., chuter, a., & henry, s.l. (june, 2009). shared web experiences: barriers common to mobile device users and people with disabilities. web accessibility initiative, retreived 09/21/09, from http://www.w3.org/wai/mobile/experiences kroski, e. (2008, july). on the move with the mobile web: libraries and mobile technologies. library technology reports, retrieved 02/10/09, from http://eprints.rclis.org/15024/ lippincott, j.k. (2008, december). mobile technologies, mobile users: implications for academic libraries. arl: a bi-monthly report, 261, retrieved 02/02/09, from http://www.arl.org/bm~doc/arl-br-261-mobile.pdf nielsen, j. (2009, july). mobile usability. jakob nielsen’s alertbox, retreived 09/21/09 from http://www.useit.com/alertbox/mobile-usability.html rabin, j., & mccathienevile, c. (eds.). (2008). w3c mobile web best practices 1.0, retrived 09/21/09 from http://www.w3.org/tr/mobile-bp/ west, mark andrew, arthur w. hafner, and bradley d. faust. “expanding access to library collections and services using small-screen devices”. information technology and libraries, v. 25, n. 2, june 2006, pp. 103-107. http://www.ala.org/ala/mgrps/divs/lita/ital/252006/2502jun/west.cfm resources [1] http://www.andymoore.info/download-manager.php?id=13 [2] http://smartmobtoolkit.wordpress.com/2009/01/26/not-device-detectionjavascript-perl-and-php-code/ [3] http://deviceatlas.com/ [4] http://wurfl.sourceforge.net/index.php [5] http://instantmobilizer.com/ [6] http://mymobileweb.morfeo-project.org/mymobileweb [7] http://www.hawhaw.de/ [8] http://wurfl.sourceforge.net/java/index.php [9] http://developer.apple.com/iphone/library/documentation/userexperience/conceptual/mobilehig/introduction/introduction.html [10] http://www.w3.org/tr/css-mobile [11] http://developer.apple.com/safari/library/documentation/appleapplications/reference/safaricssref/articles/standardcssproperties.html#//apple_ref/doc/uid/tp3 0001266 [12] http:// www.passani.it/gap [13] http://www.w3.org/tr/mobileok/ [14] http://ready.mobi/launch.jsp?locale=en_en [15] http://www.opera.com/mini/demo/ [16] http://informatico.altervista.org/netvibes/operamini.php [17] http://www.testiphone.com [18] http://developer.apple.com/iphone/program [19] http://www.marketcircle.com/iphoney/ [20] http://www.forum.nokia.com/info/sw.nokia.com/id/db2c69a2-4066-46ff-81c4caac8872a7c5/nmb40_install.zip.html [21] http://developer.android.com/guide/developing/tools/emulator.html [22] http://developer.openwave.com/dvl/member/downloadmanager.htm?softwareid=23 [23] http://www.blackberry.com/developers/downloads/simulators/index.shtml [24] http://www.microsoft.com/downloads/details.aspx?familyid=a6f6adaf12e3-4b2f-a394-356e2c2fb114&displaylang=en [25] http://developer.palm.com/ [26] http://mtld.mobi/emulator.php [27] http://www.rentobile.com/ [28] http://www.deviceanywhere.com/ subscribe to comments: for this article | for all articles 6 responses to "library/mobile: tips on designing and developing mobile web sites" please leave a response below: randy norwood, 2009-12-04 the oclc worldcat iphone app referenced in “the mobile experience” section goes to an obviously non-oclc site. perhaps the domain has changed hands? otherwise, a very useful article. emily, 2009-12-04 randy, thanks for letting us know. the link should really point to this news release from oclc, which includes instructions for downloading their iphone app from the app store: http://www.oclc.org/news/announcements/announcement353.htm we have fixed the link in the article itself, as well. alan dyck, 2010-03-09 nice work on this article. regarding the section on airpac from innovative interfaces, access for iphones does not require an app download, but is part of the smartphone interface. this is a web-based solution that supports multiple mobile devices using their built-in web browser. kim, 2010-03-18 alan, thanks, for the correction. the line “airpac also has an iphone application” should read “airpac also has an iphone web application.” sylvain machefert, 2010-06-02 in section “detect the device and link to an appropriate css file”, you’re missing a “-” sign in the iphone detection : max-device width:480px ==> max-device-width:480px thanks for the article, it’s very interesting lesson 11: the mobile web « lis 9763 social software and libraries, winter 2012, 2012-03-27 […] devices, there are many design principles that must be considered when creating mobile apps; read this article for a somewhat technical look at designing for the mobile web. additionally, there is a wonderful special section in the october/november 2011 issue of the […] leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – collected work clustering in worldcat mission editorial committee process and structure code4lib issue 30, 2015-10-15 collected work clustering in worldcat worldcat records are clustered into works, and within works, into content and manifestation clusters. a recent project revisited the clustering of collected works that had been previously sidelined because of the challenges posed by their complexity. attention was given to both the identification of collected works and to the determination of the component works within them. by extensively analysing cast-list information, performance notes, contents notes, titles, uniform titles and added entries, the contents of collected works could be identified and differentiated so that correct clustering was achieved. further work is envisaged in the form of refining the tests and weights and also in the creation and use of name/title authority records and other knowledge cards in clustering. there is a requirement to link collected works with their component works for use in search and retrieval. by janifer gatenby, gail thornburg and jay weitz, oclc introduction a collected work can be defined as a group of individual works, selected by a common element such as author, subject, or theme, brought together for the purposes of distribution as a new work. the ifla frbr review group established a working group to review collected works, the working group on aggregates. in 2011, this group produced a report (ifla 2011) after six long years of study and thirteen years after the frbr model [1] was first published (ifla study group on the functional requirements for bibliographic records, 1998). the conclusion took the form of a proposal to enhance the frbr model to encompass viewing a collected work in two ways: first as a new aggregate work and expression or expressions, and second as a manifestation related to the expressions and works of its various component works. like ifla, the first implementations of the frbr model in oclc’s worldcat shied away from collected works. worldcat contains more than 335 million cataloguing records (oclc 2015) for actual publications of a work, termed “manifestations” by the frbr model. determining the actual works represented by these records has been a decade long, continually evolving task within oclc, recognizing that, for most users, the works are more important to the discovery experience than the bibliographic records. thus the 335 million-odd records have been clustered into approximately 267 million work-sets (toves 2015) and further clustered into content and manifestation clusters (called glimir clusters (gatenby 2012)). manifestation clusters are clusters of bibliographic records that describe the same publication, such as parallel records catalogued in different languages conforming to the frbr definition of “manifestation.” content clusters comprise groups of manifestations that are generally equivalent from the end-user perspective (e.g., the original print text and its microform, ebook, and reprint versions, but not new editions). they approximate the frbr “expression level” (gatenby 2012). the work-sets are used in oclc’s discovery and cataloguing services and are distributed as linked data for the benefit of systems world-wide handling bibliographic data (oclc developer network 2015). within the large work-sets in particular, sub-clustering is needed in order to present the sets in a manageable way, by surfacing various editions (for example different revisions and different annotations) and various translations of a work, by different translators into different languages, and clustering their various manifestations (reprints, electronic and text versions). until 2014, most focus was given to clustering within single works, with the understanding that collected works were more difficult but would need attention one day. it is estimated that more than 36 million bibliographic records and 11 million works in worldcat represent collected works (toves 2014). as electronic publications become more numerous, the ability to mix and mash will increase the numbers of collected works and probably produce new types of aggregations of increased complexity. it is important to master the nature of collected works as they currently present themselves in order to arm ourselves for their future evolution. as for the work-sets, the sub-work clustering being done on worldcat will be of benefit to the bibliographic community as a whole. the original glimir project created programs for making clusters of records within worldcat at the sub-work level for parallel records describing the same resource and for records that described the same content, not necessarily the same resource. early in the project, it became clear that mixing single works and collected works in the same cluster would cause serious clustering faults. the first implementation strove to focus on clustering within single works by identifying and setting aside collected works. gail thornburg’s paper, “a candid look at collected works: challenges of clustering aggregates in glimir and frbr” (2014), states: the greatest problem, and most immediate need, was to stop the snowballing of clusters. a follow on project revisited glimir clustering for collected works. the results of this project and the further work it identified are the main focus of this paper. this project aimed to: improve glimir clusters for collected works, i.e., increase clustering by reviewing singletons (work-sets containing only one record) to see if they should or could cluster reduce the size of some of the very large glimir clusters — i.e., reduce over-clustering by better detecting collected works and by better detecting their content improve, in particular, glimir clustering of sound recording records — (because sound recordings are frequently collected works). work-level clustering itself was not in scope, although, like the original glimir project, this project has provided information to improve work-level clustering of collected works. also, generating links from single works to collected works (contained in) and vice versa (contains) was not in scope, though it is envisaged for a related project. implementation and testing of the algorithms the software to identify cluster members is written in java and runs in linux. records are drawn from pools and processed in batches through the same framework that handles batch loading of records to worldcat, de-duplicating of worldcat, and other functions. speed of processing varies considerably according to the overall loads and other factors. each record processed produces a set of glimir matches written as rows to an hbase table [2]. the entries to this table are processed offline nightly in a map/reduce process which synthesizes matches to produce and update clusters. once a month, the results of the daily glimir processing are integrated with the frbr work set processing, a separate set of map/reduce processes, so that, as work ids are updated, glimir clusters that cross work sets can be aligned. for training sets, known use cases for works difficult to cluster correctly were identified by our subject experts. this included 17 sets of collected works known to over-cluster. there was one cluster included that was known not to be a collected work, thus serving as a negative test. set sizes ranged from 50 to 336 records. by the end of the test cycles, four clusters had been removed from the training sets due to invalid data. five formal test cycles were set up, and all cases were tested and reviewed by several domain expert team members to identify errors. errors were logged in our shared project environment, and expert comments were responded to by development team members. in addition, other recordings test files and regression tests were used to verify the changes, though these were not part of the formal test cycles. questionable cases were discussed and resolved at regular team meetings. the measured goals of the project included both refinement of over-clustered cases and improvement to clustering difficult cases which had been under-clustered. reports are now regularly generated for large glimir clusters which can be evaluated as needed. identifying collected works the first implementation of glimir concentrated on clustering single works by manifestation and by equal content. collected works, along with sparse bibliographic records, were the “danger zone”, needing identifying so that they could be isolated from the clustering process. special processing for sparse records has been discussed in a previous article (gatenby 2012). collected works are isolated and separated from individual works because an undetected collected work within the work cluster of an individual work can cause the snowballing of the cluster and the blurring of the distinction of the individual works that severely hampers search and discovery. not only do collected works and individual works need to be distinguished from each other, but it is equally important to distinguish among those identified as collected works (for instance, the same title but different content). glimir clustering and work clustering use similar routines for identifying collected works. a detailed table of glimir tests has been published (thornburg 2014). certain material types are systematically flagged as collected, including “mix” and “kit”. [3] particular focus was given to sound recordings, which are known to have a high proportion of collected works. smiraglia (2001) and velluci (2007) note the high incidence of derivative works in music such as adaptations, amplifications, arrangements and performances. in addition, sound recordings exhibit large problems whose solution could benefit all other material types. examples of these problems include a high number of recordings by the same performer of different works, numerous recordings of the same work by different performers, high numbers of generic titles and different arrangements. also noted are recordings of the same work by the same performer over time, such as early and late career. identifying content of collected works this project then focused on the clustering of worldcat records flagged as collected works. records are clustered where the content is determined to be equal. until now, collected work records were only clustered at the manifestation level where they were determined to be describing the exact same resource. from a pragmatic viewpoint, there are collected work records where the contained works can be identified and those where the contained works cannot be identified. the latter are exemplified by generic titles in the 245 $a such as “selected works” or “collected works” without any more precise data elsewhere in the record. beyond simple tagging these can only be ignored; they cannot enter into a content cluster unless they have matched at the manifestation level with another record. there is a case for declaring such records to be sparse records and to try to downplay them in displays, preferring other more detailed records. this is a subject for a future project. identifying the content of collected works differs from identifying content of individual works. an individual work may appear in different editions, different translations, different formats and different performances (e.g., talking book versus printed, or different orchestral performances). glimir separates these into different content clusters. thus there may be multiple content clusters within a work cluster. however, for collected works, the determination of content is largely at a higher level, i.e., the checking of the component works. though the checking is currently done by the glimir algorithms at the glimir content level, if it is determined that one bibliographic record for a collected work has different multiple work content compared with another bibliographic record, then it should not only be a separate content cluster, but it should also be in a separate work cluster. more work is required to harmonise glimir and work clustering in this area. the works within collected works can be expressed in a record in different ways, depending on the chosen cataloguing style. it is this mixture of cataloguing styles that presents a challenge to content identification and clustering. these are some of the cases where the content can be identified: the record has multiple 700s or 710s with $a (name) and $t (title) subfields the 505 (contents note) has multiple works separated consistently by punctuation or by subfielding multiple titles can be detected in the main title field (245). cast list, determined by examining the participant or performer note (511), and added creator / title entries (7xx) are considered together with contents notes (505). further tests for differentiating contents were added that were applicable to both single and collected works, looking for conductor and orchestra. where there is a different combination of conductor and orchestra, different content was determined. figure 1: ocn 40834246 example record with content expressed in 505 contents field figure 2: ocn 75301540 example record with content expressed with multiple 700 fields containing $t (and multiple titles in 245) figure 3: ocn 36906049 example record with 700 indicator 2 (and 245) = analytical work by examining the 505 content field, together with the 511 notes on participant or performer (cast), good clustering has been made possible as per the example in figure 4. figure 4: spreadsheet used for testing algorithms figure 4 illustrates part of a manually crafted spreadsheet that was used for testing the algorithms as they were developed. all seven records have been identified manually as having equal content, and this has been replicated by the algorithms; all seven now have all the same content identifier applied by the algorithms. this is despite differences in all key fields — main creator, title, cast, performance notes and content notes. some records attribute roy eldridge (the person) as the main contributor and others roy eldridge 4 (the musical group). some contents notes have titles only, and some include performers and timing. the algorithms will also treat incomplete contents notes that end in an ellipsis. though the test sets used were focusing on music (bruckner’s symphony number 9, bernstein’s mass, montreux 77, 16 greatest hits), the improvements made to creator and cast checking to identify them in new fields within bibliographic records enabled improved differentiation for illustrators of juvenile works. further work as the project deadline drew closer, focus was given to ensuring that, as good clusters were made, no bad ones were made at the same time. as with any project, some already defined enhancements were sidelined for a potential future project and some new problem areas were detected. refining the current clustering event dates, such as performance dates extracted from field 518, are currently used in duplicate detection. more use could be made of these dates in differentiating content. the weights given to the various tests could be further adjusted to achieve more content matches. specifically, some cases were noted where a mismatch on material type and extent have overruled matching content and cast notes. sparse records were identified for special treatment early in the first glimir project and isolated so that they did not cause over-clustering and also so that they could be downplayed in search and retrieval (gatenby 2012). this project is revealing that collected works require stricter sparse rules. locally bound items, particularly of musical scores, were noted as a potentially problematic case for further review. enriched authority control name / title authority records are useful in work clustering. authority records containing established facts complement clustering algorithms as they can provide the missing link between two bibliographic records, enabling them to be clustered. salaba and zhang (2007) claim that the main weakness of most current implementations of the frbr model is the reliance only on bibliographic data without authority data. ted fons (2014) argues for extracting entities from worldcat and creating rich authority records with as much context as possible to enable sharing as linked data to maximize their value to the web as a whole. a work linked-data service is already offered; it is desirable to explore the possibilities for sub-work entities. a recent oclc research project has examined the richer records in worldcat and used them to generate more than 1 million authority records for translated works (oclc research 2014-2015 and smith-yoshimura 2014). thus, the richer records in worldcat are mined to make new-style authority records that are akin to wikipedia’s and google’s knowledge cards, capable of informing work clustering algorithms and enabling them to better cluster all records. in a similar way, clustering at a work and sub-work level could greatly benefit from the extension of the authority record model to include records for performances and repertoire. like translations, these can be created by mining the most complete records in worldcat. for example there are three identified different published performances of bernstein’s mass. a knowledge card containing the conductor, lead performers, choirs and orchestra, and place and time, could be crafted that would permit better identification and clustering of bibliographic records of varying degrees of detail. similarly, knowledge cards could include titles with opus or thematic index identifiers, such as köchel identifiers, and be linked to the known recorded performances. use of knowledge cards especially for well-known, deceased creators would greatly improve the discovery experience by providing accurate clustering within large clusters. it is ironic to produce less than optimal results knowingly for fear of over-clustering by relying solely on algorithms, where rich information is widely known and available that could supplement and complement the algorithms. moreover, less than adequate clustering and presentation of well-known identities and their works weakens the perceived authority of the catalogue. the work sets associated with well-known identities tend to be large and difficult to cluster. thus exploring a combined algorithm / data mining / knowledge card approach is highly desirable. once we understand the repertoire, we are better able to identify collected works, and the ensemble of works they contain, and improve the discovery experience. container or work? optimising retrieval of contained works is a collected work really a work, or is it just a container? to date, librarianship practice has treated all collected works as new works, or, in frbr terms, aggregate works (tillett 2009). however, sometimes a collected work is merely a container, as is clearly the case for locally bound items, mentioned previously, or individual works grouped together for marketing purposes. sometimes a collection has been brought together with a critical introduction and more. thus a continuum exists between collected works that are containers and works in their own right, and the portion in the middle is fuzzy and open to interpretation. users probably do not distinguish between the two, and catalogue records are not consistent in their detail, making it hard to classify them solely from the data within a bibliographic record. for these reasons, it is pragmatic to treat all collected works as real works in their own right, separate from their component works. nevertheless, it is possible to improve retrieval of individual works by also identifying them within collected works. retrieval of individual works from within collected works is patchy. some records, as illustrated above, do not have added entries for their components, and those that do may have a variant title form — meaning that a work within a collected work can only be retrieved by that one title string. where the content of collected works has been identified, the work could be linked to the individual work records by explicit “contains / containedin” links. such links could be considered as an application of the changes to the frbr model proposed by the ifla review group working group on aggregates (ifla 2011). these explicit links can be important in finding translations of works, as publishers commonly release translations as collected works — for example, the only available russian translations of the works of dick francis are in compendiums. generating these links could be the subject of a future proposal. unification or synchronisation of algorithms another challenge is the unification or synchronisation of algorithms. currently, for practical purposes, in oclc’s worldcat, the three processes of duplicate detection and resolution (ddr), manifestation and content clustering (glimir), and work clustering, are separate. new and significantly modified records, whether added online or in batch load, are normally put through the ddr process after an interval of seven days to allow time for a possible reversal if required. work clustering is neither incorporated in batch loading nor is it part of online additions; it is currently done at monthly intervals (though more frequently is envisaged) on a dedicated machine with a database copy, and then the work identifiers are copied into the production copy. glimir simple clustering of books is treated similarly, and there are separate processes for the retrospective processing of glimir clusters for more complex material types. retrospective glimir clustering is thorough, but slow, and batches of records for the remaining 39% of records that are lacking glimir ids are being steadily processed in nightly runs. having three processes has meant duplication of code that is not always easy to fully align, resulting in discrepancies due not only to different code but also different processing timescales. ideally, when new records are loaded into the database, the three processes should be integrated into one. that is: “is this record a duplicate of an existing record? if not, is it the same manifestation as an existing record? if not, is it the same content as an existing record? if not, is it the same work?” glimir clusters are used by the work algorithms to bring works together and have caused 3.6 million bibliographic records to move work-sets to date (toves 2015). the work-set algorithms could also use the new glimir collected work content clustering, where it identifies different component works of a collected work, to realign some work clusters. cluster presentation as previously mentioned, optimizing clustering has been a decade-long event, and determining optimum ways to present the clustering information to end users has been just as challenging. oclc research has presented some prototype displays in its work records project (oclc research 2013-2015), and viaf [4] has some new work displays to present translations of works. these displays are being evaluated for incorporation in the main displays of worldcat.org as entities other than the bibliographical record become the focus (fons 2014). conclusion the identification of collected works was the key starting point of the project. the existing identification rules established for the first glimir project were confirmed and extended, and these rules have also been adopted for work clustering. the major objective was to cluster collected works by content. for the sound recordings test sets, the project has succeeded in identifying component works within collected works and has done so without making any false clusters. by extensively analysing the cast-list (508, 511, 245 $c), performance notes (518) and contents notes (505), titles (245, 246, 7xx $t), uniform titles (240, 740) and added entries (7xx), the contents of collected works could be identified and differentiated so that correct clustering was achieved. retrospective clustering of approximately 4 million musical sound recordings identified as collected works is in progress as part of a drive to process all sound recordings, musical and non musical, single and collected work. other sets of records affected by the enhancements of this project are also being identified and will be scheduled for re-processing. the project has inspired ideas for further significant development. refining the existing algorithms is desirable but more impact will come from incorporating rich authority records and knowledge cards into the clustering process in conjunction with the effort to mine new entities from the database. these new entities will lead to new ways of displaying the data and new ways of sharing with the bibliographic community world wide, particularly as linked data. acknowledgments this paper has greatly benefitted from the input, and patient and meticulous review from jenny toves and stephan schindehette. richard greene contributed significantly to the project with his considerable musical and metadata expertise. tribute is also due to the team as a whole: robert bremer, tony chirakos, bert cousins, janifer gatenby, richard greene, marianne kozsely, glenn patton, russ pollock, bob schulz, gail thornburg and jay weitz. oclc quality control provided invaluable input and review throughout the successive test cycles. references fons, theodore (2014). authorities, entities and communities: paper delivered at ifla 2014, lyon. available from: http://library.ifla.org/1034/1/086-fons-en.pdf (viewed 2015-03-11) gatenby, janifer, greene, richard o., oskins, w. michael and thornburg, gail (2012) glimir: manifestation and content clustering within worldcat. code{4}lib journal issue 17, 2012-06-01. available from: http://journal.code4lib.org/articles/6812 (viewed 2015-03-10) ifla. frbr review group. working group on aggregates (2011). final report available from: http://www.ifla.org/files/assets/cataloguing/frbrrg/aggregatesfinalreport.pdf (viewed 2015-03-16) ifla study group on the functional requirements for bibliographic records (1998). functional requirements for bibliographic records: final report. — münchen: k.g. saur, 1998. — (ubcim publications; new series, vol. 19). — isbn 3-598-11382-x [viewed 2015-03-24]. available from: http://www.ifla.org/en/publications/functional-requirements-for-bibliographic-records oclc (2014) worldcat works linked data: connecting library resource into the core of the web available from: http://www.oclc.org/en-europe/publications/nextspace/articles/issue23/worldcatworkslinkeddataconnectinglibraryresourcesintothecoreoftheweb.html (viewed 2015-03-12) oclc (2015) worldcat statistics: 2015-02-15. available from: http://www.oclc.org/en-europe/publications/nextspace/articles/issue24/worldcatstatistics.html (viewed 2015-03-12) oclc developer network (2015) oclc linked data. available from: http://www.oclc.org/developer/develop/linked-data.en.html (viewed 2015-03-12) oclc research project (2014-2015): multilingual bibliographic structure. http://www.oclc.org/research/themes/data-science/multilingual-bib-structure.html?urlm=168909 (viewed 2015-03-09) oclc research project (2013-2015) work records in worldcat. http://www.oclc.org/research/themes/data-science/workrecs.html (viewed 2015-03-31) salaba, athena and zhang, yin (2007) from a conceptual model to application and system development: functional requirements for bibliographic records. american society for information science and technology. august/september 2007. available from: http://www.asis.org/bulletin/aug-07/salaba_zhang.pdf (viewed 2015-03-24) smiraglia, richard p. (2001). the nature of “a work”; implications for the organization of knowledge. lanham, maryland, scarecrow press, 2001. smith-yoshimura, karen (2014) challenges posed by translations. hanging together may 20th 2014. available from: http://hangingtogether.org/?p=3878 (viewed 2015-03-12) thornburg, gail, and oskins, w. michael (2012). matching music: clustering versus distinguishing records in a large database. oclc systems and services, v. 28, issue 1 available from: http://dx.doi.org/10.1108/10650751211197059 (viewed 2015-03-10) thornburg, gail (2014): a candid look at collected works: challenges of clustering aggregates in glimir and frbr. information technology and libraries vol 33, no.3 september 2014 available from http://ejournals.bc.edu/ojs/index.php/ital/article/view/5377 (viewed 2015-03-10) tillett, barbara (2009) definition of aggregates and works: tillett proposal [to the ifla frbr working group on aggregates ifla conference 12 august 2009] available from http://www.ifla.org/files/assets/cataloguing/frbrrg/aggregates-as-works.pdf (viewed 2015-03-11) toves, jenny (2014) email correspondence 2014-08-14 and 2014-04-21 toves, jenny (2015) email correspondence 2015-03-10 vellucci, sherry l. (2007). “frbr and music”. in understanding frbr: what it is and how it will affect our retrieval tools, ed. arlene g. taylor (westport, cf: libraries unlimited, 2007), 131-151. notes [1] the frbr model comprises four main entities: a work, an expression, a manifestation and an item. to illustrate: emile zola wrote germinal (work) that was translated into english by roger pearson (expression) and published by penguin in 2004 (manifestation) and a particular copy can be found on this library’s shelves (item). [2] hbase is a column-oriented database management system that runs on top of hdfs. it is well-suited for sparse data sets, which are common in many big data use cases. unlike relational database systems, hbase does not support a structured query language like sql; in fact, hbase isn’t a relational data store at all. hbase applications are written in java much like a typical mapreduce application. hbase does support writing applications in avro, rest, and thrift. http://www-01.ibm.com/software/data/infosphere/hadoop/hbase/ (viewed 2015-09-08) [3] material type is a classification of format computed for each bibliographic record in worldcat based on selected values in the leader, 006, 007, 008, 33x, and occasionally other fields. it indicates certain aspects of content, medium, and carrier. tests are made for uniform titles indicating collected works by coding and by a list of generic titles such as “selections” and “oeuvres completes”. a check is made for multiple paginations. similarly added entry creator / title fields (7xx series) coded as analytic or in multiple occurrences are indicators. the title proper (245), if it contains the word “selection” or multiple occurrences of semicolons, is an indicator, as is also the occurrence of multiple 740 fields with second indicator 2 and the varying title field (246) in some circumstances. the most recent refinements involved checking contents notes (field 505). punctuation (forward slash, hyphen and semicolon) and multiple occurrences of “movement” or a statement of responsibility subfield or title subfield are used as evidence of a collected work. [4] http://viaf.org (viewed 2015-03-31) see for example: http://viaf.org/viaf/196911378/ anatole france’s les dieux ont soif. about the authors janifer gatenby works on metadata projects at the offices of oclc bv in leiden, netherlands. her main project for the last 5 years has been the development and maintenance of the isni database and system. she has been at oclc since 2000, working on data-related projects including the multi-lingual bibliographic structure project, the glimir project and standards related to system interoperability. she has contributed significantly to the development of international standards since 1993 including sru, openurl request transfer message, iso holdings, iso data elements, sru update, library registry (iso 2146), isni, isci and z30.50 among others. gail thornburg, ph.d., has taught at the university of maryland and elsewhere, and is now a senior-level developer and researcher at oclc, and author of numerous papers. in recent years her work has focused on expert software algorithms for matching and clustering data in very large datasets. she has been recipient of an oclc president’s award in 2010, and recipient of emerald publishing outstanding paper award in 2013. jay weitz is a senior consulting database specialist at oclc and was previously assistant catalog librarian at capital university, columbus, ohio. he has a ba in english from the university of pennsylvania, an mls from rutgers university, and an ma in education from ohio state university. he is the author of cataloger’s judgment, both editions of music coding and tagging, and the cataloging q&a columns of the music oclc users group newsletter and the online audiovisual catalogers newsletter. he serves as oclc liaison to numerous groups. since 1992, he has presented dozens of cataloging workshops in the u.s., canada and japan. he was the recipient of the moug distinguished service award in 2004, olac’s nancy b. olson award in 2005, and for his work on the reimplementation of duplicate detection and resolution software in worldcat, an oclc president’s award in 2010. since 1981, he has been program annotator for concerts of chamber music columbus (ohio, usa). he has been a performing arts critic in public radio, in print and on the web, and currently serves as theatre and dance writer for the weekly alternative newspaper “columbus alive” (http://www.columbusalive.com). subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – update oclc holdings without paying additional fees: a patchwork approach mission editorial committee process and structure code4lib issue 49, 2020-08-10 update oclc holdings without paying additional fees: a patchwork approach accurate oclc holdings are vital for interlibrary loan transactions. however, over time weeding projects, replacing lost or damaged materials, and human error can leave a library with a catalog that is no longer reflected through oclc. while oclc offers reclamation services to bring poorly maintained collections up-to-date, the associated fee may be cost prohibitive for libraries with limited budgets. this article will describe the process used at austin peay state university to identify, isolate, and update holdings using oclc collection manager queries, marcedit, excel, and python. some portions of this process are completed using basic coding; however, troubleshooting techniques will be included for those with limited previous experience. by nicole wood and scott shumate background adding an oclc institution symbol to a record in oclc connexion indicates that the item described in the record is owned by the corresponding library [1]. this is integral to the interlibrary loan process, in which these holdings are utilized to locate items for lending and borrowing. these holdings are added and deleted through oclc as libraries acquire and weed materials. some libraries may set holdings for their entire collection, while others may only set holdings for resources that they wish to circulate through interlibrary loan, excluding collections that are non-circulating like reference materials, or collections with copyright and delivery restrictions such as ebooks. if these holdings are not regularly reviewed, time, lack of training, and forgetfulness can culminate in a catalog that is no longer accurately depicted in oclc. oclc offers two services to bring these holdings up to date: reclamation and data sync. during reclamation, library records are matched with the worldcat equivalent and holdings are set and deleted accordingly. this provides a complete holdings overhaul and is most beneficial to libraries that “have not consistently maintained [holdings] and cannot easily isolate only the records that need to be updated” [2]. this is a fee-based service that may be cost-prohibitive to libraries with limited budgets. data sync services for ongoing collection maintenance are included in oclc cataloging subscriptions; however, this is only recommended for libraries that have maintained regular holdings updates [3]. at austin peay state university (apsu), holding updates are built into the workflow for acquiring and discarding materials. holdings for e-resources are maintained through worldshare collection manager, while print holdings are updated on a title-by-title basis as resources are added and removed from the collection. apsu’s last reclamation project was conducted in 2016. over time, interlibrary loan and cataloging personnel began to notice rogue oclc holdings that were not present in the apsu library catalog. the records covered a broad range of creation dates and bibliographic locations. it is likely that these incorrect holdings are the result of past weeding projects and the occasional instance of setting holdings for the wrong item during the initial cataloging process. due to budget restrictions, a reclamation project was not possible. in order to perform the less cost-prohibitive data sync, the entire catalog needed to be reviewed to extract the limited number of records with incorrect holdings. because e-resource holdings are automatically updated through worldshare collection manager, we only needed to update our holdings for print resources. the goal was to compare holdings in our ils with our holdings in oclc, extract non-duplicate entries, and either delete or add holdings accordingly. oclc collection manager for this project, we wanted to correct oclc holdings for the library’s print monographs. to initiate this process, we needed to get an overview of all of our current oclc holdings for this collection. oclc worldshare management services users have access to a report of all titles held by their institution, but we do not subscribe to this service. worldshare collection manager queries can provide the same information. query collections deliver sets of marc records from which the appropriate information can be extracted. our query searched for all marc records that have our oclc symbol but do not have a material type of electronic. the steps for creating the query are: create a new query collection in collection manager. under ‘worldcat selection criteria’ write the following string: li:tpa not 14:tpa not x4:digital not mt: elc. ‘li’ is the index label for the holding library symbol. tpa is our oclc symbol. ‘l4’ is the index label for local holdings records, which we only set for serials. by excluding local holdings records, we were able to exclude holdings for serials. ‘x4’ is the format/facet in worldcat.org. while we stated that we do not want a format of digital or a material type of electronic, this double clarification might be redundant. ‘mt’ is the index label for material type. while we used not mt: elc to create a query that did not include electronic resources, not mt: url could have also been used to create a query for records that do not have a url present in the 856 field. select one-time delivery to prevent ongoing delivery of this large file. figure 1. screenshot of query collection in wcm scroll down and open the marc records accordion. select ‘deliver records for this collection in a separate file’ to ensure that these records aren’t merged with any other collection manager download files. this query returned 218,741 records. according to our inventory records, we have 223,473 physical volumes in our collection that correspond to these search criteria. a portion of this discrepancy can be accounted for by factoring in the numbers for our uncataloged government documents and other non-circulating collections that do not have holdings set in oclc. marcedit the collection manager query collection provided a download file with complete marc records for each non-electronic resource that had our oclc symbol attached to it. to compare these records with those in our ils, it was necessary to discern a unique identifier, in this case the oclc control number. the oclc control number can be found in the 001 field of a marc record, which corresponds with the bibliographic utility number in sierra, our ils. depending on local cataloging practices, it may be helpful to consider the 019 (oclc control-number cross reference), 035 field (system control number), or the 010 field (library of congress control number). these numbers can be isolated in marcedit using the find function or the tab delimited records function. to use the find function: hold down ctrl + f or open the find and replace search box. search for =001 and select ‘find all.’ figure 2. screenshot of a find and replace search in marcedit. when the search results load, click on the copy icon in the lower left-hand corner of the search results box to save the content to the computer’s clipboard (this looks like two pieces of paper stacked on top of one another). open excel and paste the control numbers. the pasted data will contain the found text (=001 [control number]) and the marcedit action (jump to record#:). fortunately, the marcedit actions are in an isolated column and can easily be deleted. the found text and control numbers can be separated by using the ‘text to columns’ feature under the data tab. the format for each cell is =001 [control number] (example: =001 49632629). isolate the control number in a new column by using the space between =001 and the control number as a delimiter, and delete the column containing the =001 marc field tags. figure 3. in ‘text to columns,’ use the space as a delimiter. to use the export tab delimited records function: from the marcedit home screen, select ‘export tab delimited records’ under tools. set the file path from the collection manager download file to a new .txt file. select tab as the field delimiter and a semicolon as the in-field delimiter. add 001 as the field to export. figure 4. add 001 as the field to export when using export tab delimited records. open excel and import the external data from the new .txt file created by the marcedit export. if the data is imported in the format of [control number]\, copy and paste the following formula into an adjacent column to delete the last character (\) for each control number: =left(a1,len(a1)-1) sierra ils once the oclc control numbers for non-electronic records with tpa as the oclc symbol were identified, downloaded, and isolated, they could be compared to numbers in the 001 bibliographic utility number field in sierra. if a record is in our collection and our holding symbol is attached to a record in connexion, these numbers will match. if there is a number without a match, the holdings are incorrect – either because we own an item without the associated holdings or because holdings are set for an item we no longer have or for a record that has since been merged. consider your library’s interlibrary loan circulation habits when creating a review file in the ils. if you have non-circulating collections, such as reference or reserves, you will want to exclude these from the search. the needed bibliographic utility numbers were identified by creating a query for non-electronic resources and exporting the 001 field for all records in the output review file. the query needed to create this review file will vary by ils and by local cataloging practices. to pull the records for apsu, we ran a query for all bibliographic records that were in certain locations, such as main book collection, popular reading collection, and government documents. a similar search could have been performed for all records that did not have an 856 field. when satisfied with the contents of the review file, the bibliographic utility numbers were exported to a .txt file, and then imported the .txt file into excel. the creation of this query proved to be the most time-intensive step in this process, and the query was revised many times before the results were usable. after completing the steps below, if the file output is much smaller or larger than expected, spot check the records and adjust the search criteria accordingly. our final search criteria included: item record = exists and bibliographic location = general book collection or juvenile book collection or reference collection or government publications or video collection or popular reading collection or archives excel once lists of holdings from oclc and the ils have been acquired, excel can be used to sort out entries that do not have a match. if a control number from oclc does not have a counterpart in the ils, it can be assumed that the record should no longer have holdings attached. similarly, a control number that only exists in the ils will need to have holdings set in oclc. this does not take into account control numbers that might appear in the 010, 019, and 035 fields, but this limitation can be discussed with an oclc cataloging specialist during the data sync process. the imported data may have different formatting. our ils control numbers were exported as written, while oclc assigned prepended zeros for all numbers with less than six characters. excel will recognize 685 and 000685 as duplicate values, so there is no need to alter this formatting. if preferred, prepended zeros can be added to the ils figures to make identical entries. to do so: highlight all cells in the column of numbers without prepended zeros. right click on the column and choose ‘format cells.’ choose ‘custom’ for the category, and replace the word ‘general’ (under ‘type’:) with the pattern 000000. this makes the minimum number of digits six and fills in the needed digits with prepended zeros. at this point, values with multiple entries could hypothetically be identified by using conditional formatting to highlight cells with duplicate values and then using sort and filter to remove any entries that are highlighted. however, these actions caused excel to crash on several computers with varying levels of processing power. csv and python scott shumate converted the excel data into two csv files (one for ils numbers and one for oclc numbers) and wrote a python script that compared both files and delivered values only found on one list to a separate output file. this code requires python 3 and will need to be adjusted accordingly if using python 2. #!/usr/bin/env python3 # import library to let us read and write csv files, # and check operating system paths import csv import os.path # make an empty list/array called iii that will hold the # iii values from our csv # then iterate through the csv and append each row iii = [] with open('iii.csv') as csv_file: csv_reader = csv.reader(csv_file, delimiter=",") line_count = 0 for row in csv_reader: iii.append(row) line_count += 1 print("[iii] processed " + str(line_count) + " lines.") # make an empty list array called oclc that will hold # the oclc values from our csv, # then iterate through the csv and append each row oclc = [] with open('oclc.csv') as csv_file: csv_reader = csv.reader(csv_file, delimiter=",") line_count = 0 for row in csv_reader: oclc.append(row) line_count += 1 print("[oclc] processed " + str(line_count) + " lines.") # make an empty list for our output that will only contain # our results output = [] # count variable is just for monitoring purposes. # look at each value in the iii list, then check to see # if it is in the oclc list. if it is not, # put that value into our output variable # for monitoring purposes, this also checks our count and # prints every 100 rows count = 0 for i in iii: if i not in oclc: output.append(i) count += 1 if count % 100 == 0: print("[iii] " + str(count)) # count variable is just for monitoring purposes. # look at each value in the oclc list, then check to see # if it is in the iii list. if it is not, # put that value into our output variable # for monitoring purposes, this also checks our count and # prints every 100 rows print("done with iii.") count = 0 for o in oclc: if o not in iii: output.append(o) count += 1 if count % 100 == 0: print("[oclc] " + str(count)) print("done with oclc.") # last but not least, output our "output" variable # to an output csv file we can open in excel with open('output.csv', 'w', newline='', encoding='utf-8') as file: writer = csv.writer(file, quoting=csv.quote_nonnumeric, delimiter=',') writer.writerows(output) to get a list of records in oclc but not our ils, we ran the original script but deleted the following loop: count = 0 for o in oclc: if o not in iii: output.append(o) count += 1 if count % 100 == 0: print("[oclc] " + str(count)) print("done with oclc.") results at this stage, it is important to spot check a few records in the output file to ensure that the queries and code behaved as expected. this process required many rounds of trial and error before a feasible solution was found. this is the approach that worked for us, but libraries with different oclc subscriptions, ilss, and cataloging practices will need to tweak things accordingly. our initial file output included 34,278 control numbers without matches. we conducted a quick review of 30 randomly selected records. of these, 19 were present in both the catalog and oclc holdings. further investigation revealed that these erroneous outputs were the result of input data that needed to be cleaned up (e.g. a government document collection in which control numbers were assigned a prefix) or specific item record and bibliographic record locations that were not originally factored into the ils query (e.g. books that were temporarily assigned to a display case or featured bookshelf). our final output file returned 9,676 records that did not have a match in the ils and oclc. we conducted another spot check of randomly selected records, this time looking at 50 records. none of the records had matches in our ils and oclc. six of the 50 records had oclc holdings but no equivalent in our ils. the 44 remaining records did not match on the 001 field but had matches between the 001 field in our catalog record on the 019 field in oclc. if you are confident that your output file only contains control numbers for holdings that need to be added or deleted, this can be done quickly through a batch update in the oclc connexion client. to do this, select ‘batch holdings by oclc number’ under the batch tab, and copy and paste the relevant control numbers. this process can be used to update and delete holdings. because our 9,676 record output contained a mixture of holdings that needed to be deleted and records that needed to have control number updates, we processed these by creating a data sync collection in oclc collection manager. choosing to delete holdings through the connexion client places the responsibility on the cataloger to ensure that all of the data is accurate. when using a data sync collection to update records, an oclc database specialist is assigned to the collection to review the submitted data and provide clarification on which type of data sync is required to achieve the requested modifications. we will continue to add and delete holdings for our physical collection as items are acquired for or weeded from the collection, but this process provides the ability to spot check our collection as needed to account for the inevitable user error. additional resources: to learn more about creating query collections, visit oclc’s collection-level settings in query collections guide [4]. to view a concise table of available indexes, visit oclc’s searching worldcat indexes: quick reference guide [5]. for a comprehensive description of all indexes, visit oclc’s indexes toolbox [6]. to learn more about creating data sync collection in collection manager, visit oclc’s guide [7]. bibliography [1] oclc. cataloging: take actions on bibliographic records. 2018. available from: https://www.oclc.org/content/dam/support/connexion/documentation/client/cataloging/bibactions/bibactions.pdf [2] oclc. create a bibliographic collection for a reclamation. 2020, march 3. available from: https://help.oclc.org/metadata_services/worldshare_collection_manager/choose_your_collection_manager_workflow/data_sync_collections/create_a_bibliographic_collection_for_a_reclamation [3] oclc. available data sync collections. 2019, may 7. available from: https://help.oclc.org/metadata_services/worldshare_collection_manager/choose_your_collection_manager_workflow/data_sync_collections/about_data_sync_collections_in_collection_manager/available_data_sync_collections [4] oclc. collection-level settings in query collections. 2019, june 18. available from: https://help.oclc.org/metadata_services/worldshare_collection_manager/choose_your_collection_manager_workflow/query_collections/about_query_collections_in_collection_manager/collection-level_settings_in_query_collections [5] oclc. searching worldcat indexes: quick reference. 2020, february 25. available from: https://help.oclc.org/metadata_services/worldshare_record_manager/reference/searching_worldcat_indexes_quick_reference [6] oclc. indexes. 2017, october 12. available from: https://help.oclc.org/librarian_toolbox/searching_worldcat_indexes/indexes [7] oclc. data sync collections. 2020, may 19. available from: https://help.oclc.org/metadata_services/worldshare_collection_manager/choose_your_collection_manager_workflow/data_sync_collections about the authors nicole wood (woodn@apsu.edu) is the resource management librarian at austin peay state university in clarksville, tn. she oversees the library’s monographs collection and maintains the library’s catalog. scott shumate is the it analyst for digital services at austin peay state university. he works on digital initiatives and supports the use of technology in the library. subscribe to comments: for this article | for all articles one response to "update oclc holdings without paying additional fees: a patchwork approach" please leave a response below: tomasz, 2020-08-25 thanks for sharing your experiences syncing your worldcat holdings. just one quick suggestion. when comparing two large data sets as in your ils and oclc numbers csv files it’s much more convenient to use pandas python package with all its magic. finding duplicates, unique values, values present in one set but missing in the other, etc. is a breeze and can be reduced to a few lines of code. as your data sets get larger, advantages of pandas become more and more apparent. cheers! leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – editorial introduction – a cataloger’s perspective on the code4lib journal mission editorial committee process and structure code4lib issue 11, 2010-09-21 editorial introduction – a cataloger’s perspective on the code4lib journal on the code4lib journal, technology, and the universe of library cataloging and metadata. by kelley mcgrath coordinating editor, issue 11 i sometimes feel myself a bit of an odd duck on the code4lib journal editorial committee as i am not a coder. i am a cataloger. however, when i think about this a little more, it no longer seems so odd for a cataloger to be on the editorial committee. the future of library cataloging and metadata is inextricably bound up with technology. this is nothing new—as karen coyle points out, the card catalog was an innovative technological breakthrough in its day [1]. the pace and degree of technological change only continue to increase. the rising ubiquity of full-text searching, our expanding ability to mine large pools of electronically-manipulable data [2], the interconnected linking system of the internet, and the rise of online social networks coupled with the ability to analyze the collective output of the hive mind have all changed the information-seeking landscape. although many predict doom and obsolescence for library metadata, i do believe that bibliographic data still has a role to play in the universe of information and that libraries bring significant resources to the table. for one thing, we have a large and valuable corpus of largely-standardized bibliographic data created over a long period of time that would be expensive and time-consuming to replicate. many other entities, such as google books and the open library project, would like to utilize this data resource. libraries also have a pool of people well aware of the challenges in creating useful, consistent, accurate, and complete metadata. we are dedicated to helping people connect with information resources that they want or need rather than to commercial aims. we have been thinking about the description and organization of documents for a long time and have a rich tradition of theory and practice. however, there are significant obstacles that make it difficult for library metadata to reach its full potential in today’s world. our legacy data is a mixed blessing. we have all this bibliographic data, but our ability to use it in today’s environment is limited by the fact that our data was designed in an era of very different technological constraints. although people’s expectations have expanded along with the capabilities of technology, the traditional aims of the catalog as set out by figures like cutter and ranganathan are still valid. understandably, but unfortunately, when the library world transitioned from the card catalog to the computer-based catalog, we largely tried to carry over the same means of achieving these aims. if we were starting over today, i think we would do things very differently—not so much in terms of what, but in terms of how. we need to find ways to meet users’ needs that are more compatible with the strengths of contemporary technology. we must transition our underfunded, interdependent, cooperation-based but locally-customized, and monolithic but decentralized systems for creating, structuring, maintaining, and sharing bibliographic data. we must identify practical means to turn them into systems that work in the internet age. in many library world dialogues, such as those on ngc4lib, a list for discussing next generation library catalogs, there often seem to be battle lines drawn between traditional catalogers and the evangelists of the power of new technology. both sides often seem to be talking past each other, as well as past the budget-stressed, penny-pinching library administrators who are looking for a greater return on their investment in metadata. if we cannot build an effective contemporary framework for creating, structuring, and sharing library metadata, the path of reducing library metadata to the lowest common denominator rather than manifesting its tremendous potential power in an online environment may be chosen for us. i don’t think the aims of traditional cataloging and the changes that are necessary to optimize library metadata for today’s automated and interconnected world are in conflict. however, the cooperation and knowledge of both technology experts and cataloging and metadata experts are needed to navigate the transition from where we are now to where we need to go. for this, people from all backgrounds and positions must be willing to take part in open-minded conversations and reexamine their basic assumptions. it is necessary to understand the problems that traditional bibliographic control is trying to solve and the assumptions that are built into the historical solutions. it is also necessary to understand the potential of new tools to expose our data and to help users navigate the ever-expanding bibliographic universe. finally, it is necessary to understand how our existing data is often not well-suited for exploiting the many possibilities or exposing the many connections that computer manipulation could make possible. we need to create data that is functional both for direct human consumption and for automated processes that can enhance and mine that data for value-added forms of human consumption. i see my role on the editorial committee as a way to play a small part in helping to facilitate this conversation and to forge a path to more effective and powerful bibliographic data. the ethos of the code4lib journal manifests many of the qualities i most admire in the library world, including cooperation, free and honest sharing of ideas and solutions, open-mindedness, a supportive and collegial environment, and a focus on practical solutions. it is one of many venues where ideas about the synergy of metadata and technology are being hashed out and one that i am proud to participate in. this issue of the code4lib journal once again contains a solid line-up of useful articles, many of which discuss the interaction between metadata and technology in libraries. in “interpreting marc: where’s the bibliographic data?” jason thomale discusses the mismatch between the expectations about data structure that he brought from the modern computing environment and the structure that he found in marc. thomale aims to help other programmers frame marc/aacr records in a way that will help them more effectively manipulate library data stored in marc. the article is a good example of how different underlying paradigms held by catalogers and programmers can obstruct communication and impede effective data use. better tools for inputting and editing metadata are desperately needed. in “xforms for libraries, an introduction” by ethan gruber, bill parod, and scott prater describe the use of xforms applications to support consistent and correctly-structured metadata. teressa m. keenan discusses how the university of montana used marcedit and xslt to modify publisher-supplied metadata for the u.s. congressional serial set, 1817-1994 to make it more useful for their users and to better integrate it into their catalog in “why purchase when you can repurpose? using crosswalks to enhance user access.” marcedit, a free marc editing tool developed by terry reese, is an excellent example of how the power of technology can be harnessed to improve the quality and efficiency of library metadata. marcedit also demonstrates how mutual respect and open communication between technology and metadata experts can lead to improved results for the whole community. for example, on marcedit-l, the marcedit user community can suggest improvements and learn how to get more out of this valuable tool. in “hacking summon,” michael klein describes flexible, locally-maintainable techniques for adjusting and tweaking communication between a consolidated discovery interface, such as summon, and data stores, such as the local catalog, in order to customize library-specific data and produce more desirable results for users. najko jahn, mathias lösch, and wofram horstmann describe a project to create lists of publications of bielefeld university faculty with full text links in “automatic aggregation of faculty publications from personal web pages.” they faced a tough metadata challenge when they tried to transform the unstructured and inconsistent data they harvested into sensible citations. finally, in “managing library it workflow with bugzilla,” nina mchale takes a look at how auraria library systems department used bugzilla as a low-cost, easy-to-set-up tool to improve their workflow for tracking and solving it problems in a timely manner. notes [1] coyle, k. (2005). catalogs, card—and other anachronisms. journal of academic librarianship, 31(1), 60-62. preprint retrieved from http://www.kcoyle.net/jal-31-1.html. [2] halevy, a., norvig, p., & pereira, f. (2009). the unreasonable effectiveness of data. ieee intelligent systems, 24 (2), 8-12, retrieved from http://accelerationwatch.com/downloads/halevynorvigpereiraunreasonableeffectivenessofdatais2009.pdf subscribe to comments: for this article | for all articles 3 responses to "editorial introduction – a cataloger’s perspective on the code4lib journal" please leave a response below: jeffrey beall, 2010-09-22 hi, kelley: that’s a very well written introduction, and i look forward to reading the articles. in the piece, you say, “many other entities, such as google books and the open library project, would like to utilize this data resource.” actually, oclc is sharing metadata from worldcat with google, for the google books database. here’s a link to an announcement that describes the arrangement” http://www.oclc.org/news/releases/200811.htm cataloging futures, 2010-09-24 catalogers and coders, perfect together… if you have time to read only one thing this week, make it kelley mcgrath’s thoughtful introduction to the new issue of the code4lib journal: a cataloger’s perspective on the code4lib journal. kelley succinctly lays out the important issues the…… musa, 2010-09-27 traditonal cataloguing has persisted and refuse to die and be buried once and forever. however it is delighting to know cataloguer ve come to time with old and new system and in addition appreciate the need to merge and move on leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – consortial rightsstatements.org implementation and faceted search for reuse rights in digital library materials mission editorial committee process and structure code4lib issue 45, 2019-08-09 consortial rightsstatements.org implementation and faceted search for reuse rights in digital library materials the florida academic library services cooperative (falsc) makes available digital library hosting free-of-charge to all institutions of florida public higher education. 21 institutions participate in the islandora digital library platform hosted through falsc. centralized digital library hosting through falsc, or its predecessor consortium, has been available since 1994. meanwhile, the rightsstatements.org standard, which provides a controlled vocabulary for indicating the copyright status of digital library material, was released in 2016. after the standard was released, participating libraries expressed interest in implementing rightsstatements.org for existing digital content. during fall 2018 and spring 2019, falsc implemented rightsstatements.org values on islandora sites. this article describes the process undertaken by falsc, the lessons learned, and recommendations for libraries looking to implement rightsstatements.org values. by wilhelmina randtke, randy fischer, and gail lewis. introduction the florida academic library services cooperative (falsc) provides digital library hosting free-of-charge to all institutions of florida public higher education. through this program, 21 institutions participate in the islandora digital library platform hosted through falsc. centralized digital library hosting through falsc, or its predecessor consortium, has been available since 1994. meanwhile, the rightsstatements.org standard, which provides a controlled vocabulary for indicating the copyright status of digital library material, was released in 2016. after the standard was released, participating libraries expressed interest in implementing rightsstatements.org for the large backlog of existing digital content hosted through falsc. during fall 2018 and spring 2019, falsc implemented the rightsstatements.org values on islandora sites. this involved: adding the standard to falsc’s local metadata object description schema (mods) metadata profile. configuring solr such that the search results can be meaningfully faceted by reuse rights. there are 12 possible rightsstatements.org values, which can be confusing and difficult for researchers to understand. through group meetings and dialogue between member institutions, these were grouped into 3 categories of reuse: free re-use, limited re-use, and no-reuse. solr indexing was configured to allow users of digital library sites to limit search results by those three manageable categories and allow researchers using the digital library to quickly find materials which can be reused for presentations or to create open educational resources. adding the rightsstatements.org values to metadata forms available to librarians and others adding material to the islandora platform. assisting member libraries in implementing the rightsstatements.org values by sharing existing training resources from other groups, by providing excel spreadsheet reports of digital library metadata to assist libraries in quickly working through materials, and by coding a batch insert to allow libraries to return a completed excel spreadsheet and have falsc add the values to existing islandora materials. from the time falsc’s batch insert code became available in february 2019 through the march 2019 conclusion of the first push to add rightsstatements.org values, the standardized copyright statements were added to approximately 60% of public materials in islandora not coming through florida state university (fsu), the largest participating digital library. this paper provides information on the way rightsstatements.org values were grouped into broader, easier-to-understand reuse categories for end users and the process of communicating with libraries to empower each library to add copyright statements to a large amount of existing content. it also includes recommendations on improving the process, including addressing a digital library collection-by-collection versus item-by-item and a strong recommendation to approach rightsstatements.org by simultaneously adding a more detailed copyright statement showing the logic behind the controlled statement in order to “show your work” so the end user can more quickly confirm whether or not material can be reused. background on rightsstatements.org what does rightsstatements.org do? the core of rightsstatements.org values is that they are computer readable markup about copyright status and that they aren’t licenses. a rightsstatements.org markup on a work is a uri referencing a url which says something to the effect of: this is what we believe the copyright status of this work to be, and here are several disclaimers that this might not be accurate and you may want to do your own additional research to verify this.[1] the rightsstatements.org values are similar to the established creative commons licenses.[2] the creative commons licenses allow an author or creator of a work to attach a license onto the work in standardized markup, which can be read by a computer. the rightsstatements.org values perform a similar function, showing copyright status, but are not licenses and specifically sidestep promises and legal liability. the rightsstatements.org statements are a standardized markup that can be applied to items which a library has digitized from its collections without the library needing to own the copyright to the item. in contrast, a license requires ownership of the copyright in order to grant a license. so a creative commons license would have to be applied by the author at the time the material is given to a library or the author would later have to be contacted to provide the license. for the vast majority of digital library content, licensing is impractical or impossible. for older materials, the author may be deceased. even materials with a living author have likely been collected without the library or archive simultaneously getting a license. comprehensively implementing creative commons licenses across all digital library material is not feasible. any computer applications supported by computer-readable creative commons licenses can be supported by rightsstatements.org statements. and because the rightsstatements.org statements can be applied by a library to a backlog of materials without any action by the authors of those materials, there is a possibility of quick and comprehensive adoption for cultural heritage materials in digital libraries. by being computer-readable, the rightsstatements.org statements theoretically allow for faceting by reuse rights similar to that allowed by creative commons licenses in searches like google advanced search.[3] rightsstatements.org is also interoperable with creative commons licenses. both can be used across a set of records, with creative commons in use for items which can be licensed and rightsstatements.org in use for items which are marked up by someone other than the author and therefore can’t be licensed. history of the rightsstatements.org standard. europeana[4] is a massive centralized federated search of digital library materials in europe. rightsstatements.org is a relatively recent standard compared to creative commons, which was released in 2002.[5] in april 2016, the digital public library of america (dpla) and europeana jointly launched rightsstatements.org and the rightsstatements.org controlled values.[6] at the time of launch, both federated search projects emphasized a rightsstatements.org value as a pending future requirement for all items contributed to dpla or europeana. in the case of dpla, this requirement is coming soon with many contributing digital libraries a in the process of implementation. in the case of europeana, the requirement was to have either an existing statement in the europeana licensing framework or to have a rightsstatements.org value. as of 2019, europeana is the only major digital library search to use the rightsstatements.org values as part of the interface. the history of europeana, including the early standardization of copyright statuses within europeana materials in 2012 using the europeana licensing framework, is the most likely reason for this.[7] before rightsstatements.org was released, europeana had a critical mass of existing items with a controlled copyright statement in the europeana licensing framework and so contributing digital libraries had existing statements that could be mapped to or combined with rightsstatements.org values. significantly, not all items in europeana have a controlled rights statement. as of june 2019, running searches on europeana will result in more “hits” than are available by summing together rights statements statuses within that set of hits (i.e., a search locating 290,000 hits might have 287,000 hits with a copyright statement and a handful of hits without). running searches which retrieve a large number of hits and examining facets by copyright status reveals that many existing statements in europeana are “copyright not evaluated.”[8] nevertheless, europeana has a critical mass of controlled copyright statements to allow meaningful faceting by reuse rights and has incorporated those into the search interface. you can see this by going to europeana’s search portal at https://www.europeana.eu/portal/en and running a search. along the left hand side of your search results, there is a facet for “can i use it?” this facet is built on the controlled rights statements and maps both creative commons licenses and rightsstatements.org statements to broad categories of reuse: free re-use (public domain or attribution required), limited re-use (educational or nonprofit use allowed and contractual/legal restrictions), and no re-use (in copyright or copyright not evaluated). meanwhile, in the u.s., the controlled rightsstatements.org values are gaining traction. as they are adopted and implemented by different libraries, a critical mass of content is building in the form of digital objects labeled with a rightsstatements.org statement. assuming implementations of the rightsstatements.org statements are generally accurate, these may soon be incorporated into existing searches and into digital library interfaces to allow search by reuse. the interface on europeana shows the kind of application that is possible with widespread adoption of the rights statements as digital library technologies begin to incorporate this information into the interfaces. background on flvc/falsc hosting for digital library content the florida virtual campus (flvc) and florida academic library services cooperative (falsc) are state-established governmental entities in florida.[9] falsc is housed within flvc. falsc provides centralized technology-based services to all main campus libraries in public higher education in florida: the florida state university system and florida state college system. at this time, this represents 40 institutions total. participation in the centralized online catalog is mandatory. access to digital library hosting services is optional and is provided free-of-charge. as of summer 2019, 21 institutions use digital library hosting services through falsc’s islandora platform to publish open access content to the web. not all have a public web presence. some do not maintain their own public web presence, but rather display materials through a publication of archival library & museum materials (palmm) site where materials from several digital libraries around the state are grouped into collections around floridiana topics and are displayed together, with institutional branding appearing on individual items in the bigger digital library.[10] palmm is a long running collaborative project which launched around 2000 based on collaborative digitization efforts begun in 1998.[11] palmm contains materials related to floridiana, unique collections held by institutions in florida, and collaborative collections where materials are held across multiple institutions and are presented together on the shared site to present a unified whole. in 2014 and 2015, falsc was restructured from two predecessor organizations, the college center for library automation (ccla) and the florida center for library automation (fcla). fcla had previously provided digital library hosting services since at least 1994, with the florida entomologist journal coming online hosted by fcla prior to 1994.[12] as a result of such a long history of digital library hosting, institutions using falsc’s digital library services have a significant amount of material hosted on falsc’s servers which were uploaded long before the rightsstatements.org standard existed. these historic materials had metadata which predated standardized copyright statements and required metadata remediation to add statements. background on the ssdn and dpla in florida the dpla provides a centralized search of digital library resources. the dpla launched in 2013 after a two and half year planning and design phase. it contains records for more than 21 million items coming from more than 3,000 cultural heritage institutions around the united states.[13] the sunshine state digital network (ssdn) is florida’s dpla hub.[14] in summer 2018, the ssdn released metadata guidelines for digital libraries wishing to contribute records to the dpla through the ssdn.[15] a copyright or license statement is required by ssdn, and a preferred format for this statement is rightsstatements.org . throughout 2018 and to present, there has been widespread interest in ssdn and dpla participation among falsc members’ digital libraries. rightsstatements.org implementation adding the standard to falsc’s local metadata object description schema (mods) metadata profile. falsc-hosted islandora sites use mods to store metadata. the metadata profile for the islandora sites is posted to https://fig.wiki.flvc.org/wiki/index.php/metadata#flvc.27s_local_mods_profile. the mods profile is used for internal software development and promoted to librarians working with metadata around the state as a way to check over internal workflows and verify that the use of mods is consistent with falsc tools and software. after the ssdn released metadata guidelines in summer of 2018, falsc promptly checked over the guidelines and ensured that each standard could be met by member libraries wishing to participate in ssdn and dpla. initial analysis revealed copyright statements as a key field to focus on. the copyright statement was the only field required by ssdn which was not previously required by falsc’s mods profile or by the software implementation of the falsc islandora platform. accordingly, falsc coordinated with member libraries using the islandora platform and proceeded to implement rightsstatements.org in the islandora platform. adding the rights statements to the falsc mods profile. falsc’s mods profile lists a handful of extra requirements beyond the library of congress’s mods standard. additional requirements generally are in place because of software requirements. based on an analysis of solr and xpath used in falsc’s mods editing forms, falsc determined to store the rights statements as follows: example rightsstatements.org element in a mods record: <accesscondition type="use and reproduction" displaylabel="rightsstatements.org"> http://rightsstatements.org/vocab/inc/1.0/ </accesscondition> example creative commons element in a mods record: <accesscondition type="use and reproduction" displaylabel="creative commons"> http://creativecommons.org/licenses/by/4.0/ </accesscondition> the displaylabel attribute is available for any mods element and provides an easy way to isolate the rightsstatements.org controlled values in a record. it is more reliable than string matching, and so allows code and xpath queries to accurately isolate and manipulate the rightsstatements.org values. configuring solr such that the search results can be meaningfully faceted by reuse rights. libraries using the falsc-hosted islandora perform an advisory role for the service. any libraries eligible for services through falsc are welcome to participate in monthly meetings of the islandora subgroup (isg) of falsc. the isg considers issues that affect all sites, such as standardized settings which are configured centrally by falsc (standardized settings in the software allow falsc to provide meaningful training and troubleshooting to many campuses), prioritizing software development, and identifying issues which affect all sites. following ssdn’s release of metadata guidelines, the rightsstatements.org values were discussed at the monthly isg meeting. there was widespread interest among member libraries in implementing the statements. two main concerns came out of isg meetings. one aspect of discussions was how to represent the rightsstatements.org values in the user interface for researchers and members of the public using the islandora sites. the other was a concern from the isg with how to add the rightsstatements.org values to a large backlog of materials. the isg quickly determined that faceted search and search options built on the rightsstatements.org values were desirable. in a search interface, a large number of options can be confusing and difficult for researchers to understand. the isg quickly determined that it would be desirable to show fewer than 12 options to end users. a good model for how reuse might appear in search came in the form of google advanced search, which uses the long standing and well-established creative commons licenses to allow search by “usage rights”.the key concept desired by isg was to have many statements grouped into a smaller number of easily understandable categories. a smaller group was formed within the isg, in order to determine how to group the rightsstatements.org values into broader categories. this group was formed from any interested librarians who responded to a recruitment call at the isg meeting and through the isg’s listserv. additionally, falsc reached out to scholarly communications librarians at member institutions and requested input on how to represent the rightstatements.org values to researchers and the public. through a series of meetings and emails, the group considered different categories to break the statements into. possible sources for facets. the isg considered several possible breakdowns of rightsstatements.org values into a smaller number of easily understood and easily managed categories. google advanced search: restricting search by reuse rights. google advanced search was one possible source for categories of reuse rights. the search options in google advanced search use the well established creative commons licenses to allow searchers to filter by 5 possible reuse rights: “not filtered by license”, “free to use or share”, “free to use or share, even commercially”, “free to use share or modify”, and “free to use, share or modify, even commercially”. figure 1. screenshot of how creative commons licenses are used for searching by category of reuse in google advanced search. rightsstatements.org : the rights statements are grouped into three reuse categories. the 12 rightsstatements.org values are listed below. on the rightsstatements.org website, the statements are broadly grouped into three categories: in copyright, no copyright, and other.[16] these three categories were considered as a possible grouping for the florida system. table 1. the rights statements as grouped on the https://rightsstatements.org website. rights statements for in copyright objects in copyright uri: http://rightsstatements.org/vocab/inc/1.0/ in copyright – eu orphan work uri: http://rightsstatements.org/vocab/inc-ow-eu/1.0/ in copyright – educational use permitted uri: http://rightsstatements.org/vocab/inc-edu/1.0/ in copyright – non-commercial use permitted uri: http://rightsstatements.org/vocab/inc-nc/1.0/ in copyright – rights-holder(s) unlocatable or unidentifiable uri: http://rightsstatements.org/vocab/inc-ruu/1.0/ rights statements for objects that are not in copyright no copyright – contractual restrictions uri: http://rightsstatements.org/vocab/noc-cr/1.0/ no copyright – non-commercial use only uri: http://rightsstatements.org/vocab/noc-nc/1.0/ no copyright – other known legal restrictions uri: http://rightsstatements.org/vocab/noc-oklr/1.0/ no copyright – united states uri: http://rightsstatements.org/vocab/noc-us/1.0/ other rights statements copyright not evaluated uri: http://rightsstatements.org/vocab/cne/1.0/ copyright undetermined uri: http://rightsstatements.org/vocab/und/1.0/ no known copyright uri: http://rightsstatements.org/vocab/nkc/1.0/ europeana: the rights statements are used for faceting search results into three broad categories. europeana incorporates the rights statements into its search interface in the form of facets on the left hand side of search results. figure 2. reuse categories powered by rightsstatements.org on the europeana search interface. the facets appear on the left hand side of search results. figure 3. screenshot of the expanded facets in the europeana search interface, showing which rightsstatements.org value is grouped into which category. florida’s decision to follow europeana. after several meetings of the subcommittee within isg, the group came to the consensus to follow europeana. the reasoning behind this is that europeana was the only identified user interface built on rightsstatements.org, and the florida systems wished to follow that example on the theory that dpla and other large digital libraries likely will follow europeana and that the implementation and user interface would be more standard. a major concern expressed with europeana’s breakdown of rights statements into categories is that some statuses appear to be sorted to the wrong category. for example, europeana groups the creative commons (remember creative commons licenses are incorporated into the rightsstatements.org standard) license for attribution-sharealike into the category of “free re-use”. but, sharealike requires content that reuses or remixes an item to attach an identical license.[17] that’s a significant restriction because someone cannot take material released under a sharealike license, rework it, and then release it freely to the public. instead, the identical license must attach. the exact license which must be used for derivative works and remixes is dictated by a sharealike license. despite minor disagreements with how the rights statements were sorted by europeana, the isg decided to follow europeana in the interests of standardization. following the decision by the isg, solr fields were configured to allow faceting search results by the three broader reuse categories. the facets were added to logged in views for librarians around the state for preview and experimentation but were not immediately added to the public interface. each digital library is able to send a request at any time to make the fields available to the public. libraries which have chosen to do so have added the statements to a critical mass of items on the site before incorporating them into the public interface. timing and sequencing concerns in implementation: faceted values stored directly to solr, rather than generated on the fly from rightsstatements.org values. in parallel with the isg committee, flvc has an internal flvc islandora developers team which meets weekly. this team consists of computer programmers, librarians, and systems administrators who support the islandora platform. once the isg had settled on the idea of breaking out the 12 rights statements into a smaller number of categories for search, the flvc islandora developers team looked at how to implement this with solr. solr drives the search interface on the islandora platform. how information is stored to solr and which fields exist is set up by configuring solr directly and making changes to the solr schema. the administrative settings in the islandora web interface allow existing solr fields and the content indexed into them to be added to the search interface. figure 4. screenshot of solr admin settings on broward college’s islandora site. once a field is added to solr’s indexing, and that field has been populated, it’s easy and straightforward to incorporate that field into the user interface for islandora. each item in islandora is indexed into solr when the item is initially created and the solr index is updated each time the item’s metadata is updated. comprehensively reindexing the sites is no small task. as of summer 2019, there are approximately 200,000 mods xml records indexed to solr across the islandora sites for the 21 institutions using islandora. comprehensive reindexing of solr would require significant time, measured in days, and would have an impact on the florida islandora system performance. the reason for this is that there are many items stored in the system with full text included, so there’s more to index than just mods metadata fields. the full text records are voluminous and require more time per record than would mods metadata only. selectively reindexing items with a specific metadata field would require directly accessing and analyzing the mods xml records in order to pull a list of items to reindex. doing so would be manageable but would have been an extra step. comprehensive reindexing in smaller chunks also involves an extra level of choreography. in short, in implementing a new field, it was desirable to avoid reindexing. the way to avoid reindexing is to configure the solr schema for a new field before adding that field to records. therefore, no rightsstatements.org values were added to records until both the isg had made recommendations regarding fields and the changes had been implemented in the solr configuration. following the ssdn’s release of metadata guidelines in summer 2018, the process of determining how rightsstatements.org would be incorporated into the user interface and implementing that decision in solr was completed approximately 4 months later, in mid-fall 2018. following that, updated mods metadata editing forms were released to libraries to allow the rightsstatements.org values to be added to records. adding the rightsstatements.org values to metadata forms available to librarians and others adding material to the islandora platform. the islandora software handles metadata entry and updates using the islandora forms module.[18] librarians using falsc-hosted islandora can update metadata on records by clicking to one of two standardized forms used across all islandora sites in the falsc system and maintained centrally by falsc. unless falsc makes a separate process available focused on a specific field, there is no way to do a batch update of metadata across a site or collection. adding rightsstatements.org values to the mods editing forms did not present a streamlined path forward for digital libraries wishing to add statements to a large backlog of materials, collected and hosted beginning in the 1990s decades before the standard existed. adding the statement manually to a single record takes 8 clicks. while that can be streamlined somewhat with a direct url to open the metadata form for an object, comprehensive implementation using only the islandora mods forms would have involved a lot of clicking. assisting member libraries in implementing the rightsstatements.org values with reports of digital library objects and a batch insert. falsc supported rights statements implementations at member libraries by sharing existing training resources from other groups regarding rightsstatements.org and copyright, by providing excel spreadsheet reports of digital library metadata to assist libraries in quickly working through materials, and by coding a batch insert to allow libraries to return a completed excel spreadsheet and have falsc batch add the values to existing islandora materials. sharing training materials about the copyright statements. because librarians working with the islandora platform come from a wide variety of backgrounds and often wear many hats on their campus, falsc’s goal was to point to concise training materials. falsc selected a small set of educational materials which were concise and could be worked through in less than a day. the following three resources were promoted to libraries as good starting points for gaining the educational background needed to implement rightsstatements.org : aserl webinar about rightsstatements.org[19]: an excellent one hour introduction to rightsstatements.org it is a good resource for determining how to approach rightsstatements.org and deciding whether to implement the statements in a set of digital library materials. society for american archivists’s (saa’s) guide to implementing rights statements from rightsstatements.org[20]: only 7 pages long with a flow chart. this is a great place to start for planning workflows for assigning rightsstatements.org values to items in a digital library. because it is short, it is easy to provide to anyone at a library who will work with implementing the statements. cornell’s copyright status chart: copyright term and the public domain in the united states[21]: a long chart (14 pages if copy-pasted into microsoft word) for pinpointing the copyright status of an item that isn’t in one of the clear categories mapped out in the saa guide. this is a good reference for digging into the copyright status of a single book or item. additionally, libraries were able to meet up with falsc online through screen sharing and go over reports, collections, and the rightsstatements.org materials. reports of all materials on each islandora site. shortly after ssdn released metadata guidelines and as part of the early discussions about copyright at the isg, falsc prepared reports of items on the islandora sites with the goal of providing information that could help libraries to determine copyright status. the reports included for each item on the site with a mods record: identifier of item link to the item on the digital library islandora content model of item (ie. book, pdf, basic image). taken from the islandora content model, which is a software setting that describes what components a digital object has and how to display that item in the web interface. it is not taken from the mods metadata. collection the item was in. this allowed quickly sorting and marking the same status for homogenous collections. title any date information. because mods has several fields where a publication date can be stored, these fields were concatenated. (ie. the elements <dateissued>, <datecreated>, <datecaptured>, <copyrightdate>, and <dateother> were all provided together in a single column of an excel report.) any existing copyright statements. (ie. contents of either <accesscondition> with no type attribute or of <accesscondition type=”use and reproduction”> element) a blank area to add a rightsstatements.org value. reports were provided to the primary contact for each islandora implementation in an excel spreadsheet. islandora plays a varying role from campus to campus across the state. some campuses have daily activity with multiple full-time librarians working on the islandora platform. others have content hosted but are not actively adding content and do not regularly attend isg. because of this, falsc made sure to reach each recipient of the report by phone before sending a report. the phone call allowed falsc to explain that adding the rightsstatements.org values was voluntary so the spreadsheet did not have to be completed and that falsc was focusing on this field because of widespread interest among isg members in ssdn and dpla participation. initial phone contact was important so that libraries without staffing or desire to add the statements would not misunderstand why they were getting a spreadsheet. phone contact also ensured that each library would be aware that applying to the ssdn and dpla was a separate additional process and not mediated by falsc. along with the spreadsheet reports, falsc suggested to libraries that if a critical mass of reports were returned, and if multiple libraries across the state requested a batch insert, that coding the insert might be possible. libraries were encouraged to hold off by manually adding the statement until spring 2019, when the isg and falsc could be certain whether there was enough interest in the batch insert to justify the resources needed to code and implement a batch insert. batch insert of rightsstatements.org values. initial training materials and spreadsheet reports regarding rightsstatements.org were provided to member libraries in august 2018. user interface decisions and implementation in solr were completed by november 2018. as of december 2018, three member libraries had returned completed spreadsheets with a formal request as to whether falsc could do a batch insert, and one member library had manually implemented rightsstatements.org values comprehensively across the site using the mods forms. multiple requests for a batch insert and the fact that other libraries had indicated a future intent to return the reports justified allocating resources for the batch insert. in early 2019, work to code and promote the batch insert began. falsc provided specs to flvc’s division of information technology unit and code was available in february 2019. code was not available to member libraries and instead the insert is run by falsc librarians in command line. the batch insert script takes a .csv file as input with a list of identifiers and corresponding rightsstatements.org values. islandora 7, currently in use at flvc/falsc, is built on a fedora commons 3.7.1 framework and stores metadata as mods xml text files. the script goes row-by-row and inserts the value directly into mods xml then trips off solr reindexing for that item. during february and march 2019, falsc promoted the batch insert. the isg and individual libraries were contacted and informed that any libraries returning a completed or partially completed report would be able to have the batch insert done. on contact, several libraries returned completed spreadsheets which the libraries had prepared in fall 2018 but not yet submitted. updated reports including newly added materials were available on request. libraries expressing a need for further training in copyright were offered screen sharing sessions where falsc and the library worked through reports and identified rightsstatements.org values for specific categories of materials, such as public domain for materials published in the united states before 1924, educational use only for specific collections of homogenous materials, or in copyright for newer materials such as current campus publications. the screen sharing sessions focused on large chunks of content similar to those in the flowchart in the saa guide. some of the focus of screen sharing sessions was also on how to better group similar content in excel in order to more efficiently fill out the spreadsheet. when a library returned a completed report, falsc would check over the report at a high level. falsc checked formatting of the rightsstatements.org uris, and any irregularities were automatically corrected. if a library typed out words for a statement, for example, “in copyright” rather than providing the rightsstatements.org uri, a uri was suggested and the library was contacted to approve the uri. if a library turned in less meaningful values, for example, “copyright not evaluated” for all items, the library was asked to confirm before running the insert. on follow up, no member library opted to globally apply “copyright not evaluated.” for any inserts that looked inaccurate, falsc asked the library to confirm before running the insert. for example, a follow up question might be asked about very old materials marked as in copyright. most returned spreadsheets of values for insert were generally accurate. most libraries that were offered to meet up and review items together accepted the invitation. by the end of march 2019 the batch insert had been run to add rightsstatements.org values to 35,477 items on the islandora sites. this represented 62% of the public items across the islandora sites not coming through fsu and more than 80% of palmm islandora materials. fsu has an in-house metadata librarian who is able to use xslt to make batch inserts. fsu’s local mods profile differs slightly from falsc’s, including an implementation of rightsstatements.org values in mods. regarding fsu, falsc coordinated to ensure compatible solr indexing, but did not assist with the batch insert. only 52 of the 35,477 inserted rightsstatements.org values were for “copyright not evaluated” and the added rightsstatements.org values represent meaningful information for search. libraries not participating in the batch insert cited primarily staffing as the concern. some background on what records are shared may be helpful in understanding the scale of the content. in terms of numbers of items across the florida islandora system, the florida islandora sites are configured to share out by oai-pmh only top level records for newspapers and serials, but to not share out issue and article records. so, in the case of a newspaper, a single record can represent thousands of individual issues made available through the software. going forward, falsc offers the ability to batch insert rightsstatement.org values. since march 2019, a trickle of requests has come in for batch inserts and falsc anticipates promoting the batch insert on a regular basis. rightsstatements.org in the islandora user interface for sites which have requested to make the reuse facets available in search, the facets for reuse show up along the left hand side of search results under the heading “can i use it? rights status”. this mirrors the language used on europeana, and the breakdown of which statements map to which facet also mirrors europeana. the final groupings of rights statements and creative commons licenses into categories for faceting are shown below in table 2. as of summer 2019, facets can be seen on the digital libraries for the palmm project[22], broward college[23], and florida gulf coast university[24]. table 2. groupings of rightsstatements.org and creative commons licenses into broader reuse categories on the falsc islandora sites. free re-use rightsstatements.org: no copyright us: http://rightsstatements.org/vocab/noc-us/1.0/ rightsstatements.org: no known copyright: http://rightsstatements.org/vocab/nkc/1.0/ public domain: http://creativecommons.org/publicdomain/mark/1.0/ creative commons: cc-by-sa: http://creativecommons.org/licenses/by-sa/4.0/ creative commons: cc-by: http://creativecommons.org/licenses/by/4.0/ creative commons: cc0: http://creativecommons.org/publicdomain/zero/1.0/ limited re-use rightsstatements.org: no copyright – non-commercial use only: http://rightsstatements.org/vocab/noc-nc/1.0/ rightsstatements.org: in copyright – educational use permitted: http://rightsstatements.org/vocab/inc-edu/1.0/ rightsstatements.org: in copyright – non-commercial use permitted: http://rightsstatements.org/vocab/inc-nc/1.0/ rightsstatements.org: no copyright – contractual restrictions: http://rightsstatements.org/vocab/noc-cr/1.0/ rightsstatements.org: no copyright – other known legal restrictions: http://rightsstatements.org/vocab/noc-oklr/1.0/ creative commons: cc-by-nc: http://creativecommons.org/licenses/by-nc/4.0/ creative commons: cc-by-nc-nd: http://creativecommons.org/licenses/by-nc-nd/4.0/ creative commons: cc-by-nd: http://creativecommons.org/licenses/by-nd/4.0/ creative commons: cc-by-nc-sa: http://creativecommons.org/licenses/by-nc-sa/4.0/ no re-use rightsstatements.org: copyright not evaluated: http://rightsstatements.org/vocab/cne/1.0/ rightsstatements.org: copyright undetermined: http://rightsstatements.org/vocab/und/1.0/ rightsstatements.org: in copyright: http://rightsstatements.org/vocab/inc/1.0/ rightsstatements.org: in copyright – rights-holder(s) unlocatable or unidentifiable: http://rightsstatements.org/vocab/inc-ruu/1.0/ rightsstatements.org: in copyright – eu orphan work: http://rightsstatements.org/vocab/inc-ow-eu/1.0/ figure 5. facets on the palmm islandora search appearing to the left of search results. recommendations for others implementing rightsstatements.org in providing reports to libraries, the fields chosen (identifier of item, link to item on the digital library, islandora content model, collection the item was in, title, all date information, and existing copyright statements) were useful for quickly determining copyright status for many materials and for being able to click into items in the digital library for a closer look. one weakness in the reports was that the collection information was not emphasized. in fielding follow up questions, it became clear that sorting out by collection and determining status one collection at a time was the best strategy for quickly and accurately working through digital library materials at many institutions. it is recommended that any digital libraries looking to implement the rightsstatements.org values first examine copyright status at a collection-level and only go at an item-level when a collection contains a variety of widely varying content. this collection-level strategy should be emphasized early in any comprehensive implementation of the rights statements. running the batch insert centrally rather than providing it as a self-service tool was also desirable because the few formatting issues with the rightsstatements.org values returned by libraries could be corrected. for example, formatting issues included https instead of http for a uri and a copied and pasted url for a webpage showing the text of the statement instead of the uri. by fixing basic formatting centrally, some frustration was likely avoided across the state and it ensured that well-formatted data went into the system. the batch insert code validates against the list of rightsstatements.org uris. by running the batch insert in-house, falsc reviewed error reports and centrally corrected formatting issues. running the batch insert centrally was also desirable for being able to push out copyright training and information about the ssdn and dpla. in promoting the insert, falsc was able to field basic questions and to connect libraries with information and resources as needed. a major shortcoming of the batch insert is that falsc only inserted the controlled rightsstatements.org values and did not simultaneously insert an uncontrolled copyright statement showing the logic behind that copyright status determination. in looking at a metadata record, seeing a status of “no copyright us” is not quite as good as seeing that in combination with an uncontrolled statement like “published in the us before 1978 without a copyright notice. not subject to us copyright.” currently, in florida islandora sites, there’s no way for the library to “show the work” behind the copyright determination without manually going in and adding the uncontrolled statement. in asking questions about returned spreadsheets and in screen sharing meetings with libraries to go over materials, it is clear that libraries put thought and care into the copyright status determinations. for example, libraries marking very old materials as “in copyright” tended to do so because the digital library interpreted that specific digital image made by the campus to be under copyright, although the original digitized item was not. another example is libraries working with a collection where the donor had pledged certain rights with the donation may have an accurate rightsstatements.org value with respect to reuse rights, but without the uncontrolled statement, it’s not possible to confirm that purely by looking at the item and its metadata online. in many cases, the same rightsstatements.org value was applied across material in an entire collection. it would have been straightforward to simultaneously document the decision making process had code been planned, spec’d, and written to support that documentation in the form of allowing the library to batch insert an uncontrolled copyright statement along with the rightsstatements.org value. it is strongly recommended that any project to comprehensively add rightsstatements.org values also simultaneously allow adding an uncontrolled statement to “show the work” and demonstrate the logic behind the copyright statement. a significant weakness in the falsc implementation is that copyright training was offered later rather than earlier in the project. staffing is a big concern with a consortium supporting many institutions. in any consortial implementation, a good path to follow might be to recruit potential peer mentors around the state and locate librarians with backgrounds in copyright who are able to answer questions and work through items and collections with librarians whose strengths are in other areas. this is also something for dpla hubs to consider in an implementation. experience with copyright was highly variable among librarians across the state and mentoring, or even the ability to get a quick answer to a question, was often not available within a given member library. community networks, specifically for assistance with determining copyright status, could be helpful. it’s also important to stress that the rightsstatements.org values are specifically written to avoid liability, so adding them to materials does not trigger legal issues. something like release forms should always be sent for review within a campus or organization, but basic education and mentoring in determining status should not normally require legal review. the rightsstatements.org values are written specifically to avoid that need. references [1] see, for example, the rights statement language for “no copyright – united states” at http://rightsstatements.org/vocab/noc-us/1.0/. the notices section states, “unless expressly stated otherwise, the organization that has made this item available makes no warranties about the item and cannot guarantee the accuracy of this rights statement. you are responsible for your own use.” a standard disclaimer included on all rights statements states, “disclaimer the purpose of this statement is to help the public understand how this item may be used. when there is a (non-standard) license or contract that governs re-use of the associated item, this statement only summarizes the effects of some of its terms. it is not a license, and should not be used to license your work. to license your own work, use a license offered at https://creativecommons.org/“ [2] https://creativecommons.org/. [3] https://www.google.com/advanced_search. [4] https://www.europeana.eu/portal/en. [5] history [internet]. [updated 2011 april 28]. mountain view (ca): creative commons; [cited 2019 june 7]. available from https://wiki.creativecommons.org/wiki/history [6] fallon j, senior policy advisor, europeana foundation. 2016 april 14. rightsstatements.org launches at dplafest 2016 in washington dc. europeana pro. available from https://pro.europeana.eu/post/rightsstatements-org-launches-at-dpla-fest-in-washington-dc [7] europeana foundation. 2015 feb 1. archives europeana rights statements – for reference only. the hague (netherlands): europeana foundation; [cited 2019 june 7]. available from https://pro.europeana.eu/files/europeana_professional/ipr/archivedrightsstatementspages-2.pdf [8] for many searches, a large number of hits locate items with a copyright not evaluated status. for example, as of july 2019, searching the word “language” on europeana’s search portal homepage at https://www.europeana.eu/portal/enresults in 274,594 hits. facets show that 118,094 of these have a rights status of “copyright not evaluated”. regarding overall coverage, an empty search of europeana results in 57,714,214 hits. facets show that 1,207,685 of these have a status of “copyright not evaluated”. [9] florida statutes section 1006.73 (2018). available from http://www.leg.state.fl.us/statutes/index.cfm?app_mode=display_statute&search_string=&url=1000-1099/1006/sections/1006.73.html. [10] http://palmm.digital.flvc.org. [11] henjum, e. introducing the florida heritage collection: a cooperative digital library initiative. florida libraries 2000 fall; 43:8. [12] clement gp. 1994. evolution of a species: science journals published on the internet. database 17:44-46. [13] gore e. 2018 april 18. dpla: a look back on the last 5 years. boston (ma): digital public library of america; [cited 2019 june 7]. available from https://dp.la/news/dpla-a-look-back-on-the-last-5-years [14] https://sunshinestatedigitalnetwork.wordpress.com/. [15] sunshine state digital network metadata working group. ssdn metadata participation guidelines version 1. sunshine state digital network; 2018 july 1 [cited 2019 june 7]. available from https://docs.google.com/document/d/1wmjl8tbf3-oa3x8aspgpnhdceggm7ppvy2xu9rxjphu/edit [16] https://rightsstatements.org/page/1.0/?language=en [17] see https://creativecommons.org/licenses/by-sa/4.0/legalcode and https://creativecommons.org/share-your-work/licensing-considerations/compatible-licenses . [18] how to edit/create ingest forms. [updated 2019 april 16]. charlottetown (pe, canada): islandora foundation; [cited 2019 june 7]. available from https://wiki.duraspace.org/pages/viewpage.action?pageid=69833497 [19] gore e, hansen d. 2016 dec 1. aserl webinar: “overview of rightsstatements.org” [videorecording]. atlanta (ga): association of southeastern research libraries; [cited 2019 june 7]. available from https://vimeo.com/193942186 [20] saa intellectual property working group. society of american archivists guide to implementing rights statements from rightsstatements.org. society of american archivists; 2016 dec [cited 2019 june 7]. available from https://www2.archivists.org/sites/all/files/rightsstatements_ipwg%20guidance.pdf [21] hirtle, p. copyright term and the public domain in the united states [internet]. [updated 2019 feb 7] ithaca (ny): cornell university; [cited 2019 june 7]. available from https://copyright.cornell.edu/publicdomain [22] https://palmm.digital.flvc.org. the palmm islandora site includes contributions from university of west florida, florida gulf coast university, university of florida, university of central florida, university of south florida, florida state university, florida international university, florida agricultural and mechanical university, university of north florida, florida atlantic university, and indian river state college. [23] https://broward.digital.flvc.org. [24] https://fgcu.digital.flvc.org about the authors wilhelmina randtke is the digital library services and open educational resources coordinator at the florida virtual campus’s florida academic library services cooperative. at the consortium, she supports digital publishing and digital library activities at member libraries across florida through training, technical support, facilitating connections between campuses, and assisting in prioritizing and routing issues for attention by technologists. randy fischer is software applications engineer in the florida virtual campus’s division of information technology. he supports the islandora digital library platform through maintaining code to streamline batch ingests of content to the florida islandora sites, and streamlining processes at scale in the islandora digital library platform. gail lewis is web applications engineer in the florida virtual campus’s division of information technology. she supports a variety of solr applications at the consortium including configuring solr and monitoring indexing issues for islandora and in the statewide discovery tool, mango. she maintains the table of contents display functionality in the islandora book module and maintains the islandora ip embargo module. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – out from behind the firewall: towards better library it communications mission editorial committee process and structure code4lib issue 21, 2013-07-15 out from behind the firewall: towards better library it communications traditionally, it departments lack a strong focus on communications and promotions. numerous exciting projects and services are created by library it departments and web development teams daily, but resources for promotion are typically unavailable or deemed low priority. this article examines it-specific communications within the library context, offers a model of user-focused communications useful to libraries of any size, and discusses university of toronto libraries’ information technology services department’s efforts to increase library technology communications. by lisa gayhart introduction from basic daily functions to bleeding edge developments, technology departments are important partners in the success of libraries, big and small. ongoing communications is an opportunity for technology departments to become more engaged with the larger organization. gone are the days where technology professionals could rely solely on technical expertise: today we must engage regularly with our communities or risk the perception of being out of touch and ineffective (trubitt, 2008). through better communications, library technology departments can move from the traditional image of the gatekeeper of services, to a more inclusive image of a partner in success, while improving both staff and client satisfaction. getting the conversation started communications is a conversation: a lively and inspired two-way discussion between multiple parties, in which we both have an intended purpose and desired outcome. instead of a one-way persuasion to buy a product or use a service, as seen in marketing, communications is an ongoing dialogue between the service provider and the user. both parties benefit from the reciprocal process: by moving together towards a common goal, the relationship is strengthened through information sharing and discussion. effective communication is central to the success of entire library technology departments as well as individual projects. in educause’s top ten it issues for 2012, the number one issue is “updating it professionals’ skills and roles to accommodate emerging technologies and changing it management and service delivery models.” this issue includes “strategic planning, relationship management, and communications skills” (grajek, 2012). non-technical skills are essential for technology teams as it departments rapidly grow and change within the innovative culture of libraries. although communications efforts are often led by a dedicated communications professional in large organizations, many libraries do not have the necessary resources to make this situation a reality. however, integrating communications practices into departmental planning processes and product development cycles can be led by anyone with an interest in better departmental communications. why take the time to communicate? libraries exist to help people find and make sense of information. we are known for our continued dedication to our users and hold excellent customer service in high regard. clear and targeted communications are an extension of this long-standing philosophy: good communication practices provide users with the information needed to efficiently solve problems and find information. in library it, we are in the business of providing support to users of library technology, developing new and improved services or products, and imagining innovative solutions. combining these offerings with user-focused communication ensures that technology departments are providing the most useful services for our target audience, at the appropriate time. in the face of a widespread lack of resources, both in terms of budget and qualified staff, raising the profile and communicating the value of library technology is more important than ever. it operations consume a large amount of budgetary resources and the onus is on the department to justify as well as maximize this expenditure, in a manner that resonates with the larger organization (mcshea, 2007). increasing departmental communications efforts can help library technology departments deliver meaningful information to the larger institution. research has shown that effective communication has an impact on perceived service level satisfaction (park et all., 2012). a measurable increase in satisfied users, clients, and staff is a benefit to technology departments, one that can easily be communicated to the larger library. reaping the rewards of communications efforts often takes time, since connections with other departments or libraries, communications staff, media, users, and other stakeholders must be forged and tended. the longer and more consistent the communications efforts are, the more the department will receive in return. the communications planning process although all communications efforts should be user-led and user-focused, it helps to have a plan to guide you along the path to partnership. this model focuses on an iterative cycle of open communications that allows discussion and refinement at any and every step, as opposed to a successive process comprised of many highly structured steps. communications cycle have a vision the first step in any great plan is to decide on a goal. why are we attempting to improve communications? once articulated, distill your vision down to a single statement and use this statement to inform all of your communications planning and actions. for example, “move from image of it as a ‘gatekeeper’ and towards image of a partner, with a continued focus on user-first service” is a defined goal that can frame a communications planning process. when brainstorming around goals and plans, take the time to talk with other communications resources within your library or organization. aligning departmental communications with the brand and messages of the central institution is critical to success. collaboration strengthens your communications process, integrates technology departments into a larger communications network, and provides consistency for your intended audience. know your environment the key to crafting effective communications is understanding your audience. who are your users? talk to them and discover what they need from library technology products, services, and staff. your audience may be comprised of a variety of groups – students, faculty, community members, staff, fellow team members, partner libraries – with different expectations and needs. understanding your users doesn’t have to be a large and complicated process, bogged down by demographic analysis and research. people associated with libraries are often very interested in library technology and welcome the opportunity to engage on this topic. getting to know your audience begins with reaching out to your users, both physically and virtually. hold open houses and information sessions for an opportunity for education and discussion. improve two-way dialogue between users and it by sending a departmental representative to staff and public meetings, events, conferences, and in the professional literature. offer multiple opportunities for feedback, such as online forms, published email addresses and phone numbers, forums, and social media conversations. whenever possible, be an active part of your community. when creating a culture of feedback, ensure there is a process in place for responding to all users. all feedback requires a response, even if the response is a simple acknowledgement of receipt. people require acknowledgement of their ideas and suggestions, especially when they have taken extra time to participate in your conversation. the feedback and response loop is an essential component to improving communications. if resources are unavailable to respond to feedback in a timely manner, minimize opportunities for feedback until resources are increased. make a plan at this stage, you have a goal and you know your audience. now is the time to create a fully integrated and actionable communications plan, tailored to the needs of a specific product or service. whether communicating on behalf of an entire library, department, project, service, or event, a plan is useful in targeting specific audiences and staying on track with pre-established goals. the most effective it communications are integrated directly into the project or development lifecycle, starting at initiation. too often, we think about communicating the message after product development is complete, when the product or service is ready for release to users. in the development example, communications staff work together to create the plan by consulting all team members on the goals of the project, intended audience, timelines, content, testing, and roll out plans. it is very important to ask for feedback from the development team and revise the communications plan; speak directly with clients and potential users; and speak with other communications resources within the larger library. assessment is an essential step in any plan: casting a critical eye on the planning and implementation process reveals areas of improvement for the next project. when establishing the communications plan, set specific indicators of performance within the assessment phase. these are useful in gauging the success of the communications plan. for example, a performance indicator for our partnership goal could be “respond to 97% of feedback within 48 hours of receipt. responses may be full answers to questions or suggestions, acknowledgement of receipt, and/or connecting the user with the appropriate team resource.” in the assessment period, the communications team member would then tally all incoming feedback and determine the percentage of timely responses. outcomes of communications are often difficult to quantify, therefore ascribing a tangible measure of performance greatly assists in communicating the value of communications efforts to your department and the larger organization. communications plan outline phase what this means be at the table take part in the discussion from project initiation; meet with various stakeholders to gain a full understanding of the project and the desired outcomes know your audience understand your users’ needs and wants; get to know the language of your audience and their level of technical expertise set your goal define the desired outcome for the communications plan; align these targets with the project lead’s goals build key messages solidify two to three statements that accurately frame your goal and describe user actions (how do i participate?) and gains (what’s in it for me?) select objective and tactics outline the path to your goals; clearly articulate how to achieve these goals set timelines assign specific milestones and end dates produce collateral create the communications materials, such as websites, web and text copy, brochures, web and/or radio ads, editorials, interviews, and town halls perform rollout release the collateral to the intended audience receive feedback stay open to feedback at all times; continually acknowledge feedback perform assessment review the outcomes of the communications in relation to performance indicators; recommend improvements for next cycle; inform appropriate parties of the final outcome restart the cycle start the next communications planning process work the plan in larger libraries, a communications person may drive the communications process, but the entire department and/or specific project team must be aware the course of action and all outgoing messages. essential to any communications plan are the key messages. key messages relay the most important information about your project or service and include reasons why the user should find this information important and what they can do to act on the information. take time to build these messages, as you will use them continually and all communications will be based on these messages. over the course of the communications, continually reinforce these messages through the collateral, general conversation, conversations with team, and other appropriate avenues. repetition is necessary: people are inundated with an enormous amount of information all day, every day. ensure that your message is clear, informative, and actionable. once solidified, the team and the department should be very familiar with the key messages. everyone on staff is a brand ambassador: users see one united front and not the different levels of staff or responsibilities. tips for crafting your communications conversation: use language and positioning that enables the user – ‘help shape the future of your campus at this open info session.’ frame the conversation around your key messages tech translation – discover how much your target audience knows about your product, team, or service and tailor your communications as needed. speaking to clients who are developers requires different language than speaking with students who are end users of your product. avoid jargon, acronyms, and very technical language (unless appropriate for your audience) draft communications in plain language: direct, simple, and short solicit feedback at every opportunity use feedback to action evidenced-based, iterative design and content modifications focus on benefits of your product or service think about benefits from the user’s perspective – when crafting messages, ask “what’s in it for me?” or “why would i care?” to get a sense of the impact on the user include actionable statements to ensure the user knows how to respond to your messages and why the messages are important review the review and assessment stage is too often seen as an additional step to take, and only if time permits. when building a foundation for effective communications within a library it department, reviewing the effect of outgoing communications on your audience, internal team, and overall institution is critical. the conversation evolves and strengthens over time, as communication becomes part of the ongoing process of developing projects or promoting your department. in a foundational stage, we should strive towards continual improvement with every iteration of a plan, product, or conversation. helpful questions to ask during the review phase: how was the planning and implementing process in terms of workload on it staff? did we achieve the desired results? compare your results to the performance indicators set in earlier stages. if desired results were not achieved, why and how can we do it differently next time? what do we need to make a stronger presence/team/product? assessment may reveal a need for capacity building, such as staff training. communications training is very useful, especially for it professionals who often struggle with translating technical language for users and creating plain language messaging. other areas of development may include various soft skills essential for today’s it professionals: displaying empathy and demonstrating listening skills accurately explaining technical details in clients’ language setting realistic expectations about system performance and its technological limitations educating clients so they can make more informed technological decisions explaining how various technical options may work regularly sensing clients’ unmet expectations (park et all., 2012) additionally, review and assessment often reveals a need for strengthening the communications presence within the library. historically, libraries have not been strong at communicating with user groups, so these networks may not be established. finding and cultivating leads and contacts, working with the media, establishing basic communications tools such as media release templates will take time to create and refine. case study: realigning library technology communications for web development projects at utl at the university of toronto libraries, the information technology services (its) department faces many of the opportunities and challenges described above regarding improving departmental communications. currently, we are realigning our internal and external communications to support the partnership model of communicating library technology services. its partners with the university of toronto, university of toronto libraries, and affiliates in a variety of ways, including the provision of traditional it services, library-specific technology, and web development. the digital library and web services team within its focuses on the development of scholarly digital collections (http://onesearch.library.utoronto.ca/digital-collections) for the digital humanities community, specializing in the areas of digital preservation, data curation, visualization, accessibility, and responsive web design. the newly integrated communications practices of this area are the focus of this case study example. in addition to a variety of talented technical staff, team resources also include one full time digital communications team member from its. digital project development occurs on a continuous improvement model: a cyclical project management framework is used, as opposed to a traditional waterfall model of pre-defined, successive stages. an iterative process of development allows for staff to concurrently contribute to different areas of the project, bringing their expertise to a wide range of processes. this open ideology promotes continuous innovation and new growth in unexpected areas. excellent communications is essential to this development process. research shows that effective communications around it projects is essential to service quality, client satisfaction, and producing a quality product (park et al., 2012). integrating communications into the end-to-end development process requires dedicated staff to ensure consistent and professional strategy, messaging, execution, and assessment. the communications resource is brought in at project initiation, to ensure they have the same level of project knowledge as other team members for the duration of the project, but also to contribute to planning and strategy. each project deals with different content and all team members must be familiar with the context of the project. collaboratively, a communications plan including two to three key messages are established during the project initiation. throughout the development process, all team members are project ambassadors: willing and able to talk intelligently about the project, why it is important, and the impact the final product will have on the digital humanities community. partnership with clients is an important aspect of every development project. regular and effective communication fosters a culture of trust, resulting in heightened client satisfaction with both service level and the end product (park et al., 2012). two-way communication with clients from project initiation is encouraged: clients are supplied with the project team’s contact information; information on how to submit issues, bugs, and recommended improvements; and a future meeting schedule is solidified and distributed. the team has integrated better tracking and interaction tools over the past year. tools used to increase intra-team and intra-departmental communications include: issue tracking and project management software: atlassian suite of jira, confluence, greenhopper staff wiki (currently in mediawiki, with migration plans for early summer 2013 to confluence) wireframing and mock-ups tools in-house chat server open office atmosphere: team members meet spontaneously to share ideas and keep the development process moving forward ongoing support of open source communities and tools weekly open forum meetings, where staff are encouraged to drop in and share current projects, seek collaborators, ask questions, give updates, discuss upcoming changes, etc. these tools allow our team to communicate more effectively with each other and to provide better support to our clients. all tools are web-based, allowing for instant communication from any location. collaboration spaces, such as a project-specific confluence space, are established during the project initiation: team members immediately begin working together to share ideas and resources related to the project. online project spaces also create a living transcription of the project life cycle, which is valuable during the assessment stage after the product is launched. during the development process, all team members are encouraged to use these online spaces but in-person brainstorming and team updates are preferred. weekly updates are given to the larger its group, to encourage collaboration across the various groups within the department. larger development milestones are demonstrated at these meetings, where feedback can be gathered to refine the product before presenting to the client for review. the communications team member facilitates the project’s content management over the development period. while sites are in development, all site content is prepared in conjunction with the client, including page content, terms of use, copyright statements, contact forms. at this point the communications person recruits participants for product testing and feedback. information gathered from these sessions is given to the development team, who work to prioritize feedback and incorporate testing results into the final product. as the project is in the latter half of the development cycle, external communications are produced in conjunction with the project team and the client. following the communications plan set at project initiation, collateral such as web copy, web banner images, email copy, printed material, and media releases are created and distributed. its is increasing opportunities for feedback in all outgoing communications. any collateral created for development projects includes many opportunities for users to get involved in the process, such as opportunities to test new products, email comments, report bugs or suggestions for improvements, speak directly with the development team, or attend focus groups or information sessions. as externally focussed communications efforts are increasing, we are raising the departmental profile within the larger library system. the emphasis remains on understanding our users and designing targeted communications materials based on this data. various outreach tactics have been employed in an effort to better understand our users. an event showcasing technology services was held to explain our role among the larger library system and offer a venue for discussion and feedback. its also joined other outreach and feedback events as a partner, using these events as opportunities to talk to library staff and users, get feedback, and promote upcoming projects. an increased attendance a departmental meetings, strategic planning sessions, and town halls are additional opportunities to speak with users and simultaneously increase awareness about its projects and services. our own review of the newly actioned communications efforts revealed areas in which improvement is needed. accessibility guidelines and standards are lacking for written web content and outgoing communications messaging. communications must be developed with accessibility in mind, such as writing accessible copy for communications collateral, both printed and digital. the web style guide is scheduled for update and will include sections addressing accessible content creation and distribution. accessible content creation training will be offered to all website content managers in the summer of 2013. in the future, accessibility measures will be a standard aspect of the departmental communications package. an additional area of improvement is a heightened focus on better integration with the larger library and academic community, including increasing collaboration with other departments in the library, sharing best practices and applicable resources, and establishing deeper communications networks. finally, its does not offer an online service catalogue or knowledge base for users. this is an area we aim to develop in the future. offering an online service catalogue allows users to help themselves, when convenient, which leads to an increase in service level (alberts, 2012). offering a space for users to submit questions and answers will foster a process of self-directed teaching and learning. we also hope users will share examples of good service experiences from its with others, which improves the image of its within the library. conclusion library technology departments and teams can undertake improving communications practices at any time. begin with lower demand tasks, such as talking to users or attending library events as a technology representative. as a foundation for communications builds, include larger demand tasks such as media involvement and longer term user outreach. over time, technology departments can move towards a true partnership with the larger library and library users. references alberts, randall, and michael cato. 2012. is your it organization a marketing one? grajek s and pirani j. june 2012. top ten it issues 2012. available from: http://www.educause.edu/ero/article/top-ten-it-issues-2012 mcshea, michael. 2007. communicating it’s value in a modern business climate. it professional 9 (1) (2007 jan.-feb.): 42-5. park, jungi, jungwoo lee, hyejung lee, and duane truex. 2012. exploring the impact of communication effectiveness on service quality, trust and relationship commitment in it services. international journal of information management 32 (5) (201210): 459-68. trubitt, lisa, mur muchane 2008. in plain english, please: effective it communications. educause quarterly 31 (2) (2008): 62-65. available from: http://www.educause.edu/ero/article/plain-english-please-effective-it-communications about the author lisa gayhart is the digital communications services librarian in information technology services at the university of toronto libraries. as the dedicated communications team member in its, her number one priority is to move library it from the sidelines to the spotlight through structured and actionable communications. besides leading communications efforts for the library it group, lisa works with digital usage analytics, accessibility measures and guidelines, and project and process management. to discuss this article or current talks, provide feedback, or obtain more information on diy communications for library technology, contact lisa at lisa.gayhart@utoronto.ca. subscribe to comments: for this article | for all articles 2 responses to "out from behind the firewall: towards better library it communications" please leave a response below: jonathan rochkind, 2013-07-31 hi, you say “all tools are web-based” — does that include for the “weekly open forum meetings”, or are those done in person? if you do those weekly open forum meetings online, can you let us know what tools you use for that? if you do them only in person — have there been challenges with remotely located colleagues who don’t have the opportunity to attend those weekly open forums, so end up not as included in communications? lisa gayhart, 2013-08-01 hi jonathan, our weekly meetings take place in person. if someone is off-site, they have the option to dial in and join the group discussion, or get a recap from another team member at a later date. we hold the meetings in the middle of the week at the same time each week, which seems to catch most people who may be working remotely. often, if someone knows they will be absent, they will pass notes along in advance and we will review them as a group. at this point, we don’t have any staff members that work remotely 100% of the time. hope that clarifies things, lisa gayhart leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – autoload: a pipeline for expanding the holdings of an institutional repository enabled by resourcesync mission editorial committee process and structure code4lib issue 36, 2017-04-20 autoload: a pipeline for expanding the holdings of an institutional repository enabled by resourcesync providing local access to locally produced content is a primary goal of the institutional repository (ir). guidelines, requirements, and workflows are among the ways in which institutions attempt to ensure this content is deposited and preserved, but some content is always missed. at los alamos national laboratory, the library implemented a service called lanl research online (laro), to provide public access to a collection of publicly shareable lanl researcher publications authored between 2006 and 2016. laro exposed the fact that we have full text for only about 10% of eligible publications for this time period, despite a review and release requirement that ought to have resulted in a much higher deposition rate. this discovery motivated a new effort to discover and add more full text content to laro. autoload attempts to locate and harvest items that were not deposited locally, but for which archivable copies exist. here we describe the autoload pipeline prototype and how it aggregates and utilizes web services including crossref, sherpa/romeo, and oadoi as it attempts to retrieve archivable copies of resources. autoload employs a bootstrapping mechanism based on the resourcesync standard, a niso standard for resource replication and synchronization. we implemented support for resourcesync atop the laro solr index, which exposes metadata contained in the local ir. this allowed us to utilize resourcesync without modifying our ir. we close with a brief discussion of other uses we envision for our resourcesync-solr implementation, and describe how a new effort called signposting can replace cumbersome screen scraping with a robust autodiscovery path to content which leverages web protocols. by james powell, martin klein, and herbert van de sompel introduction the institutional repository (ir) is an ecosystem of open source software components that collectively support archiving, preservation, discovery, and dissemination of content. an ir aims to house the institutional output of an organization. but they often grapple with routing content into the archive in a timely, consistent, and efficient manner. this paper describes a work in progress effort to automatically discover and expand the holdings of an institutional repository. this pilot project uses resourcesync [1] to initialize a discovery and harvesting pipeline that programmatically evaluates discovered content for relevance, associates bibliographic and publisher licensing metadata with it, and aggregates the results in a graph store. the research library at los alamos national laboratory uses solr [2] as the foundation for web based applications for end users. metadata is retrieved from the adore repository [3], indexed, and then made available through java web applications. one of these web applications is called lanl research online (laro). it provides search and faceted browse access to lanl authored publications that have been approved for general release. lanl has long standing requirements for review and approval prior to publication or presentation of researcher content. there is now a web workflow that makes this requirement much easier to adhere to and this will eventually improve coverage. but to date there’s been no effort to retroactively populate the repository with older published content. we discovered the extent of the coverage problem as we were laying the groundwork to publicize laro content to search engines. we were planning to use resourcesync to provide an iterable list of laro publications and their metadata. resourcesync is a niso standard designed to facilitate content replication and synchronization. one use case for resourcesync is to make a collection of web resources accessible and available for discovery. laro contains metadata for about 25,415 publications and we thought a reasonable percentage of these metadata records would reference local copies of the full text item that they described. our plan was to use resourcesync in conjunction with solr to expose this collection so that it could be harvested and indexed. when we discovered that there were only 1023 full text publications in laro, we decided that there was not yet enough content to justify marketing it widely. our new goal was to increase the full text in laro and here again we thought that resourcesync might be an excellent tool for bootstrapping this effort. to explain why, we will briefly review the features and capabilities of this standard. resourcesync resourcesync is a niso standard [4] approved in 2014, and updated in 2017, as a solution to the problem of synchronizing web resources between sites. it is specifically concerned with exposing the locations of the web resources and their descriptions. to accomplish this, resourcesync defines a framework of synchronization capabilities. a content provider is referred to as the source, while a site that intends to replicate the source’s content is called a destination. resourcesync defines modular capabilities which can be implemented by the source to enable a destination to retrieve and verify the integrity of the source’s content as it changes over time. capabilities supported by the resourcesync framework include “pull” and “push” baseline synchronization and incremental synchronization, bulk synchronization, and audit/verification, for metadata and content. resourcesync specifies an xml vocabulary for describing resourcesync framework capabilities implemented by the source, and for describing the resources and metadata available for replication. this adds a few element and attribute extensions to the sitemap protocol [5]. sitemap is an xml specification for describing and itemizing the various pieces of a web site, for discovery, navigation, and indexing. resourcesync builds on sitemap [6] with elements that make it possible for a destination to make a copy of a site’s content, find the associated metadata for that content, find information about changes that have occurred at a source over time, and data that can be used to verify the integrity of the destination’s copy of a source. resourcesync is modular [7], which means that a source may be selective about which parts of the standard it implements. a simple approach is to enable destinations to stay in sync via pull requests. the source publishes a baseline list of resources (a resource list) as well as lists of updates as the site changes over time (change lists). but a source may elect to use other parts of the standard. for example, if the source collection changes frequently, then it could provide push change notifications to decrease synchronization latency. or if the source wants to support bulk replication for a large amount of content, then it could publish resource dumps and change dumps, which are collections of content and their metadata bundled together and compressed for easier transfer to a destination. any type of resource is supported by the framework as long as it can be accessed via a uri. a capability list describes which parts of the resourcesync framework a source implements. in our case, laro functions as a source, and the autoload pipeline is a destination. laro supplies a resource list for baseline synchronization, and change lists which contain new entries when new items are indexed in laro. so the autoload process starts by retrieving a resource list or change list. resourcesync and solr many irs, including fedora/hydra and duraspace/dspace use solr. solr is an open source application that provides indexing and searching functionality which utilizes the apache lucene search library. lucene is a mature platform for textual search, and solr adds simple forms-based support for indexing textual elements of structured content, such as metadata. solr can be used to provide a variety of field-specific, cross-element, value-bounded, and faceted queries. this functionality is available not only to end users but also to applications through a rest interface. at the research library, our metadata records reside in our adore repository. adore uses the mpeg-21 digital item declaration language (didl) as a container for metadata and digital objects. the didl is referenceable via a digital item identifier (did). in this way, adore can accommodate any type of digital objects and any type of metadata. we store the marc xml file inside the didl. for many end-user applications, we pull this marc xml metadata from the repository and index this data with solr. although laro is technically our resourcesync source, we actually implemented the source as an intermediary, external to laro. it queries laro’s solr instance to generate resourcesync lists. we did this for practical reasons – we did not want to modify the repository codebase or any existing applications in order to support resourcesync. laro is a virtual collection which has no explicit identification within the repository. metadata resides in the adore repository and it is extracted according to certain selection criteria, indexed with solr and made available under the laro banner. so effectively, the solr index that supports laro is the only record of membership for the laro collection. solr provides a convenient interface upon which to build resourcesync capabilities for a couple of reasons. first, it has two characteristics which are essential for building “pull” synchronization components – unique web referenceable identifiers, and timestamps. timestamps are necessary in order to provide temporal details about the assembled collection of uris in the resource list and the change lists to destinations. second, solr provides a rest interface for resource uris which the source intermediary can query, thus avoiding the need to build new hooks into the repository or touch any existing code associated with the repository or laro. from resync import resource,resourcelist rl = resourcelist() thismoment = str(datetime.now()) resourcelist_timestamp = thismoment rl.up = "http://example.com/dataset1/capabilitylist.xml" rl.md_until = resourcelist_timestamp thisresource = resource(uri=linkval, lastmod=timestamp) thisresource.link_set(rel="describedby", modified=timestamp, href=recmetadatauri) thisresourcerecip = resource(uri=recmetadatauri, lastmod=timestamp) thisresourcerecip.link_set(rel="describes", href=linkval, modified=timestamp) thisresourcerecip.link_set(rel="profile", href="http://www.w3.org/2001/xmlschema-instance") rl.add([thisresource, thisresourcerecip]) listing 1 using the resync library to create resource list entries overview of the solrsync application various software libraries exist for implementing resourcesync capabilities including the python resync library [8] for building and testing various resourcesync framework capabilities, and rspub-core [9] which is a comprehensive, extensible framework for adding resourcesync capabilities to an existing web application, repository, or database via plug-ins. there is also a library for push notifications called resourcesync_push [10] (based on pubsubhubbub). our implementation, which we refer to as solrsync, was implemented in python using the resync library, atop the django web framework [11]. our goal was to implement resource lists and change lists as dynamic elements constructed from the laro solr index. for this test collection which contains around 25,000 items, this approach seems to work quite well. the solrsync prototype’s codebase is quite modest and demonstrates that resourcesync is easy to support with a very small amount of effort (see listing 1). no existing systems had to be modified to support resourcesync, demonstrating that resourcesync capabilities can be easily added to an existing ir ecosystem. here are the steps that are performed when a resourcelist.xml is requested from solrsync: when a request is made for resourcelist.xml, the django solrsync application resolves this via a pattern for resourcelist.xml in its urls.py file this triggers the execution of a method which queries solr for a set of results up to a given date it parses the solr results set and locates a resumption token which occurs as the text node of a str element with an attribute name that contains the value “nextcursormark” it iterates through the doc element blocks in the solr results and for each, it locates the str element that has a name attribute with the value “recid” (this is specific to our instance of solr) it uses a base uri value to create a direct reference to this item’s metadata in the adore repository it constructs a resync resource item based on this data it examines each solr result entry for a reference to a full text report or other full text content, and if it exists, adds this to the resource list as well (see below example). these are items that the autoload pipeline will skip as it consumes the resource list. it repeats this process from step 2 onward until it exhausts the solr result set finally, solrsync outputs xml for the resource list and returns it to the requester there are several important considerations for building a resource list or change list from a solr results set. a solr result set includes an important piece of information that becomes crucial when dealing with large numbers of items: the numfound attribute. the sitemap protocol limits the number of uris it can reference to 50,000. if numfound exceeds this value directly, or when combined with references to both full text and metadata, then it is necessary to generate multiple resource list or change list files. these are itemized in an index referenced from the capability list, which could also be dynamically generated. if this information cannot be computed, then it is not possible to generate resource lists and change lists that conform to the sitemap protocol. the solrsync approach will not work if the solr index does not include a meaningful timestamp per item indexed. this value must be searchable, and it must be returned as part of the result set. this approach will also fail if the solr index does not contain and does not return the identifiers for indexed items. this is generally not a problem with bibliographic metadata indexes since the point of the solr index is to facilitate content discovery. listing 2 provides an example solr query parameter list for solrsync: q=title:* and timestamp:[* to 2017-02-08t21:18:33z] &wt=xml&indent=true&sort=recid+asc&cursormark=* listing 2 solr query example this query, which is used to generate a resource list, retrieves all records that have a timestamp up to and including the value specified, returns the results as xml, sorts on the unique identifier “recid”, and indicates that the result set should be paged using cursormark. the timestamp parameter for a solr query for a change list is slightly different than for a resource list. in addition to specifying a date range, it uses curly brackets to indicate that the result set should contain items were added or changed since the value specified untill the present moment, as illustrated in listing 3: timestamp:{2016-08-20t16:33:33z to now} listing 3 alternate temporal parameter for solr query to generate a change list solrsync uses the python requests library to submit a query to solr via http, and then uses the python library xmldict to load and parse [12] the solr result set (listing 4). the solr results are transformed into a resourcesync resource list or change list. since most of the laro collection is represented only by metadata, the laro resource list contains mostly metadata uris, which point back to the adore repository (listing 6). when there is full text available, this information is included in the laro solr index. the index will contain a reference to a permalink, a shareable and bookmarkable url that can be derefenced to access full text content on a local content server. when full text does exist for a laro record, two resourcesync loc elements are included for this solr result entry. an example is provided in listing 5. the resourcesync rs:ln is a child element of the loc element that contains a uri. this uri has a relationship with the resource indicated by the rel attribute. if the rel attribute value is “describes” then the rs:ln uri is for the content. if the rel attribute value is “describedby” then the rs:ln uri is a pointer to metadata about the content. solrsync generates a resource list or change list using the resync python library, which builds a python resource list object in memory, and then outputs it as xml. the resync library can be used to generate other documents for resourcesync such as a capability list, which contains links to a source’s resource lists and changelists. <response> <lst name="responseheader"> <int name="status">0</int> <int name="qtime">83</int> <lst name="params"> <str name="sort">recid asc</str> <str name="indent">true</str> <str name="q">title:* and timestamp:[* to 2016-07-05t23:23:54z]</str> <str name="wt">xml</str> <str name="rows">25500</str> </lst> </lst> <result name="response" numfound="25415" start="0"> ... <doc> <str name="recid">info:lanl-repo/lareport/la-ur-14-29153</str> <str name="dataset">rassti</str> <str name="displayname"> guzik, joyce ann ; bradley, paul andrew ; jackiewicz, jason ; molenda-zakowicz, joanna ; uytterhoeven, katrien ; et al. </str> <str name="displaytitle"> the occurrence of non-pulsating stars in the gamma dor and delta sct pulsation instability regions: results from kepler quarter 14-17 data </str> <arr name="publication"> <str>astronomical review</str> </arr> <arr name="publication_browse"> <str>astronomical review</str> </arr> <arr name="publication_rbrowse"> <str>zhgilmlnrxzo ivervd</str> </arr> <str name="doi">10.1080/21672857.2015.1023120</str> <str name="displaysource"> astronomical review ; vol.9, iss.1, p.41-65, jan. 2014 </str> <str name="openaccess">true</str> <str name="afv_flag">true</str> <str name="embargodate">2015-04-21</str> <arr name="url"> <str>url accepted manuscript | http://arxiv.org/abs/1403.8013</str> <str> accepted manuscript | http://permalink.lanl.gov/object/ampdf?what=info:lanl-repo/lareport/la-ur-14-29153 </str> </arr> <long name="_version_">1522819486080565248</long> <date name="timestamp">2016-01-08t17:23:52.373z</date> </doc> </result> </response> listing 4 solr results excerpt <url> <loc>http://lastage.lanl.gov:8080/adore-disseminator/service?url_ver=z39.88-2004&amp;amp;rft_id=info:lanl-repo/lareport/la-ur-14-29153&amp;amp;svc_id=info:lanl-repo/svc/xml.format.full</loc> <lastmod>2016-01-08t17:23:52.373000z</lastmod> <rs:ln href=" http://permalink.lanl.gov/object/ampdf?what=info:lanl-repo/lareport/la-ur-14-29153" modified="2016-01-08t17:23:52.373z" rel="describes" /> <rs:ln href="http://www.w3.org/2001/xmlschema-instance" rel="profile" /> </url> <url> <loc> http://permalink.lanl.gov/object/ampdf?what=info:lanl-repo/lareport/la-ur-14-29153</loc> <lastmod>2016-01-08t17:23:52.373000z</lastmod> <rs:ln href="http://lastage.lanl.gov:8080/adore-disseminator/service?url_ver=z39.88-2004&amp;amp;rft_id=info:lanl-repo/lareport/la-ur-14-29153&amp;amp;svc_id=info:lanl-repo/svc/xml.format.full" modified="2016-01-08t17:23:52.373z" rel="describedby" /> </url> listing 5 resource list entry corresponding with the solr result from listing 4 <record> <title> the occurrence of non-pulsating stars in the gamma dor and delta sct pulsation instability regions: results from kepler quarter 14-17 data </title> <authors> guzik, joyce ann (xtd-nta: xtd nuclear threat assessment) (joy@lanl.gov) ; bradley, paul andrew (xcp-6: plasma theory and applications) (pbradley@lanl.gov) ; jackiewicz, jason (new mexico state u., las cruces, nm) ; molenda-zakowicz, joanna (uniwersytet wroclawski, wroclaw, poland) ; uytterhoeven, katrien (instituto de astrofisica de canarias, tenerife, spain) ; kinemuchi, karen (apache point observatory, sunspot, nm) </authors> <institutions> xtd-nta: xtd nuclear threat assessment ; xcp-6: plasma theory and applications ; new mexico state u., las cruces, nm ; uniwersytet wroclawski, wroclaw, poland ; instituto de astrofisica de canarias, tenerife, spain ; apache point observatory, sunspot, nm </institutions> <source>astronomical review ; vol.9, iss.1, p.41-65, jan. 2014</source> <pubdate>2015-02-09</pubdate> <osti_subjects>astronomy &amp;amp; astrophysics(79)</osti_subjects> <keywords>variable stars ; stellar pulsation ; kepler spacecraft</keywords> <document_type>journal article</document_type> <report_number>la-ur-14-29153</report_number> <pages>29 p.</pages> <sponsor>nasa</sponsor> <openaccess>true</openaccess> <issn>2167-2857</issn> <doi>10.1080/21672857.2015.1023120</doi> <lapr_id>lapr-2014-020948</lapr_id> <accepted_manuscript>true</accepted_manuscript> <program_code>5j880a-rkwc; 5j880a-rmdb</program_code> <restrictions> this report has access restrictions. see los alamos authors database record for details. </restrictions> <repository_id>info:lanl-repo/lareport/la-ur-14-29153</repository_id> <dataset>rassti</dataset> </record> listing 6 adore disseminator xml metadata serialization for uri in <loc> element from listing 5 the autoload pipeline the goal of the autoload pipeline (figure 1) was to establish a self-sustaining pipeline that would monitor laro and update it with full text content. feeding the pipeline with resourcesync would allow it to function as a “machine that would go of itself.” the overall effort is similar in its goals to that described by flynn et al, 2013 [13]. the authors describe a two phase approach to populating an ir – first harvesting, transforming and ingesting metadata, and second, retrieving licensing information from sherpa/romeo [14] to aide staff efforts to manually populate their ir with full text content. their goal was to make populating the ir less staff intensive. our effort diverges markedly in many respects, and these differences stem from our effort to use the resourcesync standard together with various web services to develop a self-sustaining automated pipeline to populate our ir. functioning as a resourcesync destination, the first thing the autoload pipeline does is request a resource list to establish a baseline, or a changelist upon subsequent executions, which will provide information about changes that occurred since the baseline was established. as noted above, these are xml documents which itemize a list of resources available via http. autoload parses this document using the python xmltodict library. we are interested in the items that lack full text, so when a resource or change list loc element has only a metadata link, autoload injects this into the pipeline. when this file contains <loc> elements that have associated full text content (reciprocal describes and describedby relationships), they are skipped. the pipeline formulates a request to the adore repository disseminator for the metadata pointed to by that uri. in the adore architecture, disseminators have the ability to generate different serializations of the archived metadata. autoload requests a basic xml serialization which includes a reference to the object’s doi, if known. listing 6 illustrates an xml metadata record from the repository. this doi (contained within the <doi> element) is used in subsequent steps in the pipeline. only items that have associated dois, and which currently lack full text, are processed by the autoload pipeline. figure 1 autoload pipeline using crossref, oadoi, sherpa/romeo in the autoload pipeline an initial execution of the autoload pipeline discovered 21,798 unique dois, most of which lacked full text. with this list, autoload used several approaches in succession in an effort to discover and retrieve full text of the publication or related content. the pipeline utilizes data from crossref [15], oadoi [16], sherpa/romeo, and it also attempts direct retrieval of content via doi resolution and screen scraping. sometimes it discovers related content, rather than the full text of the publication. these can include supplemental figures, charts, or appendices. this was not expected, but we do intend to archive these items as well. this finding reinforces the particular order in which steps are performed in the autoload pipeline as compared to other similar efforts. these materials would probably not have been discovered if the sherpa/romeo license categories were applied prior to the discovery phase, rather than afterward. on the other hand it does mean that some content may require human review. autoload first uses the doi to retrieve crossref metadata for the content. crossref returns json-encoded metadata that describes the content. this metadata includes various bibliographic metadata, as well as links to full text content if known, although in practice this element is rarely populated. crossref metadata provides semantic information about the object itself. it also provides publisher information. at this point in the process, the autoload pipeline uses values from crossref, specifically the issn, to query sherpa/romeo to retrieve license information that pertains to the content. the next step in the autoload pipeline is to resolve the doi to a uri. one technique it uses is to request the doi via the oadoi web service (listing 7). oadoi returns various pieces of information about how to access the object pointed to by a doi, including, if available, a link to a uri that has been vetted and determined to be a free_fulltext_uri. in practice, this uri points to a landing page intended for humans, so it still needs to be parsed. we once again apply the simple recursive anchor parser to this page, which as before, will retrieve and parse a target page if “pdf” occurs in the link text, or retrieve and store the target file if “pdf” occurs in the link target. we found oadoi returned free_fulltext_url entries for over 5300 items, about 25% of the dois in laro. http://api.oadoi.org/v1/publication/doi/10.1080/21672857.2015.1023120 { "results": [ { "_title": "the occurrence of non-pulsating stars in the \u03b3 dor and \u03b4 sct pulsation instability regions: results from kepler quarter 14\u201317 data ", "doi": "10.1080/21672857.2015.1023120", "doi_resolver": "crossref", "evidence": "oa repository (via base-search.net oa url)", "free_fulltext_url": "http://arxiv.org/abs/1502.00175", "is_boai_license": false, "is_free_to_read": true, "is_subscription_journal": true, "license": null, "oa_color": "green", "url": "http://doi.org/10.1080/21672857.2015.1023120" } ] } listing 7 a response from oadoi identifying access points for a paper screen scraping is the autoload pipeline’s technique of last resort but unfortunately for now, it remains a necessary evil. screen scraping is used for oadoi responses that contained a free_fulltext_url entry which points at an html document rather than a pdf file. it is also used for items for which no full text links were found in crossref or oadoi. in the latter instances, the doi is resolved to a uri, which is then followed using a simple recursive routine. this process uses beautifulsoup, a python screen scraping library, to parse html content. it iterates through this list, inspecting each link. it examines both the anchor text and the anchor target uri, in an attempt to locate references to pdf. if the anchor text contains a reference to “pdf” but the anchor target does not, then it retrieves that page and repeats the process. if the anchor target contains the string “pdf”, then it retrieve the file it points to and store it locally. as mentioned earlier, this strategy is effective at not only retrieving the primary pdf document if available, but also auxiliary information that an author may have provided, if it is in pdf format. since we had used microsoft academic [17] in a previous project with similar aims, we also evaluated its successor, microsoft cognitive services for use in the autoload pipeline. the results were tantalizing, yet frustrating. bing, which appears to be the source of some of the service’s data, aggregates links to full text when it is able to disambiguate a title (see example in listing 8). this means that a title search may return more than one full text link for a given paper. however the service had several limitations. dois are not indexed so they are not directly searchable. the doi is considered extended metadata, so it can only be retrieved once a search has been performed on an indexed field such as author or title. furthermore, partial title searches yielded very poor results for the laro collection – in fact, a partial title search for a random sample of laro items yielded zero results. it may be that the content we were searching for had yet to be indexed by microsoft cs or bing. another consideration is that it is a fee-based service. while we plan to keep an eye on this service for future projects, because of these limitations we abandoned plans to incorporate it into the autoload pipeline. https://api.projectoxford.ai/academic/v1.0/evaluate?expr=ti=’graphs in libraries’…&model=latest&count=10&offset=0&attributes=id,e { "expr": "ti='graphs in libraries'...", "entities": [ { "e": "{ \"dn\":\"graphs in libraries: a primer\", \"d\":\"....\", \"s\":[ { \"ty\":3, \"u\":\"http://ejournals.bc.edu/ojs/index.php/ital/article/download/1867/1705\" }, { \"ty\":1, \"u\":\"https://www.questia.com/library/journal/1g1-272739694/graphs-in-libraries-a-primer\" }, { \"ty\":1, \"u\":\"http://www.researchgate.net/profile/james_powell8/publication/228776771_graphs_in_libraries_a_primer/links/00b4953500690bfd15000000.pdf\" }, { \"ty\":1, \"u\":\"http://ejournals.bc.edu/ojs/index.php/ital/article/view/1867\" } ], \"vfn\":\"information technology and libraries\", \"v\":30, \"i\":4, \"fp\":157, \"lp\":169, \"doi\":\"10.6017/ital.v30i4.1867\" }" } ] } listing 8 a request and response example for microsoft cognitive services when candidate full text is discovered for a doi, the next step is to analyze this content and assign a match score as compared to its metadata. for this step, autoload uses the previously retrieved crossref metadata and compares this to the plain text it extracts from the pdf document. it extracts values from various json attributes for the object metadata and subjects this to several data cleaning steps: it tokenizes the content, eliminates stop words, and applies an implementation of the porter stemming algorithm. the results are a word list containing each unique word that resulted from the data cleaning steps. next, the pipeline extracts the textual contents of the discovered pdf document. we use a python library called pdfminer to extract text from the pdf document. this data is also cleaned as per above, but then an additional step is applied: only words that occurred in the metadata word list are included in the document word list. if a word is missing from the latter, then it is included but assigned a value of 0. these are used to evaluate the extent to which the metadata and the discovered document share information. the cosine distance is calculated between the vector representations of the document and the metadata content, results in a similarity score [18]. this score is subsequently associated with the discovered object. the next step in the autoload pipeline is retrieval of licensing information from sherpa/romeo. this service has an easy to use web services interface and returns an xml document which describes the license associated with a particular issn. the sherpa/romeo color is a strong indicator of whether an item can be archived, with green reflecting the licenses which are most amenable to local archiving of some version of the publication. but there are other elements in the response that must be considered. other license details of interest include information about which version of a publication can be archived, whether there is an embargo period, and additional free text descriptions about the license and the publication. we retrieve this information even for items for which no full text was retrieved, because eventually it may be used to trigger additional discovery steps, for example for discovering preprints. figure 2. neo4j web interface the end of the line autoload aggregates retrieved metadata, licensing information, and details about discovered content in a property graph. this graph has five node types: author, publication, publisher, subject and file. this model is derived from the model utilized in a previous project, described in egosystem: where are our alumni [19]. the autoload property graph model differs in that it is concerned with publications, publishers, and discovered content, rather than people, institutions, and publications. each publisher node is uniquely identified by issn, and has associated with it various data from sherpa/romeo. each file node is uniquely identified by a filename (uuids are employed to avoid collisions) and each has a similarity score property with a value computed as described above . publication nodes are uniquely identified by doi and have properties that contain values from crossref. this property graph aggregates all the various information that was discovered for each entry in a resource list or change list that lacked full text. it is intended to facilitate validation of discovered content and verification of its archivability, but other uses have been proposed, such as a review of crossref as compared with locally assigned subjects. populating and updating the graph store is the last step in the pipeline before items can be cleared to be added to laro. autoload generates csv files describing the nodes and relationships for the graph. these files are imported into a neo4j graph database [20]. it has a visual query and browse interface, as illustrated in figure 2. neo4j provides a web services interface which we used to build a simple web view of harvested content. this interface displays a tabular view of discovered content together with relevant details about each file. just like the system described in [13] the web view of harvested content highlights entries using their sherpa/romeo license color code. the web interface (figure 3) includes the vector similarity score for the object as compared to its metadata, the object’s original filename, and a link to the object which a reviewer can access in order to visually inspect its contents. these elements allow a reviewer to make a final determination as to whether this object ought to be added to the repository. when these items are added, they will be indexed by solr, and subsequently become part of the resource list or change list that the pipeline consumes. these items will be skipped by the autoload pipeline in future iterations because they were processed successfully in a previous iteration. figure 3. autoload web review interface conclusion autoload is implemented in python, with much of the work handled by three classes: a harvester class, a comparison class and a graph generation class. an initial execution of the pipeline identified and harvested nearly 4,000 objects. as autoload is a work in progress, we do not yet automatically ingest the objects it discovers, but instead archive them for review. if even half of them were deemed appropriate copies and archivable, autoload will have tripled laro’s full text holdings. this project motivated the solrsync proof of concept, which can generate resourcesync resource lists and change lists dynamically. it also serves to demonstrate an extremely useful non-replication use case for resourcesync: sustaining a workflow that acts upon resource lists and change lists. it points the way forward for developing self-sustaining processes based on resourcesync. it is likely that we will adapt this approach to support several other efforts, including providing bibliographic metadata for characterizing institutional output and expertise, generating a corpus of locally authored publications for text and data mining (tdm) collection – where the tdm will aggregate and perform additional processing on content from several resourcesync sources, as well as other more typical data replication use cases, such as exposing laro to google for harvesting and indexing. the weakest link in this pipeline is the autoload harvesting component, because it essentially emulates user interactions and uses screen scraping and brittle heuristics. this situation will hopefully improve as content providers begin to adopt the signposting model [21]. signposting uses typed links in http response headers to address some common information discovery patterns, including discovering bibliographic metadata and locating full text from a landing page. this would take much of the guess work out of the process of discovering full text for a resource. the autoload pipeline would also benefit from the availability of a licensing pattern (rfc5988 includes a “license” relation type which could be adapted for this purpose), which might reference a record in sherpa/romeo or perhaps a creative commons license, as applicable. at this point we have come full circle: the autoload pipeline, initiated by resourcesync and applied to laro generates a list of new content that can be ingested into laro. as autoload expands laro’s full text holdings, it is becoming increasingly worth our while to return to our initial goal: using resourcesync resource lists and change lists to enhance the discoverability of content in laro. notes “a machine that would go of itself:the constitution in american culture” is the title of a book by michael kammen. references [1] van de sompel, h, sanderson, r, klein, m, nelson, m.l., haslhofer, b, warner, s, lagoze, c. 2012. a perspective on resource synchronization. d-lib magazine [internet] volume 18 number 9/10 [cited 2017 march 21]. available from http://www.dlib.org/dlib/september12/vandesompel/09vandesompel.html [2] niso standard z39.99.2017 http://www.niso.org/standards/z39-99-2017/ [3] van de sompel, h., bekaert, j., liu, x., balakireva, l., schwander, t.. adore: a modular, standards-based digital object repository, the computer journal, september 2005, volume 48, number 5 [cited 2017 april 20], http://dx.doi.org/10.1093/comjnl/bxh114 [4] apache solr reference guide. 2015. [cited 2017 feb 7]. available from: https://cwiki.apache.org/confluence/display/solr/apache+solr+reference+guide [5] sitemaps xml format. 2016. [cited 2017 feb 7]. available from: https://www.sitemaps.org/protocol.html [6] klein, m, van de sompel, h. 2013. extending sitemaps for resourcesync. jcdl ’13. available from: http://doi.acm.org/10.1145/2467696.2467733 [7] klein, m, sanderson r, van de sompel, h, warner, s, lagoze, c, nelson, m. l. 2013. a technical framework for resource synchronization. d-lib magazine [internet] volume 19 number 1/2 [cited 2017 feb 7]. available from: http://www.dlib.org/dlib/january13/klein/01klein.html [8] warner, s. resync. 2016. [cited 2017 feb 7]. available from: https://github.com/resync/resync [9] shankar, h., klein, m., van de sompel, h. resourcesync push. 2013. [cited 2017 apr 20]. available from https://github.com/resync/resourcesync_push. [10] shankar, h., klein, m., van den burg, h., rspub-core. 2017. [cited 2017 apr 20]. available from https://github.com/ehri/rspub-core [11] django documentation. [cited 2017 feb 7]; available from: https://docs.djangoproject.com/en/1.10/ [12] reitz, k. 2016. the hitchhiker’s guide to python: xml parsing. available from: http://docs.python-guide.org/en/latest/scenarios/xml/ [13] flynn, s, oyler, c, miles, m. 2013. using xslt and google scripts to streamline populating an institutional repository. code4lib [internet]. [cited 2017 feb 7]; issue 19. available from: http://journal.code4lib.org/articles/7825 [14] sherpa/romeo applications programmers’ interface. 2013. [cited 2017 feb 7]. available from: http://www.sherpa.ac.uk/romeo/apimanual.php?la=en [15] crossref rest api. 2016. [cited 2017 feb 7]. available from: https://github.com/crossref/rest-api-doc/blob/master/rest_api.md [16] introducing oadoi: resolving a doi straight to oa. 2016. impactstory blog [internet]. [cited 2017 feb 7]. available from http://blog.impactstory.org/introducting-oadoi/ [17] microsoft academic services documentation. [cited 2017 feb 7]; available from: https://www.microsoft.com/cognitive-services/en-us/documentation [18] muhlestein, d. 2007. similarity of texts: the vector space model with python. all my brain blog [internet]. [cited 2017 feb 7]; available from: http://allmybrain.com/2007/10/19/similarity-of-texts-the-vector-space-model-with-python/ [19] powell, j, shankar, h, rodriguez, r, van de sompel, h. 2014. egosystem: where are our alumni? code4lib [internet]. [cited 2017 feb 7]; issue 24. available from: http://journal.code4lib.org/articles/9519 [20] using the neo4j rest api. 2012. hack sparrow blog [internet]. [cited 2017 feb 7]. available from: http://www.hacksparrow.com/neo4j-tutorial-rest-api.html [21] signposting the scholarly web [internet]. [cited 2017 march 21]. available from http://signposting.org about the authors james powell (orcid: 0000-0002-3517-7485) is a research technologist and a member of the digital library research and prototyping team at the research library at los alamos national laboratory. his latest book is powell, james e., and matthew hopkins. a librarian’s guide to graphs, data and the semantic web. chandos information professional series. waltham, ma: chandos, 2015. martin klein (orcid: 0000-0003-0130-2097) is a scientist in the prototyping team of the los alamos national laboratory research library. his research interests include web-based scholarly communication and system interoperability, temporal aspects of the web, and information retrieval and extraction. he is the lead editor of the resourcesync specification and is involved in work on robust links and memento. herbert van de sompel (ordic: 0000-0002-0715-6126) is an information scientist at the los alamos national laboratory and leads the prototyping team in the research library. the team does research regarding various aspects of scholarly communication in the digital age. herbert has played a role in various interoperability efforts (oai-pmh, openurl, oai-ore, info uri, open annotation, resourcesync, sharedcanvas, memento) and in the design of scholarly discovery tools (sfx linking server, bx recommender engine). currently, he works on the robust links effort, aimed at addressing reference rot, and contemplates about archiving the web-based scholarly record. more information about herbert can be found at http://public.lanl.gov/herbertv/ subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – preparing existing metadata for repository batch import: a recipe for a fickle food mission editorial committee process and structure code4lib issue 42, 2018-11-08 preparing existing metadata for repository batch import: a recipe for a fickle food in 2016, the university of waterloo began offering a mediated copyright review and deposit service to support the growth of our institutional repository uwspace. this resulted in the need to batch import large lists of published works into the institutional repository quickly and accurately. a range of methods have been proposed for harvesting publications metadata en masse, but many technological solutions can easily become detached from a workflow that is both reproducible for support staff and applicable to a range of situations. many repositories offer the capacity for batch upload via csv, so our method provides a template python script that leverages the habanero library for populating csv files with existing metadata retrieved from the crossref api. in our case, we have combined this with useful metadata contained in a tsv file downloaded from web of science in order to enrich our metadata as well. the appeal of this ‘low-maintenance’ method is that it provides more robust options for gathering metadata semi-automatically, and only requires the user’s ability to access web of science and the python program, while still remaining flexible enough for local customizations. by william roy and chris gray introduction the confederation of open access repositories (2012) acknowledges that, “most content recruitment practices are fairly resource intensive and involve staff resources” (p. 2). at the university of waterloo, mediated deposit is a multi-stage process consisting of communications and promotions to researchers, review of faculty cv’s for eligible works to deposit, checking for copyright and licensing considerations, and the manual entry of metadata in order to be consistent with local repository metadata standards. mediated deposits have proven to be a useful way to stimulate usage by faculty. however, this option requires a far greater investment of time for library staff than self-depositing workflows require, because users are not required to enter metadata at the time of ingest. at the university of waterloo, it was discovered that staff were spending close to 50% of their time on the manual collection, clean-up, and entry of publisher metadata into spreadsheets. this part of their mediated deposit service required rethinking so that staff could spend more time on the rest of the outreach and copyright-related activities required for recruiting content and less time on data-entry. there have been numerous and notable efforts made by institutional repository (ir) managers and technologists to streamline the collection and harvesting of publications’ metadata. ir’s collect and provide access to a large variety of original scholarly works, but a primary focus still remains on providing access to published works with ‘green’ open-access friendly versions, or openly-licensed publisher versions that are free to redistribute and share on their platforms. the accompanying descriptive metadata needed for most repository objects is already existent in some shape or form in citation or registration agency databases, but is likely in need of transformation for ingest into a repository. examining the previous literature demonstrates the varying lengths to which repository personnel have gone to retrieve and transform data into formats useful for their local context. the college of wooster used an xml export of selected citations from the commercial program refworks and transformed the resulting citations metadata into a repository-ready dublin-core xml format using custom-written xslt (flynn, oyler & miles, 2013). yuan li (2016) successfully used soapui with the web of science web services to harvest publications metadata in xml and similarly employed a custom xslt to transform the output so that it conformed to local metadata standards. finally, a most recent and robust solution from powell, klein and van de sompel (2017) automated metadata collection, along with several other aspects of the content recruitment process by using resourcesync, django solr sync, multiple api’s (e.g. sherpa romeo, crossref & oadoi), and a neo4j graphing database. rationale we considered similar routes as discussed above, but due to the nature of how different citation services package and output metadata, it was found that no single source (e.g. web of science, scopus, and various citation management software packages) could be relied on to consistently contain the fields that were of interest to our local repository schema. of considerable challenge were metadata containing funding references, creative commons licenses, and affiliation information. xslt, although a powerful timesaver, is not easily editable for those not trained in the language. we assumed using xslt would sacrifice support staff’s ability to manually edit or supplement retrieved metadata that was not ideal upon initial collection. support staff who edit repository metadata may not have the training to dissect potential problems and edit the resulting xml documents but would be much more proficient working with spreadsheets. the method of using spreadsheets to organize metadata for import into various repositories and information systems can be a popular choice for libraries. due to the low barrier for adoption by new and contract staff members, using a data-entry workflow that consists of entering metadata into spreadsheets was a natural choice for our library. dspace, the repository used at the university of waterloo, utilizes a tool called the safbuilder [1]. safbuilder, available in bash script or windows batch file format, accepts a csv document’s filename as its argument and packages metadata and file names indicated within the spreadsheet into separate dspace ready archive folders, complete with properly-formed metadata in xml. many other repositories have similar options for batch metadata entry. for example, digital commons offers a “batch upload excel” option [2], and islandora modules exist for importing mods metadata through csv [3]. it is our hope that by utilizing a method of semi-automated metadata collection in harmony with a typical spreadsheet workflow, we can provide a relatively platform-agnostic approach that is simple for support staff to use and to alter as needed. in addition, because this method allows for communication with apis, it remains flexible enough to connect to other emerging scholarly services. there are a variety of interesting services arising and growing each day that source value-added scholarly metadata using apis such as: orcid, the scholix: framework for scholarly link exchange, sherpa romeo, and datacite. since the urllib python package that the script uses is able to make web requests and the json package can communicate and parse json data, this method, given some additional tweaking, could also allow for easy association of items to things such as: linked datasets, orcids, and sherpa romeo information via the list of dois supplied to the script. defining the problem as mentioned earlier, no single source was all-encompassing in offering the type of metadata fields our repository hoped to populate. some pain-points in particular were: funding information creative commons licenses affiliation information full bibliographic citations citation software, such as zotero or refworks, do not capture these fields by default. we found web of science (wos) to be the most complete source of information that staff would rely on for the retrieval of metadata [4]. despite being able to export a tsv from wos with a reasonable amount of the metadata we required, the above fields were either non-existent in wos, or required further editing to be ready for packaging by safbuilder. additionally, the tsv exported from wos contained numerous extraneous columns that were of no use. in summary, we wanted to attempt to populate all relevant fields (fig. 1) with metadata that we deemed consistent with our local standards and specifications. figure 1. column structure showing metadata fields our script targeted. click to enlarge before implementing our python script, the workflow [5] was as follows: locate a list of titles on wos that are ready for import into the repository and add them to a marked list. use the marked list parameters to select all applicable metadata fields to capture. export a tsv file of the marked list. open the tsv as a spreadsheet and copy and paste the fields of interest into a safbuilder template, avoiding extraneous columns. the column headers of the safbuilder csv represent dublin core field names, so staff must discern the appropriate matching column in the wos tsv, and copy and paste accordingly. after transferring useful information from wos tsv, identify missing metadata and fill in remaining fields manually by locating the information on publisher’s website. manually change formatting to be compliant with local standards (i.e. switch semicolons to double pipe symbol, use concatenation formula to make dois resolvable, alter date fields to display in one column rather than two, and use iso 8601 format). when complete, run the safbuilder script with the newly created csv and import the resulting zip folder to uwspace. our proposed solution overview of method the scripts use standard python libraries with the exception of the third-party habanero [6] library. this library has a crossref module that is imported to request json data from the crossref api. pip was used to install habanero. alternatively, if you do not want to install habanero, it is possible to access the crossref api by just using the standard urllib.request library. the python script starts from a spreadsheet (tab delimited, utf-8) downloaded from wos. the wos metadata export should include the following fields. we prefer to obtain author information, keywords and abstract metadata from wos because information for these fields is often more complete, or formatted in ways that is preferred by our repository. dois are obtained at this step for the purposes of creating the crossref lookup, and titles are obtained as a useful point of reference for troubleshooting if the doi is found to be non-existent. the wos metadata export should include the following fields: author title author keywords keywords plus ® abstract digital object identifier these fields appear in the tab delimited file as: af, ti, de, id, ab & di. the script then takes each doi from that data, and uses the dois to get citation information for each record using crossref content negotiation [7]. it – obtains supplementary metadata fields for each record from crossref by converting the json returned from crossref into python objects. this is done by a call to another module named doilookup. within the scripts, data is always handled as dictionaries with the keys in the dictionaries representing the heading of columns in a tab delimited file. the python csv module contains two utilities called dictreader and dictwriter that read and write files to and from dictionaries. dictreader provides a fieldnames list that contains the column headers in order and dictwriter must be provided with a similar list that is used as the column headers and sets their order. running the script with wos integration the main script called tsv4saf [8] takes two arguments: the path and filename of the wos input file, and the path and filename of the output file: python3 tsv4saf.py [your input file] [your output file]. your input file = the path to the tab-delimited file downloaded from wos your output file = the desired name of your file in tsv format for example: python3 tsv4saf.py wosdata/savedrecs.txt foringest/output.tsv as was desired, the output of this script will return synchronized rows of metadata for the targeted dublin core fields, as well as some fields that are specific to local schemas and workflows. figure 2. screenshot of the script’s output shown in excel for a set of sample data. click to enlarge when viewing the output, it is noticeable that certain cells are returned empty. since this script relies heavily on the existence of dois, if the supplied metadata from wos is missing dois, the script is programmed to only output wos metadata for those particular rows, and will skip columns normally obtained by crossref rather than returning an error. similarly, the script will output blank cells for data that is not existent in crossref, as information is not always supplied. nonetheless, this saves significant amount of time, and helps support staff perform edits to the metadata at a much quicker rate than was possible before the implementation of the python script. running the generalized script on a single doi the python module, tsv4saf, calls a separate module called doilookup. the doilookup module can be used by other python scripts or used as a standalone module from the command line. this may be useful to users who are not starting with a wos file or targeting a csv import process for dspace. the module, doilookup, can be used from the command line to fetch the crossref data for a single doi. python3 doilookup.py [your doi] when used from the command line, the script outputs a printed list of the metadata fields (using field names modeled on the crossref json output) and the value found for each field (fig. 3). $ python3 doilookup.py 10.1002/cbic.201800049 pb: wiley ty: journal-article id: https://dx.doi.org/10.1002/cbic.201800049 fu: natural sciences and engineering research council of canada|china sponsorship council li: http://doi.wiley.com/10.1002/tdm_license_1.1|http://onlinelibrary.wiley.com/termsandconditions#vor issn: 1439-4227 ct: chembiochem af: department of chemistry and waterloo institute for nanotechnology; university of waterloo; 200 university avenue west waterloo ontario n2l 3g1 canada|xiangya school of pharmaceutical sciences; central south university; 172 tongzipo road changsha hunan 410013 china issued: 2018-04-26 bc: yu, t., zhou, w., & liu, j. (2018). an rna-cleaving catalytic dna accelerated by freezing. chembiochem, 19(10), 1012–1017. doi:10.1002/cbic.201800049 figure 3. example of how to use this script on its own and the resulting output utilizing the doilookup for customized scripts the doilookup script can be called from within another python script. we have done this with our tsv4saf script which is invoked by the following method: import doilookup data = doilookup.lookup('[your doi]') within doilookup, a function lookup() takes a doi as its argument and returns the data in a python dictionary. as the tsv4saf script does, this can be called within a loop to handle multiple dois. assuming the variable rows is a list of dictionaries and each dictionary has a key of di (where a doi is stored), a new list called ‘newrows’ can be built with dictionaries that have the data from ‘rows’ together with data retrieved from crossref: newrows = [] for row in rows: rowaug = doilookup.lookup(row['di']) newrows.append({**row, **rowaug}) the variable data will then contain a python dictionary with the metadata in it. the handling of the metadata within the scripts and the output of the scripts are documented by certain fixed dictionaries and lists. they are: in tsv4saf.py fields: a list of the fields of interest in the wos files and in the output of the doi lookup that are used in the output mapping: a dictionary mapping the fields of interest to output fields for dspace newfieldnames: a list of the output field names produced by applying mapping to fields in doilookup.py addedfields: a list of fields that are constructed from crossref data as an easy example of customization, your site will not be using our local metadata field uws.contributor.affiliation2. your site can change the entry in mapping from: uws.contributor.affiliation2[en] to instead use a more standard field like: dc.contributor further considerations the heavy lifting of parsing the crossref json data is included in the doilookup script. the difficulty is in correctly parsing the json data returned by crossref and finding the data of interest. this is sometimes nested within the hierarchy of the json data and is not always present in all records, so a certain amount of drilling down and testing of conditions is necessary. the doilookup we have provided does this for several key metadata fields of interest, but there are of course many more data options available through crossref that could be incorporated into future versions of the script. for those wishing to add custom fields, they should be aware that some of the json returned by crossref is missing data and needs to be formatted in a way that accommodates output to csv (or other formats). this will be the hardest part of the script to modify for people looking for different metadata and who want to format the data on output. those customizing the output format should be aware that our script assumes an input file that is utf-8 encoded, and also assumes that output is done to a tab delimited file. if choosing other input and output parameters in your own project, you will need to change the parameters to the open command to read and write files or to the dictreader or dictwriter. we also recommend that certain rules of netiquette be applied when making calls to the crossref api [9]. this includes supplying a valid email address within your script where crossref can reach you if any problems occur. if you wish to provide an email address, you can add a file called polite_email.txt containing your contact email. the doilookup script looks for this file and provides the address to crossref if the file exists. in addition, adding pauses between doi lookups will help avoid overloading of the api. the loop within tsv4saf begins with a call to the python sleep function to pause for a tenth of a second between loops. conclusion cory doctorow has said, “a world of exhaustive, reliable metadata would be a utopia. it’s also a pipe-dream” (2001). as with many projects aimed at automating the collection of metadata, it’s not uncommon to encounter imperfections, and to ultimately learn that humans are still helpful and necessary intermediaries in the quality control process. our methodology succeeds in offering a holistic approach to how most repository staff work, and offers an attractive option for those repository managers who want to streamline a very tedious and low-level part of the otherwise challenging and rewarding work that themselves and their staff members take part in. by using the semi-automated tsv4saf script, you can quickly collect large volumes of well-formed metadata, and have it delivered in a format that remains flexible enough to be vetted by a pair of human eyes who can inspect and improve the delivered content as needed. further, because python easily communicates with apis, this method leaves open the possibility of connecting this script to many other value-added scholarly services in the process, thereby offering the potential for further enrichment of metadata, and improvements to the experiences of repository users in the long term. bibliography averkamp s, lee j. repurposing proquest metadata for batch ingesting etds into an institutional repository. 2009 [accessed 2018 sep 7];(7). https://journal.code4lib.org/articles/1647 confederation of open access repositories (coar). sustainable best practices for populating repositories: preliminary report. 2012 [accessed 2018 sep 7]. p. 1–11. https://www.coar-repositories.org/files/sustainiable-practices-preliminary-results_final.pdf doctorow c. metacrap: putting the torch to seven straw-men of the meta-utopia. 2001 aug 1 [accessed 2018 sep 7]. https://people.well.com/user/doctorow/metacrap.htm li y. harvesting and repurposing metadata from web of science to an institutional repository using web services. 2016 [accessed 2018 sep 7];22(3/4). (d-lib magazine). doi:10.1045/march2016-li powell j, klein m, sompel hv de. autoload: a pipeline for expanding the holdings of an institutional repository enabled by resourcesync. 2017 [accessed 2018 sep 7];(36). https://journal.code4lib.org/articles/12427 notes [1] safbuilder documentation is available at: https://github.com/dspace-labs/safbuilder [2] for further information about the digital commons excel batch upload process, see: https://vimeo.com/79930071 [3] an example of an islandora csv import module: https://github.com/islandora-collaboration-group/icg_csv_import [4] scopus is likely comparable in this respect but was not tested. [5] for brevity, we have intentionally removed certain steps that do not pertain to metadata collection such as gathering files, checking copyright, etc. [6] documentation (including installation) of the latest version of habanero is available at https://habanero.readthedocs.io/en/latest/index.html [7] for a useful discussion of doi content negotiation through crossref see: https://crosscite.org/docs.html [8] github repository for both the doilookup and tsv4saf scripts: https://github.com/cpgray/tsv4saf [9] link to crossref api documentation, including etiquette: https://github.com/crossref/rest-api-doc about the authors chris gray is a web and database developer for the university of waterloo library with a strong interest in data wrangling. he prefers python and postgresql more than any ordinary brands and thinks “user-friendly” is too often used to avoid automation. he only speaks of himself in third person in special circumstances. william roy is digital repository scholarship specialist for the university of waterloo library and handles recruitment, promotion and depositing of scholarly works for the institutional repository. he holds a masters of library and information science from the university of western ontario but is earnestly working on earning his chops on the technical side of librarianship. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – editorial mission editorial committee process and structure code4lib issue 48, 2020-05-11 editorial an abundance of information sharing. as a relatively new member of the code4lib journal editorial committee, i had the privilege of coordinating issue 48. it’s been a great learning experience and also solidifies for me the idea of code4lib as a community focused on sharing information and experiences about library technology. we had 30 submissions for this issue of the journal, a number far beyond the capacity of any one issue. and since the number of editorial committee members, all of whom volunteer their time on top of their regular jobs, is also limited, a number of submissions that would have otherwise been accepted were not able to be included. it’s unfortunate since the mission of the journal is about sharing information, however, the number of authors willing to share their work, their experience, and their code is inspiring. i hope these authors will resubmit for upcoming issues of the journal and continue their efforts to share their knowledge with the library technology community. issue 48 includes articles about the use of existing technology to build, enhance, and manage digital libraries (“leveraging googledrive for digital library object storage“, “building a library search infrastructure with elasticsearch“) and website applications (“how to use an api management platform to easily build local webapps“, “git and gitlab in library website change management workflows“), as well as the development of new tools for automation and resource management (“tweeting tennessee’s collections: a case study of a digital collections twitterbot implementation“, “experimenting with a machine generated annotations pipeline“, “operationalizing the acrl/rbms latin place names file with python“, “building strong user experiences in libguides with bootstrapr and reviewr“). also included are articles on data-informed decision making (“iiif by the numbers“) and the verification of vpat accessibility claims (“trust, but verify: auditing vendor-supplied accessibility claims“). i hope readers will find something interesting, inspiring, or helpful in this issue and will consider submitting a proposal about their own experiences with library technology to continue to add to the collection of freely shared information available to and by the code4lib community. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – editorial introduction – code4lib: long may you run mission editorial committee process and structure code4lib issue 7, 2009-06-26 editorial introduction – code4lib: long may you run the code4lib journal mirrors the diversity and depth of interests and expertise of its readership. our successes, indeed, are yours. by tom keays the code4lib journal website has been using google analytics since it first went live and we’ve found it fascinating to see how the site is used. here are a few highlights. during the 18 month period from december 2007 through may 2009, the journal has had almost 69,000 visitors for a total of 94,000 visits and almost 181,000 page views. although 69% of our readership views only one page (this is what google terms the “bounce rate”), a sizable proportion of you view 2-5 pages per session. google analytics: visitor overview, december 1, 2007 – may 31, 2009 the journal’s average number of visitors per day is about 172 since issue 1 in december 2007. however, if you compare the most recent 12 months from june 2008 through may 2009 with the previous 6 months, readership increased 138% from 76 visits/day to 182 visits/day. not surprisingly, the days surrounding publication of each issue sees a huge spike up to peaks of over 1,100 views per day. google analytics: visits for all visitors compared, december 1, 2007 – may 31, 2009 of the journal’s 75 html-formatted pages (as of issue 6), including the homepage, 61 articles, 6 table of contents, and 6 utility pages, 7 of the top 8 pages viewed have been articles. these articles included: “free and open source options for creating database-driven subject guides” by edward m. corrado and kathryn a. frederick, issue 2. “using google calendar to manage library website hours” by andrew darby, issue 2. “libraryh3lp: a new flexible chat reference system” by pam sessoms and eric sessoms, issue 4. “beyond opac 2.0: library catalog as versatile discovery platform” by tito sierra, joseph ryan, and markus wust, issue 1. “the rutgers workflow management system: migrating a digital object management utility to open source” by grace agnew and yang yu, issue 1. “column: we love open source software. no, you can’t have our code” by dale askey, issue 5. “facet-based search and navigation with lcsh: problems and opportunities” by kelley mcgrath, issue 1. google analytics: top content, december 1, 2007 – may 31, 2009 articles from the first 2 issues are still being read over a year later and more current articles are also claiming significant readership. the editors are understandably pleased with this trend and see it as a measure of the high quality of the articles. but it is the code4lib community itself that has all the bragging rights. our successes, indeed, are yours. the journal’s mission from the very beginning has been “to foster community and share information among those interested in the intersection of libraries, technology, and the future.” the code4lib community continually shows the depth and diversity of knowledge of the librarians and coders of which it is made, not just here in the pages of this journal, but in the email discussions on the listserv, in conversations on the irc channel, at the annual conference and a growing number of regional meetings, and wherever and however we connect, interact, and share our expertise and knowledge. it is a pleasure to be a part of it all. long may you run! code4lib journal editorial changes this issue sees the departure of one editor and arrival of another. we gratefully acknowledge the contributions of christine schwartz, who served as an editor for issues 3 – 7 and was the coordinating editor for issue 6. christine, who also blogs at cataloging futures about rda, marc21, ils and opacs, and a variety of topics centered on cataloging and metadata, is the metadata librarian at princeton theological seminary libraries. since many of the journal’s articles have touched in these areas, her knowledge and experience has been invaluable. we are pleased to welcome kelley mcgrath, who is cataloging & metadata services librarian (audiovisual) at ball state university and also serves as the chair of online audiovisual catalogers (olac) cataloging policy committee. kelley has authored two articles in code4lib journal, including “facet-based search and navigation with lcsh: problems and opportunities” in issue 1 and “identifying frbr work-level data in marc bibliographic records for manifestations of moving images” in issue 5. as always, if you would like to become involved with the production of the code4lib journal, email the editorial committee at c4lj-discuss@googlegroups.com and let us know how you are able to contribute. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – using scalable vector graphics (svg) and google sheets to build a visual tool location web app mission editorial committee process and structure code4lib issue 58, 2023-12-04 using scalable vector graphics (svg) and google sheets to build a visual tool location web app at the university libraries at virginia tech, we recently built a visual kiosk web app for helping patrons in our makerspace locate the tools they need and assist our staff in returning and inventorying our large selection of tools, machines, and consumables. the app is built in svelte, and uses the google sheets “publish to web as csv” feature to pull data from a staff-maintained list of equipment in the space. all of this is tied to a scalable vector graphics (svg) file that is controlled by javascript and css to provide an interactive map of our shelving and storage locations, highlighting bins as patrons select specific equipment from a searchable list on the kiosk, complete with photos of each piece of equipment. in this article, you will learn why the app was made, the problems it has solved, why certain technologies were used and others weren’t, the challenges that arose during development, and where the project stands to go from here. by jonathan bradley in the spring of 2022, the university libraries at virginia tech (vt) opened its prototyping studio to the vt community and public writ large. the space provides large machines like cnc milling machines, over a dozen 3-d printers, vacuum forming machines, a pick-and-place machine, and others, but in addition, the space provides access to thousands of hand tools and consumable items, everything from microelectronic components to fasteners. given the huge scale of the number of items kept in the libraries’ inventory and made available, organization and record-keeping quickly became an issue for the space. however, this issue existed both for us as the maintainers of the space, who needed a way for the studio’s student workers to know where to return items left out in the space and understand when tools go missing or consumables are running low, and for the studio’s patrons, who needed to be able to quickly locate the tools or search for options when they may not be sure what tool they need for the given situation. we set out to build a visual map and searchable inventory of the items in the space that could be made available on a public kiosk for patrons and employees to use. figure 1. a photo of the tool finding kiosk in the prototyping studio in newman library figure 2. a photo of one of the cabinets in the prototyping studio, complete with tool baskets the data a frequent problem that i’ve faced over the years building applications or systems for use in vt’s libraries is how to handle the fact that very often non-technical people are being asked to maintain the data. this drastically narrows the field of approaches that are viable. many ways of manipulating data that a developer might be comfortable with are well outside the reach of the average user, and ease of use is also a huge factor in getting people on board with your solution, particularly when those people are student workers. one option is to build a back end structure and interface for a database that allows users a gui to work with, but there is significant overhead with an approach like that, such as building authorization and authentication methods and maintaining them and asking people to go to one more new location as part of their jobs. i’ve used this approach over the years, but recently i’ve decided on a different approach for new projects that has been working really well, and that is just using a google sheet. i primarily build web applications, so i prefer my data coming in as json data. a google sheet can be published to the web as a csv file, which can easily be converted to json data using a library like papa parse. this makes getting data a single fetch call away with minimal setup, and while there are downsides, the benefits are numerous. some of the downsides include lack of type verification on data fields, relying on google’s existing structures and distribution mechanisms staying constant, and the ability for users to put invalid data into the sheet. we have run into all these issues while using google sheets in this way; we’ve had students and staff put a type of data into a field in the spreadsheet that isn’t what we are looking for and it causes a problem on the front end. usually, this sort of issue makes itself known quickly, and luckily google sheets maintains a history so you can see who made the change and undo it quickly and easily. i often put comments on fields to let people know any rules i need them to follow about putting data in a field, or, if it becomes a recurring problem, i will put data validation rules within google sheets on a column to try to cut that behavior off before it can keep popping up. honestly, though, this rarely comes up, and when it does, it is usually a matter of sending a message to the user who put the data in wrong, explaining what you need from them, and it usually doesn’t happen again. we’ve also run into the issue in the past of google changing its apis or altering the way urls are generated and by extension breaking something in the system. but this is simply a product of relying on cloud infrastructure, and we’ve had these same issues arise using other systems from different vendors that were supposed to be designed for this sort of work. generally, i just update the code to accommodate google’s changes and move on, and this happens very rarely. in the years that i’ve been setting up spreadsheets to act as data sources this way, i’ve only run into this once. but despite these issues, the benefits are immense. google’s infrastructure already has methods for authorization and authentication, and those methods are well-tested and maintained and already available to users for accessing the sheets and controlling permissions. the interface is also very familiar and easy to use for library employees. i have yet to run across the student worker or staff member who couldn’t update and maintain a spreadsheet, and the value of familiarity is often vastly overlooked when preparing systems like this. finally, the simplicity of using a spreadsheet in this way in comparison to something like setting up a database infrastructure with api and authentication controls, is an enormous boon for small projects. i can have a data source ready for an app in less than 10 minutes. would i recommend an approach like this for a large complex app, or one in which data integrity was important? absolutely not. but for small projects, particularly the public kiosk or service point assistance apps like the ones i’ve used it for in the past, it is a very easy method to get a working app online that requires very little upkeep and overhead while also allowing other non-technical users to maintain the data. for the tool finding app, the studio’s student workers and staff for the space maintain a google sheet that was designed by myself and the space manager. it includes all the fields they needed for inventory purposes and those i required to make the app work, and once established, we’ve locked the structure so none of the other staff could alter it. if it is decided that new columns are needed, we can add those in a way that doesn’t break anything, and since i don’t have to use all of the fields available in the sheet, they can also include additional information that we don’t show on the public kiosk, like when we last ordered the tool or consumable, which is valuable information for us in record keeping and maintaining the service, but not so much for a patron trying to find a hammer. in the past, i’ve been asked how we handle data that we might not want exposed publicly, since publishing a sheet means others could access the url. the first argument i have is, if you are using it in a public app, you have already made the data available to the public via the app itself, so you should make sure any data that is there is safe to be shared in this way. the fact that someone can access the csv data from the spreadsheet is moot; anyone who knows what they are doing could have gotten that data from your app anyway. but what about the scenario in which you want to keep information in the spreadsheet, information you need as employees, but that you don’t want the public to have? in this situation, we have created a second sheet in the google sheet and mapped only the fields that we want to make publicly available in the app into this second sheet. google sheets allows you to only publish a specific sheet when deciding to publish the csv data, so this way you can release only a subset of the total information you are keeping for your records if need be. the front end for the front end of the app, i went with what is quickly becoming my default solution for small apps: svelte. as a note, i am using svelte and not sveltekit, as i don’t feel like for these small projects, many of which only have one or two pages, i need the extra infrastructure, and by extension overhead, that sveltekit offers, though i would definitely consider it for a larger project. i used to build these sorts of applications in react, and consistently found that react, especially if i needed global state via store management, was just a lot more complex than what i needed for my projects. svelte, on the other hand, has a small setup process, and store management is far simpler than with a framework like react, providing benefits like two-way data binding without the additional features that are really targeted toward a much larger app. it also doesn’t hurt that svelte is a lot easier to learn and looks a lot more like regular html, css, and javascript (js), than react, and i am confident that if needed a new programmer in my organization could look through the code and gain an understanding of what is going on without much trouble. the app is composed of six components, a main app component that controls which of the other components are on display and handles initial data fetching, a footer, a greeting page, the scalable vector graphics (svg) map of the shelving, and a table or catalog view for combing through the available tools. the footer is simply vt libraries’ logo lockup–a special version of virginia tech’s logo with an additional text treatment created by the university relations department to include “university libraries” as part of the logo–which is displayed at the bottom of every screen. the app component contains logic for switching between different views of the app. when the app is loading and the data is being fetched, it displays a loading spinner. it then has three switches for whether to display the greeting page, the table, or the catalog. the app loads the greeting page by default, which was a suggestion by a member of our libraries’ communications team who thought it might be nice for patrons to see a very short explanation of what the kiosk is for and presenting them with some prefilled search options instead of always expecting them to type in the search box. the app asks patrons what kind of project they are working on, and provides some buttons with common project types the prototyping studio receives, such as woodworking, electronics, 3d printing, or measurement. libraries’ staff intend to do some usability testing on the app, and these suggestions are one of the areas my unit are hoping to get feedback on. are these categories helpful? are there categories that definitely should be included, or should we reword those we have? figure 3. a screenshot of the welcome screen of the kiosk app, including footer in addition to prefilled buttons, patrons also can just go ahead and search for tools, and if they click the search box an onscreen keyboard appears. whether they use the search functionality or prefilled options, the system takes them to the next screen, which is a combination of the map of the shelves and the catalog, which is the default view method. users can then switch to a table view by using a button in the top right-hand corner if they choose. figure 4. a screenshot of the catalog view of the app both the table and the catalog use two-way data binding with the search field to filter out results, and the filtering function is a custom one that searches specific data fields chosen by myself and the space manager. the primary difference between the two display methods is that the catalog shows photos of the tools in question, and the table is more compact and displays more information at once. we chose the catalog to be the default view because the initial reaction from patrons was that they appreciated being able to confirm if the tool was the one they needed by looking at it, but which view should be the default is something we plan to look at during the usability testing. figure 5. a screenshot of the table view of the app the final component, and by far the most complex one from a coding perspective, is the map of the shelving. it displays at the top of the screen regardless of which view you are using and by default displays a vector graphics facsimile of the shelving and storage we have in the space. by default, all the shelves are empty. however, when a patron taps a tool on the touchscreen, either via the table or catalog, a location highlights on the shelving for where the tool should be. this highlighting takes the form of a color-coded storage basket, which is how we store the vast majority of the tools. this visual indicator is intended to immediately tell a patron something along the lines of, “the tool you are looking for is in the last cabinet on the right, second shelf down, in the red basket.” by extension, this system can also be used by the studio’s own student workers when trying to return tools to the cabinets that have been left out. figure 6. a screenshot of an item in the catalog selected and the resulting storage basket appears for the most part, the front end of the code is fairly standard single-page application (spa) running svelte, and its primary interest lies in the control of the svg file that the map is made of. the svg during development, a rough prototype of the needed svg that i created was used for testing purposes, but the final production graphic was created by a library employee with a design background and skill in working with vector graphics. the file was created in inkscape, a free and open-source vector graphics program that is very good at creating files for use on the web, unlike adobe illustrator, which often produces malformed svg files when exporting. during the process, i talked with the designer about the requirements of the file. it needed to have a separate layer that contained all the background shelving elements. in the studio space, the shelving has doors attached, but we chose to omit them in the graphic as they caused additional visual noise that we didn’t want the patron to have to deal with. in addition to the background layer, we needed another layer containing all the baskets, each one a separate object with a unique id and an id attached to the entire layer. the designer ended up putting together one basket design and just copying it, changing the metadata information, and editing the color information for it when needed. they also needed to go through and place all the baskets on the shelves where they appear in real life. this was a significant amount of work, but luckily was one-time work, and barring some sort of very large overhaul of the studio’s organizational structure for the space, should only need maintenance moving forward. the ids for the baskets is what makes all of this possible. each one has an id similar to “blue-01” and “blue-02”. these ids are all also used in the spreadsheet of data on the back end to indicate which basket a tool is located in. the layer that contains all the baskets also has an id of “baskets” so it can be accessed directly. for those unfamiliar with working with svgs, these ids, classes, and other elements are all treated like html elements when loaded in the browser, though this only works if you load it in specific ways. for example, loading the svg file using an <img> tag will result in you being unable to access the ids within the svg the way you would any other html element. however, simply copying the content of the svg file into the html file makes all its elements available for dynamic manipulation. luckily, svelte offers an {@html …} template option that will take whatever content is provided and drop it in as html, meaning we can load the svgs into variables using import syntax and then drop them into the html where they belong. the benefit to doing it this way, in addition to the svg being available to the css and javascript, and being able to handle files programmatically instead of hard-coding, is that we can leave the svg as a separate file that can be edited and maintained as a modular component and, perhaps most importantly, tracked by git in a repository. svg files, unlike raster image types like jpg or png, are a plaintext set of instructions on how to create the graphic, not a series of data points on which pixels are which color, meaning they mesh very well with a git and plaintext workflow, allowing you to see the specific changes happening to the file over time via a git diff. when the tool-finder app is loaded, a bit of javascript cycles through all the child elements of the id “baskets” and changes the visibility of all those elements to hidden, meaning the user is presented with an empty set of shelves. when a patron taps on a tool either in the table or catalog view, we grab the id that is associated with that tool (included in the data from the spreadsheet), find the element with that id in the svg file, and change its visibility to visible. thus, the user gets a dynamically changing set of shelves that can show any one of the potential locations for tools based on the spreadsheet data. finally, after a period of inactivity on the page, we reload the page to reset the system for the next patron using a standard js settimeout() function. challenges this project presented several challenges, but few were actually the product of the code and instead more the logistics of setup. since working with svgs is so easy with html, css, and js, the code to provide the expected behavior was done fairly quickly. however, we have had some issues with how the project appears on the kiosks. the code for the project is being served on a standard lamp stack cpanel server owned and maintained by the university libraries, and the kiosk just loads the url in a kiosk browser for display in full screen mode. we’ve done this for most of my unit’s public service applications and it works really well, but the lingering problems that come up are two-fold. the first is on-screen keyboards. it can be hard to get an on-screen keyboard on a kiosk to show up in the right area and not block important information, and this project was no exception. this, however, can be solved with a trial-and-error session with the university libraries’ it hardware team. the second problem that often arises is the browser on the kiosk. i usually test the application on my computer in different browsers using the resolution of the kiosk we have purchased, but each kiosk is a little different in terms of the operating system on the device, and some have their own versions of browsers or particularly old versions of browsers on them, so getting everything to look and behave just right is always a bit of trial and error as well once it is on the specific machine it will run on. the other major challenge has been the data for this project. the spreadsheet is complex and the number of items we have that need to be tracked is quite large, roughly 325 discrete types of items and thousands of individual pieces when small consumables are considered. and this number is continuing to grow. it is maintained by student staff, part-time wage, and one full-time employee, and the result has been a need for ongoing improvement. items including long or unhelpful titles, poor tagging, and lack of photos for some equipment have resulted in a struggle to have the information presented in a helpful manner that is easy to search, but this is a problem that will improve over time as more people work on it. the future for the future of this project, in addition to improving the overall data that feeds into the system as mentioned above, we have two main ideas we’re planning to pursue. one is the need for usability testing, mentioned earlier, with the hope that we will get feedback to help improve the interface and maybe make the whole system more attractive to patrons as they enter the space. the testing is currently scheduled for the end of the university’s fall 2023 semester, so we may have improvements implemented by spring 2024 if all goes well. the second goal for the future is to add a mechanism for finding machines outside of the shelving storage system. we have machines like the cncs and 3-d printers that aren’t on the shelves, but people may be interested in. the goal right now is to add an icon to the svg map of a person with a speech bubble that says “ask an attendant” that will appear when a patron has selected a tool or machine that requires special assistance, so they don’t keep searching needlessly. this feature will also provide the prototyping studio an option for handling items in the inventory that don’t have a location currently because they aren’t on any of the shelves but need to be kept track of by staff. this system is currently in production and seems to be helpful for patrons. as time passes, we will continue assessing if using the kiosk is actually helping guests; for example, do repeat users of the space who used the kiosk once use it a second time, or is it viewed as not worth it? the project is very much an experiment in how digital interfaces can improve a patron’s experience at a service point, and like all experiments, if it isn’t successful, we will adapt or drop it completely. and this mindset acts as an explanation for why it is so important for us to use these simple, quick development options like svelte or google sheets as a back end. the more time, effort, and infrastructure that was put into a project, the more invested people feel in them and the less likely they are to be willing to call it a failure and move on or make drastic revisions. so often projects are propped up by a sunken cost fallacy long after their lack of effectiveness has become clear, but these development approaches free projects from this burden and allow them to be assessed more faithfully. the code for this project can be found at: https://github.com/vtul/proto-tool-finder. about the author jonathan bradley is the assistant director for learning environments and innovative technologies at the university libraries at virginia tech. in this position, he oversees the libraries’ six studio spaces, which focus on technologies ranging from virtual reality to media production to maker tools. he also does web development for the studios, creating various tools to gather feedback from patrons and improve the service point experience. he earned his doctorate from middle tennessee state university in 2013. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – partnering for discoverability: knitting archival finding aids to digitized material using a low tech digital content linking process mission editorial committee process and structure code4lib issue 34, 2016-10-25 partnering for discoverability: knitting archival finding aids to digitized material using a low tech digital content linking process as libraries continue to ramp up digitization efforts for unique archival and special collections material, the segregation of archival finding aids from their digitized counterparts presents an accumulating discoverability problem for both patrons and library staff. for utah state university (usu) libraries, it became evident that a system was necessary to connect both new and legacy finding aids with their digitized content to improve use and discoverability. following a cross-departmental workflow analysis involving the special collections, cataloging and metadata, and digital initiatives departments, a process was created for semi-automating the batch linking of item and folder level entries in ead finding aids to the corresponding digitized material in contentdm. in addition to the obvious benefit of linking content, this cross-departmental process also allowed for the implementation of persistent identifiers and the enhancement of finding aids using the more robust metadata that accompanies digitized material. this article will provide a detailed overview of the process, as well as describe how the three departments at usu have worked together to identify key stakeholders, develop the procedures, and address future developments. by liz woolcott, andrea payant, sara skindelien introduction utah state university (usu) library’s special collections and archives and digital initiatives departments work together to provide scans of archival material on request to patrons in the special collections’ reading room. one problem that repeatedly surfaces is the request to digitize resources that are already scanned and available online. what usu library found was an emerging pattern, where neither the patrons nor the staff members serving on the front lines (such as the archive’s reference desk) were consistently aware of what resources were already available in digital format. despite efforts to educate staff about local resources and also to make digital collections more discoverable by contributing them to online discovery portals such as the mountain west digital library (mwdl) and the digital public library of america (dpla), usu’s digital program contained blind spots with key potential access points that remained untapped. from both a staff and patron perspective, reasons for this issue are readily apparent. usu special collections staff, who assist patrons in their research, primarily utilize the online finding aids as their chief research tool.[1] [2] finding aids are hosted online and linked from the usu special collections’ homepage. however, usu’s digitized archival material is housed in a separate content management system (contentdm) and cannot be searched in conjunction with the finding aids. to add to this, the online finding aids rarely contain any notation that a digital component exists. patrons and staff need to reconstruct their original searches once again within contentdm in order to see if a digital version is available. as the oft cited 4th law of ranganathan states, the library should save the time of the reader (or, in this case, the user) (ranganathan, 1931). requiring duplication of searches to accommodate the cobbled together infrastructure for digital and physical formats of archival material breaches the spirit of this law. in order to bridge the gap between the digital content and the archival description, the special collections and archives staff and the digital initiatives staff needed to work together to make digital materials accessible from multiple entry points in the simplest possible manner. background usu library formed a special working group to address workflows between special collections and archives and digital initiatives. the working group’s name was shortened to the more tongue-in-cheek acronym of “scad.” at the time that the scad working group was investigating workflows, metadata creation was coordinated by the digital discovery librarian, but the day-to-day metadata work was split between three departments: cataloging and metadata services, digital initiatives, and special collections and archives. at the same time, the library was also experimenting with a multi-departmental team that was cross-trained in the standards and practices of all three areas.[3] this strategic works and tactics (swat) team was organizationally housed within special collections and archives, and trained in machine readable cataloging (marc), dublin core, and encoded archival description (ead). members of the swat team were charged with focusing on the backlogs in all three departments with the end goal of using their holistic training to provide feedback on how the library could streamline the workflows between the departments. this three-member swat team represented half of the scad working group with the other half composed of a digital initiatives assistant, an archival ead specialist, and the digital discovery librarian who chaired the group. figure 1. diagram of the scad working group over the course of a year, the group analyzed the stages of collection description at usu libraries from the initial archival processing to the final metadata creation for the digitized surrogates. the group briefly reviewed archival management software, explored some training opportunities, and ultimately made several recommendations about practices that could be streamlined or improved. there were a lot of potential avenues to explore, so the group narrowed its focus to streamlining metadata workflows. the main product that came out of the scad working group effort was identifying the necessity of linking the online finding aid to the digital copy (otherwise referred to as digital content linking) and the development of a process to create those links both individually and in batches. technological considerations described below is the process that was developed and is currently in use by utah state university libraries. before diving into that process, a few structural and technological background items should be noted. first, the process outlined below is intended to be “low tech” wherever possible in order to reduce the technological barrier for staff and especially for the student help who do the bulk of the processing and inventory creation. procedures requiring additional training or education were found to be too cumbersome to maintain long term with the naturally high turnover rates from graduating student workers. turnover rates in front line staff are also high enough to make extensive or highly technical training unsustainable in the long run. second, utah state university library does not currently and has not historically employed any archival management software, although there is a plan for migrating to archivesspace in the next two years as part of a consortia project.[4] management of archival material is carried out through a combination of processes beginning with accessioning collections using an access database, proceeding to physical processing and inventory creation, and ending in the construction of ead finding aids which are stored locally on a shared drive but contributed to a consortium for hosting and public viewing. third, all finding aids are hand coded using the ead 2002 xml standard.[5] there is no set process for creating ead finding aids at usu. the process differs depending on which archival team does the processing. when developing the digital content linking process, the scad working group had to be cognizant of the various ead creation styles employed by the archival teams. fourth, the usu special collections and archives department is part of the archives west consortium and adheres to the best practices guide developed by the content creation and dissemination program of that consortium.[6] the archives west consortium is also responsible for developing the style sheet that interprets the xml structure of the finding aid and renders the final display for the viewer. the digital content linking process that was developed had to adhere to both the standards of the best practices guide and also respect the capabilities of the style sheet for the consortium. linking digital objects to corresponding items within ead finding aids tools required: microsoft excel microsoft word xml editor (such as oxygen) this is an overview of the process which utilizes robust digital collection metadata to create ead finding aids linked to digital objects at the item or folder level using the <dao> tag.[7] there are three main steps to the process: first, preparing and exporting digital collection metadata. second, editing metadata to repurpose it for digital content linking. third, using the mail merge function in microsoft word to automatically create an xml format container list that can be pasted directly into the primary xml document for the finding aid.[8] while there are undoubtedly scripts that can perform a number of the steps presented here, this process was intentionally designed to be “low tech,” requiring only familiarity with or training on spreadsheets, microsoft word mail merge functions, and a small bit of xml. this decision was reached by the scad working group after realizing one of the largest barriers to performing this process was the constant re-training of staff and student workers who have high turnover rates. this process was also more flexible when it came to adapting to the variety of hierarchical depths found in legacy finding aids. process overview [9] step 1. prepare and export collection metadata from the digital asset management system in a format that can be read as a spreadsheet. metadata for digital objects need to include the following fields: title, date, format, call number (or at minimum the collection, series, box, or folder information), and most importantly, the url to access the item. in the contentdm database, this is found in the reference url field and is automated by the system. usu library chose to use an independent persistent identifier through the ezid service, as a best practice for long term sustainability when linking digital content, instead of relying on the reference url supplied by contentdm. should there be a system change, staff will not need to individually modify each of the ead guides, catalog records, exhibits, or other haphazard places where digital content might be linked. instead, staff can simply update the ezid database with the current location of the item and all other links will redirect correctly (deridder, presnell, and walker, 2012). ezid is a subscription based service provided by the california digital library that mints dois and arks as unique identifiers. to create a unique identifier, a metadata creator enters basic information about an object (name, date, creator, etc.) along with the object’s url location.[10] once entered, the ezid system generates both a unique identifier in the form of an ark (e.g. ark:/85142/t4059n) and a url which incorporates the identifier (e.g http://n2t.net/ark:/85142/t4059n). when a user clicks on the url generated by ezid, it redirects to the current location of the digital item. usu libraries has traditionally embedded links to digital content in marc records, ead finding aids, related dublin core records, online exhibits, social media posts, etc. we are currently transitioning existing urls for digital content to the ezid-generated url in order to manage links with one service. if an independent identifier is going to be used, it needs to be generated and recorded in the metadata for each item before the digital linking process begins. for the purpose of our process, we needed to be able to assign ark ids for single items or batches of items, depending on the content we were linking. for single items, the process is very straightforward – we simply filled out the ezid online form with the title, creator, date, and url for the item and received back the ark and url, which were then inputted in the metadata record. for batches of items, however, usu’s systems administrator developed an interface through the ezid api to load the same fields (title, creator, date, and url) for a batch of items. the ezid service then returned a csv file with the newly minted arks and urls.[11] once created, the ark and new url are entered into the item level metadata by a student employee. while this process of assigning ezid arks to each digital item is extremely useful in many ways, it should be noted that for the digital content linking process presented here, simply using the system-supplied url will also work. after ensuring the title, format, date, call number, and url fields were correct, we exported the metadata for the collection from the digital asset management system to a tab-delimited or text file. this process varies depending on the digital asset management system used. for the bulk of the collections that usu libraries will be linking, the metadata is downloaded from contentdm.[12] this file was then opened in an excel spreadsheet for ease in editing. step 2. edit the metadata for ead we edited the spreadsheet to only include the title, format, date, call number, and url fields necessary for the ead finding aid and the digital content linking. based on the complexity of the collection’s hierarchy, we inserted five empty columns at the beginning of the sheet and titled them as follows (*columns with an asterisk could vary depending on the hierarchical depth of the physical collection): component level component number box* folder* item* next, we separated the call number information into series, box, folder, item, etc. there are several excel formulas that help automate this. when all of the separated information was sorted into the appropriate columns and rows, we copied and re-pasted the columns as values, since formulas are problematic to work with in excel. we then sorted the entries in call number order to reflect the hierarchical structure of the physical collection. in the component level column, we indicated the component type using the approved ead level elements.[13] in the component number column, we indicated the component level number (i.e., <c01> through <c12>.) however, we only listed the variable number and did not include the “c” or “c0”portion of the element. (this was added at a later stage in the process.) figure 2. adding component level and component number information (note: this particular example has no hierarchy) next, we added a column at the end with the header: “resource label.” underneath that column header, we entered the text “click to access” for each row in the column. this is the hyperlinked text that is displayed to the user. the final spreadsheet had the following columns (*columns with an asterisk varied depending on the physical collection’s hierarchical depth) component level component number box* folder* item* title format date url resource label finally, we added additional rows as necessary to create level information for series, boxes, or folders that are not digital objects, but would serve as headers in the ead finding aid. (see rows 2 and 3 in figure 3 below for an example.) figure 3. example of a final spreadsheet (using collection data as seen in figure 1 – note: this collection has a hierarchy). click image for larger view. step 3. use the mail merge function to create a new xml container list with links to digital content once finished making the necessary edits to the spreadsheet, the next step was to utilize the mail merge function in microsoft word to create a new xml container list for the finding aid embedded with the new links to digital content. to begin, we opened a new word document and inserted an xml template like the one listed below. this template represented the xml coding needed for a single item in a collection. we were sure to include the digital archival object and any attribute tagging necessary for the content linking to operate effectively. template for use with xlink namespace: <c0«component_number»level="«component_level»"> <did> <container type="box">«box»</container> <container type="folder">«folder»</container> <unitid>«item»</unitid> <unittitle encodinganalog="title">«title»</unittitle> <daogrp> <resource xlink:label="start">«resource_label»</resource> <daoloc xlink:label="image" xlink:href="«ark_url»" xlink:title="digital image of «title»" xlink:role="«format»"/> <arc xlink:form="start" xlink:to="image" xlink:show="new" xlink:actuate="onrequest"/> </daogrp> <unitdate>«date»</unitdate> </did> </c0«component_number»> template for use without xlink namespace <c0«component_number»level="«component_level»"> <did> <container type="box">«box»</container> <container type="folder">«folder»</container> <unitid>«item»</unitid> <unittitle encodinganalog="title">«title»</unittitle> <daogrp> <resource label="start">«resource_label»</resource> <daoloc label="image" href="«ark_url»" title="digital image of «title»" role="«format»"/> <arc form="start" to="image" show="new" actuate="onrequest"/> </daogrp> <unitdate>«date»</unitdate> </did> </c0«component_number»> the component_number, component_level, box, title, ark_url, title, format, and date elements in the templates above represent where the data from each column in the spreadsheet will be placed. as each row in the spreadsheet represents a new digital item, it creates a new <c0> unit for each row. using the mail merge function in word, we assigned columns from the spreadsheet to the corresponding ead elements in the xml template and ran the merge. this produced a new word document. the xml coding for each of the items in the collection was visible with information inserted from the spreadsheet. one entry was displayed on each page of the document. we then removed the extra blank space between entries by using the find and replace function in word (using ^b to find blanks, and replace with ^|^|) to insert two empty lines or manual line breaks between the entries. the end result looks like this: figure 4. a completed xml container list in word. once finished, we copied the container list in word and pasted it into the <dsc> section of the xml file for the collection’s finding aid. depending on the collection’s need, we might overlay the entirety of the previous <dsc> content or just insert individual links (or smaller groupings of links.) we then performed quality control on the xml. note: if the digital collection is a not a 1-to-1 relationship with the physical collection, some edits may need to be made. it may be advisable to only copy and paste specific sections of the newly minted xml. if the digital content only reflects random individual items in a collection, the process would need to be adjusted to copy and paste single items at a time. figure 5. the finding aid complete with links to digital objects in archives west (note: folder information is not seen here because it was added in the above procedures for demonstration purposes only. for this collection, “1972.001.003” is a single item number and not a box or folder number.) notes on the benefits and considerations of digital linking this batch process can be used in two ways to provide digital content links: to formulate a brand new xml container list for an ead finding aid or to update an existing container list. with regard to updating an existing ead container list, this process is most accommodating for digital collections that share a 1-to-1 relationship with the physical collection, where all items in the physical collection have been digitized and are housed in one single digital collection. digital collections that contain more than one physical collection will be more problematic and require additional steps to sort. likewise, digital collections with content that comprise only a portion of a physical collection, instead of all of it, will also require additional steps. while the overall benefit of creating digital links between a finding aid and digital images may appear quite obvious, there were several elusive benefits and considerations we uncovered as we developed this process. robust finding aid inventories prior to standardization through the advent of dacs (describing archives: a content standard) or ead (encoded archival description), finding aids were as unique as the people arranging and writing them. these legacy finding aids were often composed with seasoned researchers as their target audience—researchers who journeyed into special collections to view materials first hand. the digital age has changed all that and the expectation to have more materials in a digital format has increased. by linking individual items from an archival finding aid to a digital object, researchers are not only provided with the organizational context of archival materials found in a finding aid, but also a visual object that otherwise may not be easily accessible. described by zhang and mauney (2013) as the parallel model, digital linking on the item, folder, or box-level allows the archival materials to be presented in two distinct ways without forsaking one format for another. the finding aid preserves the contextual information researchers need to understand provenance and historical information about the collection, while the metadata associated with the digital file provides more detailed, granular, and controlled descriptions that can be faceted within and across collections to tie disparate items together. this process allows us to retain the benefits inherent in both of the parallel systems we were using and also unites them together at the point of discovery. without any training at all, front line staff or users who have stumbled upon the ead guide can access the digitized version of an item without having to re-do their searches in a separate content management system (assuming that they know that separate system exists in the first place.) due to the nature of descriptive metadata creation, titles for digital assets are often more detailed than what is recorded in the ead guide. when digital asset metadata is transformed into an ead environment, these more detailed titles are transferred with them and provide important keywords to improve discoverability. this process can serve as a complement to the archival “more product, less process” or “mplp” philosophy. (greene and meissner, 2005) material can be inventoried in an ead guide using the mplp process, but then be iteratively upgraded using metadata descriptive practices when a collection is deemed worthy of digitization. new accessioning opportunity the digital content linking process also allows for the “quick” creation of finding aids for newly acquired or born-digital collections. usu libraries experimented with taking newly created born-digital objects from an oral history project conducted by our fife folklore archive and accessioned them using the digital content linking process. in effect, the digital files were uploaded to contentdm and described by metadata specialists rather than an archival processing team. once completed, the metadata was exported, edited as described above, merged into an xml template using the mail merge process and then inserted into the portion of a brand new finding aid. the curator then created the “front matter” (or background, scope and content, and other information regarding the provenance of the collection.) this streamlined process provided a new potential model for accessioning collections where metadata specialists and curators worked in concert to bring both the digital collection and the finding aid online simultaneously. of course, this will not work for all newly acquired collections, particularly large ones or collections where patron demand is unknown, but it does provide a possible new model for smaller collections where high patron demand is anticipated. feedback in may of 2016, the authors presented this process at the conference of inter-mountain archivists (cima) as part of a panel on re-purposing archival metadata. immediately following the presentation, we received multiple requests for the step-by-step process instructions as many institutions were interested in how we maintained a low technological bar by using the programs and software they also use. library and archival staff have also been quite receptive to the process of inserting digital links between finding aids and digital content. they appreciated the fact that student training is very minimal and the tutorial developed for this process allows for quick training sessions, particularly since the student employee turnover rate is high in the academic world. as a result of this staff support, we have begun to collaborate with other departments outside of the archives to train more student workers on the process. to manage the “backlog” of finding aids needing digital linking, we organized our finding aids into three phases according to the level of complexity. phase i is the direct one-to-one matches between the finding aid and the digital collection (where a single archival collection has been digitized and put online as a single digital collection); phase ii includes all of the digital collections that contain multiple finding aids (where a single digital collection has items from more than one physical archival collection); and phase iii is comprised of digital collections that do not have a corresponding finding aid but are excellent candidates for the reversal process of creating a finding aid inventory from the digital collection metadata while simultaneously inserting digital links (described in the “new accessioning opportunity” listed above). conclusion digital content linking provides the missing step between ead finding aids and digital content that is housed outside of the ead environment. it allows users and library staff to know which items in a collection have been digitized and brings them directly to the digital objects. providing cross linking of finding aids to digital content and vice versa creates cyclical access to accommodate users from any discovery point. creating digital content links one by one may be appropriate for situations where only select items in a collection are digitized, but for collections that have large portions of content digitized, bulk procedures are the most efficient means for creating access points. the procedures outlined in this article can prove to be beneficial by adding valuable information to the ead finding aid. however, changes may affect the look and feel of the finding aid and should be discussed with archivists and curators before proceeding. additional discussions about the appropriate level to digitize and describe an item are also imperative. while these procedures were developed for archival collections with existing finding aids, they were also found to be easily adapted to help streamline workflows for born-digital content. notes [1] also called inventories, finding aids, finding guides, registers, etc. in this paper, we will refer to them as finding aids for easy reference. they are often marked up in encoded archival description (ead ) xml. when discussing the xml, we will refer to it as an ead finding aid. [2] it should also be noted that patrons are also accessing the finding aids when not in the reading room, as well. analytics for finding aid access shows that they are receiving a robust amount of traffic off campus. [3] the swat team pilot project was reported on at the american library association annual conference in 2014. to see more information on the structure and planning of it, please visit: http://digitalcommons.usu.edu/lib_present/61/ [4] this migration will change the procedures outlined below in several substantive ways, but we anticipate that much of the process can still be retained after adapting it to the archives space requirements. [5] in concert with partners in the archives west consortium, usu library will not be migrating to the new ead3 standard until after 2018. [6] the best practices guide is available here: https://www.orbiscascade.org/file_viewer.php?id=2923 [7] to see the specifications for the tag in ead 2002, please visit: https://www.loc.gov/ead/tglib/elements/dao.html [8] the mail merge process was not created by utah state university, but was shared by the utah state archives with usu libraries in 2008. it can be found here: http://archives.utah.gov/research/inventories/ead.html [9] to see the step-by-step instructions for this process, please visit the usu cataloging and metadata services blog: https://usucataloging.wordpress.com/2016/09/26/digital-content-linking-workflow/ [10] you can find out more about ezid at their website: http://ezid.cdlib.org/ [11] this code is currently not publicly available. usu libraries is investigating the possibility of opening the source code in the future. [12] this process will vary depending on the digital asset management system. for contentdm users, the process is outlined here: https://www.oclc.org/support/services/contentdm/help/collection-admin-help/exporting-metadata.en.html [13] ead tag library: https://www.loc.gov/ead/tglib/att_gen.html references deridder, j. axley, a.a. and walker, k.w. (2012). leveraging encoded archival description for access to digital content: a cost and usability analysis. the american archivist, 75, pp. greene, m., & meissner, d. (2005). more product, less process: revamping traditional archival processing. the american archivist, 68(2), 208-263. ranganathan, s. (1931). the five laws of library science. madres library association, p. 337. available at: http://hdl.handle.net/2027/uc1.$b99721?urlappend=%3bseq=383 zhang, j. and mauney, d. (2013). when archival description meets digital object metadata: a typological study of digital archival representation. the american archivist, 75. pp. 174-195. about the authors liz woolcott liz.woolcott@usu.edu liz woolcott is the head of cataloging and metadata services for utah state university libraries. she has worked in cataloging and metadata coordination for the last 12 years and is the co-founder of the library workflow exchange. andrea payant andrea.payant@usu.edu andrea payant is the metadata coordinator for utah state university libraries and has worked for usu for 10 years. her most recent work has focused on data management and consulting for library staff as well as students and faculty campus-wide. sara skindelien sara.skindelien@usu.edu sara skindelien is the special collections and archives library assistant and specialist in ead for utah state university libraries. she holds an ma in public history from the university of northern iowa. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – editorial: new editors, diversity, and representation mission editorial committee process and structure code4lib issue 44, 2019-05-06 editorial: new editors, diversity, and representation welcoming new editors, new surveys, and thinking about diversity at code4lib journal. by junior tidal, issue 44 coordinating editor code4lib journal welcomes two new editors to the editorial committee. brighid gonzalez(our lady of the lake university) and carolyn moritz(vassar college libraries). the process of reviewing and discussing articles, managing the journal, and contacting authors is an arduous one and we welcome their hard work and expertise. there are also plans to have another call for code4lib journal editors, specifically looking for editors outside of the united states, which will be posted soon. coincidentally, the last time we had new editors come aboard, sara amato wrote a follow-up to ron peterson’s examination of diversity in the journal. this seems like an appropriate time to look at the makeup of authors and editors. looking at the editorial committee, members have self-reported that 63% identify as white. gender balance remains unchanged with 40% of editors identifying as female and 60% identify as male. based on collected data, the majority of editors hail from academic institutions, with public libraries, other research institutions, and vendors in the minority. when authors contribute to the journal, they are given an optional demographic survey that collects information including ethnicity/cultural identity, gender, location (country), disability, and institution type. the data collected also comes from contributors whose articles may not necessarily be published due to rejection or lack of editors. looking at this data, not much has changed since sara explored it 2 years ago. data that has been disclosed indicates that a majority of authors (52%) identify as white (see figure 1), with the second most largest group that responded to the survey indicated “no response.” contributors’ self-reported gender identities show that this has slightly changed. the majority of contributors identify as male (see figure 2), yet 2% of respondents have identified as neither, non-binary, or other. on the survey, this particular field is an open text field, with suggestions listed as “female, male, neither, other, both.” data also shows that authors mostly come from the united states (69%), are from academic institutions (70%), and do not report or have no disabilities (98% of respondents). figure 1. self-reported ethnicities/cultural identities of code4lib journal contributors. figure 2. self-reported gender identities of code4lib journal contributors. contributors. now that we have this data, what can we do with it? while the editorial committee is discussing this, we were also looking into what the purpose is to gather demographics from contributors in the first place. i think collecting this data will illuminate faults in our lack of inclusiveness, and may also reveal how the committee can support a wider range of diverse voices in code4lib journal. to support this, the editorial board is revamping the demographics surveys for both authors and editors. our initial surveys had flaws in need of repairing. for instance, it has been suggested that the gender identification field language has been updated from changing the language from “other” to “non-binary, third-gender, self-describe, prefer not to say.“ the location field initially only had the options of either “united states” or “other,” paired with an open text-field. this will be altered away from an americentric line of questioning to a drop-down list of countries. we have changed the question of respondents’ self-reported disability to a yes or no answer. there will also be other changes to the survey which will be used after the publication of issue 45. revisiting the survey at periodic points will be valuable, so that other points of data can be amended to better examine our community. code4lib journal has a wealth of invaluable knowledge to the information professional community, and i hope that it reflects the inclusive values of that community. as a librarian of color, i feel that representation is not just important, it is essential. the work that information professionals conduct, especially those working within technical and technological services, can at times be invisible. this labor may be invisible, but those behind the scenes do not have to be. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – getting what we paid for: a script to verify full access to e-resources mission editorial committee process and structure code4lib issue 25, 2014-07-21 getting what we paid for: a script to verify full access to e-resources libraries regularly pay for packages of e-resources containing hundreds to thousands of individual titles. ideally, library patrons could access the full content of all titles in such packages. in reality, library staff and patrons inevitably stumble across inaccessible titles, but no library has the resources to manually verify full access to all titles, and basic url checkers cannot check for access. this article describes the e-resource access checker—a script that automates the verification of full access. with the access checker, library staff can identify all inaccessible titles in a package and bring these problems to content providers’ attention to ensure we get what we pay for. by kristina m. spurgin, e-resources cataloger, university of north carolina at chapel hill introduction libraries have historically exerted significant control over their print (and other tangible format) collections. traditionally, libraries have had print workflows involving checking in the individual physical items purchased to ensure all ordered materials were received; cataloging each item to provide intellectual access to the material via the library catalog; physical processing (affixing spine labels, barcodes, security devices) to prepare the material for library use; [1] and shelving each item somewhere in the library to provide physical access to the material. our workflows for processing electronic (e-) resources tend in part to mirror the ones we use for processing print, but the scales are vastly different. most libraries that provide access to e-resources obtain many of them by purchasing or subscribing to e-resource packages. each package may contain hundreds (and sometimes hundreds of thousands) of titles—the individual e-books, online journals, streaming media, online datasets, and other electronic resources we hope our patrons want. [2] when purchasing or subscribing to e-resource packages, libraries cede much of their traditional control to content providers, mainly due to the size of e-resources packages and the growing need for libraries to do more with less. to give a sense of the scale we are talking about, here is a snapshot of what we are dealing with at my library. as the e-resources cataloger at the university of north carolina at chapel hill libraries, [3] i keep track of about 700 separate e-resources packages. [4] these packages were represented by about 1,250,000 records in our catalog in our last yearly catalog record count-by-package, conducted in july, 2013. the smallest packages had only two associated records; the largest package had 312,464. in the first four months of 2014, i loaded 112 separate batches of catalog records for 36 different packages of e-books, e-journals, streaming media, and web resources—a total of 102,252 new catalog records added. this means my two-person section has been responsible for acquiring, checking, batch editing, and adding to our catalog over 25,000 marc records per month on average this year. smaller libraries may have smaller numbers of e-resources to handle, but they will also likely have fewer people to work on e-resources management. perhaps one person will be tasked with handling the purchasing, licensing, cataloging, and troubleshooting of e-resources on top of other job duties. in no library does anyone have the ability to manage e-resource packages at the individual title level. and yet, our patrons are looking for those specific titles relevant to their needs and interests. workflows for processing electronic resources checking in the contents of an e-resource package entails verifying the receipt of every individual title included in the package based on a title list made available by the content provider. in the context of e-resources, “to receive” a title means that the content provider has given us—and our patrons—full access to the online content of that title: we can read the whole book, watch the entire film, etc. unlike with print resources, we cannot manually check that we have “received” each of hundreds or thousands of titles included in a package. usually we look at a few of the titles we know are part of a package to verify that the provider has activated our access to that package and then move on. we trust that the content provider is indeed giving us full access to all the titles we have paid for. providing intellectual access to the contents of e-resources packages still happens via the library catalog. [5] we typically rely on content-providers or other 3rd party vendors to provide batch-files of catalog metadata for the individual contents of e-resource packages. usually we deal with multiple batches of metadata for a given package as metadata are gradually created and/or new titles are added to a package. unlike with print materials, we do not have the resources to compare each metadata record with the item it describes. we trust that we will eventually receive records for all of the titles included in a package and that we are not receiving records for titles our patrons won’t be able to access. shelving e-resources obviously does not happen literally, but we do rely on content providers to host the individual titles within packages so they are accessible to our patrons. the content providers maintain the shelves, as it were. we trust they will arrange the titles in some sensible manner and maintain consistent arrangement of those titles. whatever link resolving and redirecting content providers implement, we trust that the urls we are given in title lists and metadata records will continue to point our patrons to the full content of titles, much as call numbers point patrons to specific books on shelves. finding e-resource access problems the scale at which libraries routinely purchase e-resource packages and the very nature of electronic resources require that we shift much of the responsibility for processing resources onto content providers. this also largely precludes us from checking over the results as thoroughly as we do with print resources. for our own sanity, we often have to assume that content providers have processed things correctly and that the resources we’re paying for work. however, we know from experience that this is not always the case. sometimes we stumble over problems ourselves; other times we find out about access problems from patrons. when a properly-authenticated patron reports that they have hit a paywall or other “you do not have access to this content” barrier within a resource we’ve paid for, it puts us in a reactive position. content providers usually resolve known-and-reported access problems quickly, but we are still nagged by the suspicion that what we’re aware of is only the tip of the iceberg. trust alone is clearly not a viable option: we need a way to verify that we actually have access to everything we’ve purchased. this is why i’ve developed the e-resource access checker script, a ruby script that automates checking availability of full access to electronic resources. it allows us to take a more proactive approach to finding access problems at each stage of our e-resource workflow. first, the access checker provides an automated method for verifying full access to titles within a package based on a title list containing the url for each individual title. in this sense, it can be conceptualized as a tool for checking in the contents of e-resource packages. note that it is called an “access checker” because it goes beyond simply checking urls, which would not reveal access problems per se. this difference is described more fully below. second, it can help resolve some metadata problems. while it can’t find missing metadata records, it can identify individual records within a batch that link to resources lacking full access. it can also flag records containing invalid urls or doi-based urls that fail to resolve. third, it can help find problems for resources that are already “on the shelf,” so to speak. the nature of e-resources means that access problems may arise at any time, even if we verify that we have full access to all titles in a package at the time of purchase. the content provider may have server trouble. errors may be introduced in our proxy or ip configurations. titles may be removed from packages, even if we have purchased “perpetual access,” and some providers are more proactive than others about notifying libraries about such withdrawals. if we have catalog records linking to titles in such situations, these titles are effectively missing from the shelves. via a kind of virtual shelf-reading of a particular package’s contents, the access checker can help determine how widespread access problems are for that package. it can also help identify which existing catalog records may need to be suppressed from view or deleted from the catalog because the “volumes” are now missing. verifying full access vs. verifying urls having full access to an e-resource means you can use that resource in its entirety: e.g., you can read the whole book or watch the whole video. if you can only read table of contents and the first chapter of an e-book, you don’t have full access to that e-book. if the url in a catalog record takes you to a doi-resolution error or a “no item found” message within the content provider’s site, you effectively do not have access to that e-book unless you think to search the provider’s site for the title (and know how to do so). note that all of the problematic situations in the previous paragraph involve the patron being pointed to valid web resources. for this reason, integrated library system-based or stand-alone url checkers do not solve the problem addressed by the access checker. these url checkers mainly report whether a url points to a valid web resource or not based on the http status code the server returns (404, 502, etc). but e-resource content providers rarely (if ever) use http status codes to indicate whether or not an e-resource is accessible. when they provide restricted-access messages about their e-resources, they generally send them with a 200 (ok) status, so traditional url checkers fail to report them as problematic. furthermore, a url checker might report that a url redirects to another url, but it won’t actually follow the redirect. with e-resources, we expect that most urls from title lists or in marc records will redirect to a different url on the content provider’s platform, so telling us that a url redirects is uninformative. we need to know what content is found at the end of the redirect chain. in 2013, the code4lib journal published an article describing a tool called normac, which includes an access checker that works on the same principles as the one described here. [6] this was the first i had heard of any other solution for automated access verification, and i had developed the first versions of my access checker in late 2010. normac’s access checker is part of a more complex tool to automate e-resource metadata batch processing, and requires users to configure their own access checking logic profiles. the access checker i’ve developed is a simpler solution that staff at any library can use to verify access to their e-resources (on the supported platforms) without any configuration. the downside to this is that users of this access checker who want to check access on platforms not currently supported need either a) to edit their version of the script locally; b) to contribute changes to the script via github; or c) to ask me to add support for the desired platform (and wait for me to make that happen). how the e-resource access checker works on many e-resource platforms, the descriptive landing page for an e-resource to which a patron has full access includes a clear indication of its accessibility: a green check mark, for instance, or a text string such as “access to this e-book is provided by…” or “full text.” this access signifier is different on each content-provider platform—and even sometimes different for individual packages offered by the same content provider. the presence of the access signifier (or the string representing it in the html source) is what the access checker pays attention to. however, verifying access isn’t always a binary decision, and it isn’t always as clear as affirming the presence of a positive indicator of full access. on some platforms, there is no positive access signifier; in these cases, the absence of an error or “you do not have access” message is assumed to indicate full access. in some packages, we see multiple types of access problems. springerlink is a good example of this. typical access problems on this platform fall into four categories: a url points to the e-book title on springerlink, but access is restricted (only the table-of-contents and front matter are available). a url points to a “not found” page within springerlink (which returns a 200 (ok) http status code). a doi-based url does not resolve and points to “doi not found” page. a doi-based url resolves to the e-book on a partner publisher’s site, where our successfully authenticated users are asked to purchase the book in order to view its contents. resolving each of these types of access problem requires different steps, so we need to know which category a given problem falls into. the e-resource access checker is a simple jruby script [7] that automates checking the access status of individual e-resource titles. you run it from the command line using the input and output files as parameters: > jruby access_checker.rb urls_to_check.csv access_results.csv the access checker takes input from a .csv file, which may contain any number of columns (title and record id are commonly included). the last column must contain one url associated with the title, [8] and all the urls in the file should be on the same content-provider platform. the title lists obtained from content providers are usually spreadsheets that are easily saved as .csv files for use as access checker input. bibliographic data and urls can also be extracted or exported from marc records in a spreadsheet-like format easily convertible to .csv from marcedit and some integrated library systems. when a user starts the script, she is presented with a list of platforms supported by the access checker and must type which platform she wants to check for access. the script checks the html source code that each url points to for appropriate access-signifier string(s) for the specified platform. if a string matches, the corresponding access result is returned. if no match is found for any of that platform’s access signifiers, the script returns the result: “check access manually.” details on the access signifiers used for the platforms currently supported by the access checker are found in the table below. while the access checker is running, the result for each url is printed to the screen so the user can get a general sense of the progress and the severity of the access problems. the access result for each url is also appended to the output .csv file. the output file is equivalent to the input file, with a new “access” column added to the end. we have been using the access checker heavily at unc chapel hill since late 2010. it has been used successfully by at least three other institutions. access checking for ebscohost ebooks was added at the request of a colleague at one of these institutions just before unc began purchasing titles on that platform. platform string to match access result returned if matched apabi type="onlineread" access probably ok alexander street press page not found page not found alexander street press error error returned alexander street press browse full access duke university press (highwire) doi not found doi error duke university press (highwire) log in to the e-duke books scholarly collection site no access duke university press (highwire) t-page-nav-arrows full access ebrary document unavailable. no access ebrary date published full access ebscohost class="std-warning-text">no results no access ebscohost ebook full text full access sciencedirect <td class=nonserialentitlementicon><span class="sprite_nsubicon_sci_dir" restricted access sciencedirect title="you are entitled to access the full text of this document" full access sage knowledge page not found no access – page not found sage knowledge add to my lists probable full access sage research methods page not found no access – page not found sage research methods add to methods list probable full access springerlink viewtype="denial" restricted access springerlink viewtype="full text download" full access springerlink doi not found doi error springerlink bookshop, wageningen wageningenacademic.com serialssolutions ss_nojournalfoundmsg no access indicated serialssolutions ss_holding access indicated university press scholarship <div class="contentitem"> full access wiley online library doi not found doi error wiley online library you have full text access to this content full access table 1: access-signifier strings checked for each platform. decisions made and challenges faced in development use of celerity and jruby i began writing the access checker in ruby while teaching myself the language, but i ran into many errors i didn’t know how to resolve when using its standard libraries to handle the urls and the content they returned. most content provider urls redirect at least once before landing on the page a patron sees in her browser. further, what the patron sees is often rendered after javascript or some other wizardry happens at load time. some content provider sites are very slow to respond and would time out under the standard libraries. the access checker also needed to work with urls containing proxy strings, which were problematic for the standard libraries. my research turned up a jruby library called celerity that solved the above problems. “celerity is a jruby wrapper around htmlunit – a headless java browser with javascript support. it provides a simple api for programmatic navigation through web applications.” [9] essentially, celerity allows your program to “see” a web page as an end user would see that page in a browser, so it can test for the presence of elements in a final rendered web page. using celerity, i could still write my script in ruby; it only required i install and run the script in jruby. the major downside of using celerity is it adds a lot of complexity to an otherwise extremely simple program. [10] sometimes the processes handled by celerity fail in ways i haven’t been able to address in the access checker, partly because they don’t always fail again in quite the same way if tried again a minute later. sometimes these failures cause the access checker to stop running. to mitigate this issue, the access checker writes the access result for each resource in the input file to the output file as it is checked. this way, if the script fails, you can just remove the already-checked resources from the input file and restart the script. if you give the same output file when calling the script, it will not be overwritten; the rest of the access results will be appended to it. this is also handy if you need to verify access to a large number of resources—you can break your input files down into smaller lists for checking [11] but write all the results to one file. creating a well-behaved script from the beginning, i was concerned with creating a tool that could be used without drawing the attention or ire of our content providers. the access checker is fairly slow due to the way celerity works. each page is loaded into the virtual java browser behind the scenes, one at a time, and checked for access. the access checker also adds a brief sleep time after checking each url. this sleep time is currently hard-coded into the script and is the same for all platforms, but i have considered allowing it to be set interactively at run-time or configured per platform. the access checker’s behavior resembles that of a preternaturally focused person clicking through a list of links more than a typical link checker or web robot. furthermore, for all but one platform, the access checker never touches the actual full content of the resource. only the descriptive landing page for the e-resource is visited. the exception is duke university press ebooks via their new highwire platform. at the time i added access checking logic for this platform, i found no differences in the source code of the landing page for a resource to which we had full text access, and one we had no access to. the only way i could manually check for access was to click the “read now” button on the landing page and see if i was asked to log in (no access), or shown the first page of the book (full access). the access checker approximates this process by requesting page 25 of each title and seeing checking for access signifiers on the resulting page. i have requested of duke university press that they add a visual indicator of full text access to the e-book landing pages. ideas for improving the access checker (or building a better one!) the access checker was one of my first real coding projects, so i’ve been very pleased to make something that solves problems and meets needs at my institution and beyond. however, especially as one of my first projects, the tool can certainly be improved. i welcome suggestions, collaborators, and other people running with this “access checking” concept to create much nicer tools than i have the skills to build. some of the possible improvements that loom largest in my mind follow. earlier i described the occasional and seemingly random errors thrown by celerity that sometimes cause the access checker to fail. while the access checker is designed so that you don’t lose the results of any already-checked resources, this sort of failure lacks grace and doesn’t engender confidence in a user. the access checker’s responses to celerity’s errors should be improved, and figuring out how to do this has been on my to-do list for a long time. until i have time to improve my coding chops, it will likely remain on the list. the access checker can be further improved by adding support for more platforms, which is quite simple to do. once you have html source examples for (1) a resource with full access; (2) a resource with restricted or no access; and (3) any other category of known error occurring on a given platform whose status should be reported separately, you examine the html source to identify the access signifiers for each of those states. after that, adding the access checking logic to the script is trivial. fortunately, the access checking logic is not institution-specific, so it’s easy to add support for platforms to which my institution does not have access, if you have html source examples. currently, the access-checking logic for each platform is hard coded in the body of the access checker script, which isn’t ideal practice. further, many libraries purchase e-resources on just a handful of platforms. presenting every user with a list of all supported platforms becomes unwieldy and unfriendly as the list of supported platforms grows. i have considered moving the logic for each platform into a separate platform configuration file. users would download only the files for the platforms relevant to their institutions but wouldn’t have to figure out the access checking logic for themselves, which is one of the strengths of this script over normac’s access checker. finally, the access checker must be run from the command line, which is highly intimidating to many of the technical services staff who would benefit most directly from using this tool. as much as i’d love to encourage everyone to embrace the power of the command line, cynthia ng’s presentation at the 2014 code4lib conference [12] drove home the fact that the access checker isn’t easily accessible to its main target users. creating and distributing a gui-based access checker is quite a bit beyond my ken at this point, but anyone who could make that happen would be a hero to e-resources librarians. conclusion the access checker script i’ve described here has helped unc chapel hill regain some semblance of control over our e-resources packages at very little additional cost. the script can take a long time to run, but the staff time needed to start it and process the results is minimal—we always include the content provider’s unique id for each resource in our input file, so filtering the output results to produce a list of problems that can be sent to a content provider is trivial. and the results are well worth it. although we have not kept systematic statistics on access problems identified using the access checker, here are a few examples to give a general idea of how we use the tool and what we’ve found. first, only one of our large e-book packages necessitates running the access checker routinely: we frequently receive new marc records for it, and we know the titles and marc records in this package often demonstrate access issues. we’ve run the access checker on the entire collection retrospectively, and we run it on all urls provided in new marc records before we load them. currently we have 38,447 marc records in our catalog for this package, of which 1,482 (nearly 3.9%) are suppressed due to discovering via the access checker that access is restricted. these are reported to the content provider, and the records are eventually unsuppressed (if erroneously restricted access is restored) or deleted (if we received records for titles we didn’t purchase). there are also 33 marc records in which we’ve provided a non-doi url due to issues with the doi-based urls provided in the package (e.g., doi error or resolution to partner publisher page). we report doi problems and revert the urls in the records to the doi format as issues are resolved. second, for other packages, we tend to use the access checker to identify the scope of access problems reported to us. for example, within one day, we received two user reports about issues with a platform that has 2,031 marc records in the catalog. we ran the access checker on the urls from all the records and discovered a total of nine access problems that we then reported to the content provider. full access to the titles was quickly restored. titles in this package are heavily used, so it was important to identify and rectify all access problems, even if there were relatively few. finally, the access checker is useful for verifying title lists from content providers, especially when the number of marc records we receive for a package doesn’t match the number of titles in the package’s title list. for one package, we received a list of 786 titles but 792 marc records. using the access checker on urls extracted from the marc, we identified that access to 120 of the titles included on the title list was restricted but we had full access to six titles not included in the title list. we sent these discrepancies to the content provider, who corrected the title list. if a package contains significantly fewer titles than claimed by the title list provided at the time of purchase, there may be an opportunity to negotiate a refund or credit with the content provider. as you can see, with the access checker at our disposal, we can have more confidence that the e-resources we license are actually accessible to our patrons. although we do trust our content providers and we realize they are doing their best in good faith to ensure access to their resources, we also know that it’s impossible for them to be aware of every problem that crops up. this tool helps us proactively identify the problems that inevitably do occur so we can report them to content providers for resolution. in turn, this improves their products and service, possibly for all libraries buying the content. it also ensures that we are getting all that we are paying for with our e-resources packages. endnotes [1] the “shelf-ready” services offered by major vendors of print library materials increasingly obviate the individual cataloging and physical processing steps for libraries that can pay for such services; however, every print book a library receives must still be touched and placed somewhere within that library. [2] the issues and solutions discussed in this paper apply equally across e-resource formats. where i revert to text-focused language, it is for brevity. [3] the following reflects my responsibility for all e-resources purchased by the academic affairs library (ie. not by unc’s health sciences or law libraries). [4] what counts as a separate e-resources package is by no means clear, and differences in interpretation here endlessly complicate the answers to seemingly obvious questions such as “how many e-book collections do we own?” in general, i count something as a separate package if (a) it was purchased separately (business expert press 2011 collection is separate from business expert press 2013 collection); (b) it is a segment of a purchased collection with a separate source of cataloging (the e-journals from agu digital library are cataloged via serialssolutions, but the monographs from the same are cataloged via oclc worldcat collection set); or (c) it is a segment of a purchased collection for which cataloging is handled by another section or accounting unit. [5] we also increasingly employ non-catalog discovery tools such as proquest summon or ebsco eds to provide intellectual access. this article will not discuss these directly, as i have not personally worked on access checking based on data exported from these tools, but this is certainly possible. [6] http://journal.code4lib.org/articles/8375 [7] initially named “ebook access checker,” this script and instructions for its use are available via github: https://github.com/unc-libraries/ebook-access-checker [8] there can be multiple urls for a single title, particularly when the input file is derived from marc records. [9] https://rubygems.org/gems/celerity [10] in addition, while this article was in revision, an explicit statement that celerity is no longer being maintained was added to its github homepage. this provides further encouragement to investigate other options for use in the access checker. [11] i normally cap my input files at 3000 titles. this is due to the time it takes to check that many titles for access rather than any performance concerns. [12] ng, cynthia. (2014) “we are all disabled! universal web design making web services accessible for everyone.” presented at code4lib 2014. (slides | video stream) about the author kristina m. spurgin has been the e-resources cataloger in the university library at university of north carolina at chapel hill since 2010. soon after finding herself in this position, she taught herself some basic coding. she has been headily solving easy problems by throwing code at them ever since. subscribe to comments: for this article | for all articles 2 responses to "getting what we paid for: a script to verify full access to e-resources" please leave a response below: jrochkind, 2014-09-03 great work, and a great write up, thank you! ethan fenichel, 2014-10-01 really interesting. will set about seeing i can apply in my tech services directly! hopefully, i might even be able to contribute back. thanks leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – what’s in a name? on ‘meaningfulness’ and best practices in filenaming within the lam community mission editorial committee process and structure code4lib issue 40, 2018-05-04 what’s in a name? on ‘meaningfulness’ and best practices in filenaming within the lam community cultural institutions such as libraries, archives and museums (lam) face many challenges with managing digital collections, particularly when it comes to organizing the individual files that make up each collection. while tools such as metadata and collection management systems support identification and arrangement for digital files, administrative control depends significantly on the mere filenaming in use beneath the surface. anecdotal evidence has shown that many lam institutions have specialized filenaming schemes in place for their digital collections. this paper includes a literature review of filenaming practices in the lam community, followed by a description and analysis of survey data regarding filenaming practices in the lam community. the purpose of the survey was to learn about filenaming conventions in use within lam organizations who have filenaming policies in place. the data suggests that: similarities and differences exist in filenaming approaches between museums/galleries, archives/special collections, and academic institutions; it is preferred that filenaming be simultaneously meaningful to both humans and computers; and conventions that affect sortability are deemed more important than those that affect readability. the data also indicate several subtopics related to filenaming that would benefit from further study. by drew krewer and mary wahl introduction anyone who has worked with born-digital archives or who has inherited hard drives would tell you that people organize and name files in vastly different ways. some filenames appear standardized, such as those with uniformly formatted dates; others are more haphazard, such as those with long strings of meaningless numbers and punctuation. often filenames rely on the memory of the file’s creator to interpret, such as those that include subjective descriptions. when it comes to lam institutions, many have the added challenge of working with multiple versions of files. for instance, a single high resolution digital photograph master file may have a corresponding low resolution copy, a color corrected copy, a thumbnail, and perhaps several intermediary files. librarians, archivists, digital asset managers and other related practitioners are faced with not only developing filenaming schemes for individual files, but with determining how their files make sense in context to other files, hierarchies, collections and perhaps even outside the institution.   as our technologically-driven cultures continue to rapidly increase their knowledge-production, records creators will be continually faced with the important decision of selecting and creating filenaming schemes within many varied contexts. this paper attempts to go beyond the commonly referenced iso standard 9660:1999 (level 2) by assembling a collection of best practices that address logistical issues of filenaming within different organizational and workflow contexts. first, a literature review of filenaming practices in both the lam and greater computing communities is provided. this is followed by a description and analysis of survey data regarding filenaming practices in the lam community. literature review while archival and library scholarship has touched on filenaming within the context of larger studies, few, if any, have focused heavily or primarily on the development of filenaming schemes. this literature review includes grey literature found online such as guidelines and project documentation created by lam institutions; informal content on listserv archives from organizations such as code4lib and society of american archivists; and general computing documentation such as that found in windows dev center.   first, by examining the literature, one can determine which filenaming conventions are most frequently mentioned. it is most useful to survey the recommendations based on their frequency of appearance, so they are grouped into three tiers below.   most frequently mentioned: iso 9660:1999 (level 2): the elements of this standard are widely suggested (in full or altered slightly). this standard includes: a maximum of 207 characters in the filepath; use of lowercase a-z, 0-9, underscore (_), and period; no spaces, do not begin or end with a period, must contain no more than one period; limit directory hierarchy to 8 levels, and directory names should not use periods. brevity: many emphasize keeping the filenames as short as possible. standardized date: many require that dates be standardized. iso 8601 seemed to be by far the most popular date standard, likely due to it providing the functionality of sorting by year, then by month, then by date. iso 8601 includes date formats such as yyyy-mm-dd (e.g. 2018-03-24 for march 24, 2018). version indicator: many frequently stress the importance of version control, usually by the use of an appended indicator such as _01, _02; or qualifiers such as lo and hi for low and high resolution, etc. no special characters: it is often considered best practice to ensure that special characters (e.g. #, %, &) are not used. issues can occur when filenames use characters that are reserved by operating systems. underscores and hyphens: though underscores are mentioned in iso 9660, both hyphens and underscores are mentioned as ideal replacement for spaces.   frequently mentioned: meaningfulness: a moderate amount of informal literature suggests making filenames “meaningful.” this usually corresponds with aligning filenames with existing institutional knowledge such as keywords, call numbers, and accession numbers. leading zeros: many suggest adding leading zeros (e.g., 01, 001, 0001) to aid in appropriate sorting. the number of zeros used should complement the quantity of files being named. unique: no two files should have the same name.   occasionally mentioned: camel case: the use of camel case (e.g. gmailaccount, donornumber) is occasionally mentioned to denote spacing. some state that camel case is preferable to underscores/hyphens (university of edinburgh 2015), while others suggest the appropriateness of both camel case as well as underscores and hyphens (borer et al. 2009). order of elements: elements that construct the filename should appear in order from left to right in a way that allows users to most easily find and sort the digital assets. care should be taken to understand how an institution retrieves information to determine this order. scalability: some literature mentions that filenaming guidelines should be extensible to include outlying materials and growing amounts of digital assets and information. do not use values that could change: using values that may change over time within a filename would create unnecessary retroactive work in the future.   secondly, the literature review found four primary contexts in which filenaming documentation was established: digitization programs, born digital archives, business protocol, and data management services. of the four, data management filenaming had slightly different conventions than the others. for instance, best practices recommend including information such as project or experiment names and testing conditions. concerns with consistency across the lifespan of an experiment are also stressed (stanford libraries; university of oregon libraries).   lastly, a primary point of contention around filenaming centers around what seems to be simultaneously both a theoretical and practical issue: whether or not unique identifiers should be “meaningful” or randomly assigned, and under what circumstances. this has been discussed as far back as 2007, for instance: the creation of persistent identifiers generally follows one of two conventions. the first school of thought is that the identifier should be randomly generated, embedding no semantic meaning. the second school of thought advocates that unique identifiers should carry some meaning that is understandable to those working intimately with the collection (schreibman).   and in 2008: meaningful file names work best for medium to small collections and make it easier to identify and manage the digital files. meaningful file names don’t work as well for large collections because it is more difficult to create unique names for all items. it is also possible that the name’s meaning will be lost or change connotation over time (university of colorado, boulder libraries).   the latter quote moves forward to concede that because non-descriptive filenames provide no identifying information, “the files are harder to manage outside of the database that contains the associated metadata.” it also concludes that collection characteristics and project specifications should dictate the decision to use a meaningful name vs. a non-descriptive name (university of colorado, boulder libraries).   more recently, others seem to view “meaningfulness” as being a two-way street: “in the interest of preserving access to digital files, we choose file name components that are least likely to cause a problem in any environment. file names should provide context and be easily understood by humans and computers” (frazer 2013). this positioning seems to argue that meaningful identifiers are more desirable than non-descriptive ones, as they are able to provide meaningful information to both humans and computers. survey the survey (see appendix a) was designed to gather data regarding filenaming practices in use within libraries, archives and museums. the target audience included those working with digital collections in the lam community. in particular, we sought to learn: 1) what filenaming conventions are in use within lam organizations? 2) do lam organizations have filenaming policies in place? and 3) what is the significance of various filenaming concepts? for the purposes of the survey, a filenaming policy was defined as a set of specified rules or procedures for guiding the creation of filenames. the online data collection tool qualtrics was used to create and administer the survey, which consisted of 30 questions. the survey comprised primarily of multiple choice, “select all [options] that apply”, and open text questions. several 5-point likert scale questions and one forced ranking question were also included. the survey was divided into four main sections: 1) background information regarding the role and the lam organization in which the respondent works, as well as general filenaming practice in that organization, 2) specific filenaming conventions in use and the context(s) in which they’re used, 3) the importance placed on various filenaming concepts, and 4) additional comments from participants. each question in the survey was voluntary and did not require a response in order to make a final submission. the survey remained open for ten weeks, from february 3 through april 15, 2016. to invite participation, a description and link to the survey was shared to listservs and online newsletters whose readership emphasized the target audience. a total of 398 responses to the survey was collected. survey results and commentary background information this section of the survey aimed to gather data regarding respondents’ lam organization in which they work, as well as general filenaming practice in that organization.   participants were first asked to select a description of the overall type of organization in which they work (n=371). see table 1.   table 1: demographic breakdown of respondents respondents’ type of organization percentage of responses   academic library 44.2% archives/special collections 25.6% other 13.2% museum/gallery 12.1% public library 4.9%   participants were asked about their role in their institution. over one-third of respondents (n=330) identified themselves as librarians, followed by over one-quarter as archivists. about 20% selected other and supplied an open text explanation. upon review of these responses, we identified a number of these to fall within another category, bringing the other category down to 17% (and increasing several of the other categories). remaining open text responses comprised primarily of photographer, as well as some variation of the title digital assets manager or digital curator. see table 2. table 2: professional role breakdown of respondents respondents’ role percentage of responses librarian 37.6% archivist 29.4% other 17.3% it professional 6.1% project manager 6.1% assistant/technician 3.6%   participants were also asked about the existence (or non-existence) of filenaming policies in place in their organizations (n=331). participants who selected no or i don’t know were given no further questions and prompted to complete the survey by submitting their responses up to this point. remaining respondents (n=313) were asked what kinds of policies existed and could choose more than one type. see table 3.   table 3: filenaming policy types in use by respondents type of filenaming policy number of respondents one or more organization-wide policies 28.4% one or more department-wide policies 37% one or more filenaming policies per project/collection 34.6%   remaining respondents (n=313) were also asked whether they felt that their filenaming policies adequately serves their needs. see table 4.   table 4: breakdown of if current policies served respondents’ needs does current policy(ies) serves organization’s needs? percentage of respondents yes 75.3% no 17.3% i don’t know 7.4%   participants who selected no were prompted to explain why their filenaming policies may not adequately serve their needs in an open text box. fifty-one respondents provided explanations that ranged from current schemes not being extensible enough to a need for more training of staff members who are on the front lines of creating filenames. several themes emerged from the responses. first, eight respondents mentioned the need for more buy-in within the institution to follow established guidelines. some of these expressed that adoption was “casual” or that policies were followed as “suggestions”. one response mentioned that “compliance with naming rules is very difficult to impose or enforce without full knowledgeable support from senior management”. another theme that appeared was the desire to have either an institution-wide policy that can be modified for smaller work units or projects, or a larger, profession-wide set of best practices. one response stated that they would prefer to have their filenaming schemes “standardized according to professional standards”. lastly, five of the respondents expressed the challenge of having schemes in place for some collections but not others. among these responses, filenaming policies were typically in place for digitized materials, but not others. filenaming characteristics this section of the survey aimed to gather data regarding filenaming conventions in use in the lam community and the context(s) in which they are used. organizational approaches to naming digitized and born-digital files participants were asked to select what types of filenaming characteristics were being employed at their institutions according to material type. when we further drilled down to look at how specific types of institutions were responding to these questions, both similarities and differences were found in institutional approaches. due to the low response rate of public libraries, the responses of academic libraries, archives/special collections, and museum/galleries will primarily be discussed.   table 5: types of filenaming characteristics employed – digitized material organization type academic library public library archives/special collections museum/gallery count column n % count column n % count column n % count column n % for digitized material, please select the types of elements in use in your filenames date a physical item was created 16 18.8% 1 20.0% 13 25.0% 4 10.8% date a digital file was created 13 15.3% 1 20.0% 10 19.2% 8 21.6% date a digital file was last modified 1 1.2% 0 0.0% 5 9.6% 0 0.0% institution identifier (including any subunits of the institution) 30 35.3% 2 40.0% 23 44.2% 8 21.6% collection identifier (i.e. call number, accession number) 71 83.5% 4 80.0% 43 82.7% 32 86.5% item identifier 71 83.5% 5 100.0% 44 84.6% 24 64.9% item subunit identifier (i.e. recto, verso, page 1, side a) 52 61.2% 4 80.0% 31 59.6% 23 62.2% staff member identifier 2 2.4% 0 0.0% 3 5.8% 6 16.2% physical location identifier 15 17.6% 1 20.0% 14 26.9% 2 5.4% digital location identifier 6 7.1% 1 20.0% 9 17.3% 1 2.7% an indication of the file’s purpose (i.e. preservation master, access copy, thumbnail) 20 23.5% 1 20.0% 16 30.8% 17 45.9% an indication of the file’s quality (i.e. high resolution, lossy) 7 8.2% 3 60.0% 5 9.6% 6 16.2% other (please explain) 15 17.6% 0 0.0% 7 13.5% 7 18.9%   institution identifiers for digitized materials seemed to be more prevalent in academic libraries and special collections/archives rather than in museums/galleries. this may be due to the fact that educational institutions may have numerous campuses or libraries, and these identifiers perhaps aid in distinguishing content across a larger institutional network. the most consistent filenaming characteristic in use seems to be the collection identifier, indicating that institutions are using identifiers to make meaningful links between digital and physical collections. item identifiers also seem particularly common in academic libraries and archives/special collections, and museums tend to use these as well, but not as frequently as the other institutional types. museums/galleries also seem to use indicators to distinguish various versions of a file—preservation masters vs. access copy vs. thumbnail, etc. this practice may be due to the fact that museums and art institutions frequently license out images for publication and other purposes—perhaps distinguishing versions of an image has use within a licensing workflow. see table 5.   table 6: types of filenaming characteristics employed – born-digital material organization type academic library public library archives/special collections museum/gallery count column n % count column n % count column n % count column n % for born-digital material, please select the types of elements in use in your filenames date a digital file was created 16 32.0% 3 50.0% 18 45.0% 10 34.5% date a digital file was last modified 2 4.0% 0 0.0% 7 17.5% 3 10.3% institution identifier (including any subunits of the institution) 20 40.0% 1 16.7% 17 42.5% 6 20.7% collection identifier (i.e. call number, accession number) 37 74.0% 3 50.0% 30 75.0% 20 69.0% item identifier 41 82.0% 5 83.3% 29 72.5% 18 62.1% item subunit identifier (i.e. recto, verso, page 1, side a) 17 34.0% 4 66.7% 16 40.0% 14 48.3% staff member identifier 4 8.0% 1 16.7% 3 7.5% 5 17.2% digital location identifier 3 6.0% 3 50.0% 5 12.5% 2 6.9% an indication of the file’s purpose (i.e. preservation master, access copy, thumbnail) 9 18.0% 3 50.0% 13 32.5% 9 31.0% an indication of the file’s quality (i.e. high resolution, lossy) 3 6.0% 2 33.3% 3 7.5% 2 6.9% other (please explain) 7 14.0% 0 0.0% 8 20.0% 6 20.7%   upon review of the data in table 6 and comparing it to table 5, it appears there is a sharp increase (compared to digitized material) in filenaming characteristics including the date a digital file was created, though using a modification date in the filename seems to still not be a priority. again, institutional identifiers seemed to be more prevalent in non-museum/gallery settings, and both collection and item identifiers are common practices for born-digital filenaming. the use of sub-unit identifiers (recto, verso, page 1, side a, etc.) does become less prevalent with born digital materials when compared to digitized materials, likely due to the significant characteristics of digital materials vs. physical materials. due to the phrasing of the survey question, participants’ answers may refer to either born digital files given to the institution (and then renamed) and born digital files created by the institution (therefore named upon creation). organizational beliefs on the purpose and characteristics of filenaming in addition to benchmarking how organizations name digitized and born-digital files, the survey sought to gauge foundational beliefs about filenaming that could influence an organization’s approach to creating filenaming convention(s).   the data in table 7 supports the idea that filenames should be both human and machine-readable. the majority of institutions either agreed or strongly agreed that filenames should be meaningful to both humans and computers. no respondents strongly disagreed, and a few respondents were neutral in their opinion. this desire for meaningfulness in filenaming is also supported by the findings in table 8, which indicates that most institutions either disagree or strongly disagree that filenames should be randomly generated by software or a system managing the files.   questions in the survey also addressed whether the meaning of filenames should be understood both inside and outside the organization. most respondents either agreed or strongly agreed that filenaming should primarily be meaningful to the person or organization managing the files (see table 9). between 19.6% and 27.6% of each type of organization were neutral in their opinion. it is interesting to note, however, that 11.8% of respondents working in an archives/special collections setting and 15.8% of respondents working in a museum/gallery setting disagreed that filenames should mainly serve to provide meaning to internal stakeholders. this might indicate that the filenames in use might be serving as an access point for the users of these specific institutions. in looking at all organizational attitudes toward whether filenames should be understandable to individuals outside of an organization, institutions seem particularly divided in opinion (see table 10). this could indicate that decisions to make files externally meaningful are intimately linked to service models, workflows, access points, and other defining qualities of an institution’s operational approach.   table 7: levels of agreement regarding human vs. machine-readable organization type academic library public library archives/special collections museum/gallery other (please explain) count column n % count column n % count column n % count column n % count column n % filenames should be both meaningful to humans/organizations and machine readable. strongly disagree 0 0.0% 0 0.0% 0 0.0% 0 0.0% 0 0.0% disagree 3 3.4% 0 0.0% 0 0.0% 5 13.2% 2 6.9% neither agree nor disagree 13 14.9% 2 40.0% 6 11.8% 3 7.9% 1 3.4% agree 27 31.0% 2 40.0% 20 39.2% 12 31.6% 10 34.5% strongly agree 44 50.6% 1 20.0% 25 49.0% 18 47.4% 16 55.2%   table 8: levels of agreement regarding name assignment organization type academic library public library archives/special collections museum/gallery other (please explain) count column n % count column n % count column n % count column n % count column n % filenames should be randomly assigned by the software or system managing the files. strongly disagree 29 33.3% 1 20.0% 22 43.1% 16 42.1% 11 37.9% disagree 29 33.3% 3 60.0% 16 31.4% 9 23.7% 7 24.1% neither agree nor disagree 17 19.5% 0 0.0% 9 17.6% 9 23.7% 8 27.6% agree 9 10.3% 0 0.0% 3 5.9% 2 5.3% 2 6.9% strongly agree 3 3.4% 1 20.0% 1 2.0% 2 5.3% 1 3.4%   table 9: levels of agreement regarding external understandability organization type academic library public library archives/special collections museum/gallery other (please explain) count column n % count column n % count column n % count column n % count column n % it is important for filenames to be understandable to individuals outside of my organization. strongly disagree 7 8.0% 0 0.0% 5 9.8% 4 10.5% 0 0.0% disagree 18 20.7% 1 20.0% 9 17.6% 8 21.1% 9 31.0% neither agree nor disagree 27 31.0% 2 40.0% 11 21.6% 14 36.8% 6 20.7% agree 28 32.2% 1 20.0% 17 33.3% 9 23.7% 6 20.7% strongly agree 7 8.0% 1 20.0% 9 17.6% 3 7.9% 8 27.6%   table 10: levels of agreement regarding internal understandability organization type academic library public library archives/special collections museum/gallery other (please explain) count column n % count column n % count column n % count column n % count column n % filenames should primarily be meaningful to the person or organization managing the files. strongly disagree 1 1.1% 0 0.0% 0 0.0% 2 5.3% 1 3.4% disagree 8 9.2% 0 0.0% 6 11.8% 6 15.8% 2 6.9% neither agree nor disagree 20 23.0% 0 0.0% 10 19.6% 9 23.7% 8 27.6% agree 35 40.2% 4 80.0% 15 29.4% 10 26.3% 12 41.4% strongly agree 23 26.4% 1 20.0% 20 39.2% 11 28.9% 6 20.7% job roles and importance of filenaming to workflows the survey also sought to understand if individuals with particular job roles might have different perspectives on the importance of filenaming. several statements were listed to gauge how filenaming supports and enhances both shortand long-term job functions. some of these statements are listed below, along with an interpretation of the collected data.   filenaming is important to the productivity of the unit producing the files. most respondents either agreed or strongly agreed with this statement (see table 11). over 70% of both project managers and individuals in a digital collections role strongly agreed. this data seem to recommend that if one is attempting to establish a filenaming convention for a digital services unit or other similar department, it would be wise to receive feedback from those intimately involved in file creation and management to determine how to create a filename that optimizes efficiency.   table 11: levels of agreement regarding filenaming and productivity which of the following best describes your role where you work? (select one) librarian archivist project manager it professional assistant/technician other (please explain) digital collections position count column n % count column n % count column n % count column n % count column n % count column n % count column n % filenaming is important to the productivity of the unit producing the files. strongly disagree 0 0.0% 0 0.0% 0 0.0% 0 0.0% 0 0.0% 0 0.0% 0 0.0% disagree 1 1.5% 2 3.3% 0 0.0% 0 0.0% 0 0.0% 0 0.0% 1 7.1% neither agree nor disagree 9 13.6% 10 16.7% 0 0.0% 0 0.0% 4 18.2% 2 9.5% 1 7.1% agree 25 37.9% 32 53.3% 3 25.0% 4 50.0% 9 40.9% 7 33.3% 2 14.3% strongly agree 31 47.0% 16 26.7% 9 75.0% 4 50.0% 9 40.9% 12 57.1% 10 71.4% filenaming is important to relating digital files to other related content (i.e. finding aids, marc records, mets records). a sizeable number of respondents neither agreed or disagreed with this statement, but a larger amount of respondents either agreed or strongly agreed (see table 12). this indicates a wide range of practice; filenames may either not be used to relate files to descriptive information or they may be used to serve as an identifier that more broadly integrates the file into various operational functions within the institution.   table 12: levels of agreement regarding relatability between files which of the following best describes your role where you work? (select one) librarian archivist project manager it professional assistant/technician other (please explain) digital collections position count column n % count column n % count column n % count column n % count column n % count column n % count column n % filenaming is important to relating digital files to other related content (i.e. finding aids, marc records, mets records). strongly disagree 1 1.5% 1 1.7% 0 0.0% 0 0.0% 1 4.5% 0 0.0% 0 0.0% disagree 6 9.1% 3 5.0% 0 0.0% 0 0.0% 2 9.1% 2 9.5% 2 14.3% neither agree nor disagree 23 34.8% 19 31.7% 3 27.3% 3 37.5% 6 27.3% 6 28.6% 3 21.4% agree 25 37.9% 20 33.3% 5 45.5% 3 37.5% 9 40.9% 4 19.0% 5 35.7% strongly agree 11 16.7% 17 28.3% 3 27.3% 2 25.0% 4 18.2% 9 42.9% 4 28.6% filenaming is important to the long-term management of digital files. an overwhelming number of respondents either agreed or strongly agreed that filenaming is important to the long-term management of digital files (see table 13). with such a strong response to this statement, it is clear to professionals in the field that a filenaming convention is essential to the long-term success of digital programs. as institutions establish programs that produce large quantities of files, it becomes very important to establish meaningful filenames as early as possible.   table 13: levels of agreement regarding long-term management of digital files which of the following best describes your role where you work? (select one) librarian archivist project manager it professional assistant/technician other (please explain) digital collections position count column n % count column n % count column n % count column n % count column n % count column n % count column n % filenaming is important to the long-term management of digital files. strongly disagree 1 1.5% 0 0.0% 0 0.0% 0 0.0% 0 0.0% 0 0.0% 0 0.0% disagree 0 0.0% 0 0.0% 0 0.0% 0 0.0% 1 4.5% 0 0.0% 1 7.1% neither agree nor disagree 1 1.5% 1 1.7% 2 16.7% 1 14.3% 1 4.5% 0 0.0% 0 0.0% agree 15 22.7% 16 26.7% 2 16.7% 3 42.9% 3 13.6% 5 23.8% 2 14.3% strongly agree 49 74.2% 43 71.7% 8 66.7% 3 42.9% 17 77.3% 16 76.2% 11 78.6%   significance of filenaming survey participants were asked to rank seven filenaming conventions in order of importance, with 1=most important and 7=least important. table 14 provides these conventions in order of most to least important based on mode, then mean (n=214).   table 14: ranking of filenaming conventions filenaming convention mode mean standard deviation removing and/or replacing punctuation 1 2.71 1.48 formatting dates (yyyy-mm-dd, yymmdd, etc.) 1 3.56 1.96 removing and/or replacing spaces 2 2.95 1.42 using leading zeros (0001, 0002, 0003, etc.) 2 3.5 1.95 using only lowercase characters 6 4.78 1.61 keeping filenames under a limited number of characters 7 4.63 2.05 using camelcase characters 7 5.87 1.33 removing and/or replacing punctuation was most often ranked as most important, followed by formatting dates, removing and/or replacing spaces, and using leading zeros. a gap is evident between these four conventions and the last three (using only lowercase characters, keeping filenames under a limited number of characters, and using camelcase characters) indicating a clear divide between these two groups. perhaps relevant here is that the first four conventions mostly affect sortability, while the last three conventions mostly affect readability. interestingly, both groups include a convention that affects functionality: removing and/or replacing punctuation, and keeping filenames under a limited number of characters. if one is deemed most important, it is compelling to wonder why the other is not. perhaps it is because institutions simply do not often have filenames that would surpass a limit on the number of characters.   respondents were also asked about the importance of eight particular filenaming concepts. responses (n=203) were recorded using a 5-point likert scale with 1=”strongly disagree” and 5=”strongly agree”. table 15 provides a breakdown, in order of highest to lowest mean.   table 15: importance of various filenaming concepts filenaming is important to… mode mean the long-term management of digital files 5 4.66 the digitization and/or reformatting of materials 5 4.35 the ingest processes of digital files 5 4.31 the productivity of the unit producing the file 5 4.28 relating digital files to their derivative files 5 4.21 relating digital files to their physical counterparts 5 4.14 the management of “use copy” digital files 4 3.84 relating digital files to other related content (i.e. finding aids, marc records, mets records) 4 3.74 respondents most often chose strongly agree to filenaming being important to the long-term management of digital files, which also aligns with the data in table 13. each of the concepts fell within the strongly agree or agree range, thus indicating that each of them are deemed rather closely in importance. it is somewhat noteworthy that the importance of using filenaming to relate to derivative files fell higher than relating to physical counterparts, which fell higher than relating to other related content (finding aids, etc.), though the range is narrow. additional comments this section invited survey participants to share additional comments regarding filenaming in the lam community. seventy-nine respondents provided comments. while most comments further iterated preferences for conventions already covered in the survey, two main themes emerged.   first, it was overwhelmingly apparent that respondents were simply eager to learn the specifics of how others in the lam community are developing filenaming schemes. comments included: “i have done a bit of research to see how others are creating filenames; i found some resources, but not as many as i had hoped,” “i always second guess if my practice aligns with what my colleagues are doing… i’d be very interested to see others responses and rationale for their schemas,” and, “i would love to find a set of relevant guides that i can easily adapt and disseminate.” this points to a need for an online, shared registry or repository of filenaming guidelines, or perhaps sets of guidelines from some of the prominent professional organizations in the lam field. a repository would be particularly useful, as it could be applicable to both the lam community and broader fields such as personal archiving which includes non-lam and even non-technical individuals. additionally, survey respondents were eager to share their own filenaming schemes, and about a dozen respondents shared details on their conventions used or links to their schemes online.   second, the challenge of getting buy-in from others was shared by a number of respondents. comments included: “while we have a file naming policy in place and have had one for a number of years, it is difficult to get all staff and librarians to fully comply” and “filenaming practices are difficult to enforce/get buy in for from people new to digital projects”. it would be enlightening to learn how prevalent this challenge is across the lam community, and if there are institutions who have found solutions to communicating the value of implementing filenaming schemes. perhaps this is an area for future study. conclusion librarians, archivists, digital asset managers and related practitioners will no doubt be faced with a growing need to develop intricate filenaming schemes. though the topic of meaningfulness is covered informally in the lam community, and a few suggestions for and against such methods presented themselves in the survey data, several questions still remain. for instance, how would the use of both meaningful and non-meaningful identifiers within a digitization program affect productivity and workflow? is it better to select one method for all assets, or to select the method on a case-by-case basis? and if selecting only one method, which method—meaningful or non-descriptive—should one choose? we hope that the data analysis above, combined with a consideration of one’s current conventions, may assist in constructing a more nuanced and established way of considering, selecting, and applying filenaming procedures and guidelines. references borer, e. t., seabloom, e. w., jones, m. b. and schildhauer, m. (2009). some simple guidelines for effective data management. the bulletin of the ecological society of america, 90: 205-214. doi:10.1890/0012-9623-90.2.205.   frazer, meghan. (2013). an elevator pitch for file naming conventions. acrl techconnect blog. retrieved october 6, 2017 from http://acrl.ala.org/techconnect/post/an-elevator-pitch-for-file-naming-conventions   schreibman, susan, ed. (2007). best practice guidelines for digital collections at university of maryland libraries, 2nd ed. retrieved october 6, 2017 from http://cs.mty.itesm.mx/profesores/pverdines/tcs/p2/digitalcollections.pdf.   stanford libraries. (n.d.). best practices for file naming. retrieved october 6, 2017 from http://library.stanford.edu/research/data-management-services/data-best-practices/best-practices-file-naming   university of colorado, boulder libraries. (2008). guidelines on file naming conventions for digital collections. retrieved july 11, 2014 from http://ucblibraries.colorado.edu/systems/digitalinitiatives/docs/filenameguidelines.pdf   university of edinburgh. (2015). use capital letters to delimit words. retrieved october 6, 2017 from http://www.ed.ac.uk/records-management/records-management/staff-guidance/electronic-records/naming-conventions   university of oregon libraries. (n.d.). data management best practices: file naming & version control. retrieved october 6, 2017 from https://library.uoregon.edu/research-data-management/best-practices about the authors drew krewer has worked with both digital and physical collections at a range of cultural heritage institutions, including the getty research institute, the university of arizona poetry center, and the university of houston libraries. he is currently an analyst at the university of arizona foundation. mary wahl is technical services librarian at pasadena city college where she oversees cataloging and metadata. her professional interests include metadata wrangling, digital preservation, media collections and personal digital archives. appendix a – filenaming survey study on filenaming practices within the lam community section i: background information this section aims to gather data regarding the lam organization in which you work, as well as current filenaming practice in that organization. which of the following best describes the overall type of organization in which you work? (select one) academic library public library archives/special collections museum/gallery other (please explain) please select the size of your academic institution from the choices below: fewer than 500 students 500-1,999 students 2,000-4,999 students 5,000-9,999 students 10,000 students or more how many individuals are employed at the library/archive/museum for which you work? this figure should include paid positions only. which of the following best describes your role where you work? (select one) librarian archivist project manager it professional assistant/technician other (please explain) do you and/or your organization employ a filenaming policy, either organization-wide, departmentwide, or on a per-project basis? for the purposes of this survey, a filenaming policy is defined as a set of specified rules or procedures for guiding the creation of filenames. yes no i don’t know do you employ (select all that apply) one or more organization-wide filenaming policies one or more department-wide filenaming policies one or more filenaming policies per project/collection do you feel that your current filenaming policy(ies) adequately serves your needs? yes no i don’t know please explain why you feel your current filenaming policy(ies) may not adequately serve your needs.   section ii: filenaming characteristics this section aims to gather data regarding filenaming conventions in use in the lam community and the context(s) in which they’re used. for which of the following types of digital collections do you employ filenaming policy(ies)? (select all that apply) digitized material born-digital material business data (i.e. institutional records, digital marketing assets, etc.) research data other (please explain) for digitized material, please select the types of elements in use in your filenames: date a physical item was created date a digital file was created date a digital file was last modified institution identifier (including any subunits of the institution) collection identifier (i.e. call number, accession number) item identifier item subunit identifier (i.e. recto, verso, page 1, side a) staff member identifier physical location identifier digital location identifier an indication of the file’s purpose (i.e. preservation master, access copy, thumbnail) an indication of the file’s quality (i.e. high resolution, lossy) other (please explain) for born-digital material, please select the types of elements in use in your filenames: date a digital file was created date a digital file was last modified institution identifier (including any subunits of the institution) collection identifier (i.e. call number, accession number) item identifier item subunit identifier (i.e. recto, verso, page 1, side a) staff member identifier digital location identifier an indication of the file’s purpose (i.e. preservation master, access copy, thumbnail) an indication of the file’s quality (i.e. high resolution, lossy) other (please explain) for business data, please select the types of elements in use in your filenames: date a physical item was created date a digital file was created date a digital file was last modified institution identifier (including any subunits of the institution) collection identifier (i.e. call number, accession number) item identifier item subunit identifier (i.e. recto, verso, page 1, side a) staff member identifier physical location identifier digital location identifier an indication of the file’s purpose (i.e. preservation master, access copy, thumbnail) an indication of the file’s quality (i.e. high resolution, lossy) other (please explain) for research data, please select the types of elements in use in your filenames: date a physical item was created date a digital file was created date a digital file was last modified institution identifier (including any subunits of the institution) collection identifier (i.e. call number, accession number) item identifier item subunit identifier (i.e. recto, verso, page 1, side a) staff member identifier physical location identifier digital location identifier an indication of the file’s purpose (i.e. preservation master, access copy, thumbnail) an indication of the file’s quality (i.e. high resolution, lossy) other (please explain) for the type of digital collection that you considered “other,” please select the types of elements in use in your filenames: date a physical item was created date a digital file was created date a digital file was last modified institution identifier (including any subunits of the institution) collection identifier (i.e. call number, accession number) item identifier item subunit identifier (i.e. recto, verso, page 1, side a) staff member identifier physical location identifier digital location identifier an indication of the file’s purpose (i.e. preservation master, access copy, thumbnail) an indication of the file’s quality (i.e. high resolution, lossy) other (please explain) please rank the following filenaming conventions in order of importance, with 1 being the most important: removing and/or replacing punctuation removing and/or replacing spaces using only lowercase characters using camelcase characters formatting dates (yyyy-mm-dd, yymmdd, etc.) using leading zeros (0001, 0002, 0003, etc.) keeping filenames under a limited number of characters how important are the following filenaming concepts to you/your organization (using a scale of not important; slightly important; moderately important; important; very important): relating a digital file to its physical original relating a digital file to its digital derivatives scalability of naming conventions sortability to what extent do you agree or disagree with the following statements (using strongly disagree; disagree; neither agree nor disagree; agree; strongly agree): filenames should primarily be meaningful to the person or organization managing the files. filenames should primarily be machine readable. filenames should be both meaningful to humans/organizations and machine readable. filenames should be randomly assigned by the software or system managing the files. it is important for filenames to be understandable to individuals outside of my organization. who assigns filenames, either as a whole or in part? (select all that apply) project manager for the collection technician(s) equipment (i.e. camera, scanner) other (please explain) does your content management system influence your filenaming? yes no sometimes not applicable please explain how your content management system influences your filenaming. under what circumstances does it influence your filenaming? when does it not influence your filenaming? for digitized material, does physical format of the original influence your filenaming? yes no sometimes not applicable please explain how the physical format of the original influences your filenaming. what qualities of the physical formats influence your decisions? please explain when physical formats would influence your filenaming and when they would not influence your filenaming. what qualities of the physical formats guide these decisions? for both digitized and born-digital material, does file format influence your filenaming? yes no sometimes not applicable please explain how file format influences your filenaming for both digitized and borndigital materials. when does file format influence your filenaming for both digitized and born-digital materials? when does file format not influence your filenaming? section iii: significance of filenaming this section aims to gauge the importance of various filenaming concepts in the lam community. to what extent do you agree or disagree with the following statements (using strongly disagree; disagree; neither agree nor disagree; agree; strongly agree): filenaming is important to the productivity of the unit producing the files. filenaming is important to the digitization and/or reformatting of materials. filenaming is important to the ingest processes of digital files. filenaming is important to the long-term management of digital files. filenaming is important to relating digital files to their physical counterparts. filenaming is important to relating digital files to their derivative files. filenaming is important to relating digital files to other related content (i.e. finding aids, marc records, mets records). filenaming is important to the management of “use copy” digital files.   section iv: additional comments this section is to allow survey participants to share any additional comments regarding filenaming in the lam community. please share any additional comments you may have regarding filenaming practice.   subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – joining an open source community: creating a symphony connector for the xc ncip toolkit mission editorial committee process and structure code4lib issue 14, 2011-07-25 joining an open source community: creating a symphony connector for the xc ncip toolkit when the pennsylvania academic library consortium, inc. (palci) decided to upgrade its resource sharing software (ez-borrow) all of the participating libraries – among them lehigh university – were responsible to have in place an implementation of the ncip protocol to provide communication between the new ez-borrow software developed by relias international and their respective ils. this article presents the process of lehigh choosing to adopt the extensible catalog ncip toolkit, and the technical details about building a connector with the sirsidynix symphony ils. by michelle suranofsky background when the pennsylvania academic library consortium, inc. (palci) [1] decided to upgrade its resource sharing software (ez-borrow) all of the participating libraries – among them lehigh university – were responsible to have in place an implementation of the ncip protocol to provide communication between the new ez-borrow software developed by relias international [2] and their respective ils. weighing our options lehigh had several options to consider for meeting this requirement: license the sirsidynix [3] ncip web service software use the extensible catalog (xc) [4] ncip toolkit (an ncip web services implementation written in java) create our own solution licensing the sirsidynix web services appeared to involve the least amount of effort, but it had a price tag. using the xc toolkit meant that we would need to write java code to make the calls to the sirsidynix catalog api.  this is because the xc ncip toolkit ‘core code’ takes care of the mechanics of the ncip web services but the connector (additional java) code needed to communicate with the sirsidynix api had not been developed yet.  often a picture is worth a thousand words so i’ll throw one in now.  the diagram shown in figure 1 gets ahead of the story but paints a picture of the xc ncip toolkit core code functionality and the role of the needed connector. figure 1. diagram of the integration between ez-borrow, xc ncip toolkit and sirisdynix symphony to bridge the toolkit to the sirsidynix catalog we needed to write java code that would call the catalog’s api and return the results back to the core toolkit. the extensible catalog organization is building a library of open source connectors for various catalog products.  at the start of this project, a sirsidynix connector had not yet been written. when we learned that we would not have to support the entire ncip protocol, but rather only four of the services, we knew the job of writing connector code for the toolkit or creating our own solution would be achievable in the amount of time we had (a couple of months).  if the new relais ez-borrow software had required us to support the entire ncip protocol, we may have taken an entirely different approach.  that undertaking would haven taken much more time.  since the xc toolkit was available at no cost, we thought we should explore its capability before thinking about how we could possibly write our own solution. so lehigh decided to put its newly hired java developer (me) and its sirsidynix symphony api expert (mark canney) to work exploring the toolkit and developing a solution. our initial plan was to create a proof-of-concept to confirm that the toolkit would work for us by: downloading the toolkit code and getting it running locally (on my workstation) taking a look at  the existing ‘connector’ code that had been developed for other catalogs reviewing the guidelines for connector code development provided by xc writing connector code that would make the api call to sirsi, parse those results, and return the results to the core toolkit code. the proof-of-concept development went well.  i found the xc toolkit code easy to work with and the performance tests i ran against it looked good.  we demonstrated the prototype to our library management team. they were happy…i was happy…so we decided to move forward with the project using the xc toolkit.  i should also mention that xc has written two versions of the toolkit.  the second version developed to support the ncip 2 protocol.  it was just around the time of this project that xc was finishing up work on version 2.  we decided to move forward with version 2 since the relais software would support either ncip version, and because we knew eventually we would want to use the latest protocol. my goal in writing this article is to share the technical details of this project’s implementation in order to bring a level of clarity to other organizations looking for a way to communicate with their catalog via ncip.  to that end, the sections of this article that follow elaborate on all of the technical details of the setup, coding, testing and deploying of this project.  i have previously installed eclipse, tomcat, maven and subversion. setting up the toolkit in eclipse & tomcat to get started i wanted to get the toolkit (core code) running as a project in eclipse using my local tomcat server.  this setup would allow me to easily debug the entire process (not just the connector code i would be writing).  it would also allow me to make any changes (that i might need to make) to the core code. initially, because of something not expected in the xml request from relais, i did have to modify the core code.  as the project progressed relais adjusted their xml request which enabled me to run the toolkit code without any modifications. there are many different ways to download the ncip xc toolkit.  in fact, there are multiple approaches and tools for much of what is included in this article. take these ‘steps’ with a grain of salt and adapt and implement as you like. to download the code outside of eclipse use this subversion call at the command prompt: svn checkout https://xcncip2toolkit.googlecode.com/svn/core/trunk/ xcncip2toolkit-read-only next…call the this maven command from within the directory where you have downloaded the code:  mvn clean install this gives you all of the xc ncip2 toolkit core code.  my next steps could be somewhat controversial.  i created a new project in eclipse to import this code into. however, i changed the project structure (folders that the files live in) just slightly to create a structure that i prefer mainly so that i can easily debug and ‘hot deploy’ any code changes i want to make without having to wait for a ‘build’.  this structure allows me to do so without having to do any extra tomcat/maven configuration. in other words, in my eclipse development environment (on my workstation) – for this project – i take maven out of the picture. this is just my preference for this project. in eclipse, create a new dynamic web project named ncipcore  leaving all of the default settings except the default output folder.  change that to: ncipcore/webcontent/web-inf/classes/ copy: /folder_where_i_downloaded_code/web/target/web-0.0.1-snapshot/web-inf /folder_where_i_downloaded_code/web/target/web.0.0.1-snapshot/index.html /folder_where_i_downloaded_code/web/target/web.0.0.1-snapshot/stylesheet.xsl into my eclipse project ncipcore/webcontent/. create the following packages in the ncipcore project, src folder, as shown in figure 2. figure 2. create these packages within the ncipcore project copy all of the .java files from /folder_where_i_downloaded_code/ into packages (as illustrated above) within the /ncipcore/src folder.  the project should look similar to figure 3. figure 3. the ncipcore project setup in eclipse next, clean up a couple of the errors in the project and get it running on tomcat. add tomcat’s servlet-api.jar file to the build path of the project (figure 4): figure 4. add the service-api.jar file to the build path of the project add the ncipcore project to the list of deployed projects on your tomcat server (figure 5): figure 5. add the ncipcore project to the tomcat server we are almost there, just a few last steps to configure.  create a new environment variable named ncip_root_dir.  point that environment variable to a directory of your choice.  you’ll need to put the ncip configuration files in that folder as follows: copy /folder_where_i_downloaded_code/ncip-instance/ to the folder that your environmental variable points to. [ncip_root_dir] rename the ncipv2 folder ncipcore modify the log4j.config.txt file to point to your tomcat log file start up tomcat – you should not see any errors in the console.  the console messages should look similar to figure 6: figure 6. console messages of the ncipcore project point your browser to http://localhost:8080/ncipcore/.  your port may vary, but typically tomcat is configured to run on port 8080.  you should see something like the a simple web page  shown in figure 7: figure 7. index.html file that comes with the xc ncip toolkit. figure 7 shows the index.html file that comes with the toolkit.  you can use it to send test ncip xml requests to the application.  sample xml can be found on the xc ncip2 toolkit google code site [5]. by default the xc toolkit is configured to use the “dummy connectors” that come with the project.  these configuration settings are located the driver_config.properties file (figure 8) which was placed into the  ncip_root_dir environmental variable folder. figure 8. driver_config.properties file now that the toolkit is running on your tomcat server, if you are interested in walking through the code, a good place to start is to place a breakpoint in the “dopost” method of the ncipservlet class.  this is the class that catches all of the xml requests.  in a sense, it is the front door to the application: @override public void dopost(httpservletrequest request, httpservletresponse response) throws ioexception, servletexception { response.setcontenttype("application/xml; charset=\"utf-8\""); servletinputstream inputstream = request.getinputstream(); ncipinitiationdata initiationdata = null; try { initiationdata = translator.createinitiationdata(inputstream); } catch (serviceexception e) { returnexception(response, "exception creating the ncipinitiationdata object from the servlet's input stream.", e); } if (initiationdata != null) { ncipresponsedata responsedata = messagehandler.performservice(initiationdata); try { inputstream responsemsginputstream = translator.createresponsemessagestream(responsedata); try { as illustrated in the code snippet below, the jaxbtranslator class detects the type of ncip service being called and transforms the xml from the request into the appropriate type of request object based on the service type: public org.extensiblecatalog.ncip.v2.service.ncipinitiationdata createinitiationdata(ncipmessage initmsg) throws bindingexception { org.extensiblecatalog.ncip.v2.service.ncipinitiationdata initiationdata; if (initmsg.getacceptitem() != null) { initiationdata = createacceptiteminitiationdata(initmsg.getacceptitem()); } else if (initmsg.getcheckinitem() != null) { initiationdata = createcheckiniteminitiationdata(initmsg.getcheckinitem()); } else if (initmsg.getcheckoutitem() != null) { initiationdata = createcheckoutiteminitiationdata(initmsg.getcheckoutitem()); } else if (initmsg.getlookupitem() != null) { initiationdata = createlookupiteminitiationdata(initmsg.getlookupitem()); } else if (initmsg.getlookuprequest() != null) { initiationdata = createlookuprequestinitiationdata(initmsg.getlookuprequest()); } else if (initmsg.getlookupuser() != null) { initiationdata = createlookupuserinitiationdata(initmsg.getlookupuser()); } else if (initmsg.getrenewitem() != null) { initiationdata = createrenewiteminitiationdata(initmsg.getrenewitem()); } else if (initmsg.getrequestitem() != null) { initiationdata = createrequestiteminitiationdata(initmsg.getrequestitem()); then, the performservice method of the messagehandler class uses the configuration files to detect which connector service will process this request.  out-of-the-box this will be one of the ‘dummy’ services that come with the xc toolkit core. public ncipresponsedata performservice(ncipinitiationdata initiationdata) { ncipresponsedata responsedata; ncipservice<ncipinitiationdata, ncipresponsedata> service = null; if (supportedservices != null) { service = supportedservices.get(initiationdata.getclass().getname()); } // todo: the dummyservice shouldn't require this special handling it should return the unsupported service error itself if (service != null &amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp; !service.getclass().getname().equalsignorecase("org.extensiblecatalog.ncip.v2.service.dummyservice")) { try { responsedata = service.performservice(initiationdata, servicemanager); the extensible catalog organization has written documentation to explain these classes and relationships in much more detail.  you can find this documentation on their google code site [6]. this is where the xc toolkit core functionality needs the connector code (to replace the dummy connector classes).  the connector code must consist of one class for each supported service.  the connector classes are the bridge.  they are responsible for communicating with the catalog, parsing those results and returning a response object back to the messagehander.performservice method as you see above. the symphony connector code & configuration files to replace the ‘dummy’ connector services with the symphony connector services, replace the ‘dummy’ configuration file that comes with the toolkit with the symphony configuration file.  place the symphony (or your own) configuration file here: [ncip_root_dir]/ncip-instances/ncipv2/driver/driver_config.properties. this will cause the toolkit to call the symphony (or your own) service classes instead of the ‘dummy’ service classes.  the configuration file for the connector also contains other values that allow the services to be customized for each sirsidynix customer, e.g. perl script url (figure 9): figure 9. driver_config.properties file for the symphony connector in eclipse, set up the symphony (or your own) connector code in its own ‘java’ project as shown in figure 10: figure 10. illustrates the ncipcore and symphonyconnector projects within eclipse as you can see there are classes for each of the four services needed for the project here at lehigh: symphonyacceptitemservice symphonylookupuserservice symphonycheckinitemservice symphonycheckoutitemservice in addition to the service classes there is a symphonyremoteservicemanager and several other classes created to hold results, errors or constants.  the service classes (one for each supported service) and the symphonyremoteservicemanager class are part of the framework suggested by the extensible catalog organization for any connectors developed to extend the toolkit.  this framework is consistent among the other connectors that have been developed to date. the extensible catalog organization does an excellent job of explaining the framework and guidelines for connector development [7]. in the symphony implementation, the service classes validate the input (when needed) and then makes a call to the symphonyremoteservicemanager which calls the api and parses the results: @override public checkoutitemresponsedata performservice(checkoutiteminitiationdata initdata, remoteservicemanager servicemanager) { final checkoutitemresponsedata responsedata = new checkoutitemresponsedata(); userid userid = this.retreiveuserid(initdata); userid.setagencyid(new schemevaluepair(symphonyconstants.agency_id)); itemid itemid = initdata.getitemid(); responsedata.setuserid(userid); itemid.setagencyid(new schemevaluepair(symphonyconstants.agency_id)); responsedata.setitemid(itemid); checkouttransactionreults checkouttrans = new checkouttransactionreults(); try { validateuserid(userid); validateitemid(itemid); } catch(exception exception) { if (responsedata.getproblems() == null) responsedata.setproblems(new arraylist<problem>()); problem p = new problem(new schemevaluepair(symphonyconstants.check_out_problem),symphonyconstants.check_out_input_problem,exception.getmessage(),exception.getmessage()); responsedata.getproblems().add(p); return responsedata; } try { checkouttrans = this.callsirsi(servicemanager,userid,itemid); checksirsiresponseforerrors(checkouttrans); } private checkouttransactionreults callsirsi(remoteservicemanager servicemanager,userid userid,itemid itemid) { symphonyremoteservicemanager sirsiservicemanager = (symphonyremoteservicemanager)servicemanager; checkouttransactionreults checkouttransaction = sirsiservicemanager.checkoutitem(userid, itemid); return checkouttransaction; } the call to the symphony api is accomplished by calling the url of a perl script which makes the direct api call: public checkouttransactionreults checkoutitem(userid userid,itemid itemid) { checkouttransactionreults checkouttrans = new checkouttransactionreults(); try { url url = new url(symphonyremoteservicemanager.scripturl + "checkoutitem.pl?id="+userid.getuseridentifiervalue()+"&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;itemid="+itemid.getitemidentifiervalue()); stringtokenizer stringtokenizer = this.callurl(url); heckouttrans = this.parsecheckoutresults(stringtokenizer); } catch(exception e) { checkouttrans.setstatuscode(symphonyconstants.sirsi_failed); } return checkouttrans; } parsing the results… private checkintransactionresults parsecheckinresults(stringtokenizer stringtokenizer) { checkintransactionresults checkintransaction = new checkintransactionresults(); hashtable<string,string> ht = new hashtable<string,string>(); ht = this.stringtokenizertohashtable(stringtokenizer); checkintransaction.setuserid((string)ht.get(symphonyconstants.sirsi_userid)); checkintransaction.setcallnumber((string)ht.get(symphonyconstants.sirsi_call_number)); checkintransaction.setauthor((string)ht.get(symphonyconstants.sirsi_author)); checkintransaction.settitle((string)ht.get(symphonyconstants.sirsi_title)); checkintransaction.setstatuscode((string)ht.get(symphonyconstants.sirsi_message_number)); return checkintransaction; } …which are then returned to the service class.  the connector service class populates the response data object which will be returned back to the toolkit core code.  the toolkit core code will then transform the response object into xml to send back as the response. setting up the symphony project in eclipse it can sometimes be difficult to get multiple projects ‘talking’ to each other and deployed to tomcat within eclipse.  for those that wish to duplicate this setup, please find below the configuration settings i used get this all working.  the symphonyconnector project needs to know about the ncipcore project (figure 11): figure 11. adding the ncipcore project to the build path of the symphony connector project the ncipcore project does not need to know about the symphonyconnector project (figure 12): figure 12. projects on the build path of the ncipcore project i have included both of the projects in the tomcat classpath (figure 13): figure 13. tomcat classpath testing much of the testing was done using an enhanced version of the index.html file that comes with the core toolkit.   as shown below in figure 14, i added javascript that allowed me to ‘toggle’ between each of the four service calls.  this saved me the steps of cutting and pasting. figure 14. a modified version of the toolkit test page the javascript required for this functionality: <script type="text/javascript"> xmlarray = new array(); xmlarray['lookupuser']="<?xml version = '1.0' encoding='utf-8'?><ncipmessage version='http://www.niso.org/schemas/ncip/v2_0/ncip_v2_0.xsd' xmlns='http://www.niso.org/2008/ncip' xmlns:xsi='http://www.w3.org/2001/xmlschema-instance'><lookupuser><userid>  <agencyid>nfsm</agencyid>  <useridentifiervalue>123456789</useridentifiervalue></userid><userelementtype>name information</userelementtype><userelementtype>user address information</userelementtype><userelementtype>user language</userelementtype><userelementtype>user privilege</userelementtype><userelementtype>user id</userelementtype></lookupuser></ncipmessage>"; xmlarray['checkinitem']="<?xml version = '1.0' encoding='utf-8'?><ncipmessage version='http://www.niso.org/schemas/ncip/v2_0/ncip_v2_0.xsd' xmlns='http://www.niso.org/2008/ncip' xmlns:xsi='http://www.w3.org/2001/xmlschema-instance'> <checkinitem><itemid>  <agencyid>your agency id</agencyid>  <itemidentifiervalue>39159999985</itemidentifiervalue></itemid><itemelementtype>bibliographic description</itemelementtype><itemelementtype>item description</itemelementtype></checkinitem></ncipmessage>"; xmlarray['checkoutitem']="<?xml version = '1.0' encoding='utf-8'?><ncipmessage version='http://www.niso.org/schemas/ncip/v2_0/ncip_v2_0.xsd' xmlns='http://www.niso.org/2008/ncip' xmlns:xsi='http://www.w3.org/2001/xmlschema-instance'><checkoutitem><userid>  <agencyid>your agency id</agencyid>  <useridentifiervalue>123456789</useridentifiervalue></userid><itemid>  <agencyid>your agency id</agencyid>  <itemidentifiervalue>39999995</itemidentifiervalue></itemid></checkoutitem></ncipmessage>"; xmlarray['acceptitem']="<?xml version = '1.0' encoding='utf-8'?><ncipmessage version='http://www.niso.org/schemas/ncip/v2_0/ncip_v2_0.xsd' xmlns='http://www.niso.org/2008/ncip' xmlns:xsi='http://www.w3.org/2001/xmlschema-instance'><acceptitem><requestid><agencyid>relais</agencyid><requestidentifiervalue>reg-12345679</requestidentifiervalue></requestid><requestedactiontype>hold for pickup</requestedactiontype><userid><agencyid>your agency id</agencyid><useridentifiervalue>123456789</useridentifiervalue></userid><itemid><agencyid>your agency id</agencyid><itemidentifiervalue>39999995885</itemidentifiervalue></itemid><itemoptionalfields><bibliographicdescription><author>author</author><title>title</title></bibliographicdescription><itemdescription><callnumber>call number</callnumber></itemdescription></itemoptionalfields><pickuplocation>your pickup location code</pickuplocation></acceptitem></ncipmessage>"; function  changexml(servicetype) { document.getelementbyid('ncipdoc').value = xmlarray[servicetype]; } </script> the clickable links that call the javascript changexml method: <a href="javascript:changexml('lookupuser')">show lookup user xml</a>  | <a href="javascript:changexml('checkinitem')">show checkin item xml</a> | <a href="javascript:changexml('checkoutitem')">show checkout item xml</a>  | <a href="javascript:changexml('acceptitem')">show accept item xml</a> for performance testing the services, i use jmeter [8] which is an apache open source project that allows me to create repeatable tests for valid input, invalid input and also stress testing (figure 15): figure 15. apache jmeter deploying the software (to jetty 7.2.0) our implementation of the toolkit and connector is deployed in our production environment using jetty.  i do not ‘package’ the ncip core code from eclipse to deploy.  instead i simply download the .war file provided by the extensible catalog.  i do package the connector code from eclipse by exporting the project (class files only) as a .jar file.  the toolkit .war file is placed in the webapps directory of jetty.  the connector code, packaged as a .jar file, is placed in the [ncip_root_dir]/ncip-instances/ncipv2/driver/ directory together with the driver configuration file.  (ncip_root_dir represents the directory your environmental variable points to.) open sourcing the connector code & perl scripts all of the symphony connector java code has been open-sourced.  it can be found on the xc ncip2 toolkit google code website [9]. the perl scripts that are called by the java connector code have also been made available to any symphony api customer on symphony’s ‘sirsiapi.org’ site [10]. a user id and password provided by symphony are required to access this site.  this is the best we could do as far as open-sourcing these scripts, due to the use of the proprietary sirsidynix api.  the scripts consist of a call to the symphony api and a return of the resulting string to the java method calling it as show below in figure 16: figure 16. perl script that calls the proprietary sirsidynix api results of the project and getting involved we have been using the toolkit and the connector code in a production environment (on jetty) since late march 2011.   since then, the services have been called approximately 4,000 times without any downtime or issues.  while this is not a terribly high volume, we have been happy with the stability and performance. in the end, i can’t help but to reflect upon the simple process of the symphony connector code.  it validates input, calls an api, parses the results and constructs and populates the response object.  there are approximately fifty services that make up the entire ncip protocol.  the code that makes up the symphony connector – just four services –  is reliable and does exactly what we needed it to do support our patrons use of the relais ez-borrow software. however, as the symphony connector code grows to fully support all of the ncip services, i would like to see the design be considered so that it scales in a clean and elegant manor.  i am thinking along the lines of mapping the xml elements to the sirsi api codes to maximize reuse of the java clases. if you are interested in getting involved with the xc ncip2 toolkit project, there is a bi-weekly conference call managed by randy cook [11] who is the project manager for xc.  randy and the developers that work on this project have been wonderful to work with and extremely responsive to all of my questions.  this is a worthy project with the potential to benefit any organization looking for a way to access their catalog using ncip. software list eclipse java ee ide for web developers, version: helios release, build id: 20100617-1415 apache tomcat 7.0.2 jetty 7.2.0 apache maven 2.2.1 apache jmeter 2.4 r961953 subversion 1.6.12 references [1] pennsylvania academic library consortium, inc. http://www.palci.org/ [2] relais international http://www.relais-intl.com/relais/home/index.htm [3] sirsidynix http://www.sirsidynix.com/ [4] extensible catalog organization http://www.extensiblecatalog.org/ [5] extensible catalog xc ncip 2 toolkit google code website – xml samples http://code.google.com/p/xcncip2toolkit/wiki/samplexml [6] extensible catalog xc ncip 2 toolkit google code website – class diagram http://code.google.com/p/xcncip2toolkit/wiki/classdiagram [7] extensible catalog xc ncip 2 toolkit google code website – how to write an ils connector for an ncip service http://code.google.com/p/xcncip2toolkit/wiki/writeconnector [8] apache jmeter http://jakarta.apache.org/jmeter/ [9] extensible catalog xc ncip 2 toolkit google code website http://code.google.com/p/xcncip2toolkit/source/browse/ [10] sirsi api voluntary code repository (*password protected site*) http://sirsiapi.org/ [11] extensible catalog organization – randall cook http://www.extensiblecatalog.org/about/people/randall-cook about the author michelle suranofsky is a sr. developer at lehigh university in bethlehem, pa.  before joining lehigh in 2010, michelle worked for 10 years developing and maintaining web applications primarily with java at chubb insurance.  she can be contacted at mis306@lehigh.edu. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – discovering digital library user behavior with google analytics mission editorial committee process and structure code4lib issue 17, 2012-06-01 discovering digital library user behavior with google analytics google analytics has advanced features for tracking search queries, events such as clicking external links or downloading files, which you can use to track user behavior that is normally difficult to track with traditional web logging software. by tracking behavior, you can use google analytics api to extract data and integrate it with data from your digital repository to show granular data about individual items. using this information, digital libraries can learn how users use the site without extensive hci studies, and can use this information to improve the user experience. by kirk hess introduction in my first digital library job, every so often i would have to compile reports based on the server logs to answer a question about the site. questions such as “how many times was x document downloaded”, or “what’s the most popular item in our collection?” were common and i answered them using awstats data collected from our server logs. however, i was concerned the results were not accurate because of many inconsistencies, such as how many of the ip addresses were from non-.edu domains, or the page entry and exit numbers didn’t match up. while working on the illinois harvest portal i found there was no web analytics solution enabled on the website, and the web logs were no longer easily accessible. i turned to google analytics, which allowed me to answer simple questions about numbers of user hits. but one of the things i found that google analytics didn’t do well out of the box was tracking two types of events: clicks on links to other domains, such as the internet archive, and clicks which downloaded files. by implementing event tracking, i was able to determine which items were used and which ones weren’t being used, and i could track complete user interactions with the website. the goal of this paper is to help librarians and programmers who develop and maintain digital library websites to use events and site search within google analytics. by gathering better intelligence about what users are doing, you can determine what you should change on the website to make it more useful to users without having to do extensive usability tests or human-computer interaction (hci) studies. web analytics the digital analytics association defines web analytics as: … the measurement, collection, analysis and reporting of internet data for the purposes of understanding and optimizing web usage. [1] there are many potential solutions, both log based and script based for web analytics. logging solutions use raw server logs to mine the information for usage trends. your typical awstats logs will track hits, which usually are http get requests for something, a page, an image, a file, etc. hits on your webserver are going to be from humans, but because of the way the web is crawled, a substantial portion will often be from various automated systems like googlebot or bingbot. solutions like awstats do a good job of trying to filter out all this noise, but the problem is you never get rid of 100% of the noise. in addition, a significant subset of bots try their best to avoid detection. google analytics uses a javascript-based approach that injects a tiny image onto each page for tracking. because most automated crawlers do not have javascript or cookies enabled, and most modern browsers do, you have a high level of confidence that the hits captured are human users. google analytics is a simple, inexpensive, and effective way to capture data about your users. google analytics basics google analytics is easy to set up on any website following the basic instructions. generally, you sign in, input the information about your site and put the javascript <script> element in the <head> portion of your website. of course, every system builds the <head> element for pages a little differently, and since you want to include the tracking cookie on every page in a consistent way you’ll need to figure out how to do this in your system. this hopefully is as simple as putting it in an include file which is included when the page is rendered. <script type="text/javascript"> var _gaq = _gaq || []; _gaq.push(['_setaccount', 'ua-xxxxxxxx-1']); _gaq.push(['_setallowlinker', true]); _gaq.push(['_setdomainname', 'illinoisharvest.grainger.uiuc.edu']); _gaq.push(['_setallowhash', false]); _gaq.push(['_trackpageview']); (function() { var ga = document.createelement('script'); ga.type = 'text/javascript'; ga.async = true; ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js'; var s = document.getelementsbytagname('script')[0]; s.parentnode.insertbefore(ga, s); })(); </script> finally, once you figure this out it’s hard to tell if tracking actually works because there is usually some minor amount of lag in between installing the code and actually getting data in the online tool. to see the tracking cookie using firebug (or its chrome/ie/safari equivalents), you simply refresh the page with the net, images tab visible. if you see an image starting with _utm.gif, you are tracking that page. figure 1. firebug showing _utm.gif?… tracking cookie. tracking events to gather information such as clicking external links or downloading files, we’ll enable event tracking to track those events. again, google has this documented pretty well online. events are segmented into categories, actions, and labels. for our site, i chose two types of categories of events, external and download. for actions, i chose click, and click-, extension varying based on the type of file downloaded. the label is optional, however i chose to include the link url or the file name. one thing i would have probably preferred was to include the internal identifier for the item that would have made linking files to items easier to manage. in your system you may simply be able to change the code that displays links, however in our system that was too complex so instead i wrote a jquery script that does the work after the page has been loaded, which would also be useful for tracking a vendor system that doesn’t allow changes to the code. jquery binds an event handler to the “click” javascript event: <script type="text/javascript"> if (typeof jquery != 'undefined') { jquery(document).ready(function($) { var filetypes = /\.(pdf|txt|dijv|xml)$/i; var basehref = ''; if (jquery('base').attr('href') != undefined) basehref = jquery('base').attr('href'); jquery('a').each(function() { var href = jquery(this).attr('href'); if (href & (href.match(/^https?\:/i)) & (!href.match(document.domain))) { jquery(this).click(function() { var extlink = href.replace(/^https?\:\/\//i, ''); _gaq.push(['_link', href]); _gaq.push(['_trackevent', 'external', 'click', extlink]); if (jquery(this).attr('target') != undefined & jquery(this).attr('target').tolowercase() != '_blank') { settimeout(function() { location.href = href; }, 200); return false; } }); } else if (href & href.match(filetypes)) { jquery(this).click(function() { var extension = (/[.]/.exec(href)) ? /[^.]+$/.exec(href) : undefined; var filepath = href; _gaq.push(['_trackevent', 'download', 'click-' + extension, filepath]); if (jquery(this).attr('target') != undefined & jquery(this).attr('target').tolowercase() != '_blank') { settimeout(function() { location.href = basehref + href; }, 200); return false; } }); } }); }); } </script> the result is you get code like this injected onto your links: _gaq.push(['_trackevent', 'external', 'click', extlink] site search another useful option we can enable is search tracking, and this is covered pretty well by the online documentation. here’s an example of our configuration for tracking site search. figure 2. site search configuration. collect data once everything is set up correctly, you will need to wait as activity is collected. google analytics has some very nice dashboard utilities to view the results of the site in real-time. below are a couple of images of the results that i will discuss further in the analysis section: top pages by events and search results. figure 3. report of top pages with events from october 1 2011 to april 1 2012. figure 4. report of search queries from october 1 2011 to april 1 2012. browse results not displayed. export data by only using the dashboard utility, i found that i wasn’t getting the answers i wanted–especially for downloads. the downloads reports either were too granular (by extension) or too broad (all events), and they couldn’t answer one important question: what wasn’t being used? google provides an api to extract data into whatever format you like; since our data is stored within a ms sql database i wanted to export it into a table so i could correlate the results to all items in a particular collection. i found it easier to do with java; other programming languages are supported. code example: java class for extracting data /** * retrieve data from the google analytics api. */ public class googleanalytics { // credentials for client login authorization. private static final string client_username = "xxx"; private static final string client_pass = "xxxx"; private static final string[] eventdates; //define as needed // table id constant private static final string table_id = "ga:123456789"; public static void main(string args[]) { try { // service object to work with the google analytics data export api. analyticsservice analyticsservice = new analyticsservice("gaexportapi_acctsample_v2.0"); // client login authorization. analyticsservice.setusercredentials(client_username, client_pass); // get data from the account feed. getaccountfeed(analyticsservice); // access the data feed if the table id has been set. if (!table_id.isempty()) { // get profile data from the data feed. getdatafeed(analyticsservice); } } catch (authenticationexception e) { system.err.println("authentication failed : " + e.getmessage()); return; } catch (ioexception e) { system.err.println("network error trying to retrieve feed: " + e.getmessage()); return; } catch (serviceexception e) { system.err.println("analytics api responded with an error message: " + e.getmessage()); return; } } /** * @param {analyticsservice} google analytics service object that * is authorized through client login. */ private static void getaccountfeed(analyticsservice analyticsservice) throws ioexception, malformedurlexception, serviceexception { // construct query from a string. url queryurl = new url( "https://www.google.com/analytics/feeds/accounts/default?max-results=50"); // make request to the api. accountfeed accountfeed = analyticsservice.getfeed(queryurl, accountfeed.class); // output the data to the screen. system.out.println("-------account feed results --------"); for (accountentry entry : accountfeed.getentries()) { system.out.println( "\naccount name = " + entry.getproperty("ga:accountname") + "\nprofile name = " + entry.gettitle().getplaintext() + "\nprofile id = " + entry.getproperty("ga:profileid") + "\ntable id = " + entry.gettableid().getvalue()); } } /** * @param {analyticsservice} google analytics service object that * is authorized through client login. */ private static void getdatafeed(analyticsservice analyticsservice, string[] eventdates) throws ioexception, malformedurlexception, serviceexception { stringbuffer skippedrecords = new stringbuffer(); int skipcount = 0; // create a query using the dataquery object. for(string eventdate : eventdates){ dataquery query = new dataquery(new url( "https://www.google.com/analytics/feeds/data")); query.setstartdate(eventdate); query.setenddate(eventdate); query.setdimensions("ga:eventlabel,ga:eventcategory,ga:pagepath"); query.setmetrics("ga:totalevents,ga:uniqueevents"); query.setsort("-ga:totalevents"); query.setmaxresults(500); query.setids(table_id); // make a request to the api. datafeed datafeed = analyticsservice.getfeed(query.geturl(), datafeed.class); } } analyze data and next steps one of the things that i wondered about with illinois harvest on my first day was did we have any users at all? the basic statistics showed over the previous 6 months a total of 15k unique users visited the site. on average, those users visited about 4 pages, with a bounce rate (a.k.a. a single-page visit) of approximately 35%. figure 3 shows that we had many users clicking on external links and download links – the top 3 pages were digitized book landing pages, while the 4th page for the collection summary contains many external links. specifically, the reason the top page had so many hits is it’s one of the references for a wikipedia page about the iroquois theatre fire. figure 4 shows an interesting yet disturbing result – after removing browse results, the site search is rarely used. traffic to the site lands deep into the website, either at a location determined by an external search engine, the library catalog, or other direct links. the site was designed about 6 years ago when portals were considered important and it doesn’t appear to be working for users who arrive from a search engine. by exporting the data and comparing to our item level records, i was able to see that most items were never used, where use being defined as a click to an external link or downloaded a file. figure 5 shows one of our collections, and while a couple of links get a fair amount of clicks, the vast majority of items in this collection of about 23k items are never clicked or downloaded. the disparity between the size of a collection and use my colleagues harriet green and richard hislop have termed ‘effective collection size’ or ecs, is something we are continuing to investigate as part of larger datasets. figure 5. graph of item usage from ideals collection on illinois harvest, july 1 2011 – november 1, 2011 next steps the illinois harvest user interface will be redesigned in 2012 and hopefully some of these lessons will be part of those efforts. specifically, i think site search needs to be examined more closely, probably as a full hci study, to make sure it’s useful for users. by analyzing a sample set of 40k items held by our literature and language library, we’ve found a similar disparity between the size of a collection, in this case both physical and digital items, and use. we’re going to be exploring how to improve ecs in a collection by examining a much larger data set of 22 million items indexed in the university of illinois library catalog. based on the network analyses resulting from this project, we will begin development of an enhanced recommender system for library catalogs and digital libraries that retrieves richer search results from a library collection search based on network analysis of subject relevancy, circulation data of items, and usage data for items that share interrelated subjects. notes [1] about us. digital analytics association. available from http://www.digitalanalyticsassociation.org/?page=aboutus appendix: resources tools used javascript jquery java mssql code examples are available on github further reading about google analytics and libraries prom, christopher j. using web analytics to improve online access to archival resources. american archivist, spring/summer 2011, vol. 74 issue 1, p158-184 marshall breeding, “an analytical approach to assessing the effectiveness of web-based resources,” computers in libraries 28 (january 2008): 20–22 feng wei, “using google analytics for improving library website content and design: a case study,” library philosophy and practice (july 2007), http://digitalcommons.unl.edu/libphilprac/121 about the author kirk hess (kirkhess@illinois.edu) is digital humanities specialist at the university of illinois urbana-champaign, and holds a mlis from the university of illinois urbana-champaign. back subscribe to comments: for this article | for all articles 4 responses to "discovering digital library user behavior with google analytics" please leave a response below: new code4lib issue out | learn@the corkboard, 2012-06-25 […] discovering digital library user behavior with google analytics […] judah, 2012-06-28 great article. very interesting to learn that search was so under-utilized. weekendvitaminen #33 « zeemanspraat, 2012-07-06 […] kirk hess, discovering digital library user behavior with google analytics (code4lib, issue 17) […] ux and seo — scott w. h. young, 2015-04-27 […] the role and impact of seo has been a focus of study for some time now. the conversation around seo is dominated by digital marketers and others with commercial interests, but librarians have been drawn to this discussion after observing google’s impressive abilities and accomplishments (to the point of near-total domination of search traffic). our own library dean has published a paper addressing the low indexing rate of institutional repository content in google scholar. others within libraries have explored social traffic for digital collections, seo for library web content, and the use of google analytics for understanding user behavior. […] leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – annif analyzer shootout: comparing text lemmatization methods for automated subject indexing mission editorial committee process and structure code4lib issue 54, 2022-08-29 annif analyzer shootout: comparing text lemmatization methods for automated subject indexing automated text classification is an important function for many ai systems relevant to libraries, including automated subject indexing and classification. when implemented using the traditional natural language processing (nlp) paradigm, one key part of the process is the normalization of words using stemming or lemmatization, which reduces the amount of linguistic variation and often improves the quality of classification. in this paper, we compare the output of seven different text lemmatization algorithms as well as two baseline methods. we measure how the choice of method affects the quality of text classification using example corpora in three languages. the experiments have been performed using the open source annif toolkit for automated subject indexing and classification, but should generalize also to other nlp toolkits and similar text classification tasks. the results show that lemmatization methods in most cases outperform baseline methods in text classification particularly for finnish and swedish text, but not english, where baseline methods are most effective. the differences between lemmatization methods are quite small. the systematic comparison will help optimize text classification pipelines and inform the further development of the annif toolkit to incorporate a wider choice of normalization methods. by osma suominen, ilkka koskenniemi introduction libraries and other cultural heritage institutions are increasingly adopting natural language processing (nlp) and machine learning technologies to assist in the production of metadata. a common need in such settings is to automatically or semi-automatically classify text documents, for example performing subject indexing or to classify documents into mutually exclusive classes. one practical tool for these kinds of tasks is the automated subject indexing toolkit annif (suominen, inkinen and lehtinen 2022; suominen 2019). this study concentrates on a small but nevertheless important aspect of automated text classification systems, namely the use of different preprocessing and normalization methods such as stemming and lemmatization. these are particularly important in the traditional nlp setting, where text classification pipelines commonly include text preprocessing (e.g. tokenization and word normalization such as stemming or lemmatization), vectorization (turning words into numbers, often represented as vectors using a tf-idf transformation), model training and inference. the goal of word normalization in such a pipeline is to reduce the amount of unimportant linguistic variation by excluding aspects such as singular vs. plural and inflected word forms. for example, the words eat, ate, eats, eating and eaten all share the same base form eat and refer to the same activity, yet without normalization they would all be considered completely different words and a classification algorithm would need to learn their meanings separately. normalizing them all into the same form with a stemming or lemmatization algorithm reduces the amount of noise in the data and will in many cases both reduce the required amount of training data and improve overall classification results. most of the algorithms implemented in annif follow this traditional nlp paradigm, and in the modular architecture of annif, the text preprocessing is performed by an analyzer module. this study explores the effect of different text normalization choices on the quality of the resulting automated subject indexing suggestions. the comparison includes both algorithms currently implemented as analyzer modules in annif as well as several other lemmatization algorithms. we mainly selected open source lemmatization tools that were easy to use from python code. as this study was performed in a collaborative project between the national library of finland and lingsoft inc., we also included the proprietary lingsoft language management central (lmc) platform for language structure analysis in the comparison. in order to compare the effect of different analyzer methods, we set up a series of experiments where the setting (corpora, algorithms, parameter settings, evaluation metrics) is static except for the text normalization method. we chose two subject prediction tasks, informed by the requirements of the finto ai automated subject indexing service. the first task is an example of subject indexing with a thesaurus, where the goal is to assign a small number (typically 5–12) of relevant subjects to each document, which will improve the discoverability of the collection. the second task is an example of library classification, where the goal is to place each document in a single class or category – which, in a traditional library setting, would determine the location of a book on the shelf. we selected appropriate corpora, algorithms and other settings for each of the two tasks: automated subject indexing with the general finnish ontology yso (including yso places). by now, this is a well established task which has been supported by the finto ai service, and its experimental predecessor systems, for several years. this task is further divided by language: finnish, swedish and english. automated classification with the finnish public library classification ykl, which is closely related to the dewey decimal classification used around the world. this is currently a more experimental task which, at the time of writing, has not yet been turned into a production service. however, we have collected suitable training and evaluation data and tested different algorithms on this task. we only perform this task in finnish due to the lack of enough training data in other languages. the goal of these experiments is to find out what kind of lemmatization methods (or other kinds of text normalization) give the best results when used in a typical nlp pipeline that performs text classification. the experiments are not intended to benchmark the classification algorithms themselves, as this has been done before (see, e.g., suominen, inkinen and lehtinen 2022; uhlmann and grote 2021), but to highlight the effect of the lemmatization method on the downstream evaluation results. we hope that the experiment will help inform choices about which lemmatization method to select for a particular type of task and classification setting and to quantify how much that choice matters for the success of the classification task. although the experimental setting is focused on the annif tool, the results should generalize to other text classification settings that follow the traditional nlp paradigm, for example custom text classifiers implemented with the scikit-learn machine learning toolkit. the experiment compares a wide range of lemmatization methods, ranging from very simplistic rule-based approaches to computationally intensive state-of-the-art neural network pipelines which perform many other functions than lemmatization, including part-of-speech (pos) tagging, dependency resolution, named entity recognition (ner) and named entity linking (nel). for this experiment, only the result of the lemmatization matters, and it doesn’t even have to be linguistically correct, just consistent enough to adequately support the classification algorithms that make use of the lemmatized output. as an example, simple stemming algorithms convert words into stems (e.g. “qualities” becomes “qualiti” using snowball stemming) that often aren’t valid word forms but can still be used to group together related words. corpora for training and evaluation we selected preexisting corpora that we had already been using to train and evaluate annif models, in particular in the finto ai service for automated subject indexing. the corpora consist of manually indexed raw text in the form of txt and tsv files. table 1 gives an overview of the corpora used in the experiments. the total amount of raw text included in all the corpora was approximately 1.1 gb. task / language data set train set size used for training test set size raw text yso finnish finna 7.0m records parabel, bonsai – 360 mb jyu-theses 500 documents mllm 764 documents 270 mb bookdesc 500 documents mllm 1311 documents 1.9 mb yso swedish finna 920k records parabel, bonsai – 50 mb jyu-theses 103 documents mllm 32 documents 25 mb bookdesc 744 documents mllm 560 documents 1.1 mb yso english finna 2.0m records parabel, bonsai – 120 mb jyu-theses 498 documents mllm 300 documents 190 mb bookdesc 826 documents mllm 447 documents 1.3 mb ykl finnish bookdesc 51187 documents parabel, bonsai 4237 documents 46 mb table 1. overview of document corpora used in the experiment. to test subject indexing using concepts from the general finnish ontology yso, we chose a snapshot of yso and yso places dated 2021-09-23, training corpora based on metadata records collected from the finna.fi discovery system (finna), and two sets of manually indexed full text documents: master’s and doctoral theses from the university of jyväskylä (jyu-theses), as well as titles and short descriptions of non-fiction books collected from the database of kirjavälitys oy (bookdesc). these corpora include documents in three languages (finnish, swedish and english) and have been used to train and evaluate the models in the annif-based automated subject indexing service finto ai. we also included a finnish language corpus which is based on the same book descriptions as the bookdesc data set, but classified using the finnish public library classification ykl, in order to test the effect of text preprocessing strategies in an automated classification setting. text preprocessing methods we included nine text preprocessing methods in the comparison: the baseline methods simple and snowball as well as the more advanced lemmatization methods voikko, spacy, simplemma, udpipe, stanza, lmc and tnpp. not all methods work on all three languages, as detailed below. the first four methods are implemented in annif (version 0.57.0 was used for the experiments), so we relied on the built-in support. for the other methods, we performed the lemmatization of the data files outside the annif tool, in most cases using jupyter notebooks (google colaboratory in the case of tnpp and a custom python script in the case of lmc), and stored the results in corpus files that were subsequently used instead of the original, unlemmatized files. we configured the corresponding annif projects to use the simple analyzer, which converts the input text into lower case. baseline methods the simple analyzer, implemented in annif, is a baseline, language-agnostic method that folds the input text into lower case but does not attempt to perform any stemming or lemmatization. the snowball analyzer in annif is another baseline method that converts words into lower case and additionally performs stemming using the snowball algorithm, as implemented in the python natural language toolkit (nltk). the stemming rules are specific to each language and the implementation in nltk supports at least 16 languages, including finnish, swedish and english. lemmatization methods the voikko analyzer in annif is based on the voikko morphological analysis tool, which was originally developed as a finnish language spell checker for the openoffice and libreoffice applications. this analyzer also converts individual words into lower case. voikko only supports the finnish language. the spacy analyzer in annif performs lemmatization using spacy (version 3.2.4), which is a popular, multilingual nlp toolkit. lemmatization requires installing a language-specific pretrained model, but among the languages used in this comparison, only english was supported (support for finnish and swedish was added in spacy 3.3, which was released after the experiments had already been completed). we tested two variations of this method: with and without forced lowercasing of individual lemmas. the lowercased variant is called spacy_lower in the remainder of this paper. the simplemma method was implemented using the simplemma library (version 0.6.0), a simple multilingual lemmatizer for python which does not need morphosyntactic information and works on the level of individual words, which makes it extremely fast. it has support for 38 languages, including the three languages used in this comparison. the following code demonstrates how simplemma was initialized and used for lemmatization: # load the english model import simplemma langdata = simplemma.load_data('en') # lemmatize text using simplemma def lemmatize(text): lemmas = simplemma.text_lemmatizer(text, langdata) return " ".join(lemmas) the udpipe method was implemented using the spacy + udpipe package (version 1.0.0) that wraps the udpipe nlp pipeline (version 1, originally implemented in c++ and accessible through python bindings) so that it can be used via the spacy api. it supports more than 100 languages or variants, including all three languages used in this comparison; the language-specific models need to be downloaded separately. the following code demonstrates how spacy + udpipe was initialized and used for lemmatization: # download the english model import spacy_udpipe spacy_udpipe.download('en') # load the english model nlp = spacy_udpipe.load('en') # lemmatize text using spacy + udpipe def lemmatize(text): lemmas = [token.lemma_ for token in nlp(text)] return " ".join(lemmas) the stanza method was implemented using the multilingual stanza nlp toolkit (version 1.3.0) to perform lemmatization. stanza is based on a neural network architecture and like spacy and udpipe, it requires downloading language-specific pretrained models. it supports all three languages that were used in this comparison. the following code demonstrates how stanza was initialized and used for lemmatization: # download the english model import stanza stanza.download('en') # limit the maximum number of threads for best performance import os os.environ["omp_num_threads"] = "1" # initialize the stanza nlp module for english # only include the processors necessary for lemmatization nlp = stanza.pipeline(lang='en', processors='tokenize,mwt,pos,lemma') # lemmatize text using stanza def lemmatize(text): doc = nlp(text) lemmas = (word.lemma for sent in doc.sentences for word in sent.words if word.lemma) return " ".join(lemmas) the lmc method refers to lemmatization using lingsoft’s proprietary text analysis and proofing platform, the language management central (lmc). it is a language technology platform through which various nlp tools are made available as cloud services, including lemmatization, pos tagging, syntactical disambiguation, ner and nel, keywording and sentiment analysis. supported languages include finnish, swedish, english, norwegian, and danish. its lemmatization capabilities are based on lingsoft’s fintwol, a rule-based morphological model of the finnish language and fincg, a constraint grammar disambiguator for finnish. the twol and cg for the lemmatization are based on the two-level morphology (koskenniemi, 1983) and constraint grammar (karlsson, 1990) research by two of lingsoft’s founders, professors fred karlsson and kimmo koskenniemi. in lmc_yso, a semantic layer extension is also used, which marks the token sequences in running text where it finds labels of terms in a given terminology/ontology like yso. for example: “negatiivinen paine ilmenee pimeässä energiassa, joka työntää galakseja kauemmas toisistaan.” (english translation: negative pressure occurs in dark energy that pushes the galaxies further apart.) would be tagged as “negatiivinen paine ilmetä pimeä energia [pimeä_energia] , joka työntää galaksi kauemmas toinen .” (english translation: negative pressure occurs in dark energy [dark_energy] that pushes the galaxies further apart.) it attempts to mark only the sequences where the term’s label appears in a syntactically valid manner (e.g. proper congruence between constituents of a noun phrase). the idea of using the yso tagging in this test was to attempt to make it easier for the text classification algorithm to distinguish between cases where the different parts of a multi-word term label appear independently (e.g. “led-valaisin valaisee pimeässä energiaa säästäen.”) in the text from cases where the term is actually being used. the analysis run was done as a batch run on a desktop linux computer at lingsoft’s premises. the various source files were processed using a top-level shell script that sent them to a command-line wrapper tool (a python script). this tool performed the actual calls to the lmc api. this run utilized a feature in lmc that processes the tab-separated values (tsv) format. the tnpp method was implemented using the turku neural parser pipeline (tnpp), a nlp pipeline based on neural networks, to perform lemmatization. pretrained models were available for finnish and english, but we chose to test this method only for the ykl classification task because of its high resource requirements, including the need for gpu computing. we used google colaboratory to perform the lemmatization of train and test documents, adapting an example notebook (tnpp-diaparse.ipynb dated 2022-01-19) provided by the tnpp project. the notebook provides functionality to parse text and print the parsed structure in the conllu output format. for lemmatization, we used the following function whose main purpose is to convert the conllu format into a string of lemmas: from tnparser.pipeline import read_pipelines, pipeline # what pipelines do we have for the finnish model? available_pipelines = read_pipelines( "models_fi_tdt_dia/pipelines.yaml") # instantiate one of the pipelines p = pipeline(available_pipelines["parse_plaintext"]) # lemmatize text using tnpp def lemmatize(text): parsed = p.parse(text) # parsed string in conllu format lemmas = [] for line in parsed.split("\n"): line = line.strip() if not line or line.startswith("#"): continue flds = line.split("\t") lemmas.append(flds[2]) return ' '.join(lemmas) example output of each method in the following tables, we show the output of the different preprocessing methods when applied on example text. the example texts are titles of books selected from the book description data set. we picked examples that illustrate the differences between the methods; usually the differences in the output of the different methods are more subtle. note that in these examples, we have retained the original case of words as returned by the normalization algorithms, even though text was subsequently folded to lower case in most of the annif experiments. original hääd pulmapäeva ja yli 1000 muuta viron kielen riskisanaa simple hääd pulmapäeva ja yli 1000 muuta viron kielen riskisanaa snowball hääd pulmapäev ja yli 1000 muuta viro kiele riskisan voikko hääd pulmapäeva ja yli 1000 muuttaa viro kieli riskisana simplemma hääd pulmapäeva ja yli 1000 muu viro kieli riskisana udpipe hääd pulmapäeva ja yli 1000 muu viro kieli riskisana stanza hääd pulma#päeva ja yli 1000 muu viro kieli riski#sana lmc hääd pulmapäeva ja yli 1000 muuttaa viro kieli riskisana lmc_yso hääd pulmapäeva ja yli 1000 muuttaa viro kieli [viro_kieli] riskisana tnpp hääd pulma#päeva ja yli 1000 muu virko kieli riski#sana table 2. original finnish input text and the output of each preprocessing method. the finnish input text used as an example in table 2 is a bit tricky to analyze, since the first two words are actually in estonian language, which is closely related to finnish; the book is a collection of estonian words and phrases that resemble finnish, but whose meaning is different from what a finnish speaker would expect. the stanza and tnpp analyzers split compound words such as pulmapäev (estonian for ‘wedding day’) and riskisana (‘risk word’, i.e., a word that is easily misinterpreted) into their constituent parts separated using a hash character. the lmc_yso analyzer has noticed that the phrase viron kielen matches the yso concept viron kieli (‘estonian language’). some words are wrongly lemmatized: muuta (‘other’) is lemmatized into muuttaa (‘to change’) by voikko and lmc, while tnpp lemmatizes viron (‘estonian’) into virko which is a nonsensical word. original knut pipping och etableringen av den moderna sociologin vid åbo akademi simple knut pipping och etableringen av den moderna sociologin vid åbo akademi snowball knut pipping och etablering av den mod sociologin vid åbo akademi simplemma knut pipping och etablering av en modern sociologi vid åbo akademi udpipe knut pipping och etablering av en modern sociologi vid åbo akademi stanza knut pipping och etablering av en modern sociologi vid åbo akademi lmc knut pipping och etablering av den modern sociologi vid åbo akademi lmc_yso knut pipping och etablering av den modern [den_modern] sociologi vid åbo akademi table 3. original swedish input text and the output of each preprocessing method. the swedish input text in table 3 is the title of a book about the history of sociology. again, lmc_yso has noticed that the phrase den moderna matches the yso concept det moderna (‘modern’, referring to the modern age). there are no obvious mistakes in the lemmatization, although lmc has chosen a different lemma for den moderna than other lemmatizing methods; simplemma, udpipe and stanza selected en modern (‘a modern’) instead of den modern (‘the modern’). there are also differences in the capitalization of the names knut pipping (a person) and åbo akademi (a university), although these are not relevant for this study due to the case folding performed later in the process. original health-related quality of life and functioning in patients with spinal fusion simple health-related quality of life and functioning in patients with spinal fusion snowball health-rel qualiti of life and function in patient with spinal fusion spacy health – relate quality of life and functioning in patient with spinal fusion simplemma health-relate quality of life and function in patient with spinal fusion udpipe health – related quality of life and functioning in patients with spinal fusion stanza health -related quality of life and function in patient with spinal fusion lmc health-relate quality of life and function in patient with spinal fusion lmc_yso health-relate quality of life [quality_of_life] and function in patient with spinal fusion table 4. original english input text and the output of each preprocessing method. the english input text in table 4 is the title of a doctoral dissertation in the field of medicine. there are differences in how the different methods handle the compound expression health-related, as well as in the capitalization; the original text was given in title case and many methods have preserved at least some of the uppercased words. again, the lmc_yso method has matched the expression quality of life to the yso concept with the same label. model training and evaluation we used the data version control (dvc) tool to orchestrate the annif experiments. all the corpus and vocabulary preparation, training and evaluation steps were specified in a single dvc.yaml configuration file that eventually grew to 979 lines. the dvc repro command was used to run the necessary annif commands. results of the evaluation runs were written by the annif tool into json files. finally, a jupyter notebook was used to collect the results from the json files and, with the help of the pandas data analysis library, format them into tables that were then copied into the results section of this paper. the dvc automation made it easy to ensure that all experiment runs were performed consistently and correctly. all experiments (both model training and evaluation) were repeated five times and the results of the evaluations averaged in order to minimize the effect of random fluctuations. algorithms for text classification we chose to include two algorithms supported by annif in the comparison, in a total of three different configurations: the mllm algorithm (maui-like lexical matching) originally implemented in annif is a lexical automated subject indexing algorithm, inspired by the maui tool (medelyan 2009). it relies on matching terms (concept labels) in the indexing vocabulary to terms (words and combinations of words) in the document text, so it is important that the normalized terms from the vocabulary match the normalized terms from the text. without text normalization, the algorithm doesn’t perform well, as it then cannot match e.g. singular terms that appear in document text to plural form terms in the vocabulary or vice versa. omikuji is a reimplementation of a family of efficient tree-based machine learning algorithms for multi-label classification, including parabel (prabhu et al. 2018) and bonsai (khandagale, xiao, and babbar 2020). when used with annif, it can be configured with settings that constrain the size and shape of the decision tree in a way that emulates parabel or bonsai, with parabel being the default configuration. both the parabel and bonsai variations were used in these experiments, as they have slightly different strengths. these algorithms need to be trained on a large number of manually indexed samples and they learn to recognize correlations between words (or word n-grams, i.e. combinations of consecutive words) that appear in training documents and subjects or classes those documents have been manually assigned, without looking at the terms in the vocabulary at all. mllm parabel bonsai backend=mllm backend=omikuji transform=limit(5000) backend=omikuji transform=limit(5000) cluster_balanced=false cluster_k=100 max_depth=3 ngram=2 min_df=2 table 5. configuration settings for the parabel and bonsai backends used for both yso and ykl tasks. the settings language, vocab and analyzer have been omitted as they vary depending on the task setup. the settings of the annif projects used for the three types of algorithm are shown in table 5. the mllm configuration relies entirely on default values. the parabel configuration is also very simple and mostly relies on default values, except for the transform setting which truncates long documents to the given length (5000 characters), which we have previously found to improve results. the bonsai configuration is more advanced: in addition to the transform setting for truncating documents, the cluster_balanced, cluster_k and max_depth settings constrain the shape of the decision tree to emulate bonsai; the ngram=2 setting causes bigrams (combinations of two consecutive words/tokens) to be used as features in addition to unigrams (single words); and the min_df=2 setting requires that n-grams (unigrams or bigrams) must appear in at least two training documents in order to be included as features, thus excluding very rare words and expressions that could add noise and make the model unnecessarily large. these settings represent a simplified version of the models that are either currently used in production in the finto ai service (mllm, bonsai) or have been used in the past (parabel). yso subject indexing task for the yso subject indexing task in all three languages, we used the mllm lexical algorithm as well as both the parabel and bonsai variations of the omikuji algorithm, with the settings shown in table 5. the mllm models were trained on the training documents combined from both the jyu-theses and bookdesc corpora. the parabel and bonsai models were trained on the finna corpus. we did not use the lmc_yso analyzer with mllm, as the mllm algorithm itself identifies yso concepts within the document text and the additional matching performed by lmc_yso most likely would have interfered with that mechanism. all three types of yso subject indexing models were evaluated using a combination of test set documents from both the jyu-theses and bookdesc corpora: 764+1311=2075 documents in finnish, 32+560=592 documents in swedish, and 300+447=747 documents in english. we chose two metrics for the evaluation: f1@5 (f1 score – harmonic mean of precision and recall – calculated by comparing the top 5 suggestions of the algorithm to the manually assigned subjects) and ndcg (normalized discounted cumulative gain), a ranking measure where a higher score indicates that the manually assigned, “correct” subjects are closer to the top of the ranked list of suggestions given by the algorithm. ykl classification task for the ykl classification task in finnish, we used the parabel and bonsai variations of the omikuji algorithm, with the settings shown in table 5; mllm was not used for this task, as it is not very well suitable for single-class classification. both models were trained on the train set consisting of 51187 book descriptions manually classified using ykl. the models were evaluated on the test set of 4237 similarly classified book descriptions. we chose two metrics for the evaluation: accuracy, which is the proportion of documents for which the top suggestion of the algorithm matches the manually assigned class (equivalent to precision@1 in this setting where each document is assigned exactly one class), and the same ndcg ranking measure that was also used for the yso subject indexing task. results yso subject indexing in finnish mllm algorithm parabel algorithm bonsai algorithm f1@5 ndcg f1@5 ndcg f1@5 ndcg simple 0.221 0.290 0.415 0.563 0.454 0.615 snowball 0.269 0.364 0.424 0.575 0.471 0.636 voikko 0.305 0.413 0.423 0.577 0.472 0.639 simplemma 0.302 0.411 0.418 0.567 0.465 0.630 udpipe 0.287 0.389 0.407 0.555 0.450 0.609 stanza 0.301 0.412 0.397 0.542 0.466 0.631 lmc 0.313 0.420 0.426 0.579 0.470 0.636 lmc_yso – – 0.427 0.582 0.473 0.638 table 6. evaluation results for finnish yso subject indexing using different analyzers. the results of the finnish language yso subject indexing task are shown in table 6. the differences between analyzers are most apparent with the mllm lexical algorithm, which performs very poorly with the baseline methods (simple and snowball). the lmc method achieved the highest scores with mllm. for the parabel and bonsai algorithms, differences between analyzers were smaller; the lmc_yso, lmc and voikko analyzers as well as the baseline snowball analyzer all achieved high scores. the results for udpipe and stanza were in many cases worse than those of the baseline methods. yso subject indexing in swedish mllm algorithm parabel algorithm bonsai algorithm f1@5 ndcg f1@5 ndcg f1@5 ndcg simple 0.171 0.242 0.409 0.564 0.454 0.623 snowball 0.208 0.300 0.425 0.583 0.464 0.633 simplemma 0.207 0.297 0.423 0.579 0.458 0.625 udpipe 0.204 0.292 0.408 0.565 0.438 0.601 stanza 0.211 0.307 0.420 0.577 0.454 0.620 lmc 0.217 0.310 0.425 0.585 0.464 0.629 lmc_yso – – 0.428 0.591 0.462 0.631 table 7. evaluation results for swedish yso subject indexing using different analyzers. the results of the swedish language yso subject indexing task are shown in table 7. as with finnish, the lexical mllm method appears to be most sensitive to the choice of analyzer, with the simple baseline method performing poorly. the lmc and lmc_yso analyzers performed best, but also the snowball analyzer gave surprisingly good results. yso subject indexing in english mllm algorithm parabel algorithm bonsai algorithm f1@5 ndcg f1@5 ndcg f1@5 ndcg simple 0.235 0.321 0.417 0.571 0.500 0.678 snowball 0.249 0.343 0.396 0.550 0.492 0.670 spacy 0.241 0.334 0.400 0.553 0.495 0.675 spacy_lower 0.242 0.337 0.405 0.555 0.496 0.675 simplemma 0.243 0.336 0.407 0.559 0.493 0.674 udpipe 0.235 0.325 0.397 0.546 0.476 0.656 stanza 0.244 0.337 0.398 0.550 0.485 0.665 lmc 0.247 0.340 0.401 0.556 0.486 0.667 lmc_yso – – 0.415 0.574 0.488 0.671 table 8. evaluation results for english yso subject indexing using different analyzers. the results of the english language yso subject indexing task are shown in table 8. also in this case, the mllm lexical algorithm doesn’t perform well with the simple analyzer, but works surprisingly well with the snowball analyzer. for the parabel and bonsai algorithms, the simple analyzer gives the best results. the lmc and lmc_yso analyzers are not far behind. in general, it appears that for the english language subject indexing task, the more advanced analyzers for the most part do not produce results that would be better than those of one of the baseline methods. ykl automated classification in finnish parabel algorithm bonsai algorithm accuracy ndcg accuracy ndcg simple 0.622 0.725 0.612 0.724 snowball 0.631 0.738 0.627 0.743 voikko 0.637 0.747 0.632 0.750 simplemma 0.638 0.748 0.632 0.750 udpipe 0.631 0.742 0.629 0.746 stanza 0.634 0.752 0.639 0.763 lmc 0.637 0.747 0.636 0.753 lmc_yso 0.639 0.748 0.636 0.753 tnpp 0.629 0.750 0.639 0.763 table 9. evaluation results for finnish ykl classification using different analyzers. the results of the finnish language ykl classification task are shown in table 9. the best results were obtained with the lmc_yso, tnpp and stanza analyzers. as with the finnish language subject indexing task, the baseline analyzers (simple and snowball) performed worst. the differences between different lemmatizer methods were quite small, with the possible exception of udpipe, which was not much better than the snowball analyzer. computational cost we did not specifically measure the performance of each method in terms of computing resources because the methods were run in different computing environments and we didn’t put much effort into optimizing them. however, it was apparent that some methods were much more resource-intensive than others. we can roughly divide the methods into three categories based on resource usage: the lightweight methods include the baseline methods simple and snowball as well as the lemmatizers voikko and simplemma. the lmc analyzer (and its lmc_yso variant) is also quite fast, with the full set of corpora lemmatized in approximately two hours on a desktop pc. with the lightweight preprocessing methods, we can expect the execution time of typical annif training and evaluation runs to be dominated by other processing steps including text vectorization, model training and prediction. the methods with moderate resource requirements include spacy (along with the spacy_lower variant) and udpipe. the spacy analyzer is an order of magnitude more resource-intensive than the lightweight methods, adding several hours to the runtime of the english language training and evaluation runs where it was used. likewise, udpipe was much slower than the lightweight methods, with a full lemmatization of the corpora taking more than an hour on a high-end server with 48 cpu cores used in parallel. the most resource-intensive methods were stanza and tnpp. the stanza lemmatization of the full set of corpora took several hours with 48 cpu cores used in parallel, while the tnpp analysis of just the small ykl corpus (46 mb of raw text) took six hours on the google colab platform using gpu computing. in practice, these two methods are unlikely to be suitable for real time usage due to their high computational cost. conclusion in this comparison of text normalization methods, we expected that more advanced lemmatization methods would result in better downstream quality for text classification tasks compared to simple baseline methods (only case folding and simple rule-based stemming). by and large, this was the result we got, with some notable exceptions: in particular, the baseline methods simple and snowball performed well for the english language subject indexing task whereas even the more advanced lemmatizers performed relatively poorly, rarely scoring above the baseline methods. this may be due to the more analytic nature of the english language, which implies that there are fewer inflected word forms than in a synthetic, heavily inflected language such as finnish. the clearest benefit of lemmatization over baseline methods was seen for the finnish language, with swedish falling somewhere in between finnish and english. the lexical algorithm mllm was clearly more sensitive to the choice of text normalization than the omikuji variants, which was expected, because mllm needs to be able to accurately match terms in the vocabulary to terms that appear in the text. the omikuji algorithms, in contrast, are associative and only concerned with words in the text, so the particular text normalization method makes less of a difference. the differences between lemmatization methods were relatively small, although lmc and lmc_yso usually ranked at or near the top and udpipe did not perform as well as the other methods. since the relative performance of different analyzers appears to be different for each task, algorithm and language, it is difficult to predict in advance which method will work best in a particular situation. the best method has to be chosen experimentally, by training and evaluating models using both baseline methods and more advanced lemmatization methods. when using the annif toolkit, this can be performed by adjusting the analyzer setting of a project; in custom nlp pipelines, the normalization method can be changed by changing the code for text preprocessing. the study was limited to two types of tasks, corpora from three source systems, nine text normalization methods, three text classification algorithms and three languages. it is by no means a comprehensive comparison of text normalization methods, although even with these options the number of individual evaluation runs was 87 due to combinatorial explosion. this figure was further multiplied by the five repetitions we performed for the purpose of smoothing out random variation, so the full runs took several days on a server with 48 cpu cores. a larger comparison would have required access to training and evaluation corpora in other languages, more normalization methods to test or other possible algorithmic variations. among lemmatizers, we excluded the uralicnlp toolkit, which we didn’t have time to integrate and whose cc by-nc-nd license could be problematic; lemmy, which could have been used for swedish but usage was not straightforward as all the working examples were for the danish language; as well as some advanced nlp toolkits that would probably have been extremely resource-heavy, e.g. udpipe version 2 and sparknlp. a core reason for setting up this study was to determine the feasibility of adding additional analyzer modules to the annif toolkit. of the lemmatizer methods that were not yet implemented in annif, we determined the simplemma method to be the most promising, as it is very simple and fast, it supports a large number of languages (currently 38), and the empirical results in this study were rather good even though it was never the top performer for any particular task. we have already implemented an analyzer module based on the simplemma library and it was included in the annif 0.58 release. other analyzer integrations may follow later. references khandagale s, xiao h, and babbar r. 2020. bonsai: diverse and shallow trees for extreme multi-label classification. machine learning [internet]. [cited 2022 june 22]. 109 (11): 2099–2119. available from: https://doi.org/10.1007/s10994-020-05888-2. medelyan o. 2009. human-competitive automatic topic indexing [dissertation], the university of waikato. available from: https://researchcommons.waikato.ac.nz/handle/10289/3513. prabhu, y, kag a, harsola s, agrawal r, and varma m. 2018. parabel: partitioned label trees for extreme classification with application to dynamic search advertising. in proceedings of the 2018 world wide web conference, 993–1002. www ’18. lyon, france. available from: https://doi.org/10.1145/3178876.3185998. suominen, o. 2019. annif: diy automated subject indexing using multiple algorithms. liber quarterly: the journal of the association of european research libraries [internet]. [cited 2022 june 22]. 29 (1), 1–25. available from: https://doi.org/10.18352/lq.10285 suominen o, inkinen j, lehtinen m. 2022. annif and finto ai: developing and implementing automated subject indexing. jlis.it [internet]. [cited 2022 june 22]. 13 (1), 265–282. available from: https://doi.org/10.4403/jlis.it-12740 uhlmann s, grote c. 2021. automatic subject indexing with annif at the german national library (dnb) [presentation slides]. [cited 2022 june 22]. swib21 online conference, december 2021. available from: https://swib.org/swib21/slides/03-02-uhlmann.pdf koskenniemi k. 1983. two-level morphology: a general computational model for word-form recognition and production [dissertation]. (publications; no. 11). university of helsinki. department of general linguistics. available from: http://hdl.handle.net/10138/305218 karlsson f. 1990. constraint grammar as a framework for parsing running text. in: karlgren h, editor, proceedings of the 13th international conference on computational linguistics, volume 3, pp. 168-173, helsinki, finland. available from: https://doi.org/10.3115/991146.991176 about the authors osma suominen is working as an information systems specialist at the national library of finland. he is currently working on automated subject indexing, in particular the annif tool and the finto ai service, as well as the publishing of bibliographic data as linked data. he is also one of the creators of the finto.fi thesaurus and ontology service and is leading development of the skosmos vocabulary browser used in finto. osma suominen earned his doctoral degree at aalto university while doing research on semantic portals and quality of controlled vocabularies within the finnonto series of projects. ilkka koskenniemi is working at lingsoft as a technical architect and currently in charge of developing lingsoft’s text analysis and proofing tools and their cloud-based interface lmc. he has a background in computer science at the university of helsinki where he also worked briefly as a research assistant before starting full-time work at lingsoft. acknowledgements we thank filip ginter and jenna kanerva for assisting with tnpp technical issues, michael stormbom and sebastian andersson for providing background information about lmc, and juho inkinen for constructive comments. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – relevance and phrase searching in summon: looking under the hood mission editorial committee process and structure code4lib issue 21, 2013-07-15 relevance and phrase searching in summon: looking under the hood this article briefly examines the mechanisms behind seemingly counter-intuitive phrase search results in serials solutions’ discovery platform summon. the authors use the platform’s search api to explain why users sometimes encounter greater numbers of results when typically they would expect fewer. the article explores the reasons behind the search results and the implications for library instruction. by thomas hodge & james rw macdonald introduction if you’ve done much teaching you may have encountered one of those embarrassing moments when you attempt to demonstrate a concept and the demonstration produces unexpected results. when this happens the seasoned teacher may turn this failure into a learning opportunity. what went wrong? what might explain the unexpected results? when the answers seem to contradict the principle the teacher was trying to deliver, confusion rather than understanding may be the result. this paper examines the behavior of phrase searching in summon and how that may place librarians in one of these awkward teaching moments. the american university of sharjah library (aus) logged a call with serial solution’s help desk in february 2013. aus’ information literacy librarians had reported that summon searches containing phrases sometimes produced inexplicable results. for example: students bullying test scores – retrieved 676 results students bullying “test scores” – retrieved 1,361 results [1] the librarians expected that quoting the term “test scores” would reduce rather than increase the number of results. this is an expectation borne from past experience with databases, which is reflected in the way we have taught boolean searching to our students. the number of results in a search should decrease as the search becomes more specific. we can illustrate this with a familiar venn diagram, where items containing the phrase “test scores” and the terms “students” and “bullying” are a subset of the items containing all four terms: figure 1. when a more explicit search returns a larger result set librarians face an interesting instructional problem. the results are unexpected and the librarian may be hard pressed to explain why. many librarians teach their students that when they are using a phrase search properly they should notice a reduction in the number of search results. with summon this no longer appears to be true in all cases. using serial solutions’ search api to analyze search results, this brief paper explains why phrase searching in summon sometimes produces more results than a search for the same terms on their own. the approach to phrase searching serial solutions employs has implications for library instruction and more broadly on how to manipulate and interpret the results produced by a summon search. serials solutions’ response shortly after the support call to serial solutions we received this answer: … looking for the exact phrase, (in double quotes) will emphasize that phrase over “test scores” and will yield better results than a search for the same words without double quotes. note that even without the double quotes, verbatim word matches score higher than their non-verbatim matches. adding double quotes increases the emphasis on the verbatim matches, as well as treating those words as a phrase.[2] this response from the help desk was accurate and contained the answer we eventually arrived at. however, as it represents a change in how phrase searching has generally been taught, it simply did not make sense to the librarians examining the issue. the help desk also referenced answer 1068 from the support center which added some context: exact matches on words within a phrase search (with double quotes) get a higher relevancy boost over the same words in non-phrase searches (without double quotes). [new june 2012] [3] a follow-up call on march 6 produced more explication: i took a look at the later results of your examples (past page forty with fifty results per page) and by that point there seems little difference in the results for either search. i suspect the number of results then differs due to how the full text searching relevance is weighted. so for instance, it is very likely that those four terms would be found within a book, with each word hundreds of pages apart. in that case the result may not be returned at all since it would likely not be relevant. however, it may be more likely that “academic achievement” and one of the other terms are found on the same page, so then there would be more results. [4] this gave us a more concrete example to think about. when looking at the full text of an item, the fact that all of the words appear somewhere in the text does not mean it is a relevant result. we have probably all searched a full-text index in an ebook database and seen numerous results that are not actually relevant. serial solutions seems to have introduced a feature into their search to not only rank results by relevance but to retrieve by relevance, preventing not-really-relevant items from making it into the results set. we still needed a better understanding of the mechanism behind the results. during an email exchange on the summon-clients list (“counter-intuitive result counts for keyword/phrase searches”, 28 feb 2013), one participant noted that: with the advent of databases/services like google (and even summon), we have arrived, for better or worse, in an age where, try as we might, we really don’t know what’s going on. we still enjoy telling ourselves that we do, but we don’t. and it’s ok. [5] this is a very useful insight, especially considering the secrecy that vendors must exercise to protect the intellectual property in their relevancy algorithms. however, this should not keep us from utilizing the tools at our disposal to understand all we can about what is happening behind the scenes. at the very least information literacy librarians need to be able to teach how summon searching differs from searching in our webopac and other databases, how to evaluate the results, and what the quotes around a phrase will actually do for the searcher. analysis with the summon search api fortunately, summon offers a useful search api that lets us reproduce the results from the client and examine them more closely. to this end we modified the script we have been developing for the api to display the relevance score for each item, and to fetch the full item record and display it. applying an example from the above referenced email thread in our summon instance, we found: sheep dip flies retrieved 138 results “sheep dip” flies retrieved 155 results figure 2. we noticed in the search without quotes that the first item had a relevancy of 7.3483596 [figure 2], while in the search with quotes the top item had a relevancy score of 14.517519 [figure 3]. this suggests that quotes do indeed increase relevancy scores which is further confirmed by the fact that the item from the quoted search with the top score, (“firms withdraw sheep dip products”), appears in the search without quotes with a score of only 1.5343612. figure 3. looking at the end of the results set retrieved by the phrase search (“sheep dip” flies), we identified items from our collection like the australian oxford dictionary with a score of 0.06656703 [figure 4], and ashford’s dictionary of industrial chemicals with a score of 0.024393521. however, when we searched for: sheep dip flies and title: australian oxford dictionary we retrieved no results. manually examining the results from the search without quotes confirms that this title (australian oxford dictionary) does not appear. looking at the actual summon record we can see that the terms were (unsurprisingly) not present in the metadata, indicating that the phrase search result was probably coming from the full text. figure 4. items present at the end of a results set tended to be ones that could plausibly contain the search terms. they also tended to be from our local collection (a factor which also boosts the relevance score). it seems this combination is raising the relevance score of these items enough to cause their inclusion in the set, thereby increasing the total number of items. while items that don’t contain the words as a phrase fall out of the set, more items come in than fall out. observations / conclusions utilizing summon’s api we have demonstrated a plausible explanation for counter-intuitive results in phrase searching. we hope these findings will aid librarians as they teach this concept. as the algorithms controlling relevancy increase in complexity, librarians should do all they can to understand how those algorithms affect search results. our investigations have led to at least three observations: we are used to thinking of relevance scores as something applied to items in a search set after retrieval not to determine whether they should be included in a results set in the first place. serials solutions’ strategy of excluding items that technically match the search but are not actually very relevant is probably better for producing useful results, but it is also harder to explain to users. after examining the relevant sources and doing some testing, it seems that the quote operator in summon is being used for at least three things: specifying a set of terms as a phrase increasing the relevance of whatever is inside the quotes (including a single word) indicating that the exact term(s) in quotes is what is desired, rather than other forms of the word taken together, what we have is a “this is really important to me” operator. we found this interesting, as one of the information literacy librarians pointed out that this is exactly what their students seemed to think that quotes around terms did in a query. anecdotally, they have observed students consistently putting single terms in quotes while searching. this behavior seems to indicate that students do not have the same understanding of phrase searching as librarians. summon’s current use of the quote operator seems to favor the students’ concept. for example, we note that the search: (“sheep” flies) returns more results than simply (sheep flies). it is easy to conclude that a product is broken when we encounter an issue like the one described here. we hope that the analysis of this case will encourage librarians to consider alternate explanations and methods of analyzing which hypothesis is correct, and to focus on the relevant questions. here the issue is whether serials solutions’ approach to phrase searching is delivering better results. if it is, the fact that the number of results increases is simply a new reality that we must incorporate into our instruction strategies. notes [1] this search and the others used as examples were conducted with the filter option “exclude newspapers” set to true. phrase searches are less likely to produce the counter-intuitive effect of a larger results set without excluding newspapers. we believe this is a result of the large numbers of newspaper articles being searched. [2] aus call to serial solutions help desk : “summon operators” (02/21/2013), retrieved from http://support.serialssolutions.com/app/account/questions/detail/i_id/283944 (requires serials solutions support center login) [3] help center document – “summon: boolean, phrase, wildcard and proximity searching” (answer id: 1068, date created: 15 mar 2010, date updated: 03 jul 2012) retrieved from: http://support.serialssolutions.com/app/answers/detail/a_id/1068#phrase_searching (requires serials solutions support center login) [4] aus call to serial solutions help desk : “note from our information literacy librarian (03/06/2013), retrieved from http://support.serialssolutions.com/app/account/questions/detail/i_id/287442 (requires serials solutions support center login) [5] email thread – [summonclients] counter-intuitive result counts for keyword/phrase searches (02/28/2013). retrieved from http://mail.lists.summon.serialssolutions.com/sympa/arc/summonclients/2013-02/msg00062.html (requires list login) about the authors thomas hodge (thodge@aus.edu) is the assistant university librarian for technology and technical services at the american university of sharjah (aus) in the united arab emirates. thomas has been with aus since 2007. james rw macdonald (jmacdonald@aus.edu) is the web services librarian at the american university of sharjah (aus) in the united arab emirates. he has been with aus since 2012. james’ research interests currently focus on usability and the user experience in digital environments. subscribe to comments: for this article | for all articles one response to "relevance and phrase searching in summon: looking under the hood" please leave a response below: dave pattern, 2013-07-19 just a quick comment! summon uses lucene/solr and lucene “score” values exposed by the api aren’t comparable across different searches. to quote the lucene wiki: “score values are meaningful only for purposes of comparison between other documents for the exact same query and the exact same index” leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – a systematic approach to collecting student work mission editorial committee process and structure code4lib issue 43, 2019-02-14 a systematic approach to collecting student work digital technology has profoundly changed design education over the past couple of decades. the digital design process generates design solutions from many different angles and points of views, captured and expressed in many file formats and file types. in this environment of ubiquitous digital files, what are effective ways for a design school to capture a snapshot of the work created within their school, and to create a long-term collection of student files for purposes of research and promotion, and for preserving the history of the school? this paper describes the recent efforts of the harvard graduate school of design in creating a scalable and long-term data management solution for digital student work files. the first part describes the context and history of student work at the harvard graduate school of design. the second section of the paper focuses on the functionality of the tool we created, and lastly, the paper looks at the library’s current efforts for the long-term archiving of the collected student files in harvard’s digital repository. by janina mueller introduction as a result of a collaboration between the library, instructional technology, the it department and the software company instructure, the graduate school of design (gsd) at harvard university successfully designed and implemented a school-wide software tool that allows for the systematic collecting of student assignments from courses, embedded in harvard’s chosen enterprise learning management system, ‘canvas’ by instructure. for the first time in the history of the gsd, a school-wide system is now in place that allows the library to build an archival collection of digital student work. the collected student works and their metadata are used by various departments within the gsd for a range of activities such as exhibitions, publications, marketing of academic programs, the accreditation of academic programs and for archival purposes. this paper outlines the history of student work collecting at the gsd, and the processes that led to the successful implementation of the tool we call the “student work harvester” (swh). the goal of the new collection process was to develop a simple, intuitive and flexible tool that facilitates the selecting and gathering of student work, and allows for the creation of a centralized depository of student work that is searchable and archived long-term. context and history of student work at the frances loeb library when the harvard graduate school of design (gsd) was founded in 1936, it brought together harvard’s departments of architecture, landscape architecture and urban planning into one graduate school. the gsd’s frances loeb library has samples of student work going back to the 1920s. however, student work had been collected in a haphazard and uncoordinated way throughout the decades. thus, the collection of student work is inconsistent in content and format.[1] until 2013, the frances loeb library at the gsd had a passive role in the collecting of student work. academic departments that needed student work for various purposes, including for publications, accreditation and promotion, collected the files themselves. often times this meant that administrative staff and faculty had to contact students who graduated several years prior and ask for examples of their course work. since there was no or little coordination, students were often contacted multiple times by different people at the gsd. recognizing the redundancy caused by decentralized and uncoordinated processes, the library proposed to manage the process of collecting student work at the gsd. as harvard began rolling out a new learning management system (lms) in 2015, the library and the instructional technology department decided to explore the feasibility of using course websites as a way to gather student files, reasoning that collecting items and metadata as closely as possible to the source, and requiring no extra work on the students’ part, was the most efficient. by working with different stakeholders both inside and outside of the library, we worked with the software company instructure and designed a learning tools interoperability (lti) plug-in tool within the canvas lms.[2] our new tool allows us to collect all student work submitted as course assignments, or subset collections based on specific metadata criteria. instructure developed this customized, proprietary solution for us; regrettably, it is not open source and we are not able to share the tool. functionality of the student work harvester the student work harvester (swh) allows gsd instructors to identify student submissions that meet certain criteria and tag them so they can be made available as needed for inclusion in exhibitions, publications, accreditation or open house presentations. once work projects have been identified and tagged by instructors, canvas administrators are then able to search for student work that matches their search criteria and export it to a cloud-based storage location. figure 1. workflow diagram. the student work harvester has the following two interfaces: a tagging or selecting interface, and an export interface. the tagging interface is accessible only to teaching faculty and teaching fellows (tfs) of a course. in this interface, student files can be nominated by assigning tags to assignment documents. figure 2. student work harvester tagging and selecting interface. in the tagging interface, for any course, an instructor selects an assignment on the left-hand side of the screen from the assignments entered in canvas and displayed on the course website. this action populates to the right a list of students who have submitted at least one file against the assignment. the instructor then selects a student’s name. by clicking on the name, a table below populates with all the files the student submitted for that assignment. the instructor then checks a box according to which tags should be associated with the file. this is the way that student work is selected and tagged, for each course, for each assignment, for each student. the swh lists student file uploads that are submitted as “assignments” within canvas. to be specific, student submissions must be a “file upload” submissions type as other types (such as “text entry”, “website url” or “media recording”) are not recognized formats for the swh. the export interface is accessible to the library staff and a selection of gsd administrative staff who have ‘administrator’ roles in canvas. this interface allows administrators to search for student work and export it to a cloud-based storage system, which, in our instance, is an amazon s3. figure 3. export interface. the export view has a sequential selection process and works in the following manner: one or more terms can be selected in the term(s) field, after which all people enrolled as “teachers” in any of the courses from those terms are listed in the instructure name field. the export view allows you to select all or specific instructors, after which the course number field is populated. courses that the specified instructors taught in that term are listed. the student name and assignment name fields work in the same manner; the list of student names is determined by the selection made for the previous field, and either all or a selection of students can be selected. the same goes for the assignment name field. finally, the user of the export view has the option to export all files or only files marked with specific tags for specific purposes. when an export starts to run, the swh selects the indicated student files together with relevant student and course information from canvas. those files are placed in a temporary, cloud-based application platform, in our case heroku. once the processing is complete on heroku, the final export, in the form of a zipped folder, is placed on the amazon s3. when the processing is complete, an email notification with a link to that zipped file is sent to the person who ran the export in the student work harvester. figure 4. file hierarchy example. the top-level folder can be named by the exporter when running the export; its default name is the timestamp of when the export was created. within that folder are the following hierarchical folders: term name > course number > assignment name > student name. student work files are stored in the lowest folder (student name). the names of the files are the original filename from canvas. additionally, a json metadata file is generated and stored in each folder. the json file in the top-level folder contains a hierarchical representation of all data in the entire export. going one level down, json files within a subfolder contain metadata from the export scoped to that particular term [see figure 5]. the information for the metadata fields is pulled from the course information on canvas. figure 5. json metadata example. customization several parameters controlling swh’s functionality are customizable, including a) the tags that can be assigned to student files, and b) the server destination where the exports are placed. the tags are customizable in the export interface by administrators of the swh. the tags currently used were agreed upon by the academic departments and they reflect their needs for the use of student work. the tags label student work to be used for accreditation, in publications and the school’s outward-facing website. additionally, the final destination of the file exports, which currently is set to an aws s3 instance, is customizable. the export specifications are flexible enough to export to other endpoints. challenges even though we wanted our lti tool to be open source, we ultimately did not have a choice in this matter. because our lti tool is based on instructure’s canvas platform, working with instructure in creating this tool was the most efficient and cost-effective approach in the long run. canvas is not an open source platform, and the code for the swh was not shared with us either. this, in addition to the fact that an lti tool is inherently based on the functionality of the original platform (in this case canvas) can pose limits the tool. for example, the swh is based on roles and permissions of the gsd’s canvas instance; these, in turn, are in tune with other systems that are used at harvard, including the student information system. changes that are made to any of those systems can affect the functionality of the swh. many times we are not informed about changes that could affect the swh and accidentally find out about broken features of the tool. due to the fact that the swh is not open source, debugging is dependent on developers and scheduling priorities from instructure. additionally, because the swh is a customized lti tool, reporting a bug to instructure often requires us to repeatedly explain the swh’s functionality and purpose to instructure’s employees. this probably has to do with the fact that the swh is not a core functionality of canvas and consequently, customer support staff and developers at instructure are unfamiliar with it. implementation: the need for cultural change the implementation of the swh in the gsd’s canvas instance took place at the beginning of december 2015. getting teaching faculty to use a new tool, including a new lms, can be very challenging.[3] people get used to a system and its interfaces, and are often not interested in learning a new tool. faculty members generally care much less about the technical platform of their course websites than the content they teach. in other words, their priorities do not lie in spending the time to learn a new system. in order to minimize some of the frustrations that come with the implementation of new courseware and a new lti tool, we worked with the administrative staff of the departments to encourage and support the use of the swh. we also presented the tool in faculty and departmental meetings and offered faculty individual assistance if needed. one of the biggest takeaways from the implementation process of both canvas and the swh is that getting users to accept a new workflow and system takes time; faculty needed time to learn and understand the new platform, as well as gain an understanding of our intentions behind them using the swh tool. today, canvas is an integral part of any gsd course, and faculty are aware that part of their responsibility is to nominate student work at the end of their courses by means of the swh. additionally, the benefit of the swh for administrative staff and publication editors has led them to promote the tool to faculty as well. other major advantages one of the biggest advantages of collecting student work through the swh is that we can run very large exports, well above 100gb. this is a major advantage as it allows us to export not only a selection of files but potentially all files submitted to a canvas course. in our current environment of ubiquitous digital files – particularly pronounced in fields such as architecture, landscape architecture and urban planning – student assignment submissions are often made up of a number of files in different formats and sizes, and image files can be very large! our canvas assignment batch export mechanism accommodates these file size and format conditions. by using existing permission roles within canvas, we were able to have access controls to both the tagging and export interfaces. this was important, as we wanted to make sure that only teaching faculty and fellows – and not the students themselves – were able to assign tags to student files and that only administrators can trigger exports of files. future: what lies ahead now that we have a robust collection platform and workflow in place, we plan to work with a digital archivist in developing a workflow for the long-term archiving of the student files in harvard’s digital repository. our goal is to record the history of the creative output of the gsd and to archive a selection of student work files for perpetuity. challenges we have encountered thus far include the fact that harvard’s current digital repository only accepts a limited number of file formats. as noted previously, one of the advantages of our canvas lti collection tool is that it can accept any file format. thus, we need to find a way to archive all of the relevant files in their various file formats. moreover, we need to work on mapping our metadata json file to existing metadata schemas that are used for files in the digital repository. connected yet separate from that issue are questions of access to the archived files and their metadata: when, who and how should access to student files be granted? in other words, in the future we will have to work on higher level issues such as the scope of the long-term archival collection, access and discovery policies and technologies, as well as technical issues of metadata crosswalks and file formats. conclusion and summary the development process for our customized lti plug-in tool followed a typical software development cycle; however, it was constrained by proprietary software. the effort for developing this tool was well worth it for the entire graduate school of design, as it has centralized workflows and thus eliminated redundancies. moreover, we have witnessed a change in the expectations and interest of students and faculty in the area of data management and archiving. ideally, in the future, we hope to do more enhancements to the swh, including adding additional content metadata that describes the geographic location and type of problem for each assignment. to sum up, for the first time in the history of the gsd, a school-wide system – the student work harvester – is now in place that allows the library to build an archival collection of digital student work. the collected student work and their metadata are used by various departments within the gsd for a range of activities such as exhibitions, publications, marketing of academic programs, the accreditation of academic programs, and for archival purposes. this paper outlined the history of student work collecting at the gsd, and the processes that led to the successful implementation of the shw tool within harvard’s enterprise learning management system, canvas. the goal of the new collection process was to develop a simple, intuitive and flexible tool that facilitates the gathering and selecting of student work, and allows for the creation of a centralized depository of student work that is searchable and archived long-term. so far, that goal has been realized. references [1] the gsd’s frances loeb library has a special collections that includes a collection of student work in both analog and digital formats. this collection contains files going back to before the founding of the gsd, when the three disciplines were housed in different buildings. the earliest student work examples are student prize-winning projects from the 1920s. the library also holds a representative sample of student assignments from the 1940s and 1960s. in the 1980s, the gsd started to publish yearly print publications with examples of work students produced at the school. the library holds copies of these publications, as well as some digital files of the student work displayed in them. [2] the stakeholders included: a) gsd’s computer resources group; b) harvard’s teaching, learning, technology (tlt)-group, which managed the university-wide canvas roll-out; and c) instructure, the software company owning canvas. [3] conrad, suzanna (2018). spinning communication to get people excited about technological change. code4lib journal, (41). https://journal.code4lib.org/articles/13641. about the author: janina mueller (mueller.janina@gmail.com) has been working at the harvard graduate school of design (gsd) since 2013. she worked as the design data librarian from 2013-2015, and has since then worked as the gis & data librarian. as the design data librarian, she spearheaded a new, digital collection of student work, which centralized a number of disparate collection and work processes. she describes the design and implementation of this project in this article. today, mueller works as the gis and data librarian at the gsd. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – open journal systems and dataverse integration– helping journals to upgrade data publication for reusable research mission editorial committee process and structure code4lib issue 30, 2015-10-15 open journal systems and dataverse integration– helping journals to upgrade data publication for reusable research this article describes the novel open source tools for open data publication in open access journal workflows. this comprises a plugin for open journal systems that supports a data submission, citation, review, and publication workflow; and an extension to the dataverse system that provides a standard deposit api. we describe the function and design of these tools, provide examples of their use, and summarize their initial reception. we conclude by discussing future plans and potential impact. by micah altman, eleni castro, mercè crosas, philip durbin, alex garnett, and jen whitney introduction academic libraries have been increasingly engaged in the open access movement and in stewarding content published as open access. increasingly, funders have taken note of this movement — and a slew of policies issued by federal agencies and by foundations have incentivised both open access publishing and data sharing. furthermore, such policies increasingly require access to research data.   thus, libraries increasingly have the opportunity to engage with the fields of open access, publishing, and open data. these are all complex areas and, although overlapping, to the authors’ knowledge at the time of writing, have not heretofore been integrated with a free and open source platform in a tangible and automated way. this project provides a framework and operational tool to integrate open access journal publishing, data publishing, and data preservation. with funding from the alfred p. sloan foundation, the dataverse project (king 2007; crosas, 2011, 2013) from harvard university collaborated with the public knowledge project (pkp) (macgregor, stranack & willinsky, 2014) from simon fraser university and stanford university on an open source project to enable open access journals to seamlessly manage the submission, review, and publication of data associated with published articles. the result has been the development of software that editors of journals using pkp’s open journal systems’ (ojs) (willinsky, 2005) can easily install and use to adopt appropriate data replication, citation, archiving and publication policies. this goal has been achieved — ojs now supports a complete data publishing workflow that is shipped as part of the standard ojs distribution and which automatically and seamlessly integrates with dataverse for long-term access to research data associated with published articles. the project resulted in two main outputs: a plugin for ojs that supports a data submission, citation, review, and publication workflow; and an extension to dataverse that provides a standard deposit api. these extensions have been integrated into development and maintenance processes of their respective open source packages. furthermore, these extensions are being used to support interoperability with other systems such as the center for open science’s open science framework (osf) and archivematica starting in the summer of 2015. extending dataverse with sword the dataverse project started in 2006 at harvard’s institute for quantitative social science (iqss) as an open­ source software application to share, preserve, cite, explore and analyze research data. this effort was the continuation of a substantial sequence of development efforts at iqss, harvard and beyond (altman, et al 1999; altman 2002). from its beginnings, dataverse (then called the ‘dataverse network’) has provided a robust infrastructure for data stewards to host and archive data while offering researchers an easy way to share and get credit for their data. there are now over ten dataverse repositories that share metadata with each other hosted in institutions around the world, which together serve more than 59,250 datasets with 282,627 data files (http://dataverse.org). these dataverse repositories are using the dataverse software in a variety of ways — from supporting existing large data archives, to building institutional subject specific repositories, to use as institutional repositories. one of these dataverse repositories is the harvard dataverse, which alone hosts more than 1,000 “dataverses” (containers of datasets) owned and managed by researchers, research groups, organizations, departments and journals. the harvard dataverse has so far served more than a million downloads of its datasets, allowing researchers around the world to reuse the data, discover new findings, and extend or verify previous work. while the dataverse project was started by the social sciences for the social sciences, with the recent version 4.0 release (https://dataverse.harvard.edu/) it has now expanded to benefit a wide range of disciplines and scientific domains (astronomy, life sciences, etc), leveraging its progress in the social science domain to define and enhance data publishing across many research communities. through this project, we have enabled dataverse to federate with other systems not only as a data provider but also as a data store. prior to this project, dataverse did not provide a standard api–all deposits were accomplished through gui interfaces or through batch processes executed on the dataverse server itself. implementing a data deposit api in dataverse enables the ojs system to federate with any instance of the dataverse software in a standard, seamless way. for interoperability, we selected the sword2 protocol (stevenson, 2009). sword (simple web-service offering repository deposit) is a protocol specification used by digital repositories to accept the deposit of content from multiple sources in different formats.  repositories such as arxiv, dspace, fedora, and microsoft zentity support sword. through sword, data files, documentation, supporting materials and its corresponding descriptive metadata can be deposited into the dataverse system. we implemented version 2.0 of the sword protocol, which was originally developed by a collaboration of jisc funded efforts and other non-profits. sword2 supports a full set of crud (create, read, update, delete) operations – and thus provides a very general way of interacting with dataverse. as a result, it has enabled other projects to more readily federate with dataverse, including the (sloan supported) open science framework and the ropensci package for r-language integration with dataverse. dataverse implements a full sword deposit interface. the core supported functions are: retrieve sword service document create a dataset with an atom (http://bitworking.org/projects/atom/rfc5023.html) entry dublin core terms (dc terms) qualified mapping – dataverse db element crosswalk list datasets in a dataverse add files to a dataset with a zip file display a dataset atom entry display a dataset statement delete a file by database id replacing metadata for a dataset delete a dataset determine if a dataverse has been published publish a dataverse publish a dataset sword is a restful api that is easily accessed through standard libraries or command line clients such as curl. for example, the following command will list all datasets in a dataverse: curl -u $api_token: \ https://$hostname/dvn/api/data-deposit/v1.1/swordv2/collection/dataverse/$dataverse_alias or, to create a dataset, using an atom entry: curl -u $api_token: \ --data-binary "@path/to/atom-entry-study.xml" \ -h "content-type: application/atom+xml" \ https://$hostname/dvn/api/data-deposit/v1.1/swordv2/collection/dataverse/$dataverse_alias example atom entry (xml) <?xml version="1.0"?> <entry xmlns="http://www.w3.org/2005/atom"       xmlns:dcterms="http://purl.org/dc/terms/">   <!-some embedded metadata -->   <dcterms:title>roasting at home</dcterms:title>   <dcterms:creator>peets, john</dcterms:creator>   <dcterms:creator affiliation="coffee bean state university">stumptown, jane</dcterms:creator>   <!-dataverse controlled vocabulary subject term -->   <dcterms:subject>chemistry</dcterms:subject>   <!-keywords -->   <dcterms:subject>coffee</dcterms:subject>   <dcterms:subject>beverage</dcterms:subject>   <dcterms:subject>caffeine</dcterms:subject>   <dcterms:description>considerations before you start roasting your own coffee at home.</dcterms:description>   <!-producer with financial or admin responsibility of the data -->   <dcterms:publisher>coffee bean state university</dcterms:publisher>   <dcterms:contributor type="funder">caffeineforall</dcterms:contributor>   <!-production date -->   <dcterms:date>2013-07-11</dcterms:date>   <!-kind of data -->   <dcterms:type>aggregate data</dcterms:type>   <!-list of sources of the data collection-->   <dcterms:source>stumptown, jane. 2011. home roasting. coffeemill press.</dcterms:source>   <!-related materials -->   <dcterms:relation>peets, john. 2010. roasting coffee at the coffee shop. coffeemill press</dcterms:relation>   <!-geographic coverage -->   <dcterms:coverage>united states</dcterms:coverage>   <dcterms:coverage>canada</dcterms:coverage>   <!-license and restrictions -->   <dcterms:license>none</dcterms:license>   <dcterms:rights>downloader will not use the materials in any way prohibited by applicable laws.</dcterms:rights>   <!-related publications -->   <dcterms:isreferencedby holdingsuri="http://dx.doi.org/10.1038/dvn333" agency="doi" idno="10.1038/dvn333">peets, j., &amp;amp; stumptown, j. (2013). roasting at home. new england journal of coffee, 3(1), 22-34.</dcterms:isreferencedby> </entry> during the project, sword support has been prototyped, beta-tested, released, and updated. we also contributed minor extensions to the sword protocol and codebase, which have been accepted by the sword maintainers.   this api is now fully integrated into the current dataverse release, and is a fully supported and maintained module in the dataverse codebase. we have also deployed it in production, and it is being actively used to integrate ojs journals with the harvard dataverse instance. sword does impose some practical limitations. the most significant is with metadata — the standard implementation of the sword library uses simple dublin core terms for metadata, which are considerably less rich than the data documentation initiative (ddi) schema used by dataverse.[1] a typical way of approaching this to include metadata in a packaging format, such as mets or mpeg didl. however the simplezip packaging format used by ojs and dataverse does not have strong support for embedded metadata. to bridge this gap, project developers worked with the sword community to extend the api to support the addition of attributes to dcterms:[2] this enables richer metadata to be used directly in the client-server transaction, and dataverse uses this as the foundation for a more detailed translation to the to ddi. notwithstanding, dataverse 4.0 supports additional apis that are often more convenient for direct access to and manipulation of individual data files. dataverse also supports other apis such as oai-pmh for harvesting of information; lockss manifests and crawler apis for preservation;  and a set of native json apis for fine search and fine-grained deposit and access. for detailed information on all dataverse apis see: http://guides.dataverse.org/en/latest/api/index.html. designing an ojs dataverse plugin for data publication since 2001, pkp has maintained and developed ojs with a team of developers, librarians and library staff at simon fraser university, stanford university, and elsewhere. ojs is an open source journal management and publishing system which provides an online platform for journal editors to accept submissions, perform double blind peer review, edit, publish, and disseminate journal articles. pkp makes ojs freely available with the “purpose of making open access publishing a viable option for more journals” (ojs website: https://pkp.sfu.ca/ojs/). more recently, pkp has also developed other open source tools for managing conferences (open conference systems (ocs)), managing book publications (open monograph press (omp)), and metadata harvesting (open harvester systems (ohs)). although ojs allows for the publication of supplementary files alongside articles, it previously had no special provision for the review, publication, or presentation of common research data formats. this integration facilitates data sharing and archiving, allowing authors to be able to deposit their research data at the same time as their article. this streamlines the data submission process and provides an indexed, permanent identifier for published data. in keeping with ojs’ focus on a robust scholarly workflow, dataverse integration allows for consistent editorial policies to be applied to the submission and exposure of supplemental data; for example, data may or may not be subject to peer review, and in some cases it may be desirable to publish and provide an identifier to a dataset even when its corresponding article is rejected. it also allows embargoes of content as appropriate. users are also able to configure whether they wish to make article data available immediately upon acceptance of an article by a journal editor or to wait until the article itself is published, enabling faster-moving scholarly workflows. the diagram below illustrates an information lifecycle workflow, which we envisioned for this project, and demonstrates what a generalized ideal automated and integrated journal and data publishing workflow would look like (castro, 2015, also see codata 2014 for more details on the data lifecycle). this type of workflow enables journal editors and reviewers the ability to seamlessly manage the submission, review, and publication of data associated with published articles, and streamlines the author’s article and corresponding data deposit process by automating deposit from ojs into dataverse at the time of journal publication or journal article acceptance. figure 1. lifecycle of an automated and integrated journal and data publishing workflow. how does it work? to make data sharing and archiving as easy as possible for authors, data files are deposited in conjunction with the journal’s article submission process, resulting in a permanent 2-way linking between an article and its corresponding data. the full supported workflow is illustrated below: figure 2. data submission, review and publication workflows. there are four major roles addressed by these workflows: the author submits their article and research data to the journal’s ojs article submission system. (note that the article and data do not have to be submitted at the same time. authors can also submit data at a later time, or they can just provide a persistent link with a data citation pointing to the repository that their data is currently in.) editors and/or peer reviewers review the article and data. if the article and corresponding research data are approved for publication, the authors’ research data and its corresponding metadata is automatically deposited from ojs into the dataverse through the api. no redundant information need be entered. a permanent identifier (doi) will be automatically included that allows the data to be cited and tracked. there will be a data citation included in the journal article page in ojs (and ideally within the reference section of the article) enabling readers of the article to quickly access the data. the dataverse stores the dataset metadata and files (including raw data, documentation, code, etc). there will also be a permanent publication citation link within the dataverse for researchers to access the article in ojs that corresponds to this research data. editors, authors, and reviewers interact with the system through extensions to the ojs web-based interface.  for example, editors can set the data deposit and citation policies for journals, configure data review workflows, and designate a preferred repository using the following configuration dialogs: figure 3a. editorial configuration dialog in ojs, part 1. figure 3b. editorial configuration dialog in ojs, part 2. authors submit data through ojs as part of its standard manuscript submission workflow: figure 4. author submission interface in ojs. after publication, published data can be revised and updated using the standard dataverse interfaces, as illustrated below: figure 5. sample data management interface in dataverse. development considerations development work was jointly undertaken by iqss and pkp. iqss development resources were mostly dedicated to implementing, then expanding and documenting, the dataverse api; ojs developers focused on the creation of the ojs dataverse plugin itself. both teams contributed project management expertise, consultation with their respective communities, and presentation / promotion opportunities. both teams also benefited indirectly from this development work through the expansion of generic swordv2 api functionality in both ojs and dataverse. in designing the dataverse plugin for ojs, care was taken to ensure that as many administrative functions as possible could be performed from a single interface of a user’s choosing (e.g. electing to release or restrict data in dataverse from the ojs side), and that any file or metadata updates made in the ojs interface after the time of publication would be properly reflected in dataverse. additionally, a schema was developed to maintain compatibility between dataverse’s native support for ddi metadata and ojs’ dublin core metadata; this schema is also now openly available and is sufficiently generic to be useful outside of the ojs/dataverse context. through the use of the ojs dataverse plugin, ojs is also capable of automatically “mirroring” data terms of use and other related access policies from a linked dataverse in its existing policy text blocks. all of this was done in order to permit users to remain in a single coherent interface and to make the integration between the two platforms as natural as possible. dissemination the ojs dataverse plugin that constitutes the main user interface component of this work is now integrated into the main ojs development branch.  as of ojs 2.4.4, released in may 2014, the dataverse integration functionality is distributed in the core set of ojs plugins so that it can be enabled with a few clicks by an administrator, provided that there is a corresponding dataverse to associate. there have been two full ojs point-release cycles (2.4.5 in september 2014 and 2.4.6 in march 2015) since then, and both have included minor workflow improvements (and other bug fixes) suggested by the dataverse and ojs user communities following on from the initial release. the ojs dataverse plugin is also compatible with ojs as far back as version 2.3.0, and with both the stable, widely used dataverse 3.6.x branch and the recently released dataverse 4.0. the ojs dataverse plugin will be maintained by pkp throughout future releases, providing support for dataverse api changes and other minor improvements. with respect to dataverse, all of the functionality is now incorporated in the main dataverse codebase, bundled (modularized) into the sword api support, and will be maintained in 4.0 and future releases access to the dataverse code is provided through github. the long term sustainability of software generally relies on adoption and use by organizations that contribute to development and maintenance. while there can be no guarantees, both dataverse and ojs have large user communities that are continuously expanding. the dataverse software is licensed under the apache license, version 2.0 (the “license”); and may not be used except in compliance with the license. furthermore, any journal (regardless of institutional affiliation) using the plugin has the option to deposit data to the harvard dataverse, which provides permanent access.[3] the harvard dataverse relies on multi-institutional replication and a permanent endowment to guarantee bit-level preservation. supporting information we created documentation and policy templates in support of the software tools developed through the project. this includes documentation for developers, such as api implementation and testing; for copy editors and reviewers that provide policies, guidelines, and detailed workflow help for areas such as citing data; for authors submitting data associated with an article; linking articles to existing data; preparation of data and documentation data archiving; data confidentiality; dealing with licensing and other intellectual property restrictions; and other data use restrictions. the materials for end-users are integrated into the built-in help and template systems for the ojs plugin and dataverse. documentation for developers is integrated through github. furthermore, the project web site (http://projects.iq.harvard.edu/ojs-dvn) provides links to all documentation, templates, and training material.   the project website is the single location to find all information and resources related to the project, which includes presentations, publications, documentation related to the open source code for the plugin and the api, documentation for users setting up and using the plugin in ojs (updated (whenever relevant) with each new release of ojs), a video tutorial of the plugin, and a blog highlighting project milestones and events. reception outreach was integrated into the project at its early stages. we originally targeted approximately 50 journals that expressed interest in adopting the plugin when we completed our first phase of development in winter 2014. these journals tested a beta version of the software and provided feedback on the plugin’s core functionality and on our proposed data publishing workflows. we directly reached out to over 566 individual journals within ojs through one-on-one in person conversations at conferences, phone/web conference calls and email correspondences. on top of the above mentioned number we were also approached by an aggregator of ojs journals that supports hundreds of journals all over latin america. throughout the project we have been dedicated to providing one-on-one training and support for all journals and publishers who were interested or were already trying out the plugin. this included email, phone, in-person, and virtual video conference calls depending on the type of assistance needed, as well as provisioning of test server instances. in the fall of 2014, we conducted online surveys of representatives from journals that had previously expressed interest in the tool. we received responses representing over 200 journals (each respondent represented a cluster of journals, typically those hosted by a particular association or collaborative). results from this survey (altman, et al 2015) indicated that 28% were currently evaluating the newly released tools, 29% were in active planning stages, and the remainder were in early planning stages.   respondents indicated almost universally that the tool was not difficult for editors and administrators to configure or for authors and readers to use. a small minority encountered difficulty with installation of the plugin which subsequently has been resolved through its inclusion in the main ojs distribution. however, the surveys did indicate that there was room to improve the interfaces as only a minority found the system “very easy” to use (on a likert scale). as a result of this feedback, we implemented a number of feature improvements and bug fixes to improve the interfaces. there remains substantial opportunity for user interface refinement — especially for the reviewer workflow — but most of this goes hand in hand with the user experience improvements due for ojs 3.0. moreover, a substantial majority (72%-95%) of journals responding agreed (or strongly agreed) that data sharing (75%), citation (95%), and replicability (72%) were important to the journal. (the responding journals likely reflected those with the most interest — based on a nonrespondent follow-up of 191 journals, a third to a half of all journals regard these issues as important.) journals almost universally agreed (99%) that supporting authors of articles with submission of their data was important to their journals, even in the absence of a data sharing policy; we now provide easy-to-implement policy templates within the plugin. future development preliminary work has also been done on porting the ojs dataverse plugin to open monograph press, which uses the same plugin architecture as the upcoming ojs 3.x branch. it is expected that pkp will be able to use this preliminary spec work to provide a version of the ojs dataverse plugin for ojs 3.x (and open monograph press) within a year of its planned release this autumn.  more generally, the codebase that supports integration with dataverse through the standard sword interface for data deposition offers the potential to integrate ojs with other repository and data management systems (such as dash, harvard’s institutional repository which is built on dspace, and archivematica) with substantially reduced effort. with respect to dataverse development, the sword functionality developed through this project also significantly facilitates integration with additional systems. for example, preliminary work has been completed in integrating dataverse with the open science framework using this standard. with the support of an additional award from sloan, dataverse is building upon this codebase to test integrations with other publishers such as sage publishing, plos, elsevier, nature, and f1000research.    expected future impact a number of direct impacts of this project should become visible over the next three to five years. based on preliminary estimates of a broader survey of open access journals we recently conducted (and are preparing for forthcoming submission), there are hundreds of ojs and doaj listed oa journals with some form of data replication policy. currently, there are over 8200 journals that are actively using ojs (defined as publishing more than 10 articles in a year for more than one consecutive year, as of the end of 2014) and have access to the plugin or will automatically have access following their next upgrade cycle.  we expect to continue to monitor measures of adoption, datasets deposited through the tool, and citations to data and articles published through journals using the plugin. in the medium term we expect these and other measures to show significant impact. the broadest long-term outcome of the project will be to increase the replicability and reuse of research. we expect many of these broader impacts to develop over the course of the next ten years. the automated support for data publication, sharing, and citation provided by these tools makes it substantially easier for journal editors to adopt effective data citation, sharing, and replication policies. these policies are becoming increasingly salient as major stakeholders have issued policies and mandates supporting open data. within the last year alone, a number of federal agencies (most recently dod), independent funders, and professional societies have issued policies requiring data sharing, with greater movement in this direction clearly on the horizon. endnotes [1] the recently released version of dataverse, 4.0 uses an internal schema, but still supports ddi for export and import. [2] these extensions, along with bug fixes to the sword php library were accepted into the community supported sword codebase. [3] see http://best-practices.dataverse.org/harvard-policies/ for harvard dataverse preservation terms. references altman, micah, leonid andreev, mark diggory, gary king, elizabeth kolster, a. sone, sidney verba, daniel kiskis, and michael krot. (2001). “overview of the virtual data center project and software.” in proceedings of the 1st acm/ieee-cs joint conference on digital libraries, pp. 203-204. acm. altman, micah. (2002). “open source software for libraries: from greenstone to the virtual data center and beyond.” iassist quarterly 25. altman, micah; castro, eleni; crosas, merce; durbin, philip; whitney, jen, 2015, “replication data for: open journal systems and dataverse integration– helping journals to upgrade data publication for reusable research”, http://dx.doi.org/10.7910/dvn/y3wooe, harvard dataverse, v1 castro, e. (2014). connecting journal articles with their underlying data: encouraging sustainable scholarship through the pkp-dataverse integration project. in: jankowska, m. (ed.), focus on educating for sustainability: toolkit for academic libraries (pp-141-149). sacramento, ca: library juice press. castro, e, & garnett, a. (2014). building a bridge between journal articles and research data: the pkp-dataverse integration project. international journal of digital curation [internet]. 9(1):176-184. castro, e. (2015). a bridge from publishing words to publishing data [poster]. force2015: research communications & e-scholarship conference, oxford university. http://datascience.iq.harvard.edu/presentations/bridge-publishing-words-publishing-data-poster codata. (2014) task group on data citation standards and practices, codata-icsti. (2013). “out of cite, out of mind: the current state of practice, policy, and technology for the citation of data.” data science journal 12.0: cidcr1-cidcr75. stevenson, a. (2009). sword2 project final report. http://www.webarchive.org.uk/wayback/archive/20140615083222/http://www.jisc.ac.uk/media/documents/programmes/reppres/sword2finalreport.pdf crosas, m. (2011). the dataverse network®: an open-source application for sharing, discovering and preserving data. d-lib magazine, 17(1/2) doi:10.1045/january2011-crosas crosas, m. (2013). a data sharing story. journal of escience librarianship, 1(3) doi:10.7191/jeslib.2012.1020 king, g. (2007). an introduction to the dataverse network as an infrastructure for data sharing. sociological methods and research, 36(2), 173-199. macgregor, j., stranack, k., & willinsky, j. (2014). the public knowledge project: open source tools for open access to scholarly communication. opening science (pp. 165-175) springer international publishing. willinsky, j. (2005). open journal systems: an example of open source software for journal management and publishing. library hi-tech, 23(4), 504-519. about the authors authors are listed in alphabetical order. we describe contributions to the paper using a standard taxonomy (allen et. al 2014). micah altman and eleni castro were the lead authors, taking responsibility for content and revisions. micah altman and merce crosas were responsible for the initial conceptualization of the work. eleni castro was primarily responsible for data collection. micah altman and eleni castro were responsible for data analysis. phil durbin, alex garnett, and jen whitney were primarily responsible for the software development described.  micah altman and eleni castro authored the first draft of the manuscript. all authors contributed to the writing through critical review and commentary. micah altman is the director of research at mit libraries (http://informatics.mit.edu). eleni castro is the research coordinator, iqss at harvard university (http://www.iq.harvard.edu/people/eleni-castro. mercè crosas is the director of data science, iqss at harvard university (http://scholar.harvard.edu/mercecrosas). philip durbin is the software developer,  iqss at harvard university (http://www.iq.harvard.edu/people/philip-durbin). alex garnett is the data curation & digital preservation specialist at simon fraser university (http://webambler.com/). jen whitney is the systems librarian at carleton college (https://ca.linkedin.com/in/jenwhitney). subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – using xslt and google scripts to streamline populating an institutional repository mission editorial committee process and structure code4lib issue 19, 2013-01-15 using xslt and google scripts to streamline populating an institutional repository the college of wooster has created a process that allows library staff to quickly populate institutional repositories. an xslt script is used to transform refworks citations into dublin core xml and batch load those records into the institutional repository. a second script in a google docs spreadsheet then looks up publisher permissions in sherpa/romeo. the resulting workflow has dramatically reduced the amount of time necessary to populate an institutional repository with faculty scholarly articles. by stephen x. flynn, catalina oyler, marsha miles the open access movement offers academic libraries an opportunity to play an integral role in promoting and preserving faculty scholarship. although institutional repositories (ir) are prevalent among academic libraries, one challenge identified in a 2012 confederation of open access repositories report is that populating repositories is staff intensive (coar, 2012). faculty are often recruited to submit their manuscripts to irs, but this has led to low participation rates at some institutions. (jihyun, 2011) the majority of staff work involves determining what articles faculty have published, and whether or not libraries may upload the publisher’s version of those articles to the ir. this article describes two scripts developed at oberlin college and the college of wooster. one script packages author affiliation database searches into a dublin core metadata batch load, and another identifies journal copyright policies in sherpa/romeo to help staff determine which journals permit the published pdf to be uploaded to a repository. we describe how we used both scripts at the college of wooster to streamline importing faculty article metadata and uploading the published version of a pdf into an ir. batch loading faculty article metadata at the college of wooster, we implemented a metadata batch load script developed in 2010 at oberlin college library. library staff manually entered metadata into their ir, oberlin shares (sciences, humanities, and arts: repository of expression and scholarship), from faculty curriculum vitae (cv), copying and pasting or re-keying information into an online form. this process was time intensive, and they looked to alternative sources for metadata on faculty publications, specifically reference management software, that was able to bulk ingest citations from journal databases. eventually, refworks was selected to gather their citations as it was already used on campus and, unlike other software considered, offered an easy xml export function. the data could then be transformed into dublin core using an xslt script and ingested by dspace. importing data into refworks involved running an author affiliation search in large index databases such as ebsco, scopus and web of knowledge, then exporting the results to refworks and using refwork’s deduping feature. figure 1. refworks’ duplicate detection feature made it possible to run author affiliation searches in multiple databases, export to refworks, and delete duplicates. we first generated a schema for the refworks xml and then created an xslt stylesheet to transform the xml into dublin core ready for import into dspace. once finished, these documents can be used to run the transformation multiple times to continue adding recently published articles to the database. when run, the xslt completes three tasks: dividing the single refworks file into multiple dublin core records, mapping refworks elements to their equivalent dublin core fields, and cleaning up the exported text. the division of the refworks export into multiple dublin core files is accomplished with a simple xsl:result-document command. to match the dspace upload format, these documents are all titled as dublin_core.xml and placed into sequentially numbered folders, also generated by the xslt. a blank contents folder is also generated and included in each number folder for the upload process as well. in most cases, the map from refworks xml to dublin core is clear with a one-to-one transformation for fields such as title, contributor, abstract, publisher, digital object identifier (doi) and others. the single field that involved multiple mappings is the citation in dublin core (dc.identifier.citation). the citation field brings together data from various refworks fields and creates a single well-formatted reference to the article despite the many variables in the metadata itself, such as number of authors and the possibility of missing fields. in addition to mapping fields, the field contents are also transformed in this process. the text from the refworks export varies in quality, but the xsl transformation corrects the most common errors. these corrections include changing titles, publishers, and journals from all capitals into title case, and replacing corrupted html character codes. further changes include ensuring authors are listed with their last name first, changing dates to numerical year-month-day format, translating given languages to their iso code, checking that dois are correctly formatted for linking, and removing stray characters that occasionally appear. in addition, fields that are consistent for all records are written by the stylesheet, such as dc.type which is set to “article” and the institution name. <!-author field --> <xsl:for-each select="a1"> <dcvalue element="contributor" qualifier="author"> <xsl:choose> <!-adds a space after the comma that seperates first and last names --> <xsl:when test="contains(.,',')"> <xsl:variable name="last" select="substring-before(.,',')"/> <xsl:variable name="first" select="substring-after(.,',')"/> <xsl:value-of select="concat($last,', ',$first)"/> </xsl:when> <xsl:otherwise> <xsl:apply-templates/> </xsl:otherwise> </xsl:choose> </dcvalue> </xsl:for-each> figure 2.the xslt script standardizes author metadata from a refworks xml export into last name, first name (with a space after the first comma). the text corrections catch the majority of errors in the exported records, but staff members still need to double check the imports. errors that have passed through the transformation include unexpected or new stray characters and corrupt html character codes, typos from the original records, and incorrectly associated information, such as an abstract from a different record. in our workflow, we loaded the xslt output into dspace before reviewing the data for errors, but this step should be completed at an earlier point in the process. while staff time is still required to run the script, load, and review records, this process removes the largest burden of work by automating the record creation. automating journal copyright policies in addition to obtaining metadata, there are copyright issues to consider when archiving faculty articles. the rights of authors to self-archive and post their work in an ir vary among publishers. some allow authors to post their published pdf to a repository, some only permit the posting of author pre-prints (the version of an article before peer review), while others restrict all author self-archiving. determining the copyright policy of every journal in which faculty have published would be time consuming. we created a script in google docs that searches sherpa/romeo, a database of publisher copyright policies, to determine whether or not a journal permits authors to upload published pdfs to irs. scripting journal copyright policies after using refworks and xslt to batch load the college of wooster faculty’s article metadata into our ir, we exported the resulting metadata into a google docs spreadsheet with columns for author, journal title, article title, and the journal’s issn. sherpa/romeo provides an api that allows developers to link journal copyright permissions into their software, by sending an http request to sherpa’s server which then returns xml data. we used google apps script, the scripting feature built into google docs, to write javascript functions that, when invoked in a spreadsheet cell, search journal copyright policies in sherpa using the issn numbers from our spreadsheet. figure 3.invoke the pubpdf() function and select the corresponding issn field. figure 4.copy the function down the list and pubpdf() will automatically check sherpa/romeo for whether the journal permits archiving the publisher version. the script performs the following (see appendix a for source code): inputs the journal’s issn. sends a get request to sherpa/romeo’s api with the issn embedded. retrieves xml from sherpa/romeo containing the journal’s policies. parses the xml looking for <pdfarchive>can</pdfarchive> which broadly indicates the journal’s willingness to permit uploading their published pdfs into a repository. returns a “yes” or “no” statement to the spreadsheet. a secondary function also searches for the word “embargo” anywhere in the xml, and returns a “yes” if found. this would be run in a separate column. with over 2700 articles written by college of wooster faculty discovered in our library databases, this script allows us to determine which article pdfs we have permission to upload to our ir at a far greater speed than manually searching journal copyright permissions. limitations in the script the script is not completely automated. our staff must double check every article marked “publisher’s version/pdf may be used”. for example, the american journal of physics permits authors to archive publisher pdfs on their personal website, but not on an e-print server or institutional repository. other journals have similar restrictions that are not universally keyed the same way in sherpa/romeo’s xml feed, and thus need to be double checked. in addition, google limits how often you can run the script. google’s urlfetchapp() function, which we used to fetch the xml from sherpa/romeo’s api, is limited to 20,000 calls per day. we encountered errors running the script on all 2700 articles at once. an ideal solution would be to cache multiple responses coming from the same journal. right now, if a spreadsheet contains 20 articles published in journal of chemical physics, the script will retrieve and parse the xml for each of those articles, twice over if we’re also checking for embargo language. using google’s scriptdb() function, we could cache a journal’s policies so that the lookup only occurs once, greatly increasing the speed. we haven’t yet developed this capability. figure 5.google script doesn’t permit running a call excessively in a short period of time. finally, google spreadsheet cells containing a function will recalculate if the browser is refreshed or the sheet resorted, which would increase the likelihood of exceeding the daily limit. the workaround we use is to copy the results into their own cells by copying and pasting the values under the paste special menu, or creating a “static” column to paste the values. implementing the script using student assistants after testing the script on a small list of faculty publications, we ran it on a complete list with the help of our student assistants. we sorted the google spreadsheet by issn, which groups multiple articles from the same journal together. due to some bad metadata on initial import, such as combining an issn and isbn number in the same field, we couldn’t determine the journal copyright policies of certain articles, and skipped over them with the goal of cleaning up metadata during a second phase of our project. attempting to run the script on more than approximately 100 articles at a time resulted in a “service invoked too many times” error message described above. our student worker completed the steps in appendix b in small batches. we created a second sheet in the spreadsheet to collect what we term “green” articles, or the articles that sherpa/romeo says can be archived (pre-print, post-print or publisher’s version/pdf). the student sorted the spreadsheet by the permission column and copied the “green” articles to the new sheet. as stated previously, we double-checked the permission language of “green” articles. also, language inconsistencies prevent the script from being accurate enough to catch all of the embargoed articles. because we have to check the articles manually on sherpa/romeo to determine if the article is in fact a “green” article, we saved time by not having the student copy and paste the embargo detection script. figure 6.we double check permissions on the sherpa/romeo website, since some journals attach conditions to the use of their published pdf, such as requiring that you link to the journal’s home page. figure 7.we insert copyright caveats into our spreadsheet. ideally, the script would automatically load the “general conditions” section from each sherpa/romeo record into the spreadsheet to eliminate having to manually check each one on the website. however, using the script as currently written still saves time. by ignoring journals that do not permit archiving published pdfs, we can focus on journals that do. after double checking the permission language, we add articles to a new sheet called “articles to find,” and the student searches for and downloads article pdfs from library databases. local pdf archiving file naming when article pdfs are downloaded, we give them a more meaningful file name. the college of wooster adapted the file naming structure used in oberlin college’s shares collection. the file name is built using the first-listed local author’s last name and a brief part of the article title. the volume number and issue number (or year if used by the journal in place of issue number) are included to prevent the possibility of any duplicates if the author published an article with a similar title. because we batch load article metadata in our repository independently as a first step, adding pdfs to dspace is as simple as attaching the corresponding publisher pdfs as bitstreams to the item records. for a second phase, we plan to fill in missing issns and fix other metadata issues, an inevitability when working with automated data. finally, in future years we will batch load that year’s faculty publications and check sherpa/romeo for pdfs we can upload. conclusions our goal at the college of wooster was to streamline the process of populating our ir. the refworks xslt script developed at oberlin college helped us quickly create dublin core metadata for our faculty’s articles to batch load into dspace, and our sherpa/romeo script helped us determine the articles published in journals that allow pdfs to be uploaded to an ir. despite being slightly cumbersome to use, it is a significant improvement over completing these tasks manually. our workflow is specific to some products, such as refworks 2.0, dspace 1.6 and google docs, but this process could be altered to work with other systems. one could first perform an author affiliation search in web of knowledge, upload the resulting spreadsheet to google docs, identify your “green articles” using the sherpa script and then batch load those pdfs directly into a repository, with metadata attached. an upgrade to dspace 3.0 would allow the ability to batch import metadata from common bibliographic formats including endnote, ris and .csv files directly into a repository. uploading our faculty’s published pdfs to our ir is one step we are taking to promote open access to faculty scholarship. we will eventually recruit author pre-prints from the faculty, and our sherpa/romeo script can easily be retooled to look for faculty articles in journals where pre-print and post-print archiving is permitted. using this information, libraries could focus their recruitment efforts on faculty with large numbers of pre-print eligible articles. we haven’t yet formally assessed the impact of our fledgling ir on faculty attitudes towards the role of libraries in scholarly communications, but we look forward to reading comments and suggestions from faculty, librarians and technologists who might use our two scripts to streamline populating an ir. references “preliminary report – sustainable best practices for populating repositories” confederation of open access repositories. 2012 [http://www.coar-repositories.org/working-groups/repository-content/preliminary-report-sustainable-best-practices-for-populating-repositories/] jihyun, kim. “motivations of faculty self-archiving in institutional repositories” the journal of academic librarianship. 37:3. 2011 “batch import from basic bibliographic formats (endnote, bibtex, ris, tsv, csv)” duraspace issue tracker. [https://jira.duraspace.org/browse/ds-1226] appendix a – source code refworks xml to dublin core transformation script download the .zip file which contains detailed pdf instructions and the original source code: https://sites.google.com/site/drmcbackup/documentation/batch-submission-from-refworks sherpa/romeo script source code copy the code here – http://pastebin.com/sxknbhdq in google spreadsheets, go to tools -> script editor, and paste the entire script, replacing the myfunction() placeholder. if you plan on using the script in a project, you will need to register an api key with sherpa/romeo (link included in the source) and insert it into the script as indicated, or else your xml requests may time out. if you experiment with a small number of records, an api key won’t be necessary. appendix b – ir script instructions provided to students in addition to the metadata fields imported from refworks, we added “pubpdf” and “embargo” columns to run the pubpdf() and embargo() functions, and columns next to each where we paste the static values to prevent the script from running excessively. phase i 1. open the ir_collection spreadsheet on google drive 2. select cell a2 and type: =pubpdf( 3. click on cell e2 (issn column) and hit return 4. copy cell a2 down approximately 100 rows 5. copy the results and select cell c2 in the “pdf static” column 6. go to the edit menu and select paste special ? paste values only 7. delete the results from the “pubpdf” column 8. continue steps 2-7 for the rest of the spreadsheet 9. sort by the “pdf static” column a. add the green rows to the “green articles” sheet 10. sort by the “journal” column 11. double check the permission language a. update the “pdf static” column if needed 12. search for pdfs for articles on the “green articles” sheet a. rename the files i. lastname_partoftitle_vol#_iss#.pdf b. add the file name to the “green articles” sheet 13. add pdfs to the corresponding metadata in the ir collection phase ii 1. find issns for articles on the spreadsheet that were missing issns 2. fill in other missing information (journal titles, etc.) 3. complete steps 2-9 above for these articles about the authors stephen x. flynn (http://www.sxflynn.net) is emerging technologies librarian at the college of wooster in wooster, ohio. he also created opencoverletters.com, a repository of successful cover letters for library jobs. catalina oyler is a technical support specialist for orcid (open researcher and contributor id). previously she worked as digital initiatives coordinator for the five colleges of ohio. marsha miles is the digital initiatives librarian at cleveland state university in cleveland, ohio. previously, she was the visual resources curator and digital services associate at the college of wooster in wooster, ohio. subscribe to comments: for this article | for all articles 5 responses to "using xslt and google scripts to streamline populating an institutional repository" please leave a response below: tom at iso15926, 2013-11-13 excellent, i wasn’t aware of the sherpa api – exactly what i need. thanks for such a thorough write up. lilly, 2014-06-26 hi there, i am very interested in this article. the link to the code, specifically, “refworks xml to dublin core transformation script” doesn’t work anymore, could you please email me the code? thank you very much, lilly stephen flynn, 2014-07-21 a backup of the file can be found here https://sites.google.com/site/drmcbackup/documentation/batch-submission-from-refworks thanks for bringing this to my attention. steve johnson, 2014-11-18 the backup site is available, but google returns the same 415 error when one attempts to download the scripts. thank you, steve johnson stephen flynn, 2015-02-23 i apologize for the links being down. here is a mirror copy – https://archive.org/details/refworks2dc leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – extracting user interaction information from the transaction logs of a faceted navigation opac mission editorial committee process and structure code4lib issue 7, 2009-06-26 extracting user interaction information from the transaction logs of a faceted navigation opac this paper discusses the analysis of apache web server logs from a faceted catalog interface (opac) at north carolina state university. by grouping individual http requests into user sessions and analyzing in that context, requests can be understood as particular user actions, with more specificity as to purpose and effect of an action. client ip address and time are used as a sufficient proxy for determining user sessions from logs. some initial exploratory findings of user behavior in the ncsu opac are provided, including that users make use of facets less than of text searching, and that some facet groups are used significantly more than others. links are provided to the scripts used to make this session-based analysis, which could be modified for use with other facetted opacs which use an apache front-end. by cory lown and brad hemminger introduction the purpose of this paper is to demonstrate a method for harvesting, storing, and analyzing data from the transaction logs of modern, faceted, search and browse online public access catalogs (opacs). faceted navigation opacs, such as the ones found at north carolina state university (ncsu) [1] and triangle research libraries network (trln) [2], provide users with the ability to explore library collections by text searching and facet selection. by now, faceted opac interfaces are common, found everywhere from public libraries to research libraries. the interfaces of such catalogs provide a significant advancement for the opac, giving users access to better tools for item discovery. there is also, among libraries, an increasing need to understand how the tools it provides are being utilized. this need is driven by increasing emphasis on optimizing the user experience and demonstrating to library stakeholders how the library’s resources are being used. the method described in this paper is flexible enough that the output can be used for a one-off analysis of particular statistics of interest, stored and queried as a research database, or used to feed a larger data store for library statistics, one that might be used to store and report many other statistics about the library and its operations. the endeca software that powers ncsu’s faceted opac includes some capacity for reporting usage. the reports we examined indicated that users were utilizing facets as well as text searching. the reports also indicated the most popular facets, counts of text-search only requests, facet selection-only requests, and requests that combine navigation techniques, and other statistics. however, the reporting functionality built into endeca had no way to track sessions. as a consequence, it was impossible to know such things as how long people were spending interacting with the site, how iterative the search process was, and whether certain features of the site were more utilized at the beginning or the end of the search process. it was also a black box, meaning we had no good way of determining the accuracy of the statistics reported, and we did not know whether robot activity on the site might be affecting the reports. we were interested, broadly, in trying to understand how people were making use of text searching and facet selection in combination as a searching technique. we were also interested in whether certain search strategies might be more suitable to particular kinds of tasks. answering such questions would require direct user research. as a first, exploratory step in the research, and as a means to verify and enhance the information available to us in the endeca reports, we set out to process the server logs to generate our own reports on usage of the catalog. the techniques described in this paper rely on server transaction logs, which are a record of requests received by a server. the format of transaction logs is relatively standard, though logging may be configured somewhat differently from server to server. each line of text in the transaction log represents an individual request. these individual requests can be grouped into sessions, series of requests initiated from the same ip address. once the requests in the transaction logs are grouped into sessions, the individual requests that make up each session can be analyzed as a sequence of actions performed by the user. the change in parameters applied in the request url from one request to another within a session enables us to infer the specific action taken by the user at each step in her interaction with the opac. when this method of analysis is carried out programmatically on a large number of requests one can learn a great deal about how users interact with faceted navigation opacs. identifying sessions, and in turn, coding each request as one of a finite set of actions the user can take on the site opens up many possibilities for tracking ways that visitors make use of the opac. some of the statistics that it is possible to generate from applying this method of log analysis are: number of requests number of sessions sessions by known ip address ranges sessions by known referring pages counts of each coded interaction type counts of sessions that include text searches counts of sessions that include browsing (facet selection) counts of mixed text search and browse sessions tracking the index used for searching (keyword, author, subject, etc.) tracking popular search terms statistics can also be tracked over time to show differences in use from week to week throughout the academic year, or from one year to another. methods this method of analysis of user behavior relies on the availability of standard apache server logs, which track user requests in chronological order. a single line from a typical server log will be similar to this: www2_search-access.log.1169596800: 44.154.268.143 www2.lib.ncsu.edu [24/jan/ 2007:00:03:04 -0500] "get /catalog/?n=0&nty=1&ntk=author&view=full&ntt=pettersmann http/1.1" 200 23888 "http://www2.lib.ncsu.edu/catalog/?n=206417" "mozilla/5.0 (windows; u; windows nt 5.1; en-us; rv:1.8.0.9) gecko/20061206 firefox/1.5.0.9" definition log segment host www2_search-access.log.1169596800: 44.154.268.143 www2.lib.ncsu.edu date and time [24/jan/2007:00:03:04 -0500] request “get /catalog/?n=0&nty=1&ntk=author&view=full&ntt=pettersmann http/1.1” status code 200 bytes sent 23888 referring page “http://www2.lib.ncsu.edu/catalog/?n=206417” user agent “mozilla/5.0 (windows; u; windows nt 5.1; en-us; rv:1.8.0.9) gecko/20061206 firefox/1.5.0.9” table 1: typical transaction log components of primary importance for this log analysis technique are the ip address present in the host information section, the date and time stamp, the complete request, and the referring page (table 1). the parameters supplied in the request url must also be tracked. the possible request parameters for this analysis, which was conducted against an endeca powered opac, are listed in table 2. parameter meaning n unique id of facet applied to query no the offset, used for pagination ntk the index for text searching (author, keyword, etc.) ntt text search terms applied to query ne overrides the default sort order for displaying results sort request to expand a facet group ns applies a sort to the results view request to switch result view from brief to full, or vice versa table 2: query parameters (endeca/ncsu specific) the parameters present in the request will vary depending on the software used to power faceted search and text search, however the same basic principles should apply. request parameters are used to code the action the user took that generated the request to the opac. for this log analysis we chose to define a “session” as a series of consecutive requests from a single ip address with no more than 30 minutes passing between each request. the literature related to transaction log analysis provides little consensus about what time to use for distinguishing sessions. generally, researchers using transaction log analysis have used times from 5 to 60 minutes. we chose 30 minutes because it falls near the center of the range of typical times used in studies. for more information about this issue see silverstein (1999) [3], göker (2000) [4], hert (1997) [5], marchionini (2002) [6], mat-hassan (2005) [7], and chau (2005) [8]. the ip address and date/time stamp are used to group discrete requests into sessions. some catalogs may themselves assign http cookies to users to identify a session. where possible, configuring apache to include cookie information in its logs may allow one to use the catalog’s own session tracking mechanism. this may be preferable as a means to track users because it avoids potential problems where an ip address does not in fact uniquely identify a user due to proxy servers or other network configurations. in our log analysis study, we used already captured historical logs which did not include any cookies which the catalog application might apply. we do not think proxy servers or other network configurations that would render the ip address meaningless as a unique identifier of a computer affected our results significantly, since we expected most of the requests to come from ncsu’s campus, dorms, and from homes. it is also possible for users to share computers, which could result in requests from two or more different users appearing in the logs from a single ip address. additional measures we took to remove robot activity (discussed later) also have the effect of removing heavily used public computers from the log, so we think the effect of public computers is small. the term “dwell time” refers to the difference in time between two requests from the same ip address. by “action” we mean the user’s interaction with the search interface. there is a user action generated for each request recorded in the transaction log. “action code” refers to the one action the user took that generated a request. this is deduced by comparing the present set of request parameters with the previous set of request parameters. the difference between these two requests reveals the action the user took to generate the current request. for the opac used in this analysis, there were 12 possible user actions of interest. the codes possible for other systems will vary depending on the parameters available for modification by the user interface. we also assume a stateless system, in which each action generates a separate request to the server handling requests for the opac. table 3 illustrates the action codes possible in the ncsu opac at the time it was analyzed. it is worth noting that only searches, and not item record views, were captured by the log files we examined. additionally, website interactions that include the use of a browser’s “back” or “forward” button are not captured in the logs if the browser uses a cached version of the page. the logical rules in table 3 define how the action codes are determined. coded action explanation logical rule text search first appearance of a search string within a session ntt has a value. value is different from previous ntt value. ntt value has not appeared previously in the same session. if multiple values have changed, ntt change takes precedence. facet search refines the result set by selecting a facet n has a value other than “0”. n value is different from previous n value, or n has an additional value that was not present in the previous n value. if multiple values have changed, takes precedence over all but a change in ntt value. begin text facet search begins a search with a facet and a text string first step in a session. n has a value other than “0” and ntt has a value. refresh no change in search state; suspect user reloaded/refreshed the page not the first step in a session. present and previous parameters have the same values. switch field switches the index used for text search if none of the previous conditions are met, ntk has a value, and that value is different from the previous ntk value. next page views a different page of results if none of the previous conditions are met, no has a value and that value is different from the previous no value. sort changes the sorting of the results if none of the previous conditions are met, sort has a value, and that value is different from the previous sort value. switch view switches record view from brief to full, or vice versa if none of the previous conditions are met, view has value, and that value is different from previous view value. begin full set begins a search by selecting search without entering a search string; this returns all possible results first step in a session. n has a value of “0” and ntt is not present in the url or has a value of “-“. previous term searches on a term again ntt has a value. ntt value has appeared previously in the same session. remove facet removes a facet from the search (either by clicking the “x” in the interface, or by clicking the “back” button in the browser) determined by counting the number of concatenated facet ids in n value. if present count is less than previous count, then this condition applies expand facet group expands a facet group by clicking “show more …” none of the other conditions are met, ne has a value, and that value is different from previous ne value. table 3: example coded user actions the primary benefit of this method over other simpler methods, such as searching for instances of parameters in the log, is the addition of the concept of a session, so that statistics can be generated on a per session rather than just per request basis. an additional benefit of this method is that one can track session characteristics, such as sessions that include only certain kinds of queries. also, because each request is coded, one can eliminate duplicating counts of parameters after they are applied. in effect, this is a way of eliminating counts of parameters still present in the request url that were applied earlier in the session. to illustrate, if a user begins a session by searching for the string “photography,” then applies a facet, applies another facet, and then views 3 pages of results, the initial parameter that holds the query string “photography” would be present throughout all 6 requests of this user’s session. a simple count of the parameters present in this set of requests would count the “ntt=photography” parameter 6 times rather than 1 time. by counting logically derived codes instead of parameters we are able to provide a more accurate count of user interactions with the site. the basic method applied in this approach of parsing a standard server log was adapted from callender’s perl for web site management [9], which includes an excellent description of log formats and approaches to parsing them. some pre-processing of the logs may be required to eliminate requests not related to search (such as requests for images). if there is known robot activity present in the logs, and the ip address of the robot making requests is known, these requests may be eliminated as well. further automated techniques will be used later in the log processing to eliminate most non-human activity from the logs. we processed the transaction logs in several passes. in total, three scripts were written to process the logs. all the scripts, a sample log file, and sample output files may be downloaded from the web [10]. the first script parses the raw log file to prepare it for further processing. this includes converting the date to unix epoch time for time calculations, extracting parameters from the request url, and coding the referring page to track how users are entering into the faceted opac. the second script takes as input the output from the first script. this script’s primary purpose is to parse separate requests into sessions. requests from a single ip address with not more than 30 minutes passing between a request are counted as being part of the same session. each session is assigned a unique identifier and each request within a session is assigned a number from 0-n to capture the order of the requests within each session. the time elapsed between each request within a session is also recorded by the second script. the third script takes as input the output from the second script. with requests grouped into sessions and ordered sequentially, the request parameters can be analyzed for the purpose of assigning a coded action to each request. the code associated with a request is determined by comparing the current state with the previous state. for instance, consider this sequence of two requests: previous state: /catalog/?n=0&view=full&ntk=keyword&ntt=photography current state: /catalog/?view=full&ntt=photography&ntk=keyword&n=201015 previous state current state n=0 n=201015 view=full view=full ntk=keyword ntk=keyword ntt=photography ntt=photography table 4: query parameter state comparison in this case, the difference between the previous state and the current state is the “n=” value, which in the first state is “0” and in the second state is “201015”. because only the “n=” value has changed, the current state is coded as facet search (table 4). it is necessary to account for any default parameters, which apply to the request whether or not they are applied explicitly in the request url. although requests from known robot ip addresses were eliminated prior to processing, there is also an automated way of minimizing the presence of robot activity in the processed logs. jansen found that eliminating sessions with over 100 requests was effective for removing most robot activity, while not excluding most human activity [11]. processed data may be stored and used in multiple ways. we chose to store the processed data in a mysql database for ease of querying the data, and also the ability to output the data in other formats. queries can be written to output the processed log data from a database in a format suitable for analysis in a statistical package, such as sas. the database we created consists of a single table with the fields listed in table 5: field explanation request id an auto incremented id assigned to each request. ip address the ip address originating the request date human-readable date time human-readable time epoch time stamp unix epoch time stamp url the complete request url search term the search term (if any) extracted from the request url index the index (author, subject, keyword, etc.) applied (if any) extracted from the request url facet the id of the facet applied (if any) extracted from the request url view the current view applied (full or brief) extracted from the request url page the offset for viewing results extracted from the request url expand facet group the id of a facet group (id any) extracted from the request url, indicating a request to view the full list of possible values visit number an id assigned by the processing script to every request that belongs to the same session visit step a number from 0-n that maintains the order of requests within a session. elapsed time the difference in time between the present request and the previous request within the same session. referring page the referring page url (if any) that indicates the url of the page that originated the request. action code the code assigned to the request, determined by the processing script by deducing the change in state between the current request and the previous request. table 5: output from log processing results originally, the techniques described in this paper were developed to conduct a one-time analysis of the transaction logs of ncsu’s opac from a four month period. for a more complete overview and discussion of the statistics generated see lown’s master’s paper [12]. the following statistics illustrate the range of data it is possible to generate from the processed logs. first, general observations about user behavior can be made. from the four month period there are 938,848 requests. after removing robot activity and excluding sessions with over 100 requests, there are 130,482 separate sessions. most sessions are short, consisting of a median of 2 requests and lasting about 45 seconds, with just over 22 seconds passing between each request. the most frequent actions are text search (39%), next page (25%), and facet search (18%) (figure 1). it is notable that “refresh” accounts for 7% of requests. a request was counted as a refresh if the present url contained the same parameter values as the previous request. an examination of the data shows that in these cases the unprocessed server logs recorded two consecutive requests from the same ip address with the same parameters applied. because refresh is the correct code for this circumstance, the log processing logic appears to be working as expected. we cannot be sure whether the high number of refreshes present in the logs represents users actually refreshing the page, a problem with the way the server was logging requests, or some other anomaly. figure 1: action code frequency percent text searching is the most frequent request parameter applied to a search on the opac. when users text search they have the option of selecting a specific index to search against. the default index is keyword, which searches across a range of fields. figure 2 illustrates the most frequently searched indexes. figure 2: index searched frequency percent the average number of terms used when searching particular fields can also be determined (table 6). search index average # terms applied title 3.01 multi index 2.87 keyword 2.51 subject 1.97 author 1.90 isbn 1.09 table 6: average terms applied per index these findings are consistent with studies that indicate the average number of terms applied to web searches. for instance, jansen reports that internet searchers use an average of 2.21 terms per query, which is close to the 2.51 average terms applied to the keyword index of the ncsu opac [13]. one can also determine the average number of facets applied to queries that include at least one facet parameter (table 7). number of facets frequency % 1 facet 58% 2 facets 23% 3 facets 12% 4 facets 6% >4 facets 1% table 7: average number of facets applied because requests are grouped into sessions, one can also categorize sessions based on the kinds of requests they include or exclude. for instance, figure 3 compares the percentage of sessions that include at least one text search with the percentage that include at least one facet search. figure 3: percent of sessions including action code figure 4 shows the distribution of action codes by the order in which they occur within sessions. this reveals that text searches decrease and next page views increase over the course of a session. there is also a slight trend that facet searches decrease over the course of a session. note that the first step is excluded because the interface studied provided different options for entry into the opac interface. figure 4: action percent by visit step within sessions the endeca software used to provide search and navigation on the ncsu opac includes some reporting capability built-in. however, the reporting features are essentially a black box; data goes in and reports come out, but it is not possible to examine how the reports are generated.  additionally, building custom log analysis tools enables data cleansing to remove most robot activity. studies of user behavior benefit from minimizing non-human activity. figure 5 compares relative usage of different groups of facets. both the endeca-generated statistics and the data generated from this log analysis are shown. while it is not possible to know how endeca’s reporting function is generating its numbers, we can explain the differences in the numbers generated by our reporting techniques and the endeca reports. for instance, if we calculate facet group usage without eliminating robot activity and without filtering for actions that are coded as facet searches, we find that our numbers are much closer to those in endeca’s reports. we think this illustrates both the value of eliminating robot activity and of coding user actions by comparing urls. figure 5: facet group usage – logs vs. endeca reporting discussion the results generated from our log analysis work are an improvement over the endeca-generated statistics in that they eliminate activity attributable to robots. additionally, the statistics are a more accurate representation of user behavior because they can account for sessions. more specifically, a few results are worth discussion as they will help inform research questions for further studies, especially observational studies, in which we will be able to control search task types and rate the relative success of different search strategies for different tasks. one of the significant limitations of log analysis is that it is impossible to know how successful the user’s search was from the logs. the results show that while users do make use of facets, they do so less than text searching. this holds whether one looks at overall usage of search features as in figure 1 (where text searching occurs 39% of the time and facet searching occurs 18% of the time), or whether one looks at occurrences within sessions as in figure 3 (where text searching occurs at least once in 97% of sessions and facet searching occurs at least once in 34% of sessions). future research studies would be required to explain why text searching occurs more frequently. it may be that text searching is a more effective strategy in more common search tasks, while facet application only helps under different, less frequent circumstances. it is also possible that text searching is more iterative and requires more trial and error and so requires more individual requests to retrieve an acceptable result set. additionally, we know that use of text searching tends to decrease over the course of a session (figure 4). it appears that text searching decreases in proportion to an increase in next page views. it is possible that this is explained by increased exploration of results and less modification of search strategy over the course of a session. however, facet search use decreases less dramatically. this may also indicate a difference in search behavior between shorter and longer sessions. we would have to observe real users to uncover why text searching is used less frequently in later stages of searching. the results also show that some facet groups are used significantly more frequently than others. subject-topic, lc call number, and format together account for about 67% of all facet selections (figure 5). in a future user study we could test whether the placement of these facets on the page affects their usage, whether particular kinds of tasks influence which facet groups are more likely to be used, or whether some facet groups are confusing to users and so are applied less frequently. testing such questions could help designing future interfaces in terms of which facet groupings should be displayed and how they should be displayed on the page. derek rodriguez of the triangle research libraries network (trln) adapted the scripts for use with the trln log format, and also to continually update a database with new log activity for ongoing analysis of user behavior on the trln opac. a web-based interface (under development) enables trln administrators to view statistics within a user-specified date range. (the reporting interface is not accessible publicly.) trln plans to share the logging scripts and database schema with all trln member libraries so that usage statistics can be shared and compared across the trln libraries. these scripts have been enhanced further by hemminger and xi niu at unc chapel hill to process the unc and trln data into a finer grained set of actions. they are performing clustering analysis with their revised coding scheme against the complete log history of endeca searches at unc. their aim is to determine the different types of search behaviors in use, and to better understand how facets are used in conjunction with text searching. conclusion transaction log analysis provides a scalable means to study and monitor how visitors to a faceted navigation opac make use of interface features. programmatically grouping requests into sessions and eliminating non-human requests produces more accurate and more realistic statistics than some out-of-the-box reporting tools. additionally, by coding changes-in-state between consecutive requests within sessions, each request can be reduced to one of a finite set of possible user actions. logs may be parsed and analyzed for one-time studies to inform research questions, or can be continually processed and stored in a database for ongoing monitoring of how users are interacting with the opac. storage in a database table enables querying for customized analyses, as well as building standardized reports viewable through a data dashboard. notes [1] ncsu’s opac, http://www.lib.ncsu.edu/catalog [2] trln’s opac, http://search.trln.org/search.jsp [3] silverstein, c., et al. (1999). “analysis of a very large web search engine query log.” sigir forum, 33(1), 6-12 (coins) [4] göker, a., & he, d. (2000). “analysing web search logs to determine session boundaries for user-oriented learning.” proceedings of adaptive hypermedia and adaptive web-based systems, 319-322. (coins) [5] hert, c. a., & marchionini, g. (1997). “seeking statistical information in federal websites: users, tasks, strategies, and design recommendations.” final report to the bureau of labor statistics. http://ils.unc.edu/~march/blsreport/blsmain.htm [6] marchionini, g. (2002). “co-evolution of user and organizational interfaces: a longitudinal case study of www dissemination of national statistics.” journal of the american society for information science and technology, 53(14), 1192-1209. (coins) [7] mat-hassan, m., et al. (2005) “associating search and navigation behavior through log analysis.” journal of the american society for information science and technology. 56(9), 913-934. (coins) [8] chau, m., fang, x., & sheng, o. r. l. (2005). “analysis of the query logs of a web site search engine.” journal of the american society for information science and technology, 56(13), 1363-1376. (coins) [9] callender, j. (2001). perl for web site management. o’reilly media, inc. (coins) [10] log processing scipts: http://www.lib.ncsu.edu/staff/cwlown/ncsu_log_proc.php [11] jansen, b. j. (2006). “search log analysis: what is it; what?s been done; how to do it.” library and information science research, 28(3), 407-432. (coins) [12] lown, c. (2008). “a transaction log analysis of ncsu’s faceted navigation opac.” sils-edt: http://hdl.handle.net/1901/488 [13] jansen, b. et. al. (2000). “real life, real users, and real needs: a study and analysis of user queries on the web.” information processing and management, 36, 207-227. (coins) about the authors cory lown is a library fellow in digital library initiatives and collection management at north carolina state university libraries, where he contributes as a designer and developer of a variety of digital library services and applications. brad hemminger is an associate professor at the school of information and library science at the university of north carolina at chapel hill. he is the director of the informatics and visualization research laboratory and the center for research and development of digital libraries. his research interests include scholarly communications, information seeking, user interface design, information visualization and bioinformatics. subscribe to comments: for this article | for all articles 2 responses to "extracting user interaction information from the transaction logs of a faceted navigation opac" please leave a response below: google analytics pour primo : étudier le comportement des internautes « bibliothèques [reloaded], 2011-07-07 […] méthode est celle déjà décrite (plus élaborée) dans cet article de 2009, utilisée à la north carolina state university pour leur opac […] continued adventures in integrated article search | bibliographic wilderness, 2013-09-04 […] searches and boolean connectors were used very rarely, in only about 5% of searches.  there are other studies that exist of this sort of thing too, based on analytics, with roughly compatible […] leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – libraryh3lp: a new flexible chat reference system mission editorial committee process and structure code4lib issue 4, 2008-09-22 libraryh3lp: a new flexible chat reference system libraryh3lp is an integrated im and web chat system designed specifically for virtual reference services in libraries. the software was designed for, and is currently used by, a night-time chat reference collaboraton between several large academic libraries. libraryh3lp is designed for the workflow of chat reference, supporting multiple simultaneous operators and routing to queues of operators in a particular service area. it also supports web page embeddable chat ‘widgets’, as well as simultaneous gateways to multiple im protocols. this article discusses the motivation for the development of the software, and provides an overview of libraryh3lp’s features and technical architecture. parts of libraryh3lp are available as open source. the complete application is available as a low-cost hosted service, and will eventually be available to be licensed for local hosting. by pam sessoms and eric sessoms introduction libraryh3lp is an integrated im and web chat system designed specifically for virtual reference services in libraries. libraryh3lp’s evolution has largely been driven by the needs of a night time virtual reference (vr) collaboration between the libraries at duke university, north carolina state university, and the university of north carolina at chapel hill. this “night owl” chat collaboration began in 2003 and allows the libraries to offer their virtual reference services later in the night by sharing the task of staffing it. during the day, each library staffs its service separately, only answering questions from its own patrons. originally, like many libraries around the world, all of the libraries involved in this collaboration offered web chat through a commercial virtual reference provider. web chat services are offered to patrons visiting the library’s website and typically present the patron with a brief entry form containing fields such as name, e-mail address, and question before the chat begins. while some systems allow patrons to be completely anonymous, patrons are typically not able to bypass the entry form altogether. commercial web chat systems typically provide features such as administrative reports, transfer of chats among operators, and multiple simultaneous operators to flexibly handle increases in chat volume. the core technical feature that allowed the night owl collaboration to succeed was support for multiple simultaneous operators and routing to specific “queues,” such as home institutions or departments. enter im in the early-mid 2000’s, many libraries launched very successful im (instant messaging) services for their patrons by registering identities on popular networks like aim, yahoo!, msn and google talk. these im services helped libraries better assist their patrons by integrating the library with the patron’s existing im network. rather than remembering to visit the library’s website in order to ask questions, patrons could im the library directly using their buddy list, where the library could be available in addition to their other im friends. by the end of 2005, the libraries involved in the night owl collaboration had all developed popular im services, but they were able to offer them to their patrons only during the daytime. im software is designed for use by one individual, and it does not support the multiple simultaneous operator and routing to specific queues—features our commercial vr system did. there are a variety of problems associated with staffing a virtual reference service using an im system for this reason. busy services requiring more than one librarian at a time to answer questions, or staffing models that involve brief overlap of librarians at shift change, are very difficult to implement because the second login will disconnect the first, or at the very least generate a confusing error message. staffing im services during the day at each institution was challenging enough, and extending the im service into the night owl collaboration seemed impossible given the single operator limitation. during this time, librarians staffing their library’s vr service during the day had to monitor two separate pieces of software: an im client and the software for the commercial vr system used for web chat. these separate systems had completely different interfaces and features, which made it a bit confusing for the librarians answering questions using both systems. while trying to concentrate on their online reference interview and answering the patron’s actual question, they were also juggling different software packages. enter meebo me meebo me widgets became extremely popular for web chat in 2006. meebo.com is designed to consolidate various im services as well as provide im-style chat through a web page widget, meebo me, so the user does not need a separate im client. on the librarian’s end, meebo me communications can be monitored using the meebo.com website or a standard jabber im client. rather than presenting the patron with an initial form (for e-mail address and question, for instance), meebo me widgets simply provide a place for the patron to ask their question anonymously (see attached figure 1). many libraries saw a sharp increase in their web chats when meebo me widgets were put in place. meebo.com also provided convenience for librarians by allowing consolidation of standard im and meebo me traffic in one interface. we found gaim, an open source im client that later changed its name to pidgin, particularly useful, especially with a plug-in[1] developed by one of this paper’s authors, eric, to provide some changes to gaim’s handling of meebo me specifically useful for vr. however, meebo is still essentially an im solution exhibiting the same problems of a one-to-one communications model, rather than supporting multiple operators and queues. while librarians using meebo me widgets rather than a commercial web chat system now only had to monitor one piece of software to handle chat and im services, those one-to-one limitations continued to make effective staffing a challenge, especially as the services grew and became more popular with patrons. librarians with busy services tried to compensate in a number of ways. for instance, they might monitor separate protocols using separate accounts. in this way, they could have more than one librarian answering patrons at a time. however, this was only a partial solution because often more than one patron on a given popular protocol showed up simultaneously, and the librarian would have to juggle between them. if the library staffed the service in shifts—for example from staff offices—changing shifts was problematic. librarians would have to communicate with each other at each shift change to be sure that the second librarian would not disconnect the first librarian from an ongoing chat. if the first librarian had another commitment, such as a meeting, at shift change, it was difficult or impossible for that librarian to extricate him or herself from the chat session so that the second librarian could take over. additionally, while very popular, meebo me widgets are not without some usability concerns. first, meebo me widgets are flash, and the implementation is not accessible to screen readers used by visually impaired patrons. second, links sent to the patron by the librarian are not clickable. while patrons can copy and paste these links, this often results in the patron accidentally navigating away from their chat when they paste the link to the same browser window they were using for their chat. however, the success of the core functionality offered by a meebo me widget, that of a fast, responsive one-on-one conversation without an entry form barrier, was undeniable. meanwhile, back at the ranch… the libraries in the night owl collaboration all had popular daytime-only im services that could not realistically be offered through the collaboration because of the technical barriers described above. even by day all of the libraries were struggling with their own in-house im traffic; they had staff willing to answer the increasing volume of questions, but lacked the technology to do so efficiently. additionally, use of the commercial web chat system had plateaued, and everyone involved wanted to switch to a widget-based format for web chat to see if usage would increase with the removal of the patrons’ entry form. it was clear that the needs of the service had outgrown the systems in use. extensive research and testing showed that there were no good technical systems in place that addressed the problem, and it sounded like no one was very far along in building one; development of specialized software was needed. eric wrote a very modest predecessor to libraryh3lp that allowed the night owl collaboration to move forward. with the start of the fall semester in 2007, the libraries went live with pidgin4lib, eric’s peer-to-peer solution that first broke down the “one to many” problem plaguing growth of the service by providing support for multiple operators and a queue for each institution. for the first time, the night owl collaboration was able to include im services. all of the institutions also switched to meebo me widgets rather than their commercial vr package for web chat. multiple librarians could be signed in and receive ims and meebo me chats from patrons, and librarians could come and go in shifts without disconnecting each other. usage statistics increased dramatically. for example, unc-chapel hill’s traffic for the time period covered by the night owl collaboration increased 300% over what it had been the previous semester. however, pidgin4lib was constrained by its peer-to-peer architecture; upgrades had to be done on many, many computers across three major university libraries. there were also some bugs that would be difficult or impossible to address within its framework. two things were clear: we were on the right track, and the system needed to be nearly completely re-written for better scalability. this was the motivation for the development of libraryh3lp, the system currently in use by the night owl collaboration as well as for regular vr services during the day at all three institutions. libraryh3lp service, technical, and design goals we set out to write a system capable of routing and dispatching both web-chat and instant messaging traffic. as a secondary goal, we wanted the system to be able to handle sms, irc, and whatever trendy new communication channels should come down the pipeline in the future. in software, it never pays to worry over much about the future—that is usually done at the expense of the present—but some choices are more flexible than others, so we wanted to be aware of the possibilities. these goals led us to seek out a standard platform with a well-defined interface to external systems. as work on libraryh3lp was not funded by any library or grant source, another goal was to do as little reinventing the wheel as possible. this led us to look for open-source software solutions that we could build on. together these objectives led us to the extensible messaging and presence protocol (xmpp, known as “jabber”). xmpp is an open standard with a defined protocol for interfacing to legacy im protocols such as aim. there is a wealth of software supporting xmpp, including open-source servers and clients, as well as transports implementing the gateway protocol. in truth, we never really considered any solution other than xmpp. it fit with our problem so well that even now it is hard to imagine that any other solution could have worked out. given our previous experience with pidgin4lib, we had a good idea of how libraryh3lp needed to operate. some of our overall system design goals were to: 1. move to a centralized architecture with a web administrative interface. 2. provide flexible routing of ims and web chats to multiple librarian operators. 3. limit specialization of locally-installed jabber clients such as pidgin to plugins if possible (do not fork the main program). 4. eventually provide a centralized web client for more extensive jabber client customization. 5. place the design and management of a vr service in the hands of the local administrators. 6. allow easy creation and deletion of queues with assignment of one or more operators to each queue. for example: reference, circulation, youth services, etc. 7. provide internal im functionality for inter-librarian communication. 8. provide the ability to transfer any chat or im to other librarians under the same “administrative domain” (such as a university or larger collative environment). 9. provide hierarchical permissions in the system to facilitate collaborative services that span multiple institutions, such as needed by the night owl collaboration. for our web chat widget, some of our goals were: 1. allow web page authors to easily either embed the web chat widget or have it pop-up from an image or link of their own design. 2. develop a presence api so that web page authors can creatively assign dynamic behaviors based on whether a chat queue is online or offline. 3. have typing notification for web chats since it helps the conversation flow so nicely in ims. 4. strive for accessibility for users of screen readers. 5. make links sent to the patron clickable for usability. 6. empower patrons as much as possible by providing convenient features such as pop out, send file, e-mail transcript, and alert noise. libraryh3lp overview we are successfully meeting these goals by developing libraryh3lp. libraryh3lp has evolved into a large system, and it is continuing to improve, with frequent updates. this overview section will describe many of its features, and the next section will describe the underlying technology. accounts, queues, and gateways librarians in charge of running vr services get started with libraryh3lp by registering an admin account on the libraryh3lp.com website. this account is used to create additional user accounts (see attached figure 2) and queues. the main admin account and the user accounts are jabber accounts that can be used to staff the service in addition to being used on the libraryh3lp website for various purposes. users can be assigned to zero or more queues to receive incoming chats from patrons. users assigned to zero queues can still be buddied with other operators to facilitate back office communications, and they can receive transferred chats from other operators in the same administrative domain. queues (see figure 3 and 4) are a central part of libraryh3lp. they can be easily created and deleted, and they serve as a way to connect patrons to the most appropriate librarian for their question. a queue’s name is specified when creating a web chat widget, and patron traffic coming into a widget is thereby routed to the librarians assigned to the specified queue. most libraries will have at least a general ask-a-librarian queue. specialized queues can also be created for other departments, such as circulation; services, such as assistance with tools such as endnote or refworks; or areas of subject expertise, such as literature or history. queues are online if at least one operator assigned to the queue is available. im gateways are integrated at the queue level. the admin enters the im identity’s credentials, which are stored in a database and passed off to the home service (such as aol, yahoo!, or google) for login when a queue changes to online status (see figure 5). in this way, the one-to-many problem with staffing an im service is overcome. all of the librarians online for a queue will receive the ims on its gateways; librarians will not disconnect each other at shift change. figure 3: admin view of queues. colors indicate online or offline status. figure 4: refworks queue. three operators are assigned to this queue. the queue is online because an operator is online. additional operators are added using the drop-down list at the bottom. figure 5: im gateways on a queue. when this queue is online, all of these im accounts will also be online. figure 6: admin’s call history/transcript management interface. also at the queue level, administrators can opt into storing their transcripts on the libraryh3lp server (see figure 6). by default, transcript storage is turned off. the admin interface includes detailed call history management. this allows administrators to save call histories to a csv file, to sort by a number of different criteria including queue, protocol, and operator, and to manage their transcripts. administrators can quickly see chat duration, how long a patron waited before being answered, and chats that went unanswered. transcripts can be deleted or downloaded in bulk; it is also possible to delete entire call histories, which deletes not just the transcript but the full record of the chat. jabber client, incoming messages, and transfers now that we’ve seen a bit of the admin side and system setup, we will look at what librarians working with patrons see. librarians use a jabber client such as pidgin to monitor their services.[2] they can also use their client to im other librarians on the libraryh3lp system. pidgin is a multi-protocol client which allows users to also simultaneously monitor their own personal im accounts on other systems, such as aim. we provide a custom libraryh3lp plugin for pidgin that enables additional functionality not available to other clients.[3] a full-featured web client is in the works. when a patron sends a chat using a widget, or a patron sends an im to an account registered with a gateway, the chat is received as an im in the librarian’s jabber client. if more than one librarian for a queue is online, they will all receive the incoming messages for that queue, but only the first to send a response to the patron will become connected to the patron. the others will be disconnected from that patron and will receive a message indicating which librarian took the chat. figure 7: pidgin buddy list showing libraryh3lp buddies as well as personal contacts on other protocols. figure 8: anatomy of an incoming chat as seen by the librarian. incoming messages are comprised of three lines of text (see figure 8): 1. the patron’s message. 2. a note stating which queue the patron used to enter the system. this provides helpful contextual information for the librarian. 3. a link to the libraryh3lp website for managing that specific chat, for example, by transferring it to another operator or queue. the system also sends a number of useful messages to librarians under various circumstances. first, when more than one librarian is online for a queue, the librarians who did not respond to the patron first, and thereby do not become connected to the patron, receive notification of which librarian did successfully answer the chat. next, when patrons using the libraryh3lp web chat widget close their window or navigate away from a page containing an embedded widget, librarians are notified that the patron has “left the conversation.” last, if a message fails to make its way to the patron, the librarian receives “message not received;” this usually happens when an im patron has signed out of their account or the librarian tries to send a message to a patron that has already navigated away from their web chat. many chat and im sessions can be handled entirely within the jabber client, by sending messages and links back and forth. sometimes the librarian may need to perform more advanced tasks, and the “chat management” link sent as the third line of the chat facilitates these tasks by taking the librarian to a web page providing several additional features (see figure 9). once signed into the website, the librarian can transfer the chat to another queue or individual in the same administrative domain, send a file to the patron, e-mail a transcript, or view any previous chats from this patron, assuming transcripts have been turned on for that queue. repeat im patrons receive a stable guest id number, and it is possible to read transcripts from any stored earlier session with that patron. patrons on web chats are identified by an anonymous guest id number stored in a session cookie, so these returning patrons can only be identified for the duration of their browser session. figure 9: chat management page with transfer, e-mail transcript, and send file options. the widget, presence api, and web pages the patrons’ widget is simple to use, and it provides typing notification for both the patron and the librarian. links sent to the patron are clickable and open in a new window or tab, depending on the patron’s web browser. it provides a number of advanced features as well. currently, patron widget controls include: 1. “pop out” an embedded widget. many embedded widgets tend to be very small because of limited real estate on web pages. it can be very difficult to follow a detailed chat conversation in a small widget because of the scrolling required. once popped out, widgets can be easily resized by the patron, who can then toggle between their chat and the research they are conducting. 2. send a file to the librarian. this can be helpful when the patron needs to show the librarian a problem they are having with a bibliographic citation or with a bad pdf from a library database. 3. send an e-mail transcript of the chat. this saves the patron from needing to copy and paste the text of their chat into an e-mail or text file in order to refer to it later. 4. toggle an alert noise for new messages from the librarian. this helps the patron alternate between their chat and their research since they don’t have to keep checking back for new messages from the librarian. the widget is highly customizable, and the libraryh3lp administrative interface includes a tool called the configurator that allows wysiwyg editing of widgets. it is also possible to provide customized css to change a widget’s appearance. the graphical appearance can be stored on the server and referenced in web pages, and these skins can be shared with other libraryh3lp users. internationalization and localization, to display text in the widget in the language indicated as preferred in the patron’s web browser, are also available (see figure 10). currently, translations are available for czech, french, and spanish, and we will install additional translations provided to us.[4] figure 10: patrons’ widget illustrating czech localization and patron control icons. libraryh3lp provides a presence api that allows a great deal of customization of a service. it is very easy to present a widget when a queue is online, and when it is offline, present something else entirely, such as an e-mail form or an entry point into a 24/7 backup service (see figures 11a and 11b). the administrative interface includes a tool called the servinator that assists with generation of html skeletons using the api. using the servinator, it is easy to create code for embedded widgets or pop-up widgets (see figure 12), and it is easy to create html that produces tiered service levels. for instance, a tiered service level might show a subject expert queue’s widget if they are online, but if they are not, will show the general reference desk queue’s widget; if the general reference desk is offline as well, an offline option can be specified (see figure 13). because html is generated, it is very easy to modify the skeleton code provided in order to give the patron contextual information that will help set service expectations. figure 11a: presence api: online and offline views of main unc-chapel hill ask-a-librarian queue. when this queue is offline, an entry point into the state-wide collaborative system, which features 24/7 backup service, displays. figure 11b: presence api: online/offline example from auroria library. figure 12: online/offline button inside catalog. if online, the button generates a popup window for chat. if offline, a different button displays and takes the patron to an e-mail reference form. figure 13: presence api: service rollover. if the specialist is online, show her widget; if she is not online, show the general reference service widget; if both are offline, indicate chat is offline. code: getting started there are a number of free, open-source xmpp servers available.[5] like many open-source efforts, xmpp server implementations are available in a variety of languages and vary in their degree of completeness, correctness, and dependability. the two most featureful servers available when we started this project were ejabberd by process one and openfire by jive software. ejabberd and openfire have more-or-less equivalent functionality, and both are open source. they differ primarily in that openfire is written in java and ejabberd is written in erlang (thus, the erlang jabber daemon). the erlang language is initially shocking to some programmers, as it lacks many basic things most programmers take for granted, such as loops, variables, and assignment. it also lacks static typing, type declarations, classes and objects, thread synchronization primitives, fancy graphical integrated development environments, and armies of overpriced consultants. the two servers also differ in that ejabberd is written in only about 40,000 lines of code where openfire requires 200,000. ejabberd is faster, supports more simultaneous users on the same hardware, and has built-in support for clustering which the open-source openfire lacks. clustering is important for scalability and reliability. openfire is a bit easier to setup and install. we owe most of our success to picking the erlang option. erlang erlang is a soft real-time programming language developed by ericsson in the 80s for telecommunications applications. ericsson developed the language because nothing else available at the time fit the needs of their applications. erlang is named for the danish mathematician a.k. erlang, not the company ericsson.[6] erlang is known primarily for its performance, scalability, and famous “nine-9s” of uptime.[7] like many of the more successful programming languages, erlang is built around a small set of core concepts. erlang is designed for fault-tolerance and reliability, and most features of the language flow naturally out of this choice: hot code swapping. code can be added to, removed from, or replaced in an erlang system at run-time, with no restart needed. versioning of executable modules is built-in. distributed computing. an immediate consequence of designing for fault-tolerance is the need to be able to distribute a program across multiple machines. networking. high-performance networking primitives are built into the language as a consequence of the need to distribute computations. binaries. to simplify the implementation of networking protocols, erlang has a powerful built-in syntax for low-level manipulation of bits and bytes. message-passing concurrency. erlang systems support tens of thousands of lightweight threads, leading to a style that has come to be known as “concurrency-oriented programming.” dynamic typing. neither hot code swapping nor message passing concurrency can reach their full potential in a statically-typed system: too much has to be decided up front. an advantage of dynamic systems is that they can grow and change over time. pattern matching. a syntactic feature borrowed from prolog, pattern matching greatly simplifies many types of code. immutability. lacking data synchronization primitives, erlang needs to avoid all the problems that can arise when multiple threads of execution modify the same data. it does this by keeping all data immutable, thus there is no assignment in the language. functional programming. having thrown out mutable data and assignment, many other language features cease to make sense, including primitive things like control structures (that is, loops: for, while, do/until). while functional programming is usually described in terms of negatives—what the language doesn’t have—in practice it has many advantages over procedural programming [7a]. a practical consequence in erlang is that code, not merely data, can be sent around the network to run on different compute nodes. this opens up all kinds of exciting options, one of which is… distributed transactional database. a program that doesn’t modify any data isn’t useful for much other than generating heat from the cpu. erlang comes bundled with a distributed (non-relational) database system called mnesia that allows for persistence. it has built-in support for replicated tables (for reliability and speed of reading) as well as fragmented tables (for speed of writing) to handle most of the database needs of any erlang program. all of these features are relevant and contribute quite directly to the implementation of an xmpp server. they go a long way to account for the success of ejabberd, and provide a great foundation for the libraryh3lp system. we want to emphasize, however, that the reason we picked erlang, and by extension ejabberd, was its support for functional programming. all those other great features are just gravy. functional programming it is a given that one should always program at the highest level of abstraction available. greater abstraction leads to less code and better organization. less, better organized code is easier to understand, maintain, test, and debug. functional programming provides new tools for abstraction that are not available in procedural languages. antecdotally, functional programmers often report a factor of 10 improvement in speed of development and size of the developed system in comparison to procedural programmers. in a study conducted by ericsson, erlang was found to have a factor of 4 edge over other languages.[8] in the case of ejabberd, we can see that the resulting program is 1/5th the size of its nearest compettitor, but is faster, more scalable, and more reliable. (the speed comes not from the intrinsic speed of the erlang byte-code intepreter, but from erlang’s lightweight threads and high-performance networking.) all of this means two things that very directly impacted the libraryh3lp project: we found ejabberd to be a well-crafted, modular, easily extensible system. it was obviously written by experienced, capable software engineers who took advantage of the organizational possibilites offered by the language. the productivity boost offered by the language meant that it was possible and realistic for one programmer to do interesting and useful work. libraryh3lp technical architecture we will begin by briefly recapping the features described above in the overview section in more technical terms. to staff the system and receive chats, librarians log into the ejabberd server using their jabber client. we provide a plugin[9] for the pidgin multi-protocol client that enables additional functionality not available to other clients. when it is necessary to access additional functionality beyond messaging, as when setting up the system or dealing with exceptional situations, librarians can log into the web site using any standards-compliant browser to review transcripts, transfer chats, and send files to patrons. additionally, vr administrators can use the web site to generate reports, monitor their queues and librarians, and see chat traffic in real-time. the web site communicates with the jabber server both directly, using erlang’s native remote procedure call mechanism, and indirectly through the shared mnesia database. patrons can enter the system in two ways. im patrons talk to gateways using their protocol of choice, and the gateways translate their traffic into xmpp packets and forward them on to the jabber server, where libraryh3lp takes over. web patrons start with a chat widget generated by the web server, which then proxies their communication with the jabber server. the widget natively speaks xmpp, and so does not go through any translation; the proxying is simply there to work around browser security policies preventing cross-site scripting (talking to two ports on the same server is, unfortunately, “cross-site” to most browsers). figure 14 provides an overview of the architecture of a single libraryh3lp server. figure 14: internals of a single libraryh3lp server. ejabberd and custom extension the core of libraryh3lp is a flexible and intelligent routing system written as an extension module to ejabberd. by writing the routing piece at this level, we are guaranteed to handle not only web-chat traffic, but also instant messaging and whatever other communication mechanisms can be gatewayed into xmpp. to interface with non-xmpp instant messaging systems, we originally used the py-transports family of gateways. we have since developed a libpurple-based multi-protocol gateway (libpurple is the gui-less core of pidgin) that allows us to connect to and have a presence on any protocol supported by the pidgin multi-protocol client.[10] this gives us greater reach and breadth than any other chat/im system. certain other decisions followed easily on the decision to base our system on erlang and ejabberd. erlyweb mvc framework erlyweb is an mvc (model-view-controller) web development framework for erlang, in the style of rails or django. it provides generators for the database access code (model), templates for the view, and a flexible framework for url routing (controller). we use this to provide libraries with a full-featured administrative interface. the goal of this interface is to allow libraries to have complete control over the services they offer, and to add and change services at any time, in the hope that they will find the best service and staffing model for themselves through experimentation. we do not dictate anything about how libraries use the service. libraryh3lp also uses the routing engine to provide restful resources in support of programmatic interaction with our server. by adhering to rest design principles, it was very easy to build an api on top of our existing web interface. the possibilities for using this api are open-ended, but one development we are already seeing is the use of scripts to automate common administrative tasks. for example, instead of a vr administrator logging into the web site daily to download transcripts and usage statistics, she can run a simple script that accesses the web site to download the same information and automatically import it into her local reference tracking tools. if the script is in cron, the whole process becomes automated. erlyweb made it easy to provide this feature. one of the nicer features of erlyweb is support for component-oriented development, which aids in the factoring of the server-side code. in my experience with other web-development frameworks, only rails comes close in terms of maintainability of the code, but ruby has nothing on erlang when it comes to performance. yaws web server erlyweb depends on yaws (yet another web server), so that decided our choice of server software. yaws is an extremely high-performance server that supports many times more concurrent connections than apache.[11] it is true that apache and other, newer web servers (nginx) are doing a lot of work on performance, particularly handling massive numbers of simultaneous connections. yaws was the existence proof that showed this level of performance was possible. in order to provide real-time response to web users, we ported erlycomet to erlyweb (and, thus, yaws). comet refers to a family of strategies for developing “live” web pages. whereas ajax techniques can be used to make web pages highly responsive to user-interaction, ajax has limitations related to the display of real-time data from the web. the most common ajax technique for live web pages is polling, but this is both slow (because of the polling interval) and high-bandwidth (because you keep polling even though most of the time nothing ever happens). comet is being developed to work around these limitations. comet is still immature, with many people trying many varied approaches, but the most mature take on comet (so far) is the so-called bayeux protocol. bayeux is supported by the jquery and dojo javascript libraries, and increasingly by server-side components. erlycomet is an implementation of the bayeux protocol for comet. by adhering to a standard, we were able to avoid rolling our own comet implementation in the browser. this saved us a lot of work. erlycomet allows us to provide a real-time monitoring feature in the admin interface, so that libraries can see and manage their chat traffic in real-time. the erlyweb system plays two roles in libraryh3lp. on the one hand, it provides the administrative interface by which librarians setup and monitor their vr services. on the other, it produces code for the web-chat widget. javacript jabber client (jsjac) the client-side of the web-chat widget is written in jsjac (javascript jabber client), a library that communicates with the server using the xmpp bosh (bidirectional-streams over synchronous http) protocol. bosh is a comet-style protocol that allows us to avoid polling. bosh keeps the web chat fast and responsive while at the same time reducing server load. glue code in the client is written in jquery, a lightweight javascript library. we made every effort to comply with all applicable standards in the development of the widget: it’s xhtml with css and javascript only. this has resulted in an application that works with screen readers. it has also yielded what we like to call “the hackable widget”, as it is very easy for libraries to customize the chat however they desire. there is no libraryh3lp branding in the widget, leaving libraries free to fully make their widgets their own. we are working on a system to allow libraries to host the widget themselves, so they can do even more extensive customizations. conclusion as of this writing, approximately 30 libraries[12] are staffing over 60 queues, and there is an average of 150 chats per day on the system as a whole. however, we anticipate a large increase in traffic with the fall academic semester. to prepare for this load, we have already started migrating parts of the system to amazon’s elastic compute cloud (ec2) (see figure 15). current projects include adding additional gateway protocols, creating features to support larger-scale collaborative services, creating a web client for librarians staffing the service, and improving the local administrator’s interface by providing additional tools for reporting, including basic automated anonymization of transcripts. libraryh3lp is built on a foundation of open source projects, and we have in turn released a number of components back to the open source community. so far, these include the full web chat widget, a port of erlycomet to erlyweb, the plugin for pidgin, and patches to ejabberd, yaws, and pytransports.[13] we plan to release more components soon, including our libpurple multi-protocol gateway. those interested in testing libraryh3lp can freely register for an admin account at http://libraryh3lp.com/. to help keep the system sustainable, nub games, inc., the company funding libraryh3lp’s development and providing libraryh3lp’s professional web hosting, has always planned to begin implementing a modest annual hosting fee once a useful set of features was in place; this should happen this fall.[14] libraries can freely experiment with the system for a trial period, with the hosting fee required if a library finds libraryh3lp to be a suitable solution for production services. eventually, most of the system, with the exception of the routing component, will be open-sourced under the gnu affero general public license version 3 (agpl3). it will be possible to license the routing component as a binary, and interested parties should contact nub games, inc. those interested in running a complete libraryh3lp system on their own hardware will also be able to purchase licensing and support from nub games, inc. libraryh3lp was written to address the needs of real, working librarians who were struggling with using the existing tools while striving to expand their growing services. we certainly did not originally set out to write a fully-featured vr system, but that is how the project has evolved. in sum, we hope we are producing a platform for freedom and creativity in the provision of virtual reference services. we strive to take the technical limitations out of the way so that librarians can concentrate on the best way to provide services to their patrons. more about libraryh3lp http://libraryh3lp.com/ main site. http://groups.google.com/group/libraryh3lp group http://libraryh3lp.blogspot.com/ blog http://code.google.com/p/libraryh3lp/ documentation figure 15: multi-server architecture for the cloud. notes [1] plugin and source code available from http://www.lib.unc.edu/reference/eref/pidgin/meebomewidget.html [2] http://code.google.com/p/libraryh3lp/wiki/suitablejabberclients [3] http://code.google.com/p/libraryh3lp/wiki/clientsetuppidgin [4] http://code.google.com/p/libraryh3lp/wiki/localization_and_internationalization [5] http://en.wikipedia.org/wiki/list_of_jabber_server_software [6] http://www.erlang.org/course/history.html [7] http://www.cincomsmalltalk.com/userblogs/ralph/blogview?entry=3364027251 [7a] http://www.cs.chalmers.se/~rjmh/papers/whyfp.html [8] http://www.erlang.se/publications/ulf_wiger.pdf [9] http://code.google.com/p/libraryh3lp/wiki/clientsetuppidgin [10] http://pidgin.im/ [11] http://www.sics.se/~joe/apachevsyaws.html [12] some have listed themselves on the library success wiki: http://www.libsuccess.org/index.php?title=online_reference#libraries_using_libraryh3lp_for_embedded_chat [13] http://code.google.com/p/libraryh3lp/wiki/opensourcecontributions [14] http://code.google.com/p/libraryh3lp/wiki/free_or_fee_full_disclosure about the authors pam sessoms is electronic reference services librarian at unc-chapel hill’s davis library. her professional interests include virtual reference services, bibliographic management utilities, maintaining access to legacy research databases and software through the use of virtualization software, and providing library services to patrons with disabilities. pam can be reached at psessoms at gmail dot com. eric sessoms is president of nub games, inc., and is currently executive consultant for end-to-end data operations for the us national radio astronomy observatory, where he spends his time worrying about astronomical data archiving and virtual observatories. formerly, he was chief software architect for the nrao and worked on the green bank telescope, primarily in areas of usability and data reduction. in previous lives, he was lead programmer for sinister games, technology evangelist for stingray software, and worked on microsoft operations manager. eric can be reached at nubgames at gmail dot com. tags: libraryhelp subscribe to comments: for this article | for all articles 10 responses to "libraryh3lp: a new flexible chat reference system" please leave a response below: kim shreve, 2008-10-09 i work for the hesburgh library at the university of notre dame as their desktop computer consultant and we are currently using meebo, but find it is unstable and not reliable at all times. your products sounds interesting and i would like to know how much it costs to install it and use it and if there is any support on it? thank you, kimberly a. shreve technical support analyst – consultant information, research & instructional services 116 hesburgh library, university of notre dame phone: (574) 631-6640 eric sessoms, 2008-10-09 hi kim, pricing information is available on our wiki here: http://code.google.com/p/libraryh3lp/wiki/free_or_fee_full_disclosure the service is fully hosted, including a librarian’s web client that we’ll be releasing in the next few days, so it is not necessary to install anything. support is available through our newsgroup: http://groups.google.com/group/libraryh3lp commercial support is also an option for users who desire it through altarama’s refchatter product: http://refchatter.net/ hth, eric stephanie beene, 2008-11-25 dear pam and/or eric: i am conducting research at the university of texas at austin on the current embedded im chat service, which uses meebo. i have some usage statistics from the head of reference services and we will be conducting a series of usability studies from our end, comparing this service and meebo. do you have any usage statistics from your library? do you have any testimonies from users or any satisfaction survey results? thank you! stephanie beene msis, the university of texas at austin, 2009 pam sessoms, 2008-11-30 hi stephanie, yes, i have lots of usage statistics for unc-chapel hill. i’ll get in touch with you during the week to see what will be most useful to you. user testimonies at this point are just informal statements given during typical chat sessions. we have not done user satisfaction surveys, favoring instead reading all of our transcripts by hand. we have done a couple of different usability studies, or rather, we’ve tested a few different libraryh3lp chat widget options as part of larger usability studies. libraryh3lp provides code for creating embedded widgets, pop-up widgets, and “follow-me” widgets. the idea here is that there probably isn’t a one size fits all solution, and we want people to be able to experiment and see what might work best. these widget types are described in more detail at http://libraryh3lp.com/docs/h3lp the first usability study was to determine user preference between an embedded widget and a pop-up widget on a no-results page inside search trln (union catalog). users preferred the pop-up widget overall for a variety of reasons. they typically didn’t see much point in having chat on the no-results page in particular but liked the idea of having it as a general part of the catalog user interface. these results were presented at lida 2008 — i can send you a copy of that paper if you’d like. the second usability test was to determine user preference between an pop-up widget and a “follow-me” widget inside the unc-chapel hill catalog. here, most users preferred the pop-up widget, but i think that our test setup was flawed. we asked the first four (out of five total) subjects to use the pop-up widget first, and then the follow-me widget. all of these users preferred the pop-up widget. when using the follow-me widget, they seemed to think that the usability test task was to see if they could get the follow-me widget to act like the pop-up widget, so they used the pop-out option. i had a hunch, and i asked the fifth subject to use the follow-me widget first, followed by the pop-up widget; that user strongly preferred the follow-me. so, i think that this needs more testing. :) these results have not been published. thanks for your interest! -pam. natalie, 2009-03-03 hello, i would like to do a demo with library h3lp. (we’re having a sort of tech petting zoo at a conference.) is it ok if i set up an account for this purpose? will there be any charges? thanks, natalie tagge pam sessoms, 2009-03-09 hi natalie, i think we spoke in email, but just for the record here… no charges for demos, feel free to get it all set up. good luck and have fun! donald woeltje, 2009-04-06 i’ve seen where you call this product secure….but i don’t see anything secure about it. im is rapidly becoming the attack vector of choice for individuals trying to compromise other peoples systems. it doesn’t appear as if your product does any kind of traffic encryption. but my main issue is who it allows to talk to who. since i’m not really familiar with your system, i’d like to know that information. who can im to who using your system? can joe blow on the internet use your system as an im gateway to im student jane doe at xyz university? or can joe blow on the internet only communicate with a librarian at xyz university? what about auditing? we need to audit that traffic to meet legislative requirements as well as certain industry standards. are we going to be able to audit that traffic and obtain audit reports? what kind of authentication capability is there? use of this kind of technology without any kind of security controls can cause issues related to ferpa’s requirement for protecting student information. pam sessoms, 2009-04-11 donald, libraryh3lp is really a platform for building chat and im reference services. if you don’t want to offer commercial im services through aim, yahoo!, etc, there is nothing in libraryh3lp forcing you to do so. if security and encryption are your main concerns, you can built a libraryh3lp service using secure sockets (https) end-to-end, from the patrons’ widget to the librarians’ client to the admin interface. typically, libraries using im gateways were already offering im services before using libraryh3lp, but once the service became popular, it became difficult to staff because of the one-to-one limitation. libraryh3lp’s im gateways make staffing an im service easier since you can have more than one librarian at at time monitoring your public-facing im buddy names. to your specific questions: “who can im to who using your system? can joe blow on the internet use your system as an im gateway to im student jane doe at xyz university?” no. im gateways are for public-facing accounts such as “my-university-librarian” on aim. these public-facing accounts must get added to the system deliberately, and of course, they must have originally been created by registering with the relevant im network (such as aol for aim). “or can joe blow on the internet only communicate with a librarian at xyz university?” yes. “what about auditing? we need to audit that traffic to meet legislative requirements as well as certain industry standards. are we going to be able to audit that traffic and obtain audit reports?” well, i don’t know what specific industry standards you’re alluding to here, but most college and university it staff we’ve spoken with so far have been concerned with the security of the chat and im transcripts (conversations that take place between students and librarians). here are some details about that: – transcript storage is turned off by default; the local admin has to turn it on deliberately for each queue. you can certainly opt to trap that data locally, perhaps using a locally-installed jabber client and a local network drive location, instead of it staying on the libraryh3lp servers if you prefer. – if transcripts are opted into, they are not currently being anonymized. this means that things like patron im buddy names and any identifying information revealed during the chat or im conversation are stored. only users with login information for your school’s libraryh3lp acount(s) can read the transcripts. – transcripts can be downloaded in bulk and/or deleted from libraryh3lp at any time. deleting the transcripts leaves the statistical record of the chat itself, which can then be used for gathering statistics and using the reports feature. – it is also possible to delete the entire record, including statistical data, of the chats (“delete calls” under the monitor activity tab). – csv files of the statistical data can be downloaded across time periods and imported into excel. the transcripts and csv files of statistical data are what is available to librarians administering their services through the libraryh3lp platform. we don’t currently provide any kind of additional auditing tools for libraries. “what kind of authentication capability is there?” i’m not certain what kind of authentication you mean here. i mentioned secure sockets previously. there are a variety of jabber clients that librarians can use to staff the service. if you mean authentication for your patrons/students, that’s really up to you and how you implement the service. for im services, your patrons aren’t authenticating other than by providing their login credentials to aim, msn, etc. if you mean authentication for your patrons/students using the webchat widget (the little chat box placed on the library web page), then there is none by default. however, since you’re in control of your web pages, you can put authentication in front of it using your own proxy server or whatever method you’d like. that’s up to you. hope this helps a bit. -pam. university of alberta libraries « imavailable4u, 2011-04-10 […] library routes messages from other im networks such as aim, yahoo! msn and gtalk! as mentioned on code {4} lib, libraryh3lp is a new flexible reference system geared towards virtual services in libraries. need […] issues with inevitability in info tech discourse (pt. 1) | gnovis journal, 2011-08-12 […] pam and eric sessoms. “libraryh3lp: a new flexible chat reference system.” code4lib journal 4 […] leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – from the catalog to the book on the shelf: building a mapping application for vufind mission editorial committee process and structure code4lib issue 17, 2012-06-01 from the catalog to the book on the shelf: building a mapping application for vufind at yale university library (yul), recorded reference transactions revealed that after finding a book in the catalog patrons had difficulty knowing how to use the call number to find the book on the shelf. the library created a mobile service to help locate the call number in the library stacks. from any call number of a book in sterling memorial library at yul, a map will be displayed which highlights that call number’s general area on a floor in the stacks. yul introduced the mapping application in yufind, a catalog in place at yale since 2008 which is based on vufind. by kathleen bauer, michael friscia, and scott matheson ubiquitous smart phone and mobile device use has raised a new opportunity for libraries.  current mobile options provide patrons with better access both physical and virtual library resources. the handheld, portable smart phone is a perfect technology to help patrons discover material online and then successfully navigate physical stacks locations. libraries have begun to use this technology, offering views of their online catalogs optimized for mobile access, mobile tours of libraries and qr codes as ways to connect patrons to their physical libraries. notable examples include the new york public library (www.nypl.org/mobile-help) and north carolina state university (http://www.lib.ncsu.edu/dli/projects/librariesmobile/).  maps are an obvious match for mobile development because a person can easily use a map on a handheld device to guide them through streets, or for that matter the shelves in a library. at yale university library (yul) it was apparent from recorded reference transactions that, after finding a book in the catalog, patrons had difficulty knowing how to use the call number to find the book on the shelf. library staff realized that the logic of the call number lent itself to a computer algorithm to help locate sets of call numbers, and that linking call number maps of the library’s stacks from the online catalog would be an effective mobile service to offer patrons. given any call number of a book in sterling memorial library at yul, a map will be displayed which highlights that call number’s general area on a floor in the stacks. yul introduced the mapping application in yufind, a catalog in place at yale since 2008 which is based on the vufind catalog developed at villanova [1].  the maps are linked from each record in the yufind catalog and the legacy voyager catalog (called orbis), and also from an ipad kiosk, where a patron can enter a call number and access the same maps. the mapping applications have helped reduce the number of questions about finding call numbers at the sterling memorial library information desk by 20.6%. this article describes how maps were designed, the logic behind the application, use of yul’s classification system, and how the algorithm was built using a web application that accepts restful urls and an sql database that stores relational data about the book locations, call numbers and maps. background mobile technology mobile technology has become ubiquitous in the college-age population: according to some surveys, close to 100% of undergraduate students worldwide use at least one smart phone or other mobile device such as the ipad, with the highest use occurring in north america. [2] [3] given the popularity and growth of mobile and smart phone technology, universities will increasingly make a priority to incorporate mobile computing into academic life(educause horizon report 2011). efforts to simply put web pages onto a mobile device do not work well, as text or image-heavy pages may not make sense on a mobile device, and some services are not well-suited to mobile access. mapping applications do work well for mobile computing, as a handheld device and an online interactive map are a great way to find your way in unfamiliar surroundings. it is therefore not surprising that maps have been used widely on smart phones for years. setting: at yul reference transactions revealed that patrons had a need for directional information in the library, especially to find books. in fall 2010, 7.3% of all questions at the sterling info desk dealt with finding a particular call number. a number of factors contributed to the problem: signage was poor or nonexistent, some parts of the collection were tucked away in small reading rooms, and two call number systems (library of congress and yale classification) added to general confusion. a patron who found a record in the online catalog would know the library (yale has eighteen) and the call number but no information that would help the patron decode the call number to find it on the shelf (there are 15 floors of stacks in the largest library). for the reasons discussed above, library staff knew that maps would be a good application to present both on the regular web and on mobile devices, and the library’s yufind catalog seemed a perfect place to experiment with linked maps. library classification: books at yale university library (yul) are mainly arranged on shelves according to the library of congress (lc) classification system.  lc classification builds on the work of melvyl dewey, who created a system using base ten mathematics to cover all the world’s ideas. he first divided knowledge into ten general classes, and then each of those into ten more classes, giving one hundred total classes and so on, always subdividing by ten, as described in the 1876 work a classification and subject index for cataloguing and arranging books and pamphlets of a library [4]. dewey extolled the simplicity of his system, and also claimed that it would allow books in a topic to be generally kept together in a library, aiding any patron wishing to discover material by simply browsing a section in the library. the library of congress created a different system using letters instead of numbers to signify broad topical areas [5]. for example in lc classification, b denotes material in philosophy, psychology, and religion, while both e and f cover works in american history. lc is north american-centric (hence two categories for history of americas, but one class d for all the rest of the world). all of science is in one category q, but military and naval science each has its own class, u and v respectively. there is a well-developed literature on the strengths, weaknesses and cultural biases of various classification schemes [6] and a full discussion is outside the scope of this article. in addition to lc classification, a yul-developed yale classification system was used until 1970. while no new books are given a call number using  yale classification, older materials remain in the collection with their original yale class call numbers. in addition the large yul system involves many buildings and special collection locations which must be evaluated along with call numbers to determine actual locations. even taking into account the two classification schema and numerous yul  locations it is still fairly straightforward to follow a set of rules in sequential order to decode any call number for a book in the yale library system and determine its general physical location. further these rules are easily translated into a computer algorithm. this ability was used in the project described here to help the patron move from a call number in the online catalog, to the correct map of that books floor location, and to the appropriate section of that floor, all through the use of its assigned call number (both lc and yale). building the application a project was started with these goals: produce physical and online maps for every stack floor in the sterling memorial library (sml), 15 floors total. provide links from every yufind record to the correct map in sml or to a general map of separate library locations external to sml. highlight the general area of a call number on each floor map. build an ipad kiosk where a user could enter a call number and find the correct map. develop the framework to accommodate, at least at a high-level, other classification and shelfmark schemes used in the yale system, including sudocs, hicks and accession number-based collection items. maps and supporting tables a graduate student in art history was hired to create maps for every floor in the library. the implementation team worked from maps used by staff which indicated call number ranges in the stacks. early in the project a decision needed to be made about how precise the maps should be in depicting call number ranges. a very precise approach would have tried to locate the specific shelf holding a book. a very imprecise but simple approach would have shown the floor for the book with no details about call number sections within the floor. a medium approach was chosen for this project, in which fairly large sections within a floor are shown with their call number ranges. the fifteen stack floors in sterling were produced in adobe illustrator, with each stack range included, along with elevators, bathrooms, computers, and exits. each level has a unique color associated with it, and that color is used on the print map and the directory for that floor. the same maps used online are posted on each floor in at least two areas near main elevators. this provides a consistent experience for the patron as they move from online to orienting themselves in the physical space. in practice only information found before the period or decimal point in call numbers were used in creating the maps and linking algorithm. in some cases only the first letter was used along with information about whether a book is oversize or not (indicated by a plus sign). in the case of the floor 5m (figure 1), only the first letter is used because relatively small numbers of books are found in these ranges, so a single range may have material for ba, bc, bd and so on. when the letter part changed so frequently there seemed little point in trying to show these changes on a map, and the relative locations would frequently change due to changes in the collection. thus not much information is used beyond that the floor is 5m and the location of “a” call numbers versus “b” call numbers. figure 1. floor 5m in sterling memorial library showing very large blocks of call number ranges. the plus sign (+) indicates oversize. maps for other floors contain more detailed information and use more than the first letter of call numbers. a more typical map is shown in figure 2. this map provides more information within the p class, using the second letter following the first p. by using large chunks of areas the maps do no not need to be updated frequently when small shifts occur among the shelves (as material is added or moved). the “chunking” approach aims to get the patron close enough to use the call number guides found on the end of every range of shelves. figure 2. map for floor 1mb shows blocks for smaller call number ranges, with the range ps highlighted. sql tables contain information for each map necessary for the correct functioning of the online application. in the stackmap table every call number box is assigned an arbitrary identifying number and the file name and location for the corresponding floor map image. for example in the map in figure 2, there are seven boxes, and the pt-pz box might be internally labeled number 2. that box 2 is then linked to all the call number ranges in it, from the “smallest” call number to the largest thusly: pt0000 to pt99999, pu0000 to pu99999, pv0000 to pv0000, pw0000 to pw9999, px0000 to px9999, py0000 to py9999 and pz0000 to pz9999, even though there may not be any books currently in some of these ranges, every eventual possibility is covered. numbers and letters after the decimal are ignored. when a call number is within a box, the block should be highlighted. to do this each block is broken into rectangles, and the coordinates of the upper left and lower right part of each rectangle in the box are also recorded in a separate table, stackmap_coordinate. there are some blocks that require more than one rectangle to be drawn because the stacks for that call number range are l shaped or spread out in different areas of a floor. this is why the stackmap table does not store coordinates since there is a one-to-many relationship with the stackmap_coordinates table. with this information any call number of a book in sterling memorial library can be matched to the appropriate map and box, and that map will display highlighting the correct box. for books in other libraries, two other sql tables are needed. one stores all the individual location codes used in the library catalog, e.g., “beinecke (non-circulating)” paired with the library that contains that location, e.g., “beinecke”. the second table links libraries to urls for those libraries. generally these pages list the address of the library, hours of operation and contact information, as in the example in figure 3. figure 3. the page linked for call numbers of items in the beinecke rare book & manuscript library, a library without open stacks and no associated stack maps. the algorithm the application that selects the correct map location given a call number consists of two parts: a web application that accepts restful urls and an sql database that stores some relational data. the web application was written in c# and consists of a single class containing the algorithm. the main file, default.aspx displays an image gif file showing the map of the floor the item is on and css is used to place a div over the map which highlights (25% opacity) the matching call number box. the map image is still fully visible under the highlighting. the sql database uses four relational tables (explained above in maps and supporting tables) that store information about which gif image to use, which areas to highlight and in cases where the location is not associated with a map, a url to provide more information about the library location where the materials are held. two key arguments are needed in the web application: the location code of the item from the opac and the call number. the url is received and the arguments are processed by the algorithm. there are just a few vital pieces of information needed to make the application work. first, it must determine whether the call number uses yale or lc classification. there are some cases in which the yale and lc classifications overlap making it impossible to tell them apart. additionally many classifications are held in various libraries at yale. luckily this information is included in the location code; and therefore this code is examined first. if the location fails to identify the classification, then regular expressions are used to determine it. the algorithm checks for special cases such as oversized items and folios, which may be indicated by special characters or keywords in the call number. with the classification scheme, oversize information, and the first letters and numbers of the call number it is possible to determine which map the call number belongs to. at this point most of the work is sent to the sql tables to figure out where the system thinks the item is located. exceptions to be handled include temporary shelving of materials, e.g., course reserve items or other special circumstances, which can be determined by the location code. the first query run will return a list of library locations, floors and the map sections for where the item is located. for example the following may be returned for an item: sml, 1mb, 1. where sml is the building code for the library, 1mb is the floor and 1 is the boxed section of the map containing the call number. the first two set the name of the map file to be used. in this case the file is named sml_1mb.gif. the box number 1 indicates the area that will be highlighted on the map. the highlighted boxes are drawn using html div tags with absolute css placement. the stackmap_coordinate sql table links a map section id with the top, left, width and height of the div(s) to be drawn. when the call number lookup fails to find an associated map to display or when a map exists but the call number range does not fall onto the map stored, the patron is redirected to a url with additional information. in the worst case, where the location is not recognized, the patron will be redirected to the entire list of all yale library locations. putting call number maps into yufind and orbis the project team needed to decide where in the yufind catalog links to maps would be displayed. at the title display level, maps could not be displayed because a single title may be held in several locations. the more logical place was to display the link to maps was from the item level, where only one location is possible. once this was determined it was simply a matter of adding code to the yufind bibliographic display template to build a call to the mapping web service. this meant building a url that contained the location name and the call number, properly encoded. we experienced a small glitch due to the different ways that solaris/php yufind code encoded the url and the way that the windows/.net web service decoded the url on the receiving end. once we identified this problem, we were able to code around it. figure 4. an item level record in yufind with the call number and link to the associated map. once the implementation was done in yufind and had been well tested, it was simple enough to add the same link to the library’s legacy voyager online public access catalog (opac). because there was no logic required on the opac side and the information needed by the web service was included in the bibliographic display, it was easy to replicate the simple link created in yufind. after initial testing, we decided to also pass the unique record number of the item to the mapping service so that we can display a link back to the catalog. this allows patrons to request staff paging of an item if they decide not to follow the map and retrieve the book themselves. the web service call, including location, callnumber and bibid parameters, looks like this: http://resources.library.yale.edu/map/?location=sml,%20stacks,%20yale%20classification&callnumber=%20ix%20c591%20884i&bibid=5415792 note the simple urlencode of the existing human-readable location code and call number. because the web service handles these parameters, deployment to different catalog systems is simplified. we chose to have the map display in a new window for the basic rollout for both yufind and orbis, but as we move to a more mobile-optimized display, we will revisit the exact mechanics of the display. ipad kiosk finally, to create the kiosk for patrons who had written down a location and call number and were in the sterling memorial library but weren’t sure which floor to go to, we coded a simple web form. since the web service depends on perfectly formatted location names for choosing between the yale and lc class schemes, we created a drop-down that requires a patron to identify the scheme, based on the location they have written down. patrons must enter “at least the first letters and numbers” of the call number in a text entry box, and this completes the form. the lookup is available at http://www.library.yale.edu/m/call.html figure 5. call number look up page http://www.library.yale.edu/m/call.html. results in the fall semester 2010, 7.3% of all questions at the information desk at sterling memorial library were questions relating to finding a specific call number. after the mapping project was fully implemented in fall 2011 the number of call number related questions was reduced to 5.8% of all questions at the same desk, a 20.6% reduction in patron inquiries on this topic. more publicity for the mapping functionality and the ipad kiosk should further help reduce the number of requests. based solely on the reduction in requests for help finding call numbers in sterling memorial library the improved mapping can be judged successful and worth replicating elsewhere in the yale library system. future directions the first extension of this project will be to map all collection areas in the yale library and provide location links for all of them in the library’s catalog. future development will include using the same lookup for other types of resources such as reading rooms, special collections and restrooms outside of the stacks areas. the call number maps will also be used to help students find work carrels near their subject areas, something students have told us they want. finally, staff are interested in reversing this project, providing qr codes near call number ranges which will lead patrons to virtual shelf browsing in the catalog, another feature patrons have requested. this project has opened new possibilities in exploiting data which help to link the online world and the physical in new and interesting ways and we imagine that more ways to use this functionality will become apparent as we investigate mobile technologies. code view the code. notes [1] houser j. 2009. the vufind implementation at villanova university. library hi tech 27(1):93-105. [2] cummings j, merrill a, borrelli s. 2010. the use of handheld mobile devices: their impact and implications for library services. library hi tech 28(1):22-40. [3] gsm. 2011. mobile phone usage report 2011: the things you do. http://www.gsmarena.com/mobile_phone_usage_survey-review-592.php [4] dewey m, comaromi jp, beall j, matthews we, new gr, forest press. 1989. dewey decimal classification and relative index. albany, n.y.: forest press, a division of oclc online computer library center. [5] taylor ag. 1999. the organization of information. englewood, colo.: libraries unlimited. [6] mai je 2010. classification in a social world: bias and trust. journal of documentation 66(5):627-642. about the authors kathleen bauer is the director of usability and assessment at yale university library, a department she created. she is currently a doctoral student at simmons college, has an ms in mathematics from rensselaer polytechnic institute and a ba from mount holyoke college. for more information about usability and assessment at yale see http://www.library.yale.edu/usability. michael friscia is the manager of digital library and programming services at yale university library. his department focuses on the library website, interfaces and digital asset management. michael is the lead programmer for ladybird, a new tool at yale university library that’s being developed to centralize digitization workflows, integrate with digital preservation systems and route digital objects to discovery interfaces. he can be contacted at michael.friscia@yale.edu. scott matheson is librarian for digital resources at yale law library. he has experience in public services and in library it where he was web manager at yale university library. he has a mlis and a jd from the university of washington. he can be contacted at scott.matheson@yale.edu. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – managing discovery problems with user experience in mind mission editorial committee process and structure code4lib issue 44, 2019-05-06 managing discovery problems with user experience in mind williams libraries recently developed a system for users to report problems they included while using the library catalog/discovery layer (primo). building on a method created by the orbis cascade alliance, we built a google form that allows users to report problems connecting to full text (or any other issue) and automatically includes the permalink in their response. we soon realized that we could improve the user experience by automatically forwarding these reports into our ask a librarian email service (libanswers) so we could offer alternative solutions while we worked on fixing the initial issue. the article will include an explanation of the process, reactions from public service staff, methods for managing the problems once submitted, and code shared on github for those interested in implementing the tool at their own library. by emery shriver introduction first, a fairy tale: once upon a time (july 2016), williams libraries launched a new ils and discovery layer. the new system, simply called “the library catalog,” was our first web-scale discovery tool. its debut heralded the beginning of a new era of streamlined discovery. users would no longer have to endure the archaic “search here for articles and search here for books” searching method. they could harness the power of the “next-gen” discovery system to search for almost anything, and be seamlessly connected to that perfect, peer-reviewed article that they never knew they always wanted. when the system went live, complete with a “tell us what you think!” link, we expected the glowing reviews to pour in, and then we would all live happily ever after. ok, that’s the fairy tale version. what actually happened is that feedback came trickling in, and most of the submissions were reports of problems connecting to full-text electronic resources. although many users were experiencing streamlined discovery and seamless connection, some were instead getting dead links, paywalls, or error messages. as the newly-assigned person responsible for discovery, it was my job to troubleshoot the problems. working in a library with a culture of timely, user-centered service, it was also my job to respond quickly to users with either a solution to the problem or an alternative method of access (such as interlibrary loan). i also needed a way to triage and distribute the problems reported between myself and staff in other departments, and i had other job responsibilities to juggle. after two years and a few iterations, we now have an efficient e-resources troubleshooting workflow that foregrounds the user experience. version 1.0: tell us what you think before the discovery layer, we used a third-party link resolver to connect citations from databases to our full-text holdings. when there were problems connecting to the full-text, library staff reported them to an email address, which was monitored by the systems librarian and the head of collections. the email address was not publicized to users, and only a few problems were reported each month. after the launch of the new catalog, traffic coming through the “tell us what you think” feedback form, linked in the footer, continued at the same pace. in the first year, we received 70 entries, and 90% of them were from library staff. anecdotally, however, i was hearing from my public service colleagues that users were having trouble connecting to full-text electronic resources, but they weren’t using the feedback form to report them. figure 1. screenshot of the “tell us what you think” form link. i did some impromptu usability testing with some of our student employees, and it was clear that they didn’t notice the feedback link, and they also didn’t realize it could be used to report a problem. moreover, once they landed on the feedback form, there were too many questions, and they needed to go back to the catalog record to get the url of the problematic item, which was cumbersome. it was likely that users who were using this form were already a bit frustrated, so we needed to make the problem reporting process quicker and easier. version 2.0: report a problem in august 2017, we moved to a new user interface (ui) offered by our discovery vendor. the out-of-the-box (otb) version of this new ui did not include a footer by default, so we had to move our feedback form link elsewhere. this forced change was also an opportunity: i could use some of the data i had gathered from our student workers to improve the feedback experience on the user side. at the same time, we also needed to update other custom add-ons to be compatible with the new ui, which was built using the angularjs framework. while researching what other libraries who had already moved to the new ui had done, i discovered the online primo toolkit [1] provided by the orbis cascade alliance. in a post called “add a ‘report a problem’ link in primo actions menu [2],” i found my solution to the feedback form dilemma. the post, which was designed for the classic ui, explained how to prepopulate a form with the record’s permalink. although the orbis cascade feedback form was drupal-based, a quick search of google’s documentation [3] told me how to prefill fields in a google form. changes made based on the feedback from our student workers, i knew version 2.0 needed to be shorter. in addition, my experience receiving the form responses had made it clear that a form combining access problems with general catalog feedback was confusing and difficult to manage. with some input from our web advisory group, i split the feedback gathering process into two forms: one to gather general feedback about the catalog, and another to report access problems with specific records. i used part of the original “tell us what you think” form for the general feedback form, which is linked from the catalog landing page, and i created a new “report a problem” form based on the orbis cascade post. the new form had four questions: please tell us what’s happening so we can fix it: permalink (pre-filled from catalog record) your email are you library staff? i included the last question so i knew the time expectations for getting back to the reporter. we typically try to respond to users from outside of the library within one day. library staff know that someone is working on the problem, so they don’t expect an answer until it has been resolved. this new form was added to each catalog record, under what our catalog vendor calls the “actions menu.” figure 2. screenshot of “report a problem” form link. i added two custom scripts to the google sheet that held the form submissions. one forwarded the reported problem to the abovementioned email list, a holdover from our old ils, which was now monitored by the systems librarian, the head of collections, and the catalog librarian responsible for electronic resources. the second script constructed an email to the problem reporter to let them know when the problem was resolved. both scripts were triggered manually as part of the triage process. in addition to these semi-automated emails, i was also emailing problem reporters outside of the library at other points: when the problem report was received, if what was reported was user error, and to send them suggestions of what to do to fill their initial need while we were troubleshooting the issue. results a soft rollout of this service resulted in a dramatic increase in use. in the first 30 days, we received 35 submissions. over half of the submissions (17/35) were from users outside the library. this was a great problem to have (users were seeing the link and using it) but it was still a problem. the workload increased for those involved in troubleshooting. i felt like i needed to drop everything when a submission came in so i could get back to the problem reporter. most interestingly, users’ definition of a “problem” was much broader than before. students used the form when they couldn’t find a book on the shelf. faculty used the form to request changes to how their own books were represented in the catalog. i was happy that our users were using this form. after all, all feedback is useful feedback, right? on the other hand, in handling this variety of problems, i was doing a lot more “reference” work than troubleshooting work. i needed a way to make sure that users were being taken care of, but that also freed me up to actually fix the problems. version 2.1: forwarding to libanswers i examined the new workflow and realized that there were parts of the process that i didn’t need to be involved in. for example, i didn’t need to communicate with the reporters outside the library, thanking them for their submission and offering alternative ways of access. someone needed to be in contact with them, but that someone didn’t have to be me. furthermore, we already had a system in place for communicating with users: our ask a librarian service, staffed whenever the research help desk is open. better yet, this system, libanswers, had its own email address, and i had already written an email-building script. all i needed to do was make some slight modifications to automate it. changes made i copied and modified one of the manually triggered email scripts to forward problems to the ask a librarian service. every 15 minutes the script checks to see if any reports have come in from users outside the library since the last time the script ran. if a report has come in, it builds an email and sends it to our libanswers queue. the subject of the email is “catalog problem reported,” and the sender is the reporter who submitted the problem. the email includes a description of the issue and a permalink to the problem record. the reference librarian staffing the email queue when the report comes in does a little exploration, such as checking to see if they can recreate the problem, or finding a means of alternative access. they reply to the user with suggested next steps, and log the transaction in reference analytics. results automating and delegating the user communication part of the triage process was a significant improvement. i spent much less time communicating directly with users, and could focus on fixing the problems. users received help accomplishing what they set out to do before encountering the problem. the problems that are not related to full-text access could be easily forwarded to other library departments. finally, because transcripts of the transactions are logged in reference analytics (which was shared with all reference librarians and student employees) everyone had a better idea of the types of problems we were encountering, and their possible workarounds. examining the daily digest of reference transactions led me to add two additional questions to the form to improve the user experience: are you on campus or off campus? have you requested this article through interlibrary loan? the first question was added to help reference librarians in their initial troubleshooting, when trying to recreate the problem reported. the purpose of the second question was twofold: to remind the user that the interlibrary loan service exists and that they can use it to request articles that they are having problems accessing, and to prevent librarians from offering redundant advice if the user had already used the interlibrary loan service.seeing the trends in librarian answers to reported problems gave me a different perspective than what i was seeing in the form submissions, enabling me to make these small, iterative changes. there was, however, still a problem to tackle: the mountain of emails sent and received by me and my troubleshooting colleagues, known as the “e-problems group.” every time i forwarded a problem from the spreadsheet to the e-problems email list, it went to three people. it wasn’t clear which of the three people were working on the problem, and many times more than one person would start troubleshooting the same issue at the same time. because of the way i built the email script, every email had the same subject, so it was hard to know which problem was which, especially for those using gmail conversation mode. it was also hard to share notes among the group about what we had already tried. what we really needed was a more formal ticketing system to assign problems, add notes, and track trends. version 2.2: further forwarding to libanswers so far, i had created a reporting system that was quick and easy for people to use, and provided a user experience that fit our prompt and personalized service model. it was time to focus on improving the experience of the staff troubleshooters (myself included) who were struggling to manage the mountain of emails generated by the problem reporting system. once again, libanswers was the solution. the same email queue system that reference librarians were using to answer email reference queries (and respond to problem reporters) could be used as a ticketing system to organize the problems reported. it would allow us to assign and transfer tickets to individuals based on expertise, make notes about the issue and what we’d already tried so far, and add tags and gather data for analysis of trends. implementing this solution was fairly simple on my end. as a springshare administrator, i added another email queue (at a cost of $100). each libanswers queue has its own email address, so i just needed to change the email address in the version 2.0 script. since i was still triaging the problems, i didn’t need to set any triggers to automate this script. staff in the e-problems list had not used the email ticketing system in libanswers before, so i met with them to set up the queue and train them on how to use it. results moving the management of troubleshooting out of email and into libanswers has improved our efficiency. there was a small learning curve for staff not familiar with the system. after using it for a few weeks, we were able to settle on common practices that ensure we’re all using the system in the same way (e.g. when to “close” a ticket or when to leave it as “pending”). during that time, i also made some small adjustments to the email forwarding script so that it was easier to tell one problem from another in the queue. since it is an internal workflow with only three people involved, we can continue to make small, iterative changes when necessary. future plans after more than a year of using the “report a problem” feedback system, we’re generally happy with how the system works: we’re able to respond to users promptly, track problems efficiently, and we’re building a system that lets us keep track of problem trends and solutions. yet, there’s always room for improvement. my colleagues in research services have suggested that we create some canned responses in libanswers so we can provide quick, consistent solutions to frequent problems. syncing the google sheet with the e-problems queue could also be improved. currently, i’m using the google sheet to track problems and generate statistics, since not all problems reported get forwarded to the e-problems group, but that’s not very efficient. finally, it would be great to start being proactive in our troubleshooting, and set up a testing schedule, so we can try to identify and fix problems before our users encounter them. what we learned users are not used to receiving prompt, personalized responses when they use an online form to report a problem. many of our users are pleasantly surprised when they hear from us right away, and they thank us for the prompt reply. while this may be outside the norm for online support in general, it’s one of many ways we show our users that we appreciate them. when we rely on our users to help us identify problems in our catalog, we need to make it easy for them to do so. by putting them at the center of the troubleshooting process, we can continue to uphold our personalized service ethic, and ensure that users will continue to partner with us to make our catalog the best it can be. about the author emery shriver is the reference and web development librarian at williams college in williamstown, massachusetts. his work resides at the intersection between critical librarianship and user-centered services. he received his msi from the university of michigan school of information. the code can be found on github at https://github.com/emery-williams/reportproblem. references [1] primo toolkit. 2016-2019. eugene, or: orbis cascade alliance; [accessed 2019 march 8]. https://www.orbiscascade.org/blog/9/. [2] add a “report a problem” link in primo actions menu. 2016. eugene, or: orbis cascade alliance; [accessed 2019 march 8]. https://www.orbiscascade.org/blog/9/?bid=119. [3] send your form to people. google; [accessed 2019 march 8]. https://support.google.com/docs/answer/2839588?hl=en. further reading subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – tweeting tennessee’s collections: a case study of a digital collections twitterbot implementation mission editorial committee process and structure code4lib issue 48, 2020-05-11 tweeting tennessee’s collections: a case study of a digital collections twitterbot implementation this article demonstrates how a twitterbot can be used as an inclusive outreach initiative that breaks down the barriers between the web and the reading room to share materials with the public. these resources include postcards, music manuscripts, photographs, cartoons and any other digitized materials. once in place, twitterbots allow physical materials to converge with the technical and social space of the web. twitterbots are ideal for busy professionals because they allow librarians to make meaningful impressions on users without requiring a large time investment. this article covers the recent implementation of a digital collections bot (@utkdigcollbot) at the university of tennessee, knoxville (utk), and provides documentation and advice on how you might develop a bot to highlight materials at your own institution. by meredith l. hale literature review are bots effective? before launching into the process of making a twitterbot, some evidence on their effectiveness in creating meaningful impressions on patrons is warranted. existing literature indicates that bots, generally, are positively received and that they have had an impact specifically on library marketing and communication. in addition, bots are becoming more and more prevalent. as of 2017, it was estimated that 15% of twitter’s users were bots rather than humans (veale and cook 2018). a common initial reaction to proposing that library social media accounts consist of bot-created posts instead of solely human-created content is that automated content creation will not be as accepted or interesting for patrons. a study by edwards et al. disproves this assumption; people typically interact with bots in the same way that they communicate with accounts run manually by humans (edwards et al. 2014). its results align with the computers are social actors (casa) paradigm. in their study, edwards et al. had college students rate the credibility, interpersonal attraction, communication, and following likelihood of two twitter feeds, one generated by a bot and another by a human. participants were informed of the source of the feed upon viewing screenshots of each. both feeds appeared to be generated by the center for disease control (cdc) and shared content on stds. the authors note that using the cdc as their test feed might have resulted in higher credibility scores than normal and indicate that future research should focus on more socially-focused agents more typical of twitter. because library accounts are often associated with established academic, community, or cultural heritage institutions, this attribution of credibility might also be given to library twitterbots. ultimately the study concluded that twitterbots are viable substitutes for humans in transmitting credible information in a relatable way on social media. in considering bots and how they are perceived by the twittersphere, one also should recognize that human language itself is not always creative. just like with tweets from bots, “natural” human products often rely on socially constructed clichés and catchphrases. bots are made by humans and therefore project many human idiosyncrasies. veale and cook suggest that those skeptical of the creativity of twitterbots compare the language they tweet to conversations at parties and country clubs. formulaic tweets are also likened by the authors to the language used by children with chatterbox syndrome, which is characterized by the use of words that are not understood by their speaker for the sake of impressing others or socially passing (veale and cook 2018). like with bots, these children repeat phrases they have heard from others or learned from media, but often the context in which the phrases are delivered is not quite appropriate. these children are considered smart and likeable, despite the lack of creativity and social awareness in the words they share. these examples show that comparing human and automated content may make individuals reevaluate what it means for something to be “creative.” in this way, bots can be used to spur critical conversations. ultimately, the goal of making a twitterbot is not to see if it can pass as human, but to share content that can be trusted and spark ideas. as veale and cook state, “twitterbots are not fake humans, nor are they designed to fool other humans” (veale and cook 2018). in fact, many established standards guiding how bots are named and used make no attempt to conceal their automated nature. instead these characteristics are highlighted. for instance, it is typical protocol for a twitterbot’s handle to include the word “bot” and for the description to indicate that tweets are automated rather than individually curated by hand. identifying bots as bots actually makes users more comfortable about interacting with them (veale and cook 2018). twitter & cultural heritage institutions the examples given above provide evidence that twitterbots can make meaningful impressions on patrons, but this research and assessment does not include examples directly relevant to cultural heritage institutions. there are many instances of library and museum twitterbots that can be used to assess the effectiveness of this communication method and for inspiration in making your own. included is a brief list of some favorite bots tied to cultural heritage materials along with the creator and numbers of followers associated with each. in creating this list, the method was to find any and all twitterbots in existence that are associated with a gallery, library, archive, or museum (glam) that focus on sharing images from established collections on a regular basis. while sources like veale and cook’s bot index were consulted to find more instances of glam bots, ultimately the bots noted below came from my own efforts over the past year and a half to identify bots similar in function to @utkdigcollbot. other twitterbot listings focus on bots that meet different goals than those typically desired by glams. this table shows that it is possible to reach hundreds, thousands, or even occasionally tens of thousands of people with a twitterbot. at the very least, this means that having a twitterbot raises the likelihood that the public will encounter content from your institution. both @pomological, which shares images from the usda agricultural library, and @boschbot, which shares tiny sections of paintings by hieronymus bosch, are noteworthy bots in terms of their total followers. every bot listed creates posts that include an image and text, typically a title pulled from the metadata. because the bots essentially tweet out existing data, they are characterized as “feed bots” (veale and cook 2018). these bots are passive in nature, in that they do not react to other users on twitter. outside of the cultural heritage realm, there are “watcher bots” that actively respond to the tweets of public twitter users based on some criteria. for instance, @stealthmountain seeks out tweets with the phrase “sneak peak” and asks users if they’ve made a spelling mistake (e.g. “sneak peek”). in libraries and museums, it seems that this kind of interaction might be frowned upon as invasive as no instance of this type of bot in those environments exists. a 2012 study of academic library twitter use broadly found that very few libraries use the social media platform for two-way conversation (del bosque et al. 2012). beyond anecdotal evidence, there is no research showing why glams have limited direct interaction with followers, but twitter accounts managed by cultural heritage institutions routinely show the same distanced behavior. table 1. cultural heritage twitterbots listed alphabetically (march 2020). of those listed, @nyplemoji is the only bot that breaks away from the “feed bot” characterization. when users tweet an emoji to @nyplemoji, the bot returns an image from the new york public library’s collections that relate to the emoji initially sent. for instance, if a user tweets a calendar emoji, a photograph or drawing in nypl’s collections that depicts a calendar will be tweeted. while often simply a matter of visually matching, occasionally there’s less of a direct relationship between the visual content of the emoji and the resulting tweet. for example, in response to the “face with tears of joy” emoji, @nyplemoji sent a piece of sheet music titled “tears of joy and sorrow” (figure 1). as all of the emoji matches were hand coded, relationships like these are possible. a json file of emojis and their matching collection items can be found on lauren lampasone’s github page for the bot [1]. figure 1.tweet from @nyplemoji posted on 01-08-2020 although randomness is a key feature of many feed bots, posts by @nyplemoji are the result of a considerable amount of human control. beyond having a curated list of emoji-image matches, the code also includes a few exclusions. in the nypl-emoji-bot github repository a file named “blacklist.json” shares five emojis that the bot will not post a reply to. the list includes the gun, knife, bomb, no one under eighteen, and poop emojis. as long as content can be programmatically identified, it is possible to exclude sensitive or undesired content from a twitterbot. one interesting feature of twitterbots, supported by the table above, is that typically creators of twitterbots have no direct affiliation with the institutions whose content they are promoting. in most instances, this information on the creator is given in the twitterbot’s profile in order to be transparent. for instance, darius kazemi, perhaps better identified here as @tinysubversions, created two of the bots listed in table 1. ten of the bots were also created by john emerson, whose handle is @backspace. kazemi is a computer programmer and artist who previously organized the bot summit. emerson is the founder of backspace, a design consultancy, and he identifies himself an “activist, graphic designer, writer, and programmer” [2]. both individuals have strong interests in art and the public good and try to share the open access resources cultural heritage institutions make available. while individuals like these are disseminating images from major art museums, archives, and aggregators across the twittersphere, smaller institutions are not as likely to be represented. even when collections from an institution are represented in an aggregator bot, like @dplabot, this type of an account does not have the same focus as an institution-specific account. while kazemi, emerson, and others should be applauded for promoting (and having fun with) the digital content of glams, i would urge cultural heritage institutions to actively share their own digital materials. this gives institutions more control over what materials are being shared and how, while also making it easier to attract users through an institution-specific account that has targeted content. copyright and image sharing one of the major concerns most librarians have when dealing with digital collections is copyright. so, if you decide that you do want to invest the initial effort in creating a twitterbot, you may likely ask if you need to consider any legal issues. first, before posting any digital items online, the institution should have ascertained whether or not it has the right to do so. assuming this work has been done, creating a twitterbot that further promotes these materials should cause no problems. due to open graph tags [3], you can “share” an image on twitter without actually uploading any content to the site. the link you include from your digital repository will allow a preview of the image to appear on twitter. therefore, if you have the right to share the image on your local repository, you have the right to create a twitterbot that shares image previews through links. in a society focused more and more on “user-found” rather than “user-created” content (carpenter 2013), twitterbots are a legal source of information for users to both digest and retweet. method to create utk’s digital collections bot, i began by creating a new twitter account and activating developer access for it on the twitter developer website [4]. within twitter apps, you can retrieve all of the codes needed to interact with twitter’s api (consumer key, consumer secret, access token, and access token secret). i used pycharm as my ide for python and heroku for hosting. if you are interested in a less technical solution for a twitterbot at your institution, there are several options you can consider. tracery [5] and google sheets [6] are two options that have extensive documentation already present online. in terms of documentation, i personally benefited greatly from jeanette sewell’s presentation “the wonderful world of bots” given for an online amigos library services conference [7] as well as scott carlson’s “you should make a twitter bot for your g/l/a/m’s digital collection(s) [8]. in order for you to be successful in making a twitterbot, it is critical that certain technologies have been implemented for your digital collections. first and foremost, open graph tags must be present in the metadata online for each record page in order for an image to appear when you tweet a link. to check and see if you have open graph tags, simply go to “view page source” or test a link to a digital object in twitter’s card validator [9]. if you have open graph tags, you will see some of the following tags in your page source:,, etc. twitter’s card validator is also very simple to use to check this. simply insert a link to a digital object to preview what the tweet will look like. the validator also indicates how many metatags are present and reports if the twitter card can be loaded successfully. when first beginning this project, open graph tags were not present in utk’s islandora records. our digital initiatives department helped add these tags using islandora’s social metatags module [10]. this addition was not only helpful to the twitterbot, but enhanced the use of digital collections links across our social media outlets through the inclusion of more images. another technology that i relied heavily upon is the open archives initiative protocol for metadata harvesting (oai-pmh). in order to avoid having to manually add links and titles to a spreadsheet, you should find an automated way of generating this information. for institutions that contribute to the digital public library of america (dpla), the organization’s api [11] is a great alternative to oai-pmh. visit @dpl_eh (a canadian twitterbot) to see how this can be executed. code is also present online in github for this implementation [12]. following the precedent set by many of the cultural heritage feed bots mentioned in the previous section, i decided to keep the output of the twitterbot simple and focused on highlighting digital collections content. each tweet included the title taken from the metadata followed by the “find it at:” with the link. because open graph tags have been implemented, the image shows rather than the link text. below is an example tweet that demonstrates the daily product the twitterbot shares (figure 2). the complete code for @utkdigcollbot can be found on github [13]. figure 2. example tweet from @utkdigcollbot. challenges while there is significant documentation in existence to help guide the creation of bots, several issues arose in my work that required novel solutions. in initially testing the bot, i found that some of the images i shared on twitter were appearing grainy when posted. by going to the webpage and opening “view page source” to look at the files directly, i discovered that these images were thumbnails rather than full jpegs. upon looking through the collections affected by this issue, i quickly discovered that this was due to the islandora content model selected for each collection. collections using islandora’s compound or book models had thumbnails present in their open graph tags while collections using the large image content model supported high quality image sharing. to address this, i altered my program so that it only selected random images from particular collections by including only a portion of our oai sets. as the libraries continue to add collections, i also need to update the code to include this oai set name so that this new content finds its way into the twitter feed. other twitterbots seem to face similar image quality issues, but not all decide that displaying high quality images is important. the feed for @dplabot for instance, includes many grainy images (figure 3), though this may be because the varied nature of its content from all dpla institutions made it impossible to troubleshoot this problem effectively. figure 3. fuzzy image from @dplabot reflecting the importance of considering image quality. being able to select particular collections to include in the twitterbot also made it possible for me to avoid collections with known metadata issues and discover new issues. throughout my time working at the university of tennessee, i have worked to remediate metadata for all of our collections following migration to islandora and a transition from following the dublin core schema to mods. through completing this project, i also became aware of new metadata issues that i needed to address in remediation. for instance, figure 2 demonstrates that there are metadata records in our collections that have no abstract or description value. because there is no value given in the mods metadata, no description value can be included in the open graph tag. this results in the text below the image stating, “no description given.” while the visual content of the image may be able to overcome this lack of metadata and still gain the interest of the user, posting objects without descriptions likely does not promote engagement and may reflect badly on the institution creating the records. since starting the twitterbot, i have pointed to some of the empty descriptions highlighted through the twitter posts as a way to advocate for spending more time to provide this information in metadata creation. even a blanket description used for each and every item in a collection provides users with more context than “no description given.” in addition, having the ability to select particular sets has also made it possible to exclude potentially sensitive or problematic content. like many institutions, the university of tennessee’s digital collections include racist content. for instance, within the university’s collection of sheet music, there are numerous illustrations of minstrelsy with degrading lyrics. photographs of blackface are also present. while these digital objects are made publicly available in the digital collections interface for research purposes, i felt strongly that highlighting individual works of this type without context on twitter would both portray the university in a negative light and not meet the expectations of bot followers. in the same way that @nyplemojibot forbids image results from certain emojis, i decided to not include any sets that had content which might be offensive. while i believe that this type of material needs to be preserved and shared as part of the historic record, it seemed unproductive to share these images on twitter without being able to provide additional information. the most challenging aspect of this project was figuring out how to host the bot so that it was truly automated and did not require any intervention. a simple method is to run your program on a computer that is always on, although posts may be interrupted due to computer maintenance. rather than relying on a spare computer or using library server space, i used heroku, a free cloud application program, to run my python program continuously. running a program periodically can be achieved in heroku by using either apscheduler or the heroku scheduler add-on [14]. heroku scheduler allows you to run jobs at pre-established intervals while apscheduler is more customized and allows you to specify any time or frequency for your job. while more complex than necessary for my needs, i am running utk’s digital collections bot using apscheduler so that it is more flexible in the future. another method for hosting you can explore is amazon web services (aws) lambda [15]. titles also posed an additional metadata-related obstacle to be overcome. initially after completing the code and implementing the twitterbot, there were times when no tweet would be posted for the day. looking through error messages in the log on heroku, it was determined that the code did not account for the possibility of multiple titles being returned. when additional handling was added to account for a list of titles being returned, as the metadata often contained both supplied and transcribed titles, the program successfully ran again. figure 4 shares the code that handles these possibilities. in the “get_title” function, the first “if” statement accounts for the metadata that contains only one title. in this case, the function will retrieve a key-value pair following the dictionary data type. the second “if” statement is included to find instances in which multiple strings are associated with the title field in the form of a list. only the first title in this list is retrieved, as is indicated by including an index of zero. finally, an else statement is included to deal with the possibility of the title data not conforming to either a dictionary or a list format. figure 4.handling multiple data types possible for title values. assessment at the writing of this article, utk’s digital collections bot had been regularly posting for nineteen months. in this time, the bot gathered a total 100 followers. twitter analytics were used to gather more granular information about engagement with the twitterbot [16]. a total of 46,197 impressions were collected, resulting in 838 engagements. an “impression” simply refers to the number of times users saw the tweet while an engagement records a more active interaction. on average, a single post resulted in 77 impressions and 1.5 engagements. the 838 engagements are composed of the following actions: 459 url clicks, 246 likes, 62 expand details clicks, 51 user profile clicks, and 19 retweets. these numbers show that the twitterbot brought 459 users to the libraries digital collections website. some of these users potentially were viewing the site for the first time. the most popular link was clicked a total of nine times and was associated with a sketch from the anna catherine wiley sketches collection titled “charcoal figure drawing of woman reclining” [17]. while the bot did gain interest from the twittersphere, active engagement through comments were completely lacking. a few retweets elicited written responses from users, but no comments were made directly on @utkdigcollbot. given that the objective for this bot was to increase the visibility of the university of tennessee’s digital collections without daily upkeep, not having to respond to comments can be seen as an asset. still, for those that want to foster an active conversation through twitter, a bot that simply posts materials may not meet your needs. the metrics reported above did meet expectations, which, admittedly, were somewhat low as i was unsure what effect the bot might have given the paucity of information on the engagement library twitterbots have produced at other institutions. discussions have been had in the past on ways to more fully integrate the twitterbot’s content with the libraries’ official handle (@utklibraries), but the planning and curating of tweets for this main feed has proven to be a bit incongruent with the random tweets of @utkdigcollbot. on occasion though, randomness and meaning have collided, which has made the tweets even more special. for instance, the twitterbot tweeted an image from the virginia p. moore collection the same day a colleague was giving a presentation on it. when this happens, retweets and smiles follow. conclusion this paper has demonstrated that creating a twitterbot is an achievable goal for librarians. while they can be technical undertakings, there are also lower barrier solutions that can be implemented using tracery or google sheets. currently cultural heritage twitterbots typically simply share content from cultural heritage institutions, but there are instances that demonstrate that glams can create more dynamic and interactive feeds. @nyplemoji is an example of a cultural heritage bot that both matches visual content and elicits active posting from followers. going forward, it would be helpful to better define what the engagement goals of twitterbots should be for a particular institution to determine if these creations are meeting their purpose. the fact that many cultural heritage bot creators have no direct association with the institutions whose content is being tweeted suggests that glams have yet to fully embrace this technology and that bots have potential to be a fruitful opportunity for outreach. it also indicates that for many the creation of a twitterbot is a fun technical experiment more akin to art than a part of a formalized social media campaign. and, as veale and cook’s writings advocate, twitterbots at their core will always have a bit of whimsy. now that you can make a twitterbot, it is important to seriously consider what it is you hope to achieve. acknowledgements considerable thanks are due to mark baggett, department head of digital initiatives at the university of tennessee at knoxville, for making @utkdigcollbot a reality. his guidance in coming up with a solution for hosting as well as his development of the code were essential contributions to this project. about the author meredith l. hale is the metadata librarian at the university of tennessee, knoxville. she is responsible for managing the creation and sharing of special collections metadata and contributing to the state’s dpla service hub and the digital library of tennessee (dltn) committee. notes [1] https://github.com/lolibrarian/nypl-emoji-bot [2] https://backspace.com/is/in/the/house/address.html [3] http://ogp.me/ [4] set up developer access at https://developer.twitter.com/en/apps [5] https://cheapbotsdonequick.com/ [6] http://www.zachwhalen.net/posts/how-to-make-a-twitter-bot-with-google-spreadsheets-version-04/ [7] https://www.amigos.org/node/4753 [8] http://www.scottcarlson.info/you-should-make-a-twitter-bot [9] https://cards-dev.twitter.com/validator [10] https://github.com/islandora-labs/islandora_social_metatags [11] https://pro.dp.la/developers/api-basics/ [12] https://github.com/ruebot/dpleh [13] https://github.com/mlhale7/utkdigcollbot [14] https://devcenter.heroku.com/articles/clock-processes-python [15] https://aws.amazon.com/lambda/ [16] https://analytics.twitter.com/ [17] https://digital.lib.utk.edu/collections/islandora/object/acwiley%3a322 bibliography api basics [internet]. dpla; [cited 2020 feb 6]. available from: https://pro.dp.la/developers/api-basics/ aws lambda [internet]. seattle (wa): amazon; [cited 2020 feb 5]. available from: https://aws.amazon.com/lambda/ buckenham g. cheap bots, done quick! [internet]. [cited 2020 feb 4]. available from: https://cheapbotsdonequick.com/ carlson s. you should make a twitter bot for your g/l/a/m’s digital collection(s) [internet]. [cited 2020 feb 6]. available from: http://www.scottcarlson.info/you-should-make-a-twitter-bot carpenter c. 2013. copyright infringement and the second generation of social media: why pinterest users should be protected from copyright infringement by the fair use defense. internet law 16.7:9-21. del bosque d, leif sa, skarl s. 2012. libraries atwitter: trends in academic library tweeting. reference services review 49(1): 165-183. edwards c, edwards a, spence pr, shelton ak. 2014. is that a bot running the social media feed? testing the differences in perceptions of communication quality for a human agent and a bot agent on twitter. computers in human behavior 33:372-376. emerson j. backspace [internet]. new york (ny): backspace; [cited 2020 feb 1]. available from: https://backspace.com/is/in/the/house/address.html hale m. utkdigcollbot [internet]. san francisco (ca): github; [updated 2020 march 10; cited 2020 march 10]. available from: https://github.com/mlhale7/utkdigcollbot lampasone l. nypl emoji bot [internet]. san francisco (ca): github; [updated 2018 july 24; cited 2020 march 12]. available from: https://github.com/lolibrarian/nypl-emoji-bot the open graph protocol [internet]. [cited 2020 feb 5]. available from: http://ogp.me/ sewell j. the wonderful world of bots: innovative ways to automate library marketing with metadata and twitter. paper presented at: amigos library services, innovating with metadata 2017. ruest n. dpleh [internet]. san francisco (ca): github; [updated 2017 jul 9; cited 2020 feb 2]. available from https://github.com/ruebot/dpleh scheduled jobs with custom clock processes in python with apscheduler [internet]. san francisco (ca): heroku; [updated 2018 aug 2; cited 2020 feb 6]. available from: https://devcenter.heroku.com/articles/clock-processes-python twitter card validator [internet]. san francisco (ca): twitter; [cited 2020 feb 3]. available from: https://cards-dev.twitter.com/validator twitter developer apps [internet]. san francisco (ca): twitter; [cited 2020 mar 9]. available from https://developer.twitter.com/en/apps veale t, cooke m. 2018. twitterbots: making machines that make meaning. cambridge (ma): mit press. p. 19-29, 351-352. weigel b. islandora social metatags [internet]. san francisco (ca): github; [updated 2019 mar 28; cited 2020 feb 4]. available from: https://github.com/islandora-labs/islandora_social_metatags whalen z. how to make a twitter bot with google spreadsheets (version 0.4) [internet]. fredericksburg (va): university of mary washington; [cited 2020 feb 2]. available from: http://www.zachwhalen.net/posts/how-to-make-a-twitter-bot-with-google-spreadsheets-version-04/ wiley ac. charcoal figure drawing of woman reclining [internet]. knoxville (tn): university of tennessee, knoxville. libraries; [cited 2020 feb 2]. available from: https://digital.lib.utk.edu/collections/islandora/object/acwiley%3a322 subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – editorial introduction: it is volunteers all the way down… mission editorial committee process and structure code4lib issue 20, 2013-04-17 editorial introduction: it is volunteers all the way down… by peter murray a well-known scientist (some say it was bertrand russell) once gave a public lecture on astronomy. he described how the earth orbits around the sun and how the sun, in turn, orbits around the center of a vast collection of stars called our galaxy. at the end of the lecture, a little […] by peter murray a well-known scientist (some say it was bertrand russell) once gave a public lecture on astronomy. he described how the earth orbits around the sun and how the sun, in turn, orbits around the center of a vast collection of stars called our galaxy. at the end of the lecture, a little old lady at the back of the room got up and said: “what you have told us is rubbish. the world is really a flat plate supported on the back of a giant tortoise.” the scientist gave a superior smile before replying, “what is the tortoise standing on?” “you’re very clever, young man, very clever,” said the old lady. “but it’s turtles all the way down!” –  stephen hawking’s 1988 book a brief history of time the code4lib community is wonderful and a wonder, as numerous conference keynote speakers and journal editors have noted.  it started first as a mailing list then quickly branched out to an irc channel, a website and wiki, an annual meeting with regional and international flair, this journal, and countless other opportunities to network, share ideas, and build on the work of each other.  as an active participant in the community, what i find even more fascinating is the sense of dedication that other like-minded people have for the community.  i can imagine that this is due in some part because their employer has an enlightened sense of the importance of community participation to give staff release time to engage in code4lib activities.  in other cases, the participants themselves cherish something of value and squirrel away unofficial time (perhaps evening hours, perhaps daytime activity under the broad heading of “professional development”) to use in community activities.  some have donated server space, others funds towards a domain registration.  a few years back we had a professional graphics designer volunteer time to create the code4lib logo and style guide. some make a formal expression of their volunteer contribution, like the group that is looking at conference options, the journal editorial board, and the local conference arrangement communities through the years.  others are more informal and ad hoc, like subscribers to the code4libcon mailing list that guides conference activities or the journal’s general discussion list.  some are short term, like those that volunteer for conference activities, or long term, like commitments to run the zoia irc bot. and it is not only our own volunteers.  some of the community’s web servers are hosted by the oregon state university open source lab at no cost to us.  pdpc, the home of the freenode irc network, recently announced that it was formally disbanding as an organization, but the freenode network will continue — run by its volunteers. if you step back and look at it, i hope you can’t help but be amazed at what has been accomplished by “rough consensus and running code” backed by the volunteers that make it happen — even as i have undoubtedly forgotten to list above major community projects that deserve mention.  (hey, the comments are open below — add your favorite notable code4lib volunteer activity!) i wonder, though, if we can continue to scale up to more participants and more activities without some sort of guiding structure.  for instance, an anti-harassment policy was created after one of the members of the community expressed the concern that “even in a wonderful community where we mostly share the same values, not everyone has the same definitions of acceptable behavior.”  a link to the code of conduct was placed on the 2013 conference home page and was highlighted during the conference opening announcements.  but other than the “rough consensus and running code” nature of the policy, is there any assurance that future conferences will use it or a way to appeal decisions made by meeting organizers? also related to the conference, we left the meeting this year without anyone stepping up to host the meeting next year.  in the aftermath of the 2013 meeting, at least two groups began efforts to create a hosting proposal and the infrastructure required to host a meeting.  one put out a proposal and requested a decision deadline to meet commitments for venue space; another was not quite so far along and needed time to finish their proposal. since we were outside of precedent for making a decision for a meeting host beyond the point of the current conference, we started making stuff up out of the blue.  was the meeting hosting to be given to the group that announced their proposal first?  what was an acceptable period of time wait for other proposals to be announced before starting a vote?  we worked out an acceptable timeline in the end, but i wonder about the next time. i think these are two examples of the difficulties of making decisions as we have grown.  (and we do continue to grow…over half of the conference attendees each year i’ve participated have come to their first meeting.)  is it time for the community to empower a group to make decisions on behalf of the community?  to have a transparent, accountable, and established procedure for addressing issues when “rough consensus and running code” isn’t good enough? i’ll admit to being somewhat enamored with the apache foundation process and how it works.  it is a merit based process (“government by merit”) where leaders are drawn from various parts of the community based on their commitment to the community and demonstrated ability to work with others in the best interests of the community (my paraphrasing of the apache governance model). apache is an irs-recognized non-profit corporation — that kind of independent foundation structure is certainly overkill for our community’s needs and i’m certainly not advocating that we would go that far.  but if we were to find a way to identify and agree on meritorious members of the community and grant them with the responsibility and trust to facilitate key decisions for the community, i think we will be able to ease some of the growing pains that we have been experiencing.  at the very least, it gives the community a process through which to make key decisions in a timely and transparent manner. summary of issue #20 the code4lib journal editors are excited to bring you this latest issue with seven articles.  the first set of articles show ways to manipulate metadata records.  in workflow tools for digital curation andrew james weidner and daniel gelaw alemneh describe how they use autohotkey and selenium ide at the university of north texas to automate various aspects of manipulating digital objects.  heidi frank show how to process marc records from archivists toolkit in augmenting the cataloger’s bag of tricks; the techniques – using marcedit, python, and pymarc – are transferrable to other sources of records as well.  in keeping up with ebooks kathryn lybarger introduces a tool for updating batches of vendor-supplied records through a set of normalization routines. jason clark show how montana state university is using youtube as a digital video platform in developing a digital video library with the youtube data api. getting users what they want without extraneous hits is always a challenge, and in better search through query expansion using controlled vocabularies and apache solr demonstrates how to configure solr to make the best use of a hierarchical controlled vocabulary.  in breaking up with contentdm heather gilbert and tyler mobley lead us through the migration of a repository to fedora commons using drupal, blacklight and rutgers’ openwms software.  and for the hardware geeks, tim ribaric and jonathan younker describe how to build a simple desk counter tied to a google spreadsheet in arduino-enabled patron interaction counting. finally, i am using this space to extend my gratitude to the code4lib journal editorial committee for their patience in guiding me through my first experience as coordinating editor for this issue. there is a lot of caring that goes into each issue — more than i appreciated from the outside looking in. they are just a few among the many others that step up in our community to make code4lib the useful, vibrant, and engaging place that it is today. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – docker: a software as a service, operating system-level virtualization framework mission editorial committee process and structure code4lib issue 25, 2014-07-21 docker: a software as a service, operating system-level virtualization framework docker is a relatively new method of virtualization available natively for 64-bit linux. compared to more traditional virtualization techniques, docker is lighter on system resources, offers a git-like system of commits and tags, and can be scaled from your laptop to the cloud. by john fink, digital scholarship librarian, mcmaster university introduction if you were working in library it in the last millennium, you’ll likely remember what your server room used to look like. pc towers running novell netware attached to huge multi-disc cdrom arrays. refrigerator-sized sun boxes. digital alphaservers running your library catalogue. to run most of the serious business of libraries, you needed serious equipment, and, as a result, machine rooms were often jumbled messes of shelves, wires and air conditioning units. with the advent of linux running on microcomputers, these rooms became slightly smaller and maybe slightly less complex, but it wasn’t until the early 2000s that the real sea change came for the server room — the rapid adoption of easily implementable virtualization [1], which allowed running multiple discrete operating systems on a single machine. although virtualization in the modern sense actually happened as early as the 1960s with vms for ibm system/360 machines [2], and both the 286 and 386 chips contained some species of virtualization [3], it wasn’t until 2001 when vmware introduced its x86 virtualization products that linux-based virtualization really took off. with technologies like kvm [4] and xen [5] looming large for local deployment and products like amazon web services [6] and the openstack [7] framework claiming the cloud, the jumbled, messy, white-box-pc-and-cdrom-tower-laden server room is more or less extinct; today, you can create multiple virtual machines out of minimal physical machines. virsh # list id name state --------------------------------- 1 trotsky running 2 eris running 3 funhouse running 4 gorgar running 5 shoah running 6 balder2 running 7 asgard running a listing of virtual machines on a single server at the author’s workplace. virtualization in libraries modern virtualization needs for libraries vary greatly from institution to institution. a small to midrange it shop that doesn’t do a whole lot of software development — as what you’d find traditionally in many libraries — might only require a handful of virtual machines. it may need one for the library’s web presence, one for the ils, and one to control images for patron workstations, for example. as much as possible, these would be drop-in replacements for those old white boxes. some instances, like for a given ils, would be bound by convention or licensing to have as little deviation from a traditional install as possible. for others, the effort to customize an install to a virtualization framework might not be worth the payoff; gains in performance or convenience may very well be minimal. libraries with staff who develop software likely have more complex requirements. the modern development environment is one of constant iteration. write code. write tests. code breaks. write more code. write more tests. break more things. imagine manually having to set up a new machine, even a virtual one, each time you wanted to test your code from scratch. anything that makes this process more efficient lets developers spend more time doing the specific work that they’re good at. the popular virtualbox [8] software, for instance, is very heavily gui-oriented [9] and has therefore had an entire framework [10] built around it designed to make the creating and destroying of virtualbox vms scriptable. typical library virtualization schemes mostly utilize machine-level virtualization. the systems already mentioned — kvm, vmware, xen, virtualbox, along with products like dosbox [11], a multiplatform emulator specifically written to run dos games — are machine-level emulators. they attempt to emulate as much as possible about a computing environment in software, down to disk drives, ram allocation, graphics, hard drive space, and even processor type. some machine-level virtualization platforms, for instance, make it possible to emulate an arm-based system like a raspberry pi [12] on an intel-based platform like a desktop pc. but this type of virtualization can be resource-intensive and inefficient. when running multiple virtual machines on a single computer, you might quickly run into the limits of the physical machine by allocating large disk drives or carving out multiple gigabytes of dedicated ram for each vm instance. workarounds for some of these problems are relatively simple (like mounting external drives as network shares), but others are more difficult (such as dealing with limited ram). enter operating system-level virtualization [13], which tries to share resources amongst instances rather than emulate as much of an actual machine as possible. typical operating system-level virtualization schemes will share ram, disk space, and kernel with guest instances. consequently, an arbitrary number of operating system-level virtualization instances will be less likely to run out of host system resources than an equivalent number of machine-level virtualization instances. but that flexibility comes at a cost. because guest instances must share a kernel and therefore both a processor and operating system type [14], you could not run that virtual raspberry pi on x86 or windows under a linux host. despite these limitations, operating system level virtualization is emerging as a very attractive workflow option for development work due to its ease of deployment and lightweight nature. virtualization with docker docker [15], an open-source implementation of operating system-level virtualization, is rapidly gaining mindshare amongst developers. like most good open source projects, docker incorporates a lot of existing linux technologies along with new functionality; it uses already existing technologies like copy-on-write union filesystems (usually aufs [16]) and linux containers [17], and it couples that with a number of features that make it developer-centric (and therefore distinct from traditional virtual machines that attempt to hew as much as possible to the metaphor of machine): like deployment portability, versioning, re-use, and repeatability [18]. these features are worth considering. they transform the notion of virtual machine provisioning from a time-consuming, sysadmin-centric model to one focused more on a developer-oriented workflow, [19] as evidenced in particular by the git [20]-like nature of docker’s versioning system (with its diffs and tags). at its core, docker is a virtualization framework focused around running applications and not around emulating hardware, which seems facile at first but underscores the critical difference between operating system-level virtualization software like docker and machine-level virtualization. machine-level vms are about faithful recreation of hardware — right down to ram allotment, how many cpus to assign, emulating nics, and so forth — and operating system level virtualization is about applications, not machines [21]. most of the time when we’re developing, testing, and releasing software we care about applications and not really the specific hardware environment — real or virtual — that we’re developing in, perhaps except with emulating historical hardware [22] and other edge cases. when we write to mailing lists asking for help with drupal, we don’t say “i have a dell poweredge 5100 with 4 intel core2duo processors and 16gb of 70ns ram and two atheros 100gb nic cards and i can’t get this drupal module to work correctly.” unless we have a pretty strong indicator that our problem is hardware bound, we focus on software. as it happens, this is what docker does also. there is usually no reason to try to define for docker how much memory it should have, how large its hard drive should be, or how much cpu it should take up; you don’t generally do this for the code you write on your desktop either. in practice, this means a computer running docker can run many more simultaneous virtual instances than the same computer running typical virtual machines. to illustrate: while wrestling with machine-level virtualization on an anemic desktop dating from 2007, i had trouble running two virtualbox instances concurrently. but, on the same host, i was able to run over 40 docker containers without a noticeable drop in host system performance. running docker what does it look like to run docker? after the docker software is installed [23], you’re left with a primary binary (“docker”) with which you can start, stop, import, export, and do other [24] operations. here’s an example of a docker host running a few containers, which we can see by running docker ps on the comand line. container id image command created status ports names 1bc191f4cdbb bedework:latest supervisord -n 11 days ago up 11 days 0.0.0.0:8080->8080/tcp sick_newton b58946da298c papyrus-demo:port6000 /bin/bash 13 days ago up 13 days 3000/tcp, 6000/tcp, 0.0.0.0:6001->6001/tcp drunk_bell c90c0a6be88f saucy-csclub:latest /bin/bash 13 days ago up 13 days 0.0.0.0:9999->9999/tcp angry_shockley e5a0f8a71f7e papyrus-demo:in-process /start.sh 2 weeks ago up 13 days 0.0.0.0:3000->3000/tcp mad_poincare f752161937c6 ldap_update_pw:latest supervisord -n 5 weeks ago up 13 days 0.0.0.0:5000->5000/tcp distracted_nobel 33cf9eb89073 catmandu:in-process /bin/bash 6 weeks ago up 13 days cranky_mccarthy individual docker instances are split up into images and containers. containers run instances of images. you can have several containers that come from the same image or variants of that image. in the above list, we can see that container id b58946da298c and e5a0f8a71f7e are running versions of the image “papyrus-demo,” demonstrating that images can have tags that act similarly to git tags, representing different states of a common image. in the papyrus-demo’s image, there’s a tag “in-process” and a tag “port6000”; an image without a distinct tag is always “latest”. a simple example: article-as-container to demonstrate how docker works, i’ve created a container that converts the markdown that this article was originally written in into html via pandoc [25] and serves it using the python simplehttpserver module. you can get the docker container from the github repository [26] for this article. it’s a very simple docker container and is built with two components. the first of which, the dockerfile, runs when the image is first built and the second, start.sh, runs at each instantiation of image into container. the article text is in c4l-docker-article.md, and it’s this file that gets converted by pandoc and served with simplehttpserver in start.sh. let’s break down the components and how to run them. dockerfile we’ll start by looking at the dockerfile line by line. from ubuntu:latest this is the image to use as a base. in this case, it’s building from docker’s stock ubuntu image. maintainer john fink <john.fink@gmail.com> the maintainer of the created image. run echo "deb http://archive.ubuntu.com/ubuntu precise universe" >> /etc/apt/sources.list the run statement runs a command from within the image being built. in this case, it’s adding the universe repository to ubuntu’s source list, so later on we can install pandoc. run debian_frontend=noninteractive apt-get update run debian_frontend=noninteractive apt-get -y install python git pandoc two run statements, updating ubuntu and installing the necessary packages. add ./start.sh /start.sh add inserts files from outside the building image. here, start.sh is added to the image. run mkdir /article/ add ./c4l-docker-article.md /article/c4l-docker-article.md a run and an add together, making the directory and adding the article markdown to it. expose 8888 expose tells docker that containers built from the image will have programs running that need access to port 8888; in this case, for python’s simplehttpserver. cmd ["/bin/bash", "/start.sh"] the cmd statement defines a default program to run inside the container if no specific command is given; here, we want containers to run the start.sh file using /bin/bash. start.sh the extremely basic start.sh files looks like this: cd /article/ pandoc c4l-docker-article.md -o index.html python -m simplehttpserver 8888 all it does is go to the article’s directory, convert the article from markdown to html, and then spawn python’s simplehttpserver to serve the article. building the article container running the docker build command builds the container. the command parses each line of the dockerfile, saving each step as a commit and layering the next commit on top. this makes it easy to roll back if errors are detected. as a performance bonus, subsequent invocations of docker build will use the prior layers as a cache, only building new layers when a line is changed. this usually makes building images from edits very fast. docker build -t c4l-docker-wordpress git://github.com/jbfink/c4l-docker-article and after the build, launch the container with: docker run -pd c4l-docker-article every time a new container is created from the c4l-docker-article image — as we do when we issue the docker run command — the cmd statement from the dockerfile is executed; in this case, our start.sh file. this file converts the article markdown to html and runs simplehttpserver to serve the file. the flag p tells docker to expose a random port on the host to the container port 8888 so simplehttpserver can be reached from outside the container, and flag d tells docker to run the container detached in the background. running docker ps should show a line with the new container and the random port assigned to it; going to the docker host at that port with any web browser renders the article. a real-world example: wordpress in the spring of 2013 i started building a docker wordpress container with an eye towards using it in-house for development projects. why wordpress? wordpress is the white lab rat of library software — it’s used everywhere, is well supported, is well understood, is generally easy to take care of, and has a huge host of ancillary software behind it. it’s a good real-world example for testing docker’s capabilities. initially i built the container manually — that is, by launching a single docker container running a bash shell and basically performing the steps in the above dockerfile by hand (doing the normal apt-gets and vim editing of config files). i put it up on docker index [27] and was contacted by a few folks in email about how i built it. in august of 2013 i started work on docker-wordpress [28], a structured way of building what i had done manually that people could play with and build on. the key problem with setting up wordpress in a normal fashion, freezing it in a docker image, and then running that elsewhere is that the configuration would remain the same across containers — same mysql passwords and wordpress salts and keys in php. ideally every time docker-wordpress is run there should be different values for all the fiddly wordpress configuration options, so docker-wordpress contains start.sh [29], which runs a series of commands at first inception to set values for things like salts in wp-config.php. sed -e "s/database_name_here/$wordpress_db/ s/username_here/$wordpress_db/ s/password_here/$wordpress_password/ /'auth_key'/s/put your unique phrase here/`pwgen -c -n -1 65`/ /'secure_auth_key'/s/put your unique phrase here/`pwgen -c -n -1 65`/ /'logged_in_key'/s/put your unique phrase here/`pwgen -c -n -1 65`/ /'nonce_key'/s/put your unique phrase here/`pwgen -c -n -1 65`/ /'auth_salt'/s/put your unique phrase here/`pwgen -c -n -1 65`/ /'secure_auth_salt'/s/put your unique phrase here/`pwgen -c -n -1 65`/ /'logged_in_salt'/s/put your unique phrase here/`pwgen -c -n -1 65`/ /'nonce_salt'/s/put your unique phrase here/`pwgen -c -n -1 65`/" /var/www/wp-config-sample.php > /var/www/wp-config.php this ensures that different values for important variables get loaded each time docker-wordpress is first run. combining this with the output from the initial build [30] from source means a fairly long startup time, but subsequent runs usually take less than 30 seconds, and rebuilds (which can be cached) really only need to happen when major updates to component software happen. running the docker-wordpress image is a fairly simple affair, and new containers take about a second to spawn, after which they can be accessed via internal ips or given outward-facing ports on the host machine. docker-wordpress has a great advantage in that it’s one image, can be run in one container, and is easy to understand. but it would be a mistake to consider it a good model for a production-type instance. in particular, its slapdash approach to logging (something logstash [31] could go a long way towards fixing) and inclusion of a local mysql instance make it a difficult sell for a production environment as it is currently written. consider running 20 docker-wordpresses, each with its own database; it would make much more sense to have a single mysql server serving multiple wordpress instances. conclusion docker has been under heavy development for over a year, and during that time its creators have more or less discouraged people from attempting to run docker as a production-ready framework. however, on june 9th, docker reached 1.0 [32] status, and as such should be considered ready for production instances. importantly, docker 1.0 is compatible with all prior versions of docker, so projects that started when docker was a moving target can still be used going forward. however, porting more esoteric applications to docker is not yet an easy procedure. docker wants to run things in foreground processes, making it necessary to convert common programs like mysql and apache from their usual background modes to foreground ones, and docker’s focus on one application per container (achieved in docker-wordpress and many other docker applications through judicious use of supervisord) makes running products with complicated install procedures somewhat less than optimal and perhaps more suited to traditional vm or bare iron deployments for now. despite the work yet to be done on docker, it is rapidly approaching a stable state [32] and shows definite promise for incorporating into libraries’ virtualization activities. for software development, when programmers check their code into git, a dockerfile could be included in the source code, allowing for quick testing of code on remote servers or as a demonstration tool to let others quickly bring up their own versions of an application without having to worry about specific building instructions or dependency management. docker could be used as a backup strategy — as part of a backup script for a library’s web site, a docker image could be built for a rapid deployment in case of service outage. and for regular systems administration tasks, who wouldn’t be happy to see a huge installation procedure [33] boiled down to a single command? endnotes [1] http://en.wikipedia.org/wiki/virtualization [2] http://www.theregister.co.uk/2011/07/14/brief_history_of_virtualisation_part_2/ [3] http://www.theregister.co.uk/2011/07/11/a_brief_history_of_virtualisation_part_one/ [4] http://www.linux-kvm.org/ [5] http://www.xenproject.org/ [6] http://aws.amazon.com [7] http://www.openstack.org [8] http://virtualbox.org [9] https://www.virtualbox.org/attachment/wiki/screenshots/gnome.png [10] http://vagrantup.com [11] http://www.dosbox.com [12] http://cronicasredux.blogspot.ca/2011/09/installing-and-running-debian-armel-on.html [13] http://en.wikipedia.org/wiki/operating_system-level_virtualization [14] you can run different linux versions even though you might commonly consider them different oses; a red hat guest could run under an ubuntu host, for instance. [15] http://docker.io [16] http://www.thegeekstuff.com/2013/05/linux-aufs/ [17] http://linuxcontainers.org/ [18] http://stackoverflow.com/a/18208445/380282 [19] http://radar.oreilly.com/2012/06/what-is-devops.html [20] http://git-scm.com/ [21] https://stackoverflow.com/a/22370529 [22] http://simh.trailing-edge.com/ [23] https://www.docker.io/gettingstarted/#h_installation [24] http://docs.docker.io/reference/commandline/cli/ [25] http://johnmacfarlane.net/pandoc/ [26] http://github.com/jbfink/c4l-docker-article [27] https://index.docker.io/u/jbfink/wordpress/ [28] http://github.com/jbfink/docker-wordpress [29] https://github.com/jbfink/docker-wordpress/blob/master/scripts/start.sh [30] https://gist.github.com/jbfink/9054707 [31] http://logstash.net/ [32] http://blog.docker.com/2014/06/its-here-docker-1-0/ [33] http://docs.evergreen-ils.org/2.5/_installing_the_evergreen_server.html about the author john fink is the digital scholarship librarian in the sherman centre for digital scholarship at mcmaster university in hamilton, ontario. he has been working with open source and libraries since 1995. subscribe to comments: for this article | for all articles one response to "docker: a software as a service, operating system-level virtualization framework" please leave a response below: david kinzer, 2014-07-21 this is a very good article on this subject. thanks for taking the time to write it! i wanted to add that now there is also a docker/container chef tool (in beta) that will be beneficial for those who already use chef to manage their systems. http://www.getchef.com/blog/2014/07/15/release-chef-container-0-2-0-beta/ leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – xforms for libraries, an introduction mission editorial committee process and structure code4lib issue 11, 2010-09-21 xforms for libraries, an introduction xforms applications can be used to create xml metadata that is well-formed and valid according to the schema, and then saved to (or loaded from) a datastore that communicates via rest or soap. xforms applications provide a powerful set of tools for data creation and manipulation, as demonstrated by some projects related to library workflows that are described in this paper. by ethan gruber, chris fitzpatrick, bill parod, and scott prater i. introduction libraries have become repositories for increasingly complex sets of information. several decades ago, libraries typically only managed printed materials: books, periodicals, maps, etc. a single cataloging record described each object, and this record rarely had to change since later editions of the work were assigned new records. today, libraries manage traditional materials in addition to vast sets of digital resources: databases, geographical information system datasets, images, audiovisual materials, and many others. each intellectual object requires metadata for searchability, and the better the quality of this metadata is, the more likely library patrons will be able to access objects through public interfaces.[1] xml has become the standard for metadata encapsulation, lending itself easily to internet transmission and machine processing. libraries throughout north america and europe commonly use dublin core, mods, tei, ead, and vra core[2] to describe various types of bibliographic, archival, or visual resources collections. these metadata records are often not static, but rather need to be modified over time, and therefore institutions have devoted resources to developing applications and workflows for editing them. structurally complicated xml (which describes all those listed above with the exception of dublin core) is difficult to author or edit in traditional html forms, which also have limited validation capabilities. a working group was formed to devise a new standard, xforms, to address the inadequacies of earlier form types. xforms 1.1 is a world wide web consortium (w3c) xml specification, published october 2009[3], that defines the operation of a form with the model-view-controller (mvc) relationship. xml structure is encapsulated in the model while controllers manage interaction with web services, action handlers, background processing, and other features to present a form to the end user. the view enables repeatable elements and real-time validation. essentially an xforms application can be used to create xml metadata to the fullest extent the schema allows and then saved to (or loaded from) a datastore that communicates via rest or soap.[4] the xforms submission enables your xforms model, e. g., a mods record, to be processed against an xml pipelining language (xpl) call to transform that instance into a solr document and post it to an index. xforms applications provide a powerful set of tools for data creation and manipulation, as demonstrated by some projects related to library workflows that are described in this paper. ii. support a. required expertise xforms is a complex standard that was designed to be powerful and flexible enough to fill the needs for complex, dynamically-created and modifiable forms. like its close cousin, xslt, it inhabits the grey zone between being a programmable toolkit and a data serialization standard. simple forms can be created fairly rapidly, though they will be more verbose than their html counterparts. however, the benefits of xforms do not really begin to manifest themselves until you begin to design forms with complex structures, dependencies, and runtime behaviors. if all you want to do is create a simple, static address form, you would be better off sticking with html. however, if you want to create a metadata editor that fully encapsulates and enforces the constraints of a mature and rich standard, such as mods or ead, the time spent mastering xforms will pay off in the long run. so what are the skills an xforms designer should have? first and foremost, they should be familiar with xml, and comfortable navigating complicated xml documents. xforms uses xpath syntax and functions to express relationships and behaviors, so the designer should have a good working knowledge of xpath. the designer should also understand the xml schema standard, as it is used to express constraints and define validation rules. the designer should also be familiar with the metadata standard(s) that the form will generate. in the context of libraries, this means a good working knowledge of dublin core, mods, ead, tei, vra core, and other more specialized xml metadata standards used in the library field. these standards will be used to create the model (explained below); data entered in the form will be serialized into the xml standard defined in the model. finally, but perhaps most importantly, the xforms designer should be a skillful, sensitive user interface designer. no matter how large or complex the form, or how rich the metadata that is generated from it, the form will be of no use whatsoever if the end users find it clunky, non-intuitive or difficult to use. in this regard, the power and flexibility of xforms can be seductive, especially to the technically proficient xml wrangler. the designer cannot lose sight of the fact the form will be used by people who are more interested in entering data quickly and accurately than they are in fancy interface tricks and ornate data structures. if you’re fortunate to have on staff a developer with all these qualities embodied in one person, then you’re in a very enviable position. if the expertise in your library is more diffusely distributed, as is the case in most places, then the work of designing an xforms-based interface may fall to three people: a primary developer, who will create the xforms document, a metadata librarian, who will communicate the functional requirements of the form, and help guide validation rules and local encoding practices, and an interface designer, who will work with the xforms developer to map the metadata model onto the controls to produce a user-friendly form. depending on the degree of overlapping expertise, the core job of developing the form may be shared to a lesser or greater extent; but no matter who does the work, about 35% of the time spent creating the first complex xforms interface will be consumed with defining the metadata model at one end (the output), and the user interface design at the other end (the input screen). the remaining 65% of the time will be used to actually create the xforms to bind the two together. as time goes by, the roles of the metadata librarian and interface designer will diminish, as a body of decisions and practices, embodied in a library of xforms snippets, takes shape. b. software so you have the expertise in-house, ready to create an xforms-based metadata entry web application. now what? can you just write an xforms document, put it on your webserver, and see the form magically appear in your browser? unfortunately, that is not the case today. xforms was going to be the new standard for web forms back in the xhtml days, but as of 2009, xhtml (both version one and two) has been deprecated by the w3c in favor of the next-generation html standard, html 5. when the w3c dropped xhtml, it also abandoned plans to incorporate xforms into the next-generation html standard (opting instead to expand the current html forms tagset for html 5).[5] in real world terms, this means that browsers are unlikely to offer native xforms processing any time in the future. but xforms itself is alive and well, it continues to be developed by the w3c xforms working group and there are a number of products available that implement the most recent xforms 1.1 standard. as xforms is its own xml specification with its own namespace, independent of any other markup standard, it can be encapsulated in an xml document of any type, not just xhtml. for example, openoffice 3.x+ implements the xforms 1.0 standard to create form documents. however, for the purposes of web development it makes the most sense to embed xforms markup in xhtml documents, and that’s where you’ll see 99% of the xforms out in the wild. there are a number of third-party browser plugins available, as well as server-side applications that can process an xforms document, translate it into ajax controls with additional css and javascript, and send it to a browser for display without the need for any client-side software.[6] iii. metadata editors all of the applications and examples discussed below were implemented using the orbeon forms application, a server-side web application that transforms xhtml+xforms documents into html documents with ajax controls for display in the browser. orbeon forms is an open source product; the company offers a gnu lesser general public licensed (lgpl) community edition for free download and a professional edition for purchase.[7] the authors have opted to use orbeon due to its active and growing user community, the responsiveness of the software developers to questions, the rich set of examples and documentation created for the product and for xforms development in general, and the stability and maturity of the software itself. moreover, orbeon is a java-based application that runs in apache tomcat, like numerous other applications that libraries use (solr, fedora, cocoon, etc.). while the orbeon forms source code is publicly available on github, the core of the orbeon forms application is maintained by a small group of committers. this makes the core java libraries of the application very stable. application upgrades in most circumstances only require moving the xforms templates to the new application. the orbeon developers have also included over 400 unit tests, which are automatically run by orbeon at build time but can also be run by individual users on their own systems. however, as with all software, there are some maintenance issues of which developers need to be aware. initially at stanford, some of the most time consuming issues were not related to maintaining the code, but rather to supporting all the various forms that had been released. this was the result of a deployment decision to not attempt to create a single mods form for all users, but rather deploy multiple slightly modified mods forms in order to address a collection’s particular needs. while this is very easy to do in the orbeon application, it does create some support issues. since each form is essentially a markup document that configures the mvc that is interpreted by the application, the vast majority of any desired behavioral changes must be done by editing these configuration settings, not by recoding the application. since this can require changes to multiple configurations, this can make pushing requested global changes, such as structural changes to the metadata output, modifications to a third-party api, or a new feature desired for all collections, somewhat time consuming. in attempting to address this issue, in august 2009 stanford migrated to the form runner environment, which is a part of the orbeon forms ce core distribution. as stated on the orbeon website, “form runner manages form types and form data, handles search, validation, and takes care of the plumbing necessary to capture, save, import and export form data.”[8] primarily, the form runner code accomplishes this by making more of the mvc dynamically built, as well as adding a commonly used persistence layer for all forms running in the environment. forms running under the form runner environment are rendered at runtime by common xslt stylesheets that can dictate much of the structure of the form. many desired global changes can therefore be made at the form runner level, rather than in the individual form. in addition, interactions between the forms and their datastore’s apis can be configured in the form runner’s persistence layer settings. therefore, a change to an api can be applied across the entire application rather than having to edit the rest calls defined in each of the form’s markup. however, committing to the form runner environment also has its trade-offs. most notably, the orbeon form runner is currently in a beta release, meaning much of the code that drives the templating is still undergoing semi-frequent changes. while unit testing does greatly help limit unforeseen errors, there are cases where a change to a feature can unknowingly and detrimentally impact the application. developers investigating orbeon are encouraged to take into consideration these maintenance factors in formulating an approach. currently, using “traditional” xforms in orbeon provides a very high level of application stability, but can increase the time required for administration if the number of actively deployed forms proliferate. form runner, while still an emerging technology with some of the pitfalls of beta software, has the upside of offloading much of the maintenance to common templates files. a. simple xforms document let’s start with a simple xforms document that outputs a snippet of mods metadata. <xforms:model id="hello_world"> <xforms:instance id="mods"> <mods:mods xmlns:mods="http://www.loc.gov/mods/v3"> <mods:title/> </mods:mods> </xforms:instance> <xforms:bind id="my_title" ref="instance('mods')/mods:title" required="true" type="xs:string" /> <xforms:submission action="http://my.web.service/eat/mydata" method="put" id="send_it_off" /> </xforms:model> <xforms:input bind="my_title" /> <xforms:submit submission="send_it_off"> <xforms:label>send it off</xforms:label> </xforms:submit> at its most basic, an xforms application does nothing more than take the data input via controls, validate it, put it into the model, and send it off as a stream of xml. lines 1 – 8: the model defines the serialization of the data, its bindings and constraints, and what happens to it when it is ready to be sent off. what you put in the model is what will be sent to a web service somewhere when the user clicks “submit”. the serialization format itself is set in the elements. each includes a snippet of xml in the metadata format you want to output; in this case, we’ll output a mods document. in more complex documents, you may have many models, and dozens of instances in each model. not all models and instances need be part of the output, and you have complete control over which models and instances are used to build the final serialization. the optional element on line 7 ties the model to a control, and defines certain constraints on this bit of the model. in this case, we declare that the /mods/title element in the model is required, and that it must be a valid string, as defined by the xml schema standard. the <xforms:submission> element on line 8 determines where the data will be sent once a submit event occurs. a model may have more than one submit event. xforms gives you a great deal of flexibility in chaining submissions together, sending different pieces of a model to different services simultaneously, or only submitting data upon the fulfillment of certain conditions (including, but not limited to, successfully completing and receiving a response from earlier submissions). lines 9-12: this is the “view/controller” part of the xforms, where the model gets embodied in traditional interface form controls. every control in an xform is bound to some instance in a model. the input to the control becomes the value of the bound element of the model. in line 9, the element displays a text input field on the web page. the data that was typed into the text input field becomes the value of the /mods/title element. the “bind” attribute refers to an element with the corresponding id in the model. the element, in turn, refers to an element in an instance in the model, as expressed using xpath. lines 10 -12 display a submit button on the web page, with the label “send it off”. the “submission” attribute ties the button to the submit action with the id “send_it_off” in the model. this xforms is a bare bones document. to be properly displayed in a browser, it should be combined with xhtml. here’s the above form rendered in a html page: <html xmlns="http://www.w3.org/1999/xhtml" xmlns:xforms="http://www.w3.org/2002/xforms" xmlns:xs="http://www.w3.org/2001/xmlschema" xmlns:xsi="http://www.w3.org/2001/xmlschema-instance" xmlns:mods="http://www.loc.gov/mods/v3"> <head> <title>mods metadata editor</title> <xforms:model id="hello_world"> <xforms:instance id="mods"> <mods:mods xmlns:mods="http://www.loc.gov/mods/v3"> <mods:title/> </mods:mods> </xforms:instance> <xforms:bind id="my_title" ref="instance('mods')/mods:title" required="true" type="xs:string" /> <xforms:submission action="http://my.web.service/eat/mydata" method="put" id="send_it_off" /> </xforms:model> </head> <body> <xforms:input bind="my_title" /> <br/> <xforms:submit submission="send_it_off"> <xforms:label>send it off</xforms:label> </xforms:submit> </body> </html> if you were to type “codex seriphinianus” in the input field, then click “send it off”, your web service would receive the string “codex seriphinianus” in the body of a put request. b. use case examples the following use cases from the university of virginia, northwestern, and stanford demonstrate the kinds of problems encountered when creating metadata editors, and how xforms were used to address those problems. some common themes emerge: mastering xforms is a continual process, and as expertise develops, forms can be more modular, with reusable form elements; that using xml to process xml is preferable to creating models of the data outside the data; and that the real power and flexibility of xforms manifests itself when it comes time to express arbitrarily complex, hierarchical and recursive relationships in a form. encoded archival description (ead) one of the major obstacles to streamlining electronic finding aid production in archives has been the technical barrier of subject specialists learning ead xml in order to create valid documents that make use of elements in a semantically appropriate way. while as-you-type-validation in oxygen has improved quality control, many institutions are using text editors or microsoft word to author electronic finding aids. an xforms application that enables archivists to create complex, hierarchical finding aids without needing actual knowledge of the xml schema would be of great use to the community. the eaditor project was established nearly a year ago to tackle this challenge and has progressed significantly since its inception.[9] much of the ead 2002 schema is represented in the web form, but mixed content (a mix of plain text and stylistic elements, much like html or tei paragraphs) is not yet supported. the application enables uploading ead 2002-compliant guides from the “wild,” i.e., human-generated files. these ead files cannot easily be imported into database-driven systems such as archon or archivists’ toolkit since the source document is broken down into 20 or more tables, heightening the potential for data loss. eaditor includes as part of its suite of xforms applications a form for customizing the default ead templates so that archivists can define their own data model for the core of the finding aid or subcomponents of the collection. for example, a repository may require a <unitdate> and access rights information for each item. the archivist can build this into the model so that individual elements do not have to be inserted for each new finding aid. additionally, items can be “published” to a solr index, which the public interface for searching, browsing, and displaying finding aids relies upon. in this sense, there is not just one web form for creating and editing ead, but rather numerous xforms applications that the user encounters, depending on the task at hand. some of these are as simple as a pop-up window that asks the user to choose to add or remove a solr document or delete the xml file from the datastore. one of the most important features of eaditor is a controlled vocabulary management system. in one of the first informal demonstrations of eaditor, it was pointed out that there should be strong authority control in several sections of the document. subjects, along with personal, geographic, corporate, and family names should be restrained by localized university of virginia or virtual library of virginia consortium controlled vocabulary. authorized library of congress subject headings (lcsh) terms were also requested. termscomponent in solr 1.4 was used in conjunction with orbeon’s form runner autocomplete widget to deliver this functionality. as a user enters text into an input bound to the widget, the value of the input is passed to a submission that queries solr to return terms that begin with the letters that the user has typed. the solr response is placed into an instance which populates a dynamic set of items that is suggested to the user. the controlled vocabulary manager of eaditor enables one to scrape all of the access terms from ead files within the datastore (exist, in this case) and post them en masse to solr. new terms can be created and posted to solr in addition to deleting terms. the library of congress provides an atom feed for updates to the lcsh dataset, which is processed by an xslt stylesheet called within the vocabulary manager to add or update to the solr index all the terms that were modified after the previous time the process was executed. although elements in eaditor do not query the service maintained by the library of congress directly, they do query a service running on local hardware that can always be kept up to date with id.loc.gov. additionally, localized and lcsh terms can be suggested in parallel since they are contained in the same solr index. figure x: lcsh linked data processing in eaditor vra core northwestern university library (nul) is in the process of converting its art history image collection from marc maintained in its ils (integrated library system) to vra core in fedora/solr. moving away from the ils leaves a functional gap in the availability of a suitable cataloging tool. the choice of vra core as the archival format has focused editor requirements on its descriptive model and prevalent cataloging practice in visual resource collections. this case study describes the motivations to adopt xforms and the use of xforms to create an editor for vra core. reusable interface widgets vra core organizes its larger descriptive model into sets, such as titleset, agentset, descriptionset. each set contains repeatable groups of descriptive fields, an optional note field, and a free-form ‘display’ field that is used to combine the set’s description for public display purposes. while each set has different fields in its repeatable groups, the consistent overall structure of vra sets means that a reasonably abstracted and repeatable ‘set widget’ can be used to support the entire schema. for example, here is the agentset in such a widget: controlled vocabularies many fields in an item’s description are under authority control. at northwestern, authorities management is done in the ils (ex libris’ voyager). to provide authorities access to this xforms editor and other collection management systems, authorities services are exposed through a locally developed web service. since multiple authorities are available for name/term selection, a dialog panel allows the user to select authorized names and terms. the dialog includes drop-down lists to select the source authority (library of congress or getty) and the type of authority record. once the authority is selected, an autocomplete mechanism similar to the one in the eadeditor application iinteracts with the the authorities services. for example: custom xbl components since controlled vocabularies are used for so many fields, there’s a strong incentive to make the authority files dialog easy to invoke throughout the editor. this can be accomplished by first encapsulating the dialog, its fields, authorities service interactions, and selection mechanisms in a distinct xforms component. there then needs to be a way to associate the encapsulated dialog with controlled fields, then invoke it. orbeon supports the w3c xml binding language (xbl) for binding an xforms mvc component (the authorities dialog in this case) with a custom namespaced element. by using xbl to create a custom xml element for field input which invokes the dialog widget when entered, that element can easily be used for any field that should be under authority control. the interface, local data, and behavior is encapsulated, then made available in an editor form by using a simple custom xml element. here is an example structure to declare a custom xbl widget and bind it to a custom xml element. the authorities dialog is defined in a custom component xbl file that provides its models and views between <xbl:template/>: <xbl:xbl xmlns:xbl="http://www.w3.org/ns/xbl" xmlns:nulauthlkup="our_custom_namespace_uri"> <xbl:binding id="nulauthlkup-authorities-lookup" element="nulauthlkup|authorities-lookup"> <xbl:template> dialog's view and model elements in here </xbl:template> </xbl:binding> </xbl:xbl> the authorities dialog is included in the editor form: <xi:include href="oxf:/xbl/nul/authorities-lookup/authorities-lookup.xbl" xxi:omit-xml-base="true"/> the custom element and its reference to the vra element it manages is declared for any field under authority control that should provide the dialog for entry. for example: <nulauthlkup:authorities-lookup authtype="subject" ref="vra:term"/> search integration an important feature of vra core is its support for many-to-many work / image relationships. having separate work and image records facilitates descriptive consistency and efficiency across images of the same work, while affording descriptive differentiation where needed for different views of the work. catalogers must be able to efficiently assert relationships between work and image records. the editor supports editing separate work and image records simultaneously. work records receive full vra cataloging using all of the vra ‘sets’ grouped in separate tabbed panels. a separate image record can be loaded in the ‘image’ tab, displaying its variable resolution zoomable image along with its vra title, description, subject, and relations sets. these sets are the only ones currently provided in an image record, but additional sets can be easily added if needed. both works and images can assert relationships to multiple image and work records respectively. to do this one needs to be able to search the collection, select appropriate items, and assert their appropriate relationships. having a search feature in the editor with access to the entire collection in the context of editing a specific record makes this possible. it also facilitates other useful operations that draw on the larger collection. the editor supports the following actions on search results: add this image to the currently editing work record add this image’s primary work record to the currently editing image edit this image record edit this image’s primary work record clone this image’s primary work record and edit it display this image’s public view in a separate window the “public view” mentioned above is the “read only” html dissemination of the image with zooming apparatus, full metadata, and thumbnails of its related images. relationships are edited in the work and image records’ vra relationshipset. there is a widget for these shown below. work and image records in this display can also be selected for editing, cloning, or full public display. workflow and repository integration cataloging takes place in the larger context of ingest workflow and publication. cataloging is, of course, just one aspect of collection management. cataloging is embedded into the workflow process in two ways. image items in the workflow system, which manages scanning jobs and their processing towards publication or fulfillment, include a link to the editor with the workflow item identifier and repository identifier as parameters. catalogers can inspect their workflow task queues and jump directly to items loaded into the editor. when a workflow item identifier is available to the editor, the editor displays an additional “cataloged” button which the cataloger can use to notify the workflow system when the item’s cataloging is complete. the workflow system can then advance the item if cataloging was the only process pending for its publication or fulfillment. this way a “save” action isn’t overloaded with both saving and indicating that the item is ready. catalogers may catalog (and save) an item over multiple edit sessions before it is deemed ready for publication. the repository identifier is used to retrieve and store the vra record in a fedora repository. when records are saved, using the “save” button, they are forwarded to a jms (java message service) queue for ingest processing. that process takes care of updating the item with the new vra core metadata in fedora. thoughts on this xforms experience this vra core editor is northwestern’s first use of xforms in application development. northwestern came to xforms fairly recently, after somewhat unsatisfying experiences in building web-based database applications on other platforms and encouragement from the work at virginia, stanford, and wisconsin. while each platform has its strengths and weaknesses, using non-xml platforms to implement non-trivial xml data models inevitably requires just that: a non-trivial data model to implement. if you’re supporting a relatively simple descriptive model, it’s reasonable to develop a web-form application in whatever platform seems expedient and then crosswalk that simple model to a standards-based xml schema for archiving or transmission. however, when you start to use more and more of any expressive schema, the development and maintenance complexity of your model software can increase a lot. you typically wind up either over-simplifying the model for expediency, codifying the full schema in custom software, or writing a general purpose xml editor. in the end none of these approaches are very palatable. xforms’ direct support for the target xml schema as the application data model removes all of that, and is one of the principal attractions of xforms. it’s true that the flip side of that trade-off is that your view – the form itself – is tightly bound to the specific schema you’re supporting. however, the bulk of the work in xforms development is in developing and arranging interface widgets – combinations of primitive interface elements that combine to address a complex descriptive task. designing such widgets for reuse should make their deployment in forms supporting other schemas a relatively low-cost activity. the combination of widgets like those for vra sets, an authorities dialog, and a search widget will likely form the core functionality of a variety of item level collection management applications at northwestern. another benefit in xforms development worth pointing out is the quality and responsiveness of the mailing lists. technology principals from orbeon frequent the ops-users list and have been very helpful and timely in their replies to questions. metadata object description schema (mods) and text encoding initiative (tei) recently, several institutions have implemented xforms applications to edit mods metadata. as mentioned above, stanford has been using an orbeon xforms to create mods records for over two years. the original mods editor was inspired by mods editor developed by michael park at brown university.[10] the original application was a single form that exposed the most basic set of mods elements to users for metadata creation. early attempts focused on creating a single form that could render all the mods elements in a way that was universal for all projects. however, after analyzing the needs of several project managers, it was realized that creating a single mods form for all collections was much harder than simply creating several variant mods forms. with the migration to the form runner environment, much of the form creation process has been greatly streamlined, and a similar approach has recently been applied to forms used to also generate tei records. the process of developing a new form usually entails modifying an existing form to have the structure outlined by the project manager’s requests. default value lists, section labels, and field data types are all specified by the project manager. these specifications are then applied to a generalized mods form and deployed on the server as its own form in the orbeon application. overall, the creation of individual forms for each collection goes very smoothly, since most collections’ mods metadata are similar in overall structure. the differences tend to be found in the value constrained lists, default values, and editable/non-editable fields. very often, collections already have some metadata created, although this can be in a wide variety of forms and conditions. for example, some collections have marc records already created in the university’s integrated library system, while other collections have records stored in filemaker pro databases. for these collections, often an export is converted to mods and pre-populated into the exist database. end-users are then able to go through the records in order to clean up or enrich the metadata. in this type of situation, xforms really shows some of its advantages since it stores xml in a server-side dom object rather than in a client-side javascript object. when metadata already exists for a collection, often it is for cataloged items sent to the stanford scanning labs. in these situations, the bulk of the record does not need to be changed, since only a few elements (such as a url or an updated copyright notice) are added. exposing the entire record in a form can be overkill, especially if a large part of an item’s mods is collection-level boilerplate. using most standard web application languages, this can be problematic, since standard html forms do not have an mvc. in orbeon, it’s possible to only show a small subset of the mods and hide the rest of the xml. when the xml is edited and submitted to the datastore, the entire xml document is sent. since all this is done by the core orbeon application, developers do not have to resort to scripting ajax functions to handle xml manipulation. like the northwestern’s vra core application, the stanford metadata toolkit also uses xbl to bind a form’s mvc to third-party web services. one of the most helpful features the toolkit provides is the ability to import mods records from the university’s ilms. by keying on particular mods identifiers, this service pulls a marc record from an instance of sirsidynix symphony, converts it to mods using the library of congress marcxml to mods stylesheet, then populates the form. the way orbeon’s mvc operates is important for facilitating the desired behavior. since a core part of the model is simply the mods schema, the form does not have to declare all the possible elements that are present in the mods record. when the form loads a new mods record, defined elements will just populate the form as described in the view, but the rest of the xml structure will persist as loaded in the dom object. this keeps developers from having to explicitly declare all the possible elements in a mods document in order to ensure imported xml data does not break their mvc or is lost. overall, this ability to handle complicated xml structures is xforms’ biggest asset. in the past, attempts to develop similar applications required either breaking the xml into smaller chunks, a strong reliance on customized client-side ajax, or both. in the metadata toolkit, the application is allowed to interact with the data with an xml-based approach. in xforms, the challenge usually comes in attempting to sort out the particulars of how the view represents your model, which can be difficult considering most schemas are not developed with data entry in mind. however, while the learning curve can be steep, it does free the developer up from having to jump through hoops to build a language-specific model that is replacing an xml schema. iv. enterprise integration once your xforms are up and working, and your metadata librarians are happily creating metadata, what do you do with the metadata they create? fortunately, handling xforms output is surprisingly simple, as all that is produced is a stream of xml that can be posted or put via http requests (rest or soap) to any web service that is set up to process incoming streams. the xforms submission methods are considerably richer than standard html form submission elements; submissions can be chained together, the same xml stream can be sent off in parallel to different endpoints, submissions can happen behind the scenes as users type (as is the case with the autosuggest functions, mentioned above), and submission widgets (buttons, etc.) can be activated/deactivated depending on the validity of the data entered into the form, or other criteria. orbeon ships with an embedded exist[11] xml database. in its samples and tutorials, data is put in the database via a restful request. then metadata can be retrieved when the form loads, and the fields can be pre-populated, via a get request to the database. in a production environment where manually creating or editing metadata is a piece of a larger workflow, endpoints will vary: data can be sent to a fedora repository to update a datastream, sent to dspace, become part of an ils catalog record, and so on. with the introduction of form runner, the use of an application-wide persistence layer allows the forms to seamlessly interact with different datastores. by default, orbeon ce form runner uses the embedded exist database, although it also ships with a persistence layer that is configured for a mysql database. orbeon professional edition ships with both oracle integration, as well as a persistence layer configured for the alfresco content management system. one approach currently being investigated is developing a persistence layer for a fedora repository. fedora offers some interesting possibilities, since it would be able to utilize the versioning, optimistic locking, and messaging for both the metadata and form markups. at stanford, initial development integration has utilized a middleware application written in sinatra that brokers communication between orbeon and fedora. this was done in order to more seamlessly integrate the stanford unique identifier and workflow services. in this approach, a form’s markup is stored in the fedora repository as its own object. a particular collection that utilizes this form is given relationships in fedora’s rel-ext that ties the form object to collection object. the collection objects also have relationships to their member items. this allows an application to automatically pull a particular collection item’s form from fedora, as well as generate lists of items related to a particular form or collection. since the application uses the stanford workflow service, it is also possible to generate collection or form-based queues on workflow status. the fedora work also allows developers to take advantage of the efforts being made by the fedora community to incorporate the fedora security layer (fesl).[12] the orbeon application does offer a standard authentication mechanism provided by any j2ee application server, but this can be difficult to integrate into a large organizational authentication strategy. at stanford, currently the web application is behind a web authentication layer that only allows access to whitelisted users. however, authentication has not yet been fully incorporated into the application. for example, from the datastore’s perspective all records are made by a single user that is held by the application. by using fesl, access control would be able to be set at the fedora object level, which would not only make for better security policy, but also allow separate applications to share common access control decisions. another promising avenue some of the authors have been exploring is integrating xforms into a workflow designed around an enterprise service bus. in this case, xml streams are sent as “messages” to the bus, where they are then processed and kicked out for further processing and storage according to rules and paths defined in the bus. one important consideration that has become clear to the authors is the need to distinguish between data in an unfinished state, and data that is ready for further processing and storage. in that regard, exist or some other xml database can be useful in a production environment as a temporary storage area, a buffer where metadata that is in the process of being created can be stored and updated, before it is marked as ready to move on. using an intermediary buffer for editing metadata can offset some of the problems with updating “live” data, insofar as a working copy is made from live data, the working copy is modifiable offline, and when it is deemed ready to go live, it is copied back into the “live” location. issues of locking and simultaneous edits must be addressed, however, as should questions of authorization. v. conclusion despite its power and flexibility, xforms has yet to become a mainstream standard. this is mostly due to its unfamiliarity, as it is a standard that has not received much attention in the web development community. of those who have had an opportunity to explore the technology, some have raised objections to it. the objections to xforms can be boiled down to the following three points: it is a dying (or dead) standard, with little support. it is difficult and complex. the infrastructure necessary to process xforms adds extra support overhead to your workflow environment. the first point has been addressed above: the standard itself is alive and well, and despite the lack of native support for it in browsers, there is a small but active and growing community of developers who have adopted the standard, and a variety of open source and vendor applications under active development that process xforms. many large enterprises and public institutions across the world have adopted xforms for building forms, for example: the national archives and records administration, cisco, pfizer, and the university of california santa barbara.[13] regarding the second objection, a perceived weaknesses of xforms which has lead to reluctance to adopt the standard and commit to building applications around it is that libraries lack personnel who are experienced with it. certainly many libraries have committed to php, perl, and ruby on rails development and, in some cases, have been developing applications based on those languages for more than a decade. however, many libraries already have potential xforms developers. as mentioned above, xslt programmers with a concept of the document object model and xpath would have little trouble adapting to xforms development. there are numerous resources for xforms development online, including the great support community surrounding orbeon. with many institutions moving progressively toward open source releases of software applications, this will save time, money, and effort for many libraries that intend to create metadata records. the brown university mods editor, the stanford mods editor and the university of virginia ead editor are freely available, for example, and the northwestern university vra core editor will be available in the future. once the structures of the xforms documents are understood, designers can tweak the model and view, and begin to integrate the application into current workflows, rather than starting from scratch to build something completely new. xforms are certainly non-trivial to master, but its difficulty must be evaluated in the context of your entire application environment and support of the software that underlies it. there are many individuals and organizations throughout the library, archive, and museum communities that are involved in cutting-edge tool development; the wheel certainly could be re-invented, and applications that fill the same purposes as xforms could be developed and implemented using the language du jour, be it ruby, java, perl, groovy, lisp, or what-have-you. however, one should be mindful of the long-term maintainability of such projects. a metadata editor built in a ruby on rails or php framework with a lot of javascript attached may be able to generate the same mods data model as an xforms-based editor. the difference is that the xforms developer is mostly concerned with writing solely xforms xml. the server-side processor includes the javascript necessary to effectively render and validate the form in the browser. in xforms, the data model is seamlessly bound to the interface (inputs) and to the data serialization (output), removing layers of programmatic abstraction that require ongoing maintenance (serializations of data objects in the model to the desired format, updates to the objects and backend object storage whenever the model is changed, etc.) this means that the xforms application can be more easily migrated to new platforms or upgraded to newer versions of the processor, as it is primarily xml, with little or no storage and language dependencies, beyond the ones bound with the back-end processor. a rails application with a great deal of highly customized javascript may not function as desired five or ten years from now, which requires developers to maintain it regularly in the long-term. the third point of criticism listed above is the perception that xforms complicates the application stack and convolutes the cyber-infrastructure with yet another platform that systems administrators have to manage. as mentioned previously in this paper, orbeon runs in apache tomcat. so does chiba, another open source server-side xforms processor (which does not have the same market share or support base as orbeon).[14] fedora and solr, two potential building blocks for contemporary institutional repositories, also function in a tomcat production environment. in this case, using xforms with orbeon as opposed to a rails or php application for depositing digital materials into a repository actually simplifies the infrastructure, rather than complicating it. one of the main benefits of xforms is the facility with which you can leverage the work others have done to implement forms that can then in turn be shared within a wider community of users outside the walls of your own library. in this respect, the potential disadvantage of introducing another application layer into your workflow environment can become an advantage, as it helps to keep your forms and metadata serializations loosely coupled with the other pieces of your environment. more importantly, it offers you the opportunity to participate in an active and growing community of xforms developers working to solve the same problems that confront you. in this sense, as the authors of this paper have discovered, the true power of xforms is in its community; working together, they have solved problems more efficiently and rapidly than any of them could have done on their own. given the sustainability of the standard and the ability to create and edit complex xml models, xforms has great potential for the library community. not only can input forms and xml serializations be created with xforms, but applications can be woven into digitization and publication workflows, controlled vocabularies can be managed, and web services commonly used in libraries can be easily hooked to the forms. with numerous institutions exploring the standard, xforms holds promise for becoming a mainstream form of application development over the next several years. references [1] bruce, thomas and dianne hillman. “the continuum of metadata quality: defining, expressing, exploiting.” metadata in practice. ala editions, 2004. (coins) [2] dublin core: http://dublincore.org/ metadata object description schema (mods): http://www.loc.gov/standards/mods/ text encoding initiative (tei): http://www.tei-c.org/index.xml encoded archival description (ead): http://www.loc.gov/ead/ vra core: http://www.vraweb.org/projects/vracore4/ [3] http://www.w3.org/tr/xforms11/. ibm continues to play a large role in maintenance and support of the standard. ubiquity xforms (http://code.google.com/p/ubiquity-xforms/) is a javascript library that plugs into existing javascript apis for browser-based xforms rendering. [4] for more information on representational state transfer (rest), see http://www.ics.uci.edu/~fielding/pubs/dissertation/top.htm; for information on simple object access protocol, see http://www.w3.org/tr/soap/. [5] recent history of xhtml and html5 and xforms: http://diveintohtml5.org/past.html [6] xforms software catalog: http://www.w3.org/markup/forms/wiki/xforms_implementations [7] orbeon download: http://orbeon.com/forms/download [8] vorbeon form runner: http://www.orbeon.com/forms/orbeon-form-runner [9] see: http://code.google.com/p/eaditor/ [10] brown mods editor: http://library.brown.edu/its/software/metadata/ [11] http://exist.sourceforge.net/ [12] fedora security layer: https://wiki.duraspace.org/display/fcr30/fedora+security+layer+%28fesl%29 [13] http://www.orbeon.com/company/customers [14] http://chiba.sourceforge.net/ about the authors ethan gruber: ethan is a web application developer in the scholars’ lab of the university of virginia library. he specializes in creation and delivery of cultural heritage digital content, including metadata modeling and information architecture. his website is http://people.virginia.edu/~ewg4x/ chris fitzpatrick: chris is a developer at the stanford university digital library systems and services group. he is a graduate of portland state university, as well as san jose state university’s mlis program. he enjoys programming in both ruby on rails and xforms. his blog can be read at http://worldonawire.info/. bill parod: bill is a software developer at northwestern university library. he helps design and implement systems for managing and delivering digital collections. his previous work includes software development in text analysis, aerospace, and the coin-op game industry. scott prater: scott is a digital library applications developer and project manager at the university of wisconsin – madison. he currently specializes in designing and implementing software infrastructures for digital collections. he has worked as a gold miner and system administrator, among other professions, in past lives. his work website is: http://sdg.library.wisc.edu/. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – oss4eva: using open-source tools to fulfill digital preservation requirements mission editorial committee process and structure code4lib issue 34, 2016-10-25 oss4eva: using open-source tools to fulfill digital preservation requirements this paper builds on the findings of a workshop held at the 2015 international conference on digital preservation (ipres), entitled, “using open-source tools to fulfill digital preservation requirements” (oss4pres hereafter). this day-long workshop brought together participants from across the library and archives community, including practitioners, proprietary vendors, and representatives from open-source projects. the resulting conversations were surprisingly revealing: while oss’ significance within the preservation landscape was made clear, participants noted that there are a number of roadblocks that discourage or altogether prevent its use in many organizations. overcoming these challenges will be necessary to further widespread, sustainable oss adoption within the digital preservation community. this article will mine the rich discussions that took place at oss4pres to (1) summarize the workshop’s key themes and major points of debate, (2) provide a comprehensive analysis of the opportunities, gaps, and challenges that using oss entails at a philosophical, institutional, and individual level, and (3) offer a tangible set of recommendations for future work designed to broaden community engagement and enhance the sustainability of open source initiatives, drawing on both participants’ experience as well as additional research. by marty gengenbach, shira peltzman, sam meister, blake graham, dorothy waugh, jessica moran, julie seifert, heidi dowding, and janet carleton. edited by marty gengenbach, shira peltzman, and sam meister introduction open-source software (oss) has played an increasingly prominent role in digital preservation over the past two decades.[1] starting with lockss, dspace, and droid, oss was integrated into a number of high profile preservation initiatives in the early 2000s, and quickly gained traction within the preservation community thereafter for reasons that ran the gamut from greater accessibility to lower costs. in the years that followed, oss came to play an increasingly integral role in the back-end technology that comprised the preservation landscape. as the adoption of oss solutions to support the curation of digital collections grew, and as both the number and variety of oss tools increased, there was a growing need among preservationists to assess how and when to adopt particular tools so that they could better support their institutions’ specific requirements and workflows. in response to these growing needs, organizers convened a workshop at the 12th international conference on digital preservation (ipres2015) entitled, “using open-source tools to fulfill digital preservation requirements” (oss4pres hereafter). oss4pres, which was promoted as “a space to talk about open-source software for digital preservation, and the particular challenges of developing systems and integrating them into local environments and workflows,”[2] was a unique event in two ways: whereas most oss workshops or events typically focus on a specific tool, user group, or region, this one was international in scope and brought together participants from across the oss community, including practitioners, proprietary vendors, and representatives from open-source projects and tool providers. not only did the diversity of participants enable a uniquely productive discussion, but it also resulted in an evenhanded consideration of the issues at the heart of the oss tools discussed. another factor that set oss4pres apart was its design. the day-long workshop was comprised of morning and afternoon sessions. while the morning was devoted to lightning talks–principally in the form of demos and case studies–that addressed a particular usage or implementation of oss, the afternoon was devoted to focused group discussions on oss’ opportunities, gaps, and challenges. workshop organizers strove to make the event participatory and placed a heavy emphasis on encouraging participants to interact with one another. this format allowed for dynamic and engaging discussions around the selection and integration of oss into preservation workflows. the resulting conversations were surprisingly revealing as much for their depth and breadth as for their consistency. the striking universality of the themes, concerns, and questions raised during oss4pres indicates a need to take stock of the current state of oss as it has been applied within the digital preservation community, particularly in order to identify and highlight the challenges and opportunities associated with implementing these tools. while oss’ significance on both a practical and theoretical level within the preservation landscape is clear, currently there are a number of roadblocks that discourage or altogether prevent its use.[3] the persistence of these challenges suggests that solutions must encompass voices from across and beyond the field, and must specifically address the stumbling blocks that participants have identified in conversation with one another at forums such as oss4pres. this paper will build upon the rich and multi-layered discussions that took place at oss4pres by offering a set of recommendations designed to illuminate a path forward for the preservation community. in order to achieve this task, the authors will begin with an analysis of the opportunities and challenges encountered by many organizations implementing oss, followed by a summary of the gaps in the oss landscape that have yet to be filled. in closing, the authors will draw on the conversations that took place at oss4pres along with additional research to offer a tangible set of recommendations for future work that will: (1) bolster existing open-source initiatives; (2) broaden community engagement; and (3) provide preservationists with a more sustainable model for open-source projects in the future. opportunities the collaborative model upon which the concept of oss relies offers numerous opportunities for fostering communities of practitioners and promoting knowledge and resource sharing at the local, national, and international levels. one example of this is the bitcurator software environment, a suite of digital forensics tools that have been modified and packaged for increased accessibility. cal lee and kam woods’ presentation on bitcurator at oss4pres focused on the twin goals of community engagement and making digital forensics tools more accessible to digital preservationists.[4] of particular note with regard to this software is the central role that the bitcurator consortium (bcc) plays as both the tool’s host and center for administrative, user, and community support. bcc is an independent, community-led membership association geared toward libraries, archives, museums, and other institutions that seek a collaborative approach to exploring and applying forensics solutions to their digital collections. the bcc maintains an active membership of libraries and archives that strive to increase the accessibility of the bitcurator software suite through outreach and documentation. the ethos of ‘give as you can’ allows institutions to develop specific tools that they can afford, while also providing these tools to organizations that cannot engage in development activities. open-source projects also provide an avenue for smaller or under-resourced institutions to actively participate in the development and improvement of tools by reporting bugs and providing feedback. in theory, the benefit of this arrangement is that the user community can contribute by developing and documenting tools so that there are resources available for all institutions working with oss. this points to another key opportunity for using oss in preservation environments: the ability to share experiences and build relationships among fellow practitioners and tool developers. some of the best examples of this at the workshop were conversations focused on the establishment of institutional workflows. andrew berger, for example, described the challenges he faced implementing archivematica, an open-source preservation management tool, at the computer history museum.[5] berger’s lightning talk touched on the usefulness of reaching out to colleagues and institutions working on similar projects and the importance of engaging with and participating in the wider open-source community. in addition to creating a more useful product, participating in a tool’s development contributes to the institution’s increased sense of commitment, both to that oss tool and to the community supporting it. there are mutual benefits to participation in an active oss community: practitioners can have a stronger voice in guiding the development of tools that fit their needs, while developers can refine their work and have more productive development cycles based on clear feedback from users. this relationship between tool users and developers is particularly important within the field of digital preservation, where adherence to community-developed practices and standards is paramount to success. by having direct relationships with developers, practitioners are able to advocate for tools that meet field-specific standards. oss4pres also demonstrated a number of tangible opportunities that exist at an organizational level. chief among these is the possibility of intra-institutional collaboration. using oss, particularly when it comes to installing, hosting, maintaining and upgrading software, often requires the skillsets and expertise of staff across multiple departments. this presents staff with a valuable opportunity to leverage one another’s knowledge in order to work toward a shared goal. for example, ucla library recently embarked on a project to implement archivematica so that it can be used by several different departments throughout the library.[6] although the digital archives program within library special collections (lsc) will be the primary user this software, the library’s core operations and developer operations teams will be responsible for implementing, virtualizing, and ultimately supporting it. in order for this project to be effective, the two departments have worked together closely. this process has required that core ops and dev ops staff become familiar with digital preservation tools and concepts like the reference model for an open archival information system (oais)[7], while lsc staff have had to become familiar with linux-based operating systems. there are two results of cross-pollinating the various knowledge bases within an organization. firstly, interdisciplinary collaboration can help prevent the development of departmental or organizational silos of information. secondly, projects that incorporate staff from a wide range of positions and departments throughout an organization reinforce the notion that, in order to be successful, digital preservation must be supported at all levels of an organization. the more people involved in preservation activities, the higher preservation’s profile within an organization will be–an important component in advocacy for oss implementations. in addition to the opportunities that intra-institutional collaboration affords, there’s even greater potential for collaboration on oss projects across institutions. in fact, many oss projects have been the result of inter-institutional collaboration. the ability to combine several organizations’ experience, resources, and existing knowledge bases can be a powerful tool that results in something much greater than the sum of its parts. a particularly good example of inter-institutional collaboration is the hydra project. a multi-institutional collaboration with over 30 listed partners, the hydra project “gives like-minded institutions a mechanism to combine their individual repository development efforts into a collective solution.”[8] banner projects like this one that leverage the resources of multiple institutions are particularly valuable because the final output–in this case a growing suite of tools for digital asset management and digital preservation, among many others–can be used by other organizations in the hydra community. the motto of the whole project is “if you want to go fast, go alone. if you want to go far, go together.”[9] as one of the hydra project partners puts it, “working collaboratively makes us work better. some of the best and brightest technologists in libraries are engaged in hydra, and the community model puts them on ‘our team,’ states one of the founding organizations.”[10] in addition to the philosophical and institutional opportunities described above, oss4pres workshop participants saw many benefits for digital preservation practitioners related to using open-source tools in both existing and developing workflows. many participants stated that using oss helped them share and improve workflows, gave them greater flexibility in establishing their own workflows, and improved the visibility of workflows across institutions, making them easier to comprehend. participants also cited the ability to freely download and experiment with open-source tools to determine how well they fit into existing workflows. they also appreciated the option to implement open-source tools incrementally, and as the tools become more familiar. the advantage of trying out and adding tools to a workflow as needs and requirements change is that practitioners do not have to implement an entire system at once to start testing and experimenting with oss, as is the case with some commercial software. as an added benefit, once an open-source tool has been tailored to fit local needs, these customized solutions can be shared with the preservation community at large and adapted by other individuals or organizations with similar needs. challenges for all the opportunities that arise from the community-driven nature of oss, there are challenges as well. indeed, the underlying commitment to freely-shared resources and collaborative development can be a double-edged sword: oss4pres participants agreed that oss both fosters and depends on an engaged community of practitioners. without continued involvement in and support from users, funders, and developers, open-source tools can flounder and may become defunct or orphaned. while there are many variables that can contribute to decreased levels of active community support in oss projects, oss4pres participants identified four primary factors: (1) lack of clear leadership and governance structures for oss; (2) administrative misunderstanding and/or lack of institutional buy-in; (3) limited or uncertain funding for the project; and (4) the unique challenges of oss implementation within a specific institutional context. of these factors, a lack of clear leadership is perhaps the most philosophical in nature. this is a dilemma that arises as a result of the oss model of distributed ownership, wherein responsibility for funding, development, and outreach is shared across several groups or institutions. although there are benefits to such a model, it can also be a source of confusion as to the direction and objectives of the tool in question for both developers and users alike. similarly, participants observed that the organizations behind existing oss may exhibit very strong leadership in certain areas, such as technical development, but may lack the skills or resources necessary to secure extended funding or provide training and documentation. this becomes an increasingly serious problem as tools become more widely used, and if software developers don’t have the resources necessary to keep up with demand. another challenge that can arise from the collaborative nature of oss is the perception–particularly among administrators–that because of oss’ decentralized ownership, these tools are inherently unstable and therefore present a risk. participants noted that as a result of this belief, administrators often express reluctance to commit resources to an oss project because they see it as a ‘bottomless pit’ that will drain financial resources. conversely, some administrators assume there will be no costs associated with using oss. this mistaken belief is the genesis of the oft-referenced anecdote about the difference between “free beer” and “free kittens”: namely, that while “free beer” can be unconditionally free, “free kittens” come with associated long-term costs that ensure that the animal survives and thrives. in both cases, these misperceptions of oss require practitioners to engage administrators in order to ensure institutional buy-in. advocacy for and education about open source projects must demonstrate both the return on investment in oss and the significant benefits to be gained from a collaborative, community-based approach to software development. thus, a key component of successful institutional oss adoption lies in ensuring that major stakeholders understand what using a particular open-source tool will mean. this includes an appreciation of the required investment over time to install, maintain, upgrade, and potentially further develop open-source tools. the last point is especially important because, like any software product, oss requires regular patches, updates, and upkeep. ignoring these over time may leave an institution with an oss tool that is outdated and therefore subject to bugs and security holes. the choice to maintain an unsupported version of a particular open-source tool simply because it meets (or has been customized to meet) an organization’s needs is problematic. for what an institution may stand to gain from this tool in terms of functionality and local integration, it may stand to lose in terms of the stability of a mainstream code release, the risk to information security, and the likelihood that the tool in question will become increasingly less functional and reliable as it ages. the source and duration of funding are other factors that can threaten the long-term sustainability of oss. tools that begin as grant-funded projects face the challenge of identifying and securing lasting sponsorship after the seed money dries up. one solution to this challenge lies in making a successful transition from a grant-funded project to a member supported consortium-based model. participants pointed to bitcurator as an example of a tool that has made this transition effectively. within the bcc, an elected executive council comprised of nominees from member institutions helps govern ongoing development and outreach. distributing this responsibility among consortium members serves to help ensure the tool’s continued upkeep and viability. the final challenge participants identified is in integrating open-source tools into institutional workflows. understanding the software dependencies, system requirements, and local configuration necessary to bring many oss tools online in a production environment can require a considerable investment of both time and resources. participants were quick to note that while one of the most significant benefits that oss has to offer is the ability to customize a tool for use within a specific context, the decision to do so comes with its own hurdles: while customization may result in a solution that is ideally suited to that particular organization, it may also reduce the potential value of other oss strengths, such as the benefit of a wide user community from which to draw experience and insight. furthermore, highly customized open-source tools may require a lengthy period of time to properly test, implement, and integrate. even if this is not the case, open-source tools that have been heavily tailored to a specific institutional context may have limited documentation that applies only to a small subset of users or be vulnerable to single points of technological and/or human failure. this siloization increases the odds that a particular tool will eventually become obsolete and therefore represents a risk to the tool’s long-term sustainability. workshop participants noted two related issues: first, there are a limited number of tools and documented workflows for pre-ingest tasks, which is perhaps due to the inherent need for customized implementation during this phase. particularly with regard to born-digital materials, pre-ingest activities were described by one presenter (to much agreement around the room) as the “wild west” of digital preservation. secondly, most of the tools that are available have been built on the assumption of simple and increasingly automated workflows. although automation is useful for organizational contexts with sufficient resources and technical expertise, for many practitioners the reality is far more rudimentary. the heterogeneity of born-digital content requires intervention or assessment at the file level, and the set of activities involved in this process are currently difficult to automate. in addition, many institutions utilize multiple workflows depending on the type of digital material being processed (e.g. digitized material, published and unpublished born-digital, audio/visual material, research data, theses, reports, etc.). in each case, the diversity and specificity of potential workflows make the development of a one-size-fits-all tool nearly impossible. gaps the preceding sections in this article point to a number of gaps in the current landscape for archives, libraries, and museums attempting to implement oss. perhaps the most significant of these is the paucity of successful models for oss leadership and governance. as suggested above, this is likely a result of the inevitable fluctuations in oss’ funding, development and maintenance. however, there are exceptions to this rule which, albeit still relatively few in number, may serve as a valuable stepping stone to a more widely replicable solution. these include the growth of the hydra community and the governance models of duraspace and the bcc, where development activities are coordinated via steering groups or similar leadership bodies. building on and extending these governance models could assist in fostering collaboration across the digital preservation oss development community. at the same time, there is a desperate need for transparent examples of effective cost-modeling for oss development projects. it is hard to judge, and therefore plan, the amount of time and resources it may take to either develop or implement an open-source solution for digital preservation. this can result in a lack of appropriate technological, financial, or staffing resources available to institutions–especially those that are less well funded. identifying and aggregating successful models for cost-sharing development and maintenance of oss efforts could prove enormously beneficial to practitioners in need of more realistic estimates. this could also be used to advocate for oss in meetings with administrators who control the budget. another gap is the absence of a centralized hub or forum where practitioners could share software and workflow documentation, post case studies, and address technical implementation challenges. recognizing the existence and value of current forums, such as the community owned digital preservation tool registry (coptr), participants articulated the need for such a space. shared evaluation of individual tools is valuable, but an illustration of the variety of approaches for sequencing and integrating tools is clearly a current priority for digital preservation practitioners. a centralized resource would assist in informing institutional decision-making about local adoption, implementation, and integration amongst multiple oss tools options. when it comes to specifics in functionality and workflows, requirements don’t always translate well. particular examples cited at oss4pres were requirements for tools that address rights management and intellectual property values, redaction and versioning, and security. along with an expressed lack of standardization around metadata exists the desire and need for an integrated or accessible way to reuse, map, and/or synchronize metadata between systems and tools. many workshop participants brainstormed ideas for bridging procedural and technical gaps at the local or individual level. several of the discussions about gaps focused on metadata. although a wide range of metadata standards and guidelines exist, some practitioners lamented that these documents do not always provide clear roadmaps for adopting new metadata standards and practices into existing environments. detailed guides for integrating, or entirely replacing, standards and practices seem either nonexistent or unidentifiable. moreover, metadata-integration methods between software applications and databases were also identified as a challenge. in order to bridge this gap, participants suggested that stronger communication lines be forged amongst practitioners for sharing issues, obstacles, and problem-solving activities for micro-tasks. one of the core features of, and benefits to, adopting oss is the willingness to share knowledge. it is up to the professional community to exploit multiple communication methods and platforms, including documentation, in a way that addresses the recurrent metadata-application challenges that surface across institutions. there is also a need for resources to support succession planning and the long-term sustainability of oss tools designed to achieve archival goals. this could include successful case studies for succession planning, which may help highlight the necessary components to ensure the sustainability of digital preservation tools over time. succession planning materials should also look at failed projects, in order to provide as much information as possible into what makes tools succeed or fail in the long run. in order to be truly effective, succession planning must occur at multiple levels. put another way, it is not enough for a grant-funded project to have a succession plan; any institution choosing to adopt oss must understand the tool’s functionality and requirements, and must establish a plan to mitigate any loss of productivity in instances where the institution is no longer able to meet those requirements. such planning could guard against loss of service by identifying similar tools with different requirements or technical components and providing technological road-mapping for institutions to navigate large-scale infrastructure changes. recommendations through the wide-ranging and dynamic conversations at oss4pres, participants brainstormed a number of different challenges, opportunities, and gaps in the existing digital preservation oss landscape. by compiling and distilling the resulting notes, the authors of this work have identified three recommendations to ensure the long-term viability of open-source projects. this section will provide additional detail for each of these recommendations, including references to similar resources where applicable. 1. develop sustainable and replicable models for governance, collaboration, and cost-sharing this recommendation is most concerned with the administration and organization of both interand intra-institutional oss projects. oss4pres participants noted a dearth of resources (or a lack of familiarity with existing resources) for organizations in the planning phases of oss projects. the lack of material to help organizations plan ahead at this stage is problematic since this early phase is where many of the most fundamental factors to long-term success are determined, including: deciding whether the project will be independently or collaboratively run–whether it is intraor inter-institutional–and what that will mean for governance and cost-sharing; understanding and clearly documenting the roles and responsibilities of each participating party; establishing clearly defined goals and criteria for success. to ensure the long-term success of future projects in the digital preservation space, there is a need for more examples of successful models of oss project management and project planning. one potential starting point for this work is the arl spec kit 340: open source software. the spec kit provides examples of oss contributor agreements that can help define how an institution will contribute to oss projects.[11] another useful resource are the many existing examples of tools and strategies to use in project scoping and management–from the mnemonic smart goals (specific, measurable, assignable, realistic and time-related) to highly sophisticated project management systems such as jira or daptive ppm. what’s missing, however, are documented examples of the implementation of these tools and systems in an organizational setting to support digital preservation oss. 2. establish a centralized resource for oss documentation and workflows while there is significant potential for sharing experiences about adopting and enhancing oss, there are still many issues that continue to hinder the implementation of a specific tool in a particular institutional context. whether it is an improper software configuration, incompatible hardware, or a limited understanding of a successfully installed tool, oss workflows for digital preservation may break down in any number of ways. one of the most commonly voiced recommendations from oss4pres attendees was the desire for a centralized location for technical and instructional documentation, end-to-end workflows, case studies, and other resources related to the installation, implementation, and use of oss tools. this resource–which participants envisioned as perhaps taking the form of a wiki–would serve as a hub that would enable practitioners to freely and openly exchange information, user requirements, and anecdotal accounts of oss initiatives and implementations. many oss tools have their own sites dedicated to hosting documentation, and there are other resources that are dedicated to hosting and aggregating wide-ranging technical and user-oriented information about tools, formats, and platforms for digital preservation.[12] however what is notable about the proposed hub is its unique emphasis on support throughout all stages of the oss lifecycle, including but not limited to: strategies and resources for successful oss advocacy with administrators, funders, and other stakeholders, including faqs about oss projects; case studies that include software and hardware requirements, estimated time from testing to production, and project budget information that other organizations could use or re-purpose in scoping or executing similar oss projects; long-term cost estimates that take into account factors like technical resources for installation and maintenance, and recommended hardware or software configurations and upgrades; information on oss project status–is it currently supported? if not, when was it last updated?; end-to-end workflows for acquiring, processing, preserving, and providing access to born-digital materials using oss tools. it’s also important to note that this proposed hub does not need to be standalone; in fact, it would be better integrated with existing listings of digital preservation tools, such coptr or the preserving digital objects with restricted resources (powrr) project. an example of such a resource for the library community includes the lyrasis-supported foss4lib site. as xkcd, the “webcomic of romance, sarcasm, math, and language” has observed, attempts to counter proliferation may inadvertently exacerbate the problem they intend to solve.[13] a recent code4lib journal article on the barriers to initiation of oss projects notes that the use of github as a widely accepted repository for oss project code has largely mitigated concerns about common platforms for sharing code; the authors of this article now seek a similar resource that will enable the free and open exchange of documentation, user requirements, and workflow information.[14] 3. develop resources to improve succession planning in oss projects the third recommendation is to promote succession planning as an integral part of oss development, and specifically, to ensure that after the project has ended there is a clearly documented strategy for maintaining the oss code base. just as the national institute of health, national science foundation, and an increasing number of other research-funding enterprises have mandatory data management and reproducibility requirements to ensure the data created by those projects remains available for the long-term, so too should digital preservation-focused oss have viable succession plans in place from their outset.[15] fortunately, there are many data management plans from which the digital preservation community can draw. ironically, many of these data management requirements have been established by archivists and records managers. this raises broader questions, outside the scope of this work: is succession planning less an issue of adequate resources (guidelines, best practices, etc.), and more an issue of not practicing what we preach? or, as bram van der werf suggested in a 2012 blog post, is it the limited number of skilled practitioners with the knowledge and ability to maintain the tools that are created in the digital preservation community?[16] regardless of the source of these issues, organizations approaching open-source projects should closely examine those that have made a successful transition from grant-funded standalone initiatives to robustly supported ongoing programs, and consider how to best emulate them. conclusion as the digital preservation field has matured so has the purpose and function of related oss. in the span of fewer than 15 years, the oss landscape has evolved from a scattered set of standalone tools designed to accomplish discrete tasks (e.g., generating and validating checksums, directory mapping) to complex software environments that bundle multiple open-source tools together to provide a suite of preservation services (e.g., archivematica, bitcurator). projects like these have radically expanded the horizon of possibilities for preservationists and provide exciting new possibilities. nevertheless, these tools still are not watertight. reflecting on the work that remains to be done, oss4pres participants noted that there are real concerns about open-source tools at both the individual and institutional levels. often the root cause of these issues stems from the philosophical underpinnings of oss–particularly its distributed ownership model. additionally, as participants were quick to point out, many of these challenges are not minor; they pose serious risks to collections both large and small. but for all these threats, participants also recognized that oss represents an unprecedented opportunity for practitioners. implementing the recommendations outlined in this paper will require time and consideration, and will likely entail its own set of challenges. but given everything that is stood to be gained from oss, it is imperative that progress be made toward improving the arsenal of resources available for implementing and supporting oss initiatives. it is the authors’ hope that realizing the above recommendations will achieve this, and thereby enable the field at large to move forward toward a better, more sustainable open-source landscape. acknowledgement the authors wish to give sincere thanks to the numerous editors, note-takers, and oss4pres participants that made this article possible. endnotes [1] the 2014 arl spec kit on open source software notes that 53% of 66 academic and public libraries responding to its survey use a locally hosted and supported oss solution for digital preservation. see curtis j. thacker, charles d. knutson, and mark dehmlow, “arl spec kir 340: open source software,” association of research libraries (2014). available: http://publications.arl.org/open-source-software-spec-kit-340/. [2] courtney mumma et al, “using open-source tools to fulfill digital preservation requirements.” proceedings of the 12th international conference on digital curation. chapel hill, nc: university of north carolina, school of information and library science, 2015. available: http://phaidra.univie.ac.at/o:429623. workshop proceedings are available at: http://oss4pres2015.web.unc.edu/. [3] other code4lib journal articles have discussed the challenges to implementing and sustaining oss projects. see: dale askey, “we love open source software. no, you can’t have our code,” code4lib journal no. 5, december 15, 2008: http://journal.code4lib.org/articles/527#correction; sibyl schaefer, “challenges in sustainable open source: a case study,” code4lib journal no. 9, march 22, 2010. available: http://journal.code4lib.org/articles/2493; curtis thacker and charles knutson, “barriers to initiation of open source software projects in libraries,” code4lib journal no. 29, july 15, 2015. available: http://journal.code4lib.org/articles/10665; bret davidson and jason casden, “beyond open source: evaluating the community availability of software,” code4lib journal no. 31, january 28, 2016. available: http://journal.code4lib.org/articles/11148. [4] cal lee and kam woods, “demonstration of the bitcurator environment,” november 6, 2015. available: http://oss4pres2015.web.unc.edu/contributed-talks/#lee [5] andrew berger, “implementing archivematica at the computer history museum,” november 6, 2015. available: http://oss4pres2015.web.unc.edu/contributed-talks/#berger [6] one of the co-authors of this work is employed by ucla library. [7] consultative committee for space data systems. reference model for an open archival information system (oais), ccsds 650.0-m-2, magenta book, issue 2, june 2012. available: https://public.ccsds.org/pubs/650x0m2.pdf [8] hydra project homepage: https://projecthydra.org/ [9] ibid. [10] why use hydra at stanford?, available: https://projecthydra.org/about-hydra-2/why-use-hydra/. [11] spec kit 340: open source software, 16. [12] for example, see: community owned digital preservation tool registry (coptr); open preservation foundation knowledge base wiki; digital curation centre; and program for electronic records training, tools, and standards (pertts). [13] xkcd: standards, july 20, 2011. available: https://xkcd.com/927/. [14] curtis thacker and charles knutson, “barriers to initiation of open source software projects in libraries,” code4lib journal no. 29, july 15, 2015. available: http://journal.code4lib.org/articles/10665. [15] butch lazorchak, “all that big data is not going to manage itself, part one,” the signal, may 27, 2014. available: https://blogs.loc.gov/digitalpreservation/2014/05/all-that-big-data-is-not-going-to-manage-itself-part-one/. [16] trevor owens, “open source software and digital preservation: an interview with bram van der werf of the open planets foundation. the signal, april 4, 2012. available: http://blogs.loc.gov/digitalpreservation/2012/04/open-source-software-and-digital-preservation-an-interview-with-bram-van-der-werf-of-the-open-planets-foundation/ van der werf suggests, “maybe we have a need for tools but we have a much bigger need for skilled people and an active preservation community.” bibliography askey, dale. “we love open source software. no, you can’t have our code,” code4lib journal no. 5, december 15, 2008: http://journal.code4lib.org/articles/527#correction. consultative committee for space data systems. reference model for an open archival information system (oais), ccsds 650.0-m-2, magenta book, issue 2, june 2012. available: http://public.ccsds.org/publications/archive/650x0m2.pdf. davidson, bret and jason casden. “beyond open source: evaluating the community availability of software,” code4lib journal no. 31, january 28, 2016. available: http://journal.code4lib.org/articles/11148. lazorchak, butch. “all that big data is not going to manage itself, part one,” the signal, may 27, 2014. available: https://blogs.loc.gov/digitalpreservation/2014/05/all-that-big-data-is-not-going-to-manage-itself-part-one/. mumma, courtney, et. al. “using open-source tools to fulfill digital preservation requirements.” proceedings of the 12th international conference on digital curation. chapel hill, nc: university of north carolina, school of information and library science, 2015. available: http://phaidra.univie.ac.at/o:429623 owens, trevor. “open source software and digital preservation: an interview with bram van der werf of the open planets foundation. the signal, april 4, 2012. available: http://blogs.loc.gov/digitalpreservation/2012/04/open-source-software-and-digital-preservation-an-interview-with-bram-van-der-werf-of-the-open-planets-foundation/. schaefer, sibyl. “challenges in sustainable open source: a case study,” code4lib journal no. 9, march 22, 2010. available: http://journal.code4lib.org/articles/2493. thacker, curtis and charles knutson. “barriers to initiation of open source software projects in libraries,” code4lib journal no. 29, july 15, 2015. available: http://journal.code4lib.org/articles/10665. thacker, curtis j., charles d. knutson, and mark dehmlow, “arl spec kir 340: open source software,” association of research libraries, 2014. available: http://publications.arl.org/open-source-software-spec-kit-340/. about the editors martin gengenbach is an archivist at the gates archive in seattle, wa. he holds an msls with a concentration in archives and records management from the school of information and library science at the university of north carolina at chapel hill. shira peltzman is the digital archivist for the ucla library where she leads the development of a sustainable preservation program for born-digital material. shira received her m.a. in moving image archiving and preservation from new york university’s tisch school of the arts and was a member of the inaugural class of the national digital stewardship residency in new york (ndsr-ny). sam meister is the preservation communities manager, working with the metaarchive cooperative and bitcurator consortium communities. previously, he worked as digital archivist and assistant professor at the university of montana. sam holds a master of library and information science degree from san jose state university and a b.a. in visual arts from the university of california san diego. sam is also an instructor in the library of congress digital preservation education and outreach program. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – book review: two books about frbr, compared mission editorial committee process and structure code4lib issue 4, 2008-09-22 book review: two books about frbr, compared this article reviews 2 books on frbr published in the past year. although both books aim to be introductions to frbr, their approaches are very different. one is sort of a frbr study guide with commentary, the other a collection of essays. robert maxwell’s book, frbr: a guide for the perplexed, takes the study guide approach. arlene taylor edited understanding frbr: what it is and how it will affect our retrieval tools, a book of essays about frbr and frad, written by cataloging experts, aimed at a broader audience, not just the cataloging specialist. the first seven chapters lay out the basics: introductions to frbr and frad, frbr research, frbr and the history of cataloging, frbr and rda. these chapters provide an excellent introduction for those new to frbr. the last seven chapters each look at different types of resources in relation to frbr. maxwell, robert l. frbr: a guide for the perplexed. chicago: american library association, 2008. (coins) taylor, arlene g. taylor, ed. understanding frbr: what it is and how it will affect our retrieval tools. westport, conn.: libraries unlimited, 2007. (coins) by christine schwartz after being around for ten years, functional requirements for bibliographic records (frbr) is finally coming into its own. the cataloging community is starting to realize that in order to understand the new content standard, resource description and access (rda), you need to understand frbr (and frad—functional requirements for authority data). this came home to me at a library conference earlier this year when i tried to buy these two books. the taylor book was sold out and the book vendor mentioned that they’re not used to selling so many lis books at conferences! so the advent of frbr is upon us and these two relatively new introductory texts arrived just in time. a lot has been written about frbr (its official bibliography is 56 pages long), but until now, there have been very few full-length books on the subject. the frbr model was developed in the 1990’s by a study group of the international federation of library associations and institutions (ifla). its impetus was a growing concern for the cost of cataloging, the proliferation of various and new library resources (especially electronic), and the need to explore changing technologies to find efficiencies and ways to share bibliographic records internationally. a study group was formed to look at these issues and to develop a set of functional requirements for bibliographic records. however, the study group did not just dig in to look at cataloging nitty-gritty. they stepped back to look at the conceptual framework for how we understand bibliographic records—the bibliographic universe so to speak. and they chose the entity-relationship model (e-r model), borrowing from computer science and database modeling, as the structure for a new conceptual model for how we look at bibliographic records with emphasis on the user. the initial development was by a small group of cataloging experts, the study group and consultants. but the draft document of frbr did go through a period of public review and comment. the final report was published in 1998. although not changed significantly, it was recently amended and corrected in february 2008. but it’s not only affecting cataloging, the frbr model is influencing several aspects of the library world: metadata standards, integrated library system design, etc. i’d like to start out with one caveat—the best way to understand the frbr model is to read the text itself. yes it’s theoretical and a little dry, but it’s good to know what it really says before you start reading all the interpretations. and being a conceptual model, frbr is very much open to interpretation. i think that’s why over the last ten years there’s been some confusion about what frbr is and is not trying to do. that being said, if there was ever a time when the library world needed frbr it’s now. we’re dealing with the question of how to provide access and create discovery tools for users in a rapidly changing library landscape. we’ve been slow to exploit the capacities of the web for collaboration, sharing metadata and content, linking, etc. so frbr’s entity-relationship framework and emphasis on bibliographic relationships may be its most important contribution to how we think about library resources in the digital, hyperlinked environment. although both books aim to be introductions, their approaches are very different. one is sort of a frbr study guide with commentary, the other a collection of essays. robert maxwell’s book, frbr: a guide for the perplexed, takes the study guide approach. if anyone should attempt to write a step-by-step guide to frbr, it’s maxwell. he has a gift for making the complexities of cataloging and metadata clear. so, much like maxwell’s well-known book on the anglo-american cataloguing rules (aacr2), this new book successfully explains and illustrates the text of frbr. in addition, maxwell takes a further step to speculate what frbrized metadata as well as a database based on frbr might look like. with the use of e-r diagramming, he presents a more visual approach than the original frbr study text. these diagrams are excellent and go a long way toward helping to understand the model better. his book is made up of six short chapters. after the introduction in chapter 1, maxwell explains the entity-relationship model in chapter 2. both entity-relationship diagramming and frbr diagramming are explained. the focus of chapter 3 is the frbr entities. maxwell carefully goes over each entity and its attributes. chapter 4 deals with bibliographic relationships. this was an excellent chapter. however, i did find one typographical error: systematic is used incorrectly several times, starting on page 94. in the original frbr text it is systemic. chapter 5 covers the frbr user tasks (my least favorite part of frbr). the last chapter, chapter 6, looks at the frbr model in relation to the marc and aacr2-based cataloging model. he concludes by arguing in favor of implementing a frbr data model for library metadata and database structures. maxwell comes off as being in the pro-frbr camp. i wouldn’t call maxwell’s book an easy read, but if you want to get a better handle on frbr concepts and possible future implementations, it’s a must-read. of the two books, this is the better choice for coders because the author takes a close look at database models. arlene taylor edited understanding frbr: what it is and how it will affect our retrieval tools, a book of essays by leaders in the cataloging community. the essays (which cover both frbr and frad) aim at a broader audience, not just the cataloging specialist. the first seven chapters lay out the basics, for example: introductions to frbr and frad, frbr research, frbr and the history of cataloging, frbr and rda. these chapters provide an excellent introduction for those new to frbr. i particularly liked bill denton’s adventuresome tour of the history of cataloging leading up to the development of frbr. the last seven chapters each look at different types of resources in relation to frbr. the introduction describes them as non-traditional library resources: archival materials, art resources, cartographic materials, moving images, music, and serials. i found these chapter interesting, but less satisfying toward meeting the goal of understanding frbr. the authors of chapters 8-10 all conclude that the frbr model does not accommodate their resources. each author displays a slightly different interpretation of the frbr model. the most interesting chapter from this half of the book is martha yee’s essay on frbr and moving images. yee definitely has an ax to grind with regard to rda development and she clearly lays out her position in this essay. one of my criticisms of the taylor book is not what is there, but what’s missing. there’s no chapter on frbr and integrated library systems (ils) or frbr and digital resources. covering these two important areas would have made it a more satisfying book overall. i found both books well worth reading. i think maxwell’s book in particular will become a standard reference tool in library cataloging and metadata departments. on a personal note, the frbr model has only recently started to make more sense to me since my work has shifted to digital collections and non-marc metadata. the fact is, i have to think about data models in my everyday work (finally). so i think that the frbr model really has legs to grow beyond its 1998 original text. based on the entity-relationship model, frbr has some street cred in the it world. it can function as a bridge beyond libraries to the world of information technology (much the way dublin core has done with metadata). it’s a conceptual model that can be used to develop different data models. i think we will see more development of the frbr model as time goes on and it may be that frbr and its companion models, frad and functional requirements for subject authority records (frsar), will be the most important data models for future library metadata structures and standards. so, if you want to start thinking about frbr (thinking being the operative word), these two books are a good first step! about the author christine schwartz is metadata librarian at princeton theological seminary libraries and the author of the blog cataloging futures. she is a member of the editorial committee of the code4lib journal. tags: book review, frbr subscribe to comments: for this article | for all articles 3 responses to "book review: two books about frbr, compared" please leave a response below: irvin flack, 2008-09-23 thanks for these reviews. on frbr: i think the model is very useful but is still very much based around the print monograph as the default object of description; understandably, given the history of libraries. the wemi model can be used on non-textual resources but not without some awkward shoehorning, as martha yee and others have pointed out, eg in this great paper on lubetsky’s work principle: . the work concept, which is quite straightforward when applied to a published text, gets complicated when applied to a movie, serial, website, photograph of an art work, recording of a opera performance, etc. as far as rda goes, it seemed early on that it was adopting a half-hearted approach to frbr, but now it seems more fully committed to it, which will at least enable the model to be more fully tested. -irvin chris schwartz, 2008-09-24 hi irvin, i just printed off martha yee’s article, “lubetzky’s work principle.” thanks for recommending it. several of the authors in the taylor book make the same point as yee concerning the procrustean bed of work-expression-manifestation-item. in fact, a lot of monographs would work better with just work-manifestation-item, imho. jonathan rochkind, 2008-10-27 but if some works (monographic and otherwise) work better with the four levels, and many monographs are fine with three of those four–then that to me says four is good. it doesn’t hurt the works that don’t need it. on the other hand, perhaps there are some works that could really use five, or six, levels. a model is always just an approximation of reality, but four levels seems a pretty good compromise to me. there are always the ability for other relationships to be drawn other than the ‘level 1’ sets, to specify things further. although the whole model is, for better or for worse, based on a formal abstraction of traditional cataloging practice, that was mostly designed for monographs, with everything else being an ‘edge case’. but i’m not sure what to do that’s better. if there are human catalogers describing the bibliographic/documentary/information universe, we’re a lot better off if they’re all using the same formal model, so we can have all agreed on what the data we’re recording actually means. but maybe there’s a better way to do it? if there is, nobody’s supplying it. leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – bibliographic metadata extraction from theses mission editorial committee process and structure code4lib issue 7, 2009-06-26 bibliographic metadata extraction from theses this article presents the application of part-of-speech (pos) based statistical text analysis to the task of bibliographic metadata extraction from electronic dissertations. by using the approach described here it is possible to detect the title of a ph.d. paper with an accuracy of about 80%. the accuracy measurements are done using a conceptually simple approach and implementation. by götz hatop introduction electronic documents are an essential part of each modern library. metadata about documents are usually created manually by the author himself or by a librarian before or after uploading a paper to an institutional repository. bibliographic metadata extraction can help to improve user interfaces by presenting reasonable suggestions of metadata. it also reduces the need to type information about title, author and other metadata into a form manually. the text in a document is written in a natural language and not understandable for a computing system. in general, even the task of extracting bibliographic metadata as simple as dublin core is extremely difficult. in recent years, much research has been done in statistical text analysis and, with respect to the complex task of identifying relevant information in a text, there has been considerable improvement. the language a text is written in is an example of metadata that can be retrieved from a document using statistical methods with nearly perfect accuracy. most algorithms for language detection are based on bigrams, which are groups of two written letters, two syllables, or two words. a good example of a program to determine the language of a text is the mguesser program [1]. also, the problem of automatically assigning a library of congress classification (lcc) or a dewey decimal classification (ddc) to a given work has been tackled by statistical learning methods with some success. these classifications can be used as the dublin core “subject” metadata to describe the topic of a resource. in statistical text classification, a number of already labeled documents is needed for each class. these so-called training documents have been labeled manually, but once a good training set is available, classification can be done by machines. manning et. al. give an introduction to the text classification problem [2]. this article describes a method to extract the title of a ph.d. paper by bigram analysis of the grammatical structure of the sentences found on the title page of a document, using an approach derived from statistical learning methods. metadata in an institutional repository in an institutional repository, metadata play an important role. they are used to navigate between documents, are propagated via oai interfaces and are traditionally the most important way to access a document in a library. at the university library of marburg/lahn in germany/europe over 2000 electronic dissertations were collected over the last 10 years. most of the documents are in pdf format and stored within an institutional repository. after uploading documents to the repository, the user has to enter the bibliographic metadata into a web form. a great subset of the documents can be transformed to unicode text with publicly available text converters. for some documents, transformation fails because of the usage of special pdf characteristics, which make conversion difficult. upcoming standards such as pdf/a will hopefully help to increase the machine readability of documents from the repository. nearly all documents contain a title page as their first page, since this is standard academic practice. proper title pages that appear as the first page of a thesis can be separated from other text with publicly available programs. we have used the pdftotxt program [3] and appropriate command line flags for this task. only a few documents have an empty page or some other information like a short dedication as their first page. the first page of the document (often called ‘title page’ or ‘front page’) itself contains much more information than just the title of the work and the name of the author. frequently, the university institute where the dissertation has been done is mentioned. however, in principle a title page is not limited to a certain quantity of information. also, the order of information is arbitrary and not the same on all title pages. while a human being can decide within seconds which part is the title of the thesis and which is not, this information is hidden to a computing system. until now no general algorithmic method of understanding for even such a limited amount of text was known. pos tagging of natural language sentences pos is an acronym for part-of-speech and pos tagging is the task of setting tags to text entities, sometimes called word-category disambiguation. in this article, pos tagging stands for tagging words with their grammatical sense. an example is the sentence ‘this/dt is/vbz an/dt example/nn’ which in this case is tagged with tags from the penn tagset (where dt annotates a determiner, vbz a verb, and nn a noun). modern pos taggers are able to tag words with an accuracy of about 97 percent and are almost always implemented using hidden markov models (hmms). larsen et al. present a good introduction to hmms and their applications [4]. the sentences found on a typical title page of a ph.d. thesis are different in many ways. for example, a statement like ‘from the clinic of ophthalmology of the university of marburg’ can be found on thesis title pages. this sentence differs from a typical title sentence grammatically; it contains no verb at all and 5 nouns out of a total of 10 words. the question we are trying to answer is, whether the sentences containing the title information can be identified only by analysis of the pos-sequence of each sentence. the short answer is: yes, we can. statistical analysis of pos data from a set of 256 thesis papers we extract the title pages as explained above. in a first step, the different sentences from each title page are separated. in written text, sentences are usually separated by special symbols like a period or an exclamation mark, but this is typically not the case on a title page. while sentence boundary detection is worth its own research, it is not our main focus here. we therefore restrict our experiments to those thesis papers where sentence boundaries can be identified easily by two or more carriage return symbols. we use the stanford pos tagger [5] to tag the words with their grammatical sense. the majority of the investigated title pages are written in german and so the tagger is configured to use a german tagger based on the negra corpus. the tagset used for tagging is the so called stuttgart/tübinger tagset (stts) [6]. from each tagged sentence we extract the pos-sequence. for example, the pos-sequence for the sentence ‘aus der klinik für augenheilkunde’ (‘from the clinic of ophthalmology’) is ‘appr art nn adja nn’ (preposition, article, normal noun, attributive adjective, normal noun). the documents from the test set are held in an institutional repository and metadata are known in advance. we label each pos-sequence of a title page with the line numbers referring to the original page. if a pos-sequence belongs to the title we are looking for, this information is written ahead of the sequence. with sentence detection and pos tagging it is possible to transform each title page to a form where statistics can be computed easily. a transformed title page looks like the one shown below. line 1-1: appr art nn nn adja nn line 4-4: adja nn nn nn ne line 7-8: nn ne ne ne nn nn ne line 11-13: art nn nn art ne ne appr nn appr art nn ne kon ne ne nn ne title 17-18: nn appr nn appr nn adv line 23-27: nn apprart nn art nn art adja nn art nn art nn art ne ne vvpp appr author 30-31: ne ne ne appr ne line 35-35: ne card figure 1: a pos-transformed and labeled title page (pos-page) the java program used for this transformation merely consists of the embedding of the stanford pos-tagger, some metadata lookup, and the usual input/output routines. from a training set of 8 pos-pages we train a statistical model, where ‘model’ refers to a collection of bigram frequencies. the computation of the bigram statistics is done with the jgram package, an open source java markov-chain n-gram library [7]. the size of the training set as well as the pages we choose is completely arbitrary. while this setup is good enough to get an impression about pos based data detection, in a real world production system the training base should eventually be somewhat larger. with l1 we denote the bigram statistics from title sentences, l2 denotes the bigram statistics of all other (non-title) sentences. the first few lines of the computed bigram frequencies, the heart of the statistical model, are shown below. %% cat title.stats | sort -t: -k 3 -r | head -5 adja:nn:0.11023622047244094 nn:appr:0.10236220472440945 nn:art:0.05511811023622047 fm:fm:0.05511811023622047 start:nn:0.047244094488188976 %% cat notitle.stats | sort -t: -k 3 -r | head -5 ne:ne:0.1444866920152091 nn:art:0.07034220532319392 nn:nn:0.06653992395437262 nn:ne:0.06463878326996197 art:nn:0.060836501901140684 the data can be interpreted such that in a typical title sentence from our training set a noun (nn) is preceded by an attributive adjective (adja) with a frequency of about 11 percent. sentences not found on title pages contain bigrams of named entities (ne) with a frequency of 14 percent. experimental results in title detection we are now in a position to evaluate whether the computed bigram statistic of pos-sequences is able to distinguish between title sequences and other sequence types. from a test set of 256 documents we extract each title page and transform it to a pos-page as done for model training. to calculate an acceptance value for each line, each bigram in the pos-sequence is weighted with the corresponding frequency found in the language statistic files. the pseudocode for this algorithm is shown below. result := null; maximumscore := 0.0; for each line from input file do begin head := line.substringbefore(":"); pos-sequence := line.substringafter(":"); for each bigram b from pos-sequence do a1 := a1 + frequency of bigram b as noted in file l1 a2 := a2 + frequency of bigram b as noted in file l2 done; score := computescore(a1, a2); if (score > maximumscore) begin maximumscore := score result := line end end if ( result.startswith("title") ) print success; else print fail; as it is not known in advance whether both of the language statistic files are really necessary, we check three different scoring models. in a first run, we accept the sequence with a maximum a1 value as the title sequence. from all 256 papers, only 68 titles could be extracted correctly. the second run uses the value a2 as the indicator for whether a pos-sequence belongs to a title sentence or not and retrieved only 83 titles. in the third experiment, a combination of the two values is used. the scoring value is set to a1 + 1 – a2 which results in correct detection of 205 out of 256 title sentences. to check whether a more complete statistical model can help to improve title detection rate, we changed the training set to include the whole set of 256 title pages. the computed bigram frequencies changed, the most popular pos bigram in all title sentences is an infinitive followed by a preposition. in non-title sentences a past participle followed by a named entity is the pos bigram with the highest frequency. however, running the experiments again shows a rather unspectacular improvement to 213 detected title sentences. conclusion one of the main intentions for doing the described experiments was to evaluate the idea of pos-based metadata detection and mining. the results of the experiments are positive. they show that the grammatical structure of a title sentence is different enough from other sentences that appear on a ph.d. thesis title page that title extraction is possible in many cases. only pos-tagging and simple bigram statistics are used to get the result. we feel that special tagging of words frequently found in sentences which are not the title (like ‘dissertation’ or ‘institute’) is capable of improving results further. these lines are the ones which are often detected false positive, that is, they are detected as title sentence but are not. but, such special tagging goes beyond plain grammatical based data detection and makes implementation more complicated and less general. it was relatively easy to derive a prototype of a title extraction program from the java classes written to train and evaluate the model. this prototype together with all other programs written for this project is available from bibtrack [8]. references [1] alexander barkov, mguesser. mguesser is a tool to guess a text’s character set and language. http://www.mnogosearch.org/guesser [2] christopher d. manning, prabhakar raghavan and hinrich schütze. introduction to information retrieval. cambridge university press, 2008. http://www-csli.stanford.edu/~hinrich/information-retrieval-book.html [3] pdftotext. the pdftotext program is part of a larger pdf related package called xpdf. http://www.foolabs.com/xpdf [4] richard j. larsen and morris l. marx. an introduction to mathematical statistics and its applications. 2006. isbn 0-13-201813-6. [5] stanford log-linear part-of-speech tagger: http://nlp.stanford.edu/software/tagger.shtml [6] stts tag table: http://www.ims.uni-stuttgart.de/projekte/corplex/tagsets/stts-table.html [7] jgram: a java markov-chain n-gram library. https://jgram.dev.java.net [8] bibtrack: a java program for pos-based title extraction from theses. ftp.ub.uni-marburg.de/pub/research/bibtrack.tar.gz about the author götz hatop is a developer at the it department of the university library of marburg/lahn. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – medici 2: a scalable content management system for cultural heritage datasets mission editorial committee process and structure code4lib issue 36, 2017-04-20 medici 2: a scalable content management system for cultural heritage datasets digitizing large collections of cultural heritage (ch) resources and providing tools for their management, analysis and visualization is critical to ch research. a key element in achieving the above goal is to provide user-friendly software offering an abstract interface for interaction with a variety of digital content types. to address these needs, the medici content management system is being developed in a collaborative effort between the national center for supercomputing applications (ncsa) at the university of illinois at urbana-champaign, bibliotheca alexandrina (ba) in egypt, and the cyprus institute (cyi). the project is pursued in the framework of european project “linking scientific computing in europe and eastern mediterranean 2” (linksceem2) and supported by work funded through the u.s. national science foundation (nsf), the u.s. national archives and records administration (nara), the u.s. national institutes of health (nih), the u.s. national endowment for the humanities (neh), the u.s. office of naval research (onr), the u.s. environmental protection agency (epa) as well as other private sector efforts. medici is a web 2.0 environment integrating analysis tools for the auto-curation of un-curated digital data, allowing automatic processing of input (ch) datasets, and visualization of both data and collections. it offers a simple user interface for dataset preprocessing, previewing, automatic metadata extraction, user input of metadata and provenance support, storage, archiving and management, representation and reproduction. building on previous experience (medici 1), ncsa, and cyi are working towards the improvement of the technical, performance and functionality aspects of the system. the current version of medici (medici 2) is the result of these efforts. it is a scalable, flexible, robust distributed framework with wide data format support (including 3d models and reflectance transformation imaging-rti) and metadata functionality. we provide an overview of medici 2’s current features supported by representative use cases as well as a discussion of future development directions by constantinos sophocleous, luigi marini, ropertos georgiou, mohammed elfarargy, and kenton mchenry introduction cultural heritage research is producing huge amounts of data at an ever-increasing rate and size due to the continuous development of more advanced and detailed data extraction technologies and methods. such research efforts include the pilot projects undertaken by the cyprus institute during the establishment of an advanced imaging center for ch and archaeology as part of the linksceem-2 cultural heritage work package. the three pilot projects used cutting-edge visualization technologies such as refectance transformation imaging. they generated scientific digital data of archaeological objects and artifacts, such as cylinder seals, ceramics, jewelry and other museum collections, as well as wall paintings and icons from the world known medieval churches of cyprus. in these contexts, the digital documentation, storage and management of the aforementioned data as well as providing easy access to researchers is becoming ever more critical. flexible database storage, processing, metadata extraction and presentation frameworks are needed in order to satisfy these requirements. the development of such database/data repository systems establishes a long-term solution for accessing and storing various kinds of ch data and metadata. these systems must be flexible in terms of being able to have additional data formats support incorporated over time, thus ensuring their continuous update with the latest technological developments, thus maintaining the ch scientific communities’ interest. the developed framework will be complementary to repositories such as the inscriptifact database and online library [1]. having a specialized resource repository for the needs of specific datasets (such as inscriptifact) must be coupled with a general digital storage and management framework also capable of handling types of data that are not the focus of inscriptifact. this collaboration will immediately allow the scientific exploration of newly generated ch datasets by an already established user community. medici has been and is being developed to address the above. it is a web 2.0-based general multimedia content management system capable of semantic content management and service/cloud-based workflows. it supports a broad range of throughput-intensive research techniques and community data management. medici provides scalable storage and media processing, straightforward user interfaces, social annotation capabilities, preprocessing and previewing, metadata extension and manipulation, provenance support and citable persistent data references [2]. through medici, users can upload datasets in a variety of formats, including 3d and rti formats widely used in cultural heritage research and collections [3][4]. dataset metadata is obtained by extraction services run as data preprocessors and presented to the user along with metadata input by the community for each dataset. utilizing these extractors medici is capable of deriving semantically meaningful features for content-based comparison between datasets (e.g. textual data from handwritten text). previews of large datasets in a variety of formats are also extracted and viewed to avoid the need of (down) loading the whole content on the user’s system or finding the needed software to examine the contents of a file. medici 2.0’s scalability/parallelization, flexibility, and robustness, as well as its overall performance, are improved by decoupling the preprocessor software from the main server (with extractors being allowed to run on different machines in a distributed architecture). a powerful and flexible resource description framework (rdf)-enabled database management system is used for storing datasets. dataset files can be uploaded using a variety of interfaces. use of the rdf standard makes the datasets’ metadata open and portable. medici uses the latest web technologies to display large images smoothly (seadragon, zoom.it [5]), 3d graphics (x3dom [6]), video (ffmpeg[7]) and rti (spidergl[8]) along with a special feature added to the rti web visualization developed by bibliotheca alexandrina (ba) to extract 3d models from rti files (rti-to-3d) in order to enhance the possibilities of the technology. different deployments of medici can be hosted by different institutions and parts, thus allowing the hosting party better control of access to resources. architecture a diagram describing the dataset upload, preprocessing and previewing process is displayed in figure 1. figure 1. dataset upload, preprocessing, and previewing process (click image to enlarge). the system is composed of the following parts: web server functions and architecture datasets and files are uploaded to the web server using one of an array of methods. one method uses regular hypertext markup language (html) forms (both when creating a new dataset to upload its first file as well when viewing a dataset to add files to the dataset). however, files can also be uploaded in other ways, including uploading individual files that do not belong to a dataset. the web server is responsible for sending a request to the database to store the file or dataset and sending a message to the rabbitmq message broker [9] defined in the server’s configuration. this message will contain the necessary information pertaining to the file or dataset that is needed by the rabbitmq broker to distribute the file or dataset’s preprocessing jobs to the available extractors/preprocessors, as well as by the extractors/preprocessors themselves to decide how to process the file or dataset. datasets contained as files in a zipped archive are inspected by the web server. these files are unzipped to identify the type of dataset whose preprocessing will be handled by extractors on the rabbitmq bus. the web server can then passively wait for any auto-extracted file or dataset metadata or previews (generated by the extractors/preprocessors that received a job for the file or dataset) to be uploaded to the server through a representational state transfer (rest) application programming interface (api) [10]. when an upload occurs, the web server calls the database to save the previews/metadata and associate them with the file or dataset. extractors can be chained. this means that the resulting file output from an extractor may be fed as input to another extractor, thus allowing even greater flexibility regarding execution of preprocessing subtasks common to the preprocessing of different file or dataset types. for this to happen, an extractor that outputs an intermediate result can upload it as an intermediate result for the file or dataset using the server’s rest api. this allows scientific preprocessing workflows to be implemented as a preprocessing chain or tree of different extractors. for example (common actual use case), extractor a may convert an rti (see section “extracting 3d models from rti”) to an x3d 3d model. then, that x3d model may be forwarded to extractor b to extract an html preview of the model, as well as to a metadata extractor to extract metadata from the 3d model, at the same time. this can generate a preprocessing tree of parallel preprocessing on different extractors. the only information that has to be hardcoded in the extractor itself is that it needs to upload an intermediate result, without needing to know exactly where that result will be forwarded to. the server does not need to know anything either! the intermediate result is forwarded by the server back to the job processing queue, which uses a job key to forward it to the right extractors for further preprocessing (see “job broker” section below). the server then calls the database to save the intermediate result and a new job description is sent to rabbitmq, now containing the identifiers of both the original file or dataset (with which the final result will be associated) and the intermediate file (which the extractor(s) working on the new job need to download and process). the web server is also responsible for selecting and setting up the previewers that will display the contents of a file or dataset based on the type (i.e. file format) of a file or dataset. other functions include working with custom metadata and searches, an interface for adding metadata to a dataset by users based on mapping depending on the institution managing the medici implementation, searching for datasets having metadata satisfying a query, and text-based search. the server also allows social annotation of datasets (i.e. tagging, notes, comments, custom metadata). technologies used the web server is built on the play web application framework [11][12], written for allowing coding in scala and java. as supported by the framework, the server uses the model-view-controller (mvc) architectural pattern [13]. it relies on a number of plugins for communicating with the rabbitmq broker, communicating with the database, and authenticating users. it uses dynamic html (html version 5) webpage generation (views) according to the results of the processing of the input (scala-based controllers). the models (defined in scala) are closely associated with collections in the database. preprocessors and scripts (i.e. previewers) running on users’ browsers communicate with the server using a rest api. preprocessors/extractors preprocessors are used for both extracting metadata from datasets and files as well as generating previews for them. each extractor listens to a rabbitmq broker via its own named job queue, which is bound to particular routing keys. it is by these keys that rabbitmq forwards the extraction job messages to the extractors’ queues. each job message is handled separately. whenever a job is received, the extractor first downloads the dataset or file having the identifier stated in the job description from the web server via the server’s rest api. what follows depends on the task each extractor is responsible for. extractors in general use integrated libraries or external system calls to third-party software to process dataset files to generate the previews or extract the metadata. the third-party software include, for example, 3d model and video processing programs that are installed to the extractor’s environment and are called through java command-line calls. finally, the extractor returns the result back to the web server: if the result is an intermediate file for use in extractor chaining, the file is uploaded to the server as an intermediate result to be forwarded to other extractors through rabbitmq to continue processing. if the result is a final preview, it is returned to the server as a preview and then a command is sent to the server to associate the preview with the original dataset or file. if the result is a set of dataset or file metadata, a command is sent to the server to associate the metadata with the original dataset or file. many extractors of the same type can be run simultaneously on the same or different machines and have the jobs sent by a medici web server be distributed among them by rabbitmq as long as they are registered with the same rabbitmq routing exchange. also, an extractor may receive messages from more than one medici web server instance as long as the web servers are registered with the same rabbitmq routing exchange. this allows for great flexibility, scalability and robustness through a distributed architecture. extractors can be implemented using java or python. previewers previewers are used for viewing the contents of a dataset or a file from within the user’s browser. the previewers have been successfully tested on firefox and chrome, though with minor adaptations they should work on any modern html5-supporting browser. on the server side, the server selects which previewer to be launched for each preview of each file in the dataset when a dataset page is requested from the web server. the selection is done by comparing each preview’s file type with the file types accepted by each previewer. the latter information is held as javascript object notation (json) data in a public file for each previewer. before the previewer scripts are sent to the browser along with the rest of the dataset or file page to be executed, certain variables are set dynamically by the server, mostly for the user’s browser to be able to find the files needed by the previewer on the server (e.g. the dataset files’ urls). the previewers use dynamically-added javascript scripts downloaded from the web server at runtime and added to the webpage’s dom (document object model) using jquery [14]. the previews to be displayed, as well as any other needed files, are also downloaded at runtime using asynchronous javascript and xml (ajax) calls. after the preview is ready to be displayed, the preview html is appended to the webpage’s html. the previewers use browser-side javascript and jquery. database the nosql mongodb is used as the system’s database management system (dbms). selecting a nosql dbms offers advantages important for medici. flexibility is a key attribute for medici, as ch community requirements change constantly as described in the introduction. this gives value to the horizontal scalability of nosql. simplicity of design of nosql databases allows the addition of complex features to medici (such as community-generated dataset metadata) without too much difficulty. job broker if the web server is the heart of medici 2, the job broker is its veins. the role of the rabbitmq message broker is to take preprocessing messages from the web server that are sent once a dataset or file is uploaded and distribute them to the extractors that can handle the jobs. each extractor, once activated, registers one or more delivery queues on rabbitmq, on which it listens. each queue is associated with a particular routing key set, which defines which routing keys a job can have in order for it to be routed to that queue (and thus to the extractor listening to the queue). each routing key set is defined by a four-field term like the following: localhost.file.model.obj.# in the above, the first field defines the url or ip of the machine on which the medici web server implementation sending job messages for the extractors is based. the second field defines whether the extractor takes jobs pertaining to preprocessing a file or a whole dataset. the third and fourth field are for the mime types of the files (or the types of datasets) that the extractor accepts, and the fifth is the possible subtype of the datasets or files. the syntax of the set definitions uses the conventions specified in the rabbitmq tutorials. metadata medici can accept, persist and process two kinds of metadata for each dataset or file: automatically-generated: generated by metadata extractors and associated with the dataset or file through a rest api call to the web server by the extractors. they usually pertain to the more technical aspects of datasets and files. their subjects, predicates, and objects can vary from dataset or file type to dataset or file type and from medici implementation to medici implementation depending on what metadata are available for each dataset type and dataset, as well as the type of the extractor which extracted the metadata. they cannot be modified by users. community-generated: generated by the users through completion of an html form for each dataset.their allowed subjects, predicates, and objects are defined on an institutional schema that can differ for each medici deployment. the schema is formulated by each deployer or institution at its discretion, however it is recommended for the schema to be compatible with the dublin core content description model [15] to facilitate metadata sharing between different repositories. the cidoc conceptual reference model is based on dublin core and is specialized for ch metadata and can also be used to facilitate metadata sharing among the ch scientific community [16]. the input metadata, either coming from an extractor or an html form, are converted to json structures by the extractor or a browser-side script respectively and sent to the web server, where the json is parsed back into a binary representation able to be stored in the database. metadata is understood by the web server and the javascript script manipulating them on the browser-side (for community-generated metadata) as a tree structure (for community-generated metadata) or a list of tree structures (for auto-generated metadata). this distinction applies because different extractors may generate different sets of metadata with no contact points between them for the same dataset or file and so the reasonable option here is to present them as a list. on the other hand, community-generated metadata are generated by users by modifying the whole metadata structure of a dataset using an html form at the same time, meaning that they can be merged. figure 2. a simple metadata tree for community-generated metadata. metadata are formulated as tree structures because each predicate of a metadata subject is either a property of the file or dataset or a subproperty of another property, all describing the same file or dataset. an example of a simple metadata tree (here for community-generated metadata) is presented in figure 2. the tree structures (one for each dataset) are stored on the web server’s application layer as scala maps. automatically-generated metadata there are currently metadata extractors for images and rtis, with more to be added in the future. for a discussion of the metadata extractors for each file type, see the sections discussing medici’s features for their respective types. community-generated metadata the schema for the community-generated metadata is defined in two comma-separated value files. the first file stores the name of each metadata node along with whether the node is a leaf in the metadata tree or a higher-level node. leaves consist of string data which are input by users and serve as the objects of the description of the dataset, answering the questions generated by the tree paths leading to them. branches are subproperties of higher-level properties or top-level properties. in the example shown in figure 2, the leaf node “title: ancient mari” is a leaf answering the question “what is a/the name of a/the project location of a/the project associated with this dataset?”. the second file stores the possible relationships associating nodes with nodes or leaves, along with the cardinalities of each relationship. the above are automatically enforced by the system when the user adds or modifies a dataset’s community-generated metadata. metadata search users can search for datasets that contain user-entered metadata that satisfy formulated queries. figure 3. example community-generated metadata search. the queries are formulated by filling out an html form similar to the html form for adding/ modifying community-generated metadata. for each level of the metadata tree, the user can select which properties must be satisfied by selected nodes at that level. for example, the query in figure 3 searches for datasets having a project, having cyprus as a country and a project location with “choirokitia” as a name, and the project also having responsible institution with a name, in this case, of “example institution.” users will be able to use a logical expression-like query formulation to generate queries. for example, he/she may wish to search for datasets satisfying the query above but having cyprus as a country or having a project location for which the rest apply, or having cyprus as a country and not having any project locations for which the rest apply. the json-formulated query is sent to the web server, which then parses it as a scala map, compares it to the community-generated metadata maps of each existing dataset, and returns the list of datasets satisfying the query. the comparison is done using a depth-first-search-esque algorithm [17] that recursively checks for all paths in the query tree where there is a path in the dataset’s community-generated metadata tree that satisfies the path. the server formats the result list as html and returns it to the client through ajax to be displayed. images the support of multi-format and multi-resolution image storage and presentation is highly beneficial to ch communities due to the development of image acquisition instruments that satisfy current demands of research analysis in archaeology. medici currently supports the jpeg/jpg, png, tiff, gif and bmp image formats. moreover, image processing, pyramidal multi-resolution conversion and consequently, optimization of high-resolution images (giga-images), is carried out by the extractor on the virtual server in order to store all necessary image datasets to the appropriate multi-dimensional matrix array and to be delivered to the user when needed thus creating a suitable and efficient way to draw high-resolution web images on screen without affecting the high frame-rate and fast open experience. deep zoom viewing of large images efficiently and without loss of quality is achieved by constructing an image pyramid using a technique similar to frustum culling. specifically, when a large image is to be viewed in a limited screen space, only a part of it can be viewed in full quality and the rest of the information in the image is essentially occluded. consequently, data transmission size is limited by tiling the image at various scales in a preprocessing step executed server-side so that only the tiles needed for the currently shown scale and position are transmitted. deep zoom, a technology developed by microsoft as part of microsoft silverlight [18], works exactly like thatwith the user initially seeing a lower-resolution view of the entire image which they can then zoom in to see portions of the image in more detail. zoom.it zoomit.js (http://zoom.it/h.js) is a javascript library also used by the zoom.it web service. it relies on the technology behind the seadragon gigaimaging viewer, which utilizes deep zoom. this library is used by medici 2 to view large images that have already been split up into various levels of deep zoom tiles (i.e. the image pyramid) by a dedicated image pyramid extractor. the pyramids are loaded on the web server together with an xml deep zoom image file (dzi) containing basic image metadata needed by zoom.it. gigaimage processing when an image is uploaded on the web server, a message is sent to rabbitmq for any extractors that can process images. if there are any active gigaimaging extractors registered with rabbitmq, a gigaimaging extractor will receive the job message for the image. the job message will contain the image’s size. if the image’s size is below a threshold registered value, the job is rejected and no xml file for zoom.it or the image pyramid tiles are generated. consequently when a user requests to view the dataset containing the image, no zoom.it preview will be found by the web server and therefore the original image will be shown, embedded with an html tag. if the size of the image exceeds the threshold, it is given an identifier and a zoom.it xml file is generated and registered. that identifier will be used to associate the image tiles with the preview and thus with the original image. then the image tiles for all the image zoom levels are generated, concurrently uploaded back to the web server, and associated with the appropriate preview. zoomit.js accesses the image’s pyramid tiles via a virtual directory structure generated on the web server for each preview. this structure is implemented with scala functions that take the parts of the virtual directory path as parameters and return the requested tile after querying the database where the tiles are stored. these parameters include the tile’s level and 2d-coordinates of its position on the level. an example of a zoom.it preview can be seen in figures 4 and 5, where it is used for visualizing ancient cylinder seals and byzantine wall paintings from cyprus. figure 4. a two-dimensional giga-image of a cylinder seal dating to the 13th century bc taken with a large format camera system and previewed in medici’s 2.0 web viewer. courtesy of the bank of cyprus cultural foundation. figure 5. giga-image of byzantine wall paintings taken inside the monastery of st. john lambadistis in cyprus and previewed in the web viewer. courtesy of the department of antiquities in cyprus. image technical metadata there are currently two different image metadata extractors, either of which can be used at the discretion of the deploying institution, each extracting different sets of metadata. the first uses the imagemagick image processing suite [19] and the second uses the java-based image metadata extractor library developed by drew noakes [20]. the imagemagick extractor extracts many more details, even though it needs imagemagick to be installed on the machine running the imagemagick extractor. 3d models complete digital documentation in archaeological and cultural heritage research is a multidimensional process. new opportunities and challenges for the development of web-based virtual reality (vr) applications in these research fields have been the direct consequence of advances in the field of three-dimensional representation and internet-related technology [21]. high accuracy and multi level of detail (lod) models can be obtained by using various methods for 3d model creation such as laser scanning technology, photogrammetry and 3d modeling (architectural reconstruction techniques) [22]. nonetheless, the fusion of these techniques during post-processing may occur, thus leading to the creation of highly complex models with a diversity of information encapsulated within a single file. the development of a web-based application for user access and interactive exploration of threedimensional models has been studied and worked upon since 1995. the web 3d consortium composes open standards for real time 3d data and models exchanged over the internet. virtual reality modeling language (vrml) became the first web 3d format [23]. the latest successor of vrml is x3d, providing a flexible solution for real time 3d representation and communication for medici. the system architecture of the 3d processing pipeline consists of open source tools and free software, thus providing full transparency on adopted methodology and data processing methods providing a cost effective solution both for server and client. the main feature of this web vr system previewer is to provide the user with a completely new visit experience based on a free interactive exploration interface of the object (i.e., not constrained by any predefined pathway) and the opportunity to get more detailed information on specific parts of interest. medici currently supports 3d file formats that are widely used in ch such as wavefront (obj), polygon file format (ply), x3d (x3d) format and 3d models embedded in pdf (3dpdf). models in the obj or ply format are converted to x3d and then translated to their center of rotation. “virtual scenes” containing multiple objects can also be previewed in 3d fly-through and walk-through mode (primary mode is “examine” mode). if the model needs optimization for smooth transmission over the internet, it is converted in an adequate lod, before being converted to the x3dom format (that is, html5 files) and sent back to the server as a preview to be added to the dom of the 3d previewer and displayed upon request by the user. there are also previewers for 3dpdf (also used for general pdf viewing). one of them was written by ncsa and uses mozilla’s pdf.js library [24], while the other is a simple integration of adobe acrobat using the associated acrobat browser plugin. x3d extraction the x3d extractor is responsible for converting obj and ply models to x3d and converting them if needed through external calls to meshlab’s meshlabserver [25] and setting job identity flags on each of the models uploaded by the client. for the x3d extractor to work, meshlab must be installed on the extractor’s machine. the extractor accepts jobs from rabbitmq for obj, ply and x3d. however, if the 3d model is not in x3d form, meshlab’s meshlabserver command-line interface is called by the extractor to generate the equivalent x3d model. on the other hand, if the model’s file size is above a pre-register boundary size, the client is responsible for calling/activating the extractor program as an input parameter and a dynamically-generated meshlabserver re-meshing/simplification script is used. a new call to convert the model to the adequate lod is then generated according to the client’s preset boundaries in order to reduce its complexity and thus its size. finally, a flag is set on the job received by the extractor indicating that the first phase of the processing of the model was completed and the post-processed model is uploaded back to the web server as an intermediate result together with the model’s new job flag. the web server then transmits the intermediate result to the html5-x3dom extractor for preparation for front-end display html5-x3dom extraction converting x3d models to x3dom/html5 facilitates and optimizes their integration with the x3d previewer’s html dom structure. this is done by the x3dom extractor, which uses aopt, a command-line tool bundled with the instantreality framework [26]. for the html5-x3dom extractor to work, instantreality must be installed on the extractor’s machine. models with separate materials/texture files in special cases and in order to have obj and x3d files display their geometry with the adequate color information, models at times have separate material/texture files. medici 2.0 considers this case. those files can be still previewed correctly if they are uploaded contained in a zip file with their corresponding materials/textures. the web server uses a utility function that unzips the file and uses rules regarding the existence of files of certain formats in the zipped dataset to resolve the type of model contained in the zip file (whether it is a zipped obj, a zipped x3d, or something different). the type discovered is sent as part of the routing key to rabbitmq when the file’s extraction job is sent. after the automated post-processing of the model with its textures from the x3d extractor, the html5-x3dom extractor completes work on the x3d preview and sends back the html5 x3dom as the file’s final preview. x3dom previewer the x3dom previewer works by dynamically downloading the html5 file in which the x3d preview of the model is encapsulated, and embedding it into an x3d html5 element. afterwards, events generated by the new elements activate functions in the x3dom javascript library, initializing the model’s preview through an x3d scene. the preview allows (among other functions) rotating the model, zooming, panning, displaying model statistics, changing rendering geometry mode from default view to wireframe or vertex view, with or without texture, and surface depth map simulation. x3dom.js (http://www.x3dom.org/download/dev/x3dom.js accesses any image textures via a virtual directory structure generated on the web server for each preview, which, for files uploaded as zip files, simulates the original zip file’s internal directory structure. this structure is implemented with scala functions that take the parts of the virtual directory path as parameters and return the requested image texture after querying the database where the image textures are stored. these parameters include the (virtual) path to the image texture, as it was in the original zip file containing the textures of the original model. 3d model preview examples (with the depth maps and model statistics also being displayed) are shown in figures 6 and 7. figure 6. hellenistic-roman theater, 300 bc-365 ad, previewed in the web 3d viewer. courtesy cyprus dept. of antiquities-university of sydney. figure 7. ancient vessel from the pyrgos area of cyprus, previewed in the web viewer. courtesy of the 3d-coform eu project. rti the application of reflectance transformation imaging technology (rti) has offered great possibilities for research as well as for the documentation and preservation of cultural heritage objects and works of art. polynomial texture map (ptm) is a subset of the rti method and was developed by hewlett-packard imaging labs at the beginning of the past decade [27][28] and enhanced by the west semitic project at university of southern california (usc) and cultural heritage imaging (chi). ptm addresses the challenges in the photography of objects’ faded, damaged or badly preserved surfaces especially when they feature inscription, decorative patterns and designs. the surfaces of an array of archaeological objects and works of art such as stone or clay, marble, plaques, coins, paintings, mosaics, sculpture, jewelry and other small objects (see figure 8), present optimal study cases for the utilization of rti technology. figure 8. an ancient replica coin that shows the capabilities of rti photography. self-shadows and interreflections are such derivatives presented on the upper-left corner in comparison with the default mode on the lower-left side that has no such properties. rti/ptm viewer the ptm algorithm synthesizes the data from images taken under varying lighting directions to create a single image that can be examined on a rti/ptm viewer with a “virtual spotlight”. the viewer allows the user to move the light angle intuitively in real time, so that the combination of light and shadow representing the relief features of the object’s surface can be freely altered. rti also permits the enhancement of the subject’s surface shape, color and luminance attributes, which allows one to extract detail out of the surface that cannot be otherwise derived (see figure 9). figure 9. this snapshot is taken from the inscriptifact desktop rti viewer developed by west semitic project at usc where the user is able to interact with the artifact using a “virtual torch” for dynamic illumination and surface enhancement. medici currently supports viewing real-time web rti through a java applet developed by clifford lyon [29] (see figure 10). there are three stand-alone desktop viewers that can interpret rti/ptm files but for the web just the viewer mentioned above [30][31][32] is used. the current online rti viewer has pitfalls concerning incompatibility with current w3c standards. figure 10. replica of an ancient coin from petra, jordan, using the current ptm viewer. in the future, the current viewer will be replaced with one based on the spidergl library, which is based on webgl. this will allow direct execution of the viewer by the user’s browser without the need to use a third-party plugin, improving cohesion between the user’s graphics processing unit, the viewer, and the user’s browser, and also improving security. medici 2.0 will currently avoid the use of the rti java applet, but instead will use an innovative method for extracting 3d models from rti files for front-end previewing. based on this the user can recognize the object and then (down)load the original high-resolution rti file to his/her desktop for actual interaction. rti metadata rti technical metadata is extracted by a standalone preprocessor in the same manner as metadata for traditional web images are. the preprocessor downloads the rti file from the web server and then reads its topmost lines for the types of metadata included in the most common rti file formats as defined in the ptm file format specification by hewlett-packard. the metadata is parsed to json and returned to the web server to be associated with the rti graphic. extracting 3d models from rti the ability to interactively change light direction and apply various filters makes rti one of the best techniques for examining archaeological artifacts. human perception of highlights and shadows in a 2d image of a surface helps the viewer interpret the surface topology. however, this ability to perceive surface topology varies from person to person. also, some surfaces with varying contrast and surface characteristics might be difficult to perceive even with dynamic lighting [33]. rtis capture both color and geometric properties of the object, which makes it possible to use them to reconstruct 3d approximations of the original surfaces. doing this greatly improves the perception of the artifacts’ surface details. the approach used is based on an approach similar to that of photometric stereo [34] and uses three 2d texture maps that are extracted from the rti to then reconstruct the final 3d surface given known lighting directions. luminance info is used to estimate a normal map which is integrated to obtain a height map. the height map is used to build a 3d surface by shifting vertices of a delaunay mesh. color info is used to extract a uniformly lit diffuse map that is used for surface texture mapping. maps used diffuse map diffuse maps (a.k.a. albedo maps) define the main color of the surface. a good diffuse map should contain only color information without any directional light effects, inter-reflections, specular highlights or self-shadowing. having any of these in a diffuse map means that the object will respond in an incorrect way to virtual incident light; such as casting shadows in the wrong directions or showing highlights where no direct incident light hits the object. ptm allows the extraction of accurate diffuse maps because the luminance and chromaticity information are stored separately. in lrgb ptm format the color information exists out of the box. for rgb ptm format, similar results can be obtained by casting light perpendicular to surface per texel to make sure all texels get the same amount of light. this results in a uniformly lit texture that is ideal for use as a diffuse map. normal map a normal map is a texture map containing surface normals at each texel stored as rgb. directional lighting information for each pixel is already stored in a ptm, which makes it possible to get a very good estimate of the surface normal at that pixel. since luminance is maximum when incident light is perpendicular to surface (i.e. incident light vector = surface normal), the surface normal at each texel can be estimated as the orientation maximizing light response. height map height maps are grayscale texture maps in which each pixel’s white level corresponds to the height value of a vertice of a 3d grid mesh (usually in the z direction). height maps used here are generated by integrating normal maps obtained as described in the previous section. normals at every surface point are perpendicular to the height map gradient. integration is not always guaranteed to yield precise results since it is based on an estimated normal map. also, information about surface discontinuities is lost in normal maps. the 3d models generated are good approximations of the real surface. height map generation is implemented in an iterative manner in which each iteration improves contrast between low and high points. height map pixels are initiated at zero height. on each new iteration, each pixel’s height is slightly modified according to the slopes of the surrounding pixels’ normals and their heights in the current iteration. contributions from all eight surrounding pixels will be averaged and added to the current height. see figure 11 for a summary of surrounding pixels, contributions, and associated signs, where nx and ny are the x and y components, respectively, of the surface normal. figure 11. contributions of surrounding pixels to the current pixel height when generating the height map from the normal map. testing the algorithm in testing done by bibliotheca alexandrina it research center (ba), a 2d delaunay triangulation was used to generate rectangular grids that were deformed using resulting height maps, and texture-mapped using diffuse and normal maps. the quality of the generated models improved proportionally with the number of iterations used. iterations beyond 100 000 iterations had no noticeable effect. further testing of the algorithm took place at the cyprus institute’s imaging cluster for archaeology and cultural heritage (icach) [35]. the testing showed that for low and mezzo-relief rti the algorithm generated 3d models of good quality with no more than 10 000 iterations needed. an example of testing the algorithm is shown in figure 12. figure 12. example of a ptm file and its 3d model generated by the rti-to-3d algorithm. implementation the algorithm will be implemented as a standalone command-line program that will take as input the desired number of height map generation iterations, the desired height modifier for the generated 3d model, and the rti file and output the generated 3d model. this program will be called using a standard extractor taking the above input parameters as its input. the result of the extraction will be sent to the 3d extractors for further processing, as it is for all 3d models. future directions immediate plans the community-generated metadata search will be enhanced to enable use of logical expression-like query formulation to generate queries with and, or, not operators. technical metadata will also be searchable. integration of more meshlab features in medici will be attempted (if supported by x3dom). these may include shading modes, a measuring caliber, community-generated annotations on the model display, and being able to change the view’s light source. support for more file formats and model types will be added, including printer-ready stereolithography (stl) diagrams, virtual reality spherical panoramas, and videos. further designs medici metadata will be able to be integrated with inscriptifact and other cultural heritage repositories. concerning 2d high-resolution imaging, the iipimage framework [36] will serve as a mediator for flexibility on the presentation of more specialized ch imaging (e.g., able to support multispectral imaging and real time annotation). regarding 3d, technical metadata extractors may be added and also the capability to generate 3d models from image datasets produced by photogrammetry similar to the arc 3d web service [37]. furthermore, the use of medici’s interactive system could therefore be potentially extended to more complex virtual exploration such as a digitized archaeological site, to serve not just as a previewer but as an intuitive highly vr environment for web and thus becoming an interactive learning environment, a “virtual world” [38]. a spidergl-based viewer will replace the current rti java applet once the viewer is developed and made stable by the digital ch communities. medici will also be able to extract semantically-meaningful features for content-based comparison (e.g. textual data from handwritten text). this will be made possible by constructing descriptors of each file according to derivations of semantically-meaningful features from a file’s data[39][40][41][42]. the generated descriptors for each file can be compared with descriptors of the same type generated from search query data. conclusion though still in development, medici 2 already supports a broad range of throughput-intensive research techniques and community data management. many file and dataset formats can be uploaded, analyzed and visualized. these include the latest formats used by the ch scientific community, such as large, detailed 3d models and rti, as well as the ever-present large images and audio/video. not only can medici 2 extract technical metadata from files and datasets, it also allows searching of metadata and social metadata generation according to each implementer’s user-input metadata schema. the above, together with the many important additions scheduled for the future, uniquely sets medici 2 apart in satisfying the ch scientific community’s dataset management, analysis and visualization needs. acknowledgments the authors thank the cyprus institute, the linksceem-2 project and its partners, the national centre for supercomputing applications (university of illinois) (ncsa) and bibliotheca alexandrina as organizations for providing them the means and guidelines enabling their contributions to the development of medici. moreover, they thank every member of staff in cyi, ncsa and bibliotheca alexandrina who provided them with user requirements and technical assistance. notes [1] l. hunt, m. lundberg, and b. zuckerman, “inscriptifact: a virtual archive of ancient inscriptions from the near east”, in international journal on digital libraries, vol. 5, no. 3, pp. 153-166, may 2005. [2] l. marini et al, “medici: a scalable multimedia environment for research”, white paper, presented at microsoft escience workshop 2010, berkeley, ca, oct. 2010. [3] the international committee for documentation of cultural heritage (cipa). http://cipa.icomos.org [4] m. mudge et al, “image-based empirical information acquisition, scientific reliability, and long-term digital preservation for the natural sciences and cultural heritage”, in eurographics 2008, hersonissos, greece, 2008. [5] zoom.it http://zoom.it [6] j. behr, p. eschler, y. jung, and m. zöllner, “x3dom: a dom-based html5/x3d integration model”, in proc.of the 14th int. conf. on 3d web technology, darmstadt, germany, jun. 2009, pp. 129-135. [7] s. tomar, “converting video formats with ffmpeg”, in linux j., vol.2006, no.146, pp. 10, jun. 2006. [8] m. di benedetto, f. ponchio, f. ganovelli, and r. scopigno, “spidergl: a javascript 3d graphics library for next-generation www”, in proc. of the 15th int. conf. on web 3d technology, los angeles, ca, jul. 2010, pp. 165-174. [9] a. videla and j. j. w. williams, rabbitmq in action: distributed messaging for everyone, shelter island, ny: manning, 2012. [10] r. t. fielding, “architectural styles and the design of network-based software architectures”, ph.d. dissertation, dept. comput. sci., univ. california irvine, irvine, ca, 2000. [11] p. hilton, e. bakker, and f. canedo, play for scala (early access edition), shelter island, ny: manning, 2012. [12] n. leroux and s. de kaper, play for java (early access edition), shelter island, ny: manning, 2012. [13] a. leff and j. watson, “web-application development using the model/view/controller design pattern”, in proc. of the enterprise distributed object comp. conf. 2001, seattle, wa, sep. 2001, pp. 118-127. [14] jquery. http://jquery.com/ [15] s. weibel, “the dublin core: a simple content description model for electronic resources”, in bulletin of the amer. soc. for inform. sci. and technology, vol.24, no.1, pp. 9-11, oct./nov. 1997. [16] m. doerr, c. e. ore, and s. stead, “the cidoc conceptual reference model: a new standard for knowledge sharing”, in tutorials, posters, panels and industrial contributions at the 26th int. conf. on conceptual modeling, auckland, new zealand, nov. 2007, pp. 51-56. [17] r. tarjan, “depth-first search and linear graph algorithms”, in siam j. comput., vol.1, no.2, pp.146-160, jun. 1972. [18] deep zoom-features-microsoft silverlight. http://www.microsoft.com/silverlight/deep-zoom/ [19] m. still, the definitive guide to imagemagick, new york city, ny: apress, 2005. [20] drewnoakes.com – jpeg exif / iptc metadata extraction in java. http://drewnoakes.com/code/exif/ [21] a. guarnieri, f. pirotti, and a. vettore, “cultural heritage interactive 3d models on the web: an approach using open source and free software”, in j. of cultural heritage, vol. 11, no. 3, pp. 350-353, jul.-sept. 2010. [22] f. bernardini and h. rushmeier, “the 3d model acquisition pipeline”, in comput. graph. forum, vol. 21, no. 2, pp. 149-172, jun. 2002. [23] web3d consortium: open standards for real-time 3d communication. http://www.web3d.org/standards/version/v3.3 [24] pdf.js by andreasgal. http://mozilla.github.io/pdf.js/ [25] p. cignoni et al., “meshlab: an open-source mesh processing tool”, presented at eurographics italian chapter conf. 2008, salerno, italy, jul. 2008, pp. 129-136. [26] j. behr, u. bockholt, and d. fellner, “instantreality — a framework for industrial augmented and virtual reality applications”, in virtual reality & augmented reality in industry. berlin, germany: heidelberg, 2011, ch. 5, pp. 91-99. [27] t. malzbender, d. gelb, and h. wolters, “polynomial texture maps”, in siggraph ’01 proc. of the 28th annu. conf. on comput.. graph.. and interactive techniques, los angeles, ca, aug. 2001, pp. 519-528. [28]t. malzbender, d. gelb, and h. wolters, “polynomial texture map (.ptm) file format”, client and media systems laboratory, hp laboratories, palo alto, ca, tech. rep. hpl-2001-104, apr. 2001. [29] ptm web viewer by clifford lyon. http://materialobjects.com/ptm/ [30] hewlett-packard labs ptm desktop viewer. http://www.hpl.hp.com/research/ptm/downloads/download.html [31] cultural heritage imaging (chi) rti desktop viewer. http://culturalheritageimaging.org/technologies/rti/ [32] inscriptifact desktop viewer. http://www.inscriptifact.com/instructions/ [33] j. t. todd, “the visual perception of 3d shape.”, in trends in cognitive sci., vol. 8, no. 3, pp. 115-121, mar 2004. [34] r. j. woodham, “photometric method for determining surface orientation from multiple images.”, in optical eng., vol. 19, no. 1, pp. 139-144, feb 1980. [35] imaging cluster for archaeology and cultural heritage (icach), science and technology in archaeology research center, the cyprus institute, http://imlab.cyi.ac.cy/ [36] iipimage framework. http://iipimage.sourceforge.net/ [37] arc 3d web service. http://homes.esat.kuleuven.be/~visit3d/webservice/v2/index.php [38] b. harper. j. g. hedberg and r. wright, “who benefits from virtuality?”, in comput. & educ., vol. 34, no. 3/4, pp. 163-176, apr./may 2000. [39] l. marini et al, “versus: a framework for general content-based comparisons”, in ieee escience, chicago, il, 2012. [40] l. diesendruck et al, “digitization and search: a non-traditional use of hpc”, in ieee escience workshop on extending high performance computing beyond its traditional user communities, chicago, il, 2012. [41] l. diesendruck et al, “a framework to access handwritten information within large digitized paper collections”, in ieee escience, chicago, il, 2012. [42] l. diesendruck et al, “using lucene to index and search the digitized 1940 us census”, in extreme science and engineering discovery environment, san diego, ca, 2013. about the authors constantinos sophocleous, computation-based science and technology research center, the cyprus institute, nicosia, cyprus luigi marini, national center for supercomputing applications, university of illinois at urbana-champaign, urbana, illinois, usa ropertos georgiou, science and technology in archaeology research center, the cyprus institute, nicosia, cyprus mohammed elfarargy, international school of information science, bibliotheca alexandrina, alexandria, egypt kenton mchenry, national center for supercomputing applications, university of illinois at urbana-champaign, urbana, illinois, usa subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – wikidata: a platform for your library’s linked open data mission editorial committee process and structure code4lib issue 40, 2018-05-04 wikidata: a platform for your library’s linked open data seized with the desire to improve the visibility of canadian music in the world, a ragtag band of librarians led by stacy allison-cassin set out to host wikipedia edit-a-thons in the style of art+feminism, but with a focus on addressing canadian music instead. along the way, they recognized that wikidata offered a low-barrier, high-result method of making that data not only visible but reusable as linked open data, and consequently incorporated wikidata into their edit-a-thons. this is their story. by stacy allison-cassin‡, dan scott.‡ ‡ both authors contributed equally to this work. introduction creating and using linked open data (lod) in library and glam (galleries, libraries, archives, museums) projects has historically been associated with a high level of institutional requirements. erik et al (2015 [3]) asserted “the fact that lam institutions are still having to select triplestores, sparql engines, indexing platforms, and other services means that there is still a relatively high bar for institutions to cross in taking up ld projects”. the requirement to select, host, and administer all of these systems establishes technical and resource barriers that can prevent organizations and individuals from participating in lod entirely (goddard and byrne, 2010 [5]). creating and publishing lod has traditionally required technical skills to transform relational data, to support content-negotiation and alternate serializations, and to understand vocabularies and ontologies typically documented in rdfs or owl. wikidata, launched in 2012 by the wikimedia foundation as the machine-readable store for all wikimedia foundation projects, is a freely available hosted platform that anyone–including libraries–can use to create, publish, and use lod. powered by blazegraph, the platform offers a triplestore and high-availability sparql endpoint that (as of april 2018) has served roughly 3 million queries per day over the past year (wikimedia foundation, n.d. [11]); a full text search engine; and is administered by the wikimedia foundation. its vocabulary is published and editable alongside other items in the platform using the same relatively user-friendly interfaces. in effect, wikidata has responded to the barriers identified by erik et al and goddard and byrne by providing a ready-made platform for any person or organization that wants to create, publish, and use lod, including libraries. in their 2016 ifla discussion paper, bartholomei et al noted “[t]he potential of wikidata to draw linked open data and linked open data authorities together across the world’s languages and many different ontologies and taxonomies has enormous potential to support researchers around the world.” [1] the platform is increasingly important as a general lod resource and as a “linking hub”, recognized in 2013 by klein and kyrios as they worked on integrating viaf data into wikipedia [7]. at the first wikidata conference, van veen (2017 [10]) boldly suggested wikidata could be both a linking hub and source of library authority data. as of february 2018, wikidata now offers links to external data with more than 2,500 identifiers. an international, multilingual, community-based project, wikidata is a practical choice for use by libraries, and requires all contributions to be licensed under the creative commons cc0 “no rights reserved” licence. this licence allows the contents (49 million items as of february 2018) to be used in any project without the cumbersome attribution requirements of other open data licenses, and ensures that every contribution to the repository broadens the range of freely available data. background in 2015 allison-cassin began a lod project focusing on the mariposa folk festival. the project initially focused on creating a dataset of entities related to the festival. project plans included the enhancement of existing data through biographical snippets drawn from wikipedia via dbpedia and links to bibliographic data stores such as the virtual international authority file (viaf). early findings of the project revealed there were few wikipedia articles and little to no available data in standard lod stores such as viaf. this was especially the case for performers who were not white american males. as one moves into other categories or intersections of categories, for example canadian women performers, the amount of data became drastically reduced. this echoes the findings of pattuelli et al (2017), who reported a lack of linkable data for and about female jazz performers in contrast to the availability of data for male performers. the scarcity of linkable data is correlated to areas where structural inequalities in wider society exist. inspired by the effectiveness of focused edit-a-thon campaigns such as art+feminism to increase content in areas where representation is problematic, allison-cassin organized a year-long “music in canada @ 150” campaign to coincide with the 150th anniversary of the confederation of canada to organize a group of music librarians across canada to add content on canadian music to wikipedia and wikidata. the music in canada @150 campaign did not specifically engage with first nations, métis and inuit (fnmi) communities as the focus was on music librarians and cataloguers and their communities (i.e. music students and music faculty.) however, the organizers did try to encourage participants to focus on local collections and communities, with particular attention to those who might be considered underrepresented in dominant reference sources. the campaign also became a way to experiment with the creation of linked data and introduce wikidata and structured data to an audience that was primarily made up of participants unfamiliar with lod and metadata library practices. lowering barriers to lod beyond relieving institutions of the need to maintain their own lod infrastructure, wikidata is designed to be usable by novices and support their growth as editors, community members, and contributors to the platform and lod. ease of use with progressive capabilities compared to classic methods of generating lod, such as hand-crafted rdf/xml or automated transformations of legacy relational data that require developer support, wikidata’s editing interface (figure 1) presents a comparatively simple key-value approach for creating and editing statements. similar to the vitrolib lod editor’s custom work forms (khan et al, 2018[4]), it uses autocompletion to suggest matching items for the values of most statements. this approach makes it easier to create and edit lod: contributors are not required to comprehend lod principles before they can add a statement to an existing item, or before they can create a new item where one does not already exist. during the workshops, we demonstrated adding a statement such as participant for a given iteration of a music festival: the result would either be that the desired value to complete that statement already existed in wikidata; or if the value did not exist and needed to be added, led naturally to an opportunity to show the creation of a new item from scratch. after creating an item with a label, and (optionally) a description and aliases, we showed that the new item had its own uri and was immediately ready to both fulfill the statement from the initial item and to be fleshed out with more structured data. as editors gained experience, they were able to engage in more complex editing tasks such as entering multiple values for a single statement, adding qualifiers like start and end dates to statements, and adding references to provide verifiability for statements. from an editor’s perspective, they are merely qualifying their statement that a given person was part of a band with an additional assertion that the member’s start date was 2007. the corresponding lod generated by these more advanced tasks is highly complex, requiring reification and a mix of namespaces, but none of this is evident to the editors, who are guided by the user interface in relatively user-friendly forms located within the same statement box. figure 1. wikidata editing interface for northern lights festival boréal 2017 wikidata also errs on the side of usability and progressive disclosure by allowing editors to create statements that initially violate property constraints, such as establishing inverse relationships between items or allowing only one item to use a unique value. when the editor reloads the page, wikidata displays a warning next to the statement indicating that a violation may need to be resolved, with a link to the documentation describing the potential violation. in this way, the platform introduces the basic ontological principle that properties and classes can have additional requirements without frustrating new contributors at the point of data entry. as they gain experience with the platform, editors can generate lists of properties that violate constraints for a given project and address those violations by either manual or automated means: for example, missing inverse relationships could easily be added to target items. contextual vocabulary discovery many lod vocabularies separate their development, documentation, and community discussion, an approach which makes it challenging for newcomers to find help in using or contributing to a given vocabulary. for example, all decisions about the rda vocabulary are made by the rda steering committee, while code is developed on github, documentation is hosted at http://www.rdaregistry.info/, and usage is discussed primarily on an ala mailing list. the wikidata platform, in contrast, aims to be self-documenting: vocabulary classes and properties of items such as related wikidata properties are all described as entries in the repository itself and displayed in a format readable and editable by humans (figure 2). during the workshops, we used this visual evidence to highlight wikidata’s knowledge organization system based on items that serve as classes, instances, and properties: class items organize wikidata’s items hierarchically and can be distinguished by the presence of one or more subclass of statement values as their primary property instance items provide specific data about one individual item and can be distinguished by the presence of one or more instance of statement values as their primary property property items classify statements about items, including instances, classes, and properties, and can be distinguished by their identifier prefix of p instead of q for class and instance items wikidata surfaces the evolution of its vocabulary in various ways, each of which invites editors to explore decisions that have been made, to learn a little more about the vocabulary, and to help organize wikidata’s knowledge as well as contribute data. every item displays a change log via a view history link. similarly, a discussion link puts the community’s ongoing decision-making processes for a given item in context and invites editors to join the conversation and contribute their perspectives. editors comment based on their domain expertise, the project’s prior practice, and general ontology engineering principles. property items include a special property property proposal discussion (p3254) that links to the original proposal and discussion for the creation of the property, thereby providing examples for editors that might want to propose a new property. figure 2. related external and wikidata properties for the genre (p136) property the same vocabulary data, which can include more abstract statements such as equivalent property (p1628) and external subproperty (p2236) for aligning the vocabulary with external vocabularies, is also available in machine-readable serializations such as json-ld, turtle, and rdf/xml. the following example shows the subset of data that corresponds to the human-readable representation displayed in figure 2: $ curl -l -h "accept: text/turtle" https://www.wikidata.org/entity/p136 @prefix wd: <http://www.wikidata.org/entity/> . @prefix wdt: <http://www.wikidata.org/prop/direct/> . wd:p136 a wikibase:property ; wdt:p1628 <http://schema.org/genre> ; wdt:p1659 wd:p921, wd:p415, wd:p135 ; wdt:p2875 wd:q20990014 ; wdt:p2236 <http://rdf.freebase.com/ns/book.book.genre>, <http://comicmeta.org/cbo/genre> . figure 3. turtle serialization of related properties similarly, the wikidata platform surfaces domain and range constraints directly in vocabulary items as statements, shown in figure 4. figure 4. domain and range constraints for the genre (p136) property every item has a discussion page, which often includes autogenerated documentation providing useful links for displaying instances of a given class, showing class hierarchies, and listing reports of constraint violations. discussion pages can also host conversations between users about topics and issues related to the item. for example, the discussion page for singer (q177220) contains a topic from 2013 titled “vocalists” in which littledogboy asserts “singer and vocalist are synonyms.” and infovarius replies “rapper is not singer but vocalist.” these exchanges provide insight into the wikidata community’s evolving positions on items and their usage, as well as the challenges of working in an international, multilingual context. proposals for new properties follow an open community process, and the results of the proposals are archived; for example, you can search for all property proposals relating to music to see the history of their development and the outcomes. any wikidata editor can propose a new property, contribute their expertise to the discussions, suggest amendments or alternatives, and indicate their support or opposition to the proposal. if a consensus is reached, as in the case for sheet music (p3030) (which had been proposed as “musical score”), the new property may be created and used. just-in-time vocabulary evolution given the community-driven process of ontology development, alternative and potentially conflicting modes of representation can evolve over time. for example, the college library (q1622062) and academic library (q856234) types are currently both subclasses of the library (q7075) type, but follow separate hierarchies: library research library academic library school library college library a wikidata editor trying to represent a library attached to an institution with the word “college” in its name must decide whether to classify it using the category most closely matching the institution’s name (college library), a category reflecting the academic and research orientation of the library (academic library), classify it as an instance of both academic library and college library, or engage with the community to resolve the issue for the entire platform by either merging college library with academic library or making college library a subclass of academic library. viewing the history of college library shows that in 2017 it changed multiple times between being a subclass of library, scientific library, school library, and academic library. while a wikidata editor may feel empowered recognizing that they have the ability to directly improve the vocabulary, they may also be reluctant to attempt to resolve issues with an item that has a complex history. many editors choose a prudent strategy of satisficing–choosing an existing option that is close enough to their intended meaning to convey the relationship to the data recipients–and trust in the broader community to resolve the potential issues in the future. uris and linking while wikidata’s lower barriers to accessing and editing existing lod is helpful, libraries still need uniform resource identifiers (uris) to link their own collections and contributions of data. unfortunately, uris are still lacking for much of the long tail of topics, domains and areas of local interest that are central to the unique materials held by libraries and archives. libraries can mint their own uris, of course, but they may not have the resources to preserve the integrity of their namespaces over time, and those smaller namespaces may lack visibility. creating an item in wikidata immediately mints a uri with the pattern https://www.wikidata.org/entity/q####, and offers the advantages of relatively higher visibility and stability. the music in canada @ 150 campaign organizers agreed that, while some of wikidata’s music-related data (such as its almost 1,500 loosely organized music genres) might concern classically trained music cataloguers, it held great potential for addressing the social justice issues that had motivated allison-cassin to launch the campaign in the first place. as scott wrote to the organizing committee on 2017-05-11: the more minute details of, say, merging and creating music genres is definitely interesting and worthwhile, but when the entity for multi-juno award-winners a tribe called red doesn’t link to ian “dj ndn” campeau or any of its other musicians because they don’t exist as entities in wikidata, or reflect those awards, i want to tackle the more fundamental work of visibility first! in preparation for the music in canada @ 150 campaign some initial analysis was conducted of coverage in wikidata. carolyn doi conducted a survey of associate composers of the canadian music centre (cmc) and with a sample of 3.85% of total entries in the cmc database only 40% had pages in wikidata. findings were similar with the mariposa folk festival participants for 1961 to 1971. of the 469 performers, only 33% had an item in wikidata, and 49% in viaf. even more problematically, initial surveys of the native peoples area, a section of the festival which ran between 1972 and 1978, of 42 individual participants only 5 had a match on viaf. these two examples reinforce the ways that many library catalogues and wikipedia continue to reify dominant cultures and publication practices. using wikidata to create structured data is a way local communities can have a global impact. the canadian music @150 campaign was specifically focused on creating content about notable local musicians, composers and organizations that little visibility in libraries and online. the juno awards, “presented annually to canadian musical artists and bands to acknowledge their artistic and technical achievements in all aspects of music” (“juno award”, n.d. [6]), represent a base level of canadian culture for which information should be available in a global context. many groups accordingly focused on filling in information about juno award winners to local bands and traditional musicians. the group at memorial university in newfoundland focused on creating wikidata items and created 40 items, linking local musicians from wikidata out to the lod cloud. while these actions individually can seem small, they increased the amount of open data and helped create a community practice. since the campaign, allison-cassin has been working on wikidata and content related to indigenous communities in north america. in 2017, the association of research libraries approved a project focused on social justice, linked data, indigenous peoples and wikimedia [9]. this project is specifically intended to engage with indigenous communities in the creation of data. a wikidata project was started by allison-cassin as a place to collect and discuss properties, data models and foster community. allison-cassin is also working with indigenous community members, local glam institutions and wikimedia canada to start “indigiwiki” to work on indigenize wikimedia projects. the first meeting was held at the toronto reference library in march, 2018.[13] these activities are intended to not just add to the availability of linkable data, but to actively engage communities in the description and content itself. this approach builds on the example of other projects actively engaged in decolonization of wikimedia projects and the web, such as whoseknowledge? and wikimedia canada’s aboriginal community outreach projects to give greater visibility to indigenous peoples and culture. given its visibility as a central lod platform, wikidata’s uris and data are being used in many projects, including the glam sector in projects such as snac, and in commercial efforts such as songkick. recognizing the increased visibility and likelihood of interlinking that results from uploading vocabularies to wikidata, europeana (2017 [2]) has called on the glam community to “get [their] vocabularies in wikidata.” this acknowledges the importance of local data about collections for enriching the network, as well as the current challenge of finding sources to which those collections can be connected. with wikidata acting as a linking hub for locally held data, the “there’s nothing to link to” problem becomes much easier to address. wikidata as structured data community outreach wikidata is a good platform for helping librarians, library staff and community members to understand structured data, the impact of structured data on the web, and the wikimedia foundation’s rationale for starting the wikidata project in 2012. the music in canada @ 150 campaign included wikipedia as well as wikidata, and was concerned about surfacing content in both of canada’s official languages (french and english). we highlighted the potential benefit of adding structured data to wikidata specifically by showing the discrepancies between band member timelines that almost inevitably creep into the over 200 language variants of wikipedia due to the complex timeline markup and the manual effort of synchronizing the markup across languages. in figure 5, a snapshot of english wikipedia shows only 8 members as having participated in the québec-based band voivod, while in figure 6 the french wikipedia entry captured at the same time shows 10 members. if the corresponding wikidata item listed the participants with qualifiers for start and end dates, aliases, and instrument played, then the timelines could instead be automatically generated for all wikipedia instances, ensuring that a single edit to wikidata would be reflected in all of the wikipedia instances. figure 5. voivod band member timeline (from english wikipedia) figure 6. voivod band member timeline (from french wikipedia) the canadian music campaign reassured those who do not have a background in metadata that they can contribute structured data; specifically that subject specialists can make important contributions through their domain knowledge. along with band member timelines, the workshop leaders showed how wikipedia’s {{authority control}} macro (“wikipedia:authority control”, n.d. [12]) automatically displays selected external identifiers such as library of congress and virtual international authority file (viaf) stated in the wikidata item associated with a given wikipedia article, and the more than 100 wikipedia infoboxes that draw data partially or entirely from the article’s associated wikidata item. the english wikipedia article for the band voivod had manually asserted in its infobox that the official website for the band was http://www.voivod.com/–which is just a page that loads http://www.voivod.net/ (the uri that had been stated in the wikidata item) in an <iframe>–while the french wikipedia article had asserted the website was http://www.www.voivod.net/ (which does not resolve). the infobox for the english article now uses the {{official url}} macro, and the french article now uses the {{#invoke:wikidata|framefun|formatstatements|property=p856|}} macro, to instead pull the correct website from the band’s wikidata item. participants only need a web browser and a basic guide, and through wikidata they can begin creating lod a statement at a time. our workshops did not focus on publishing lod, however: our materials for the workshops only mentioned lod once. instead, we showed how data stored in wikidata could be integrated into library systems (per seeing is believing: wikidata in production library systems), encouraged participants to collaborate on editing tasks as a local example of how others can extend and improve their data, and provided guides with suggested instance of values and recommended properties for each of the data types we covered. lod was therefore a secondary outcome of their contributions, not the primary goal for the participants–but we then briefly pulled back the covers to demonstrate how developers access the same data through apis and sparql queries. in the superconference workshop, we prepared sparql queries that used wikidata’s built-in map visualization to plot the canadian libraries found in wikidata, and recommended several items that were missing either coordinates or country: canada statements that participants could edit so that they would immediately be added to the map. wikidata’s sparql endpoint supports a number of useful visualizations to help build confidence in wikidata specifically, and lod in general, as a platform for library data, including maps for result sets that include coordinates, timelines for items with dates, galleries for items with images, and a variety of charts and graphs. to help support the music in canada edit-a-thon, scott created an introduction to wikidata workshop that included a summary of the most relevant properties for describing musicians and music festivals (workshop notes) that he compiled by comparing and evaluating existing wikidata items. while the discovery process can be enjoyable, it can also be frustrating, particularly if there is not a richly described existing template item that one can use as a model for creating new items. recipients therefore appreciated the prescriptive guidance as a pragmatic way to speed up their productivity rather than getting mired in discovery. for a subsequent workshop, scott created a more thorough guide for describing libraries and catalogues (excerpted in figure 7). figure 7. an excerpt of scott’s guide for describing libraries in wikidata music librarians and scholars participating in the project were creating or enhancing biographical articles in wikipedia. we paired this with manual revision of the wikidata items in focused areas and encouraged participants to add standard identifiers for sources such as viaf, musicbrainz, and songkick. in the canadian context, it was useful to demonstrate the way wikidata can assist with bilingual work. it also facilitated more general conversations about openly available and licensed data, and how these approaches contrast to data in proprietary systems. seeing is believing: wikidata in production library systems library workers sometimes understandably question the call to add enriched metadata–for example, relator codes or rda elements to marc21 records–if they do not immediately result in a change to the publicly visible display. similarly, one of the challenges of embracing lod is demonstrating why that effort improves the experience of library users. at the 2017 evergreen international conference, scott witnessed a challenge to the efforts the open source library system developers had been putting into enhancing its support for lod, so as he prepared his workshop materials for the music in canada edit-a-thon training session, he built a proof-of-concept extension to evergreen (his library’s public catalogue) that would integrate musician and band data from wikidata to create infocards in real-time within the detailed display of music records, as seen in figure 7. the code for the standalone proof-of-concept, written in vanilla javascript, is available at https://gitlab.com/denials/wikidata-music-infocard. the production code used in laurentian university’s catalogue can be found in a feature branch of the evergreen git repository. figure 8. laurentian university library catalogue showing wikidata infocards a single javascript file, added to the record display page, leverages the schema.org structured data that evergreen already expresses in its html by determining if the page describes a schema.org musicalbum type. if so, it iterates through each of the contributors to the resource to add a wikidata icon that, when clicked, checks wikidata for a musician or band with a matching name and retrieves a subset of the possible information for display as an infocard. figure 9 shows a simplified html example of the evergreen page: <body vocab="http://schema.org/" typeof="musicalbum"> <h1 property="name">the 1985 juno awards collection</h1> <section id="contributors"> <h2>contributors</h2> <div typeof="person" property="contributor" resource="#schemacontrib3"> <span property="name">silver, liberty</span> (<span property="description">singer</span>).</div> <div typeof="person" property="contributor" resource="#schemacontrib8"> <span property="name">lang, k.d. (kathryn dawn),</span> <span property="birthdate">1961</span>-</span> (<span property="description">singer</span>).</div> <div typeof="organization" property="contributor" resource="#schemacontrib10"> <span property="name">the family brown</span> (<span property="description">performer</span>).</div> </section> figure 9. simplified html for evergreen music album example to encourage reuse and adoption, scott wrote the javascript without using external libraries, and factored out the variable elements of the implementation into five css selector paths, per figure 10: /* path for the schema.org type declaration */ var type_path = '[typeof~="musicalbum"]'; /* list of elements describing contributors */ var contributors_path = 'div[resource^="#schemacontrib"]'; /* path to find the name within each contributor element */ var contributor_name = 'span[property="name"]'; /* within each contributor element, where we can append a wikidata clickable icon */ var name_path = 'span[property="description"]'; /* path for appending the requested infocards */ var infocard_location = '#contributors'; figure 10. css selector paths for finding contributors and appending infocards wikidata offers several different methods of accessing its data, but for this implementation scott chose the sparql endpoint at https://query.wikidata.org/sparql, requesting the json-ld serialization for easy manipulation in javascript. the sparql query in figure 11 is implemented as a union of three different subqueries, to find musicians, bands, or musical ensembles with names or aliases that match the requested contributor’s name, and to potentially return supplemental data to include in the infocard, along with a possible link to a corresponding wikipedia entry (?wplink). select ?item ?itemlabel ?itemdescription ?image (group_concat(distinct ?instrumentlabel;separator="; ") as ?instruments) ?birthplace ?birthplacelabel ?website ?musicbrainz ?songkick ?twitter ?facebook ?wplink where { ?item rdfs:label|skos:altlabel|wdt:p1449 'a tribe called red'@en . # instance of = any subclass of band { ?item wdt:p31/wdt:p279* wd:q215380 . } union # occupation = any subclass of musician { ?item wdt:p106/wdt:p279* wd:q639669 . } union # instance of = any subclass of musical ensemble { ?item wdt:p31/wdt:p279* wd:q2088357 . } optional { ?item wdt:p3478 ?songkick } . optional { ?item wdt:p19 ?birthplace } . optional { ?item wdt:p1303 ?instrument } . optional { ?item wdt:p856 ?website } . optional { ?item wdt:p434 ?musicbrainz } . optional { ?item wdt:p2002 ?twitter } . optional { ?item wdt:p2013 ?facebook } . optional { ?item wdt:p18 ?image } . optional { ?wplink schema:about ?item . ?wplink schema:inlanguage "en" . ?wplink schema:ispartof <https://en.wikipedia.org/> . } service wikibase:label { bd:serviceparam wikibase:language "en". ?instrument rdfs:label ?instrumentlabel. ?item rdfs:label ?itemlabel. ?item schema:description ?itemdescription. ?birthplace rdfs:label ?birthplacelabel } } group by ?item ?itemlabel ?itemdescription ?image ?birthplace ?birthplacelabel ?website ?musicbrainz ?songkick ?twitter ?facebook ?wplink limit 10 figure 11. sparql query to find musician info based on matching name or alias scott showed an initial prototype to the cataloguers at his library and had a positive reception, as it demonstrated the value of not only being able to combine freely licensed data with their library’s catalogue, but of contributing to that body of data rather than keeping it confined within their own library system. later iterations received positive feedback at the music in canada @ 150 edit-a-thon training session and during a wikidata breakout session at swib17. one of the suggestions was to incorporate a description from wikipedia, if available, as wikidata item descriptions are deliberately terse: they are intended only to disambiguate items that have identical labels, so many descriptions are as simple as “canadian musician”. by comparison, wikipedia provides richer descriptions that normally include a brief amount of biographical information. having added the ?wplink attribute to the sparql query to retrieve a direct link to the associated english wikipedia article, scott used the mediawiki api to extract the opening text for article and add that to the infocard with a link to the article, per figure 12. function addwikipedia(musician, r) { var wpapi = 'https://en.wikipedia.org/w/api.php?origin=*' + '&amp;format=json&amp;action=query&amp;prop=extracts&amp;explaintext=true&amp;' + 'exintro=true&amp;titles='; var wplink = r.wplink.value; // strip the wp title from the link wptitle = wplink.substring(wplink.lastindexof('/') + 1); var req = new window.xmlhttprequest(); req.open('get', wpapi + wptitle); req.onload = function (evt) { var response = json.parse(req.response); var key = object.getownpropertynames(response.query.pages)[0]; var description = response.query.pages[key].extract; // grab the first line of the article // link to article via wikipedia icon // append to the infocard } req.send(); } figure 12. retrieving and displaying the wikipedia description results testing the widget with a sample of 100 records for music recordings held by laurentian university, a collection heavily weighted towards classical and jazz, showed that 69.8% of contributor entries (1xx / 7xx fields) in the marc records returned valid data from wikidata (spreadsheet). some of the reasons for failures to find a match in wikidata included: aliases missing from an existing wikidata item. for example, in the record on which figure 5 was based, the sparql query searched for “k.d. (kathryn dawn) lang” when the label for q230454 was “k. d. lang” and the only alias was “kathryn dawn lang”; simply adding an alias to the wikidata entry resolved the problem. errors in the marc record. in the same record, “orford string quartet” had been incorrectly transcribed as “oxford string quartet”; investigating the reason for the lack of a match resulted in a correction to the source marc record. failure of the code to correctly order names including more than one comma. for example, “marriner, neville, sir” resulted in a search for “marriner, neville, sir” instead of “sir neville mariner”. resolving these failures suggests that the match success rate would have been close to 80% in this sample of records. while the name-based match algorithm can be improved, the best path forward would be to reconcile the records against wikidata and add $1 subfields with the https://www.wikidata.org/entity/q### real world object uri to each of the matching 1xx/7xx entries. this would provide the classic advantages of authority control, prevent users from requesting an infocard for an entry with no match, and enable libraries to identify entries that either need to be corrected or that need to be reflected in wikidata. the remaining 20% of the sample that is not easily matched in wikidata represents much of the long tail of topics, domains and areas of local interest, such as french-canadian composers, missing from the cultures that have dominated wikidata and wikipedia, and therefore would benefit most from having effort invested in ensuring they are represented in wikidata as a first step towards global recognition of these locally notable contributors. future goals scott’s library is currently adding wikidata ids to the authors and journals in the canadian labour studies index database, with the goal of contributing to the wikicite project to build a bibliographic database in wikidata. he also plans to use wikidata as an lod platform for a collection of grey literature about the environmental reclamation of mining sites that would benefit from linkage to concepts and authors not generally available in the broader scientific literature, as well as storing statements about specific mine sites which are currently identified only by rough coordinates. longer term, scott hopes to enhance his library system’s marc records with wikidata uris in the $1 to identify real-world objects, following the recommendations of the pcc uri task group on uris in marc, surfacing those entries for which no match can be found as potentially valuable local information that his library will then prioritize for adding to wikidata. allison-cassin’s library is working on several initiatives related to wikidata. york is collaborating with the association of research libraries (arl) on the previously discussed project focused on exploring wikidata as a platform for social justice and lod. this project is using a test case approach to model inclusive, community-led metadata in archives and special collections through the creation of a dataset of just over 200 indigenous performers and assessing and expanding properties, qualifiers and other terminology. she is also exploring ways data and applications might be used by communities, students and scholars in a pedagogical environment. conclusion wikidata is one means by which libraries can surface the long tail of subject matter and address the systemic biases and exclusions often present in our systems of description and metadata and society at large. this works at several levels. it addresses the exclusion from participation in the creation of lod, allows for larger community participation, encouraging the participation of people beyond those involved in metadata or systems work to get involved and understand the role of structured data and allows for the creation of rich data that is often excluded or unavailable in traditional bibliographic data, allowing for a multiplicity of viewpoints and modes of description. this article described and reflected on some of the ways the authors have used wikidata as a low-barrier method for creating and using lod in libraries. the platform enables libraries and other glam organizations to easily publish lod for collections and content. there are many pathways to get involved in the wikidata community. creating and editing wikidata can begin with choosing a thematic area related to a collection of interest or group of scholars or individuals. by actively contributing data, creating applications participating in community initiatives and creating documentation libraries can help to improve the quality, impact and sustainability of the platform for libraries. tools and resources news and overviews: asof bartov’s “a gentle introduction to wikidata for absolute beginners [including non-techies!]” the wikidata mailing list is a weekly newsletter detailing events, resources and new properties and developments. there is an active facebook group and a twitter community around the #wikidata hashtag development: module:wikidata – english wikipedia module for accessing data from the wikidata item associated with a given article (or any article) reconciling datasets with wikidata via openrefine wikidata sparql query help wikidata is an instance of the wikibase software, a specialization of mediawiki, and supports the mediawiki api with a module that adds wikibase-specific “wb” actions: https://www.wikidata.org/w/api.php articles & blog posts of interest: the tom longboat awards as wikidata by mita williams research libraries and wikimedia: a shared commitment to diversity, open knowledge, and community participation by stacy allison-cassin wikidata in collections: building a universal language for connecting glam catalogs by alex stinson about the authors dan scott (https://dscott.ca/#i) is an associate librarian and chair of the department of library and archives at laurentian university, where he is responsible for systems and scholarly communications. he is passionate about software freedom and open data, and is perhaps most notable for his contributions to the evergreen library system and schema.org vocabulary. dan is still pumped after organizing (with a great deal of assistance) his faculty association’s quite successful strike in the fall of 2017. stacy allison-cassin is an associate librarian and digital pedagogy librarian at york university. previously she was the w.p. scott chair in e-librarianship and a cataloguing librarian for music materials and as co-founder of the scholarly communication initiative at york. stacy is passionate about metadata, music and critical theory. as an active member of the wikimedia community, stacy writes and contributes to articles and content on first nations, métis and inuit peoples and cultures as well as canadian music. references 1. bartholmei, stephan, rachel franks, james heilman, mylee joseph, vicki mcdonald, anna raunik, mia ridge, and mark robertson. “opportunities for academic and research libraries and wikipedia.” columbus, ohio, 2016. available from: https://2016.ifla.org/wp-content/uploads/sites/2/2016/08/112-iflawikipediaacademicandresearchlibrariesdiscussiodraft.pdf. 2. europeana. “get your vocabularies in wikidata…” europeana (blog), september 1, 2017. https://pro.europeana.eu/page/get-your-vocabularies-in-wikidata. 3. erik t, mitchell. “chapter 4. the evolving direction of ld research and practice.” library technology reports 52, no. 1 (december 17, 2015): 29–33. 4. khan, h. j., folsom, s., snyder, t., clark, r., kelley, b. a., & nichols, m. f. (2018). shacl semantics for form building. presented at the u.s. semantic technologies symposium 2018, wright state university, dayton, ohio. avaiable from: https://drive.google.com/file/d/1m_xhng8qyl7m9akvirsetfogesefs9oh. 5. goddard, lisa, and gillian byrne. “linked data tools: semantic web for the masses.” first monday 15, no. 11 (november 7, 2010). available from: http://firstmonday.org/ojs/index.php/fm/article/view/3120. 6. juno award. (n.d.). in wikipedia. available from: https://en.wikipedia.org/w/index.php?title=juno_award&oldid=828286208. 7. klein, maximilian, and alex kyrios. “viafbot and the integration of library data on wikipedia.” the code4lib journal, no. 22 (october 14, 2013). available from: http://journal.code4lib.org/articles/8964. 8. pattuelli, m. c., hwang, k., & miller, m. (2016). accidental discovery, intentional inquiry: leveraging linked data to uncover the women of jazz. digital scholarship in the humanities, fqw047. available from: https://doi.org/10.1093/llc/fqw047. 9. surfacing knowledge, building relationships: indigenous communities, arl and canadian libraries. availble from: https://scbrwiki.library.yorku.ca/. 10. van veen, t. (2017, october). wikidata as a universal (library) thesaurus. presented at the wikidata conference. available from: https://commons.wikimedia.org/wiki/file:wikidata_as_universal_library_thesaurus_-_tvv.pdf. 11. wikimedia foundation. (n.d.). wikidata query service (wdqs) endpoint usage. available from: http://discovery.wmflabs.org/wdqs/#endpoint_usage. 12. wikipedia:authority control. (n.d.). in wikipedia. available from: https://en.wikipedia.org/w/index.php?title=wikipedia:authority_control&oldid=833261359 notes 13. collaborators include trina gover, ryerson university and melanie ribeau, toronto public library. jamie lee morin, the indigeneous digital collections assistant came up with the term “indigiwiki” for the group. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – using web services for a mobile opac mission editorial committee process and structure code4lib issue 13, 2011-04-11 using web services for a mobile opac the purpose of this paper is to discuss the creation and intended evolution of the rice university mobile online public access catalog (opac). the focus of the article is on how sirsidynix’s symphony web services can be used to create a mobile opac. by denis galvin and mang sun background why create a mobile website the use of mobile technologies has exploded. in 2008 four in five people in the americas used a cellular phone, an increase from one in three in 2003 (international telecommunication union (itu), 2009). [1] for an academic university the numbers are even more staggering. according to rice university’s annual it survey in 2010, 100% of undergraduates brought a cell phone and 72% of them have a phone with a data plan (new undergraduates [updated 2010]). [2] virtually all of our library customers are using cellular devices and most of them have data plans. rice library mobile opac in the summer of 2009 the library decided to create a mobile website. it was determined that the site would include some version of a library catalog search. currently the library uses sirsidynix’s ilink for its opac. for most mobile devices, this is not a satisfactory interface. the screen is too large, and the graphics are too cumbersome. the question became how we could get search results out of the catalog that would fit on a device with a smaller screen and more limited resources. it was determined that if we were able to use web services to search the catalog, we could reformat the interface into something more appropriate. at the time, sirsidynix did not offer a web services product, so the first rice library mobile opac was completed based on a duplicate opac using hard-coded css files. when sirsidynix published symphony web services 2.0 in early 2010, we began looking at using web services to create a better, more flexible mobile opac. using web services for the mobile opac sirsidynix’s implementation of web services sirsidynix published symphony web services 2.0 in early 2010. going through several minor revisions, they finally published version 3.0, a greatly improved version, in late 2010. there are three application components to sirsidynix symphony web services (symws): the web services/servlet container, the netserver server process, and symphony ils server. these components can be installed on separate boxes to distribute the transaction load. symphony web services interacts with symphony ils server via the netserver process at configurable ports. in our implementation, we install symphony web services on a server other than the ils server. the installation and set up of web services is fairly straightforward. there are only two requirements: the java se development kit (jdk) version 1.6.0 update 5 or higher and the apache tomcat server. the tomcat server comes with the installation package so the only thing that needs to be done is to download and install the jdk. after this has been accomplished there is a script that must be run on the server which guides the user through the installation process. once installed, symphony web services exposes three interfaces: patron services, security services, and the standard service. currently there is no documentation published by sirsidynix on their web services, but it can be explored through the web services description language (wsdl) along with the simple object access protocol (soap). soap is an http protocol that is used for exchanging information through web services. wsdl provides an xml grammar for describing all the pertinent information about a web service, including its name, transport binding, and message format (ma k 2005). [3] soap does the exchanging of information and wsdl describes how it should be done. (for more information, see http://en.wikipedia.org/wiki/soap) symphony web services’ three wsdl interfaces can be found at the following three urls: http://host:port/wscontext/soap/security?wsdl http://host:port/wscontext/soap/patron?wsdl http://host:port/wscontext/soap/standard?wsdl in this example “wscontext” is the name of the context path where symphony web services was installed. by exploring the wsdl interfaces a complete list of the available web services and operations can be obtained. with the aid of an integrated development environment (ide), such as oracle netbeans, client code can be auto-generated based on the exposed wsdl interfaces. up through version 2 of sirsidynix’s symphony web services it is free to access standard services (i.e., searching catalog, looking up title information, etc.) and security/patron services (i.e., user authentication, login/logout, listing/renewing checkouts, etc.). however, starting with the current version (3.0), libraries must acquire an additional license to program against and access patron services while the standard services will remain free. choosing a web services toolkit there are many existing toolkits available for developers of web service clients and servers. after a period of exploration we decided to adopt jax-ws 2.x (java api for xml web services) as our developmental toolkit for our web services client to communicate via xml-based soap messages. the reason jax-ws was chosen is because it allows a programmer to avoid dealing directly with soap generation and parsing. developing a jax-ws style web services client without the aid of ide can be compared to driving a car to a restaurant for the first time in new york without a gps device. without a gps device, you have to either remember names of major streets where turns will be made or look at a map from time to time. you will eventually make it to the restaurant, but the process may be slow and potentially fraught with error. an efficient and smart ide takes the role of a gps in software development. for this project, we are using oracle netbeans 6.8 as our ide tool. implementing the mobile opac the project can be divided into two independent tasks: developing the user services interface and developing the search catalog & place hold interface. user services interface through this interface users are able to list their checkouts, holds, overdues and bills. they are also able to renew their checkouts and manage (view and remove) their current holds. search catalog & place hold interface through this interface users are able to search our catalog via the quick search page or the advanced search page. users can also list title holdings information and place holds on holdable items. application design approach the design approach is based on model, view, controller (mvc) (see http://en.wikipedia.org/wiki/model-view-controller for a quick explanation) which is an architecture that logically separates the application into model layer, view layer and controller layer. the model layer implements the actual business logic. the view layer provides user interfaces to interact with the model layer, sending data (i.e., search terms) to the model layer to process and display data such as search hits returned from the model layer. the controller layer sits between the model layer and view layer and determines which model object a request should be directed to and which view object should be called to display a data output (i.e., a user’s checkout list object). in order to implement the mvc architecture, we decided to use an existing web application framework. frameworks are building blocks and services that a developer can use to solve problems in repeatable ways. we developed this application using the apache struts framework because it is mature, well documented, and relatively simple to learn and use. it is also supported by oracle net beans. with struts, we can define action objects to interact directly with the controller servlet component to achieve complex mvc flow control which is the advantage that struts brings to this project. other frameworks can also accomplish this; struts was selected based on the developer’s personal preference and experience. this project is moderately complex, and most popular mainstream web frameworks can do the job. the application flow the two separate interfaces, user services and catalog search & place hold, have their own application flow but as a whole share commonalties in architecture and code. the following section introduces the struts components shared by the entire application, and then outlines the general steps on how to call symphony web services. the struts components in figure 1 and figure 8, the actionservlet class is the key component of the controller layer of the struts framework and is configured as a servlet in the web.xml file as shown in the following code: <servlet> <servlet-name>action</servlet-name> <servlet-class><strong>org.apache.struts.action.actionservlet</strong></servlet-class> <init-param> <param-name>config</param-name> <param-value>/web-inf/<strong>struts-config.xml</strong></param-value> </init-param> <servlet-mapping> ... <servlet-name>action</servlet-name> <url-pattern><strong>*.do</strong></url-pattern> </servlet-mapping> ... in this application any request with the url pattern of *.do (e.g. login.do) will invoke the actionservlet controller. the actionservlet consults the struts configuration file named struts-config.xml (also defined in the web.xml file in the code above) to determine which action object should be called in response to the incoming request. the struts-config.xml file holds the request action mapping definitions. the following code snippet from struts-config.xml defines the action class (loginaction) to call in response to a request submitted via login.do: <action-mappings> ... <action name="loginform" path="/login" scope="request" type="edu.rice.library.fondrenapps.struts.loginaction" validate="false"> <forward name="success" path="/web-inf/userservices.jsp"/> <forward name="retry" path="/login.jsp"/> <forward name="page4placehold" path="/confighold.jsp"/> <forward name="failure" path="/faileduserservice.jsp"/> </action> ... </action-mappings> loginform is a form bean (extends actionform class) representing the login form that is presented and submitted via login.jsp; it transports form property data (e.g. user name) back and forth between the view (login.jsp) and the action class (loginaction). also defined in this sample code are four named forward paths – success, retry, page4placehold and failure which point to four jsp pages respectively – success.jsp, login.jsp, confighold.jsp and faileduserservice.jsp. these forwards will be used by the action class (loginaction) to determine which jsp page will be used based on the results from loginaction. take note that action classes according to the struts framework are not intended to hold actual business logic. instead, action classes are designed to hold control logic (i.e. the loginaction action class invokes the loginuser method exposed from the security web services interface) to interact with actual business logic (in this example it checks whether library user credentials can be honored or denied) which resides in the model layer. therefore the action class in this example does not belong to the model layer. because symphony web services encapsulate all library opac business logic and expose them through industry standard compliant web services interfaces, it is fine to view the web services as the model layer of our application. the actionservlet servlet, web.xml file, struts-config.xml file, form beans and user defined action classes comprise the controller layer (areas circled by ellipse in figure 1 and figure 8) whereas all jsp pages form the view layer. call the symphony web services before any call to the symphony web services can be programmed, netbeans is used to generate portable java artifacts for the web services client by reading the wsdl of the symphony web services. behind the scenes, this step is actually accomplished by a wsdl to java mapping tool (i.e., the jax-ws wsimport utility) picked by netbeans. among the generated java artifacts, the following major artifacts are listed below. service endpoint interface (sei) service jaxb generated class exceptions class (for brief explanation to the above java artifacts, please visit the link http://publib.boulder.ibm.com/infocenter/wasinfo/v6r1/index.jsp?topic=/com.ibm.websphere.wsfep.multiplatform.doc/info/ae/ae/rwbs_wsimport.html) with the java artifacts being generated, we can program calls to the symphony web services. the code snippet in example 1 shows how to make use of the generated artifacts to call the loginuser method on the symphony security service. com.sirsidynix.symws.security.securityservice service = new com.sirsidynix.symws.security.securityservice(); com.sirsidynix.symws.security.securityendpoint port = service.getsecurityport(); com.sirsidynix.xmlns.common.header.sdheader header = new com.sirsidynix.xmlns.common.header.sdheader(); com.sirsidynix.schemas.symws.security.loginuserrequest body = new com.sirsidynix.schemas.symws.security.loginuserrequest(); header.setclientid("xxxx"); body.setlogin(userid); body.setpassword(userpass); com.sirsidynix.schemas.symws.security.loginuserresponse result = port.loginuser(body, header); session.setattribute("session_no", result.getsessiontoken()); session.setattribute("authuserid", result.getuserid()); example 1 the first two code lines, at run time, create a proxy instance named “port” that implements the security endpoint interface (sei). lines 4-5 create the header instance from the generated jaxb wrapper class sdheader. the secret string required by the symphony web services is assigned to the header instance on line 8. lines 4-5 and line 8 are commonly required by all kinds of calls made to symphony web services and remain almost identical across all symphony web service calls. lines 6-7 create the body instance from the generated jaxb wrapper class loginuserrequest which behaves as a wrapper to contain all request parameters specifically required for a web service call. lines 9-10 assemble user id and user password into the body request wrapper object. lines 11-12 pass the standard request header object and request body object as the required reference type parameters and invoke the loginuser method on the port proxy instance of security sei. if successful, the invocation returns a result object that is an instance of the jaxb generated loginuserresponse wrapper class. lines 13-14 retrieve the sessiontoken and userid property values from the result object and ingest them into the session scope of the httpservletrequest object for sharing by other components of this application. user services figure 1. user services flow chart in user services (figure 1), users authenticate into their accounts through the page login.jsp (figure 2). behind the scenes, user credential data is carried by the loginform form bean. then, the loginaction action object is dispatched to call the authenticateuser and userlogin method of the symphony security service. figure 2. user services page the code in example 2 is called in the loginaction action object to test whether the supplied user id and user password are valid. if the boolean type return value – authresult is false, the local forward path – retry will be selected and login.jsp will be redisplayed asking users to retype their credentials. com.sirsidynix.schemas.symws.security.authenticateuserrequest authbody = new com.sirsidynix.schemas.symws.security.authenticateuserrequest(); authbody.setlogin(userid); authbody.setpassword(userpass); boolean authresult = port.authenticateuser(authbody, header); example 2 otherwise, if the value of the returned authresult variable is true, the call to userlogin method will be made (see the code in example 1 for detail). upon successful invocation to the userlogin method, a session scoped bean – session_no is generated. the session_no bean contains the session value, assigned by symphony security service, pointing to the currently valid authenticated user session. then, the successfully authenticated and logged in user is forwarded to the protected user services page which presents four task options: user status inquiry, view/remove holds, renew my materials and logout (figure 3). figure 3. user services page the user status inquiry (figure 4) link calls the listckoaction action object. the code snippet in example 3 shows the steps to call the lookupmyaccountinfo method on the patron service. header.setclientid("xxxx"); header.setsessiontoken(session.getattribute("session_no").tostring()); com.sirsidynix.symws.patron.patronservice service = new com.sirsidynix.symws.patron.patronservice(); com.sirsidynix.symws.patron.patronendpoint port = service.getpatronport(); com.sirsidynix.schemas.symws.patron.lookupmyaccountinforequest body = new com.sirsidynix.schemas.symws.patron.lookupmyaccountinforequest(); body.setincludefeeinfo(feeinfofilter.unpaid_fees); body.setincludepatroncheckoutinfo(checkoutinfofilter.all); body.setincludepatronstatusinfo(boolean.true); body.setincludepatronholdinfo(holdinfofilter.active); com.sirsidynix.schemas.symws.patron.lookupmyaccountinforesponse result = port.lookupmyaccountinfo(body, header); list ckolist = result.getpatroncheckoutinfo(); list feeinfo = result.getfeeinfo(); request.setattribute("listcko", ckolist); request.setattribute("num_holds", result.getpatronholdinfo().size()); request.setattribute("patronstatus", result.getpatronstatusinfo()); request.setattribute("feeinfo", feeinfo); example 3 figure 4. list checkouts page aside from the secret string required by the symphony web services on line 2, the value of the current authenticated user session is also assembled into the request header object. without this session value being assembled, the call to the lookupmyaccountinfo method will not be allowed and will throw an exception. lines 8-11 prepare the request body object of the jaxb generated lookupmyaccountinforequest class by setting values of the include properties of the body object, indicating the response should contain fee (bill) , patron status, and active hold information. lines 12-13 make the actual call to the lookupmyaccountinfo method of the patron service. lines 14-15 retrieve the patroncheckoutinfo and feeinfo information and save them into the ckolist and feeinfo array list respectively. lines 16-19 place the acquired information into the request scoped beans which will be consumed by the list4cko.jsp page. renew my materials (figure 6) needs a little bit more work than user status inquiry. the listcko4renewalaction action object is dispatched to get a list of checkouts for renewing and forwards the list to the renewcko.jsp page for display (figure 5). the code in listcko4renewalaction class is pretty much identical to the code shown in example 3, excluding the hold and patron status information. the reason we repeat the web services call performed by the listckoaction action object is to guarantee the list of checkouts for renewal is updated rather than cached. when the form with selected checkouts to renew is submitted, the item ids captured and carried by the renewalform form bean are sent to the renewalaction action object. figure 5. view/remove current holds page figure 6. select items to remove page the code snippet in example 4 shows the steps to call the renewcheckout method on the patron service. map failedrenewals = new linkedhashmap(); int failcnt=1; com.sirsidynix.symws.patron.patronservice service = new com.sirsidynix.symws.patron.patronservice(); com.sirsidynix.symws.patron.patronendpoint port = service.getpatronport(); com.sirsidynix.schemas.symws.patron.renewcheckoutrequest body = new com.sirsidynix.schemas.symws.patron.renewcheckoutrequest(); com.sirsidynix.xmlns.common.header.sdheader header = new com.sirsidynix.xmlns.common.header.sdheader(); header.setclientid("xxxx"); header.setsessiontoken(session.getattribute("session_no").tostring()); body.setuserid(session.getattribute("authuserid").tostring()); for (int i=0; i &amp;amp;lt; renewalform.getselectedckos().length;i++) { body.setitemid(renewalform.getselectedckos()[i]); try { com.sirsidynix.schemas.symws.patron.renewcheckoutresponse renewedresult = port.renewcheckout(body, header); renewedlist.add(renewedresult); } catch (exception e1) { for (checkoutinfo cko : ckolist) { if (cko.getitemid().equals(renewalformbean.getselectedckos()[i])) { failedrenewals.put(failcnt+". "+cko.gettitle(), e1.getmessage()); failcnt++; } } } } ... request.setattribute("renewedlist", renewedlist); request.setattribute("failedrenewals", failedrenewals); example 4 line 12 assembles the current authenticated user id into the body request wrapper instance of the jaxb generated renewcheckoutrequest class. line 13 starts a loop block to traverse the selected checkouts (item ids) captured in the selectedckos list properties of the renewalform bean. for each loop, line 15 assembles an item id value to the request body object and then lines 17-18 make the actual call to the renewcheckout method of patron service. should no exception be found, the return object of the jaxb generated renewcheckoutresponse will be pushed into the renewedlist array list; otherwise an exception object will be captured by the catch block and the failing reason and title will be pushed into the failedrenewals linkedhashmap object which is declared at line 1. lines 34 and 35 place the renewedlist and failedrenewals object into two request scoped beans which will be consumed by the resultofrenewals.jsp to display the renewal results (figure 7). figure 7. items renewed page view/remove holds link first calls the listholdsaction action object to get a list of the currently authenticated user’s active holds (figure 5). the code in example 5 shows the step to call the lookupmyaccountinfo method of the patron service. it’s much like an abridged version of the code listed in example 1. com.sirsidynix.symws.patron.patronservice service = new com.sirsidynix.symws.patron.patronservice(); com.sirsidynix.symws.patron.patronendpoint port = service.getpatronport(); com.sirsidynix.schemas.symws.patron.lookupmyaccountinforequest body = new com.sirsidynix.schemas.symws.patron.lookupmyaccountinforequest(); com.sirsidynix.xmlns.common.header.sdheader header = new com.sirsidynix.xmlns.common.header.sdheader(); header.setclientid("xxxx"); header.setsessiontoken(session.getattribute("session_no").tostring()); body.setincludepatroncheckoutinfo(checkoutinfofilter.all); body.setincludepatronholdinfo(holdinfofilter.active); com.sirsidynix.schemas.symws.patron.lookupmyaccountinforesponse result = port.lookupmyaccountinfo(body, header); list holdlist=result.getpatronholdinfo(); request.setattribute("holdlist", holdlist); example 5 line 15 places the retrieved holdlist list array, which contains wrapper objects of the jaxb generated patronholdinfo class, into a request scoped bean – holdlist, which will be consumed by the list4holds.jsp page to display the user’s current holds (figure 5). when the form with selected holds (hold keys) to remove is submitted, hold keys captured and carried by the list4holdsform form bean are sent to the removeholdsaction action object. the code snippet in example 6 shows the steps to call the cancelmyhold method on the patron service. list4holdsform listholdsbean=(list4holdsform)form; com.sirsidynix.symws.patron.patronservice service = new com.sirsidynix.symws.patron.patronservice(); com.sirsidynix.symws.patron.patronendpoint port = service.getpatronport(); com.sirsidynix.schemas.symws.patron.cancelmyholdrequest body = new com.sirsidynix.schemas.symws.patron.cancelmyholdrequest(); com.sirsidynix.xmlns.common.header.sdheader header = new com.sirsidynix.xmlns.common.header.sdheader(); header.setclientid("xxxx"); header.setsessiontoken(session.getattribute("session_no").tostring()); for (int i=0; i &amp;amp;lt; listholdsbean.getselectedholdkeys().length;i++) { body.setholdkey(listholdsbean.getselectedholdkeys()[i]); boolean result = port.cancelmyhold(body, header); } example 6 the loop block starting from line 11 traverses hold keys captured by the list4holdsbean instance of the list4holdsform form bean class. line 13 assembles the selected hold key into the request body object of the jaxb generated cancelmyholdrequest class. line 14 makes the actual call to the cancelmyhold method of the patron service with each selected hold key. when the loop ends, the user services page (figure 3) will be brought back. the logout option ends the user’s current session by dispatching the logoutaction action object to call the logoutuser method of the security service. failed user service operations due to runtime exceptions are handled by the faileduserservices.jsp page. search catalog & place hold interface as shown in figure 8, two search pages – the quick search page and the advanced search page are accessible for catalog search. figure 8. search catalog & place hold flowchart the quick search page — quicksearch.jsp (figure 9) presents a single search term box along with six search index options (word anywhere, author, title, subject, series and periodical title) while advanced search (figure 10) — advancedsearch.jsp allows users to build search terms from the three search term boxes in boolean mode. up to three different search indexes can be selected and searched concurrently in boolean mode with the advanced search page. the advanced search page also presents three search qualifier (filters) selectors – item type, location and publication year to help users refine their searches. figure 9. quick search page figure 10. advanced search when the quick search or the advanced search page is submitted, search request data (search terms, search index type(s), boolean operators and search (filters) qualifiers) captured and encapsulated by the searchform form bean is sent to the searchaction action object. the code in example 7 shows the steps for how to assemble search request data from the advanced search page into a search request object and then call the searchcatalog method on the standard service. searchform searchform=(searchform)form; ... com.sirsidynix.symws.standard.standardservice service = new com.sirsidynix.symws.standard.standardservice(); com.sirsidynix.symws.standard.standardendpoint port = service.getstandardport(); com.sirsidynix.schemas.symws.standard.searchcatalogrequest searchrequestbody = new com.sirsidynix.schemas.symws.standard.searchcatalogrequest(); ... com.sirsidynix.schemas.symws.standard.searchcatalogrequest.query qterm1 = new com.sirsidynix.schemas.symws.standard.searchcatalogrequest.query(); ... qterm1.setterm(searchform.getsrchdata1()); ... if (!searchform.getsrchfield1().equals("general")) { qterm1.setsearchtype(searchform.getsrchfield1());} qterm1.setoperator(searchform.getsrchop1()); searchrequestbody.getquery().add(qterm1); ... com.sirsidynix.schemas.symws.standard.searchcatalogrequest.query qterm2 = new com.sirsidynix.schemas.symws.standard.searchcatalogrequest.query(); ... searchrequestbody.getquery().add(qterm2); ... com.sirsidynix.schemas.symws.standard.searchcatalogrequest.query qterm3 = new com.sirsidynix.schemas.symws.standard.searchcatalogrequest.query(); ... searchrequestbody.getquery().add(qterm3); ... searchcatalogfilters searchfilter = new searchcatalogfilters(); ... searchfilter.setitemtypefilter(searchform.getsrchitype()); ... searchfilter.sethomelocationfilter(searchform.getsrchloc()); ... searchfilter.setpubyearfilter(searchform.getpubyear()); ... searchrequestbody.setfilters(searchfilter); ... com.sirsidynix.schemas.symws.standard.searchcatalogresult searchresult = port.searchcatalog(searchrequestbody, header); ... request.setattribute("queryid", searchresult.getqueryid()); request.setattribute("totalhits",searchresult.gettotalhits()); request.setattribute("hitsperpage",hitsperpage); if (searchresult.gettotalhits()&amp;amp;lt;= hitsperpage ) { request.setattribute("hitlist", searchresult.gethitlisttitleinfo()); ... return mapping.findforward(success); } ... request.setattribute("currentpageno","1"); return mapping.findforward(pagingrequest); example 7 line 12 assigns the term from the first search box field of the advanced search page to the term property of the qterm1 wrapper object of the jaxb generated query class. lines 14 and 15 determine the search index value which is captured by srchfield1 of the searchform bean object. the search index value selected for the first search box on the advanced search page is assigned to the searchtype property of the qterm1 object. searchtype is called search index in our own application code. because general is the default index, it doesn’t need to be assigned to the searchtype property of the qterm1 object. this is done through the usage of the “if” conditional on line 14. other values that can be assigned on the advanced search page that work with the searchtype property of query wrapper object are, title, author, subject, series, pertitle (periodical title). line 16 assigns the boolean operator value that has been selected for the first search box to the operator property of the qterm1 object. line 17 pushes the qterm1 object which has been populated with the search data from the first search box to the query property (query typed array list) of the searchrequestbody body request object of the jaxb generated searchcatalogrequest class. following the same logic, lines 19-27 collect and assemble search data from the second and third search boxes into the query property of the searchrequestbody request body object. lines 29-35 assign the selected search qualifiers to the corresponding properties of the searchfilter object of the jaxb generated searchcatalogfilter. line 37 assigns this searchfilter object to the filters property of the searchrequestbody request body object. lines 39-40 make the final call to the searchcatalog method on the standard service. lines 42-43 create two request scoped beans queryid and totalhits from the corresponding properties of the searchresult return object of the jaxb generated searchcatalogresult class. the value of the queryid bean points to the current search result set. line 44 assigns the value (10 in our application) of the hitsperpage variable into the request scoped hitsperpage bean. if the number of total search hits is not greater than the value (10) of the hitsperpage variable (line 39). line 49 forwards directly to the hitlistdisplay.jsp page (figure 11) to display the title list which is contained in the request scoped hitlist bean created from the value of the hitlisttitleinfo property of the searchresult object on line 47. if the number of total search hits is greater than 10, line 53 will forward the processing to the searchpaging link where the searchpagingaction action object can be called. figure 11. hit list page with hits less than or equal to 10 the code snippet in example 8 shows how to call the searchcatalogpaging method on the standard service. hitsperpage=integer.parseint(request.getattribute("hitsperpage").tostring()); queryid=request.getattribute("queryid").tostring(); hitsperpage=integer.parseint(request.getattribute("hitsperpage").tostring()); numofpages=long.parselong(request.getattribute("numofpages").tostring(),10); currentpageno=long.parselong(request.getattribute("currentpageno").tostring(),10); ... com.sirsidynix.xmlns.common.header.sdheader header = new com.sirsidynix.xmlns.common.header.sdheader(); com.sirsidynix.symws.standard.standardservice service = new com.sirsidynix.symws.standard.standardservice(); com.sirsidynix.symws.standard.standardendpoint port = service.getstandardport(); com.sirsidynix.schemas.symws.standard.searchcatalogpagingrequest pagingrequestbody = new com.sirsidynix.schemas.symws.standard.searchcatalogpagingrequest(); header.setclientid("xxxx"); pagingrequestbody.setqueryid(queryid); pagingrequestbody.setfirsthittodisplay((currentpageno-1)*hitsperpage+1); if (currentpageno*hitsperpage &amp;amp;lt;= totalhits) pagingrequestbody.setlasthittodisplay(currentpageno*hitsperpage); else pagingrequestbody.setlasthittodisplay(totalhits); pagingrequestbody.setincludeavailabilityinfo(true); com.sirsidynix.schemas.symws.standard.searchcatalogresult pageresult = port.searchcatalogpaging(pagingrequestbody, header); ... request.setattribute("hitlist", pageresult.gethitlisttitleinfo()); ... return mapping.findforward(success); example 8 line 15 sets the queryid acquired from the searchaction object to the queryid property of the pagingrequestbody request object of the jaxb generated searchcatalogpagingrequest class. lines 16-20 calculate the first and last hits to display for the current page and assemble them into the pagingrequestbody request body object. line 21 includes title availability information (i.e. is a title holdable?) within the return object. lines 22-23 finally make the call to the searchcatalogpaging method on the standard service. line 25 creates a request scoped bean (hitlist) from the value of the hitlisttitleinfo property of the pageresult return object of the jaxb generated searchcatalogresult class. this request scoped hitlist bean contains a subset which corresponds to the currently selected page of the original search result set. line 27 forwards back to the hitlistdisplay.jsp to display the current hitlist along with the updated page navigation buttons (figure 12). figure 12. hit list page with hits greater than or equal to 10 when a title link on a hitlist page is clicked to show a title’s holding information, the checktitleinfoaction object behind the /checktitleinfo link is called along with several request parameters. the code snippet in example 9 shows how to call the lookuptitleinfo method of the standard service. com.sirsidynix.xmlns.common.header.sdheader header = new com.sirsidynix.xmlns.common.header.sdheader(); com.sirsidynix.symws.standard.standardservice service = new com.sirsidynix.symws.standard.standardservice(); com.sirsidynix.symws.standard.standardendpoint port = service.getstandardport(); request.setattribute("titleinfo", result.gettitleinfo()); com.sirsidynix.schemas.symws.standard.lookuptitleinforequest body = new com.sirsidynix.schemas.symws.standard.lookuptitleinforequest(); header.setclientid("xxxx"); body.gettitleid().add(titleid); body.setincludeavailabilityinfo(true); body.setincludeiteminfo(true); body.setincludecataloginginfo(true); body.setmarcentryfilter("full"); com.sirsidynix.schemas.symws.standard.lookuptitleinforesponse result = port.lookuptitleinfo(body, header); request.setattribute("titleinfo", result.gettitleinfo()); example 9 line 10 adds the selected titleid to the titleid property of the body request object of the jaxb generated lookuptitleinforequest. lines 11-14 include what kinds of title information (i.e., item information, marc information) should be included in the return object. lines 15-16 make the actual call to the lookuptitleinfo method of the standard service. line 17 generates the titleinfo bean from the value of the titleinfo property of the result object of the jaxb generated lookuptitleinforesponse before being forwarded to the displaytitleinfo.jsp page (figure 13). figure 13. title holding information page the failedsearch.jsp page is used to handle failed searches and other exceptions. if the currently displayed title has a holdable copy, a place hold link will show (figure 13) for this copy. once the place hold link is clicked, the configholdaction action object is called. this action object is also used to determine whether there is an authenticated user session. if no valid authenticated user session can be found, the login.jsp will be called to ask the user for their credentials. if authentication succeeds, or if there is a valid authenticated user session, the confighold.jsp will be called to let the user choose pickup library and hold expiration date (figure 14). figure 14. place hold page when the confighold.jsp page is submitted, data for placing a hold (i.e. title, itemid, expiration date) is sent to the placeholdaction action object. the code snippet in example 10 shows how to call the createmyhold method on the standard service. com.sirsidynix.symws.patron.patronservice service = new com.sirsidynix.symws.patron.patronservice(); com.sirsidynix.symws.patron.patronendpoint port = service.getpatronport(); com.sirsidynix.schemas.symws.patron.createmyholdrequest body = new com.sirsidynix.schemas.symws.patron.createmyholdrequest(); com.sirsidynix.xmlns.common.header.sdheader header = new com.sirsidynix.xmlns.common.header.sdheader(); header.setsessiontoken(session.getattribute("session_no").tostring()); header.setclientid("xxxx"); ... simpledateformat formatter=new simpledateformat("mm/dd/yyyy"); date utildate=formatter.parse(expdate4hold); gregoriancalendar gcdate = new gregoriancalendar(); gcdate.settime(utildate); javax.xml.datatype.xmlgregoriancalendar xmlgcdate = new com.sun.org.apache.xerces.internal.jaxp.datatype.xmlgregoriancalendarimpl(gcdate); nillabledate expiredate = new nillabledate(); expiredate.setvalue(xmlgcdate); body.setitemid(itemid4hold); body.setpickuplibraryid(picklibrary4hold); body.setexpiresdate(expiredate); body.setholdtype(holdtype.system); body.setholdrange(holdrange.system); long result = port.createmyhold(body, header); example 10 line 8 assembles the authenticated user session value into the header object. lines 11-18 convert the text string of the hold expiration date (expdate4hold) into a nillabledate object. lines 19-23 set property values of the request body object of the jaxb generated createmyholdrequest class. line 24 makes the call to the createmyhold method on the standard service. the successfully returned result is a long type value. the resultofplacehold.jsp then is called to show the hold has been successfully placed (figure 15). otherwise the failedplacehold.jsp will be called to show the failed hold and the reason why it did not succeed. figure 15. result page for place hold problems the mobile opac only consumes a small set of available methods exposed from sirsidynix symphony web services’ three interfaces: security, patron, and standard services. while including all functionality that was available in rice university’s first mobile opac created using css stylesheets, it also overcomes problems with that type of mobile interface: the css mobile opac was mainly targeted for iphone/ipod touch devices. fat footprint — only hiding screen elements not eliminating them. potential breakage caused by future opac upgrades or redesigns. initially when this application was developed we ran into a number of problems with symphony web services 2.0. that version did not support system level holds and attempting to place a hold or to list holds caused an exception to be thrown. blocked and barred patrons could not log into their accounts. users who had an item recalled could not check their accounts without the system throwing an exception. as of version 3.0 all of these issues have been fixed. conclusion web services have been in existence for more than a decade. they have been widely adopted and heavily utilized by numerous companies to streamline and strengthen their business workflow internally and externally. in recent years, web services have started to be adopted by library vendors (e.g. the xisbn interface by worldcat). that ils vendors have begun to expose opac interface functions through web services is another leap into the library world for this technology. this mobile opac, consuming methods and data exposed via another library system’s web services interface, is an example of a project that can be built with little trouble even without technical documentation available from the vendor. by integrating web services into their products library vendors are using an industry standard. they present an opportunity for libraries to streamline work across many types of library systems – discovery tools, erms, e-reserves and courseware management, amongst others. web services are a step in the right direction. references [1] international telecommunication union (2009), information society statistical profiles 2009: americas. [internet]. [cited 2010 sept 8] available from: www.itu.int/dms_pub/itu-d/opb/ind/d-ind-rpm.am-2009-e09-pdf-e.pdf [2] new undergraduates [internet]. [updated 2010] rice university [cited 2010 nov 30] available from: http://it.rice.edu/newstudent.aspx [3] ma, k. (2005). web services: what’s real and what’s not? it professional, 7(2), 14-21. doi:10.1109/mitp.2005.47. [internet]. [cited 2010 nov 30] available from: http://ieeexplore.ieee.org.ezproxy.rice.edu/stamp/stamp.jsp?tp=&arnumber=1425418 additional reading hansen, md. soa using java web services. upper saddle river, nj: prentice hall, 2007. print.[cited 2010 nov 30] (coins) schlimmer jc (2002), web services description requirements. [internet]. [cited 2010 sept 16] available from http://www.w3.org/tr/ws-desc-reqs sanders dt, hamilton ja jr., macdonald ra. 2008. supporting a service-oriented architecture. in: proceedings of the 2008 spring simulation multiconference (springsim ’08). society for computer simulation international, san diego, ca, usa, 325-334. [internet]. [cited 2010 nov 28] available from: http://delivery.acm.org/10.1145/1410000/1400595/p325-sanders.pdf?key1=1400595&key2=5176311921&coll=dl&dl=acm&cfid=116583772&cftoken=82552286 sprott, d., wilkes, l., (2004) understanding service-oriented architecture cdbi forum .net architecgure center. [internet]. [cited 2010 sept 10] available from: http://msdn.microsoft.com/en-us/library/aa480021.aspx yu, q., liu, x., bouguettaya, a., & medjahed, b. (2008). deploying and managing web services: issues, solutions, and directions. the vldb journal, 17(3), 537-572. [internet]. [cited 2007 sept 10] available from: http://www.springerlink.com/content/c2k20771767l2502/fulltext.pdf about the authors denis galvin works for fondren library at rice university where he is the ils supervisor. prior to rice he worked for the library at carnegie mellon university doing similar work. he enjoys dabbling in anything open source and is always interested in topics on digital preservation. mang sun works for fondren library at rice university where he is the systems librarian. his main job duties include administrating several library systems including the ils system and developing in house applications and solutions when needed. prior to rice he worked for saskatchewan provincial library in canada as a systems librarian (information resources specialist). his research interests cover a wide spectrum of library and it technologies. he got his mlis degree from dalhousie university in 2006. subscribe to comments: for this article | for all articles 4 responses to "using web services for a mobile opac" please leave a response below: eduardo, 2011-10-04 perfect! very good user interface. i hope you don’t mind if i copy some of them! it’s very interesting that there is no comments here. your article was really helpful. thanks! szerelmi bánat, 2011-11-29 love it! as eduardo i really consider to copy a few parts myself. very very very cool! thanks a lot! and where are the other people? i expected more raving comments here… shame on them! :) you guys just keep up this good job! szerelmi bánat from hungary e-portfolios: mobile | library, baby!, 2013-01-10 […] libraries need to and are working on creating mobile solutions for their users. libraries can use mobile opacs or they can use qr codes for digital tie ins to physical […] an app for that: making your library (more) mobile-friendly | berry picked, 2014-03-12 […] – if you’re developing your own opac: http://journal.code4lib.org/articles/4810 […] leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – topic space: rapid prototyping a mobile augmented reality recommendation app mission editorial committee process and structure code4lib issue 30, 2015-10-15 topic space: rapid prototyping a mobile augmented reality recommendation app with funding from an institute of museum and library services (imls) sparks! ignition grant, researchers from the university of illinois library designed and tested a mobile recommender app with augmented reality features. by embedding open source optical character recognition software into a “topic space” module, the augmented reality app can recognize call numbers on a book in the library and suggest relevant items that are not shelved nearby. topic space can also show users items that are normally shelved in the starting location but that are currently checked out. using formative ux methods, grant staff shaped app interface and functionality through early user testing. this paper reports results of ux testing; a redesigned mobile interface, and provides considerations on the future development of personalized recommendation functionality. by jim hahn, ben ryckman, and maria lux introduction libraries have sought to leverage advances in mobile-networked technologies since users began to access the library homepage and licensed databases from the mobile browser. several libraries extended their mobile development into the application development area[1]. while app development can be resource intensive, the benefits for native mobile digital services can make great advances in services to library patrons. new library services are possible through native mobile apps and those native affordances; the native hooks available include accessing hardware of the device like the camera for optical character recognition, as well as inferring location based on gps, wi-fi, or other beacons. this makes possible an exciting ux research space since both aspects of digital and physical touch points are the basis for inquiry. one overarching concern within the library ux field is how to integrate digital touch points with the physical experience and interactions among both. research questions mobile augmented reality is a challenging ux area. augmented reality (ar) requires overlaying digital content onto markers or tags in physical space. the challenges in creating augmented reality services for mobile include the small form factor for the phone and the interactivity of utilizing the camera or video feed for augmenting the space. by using an iterative rapid prototyping methodology our mobile ux augmented reality research questions were investigated. specifically, researchers sought to explore how users would interact with augmented reality tools designed to assist in the navigation of the physical space while undertaking authentic course-based assignments. thus our research question grew out of the authentic need of students to find related material in the physical browsing area, especially since students are increasingly being asked to complete interdisciplinary assignments. one of the defining features and departure points for this project was the underlying concern that students may overlook related items in the print book stacks if they only focus on one area — and that several areas may be relevant to their topics. our focus was on undergraduate collections and in designing apps that support first-year students. our research questions included: inquiring if, based on the recommendations, students could tell us other subjects that may be of interest to their topic. from the results of the mobile app, could research participants tell us what books were not available — e.g. could they tell from the app what books were currently checked out. we inquired if seeing these recommendations led to a broader understanding of what their research topic was about. we wanted to know if such an app was useful; thus we inquired if the books that were recommended would be utilized in their research. we asked students to tell us if the app was a worthwhile tool for the library to develop. the recommendation middleware leveraged both library of congress subject headings as well as historical circulation data as a basis for recommendations. grant staff leveraged all historical checkout trends available in the library, beginning from 2003 through 2014. historical checkout data helped to show users what others items have been popularly circulating. since the broader ux field is coming to understand how best to make personalized recommendation services in library and educational settings, a benefit of this research is to demonstrate student preferences. we demonstrate results of one approach to location-based personalized recommendations, and make suggestions to future system designers in how best to leverage legacy circulation data within the on-site browsing experience. contemporary library recommendation systems have not been sufficiently or closely integrated into the on-site browsing experience in library books stacks. grant goals with successful funding from an imls sparks! ignition grant, researchers from the university of illinois library were able to prototype an augmented reality app utilizing optical character recognition software embedded into a native android app as the “topic space” module. the funding source paid for hourly programmers sourced from the school of computer science at the university of illinois who worked on the project for three semesters. additional partnerships for the grant included collaboration with the graduate school of library and information science where researchers in hci and rapid prototyping helped shape the formative evaluation component of this research and features for recommendations in the library book stacks. goals of the imls grant included the following: create shelf recognition software develop directional wayfinding support that shows users how they are physically navigating an “idea space” complete rapid iterative studies of mobile software with library patrons identify technical infrastructure needed in order to make ar an ongoing part of technology in libraries make available ar apps widely at the end of the grant the augmented reality app functionality was accomplished, allowing a user to take a picture of a call number on a book in the undergraduate book stacks, which then provides recommendations based on the subject area of the call number. figure 1. the topic space software can recognize call numbers of an item when a student successfully takes a picture of the book in the undergraduate library. optical character recognition (ocr) is processed on that call number and from this input the app is able to infer the shelf location of the user. figure 2. using the call number, the server-side web service runs queries, which are based off of the lc classification system. the app then downloads and displays the related media to the end-user. figure 3. students can also see books that are normally shelved here but that are currently checked out, (displayed in the app as: “related booksunavailable @ ugl”) achieving one of the aims of augmented reality, which is to integrate digital information with objects in the physical world. the value of this service is increased access to collections that may normally go overlooked. figure 4. these suggestions are also plotted on a map so that the user can be guided to the items in the building. this paper progresses with a literature review putting this grant work into context with other ux mobile augmented reality projects, followed by a review of methodology relevant to the formative prototype evaluation. following this are results of the study including interface revision and functionality redevelopment. the paper concludes with lessons learned, considerations of extensibility of this work in other library recommendation services including laying the foundations of personalized recommendations throughout the library online catalog and web presence. literature review azuma (1997) defined augmented reality (ar) in the classic work on the topic in presence: teleoperators and virtual environments as allowing “the user to see the real world, with virtual objects superimposed upon or composited with the real world. therefore, ar supplements reality, rather than completely replacing it. ideally, it would appear to the user that the virtual and real objects coexisted in the same space.” at the time that azuma wrote his paper (1997) app phones were not yet commonplace, so that augmented reality generally was considered in the context of peripherals such as head mounted displays. only recently, in the past decade (the iphone was launched in 2007) have mobile augmented reality services become a possible arena for development. in mobile augmented reality, for this paper’s purposes, we attempt to supplement the on-site browsing experience in the library book stacks, looking to augmented collocation features of library collections. collocation objectives in library science have been well researched and established as supporting serendipity while obtaining a known item (svenonius, 2000, pp.21-22) and so this applied research project references and extends the collocation objectives in library collections and descriptions. a further extension considering the range of probable use cases for mobile augmented reality applications for library services has been theorized (hahn, 2012) and experimented within course assignment sheets and syllabi (hahn, 2014). the shelvar app is another significant addition to uses in book stack based ar technologies within library settings. ux in mobile augmented reality in bai & blackwell’s (2012) analytic review of usability evaluation in ismar, the authors examine a corpus of papers from the international symposium of mixed and augmented reality proceedings. the authors note that —“many users are initially impressed by the unique user experience of ar. however they may also be surprised by its technological limitations such as rendering latency, intermittent tracking failure, registration error and perception issues they would not normally experience in a conventional interaction environment,” (bai & blackwell 2012, p457). the notion of rendering latency is reported later in our results section as we found similar themes when users attempted to identify the call number on books with ocr. methodology grant staff utilized rapid prototyping methods (jones & richey, 2000) in designing augmented reality features. staff sought to understand user preferences of augmented reality mobile interfaces that best support location-based recommendations early in the design process. using early functional prototypes, researchers observed and interviewed students before a final design was completed (revised mobile interface shown in results section). by forming interface app functionality early in the design phase we are able to iterate working prototypes into a viable and useful recommendation app. for this study we loaded the topic spaces module onto a library owned android development phone. users did not have to download the app to their device to participate in the study. tests for the topic space module took place in the undergraduate library at the university of illinois. the collection includes a separate run for media (30,000 items), journals (400 subscriptions), and monographs (150,000). the monograph collection was the focus of this recommendation study. with institutional review board approval, researchers were able to approach anyone in the library in order to recruit them into the study. researchers chose to approach those users who were actively looking for books in the library. while this recruitment methodology can be more time intensive, finding users who were actively searching for books can result in more precise findings for the anticipated use case for the app. in library assessment methodology, this recruitment technique is known as intercept testing (jones & sinclair, 2011, p24). the consent form also stipulated that participants must be currently enrolled students to participate in the study. a total of five student participants provided feedback on the initial prototype interface. we chose this number based on the classic usability engineering work by nielsen, where most usability pain points can be uncovered with a small test set of users (1993). students received a $10 gift card for their participation. each student interview and observation lasted no longer than thirty minutes. the task included users taking a picture of a library call number with the android development phone in a starting location and then following the recommendations. the phone would then provide two areas of recommendations for books along with specific recommendations for an item to look at which was not shelved nearby. we asked users if they felt the recommendations were relevant to their starting location and if they felt they would use these recommendations for research papers. for ux prompt, observation form, and debriefing interview questions, see appendix. results we collected formative evaluations of what students liked about the app, what they found confusing, and if they felt the app would be worthwhile to develop. these data are presented as aggregate trends. unexpected results include the observation that students moved their starting locations to be nearest to books that they had previously read, since they wanted to use the app to find other related books that were similar to those they had enjoyed in the past. in our debriefing interviews we learned that some students would make use of the recommendation features and other topic areas that the app suggested. students increasingly expect their digital experiences to be similar to those of google. for example, students expected that the app would display the location of the phone as it moves through the bookstacks; similar in nature to google maps functionality. this type of indoor positioning is feasible to prototype using beacon technology[2]. the biggest hurdle for students in utilizing this app was their expectations for the camera phone; students expected the ocr process to be instant. in practice, we found that the app takes a few seconds to focus and then send the string of captured text to the web server; the web server then takes an additional couple seconds to gather candidate recommendations. as a result, we moved toward more instant scanning with a barcode scan input, which is much faster and provides more reliable metadata to use to suggest items. revised interface based on our observations and interviews we considered the following in our next iterations: easier and more efficient input streamlined result sets supplemental book information ability to easily request materials easier to read mapping functionality figure 5. a revision to the first level of the app. in the revision, items get input into the recommendation system by way of barcode input. scanning the item this way ensures quicker recommendation that students expect. figure 6. revised first level view. after scanning a barcode, we reduced some of the cognitive overload by showing recommended books and ranges in a more common list view design pattern. the list view is expandable to provide additional details. as an added benefit students are able to see the number of recommended items in a topic area so that they have a sense of the total number of recommendations before expanding the view. figure 7. revised detail view provides expansion support for exploring additional recommendation details. figure 8. revised undergraduate library map. this design provides streamlined recommendations so that only one relevant collection location in the undergraduate library is suggested per map view. this also helps reduce the cognitive load of wayfinding, since only one point (red dot) is plotted. the “start shelf” corresponds to a shelf range number in the physical library. discussion outcomes of the grant include the availability of the topic space module within minrva app on the android play store and an experimental backbone.js based topic space web app. current implementation of the topic space module requires that users select the undergraduate library location. while other locations are planned, the module is being developed in this pilot location first before launching library-wide. in order to complete grant tasks there were several unexpected useful technical developments. the technical developments include the designing of a topic space api, a web app component, and the move to barcode scanning. after the technical findings that follow, we note several next steps for recommendations across library platforms noting that the additional usefulness of the recommendation components for this tool are cross-channel, and extend outside of the mobile environment. topic space api in order to generate the recommendations data required by the native android application we created new server side middleware that would combine several data sources into the app. the app required data from the vufind searches to gather candidate recommendations, data from the voyager oracle reports server in order to rank candidate recommendations based on popularity (sum circulations), and a small sqlite table of the stack arrangement in the undergraduate library. together these tools support suggesting items that were not near the student, but within the library. the service also helped to supplement ocr intelligence by way of a customized data table of the library of congress classifications, including classes and subclasses, to help map out distinct subject areas within our own library. the functions, with comments referencing the recommendation process are coded as follows: server side recommendations processing //we form the search terms that will be used, using our custom db table, which is based off of library of congress subject headings. string startsearchterms = getsearchterms(context, search, scannedormanual, classifications); //using vufind, we take the resulting search terms from the previous method, search all of our available libraries, and return a list of starting bibids that relate to our search. arraylist bibids = bibid (startsearchterms); //we now use the starting bibids to create a new list of focused search terms. we base these new search terms off of the topics of each book, found on vufind. arraylist topicsearch = topics (bibids); //this method cleans up the “topicsearch” list, and builds a final list of links to extract information from. arraylist finalwebpages = finalwebpagelist(topicsearch); //based off of an algorithm we developed, “filterlist” filters out links that don't appear to be closely related to the originally scanned book. arraylist filteredlist = filterlist(finalwebpages); //we now create and rank a list of bibids by popularity. this ranking is based off of the circulations of the material from the voyager oracle reports. arraylist bookinfoarray = bookarray(filteredlist); //based off of our aggregated results from the previous methods, we are able to return information about the top two suggested topic spaces. arraylist bookslist = booklist(bookinfoarray); //to help users determine, ahead of time, the types of books they should expect in the resulting suggested topic space, we also include a sample book and its information. list books = finalresults(bookslist, context); details of connecting to the custom sqlite lc table. //load the jdbc driver string drivername = "org.sqlite.jdbc"; class.forname(drivername); //connection string string url = "jdbc:sqlite:" + filepath; //create connection &amp;amp; statement conn = drivermanager.getconnection(url); statement stmt = conn.createstatement(); //build query string qry = "select * from ” + qryend; //run query rs = stmt.executequery(qry); /*the “rs” set includes classification information that has been gathered and organized beforehand, from the library of congress classification outline (http://www.loc.gov/catdir/cpso/lcco). an example of the saved information, which help decide the search terms, would include: subclass s s1-(972) agriculture (general) s21-400.5 documents and other collections s403 agricultural missions, voyages, etc. s419-482 history s530-559 agricultural education s539.5-542.3 research. experimentation s544-545.53 agricultural extension work s548-548.6 historic farms s550-559 exhibitions. fairs s560-571.5 farm economics. farm management. agricultural mathematics including production standards, record keeping, farm-work rates, marketing s583-587.73 agricultural chemistry. agricultural chemicals*/ by developing a server-side api to power the suggestions, we were able to enlist the support of the library’s infrastructure management and support department who are currently patching security updates in the redhat server running the apis. as is common practice in the software developer group in library it, we have also documented the web apis in the library’s micro-services wiki, ensuring that others could build off of this work. the public link to the topic space api is available on the minrva project website, under their “minrva services” section. the skills required to continue ar development include middleware development so that multiple data sources (ils checkout data, subject headings, map data) can be combined into a data object consumed by the mobile or web app. figure 9. modular minrva apis are exposed restfully and allow extensible applications to be developed for multiple platforms. the extensibility of a restful api made it possible to re-use the middleware for a web-based recommendation service based on user location. the web-based version is enabled in the undergraduate library location. after searching for an item and viewing item details, a user can continue to find related items by tapping the “topic space” recommendation button in the detail view. figure 10. the business logic in the tomcat web app has proven versatile enough to make ios, android, and web apps from one middleware solution. topic space web app the web app component will be incorporated within a kiosk in the undergraduate library. by accessing the app from the tablet kiosk in the undergrad library (e.g. selecting the undergraduate library location) users will have access to the same topic space suggestions as they would if they had taken a picture of the call number in the library bookstacks, thus continuing to support users access to collections they may not have been aware. figure 11. an image of topic spaces recommendation for the web. the display indicates recommendations for the title, “the pizza bible,” based on circulations and lc subjects of items in the undergrad library. see the item detail page for the first level recommendation. a key unexpected finding of this grant project was the ongoing personalized recommendations research to pursue, which we discovered by developing an api to power the native app. we have done some preliminary experimentation with ways to re-use the app data feeds in our library login. our sustainability plan is to re-use, re-purpose, and extend the infrastructure from this project for improving access and discovery of collections library-wide. for searching in targeted subject areas, contemporary catalog systems and discovery environments do not make it easy to drill into targeted subjects before you search — typically this must be done after the search, after the result set has already become overwhelming to filter. utilizing a custom built catalog api we provide an example of ways to narrow, and drill into the preferred subject area in the first set of results. figure 12. an example of topical filtering before a user begins her search. when researchers began this grant one of the key motivating factors for developing augmented reality was the use of ocr. while we were funded to fully exploit the possibility of optical character recognition software, its use brought up more user challenges than it solved. in practice, as we matched up the user requirements of capturing the target item quickly and seamlessly, we found that barcode scanning could achieve this aim. since we could not achieve user requirements after initial testing with character recognition alone we decided to chose seamlessness of scanning over the more technically challenging ocr approach. further development of recommendations in the scanner module of the minrva app may allow a more precise set of recommendations, since ocr is dependent on broad subject associations by lc letter or subject, and the scanner software package is designed to interface directly with the subject areas of a bibliographic identifier. next steps currently, topic space relies on the library of congress classes and subclasses to help generate “topic spaces.” in future versions of the topic space module, we would like to include the functionality for dewey call numbers as well. after both classification systems are incorporated into the topic space module, we would then be able to map out the relational areas between dewey decimal classes and library of congress classes. this would allow the use of topic space in additional university of illinois libraries, incorporating cross-unit results between the library system’s 32 collection/service points. figure 13. a proposal for cross-library recommendations and novel location-based recommendation approach for large multi-unit research libraries. demo represents recommendations by location and popularity through undergrad, main library book stacks, and the music & performing arts library. personalized recommender through library account the grant work from developing topic space will continue by way of ongoing library funding supporting student developers in the library. part of this project’s sustainability plan will include keeping student developers active in development of “more like this” functionality within portions of the main library homepage. this will be investigated within the account login screen, where a user logs in to renew items or place requests. the single page web app could provide recommendations from the library homepage account login — since recommendations are powered by supplying an item’s unique bibliographic identifier and user accounts are populated with bibliographic identifiers of checked out items. the personalized recommendations can also be informed by items on the user’s favorite list and generated when an item in the account is renewed. a renewal of an item may be an indication of a topic that the user would like to know more about. since the user may not be able to take advantage of collocation from her account login, the recommendation service may help support this important feature that is lost in the digital library experience. next steps after implementing personalized recommendations require further inquiry into how best to use the data generated by user clicks in the recommender. a truly powerful recommender could result if the data were mined and utilized in providing future recommendations for users. the basis for mining could be check out and requests of the recommendations provided to users. since few libraries have implemented such recommender systems from the account login, we believe sustained research in this area and further study of user behavior will provide the ux community and system designers valuable models for bringing library systems in line with contemporary information systems employed by amazon, netflix, and google. figure 14. prototype of “more like this” services from within a user account. the recommendations utilize a modified topic space api that can suggest items that are related (by subject) and are popular (by circulation count). about the authors jim hahn is orientation services and environments librarian and associate professor at the university of illinois undergraduate library. his research into technology-enhanced learning investigates the development of mobile software applications within library settings and provides unique insights into new students’ expectations and needs. he currently leads the technology prototyping service at the university of illinois. benjamin ryckman, lab manager & lead android developer, has worked with jim and the minrva project team since 2012 to develop mobile applications. he graduated with a bba from eastern illinois university, and is self-taught in the java programming language. ben leads in the development and upkeep of the minrva app (android), regularly programs new apis to provide data for upcoming minrva modules or other university of illinois library applications, and helps coordinate and manage the software developers. maria lux, graphic designer, has a bfa in graphic design and studio art from iowa state university and recently earned her mfa at the university of illinois. maria sees graphic design as a method for making information accessible and usable, and working in the university of illinois library technology prototyping service gives her the opportunity to contribute design approaches to a student-centered project such as minrva. maria is involved with user interface design and the creation of supplementary materials to help share minrva with a wider audience. references azuma, r. (1997). “a survey of augmented reality” presence: teleoperators and virtual environments 6, 4 355-385. bai & blackwell (2012). “analytic review of usability evaluation in ismar” interacting with computers 24, 6 450-460. hahn, j. (2012). “mobile augmented reality applications for library services” new library world 113, 9/10 429-438. hahn, j. (2014). “undergraduate research support with optical character recognition apps” reference services review 42, 2 336-350. jones, j.l., & sinclair, b. (2011). “assessment on the go: surveying students with an ipad” journal of library innovation 2, 2 22-35. jones, t. & richey, r. (2000) “rapid prototyping methodology in action: a developmental study,” educational technology research and development 48, 2 63-80. nielsen, j. (1993). usability engineering, academic press, boston, ma. svenonius, e. (2000). the intellectual foundation of information organization, mit press, cambridge, ma. notes [1] see for example ncsu’s wolfwalk, and a variety of several other native apps on the library success wiki on “m-libraries.” [2] see for example, off the shelf indoor location tools for mobile apps provided by estimote. appendix ux prompt/scenario instructions prompt: you are assigned an interdisciplinary research topic requiring multiple sources. your professor has asked you to find print sources for the paper. you can start from any subject location in the undergrad library book stacks. 1. search for books on your chosen topic using the topic spaces app. to use the app select the topic space icon and then take a picture of call number on a book. from the result of the mobile app can you tell us what other subjects may be of interest to this topic? from the results in the mobile app can you tell us what books are not available? do any of the unavailable books look helpful for this area? from the search results are you able to tell us more about what your topic is about? 2. go to one of the sections of the library that was identified as relevant. can you tell us if the books seem related to your first topic? based on your starting location would you use these books in your research paper? by viewing books in this area, are you able to tell us more about what your topic is about? 3. go to the second section of the library that was identified as relevant. can you tell us if the books seem related to your first topic? based on your starting location would you use these books in your research paper? by viewing books in this area, are you able to tell us more about what your topic is about? interview questions how easy is the app to use? what would make it easier to use? what was hard to do with the app? what was confusing? what was surprising? what do you wish you could have done with the app while you were using it? how useful do you find the app? what would make it more useful? would you recommend it to friends? what do you actually want from a library recommendation app? is there something else that should be here that is not here? is this app a worthwhile tool for the library to develop? observation form *note starting call number: ____ how easy to use is the app? what unexpected things occur? how do students react when the app does not work as they expect? how do students make use of the recommendation features? note any additional observations of student use of the app: subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – enviropi: taking a diy internet-of-things approach to an environmental monitoring system mission editorial committee process and structure code4lib issue 42, 2018-11-08 enviropi: taking a diy internet-of-things approach to an environmental monitoring system monitoring environmental conditions in cultural heritage organizations is vitally important to ensure effective preservation of collections. environmental monitoring systems may range from stand-alone data-loggers to more complex networked systems and can collect a variety of sensor data such as temperature, humidity, light, or air quality measures. however, such commercial systems are often costly and limited in customizability and extensibility. this article describes a do-it-yourself network of bluetooth low energy-based wireless sensors, which seeks to manage earlier-identified trade-offs in cost, required technical skill, and maintainability, based on the raspberry pi™ single-board computer and a series of microcontroller boards. this builds on the author’s prior work exploring the construction of a low-cost raspberry-pi™-based datalogger, iterating upon reviewer and practitioners’ feedback to implement and reflect upon suggested improvements. by monica maceli introduction controlling environmental conditions is an important tool used in preserving collections, with building design and hvac systems widely used to achieve the desired conditions.  tools used in conjunction with hvac systems are independent datalogger devices, which store and monitor data such as temperature, relative humidity, light, and air quality.  commercial devices in this realm are available from vendors such as onset® (with hobo® dataloggers) and the image permeance institute (ipi). these range from simple dataloggers with usb access (such as the pem2® dataloggers) to more complex and costly networked solutions that include wifi or bluetooth capabilities. [1]  used in combination with data collection devices are tools for reporting and data analysis.  the ipi’s eclimatenotebook® is a popular choice for integration with ipi’s dataloggers or other devices whose data can be imported in a suitable text file format. challenges have emerged over time with the use of these devices in practice, with archival professionals occasionally turning to devices in other domains, such as weather monitoring stations or low-cost generic consumer temperature sensors to overcome these issues.  a review of related literature and the findings of a focus group with archival practitioners revealed notable problems including difficulty monitoring collections that are housed off-site, budgeting for costly devices, excessive time spent in collecting data by hand from dozens of devices, and maintaining devices that are unable to be recalibrated easily. [2]  the available commercial devices can be both costly and inflexible in supporting new uses or novel data analysis tools and methods. though there is existing research in this area that has focused on developing and deploying wireless sensor networks (wsns) for environmental monitoring within cultural heritage organizations (e.g., londero et al., 2016; klein et al., 2017) these have not yielded reusable, open hardware and software solutions that could more widely serve cultural heritage organizations.  on a much smaller scale, a simple temperature and humidity logger is a popular do-it-yourself (diy) project that many tinkerers with interest in microcontrollers have created.  there are numerous open source projects, tutorials, and github repositories with creators showcasing their code and solutions in this area, though the majority approach these projects from the home automation perspective and do not specifically address the needs of cultural heritage organizations. the beginning: a simple diy datalogger the author’s prior research has explored stand-alone alternatives to popular and easy-to-implement dataloggers which serve small collections well without the need for complex networked features.  in the summer of 2017 a prototype datalogger was constructed and tested using the raspberry pi™, a popular low-cost single-board computer, which provided temperature and humidity sensing and stored data locally and to the cloud in a google sheet.  more detailed findings are reported in maceli, m. & cocciolo, a. (2017). the device performed comparably to a similarly-featured commercial device (the pem2® datalogger) for approximately ¼ the price.  reviewer and practitioner feedback suggested that the device should be independent of power and networking requirements, have better calibrated sensors, be smaller and more unobtrusive, and have the ability to read data from multiple sensors distributed throughout a larger monitored space.  in addition, practitioners noted the staff time lost in needing to physically collect data from each device via usb drive, as is a common feature with such dataloggers.  for organizations with 60+ devices, an entire staff member’s day might be taken up with collecting data and replacing batteries. exploring wireless solutions the next iteration on the environmental monitoring system, dubbed the “enviropi”, intended to manage the tradeoffs identified in prior research work, as well as support the needs of practitioners expressed in a critique of the raspberry pi™-based datalogger.  the goal of the project was to create an environmental monitoring system made with readily-available off-the-shelf components, capable of being constructed using hobbyist level skills, and using open protocols, software, and hardware as much as possible. a popular protocol for internet of things devices is bluetooth low energy (ble), which was released in 2009.  distinctly different than the original classic bluetooth, ble enables smaller, more power optimized devices and is natively supported by most modern mobile and desktop operating systems.  within ble specifications, generic access profiles (gap) allow ble devices to advertise their name and services, with devices serving as peripherals that connect to a more powerful central device. bluetooth low energy specifications also detail generic attributes (gatt) services: collections of characteristics and relationships to other services, for example, heart rate monitor data in a wearable device.  of relevance to this project, the environmental sensing service (ess) exposes measurement data from an environmental sensor that can include temperature and humidity.  by using gatt standard services, any ble standard device capable of serving as a central device, such as a smartphone or laptop, can view the sensor data provided. furthermore, due to the use of the standard ble identifiers for temperature and humidity, the data is automatically recognized as environmental sensing data (e.g., as seen in the mobile app screenshot in figure 1, below).  given the many potential benefits of ble in lightweight sensing projects, this project explored using ble-enabled arduino devices as mobile battery-powered data collection stations, placed in the environment as needed, that advertise and transmit data using the standard ble environmental sensing service. figure 1. bluetooth low energy (ble) device advertising its name “bluefruit es” as well as temperature, humidity, and battery level services (at the bottom of the screenshot), as viewed within a general-purpose ble mobile app employing arduino and raspberry pi™ arduino is a commonly-used open source electronics platform, with robust software support and numerous expert communities and resources online.  there are a dizzying array of arduino devices and compatible sensors available with different hardware features and price points.  in the case of this project, only arduinos with an integrated bluetooth low energy module were considered, which narrowed the field considerably.  the potential devices were further assessed based on size, ease-of-use (e.g., those with pre-existing software libraries, minimal soldering, vendor-provided documentation, and public forums), use of open source hardware, and inclusion of bluetooth low energy (ble) modules to transmit data wirelessly in a potentially non-networked space.  most devices in this category use nordic semiconductor’s popular nrf51822 bluetooth low energy chip to provide ble capabilities. though many potential devices (figure 2, below) were reviewed and experimented with, the adafruit® feather 32u4 bluefruit with the adafruit® si7021 temperature & humidity sensor breakout board ultimately fit the requirements best and were simplest to implement using adafruit software libraries.  the temperature and humidity sensor chosen provided higher accuracy than those used in the previous datalogger, with ± 3% relative humidity accuracy and ±0.4 °c temperature accuracy.  many of the devices assessed can serve only as a ble peripheral that transmits data and have limited storage space and features, so the system was designed such that all of the sensing devices connect wirelessly via ble to a more powerful central hub, powered by a raspberry pi™. the raspberry pi™ 3 model b+ supports ble natively (as does the smaller raspberry pi™ zero w model) and provides the more robust computing power necessary to host a mysql database to store data, as well as a web server to present data visualizations to viewers or json endpoints for use by other services.  this also provides the future flexibility to integrate with many popular and powerful data dashboard and analysis tools, most of which can read data live from a relational database or json endpoint as needed. figure 2. sample of ble devices assessed, including (from left to right): the adafruit® feather nrf52 bluefruit, the rfduino, an nrf51 sensor tag, and the adafruit® feather 32u4 bluefruit. prototype design the final prototype system design (figure 3, below) consisted of a raspberry pi™ 3 central device, running a mysql database and an apache web server on the raspbian operating system, along with a python 3 script utilizing the pygatt library to read data from the ble sensors at 10-minute intervals initiated by a cron job.  the raspberry pi™ is powered by an electrical outlet and included a small touch-screen, as well as an external wi-fi antenna to boost its ability to connect to the available wireless lan from a further distance.  a simple touchscreen interface, displaying the latest device readings and buttons to control the raspberry pi™, was created using python’s tkinter graphical user interface package (figure 4, below). figure 3. prototype system design figure 4. raspberry pi™ central device (left) and one adafruit feather 32u4 bluefruit-based sensor device (right) concealed in a project box enclosure an automated install script provides for initial setup of a customizable number of ble device addresses, corresponding to the sensor in use. each remote sensor is housed in a plastic project box enclosure and includes an adafruit® si7021 temperature sensor attached to an adafruit feather 32u4 bluefruit board and a lithium polymer (lipo) battery (figure 5, below).  at 10-minute intervals, the python script reads from each configured and available ble device and writes the data to the mysql database.  php and chart.js were used to create a simple web-based chart interface (figure 6, below) and json endpoints, though any combination of web technologies capable of connecting to a relational database or reading json data could be used in the future. figure 5. components of each sensor, including (from left to right): the adafruit® si7021 temperature & humidity sensor breakout board, the adafruit® feather 32u4 bluefruit, and a lithium polymer (lipo) battery. figure 6. web interface hosted on raspberry pi, with two sensors reporting data on the battery-powered remote sensor devices, adafruit and sparkfun software libraries for working with the feather 32u4 and the si7021, respectively, were incorporated into the arduino sketch.  the ble specifications were used as a reference when creating the environmental services and characteristics (e.g., for the temperature characteristic), then the code was written and uploaded to the devices using the arduino ide.  a snippet of the arduino code to create the ess service and temperature is included below, with the full sketch file available on github: /* add the environmental monitor service */ // environmental sensing org.bluetooth.service.environmental_sensing 0x181a  gss serial.println(f("adding the environmental sensing service definition (uuid = 0x181a): ")); success = ble.sendcommandwithintreply( f("at+gattaddservice=uuid=0x181a"), &amp;esserviceid); if (! success) { error(f("could not add es service")); } // temperature ess characteristic serial.println(f("adding the temperature measurement characteristic (uuid = 0x2a6e): ")); success = ble.sendcommandwithintreply( f("at+gattaddchar=uuid=0x2a6e, properties=0x10, min_len=1, max_len=5"), &amp;tempmeasurecharid); if (! success) { error(f("could not add temperature characteristic")); } as mentioned earlier, using the ble standardized identifiers for environmental services and characteristics, e.g. temperature characteristic identified as “0x2a6e”, allows the data advertised by the device to be recognized as such by any ble-enabled device, such as a mobile phone ble scanner app.  each sensor runs the same code but may be distinguished by its unique bluetooth hardware address, for scanning and data storage purposes. in the current prototype, the raspberry pi™ connects to an existing lan via wi-fi or ethernet.  if no lan connection is available, say in remote storage locations, there is also potential to add a cellular shield for cell phone network access to the raspberry pi™ central hub.  the sensing devices may connect to any central ble device in range, such as one’s smartphone, to obtain sensor readings directly off the device.  as discussed later in the future work section, new tools exist for rapid mobile app development, giving the option of creating and deploying one’s own simple mobile app for reading data directly from sensor devices. the unobtrusive case and a touchscreen for the raspberry pi™ could be considered optional add-on features, as the device’s data is web accessible.  however, practitioners’ feedback in the focus group emphasized the need to walk around the building space and easily view environmental condition data directly on the device.  also, the need for an aesthetically pleasing, or at least unnoticeable, device enclosure was stressed as well, so these features were required. testing the prototype though the devices performed reliably over a test period of several months within the expected accuracy parameters (gathering tens of thousands of readings), two main challenges during operation emerged: 1) managing battery life versus range, and 2) the relative flakiness of the ble connections.  minimizing power usage is a key concern in iot devices generally, many of which are expected to run without user intervention for months or years at a time.  though wireless protocols designed for use with iot devices are relatively low power, there is a tradeoff between wireless range and battery usage.  the arduino code written employed several power-saving measures, such as turning off leds and using a low power library to sleep the device between reads. additionally (and with a more significant impact on battery usage), the transmitting radio power of the ble device can be lowered by configuring the arduino’s code to preserve battery usage, but this, in turn, reduces the signal range.  this can lead to inconsistencies in battery life and sensor reading reliability depending on the physical layout of the space in which they are deployed.  however, the adafruit devices chosen may also be powered from a computer’s usb port, which provides a means of avoiding battery constraints entirely if archival storage locations are co-located with computer workstations. several types of batteries and connectable battery packs were tested, including lipo, alkaline, and rechargeable batteries.  though all types could be suitably connected to and run the feather device (anywhere from one week to several months depending on battery type and capacity), the lipo battery proved to supply the longest-lasting power in the smallest form factor with the added benefit of being able to be recharged from the usb cable.  however, lipo batteries require special care in charging and storage, to reduce fire risk, which may be unacceptable to an organization with valuable collections. though development is quick and easy with prototyping boards, such as those from adafruit, the downside is that such devices often include extra features that consume power but may not be needed for the final project.  designing and manufacturing a custom board with the bare minimum features needed would allow for reducing the power needs more drastically, as is seen in commercial products.  for the more robust central device, the raspberry pi™ running the web server and database, battery power is not a viable option, unless the device is to be run for short periods.  even the smaller and less power-hungry raspberry pi™ zero w has relatively high-power needs compared to the arduino.    hopefully, it is reasonable to assume that the one central device would be able to have access to a power outlet and thus avoid the need for battery power. another issue observed was the tendency of the bluetooth connection to drop mid-stream or fail to be made when attempting to read data from the sensors.  to combat this, the final python script to read data allowed for several tries to be made during each read and included error handling for the various exceptions that may be commonly raised.  despite some intermittent failures in reading the data at each interval, given a few tries the data was always able to be successfully read eventually, so no data points were ultimately missing. more generally, due to the various constraints and tradeoffs around using battery-power and achieving the needed functionality, the project necessarily employed several different technologies and skillsets, from designing a (simple) relational database to writing and modifying code in multiple languages.  with the installation script, a raspberry pi™ device and its sensors could be set up and operational in a few hours, but only if the creator was familiar with the techniques and tools used (for example – imaging an sd card for raspbian, light through-hole soldering, etc.) conclusions and future work in the world of manufacturing, supply chain solutions, and elsewhere, iot sensor networks for environmental monitoring have become commonplace.  however, these don’t necessarily fit the needs of cultural heritage organizations, which are often limited by budget and tend to develop their environmental monitoring solution in a piecemeal fashion over time.  additionally, preservation metrics that assist archivists in identifying problematic environmental conditions for artifacts of various materials are specific to the cultural heritage realm.  of the devices and systems that are oriented towards the monitoring work of cultural heritage organizations, the practitioner focus group detailed many desired improvements that are either currently unavailable or unaffordable to them.  as a result, many are turning to cheap generic consumer devices or ones designed for other domains, such as weather-monitoring. the focus group also detailed novel features that would expand the scope of the project immensely.  many practitioners reported checking in on environmental conditions from home or on weekends and advocated for a remotely-accessible mobile app and responsive web interface to make this work more efficient.  though most reported that one staff member took the lead in monitoring activities, they also expressed a desire for the ability to allow other users access to the data, with the available features restricted to different user roles, such as themselves as an administrator or a board member as a viewer. the cultural heritage field, more generally, supports many popular and active open source software projects.  a goal of this project was also to employ open source hardware, which opens future possibilities for modification and customization.  the full set of requirements generated by the practitioner focus group is ambitious, in suggesting the development of a mobile app and web interface, as well as multiple iot devices that must run without user intervention for significant time periods.  these are ongoing challenges in the wider realm of iot work and, in particular, dealing with the power-related tradeoffs may be a difficult task.  in the context of a small-scale open source, low-cost project, these requirements may currently be out of scope for a project without the funds and ability to design novel hardware and/or engage in significant mobile app or web development. as mentioned earlier, though, new services and tools are emerging on a frequent basis, that begin to tackle some of the common needs of iot projects, particularly those run by the hobbyist.  on the data dashboard side, new cloud services, such as the adafruit io project, aim to provide “the easiest way to stream, log, and interact with your [iot] data.”  these may provide the means to avoid reinventing the wheel in developing a novel mobile app or responsive web interface.  moreover, tools such as adafruit io also offer appealing data ownership and retention policies. an experimental dashboard was created using this service, by adding an adafruit-created python library to the raspberry pi™’s code, such that it would take the additional step of pushing data to the cloud service each time it was read.  an adafruit.io user account was created, then several feeds were configured for temperature and humidity data collection. this feed data was then visualized through a widget-based dashboard interface.  though the features of the adafruit.io service are still in development, a simple “live” data dashboard was created with relative ease (figure 7, below).  this would remove the need for a web server and custom web interface hosted on the central hub device computer. figure 7. example experimental data dashboard created through adafruit.io with live enviropi data another promising tool is the blynk platform which allows for quick iot mobile app prototyping and deployment.   the platform facilitates easy creation of mobile apps that directly connect to ble devices or control devices remotely via wi-fi (if the device is wi-fi-capable).  utilizing arduino devices with wi-fi capabilities (though these tend to require more power) would allow easy integration with such services. apps could also be used to read data directly off devices when traversing the physical space. a simple example (figure 8, below), created in under an hour using the blynk arduino library, is able to automatically connect to the device over ble, when in range, and read from the sensors at the desired time periods.  another option is running one’s own blynk server, which can be installed on the raspberry pi™, removing the need for a cloud host and handling all the interactions and data between the mobile app and connected sensors. figure 8. a minimal android mobile app created in blynk, connected to sensor device and displaying live readings of the temperature/humidity sensor these general-purpose iot services, and a growing number of other alternatives, offer many novel possibilities in integrating with the variety of tasks involved in environmental monitoring, from data collection to analysis and sharing, while allowing for a more open community-driven codebase.  in addition to services, new iot hardware devices for wireless and mesh networks offer many future possibilities, such as the particle mesh microcontrollers.  the author’s research work in this area will be continuing, and additional ideas, comments, or contributors are very welcome. endnotes [1] see http://www.onsetcomp.com/ and https://www.imagepermanenceinstitute.org/ for examples. [2] maceli, m. (2018). [environmental monitoring focus group]. unpublished raw data. bibliography klein, l.j., bermudez, s.a., schrott, a.g., tsukada, m., dionisi-vici, p., kargere, l., marianno, f., hamann, h.f., lopez, v. & leona, m. (2017). wireless sensor platform for cultural heritage monitoring and modeling system. sensors, 17(9), 1-21. londero, p., fairbanks-harris, t., & whitmore, p. m. (2016). an open-source, internet-of-things approach for remote sensing in museums. journal of the american institute for conservation, 55(3), 166-175. maceli, m. & cocciolo, a. (2017). monitoring environmental conditions with low-cost single board computers. preservation, digital technology & culture, 46(4), pp. 124-131. about the author monica maceli is assistant professor at pratt institute school of information, focusing on emerging technologies in the information and library science domain. she earned her ph.d. and msis from the college of information science and technology (ischool) at drexel university. she has an industry background in web development and user experience, having held positions in e-commerce, online learning, and academic libraries. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – editorial introduction – summer reading list mission editorial committee process and structure code4lib issue 33, 2016-07-19 editorial introduction – summer reading list new additions for your summer reading list! by ron peterson it is time for the summer edition of the code4lib journal. while you have been making vacation plans, the editors of the code4lib journal have been busy putting together issue 33 of the code4lib journal. if you are looking for something to inspire a summer project or just looking for something to read on the beach, we have a collection of page-turners for you. you may want to start by reading netanel ganin’s post-mortem on the development of a web-based dvd browsing interface, “emflix – gone baby gone.” netanel takes us through the triumphs and defeats of his project and reminds us that “enthusiasm is no substitute for experience.” i’m sure that many of us can relate to his experience. if you have been considering diving into r during the summer, monica macelli’s “introduction to text mining with r for information professionals” tutorial may be just the thing that you are looking for. monica shows us how she used the tm: text mining package and rstudio to analyze the text of 51 course catalogs and create visualizations of the data. in his article, “data for decision making: tracking your library’s needs with trackref,” michael carlozzi describes how he developed trackref in order to get a better understanding of the types information that users were seeking. with a better understanding of how users were using the library and what issues they were running into, michael’s library was able to redeploy staff in ways that would better address the needs of their community, such as freeing up staff to work one-on-one on technology. looking to add a little more fun to your summer? read about how the biodiversity heritage library used games to improve access to their digital texts. max j. seidman, dr. mary flanagan, trish rose-sandler, and mike lichtenberg describe the development of two online games, smorball and beanstalk, and how they used them to improve the accuracy of ocred texts in “are games a viable solution to crowdsourcing improvements to faulty ocr? – the purposeful gaming and bhl experience.” for those who are looking for ideas on how to improve the efficiency of processing their electronic theses and dissertations, we have, “from digital commons to oclc: a tailored approach for harvesting and transforming etd metadata into high-quality records” by marielle veve. marielle takes us through the process that the university of northern florida went through to adapt a workflow for transforming dspace’s qualified dublin core records into marc records so that it would work with digital commons. also in this issue, zsolt bánki, tibor mészáros, márton németh, and andrás simon describe how they created a namespace for authors of texts written in hungarian in ther article, “checking the identity of entities by machine algorithms: the next step to the hungarian national namespace.“ they take us step by step through the process of merging and deduplicating data from multiple sources in order to create a usable, semantic web related namespace. they hope that the namespace will provide some foundation for establishing a national name authority in hungary. for the quintessential beach read, pack your blanket, your e-reader, and a copy of corey harper’s “metadata analytics, visualization, and optimization: experiments in statistical analysis of the digital public library of america (dpla).” the article covers processing the metadata to prepare it for analysis, creating visualizations, and optimizing the data. if you have been looking to dive into statistics more deeply, this is the article for you. corey discusses various statistical models and their applicability to predicting item usage. you won’t be able to put it down. i hope that you enjoy the articles presented in issue 33. the authors and the editors have worked very hard to make them available to you. follow up on the gender representation in the code4lib journal as i was writing this introduction, i was reminded of the introduction that i wrote for issue 24, which addressed the gender diversity of the code4lib journal. i thought it would be interesting to see how we are doing now. sadly, there doesn’t seem to have been much movement towards a more inclusive journal. of the 7 articles in this issue, only 3 of them include female authors – 2 solo authors and 1 article authored by both males and females. in contrast, men are included as author on 5 of the 7 articles (3 solo male authors, 1 authored by 4 men, and 1 authored by both). the makeup of the editorial committee has also not changed much since then. the current editorial committee is made up of 4 women and 8 men. the number of female editors is down from issue 24, when it was 5. since issue 24, we have added 3 female editors, but we have lost 4. i was surprised to see that women made up only a third of the committee because my sense was that they accounted for the majority of the activity for issue 33. as it turns out, my sense was correct and women were the primary editors of 4 of the 7 articles in issue 33. while these numbers can only tell us part of the story, i do think they are an indication that there is still work to be done to make the code4lib journal better reflect the diversity of the code4lib and library communities.   subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – large-scale date normalization in archivesspace with python, mysql, and timetwister mission editorial committee process and structure code4lib issue 44, 2019-05-06 large-scale date normalization in archivesspace with python, mysql, and timetwister normalization of legacy date metadata can be challenging, as standards and local practices for formulating dates have varied widely over time. with the advent of archival management systems such as archivesspace, structured, machine-actionable date metadata is becoming increasingly important for search and discovery of archival materials. this article describes a recent effort by a group of yale university archivists to add iso 8601-compliant dates to nearly 1 million unstructured date records in archivesspace, using a combination of python, mysql, and timetwister, a ruby gem developed at the new york public library (nypl). by alicia detelich introduction in august 2017, representatives from across the yale university library (yul) system began a year-long effort to implement the archivesspace public user interface (pui) as the primary search and discovery platform for special collections materials at yul. the customized interface, dubbed archives at yale, launched in january 2019 replacing an access system that had been in use, mostly unchanged, for nearly a decade.[1] the project team consisted of more than 35 yul staff divided into six working groups, including the five-member data cleanup and enhancements workgroup. this workgroup was tasked with creating and updating metadata stored in archivesspace to optimize search and display functionality in archives at yale. during an initial planning meeting, the group identified and prioritized a number of data elements on which to focus cleanup and enhancement work. several major differences between yul’s legacy finding aid database and the pui guided the group’s decision-making, including a feature of the pui which enabled searching by date.[2] utilization of the pui’s search-by-date feature requires that iso 8601-compliant dates be present in the structured ‘begin’ and ‘end’ fields of archivesspace date subrecords. the group was aware that much, if not most, of the date metadata in archivesspace was represented only in the free-text ‘expression’ field, and that the majority of this data was not in iso 8601 format. numerous discussions about normalizing these date records had taken place since yul migrated its data to archivesspace in 2015, and the pui implementation project presented a good opportunity to complete the task. [3] the archivesspace data model the archivesspace application is based on a data model comprised of numerous records, which are stored in a mysql database, and which can also be created, updated, or deleted via a robust api. these records support an array of archival functions, including accessioning, description of analog and digital materials, authority control, and collection management.[4] at a very basic level, records in archivesspace can be divided into three categories: “top-level” records, subrecords, and enumerations. top-level records are the primary record types in archivesspace. they are assigned uris, and their json representation can be accessed directly via the archivesspace api. top-level records can be further divided into three types according to their function: functions record types description resources, archival objects, accessions, classifications authority control agents, subjects collection management top containers, digital objects, locations, events, collection management, repositories the most commonly used top-level records are resources and archival objects, which correspond, respectively, to collection-level descriptive records and their hierarchical components. archival objects must always be associated with a resource, and cannot be created independently. most record types are linked to a particular repository, though agents, subjects, and locations are shared across all repositories. numerous subrecord types exist in the archivesspace model. a few examples: subrecord type description dates dates associated with material creation, collection management events, etc. extents material type and/or quantity. 15 linear feet, 25 photographs notes free-text description fields. scope and content, conditions governing access, etc. subcontainers what’s inside a top container. folder, item instances link between top  containers and descriptive records subrecords cannot exist on their own, but must be associated with a top-level record. as such, they do not have their own uris, and can only be accessed via the api by retrieving the top-level record with which they are associated. subrecords are repeatable, so their json representations are stored in lists within an associated top-level json record. the various subrecord types have their own database tables, are assigned primary keys, and contain foreign keys which link them to top-level records. figure 1. date subrecords in a resource record json response the archivesspace data model is also comprised of many types of controlled values, or enumerations. enumeration values are assigned an identifier in the database, and this identifier is used as a foreign key in many top-level records and subrecords. the json response from the api represents enumerations as strings. a few controlled value types: enumeration values extent type photographs, linear feet, videocassettes (vhs) name source lcnaf, viaf, snac archival record level collection, series, sub-series, file, item archivesspace comes with a standard set of controlled values, but these can be modified by users. date records in archivesspace date subrecords in archivesspace are comprised of several writeable fields. the ‘expression’ field can contain any free-text date formulation, and there is no constraint on adding non-date values to this field. the ‘begin’ and ‘end’ fields must contain iso 8601-formatted date strings. three types of controlled values – date type, date label, era, and calendar – are also included in the record. in the database, these fields contain foreign keys that link the date table to the enumeration_value table. in the json response from the api controlled values are represented as text. figure 2. date table structure identifying unstructured dates yul migrated its archival metadata into archivesspace june 2015. a single production instance consists of 15 repositories representing 12 of yale’s special collections repositories. prior to 2015, a wide range of systems were used to manage archival metadata at yale, including but not limited to three separate archivist’s toolkit databases. practices for creating records within these systems varied widely across repositories and over time. the resulting lack of conformity meant that most date metadata was migrated into the ‘expression’ field, and not the structured ‘begin’ and ‘end’ fields. dates records created since 2015 more often, but not always, included structured dates. to determine just how many of our date records were unstructured, we queried the archivesspace database to identify all dates with a value in the ‘expression’ field but null values in both of the structured fields.[5] the resulting report contained three columns – the database identifier for the date subrecord, the uri for the top-level record to which it was linked, and the value of the ‘expression’ field. figure 3. snippet of get_unstructured_dates.sql the results of the query confirmed that the majority of our date records were unstructured – of the 1.65 million date records in the database in november 2017, 1.08 million, or about 66%, had a date value in the ‘expression’ field only. searching for a scalable solution the large number of unstructured dates and the enormous variety in the content of the ‘expression’ field led us to experiment with a number of approaches before finding one that met all of our requirements. our goal was to complete most or all of the work in-house, using our existing skill sets, in a relatively short period of time. we also wanted a mostly automated process which facilitated the re-use of data through various processing steps. and, of course, we needed a solution which could accurately convert a wide variety of dates to iso 8601 format. openrefine is a very useful tool that we all had experience using in the past, but it was not ideal for processing such a large dataset, even after allocating additional memory to the application. we also found writing a comprehensive number of grel expressions from scratch to be very time consuming, and we preferred not to recreate something that might have already been done well by someone else.[6] in addition to its dateutil module, python has a number of third-party date parsing libraries, including dateparser and daterangeparser.[7] python is capable of processing large amounts of data very quickly, but during testing we found the date parsing functions to be less comprehensive than other methods, and significant additional processing was often required to output accurate results. we were also reluctant to rely on a hodge-podge of third-party solutions, some of which did not appear to be widely used, as they presented issues for the future maintenance of any tool we created for this project. we first heard of the timewalk date parsing plugin for archivesspace from discussions with fellow members of the user community, and we considered ways we could use the plugin to work with legacy data.[8] we knew that the parsing function was triggered upon saving a record in the archivesspace staff interface, so we attempted, in a test instance, to get/post a number of records via the api without making any changes. the legacy dates we tested were parsed by the timewalk, and though the results were generally accurate, we were left with a number of errors. we decided that we needed to parse the dates outside of the application and perform quality control on the results before pushing the changes to production. timetwister the timewalk plugin is based on timetwister (itself based on chronic), which was developed by alex duryee, trevor thornton, and others at the new york public library (nypl). timetwister is a ruby gem with a command-line interface that provides powerful date parsing capabilities, including date ranges and multiple dates.[9][10] while most members of the group – all technologically-oriented archivists – had a basic knowledge of ruby and were comfortable working on the command line to accomplish routine tasks, we felt that writing a ruby application or complex bash script that employed timetwister would have been overly time-consuming considering our skill levels in these areas. our established skill sets and workflows relied heavily on python and sql, and we wanted a solution which would allow us to use tools while also taking advantage of the powerful and accurate parsing capabilities of timetwister. python + timetwister = parse_dates.py one of the most useful features of python is its extensive standard library, which provides many advanced capabilities for working with data and operating systems. one of these, the subprocess module, enables users to run external commands from within a python program, similar to how one would on the command line. [11] having employed this module in other contexts, we knew that it we could use it to make repeated calls to the timetwister command from within a python for-loop. with that in mind, we wrote a script, parse_dates.py, which performs the following sequence of actions:[12] connect to the archivesspace database using the pymysql and paramiko modules execute the get_unstructured_dates.sql query and store the results in a list iterate over the result list and call the timetwister command with the subprocess module’s ‘popen’ class, using each date expression as an argument parse the ‘original_string’, ‘date_start’ and ‘date_end’ values from the resulting json response into a list. the ‘original_string’ field was retained to compare what was actually parsed with the original query data. append the timetwister data to the query output write the output to either a single csv or a series of csv files based on type. if split into multiple csvs, append the date type id value to the output figure 4. snippet of parse_dates.py showing steps 3-5 the total runtime of the script was approximately 12 hours. we split the output into several csvs in order to make it easier for the group members to review the output, and, as will be explained later, to perform the updates in chunks. the following table summarizes the results (with estimated counts): date type count percentage single: value only in ‘date_start’ field 330,000 30% inclusive: : values in both the ‘date_start’ and ‘date_end’ fields 490,000 45% inclusive-begin: value like ‘2014-’ in the ‘date_start’ field 125 .01% inclusive-end: value like ‘-2014’ in the ‘date_end’ field 5 .0005% multiple: multiple date values in a single record,  i.e. ‘october 2010, november 2016’ 85,000 8% unparsed: values could not be parsed by timetwister, i.e. ‘undated’, ‘april 5’, ‘tuesday’ 175,000 16% errors: any other errors 250 (resolved by re-running parse_dates on subset of records) .02% totals ~1,080,280 ~100% the group then manually reviewed the output. because of the large quantity of dates, we used a sample size calculator to determine a reasonable number of records to review in order to be confident in the results – approximately 15,000 divided between five group members.[13] we excluded the unparsed, inclusive-end and inclusive-begin reports from the review. with the former, we wanted to return the results to the yul repositories that created the records and ask them to consider supplying estimated dates. the latter two we also felt should not be updated until they were corrected to include a full date range. we set aside the multiple date report, as well, since updating those would require correcting the expression field and creating a number of new date records, necessitating a different approach than the majority of the date records. all of the date records we reviewed were accurately parsed. our overall success rate was 75% – of the 1.08 million date records processed by timetwister, over 820,000 were converted into iso 8601 format. another 8% could be updated with additional processing. the majority of the dates which timetwister could not parse would not have been parseable by any tool. choosing an update method the group considered both of our options for ingesting the output of the timetwister script into archivesspace: either via a database update or the api. archivesspace database access for yul staff is read-only, so we first considered using the api to make the updates.[14] however, because subrecord ids are not included in the json response from the api (except, for some reason, in digital object file version subrecords), and a single top-level record can have multiple associated dates, we weren’t sure how to precisely locate the dates we wanted to update. we considered using the date expression string as a match point, which is probably sufficient in most cases, but also seemed less accurate than matching on the identifiers, as the original expressions could be changed in archivesspace at any time prior to us running the update. the api can be rather slow, as well – we estimated that the update script would take between 3-7 days to run, during which time a host of connection issues and other errors could occur. figure 5. snippet of api update script a database update seemed more logical, as it runs much faster, and the database identifiers could be used to ensure precision. however, while we were testing this option on a local copy of the database, we discovered an interesting property of archivesspace subrecords – they are not persistent. when a top-level record is saved, all of its associated subrecords are deleted and recreated, and the “new” subrecords are assigned new database identifiers. this design choice was likely made to improve application performance, but it seemed odd, and required us to reconsider whether our proposed method was feasible. after some discussion, we concluded that if we re-ran the parse_dates.py script relatively close to when we made the update to our production database, the number of changed date identifiers would be acceptably small.[15] the time savings and limited error potential of an sql update as opposed to the api was a major factor in this decision. we wrote two sql scripts which created a temporary date table, imported the timetwister output, and then updated the original date table with the values from the temporary table, without modifying any of the other fields in the table. we tested these scripts on the local database and were satisfied with the outcome. the scripts took just a few minutes to run and all records updated successfully. figure 6. update_dates.sql because we split the output into multiple csvs by date type for quality control purposes, we decided that it would be easiest to create two temporary tables and run the update script in two chunks. this required us to slightly modify the update script depending on the date type, as single dates do not have an ‘end’ field. it is also possible to run the update using a single csv and temporary table by adding a line which converts any blank fields in the csv file to null values. because yul’s archivesspace database is read-only, once we finished testing locally we contacted our hosting provider, lyrasis, to coordinate the production update. we provided them with our sql scripts, which they modified and executed, first on our test instance of archivesspace, and then, after we reviewed the output, in production. we are confident in the accuracy of the parser, but since it was not feasible to manually review every record, we retained the original date expression value in addition to the newly-added ‘begin’ and ‘end’ values. this makes it possible for us to identify and correct any errors that may have occurred. while we were concerned about changing date identifiers leading to failed updates, fewer than 1.5% – about 12,000 – of the database updates failed due to this issue, despite a (longer than planned) 3-week gap between data processing and update. based on our previous experience, this failure rate is likely much lower than if we had performed the update via the api. analysis the workgroup was very pleased with the results of this project. we increased the percentage of iso 8601-compliant dates in yul’s archivesspace database from 33% to 83%, vastly improving the utility of the pui’s search-by-date function, and thus the ability of our users to locate relevant archival and manuscript materials. our work has also enabled yul staff and other interested parties to perform more robust temporal analyses on our archival metadata. by using an existing tool in a novel way, capitalizing on our skill sets, and creating an efficient data processing workflow, we were able to normalize a large quantity of date records with just 15-20 hours of staff time over two months. additionally, we saved yul thousands of dollars which had been budgeted for a vendor to do the normalization work. we also worked to make our process reproducible by ourselves and others, and have continued to refine the scripts and our workflows. the parse_dates.py script can now be run interactively on the command line by users with relatively little technical experience, and we’ve made it easier to use the combined date output and perform just a single database update, among other improvements. figure 7. project workflow (revised) the update was completed in february 2018, and to maintain the work we did during this project in 2019 and beyond, we began executing the normalization process, among other data auditing and cleanup tasks, on a quarterly basis. we continue to work through the dates which could not be parsed by timetwister and those which require additional processing. our experiences on this project also led us to consider ways in which the archivesspace application could be improved to better handle dates and other subrecords. while it would be ideal for subrecord identifiers to be persistent, this would likely require a level of development resources and community backing that is not currently available. adding ids to subrecord json responses would be a much easier task to accomplish, and could also be done via a simple plugin. perhaps the best (realistic) option is a system-generated persistent identifier field, like those that already exist for note subrecords, which could be added to all other subrecords, as well. this could also be done with a local plugin, but would ideally be added to the core code, since it would improve the precision of subrecord updates made via the api for all users. source code and documentation ultimately, the point of this project was to provide a new path for researchers to discover special collections materials, and by this measure we feel that we were very successful. during the process we learned a great deal about creating efficient data processing workflows that rely on existing tools and skills to make the most of limited time and resources. we hope that others can make use of the tools we created as part of this work. all of the scripts and a tutorial are available on our github page: https://github.com/yalearchivesspace/yams_data_auditing/tree/master/dates about the author alicia detelich is a metadata archivist in the manuscripts and archives department at yale university’s sterling memorial library, and she was co-leader of the data cleanup and enhancements workgroup of yul’s archivesspace public user interface implementation project. her work is focused on developing computational tools and automating workflows for creating and enhancing archival description. references [1] archives at yale home page: https://archives.yale.edu/ [2] yale university library public user interface project data cleanup workgroup blog post: https://campuspress.yale.edu/yalearchivesspace/2017/11/29/cleaning-data-to-enhance-access-and-standardize-user-experience-part-i-planning-and-prioritization/ [3] iso 8601 wikipedia page: https://en.wikipedia.org/wiki/iso_8601; iso 8601 standard web page (full text behind paywall): https://www.iso.org/iso-8601-date-and-time-format.html; https://xkcd.com/1179/ [4] archivesspace documentation: https://archivesspace.github.io/archivesspace/doc/ [5] get_unstructured_dates.sql: https://github.com/yalearchivesspace/yams_data_auditing/blob/master/dates/get_unstructured_dates.sql [6] openrefine application documentation: http://openrefine.org/documentation.html [7] dateutil library documentation: https://dateutil.readthedocs.io/en/stable/ dateparser library documentation: https://pypi.org/project/dateparser/; daterangeparser library documentation: https://pypi.org/project/daterangeparser/ [8] timewalk source code and documentation: https://github.com/alexduryee/timewalk [9] chronic source code and documentation: https://github.com/mojombo/chronic [10] timetwister source code and documentation: https://github.com/alexduryee/timetwister [11] subprocess library documentation, python 3.7: https://docs.python.org/3/library/subprocess.html [12] parse_dates.py: https://github.com/yalearchivesspace/yams_data_auditing/blob/master/dates/parse_dates.py [13] surveymonkey sample size calculator: https://www.surveymonkey.com/mp/sample-size-calculator/ [14] update_dates_by_expression.py: https://github.com/yalearchivesspace/yams_data_auditing/blob/master/dates/update_dates_by_expression.py [15] create_datetemp_table.sql: https://github.com/yalearchivesspace/yams_data_auditing/blob/master/dates/create_datetemp_table.sql update_dates.sql: https://github.com/yalearchivesspace/yams_data_auditing/blob/master/dates/update_dates.sql subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – iiif by the numbers mission editorial committee process and structure code4lib issue 48, 2020-05-11 iiif by the numbers the ucla library began work on building a suite of services to support iiif for their digital collections. the services perform image transformations and delivery as well as manifest generation and delivery. the team was unsure about whether they should use local or cloud-based infrastructure for these services, so they conducted some experiments on multiple infrastructure configurations and tested them in scenarios with varying dimensions. by joshua gomez, kevin s. clarke, anthony vuong i. introduction organizational context in the fall of 2018, the ucla library’s software development team underwent a change of leadership. with that change came an effort to modernize the team by adopting agile planning and a devops culture. see (gomez, 2020) for a discussion of that process. along with a change in our processes came a change in our software architecture goals, in which we aspire to adopt “evolutionary” architectures, such as event-driven and microservices as described in (ford 2017). by definition, evolutionary architectures are more maintainable and easier to change over time. the goal is to avoid “setting and forgetting” a large portfolio of fragile legacy monoliths that are difficult to maintain let alone improve upon, which is the situation we are in now. instead, we hope to continually update an ecosystem of small components. architecture prior to the new leadership and organization, the larger team had already begun work on a new digital library platform, an implementation of samvera [1]. in an initial effort to make the new digital library less monolithic, the services team was tasked with designing and building an independent suite of services to support iiif [2], and to relieve the samvera stack of those duties. this suite consisted of three independent services: image transformation service. this service accepts a csv containing a row for each digital image, locates the source images files, generates derivatives, and pushes them to a designated destination. we will be migrating over 3 million items out of our old legacy digital library platforms, which made heavy use of tiffs. therefore, if we choose to use jpeg2000, a robust transformation service capable of high throughput will be necessary. image delivery service. this service accepts a iiif image api [3] request and delivers a jpeg derivative to match. manifest generation & delivery service. this service accepts a iiif presentation api1 request and delivers the appropriate manifest. it also accepts a csv containing a row for each collection, each work within a collection, and each image within a work, generates the iiif manifests and stores them for later retrieval. this paper is a description of the approach our team took when designing these services. ii. system design decisions when it comes to implementing iiif infrastructure there are many decisions to be made, including (but not limited to): which file format should we use: tiff, pyramidal tiff, jpeg2000 (jp2k)? if we choose jp2k, should they be lossy or lossless? should we use openjpeg or kakadu for transcoding? which image server should we use: loris, cantaloupe, or iip? should we do file conversion in the local data center or in the cloud? if in the cloud, with dedicated vms or with serverless functions? should we put our images and server in our local data center or in the cloud? if we use cloud computing, which one should we use (aws, gcp, azure)? should we scale the services vertically or horizontally? what should we use for caching: redis, s3, cloudfront? we knew that we were facing more decisions than we could possibly try to answer definitively while still delivering on the implementation in a timely manner. so we had to decide what to decide. we had to ask ourselves “which questions are critical to start work? which questions can we make a quick decision on now that would not be too difficult to change later?” quick decisions tooling seemed the easiest set of choices for us to make as we already had some experience with them and if we changed our minds the switching costs would not be insurmountable. kakadu. we chose to use kakadu [4], which is proprietary, instead of openjpeg [5], which is open source. there have been recent efforts to improve the performance of openjpeg, and we hope that these continue, but in our own trial runs we found kakadu to still be faster. furthermore, we found a pair of limitations with the obj_compress function in openjpeg. it was unable to perform multithreaded processing and could not work with cielab color images. since the price of kakadu was quite reasonable, we deemed it an easier path. getting an overseas software vendor approved by the campus purchasing office did present us with a six month delay, but the vendor was very accommodating with an extended development license throughout that phase. cantaloupe. we chose to use cantaloupe [6] as our image server because our team is well versed in java and we liked the flexibility that cantaloupe offers with its delegate scripts. aws. our campus already had a discount with amazon web services (aws), so we knew that if we decided to move any portion of our services to the cloud, that aws would be the cheapest option for us. furthermore, it is the most mature cloud provider and we already have some applications running there, so the path to adoption was an easy one for us. decisions needing analysis we had a lower level of confidence for the remaining decisions. unlike those above, these decisions were not influenced by campus level agreements (like aws). and unlike swapping out standards-compliant components with others (like cantaloupe and kakadu), the remaining decisions affected our own software designs, and would therefore have significant costs in developer time if we decided to backtrack. these were the issues for which we decided to run a few experiments. in these experiments we ran tests on-site and in the cloud, but the analysis was essentially about vertical versus horizontal scaling (i.e one big machine versus multiple smaller ones working in parallel). image transformation. we compared processes for creating jpeg2000 derivatives on-site versus using aws lambdas, and to do so for varying sizes of images and also to compare results for lossy versus lossless compression. image delivery. we compared performance of the image server on-site and in the cloud and compared the performance for varying image sizes and traffic levels. iii. experiment #1 – image transformation in this experiment we measured the length of time required to transform 1000 source tiffs into jpeg2000 derivatives with variations along three dimensions: hardware/scaling: local (vertical) vs. serverless (horizontal) derivative quality: lossy vs. lossless source file size: large vs. medium the large images ranged in size from 110 to 130 mb. the medium sized images ranged from 50 to 60 mb. the kakadu compression rate used was 3, which approximately results in an 8:1 ratio. the first hardware setup was a powerful vm in our datacenter to run the transformation script on all 1000 images and push the derivatives to s3 buckets. the second setup made use of aws serverless functions, called lambdas, to distribute the load. the tiffs were pushed to an s3 bucket, which triggered the lambdas to run the transformation and put the derivatives in the destination s3 bucket (the original tiffs were then deleted from the staging s3 bucket). we also ran the horizontal scaling process in a slightly altered scenario in which the concurrency on the source image machine was limited and thus could not push images up to the s3 buckets as quickly. setup vertical scaling horizontal scaling redhat 7 vm in local data center 8 cores (e5-2630 v3) @ 2.40 ghz 8 gb memory ddr4 2133 mhz compiled version of kakadu aws lambda 2 core lambda function 1024 mb memory compiled version of kakadu process vertical scaling horizontal scaling run script for tiff to jp2k conversion read tiffs from netapp file system in the local data center have kakadu convert them into jp2ks upload the jp2ks to cantaloupe’s s3 source bucket measure time to process 1000 images using unix time function upload tiffs into an s3 bucket from the local netapp file system lambda function is triggered by the bucket event kakadu in lambda function converts tiff into jp2k lambda function stores jp2k in cantaloupe’s s3 bucket measure time to process 1000 images using cloudwatch [7] data it should be noted that our intention was to find the total time needed to perform the transformations, not to find bottlenecks within each process. however, we did measure the individual parts of each process. for instance, in the local conversion, kakadu conversion was one measurement and upload to s3 was another; for the lambdas setup, upload time of the tiffs was one measurement and lambda processing was another (which included s3 upload because of the way our lambda was written). we expected upload of tiffs to be what was the time consuming part of the lambda processing and the processing of the tiffs to be what was the time consuming part of the local process — both of which proved true. we did a little tweaking of the bottlenecks (giving more cpu cores to the local processing and re-writing the upload script to take full advantage of threaded uploads), but that wasn’t really our focus. we just wanted the best performance for each method within reason. for instance, we didn’t allocate 20 cpu cores to local processing, which of course would have sped up the kakadu processing. we stayed within the realm of what we’d normally do for a local vm. results below are several charts comparing the performance of the local (vertically scaled) setup and the serverless lambda (horizontally scaled) setup. our results show that the lambdas perform 4 to 5 times faster than the large local machine even with the time it took to upload the tiffs. figure 1. local vs. lambda: lossy medium image run times figure 2. local vs. lambda: lossless medium image run times figure 3. local vs. lambda: lossy large image run times figure 4. local vs. lambda: lossless large image run times we also ran the tests on the lambda setup with limited concurrency on our end for the push of the tiffs from the local data center to the s3 bucket. in these tests the local setup outperformed the cloud. thus, the nuance here is that using the lambdas is faster, but only if you can feed them fast enough. our first configuration usually spun up 10 to 15 individual lambdas. the aws cap was 1000 lambdas. so our highest throughput setup could have gone even faster if we had shoveled content into the s3 buckets quicker. figure 5. local vs. lambda: lossy medium image run times (limited concurrency) figure 6. local vs. lambda: lossy large image run times (limited concurrency) iv. experiment #2 – image delivery in this experiment we measured response times for image requests to cantaloupe with variations along several dimensions: hardware/scaling: local (vertical) vs. cloud (horizontal) (3x and 10x) source image format & quality: tiff vs. lossless jp2k vs.lossy jp2k source image size: large vs. medium requested image size: 50% vs. full traffic: single request vs. concurrent users again, the large images ranged in size from 110 to 130 mb. the medium sized images ranged from 50 to 60 mb. the first hardware setup made use of 2 powerful vms in our data center running our docker container with cantaloupe inside. the second setup made use of aws fargate [8] to deploy 3 of the cantaloupe containers. finally, we tested fargate again, but with 10 containers instead. for the concurrent user experiments, we used the load testing tool locust [9] to simulate traffic patterns of 20, 50, 100, and 200 concurrent users. the users made requests from a list of approximately 1000 urls picked at random from the collection. each user waited between 5 and 15 seconds between their own requests. the locust load test was left running for 5 minutes for each round of testing. it should be noted that all requests were made with no caching enabled. that is, all requests required a read of the source tiff or jp2k and on-the-fly generation of the derivative jpeg. several of the collections in ucla’s digital collections are hosted on behalf of other entities across the globe. we have some projects hosted in other regions to improve performance for the primary stakeholders. therefore, to test local response times to content in regions beyond the west coast, we also decided to test traffic going to a cantaloupe server in two other regions: oregon and germany. setup local fargate 3x fargate 10x 2 redhat 7 vms 1 container per vm 3 fargate containers 10 fargate containers cantaloupe 4.1.1 container specs 8gb memory 6 cores cantaloupe 4.1.1 container specs 8gb memory 4 cores (fargate max) cantaloupe 4.1.1 container specs 8gb memory 4 cores (fargate max) total 16gb memory 12 cpu cores total 24gb memory 12 cpu cores total 80gb memory 40 cpu cores results below are several charts comparing the performance of the local setup and the two fargate setups. round 1 – single user requests in the single image tests we found that the local setup almost always outperformed the fargate setup. one surprising discovery was that responses for full size jpegs derived from tiffs were faster than for images derived from full size jp2ks no matter the setup. kakadu transformations are fast, but reproducing the entire image at full scale is not its strength. we suspect that the limiting factor in the response times is network bandwidth to retrieve the source images, not the computing resources to process them. during these trials we noticed that the speed for an s3 get request appears to be a gigabit at the most. the usual speed was 30-50mb/s whereas on our local netapp storage we saw an average of approximately 400mb/s. this made a big impact on the requests for larger images, but not for the medium images. the wait times for these large file retrievals may be bogging down the containers’ resources. thus, the 10x fargate setup performed better than the 3x setup as it was able to distribute the load across more resources. figure 7. large image 50% request figure 8. large image full request figure 9. medium image 50% request figure 10. medium image full request round 2 – multi-region single user requests in this round we performed the same tests as before, but sent requests from servers in oregon and germany. once again, we found that full size tiff requests were faster than full size jp2k requests. in general, the local environment has smaller response times than the fargate setup but the distinction was negligible, especially for lossy jp2ks. the distinction between traffic from oregon and traffic from germany was less noticeable on the fargate setup than the local setup. figure 11. large image 50% request – oregon/germany figure 12. large image full request – oregon/germany figure 13. medium image 50% request – oregon/germany figure 14. medium image full request – oregon/germany round 3 – concurrent user requests in general, we found that the scaled out fargate setup (10x) performed about the same as the local vmware setup. the vmware had a very slight advantage with lossless jp2k requests and fargate (10x) had a very slight advantage with lossy jp2k requests. figure 15. randomized – large lossless jp2 results (1k+ urls / requests) figure 16. randomized – large lossy jp2 results (1k+ urls / requests) figure 17. randomized – medium lossless jp2 results (1k+ urls / requests) figure 18. randomized – medium lossy jp2 results (1k+ urls / requests) figure 19. randomized – large tiff results (1k+ urls / requests) figure 20. randomized – medium tiff results (1k+ urls / requests) v. conclusions other work caching. we tried out a few forms of caching in both our local and cloud setups of the image api service. in the local system, we configured cantaloupe to store and deliver cached objects on the local filesystem by configuring filesystemcache. in fargate, we configured cantaloupe to store and deliver our cached objects from an s3 bucket by configuring s3cache. fargate gave us some flexibility to also put the cached images in cloudfront, the aws content delivery network (cdn). however, we found that cloudfront could not serve as a permanent cache store as there are no guarantees on how long the cdn keeps a cached object alive. cloudfront also serves as a proxy to the backend. the connection between cloudfront and the backend has a configurable timeout limit of up to 60 seconds. this timeout proved to be an issue for our microservices requirements as we needed a larger connection timeout threshold. as a result, we dropped cloudfront and implemented cantaloupe’s built-in s3cache. in the future, we may address our batch timeout requirements and look into reimplementing cloudfront with s3cache to improve performance worldwide. the cached objects stored in s3cache do not expire and are manually purged as needed. as of now, this setup works fine for us as we do not have much volatility in the content. getty. while we were running these tests we met with our colleague stefano cossu, who works across the highway at the getty. he was also running a series of experiments to determine the design of the getty’s new iiif infrastructure. we learned from him that he found pyramidal tiffs to be the best performing format, and he found iip to be the best performing image server. (cossu 2019) so it appears we bet on the wrong horses with our quick tooling decisions. final decisions after conducting the tests described above we made the following decisions. format: lossy jp2k. this decision was influenced by administrative policymakers, who did not wish savvy users to be able to reconstruct high resolution images from the tiles for certain collections that aren’t fully open access. the lossy jp2k enables us to give our users high quality images without enabling reconstruction of an image that is very near to the original lossless format. derivative generation: use lambdas. this was the easiest choice for us to make. the format choice above determined the desire to use jp2ks, and the results from test #1 clearly showed that using lambdas was superior for transforming large batches of tiffs to jp2ks. this fit our needs for the upcoming mass migration, but would not be the best choice for smaller scale migrations or systems with only a moderate influx of new content at any given time. image delivery: use fargate (for now). the results of test #2 were less compelling. the local setup performed about as well as the fargate 10x setup. however, our local setup was configured with docker compose, a tool that our team was not equipped to support at a production level. therefore, we opted to stick with the fargate setup in the near term. figure 21. architecture diagram outcomes decoupling. in the time since the experiments described in this paper were conducted the team has successfully deployed production instances of the three services in our iiif suite. these include the image transformation and storage service, named bucketeer [10], the manifest generation and presentation api service, named fester [11], and the image api, which is simply our deployment of cantaloupe. this work, along with changes made to the frontend by the applications team, has enabled a complete decoupling of the frontend of the digital library from the backend samvera and fedora stack. the public interface (an instance of blacklight) talks to its own instance of solr and the embedded image viewer requests manifests and images from the new iiif services. this decoupling will enable an easier evolution of our system beyond the current fedora/hyrax/blacklight setup. migration. our primary goal over the next year is to migrate content out of our legacy digital library platforms. therefore our services are designed for uploading csv exports out of those old systems. upon completion of the migration we will shift focus to new content ingest and handling one-off requests to bucketeer and fester directly from the backend repository rather than bulk requests. in the meantime, the new services perform well and have enabled us to remove several steps performed by hyrax during the ingest process. this was a crucial development, as we have experienced unacceptably long ingest times for bulk ingest into hyrax. future work kubernetes. while our experience with fargate was a good one, we are cautious about vendor lockin. we have therefore begun reimplementing our orchestration of the containers for the suite of services using kubernetes. we will then compare the performance of the system with a local kubernetes cluster against a cluster in amazon’s elastic kubernetes service (eks). new iiif services. now that we have deployed the iiif suite into production we will begin work on the next phase of our iiif services, namely auth and audio/visual support. several collections that will be migrated have restricted access policies due to copyright or sensitivity of the material. we therefore need to enable restricted access to our iiif resources as outlined in the iiif auth specification. we also have several audio and video collections that will be migrated out of the legacy systems. furthermore, as of october 2019 the ucla film & television archive is now part of the library. the archive is vast and demand for the content is high. we have therefore already begun updating fester to support version 3.0 of the iiif presentation api, which accommodates time-based media in manifests. we will also explore options for our a/v counterpart to cantaloupe. we currently use wowza [12] for streaming media, but we will continue with our spirit of experimentation and perform some analysis to determine if it is the best fit for our needs. centralized search index. it is time for us to design and build a new library website. the current site is getting on in years, we now have the film & television archive joining us, and the current cms (drupal 7) will be end of life at the end of 2021. one potential goal of the new site is to create a single search interface for all library resources. to support this, the services team will begin work on an aggregation service and central index. metadata outside samvera. in an effort to continue removing responsibilities from the backend repository, we will explore using archivespace [13] for item level metadata instead of using hyrax. this metadata would then be harvested by the central search index mentioned above. vi. acknowledgements the authors would like to thank the following people: hardy pottinger, a former developer on the services team, for his contributions to the services described here bill hackenberg, our project manager, for his help in keeping the project going lisa mcaualy and dawn childress, from ucla’s digital library program and our primary stakeholders, for their help in defining, prioritizing, and testing the service features. vii. references cossu s. 2019. getty common image service: research & design report [internet]. [modified 2019-06-27]. [cited 2020-03-09]. available from: https://drive.google.com/file/d/1pb2eqlslc4ua5zreeedjtbofl0wddapg/view?usp=sharing ford n, parsons r, kua p. 2017. building evolutionary architectures: support constant change. sebastopol: o’reilly media. gomez j. 2020. modernizing a software development team [internet]. [modified 2020-05-02]. [cited 2020-05-02]. available from https://misc.dev/posts/modernizing-a-dev-team/ viii. notes [1] https://samvera.org/ [2] https://iiif.io [3] the image api specifies how to request an individual image, whereas the presentation api specifies how a set of images should be presented together (i.e. sequential order, paginated vs. single images, etc.) [4] https://kakadusoftware.com/ [5] http://www.openjpeg.org/ [6] https://cantaloupe-project.github.io/ [7] cloudwatch is a monitoring service offered by aws [8] https://aws.amazon.com/fargate/ [9] https://locust.io/ [10] https://github.com/uclalibrary/bucketeer [11] https://github.com/uclalibrary/fester [12] https://www.wowza.com/ [13] https://archivesspace.org/ ix. about the authors joshua gomez (joshuagomez@library.ucla.edu) is the head of software development and library systems at the ucla library and a lecturer in the department of information studies at ucla. he holds a masters of information management & systems from uc berkeley and a ba in english from usc (ucla’s cross-town rival). kevin s. clarke (ksclarke@library.ucla.edu) is a digital library software developer at the ucla library. he holds an ms in information science from unc chapel hill and a ba in english from guilford college. anthony vuong (avuong@library.ucla.edu) is the development support systems architect at the ucla library. he studied computer science at uc irvine. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – editorial introduction: conscious resolutions mission editorial committee process and structure code4lib issue 23, 2014-01-17 editorial introduction: conscious resolutions hack your life with 10 new year’s resolutions from code4lib journal. by shawn averkamp coordinating editor, issue 23 what are your new year’s resolutions? to start blogging? contribute code? learn how to version control? it’s that time of year when we reflect on the past and resolve to improve. this issue features 10 actions you can take today to reboot your practices and perspectives. get in shape: want to get your library website into shape? in “the road to responsive: university of toronto libraries’ journey to a new library catalogue interface,” lisa gayhart, bilal khalid, and gordon belray offer their regimen for creating a lean and flexible discovery layer. clean up: isn’t it time you cleaned up your metadata? mark phillips, hannah tarver, and stacey frakes show you how to use open refine and google fusion tables to combat metadata clutter with “implementing a collaborative workflow for metadata analysis, quality improvement, and mapping.” in “recipes for enhancing digital collections with linked data,” thomas johnson and karen estlund share this one weird trick for boosting your quality control. using an rdf statement-based approach, you can swiftly normalize values and supercharge your controlled vocabularies. get organized: ivey glendon and melinda baumann tell you how they crafted the ultimate project workflow in “how the wsls-tv news digitization project helped to launch a project management office.” learn how a new project proposal process helped to set priorities and streamline production. be more productive: master your time with a productivity hack from austin dixon. “use of cue sheets in audio digitization” shows how you can efficiently split out audio tracks in just 3 easy steps. make a difference: in “a video digital library to support physicians’ decision-making about autism,” matthew a. griffin, dan albertson, and angela b. barber detail a prototype they built to assist physicians from underserved areas in assessing autistic patients. learn something new: with this issue we bring you back to the basics with two instructional articles. “unix commands and batch processing for the reluctant librarian or archivist” by anthony cocciolo provides a gentle introduction to performing common digital library-related tasks at the command line. in “automated processing of massive audio/video content using ffmpeg,” kia siang hock and li lingxia teach you how to manipulate audio-visual content at the command line with the open-source software ffmpeg. while these topics may feel rudimentary to many seasoned code4libbers, the decision to include them in this issue was, in part, based on conscious resolution. challenge your assumptions: when the question of publishing a how-to article on basic unix commands was proposed, i will admit that, even as a relative newbie to unix who would benefit from such an article, my knee-jerk reaction was to dismiss the topic as too elementary for this audience. but when i turned around to question those assumptions, i was pleased to see other committee members already questioning them for me. it can be easy to forget that the immense body of knowledge contained in the journal, on the code4lib listserv, and at the code4lib conferences represents the collective contributions of many, and it’s tempting to imagine the whole of that knowledge embodied within each of our peers. the reality–that we are each both beginner and expert alike–can be difficult to parse when so much of our interaction happens between our online identities. it can be even harder to gauge the composition of our readership when many of our peers aren’t raising their voices in these venues at all. despite our best intentions to be inclusive, it is often the most vocal participants who shape our perceptions of a community. foster community: one of the great things about the code4lib community, and perhaps what makes it more appreciative of diverse views than other tech communities, is the difference in the paths we all followed to get here and the respect for the range of skills and perspectives each of us picked up along the way. within the editorial committee alone, we have taught english abroad, product planned, crunched numbers, tuned tiny violins, raised children, traveled the world, and composed electronic sonatas. in choosing which proposals make it to publication, the editorial committee is usually its own sounding board. the range of ideas you see in the code4lib journal reflects vetting by a unique blend of histories, personalities, interests, and expertise. i am proud to have the opportunity to work with a group where this diversity of backgrounds sparks lively and challenging discussions on content, scope, and editorial process, where the response “well, this is how it’s always been done” is rarely, if ever, the final word. while we may not be a completely representative sample of the audience we serve, our differences inspire a willingness to challenge our processes and scope in order to support a diverse readership. be a mentor: learning new skills is all fine and good, but why not share some of your own? sure, some of us have been hacking on computers since age 13, but many of us haven’t, and we all still have much to contribute to the code4lib community. make a resolution this year to submit a proposal to the code4lib journal. meet new people: finally, with the new year, we also welcome new editors. heidi elaine dowding, bill dueber, dan scott, and jason thomale join us to add their unique perspectives and fresh ideas to the editorial committee mix. we look forward to having them on board as we embark on another year of learning, evolving, and improving. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – editorial: the economics of not being an organization mission editorial committee process and structure code4lib issue 38, 2017-10-18 editorial: the economics of not being an organization our successes have caught up with us. now we get to choose the next step in our evolution. by carol bean the code4lib community is at a crossroads. we’re not an organized library organization. we don’t even know how many members there are, because membership is by self-identification. but the loose ties that bind us together are straining under under the dichotomy of acting like an organization without being one. we’re pretty young as library groups go, but we’ve done a lot since arriving on the scene. 14 years ago, code4lib started as a listserv. within 5 years we had annual conferences, a journal with (more or less) quarterly issues, and had set up and migrated a web site. that first 5 years didn’t require a lot from too many people. the code4lib list has done pretty well, with fairly minimal effort required for maintenance. the web site and wiki haven’t required much ongoing effort. the journal did have a rather steep learning curve the first few issues as we hammered out workflows and policies, but works well now with minimal tweaking. all of these components, of course, have drawn directly on the skills of code4lib members. we know technology. we try out new stuff without thinking twice, share those ideas and tweak them till they work for us. the evolution of the code4lib conference, however, has brought us face to face with an important reality which required much more than the technology space we’re so comfortable in. its success has thrown us into the real world of obligations, commitments, and money. in actuality, all the components of code4lib are essentially standalone units: the email list, slack and irc channels, the journal, regional groups, the conferences. the thing is, any of those components could cease, and the group collectively known as code4lib would continue to exist. and this is a good thing! the question is, how much do we want those components to continue, and what will the economic cost be? because everything has an economic cost, and things continue to exist when we, as a collective, decide the benefits outweigh the costs. for the email list, the benefits are huge and the cost is minimal. likewise for the irc and slack channels. the economic cost of the journal is much higher, involving quite a bit of work up front for new editors and fairly significant time commitment for existing editors. but so far we deem the benefits worth the cost. the economic cost of the annual conference, however, has ballooned from moderate to staggering. yet the conferences have continued, because the benefits of the conference, until now, have been worth the cost. last year a fiscal continuity interest group formed, as usual with members of the group self-selecting from the larger code4lib community, to take a deeper look at the economics of running an annual conference when there is no formal organization to support it (see their report and recommendations: https://wiki.code4lib.org/fcig_report). three options emerged, which the code4lib community is now being asked to vote on [1]: do nothing/maintain status quo: local conference planning committees bear burden of finding and recruiting a local “fiscal host” for conference each year secure an ongoing fiscal sponsorship for c4l: pursue one of the offers from possible fiscal sponsors outlined in the fcig report incorporate c4l as a nonprofit entity option 1 is unfeasible, even irresponsible, and could result in the end of code4lib annual conferences. option 2 and option 3 are not mutually exclusive. i would point out, however, at this point the amount of work required to implement option 3 far exceeds that for option 2. and we need to do something now if we want the annual conferences to continue. given the time factor, the only path that makes sense is to secure an ongoing sponsor with the resources we as a group lack. we need their expertise and experience now, whether or not the third option is pursued in the long run. and i encourage those interested in that path to take the time and make the effort to pursue option 3, after we have secured a sponsor. let us continue to evolve! meanwhile, if you consider yourself part of the code4lib community, or if you even aspire to be part, take a few minutes (or more) to read about the options, and vote: https://www.surveymonkey.com/r/k5mwgnc. we’re that type of community. notes [1] taken from the surveymonkey ballot: https://www.surveymonkey.com/r/k5mwgnc. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – metadata analysis at the command-line mission editorial committee process and structure code4lib issue 19, 2013-01-15 metadata analysis at the command-line over the past few years the university of north texas libraries’ digital projects unit (dpu) has developed a set of metadata analysis tools, processes, and methodologies aimed at helping to focus limited quality control resources on the areas of the collection where they might have the most benefit. the key to this work lies in its simplicity: records harvested from oai-pmh-enabled digital repositories are transformed into a format that makes them easily parsable using traditional unix/linux-based command-line tools. this article describes the overall methodology, introduces two simple open-source tools developed to help with the aforementioned harvesting and breaking, and provides example commands to demonstrate some common metadata analysis requests. all software tools described in the article are available with an open-source license via the author’s github account. by mark phillips introduction the unt libraries’ digital libraries division is responsible for the creation and quality review of the majority of metadata records in the unt libraries’ digital collections. these collections contain items of similar format to other university library collections of comparable size. items in the collections include digitized and born-digital photographs, letters, documents, maps, ledgers, technical reports, and theses and dissertations. the size and scope of these collections continue to grow at an increasing rate for the past three years measuring 83,000, 93,000, and 120,000 items added per year for the fiscal years 2010, 2011, and 2012. the continued growth in these collections means that there are a greater number of metadata records created by an increasing number of metadata creators, which in turn causes a wider variance in quality. the need to analyze and report statistics for these metadata records has lead the unt libraries to develop new tools and processes to ensure that high quality metadata records are used throughout its digital library collections. this article describes an approach in use at the unt libraries for harvesting metadata records from an oai-pmh repository and then transforming them into a simpler text format, which can easily be consumed by a number of standard command-line tools available freely on most unix and linux based systems. metadata quality, as defined for this article, falls into three major areas. collection level analytics – how many of something are in the entire collection of metadata records? for example, how many unique creators are represented by the collection? which creator is associated with the most items? what item in the collection has the most creator, subject or coverage instances? these analytics are helpful in communicating metrics about the collection to others. metadata completeness – how well does the collection’s item-level metadata conform to various measures of completeness? what are required fields for a given subset of metadata, and how well do the collection’s metadata records adhere to these requirements? how does this collection of metadata meet both metadata creator and metadata consumer ideals of value? metadata value quality – based on local requirements, how well do the values within a given metadata record match those of standard or defined metadata specifications? for example, if a collection of metadata records utilizes the extended date time format for the date values in the collection, how well does the metadata collection meet the requirement of that format? which values need to be changed in order to meet more of the requirements? the tools and methodology explained in this article provide a way of identifying these areas of metadata cleanup to focus our attention, and help answer the question, “if there is a limited amount of time to spend on metadata cleanup, what is the best use of this time?” getting the data to the right tools the command line in the unix/linux environment offers a number of tools which when used in different combinations provide a useful pipeline for manipulating data. the workflow described below makes use of many standard unix/linux tools: sort – sort each line of a text file uniq – report or omit repeated lines wc – count the number of words or lines awk – pattern-directed scanning and processing language grep – print lines matching a supplied pattern these tools work by having the output (standard output or stdout) of one tool act as the input (standard input or stdin) of the next tool, and chaining these tools together by directing the output and input in a series of processes often referred to as a pipeline. the tools generally operate on simple text formats. the most challenging aspect of working with these tools in a metadata analysis workflow is converting the metadata from a metadata repository into a format that can be easily consumed by the tools. many repositories have adopted the open archives initiatives protocol for metadata harvesting (oai-pmh) [1] as a way to share metadata with others. at this time there are over 1,850 repositories worldwide that make metadata available with this protocol. the methodology described in this article makes use of oai-pmh as the way to retrieve records from a repository. a concise view of the methodology used by the unt libraries is to provide a straightforward workflow for harvesting records with a simple oai-pmh harvester into an xml file referred to as a “repository file,” then using another small tool to convert this repository into a text format that is easily consumed by common command-line tools. once metadata records have been collected the next step is to convert from the format presented by the oai-pmh repository into a format that is usable by the previously mentioned command-line tools. an example of the difference in representation can be outlined in the following two examples: <record> <header> <identifier>info:ark/67531/metadc97952</identifier> <datestamp>2012-08-17t12:16:00z</datestamp> <setspec>partner:untcas</setspec> <setspec>collection:untsw</setspec> <setspec>access_rights:public</setspec> </header> <metadata> <oai_dc:dc xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:oai_dc="http://www.openarchives.org/oai/2.0/oai_dc/" xsi:schemalocation="http://purl.org/dc/elements/1.1/ http://www.openarchives.org/oai/2.0/oai_dc.xsd"> <dc:title>fenología de tayloria dubyi (splachnaceae) en las turberas de la reserva de biosfera cabo de hornos</dc:title> <dc:title>phenology of tayloria dubyi (splachnaceae) in the peatlands of the cape horn biosphere reserve</dc:title> <dc:title>sub-antarctic biocultural conservation program</dc:title> <dc:creator>jofre, jocelyn</dc:creator> <dc:creator>massardo, francisca</dc:creator> <dc:creator>rozzi, ricardo</dc:creator> <dc:creator>goffinet, bernard</dc:creator> <dc:creator>marino, paul</dc:creator> <dc:creator>raguso, robert</dc:creator> <dc:creator>navarro, nelso p.</dc:creator> <dc:subject>bryophytes</dc:subject> <dc:subject>cape horn biosphere reserve</dc:subject> <dc:subject>phenology reproduction</dc:subject> <dc:subject>splachnaceae</dc:subject> <dc:subject>sub-antarctic magellanic ecoregion</dc:subject> <dc:description>this article discusses the phenology of tayloria dubyi (splachnaceae) in the peatlands of the cape horn biosphere reserve. the sub-antarctic magellanic ecoregion harbors a high diversity of bryophytes, greater than the species richness of vascular plants. despite this fact, phenological studies on bryophytes are lacking for this ecoregion and chile. based on the study of the sporophytic phase of tayloria dubyi, an endemic moss from the sub-antarctic magellanic ecoregion, the authors propose a methodology for phonological studies on austral bryophytes. the authors defined five phenophases, easily distinguishable with a hand-lens, which were monthly recorded during 2007 and 2008 in populations of t. dubyi at the omora ethnobotanical park and mejillones bay on navarino island (55°s) in the cape horn biosphere reserve. the sporophytic (or reproductive) phase of t. dubyi presented a clear seasonality. after growing in november, in three months (december february) of the austral reproductive season the sporophytes mature and release their spores; by march they are already senescent t. dubyi belongs to the splachnaceae family for which entomochory (dispersal of spores by insects, specifically diptera) has been detected in the northern hemisphere. the period of spores release in t. dubyi coincides with the months of highest activity of diptera which are potential dispersers of spores; hence, entomochory could also take place in sub-antarctic magellanic ecoregion. in sum, the authors' work: (i) defines a methodology for phenological studies in austral bryophytes, (ii) it records a marked seasonality ion the sporophyte phase of t. dubyi, and (iii) it proposes to evaluate in future research the occurrence of entomochory in splachnaceae species growing in the sub-antarctic peatlands and forest ecosystems in the southern hemisphere.</dc:description> <dc:publisher>sociedad de biología de chile</dc:publisher> <dc:date>2010</dc:date> <dc:type>article</dc:type> <dc:format>12 p.</dc:format> <dc:format>text</dc:format> <dc:identifier>http://digital.library.unt.edu/ark:/67531/metadc97952/</dc:identifier> <dc:identifier>ark: ark:/67531/metadc97952</dc:identifier> <dc:source>revista chilena de historia natural, 2010, santiago: sociedad de biología de chile, pp. 195-206</dc:source> <dc:language>spanish</dc:language> <dc:rights>public</dc:rights> </oai_dc:dc> </metadata> </record> info:ark/67531/metadc97952 {http://purl.org/dc/elements/1.1/}title fenología de tayloria dubyi (splachnaceae) en las turberas de la reserva de biosfera cabo de hornos info:ark/67531/metadc97952 {http://purl.org/dc/elements/1.1/}title phenology of tayloria dubyi (splachnaceae) in the peatlands of the cape horn biosphere reserve info:ark/67531/metadc97952 {http://purl.org/dc/elements/1.1/}title sub-antarctic biocultural conservation program info:ark/67531/metadc97952 {http://purl.org/dc/elements/1.1/}creator jofre, jocelyn info:ark/67531/metadc97952 {http://purl.org/dc/elements/1.1/}creator massardo, francisca info:ark/67531/metadc97952 {http://purl.org/dc/elements/1.1/}creator rozzi, ricardo info:ark/67531/metadc97952 {http://purl.org/dc/elements/1.1/}creator goffinet, bernard info:ark/67531/metadc97952 {http://purl.org/dc/elements/1.1/}creator marino, paul info:ark/67531/metadc97952 {http://purl.org/dc/elements/1.1/}creator raguso, robert info:ark/67531/metadc97952 {http://purl.org/dc/elements/1.1/}creator navarro, nelso p. info:ark/67531/metadc97952 {http://purl.org/dc/elements/1.1/}subject bryophytes info:ark/67531/metadc97952 {http://purl.org/dc/elements/1.1/}subject cape horn biosphere reserve info:ark/67531/metadc97952 {http://purl.org/dc/elements/1.1/}subject phenology reproduction info:ark/67531/metadc97952 {http://purl.org/dc/elements/1.1/}subject splachnaceae info:ark/67531/metadc97952 {http://purl.org/dc/elements/1.1/}subject sub-antarctic magellanic ecoregion info:ark/67531/metadc97952 {http://purl.org/dc/elements/1.1/}publisher sociedad de biología de chile info:ark/67531/metadc97952 {http://purl.org/dc/elements/1.1/}date 2010 info:ark/67531/metadc97952 {http://purl.org/dc/elements/1.1/}type article info:ark/67531/metadc97952 {http://purl.org/dc/elements/1.1/}format 12 p. info:ark/67531/metadc97952 {http://purl.org/dc/elements/1.1/}format text info:ark/67531/metadc97952 {http://purl.org/dc/elements/1.1/}identifier http://digital.library.unt.edu/ark:/67531/metadc97952/ info:ark/67531/metadc97952 {http://purl.org/dc/elements/1.1/}identifier ark: ark:/67531/metadc97952 info:ark/67531/metadc97952 {http://purl.org/dc/elements/1.1/}source revista chilena de historia natural, 2010, santiago: sociedad de biología de chile, pp. 195-206 info:ark/67531/metadc97952 {http://purl.org/dc/elements/1.1/}language spanish info:ark/67531/metadc97952 {http://purl.org/dc/elements/1.1/}rights public info:ark/67531/metadc97952 {http://purl.org/dc/elements/1.1/}description this article discusses the phenology of tayloria dubyi (splachnaceae) in the peatlands of the cape horn biosphere reserve. the sub-antarctic magellanic ecoregion harbors a high diversity of bryophytes, greater than the species richness of vascular plants. despite this fact, phenological studies on bryophytes are lacking for this ecoregion and chile. based on the study of the sporophytic phase of tayloria dubyi, an endemic moss from the sub-antarctic magellanic ecoregion, the authors propose a methodology for phonological studies on austral bryophytes. the authors defined five phenophases, easily distinguishable with a hand-lens, which were monthly recorded during 2007 and 2008 in populations of t. dubyi at the omora ethnobotanical park and mejillones bay on navarino island (55°s) in the cape horn biosphere reserve. the sporophytic (or reproductive) phase of t. dubyi presented a clear seasonality. after growing in november, in three months (december february) of the austral reproductive season the sporophytes mature and release their spores; by march they are already senescent t. dubyi belongs to the splachnaceae family for which entomochory (dispersal of spores by insects, specifically diptera) has been detected in the northern hemisphere. the period of spores release in t. dubyi coincides with the months of highest activity of diptera which are potential dispersers of spores; hence, entomochory could also take place in sub-antarctic magellanic ecoregion. in sum, the authors' work: (i) defines a methodology for phenological studies in austral bryophytes, (ii) it records a marked seasonality ion the sporophyte phase of t. dubyi, and (iii) it proposes to evaluate in future research the occurrence of entomochory in splachnaceae species growing in the sub-antarctic peatlands and forest ecosystems in the southern hemisphere. the first example displays a common oai_dc record produced by an oai-pmh repository. the second example shows the way that many of the command line tools expect to receive text: flattened and consistently delimited. by utilizing oai-pmh for this workflow, the concepts, methods, and tools are more portable and work with any repository that supports this protocol. this approach can therefore be more broadly applied than a workflow tightly integrated into an individual platform’s infrastructure. the tools developed for this workflow are described below with examples of common metadata analysis operations following the tool description. introduction to the tools oai-pmh harvester the first tool is an oai-pmh harvester written in python and based significantly on the two page oai harvester from oclc research [2]. the harvester, while simple, offers a set of options that cover many standard use cases for harvesting metadata records from an oai-pmh repository. the harvester takes as input the url for an oai-pmh repository and the name of the output file for storing the results of the harvest. the output of the oai-pmh response is stored as a single xml file enclosed in a set of <repository> tags. mphillips$ python pyoaiharvest.py -l http://digital.library.unt.edu/explore/collections/untsw/oai/ -o untsw.dc.xml the above command will harvest all records from the oai-pmh repository available at the url http://digital.library.unt.edu/explore/collections/untsw/oai/ and save them as a file named untsw.dc.xml. the default metadata format of oai_dc is requested from the repository and transmitted back to the harvester. for a full list of command line options see the help screen for the script: mphillips$ python pyoaiharvest.py -h usage: pyoaiharvest.py [options] options: -h, --help show this help message and exit -l link, --link=link url of repository -o filename, --filename=filename write repository to file -f fromdate, --from=fromdate harvest records from this date yyyy-mm-dd -u until, --until=until harvest records until this date yyyy-mm-dd -m mdprefix, --mdprefix=mdprefix use the specified metadata format -s setname, --setname=setname harvest the specified set the tool supports requesting a specific setspec, a different metadata format, or limiting to a date range for updating collections of existing records. repository breakers once a set of metadata records have been harvested, the next step of processing metadata records is converting them into a text format that can act as input to common command line tools. the tool dc_breaker.py is used for this function. this tool efficiently consumes the output format from the pyoaiharvester.py script as input and provides a set of options for converting this into formats easily used by other command-line tools. mphillips$ python dc_breaker.py untsw.dc.xml 1000 records processed {http://purl.org/dc/elements/1.1/}contributor: |============== | 1032/1835 | 56% {http://purl.org/dc/elements/1.1/}coverage: |== | 172/1835 | 9% {http://purl.org/dc/elements/1.1/}creator: |======================== | 1834/1835 | 99% {http://purl.org/dc/elements/1.1/}date: |======================== | 1807/1835 | 98% {http://purl.org/dc/elements/1.1/}description: |======================== | 1832/1835 | 99% {http://purl.org/dc/elements/1.1/}format: |=========================| 1835/1835 | 100% {http://purl.org/dc/elements/1.1/}identifier: |=========================| 1835/1835 | 100% {http://purl.org/dc/elements/1.1/}language: |=========================| 1835/1835 | 100% {http://purl.org/dc/elements/1.1/}publisher: |========= | 681/1835 | 37% {http://purl.org/dc/elements/1.1/}relation: |==== | 351/1835 | 19% {http://purl.org/dc/elements/1.1/}rights: |====================== | 1671/1835 | 91% {http://purl.org/dc/elements/1.1/}source: |=================== | 1426/1835 | 77% {http://purl.org/dc/elements/1.1/}subject: |=========================| 1835/1835 | 100% {http://purl.org/dc/elements/1.1/}title: |=========================| 1835/1835 | 100% {http://purl.org/dc/elements/1.1/}type: |=========================| 1835/1835 | 100% dc_completeness 79.258856 collection_completeness 84.250681 wwww_completeness 99.604905 average_completeness 87.704814 the example above runs the tool without any options selected. this will generate an output that can be helpful for seeing the overall utilization of fields within a collection of metadata records. it shows the fifteen elements of the oai_dc metadata scheme, as well as a visualization of the percentage of records in the repository file which have at least one value in this element. next, the output presents a column showing the number of records in the repository that contain this field related to the total number of records in the collection and, finally, a percentage of utilization of this field in the collection. next, the tool provides a set of completeness scores, which give another type of overview of the collection. the first score is a percentage of completeness assuming that all fifteen of the oai_dc elements are required, the second is a measure of completeness as a collection. this percentage takes into account the fields used within the collection and generates a percentage of completeness based on those fields being present. for example, if a collection of metadata records uses the fields title, creator, description, and data exclusively, only these fields are used in the calculation of collection completeness. finally, the percentage of completeness is based on the recommendation by the kernel metadata and electronic resource citations (ercs) [3] community, which state that the who, what, where, when of an item are required for adequate access and citation of an item. in this case who is mapped to creator, what is mapped to title, where is mapped to identifier and when is mapped to date. in addition to these completeness values there is an average of the three scores presented. these completeness measures are useful as an overview of collection level metadata and in showing improvement resulting from metadata cleanup activities. other sample uses of this tool are printing to standard output each value of a specific element either by itself or in the format of identifier <tab> value. mphillips$ python dc_breaker.py -e creator untsw.dc.xml tummala, dinesh li, xinrong nguyen, son akl, robert g. garlick, ryan akl, robert g. li, wenming kavi, krishna akl, robert g. alhabsi, amer h. al-rizzo, hussain m. akl, robert g. akl, robert g. parvez, asad nguyen, son akl, robert g. naraghi-pour, mort hegde, manju haidar, mohamad akl, robert g. al-rizzo, hussain chan, yupo ... this example shows each instance of the creator element in the repository with one instance per line. mphillips$ python dc_breaker.py -e creator -i untsw.dc.xml info:ark/67531/metadc30827 tummala, dinesh info:ark/67531/metadc30827 li, xinrong info:ark/67531/metadc30820 nguyen, son info:ark/67531/metadc30820 akl, robert g. info:ark/67531/metadc30828 garlick, ryan info:ark/67531/metadc30828 akl, robert g. info:ark/67531/metadc30824 li, wenming info:ark/67531/metadc30824 kavi, krishna info:ark/67531/metadc30824 akl, robert g. info:ark/67531/metadc30829 alhabsi, amer h. info:ark/67531/metadc30829 al-rizzo, hussain m. info:ark/67531/metadc30829 akl, robert g. info:ark/67531/metadc30826 akl, robert g. info:ark/67531/metadc30826 parvez, asad info:ark/67531/metadc30826 nguyen, son info:ark/67531/metadc30823 akl, robert g. info:ark/67531/metadc30823 naraghi-pour, mort info:ark/67531/metadc30823 hegde, manju info:ark/67531/metadc30835 haidar, mohamad info:ark/67531/metadc30835 akl, robert g. info:ark/67531/metadc30835 al-rizzo, hussain info:ark/67531/metadc30835 chan, yupo ... this is very similar to the previous example but it prepends the records identifier from the oai record identifier to the beginning of each line and separates the identifier from the value by a tab character. when working with metadata record cleanup it is sometimes necessary to identify which records in a set do not have a value. this can be accomplished by adding the –p flag which displays if an instance of the specified element is present in the record. mphillips$ python dc_breaker.py -e creator -p untsw.dc.xml info:ark/67531/metadc27055 true info:ark/67531/metadc27052 true info:ark/67531/metadc27051 true info:ark/67531/metadc27057 true info:ark/67531/metadc27050 true info:ark/67531/metadc27058 true info:ark/67531/metadc27054 true info:ark/67531/metadc27059 true info:ark/67531/metadc27056 true info:ark/67531/metadc27053 true info:ark/67531/metadc27175 true info:ark/67531/metadc27172 true info:ark/67531/metadc27171 true info:ark/67531/metadc27177 true info:ark/67531/metadc111250 false ... this displays the identifier for the record followed by a true if the record has value for that element or false if the record does not have value for the specified element. finally an option to dump all data in the repository file into a tab-delimited format is available by using the flag –d. mphillips$ python dc_breaker.py -e creator -d untsw.dc.xml info:ark/67531/metadc27055 {http://purl.org/dc/elements/1.1/}title [handmade paper and silkscreen on cotton velveteen artwork] info:ark/67531/metadc27055 {http://purl.org/dc/elements/1.1/}creator spear, shigeko info:ark/67531/metadc27055 {http://purl.org/dc/elements/1.1/}subject surface design info:ark/67531/metadc27055 {http://purl.org/dc/elements/1.1/}subject laid paper info:ark/67531/metadc27055 {http://purl.org/dc/elements/1.1/}subject velveteen info:ark/67531/metadc27055 {http://purl.org/dc/elements/1.1/}description this detail view of the artwork shows the geometric shapes that float through the atmospheric landscape. the colors are in the dull pink, muted blues and beige ranges. deep space is depicted with mountainous shapes reaching into water. info:ark/67531/metadc27055 {http://purl.org/dc/elements/1.1/}contributor wolfe, deb info:ark/67531/metadc27055 {http://purl.org/dc/elements/1.1/}date 1987~ info:ark/67531/metadc27055 {http://purl.org/dc/elements/1.1/}type artwork info:ark/67531/metadc27055 {http://purl.org/dc/elements/1.1/}format 1 art original : handmade paper, cotton velveteen info:ark/67531/metadc27055 {http://purl.org/dc/elements/1.1/}format image info:ark/67531/metadc27055 {http://purl.org/dc/elements/1.1/}identifier http://digital.library.unt.edu/ark:/67531/metadc27055/ info:ark/67531/metadc27055 {http://purl.org/dc/elements/1.1/}identifier ark: ark:/67531/metadc27055 info:ark/67531/metadc27055 {http://purl.org/dc/elements/1.1/}language no language info:ark/67531/metadc27052 {http://purl.org/dc/elements/1.1/}title [full moon in front of grids] info:ark/67531/metadc27052 {http://purl.org/dc/elements/1.1/}creator spear, shigeko info:ark/67531/metadc27052 {http://purl.org/dc/elements/1.1/}subject surface design info:ark/67531/metadc27052 {http://purl.org/dc/elements/1.1/}subject silk (textile) info:ark/67531/metadc27052 {http://purl.org/dc/elements/1.1/}subject laid paper ... metadata analysis examples the examples below make use of records harvested from the unt scholarly works and unt theses and dissertation collection hosted by the unt digital library [4]. the oai-pmh repository links are available at the following urls: unt scholarly works – http://digital.library.unt.edu/explore/collections/untsw/oai/ unt theses and dissertations – http://digital.library.unt.edu/explore/collections/untetd/oai/ these metadata collections are saved as untsw.dc.xml and untetd.dc.xml respectively for the purpose of these examples. find most prolific creators in a set of records mphillips$ python dc_breaker.py -e creator untsw.dc.xml | sort | uniq -c | sort -n -r | head 107 mihalcea, rada 93 cundari, thomas r. 90 falsetta, vincent 89 phillips, mark edward 85 spear, shigeko 77 moen, william e. 61 marshall, james l., 1940 59 murray, kathleen 58 marshall, virginia r. 57 eve, susan this pipeline of commands will display back the ten most frequent creators. here is what the pipeline is doing: first the dc_breaker script is used to extract creator elements from each record. the output of all creators is sorted by the sort command. the output of the sort command is then made unique by the unix uniq command which is given the flag to append the number of times that value is present. the resulting set is sorted again this time by the count of each value from highest value to lowest using the –n and –r options. finally the first ten values are returned along with their counts. find the number of unique creators in a dataset mphillips$ python dc_breaker.py -e creator untsw.dc.xml | sort | uniq | wc -l 1592 this example extracts all of the creators in the repository, sorts the values, makes them unique and finally passes these to the wc (word count) utility with the –l option selected to count only lines. this returns the number of unique creators in a dataset, which can answer questions such as “how many creators have content in the unt scholarly works repository?” find the number of subjects per record mphillips$ python dc_breaker.py -e subject -i untsw.dc.xml | cut -f 1 | sort | uniq -c | sort -n -r 24 info:ark/67531/metadc111236 21 info:ark/67531/metadc29400 21 info:ark/67531/metadc111261 20 info:ark/67531/metadc111271 20 info:ark/67531/metadc111269 20 info:ark/67531/metadc111200 17 info:ark/67531/metadc29401 17 info:ark/67531/metadc29393 17 info:ark/67531/metadc111212 17 info:ark/67531/metadc111177 16 info:ark/67531/metadc29399 16 info:ark/67531/metadc111216 16 info:ark/67531/metadc111206 ... this example extracts all subjects in the repository with the identifier appended to the beginning of the line. this identifier is extracted from the line using the cut utility and then sorted, made unique and finally sorted again. the result is a list containing the number of subjects in a record followed by that records identifier. this output is useful when a collection has requirements pertaining to the number of subjects per record such as, “a minimum of three subjects per record, or no more than six subjects per record”. find the average number of subjects per record mphillips$ python dc_breaker.py -e subject -i untetd.dc.xml | cut -f 1 | sort | uniq -c | awk '{sum+=$1} end { print &quot;average = &quot;,sum/nr}' average = 5.79016 this example builds on the previous by adding the use of the awk utility to calculate the average number of subjects per record in the repository. this information is especially useful in analyzing the overall subject usage patterns across various collections. find the records without any creators mphillips$ python dc_breaker.py -e creator -p untsw.dc.xml | grep false info:ark/67531/metadc111250 false the output of this example is a record identifier followed by true or false based on the presence of the specified field, this time creator. after piping this output to grep and searching only for lines that have false in them, we can see which records do not have any creator values. if each record in a collection should have a creator element present, this output makes it easy to find records lacking this value. list creators sorted by creator value mphillips$ python dc_breaker.py -e creator untsw.dc.xml | sort aeco economic and community development class fall, 2009 aars, christian abel, mickey abel, mickey acevedo, mitzi adada, rami adalar, mehmet adams, mark adams, mark ... zhang, cankui zhang, jubo zhang, xue zhang, xue zhang, xue zhang, xue zhang, xue zhao, yong zhi, miaochan zhou, tie zhou, tie zhou, xin zhou, xin zhou, xin zhu, yuntian this example demonstrates a common usage of the dc_breaker tool in which all values of a field are printed to the screen and then sorted. it is useful for identifying slight misspellings between adjacent values. this is particularly helpful with the creator and contributor fields when trying to normalize names values that are similar, which should be replaced with an authoritative version. list creator values sorted by the number of times they occur mphillips$ python dc_breaker.py -e creator untsw.dc.xml | sort | uniq -c 1 aeco economic and community development class fall, 2009 1 aars, christian 2 abel, mickey 1 acevedo, mitzi 1 adada, rami 1 adalar, mehmet 2 adams, mark 1 aerts, andrea ... 2 zeug, nicole m. 1 zhang, cankui 1 zhang, jubo 5 zhang, xue 1 zhao, yong 1 zhi, miaochan 2 zhou, tie 3 zhou, xin 1 zhu, yuntian this example adds the uniq tool to the pipeline from the previous example. the uniq tool with the –c (count) option takes a sorted list (supplied by the sort command) and sums the value instances before returning a row for each unique value preceded by the number of times it is present in the dataset. this shows the number of time each value occurs in the dataset, from which it is possible to derive the relative importance of one value over another. mphillips$ python dc_breaker.py -e contributor untetd.dc.xml | sort | uniq -c 8 mikler, armin 48 mikler, armin r. 1 mikler, susie 1 ramisetty-mikler, suhasini 1 ramisetty-mikler, susie this example shows that one value is most likely incorrect based on the fact that there are so many instances of the other version of the value (“mikler, armin”, 8), (“mikler, armin, r.”, 48). list field values sorted by anagram hashes another variation on the previous models is helpful in certain situations. when comparing names in a dataset you may come across versions of names that are inverted (last, first) as well as natural order (first last). when identifying possible metadata changes based on adjacency in a list, these values would not appear next to each other based on a normal sort order. mark phillips phillips, mark by feeding these values into a simple anagram hash function and then sorting the values based on the hash it is possible to get these values to group together and helpful to identify related problems. aihkmlpsr mark phillips aihkmlpsr phillips, mark import sys import re def anagram_string(string): string = string.lower() string = re.sub("[\w\d]", "", string) return "".join(set(string)) # input comes from stdin for line in sys.stdin: # remove leading and trailing whitespace print "%s\t%s" % (anagram_string(line.strip()), line.strip()) mphillips$ python dc_breaker.py -e contributor untetd.dc.xml | python example-018.txt | sort | uniq -c 1 abeihlsru hariss, beulah 1 abeihlsru harris beulah 1 abeihlsru harris, beulah a. 6 abeihlsru harriss, beulah 1 abeihlsru harriss, beulah a. 1 abeihlsru harriss, beulah a 58 abeihlsru harriss, beulah a. field validation standard format – extended date time format (edtf) these tools can also be helpful in validating a given value in a record based on a list of known values, a standard format, or a well-known feature. the following examples demonstrate the usage of these tools for this purpose: mphillips$ python dc_breaker.py -e date untsw.dc.xml | python valid_edtf.py 1987~ true 1987~ true 1981~ true 1987~ true 1990~ true 1987~ true 1987~ true 1987~ true 1987~ true 1986~ true 2003 true 2002 true 2002 true 2003 true 2001 true 2003 true 2002 true ... a little background is required for this example. the unt libraries are moving toward the adoption of the extended date/time format (edtf) [5] for encoding date information in their digital library. in order to identify values that comply with this specification a simple validator was created that compares a given string against the edtf draft specification. the output of this script is in the following format: 2012 true 2001~ true 2012-12 true 2012-15 false 2012/12/12 false it is then possible to filter the output to just instances that are invalid and investigate those in the dataset as examples of values to change. does the creator field have a comma (are names properly inverted) mphillips$ python dc_breaker.py -e contributor untetd.dc.xml | grep -v &quot;,&quot; william. d. deering jesus rosales-ruiz brian richardson sue bratton may. andrew nann goplerud webb james f. blackburn s. a. harris beulah harris b. b. silvey j. k. g. silvey j. k. g. sharp l. a. [silvey j. k. g.] in many digital library metadata settings names are notated in inverted fashion, separated by a comma. for example a name like “mark phillips” is notated as “phillips, mark” this example shows a quick way to identify values in a repository that are not formatted in this way. it prints the instances of creator names in the metadata records, which do not have commas. (this assumes that the presence of a comma is an indication of correct formatting.) longest title another use for this set of tools is to answer fairly obscure but important questions that arise in the library universe. one question that came up in the creation of the unt digital library was, “what is the length of the longest titles we have in the system?” the user interface designer for the metadata display pages for records needed this. by using these tools it was relatively easy to find this answer. mphillips$ python dc_breaker.py -e title untetd.dc.xml | awk '{ print length($0) &quot; &quot; $0 }' | sort -nr | head 354 greek texts and english translations of the bible: a comparison and contrast of the textus receptus greek new testament of the sixteenth century and the alexandrian text of westcott and hort (nineteenth century) and aland and metzger (twentieth century) concerning variant texts that pertain to the orthodox christology of the council of nicea, a.d. 325. 353 a comparative study of the habits, attitudes, and opinions in regard to cigarette smoking on the part of three hundred freshman and sophomore women students and three hundred freshman and sophomore men students enrolled in physical education activity classes during the spring semester of the 1948-1949 session at north texas state college, denton texas 349 synthesis and characterization of platinum(ii)(2-(9-anthracenylylidene)-4,5-bis(diphenylphosphino)-4-cyclopenten-1,3-dione)(dichloride), platinum(ii)(2-(9-anthracenylylidene)-4,5-bis(diphenylphosphino)-4-cyclopenten-1,3-dione(maleonitriledithiolate), and platinum(ii)(4,5-bis(diphenylphosphino)-4-cyclopenten-1,3-dione)(4-methyl-1,2-benzene dithiol) 336 a performer's analysis of lili boulanger's clairières dans le ciel: song cycle for high voice and piano; a lecture recital together with the role of blanche in dialogues of the carmelites by f. poulenc and two recitals of selected works by h. purcell, f. schubert, s. prokofieff, e. chausson, w. a. mozart, r. schumann and g. fauré 332 jules massenet's musical prosody focusing on his eight song cycles and a collection, expressions lyriques: a lecture recital, together with recitals of selected works of w. a. mozart, f. schubert, c. debussy, r. strauss, d. argento, v. bellini, j. marx, w. walton, c. gounod, a. scarlatti, g. fauré, j. rodrigo, h. wolf, and others 325 the 1986 national endowment for the arts commission: an introspective analysis of two marimba works, reflections on the nature of water by jacob druckman and velocities by joseph schwantner, together with three recitals of selected works by keiko abe, christopher deane, peter klatzow, wayne siegel, gitta steiner and others. 302 the influence of selma meerbaum-eisinger's death on xaver paul thoma's composition of ich bin in sehnsucht eingehüllt: sieben lieder für sopran und klavier, a lecture recital together with three recitals of selected works of o. messiaen, g.f. handel, a. scarlatti, j.s. bach, w. latham, and others 282 supplemental studies for mastering extended techniques in three late twentieth-century works for solo trombone: luciano berio's sequenza v, folke rabe's basta and mark phillips' t. rex, together with three recitals of selected works by wagenseil, grøndahl, gotkovsky, and others 276 gradus ad parnassum of modern flute technique: an explication of musical intention and design in 30 capricen für flöte allein, opus 107 by sigfrid karg-elert, with three recitals of selected works by schulhoff, telemann, berio, bach, rodrigo, gieseking, reinecke, and others 276 a study to determine the characteristics of effective application letters for teachers, with particular reference to college positions, as influenced by reports from 65 college and university presidents and an analysis of 65 sample letters of application for college positions this example uses a combination of awk to append the number of characters in each title to the beginning of each line, numerically sort the lines in reverse order with sort, and finally use the head utility to print the top 10 lines of this new list. conclusion harvesting metadata records from oai-pmh repositories and then transforming these records into simple statements easily consumed by common command-line tools has significantly improved the workflow for identifying problems in metadata record collections and has allowed the unt libraries to utilize metadata enhancement resources in the most beneficial way possible. we have found that maintainers of metadata collections are eager to modify metadata records needing cleanup once identified using the methods described above. by using the methodology described in this article, the unt libraries is able to focus technology development resources on tools that can easily be integrated into this command-line environment. currently in development are tools that help to further identify name collisions in creator, contributor, and publisher fields as well as validators for various fields and values present in the metadata used at the unt libraries. obtaining code mentioned in this article all code mentioned in this article is available freely at the author’s github repository: http://github.com/vphill/. notes [1] open archives initiative – protocol for metadata harvesting (oai-pmh) http://www.openarchives.org/pmh/ [2] 2pageoai – http://www.oclc.org/research/activities/oai2page.html [3] kernel metadata and electronic resource citations (ercs) http://dublincore.org/groups/kernel/spec/ [4] unt digital library – about the technology http://digital.library.unt.edu/about/digital-library/technology/ [5] extended date/time format (edtf) http://www.loc.gov/standards/datetime/ about the author mark phillips is assistant dean for digital libraries at the university of north texas libraries. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – building a location-aware mobile search application with z39.50 and html5 mission editorial committee process and structure code4lib issue 10, 2010-06-22 building a location-aware mobile search application with z39.50 and html5 this paper presents mytpl (http://www.mytpl.ca), a proof-of-concept web application intended to demonstrate that, with a little imagination, any library with a z39.50 catalogue interface and a web server with some common open-source tools can readily provide their own location-aware mobile search application. the complete source code for mytpl is provided under the gnu gplv3 license, and is freely available at: http://github.com/mjsuhonos/mytpl by mj suhonos introduction the toronto public library (tpl) has 100 branches [1] spread relatively evenly over nearly 450 square kilometres (figure 1).  with over 11 million items in this highly distributed collection, a common question among users is “where is the nearest copy of an item to where i’m currently standing?”   given the proliferation of location-aware mobile devices, such as gps-enabled smartphones, and the ubiquity of urban wireless internet access, the solution to this type of known-item query should be readily available to the library or any motivated programmer with some technical savvy and access to open data and technology. figure 1. map of tpl branches in december 2009, i decided to undertake this challenge with the goals of determining: a) whether it was technically possible to construct a tool-chain which could address this specific kind of query; b) what organizational or technical obstacles might exist in implementing such a tool-chain; and c) how reproducible such an approach would be for other libraries in north america and even internationally. the result of this project is a lightweight mobile web application, called mytpl (http://www.mytpl.ca/), which queries the z39.50 interface of tpl’s sirsidynix symphony ils based on user-provided search terms, and returns a list of matching items in the library’s collection.  when an item is selected, mytpl displays a detailed list of branches which possess a copy of the item, a current holding status, and – if the device’s gps hardware is enabled – links to display a map with directions from the user’s current location. this project was specifically done without any privileged knowledge or access otherwise unavailable to a member of the general public.  although i am an employee of tpl, mytpl is not officially acknowledged or endorsed by the toronto public library. mobile applications for libraries the growth of smartphone use in recent years has led to a surge in popularity of mobile applications for countless uses, including providing access to library holdings and services. griggs, bridges, and rempel discuss, in a previous code4lib journal article, the unique features provided by mobile devices (including location awareness), as well as the limitations of existing hardware and usability issues imposed by the small form factor [2].  perhaps more importantly, there is an active discussion at commonplace.net concerning strategies and best practices for developing mobile tools, including debate around when to deploy them as platform-specific applications, versus mobile-tailored web applications [3]. to date, the trend has strongly been toward developing iphone-specific applications, most likely due to the device’s dominance in the mobile market over the past several years.  some of the applications in the iphone app store include worldcat mobile from oclc [4] (figure 2), local books from librarything [5], and institution-specific applications like spl mobile from the seattle public library [6].  these applications, however, are limited to a small number of apple-made devices: the iphone, the ipod touch, and the ipad.  with the introduction of the apple ipad sdk in january 2010, even development for devices based on this platform has begun to fragment.  additionally, given the long-standing and ever-present critique of apple’s review policies for iphone applications [7], the greater flexibility and lower risk for multi-platform – or at least web-based – mobile applications seems to provide a more promising way forward. figure 2. worldcat mobile app ui for search, list, and detail pages while some institutions are developing simplified versions of web interfaces as a means of providing basic cross-platform support for their services, these implementations tend to be inconsistent in user experience and interface design.  worse, they can border on the useless –  librarything mobile [8], for example, begins with an uninformative “user name” entry box on the login screen and provides an extremely basic display in the detail view (figure 3).  similarly, mississippi state university’s mobile online catalog [9] uses a customized version of the sirsidynix ibistro web application, based on the outdated common gateway interface (cgi), and is difficult to navigate and performs poorly as a result. figure 3. librarything mobile ui for list and detail pages many cross-platform implementations are done through the services of third-party development vendors such as boopsie, who claim “if you can create a spreadsheet, you can create a mobile application; one that works across any phone with a data plan” [10].  certainly the promise of low-cost, universally-compatible mobile applications is appealing, but one of the major trade-offs of such an approach is it necessarily limits hardware functionality to the smallest subset available on all devices.  the worldcat mobile app, for example, cannot access the iphone’s gps hardware, significantly impairing the user experience. in january 2010, librarything announced library anywhere, a paid service promising “a mobile catalog for any library” and “mobile web and apps for iphone, blackberry and android” [11]. ala techsource provided pricing and technical details for library anywhere in march 2010, along with an expected release in april [12], but as of june 2010, it has not been formally released. problem domain and approach taken while there is an important need for libraries to clearly understand which services users desire in a mobile application on any platform [13], the service provided by the mytpl project was intentionally chosen to be basic: it was specifically restricted to known-item searches.  from a technical perspective, the overall objective was to translate a set of search terms into a set of geographic coordinates corresponding to a physical copy of an item. given the considerations and restrictions of developing an iphone-specific application outlined above, the technical requirements for the application were: develop mytpl as a web-based application targeting the webkit browser engine [14], which is used in the iphone’s browser, mobile safari. reduce barriers for replicating the application in other libraries by using open-source tools and open standards whenever available. in order to provide a user experience consistent with native iphone applications, i searched for a ui framework based on the iphone interface guidelines and found two prominent frameworks: universal iphone ui kit (uiuikit) [15] and iphone user interface framework (iui) [16].  based solely on the google code activity rating of “high” – indicating development on the framework was relatively current – i chose to implement the iui framework. the next issue was determining what access, if any, was available to tpl’s catalogue records.  while a number of libraries have provided deposits of their complete catalogue in repositories such as the internet archive [17], such records are static (i.e. not reflective of real-time holdings data), do not necessarily contain branch information, and most importantly, tpl had not made any such submission. like a majority of libraries, however, tpl operates a z39.50 server to cooperatively share access to their catalogue records with other libraries in real-time [18].  moreover, public access to the z39.50 interface is not restricted in any way.  based on some rudimentary experimentation and using some sample php code from a library colleague [19], i was able to establish a set of parameters which allow simple z39.50 queries to return marcxml from tpl’s catalogue, including branch holdings information.  while not every library ils will encode real-time holdings information in the marcxml records provided via its z39.50 interface, there is still enough information to make the application useful, e.g. for single-location libraries. the bulk of the remaining effort was to design a framework which would integrate the iui interface with the z39.50 query back-end in a lightweight and logical manner.  owing to my familiarity with the lamp (linux, apache, mysql, php) web platform, i created a small linux virtual machine (vm) on an apple mac mini to begin mytpl development.  lastly, thanks to the open-source yaz toolkit from index data, the php/yaz extension [20] was easily integrated.  php/yaz is a client library for connecting to and querying z39.50/sru/srw servers directly from php, and is readily available on many linux distributions. system design and architecture the design of mytpl is a classic three-tier client-server architecture (figure 4).  the data tier exists behind tpl’s sirsidynix z39.50 interface – in essence, treating it like a precursor to today’s modern web services.  due to the variant nature of z39.50 implementations between ils vendors, the data tier acts much like a “black box” in terms of being largely opaque to how it functions, for example, how retrieved search results are ordered. figure 4. mytpl system architecture the application tier resides on the web server, with application logic written in php and making use of a number of common extensions: dom/xml, xslt, and php/yaz.  in essence, the application is a linear transformation pipeline, with three functions: search (home/landing page): pass query terms from an html form to the z39.50 server list (search results page): display summary metadata from z39.50 marcxml response detail (detailed view page): display detailed metadata from above, as well as from external web services (e.g. cover artwork from open library), and geocoded links if gps is enabled screenshots of each of these three functions can be seen in figure 5. figure 5. mytpl ui for search, list, and detail pages the data transformation pipeline within the php application serves to convert binary marc21 delivered via z39.50 from the data tier to a simpler and more presentable html format appropriate for presenting to users (figure 6).  the conversion from marc21 to marcxml (and any encoding from marc-8 to utf-8) is handled transparently by the php/yaz library.  next, the real-time branch holdings information is extracted from marc field 926, and the record is transformed into mods xml using xslt provided by the library of congress [21].  the major reason for this transformation is to simplify the metadata being displayed, e.g. removing isbd punctuation, re-mapping iso-639 codes to language names, general string scrubbing, etc.  finally, some xpath-based selection is performed using document object model (dom) on the mods record to populate variables for displaying in html, both for the compact search results list, as well as the detailed view page. figure 6. mytpl transformation pipeline the presentation tier consists of the iui framework using css and javascript on the mobile device.  access to the device’s gps or other location-gathering hardware is done with javascript via the w3c geolocation api [22] as provided with the html5 specification.  the user is presented with a dialogue box to provide their latitude and longitude when the mytpl home page is loaded.  if the user declines, or if gps hardware is not available, the application still functions normally, and item holdings are presented in alphabetical order.  if the user allows mytpl to determine their position, then holdings are presented in order of geographic distance, closest being first, with an approximate distance shown in kilometers. the fourth function of mytpl, displaying directions between the user’s current location and a selected branch, is handed off to google maps and displayed either in a new browser window or by a native application on the mobile device (figure 7).  on the iphone platform this is done using a standard google maps url containing the latitude and longitude of start and end points, which allows the maps application to handle logic and presentation around directions, travel time estimates, etc.  the main drawback, however, is that it requires the user to leave the mytpl application entirely, and they must switch applications if they wish to begin their search anew. figure 7. iphone (google) maps application showing directions discussion and future directions when developing for a mobile platform, particularly for wireless internet access, performance considerations are paramount to delivering an effective user experience.  mytpl makes heavy use of simple caching at every point in the transformation pipeline to optimize response times as much as possible.  this is an especially important consideration given the legacy nature and typically slow response of the z39.50 protocol.  in addition, cover art is retrieved asynchronously from a number of open web apis, such as librarything, open library, google books, and movie poster db.  by returning, and subsequently caching, the first image available, page requests are kept lightweight and minimal.  thus, mytpl performance will actually benefit from additional use, as more metadata and images are stored locally over time. based on limited informal user feedback, cover art availability is far from ideal – in many cases, it can even be completely erroneous, depending on the source – but is extremely useful in providing visual cues for identifying and selecting a user’s desired item.  in many ways, it acts as a surrogate for richer metadata, allowing a minimal number of elements to be used: title, author, format, language, year, and number of copies.  a full mods description is available on the detailed view page, and content summaries or abstracts are provided for certain items where available.  in this particular area, there remains much room for improvement; for example, a sample integration with last.fm could provide high-quality music album descriptions with very little effort.  one can only imagine what kind of catalogue enhancement would be possible by retrieving additional metadata for movies or music from other openly-available web services. one particularly serendipitous aspect of mytpl discovered early on was that branch holdings information was, in fact, provided in real-time.  although a static mapping had to be developed between branch names embedded in marc data and geographic coordinates of the branch locations, this functionality gives mytpl the bulk of its usefulness.  while many library ilses will not encode branch holdings information in marcxml records provided via their z39.50 interface, in practice there is often information within the marc data that can be used to similarly identify points that can be used for item location.  perhaps more promising would be investigating the use of the z39.50 opac record format [23], which is designed specifically for exposing real-time holdings information. also serendipitous was the discovery much later that geolocation fully works with location-aware desktop browsers, such as firefox 3 on mac osx 10.6 using the core location framework.  mytpl was also later verified to work equally well on the browser application on mobile devices using the android operating system, which also uses the webkit browser engine.  although the ui is specifically designed for a mobile handheld display, it is not unreasonable to assume the application will continue to work equally well on a variety of w3c-compliant browsers and platforms, both desktop and mobile. additionally, while the w3c geolocation api ensures a degree of cross-platform compatibility among browsers, it also allows the possibility of keeping all location information – considered potentially sensitive by some – entirely on the client.  while mytpl does not take the approach to perform distance calculations on the server, it would not be technically challenging to modify the application to do so.  there are several additional alternatives for determining location, including ip address, wi-fi connection (such as skyhook wireless [24] as used by core location), and of course, the user self-disclosing a postal code or address. conclusion “everything should be made as simple as possible, and no simpler.” – albert einstein mytpl takes a ”less is more” approach to mobile library application design.  by their dynamic nature, online search tools lend themselves well to web-based applications.  while platform-specific mobile applications can provide a richer and more tightly-integrated interaction, judicious use of modern web standards can still deliver a satisfying user experience.  also key to building a successful mobile search application is delivering information in as rapid and minimal a way as possible.  when complexity cannot be avoided, it should be hidden from the user; instead, the user interface should provide a familiar and consistent style and layout. perhaps the greatest strength of mytpl as an application is that it is built entirely on open-source software and technology, freely available to anyone wishing to implement it.  moreover, because the z39.50 interface is common among most libraries, mytpl should be readily deployable with an absolute minimum of effort and configuration.  to date, it has already been tested with three canadian academic libraries, and while the tpl-specific location mapping generally does not function, the rest of the application performs as expected.  thus, mytpl provides a simple yet robust foundation for a large number of libraries to begin developing their own location-aware mobile catalogue search. like the tools it is built upon, mytpl is available openly and freely, in a spirit of sharing and mutual experimentation.  the complete source code is provided under the terms of the gnu gplv3 license [25], which provides the ability to relicense software based on mytpl under the gnu affero gpl v3 license [26].  the gnu affero clause requires servers running the code to continue to make it available in whole, including any modifications that have been made.  in this way, improvements to mytpl may be guaranteed to be perpetually available to the international library community. a common tenet of effective software, especially among the open source community, is to build tools that “scratch your own itch.”  mytpl is one example of this philosophy put into practice.  by focusing on solving the user’s specific problem using the tools available at hand – not necessarily the most complex, or the most cutting-edge, but rather an efficient combination of old and new – libraries can conceive of new ways to deliver modern services, while at the same time, leveraging those established technologies which would otherwise be relegated solely to their original purpose. references 1. hours & locations [internet]. toronto public library; [cited 2010 jun 08]. available from: http://beta.torontopubliclibrary.ca/branches-and-hours.jsp 2. griggs k, bridges l, rempel h. 2009. library/mobile: tips on designing and developing mobile web sites. code4lib journal [internet]. [cited 2010 jun 08]; issue 8. available from: http://journal.code4lib.org/articles/2055 3. mobile app or mobile web? [internet]. commonplace.net; [cited 2010 jun 08]. available from: http://commonplace.net/2010/02/mobile-app-or-mobile-web/ 4. worldcat mobile [internet]. itunes preview; [cited 2010 jun 08]. available from: http://itunes.apple.com/us/app/worldcat-mobile/id309643302 5. local books [internet]. itunes preview; [cited 2010 jun 08]. available from: http://itunes.apple.com/us/app/local-books/id335363746 6. spl mobile [internet]. itunes preview; [cited 2010 jun 08]. available from: http://itunes.apple.com/us/app/spl-mobile/id364019201 7. chen b. 2010. a call for transparency in apple’s app store. wired [internet]; [cited 2010 jun 08]. available from: http://www.wired.com/gadgetlab/2010/04/app-store-transparency/ 8. librarything mobile [internet]. librarything.com; [cited 2010 jun 08]. available from: http://www.librarything.com/m/ 9. mobile online catalog [internet]. mississippi state university; [cited 2010 jun 08]. available from: http://library.msstate.edu/m/ 10.  spalding t. 2010. library anywhere, a mobile catalog for everyone. the thingology blog [internet]; [cited 2010 jun 18]. available from: http://www.librarything.com/blogs/thingology/2010/01/library-anywhere-a-mobile-catalog-for-everyone/ 11.  breeding m. 2010. librarything delivers mobile access to library catalogs. ala techsource [internet]; [cited 2010 jun 18]. available from: http://www.alatechsource.org/blog/2010/03/librarything-delivers-mobile-access-to-library-catalogs.html 12. boopsie [internet]. boopsie.com; [cited 2010 jun 08]. available from: http://www.boopsie.com 13. rochkind j. 2010. mobile trendiness. bibliographic wilderness [internet]; [cited 2010 jun 08]. available from: http://bibwild.wordpress.com/2010/04/08/mobile-trendiness/ 14.  the webkit open source project [internet]; [cited 2010 jun 08]. available from: http://webkit.org/ 15.  universal iphone ui kit (uiuikit) [internet]; [cited 2010 jun 08]. available from: http://code.google.com/p/iphone-universal/ 16.  iphone user interface framework (iui) [internet]; [cited 2010 jun 08]. available from: http://code.google.com/p/iui/ 17.  open library data [internet]. internet archive; [cited 2010 jun 08]. available from: http://www.archive.org/details/ol_data 18.  catalogue help [internet]. toronto public library; [cited 2010 jun 08]. available from:  http://www.torontopubliclibrary.ca/hel_tro_index.jsp#z3950 19.  jordan m. 2007. php/yaz z39.50 client. interoperating.info [internet]; [cited 2010 jun 08]. available from:  http://interoperating.info/mark/node/71 20.  php/yaz [internet]; index data; [cited 2010 jun 08]. available from: http://www.indexdata.com/phpyaz 21.  conversions: metadata object description schema: mods [internet]; library of congress; [cited 2010 jun 08]. available from: http://www.loc.gov/standards/mods/mods-conversions.html 22.  geolocation api specification [internet]; w3c; [cited 2010 jun 08]. available from: http://www.w3.org/tr/geolocation-api/ 23.  consolidated asn.1 z39 recordsyntax-opac [internet]; library of congress; [cited 2010 jun 08]. available from: http://www.loc.gov/z3950/agency/asn1.html#recordsyntax-opac 24.  skyhook wireless: how it works [internet]; skyhook wireless; [cited 2010 jun 08]. available from: http://www.skyhookwireless.com/howitworks/ 25.  the gnu general public license [internet]; free software foundation; [cited 2010 jun 08]. available from: http://www.gnu.org/licenses/gpl.html 26.  the gnu affero general public license [internet]; free software foundation; [cited 2010 jun 08]. available from: http://www.gnu.org/licenses/agpl-3.0.html source code http://github.com/mjsuhonos/mytpl/ about the author mj suhonos is the metadata specialist at toronto public library (http://tpl.ca).  while he provides tpl with guidance on best practices for creating, managing, and sharing metadata with the broader internet community, he is unable to completely eschew his programming background and spends his free time hacking library metadata whenever possible. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – fractal in detail: what information is in a file format identification report? mission editorial committee process and structure code4lib issue 53, 2022-05-09 fractal in detail: what information is in a file format identification report? a file format identification report, such as those generated by digital preservation tools, droid, siegfried, or fido, contain an incredible wealth of information. used to scan discrete sets of files comprising a part of, or the entirety of a digital collection, these datasets can serve as entry points for further activities including appraisal, identification of future work efforts, and the facilitation of transfer of digital objects into preservation storage. the information contained in them is fractal in detail and there are numerous outputs that can be generated from that detail. this paper describes the purpose of a file format identification report and the extensive information that can be extracted from one. it summarizes a number of ways of transforming them into the inputs for other systems and describes a handful of the tools already doing so. the paper concludes that describing a format identification report is a pivotal artefact in the digital transfer process, and asks the reader to consider how they might leverage them and the benefits doing so might provide. by ross spencer introduction a file format identification report is a specific type of dataset produced by format identification tools such as droid[1], siegfried [2], or fido[3]. each of the tools, droid, siegfried, and fido are cross-platform (windows, linux, macos) utilities designed to scan individual files or collections of files. the output of this “scan” is called an identification result – a small chunk of metadata that describes a file or set of files in more detail. a single identification result might include the following: absolute path to the file on the local filesystem. filename. filesize. checksum (e.g. md5, sha256, or sha512). date last modified. pronom[4] unique file format identifier (puid). file format name. file format version. mimetype. a collection of file format identification results will usually output this information in a two-dimensional (table) format, with each row describing a different file. optionally, directory entries will also be described [5]. in aggregate, a file format identification report is an important artefact in the process of understanding and preserving a set of digital objects. this paper will describe the importance of this information and the ways it can be leveraged and used as inputs to other stages of the digital transfer process. droid, siegfried, and fido there are three primary format identification tools used in the digital preservation community across our respective glam (galleries, libraries, archives, museums) sectors[6]. these tools share a single property that distinguishes them from other utilities available to users, for example, the “file” command in linux, freebsd, or unix environments [7]. each of these tools attempts to resolve file format identification down to a single puid from the pronom file format registry [8]. puid (pronom unique identifier) puids are “minted” by the developers of pronom. they are identifiers for file formats and tend to have granularity down to version of the format [9]. puids were first designed to be uris in their own right in the info namespace[10]. in practice, the info namespace was not widely adopted and in 2010 it was recommended that adopters of the namespace migrate to http links following the publication of recommendations around http uris that supported identification (the info part) and resolution (the association of links with human or machine readable data)[11]. puids in practice do resolve as http uris with a human readable and machine readable description available to the caller. e.g. human readable (xhtml) and (xml), respectively, below[12]: https://www.nationalarchives.gov.uk/pronom/fmt/1 https://www.nationalarchives.gov.uk/pronom/fmt/1.xml a puid is part of a pairing between itself and a file format signature – a series of bytes written in pronom regular expression format that identify specific versions of a file format. the purpose of a puid therefore is to pair data on a caller’s hard drive, namely a users’ files and file formats, with the pronom registry – thus creating an unambiguous identification. jackson (2010) describes their hope for a linking model for file format registries[13]: “my current thinking on the data model is that the format registry design should only be concerned with minting identifiers for formats, and collecting the minimal representation information required to support this. i think that if we can get this right, then we can use those identifiers to describe the other entities, like software tools and digital object characteristics, but without forcing them all into the same system.” a linking model like this starts to look like a graph. the puid is the first node on that graph. with it in place, and returned by the aforementioned tools, it is possible to create connections between our collections and the technical metadata required to maintain them [14]. droid droid[15][16] was the first tool that was developed that utilizes the pronom file format registry. it was created in partnership with the national archives, uk (tna) by tessella scientific software solutions[17]. recent adaptations have been developed in-house by tna and with support around the identification engine (byteseek) from matthew palmer[18]. droid’s support by a non-ministerial government department, and its widespread use across the uk government for the transfer of digital records means that it comes with a strong reputation for the job that it does. it is written in java and works cross-platform, on windows, linux, and macos. droid is capable of scanning the contents of aggregate file formats [19]. its primary output is a comma-separated-values (csv) table describing files and folders row by row. droid describes the structure of files and directories using id and parent_id. an id that is of type folder and has no children, can be reliably identified as an empty folder in the droid csv. droid also has a field that describes empty directories as “empty”. the fields used in the droid output are as follows: id parent_id uri file_path name method status size type ext last_modified extension_mismatch hash format_count puid mime_type format_name format_version fido fido[20] was created in 2010 by adam fraquhar of the british library. fido was developed in python. where pronom specifies its own regular expression language for describing file-format signatures, fido converts pronom file format signatures into python regular expression syntax and uses a standard regular expression matching engine to perform the same function as droid. theoretically, as there is greater standardization in regular expressions used by python, there is a greater opportunity for others to write file format signatures for pronom. providing a host system has a python interpreter, fido is cross-platform just like droid. fido is capable of scanning (recursing) into aggregate file types just like droid. fido is focused on identifying the file format of files. if a set of nested directories existed that are all empty, e.g. $ tree empty_folder_one/ empty_folder_one/ ----> empty_folder_two --------> empty_folder_three 2 directories, 0 files fido would output no identification row for this set of folders. as such it is not possible to reliably determine whether a collection contains empty directories[21]. fido outputs a csv file without a header, however, its fields are as follows: status time puid format_name signature_name file_size file_name mimetype match_type the fido documentation describes them using the following syntax: “ok,%(info.time)s,%(info.puid)s,%(info.formatname)s,%(info.signaturename)s,%(info.filesize)s,\”%(info.filename)s\”,\”%(info.mimetype)s\”,\”%(info.matchtype)s\”\n” siegfried siegfried[22] is developed and maintained by richard lehane and was first announced in 2014. siegfried has been written in golang. this provides different benefits to java and python, but it still provides all important cross-platform compatibility. siegfried is compiled and released for windows, linux, and macos. siegfried offers alternative identification options including freedesktop’s mime-info, the library of congress fdd (file format definitions), and wikidata. siegfried also implements a pronom identifier using file format signatures from the registry but using a different pattern matching algorithm to droid or pronom to return its results based on those definitions. siegfried’s primary output is yaml[23], a plain-text structured key-value serialisation language. siegfried’s output is structured into a header which describes how siegfried was configured to run, e.g. it will describe siegfried’s version, and the version of the pronom signature file used. following the header, siegfried outputs blocks of identification results. siegfried’s identification fields are as follows[24]: filename filesize modified errors md5 match: ns match: id match: format match: version match: mime match: basis match: warning including the header section, the result for a single digital object might look as follows: --siegfried : 1.9.2 scandate : 1970-01-01t00:01:00 signature : default.sig created : 1970-01-01t00:00:02+00:00 identifiers : name : 'pronom' details : 'droid_signaturefile_v100.xml; container-signature-20211216.xml' --filename : 'filename.htm' filesize : 42 modified : 1970-01-01t00:00:00z errors : md5 : efe3693aa7019eb73969a0418e380e04 matches : ns : 'pronom' id : 'fmt/471' format : 'hypertext markup language' version : '5' mime : 'text/html' basis : 'extension match htm; byte match at 0, 15 (signature 1/2)' warning : siegfried also outputs its own version of a csv file, as well as a csv file conforming to droid’s specification. like fido, siegfried only outputs rows for files that it discovers, and so it is not possible to reliably determine empty folders using this report. droid, siegfried, or fido for the purposes of the remainder of this overview, the focus will be on droid and siegfried. while each of the tools discussed have their merits in terms of performance, how they can be implemented, and their ease of use in different contexts, both droid and siegfried have support for different hashing (checksum) algorithms. a checksum is unique to a file whereas a filename is not. we will go on to describe where each of the checksum, filepath, and filename are used in different, flexible ways in support of leveraging these reports for different needs and so having access to a file hash is one of the single most important aspects of a file format identification report if we are going to extract as much potential out of them as possible. fractal in their details once a collection has been scanned, then it is possible to drill down into the details of the reports generated, including the implications of those details which will help with understanding content, and work that may need to be planned in future to better look after the materials transferred, donated, or received. format identification format identification is perhaps the first detail that should be focused on. format identification tells us the file format that a record is encoded in. for example, this paper may be encoded in html (hypertext markup language) or pdf (portable document format) at some point. if there was a dataset associated with this paper it might be encoded in a tabular format such as csv (comma separated values), or json (javascript object notation). format identification is perhaps the first most important part of the conversation around maintenance of digital objects. as described, the puid is the first node in the graph of dependencies that build a picture of an operating environment where digital records were first created, e.g. format ‘a’ ? was created on software‘b’ ? was developed for operating system ‘c’ strength of identification the file format identification report will contain indicators of the strength of a file format identification. a container identification will normally indicate that the base format of a file format is zip, or ole2[25]. these are both “container” objects whose contents determine a sub-type of file format, e.g. microsoft word 97 utilises ole2 to create its doc file format; and microsoft word 2010 utilises zip to create its docx file format. a signature based identification is more simple and uses identifying byte-sequences within a file’s binary stream, to determine what file format it might be. container and signature based identification may be considered the most accurate form of identification using the tools discussed in this paper. extension based identification merely maps a file format extension, e.g. ‘.docx’, ‘.doc’, ‘.csv’, to a list of “known” extensions and their file format name, and no content analysis is completed, meaning that the file may not be what it purports to be. an “extension mismatch” will be returned when a signature or container identification is returned but the mapped extension is considered to be incorrect. beyond that, formats may also remain “unidentified” which raises the potential of other work to be completed to look after a digital object. format identification combinations can be summarized as follows: unidentified file formats as an indicator of further work unidentified files are an important detail in a format identification report when it comes to determining the work that needs to be done to stabilize a collection. unidentified files represent an unknown that in time needs to be corrected. various different pathways may exist: understanding an exception flow for information into a digital repository, i.e. anticipating that something like rosetta’s technical workbench[26] will raise an exception for unknown objects unless already anticipated. manually identifying the format of the unknown objects, e.g. through forensic analysis, or the use of other tooling, e.g. exiftool, linux file command, etc. preparing a file format signature submission to the pronom development team along with a collection of samples of the file format, either from your collection, or from elsewhere, e.g. uk web archive[27]. identified file formats as an indicator of unwanted files format identification when compared with an acceptable file formats policy may determine that there is work required to “weed” or destroy files in a given collection. this might result in two reports – a list of file paths of objects to be maintained and a list of those that should be removed. the latter being documented and agreed with the agency or donor, implicitly through legislation or more explicitly recorded. identified file formats as an indicator of the complexity of a collection identified files are an important detail in a format identification report when it comes to understanding the complexity of a collection and the work required to maintain it. each different file format becomes an aspect of the collection that will be monitored over time. preservation plans and policies might already indicate file formats that are associated with certain risks around sustainability[28]. aligning the results of the identification report with those policies helps us to identify gaps, and new formats that come under our custodianship from this point on. file and directory names depending on the source of the identification report, only a small section of a filesystem’s structure is described. take for example, the following two excerpts taken from an archives new zealand series[29]: one record might arrive with the following file path output in a format identification report: /speeches/culture and heritage/speech notes for launch of nga iwi o aotearoa-book.doc from this detail we can derive: the record is part of the ‘speeches’ structure in the minister’s folder structure. it is connected to ‘culture and heritage’. the document is presumably: speech notes. related to the launch of the nga iwi o aotearoa (maori peoples of new zealand) book. it was likely typed in microsoft word (.doc) format. /speeches/conservation/nzcps notes for iwi hui kaikoura.doc from the second structure we can derive: the record is also part of the minister’s speeches. however, it is connected to ‘conservation’. from the title of the document it may be: about nzcps (new zealand coastal policy statement). and to be presented at a maori iwi (tribe/s) hui (meeting). in kaikoura, on the east coast of new zealand’s south island. it was likely typed in microsoft word (.doc) format. this information can form part of the archival record with culture and heritage, and conservation forming sub-series. the filename becomes the title in the archival record. if we wanted, however, we can begin to draw potential lines of connection in the archival description between other access points and authority records, such as kaikoura the place, and more general classifications of maori record and collections related to maori people: with more granular description possible, once the content is described more thoroughly, such as specific iwi, geographical regions, and so forth. other lines of investigation around full file-path information may be possible, for example, natural language processing for classification of the different parts of the structure. this can potentially pick up people, places, and other different topics or facets. encoding file paths and filenames encapsulate different encoding issues. paraphrasing the archivist summaries of demystify[30]: paths may cause issues across different systems and applications. these could be filenames that include utf-8 characters such as macrons and other diacritics, or incidences of filenames with multiple space characters following one after the other. filenames may have explicit recommendations against using them, e.g. reserved names described by microsoft[31]. the early detection of this information can be achieved in many different ways but using the format identification report as a pivotal part of other digital transfer activities is one approach. with improving capabilities for handling unicode in modern programming languages, and the long, and long-overdue process of moving away from ascii (american standard code for information interchange), quality control and assessment of the accurate transfer of file paths from one encoding or language into other systems of record, be those digital preservation, or archival description, or some other host platform, should merely be a formality. that being said, the deliberate act of looking after this information[32] ensures we do not lose data. the purposeful management of “problematic” file names such as those described as reserved by microsoft, can only further increase our ability to maintain our digital record. empty directories empty directories can show up in a format identification report. directories that contain only other directories and no files can show up also. empty directories may be an indication of a file not transferring correctly or not being picked up correctly before being transferred. one might cross-check the format identification report with any other manifest sent with a collection of items. empty directories are difficult to record but they also have informational value as described in the file path examples above. they are part of the original order of a transfer and so a deliberate decision must be made about how to maintain that information – and/or how to document their removal from a transfer, as part of an appraisal process, or another process somewhere down the line. specifications such as bagit[33] or ofcl[34] (oxford commons file layout) do not have an easy task looking after empty directories and suggest workarounds. standards such as mets provide ways of documenting folder structures that can accommodate empty directories. file sizes file size in format identification reports help us to determine the volume of digital storage required. they help us to predict and request more storage resources. we may record the aggregate value as part of the accession record, and in archival description as we record the size and scope of a collection. file-sizes that are too large may be indicative of files that need more analysis and attention. for example, as one becomes more familiar with what to expect from different file-types, take a word document, if its size starts to creep into the tens of megabytes or more, then it might warrant review for embedded content and complexity. file-sizes that are too small may be indicative of a need for further work as well; an mp3 (mpeg-2 audio layer iii) or flac (free lossless audio codec) file that is only bytes or kilobytes in size may not have any significant intellectual payload and could be reappraised. files, in general, that are too small[35] may benefit from further forensic investigation to determine if they have been transferred correctly or incorrectly selected for transfer. zero-byte files zero-byte files are a specific example of where file sizes are important to recognize and understand up-front. it is also a concrete value that can be easily detected versus the fuzzier concept of a file being “too large’ or “too small”. zero-byte objects may have sensible naming conventions and sensible file-format extensions but have zero-payload. they would render blank in a hexadecimal (hex) editor if they were to be opened for forensic analysis. zero-byte files, perhaps more so than empty directories, may be indicative of something that may have gone wrong in a file transfer. it is something to ask of the agency or donor, or of any additional manifests sent with the digital objects. documenting these may be challenging, and if they are to be “preserved” it is important to make sure receiving systems, e.g. your digital preservation platform, understand how to look after them and do not remove them from a transfer. ultimately, it may simply be a further discussion with the agency or donor about their selection for transfer and may eventually be removed from the collection. having this information in the format identification report may help the agency or donor in the appraisal process. it helps us as long-term custodians as we process these collections. date last modified date last modified is a date that can be accessed consistently across the widest range of file systems[36]. for archival purposes the earliest date and latest date in a digital collection may accurately map to the beginning and end range of a sub-series description. from a debugging perspective the date last modified of different files should be monitored as the first indicator of a problem during transfer, appraisal, or analysis – homogenized (all dates are the same), or dates that are too recent compared to the others in the transfer may point to unintended changes to data, and some of those changes might have been made before fixity was truly in place for a transfer. it may also mean there is a problem somewhere in the digital transfer workflow where we attempt to preserve this information across different filesystem boundaries. checksum analysis while creating checksums cause a loss of performance when identifying digital collections, because of the need to read an entire file from disk, with respect to this paper, a checksum is one of the most powerful features of the format identification report. droid and siegfried share ‘md5’, ‘sha1’, ‘sha256’ as checksum options, and siegfried also offers ‘sha512’, and ‘crc’ (cyclic redundancy check). it will largely depend on institutional policy which one is adopted. when we look to match objects against “known” checksums as is discussed later on, it may be important to match the algorithm used by the source registry of those checksums. a checksum, as we know, describes the fixity of a digital document. they describe the uniqueness of the content of a digital record as it was at a given point in time. if a checksum for a digital object changes then it is indicative of something changing in the file, either by design or through corruption. a checksum is a digital fingerprint for a file. in combination with a file path, we can identify a file uniquely in a collection of digital objects. we can use both as a fingerprint that enables us to describe the digital object in front of us. at the most fundamental level: {<checksum> + <file path>} allows us to unambiguously attach the rest of the information in a format identification report with that row of data. the combination enables the following: {<checksum> + <file path>} format identification additional descriptive information about the digital record, e.g. archival description; additional technical metadata. file size date last modified duplicate detection the importance of distinguishing checksum + file path is that a collection may contain duplicates of other files. they may sit at different paths in the collection structure, but they are byte-for-byte duplicates. having the checksum in a report allows us to identify the byte-for-byte duplicates[37]. having the file path allows us to understand the context. once the purpose of a duplicate can be ascertained they are likely to be important enough to maintain, e.g. given enough context we might find enough information to distinguish different purposes for something. in the fictional example below, we can see a speech may have been recycled and used at a museum event in december 2021, and reused at a library opening january 2022[38]: /speeches/culture and heritage/museum_speech_dec_21.txt /speeches/culture and heritage/library_opening_speech_jan_22.txt maintaining duplicates may be an expensive undertaking and if context cannot be properly established, or we find the context largely identical, or duplicates are numerous enough, then there are disposal activities we can perform around this. connecting collections perhaps one of the most exciting prospects using checksums is to identify provenance across collections. i discuss in a previous paper the ways in which checksums might be used to identify links across series in archival collections[39]. taking one example, checksum duplicates received from two different agencies may tell us something about how both agencies were in communication with one another and then the presence of a document in two different contexts may illuminate more about that interaction. system files operating systems and different applications software are distributed with default files that are helpful to the user, or support the software itself. these files are best characterised as system files. the number of examples of “system files” is extensive. the true-type font “arial” that is distributed with microsoft windows could be classified as one, windows’ default wallpapers or executables (ms paint, minesweeper, and so forth). likewise, distributions of linux or macos will have common files and applications. common to specific versions of these “off-the-shelf” products, these files with few exceptions will share the same checksum with each other, and so, cmd.exe on my copy of windows 95 is going to have the same checksum as on your copy of windows 95. when i download droid 6.5 from the national archives, uk, it will also share the same checksum as your copy of droid 6.5, and so will all of droid’s supporting files. these files do not usually need to be maintained with a collection as they do not represent an intellectual output of the creator. if we can identify these files they can be weeded, or recommended for destruction, ideally during appraisal, but potentially at any time during the transfer lifecycle. registries for the checksums of these commonly distributed system files exist. a checksum in a file format identification report can be looked up against those registries and can help with further decision making around appraisal and disposal. later we will look at steffen fritz’s tool “filedriller” which recognizes this within the digital heritage context and attempts to help with just this. zero-byte-ness a final, but curious property of checksums, that may be of interest to some, is that a checksum can also identify whether a file is zero-bytes. another way of identifying zero-byte files, albeit one that is less practical than looking at the size if it is available, is by looking for a zero-byte checksum in a report. the checksum will always be consistent with the algorithm. a mapping of some zero-byte checksums is shown below: sha256  e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 sha1      da39a3ee5e6b4b0d3255bfef95601890afd80709 md5       d41d8cd98f00b204e9800998ecf8427e distance scanned in file a small but important part of identification is its accuracy. identification is largely dependent on how much information needs to be scanned to return an identification result. a feature that exists in both droid and siegfried is a limit on how many bytes are scanned when running identification. droid introduces this feature to users “out of the box” and limits scanning to 65535 bytes. users who want to make sure a file is scanned in its entirety need to set this value to -1. siegfried works the other way around and scans the entirety of a file out of the box and asks the user to configure a maximum number of bytes if it is desired. when a max scan value is set, there is always a possibility that a file will not be correctly identified. in reality, it is likely to be partially identified using indicators of another format. an example might be the pronom record for pdf/a-1a[40] which describes a signature as: identifies the byte sequence which equals – xmlns:pdfaid=”http://www.aiim.org/pdfa/ns/id/ which is the pdf/a identification namespace within the pdf/a meta-data, followed by 1 and a, which are the two mandatory elements and values for pdfa/1a. see iso 19005-1:2005 and technical corrigendum 1 for further information. the sequence is “variable” according to pronom which means that it can be found anywhere in a file. if that offset is over 65535 bytes it can be missed using default droid configuration. a more-likely result if this variable sequence is missed is a more standard pdf identification which may or may not be enough information for some users. introduced into siegfried was the return of information about how many bytes have been scanned when generating an identification result. this information can be used to configure a maximum byte scan value by finding an average for all the identification results in your collection. this could potentially be used to calibrate variable signatures in pronom in the future as well. if a balance of speed and accuracy is important, then these settings are worth looking into in more detail. codifying the details acknowledging the summary of details described so far, it becomes important to extract information from a file format identification report in a fast, consistent, and repeatable way. a collection can then be evaluated and reevaluated throughout appraisal and further processing activities using the same format identification report or updated copies based on the same collection of objects. a spreadsheet can only do so much, and it is difficult to repeat large numbers of queries and ensure those queries are always correct. it is also difficult to do without a consistent mental model of the information that you are looking at. for example, how you anticipate teasing apart information on identification accuracy, e.g. by container or signature, or both, or including extension-only identification as well. one approach this author has investigated is to convert the spreadsheet into a standardised database and then run sets of queries against that database. the database conversion component is called sqlitefid[41]. sqlitefid will convert droid and siegfried reports into a consistent database schema that can then be queried. the database schema looks as follows: figure 1. fields and tables created by sqlitefid. the abstraction described in this schema is already used to normalise droid and siegfried reports and can also be used for fido or any others outputting similar information. the tool demystify[42] uses sqlitefid and was first written in early 2014 and attempts to find a consistent set of queries to ask the database it creates. a summary of these queries is documented below. count all files in the database count all identification results for all identifiers as one count all rows in the id data table count of all extension mismatches count of extensions for all signature and container identified files in the database count of files from the database identified using signature or container methods for a given namespace count of files from the database with multiple identifications for a given namespace id count of files with multiple identifications count of non-pronom signature or container identification results count the number of container files in the database count the number of distinct files in the database count the number of files in container objects, e.g. zip count the number of namespaces used by an identification report count the number of unique directories in the database count the number of unique identifications count the number of zero-byte files in the database count of pronom signature or container identification results create a frequency list of unique errors output by the identification tool create a frequency listing of all ‘last-modified’ years in the database create a frequency listing of duplicate files in the database create a frequency listing of file format extensions in the database identify gaps across all namespaces used in an identification run list extensions of container objects in the database list the paths of the zero-byte files in the database select all ‘folder’ file paths from the filedata table select all filenames from the filedata table select all identification results across the database and order by namespace select all namespace data select all non-folder file paths from the filedata table select all unique file format extensions in the database select distinct file names from the database select file and id metadata from the database for a set of files in a given list of format ids select file paths and file names from the database for a set of files with extensions in a given list select file paths and names from the database for file names in a given list select file paths and names from the database for folder names in a given list select file paths from the database where the name is in a given list select files from the database that have extension mismatches select files identified using signatures or containers using pronom select information about all mimetypes recorded in the database where a mimetype is listed select information about all signature or container identified files in the database select information about files identified by extension only select information about namespace and identification based on the given identification method select metadata about all results ordered by namespace select metadata about identifications for a given identification method select metadata about objects with a byte match identification select paths from the database for files with a given checksum select the checksum type used by the database select the format identification tool used sum the total bytes used by all files in the database the sql for these queries can be found in this gist: https://perma.cc/n4pt-ckng[43]. brunnhilde[44] will be discussed as a stand-out example of the approach of using a database as an interface below. brunhilde was created by tessa walsh in 2016 and uses an in-memory sqlite database to function as an intermediary between a siegfried format identification report and additional reporting on top of that[45]. an api for querying format identification reports it is this author’s contention that a more traditional api driven approach may be feasible, and an improvement over the existing implementation in sqlitefid. an api approach would enable a format identification report to be queried using http queries and would offer users and developers more granularity with how they access results and piece them together to generate some of the outputs that will be discussed in more detail below. outputs become inputs with the potential of our format identification report becoming clearer, we can explore how the report is used, or can be used as an input to different tools. accession inventory / manifest with a checksum and file path, the format identification report can first function as a reliable inventory. a bureaucratic piece of information that describes what was received, and formatted correctly, and what was transferred into digital storage. at archives new zealand, droid was previously converted from csv to their archival description system’s “archway” input format[46]. the mapping required adding series and repository information to the data already contained in the report. file paths were split into sub-series, and file names formatted to record titles. the checksum in the report became a way of tying the digital record to this report and was important for aligning the archway import sheet with the input required for the rosetta digital preservation system[47]. at the end of a transfer process the report could then be formatted again in such a way to be returned to the transferring agency or donor to confirm what was received, transferred into preservation storage, and create a complete record of the digital transfer. droid-convert where the “archwayimportgenerator” was specific to archive new zealand’s archival description system at the time, droid-convert is a more recent tool developed by the national archives uk that might provide a useful reference to readers. the approach is similar and takes a droid report and transforms it into a format for transfer into tna’s digital preservation system[48]. appraisal the format identification report supports appraisal and re-appraisal. as previously discussed, the data in the report provides a snapshot of how a transfer needs to be stewarded and a collection maintained. demystify as discussed, demystify was first created in 2014 and takes the information in a format identification report and queries it systematically to provide comprehensive consistent reporting from the data. it endeavours to provide a static analysis of a collection. demystify outputs a html human readable summary of all of the information it can gather. demystify can be run on different versions of a collection’s format identification report as preparation work on the collection progresses. demystify was first inspired by c3po[49] (clever, crafty content profiling of objects). c3po it was decided could not easily be run in restricted government compute environments and so similar capability was translated to python standard library components in demystify wherever possible. demystify can help users filter collections into two different parts [50], quickly summarised as “files that we do not need to investigate further”, and “files that might need additional effort to transfer safely or remove from the collection”, namely “heroes” and “rogues”. the feature was created by this author and andrea k. byrne, and uses an allowlist and denylist feature to help archivists flag different issues they would like to investigate further in an isolated part of the file structure. brunnhilde brunnhilde is distributed with bitcurator[51] and was created by tessa walsh. brunnhilde is best described by the creator, as follows [52]: “brunnhilde was written to supplement existing file format identification and characterization tools, with a focus on producing human-readable high-level reports for digital archives and digital preservation practitioners.” brunnhilde provides collection level statistics, as well as augmenting this with file level information about whether viruses can be detected using clamav[53], or personally identifying information using bulk extractor[54]. freud freud is another analysis tool[55]. developed by paul young at the national archives, uk in 2018 it takes a droid report as an input and returns information about: unidentified formats. extension only identification. multiple ids. extension mismatches. compressed archival container formats. duplicate files. zero byte files. formats not on an allow list. filedriller filedriller[56] uses the checksum output of siegfried to compare file hashes associated with files to those in the national software reference library (nsrl) database[57]. the nsrl is one example of the aforementioned databases to capture information about commonly found files in different computing environments. fonts, or wallpapers, for example, distributed with windows are recreated many times across all of the numerous deployments of windows systems. these files all share the same checksum and so if they are identified in a collection it is highly likely they are not considered important in the transfer. with this information identified, a curator or archivist can begin to make this decision. filedriller can help with appraisal and removal of unnecessary information from our collections. rosetta ingest sheet a “rosetta ingest sheet” is a very specific example that was primarily used in archives new zealand, but the codebase has been used by a few other institutions running rosetta as well including the j. paul getty trust in california and the centre for jewish history in new york. rosetta provides csv as one mechanism for the ingestion of large numbers of digital objects. with the format identification report, an artefact that existed at the beginning of a transfer from a government agency to re-appraisal, and ingestion, it could be transformed into a csv that matched the specification expected by rosetta. paths needed to be modified to match those of a zip-based structure, and information would be optionally augmented with information provided by the archival description system. data was tied together with checksums and combined with a zipped directory of all the digital objects in a transfer provided rosetta with everything needed to store everything securely in preservation storage [58]. depending on the requirements of other digital preservation systems, this approach could be adapted to those as well. the key is treating this information we have at our fingertips as the fulcrum of that workflow. simulating a collection the current generation digital preservation tooling might not always work out of the box. as developers we have to find different ways of debugging systems and workflows. one limitation to this is that the final line of debugging, accessing original files, is not always possible. one approach this author has adopted in the past is when failing to get access to original digital objects, asking for a droid or siegfried report from the collection. a report might provide enough information to tease out some potential issues with a digital transfer. as the report has file-paths including file names and file sizes, it is possible to programmatically recreate a directory structure including mock files and mock data matching the size of the original collections. this data can then be wrapped in such a way as to mimic a transfer to help us understand if there is anything inherent in this detail that might be causing a problem somewhere. sometimes it has been feasible to create skeleton file format samples mimicking specific file-formats[59]. mimicking collections more closely using forensically detailed techniques such as this can help to generate more debugging information (potentially more realistic debugging) as these mock collections move through a transfer workflow. it may not always be possible to identify a problem this way but it has been useful for providing a means for more realistic bulk testing of systems, and to remove lines of investigation from a debugging analysis in the past. conclusion the file format identification report can be described as a primitive of the digital transfer process. a single format identification on its own might seem too atomic and have limited reuse value, but reports collecting this information in aggregate can make them a pivotal asset in the digital transfer process. as an abstraction of a filesystem and a digital collection, perhaps one of the most interesting aspects of the format identification report is that it provides a static view of what can be a lot of data, and what can be used in many different contexts. a file format report can be used to understand a digital collection even before an archive or library has access to those digital objects. they provide a picture of a world that can be used to analyze, debug, and maintain. it can be output once and revisited multiple times saving processing time and resources along the way. a fascinating tease for the future of these reports from the maintainers of droid might be to include “format type/classification” in the output one day[60]. one’s imagination starts to work overtime with the thoughts of the ways this information can be presented in a static analysis to provide different perspectives of a digital transfer and to understand its potential descriptive uses as well as understanding the habits and practices of a donor or agency. the format identification report is a dataset and treated as such we find that it is fractal and illuminative in its detail. while this paper covers a lot of potential uses, it is hoped that it has not uncovered all the potential uses. and it is hoped the reader will be inspired to provide examples to counter international biases in some of the tooling discussed here, and complete any gaps that are undoubtedly found herein. references acrobat pdf/a – portable document format 1a. (2022, february 11). pronom. retrieved march 7, 2022, from https://web.archive.org/web/20220307081426/https://www.nationalarchives.gov.uk/pronom/fmt/95 aggregate >> quality and functionality factors . (2022, 1 28). sustainability of digital formats: planning for library of congress collections. retrieved march 6, 2022, from https://www.loc.gov/preservation/digital/formats/content/aggregate_quality.shtml archives-new-zealand/archives-nz-rosetta-csv-ingest. (2014, august 13). github. retrieved march 13, 2022, from https://github.com/archives-new-zealand/archives-nz-rosetta-csv-ingest archives-new-zealand/archwayimportgenerator: code to create an archway import sheet from droid csv + external csv. (2017, may 31). github. retrieved march 7, 2022, from https://github.com/archives-new-zealand/archwayimportgenerator brown, a. (2006, july 27). the pronom puid scheme: a scheme of persistent unique identifiers for representation information. retrieved march 6, 2022, from https://perma.cc/axw5-8r6s brunnhilde. (2022, february 4). coptr. retrieved march 13, 2022, from https://coptr.digipres.org/index.php?title=brunnhilde&oldid=5570 bulk-extractor. (2021, november 15). kali linux. retrieved march 13, 2022, from https://www.kali.org/tools/bulk-extractor/ clamav about. (2022, march 13). clamavnet. retrieved march 13, 2022, from https://www.clamav.net/about clipsham, d. (2021, december 1). droid to additionally provide info on format type/classification? github. retrieved march 7, 2022, from https://github.com/digital-preservation/droid/issues/663 comparison of file systems. (2022, january 31). wikipedia.org. retrieved march 7, 2022, from https://en.wikipedia.org/w/index.php?title=comparison_of_file_systems&oldid=1069049316 demystify, tool for static analysis of format identification reports. (2013, december 10). github. retrieved march 7, 2022, from https://github.com/exponential-decay/demystify digital-preservation/droid: droid (digital record and object identification). (2012, april 25). github. retrieved march 6, 2022, from https://github.com/digital-preservation/droid droid (digital record object identification). (2021, august 17). coptr. retrieved march 6, 2022, from https://coptr.digipres.org/index.php?title=droid_(digital_record_object_identification)&oldid=5245 droid: file format identification tool. (2022, march 8). the national archives. retrieved march 8, 2022, from https://www.nationalarchives.gov.uk/information-management/manage-information/preserving-digital-records/droid/ exlibris. (2018, august 14). technical analyst workbench. exl-edu. retrieved march 13, 2022, from http://exl-edu.com/12_rosetta/rosetta%20essentials/sip%20processing/ta%20workbench/story_html5.html exponential-decay/moonshine: given four bytes, download a random file from web archives implementing the ukwa shine interface. (2018, march 14). github. retrieved march 7, 2022, from https://github.com/exponential-decay/moonshine exponential-decay/rosetta-csv-ingest: generate a rosetta csv ingest sheet with just a droid export csv. (2017, may 10). github. retrieved march 7, 2022, from https://github.com/exponential-decay/rosetta-csv-ingest exponential-decay/sqlitefid: normalize file format identification results (droid, siegfried) into a single sqlite db. (2017, may 26). github. retrieved march 7, 2022, from https://github.com/exponential-decay/sqlitefid farquhar, a. (2010, october 13). openpreserve/fido: format identification for digital objects (fido) is a python command-line tool to identify the file formats of digital objects. it is designed for simple integration into automated work-flows. github. retrieved march 6, 2022, from https://github.com/openpreserve/fido fido (format identification for digital objects). (2021, april 22). coptr. retrieved march 6, 2022, from https://coptr.digipres.org/index.php?title=fido_(format_identification_for_digital_objects)&oldid=4293 file (command). (2021, october 11). wikipedia.org. retrieved march 6, 2022, from https://en.wikipedia.org/w/index.php?title=file_(command)&oldid=1049303302 file format identification. (2021, april 20). coptr. retrieved march 6, 2022, from https://coptr.digipres.org/index.php?title=file_format_identification&oldid=3863 file format registry: new version released by the uk national archives. (2005, november 18). cambridge network. retrieved march 6, 2022, from https://perma.cc/lgf4-7h4y formats, evaluation factors, and relationships. (2017, march 2). library of congress. retrieved march 7, 2022, from https://www.loc.gov/preservation/digital/formats/intro/format_eval_rel.shtml#factors fritz, s. (2020, november 25). steffenfritz/filedriller – filedriller. codeberg. retrieved march 7, 2022, from https://codeberg.org/steffenfritz/filedriller implementation notes. (2022, february 3). oxford common file layout. retrieved march 7, 2022, from https://ocfl.io/0.1/implementation-notes/#empty-directories jackson, a. (2010, december 8). breaking down the format registry. open preservation foundation blogs. retrieved march 6, 2022, from https://perma.cc/87s5-a77k mcgath, g. (2013, january 15). the format registry problem. the code4lib journal, (19). https://journal.code4lib.org/articles/8029 microsoft compound file – just solve the file format problem. (2021, september 8). just solve the file format problem. retrieved march 7, 2022, from https://web.archive.org/web/20210119190525/https://www.nationalarchives.gov.uk/pronom/fmt/754 murphy, n. (2022, february 16). a beginner’s guide to brunnhilde: reasons for using the brunnhilde software. digital preservation coalition. retrieved march 7, 2022, from https://www.dpconline.org/blog/blog-niamh-murphy-brunnhilde-reasons naming files, paths, and namespaces – win32 apps. (2021, january 4). microsoft docs. retrieved march 7, 2022, from https://docs.microsoft.com/en-us/windows/win32/fileio/naming-a-file?redirectedfrom=msdn the national archives, uk. (2018, january 19). digital-preservation/droid-convert. github. retrieved march 7, 2022, from https://github.com/digital-preservation/droid-convert national software reference library (nsrl). (2016, april 19). nist. retrieved march 7, 2022, from https://www.nist.gov/itl/ssd/software-quality-group/national-software-reference-library-nsrl oclc. (2010, may 22). notice to freeze “info” uri scheme. “info” uri registry (frozen). retrieved march 6, 2022, from https://perma.cc/6evn-6bc6 palmer, m. (2010, december 8). nishihatapalmer/byteseek: a java library for byte pattern matching and searching. github. retrieved march 6, 2022, from https://github.com/nishihatapalmer/byteseek petrov, p. (2011, may 4). peshkira/c3po: clever, crafty content profiling of objects. github. retrieved march 7, 2022, from https://github.com/peshkira/c3po political papers – speeches and press releases. (2002-2008). archives new zealand. retrieved march 7, 2022, from https://collections.archives.govt.nz/web/arena/search#/item/aims-archive/22090/political-papers—speeches-and-press-releases pronom changes and droid signature file release notes. (2022, february 22). the national archives. retrieved march 6, 2022, from https://web.archive.org/web/20220306185322/https://www.nationalarchives.gov.uk/aboutapps/pronom/release-notes.xml puid for encrypted pdf’s . (2016, november 22). google groups. retrieved march 6, 2022, from https://web.archive.org/web/20220306190252/https://groups.google.com/g/droid-list/c/ogejx6oeaac/m/e01wacksawaj rfc 8493 – the bagit file packaging format (v1.0) . (2018, october). ietf tools. retrieved march 7, 2022, from https://datatracker.ietf.org/doc/html/rfc8493 richardlehane/siegfried: signature-based file format identification. (2014, february 28). github. retrieved march 7, 2022, from https://github.com/richardlehane/siegfried siegfried. (2021, april 21). coptr. retrieved march 6, 2022, from https://coptr.digipres.org/index.php?title=siegfried&oldid=4106 spencer, r. (2013, june 14). generation of a skeleton corpus of digital objects for the validation and evaluation of format identification tools and signatures. international journal of digital curation, 8(1). https://doi.org/10.2218/ijdc.v8i1.249 spencer, r. (2015, august 25). hero or villain? a tool to create a digital preservation rogues gallery. open preservation foundation blog. retrieved march 7, 2022, from https://openpreservation.org/blogs/hero-or-villain-a-tool-to-create-a-digital-preservation-rogues-gallery spencer, r. (2017). binary trees? automatically identifying the links between born-digital records. archives and manuscripts, 45(2), 77-99. 10.1080/01576895.2017.1330158 spencer, r., thornton, k., lehane, r., & cochrane, e. (2021, october16). wikidata: a magic portal for siegfried and roy. ipres proceedings 2021. https://osf.io/t8qab/ walsh, t. (2016, january 9). tw4l/brunnhilde: siegfried-based characterization tool for directories and disk images. github. retrieved march 7, 2022, from https://github.com/tw4l/brunnhilde walsh, t. (2022, march 8). brunnhilde overview. bitarchivist.net. retrieved march 8, 2022, from https://www.bitarchivist.net/projects/brunnhilde/ yaml ain’t markup language (yaml™) version 1.2. (2021, october 1). the official yaml specification. retrieved march 7, 2022, from https://yaml.org/spec/1.2.2/ young, p. (2018, february 15). digital-preservation/freud. github. retrieved march 7, 2022, from https://github.com/digital-preservation/freud footnotes [1] (droid on the coptr registry, 2021) [2] (siegfried on the coptr registry, 2021) [3] (fido on the coptr registry, 2021) [4] introduction to file format registries and describes the importance of pronom: (mcgath, 2013) [5] as-is the case with droid and will be discussed throughout this paper. [6] a more complete collection of format identification tools is listed on the coptr registry: (file format identification tools on the coptr registry, 2021) [7] file is just one example outside of the glam community.. file uses similar techniques to the others we will describe, and has been around since the 70s through different implementations. file attempts to resolve a file format identification to a descriptive string, so a web-page might be identified by file as: `index.html: html document, utf-8 unicode text, with very long lines`, rather than, say, fmt/471 (html 5.0) in the pronom registry. (file (command) described on wikipedia, 2021) [8] other tools popular in the glam sector, such as jhove can assert something about a format’s identification and return a puid. a key differentiation for the purpose of this paper is the identification coverage with those drawing directly from the pronom registry able to identify in upwards of a thousand different formats. pronom v102 has over 1600 file formats at time of writing: (pronom release notes, 2022) [9] an example of a format description at the feature level: (microsoft word document (password protected) 97-2003, 2015) and related thread on the droid-list discussing the use of pronom for further “characterisation” like identification: (puid for encrypted pdfs discussion on google groups, 2016) [10] (brown, 2006) [11] (oclc, 2010) [12] pronom has multiple namespaces. for the purpose of this paper, the file format namespace is fmt and x-fmt and the distinction between the two is largely historical. x-fmt records are no longer created, only updated or deprecated, and fmt records are created for new formats. [13] (jackson, 2010) [14] beyond the scope of this paper, other registries may yet complement pronom and enable resolution of a file format identification to a single web-resource that can unambiguously identify a file format. see (spencer et al., 2021) [15] (droid: file format identification tool, 2022) [16] (digital-preservation/droid: droid (digital record and object identification), 2012) [17] (file format registry: new version released by the uk national archives, 2005) [18] (palmer, 2010) [19] (definition of an aggregate format type from the library of congress, 2022) [20] (farquhar, 2010) [21] from both a forensic, and an archival perspective, this asks us to think carefully about how we describe the original order of a collection, which may negate, for a few, some of the positives described around file format identification reports later on in this paper. [22] (richardlehane/siegfried: signature-based file format identification, 2014) [23] (yaml ain’t markup language (yaml™) version 1.2, 2021) [24] “match” is used for convenience where “matches” describes a collection (nested sequence) of format identification results in siegfried with a relationship of one collection to many potential ids. [25] (ole2 described on the just solve it file formats wiki, 2021) [26] the rosetta technical analyst workbench is part of the rosetta digital preservation system from exlibris. files that demonstrate issues, such as being unidentified, are captured in the workbench for further analysis. files can be treated in the workbench, or moved into long-term preservation storage. if the problems are understood, exceptions can be created before a digital transfer is made so that known problems with treatments already in progress can be ignored. publicly available documentation on the technical analyst workbench feature is minimal but it can be observed in various tutorials and videos (exlibris, 2018) [27] moonshine is an example of a utility that can download objects from the uk web archive based on matching four byte sequences using a sparsely documented part of the uk web archive’s api: (exponential decay/moonshine: given four bytes, download a random file from web archives implementing the ukwa shine interface, 2018) [28] see (library of congress’ factors to consider when evaluating a digital format, 2017) [29] (political papers – speeches and press releases, 2002-2008) [30] (demystify, tool for static analysis of format identification reports, 2013) [31] (microsoft file naming guide, 2021) [32] for example, proper quoting, and escaping of file path strings in software and scripts that interact with this data. [33] (rfc 8493 – the bagit file packaging format (v1.0), 2018) [34] (description of empty directory handling in the ocfl specification, 2022) [35] bytes in size (less than one kilobyte) may be a good rule of thumb. [36] (metadata features of different file systems, 2022) [37] the droid user interface for example takes on the form of a table that allows for sorting on different columns. if the checksum column is sorted (labelled ‘hash’) duplicates will line up underneath each other in alphanumeric order. [38] it also tells us something about this individual’s habits. [39] (spencer, 2017, 77-99) [40] (pdf/a-1a on pronom, 2022) [41] (exponential-decay/sqlitefid: normalize file format identification results (droid, siegfried) into a single sqlite db, 2017) [42] (demystify, tool for static analysis of format identification reports, 2013) [43] the queries are generated from sample data. [44] (walsh, 2016) [45] (murphy, 2022) [46] (archives-new-zealand/archwayimportgenerator: code to create an archway import sheet from droid csv + external csv, 2017) [47] (archives-new-zealand/archives-nz-rosetta-csv-ingest, 2014) [48] (the national archives, uk, 2018) [49] (petrov, 2011) [50] (spencer, 2015) [51] (brunnhilde on the coptr registry, 2022) [52] (walsh, 2022) [53] (clamav about, 2022) [54] (bulk-extractor, 2021) [55] (young, 2018) [56] (fritz, 2020) [57] (information about the nsrl reference library, 2016) [58] (exponential-decay/rosetta-csv-ingest: generate a rosetta csv ingest sheet with just a droid export csv., 2017) [59] (spencer, 2013) provides a description of how to generate mock files simulating many different file formats. [60] (clipsham, 2021) about the author ross spencer (orcid: 0000-0002-5144-9794) is a digital preservation specialist for the international games and book publisher, ravensburger ag in germany. ross has been part of the digital preservation community for over a decade and has formerly been a digital preservation analyst and programmer in the teams at the national archives uk, archives new zealand, and artefactual systems inc. acknowledgements this author would like to acknowledge jan hutar and andrea k. byrne for their friendship, debate, and their persistence and contribution to some of the tools in this paper. to talei and anna and the team at archives new zealand for refining many of the concepts in demystify. to joshua ng for the push to refactor (and name!) demystify and for providing me with the inspiration to talk about it in a modern light. and to richard lehane for his patience with the bug reports and the opportunity to work alongside him on siegfried over these years. finally, to sara amato for their feedback editing this piece for code4lib. thank you all! subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – processing government data: zip codes, python, and openrefine mission editorial committee process and structure code4lib issue 25, 2014-07-21 processing government data: zip codes, python, and openrefine while there is a vast amount of useful us government data on the web, some of it is in a raw state that is not readily accessible to the average user. data librarians can improve accessibility and usability for their patrons by processing data to create subsets of local interest and by appending geographic identifiers to help users select and aggregate data. this case study illustrates how census geography crosswalks, python, and openrefine were used to create spreadsheets of non-profit organizations in new york city from the irs tax-exempt organization masterfile. this paper illustrates the utility of python for data librarians and should be particularly insightful for those who work with address-based data. by frank donnelly, geospatial data librarian, baruch college cuny 1  introduction one aspect of a data librarian’s mission is to help make datasets more accessible to users, which is not always a simple matter of teaching users how or where to search for them; it can also involve processing and cleaning datasets to make them readily usable. in the so-called age of “big data”, the traditional roles that relegated programming to programmers, analysis to statisticians, and finding and archiving to librarians is breaking down, as the need for interdisciplinary professionals who can understand and manage all of these related processes grows [stanton 2012]. with a few programming skills and some subject knowledge, be it bibliographic, geographic, scientific, or something else, data librarians can play a pivotal role in connecting users to useful data (for a discussion of evolving data services in academic libraries see [wang 2013], [mooney and silver 2010], and [read 2007]). this case study illustrates a process used to create a useful subset of the irs tax-exempt masterfile for new york city and a summary of the records by borough in one spreadsheet, as an example of how data librarians can add value to unrefined datasets.  this process involved three stages: using tools created by the missouri census data center to crosswalk zip codes with other types of geography, using python to select records within the geographic area of interest and to append geographic identifiers that would be helpful to users, and using openrefine to clean up data in the city address field of the dataset.  this study demonstrates the utility of specific tools like python and openrefine that data librarians and researchers can use to process data, and provides a process and script that can be used to produce this irs dataset for other states. common issues faced when working with address data are also illustrated; address-based datasets are widely available, but us zip codes pose unique challenges for researchers [grubesic 2008]. baruch college, located in midtown manhattan, is one of the senior colleges in the city university of new york (cuny) system. students and faculty from disciplines as diverse as marketing, real estate, entrepreneurship, public affairs, and sociology are often interested in finding neighborhood-level data within the city. many of these questions involve generating lists of different institutions for specific places; in particular there is a keen interest in studying non-profits. the tax statistics division of the irs publishes a dataset called the tax-exempt organization masterfile [irs 2014], which was created by scraping data from the 990 forms that all organizations must file if they are seeking tax-exempt status. the masterfile contains the name and address of every organization, along with a number of fields that classify organizations by tax-exempt status. while this resource does not include every non-profit (religious organizations, state and federal government institutions, and small charities with annual revenue less than $50,000 are not required to file – but some do anyway), it is an extensive and solid source for this type of information. guidestar, an information services and analysis organization for non-profits, uses this dataset as a foundation for building some of their products [guidestar 2013]. the irs masterfile data is available for download in state-based csv files[i]. in many instances a user will want to capture all organizations in a particular city. however, a major challenge is that a city, as represented in a us postal address, is often not the same as an actual municipality [stevens 2006]. postal city names will often refer to places that are unincorporated or that appear inside incorporated places, so that a user would have to know something about how the zip code[ii] coincides with municipal boundaries. the city of new york is a particularly complex example. new york, ny, only refers to addresses in manhattan, while addresses in three other boroughs are referred to by their borough name: bronx, brooklyn, or staten island. in queens, the city name can be queens, or it can be one of 42 officially recognized names assigned to different neighborhood post offices. so at minimum a user wishing to capture every record in new york city would have to look for at least 47 different possibilities for city name. unfortunately the situation is even more complicated, as residents (including the filers of the 990 forms) may use place name variants as a city address (such as harlem in manhattan or riverdale in the bronx), or may abbreviate (bklyn, blyn) or misspell (broooklyn, brooklin) the city name. rather than using the city name, users can extract data using the zip code, but in this example one would need to know what the 300 plus zip codes in new york city are. rather than forcing many users to repeat this arduous task, a process was created to identify zip codes by county, process the state-based irs file to select records with just those zip codes of interest, clean the city field to standardize it, and provide an easily accessible spreadsheet file.  while this script was created to generate specific output for new york city, it was generalized so that other data librarians and researchers can use it to generate data for their own geographic areas without having to modify it. you would just need to generate a table that relates counties to zip codes for your area of interest, which the script requires for input; the process for doing that is addressed in the next section. 2  zip codes a basic web search for a list of all zip codes in new york city reveals countless sources, of various and sometimes dubious quality. the missouri census data center (mcdc) has been providing invaluable tools, data, and advice to us census data users for many decades, and is a solid, definitive source [mcdc 2013a]. one of the many resources they provide is a master list or crosswalk of all 5-digit zip codes that contains every zip code, the type of zip code it is, the postal city name, a state abbreviation, and a census zcta number[iii]. the challenge here is the same one that faces us in parsing the irs file, specifically having to select all zip codes using 47 different city names. it would be preferable if we could select all zip codes in a particular county[iv]. to determine what county each zip code is in, we can use another mcdc tool called mable / geocorr to relate different census geographies. in this case we can look up the county code for every zcta, and then relate the county code to the zip code in the crosswalk via the zcta number; zip codes can’t be directly related to counties as they are not a census geography. what is a zcta?  zip code tabulation areas were created by the us census bureau to approximate us postal service (usps) zip codes. contrary to popular belief, zip codes have no geographic area; the usps assigns zips to ranges of addresses associated with street segments. private firms will use different methods to convert this data into actual geographic areas, and the us census bureau achieves this by aggregating census blocks where addresses in the blocks share common zip codes [census 2013]. some zip codes have no tangible area, either because they represent clusters of post office boxes or they are assigned to large, individual organizations that process a lot of mail. as a result there are fewer zctas than zip codes, as zctas only represent zip codes that occupy geographic space. the mcdc crosswalk can be used to assign all zip codes to zctas based on their location, so that po box and institutional zips are assigned to the zcta where they are physically located [mcdc 2013b]. this enables users working with address-based data to aggregate zip codes to zctas, in order to associate their data with census data published at the zcta level and to map their data in geographic information systems (gis) using zcta boundaries from the census tiger files. for this project the zcta number acted as a bridge to connect the usps zip codes in the crosswalk with the table output from the mable / geocorr tool that lists which county each zcta is in, so that we can assign a county to every zip code. this makes it possible for us to select all of the new york city zip codes from the crosswalk, and will allow us to append the county codes to every record in the irs file, so that we can summarize data by counties and so that users will be able to select records by county. using the mable / geocorr interface (at http://mcdc.missouri.edu/websas/geocorr12.html), you select a source (zcta) and a target (county) geography, and specify how to assign one to the other (by land area or by population). since some zctas have no residential population, the option to include records with zero population is checked. you select a state and use a filter box to input ansi / fips codes [v] to limit the output to specific counties. the output table will list every zcta / county combination. zctas don’t cross state boundaries but may cross county boundaries; in those instances there will be more than one record for each zcta, and the apportionment column will display the percentage of the population (or land area) that is in each county. once the file was downloaded and opened in a spreadsheet, duplicate records were deleted so that a zcta was assigned to only one county. in new york city there are only a handful of cases where zctas cross county boundaries and in this case the effect was marginal[vi]. the zip crosswalk and the geocorr table were imported into a sqlite database using the sqlite manager, an extension available for firefox. a basic sql query was executed that related the two tables based on zcta number (see figure 1). since the geocorr table only contains records for nyc, all non-matching records outside the city fall away, and the final list consists of one record for each nyc zip code, with its associated zcta and county. the query was exported out as a text file (zip_county_nyc.csv) that contained each zip – county combination. select zip, county from geocorr12_nyc, zipcodes where zcta5=zcta order by zip figure 1: mcdc crosswalk on left gets joined to gecorr table on right using the zcta number   3  processing with python a python 3 script was written to serve two purposes: create a subset of the irs file that contains just records for nyc, and create a summary table that counts the number of records for each county by type of organization. the entire program (irs_exempt_csv.py) was written as a function (irs_process) to process one state-based irs csv file, stored in a folder that’s named for the month and year (the irs seems to update the data on a monthly basis). the function takes four arguments from the user. the first is the name of the text file that contains the zip code to county relationship. the second is a month and year combination (mar2014) to identify which folder to draw the irs data from and to append the date to the name of each output file. the third is the two-letter state code that identifies which state-based irs file to process. the last element is a place name created by the user that generally describes the location of the records. process_irs_csv('zipcounty_nyc.csv','mar2014','ny','nyc') the path method from the python os module is imported first, which allows for the construction of cross-platform paths for reading and writing files using the parameters supplied by the user and static strings. the first part of the program defines some general functions that will be called subsequently. #!/usr/bin/python import os.path def process_irs(zipfile,monthyear,state,place): #functions def dictreader(file,delim): """read a file that has two values per line as a dictionary""" dictionary={} readfile=open(file) for line in readfile: key,value=line.strip().split(delim) dictionary[key]=value readfile.close() return dictionary def output_wlist(file,header,somelist): """write a list out to a file, with a header row""" writefile=open(file,"w") writefile.writelines("\t".join(header)+"\n") for records in somelist: writefile.writelines("\t".join(records)+"\n") writefile.close() def addto_dict(somedict,somelist): """add elements to a list value in a dictionary based on the number of items in another list""" for values in somedict: somedict[values]=somedict[values]*len(somelist) the second part of the program creates the subset file that has the individual organization records for our area of interest. the zip to county relationship text file is read in as a python dictionary called zipcounty using the dictreader function. in other programming languages the dictionary data structure is known as a hash or an associative array, but the concept is the same. a dictionary is an unsorted series of key / value pairs that are indexed by key, so that the values can be quickly retrieved by invoking their key. in this example the zip code serves as the key and the county ansi / fips code is the value, and the dictionary looks like this: {’10001’:’36061’,’10301’:’36085’, ’10034’:’36061’…} the irs file is read in next. the header row is captured first and saved in a list called header, and then as the program loops over each line in the file it checks the zip code to see if it is included in the zipcounty dictionary; if it’s in the dictionary then it’s an nyc zip, and the record is appended to a new list called keeplist that will contain just the nyc records. thus, each record is parsed using commas and stored as an individual list within keeplist, which is a master, nested list. an individual record list looks like this: [’133833921’, ’w b yeats society of n y’, ’% national arts club’, ’15 gramercy park s’, ’new york’, ’ny’, ’10003-1705’, ’0000’, ’03’, ’3’, ’2000’, ’199810’, ’1’, ’16’, ’092000000’, ’1’, ’13’, ’201306’, ’0’, ’0’, ’02’, ’0’, ’06’, ’0’, ’0’, ’0’, ’a20z’, ’national arts club’] this format makes it easy for the program to loop through the list to select the list element that contains the zip code, by using the index number for that element (in this case the zip code is index number 6; in python the first index is always 0). in the irs file the zip code is stored as a zip+4 number, so in the script we had to specifically check the first five digits for the values in index 6 to compare them against the dictionary to decide whether to keep the record or not. subsequently we modify keeplist to create a five digit zip code for each record as a new list element, and we get the associated county code for that zip from the zipcounty dictionary and append it as a new element. we also have to make sure we add headers for our two new elements in the header list. the list format also makes it simple to write the output; keeplist is passed through a simple function that writes the column header and each record out on its own line, with each element separated by a tab. the output file’s name is appended with the date and place name initially supplied by the user (i.e. irs_xorgs_mar2014_nyc.txt). zippath=os.path.join('z_geo',zipfile) zipcounty=dictreader(zippath,',') taxpath=os.path.join('%s','eo_%s.csv') %(monthyear,state.lower()) readfile=open(taxpath) keeplist=[] counter=0 header=readfile.readline().strip().split(",") for line in readfile: record=line.strip().split(",") if any(v==record[6][0:5] for v in zipcounty.keys()): keeplist.append(record) counter=counter+1 readfile.close() for record in keeplist: record[2]=record[2].strip('% ') zip5=record[6][0:5] record.append(zip5) record.append(zipcounty.get(zip5)) header.append('zip5') header.append('county') outpath=os.path.join('%s','irs_xorgs_%s_%s.txt') %(monthyear,monthyear,place) output_wlist(outpath,header,keeplist) print(counter,'records have been written to',outpath) the third and final part of the program creates a summary of the records. the irs uses exempt-organization codes to classify each organization into categories based on irs regulations. most of the organizations in the masterfile are classified as 501(3)c (abbreviated in the data as 03), which includes most public charities and private foundations. this part of the program counts records by eo code for each county. first, the county codes in the zipcounty dictionary are converted to a data structure called a set, which gives us a unique list of every county code in our area of interest. that set is then converted into a sorted list; this conversion is necessary because sets cannot be sorted. next, a dictionary is created for each of the eo codes, where the code is the key and the value is a list with a zero as the only element. a second dictionary is created to hold the sum total of records by county. both dictionaries are modified by passing each one into the addto_dict function; this function counts the number of counties in the county list and adds the appropriate number of values to each code in the dictionary. so in this instance, since there are five counties in nyc, each eo code in the dictionary eocodes will have a list value with five elements. each element will hold a count of the number of records for each county. once the dictionaries are ready, the program can loop over the keeplist of organization records. it looks at each eo subcode and county ansi / fips code in keeplist and checks for the same eo subcode in the dictionary; once it finds the matching code it adds a 1 to the appropriate element in the list, based on the index number of the matching fips in the county list (the county codes are sorted in numeric order, so the value that’s updated in the dictionary is in the same position as the value in the county list). if there isn’t a matching code in the dictionary the value is dumped into an unknown category. once the value is accounted for the sum total dictionary is updated as well. fips=set(zipcounty.values()) counties=sorted(list(fips)) eocodes=dict.fromkeys(['01','02','03','04','05','06','07','08','09','10','11','12', '13','14','15','16','17','18','19','20','21','22','23','24', '25','26','27','29','40','50','60','70','71','81','92', 'unknown'],[0]) summary=dict.fromkeys(['total'],[0]) addto_dict(eocodes,counties) addto_dict(summary,counties) for record in keeplist: subcode=record[8] fips=record[29] if subcode in eocodes: eocodes[subcode][counties.index(fips)]=eocodes[subcode][counties.index(fips)]+1 else: eocodes['unknown'][counties.index(fips)]=eocodes['unknown'][counties.index(fips)]+1 summary['total'][counties.index(fips)]=summary['total'][counties.index(fips)]+1 the remainder of the program formats the data for output. a list from the census bureau of all the county codes and their names [census 2010] is read in as a dictionary using dictreader, and the names are used as column headers. additional headers are inserted for the eo code number and for a description of the categories; the category names are also read into a dictionary using dictreader and are inserted into the eocode dictionary in front of the county totals for each code. since we are writing output from a dictionary and not a list we can’t use the output_wlist function defined earlier. in writing the output from the dictionary, every key / value pair is written to one line, but since the value is a list the output would appear in list notation, with elements separated by commas and enclosed in brackets. as a hack we can strip out the brackets, and by default the remaining commas will serve as delimiters. countyname=dictreader('z_geo/county_name.txt','\t') sumheader=[] for fips in counties: sumheader.append(countyname.get(fips)) sorted(sumheader) sumheader.insert(0,'eo subcode') sumheader.insert(1,'description') descriptions=dictreader('z_geo/irs_501.txt','\t') for key in descriptions: eocodes[key].insert(0,descriptions.get(key)) summary['total'].insert(0,'') outpath2=os.path.join('%s','irs_xcounts_%s_%s.txt') %(monthyear,monthyear,place) writefile=open(outpath2,'w') writefile.writelines(','.join(sumheader)+'\n') for key, values in sorted(eocodes.items()): writefile.writelines(', '.join([key,(str(values)).strip('[]')])+'\n') for key, values in (summary.items()): writefile.writelines(', '.join([key,(str(values)).strip('[]')])+'\n') writefile.close() print('summary has been written to',outpath2) the output file name is appended with the date and place name supplied by the user (i.e. irs_xcounts_mar2014_nyc.txt), and the content looks like this: eo subcode, description, bronx county, kings county, new york county, queens county, richmond county 01, ’501(c)(1) – government instrumentality’, 1, 0, 1, 1, 0 02, ’501(c)(2) – title holding corporations for single parent organizations’, 5, 16, 117, 30, 4 03, ’501(c)(3) – charitable and religious organizations’, 2328, 9230, 16725, 4533, 988… 4  cleaning with openrefine the last step of the process was to clean the city field in the organization records from our python output using openrefine (http://openrefine.org/), which would make it easier for end users to select and manipulate records using this field. originally a google product called googlerefine, the project was reorganized and released as open source in 2012. it is a desktop application that works within a web browser. you connect to a data file (which can be stored in any number of formats) and operate in a spreadsheet-like view. for this project the text-facet tool was used to group records into categories based on city name, which gave us the ability to see every single variant (and error) associated with this field so that they could be corrected. figure 2: openrefine interface   openrefine has a cluster tool that uses heuristic algorithms to select several clusters of text that it considers similar enough to possibly be identical, and the end user has the ability to correct them all en-masse. for items the cluster tool does not catch, the user can select individual facets and correct all the records that use that facet. for example, the different abbreviations and misspellings of brooklyn could be quickly identified, and each cluster of records could be corrected en-masse to match the correct term (see figure 2). using this tool, all misspellings were corrected, abbreviations standardized, and data that fell into the wrong field (i.e. a portion of the street address) adjusted. unofficial variants of city names were left as is. 5  conclusion the cleaned text file of organizational records was imported into excel using the text import wizard, and care was taken to assign the zip and county code fields as text so that leading zeros would not be dropped. the organizational summary file was imported into a second sheet in the workbook and cosmetic adjustments were made to each sheet. lastly, a third sheet was inserted in front of the data sheets that contained a brief description of the file, some metadata, and links to the original source. the file was stored in the older .xls format to insure greater cross and backward compatibility. the data file was uploaded to the library’s libguide platform and embedded in a guide for finding data for new york city at http://guides.newman.baruch.cuny.edu/nyc_data/hhs, a likely location where users doing city research would find it. a dedicated libguide box was created to hold the file, so that other librarians could link to it if their guide carried related content. an annotated copy of the python 3 script is included with this article, along with the necessary input files (that are read in as lists), a copy of ny state’s data for mar 2014, the zip to county relationship file for nyc, and the sqlite database that contains the list of all zip codes and zctas in the us. to generate data for other parts of the country, download the irs csv file for your state and follow the steps in the zip code section of this paper to generate a zip code to county relation file for your area of interest. create the same folder structure so that you don’t have to modify path names in the script: place the script in a folder, and in subfolders store the relation file and other input files in a folder called z_geo and the irs data in folders named for the date, i.e. mar2014. you can import the function process_irs or run the script irs_exempt_csv.py to load it in memory, then call the function with the necessary arguments: process_irs(’zipfile’,’monthyear’,’state’,’place’). references [census 2013] u.s. census bureau [internet]. (updated 2013). zip code tabulation areas (zctas) in geography – reference; [cited 2014 may 20]. available from http://www.census.gov/geo/reference/zctas.html. [census 2010] u.s. census bureau [internet] (updated 2010). 2010 fips codes for counties and county equivalent entries; [cited 2014 may 20]. available from: http://www.census.gov/geo/reference/codes/cou.html. [grubesic 2008] grubesic, th. zip codes and spatial analysis: problems and prospects. socio-economic planning sciences 42(2008): 129-149. [guidestar 2013] guidestar [internet] (updated 2013). what is the irs business master file, and why is it important?  in faqs: due diligence and guidestar charity check overview; [cited 2013 oct 23]. available from: http://www.guidestar.org/rxg/help/faqs/guidestar-charity-check/overview.aspx. [hayter 2005] hayter ct. 2005. adding a little zip to the mail: the development and growth of the zip code. dttp: documents to the people 33(3): 43-51. [irs 2014] internal revenue service [internet]. (updated 2014). soi tax stats – exempt organizations business master file extract; [cited 2014 apr 23]. available from: http://www.irs.gov/charities-&-non-profits/exempt-organizations-business-master-file-extract-eo-bmf.. [mcdc 2013a] missouri census data center [internet]. (updated 2013). [cited 2013 oct 17]. available from: http://mcdc.missouri.edu/. [mcdc 2013b] missouri census data center [internet]. (updated 2013). all about zip codes: 2010 supplement; [cited 2013 oct 17]. available from: http://mcdc.missouri.edu/pub/allabout/zipcodes_2010supplement.shtml. [mooney and silver 2010] mooney h, silver b. 2010. spread the news: promoting data services. college & research libraries news 71(9): 480-483. [read 2007] read ej. 2007. data services in academic libraries: assessing needs and promoting services. reference & user services quarterly 46(3): 61-75. [stanton 2012] stanton jm. 2012. data science: what’s in it for the new librarian?  information space, july 16. [internet]. [cited 2014 may 20]. available from: http://infospace.ischool.syr.edu/2012/07/16/data-science-whats-in-it-for-the-new-librarian/. [stevens 2006] stevens n. 2006, june 23. changing postal zip code boundaries. washington dc: congressional research service. order code rl33488. [internet]. [cited 2013 oct 17]. available from: https://www.fas.org/sgp/crs/misc/rl33488.pdf. [wang 2013] wang m. 2013. supporting the research process through expanded library data services. program: electronic library and information systems 47(3): 282-303. about the author frank donnelly (francis.donnelly@baruch.cuny.edu) is the geospatial data librarian and an assistant professor at the william and anita newman library at baruch college, city university of new york (cuny). he is a subject specialist in geography and geographic information systems (gis) and has extensive experience working with us census data. he holds an mlis from the university of washington and an ma in geography from the university of toronto. notes [i] the irs began publishing the data in this format in january 2014 to make it more accessible. prior to that time the data was available in state-based fixed-width text files that required a codebook to parse, or as a series of excel spreadsheets, with data for large states divided into several files. [ii] a zip code is a us postal code. an abbreviation for zone improvement plan code, zips were introduced in 1963 to speed mail processing and delivery. for a brief history see [hayter 2005]. [iii] the original source of the crosswalk is the uds mapper at http://udsmapper.org/zcta-crosswalk.cfm. [iv] the five boroughs of nyc are also individual counties within the state of new york; in some instances the borough names differ from the county ones. in this paper i use the term county, since that is the common term others would use in applying this project to their own geographic area. [v] every piece of census geography is assigned an ansi / fips number to uniquely identify it. every state is assigned a two digit code and every county a three digit code; for example 36 is the code for new york state, and within new york state 005 is the code for the bronx. for a list of state and county codes see [census 2010]. [vi] the effect may be greater in other places, for instance if a zip code is split in half by a county. the mcdc estimates that 10% of all zip codes cross county boundaries. if a zip code split introduces too much error, records for those organizations would have to be spot-checked or geocoded to unequivocally identify which county they are located in. in nyc there are three zip codes (11001, 11003, and 11040) that are partially located in queens but are primarily in nassau county on long island. these zips were manually deleted from the zipcounty_nyc.csv file so that the irs records in these areas would not be included in the final output. sample file a copy of the sample data files and scripts can be found here: http://journal.code4lib.org/media/issue25/donnelly/article_asset_files.zip   subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – augmenting the cataloger’s bag of tricks : using marcedit, python, and pymarc for batch-processing marc records generated from the archivists’ toolkit mission editorial committee process and structure code4lib issue 20, 2013-04-17 augmenting the cataloger’s bag of tricks : using marcedit, python, and pymarc for batch-processing marc records generated from the archivists’ toolkit catalogers have traditionally created and edited marc records on a one-by-one basis. recently, it has become more common for catalogers to delve into scripting and programming tools in order to automate the processing of large numbers of records simultaneously. this article provides a case study showing how marcxml archival records generated by the archivists’ toolkit (at) can be modified in batches using the marcedit software, python scripting and the pymarc module in python. it analyzes selected problems with the marc records exported by the at and shows how marcedit and python are used to resolve them so that the records are formatted correctly for loading into the library’s local catalog. similar methods could be used by catalogers dealing with any large set of marc data, such as ebook records from vendors. by heidi frank introduction the tools for catalogers have traditionally included content standards, such as aacr2, lcsh, lc classification, and now rda, and the medium for these standards has been the marc format. however, with more and more resources to describe and catalog, the approach of dealing with them one-by-one is not sustainable, and many catalogers who work with marc records in traditional library settings are now finding that they may need a new approach, one that combines traditional cataloging standards and programming, in order to efficiently manage their workflows. although the combination of these two activities can make a cataloger’s workload more manageable, many catalogers do not yet have the programming skills and find themselves needing to work with programmers to achieve their goals. unfortunately, there is often a disconnect when it comes to communication between the two groups, especially when dealing with legacy marc data. jason thomale articulates this in his article on interpreting marc, stating, “library catalogers and programmers clash (often passionately) about what constitutes ‘good library metadata.'”[1] he demonstrates how certain cataloging rules, such as the varied location of “=” within the marc 245 title field depending on which subfields are present, as well as the complexity of the marc format, can be a hindrance to automation. he also explains how assumptions made by a programmer who is unfamiliar with those complexities (as convoluted as they might be), also creates obstacles in technological development. fortunately, opportunities have become more prevalent in recent years to pull these two communities together, such as the curatecamp event for catalogers and coders [2] held as a pre-conference to the dlf fall forum in 2011. other examples are the ever-growing cataloging community involved in code4lib, the catcode [3] wiki space, and the number of catalogers participating in the codeyear project.[4] in addition, programs for manipulating marc data, such as marcedit, continue to be developed that make batch-cataloging capabilities more accessible to the non-coder. while a new environment for library metadata is under development in the form of the bibliographic framework initiative [5] and the linked data [6] model, catalogers will continue to work with marc records for a long time to come. therefore, a cataloger’s ability to add to their arsenal additional tools and skills to automate workflows by analyzing and correcting marc records in bulk is becoming more critical than ever. this article presents a look at applying some batch-processing methods – specifically using marcedit and the pymarc module in python – to automate the analysis and clean-up of large sets of collection-level marc records generated from the archivists’ toolkit (at). although the example presented is specific to archival collections described in the at, the ideas demonstrate how even a little bit of coding knowledge can provide a cataloger with a whole new approach towards efficiently working with any large set of marc records, providing quality metadata for any collection. marcedit marcedit [7] is a windows-based software program used for editing and manipulating marc data, and was developed in 2000 by terry reese, who continues to maintain and improve it. the basic gui interface is designed for a non-coder and provides many convenient analysis and conversion capabilities that do not require any direct knowledge of programming or scripting. for example, brandy klug provides some common examples for customizing the urls for electronic resource records supplied by vendors using marcedit’s built-in functions for adding field data or editing subfield content.[8] while these built-in functions are extremely powerful on their own, the workflow being described here additionally incorporates the use of complex regular expressions (regexes) that are based on the microsoft .net regular expression syntax.[9] and although the use of regexes on their own is not actually programming, it is a common method used in scripting languages when dealing with large amounts of textual data. marcedit is also used to convert files between various metadata schemas, and allows for the marcxml format generated by the at to be converted into marc source files (i.e., “raw marc”), and then into the mnemonic .mrk format for readability within the marceditor module. additionally, the software allows you to join multiple marc files into a single file for convenient batch-processing and analysis and loading into a local system. python scripting with pymarc the python scripting language [10] has numerous applications, from web development and design to text analysis. the pymarc python library [11] was created in 2005 by gabriel farrell, mark matienzo, and ed summers. an open source project developed specifically to work with marc record data, it provides common methods of manipulating field indicators, subfields, and field content. archivists’ toolkit the archivists’ toolkit (at)[12] is a software program collaboratively developed in 2006 by the ucsd libraries, the nyu libraries, and the five college libraries through a grant awarded by the andrew w. mellon foundation. the program allows archivists to describe and manage their collections, and also to generate encoded archival descriptions (ead) in an xml format for display and discovery on the internet. the at also has a feature that provides for the export of equivalent marcxml records for these collections, which is useful for the cataloger who primarily deals with the marc format and with traditional library catalogs because it eliminates the need for manual creation of the marc data. using the at as the source record also keeps the marc record in synch with the ead description. while the at has definitely proven its value for archival management, it introduces errors into the marcxml records that it is capable of exporting. some of these errors are simply the result of incorrect settings of the marc field data such as field indicators upon export of the record, but other errors have to do with how the data must be entered into the at interface, such as the single-box entry for multi-level subject headings. inconsistent data entry methods by at users, such as the use or lack of parentheses around data elements, can also be problematic. there have been other discussions about the formatting issues with at-generated marcxml records, such as the lack of subfield divisions for subject headings when trying to load records into oclc worldcat. [13] in that case, the records were ultimately edited manually in oclc’s connexion interface and the process continued to be on a record-by-record basis. the logical approach to resolve these issues would be to update the interface and the underlying configuration of the at program itself in order to provide fully valid marc data and content standards directly upon export. the reason this has not happened may be twofold. first, many catalogers who need to work with at-generated marcxml may not have access or permissions to edit records directly in the at, nor be able to collaborate with the archivists who do create the records in the at, and also they may not be very involved in the archival community. so they may feel it is easier to handle corrections to the marc data on the cataloging side of things. second, program development for the at software ended in 2009 due to the fact that many of the implementers are now working on the development of a new product called archivesspace, [14] which intends to supersede both the at and a similar program, archon. the archivesspace project was funded in mid-2011, and the core development began in 2012. archivesspace version 0.4 has just been released in march 2013, and includes some of the initial marc export features. while marc catalogers can now be involved in testing this new software in order to ensure valid formatting, [15] it may still be a while before current users of the at have fully implemented and migrated to archivesspace, leaving the need to deal with marc data issues from the at for some time to come. putting it all together using marcxml records exported from local instances of the at, the marc output was analyzed in depth in order to identify areas where content or marc standards were not followed. methods for correcting these problems using marcedit and a python script that includes the pymarc library functionality were identified. the diagram below demonstrates the basic steps in our workflow for transforming the marcxml records exported from archivists’ toolkit into a single file of corrected, standardized marc records, which is ready to be loaded into a local catalog. overall workflow for processing marcxml records from the at figure 1. diagram of overall workflow. ultimately, at the end of the workflow, the file structure will look something like this: figure 2. sample file structure. archivists’ toolkit application – marc & ead xml files the first step in preparing archival collection records from the at for marc cataloging is to export the individual eadxml and marcxml record sets from the at. when exporting multiple resource records from the at, the exported files are automatically named using the resource ids for each of the collections, followed by either “-ead.xml” or “-marc.xml”. this file-naming pattern is necessary for the python script since it relies on the resource id pulled from the marc 099 field to identify the corresponding xml files. so if exporting a single record, the same pattern must be followed when manually entering the filename, using the same capitalization, spacing and punctuation found in the 099 field (e.g., rg 9.7-ead.xml). these xml file pairs (e.g., 1973.071-ead.xml and 1973.071-marc.xml) are saved separately into the first two folders within the input_files folder, separating the ead from the marc. when capturing the ead file content, the python script looks for files in the eadxml folder that match the corresponding marc record being processed based on the resource id plus the extension “-ead.xml”. marcedit application – file format conversion the next steps use marcedit tools to convert file formats and create a single file of marc records to be modified in batch. to start, the set of marcxml files generated from the at are converted to individual .mrc files using marcedit’s batch process records option in the tools menu, pointing the source directory to the folder containing the marcxml files and setting the function to “from marcxml to marc”. figure 3. marcedit’s batch process marc records utility window. the batch process marc records function creates the folder within the marcxml folder called processed_files, which contains the individual .mrc formatted files. these individual .mrc files are then combined into a single file using marcedit’s marcjoin function. figure 4. marcedit’s marcjoin utility window. in the marcjoin utility window, the source file is actually set to the filename for the single .mrc file that will be created. based on the file structure example above, this would be set to:        \atrecords_2013_03\input_files\2013_03_all.mrc the “file(s) to join” is set by navigating to the processed_files folder containing the individual .mrc files and selecting all of the files in the folder. the result of this process is the single .mrc file found in the directory specified above. when this .mrc file is opened it will invoke marcedit’s marcbreaker utility window to convert the .mrc file to the eye-readable .mrk mnemonic format. figure 5. marcedit’s marcbreaker utility window. the resulting .mrk file can be accessed from the input_files folder, or upon execution of the conversion above, the edit records button will become active in order to immediately open the .mrk output file and begin the batch-editing tasks using the marceditor module. prior to editing the records, it is good practice to save a copy of the original .mrk file under a new filename, such as “2013_03_all_me.mrk” shown in the sample file structure. cleanup of marc data issues in records generated by the at a full list of the issues encountered with the marc data due to the configuration of the at or to data entry methods of at users, was compiled, [16] including descriptions of the cause of the problems and resolutions implemented using either marcedit or python scripting with the pymarc library. a few examples of these issues and their resolutions are provided below. collection dates (marc 008) problem 1: in order to populate the date1/date2 bytes of the marc 008 field, the inclusive begin and end dates must have been entered in the at resource record – i.e., the date expression field in the at is not sufficient. if the inclusive dates are not explicitly specified in the at, then the date1/date2 bytes export as blanks in the marc record. many search and discovery systems rely on these date fields for sorting and for limiting searches, so these fields are crucial for the user. resolution: since entry of the inclusive begin/end dates in the at depends on the archivist creating the resource record, and not on the configuration of the at interface, local guidelines were documented on the proper entry of dates in the at. [17] anticipating that these guidelines may not be followed consistently, it is desirable to check for the absence of dates in the 008 using the find function in marcedit and the following regex:        find: =008 .{6}i\\\\ this regex looks for an 008 field, followed by two spaces, then skips the first six characters of the field using “.{6}”, and finds the date status code “i” followed by four blank characters “\”. it identifies any records where bytes 07-10 of the 008 field (i.e., date1) are blanks, thereby providing a list of records needing dates to be input into the at. this check could also be incorporated into a python script, using pymarc to check for records where the date1 bytes are blanks, as follows: for field008 in rec.get_fields('008'): field008data = field008.value() field008date1 = field008data[7:11] if field008date1 == ' ': # write record info to file this script finds the 008 field in the given record and captures the string of data held there. here are data string samples for four 008 fields: 120410i18051805xx eng d 120410i18561856xx eng d 120925i xx eng d 120106i16331929xx eng d the script then creates a substring from position 7 (the first position is 0) up to position 11, which corresponds to bytes 07-10 of the 008, and checks whether those four bytes are space characters, as shown above in the third 008 data string. if there are spaces in those positions, then the record’s descriptive information can be written to a text file, and this marc record can be output to a “reject” .mrc file so that the at records can be populated and re-exported. problem 2: for collections that span only a single year, the at still requires that both an inclusive begin and end date be entered (i.e., you are not allowed to leave the end date blank if there is a value entered in the begin date field). in these cases, the date status field (dtst – byte 06 of the 008 field) is still coded upon export as “i” for inclusive dates, and the same year is entered in both date1 and date2 of the 008 field. the correct marc format for single year spans would be to have the date status coded as “s” for a single date and the year entered only in date1. resolution: a simple find/replace function can be used in marcedit to convert the 008 field to a single date format, as follows:        find: (=008 .{6})i(\d{4})\2        replace: $1s$2\\\\ this find regex looks for 008 fields that have the same set of 4 digits consecutively after the date status code “i”, where “\d{4}” finds the first 4 digits and “\2” finds that same pattern directly after. the replace syntax then copies the content of the first parenthetical section using the reference $1, replaces the date status code with an “s” (for single date), followed by the first set of 4 digits (date1) referenced by $2, and replaces the second set of 4 digits (date2) with blanks (“\\\\”). non-filing characters of titles (marc 245, 630) problem: the 245 main title field of the marc record exports from the at with default indicators of “1\”, whereas in actual cataloging practice the second indicator is meant to indicate the number of non-filing characters at the beginning of titles, allowing systems to skip leading articles such as a, an or the. for example, the second indicator would be a “4” if the title begins with “the”. most of the time, this second indicator should be a zero, but in cases where there is a leading article, the title would not appear correctly in an alphabetical listing of titles (unless the search interface is already programmed to recognize common articles). resolution: the first step would be to use marcedit’s edit indicator function to change all of the second indicators of the 245 fields to zero, setting field=245, indicators=1\, and replace with=00. (note: the first indicator, which designates whether the 245 field is used as the main entry, can also be changed to a zero or left as a 1, depending on local decisions for handling main entries.) once all of the second indicators are set to zero, then a series of find/replace commands can be used to change indicators for those titles that begin with a common article, as follows: find replace (=245 .)0(\$aa ) ${1}2$2 (=245 .)0(\$aan ) ${1}3$2 (=245 .)0(\$athe ) ${1}4$2 the find regex looks for a 245 field with the second indicator equal to zero and the first letters of the subfield $a equal to either a, an, or the. you must include a single space following the article in the find regex. the replace syntax references the first and second parenthetical sections of the find regex using “${1}” and “$2”, and the middle number is changed to reflect the correct indicator value (2, 3, or 4, respectively). the presence of non-english articles, such as la, las, or le, must also be taken into account. these articles may or may not need to be skipped – as in the case of a title such as “las vegas.” one could also write a python script that would first determine the language of the material based on the lang code in bytes 35-37 of the 008 field, and then modify leading articles accordingly. given the presumed rarity of these occurrences, however, it may be acceptable to ignore them or to search for them separately on a case-by-case basis. the indicators of the 630 fields for uniform titles pose a similar problem to those in the 245 main title field and can be fixed analogously. abstract versus scope and contents notes (marc 520) problem: although the notes for both the abstract and the scope and contents are specified as such in the at interface, upon export to marcxml, these notes both get entered into marc 520 fields without the proper indicators to identify which one is which. they are both assigned blanks as their first indicator, rather than either a 3 or 2 for the abstract or scope and contents, respectively. the scope note is often too large for the maximum size of the local catalog’s record fields, so if one wanted to delete the scope note from the marc record, but keep the abstract, there is not a way to easily identify which one needs to be deleted. resolution: the pymarc and python scripting is used to extract the 520 fields from the marc record and both the abstract and scope and contents notes from the corresponding eadxml record. it then normalizes the content of each down to basic alphanumeric characters (i.e., deleting any html coding, extra spacing, etc.), and compares the 520 content with each of the notes from the ead using a text-comparison analysis tool to determine whether or not they match. the geteaddata python method below opens and reads the eadxml file that corresponds to the marc record under review using the resource id as the filename, and then parses the eadxml data to capture the resource id from the <num> tag, plus the <scopecontent> and <abstract> tags. the content of these three fields is normalized using the normalizetext method to remove extra white space, non-alphanumeric characters, html tagging, etc., and then they are copied into the eaddata variable, a python dictionary structure, as key/value pairs. import re from xml.dom.minidom import parsestring ###################################################################### ## method geteaddata ## get resource id <num>, scopecontent & abstract from ead xml file ###################################################################### def geteaddata(resid): eaddata = {} # dictionary variable to hold ead field data eadfilename = foldername+'/input_files/eadxml/'+resid+'-ead.xml' eadfile = open(eadfilename, 'r') eadstr = eadfile.read() eadfile.close() xmldom = parsestring(eadstr) # get the resource id from the ead <num> tag if xmldom.getelementsbytagname('num'): numtag = xmldom.getelementsbytagname('num')[0].toxml() numdata = numtag.replace('<num>', '').replace('</num>', '') numdata = numdata.encode('ascii', 'ignore') else: numdata = '' # get the scope note from the ead <scopecontent> tag if xmldom.getelementsbytagname('scopecontent'): scopetag = xmldom.getelementsbytagname('scopecontent')[0].toxml() scopedata = normalizetext(scopetag) else: scopedata = '' # get the abstract note from the ead <abstract> tag if xmldom.getelementsbytagname('abstract'): abstracttag = xmldom.getelementsbytagname('abstract')[0].toxml() abstractdata = normalizetext(abstracttag) else: abstractdata = '' # create an entry in the eaddata dictionary for this record eaddata['id'] = numdata eaddata['scope'] = scopedata eaddata['abstract'] = abstractdata return eaddata ###################################################################### ## method normalizetext ## strip and normalize text, removing extra white space, non-alpha chars, html codes, etc. ###################################################################### def normalizetext(text): # create variables for regex patterns that need removed headtag = re.compile(r'<head>[^<]*</head>') htmltags = re.compile(r'<[^>]+>') htmlampcodes = re.compile(r'&[^;]+;') nonalphanum = re.compile(r'[^\sa-za-z0-9]') newlinechar = re.compile(r'\n') whitespace = re.compile(r'\s+') # remove or replace any html tags, non-alpha chars, and extra white space, etc. text = headtag.sub('', text) text = htmltags.sub('', text) text = htmlampcodes.sub('', text) text = nonalphanum.sub(' ', text) text = newlinechar.sub(' ', text) text = whitespace.sub(' ', text) text = text.strip() text = text.encode('ascii', 'ignore') return text once the eaddata dictionary is created, the following python script then loops through the 520 fields for each record using the pymarc library, normalizes the text of the 520 note using the normalizetext method above, and then uses the difflib library [18] to compare this text to both the abstract and scope notes captured from the ead. if the difference ratio is greater than 0.9, then the 520 text is considered a match, and the first indicator of the 520 field is modified accordingly. if the 520 text does not reach this comparison ratio level for either note, then it is identified as not a match and the record information is again written to a text file for manual analysis. import pymarc from pymarc.field import field import difflib marcrecsin = pymarc.marcreader(file(foldername+'/input_files/'+mrcfilename), to_unicode=true, force_utf8=true) # iterate through all the marc records in the specified .mrc file for record in marcrecsin: resid = record['099']['a'] eaddict = geteaddata(resid) for field520 in record.get_fields('520'): field520a = field520.get_subfields('a')[0] # get the first item in the list of all subfields $a (there should be only 1 in list) altfield520a = field520a # capture a copy of the 520 text that will be altered for comparison altfield520a = normalizetext(altfield520a) # strip and normalize the 520 text for comparison diffratioabstract = difflib.sequencematcher(none, altfield520a, eaddatadict['abstract']).ratio() # returns a value of the similarity ratio between the 2 strings diffratioscope = difflib.sequencematcher(none, altfield520a, eaddatadict['scope']).ratio() # returns a value of the similarity ratio between the 2 strings if diffratioabstract >= 0.9: # this 520 matches the abstract if len(altfield520a) <= 1900: # this 520 is within 1900 characters new520abstract = field(tag='520', indicators=['3',' '], subfields=['a', field520a]) # create a new 520 field for the abstract with 1st indicator=3 else: abstracttoolg = lgabstracts(rec=record, text=altfield520a) # write to file when abstract is greater than 1900 characters, and return true elif diffratioscope >= 0.9: # this 520 matches the scope if len(altfield520a) <= 1900: # this 520 is within 1900 characters new520scope = field(tag='520', indicators=['2',' '], subfields=['a', field520a]) # create a new 520 field for the scope note with 1st indicator=2 else: nomatch520 = nomatch520s(rec=record, eaddata=eaddatadict, alt520=altfield520a, diffabstract=diffratioabstract, diffscope=diffratioscope) record.remove_field(field520) # remove the old 520 field that has blank indicators if new520abstract: record.add_field(new520abstract) if new520scope: record.add_field(new520scope) subject heading subfields (marc 6xx) problem: the interface in the at for entering subject headings is not granular, and there is only a single text box for entry, so it is not possible to distinguish marc subfield codes for subject headings that include any form of subdivision. resolution: local guidelines [19] were developed directing the at user to separate subject subdivisions with ” |x ” (space–vertical bar–subfield code–space) when entering subject headings in the at that include subdivisions (i.e., marc subfields), such as these headings:        actors |x labor unions |z united states        cemeteries |z new york (state) upon processing, marcedit is able to use the find/replace function to convert the ” |x ” syntax to “$x”, using this regular expression (regex) combination:        find: ( \|)([0-9a-z])( )        replace: $$$2 this find command looks for a space followed by the vertical bar, followed by either a digit or lowercase letter, followed by another space, and then replaces that combination with a dollar sign using the escape syntax “$$”, followed by the content of the second parenthetical section of the find regex using the reference “$2”, which is the alpha-numeric subfield code. the subject heading examples above would become:        actors$xlabor unions$zunited states        cemeteries$znew york (state) while this fix satisfies marc standards, it becomes a problem for the ead, which is usually formatted with double-dashes separating the subject heading subdivisions (e.g., cemeteries — new york (state)). the css managing the ead display was changed to convert the new syntax back to double-dashes. name heading punctuation (marc 600/610, 700/710) problem: when name headings are exported from the at, they are lacking the appropriate punctuation between subfields, and in some cases, the subfields are not in marc order. for example, when subfield $q is present in a 600 or 700 field, it is not enclosed in parentheses (unless the parentheses are explicitly added into the name record in the at), and it appears after subfield $d when present, instead of before. another issue that had to be dealt with locally was with records that were originally ingested into the at database, which had the complete content of each of the personal name headings (i.e., both last and first names, dates, fuller forms of names, etc.) entered into the single field for primary name (i.e., last name). so upon export, the full name headings would be in subfield $a, and there was no easy way to parse the subfield content. these non-standardized name headings prevent collocation with their authorized versions, which affects faceting and searching and becomes an issue for the front-end user. resolution: the first step locally was to retrospectively go through all of the personal name headings in the at, manually moving the different parts of the headings into their appropriate fields – such as the dates, the first and middle names to the “rest of name” field, etc. local guidelines were written on the proper entry of name headings in the at [20] and were provided to the at users for this retrospective cleanup of the headings and for new headings to be added. to address the punctuation issues, a long series of find/replace commands were constructed that look for various subfield combinations and reorder and replace them with the appropriate punctuation. here is a sample: find replace (=700 ..)(\$a[^\$]+)(\$d[^\$]+)(\$q)([^\$]+)(\$e.*$) $1$2$4($5),$3,$6 (=700 ..)(\$a[^\$]+)(\$d[^\$]+)(\$e.*$) $1$2,$3,$4 (=700 ..)(\$a[^\$]+)(\$c[^\(][^\$]+[^\)])(\$d[^\$]+)(\$q)([^\$]+)(\$e.*$) $1$2,$3$5($6),$4,$7 the three find/replace commands above address the following subfield combinations: find 700 subfield pattern: convert to: $a $d $q $e $a $q (), $d, $e $a $d $e $a, $d, $e $a $c $d $q $e $a, $c $q (), $d, $e a similar set of find/replace commands were written to insert the appropriate punctuation into the 600, 610 and 710 fields. marcedit application – processing tasks many of the resolutions to the marc data problems involve the use of marcedit functions, which can be automated using the manage tasks tool of the marceditor module. this tool provides the ability to combine editing tasks into a list that can be initiated in a single step. figure 6. marcedit’s marceditor tools menu – manage tasks. figure 7. marcedit’s task manager tool. the task manager is the most powerful component used to resolve the majority of the problems that have been described with at marcxml records. for this workflow, a single task list was created which incorporates all of the find/replace commands mentioned, as well as a number of other minor corrections and enhancements to the records. the full list of tasks and descriptions of the modifications being made are maintained in a spreadsheet [21], but a portion of this task list when viewed in the task manager is shown below. figure 8. sample task list in marcedit’s marceditor task manager. once the task list has been applied to the .mrk file of marc records, the file is then converted back to .mrc format for python processing by choosing “compile file into marc” in the file menu of the marceditor module, and saving the file in the input_files folder. in the file structure example, this file is the one named “2013_03_all_python.mrc” and is the filename that will be entered when running the python script. python processing scripts with pymarc the key functions of the full python script [22] are to identify records that may need to be edited in the at. among other things, the script looks for records missing dates in the 008 field, lacking an abstract, or not having any named creator (i.e., no 1xx or 7xx). it also analyzes the 520 notes to identify which are abstracts and which are scope notes, as well as deleting any 520 or 545 notes that are too long for the field size limitations in the local system. it finally evaluates the 7xx fields to de-duplicate any name headings that have multiple roles assigned. when the script is run, it prompts for a folder name and then a filename. the folder name will be the top level folder in the file structure, and the filename is the name of the .mrc file of marc records to be processed – in the sample file structure shown above, the folder name is “atrecords_2013_03” and the filename is “2013_03_all_python.mrc”. once the analysis is complete and the modifications are made, the script outputs the following list of files to the output_files folder shown in the sample file structure: 2013_03_all_python_good.mrc 2013_03_all_python_lgabstracts.txt 2013_03_all_python_no7xxs.txt 2013_03_all_python_no008dates.txt 2013_03_all_python_noabstracts.txt 2013_03_all_python_nomatch520s.txt 2013_03_all_python_rejects.mrc the five .txt files provide lists of records identified as having problems, which can be given to a cataloger or archivist to check and modify in the at as needed. all of the records having no dates in the 008 field, or either an abstract that is too long or not having an abstract at all, are output to the rejects.mrc file. lists of these records are found in the three corresponding .txt files – lgabstracts.txt, no008dates.txt, and noabstracts.txt. the lists of records in the no7xxs.txt and the nomatch520s.txt files are used as alerts for records that may need reviewed, but the records are not considered rejects. the rest of the records are output to the good.mrc file. the file of “good” marc records can then be re-analyzed and validated again in marcedit if desired, and then used for batch-loading into a local catalog. conclusions while this paper presents a very specific example for batch-editing at-generated marc records, the concepts are applicable to any large set of marc data, from ebook vendor records to collections of mods records (or those conforming to other standard descriptive metadata schemas) that can be converted to marc and vice versa using appropriate crosswalks. the key idea is that the use of some minimal knowledge of regular expression syntax, and possibly some scripting language – be it python, xslt, javascript, ruby, or any other language – would prove beneficial to any cataloger. from the perspective of a traditional marc cataloger who has had the opportunity to learn a little bit of programming on the side, it is now hard to imagine a world of cataloging that does not incorporate some of the methods and tools that have been described. furthermore, the more catalogers are able to speak the language of coding, the better they can communicate with the technologists who have the deeper skills to develop the systems that represent the bibliographic data catalogers hold so dear. catalogers who truly understand the long-standing rules and standards of traditional marc cataloging will be able to contribute this knowledge to the larger objective to bring bibliographic data into the technological capabilities of the 21st century. the increase in collaborations between catalogers and coders, as well as the increase in catalogers who are coders, is proving to be the next logical direction for the field of bibliographic description and access. endnotes [1]. thomale, jason. 2010 sept. 21. “interpreting marc: where’s the bibliographic data?” code4lib journal [internet], issue 11. [cited 2013 march 7]. available from: http://journal.code4lib.org/articles/3832 [2]. curatecamp: catalogers + coders; 2011 oct. 30; baltimore, md [internet]. [cited 2013 march 7]. available from: http://curatecamp.org/content/welcome-curatecamp-catalogers-coders [3]. catcode wiki [internet]. [cited 2013 march 7]. available from: http://catcode.pbworks.com/w/page/49328692/welcome%20to%20catcode%21 [4]. mcdanold, shana. 2012 jan. 3. codeyear catalogers [internet, blog post]. [cited 2013 march 7]. available from: http://slmcdanold.blogspot.com/2012/01/codeyear-catalogers.html [5]. library of congress. 2013 feb. 25. bibliographic framework transition initiative [internet]. [cited 2013 march 7]. available from: http://www.loc.gov/marc/transition/ [6]. heath, tom (site administrator). linked data [internet]. [cited 2013 march 7]. available from: http://linkeddata.org/ [7]. reese, terry. marcedit [internet]. [cited 2013 march 7]. available from: http://marcedit.reeset.net/ [8]. klug, brandy. 2010 march 22. “wrangling electronic resources: a few good tools.” code4lib journal [internet], issue 9. [cited 2013 march 7]. available from: http://journal.code4lib.org/articles/2634 [9]. msdn. microsoft .net regular expression language: quick reference [internet]. [cited 2013 march 7]. available from: http://msdn.microsoft.com/en-us/library/az24scfc.aspx [10]. python programming language [internet]. [cited 2013 march 7]. available from: http://www.python.org/ [11]. pymarc [internet]. [cited 2013 march 7]. available from: https://github.com/edsu/pymarc [12]. archivists’ toolkit [internet]. [cited 2013 march 7]. available from: http://www.archiviststoolkit.org/ [13]. audra. 2010 nov. 23. touchable archives : sharing marc from archivists’ toolkit [internet, blog post]. [cited 2013 march 7]. available from: http://librarchivist.wordpress.com/2010/11/23/sharing-marc-from-archivists-toolkit/ [14]. archivesspace [internet]. [cited 2013 march 7]. available from: http://www.archivesspace.org [15]. archivesspace: get involved [internet]. [cited 2013 march 7]. available from: http://www.archivesspace.org/get-involved/ [16]. frank, heidi p. 2013. archivists’ toolkit generated marc : a compilation of issues and resolutions [internet]. [cited 2013 march 7]. available from: https://github.com/hfrank71/at_marc/wiki/at-documentation [17]. frank, heidi p. 2012. archivists’ toolkit documentation: guide on entering collection dates in the at [internet]. [cited 2013 march 7]. available from: https://github.com/hfrank71/at_marc/wiki/at-documentation [18]. python software foundation. 2013. python documentation, 7.4 difflib: helpers for computing deltas [internet]. [cited 2013 march 7]. available from: http://docs.python.org/2/library/difflib.html [19]. frank, heidi p. 2010. archivists’ toolkit documentation: guide on entering subject headings in the at [internet]. [cited 2013 march 7]. available from: https://github.com/hfrank71/at_marc/wiki/at-documentation [20]. frank, heidi p. 2009. archivists’ toolkit documentation: guide on entering name headings in the at [internet]. [cited 2013 march 7]. available from: https://github.com/hfrank71/at_marc/wiki/at-documentation [21]. frank, heidi p. 2013. marcedit task list for modifying archivists’ toolkit marc records [internet]. [cited 2013 march 7]. available from: https://github.com/hfrank71/at_marc/wiki/at-documentation [22]. frank, heidi p. python script for modifying archivists’ toolkit marc records [internet]. [cited 2013 march 7]. available from: https://github.com/hfrank71/at_marc/blob/master/atmarc.py about the author heidi frank (hf36@nyu.edu) is the electronic resources & special formats cataloger at new york university libraries. while she has primarily dealt with cataloging non-print materials, such as videos and cd/dvd-roms, she also works on metadata projects involving the mappings of marc to ead and mods, marc data analysis for ebooks, the development of workflows to assist in automating the cataloging process, and resolving display and navigation issues found in the library’s aleph and primo front-end interfaces. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – editorial: looking to the past to find the future mission editorial committee process and structure code4lib issue 41, 2018-08-09 editorial: looking to the past to find the future i reflect on my 10+ year tenure with the code4lib journal. ponder the work of our editors and authors. and come out the other side ready for 10 more years. i’m pleased to present the 41st issue of the code4lib journal, which is packed with useful information that i am sure will help you at your libraries. the editorial introduction for each issue of the code4lib journal is an opportunity to reflect on libraries, coding, and the journal. it’s an opportunity to step back and think about where we have been, where we are now, and where we are going. recent internal discussions have gotten me thinking about my tenure at the journal. i have been a part of the code4lib journal since the beginning, for more than 10 years, longer than i have had any one position. longer than about anything else that i have done. i have watched it grow from infancy, when jonathan rochkind put a call out to volunteers, through 346(!) published articles, 34 different editors, a special issue on diversity, and much more. i have been able to watch the journal mature. it has been a gratifying experience. but it isn’t just the journal, i’ve gotten to know my fellow editors and the authors – well, at least through email – and i’ve gotten to see them develop. for example, one of the authors in our current issue, kelley mcgrath, was also an author in our very first issue – she also served as a editor for awhile. in the current issue, she writes about “getting more out of marc with primo” with lesley lowery, a first-time author. it’s rewarding to see returning authors, like kelley, continue to work with the journal. for this issue, denis galvin and mang sun, who first wrote in our 13th issue, teamed up with their colleague, hanjun lee, to write about room reservation systems. suzanna conrad contributed to issue 27 and, for the current issue, helps us with “spinning communication to get people excited about technological change.” bringing with him two new authors, véronique lecat and veronica ramshaw) for “wms, apis and libguides: building a better database a-z list” is returning author thomas hodge. this issue also include a veritable super group made up of code4lib journal alums margaret mellinger (issue 2), hui zhang (36), and ryan wick (editorial committee) who are joined by new contributors steve van tuyl, josh gum, gregorio luis ramirez, and brandon straley to perform “are we still working on this? a meta-retrospective of a digital repository migration in the form of a classic greek tragedy.” one of the things that i really enjoy about serving on the editorial committee of this journal is that it exposes me to topics i wouldn’t come across in my regular job, such as returning author michael bennett’s article on high efficiency video coding. it is also exciting to work with authors that we haven’t worked with before. particularly when they can help you understand a topic better, like charlie harper does in “machine learning and the library or: how i learned to stop worrying and love my robot overlords.” pretty much everything that i know about digital repositories, i learned by editing and reading articles in the code4lib journal. that trend continues with three articles from from authors who are new to the journal. lisa lorenzo taught me about “using xml schema with embedded schematron rules for mods quality control in a digital repository.“ i learned about “extending and adapting metadata audit tools for mountain west digital library members” from teresa hebron. and saskia van bergen and lucas van schaik walked me through “copyright and access restrictions–providing access to the digital collections of leiden university libraries with conditional access rights.” finally, i learned how the nypl manages their web site designs in the article, “adaptation: the continuing evolution of the new york public library’s digital design system,” by jennifer l. anderson & edwin guzman. working with authors, returning and new, is often the most rewarding aspect of being part of the editorial committee. it is also time-consuming and difficult work, as we all want to make sure that we are putting out the best product that we can and that the articles will be useful to people in accomplishing their goals. the patience and fortitude that the authors and editors put into every issue never fails to impress me. it’s invisible for most people, but we couldn’t put out an issue without the hard work of everyone involved. i’ve had the pleasure of getting to meet several of the authors that i have worked with and talk with them about their experiences publishing in the code4lib journal. it’s usually a positive review, even when the article was challenging. the positive review may be the result of hindsight softening the edges or just the author being too polite to complain, but i find it encouraging to hear that they had fruitful experience working with me and the rest of the committee. it has always been a goal of the journal to lower the barriers to publication. to get useful information out to the people who need it, when they need it. judging from the frequency that i see articles in the code4lib journal cited in other work, i think we have been successful at getting people the information they need. it makes me proud when i see one of the articles from the code4lib journal cited in an article or presentation. when someone tells me that something that they learned in the code4lib journal made their work or their life even a little bit easier, i know that all the hard work that the authors and our editors do is worth it. a few days ago, my wife asked me how the journal was doing. i hesitated. when you are caught up in the work of putting together an issue, it can seem precarious or even unsustainable. but now that it is ready to launch, i feel much more confident about the future of the journal. looking back over the 10+ years that i have worked on the journal, there have been many challenges – and there still are. but the energy and commitment of my fellow editors and our authors shows me that the code4lib journal is valued and, as long as there is a need, the journal will continue to “foster community and share information among those interested in the intersection of libraries, technology, and the future.” thanks, rp subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – distributed version control and library metadata mission editorial committee process and structure code4lib issue 3, 2008-06-23 distributed version control and library metadata distributed version control systems (dvcss) are effective tools for managing source code and other artifacts produced by software projects with multiple contributors. this article describes dvcss and compares them with traditional centralized version control systems, then describes extending the dvcs model to improve the exchange of library metadata. by galen m. charlton introduction libraries have a long history of sharing their metadata, from the distribution of catalog cards by the library of congress (lc) to the development of marc to the sharing of records via bibliographic record services such as oclc. however, the flow of bibliographic information is usually top-down. to give a u.s.-centric example, a book may be catalogued by lc or a bibco library [1], contributed to oclc, perhaps updated by a few catalogers in oclc (but only if they have the privilege to do so), and then downloaded into an individual library’s ils or integrated library silo. from there, the record may be sent out for authority control (drawing from a central headings database maintained by the vendor) and edited by a library cataloger. next, the cataloger may improve the local copy of the record by adding a table of contents or better subject headings. however, the improved record can then languish in an individual library’s ils. it often takes a special effort to contribute a revised record back to a bibliographic utility’s database; contributing an entirely new record may be easy, but making a change to an existing record is often less so. if the library runs an open z39.50 server, a few other libraries may download the record, but most libraries will not see the improvement. even if the record is improved in oclc’s database, that improvement may not make its way down to other local catalogs as individual libraries may not subscribe to oclc’s change notification service or may not have the tools with which to load the improvement without losing their local changes. software development projects, particularly ones with a large number of contributors, have similar issues. a programmer will usually work on a local copy of a project’s source code. once she has completed a new feature or bugfix, the change has to get back into the main project, and the programmer has to be able to retrieve the work that others did in the meantime. in a well-managed software project, this process is mediated by a version control system (or vcs). this article will describe a particular kind of vcs, namely a distributed version control system (or dvcs), give some examples of how a dvcs can be used in a software project, then propose how a dvcs can be used for the global, collective project of creating and maintaining library metadata. version control systems broadly speaking, a version control system keeps track of a set of objects and the history of changes to these objects. in the case of a software project, the objects are typically files containing source code, testing scripts, documentation, external software libraries, and so on. a traditional version control system such as cvs or its inheritor subversion is based on a central repository. if you want to work on a project that uses cvs, you must gain access to the repository and identify the current version of the files maintained in that repository. you then check out the files of interest into a local working directory and start hacking away. once you’ve completed your changes, you must commit or check the revised files back into the cvs repository for the change to be recorded. cvs keeps a log of all of the history of changes and lets you generate diffs (comparisons of different versions of a file), retrieve older versions, or undo changes. however, once you start adding more programmers, the workflow can get more complicated. if two programmers need to work on the same file, several things can happen. with some version control systems, a programmer can get an exclusive lock on a file, forcing other developers to wait for the first programmer to finish a change and unlock the file before they can commit their work. cvs supports file locks, but by default it will not use locks. instead, if two developers try to commit a change to the same file, cvs will try to merge the second developer’s changes with the first changes. if the automatic merge fails, it will require the second developer to manually fix the conflicts. but where two or more programmers gather, a quality control process necessarily arises. for a large project with multiple developers, having a central repository means that whatever gets committed in the main line really ought to work, or at least not break the build. in practice, for a large open source project, two classes of programmers emerge – committers who are considered trustworthy enough to be given rights to commit their changes directly to the repository and everybody else. a non-committer who wants to contribute typically submits patches or diffs representing their changes to the committers or project mailing list. if the change is good, a committer commits it. send enough good patches, and a non-committer can be granted the keys to the kingdom and become a committer. thus, a traditional, centralized version control system by its very nature encourages certain restrictions on the social aspects of programming. while peer review of some kind is necessarily a part of quality software development process, forcing the existence of a class of committers can sometimes lead to problems. if a project has a small number of committers, there can be a bottleneck in pushing up work done by other contributors. sometimes the wrong person is promoted to committer status; sometimes good programmers are passed over for bad reasons; occasionally disputes about commit access lead to the foundering or forking of an open source project. the notion of commit access is similar to certain kinds of gatekeeping involving library metadata. for example, while any full oclc member can always add a new record or add various fields to a master record if not already present [2], fixing a mistake in a record originally contributed by lc is generally not possible; instead a library must submit a request to have a record corrected. to be able to directly correct various marc fields in a master record in oclc, a library must qualify for oclc enhance or conser status [3]. to be able to contribute authority records directly to lc’s authority file, naco or saco status [4] is required. an unprivileged cataloger will sometimes decide that it is quicker to create a new, duplicate record in the oclc database than to edit an existing record. while this practice is not recommended, the relatively centralized and hierarchical approach to metadata quality control leads to this temptation. distributed version control systems a distributed version control system, such as darcs, git, mercurial, or bazaar [5], does not depend on a central repository of changes. instead, each contributor has her own repository, into which she will commit her changes. when she is ready to share her work, other contributors can pull changes from her repository or she can push changes to another’s repository or send patch files generated from her repository. to give a concrete and somewhat simplified example of what the workflow might look like, consider the koha ils project, which uses git for version control [6]. to start contributing, a programmer (or web designer, or documenter, or user) would clone the koha git repository. assuming that she is working from a unix shell and has git installed, this would be accomplished by: $ git clone git://git.koha.org/pub/scm/koha.git koha this will create a new, local git repository in the koha subdirectory that contains a copy of the public git repository for koha (located at git://git.koha.org/pub/scm/koha.git). from there, she can change to the koha directory, run the koha installation process if necessary, and start hacking away. $ cd koha $ $insert_favorite_editor_here this_and_that once some work is ready to check in, she can commit it like this: $ git add a_new_file   # tell git about a new or updated file $ git commit -a        # tell git to commit all updated files upon running git commit, the programmer will be prompted to enter a commit message describing the changes. the changes are saved in the programmer’s local git repository; unless she chooses to share her repository with others, nobody else can see them yet. for certain situations, that could be all there is to the workflow – a distributed version control system can be used to easily copy the revision history of a software project and then simply manage local changes. this would be analogous to the common copy cataloging practice of downloading bibliographic records from a provider and making changes in the local ils without contributing the changes back to a central system. however, to stay in sync with changes made by others, the programmer could periodically bring in changes that were added to the central repository from which she originally cloned from the project. in git, this might be done like this: $ git pull or $ git fetch $ git rebase origin either command will query the source repository for changes made since the last clone or fetch and merge those changes with the local work done by the contributor. a merge is a very important operation in a version control system and one that a dvcs in particular needs to do efficiently and intelligently as possible. for example, suppose the programmer commits a change to an (imaginary) file in the koha project called adams.pl. when the programmer pulls the latest changes from the public repository, if one of those changes also involves adams.pl, git will attempt to combine both the local changes and the remote changes. for source code files, this is normally done via a three-way merge. a three-way merge works like this. let’s call the version of adams.pl at the time that the programmer made her clone adams.plorig. let’s call the version after she commits her change adams.pla, and the version committed by somebody else and pushed to the public repository adams.plb. adams.plorig adams.pla adams.plb #!/usr/bin/perl -w # answer a question. # what question? # what else? print "41\n"; #!/usr/bin/perl -w # answer a question. # what question? # what else? print "42\n"; #!/usr/bin/perl -w use strict; # answer a question. # what question? # what else? print "41\n"; the merge process considers the differences between the original ancestor version and the two variations: adams.plorig to adams.pla adams.plorig to adams.plb diff --git a/adams.pl b/adams.pl index 59446bb..0392661 100644 -- a/adams.pl +++ b/adams.pl @@ -4,4 +4,4 @@  # what question?  # what else? -print "41\n"; +print "42\n"; diff --git a/adams.pl b/adams.pl index 59446bb..6784430 100644 -- a/adams.pl +++ b/adams.pl @@ -1,5 +1,7 @@ #!/usr/bin/perl -w+use strict; +  # answer a question.  # what question?  # what else? after applying the programmer's change (adams.pla) and the remote change (adams.plb), the end result will be a merged version, automatically made both strict and accurate: #!/usr/bin/perl -w use strict; # answer a question. # what question? # what else? print "42\n"; most dvcss will try to resolve merges automatically, but in some cases, a merge conflict may arise which the programmer must then resolve manually. for example, if adams.plb had also changed the answer to "43", a three-way merge would not be able to automatically determine if "42" or "43" was correct and would leave it to the programmer to tell git what the correct result should be. the programmer is not restricted to pulling changes from the repository she originally cloned from she can pull from the repository of any other koha developer who makes it publicly available. for example, if someone else is working on an automatic library police dispatch feature for koha, the programmer could pull from that repository to get a sneak peek at the changes: $ git pull git://git.librarypolice.com/pub/scm/koha.git conversely, others could pull changes from her repository. to go further and submit her changes to the public repository so that they are added to the public or consensus version of koha, she could tell git to prepare and e-mail patch files. $ git format-patch origin $ git send-email *.patch these two commands generate a set of patch files, then email them to an address specified interactively in this case, to the koha patches list, or to another koha developer for a review. if, after the koha qa and release managers review and approve the patch, nobody in the koha developer community objects, it will generally be pushed to the public repository and thus made available for others to download. advantages of distributed version control systems the fact that each dvcs repository can store the complete history of a project provides several advantages. these advantages include independence for local developers (while still maintaining a relationship with the larger project), a distributed backup of the central project, and de-centralized management of user permissions. first, each developer can work independently without having to rely on constant network access to a central repository. if a programmer is working on a large feature, she can control how and when to merge changes made in the central repository in the meantime into her local repository. if the feature is speculative, she can make her local repository public and ask other contributors to look at her work. if the feature concerns a distinct module of the central project, she can readily use her local repository to create a subproject, pulling changes from the repositories of other developers working on the subproject. once the subproject is in a completed state, she can then ask the maintainer of the central repository to pull in all of the changes related to the subproject. second, each programmer's repository is effectively part of a distributed backup network for the project as a whole. if the public repository is lost, all of the changes can be recovered from the local repositories of the active contributors. finally, decisions about who to give commit access to a single central repository can be mostly avoided. of course, the decisions about which patches to accept into the public repository cannot be avoided, but this applies to any software project. every contributor has commit access to her own local repository, and submits changes to the consensus repository or to the maintainer of one of the project's modules. the maintainers of the consensus version can focus on whether particular changes are generally useful instead of who qualifies for committer status. if the programmer's work is of only limited or local interest, she can still continue to maintain her local changes while tracking the public changes. distribution of version control does not necessarily mean fragmentation of a project. the linux kernel project, for which linus torvalds originally wrote git, consists of a wide-variety of subprojects (e.g., the networking subsystem), many of which are run semi-independently of one another. whenever a subproject is ready, its manager will ask linus to pull from their integration repository to his own repository, which is one of the primary repositories that is considered a canonical source of linux. the distribution of the repositories mirrors the distribution of responsibility and trust a driver patch by a new kernel hacker does not have to be personally vetted by linus, but instead can be checked by the person or group maintaining the affected subsystem [7]. distributed version control and library metadata use of a dvcs is a valuable tool for many software projects and could also be a valuable tool to support the exchange of library metadata. at the moment, library metadata practice is in a state of flux. in addition to the development of a new content standard for bibliographic metadata, several trends are encouraging the decentralization of metadata creation and maintenance. for example, if the library of congress follows the recommendations of the working group on the future of bibliographic control, it would seem to be on a course to reduce its role in the production of original cataloging for english language monographs and has already ceased creating new series authority records [8]. that will leave a gap to be picked up by somebody, and in the absence of proven techniques for automatically creating bibliographic metadata or repurposing publisher metadata, it will put pressure on library cataloging departments to produce more original cataloging. most ilss currently maintain separate silos of bibliographic data, including original cataloging, but not all libraries participate in central bibliographic utilities. on the other hand, some centralizing projects to eliminate ils metadata silos are underway, including oclc's development of worldcat local [9] and the internet archive's open library [10]. both projects offer models for sharing metadata from big centralized databases, but it is an open question whether such projects can achieve complete coverage of all library materials and whether libraries can or should collectively tie their fortunes to a small number of central providers. in a distributed model, libraries would continue to maintain separate repositories of their metadata. however, unlike most current ilss, which may offer z39.50 services but are not designed for the continual update of records and the transmission of improvements, each repository would be able to: maintain a complete change history of each metadata record. generate diffs and patches. clone records from another repository. pull changes from records in other selected repositories. allow other repositories to pull changes. perform the equivalent of three-way merges. this approach assumes that there is a core set of records that multiple libraries are interested in and need to share changes. this approach also assumes that a record may change frequently. this is already the case under current marc and aacr2-based practices. the first library to catalog an anthology may only create a minimal marc record with basic descriptive cataloging. another library may add a table of contents, another may add additional subject headings, and a third may fully analyze it. the introduction of ideas from frbr will likely only increase the changes; if you are cataloging a new translation, you may not be able to identify the existing work that it is an expression of it may take some time for the relationship to become apparent. a workflow for copy-cataloging in a distributed environment might look like this: the cataloger selects and clones a record. if the item is an academic monograph, it might be cloned from the repository of the nearest university; if it is a dvd, it might be cloned from a public library. the cataloger makes a local change such as adding an acquisitions note. periodically, the cataloger pulls changes from a table of contents provider. periodically the library's local repository pulls changes from the original repository. for example, one day the record in question may get an additional subject heading contributed by another cataloger. the cataloger notices a typo and corrects it, then pushes the change back to the source repository. a distributed model for maintaining version control of metadata and sharing changes would have several advantages: economies of distribution could be better realized. currently, a bibliographic record may initially lack a table of contents; before one is added to the version maintained by a bibliographic utility, a number of libraries may add one to their local copy without being able to readily contribute their improvement. a model that encourages periodically bringing in improvements while preserving local changes would improve the quality of metadata stored in each library's repository, and allow all libraries in the network to benefit from changes. contributions from smaller cataloging departments could be more readily integrated into the bibliographic consensus. the dvcs platform could support the creation specialized repositories. for example, a consortium of libraries could set up a repository of records enriched by the addition of spanish-language subject headings. full change history for each bibliographic record would be available a feature lacking from many ilss. there would also be some challenges: issues surrounding the ownership of bibliographic records would need to be resolved. if we were to view library metadata as a shared, perhaps global, project the efficient exchange of records and patches would require that the exchange not be constrained by overly restrictive licensing terms. for example, if a library receives marc 505 fields from a table of contents service, the terms of their agreement with the enhanced content provider may not allow them to redistribute the tocs. large central repositories would continue to serve a valuable function, acting as both primary loci for cloning records and as targets for large-scale data-mining and record improvement projects. however, those repositories would have to be supported, both with staff and technical resources. the user interface for processing large numbers of updates and merges would have to be very efficient and easy to use. towards a library metadata dvcs many components required to build a dvcs for library metadata already exist: several dvcs platforms could be built upon for a prototype and most library metadata formats can be expressed as xml, for which toolkits for diffs and merges already exist. a cataloging editor that uses a dvcs repository would need to be able to query multiple target repositories, but existing search standards and metasearch tools could handle that. there are some cases where a library dvcs should behave differently from a traditional dvcs. for example, it may be impractical for a library to clone the entire metadata history from a large repository, so it will be necessary to be able to clone on a record-by-record level. bibliographic records typically contain both metadata of general instance and metadata that is useful only to one library, so a dvcs may need to be able to keep track of purely local changes. for example, it may not be useful to send marc 856 fields containing urls referencing a library's proxy server to other repositories. in order to take advantage of dvcss for library metadata, the process of pulling and pushing changes will need to be automated. once tools for managing diffs and merges of library metadata records are created, automating a periodic update of changes from other repositories is in principle quite simple for example, one might back up a git repository by writing a script to periodically fetch changes into the backup repository. if this notion is applied to library metadata, automatic merges have implications for metadata quality control. no automatic merge algorithm is perfect, so catalogers would have to resolve merge conflicts. even in the absence of merge conflicts, a cataloger may wish to monitor the changes to verify that they are correct, a task that could prove daunting if distributed version control becomes popular and the frequency of tweaks to records increases. consequently, quality control may end up depending on the creation of better automatic tools for verifying the correct structure of metadata records, but it may also depend on catalogers distributing quality control. in other words, if a library pulls two hundred changes to its records a day, the cataloger may only be able to examine ten of those changes in great detail and must trust that the rest are either correct or will be corrected later. however, a dvcs enables each library to choose which repositories (or institutions, or even individual catalogers) will be used as a source for pulling record updates automatically. utilizing these as trusted repositories could help facilitate the distribution of of quality control across institutions. conclusion distributed version control for library metadata offers many advantages over the current approach where individual ils silos at multiple institutions pull records from central bibliographic utilities, often manually adding local enhancements are never shared with others. distributed version control tools promote the propagation of improvements to metadata records back to central repositories and among metadata users. libraries of any size can take ownership of improving a portion of bibliographic metadata and sharing those improvements freely with others. record enhancements can come from the bottom up, not just the top down. each local metadata repository will have a complete history of changes to its records, which is a capability that not all bibliographic record editors and databases currently have. each local metadata repository is automatically part of a distributed backup network for metadata records lockss [11] can apply to metadata as well as electronic publications and scholarly data. while any sensible model for metadata creation has an important place for large central repositories, a distributed model is not completely dependent on any one entity. if a large contributor of metadata decreases its contributions, or if a central repository experiences difficulty, a distributed model allows work to be more easily shifted to other contributors and other core repositories. with the many possible advantages of a dvcs model, it makes sense for institutions to begin experimenting with the creation of distributed repositories containing library metadata records. repositories based on the dvcs model can promote metadata sharing beyond the level attained by current cooperative cataloging models, thereby improving the quality of metadata available to aid patrons and further distributing the cost of cataloging over many institutions. resources open source library software projects using dvcs indexdata projects, including yaz and zebra (git, http://git.indexdata.com) koha (git, http://git.koha.org) pymarc, a python marc library (bazaar, http://fruct.us/bzr/pymarc) notes [1] bibco, the monographic bibliographic record program of the program for cooperative cataloging (pcc) of the library of congress: http://www.loc.gov/catdir/pcc/bibco/. [2] "database enrichment" section of the "quality assurance" section of oclc's "bibliographic formats and standards" guide. http://www.oclc.org/bibformats/en/quality/default.shtm#databaseenrichment. viewed 13 june 2008. [3] "enhance training outline." http://www.oclc.org/support/training/worldcat/enhanceoutline/. viewed 25 april 2008. [4] naco is the name authority cooperative program of the pcc (http://www.loc.gov/catdir/pcc/naco/) and saco is the equivalent program for subject authority records (http://www.loc.gov/catdir/pcc/saco/). [5] dvcs bazaar: http://bazaar-vcs.org darcs: http://darcs.net git: http://git.or.cz/ mercurial: http://www.selenic.com/mercurial/ [6] "version control for koha using git." http://wiki.koha.org/doku.php?id=en:development:git_usage. viewed 25 april 2008. [7] "re: clarification on git, central repositories and commit access lists." posting to kde-core-devel mailing list by linus torvalds. http://lwn.net/articles/246381/. viewed 25 april 2008. [8] "on the record: report of the library of congress working group on the future of bibliographic control." http://www.loc.gov/bibliographic-future/news/lcwg-ontherecord-jan08-final.pdf. viewed 15 june 2008. [9] "overview [oclc worldcat local]." http://www.oclc.org/worldcatlocal/overview/. viewed 17 june 2008. [10] "about us (open library)." http://openlibrary.org/about. view 17 june 2008. [11] "lots of copies keep stuff safe". see http://www.lockss.org about the author galen charlton is a developer at liblime, working on the koha open source ils project. e-mail: galen.charlton@liblime.com url: http://www.galencharlton.com/blog/ subscribe to comments: for this article | for all articles 4 responses to "distributed version control and library metadata" please leave a response below: jon gorman, 2008-06-30 love the idea and i’ve been arguing for something similar for a while. (i’d love a change history for our records). one other possible issue that popped into my head this weekend is the possibility two different records get created for the same resource at about the same time in two different libraries. say susan at x makes a record for “how to conquer the world in thirty days; by khan”. bill at y got the book around the same week or so and susan hadn’t actually submitted her file (1234.mrc) upstream yet. bill doesn’t find any existing records and creates a new one as well (24444.mrc). now we start getting fragmentation. the way most dvcs work on is based off filename i believe. this could be an issue if each institution assigns a record for the same source with different conventions. we could try to use an identifier with ways of correcting for conflicts with identical isbns in different books, etc. this makes me a little worried we’d have to do some custom tweaking of a dvcs, which might be nightmarish. i suppose we’ll probably need to create custom searches and indexes anyhow from the system, but we can reuse some tools for that. jakub narebski, 2008-07-20 @jon gorman: git does rename detection based on similarity score, so it should not matter whether one person caled file ‘foo’ and second ‘bar’ when creating almost the same file: they should merge correctly. jon gorman, 2008-07-21 ah, interesting. i haven’t done as much with the distributed version control systems as the…well, traditional ones i guess you would call them. obviously i need to play around with them a little more. idle hands are the devil’s plaything « thesecretmirror.com, 2011-06-07 […] (ahem, vote for mine, if you're so inclined): building on galen charlton's investigations into distributed version control systems for metadata management, i offer a prototype system for managing archival finding aids in ead (encoded archival […] leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – alma enumerator: automating repetitive cataloging tasks with python mission editorial committee process and structure code4lib issue 42, 2018-11-08 alma enumerator: automating repetitive cataloging tasks with python in june 2016, the warburg college library migrated to a new integrated library system, alma. in the process, we lost the enumeration and chronology data for roughly 79,000 print serial item records. re-entering all this data by hand seemed an unthinkable task. fortunately, the information was recorded as free text in each item’s description field. by using python, alma’s api and much trial and error, the wartburg college library was able to parse the serial item descriptions into enumeration and chronology data that was uploaded back into alma. this paper discusses the design and feasibility considerations addressed in trying to solve this problem, the complications encountered during development, and the highlights and shortcomings of the collection of python scripts that became alma enumerator. by nausicaa rose introduction in june 2016 vogel library at wartburg college migrated its integrated library system from millennium to alma. for the most part, the migration went smoothly. most records transferring over correctly, but as with any large-scale data migration various problems arose. among the thorniest of these issues was the loss of enumeration and chronology data for roughly 79,000 print serial items housed in millennium’s serials module. this meant vogel library’s new discovery layer, primo, couldn’t filter serials by volume number or year. fortunately, the enumeration and chronology data was preserved in each item’s free text description field in strings like “v.2 no.10 sum 2010.” the problem lay in how to best extract this data and re-enter it into the appropriate fields in each item record. considerations as vogel library’s lone technical services staff member, the responsibility of updating these records largely fell to me. given the large number of records that needed to be updated, manually editing each record was not a realistic option, not even with the help of student employees. fortunately, alma has a well-documented rest api that made large-scale programmatic updates possible. alma stores information for physical materials as bibliographic records, with one or more associated holding records, which in turn have associated item records. in each item record, alma stores volume, issue, and other enumerative numbers in enumeration_a through enumeration_h fields, and stores date information in chronology_i through chronology_m fields, as shown in this xml snippet: <enumeration_a>44</enumeration_a> <enumeration_b>3</enumeration_b> <enumeration_c></enumeration_c> <enumeration_d></enumeration_d> <enumeration_e></enumeration_e> <enumeration_f></enumeration_f> <enumeration_g></enumeration_g> <enumeration_h></enumeration_h> <chronology_i>2014</chronology_i> <chronology_j>11</chronology_j> <chronology_k>12</chronology_k> <chronology_l></chronology_l> <chronology_m></chronology_m> <description>v.44 no.3 nov 12 2014</description> to correctly populate these fields i needed a way to programmatically extract the enumeration and chronology data and place it in the correct item record fields. this was more easily said than done. while i had some prior programming experience, i had minimal experience working with rest apis, had not done any complex text processing, and had not worked with xml, which alma’s api used for data exchange. there was a reasonable possibility that i’d be unable to successfully write a program that could update item records as required. i decided to give myself a couple weeks. if i couldn’t build something that was at least promising in that time, i’d give up and start looking for another solution. if the project turned out to be beyond my skills, being two weeks behind on updating 79,000 records wouldn’t matter much. on the other hand, if i could get something working in that time i might be able to have all, or at least most, of the item records updated before classes started in late august. to begin the project i decided to use python. i had some prior experience with python and it had a reputation as an easy-to-use and easy-to-learn programming language with a wide variety of useful libraries. since library staff did not have administrative privileges and i was unsure of what libraries i would need, i chose to use winpython, a portable distribution of python that didn’t require administrative privileges to install and provided a large number of libraries pre-installed. a similar distribution i considered was anaconda, but at the time i did not realize it could be installed without administrative privileges. design and redesign the initial design involved a two-stage process that remains the basis of how the software that came to be called alma enumerator works. the first stage was to download all the item records connected to a bibliographic record in alma, parse the description field in each record, and save the resulting data to a file that can be checked for errors. the second stage was to update the item records in alma with the data from the saved file. to accomplish these tasks i identified what libraries i would need. to handle requests to the alma api, i used the requests library which simplifies the process of handling http requests over python’s default libraries. to process the xml records i used beautiful soup, a library designed for parsing and manipulating html and xml documents. after deciding on a basic structure and the needed python libraries, i developed the alma enumerator primarily via exploration rather than conscious design. while i did make design decisions throughout the process, they were more often driven by trial and error than through planning and careful consideration. this led to frequent redesigns, especially for the description parsing algorithm which went through three major iterations. the first iteration of the description parsing algorithm was a single regular expression. i assessed the item descriptions for a small number of bibliographic records. each description used a set pattern. as long as the pattern held, a regular expression would be sufficient to extract the desired fields. this worked quite well for the first few titles that i had assessed. unfortunately, after i began working on titles beyond my initial sample, i began to discover that there was a much wider variety of description formats than i had anticipated. i began to expand the application’s regular expression to handle these new variations, but this approach soon proved unworkable. the vogel library catalog had descriptions that ranged from the simple “v 149” to the complex “v.302 no.2-3 jan 11 2016 – jan 18 2016” along with outliers like “v.85 no.6 2009 international focus issue” and typos like “\v.523 no.7560 jul 16 2015”[1]. there was no regular expression that i could write and maintain that would handle the level of variation in item descriptions, so i had to find another approach. my second approach was to split each description into a list of tokens to sanitize the descriptions, removing undesirable punctuation and words like “index” or “international focus issue” that did not provide enumeration or chronology information. the majority of descriptions i’d encountered up to that point still fell within a fairly small number of patterns that varied primarily in the number of tokens in the description. to handle different descriptions, i wrote a series of if-else statements that first checked the length of the token list then had nested if-else clauses to handle different patterns of the same length. this approach worked, but was inflexible. for each new description format, i had to add a new series of if-else statements. this was the approach i used for most of the summer of 2016 to update item records, but by august i decided there had to be a better way. # ['v.65', 'no.3', 'may', '2012', '-', 'jun', '2012'] if len(info) == 7: if has_digitsp.match(info[2]) == none and has_digitsp.match(info[5]) == none and info[4] == '-': print('seven. matched.') item_info['enumeration_b'] = snarf_numerals(info[1]) if info[3] == info[6]: item_info['chronology_i'] = snarf_numerals(info[6]) else: item_info['chronology_i'] = '/'.join([snarf_numerals(info[3]), snarf_numerals(info[6])]) item_info['chronology_j'] = '/'.join([info[2], info[5]]) else: print('{} appears to be irregular. please correct by hand.'.format(item)) item_info['error'] = item listing 1. a sample of a small portion of the second approach to description parsing in alma enumerator. i decided to rewrite the parsing function once again. the new approach split each description into tokens and sanitized them, as in the previous approach. instead of writing a custom series of if-else statements for each description pattern, however, i took a more generalized approach. now the function iterated over the list and attempted to identify and collect years and month/season words like ‘may” or “winter,” and pagination data. it also flips a boolean flag if a description includes the day of publication in its chronological data. after this initial parsing, the remaining tokens were apportioned based on the number of remaining tokens and some logic that attempts to determine if excess tokens are days of the month. although i did need to make some tweaks to handle various unusual descriptions, this new algorithm was generally able to handle new description formats without requiring a new series of if-else statements. # go through each item in the list for i in info: # if it's a hyphen, ampersand, or slash, add it to the deletion list. if r_exp.match(i): delete_me.append(i) # if it's not a numeric value, check to see if it's a pagination marker, # otherwise, add it to the month/season list. in either case, mark it # for deletion. elif not has_digitsp.match(i): if has_pagination(i): pp = snarf_numerals(info[info.index(i) + 1]) info[info.index(i) + 1] = pp pages.append(pp) delete_me.append(i) delete_me.append(pp) else: for key in date_patterns: if key.match(i): mo_season.append(i) # check the item after this one. if last_index > info.index(i): look_ahead = info[info.index(i) + 1] # if it looks like a numeric value and not like a year. mark it # as a chronology_k (day) value. if has_digitsp.match(look_ahead) and not is_yearp.match(look_ahead): has_chron_k = true break delete_me.append(i) # if it is a numeric value, sanitize it by running it through # snarf_numerals() else: info[info.index(i)] = snarf_numerals(i) i = snarf_numerals(i) # if it looks like a year, add it to the years list and the deletion # list. if is_yearp.match(i) and i not in pages: years.append(i) delete_me.append(i) # only month values should have leading zeros if anything left in this list # has a leading zero, it's probably something like an abbreviated year that # has been misidentified as not a year (ie 07 for 2007). treat it as an # error. if has_leading_zero(i): item_info = handle_record_error(item, item_info) listing 2. the portion of get_info_from_description that iterates over each token in a description and attempts to identify what enumeration or chronology field the token belongs in. this third approach remains the way alma enumerator parses descriptions. it transforms the item description, as the second approach did, to convert a description like ‘v.44 no.3 nov 12 2014 pp 205/298’ along with the item’s mms id into a python dictionary with the following structure: {'id': 999999999, 'enumeration_a': 44, 'enumeration_b': 3, 'chronology_i': 2014, 'chronology_j': 11, 'chronology_k': 12, 'pages': 205/298,} it also attempts to identify descriptions it can’t correctly parse. alma enumerator then writes the dictionaries created from item descriptions to a csv file and saves the errors it identifies to a separate file. to update the item records in alma, the program reads from the saved csv file, converting each line back into a python dictionary and then inserting the data from each key-value pair in the dictionary into the matching element of the item record. the code to handle requests to the alma api is relatively simple, although it did take a little thought to handle bibliographic records with multiple holding records, and to retrieve a full item list since alma’s api limits its responses to 100 items and wartburg had several print serial titles with more than 100 items. matching the two stage process, alma enumerator initially began as two separate programs: one that downloaded and parsed descriptions and another that updated the records based on the output of the first. however, there was enough overlapping code in each of these that i combined them into a single library called ae.py. during the early stages of development, i used alma enumerator mainly by running it directly from my ide or by loading it into a python repl and running its fetch and update functions from within the repl. however, as the project became more mature, i realized it might prove useful to others. i decided to provide interfaces for using the alma enumerator library: a set of command-line scripts that could accept arguments (i already had a couple primitive scripts that relied on edits to a settings document to change behavior) and to build a gui that could be used by non-technical library staff. for the command-line scripts, i initially used docopt to parse command line arguments. i found this library easier to wrap my head around than argparse, the recommended command-line argument parser in python’s standard library. docopt comes as part of winpython and anaconda, so i thought it would be reasonable to assume that it would be likely that others would be able to install the package. however, i had difficulty getting docopt to handle arguments the way i wanted. in march 2018, i rewrote the argument parsers for alma enumerator’s command-line scripts to use argparse. for the gui, i used tkinter, a gui toolkit that is part of python’s standard library. by the time i decided to build a gui i’d realized that if alma enumerator was used by other libraries, it would most likely be in situations similar to mine at wartburg: a windows (or perhaps apple) computer that wasn’t set up for non-technical users to use python or install new libraries. wanting to minimize the barriers for use in such a set up, i opted for the default gui library over the various other options available outside python’s standard library. figure 1. screenshot of the alma enumerator gui. analysis for my immediate goal of updating 79,000 item records, alma enumerator was a success. i didn’t manage to have the records for all the print serials updated by the start of fall semester 2016 as i had hoped, but i had made something that could update item records much faster than i could have done manually. as i refined alma enumerator over the summer, i’d processed a number of print serial titles, starting with smaller collections that would be easier to correct by hand if errors arose. once alma enumerator was reasonably reliable, i prioritized the titles that were current subscriptions followed by those that had the largest number of items, assuming these would be the titles most commonly used by students and faculty. i was able to complete the updates for these first two categories by the time classes began. i updated the remainder over the course of the fall semester. figure 2. screenshot showing primo’s drop-down menu for filtering print serial items by volume number. the figure demonstrates that the menu uses item description text when there is no data in an item record’s enumeration_a field. figure 3. screenshot showing another example of primo’s drop-down menu for filtering print serial items by volume number. the figure demonstrates how item records can be filtered by volume number when there is data in the record’s enumeration_a field. figure 4. screenshot of primo’s drop-down menu for filtering print serial items by year. the figure demonstrates that items cannot be filtered by year when there is no data in an item record’s chronology_i field. figure 5. screenshot of primo’s drop-down menu for filtering print serial items by year. the figure demonstrates how item records can be filtered by year when there is data in the record’s chronology_i field. although i consider alma enumerator a success, it is not without its shortcomings. due to the exploratory nature of much of the design process, the code tended to grow organically rather than following a clear structure. some functions, especially get_info_from_description, which parses item descriptions, are overly long and complex and could probably be split into smaller functions. the code is messy in places and variable names like rp, r_exp, and info are not exactly intuitive. i suspect the description parsing process could be further refined to be less convoluted. there were also trade offs made during the development process that make alma enumerator less efficient than it could be. early in the design process, my primary goal was to get something working as quickly as possible, so many early design decisions emphasized development efficiency over efficient operations. one of the places this is the most obvious is in the way alma enumerator updates item records. for each item record that needs to be updated alma enumerator downloads one existing item record, updates it, and uploads that updated record back to alma before repeating the process for each subsequent record. this requires two http requests per item record and each request is made in sequence as each item is processed in sequence. for bibliographic records with only a few items, this doesn’t matter much, but for titles with dozens or hundreds of items an update would take minutes to complete. in the case of nature which had 657 items, the time it took to update each record was approximately half an hour. one option to speed up this process would be to save the item record xml files downloaded during alma enumerator’s first stage. another would be to use multiple threads or processes to make the http requests so that the main thread can continue. another issue is the number of description patterns that alma enumerator can’t handle. some of these are outliers or outright errors like “100,02/04,1997/1998,v 100 no 2 feb 1997-v 101 no 2 april” or “v 67 #3 aug 986” that are better left to be handled manually. when these are correctly identified as errors, things are still working as intended. the more troubling class of problem descriptions are those that pass through without correctly being identified as errors and thus must be caught by manual inspection lest inaccurate data is added to an item record. some of the descriptions that pass through are outliers and errors as above, but others are descriptions like “v.302 no.21 & 22 may 23/30 2016” that alma enumerator should ideally be able to handle. to fix these issues would require further tinkering with the way the program parses item descriptions. i have also not written enough tests to make sure that alma enumerator is robust. the only part of the project that is tested at all is the get_info_from_description function and even there the testing is incomplete. if i had taken a more test-driven approach, i expect the development time it took to create alma enumerator may have been reduced at certain points where i became mired in trying to root out unexpected bugs. more thorough test coverage would also help to minimize the potentially large number of still unknown bugs that may lurk in the code. perhaps the most troubling shortcoming is that it is not all that easy to use alma enumerator despite my efforts to make it reasonably simple to set up and run. on and off over the past year and a half, i have been trying, with a colleague at a university that migrated to alma at the same time as wartburg, to get alma enumerator to work for his institution. it was not until early october, 2018, that we were able to get alma enumerator to work for this institution. part of the problem was due to a lack of testing, as he has encountered problems i did not at wartburg. the larger part of the problem was that i overfit my code to the way i needed it to work. for example, i had initially hard-coded a slash character as the separator for multiple numbers, a description like ‘v. 100-102’ would result in an enumeration_a value of 100/102. his library preferred to use a hyphen, which meant i had to rewrite several parts of the code to accommodate the possibility of different separator characters. there may be additional code that similarly hard codes other local idiosyncrasies. conclusion alma enumerator is far from perfect, but it did accomplish the job it was meant to do. the process of creating it has also helped me gain a deeper understanding of python and the programming process from design to coding, testing, and refactoring that i can apply to other projects and hopefully avoid some of the pitfalls that affected this project. it also helped me start thinking about other ways to apply automation to large-scale and tedious tasks. as for alma enumerator itself, the code lives on on github and has been released under the terms of the unlicense. i’m still actively, though minimally, maintaining it and dreaming of that day when i’ll have the time and energy to fix all its various flaws. notes [1] for a fuller picture of all the various descriptions encountered in the vogel library catalog, see https://github.com/wtee/alma_enumerator/blob/master/notes.md. about the author nausicaa l. rose is a metadata librarian at parks library at iowa state university. previously, she served as the technical services supervisor at vogel library at wartburg college. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – building bridges with logs: collaborative conversations about discovery across library departments mission editorial committee process and structure code4lib issue 32, 2016-04-25 building bridges with logs: collaborative conversations about discovery across library departments this article describes the use of discovery system search logs as a vehicle for encouraging constructive conversations across departments in an academic library. the project focused on bringing together systems and teaching librarians to evaluate the results of anonymized patron searches in order to improve communication across departments, as well as to identify opportunities for improvement to the discovery system itself. by jimmy ghaphery, emily owens, donna coghill, laura gariepy, megan hodge, thomas mcnulty, erin white[1] the efficacy of library discovery tools can be a contentious topic within libraries, and the dissimilar experiences and attentions of technical librarians who support the system, and teaching librarians who use the system daily with patrons, can hinder communication between these two essential groups. at vcu libraries in the fall of 2013, confidence in our discovery system, ex libris’ primo, was low, especially amongst our teaching librarians. even after ironing out some stability issues, there were lingering questions about the quality of the search results. seeking ways to pinpoint search relevancy issues and enhance communication between teaching librarians and it, a team of systems and teaching librarians agreed to jointly assess and discuss a random sample of actual search queries from primo. this paper describes that process, compares the initial perceptions of relevancy between each group of librarians, and reflects on the success of both improving communication across departments and the library discovery system itself. selected literature review the topic of librarian attitudes toward library discovery systems has been an area of recent research, and certainly one of frequent mention on listservs and at conferences. in exploring teaching librarian attitudes toward discovery systems, allen (2015) states that “librarians need to be comfortable with a resource in order to teach and advocate for it.” on the other side of the public/technical divide, gayhart (2013) notes that user­centered library­wide staff communication is critical for the success of technology departments and the products they support and calls for library it departments to “move from the traditional image of the gatekeeper of services, to a more inclusive image of a partner in success”. in his editorial about library discovery, demlow (2013) sums up one of the key barriers to librarian adoption: “in part, discovery systems induce discomfort for library professionals because the ways in which relevance is calculated are becoming more opaque” (p. 3). there is also substantial library literature surrounding discovery systems. two recent articles on selection of discovery systems provide literature reviews that include history, selection, and evaluation (deodata 2015, ellero 2013). ellero’s work is also noted for the evaluation method including “actual reference questions” (p. 324) that resulted in the library’s decision to not implement a discovery system. asher, duke, and wilson (2012) noted that their study comparing various discovery systems with google scholar and other library databases was new ground beyond the previous literature on selection and individual usability studies of various systems. ciccone and vickery (2015) use a very similar methodology to our study in randomly selecting user queries from search logs to compare two library discovery systems and google scholar. interestingly (and consistent with our findings), google scholar outperformed the discovery systems for topical searches. logs from systems have long provided rich data for understanding user behavior and thus offering direction for system improvement. jansen (2006) provides a strong review of search log analysis history with a detailed literature review across various disciplines including libraries. more recently, park and lee (2013) investigate institutional repository interfaces through log analysis, and in doing so also provide a substantive literature review. traditionally, however, logs have been the turf of systems staff and systems administrators (in many cases due to valid privacy concerns). this project pulled together systems librarians and teaching librarians as partners in sharing attitudes toward a sample of anonymized logs, with the goal of improving cross departmental communication against the backdrop of improving the discovery service. in reaching across a potential technical, non­technical divide this work leverages transaction logs in a novel manner as a discussion starter toward those goals. background vcu libraries first fielded a discovery solution (ex libris’ primo, with aleph as the underlying ils) as a “beta” service to patrons in october of 2011. by january 2012, discovery search became the primary option presented to users on the library homepage. we made the conscious decision to present users with a blended search of local holdings and articles, and not to offer “tabbed” searching options such as “catalog only.” in october of 2012, vcu libraries became the third institution in north america to move to production with ex libris’ alma unified management system. at that time we introduced a new primo instance which was fed by alma. following this period of rapid change, we experienced a string of primo stability issues in the fall of 2013. these issues included system slowness, incompleteness of results, and temporary outages. our teaching librarians felt the brunt of these problems when the system was used heavily during library instruction sessions. even after those issues were resolved, there were significant doubts about primo, particularly among teaching librarians. further, we no longer had the option of a traditional local catalog search since alma does not have a public catalog and we were still pursuing a blended, single­-search philosophy. all of these factors put additional pressure on the discovery service to perform optimally. systems and teaching librarians decided to collaborate on an evaluation project with the goals of comparing perceptions of system quality, increasing communication between departments, and identifying particularly egregious problems. the project to get the sample of queries, we pulled a report of all the searches that came from the primo search logs during our library’s statistical “typical week” in october 2013. the searches were decoupled from any identifying information such as ip address. from those 68,579 queries, we used a python script to filter and randomly select 200 search queries, and format the list of queries as a hyperlinked web document for easy review.[2] the first batch of randomly selected searches from the logs included few limits. after initial testing and meeting we identified areas for improving the selection. in particular we found that the sample included results that had been secondarily filtered by the user (facets, next page, etc), and were thus not a good gauge for measuring immediate search relevancy. in our subsequent evaluation processes, we limited the search logs to initial searches only: both advanced and simple searches were included, but only un­faceted, first pages. based on the time it took to rank each search, we also agreed that 100 searches was a sample sufficient for our qualitative purposes of sharing perspectives on primo’s relevancy. for the test run we had also developed and agreed upon a scale for determining what might constitute a “relevant” result. the test scale was: very poor (no relevant items, misleading results, known item not findable) below average (few relevant items, known item buried) average (some relevant items? known item on first page or accessible from reasonable use of facets) above average (topic search returned mostly relevant results, known item on top 5 results) excellent (topic search returned very relevant results? known item search retrieved item in top 2 results) in our initial meeting, group discussion about how to define relevancy proved useful in and of itself. for the subsequent evaluation, we agreed to use a less prescriptive rating scale (figure 1). while the scale was certainly open to interpretation, the previous definitions and collegial development of the scale gave us a reasonable agreed upon framework for future efforts at determining the search performance. we certainly found that defining relevancy for different user populations, based only on an anonymized search, is difficult. however, many of the searches spoke for themselves in terms of what any higher education user might hope to retrieve. this was true especially for known item searches, as well as searches with very focused parameters. in addition, our discussion of the pilot phase led us to include a “flag for followup” option to help highlight specific searches for either additional discussion or potential reporting to the vendor. by the spring of 2014, we were ready for each librarian to independently review the search results for each query. figure 1. survey instrument please indicate how strongly you agree with the following statement: primo produced relevant results for this search: 1 = strongly disagree 2 = disagree 3 = neither agree nor disagree 4 = agree 5 = strongly agree flag for follow­up: yes/no comments (optional) results after the librarians had completed ranking of the search terms, we held a meeting to discuss the searches. prior to the meeting, average ranking scores were released to the group. interestingly, scores were generally close for all librarians, with a little more pessimism from the systems librarians. this in and of itself served as an initial team building acknowledgment that there was common ground across both groups. individuals average of result relevancy round 1 average of result relevancy round 2 systems librarian #1 3.27 3.68 systems librarian #2 3.56 4.23 systems librarian #3 3.54 3.81 systems librarian #4 3.94 3.64 teaching librarian #1 3.77 3.91 teaching librarian #2 3.66 4.06 teaching librarian #3 3.86 outreach liaison librarian 3.33 digitization librarian 3.63 average 3.66 3.79 departments systems 3.57 3.84 teaching 3.76 3.96 new librarians (outreach + digitization) 3.48 search type known item 3.90 4.24 the resulting conversations about the searches themselves were even more fruitful. on an actionable level, some searches were forwarded to ex libris as examples of where search relevancy could be better. it was especially useful to start with real users’ searches for both groups to understand the typical use of the system, outside of each librarians’ domain (systems or teaching). much like a usability test, this process centered the conversation on the user by way of searches themselves. a number of trends were identified including known item versus topic searches, usage of advanced search, problematic punctuation, misspellings, and ambiguous searches. a year later in the spring and summer of 2015, the same survey was completed again with the same search terms. there are inherent scientific problems with such repetition due to librarians remembering previous searches, librarians being influenced by past ranking or discussion, and the potential that the vendor could have fixed previously reported problems. further, it is important to note that the systems department initiated other outreach within the library during this period including regular staff blog postings, an aggressive patching schedule for primo to take advantage of new releases, and a monthly open staff meeting to discuss web and discovery issues. nonetheless, the repetition of search rankings served as a good closure to the project. we were indeed curious to get a sense of how the search tool (or our attitudes) may have changed given the passage of time and a number of primo upgrades which focused on search relevancy and known item retrieval. we decided to open the second round up to all library faculty in order to broaden the potential discussion. all four of the original systems librarians, two of the original three teaching librarians, and two new librarians (one outreach liaison librarian and one digitization librarian) completed the second round of testing. for this round, summary results were not shared before discussion. anticipating a larger group, one of the systems librarian pulled out a number of searches as a basis for suggested discussion. this set included queries that had garnered many comments or large variations in ratings between librarians. in general, librarians returning to the results a year later did not feel that previous rankings had a big impact on their scores. by and large, the perception was that performance of the discovery tool was better. the average of all scores was slightly higher (3.79 in 2015 vs. 3.6 in 2014), and likewise both of the repeating group averages were higher in the second round (teaching librarians went from 3.7 to 3.96 and systems librarians from 3.53 to 3.84). the two new librarians to the project recorded the lowest average score (3.48). acknowledging all of the validity flaws, the comparisons must be made with a very forgiving brush. nonetheless, there does seem to be more agreement amongst the various librarians with results clustered close to each other in a consistent range of mid to upper 3’s. in the second round we analyzed the data further and found that overall confidence in the quality of the results was significantly higher for searches that were identified as known item searches (4.24 as opposed to the overall average of 3.79). this was also consistent with qualitative impressions from the discussion sessions where a number of librarians in the repeating group remarked that results for known item searches seemed more relevant in the second round. a common theme in both sets of discussions was the comparison of topic relevance with results from google scholar. even though it was not part of the agreed­ upon procedure, several librarians (both teaching and systems) reported trying the same searches not only in our implementation of primo, but also other institutions’ discovery platforms, and google scholar. in all cases, the feeling was that google scholar outperformed vendor discovery systems for topic searches. this points to the challenges and expectations that discovery services must meet in satisfying known item and topic searches (and being able to tell the difference between the two). qualitative librarian comments when asked “what would you like to share in this article about the project?” the librarian participants added the following comments. systems department head. search rounds 1 and 2. i very much value the time from all of the participants and the productive conversations that resulted from the project. i am especially interested in how focused some of our discussions were that illuminated some of the unique challenges that library discovery services can have. while we did some benchmarking against google and google scholar, i think that we all came away with a more nuanced appreciation for library discovery and increased respect for each other’s opinions. teaching and learning department head. search rounds 1 and 2. this was about as ideal a collaborative improvement process as i could imagine. not only did it result in improvements to our local implementation of primo (and based on some of our feedback, perhaps improvement of the product at large), but it also created a sense of shared understanding and concern for how we might continue to work together to improve our enterprise systems. prior to the project, i thought quite wrongly, as it turns out that our systems librarians and teaching librarians had very different perspectives on primo’s overall performance. despite their continual support and collegiality, i thought that the teaching librarians may have been a bit of a holy terror to our systems colleagues with our concerns about primo. turns out, we were on the same page. i suspect this is true more often than not, and i’m grateful for the the opportunity to have deepened our collaborative relationship. integrated systems librarian. search rounds 1 and 2. i was happy to be included in a project that was beneficial in many different areas. in systems we tend to only hear from our colleagues in other departments when there is something wrong. working together on this project to not only find what we all thought could be improved in the system but also finding what was working well was refreshing. while everyone brought different perspectives to the table it was good to see that in the final analysis our opinions were very similar. it was also a positive outcome that we could provide real and meaningful feedback to our vendor in the hopes of improving the discovery product for everyone. i think this project helped reinforce the idea that we are all on the same team and though we approach the issues from different perspectives we share the same desired outcome ­ the best discovery service possible for our users. teaching & learning librarian. search round 1. siloed as different departments often are in large libraries, this exercise was valuable not just because it offered a glimpse into patrons’ real­life searching habits, but because of the interdepartmental relationship building aspect. as a new librarian i’d only been employed by vcu libraries for 6 months at the project’s start it was especially useful. i can see this exercise being replicated for myriad purposes? i would be especially interested to see how patrons’ search behavior changes over time based on future improvements to our discovery service and teaching practices. systems librarian. search rounds 1 and 2. it was extremely valuable to see how the system behaved in response to actual user queries. as a systems librarian, most of my experience evaluating discovery has been based on my own use and the reports of my librarian colleagues (which are usually prompted by problems). how librarians use the discovery system often differs from how our patrons use it, and because i don’t do instruction or reference, this exercise provided a much needed education in current patron search behavior. i was reminded of the value provided by certain discovery features, such as “did you mean” suggestions and misspelling corrections, as well as the frequency and effectiveness of different categories of query, beyond the narrow topic or known item searches: it was enlightening to see how the system responded to broad search terms (“sociology”), known items not in our collections, and copy­pasted citations alike. web systems librarian. rounds 1 and 2. i’ve been interested in the dynamic between “traditional librarians” and it since graduate school and have found this project exceedingly helpful in opening up lines of communication between two functional areas in our library, not only around these relevancy assessments but in general. the bigger benefit of this project was that everybody seemed to gain an understanding of our shared goals and priorities while grappling with software questions that sometimes not even the systems librarians could answer. the de­centering of it as “all knowing” really helped here. though we systems folks are not incompetent, we often don’t have answers or solutions right away, or even know there are problems with systems we manage until we can see them through users’ eyes, and this insight is often being conveyed to us through teaching librarians. we all gained valuable insight from seeing actual users’ queries and understanding how our system is used? that can inform our future work in systems and in teaching scenarios with library users. teaching & learning librarian. search rounds 1 and 2. systems librarians and teaching librarians don’t often get to work together as directly as we did on this project. the obvious benefit of such a collaboration is the shared goal of analyzing and improving our searches in primo. though we came at the search results with differing knowledge bases, we found a useful middle ground and language that, in the end, resulted in an ideal collaborative project. conclusion while we by no means claim to have solved discovery, we are pleased to report a number of successful outcomes. even with such a small random sample, we were able to identify a number of problematic searches to bring to our vendor’s attention. likewise, this effort has encouraged ongoing local development of add­on javascript improvements to offer search suggestions. culturally, the project itself helped us move beyond a systems and teaching librarian dialectic in addressing our discovery tool. the resulting discussions helped raise sensitivity across both groups toward the challenges of discovery by placing genuine user queries at the conversation’s center. as a group we left the project with a sense of progress and a desire to see our discovery tool further improve. cultivating this type of collegiality, shared responsibility, and high expectations across the profession is key to future improvements across all of our systems and services. the use of anonymized logs as a focal point for discussion proved successful for us in both building partnerships across technical and non­technical departments and improving system usability. endnotes [1] coghill, gariepy, hodge, mcnulty, and white listed in alphabetical order. [2] a potential pitfall for discovery system log analysis is the introduction of auto­suggested searches. in essence, the search terms pulled in 2013 are somewhat special in that we had not yet introduced an auto­suggest feature. for future projects, we would need to manually check to try to determine if searches were influenced by autosuggest. utility and use of autosuggest in general seems like a rewarding research area for the future. references allen nd. 2015. utilizing discovery tools for classrooms: how do librarian attitudes on discovery impact tools they teach? library hi tech news [internet]. [cited 13 oct 2015]?32(1):8­12. available from: http://dx.doi.org/10.1108/lhtn­09­2014­0078 asher aa, duke ll, wilson ss. 2013. paths of discovery: comparing the search effectiveness of ebsco discovery service, summon, google scholar, and conventional library resources. coll & res libraries. 74(5):464­488. carson p, little g. 2014. re­framing librarians’ identities and assumptions around it. j library adm. 40(3­4):405­407. ciccone k, vickery j. 2015. summon, ebsco discovery service, and google scholar: a comparison of search performance using user queries. evid based library and inf practice [internet]. [cited 13 oct 2015]?10(1):34­49. available from: http://ejournals.library.ualberta.ca/index.php/eblip/article/view/23845 dehmlow m. 2013. editorial board thoughts: services and user context in the era of webscale discovery. inf technology and libraries. 32(2):1­3. deodato jj. 2015. evaluating web­scale discovery: a step by step guide. inf technology & libraries. 34(2):19­75. ellero nn. 2013. an unexpected discovery: one library’s experience with web­scale discovery service (wsds) evaluation and assessment. j library adm. 53(5/6):323­343. gayhart l. 2013. out from behind the firewall: towards better library it communications. code4lib j [internet]. [cited 13 oct 2015]?21:8 pages. available from: http://journal.code4lib.org/articles/8741 jansen bj. 2006. search log analysis: what it is, what’s been done, how to do it. library & information science research [internet]. [cited 13 oct 2015]?28(3):407­432. available from: http://faculty.ist.psu.edu/jjansen/academic/pubs/jansen_search_log_analysis.pdf park m, lee t. 2013. understanding science and technology information users through transaction log analysis. library hi tech [internet]. [cited 13 oct 2015]?31(1):123­140. available from: http://dx.doi.org/10.1108/07378831311303976 subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – semi-automatic citation correction with lemon8-xml mission editorial committee process and structure code4lib issue 6, 2009-03-30 semi-automatic citation correction with lemon8-xml the lemon8-xml software application, developed by the public knowledge project (pkp), provides an open-source, computer-assisted interface for reliable citation structuring and validation. lemon8-xml combines citation parsing algorithms with freely-available online indexes such as pubmed, worldcat, and oaister. fully-automated markup of entire bibliographies may be a genuine possibility using this approach. automated markup of citations would increase bibliographic accuracy while reducing copyediting demands. by mj suhonos history scholarly practice has, for decades, used a set of discipline-centric bibliographic styles (e.g., apa, mla, chicago, etc.) to codify the metadata elements in reference lists, through specific typographic and punctuation conventions. one of the most time-consuming and repetitious aspects of scholarly publishing is the validation and correction of this information. generally, copyeditors must manually inspect each element of a citation and compare it visually to a record retrieved through an online indexing service. with the advent of word-processing software and the internet, a number of reference-management applications have appeared. historically, this software has been proprietary and expensive, averaging from $100-300 for each copy installed [1]. more recently, inexpensive alternatives have arisen, such as the open-source, browser-based zotero [2] and low-cost, web-based refworks [3]. although these applications have given authors the ability to store and edit citations, they typically have very limited ability to search online citation indexes or export citations in a standardised, structured xml format. as a result, there are a number of challenges for authors trying to manage their personal bibliographies and ensure an accurate and well-formatted reference list. copyeditors, in turn, are constrained by their ability to read the reference format supplied and by their access to online databases for validating citation data. why lemon8-xml? in 2000 pubmed central (pmc) was launched as a “free digital archive of biomedical and life sciences journal literature” [4]. the growth of pmc and similar archives led to a need for fully-structured documents containing not only indexing metadata, but marked-up content of the entire article, including bibliographic citations. although nlm maintains a rigorous review process for inclusion in pmc, there is also a substantial technical challenge involved in generating fully marked-up xml for complete journal articles. commercial products and services are available for this purpose, but they tend to be very expensive, as with inera’s $10,000 extyles software [5], or involve a pricey consulting contract for typesetting services. in addition, this work is often done manually or by outsourcing to workshops in developing nations, causing the quality to vary from vendor to vendor. in 2003, crossref, a collaborative reference linking service for scholarly articles using the digital object identifier (doi) system, added a fee of $0.22 per article, per year, for publishers who did not (or could not) provide fully-structured reference lists with each article they deposited in their system [6]. open-access journals and small publishers in this situation find themselves in a difficult spot: they can’t afford the expensive services to get into the most prevalent indexes and archives, yet inclusion in those services would undoubtedly increase their exposure, academic impact, and for commercial journals, revenue. there is clearly a need for a low-cost, easy-to-use, software application that would lower the technical and financial barriers for journals seeking inclusion in these services. the public knowledge project (pkp) created lemon8-xml (l8x) as a companion to its successful journal management software, open journal systems [7]. the goal of l8x is to provide a freely-available, web-based article editor that gives users the ability to generate fully nlm-compliant xml with minimal effort, and provide assistance with correcting article and citation metadata. l8x is built upon common lamp technology, and is currently being used by at least one journal in production [8]. citation parsing and online indexes a number of research projects, including paracite [9], freecite [10], and parscit [11], have attempted to create software which, based on established citation styles, can extract the requisite metadata elements in a way that is tolerant to aberrations and errors. at the same time, there has also been a rapid rise in the number of freely-available online indexes, such as oaister [12], citeseer [13], and isbndb [14], that provide web-based search apis and well-structured indexing metadata in xml response format. the intent of l8x has been to, wherever possible, integrate the results of these efforts in an extensible manner. by building on the work of others, we can focus on making their systems and data more accessible to authors and copyeditors as efficiently and user-friendly as possible. in this way, we are helping to disseminate the citation parsing algorithms, increase visibility of the online indexes, and provide a pragmatic combination to automate the work that is already being done manually. system design l8x is broadly designed as an application to assist in creating structured markup for an entire scholarly article, not just reference lists. the scope of the software also includes semi-automatic detection of article sections, figures, tables and metadata. among the library and publishing communities, however, it has been the citation parsing and editing features which have garnered the most attention. the general approach involves the following steps: identify a portion of text that represents a formatted citation parse this text into discrete metadata elements identify the genre of the citation (journal article, book, etc.) search available databases for a full record of the citation provide a measure of estimated accuracy in parsing and searching provide an intuitive user interface to manually inspect the results at each step as can be seen in figure 1, this is essentially a two-stage process: parsing and searching (“lookup”). the user interface is provided to allow editors to apply their knowledge and interject their corrections during the process if they choose, or it can be used to enter and edit a citation entirely manually – the parsing and lookup functions are assistive and optional. likewise, by setting a threshold for “scoring” the accuracy of the parsing and lookup stages, an editor can choose to automatically process an entire reference list without using the user interface whatsoever. parsing and searching functionality is provided to l8x by way of “components” — essentially, plugins for accessing a particular parsing algorithm or online citation index. each plugin is tailored to the api of the specific system it is contacting, including providing any authentication keys or schema mappings required to get data between l8x and the corresponding service accurately. by using this approach, new services can be added to the system by developing a new plugin and installing it. similarly, journals or organizations can develop or maintain their own plugins as appropriate, and choose whether or not to release them, and under what terms if they choose so. figure 1. system design and data flow. [view full-size image] the guiding design principle behind this approach has been to minimise the amount of work required by the editor, not eliminate it altogether; to allow the computer to do as much as possible, but also allow users to focus their effort on those citations which are problematic or otherwise require human judgement. as the overall quality of the software improves, including additional citation indexes and improved parsers, the amount of human work required will decrease accordingly. a common metadata format in order to aggregate the results of each citation parser, we had to choose a metadata schema for representing the elements of a citation. this schema had to contain all of the elements in a citation of the genres supported by l8x: journal article, conference proceeding, book, and cited website. in addition, the schema had to be extensible enough to include possible item identifiers, such as isbn, doi, or pubmed id (pmid). the simplest schema to meet these requirements was a metadata format based on the openurl 1.0 scholarly formats (journal, book, dissertation) [15] in kev (key/encoded-value) encoding [16]. the choice to use openurl 1.0 was reinforced by the fact that two of the citation-parsing methods initially chosen for l8x were based on the openurl metadata elements. as a result, it was simplest to write a crosswalk from the parsers that returned non-openurl elements, and allow openurl-compliant parsers to pass data through directly. table 1 shows a list of the parsing systems incorporated into l8x, and the metadata schemas used by each. parser language/procotol schema freecite post xml custom lemon8-xml php / regular expression openurl 1.0 paracite perl openurl 0.1 parscit post xml openurl 1.0 table 1. citation parsing services/algorithms. one of the main considerations for representing citations in the openurl schema and developing crosswalks was choosing elements for each genre of citation. this played an important part in the development of the citation editor ui, as well as the way in which scoring and comparison are done by l8x. the openurl 0.1 standard identifies journal, book, and conference proceeding citations as individual genres using the same metadata profile. however, the openurl 1.0 standard considers book and journal citations as different genres with corresponding metadata profiles, and conference proceedings as a variant of a journal article citation [17]. the mapped metadata profiles used in l8x can be seen in table 2. element journal article book conference proceedings web citation authors required required optional required atitle required optional optional required title required required required optional year optional volume required optional issue required optional edition optional editor optional optional required chapter optional publoc required required date required required required sponsor optional optional optional artnum optional optional publisher required required spage optional optional optional epage optional optional optional isbn optional pmid optional doi optional optional optional targeturl optional required access-date required comment optional optional optional optional table 2. citation metadata schema (elements in italics are mapped to openurl 1.0). the original citation string is passed to each parser in turn: the results are serialised to xml, passed through an xslt stylesheet mapping them to the l8x openurl schema (where superfluous elements are discarded), an accuracy score calculated, and added to an array of results. rudimentary de-duplication and data cleaning is then performed. for example, ensuring the “year” only consists of a four-digit value, and the results are presented in the ui as an editable drop-down list. very often, some parser plugins will produce highly inaccurate results, and although these are easily ignored by a human editor, additional data cleaning/checking could reduce this issue. overall, however, each parser will extract most of the citation correctly, and the combined results usually average toward an accurately-parsed citation, reflected in an overall parse score. the editor can then make any corrections they see fit – often, simply selecting the correct value for elements with multiple parsed results – and save the values to the database. aggregating web service results as in the parsing stage, results from online index databases are serialised to xml, passed through an xslt stylesheet mapping them to the l8x openurl schema, and returned with an accuracy score as an array of results. however, obtaining structured citation information from online databases is more complex than parsing for a number of reasons. to begin, each database has its own licensing restrictions, even “free” ones, and some require registration to obtain an api key for access to their data. at present, many large databases with a web service api provide some level of free access. for example, worldcat provides access to registered (and thus, paying) oclc members who request an api key for limited use. oclc does, however, provide a limited set of free services with its xisbn offering. generally, users must agree to terms of service which mandate non-commercial use of the data and limit the number of requests made on a daily or monthly basis [18]. in addition, obtaining citation information is itself a two-stage process: querying the database, and retrieving a structured record. each database uses its own search interface and syntax, and the accuracy of the search is directly affected by the accuracy of the parsed citation, as well as the manner in which the search query is constructed. moreover, the challenge in developing a search query lies in attempting to match a single record, not a list of possible results. when necessary, the l8x lookup plugins order results by relevance, and take the first (i.e., most relevant) in the list. the request mechanism and result schema for searching and retrieving records differ for each online database. most index apis use a restful uri query syntax, but some require oai http post or json. similarly, most return results in a standardised xml schema (e.g., marcxml, oai-dc), but some use a custom schema or non-xml syntax like json. this means that each lookup plugin must contain the necessary metadata crosswalk to map results into the l8x openurl schema. table 3 provides a list of the online databases currently supported by l8x, as well as a list of proposed plugins for development in the near future. service scheme license content crossref openurl free journal articles isbndb url free registration books pubmed (medline) url free journal articles, conference proceedings worldcat api url oclc members only books, journal articles worldcat xisbn url free books citeseer oai free journal articles, conference proceedings oaister openurl free journal articles, misc librarything url, json free registration books openlibrary json free books google books url free books amazon url free registration books, misc table 3. web-based metadata index api services (items in italics are in development). when a result is obtained from an online database, it is compared to the stored (parsed) citation by individual element and overall content, and a similarity score assigned. this score is intended to identify whether the record retrieved actually matches the citation, and whether any additional information (e.g., a url, doi, or full list of author forenames) was returned as well. like the parser results, rudimentary data cleaning is performed, and the results displayed in the ui as an editable drop-down list. this allows the user to use the same interface for both parsing and searching. an additional consideration with obtaining citation metadata lies in correctly identifying the genre of the citation at the parser stage. since database content is generally genre-specific, the genre must be correctly identified in order to query the appropriate databases. as well, because content coverage depends on the parser plugins available, results will be improved over time as more online services are made available and plugins are developed. threshold-based scoring system both citation parsing and lookup functions generate an accuracy score for each plugin —methods differs between parsing and lookup, but both have been developed to return a percentage value between 0 and 100—which generally falls between 50 and 90 for relatively well-formed citations. the intent is not to present an absolute measure of the quality of the citation, but rather a relative indication of how well the parser or lookup service was able to generate a useful result. parser scores are calculated based on the number of metadata elements successfully extracted from a citation compared to the number of expected elements (e.g., 6 of 8 for a book citation would score 75). lookup scores are calculated based on the percentage difference in content between the parsed citation and the result from the online index (e.g., a citation containing 100 characters that returns a result with 120 characters would score 20). this score is added to the aggregate parse score to indicate whether it is improved or not (e.g., 75 + 20 = 95, indicating that the lookup improved the citation’s data). scores are aggregated across all plugins and an aggregate score is calculated based on the value between the mean and maximum scores (i.e., maximum-weighted mean score). this reflects the fact that only one “good” result is required to be useful, while multiple partial results are more likely to improve the overall result than not. empirically, scores from 50-70 tend to be good, 70-90 tend to be very good, and above 90 are excellent, essentially “perfect”. this general ranking allows l8x to introduce the concept of thresholds into the parsing process to improve automation. by setting a minimum value for parse and lookup scores, the user can set a threshold for what they consider to be acceptable automatic parsing and correction: for example, deciding all parsed citations with a score above 60 to not require human editing and deciding all citations with a returned lookup score above 80 to have been correctly identified in the online databases. using this threshold system means that l8x can automatically process the majority of the citations, and flag those that do not meet the thresholds for the editor to inspect manually via the ui. citation editing interface another way in which l8x provides feedback to the editor about the accuracy of its parsing and lookup functions is to visually mark the elements of the citation that it was able to extract into individual metadata elements. figure 2 shows an example of a citation (#59) that was successfully parsed into 7 elements (identified by coloured highlighting), with a score of 90 (in this case, the 90 comes from a successful lookup as well). by presenting information this way, and using simple iconography (i.e., a check-mark or red x icon), editors can see at a glance which citations require their attention, and where the problems most likely are. figure 2. citation editor showing reference list and colour-marked parsed citation. [view full-size image] when the editor does decide to edit a citation, he clicks on an icon next to the marked contents, and the interface expands to display a list of the metadata elements and their values, as well as buttons for re-parsing or re-searching the citation metadata. when results are returned, they are displayed in this same interface; the combo boxes next to each element are populated with the results of the parse or lookup. a flashing asterisk appears next to any values that have been modified to visually draw the editor’s attention (see figure 3). figure 3. citation editor showing expanded elements and combo boxes with online lookup results. [view full-size image] several other visual elements are also used to assist the editor: the background colours of each element match those in the marked citation and are grouped chromatically, editable combo boxes are used to allow the editor to simply select from a drop-down or make textual corrections (or both), and, the aggregate score is immediately displayed to let the editor determine the quality of the result. the current user interface is the result of rapid user feedback during the development process, and contributes significantly to the effectiveness of the citation editing component of l8x. at the same time, there are still known deficiencies that are yet to be addressed. as a work-in-progress, it presents both an example of a successful user-driven interface as well as an opportunity for great refinement. future directions as lemon8-xml is intended to be much more than a citation editor, there are several other areas of active development beyond those mentioned here. the goal of l8x remains to provide a tool for developing structure in scholarly articles, and many of the principles learned so far have application in other areas. for example, validating quotations against their canonical records in online databases, verifying author or cited speaker names against online name authority indexes, or even degrees of plagiarism detection and warning. for citation editing, however, the main focus continues to be the addition of more online index services, particularly multilingual and non-english databases to improve the international applicability of l8x. in particular, there is great potential for a generic oai or openurl-based lookup plugin, to allow institutions and publishers to access metadata in their own knowledge bases and union catalogues. adding more support for additional citation styles—again, that are often non-english or specific to smaller disciplines, is another area for development. finally, as part of the plan for developing integration between l8x and ojs, it may be desirable to provide the citation editing framework as a stand-alone service that can be installed or otherwise used independently. the most obvious area for improvement remains the user interface for the citation editor. more visual indicators of the software’s progress and function are desirable, as is the ability for the user to configure the parser and lookup plugins’ function more specifically. for example, different thresholds for each plugin; re-ordering or disabling plugins on a per-article, per-citation, or dynamic basis; and dynamic feedback on each plugin’s results and score. one area that holds particular promise is the ability to cite a resource by identifier alone; for example, using a pmid, isbn, or doi to uniquely identify a citation. since l8x can resolve these to a full set of citation metadata, it is not inconceivable to imagine an entire reference list composed entirely of identifiers provided by the author. this itself raises another possibility: integration of embedded citation formats such as those provided by the zotero openoffice plugin [19] to obviate the need for parsing altogether. conclusion during the time lemon8-xml has been developed, a number of reference management applications, including extyles and bibsonomy, have begun to incorporate limited parsing and lookup methods [20,21]. however, these tend to be limited to some combination of pubmed or crossref, far short of the roster intended by l8x. it is clear that this approach has great potential, and will only gain wider appeal as parsing algorithms improve and more databases become accessible. improvements to the threshold scoring system will bring the software incrementally closer to fully automated citation correction, although it is probably naïve to think that it will ever remove the need for a human review. as open source software, lemon8-xml is freely available to download for anyone wishing to use it, as well as to those interested in collaborating on its development. the public knowledge project maintains documentation and support for l8x [22], as well as a publicly-accessible demonstration instance at: http://www.lemon8.org. we would be happy to receive feedback from anyone interested in testing or developing l8x, either for their own purposes, or in collaboration with pkp. resources [1] thomson researchsoft: endnote, procite, reference manager, refviz. http://scientific.thomsonreuters.com/info/rs_purchase_direct/ [2] zotero browser toolbar. http://www.zotero.org/ [3] refworks. http://www.refworks.com/ [4] pubmed central overview. http://www.pubmedcentral.nih.gov/about/intro.html [5] inera aids production editors with extyles. http://www.inera.com/seybold.pdf [6] crossref publisher fees. http://www.crossref.org/02publishers/20pub_fees.html [7] open journal systems. http://pkp.sfu.ca/ojs [8] no budget, no worries: free and open source publishing software in biomedical publishing. http://www.openmedicine.ca/article/viewarticle/276 [9] paracite paratools. http://paracite.eprints.org/developers/ [10] freecite citation parser. http://freecite.library.brown.edu/ [11] parscit: an open-source crf reference string parsing package. http://wing.comp.nus.edu.sg/parscit/ [12] oaisterunion catalog of digital resources. http://www.oaister.org/ [13] citeseer.ist scientific literature digital library. http://citeseer.ist.psu.edu/ [14] isbndb.com. free isbn database. http://isbndb.com/ [15] registry for the openurl framework. http://alcme.oclc.org/openurl/servlet/oaihandler?verb=listrecords&metadataprefix=oai_dc&set=core:metadata+formats [16] the key/encoded value (kev) format implementation guidelines. http://library.caltech.edu/openurl/standarddocuments/kev_guidelines-20041209.pdf [17] openurl syntax description. http://www.exlibrisgroup.com/?catid={2f09a3e3-a22b-4cb1-8434-930cdb7264a7} [18] xisbn web service http://www.worldcat.org/affiliate/webservices/xisbn/app.jsp [19] zotero openoffice integration. http://www.zotero.org/support/openoffice_integration [20] what’s new in extyles? automatic reference correction. http://www.inera.com/refcorrection.shtml [21] bibsonomy scraperinfo. http://www.bibsonomy.org/scraperinfo [22] lemon8-xml. http://pkp.sfu.ca/lemon8 about the author mj suhonos is a system developer and librarian with the public knowledge project at simon fraser university. he has served as technical editor for a number of open access journals, helping them to improve their efficiency and sustainability. more recently, he leads development of pkp’s lemon8-xml software, as part of their efforts to decrease the cost and effort of electronic publishing, while improving the quality and reach of scholarly communication. subscribe to comments: for this article | for all articles one response to "semi-automatic citation correction with lemon8-xml" please leave a response below: zayed, 2011-11-12 sir please tell me the parser setting required for lemon8 xml openoffice is an essential for proper visuallisation of document leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – automating reference consultation requests with javascript and a google form mission editorial committee process and structure code4lib issue 53, 2022-05-09 automating reference consultation requests with javascript and a google form at the cuny graduate center library, reference consultation requests were previously sent to a central email address, then manually directed by our head of reference to the appropriate subject expert. this process was cumbersome and because the inbox was not checked every day, responses were delayed and messages were occasionally missed. in order to streamline this process, i created a form and wrote a script that uses the answers in the form to automatically forward any consultation requests to the correct subject specialist. this was done using javascript, google sheets, and the google apps script backend. when a patron requesting a consultation fills out the form, they include their field of research. this field is associated in my script with a particular subject specialist librarian, who then receives an email with the pertinent information. rather than requiring either that patrons themselves search for the right subject specialist, or that library faculty spend time distributing messages to the right liaison, this enables a smoother, more direct interaction. in this article, i will describe the steps i took to write this script, using only freely available online software. by stephen zweibel context like many other academic libraries, we offer consultations with our subject specialist librarians. due to the fact that we are a school for phd and master’s students, most questions we receive are particular to the individual student’s field of research. for this reason, we have found that it saves time to have students directly request help from their liaison librarian, rather than meeting first with whomever happens to be at the reference desk. this offers the additional potential benefit of fostering personal relationships between librarians and students that can last through their doctoral programs. problem our central problem was: how do we make it as easy as possible for students to reach us? we have decided to trust our students to know whether their question is worth asking a subject specialist directly, so how do we enact that in reality? when i came to the cuny graduate center library, the ad hoc solution was to have a google form with prompts about the student’s question, topic area, and other general queries. on submission, this form would be sent to our reference desk email address, monitored by several librarians but primarily our head of reference. whichever librarian saw the consultation request email would read it, and depending on its contents, forward it to the relevant subject librarian. this of course did work, and had the advantage of a personal touch. the main disadvantage of this manual process was its potential for human error — sometimes emails would languish in the inbox for days -– and the added administrative burden it placed on librarians. design the solution as i saw it was to remove the librarian intermediary to allow students and faculty to get their consultation request directly to a subject specialist. one proposal was to do away with the request form and direct patrons to our list of liaison librarians, with instructions to reach out to the librarian for their program. although this could work, it would add friction to the process for patrons, and thus could risk reducing the number of requests we would receive. instead, i wanted to make the request process as smooth as possible for both patrons and librarians. the required solution would need to make it possible for: patrons to be able to get their request to the correct subject specialist without knowing who that specialist was subject specialists to receive the correct requests without an intermediary librarian changes in subject specialist status (like sabbaticals or new responsibilities) to be easily reflected without large changes to the system therefore, the solution would have to be automated, and able to determine the correct subject specialist from the information provided by the patron in the form. for a while i entertained a kind of machine learning solution, where topic modeling would be trained on the long-form description the patron gave of their problem, and compared to a list of our experts. after consideration i determined this to be unreliable and much too complicated! fairly quickly i came to the solution of adding a single question to the form: “which academic program do you belong to?” because each academic program has an assigned librarian liaison, this provides a 1:1 determinant of who would be responsible for handling a consultation request. this piece of information, supported by the other results of the form, would enable us to match requests to the appropriate subject specialist. the challenge was to do that in real time, as requests came in, and to generate a message that would reach the desired librarian via a channel they were accustomed to checking regularly. google forms and google sheets the choice i made was path dependent, in that we were already using google forms. a solution suggested itself: to use google’s free tools to get an automatic email to send every time the form is submitted by a patron. our form looks like this: figure i. form content example. on the form management page, there is a little green icon that links to a spreadsheet of the responses. figure ii. form management page. on the spreadsheet page, there is a menu called ‘extensions’. figure iii. extensions menu. using ‘apps script’ link leads to google’s apps script code platform, which uses a slightly modified/specialized form of javascript that allows us to interact with the form results. essentially, we have written a javascript function that will run when a form is submitted, taking the form inputs as values to decide its behavior. function construction we start by naming our function and making a ‘librarian’ object: function emaillibrarian(e){ var librarians = []; function librarian(name, email, programs, available){ this.name = name; this.email = email; this.programs = programs; this.available = available; librarians.push(this); //adds librarian to array } breaking this down somewhat, in the line: function emaillibrarian(e){ the ‘e’ is the ‘event object’ passed to us by google forms (i.e., actually the content of the form submitted by a patron). we will use this a bit later in the script. this line: librarians.push(this); //adds librarian to array adds each created librarian to the list of librarians for later. function librarian(name, email, programs, available) this librarian object is used to represent each of our subject experts, like this: var stephenzweibel = new librarian("stephen zweibel", "szweibel@gc.cuny.edu", [ "comparative literature", "data analysis and visualisation", "data science", "data visualization", "demography", "digital humanities", "french", "geography/gis", "language and literacy", "master of arts in liberal studies (mals)", "music", "psychology of political behavior"], true); here we have in order: subject expert name email a list of fields this librarian is responsible for and ‘true’, which represents availability. this is set to ‘false’ if the librarian is on sabbatical, etc. so a librarian object is created for each librarian, and that librarian is added to a list of librarians, i.e.: var librarians = [stephenzweibel, …]; from here, it’s a matter of grabbing the relevant subject area of the form request and matching it to a librarian. mals: e.values[10],status: e.values[5], purpose: e.values[6], availability: e.values[7], topic: e.values[8], "steps taken": e.values[9]} this variable takes the responses from the form, ‘e’, and splits it out into attributes that i can read and do something with. ‘program’ is important, as this is what we’ll match our liaison emails with: var liaison = librarians.filter(function(liaison){ return liaison.programs.includes(program); })[0]; var liaisonemail = liaison.email; this takes advantage of a list method that filters according to some rule, in this case whether or not the program (anthropology, english, comp lit, etc.) chosen by the patron is included in the listed responsibilities of the librarian. i have also added a couple of overrides, in cases where other answers in the form are more salient than the student’s academic program, for instance: if (values.purpose == "data management"){ liaisonemail = "szweibel@gc.cuny.edu"; } this would choose me as the recipient based on a different criterion than the program. another important intervention: if (liaison.available == false){ liaisonemail = "ref@gc.cuny.edu"; }; in this case we account for unavailable librarians, and send the email to our catchall email. and to construct and send the email: // construct the email text var text = "hello,\n\n" text += values.name + " has requested a research consultation.\n\n" for (var item in values) { text += item+ ": "+values[item]+"\n\n"; } mailapp.sendemail (liaisonemail, subject, text, {cc: "ref@gc.cuny.edu"}); }; this translates the values into text, and we take advantage of google’s mailapp toolset to send an email with that text to the correct liaison librarian, cc’ing to the reference desk email. an example email: hello, **** has requested a research consultation. name: **** patron email: ****@gc.cuny.edu phone: **** program: digital humanities status: student purpose: dissertation, thesis, or capstone project availability: mon, oct 18th 2-6 pm; wed, oct 20th 2-6 pm; thu, oct 21st 2-6 pm topic: i am working on my capstone project and i wanted to have a consultation on the best way to submit/preserve my digital project. steps taken: i have talked with the digital fellows about audio techniques. settings there is one more important step. we need to set the function we have made to fire on a trigger; navigating to the ‘triggers’ section: figure iv. selecting triggers from settings. here are our settings: figure v. trigger settings. now the function will run every time the form is submitted by a user. luckily this also will send us an alert if the function fails for whatever reason, usually a misspelling on my part of an academic program. advantages/disadvantages for the most part, this script runs without any intervention, and has for several years. it does need to be updated periodically, to account for librarian availability and changes in responsibilities. relying on free tools that are outside our control is the largest concern with this solution: once already, google has made a big update to the way their tools work, which required some minor adaptations on our end. another potential disadvantage regards continuity: if i were to leave my position at the library, this process might not be transparent to a successor librarian. this paper will hopefully serve as documentation. since implementing this solution, our librarians have come to see the form emails as infrastructural and to take them for granted. we receive maybe ten emails a month, a good rate for our small patron base. resources link to the reasearch appointment request form: https://docs.google.com/forms/d/e/1faipqlsdnpu7vwgbep9frzggtrqbdgeugz7qbqzwtnaeex0rvez8bha/viewform javascript code source about the author stephen zweibel is digital scholarship librarian at the cuny graduate center, where he supports students and faculty in developing digital projects and working with data, individually and through workshops and classes on research skills and tools. stephen built dh box, which won a national endowment for the humanities start-up grant. subscribe to comments: for this article | for all articles one response to "automating reference consultation requests with javascript and a google form" please leave a response below: gladys aboude, 2023-01-11 dear mr.zweibel, i hope that at the recpetion of this email you are well and safe. i am working on a project where i should be able to demonstarte how google sheets is a collaborative tool and how does it work in educaction. do you have any book or suggestion i could look for? please a digital book in free version would be useful, since i live in venezuela and books are not priority in our libraries, specially in the city we are living. i appreciate your support, sincerely, gladys aboude de orah djkian. br. science of the nutrition. leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – using the ‘rentrez’ r package to identify repository records for ncbi linkout mission editorial committee process and structure code4lib issue 38, 2017-10-18 using the ‘rentrez’ r package to identify repository records for ncbi linkout in this article, we provide a brief overview of the national center for biotechnology information (ncbi) linkout service for institutional repositories, a service that allows links from the pubmed database to full-text versions of articles in participating institutional repositories (irs). we discuss the criteria for participation in ncbi linkout for irs, current methods for participating, and outline our solution for automating the identification of eligible articles in a repository using r and the ‘rentrez’ package. using our solution, we quickly processed 4,400 open access items from our repository, identified the 557 eligible records, and sent them to the nlm. direct linking from pubmed resulted in a 17% increase in web traffic. by yoo young lee, erin d. foster, david e. polley, and jere odell introduction in 2017 the national library of medicine (nlm) announced that it would link pubmed records to open access versions of articles in institutional repositories (irs) through the national center for biotechnology information (ncbi) linkout service. this service benefits readers by identifying open access versions of articles that are otherwise hidden behind paywalls. at the same time, linkout rewards both the authors that contribute articles to irs and the ir service itself. authors benefit from the increased discoverability of their articles and from the increased readership that results when an open access version is easy to find. irs may also benefit from an increase in web traffic, but also from a general increase of trust and interest in the ir service among potential contributors. in order to participate in the service, ir managers are required to identify articles in their holdings that a) are open access, b) have a citation in pubmed, and c) are not included in pubmed central (an nlm-managed, public access repository). mature irs may include thousands of open access articles. manually checking each article by searching for it in both pubmed and pubmed central is time intensive and presents a barrier that may prevent irs from participating in linkout. in this article, we describe some existing solutions, provided by nlm, that allow for the identification of articles in an ir that are eligible for inclusion in the linkout service. we then provide our solution and document the process that we used to export the metadata of over 4,400 open access articles from a dspace repository and the development of an r script using the ‘rentrez’ package, which provides functions for applying the ncbi entrez utilities api (winter, 2017). using this approach, we can quickly identify articles that are eligible for inclusion in the linkout service, prepare the records, and send them to the nlm. by sharing our process, we hope to provide a useful and accessible solution for ir managers and staff that encourages further participation in, and use of, the linkout service. background indiana university – purdue university indianapolis (iupui) is a public university campus that offers 243 undergraduate, graduate, and professional degree programs. the iupui campus also serves as the main location for indiana’s dental, health & rehabilitation, nursing, public health and medical schools. the iupui campus includes four libraries: ruth lilly law library, ruth lilly medical library, school of dentistry library, and university library. all schools on the iupui campus use iupui scholarworks https://scholarworks.iupui.edu/ as an institutional repository. iupui scholarworks runs on dspace and is hosted and managed by iupui’s university library. librarians from all campus libraries participate in providing outreach and assistance with deposit of works in the repository. iupui scholarworks provides open access to over 12,000 contributions from iupui constituents–including: historical documents, student theses and dissertations, conference proceedings, working papers, reports, and scholarly articles. launched in 2004, the majority of the initial deposits to iupui scholarworks were historical documents and student theses and dissertations. in recent years, however, the deposit of scholarly articles by faculty authors has greatly increased. this change is largely a response to iupui’s adoption of an open access policy. on october 7, 2014 the iupui faculty council voted unanimously to adopt an opt out open access policy based on the harvard model open access policy https://osc.hul.harvard.edu/modelpolicy/ (iupui open access policy, 2014). under the policy, faculty authors retain their copyrights to scholarly articles and agree to deposit them no later than the date of publication in an open access repository, such as iupui scholarworks. the policy also gives the library permission to make these deposits on behalf of faculty authors. if authors can not or do not want to participate in the policy, they are given an opportunity to opt out for individual works. in an effort to make participation as easy as possible for faculty authors, articles that are published in open access journals and articles that are uploaded in copyright-permissible forms to open websites are harvested by the library and archived in iupui scholarworks. pubmed central is a leading source of accepted manuscripts harvested for deposit in iupui scholarworks, due to the fact that the majority of iupui faculty work in the health sciences. when an open access version of an article cannot be found in pmc, arxiv, or another source, the campus author of the work is notified and asked to deposit or to opt out. as a result, iupui scholarworks is the only open access source for many of the article manuscripts authored by the faculty. a significant number of these articles are indexed in pubmed. relative to the annual output of scholarly articles by iupui faculty (~3,000 articles per year), the open access policy workflow and systems are maintained by a small group of people–less than 3 fte. the library, therefore, has a strong interest in finding ways to reduce the labor of supporting the open access policy while also ensuring that there are few barriers to faculty participation. what is the linkout service? ncbi’s linkout service allows direct linking between nlm’s ncbi databases (e.g., pubmed, genbank, dbgap) to information and services provided by external institutions and/or organizations (linkout help, 2005). this service enables, among other things, the inclusion of links to full-text publications in pubmed citation records. for example, a library can coordinate with nlm to ensure that the library’s subscription holdings are linked to from pubmed, most commonly through an institutionally-specific icon that is displayed in a pubmed citation record. this allows users to access full-text holdings via pubmed based on their institutional affiliation and their library’s subscriptions. in addition to full-text publications, the linkout service provides access to other types of information, such as consumer health information, research tools, and links to other biological databases and repositories (linkout help, 2005). ncbi’s linkout service recently expanded to allow linking between pubmed and the publicly available content (i.e., full-text articles) held in institutional repositories (irs) (bastian and kwan, 2016). this allows users to access full-text publications in open-access repositories, provided they are not already freely available from a journal and/or via pmc. for irs that participate in the linkout for ir program, an ir-specific icon is visible in pubmed citation records (see figure 1). figure 1. screenshot of pubmed citation with link to full-text through iupui scholarworks (see icons) requirements for participating in ncbi linkout for irs in order to participate in linkout for irs, nlm requires basic information about the repository (e.g., name, url, contact information), documentation about the ir’s copyright policy, and identification (on the part of the ir) of the number of articles that have a pubmed citation, but do not have an existing publication freely available in pmc. as of earlier this year, irs that wish to participate in this service must have at least 1,000 articles that fit that criteria (i.e., pubmed citation, not in pmc). this requirement may present the first barrier for an ir’s participation in linkout, and readers should take this into consideration before applying to the service or implementing any of the strategies discussed in this paper. the full listing of requirements and contact information for ncbi’s linkout is available here: https://www.ncbi.nlm.nih.gov/projects/linkout/doc/ir-application.shtml. existing solutions there are several methods for identifying the number of items in an ir that have a citation record in pubmed and are not in pmc, but most require a combination of solutions in order to reach the desired result. additionally, the time intensiveness of certain methods may factor into their feasibility as a solution for some ir managers. in the following, we provide a brief overview of some existing methods in addition to documenting the process involved in developing our solution. the most straightforward method is manually searching pubmed to determine a) whether an article has a pubmed record and b) if that article is linked to the full-text publication in pmc. using the pubmed interface is arguably one of the most time intensive methods, particularly since a corpus of over a 1,000 items is now a requirement for participating in linkout for irs. the following methods are more sophisticated and leverage nlm ncbi tools and apis to automate portions of the task. this is not a comprehensive listing, but rather an overview of some of the well-documented ways of accomplishing this work: batch citation matcher this tool uses basic bibliographic information (e.g., author, journal title, year) to find, if available, pmids and pmcids for citations. this information is formatted into strings that can be entered into a text box on the batch citation matcher https://www.ncbi.nlm.nih.gov/pubmed/batchcitmatch page or uploaded as a separate file. id converter api this api allows for identification of multiple ids used by ncbi systems. in this case, using dois, one can identify corresponding pmcids, if available. the pmcid-pmid-manuscript id-doi converter https://www.ncbi.nlm.nih.gov/pmc/pmctopmid/#converter tool leverages the id converter api https://www.ncbi.nlm.nih.gov/pmc/tools/id-converter-api/  and allows input of ids (e.g., dois) into a text box, which then generates a file of corresponding pmcids. if a pmcid is available, the results will also list the corresponding pmid. however, this is limited to articles in pmc so if the doi is not found in pmc, no ids are captured. esearch query https://www.ncbi.nlm.nih.gov/books/nbk25499/#chapter4.esearch using the ‘esearch’ utility of the entrez utilities api provided by ncbi, one can check if an article is in pubmed and retrieve the pmids while also determining whether or not the same article is in pmc. for example,a query can be built using dois to search pubmed to identify corresponding articles while simultaneously excluding them if they are already available in pmc (see figure 2). results from these search queries are reported as xml. https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi?db=pubmed&term=10.1089/neu.2014.3454[doi]+not+pubmed+pmc+all+[filter] figure 2. sample esearch query using a doi and filter to not pmc while these solutions may ultimately speed up the process of checking ir articles against pubmed/pmc, using any of these methods requires varying amounts of time and effort in learning and establishing a sustainable workflow. in the next section, we provide a detailed breakdown of our solution. our solution in order to implement our solution, an ir manager must be able to generate a csv file from the repository that contains the following basic metadata: unique repository id, a url to the repository landing page for the article’s full text, and a doi for the article. certain repositories may make it easier to export the necessary metadata. for example, dspace (the repository software used by iupui) makes exporting item metadata and embargo dates in the same file difficult. we provide our solution to this problem. [1] our solution relies on r, an open source programming language for data analysis. we use two packages that are not part of the base r package, rentrez and plyr, which help us automate the process for looking up dois and returning their associated pubmed ids (pmids) and pubmed central ids (pmcids). the rentrez package is developed by ropensci https://ropensci.org/ and it enables r community users to interact with the ncbi e-utilities https://www.ncbi.nlm.nih.gov/books/nbk25501/ (entrez programming utilities help) directly in r environment (winter, 2017). e-utilities consist of nine server-side programs via web interface including: einfo esearch epost esummary efetch elink egquery espell ecitmatch the rentrez package provides functions that communicate with e-utilities via a rest api. some of the functions that rentrez package supports include the following: entrez_db() entrez_search() entrez_post() entrez_summary() entrez_fetch() entrez_link() among many useful functionalities, entrez_search() and entrez_summary() are incorporated in the r script presented in this article. the first step in our solution is to read a csv file of articles from the repository that may be eligible for the linkout service. minimal metadata is required for this solution and the csv file only contains columns for a dspace record id, handle url, and doi. when read into r, this csv file is transformed into a data frame called pmid_pmcid. data frames are a commonly used data structure in r that allow for the storing of lists of vectors of equal length. next, two empty columns are added to the data frame. these columns will hold the pubmed ids and pubmed central ids that are retrieved from e-utilities. pmid_pmcid <read.csv("filename.csv", header=true) pmid_pmcid$pmid <" " pmid_pmcid$pmcid <" " the next step is to identify pubmed ids (pmid) for articles available in the pmid_pmcid data frame by searching e-utilities using the articles’ dois. the function entrez_search() allows searching a designated database for records containing specific terms. for example, entrez_search(db=”pubmed”, term=”science”) returns 1,957,121 pubmed records that contain a match for the keyword ‘science’. entrez_search(db="pubmed", term="science") ## entrez search result with 1957121 hits (object contains 20 ids and no web_history object) ## search term (as translated): "science"[mesh terms] or "science"[all fields] in our solution, we developed a custom function mypubmedid(). this function searches e-utilities for a doi and returns the pmid for associated articles that are found. if no articles are found, the function returns a value of ‘none’. mypubmedid <function(x) { pubmed <entrez_search(db="pubmed", term=x) pubmed_id <pubmed$ids if (length(pubmed_id) == 1 ) { pmid <pubmed_id } else { pmid <"none" } return(pmid) } once the function mypubmedid() is declared in the r workspace, the function is called iteratively using lapply() function and returns either the pmid or ‘none’ for each record in the data frame. the result is saved as a list, which is a specific type of data structure in r. before the values in this list can be added to the pmid column in the data frame, it must be converted to a vector by using the unlist() function. pmid_list <lapply(pmid_pmcid$doi, mypubmedid) pmid_list <unlist(pmid_list) pmid_pmcid$pmid <pmid_list some of the articles fed into this process are not indexed in pubmed and ultimately we will ignore these articles. however, for some of the records, when there is no exact match for the doi provided to the mypubmedid(), e-utilities defaults to searching all fields. e-utilities takes the prefix of the doi and looks for a ‘best match’, which it usually finds in the pmid field of an unassociated article. for instance, searching for the dois 10.1179/2159032x15z.00000000038 and 10.1179/1059865014z.00000000074 returns pmid 15017085. mypubmedid("10.1179/2159032x15z.00000000038") ## [1] "15017085" mypubmedid("10.1179/1059865014z.00000000074") ## [1] "15017085" this tendency creates duplicate pmids for articles that are not actually available in pubmed if their doi prefix is identical. as an example above, e-utilites takes the doi prefix for some records and looks for a best match. our dataset has multiple dois with the same prefix and the duplicated pmid is returned for every doi with the same prefix if this article is not indexed in pubmed. to facilitate their removal, the duplicated pmids should be transformed to a value of ‘none’ using duplicated() function from the r base package and count() function from plyr package (wickham, 2011). duplicated_ids <data.frame(pmid_pmcid$pmid[duplicated(pmid_pmcid$pmid)]) # this returns n-1 colnames(duplicated_ids) <c("value") # change the column name duplicated_ids <subset(duplicated_ids, value != 'none') # remove none duplicated_ids <data.frame(count(duplicated_ids)) # change duplicated ids to none value pmid_pmcid <within(pmid_pmcid, pmid[pmid==23104645] <c("none")) in order to get pubmed central ids (pmcid), we could apply the entrez_search() that we used in the mypubmedid() function. however, this returns a pmcid only when the item is not under embargo in pmc and an empty value when the item is under embargo. instead of using entrez_search() for pmcid, we use the function entrez_summary(), which supplies information about an item with the provided parameters database and id. entrez_summary() provides not only pubmed central id of the article but also its embargo date. with the unique pubmed id identified in the previous step, we can use entrez_summary() in order to get summary information about the article, which includes the pubmed central id. for example, a new variable called summ1 contains information for entrez_summary(db=”pubmed”, id=”25622111″) and this returns the following information: summ1 summ1 ## esummary result with 42 items: ## [1] uid pubdate epubdate ## [4] source authors lastauthor ## [7] title sorttitle volume ## [10] issue pages lang ## [13] nlmuniqueid issn essn ## [16] pubtype recordstatus pubstatus ## [19] articleids history references ## [22] attributes pmcrefcount fulljournalname ## [25] elocationid doctype srccontriblist ## [28] booktitle medium edition ## [31] publisherlocation publishername srcdate ## [34] reportnumber availablefromurl locationlabel ## [37] doccontriblist docdate bookname ## [40] chapter sortpubdate sortfirstauthor one of the available objects from entrez_summary() is articleids. the variable summ1$articleids shows other article identifiers associated with the pmid, such as doi, pmcid, rid, eid, and mid. summ1$articleids ## idtype idtypen value ## 1 pubmed 1 25622111 ## 2 pii 4 2091742 ## 3 doi 3 10.1001/jamainternmed.2014.7667 ## 4 pmc 8 pmc4346513 ## 5 mid 8 nihms663485 ## 6 rid 8 25622111 ## 7 eid 8 25622111 ## 8 pmcid 5 pmc-id: pmc4346513;manuscript-id: nihms663485; in order to capture the pmcids for articles in the data frame, we create a custom function called mypmcid(). first, this function returns the summaries of all articles by using the entrez_summary() function and the pmids identified with the previous function. then, the mypmcid() function uses a regular expression, with the grep() function from the r base package, to parse the summaries and return the pmcid or a value of ‘none’, if no pmcid is available. mypmcid <function(x) { if (x == "none") { pmcid <"none" } else { taxize_summ <entrez_summary(db="pubmed", id=x) pmc <data.frame(taxize_summ$articleids) pmc_id <grep("pmc-id:", pmc$value, value=true) if (identical(pmc_id, character(0))) { pmcid <"none" } else { pmcid <pmc_id } } return(pmcid) } the mypmcid() is declared in the current r workspace before the function is called with lapply() function. this is essentially the same process used with the mypubmedid() function. pmcid_list <lapply(pmid_pmcid$pmid, mypmcid) pmcid_list<unlist(pmcid_list) pmid_pmcid$pmcid <pmcid_list in order to meet the requirements ncbi linkout for irs program, the datasets with pmid and pmcid identified are cleaned up. first, the items should be available in pubmed, so the records with a pmid value of ‘none’ are deleted. second, the items should not be available in pubmed central, which means the records with pmcids should be removed. # delete all rows with pmid value none pmid_pmcid <pmid_pmcid[ which(pmid_pmcid$pmid != "none"), ] # delete all rows with pmcid value not none pmid_pmcid <pmid_pmcid[ which(pmid_pmcid$pmcid=="none"), ] the full code is available in github. https://github.com/yooylee/pmid_pmcid conclusion by using the ‘rentrez’ package for r and the nlm’s national center for biotechnology information (ncbi) entrez utilities api, we are able to efficiently and accurately select repository records for inclusion in the ncbi linkout for irs program. after running the script across more than 4,400 repository records, we identified 557 items that were immediately eligible for a linkout from pubmed.[2] we also identified more than 100 items that will be eligible for linkout for irs inclusion when embargoes on fulltext access expire. processing time for this task in r requires less than 10 minutes. by our estimate, the task of manually searching pubmed using an article’s doi and recording its eligibility for pubmed linkout for all 4,400 records would require more than 22 hours of work. [3] librarians without a basic understanding of r and apis, may face a steep learning curve for using our approach. learning the basics of r, however, will provide skills that can be applied to other projects. furthermore, with our r script written and our process documented, we will be able to send future repository works to ncbi with very little effort. by reducing the labor of participating in the ncbi linkout program, we increase the likelihood that authors, readers, and the libraries will benefit from the program. by including a link in pubmed to a free fulltext version of an article in the repository, we are increasing the discoverability of an open access work authored by a faculty member on our campus. at the same time, increased discoverability provides incentives for campus authors to deposit a version of their articles in the institutional repository. in our case, we are already seeing an increase in web traffic to iupui scholarworks. soon after successfully sending 557 records to ncbi, pubmed became a leading source of web referrals to the repository–providing 17% of all referrals to the site, second only to the u.s. version of google scholar (24% of all referrals).[4] we also expect that repository works included in the linkout program will likely see increased downloads. in sharing our process for identifying eligible repository items for inclusion in the ncbi linkout for irs program, we hope that other libraries, repository managers, and staff will participate in the program. assuming that an institutional repository meets ncbi’s current requirements for participation, the process that we have documented here can save libraries the tedious effort of manually checking the eligibility of individual items. as more repositories join the ncbi linkout service, the increased discoverability provided by the service will reduce barriers to scholarly literature for readers around the world. notes [1] dspace makes it difficult to export metadata, including dois, along with embargo dates for those items. our approach to dealing with embargoed records is to compare two data frames by the same dspace id using the intersect() function. the data frame of the embargo contains dspace record id, handle and embargo date. this allows us to remove embargoed items before processing. # load csv file of items under embargo and create a data frame embargo # identify items under embargo and change column name items_under_embargo colnames(items_under_embargo) # remove items under embargo from pmid_pmcid data table pmid_pmcid [2] the 557 items deposited falls below the 1000-item threshold now required by the nlm. when iupui began this project, the 1000-item requirement was not in place. we committed to sending additional records on a monthly basis and expect to meet the 1000-item threshold soon. [3] to estimate how long manually searching pubmed might take, we timed ourselves searching for a random sample of 10 articles from our ir. for each article, we searched pubmed using the article’s doi followed by the [doi] field tag, ex. 10.5489/cuaj.4090 [doi]. we noted the presence of a pmcid, indicating ineligible articles, and recorded the pmid for eligible articles. on average, it took us 3 minutes and 19 seconds to search for the 10 articles. extrapolating from this average, it could take 5 hours 31 minutes and 40 seconds to search pubmed for 1000 articles. this calculation potentially underestimates the true amount of time it would take library staff to complete these searches, as an individual is likely to execute searches at varying speeds throughout a search session. when we compare this with our solution, which processes 1000 records in 2 minutes and 1 second, the time savings are evident. [4] the linkout to iupui scholarworks in pubmed was officially available on august 8, 2017. according to google analytics, 190 sessions came from pubmed between august 8 and august 16 while there was 0 traffic referral in the previous period (july 30 – august 7, 2017). this also increases the overall referral traffic by almost 40%. when we compared the same period of the last year (august 8 – august 16, 2016), there was no traffic referral from pubmed during this time. references bastian h, kwan y. 2017. institutional repository linkout: a new full text access feature in pubmed. nlm tech bull [internet]. [cited 2017 aug 15]; 2017 march-april(415):e3. available from https://www.nlm.nih.gov/pubs/techbull/ma17/ma17_linkout_institutional_repository_icon.html iupui open access policy [internet]. 2014. open access at iupui. indianapolis (in): indiana university – purdue university indianapolis; [cited 2017 sep 1]. available from: https://openaccess.iupui.edu/policy linkout help [internet]. 2005. bethesda (md): national library of medicine; [cited 2017 aug 15]. available from https://www.ncbi.nlm.nih.gov/books/nbk3805/. wickham h. 2011. the split-apply-combine strategy for data analysis. j statistical sftw [internet]. [cited 2017 aug 17];40(1):1-29. available from http://www.jstatsoft.org/v40/i01/. winter d. 2017. rentrez: entrez in r. r package version 1.1.0 [internet]. berkley (ca): ropensci; [cited 2017 aug 15]. available from https://cran.r-project.org/package=rentrez about the authors yoo young lee http://orcid.org/0000-0002-9867-6070 is the digital user experience librarian and it analyst developer at iupui. she works with the university library’s center for digital scholarship to provide skills and knowledge in user experience, application development, project management, and data analysis. erin d. foster http://orcid.org/0000-0001-6908-9849 is the data services librarian for the indiana university school of medicine. she works, in collaboration with the university library’s center for digital scholarship, to develop policies, services, and infrastructure that enable faculty and students to manage, share, and preserve their research data at iupui. david e. polley http://orcid.org/0000-0003-3595-9708 is the social sciences & digital publishing librarian at iupui. he works with the university library’s center for digital scholarship, managing the center’s open access journal publishing service and providing consultation services in data analysis and visualization. jere odell http://orcid.org/0000-0001-5455-1471 works as a scholarly communications librarian to support open access publishing practices. at iupui, he works with the university library’s center for digital scholarship to implement the iupui open access policy, to manage the library’s open access publishing fund, and to provide other library publishing services. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – solving advanced encoding problems with ffmpeg mission editorial committee process and structure code4lib issue 25, 2014-07-21 solving advanced encoding problems with ffmpeg previous articles in the code4lib journal touch on the capabilities of ffmpeg in great detail, and given these excellent introductions, the purpose of this article is to tackle some of the common problems users might face, dissecting more complicated commands and suggesting their possible uses. by josh romphf introduction ffmpeg has been an important encoding tool for technologists, preservationists, and hobbyists for well over a decade. it is a powerful, multi-purpose open-source library that operates from the command-line, and while a basic knowledge of the command-line environment is useful, ffmpeg can be effectively utilized with little to no programming experience.   the source files are easily compiled on all major platforms, and static builds are available for those who prefer to install from an executable[1]. two articles in code4lib issue 23 touch on the capabilities of ffmpeg in great detail[2], and given these excellent introductions, the purpose of this article is to tackle some of the common problems users might face, dissecting more complicated commands and suggesting their possible uses. to refer to the ffmpeg community as an active one is a vast understatement, and it not surprising how quickly one can fall down an encoding rabbit hole when trying to search forums for solutions to their challenges. this article serves to provide some useful suggestions for transcoding techniques within a specific library or archival context. note: the following commands rely heavily on videolan’s high performance x264 library, which is licensed under the gnu gpl and can be compiled with ffmpeg, but is not added to the standard build. the x264 library is used for encoding h.264/mpeg-4 avc, and undergirds some of the most high profile streaming operations on the web, including youtube, vimeo, and hulu. when compiled with ffmpeg, it is capable of high quality compression at relatively high speeds. understanding your project’s needs before embarking on an encoding project, consider whether or not ffmpeg is the right tool to be used. for instance, ffmpeg is excellent for batch processing large collections, but is not ideal for detail oriented digital restoration work. similarly, it may prove to be overkill for simple transcoding projects, such as on-demand delivery of a single file for a patron.  it may be more efficient to use a transcoding program such as handbrake (which is built on ffmpeg and x264), which has a gui and significantly less of a learning curve, instead of  a command-line application that can give you highly customizable results. one of the key steps in helping to make this call is defining the desired output. while this may sound obvious, it is important to tailor your tools to the task at hand: what is the method of delivery? are you producing preservation quality files? are you streaming? what type of player will be used? does file size matter? if so, is there an optimal file size? for those who opt for ffmpeg, what follows are some scenarios accompanied by possible solutions. preservation and access basics: input, output, preservation, and streaming arguably the most common use of h.264/mpeg-4 avc encoding is for streaming and access solutions, but it can also be used to transcode lossless preservation masters and high quality mezzanine files[3]. when the ultimate goal is preservation, the x264 codec supports lossless encoding, yet it should be noted that a number of streaming services, with the exception of youtube, are capable of decoding lossless h.264. in addition, a new container format may be desired for posterity; one such format is matroska (mkv), an open source container designed with the intention of becoming the defacto standard for multimedia containers. it has become known for its support of  “all known video and audio compression formats” and has been adopted by a number of archival projects [5]. as libraries and archives begin to digitize their moving image collections and provide online access to them, the quality of streamed files becomes more and more important. depending on whether or not the file size of the output is an issue, there are a few paths one can take with x264. 1) transcoding for quality: ffmpeg -i input -c:v libx264 -preset slow -crf 18 -vf yadif -strict -2 output.mp4 when file size is not an issue, this is a quick command that will produce a visually lossless h.264 encoding with an mp4 container. c:v is the video codec of choice, preset is the compression preset (in this case slow for higher quality compression), and crf is the constant rate factor, which preserves an overall level of quality throughout the file by adjusting each frame’s bitrate based on the given quality level. consequently, the higher the crf, the lower the overall quality level. the video filter flag (-vf) is used to call ffmpeg’s pre-bundled video filters, while yadif (yet another deinterlacing filter) deinterlaces an interlaced input, as progressive video is not only easier to compress and most current computer monitors and televisions are progressive scan. finally, when encoding with h.264/mpeg-4 avc, the audio format used is aac (advanced audio coding); in order to enable ffmpeg’s experimental, native aac encoder, -strict -2 needs to be added to the command. an external library such as libfaac [4] can also be used, and –strict -2 can be omitted. a preservation quality master can be output using the same principles, setting the –crf to 0, the preset to veryslow, and the output container to .mkv: ffmpeg -i input -c:v libx264 -preset veryslow -crf 0 -vf yadif -strict -2 output.mkv a preservation quality master can be output using the same principles, setting the –crf to 0, the preset to veryslow, and the output container to .mkv: ffmpeg -i input -c:v libx264 -preset veryslow -crf 0 -vf  yadif -strict -2 output.mkv 2) streaming for file size: in a scenario where server space or bandwidth is limited, a desired file size may be required. to calculate the output file’s bitrate based on a target size, the following equation can be used: bitrate = output file size in mb * 8192 (convert mb to kb) / duration in seconds from there, two-pass, variable bit rate (vbr) compression can be used to yield an approximate file size: ffmpeg -i input -c:v libx264 -preset slow -b:v (calculated bitrate) -strict -2 -pass 1 -f mp4 /dev/null && ffmpeg -i input -c:v libx264 -preset slow -b:v (calculated bitrate) –strict -2 -pass 2 output.mp4 in the above example, the first command is passed to a null device, the && operator then executes the second command, which yields the mp4 file. 3) streaming within a browser if a streaming service such as youtube or vimeo is not being used, an option is to add –movflags faststart to the command, which shifts the moov atom (the data unit that defines the file’s timescale, duration, and information for each track)[7] to the beginning of the mp4 container. this allows for playback and seeking to be started before the video has fully downloaded – a feature that is particularly useful when streaming content.  although some delivery mechanisms do not require the position of the moov atom to be located at the beginning of the container, it is required by both flash video (.flv) and html5 video, which are two of the most common web-based video options available. most other web-based players will be able to function regardless of the moov atom’s position. concatenating files another essential use of ffmpeg lies in its ability to concatenate (join) multiple video files. the need to concatenate may arise when content is ingested in the form of multiple files, when a patron requests a sequence of clips, or when an image sequence needs to be animated. the use of the concat filter is outlined in automated processing of massive audio/video content using ffmpeg, and is suitable when concatenating at the file level, yet it can be lossy and uncooperative with some containers, especially mp4. the most straightforward means of concatenating files produced with the same codec is by using the concat demuxer, which calls on an external .txt file for orders on which files to join and in what order. the text file should appear as follows: #list of files to join (comment) file ‘/usr/path/to/file1’ file ‘/usr/path/to/file2’ the command itself is quite simple, and can be used as a stream copy for efficiency. –c copy moves all of the streams over to a new container without decoding and encoding the input: ffmpeg -f concat -i /path/to/list/concatlist.txt -c copy out.mp4 the unix cat command can also be used for concatenating files, and is particularly helpful when pulling .vob files together from a dvd. a handbrake alternative, this method is ideal for transcoding to containers not offered by the program, such as .mov and .mp4: cat /volumes/yourdvdtitle/video_ts/vts_01_[1234].vob | ffmpeg -y -i -c:v libx264 -preset slow -crf 18 -strict -2 output.mp4 the .vob files are concatenated in order, where [1234] corresponds to the filename; for instance, to concatenate vts_01_1.vob and vts_01_2.vob, the command is cat /volumes/yourdvdtitle/video_ts/vts_01_[12]. this is then piped to ffmpeg, setting the .vob files as the input. the user can then customize the remainder of the command in any way. another means of concatenating files with the same codec is the concat protocol in ffmpeg, which works at the file level as opposed to the stream level. unlike the unix cat command, the concat protocol uses pipes “|” to delineate input files: ffmpeg –i “concat:input1|input2|input3” -c copy out.mp4 be aware that some codec and container combinations may not work properly. for instance, h.264 encoded files cannot be losslessly concatenated in ffmpeg. there is, however, a workaround for this which is achieved by first copying them to mpeg2 transport streams (.ts). using –c copy will yield an error that can be solved by the use of a bitstream filter (–bsf), which performs modifications at the bitstream level, foregoing any decoding and producing the same results. the transport streams can then be written to a temporary directory, concatenated, and removed: ffmpeg -i input1.mp4 -bsf:v h264_mp4toannexb -f mpegts -c:v mpeg2video -b:v 2500k /path/to/temp/directory/intermediate1.ts && ffmpeg -i input2.mp4 -bsf:v h264_mp4toannexb -f mpegts -c:v mpeg2video -b:v 2500k /path/to/temp/directory/intermediate2.ts && ffmpeg -probesize 100m -analyzeduration 250m -i "concat:/path/to/temp/directory/intermediate1.ts|/path/to/temp/directory/intermediate2.ts" -c:v libx264 -crf 18 -preset slow -strict -2 -movflags faststart output.mp4 && rm -rf /path/to/temp/directory the use of –probesize and –analyzeduration help ffmpeg to recognize the audio and video stream parameters of a file, and while they are by no means necessary, their use can help reduce input processing errors, especially when moving across codecs. finally, libx264’s powerful animation capabilities can prove incredibly beneficial for preservation projects. for instance, film scans that produce individual frames (for instance, cineon, jpeg, dpx or tiff sequences) can be recombined and compressed for access. another use could be for pulling jpeg2000 images from a dcp (digital cinema package) and recombining them for viewing. to join image files, use the command: ffmpeg -r 24/1 -i filename%04d.jpg -c:v libx264 -r 24 -pix_fmt yuv420p output.mp4 where –r 24/1 refers to the duration of each image, in this case 1/24th of a second, filename%04d.jpg finds all jpeg files with filename followed by a 4 number sequence (i.e. filename0001.jpg), and –r 24 refers to the framerate. –pix_fmt is also important, as the pixel format must be set; in this case, the yuv color space with 4:2:0 chroma subsampling is used to maximize playback compatibility. ffmpeg also supports globbing for image sequences, where –i is supplemented with –pattern_type glob –i, whereby wildcards (i.e. *.jpg) can be used. scripting and batch processing  an even greater level of automation can be achieved by scripting ffmpeg and unix commands, freeing up time for other tasks. all of the above commands can be scripted and combined using a number of scripting languages. for example, a script can be written to batch transcode a preservation mkv and streaming optimized mp4 from any type of source file. two of the more popular methods are to use a conventional shell script or a python script with subprocess calls. for instance, a simple shell script with a for loop can be written as such: #!/bin/bash #convert all mov files to mp4 in a directory for x in /path/to/files/*.mov; do ffmpeg -i “${x}” -c:v libx264 -crf 18 -preset slow –strict -2 –pix_fmt yuv420p “$x.mp4”; #insert other commands, such as moving files or creating a lower quality copy done the file is then saved with the .sh extension and can be opened in a bash shell. this is by far the simplest method of the two, as it is language that ffmpeg users are already familiar with. python can be used for more complex scripting, such as reading files through ffmpeg, processing them with python, and piping them back out to ffmpeg for an output. with the inclusion of the os and subprocess modules, one can execute shell commands with all the functionality of a shell script. subprocess calls can also be couched within a for loop or stored in variables for more efficient coding. a simple python script with multiple commands would be written as follows: #!/usr/bin/env python import os import subprocess #script to compile image sequence and add audio #if ffmpeg is not in $path, use os.chdir(‘path/to/ffmpeg/binary/directory/’) #compile image sequence subprocess.call([ 'ffmpeg', '-r', '24/1', '-i', 'filename%04d.jpg’, ‘-r’, ‘24’, '-c:v', 'libx264', '-preset', 'slow', '-crf', '18', '-pix_fmt', 'yuv420p' '-s', '1920x1080', ' output.mp4']) #add audio track to output video subprocess.call([ ‘ffmpeg’, ‘-i’, ‘output.mp4’, ‘-i’, ‘audiotrack.wav’, ‘-c:v’, ‘libx264’, ‘-c:a’, ‘aac’, ‘-strict’, ‘-2’ ‘-b:a’, ‘-192k’, ‘composite.mp4’]) the script is saved with the extension .py, and can be run in the same manner as any other python script: python /path/to/script/videoscript.py conclusion one can imagine the possibilities of ffmpeg, especially when combined with a powerful, yet accessible programming language like python. this article merely scratches the surface of what can be done. an indispensable tool for libraries and archives – especially those with limited technical staffing –  ffmpeg can be utilized to solve nearly all the needs of a digital video project. again, the learning curve can be daunting, and the end result is largely what you make of it. it is hoped that this article served as a gentle introduction to both video encoding principals, as well as a companion for some of ffmpeg and x264’s idiosyncrasies. please consult the references section for more specific, in-depth commands and explanations. notes [1] http://www.ffmpeg.org/download.html features downloads for nearly all operating systems, and https://trac.ffmpeg.org/wiki/compilationguide provides easy to follow compilation guides. [2] automated processing of massive audio/video content using ffmpeg (kia siang hock and li lingxia, january 2014) contains several important, common commands and provides explanations, while unix commands and batch processing for the reluctant librarian or archivist (anthony cocciolo, january 2014) explains how to batch transcode files using ffmpeg commands with a bash for loop. [3] mezzanine files are intended to be used as your input when transcoding access files to lessen the risk of damaging the master. [4] libfaac (freeware advanced audio encoder) –enable-libfaac –enable-nonfree [5] the matroska (http://matroska.org/technical/specs/index.html) container format has gained a great deal of momentum over the last few years, and has been adopted as the preservation container within the infrastructres of open source digital asset management systems such as archivematica (https://www.archivematica.org/wiki/main_page) and islandora (https://discoverygarden.ca/). [6] visit http://trac.ffmpeg.org/wiki/how%20to%20quickly%20compile%20libx264 for instructions on how to easily compile x264 with ffmpeg. as a side note for mac users, the x264 library (and a number of other dependencies) are included in the homebrew installation. [7] for an excellent article on the moov atom, see understanding the mpeg-4 movie atom (maxim levkov, adobe developer connection, october 2010) http://www.adobe.com/devnet/video/articles/mp4_movie_atom.html. references: several ffmpeg tutorials are available through the official bug tracker and wiki: https://trac.ffmpeg.org/. another helpful resource is video engineer fabio sonnati’s excellent ffmpeg – the swiss army knife of internet streaming (http://sonnati.wordpress.com/2011/07/11/ffmpeg-the-swiss-army-knife-of-internet-streaming-part-i/). videolan also maintains excellent documentation on the x264 library: http://www.videolan.org/developers/x264.html. as always, stack overflow, albeit somewhat difficult to navigate, offers a great deal of tips and tricks http://stackoverflow.com/questions/tagged/ffmpeg. after dawn http://www.afterdawn.com/ also maintains a forum for video encoding related questions and issues. about the author josh romphf is a programmer in the digital humanities center at rush rhees library (university of rochester), specializing in web development, 3-d modeling, video encoding, and preservation solutions. subscribe to comments: for this article | for all articles one response to "solving advanced encoding problems with ffmpeg" please leave a response below: sanchit sharma, 2020-12-20 hi josh, this journal really help me a lot, but i have a question for you. as i am worked on ffmpeg before i’ve stuck on that particular part at that time. so, here is a scenario i have 100’s of small chunks of an video but they’ve different bitrate (audio), and when i am trying to merge then they merge without any error, but issue is that after the merger the 3 minutes of video becomes 9 minutes. what do you what will be the issue here, if there is any unique or unknown command that i didn’t know, then please let me know. thanks, sanchit sharma leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – better search through query expansion using controlled vocabularies and apache solr mission editorial committee process and structure code4lib issue 20, 2013-04-17 better search through query expansion using controlled vocabularies and apache solr this article describes how the university of pennsylvania museum of archaeology and anthropology (penn museum) modified its solr-based discovery interface to improve recall and enable end users to benefit from the power of their in-house controlled vocabularies. these modifications automatically expand the query generated by any search term that matches their controlled vocabulary to include all related alternate and narrower terms. for example, if a user enters ohio, that search will retrieve the record for an arrowhead found in cincinnati (a narrower term of ohio) even if that record does not include the term ohio. by scott williams introduction this article discusses one solution to a very common problem in resource discovery. how do museum professionals make it easy for the general public to benefit from the power of rigorous, discipline-specific controlled vocabularies? museums have confronted the issues of establishing and conforming to descriptive metadata standards and the role of controlled vocabularies in different ways. some build their own controlled institutional standards, some rely on external standards like the getty’s art and architecture thesaurus[1], the thesaurus of geographic names (tgn)[2] or the nomenclature 3.0 for museum cataloging (chenhall)[3] and others prefer the freedom and flexibility of uncontrolled descriptive metadata. these variations lead to inconsistent descriptive metadata being applied to similar objects in different institutions.  when institutions make their object metadata searchable online, these variances pose significant barriers to discovery for users. this problem is heightened in a research institution, like the university of pennsylvania museum of archaeology and anthropology (penn museum), where metadata for the 330,000 object records is often discipline-specific, and the vocabulary used does not overlap with terms that would be familiar to an average user. users are then left to guess at keywords and hope that all the relevant descriptive information exists within the item they want to find.  over the last year, penn museum has confronted these issues as we have worked to reconcile our legacy as a research institution and the mandate to be a museum with a strong focus on public education and make the information about the objects in the collection more accessible. in this article, i will explain how our approach utilized our internal hierarchical vocabulary and apache solr in order to provide a more inclusive search, resulting in improved resource discovery in our online collection (penn.museum/collections). history in the early 1980s, individual curatorial departments at penn started to implement content standards for cataloging objects. curators developed “approved terms lists,” but lacked a mechanism for enforcing these standards. in 1993, penn museum began using questor systems’ collections information management system (cims), argus. in the museum world, a cims is a repository and cataloging management application. in addition to the standard functionality of object cataloging, it is used to manage and document a range of collections management related activities such as accessions, deaccessions, loans, insurance, inventories, exhibitions, scientific testing, and conservation. the argus cims also provided a robust hierarchical vocabulary management system, a lexicon, and penn museum used this lexicon to develop, refine and build upon the existing approved term lists.  the lexicon allowed users to add non-preferred terms and link them to a preferred term. when a non-preferred term was entered by cataloger it would be replaced by the preferred term, data entry was controlled and internal standards enforced. in addition to the lexicon, questor provided a terms list of broad concepts and in the early 1990s, the original museum-developed approved term lists were integrated into the questor supplied terms list. over the last thirty years this hierarchical vocabulary has grown and today contains 62,000 local terms as well as a modified version of the 1978 chenhall nomenclature. today, penn museum uses ke emu[4] as a cims but the role of the vocabulary has remained the same. the vocabulary controls the content in core object cataloging fields: object name, provenience (a term used by archaeologists to describe where an object was excavated), material, technique, culture, period, dynasty, subject, iconography, maker, royal name, and manufacture location. museum staff rely heavily on the vocabulary for cataloging objects and searching. within the cims, when a field that is controlled by the vocabulary is searched, the system expands the query to include all narrower, non-preferred and alternate terms.  a single search term like “greek” might actually search for hundreds or thousands of additional terms in the background without the user being aware of what terms are included in the search. over the last twenty years, museum catalogers have become accustomed to describing an object with limited record-level metadata. the search functionality provided by the cims and structured vocabularies in the lexicon allowed catalogers to only enter the most granular information in the object record and the lexicon worked in the background to help searches. for instance, if a user were to search for “ohio” in the provenience field, the cims would expand the query search for all the terms organized underneath ohio in the lexicon, e.g. “cincinnati,” “hamilton county (ohio),” “cleveland,” “porkopolis” and “columbus (ohio).” objects in the collection whose provenience was described using only one term, say “cincinnati,”  would be returned by a search on “ohio” because of this expansion. the cims vocabulary management, object cataloging and search systems worked together seamlessly. users could find objects even though the item level record did not contain the term that was searched upon. problem when the object catalog data was put online in january 2012 (penn.museum/collections), long-time users of the cims were frustrated to find that their search would only return records that contained the exact terms in their search — not narrower representations or alternate forms. for instance, a search for “egypt” wouldn’t return an item that had been cataloged as being excavated in “misr” (the romanized spelling of the arabic name for egypt) or “mit-rahina” (a town in egypt). this was a complete departure from how the cims search has worked for twenty years. it was as though huge portions of the collection had been un-cataloged because the data was now in a context where resource discovery required all the relevant object metadata needed for discovery to exist in each record. in the months following the initial site launch, we analyzed search logs and found that 83% of the top 400 search terms matched an existing term in our vocabulary. of those matching terms, the median number of search results was forty-five objects. however, each term had, on average, thirteen narrower or alternate terms, and if those additional terms were included in the query the median number of results jumped to 123. the benefit of expanding queries was made particularly clear for the search “greek,” which is one of the most frequently run searches. with the old full text search system only 1,400 objects were returned, but if narrower terms like “aegean,” “late minoan,” and “mycenaean” were included, nearly 7,000 objects were returned. this expanded result set is much closer to the actual size of our greek collections than 1,400 objects and presents a much more complete picture of “greek” objects in the collection. we were able to use the existing vocabulary structure and apache solr to create a search field that would replicate the functionality already present in the cims and return objects that matched the searched-for term, alternate forms, and narrower terms. how? the typical implementation of solr’s synonymfilterfactory involves indexing equivalent but variable formats of a term or normalizing searched-for-terms to a standard form. in our implementation, the synonymfilterfactory was used to index the occurrence of each thesaurus term with all of its broader and alternate forms. <!- for query expansion searching --> <fieldtype name="queryexpansion" class="solr.textfield" positionincrementgap="100"> <analyzer type="index"> <tokenizer class="solr.keywordtokenizerfactory"/> <filter class="solr.synonymfilterfactory" synonyms="index.hierarchy.txt" ignorecase="true" expand="true" tokenizerfactory="solr.keywordtokenizerfactory"/> </analyzer> <analyzer type="query"> <tokenizer class="solr.keywordtokenizerfactory"/> <filter class="solr.lowercasefilterfactory"/> </analyzer> </fieldtype> the first step in the process of creating the index.hierarchy.txt file is to move the vocabulary data from ke emu to solr. emu comes with an api, imu, and using in-house developed scripts and this api, the entire vocabulary is exported into solr. each term record in emu is added as a record to solr as well as the term metadata (description, broader terms, etc.) this is done for two reasons. first, having the data in solr makes the process of creating the index.hierarchy.txt file for synonymfilterfactory much faster. second, one of our future development goals is to make the vocabulary a more visible and tightly integrated component of the online collections and having the data in solr would be required to implement those features. <doc> <str name="term">athena</str> <str name="description">goddess of wisdom</str> <str name="emuirn">12786</str> <str name="key">upmaa_emu_ethesaurus_12786</str> <arr name="termhierarchy"> <str>1:48288:42506:45572:2385:5750:12101:29968:3917:47172</str> <str>28673:3917:47172</str> </arr> </doc> solr record for the term record representing the greek goddess athena after all of the terms are inserted into solr, the index.hierarchy.txt file is created using the vocabulary data in solr. the values in this file explicitly index each occurrence of a term with its alternates, non-preferred versions, and parent terms, up to and including the root term of the vocabulary. for the purposes of query expansion, the order of these terms does not matter. using the previous example, this is the row for athena in index.hierarchy.txt which indexes every occurrence of “athena” with the broader terms “greek god,” “greek mythology,” and “olympian.” athena=>athena,athena (uncertain),base lexicon,belief and tradition,classical mythology,god,god (uncertain),gods,greek god,greek mythology,greek pantheon,mythology,mythology by culture,olympian,olympians,upmaa lexicon with the original full text search, a query for “greek god” would not return any object records because we do not describe iconography or subjects at such a broad level. however, with query expansion 1,096 object records are returned because the more specific terms that are used in the catalog records have been indexed with all of their broader parent terms. homographs how did we deal with the problem of homographs producing false or misleading results? we didn’t address this problem directly because the problem is mitigated by requiring primary and alternate terms to be unique. if a new term is added to the vocabulary and it is a homograph of an existing term, the cims and internal data standards force the user to disambiguate the new term by appending some clarifying term (e.g. “memphis (egypt)” and “memphis (tennessee)” and “memphis (greek goddess)”). while this can make fully utilizing the query expansion tool more difficult for users because they won’t know how the term has been disambiguated, it reduces the frequency and number of false homograph matches. but in the end the problem of searching “memphis” and getting all three types of memphis is the same as what happens with a typical full text search. in the previous example search of “greek god,” object number 16706 (http://www.penn.museum/collections/object/67074), was included in the result set because it contains the word “atlas” in the description and iconography fields. when that data was indexed by the queryexpansion field, “atlas” was indexed to multiple parentings because the term was never disambiguated and exists as a homograph for what should be three distinct terms in emu. the term is tripled parented in emu as type of map, greek deity, and cervical vertebra. in our testing we found that the number of false hits are relatively small compared to the increase in relevant, useful results and the overall accessibility to our collections data outweighs these inaccurate cross-references. notes on testing at the time of writing, we were providing 332,025 object records online but a search for “upmaa lexicon” returns 331,884 objects. this means that those 141 objects which are not returned by the search “upmaa lexicon” have only been catalogued with terms that are not parented to the root term, “upmaa lexicon”. identifying these records and unparented terms within the cims would be impossible but with solr we can, and we are working to fix those problem records and terms. additionally, excluding branches of the hierarchy can be achieved by adding a minus sign (the standard lucene negation operator) in front of the term you want to exclude. for example, to search for baskets from north america but not from the state of california enter:        “north america” -“california (state)” basket penn museum’s vocabulary is not exempt from bad legacy data and contains some terms that have a comma as a part of the term.  this creates a problem for the index text file because the synonymfilterfactory uses a comma to separate terms and having a comma in the term creates very real problems for searches. we are addressing this through data cleanup (removing commas and other punctuation from terms) and revised internal data standards. as noted in the solr documentation for the synonymfilterfactory[5], a common pitfall of expanding synonyms is inaccurate relevancy rankings for synonymous terms because the default behavior of solr is to treat less frequently used terms with more weight that more commonly used terms. this ranking strategy is known a inverse document frequency (idf). we have attempted to mitigate this problem by combining our queryexpansion fields with standard full text fields in a dismax[6] query parser. dismax is a customizable feature in solr that makes searching across multiple fields much easier. it also allows users to specify how those searches are ranked. our implementation of dismax boosts the results from the full text search fields, especially the object name, material and provenience fields over those from the queryexpansion fields. without heavily weighting the results of a full text fields the top results of a search for “vessel” would be records that use the less commonly used terms like phiale, larnax or balsamarium and records that use more common term “vessel” would be at the bottom of the results list. by boosting searches on the full text fields the top results are records that contain the term “vessel” and not the less frequently used forms of “vessel.” while this strong-armed manipulation of the dismax query parser may not be ideal, this method has mitigated the idf problem inherent with expanding synonyms. future development the next steps for improving resource discovery will certainly involve our vocabulary, exposing more of the structure to the public and making it a more visible part of our online collections. users would benefit from seeing the terms in context (what are the broader or related terms?). providing the vocabulary in a clickable, browsable tree structure would allow users to see what the top level terms are because currently there is no way for them to know they need to search for “armament t&e” and not “weapon” to find all weapons. another possible supplement to the existing user interface would be to provide users with the search results of the vocabulary along with their object results (prototypes shown below). showing them terms in the vocabulary that are similar to their searches as well as all of the narrower terms of their search (if their searched-for-term matches a term in the vocabulary) could provide useful indirect exposure to our content standards and make discovery more “clickable.” figure 1. suggesting similar and more specific search terms using the controlled vocabulary. this type of interface would also help address the problem of homonyms. when a user searches for a homonym, like “memphis,” they would then be prompted choose the most appropriate term from the vocabulary and by using the more specific term be able to take full advantage of the query expansion functionality. figure 2. using the controlled vocabulary to differentiate between homonyms. conclusion while there is certainly room for improvement, our approach to the problem of resource discovery in a diverse museum collection was built upon existing content and functionality. we took a feature that has been at the core of our museum practice and replicated it online with little cost or overhead and improved the discovery process for users searching our online collections. end notes [1]. http://www.getty.edu/research/tools/vocabularies/aat/ [2]. http://www.getty.edu/research/tools/vocabularies/tgn/index.html [3]. http://aaslhcommunity.org/nomenclature/ [4] http://www.kesoftware.com/emu [5] http://wiki.apache.org/solr/analyzerstokenizerstokenfilters#solr.synonymfilterfactory [6] http://wiki.apache.org/solr/dismaxqparserplugin about the author scott williams (scottwi@upenn.edu) is the collections database administrator at the university of pennsylvania museum of archaeology and anthropology. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – look what we got! how inherited data drives decision-making: unc-chapel hill’s 19th-century american sheet music collection mission editorial committee process and structure code4lib issue 13, 2011-04-11 look what we got! how inherited data drives decision-making: unc-chapel hill’s 19th-century american sheet music collection have you inherited a digital collection containing valuable, but inconsistent metadata? and wondered how to transform it into a usable, quality resource while accepting that it can’t meet your idea of perfection? this article describes such an experience at the university of north carolina at chapel hill university library with its contentdm-based 19th-century american sheet music collection, addressing issues such as field construction, the use of controlled vocabularies, development of a project data dictionary, and metadata clean-up. by renée mcbride introduction in december 2009 the university of north carolina at chapel hill (unc-chapel hill) university library completed migrating its 19th-century american sheet music collection – a collection of approximately 3,500 popular vocal and instrumental titles from the 19th century – from a locally developed php application backed by a mysql database to contentdm, a software package enabling libraries to manage and provide access to their digitized collections. unc-chapel hill adopted contentdm as its digital asset management system in 2005. since then we have engaged in a broad effort to eliminate digital silos (e.g., collections in access, filemaker pro, mysql) by migrating them to contentdm or our millennium-based library catalog, and making them available through our information access platform endeca, depending on the nature of the collection. our goals in this area are to offer our users easier access to digital collections, to create more consistent workflows and interfaces across the collections, to extend the benefits of enhancements of one collection to all collections, and to achieve interoperability. background a brief history of our project is necessary to provide the context in which our migration and project decision-making took place. in the 1940s or early 1950s, 125 bound volumes of sheet music came to unc-chapel hill’s wilson library, followed by a few additions in subsequent years, resulting in a total of 131 volumes, now housed in the unc-chapel hill music library. cataloging of various sorts occurred in the early 1950s and during the 1960s, and a major scanning and cataloging effort took place in the mid-1990s. in 2004 the collection was migrated from static html pages to a locally developed php/mysql application and provided with a custom-made public interface. scanning and cataloging continued in fits and starts and with a variety of staffing over the next few years. in 2008 serious planning for the migration of this collection to contentdm began. a student from unc-chapel hill’s school of information and library science (sils) exported the data from mysql into a format that contentdm could read and began a project wiki that we use to this day. another sils student loaded a backlog of previously scanned images – many of less than ideal quality – into contentdm and helped with the process of rescanning images and getting them into contentdm. the purpose of rescanning was to replace poor-quality with high-quality images. the rescanned images were created and stored in our digital archive as tiff images, then converted to jpeg2000 in contentdm, allowing for features such as loading only part of an image when zooming in on an image. in january 2009 i assumed management of the project. at that point in time we were in the process of rescanning images of music for which metadata had already been created. once a volume was rescanned, the images were uploaded into contentdm with their metadata. rescanning was completed in april 2010. as of the writing of this article 121 of the 131 total volumes are completed, and we anticipate finishing all cataloging by june 2011. as you may imagine, there was something of a lack of consistency in workflow, supervision, policies and procedures, and staffing over the many years this project has been in existence. by extension, there was also a lack of consistency in the quality of the metadata. project staff ranged from librarians to sils students to musicology graduate students. once in a while, we struck gold with staff experienced in both cataloging and music. students with systems expertise were invaluable in helping us learn how to scan, organize and transfer files, and work in contentdm, but were rarely involved in creating metadata. the uncertainty of our staffing was a driving force behind some of the decisions we made in this project. the other primary factor was the amount of metadata of uneven quality that we inherited. two broad areas required decision-making and subsequent implementation before we could entertain the thought of going live with the contentdm version of this collection: the technical area, responsible for the content of the collection, and the public interface area, responsible for offering an attractive and friendly product to our users. my primary area of responsibility was the technical area, although i consulted closely with our public services music librarian on both technical and public interface issues throughout the decision-making process. additionally, we received invaluable input from student staff and systems librarians. this article focuses on the following technical issues and areas of decision-making: which fields to retain, delete, or change; which dublin core fields our contentdm fields would map to; which fields required controlled vocabularies; creating and maintaining a project-specific data dictionary; which fields could reasonably be cleaned up, and which we would choose to live with (or, as one of my systems colleagues put it, which fields offered us an “acceptable level of embarrassment”); and, decisions affecting browsability. field construction and dublin core mapping contentdm allows complete freedom in the naming of fields. at unc-chapel hill we maintain a general digital collection data dictionary that is dynamic, enabling the addition of fields as required by collections with specialized information, e.g., latitude and longitude for map collections, or artist for art-related collections. a very few of our digital collections exist in non-compliance with the data dictionary, for reasons such as the inheritance of metadata of uneven quality in an otherwise valuable collection, precisely the case with our 19th-century american sheet music collection. once we determined that this collection would exist, at least for the time being, outside our ideal digital collection world, we began the process of making decisions about its content. the collection’s original 41 fields now number 35. figure 1. contentdm fields some decisions were fairly simple matters, such as deleting fields determined to have no use and containing highly inconsistent and confusing metadata. examples of such fields were series instrumentation, series dates, and series description, none of which existed in any recent project staff’s memory. over the years the terms “series” and “volume” appeared to have become tangled up with each other, so that by the time we were looking at these fields, they simply made no sense. we also did away with publisher’s quote, colophon, and advertisement, which contained no valuable metadata. we reduced five title fields to three fields, incorporated the cover design field into the illustration field, and changed the names of two fields: cover inscription became inscription, and subject became topic; these fields are discussed in more detail below. an exception to our general policy of deleting fields of questionable use and containing inconsistent and confusing metadata is the field labeled unknown – by whom, when, and for what purpose no one knows. after determining that this field contains a variety of possibly useful information, we decided to retain the field but hide it from public view and not use it for current cataloging. we hope to be able to extract useful metadata from this field at a later time and move it to appropriate fields to enhance description and access. we added a few fields to bring the collection partially in line with our digital collection practice: type (resource type, i.e., sheet music) repository (entity responsible for publishing the digital object, i.e., university of north carolina at chapel hill. music library) digital collection (name of the digital collection of which the digital object is a part, i.e., 19th-century american sheet music) host (university of north carolina at chapel hill) more challenging were fields devoted to titles, illustrations, instrumentation, and topic. our original five title fields: full title short title caption title inside title other titles were compressed into three, hopefully less confusing, fields: title (title proper) caption title (if different from title) other titles (all other expressions of title) the original fields cover design and illustration were reduced to a single field, illustration, incorporating selected content from cover design into illustration (more on this in the following discussion of metadata clean-up). we also eliminated the earlier practice of describing such illustrations as calligraphic swashes and decorative designs, limiting the use of this field to descriptions of scenes, portraits, and photographs. originally there was an instrumentation field and a subject field, with a majority of subject entries expressing in fact instrumentation or genre. we kept the instrumentation field and changed the name of the subject field to topic, using it only for topics of vocal works. no vocabulary control had been applied to either of these fields, resulting in many conflicting headings for the same concept, e.g., choir music and choral music, or solo song and solo voice. our primary purposes in all our decision-making were to simplify both the user’s understanding of the metadata and project staff’s creation of metadata. as we made decisions about field construction, we took efforts to create self-explanatory field labels. concomitant with the construction of our fields was their mapping to dublin core fields (see the “dc map” column of figure 1). our contentdm collections are harvested by the open archives initiative protocol for metadata harvesting (oai-pmh), which employs dublin core as its common metadata format, allowing the sharing of digital collections created in varying metadata formats. once mapped to dublin core, data can be harvested by anyone who has adopted oai-pmh. controlled vocabularies and data dictionary contentdm contains several thesauri, e.g., the getty research institute’s art & architecture thesaurus, the library of congress’s (lc) thesaurus for graphic materials i, and mesh (the u.s. national library of medicine’s medical subject headings). it does not contain nor can it be connected to the lc name authority file (lcnaf) or lc subject headings (lcsh). therefore, we enabled controlled vocabularies for the fields for which authority control and consistent metadata quality were most important. such control is important not only for consistent search results, but also for public browse capability. a crucial question we addressed was: what fields do we want our users to be able to browse? figure 2. sheet music collection opening screen the fields for which we offer browsing capability are shown in figure 2. effective browsability requires that metadata be consistently applied, therefore browsable fields tend to have controlled vocabularies. for example, a sheet music researcher could likely be interested in studying the history of music publishing in a particular location and would find the ability to browse this collection by publisher location highly useful. figure 3. browse by location when a field is made browsable, a hyperlinked, alphabetical listing of the contents of that field throughout the entire digital collection is made available to the user. if this particular vocabulary were not controlled, state abbreviations would appear in various forms, as would names of non-u.s. cities, e.g., mainz and mayence, germany, and brussels and bruxelles, belgium. the same logic extends to providing reliable browsing of, for instance, composer and lyricist names. as we made controlled vocabulary decisions, we developed guidelines for creating entries for all fields, controlled or not, in the form of a project-specific data dictionary. [1] (see appendix a for data dictionary.) as with our decisions regarding field construction, our primary purpose in creating the data dictionary was to simplify the creation of metadata for project staff. our fundamental principles in the construction of the data dictionary were simplicity and consistency. these principles allowed us to make steady progress dealing with a mass of inherited metadata of uneven quality with no control mechanisms, and to successfully handle our situation of uncertain staffing, which required fairly frequent training of new, and always temporary, project staff. clearly, names need to be controlled, so we enabled controlled vocabularies for our four name fields: composer lyricist arranger lithographer, engraver, or artist after completing metadata clean-up (to be discussed later) of the name fields, we combined the originally four separate name controlled vocabularies into one large shared controlled vocabulary that contains all the names used in these four fields. due to the sheer number of names we inherited, we couldn’t consider bringing them all into conformity with lcnaf, aacr2, and local digital collection practice. but we could ensure consistency within the database, making sure that one, and only one, form of each name exists in the controlled vocabulary, and adding new metadata according to the guidelines laid out in our data dictionary. as much as possible, we drew on previous project practice as our guide to constructing names in order to avoid recreating everything from scratch, and we formalized practice in the data dictionary. the “comments” column of our data dictionary contains instructions about how to construct names, with examples given in the “content examples” column and guidance about where in the musical score to locate name fields metadata in the “vocabulary source” column. other fields for which we enabled controlled vocabularies are: publisher; publisher location: left uncontrolled, these fields would give users unpredictable search results, and their browse views would be a mess, with geographical terms such as state abbreviations and names of foreign cities appearing in various forms. as with personal and corporate names, we did not follow lcnaf and aacr2 practice, but ensured consistency within the database. topic: we fairly quickly decided against using lcsh for our topic field due to training challenges with our frequently changing project staff. lcsh is simply too complex to use with student staff that experiences regular turnover. after considering several thesauri, we decided to rely on wikipedia, with webster’s third new international dictionary, unabridged (available via our library’s website) as our backup. wikipedia has served our needs beautifully. wikipedia concisely describes the usually simple topics addressed by vocal works in our collection, e.g., christmas, death, loneliness, and patriotism. more complex topics are expressed much more simply by wikipedia than lcsh, e.g., american civil war vs. united states—history—civil war, 1861-1865; battle of prague (1757) vs. prague, battle of, prague, czech republic, 1757; and silver standard vs. bimetallism, or silver question, or currency question, or money, or coinage. as in other areas of decision-making, we chose wikipedia as our primary topical thesaurus in order to simplify the creation of metadata for project staff, as well as to easily ensure consistency in the topic field. volume: consistency in this field is most important to staff working behind the scenes. for instance, when we uploaded rescanned images with their metadata, we had to make sure that the volume part of the image file names exactly matched the volume numbering in the metadata, which in turn had to be consistent throughout the individual records for the musical works from each volume. additionally, a public use is served when someone wants to look at the contents of a full volume, as at least one local musicology professor assigns his students to do. type; repository; digital collection; host: these fields contain constant data, so enabling controlled vocabularies for them ensures consistency, as well as enables extremely easy metadata input for these fields. the cataloger simply double clicks on the entry in the controlled vocabulary, and the field is accurately populated. metadata clean-up when metadata clean-up could be accomplished using contentdm’s “find & replace” function, we were able to tackle large amounts of data. the function “find and replace a single field” served our needs best, since we attacked clean-up on a field-by-field basis. figure 4. find & replace function additionally, making changes across all fields is a high risk activity for the reason that words or phrases needing correction in some fields occasionally occur in contexts not requiring correction. when clean-up could be achieved only one record at a time, we cleaned up fields involving manageable amounts of data, but sometimes had to opt for the “acceptable level of embarrassment.” examples of large clean-up efforts include the following fields: composer; lyricist; arranger; lithographer, engraver, or artist topic illustration once guidelines for creating names were established and controlled vocabularies for each name field were enabled, clean-up began with the controlled vocabularies. controlled vocabularies were originally populated with the existing contents of their corresponding fields. many names appeared in multiple forms, so duplicate forms were deleted after the authoritative form was selected. when the controlled vocabulary was clean, we proceeded to cleaning up that field’s metadata, accomplished by using the “find & replace” function mentioned above. the incorrect form of name was entered in the “find” box, and the correct form in the “replace with” box. once this process was completed, we ran the verify command for that field, comparing a field’s metadata across all records with the field’s associated controlled vocabulary. figure 5. contentdm verify command ideally the verify command delivers zero results: figure 6. verify results results indicate a mismatch, telling us one of two things: 1) an error exists in the metadata, or 2) metadata exists that has not been added to the controlled vocabulary. when we achieved zero results from the verify command, that field’s clean-up was completed. to this day we run the verify command on fields with controlled vocabularies on a monthly basis as a method of quality control. once all name fields were cleaned up, we combined the four separate controlled vocabularies into a single shared controlled vocabulary containing all the names used in the four fields. using a shared controlled vocabulary has proven helpful in streamlining data entry for these fields, because some names appear in multiple name fields. for example, the composer of a work might also have written the lyrics to that work, and might have arranged other works. as mentioned earlier, the topic field was originally called subject. we renamed the subject field topic, hoping to make it clearer that this field is used only for songs about a specific topic. we removed all non-topical terms from the controlled vocabulary, corrected terms in the controlled vocabulary as appropriate, then cleaned up metadata using the “find & replace” function. it is possible that some records that would benefit from the topic field are lacking it. in addition to making sure that correct topical terms were used, this metadata clean-up required removing numerous instrumentation and genre terms from the topic field of records. because of the magnitude of this problem, we were unable to move those terms into the instrumentation field of their records, a task that would have required record-by-record editing. the information was simply deleted so that the topic field would be clean across the database. this is an excellent example of a less than ideal decision made when the amount of inconsistent inherited metadata was simply too large to fix. what we did fix was the general integrity of the database. in contrast to the problem described above with the instrumentation field, we were able to transfer valuable metadata from the original cover design field to the illustration field because it was of a manageable amount, although the transfer of metadata did have to be done one record at a time. our public services music librarian reviewed the contents of the cover design field to determine what metadata should be retained. this metadata was moved to the illustration field and the cover design field subsequently deleted. once metadata clean-up was completed, we knew that every detail wasn’t as perfect as it might have been if we had designed things from scratch, but we also knew that the database was in stunningly better condition than when we had begun. the future as pleased as we are with the current state of the 19th-century american sheet music collection, we have plans and hopes for future improvements. most immediate is making the collection accessible via the sheet music consortium, “a group of libraries working toward the goal of building an open collection of digitized sheet music using the open archives initiative:protocol for metadata harvesting (oai:pmh).” [2] more labor intensive, but nevertheless attainable, goals are to bring names into compliance with lcnaf and topics with lcsh, which would bring the collection in line with our local digital collection practice. the importance of these goals lies in the fact that the unc-chapel hill library plans to make contentdm collections available via endeca, which offers relevance-ranked keyword search results as well as faceted navigation built on existing library metadata. it is highly undesirable to include collections lacking the authority control applied to names and subjects in our millennium-based library catalog. although our contentdm collections will not actually reside in millennium, the distinction between millennium and endeca is not apparent to our users, nor should it be. an endeca search retrieves integrated results, providing access to a variety of the library’s collections, not just those cataloged in the marc format. a current example of such resources available via endeca is our ead finding aids for special collections. a final hope for improving the sheet music collection is to populate the instrumentation field in all records. when the scanning and cataloging of all 131 volumes of sheet music is completed, we will turn our attention to improving an imperfect but highly usable and consistent collection. inherited projects like the 19th-century american sheet music collection present many challenges, some that cannot be resolved in an ideal manner, but the collections are nonetheless valuable. as heir to such a project, we strove to improve the weakest parts of the project and to ensure quality from the present day forward. end notes [1] lois schultz and sarah shaw’s cataloging sheet music (scarecrow press and music library association, 2003) was helpful in forming some of our data dictionary’s field definitions. [2] sheet music consortium: about the project. [cited 2010 march 10]. available from: http://digital.library.ucla.edu/sheetmusic/oaiproject.html acknowledgments i would like to acknowledge several unc-chapel hill colleagues for their help and advice with various aspects of this article: lynn holdzkom (head of special collections technical services) ericka patillo (ph.d. student, school of information and library science) jill sexton (information infrastructure architect) tim shearer (head, applications development team) diane steinhaus (public services music librarian) about the author renée mcbride is head of the special formats & metadata section of the resource description & management dept. at the university of north carolina at chapel hill university library. her previous positions include technical services & arts liaison librarian at hollins university, and humanities & music cataloger at ucla. appendix a general rule: if data for a particular field does not exist, do not enter any information in the field. do not enter information such as unknown, n/a, etc. the single exception is the title field, which is mandatory. see title field instructions below for guidance. controlled vocabulary: when this is listed as the vocabulary source, it refers to contentdm’s controlled vocabulary for the field in which you are working. transcribe: when this is listed as the vocabulary source, it means transcribe exactly what you see in the work. multiple entries in a field: when you have more than one entry in a field, connect the entries with ;space, e.g. piano; voice; violin (possible entry in instrumentation field), english; french (language field), standard pieces for piano; twelve standard pieces for piano (other titles field). diacritics display: if your diacritics are not displaying correctly, do the following in firefox: view/character encoding/unicode (utf-8) in internet explorer: view/encoding/unicode (utf-8) field obligation comments content examples vocabulary source identifier mandatory     system supplied composer required if available personal & corporate names only; no functions, e.g. arranger, translator, etc. with personal name, or locations with personal or corporate. function is sometimes part of a corporate name, in which case it may be included. do not use anon. or anonymous. personal names surname, forename, title (as applicable) prepositions and articles follow forename. titles/abbreviations to use col., dr., hon., maj., miss, mme., mrs., rev., sir as others arise, ask before entering. do not use esq./esquire. use mr. only when no forename or initial given. spell out abbreviated forenames unless spelling is unknown. do not use, e.g. chas, robt, wm, geo. if name consists of only initials, a surname, or a phrase, transcribe exactly, beginning phrase with articles if they exist. no spaces between adjacent initials. if more than one name in a field, connect with ;space do not use any punctuation to indicate questionable information, i.e., do not use [brackets], (parentheses) or "quotation marks." include diacritics if you always find the name with diacritics. if the name appears both with and without diacritics, do not include diacritics. information about the entities in these name fields goes in the additional information field. bellini, vincenzo edw. l. balch, music printer endicott & co. browne, miss neukomm, sigismund, the chevalier pinna, joseph de suppe, franz von hache, theodore von la becket, gilbert a, mrs. barton, j., maj. blackwood, price, mrs. drake, mr. kontski, ant. de (we aren’t sure how to spell out ant.) a lady of new york lefebure-wely m.m.o. abel, j.l. koch, h.a.r., dr. caudill, william m.; walker, william controlled vocabulary title page or caption lyricist required if available controlled vocabulary title page or caption arranger required if available controlled vocabulary title page or caption lithographer, engraver or artist required if available controlled vocabulary usually title page or caption; engraver information may also be found at bottom of title page or on the last page of music. consult the appropriate controlled vocabulary list for your person or corporate body. a correct entry from the list may not be an exact match to what is on your piece of music, e.g. lithographers’ business names appear slightly differently in different publications, or your composer’s name may have the forename initialized or abbreviated on your piece of music. if you are uncertain whether your person or corporate body is a match to a name in the controlled vocabulary, or as to how to enter the name, please consult with supervisor. manually enter a person or corporate body only if your entry is not in the list. for these four names fields, there may be one and only one correct entry. this form of name will be in the controlled vocabulary. if you encounter a new name, you must add it to the controlled vocabulary. performer required if available phrase about a performer of the work sung by miss taylor as sung by miss matthews as performed by the germania musical society transcribe from title page or caption (information located above first page of music) title mandatory title proper : subtitle capitalize only first word of title, unless: 1) title contains proper names or words commonly capitalized, e.g. i, christmas 2) title is in german, in which case nouns are capitalized include definite and indefinite articles at beginning of title. do not include non-title information, e.g. edition, composer, arranger, lyricist, publisher, dedication, etc. if a work does not have a title, supply one based on the information you have. the title field is mandatory and must contain metadata. do not enclose title in brackets. when you have to supply your own title, add in the additional information field the note: title supplied by cataloger. if a file contains more than one work, enter multiple titles connected with ;space between titles. see next column for example. 12 standard pieces for piano : floating on the wind alice : romance the devoted land meiner seeligsten gefiehlen sechs sonatinen in fortlaufender schwierigkeit come where the fountains play : la favorita; march masquerade transcribe from title page, or caption if no title page caption title required if available title at top of first page of music enter only if different from title. if caption contains lengthy other title information, enter in this field only the first part of the title; enter any further title information in the other titles field. floating on the wind alice, where art thou? transcribe from caption other titles required if available title not covered by previous title fields, or portion of title that might reasonably be searched standard pieces for piano (a portion of the title one might search) twelve standard pieces for piano (offers searching of numeral spelled out) alice, ou donc estu? (title given in french after the primary english title on caption) anywhere in score edition required if available edition statement 10th ed. fourth edition new editions carefully revised transcribe from title page, or first page of music if not given on title page publisher required if available primary publisher only first publisher listed. enter publisher name exactly as it appears in publication. be careful not to confuse publisher information with that about engravers/lithographers/printers. no spaces between adjacent initials. publishers’ names may appear in different forms and languages on different pieces of music. use the form that appears on the work you are cataloging. a.e. blackmar & bro. d.p. faulds lee & walker b. schott b. schott & sons b. schott’s sons le fils de b. schott transcribe from title page, or first page of music if not given on title page. consult the controlled vocabulary for your publisher. manually enter your publisher only if it is not in the controlled vocabulary. if you encounter a new name, you must add it to the controlled vocabulary. publisher location required if available primary publication location only first location listed. for u.s. locations, give city and state, separated by ,space, using two-letter us postal service abbreviations, available at: http://www.usps.com/ncsc/lookups/ usps_abbreviations.html for non-u.s. locations, use city and country in its english form, separated by ,space. for the name of the city, use the form found in the controlled vocabulary list, or if not in the list, the form in english, or as it appears on your piece of music if the english form cannot be determined. if you don’t know the state or country, enter only the city name. if only the state is given, enter the state name spelled out. if no place of publication at all is given, do not enter any information in this field. baltimore, md philadelphia, pa berlin, germany brussels, belgium london, england mainz, germany kentucky brunswick (state unknown) use title page, or first page of music if not given on title page, for source of information. consult the controlled vocabulary for your location. manually enter your location only if it is not in the controlled vocabulary. if you encounter a new location, you must add it to the controlled vocabulary. publication date required if available year of publication give in arabic numerals. give year only; do not include text about the year of publication or copyright. 1870 transcribe from title page, or first page of music if not given on title page sometimes given as part of copyright statement, e.g. entered according to act of congress in the year 1870. publisher’s number required if available a numbering designation assigned to an item by a music publisher, appearing on the title page, first page of music, or at the bottom of various pages of music. it may include initials, abbreviations or words in addition to numerals. do not include pub. pl. in your statement. this often exists and is not a meaningful part of the publisher’s number. g.a. 153 953.9 4924 no. 10014 transcribe from anywhere in the score pages mandatory number of pages in the score give in arabic numerals. enter only a numeral. if the pages are numbered, give the printed number of pages. this number often includes the title page and title page verso as pages. if the pages are unnumbered, count the pages, not including the cover, title page, and title page verso in your numbering. 4 67 score instrumentation mandatory instruments for which the work is composed this field will usually contain multiple entries, which are to be connected by ;space you may list instruments in any order you like. list every instrument represented in the score, including optional instruments, whether named in the score or not. sometimes an instrument is not named in the score, e.g. in the case of solo piano. use chorus for all choral works; do not specify voices. use piano, not pianoforte or piano forte. if for a single instrument, use only the name of the instrument, not "solo [instrument]." e.g. use piano, flute, voice, violin, etc. use arabic numberals when numbers are needed, e.g. 4 voices. do not use hyphens, e.g. 4 hand piano. if there is a phrase about instrumentation that you think is useful, transcribe it in the additional information field. piano; violin; harp flute; piano guitar piano; organ; voice piano; chorus piano; chorus; high voice soprano; tenor; piano (voice parts are specified) 2 voices; piano (voice parts are not specified) 4 voices; piano 2 pianos 4 hand piano 8 hands, 2 pianos chorus; piano; middle voice 2 tenors; bass; guitar chorus; piano; soprano; alto high voice; low voice; piano 2 high voices; piano score, title you are free to use terms in any order, in accordance with the instructions given in the comments column. topic required if available the topic that a vocal work is about use this field only when you are dealing with a song that is clearly about a topic. do not enter instrumentation or musical genre in this field. if there are multiple terms, connect with ;space and begin each new entry with a capital letter. american civil war battle of prague (1757) berry picking children frederick ii of prussia love minstrel shows old age patriotism sailing sea coast; seashells sunrise unrequited love title and/or text of song determine term(s) to use in this order: 1) controlled vocabulary: if a term exists that covers your topic, use it. 2) wikipedia: a) if term exists in singular form and plural is clearer, use plural form (e.g. children, minstrel shows). b) if you are redirected from a term that more accurately describes your topic than the term to which you are redirected, use the more accurate term (e.g. berry picking redirects you to berry; use berry picking). 3) webster’s third new international dictionary, unabridged if unsure what term to use, please ask before making decision. first line of text required if available first textual phrase of vocal works limit to the first grammatical phrase of the text, often made clear through punctuation such as a comma at the end of the phrase. enter in the primary language of the song if there is more than one language. oh! dearly do i love to rove among the fields of barley (first line of song entitled bonnie charlie) transcribe first line of chorus required if available first textual phrase of the chorus, or refrain, of vocal works often indicated by the term "chorus" at the beginning of this section of the music. yes, my love, we’re growing old (first line of chorus of work entitled you are always young to me) transcribe language required if available languages of text in vocal works that contain languages other than english do not use this field for works with only an english text. you may list languages in any order you like. multiple entries separated by ;space if english is one of the languages, do include it in the list of languages. if there is information about languages that you want to include that does not fit the limitations of this field, enter it in the additional information field. english; french english; french; italian score illustration required if available describe illustrations such as scenes, portraits, photographs. can be as succinct or lengthy as necessary to provide accurate description. do not describe designs such as calligraphic swashes, geometric or abstract designs, decorative designs. venice at night. several domed buildings in the background, across a broad canal with two gondolas floating. several men and women stand on a landing near the water’s edge. steps lead up to a building, with another lady standing on a balcony. color lithograph of 3 women by shore, rainbow on horizon. trees and rocks surround them. portrait of jenny lind. score inscription required if available handwritten information on or inside cover, on title page, or on caption page if illegible, do not transcribe. if partially legible, transcribe and explain illegible part in [brackets], e.g. [illegible], [cut off] this is usually information about who is being given the music, often as a gift. may be easily confused with previous owner, so be careful. if an inscription applies to the volume as a whole, enter that inscription only once, in the record for the first piece of music from that volume. presented to miss ada j. coit by reinhold f. hunt dear little one, from [cut off] a monsieur sigismond benedict transcribe from cover or inside of cover, title page, or caption page previous owner required if available name of person who owned the print music sometimes comes from print volume as a whole, or on or inside cover, on title page, or on caption page of individual pieces of music. if illegible, do not transcribe. if partially legible, transcribe and explain illegible part in [brackets], e.g. [illegible], [cut off] may be easily confused with inscription, so be careful. if it is clear that the same previous owner information applies to multiple songs in a volume, include it in every applicable record even if the name is not written on each individual piece. e.g. the volume has pieces published together as a single collection, with the owner’s name appearing only on the title page of the collection. he/she clearly owned each of the pieces within that collection. eliza c. anderson belle hannah mcgehee, burleigh, near milton n.c. transcribe, usually from cover or inside of cover, title page, or caption page; check entire score marginalia required if available handwritten information within score be careful not to confuse with inscription and previous owner. if illegible, do not enter information. if multiple entries, separate by ;space performer’s annotations; on the recto to p. 1 in pencil: gallopade quadrille ossia bars added to final measures x’s or hash marks next to pieces in the advertisement score; free text description of marginalia; transcribe if appropriate dedication required if available printed dedication statement most cordially dedicated to mrs. archibald robertson (of philadelphia) to mrs. harriet e. griffin (albany n.y.) music composed & dedicated to mrs. gardiner of rochester n.y. transcribe from title page or caption dealer’s stamp required if available dealer or distributor information hand-stamped or on an affixed label. usually on title page. geo. b. mitchell successor to zogbaum & co music dealer savannah ga gaines, riches & co., petersburg transcribe price required if available printed price usually on title page. omit words like price. spell out the word cents for the cents symbol. 50 cents $2.00 nett 50 cts. net 2 fr. one dollar transcribe, omitting words like price. defects required if available describe any substantial defects in the printed music second page torn and repaired with tape pages missing score volume mandatory numbering of print volume in which work is found the numbering in the volume field must match exactly the numbering in the volume field’s controlled vocabulary. for volumes that have yet to be cataloged, only the base volume number is in the controlled vocabulary, e.g. old series 102. when you catalog a new volume, add the appropriate song number to the base volume number, e.g. old series 102,4. enter this information in both the volume field and the controlled vocabulary. iii,31 new series xiv,9 old series 10,1 white i, 22 xxxv,8 print volume and controlled vocabulary additional information optional important information that does not fit into preceding fields any information you think will be helpful in describing and accessing the work may go into this field. english lyrics printed beneath french and italian. the popularity of this song has induced persons in philadelphia, baltimore, and new york, to publish music with the title of the ‘carrier dove.’ the publisher of this song would respectfully remind purchasers: that the genuine copy has the imprint … inside cover page provides an extract from the france musical testifying to the power of gottschalk’s music. flute part is in different key than piano and vocal music, unless intended for a transposing instrument. title page caption: this lighthouse was entirely swept away on the night of april 16th 1851. the assistant keepers joseph wilson & joseph antone were drowned, being the only persons in it at the time of the accident. score; transcribe or free text as appropriate unknown do not use because this field contains a lot of useful information in previously input records, we are retaining the field and making it searchable, but hidden from public view. we are not using this field for currently input records. do not use type mandatory constant data is: sheet music use for every digital object whether or not this information appears elsewhere in the metadata. sheet music this will always be the content of this field. use controlled vocabulary for this field. repository mandatory entity responsible for publishing the digital object on the web use for every digital object whether or not this information appears elsewhere in the metadata. university of north carolina at chapel hill. music library this will always be the content of this field. use controlled vocabulary for this field. digital collection mandatory name of the digital collection of which the digital object is a part use for every digital object whether or not this information appears elsewhere in the metadata. 19th-century american sheet music this will always be the content of this field. use controlled vocabulary for this field. host mandatory constant data is: university of north carolina at chapel hill use for every digital object whether or not this information appears elsewhere in the metadata. university of north carolina at chapel hill this will always be the content of this field. use controlled vocabulary for this field. subscribe to comments: for this article | for all articles 2 responses to "look what we got! how inherited data drives decision-making: unc-chapel hill’s 19th-century american sheet music collection" please leave a response below: cbs bibliotek blog – innovation & ny viden » blog archive » nyt nummer af the code4lib journal, 2011-04-14 […] look what we got! how inherited data drives decision-making: unc-chapel hill’s 19th-century americ… […] gabriel farrell, 2011-04-20 the use of wikipedia for the topics was a great idea. i imagine they were easier to check and input for those doing the cataloging, and were also more useful for information seekers. leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – ead mctaggart: using vba to automate ead container list tagging mission editorial committee process and structure code4lib issue 8, 2009-11-23 ead mctaggart: using vba to automate ead container list tagging faced with the prospect of converting 200-page container lists to encoded archival description (ead), the author programmed a microsoft access® database using visual basic for applications (vba) to automatically insert the necessary ead tags and their attributes. some work is still required to ensure that the container list is properly formatted before importing into the database. once formatted, the database, named ead mctaggart, will convert a 7,000 line microsoft excel® container list, where each line represents a series, sub-series, or folder title, into a properly tagged ead container list in about five minutes. as written, ead mctaggart will handle up to six component levels, but can be modified to handle more. although many institutions use archivists’ toolkit or archon for this functionality, many libraries and archives who have not implemented those tools will find that ead mctaggert minimizes the work of converting existing container lists to ead finding aids with a low time investment for implementation. by randall miles introduction when i began in my current position in november 2007, only about one-quarter of the collections had ead finding aids and not all of those finding aids were complete. many lacked container lists. generating ead for the information other than the container list is not an issue as we create marc records for any collection which would require an ead guide. i simply use terry reese’s marcedit, with a style sheet modified to conform to local standards, to convert the marc record to ead. marcedit does not help with the container list because those lists are not included in the marc record. container lists for the collections existed in the kheel labor-management documentation center and archives at cornell university in various forms: microsoft excel®, microsoft word®, and paper only. the data from these finding aids can be converted to comma separated value (.csv) files and imported into either archivists toolbox or archon, then exported as fully tagged ead. however the kheel center was not using either of these tools and any implementation of either was several years in the future. something needed to be done in the interim. the prospect of hand tagging container lists, some over one hundred pages long, and at least one over one thousand pages long, was not pleasant. importing the data into an excel spreadsheet and using the fill-down function to fill in repeated tags is a common solution, but keeping track of the closing tags, normalizing dates, and pulling out series headings which link back to the list would still be daunting for any list more than a few pages long. this chore seemed a good candidate for automation. my preferred application for this sort of project is microsoft access®, which led to the creation of an ead tagging database i call ead mctaggart. the first attempt at automating tagging was with an sql select query involving numerous immediate if (iif, an sql function that processes the if-then-else logic in the select clause of an sql query) statements in the select clause. the query worked well for most of the tagging, but could not place the component level closing tags correctly since they depend on the next line, or record, in the table. at the end of a <c01> line, if the next line was another <c01> a closing tag, </c01> was needed, but it was not needed if the next line was a <c02> line. i needed something that could look into the future to see what was coming up next. using visual basic for applications (vba) in an access database i was able to have the program evaluate the next line before finishing the current line. this was done by moving the cursor forward in a record set containing all the lines for the container list. with careful use of if-then-else statements, unneeded tags can be automatically left out, or the tag’s attributes can be automatically adjusted to match the tag’s value; i.e., the “normal” attribute for a single date versus a date range in the <unitdate> tag. by using the unicode designations for line breaks and tabs in the vba code, the output can be formatted with the proper indentation for ease of use in an ead editor. this article will not provide step-by-step instructions for writing the vba programming needed to convert a container list to ead. in addition to knowledge of ead, i will assume a certain level of vba programming ability and simply cover the more important points in four basic areas: data preparation, special characters, an overview of the database design, and code examples. data preparation one key to getting good results with this database system for generating ead tagged container lists is formatting the data before importing it into the database. as stated above, the kheel center’s container lists are mainly in one of three formats: word, excel, or print. i prepare the data in excel, so that the formatting matches the format of the table in ead mctaggart. the basic spreadsheet has columns for box number, folder number, title, date, and scope content note. i also include columns for box text and folder text, which allow such notations as “box 1a: folder 1-12,” while still maintaining proper sort order, without having to left-pad with zeros. the data for these columns comes from the existing folder lists, or, if there is no folder list, the list is created in the spreadsheet. two other important columns are the “c0” column where the ead component level is recorded as an integer and the “index” column which serves to keep the entries in the proper order. formatting the data in the excel spreadsheet is quite simple. unless the data needs to be cleaned up it also goes quite quickly. folder lists in excel usually require little reformatting. print folder lists require the most reformatting. scanning, running through ocr, and proof-reading the results before importing into the spreadsheet can take quite a bit of time. given the error rate for ocr i often just have a student re-key the data into the spreadsheet. folder lists in word take a little more reformatting than those in excel, but not as much as a print list. as an example, if i have a word document with box numbers, folder numbers, folder titles, and dates, i copy and paste the data into a spreadsheet. usually, tabs separate most of the elements in the word document and i only have to separate out the dates from the titles. at this point if the collection will have series and sub-series, i replace all null values in the “c0” column with 3 so that the folder titles will end up at the <c03> component level. if the folder list already has series and sub-series titles, the value in the “c0” column is then changed to 1 for rows with series titles and 2 for rows sub-series titles. if there are not already series and sub-series titles, insert new rows where series titles are needed. series titles can then be entered and the null values in “c0” replaced with 1. then insert new rows where sub-series titles are needed, enter the titles and replace the null values in “c0” with 2. at this point the index field is numbered to maintain proper order. note that the series and sub-series rows will have null values for “box” and “folder,” but are included in the index numbering. the data are now ready for ead mctaggart. see table 1. table 1: sample of container list data formatted in excel for ead mctaggart index c0 box box text folder folder text title date scopecontent 1 1         i: general board 1970-1979   2 2         a: correspondence 1970-1975   3 3 1   1   general 1970   4 3 1   2   general 1971   5 3 1   3   general 1972   6 3 1   4   general 1973   7 3 1   5   general 1974   8 3 1   6   general 1975 january – june 9 3 1   6 a general 1975 july – december 10 3 1   7   personal 1970-1971   11 3 1   8   personal 1972-1973   12 3 1   9   personal 1974-1975   13 2         b: meeting minutes 1970-1979   14 3 2   1   official minutes 1970-1971 bound 15 3 2   2   official minutes 1972-1974 bound 16 3 2   3   official minutes 1975-1978 bound 17 3 2 a 1   personal copies 1970-1971 annotated, loose 18 3 2 a 2   personal copies 1972-1973 annotated, loose 19 3 2 a 3   personal copies 1974-1975 annotated, loose 20 3 2   4   personal copies 1976-1977 annotated, loose 21 3 2 a 5   personal copies 1978-1979 annotated, loose special characters special characters can be either problems or solutions. problems include characters, or words, which prevent sql statements or vba code from running. solutions include characters which make the output more human readable. two of the problem special characters i had to cope with were the single quote and the ampersand. the single quote, or apostrophe, will sometimes cause the sql or vba to stop running, while the ampersand will cause an error in the style sheet when the ead is converted to html for viewing in a browser. apostrophes had to be escaped in order for the program to ignore them, while ampersands had to be replaced with something more html-friendly. to escape an apostrophe, put a single quote in front of it. essentially, it is necessary to replace single quotes with two single quotes, but not a double quote. unfortunately, you cannot simply run an sql statement to change the single quote to two single quotes: update tablename set title = replace([title], " ' " ," ' ' "); when the vba code is compiled the first single quote is read as a comment indicator and everything after it is treated as a comment, leaving you with an incomplete sql statement. the solution is to declare two variables, one for the single quote and one for the two single quotes and then assign the problem characters as the values: dim strchr1 as string dim strchr2 as string strchr1 = " " " ' " " " strchr2 = " " " ' ' " " " all those double quotes identify the single quotes as strings. these variables are then used in the update sql: update collection table set title = replace([title], " & strchr1 & ", " & strchr2 & "); which runs as: update collection table set title = replace([title], " ' " ," ' ' "); without a compiling error. replacing the single quotes with two single quotes allows the program to run and drop the extra single quote, so that the output will have only the one single quote you started with. ampersands in the data strings do not cause sql or vba problems, but do prevent the guide from being displayed when the xml file is converted to html for delivery to the internet. replacing the ampersand is similar to replacing the single quote. when used outside of a variable assignment statement the compiler sees the ampersand as a reserved character, resulting in a compiling error. to get around this problem, declare two variables and give them values: dim strchr3 as string dim strchr4 as string strchr3 = " " " & " " " strchr4 = " " " &amp; " " " then use the variables in the update statement: update collection table set title = replace([title], " & strchr3 & ", " & strchr4 & "); both of these sql updates also need to be run on the scopecontent field. there is also a word commonly used in ead container lists that is reserved in ms access, “level,” which needs to be dealt with, or the code will not run. this reserved word occurs as an attribute in the component level opening tags, e.g. <c03 level=”file”>. to keep from having the code choke on the reserved word enclose it in square brackets in the vba code: <c03 [level]=”file”>. you can later remove the brackets in a text editor. i included an sql update line to remove them before exporting the data: update tblead set tags = replace([tags], '[level]', 'level'); there are a few other useful characters to be mentioned at this point. vba uses double quotes to tell the processor what in an output string is text and what is a variable. anything enclosed in double quotes is text and is be printed as is, while anything outside the double quotes is a variable and is to be replaced with the variable’s value. when you need double quotes in the text portion of your vba code, you need to use the ascii value in the chr function, chr(34). i also use the chr function and ascii values to format the output with the tags on different lines and indented, so that it is easily read in a text editor. this is done by putting chr(13) in the code where i want a line break, and chr(9) where a tab is needed. design of ead mctaggart the database has very few components. there are only two permanent tables, though at times there may be several temporary tables, one form, one query, and one report. the real workhorse of ead mctaggart is the vba coding behind the form. the form, which i call frmextractead, has one text field, where the collection identifier is entered, a button to import the data, and a second button to export the ead container list. the workflow is simple: enter the collection identifier in the text box and click the import button. the first thing i have the vba code do is create a new table using the structure of one of the existing tables, tblpattern, and name it with the collection identifier. the imported data will be put into this new table and the new table will eventually be dropped when the ead has been exported. you could import the data directly into tblpattern and just empty the tblpattern when the ead has been exported, but i wanted the option of importing several container lists before exporting anything, so i generate a new table for each collection. the import function lets you navigate to the spreadsheet and drop the data into the appropriate table. to export the ead container list, make sure the collection identifier is in the text box and click the export button. this is where ead mctaggart does his vba magic. as the data are tagged, they are placed in the four fields in the tblead table. the first field, tags, holds the opening component level tag and all other data except the scope/content note and the closing tags. the second field, scope, holds the scope/content note. the third, close, holds the closing tags. the fourth, an index value. note that ead mctaggart uses two indexes. both are consecutive numbers used to keep the data in the proper order. one is the index value from the original data, the table index. this index is the one created in the excel spreadsheet and maintains order while the data is imported into ead mctaggart and when the record sets are pulled to process the list. the other index is a new one generated by the vba code, the vba index, and is used to maintain order in tblead and the exported text file. for container lists without series the table index would work for both. however, using the table index for lists with series will not work. since series and sub-series headers appear twice in the ead guide, once as a series list showing the arrangement, and again in the container list, vba index is needed to give each instance of the repeated headers a unique index value in tblead. the first thing the vba code does is run the sql updates to deal with the special characters. then the vba pulls a record set of the index, title, date, and c0 fields for all the series and sub-series data in the collection table, sorted on the original index field. these are the rows in the table where the box value is null and the c0 value is “1” or “2.” for our sample in table 1, this record set would contain the data from only rows 1, 2, and 13. it would look like table 2. table 2: first record set, used for the series list index title date c0 1 i: general board 1970-1979 1 2 a: correspondence 1970-1975 2 13 b: meeting minutes 1970-1979 2 this record set is used to create a series/sub-series list in an <arrangement> tag, using <ref> tags with the index value as the “target” attribute’s value to provide a link to the container list. at this point the vba code generates the new, sequential, vba index number for each line added to tblead. after all records in this record set have been processed the record set is closed and a second record set opened. see flowchart in figure 1. the data in tblead at this point looks like table 3. table 3: the series list in tblead tags scope close index <arrangement><head>series list</head> <list><defitem><label><ref linktype=”simple” target=”link1″ show=”new” actuate=”onrequest”>series i: general board, 1970-1979</ref></label> <item><list>     1 <defitem><label><ref linktype=”simple” target=”link2″ show=”replace” actuate=”onrequest”>sub-series a: correspondence, 1970-1975</ref></label><item>   <lb/></item></defitem> 2 <defitem><label><ref linktype=”simple” target=”link13″ show=”replace” actuate=”onrequest”>sub-series b: meeting minutes, 1970-1979</ref></label><item>   <lb/></item></defitem></list><lb/></item> </defitem></list></arrangement> <dsc> <head>container list</head> 3 depending on the style sheet used to convert the ead file to html the series list in the user’s web browser the series list could look like this: series list series i: general board, 1970-1979 sub-series a: correspondence, 1970-1975 sub-series b: meeting minutes, 1970-1979 the ead tagging for the series list looks like this: <arrangement> <head>series list</head> <list type="deflist"> <defitem> <label><ref linktype="simple" target="link1" show="new" actuate="onrequest">series i: general board, 1970-1979</ref></label> <item> <list type="deflist"> <defitem> <label><ref linktype="simple" target="link2" show="replace" actuate="onrequest">sub-series a: correspondence, 1970-1975</ref></label> <item><lb/></item> </defitem> <defitem> <label><ref linktype="simple" target="link13" show="replace" actuate="onrequest">sub-series b: meeting minutes, 1970-1979</ref></label> <item><lb/></item> </defitem> </list> <lb/></item> </defitem> </list> </arrangement> figure 1: flowchart for series list tagging the second record set contains all the data from the collection table, sorted on the table index field. the vba code then looks at the first record. if there is a scope content note, the note is wrapped in scope content tags, <scopecontent></scopecontent>. next the c0 field is evaluated. if the value is null, ead mctaggart assigns it a value of 1. any record with a c0 value of 1 and the box value null gets tagged <c01 level=”series”>; any other c0 value with a null box value gets tagged as level=”sub-series”; any record where the box value is not null gets tagged with level=”file.” see flowchart in figure 2. the vba then moves to the next record and looks at the c0 value. if the c0 value of the new record is greater than the c0 value of the old record, no closing tag is entered. if the new value is less than or equal to the old value, closing tags are added for all component levels from the level equal to the old value, up to and including the level equal to the new value. see flowchart in figure 3. the data for the first record, (strtags, strscope, strclose, and intindex), are then inserted into tblead and the second record is processed. this continues until the end of the record set is reached. when all records have been processed, the sql update to remove the square brackets from the reserved word “level” is run and the data in tblead are exported by exporting the report, rptextract, as a text document. the report, rptextract has only one field, tags, populated by using the query, qrytags, which concatenates tlbead.tags & tblead.scope & tblead.close, sorted on index. this report is automatically exported as a text file. the beginning of the container list ead tagging looks like the following. <dsc type="in-depth"> <head>container list</head> <c01 level="series"> <did> <unittitle>series i: general board, 1970-1979</unittitle> </did> <c02 level="subseries"> <did> <unittitle>sub-series a: correspondence, 1970-1975</unittitle> </did> <c03 level="file"> <did> <container>1</container> <container>1</container> <unittitle>general</unittitle> <unitdate normal="1970">1970</unitdate> </did> </c03> <c03 level="file"> <did> <container>1</container> <container>2</container> <unittitle>general</unittitle> <unitdate normal="1971">1971</unitdate> </did> </c03> after the report has been exported as a text file, the collection table is dropped from ead mctaggart and tblead is emptied. when the text file is opened in a text editor, there are a number of blank rows inserted by access when the report was formatted. these can be removed. the ead tagged container list should now be ready for the front matter of your ead finding aid. figure 2: flowchart for container tagging figure 2: flowchart for container tagging figure 3: flowchart for closing tags code examples here are a few examples of code i used in ead mctaggart, with brief explanations. for clarity the line breaks and tabs which format the output, chr(13) and chr(9), have been left out. also, some of the examples are just the beginning of longer if-then-else statements. the remainder of the statement is represented by “. . . [additional code].” the intent here is not to provide the complete code, but to illustrate how some of the issues with automating ead tagging were addressed. first the data need to be brought into ead mctaggart. clicking the “import” button on the form runs code which creates the collection table and opens the standard access dialog box for importing an excel spreadsheet. strsql = "select tblpattern.index, tblpattern.accession, tblpattern.box, " strsql = strsql & "tblpattern.boxmodifier, tblpattern.folder, " strsql = strsql & "tblpattern.foldert, tblpattern.title, " strsql = strsql & "tblpattern.date, tblpattern.scopecontent, tblpattern.c0 " strsql = strsql & "into " & strcoll & " from tblpattern;" docmd.runsql strsql docmd.runcommand accmdimportattachexcel the variable “strcoll” is the collection identifier entered into the text box on the form. if there is no collection identifier in the text box, an if-then-else test captures the error and displays a message box requesting that a collection identifier be entered. once the data are in ead mctaggart, in order to transform the data into ead the vba code needs to determine if the record being processed is a series, sub-series, or file. if the “box” field contains a box number, i.e., is not null, then it is a file and the “level” attribute in the component level tag is set to “file.” if the “box” field is null, it is a series if the value in “c0,” otherwise known as variable “intc0,” is 1 and a sub-series if it is greater than 1. vba will not handle a null value so i use the nz() function to replace any null values in the “box” field with the word “blank.” if strbox like "blank" then if intc0 = 1 then strlevel = "series" else: strlevel = "subseries" end if . . . [additional code] the component level tag in the vba code looks like this. "<c0" & intc0 & chr(34) & "[level]=" & chr(34) & strlevel & chr(34) & ">" for intc0 = 1, with no box number, this will be output as <c01 level=”series”>. if there are series in the collection, the series titles are pulled out to be displayed at the top of the container list. to do this, the code pulls a record set of only lines with no box numbers and component level c01 or c02 from the table, and processes them as follows. first the link is formed in a variable called strtags2. this defines all the link parameters in the <ref> tag, including the target, which is simply the index number from the collection table, saved in the variable intrecindex. it also concatenates the title and date, if there is one, into the label. if strdate like 0 then strtags2 = "<defitem>" strtags2 = strtags2 & "<label><ref linktype=" strtags2 = strtags2 & "simple" & chr(34) & " target=" strtags2 = strtags2 & "link: & intrecindex & chr(34) & " show=" strtags2 = strtags2 & "new" & chr(34) & " actuate=" strtags2 = strtags2 & chr(34) & "onrequest" & chr(34) & ">" strtags2 = strtags2 & strlevel2 & " " & strtitle & "</ref></label>" strtags2 = strtags2 & "<item>" strtags2 = strtags2 & "<list>" else: strtags2 = strdbltab & strtab & "<defitem>" strtags2 = strtags2 & "<label><ref linktype=" strtags2 = strtags2 & chr(34) & "simple" & chr(34) & " target=" strtags2 = strtags2 & chr(34) & "link" & intrecindex & chr(34) & " show=" strtags2 = strtags2 & chr(34) & "new" & chr(34) & " actuate=" strtags2 = strtags2 & chr(34) & "onrequest" & chr(34) & ">" strtags2 = strtags2 & strlevel2 & " " & strtitle & ", " & strdate & _ "</ref></label>" strtags2 = strtags2 & "<item>" strtags2 = strtags2 & chr"<list>" end if once the link and label are formed they are wrapped in the appropriate opening and closing tags. if the current record is the beginning of the file, bof, the arrangement and series list header tags are added. for subsequent records these tags are left off. at this point if the next record’s component level is c01, the closing tag </list> is added; otherwise this tag is left off. to do this the component level of the current record is saved as intcurrent and the cursor is moved to the next record in the record set. the component level for this record then becomes the value in the variable intc0. if intindex = 1 then if intc0 = 1 then strtags = "<arrangement>" strtags = strtags & "<head>series list</head>" strtags = strtags & "<list type=" strtags = strtags & chr(34) & "deflist" & chr(34) & ">" & strtags2 strclose = "<defitem><label></label><item></item></defitem></list>" strclose = strclose & "<lb/></item>" strclose = strclose & "</defitem>" else: strtags = "<arrangement>" strtags = strtags & "<head>series list</head>" & chr(13) strtags = strtags & ""<list type=" strtags = strtags & chr(34) & "deflist" & chr(34) & ">" & strtags2 strclose = "" end if . . . [additional code] when the last record in the record set is reached, that is the end of file, eof, everything, including the arrangement tag, is closed and the description of subordinate components tag, <dsc>, is opened and given a “container list” header. elseif rst2.eof = true then if intc0 = 1 then strtags = strtags2 strclose = "<defitem><label></label><item></item></defitem></list>" strclose = strclose & "<lb/></item>" strclose = strclose & "</defitem>" strclose = strclose & "</list>" strclose = strclose & "</arrangement>" strclose = strclose & "<dsc type =" & chr(34) & "in-depth" & chr(34) & ">" strclose = strclose & "<head>container list</head>" elseif intc0 = 2 then strtags = strtags2 strclose = "<lb/></item>" strclose = strclose & "</defitem>" strclose = strclose & "</list>" strclose = strclose & "<lb/></item>" strclose = strclose & "</defitem>" strclose = strclose & "</list>" strclose = strclose & "</arrangement>" strclose = strclose & "<dsc type =" & chr(34) & "in-depth" & ">" strclose = strclose & "<head>container list</head>" end if . . . [additional code] when all records in the record set have been dealt with, the record set is closed and a new record set, pulling all records, not just series and sub-series headings, is created. this record set will be used to generate the actual container list. as with the first record set, closing tags are determined by moving the cursor to the next record and looking at the component level. if the component level of the next record is lower, no closing tags are indicated. if the same, only the closing tag for that level is indicated. if higher, all tags up to that level are indicated. if intcurrent = 1 then 'variable intcurrent is the component level of the current record. if intc0 = 1 then 'variable intc0 is the component level of the next record. strtags = strtags2 'variable strtags2 contains the title and date for the current record. strclose = "</c01>" else: strtags = strtags2 strclose = "" end if elseif intcurrent = 2 then if intc0 = 1 then strtags = strtags2 strclose = "</c02>" strclose = "</c01>" elseif intc0 = 2 then strtags = strtags2 strclose = "</c02>" else: strtags = strtags2 strclose = "" end if . . . [additional code] another issue involves tagging the date field. not all series, sub-series, and folders have dates, in which case i do not want the date tags included. when dates are included, i want them normalized as either single or inclusive. this was done using if-then-else statements. it should be noted at this point that i allow only years in the date field, either a single four-digit year or a range of two four-digit years separated by a hyphen. any more specific dates or outlying years are noted in the scope content note. if mid(strdate, 5, 1) = "-" then strtags2 = "<unittitle>" & strtitle & "</unittitle>" strtags2 = strtags2 & "<unitdate type =" & chr(34) strtags2 = strtags2 & "inclusive" & chr(34) strtags2 = strtags2 & " normal=" & chr(34) & left(strdate, 4) strtags2 = strtags2 & "/" & right(strdate, 4) & chr(34) & ">" strtags2 = strtags2 & strdate & "</unitdate></did>" elseif strdate like 0 then strtags2 = "<unittitle>" & strtitle & "</unittitle></did>" else: strtags2 = "<unittitle>" & strtitle & "</unittitle>" strtags2 = strtags2 & "<unitdate normal=" & chr(34) strtags2 = strtags2 & strdate & chr(34) & ">" strtags2 = strtags2 & strdate & "</unitdate></did>" end if when the end of file is reached, all open component level tags are closed and the </dsc> closing tag is added. the </archdesc>, and </ead> closing tags were already supplied by the style sheet in marcedit. however, if they were not they too can be added at this point. this is how strclose would look if the last record is at component level <c06>. elseif intc0 = 6 then strtags = strtags2 strclose = "</c06>" strclose = strclose & "</c05>" strclose = strclose & "</c04>" strclose = strclose & "</c03>" strclose = strclose & "</c02>" strclose = strclose & "</c01>" strclose = strclose & "</dsc>" end if conclusion ead mctaggart quickly and accurately generates an ead tagged container list from an easily formatted excel spreadsheet. series headings, and also sub-series headings if you like, are listed first and linked to the main body of the list so that a user can click on a heading and be taken to that section of the container list. dates are normalized to facilitate searching by date, and the tags are all properly closed. the text file exported by ead mctaggart is ready to be dropped into the ead guide right before the </archdesc> tag. while this can certainly be done using archivists toolkit or archon, not everyone who wants to create ead guides is using those tools. for those people, ead mctaggart can do most, if not all, of the heavy lifting in creating ead compliant container lists. complete code the complete ead mctaggart vba code is available at http://journal.code4lib.org/media/issue8/miles/eadmctaggartvba.html. about the author randall miles is the technical services archivist at the kheel labor-management documentation center and archives, in catherwood library, at cornell university. he can be contacted at rm527 at cornell dot edu. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – html5 microdata and schema.org mission editorial committee process and structure code4lib issue 16, 2012-02-03 html5 microdata and schema.org on june 2, 2011, bing, google, and yahoo! announced the joint effort schema.org. when the big search engines talk, web site authors listen. this article is an introduction to microdata and schema.org. the first section describes what html5, microdata and schema.org are, and the problems they have been designed to solve. with this foundation in place section 2 provides a practical tutorial of how to use microdata and schema.org using a real life example from the cultural heritage sector. along the way some tools for implementers will also be introduced. issues with applying these technologies to cultural heritage materials will crop up along with opportunities to improve the situation. by jason ronallo foundation html5 the html5 standard or (depending on who you ask) the html living standard has brought a lot of changes to web authoring. amongst all the buzz about html5 is a new semantic markup syntax called microdata. html elements have semantics. for example, an ol element is an ordered list, and by default gets rendered with numbers for the list items. html5 provides new semantic elements like header, nav, article, aside, section and footer that allow more expressiveness for page authors. a bunch of div elements with various class names is no longer the only way to markup this content. these new html5 elements enable new tools and better services for the web ecosystem. browser plugins can more easily pull out the text of the article for a cleaner reading experience. search engines can give more weight to the article content rather than the advertising in the sidebar. screen reader software can use the structural elements such as nav to make textual content more accessible to people with disabilities. while these new elements provide extremely useful extra information about the sections of content, they do not really describe what the html document is about. for example html5 does not provide a book element for use in web pages being delivered by library catalog. most web applications sit on top of databases that contain lots of structured data. however, the trip this data takes from the database into html is often lossy, as the structured data is converted to html for display. maybe a human can read your field labels on the page to understand your metadata, but that meaning is lost on machines. fortunately html5 includes a syntax called microdata, that allows web publishers to layer richly structured metadata directly into their web presentations. but before we jump into how microdata works, we’ll take a brief tour through previous attempts at making structured data available on the web. to embed or not to embed one way to communicate structured metadata in html is to link your html presentation to an alternate representation of your data. here’s a simple example of making ris formatted citation data available: <link rel="alternate" type="application/x-research-info-systems" href="/search?q=cartoons&format=ris" /> this pre-html5 approach can work in many cases, such as in the area of web syndication with rss and atom…but it has some problems. techniques like this require consumers of your data to have out of band knowledge of where to look for this particular invisible content. it also adds a layer of complexity by requiring a site to have apis or metadata gateways that can be burdensome to setup and expensive for organizations to maintain. another approach which avoids some of those problems is to embed the data directly in the html. the html representation of a resource is most visible to users, so it is also the html code which gets the most attention from developers. little-used, overlooked apis or data feeds are easy to let go stale. if the website goes down, you are likely to hear about it from multiple sources immediately. if the oai-pmh gateway goes down, it would probably take longer for you to find out about it. hidden services and content are too easy to neglect. data embedded in visible html helps keep the representations in sync so that page authors only have to expose one public version of their data. this insight has lead to a number of different standards over time which take the approach of embedding structured data along with the visible html content. microdata is just one of the syntaxes in use today. a short history of structured data in html other efforts that came before microdata have addressed this same problem of marking up the meaning of content. microformats are one of the earliest efforts to provide “a general approach of using visible html markup to publish structured data to the web.” some microformat specifications like hcard, hcalendar, and rel-license are in common use across the web. development of the various small microformat standards takes place on a community wiki. simply put, microformats usually use the convention of standard class names to provide meaning on a web page. the latest version microformats-2 simplifies and harmonizes these conventions across specifications significantly. rdfa, a standard of the w3c, has the vision of “bridging the human and data webs.” the idea is to provide attributes and processing rules for embedding rdf (and all of its graph-based, linked data goodness) in html. with all that expressive power comes some difficulty, and implementing rdfa has proven to be overly complex for most web developers. google has supported rdfa in some fashion since 2009, and over that time has discovered a large error rate in the application of rdfa by webmasters. simplicity is a central reason for the development of microdata and the search engines preferring it over rdfa. in part a reaction to greater adoption of microdata, a simplified profile of rdfa has been created. rdfa lite 1.1 provides simpler authoring guidelines that mirror more closely the syntax of microdata. it also bears mentioning that in the library space, we have developed specifications which tried to solve similar problems of making structured data available to machines through html. the unapi specification uses a microformat for exposing (using the deprecated abbr method) the presence of identifiers which may resolve to alternative formats through a web service. coins uses empty spans to make openurl context objects available for autodiscovery by machines. while there has been some deployment of unapi and coins in the library community, it in no way approaches the use that microformats and rdfa have seen in the larger web ecosystem. so what is microdata? microdata came out of a long thread about incorporating rdfa into html5. because rdfa simply was not going to be incorporated into html5, something else was needed to fill the gap. out of that thread and collected use cases ian “hixie” hickson, the editor of the html5 specification, showed the first work on microdata on may 10, 2009 (the syntax has changed some since then). the syntax is designed to be simple for page authors to implement. according to microdata terminology the things being described in an html page are items. each item is made up of one or more key-value pairs: a property and a value. the microdata syntax is completely made up of html attributes. these attributes can be used on any valid html element. the core of the data model is made up of three new html attributes: itemscope which says that there is a new item within itemtype which specifies the type of item itemprop which gives the item properties and values a first example here is a simple example of what microdata looks like: <div itemscope itemtype="unorganization"> <span itemprop="eponym">code4lib</span> </div> the user of a browser would only see the text “code4lib” on the page. the snippet provides more meaning for machines by asserting that there is an “unorganization” with the “eponym” of “code4lib.” the itemscope attribute creates an item and requires no value. the itemtype attribute asserts that the type of thing being described is an “unorganization.” this item has a single key-value pair–the property “eponym” with a value of “code4lib.” to make it clear to someone who thinks in json, here is what the item looks like: { "type": [ "unorganization" ], "properties": { "eponym": [ "code4lib" ] } } those are the only three attributes necessary for a complete understanding of the microdata model. you’ll learn about two more optional ones later. pretty simple, right? what is schema.org? while the above snippet is completely valid microdata, it uses arbitrary language for its itemtype and itemprop values. if you only need to communicate this information within a tight community and do not need anyone else to ever understand what your data means, that may be just fine. but for the most part you probably want many other machines (like web crawlers) to understand the meaning of your content. to accomplish this you need to use a shared language so that page authors and consumers can cooperate on how to interpret the meaning. this is where the schema.org vocabulary comes in. the search engines (bing, google, yahoo!) created schema.org and have agreed to support and understand it. it is unrealistic for them to try to support every vocabulary in use. schema.org is an attempt to define a broad, web-scale, shared vocabulary focusing on popular concepts. it stakes a position as a “middle” ontology that does not attempt to have the scope of an “ontology of everything” or go into depth in any one area. a central goal of having such a broad schema all in one place is to simplify things for mass adoption and cover the most common use cases. understandably, the vocabulary does seem to have a bias towards search engine and commercial use cases. the type hierarchy presented on this site is not intended to be a ‘global ontology’ of the world. it only covers the types of entities for which we (microsoft, yahoo! and google), think we can provide some special treatment for, through our search engine, in the near future. (schema.org data model) you can browse the full hierarchy of the vocabulary to get a feel of the bounds of the world according to search engines. schema.org defines a hierarchy of types all descending from thing. thing has four properties (description, image, name, url) which are inherited by all other types. child types can add their own properites and in turn can have their own children types. each property name has the same meaning when found in any type in the vocabulary. we will get to other specifics later in the tutorial. microdata and schema.org have a tight connection, though each can be used without the other. the search engines are currently the main consumers of schema.org data and have a stated preference for microdata. the schema.org examples are written using the microdata syntax. both are designed and work well together to make adoption simple (and less error-prone) for html authors. here is the above microdata example rewritten to use the schema.org organizationtype (and make more sense). <div itemscope itemtype="http://schema.org/organization"> <span itemprop="name">code4lib</span> </div> rich snippets the most obvious way that the search engines are currently using microdata is to use the embedded data on pages to display rich snippets in search results. if you have done a google search in the past two years, you have probably seen some examples of rich snippets showing up in your search results.you can see good examples of how powerful this is by doing a google recipe search for “vegan cupcakes”: this search result snippet for a recipe includes an image, reviews, cooking time, calorie count, some of the text introducing the recipe, and a list of some of the ingredients needed. this gives a lot more information to the user to help them decide whether to click on a particular result. snippets with this kind of extra information are reported to increase click through rates. google started presenting rich snippets in 2009. using embedded markup like microformats, rdfa, or microdata, page authors can influence what may show up in a search result snippet. both microformats and rdfa were promoted for rich snippets in the past using various vocabularies and continue to be supported. google added support for microdata for rich snippets in early 2010. after rdfa lite was created, the schema.org partners agreed to support that syntax as well. before schema.org, rich snippets were constrained to being triggered by the few types defined by data-vocabulary.org, which prefigured the schema.org approach. reviews, products, and breadcrumbs were the kinds of common types of data that could be marked up. at the time of this writing there is still not rich snippet support for all, or even most, of the schema.org types. the number of supported types and example rich snippets has been slowly growing. the promise is that many more of the schema.org types will begin to trigger rich snippets. microdata with schema.org is the latest search-engine preferred method for exposing structured data in html, and this is the main reason to use it. while other consumers for structured data using microdata and/or schema.org may appear in the future, the most compelling use cases are currently from the search engines, especially for rich snippets. by providing the search engines with more data on your pages, it improves the search experience of users and can draw them to your site. since most of the users of your site likely come through the search engines, this could be a powerful way to draw more users to your resources. the search engines may also start using structured data as signals for relevance, though they are unlikely to say much publicly about this. from a developer’s perspective there are many considerations beyond rich snippets for choosing a particular syntax. microdata has a natural fit with html and is designed for simplicity and ease of implementation. schema.org simplifies the documentation and choices to make. in my own experience implementing microdata and schema.org was much simpler than a past failed attempt at implementing rdfa using various vocabularies on a similar site. tutorial this tutorial will move through implementing microdata and schema.org on a pre-existing site for exposing digitized collections. the example is based on ncsu libraries’ digital collections: rare and unique materials. each section will lead you through decisions that are made and common problems that are encounted in implementing microdata and schema.org. along the way are tips, tricks, tools, and a closer look at the specifications. before microdata here’s a screenshot of the page to mark up. as we add microdata markup, the appearance of the page will not change at all. the page uses a grid system to place the image on the left and the metadata to the right. these students are jumping with excitement to learn more about html5 microdata and schema.org! here is the basic structure of the main content of the page with some sections and attributes removed with ellipses for brevity. <div id="main" class="container_12"> <h2 id="page_name"> students jumping in front of memorial bell tower </h2> <div class="grid_5"> <img id="main_image" alt="students jumping in front of memorial bell tower" src="/images/bell_tower.png"> </div> <div id="metadata" class="grid_7"> <div id="object" class="info"> <h2>photograph information</h2> <dl> <dt>created date</dt> <dd>circa 1981</dd> <dt>subjects</dt> <dd> <a href="/s/buildings">buildings</a><br> <a href="/s/students">students</a><br> </dd> <dt>genre</dt> <dd> <a href="/g/architectural_photos">architectural photographs</a><br> <a href="/g/publicity_photos">publicity photographs</a> </dd> <dt>digital collection</dt> <dd><a href="/c/uapc">university archives photographs</a></dd> </dl> </div><!-item --> <div id="building" class="info"> <h2>building information</h2> <dl> ... </dl> </div><!-building --> <div id="source" class="info"> <h2>source information</h2> <dl> ... </dl> </div><!-source --> </div> </div> adding a webpage when you are adding microdata there is a sense in which you are always just describing a web page. ian hickson, the editor of the microdata specification, has said that microdata items exist “in the context of a page and its dom. it does not have an independent existence outside the page” (ian hickson on public-vocabs list). this is different than the way rdfa may think about embedding structured data in html as part of a graph which links items together across the web. microdata is not so much linked data as it is a description of a single page. there are efforts to serialize microdata as rdf that allow microdata to be used in a linked data environment. when using the schema.org vocabularies, every page is implicitly assumed to be some kind of webpage, but the advice is to explicitly declare the type of page. when picking an appropriate type, it is best to choose the most specific type that could accurately represent the thing to be marked up. in this case it seems appropriate to use itempage which is a subclass of webpage. itempage adds no new properties to webpage, but it communicates to the search engines that the page refers to a single item rather than a search results page or other type of page. in the html snippet below the itempage is added to the div#main on the page and some properties are added within the scope of that div#main. <div id="main" class="container_12" itemscope itemtype="http:schema.org/itempage"> <h2 id="page_name" itemprop="name"> students jumping in front of memorial bell tower </h2> <div class="grid_5"> <img id="main_image" alt="students jumping in front of memorial bell tower" src="/images/bell_tower.png" itemprop="image"> </div> <div id="metadata" class="grid_7"> ... </div> </div> here we apply the itemscope and itemtype attributes at a level in the dom that surrounds the item being described in the page. since we are describing the page, we could instead add the itemscope and itemtype at a higher level. if we need to use some metadata in the head of the document, say the title element, we could apply this to the body or even the html element, though there are some limitations to using microdata within head. an itemprop is added as an attribute to the element which contains its value. within div#main we add the two itemprop attributes for the “name” and “image” properties. different elements take their itemprop value from different places. in this case the name property is taken from the text content of h2 element. for the img element the value is taken from the src attribute, which is then resolved into an absolute url. the a element uses the absolute url from the href attribute. like img and a, several elements provide their values through an attribute rather than the text content. if you start using microdata, you will want to consult this list from the spec which details how property values are determined for different elements. microdata json and dom api one of the cool features of microdata is that it is designed to be extracted into json. you can copy any html snippet with microdata markup into live microdata to see what the json output would look like. live microdata uses microdatajs, which is a javascript (jquery) implementation of the microdata dom api. the microdata dom api is another neat feature of html5 microdata which allows you to extract microdata client-side. you can test whether your browser implements the microdata dom api by running the microdata test suite created by opera. if you open a page containing microdata with opera next (at the time of this writing version 12.00 alpha, build 1191) and open up the console (ctrl+shift+i), you can play with the microdata dom api a bit. document.getitems() will return a nodelist of all items. the nodelist will contain only the top-level items; in this case, one element. it is possible to get all items of a particular type by specifying the type or types as an argument like document.getitems('http://schema.org/itempage'). this api may change in the future “to match what actually gets implemented.” >>> var itempage = document.getitems('http://schema.org/itempage') undefined >>> itempage nodelist [<html itemscope="" itemtype="http://schema.org/itempage" lang="en">] >>> itempage[0].properties htmlpropertiescollection [<h2 id="page_name" itemprop="name">, <img itemprop="image" id="main_image" alt="students jumping in front of memorial bell tower" src="/images/bell_tower.png"/>] itempage about a photograph now that we have some basic microdata about the page, let’s try to describe more items. one valid property for an itempage, is about, inherited from creativework. itempage inherits different properites from thing, creativework, and webpage. in some cases child types add in new properites, but in the case of itempage no new properties are defined. one way to express what the page is about would be to add itemprop="about" to the div#metadata containing all of the metadata. what would be extracted from the page is just the text content within the div#metadata with line breaks and all. microdata processing never maintains markup. (if you need to maintain some snippet of markup, then consider using microformats or rdfa, which have the ability to maintain xml literals.) unlike xml, microdata is designed to be tolerant of errors. so in this case a processor may still be able to do something useful with even just the text content contained by the div#metadata. processors and consumers should expect bad data (conformance). looking at how the about property of an itempage is defined, the proper value of the about property is another thing (not text content). in this way schema.org suggests how to nest items within other items forming a tree. in this case specifying thing as a valid value of the about property means that any type in the schema.org hierarchy can be selected to create a new item as the value of the about property. the photograph type is most appropriate for this example. the way to show relationships between items on a page is by nesting items. nesting is implemented by using all three attributes (itemprop, itemscope, itemtype) on the same element. doing this states that the value of a property is a new item of a particular type. since all of the metadata describes the photograph, these attributes will be applied to div#metadata which contains all of the metadata. at this point the subjects and genres can be added as properties of the photograph. subjects map well enough to the keywords property of photograph. the older practice of using invisible meta keywords that had nothing to do with the content in the body were used to try to game the search engines, so they were ignored. the keywords on the example page are visible to users, so the values are less likely to be used to try to trick search engines to rank for particular terms. google advises page authors to not mark up non-visible content on the page, but to stick to adding microdata attributes to what is visible to users. we could attach the “genres” itemprop to the dd element, but then a processor would extract all the text contained within as a single chunk of text. the genres have some spaces within each term, so rather than leaving it up to a post processor to handle that, we apply the same itemprop to each term separately. this ensures that the multiword genres remain intact. at this time regardless of the use of singulars or plurals for property names in the schema.org documentation, it is allowable to repeat all schema.org properties like this. when these repeated properties are parsed, the values are merged under a single property key to form a list/array of values. the “keywords” itemprop cannot be applied to the a element that surrounds each term. the gotcha here is that the property value of an a element is the value of its href attribute. to work around this we use the common pattern of adding some extra spans around the text. adding the itemprop to the spans allows the text content to be selected instead. here is the markup so far: <div id="main" class="container_12" itemscope itemtype="http:schema.org/itempage"> <h2 id="page_name" itemprop="name"> students jumping in front of memorial bell tower </h2> <div class="grid_5"> <img id="main_image" alt="students jumping in front of memorial bell tower" src="/images/bell_tower.png" itemprop="image"> </div> <div id="metadata" class="grid_7" itemprop="about" itemscope itemtype="http://schema.org/photograph"> <div id="object" class="info"> <h2>photograph information</h2> <dl> <dt>created date</dt> <dd>circa 1981</dd> <dt>subjects</dt> <dd> <a href="/s/buildings"><span itemprop="keywords">buildings</span></a><br> <a href="/s/students"><span itemprop="keywords">students</span></a><br> </dd> <dt>genre</dt> <dd> <a href="/g/architectural_photos"><span itemprop="genre">architectural photographs</span></a><br> <a href="/g/publicity_photos"><span itemprop="genre">publicity photographs</span></a> </dd> <dt>digital collection</dt> <dd><a href="/c/uapc">university archives photographs</a></dd> </dl> </div> ... </div> </div> you can see what has been implemented in microdata so far as json. picking types and properties (and more nesting) after nesting a photograph within an itempage, the nesting can continue further. the photograph has the memorial bell tower at north carolina state university in the background. the photograph is about the memorial bell tower. in this case, landmarksorhistoricalbuildings seems to work well enough as a type. <div id="building" class="info" itemscope itemprop="about" itemtype="http://schema.org/landmarksorhistoricalbuildings"> once we add an item for the building, it is easy to add name and description properties for the building. we also have the data to add the address (as a postaladdress) and geographic location (geo property with a value of a geocoordinates) as further nested data items. picking appropriate types is an important issue for describing the historic record. at ncsu libraries we are describing lots of buildings and making related drawings accessible. some of these buildings are on the national register of historic places, but many are buildings of lesser note. some were never built or have since been demolished. it could be that for most of the records of buildings in an architectural collection, that a more generic item type like place would be a more appropriate fit. for other buildings there are definitely more specific types like airport, cityhall, courthouse, hospital, and church. using a more specific type, it would still be impossible for the search engines to recognize that the items refer to the historic record of these places rather than to their current services. we could be giving exact geographic coordinates for a hospital which was demolished or no longer in operation! also, this is a new metadata facet that may not already be captured in a way that makes it easy to convert to schema.org types. some retrospective and ongoing work would be needed to choose the correct type for every building. for now ncsu has made the decision, that will likely get revisited, to always pick landmarksorhistoricalbuildings for all the buildings we describe as something of a compromise. it almost communicates that the resource is part of the historic record, but it does lose on the specificity we might like to have. picking appropriate types is one area where schema.org does not provide anything near the level of granularity at which archives and museums often specify the types of objects they describe. schema.org has types for painting, photograph, and sculpture. some types of objects like drawings, vases, and suits of armor may have to move back up the hierarchy to use the generic creativework type. with the thousands of types of objects that may be held by libraries, archives, museums and historical societies (lams), it is not feasible to add every type to schema.org. one suggestion made by charles moad, director of the indianapolis museum of art lab, is to extend creativework with an “objecttype” property (private correspondence). if this “objecttype” property were part of schema.org, then the lam community could suggest that the value of the property come from a vocabulary that covers the kinds of objects lams collect. this could go a long way towards expanding the options for lams to describe their materials. this pattern of using external vocabularies maintained by domain experts is used in other places within schema.org. the jobposting type includes an occupationalcategory property which suggests the use of the bls o*net-soc taxonomy (though the exact format this value should take is underspecified). this still leaves open the question of what free, open vocabulary would allow for the kind of extensibility lams need to cover all of their object types at a good enough level of specificity. it is not feasible to create a vocabulary to cover and keep up with every type of thing lams may collect. the product ontology provides a possible model for how to create such an extensible vocabulary by reusing the names of wikipedia articles to identify types of objects. something like the following code could be done to simply reuse the name of the plate_armour article to note that the type of object in the collection is plate armour. the url http://objectontology.org/id/plate_armour could resolve to useful information and a tutorial similar to the product ontology page for plate armour. an object in a lam collection differs enough from a product for sale to need a new namespace to explain the difference, especially to machines. this kind of link also begins to make microdata more linkable data. <div itemscope itemtype="http://schema.org/creativework" itemid="http://www.philamuseum.org/collections/permanent/71286.html"> <span itemprop="name">gorget (neck defense) and cuirass (torso defense), for use in the field</span><br/> item type: <link itemprop="objecttype" href="http://objectontology.org/id/plate_armour"> plate armour </div> although it could be argued that the addition of objecttype is not actually necessary since the microdata data model allows for itemtype to be used with wikipedia urls. if compatibility with schema.org is not a concern the following is another option: <div itemscope itemtype="http://en.wikipedia.org/wiki/plate_armour" itemid="http://www.philamuseum.org/collections/permanent/71286.html"> <span itemprop="name">gorget (neck defense) and cuirass (torso defense), for use in the field</span><br/> </div> another possibility that would maintain compatibility with schema.org is the following based on a product ontology example. the example objectontology.org url is used here, but the wikipedia page url may work as well. <div itemscope itemtype="http://schema.org/creativework" itemid="http://www.philamuseum.org/collections/permanent/71286.html"> <span itemprop="name">gorget (neck defense) and cuirass (torso defense), for use in the field</span><br/> item type: <link itemprop="http://www.w3.org/1999/02/22-rdf-syntax-ns#type" href="http://objectontology.org/id/plate_armour"> plate armour </div> because there are multiple possibilities to express more specific types of objects, each with their own strengths, this is definitely an area where the cultural heritage community could come to some agreement and promote a shared approach to the problem. result so far here’s what our marked up snippet looks like so far: <div id="main" role="main" class="container_12" itemscope itemtype="http:schema.org/itempage"> <h2 id="page_name" itemprop="name"> students jumping in front of memorial bell tower </h2> <div class="grid_5"> <img itemprop="image" id="main_image" alt="students jumping in front of memorial bell tower" src="/images/bell_tower.png"> </div> <div id="metadata" class="grid_7" itemprop="about" itemscope itemtype="http://schema.org/photograph"> <div id="object" class="info"> <h2>photograph information</h2> <dl> <dt>created date</dt> <dd>circa 1981</dd> <dt>subjects</dt> <dd> <a href="/s/buildings"><span itemprop="keywords">buildings</span></a><br> <a href="/s/students"><span itemprop="keywords">students</span></a><br> </dd> <dt>genre</dt> <dd> <a href="/g/architectural_photos"><span itemprop="genre">architectural photographs</span></a><br> <a href="/g/publicity_photos"><span itemprop="genre">publicity photographs</span></a> </dd> <dt>digital collection</dt> <dd><a href="/c/uapc">university archives photographs</a></dd> </dl> </div><!-item --> <div id="building" class="info" itemprop="about" itemscope itemtype="http://schema.org/landmarksorhistoricalbuildings" itemref="main_image"> <h2>building information</h2> <dl> <dt>building name</dt> <dd><a href="/b/memorial_tower"><span itemprop="name">memorial tower</span></a></dd> <dt>description</dt> <dd itemprop="description">memorial tower honors those alumni who were killed in world war i. the cornerstone was laid in 1922 and the tower was dedicated on november 11, 1949.</dd> <dt>address</dt> <dd itemprop="address" itemscope itemtype="http://schema.org/postaladdress"> <span itemprop="streetaddress">2701 sullivan drive</span><br> <span itemprop="addresslocality">raleigh</span>, <span itemprop="addresslocality">nc</span> <span itemprop="postalcode">26707</span> </dd> <dt>latitude, longitude</dt> <dd itemprop="geo" itemscope itemtype="http://schema.org/geocoordinates"><span itemprop="latitude">35.786098</span>, <span itemprop="longitude">-78.663498</span></dd> </dl> </div><!-building --> <div id="source" class="info"> <h2>source information</h2> ... </div><!-source --> </div> </div> all told there are five microdata items on the page (itempage, photograph, landmarksorhistoricalbuildings, postaladdress, geocoordinates). in part, this snippet now basically says something like this in english: this page is an itempage about a photograph. the photograph is about a landmarksorhistoricalbuildings. the itempage has a “name” of “students jumping in front of memorial bell tower” and an “image” at “http://example.com//images/bell_tower.png.” the photograph has some “keywords” and “genre.”the landmarksorhistoricalbuildings has a “name” of “memorial tower” and a “description.” the landmarksorhistoricalbuildings has an “address” which is a postaladdress item (with its own properties), as well as, a “geo” property which is a geocoordinates item. the nested item types could be represented by this image: you can see the extracted json at live microdata under the json tab. itemref so far the image is not associated with the photograph or any child items on the page. that needs to be fixed, because a rich snippet for a photograph is unlikely to show up without a value for the image property. the problem is that the image is not nested within the same div where the photograph is defined. valid html is particularly important in pages that contain embedded markup. all methods of embedding data within html use the structure of the html to determine the meaning of the additional markup. (choosing an html data format) while we have the content on our page relatively well organized to contain our items, our layout and grid system result in the image of the photograph not being nested within the same div as the metadata about the photograph. when coding new pages, it is important to think about how to structure the page with both presentation and data in mind. it makes marking up data easier if the properties of a thing are grouped together. while in most cases it will be easier to arrange our markup so that the properties of an item are all within the same block on the page, there are times like this where the style and layout of the page determines the structure. microdata provides a rather simple mechanism for including properties that are outside of an item’s scope. microdata uses the itemref attribute to make this more convenient. note: the itemref attribute is not part of the microdata data model. it is merely a syntactic construct to aid authors in adding annotations to pages where the data to be annotated does not follow a convenient tree structure. (microdata specification itemref attribute) in our example above the img already has an id of main_image and an itemprop with the value image. all that we need to do to use that image property for the photograph is add itemref="main_image" to div#metadata. the itemref adds elements with that id attribute to the queue of locations in the dom to check for properties for the item created on the same element. here is the relevant markup: <img id="main_image" alt="students jumping in front of memorial bell tower" src="/images/bell_tower.png" itemprop="image"> ... <div id="metadata" class="grid_7" itemprop="about" itemscope itemtype="http://schema.org/photograph" itemref="main_image"> ... </div> the div#metadata which contains all of the information about the photograph now uses four microdata attributes. the same itemref value could be added to the building to associate the page image with that item as well. problems with rich snippets no rich snippet would show up in search results or the rich snippets testing tool for this example right now. even though the microdata is valid and multiple schema.org items properly marked up, at the time of this writing, google would not use data from any of the possible items to create a search result snippet. google currently only supports rich snippets for several of the schema.org types (applications, authors, events, movie, music, people, products, products with many offers, recipes, reviews, tv series– not all of which are schema.org types). for each of these item types google requires that certain properties be present in order for a rich snippet to have a chance of showing up. even then a rich snippet may only show up for a page if an item is relevant to the users query. but using the structured data linter we get this possible preview for the landmarksorhistoricalbuildings item. this is exactly the kind of attractive rich snippet we want users to see for digitized resources. the snippet includes the image and address to make a more clickable target. hopefully the search engines will begin showing snippets for some of the item types cultural heritage organizations are most likely to be making available. there may also be an opportunity for cultural heritage organizations to build their own local and cross-institutional search engines that take more advantage of more detailed microdata. time and datatypes another piece of data which would be good to add to the photograph is the date created using the datecreated property with an expected value of date. the microdata specification (unlike rdfa and microformats-2) does not provide a generic mechanism for specifying data types. instead microdata relies on a vocabulary to define expected data types. schema.org has four basic types: boolean, date, number, and text, but none of them are very well documented. html5 has some new elements to allow inclusion of machine-readable data in html and these can be used for the schema.org datatypes. the time and data elements have been added to html5. the specification of time has not been stable. in fact the time element was removed from html5 with some strong objections and lots of exciting specification drama. so while the time element is in the specification again, the schema.org and google documentation add confusion to the matter by using the meta element instead when the content is a date. lots of the examples look like this: <meta itemprop="startdate" content="2016-04-21t20:00"> thu, 04/21/16 8:00 p.m. using the meta and link elements within the body element of a document is allowed in html5 when used with an itemprop attribute. these elements can sometimes be useful for microdata in expressing the meaning of content or a url which has no usefulness to a human reader. the meta element above is used to give the machine readable date near the date visible to the user. it is hidden on the page, so goes against the general recommendation to not use hidden markup for microdata. further, the meta element does not surround the free text version of the date disassociating them from each other. the same data as above could be marked up using <time> like so: <time itemprop="startdate" datetime="2016-04-21t20:00"> thu, 04/21/16 8:00 p.m. </time> the time element has the correct semantics for the photograph: <dd> <time itemprop="datecreated" datetime="1981"> circa 1981 </time> </dd> i have made no attempt to handle the approximate date (“circa”). the current processing rules in the specification do not handle many valid iso8601 dates. as dates and ambiguity about dates is important for describing cultural heritage materials, hopefully the html5 processing rules can be adjusted to handle more valid iso8601 dates. it seems as if the whatwg has accepted a proposal to support year only dates, which is a start. itemid some items have unique identifiers or canonical representations elsewhere on the web that can be used to link resources together. the memorial bell tower is a unique landmark that could be linked together with other representations of the same place. this linking could help machines to make connections between resources or users to follow their nose to related representations. in microdata the itemid attribute can be used to associate an item with a globally unique url identifier, “so that it can be related to other items on pages elsewhere on the web.” this is the main mechanism by which microdata natively supports something like linked data. when serialized to rdf the itemid of an item would become the global identifier used as the subject of rdf assertions. the other urls used as values throughout microdata do not provide this linkability, and the assumption is that consumers will have a built-in knowledge of the vocabulary used for item types and properties in order to make sense of the data. microdata does not adhere to the “follow-your-nose principle, whereby vocabulary authors are encouraged to provide a machine-readable description of classes and properties at the url used for the class or property,” like rdf does (html data guide, w3c editor’s draft 08 january 2012). item types are opaque identifiers, and user agents must not dereference unknown item types, or otherwise deconstruct them, in order to determine how to process items that use them. (html microdata: items) the meaning of the itemid attribute is determined by the vocabulary. unfortunately, schema.org does not document any use or support for the itemid attribute at this time, though they “strongly encourage the use of itemids.” the semantics of itemid in the schema.org context seems uncertain and overlapping with the consistent use of url property names. we’ll add an itemid in any case. the memorial bell tower can be represented by this freebase uri: http://www.freebase.com/m/026twjv <div id="building" class="info" itemprop="about" itemscope itemtype="http://schema.org/landmarksorhistoricalbuildings" itemid="http://www.freebase.com/m/026twjv"> extending schema.org another kind of property a cultural heritage organization might like to add to a landmark or building like the memorial tower are the events related to the building. in this case the cornerstone was laid in 1922 and the tower dedicated on november 11, 1949. other buildings could have events in their history like the dates they were designed or dates of renovations, derived from the drawings and project records. museums may be interested in various events in the history of a painting including provenance and restorations. history museums and historical societies may also want to refer to various historical events that relate to their exhibits. each of these institutions may also want to promote various future events like movie screenings, limited time exhibits, and tours. so it may be important to be able to disambiguate whether an event is in the future or of some historical significance. schema.org has an event type defined as: “an event happening at a certain time at a certain location.” that seems broad enough to apply it to either future or historic events. but the message from google’s support page on rich snippets for events gives more specific guidance: the goal of events snippets is to provide users with additional information about specific events not to promote complementary products or services. event markup can be used to mark up events that occur on specific future dates, such as a musical concert or an art festival. google has a different view of the world than cultural heritage organizations and historians. because of the focus on the future, we may not want to try to mark up historic events, as rich snippets may be unlikely to show up for past events. another option for publishing data about this kind of event in the life of an object may be to use the schema.org extension mechanism to try to make it clearer to the search engines that certain types of events are different. as schema.org is intended as a web-scale schema, there is no possibility of having it fit every kind of data on the web. the basic mechanism for extending an item type is to take any schema.org type url, add a forward slash to the end, and then add the camel cased name of the extension. so one possibility to handle historic events would be to extend event: http://schema.org/event/historicevent at least the search engines will understand that these items are some type of event. if enough other folks use the same extension and the search engines notice, then the search engines may start using the data in a meaningful way. this is one way to grow the schema organically with actual use influencing the vocabulary, though there are those who question this extension mechanism and there are no assurances that extensions will get used. schema.org lacks a good, public, formal process for sharing extensions and advocating for their inclusion in the vocabulary. work still needs to be done to build up a clear central location to share extensions or have a community process to work out new extensions. dan brickley has noted that the properties available in the schema.org types for describing museum objects are a little “sparse” compared to some controlled vocabularies used by museums. for properties there are two options for mixing in a new property for an existing (or extended type). schema.org prefers page authors to add the new property name as if it were defined by schema.org. so our rights statement could be given this markup: <dd itemprop="rights"> reproduction and use of this material requires permission from north carolina state university. </dd> the other option provided by the microdata specification is to add a full url for the property like this: <dd itemprop="http://purl.org/dc/elements/1.1/rights"> reproduction and use of this material requires permission from north carolina state university.</dd> so while anyone can easily extend schema.org types and properties, there needs to be some community using or consumer understanding the same extensions in a consistent way for the extensions to have any usefulness. do not expect the search engines to do anything with property names that are not documented on schema.org. another way forward for the cultural heritage sector? these are just some of the ways in which schema.org may not fit the needs of cultural heritage organizations very well. other communities have already voiced their concerns that their domains are not adequately specified. for instance, the rnews standard of the international press and telecommunications council was in large part added to schema.org. this was the first industry organization to work with the schema.org partners to publish an expansion of schema.org to make it more robust and useful in a particular domain. the adoption of much of rnews has resulted in news and publishing-related types being more expressive and better meeting the needs of news organizations. another change to the schema was the result of a collaboration between schema.org and the united states office of science and technology policy to add support for job postings. these additions were immediately put to use to create a job search widget for government web sites to highlight job listings from employers who commit to hiring veterans. (see the veterans job bank.) so this kind of partnership with domain experts seems like a way forward for other groups. libraries, museums, archives, and other cultural heritage organizations could start to enumerate the places where the vocabulary could be modified to better fit their data and use cases. there has been some suggestion for future discussions with the cultural sector to this end. one suggested location for this kind of activity is the w3c wiki, which points to the simplicity of the submission for job postings. another place this work is being done is in the schema.org alignment task group of the dublin core metadata initiative, which has a multi-decade history of working internationally with web metadata in the cultural heritage sector. some of this work may already be available. work is ongoing to map other vocabularies to schema.org. this provides a simpler way for organizations to expose their data through microdata and schema.org while still maintaining their data in their current schema. in each of these mappings there are certainly some areas where there is not overlap, so there is potential for expanding schema.org in those directions. conclusion microdata and schema.org provide a relatively simple way for libraries, archives, and museums to begin to expose their data in new ways and make their collections more discoverable and useful. the semantic markup of data embedded in html is a rapidly changing area, and much of what is written here is likely to change. while this is challenging for implementers, it also provides a chance for cultural heritage organizations to enter the conversation, and build new tools that actually use the newly available structured data. there is a huge opportunity to have an impact on these technologies to improve the discoverability and use of our collections and services. appendix: resources examples from cultural heritage organizations ncsu libraries’ digital collections: rare and unique materials the example in this tutorial is based on this site. indianapolis museum of art uses creativework and extends the type by adding the following properties: accessionnumber, collection, copyright, creditline, dimensions, materials. biodiversity heritage library adds an “oclc” property for a book. sudoc french academic union catalogue seems to only show the microdata representation to crawlers? more information sindice search this search can be adjusted to find a specific schema.org type (class here). faceting by the microdata does not always seem to find sites that predominantly use microdata rather than rdfa. tools these are tools which i have regularly used. rich snippets testing tool note that while it may not currently show a rich snippet example for every schema.org type, you can use the data at the bottom of the page to insure that your microdata is being parsed as you intended. the format here breaks every item out and shows references between them in a flat way. structured data linter the best feature of this tool is the way that it displays your nested microdata as nested tables, making it easy to spot problems. if the rich snippets testing tool doesn’t show a rich snippet for your content, this is a good alternative to see what your snippets might look like.the snippets here do not cover every type either, but they cover a few different types from what the rich snippets testing tool does, for instance it will show images for more types. the code is open source, so you can run your own instance to be able to check your syntax while you are in development. this is written by folks who have been part of the conversations around web vocabularies and structured data in html. live microdata a good open-source tool for testing snippets of html marked up with microdata. the microdatajs source code allows you to implement the microdata dom api on your own site, similar to how this tutorial outputs the json from parsing the page. html5 living validator schema.rdfs.org list of tools and libraries mappings there are many efforts to map schema.org to other vocabularies. if you maintain your metadata in one of these vocabularies, you can use these to help expose your data in a way that the search engines understand. dublin core alignment task group partial bibo mapping alignment between rnews and schema.org schema.rdfs.org list of mappings other tutorials google rich snippets videos: this series of short tutorial videos uses microdata and schema.org. getting started with schema.org the schema.org examples have been reported to include some bugs. html data guide: this guide helps producers and consumers determine which structured data syntax to use. it covers microformats, rdfa, and microdata, and is highly recommended. why rnews? is a good tutorial on how organizations often have good structured data that when made accessible through html loses its meaning. dive into html5: “distributed,” “extensibility,” and other fancy words spoonfeeding library data to search engines extending microdata vocabularies discussions public vocabs list schema.org feedback and discussion has moved over to this w3c list, and other web vocabularies may join them in the future. w3c html data task force w3c web schemas task force about the author jason ronallo is associate head of digital library initiatives, ncsu libraries. he also writes on his blog preliminary inventory of digital collections. subscribe to comments: for this article | for all articles 16 responses to "html5 microdata and schema.org" please leave a response below: dan brickley, 2012-02-07 hey, thanks for sharing such a in-depth study of this, and for encouraging the lam community to engage with schema.org rather than sit on the sidelines. this is the perfect time to be finding middle-ground and useful compromises… zazi, 2012-02-07 nice write up and interesting that you’ve choosen a json translation in your first example section. i guess, the same aha effect could be achieved via showing an rdfa snippet first and then it’s turtle translation. … it’s always the same game. there is no real magic in microdata :) niklas lindström, 2012-02-08 hi jason; thanks for the interesting write-up! regarding: ” because rdfa simply was not going to be incorporated into html5, something else was needed to fill the gap.” thankfully, this isn’t the case nowadays. see e.g.: http://dev.w3.org/html5/rdfa/ rdfa 1.1, now in last call, is rapidly approaching a final recommendation. it addresses a huge amount of the criticism of 1.0. most importantly, by virtue of rdf, it already handles all kinds of data challenges by design, such as schema extensibility and making precise, decentralized statements about any kinds of things (i.e. not only the pieces of a page). not to mention the huge toolset and storage implementations available, and the general linked data movement (and research space) at large. i really recommend trying out rdfa 1.1. note that lite is mainly designed for newcomers to grasp the basic shape and features of rdfa. any rdfa 1.1 processor shall support rdfa 1.1 in full though, so depending on the richness and variety of your data, you have the precision available to you if you need it. that said, the lite subset should cover most regular needs, unless you use e.g. specific datatypes (other than date and time types, which should be covered by @datetime for rdfa in html5) or need @rev. i have made an rdfa 1.1 version of your example which i put at together with a turtle extract. (along with attribution of course, as per your license. naturally i’ll remove it anyway if you don’t approve of it being there.) it is mostly rdfa lite, but i used @datatype to capture the geo coordinates as decimals, and i used @resource throughout instead of @about, which is currently under debate. of course in any case it should be valid rdfa 1.1. (granted, there may be some conflations of identity here (the png not being the original photograph and the freebase resource possibly not being the landmarksorhistoricalbuildings but a record about it). but that’s another discussion.) niklas lindström, 2012-02-08 ups, i put the link to my rdfa 1.1 version in angle brackets. here it is: https://gist.github.com/1775752 bob bartley, 2012-08-15 enjoyed your article/site on microdata and rich snippets! it was concise yet very informative both in syntactical explanations and marked-up html coding examples. website also allowed convenient copy and print functions to manipulate scource code examples as needed. right on! microdata and schema.org?? | web crawling, 2013-02-10 […] it has been a day or two or a year… so i jumped at the chance to read jason ronallo’s post on microdata!  it was very difficult to get through at first.  i actually read it about three times and then […] more on microdata « a wild metadata appears!, 2013-02-12 […] annie, i found the jason ronallo post a bit over my head (i’m currently taking ls 560, so not having that in my background may […] peter, 2013-05-09 hi, all these news capabilities with html 5 and javascript have brought the same problems raised in the past with adobe flash websites. everything is limited to only one page and all content gets loaded through ajax or flash calls to the server. we know that search engines crawl servers for what google calls a “html snapshot” in order to understand the data a website holds. problem is that when you’re using a javascript page that has nothing to show up until there is an ajax call with or without human interaction nothing exists to be analised. there are techniques for creating deep linking by using the hashtag but none of this is used at server level. so to get all content of this kind of website indexed on google you need to go through these steps shown at https://developers.google.com/webmasters/ajax-crawling/docs/getting-started now this is a process that requires some knowledge programing server side scripts and forces a swap of the page location in order to provide a version for servers and a version for javascript, and therefore to make a website have javascript deep linking while still being indexed at all it’s content in search engines. here comes my question: would it be possible to provide snippets of content inserted in one page only (index.html), having all the schema.org provided for each content with the related link containing the hashtag logic, and in the end get indexed for all content? an example would be a website containing an animated timeline with: avatar read gladiator read …. and in the end have google recognize the content for: avatar http://www.domain.com/#/event-01 and gladiator http://www.domain.com/#/event-02 thanks peter peter, 2013-05-10 my comment got parsed and is not correctly displayed on the html example. you can find my example displayeda at http://jsfiddle.net/petermonte/25mxy/1/ the idea is to show the html markup on the top left textarea. by reading the diference between schema.org and facebook open graph i can see that the first one is better suited to list more than one entity in a single page, i quote: “q: how does schema.org relate to facebook open graph? facebook open graph serves its purpose well, but it doesn’t provide the detailed information search engines need to improve the user experience. a single web page may have many components, and it may talk about more than one thing. if search engines understand the various components of a page, we can improve our presentation of the data. even if you mark up your content using the facebook open graph protocol, schema.org provides a mechanism for providing more detail about particular entities on the page. for example, a page about a band could include any or all of the following: a list of albums a price for each album a list of songs for each album, along with a link to hear samples of each song a list of upcoming shows bios of the band members” could the correct use of schema.org microdata and a well structured sitemap.xml serve the purpose of getting a one single page indexed in google for every data it contains seperatly with a link containing the hashtag? sorry if i’m pointing an issue that this article doesn’t aim. is your library using schema.org? | eduhacker, 2013-06-10 […] looking for an introductory article and some background on schema.org, please read “html5 microdata and schema.org” by jason ronollo published in the code4lib journal. months after ronollo’s article, […] analysis of the ncsu library urls in the common crawl index – common crawl – blog, 2014-09-30 […] the discoverability of this content on the open web. i implemented embedded semantic markup, microdata and schema.org, on this […] ester, 2014-12-08 thank you so much for the article, i am writing about the microdata and schema.org and with you permission i will quote you :) html5 microdata and schema.org | thoughts on metadata, 2015-03-28 […] http://journal.code4lib.org/articles/6400 […] learning about html5 from the w3c (on edx) |, 2015-06-18 […] most interesting concept so far (to me) is that of microdata. i don’t why it interests me, exactly, but it seems like a neat concept. along with the new […] no deed goes unpunished | gavia libraria, 2015-09-01 […] the internet movie database, perhaps? but even in the placid waters of library-and-archives-land, microdata is helping with search-engine optimization here and […] analysis of the ncsu library urls in the common crawl index – common crawl, 2016-03-07 […] the discoverability of this content on the open web. i implemented embedded semantic markup, microdata and schema.org, on this […] leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – exploring information security and shared encrypted spaces in libraries mission editorial committee process and structure code4lib issue 29, 2015-07-15 exploring information security and shared encrypted spaces in libraries libraries are sensitive to the need to protect patron data, but may not take measures to protect the data of the library. however, in an increasingly collaborative online environment, the protection of data is a concern that merits attention. as a follow-up to a new patron privacy policy, the oakland university william beaumont medical library evaluated information security tools for use in day-to-day operations in an attempt to identify ways to protect private information in communication and shared storage, as well as a means to manage passwords in a collaborative team environment. this article provides an overview of encryption measures, outlines the medical library’s evaluation of encryption tools, and reflects on the benefits and challenges in their adoption and use. by keith engwall introduction privacy is a fundamental aspect of libraries. information security has been a concern of libraries for over 75 years, particularly in terms of patron privacy, as evidenced in the very first code of ethics adopted by the american library association in 1939: “it is the librarian’s obligation to treat as confidential any private information obtained through contact with library patrons” (ala 1939). however, whereas information security in 1939 consisted mainly of locking paper records in a file cabinet, the challenges in today’s digital world are far different. in addition to protecting patron information, libraries must maintain other forms of sensitive information – account numbers, server passwords, etc. keeping this information secure can be a challenge, particularly when it must be shared. managing collaborative environments can be difficult in its own right, and most privacy technology is developed for use by individuals. on top of this, other more subtle issues may stand in the way of our efforts at information security. as our interactions through technology have increased, so has our acceptance of risk as it pertains to privacy. in our personal lives, we conduct financial transactions, log our searches, map our comings and goings, and share the details of our day-to-day lives, exposing a trove of private information to the world without much consideration as to how it may or may not be protected. meanwhile, at work, librarians seek opportunities to collaborate, and there are a plethora of free tools available to us for that purpose. we use tools like twitter, anymeeting, evernote, trello, google docs, and others to seamlessly communicate, coordinate, and create across great geographical distances. in both cases, we allow ourselves to be fully reliant on policies and measures put in place by others, often without sufficient understanding of the measures used to adequately evaluate their efficacy. in doing so, we prioritize the benefits we receive from these services over knowledge and/or control of the security protecting the information we put therein. to be sure, we understand the difference between putting our own information at risk and that of our patrons, but an argument can be made that our relaxed attitudes towards our own information security may leave us less prepared when it comes to protecting the information of our patrons. conversely, by developing better security habits for ourselves, we may improve our ability to protect the information of our patrons in our digital environment. as librarians, we draw great benefit from collaborative efforts. together, we are able to overcome limited time and resources to make better decisions, create stronger and more consistent policies, and conduct complex research on extremely limited budgets. however, due to limited budgets, we may not be able to avail ourselves of enterprise-level tools with built-in security, such as those used by corporations, hospitals, and other large institutions to protect the information of customers, patients, and corporate research and development. instead, we tend to rely on free or inexpensive software or services designed primarily for light, personal or casual use, such as basecamp, google drive, evernote, trello, and others. some of these services and/or software, such as evernote, provide information about their security measures, while others provide little or no information. we must trust that our information is being protected and is not vulnerable to theft or disclosure. fortunately, there are free and open-source tools available to allow us to take information security into our own hands. however, the burden of locating, implementing and using security software is on us. in 2014, the medical library at oakland university william beaumont (ouwb) school of medicine took part in a task force, alongside the broader university library, to investigate patron privacy. the task force produced a joint patron privacy policy for the libraries, as detailed in a recent article in the journal of academic librarianship (hess 2014). as next steps in this process, the medical library was tasked with taking measures to identify gaps in information security and improving protections of patron information. as an extension of this process, we decided to consider other ways in which we might improve information security, not only for patrons, but for other areas, such as securing research data and internal information. by taking a broader, more comprehensive approach to information stewardship, we would develop better habits and be more successful in protecting not only our patrons’ information, but that of the library and ourselves, as well. the medical library decided to investigate how it might be able to provide a better information security infrastructure, not necessarily to dictate specifically how information is secured, but rather to provide tools to support information security. in some cases, available security measures do not fit into our current workflow, while others can be more readily implemented. we do not have any comprehensive solutions to the problem of information security, but we are taking steps to become more security conscious and will be evaluating our general information security practices alongside our efforts to protect patron privacy. our investigations covered secure information, secure information storage, and password management. information security basics to better understand how security software protects data, it helps to have a basic understanding of how data is protected. this section provides an overview of encryption and how it is used by security tools, such as those covered in this article. keys and key lengths in encryption, an unencrypted, sometimes called plaintext, object, such as a document, file, or string of data, is scrambled using mathematical algorithms. the information within an object (characters, pixels, sequences of code, etc.) is represented digitally by a sequence of numbers. by passing these numbers through a complex mathematical algorithm along with a unique, random, very long number, they become scrambled, and thus, encrypted, as shown in diagram 1. this unique, random, very long number is called a key. diagram 1: an object is encrypted using an algorithm and a key. the strength of the encryption is dependent on the length of the key. the length of a key is measured in bits, which use the digits 0 and 1. a key of length n bits has 2^n possible values (for example, a 1-bit key has 2 possible values: 0 or 1). a 128-bit key has 2^128, or 340,282,366,920,938,463,463,374,607,431,768,211,456 possible values. a 256-bit key has the square of that many values. the longer the key, the longer it would take someone to guess that key and decrypt the encrypted object. symmetric encryption symmetric encryption uses the same key to encrypt and decrypt an object, as shown in diagram 2. diagram 2: the same symmetric key is used for encryption and decryption. the advanced encryption standard (aes) and twofish are symmetric encryption algorithms. aes-128 or aes-256 are common implementations of this algorithm, using 128-bit or 256-bit keys, respectively. twofish uses a similar naming convention (e.g. twofish 256). symmetric encryption is fast and very secure, and is what is typically used to encrypt files, text, and other objects. each encrypted object is typically assigned a new key by the software handling the encryption, and may change each time the object is encrypted (symmetric-key… 2015). asymmetric encryption asymmetric encryption uses a pair of keys to handle encryption and decryption. it is often used in the transfer of data from one party to the other. each party has a pair of keys. one of these keys is public and can be made available to anyone. the other is private and kept secret (public-key cryptography 2015). rsa encryption rsa (named for its authors: ron rivest, adi shamir, and leonard adleman) encryption uses one key to encrypt data and another key to decrypt it. typically, the sender will encrypt an object using the recipient’s public key, and the recipient will decrypt the object using their own private key. this is shown in diagram 3. diagram 3: in asymmetric encryption, different keys are used for encryption and decryption because asymmetric encryption is far slower than symmetric encryption, it is sometimes used to encrypt a symmetric key rather than the data itself, as shown in diagram 4. the encrypted key is transferred and decrypted and then can be used to encrypt and decrypt the data, as shown in diagram 1 above. diagram 4: rsa encryption can be used to encrypt symmetric keys, rather than data. dh encryption dh (named for its authors: whitfield diffie and martin hellman) also uses asymmetric public and private keys, but combines each party’s public key with the other’s private key to create a secret shared key known only to the two parties, as shown in diagram 5. diagram 5: dh encryption combines private and public keys into a shared secret key. the shared key is then used as a symmetric key that both parties can use to encrypt and decrypt data transferred between them, as shown in diagram 6. diagram 6: the shared secret key is used for symmetric encryption. both rsa and dh make use of prime numbers, and since the number of prime numbers is relatively small in comparison to all numbers, asymmetric keys must be extremely long, anywhere from 1024 to 4096 bits long. in most cases, a new key pair is generated for each secure session. there are many other encryption algorithms in use, but aes, rsa, and dh are very common. digital certificates digital certificates are important to confirm the identity of a server with which a client is attempting to establish a secure connection. in these cases, the server’s certificate is issued by a trusted certification authority, which can be checked during the initial connection to validate the certificate. a less common use of digital certificates is for users to create their own (self-issued) certificates for use with some secure communication protocols (public-key certificate 2015). tls and ssl transport layer security (tls) and its predecessor, secure sockets layer (ssl), are cryptographic protocols for secure communication (transport… 2015). ssl has been in use since the mid 90’s, but has been superseded by tls. both use certificates and asymmetric encryption to establish secure communication channels, but tls contains a much larger set of encryption algorithms that the server and client may use for encrypted communication. when you connect to a website with a browser, if the url begins with https:// instead of http://, you are making a secure connection that will use either ssl or tls, depending on the server and the browser (browser support of tls has been ubiquitous for about a year). authentication, hash and salt just as digital certificates confirm the identity of servers, authentication is used to confirm the identity of a user on that server. users create a password, which is passed through a hash algorithm and the hash is stored on a server. unlike encryption, which is an algorithm intended to be reversible (decrypted), a hash is a one-way algorithm that can never be reversed. the output of this algorithm is a hash, which is stored on the server. each time a person enters their password, it is passed through the same hash algorithm and produces the same hash. by comparing the hashed password with the hash on file, the server is able to confirm the identity of the user. this protects the user in the instance of a security breach. even if an attacker gains access to the password hash file, they would still need the original passwords to gain access to user accounts. they would not be able to derive those passwords from the hashes, except through a “brute force” algorithm that guesses passwords, runs them through a hash algorithm, and compares the hash to the stolen hash table. a sufficiently strong password would take hundreds or even billions of years to guess. by itself, a hash algorithm would generate the same hash for the same password chosen by multiple users. repetition of hashes would tip off an attacker that those users have likely chosen weak and/or common passwords. to prevent this, extra data (usually random), called a salt, is added to the password before it is passed through the hash algorithm, thus producing different hashes. in order to replicate the hash during authentication, the salt must be stored on the server along with the username and password (salt… 2015). multifactor authentication another security measure that has started to show up as a feature in a variety of software and websites is multifactor (or 2-factor, 2-step, etc.) authentication. this is an additional layer of security that is used to confirm your identity. this additional authentication step may be handled through a security question you set up ahead of time, entry of a code texted to your phone, a physical key such as an encoded usb device, or an application such as google authenticator. the first time you log in on a new computer or device, you are prompted to provide the second authentication step. you generally can choose at that time whether to register the new computer or device (such as when you purchase a new one), so that it is recognized on subsequent logins, removing the necessity of the additional authentication step. you can elect not to register the computer or device (such as when you are using a public computer), which will cause you to be prompted for the additional authentication step the next time you log in. the idea is that someone who had discovered your password would still be unable to authenticate without possession of a physical object, such as your phone or usb key. how it fits together so, how does everything fit together? whether through a browser or through client software, initial connections to online services are typically encrypted using tls or ssl, during which the validity of the server’s certificate is likely to be confirmed, and private and public session keys are established and exchanged to create a secure connection. the user logs in with a username and a password. if the server uses salt, it uses the username to look up the salt for the hash. the password is hashed and compared to the hash on file. once the user has been authenticated, access to the service is granted. all information transferred to and from the service will at least be encrypted during the transfer using the session keys established through tls/ssl. depending on the type of service being used (communication, social media, productivity, cloud storage, etc.) and the security policy of the service, information (files, images, text, etc.) stored within the service may or may not be encrypted on the server. if it is encrypted, it will likely be encrypted using symmetric cryptography, such as aes. this encryption will occur on the server and the service provider will be responsible for managing the keys for the encrypted data. one major risk to storing sensitive information online is that an attacker who gains access to either the server or to the user’s account may also be able to gain access to the unencrypted content. last year, the discovery of the heartbleed vulnerability, as described in a paper presented at the 2014 conference on internet measurement (durumeric 2014), highlighted how insecure this model can be. through a bug in openssl, an attacker would be able to silently and repeatedly gain access to the contents of server memory, around 64kb at a time, which may contain cryptography keys, usernames, and passwords. thus, an attacker might be able to obtain keys to decrypt individual files (which would require gaining access to the server) or to a user’s account. the latter scenario is particularly serious, since upon successful user authentication, the server would automatically decrypt any content requested by the attacker. another security risk to consider is that if a service provider has full control over the content, it is able to decrypt information as it deems necessary, such as in response to pressure from government agencies. although multifactor authentication would provide a measure of protection against vulnerabilities such as heartbleed, it would not provide any protection against intentional disclosure. zero-knowledge encryption zero-knowledge encryption is a term describing the process of encrypting information offline and storing the encrypted object on a server without also uploading the symmetric key. the server, therefore, has zero knowledge of the key and cannot decrypt the content. an attacker who gains access to the server or to the user’s account would only be able to access the encrypted content, and since the key is never transferred to the server, it is not vulnerable to discovery through heartbleed or other attacks. software that encrypts information on the user’s computer or device provides significant added security. zero-knowledge encryption is not bullet-proof. even though the security key is not stored on the server, a brute force hack of a password will generate the key necessary to gain access to the encrypted data. however, that is the only method of attack that will work. other methods, such as tricking the service into letting an attacker change passwords or hacking the authentication process, may gain the attacker access to the service, but will not produce the key used by the local client to decrypt information contained within the user’s account. medical library findings on information security secure communication one of the first areas we investigated was secure communication. we looked at various security measures we could apply to chat and email, but in the end did not pursue them for a variety of reasons. the following is a brief summary of our investigations. otr chat chat can be encrypted using clients supporting the “off the record” (otr) protocol. programs such as adium for mac (v1.5.10) and pidgin for windows (v2.10.11) and linux, support otr, either natively or via plugin. otr chat works using aes (128 bit) and dh (1536 bit) encryption. each participant must generate a private key and a public key, called a fingerprint. otr-compatible chat clients have options for generating these keys and handling the exchange through an authentication process. once fingerprints are exchanged, participants can enter into a “private” (encrypted) chat session. instructions for setting up otr are available online (hoffman 2014). although the oakland university (ou) library does use chat for reference and some internal communication, students and faculty in the medical school do not. for those who do wish to use chat to communicate, it is worth considering whether chat would be limited to internal communication or would be used with the public. internal communication would likely occur within an institution’s intranet and would likely be protected by its firewall. also, policies could be put in place to omit sensitive information within chat. on the other hand, any reference chat with a patron could be considered, by its very nature, to be sensitive. furthermore, libraries implementing reference chat would probably want to make it available to patrons outside the institution. in these situations, otr-protected chat could be offered as an option. however, it would be up to the patron to install an otr-compatible chat client, participate in the exchange of keys, and enter a private chat session. currently, this is a fairly cumbersome process, but it is at least an option that libraries can offer. the medical library did attempt to use otr chat to communicate internally with members of the university library, but found that it was not a good fit, and since our patrons do not use chat either, we elected not to explore it further. pgp-encrypted email along with walk-ins and phone calls, the medical library receives most of its interactions with patrons through email, either directly or to a reference email account. in our evaluation of patron privacy, we identified email as one of the primary ways in which personally identifiable patron information might be exposed. we investigated using open pretty good privacy (openpgp) encryption to encrypt email as a means to protect this information. openpgp is not typically built into email clients, though several email clients are compatible with openpgp. in windows, the mozilla thunderbird client (v38.0.1, www.mozilla.org/en-us/thunderbird/) can be set up to use openpgp through an extension. mac mail, built into the operating system, can be configured for openpgp by installing gnu privacy guard (gpg) suite (v2015.06, gpgtools.org). browser plugins such as mailvelope (v0.13.0, www.mailvelope.com) can be used on chrome or firefox to provide pgp support for web-based email. although documentation is generally available from the creators of these products, the complexity of using openpgp is sufficient to merit additional instruction. lifehacker has instructions for setting pgp up with thunderbird (trapani 2006), and a walkthrough for setting up openpgp on a mac is available as well (gangi 2014). the openpgp standard supports several different kinds of symmetric and asymmetric keys, and these may vary depending on the program used. similar to otr-protected chat, using openpgp requires both parties wishing to encrypt an email exchange to install openpgp-compatible software and then create and exchange keys. once that is done, someone writing an email could encrypt the content of a message for a specific recipient. this process is cumbersome to set up and use. furthermore, a major drawback of openpgp-encrypted email is that it only is able to encrypt content within the body of the email, not the email header. unfortunately, the header contains precisely the kind of information (name, email address, subject) a library would wish to protect. this severely limits the protection openpgp can provide. given these limitations and how cumbersome openpgp is to set up and use, we do not expect our patrons to embrace and use it. the medical library may look into it further, but for now we do not expect to implement openpgp-encrypted email as a security measure. since our evaluation of openpgp, we have learned that google uses tls to secure email on its internal servers (lidzborski 2014), though there is no guarantee that the email will remain encrypted when it leaves the gmail servers. outlook also has adopted tls and other security measures (sawers 2014), and we are hopeful that this is a trend that will be adopted more broadly. that leaves the question of what to do about email. for now, the medical library is developing a set of practices to reduce the amount of patron email left lying around in our email accounts. first, we need to be able to identify email containing patron information. the university uses enterprise-level google mail, which allows you to apply labels to emails. by creating a patron label and applying it to emails from patrons, we should be able to flag these emails. the next steps are to develop guidelines for retention and disposal of email, and decide how best to inform patrons as to good information security practices in their interactions with the library. we will review the institutional policy on data retention and determine whether it is sufficient. we do not want to delete an email thread if we think that the patron might come back later with a follow up question. our students conduct a capstone research project that spans all four years of medical school, and they often need to revisit their literature searches from year to year. faculty ask questions related to course development, and we may want to refer to research or recommendations in a previous year. although we do not have a solution at this time, one thing we might do is apply labels to email threads that we believe to be complete and periodically review these to see which ones we feel comfortable deleting. another good security practice the medical library is trying to employ is to limit the amount of sensitive information contained either within the body of emails or as attachments. instead, this information could be placed in a shared secure location and a link provided. anyone intercepting the link would have to authenticate on the shared secure location in order to access the information. shared secure information storage the next facet of information security that the medical library investigated is secure information storage. while several options are available for encrypting information on a local computer, encryption of shared storage is more complicated. to share files, the medical library uses shared folders on the university intranet and cloud storage services such as google drive (drive.google.com) and dropbox (dropbox.com). the university maintains an internal encrypted storage service using xythos software, which has since been purchased by blackboard (www.blackboard.com/platforms/learn/products/blackboard-learn/blackboard-xythos.aspx). although the university recommends use of this service for secure online storage, the interface is clunky and storage is limited by default to 200mb. the university has an enterprise subscription to google email, which includes google drive, which recently increased its quota from 30gb to unlimited storage, and we make heavy use of the collaborative features inherent in google documents, forms, etc. although the university discourages the use of dropbox, some medical library staff use it for collaborative projects managed by others outside of our institution. the university encrypts its centrally-managed shared file servers, which are accessed through network shares. remote access to these folders requires a virtual private network (vpn) connection. when working with groups outside of the institution, it is sometimes necessary to use cloud storage services such as google drive and dropbox, since individuals not affiliated with ou do not have access to the university intranet. although these services may encrypt data, that encryption is performed by the vendor and the keys are stored on their servers. given that users have no knowledge or control over the encryption process, there is no way to be confident that data stored on these services is truly secure. the medical library investigated a variety of zero knowledge encryption options for encrypting cloud data, including software that encrypts data locally for existing cloud storage, as well as stand-alone cloud storage services that provide a way to encrypt data before syncing it to the server. it should be noted that data stored in the cloud, even encrypted data, may not satisfy security requirements of your institutional review board (irb) for personally identifiable data in research. furthermore, it may not meet requirements for protecting patron, patient or student data. there are still situations, however, when you may wish to secure information that you share with colleagues, and the following solutions should suffice for this purpose. the medical library investigated both encrypted cloud storage services and encryption software to use on windows and mac with either dropbox or google drive. for testing, we used two computers, a 2011 imac running os x 10.10.3 yosemite and a 2013 macbook air running windows 7 enterprise (service pack 1). all of the solutions we tested use aes-256 encryption and rsa encryption. in addition to working from our own experience with encryption software, we reviewed online recommendations for secure online storage, including an article from lifehacker (henry 2013), and selected the following solutions to evaluate. encryption software viivo by pkware, inc. (v2.6) – viivo.com viivo is commercial software with a free version whose terms of service do not restrict how it is used (personal vs. business). additional features and support are available with paid subscription versions. viivo uses aes-256 and rsa (2048 bit) encryption and works by syncing files between a decrypted folder and an encrypted folder. specifying a cloud service will place the encrypted folder within the local files for a cloud service such as google drive or dropbox. in testing, we found viivo to be extremely clunky, and the software does not seem to be fully developed. we had great difficulty getting the mac version to connect to google drive and were not successful in getting multiple users to access and decrypt content on google drive (although it is unclear whether the issue was with google drive or viivo, we did not have difficulty connecting to google drive using other software). we had better luck with dropbox, and were able to set up an encrypted share between our mac and windows computer, but the process for setting up the share was too cumbersome to expect others to tolerate. cloudfogger by cloudfogger gmbh (v1.4 for windows, v1.0 for mac) – cloudfogger.com cloudfogger is free commercial software, primarily for windows. cloudfogger uses aes-256 and rsa (unspecified) encryption. the windows version works by monitoring a folder on your computer and automatically encrypting every file and subfolder within. so long as cloudfogger is running in the background, it will automatically decrypt a file when you access it. by monitoring a folder within the local files for a cloud service, those files are encrypted and only accessible while cloudfogger is running. if you share a folder with someone through your cloud service, you can add them to a share list in cloudfogger, and when they log into their cloudfogger account, they will be granted access to the folder. unfortunately, the mac version has none of these features and only allows you to manually decrypt and encrypt files. boxcryptor by secomba gmbh (v2.1) – boxcryptor.com boxcryptor is free for personal use on a single device with a single cloud storage provider. paid subscriptions are available for personal use on unlimited devices with unlimited cloud storage providers and for unlimited business use. boxcryptor uses aes-256 and rsa (4096 bit) encryption, and works by mounting a virtual drive on your computer (through a drive letter in windows and through a mounted volume on mac) that provides decrypted access to an encrypted folder. files added to the virtual drive are automatically encrypted and can only be viewed by accessing them from the encrypted drive. in boxcryptor’s settings, you select a cloud service (the paid version lets you select more than one), and the local folder of that service is shown in the virtual drive. any file or folder within can be encrypted. unfortunately sharing is only available with the paid version, which is prohibitively expensive for our purposes. encrypted cloud services spideroak by spideroak, inc. (v5.2) – spideroak.com spideroak provides 2gb of online storage for free, with additional storage and features available at various levels of paid subscriptions. terms of service indicate no restrictions on type of use (personal vs. business) for the free version. client software creates a local unencrypted folder on your computer. files placed within the folder are synced up to the cloud service after being encrypted locally. spideroak uses aes-256 and rsa (3072 bit) encryption. unfortunately, in testing, we discovered that the only sharing available in the free version is to provide read-only access to others through a url, which would not be sufficient for collaboration. tresorit by tresorit ag (v2.0) – tresorit.com the free tresorit account provides 3gb of storage and limits some features, with additional features and storage available for a paid subscription. terms of service indicate no restrictions on the type of use (personal vs. business) for the free account. tresorit uses aes-256. the tresorit client lets you access and manage containers called tresors (german for “vault”), each of which may be configured to sync to a folder on your local computer. if you do not wish to keep a local decrypted copy of the files on your computer, you can turn sync off and access the files through the client. each tresor can be shared with other tresorit users. the free account lets you have three tresors, each of which can be shared by up to 10 users. the tresorit client has a clean and simple interface, is easy to set up on both mac and windows, can be configured to use multifactor authentication (explained further in the section on password management) for extra security, and can be configured to automatically run and log in on startup, allowing you to interact with files in a synced tresor folder without touching the client. based on our investigations, we found tresorit to be the most promising free option for setting up an encrypted share for a small group. sharing with larger groups would only require the owner of the share to pay for a subscription, allowing other users to access the share for free. password management the medical library was also interested in using a shared password management system to allow the library and smaller groups therein to manage passwords for services, such as the reference email account, and other resources, such as web servers and database users. as it happened, the university library had already investigated password managers for its computer support department, including keepass, dashlane and lastpass. after some discussion, it was decided to share a group purchase of a password manager. keepass professional edition (v2.29) – keepass.info keepass professional edition is free, open source software that runs under windows. it supports aes and twofish symmetric key encryption. keepass works by creating an encrypted database which can be unlocked using a master password and/or a key file. records containing information such as the name of the program or website, login, password, website url, etc. are stored in the database. keepass records must be created manually, but keepass does have features to autofill logins and passwords if you initiate connection to a web site from the program (which opens the site in your default browser) and there are third party plugins to provide additional browser integration. a keepass database can be shared by storing the password database in a shared location. multi-user support is provided through plugins for the program. since keepass is locally installed software, no user registration is required, but multiple users must share a single password to access the database. keepass uses the .net environment, which is built into windows (vista and higher). it does run on mac computers, using mono (v2.6 and higher), an open source .net framework. however, according to the keepass website, the official site supporting the mac version of keepass is no longer available. an open source project, keepassx (alpha v0.4.3), based on keepass 1.x, is available for the mac. the developer describes it as a hobby project, and it hasn’t been updated in four years, but the underlying code is relatively stable. dashlane by dashlane, inc. (v3.5 for mac, 3.2.5 for windows) – dashlane.com dashlane is a password management service that provides a free account for personal or internal business use. it runs on windows, mac, ios and android. dashlane is free when used on a single computer or device. for a subscription, dashlane premium provides syncing between computers/devices, unlimited sharing of passwords, and other features. dashlane integrates well with a variety of browsers. it uses aes (256 bit) to encrypt its database, and even if you subscribe to the premium version, you can choose to restrict your database to your local computer (instead of uploading the locally encrypted database to its servers). dashlane uses an auto-create feature to add records to its database when you log into a website, as well as an auto-fill feature to enter login and password data into a website from an existing record. individual password records may be shared with other users. dashlane requires users to register in order to create an account, but authentication is handled locally (no internet connection is required to open the password database). it supports multifactor authentication. lastpass by lastpass (v3.1.95) – lastpass.com similar to dashlane, lastpass is a password management service that runs out of your browser through a toolbar icon. the free account allows you to install lastpass on multiple computers and provides web access to your password database. different subscription levels provide additional features such as use of mobile apps (premium subscription) and sharing features (enterprise subscription). it uses aes (256 bit) to locally encrypt the password database before uploading it to its servers. lastpass does not provide a local-only option for its database. it provides similar auto-create and auto-fill features as dashlane. it also supports multifactor authentication. lastpass enterprise provides a robust set of features. individual records can be shared, or shared folders can be created to organize records and automatically provide access to users with whom a folder is shared. lastpass users who have a personal account as well as an enterprise account can link the accounts so that when they log into their enterprise account, they will be able to access their personal database as well. lastpass can audit your password database and alert you to bad security practices (weak passwords, using the same password for multiple accounts, etc.). all of the above password managers were given consideration. at least one person on staff already used each one. given the limited sharing capability of keepass, as well as the uncertainty of mac development, keepass would not work well in the medical library’s mac-heavy environment. keepassx was excluded from consideration due to its lack of development. dashlane was more promising, but lastpass was ultimately chosen because of its more robust sharing capabilities and the proactive way in which lastpass handled the heartbleed vulnerability (kovach 2014). conclusion after evaluating the above security measures, we have had success using lastpass as a shared password manager, but have not found ways to fit other measures into our workflow. we ended up not using chat at all and were disappointed in the email encryption options available because they are cumbersome and do not protect the identities of the sender and recipients. we are encouraged that email encryption is starting to take hold among major providers, but it is unclear how quickly this will be more broadly adopted or how transport security might apply to stored emails. for file storage, we continue to primarily use institutional file servers to store internal, sensitive information. we are hoping to pilot use of tresorit with an external group if we are able to find willing participants. still, we are pleased with the results thus far. about half of the staff in the medical library have begun to use lastpass on a daily basis. it integrates very well with the browser, so much so that one of our staff didn’t even realize they were using it. we have two shared password folders, one for the medical library, and one shared with the university library’s technology team. the technology team has made the heaviest use of the shared folders, mainly storing server passwords. the medical library has begun to populate its shared folder with passwords for shared services (such as our shared reference email account). access to passwords and folders can be configured to specific individuals, so only library staff that are authorized to have access to library resources are able to access the usernames and passwords for these resources. this, in itself, is a great benefit to managing sensitive library systems. previously, passwords would need to be exchanged verbally or written down, each of which has some degree of risk. at the time of the writing of this article, the medical library received a security alert from lastpass (siegrist 2015), notifying us that an attack was detected on their system and user password hash files and salts were compromised. the service immediately implemented a security measure, requiring any logins from non-trusted (new) computers to undergo an email verification step, unless the account already had other multi-factor authentication in place. the alert reiterated the purpose of strong passwords in the protection against such an attack, and advised users to change their account password in any case. our site administrator was able to require us to change our passwords, even though some of our passwords were strong enough that they would be at a very low risk from brute force guessing. although we were concerned about the breach, we felt that the response was timely and sufficient to protect our data. the steps we have taken thus far are small ones, and do not have a significant impact on the protection of patron information itself, for which the library continues to rely primarily on security built into library systems such as the integrated library system and interlibrary loan system. still, they move us towards building a more comprehensive approach to information security. one of the next steps recommended by the libraries’ patron privacy policy is to develop security procedures for the identification, management, and retirement of patron information, as well as a means to monitor how well the library is implementing these procedures. while use of additional security measures is outside the scope of the patron privacy policy, the medical library will look into how we might support its use through training, creation of guides, etc. one thing has become clear through this process. it is very easy to take digital information for granted and fail to give due consideration to its vulnerability. the steps we have taken thus far are modest, at best, but are also important in that they encourage us to take a proactive approach to thinking about information security, and, we hope, prompt us to take preemptive action. the first step is the most critical in protecting the sensitive information of our patrons, our institution, and ourselves. references american library association (ala) 1939. code of ethics for librarians. american library association [internet]. [cited 2015 may 12]. available from: http://www.ala.org/advocacy/proethics/history/index5 durumeric z, et. al. the matter of heartbleed. proceedings of the 2014 conference on internet measurement. 2014 [cited 2015 jun 15]: 475-488. available from: http://dl.acm.org/citation.cfm?id=2663755 gangi j. the best pgp tutorial for mac os x, ever. jerzy’s notes [internet]. [updated 2014 mar 12]. [cited 2015 jun 15]. available from: http://notes.jerzygangi.com/the-best-pgp-tutorial-for-mac-os-x-ever/ henry a. the best cloud storage services that protect your privacy. lifehacker [internet]. [updated 2013 jul 10]. [cited 2015 jun 15]. available from: http://lifehacker.com/the-best-cloud-storage-services-that-protect-your-priva-729639300 hess an, laporte-fiori r, engwall k. 2014. preserving patron privacy in the 21st century academic library. the journal of academic librarianship [internet]. [cited 2015 may 12]; 41(1). available from: http://hdl.handle.net/10323/3345 hoffman c. 2014. how (and why) to use otr for private instant messaging. howtogeek [internet]. [cited 2015 may 12]. available from: http://www.howtogeek.com/190811/how-and-why-to-use-otr-for-private-instant-messaging/ kovach s. find out instantly if a site has been infected by ‘heartbleed’. business insider [internet]. [upated 2014 apr 9]. [cited 2015 may 12]. available from: http://www.businessinsider.com/lastpass-heartbleed-checker-2014-4 lidzborski n. staying at the forefront of email security and reliability: https-only and 99.978% availability. official gmail blog [internet]. [updated 2014 mar 20]. [cited 2015 jun 15]. available from: http://gmailblog.blogspot.com/2014/03/staying-at-forefront-of-email-security.html public-key certificate. wikipedia [internet]. [updated 2015 may 16]. [cited 2015 jun 15]. available from: https://en.wikipedia.org/wiki/public_key_certificate public-key cryptography. wikipedia [internet]. [updated 2015 jun 4]. [cited 2015 jun 15]. available from: https://en.wikipedia.org/wiki/public-key_cryptography salt (cryptography). wikipedia [internet]. [updated 2015 apr 29]. [cited 2015 jun 15]. available from: https://en.wikipedia.org/wiki/salt_%28cryptography%29 sawers p. microsoft boosts outlook.com email encryption and opens its first transparency center. thenextweb news [internet]. [updated 2014 jul 1]. [cited 2015 jun 15]. available from: http://thenextweb.com/microsoft/2014/07/01/microsoft-brings-tls-encryption-outlook-com-opens-first-transparency-center/ siegrist j. lastpass security notice. lastpass blog [internet]. [updated 2015 jun 15]. [cited 2015 jun 17]. available from: https://blog.lastpass.com/2015/06/lastpass-security-notice.html/ symmetric-key algorithm. wikipedia [internet]. [updated 2015 jun 15]. [cited 2015 jun 15]. available from: https://en.wikipedia.org/wiki/symmetric-key_algorithm transport layer security. wikipedia [internet]. [updated 2015 jun 15]. [cited 2015 jun 15]. available from: https://en.wikipedia.org/wiki/transport_layer_security trapani g. how to encrypt your email. lifehacker [internet]. [updated 2006 jun 15]. [cited 2015 may 12]. available from: http://lifehacker.com/180878/how-to-encrypt-your-email about the author keith engwall, ms lis, ahip is an assistant professor at the oakland university william beaumont school of medicine, in rochester, mi. he serves as the web and emerging technologies librarian at the ouwb medical library. he can be reached at engwall@oakland.edu subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – recipes for enhancing digital collections with linked data mission editorial committee process and structure code4lib issue 23, 2014-01-17 recipes for enhancing digital collections with linked data standards-based metadata in digital library collections are commonly less than standard. limitations brought on by routine cataloging errors, sporadic use of authority and controlled vocabularies, and systems that cannot effectively handle text encoding lead to pervasive quality issues. this paper describes the use of linked data for enhancement and quality control of existing digital collections metadata. we provide practical recipes for transforming uncontrolled text values into semantically rich data, performing automated cleanup on hand-entered fields, and discovering new information from links between legacy metadata and external datasets. by thomas johnson and karen estlund introduction quality metadata is a critical part of digital library and archive collections. metadata drives both curatorial and end-user interactions with objects, playing a key role in discovery, preservation, and everything in between. within libraries, the importance of quality description to the utility of collections enjoys broad recognition. despite this, limited resources mean that, at best, only modest expert attention is given to ensuring quality as records are created and maintained. metadata is often created with inadequate training, sparse documentation, and within systems that mishandle information. quality problems are, as could be expected, pervasive. once created, records are rarely subject to systematic review or evaluation, which generally requires human resources and expertise on a scale that makes it impractical for most organizations. records are left to languish. the aim of this work is to provide practical recipes for quality control and incremental enhancement of existing digital collections metadata. the recipes rely on the rdf data model and linked data practices as both an underlying framework and a toolset for metadata enhancement processes. if libraries do not have systems supporting rdf or linked data, the recipes may still be used to clean up and normalize metadata outside the system for re-ingestion into the existing systems. by relying on a statement-centric model, using uris for controlled vocabularies, and welcoming external sources of information, libraries can improve both data quality and collection utility. the role of linked data linked data brings with it a shift from a monolithic metadata model—in which an object has a single authoritative and mostly static record—to a distributed approach centered around individual statements from diverse sources. an early expression of this shift in ethos is “anyone can say anything about anything” (berners-lee, 1997). at first blush, this might not inspire confidence that improvements in quality will result from this philosophy. the opportunity, however, lies in the possibility of sourcing quality externally and reusing information with greater accuracy, reliability, and depth than could be achieved with local resource limitations and institutional priorities. hillmann, dushay, and phipps (2004) have previously demonstrated the value of a statement-centric model as it applies to enhancement outside of the linked data context. in discussing augmentation processes intended for use on federated collections: [i]t allows us to expose the source of each statement, and to reassemble these statements about resources in a variety of ways and for a variety of purposes, rather than expose a mélange of records and expect the downstream users to perform complex dissociations and recombinations. by abandoning a record-centric model, metadata providers can begin to seek information from sources other than the human effort of staff. the reuse of outside metadata allows new information to be introduced while limiting the need for internal management. understanding quality understanding metadata quality as something that can be managed and enhanced presents a significant conceptual challenge. various authors have attempted definitions of quality, taxonomies of errors, or frameworks for assessment and analysis (moen et al. 1997; stvilia et al. 2004; bruce & hillman, 2004; margaritopoulos et al. 2008). broadly, these discussions are focused on assessing information quality in terms of detectable flaws and indicators such as “completeness, accuracy, provenance, conformance to expectations, logical consistency and coherence, timeliness, and accessibility” (bruce and hillman 2004). in lieu of further forays into this territory, we approach quality roughly and pragmatically. the quality of a statement, when viewed context-independently, is simply the amount and accuracy of information that can be reliably extracted from it. this falls short as a formal definition, but provides a realistic guideline for performing actual enhancement work [1]. contextual and domain-specific quality issues introduce an additional factor in “utility”—as explained by sutton (2008)—which is harder to intuit and important to any effort at cost-benefit analysis. still, for the purposes of the general recipes presented here, quality enhancement can be understood as a matter of introducing statements consisting of new information and removing obfuscating factors (errors and ambiguity) from existing data. beyond this, we want to call attention specifically to types and sources of errors addressed by the recipes outlined in this article: data entry errors in the form of typos, data entered into wrong fields, and statements that, for various reasons, contain no information at all. system limitations resulting in unintelligible or badly encoded data. this occurs most often with unicode translation errors. for example, “akh tã¶bei” should be “akh töbei.”: lack of control leading to inconsistent and hard to interpret data values. this also contributes to data entry errors and other accuracy issues. shifting strategies between different collections, federated collections, and within individual collections over time. this problem manifests as differing use of metadata elements, inconsistent application of controlled vocabularies, and changes in data entry practice (e.g. delimited vs. repeatable fields). these issues are usually accompanied by poor documentation, making interpretation a matter of local expertise and institutional memory. notation throughout the remainder of this article, we use turtle [2] syntax to represent rdf metadata. in our examples, curation objects are designated by the qualified name ‘:object’ (or ‘:object1’, ‘:object2’, etc… where differentiation is called for). so a set of statements about an object appears in the form: :object <metadataterm> "value" ; <metadataterm2> <valueuri> . additionally, we make use of a number of prefixes to map uris to shorter strings (curies [3]). for example, dct:type maps to http://purl.org/dc/terms/type. the prefixes used are: @prefix dce: <http://purl.org/dc/element/1.1/> @prefix dct: <http://purl.org/dc/terms/> @prefix dctype: <http://purl.org/dc/dcmitype/> @prefix foaf: <http://xmlns.com/foaf/0.1/> @prefix mads: <http://www.loc.gov/mads/rdf/v1#> @prefix cc: <http://creativecommons.org/ns#> @prefix prov: <http://www.w3.org/ns/prov#> examples of “legacy” source records are given in a simplified xml-like format. the structure typifies record-based, key-value metadata common to systems like contentdm. we avoid specific schemas and element sets, preferring a general demonstration that can be broadly applied. <record> <metadataterm>value</metadataterm> <metadataterm2>another value</metadataterm2> </record> strategies for mass enhancement approaches to performing enhancement en masse depend somewhat on the nature of the collections in question and the quality of metadata at the start. perhaps with undue pessimism, we assume a fairly messy state of affairs. different collections likely use elements or value vocabularies differently, elements are probably made up ad-hoc, and ‘note’ or ‘description’ fields are almost certainly used arbitrarily, holding any data that didn’t have an obvious home at the time. because of these factors we recommend handling enhancement on a collection-by-collection basis. applying generalized, but configurable, processes to individual collections helps to separate legacy records into groups that can be handled in roughly the same way. however, because practices are often poorly documented and inconsistent over time even within a collection, we also recommend a two-step approach to a linked data migration. the initial step converts legacy metadata into rdf, performing cleanup operations that are ‘safe’ and applicable across collections. this process should do the following: remove noise normalize presentation assign uris for curation objects map legacy elements to linked data vocabularies removing “noise” means eliminating valueless metadata entries. in some cases this means, quite literally, removing entries without any content. hillmann, dushay, and phipps (2004) identify valueless statements as those that are empty, listed as “n/a” or “unknown” (or a variation), or consist solely of punctuation [4]. other statements may be “valueless” because they contain meaningless or obfuscating information, e.g. duplicate or untimely entries. <record> <title>object with no creator</title> <creator></creator> </record> <record> <title>object with no creator</title> <creator>unknown</creator> </record> <record> <title>object with duplicate information</title> <label>object with duplicate information</label> </record> <record> <title>object with untimely information</title> <identifier>staleid</identifier> </record> normalizing presentation is the process of abstracting away presentational aspects of metadata values. this includes condensing whitespace, handling encoding issues, individuating delimited values, and stripping non-semantic punctuation. <record> <title>object with whitespace</title> <description> an object with extra whitespace </description> </record> <record> <title>object with encoding issues</title> <description>object depicts khã¶ltsã¶ã¶tiin gol with accented vowels. </description> </record> <record> <title>object with delimited creators</title> <creator>johnson, t.; estlund, k.</creator> </record> assigning uris requires both a namespace and a source of unique ids. though much effort could be given to frbr-style entity relationship modeling, we find that a one-uri per object (“record”) methodology is the best approach where something more sophisticated isn’t necessary for a specific use case. most digital collections already have some sort of unique record id that can be used as a simple source of identifiers. if there are perceived shortcomings with existing local identifiers, an identifier ‘minter’ like noid is a good idea. “safety” of automated edits automated changes to metadata frequently raise concerns about the safety of existing information. hillmann, dushay, and phipps (2004) describe an approach to “safe transforms” which carry “no risk of degradation”. the processes included in the initial step above closely align with those they outline as “safe”. still, even under a fairly loose interpretation of “risk-free” enhancement, the changes that qualify must be severely limited in order to prevent incorrect assumptions and overwriting of good data. more substantial enhancements, whether carried out by a machine or human, will invariably have some risk of data loss. updated statements that are not strictly equivalent to their former expressions could always be a source of information degradation. the approach we suggest is to understand and manage this risk. a potential loss of poor quality information should not be allowed to prevent a well-understood, systematic gain elsewhere. managing provenance one strategy for managing the non-safety of automated updates is to carefully track provenance of data. current linked data best practice recommends doing this through the use of “named graphs” (carroll et al. 2005). named graphs allow metadata statements to be grouped into sets which themselves can be the subject of statements expressing, for example, the source, licensing, or versioning of included statements (dodds 2009; shinavier 2010). we suggest using named graphs to track the source of each statement. the number and scope of graphs used can vary widely depending on local needs. a practical scheme might employ a graph for each external data source and one containing each of the following: the initial rdf expression of the original, unenhanced data. statements added or altered by human workflows. statements generated by automated enhancement processes. using turtle (trig) for named graphs, four named graphs (g1-4) describing a cleanup activity may look like: g1 (:object1 dct:title "example" ; dct:description "object depicts khã¶ltsã¶ã¶tiin gol with accented vowels." .) g2 (:g1 prov:derivedfrom :oregondigitalcontentdmexport . dct:date "2013-01-10" ; :g2 dct:date "2013-01-10" . :oregondigitalcontentdmexport prov:wasassociatedwith :oru .) g3 (:object1 dct:description "object depicts khötsöötiin gol with accented vowels." .) g4 (:g3 prov:derivedfrom :encodingcleanupscript1 ; dct:date "2013-11-15" . :g4 dct:date "2013-11-15" . :encodingcleanupscript1 prov:wasassociatedwith :tjohnson .) implementing recipes and digital asset management systems (dams) under these strategies and assumptions, the following recipes support the goal of incremental improvement of metadata and provide practicable approaches to enhancement. these recipes are designed as intermediary cleanup steps where the metadata is extracted and re-inserted in an existing system. linked data is used as a cleanup mechanism; however, the ideal solution would publish the resultant metadata as linked data, as well. recipe: learning control where controlled vocabularies are used in existing metadata, they are often undocumented and represented as hand-entered text values. accordingly, they are subject to a variety of errors. worse, they may be so poorly used that even identifying the presence of a controlled vocabulary may require specialist or localized knowledge. replacing these values with uris from a managed vocabulary offers an immediate quality improvement, turning the use of controlled terms into a self-documenting practice. the dcmi type vocabulary provides a simple example. our goal is to find legacy records using elements from the vocabulary, and replace them with rdf statements using the appropriate uris. given the legacy record set: <record> <type>image</type> </record> <record> <type>text</type> </record> <record> <type>physicalobject</type> </record> we should be able to derive the statements: :object1 dct:type dctype:image . :object2 dct:type dctype:text . :object3 dct:type dctype:physicalobject . in simple cases, this “identification” can be done with any tool capable of reasonably context aware find/replace operations. all we need is a hard-coded set of mappings between expected values and uris. since dcmi types contain only 12 terms, this is a small amount of work to power a bulk enhancement operation. we can prevent false positives by limiting the search to fields that might relate to the vocabulary. in doing similar work with dcmi types, hillmann, dushay, and phipps (2004) observed that identification scales poorly to larger and more complex vocabularies. though fairly manageable when working with a limited set of terms, hard-coded mappings become extremely work intensive as the complexity of the vocabulary grows. these scalability limitations are substantially mitigated, however, if potential vocabularies are restricted to those published as linked data. using rdf documents as a machine-actionable expression of the vocabulary, identification and enhancement becomes an automated, configurable process. to automatically convert plain text entries to their appropriate uris we need two things: an rdf graph containing descriptions of each term in the vocabulary [5]. knowledge of which properties are labels that can be matched against text values. we use the following set of labels, listed in order of priority, as defaults: skos:preflabel mads:authoritativelabel dct:title rdfs:label foaf:name skos:altlabel mads:variantlabel as an example, reconsider dcmi types. the full set of types is described in rdf/xml at http://purl.org/dc/dcmitype/ and each term has an rdfs:label. given the limited configuration below, we can automatically identify matches and replace them with appropriate uris. uri: http://purl.org/dc/dcmitype source: dce:type target: dct:type replace: true in addition to a uri for the vocabulary graph, the configuration consists of only a handful of optional fields. we limit the transformation to a set of source fields appropriate to the vocabulary. we may set a target predicate to use in generated triples if semantics are improved by the change. we can specify label uris. and, finally, we set a replace flag to determine whether old triples should be deleted as derivatives are added. figure 1: transforming dcmi type values into uris. unmatched values are ignored. following the named graph convention, we may choose to segregate and describe the resulting triples. the process now takes a defined set of steps which can be applied independently of vocabulary size. dereference the graph uri. parse the resulting graph [6]. generate a list of uris and their labels. search for legacy values across desired fields. map elements and values, adding statements to a named graph. recipe: drawing connections discovering new links perhaps the most powerful aspect of this configurable interaction with external vocabularies is that it need not be limited to working with vocabularies already used in source records. a minor addition to the process allows more complex transformation logic to be applied on a per-vocabulary basis. this can be used to implement linked data vocabularies in place of ad hoc terms where a relationship can be identified between existing data and rdf labels. the transformation logic replaces the normal text-to-label matching with arbitrary code specified in the vocabulary configuration (we implement this as a dynamic method call in ruby). generally, this code runs some combination of preprocessing source values, identifying full or partial text matches, and adding new statements. we have used this approach to transform text entries referring to creative commons licenses into links to appropriate uris. :object1 dce:rights "cc by-nc-nd 3.0" . :object2 dce:rights "cc by 3.0" . :object3 dce:rights "public domain" . these rights statements give us enough to go on to identify uris for the licenses. the logic, in this case, is a simple process breaking out the aspects of the license (‘by’, ‘nc’, ‘3.0’, etc…), recombining them, and mapping them to the license in the creative commons ‘controlled vocabulary’. it generates a single statement defining the relationship between the digital object and the license uri, then dereferences the latter, adding appropriate statements further describing the license. :object1 dct:license <http://creativecommons.org/licenses/by-nc-nd/3.0> . <http://creativecommons.org/licenses/by-nc-nd/3.0/> cc:permits cc:reproduction; cc:requires cc:attribution; cc:prohibits cc:commercialuse; dct:title . "attribution-noncommercial-noderivs 3.0 unported" . :object2 dct:license <http://creativecommons.org/licenses/by/3.0> . <http://creativecommons.org/licenses/by/3.0/> cc:permits cc:reproduction; cc:requires cc:attribution; cc:permits cc:derivativeworks; dct:title "attribution 3.0 unported" . :object3 dct:license <http://creativecommons.org/publicdomain/mark/1.0> . <http://creativecommons.org/publicdomain/mark/1.0> cc:permits cc:reproduction; cc:permits cc:derivativeworks; cc:permits cc:distribution; dct:title "public domain mark 1.0"@en . figure 2: adding creative commons licenses as linked data. the “union graph” contains the statements in the source data, as well as those generated by the configuration and logic. the pattern can be applied to more expansive vocabularies or cases where matches between existing values and uris are less straightforward. using geonames, for example, we have been able to connect uncontrolled place names across our collections and, in the same stroke, access an external and authoritative source of geographic coordinates. it must be said that the human effort needed in this process is more substantial, both in terms of metadata specialist and programmer time. the complexity of the ‘logic’ step and the need for care in provenance management grows as enhancements get more sophisticated. quality gains must be weighed pragmatically against their associated costs. the effort required is justified only when a clear use case for the new data has been identified. recipe: tidying up identifying controlled vocabularies can only provide a partial solution to quality issues arising from their undocumented or inconsistent use. unfortunately, identification works best where the data quality already meets a minimum standard. poor data quality in the form of typos, encoding issues, and inconsistent use over time presents a barrier to identifying a uri with a clear relationship to the existing text value. cleaning up, then, must happen as a gradual, incremental process. identifying exceptions searching for and marking exceptions to the use of controlled vocabularies can be a powerful tool for identifying poor quality data. generating a list of terms used outside the vocabulary allows for quick analysis. returning to dcmi types as an example, consider the following records: <record> <type>image</type> </record> ... <record> <type>img</type> </record> <record> <type>text</type> </record> ... <record> <type>txt</type> </record> <record> <type>special object</type> </record> <record> <type>other</type> </record> if we have gone through the previous recipes, the result is a set of statements with two distinct predicates: the source statements (dce:type) and the enhanced ones (dct:type). :object1 dct:type dctype:image . :object2 dce:type "img" . :object3 dct:type dctype:text . :object4 dce:type "txt" . :object5 dce:type "special object" . :object6 dce:type "other" . values from outside the controlled vocabulary can be identified trivially by querying for statements with the original predicate. outputting a list of these values lets us review bad data at a glance and map values for normalization. in the example, mapping “img” to dctype:image and “txt” to dctype:text, yields a more consistent output. data that continues to evade normalization can be left alone or identified as ‘valueless’ and deleted: :object1 dct:type dctype:image . :object2 dct:type dctype:image . :object3 dct:type dctype:text . :object4 dct:type dctype:text . :object5 dce:type "specialobject" . figure 3: identifying exceptional terms conclusion we view these recipes as providing a starting place for the continual improvement of metadata statements. using the statement-centric approach, the willingness to link to outside information sources, and the named graph provenance strategy, metadata records are no longer held static by their pretensions to “authority”. instead, they become mutable and flexible, subject to a quality feedback loop through reuse and sharing. footnotes [1] bruce and hillmann (2004) compare quality to pornography: “we know it when we see it”. [2] terse rdf triple language. a primer for rdf in turtle can be found at: http://www.w3.org/2007/02/turtle/primer/ [3] http://www.w3.org/tr/curie/ [4] generally, this is a simple matter of ignoring the field as the record is processed. an exception may arise when a value of “unknown” is intended to explicitly state that a property’s value is unknown universally, not only locally, which occurs often within our artwork collections. the distinction arises from the open world assumption, which is core to linked data. under this assumption, we may read empty values as unknown locally, but potentially known externally. [5] best practice for publishing linked data dictates that resource uris return rdf serializations via http content negotiation (bizer et al. 2007). many vocabularies return a full graph of their terms from a single uri as in the example. larger vocabularies are often published as compressed “dumps” which can be downloaded, decompressed, and used locally. [6] we use raptor (http://librdf.org/raptor/) to parse the graph. in ruby, the rdf-raptor gem will do the work of steps 1 and 2: require ‘rdf’ require ‘rdf/raptor’ graph = rdf::graph.load(“http://purl.org/dc/dcmitype”) references berners-lee, t. 1997. web architecture: metadata [internet]. [cited 2014 january 2]. available from: http://www.w3.org/designissues/metadata.html bizer, c., cyganiak, r., heath, t. 2007. how to publish linked data on the web [internet]. [cited 2014 january 2]. available from: http://wifo5-03.informatik.uni-mannheim.de/bizer/pub/linkeddatatutorial bruce, t.r, and hillmann, d.i. 2004. the continuum of metadata quality: defining, expressing, exploiting. in: d. hillmann & e. westbrooks, editors. metadata in practice. ala editions. available from: http://hdl.handle.net/1813/7895 carroll, j.j., bizer, c., hayes, p., stickler, p. named graphs, provenance and trust. in: www ’05 proceedings of the 14th international conference on the world wide web. new york, ny: acm. p. 613-622. available from: doi:10.1145/1060745.1060835 dodds, l. 2009. managing rdf using named graphs [internet]. [cited 2014 january 2]. available from: http://blog.ldodds.com/2009/11/05/managing-rdf-using-named-graphs/ hillmann, d.i., dushay, n., phipps, j. 2004. improving metadata quality: augmentation and recombination. in: metadata across languages and cultures. proceedings of the 2004 dcmi international conference on dublin core and metadata applications. available from: http://dcpapers.dublincore.org/pubs/article/view/770 margaritopoulos, t., margaritopoulos m., mavridis i., manitsaris a. 2008. a conceptual framework for metadata quality assessment. in: proceedings of the 2004 dcmi international conference on dublin core and metadata applications. available from: http://dcpapers.dublincore.org/pubs/article/view/923 moen, w.e., stewart, e.l., mcclure, c.r. (1997). assessing metadata quality: findings and methodological considerations from an evaluation of the us government information locator service (gils). in: proceedings of the advances in digital libraries conference, 1998. ieee computer society. p. 246. available from: doi:10.1109/adl.1998.670425 shinavier, j. 2010. position paper: named graphs in linked data [internet]. [cited 2014 january 2]. available from: www.w3.org/2009/12/rdf-ws/papers/ws25.pdf stvilia, b., gasser, l., twidale, m.b., shreeves, s.l., cole, t.w. 2004. metadata quality for federated collections. in: proceedings of iciq04 – 9th international conference on information quality. cambridge, ma. available from: http://hdl.handle.net/2142/721 sutton, s.a. 2008. metadata quality, utility and the semantic web: the case of learning resources and achievement standards. cataloging and classification quarterly. 46(1). available from: doi:10.1080/01639370802183065 about the authors thomas johnson is digital applications librarian at oregon state university libraries and press, where he works on digital curation, scholarly publication, and related metadata and software issues. karen estlund is the head of the digital scholarship center at the university of oregon libraries, where she provides support for faculty and student digital scholarship projects and vision and leadership for the growth of the library’s digital collections and scholarly communication activities. subscribe to comments: for this article | for all articles one response to "recipes for enhancing digital collections with linked data" please leave a response below: eric lease morgan, 2014-01-20 this is very interesting. dirty metadata is most certainly a problem that needs to be dealt with in library land, and i certainly like the idea of making it available as linked data, but alas, i was hoping for some code or demonstrations on how one or more of these recipes were put into practice. for example, how might one go about “preparing one of these meals”? any suggestions? maybe i should just dump my library catalog as marc, serialize it into rdf, process each triple, and reverse the process? –elm leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – metadata in, library out. a simple, robust digital library system mission editorial committee process and structure code4lib issue 10, 2010-06-22 metadata in, library out. a simple, robust digital library system tired of being held hostage to expensive systems that did not meet our needs, the university of alabama libraries developed an xml schema-agnostic, light-weight digital library delivery system based on the principles of “keep it simple, stupid!” metadata and derivatives reside in openly accessible web directories, which support the development of web agents and new usability software, as well as modification and complete retrieval at any time. the file name structure is echoed in the file system structure, enabling the delivery software to make inferences about relationships, sequencing, and complex object structure without having to encapsulate files in complex metadata schemas. the web delivery system, acumen, is built of php, json, javascript and html5, using mysql to support fielded searching. recognizing that spreadsheets are more user-friendly than xml, an accompanying widget, archivists utility, transforms spreadsheets into mods based on rules selected by the user. acumen, archivists utility, and all supporting software scripts will be made available as open source. by tonio loewald and jody deridder, university of alabama at the university of alabama, we have developed a simple, robust digital library system using php, mysql, and apache. the new system is free, easy to set up and administer, imposes no new work flows or skill set requirements on archivists, bypasses the need to create extra metadata wrappers, works in all major browsers, runs on the most common web server stack, and automatically creates and updates its own index database (while storing no valuable data inside it). details can be seen at http://acumen.lib.ua.edu/project. why a new system? experience and frustration with our former digital collections platform had resulted in considerable discussion as to what a perfect, or at least “good”, system might look like. by the time the first author joined the project, a number of key considerations had already been settled on, for example: ingestion needs to be streamlined and lossless (“what goes in comes out”) lightweight web services architecture modular rather than monolithic (small, individually useful components) work with the outside world (e.g. outside search engines), not against it do not place new burdens on archivists (who were already on our critical path) it seemed that what our stakeholders really wanted was something that would automatically scan repositories and then make the content searchable and readable online without requiring them to perform any extra work. no such thing appeared to exist. so we decided to build one. overview figure 1. this is a high level view of the acumen architecture. it’s built on the most widely deployed and inexpensive server stack and makes very few assumptions about the contents of the repository whose content it will index and display. click here for a larger image our solution, which we call acumen, is free, and built on top of other free, open source technologies such as php, mysql, and apache[1]. it comprises three main components: indexer, which locates files and creates index entries for them; search, which provides a user interface for searching the index database and displaying results; and presentation, which turns xml metadata files and associated media assets into something easy to view and navigate. easy to use a free tool can quickly become expensive if it is hard to use. our goal is for acumen to be as easy to install as popular, well-written free web applications (e.g. wordpress or phpbb)[2], to run on not just free but ubiquitous, popular, and well-documented server stacks (php, mysql, apache), and require no extra work or training of anyone, including the people who create and maintain metadata, system administrators, and end-users. no special apache configuration is required (a simple .htaccess file can redirect urls to the php script directory) and the only slightly unusual php module we currently use is php_xsl (which is included in default installations, but may need to be “toggled on” by changing a configuration file). ingestion metadata does not need to be prepared, tweaked, or modified in order to allow acumen to provide users with access to it; it merely needs to be valid xml. just as google (and other web crawlers) index the web without placing any special requirements on the creators of web pages, acumen indexes metadata without placing special requirements on archivists. just as google will send users to a web page that has changed since it was indexed, acumen shows the data in the repository, not the data it scanned when it created or updated its index. acumen indexes raw metadata directly by summarizing it on-the-fly using xsl. the output is a simple xml file largely consisting of some content nodes — the indexing engine parses the output and stores the content of each node as an authority record attached to the metadata file record of the specified type. acumen renders metadata as web pages the same way, and then builds a user interface around the results (inside the end-user’s browser) using javascript and css. all of the original data lives in the repository — acumen does not touch it. workflow acumen makes few assumptions about how a repository is created. a repository is assumed to be a collection of metadata and digital asset files that adhere to a naming convention. how those files are created and managed is outside its purview. as such, acumen has virtually no impact on existing workflows — produce the metadata and digital assets, name them in a specified way, and stick them on a web server where acumen can find them. that’s it. in practice, when we capture digital objects (e.g. by scanning or audio capture), we enter metadata into large spreadsheets. this metadata is then converted into standards-compliant xml. in the past this process was both labor-intensive and prone to errors. furthermore, ingestion into our previous tool had left much of our data in something of an undefined state. as of version 1.0, acumen will only index metadata that is in a format it recognizes, i.e. ead, mods, and mets xml. adding support for a new kind of xml file requires two xsl files (one for summarization and one for presentation) and a row added to the file_type table. archivist utility figure 2. this is the data view of archivist utility. on the left is a collection of spreadsheets (exported as text) ready for batch-processing. on the right is a “database” view of the contents of the selected file. click here for a larger image to streamline the process of getting raw data into xml we developed a desktop application —archivist utility[3]— which processes spreadsheet data into xml using a flexible template system that is rather more sophisticated than, but similar in concept to, mail merge. unlike typical mail merge, our template system supports conditional tags, repeating sections, transformations on incoming data, stores unused data from the spreadsheet in xml comments, strips empty tags from the resulting xml, and “pretty prints” its output (e.g. with nice indentation). a more complete description of archivist utility is beyond the scope of this article. figure 3. this is the archivist utility’s template view. the template “language” allows for conditional and looping constructs; e.g. a spreadsheet field which contains a comma-delimited set of subjects can be converted into a sequence of node hierarchies. template insertions are marked with doubled curly braces, e.g. “{{if:publisher:data iso}}” inserts the content of the “data iso” column if the “publisher” column is non-empty. the template language also supports looping structures and error reporting mechanisms. click here for a larger image we should note that archivist utility is in no way needed to use acumen. if, as of version 1.0, metadata is already in ead, mods, or mets, nothing else needs to be done for acumen to index and display it. adding support for additional types of metadata is straightforward and discussed earlier. figure 4. archivist utility allows the user to preview its output. archivist utility both strips empty sub-hierarchies out of its output, and “pretty prints” the files to make them easier for human beings to read. click here for a larger image archivist utility may also be of use even without acumen. it is useful for both converting large spreadsheets into xml via templates that can be created and tweaked quite easily (by populating a sample xml file with tags), and for interactively editing xsl and seeing its effect on source xml files. acumen’s naming convention acumen’s file-naming convention comprises four simple rules, and should make sense even without formal documentation: a file (metadata or asset) has a name comprising a base (e.g. “foo_bar”) and a tail (e.g. “_2048.jpg” or “.ead.xml”). the base is assumed to be unique for a given kind of file, and files of different type with the same base are assumed to be related (assets with a given base name are assumed to be described by metadata files with the same base name). e.g. foo_bar.xml is assumed to describe foo_bar.jpg, foo_bar.ocr.txt, foo_bar.txt, and foo_bar.mp3. the tail denotes the file’s type, and acumen allows types to be more specific than simple file type, e.g. “_2048px.jpg” is used to indicate an image scaled to fit inside a 2048 x 2048 pixel square.[6] base names are assumed to be hierarchical, comprising a series of alphanumeric identifiers separated by underscores (“_”)[7] , with the underscores denoting hierarchical relationship. e.g. “foo” contains “foo_bar” and “foo_baz”; “foo_bar” contains “foo_bar_001” and “foo_bar_002”. of these rules, the only one that acumen strictly requires is uniqueness. acumen assumes that foo_bar.xml is foo_bar.xml regardless of where it finds it. if two or more metadata files share the same base name (e.g. two files named foo.xml, or foo.ead.xml and foo.mods.xml) then this will be flagged as a potential error, and only one file will be visible to end-users (based on a configurable order-of-precedence per file types and, for files of the same type, latest modification date). because file base names are unique, acumen uses them to provide stable urls. it follows that if a metadata file is replaced by a new file of a different type, the url will stay the same. automatic indexing the following snippet of code (from our config.php source file) shows how acumen can be configured to handle local and remote repositories. a repository must have a specified web root; if no file system root is provided, acumen indexes the repository by crawling web directories (by parsing the default “index.html” listing produced by apache, and most other web servers, by default), otherwise it uses the file system (which is faster and provides better information). // $repositories is a global array of repository instances $repositories = array(); // ‘test files’ is local so we provide a file-system path to it $repositories []= new repository( 'test files', 'http://localhost:8888/repository/', '/users/tal/sites/repository/' ); // ‘acumen content’ is remote repository so we simply provide a url $repositories []= new repository( 'acumen content', 'http://acumen.lib.ua.edu/content/' ); assuming the metadata follows a specified naming convention and resides on a web server, simply point acumen at the repository and its content automatically gets indexed. once indexed, it is automatically searchable and automatically rendered for end-users. for fully-automatic indexing, one further step is needed: a cron task somewhere that periodically requests auto.php (wherever it is installed) via http. our system uses cron to request auto.php once per minute. acumen can also perform index passes interactively via its admin interface, and for testing purposes we’ve also created a simple desktop app which can poll auto.php as well. for all of this to work, acumen requires that: files in the repository adhere to acumen’s naming convention. the naming convention allows hierarchical relationships to be encoded in names (e.g. foo contains foo_bar contains foo_bar_baz) and file types and purposes to be consistently represented at the end of file names (e.g. .xml indicates an xml file, .ead.xml specifically indicates an ead, and _2048px.jpg indicates an image that fits inside a 2048×2048 pixel square).[4] in our repository, the first portion of a name reflects provenance, the second collection, the third item number, and the fourth is sequence.[5] the indexer is schema-agnostic (and can identify xml files either by naming convention or by “sniffing” for signature strings). to support a given kind of metadata one need simply provide two xsl files (one to summarize it and one to render it). we have thus far implemented xsl to support mets, mods, and ead. the digital assets exposed to end users will need to be in certain formats and also follow certain (customizable) naming conventions. these assets can be created automatically from whatever archival formats are being used. thumbnail and web-distributable images may need to be created from archived images files. again, this can be done automatically and fairly easily. as noted above, the indexing engine can be invoked manually (via a web admin interface) or automatically (via a cron job or similar). there is also the desktop application that can drive automatic indexing from any computer with web access. the indexer performs a complete scan in three stages: scanning. the indexer recursively walks each repository one folder at a time. each folder encountered is added to a task list, while files are categorized as metadata (generally xml), digital assets (e.g. jpeg, mp3, mpeg4), or something else (ignored or logged). each metadata file is summarized using xsl and the results stored in a database. each asset is simply recorded in the asset table. figure 5. assets are associated with metadata via our naming convention. in this case page one of chapter one of austen_pride has three associated assets, an image, some ocr text, and some text (the latter presumably a hand-edited version of the ocr text). click here for a larger image asset assignment. assets are assigned to metadata files based on naming convention. an asset with the same base name as a metadata file is assigned to that file; if no such file exists then it is assigned to the file which would be that file’s “parent” if it existed, and so on. e.g. austen_pride_01_01.txt has the base name austen_pride_01_01, which is identical to that of the metadata file austen_pride_01_01.xml and thus is assigned to it. if austen_pride_01_01 did not exist, the assets currently assigned to it would instead be assigned to austen_pride_01. assets with no parents are considered “orphans”. orphan assets are not visible to end-users because access to content is via metadata (and an orphan, by definition, has no metadata). during the indexing process, an asset indexed before its corresponding metadata file will temporarily be an orphan. when an indexing pass is complete, the presence of any remaining orphans is recorded, since these may be errors or omissions. figure 6. our admin interface provides access to current and past scan results. the presence of orphan assets is immediately apparent in the results.click here for a larger image when a user views a metadata file, the presentation system looks for assets already in the index that ought to belong to a metadata file and assigns them “just in time”; if no one looks at that file, the assets will still be assigned when the indexing engine gets to that file as a matter of course. because we can infer relationships between metadata files and assets purely from our naming convention, the need to create xml metadata wrappers (such as mets) for individual elements of complex objects (e.g. pages of a book) is eliminated. this system works admirably for objects that are ordered sets of things (as books are of pages or record albums are of tracks), but not all relationships between assets and metadata can be inferred purely from naming convention. we are in the process of ironing out how to handle some more complex examples, such as: assets which themselves contain metadata which might need to be assigned to the metadata file that owns the asset (e.g. text transcripts of an image or audio track) assets which are in themselves “more than one thing,” such as concert recordings which comprise performances of several different pieces. ultimately, if a relationship can be represented in xml in some satisfactory way then it can be parsed out via xsl, but we certainly haven’t covered every possible case. figure 7. if metadata files in a repository follow acumen’s naming convention the indexer can infer hierarchical relationships between them. this diagram shows how our naming convention would represent a hierarchy of related items. (file extensions have been omitted for clarity, and our actual names are not in themselves meaningful, e.g. u0003_000345_000002.) click here for a larger image metadata hierarchy. each metadata file not already owned by the “obviously correct” parent is assigned (if possible) to a parent metadata file. metadata files with no parents are considered “apical” (i.e. they are the top of a hierarchy — which makes them more prominent in collection listings). metadata files with no children (“leaves”) are suppressed in collection listings, unless they also happen to be apical. it is important to note that a complete scan against an empty database rebuilds the index from scratch while a complete scan against an existing database updates it (including flagging missing files as missing, moved files as moved, obsolete files as obsolete, and so on). search once a repository has been indexed by acumen (and during the indexing process itself) it may be searched acumen’s search interface, which allows users to create arbitrarily specific queries, refine existing queries, and exchange stable query urls. owing to our very simple database architecture (three main tables), search queries are both simple and efficient. the user interface restricts queries to ands, which keeps searches very fast. our search back end is completely flexible and allows queries to be built from urls in polish notation[8], defined thus: /s/{/[_and|_or/]_contains(_all|_any|_exact)[_considercase]//{+}} in concrete terms this ui: generates (and is generated by) this url[9]: /s/all/_and/_contains_all/all/cabanis/_contains_all/box_number/252/_contains_all/folder_number/4 which becomes this sql fragment: inner join authority as a0 on file.id = a0.file_id inner join authority as a1 on file.id = a1.file_id inner join authority as a2 on file.id = a2.file_id where ((a0.authority_type_id = '17' and (a0.value like '%4%') and a1.authority_type_id = '16' and (a1.value like '%252%') and (a2.value like '%cabanis%'))) search results are rendered conventionally on the server, but an obvious (and planned) refinement of this would be to allow xsl rendering of search listings so that different kinds of records can be rendered more flexibly. here is an example of what our search results currently look like: other features categories. acumen allows for categories to be specified simply by providing a category name and a query that will specify members of that category. collections. acumen provides support for collections, currently based on naming hierarchies. the logic which determines relationships could easily be modified by other institutions to handle other methods of representing hierarchies and other relationships, such as folder hierarchies or explicit links. discovery. acumen provides a simple discovery interface comprising randomly selected thumbnails picked from the currently selected search category. we plan to substantially improve our discovery interface and add curatorial tools. immediate indexing. in addition to automatic indexing, acumen allows archivists to index a specific directory interactively and out-of-order so that it immediately becomes available. url stability. providing stable, clean urls is a core design goal of acumen. because we require metadata files and assets to have unique names, our urls are completely independent of repository structure (so repositories can be restructured without changing object urls). exposure to search engines. acumen automatically generates site maps and complete listings of all its content in the form of stable urls. presentation once acumen is installed on a web server and has completed indexing a repository (or repositories) it will allow users to search and view the repository’s content in any modern web browser (we’ve tested it with ie7 and ie8, as well as current and recent releases of firefox, safari, chrome, and opera). when a user clicks a link, the original metadata file is located in the repository, its type is retrieved from the index database, and the appropriate xsl stylesheet is located to render it into human-readable form. at this point the original metadata is then rendered to html on the server “just in time” and sent to the user’s browser or — if the link ends in “.xml” — the stylesheet is linked to the metadata file and the file itself is sent to the user’s browser which then renders it to html in the browser. it follows that changes made to metadata files are immediately reflected on the end-user’s browser, even if the indexer hasn’t scanned them yet.[10] data access api no data access api is needed — acumen does not ingest data, and the raw data is made available to anyone. by default we create html representations of raw metadata files (such as ead and mods files) via xslt on the server, and then pass the resulting html to a user’s web browser. but if “.xml” is added to one of our metadata urls, we send the original xml with an xsl stylesheet tag inserted to the end user’s browser, which (a) gives end users complete access to raw data with no intermediate translation or loss, and (b) looks just like a regular web page.[11] moreover, we make our entire repository — both metadata and digital objects — available via the web (apache can be configured to block access to files by type, if desired, or a repository may simply be a mirror of an archive which omits some files), allowing third-party software, web crawlers, agents, and web applications to access our data. it follows that any changes to existing metadata files become available to end-users as soon as they are placed in the (web-accessible) repository and are searchable as soon as the indexer scans them. one of the fields we store as a matter of course is the full content of each metadata file stripped of tags. searching for content contained in a tag which is not handled by the existing xsl is always possible (by searching metadata file content). furthermore, if we want to store data in a new searchable field we can simply add it in the xsl (e.g. we might want to add a new field named “medium” which we could do simply by outputting in our xsl summary). when the indexing engine parses this tag it will simply create a new entry in the authority_type table for “medium” and the search interface will immediately pick this up and make it searchable. as a final note, the indexing engine skips metadata files which it has already scanned if neither they nor the summary xsl file has changed since the last time they were scanned. thus, changing a summary xsl file will force all metadata files of the type it summarizes to be reindexed. summing up and moving forward the system described has been in development for slightly more than twelve months to reach release (“1.0”) quality, having been in usable “beta” form for six of those months. moving forward we plan to make it more modular and easier to configure and use, while adding functionality. this is a project that can be infinitely refined, but our goal has always been to do as little as possible while asking as little as possible of everyone else. while we have not chosen to use an off-the-shelf product, we have made use of some very powerful off-the-shelf components, including xslt — which does most of the heavy lifting in our indexing and presentation components, and affords us almost limitless opportunities to improve both without writing another line of “code” — and jquery — which vastly simplifies the client-side interactivity layer. in essence, acumen is a lightweight wrapper around xslt, and a great deal can be achieved (e.g. different or more attractive presentation, customization, improved indexing) simply by adding to and refining our xsl. “in the long run,” to quote john maynard keynes, “we’re all dead.” it is worth considering what will be left behind when our software is no longer there. acumen specializes in discovery and presentation, effectively modularizing digital delivery so that maintenance and archiving remain outside of the system, under the direct control of the implementers. this simplifies workflows and reduces time and effort in metadata remediation, migrations, and content management. to deploy acumen, repository content must adhere to a self-explanatory naming convention, and the contents of metadata files must adhere to standards and be kept in good order, and all of this must reside on a web-accessible server. remove acumen, and what remains is repositories of standards-compliant metadata with consistent file names. notes [1] acumen requires (or at least has only been tested with) php 5.2.x, mysql 5.x, and apache 2.x. it works with current and recent installations of ubuntu linux, mamp (on the mac), and wamp (on windows). [2] our installation and hot fix processes are both documented and will be available for download. the installation process is comparable in complexity to wordpress or phpbb. [3] although this tool will be made available for free along with its source code, it was developed in realbasic (http://realbasic.com/) which is not free or open source (although the basic linux version is free). archivist utility runs on mac os x, windows, and linux. we plan to port this tool to our web platform. [4] names are assumed to be unique in the sense that if multiple things (e.g. a metadata file and an image) have the same “base name” (i.e. not including file type “tail”) then the system considers them to be part of the same object. [5] this naming convention reflects the lockss journal hierarchy, and also works well with our automated lockss system. [6] we use unique tails for the various specific types of metadata files, e.g. “.ead.xml” for eads, but because acumen started being used before these conventions were in use it is also able to identify xml files by examining their contents. [7] different characters may be used with a change to the configuration file. [8] best known for being used, backwards, in hp calculators. see http://en.wikipedia.org/wiki/polish_notation [9] note that the url explicitly conjoins the search criteria with and (rather than or, or some parenthesized combination). as of version 1.0, the back end supports arbitrarily complex searches, but the ui only supports and. [10] a metadata file’s type could (and probably should) be determined on-the-fly just as it is during scanning, which would allow a metadata file to be replaced by a new type of metadata file and the new information to be correctly presented even if the changes had not yet been scanned by the indexer. we will probably revisit this issue when we implement a caching mechanism for rendered html. [11] until supporting older versions of internet explorer became a requirement, acumen emitted xml by default and only sent html by request. it turns out that ie6 and 7 do not correctly handle even moderately large xml files. about the authors jody deridder (jlderidder@ua.edu) is head of digital services at the university of alabama libraries. she is interested in developing and supporting practical, low-cost, scalable solutions to digital library problems, from capture and description, through improved usability, to long-term access and preservation. tonio loewald (taloewald@ua.edu) is an analyst/programmer at the university of alabama libraries. he has developed educational and game software for both small children and large ones with advanced degrees, as well as a program for creating and deploying electronic questionnaires for social science research (riddlemethis). he believes that bad user interfaces are a kind of bug that manifests when software runs on people. subscribe to comments: for this article | for all articles 2 responses to "metadata in, library out. a simple, robust digital library system" please leave a response below: how does this acumen thing work? | digital services @ the university of alabama, 2011-05-19 […] live as open source.  tonio loewald is the developer, and we published an article about acumen in code4lib last year, which will contain more detail than this blog […] jody deridder, 2011-05-23 acumen has finally been released open source: http://sourceforge.net/projects/acumendls/ the long-awaited day has finally arrived. after many trials and tribulations, we have at long last released acumen digital library software open source!!! this software is light-weight, easy to install (intended for linux or osx), easy to use, and it does *not* require munging your xml metadata into some software-specific form. you can load multiple kinds of content into it — finding aids in ead, images with simple dublin core, tei transcribed documents, audio with xml play lists, any kind of digital content you want with your own form of xml metadata. you can have acumen index content all together from multiple sites!! thus this is an option for those wanting to set up state-wide digitization projects — or wanting to move away from contentdm multi-site server software. just include the base of the web-accessible directories in the configuration file. the file naming is the key to organization and to delivery. but besides that — you can add, remove, change content at will in the web directory where you put it, and acumen will automatically update the display during the next round of indexing. how easy can it get? acumen. finally out there for free!! :-) http://sourceforge.net/projects/acumendls/ leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – always be migrating mission editorial committee process and structure code4lib issue 50, 2021-02-10 always be migrating at the university of california, los angeles, the digital library program is in the midst of a large, multi-faceted migration project. this article presents a narrative of migration and a new mindset for technology and library staff in their ever-changing infrastructure and systems. this article posits that migration from system to system should be integrated into normal activities so that it is not a singular event or major project, but so that it is a process built into the core activities of a unit. by elizabeth mcaulay introduction in libraries software systems undergird the majority of our work. however, regardless of how ingrained a system may be in our daily activities, all software solutions are temporary [1]. within an organization each software system has a life cycle: (1) selection, (2) implementation, (3) use, (4) dissatisfaction, and (5) migration and decommission (see figure 1). there is the phase of the rising system: selection and implementation. then there is an era of smooth sailing in which we use the system routinely and the system is integral to our work. but soon, the system enters its waning phase, and the looming death of the system haunts us [2].  the waning phase can be precipitated by internal or external forces, such as user and/or developer dissatisfaction with the system or software end-of-life decisions from external bodies. when the external or internal pressure reaches an inflection point, organizations initiate a new software system life cycle. in their recent article, babcock et al. (2020) offer a good overview of this life cycle relative to their digital collections software system. as one system is waning, another life cycle is usually spawned that leads to the birth of a new system, and part of that process of switching systems is the awkward and onerous task of migration. migration is the one-time activity in which we port our content from the old system to the new system. figure 1. two systems life cycles chained together. software system life cycles are a natural part of library organizations and, therefore, migrations are a frequent activity in library technology groups. i see evidence of the prevalence of migration activities both in my own organization and when consulting the articles in this journal. at the time of writing, the university of california, los angeles (ucla) library is involved in migrating four of our major systems: the general website content management system, the electronic resources management system, the integrated library system, and the digital collections repository [3]. when looking to our larger code4lib community, the frequency of migration projects is evidenced by the variety of articles on the topic in the code4lib journal; at least 10 articles published within the past four years specifically report on migration tools, methods, or projects [4]. nevertheless, migration is often seen as a one-way, one-time activity — just a big project with a lot of throwaway, case-specific code. given the inevitability of migration in library technology work, i propose there is an opportunity to identify migration principles that support libraries in adopting and using systems. while specific examples and approaches to migration are useful, this article aims to serve as a companion during any migration. in this article, i will provide some specifics about the context of the migration for the ucla digital collections and then outline some proposed evergreen principles of migration [5]. case study: background the digital library program at the ucla library was established as a library unit in 2004, comprised of librarians and situated to partner with a small team of digital library software developers in the library technology unit. since 2004, the digital library program unit has created and maintained a growing corpus of unique digital collections and projects. some of these projects actually continue on the web in their near-2004 vintage, e.g., our campaign literature collection, which contains ephemera related to elections in california from 1908 to 1940. at present we manage 201 published collections comprising 3.25 million assets and 235 terabytes [6]. during the period of 2004 to the present, we have used many different systems. as with many digital library initiatives, at ucla we have endeavored to support both custom projects with unique features and also generalized systems that promote increased output [7]. there was a period in which we focused on the marriage of these two opposites by using generalized systems to manage and structure metadata and content files underneath customized interfaces to display these items and enable unique user experiences. the underlying database and web application programs retained standardized workflows that could support high volume output and aggregation, and then custom web application programming could be layered on top for specialized features and unique branding. this endeavor occupied us locally, but also matches the broader developments across the digital library field. during this period, many institutions were developing repositories and other technologies that would support increased standardization and interoperability: dspace, fedora, hydra (now samvera), islandora, and the international image interoperability framework (iiif). first, we used a one-collection-per-system approach. we developed our own custom database schema, our own custom web application, and we cloned the schema and application and customized it for each collection. each collection was a standalone instance and could be customized as much as we could afford [8]. as the collections started accumulating, we recognized that the aggregate of the collections was more meaningful than any single collection, and we re-architected our custom database plus java solution into a custom-built repository and publishing system. this new system meant that we had a locally developed repository system that allowed for greater efficiency and improved discoverability. however, soon we started to realize that we were isolated with our locally developed system. we were unable to share code with other institutions so we were always dependent on our own development personnel to accomplish any new objective. at that same time, we observed other institutions converging on shared technologies and developing features and solutions more quickly. to participate in this new way of working, we adopted islandora as our digital library solution and drupal as our new content management system for the broader library website [9]. this new solution did improve some of our capacity issues, but after roughly four years we again recognized the need for a new system that could better fit our needs. first among our reasons for selecting a new system was that the potential synergy between drupal and islandora was not as productive as imagined. second, we realized we did not effectively connect with the larger islandora community of developers and users [10]. thus, in 2016, we began our systems life cycle again. we started with a formal review and selection process for a new digital library solution. two digital library staff members were assigned to research and write a white paper; they were asked to outline our needs, conduct an environmental scan of possible systems, and offer a recommendation on how to proceed [11].  as an organization, we used the white paper to support discussion, review options, and select a new system. we chose samvera, which is actually a suite of systems, specifically hyrax for asset management and blacklight for discovery and display. since that decision, we have both adopted and begun implementation of the new system while also establishing a microservices approach. in this approach, we have developed a workflow and suite of microservices specifically around preparation and delivery of image assets using the international interoperable image framework (iiif). we have separated our iiif applications into a set of services that our digital library applications utilize. at present, our iiif-focused microservices include image preparation, iiif-manifest preparation, and a iiif image server. after images are prepared through these microservices, we ingest a comma separated values (csv) file with uris for the image asset and its corresponding iiif manifest into hyrax. our hyrax instance adds data to a solr index that is copied to a second index which drives our public blacklight instance. in this current instantiation, any one of the services can be replaced or re-architected while the other components remain unaltered. in the same fashion as the university of toronto, as reported by babcock et al. (2020), ucla library will continue to break our technical infrastructure and systems into microservices to allow for more nimble development, maintenance, and improvement going forward. as part of that transformation, we might find ourselves always migrating — at least a little bit. data must flow indeed, even while i am wrestling with the challenges and details related to our current migration to samvera, we are re-migrating some of the already migrated collections so that we keep pace with the newer microservices. i found this reality somewhat frustrating even while i am supportive of the microservices philosophy. in the midst of this reaction, i recognized an opportunity to analyze our migration activities and shift from a telic mindset to an atelic one. the greek word telos means end or purpose and telic activities would therefore be those that have an end as their goal. by contrast, atelic activities are those that are pursued for their own good and without end. while clearly the migration project we are working on at ucla for our digital library will have a completion — a successful one! — i would like to attempt to reconceive migration as a practice, like yoga, instead of a single event like a race or a match.  an always-migrating mindset might include thinking about our metadata and content as water that flows through a faucet that repository managers can turn on at any point and direct toward any garden. principles of migration from my experience through several migrations, i have learned that while system-to-system migration is always specific and non-repeatable, there are some principles that undergird every migration and are worth sharing. i propose the following principles: documentation: take the time to document and share your data with your organization. when you make decisions about how to organize your data, document those as well. leave a trail, they may be your breadcrumbs. transparency: treat your migration like something that someone else might need to take over at any time. love your data: interfaces and systems change. the data that flows through these systems is the true value in all our organizations. interoperability is a verb: moving data among systems improves its interoperability. first, systems exist to serve a purpose. in the past, that purpose and the current system were aligned and married. however, when a system approaches its end of life, it is a good moment to re-examine the purpose it is serving. ask questions. what is the mission-driven service that the system enables or supports? in what ways does the organization want to evolve that service? what are the bottlenecks that impede efficiency or degrade the quality of the service? in asking these questions, you can ensure that you and your organization are not mistaking aging technology for outdated services or processes. second, inventories of what is in a system are essential. inventories are both philosophical and practical. philosophically, an inventory should present the theory of the system: what are the things the system contains, how many types of things are there, what is the relationship between the internal inventory (often a database) and the thing it points to (a file, a link, a physical entity). your inventory needs to show you the theory of the system and then it needs to provide a scale of the system. inventories can serve throughout all stages of the system life cycle and can help you continually evaluate the efficacy of your system. third, this migration will not be your last. everything about a migration is focused on getting it done: getting to the promised land where everything will be in the new system and working better. then we can get back to our regular work and everything will go more smoothly. that ethos of migration is a partially true myth. the new system will improve some of your regular work, but you are still in a life cycle of systems. more importantly, your data and metadata are in motion constantly–serving your organization’s mission. you are always in the process of imagining new ways to orchestrate the connections among your materials, access, data, metadata, users, and staff. therefore, if another migration might be in the near future, i encourage you to prepare for it now. i see this as a gift to a future self. is there some aspect of this migration or the new system that you can arrange to make future migrations easier? some areas to consider: documentation, data integrity, inventory processes, and communications. conclusion this article is a thought piece about a standard practice in library technology: systems migration. i urge us to see migration as an activity that is a phase of a life cycle, and that we are always engaged in that life cycle. migrations are disruptions to existing workflows and work assignments with enormous effort dedicated toward them, but so frequently these migrations are filled with throw-away scripts and data or metadata mappings that are only useful for the current migration. our challenge is to extract learning from each migration that might help us when all the specifics may be completely different when the next migration rolls around.  with these principles in hand, i think we can adopt an ethos of always migrating that more closely matches the reality of systems life cycles in libraries. coda one final note about software systems: they are run by people. the best software solution for your organization’s list of requirements is not necessarily the right system for your organization. a software system and a migration strategy need to align with the motivations and aptitudes of your organization, your organization’s leadership, and your current staff. managers and decision-makers should push team members to take on new challenges and to continue to learn new technologies. however, we should ensure that the challenge aligns with the values and motivations of the leaders. moreover, if key personnel related to the system change, take a moment to evaluate whether your current trajectory still has the same alignment. even if you just finished a migration to a new system, it’s okay to always be migrating. in fact, with these best principles in hand, you are ready for it. notes [1] for a good summary of some scenarios that lead to migrations see hodges and schlottmann’s introduction (2019). in their article about their digital collections interface tool, becker, williamson, and wikle note as an aside that “no platform is future proof” (2020). [2] this life cycle is the same whether systems are vendor-supplied or locally developed or some combination of the two. likewise, the connection between an old system and a new system remains migration, and whether supported by a vendor or performed locally, the readers of this journal will find themselves involved. [3] for our general library website we are migrating from a single content management system and web publishing solution (drupal 7) to craft as our content management system and a locally developed vue.js end user interface. the migration for both our electronic resource management system (summon) and our integrated library system (voyager) is driven by the university of california’s system-wide selection of ex libris’ alma-primo solution. lastly, for our digital collections repository system, we are migrating away from our locally developed oracle-java web application to a microservices suite of applications that leverages community products, including samvera applications and cantaloupe for implementation of the specifications outlined by the international image interoperability framework (iiif). [4] i searched the code4lib journal archive using the keyword “migration” and then reviewed the introduction to each article to determine whether the article was substantially about a technology system migration. from this process, i identified 10 articles that substantially addressed migration, including migrations of repositories, archival description systems, and integrated library systems. notably, these articles were all from the past four years. these articles were: babcock et al. (2020), banerjee and forero (2017), gomez et al. (2020), hodges and schlottmann (2019), maistrovskaya and newson (2019), mayo (2019), mayo and bowers (2017), ramshaw et al. (2018), teal (2018), and tuyl et al. (2018). [5] i see this article as furthering the concepts presented by tuyl et al. so creatively in their 2018 migration as greek tragedy article. [6] the way we count our assets varies across our multiple platforms. depending on the collection, we may count an issue of a local newspaper as one item or as eight items (each page being an item). that discrepancy is caused because we currently are using several repository systems and we are building inventories based on the way those systems work instead of maintaining an agnostic and independent inventory. in the future, we will want to be able to represent our holdings in multiple counting methods: volumes, linear feet, and archival digital assets. for the purpose of evaluating the magnitude of the collections we are managing and migrating, the size in terabytes indicates the scale and the number of collections indicates the diversity of these assets. [7] this approach has been discussed in the literature in the following article: becker, williamson, and wikle (2020). [8] we used oracle database and wrote java web applications. [9] our islandora 7 instance still serves as a repository for some of our collections, but we use a non-islandora web application for public discovery. see, for example, the international digital ephemera project and the picturing ucla collection. for more about the islandora digital library software, see https://islandora.ca/. the ucla library website uses drupal 7. [10] the islandora community is robust and many creative and effective repositories have been built using islandora. i do not intend to disparage the islandora community in these comments, but i am aiming to be transparent about my experience in managing the ucla digital library program through a variety of digital repository systems. the process of selecting and evaluating systems is not the focus of this article so i am aiming to provide enough details about the context around a migration process without delving into that context substantially. [11] our process and key activities were similar to those outlined by tuyl et al. (2018) and babcock et al. (2020) in their recent articles: conduct stakeholder discussions and outline our current needs; investigate existing software solutions that might meet these needs; and provide a recommendation. the full white paper has been made available on zenodo by the authors peter broadwell and dawn childress https://zenodo.org/record/3027359#.x-pb6-lkg1s. doi:10.5281/zenodo.3027359. the white paper process worked very well for our organization because it encouraged a process that allowed for requirements gathering and an environmental scan with a resulting white paper that promoted discussion and debate. after a decision about our replacement system was made, the white paper supported communication and transparency. now that a few years have elapsed, the white paper still serves as a record of the software solutions available at that time and our organization’s articulated needs. bibliography babcock k, lee s, rajakumar j, wagner a. where do we go from here: a review of technology solutions for providing access to digital collections. the code4lib journal. 2020 [accessed 2020 nov 14]; (47). https://journal.code4lib.org/articles/15000 banerjee k, forero d. 2017. diy doi: leveraging the doi infrastructure to simplify digital preservation and repository management. the code4lib journal.(38). [accessed 2020 dec 29]. https://journal.code4lib.org/articles/12870. becker d, williamson e, wikle o. 2020. collectionbuilder-contentdm: developing a static web ‘skin’ for contentdm-based digital collections. the code4lib journal.(49). [accessed 2020 dec 29]. https://journal.code4lib.org/articles/15326. broadwell p, childress d. digital library systems and ucla: an environmental scan with suggestions for near and medium-term technology strategies and user stories / functional requirements. zenodo; 2017. https://zenodo.org/record/3027359#.x-pb6-lkg1s. doi:10.5281/zenodo.3027359. gomez j, clarke ks, vuong a. 2020. iiif by the numbers. the code4lib journal.(48). [accessed 2020 dec 18]. https://journal.code4lib.org/articles/15217. hodges dw, schlottmann k. 2019. reporting from the archives: better archival migration outcomes with python and the google sheets api. the code4lib journal. (46). [accessed 2020 dec 12]. https://journal.code4lib.org/articles/14871. maistrovskaya m, newson k. 2019. making the move to open journal systems 3: recommendations for a (mostly) painless upgrade. the code4lib journal. (43). [accessed 2020 dec 29]. https://journal.code4lib.org/articles/14260. mayo c, jazairi a, walker p, gaudreau l. 2019. bc digitized collections: towards a microservices-based solution to an intractable repository problem. the code4lib journal. (44). [accessed 2020 dec 12]. https://journal.code4lib.org/articles/14445. mayo d, bowers k. 2017. the devil’s shoehorn: a case study of ead to archivesspace migration at a large university. the code4lib journal.(35). [accessed 2020 dec 29]. https://journal.code4lib.org/articles/12239. ramshaw v, lecat v, hodge t. 2018. wms, apis and libguides: building a better database a-z list. the code4lib journal.(41). [accessed 2020 dec 29]. https://journal.code4lib.org/articles/13688. teal w. 2018. alma enumerator: automating repetitive cataloging tasks with python. the code4lib journal.(42). [accessed 2020 dec 29]. https://journal.code4lib.org/articles/13947. tuyl sv, gum j, mellinger m, ramirez gl, br, straley on, wick r, zhang h. 2018. are we still working on this? a meta-retrospective of a digital repository migration in the form of a classic greek tragedy (in extreme violation of aristotelian unity of time). the code4lib journal.(41). [accessed 2020 dec 12]. https://journal.code4lib.org/articles/13581. about the author elizabeth mcaulay is the head of the digital library program at the university of california, los angeles library. subscribe to comments: for this article | for all articles one response to "always be migrating" please leave a response below: bill hackenberg, 2021-02-17 lisa, i really enjoyed your post. your perspective to see migration as a process is very illuminating. also, really enjoyed working with you in a project manager role for your first implementation of samvera at the ucla library. your post really got me thinking. what about a progress metric of some sort? it might provide some additional insight into the migration process. how much content has finished? how much is left? what is the pace of migration? here’s a recent example of the potential benefits of a progress metric. early in the recent 2021 covid-19 vaccination phase, a leader was heard to say something along these lines: if we continue at the current pace, we will finish in late 2022. this lead to improvement! progress metrics can guide practitioners and inform leadership. here’s another example. i am not sure is this is still true, but at one time the size of the team tasked with painting the golden gate bridge was determined so they could just start again when finished. this was a factor of the life-span of the paint divided by the size of the team. with a good progress metric, you may be able to synchronize with the system life cycle. progress metrics serve all sorts of purposes and are frequently used by project managers for planning and tracking purposes. sometimes they lead to change. sometimes not. in other words, quantifying progress can be helpful! leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – book review: html5: up and running mission editorial committee process and structure code4lib issue 13, 2011-04-11 book review: html5: up and running mark pilgrim’s html5: up and running was one of the first books published on the subject. if you’re looking for a really good, well-written, entertaining, concise overview of what’s going on right this very minute with html5 technologies and techniques, this is a good book to have. mark pilgrim. html5: up and running. sebastopol, ca: o’reilly and google press, 2010. 205p. isbn: 978-0-596-80602-6 by mark cyzyk reading the first chapter of mark pilgrim’s html5: up and running, i kept thinking to myself, “what kind of geek would you have to be to find this the least bit interesting?” my kind of geek, i guess, because i found it riveting. chapter 1 of this slender volume traces, among other things, the genesis and evolution of the html <img> element, starting with an initial usenet post by marc andreesen back in 1993. for those who don’t know, marc andreesen was the university of illinois programmer who cowrote, with eric bina, the world’s first popular graphical web browser, mosaic. for a time, back in the early-mid nineties, he was the poster boy for all things web. this first chapter traces his initial suggestion for an <img> element, its reception by the usenet group, the group’s competing suggestions, its ultimate inclusion in mosiac, and then a nice historical overview of the development of html and xhtml standards, all as context for the recent advent of html5 and a prelude to the rest of the book. as someone who appreciates that most things we take for granted in our workaday worlds – tools, technologies, techniques, policies – do not fall in final form out of the sky, but instead have idiosyncratic histories and individual evolutions, i found pilgrim’s narrative of the evolution of a single html tag instructive, illuminating, even fascinating in its drama. here is one small, microcosmic example of how all that we are familiar with in the web world came into being. interesting stuff. chapter by chapter, the author then takes us systematically through the new features of html5. chapter 2 focuses on techniques for detecting browser support of particular html5 functions. the author usefully provides raw javascript snippets for doing this, but then introduces us to the even more useful “modernizr” library for html5 feature detection [1]. while browser compatibility for html5’s new features and functions is described throughout the book, chapter 2 emphasized general coding techniques that help ensure detection and graceful degradation when html5 is not supported. chapter 3 starts digging into the new functionality of html5. after an illuminating discussion of the importance of proper <!doctype> declarations, it discusses and illustrates the new semantic tags in html5. use cases and code illustrations for <section>, <header>, <footer>, <article>, <time> tags are provided. chapter 4 addresses the new <canvas> element, a tag that defines a region on the page for drawing and scripting 2d shapes and bitmap images. drawing in 2d is useful and cool, and the book shows how to draw lines, circles, gradients, etc., but the html5 specs leave room for a <canvas> that supports 3d, and when that happens i think we are all going to witness great things. back in the day, i was always intrigued by the virtual reality modeling language (vrml). i wonder how this new technology might compare? we’ll see. chapter 5, video on the web, i found to be really interesting. it not only emphasizes html5 support for the various video/audio containers and codecs, but it explains what those containers and codecs are, how they relate to one another, and how to plan for the graceful handling of idiosyncratic browsers by adhering to a recommended hierarchy of video/audio formats; page 90 lists a useful workflow for creating multiple versions of video clips and streams in differing formats so you squeeze the best performance possible out of every browser/codec format combination. chapter 6 addresses the html5 geolocation api which enables web browsers and their users to share location information with trusted websites, and chapter 7 delves into the topic of local storage for web application use, its history and evolution from the ubiquitous cookie to the new web storage feature allowing name/value pair storage of a default, relatively whopping, 5mb of data. chapter 8, “let’s take this offline”, ponders the notion of an “offline web application”. browsers supporting this will look for a linked manifest file in a page’s <head> section. any files linked from this manifest will then be downloaded and cached for offline use. a slew of potential problems surround this behavior which the author admirably dissects and debugs. i’m betting the features discussed in chapter 9, “a form of madness”, will be the most appreciated in the promising new html5 world. here are enumerated the multiple new html form input types available to html5-compatible browsers. these include support for an autofocus formfield, placeholder text in a formfield, new input types for email addresses, urls, numbers as spinboxes, a slider input type, and a nifty date picker input type. all of this simply makes programming those html forms easier for you, the programmer. no more javascript snippets required when these sorts of functionalities are built directly into the browser. the final chapter, “‘distributed,’ ‘extensibility,’ and other fancy words,” introduces the notion of “microdata”. essentially, microdata is metadata about what’s on the page, integrated directly with data on the page; “microdata” refers to the use of custom vocabularies to semantically markup data on a page. an example in the book suggests a vocabulary for the concept of “person”, including such terms as “name”, “nickname”, “affiliation”, and “title”. this chapter shows how to create a namespace for the “person” vocabulary, then how to mark up data on the page accordingly. so here we find, on page 171, the following illustration, with semantic markup (“name”, “title”, “affiliation”) intermixed with structural/presentation markup (dd, dt): <dd itemprop="name">mark pilgrim</dd> ... <dt>position</dt> <dd><span itemprop="title">developer advocate</span> for <span itemprop="affiliation">google, inc.</span></dd> why is this important? this technique lends semantic meaning when being programmatically processed by, say, a search engine. a program looking at your page can tell, via microdata markup, that this-here-is-an-actual-name-of-a-person rather than merely being a string of characters on the page. pilgrim, in this chapter, then illustrates how this notion of “microdata” is supported by google’s rich snippets technology [2]. basically, google will crawl your page, determine that there is microdata present, index the values contained within this semantic markup, and usefully customize its presentation of search results accordingly. so searching for a properly-marked-up “person”, google will return a search results screen in the usual format, but will also set aside screen real estate to highlight the fact that the string being searched is about an actual person, someone with a name, title, and position in an organization. the book concludes with an appendix recapping the new features and elements in the html5 specification. just glancing through this list gives you a quick impression of the scope of changes in the new spec. as always, o’reilly books qua technical books set the standard for page layout, code quotation conventions, sidebar formatting, etc.; things you might not think about, but they add greatly to the readability and effectiveness of the book as a whole. i raise my glass to the book designers of o’reilly. my sense, heightened and honed by this book, is that all of this html5 business is rapidly morphing and evolving. although tech books eventually go the way of the <blink> tag, i’m sensing that this one will go quicker than, say, your beloved and venerable o’reilly dynamic html: the definitive reference. nevertheless, if you’re looking for a really good, well-written, entertaining, concise overview of what’s going on right this very minute with html5 technologies and techniques, this is a good book to have. an evolving version of the book freely is available from the author at http://diveintohtml5.org/. references [1] modernizr javascript library. http://www.modernizr.com/ [2] rich snippets (microdata, microformats, and rdfa). google webmaster central. http://www.google.com/support/webmasters/bin/answer.py?answer=99170 about the author mark cyzyk is the scholarly communication architect in the sheridan libraries, johns hopkins university, baltimore, maryland, usa. in 1994 he installed trumpet winsock and an early version of the ncsa mosaic web browser on a computer at the reference desk of the albert s. cook library, towson state university, towson, maryland, at which point he gathered everyone around, hit the button, and lo and behold, a pretty picture on the screen! subscribe to comments: for this article | for all articles one response to "book review: html5: up and running" please leave a response below: usenet, 2011-08-26 why is this important? this technique lends semantic meaning when being programmatically processed by, say, a search engine. a program looking at your page can tell, via microdata markup, that this-here-is-an-actual-name-of-a-person rather than merely being a string of characters on the page. leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – optimizing elasticsearch search experience using a thesaurus mission editorial committee process and structure code4lib issue 51, 2021-06-14 optimizing elasticsearch search experience using a thesaurus the belgian art links and tools (balat) (http://balat.kikirpa.be/) is the continuously expanding online documentary platform of the royal institute for cultural heritage (kik-irpa), brussels (belgium). balat contains over 750,000 images of kik-irpa’s unique collection of photo negatives on the cultural heritage of belgium, but also the library catalogue, pdfs of articles from kik-irpa’s bulletin and other publications, an extensive persons and institutions authority list, and several specialized thematic websites, each of those collections being multilingual as belgium has three official languages. all these are interlinked to give the user easy access to freely available information on the belgian cultural heritage. during the last years, kik-irpa has been working on a detailed and inclusive data management plan. through this data management plan, a new project hescida (heritage science data archive) will upgrade balat to balat+, enabling access to searchable registries of kik-irpa datasets and data interoperability. balat+ will be a building block of digilab, one of the future pillars of the european research infrastructure for heritage science (e-rihs), which will provide online access to scientific data concerning tangible heritage, following the fair-principles (findable-accessible-interoperable-reusable). it will include and enable access to searchable registries of specialized digital resources (datasets, reference collections, thesauri, ontologies, etc.). in the context of this project, elasticsearch has been chosen as the technology empowering the search component of balat+. an essential feature of this search functionality of balat+ is the need for linguistic equivalencies, meaning a term query in french should also return the matching results containing the equivalent term in dutch. another important feature is to offer a mechanism to broaden the search with elements of more precise terminology: a term like “furniture” could also match records containing chairs, tables, etc. this article will explain how a thesaurus developed in-house at kik-irpa was used to obtain these functionalities, from the processing of that thesaurus to the production of the configuration needed by elasticsearch. by emmanuel di pretoro, edwin de roock, wim fremout, erik buelinckx, stephanie buyle, and véronique van der stede introduction with the hescida project [1], kik-irpa [2] wishes to transform how users search across its collections. now, the search platform, called balat [3], queries the api of the collections management system (cms), namely adlib xplus [4]. a first step towards achieving a new system, called balat+, is to replace those api calls with another component, based on elasticsearch [5], which is a search engine based on lucene that is available via a rest api. using elasticsearch will allow more fine-tuning about the way data are indexed. the next step consists of integrating other datasets in balat+, such as the data produced from the scientific and technical research performed by kik-irpa researchers. balat+ must offer at least features on par with the current system but should eventually improve the overall search experience. during the initial installation and configuration of elasticsearch it became obvious that it was possible to exploit and profit from the work already done within the library and photo library part of the current system. the latter contains 98,037 entries, which can be terms coming from dutch, english, or french. this thesaurus is used to describe the works of the library, and the resources of the photo library. in this article, the exploratory work done to configure elasticsearch and thus offer the expected functionalities for balat+ will be presented. overview of the method the task at hand consists of generating configuration files for elasticsearch based on a thesaurus available in a cms. one solution would be to devise a specific algorithm to obtain that result, for instance by creating a data structure and querying it to obtain linguistic equivalences and narrower terms, an alternative would be to benefit from existing technologies to get to the results quicker and with less effort. to do so, a resource description framework (rdf) graph [6] will be built by using a specialized vocabulary named simple knowledge organization system (skos) [7], that help express basic structure and content of concept schemes such as thesauri [8]. in practical terms, the thesaurus will be converted in skos as an rdf/turtle file to be later imported in an rdf graph, sparql [9] queries will then be used to get the needed information, and some basic processing will take place to format the results of queries as synonym files for elasticsearch. this solution is much more flexible than an ad hoc algorithm, as only the sparql queries will need to be updated if new situations need to be handled. and it is also a much stronger solution as proven technologies are used. collecting the thesaurus from adlib xplus as mentioned, the cms used by kik-irpa is adlib xplus from axiell. to retrieve the thesaurus, the adlib web api [10] was chosen as the easiest method. that api offers access to all the databases present in the cms. to achieve this, a simple query to the search [11] command of the api is needed which looks like this query: http://adlib_server/wwwopac/wwwopac.ashx?database=thesau&search=all. the url can be explained as follow: the adlib xplus installation part: http://adlib_server/wwwopac/wwwopac.ashx; the query part: the database parameter with the name of the database to query and the query parameter. a special keyword, all, is used to ask adlib xplus to return all records from the database. it returns an xml file that looks like: <adlibxml> <recordlist> <record priref="2" created="2013-08-17t21:28:40" modification="2020-07-18t17:07:38" selected="false"> <broader_term>philosophie</broader_term> <broader_term.lref>3</broader_term.lref> <equivalent_term>metafysica</equivalent_term> <equivalent_term.lref>1187</equivalent_term.lref> <narrower_term>âme</narrower_term> <narrower_term>beauté</narrower_term> <narrower_term>cosmologie[f]</narrower_term> <narrower_term>esthétique</narrower_term> <narrower_term>mort</narrower_term> <narrower_term>vie</narrower_term> <narrower_term>vitalisme[f]</narrower_term> <narrower_term.lref>39</narrower_term.lref> <narrower_term.lref>65</narrower_term.lref> <narrower_term.lref>121</narrower_term.lref> <narrower_term.lref>63</narrower_term.lref> <narrower_term.lref>83</narrower_term.lref> <narrower_term.lref>84</narrower_term.lref> <narrower_term.lref>35152</narrower_term.lref> <priref>2</priref> <term> <value lang="">métaphysique</value> </term> <term.code>fr</term.code> </record> . . . </recordlist> <diagnostic> <hits>98037</hits> <xmltype>grouped</xmltype> <first_item>1</first_item> <search>all</search> <sort/> <limit>10</limit> <hits_on_display>10</hits_on_display> <response_time unit="ms" culture="nl-be">2850,285</response_time> <xml_creation_time unit="ms" culture="nl-be">35,0035</xml_creation_time> <link_resolve_time unit="ms" culture="nl-be">25,0025</link_resolve_time> <dbname>thesau</dbname> <dsname/> <cgistring> <param name="database">thesau</param> </cgistring> </diagnostic> </adlibxml> the xml returned for the query consist in two parts. the first part is the recordlist element that will be either empty or composed of one or multiple record elements, which are the data needed. the second part is the diagnostic element that contains various information related to the execution of the query on adlib xplus: the response time, the total number of results, the query, etc. the entire result of this query was stored in a file after processing it with the python module xmltodict [12] and some basic modifications such as normalizing the keys of the returned dictionary, or the deletion of useless information. the json format was chosen because it is easier to work with and is the default format used by catmandu [13], the tool chosen to transform our data into skos. data conversion the json file containing the entire thesaurus needs to be transformed into skos so that it can be imported into an rdf graph and be queried easily with sparql. to do the conversion, catmandu was used. it is a specialized tool aimed at facilitating access and conversions of data from different formats and origins. before diving into the technical stuff, here is a short explanation of the approach used: a skos representation of the data is required to be easily imported in an rdf graph; the specific serialization format is not relevant as the chosen tools are able to work as easily with rdf/xml, rdf/turtle or any other rdf serialization format. rdf/turtle was chosen because it is more compact and more human-readable; a template [14] will be applied to transform each record into an rdf/turtle representation; the result of this processing will be saved into a file for later use. so, consider the following data as input in json: [ { "priref": 1, "term": "astronomie", "broader_term_lref": [ 2 ], "narrower_term_lref": [ 3 ] }, { "priref": 2, "term": "sciences naturelles", "narrower_term_lref": [ 1 ] }, { "priref": 3, "term": "astrolabe", "broader_term_lref": [ 1 ] } ] the following result in rdf/turtle is expected: @prefix skos: <http://www.w3.org/2004/02/skos/core#> . @prefix thesau: <http://hescida.kikirpa.be/thesau/skos/term#> . thesau:1 a skos:concept ; skos:broader thesau:2 ; skos:narrower thesau:3 ; skos:preflabel "astronomie" . thesau:2 a skos:concept ; skos:narrower thesau:1 ; skos:preflabel "sciences naturelles" . thesau:3 a skos:concept ; skos:broader thesau:1 ; skos:preflabel "astrolabe" . this was achieved by catmandu using the following command: catmandu convert json --file ../data/thesau.json to template --template_before skos_turtle-before.tt --template skos_turtle.tt --file thesau_skos.ttl and the explanation of this command: convert is the command provided by catmandu that allow us to specify format to use via an architecture of importer and exporter plugins: json: the importer to use; –file ../data/thesau.json: a parameter to locate the json file to process; in our case, this is the export of the thesau database as mentioned previously; to: a part of the convert command letting us specify the name of the exporter to use; template: the exporter to use; as the name suggest, text templates will be used to transform our data into another form; two templates will be used: one will be a preamble and will be called only once, and the other will be called for each record in the data; –template_before skos_turtle-before.tt to specify the template to apply before the beginning of the processing; –template skos_turtle.tt to specify the template file to apply to each record; –file thesau_skos.ttl to specify to the template exporter where to store the result of the processing. in our case, a file named thesau_skos.ttl. here is the content of the template toolkit [15] templates used. firstly, the skos_turtle-before.tt template used as a header for the result file, in which prefixes are defined that will be used later when creating skos concepts. these are respectively: @prefix skos: <http://www.w3.org/2004/02/skos/core#> . @prefix thesau: <http://hescida.kikirpa.be/thesau/skos/term#> . secondly, the skos_turtle.tt template that was used to convert each json object into turtle representation. here is the content of the template: [% if priref.0 %] thesau:[% priref.0 %] a skos:concept ; [%foreach bterm in broader_term_lref %] skos:broader thesau:[% bterm %] ; [%end %] [%foreach nterm in narrower_term_lref %] skos:narrower thesau:[% nterm %] ; [%end %] [%foreach eterm in equivalent_term_lref %] skos:exactmatch thesau:[% eterm %] ; [%end %] [%foreach rterm in related_term_lref %] skos:relatedmatch thesau:[% rterm %] ; [%end %] [%if term_code %] skos:preflabel "[% term.0.dquote %]"@[% term_code.0 %] . [%else %] skos:preflabel "[% term.0.dquote %]" . [%end %] [%end %] elasticsearch configuration & the notion of elasticsearch synonyms elasticsearch is positioned as a search engine accessible via a restful api. the index configuration allows for extremely detailed control to offer a rich and flexible search experience for users. in this article, one of the configuration possibilities of elasticsearch is of particular interest to us: the concept of synonyms [16]. this works as follows: when documents are submitted to elasticsearch, the latter will apply several classic search engine treatments. one of these processes is tokenization: the production of strings (tokens) that will be integrated into the inverted index, the heart of any search engine. synonyms are used as a token filter [17] and allow the interpolation of values. this is useful when variations of writing the same concept existed, for example: ipod, i-pod, i pod. in this situation, a list of correspondences is given that will match any of the entries in the list. elasticsearch allows us to structure the matching list in two ways. the first is to write the values on a line, separated by commas. this method, called synonym equivalences, means that all the values will be used in the index, and will thus allow finding results using any of the values. in this article, that solution is used for bilingual synonyms that will give us the multilanguage search feature. the second way to structure the list of synonyms is to use the syntax of explicit matchings. here is an example of this syntax: sea cookie, sea biscuit => seabiscuit. this way, the values on the left side of the => are replaced by the values to the right of the symbol. this syntax is used for hierarchical synonyms that will give us the broaden search feature. generating bilingual synonyms once the concepts in the json file have been converted to the rdf/turtle format, the necessary configuration for elasticsearch can be generated. first, the equivalences between languages are generated to provide the multilanguage search functionality. to achieve this, a graph was built with rdflib [18], the turtle data were imported, a sparql query was submitted and finally search results were processed to produce the file in the format required by elasticsearch, namely “term1,term2”. here is the excerpt of the script that generates our list of bilingual synonyms: def generate_bilingual(graph, output): qres = graph.query(""" prefix skos: <http://www.w3.org/2004/02/skos/core#> prefix thesau: <http://hescida.kikirpa.be/thesau/skos/term#> select ?label ?equivalent where { ?term skos:preflabel ?label ; skos:exactmatch ?concept . ?concept skos:preflabel ?equivalent . filter((lang(?label) = "fr" && lang(?equivalent) = "nl") || (lang(?label) = "nl" && lang(?equivalent) = "fr")) . } """) seen = {} with path(output).open(mode="w") as outfile: for row in qres: term1 = synonym_cleaning(row[0]) term2 = synonym_cleaning(row[1]) if term1 != term2 and term1 not in seen and term2 not in seen: outfile.write(f"{str(term1)},{str(term2)}\n") seen.update({ term1: 1, term2: 1 }) the complete script is available on the companion github repository of this article [19]. here are some explanations on the main aspects: the sparql query to obtain the list of linguistic equivalences is as follows: prefix skos: <http://www.w3.org/2004/02/skos/core#> prefix thesau: <http://hescida.kikirpa.be/thesau/skos/term#> select ?label ?equivalent where { ?term skos:preflabel ?label ; skos:exactmatch ?concept . ?concept skos:preflabel ?equivalent . filter( (lang(?label) = "fr" && lang(?equivalent) = "nl") || (lang(?label) = "nl" && lang(?equivalent) = "fr") ) . } currently, queries are explicitly limited to french and dutch concepts with the filter part of the query; the synonym_cleaning() function is used to remove any syntax elements that are forbidden by elasticsearch in a list of synonyms. the following cleaning operations are performed: lowercasing, deletion of commas, square brackets and slashes; the result of this processing is stored in a file for later use. generating hierarchical synonyms to produce hierarchical synonyms, the same approach as for bilingual synonyms is adopted. in fact, the script is almost identical, except for the data generation function, presented below: def generate_hierarchical(graph, output): h = dict() qres = graph.query(""" prefix skos: <http://www.w3.org/2004/02/skos/core#> prefix thesau: <http://hescida.kikirpa.be/thesau/skos/term#> select ?term ?preflab (count(?middle)-1 as ?distance) where { ?category skos:preflabel ?term ; skos:narrower* ?middle . ?middle skos:narrower* ?concept . ?concept skos:preflabel ?preflab . } group by ?term ?preflab order by ?term ?preflab """) print(f "we have {len(qres)} results when processing narrower terms.") for row in qres: if 0 < int(row[2]) < 4: term1 = synonym_cleaning(row[0]) term2 = synonym_cleaning(row[1]) h.setdefault(term1, {"narrower": []})["narrower"].append(term2) with path(output).open(mode="w") as outfile: for k in h: terms = { term: none for term in list([ *h[k].get('narrower', []), k ]) } if len(terms.keys()) > 1: outfile.write(f"{k} => {', '.join(terms.keys())}") obviously, the sparql query is different since specific terms are needed, as well as an indication of the distance of the specific term from the concept being processed. this information is obtained by adding a temporary variable, ?middle, which is counted to get the distance between the concept (?term) and the specific term (?preflab). in the second part of the function the result is processed. firstly, only specific terms with a distance lower than 4 are retained. and secondly the result is formatted as required for elasticsearch’s synonyms syntax. this creates rows in the format “concept => specific term 1, specific term 2, concept”. this is also available on the companion github repository of this article [19]. integrating the configurations into elasticsearch to integrate the synonyms in elasticsearch, elasticsearch_dsl [20] is used as a high-level wrapper around elasticsearch, which makes it possible to create simple python classes to execute queries against elasticsearch. and in those classes, it is possible to integrate the bilingual and hierarchical synonyms as custom analyzers. in the context of elasticsearch, an analyzer [21] is the set of rules to apply when a text is process during the indexing or searching phases. it is possible to configure how tokens are produced, normalized or filtered. in our cases, synonyms are applied as a token filter. here is an example: from elasticsearch_dsl import ( analyzer, date, document, keyword, normalizer, text ) class example(document): title = text( analyzer=text_analysis, fields={ "bsyn": text(analyzer=bsynonym), "hsyn": text(analyzer=hsynonym), "raw": keyword() } ) class index: name = "example" the example class inherits from the document class defined in elasticsearch_dsl. within this class, an attribute title is defined that is configured as a text field whose analysis will be done with text_analysis. three fields are defined: bsyn for bilingual synonyms, where a text field is defined the bsynonym analyzer; hsyn for hierarchical synonyms, where a text field is defined using the analyzer hsynonym; raw which is defined as a keyword field, thus a direct copy of the data. next, text_analysis also needs to be defined: text_analysis = analyzer( "text_analysis", tokenizer="lowercase" ) a tokenizer is used to convert the tokens to lowercase. as a reminder, when processing synonyms, the terms were also transformed in the same way when generating elasticsearch configurations. finally, the analyzers for the bilingual and hierarchical synonyms are defined. both require a token filter that uses the list of synonyms (respectively bsynonym and hsynonym). the code for the filters and analyzers is shown below: bsynonym_filter = token_filter( "bsynonym_custom_filter", type="synonym", synonyms=bsynonyms ) bsynonym = analyzer( "bsynonym_custom", tokenizer="letter", filter=[ "lowercase", bsynonym_filter ] ) hsynonym_filter = token_filter( "hsynonym_custom_filter", type="synonym", synonyms=hsynonyms ) hsynonym = analyzer( "hsynonym_custom", tokenizer="standard", filter=[ "lowercase", hsynonym_filter ] ) when the elasticsearch index is created and configured, queries are submitted as follows: example = example() results = example.search().query( q("simple_query_string", query="query string") ) it is also possible to weight the fields and subfields: example = example() results = example.search().query( q( "simple_query_string", query="query string", fields=[ 'title.bsyn^4', 'title.hsyn^2', 'title.raw^2' ] ) ) where the ^ is the prefix to the ponderation to use for this specific field or subfield. first results and conclusion the whole process can be summarized by the following diagram: figure 1. summary of the workflow used to configure elasticsearch based on our internal thesaurus. based on the 98,037 entries of the internal thesaurus, 12,312 bilingual synonyms and 9,132 hierarchical synonyms were produced to configure elasticsearch. a prototype was developed to collect feedback from our internal users about the search experience. here is a screenshot of that prototype: figure 2. screenshot of the application developed to get feedback from our internal users. that prototype is simply a web application querying the photo library indexed in elasticsearch, which is configured with the generated list of synonyms. while this experiment is still in progress, some highlights can already be highlighted. first, the multilanguage search functionality gives the expected results. if there are differences, those can be explained, among others, by the absence of the term in the thesaurus, and thus in the configuration of elasticsearch. a possible solution would be to add these terms to the list and update the elasticsearch configuration. second, the broaden search using the narrower terms as synonyms is powerful but needs some more configuration. for this exploratory work, the complete thesaurus was used, but a more precise approach needs to be set up to avoid terms with too many narrower terms. a solution would be to analyse queries submitted to balat, and build a list of terms that need to be expanded with narrower terms. in conclusion, synonym generation for elasticsearch configuration from an existing thesaurus does not result in a perfect configuration. these configurations must be adapted according to the collection to be indexed, but also according to the users’ feedback. nevertheless, the use of the thesaurus makes it possible to exploit the work already done during the development of the thesaurus and thus to quickly have a first configuration specific to the collection to be processed. bibliography [1] the project. hescida. [internet] [accessed 25/4/2021]. available from: http://hescida.kikirpa.be/ [2] kik/irpa. [internet] [accessed 25/4/2021]. available from: http://www.kikirpa.be/ [3] balat kik-irpa. [internet] [accessed 25/4/2021]. available from: http://balat.kikirpa.be [4] adlib. axiell. [internet] [accessed 25/4/2021]. available from: https://www.axiell.com/solutions/product/adlib/ [5] recherche open source : les créateurs d’elasticsearch, de la suite elk et de kibana | elastic. [internet] [accessed 25/4/2021]. available from: https://www.elastic.co/fr/ [6] rdf 1.1 concepts and abstract syntax. [internet] [accessed 9/5/2021]. available from: https://www.w3.org/tr/rdf11-concepts/ [7] skos simple knowledge organization system reference. [internet] [accessed 9/5/2021]. available from: https://www.w3.org/tr/skos-reference/ [8] skos simple knowledge organization system primer. [internet] [accessed 6/5/2021]. available from: https://www.w3.org/tr/skos-primer/ [9] sparql 1.1 query language. [internet] [accessed 9/5/2021]. available from: https://www.w3.org/tr/sparql11-query/ [10] axiell webapi site – welcome. [internet] [accessed 25/4/2021]. available from: https://api.adlibsoft.com/ [11] axiell webapi site – search. [internet] [accessed 25/4/2021]. available from: https://api.adlibsoft.com/api/functions/search [12] blech m. 2021. martinblech/xmltodict. [internet] [accessed 25/4/2021]. available from: https://github.com/martinblech/xmltodict [13] librecat/catmandu data processing toolkit. [internet] [accessed 25/4/2021]. available from: https://librecat.org/ [14] catmandu::template – catmandu modules for working with templates – metacpan.org. [internet] [accessed 25/4/2021]. available from: https://metacpan.org/pod/catmandu::template [15] template toolkit home page. [internet] [accessed 25/4/2021]. available from: http://www.template-toolkit.org/index.html. [16] synonyms guide | elastic app search documentation [7.12] | elastic. [internet]. available from: https://www.elastic.co/guide/en/app-search/current/synonyms-guide.html [17] token filter reference | elasticsearch guide [7.12] | elastic. [internet] [accessed 25/4/2021]. available from: https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-tokenfilters.html [18] rdflib 5.0.0 — rdflib 5.0.0 documentation. [internet]. available from: https://rdflib.readthedocs.io/en/stable/ [19] kikirpa/thesaurus_elasticsearch_c4lj. 2021. royal institute for cultural heritage (kik/irpa). [internet]. available from: https://github.com/kikirpa/thesaurus_elasticsearch_c4lj [20] elasticsearch dsl — elasticsearch dsl 7.2.0 documentation. [internet] [accessed 25/4/2021]. available from: https://elasticsearch-dsl.readthedocs.io/en/latest/ [21] analyzer | elasticsearch guide [7.12] | elastic. [internet] [accessed 25/4/2021]. available from: https://www.elastic.co/guide/en/elasticsearch/reference/current/analyzer.html about the author emmanuel di pretoro holds bachelor’s degrees in library science and in computer science, as well as master’s degrees in library and information science. he worked as a librarian and was involved in the training of future librarians where he taught various courses related to cataloguing, indexing, database modelling and on other subjects related to computer literacy and library science. he gained experience with web archiving when working on the preserving online multiple information: towards a belgian strategy (promise) project. since 2020, he works as developer on the hescida project. edwin de roock. wim fremout graduated as a msc in chemistry (ghent university, 2002) and holds a phd in chemistry (ghent university, 2014). since 2003, he works as a conservation scientist in the laboratories department of the royal institute for cultural heritage (kik-ipra, brussels, belgium). he is specialised in the study and identification of pictorial layers and the characterisation of plastics and contemporary materials used in arts, applying a broad range of invasive and non-invasive techniques. in several international projects he is involved in research data management and digital access initiatives, and in the development of the digilab platform in the european research infrastructure in heritage science (e-rihs), which will provide access to digital resources in heritage science. since 2019 he coordinates the hescida project at kik-irpa that aims to create a data repository integrated in the institutional collection portal balat, which will become a local repository for digilab. erik buelinckx is scientific researcher at kik-irpa. he holds master’s degrees in information and library science, documentation and library science and in art history and archaeology. he is responsible for documentary image databases and digitization projects in the department of documentation. he has experience in european projects (dch-rp, partage plus, athenaplus, preforma, iperion, fifties in europe kaleidoscope, pagode – europeana china) and is a member of dariah.be, of the europeana members council (2nd term), of the time machine consortium. he organises and participates in several interregional and international collaborations on multilingual thesauri in the field of ch. he is involved in the belgian hescida-project (heritage science data archive, 2019-2022) in the frame of e-rihs. stephanie buyle studied history at the university of antwerp and archival science at the free university of brussels. since 2016 she was appointed as scientific researcher at the royal institute for cultural heritage (kik-irpa). her tasks mainly consisted of mapping data production and the problems associated with it. since 2019, she works on the hescida project as data manager. she is also involved within iperion-ch and iperion-hs projects (especially the topics related to research data management). véronique van der stede is an archaeologist (phd in art history and archaeology, université libre de bruxelles, 2003) and an assyriologist (master degree in assyriology, université libre de bruxelles, 1996) whose research focuses on several aspects of the mesopotamian civilization. she teaches akkadian at the “université libre de bruxelles”. director of the ulb-team of the euro-syrian excavation of tell beydar (syria), she took also part in archaeological missions in thasos (greece), chagar bazar (syria) and mleiha (uae). scientific researcher at the art and history museum (brussels) between 2014 and 2019, she was involved in the coordination of the bareo project (brain-be, belspo), an interdisciplinary research project aiming at ensuring the preservation of archaeological records produced during belgian archaeological expeditions to the near east and iran. she is currently scientific researcher at kik-irpa in the frame of the hescida-project (heritage science data archive, 2019-2022). subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – how to build an xml web client for the gold rush link resolver’s xml gateway web services layer mission editorial committee process and structure code4lib issue 6, 2009-03-30 how to build an xml web client for the gold rush link resolver’s xml gateway web services layer the gold rush link resolver (grlr) is part of a suite of programs developed by the colorado alliance of research libraries (carl) which help manage a library’s electronic resources. it contains the essential features required to perform link resolution, and comes at a substantial discount compared to other commercial link resolvers. after a comprehensive review of the available options, the library at the university of tennessee at chattanooga (utc) chose to implement gold rush over the summer of 2008. the utc library also decided to take advantage of the release of the gold rush xml gateway web services layer by the colorado alliance in the spring of 2008. this article is a case study of how the utc xml web client was built and the steps necessary to successfully deploy such a client. by brian kysela 1. introduction this article will focus on the actual steps, with code examples, taken to build the xml web client, through which i hope to demonstrate a model of good project planning to create a serviceable application.  for a brief overview of an xml gateway and link resolvers, see the recent code4lib journal article “auto-populating an ill form with the serial solutions link resolver api” by daniel talsky [1]. 2. the project requirements we began the project by drawing up a requirements document.  when we decided to implement the gold rush link resolver, we determined that, if possible, the new resolver should resemble the former vendor’s look and feel closely while allowing for the necessary transition by our users. we would introduce newer features and designs gradually. this is working well in practice. shortly after our selecting gold rush, carl announced that their xml  gateway web services layer was ready for production use. this was excellent news for us, because we wanted a maximum of flexibility in designing the look of the new link resolver.  this would allow me to build an xml client and preserve what we needed of the previous vendor’s web interface. with this in mind, it was fairly simple to write up the project requirements. without reproducing the full document, suffice it to say that the main   requirement of this project was to implement an xml web client that would allow for a gradual transition of the web interface while taking advantage of  the new xml gateway technology. 3.a the project specification the next traditional step in programming is to develop a project specification that outlines exactly what the program will do. the project specification is especially relevant here, because it is a document for which the programmer takes responsibility. the most important part of the specification is determining the scope of the project. that is, just what will the program be doing?  naturally, the specification draws on the project requirements. over several meetings, we determined how the new interface would appear to users. some parts of the design were left to be worked out when the vendor’s xml gateway could be understood in greater detail. all in all, the project specification was fairly neatly bound by what our former link resolver was capable of doing. new features would be determined at a later date. 3.b the vendor api the vendor api is fairly straightforward. following are brief summaries of a couple of methods that have a different calling syntax from the typical gold rush api. full details can be found in their xml gateway documentation [2]. only a few methods have a different calling syntax: browsejrnlsubjects and grlresolvearticle. browsejrnlsubjects as the name implies, this is a left-anchored search of journal subjects. this method only requires that a search string be sent with a query. grlresolvearticle this is the critical method for link resolution. it requires article metadata to be sent so that a proper look-up can be performed. this includes, at a minimum, the issn and journal title. 4. the minimal prototype most anyone who enjoys programming enjoys seeing results, and those as  quickly as possible.  with xml gateways, a variety of methods are open to a programmer. i have a long history of working with linux and have been using perl for as many years.  perl has an excellent library, or, more  properly called, perhaps, a module for dealing with xml called  soap::lite [3].  do not be mislead by the name–it is a heavy duty library that accomplishes much while providing the user of the library with a lightweight interface. it can be a daunting task to have 70 or so pages of vendor documentation and only a few documents that will guide a project not yet begun. i like to start simply by having something that can immediately show several things: (1) that the xml gateway is up and running; (2) that the client code i am writing functions; and (3) that the client code returns appropriate results. to properly determine all this required only a few lines of code as follows: #! /usr/bin/perl use warnings; use strict; use soap::lite; use data::dumper; $data::dumper::indent = 1; my $gw = "http://grx.coalliance.org/grx.php"; my $soap = soap::lite->proxy($gw); my $result = $soap->fulljtrec("001_aur", 26707)->result; print dumper($result); only four lines of code are needed, after line 1 that lists the interpreter, and lines 3-7 that list the required perl pragmas and modules to use in this program. line 9 notes the http address of the gold rush xml gateway. line 10 sets up the soap::lite script to be called with the gold rush gateway method ‘fulljtrec’ in line 11. the variables ‘001_aur’ and ‘26707’ are specific to the ‘fulljtrec’ method. the first is the gold rush assigned id that will identify your particular data store with the vendor. the second variable, ‘26707’, is an id for a particular resource that i happen to know will work with the data source cited. if you are not sure what would work when setting this up, a general purpose search would work as well. a generic method like the following could be substituted for line 11. my $result = $soap->browsejrnlft("001_aur", "library")->result; this method, ‘browsejrnlft’, does a browse of journals that have full text articles indexed. line 11 is really the only line that runs. the others merely set up the environment.  the last line uses the perl data::dumper module to print the results that are stored in the variable $result. lines 6 and 7 bring in this convenient library and give it a nice level of indentation when   printing. running this script will allow inspection of the results returned from the gold rush gateway. results from a search like this come back in a raw form. here is a selection from the browsejrnlft search noted above: $var1 = bless( { 'grx_error' => bless( {}, 'grxerror' ), 'brief_recs' => [ bless( { 'issn' => '0041-7912', 'jtitle' => 'lc classification, additions and changes', 'id' => '90134' }, 'browselistitem' ), bless( { 'issn' => '1756-1086', 'jtitle' => 'library & information research : research into practice for information & library services', 'id' => '312136' }, 'browselistitem' ), doesn’t look like xml, does it? we let soap::lite deal with that. the logic of what data goes where is more interesting. the data returned above has been stuffed into an easily parsed perl hash. now that we know the gateway server is up and running, we now have a client that not only works properly, but it also can return interesting results that relate to how it is called. 5. the maximum prototype and a variation once i understood how to get results out of the gold rush gateway, it was only a matter of determining what the patterns in the gold rush gateway methods were. with that information, i built a generalized prototype that could easily call any of the possible methods and display the results. notice that this prototype works from the command line and will not be related directly to the finished client. this will be invaluable in saving time in troubleshooting the finished client. in order to troubleshoot with any precision, it is necessary to know what is happening at as low a level as possible. the soap::lite module has   additional means for giving low-level information, if need be. for example, it is possible to trace many of the events or parts of soap::lite as it processes requests. at times this too proved useful in determining the causes of errors in data returned by the server. i generally used the prototype to view raw data since i was mainly interested in transforming that data. the large prototype evolved in tandem with the development of the web client. the finished prototype rendered a command line menu-driven interface to allow easy testing of all the gateway methods: enter a number for the corresponding action: [0] quit [1] browsejrnlft [2] browsejrnl [3] browsejrnlsubjects [4] browsedb [5] browsedbcategory [6] searchbyissn [7] searchbydbtitle [8] searchbyjrnltitle [9] searchbykeyword [10] searchbysubject [11] journallista2z [12] databaselista2z [13] grlresolvearticle [14] fulljtrec [15] fulldbrec [16] getdbtitlelist enter: as you can see, the menu presents the user with a simple list that can be called by number. you may also note that not all gateway methods are implemented. the methods not implemented include those that return extremely large data dumps of the client datasource. even if we had intended to use those methods, this prototype would not have been the place to run and test them. one important variation on this prototype is a web version that made the debugging of openurl links much easier. these openurl links would be sent via email, but different email clients would mangle them in different ways. it became quickly tedious to unravel these links and insert them into a proper form for loading into a browser and debugging. so, out of urgent need to retain sanity, i hacked together a url demangler and debugger. it is not the most elegant of solutions, but it works well and allows much easier debugging. 6. the web client while having a working prototype is satisfying, the real project is the working web client that patrons will eventually be using. to produce this required more than just an understanding of how the gold rush xml gateway server and a client can be built. it required thinking through how to bring what goes on in the engine, so to speak–the client/server interaction– together with the search and results screens for the users–what one might consider the controls. providing an interface that was consistent with the previous vendor’s controls was one of the most difficult parts of the project, since not all the methods of the new service mirrored exactly what we had had previously. with some ingenuity we were able to bring the functionality relatively close, but not in every case. a simple example is that a browse by title in the new interface would bring entries up that included titles indexed under an alternate title.  so if a user does a search beginning with the letter ‘b’, for example, the result set will begin with alternate titles that begin with an ‘a’. for the future, we have asked the vendor to index the results under the actual alternate title, so that these results will appear consistent. turning to the overall architecture of the web client, the results were not what we initially had in mind, but are conceptually very similar. in considering how to design this project, i was guided by a thoroughly traditional idea of design: i wanted a very simple structure which would allow me to upgrade the project software easily, while retaining configuration directives across upgrades. the solution turned out to be a simple directory structure which would allow for a modular design approach. clearly a configuration file would be needed, but it could not occupy a place where it might be overwritten during an upgrade.  coming from a linux background, this meant that the configuration file would be placed in ‘/etc’ on the web server, a typical place for configuration files in the directory hierarchy of a linux or unix server.  this turned out to be the only file outside the project directory. 7. the project directory below is a list of all the directories within the main ‘grx’ project directory: -rw-rw---1 apache web  4217 2009-01-13 16:55 install -rw-rw---1 apache web 20217 2008-07-16 12:01 license -rw-rw---1 apache web  5004 2008-12-17 16:42 readme drwxrwx--2 apache web    96 2009-01-14 15:09 bin drwxrwx--3 apache web  4096 2008-07-09 10:01 cache drwxrwx--2 apache web    43 2008-07-24 11:59 conf drwxrwx--3 apache web    29 2009-01-13 17:52 lib drwxrwx--4 apache web    28 2008-04-28 09:46 tt drwxrwx--6 apache web    61 2009-01-15 15:59 www the ‘bin’ directory holds the standalone utility programs like the full prototype for debugging, and scripts for building the cache that will be discussed further below. the ‘cache’ directory holds cache files that are accessed by the web client. since the actual configuration file is not part of this project directory, the sample configuration file is included in the conf directory. the ‘lib’ directory contains two files that are the bulk of the project: ‘lib/grx.pm’ and ‘lib/grx/resolvearticle.pm’.  these two files are the implementation of the list of methods seen above in the prototype menu.  the ‘tt’ directory represents the html template file directory. i labeled it ‘tt’ since i make use of the template::toolkit module for perl. this is where the pages to be customized for the user screens are kept. all the html files are here rather than in the last directory, ‘www’, which contains the files that the web server must have access to. those files include the programs that make everything else work, as well as the css, javascript and image directories. one of my goals while creating this project was to keep it as compact as possible. if we only consider the ‘lib’, ‘tt’ and ‘www’ directories (all that is truly needed to make everything go) the project is around 60kb in size. 8. the web directory whenever i read about a new software project, i am always most interested in how it works: what makes the wheels turn? for my project, it is very simple. the web server is set up to serve the ‘/grx/www/’ directory. on linux, using apache, i just place the project in ‘/var/www/’ and set up apache to run what it finds in there by adding the following directives to the apache httpd.conf file: alias /grx /var/www/grx/www/ <directory /var/www/grx/www/> allowoverride none directoryindex index.cgi order allow,deny allow from all options +execcgi addhandler cgi-script .cgi </directory> setting it up like this will allow the apache web server to serve both files that control what i call the ‘grx’ application: /grx/www/index.cgi /grx/www/grl/index.cgi the first handles all the browsing and searching methods for journals. the second one deals with the link resolving method. this method, lightheartedly enough, is listed under ‘miscellaneous functions’ in the gold rush xml gateway manual. this function was where i spent the bulk of my development time. if you would like to follow along at home,  i use the git version control system to manage the project and have a public git repository available with the latest development snapshot [4].  the key to getting started is the installation of several perl modules. this can be automated easily and is covered in the install file that you will find in the git repository. below is the first index.cgi file that drives the searching: #!/usr/bin/perl use warnings; use strict; use lib '../lib'; use grx; my $grx = grx->new( {   'browsejrnlft'       => 1, 'browsejrnlsubjects' => 1, 'searchbyissn'       => 1, 'searchbysubject'    => 1, 'searchbykeyword'    => 1, 'fulljtrec'          => 1, 'grlresolvearticle'  => 1, } ); $grx->run(); as you can see, it is very brief. the real work is done in the grx.pm library that is called on line 6. lines 8-17 are merely for setting up what methods can be called on the new $grx object. i implemented most of the methods, although we did not actually use them in our final design. line 19 is what makes everything go. the other cgi file, for article resolution, is even more brief: #!/usr/bin/perl use warnings; use strict; use lib '../../lib'; use grx::resolvearticle; my $gra = grx::resolvearticle->new(); $gra->run(); notice that the real work is done in the grx::resolvearticle library. when i first started building the application, i did not use an object oriented design, but quickly realized its benefits and switched over to that approach. it made the development infinitely easier. 9. the template directory the ‘tt’ directory contains two directories with only a few files in each: tt: total 0 drwxrwx--2 apache web 69 2008-12-10 09:59 lib drwxrwx--2 apache web 40 2008-12-10 11:34 pages tt/lib: total 20 -rw-rw---1 apache web 1894 2008-07-24 11:43 foot.html -rw-rw---1 apache web 4354 2008-09-22 10:37 full_jt_rec_display.html -rw-rw---1 apache web 6290 2008-06-04 15:00 head.html tt/pages: total 16 -rw-r--r-1 brian  users 4657 2008-12-09 09:52 grl.html -rw-rw---1 apache web   6971 2008-07-25 15:18 results.html the library directory (tt/lib) has parts of pages, like the header and footer, and also has one piece that turned out to be used in two different parts of the html pages. i abstracted it and put it in the library rather than duplicating the code. the second directory contains the pages of the user interface which combine with the boilerplate found in the library directory.  the ‘results.html’ page contains the search and browse interface. i conveniently had the search interface also display the results when they became available.  this cuts down on the potential proliferation of pages. the ‘grl.html’ page serves as the result, or, locally termed, the ‘landing’ page for article resolution. 10. the grx perl modules without going into great detail about each of these libraries, i will sketch out the form each one took in order to communicate something of the overall design. 10.1 the grx.pm module this library contains all the functions that control the searching. if you review the code for ‘/grx/www/index.cgi’ above, you will see that it creates a new object. this calls the following method in the grx.pm module: 24 sub new { 25     my ($invocant, $arg_ref) = @_; 26     my $self = bless( {}, ref $invocant || $invocant ); 27     $self->init($arg_ref); 28     return $self; 29 } from here the methods are passed in and then sent along to the ‘init’ method: 31 sub init { 32     my ($self, $arg_ref) = @_; 33     $self->ok_methods($arg_ref); 34 } the init method determines what methods can be called from how it was called in index.cgi. the next step, as per the cgi script in section 8 above, is to set everything moving with the ‘run’ method, which calls this function in grx.pm: 71 sub run { 72     my $self = shift; 73     $self->get_vars(); 74     $self->check_method(); 75     $self->get_results(); 76 } this function or subroutine does an excellent job at summing up the rest of the program’s execution: (1) obtain any needed variables, whether from the configuration file or sent in by a form; (2) check to make certain we have a valid method request; (3) finally, go and make a request from the gold rush xml server gateway. the get_results method calls the appropriate method. since the methods for searching are generally the same, that is, send the gold rush datasource id and the search string with the method, i created a default search routine for all but the two that have a different syntax in the gold rush xml gateway api: browsejrnlsubjects and grlresolvearticle. the default method requires the institution id string be sent with the search string. the ‘browsejrnlsubjects’ method only requires a search string. the ‘grlresolvearticle’ method requires a set of article metadata be sent. this is the extent of the gold rush xml gateway api. the last thing that is done in grx.pm is the get_results subroutine, which calls the process_template subroutine that generates the output for the web server to display to the user: 304 sub process_template { 305     my $self = shift; 306     my $path = $self->grx_path(); 307 308     print header(); # a cgi.pm method 309     $| = 1; # turn buffering off 310 311     my $template = template->new( 312         { include_path => "$path/tt/lib:$path/tt/pages" } ); 313     $template->process( $self->tt_file(), $self->tt_vars() ) || die $template->error(); 314     return 1; 315 } this subroutine is fairly short and straightforward. the important line is line 313, which calls the template::toolkit method ‘process’ that actually processes the template files in the tt/lib and tt/pages directory discussed above. 10.2 the resolvearticle.pm module the resolvearticle.pm module is strikingly similar to the previous module, having ‘new’, ‘run’, ‘get_vars’ subroutines and, in this case, one method call: grlresolvearticle. since there is some bit of overlap between the two modules, i will concentrate on explaining the grlresolvearticle method call which is unique to this module: 147 sub grlresolvearticle { 148     my $self = shift; 149 150     my @vars = ( $self->grx_id(), $self->grl_ref() ); 151 152     eval { 153         $self->grx_result( 154             $self->soap_script()->grlresolvearticle(@vars)->result ); 155     }; 156     if ($@) { 157         $self->error_string($@); 158         $self->do_error(); 159     } 160     elsif ( !defined $self->grx_result()->{'citation'} ) { 161         my $cite = get_citation($self); 162         $self->tt_vars( { 'result' => $cite } ); 163         $self->tt_file('grl.html'); 164 165         if ( $self->email_errors() ) { 166             my $subject = $self->subject_line() . " " . $self->remote_host(); 167             my $message = $self->url(-full=>1, -query=>1) . "\n\n" . $self->referer(); 168             $self->mail_it($subject, $message); 169         } 170         return 1; 171     } 172     else { 173         $self->tt_vars( { 'result' => $self->grx_result(), } ); 174         $self->tt_file('grl.html'); 175         return 1; 176     } 177 } this is the same pattern as before: retrieve variables, the datasource id and the search parameter. this is done on line 150. line 154 makes the request to the gold rush xml server. this is wrapped in a perl eval to make sure it works; if not, it throws an error. otherwise it continues, obtaining a citation if the xml gateway does not provide one on line 161. (this had shown up during development and was used to work around a problem that was later solved. i left it in just in case it should ever turn up again.) a configuration option allows  me to set whether it will throw an error and who will receive it if this does happen. template variables are set up on lines 162-163. after the grlresolvearticle method runs, the results are displayed by calling the process_template subroutine contained in the grx.pm module, since we use that module as a base to this one. this means we can call its methods from this perl library, and conveniently, even override them as we like. 11. conclusion with a minimum of code i was able to implement an xml web client that provided the ability to customize the utc link resolver interface as needed during the transition from the previous link resolver. all the program logic is found in two library files, while the appearance is updated by modifying the template files. this design provides a clean separation between program logic and presentation. the flexible design also allows for ease of maintenance and seamless upgrading of the code base. in closing, i would like to emphasize that the project grew organically, but was structured according to the project requirements and specification. i placed special emphasis on having good debugging tools close at hand and creating a design that would allow me to not only share the code, but also to drastically change any one aspect without affecting any other part of the code base unnecessarily. keeping the design simple, and the program short was also key. licensing information with respect to the code, i offer it under the same terms as perl itself.  there is a license file included with the distributed code for further reference. about the author brian kysela is the web technologies librarian at the university of tennessee at chattanooga. —footnotes [1] http://journal.code4lib.org/articles/108 [2] http://grx.coalliance.org/grxtest/dsp/grx_documentation.pdf [3] http://search.cpan.org/~byrne/soap-lite/lib/soap/lite.pm [4] http://www5.lib.utc.edu/pub/scm/grx/core/git/bkysela/grx.git/ subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – integrating process management with archival management systems: lessons learned mission editorial committee process and structure code4lib issue 6, 2009-03-30 integrating process management with archival management systems: lessons learned the integrated digital special collections (indi) system is a prototype of a database-driven, web application designed to automate and manage archival workflow for large institutions and consortia. this article discusses the how the indi project enabled the successful implementation of a process to manage large technology projects in the harold b. lee library at brigham young university. it highlights how the scope of these technology projects is set and how the major deliverables for each project are defined. the article also talks about how the indi system followed the process and still failed to be completed. it examines why the process itself is successful and why the indi project failed. it further underscores the importance of process management in archival management systems. introduction archivists are constantly looking for ways to better manage the resources that they are responsible for. the integrated digital special collections (indi) was envisioned as a prototype of a database-driven web application to automate and manage archival workflow for large institutions and consortia. this article discusses how the indi project enabled the successful implementation of a process for managing large technology projects in the harold b. lee library at brigham young university. it highlights how the scope of these technology projects is set and how the major deliverables for each project are defined. the article also talks about how the indi project followed the process and still failed to be completed. it examines why the process itself is successful and why the indi project failed. it further underscores the importance of process management in archival management systems. the indi project developed out of ongoing efforts to improve the management of archival and manuscript resources in the l. tom perry special collections (hereafter perry special collections) at brigham young university. the perry special collections is a mid-sized repository that makes available primary source materials (including rare books, manuscripts, photographs, folklore, and corporate archives) to students, faculty and other researchers at brigham young university. the perry special collections employs twelve full-time curators with responsibility for a variety of collecting areas ranging from arts and communications to 21st century utah history. several paraprofessionals help us with processing manuscript collections and managing our workflow. a large number of students also help process collections, provide reference service and perform a variety of other tasks. in order to maximize efficiency the staff in the perry special collections developed a distributed workflow designed to leverage the unique abilities and different levels of training of our employees. this workflow segmented traditional archival processes and assigned each segment to specific individuals based on their training and experience. let’s examine how this workflow occurs with a manuscript acquisition. the manuscript curator contacts a potential donor and works with them to facilitate the transfer of their materials to special collections. once the material is physically in our custody, the curator creates a case file that tracks the various steps that occur before a collection is placed in our stacks. the curator makes sure that a deed of gift or other transfer instrument is obtained and that the donor has been appropriately acknowledged. the curator then assigns a processing student to accession the material, minimally process it, and create a basic finding aid (either a basic inventory or a catalog record) in xml utilizing the encoded archival description (ead) dtd. the student then takes the case file to our workflows technician (another student employee). the workflows technician reviews the case file checklist to make sure that all of the necessary actions have been taken by the curator and the processing student. the technician also performs an editorial check of the finding aid. the technician then retrieves the collection and makes sure that it is labeled, bar-coded and placed in the stacks. the technician then passes the case file off to our manuscripts cataloger so that a collection catalog record can be entered into the library’s on-line catalog. the technician also makes sure that the ead file is sent to the appropriate individual so that it can be placed on the world wide web. if conservation work, digitization work or microfilming is necessary, additional individuals become involved in the process. project background in its earliest incarnation indi was a simple automation of this paper-based workflow designed to ensure that established procedures were followed in preparing archival and manuscript collections for use by our research base. we quickly realized that the automation of a paper-based workflow would not meet our underlying needs. these needs included the ability to manage a series of tasks that occur for each archival or manuscript collection, the need to manage the distribution of these tasks among several different contributors before the collection was considered finished and made available to researchers, and the need to ensure that each of these tasks was completed properly as a large portion of them were completed by student employees and interns. we also needed to develop a system that would allow us to work more cooperatively with the other two campuses of the brigham young university system.[1] finally, we needed a system that would gather all of the information that we maintained about our archival and manuscript collections into one central location.  we needed to eliminate the various standalone databases used in the department to accession items, to create encoded archival description (ead) tagged finding aids, to manage location information, and to handle other associated tasks. these standalone databases were not only unsupported by our library’s information technology unit, but they also fostered the redundant gathering of information and wasted time that could be spend on other activities. once these needs had been articulated in mid-2005, a project team was formed in the perry special collections to determine whether or not there were any open source systems for the management of archival materials. the project team discovered that there was only one open source tool for archival management under development: the archivist’s toolkit.[2] the project team carefully analyzed the documentation for this application and found that many of our needs would be met by the completed version of this project. however, it would not meet what was emerging as one of our most important needs—managing our distributed workflow. given this information, the project team recommended to the chair of the perry special collections that we approach our library administration with a proposal to build a system internally that would meet all of our needs. the department chair took this proposal to the library administration and we received permission to assemble a new project team and begin planning in september of 2005. the project team was comprised of subject specialists from the perry special collections and technology experts from the library information technology (lit) division. the indi project was the largest programming effort that the lit division of harold b. lee library had undertaken and they recommended that we utilize project management processes to manage the project. project management provided us with processes and techniques for defining the project scope, estimating the resources necessary to complete the project, tracking our progress, and developing adequate documentation. the first step in this process is project definition. project definition has three components: defining the project objective, creating a flexibility matrix, and the identification of appropriate team members. our project objective was to “streamline and improve special collections (sc) workflow processes and integrate sc best practices into a workflow database system.” the flexibility matrix forced us to decide what our constraints were in building the system. questions that had to be answered included: is the scope of the project the most flexible, are there sufficient resources available and are these resources limited or expandable, and what deadlines are associated with the project-are they firm or flexible? after much deliberation we decided that our scope was the most flexible, our schedule would allow us a little wiggle room and that the resources available to us were unlikely to change. finally, we decided that the project team would be composed of lit staff, special collections staff, and non-departmental staff. once we completed the project definition, we turned to developing requirements for the indi system. we used the “is/is not” process to identify and define the major deliverables for the indi system. the is/is not process works like this in the harold b. lee library. giant sticky notes are placed on the walls of the meeting location. each sticky note has one major deliverable written on it and then two columns: is and is not. team members are given small sticky notes and then write what they think the deliverable should be and place it in the appropriate column. if they are not sure which column to place an item in, then the sticky note is placed on the line dividing the columns for further discussion by the project team. eventually all of the sticky notes must be placed in either the is column or the is not column. this process played out over the course of several meetings. the major deliverables included the core functionality of the indi system and the various modules that would allow us to complete important archival tasks. they also included the ability to manage archival business processes. once the major deliverables were identified we had to define each of them. we utilized the is/is not process to do this as well. the most important of the major deliverables defined the core functionality of indi and the series of modules that would enable curators, paraprofessionals, and students to complete their work. it was during this phase that it became clear that one of the most important deliverables of the project was the ability of the indi system to manage business processes. we were particularly interested in managing workflows. workflows are a very specific type of business process and can be defined as business processes that deliver services or informational products. by integrating workflow management into the indi system, the system would deliver the right piece of work to the right resource at the right time. this would enable us to achieve maximum efficiency in our archival processes. workflow management would enable us to define specific tasks in the system and associate those tasks with specific resources that had specific roles. this would allow us to fully manage our distributed workflow and it would give us the ability to monitor key indicators to ensure that we were meeting our established goals. integrating workflow management directly into the indi system would also enable us to avoid having to rekey data multiple times in multiple systems or having to programmatically transfer data from one system to the next. finally, integration would allow us to monitor who was completing which tasks and design workflow logic that would help the system know who to give a task to once the previous task was completed. based on the information gathered during the definition of the major deliverables, we decided to divide the project into a number of phases that would be completed sequentially. we also determined that the is/is not process would be repeated for each module in order to help us carefully define them. phase i was to consist of pre-arrangement (acquisition and accessioning, location guide, import process, reports, workflow management, and contacts management) while phase ii was to consist of arrangement and description (finding aid creation, catalog record creation, metadata creation, etc.). business process management was to be built into the underlying architecture of the system and was included in phase i. once the major deliverables were defined the project team turned to creating documentation to help the programmer understand how archival processes worked within the perry special collections. a number of workflow diagrams were created to graphically illustrate the archival processes and the steps that were required to complete each process (see figure 1). these workflow diagrams were supplemented by textual descriptions of the processes. these materials were handed over to the programmer assigned to the project and he compiled them into the first version of the indi design document. the design document described the overall project and the inter-relationship between the core functionality and the various modules. the documentation process took most of the calendar year 2006. figure 1. acquisition and accession workflow diagram (full-size image). with the completion of version one of the design document, the programmer began to program the indi system using a pre-existing php framework of his own design. the core functionality of indi was completed and presented to the indi committee for review during fall semester 2006. the committee requested several changes to the main indi tools and recommended that the pre-arrangement module be separated out into component parts to be built individually. the programmer took this information and went to work building a prototype.  there was very little communication between the project team and the programmer for several months while the prototype was being built. the programmer presented the indi prototype to the committee in late march 2007 and the committee was satisfied enough with the results to recommend internal testing. the prototype included the three core indi tools and the acquisition and accession module. several curators and students were assigned to test the prototype vigorously between april 2007 and august 2007. the curators and students testing the prototype identified several minor bugs which were reported to the programmer and fixed. in august 2007 the indi committee recommended that the prototype be released for production use in the perry special collections and that planning begin for the programming of the arrangement and description module. although not perfect, the project management process that the indi committee followed was extremely useful and helped the project team successfully create enough documentation for the programmer to begin building the indi system. the process of defining the entire project and its component parts enabled team members to gain a better understanding of what we needed from an archival management system. the process utilized to define the indi project has been adapted for use by another ongoing technology project in the perry special collections. the critical problem with indi was not process-based but rather technology-based, a fact that wasn’t discovered until after the indi prototype had been released. system features at the time of its implementation in the perry special collections, most of the core functionality of the system was complete. this included three main tools: a contact manager, a task manager, and a search application. these were built as plug-ins to the underlying rich internet application (ria) user interface (ui) used to develop the system. the system also took advantage of other features of the ui, though these were not the focus of our development work. at the base of the system is sierra-php[3], a php framework developed by our programmer prior to his employment at the library. sierra-php is an open source project that provides indi with web services, authentication, database abstraction, and other features. it also provides workflow management functionality, using xml descriptor files. these descriptions (as shown in sample 1) included data entry, data validation, and task assignment based on decision points in the workflow. <?xml version="1.0" encoding="iso-8859-1"?> <!doctype workflow public "-//sierra//dtd sierra workflow//en" "http://sierra-php.googlecode.com/svn/trunk/lib/workflow/workflow.dtd"> <workflow due-date="+0-+0-+30" notify-from="indi@lib.byu.edu" resource="prearrangement.type" resources="etc/plugins/indi/l10n/pre-arrangement etc/plugins/indi/l10n/indi" role-attr="name" role-attr-members="members" role-entity="osgroup" start="startprearrangement" user-attr="username" user-attr-displ="name" user-attr-email="email" user-entity="osuser" user-key="user" user-key-type="global"> <!-acquisition information --> <step key="accessioninfo" entity="accession" entity-id="accession" next="commitaccession" resource="step.accessioninfo"> <decision next="bocreviewform"> <!-boc approved !== true --> <constraint attr="bocapproved" operator="832" /> <!-boc approval applies only to byu --> <constraint entity-id="collection" attr="institution" value="byu" /> <!-criteria for boc approval --> <constraint-group connective="or"> <!-changes to the contract --> <constraint attr="changestocontract" operator="64" /> <!-value gt $1000 --> <constraint attr="value" operator="2" value="1000" /> <!-size is gt 10 linear feet and material type is not ua --> <constraint-group> <constraint attr="linearfeet" operator="2" value="10" /> <constraint attr="type" entity-id="collection" operator="257" value="ua" /> </constraint-group> </constraint-group> </decision> <task resource="task.descriptivesummaryform" validate="1" view="prearrangementaccessionform" /> <task resource="task.acqinfoform" validate="prearrangementacquisitionform" view="prearrangementacquisitionform"> <eval-before><!&#91;cdata&#91; // first assign collection to entity $data = array(); $data&#91;'collection'&#93; =& $collection; $entity->updateentity($data); return true; &#93;&#93;></eval-before> </task> <task resource="task.mergeappraisal"> <eval-before><!&#91;cdata&#91; // first assign collection to entity if ($appraisal) { sra_util::mergeobject($accession, $appraisal); $entity->setentity($accession); $entity->update(); } return true; &#93;&#93;></eval-before> </task> </step> ... sample 1. xml description of the acquisition and accession workflow. descriptive xml files were also used to define the plug-in application data model and to generate forms for data entry (see sample 2). ... <view key="prearrangementaccessionform" template="model/sra-grid.tpl"> <param id="class" type="table-attrs" value="indiformtable"/> <param id="colsetlabelclass" type="col-set-config" value="mycontactshelplink"/> <param id="colsetlabelonclick" type="col-set-config" value="core_helpmanager.load('indi', 'accessioninghelp', '&#91;entity&#93;.&#91;attr&#93;')"/> <param id="colsetformat" type="col-set-config1-0" value="horz"/> <param id="title" type="col-set1-0" value="input"/> <param id="collectionarea" type="col-set1-0" value="input"/> <param id="colsetformat" type="col-set-config2-0" value="horz"/> <param id="colsetwidth" type="col-set-config2-0" value="2"/> <param id="creators" type="col-set2-0" value="input"/> <param id="colsetformat" type="col-set-config3-0" value="horz"/> <param id="linearfeet" type="col-set3-0" value="input"/> <param id="volumes" type="col-set3-0" value="input"/> <param id="colsetformat" type="col-set-config4-0" value="horz"/> <param id="spandatestart" type="col-set4-0" value="input"/> <param id="spandateend" type="col-set4-0" value="input"/> <param id="colsetformat" type="col-set-config5-0" value="horz"/> <param id="spandatestartformat" type="col-set5-0" value="input"/> <param id="spandateendformat" type="col-set5-0" value="input"/> <param id="colsetformat" type="col-set-config6-0" value="horz"/> <param id="bulkdatestart" type="col-set6-0" value="input"/> <param id="bulkdateend" type="col-set6-0" value="input"/> <param id="colsetformat" type="col-set-config7-0" value="horz"/> <param id="bulkdatestartformat" type="col-set7-0" value="input"/> <param id="bulkdateendformat" type="col-set7-0" value="input"/> <param id="colsetformat" type="col-set-config8-0" value="horz"/> <param id="colsetwidth" type="col-set-config8-0" value="2"/> <param id="materialcondition" type="col-set8-0" value="input"/> <param id="colsetformat" type="col-set-config9-0" value="horz"/> <param id="colsetwidth" type="col-set-config9-0" value="2"/> <param id="intellectualdescription" type="col-set9-0" value="input"/> <param id="colsetformat" type="col-set-config10-0" value="horz"/> <param id="colsetwidth" type="col-set-config10-0" value="2"/> <param id="restrictions" type="col-set10-0" value="input"/> </view> ... sample 2. xml definition of accession data form this framework is overlaid by sierra-os[4], a php-based ria ui built as part of the indi development process. sierra-os provides user management functions to the system, as well as windowing and search functionality. sierra-os was designed to allow developers to build applications as plug-ins, which could both extend system features and leverage existing functionality. the system features a core plug-in, which includes a federated search tool, spell checker, help manual, and unix-style terminal (see figure 2). indi itself was built as a plug-in application to sierra-os. figure 2. terminal component of the sierra-os core plug-in the first of the tools included in the indi plug-in was indi desktop, designed for searching and displaying accession data. this tool works independently of the federated search, and queries information associated with an accession rather than with associated projects. indi desktop allows for fielded searches, as well general keyword and saved searches (see figure 3). as part of its display, the application provides summary information together with the status of pending tasks. figure 3. indi desktop search results the second tool, myprojects, is a project management tool, tracking and providing access to tasks defined in a workflow. in this project management approach, each workflow is treated as a project and managed under that paradigm. tasks are assigned to users sequentially as defined in the xml workflow descriptor, and completion time is tracked. myproject provides views of project information, including discussions, tasks, and files associated with projects. its dashboard view tracks latest activity, as well as tasks that are upcoming or late (see figure 4). while data entry forms and the information they hold are managed separately by the system, myprojects provides users access to them. in order to further improve access to project data, the myprojects tool also includes search and serialization services. basic and advanced search capabilities are included, allowing users to complete keyword and fielded searches of project data. users may also save search queries, which can then be used to generate rss and icalendar feeds. figure 4. myprojects tool (full-size image). the third tool included in the indi plug-in is mycontacts, a contact management tool. based on the vcard standard, and supplemented with fields from the encoded archival context (eac) beta, mycontacts manages information about individuals and corporate bodies that can then be associated with various projects (see figure 5). contacts can also be associated with groups, either for management by the user or based on system linkages. the system also allows users to define relationships between contacts, creating a rich web of references between entries. figure 5. view of vcard data in the mycontacts tool (full-size image). using the functionality of the indi plug-in tools, we have implemented two workflows: appraisal, and acquisition and accession. the appraisal workflow includes tasks involved in assessing potential acquisitions, such as documenting collection information and appraisal decisions. the acquisition and accession workflow walks users through the process of acquiring materials and documenting their arrival at the institution. planning has been completed for an arrangement and description workflow, and preliminary plans were in place for other managing other related archival activities (collection management, digitization, conservation, etc.). project termination as the core development of indi was coming to a close the project began to encounter problems. in september 2007, our programmer announced that he was taking another position effective the end of october and that he would be leaving the lee library it staff.  a new programmer was hired quickly and worked closely with the departing programmer for about a month to learn how to continue the indi development. together they built the appraisal workflow, which was tested and approved by the indi project team. unfortunately, it quickly became apparent after the original indi programmer left in late october that the indi system was so complex that the new programmer would need between six to nine months to begin to understand how to support the system and it wasn’t clear if the programmer would be able to continue development utilizing the existing indi framework. between november 2007 and march 2008 the indi committee and the staff of the perry special collections identified several bugs in the indi system and reported them to the programmer. the programmer spent a lot of time on trying to fix these bugs and on several occasions the library it unit had to contract with the previous programmer to fix bugs. very little time was spent on programming additional components of the indi system. by the end of april, due to the inability of the new programmer to further develop the existing system, our it department recommended halting work on the existing code base and beginning planning for a replacement. the existing indi system would be maintained until the new application was complete and the data was migrated. nevertheless, to meet our original project goals, the committee completed basic user documentation for the existing system, and the source code for the project was made available to the archival community in august 2008.[5] lessons learned during the course of developing indi, the most significant lesson learned by the project team was the need for adequate documentation throughout the development process. when the project was begun the committee members had never been involved in a large-scale software development project, and did not understand the level of documentation needed to support that effort long-term. on the committee’s side, critical documentation to guide the programmer’s efforts was lacking. the initial design documents primarily consisted of the project definition and workflow diagrams. no functional requirements document or software requirements specification were provided, leaving the programmer unclear what system features were needed to meet the project’s goals. this directly contributed to the second programmer’s inability to extend the application as the committee hoped, as critical features were not included in the underlying framework. however, after the departure of our initial programmer it became clear that there was also inadequate documentation for the existing code base as well. while the decision to build indi on our programmer’s own php framework reduced development time significantly early on, the lack of easy documentation for the system hindered further development when he left the project. in order to learn the system, the new programmer had to rely on the code itself—reading in-line comments in the code and xml dtds—to understand the system’s structure and functions. it also became clear that there was a greater need for communication between the different parts of the project team. throughout the project there were disagreements over the development schedule, which were compounded by a lack of communication between the it staff and the subject specialists. decisions about the development of the system, such as the use of sierra-php and the temporary use of a proprietary xsl-fo processor, were also made by programming staff without consultation with the rest of the committee. while these development choices were clearly within their purview, the ramifications of these decisions ultimately undermined the project. an open discussion with the committee about how the system was being developed might have led to a different solution. developing the indi system also helped us learn the importance of design in software development. in the course of user testing of the completed indi system, it became clear that there were many aspects of the interface design that our staff found confusing. much of this came from the project management concepts used throughout the application, and the flexibility that it provided. in the end, the system did far more than our staff wanted or needed it to do. finally, we came to understand the need to plan for sustainability. throughout the development cycle concerns about the long-term viability of the project often intruded in discussions within the committee. these included the need to plan for platform upgrades, security patches, and the addition of new features. in spite of the fact that these issues were discussed, solutions were never agreed upon and these longstanding questions about the long-term maintenance of code generated during the project contributed directly to the ultimate suspension of development work. conclusion while the development of the indi system as an archival management application was unsuccessful, the process has been a learning experience for all involved. it has led to a deeper understanding of archival processes and descriptive standards among the subject specialists that served on the committee, and has led to the adoption of development and documentation standards by our library information technology division. it has also resulted in a greater integration between the archives and it staff within the library as we have come to understand each other’s work better. as the indi project came to a close, we began planning for its replacement. we began by developing the functional requirements documents and software requirements specifications that had been missing from the original indi project, as well as building case studies, refining workflow models, and building a new data model. however, with the recent economic downturn the committee has decided to suspend development work. in its place we are planning to adopt a two-application solution, using separate workflow management and archival management tools. with a two-system solution, information about the processes would be managed apart from the archival information. in this scenario, a process such as accessioning or processing would be initiated in a business process management (bpm) application such as processmaker or intalio bpm. this application would assign tasks to archives staff, and could notify subsequent users of pending tasks. but to complete the assigned tasks, staff would need to use a separate archives management tool, such as the archivist’s toolkit or ica-atom.[6] we are currently involved in a review of existing products, using our documentation as selection and testing criteria. although this configuration may appear workable, it is far from optimal. by maintaining the data in separate silos, the bpm system would be unable to automatically process decision points in the workflow as the information needed to make these decisions would be held independently in the archival management system. in order for the bpm application to complete a process, information about the archival materials would need to be entered into that system, leading to data redundancy and constant rekeying of descriptive information. while it may be possible to integrate these two systems using web services or apis, current archival management systems have not implemented such interfaces. while dividing the workflow from the data is not ideal, there is still no tool available that combines the two for archival management. in other industries system designers are beginning to integrate bpm software into applications to manage workflow.[7] in the library environment, the development of the web interface for resource description and access (rda), the proposed replacement of the anglo-american cataloging rules, features workflows as a central component in the use of the proposed standard. the indi project prototype has demonstrated the power and flexibility provided to archivists when business process management is integrated into archival management systems. it is our hope that the archival management systems currently available consider how to integrate bpm/workflow management into their systems. about the authors j. gordon daines iii is the brigham young university archivist and assistant department chair, manuscripts in the perry special collections at brigham young university. he holds a master’s degree in history from the university of chicago and received archives and records management training at western washington university. cory l. nimer is manuscripts cataloger and metadata specialist for the perry special collections at brigham young university. he holds a master’s degree in history from sonoma state university, and a master’s in library and information science from san josé state university. notes brigham young university hawaii in laie, hawaii and brigham young university-idaho in rexburg, idaho. information about the archivist’s toolkit project is available online at http://www.archiviststoolkit.org/. additional information about sierra-php is available on-line at http://code.google.com/p/sierra-php/. additional information about sierra-os is available on-line at http://code.google.com/p/sierra-os/. the source code for the indi plug-in is available on-line at http://code.google.com/p/sierra-indi/. information on the ica-atom project is available on-line at http://www.ica-atom.org/. one example is the integration of processmaker (bpm software) into knowledgetree (document management software). subscribe to comments: for this article | for all articles one response to "integrating process management with archival management systems: lessons learned" please leave a response below: cory nimer, 2009-04-04 additional information about the integrated digital special collections (indi) is available at the project web site: http://lib.byu.edu/indi/ leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – how the wsls-tv news digitization project helped to launch a project management office mission editorial committee process and structure code4lib issue 23, 2014-01-17 how the wsls-tv news digitization project helped to launch a project management office this article discusses how the wsls-tv news digitization project at the university of virginia libraries was the catalyst for creating a more formalized project workflow and the eventual creation of a project management office. the project revealed the need for better coordination between various groups in the library and more transparent processes. by creating well documented policies and processes, the new project workflow clarified roles, improved communication, and created greater transparency. the new processes enabled staff to understand how decisions are made and resources allocated which allowed them to work more efficiently. by ivey glendon and melinda baumann introduction since 2012, the university of virginia (u.va.) library has invested heavily in two initiatives: providing access to some 13,000 film clips associated with the wsls-tv news film collection, 1951-1971, and launching a project management office in support of this and future projects. the complexity of the wsls-tv project has been a catalyst for building a project management program at the university of virginia library. a new project management office, currently housed in the library it department, is putting procedures into place that change the way projects are prioritized and how collections are made discoverable. this paper will discuss best practices for managing a multiple-department project team and lessons learned about creating robust project plans. the wsls-tv news film collection, 1951-1971 in 2007, the wsls-tv station in roanoke, virginia donated a large and comprehensive collection of news films and scripts of mid-20th century broadcasts to the u.va. library. the wsls-tv collection consists of approximately 13,000 clips of 16mm film and over 18,000 pages of accompanying anchor scripts, and has the only known surviving local news reels from the civil rights era produced in virginia. the content was unique and a good fit for the media collection, but after the donation was accepted it languished without a clear plan for processing and delivery.  while the library had undertaken many successful digitization projects in past years, this large and complex project required considerable time and resources from multiple departments with their own priorities. in 2010, the library’s preservation services unit took ownership of the project and received a $254,600 grant from the national endowment for the humanities (neh) in support of the preservation and digitization of the film and the paper scripts. from 2010 to 2012 a small number of staff members, primarily from preservation services, worked with a vendor to digitize the collection as they generated the descriptive metadata in-house. library it and other departments worked on how to deliver a digital video collection through the library’s online catalog, virgo. library leadership deemed it a priority to make the collection available, but progress was slow as the small team grappled with challenging issues: collection organization. there were three primary categories of content in the collection:  clips with matching scripts, clips without scripts, and scripts without clips. each object required manual sorting to place it in one of these categories. metadata. the pbcore metadata standard was chosen as most appropriate for a public broadcast and moving image collection; however, because it was the library’s first implementation of pbcore there was little familiarity with the standard. access. copyrighted content was discovered embedded in the collection. additionally, multiple vacancies in the library’s user experience team slowed decisions about the user interface. in late 2012, when it became clear that the processing, description, and access challenges threatened to overwhelm the project, a formal project team was formed and the library’s metadata librarian was asked to serve as project manager (pm). experts from five departments – library it, metadata management services, preservation services, digital curation services, and the arts & media group – were charged by the library’s new chief technology officer with making the collection’s content available online. the formality of charging a project team helped to focus the team on the common goal of moving the project forward. revised goals and principles upon convening the project team in october 2012, the pm worked with stakeholders to fully assess the status of the collection, its metadata, and the amount of staff resources allocated to the project. the review revealed that the project was understaffed and far more complicated than originally thought. consequently, several staff members from the metadata management services unit were added to the project team to devise and execute workflows for sorting content, matching film clips to their scripts, and creating standardized metadata for the entire collection. in addition to convening the team for a full scan of project activity, the pm created a dedicated mailing list, a confluence-based wiki for recording meeting notes and other documentation, and established regularly scheduled face-to-face check-in meetings with the project team. the regular meetings – initially viewed skeptically in a meeting-averse culture –proved critical to the success of a project with a large, complex collection and staff from numerous departments.   scheduled face-to-face interaction with the whole team improved communication efficiency by eliminating the need for smaller sub-group meetings and also fostered accountability for decision-making and task implementation because all project members began working with the same set of information and expectations. the data architecture group, the working group that had responsibility for all new and existing metadata standards, was still in the process of forming at the start of the project and didn’t have a clear path for implementing new standards. how would this important content be made available in the online catalog when there was no existing mechanism for the delivery of video with pbcore metadata? how would the scripts be displayed in relation to the video clips? new problems emerged as the project team was addressing existing collection, metadata and access challenges. the library still lacked a formal process by which scarce staff resources could be assigned to a project or collection, and not enough attention was paid to developing strong user requirements. what was needed was a process that took all expert opinion into consideration at the start of the project and widespread acknowledgement that prioritization sometimes necessitates that other projects must go on hold. project management office informs new workflow as the wsls-tv project team labored to complete their charge, the consolidation of the library’s it departments under a newly-hired library cto provided an opportunity to rethink the workflows for projects similar to the wsls-tv effort. the time was ripe to develop a more efficient and transparent process for vetting and prioritizing projects and allocating resources with support from a new project management office (pmo). the wsls experience has been the cornerstone of this new project proposal process insofar as it has exposed needs for robust project planning and significant institutional support (in the form of technical and human resources, project prioritization, and space for documentation) in ways that no other collection had before. we know that the wsls-tv project, while complex, is not unique in its complexity and that future projects at the library will need similar support from units across the library. the pmo, in response to the wsls-tv project having exposed resource gaps in the organization, now leads a new workflow wherein staff proposing projects define desired needs and outcomes much earlier in the planning process. each new project proposal requires a sponsor to provide some resources and a business process owner to provide early details to flesh out the proposal and to help develop user requirements. projects require early assessment and planning, not “just-in-time” assessment and planning. transparency makes the resource allocation process less mysterious. any deadlines or other urgencies need to be stated up front. finally, all proposals must link to at least one library strategic direction (https://www.library.virginia.edu/assets/strategicdirectionssep2013.pdf) to support its prioritization. the new u.va. process involves over 50 people (or almost 25% of full-time staff) in the decisions to approve, inform, define, and prioritize projects. earlier efforts to ask managers to prioritize were largely unsuccessful. most felt they didn’t have either the technical expertise or enough information about the impact of projects to consider their merit or to rank them in relation to each other. but with a more robust and well-documented vetting process in place, staff from across the library are more willing to delve into the complex, granular proposal details and form opinions on what projects are truly priorities. here’s how it works: proposals are submitted on a web form that is sent to the pmo. the pmo assigns the proposal a tracking number and creates a wiki site for documentation and updates. the pmo then sends the proposal to one of three service teams representing the library’s major content delivery mechanisms: the online catalog (virgo), the institutional repository (libra), and the other critical web services such as the library home page and online exhibit omeka installation. recently a fourth service team was added to review projects that support business operations, grants, and external collaborations. each service team oversees a portfolio of projects to completion. if it aligns with library strategies, the service team conditionally approves the proposal within two weeks. service teams then send representatives to meet with working groups responsible for user experience, collections, data architecture, and systems. these working groups have the expertise to flesh out proposals with resource requirements and to comment on potential project constraints, assumptions, and risks. the collections group has the additional responsibility to consider the impact and value of the content on u.va. scholarship.  the working groups have at least four weeks (concurrent) to review and comment. the newly-compiled proposal, when returned to the overseeing service team, is then deemed a project. the project is ready for prioritization against the other new projects approved that quarter and against projects already underway. prioritization efforts are successful because the service teams are very familiar with the project details. library leadership teams are also updated and provide some input on prioritized lists on a quarterly basis. after prioritization the pmo assigns a project manager and provides administrative support to the project manager and team. u.va. library workflow: proposal to project diagram legend sponsor: usually a department head who conceives of a library-wide project and can provide some resources bus. proc. owner: business process owner, usually the proposal author and champion of the project, is most knowledgeable about the requirements and desired outcomes pmo: project management office, six members of library it department who administer library projects with an it component from inception to conclusion lib-it mgmt: staff reporting to the head of library it service teams: groups with in-depth knowledge of major library content delivery systems and infrastructure that evaluate and approve project proposals and oversee projects to their completion working groups: groups with in-depth knowledge of systems, metadata, collections, or user experience lib-ops: library operations group, a decision-making body responsible for translating the library’s strategic directions into user-centered services slt: senior leadership team, a decision-making body comprised of university librarian and other library leaders as of november 2013 there were over a dozen active projects that the project management office has shepherded through the new proposal process. there are also several dozen other active projects and project teams that predate the new process but are benefiting from the improved prioritization method. resources are being used carefully and effectively and library leaders see that more is getting done. the pmo is currently working with service teams, individuals, and library departments to identify staff interested in becoming project managers and to provide baseline training. departments that intend to submit proposals in the future are particularly encouraged to allocate staff time to project management. providing staff that are knowledgeable about a proposal and its desired outcomes will greatly help to put the project on track and keep it there. over time, the intent is to build a community of skilled project managers willing to take on ambitious projects from across the library.  conclusion the wsls-tv news film collection, 1951-1971 has been an exciting, complex and challenging collection for the library. the first batch of several thousand clips and scripts became publicly available in august 2013, less than a year after the project team formed – proving that new collections to support research and scholarship can be adequately described and made accessible in a timely manner. the team now adds new content in two releases each month. the wsls-tv project is on schedule to conclude in may 2014, thanks to good project management and the willingness of the project team to communicate continually and rethink long-standing workflows. other institutions may also find that a more robust project workflow can help change organizational culture. the team model in place for the wsls-tv project has demonstrated to library staff a need for centralization around projects, and the new pmo processes have added structure, clarified roles, and led to greater efficiency for the organization. the transparency created by the documented policies and processes allows library staff to understand how decisions are made and resources are allocated, enabling staff to create more effective proposals and feel confident that their proposals have been given adequate attention. at u.va., the immediate goal remains to develop a process that is efficient and sensible by using the wsls-tv project as a model. the larger, long-term goal is to build trust, to foster greater ownership of projects, and to support and celebrate their successful outcomes. as this new project workflow develops, so too will staff expertise and comfort in prioritizing disparate projects and initiatives. further, participation from staff in all areas of the library is expected to foster greater understanding of and appreciation for interdependencies between departments and help to ensure that the decisions about how time and money are spent are made fairly. about the authors ivey glendon (img7u@virginia.edu) is a metadata librarian at the university of virginia library, where she consults with faculty, students and staff on metadata schema and controlled vocabularies, suggests tools for metadata creation and management, and provides project management support. prior to joining u.va. in 2011, she served as a digital conversion specialist for the national digital newspaper program at the library of congress. melinda baumann (baumann@virginia.edu) is librarian for project and service management in the library it department at the university of virginia, where she has worked since 1993. in addition to managing technology projects she has been instrumental in creating a new paradigm for project workflow being adopted by the university library. baumann has also worked as a reference librarian and a director of digital production. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – jupyter notebooks and institutional repositories: a landscape analysis of realities, opportunities and paths forward mission editorial committee process and structure code4lib issue 58, 2023-12-04 jupyter notebooks and institutional repositories: a landscape analysis of realities, opportunities and paths forward jupyter notebooks are important outputs of modern scholarship, though the longevity of these resources within the broader scholarly record is still unclear. communities and their creators have yet to holistically understand creation, access, sharing and preservation of computational notebooks, and such notebooks have yet to be designated a proper place among institutional repositories or other preservation environments as first class scholarly digital assets. before this can happen, repository managers and curators need to have the appropriate tools, schemas and best practices to maximize the benefit of notebooks within their repository landscape and environments. this paper explores the landscape of jupyter notebooks today, and focuses on the opportunities and challenges related to bringing jupyter notebooks into institutional repositories. we explore the extent to which jupyter notebooks are currently accessioned into institutional repositories, and how metadata schemas like codemeta might facilitate their adoption. we also discuss characteristics of jupyter notebooks created by researchers at the national center for atmospheric research, to provide additional insight into how to assess and accession jupyter notebooks and related resources into an institutional repository. by adrienne vandenbosch (0000-0001-8777-8699), keith e. maull (0000-0002-3459-5810), and matthew mayernik (0000-0002-4122-0910) introduction jupyter notebooks have seen widespread adoption in scientific disciplines, from geosciences and astrophysics, to computational biology and others. their ease of use, support for the python programming language (among others), portability, shareability and easy deployment onto a wide range of hardware and into execution environments from local to remote, has made them popular among novice and expert users alike. the proliferation of notebooks, however, has produced a number of challenges. the enthusiasm for sharing notebooks on software repository platforms like github (https://github.com), have made things increasingly complex. github, while a platform likely to exist for a long time, is not an institutional repository (ir), and does not share the same commitment to preservation and permanence as traditional irs. to address this issue, the software heritage service (https://www.softwareheritage.org/) has been created to preserve the contents of github repositories, among others. it operates on a massive scale, duplicating github at large without any filtering. it thus provides preservation capabilities, but not access, curation, or support. therefore, we ask an important question for scholarly libraries: how do institutional repositories play a role in preserving these important digital outputs of scholarly work and making them more accessible and usable? institutional repositories typically house scientific and scholarly output as the permanent record of comprehensive scholarship amongst other related and relevant digital assets such as publications or data. it is therefore not unreasonable for traditional ir platforms to play an active role in supporting notebooks despite their relative novelty in the modern scholarly workflow. the challenges of doing so are significant, but the tools, techniques and technologies already exist for doing so. thus, in this paper we investigate what it might take for notebooks to be accessioned into institutional repositories as first-class scholarly assets. many questions remain around just how these new outputs of 21st century scholarship fit into the landscape of other more traditional assets like publications or even data. it is clear, however, that before we can understand their place among assets which are already accepted, we must address one of the primary barriers to accepting notebooks into traditional repository infrastructure: metadata. whether inconsistent or altogether absent, any digital asset without proper and appropriate metadata is as good as a ghost within the wide and vast repositories operated by the best academic institutions around the world. the optimal solutions for managing, preserving, and providing access to computational notebooks lie within some combination of tools, standards, best practices, guidelines, incentives, penalties or any number of remedial strategies to help certain communities realize the importance of these digital products. but which tools and standards? what best practices? whose guidelines? at the moment, there are standards and best practices that could be applied, yet little consensus has grown around their adoption and use. likewise, tools and schemas that could generate appropriate metadata are either not widely used or known to end users. furthermore, many standards, guidelines, and best practices require that communities adopt them, and often this is driven by governance structures and community consensus. more needs to be done to accelerate and extend the range of existing solutions, best practices and tools, as well as consider more rigorous approaches to provide the highest value for notebooks to take their place among more traditional digital assets within institutional repositories. in this paper, we will: detail the state of jupyter notebooks (and more generally, executable code) in the context of managed institutional repository assets through careful examination of best practices and community knowledge, explore how existing institutional repositories are implementing the accessioning of notebook assets, examine the concrete application of codemeta schema metadata mappings to jupyter notebooks, and explore how a single scientific institution, the national center for atmospheric research (ncar), might assess and prepare a collection of their own github notebooks for accession and permanent deposit into their opensky institutional repository. given the relative sparseness of work on jupyter notebooks in irs, this project is necessarily exploratory of a broad landscape. but we hope that our work can point out current trends, challenges, and opportunities that offer insight for next steps within the digital library community. jupyter notebooks jupyter notebooks have become the de facto tool in many modern data science workflows, and have become an important tool in education and training across many disciplines. their popularity is owed to the confluence of several factors. first, the rise of computational thinking in modern scientific disciplines is now unavoidable (wing, 2011). the size and scope of data requirements to conduct scientific inquiry require active application of computational techniques. while the use of computation in science is not new, the scope of its use is, requiring more domain experts than ever to learn how to write software to conduct the kind of research necessary to make field-relevant advances. the python programming language is the second factor contributing to the broad adoption of jupyter notebooks. python is a language that was initially developed as an alternative to languages like c to lower the syntactic complexity and simplify the semantic expressiveness of programs. while over 30 years old as of 2023, in the past decade, python has enjoyed a dramatic increase in popularity and use among computational newcomers because of its simple and unimposing syntax, natural expressiveness, intuitive data structures and robust software libraries for a broad range of scientific disciplines from astrophysics, genomics, geoscience to others. jupyter notebooks are executed within a portable, web-based platform that combines into a single tool the execution capabilities of typical programming interfaces, with the expressive capabilities of narrative structure. a notebook consists of a series of cells that contain program fragments intended to be executed sequentially. the program fragments can contain any valid code. though jupyter supports many more languages, python is the most popular within the platform. a cell may also contain non-code fragments of text, possibly technical expositions that either expand details of the code, or provide theoretical or any other explanation of the contents of the notebook. when seen as a whole, these cells form the “notebook”, and a form of what can be called computational narrative. notebooks have a tremendous amount of appeal because, not only do they capture the execution of the cells themselves (and hence the “executable” nature of notebooks), but they may contain a variety of cell outputs such as images, diagrams, graphics and even interactive widgets that allow the end user to actively change inputs and see corresponding changes in real-time (see figure 1 for an example). many feel that these notebooks and their interactive features hint at the future of all scientific explanation that provides the transparency and auditability necessary for computational-centric science to progress (gil et al., 2007; stodden et al., 2016; stodden et al., 2018; brinckman et al., 2019). indeed, on platforms like github, there are now millions of such notebooks (rule et al., 2018), some of which contain important scientific exposition. jupyter notebooks of scientific value are the ones that we are most interested in for this work, and specifically, how they become part of the permanent scientific record by way of irs. figure 1. example jupyter notebook. jupyter notebooks and institutional repositories jupyter notebooks are exploding in number and variety, and are being used as part of scientific workflows and in other data science applications. these resources are highly variable in size, content, and utility, however (wang et al., 2020). many jupyter notebooks that are publicly available on github appear to serve as exploratory tools for their creators, and have not been created with any intention of use by anybody but the creator (rule et al., 2018). further, despite their promise as sharable resources that can be reused and re-executed by known or unknown collaborators, many publicly available jupyter notebooks have problems that prevent them from being re-executed in any straightforward fashion (pimentel et al., 2021). for example, many notebooks do not indicate the versions of the libraries used, making precise reproduction tricky. additionally, when there are errors due to missing files, for example when data files are imported using hard-coded paths instead of relative or url paths, re-execution, let alone reproduction becomes even more challenging. institutional repositories come into the discussion as a potential home for long-term curation and preservation of jupyter notebooks. this discussion is stimulated by the widespread use of github as a place to store and share jupyter notebooks, and by the increase in links to github within scholarly publications (escamilla et al., 2022). github has many great features to support software development and file sharing generally. but it is owned by microsoft, a for-profit corporation, and is not designed or operated to be a preservation environment. microsoft has made gestures toward preservation of material hosted in github, such as creating an “arctic code vault,” in which a snapshot copy of github was deposited into a frozen norwegian mine in 2020. but such publicity stunts belie the lack of any practical digital preservation strategies built-in to github (rosenthal, 2019). can irs have a role in addressing this challenge? although the ir landscape has evolved significantly over the past 20 years, the basic functions of irs remain consistent with clifford lynch’s formulation from 2003 (pg. 328): an ir consists of “a set of services that a university offers to the members of its community for the management and dissemination of digital materials created by the institution and its community members. it is most essentially an organizational commitment to the stewardship of these digital materials, including long-term preservation where appropriate, as well as organization and access or distribution” (lynch, 2003). this organizational commitment has thus far focused on publications, gray literature, and digital data sets. where do software and computational notebooks fit within organizational commitments regarding digital curation? it may be that irs are not the optimal approach for curation and preservation of jupyter notebooks. perhaps centralized services like zenodo (https://zenodo.org/) and software heritage have benefits regarding economies of scale and sustainability planning, but this is an open point of debate (arlitsch and grant, 2018). one central finding of ir research and operations, however, is that irs are continuously needing to be reassessed, iterated, and evolved over time, with this evolution encompassing the ir technologies, scale and scope, services, and the collections themselves (cragin et al., 2010; borgman et al., 2015; fallaw et al., 2021). in the remainder of this paper, we will survey the terrain of software for institutional repositories along four strata. stratum one explores the community knowledge and best practices within ir communities for software assets. given the increased general demand for software deposits, a unique set of community behaviors and knowledge has emerged that requires deeper understanding. along stratum two we narrow the scope of investigation to five large irs to learn more about what concrete repository practices can be observed for jupyter notebook deposits in the wild. learning how best practices are synchronized with implementation provides important links to the realities facing irs. within stratum three we dive deeper into codemeta (https://codemeta.github.io/) as an important tool for capturing jupyter notebook metadata. here we explore how this schema can be crosswalked to the metadata object description schema (mods). this effort explores how software-focused metadata might fit within typical ir metadata structures. in the fourth stratum, we examine how a single ir might act as a case study in decision making by looking at the national center for atmospheric research (ncar) github organization for jupyter notebook candidates that might be deposited into the ir at ncar. this exercise demonstrates some of the complexity of the decision to accession notebooks, as well as the nuance required to be successful. community knowledge and best practices an investigation of community knowledge was conducted through the analysis of relevant literature to gather current ideas, best practices, and theories about software and executable code as assets in irs. publications regarding the use of jupyter notebooks as scientific objects were also analyzed to gather information on how creation and use may affect decisions around managing them in irs. jupyter notebooks are used for analysis of results, as supplements for publications and as the source of books, dissertations, theses and in many other research contexts (kluyver et al., 2016). increasingly, they are being integrated into and may form the backbone of complex scientific workflows (sullivan et al., 2019) and hence the target of recommendations for how such workflows are reproduced and what tools to use. a typical notebook is run in a specific computational environment with an underlying hardware and software configuration. specific versions of modules and libraries that are necessary for proper (and expected) functioning of a notebook are crucial, since if even versions of imported modules are different, execution outputs, and hence reproducibility and importantly replicability may be impossible, leading to undesirable outcomes that are increasingly plaguing scientific research (reality check on reproducibility, 2016; resnik and shamoo, 2017). capturing, codifying and standardizing these configurations has been the topic of much research in software generally and jupyter notebooks, specifically. for example, bouquin, et al (bouquin et al., 2019), suggest that notebook metadata include version information for jupyter kernel and distribution, while erdmann, et al (erdmann et al., 2021) recommend, in addition to module versions and dependencies, that zero-install cloud environments and configuration be considered to facilitate running and executing notebooks reliably. still too, beg, et al (beg et al., 2021) recommend using mybinder (https://mybinder.org/) to ensure notebooks are rerunnable, but acknowledge that relying on third party tools may introduce unresolved complications. similarly, containerization methodologies such as data and software preservation for open science (daspos) (thain et al., 2015) provide a glimpse of what might be considered (and which tools to use) to specify, assemble and execute software, including all dependencies. these are just a few of the ongoing efforts to address the executable reproducibility of notebooks. storing notebooks and their corresponding metadata is necessary for discoverability and preservation. at the moment, many jupyter notebooks are created (and hence stored) on the github platform, which provides reliable, yet impermanent storage for notebooks. notebooks stored on github can be removed at any time by their creator (and infrequently, by github), and while the platform provides an incredible level of detail about changes to a repository and any other asset in them, an internal github-maintained metadata schema and data format for storing and tracking data is used. concerns about longevity and metadata of assets in github have encouraged several community efforts. for example, zenodo is a platform supported by the european commission (ec) and operated by openaire and cern (conseil européen pour la recherche nucléaire) that focuses on permanence and preservation of github repositories. not only does the zenodo platform provide automated mechanisms for providing versioned snapshot archives of github repositories, it provides persistent identifiers (as dois) and metadata in schemas common to institutional repositories, such as marcxml, dublin core (dc) and datacite. when used in conjunction with recommendations like those suggested by greenberg, et al (greenberg et al., 2021), zenodo reveals an important step that shows how normalizing both end user deposit behavior, best practices and the structure of metadata might inform institutional repository decision making for software as well as notebooks. unified frameworks for software citation and attribution have been proposed by force11 (katz et al., 2019) who identify six essential dimensions of software citation: (1) importance, (2) credit and attribution, (3) unique identification, (4) persistence, (5) accessibility, and (6) specificity. these same dimensions are applicable to jupyter notebooks, and provide guidelines for curation, standardization of metadata and deposit of notebooks along these dimensions. for example, given the broad proliferation of notebooks, importance criteria are especially necessary for institutional repository curators to scrutinize what is deposited, as resource limitations require adequate filters on accessioned assets. furthermore, credit and attribution as well as unique identification have been the subjects of robust debate. dois (digital object identifiers) have been used since their creation as a robust mechanism to ensure the attribution, persistence and resolution of digital objects, and are recommended as the preferred identifier for many assets deposited into irs (banerjee and forero, 2017), though other persistent identifiers (e.g. handles) could be equally valid in many ir contexts. however, dois typically resolve to a single digital object that is not subject to significant change (e.g. a publication), yet software in general is often changing and the versions of those changes are important for research, putting unique pressure on the concept of using a single doi for an evolving object. recommendations for citing versions of software often follow patterns established for versioned data citation, though consensus on the accepted strategy for minting dois for software versions has yet to emerge. furthermore, unique challenges emerge with the use of microcitation or highly granularized citations that might point to individual lines of software source code (data subsets in the case of data) or in the case of jupyter notebooks, output cells containing figures, graphs or computation outcomes. it is acknowledged that this type of citation is more difficult to achieve with a doi, and in many research communities where dataset (subset and derivative) citation is acknowledged as necessary for research reproducibility (such as earth and life sciences communities), important recommendations (patrinos et al., 2012; esip software guidelines draft, 2016) have emerged to address some of the underlying issues. for software, these issues have been aggressively addressed by the non-profit organization software heritage to focus on the unique characteristics of software and the corresponding wide range of citation use cases. their swh-id uri schema (di cosmo, 2020), which also permits microcitation, is an important step forward, even if it breaks from the conventional use of dois. important work in deposit guidelines for software digital assets has been innovated by the software sustainability institute (brown et al., 2018), who provide a detailed framework for why, when, what, where and how software might be deposited into a repository by researchers and curators alike. their framework proposes that a deposit adhere to one of three levels: minimal, runnable and comprehensive – where a comprehensive deposit would not only include the most detailed metadata, but also detailed documentation about the software, its purpose, function, and ecosystem (e.g. dependencies) as well as test code such that expected outputs are well understood and inspected against running code. this can be especially important in a jupyter notebook context, where a notebook can be deposited with the executed cells as well as unexecuted cells, and if a notebook is executed and any cells fail, it can be considered as potentially unfit for deposit according to ssi guidelines. knowing whether a notebook has failing cells would also be necessary to develop a larger strategy for assessing quality standards of deposits. similar efforts to encourage software deposit and lift software to first class status can be seen with the peer-reviewed journal of open source software (joss) (smith et al., 2018), which accepts software submissions of all kinds to normalize software as an important asset in scientific communication at the same level of importance as publications. the journal enforces systematic and consistent structure, description and execution standards of software submissions to encourage a community of practice built around high quality software norms and guidelines. while deposits into joss stay within their transparent community-centered platform, any of its deposits could be accessioned into a relevant ir, but importantly, joss demonstrates a comprehensive approach to deposit, metadata, preservation and presentation – which might inform irs of strategies that may be implemented in their own environments adapted to the needs of their specific communities. finally, metadata for notebooks is an important, but often neglected topic. building more on the software sustainability institute’s typology of minimal, runnable, and comprehensive software deposits, metadata for software minimally requires readme files with the software name, description, contact information, and link to the live project or source code; they should also include explicit copyright and license information, author and contributor information, and the source code (jackson, 2018). for a runnable deposit, a pdf or static html version of the notebook, a full dependencies list, a citation file, a codemeta/json-ld file and a container file where applicable (rule et al., 2018 oct). in a runnable deposit the readme should include a version number, copyright statement, user documentation, and a summary of other deposit content (jackson, 2018). even without runnable or executable software, source code can have long-term value as data (esip software guidelines). runnable and comprehensive deposits include richer metadata about dependencies and programming languages, more in depth documentation, sample data, and a containerized version/emulation of the software. one set of metadata guidelines explicitly focused on software is the codemeta framework. we discuss codemeta in detail in a later section, as it defines metadata elements that are most relevant, adequate and appropriate for software. all of these efforts reveal the inherent complexity and limitations of current strategies for citing, describing and preserving software, and more generally the unique features of emerging digital objects that must be curated and preserved as part of the 21st century scientific record. while many of these efforts are maturing, and their communities still growing, there exists substantial demand and enthusiasm to build techniques, tools and strategies that demonstrate the importance of software and executable digital objects such as jupyter notebooks. jupyter in existing irs we looked at the institutional repositories of five large research institutions to see what examples of jupyter notebooks in irs already existed and what could be learned from them. caltech data, michigan deep blue data, stanford searchworks, virginia tech data repository, and university of california san diego digital collections were chosen because they were known to have software or notebook files. this was not intended to be a systematic examination of the entire ir landscape, but to provide insight into key characteristics of jupyter notebooks as they exist in irs. from these observations, we found that deposits containing jupyter notebook files did not follow uniform practices, even when existing in the same ir. in one repository, notebooks were categorized as three different types of objects (data, software, and workflow), perhaps referencing their intended use case and therefore a need for different preservation practices. trying to understand decisions made regarding the handling of notebooks and related objects was especially challenging since the irs had few notebooks to compare for consistency of practices. future investigation would likely benefit from additional insight into guidelines and decisions for preservation for these items. table 1 summarizes the characteristics of the jupyter notebooks that existed in the irs we examined. a few deposits included a static representation of the executed notebook. in the majority of cases this was as a pdf, although one included a text file of the notebook json. this follows recommendations for including static representations, so that even if the notebook is no longer executable, there is some sort of representation of its original content. other ir deposits in the results of our search were data sets, mentioning ipynb or jupyter notebooks in their description, and linking to github repositories where the actual notebook files were stored and could be previewed. the virginia tech ir was powered by figshare (https://figshare.com/), which allows for an in-browser preview of ipynb files. here, a search for the python language unearthed a result that contained 6 zipped ipynb files that did not show up in searches for ipynb or jupyter. one repository, caltech data, provided links to mybinder for resources that contain an ipynb file, which enables users to invoke an online environment that allows for viewing, editing, and running the notebook file (morrell, 2019). caltech data has since moved to a new platform where mybinder links do not currently appear to be supported. table 1. high-level characteristics of jupyter notebooks in the examined irs. *limited to the resource type of ‘dataset’, due to full integration of the library catalog. **no individual ipynb files were available to test interactions, consisting of compressed/non-ipynb files. ir # results for ipynb # results for jupyter # results for python interaction, method metadata caltech data 9 14 – invocable, binder not discernible stanford searchworks* 2 2 19 preview, catalog** mods virginia tech 1 1 2 preview via figshare datacite, dublin core, nlm (figshare options) michigan deep blue data 1 2 19 preview, catalog** not discernable uc san diego 4 32 – preview, catalog** not discernable metadata schemas were unable to be determined for the majority of the repositories that we examined, making it difficult to find out if the information needed to access or support reproducible notebooks would be available. with most of the files in compressed form, searching for .ipynb would not necessarily return all results that contain jupyter notebooks without a complete list of files in a field that is searchable in the online catalog. once located, there is still the question of whether or not these notebooks are executable — are they provided with the information needed to maintain their relationships to external objects? notebooks require specific versions of external objects (libraries, data, etc.) and a particular ordering of cells to be reproducible. in the few instances where metadata schemas were distinguishable, it is clear that end users would struggle to find the necessary information to make these objects function. codemeta schema for jupyter what metadata should exist for jupyter notebooks? examples of relevant metadata for jupyter-based tools might include metadata about the inputs to the notebooks (data), the products or outputs of the notebook cells, the structure of the notebook itself, including linked or supplemental notebooks and, of course, metadata about the tools themselves (leipzig et al., 2021). jupyter notebooks are generally categorized as software, thus metadata recommended for software provides a good starting point in considerations for metadata that enable the preservation and reuse of these resources. the codemeta (jones et al., 2017) metadata framework provides one of the most robust platforms for describing software using json-ld and xml. for example, it has been used in the r community to automatically generate metadata for r software packages (boettiger, 2017). it is appropriate to discuss it regarding jupyter notebooks, because it captures a broad range of features that target the essential concerns for notebook content, including notebook configuration, such as language versions, target operating platforms and software dependencies. a crucial challenge remains unsolved, however: crosswalking codemeta to schemas commonly found in irs (mods, dublin core, etc.). here we describe a crosswalk between codemeta and the metadata object description schema (mods, https://www.loc.gov/standards/mods/), as a demonstration of the challenges and possibilities involved in applying the codemeta framework within a metadata schema that is commonly used in an ir. mods was chosen because at the time of this writing mods is the underlying metadata framework in the ncar library’s ir, called opensky (https://opensky.ucar.edu/). in order to complete the mapping, codemeta and mods documentation were consulted in-depth. sample xml files from three objects in opensky were used to verify the compatibility of crosswalk recommendations with actual repository records. a snippet of this mapping is shown in table 2. table 2. sample table of crosswalk from mods to codemeta. codemeta mods author name/role/roleterm(type?, =’code’)/[.=’aut’] citation relateditem(othertype?, =’references’) contributor name/role/roleterm(type?, =’code’)/[.=’ctb’] copyrightholder accesscondition/rights.holder/name copyrightyear accesscondition/creation/year.creation datecreated origininfo/datecreated mods has 20 top-level elements, while codemeta contains 67, and this fidelity mismatch requires any functional crosswalk to map top-level codemeta elements to attribute level metadata from mods. mods allows for multiple uses of the same top-level element in a record (e.g. multiple name tags for more than one author), that can then be distinguished using sub elements and attributes. for an item with two authors and one sponsor you would have three name tags, two with the roleterm type as author (aut in marc relator terms), and one with the roleterm type as sponsor (spn). where the mapping struggles the most is when these sub-elements and attributes do not yet have the controlled vocabulary to ensure consistency and accuracy of the crosswalk. for example, mods defines the affiliation element as one that can contain address and email as well as organizational affiliations using that same subelement. however, there is no attribute of affiliation that allows for the direct mapping of email from mods to the codemeta email property. where these attributes and controlled vocabularies don’t yet align with codemeta properties, mods does have the extension element. the mods extension element creates an opportunity to preserve all unmapped codemeta properties within that element tag. this is not ideal, but would ensure that if a notebook were to be ingested from an environment using the codemeta into an ir using mods, the needed information for notebook understanding and execution would not be lost. alternate approaches could be to map key metadata fields to mods as noted above, and upload a full codemeta with all of the relevant fields as a separate json file into the repository, or include the metadata directly in the notebook as a header cell. identification & extraction: ncar github when choosing to ingest from github into an ir for long-term preservation and access, it is important to assess how ingest-ready a jupyter notebook might be. the github api provides access to internal metadata which serve as markers of notebook readiness. in order to see what markers and additional information would be discoverable, we used the github api to look specifically at the ncar organizational github. limiting our assessment to the ncar organizational github also gave us the ability to look at the variation of practices within a fairly controlled environment, putting into perspective just how much variation will likely exist in larger organizations. summary characteristics of the jupyter notebooks found in the ncar github are shown in table 3. table 3. summary of jupyter-centric repos in the ncar organizational github. repository characteristic notebook count repo was identified as containing jupyter nb 93 repo was identified as having majority of jupyter nb language ≥ 60% jupyter nb 89 had description 70 indicated license 51 project potentially stable/candidate for ingest created before jan. 1, 2020 39 project is current/candidate for consult updated after jan. 1, 2022 41 repo with higher interest level 25+ watchers 5 associated with a doi doi in readme file, either assigned or to a related object 8 our api query found repositories within the ncar github that were “jupyter-centric,” having at least 60% of their identified languages as jupyter notebook. it is important to note, that the languages in a repository are determined by github and may change over time or differ from individual identification. from our query, we identified 89 of the 93 repos containing jupyter notebook language, with at least 60% jupyter notebook as the determined language. we gathered the count of watchers and forks, open issues, owner info, object license, and if a readme or cff file was present. these data, we felt, would be good indicators of the dimensions of importance through follower counts, credit and attribution through citation files and licensing information, unique identification and persistence through an assigned or related doi, and persistence through creation date and last update as well. from the 93 repos, 51 had licenses, eight had dois present in the readme file, and five had 25 or more watchers. the date of creation or last update for a repo could be used to identify long standing projects that might be at a stable state for ingest into an ir. alternatively, a repo with a large following and frequent updates may indicate a project that would be a future candidate for ingest. gathering this information could flag repositories to approach about possible ingest. this information could also provide targets where an ir could offer support to improve the quality of metadata so that deposits are received in a more complete state. further work is necessary to assess the extent of notebook accessibility and executability, looking closer at content and metadata. discussion and conclusions in this final section, we discuss several types of considerations for archiving and providing access to jupyter notebooks via institutional repositories. we discuss in turn conceptual, technical, process, and preservation considerations. conceptual considerations. one of the most important aspects of archiving and storing assets in a repository is the conceptual integrity of the object — that is, when it is accessed, does it lie with a range of acceptable and expected forms. jupyter notebooks can take on a broad array of forms, from pure programming content to pure narrative content, with a broad spectrum between. deciding what a jupyter notebook is precisely, poses many challenges, especially when trying to classify for accessioning into a repository. for example, existing repositories studied in this paper classify jupyter notebooks in various ways. some contain a large amount of data (or heavily rely on it). some contain mostly software code (often processing data from somewhere else). some expose particular processes or workflows for doing a computational task, while others are narrative in scope and contain little code or data. with such varied and flexible forms, it becomes challenging to classify a notebook as predominantly data, software, narrative, workflow, or some ratio of all of these. this complexity puts a special burden on deposit decision making, since each of these may look quite different because of the files that must be deposited into a repository to maintain the necessary conceptual integrity (and usefulness) of the notebook. indeed, a deposit may include one, some or all of .ipynb, python source files (.py), json, csv, pdf files and more. second, should jupyter notebooks be dealt with as solitary or compound objects? notebooks can be individual files, but are often part of larger collections of objects. they therefore can have important dependencies and interrelations with other software, data, and metadata. repositories must plan and develop strategies for accommodating jupyter notebooks as sets of objects, and for potentially managing relationships therein, or to external objects (such as data sets provided by external organizations). technical considerations. from a technical point of view, since jupyter notebooks are executable and it is indeed this executability that make them so important in scientific reproducibility, repositories must consider whether to simply provide access to the jupyter notebook files, or whether to enable users to interact with and execute those files via the ir platform (or through some third party platform). tools like binder (https://mybinder.org) exist for invoking jupyter notebooks in interactive computational environments, but require financial and computational resources that may be hard to predict as they are dependent on the size and complexity of the jupyter notebooks and associated data. another important reality is that many projects that create jupyter notebooks use github to enable active software development, collaboration, discoverability and public exchange. irs clearly have a different set of technical capabilities (and function) than github, and as such provide different services. enumerating these services, while providing bridges to the unique capabilities of github (e.g. automation, enabling programmatic connections to irs), is yet unexplored territory. for example, irs may act as third party platforms to github, providing services that programmatically ingest github repositories for access and preservation purposes. this could complement the capabilities of other github preservation services, such as zenodo (https://zenodo.org) and software heritage (https://softwareheritage.org). process considerations. to establish an ir as a routine location to archive jupyter notebooks, establishing selection criteria will be important. first, identifying what to preserve is of paramount importance. the explosive number of notebooks now being created by researchers in almost every academic field, make process improvement a vital focus for irs. for example, self-deposit (e.g. github-to-zenodo) may be one option, where the creators of resources decide what and when to archive their materials, but since many irs operate via a mediated deposit strategy, irs would need to prioritize and develop clear selection and evaluation processes. such selection and evaluation considerations also lead to another key question: what criteria are used to accept jupyter notebooks for preservation? it is known that the quality and completeness of notebooks on github vary greatly (rule et al., 2018) (and elsewhere), so some standard must be developed by target irs to guide a minimum acceptance criteria (or such exception guidelines otherwise provided). there is little value to preserving notebooks that are of low significance, incomplete, partially executable, or otherwise plagued with quality problems. irs therefore must develop a process for evaluating and curating notebooks during the deposit stage. preservation considerations. in some sense, most of the considerations thus far have implications for preservation. but additional preservation considerations were uncovered through this project. first, and importantly, there is not yet a standardized metadata schema used to describe and preserve jupyter notebooks. as noted above, connecting metadata conventions for notebooks to the metadata standards imposed by irs is complicated. while the codemeta framework proposed here provides a useful starting point, it still requires many metadata schema implementation decisions, which further require expertise and careful consideration of current and future metadata needs. a second key preservation consideration is persistent identification — that is, how should persistent identifiers like dois be assigned to jupyter notebooks given some of the conceptual considerations noted above? consensus has not yet emerged on what a persistent identifier to a jupyter notebook is pointing to — particularly when the notebook is part of a compound object. do identifiers point to the compound object, the notebook or outputs within a notebook? how are complexities like notebook versions handled? do identifiers maintain any integrity with the inaugural version identifier? these crucial questions do not have clear answers, but need to be addressed urgently, given the rapid acceptance of notebooks in scholarly contexts. management processes to facilitate notebook capture, review and deposit, as well as technical processes and best practices that encourage predictable notebook structure and quality must be prioritized. tips for moving forward. bringing jupyter-based resources into ir collections will likely be easier to implement via progressive policies that provide iterative approaches that allow rapid response and adaptation to the needs of ir operators and depositors alike. emphasizing the partnership between the ir and depositor will also lead to better outcomes. this study has uncovered the wide variance in existing practices. several recommendations are now provided to guide irs toward developing practices and policies for accepting jupyter notebooks. recommendation #1: develop a recommended file/folder structure for github repositories housing jupyter-based assets for deposit or accession into an ir. as previously mentioned, many scholarly materials, especially those that are based on jupyter notebooks, are stored in github. the advantages of github are clear for open source (and open science), but the platform itself provides little guidance on how jupyter-based repositories should be structured. furthermore, github itself makes no guarantees on the long-term availability or accessibility of public code repositories, making their deposit and accession into irs even more important. providing a recommended template structure for github repositories which would be deposited into an ir would accelerate and standardize jupyter-based assets. if a deposit has its own structure different from the recommended template, it is important to encourage the depositor to reveal the details and logic of that structure so it might become part of the metadata of the deposit. recommendation #2: leverage codemeta mappings to ir metadata schemas where possible, or develop an ir-specific schema that fits the needs of the target repository environment. while codemeta has not become a standard for jupyter-related metadata, at this time it is the most appropriate schema for these types of digital assets. developing a custom schema is largely unnecessary as codemeta provides enough flexibility to integrate relevant domain and ir specific metadata. connecting codemeta concepts to ir metadata frameworks still presents challenges, but the more codemeta is used, the more these issues can be ironed out. there is not yet agreement whether some minimal metadata responsibility should be required of depositors or even how such metadata should be produced. basic notebook metadata could be created by depositors and included as a file in the source github repository containing a jupyter notebook deposit. for example, the citation file format (cff) (druskat, 2021) for software citation metadata, might provide strategic hints for storing jupyter metadata in machine-readable files which ir operators could then use to build a proper and more complete codemeta metadata as suggested in this paper. there are other technical solutions that are also possible, including embedding metadata directly into the jupyter notebook with a plugin or extension that queries notebook authors for basic notebook metadata. recommendation #3: develop and publish transparent deposit guidelines for jupyter-based submissions that enumerate the minimal required metadata and structural requirements in recommendation #1 above. encouraging ir deposits of jupyter notebooks becomes easy once a consistent github repository structure (or any notebook submission outside of github) is developed. when deposits are reviewed for official accession into the ir, if there are issues why submissions are not accepted, ir operators need to provide transparent processes for concrete feedback on how submissions can be improved for successful accessioning — relying as much on the published guidelines set forth to reduce ambiguity and frustration. furthermore, providing pre-submission checklists and guides well in advance of submission, should go a long way to improve deposit quality and acceptance rates. recommendation #4: develop policy guidelines that provide intuitive frameworks for decision making within ir management. choices and policies regarding the management of jupyter notebooks must be informed by their relationships to external objects and their intended use cases. if the ir does not support execution of notebooks, for example, this decision should be communicated within ir management and expectations properly set. this paper has presented a broad perspective on the landscape and challenges of modern scholarly products which should not go unnoticed by academic institutional repositories. jupyter notebooks provide an important potential use case for irs, not only because of their wide adoption and increased growth, but also because they represent a class of digital object that is as of yet underserved by well-curated digital repositories. the variation of objects, metadata, and practices, even within singular institutions, points to an urgent need for action in developing policy and guidelines that their respective ir can support. there is yet a lot of work to be done in this area, but the time has come for ir operators and depositors alike to begin taking assets like jupyter notebooks more seriously so that important academic products can be preserved for future generations. acknowledgements we thank peter organisciak for input during early stages of this work. this work was supported by the collaborative analysis liaison librarians (call) project, made possible in part by the institute of museum and library services (#re-13-19-0027-19). this material is based upon work supported by the national center for atmospheric research, which is sponsored by the national science foundation under cooperative agreement no. 1852977. references arlitsch k, grant c. 2018. why so many repositories? examining the limitations and possibilities of the institutional repositories landscape. journal of library administration. 58(3):264–281. https://doi.org/10.1080/01930826.2018.1436778. [accessed 2023 jan 27]. banerjee, k., forero, d. 2017. diy doi: leveraging the doi infrastructure to simplify digital preservation and repository management. code{4}lib, issue 38. https://journal.code4lib.org/articles/12870. [accessed 2023 oct 31] beg m, taka j, kluyver t, konovalov a, ragan-kelley m, thiéry nm, fangohr h. 2021. using jupyter for reproducible scientific workflows. computing in science & engineering. 23(2):36–46. https://doi.org/10.1109/mcse.2021.3052101. boettiger c. 2017. generating codemeta metadata for r packages. the journal of open source software. 2(19):454. https://doi.org/10.21105/joss.00454. [accessed 2023 sep 28]. borgman cl, darch pt, sands ae, pasquetto iv, golshan ms, wallis jc, traweek s. 2015. knowledge infrastructures in science: data, diversity, and digital libraries. international journal on digital libraries. 16(3-4):207–227. https://doi.org/10.1007/s00799-015-0157-z. [accessed 2023 jan 27]. bouquin d, hou s, benzing m, wilson l. 2019. jupyter notebooks: a primer for data curators. https://hdl.handle.net/11299/202815 [accessed 2022 feb 1]. brinckman a, chard k, gaffney n, hategan m, jones mb, kowalik k, kulasekaran s, ludäscher b, mecum bd, nabrzyski j, et al. 2019. computing environments for reproducibility: capturing the “whole tale.” future generation computer systems. 94:854–867. https://doi.org/10.1016/j.future.2017.12.029. [accessed 2023 sep 28]. brown c, hong nc, jackson m. 2018. software deposit and preservation policy and planning workshop report. zenodo. https://doi.org/10.5281/zenodo.1304912 [accessed 2022 feb 1]. cragin mh, palmer cl, carlson jr, witt m. 2010. data sharing, small science and institutional repositories. philosophical transactions of the royal society a: mathematical, physical and engineering sciences. 368(1926):4023–4038. https://doi.org/10.1098/rsta.2010.0165. [accessed 2023 jan 27]. di cosmo r. 2020. archiving and referencing source code with software heritage. in: bigatti am, carette j, davenport jh, joswig m, wolff t de, editors. mathematical software – icms 2020. cham: springer international publishing. (lecture notes in computer science). p. 362–373. druskat s. 2021. making software citation easi(er) – the citation file format and its integrations. in: zenodo. https://doi.org/10.5281/zenodo.5529914 [accessed 2023 sep 19]. erdmann c, stall, shelley, gentemann, chelle, holdgraf, chris, fernandes, filipe p. a., gehlen, karsten peters-von, corvellec, marianne. 2021. guidance for agu authors – jupyter notebooks. https://doi.org/10.5281/zenodo.4774440. [accessed 2022 feb 1]. escamilla e, klein m, cooper t, rampin v, weigle mc, nelson ml. 2022. the rise of github in scholarly publications. in: silvello g, corcho o, manghi p, di nunzio gm, golub k, ferro n, poggi a, editors. linking theory and practice of digital libraries. vol. 13541. cham: springer international publishing. p. 187–200. https://doi.org/10.1007/978-3-031-16802-4_15 [accessed 2023 jun 2]. esip software guidelines draft. 2016. https://esipfed.github.io/software-assessment-guidelines/guidelines.html#h.pzfz7hkzxqus. [accessed 2022 feb 1]. fallaw c, schmitt g, luong h, colwell j, strutz j. 2021. institutional data repository development, a moving target. code{4}lib.(51). https://journal.code4lib.org/articles/15821. gil y, deelman e, ellisman m, fahringer t, fox g, gannon d, goble c, livny m, moreau l, myers j. 2007. examining the challenges of scientific workflows. computer. 40(12):24–32. https://doi.org/10.1109/mc.2007.421. [accessed 2023 sep 28]. greenberg j, hanson k, verhoff d. 2021. guidelines for preserving new forms of scholarship. new york, ny: new york university. https://doi.org/10.33682/221c-b2xj [accessed 2022 feb 1]. jackson m. 2018. software deposit: what to deposit. https://doi.org/10.5281/zenodo.1327325. [accessed 2022 feb 1]. jones mb, boettiger c, mayes ac, smith a, slaughter p, niemeyer k, gil y, fenner m, nowak k, hahnel m, et al. 2017. codemeta: an exchange schema for software metadata. https://doi.org/10.5063/schema/codemeta-2.0. [accessed 2022 jun 25]. katz ds, bouquin d, hong npc, hausman j, jones c, chivvis d, clark t, crosas m, druskat s, fenner m, et al. 2019. software citation implementation challenges. arxiv:190508674 [cs]. [accessed 2022 feb 1]. http://arxiv.org/abs/1905.08674. kluyver t, ragan-kelley b, rez f, granger b, bussonnier m, frederic j, kelley k, hamrick j, grout j, corlay s, et al. 2016. jupyter notebooks – a publishing format for reproducible computational workflows. positioning and power in academic publishing: players, agents and agendas.:87–90. https://doi.org/10.3233/978-1-61499-649-1-87. [accessed 2022 jun 24]. leipzig j, nüst d, hoyt ct, ram k, greenberg j. 2021. the role of metadata in reproducible computational research. patterns. 2(9):100322. https://doi.org/10.1016/j.patter.2021.100322. [accessed 2023 feb 21]. lynch ca. 2003. institutional repositories: essential infrastructure for scholarship in the digital age. portal: libraries and the academy. 3(2):327–336. https://doi.org/10.1353/pla.2003.0039. [accessed 2023 jan 27]. morrell t. 2019. the opportunities and challenges of research data and software for libraries and institutional repositories. against the grain: linking publishers, vendors and librarians. 31(5):30–32. https://resolver.caltech.edu/caltechauthors:20200116-114211107. patrinos g, cooper d, mulligen e van, gkantouna v, tzimas g, tatum z, schultes e, roos m(marco), mons b. 2012. microattribution and nanopublication as means to incentivize the placement of human genome variation data into the public domain. human mutation. 33(11):1503–1512. https://doi.org/10.1002/humu.22144. pimentel jf, murta l, braganholo v, freire j. 2021. understanding and improving the quality and reproducibility of jupyter notebooks. empirical software engineering. 26(4):65. https://doi.org/10.1007/s10664-021-09961-9. [accessed 2022 jun 25]. reality check on reproducibility. 2016. nature. 533(7604):437–437. https://doi.org/10.1038/533437a. [accessed 2022 jun 24]. resnik db, shamoo ae. 2017. reproducibility and research integrity. accountability in research. 24(2):116–123. https://doi.org/10.1080/08989621.2016.1257387. [accessed 2022 jun 24]. rosenthal, d. 2019. seeds or code?. dshr’s blog, nov. 19, 2019. https://blog.dshr.org/2019/11/seeds-or-code.html rule a, birmingham a, zuniga c, altintas i, huang s-c, knight r, moshiri n, nguyen mh, rosenthal sb, pérez f, et al. 2018 oct. ten simple rules for reproducible research in jupyter notebooks. arxiv:181008055 [cs]. [accessed 2022 feb 1]. http://arxiv.org/abs/1810.08055. rule a, tabard a, hollan jd. 2018. exploration and explanation in computational notebooks. in: proceedings of the 2018 chi conference on human factors in computing systems. montreal qc canada: acm. p. 1–12. https://doi.org/10.1145/3173574.3173606 [accessed 2023 jan 27]. smith am, niemeyer ke, katz ds, barba la, githinji g, gymrek m, huff kd, madan cr, mayes ac, moerman km, et al. 2018. journal of open source software (joss): design and first-year review. peerj computer science. 4:e147. https://doi.org/10.7717/peerj-cs.147. [accessed 2022 jun 25]. stodden v, mcnutt m, bailey dh, deelman e, gil y, hanson b, heroux ma, ioannidis jpa, taufer m. 2016. enhancing reproducibility for computational methods. science. 354(6317):1240–1241. https://doi.org/10.1126/science.aah6168. [accessed 2023 sep 28]. stodden v, seiler j, ma z. 2018. an empirical analysis of journal policy effectiveness for computational reproducibility. proceedings of the national academy of sciences. 115(11):2584–2589. https://doi.org/10.1073/pnas.1708290115. [accessed 2023 sep 28]. sullivan i, dehaven a, mellor d. 2019. open and reproducible research on open science framework. current protocols essential laboratory techniques. 18(1):e32. https://doi.org/10.1002/cpet.32. [accessed 2022 jun 25]. thain d, ivie p, meng h. 2015. techniques for preserving scientific software executions: preserve the mess or encourage cleanliness? proceedings of the 12th international conference on digital preservation. https://hdl.handle.net/11353/10.429560 [accessed 2022 jun 24]. wang j, li l, zeller a. 2020. better code, better sharing: on the need of analyzing jupyter notebooks. in: proceedings of the acm/ieee 42nd international conference on software engineering: new ideas and emerging results. seoul south korea: acm. p. 53–56. https://doi.org/10.1145/3377816.3381724 [accessed 2023 jan 27]. wing jm. 2011. computational thinking. 2011 ieee symposium on visual languages and human-centric computing (vl/hcc).:3–3. https://doi.org/10.1109/vlhcc.2011.6070404. [accessed 2023 sep 28]. about the authors adrienne vandenbosch is the reference services coordinator at the university of colorado boulder working to assess library services. adrienne has a specialization in stem liaison librarianship as a graduate from the cohort for the collaborative analysis liaison librarianship instructional project. at the university of denver, she has also worked on projects for the massive texts lab. keith e. maull is a data scientist and software engineer at the national center for atmospheric research in boulder colorado, usa, where he has held a position for 10 years in the ncar library exploring the scientific research impacts of the organization. his research lies at the intersection of open science, open data and open analytics and he has spent significant time exploring the value and impact of computational narratives and jupyter on 21st century science. matthew mayernik is a project scientist and research data services specialist in the ncar/ucar library. he is also the deputy director of the ncar library. his work is focused on research and service development related to research data curation and digital scholarship broadly. he has taught research data curation courses at the university of denver and university of washington. he is the joint editor-in-chief of the data science journal. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – isbn and qr barcode scanning mobile app for libraries mission editorial committee process and structure code4lib issue 13, 2011-04-11 isbn and qr barcode scanning mobile app for libraries this article outlines the development of a mobile application for the ryerson university library. the application provides for isbn barcode scanning that results in a lookup of library copies and services for the book scanned, as well as qr code scanning. two versions of the application were developed, one for ios and one for android. the article includes some details on the free packages used for barcode scanning functionality. source code for the ryerson ios and android applications are freely available, and instructions are provided on customizing the ryerson application for use in other library environments. some statistics on the number of downloads of the ryerson mobile app by users are included. by graham mccarthy and sally wilson introduction ryerson university is a mid-sized urban campus, located in the multicultural heart of toronto, canada.  it is known for innovative programs built on the integration of theoretical and applied learning.  more than 28,000 undergraduate and 2,000 graduate students are enrolled in over 95  programs that are distinguished by a professionally focused curriculum and strong emphasis on excellence in teaching, research and creative activities. ryerson is also a leader in adult learning, with the largest university-based continuing education school in canada. the library & archives has a staff of more than 85 including 25 librarians. the collection includes over 600,000 books, 1,700 hard copy serials and an impressive electronic resources collection of over 33,000 full text journals. the library has many innovative and supportive services, including a strong reference and instruction presence, which is increasingly mobile and virtual. a rapidly expanded research mandate complements the university’s traditional undergraduate focus with new frontiers for emerging digital library support. objectives the ryerson university mobile app was initially conceived to leverage the use of mobile phones to provide improved access to the library’s resources for patrons in any location. by taking advantage of the built-in camera, which is available in most modern mobile phones, we were able to provide an easy way for patrons to check library item availability by scanning the isbn of a book (rather than laboriously keying a title or isbn into the mobile version of our catalogue). to reach the widest possible audience, the app was enhanced with additional functionality to provide a convenient way for students to store their login credentials for quick access to the password-controlled portions of the popular ryerson mobile website. qr codes the idea to incorporate a qr code scanner into the app arose from the library’s experimentation with qr codes as an alternative method of accessing library resources.  the library had used qr codes in a minor way to link users to a downloadable audio tour of the library and had experimented with using them on bookmarks and for contests.  in the fall of 2010 the library added qr codes to its catalogue so that patrons could use their phones to capture title, call number and location information about the item being viewed.  this information could then be used later to locate the item in the library.  prior to introducing qr codes in the catalogue the library students had been using a text messaging functionality as a method of transferring this same information to a patron’s phone.  this sms service was well-received and was increasing in popularity, so the logical next step was to expand the service via qr codes. qr codes in the catalogue qr codes are incorporated into the catalogue records on the fly using javascript, which extracts title, call number and location information from the bibliographic record and sends it to the google charts tool api (see http://code.google.com/apis/chart/docs/gallery/qr_codes.html). a qr code is returned for insertion into the display of the bibliographic record. it was logical that since we had qr codes in the catalogue, we should also provide an app that could be used to interpret them. technical framework library record finding script the barcode scanning application is a fairly simple concept. using a mobile device’s camera feature, the application takes temporary snapshots and looks for numeric, alphanumeric, or two-dimensional barcode data. if a barcode is found, the application stops looking and interprets the symbol into its representative data.  if the data contains text, it will be displayed on screen; if the data contains a url, the web page will be launched automatically by the application and if the data contains a 10, or 13 digit isbn number, it will send that isbn off to a php script running on a library server to process the request and find related items.  figure 1 shows an overview of the data processes. figure 1. system interaction diagram this script connects to the xisbn web service provided for free non-commercial use by worldcat (oclc, http://xisbn.worldcat.org) and looks for related isbns.  the script then loops through all of the isbns it has tabulated and checks the library catalogue for item availability. finally, the script does a google books search for the isbn to retrieve the title, author, book jacket and google books full text and preview availability. the script then returns an xml document to the requesting mobile device with information about the resource (title, author, book jacket) and its item availability within the library. returned xml <results count="3" libcount="2" message="ok"> <record> <title>programming in objective-c 2.0</title> <author>stephen kochan.</author> <location>9th floor</location> <callno>qa76.73.o115 k63 2009</callno> <availability>due 04-24-11</availability> <isbn>9780321566157</isbn> <recordno>b1926238</recordno> </record> <record> <title>programming in objective-c 2.0</title> <author>stephen kochan.</author> <location>internet resource</location> <callno>qa76.73.o115 k63 2009</callno> <availability>online</availability> <isbn>9780321605559</isbn> <recordno>b2027696</recordno> </record> <record> <bib_key>9780321566157</bib_key> <info_url>http://books.google.com/books?id=4-gnnqaacaaj&source=gbs_viewapi </info_url> <preview_url>http://books.google.com/books?id=4-gnnqaacaaj&source=gbs_viewapi </preview_url> <thumbnail_url>http://bks8.books.google.com/books?id=4-gnnqaacaaj&img=1&sig=acfu3u051un7n4zvc2oroeewopmgcqg4xg </thumbnail_url> <preview>noview</preview> <embeddable/> </record> </results> figure 2. sample xml code returned for a resource . response from http://news.library.ryerson.ca/api/isbnsearch.php?isbn=9780321566157 for “programming in objective-c 2.0” by stephen kochan.  <results count="0" libcount="0" message="unrecognized isbn"> </results> figure 3.sample xml code of non isbn item. response from http://news.library.ryerson.ca/api/isbnsearch.php?isbn=9780431566157. <results count="2" libcount="0" message="item not available in ryerson university library"> <record> <title>secret historian</title> <author>justin spring</author> <isbn>9781429932943</isbn> </record> <record> <bib_key>9781429932943</bib_key> <info_url>http://books.google.com/books?id=4hy7aqaaiaaj&source=gbs_viewapi </info_url> <preview_url>http://books.google.com/books?id=4hy7aqaaiaaj&cd=1&source=gbs_viewapi </preview_url> <thumbnail_url>http://bks9.books.google.com/books?id=4hy7aqaaiaaj&printsec=frontcover&zoom=5&sig=acfu3u0_qt0qttepgmxyiycizilmarxdqg </thumbnail_url> <preview>noview</preview> <embeddable/> </record> </results> figure 4. sample xml code for valid isbn, but not available at the library. figure 2 shows an example of the data the mobile device would receive from the library record finding script if the library had a copy of the item.  in this example, there are two copies of the book, one due back on april 24th 2011 and one available online.  the current version of the application does not fully utilize the google books preview, info and full text code, but there are plans to include this functionality in a subsequent version.  figures 3 and 4 show what the xml would look like for isbn searches that are invalid and return no results, respectively.  the ‘message’ attribute of the ‘results’ element provides the mobile application with information to display the user if no items are found in the library (i.e. if the ‘libcount’ attribute is 0). record finding script modification to modify the isbnsearch.php script for your institution, you need to point the record look-up url to your own catalogue in two specific locations. line 45 : $html = file_get_contents("http://catalogue.library.ryerson.ca/search/i?search=".$similarisbn."&submit=search"); line 64 : $html = file_get_contents("http://catalogue.library.ryerson.ca".$newurl); line 64 accounts for the potential case when two records are returned for one isbn. lines 71 to 101 perform screen scraping to get the floor, call number and item availability from the catalogue.  this section of code will need to be customized for your specific institution if you are not an innovative interfaces catalogue user. mobile applications overview the following sections outline the two application frameworks and point out their key differences.  this article assumes a basic knowledge of both objective-c and java and will not go into much detail regarding basic app development, acquiring developer accounts, or application publishing.  the notes section of this article provides links to the resources that we used to accomplish these tasks. we have developed this barcode scanning application for ios and android devices, with a blackberry version coming in the near future.  the ios application has been written in objective-c and the android version in java.  figure 5 shows three views from the application, two from the android version and one from the iphone version.  aside from a few variations, both applications look and feel the same.  figure 5a shows an example of what the application looks like once launched from the home screen.  the majority of the screen is a webview (see figure 5b) which calls an html5 mobile version of our library website (http://m.library.ryerson.ca).  the bar across the bottom of the application is the native component which contains two arrows for moving forward and backward within the application and the barcode scanner.   figure 5c shows the results from scanning a library book barcode. figure 5a. application layout (android) figure 5b. layout with webview (android) figure 5c. isbn scanned item found in library catalogue (ios) android framework the google android app uses the zxing barcode scanning application to read the qr and isbn codes.  zxing was chosen because it is a free, open source, multi-format image processing library written in java which made the integration into the android application fairly easy to accomplish. our application calls a function from zxing which opens the application, or if the user does not have the zxing application, prompts the user to download the free application from the android market.  we chose to implement zxing as a helper application to ours because it was intended to be used this way by the zxing developers and includes an integration library specifically for this purpose.  moving between applications on the android does not really disrupt the user experience because of android’s fast app switching support, where the user can move back and forth between applications seamlessly. the following code is the one line that attempts to initiate the barcode scanning functionality. if the user does not have the barcode scanning application, it prompts the user to install it from the android market. file: rmobile.java line 211: intentintegrator.initiatescan(rmobile.this, "install barcode scanner?", "this application requires you to install 'barcode scanner'. would you like to install it?", "yes", "no"); once an isbn or qr code has been scanned, the zxing application exits and a listener running within our application automatically takes over and returns the results to the screen. the following code is the function that triggers off the listener and handles the incoming data from the barcode scan.  the onactivityresult method shows that the resulting data is identified based on its content (i.e. string content, a url, or isbn).  if it is a string of text, it displays the data on screen.  if it is a url, it instructs the webview to display the web page.  if it is numeric data, it sends the isbn to the library’s record finding script which returns an xml feed containing the item availability within the library. file: rmobile.java line 347 – 387: //handle the result of the barcode reader protected void onactivityresult(int requestcode, int resultcode, intent data) { switch(requestcode) { case intentintegrator.request_code: { if (resultcode != result_canceled) { intentresult scanresult = intentintegrator.parseactivityresult(requestcode, resultcode, data); string result =""; if (scanresult != null) { result = scanresult.getcontents(); } else return; result = result.trim(); //check to see if it is a url if(result.trim().matches("^http.*://.*")) { if(isonline()) mwebview.loadurl(result.trim()); else mwebview.loadurl("file:///android_asset/noconnection.html"); } //check to see if it is a isbn (checks if it is numeric) else if (result.trim().matches("[0-9]*")){ intent intent = new intent(rmobile.this,barcoderesult.class); //pass the isbn to the barcoderesult form bundle bundle = new bundle(); bundle.putstring("isbn",scanresult.getcontents()); intent.putextras(bundle); startactivity(intent); } //assume that it must be a string else{ alertbox.show("found message", result, this); } } } break; } } android application customization to modify the android application for your own institution update the following: change the mobileurl string constant to point to your mobile website file: rmobile.java line 51 (objective-c): final private string mobileurl = "http://m.library.ryerson.ca/"; change the url variable to point to the updated isbnsearch.php script that you have modified for your own catalogiue file: xmlparser.java line 38: url url = new url("http://news.library.ryerson.ca/api/isbnsearch.php?isbn="+isbn); ios framework the ios version operates in a similar manner to the android version, except it does not rely on an external barcode scanning application to read the qr and isbn barcodes.  the ios version uses the zbar barcode reader ios library embedded within the ios app. when users download this application they are not required to download a separate barcode reader application.  the ios application uses the zbar libraries instead of zxing because there was already an open source library created in objective-c for the iphone released under a gnu lgpl 2.1 license for open source and commercial projects.  this gave us a good head start in the applications development. the zbar sdk integration tutorial (see notes section) describes step by step how to integrate the library into your ios application. the following code sample describes the actions associated with launching the barcode scanner from the application.  if the user’s device contains a camera, it creates a view controller called reader, configures the scanner to process the image for specific barcodes, and presents the reader to the user. file: specialappsviewcontroller.m (obj-c) lines 90 – 120: -(ibaction) barcodescanner:(id)sender { //see if scanner is supported on this device if ( [self scanningavailable] ) { //create the view controller object and delegate the reader to launch from self zbarreaderviewcontroller *reader = [zbarreaderviewcontroller new]; reader.readerdelegate = self; //create the scanner object which eventually captures the barcodes zbarimagescanner *scanner = reader.scanner; //disable rarely used i2/5 barcode types to improve performance [scanner setsymbology: zbar_i25 config: zbar_cfg_enable to: 0]; //set the video capture device settings, to fix bug in iphone 3 avcapturevideodataoutput *videocapture = (avcapturevideodataoutput *) reader.readerview.capturereader.captureoutput; videocapture.videosettings = nil; // present the controller view to the user, and release it from memory. [self presentmodalviewcontroller: reader animated: yes]; [reader release]; } else { //scanner not supported uialertview *myalert = [[uialertview alloc] initwithtitle:@"scanner not supported" message:@"your mobile device does not support an auto focus camera and cannot use the barcode scanning application. we are sorry for the inconvenience." delegate:self cancelbuttontitle:@"ok" otherbuttontitles:nil]; [myalert show]; [myalert release]; } } //end function when the barcode scanner has identified barcode data, it calls the imagepickercontroller method in specialappsviewcontroller.m (line 125).  the imagepickercontroller works along the same lines as the android version.  it determines the type of data coming in and processes it accordingly.  one added feature to this application is that the scanned image is also shown to the end user for a more visually pleasing experience.  the actual code for this function is fairly involved and has been well documented to explain each action in-depth. ios application customization to modify the ios application for your own institution, update the following: change the urladdress variable to point to your mobile website file: ryersonmobileviewcontroller.m line 113: nsstring *urladdress = @"http://m.library.ryerson.ca/"; change the rulaisbnsearch variable to point to the updated isbnsearch.php script that you have modified for your own catalogue. file: specialappsviewcontroller.m line 179: nsstring *rulaisbnsearch = [nsstring stringwithformat:@"http://news.library.ryerson.ca/api/isbnsearch.php?isbn=%@", datafromsymbol]; change the bookjacket_url to point to a default book jacket image of your choosing file: specialappsviewcontroller.m line 187: nsstring *bookjacket_url = @"http://news.library.ryerson.ca/api/noimage.png"; reflections on applications developing these applications for android and ios proved fairly straightforward to accomplish and we did not have many problems along the way.  the barcode scanning frameworks have example code for android and ios devices and lots of documentation and support forms.  online tutorials and the api documentation proved very useful for solving integration and syntax issues. one of the major drawbacks we have had to deal with is the regular firmware updates for ios devices.  our first issue was with the migration from ios3.0 to ios3.1. we unknowingly used a private library to read in image data from the camera. this was acceptable for ios 3.0 but not ios 3.1, and caused our application to be rejected from the app store. we experienced another issue when moving from ios 3.1 to the ios 4.0 firmware. apple made significant changes to their sdk, requiring us to make a lot of small modifications to ensure the application works on most ios device types.  this will be an unfortunate ongoing maintenance issue that we will continue to deal with.  that said, it will cause us to continue to develop and improve the applications. integration with university-wide mobile platform for the application to be adopted campus-wide, we needed to provide functionality that would benefit the students’ entire campus experience, not just their library access.  from surveys we conducted in ‘08 and ‘09, it was apparent that students did not just want access to library services and resources via their mobile devices. they wanted a holistic university experience because they view the university as one large institution.  therefore, we provided the barcode scanning application as a feature of the university-wide mobile application which provides single-click authentication into their profile, access to their courses, booking of study rooms, departmental hours of operations, access to the campus directory, campus maps, and meal and photocopy plan transaction history and balances. the single-click authentication feature has been removed from the open source code to minimize the complexity. usage statistics it has proven somewhat difficult to measure the success of the project as the statistics provided by the app store and android market only indicate if an application was installed and uninstalled, but not usage.  to determine the usage, we need to consider the statistics for the mobile website.  however, this does not truly indicate whether users are using the native app or the web app.  the information below shows the statistics captured for the two native applications from their launch date to a similar end point and the web application from this past academic year. android market statistics (aug 18th 2010 to mar 16th 2011) 343 total downloads 202 active installs 12 reviews 4 comments apple app store statistics (oct 4th 2010 to mar 13th 2011) 4,180 application downloads 3,099 from canadian itunes accounts mobile web statistics (sept 1st 2010 to mar 16th 2011) 6,402 logged-in users 20,060 anonymous users 14 applications used 157,731 times by users. device usage 58% ios 14% blackberry 10% android 18% desktop browsers and other mobile devices conclusion since the launch of the applications, we have received expressions of interest and requests for information from other units within the university, external institutions, and public and academic libraries.  an adapted version of the application is now being used by the school of continuing education at ryerson as part of their advertising campaign that features qr codes, and the ryerson math centre is piloting a project that will use the barcode scanning application to scan student ids to expedite signing them into workshops and exams.  in february 2011, the significance of this application to the ontario library community was recognized when it won the ontario library and information technology association’s award for technological innovation. the source code for this application can be downloaded from http://www.ryerson.ca/library/mobile/rula_barcode_scanner.zip for more information please contact graham mccarthy at gmccarthy@ryerson.ca. links android developer’s guide – http://developer.android.com/guide/index.html apple ios dev centre – http://developer.apple.com/devcenter/ios/ google charts tool api – http://code.google.com/apis/chart/ gnu lgpl – http://www.gnu.org/licenses/old-licenses/lgpl-2.1.html qr codes (ryerson library) – http://www.ryerson.ca/library/qr/ ryerson library’s mobile survey results: 2008 – http://www.ryerson.ca/library/msurvey/index.html 2009 – http://www.ryerson.ca/library/msurvey2009/index.html ryerson mobile – http://m.ryerson.ca ryerson university library & archives mobile – http://m.library.ryerson.ca xisbn web service – http://xisbn.worldcat.org/ zbar – barcode scanning library – http://sourceforge.net/projects/zbar/ zbar sdk integration – http://zbar.sourceforge.net/iphone/sdkdoc/ zxing – barcode scanning application – http://code.google.com/p/zxing/ acknowledgements we would like to recognize steven marsden (steven.marsden@ryerson.ca) for his contribution to the barcode scanning application’s development. about the authors graham mccarthy (gmccarthy@ryerson.ca) is the innovative technologies librarian at ryerson university. sally wilson (swilson@ryerson.ca) is the web services librarian at ryerson university. subscribe to comments: for this article | for all articles 10 responses to "isbn and qr barcode scanning mobile app for libraries" please leave a response below: jonathan rochkind, 2011-04-12 nice article. you provide statistics on downloads of your mobile apps. however, since every isbn barcode scan ends up contacting the isbnsearch.php script server-side, i wonder if it would be possible to look at request log for that script to see how many isbn lookups have been done with a mobile app? that would be an awfully useful marker of actual adoption if you could get them. because someone downloaded a mobile app doesn’t neccesarily mean they used it often (or even at all!). mobile « pintiniblog, 2011-04-13 […] isbn and qr barcode scanning mobile app for libraries [résumé] la bu de l’université de ryerson a développé une application portable qui permet de scanner aussi bien un code isbn qu’un code qr (et d’afficher les informations de disponibilité et de localisation). deux versions de l’application sont prévues, une pour ios et une pour android. les codes de l’appli sont en open source et peuvent être adaptés à d’autres bu. […] quora, 2011-04-19 how important should mobile devices be in a museum?… you should be aware of what’s going on, and you should take some preliminary steps to support mobile devices within your normal procurement processes. augmented reality, which has awesome applications for museums is still in it’s infancy, smartphone … reference 2.0 « colin and company, 2011-05-26 […] and wilson, sally “isbn and qr barcode scanning mobile app for libraries“ code4lib journalissue 13, […] quora, 2011-10-03 what functionality would be in your dream library app?… congratulations! i’m really excited that you have the resources and ability to create a custom library app. we’re using a mobile website via libguides and summon as our primary mobile interface. 1. get the results card view right. a small thumbnail, … john bestevaar, 2011-11-25 hi people not a programmer so cant comment on the code but i love the idea of scanning book barcodes using a mobile phone application. we have a small secondhand bookshop run by volunteers. the proceeds go to a community org that restores native forests on public lands. we are looking for a way to let our customers see what we have in stock via a website. regards john bestevaar julien, 2012-11-24 hi graham, thanks for this nice article, but the download link is broken. could you fix it ? best regards, julien steven marsden, 2012-12-11 hi everyone, the new url for zip file is the following http://library.ryerson.ca/wp-content/uploads/rula_barcode_scanner1.zip cheers, steven library to barcode scanner : android community for application development, 2012-12-21 […] isbn & qr barcode scanning mobile app for libraries […] ganesh, 2014-09-18 hi every one. im just learner. i’m having one form with isbn number field. i would like to capture isbn number using my mobile camera and store the isbn number respective my form field book isbn number. can you please guide me or let me know the process. regards ganesh leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – web-scraping for non-programmers: introducing oxpath for digital library metadata harvesting mission editorial committee process and structure code4lib issue 38, 2017-10-18 web-scraping for non-programmers: introducing oxpath for digital library metadata harvesting building up new collections for digital libraries is a demanding task. available data sets have to be extracted which is usually done with the help of software developers as it involves custom data handlers or conversion scripts. in cases where the desired data is only available on the data provider’s website custom web scrapers are needed. this may be the case for small to medium-size publishers, research institutes or funding agencies. as data curation is a typical task that is done by people with a library and information science background, these people are usually proficient with xml technologies but are not full-stack programmers. therefore we would like to present a web scraping tool that does not demand the digital library curators to program custom web scrapers from scratch. we present the open-source tool oxpath, an extension of xpath, that allows the user to define data to be extracted from websites in a declarative way. by taking one of our own use cases as an example, we guide you in more detail through the process of creating an oxpath wrapper for metadata harvesting. we also point out some practical things to consider when creating a web scraper (with oxpath). on top of that, we also present a syntax highlighting plugin for the popular text editor atom that we developed to further support oxpath users and to simplify the authoring process. by mandy neumann, jan steinberg, and philipp schaer 1. introduction and motivation building up new collections for digital libraries is an expensive and demanding task. not only do digital content curators need to assess many different data sources intellectually but also need to invest a lot of time and effort to extract the available data sets. usually this is done by coding custom data handlers or conversion scripts with languages like perl or python. while this might be a trivial task for programmers, librarians and content curators are most likely overwhelmed with such a task and its complexity and pitfalls. one of the largest digital libraries that lead the way in digitizing this data extraction process is the dblp computer science bibliography [1], which built up their process chain to heavily rely on automatic metadata extraction from many different sources. ley (2009) gave an excellent overview and insight into all the traps one might fall. other examples are disciplinary open access repositories like the social science open access repository (ssoar) [2] that gather available full text items from different partner organizations like publishers, research institutes, and individuals. while some of these partners or content providers are technically and organizationally able to provide a clean set of parsable metadata, many do not have the necessary technical manpower to prepare these metadata sets. ironically, many small and medium-size publishers do have a web page or an online catalogue. the same is true for many research institutes or funding agencies. although managing and preparing high quality metadata is not their core business they are somehow able to present their content online in a rather structured form. this is an opportunity we would like to make use of and present a web scraping tool that does not demand that digital library curators program custom web scrapers from scratch but instead use a mighty, but still light-weight, toolkit that does not force them to learn to program. data curation is a typical task that is done by people with a library and information science background. these people are well-aware of markup languages like xml or html and maybe know a little about doms or processing xml-resources with xpath or xslt. but to code a full-blown web scraper for various sites, they need the support of real programmers. to bypass this involvement of software developers and to empower librarians and documentalists, web scraping toolkits are an excellent way to gather content without diving deep into programming issues like development environments or dependency management. one of these toolkits is oxpath that is presented in the second section of this article. at a first peek oxpath might look rather complicated, but keep in mind: the target audience for this tool is not full stack developers but librarians and documentalists. we have seen these people getting ready to scrape their first web resources after just a short tutorial and some demos. even lis students without any mentionable previous knowledge of web scraping and html/xml were able to write their first lines of oxpath after 2 hours. we think that their general knowledge of structure and presentation of library-related material helps them to get a handle on the task. from our understanding, this is a perfect foundation for oxpath as it is easy to install and allows users to dive into web scraping right away without learning how to install and maintain a development environment with dependencies, etc. therefore we would like to introduce oxpath to a broader community and demonstrate the workflow to gather web data for digital libraries without being a programmer. on top of that we would like to present a syntax highlighting plugin for the popular text editor atom we developed to further support oxpath users and to simplify the authoring process. 2. oxpath – a language for web scraping in cases where a metadata provider cannot provide an api or similar tool, one will have to gather the data from their public website, as it is displayed to the human user. this could be done manually, which can be rather labour-intensive, or one would need to program a scraper specifically tailored to the target website that extracts the data for you. for non-programmers, web data extraction tools provide an alternative to writing a complete web scraper from scratch. most of them, such as fminer [3], parsehub [4] or import.io [5], are commercial with rather high costs or subscription fees. another problem with those gui-based tools is that often some advanced features are missing or that one is limited by the user interface that does not allow use of all the features of the underlying scraping software. we want to introduce the free and open-source tool oxpath [6] that allows the user to define the data to be extracted in a declarative way. the language of oxpath is based on the xml query language xpath 1.0 but extends xpath’s capabilities with some elements that allow the simulation of user interaction with a web page and the extraction of data in the course of these interactions. those extensions are, for example, actions like clicking and form filling, a means for iteration to enable navigation through paginated content, and extraction markers. also, in addition to xpath functions, oxpath introduces some additional useful functions to manipulate the data before it is extracted. as an example, listing 1 shows a simple oxpath wrapper for harvesting the result list of a request to google scholar. after entering the search string ’oxpath’ into the search field and pressing enter (line 2), it navigates the paginated result list by repeatedly clicking on the ’next’ button (line 3) and extracts from each result the title and author information into title and authors attributes, respectively (lines 4–5). to extract the author information correctly, the substring-before function is used to catch only the text that occurs before a separator symbol. doc("https://scholar.google.com/scholar?hl=en") /descendant::field()[1]/{'oxpath'}/{pressenter /} /(//a[contains(.,'next')]/{click/})* //div[@class='gs_r']:<paper>[.//h3:<title=string(.)>] [? .//*[@class='gs_a']:<authors=substring-before(.,' ')> listing 1. sample oxpath wrapper for harvesting the result list of google scholar after entering a search term there are many possible usage scenarios for oxpath. for example, we used it for extending existing test collections in information retrieval with additional metadata (schaer and neumann 2017). in the field of digital libraries, oxpath can be used for harvesting bibliographic metadata from publishers’ websites, as presented by michels et al. (2017) for dblp. here, we want to guide you in more detail through the process of creating an oxpath wrapper for metadata harvesting, by taking one of our own use cases as an example. 3. hands on and walkthrough some publishers provide oai-pmh endpoints or other apis, but not all of them do, and in those cases in which an endpoint is available, sometimes not all data is provided there – but there may be more data that is displayed on the human-oriented interface, i.e. the web page of a journal. suppose you want to store metadata about a new working paper series in your digital library. the publisher has agreed to collaborate with you, but is not able to provide you their publications’ metadata neither via api nor a file dump or similar method. so you decide to write an oxpath wrapper to harvest it directly from the publisher’s website. before starting to write this oxpath wrapper, it’s worth taking a deeper look at the data. the attention should be on checking if the minimal set of metadata needed for the digital library is really available. the next look should be at the source code of the website to get information about its structure. it’s a good sign if every metadata element in each dataset is formatted or marked in the same manner. composed fields without a recognizable delimiter between the individual information can become quite complicated or even impossible to parse. to easily find the location of these fields in the code, most of the web browsers offer a possibility to jump to a clicked element’s source code location [7]. in the following part, the website of the bonn international center for conversion (bicc) [8] serves as an example. our goal is to get all working papers’ metadata including their open access full text links for importing them into our digital library. first of all, an entry web page is needed on which all the relevant publications are listed. there’s a browsing category for ‘working papers’ that leads us to such a list (https://www.bicc.de/publications/by-category/all-issues/category/working-paper/). with a text editor of our choice, we create a new text file called bicc_wp.oxp in which we can start to write our oxpath script. we define our starting page: doc("https://www.bicc.de/publications/by-category/all-issues/category/working-paper/") in the source code of the website we identify the element that contains the list of the publications. figure 1. source code excerpt of the website in question. now we tell oxpath that this is the place to start: //div[@class="search-publication-results"] in this div each document has its own container. in this case it’s another div. for each of these oxpath has to open an ‘item’ xml tag in our result document. ./div[@class="search-publication-single"]:<item> for each of the items in the list, to get to the more detailed view of the corresponding metadata the ‘more’ button has to be clicked: //p/a/{click/} on this new page our analysis made evident that every metadata element we want to harvest is a child node of a div with the class name ‘tx-bicctools-pi3’. therefore we select this node to be our starting point for the new page: //div[@class="tx-bicctools-pi3"] from there we collect the relevant data fields and output them into xml tags. [? ./h1:<title=string(.)>] [? ./div[@class="publicationfield cover-field"]/img: <number=@title>] [? ./div[@class="publicationfield date_from"]: <date_from=string(.)>] [? ./div[@class="publicationfield abstract-field"]: <abstract=string(.)>] [? ./div[1]/ul/li/a: <url=@href>] [? ./div[1]/ul/li/a: <lang=string(.)>] [? ./div[2]/ul/li/a: <author=string(.)>] [? ./div[3]/ul/li/a:<topic=string(.)>] [? ./div[4]/ul/li/a:<geocontext=string(.)>] when running the oxpath script now, the output looks a bit unclean. many of the field contents are not yet in the form normally needed for a digital library import. here another advantage of oxpath comes into play. you can make use of all the standard xpath functions like normalize-space, substring-before, substring-after, replace and so on. in addition, there is some static information that is the same for every publication on the website. this information can be added in one go as well: [? ./h1:<title=string(.)>] [? ./h1:<publisher="bicc bonn international center for conversion">] [? ./h1:<doctype="working paper">] [? ./div[@class="publicationfield cover-field"]/img:        <number=substring-before(substring-after(@title, 'wp_'),'.jpg')>] [? ./div[@class="publicationfield date_from"]:         <date_from=translate(normalize-space(substring-after(.,'release date: ')),'-','/')>] [? ./div[@class="publicationfield abstract-field"]: <abstract=normalize-space(.)>] [? ./div[1]/ul/li/a: <url=concat('https://www.bicc.de/', @href)>] [? ./div[1]/ul/li/a: <lang=string(.)>] [? ./div[2]/ul/li/a: <author=string(replace(.,'pd dr. |prof.  |dr. ', ''))>] [? ./div[3]/ul/li/a:<topic=string(.)>] [? ./div[4]/ul/li/a:<geocontext=string(.)>]   after adding the static information and introducing the additional xpath functions in the expression, the resulting output is cleaner. the required content is present and fit for further processing. below is an excerpt of the final output: <item>    <title>local security-making in kyrgyzstan and tajikistan</title>    <publisher>bicc bonn international center for conversion</publisher>    <doctype>working paper</doctype>    <number>5_2016</number>    <date_from>2016/06</date_from>    <abstract>in cooperation with researchers in tajikistan and kyrgyzstan, bicc (bonn international center for conversion) is conducting a three-year research project on everyday security practices in central asia ...</abstract>    <url>https://www.bicc.de/uploads/tx_bicctools/working_paper5-1_01.pdf</url>    <lang>english</lang>    <author>marc von boemcken</author>    <author>dr. conrad schetter</author>    <author>hafiz boboyorov</author>    <author>nina bagdasarova</author>    <author>joomart sulaimanov</author>    <topic>migration</topic>    <geocontext>kyrgyzstan</geocontext>    <geocontext>tajikistan</geocontext> </item> 3.1 some practical considerations although, as we hope to have shown, writing an oxpath expression is relatively straightforward after a little training, there are some things that have to be considered when you want to use them in production. first of all, and this is true for web scraping in general, you have to be aware of the fact that the data you get might be messy. metadata on web pages are meant to be displayed to a human user. for example, in the case of author names, the human readable form often differs from the format you would normally need to have in your digital library (e.g., prof. dr. john van doe vs. doe, john van). there may also be some metadata that is not displayed at all, e.g., unique identifiers for each dataset/publication, so you need a strategy for that. in our use case, the url of the dataset can be used as a substitute. it’s not really handy, but unique and therefore a suitable working solution. also, the volume/year and the issue of the present series cannot be reliably determined. in our example script, we try to extract these details from the filename of the front page images. for more reliable content, it is inevitable to get the information elsewhere. another point is that one should make sure that the website’s owner is well informed about the oxpath web scraping, either by a general allowance from them or a targeted agreement. crawling web resources in general should respect some rules of politeness as web servers have both implicit and explicit policies regulating the rate at which a crawler like oxpath can visit them. an example might be to respect the robots.txt or to try to scrape during off-peak hours. please be aware that these politeness policies must be respected as too many actions on a page in a short period of time can overload their servers and may cause your ip address to be blocked. it also will distort the web server statistics (hits and views of every single dataset). there are possibilities in oxpath to slow the script processing (e.g., the ‘click pause’ and the ‘wait’ command). this way we can trick web servers as they cannot distinguish between the web scraping process and a human web surfer anymore. still it’s easier and advisable to just contact the web site operators. oxpath trades speed for memory efficiency. thus, an oxpath script in some cases may take hours to days to run because internally the whole web page is rendered and processed to allow a human-comparable extraction mechanism. on the other hand, oxpath maintains a rather small memory footprint compared to other tools made for extracting bulk data from web pages, as has been proven by furche et al. (2013). in production use, one may try to distribute the whole scraping process on parallel threads but this is not a built-in feature. finally, one should be aware of the fact that websites are far more likely to change than api endpoints. it’s not only the data itself that may be changed or even deleted, but also the structure and design of the web page. luckily enough, nowadays design and structure are, in general, neatly separated from each other. as oxpath addresses elements via their structural position (in the dom tree) rather than by design features, it is, in general, immune to changes in the design of a page. nevertheless, changes in the structure may also happen, which is why a given wrapper might have to be updated from time to time. in the future, we plan to provide users with tool support for scheduling and monitoring harvesting processes which also includes notification in case of such events. in spite of its limitations, oxpath is a powerful and useful tool for harvesting semi-structured data from web resources. we think it is especially well suited for non-programming librarians with a general knowledge of xml technologies. in fact, all you need for a first start is the oxpath binaries, a java runtime environment installed on your system and a plain text editor to write the expression. however, in our ongoing project we are working on providing more tool support around it. a first step in doing so was to implement a language package for the free and open-source text and source code editor atom. 3.2 syntax coloring for oxpath as stated before, an oxpath expression can be written using any text editor. nowadays, a lot of text editors also include support for different formal languages (like programming or markup languages) by providing syntax highlighting for them. syntax highlighting, or syntax coloring, does not have any function other than assisting the human reader and writer of source code. when different structural elements and keywords are visually discriminated by different colors it helps the reader with orientation in pre-written code. also when writing code, highlighting helps spotting potential (lexical) errors – e.g., when a keyword is misspelled, it does not receive the keyword-specific coloring. as syntax highlighting is especially helpful for newcomers to a language, we decided to implement a language package for the atom editor to support and heighten the uptake of oxpath [9]. we chose atom because it is a free, open-source editor available on the major operating systems with an active community of developers, maintainers and users behind it. extensive information on how to add support for a new language in atom, however, is unfortunately missing from the current version of the documentation [10]. a helpful resource is a blog post by jérémy heleine [11] that provides a complete walk-through for creating a simple syntax highlighting package and thus serves as a good starting point. there are three important components of an atom grammar. first, it specifies file extensions such that every time a file is saved with a specific extension, the corresponding language package is automatically activated. second, it defines a set of rules in the form of regular expressions that match specific language constructs. third, it maps the text matched by a rule to a css class – this information is used by atom’s syntax themes to provide different coloring to elements of different classes [12]. for the file extensions, we defined .oxp and .oxt as those that will trigger the oxpath language package. regarding the matching rules, we first checked the original grammar (which is written in javacc) to get an overview of the most important language constructs. then we decided to create rules for the most important keywords – function names like contains or substring, the action keywords (like click or pressenter), xpath axes like preceding or ancestor, and, of course, the doc statement. strings, delimited by single or double quotes, and numbers are also recognized. finally, we decided that also the extraction markers, as an important component of any oxpath expression, should receive distinct highlighting. figure 2 demonstrates the effect of the developed language package by contrasting the non-highlighted version of our sample oxpath expression, treated as plain text, with the highlighted version. even with this rather short snippet it is apparent that the highlighted version is more readable as different structural elements are visually separated. with more elaborate expressions, we expect the effect to be even more pronounced. we thus hope that our language package for atom will be of help for all users of oxpath, especially for newcomers getting started with the language. figure 2. demonstration of the effect of syntax coloring. above: expression in plain text mode. below: the same expression with associated oxpath grammar. 4. conclusion in this article, we have presented oxpath, a declarative expression language extending xpath for web scraper creation. oxpath is an easy-to-learn, light-weight, and all-in-one technology to gather metadata from web resources. we demonstrated with a real-world example from one of our own use cases how to use it for straightforward extraction of publication metadata from a small publisher’s website. we think that oxpath is a worthy alternative to other web scraping tools especially for librarians, not only because it is free and open source, but also because it provides full control of the extraction process. a basic knowledge of xml and its query language xpath is sufficient for getting started with the oxpath wrapper language – no further programming skills are required. it also requires no installation and only a simple working environment with java and a command line. although, as we’d like to point out, the integration in a java development environment as a library is also possible for more advanced usage. we think oxpath can be useful in many scenarios in the context of digital libraries. although there are still relatively few tools to support the development process, we think that our contribution of a syntax coloring package for a popular text editor is an important step towards lowering the initial hurdle for newcomers to oxpath. there is still room for improvements, and any issues submitted through the issue tracker on github [13] are highly appreciated. in the future, we want to work on further support for oxpath users which might include scheduling and monitoring harvesting processes and/or providing a visual tool for crafting oxpath expressions via the interaction with the target site. acknowledgements this work was supported by deutsche forschungsgemeinschaft (dfg), grant no. scha 1961/1-2. appendix the complete example from this article including a current version of oxpath and instructions on how to run it can be downloaded from: https://github.com/neumannm/c4l-material currently only ubuntu 64bit operating systems are supported, which will change in the future. you will have to have java installed on that system to run oxpath. oxpath is otherwise available for download at http://www.oxpath.org/download/. at the time of writing this article, the current version is 2.0 – a newer version with a slightly different and more advanced command-line interface is in development. also, an exhaustive tutorial on oxpath will be published soon. notes [1] http://dblp.uni-trier.de [2] http://www.ssoar.info [3] http://www.fminer.com [4] https://www.parsehub.com [5] https://www.import.io [6] http://www.oxpath.org [7] in chrome: right-click ? inspect element furthermore, there are extensions to some web browsers that allow to find specific elements via xpath or show the xpath to a given (clicked) location. see firepath for firefox or chropath for chrome. [8] https://www.bicc.de [9] https://atom.io/packages/language-oxpath [10] http://flight-manual.atom.io [11] https://www.sitepoint.com/how-to-write-a-syntax-highlighting-package-for-atom [12] note that this fact gives us only indirect influence on the coloring of elements. it might still be the case that a syntax theme gives the same color to different css classes. [13] https://github.com/neumannm/language-oxpath/issues references furche t, gottlob g, grasso g, schallhart c, sellers a. 2013. oxpath: a language for scalable data extraction, automation, and crawling on the deep web. the vldb journal 22:47–72. doi:10.1007/s00778-012-0286-6. michels c, fayzrakhmanov, r.r., ley, m., sallinger e, schenkel r. 2017. oxpath-based data acquisition for dblp. in: jcdl ’17: proceedings of the 17th acm/ieee-cs on joint conference on digital libraries; 2017 june 19-23; new york, ny, usa. acm. p. 319-320. to appear. ley m. 2009. dblp: some lessons learned. proceedings of the vldb endowment 2(2):1493–1500. doi:10.14778/1687553.1687577. schaer p, neumann m. 2017. enriching existing test collections with oxpath. in: jones gj, lawless s, gonzalo j, kelly l, goeuriot l, mandl t, cappellato l, ferro n, editors. experimental ir meets multilinguality, multimodality, and interaction. proceedings of the 8th international conference of the clef association, clef 2017; 2017 sept 11-14; dublin, ireland. to appear. about the authors mandy neumann is a research associate at the institute of information science at th köln (university of applied sciences) in the information retrieval research group. she studied information processing and linguistics/phonetics at the university of cologne where she received her master’s degree in these subjects. mandy.neumann@th-koeln.de jan steinberg is a research associate and software developer with the department computational social science (css) at gesis – leibniz institute for the social sciences. he graduated his studies in information and communication sciences at th köln (university of applied sciences) with a diplom and in library and information sciences at humboldt university of berlin with a master’s degree. he pursues special interests in information retrieval, open science, and natural language processing. jan.steinberg@gesis.org philipp schaer is professor for information retrieval at the institute of information science, th köln (university of applied sciences) where he teaches courses on information retrieval, database systems, and search engine technology. he studied computer science with special interest in information retrieval and human factors in information systems and graduated at university of koblenz-landau where he received his degree as diplom-informatiker and later his doctorate in computer science. philipp.schaer@th-koeln.de, http://www.schaer.de subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – digitization selection criteria as anti-racist action mission editorial committee process and structure code4lib issue 45, 2019-08-09 digitization selection criteria as anti-racist action by deciding what to digitize in special collections and archives, we choose what narratives to promote, what history to highlight, and what legacies to further. this paper details a new initiative at lsu libraries to integrate diversity and inclusion goals into digitization policies. after reviewing examples of how digitization can be either beneficial or harmful to individuals represented in the historical record, the author uses ibram kendi’s definition of racist policy — that which leads to racial inequalities — as a starting point for exploring how digitization selection can help counteract histories of exclusion. by s.l. ziegler introduction by deciding what, among the many possible choices, to digitize in special collections and archives, we choose what narratives to promote, what history to highlight, and what legacies to further. louisiana state university libraries’ special collections is in the midst of creating new digitization prioritization policies for our special collections, and to inform our decisions we’re looking closely at our legacy of racist and exclusionary practices. specifically we are (1) including diversity and inclusion goals as a standard criteria for digitization selection, and (2) reviewing past restrictions on collections to investigate their role in upholding racial narratives. a note about my whiteness this paper describes a project designed to counteract racism. as such, beginning with a note about the author’s whiteness is essential: i am white. i benefit from the protections that whiteness afford. i regularly talk about racism and anti-racism without being accosted in person or online. this safety is a privilege and a responsibility. i benefit from the hard work of many scholars and practitioners, including (but certainly not limited to) dorothy berry, jarrett drake, fobazi ettarh, april hathcock, sofia leung, tonia sutherland, and baharak yousefi. as a white person in a management role in an academic library, i am in a position to have some say over what portions of our large and diverse collections will be digitized. i fulfill this responsibility by finding ways to operationalize the lessons i learn from these and other thinkers. deciding what to digitize starting in the late 1990s, institutions such as harvard and oxford started publishing digitization criteria policies.[1] a primary focus of these policies is the argument that digitization creates new collections and is thus akin to collection development for analog material. similar to collection development for physical material, decisions related to what material to digitize should consider the institution’s mission statement, collecting scope, and audience. while many institutions currently have some form of decision-making practice related to digitization, public documentation is often hard to find. notable exceptions include dartmouth’s selection policy for digitization projects[2] and university of california’s selection criteria for digitization.[3] both provide digitization selection guidance by applying the overall mission of the respective libraries to the specific considerations of any given project. dartmouth’s policy aims, “to create a consistent, structured approach to reformatting our collections,” and does so by reference to the institution’s core mission. the primary goal of collection development at dartmouth college library is to create collections of enduring value in support of research, teaching, and learning at dartmouth college. the library’s digitization program is designed to distribute its collections widely and to bring them to the attention of scholars and students within and beyond dartmouth, enhanced with tools that will enable transformative uses of those materials.[4] in addition to tailoring needs to specific institutional settings, many common criteria are shared among different institutions. the policies from harvard, university of california, and dartmouth all contain considerations of copyright and access restrictions, preservation of source material, research use of the collection, and projected use of digital content.[5] among libraries specifically tailoring digitization priorities to match institutional goals, duke university libraries offers a significant example. in 2017, drawing on the libraries’ strategic plan as well as the university’s statement on diversity and inclusion, the advisory council for digital collections (acdc) released a call for proposals on the theme of diversity and inclusion. “the diversity and inclusion digitization initiative” states the call “will further demonstrate dul’s commitment to diversity … by focusing digitization resources on improving access to historically overlooked voices and experiences found in duke university library collections.”[6] the call for proposals resulted in seven new digital collections that promote diversity and inclusion and established new workflows for soliciting and prioritizing digitization recommendations from staff across the libraries.[7] lsu libraries is currently re-working our digitization priority policy. much like other libraries, we’re considering copyright, conservation, projected future use, and other criteria; and like duke university libraries we’re drawing heavily from our strategic plan. during this process we’ve learned a lot from the published policies of other libraries as well as the body of literature related to selection for digitization.[8] it matters what we decide to digitize we have also benefited greatly from a growing body of literature related to both the harm that digitization can cause to particular communities, as well as examples of the benefits it can bring. recently, zinaida manžuck undertook a review of the literature related to ethical digitization.[9] this literature, manžuck claims, shows a change in the thinking of professionals working in libraries, museums, and archives. “this shift,” writes manžuck, “is demonstrated by attention to the impact of digitization on the lives/values/behavior/needs of specific communities and people and numerous instances of working for and with communities to develop methods of selecting, presenting, and organizing digitized heritage content online to provide richer, more dynamic, and inclusive engagement with the past.”[10] when the impact of digitization is not acknowledged, people can be harmed. writing about the decision to digitize on our backs, a lesbian porn magazine, tara robinson urges librarians to consider the possible harm to participants in the magazine. “consenting to a porn shoot,” writes robinson, “that would be in a queer print magazine is a different thing to consenting to have your porn shoot be available online.”[11] debates about what information should be shared digitally is an active area of research for material related to indigenous communities,[12] zine librarianship,[13] and many other fields. concurrently, there are many examples of digitization projects that are working to fill gaps in the archival record by telling new stories through digitizing archival records, gathering undertold stories from various institutions via digitization, and offering a digital front door to widely dispersed material. black at bryn mawr, umbra search, and in her own right serve as examples. black at bryn mawr is a project started by bryn mawr students after other students hung a confederate flag on campus. the project is an attempt to “build institutional memory of the college’s black students, faculty and staff from bryn mawr’s founding” to the present day. to do this, material from the college’s special collections and elsewhere are being digitized[14]. umbrasearch.org brings together hundreds of thousands of digitized materials from over 1,000 libraries and archives across the country, and as such serves as a front door to search material related to african american history.[15] in her own right brings together material highlighting women’s struggle leading to the passage of the 19th amendment. the project makes material from multiple institutions available through a common interface.[16] moreover, our digitized collections become the building blocks of ongoing digital projects. as many in the code4lib community know, having collections digitized enables a wide variety of new work, from computational access[17] to deep learning.[18] digitization selection as anti-racist action digitization matters. as we’ve seen in the examples above, it can help further institutional missions. it can cause significant harm to communities when done unreflectively. it can bring together otherwise inaccessible stories when done well. digitization can also counteract long-standing institutional practice. this is what we hope to do at lsu libraries by integrating our diversity and inclusion goals into our digitization prioritization policy. lsu library’s current strategic plan includes among its goals, (1) adapting and augmenting of description of collections to expose otherwise hidden materials relating to diverse or marginalized populations; and (2) strengthening the libraries’ culture related to diversity through appropriate programming, policy development and mentoring.[19] recognizing that “all research libraries suffer from past failures to recognize, routinize, and celebrate diversity throughout their operations,” the library’s strategic plan aims to develop, “a durable and pervasive culture of inclusion for underrepresented or marginalized groups, be they racial, ethnic, gender, sexual orientation, socioeconomic, or geographic in nature.”[20] this culture of inclusion involves collection development. as such, it relates directly to selection for digitization. for all of its benefits, the strategic plan doesn’t name the problem its trying to solve. it acknowledges a lack of diversity without specifying the history of racist collection development and descriptive practices. this history is not unique to lsu libraries, nor is it unique to the u.s. south. many critiques of standardized description draw attention to biases in the way many types of communities are described[21], and/or simply excluded completely.[22] if we are to take seriously our “past failures to recognize, routinize and celebrate diversity,” as our strategic plan encourages us to do, we must acknowledge our library’s specific practices and build new practices. to routinize a celebration of diversity, we need policies that advance anti-racism. for the purposes of this paper, i’m using anti-racism in a general sense: the active resistance against recognized racist practices and policies. by racism, i mean both overt racism in collection description, and also the established practices that maintain racist outcomes. as race scholar ibram x. kendi writes, there is no possibility of being “not-racist” there is only racist and antiracist action. “all policies, ideas and people are either being racist or antiracist. racist policies yield racial inequity; antiracist policies yield racial equity.”[23] those of us working in the field of cultural institution management, those of us who choose what to digitize and thus what narratives to promote, what history to highlight, and what legacies to further have the opportunity to enact anti-racist action through digitization prioritization policies that counteract our racist past. this is a small corner of the world, to be sure, but it’s the corner we have some say over. implementing anti-racism we are still at the early stages of developing a means of implementing anti-racism into our digitization selection policy; time and experience will no doubt help us shape our process in the future. our initial efforts focus on two areas. the first is to prioritize collections that contain underrepresented communities in general, and african americans specifically. as a predominantly white institution in the u.s. south, there is no difference between our story and the story of african americans. our collections have many african american individuals and communities in them, though we rarely center these stories. to operationalize anti-racism, we’re adding an inclusion criteria to our digitization prioritization checklist. much like the other institutions explored above, we have criteria related to copyright and access restrictions, preservation of source material, research use of the collection, and projected use of digital content. to this list we’re adding a criteria to allow us to evaluate how, if at all, any given collection can help us counteract traditional erasure. discovering which collections have communities we want to highlight can be difficult because manuscript collections often contain a wide variety of material relating to many different topics, events, people and places. deciding what to describe is often a challenge and no description can contain everything. add to this the traditional exclusion of non-white professionals in the field and it is littler wonder that we have a history of collection development and description that have favored a representation of whiteness.[24] however, there are ways to work with extant descriptions. dorothy berry, in effort to find collections related to african americans for the umbra search project, emphasizes the need to look beyond immediate descriptions. “a list of potentially relevant words used across time to refer to african americans and their history was devised and each collection finding aid available online was individually searched.” terms included “negro, colored, black, afro-american, african american, interracial, civil rights, human relations, & intergroup.”[25] the second approach, spearheaded by jenny mitchell, head of manuscript processing at lsu libraries, is to review the institution’s history of collection restrictions. self-censorship has long been an issue for collection development in all fields of librarianship. often this self-censorship takes the form of librarians deciding not to acquire particular books for fear of someone objecting to the title being in the collection.[26] for special collections and archives, self-censorship can mean failing to describe certain portions of a collection in a finding aid or restricting access to material.[27] by investigating historic restrictions on the material held in lsu libraries’ special collections, we can explore the type of self-censorship the staff were imposing over the decades. did restriction policies uphold racist narratives of white supremecy, or erase racial violence? by reviewing the policies of the past, we’re asking in what way digitization can counteract their racist outcomes. work in this area is still in its early phase. by implementing specific steps to explore our own collections and counteract our history, we’re taking the first steps in reaching the goals set forth in our strategic plan. future steps should and will include looking outside of the library to work with the communities that we hope will see themselves reflected in the collections. many excellent examples exist of community involvement in collection building.[28] however, there is always a danger that predominantly white institutions lean too heavily on traditionally excluded communities to help us fix our self-imposed racism problems. by beginning this process as a deep-dive into our own history and practices, we hope that when it’s time to reach out for help, our partners will trust our sincerity. conclusion deciding what to digitize, and in what order, has long been a concern in special collections libraries, archives, and related cultural heritage organizations. as lsu libraries special collections is reevaluating in-house policies related to these decisions, we are following the lead of many institutions by considering a set of criteria including copyright and access restrictions, preservation of source material, research use of the collection, and projected use of digital content. to these considerations we’re adding a criteria that will help us evaluate a collection’s relation to our diversity-oriented strategic plan. by building this criteria directly into the policy, we hope to take one small, but significant, step toward counteracting a racist history of exclusion. the power of building this into a policy is to make it bigger than the interest of one or two staff members. the history we’re counteracting was possible because of the policies in place; the routine, everyday practice of choosing some collections over others, of describing some parts of collections and not others. it’s our hope that new policies help routinize anti-racism and yield racial equity in our collections. [1] “selecting research collections for digitization-full report,” https://www.clir.org/pubs/reports/hazen/pub74/; “scoping the future of the university of oxford’s digital library collections,” http://www.bodley.ox.ac.uk/scoping/report.html [2] “selection policy for digitization projects.” accessed june 10, 2019. https://www.dartmouth.edu/~library/digital/about/policies/selection.html. [3] “university of california selection criteria for digitization (pag): uc libraries.” accessed june 10, 2019. https://libraries.universityofcalifornia.edu/content/university-california-selection-criteria-digitization-pag. [4] “selection policy for digitization projects.” accessed june 10, 2019. https://www.dartmouth.edu/~library/digital/about/policies/selection.html. [5] additional examples of selection criteria exist for external partners to suggest digitization. see, for example, the “digital project selection criteria” chapter in ochoa, marilyn n., laurie n. taylor, and mark v. sullivan. “digital collections assessment and outreach, spec kit 341 (august 2014),” august 29, 2014. https://publications.arl.org/digital-collections-assessment-outreach-spec-kit-341/. i’m thankful to molly bragg for drawing my attention to this resource. [6] “call for proposals – diversity inclusion digitization initiative.” accessed june 20, 2019. https://duke.app.box.com/s/vvftxcqy9qmhtfcxdnrqdm5kqxh1zc6t. [7] molly bragg, head, digital collections and curation services, duke university libraries, conversation with author, june 18, 2019. [8] particularly helpful is the bibliography prepared by the cultural assessment working group of the digital library federation, available here: https://docs.google.com/document/d/18ebvhowlbnx5-na5_llm9bqqx3rxoesalrw5-jd8o4o/edit# [9] manžuch, zinaida. “ethical issues in digitization of cultural heritage.” journal of contemporary archival studies, 4 (2017) [10] ibid. p. 13 [11] robertson, author tara. “digitization: just because you can, doesn’t mean you should.” tara robertson (blog), march 20, 2016. http://tararobertson.ca/2016/oob/. [12] the literature on indigenous communities and library digitization is large; see, for example: christen, kimberly a. “does information really want to be free? indigenous knowledge systems and the question of openness.” international journal of communication 6, no. 0 (november 30, 2012); carpenter, brian. “archival initiatives for the indigenous collections at the american philosophical society.” american archivist, case study, https://www2.archivists.org/sites/all/files/case_1_archival_initiatives_for_indiginous_collections.pdf. [13] “code of ethics | zinelibraries.info.” accessed june 10, 2019. http://zinelibraries.info/code-of-ethics/. [14] “black at bryn mawr | past as legacy and project: re-remembering black experiences at bryn mawr college.” accessed june 10, 2019. http://blackatbrynmawr.blogs.brynmawr.edu/. [15] “umbra search african american history.” umbra search african american history. accessed june 10, 2019. https://www.umbrasearch.org/. [16] “in her own right.” accessed june 10, 2019. http://inherownright.org/. [17] “providing computational access to records of american capital punishment.” accessed june 10, 2019. https://2019.code4lib.org/talks/providing-computational-access-to-records-of-american-capital-punishment. [18] “deep learning and historical collections.” accessed june 10, 2019. https://2018.code4lib.org/talks/deep-learning-and-historical-collections. [19] “lsu libraries strategic plan, 2017-2022.” accessed june 10, 2019. https://www.lib.lsu.edu/sites/default/files/info/lsulibrariesstrategicplan17-22.pdf [20] ibid. p. 6 [21] see, for example, berman, sanford. 1971/1993. prejudices and antipathies: a tract on the lc subject heads concerning people. metuchen, nj: scarecrow; drabinski, emily. “queering the catalog: queer theory and the politics of correction,” the library quarterly: information, community, policy 83, no. 2 (april 2013): 109. http://www.jstor.org/stable/10.1086/669547; koford, amelia. “engaging an author in a critical reading of subject headings.” journal of critical library and information studies 1, no. 1 (2017). doi: 10.24242/jclis.v1i1.20. rinn, meghan r. “nineteenth-century depictions of disabilities and modern metadata: a consideration of material in the p. t. barnum digital collection.” journal of contemporary archival studies 5 (2018): article 1. https://elischolar.library.yale.edu/jcas/vol5/iss1/1. [22] see, for example, robinson-sweet, anna. “truth and reconciliation: archivists as reparations activists.” the american archivist 81, no. 1 (march 2018): 23–37. https://doi.org/10.17723/0360-9081-81.1.23. poole, alex. “the strange career of jim crow archives: race, space, and history in the mid-twentieth-century american south.” the american archivist 77, no. 1 (april 2014): 23–63. https://doi.org/10.17723/aarc.77.1.g621m3701g821442. sutherland, tonia. “archival amnesty: in search of black american transitional and restorative justice.” journal of critical library and information studies. 1 no. 2 (2017). accessed february 27, 2019. http://libraryjuicepress.com/journals/index.php/jclis/article/view/42. [23] kendi, i. x. (2018, december 6). this is what an antiracist america would look like. how do we get there? the guardian. retrieved from https://www.theguardian.com/commentisfree/2018/dec/06/antiracism-and-america-white-nationalism [24] see, for example, sofia leung. “whiteness as collections.” sofia leung (blog), april 15, 2019. https://sleung.wordpress.com/2019/04/15/whiteness-as-collections/. [25] berry, dorothy. “digitizing and enhancing description across collections to make african american materials more discoverable on umbra search african american history” the design for diversity learning toolkit. (n.d.). retrieved may 17, 2019, from https://des4div.library.northeastern.edu/digitizing-and-enhancing-description-across-collections-to-make-african-american-materials-more-discoverable-on-umbra-search-african-american-history/. [26] see, for example, andrea jamison. “librarians beware: self-censorship.” intellectual freedom blog (blog), may 8, 2018. https://www.oif.ala.org/oif/?p=13550; school library media research vol. 13 (2010); yorio, kara. “diverse characters impact decisions to buy books.” school library journal. accessed july 22, 2019. https://www.slj.com?detailstory=diverse-characters-impact-decisions-to-buy-books. [27] by way of example, see “precious and adored: reading victorian love letters, writing queer history.” autostraddle, june 12, 2019. https://www.autostraddle.com/precious-and-adored-reading-victorian-love-letters-writing-queer-history/. this article tells the story of love letters between rose cleveland, a sister of grover cleveland, and her lover evangeline simpson. these letters were discretely moved to a different box in the minnesota historical society in 1969. to counteract this erasure, an edited volume of the letters is now available. [28] many initiatives related to indigenous communities are ongoing. see, for example, blackburn, fiona. “an example of community engagement: libraries act and the act aboriginal and torres strait islander communities.” australian academic & research libraries 45, no. 2 (april 3, 2014): 121–38. https://doi.org/10.1080/00048623.2014.908497; patterson, lotsee. “indigenous librarianship: a global perspective.” text. about ala, march 29, 2007. http://www.ala.org/aboutala/offices/olos/olosprograms/jeanecoleman/02patterson. as well as the american philosophical society’s center for native and indigenous research, https://www.amphilsoc.org/library/cnair. about the author s. l. ziegler is the head of digital programs and services at lsu libraries, where they develop strategies for digitization and preservation operations. ziegler has an mls from drexel university and a ma in philosophy from lsu. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – audioregent: exploiting simpleadl and sox for digital audio delivery mission editorial committee process and structure code4lib issue 10, 2010-06-22 audioregent: exploiting simpleadl and sox for digital audio delivery audioregent is a command-line python script currently being used by the university of alabama libraries’ digital services to create web-deliverable mp3s from regions within archival audio files. in conjunction with a small-footprint xml file called simpleadl and sox, an open-source command-line audio editor, audioregent batch processes archival audio files, allowing for one or many user-defined regions, particular to each audio file, to be extracted with additional audio processing in a transparent manner that leaves the archival audio file unaltered. doing so has alleviated many of the tensions of cumbersome workflows, complicated documentation, preservation concerns, and reliance on expensive closed-source gui audio applications. by nitin arora introduction in late 2009, after nearly a year supervising the digitization of audio content for our department, i had observed that most of our issues arose in the post-digitization phase of processing. worse yet, every explored solution seemed to spawn new problems. at the heart of the matter was the fact that digitized audio content in its virginal form often makes for poor deliverable files due to long silences, distracting peaks, and inconsistent volume levels between sections of the same audio file. certainly, many of these characteristics can justifiably be considered to possess narrative qualities of interest to researchers, yet there still remains the practical matter of presenting content online in a presentable fashion. and to complicate matters it was, from a preservation perspective, undesirable at the least and unacceptable at the most to edit the archival file for the sake of making more suitable derivatives. since we seemed trapped between the mythological “rock and a hard place”, one solution considered was to essentially embrace our medial position and use a “middle man” audio file, an edited offspring of the archival file that would yield more suitable online derivatives given that the aforementioned offending audio would be removed and some additional light audio processing added. this too was problematic, however, in that retaining this intermediary file would not only increase local storage requirements but would also increase storage requirements for each of our peers in our lockss [1] digital preservation network. even if, for storage concerns, these intermediary files were to be deleted after the creation of web-deliverable files, responsibility would still require copious documentation to inform us or our predecessors of how we got from point a to point b, so to speak, in the event that the web-deliverables would have to be reconstituted at a some point – in other words, if much of the work needed to one day be repeated. differing from the “middle man” approach, which essentially advocates the removal of unwanted content -albeit to a new file, is the approach that advocates the extraction of desired content. in other words, the idea is to simply extract what is desired rather than delete and edit that which is undesirable. for the purposes of explanation, we can respectively call these the non-destructive and destructive approaches. the non-destructive approach offers distinct advantages over the destructive approach. quite simply, documentation is much simpler and there is no need for an intermediary “middle man” file. consider, for instance, the following analogy: if the word “computer” is analogous to our archival audio file and the word “compute” is analogous to the desired derivative, documenting either approach in terms of individual letter position would be quite simple: non-destructive approach: “extract 1st, 2nd, 3rd, 4th, 5th, 6th, and 7th letters.” destructive approach: “remove 8th letter.” if, however, the desired output is the word “cute”, the situation becomes much more complicated. non-destructive approach: “extract 1st, 5th, 6th, and 7th letters”. destructive approach: “remove 2nd, 3rd, 4th, and 8th letters”. at this point, one might ask “so, what’s the problem?” the problem is that the destructive approach refers to letter positions in the original file. when one starts actually deleting these letters in the “real world”, the instructions might be more useful if they read “remove 2nd, 2nd, 2nd, and 5th letters” as such: computer cmputer cputer cuter cute when considering that with audio one will use timecodes rather than simple letters, the situation extends beyond the merely trivial to the downright confusing. therefore, we decided to adhere to a non-destructive approach. and our gui audio editing software, sony sound forge 9.0, seemed to support our needs. gui audio editors: solutions and limiting factors our audio editing software, sony sound forge 9.0, allowed for us to implement the non-destructive approach by supporting both the extraction of multiple regions of audio from a larger whole, so as to leave out unwanted segments and the extraction of one derivative audio file with any offending segments of the source audio stream simply omitted. in terms of extracting regions, sound forge – as well as many other gui audio applications – supports graphically denoting these regions within audio files. these regions can subsequently be extracted via batch process as separate files, omitting anything that lies between them. figure 1 below depicts the notation of two regions within one audio file. figure 1. two regions selected in the same audio file. in terms of creating one audio file that omits offending segments – essentially a mirror to the process above, sound forge supports the implementation of playlists and cutlists which allow for the creation of a derivative audio file sans the undesired segments either by rendering only what is desired (“playlist”) or omitting what isn’t (“cutlist”). in addition to such capabilities, the software can produce an easy-to-read text output denoting the region list, playlist, or cutlist details such as the name given to a region as well as the relevant time markers. in considering the need to document the process of creating web-deliverables from a master file, the retention of this text output addressed our needs in this one regard but created a new problem. specifically, the potential inclusion of this data into larger xml-based technical/preservation metadata would prove bothersome, given the non-standardized formatting of the output. furthermore, it was not desirable or even feasible to spend time experimenting with alternative and likely expensive software which may or may not have supported better options for exporting such data. in mulling over these solutions and the new problems they spawned it became clear that it seemed risky to rely on embedded region metadata which may or may not be detectable by our future audio applications and the text records exported by our software were insufficient in regard to our long-term documentation needs. a simple xml-based alternative needed to be devised to address both concerns. simpleadl simpleadl (simple audio decision list) is small-footprint xml structure for defining regions within an audio file. [2] by using a “home-grown” xml format to record region locations within the audio file, the aforementioned problems of eventually migrating the data into other xml-based metadata would become sufficiently easier. simpleadl is currently schema-less. the example below depicts the <region> and <outputastracks> elements, the respective “bread and butter” of simpleadl: <region id="_01"> <in unit="seconds">0.5</in> <duration unit="seconds">601</duration> </region> <region id="_02"> <in unit="seconds">612</in> <duration unit="seconds">299.05</duration> </region> <outputastracks>true</outputastracks> simple enough for unencumbered human reading, simpleadl’s <region> element provides an easily retained record of what portions of an archival audio file we want to present online. meanwhile, the <outputastracks> element informs future users, be they man or machine, whether these portions should be extracted as multiple audio files per region or as one audio file which would essentially be a concatenation of each region. specifically, this means that the <outputastracks> element value if “true” informs one of the desire to output one audio file per region, while an element value of “false” would tell of the desire to output only one audio file consisting of all regions spliced together. currently, the creation of simpleadl files is initiated via a small python script called “simpleadlmaker”. this small command-line program prompts the user to input the name of the corresponding wav file as well as the number of regions to define and the desired value of the <outputastracks> element. for the file foo.wav, this little helper script would make a file called foo.adl.xml, a skeletal xml file housing the document tree appropriate to the user-inputted data. given the simplicity of simpleadl, the element and attribute values can then easily be entered with no more than a basic text editor. simpleadl provides enough documentation for reliable reconstitution of online derivatives from the master should the need arise to do so – for example, if we later switch from mp3 to ogg format for online audio delivery. but in the endless cat-and-mouse game of problem-solution-problem, this begged the question, “if the need to remake web-deliverables arises, who wants to open up every simpleadl file and manually extract regions from the archival audio files?”. doing so would entail endless and costly hours of mind-numbing work, ripe for human error. in other words, the very reasons for devising simpleadl were rearing their proverbial, ugly heads once more. for the last pieces of our audio workflow puzzle, we looked to our experience with image-based collections. for our image collections, we use the open-source image processing toolkit imagemagick [3] to batch process web-deliverable jpeg files from our archival scans. it only seemed logical to seek a similar toolkit for audio processing. sox sox (“sound exchange”) is an open-source, cross-platform command-line audio editor, aptly referred to on the project website as “the swiss army knife of sound processing programs”. [4] highly powerful, its capabilities to split, splice, and apply post-production effects to audio files seem almost limitless. sox is ideally suited to the digital library environment. for an imaginary audio file, “foo.wav”, the following sox commands would produce ogg vorbis [5] audio files for each region notated in the simpleadl example above: $ sox foo.wav foo_01.ogg trim 0.5 601 $ sox foo.wav foo_02.ogg trim 612 299.05 after learning about the existence of sox, it became evident that a gui audio editor would only be needed to help a technician identify the location of desired regions within each archival audio file. after the starting point and duration of each region was manually entered into a simpleadl file, the extraction of audio regions and the application of light audio processing to them could then be automated by simply “feeding” the element values from a simpleadl file to sox. for this, a python script was created. audioregent audioregent [6] is a command-line python script that essentially serves as a moderator between archival audio files in wav format and their corresponding simpleadl files. after starting audioregent the user enters the path in which wav files and their corresponding simpleadl files are to be found. audioregent then iterates over every simpleadl file within the path, sending command-line arguments to sox, thus producing derivative audio files for each master wav file per both the instructions within the corresponding simpleadl file and the audioregent settings, which can be changed by editing a simple xml setup file called audioregent.xml. audioregent.xml allows one to control things such as the derivative output audio format and what we call the “sox options” – the effects, if any, that sox will add to the derivative audio files. these effects can include leveling the recording or padding the ends with a second of silence, etc. following batch conversion of the derivatives to mp3 format, the files are distributed to our servers and our delivery platform for users to access. it should be noted that even though our desired output format is mp3, we currently have audioregent.xml set so that audioregent creates wav derivatives. we then convert these wav derivatives in batch to mp3 file using the lame encoder. [7] this step is external to the audioregent workflow and must be done because our pre-compiled windows binary of sox does not output to mp3. we could eliminate this extra step if we were to build sox from source to include the lame libraries as this would allow mp3 files to be created natively by sox. example for a “real world” example of the combined usage of audioregent, simpleadl, and sox consider the following full example of a simpleadl file for an item from the harald rohlig organ music collection [8] at the university of alabama. the item identifier for the example below is “u0008_0000002_0000089”, hence the 1:1 archival wav file is named u0008_0000002_0000089.wav. after a metadata librarian creates the basic metadata, including the timecodes for the beginning of each track, a member of our digital services staff would use the simpleadlmaker script to create a skeletal simpleadl file, shown below. note, the <statistics> and <text> blocks are not required as they do not constitute the core elements of simpleadl and are ignored by audioregent. we simply prefer to capture statistical metadata about the audio file from sound forge and house very basic descriptive metadata about the item to help the technician keep track of the tracks, so to speak. <audiodecisionlist filename="u0008_0000002_0000089.wav"> <statistics> <channel position="mono"> <minimumsampleposition unit="seconds"/> <minimumsamplevalue unit="dbfs"/> <maximumsampleposition unit="seconds"/> <maximumsamplevalue unit="dbfs"/> <rms_level unit="dbfs"/> </channel> <length unit="seconds"/> </statistics> <region id=""> <in unit="seconds"/> <duration unit="seconds"/> <text type="xhtml"> <p type=""/> </text> </region> <region id=""> <in unit="seconds"/> <duration unit="seconds"/> <text type="xhtml"> <p type=""/> </text> </region> <region id=""> <in unit="seconds"/> <duration unit="seconds"/> <text type="xhtml"> <p type=""/> </text> </region> <outputastracks>true</outputastracks> </audiodecisionlist> with both the simpeadl file onscreen alongside an instance of sound forge, accurate start and duration times are determined and typed into the simpleadl file using the notepad ++ text editor [9] for windows. given that the care required in determining and documenting this data by hand essentially builds in some quality control measures, we have not found the manual entry of timecodes to prove troublesome. this procedure results in the corresponding simpleadl file below, u0008_0000002_0000089.adl.xml. note that the last region is commented out. it was considered undesirable for delivery, yet it is still documented. <audiodecisionlist filename="u0008_0000002_0000089.wav"> <statistics> <channel position="mono"> <minimumsampleposition unit="seconds">69.73</minimumsampleposition> <minimumsamplevalue unit="dbfs">-6.094</minimumsamplevalue> <maximumsampleposition unit="seconds">226.145</maximumsampleposition> <maximumsamplevalue unit="dbfs">-6.939</maximumsamplevalue> <rms_level unit="dbfs">-25.421</rms_level> </channel> <length unit="seconds">539.779</length> </statistics> <region id="_0001"> <in unit="seconds">8.591</in> <duration unit="seconds">83.948</duration> <text type="xhtml"> <p type="title">harald rohlig: nun danket all und bringet ehr'</p> </text> </region> <region id="_0002"> <in unit="seconds">97.059</in> <duration unit="seconds">123.315</duration> <text type="xhtml"> <p type="title">harald rohlig: gott der vater wohn uns bei</p> </text> </region> <!- <region id=""> <in unit="seconds">221.727</in> <duration unit="seconds">318.052</duration> <text type="xhtml"> <p type="title">johann sebastian bach: fugue in d major</p> <p type="comment">(incomplete; do not use)</p> </text> </region> --> <outputastracks>true</outputastracks> </audiodecisionlist> figure 2 below shows the audioregent workflow for this item after the simpleadl file is created. [10] the aforementioned extra step involving the lame encoder and the subsequent transfer to online delivery is also depicted. after mp3 files are successfully delivered online, the wav files in audioregent’s final output folder are deleted, the files in audioregent’s temp folder having already been deleted automatically. after simpleadl files are made for a batch of items in the manner described above, we run audioregent. figure 2. audioregent workflow. (full-size image) the first batch we ever used audioregent on consisted of roughly 25 items from the rohlig collection, approximately 10% of the current total of digitized items for the collection. with the simpleadl files already in place, batch processing nearly 150 derivative tracks from these 25 archival items took approximately 90 minutes on a windows computer (64-bit xp professional, 3 ghz intel core 2 duo processor, 8 gigabytes of ram). this included the added use of the lame encoder as described above, which comprised approximately one-third of the total time. future work now that audioregent has successfully been integrated into our digital audio workflow we can begin thinking about developing a formal schema for simpleadl. one of our metadata librarians at the university, shawn averkamp, has agreed to spearhead this effort given that simpleadl contains metadata of interest to metadata librarians and not just audio technicians. as long as the aformentioned “bread and butter” of simpleadl remains intact, the format is highly mutable given that audioregent ignores all but the <region> timecode and <outputastracks> values. with long-term preservation in mind, a schema would seem a requisite next step in the process. in addition to the desire for a schema, we will also look to further ways in which to help automate the creation of simpleadl files from pre-existing descriptive metadata. we shall also look for an open-source replacement for sound forge as our eventual goal is a completely open-source audio workflow. while good open-source audio editors exist, we ideally want to find an application that can conjure up the same statistical technical metadata about the audio file as sound forge. finally, given the added external step involving the lame encoder for the purposes of making mp3 files, it would behoove us to go ahead and build sox from source to include lame’s mp3 encoding functionality. this would not only allow us to directly create mp3 derivatives from our master wav files, but would also make the entire batch process cycle less time-intensive. conclusion balancing preservation and practical delivery concerns in terms of digital audio content can be a win-win situation without the need for impractical overhead and complexity. by using simple technologies – audioregent and simpleadl – that utilize a powerful, open-source audio editor in sox, the university of alabama libraries’ digital services is able to implement an easy, transparent, and effective workflow to help manage digital audio content. about the author nitin arora is a digitization specialist at the university of alabama libraries’ digital services. he recently completed his master’s in library and information studies at the university of alabama. his email address is “nitaro74 at gmail dot com”. notes [1] lockss. retrieved june 2, 2010, from http://lockss.stanford.edu/lockss/home/ [2] simpleadl. retrieved june 2, 2010, from http://blog.humaneguitarist.org/projects/audioregent/#simpleadl [3] imagemagick studio llc. imagemagick. retrieved june 2, 2010, from http://www.imagemagick.org/ [4] sox – sound exchange. retrieved june 2, 2010, from http://sox.sourceforge.net/ [5] xiph.org foundation. faq. retrieved june 2, 2010, from http://www.vorbis.com/faq/#what [6] audioregent. retrieved june 2, 2010, from http://blog.humaneguitarist.org/projects/audioregent/ [7] lame mp3 encoder. retrieved june 2, 2010, from http://lame.sourceforge.net/about.php [8] university of alabama. selections for organ by harald rohlig. retrieved june 2, 2010, from http://acumen.lib.ua.edu/u0008_0000002_0000089/ [9] notepad++. retrieved june 2, 2010, from http://notepad-plus.sourceforge.net/uk/site.htm [10] for an image depicting how audioregent can produce either many or one derivative audio file/s per simpleadl, see http://blog.humaneguitarist.org/projects/audioregent/#howworks. (retrieved june 2, 2010.) subscribe to comments: for this article | for all articles one response to "audioregent: exploiting simpleadl and sox for digital audio delivery" please leave a response below: getting my hands dirty | what jody's been up to..., 2014-10-04 […] brings this to mind is the audio transformation software that was built by one of my previous employees (a professional musician), who left us for a […] leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – making the move to open journal systems 3: recommendations for a (mostly) painless upgrade mission editorial committee process and structure code4lib issue 43, 2019-02-14 making the move to open journal systems 3: recommendations for a (mostly) painless upgrade from june 2017 to august 2018, scholars portal, a consortial service of the ontario council of university libraries, upgraded 10 different multi-journal instances of the open journal systems (ojs) 3 software, building expertise on the upgrade process along the way. the final and the largest instance to be upgraded was the university of toronto libraries, which hosts over 50 journals. in this article, we will discuss the upgrade planning and process, problems encountered along the way, and some best practices in supporting journal teams through the upgrade on a multi-journal instance. we will also include checklists and technical troubleshooting tips to help institutions make their upgrade as smooth and worry-free as possible. finally, we will go over post-upgrade support strategies and next steps in making the most out of your transition to ojs 3. this article will primarily be useful for institutions hosting instances of ojs 2, but those that have already upgraded, or are considering hosting the software, may find the outlined approach to support and testing helpful. by mariya maistrovskaya and kaitlin newson introduction while software upgrades are typically not a glamorous or exciting aspect of library technology, they are essential to providing secure and usable applications to our end users. software upgrades that do not impact the user interface or functionality may go unnoticed by users, but for upgrades that affect user experience or system functionality, special care needs to be taken in communicating these changes to users throughout the process. in the summer of 2018, the university of toronto libraries (utl) and scholars portal (sp) undertook a collaborative project to move from open journal systems version 2 to 3. open journal systems (ojs) is an open-source scholarly journal publishing platform developed by the public knowledge project (pkp) that provides functionality to facilitate peer-review, editorial, and online publishing processes. it also includes various plugins to enhance its functionality, including doi minting, subscription management, and a variety of themes. the transition from ojs 2 to 3 involved significant changes to the software, with many improvements to the user interface, navigation, and functionality. background the university of toronto libraries system is the largest academic library in canada and supports over 14,000 faculty members and 90,000 students across 3 campuses [1]. in 2005, the utl’s information technology services launched a journal hosting service on a locally hosted instance of ojs to support open access publishing among faculty, students, and staff. in 2014, the ojs instance moved to scholars portal’s shared hosting service, with the utl staff providing user support, troubleshooting, and training to the journal teams. the utl is one of 21 university libraries that make up the ontario council of university libraries (ocul), an academic library consortium which serves almost 500,000 students [2] . scholars portal is the technology service arm of ocul and provides a wide range of services to ontario’s academic community. both ocul and the utl support pkp development activities, with ocul as a development partner and utl a financial supporter. both co-authors are also involved in the pkp documentation interest group. since 2012 scholars portal has offered an ojs hosting service , which is used by 10 universities within the ocul consortium. across all ojs instances hosted at scholars portal, there are approximately 125 actively publishing journals, containing over 30,000 published articles. the utl instance is by far the largest, with 60 journals, including 37 actively publishing, and over 17,000 published articles across all journals as of december 1, 2018. scholars portal handles all updates to ojs for each instance, manages servers and hosting, installs new plugins, and troubleshoots technical issues. they also offer custom journal design services. the ojs hosting service at scholars portal is run by two librarians, along with support from the systems support team. upgrade decision ojs 3.0 was released in 2016 after almost 10 years of development, improvement, and thorough ux testing. it came as a welcome change from the version 2 that many editors found difficult to navigate and that limited the ability to customize individual journals on a multi-journal instance without affecting the default templates. at the consortial level, scholars portal upgraded 10 instances from ojs 2 to 3 from june 2017 to august 2018. each institution set their own upgrade date based on a number of factors, including publishing schedules and upgrade interest from journal editors. our first upgrade came from an instance that was keen to make the move, and they were upgraded to version 3.0. as is common with early adoption of new software, earlier upgrades encountered bugs more frequently and required more staff time to support, research, and resolve these issues. as we made our way through each upgrade, they gradually became easier, with fewer bugs found or users encountering known issues that could be resolved quickly. despite having a few keen journal teams interested in the upgrade, the utl decided to wait until version 3.1 was released, which added key missing features from the first release, such as the subscription module used by our moving paywall journals. at the time of the ojs 3 upgrade in the summer of 2018, the utl instance had 57 faculty and student journals, of which 36 were actively publishing. considering the number of journals the upgrade would impact, we had to be careful and thorough in our preparation and communication throughout the process. upgrade preparation & testing below we outline the steps in pre-upgrade planning, preparation, testing, and outreach that are summarized in checklist points at the end of each section. the full checklist is available in appendix a. assessing your upgrade readiness despite the excitement around the new release, two things to watch out for are: whether it includes the critical functionality your users depend on, and whether the release is sufficiently tested and bug-free for roll out. the pkp website and forum are great resources to keep track of new releases, reported issues, and changes in functionality. additionally, it is essential to engage journal teams in conversations about the features that are critical for them. that being said, despite soliciting such feedback before the upgrade, we found that both support staff and journal teams were more likely to identify missing features and changes during regular use post-upgrade than during the testing phase. checklist item #1: work with your journals to identify their essential features and make sure they are available in the new version cleaning up your content at the time of the ojs 3.1 release, the utl instance had 24 extra journals that had moved elsewhere, ceased publishing, or never launched after the initial setup. the utl staff took this opportunity to reconnect with journal teams and archive the journals that ceased publishing indefinitely. our archiving strategy included: depositing articles into scholars portal journals for continued open access, with proper url redirects set up in the web server configuration; capturing journals’ websites via the internet archive’s wayback machine; and revising our service agreement to include an archiving provision for inactive journals going forward. the upgrade from ojs 2 to 3 is also an opportunity to clean up any users that may be spam users, which is a common issue in ojs, particularly for instances that existed before pkp implemented recaptcha on account registration. there are several ways to identify these users, but one option is through the email domains used when the user signed up. using the following sql query, you or your system administrator can identify the domains in order to identify possible spam users: select substring_index(email, '@', -1) domain, count(*) email_count from users group by substring_index(email, '@', -1) -if you want to sort as well: order by email_count desc, domain; once you’ve identified the domains that are connected to spam users, you can use these to create a list of usernames to clean up with a query: select * from users where email like "%@spam.com" or email like "%morespam.com" ... with this list of usernames, you can then use ojs’s built-in user merging tool (in your ojs directory at tools/mergeusers.php) to clean up users. we created a small bash script to do this, and pkp also has a process that they recommend. you’ll need to create a user account that all accounts can be merged into if one doesn’t already exist. keep in mind that while it’s nearly impossible to find all the spam users in a large instance of ojs, you may be able to significantly clean these up, thereby reducing the amount of data in your instance and making it easier for journal teams to manage their users. checklist item #2: consider cleaning up the content you don’t need to carry over, including inactive journals, spam users, etc. setting upgrade timelines figure 1. utl ojs upgrade timeline in 2018. the time you are able to devote to planning and preparing for the upgrade will depend on a number of factors, including your department’s schedule and staff availability, the size of your instance and customizations to carry over, whether you will have demo instances for your journals to test, and whether you are planning on delivering preand/or post-upgrade training. at utl we gave our journals 3 months to test out their sandboxes, report issues, request theme migration assistance, and receive training, consultations, and resources. as journals encountered any issues, we reported these to scholars portal. while several journals took advantage of these resources, some did not – and subsequently required significant (and more time-sensitive) assistance after the upgrade. the bottom line is that setting aside enough time and support resources post-upgrade is just as important as pre-upgrade preparation. checklist item #3: decide if you will provide sandbox sites to journals for testing (or use pkp’s ojs test-drive instance) checklist item #4: set aside time before and after the upgrade for testing, training & troubleshooting communication and training as with any major change, advance communication with clear timelines and expectations is key. the first step is to ensure you have updated contact information and re-establish communication with any journals that may have had editorial changes. announcing your upgrade plans early (regardless of whether you have the exact date scheduled) will go a long way to set expectations. if the upgrade involves downtime or a freeze for new content, make sure to request email receipt confirmation once the dates are announced. at utl, we used a combination of our mailing list for hosted journals and personal emails to journal editors for upgrade-related communication. training and/or one-on-one drop-in opportunities for journal teams may be another helpful way of announcing the upgrade, communicating your support, and resolving issues. be prepared for some journal teams not to take advantage of these opportunities until after the upgrade happens, at which point the learning curve may be steeper and the support need more urgent. at utl we offered an ojs 3 workshop 3 months before the upgrade and sent out information about journal sandboxes that were available for testing. fifteen editors attended the 2 hour workshop from 13 journals (36% of our active journals), and these journals went on to be mostly self-sufficient throughout the upgrade. because finding another session time that would work for the rest was challenging, we opted to offer drop-in hours for one-on-one training and consultations. by contrast, multiple drop-in sessions had only 1 attendee, with most editors opting for email communication. checklist item #5: check that your contact information for hosted journals is up-to-date and re-establish contact in case of editorial changes. checklist item #6: announce plans to upgrade early, even if you don’t have exact timelines yet. checklist item #7: set downtime for production upgrade – and request confirmation. checklist item #8: set content freeze dates if necessary. checklist item #9: align your training/consultation offerings with demand – your editors may prefer emails or a call over workshops and drop-ins. defining the level of support support services and resources vary in institutions hosting multiple journals. the main change that users will experience in the upgrade is to the user interface, and effort is required to carry over any design or layout changes made in ojs 2. an institution may choose to only perform the upgrade and style its base homepage while letting journals do their own customizations, or it may be able to offer more hands-on support with theme migrations. scholars portal provided assistance with customized theme migrations to institutions on a case-by-case basis, depending on the local resources available at each institution, the comfort of the institution in doing that work themselves, and the complexity of the original themes. the utl supported its hosted journals with theme migrations on the following conditions: support is only provided to active journals requests have to be made before the cut-off date (one month before the upgrade), so we have enough time to do the styling support requests are responded to on a first-come, first-served basis if there is no response to our offer of support, we assume that the journals will do their own styling after migration additionally, training sessions, individual drop-ins, and email support were offered to empower journal teams to do their own setup. in the end, we received requests for assistance from 4 journals (10% of our active journals). two more identified the need for assistance after the upgrade. the rest either did their own styling or chose to go with the default theme. overall, this change was not as drastic or disruptive as we had originally expected, and most journals were able to handle it on their own. checklist item #10: define your level of support and communicate it clearly. checklist item #11: set the cut-off date for pre-upgrade support requests. setting up your test instance each upgrade began with a testing environment that was upgraded from the current ojs 2 instance. this provided a space for journal managers to gain familiarity with the new system, ask questions about any changes in functionality, and complete any necessary customization. while it may seem obvious to those of us that manage software, it bears repeating – create backups! back up your submission files, your ojs system code, and your database consistently. this will save you a lot of time and headaches throughout the upgrade process. in particular, it can be helpful to back up your instance when it’s ready for the upgrade script to be run so you can quickly resume the upgrade once you resolve any issues that the script encountered. as you work through any issues encountered during your test upgrades, pkp’s forums, github issues, and administrator’s guide are essential resources. older ojs instances in particular may have database encoding issues that need to be resolved before an upgrade will work successfully. after some research and troubleshooting on the pkp forums, we found a process to resolve this issue. once you’ve successfully upgraded your first test instance, there are a few settings you may wish to configure. first, disable any plugins within ojs that send data out to other services or systems, such as the crossref plugin and the acron plugin. we also recommend replacing all user email addresses in your database’s ‘users’ table so that users aren’t receiving emails. pkp has also recommended this in their upgrade guide [3]. the following database query can be used to set all user emails to mailinator addresses, which you could replace with any preferred domain: update users set email=concat(username, "@mailinator.com"); if users need to have the correct email set in the testing instance later, then you can update the email in the test database via their account management pages in the user interface; for instance, if they need to reset their account password or want to test email notifications within the system. checklist item #12: create frequent backups of your code, database, and files as you work through each upgrade. checklist item #13: when setting up your testing instance, disable any plugins that may send data out to other services or systems. checklist item #14: consult the pkp forums and open github issues when troubleshooting problems testing and feedback after scholars portal (sp) set up the utl test instance, every journal received a notice with their individual url, user credentials, and testing instructions (see appendix c). it was made clear that no changes made in the sandbox would be carried over to the upgraded instance. journal teams were encouraged to get familiar with the system, test and tweak the new look, and report any missing functionality or unexpected behaviour. as users found any potential issues or changes they’d like to make, we ensured that these were documented appropriately so they could be replicated in our production upgrade. we did so via our confluence and jira platforms. as the administrator of the ojs instance, you may also want to monitor your server logs for any errors that users encounter as they are testing the new instance. only about 10% of editors contacted us about functionality that was missing or changed (e.g. thesis abstract plugin, public folder plugin, etc.). some also kindly reported the plugins they relied on that worked fine. most reports about changes or missing features came after the upgrade. there may not be much you can do about it (other than submitting a feature request on pkp forum), but providing a venue for journals to report these issues covers your bases. checklist item #15: put together an easy to follow test plan for the journal teams to test and report issues. redesigning your homepage while most ojs 3 changes affect individual journal websites, the platform upgrade may be an opportunity to freshen up your ojs homepage, particularly if you use it as a front end for your journal publishing service. at utl we took advantage of this opportunity to redesign the homepage, for which we did a round of user testing with wireframes and subsequent modifications based on feedback. we also set up the default template that would identify the library as the host in the footer of all journals on the instance. figure 2. utl’s old homepage design on ojs 2. figure 3. utl’s new homepage design on ojs 3 checklist item #16: if you are looking to redesign your ojs homepage, consider timing it with the upgrade to reduce the impact on users. improving server infrastructure at scholars portal, we used the upgrade to ojs 3 as an opportunity to improve our server infrastructure for our hosting service. when the service began, each institution’s instance was hosted on a separate server, along with a sql database hosted on the same server. as the service has grown significantly, we decided to use the upgrade to ojs 3 as an opportunity to move to a single server to host all instances, reducing the need to maintain 10 different servers. the new server infrastructure included an updated version of php, which is essential as ojs is a php application. this new setup also allowed us to move from centos 6 to 7, and all of our databases were moved to a dedicated server hosting mariadb 10, which contains a testing and production database for each instance. we also began enforcing https across all of our ojs instances, using either let’s encrypt or certificates provided by the institution. by making these changes to our servers during the upgrade process, we reduced the need for future downtime and improved the overall performance of the system. checklist item #17: if there are aspects of your server architecture you could improve, consider making these changes during the upgrade to reduce future downtime or impact on users. dealing with pushback if you encounter pushback due to significant changes in the functionality that a journal has come to rely on, evaluate possible alternatives or workarounds. if the functionality is critical, be prepared to reschedule the upgrade until it becomes available. in the case of utl, no such changes were reported before the upgrade, and while non-critical changes were discovered afterward, we were able to come up with workarounds for all. one such example was that biographies in editors’ profiles linked to from the masthead were no longer public. we worked with the journal manager to create a page that would have the biographies outside of personal profiles and directed the links in the masthead to it. because the ojs 3 upgrade involves significant changes to the look and feel of the user interface, there may be some pushback from journals that have invested in their existing setup and may not have the resources to do this work again. in this case, consider offering assistance on a case-by-case basis. at utl, only one journal expressed concern about the upgrade since they did not want their old look to change. curiously, the old look was not a custom design but one of the ojs 2 template themes. we recreated the look in the ojs 3 sandbox ahead of time and shared it with the journal for feedback. we received a letter of gratitude from the editors after the upgrade was complete congratulating us on how smoothly it went. checklist item #18: if critical functionality is not available in the new version, consider postponing the upgrade. checklist item #19: be prepared to be flexible and offer additional assistance if the upgrade takes away previously used functionality, or where modifications are beyond the journal team’s skill level. production upgrade our estimated downtime for the utl’s instance upgrade was 2 days. we communicated the dates to the editors three months in advance to give them the opportunity to notify us of potential date conflicts. we ended up completing the upgrade in 3 days. it was partially due to the fact that the upgrade was performed concurrently with a migration to a new server, and additional time was needed for dns work that added a layer of communication with network services staff on both the sp and utl side. journal teams were understanding about the delay. technical upgrade steps below is an outline of the steps taken by scholars portal when completing a production upgrade from ojs 2 to 3. pkp has also outlined some of these steps in their documentation. get your ojs 3 code base set up in a separate directory, either from git or an official release package. disable your production ojs 2 instance and display a maintenance message for end users. copy the public and files directories from ojs 2 into their new locations for ojs 3. create your new configuration file (config.inc.php) from config.template.inc.php, leaving installed set to “off”, and copy over relevant settings from your old configuration file. due to significant changes in the configuration file from ojs 2 to 3, it’s recommended to create a new file rather than using the previous version. back up your pre-upgrade setup so you can restore quickly if anything goes wrong. check that all of your file permissions are set appropriately for your environment, including the cache and public directories. if you’re using a new database name or location, import your ojs 2 database into this location. open a browser window and navigate to your ojs 3 instance to make sure there aren’t any warnings or errors. run the upgrade script in tools/upgrade.php. set installed to “on” in your configuration file. archive your ojs 2 directory and move your ojs 3 directory into the production location (e.g. by renaming the folder to the previous name used by ojs 2). navigate to your ojs instance in a web browser and explore various journals, test logging in, make sure files are downloadable, test any custom domains, and complete any other necessary tests. apply any necessary customizations, like plugin installations, new themes, or other changes that were noted during the testing phase of the upgrade. let your journal managers and other stakeholders know that the upgrade is complete. for larger ojs instances like utl’s the upgrade script can take over 12 hours to run, primarily because of the size of the metrics tables, which is an issue that pkp is actively working to resolve. we found it useful to turn off php warnings in our php.ini file – logs can become quite large otherwise (sometimes multiple gigabytes per log file). if you’re encountering bugs, you may need to change this temporarily to see error outputs. pkp is gradually working on reducing the php warnings, so this may not be necessary in the future. error_reporting = e_all & ~e_deprecated & ~e_strict & ~e_notice & ~e_warning checklist item #20: when planning upgrade time, ensure that the time to run the upgrade script is considered for larger instances. checklist item #21: consider turning off most php warnings so log files don’t become unmanageably large – you can always enable them later as needed for debugging. post-upgrade the work does not end with the upgrade. in fact, the troubleshooting, support, and training post-upgrade turned out to be the most time-consuming phase of the process. it is important to take this into account and schedule staff time post-upgrade accordingly. troubleshooting while it would be ideal if all journal teams did rigorous testing in the pre-upgrade phase, this is frequently not the case. many of the troubleshooting questions that came in post-upgrade had to do with carrying over ojs 2 set up or changed/missing functionality. for earlier upgrades, this phase was especially challenging as users encountered bugs that existed in the earlier versions of ojs 3. as these bugs have been resolved by the pkp team, these issues became less frequent for each upgrade. for these earlier upgrades, a significant amount of time was spent investigating bugs encountered by users. before posting in the forums, make sure to search both the pkp forum and github issues. encourage your users to engage with and search through the pkp forums as well, where possible. with the move to ojs 3, pkp is updating the software more frequently now, including more small releases and bug fixes. monitoring for the latest updates will make for a more smooth user experience as bugs are resolved. we recommend managing your ojs code with git so you can pull in small code changes from the pkp repository; at scholars portal, we manage our code in a local gitlab instance. we also use rss to stay on top of major releases, and zapier integrated with github to monitor commits to the branch of ojs that we’re actively using. training and communication rather than hosting another ojs 3 workshop at utl, we opted for a combination of: online training resources via the updated journal publishing guide (thoroughly put together by our graduate library assistant) individual consultations via email and phone – that proved the most in demand scheduled drop-in sessions – that proved to be poorly attended “new in ojs” listserv series that we launched based on the frequently asked questions about the changed and new features we soon realized that the extensive pre-upgrade communication put our team on the radar of journal editors who have not been previously in touch beyond occasional troubleshooting requests – and we started receiving more in-depth questions about various aspects of the publishing process. the upgrade turned into a welcome outreach opportunity and helped us establish more meaningful connections with the journal teams and offer more valuable support. reflection, lessons learned, & next steps looking back at the ojs 3 upgrade process, we can say that it went quite smoothly, both for the utl and sp staff doing the upgrade as well as for the journal editors. considering the size of the utl instance, we were happy to only encounter minor issues throughout the process that were all efficiently resolved. we credit our success to these three major factors: generally stable and bug-free state of the version we were migrating to sufficient experience with prior upgrades, and general expertise of sp staff enough prep time and extensive communication with our journal teams that being said, there is always space for improving and reflecting on what we at sp might have done differently, such as: completing more test upgrades for each instance leading up to the final migration date. creating a testing workflow that journals can follow which might point to any issues that happened in the upgrade process, such as missing data. this can also help journal managers begin to get familiar with the new system. utl created one for their users (see appendix b), but it would have been useful to develop one at the consortial level that each institution could edit as needed. evaluate the feasibility of using automated testing tools like katalon recorder, or writing selenium tests, in order to spot-check the data migration in the upgrade process. tests could be created during the first upgrade to ojs 3, and run for each upgrade that follows. this may be especially useful for tracking persistent and recurring issues. one of the most impactful side effects of the upgrade process was establishing (or re-establishing) communication and rapport with journal teams that had previously mostly worked on their own. on one hand, this has resulted in an increase in the number of questions received by utl’s service, which have gradually morphed from upgrade-related issues to more in-depth publishing inquiries. this is something we recommend accounting for if the support for journal teams at your institution is limited. at utl, 0.3 fte is dedicated to the journal production service support, and while the increase could not be called overwhelming, we did benefit from having a student assistant help with basic triage and troubleshooting initial inquiries. on the other hand, and more importantly, this re-established communication with the editorial community has motivated the utl to proceed with planning more in-depth support and services, such as centralized doi minting, assistance with doaj applications, and additional workshops for editors, which we will be rolling out in the future. additionally, we hope that ojs 3 with its intuitive navigation and flexible interface will attract journal teams (particularly student journals) that had previously moved off of ojs 2 to other platforms in search of a simple and flexible content management system. these improvements to ojs, along with other library services for student publishers, will be highlighted at our fourth annual student journal forum planned for january 2019. in conclusion, software maintenance and upgrades may not be the most exciting aspect of library technology, but there is a rewarding challenge in making the transition seamless for end-users and painless for staff. with careful advance planning, user support, and communication, the upgrade may bring more than just an improvement in functionality – it may strengthen your connections with your users and facilitate further outreach and valuable engagement opportunities. acknowledgments thank you to the following staff and students at scholars portal and utl that helped make this upgrade a success: steve baroti, gordon belray, bartek kawula, bikramjit singh, cordelia tang, and sean xiao zhao. thank you to alec smecher at pkp for pointing us in the right direction to resolving database encoding issues. thank you to james macgregor at pkp for providing valuable feedback on a draft of this article. references [1] quick facts: student enrolment (fall 2017 – 18), faculty & staff (fall 2016) [internet]. [updated 2018]. university of toronto; [cited 2018 dec 17] available from: https://www.utoronto.ca/about-u-of-t/quick-facts [2] student populations: student population statistics by institution [internet]. [updated 2018 aug]. ontario council of university libraries; [cited 2018 dec 5] available from: https://ocul.on.ca/node/21 [3] stranack, k. 5 steps to ojs 3: upgrade planning for service providers [internet]. [updated 2018 july 11]. [cited 2018 december 18]. available from: https://pkp.sfu.ca/2018/07/11/5-steps-to-ojs-3-upgrade-planning-for-service-providers/ appendix a: upgrade prep checklist work with your journals to identify their essential features and make sure they are available in the new version. consider cleaning up the content you don’t need to carry over, including inactive journals, spam users, etc. decide if you will provide sandbox sites to journals for testing (or use pkp’s ojs test-drive instance). set aside time before and after the upgrade for testing, training & troubleshooting. check that your contact information for hosted journals is up-to-date and re-establish contact in case of editorial changes. announce plans to upgrade early, even if you don’t have exact timelines yet. set downtime for production upgrade – and request confirmation. set content freeze dates if necessary. align your training/consultation offerings with demand – your editors may prefer emails or a call over workshops and drop-ins. define your level of support and communicate it clearly. set the cut-off date for pre-upgrade support requests. create frequent backups of your code, database, and files as you work through each upgrade. when setting up your testing instance, disable any plugins that may send data out to other services or systems. consult the pkp forums and open github issues when troubleshooting problems. put together an easy to follow the test plan for the journal teams to test and report issues. if you are looking to redesign your ojs homepage, consider timing it with the upgrade to reduce the impact on users. if there are aspects of your server architecture you could improve, consider making these changes during the upgrade to reduce future downtime or impact on users. if critical functionality is not available in the new version, consider postponing the upgrade. be prepared to be flexible and offer additional assistance if the upgrade takes away previously used functionality, or where modifications are beyond the journal team’s skill level. when planning upgrade time, ensure that the time to run the upgrade script is considered for larger instances. consider turning off most php warnings so log files don’t become unmanageably large – you can always enable them later as needed for debugging. appendix b: upgrade resources the following resources are linked throughout this article and are compiled here for reference. pkp’s ojs testing instance: https://pkp.sfu.ca/ojs/ojs_demo/ fixing database encoding issues in ojs: https://github.com/kaitlinnewson/ojs-tools/blob/master/fix-database-encoding.md cleaning up spam users bash script: https://github.com/kaitlinnewson/ojs-tools/blob/master/bulkdeleteusers.sh pkp’s documented process: https://docs.pkp.sfu.ca/admin-guide/en/securing-your-system#cleaning-lots-of-users pkp upgrade documentation: https://github.com/pkp/ojs/tree/master/docs pkp community forum: https://forum.pkp.sfu.ca/ pkp administrator’s guide: https://docs.pkp.sfu.ca/admin-guide/ get emails with new github commits in zapier: https://zapier.com/apps/github/integrations/email/11485/get-emails-with-new-github-commits the university of toronto’s journal publishing guide: https://jps.library.utoronto.ca/index.php/pubguide appendix c: preand post-upgrade test plan for journal managers ojs 3.1 upgrade test plan ojs 3.1 upgrade schedule march – july – pre-upgrade preparation: editorial teams review ojs 3 documentation, explore their test instances, identify customizations and report any issues to the jps team at jps@library.utoronto.ca august 7-8 – upgrade: jps team performs platform upgrade. journal websites will not be available at this time. august 9-17 – post-upgrade set up: editorial teams test their new journal sites, apply customizations and report any issues to the jps team at jps@library.utoronto.ca drop-in sessions and consultations jps team offers drop-in sessions to help you test and set up your journal before and after the upgrade. drop-ins pre-upgrade* drop-ins post-upgrade* thursday july 12, 2-4pm tuesday august 14, 10am-12pm tuesday july 17, 10am-12pm thursday august 16, 2-4pm wednesday july 25, 2-4pm monday august 20, 10am-12pm thursday august 2, 10am-12pm wednesday august 22, 2-4pm * all drop-ins are held in study room #2, ground floor robarts library, 130 st. george st, toronto if you would like to schedule a virtual session or a meeting outside of the drop-in hours, contact us at jps@library.utoronto.ca your journal’s demo instance an upgraded demo instance of your journal is available at https://ojs.scholarsportal.info/toronto/ for testing, learning, and tweaking of your journal’s looks and settings. none of the changes made in the demo instance will be carried over to your upgraded website – they will need to be re-applied. contact the jps team if you have not received your demo instance or are experiencing access issues. pre-upgrade test checklist (on demo instance*) – report issues by july 20 access and subscriptions issue and article pages including full text files are accessible (recent issues published since the demo instance was created in the spring will not be there) for subscription journals – individual and institutional subscribers have access to their content website content the content on about, editorial team, announcements, and other pages is in place all internal and external links from website pages work all uploaded files (e.g. guidelines for authors, etc.) are accessible website appearance the items in the top navigation menu and sidebar menu are in place (these may be reset to default as a result of the upgrade and may need to be re-added in the demo and post-upgrade instance) your preferred theme is in place (you can select from one of the existing themes or customize and upload a css; save your css for re-upload after the upgrade) your journal’s logo/banner/favicon are in place (these may not display in demo instance; save them from the current ojs 2 instance for re-upload after the upgrade) submissions and review workflow submitted, under review and published articles are present in the workflow (recent articles submitted since the demo instance was created in the spring will not be there) workflow settings reflect your journal’s editorial process perform a test submission, review and publication of an article/issue if you are using a quick submit plugin rather than full review workflow – perform a test submission via the plugin user roles team members are assigned to their respective roles (journal manager, editor, reviewer, etc.) your team roles reflect your journal’s management process (these can be adjusted or removed) test roles at each level including journal manager, editor, author, reviewer, reader, etc. plugins the plugins your journal uses function as expected on the demo instance (report asap any missing or non-functioning plugins) communications notify your users and team members of the upcoming upgrade and impact on scheduling, if any post-upgrade test checklist (on upgraded live website) – after august 8 repeat testing steps from the pre-upgrade test checklist re-upload your customized css, if any, and re-add menu items where needed report any technical issues to jps@library.utoronto.ca notify your users of the completed upgrade ojs 3 resources ojs 3: setting up a journal in ojs 3 (pkp school video course, free with registration) – http://pkpschool.sfu.ca/courses/setting-up-a-journal-in-ojs-3/ ojs 3: editorial workflow in ojs 3 (pkp school video course, free with registration) – http://pkpschool.sfu.ca/courses/editorial-workflow-in-ojs-3/ pkp community forum – https://forum.pkp.sfu.ca/ utl’s journal production services support team – jps@library.utoronto.ca contact the jps team at jps@library.utoronto.ca with any questions or issues related to the upgrade or your journal website this test plan is adapted from the ojs 3.1.1test plan kindly shared by the ubc library. about the authors mariya maistrovskaya (mariya.maistrovskaya@utoronto.ca) is the institutional repositories librarian at the university of toronto libraries. she manages the tspace research repository, journal production services, and a suite of other services that support scholarly communications and advance open access at the university of toronto. kaitlin newson (kaitlin@scholarsportal.info) is the digital projects librarian at scholars portal, the technology service arm of the ontario council of university libraries. you can find her online at https://www.kaitlinnewson.com and on twitter as @kaitlinnewson. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – using a web services architecture with me, myself and i mission editorial committee process and structure code4lib issue 7, 2009-06-26 using a web services architecture with me, myself and i the uw-madison libraries library course page system is used to deliver electronic reserves materials and course-focused library instruction webpages to students. as part of a rewrite of our system we broke the application into three component pieces: a file repository, a course timetable data service, and an interface application for building and viewing individual course pages. the new three-piece system was written with an inward facing service-oriented architecture that allowed us to choose the best technologies to solve each of the tasks the entire system needs to accomplish. by stephen meyer introduction it seems counterintuitive to write code that provides web services that will only ever run across localhost connections on a single server. however, using an internal service-oriented architecture for our library course pages (lcp) application provided an extensible application structure for one of the larger locally-developed code bases at the uw-madison libraries. our lcp application delivers electronic reserves materials and custom library instruction web pages targeted at specific courses. it represents one of the core applications that the uw-madison libraries uses to support the teaching mission of the university. background the original library course page system grew out of the work of talented developers within one of the uw-madison branch libraries. after the application proved to be highly successful, and when the primary developer moved into an it management position within the law school, there was a push to move the application support to our central library it department. while preparing to move application support and evaluating pending enhancements we decided to work on a complete overhaul of the code base.  since we knew that the business rules and application workflows were solid, we chose to focus on cleaning up the underlying application architecture. reviewing the lcp application code we ascertained that there were three distinct components in play: file storage hooks into campus course metadata user interfaces for managing and accessing content while tackling the re-architecture, we decided to take a web services approach: breaking the application into three distinct applications, each designed to solve a single piece of the entire lcp system. where the original lcp system ran on a monolithic php code base our re-designed system runs on a combination of a fedora-based repository, a custom xml service for course metadata using the restlet java framework, and a traditional user interface web application written to use the ruby on rails framework. using three applications to work together to do a single job provided several advantages: process isolation – each component is able to excel at one job and one job only technology choice optimization – each component uses the technology best suited to its needs data model-based design – the data models for each component are distinct and therefore clean dividing up the application the choices that we made in picking the separate services were based on the kinds of data being managed. the primary questions we posed to different data elements included: how long will it last? what exactly does it do? where does it originate? and who or what uses it? knowing how the application should function due to the success and maturity of the original system, we used these questions to determine the three categories of data in the library course page application: files that needed to be stored and retrieved by patrons (primarily for reserves); course information so we would know which reserves and instruction web pages corresponded to which courses the structure and content to build human readable web pages storing files the original lcp system stored files on disk using php and apache to manage the file storage and access. with our rewrite we wanted to use something slightly more sophisticated to make the management of disk resources easier. file storage is part of the core functionality of most computer systems and we were not interested in reinventing wheels for this component of our system. therefore we looked for an existing solution to storing files. the lcp application stores files that are used for library reserves or library instruction sections. these will typically include reserves readings or the kinds of handouts library staff create when they teach bibliographic instruction sessions. the requirements for the system started with a well-defined api. additionally, since sharing data and making it more widely available than originally intended is a growing trend in it and at the core of the web services philosophy, we wanted to leave open the possibility that the instructional digital learning objects in the system could be retrieved in ways unknown to us at the present time. our instruction librarians contribute learning resources to the curriculum at our university. they have created various digital learning objects, which may be as simple as html code snippets that can be re-used in different contexts. some of the more complex resources combine multiple simple objects (images, videos, help and assistance web pages) into tutorials that are integrated into campus curricula. as instruction staff at the library create course pages we wanted to make sure that these digital learning objecs, large and small, could be used within the lcp application and potentially in other places on our library website or other campus websites. we chose to provide a stable store for files and to maintain a high degree of control over their use via a fedora repository. fedora has a well documented api and a stable community of developers. for this project we have a separate installation of fedora that is not used by non-lcp applications. the data modeling and business rules governing persistence and versioning of this data are dramatically different from those that govern our digital collections or institutional repository.  consequently, we chose to keep this fedora instance distinct from other uses of fedora at the uw-madison libraries. at the present time, this use of the repository is simple: librarians put things in and patrons get them out. in our application setup, fedora only listens to localhost connections for requests to create new objects or retrieve files (specifically, datastreams, in fedora lingo). those requests come from an interface component described in the section “data for presentation & interaction” below. the interface component coordinates authorization for adding to the repository and determines who has permission to retrieve items from it. course info the second major kind of data used by our application is campus course information. by working with our university’s central it department we can automatically deliver individual library course pages to students enrolled in the corresponding courses through our campus portal. there are three high-level stages that make this happen: first we pull a local copy of course metadata from our university’s central it department next the we expose the metadata pulled from step 1 to the application that creates library course pages finally, we provide a metadata feed back to our campus portal about our library course pages delivering our data feed back to the central campus portal is important because it is already serving essential course related content to students. the libraries can work easily with campus systems because we all share common metadata elements that originate within the campus timetable and enrollment system. it is key to understand that the libraries do not control the data elements, their organization, or semantics. these are determined by our local campus registrar. we use remote odbc database connections to pull a feed of the relevant course metadata from the central it department, and cache a local copy of timetable course offerings for fast use by lcp processes. we then make this data available via a web service to the application used by librarians to build a course page. the aim of this course metadata service is to make information available to people working on library reserves or course-focused instruction at the point of need. we rebuild our cache daily due to the volatile nature of our university’s course offerings; we maintain local data only for the current semester and the next semester. because of this highly localized data model, we chose to write our own custom data service for this piece of the lcp application. we wrote code for this component that pulls a fresh feed from campus it servers daily, stores the information in a local database optimized for point of need exposure in the form of xml, and which responds to requests using a simple url structure based on the campus metadata model. what we identified as particular to this component is the manner in which we wanted to expose it: by translating the elements that the uw-madison campus uses to identify courses into resources that can be accessed via restful url structures. we chose to use the restlet framework to write this part of the application. restlet is a lightweight java-based framework for writing web applications according to the representational state transfer (rest)-style architecture. it excels at mapping a url path into parameters that can then be easily accessed by application code. for example, in order to identify lecture 1 of the computer science course introduction to programming, offered during the spring 2009 semester, our campus uses 5 pieces of metadata: the term (semester) code, the session code, e.g., the standard 15 week session for spring and fall semesters, the campus subject (department) code, the course number, and the section number. we access the local cache of the xml metadata for this course by visiting the following url pattern: http://localhost/terms/<term_code>/<session_code>/<subject_code>/<course>/<section> the url for the comp sci course in question is: http://localhost/terms/1094/a1/266/302/001 the resource returned is an xml view with relevant data from our course timetable: <section> <type>lec</type> <number>001</number> <timeofday>01:00 pm 02:15 pm</timeofday> <daysofweek>tr</daysofweek> <course> <number>302</number> <title>introduction to programming</title> <department> <code>266</code> <name>computer sciences</name> </department> <session> <code>a1</code> <term> <code>1094</code> <semester>spring</semester> <year>2009</year> </term> </session> </course> <instructors> <instructor>skrentny,james douglas</instructor> </instructors> </section> other useful url-defined resources in our application include things like a list of courses offered by the computer sciences department: http://localhost/terms/1094/a1/266 or a list of departments offering courses during a given session: http://localhost/terms/1094/a1 these kinds of resources can be used to generate navigation screens when a librarian is selecting a course to create a new library course page (see figure 1). the restlet framework excels at mapping url-identified resources into programmatic objects. by splitting our lcp system into three distinct applications we were free to use this technology to solve one problem extremely well. figure 1. navigation screen for attaching course metadata to a library course page (click to enlarge) data for presentation & interaction the final component to our library course page system is an interface to all the other data. this piece provides both machine and human interaction among the different components. for this part we needed a technology that is strong at both human interfaces and machine interfaces. we chose the ruby on rails framework because it has an mvc framework designed to solve both of these tasks. the human interface our lcp system has a traditional web application component, a user interface for librarians to create course pages and the resulting set of web pages that students use to view the content. librarians build course pages to deliver reserves content or custom library instruction in the form of recommended database lists and search strategies for course topics. this portion of the lcp system acts as a highly customized content management system. common tasks for library staff building a course page might include: providing the link to reserves content via a deep link into a licensed full text database, uploading the course instructor’s syllabus into the lcp system so that it can be downloaded from the course page, providing the persistent link to a licensed electronic resource along with a description of search capabilities for the resource. patron view staff view figure 2. end user views for patrons and staff for the lcp application (click to enlarge) the machine interface the rails application is also the piece that coordinates the interaction among the different components. at a high level there are a few core tasks that this component performs in conjunction with the file repository and timetable service: generate lists of departments and courses for a given semester, store a file in the repository, verify the enrollment for a given student attempting to download a file associated with a given course, retrieve a file from the repository. the machine interaction of the ruby on rails interface with the other components lies at the center of our web service architecture and is illustrated in figure 3. figure 3. application request flows within the lcp system (click to enlarge) pros & cons to the web service architecture transitioning to a web service architecture does have a few downsides, in our experience.  the lcp system has a slightly higher degree of complexity in that we need to manage multiple applications rather than one. supporting the application requires a knowledge of the application components in order to quickly pinpoint where problems reside. this does make it a little more difficult for front-line support staff to troubleshoot what is going on with application problems. despite the upfront complexity of the three for one application structure, and as mentioned above, we chose to write the new version of our application with a service-based architecture because it provides optimal technology choices. any given application with a sufficient amount of complexity will solve many different tasks. by solving each of those functional areas individually we believe we are solving them in the most efficient manner possible. the process of designing the locally developed application structures also benefited from the web service orientation. when designing our timetable service, for example, we approached the design with a data-centric process. too often there is a temptation to begin designing web applications from the perspective of an end user at the expense of the data model. when we worked on the timetable service we were in the business of exposing marked up data in the form of xml. working with the web pages that would present the data to end users was not a factor at this stage since that was left up to the design of the user interface application. the data-driven design of this service meant we were focused on ensuring our data model was well formed and optimized prior to addressing its display issues. something went very wrong… …but it turned out ok. in the initial design of the lcp system the process of downloading was handled differently. initially downloads were processed through the restlet-based timetable service. the initial assumption was that in cases where enrollment needed to be checked for compliance with fair use copyright laws, the download process should be located as close as possible to where that enrollment data was stored. however, as the application activity picked up during the first week of classes it turned out that the initial configuration of the restlet application was struggling under the volume and kept crashing with broken pipe exceptions. fortunately, rails has native file streaming capabilities via its sendfile function. due to the service orientation of the rest of the application, all we needed to do was rearrange the directionality of the service calls that are used to verify enrollment. to understand how this works, see the modifications to the download process illustrated in figure 4. figure 4. application request for downloading files from the lcp system pre and post code refactoring (click to enlarge) instead of the timetable service asking the rails-based cms which courses a particular file is associated with, the rails application just needs to ask the timetable service which courses a given logged in student is associated with and the rails application can gather all the data it needs to do the authorization logic. if it passes, then the rails application can make the localhost request to fedora for the file and send it on to the student. conclusion & looking forward using a web service architecture for our library course page application has been a great experience for our development group. we were able to choose the best technologies to solve the different tasks at hand. we were able to design our applications based on the data models unique to each functional area. since we have isolated each functional area within its own service the data will be easier to use in other applications. specifically, our timetable service can be used by other applications that we would like to write, such as personalized library home pages. because we did not couple the timetable data tightly to the lcp application exclusively, we can make api calls against it for other projects. so even though our web services only run on localhost right now, we have built an infrastructure that can be used again and again. for further exploration w3 consortium definition of web services: http://www.w3.org/tr/ws-arch/#whatis restlet framework: http://www.restlet.org/ ruby on rails framework: http://rubyonrails.org/ fedora repository: http://www.fedora-commons.org/ appendx a: timetable service code example the following code snippets demonstrate using the restlet framework to serve up the xml needed to display a course listing for a department as illustrated in figure 1 above. these snippets are incomplete and cannot run on their own. they merely provide a demonstration of how we map a url-based resource definition to an xml represntation of a department object. the base application the following class extends the core restlet application class. it is used to define individual restlets and map them to url patterns. package edu.wisc.library.ltg.timetable.controller; import org.restlet.application; import org.restlet.restlet; import org.restlet.router; import edu.wisc.library.ltg.timetable.controller.term.*; import edu.wisc.library.ltg.timetable.model.timetable; public class baseapplication extends application { private timetable timetable; public baseapplication(timetable timetable) { this.timetable = timetable; } @override public restlet createroot() { // create a root router router router = new router(getcontext()); /* * create individual restlets for the types of responses that * we expect to see. * * the requests come in the form of * semester:session:department:coursenumber:sectionnumber */ restlet termindex = new termindex(timetable); restlet term = new termhandler(timetable); restlet session = new sessionhandler(timetable); restlet department = new departmenthandler(timetable); restlet course = new coursehandler(timetable); restlet section = new sectionhandler(timetable); // attach the handlers to the root router router.attach("/terms", termindex); router.attach("/terms/{term}", term); router.attach("/terms/{term}/{session}", session); router.attach("/terms/{term}/{session}/{department}", department); router.attach("/terms/{term}/{session}/{department}/{course}", course); router.attach("/terms/{term}/{session}/{department}/{course}/{section}",section); return router; } } the department handler the following class acts as a servlet that is handed requests to represent information about academic departments. the information it returns is a list of the courses offered by the department in question for a given semester and session. package edu.wisc.library.ltg.timetable.controller.term; import org.restlet.restlet; import org.restlet.data.mediatype; import org.restlet.data.request; import org.restlet.data.response; import org.restlet.data.status; import org.apache.log4j.logger; import edu.wisc.library.ltg.timetable.timetableexception; import edu.wisc.library.ltg.timetable.model.*; public class departmenthandler extends restlet { protected static logger log = logger.getlogger(departmenthandler.class); private department department; private session session; private timetable timetable; public departmenthandler(timetable timetable) { // the timetable is the hibernate-based orm connection to the database this.timetable = timetable; } @override public void handle(request request, response response) { try { // pull the relevant params out of the url for this handler as // defined by the base application. string termcode = (string) request.getattributes().get("term"); string sessioncode = (string) request.getattributes().get("session"); string campussubjectcode = (string) request.getattributes().get("department"); // get the session and department objects from the database session = timetable.getsession(termcode, sessioncode); department = timetable.getdepartment(campussubjectcode); // set the response object's entity to the xml representation of the department // and apply the appropriate mime type to the response. response.setentity(department.getxml(), mediatype.text_xml); } catch (timetableexception te) { // catch bad params or data from the url and set the appropriate response if( log.isinfoenabled() ) { log.warn( te.getmessage() ); log.warn( "returning response: bad client request" ); } string error = "path to department is invalid"; response.setstatus(new status(400), error); response.setentity(error, mediatype.text_plain); } } } as you can see, this is not a lot of code to run a servlet-style service. the restlet website has excellent tutorials for getting started with the framework. about the author steve meyer is a library application developer for the uw-madison libraries shared development group. you can reach him at smeyer@library.wisc.edu. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – editorial: big code, little code, open code, old code mission editorial committee process and structure code4lib issue 57, 2023-08-29 editorial: big code, little code, open code, old code paraphrasing the title of christine l. borgman’s inaugural lecture in göttingen some years ago “big data, little data, open data” i could say that the current issue of code4lib is about big code, little code, open code, old code. the good side of coding is that effective contribution could be done with different levels and types of background knowledge. the issue proves to us that even small modifications or sharing knowledge about command line usage of a tool might be very useful for the user community. let’s see what we have! stefano cossu, ruven pillay, glen robson and michael d. smith published the result of a classical measurement based experiment in evaluating htj2k as a drop-in replacement for jpeg2000 with iiif. they compared the effects of using different image formats (tiff, jpeg 2000 and high-throughput jpeg 2000 – abbreviated as htj2k) in the context of iiif requests. jennifer ye moon-chung’s article standardization of journal title information from interlibrary loan data: a customized python code approach shows that when we would like to use the log files that contains data entered by the users of interlibrary loan service we have clean the input data: normalize values such as issn and isbn numbers, and make use of external services to retrieve standardized title information. several noteworthy analyses called attention to the limitations of text corpora without proper metadata of the individual documents. erin wolfe in chrononlp: exploration and analysis of chronological textual corpora presents a web based application called chronolnp to help the usage of corpora in which the historical aspects are important. chronolnp enables the combination of temporal trend analysis and a variety of natural language processing approaches. do you use (or plan to use) folio in the backend? then aaron neslin and jaime taylor’s review a very small pond: discovery systems that can be used with folio in academic libraries is for you. they check all possible commercial and open source front-end options. for this review they had talks with library sysadmins to get informed about the practicalities. a plus: information about the support of accessibility for each tool. elizabeth joan kelly’s supporting library consortia website needs: two case studies shows how a central unit of a library consortia could support the partner institutions with customizable central services when the details of their requirements are different. the paper by vlastimil krejčíř, alžbeta strakošová, and jan adler from dspace to islandora: why and how is the only one from europe in this issue, and as far i remember the first one from the czech republic. the authors compare the two popular repository software (technological stack, data structure, customization etc.) and describe the process of migrating several services from one place to the other. creating a full multitenant back end user experience in omeka s with the teams module written by alexander dryden, daniel g. tracy highlights the problems of content and rights separation in omeka s when an institution would like to run a single instance for multiple projects. the authors not only make their usage scenarios clear, but provide with a solution, a new, open source module written by themselves. the forgotten disc: synthesis and recommendations for viable vcd preservation by andrew weaver, and ashley blewer introduces the reader the preservation of contents of video discs: what is this format, where it was popular, how to save the bitstream from it (including the metadata), and how we can view and further manipulate it. krista l. gray’s breathing life into archon: a case study in working with an unsupported system is a nice account on how a devoted archivist with some programming knowledge could keep a legacy software in sync with the changed requirements. personally for me it is very sympathetic that the author confesses her limitations. i hope it will encourage others with similar backgrounds to follow a similar path. how to select an open source software backing our service? there are some answers to this frequently asked question, but i don’t think that the library (or – thinking of the research software landscape – the whole academic) community encounters all possible aspects jenn colt’s an introduction to using metrics to assess the health and sustainability of library open source software projects sheds light to four recent metrics scrutinizing the behaviors of the development community behind a software. finally, let’s party like it’s 2023! kent fitch’s searching for meaning rather than keywords and returning answers rather than links shows his experiments with the usage of large language models (yes, including chatgpt). instead of talking from a bird’s perspective the paper discusses the advantages, disadvantages, limitations, and costs of down-to-earth use cases. many thanks to the authors for bringing all these to code4lib! subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – ‡biblios: an open source cataloging editor mission editorial committee process and structure code4lib issue 5, 2008-12-15 ‡biblios: an open source cataloging editor ‡biblios is an open source cataloging editor designed to allow libraries to perform copy and original cataloging in a web based environment. ‡biblios allows users to search for, edit, and save bibliographic records in the marc21/marcxml formats. it also allows users to send records directly to integrated library systems such as the koha ils. where most marc editors are part of an integrated library system (and therefore require logging in), ‡biblios allows users to catalog with an open source standalone system available anywhere via a web browser. unlike other cataloging editors, it offers an attractive user interface for searching, saving and editing cataloging records. this article describes the system architecture and design of ‡biblios. by chris catalfo history ‡biblios was a project to develop a web based cataloging editor, suitable for use with the koha ils or with other ilss, that i put forward to liblime as a proposal for the 2007 google summer of code. when it was accepted, i served as lead programmer for the project and joshua ferraro, ceo at liblime, mentored the work as the system architect. at the end of the summer of code, we had a minimally functional web application able to search several z39.50 targets and edit records in an integrated marc editor. the name, ‡biblios, pronounced “biblios,” has no special meaning, although it obviously invokes the idea of “books.” the double-dagger symbol is commonly used as a subfield delimiter in cataloging, and forms an ideal logo for ‡biblios because the symbol can be represented in both graphical and textual formats. inspiration for this use of a symbol in a logo was drawn from the ümlaut and *asterisk projects. since the google summer of code, ‡biblios has continued to be developed at liblime, inc. in the past year of work, we have made a number of enhancements and changes. we switched from using a perl cgi script to search z39.50 targets to using the pazpar2 metasearch tool developed by indexdata. this has greatly improved the searching experience. it has also enabled us to show search facets, which pazpar2 provides as part of its webservice. besides these changes, we also added the capability to define macros to run on records, using javascript to manipulate marcxml records. systems architecture the system architecture of ‡biblios consists of: a rich internet application using the extjs toolkit and google gears which provides the user interface a set of cgi scripts providing server side services, and the pazpar2 search middleware the rich internet application provides the front end user interface and allows the user to search, select, and edit records. google gears is a web browser extension which allows web applications to store data in an sqlite database on the user’s computer. this allows ‡biblios to save records to the user’s computer. the cgi scripts provide back end functionality (such as exporting records), while pazpar2 allows searching multiple z39.50 targets. figure 1: system architecture front end the front end is comprised of a single web page, developed using the extjs [1] javascript toolkit. the user never navigates away from this single page while using ‡biblios; in this sense it is like a desktop application. extjs provides a very rich set of widgets to use in constructing web applications. ‡biblios uses most of these widgets in its user interface, where each area utilizes an extjs panel. these panels are frames on the screen which can contain other widgets such as grids or sub panels. there are several treepanels for interacting with hierarchical data such as z39.50 search targets or folders containing saved records. treepanels can be loaded dynamically via ajax (as they are when presenting search facets) or they can be loaded from the google gears database (as when displaying folders of saved records). there are a number of gridpanels for interacting with tabular data (such as search results or lists of search targets). the gridpanels provide for handling record selection via mouse clicks or arrowing down, as well as for sorting columns of data. finally ‡biblios uses several dialog windows for actions such as uploading files. the user interface is designed to emulate a web-based email client. the left-hand sidebar offers a selection of ‘resources’ (z39.50 search targets, folders to save records into, and records to create). in the center panel there is generally a grid displaying either search results or records in a folder. when editing a record, the screen changes to show the marc editor. figure 2: ‡biblios ui [view full-size image] ‡biblios also uses extjs’s gridpanels for interacting with bibliographic records and other data stored in the google gears database, as well as to view search results. these provide a user interface for interacting with tabular data; they also provide for paging buttons, selecting of records, toolbars, and loading indicators. extjs grids can be configured to work with various data sources: simple javascript arrays of data, external data sources (with results returned by ajax calls), or custom data sources. ‡biblios makes use of a custom data store developed for google gears to allow for viewing bibliographic records and “search” and “send” targets. it also feeds the z39.50 search results returned by pazpar2 into an exjs data store for viewing. figure 3: ‡biblios extjs widgets [view full-size image] each action performed in ‡biblios that needs server-side processing uses ajax requests to send data from ‡biblios and to receive data from the server. in this way there is no need to wait for page reloads to complete actions. figure 4: ‡biblios data flow [view full-size image] use of google gears ‡biblios makes use of the google gears [2] browser plugin for storing of bibliographic records. google gears allows the browser to store data from web applications in an sqlite database available to the web application. each site that makes use of the plugin may create a database and modify that database, but may not modify the databases of other sites. google gears was chosen as a means to allow users to view and manipulate records they have previously saved from search results or to view records they have created. when returning to ‡biblios, users are able to view these previously saved records. although saving records to the user’s computer (in the form of the embedded sqlite database) is handy, in the future this code may be separated out into a plugin for allowing offline access to records. at times it has proven difficult to manage data in users’ gears databases because the database may contain stale data, such as configuration data, or the web application may be expecting more recent data. this would also allow browsers which aren’t supported by google gears to access the site. use of pazpar2 search middleware pazpar2 [3] is a server developed by indexdata which allows searching multiple z39.50 databases simultaneously and returning those results via a web service interface. ‡biblios uses pazpar2 to perform searching of user-defined z39.50 search targets. in the original design for searching from ‡biblios using pazpar2, a javascript library provided by indexdata was used to communicate with pazpar2 from the web browser. this script makes it possible for the web browser to send requests directly to the pazpar2 server, via a proxy server such as apache. because web browsers are forbidden from making ajax requests to domains or ports other than their own, it is necessary to use a proxy between the browser and the pazpar2 server. recently ‡biblios has moved to using a perl proxy script and associated perl module (paz.pl and pazpar2.pm) to route requests between ‡biblios and pazpar2. this has greatly simplified the javascript code in the browser, as it no longer has to deal with maintaining a sessions with pazpar2. by default, pazpar2 does not include full marcxml records in the brief metadata it returns for each search result. for ‡biblios , the pazpar2 configuration files were modified so that ‡biblios receives the full record upon performing a search. this retrieval of the full record slightly slows down search performance, but it greatly speeds up previewing and saving records from the search results grid. this modification also allows users to save large batches of records from their search results into either their ‘save folders’ (google gears database) or to their computer. use of cgi scripts ‡biblios uses several cgi scripts written in perl to provide some functionality that is not easily implemented in the browser, or is better implemented on the server. figure 5: cgi scripts [view full-size image] the following cgi scripts are used in ‡biblios : downloadmarc.pl: downloads records from ‡biblios to user’s computer this script accepts a post of marcxml data. it then uses the marc-record suite of perl modules [4] to convert from marcxml to a user-determined record format that is returned to the browser. download.pl: generic handler for downloading temporarily saved records. returns the name of a temporary file containing marcxml records (produced by downloadmarc.pl or uploadmarc.pl) to the browser. exportdb.pl: exports of google gears’ database data to user’s computer ‡biblios sends serialized json to this script and the script instructs the browser to download a temporary file containing that serialized json. this file may then be saved to the user’s computer and reimported into ‡biblios. paz.pl: cgi proxy script for sending requests to pazpar2 search middleware this script acts as a proxy between ‡biblios and the pazpar2 server. it accepts requests from ‡biblios and routes them to a pazpar2 server. it restarts sessions with pazpar2 as required when running new searches. pazpar2.pm: a perl module, originally developed by galen charlton for the koha ils and subsequently modified to respond to more pazpar2 request types. uploaddb.pl: script for uploading data to insert into google gears database. return data uploaded by the user to ‡biblios for entering into google gears database. uploadmarc.pl: handles uploading of files of marcxml or marc21 records into ‡biblios. accepts a file of marcxml or marc21 records, converts them to marcxml and returns this data to ‡biblios for entering in google gears database. xsltransform.pl: accepts a stylesheet and xml data to transform with that stylesheet. uses the libxml suite of perl modules [5] to perform the xslt transformation. ‡biblios uses this to generate the marc21 editor and generate previews of marcxml records. kohaws.pl: proxies web service requests from the ‡biblios koha plugin to an actual koha installation. the koha installation responds with an xml document and this script returns that document to ‡biblios for further processing or display. authoritiessruproxy.pl: proxies sru queries to an sru server. used for querying authority records when editing an authority-controlled field in the marc editor. the choice of perl as cgi scripting language for ‡biblios was pre-determined by its having started as a cataloging editor for the perl-based koha ils. perl also has robust support for marc21 record handling. the cgi scripts require the following perl modules: cgi lwp::useragent cgi::carp marc::record marc::batch marc::file::xml file::temp file::basename json cgi::session data::dumper xml::libxml xml::libxslt xml::simple communication with integrated library systems ‡biblios has the ability to retrieve records from and save records to external integrated library systems (ils). as of this writing there exists a plugin for the koha ils [6]. the plugin queries koha for the most recent version of a record found in the search results. the user is able to edit this record and then send it to koha. koha saves the record to its internal database and returns the record (with possible additions of item record tags) to ‡biblios for further editing. the plugin makes use of a simple web services api developed by galen charlton, vice president of development at liblime, inc. the api calls for the following methods, implemented in a restful way by the ils: authenticate: authenticate the ‡biblios application to the koha ils bibprofile: retrieve a list of tags and subfields from koha which must be present or which must have specified values (such as item location subfields) retrieve: given a biblionumber, retrieve the most recent version of this record from koha’s internal database save: save a marcxml record from the ‡biblios editor to koha the api is fully documented on the ‡biblios website [7]. generation of marcxml editor since marcxml is a simple xml format, ‡biblios generates an editor for marcxml records using an xslt stylesheet. the stylesheet generates input fields for each of the subfields, indicators and tag numbers in a record. it also generates a fixed fields editor for marc21 records. this editor use an xml description of marc21 fixed fields to gather data from the leader, 006, 007, and 008 fields and to generate html <select> elements for each of those. once the basic html for the editor is generated, further javascript processing creates extjs combo box elements for certain parts of the record (language and country fixed fields elements and authority controlled fields). figure 6: ‡biblios marc editor [view full-size image] at several times over the last year, generating the editor using either xslt or javascript has been explored. for a time ‡biblios relied on javascript, as there had been difficulties in getting internet explorer to to pull in multiple stylesheets into the browser’s xslt transformer. the javascript generated worked fine for a while. recently we have been experimenting with opening multiple records at a time (each in its own tab). generating multiple editors with javascript took too much processing time, and so i have moved to a server side xslt transformation: this gets around cross browser problems and is still fairly speedy. ‡biblios as an open source project recently liblime launched a website [8] dedicated to ‡biblios as an open source project. with this website we hope to foster a community of users and developers around the project. at this website we have provided some documentation as well as downloads, mailing lists and access to a public git repository (git [9] is a version control system) to allow contributions from interested developers. here are some other ideas for code contributions to ‡biblios: a nice “getting started” project would be to add record counts to ‡biblios’ save folders a “network storage” plugin to save records to an arbitrary network location a means of searching non-z39.50 databases and integrating those results into ‡biblios’ search results grid alternative types of marcxml editors: for example an editor which allows use of the editing syntax supported by terry reese’s marcedit [10] to get involved in developing ‡biblios (either the core web application or plugins), visit the ‡biblios web site at http://biblios.org/ for more detailed developer documentation and sign up to the biblios-dev and biblios-users google groups. future directions there are a number of features we would like to develop (or see developed) for ‡biblios. chief among these are support for other metadata formats and plugins to communicate with ilss besides koha. as of this writing, ‡biblios supports editing marcxml records (and ‡biblios ‘ fixed fields editor supports only marc21 fixed field elements). in the future ‡biblios should support editing other types of metadata used in libraries. editing records is accomplished by means of a plugin-type architecture, so it is quite feasible to develop a plugin to edit mods records, for example. the web services api supported by ‡biblios is quite simple and therefore should present a low barrier to entry for other ilss. i would like to put in place a plugin for the evergreen ils. of course, if ilss start supporting jangle (singer 2008), communicating with them should become even easier as there will be a well defined api for performing the kinds of actions ‡biblios’ koha plugin performs now. conclusion ‡biblios is a web based cataloging editor in the form of a rich internet application. the system architecture consists of a desktop-like web browser based front end developed using extjs, the pazpar2 search middleware server and a series of cgi scripts providing server side functionality. ‡biblios communicates with pazpar2, as well as with remote ilss through the use of ajax requests. ‡biblios will be useful to libraries who perform copy cataloging and would like to use records available from z39.50 servers. for performing original cataloging, ‡biblios can be supplemented with additional library-defined marcxml templates. since ‡biblios supports macros on marcxml records, it may also be used for batch processing. notes [1] ext js. http://extjs.com/ [2] google gears. http://gears.google.com/?hl=en [3] pazpar2. http://www.indexdata.dk/pazpar2/ [4] marc::record cpan module. http://search.cpan.org/~mikery/marc-record-2.0.0/lib/marc/record.pm [5] libxml cpan module. http://search.cpan.org/~pajas/xml-libxml-1.69/libxml.pod [6] koha open source ils. http://www.koha.org/ [7] ‡biblios ‘send’ web service api. http://biblios.org/docs/biblios-send-api [8] ‡biblios open source cataloging. http://biblios.org/ [9] git – fast version control system. http://git.or.cz/ [10] marcedit. http://oregonstate.edu/~reeset/marcedit/html/ references singer, ross and james farrugia (2008) “unveiling jangle: untangling library resources and exposing them through the atom publishing protocol.” the code4lib journal 4 http://journal.code4lib.org/articles/109 about the author chris catalfo is a developer at liblime, working on the ‡biblios open source metadata editor. tags: biblios, liblime, marc editor, metadata editor, open source subscribe to comments: for this article | for all articles 3 responses to "‡biblios: an open source cataloging editor" please leave a response below: jodi, 2010-03-17 see also chris’ code4lib 2008 talk “oss web-based cataloging tool” http://code4lib.org/conference/2008/catalfo emily lynema, 2011-06-28 i note that biblios.org no longer responds. it would appear that biblios has been fully integrated into koha, but it’s difficult to find any information about it on their website. does anyone know whether this is still available as a downloadable open source tool or its current disposition? graham seaman, 2012-07-05 the source code (last updated 2009) is currently available from http://git.librarypolice.com/?p=biblios.git;a=summary or https://github.com/ccatalfo/biblios_old this information courtesy of mj ray (slef) leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – the forgotten disc: synthesis and recommendations for viable vcd preservation mission editorial committee process and structure code4lib issue 57, 2023-08-29 the forgotten disc: synthesis and recommendations for viable vcd preservation as optical media held by cultural heritage institutions has fully transitioned from use as a digital preservation ‘solution’ to a digital preservation risk, an increasing amount of effort has been focused on exploring tools and workflows to migrate the data off of these materials before it is permanently lost to physical degradation. one optical format, however, has been broadly ignored by the existing body of work: the humble video cd. while never a dominant format in the anglosphere, the video cd, or vcd, held wide popularity from the 1990s through the 2000s in asia and other regions. as such, a dedicated exploration of preservation solutions for vcd has utility both as a resource for institutions that collect heavily in pacific rim materials, as well as a means to, in a minor way, aid in the ongoing efforts to expand the digital preservation corpus beyond its traditional focus of issues prevalent in north america and europe. this paper introduces an overview of vcd as a format and summarizes its unique characteristics that impact preservation decisions and presents the results of a survey of existing tools and methods for the migration of vcd contents. this paper conveys practical methods for migrating vcd material from the original carrier and into both digital preservation and access workflows. by andrew weaver and ashley blewer introduction despite its widespread adoption, optical media is widely recognized as being a format with severe preservation risks due to its high potential for damage, degradation and technical obsolescence (schweikert, 2018). this is particularly true in the case of recordable optical media, such as cd-rs, where the inherent vices of unstable base materials and variable recording quality combine to create a high level of uncertainty for the lifespan of these discs. (schüller, 2008). accordingly, an increasing amount of effort within the preservation field has been focused on exploring tools and workflows to migrate the data off of these materials before it is permanently lost. thanks to the generosity of practitioners, quite a lot of this work has been documented and made publicly available through resources such as george blood lp’s 2014 report for the library of congress (morel et al., 2014) and alexander duryee’s introduction to the topic in code4lib journal (duryee, 2014). one optical format, however, has been broadly ignored by the existing body of work: the humble video cd. while never a dominant format in the anglosphere, the video cd, or vcd, held wide popularity from the 1990s through the 2000s in asia and other regions. as such, a dedicated exploration of preservation solutions for vcd has utility both as a resource for institutions that collect heavily in pacific rim materials, as well as a means to, in a minor way, aid in the ongoing efforts to expand the digital preservation corpus beyond its traditional focus of issues prevalent in north america and europe. this paper will present an overview of vcd as a format, its unique characteristics that impact preservation decisions and a survey of existing tools and methods for the migration of vcd contents off of the original carrier and into digital preservation and access workflows. background & context vcd is a standard digital data format for storing video on a compact disc with the intended purpose of linear video playback. essentially, vcd was an optical media disc (similar to a cd or dvd) that was used to store and play video. as a popular format in the 1990s and 2000s, vcd usage fits between laserdisc and dvd, with an image quality comparable to vhs (davidson and lediaev, 1993). vcd discs were the same dimensions and composition of cds and dvds: discs were 120 mm (4.7 in.) in diameter and composed of a clear polycarbonate plastic substrate, a reflective metallic layer, and a clear plastic protective coating. physical preservation efforts should adhere to the same practice as other optical media formats. the authority for the vcd specification comes from the “white book” standard released in 1993 by a group of video technology companies (jvc, panasonic, philips, and sony). the title “white book” is due to this specification being part of a series of standards that were developed for compact discs around this time period, creating what is known as the rainbow books. the rainbow books started with red (cd-da digital audio) in 1980 until the purple book (for ddcd double density). the white book sits between the beige book (1992), which covers photo cd technology and was written by kodak, and the blue book (1995), which covers enhanced cd technology and was written by philips and sony. the vcd “white book” standard was based off of previous work on the karaoke-cd system. thus, vcds included and built upon similar technical features like using mpeg-1 video, closed captioning, and ability to select and start from defined sections (super video compact disc, 2001). as a format, vcds were especially popular in southeast asia and china. vcd premiered in these markets at the right time, as vhs had not yet saturated the market, as it had in other geographic regions. optical media also has less risk of mold and mildew accumulation compared to magnetic tape, making them more suitable for very warm climates. like cds and dvds, vcds were either commercially pressed or they were “burned” onto writable media. while writing this paper, the authors encountered great numbers of these “burned” vcds, both in collections that had been created by community members as well as items that were commercial releases that had been purchased for circulation. these vcds are subject to the same elevated risks noted for writable optical media formats. brief overview of vcd technical characteristics the defined length of a vcd was marketed as offering typically up to 74 or 80 minutes on one side, a bit less than a standard feature-length movie. this minute mark was just an estimate; size constraints were determined by the amount of video stored rather than video duration. technically, vcd discs could store up to 800 mb of compressed video, with the most commonly produced sizes being 650 mb (74 minutes) or 700 mb (80 minutes) [1]. so while vcds were advertised as having a certain number of minutes of video, the true maximum duration is determined by the video content and compression. vcds use the mpeg-1 encoding standard for compressing and storing both video and audio. mpeg-1 is defined under iso/iec 11172 specifications for mpeg-1. it may also be referred to as h.261 [2]. the mpeg-1 format was released the same year as vcd and sought to create video files that were of reasonable size and of sufficient quality (for the era). the structure of a typical vcd is broken up into one or more cd-rom xa (“extended architecture”) sectors: first track: contains an iso-9660 file system that stores 2048 bytes/sector. this track’s primary function is to hold metadata and pointers that point to specific content (e.g. a tracklist or chapter list). it can also hold still images or frames, mostly for use in menu screens. this track is small but crucial to the operation of a structured video cd. it uses the cd-rom xa formatting method “mode 2 form 1,” which takes up more space but allows for some error correction functionality. subsequent tracks: the second and any other additional tracks contain raw mpeg tracks that store 2324 bytes/sector, with one mpeg data packet stored per sector. these tracks are not inherently part of the disc’s file system structure and thus cannot be mounted. windows operating systems may assume the intended usage of these files, interpret the contents, and “mount” the files even though they are just raw data, similar to how some systems will virtualize the raw pcm tracks on audio cds as wav files. these tracks use cd-rom xa formatting method “mode 2 form 2”; they do not support error correction but each sector has the capacity to store more data. vcds can also contain exceptions to this standard arrangement, for example, discs containing audio (cd-da) tracks or only a single track without a file system.[3] migration considerations when browsing the contents of a vcd via the directory viewer of an operating system, there will be a folder called ‘mpegav’ (see figure 1) that contains one or more .dat “files”. these .dat files are filesystem representations of the mpeg sectors, not the mpeg sectors themselves. the other visible folders and files form the non-video elements of the disc’s structure, and include information such as playback sequence metadata [4]. figure 1. directory tree of test vcd with a relatively simple structure. any attempts to copy, open or access them on a linux system will fail with an i/o error. as noted above, the windows operating system, however, will virtualize and play this raw data. legacy methods mention workarounds for linux machines using the cdfs kernel driver, but unfortunately these are no longer viable due to cdfs being incompatible with the linux kernel since at least 2013 [5]. although directly copying data on a windows machine may be viable for accessing content on video cds, it is insufficient preservation practice to copy and paste the files from the disc onto a harddrive, and the bulk of the remainder of this paper will be focused around workflows designed for preservation minded applications where a complete and replicable raw image of the original disc is produced. similar to dvds, where information such as titles etc. may be lost if only the video information is extracted from the disc, the authors believe that the creation of an image, specifically a bin/cue pairing, is the most appropriate strategy when working with vcds for preservation purposes. the bin/cue formats work in combination to yield an exact replica of the disc. the .bin file holds the complete raw binary data of the disc contents and the .cue file stores metadata for associated track layouts.this method helps ensure preservation of the complete structure of the disc. an image, however, is only useful if it can be reliably interacted with in the future – and it is here that vcds are particularly unique when compared to other optical formats. with their combination of an iso-9660 track followed by an mpeg stream track or tracks, their images can’t be opened or played back in the same fashion as dvds or other common data disc formats, and some extra steps are necessary to interact with the contents of bin/cue pairs made from vcds. accordingly, we will compare a few possible methods of accessing vcd data both within preservation workflows as well as for simple access. tools tested for creating disc images with video cds being based on the cd-rom xa architecture, in selecting tools for comparison we sought out tools that have been used in workflows with other types of cd-rom xa-based items. mixed mode audio cds [6], being composed of a single session with a first track containing a data followed by successive tracks of pcm audio, seemed the most similar format with extant documentation available, such as the workflows shared in by johan van der knijff (van der knijff, 2015). of the tools we selected, two of them (isobuster and cdrdao) are commonly used in mixed mode audio cd preservation, and one (vcdimager) is a tool specific to vcds. figure 2. graph outlining the workflow process for transferring a vcd. (key: gray is for windows tools, blue is for linux tools, green are cross platform, black represents inputs/outputs of process. * indicates that access to the .dat files only possible in windows file browser) the sample commands in the following sections (with the exception of the description of isobuster) were written assuming the linux version of the tool is being used. isobuster isobuster is a commercial data recovery application that is capable of creating raw disc images in the bin/cue format and is specific to the windows operating system. of the tools we surveyed, isobuster is perhaps the most straightforward option for creating vcd images (see figure 2). while isobuster is commercial software that requires registration for full functionality, its licensing information notes that all vcd capabilities are within its free tier [7]. process to create bin/cue conveniently, isobuster is able to create a bin/cue pair from a vcd in a single step – the user simply selects ‘extract cd raw (*.bin)’ from the drop down menu and isobuster will output the pair in the chosen location. cdrdao cdrdao is a general use utility for writing audio and data cd-rs. cdrdao [8] (which stands for “cd-r disc at once”) is a free and open source command line utility for both win32 and linux that allows the reading and writing of cds in ‘disc-at-once’ mode [9]. this method extracts data in one pass, as opposed to the alternative methods that operate on a per-track or per-session basis. at the time of writing (july 2023) it is available in locations such as ubuntu’s package manager and has had a release this year (2023), which suggests it is still a stable project. cdrdao’s output when run on vcds is a bin/toc pair, where the bin file represents the raw data and the toc (table of contents) functions in the same manner as a cue sheet. because cdrdao does not generate a cue sheet, workflows that use it must use the associated, and descriptively named toc2cue tool as an additional step (see figure 2). process to create bin/cue the command (which assumed the default optical device) used in our testing to generate the bin/toc pair was: cdrdao read-cd --device /dev/sr0 --paranoia-mode 2 --read-raw --driver generic-mmc-raw --datafile output.bin output.toc it is worth noting that cdrdao includes the cd paranoia library [10], which is a library that aims to help achieve more accurate rips of audio cds. according to its manual page, cdrdao defaults to a setting of --paranoia-mode 3. in our testing we found that it was difficult to draw strong conclusions about the results of differing paranoia settings on damaged discs. there were a wide range of results across different discs and different drives with no setting consistently providing a clear advantage with regards to discernable a/v errors. the test command here includes the setting for level 2, but that is more to highlight the use of the flag than to endorse it – any workflow involving damaged discs would most likely benefit from some local experimentation to see how the different paranoia options interact with the drive or drives being used. the second command used to generate the bin/cue pair is toc2cue output.toc output.cue. this uses the toc2cue tool (which is a part of the cdrdao installation) and converts the .toc file created by cdrdao to a cue sheet that can be used by other tools. vcdimager vcdimager is a ‘full-featured mastering suite for authoring, disassembling and analyzing video cds’ [11] that is maintained as a part of the gnu project and has existed since the year 2000. as one of the few dedicated vcd projects that is still actively maintained it was of great interest to us as a tool. the suite includes a tool for ripping the contents out of vcds, as well as a tool for generating bin/cue images that can be written to discs from extracted vcd information. as such, it is not directly creating a bin/cue pair in the manner of the other tools tested, but rather is creating a file dump with an xml file of technical metadata that can then be used to engineer a viable vcd image. vcdxrip is also capable of extracting files from bin/cue images of vcds. process to create bin/cue the initial command to generate the bin/cue pair in our testing was vcdxrip -v -c=/dev/sr0 which generates a full dump of all the vcd’s contents and automatically converts the .dat files to .mpg for easy use. it also generates an xml file that contains technical information about the vcd which can be used both for analysis and to generate a bin/cue pair from the extracted vcd contents using the command vcdxbuild -b output.bin -c output.cue input.xml. comparison of damage handling vcdimager’s vcdxrip tool was noted in many legacy online sources as being unable to handle damaged discs, and testing confirms that its vcd extraction will abort when it hits the damaged portion of intentionally damaged test discs. additionally, beyond this limitation on physical discs, vcdxrip failed in an identical way on extracting files from a bin/cue pair generated from the damaged disc with alternate tools. figure 3. an example of error correction from one of the intentionally damaged test discs. note the visible block-like effect on the leg in the far right side of the screen caused by data from other adjacent frames being used to fill in the damaged portion of the image. cdrdao created a complete bin/cue pair from intentionally damaged test discs that was both able to be split into its component elements as well as mounted and played. as was noted in the above section, we found that using the flag --paranoia-mode 2 could at times result in different results than the tool default of  --paranoia-mode 3 depending on the nature of damage and the drive used. it is recommended that in any implementation some local experimentation be done to confirm if either setting results in more optimal results. isobuster also created a complete bin/cue pair from damaged test discs that was able to be manipulated and played back. comparison of replicability the following tables show both md5 hashes and file sizes for the bin outputs of each tool. note that isobuster and cdrdao both generated images that were identical while vcdxrip’s output (that was generated in its two step process) was different both in file size and file fingerprint. table 1. disc 1 – burned version of vcdimager’s ntsc sample disc. [12] tool md5 image size (bytes) isobuster 9aacc6b5cc28901b4a36282942750fc2 86123184 cdrdao 9aacc6b5cc28901b4a36282942750fc2 86123184 vcdimager e629f0bdddf37f81aaad74a78cb1a806 86475984 table 2. disc 2 – pressed commercial release of kikujiro no natsu disc 1 (see figure 3) [13]. tool md5 image size (bytes) isobuster 73ed9542490afc1162b87773a3cb24a6 677284272 cdrdao 73ed9542490afc1162b87773a3cb24a6 677284272 vcdimager 6026fa6dfe985601a37ff0a3f9899d92 677418336 figure 4. scanned image of one of the test discs (kikujiro no natsu) and color target card tools for manipulating and accessing disc images it is an oft repeated axiom that there is ‘no preservation without access’, and when it comes to vcds there are a few extra considerations post creation of bin/cue pairs to make their content useful and accessible. unlike disk images of dvd videos, which can be opened as-is via software like vlc and handbrake, or the wav/cue pairs often used in audio cd workflows, the bin/cue pair created from a vcd is not immediately usable and requires extra mediation. the tools and methods in this section address how to extract data and video information from vcd images. bchunk bchunk is a tool for converting bin/cue pairs into the iso format, and can be used to split apart vcd images into their data and video components. the command for this is bchunk -r input.bin input.cue output, which will result in two or more files created with the root name selected for output and the .iso extension. the -r flag used in this command is a critical element that allows bchunk to output the cd-rom xa mode 2 tracks that comprise vcds in raw format [14]. of the .iso files that are produced by this command, the first will be the data track from the vcd and the following files will actually be the video contents of the image. they can have their extension changed from .iso to .mpg and be manipulated in the same manner as a normal .mpg file. isobuster along with the creation of images, isobuster is capable of extracting information from bin/cue pairs. upon opening a bin/cue pair with isobuster (via either the bin or cue file) isobuster will display the available tracks. the first (data) track can have files directly extracted from it, while the successive video tracks will need to be extracted into either raw data, or via the likely most useful option, directly as mpg video files using the available extraction choice to ‘treat as video only.’ cdemu should it be desirable to mount the image either to play it in its native format or to extract files from it manually (for example in a situation where extraction was unusable) then vcd images can be mounted using the cdemu tool on linux and the wincdemu tool on windows (see figure 2). (the following commands are shown for cdemu) [15][16]. mounting a vcd with cdemu requires first creating a virtual device with the command cdemu add-device which will create a virtual device numbered as 00 for the first device and incrementally for successive devices. then a vcd image can be mounted using the bin/cue pair with cdemu load 00 input.cue. for successful use it is important to load the cue file rather than the bin. to easily ascertain where the virtual device has been mapped use the command cdemu device-mapping. a graphical version of cdemu called gcdemu exists and can be used as well in the same manner. once a vcd has been mounted in this manner it can be manipulated as if the original disc had been inserted into the computer. this means both that it can be played back in its original manner through functions such as the ‘open disc’ functionality of the vlc player, and also that files can be copied out of the discs. the virtualized nature of the dat files discussed earlier in this paper is still in effect even with a virtual device, and it is only possible to copy or open them on a windows system, so linux workflows will have to rely on bchunk for the extraction of video content. ffmpeg many vcds will have more than one mpg track (in the same way that dvd images have more than one vob file). should this be the case, and should it be desired to form a single mpg file for access, ffmpeg can be used to concatenate either the .dat files (if on windows) or the extracted mpg files with the following method adapted from ffmprovisr [17]. ffmpeg -i concat:input01.mpg\|input02.mpg\|input03.mpg -c copy output_file.mpg conclusion despite being based in north america where vcds were never a dominant format, the authors have encountered vcds in their institutional collections in contexts ranging from commercially purchased circulating releases to individually authored videos mixed within broader archival collections. the above tools and methods have been useful for integrating these items into our digital preservation workflows, and should serve as a good starting point for readers attempting to both preserve and maintain access to any vcds held in their collections. references davidson, andrew and lediaev, lucy. 1993. video cd: what is video cd? philips interactive media duryee, alexander. 2014. an introduction to optical media preservation. code4lib journal 24. [accessed 2023 july 7]. available from: https://journal.code4lib.org/articles/9581 morel m, blood g, foltz b, gallo b, fuhrig l, finkel p, mcgrath j, warnock r, priebe j, coates b, kelly h. preserving write-once dvds producing disc images, extracting content, and preserving write-once dvds: producing disc images, extracting content, and addressing flaws and errors. 2014. [accessed 2023 jul 17]. available from: https://www.digitizationguidelines.gov/audio-visual/documents/preserve_dvds_bloodreport_20140901.pdf super video compact disc a technical explanation. 2001. [accessed 2023 jul 05]. available from: https://wayback.archive-it.org/all/20080528131354/http://www.ip.philips.com/view_attachment/2450/sl00812.pdf schüller, dietrich. 2008. audio and video carriers: recording principles, storage and handling, maintenance of equipment, format and equipment obsolescence. [accessed 2023 jul 05]. available from: http://www.tape-online.net/docs/audio_and_video_carriers.pdf schweikert, annie. 2018. an optical media preservation strategyvfor new york university’s fales library & special collections. [accessed 2023 jul 05]. available from: https://archive.nyu.edu/handle/2451/43877 van der knijff j. preserving optical media from the command-line. 2015 nov 13 [accessed 2023 jul 18]. available from: https://www.bitsgalore.org/2015/11/13/preserving-optical-media-from-the-command-line endnotes [1] information at https://www.videohelp.com/vcd [2] for more information see https://www.loc.gov/preservation/digital/formats/fdd/fdd000035.shtml [3] http://www.mplayerhq.hu/docs/html/en/vcd.html [4] while now only available via the wayback machine, the legacy vcdimager documentation was often referenced by other sources and is a particularly good explainer of vcd structure and properties. https://web.archive.org/web/20050207052857/http://www.vcdimager.org/pub/vcdimager/manuals/0.7/vcdimager.html#sec6 [5] bug report and subsequent removal from debian packages here: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=718966 [6] https://en.wikipedia.org/wiki/mixed_mode_cd [7] https://www.isobuster.com/license-models.php [8] https://cdrdao.sourceforge.net/index.html [9] https://en.wikipedia.org/wiki/optical_disc_recording_modes [10] https://www.xiph.org/paranoia/ [11] https://www.gnu.org/software/vcdimager/#devel [12] the ntsc sample disc used in our testing was made from the test image on the legacy vcdimager website, which is a good example of a complex vcd. https://web.archive.org/web/20051229173932/http://www.vcdimager.org/pub/vcdimager/examples/demovcd/ [13] note that this item has been treated as a dvd in its catalog record for the purpose of sorting: https://orbiscascade-washington.primo.exlibrisgroup.com/permalink/01alliance_uw/1juclfo/alma99110527400001452. this appears to be not uncommon, and represents a potential challenge for any surveys of vcds contained within library collections. [14] https://linux.die.net/man/1/bchunk [15] https://cdemu.sourceforge.io/ [16] https://wincdemu.sysprogs.org/ [17] https://amiaopensource.github.io/ffmprovisr/#dvd_to_file about the authors andrew weaver is the media preservation librarian at the university of washington libraries (orcid: https://orcid.org/0000-0002-9792-4036). ashley blewer is an owner of archives of tomorrow, an archives technology company. (orcid: https://orcid.org/0009-0001-0814-5226). subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – presenting results as dynamically generated co-authorship subgraphs in semantic digital library collections mission editorial committee process and structure code4lib issue 16, 2012-02-03 presenting results as dynamically generated co-authorship subgraphs in semantic digital library collections semantic web representations of data are by definition graphs, and these graphs can be explored using concepts from graph theory.  this paper demonstrates how semantically mapped bibliographic metadata, combined with a lightweight software architecture and web-based graph visualization tools, can be used to generate dynamic authorship graphs in response to typical user queries, as an alternative to more common text-based results presentations.  it also shows how centrality measures and path analysis techniques from social network analysis can be used to enhance the visualization of query results. the resulting graphs require modestly more cognitive engagement from the user but offer insights not available from text. by james powell, tamara m. mcmahon, ketan mane, laniece miller, linn collins los alamos national laboratory semantic web representations of data are by definition graphs, and these graphs can be explored using concepts from graph theory.  this paper demonstrates how semantically mapped bibliographic metadata, combined with a lightweight software architecture and web-based graph visualization tools, can be used to generate dynamic authorship graphs in response to typical user queries, as an alternative to more common text-based results presentations.  it also shows how centrality measures and path analysis techniques from social network analysis can be used to enhance the visualization of query results. the resulting graphs require modestly more cognitive engagement from the user but offer insights not available from text. visualizations and libraries library professional staff and patrons tend to be skeptical of exotic methods of exploring bibliographic metadata, and in particular they are skeptical of visualization tools. this skepticism may in part be a reaction to numerous well-intended research projects which have delivered novel visualization tools with great potential, but with correspondingly significant limitations, including such projects as activegraph, i-scape, and envision. the envision project [nowell et al. 1996] was an early effort to build a dynamic graphical interface for exploring objects in a digital library, but as the creators note “full use of envision’s graphic view requires access to a number of document characteristics that are infrequently available in bibliographic databases or library system[s].” while users reportedly “liked the flexibility and power of the graph view” the system was ultimately too far ahead of its time, with respect to the capabilities of 1990’s desktop computers, and the aforementioned limitations of the available metadata. another challenge facing visualization tools is that some visualizations are not readily understood by novice users. envision relied heavily upon displaying individual items on a scatterplot graph, and representing other characteristics of the items by varying the individual item’s icon size, shape and color, and by supplying a relevant label for each item. while there is certainly utility in this approach to representing and exploring digital library content, and for detecting trends and relationships that would not be apparent by perusing a textual results list, it was an interface that presented a steep learning curve for a non-scientist. a better approach might involve presenting users with a traditional textual interface initially, and offering the option for exploring particular aspects of the data using dynamically generated visualizations. one type of visualization could provide users with the option of exploring the co-authorship networks of various items in the result set supplied in response to a query, using a graph (or network) visualization. social networks experts in a particular field often have strong ties to one another, and these ties represent professional collaborations. exploring these linkages can lead to serendipitous discoveries. [kules et al. 2008] but most digital libraries display authors as a simple list of names, just as they appear in the source publication. it is cognitively demanding for a user to maintain a mental map of the social network from lists of names. a potential solution is to provide users with a simple visualization of the authorship networks to which a particular author belongs, which has been trimmed so that the nodes and edges displayed to the user are for publications which match a user’s query (see figure 1). these co-authorship graphs have the added benefit of being almost instantaneously comprehensible to users. figure 1. an authorship network where the author with the highest betweenness centrality is highlighted. a network represents people and the relationships between them as a collection of nodes and edges. nodes, or vertices, are objects such as authors, subjects, or document titles. edges represent properties or relationships that bind objects to one another. if a graph does not have too many nodes and edges, it is readily comprehensible to a user, at a glance. they can see patterns such as frequent co-authors or common subject headings shared by a collection of bibliographic records, for example. denser graphs benefit from the application of centrality measures, which can be used to highlight certain special nodes. centrality measures are well-known in social network analysis, where they reveal patterns of interaction and communication among nodes (authors) in a social network. an example of a centrality measure is betweenness centrality, which can reveal, in a social network, a person who is critical to communication between two distant people, or between clusters of people. [powell et al. 2011, preprint] graph theory graph theory is at the root of network visualization tools. the authors of the textbook “graph theory, 1736-1936” note that “the origins of graph theory are humble, even frivolous… the problems which lead to the development of graph theory were often little more than puzzles.” yet graph theory, which was often referred to as the geometry of position by early mathematicians, is able to reveal valuable information about connections, roles, and relationships among objects, places, or people. leonhard euler, sometimes credited with its invention, was interested in a 16th-century puzzle about whether there was a path that would allow one to cross the seven bridges of konisberg without repeating any leg of the journey more than once. his interest led to a powerful new field of mathematics with interdisciplinary applications, including many applications in digital libraries. [biggs 1986] graphs, or network-based approaches, were first applied to library data in the mid-20th century [de solla 1965], to explore citation networks. our project provides digital library users with the ability to dynamically establish boundary conditions, or trim the graphs so that the resulting authorship network is relevant to a user’s query. boundary conditions establish constraints that determine the extent of a network, by trimming the graph of all authors and co-authors for a collection so that it only contains authors and co-authors of papers whose metadata contain some keywords related to a user’s query. these authors may have written other papers on related topics, but by constraining the network based on metadata queries, we believe we can offer networks of a reasonable size for visual exploration. results as graphs our approach is to provide search results in the form of subgraphs of a co-authorship graph for a set of bibliographic data that has been previously converted to rdf triples. users perform keyword searches against titles and/or abstracts. results are initially presented in text form, with various author graph variants available via links. for example, a user can view a clipped co-authorship network that corresponds to search results. users can optionally select graph variants where the highest degree or highest betweenness centrality node is highlighted. a degree distribution plot accompanies the graph, since this can be helpful in comprehending the overall topology of the graph. for example, does the network exhibit scale-free characteristics, that is, does it have a few hubs (highly connected nodes) and a lot of nodes with just a few, or a single, connection? (see figure 2) this can help provide context for the graph. finally, we also provide tools which allow users to see who a given author’s direct collaborators are, and to find the “degrees of separation” between two authors (if they are in fact connected). [newman 2001] figure 2. initial view of results set in infosynth. we call this co-authorship graph exploration tool the social awareness tool, or sat. it is a component of a search environment we developed for collections of semantic web data, called infosynth. infosynth is described in several papers, including [powell et al. 2010], [powell et al. june/july 2011], and [powell et al. november 2011]. for the sat we use a similar model as employed for the geographic awareness tool [powell et al. november 2011], where discrete service layers (see figure 3), each with a respective role in processing the user query, are called in order to fulfill a user request. the data required for building an authorship network are very simple, and are provided by most digital library systems exposed via oai-pmh. [lagoze et al. 2002] it is essential to have the document title, and a list of authors. each author’s name must occur separate from other author’s names – if all names are combined in a single field, then it is less likely that the names can be programmatically extracted from the metadata in a reliable fashion. some oai repositories fail to provide authorship data at this level of granularity. additionally, the first author should either be explicitly designated in some fashion, or it should reliably occur as the first name associated with the record. here is an example of what the metadata might look like in dublin core [dublin core metadata initiative]: <dc:title>emergency information synthesis and awareness using e-sos.<dc:title> <dc:creator>collins, linn marks</dc:creator> <dc:creator>james e. powell, jr.</dc:creator> <dc:creator>carolyn dunford</dc:creator> ... in this example, the first author is linn marks collins, and subsequent authors contributed to the article, but no assumptions are made regarding the nature or degree of their contributions. as you can see from this small amount of data, challenges in matching names across publications quickly emerge. for example, if linn is second author of another publication, then it is possible that her name will appear as “linn marks collins” rather than “collins, linn marks.” another common occurrence is that her name may appear with her first and middle name abbreviated, as “collins, l.m.” so the heuristics for name disambiguation quickly become complex, and the reliability of those rules decreases as the amount of information available for matching decreases. matching can be improved when there is additional information available to support a given rule, for example, if the metadata also provided the organization name to which the author is associated, or an email address for each author. author name disambiguation remains a major challenge for digital libraries. [smalheiser et al. 2009] for our purposes, we elected to use the following three rules to determine if one author name matches another: 1. does the new author match a previously detected author exactly? 2. if the name contains a comma, if we swap the portion of the name after the comma with the portion of the name before the comma, does it now exactly match a previously detected author? 3. does the name include abbreviations (a single character followed by a period)? if so, does this match an abbreviated version of a previously detected name? while not comprehensive or fool-proof, these rules represent a fair trade-off in terms of matching author names within a particular field. these rules may be good enough for content harvested via oai-pmh from a topically or institutionally focused repository. however, with larger collections of metadata spanning multiple disciplines, the likelihood of false matches becomes greater, and the resulting authorship networks become increasingly unreliable. ironically, entity disambiguation efforts can be enhanced by graph analysis techniques. this means there are opportunities to use the resulting graphs to improve author matching, for example, in instances where two entities are textually similar share many of the same relationships with other entities [bhattacharya and getoor 2007]. data conversion since we convert harvested bibliographic records into the resource description framework format (rdf) [brickley and guha 2005], using a process like that described in powell et al. [july/august 2010], we use a well-established practice of storing some of this data in a relational database. each unique name is assigned a unique author identifier, a universally unique id or uuid, which takes the form of an unresolvable uri. this identifier is used internally by our applications when they generate authorship networks data in response to user queries. it is also possible to store other data to facilitate name matching. for example, there could be a name variant table with a linker table associating all detected variants of an author’s name with an identifier, as well as other data, such as organizations the author has been associated and even temporal data indicating the period of time the author was known to be associated with that organization. the relational database supports only name matching during record conversion. dublin core metadata records are mapped to an rdf/xml serialization, and these triples are loaded into an rdf repository. rdf is especially well suited for representing this data, because the authorship data that implies relationships among authors can be made explicit using the friend of a friend (foaf) ontology [brickley and miller 200], in conjunction with dublin core metadata elements, which can be used as predicates in rdf triples. it is also easy to add new facts in the form of new statements whenever they are discovered. rdf is more readily understood by humans when triples are expressed in another standard notation, called notation-3 or n3 [palmer]. here is an n3 triple derived from the dublin core metadata example featured above: “collins, linn marks” “emergency information synthesis and awareness using e-sos”. this triple expresses the fact that linn marks collins was an author of the paper entitled “emergency information synthesis and awareness using e-sos”. in practice, associations are more typically made between object identifiers, rather than string literals. for example, the document title in this example would instead be an identifier for a set of triples about this paper. the generated triples are then used to populate an rdf/xml repository representing the oai harvested content (an oai service provider). authorship networks are expressed as a series of rdf/xml statements associated with a particular article. during the metadata to rdf/xml mapping process, the unique identifier assigned to a given author is retrieved during the name matching process, and reused across publications when an author name is determined to match a previously detected author name. in this way we can explicitly represent relationships between authors across multiple publications, in rdf. the rdf/xml example below represents a portion of the mapping of a record from the nasa public oai repository at http://ntrs.nasa.gov/. j.l. steinberg is identified as a person using foaf, and as the first author of a publication about the radiation environment in proximity to the earth. co-authors s. hoang and michelle f. thomsen are identified as co-authors, and associated with steinberg with the triple, which references the first author’s unique identifier. these triples represent a simple authorship network between the authors and co-authors of this paper. <foaf:person rdf:about="uuid:5d6d9d88-956e-4759-9bd8-5528c4239f7b"> <foaf:name>steinberg, j. l.</foaf:name> </foaf:person> <foaf:person rdf:about="uuid:b0ee82b9-befa-4917-a2b0-c90ba575e8f6"> <foaf:name>hoang, s.</foaf:name> <foaf:knows rdf:resource="uuid:5d6d9d88-956e-4759-9bd8-5528c4239f7b"/> </foaf:person> <foaf:person rdf:about="uuid:5b425905-d8a5-4fc5-9779-b5daa313ed83"> <foaf:name>thomsen, michelle f.</foaf:name> <foaf:knows rdf:resource="uuid:5d6d9d88-956e-4759-9bd8-5528c4239f7b"/> </foaf:person> <rdf:description rdf:about="uuid:3826cdbc-b63a-403d-9785-9d0ee6973b60"> <dc:identifier>info:lanl-repo/laauthors/la-ur-90-1420</dc:identifier> <dc:title>observations of the earth's continuum radiation in the distant magnetotail with isee-3.</dc:title> <dc:creator> <rdf:seq> <rdf:li rdf:resource="uuid:5d6d9d88-956e-4759-9bd8-5528c4239f7b" /> <rdf:li rdf:resource="uuid:b0ee82b9-befa-4917-a2b0-c90ba575e8f6" /> <rdf:li rdf:resource="uuid:5b425905-d8a5-4fc5-9779-b5daa313ed83" /> </rdf:seq> </rdf> </foaf> once we have mapped a set of harvested records from dublin core to rdf/xml, we can either create a new rdf repository in which to store these triples, or add them to an existing rdf repository. we chose the openrdf’s sesame triplestore to store our content and to build our authorship network service. openrdf provides an application programming interface (api) for programmatically accessing repository content. figure 3. architecture and data flow for the sat. the middleware layer the sat middleware layer supports direct retrieval of graph data in various raw formats supported by other visualization tools, or by other infosynth components. for example, pajek compatible output might be used with pajek to render different networks, explore larger networks that the guess applet can accommodate, or to merge with other networks. these graphs are generated dynamically in response to a user query. the user query is converted into a sparql query, which is a query language for rdf data. [w3c sparql query language for rdf] a very basic sparql query, which can be thought of as “select all statements” looks like this: select ?subject ?predicate ?object where { ?subject ?predicate ?object. } in practice, queries are usually more specific, and include properties such as filter clauses to narrow the results. if a triple doesn’t exist for a given set of data, it may simply be omitted. however, this possibility must be taken into account when querying the data. the sparql query language includes an optional clause, which allows a query to be executed successfully even if some data is missing. portions of the query which involve data which may or may not be present are specified using the optional clause, as this example illustrates: select ?title ?description ?placename where { ?id <http://purl.org/dc/elements/1.1/#title> ?title. ?id <http://purl.org/dc/elements/1.1/#description> ?description. optional { ?id <http://xmlns.com/foaf/0.1/#topic> ?y. ?y <http://www.geonames.org/ontology#name> ?placename. } } this query against a set of triples describing documents expects titles and description statements to occur for all objects, but topic and location information may or may not be present. using the openrdf api, we built a middleware layer to query our repository and return results as authorship networks. the api provides classes for establishing a connection to an rdf repository, constructing and submitting a query, and a data type container for results – somewhat reminiscent of what the java database connector (jdbc) api provides application developers for interacting with relational databases. the java servlet that handles queries uses a partially constructed sparql query and incorporates terms supplied by the user into a filter clause. here is the previous query, with a filter by title clause that will result in any statements where the title contains the word “mars” to be returned: select ?title ?description ?placename where { ?id <http://purl.org/dc/elements/1.1/#title> ?title. ?id <http://purl.org/dc/elements/1.1/#description> ?description. optional { ?id <http://xmlns.com/foaf/0.1/#topic> ?y. ?y <http://www.geonames.org/ontology#name> ?placename. } filter regex(?title, “mars”, “i”) } next, here is an example of an actual query used by the infosynth middleware layer (with namespaces defined with prefixes). the query retrieves only co-authors, so that the authorship network for each matching title can be constructed, with edges between co-authors and the main author. the use of the rdf:seq element in the rdf/xml ensures that the order of authorship is maintained. the filter clause indicates that only items which have a dc:title containing the word “mars” should be returned. the string “mars” is an example of a value passed in as a parameter to the java servlet, representing a query term entered by a user. prefix dc: <http://purl.org/dc/elements/1.1/#> prefix foaf: <http://xmlns.com/foaf/0.1/#> prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> select distinct ?docuuid ?title ?node ?name ?knows ?knowsname where { ?docuuid dc:title ?title. ?docuuid dc:creator ?seq. ?seq ?li ?node. ?node foaf:name ?name. optional { ?node foaf:knows ?knows. ?knows foaf:name ?knowsname. } filter (bound(?knows)) filter regex(?title, “mars”, “i”) } the social awareness middleware layer can be addressed via a rest query, such as this one: http://hostname/satrdf/srdf?q=mars&repo=nasa&format=gdf the q parameter represents the user’s query, the repo parameter is the name of the rdf repository to be searched, and the format parameter specifies the format for the results. the service can return results as graphml [brandes et al. 2005], gdf (guess data format) [adar 2006], a pajek compatible .net file, xgmml [xgmml …], or a graphviz compatible .dot file. graphml is a general purpose language for representing the edges, nodes and related values for a graph, as xml. it is supported by a variety of visualization tools (see figure 4). gdf is a native format for the guess visualization framework – a java library for rendering graphs. although guess technically supports both formats, we found that guess rendered gdf data far more rapidly than it did graphml data. figure 4. cytoscapeweb rendering of a co-authorship graph, which was output by a middleware service in graphml. the rendering layer the sat’s rendering service is what the user actually interacts with when viewing social network data. a user submits a search via a simple web form. that query is converted into a query against the middleware service layer described above, and the results are combined with the guess visualization applet, or the cytoscape web tool, and returned to the user’s web browser. that visualization shows all of the authors who are associated with papers that matched the user’s query. in many cases, it reveals connections among many authors associated with the search term, thus potentially revealing others who may have made significant contributions in this field. the visualizations are simple and easy to grasp at first glance, whereas a simple list of authors associated with various publications would result in a dense network graph and is more likely to quickly overwhelm the user and limit the ability to view collaborations among authors. a user has the option of viewing alternate versions of the clipped graph that matches a user’s query. alternate views include graphs where the node with the highest degree or highest betweenness centrality is highlighted. this is helpful when viewing large or dense graphs. as a result, it is easy to represent the graph as an adjacency matrix, and calculate centrality measures such as betweenness and degree centrality for each node in the graph. there is also the option to view a table of all selected author nodes, with their respective centrality measures displayed numerically. this data could be used as the basis for other features, such as a graph view highlighting disparate clusters, or by applying other social network analysis techniques to the data. [mika 2004] finally, the same internal representations of these graphs can also be used to explore paths between authors. we use a java library called jgrapht [naveh 2005] to internally represent clipped graphs representing the results of a user query. this library provides a set of path analysis algorithms for exploring the existence and distance of connections among nodes. we use the neighborindex class to generate a navigable view of authors and co-authors. once an author is selected, all of their collaborators are returned in a list form, and users can click on collaborator names to see their co-authors. we also use the bellmanfordshortestpath class to determine if any two authors are connected and if so, how long the path is between them (figure 5). this is the same idea behind various projects such as the erdos number and the “six degrees of kevin bacon” project. path analysis can be useful in exploring the evolution of ideas and collaborations. figure 5. degrees of separation between robert oppenheimer and another scientist. conclusion this paper describes how we used semantic web technologies to build an oai service that render dynamically generated authorship networks in response to a user query against a harvested digital library, and how these graphs can be further augmented to highlight significant contributors. it has been tested against a wide variety of oai repositories, and against rdf repositories that represent content from a single oai data provider, and repositories which contain an aggregation of content from several oai data providers. the oai service provides a middleware layer which is capable of providing standards-compliant (graphml) output, which makes it possible to use with a variety of network visualization tools, or incorporate into other service oriented digital libraries. it also provides a simple rendering layer which supports visualizations of results sets. it uses basic name disambiguation heuristics to match author names during the process of mapping harvested metadata to rdf/xml. the application layer uses the openrdf libraries, which are supported by a number of rdf repositories, including highly scalable offerings from commercial vendors such as allegrograph. these tools are designed to be both modular and scalable, to function as a standalone service, or to be integrated with an existing digital library. the rendering services provide users with simple but effective visualizations of a specific set of relationships that exist in digital library data. the granularity, specificity, and standards-compliance of the middleware services, and the fact that the rendering services can stand-alone make them easy to integrate with other tools. the simplicity of these graphs, coupled with the fact that they correspond to a user’s specific query, enhances their utility and ease of use. by normalizing topically specific, manageable sets of metadata to rdf we can offer simple, specific, and straightforward visualizations for exploring aspects of otherwise hidden social networks that exist in digital libraries. references adar, e 2006. guess: a language and interface for graph exploration. in: chi ’06 proceedings of the sigchi conference on human factors in computing systems,2006 april 22-27; montreal canada: acm. bhattacharya, i. and getoor, l.] 2007. entity resolution in graphs. in: cook, d.j., holder, l.b., editors. mining graph data. hoboken (nj): john wiley & sons, inc. p. 311-345. biggs, norman. 1986. graph theory, 1736-1936. oxford [oxfordshire] ; new york: clarendon press. brandes, u., eiglsperger, m., and lerner, j. 2005. graphml primer. [cited november 2, 2011]. available from: http://graphml.graphdrawing.org/primer/graphml-primer.html. brickley, d., and guha, r 2005. resource description framework (rdf) schema specification. [cited october 30, 2011]. available from: http://www.w3.org/tr/2000/cr-rdf-schema-20000327. brickley, d. and miller, l. 2005. foaf vocabulary specification. namespace document 27 july 2005. [cited november 2, 2011] available from: http://xmlns.com/foaf/spec/20070114.html. dublin core metadata initiative [internet].dublin core metadata initiative; [cited october 2011]. available from: http://dublincore.org/ kules, b., wilson, m., schraefel, m., shneiderman, b.] february 2008. from keyword search to exploration: how result visualization aids discovery on the web. college park (md): human-computer interaction lab technical report hcil-2008-06. lagoze, c., van de sompel, h., nelson, m., and warner, s. 2002, the open archives initiative protocol for metadata harvesting, version 2.0. [cited october 2011] available from: http://www.openarchives.org/oai/2.0/openarchivesprotocol.htm. mika, p. 2004. social networks and the semantic web, in: 2004 ieee/wic/acm international conference on web intelligence (wi’04), pp.285-291. naveh, s. 2005. jgrapht – a free java graph library. [cited november 2, 2011] available from: http://www.jgrapht.org/. newman, m. e. j. june 2006. scientific collaboration networks. ii. shortest paths, weighted networks, and centrality. physical review e 64, 016132 [cited october 30, 2011] available from http://link.aps.org/doi/10.1103/physreve.64.016132. nowell, l.t., france, r.k., hix,d., heath, l.s., fox, e.a. 1996. visualizing search results: some alternatives to query-document similarity. in: proceedings of the 19th annual international acm sigir conference on research and development in information retrieval (sigir ’96). acm, new york, ny, usa, 67-75. doi=10.1145/243199.243214 http://doi.acm.org/10.1145 palmer, s.b. a. rough guide to notation3. informesh.net. [cited march 24, 2011]. available from: http://infomesh.net/2002/notation3/. powell, j., alcazar, d., hopkins, m., olendorf, r., mcmahon, t., wu, a., and collins, l.m. 2011. “graphs in libraries: a primer” (la-ur 10-08362), information technology and libraries, december, 2011, vol30, no. 4. powell, j.e., collins, l.m., and martinez, m.l.b. july/august 2010. semantically enhancing collections of library and non-library content. d-lib magazine, [cited october 30, 2011] available from http://www.dlib.org/dlib/july10/powell/07powell.html powell, j., collins, l.m., and mcmahon, t. june/july 2011. embracing lossy. asis&t bulletin, vol. 37, no. 5. powell, j., mane, k., collins, l.m., martinez, m.l.b., and mcmahon, t. november 2011. the geographic awareness tool: techniques for geo-encoding digital library content. library hi tech news, 28(9). powell, j.e., alcazar, d.a., hopkins, m., olendorf, r., mcmahon, t.m., wu, a., & collins, l. 2011. graphs in libraries: a primer. ital: information technology and libraries (in press). [preprint cited march 24, 2011]. available from http://www.ala.org/ala/mgrps/divs/lita/ital/prepub/powell.pdf smalheiser, n. r. and torvik, v. i. 2009. author name disambiguation. annual review of information science and technology, 43:1–43. doi:10.1002/aris.2009.1440430113. de solla price, d.j. july 30, 1965. networks of scientific papers. science, 149(3683): 510-515. w3c. 2008. sparql query language for rdf. [cited november 2, 2011]. available from: http://www.w3.org/tr/rdf-sparql-query/. xgmml. [internet]. [updated may 2009]. wikipedia. [cited november 2, 2011]. available from: http://en.wikipedia.org/wiki/xgmml. about the authors james powell is a research technologist at the research library of los alamos national laboratory, and a member of the knowledge systems and human factors team where he develops digital library, semantic web, and ubiquitous computing tools to support various initiatives. he has worked in libraries off and on for over 20 years, including eight years at virginia tech university libraries, where he worked on the scholarly communications project and participated in several collaborations between the library and the computer science department’s digital library group. he later went on to assume the position of director of web application research and development at virginia tech, and to lead the internet application development group, before joining lanl. tamara m. mcmahon is a library technology professional at the los alamos national laboratory. her background in human factors and information science allows her to bring people, data and technology successfully together. her current work includes knowledge discovery systems and large scale data repositories. tamara holds a master of information science degree from indiana university. dr. ketan mane is a senior research informatics developer in the health informatics and bioscience group at renaissance computing institute (renci). his current research work focuses on applying visual analytics approaches in decision support role to help clinicians identify viable treatment options at the point of care. dr. mane has a background in biomedical engineering, and holds a ph.d. in information science from indiana university, bloomington (advisor: dr. katy borner). prior to joining renci, mane was a member of the infoviz lab at indiana university, bloomington. he has also worked as a postdoctoral research fellow at los alamos national lab (lanl) as part of the knowledge systems and human factors team. his research interest include: information visualization, visual analytics, decision support tools, health informatics, knowledge domain visualization laniece miller is a graduate research assistant the los alamos national laboratory. she recently finished a masters of science in library and information studies at florida state university. linn collins is a technical project manager at the los alamos national laboratory. her current work involves knowledge systems, information exploitation, and applying semantic web and social web technologies to challenges in national security. she received a doctorate in educational technology from columbia university in new york, where her dissertation was on semantic macrostructures. prior to lanl she worked at ibm research on eduport and the knowledge and collaboration machine, and at the massachusetts institute of technology on project athena. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – building a better book in the browser (using semantic web technologies and html5) mission editorial committee process and structure code4lib issue 29, 2015-07-15 building a better book in the browser (using semantic web technologies and html5) the library as place and service continues to be shaped by the legacy of the book. the book itself has evolved in recent years, with various technologies vying to become the next dominant book form. in this article, we discuss the design and development of our prototype software from montana state university (msu) library for presenting books inside of web browsers. the article outlines the contextual background and technological potential for publishing traditional book content through the web using open standards. our prototype demonstrates the application of html5, structured data with rdfa and schema.org markup, linked data components using json-ld, and an api-driven data model. we examine how this open web model impacts discovery, reading analytics, ebook production, and machine-readability for libraries considering how to unite software development and publishing. by jason a. clark and scott w. h. young introduction and background the library as place and service continues to be shaped by the legacy of the book. the book itself has evolved quickly in recent years, with various technologies vying to become the next dominant book form. identifying an effective and popular next step for the book has been an ongoing challenge for publishers, libraries, and content creators, all of whom have focused significant resources on developing new models for creating, publishing, and accessing book content. an overall view of this landscape is provided by the pew research center, whose survey data shows that e-reading continues to rise [1], e-reading device ownership continues to rise [2], and mobile device and tablet ownership continues to rise [3]. within this context many examples from across book publishing attest to the level of attention and resources now dedicated to developing new models for publishing. representing the efforts of content creators are services such as editorially, a collaborative web-based platform for both creation and publication (since acquired by vox media) [4]. pressbooks is a similar publication service, with the added ability to publish to various formats, including pdf, mobi, epub, and a fully web-based format that pressbooks calls a “webbook” [5]. leanpub is a self-publishing tool that the blurs the line between writer and publisher [6]. other more grassroots efforts are seen in the people’s e-book, a successful kickstarter project that promises to “make e-books better ” [7], and in futurepress, an open source html5 and javascript epub3 reader for the web [8]. project gitenberg is a new and fast-growing collection of over 43,000 free, open, collaborative, and trackable ebooks built and distributed through the git version control system [9]. in addition to e-book tools and services, recent conferences, library training, and specific employment opportunities have explored the publishability and accessibility of book content. the international “if book then” conference brings together writers, publishers, technologists, scholars, journalists, and others to imagine what happens when “we really jump away from the boundaries of the physical (and even digital) book” [10]. the new york public library hosted the open book hack week in december 2013, an event designed “to help us imagine the future of digital books, and advance the open source and open api building blocks needed for the diverse ecosystem of authors, designers, developers, publishers, libraries, booksellers, and readers.” the nypl has positioned itself as a leader in developing ebooks for libraries, led by its nypl labs digital division (a recent nypl labs job advertisement for the position of lead systems architect/engineer included, “if you’re game to help re-imagine the public library ebook experience, then make that vision real (and scaleable), then we want you”) [11]. the digital public library of america (dpla) has announced a partnership with president obama to provide children with greater access to ebooks [12], and the second annual dplafest conference this year included a multi-day workshop to explore the future of ebook publishing [13], this group identified a 10-part “ebook stack” to help frame discussion around possible shared services at a national network scale. the ebook stack includes reader interfaces, discovery systems, and assessment feedback mechanisms. wake forest university libraries is pursuing a related initiative with its mini-mooc series, “zsrx: digital publishing.” with this online learning program wake forest promises to explore the “past, present, and future perfect tenses of e-books, self-publishing, and the digital publishing landscape” [14]. other universities are experimenting with alternative modes of open publishing, such as suny’s open textbook initiatives [15]. questions of information access, ease-of-use, and content quality are of vital importance not only to libraries and publishers, but to the public as well. discussion in public forums continues to emphasize, and in some corners even agonize, over publishing and the state of the book. a recent posting on the chronicle blog network warns of “celebrating booklessness” in libraries, as such a development will damage libraries and librarians over the long term [16]. other analyses identify the evolution of technology and book publishing not as a sign of the end times, but simply as the driver of a new form of reading. a feature published on the website of random house of canada remarks, “people aren’t reading less, just differently. technology has innovated” [17]. the feature offers a quote from artist tan lin: “people forget that a book or codex is a technology.” indeed, discussion within publishing and libraries regarding this topic too often focuses on the loss of our primary artifact, the book, while neglecting new forms of publishing enabled by today’s primary and most powerful information technology, the web. for more than two decades the web has enabled alternative forms of publishing, and now newer web technologies and publishing services have been introduced that promise even further advancement of content publishing. widespread support of digital publishing, however, is not a given. criticism of book publishing in a digital environment has been recently expressed with some force by ucla professor johanna drucker, who approaches digital publishing with marked apprehension in an essay titled, “pixel dust: illusions of innovation in scholarly publishing” [18]. drucker maintains a strong skepticism of digital platforms as a cure for the crisis of publishing. she notes that most phases of the complex publishing cycle—acquisition, editing, reviewing, fact-checking, design, promotion, and distribution—remain in place in the digital environment, and that only the final stage—the form of production—experiences a foundational shift. in the context of the web, how viable is book publishing? how is content acquired and vetted? what is the sustainability model for production? what technologies underpin open web publishing? these questions are fundamental to our own project at the montana state university (msu) library. responding to these questions and to the need for a new book publishing model, we at the msu library are exploring new possibilities for publishing through the web with open standards. we were particularly interested in the possibilities that open standards have for moving outside of drm restrictions for book content and how open markup standards allow for broader access to book content for anyone with a web browser. we even speculated that these open standards would bring an easier archiving model to digital books as preserving and emulating text and simple images could be brought into web archiving routines and allow us to move away from the emulation and software restrictions of .pdf or ebook proprietary formats present in long-term archiving of book content. the initial “book objects” for our msu library prototypes represent two distinct categories of non-fiction and fiction: 1) a cookbook with essays for an msu undergraduate history course, 2) a student literary journal featuring images, prose, and poetry created by msu students. in the pipeline, are two additional academic press formats including: a textbook for a statistics undergraduate course, and an academic journal featuring the mountain science of the region [19]. one of our goals was to test the software across multiple textual formats and see how broadly our book model could be applied. our overall approach to this project is outlined within a wider framework of machine-readable, structured, semantic data (arlitsch et al. 2014). our project utilizes a new method for publishing within a web browser using html5, structured data with rdfa and schema.org markup, linked data components using json-ld, and an api-driven data model [20]. these technologies together unlock the book by transforming its content into a semantic, machine-readable, and extensible platform. drucker in fact reserves her limited praise of digital publishing for this particular aspect, “the most exciting and innovative aspects of digital presentation are the ways in which structured data—texts with humanly-embedded organization—can be searched and analyzed” [21]. we agree with this viewpoint, and have begun this work partly to demonstrate the capabilities enabled by publishing semantically structured book data within a modern web browser. in this article, we will outline the application and benefits of rdfa, schema.org, and linked data models for book production. we will detail the structured data model that can turn book content into api-enabled webpages. we will also analyze the effects of this web publishing practice for machine-understanding, search engine optimization (seo), and user experience (ux). finally, we will discuss the advantages and disadvantages of this model. this web book model can function as a demonstration of a real-world application of semantic structured data. with this project we aim to tie together three threads within the context of the evolving library: open web publishing, software development, and the book. structured data at first thought, the idea that publishing would be tied to forms of markup seems at odds with the present. modern writing software relies on text-based wysiwg interfaces (think of microsoft word or the google docs interface). however, early word processors required special tags to create text that was bold, paragraphed, paginated, and countless other styles. and even more recent textual markup standards such as sgml and eventually docbook worked by requiring strict declarative markup that described a document’s structure and attributes. it is interesting to think how these previous markup structures informed the semantics of the written word. these early notions of structured data were an inspiration as we started to think about the value of structured data for the book. for our web book prototypes we chose to build with html5 and embedded rdfa, coupled with the web-scale controlled vocabulary of schema.org. our first goal was to create markup that was rich and nuanced so that it might have its own semantic value evident to any user who viewed the source code on the page. we began by building the components of the book using epub 3.0 html [22]. this allowed us to break the book into pieces and assign common divisions that you might see for book content. for example, we were able to give our book cover (index.html) some common types such as: <main epub:type="cover" role="main" id="top"> … <section epub:type="titlepage" role="navigation" id="menu"> we followed a similar pattern for our table of contents pages (table-of-contents.html) and from there designated our item level pages (item.html) as chapter divisions. <main data-bookmark="#" id="top" epub:type="bodymatter chapter" role="main"> with a semantic html5 structure in place, we next turned to the idea of annotating page content for machine-readability. rdfa and schema.org are perfectly suited for this task of description. we gave each of the pages several schema.org types including: creativework and article. below are some of the above html tags modified to use schema.org types. <html lang="en" vocab="http://schema.org/" typeof="creativework" xmlns:epub="http://www.idpf.org/2007/ops"> … <main data-bookmark="#" id="top" typeof="article" epub:type="bodymatter chapter" role="main"> with these types in place, we looked to assign multiple properties that could bring a web-scale classification into place for all the pages in the book. we applied common properties like name (title), description, creator, datepublished to help provide minimum viable descriptive metadata, but we knew that we needed more detail to achieve the rich, semantic descriptions that a machine could interpret. this primary consideration led us towards introducing linked data into our annotation practice. using a mechanism known as external enumerations [23], we brought an externally-maintained vocabulary into schema.org markup. external enumerations are a method of introducing html markup for descriptions of web page content using externally-maintained vocabularies and/or datasets. in this case, we wanted to apply topics from dbpedia.org, as it is currently one of the most widely-implemented linked data vocabularies available. we hypothesized that the scale of dbpedia’s vocabulary would result in the best return on discoverability and findability for our page content. for our linked data model, external enumerations embedded as rdfa are implemented as attributes in html tags. the goal is to link out to the target linked data vocabulary or entity and provide a robust, machine-readable definition of the primary topics of the page. an example of this practice is below in the category and context <dt> sections. <dl> <dt>date:</dt> <dd property="datepublished">2015-02-02t01:43:19z</dd> <dt>category:</dt> <dd property="genre" resource="http://dbpedia.org/page/poetry"><a property="url" href="https://en.wikipedia.org/wiki/poetry">poetry</a></dd> <dt>context:</dt> <dd property="additionaltype" resource="http://dbpedia.org/page/baucis_and_philemon"><a property="url" href="https://en.wikipedia.org/wiki/baucis_and_philemon">baucis and philemon</a></dd> <dt>read time:</dt> <dd><time property="timerequired" datetime="pt1m">< 1 minute(s)</time></dd> <dt>page:</dt> <dd epub:type="pagebreak" class="pagination">3</dd> </dl> the <dl> markup above is the visible scope note for the page detailing the basic metadata around any book page item. fields like date, page number, and read time appear, but the more interesting data are under the “category” heading where we are able to map the genre and additionaltype properties to a linked data vocabulary and give the page a topical, machine-readable description. with the rdfa and schema.org embedded and nested correctly in the html, we then tested the page through utilities such as the rdf translator (http://rdf-translator.appspot.com/). diagnostic tools such as this allow us to view our page content as a machine would, formatted as json-ld, n-triples, or xml. we were also interested in an automated representation of our page as machine-readable linked data. to this end, we created an alternate representation of each page as json-ld specifically for bots, intelligent software agents, and developers that might have a preference for the reusable key-value pairs of a .json file. initially, we included json-ld in the <head> of each html page, but we found that this practice was confusing to bots as they had to decide when to use rdfa versus json-ld values to understand the semantics of the page. an example json-ld version of one of our book pages is represented in figure 1. <script type="application/ld+json"> { "@context": "http://schema.org", "@type": "creativework", "@id": "http://arc.lib.montana.edu/book/opsis/", "mainentity": { "@type": "book" "name": "opsis: literary arts journal at montana state university (msu)", "description": "opsis is the literature and arts magazine of montana state university", "image": "http://arc.lib.montana.edu/book/opsis/meta/img/opsis.png", "datepublished": "2015-02-02t01:43:19z", "inlanguage": "en", "genre": "journals (publications)", "learningresourcetype": "student literary journal", "creator": { "@type": "person", "name": "multiple authors" } }, "article": { "@type": "article", "@id": "http://arc.lib.montana.edu/book/opsis/item/3", "name": "bloom", "author": { "@type": "person", "name": "peter hoag" }, "image": "", "articlebody": "%3cp%3ebloom+was+more+than+one+color%2c+or+three.+no%2c+%3cbr+%2f%3e%0d%0ashe+was+a+myth+spun+from+soft+fur+and+green%3cbr+%2f%3e%0d%0apea+plants%2c+simple+things+that+saw+the+golden+grass+%3cbr+%2f%3e%0d%0aand+white+picket+fences+and+let+them+be.+%3cbr+%2f%3e%0d%0aphilemon+and+baucis+tried+to+guide+the+%3cbr+%2f%3e%0d%0achariots+from+unnecessary+heights%2c+%3cbr+%2f%3e%0d%0abut+they+soared+out%2c+further+than+we+could+see.+%3cbr+%2f%3e+%0d%0athey+became+solar+sentinels%2c+watching%2c+%3cbr+%2f%3e%0d%0ajust+out+of+reach%2c+waiting+for+us+to+fall.+%3cbr+%2f%3e%0d%0aunknown+amounts+were+lost+when+the+fire+%3cbr+%2f%3e%0d%0acovered+quiet%2c+and+quiet+covered+the+earth.+%3cbr+%2f%3e%0d%0athe+burning+bloom+peeled+the+harmony+off+%3cbr+%2f%3e+%0d%0alike+skin+and+leaped+and+danced%2c+singing+its+last+%3cbr+%2f%3e%0d%0asong.+then%2c+it+disappeared%2c+its+form+intact.+%3cbr+%2f%3e%0d%0awe+all+stopped+and+watched%2c+our+eyes+toward+the+sky.%3c%2fp%3e%0d%0a%3cp%3e%0d%0athe+world+still+remains%3b+changed%2c+not+the+same.+%3cbr+%2f%3e%0d%0athe+good+is+too+old+...+everything+was+gold.%3c%2fp%3e", "url": "http://arc.lib.montana.edu/book/opsis/item/3", "timerequired": "pt1m", "datepublished": "2015-02-02t01:43:19z", "publisher": { "@type": "organization", "name": "montana state university (msu)" }, "additionaltype": "http://dbpedia.org/resource/poetry", "additionaltype": "http://dbpedia.org/resource/baucis_and_philemon", "potentialaction": { "@type": "readaction", "actionstatus": "pendingactionstatus", "target": { "@type": "entrypoint", "urltemplate": "http://arc.lib.montana.edu/book/opsis/item/3" } } }, "publisher": { "@type": "organization", "@id": "http://dbpedia.org/resource/montana_state_university", "name": "montana state university (msu)", "sameas": "http://dbpedia.org/resource/montana_state_university" }, "url": "http://arc.lib.montana.edu/book/opsis/" } </script> figure 1: page of book using structured linked data (json-ld), https://arc.lib.montana.edu/book/opsis/item.json?id=3 as each page is defined with incredible granularity, we were even able to move outside of basic descriptive metadata and introduce action metadata. you can see this activity in the potentialaction assignment where a readaction and target are defined to show a machine what it could do with a page. machine understanding was our goal with this activity, and as we started to apply these forms of structured and linked data to book content, we could monitor how well one of our target bots, googlebot, was able to understand the linked metadata in our book content including primary keywords in the page text body as well as the assigned genres and topics. (figure 2). figure 2: basic machine understanding of book content with this information from google webmaster tools, we can start to see how our pages are understood and we can adjust our metadata practices accordingly. terms like “lost”, “garden” and “adonis” are interpreted from our linked data assignments and our book as a discoverable entity is starting to take shape. our next steps here will be to continue to analyze these results and bring in other machine learning tools to see how we might optimize the linked data for other purposes like automated classification. book-as-api another goal with this project involved connecting the practice of modern publishing with the process of software development. the markup practices and the work of linked data described above require a new view of library work from the perspective of web publishing. one of the key benefits of this web book project has been the recognition that digital library work is complementary to the act of publishing. in order for the networked web book to scale into broader production settings, we needed to consider how we might “atomize” the book content into a data store. figure 3 shows how we stored the pieces of book data into data fields. create table `bodymatter` ( `id` int(11) not null auto_increment, `name` varchar(255) default null, `creator` varchar(255) default null, `url` varchar(255) default null, `description` text, `about` text, `image` varchar(140) default null, `articlebody` text comment 'http://schema.org/article', `articlesection` text comment 'http://schema.org/article', `publicationtype` varchar(255) not null comment 'https://schema.org/publicationtype', `additionaltype` varchar(255) default null, `additionaltype2` varchar(255) default null, `isbn` varchar(40) default null, `genre` varchar(140) default null, `keywords` varchar(255) default null, `publisher` varchar(255) default null, `datecreated` varchar(255) default null, `datemodified` timestamp null default current_timestamp on update current_timestamp, `datepublished` varchar(255) default null, `version` varchar(140) default null, `learningresourcetype` varchar(140) default null, `inlanguage` varchar(5) default null, primary key (`id`), fulltext key `search` (`name`,`creator`,`description`,`genre`,`keywords`,`datecreated`) ) engine=myisam default charset=utf8; figure 3: “atomizing” the book to turn it into an api in this case, we utilized a simple mysql table, but many other types of data stores could function in a similar way in different local environments. schema.org was our guide here as well, with database field names mapping directly to schema.org properties for creativework and article. the real value in atomizing the book is in its newfound reusability, as the book in this form can be transformed into a platform and an api. the articlebody and articlesection fields above allow us to actually store the chapter or page data as text. with this fielded data in place, we can do things like count words in the articlebody field and build estimated read times for the content. more importantly, we can expose all of the fields in a rest api and make the book remixable. in addition to the json-ld format, each item/chapter page has an xml and json representation. this structured data also provides the opportunity to become clients of our own api, and in this way we have built a book search functionality using our rest api endpoint (figure 4). figure 4: search built using rest api from book data model (api.html), https://arc.lib.montana.edu/book/opsis/api?v=1&type=search&q=peter&format=json when the book becomes a platform and adapts to the open nature of the web, the remix becomes possible. new sharing and discovery opportunities become possible, and we’ll examine some of the network effects of this open model in our next section. machine-readability for seo and ux as the traditional book moves into a networked environment, new applications for seo and ux become available. seo is a method for enhancing the discovery of digital content on the web (arlitsch 2012; killoran 2013). in the context of online publishing, the pages and words of the book become digital content on the web, and can therefore benefit from optimization techniques that enhance discovery on major search engines such as google. onaifo (2013) details four seo techniques for improving library content visibility in search engine results: website design, content, external links, and social media optimization (smo). seo website design this wide-ranging category considers the size and structure of a website. for our own “web book,” we have designed individual items pages to be lightweight, and have maintained a relatively small overall size for each book. in total, our prototype 1 includes 42 pages with a median page weight of 140 kilobytes, and our prototype 2 includes 35 pages with a median page weight of 26 kilobytes. we have also employed semantic structured data in constructing the web book, as described above. notably this includes meta tags in the <head>, such as <link rel="canonical">, that enable machine-readability of page content. deep-linking book content with semantic urls and semantic markup also enhances seo, as individual pages can be indexed by search engines and subsequently discovered by users with rich supplementary data, such as cook time for a recipe page (figure 5). finally, information architecture is an important aspect of seo-effective website design. a clearly hierarchical website structure is not only human-friendly, but machine-friendly, as search engines can easily crawl and index webpages. as an example, our web book features the following hierarchical structure: title page, a table of contents, and a series of numerically-ordered item pages. this site structure is then expressed through an xml sitemap, and made available to web indexers. figure 5: search engine results page content the primary objective for web search engines is to match relevant content to user queries (onaifo 104). a website can design its content so that its pages are visible to users through search engines, and web analytics software can provide insight into which channels and keywords are used to discover web content (figure 6, figure 7). figure 6: traffic sources, available through google analytics figure 7: user-generated search queries, available through google analytics with an open web book model, content can be optimized, indexed, and discovered through the web. external links the number and quality of external sites linking to a website or webpage is a ranking and relevancy method used by web search engines, and there has been shown to be a statistically significant relationship between external links and user traffic for library web content (onaifo, 96). internal links also generate signals used by search engines, which together with external links can increase the discoverability of book content. social media optimization activity on social networking services (sns) is a factor in search engine result rankings (killoran, 62-63). where seo seeks to enhance the discoverability of content on the web, social media optimization (smo) is a related suite of web content optimization techniques that seeks to enhance the shareability of content through social networks. sharing online and through snss creates relevancy signals that affect search engines rankings. we sought to enhance the discoverability of our web book by enhancing its shareability. smo is defined by five principles (killoran, 63), one of which is to make sharing easy for users by including relevant tags on webpages so that content can be richly presented on snss. to realize this aspect of smo, we implemented machine-readable metatags specific to facebook and twitter (http://ogp.me/ & https://dev.twitter.com/cards/). <!-social media tags --> <meta name="twitter:title" content="bloom peter hoag"> <meta name="twitter:description" content="opsis: literary arts journal at montana state university (msu)"> <meta name="twitter:image:src" content="https://arc.lib.montana.edu/book/opsis/meta/img/opsis.png"> <meta name="twitter:url" content="https://arc.lib.montana.edu/book/opsis/item/3"> <meta name="twitter:card" content="summary_large_image"> <meta name="twitter:site" content="@msulibrary"> <meta name="twitter:creator" content="@msulibrary"> <meta property="og:title" content="bloom peter hoag"> <meta property="og:description" content="opsis: literary arts journal at montana state university (msu)"> <meta property="og:image" content="https://arc.lib.montana.edu/book/opsis/meta/img/opsis.png"> <meta property="og:url" content="https://arc.lib.montana.edu/book/opsis/item/3"> <meta property="og:type" content="website" /> <meta property="og:site_name" content="montana state university (msu) library"/> <!-end social media tags --> these tags allow facebook and twitter bots to fetch content from our pages and display data attractively for users on their platforms (figure 8), thereby increasing the likelihood of further sharing. figure 8: smo-enhanced book content on twitter web analytics can again help provide insights into the network effect of sns sharing (figure 9). figure 9: social network traffic, available through google analytics as the book enters a networked environment, its content can be configured for discovery and sharing. once optimized for search and social, the book is poised for wide-spread discovery. ux the flexibility of the web platform allows the book reading experience to be highly configurable. user feedback mechanisms available through web analytics and other ux research methods allow online reading interfaces to be designed from a user-centered perspective. google analytics, for instance, offers a “user flow” view that shows movement through the pages of a website (fig. 10). in the same way that a website can be studied at the page level, so too can a web book be studied at the page level. with a view such as the user flow, we can start to understand how users move through the pages of a book and which pages attract more or less attention from users. in the example shown in fig. 10, we can evaluate the effectiveness of our table of contents page with respect to our goals and our users’ goals. in this case, we expect the table of contents to be successful in moving users through to content within the book. during the sample period shown in fig. 10, of 287 sessions on the table of contents page, 96.2% moved onward through the book, and 3.8% dropped off from the web book from the table of contents page. this provides a strong clue that that the table of contents is successful in directing users to content within the book. had these percentages been inverted, we would direct more design and development effort towards the table of contents page, which we could then reconfigure based on user data such as this. figure 10: google analytics user flow view further page-level analysis is offered by web analytics software such as crazy egg [24] that creates visual maps of user click behavior (figure 11). figure 11: visual map of user click behavior, available through crazy egg in this example, we can see that the gravity of clicks on the item page of our prototype 1 are collected around the previous button (lower left), the next button (lower right), and our table of contents link (top left). this is the click behavior that we expect to see, and we can further refine our page display by increasing the size of these targets. this data view also shows us that errant clicks dot the page, perhaps indicating that users expect certain words or sentences to be linked to external entities. with this user data, we have further clues that can inform subsequent ethnographic research around the online networked reading experience. discussion we have grounded our research in the concept of the evolving book—what it means for the book as a medium to be hyperlinked, marked up, atomized, and analyzed as a networked participant in the web of data. with this approach comes both advantages and disadvantages. we see advantages to the approach in breaking free from proprietary formats that bind the book to particular platforms like amazon’s kindle or overdrive. more specifically, we see our application of open standards, like html, into book production opening up broader access for readers across platforms and building on the library goal of universal access. from an archival perspective, we see the rich markup approach to book development as a boon to preservation and long-term curation of book content as our book objects can be brought into web archiving and preservation routines free from the needs of software emulation or specialized digital archival practices. we also posit that our approach allows publishers to learn how digital library roles such as discovery and api design, data storage and archiving, metadata (structured and linked), ux and analytics, and sharing and reuse are complementary to publishing processes and offer new and valuable roles to their current publishing models. building the book within a web browser also shifts the relationship between book consumer and book publisher, and the models for payment and copyright within this evolving dynamic are open for consideration. we also see this transformation as an advantage for both book producer and consumer. the consumer gains an enhanced flexibility in reading and reusing book content, while the publisher gains an opportunity to shift compensation models toward the initial work of book creation and away from existing cost models surrounding redistribution and editions. in following this web book model, we have noted the development overhead that this type of markup and treatment requires. in many ways, we recognize this overhead as the primary disadvantage. as our research continues, we are focusing on how to scale production, ease the book creation onboarding process, and improve the metadata entry workflow for this time-intensive activity. we are also investigating reading interface design and the possible connections between online reading and comprehension. and finally, we are interested in how we might build more participatory models like open annotations into the book model and incorporate the assignment of behavioral and action metadata to take advantage of the network. our “web book” mode of publishing draws together new and existing web technologies to create a book that is accessible, discoverable, shareable, and analyzable: accessible: we have built our framework using the open web standards of html5 css3, and rdfa, enabling interoperability so that content can be accessed across devices. discoverable: we have leveraged the tools and practices of search engine optimization, linked open data, and the semantic web so that book content can be described using structured data markup and found with greater relevancy in leading search engines such as google. shareable: we have fully atomized book content so that granular portions of the book may be linked, shared, and reused through a networked environment. analyzable: we have enabled advanced analytics for our book content so that we may better understand user behavior and expectations for a reading experience on the web. this model represents just one prototype in a larger conversation, but it does start to outline some of the emerging options for the book, one of the primary data formats of library and information science. appendix live project: http://arc.lib.montana.edu/book github repository: https://github.com/msulibrary/bib-template-fiction as a point of reference, we presented our idea and model at code4lib 2015. you can see the presentation here: https://www.youtube.com/watch?v=gcfpqgxcpte&t=24m1s notes [1] http://pewinternet.org/reports/2014/e-reading-update.aspx [2] http://pewinternet.org/reports/2013/tablets-and-ereaders.aspx [3] http://pewinternet.org/commentary/2012/february/pew-internet-mobile.aspx [4] http://stet.editorially.com/ [5] http://pressbooks.com/ [6] https://leanpub.com/ [7] http://www.kickstarter.com/projects/1371597318/the-peoples-e-book [8] https://github.com/futurepress/epub.js [9] https://gitenberg.github.io/ [10] http://www.ifbookthen.com/ [11] http://www.creativeapplications.net/jobs-archive/lead-systems-architectengineer-at-the-new-york-public-library/ [12] http://dp.la/info/2015/04/30/the-digital-public-library-of-america-partners-with-president-obama-to-provide-children-with-greater-access-to-ebooks/ [13] http://dp.la/info/2015/04/29/summary-report-from-dplafest-ebook-workshop/ [14] http://events.wfu.edu/event/zsrx_digital_publishing_the_past_present_and_future_perfect_tense_of_ebooks [15] http://textbooks.opensuny.org/ [16] http://chronicle.com/blognetwork/theubiquitouslibrarian/2014/01/06/the-harm-of-booklessness/ [17] http://www.randomhouse.ca/hazlitt/feature/internet-killed-books-save-reading [18] http://lareviewofbooks.org/essay/pixel-dust-illusions-innovation-scholarly-publishing [19] funding and support for the expansion of the prototypes is provided by the institute of museum and library services (imls) sparks! ignition grant – http://www.imls.gov/applicants/detail.aspx?grantid=19. [20] you can find our current “web books” at http://arc.lib.montana.edu/book/ and our code is available at https://github.com/msulibrary/bib-template-fiction and https://github.com/jasonclark/bib-template. [21] ibid. [22] full documentation for epub 3.0 is available at http://www.idpf.org/epub/30/spec/epub30-overview.html. [23] http://www.w3.org/wiki/webschemas/externalenumerations [24] http://www.crazyegg.com/ references arlitsch k, obrien p. 2012. invisible institutional repositories. library hi tech 30(1): 60–81. arlitsch, k, obrien p, clark, ja, young, swh, & rossmann, d. 2014. demonstrating library value at network scale: leveraging the semantic web with new knowledge work. journal of library administration 54(5): 413–425. killoran, j b. 2013. how to use search engine optimization techniques to increase website visibility. ieee transactions on professional communication 56(1): 50–66. onaifo, d. 2013. increasing libraries’ content findability on the web with search engine optimization. library hi tech 31(1): 87–108. about the author(s) jason a. clark (jaclark@montana.edu) is an associate professor and the head of library informatics & computing at montana state university libraries where he builds digital library applications and sets digital content strategy. his current research interests include: linked data, search engine optimization, web services & apis, interface design, and application development. read more of his occasional thoughts or check out his code samples at www.jasonclark.info. scott w.h. young (swyoung@montana.edu) is an assistant professor and digital initiatives librarian at montana state university, where he specializes in user experience research, web development, and social media community building. read more on his website, http://scottwhyoung.com/. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – centralized accessioning support for born digital archives mission editorial committee process and structure code4lib issue 40, 2018-05-04 centralized accessioning support for born digital archives archives often receive obsolete digital storage media alongside paper acquisitions: cds and dvds mixed in with folders of correspondence, zip disks, and floppy disks set aside by the donor with the intention to review the content later. archives must not only have the expertise to work with digital media, but also the hardware and software to capture the content without the risk of altering the files merely by viewing them. this article will describe how yale university libraries and museums addressed accessioning of born-digital archival content on physical media through a centralized digital accessioning support service. centralizing the hardware and expertise required for working with physical media made it possible to accession media more quickly and return the files to the originating archives for arrangement and description. by alice sara prael archives often receive obsolete digital storage media alongside paper acquisitions: cds and dvds mixed in with folders of correspondence, zip disks, and floppy disks set aside by the donor with the intention to review the content later. however, when “later” arrives, the obsolete drive necessary to view the content is more difficult to find and so these media find their way to the archive, often with the note “i’m not sure what’s on this.” the longer the files remain on the media, the more difficult and potentially expensive it is to capture the content. archives must not only have the expertise to work with digital media, but also the hardware and software to capture the content without the risk of altering the files merely by viewing them. although there are vendors specializing in capturing content from obsolete media, these services come at a cost that increases for every additional disk. this article will describe how yale university libraries and museums addressed accessioning of born-digital archival content on physical media through a centralized digital accessioning support service. centralizing the hardware and expertise required for working with physical media made it possible to accession media more quickly and return the files to the originating archives for arrangement and description. special collections at yale there are many special collection units at yale university, each with their own collecting focus. eight special collection units agreed to participate in the pilot digital accessioning service – irving s. gilmore music library, robert b. haas family arts library, divinity special collections, beinecke rare book and manuscript unit, yale center for british art institutional archives, manuscripts and archives, medical historical library, and peabody museum archives. prior to the creation of the service, each unit worked with physical media independently and with varying levels of success. some units were able to programmatically capture content from disks, some worked with born-digital content on physical media only when access was requested by a researcher, and some units did not have the capacity to work with digital content and avoided collecting physical media. in 2014, yale university library adopted archivesspace as the system of record for archival descriptions. in 2015, yul selected the preservica digital preservation system (dps) for managing long-term preservation of born-digital and digitized content. however, when the centralized service to support digital accessioning launched in 2016, special collection units were at varying levels of implementation of both systems. while some units created and edited finding aids entirely in archivesspace, some were still working to import existing finding aids from legacy systems. at the time of this article’s publication, some units are ingesting born-digital content into the digital preservation system while others are not participating in the dps, relying on other preservation solutions instead. these differing levels of implementation and use of both systems affected the readiness of each unit to participate in the service. yale’s implementation of archivesspace is integrated with the dps, which provides the opportunity for synced description of digital files and the potential for automated access. however, working with integrated systems is also challenging since mistakes made in one system may be replicated in the other. identifying the problem initial estimates provided by special collection units identified a backlog of at least 4,000 individual pieces of media located in special collections at yale. since a significant portion of that content has not been accessed since before its acquisition, the exact volume of this at-risk data was unknown. the estimates provided were based on known acquisitions and have since proven inaccurate. at the time of this article’s publication, the service has received nearly 6,000 pieces of media. in january 2015, the born digital archives working group (bdawg) was formed to provide leadership in the area of born digital archives [1]. the group’s vision is to provide the same level of stewardship for born-digital archival holdings as is devoted to our physical collections. one of its first priorities was to address the growing backlog of born digital archives and determine how yale libraries and museums can pool resources and expertise to find a path forward. in 2016, bdawg recommended the creation of a centralized service and the hiring of a two-year digital accessioning archivist as the best way forward for capturing born-digital content on physical media. in 2015, an off-site facility was renovated and repurposed to address the need for additional space (both for archival work and other departmental needs within yale). this facility included the new di bonaventura family digital archaeology and preservation lab. the lab space is shared by the yul digital preservation unit and the beinecke rare book and manuscript unit. the lab was equipped with two digital forensic machines, custom built by the digital preservation manager, euan cochrane, and other machines and equipment moved from a previous shared lab space on central campus. in 2016, a new position, digital accessioning archivist for yale special collections, was created to develop and manage this centralized service. the service is staffed primarily by myself as digital accessioning archivist with guidance from gabby redwine, beinecke’s digital archivist, and in collaboration with bdawg. the service also employs student assistants to help with data entry and capturing content from media. the service was launched in july 2016 as a two-year pilot project to eliminate the backlog of physical media in archival collections. figure 1.the lab shared by the yul digital preservation unit and the beinecke rare book and manuscript unit building a service the first step in creating the digital accessioning service was to meet with the stakeholders to identify their needs. we identified the stakeholders as members of bdawg and each of the participating special collection units. in order to simplify communication, each unit selected a staff member to serve as the digital liaison. this position is responsible for determining the order of media sent for accessioning, communicating updates between the service and the special collection unit, and providing the service staff with access to the necessary systems (shared storage location, archivesspace instance, etc.). there is some overlap in stakeholder positions as three out of six members of bdawg also served as the digital liaison for their unit. to guide conversations with stakeholders, i created a rough sketch of how a centralized service for born digital accessioning support might run. this included a strategy for shared documentation and ongoing communication, description of the equipment already available for use in the lab, and a list of tasks and materials that were already identified as out-of-scope, such as data tapes and audio cassettes. during interviews stakeholders identified the following needs: capture content from physical media (removing the need for specialized hardware/expertise within the unit) scan content for personally identifying information (pii) create manifest of all files captured create item level descriptions for physical media update existing item level descriptions for physical media prepare content for long term preservation accommodate privacy concerns during processing reformat non-born-digital audio-visual material although we had anticipated many of these needs, and in fact many of them were the reason for the service’s creation, there were also a few surprises. although we expected the need to scan content for pii, we did not anticipate privacy concerns regarding staff working with sensitive material in a shared space. although the lab is in a secure facility, there is a large window through which material might be viewed by staff outside the service. this need for additional privacy could be met with a policy regarding use of the window blind while working with such material. some needs could not be addressed, for instance the need to create digital derivatives of non-born-digital a/v material was already defined as outside the scope of the service. another surprise was the need to update existing item level descriptions in archivesspace. some special collection units have already created item level descriptions for much of their physical media and these descriptions are often more thorough and descriptive than the minimal description required by the service. for instance, the service uses the label written on the physical disk as the item title, but the label from the record creator may not be as relevant as the title created by the processing archivist. the need for two description options was considered during the workflow creation process. to contain the scope of the service we identified a list of media types that will be accepted for accessioning. accepted media types were limited to ensure that all incoming material could be accessioned in a timely manner with the equipment available in the lab. since the extent field is a controlled value in archivesspace, the system required a controlled list of media types for the item level description. the service accepts the following media types: 3.5 floppy disks, 5.25 floppy disks, jaz disks, zip disks, optical media, hard drives, flash drives, and laptops. desktop computers are not accepted because of the limited space in the lab and potential for damage during transportation, but the service offers consultations for units that need help to remove the hard drive. creating workflows after discussing the needs of our stakeholders, we began defining the workflow. for the sake of clarity, this is presented as a linear process, but in reality some stakeholder needs did not emerge until the workflow was fully implemented. the workflow presented here required tweaks to address concerns as they surfaced. this section offers a high-level description of the work done within the service, the reason for completing each step, and the inherent organizational concerns. preparing the submission prior to submitting material, the digital accessioning service, the digital liaison must provide the digital accessioning archivist with access to the special collection unit’s archivesspace repository. access is required to create/update item level descriptions for the physical media. the liaison must also provide access to a network storage share with sufficient space for processing media. the service saves the born-digital content on this network share prior to ingest to the dps. the service recommends 3-lock security network storage particularly for units collecting personal papers or institutional records with additional security concerns[2]. for each submission to the service, the digital liaison must complete a metadata spreadsheet (whose columns are mapped to as fields) describing each disk in the shipment. this spreadsheet is submitted to the service via an online form, which also requires the liaison’s contact information and method of transportation for the shipment. each row in the spreadsheet later becomes an item level description in archivesspace. this spreadsheet fills a second function of providing confirmation that the service staff is working with the correct media. it is worth noting that not all special collection units use archivesspace as their system of record. in these cases, there is no archivesspace repository to provide the service with access to and the url column in the metadata spreadsheet may be left blank. these spreadsheets will not be imported to archivesspace but instead saved to the network share. there must be a resource record for the collection in archivesspace in order for service staff to create new item level descriptions. the spreadsheet requires the url for the parent record, i.e. the archival object one level up in description hierarchy. if the item level description already exists and is only being updated by the service then the spreadsheet should contain the url for the item description requiring an update. each piece of media must be assigned a unique identifier or media number, called the component unique identifier (cuid) in archivesspace, and the identifier must be written on the physical media. the identifier facilitates identification and management of information about the item. the service provides guidance on how to write the identifier on media without damaging the carrier or obscuring the original label, but the method is ultimately up to the digital liaison. the identifier is often a derivative of the accession number, but some units do not rely on accession numbers and instead use call numbers. the only requirement for the identifier is that it must be unique to the special collection unit. this identifier is included in the spreadsheet and used as the file name for all service output associated with that disk as well as the entered as the component unique identifier (cuid) in items description in archivesspace[3]. in the service once the metadata spreadsheet and survey form have been submitted and the shipment of media has been received in the lab, service staff confirms that the correct boxes have arrived and the content was not damaged in transit. if the materials arrive in the containers listed in the spreadsheet service staff confirms that the container numbers are correct. many times the media has been removed from many hybrid collections and placed together in a temporary container which is not numbered. in these cases, the service only confirms that the media has not been damaged. the service does not confirm each piece of media upon arrival as they are verified against the metadata spreadsheet during imaging and capture. each special collection unit has been assigned a shelf in the lab where their media is stored while in queue. shipments of media are processed in the order that they arrive; however, special collection units can request expedited accessioning. each disk is photographed by the service – these photographs serve as an access surrogate of the physical disk. physical media is often restricted from use in the reading room due to the fragility and technical complications of access. if a researcher is interested in viewing the handwritten label on a disk, this photograph can fill that need. photographing media also aids in workflow management. the photograph is helpful when the service needs to communicate with the special collection unit about a discrepancy between the physical media and the spreadsheet description. when media is accessed on a machine it must be connected via a writeblocker, which ensures that staff members are not able to alter the content or metadata on the disk. without a writeblocker, it is possible to change the metadata or reformat an entire disk volume just by connecting to the media. once the media is safely connected to a machine it is scanned for viruses and malware on a non-networked machine. if viruses are discovered the service will contact the originating unit to determine the appropriate path forward. all media except for floppy disks are virus scanned prior to capture. it was determined that viruses or malware found on floppy disks are low risk to modern technology, but the fragility of this media type means that there may be only one chance to connect to the media. all content captured from media is virus scanned upon ingest to the dps, which serves as the final security check for viruses and malware. once the media is deemed safe for use, content is captured in the form of a disk image. a disk image is an exact copy of the disk volume containing the folder structure, orphaned and deleted (but not overwritten) files, and all associated metadata. all media received by the service is imaged unless a different capture method has been requested by the originating unit. a special collection unit may request that the files be transferred without attempting to create a disk image for reasons including donor concerns with the capture of deleted files, or the disk capacity is large but contains a small amount of desired data. there is one exception to the disk image first rule, compact disc-digital audio (cd-da). this format has a higher error rate than a standard cd-rom and a disk image would have an equally high error rate. for this reason, if a cd is identified as a cd-da the service will not attempt to disk image but instead use a direct transfer method that reads the disk multiple times to capture the best possible copy. if the first attempt at imaging fails, the digital accessioning archivist will try to determine the reason for the failure and attempt a second time. this often includes returning to the photograph of the media to determine if there are any hints as to the formatting of the original disk. if disk imaging fails a second time, that imaging process is logged as a failure in the metadata spreadsheet. if imaging fails, the service then attempts to transfer the folder structure and files without a disk image. the transfer capture process is also attempted twice before it is logged as a failure. the service is limited to two attempts for imaging and transfer capture due to the significant backlog of material and the time constraints of a two-year project. any error messages or reason for the failure is logged with the item level description to aid staff that may return to the media in the future for another attempt at capturing the content. the service conducts quality assurance on content captured by mounting the disk image (if created) and viewing the folder structure. if the disk image can be mounted and a folder structure exists, that disk image is considered a success. if the files were captured via transfer, the quality assurance process is to ensure that the captured folders exist and contain data. the service is not able to conduct file level quality assurance due to the limited time and scope of the project. if content was successfully captured from the media, the disk image or transferred files are then scanned for personally identifiable information (pii). three types of pii are identified by the service: social security numbers, credit card numbers and the phrase “bank account”. unfortunately, bank account numbers are too variable to search for without returning every integer found on a disk. however, many record creators label their bank account numbers with the phrase “bank account”. when researching how other tools scan for private financial data we discovered that the common method is to search for financial terms such as ‘bank account’ or ‘account holder’. due to the time constraints of the project the service is not able to filter out false hits from this scan. instead the output is saved as a csv file for later review by the originating unit. once the above steps have been completed the physical media is returned to the originating unit, while the service workflow proceeds. the spreadsheet must be imported by service staff to archivesspace where each row in the spreadsheet becomes a new item level description, or updates an existing description. the final step in the workflow is to package the captured content along with any associated metadata files. at this point the workflow forks to provide special collection units with two options – save the package to a ‘hot folder’ for automatic ingest to the dps or save the package to network storage to review before ingest. in the first option the ‘hot folder’ must be created by the digital preservation team and the digital liaison must provide the service staff with access. this ‘hot folder’ is configured by the digital preservation team to automatically pull content into an ingest workflow for the dps. once the package is saved to this folder the service workflow ends. any issues that arise during ingest will be directed to the originating unit. the second location where packages may be saved is in the network storage share for that unit. this is the same network storage that is used for processing, so the service will create a “completed” folder and will notify the unit via email when packaging is complete for a shipment of material. this is a useful option for units that plan to deaccession content once they are able to review the files. many of the disks accessioned by the service have not been connected to a machine since before their arrival at the archive. in this instance staff may review the files and make any deaccessioning decisions, at which point they are responsible for ingesting the material to the dps. researching and developing tools once stakeholder needs and the scope of the service was defined i began researching the tools necessary to complete the work. this section will address the more technical aspects of the workflow as well as the research and development required to implement the service. photography physical media is photographed using an overhead camera on a copy stand. staff from the digital services unit (dsu) at the beinecke library were very helpful in setting up and providing access to the photography equipment. service staff had no experience with photography and relied heavily on dsu staff to configure and test the equipment. optical media is photographed using a flatbed scanner because the lighting equipment used on the copy stand creates a glare, often making the creator’s label illegible in the photographs. after consulting with external groups, we determined that a flatbed scanner is the best method to photograph optical media. the scanner still causes some glare when the top of the disk is reflective, but the glare is diminished significantly compared to the copy stand. writeblocking the lab is equipped with an ultrakit from digital intelligence, which contains a writeblocker for usb, scsi, sata and ide connectors along with various cables and adapters. this writeblocker kit is sufficient for the vast majority of material, however the version of the kit originally purchased for the lab did not include a firewire writeblocker. in 2018 the service purchased an ultrakit v4.1 to fill this gap and provide backup equipment for the lab. the new version of the ultrakit includes a firewire writeblocker in addition to the writeblockers for usb, scsi, sata and ide connectors. purchasing backup writeblockers provides more flexibility for use of equipment in the lab. since the lab is a shared space utilized by the digital accessioning service, digital preservation team, and the beinecke rare book and manuscript library, it’s important that the service has dedicated access to the necessary equipment. testing determined that hardware writeblockers are not reliable for protecting usb-connected zip disks. for that reason the service relies on software writeblockers in the bitcurator environment when working with zip disks. virus scanning virus scanning must occur on a non-networked quarantine machine. a machine from the previous iteration of the digital forensics lab was repurposed to serve as our quarantine machine. this machine has a cd-rom drive and tableau t3458is forensic bridge installed in the machine. this forensic bridge is internally installed and serves as a writeblocker with usb, sata, scsi and ide connectors. the quarantine machine is not networked, but must be re-connected to the network monthly to update the virus scanner. avast antivirus software is installed on the machine and provides the following options: auto: attempts to repair the file. if unsuccessful, moves the file to the virus chest or deletes the file if neither action is successful. delete: permanently removes the file from your pc. repair: removes malicious code if the file is only partially infected. this action is not possible if the entire code is malware. chest: sends the file to the virus chest where the file cannot harm your system nothing: makes no changes to the contents or location of the file (not recommended). if a virus is discovered these options are communicated to the originating unit and they must decide the appropriate path forward. capturing content the digital forensic machines are equipped with both 3.5 and 5.25 inch floppy disk drives installed internally and connected via a kryoflux board. kryoflux is both the software tool and hardware controller board that can be used to connect to and create a forensic disk image of a floppy disk. the kryoflux is a common tool for archives because it is fairly inexpensive, works with a variety of floppy disk formats and includes write blocking capabilities. the kryoflux is also able to capture a low-level copy of the magnetic flux on a disk, saved in a proprietary stream file format. although stream files cannot be mounted to view files like a disk image, they can be re-interpreted into a disk image using the kryoflux software if the archivist is able to identify the correct disk image format. since zip disks require the use of software writeblockers available in the bitcurator environment, zip disks are imaged using the guymager tool which is available in bitcurator. both digital forensic machines have two partitions, one in windows and one in linux bitcurator with an additional hard drive that is accessible from either partition. this allows service staff to access content from either partition without requiring network access. for all other media types, the service uses the free software tool, forensic toolkit (ftk) imager. this tool is lightweight and easy to use. it also allows service staff to automatically create a manifest of files during the image creation process. optical media creates an iso image, but all other media types create a raw disk image. although we researched the disk image creation tool guymager for creating disk images, we ultimately decided to move forward with ftk imager for two reasons. first is the ability to automatically create file manifests while imaging a disk. second, because a large portion of optical media imaging is done by student workers while logging success/failure in an excel spreadsheet which is more easily done in a window’s environment. for transferring content from physical media service staff researched the preservica submission information package (sip) creator tool. this tool can be pointed at a directory to package the folder structure and files and create preservation metadata. the issue with using the sip creator to capture original files is that the sip creator is used later in the workflow to package all service output together for ingest to the dps. this would ultimately result in double packagingthe first containing the transferred files, and the second containing the first package along with the photograph, pii scan and other metadata files created by the service. this double packaging is not ideal and may cause issues with ingest to the dps. the other drawback of this tool is the inability to de-select files or folders or skip a file if it cannot be copied. if a file is corrupted or unable to copy the sip creator will fail to create a package. for this reason, i decided to move forward with a different tool called fastcopy[4], which was already in use by the digital preservation team for transferring files. the tool is able to verify the files before and after transfer using md5 checksum validation. if a single file is corrupted or unable to copy the tool will skip that file and provide an error message with the file path. when the tool is unable to copy a file that file path and associated error message are copied into a log file and the transfer capture is logged as a partial pass. the final tool used to capture content is exact audio copy (eac), which is created specifically to work with cd-das which have a higher error rate which interferes with disk imaging. when run in secure mode[5], eac reads each sector a minimum of two times, and up to 16 times if an error is detected in the first two reads. by reading each sector multiple times eac can ensure that the service acquires the best possible copy of the audio files. eac is configured to create a single wav file from the audio disc, which is the best practice for preservation and follows our practices for preservation copies of digitized audio. the audio file may be split into track files for access later using track information in the metadata[6]. scanning for personally identifying information one machine in the lab is dedicated to running the full version of forensic toolkit (ftk). this software is much more robust than the free ftk imager used for disk imaging. service staff creates a case in ftk and adds disk images as evidence items. for floppy disks and optical media the disk images are small enough that ftk can load as many as 20 disk images or folders of transferred files in a single case, which streamlines the scanning process. only disk images or transferred files are scanned for pii; stream files captured from floppy disks cannot be scanned. for hard drives and larger storage media the service loads one disk at a time. once the captured content is added to a case the service uses the live search function to identify files containing social security numbers, credit card numbers and the phrase ‘bank account’. information acquired by the scan is exported as a csv file. if multiple disk images were searched in a batch the exported data must be separated to reflect the pii on each disk. updating description and documenting actions when the metadata spreadsheet is received by the service additional columns are added to describe the work happening within the service. these columns become event records in archivesspace and are associated with the media’s description. there are four event record types in use by the service: pii scan, image capture, transfer capture, and stream capture (which is only applicable to floppy disks). each event record includes a date and can be recorded as a pass, fail or partial pass. service staff may also add an outcome note to describe any issues that arise during that event, including error messages that arise during the capture process. once the metadata provided by the originating unit has been confirmed and the event records are complete the metadata spreadsheet is converted from excel to a csv file before it is imported to archivesspace using a tool developed locally by the beinecke metadata coordinator and archivist from manuscripts and archives. this tool was originally developed as a python script intended for use only by the service, however it quickly became clear that a tool like this would be useful to staff outside the service as well. the script interacts with the archivesspace api to create an item level description and the associated event records from the data in the spreadsheet. the script also produces an outfile spreadsheet containing all the data from the original metadata spreadsheet with additional urls to the newly created archival object and event records. the first iteration of the script was limited to creating new descriptions and was unable to update existing descriptions. since many disks already had item level descriptions in archivesspace, creating a new description would lead to additional metadata clean-up work for the special collection unit. in the second year of the service the script was further developed to improve error reporting, include a graphical user interface (gui), and add the option to update existing records. this development allowed units with existing item descriptions to send material to the service without the need to condense two descriptions into one for each disk accessioned. providing a gui also makes the tool easier to implement for yale units outside the service. preparing for ingest into digital preservation system the final step of the workflow is to prepare service output by packaging it for ingest to the dps. these packages are created using the preservica sip creator tool. decisions made during the packaging step are vital because they affect access restrictions to the digital object in the dps and the appearance of digital objects in archivesspace. this tool has an easy-to-use gui, however it would be prohibitively time-consuming to package the service output for a single disk at a time. in order to save time, the digital preservation manager developed a batch packaging tool built on top of the sip creator code. the batch packaging tool can process a directory of captured content from many disks and use the metadata spreadsheet to make the connection to archivesspace. there are three components that must be included during the packaging process. the first is the collection code, which provides the link between the digital object in the dps and the item description in archivesspace. the package must include the url for item description in archivesspace as the collection code in order for the digital object in the dps to synchronize with the descriptions in archivesspace. the batch packaging tool accesses this url from the outfile spreadsheet created by the archivesspace import tool. the second component is choosing the correct deliverable unit option in the packager. the tool provides three deliverable unit options: single deliverable unit: the dps treats the entire directory of content and metadata associated with the disk as a single unit. this appears in archivesspace as a single digital object with no components. folders as deliverable units: the dps treats each folder as a subcomponent. the sip as a whole appears as a digital object in archivesspace and each folder appears as a digital object component. files and folders as deliverable units: the dps treats each file and folder as a subcomponent. the sip as a whole appears as a digital object in archivesspace and each folder and file appears as a digital object component. after discussions with stakeholders the service staff decided to package service output with folders as deliverable units. it’s important that the components of a sip appear in archivesspace, but file names may be restricted and should not be visible to researchers. the third component is the security tag. each repository must work with the digital preservation team to create security tags in the dps in order to manage access and restrictions. it was originally thought that security tags could be applied by the ingest workflow and would be the responsibility of the unit, however it was determined that if no security tag is included in the packaging step that package would automatically be assigned the “open” tag. this tag means that anyone with access to that dps repository would have access to the service output. this meant the service had to reconsider the packaging process. each repository has an associated set of security tags in the dps, e.g. repositoryname_open, repositoryname_restricted, etc. since different materials require different levels of security one of these tags should not be applied to the entire corpus of material imaged/captured for that repository. since the security tags must be applied for each disk it makes sense to include security tags as column in the metadata spreadsheet. adding a row to the spreadsheet to include security tags is simple enough, but it also required updates to both the archivesspace import tool and the batch packaging tool. the import tool was updated to ignore this new column in the spreadsheet since it does not affect the item description. the batch packaging tool was updated to pull the security tag data from the outfile spreadsheet to apply to the package. decisions and processes for managing deliverable units and security tags were developed in conjunction with bdawg and the digital preservation team. service staff also met with digital liaisons from special collection units to apprise them of newly developed processes and provide the opportunity to ask questions and voice concerns. feedback and documentation it is vital that the service consults with stakeholders to ensure their digital accessioning support needs for born-digital material are still met. after the initial round of testing for tools and workflows, the service hosted a workshop and invited the digital liaisons for each special collection unit. the workshops focused on walking through the workflow from end to end with special attention paid to the pre-service preparation of material, the metadata spreadsheet, and the service output in both archivesspace and the dps. these workshops also provided the opportunity to view the lab space and highlight the security of the facilityboth the room and the building require id access. drafted workflow documentation was provided ahead of time and this meeting served as an additional opportunity to provide feedback. the workshop was offered three times to accommodate schedules, but it was preferred that more than one special collection unit was represented in each meeting. bringing together liaisons with differing levels of experience with digital material provided the opportunity to share questions and experiences, building a stronger community around born-digital archives at yale. since implementation of the workflow led to new developments, specifically around the archivesspace import and dps ingest, ongoing communication with stakeholders was necessary. additional updates were provided to digital liaisons via email as the workflow was changed and new tools were developed. the service also has a designated email account to help manage communication and ensure that all inquiries are answered. feedback from units often led to changes in the workflow and services available. the option to review content prior to ingest and new documentation to guide units in reviewing service output are both examples of changes made in response to feedback. since the digital accessioning service works with many special collection units it is important that the documentation is easily accessible. the born digital archives working group dedicated a portion of their libguide to the providing information about the service and its use[7]. this libguide page includes an overview of the services provided, contact information, workflow documentation, manuals used for capturing content, and a link to the separate libguide for transporting special collections material. one section of the service libguide is committed to submission documentation including the metadata spreadsheet template, the submission form, and a pdf of the label that should be attached to a shipment of physical media. this libguide serves as a central reference point for units participating in the service as well as providing thorough documentation of the service for external parties interested in how born-digital accessioning is completed at yale. although the lab is shared between the service, the digital preservation team, and the beinecke rare book and manuscript library; other archival units need access to the specialized equipment to appraise content and review media prior to submission to the service. to fill this need the lab is available to staff members on friday afternoons. due to the security of the building, staff members are required to contact the service ahead of time to provide access to the building. service staff are also available for consultation purposes during open lab hours. information about the equipment available in the lab and requirements for using lab hours are documented in the libguide. challenges building a centralized service presented many organizational and technical challenges. the first logistical challenge faced was how to move fragile physical media to an off-site location. the lab is located just over a mile away from central campus but special collection material cannot, for security and preservation reasons, be transported by foot or in a staff member’s vehicle. since the stephen f. gates ’68 library conservation lab is located in the same building as the digital archaeology and preservation lab, guidelines for transporting special collection material was already underway. the born digital archives working group advocated for the development of the special collections transport protocol and ensured the unique needs of transporting physical media were addressed. recommendations for physical media included labeling shipments as containing magnetic media, packing optical media in a vertical position rather than laid flat to minimize the surface area that could be hit in case of impact, and using foam pieces as spacers to ensure that hard drives and other media cannot shift inside the shipping container. the second organizational challenge was setting boundaries and articulating the limitations of the service. the service was planned as a two-year pilot with the goal of eliminating the accessioning backlog of physical media. in order to meet that goal there were many services that could not be provided including file level quality assurance and experimentation to acquire content from unusual digital formats. through clear documentation and ongoing communication with special collection units, the service maintained its intended scope. one major challenge that was not anticipated by the service, was that many units lack the time to prepare physical media for submission to the service. the work of completing the spreadsheet, numbering disks and sometimes creating a new finding aid in archivesspace is time-consuming and difficult to manage with a limited staff. after discovering this gap in services, the born digital archives working group requested a second student worker to help units prepare material for submission to the service. the student worker still requires staff time for management but this new position lessens the burden on special collection units. the final challenge was as much an opportunity as a drawback: the archivesspace-dps system integration. this integration provides the opportunity to sync description and content for the same archival materials in both systems. this synchronized description shows the researcher what is available to access for physical media in a collection. it also provides access services with an easy to follow link to download the content for access in the reading room. the dps can even extract files from certain disk image formats and sync those file names back to the archivesspace record of the digital object. this extraction must be configured in the dps workflow or done manually during full processing. this system integration provides many great opportunities for improved description and access, but it also raised a number of challenges. for example, system updates occur regularly for both archivesspace and the dps and their respective test and production instances. these updates created data synchronization challenges because the data in the test archivesspace instance is overwritten with data from the production instance after updates. this often slowed down the ability to test ingest and import tools and required resetting test data in the dps. although these updates and resets were announced ahead of time and service staff often created screenshots to save examples of test structures, this was still very disruptive to the implementation of the full service workflow. although content was captured from disks starting in july 2016 the first spreadsheet was not imported to archivesspace until march 2017. given the two-year timeline there was no choice but to continue to capture content and complete the rest of service workflow while the system import and integration questions were not yet answered. ingest to the dps was even further delayed as the deliverable unit options were tested and decisions were made within the units about creating and applying security tags. the deliverable unit and security tag decisions required ongoing communication between the stakeholders and the service. once these decisions were made the batch packaging tool required further development to implement the changes. adding security tags as a drop-down menu in the metadata spreadsheet meant that the template spreadsheet could no longer be publicly available on the libguide. in an effort to continue easy access for the digital liaisons, a link was provided from the libguide to a folder in box which is available to all yale staff. next steps the digital accessioning service pilot phase will officially end in june 2018. at the time of this article’s publication, the service has captured content from 5,910 pieces of media and created or updated 2,695 item level descriptions. although the backlog of physical media has been diminished significantly there is still a need for centralized accessioning support for born-digital material. in addition to the existing backlog, special collection units continue to acquire born-digital content. units have expressed an interest in acquiring more born-digital content now that the service exists to ensure responsible stewardship of the materials. starting in july 2018 the service will be a permanent function for special collection units at yale university. the organization of the service will shift to reflect existing needs and available resources. since the beinecke rare book and manuscript library has the largest backlog of born-digital content, the service will focus 75% of its resources to imaging and capturing material from the beinecke with 10% reserved for accessioning material from the other units. the last 15% of service resources will be dedicated to developing infrastructure and documentation to guide born-digital accessioning across yale’s special collections. this allocation of resources is subject to change if a special collection unit has additional needs or requires priority accessioning services. moving forward, the service will be renamed digital accessioning support services (dass). the new name reflects the reality that accessioning is not limited to the work done by the service and accessioning must start before material arrives at the lab. in addition to the functions currently performed by the service, the dass will expand service offerings to include technical accessioning support for born-digital files received via direct network transfers, enriching captured content with metadata that will facilitate preservation and access, providing software training for staff working with born-digital transfers and dass output, and growing the existing consulting role to include advising on the technical aspects of acquisitions. expanding the services offered will require ongoing research in collaboration with bdawg and members of the digital preservation team. conclusion the pilot digital accessioning service proved the value of centralizing the highly technical aspects of working with born-digital archival materials. by removing the need for specialized hardware and expertise within each unit, special collections staff can focus on the other aspects of archiving born-digital materials. the pilot made it possible for staff to view and process digital files from 5,910 pieces of media. new and updated descriptions in archivesspace provide staff and researchers with a better understanding of the digital materials available in a collection. the service also removed a barrier for special collection units that shied away from collecting born-digital materials due to lack of technology and time required for stewardship and preservation. moving forward, the dass will build on the successes of the pilot project and focus on future technical needs for acquiring and accessioning born-digital archival material. glossary bdawg: born digital archives working group dps: digital preservation system, a local implementation of preservica yul: yale university libraries ftk: forensic toolkit disk image: an exact copy of the disk containing the folder structure, orphaned and deleted (but not overwritten) files, and all associated metadata notes [1] yale born digital archives working group. (2016). “born digital @ yale: home.” yale university library research guides. retrieved from: https://guides.library.yale.edu/borndigital [2] storage options: yale its. [updated 2018] yale information technology services; [cited 2018 mar. 2] available from:https://its.yale.edu/services/storage-and-servers/server-hosting-and-administration/server-management/storage-options [3] more information about disk numbering procedures and testing is available at http://campuspress.yale.edu/borndigital/2016/10/06/a-rose-by-any-other-naming-convention/ [4] ipmsg.org. 2018. fastcopy; [cited 2018 mar. 2]. available at: https://ipmsg.org/tools/fastcopy.html.en [5] exactaudiocopy.de. (2018). “extraction technology » exact audio copy.” retrieved from: http://www.exactaudiocopy.de/en/index.php/overview/basic-technology/extraction-technology/ [6] more information about the testing process for cd-das is available at http://campuspress.yale.edu/borndigital/2016/12/20/to-image-or-copy-the-compact-disc-digital-audio-dilemma/ [7] yale born digital archives working group. (2015). “born digital @ yale: digital accessioning service.” yale university library research guides. retrieved from: https://guides.library.yale.edu/c.php?g=300384&p=3593184 about the author alice sara prael is the digital accessioning archivist for yale special collections at the beinecke rare book and manuscript library.  previously, alice was a national digital stewardship resident at the john f. kennedy presidential library, where she assessed digital preservation solutions. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – introduction to text mining with r for information professionals mission editorial committee process and structure code4lib issue 33, 2016-07-19 introduction to text mining with r for information professionals the ‘tm: text mining package’ in the open source statistical software r has made text analysis techniques easily accessible to both novice and expert practitioners, providing useful ways of analyzing and understanding large, unstructured datasets. such an approach can yield many benefits to information professionals, particularly those involved in text-heavy research projects. this article will discuss the functionality and possibilities of text mining, as well as the basic setup necessary for novice r users to employ the rstudio integrated development environment (ide). common use cases, such as analyzing a corpus of text documents or spreadsheet text data, will be covered, as well as the text mining tools for calculating term frequency, term correlations, clustering, creating wordclouds, and plotting. by monica maceli introduction text mining has become a popular approach to analyzing and understanding large datasets not amenable to traditional qualitative research techniques. these tools have been applied to a range of information problems, such as understanding themes in social media or facilitating information retrieval in unstructured data. text mining can be a highly useful tool in the beginnings of research exploration, allowing the textual data to suggest themes and concepts to the researcher during analysis. this can provide a useful starting point for framing further research questions and analysis approaches, particularly if hypotheses and questions are not known in advance (as is typical with an inductive research approach). furthermore, these tools can also assist in cleaning and structuring text-based data for future analysis in visualization or other graphical tools. and, in addition to the tangible research benefits, text mining can be a fun and fruitful process of discovery! r is both a language and environment oriented towards statistical computing and graphics creation (r core team, 2016). r is made available under the gnu general public license; as a result of strong community involvement, there have been numerous extensions, called packages, developed over time, as well as robust documentation. due to this extensibility and versatility, r has remained consistently popular for data and text mining applications across many domains, and includes powerful text mining tools (meyer, hornik, and fienerer, 2008). the following article will discuss the functionality offered by the text mining package in r, assuming little knowledge of the r language and text mining generally. although there are numerous online tutorials and resources in this area, an end-to-end exploration of the common functionalities must still be cobbled together across instructional sources, which can be difficult for novice users. the intention of this article is to cover the primary tools and process, such that readers may dive in with their own text collections immediately. those with pre-existing r experience may wish to skip the following “getting set up” section and proceed directly to the following section on “working with the corpus”. getting set up rstudio[1] is a popular and efficient integrated development environment for working with r, particularly useful for those beginning their work with the r language. in addition to the basic console, many tools are provided in the graphical user interface, including those for: plotting, viewing command history, code completion, and workspace management, e.g. viewing variables currently held in memory. these features, available in the open source and cross-platform version of the tool, can greatly reduce the learning curve; rstudio will be used throughout this article, although the r commands used may be run through any r interpreter. precompiled binary versions of r for various operating systems are available at https://cran.rstudio.com/. upon download and installation of r, the appropriate version of rstudio can then be acquired and installed from https://www.rstudio.com/products/rstudio/download/. the core interaction space in rstudio is the console in the lower left quadrant of the screen (figure 1, below). the command prompt in rstudio is the > symbol, which you will see followed by a blinking cursor. this is where all commands covered in this article should be entered. the lower right pane of rstudio includes several tabs (one of which provides the r manual). this pane opens by default to the “plots” tabs in which all graphics, e.g. wordclouds, will appear upon creation. figure 1. rstudio user interface if you enter your command across multiple lines, i.e. add a newline within your code, the command prompt will automatically change to the + symbol while you are completing the command. there are many invaluable features[2] in the console that will be familiar to those with command-line experience, including recalling previous commands with the up arrow, and tab completion. installing packages the base r package includes many useful mathematical and statistical functions. however, for text mining (and later on – wordclouds) we will need to install additional packages. r packages can be installed via the console or in rstudio through the graphical user interface. for text mining, we will use the tm: text mining package (feinerer and hornik, 2015) which can be installed by entering the following command in the console or using the “install packages” feature in the “tools” dropdown menu in rstudio. > install.packages("tm") working with the corpus the tm package is designed to analyze a collection of text documents, known as a corpus. for this article, we will use a collection of 51 scraped course catalogs to determine prominent and related topics in library and information science programs. a similar approach can be used to analyze text data from a variety of sources, such as tweets or spreadsheet column(s), with a few variations on how the data is input. examples of alternative input data types will follow at the end of this section, which may better meet your particular text mining needs. after we install the tm package (as covered in the previous section) we must then load it into the current session using the following command, again entered on the command line in the console: > library(tm) then we will use the corpus() function to create a corpus of all the text documents contained within the specified directory. > text_corpus once we create our text corpus, we can retrieve summary data about our corpus by entering the corpus name: > text_corpus <<vcorpus>> metadata: corpus specific: 0, document level (indexed): 0 content: documents: 51 vcorpus refers to a virtual corpus, meaning this data will be stored until we close our session in r[3]. we can view the content of the first document using the following command. this is particularly useful to view before (and during) text cleaning to ensure we are getting the desired results. > writelines(as.character(text_corpus[[1]]))   next, we will clean the text which serves to remove extraneous characters, unwanted words, and excessive whitespace, to ensure that words are tallied correctly. an important text cleaning task is stemming. stemming reduces common words to their root, such that the tm package can recognize them as equivalent for counting word frequencies[4]. for example, stemming would reduce "librarians" and "librarian" to the common "librarian", allowing both words to be tallied as the same term. depending on the dataset, stemming may not be desired, as it has the potential to lose subtle meaning between words. it is also common to remove stopwords, which are very frequently used terms ("and", "the", etc.), as their high frequencies can drown out other important words in our analysis. the tm package includes numerous transformations to make the process of text cleaning quick and efficient. most commands have self-explanatory names, e.g. the removenumbers transformation removes all numbers from the text corpus.  the following commands will, respectively, strip extraneous whitespace, lowercase all our terms (such that they can be accurately tallied), remove common stop words in english [5], stem terms to their common root, remove numbers, and remove punctuation. your individual needs may dictate that you exclude some of these. for example, in a corpus where numbers have meaning that you would like to retain, e.g., the term "3d" or other numeric data, you might skip the removenumbers transformation. > text_corpus <tm_map(text_corpus, stripwhitespace) > text_corpus <tm_map(text_corpus, content_transformer(tolower)) > text_corpus <tm_map(text_corpus, removewords, stopwords("english")) > text_corpus <tm_map(text_corpus, stemdocument) > text_corpus <tm_map(text_corpus, removenumbers) > text_corpus <tm_map(text_corpus, removepunctuation) from our cleaned corpus we can then create the document term matrix and view its summary information like so: > dtm <documenttermmatrix(text_corpus) > dtm <<documenttermmatrix (documents: 51, terms: 7891)>> non-/sparse entries: 35252/367189 sparsity           : 91% maximal term length: 55 weighting         : term frequency (tf) here we can see that the documenttermmatrix() function has created a matrix for us, stored in the dtm variable, with the 51 documents as rows and the 7,891 terms as columns. you'll notice that the sparsity percentage is quite high, which is common as there may be a great number of rare terms, misspellings, etc. throughout the documents that appear very infrequently in the corpus. the dim() function is a handy way to view the numbers of rows and columns of our resulting data structure: > dim(dtm) [1]   51 7891 this will be useful knowledge as we reduce our matrix a bit to clean extremely sparse terms as well as prepare for techniques such as clustering (to be covered later on in this article). if we want to reduce some of the sparse terms, and further reduce our matrix size, we can experiment with the removesparseterms() function[6]. this function takes two arguments: first, our document term matrix, and second, a number representing how sparse we want to allow a term to be. this sparsity measure ranges from 0 to 1, with numbers closer to 1 meaning a term can be more sparse, i.e. less present in the corpus but still included in the matrix. we can remove some of the sparsest terms and store our new document term matrix into a new dtm2 variable. we can then re-run the dim() function to see the effect: > dtm2 <removesparseterms(dtm, sparse=0.95) > dim(dtm2) [1]   51 2511 notice that our second document term matrix (dtm2) has become quite a bit smaller! however, one of the difficult balances in text analysis is determining whether the infrequent terms are more (or less) valuable to the particular analysis. depending on the goals of the researcher, there might be more of an interest in exploring the rare terms than the highly common terms. alternatively, we can specify the length of words we would like, as well as decide whether we want to exclude very common words. so, in this case, i might determine that i am interested in terms that appear in at least 5 documents, i.e. not too rare, but that do not appear in more than 45 of the documents, i.e. not too common, and that i only want words between 4 and 20 characters long: > dtmr <-documenttermmatrix(text_corpus, control=list(wordlengths=c(4, 20), bounds = list(global = c(5,45)))) > dim(dtmr) [1]   51 1615 notice that this approach also yields a significantly smaller set of terms, i.e. columns, in our resulting document term matrix. in particular, the word length requires a bit of forethought – if, for example, you were interested in technology terms such as xml, sql, etc. you'd want to include shorter term lengths in your term document matrix. we will continue on working with our original document term matrix, dtm, for the following commands, but all commands could be run on a matrix reduced in either (or both) of the means described above. working with the document term matrix now that we have created the document term matrix, there are a number of built-in functions to assist in understanding our corpus. first, term frequency tends to be a useful starting point in understanding what terms dominate (or trail) our corpus. the findfreqterms() function can be used to assess a particular document term matrix, with the lower bound frequency you desire: > findfreqterms(dtm, lowfreq = 200) in other cases you may wish to create a full list of term frequencies for use in r or to export; this can particularly be useful to work with or chart in other software, like excel. the following command creates the freq variable with term and frequency columns: > freq <sort(colsums(as.matrix(dtm)), decreasing=true) we can then use the head() and tail() functions to view a number (in this case 50) of the top-most and bottom-most terms, as desired: > head(freq, 50) > tail(freq, 50) this step may also be useful in indicating that further cleaning of your data is necessary and there may be additional stopwords you'd like to exclude in the cleaning process. the process of document term matrix creation tends to be naturally iterative unless the dataset is very clean and concise to begin with (which it rarely is!) optionally, we can export the frequency list to a csv (that can be read by excel or other software) using the following command: > write.csv(freq, file = "frequencies.csv") r will save your file to the current working directory; you can see what directory this is by running the getwd(), i.e. get working directory, function: > getwd() [1] "c:/users/monica/documents" if you'd like to change the directory location, you can use the setwd(), i.e. set working directory, command: > setwd("c:/users/monica/desktop") determining the most frequent terms can be a useful first step that can be followed by exploring commonly associated terms. the findassocs() function takes three arguments: 1) the document term matrix, 2) the term to find correlations with, and 3) the correlation limit, expressed as a number between 0 and 1.  a correlation limit of 1 would mean that the terms always appear together.  the particular correlation limit chosen may be dictated by the purpose of your research and is worth experimenting with at different values. you may use the frequent terms identified above as a starting point or explore terms you have a pre-existing interest in understanding. for example, the following command would allow you to explore terms associated with the word "humanities": > findassocs(dtm,term="humanities",0.8) if you have stemmed your terms in the text cleaning steps above, you'll want to use the stemmed term in your findassocs() query as that is how r will be storing the term data. so to find terms correlated with the stemmed term "information" you would use the stemmed version "informat": > findassocs(dtm,term="informat",0.8) a common response to this command is the following message: > numeric(0) this simply means that no terms met the correlation criteria with that particular term. so you might try relaxing, i.e. lowering, the chosen correlation limit or pick a more frequent term from the frequency list you've identified above. once a few interesting term correlations have been identified, it can be useful to visually represent term correlations using the plot() function. if you are using rstudio, the plots will appear in the right-hand plots pane, and can be saved as a pdf/image by using the "export" dropdown. by default the plot() function will default to a handful of randomly chosen terms, with the chosen correlation threshold, e.g.: > plot(dtm, corthreshold = 0.80) however, it is usually far more useful to plot the terms correlated with one of particular interest. instead of letting r randomly pick terms, we can instead pass in a list of terms that is correlated with our particular target term.  the following command will use the now-familiar findassocs() function to retrieve a list of terms and correlation coefficients associated with "humanities", we will then use the name()[7] function to retrieve just the list of terms returned and pass that into the plot() function. > plot(dtm, terms = names(findassocs(dtm,term="humanities",0.8)[["humanities"]]), corthreshold = 0.80) figure 2. plot of terms correlated with "humanities" we can also use the attrs argument to control many of the display characteristics of the plot, e.g. color, node shape, font size, etc. to make it more visually appealing[8]. > plot(dtm, terms = names(findassocs(dtm,term="humanities",0.8)[["humanities"]]), corthreshold = 0.80, attrs=list(node=list(label="foo", fillcolor="lightgreen", fontsize="16", shape="ellipse"), edge=list(color="cyan"), graph=list(rankdir="lr"))) figure 3. styled plot of terms correlated with "humanities" wordclouds[9] can be another useful visual tool for representing frequencies at a glance. creating wordclouds in r is quite simple using the rgraphviz library. we can use our freq variable as created above to pass the terms and frequencies to the wordcloud() function for drawing. first, we will install and load the wordcloud library (which will also load the rcolorbrewer package for us automatically), then run the wordcloud() function which takes several arguments: the list of terms, their frequencies, and the minimum frequency term you'd like included. a simple wordcloud can be created like so: > install.packages("wordcloud") > library(wordcloud) > wordcloud(names(freq), freq, min.freq=400) however, there are many attributes that can be changed to control various aspects of the resulting wordcloud. in the following example, several additions make our wordcloud more visually pleasing. > wordcloud(names(freq), freq, min.freq=400, max.words=inf, random.order=false, colors=brewer.pal(8, "accent"), scale=c(7,.4), rot.per=0) in order of appearance: the max.words argument controls how many words to plot (in this case, inf or infinite, meaning all words with greater than 400 frequency); the random.order argument set to false forces the highest frequency terms to be plotted first (and therefore appear in the center of the cloud); and the colors argument, which is commonly used with one of the pre-defined color palettes from the rcolorbrewer[10] package[11]. also available is the scale attribute to determine the size difference between the most and least frequent terms and the rot.per attribute to control how many words are plotted vertically, expressed as a numeric from 0 (all terms horizontal) to 1 (all terms vertical) (figure 4, below). figure 4. wordcloud of terms with frequency greater than 400 lastly, we will explore term clustering. clustering consists of identifying objects (in our case – terms) that have a similar presence within the documents and grouping them, using a variety of existing clustering algorithms. clustering techniques are very useful for large corpora where pre-existing categorizations are not known. first, we will create a term document matrix (with terms as rows as opposed to documents), next we'll manipulate the sparsity until we get a nice-size list of terms to work with (usually about 40 terms is the maximum to create a readable cluster size). with this particular dataset, the sparsity percentage needed to be set at 10% to get the number of terms down to a manageable 35. you may want to re-run the removesparseterms() function a few times with a different value for sparse=0.1 to get an appropriate number of terms for the particular dataset you are working with: > tdm <termdocumentmatrix(text_corpus) > tdm2 <removesparseterms(tdm, sparse=0.1) > dim(tdm2) [1] 35 51 next we will shape the data as necessary to run the hclust() function[12] which performs agglomerative hierarchical clustering on our dataset. in this clustering technique, terms are first placed in individual clusters, then the process identifies the nearest (i.e. most similar) cluster and merges the two into a new cluster. this merging process is repeated iteratively until one single cluster is created out of the entire dataset. the following commands will first create a distance matrix representing the similarity between terms, then we will use the hclust() function to apply the clustering technique: > d <dist(as.matrix(tdm2)) > hc <hclust(d) lastly, we will use the plot() function to visually display our results in a dendrogram chart (figure 5, below), which is a tree diagram typically used to graphically represent clusters for ease of analysis. > plot(hc) figure 5. denogram created through hclust we can now visually inspect the cluster dendrogram. interpreting the results of clustering is as much an art as a science. as mentioned earlier, this method of clustering begins by placing each term in its own cluster then evaluating its similarity to other terms; similar terms are then merged into clusters and the process repeats until the clusters are joined into one large cluster. when read from the bottom up, our dendrogram visually represents this step-by-step clustering process that took place. the y-axis of the dendrogram – height – represents the calculated distance between clusters. a small height measure where the terms merge means the distance between terms is small. so the clusters that merged near the bottom of the chart are most similar and tend to co-occur in the documents. the length of the vertical lines indicates how far apart (dissimilar) the clusters are. we can see that the "inform"/"librari" cluster to the right was not merged until quite high on the chart, meaning that this cluster of terms is quite dissimilar and thus forms a nicely distinct cluster. we are left with what appears to be two distinct clusters – one with "inform" and "librari" and one larger cluster containing all the other terms. we can then use the cutree()[13] function to have r do the work of dividing the "tree" of terms into two clusters and visually outline them on our plot (figure 6, below): > groups <cutree(fit, k=2) > rect.hclust(fit, k=2, border="red") figure 6. dendrograms with clusters outlined alternatively, in the above commands, we could replace the k=2 with the number of clusters we felt fit the data most appropriately (researchers may interpret a dendrogram differently). a larger and more diverse dataset would likely yield a greater number of distinct clusters than in our example above. clustering can provide a useful starting point in categorizing documents and understanding the representative terms, particularly to build on what has been gleaned from analyzing term frequencies and correlation. other input data formats we have explored basic text mining commands with a very common use case, working with a corpus consisting of a folder of text documents. another frequent need is the ability to analyze spreadsheet data. although r can read in data in excel formats, it is much easier to work with csv (comma separated value) files. in excel, the file can be saved as the .csv "comma delimited" format (using the "save as" function), which can then easily be imported into r using the following command. the header argument should be set to true or false depending on whether your data includes a header row: > text_data <read.csv("my-csv-file.csv", header = true) during the next step, creation of the text corpus, instead of the dirsource() function we used with a directory of files, we'll tell r to treat each row of our csv as a distinct document, using the dataframesource() function instead: > text_corpus <corpus(dataframesource(text_data)) the same techniques covered above can then be applied to these alternate data sources: cleaning the text, creating the document term matrix, and then conducting a variety of analysis techniques. these techniques can also be used to shape the data as a series of nodes (the terms) and edges (measure of the closeness of relationships between terms) that can be input to visualization software, such as gephi[14]. available text data sources there are many freely available corpora that can be useful for practicing these text mining techniques, as well as conducting novel research.   browsing these data sources may also spark ideas for mining your own data! as a good starting point, the usc libraries maintain an excellent research guide listing freely available corpora at http://libguides.usc.edu/textmining/tools, with a mix of full text and metadata offerings. their research guide notably includes corpora from hathitrust, project gutenberg, internet archive, and many others of relevance to the lis domain. another very common and increasingly popular use case is mining tweet data retrieved from twitter's api, using r packages for authenticating and reading twitter data[15].   no matter the source of the text data, once it is read into r, the same basic commands described above can be used to analyze and interpret the findings. conclusion there is a huge amount of online documentation, tutorials, free courseware, and other online materials dedicated towards learning both novice and advanced tools in r. for those new to the area of text mining in r, this can be overwhelming and require cherry-picking instructions from numerous resources. this article brings together the most common analysis techniques that can be applied to text data, providing an introduction to the powerful text mining functionality available with the tm package. references feinerer, i and hornik, k. (2015). tm: text mining package, r package version 0.6-2. retrieved from http://cran.r-project.org/package=tm meyer, d. and hornik, k. and feinerer, i. (2008). text mining infrastructure in r. journal of statistical software, 25 (5). pp. 1-54. r core team. (2016). r: a language and environment for statistical computing. retrieved from https://www.r-project.org/about.html notes [1] https://www.rstudio.com/ [2] https://support.rstudio.com/hc/en-us/articles/200404846-working-in-the-console [3] upon closing rstudio, you will be prompted to save your workspace if you wish. this will store your data structures to be reloaded when rstudio is reopened. [4] the tm packages uses the popular porter stemming algorithm detailed at http://tartarus.org/martin/porterstemmer/ [5]you can also create a list of your own stopwords to remove during this step, which is useful if you have words unique to your corpus that you'd like to omit, e.g.: > my_stopwords <c(stopwords('english'),'like','learn','use','better','know','etc','id','using','also') > text_corpus <tm_map(text_corpus, removewords, my_stopwords) [6] http://www.inside-r.org/packages/cran/tm/docs/removesparseterms [7] note that much existing documentation uses the rownames() function here, e.g. > plot(dtm, terms = rownames(findassocs(dtm,term="systems",0.3)), corthreshold = 0.30) however, this syntax will not work with the latest version of the tm package, as the data structure returned by the findassocs() function was changed. [8] the rgraphviz documentation explains the various attributes of the graph that can be controlled in section 4 of https://www.bioconductor.org/packages/devel/bioc/vignettes/rgraphviz/inst/doc/rgraphviz.pdf [9] https://cran.r-project.org/web/packages/wordcloud/wordcloud.pdf [10] https://cran.r-project.org/web/packages/rcolorbrewer/rcolorbrewer.pdf [11] the list of color palettes is viewable by running the command: > display.brewer.all() [12] http://www.inside-r.org/r-doc/stats/hclust [13] https://stat.ethz.ch/r-manual/r-devel/library/stats/html/cutree.html [14] https://gephi.org/ [15] a concise and illustrated guide to retrieving twitter data with r is available at https://www.credera.com/blog/business-intelligence/twitter-analytics-using-r-part-1-extract-tweets/ about the author monica maceli is an assistant professor at pratt institute school of information, focusing on emerging technologies in the information and library science domain. she has an industry background in web development and user experience, having held positions in e-commerce, online learning, and academic libraries. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – practical ways to promote and support collaborative data analysis projects mission editorial committee process and structure code4lib issue 12, 2010-12-21 practical ways to promote and support collaborative data analysis projects this article is an appeal to technically-oriented library staff to initiate collaborative, bottom-up data-analysis efforts across their libraries. we discuss successful strategies used at north carolina state university (ncsu) libraries for initiating cross-departmental outreach for data-analysis work, as well as structuring and storing data, and disseminating findings. we present several specific examples of collaborative data-analysis projects undertaken at ncsu libraries. by joyce chapman and cory lown background launching a data-analysis program is challenged in equal parts by organizational and technical considerations, and while libraries have recognized for years the importance of using data to drive decision-making, translating this recognized need to the day-to-day operations of the library can be daunting. while a full-fledged data-analysis program may not be feasible for many organizations, this should not stand in the way of grassroots data-analysis efforts. working from the bottom up to build a culture of data analysis is possible at each of our institutions, and technically-oriented librarians are in a unique position within the library to facilitate data-analysis work. most technically-oriented people bring a set of computational thinking skills to bear on problem solving that is invaluable throughout the process of data analysis: define the problem; collect, structure, and process data; understand the importance of normalization; and understand basic statistics (wing, 2006; weyland, 2009). they often have an understanding of the kind of transactional data that are produced within the library, the kind of questions that data are capable of addressing, and how to access and process that data to make it useful. in short, technically-oriented staff are in a unique position to conduct outreach and work with others who have data-analysis needs throughout the library, working beyond day-to-day roles to bring their computational thinking skills to bear on data analysis. at ncsu libraries various groups across the institution are working on data analysis to inform decision-making. while some departments are able to meet their own data-analysis needs, others are in need of help. in working with librarians throughout our organization we have found that while structured data is collected by many, it often remains underutilized. the reasons for this vary from a lack of time and resources, a lack of knowledge of how to analyze the data, or an unclear understanding of the questions that staff want the data to answer. we have also found that staff may be unaware of the various data sources available for analysis across the libraries, and that data is often collected without a clear sense of what questions the data will be used to answer. additionally, data that is collected by hand may not be normalized, either because data collectors are unclear on the ways in which data must be normalized or formatted to facilitate programmatic analysis, or because a clear purpose and process for the data collection was never established. in this article we discuss our experiences and lessons learned as technology-oriented staff working to encourage bottom-up data-analysis at our library. we focus on data analysis for low-level decision-making about library products and services; we do not address collecting data for arl statistics, total quality management analysis, libqual+®, using data to determine the value of libraries, or for high level management decisions. we are interested in the use of quantitative data that can help librarians answer questions that they can use to make effective changes to products and services on the ground. examples of such questions at ncsu include: does demand outstrip availability for items such as public computers, device lending, or books in the textbook collection at certain points in the semester or time of day? can we modify staffing models or opening hours to best suit patron needs in areas such as the special collections reading room or the reference desks? by identifying levels of use for specific features of web applications, can we modify design or content in order to maximize value to patrons? while qualitative data is important for understanding the “whys” of trends, collecting and interpreting such data requires significant time commitment. we focus on quantitative data because we perceive it to be more practical for low-barrier, ongoing analysis efforts. in many cases, the initial effort of collecting and processing data can be leveraged into the future with minimal further investment in time or development. getting started we found that a useful first step toward organizing data-analysis efforts was compiling a list of current data collection activities by conducting interviews with department heads and other staff across the libraries. not only did this generate documentation and provide a broad view of the libraries’ extant data collection activities, it served as the initial outreach to various departments and units, broaching the topic of data-analysis. the documentation became the basis of a decision matrix for assessing opportunities and challenges associated with analyzing particular data sets. the decision matrix comprised a list of the data sources and answers to a series of questions about each data source. questions we found valuable in assessing data sources included: are there clear stakeholders and technical contacts for this data source? can stakeholders articulate the questions they want answered through data analysis? can the available data answer these questions? what is the format of the data source? what is the cost of normalizing the existing data? would analyzing this data be of value to one or more stakeholders, departments, and/or the libraries as a whole? explicitly compiling this data for each data source forced us to pursue answers to unknown aspects of data sources before proceeding. the process of compiling this decisions matrix — as well as the resulting documentation — helped us identify low cost, high value opportunities for analysis. these were projects that had clean data, enthusiastic stakeholders with questions they wanted answered, and the results of which had the potential to be interesting to a broad range of staff. data sources we were more reluctant to pursue for analysis had the following characteristics: they did not have clear stakeholders or the stakeholders did not have strong interest in analysis, the data would require extensive cleanup and workarounds, or results from analysis would be of only narrow interest. however, if the results of data analysis were perceived to be of high enough value to the libraries, a high cost in one of these areas would not be considered a barrier. data structure and planning as we documented data sources, it became clear that the mere act of collecting data does not necessarily make that data well suited to answering the questions of stakeholders. while it is not always possible to know how data will be used in the future, this problem seems to arise most frequently when data collection had been identified as something that should be done but is undertaken without knowledge of what questions the data will eventually need to answer. in such cases, data elements might be missing that would be required to answer questions, the data might not be normalized, or data might not be interoperable with another data source. while impossible to avoid at times, collecting data without a clear understanding of how the data will be used can have organizational implications: primarily, limited utility of the data for answering some questions and a data format or structure that limits the possibility for programmatic analysis of the data. one challenge of trying to make use of data that has already been collected is that data collection practices can change over time. in many such cases, the data collection practices improved significantly over the course of several years. although it is desirable to improve data collection practices, shifting practices can make longitudinal analysis difficult or impossible. transactional data maintained by systems such as the integrated library system (ils) can also contain hidden challenges. it is important to know that the transactions recorded by the system accurately represent what you are attempting to measure. for instance, it would be important to know if the ils differentiates checkouts by library users and checkouts for internal processes (weyland, 2009). another pattern we discovered is that many people have difficulty conceptualizing questions that data with particular characteristics can answer. we found it to be helpful for stakeholders if we presented examples of reports from previous data-analysis projects to provide ideas about the questions data can help answer. using broad categories as a prompt for thinking about useful questions was also an effective technique when working with potential stakeholders. for example: are you interested in trends across different units of time (years, semesters, days, hours of the day)? what traits about patrons (affiliation, major, geographic location) do you think might be useful to understand the use of this service or application? over the course of conducting several data-analysis projects as collaborations between technical staff and stakeholders, we identified a number of beneficial results from conversations between these two groups. technically-oriented staff were able to gain a broader perspective on the concerns of the organization and data collection activities as they currently exist and the departments we worked with benefited from discussions about data collection techniques and structuring, as well as the kind of questions data can help answer. we are hopeful that a future outcome of these conversations will be better planning from the start for data collection efforts and that utility and ease of analysis will drive data collection practices. data storage and dissemination storage and dissemination of data should be considered carefully, especially as a portfolio of multiple data-analysis projects grows. how and where you choose to store the data impacts its accessibility and utility for analysis. for projects that involve computer-generated transaction logs (which were typically either transaction reports exported from our ils or custom transaction logs for home-grown applications), we found it useful to create a single directory on the server with sub-directories for each project or application. there are a number of benefits to this: starting a collection of curated data sets, setting access permissions can be handled on one directory, and with the data in one place you or others may find opportunities to re-purpose data in the future. we also found that a number of different methods were useful for sharing findings once the data had been analyzed. in some cases we provided a static written report to stakeholders. because analysis was usually being offered as a one-time service rather than an ongoing one, this was done in cases where significant data clean up had been required to meet analysis needs or the data could not be easily collected into the future and lent itself more readily to a snapshot of a particular service or application. in other cases the cost of continuing to collect transaction data over time was low and it was deemed beneficial to be able to view statistics about a particular application or service at any point in time. in these cases custom php scripts were written to parse the transactional data and generate web-based reports based on the most current data. we found google’s visualization api to be extremely useful for generating web-based charts and graphs based on the latest log data (google, 2010). figure 1. google visualization api interactive chart: annotated time line for quicksearch modules clicks by day when developing custom transaction logs for specific applications we found particular tab-delimited field types useful across projects. a unix timestamp makes time-based comparisons and math simple. it can also be useful to include a human readable date and time to make the logs easier to inspect. in some cases an ip address is useful, although this can be excluded in cases where the transaction may record more personal information such as query terms. these initial fields are then followed by a series of categorizing codes specifying information such as a type of action, a computer’s physical location, a query term, or other relevant information about a transaction. in developing custom logs we have attempted to balance utility and simplicity. this modest level of standardization also eased development of scripts to parse the logs and produce reports. below is an example of data collected by a custom log file. // timestamp ip address library region code computing platform action 1250266660 152.1.107.147 vml 016 pc login 1250266663 152.1.222.47 hill 003 mac logout over time log files can grow quite large. if processed without memory efficiency in mind it is easy to run into memory limitations. using php we found it useful to process the log file within a do-while loop line by line such that any calculations performed were only performed on lines meeting certain criteria and we only store values when necessary. below is an example of a simple php script to parse a typical log file. if ($input_logfile) { do { $log_input_line = fgets($input_logfile); list($timestamp, $ip_address, $library, $region_code, $platform, $action) = explode("\t", $log_input_line); // do whatever you want with the data... } while (!feof($input_logfile)); fclose($input_logfile); } else { print "unable to open logfiles\n"; return; } one advantage of using the google visualization api is that the javascript used to create the charts and graphs can be generated by php with values calculated directly from transaction logs. this ensures that the report stays up to date with the most current log data. in these cases we found it useful to create a web directory to collect these script-generated reports, with a sub-directory for each application or service. over time, as the number of these reports increases, having them in one location will simplify setting permissions and increase the discoverability and utility of the reports. below is an example of the javascript input for the google visualization api as well as an example of a php script to output data from an array into the required javascript data value format. <script type="text/javascript" src="http://www.google.com/jsapi"></script> <script type="text/javascript"> google.load("visualization", "1", {packages:["corechart"]}); google.setonloadcallback(drawchart); function drawchart() { var data = new google.visualization.datatable(); data.addcolumn('string', 'curriculum'); data.addcolumn('number', 'total requests'); data.addrows(3); data.setvalue(0, 0, 'english'); data.setvalue(0, 1, 14269); data.setvalue(1, 0, 'adult & higher education'); data.setvalue(1, 1, 11396); data.setvalue(2, 0, 'sociology'); data.setvalue(2, 1, 6132); var chart = new google.visualization.piechart(document.getelementbyid('chart_div')); chart.draw(data, {width: 600, height: 500, legend: 'right'}); } </script> <?php $i = 0; foreach ($use_by_curriculum as $curriculum => $count){ print "data.setvalue($i, 0, '$curriculum');\n data.setvalue($i, 1, $count);\n"; $i++; } ?> we have also centralized the code that parses popular units of time (such as academic years and semester boundaries) in a single file accessible to all other scripts. in this way, web-based reports can stay up to date and provide valuable information about the performance of the services and applications as needed without having to create new reports by hand. an example of the dates file is shown below. // spring semester, 2010 if (($date >= "2010-01-11") && ($date <= "2010-05-16")) { $semester = "spring"; $academic_year = "2009-2010"; if ($date >= "2010-01-11" && $date <= "2010-01-17") { $week = 1; // ... intermediate weeks removed ... } elseif ($date >= "2010-05-10" && $date <= "2010-05-16") { $week = 18; } $semester_date_information = array( 'academic_year' => $academic_year, 'semester' => $semester, 'week' => $week ); examples of collaborative data analysis at ncsu below are some examples of data-analysis projects conducted at ncsu by technical staff working together with various stakeholders throughout the libraries. not all of these projects were conducted by the authors; the examples represent a growing trend of collaborative grass-roots data analysis at ncsu libraries over the past few years. reference transactions data-analysis projects can be conducted without any programming skills or specialized software. the key is planning how data will be structured, what questions will be asked, and whether the data can answer those questions. we recently worked with the reference and information services (ris) department to analyze trends in reference transactions with patrons. questions that ris had for their data included: what longitudinal trends do we see in library reference? what trends do we see over the course of semesters and days? what trends do we see in library chat reference? what patterns exist within different types or different media of reference? do trends in library reference correspond in any way to trends in other library usage patterns, such as gate counts or computer usage? most of the data related to reference transactions is gathered manually and stored in microsoft excel spreadsheets, so we decided to create graphs and conduct most data analysis in microsoft excel as well. this tool was familiar to stakeholders and was able to meet stakeholders’ needs just as well as more complex tools would have. findings aided staff in understanding issues such as peak hours of the day and weeks of the semester for reference transactions, and trends in the content type and media of staff transactions with patrons over time. quicksearch launched in 2005, quicksearch is a home-built federated search tool designed to connect users to a variety of library resources, services, and tools (sierra, 2007). quicksearch runs the primary search box on the ncsu libraries website and provides results from various sources including serial solutions’ summon articles database, our endeca-based catalog, the library website, and journal titles. quicksearch is managed by a team of librarians from the research and information services and the digital library initiatives departments. before 2010, use statistics from quicksearch were gathered from urchin, the web analytics tool. in an effort to provide more robust and detailed reports about use of the search application we began custom logging of quicksearch in january 2010. the logs contain a timestamp, information on whether the transaction is a search (user performs a search) or a click (users follows the url of a result), and if it is a click, details about the specific sub-module and link within quicksearch that the user follows. questions the quicksearch product team wanted to be able to answer on an ongoing basis include: what is the ratio of clicks to searches within quicksearch as a measure of whether results are meeting user needs? which modules (articles, catalog, website, journals, etc.) are the most frequently selected results within the results page? how does use of the search application change over the course of the semester? what are the most commonly searched terms? the resulting reports, which were written by a member of the quicksearch product team, have been used to analyze use patterns and influenced a major redesign of the results page. the redesign prioritized giving article and catalog results primary and nearly equal real estate on the page due to evidence of close to equal utilization. prior to redesign, articles had been given primary real estate and catalog results were provided less space. additionally, the reports have been used to ensure that the redesign did not reduce the click to search ratio (0.76 clicks per search in the semester prior to redesign, 0.88 in the first 3 months of the semester following redesign), which would indicate a problem with the results pages failing to prompt a url selection by users, and thus a less effective design. additionally, a report that displays the most frequently searched terms has been used to ensure that the search tool works effectively for these frequently searched terms. figure 2. pre-redesign quicksearch results page figure 3. post-redesign quicksearch results page library course tools the library course tools application dynamically generates student-centric views of library resources and tools for each unique course taught at ncsu and was developed in 2008 through a partnership between digital library initiatives and reference and instructional services staff (casden et al. 2009). staff worked together not only to develop and implement the product, but also to identify data collection needs and implement custom logging. the stakeholders’ questions included the following: how has the use of library course tools changed since the launch? how does use fluctuate over the course of the semester? what are the most and least used curricula? what widget content is most useful to students? because the data source for library course tools is a custom transaction log on a server, the data is available for real-time analysis. for the parsing and generating of reports, the google visualization api was chosen to display the results of data analysis. because technical and non-technical staff had collaborated on the project from the beginning, the log data source was structured in a way that was accessible for programmatic data analysis. according to stakeholders, data analysis helped them understand the community’s interaction with library course tools and the ways in which the application needed to change in order to better meet student needs. data analysis showed that widgets lacking dynamic content were used far less than other types of widgets. this information led stakeholders to develop more dynamic and curriculum-focused content for widgets. the ability to view the curricula for which the most library course tools pages were used helped stakeholders understand that there was a clear correlation between librarian participation and use of the tool. for example, the very high use of library course tools by chemical engineering students (relative to the size of that population) is hypothesized to be a direct result of the enthusiasm for the project and resulting outreach and content creation by the librarians who work in this area. special collections patron base and physical materials usage analysis in 2007 the special collections research center (scrc) at ncsu libraries developed a microsoft access database in which to store information about patron registrations, material transfers, and materials use. while the department was long aware that the data could inform them of important trends in collections use and patron base, they did not have the technical resources on hand to conduct an in-depth analysis of the data. we discovered this rich data source through the previously-mentioned outreach program during which we approached different departments and discussed data-analysis services that could be offered. through a discussion with scrc curators, staff, and department administrators we gained an understanding of the department’s data-analysis needs. because the data was managed in a microsoft access database stored on a file system rather than a web server, we chose not to produce live reports with the google visualization api. instead, a one-time snapshot of use covering a three-year period (the time available since the inception of the database) was prepared as a static report. data from exported database tables was analyzed programmatically and adobe illustrator was used to create graphics. “one example of how results helped the department better understand user behavior was an analysis of the percentage of total use of physical materials by hour of the day (shown in figure 4).” figure 4. example of a graph created for scrc in adobe illustrator and photoshop challenges with this project included data that was not able to answer stakeholder questions and issues with data structure or normalization. for instance, we found that certain data elements were not tracked that would be crucial to addressing several of the questions the department wanted answered. for example, the question of which collecting areas receive the most use could not be easily answered because the collecting areas to which a collection pertains are not tracked. another example is related to types of use; while use could be broken down by frequency of material checkouts, it could not be analyzed from the point of view of discrete research projects because the concept of a multi-day research session as distinct from individual circulations of a resource is not currently accounted for in data collection. issues with data structure and normalization included problems with inconsistent primary keys between tables, a small number of missing records, and legacy data that had been imported without being updated to the current format. these were by no means insurmountable problems, but resulted in some data that could not be included in analysis. conclusion while launching a successful data-analysis program can be challenging and large-scale data-analysis programs may not be feasible for many organizations, each of us can work from the bottom up to foster a culture of cooperative data analysis at our institution. we think that technically-oriented librarians are in a unique position to facilitate data-analysis work by bringing computational thinking skills to bear in collaborative efforts throughout the library. although there is undeniably a time-commitment involved in initiating this kind of effort, especially in cases where data-analysis and interdepartmental outreach are not part of one’s usual responsibilities, we think the short term and long term benefits are significant. at ncsu libraries, a grassroots approach to data analysis with cooperation between technically-oriented librarians and departmental stakeholders has yielded positive short term results, helping a variety of departments to answer questions about how patrons use specific library applications and services. developing strong collaborative partnerships between departments is both a challenge and a positive outcome of this approach. longer term, we hope these efforts can increase awareness of good data collection practices and of the opportunities that exist to leverage data to answer questions that can help libraries improve applications and services, as well as foster a spirit of collaboration between library departments. additionally, the effort to develop custom logs with live reports programmed in php and the google visualization api will continue to pay off in ongoing assessment efforts. along with others at our library, we continue to develop a set of practices that we hope will solidify data collection and assessment as part of the planning process for services and applications from their inception. references casden j, duckett k, sierra t, ryan j. 2009. course views: a scalable approach to providing course-based access to library resources. code4lib journal [internet]. [cited 2010 dec 6]; 6. available from: http://journal.code4lib.org/articles/1218 google. 2010. google visualization api [internet]. [cited 2010 dec 6]. available from: http://code.google.com/apis/visualization/interactive_charts.html sierra t. 2007. a single search box interface to the ncsu libraries, two years later. digital library federation spring forum 2007. pasadena, california. 2007 apr 24; [cited 2010 dec 6]. available from: http://www.slideshare.net/tsierra/tsierradlfspringforum2007 weyland m. 2009. the wise use of statistics in a library-oriented environment. code4lib journal [internet]. [cited 2010 dec 6]; 6. available from: http://journal.code4lib.org/articles/1275 wing j. 2006. computational thinking. communications of the acm 49(3):33-35. (coins) about the authors joyce chapman is a libraries fellow at north carolina state university libraries where she works in the digital library initiatives and metadata & cataloging departments. she is currently working on a project to analyze and visualize library usage data. cory lown is digital technologies development librarian in the digital library initiatives department at north carolina state university libraries. at the moment, he focuses on developing supplements to special exhibits for mobile devices. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – editorial introduction mission editorial committee process and structure code4lib issue 16, 2012-02-03 editorial introduction the winter months bring us festivities like mardi gras. here at the code4lib journal, we present you with a veritable feast to indulge in as our mid-winter festival offering. consume slowly, to fully appreciate the myriad flavors and enjoy the richness of the fare. by carol bean it’s winter, a time when we all think of food, and here we offer up a full fare of delicacies to warm your soul. but before you rush headlong into this dining experience, take a moment to savor the aroma of fine scripting, the abundance of tempting ideas laid out in glorious openness, the exhilaration of anticipating the finest brain food your colleagues can offer. we bring to you, after weeks of ardent and arduous refinement, a nine course feast, replete with delectable samples of scripts, mouthwatering snippets of code, and oversized servings of data manipulation. for the amuse-bouche, may we offer creating a seamless cross-platform online experience for mobile users to stimulate the palate? katherine lynch has prepared a blend of disparate web ingredients to masterfully combine them into a pleasing display for any palate. as we move to the second course, i would suggest html5 microdata and schema.org as a delightful transition. jason ronallo presents us with results of his recipe for successfully mixing new ingredients in web presentation, including, for our dining pleasure, a discussion of ways to enhance and extend the sometimes sparse offerings of schema.org. for the third course, where we prepare for the delights of heavier fare, you will find nothing better suited than using vufind, xampp, and flash drives to build an offline library catalog for use in a liberal arts in prison program. julia bauder takes a melange of difficult circumstances to create a masterful dish that is suitable for many other settings. in the fifth course, still teasing your palate, we offer improving the presentation of library data using frbr and linked data. this timely piece, prepared at the oslo public library in norway, has been created by anne-lena westrum, asgeir rekkavik, and kim talleras, three data chefs who introduce us to their work with the pode project to wrangle marc data into a more palatable dish. as we move from the lighter fare, we lay out presenting results as dynamically generated co-authorship subgraphs in semantic digital library collections to satisfy your cravings for a richer and more complex dish. james powell, tamara m. mcmahon, and dr. ketan mane challenge our taste buds with fare from the los alamos national laboratory that combines metadata and visualization tools to give users a graphic display of author connections. then, for course number six, rather than staid duck, meat or lamb, we offer you an opportunity to extend visualization techniques with william denton’s on dentographs, a new method of visualizing library collections. our seventh course takes you to the inner workings of the virginia foundation for the humanities, with matthew gibson’s using xslt’s sql extension with encyclopedia virginia. here you will find a delectable taste of combining xml with a relational database that enhances the flavor of both. and as we move to the penultimate course, we bring you ref2ris: importing word-processed bibliographies into bibliographic management software. deborah fitchett lays out a melange of bibliographic styles, and offers us a sampling of fruit to pull the dish together. finally, as a sweet conclusion to this singular repast, consider terry reese’s purposeful development: being ready when your project moves from ‘hobby’ to mission critical. here, your meal will find a thoughtful conclusion with the juxtaposition of open and closed code. we hope you enjoy our mid-winter feast! subscribe to comments: for this article | for all articles 2 responses to "editorial introduction" please leave a response below: david, 2012-02-07 hate to be the one to spot the first error, but it’s actually the height of summer! ;-) carol bean, 2012-02-07 heh! i thought about that as i wrote it, and decided to go with a northern hemisphere bias. but you are correct. i should have said simply, at this time of year. :) my apologies! carol bean leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – communication between devices in the viola document delivery system mission editorial committee process and structure code4lib issue 27, 2015-01-21 communication between devices in the viola document delivery system viola is a newly developed document delivery system that handles incoming and outgoing requests for printed books, articles, sharing electronic resources, and other document delivery services on the local level in a library organisation. an important part of viola is the stack fetching android application that enables librarians to collect books in the open and closed stacks in an efficient manner using a smartphone and a bluetooth connected portable printer. the aim of this article is to show how information is transferred between systems and devices in viola. the article presents code examples from viola that use current .net technologies. the examples span from the creation of high-level rest-based json apis to byte array communication with a bluetooth connected printer and the reading of rfid tags. please note that code examples in this article are for illustration purposes only. null checking and other exception handling has been removed for clarity. code that is separated in viola for testability and other reasons has been brought together to make it more readable. by theodor tolstoy background overview during most of 2013, the development team at stockholm university library built an entirely new document delivery system to replace its outdated and discontinued predecessor. viola is a document delivery system that handles incoming and outgoing requests for printed books, copies of articles, the sharing of electronic resources, and other document delivery services. an important part of viola is the stack fetching android application that enables the librarians to collect books in the open and closed stacks in an efficient manner. viola is built around two main workflows: one for books coming into the library and one for books going out. it interacts with many external systems such as the local ils, the joint catalogue of the swedish academic and research libraries (libris) and the swedish interlibrary loan service (iller). this means that the users can stay in viola to do their work and not have to perform tasks in other systems. outgoing orders – incoming books figure 1: the different kinds of outgoing orders for documents in viola. viola handles both interlibrary loan requests from other libraries and requests for books to purchase from patrons. viola has its own circulation and invoicing modules, so besides user authentication, no interaction takes place between viola and the ils when it comes to outgoing orders. incoming orders – outgoing books figure 2: overview of the orders for books in the local stacks coming into viola. viola handles four types of orders for outgoing books: interlibrary loan requests from the national ill system manual orders from libraries that are not part of the national system, like special libraries or libraries located outside sweden orders for books that are to be sent to the researcher’s offices (faculty office delivery) requests for books in the closed stacks there is also a fifth type of order that handles books that are reported as missing and that are to be looked for once more. before these systems were in place, each order was printed out on paper (each type of order had its own designated printer) and then processed by different librarians. sometimes up to three librarians could meet at the same shelf in the closed stacks, looking for different books. the task of sorting the hundreds of orders a day in a way that made the fetching efficient was also very time consuming. needless to say, there was room to make theses workflows more efficient. today, all these orders are processed through viola. the orders are grouped and sorted so that the fetcher will follow a predefined route around the library. the orders are then downloaded to a smartphone for the librarian, who walks around the library and collects the books. when a book is found, the barcode is fed into the system either through scanning the rfid tag using the nfc reader or through reading the barcode using the camera. then the librarian prints a slip, using a bluetooth connected portable printer, and puts it in the book. once the books are put on the hold shelf, the fetch is reported back to viola and the external systems are automatically updated. technical overview viola is built on top of the newly open sourced [1] .net platform. it is currently run on local windows servers. viola consists technically of three parts: a web site that provides a graphical user interface for the system and also works as a web service endpoint; an android application that the librarians use when fetching books in the stacks; and a windows service that runs different periodical tasks like watching for changes in external systems and fetching new requests. behind the scenes, viola uses sql server for the database with search support provided through apache solr. the web site and the windows service share most of the code through the viola core library that handles most of the business logic and data access. figure 3: the different parts of viola and the main technologies they are built with. the android application is built as a native android application, but written in c# using the xamarin platform [2]. this enables the reuse of a lot of code from the viola core library. however, the viola core library has not yet become a portable class library [3]. examples of communication between systems and devices in viola reading a barcode with a smartphone camera few external code libraries have been used in the building of the android application since both bluetooth communication and nfc/rfid tag reading are part of the android sdk. the only significant external library used is the zxing component [4]. ”zxing (“zebra crossing”) is an open-source, multiformat 1d/2d barcode image processing library implemented in java, with ports to other languages.” [5] the example below shows a code snippet capturing a 1-dimensional barcode with the smartphone’s camera. this is a good example of a task that at first was considered large and complex but later turned out to be quite straightforward, all things considered. var scanner = new zxing.mobile.mobilebarcodescanner(this); string barcode; await scanner.scan().continuewith(t => { if (t.result != null) barcode = t.result.text; }); if (!string.isnullorwhitespace(barcode)) setbarcode(barcode); rfid tag reading the viola android application reads rfid tags using the nfc reader on the smartphone. the choice of mobile platform for viola is the scope for another article, but it is worth noting that not every mobile platform with nfc capabilities has the right sdks in place to read the iso 15693 rfid tags. and if they do, the smartphone model of choice has to have the right chipset. the code sample below reads a rfid tag that is in the smartphone’s proximity, typically a couple of centimetres or closer. the code sends out a read command, receives the byte array response and sets the barcode property of the book order. /// <summary> /// this method is called when an nfc tag is discovered by the application. /// </summary> /// <param name="intent"></param> protected override void ontagpresent(intent intent) { var readcommand = new[] { (byte)0x00, (byte)0x23, (byte)0x00, (byte)0x28 }; var tag = intent.getparcelableextra(nfcadapter.extratag) as tag; var nfcvtag = nfcv.get(tag); nfcvtag.connect(); var rawresponse = nfcvtag.transceive(cmd); var response = encoding.ascii.getstring(rawresponse); var barcode = response.substring(4, 14); setbarcode(barcode); // sets the barcode on the order } printing with a portable printer for printing slips to put in the fetched books viola uses portable printers that the android device talks to over a bluetooth connection. the pairing with the printer is handled in the android settings menu, just like pairing a bluetooth headset and it is done before every fetch when the librarian chooses the printer to use during the fetch. when a book is found on the shelf and there is a barcode connected to the order, the librarian presses a button in the android application that sends a print command to the printer. the command is written in the zebra programming language [6], zpl, and converted to a byte array before being sent to the printer. the sample below shows a method that prints the text “hello, world” on a slip using a zebra printer. ///<summary> /// tells a bluetooth connected zebra printer to print "hello, world" on a piece of paper ///</summary> void printhelloworld(bluetoothdevice device) { var printonelinecommand = "^xa^lh30,30^f020,10^ad^fd{0}^fs^xz"; var zplcommand = string.format(printonelinecommand, "hello, world"); byte[] buffer = stringtoascii(zplcommand); var uid = uuid.fromstring(device.getuuids()[0].tostring()); using (bluetoothsocket socket = device.createrfcommsockettoservicerecord(uid)) { socket.connect(); await socket.outputstream.writeasync(buffer, 0, buffer.length); socket.outputstream.close(); socket.close(); } } fetching orders from external systems viola makes it possible for external systems to add orders through a pluggable architecture. every system that wants to add an order for a book has its own class that implements the icallslipsource interface, shown below. public interface icallslipsource { /// <summary> /// unique id for the external system /// </summary> string sourceid { get; } /// <summary> /// collection of "new" call slips. deduplication is not handled by /// the call slip source /// </summary> task<ienumerable<callslip>> getnewcallslips(); /// <summary> /// handles external changes in the system. /// </summary> void handleexternallychangedcallslips(); } through reflection [7] every icallslipsource implementing class in the assembly is loaded, as can be seen in the example below. since every external system is so different in its implementation, this solution works very well. /// <summary> /// returns every class implementing the icallslipsource interface in the assembly /// </summary> public static ienumerable<icallslipsource> callslipsources { get { return assembly.getexecutingassembly() .gettypes() .where(type => typeof(icallslipsource).isassignablefrom(type) && type.isclass) .select(type => newmethod(type)) .cast<icallslipsource>(); } } with every icallslipsource loaded, viola then periodically checks for and saves new callslip objects to its database. to make the icallslipsource implementation easier, viola handles deduplication so that no duplicate orders are saved. the following code sample shows the implementation of this method that is called from the windows service. ///<summary> /// fetches, deduplicates,and saves new call slips. ///</summary> public void fetchnewcallslips() { var newcallslips = callslipsourcefactory.callslipsources .selectmany(callslipsource => callslipsource.newcallslips.result); if (newcallslips.any()) { var deduplicatedcallslips = newcallslips.where(cs => !callsliprepository.exists(cs)); callslipfetcher.savenewcallslips(deduplicatedcallslips); } } example of icallslipsource implementation every icallslipsource is implemented in a different way. the example below shows the callslipsourceiller that fetches ill requests from iller for the library’s different branches. viola interacts with iller through its rest based api. the api delivers various streams of orders like incoming, outgoing, and cancelled requests. individual orders in the system can also be updated with new data. iller has multiple data format options and implements json, xml, pdf, and yaml. since the emergence of the dynamic type [8] and the httpclient [9] in the .net framework, the interaction with external web services with json data has been much more simplified. /// <summary> /// the interface method /// </summary> public async task<ienumerable<callslip>> getnewcallslips() { var dynamicrequests = return await illerhelper.getrequestsfromiller(illerstreamtypes.incoming); return dynamicrequests.selectmany(dynamicrequest => tocallslip(dynamicrequest)); } /// <summary> /// bootstrapping for the fetching /// </summary> private static async task<ienumerable<dynamic>[]> getrequestsfromiller(illerstreamtypes illerstreamtype) { var client = new httpclient(); client.defaultrequestheaders.add("api-key", violaresources.illerapikey); var branches = branchrepository.instance.readall().where(branch => !branch.disabled); var tasks = branches.select(branch => getrequestsfrombranch(client, illerstreamtype, branch.librarycode)); return await task.whenall(tasks).configureawait(false); } /// <summary> /// fetching new ill requests for each of the branches /// </summary> private static async task<ienumerable<dynamic>> getrequestsfrombranch(httpclient client, illerstreamtypes illerstreamtype,string librarycode) { var commandurl = violaresources.urltoillerapi + string.format(illerstreamtype.commandsegment, librarycode); var getstringtask = client.getstringasync(commandurl); var jsonresponse = await getstringtask.configureawait(false); var results = jsonconvert.deserializeobject<dynamic>(jsonresponse); return results.ill_requests; } /// <summary> /// converts dynamic objects to callslips /// </summary> private static callslip tocallslip(dynamic illrequest) { return new callslip { title = illrequest.title, publicationyear = illrequest.year, pages = illrequest.pages; // ..and so on }; } the first method above shows how the interface method is implemented chaining the call to iller and the parsing of the objects together. the returned task that is wrapping the collection of callslips tells us that this code is using the recently added async and await libraries [10]. this is a much more convenient way of writing asynchronous, multi-threaded, applications with the .net framework than before, and since the async/await libraries also have benefits for web applications [11], it is used in viola quite thoroughly. the getrequestsfromiller method bootstraps an httpclient with the right headers and reads the library codes of the branches from the database. it then calls the getrequestsfrombranch for each of the branches read from the database. getrequestsfrombranch is responsible for doing the actual call to iller and turning the returned json string into a dynamic object. the last method in the code above shows how straightforward the process of converting the dynamic json object to a statically typed callslip object is. it is only to match the different properties and the type conversion is handled automatically. no casting or parsing is cluttering the code. fetching orders from viola web service to the android application one important part of mvc is the web api type controllers [12]. they make it easy to put forward rest-style apis that can work for both single page application style web projects and for communication between systems. web api comes with many features out-of-the-box, like content negotiation, routing, and authorization. the sample code below shows the implementation of the web service that returns the orders that a librarian has chosen to fetch from the stacks, all sorted in the right order. to invoke the web service, the client has to point to the base url of viola followed with the name of the controller class. the api is then reachable through making an (authorized) http get request to https://url_to_viola/api/phonecommunication?userid=user_id ///<summary> ///exposes the list of callslips (book orders) that a librarian has choosen to fetch as a ///web service.the list is read from database and then sorted ///</summary> public ienumerable<callslip> get(string userid) { return callsliprepository.instance.read(userid, callslipstatuses.fetching) .orderby(callslip => int.parse(callslip.areadefaultsortorder)) .thenby(callslip => int.parse(callslip.callnumberdefaultsortorder)) .thenby(callslip => callslip.callnumber.replace(".",string.empty), new naturalcomparer()); } as the example above shows, no data transformation is taking place in code. that is all handled by the framework. from the programmer’s perspective, the native objects are exposed to the web as they are. json and xml data formats are implemented into web api by default. the data format delivered to the caller depends on the calling application’s headers. in the android application, the code calling the web service defined in this code sample is not that different from the callslipsourceiller example in the previous sample. the data received is not parsed in this method. it is just being downloaded as a string and saved to disc. the following code sample shows the invoking of the web service defined above. /// <summary> /// fetches callslips (book orders) from viola web service and /// saves them on the phone /// </summary> async void fetchcallslips(string url, string userid, string password, string domain) { using (var handler = new httpclienthandler { credentials = new networkcredential(userid, password, domain)}) using (var client = new httpclient(handler)) { var result = await client.getstringasync(string.format(url, userid)); file.writealltext(_filepath, result, encoding.default); } } future work implement a state machine “every developer has implemented a finite state machine at least once in his or her career.” [13] for the development team, viola should have been that first time. the team did smell the need quite early on, but being burned a few times overworking and overthinking other parts of the system, they over-practised the yagni [14] principle when it came to state transition and defining strict workflows. the absence of a state machine is now very obvious, and for both maintainability and extensibility reasons this is a much needed refactoring. switch from a relational database to a document database when the choice to use an object relational mapper (orm) for viola was made, the step from manual mapping between columns and object properties was considered a big leap forward. the team has since then acquired experience with a document database and the opinion is now that this type of data store is better suited for viola. implement some kind of queue like nservicebus or masstransit the need for persistent messaging is not as important an issue as the state machine issues, but if viola is to be scaled up or shared among other libraries, implementing a queue is something that should be done in order to make the system more trustworthy and robust. sharing viola putting all this effort into a system and not sharing it with others it makes no sense to most. the ambition is to release viola as open source, but it has to be done the right way. for example, the code cannot be released before a proper state machine is in place and before all hard coded localization has been removed. sending viola to the cloud viola is currently being prepared to be used by other library organizations. the first step is to enable it to run on windows azure and make it generic enough so that all swedish libraries using ex libris voyager can use it. so far everything looks promising, and the first end users should be able to test it in early 2015. splitting the workflows into two systems viola has been well received when being presented at conferences, and the main interest seems to lie in the stack fetching part. to be able to offer this part of the system in a convenient way, viola has to become more modular and development must be pushed in a direction that makes it more of a framework. notes [1] microsoft, microsoft news center [internet] [published 2014 nov 12] microsoft takes .net open source and cross-platform, adds new development capabilities with visual studio 2015, .net 2015 and visual studio online [visited 2014 dec 10]. available from: http://news.microsoft.com/2014/11/12/microsoft-takes-net-open-source-and-crossplatform-adds-new-development-capabilities-with-visual-studio-2015-net-2015-and-visualstudio-online/ [2] xamarin inc. [internet] xamarin platform. [visited 2014 dec 10] available from: http://xamarin.com/platform [3] microsoft, microsoft developer network [internet]cross-platform development with the portable class library [visited 2014 dec 10] available from: http://msdn.microsoft.com/enus/library/gg597391(v=vs.110).aspx [4] dick, j. [internet] zxing.net.mobile [visited 2014 dec 10] available from: https://components.xamarin.com/view/zxing.net.mobile [5] zxing [internet] xzing readme file. [visited 2014 dec 10] available from: https://github.com/zxing/zxing [6] zih corp, zebra technologies [internet] programming guide: for zpl ii, zbi 2, set-get-do, mirror, wml [visited 2014 dec 10] available from: http://www.zebra.com/content/dam/zebra/manuals/en-us/software/zpl-zbi2-pm-en.pdf [7] microsoft, microsoft developer network [internet] reflection in the .net framework [visited 2014 dec 10] available from: http://msdn.microsoft.com/enus/library/f7ykdhsy(v=vs.110).aspx [8] microsoft, microsoft developer network [internet] using type dynamic (c# programming guide) [visited 2014 dec 10] available from: http://msdn.microsoft.com/enus/library/dd264736.aspx [9] nielsen, h. f. httpclient is here! msdn blogs [internet] [visited 2014 dec 10] available from: http://blogs.msdn.com/b/henrikn/archive/2012/02/11/httpclient-is-here.aspx [10] [internet] asynchronous programming with async and await (c# and visual basic) [visited 2014 dec 10] available from: http://msdn.microsoft.com/en-us/library/hh191443.aspx [11] cleary s. 2014. introduction to async/await on asp.net. msdn magazine [internet] [visited 2014 dec 10] available from: http://msdn.microsoft.com/en-us/magazine/dn802603.aspx [12] microsoft [internet] learn about asp.net web api [visited 2014 dec 10] available from: http://www.asp.net/web-api [13] bajaj, s. 2001. design patterns: solidify your c# application architecture with design patterns. msdn magazine [internet]. [visited 2014 dec 10] available from: http://msdn.microsoft.com/en-us/magazine/cc301852.aspx [14] schach, s. 2007. object-oriented & classical software engineering. p. 58 about the author theodor tolstoy is the head of development at stockholm university library in sweden. blog: http://bibliotekarien.se twitter: @tedde subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – repurposing proquest metadata for batch ingesting etds into an institutional repository mission editorial committee process and structure code4lib issue 7, 2009-06-26 repurposing proquest metadata for batch ingesting etds into an institutional repository this article describes the workflow used by the university of iowa libraries to populate their institutional repository and their catalog with the data collected by proquest umi dissertation publishing during the submission of students’ theses and dissertations. re-purposing the metadata from proquest allowed the university of iowa libraries to streamline the process for ingesting theses and dissertations into their institutional repository the article includes a discussion of the benefits and limitations of the workflow described. by shawn averkamp and joanna lee introduction the university of iowa libraries has recently established an institutional repository (ir) for archiving a broad range of scholarly output including graduate student theses and dissertations. we expect the quantity of theses and dissertations submitted electronically to increase as the graduate college begins encouraging electronic submission over traditional print submission. therefore, we needed to create an efficient workflow for batch ingesting this content into our ir (iowa research online) [1]. the university of iowa currently uses the proquest umi dissertation publishing service to handle processing of both print and electronic theses and dissertations. to submit electronic theses and dissertations (etds) to proquest, students complete a web form and upload their documents. this data is later returned to the libraries via ftp as xml metadata, a pdf of the thesis or dissertation, and any supplementary files. we developed an xslt stylesheet to convert the proquest xml metadata to an upload-ready xml schema. while it is possible to harvest this metadata in xml from catalog records in worldcat, we chose to use the proquest metadata for a variety of reasons, foremost being that we can make the electronic access copies available before local marc catalog records are created, and then generate brief marc records for our local catalog. in this article, we will present a detailed description of this process including its benefits and limitations. our repository is hosted on digital commons a platform developed by bepress (the berkeley electronic press), but the workflow we will outline, summarized by figure 1, could easily be adapted by institutions using other repository platforms such as dspace [2]. as we developed this approach, we tried to integrate and streamline existing workflows and repurpose metadata as much as possible to avoid manual processes. for example, the proquest data was an attractive source because the metadata was robust and the files were readily available through proquest’s ftp delivery. the marc records we generate, while still requiring some manual review, could enhance the previous workflow for handling etds in which individual records were created manually and added to the local catalog. we include the annotated xsl files we developed for others to use and adapt[3]. figure 1: etd workflow process unzip proquest xml metadata files proquest submits compressed folders of etd xml metadata and pdf files to the university of iowa via ftp. the first step is to unzip all of the etds that need to be uploaded to iowa research online and make sure the pdf files are stored in a publicly accessible directory so that bepress can automatically pull them in during the batch upload. combine proquest metadata into one xml file proquest returns metadata about each etd in a separate xml file. because we needed to build one batch file describing all of the etds, the next step is to combine all of the xml files into one xml file. instructions for combining multiple xml documents into one xml document using microsoft windows. copy filepaths of files to be combined into a microsoft excel spreadsheet column (on vista, hold shift while right-clicking, select “copy path.”) in excel, remove the topmost common directory from filepaths. (use find and replace to remove, for example, “l:\etd.lib.uiowa.edu\“) add a “+” to each filepath. (in the next column, b1, enter concatenate formula: =concatenate(a1,”+”) then copy formula down the column for the rest.) copy column b into a word document (using paste special–unformatted text). remove line breaks in the word document (using find and replace to remove “^p“). delete last “+” from this text string. in command line, change to the desired drive. (for example, if you are working with files on the l drive, enter: “l:“) change to topmost common directory (the one you removed from the file paths in step 2). (for example, enter: “chdir etd.lib.uiowa.edu“) use the “copy” command to combine the files (enter: “copy“) then paste in the string you created in word, then a space, then name the destination file. (for example, “etdmetadata.xml“) open your destination file in notepad and using find and replace, remove all of the “<?xml…>” headers. paste the header back in at the top of the document. just below, enter a top-level “<xml>” element and close at the end with “</xml>“. transform to bepress schema using etd-proquestxml2bepressxml.xsl because of bepress’ minimal metadata schema, crosswalking between the two schemas was fairly straightforward.  elements that are not represented explicitly in bepress’ schema can be represented with a custom field element. while this solution offers flexibility in mapping from proquest to bepress, mapping from the custom fields in bepress to dublin core is currently somewhat limited; bepress is working on their metadata export tools and will be offering more options in the future. (see table 1 for complete crosswalk from proquest to bepress schema.) proquest (source) bepress (output) format/value notes xml documents root diss_submission document item diss_description/diss_title title outputs first letter of each word in caps diss_authorship/diss_author/diss_name/diss_surname diss_authorship/diss_author/diss_name/diss_fname diss_authorship/diss_author/diss_name/diss_middle authors/author/lname authors/author/fname authors/author/mname authors/author/institution the university of iowa diss_content/diss_abstract/p abstract/p diss_description/diss_dates/diss_comp_date publication-date iso 8601 (yyyy-mm-dd) completion date from proquest record. only year is set to display in our repository, but since bepress requires the full date, we’ve defaulted to jan 1. diss_description/diss_categorization/diss_keyword keywords/keyword diss_content/diss_binary fulltext-url filename concatenated with local location on server staging area diss_description/diss_degree fields/field/@name=”degree_name”/value diss_description/diss_institution/diss_inst_contact fields/field/@name=”department”/value diss_description/diss_advisor[1]/diss_name/diss_fname diss_description/diss_advisor[1]/diss_name/diss_surname diss_description/diss_advisor[1]/diss_name/diss_middle fields/field/@name=”advisor[1]”/value first three advisors captured label item node position + total of etds already uploaded forces bepress article id on each etd in order to construct the access url before uploading table 1: proquest schema to bepress schema metadata mapping now we can transform the xml file from proquest’s schema to bepress’s general schema for importing material using the transformation we developed, etd-proquestxml2bepressxml.xsl [4]. there are a few aspects of our xsl file worth noting. we use a function to build the filepath of each pdf for the fulltext-url field based on the filename of each etd and the root directory where the file is saved. during the batch uploading process, bepress will use this path to pull in each file. our code also normalizes the degree_name field to keep those values consistent (for example, m.a. will change to ma; phd will change to phd) and uses a function to change the all-caps title field values and the variable name field values of the proquest metadata to title case (only the first letter of each word is capitalized). in order to control and predict the future url of each etd when it is uploaded, we forced incremental integers onto each record in bepress’s label field. for example, an etd with a label value of “75” will be uploaded to http://ir.uiowa.edu/etd/75. bepress automatically generates a label for each document, but we chose to force the label so that we can easily generate the url of each record for local cataloging purposes in a later transformation. each time we transform a new batch of etds to bepress’s schema, we must change the base integer in the label field of etd-proquestxml2bepressxml.xsl to ensure that each etd receives a unique label and thus, a unique url. for example, if there are 220 etds loaded into iowa research online, we must start the next batch at integer 221. it is possible to create a unique label for each etd without relying on a single base integer that must be reset before each transformation; the bepress schema supports any string in this field. for example, combining an integer and another field, such as the author’s last name, will likely generate a unique value for the label field and result in a url such as http://ir.uiowa.edu/etd/chang13. however, on bepress’s recommendation, we chose to use an integer value alone to make the etds easier to manage in the administrative interface; sorting by label reflects the order in which documents were uploaded. it also keeps the format of the resulting url consistent with other documents in our repository (http://ir.uiowa.edu/series-name/integer). output to tab-delimited file using etd-bepressxml2txt.xsl to review and revise metadata to make it easier to review the transformed metadata before batch-loading, we developed another transformation to reformat it as a tab-delimited file. opened in a spreadsheet, it is easy to check the metadata for errors and make associated changes in the transformed xml. upload batch file to repository now we simply upload our batch file to iowa research online. transform to marc21-xml using etd-bepress2marc21xml.xsl now the etds are available in the repository and discoverable in the local catalog through a pipe from iowa research online, but they do not have marc records in the local catalog. at this point, the university of iowa plans to continue building marc records for the etds, in part to easily maintain them in oclc worldcat. to make local cataloging easy, we developed another transformation to reformat the metadata from our final bepress batch xml file to marc21xml. because we forced a label on each etd during the transformation to bepress’s schema, we are able to predict and build the final url of each etd in our marc21xml. create marc records using marcedit the final step in our workflow is to use an existing transformation available from marcedit, a free program developed by terry reese at oregon state university, to generate marc records from our marc21-xml [5]. benefits the main benefit of using the proquest metadata is that we are able to provide public access to etds sooner than we would have been able to by harvesting oclc metadata post-cataloging. ir metadata records created from a proquest schema to bepress schema transformation, while imperfect, can serve as access records while the etds await local cataloging for the library catalog.  although marc records do not yet exist in the local catalog, a pipe from our bepress ir to our library’s federated search system (ex libris’ primo) allows etds to be discoverable in the local collection environment. when etds eventually receive local cataloging treatment, minor errors in title case and special characters can be corrected in the ir metadata [6]. (while the creation of duplicate records in both the ir and the local catalog may seem redundant and potentially confusing to the federated search user, current local cataloging guidelines do not allow for the substitution of an ir metadata record for a full marc catalog record.) to expedite local cataloging of etds, we created a transformation that could be used to automate some of the process of brief record creation. (this step has not yet been approved for our local cataloging workflow, but we include it for the benefit of those wishing to streamline their own workflows.) proquest metadata that has been transformed to the bepress schema can then be transformed into marc21xml, which can in turn be transformed into marc-21 brief records using the marcedit tool (developed by terry reese at oregon state university, see resources, below). brief records were previously created manually. using local thesis and dissertation cataloging guidelines and the networked digital library of theses and dissertations’ (ndltd) etd-ms interoperability metadata standards, we created a transformation to convert our bepress upload xml (after manual edits have been completed) to marc21xml [7]. this transformation captures most datafields, but a few must be entered manually, specifically physical description (300ab) and topical subject headings (650ax). also, the university of iowa libraries adds several local fields that must be populated manually. as noted earlier, the title statement (245:10abc) case must be normalized manually and the abstract (520) must be checked for mistranslated diacritics. using marcedit, a free application for editing and transforming marc records, we convert the marc21xml to a marc file (.mrc). this file can then be imported into the integrated library system (ils) where certain fields are populated automatically, and the records can be cleaned up and fleshed out. bepress marc format/value notes ldr ^^^^^ntm^ 22^^^^ ua 4500 005 auto-generated in aleph 006 m^^^^^^^^d^^^^^^^^ additional fixed data: electronic resource 007 cr^n|||||||||| physical description fixed field: electronic resource 008/15-17 xx publication place: no place of publication – i.e. unpublished 008/23 s form of item: electronic 008/24-27 m nature of contents: theses 040ac nui cataloging source authors/author/lname authors/author/fname authors/author/mname 100:1a [lname], [fname] [mname] main entry title 245:10ab title is split into title and subtitle at ‘:’, ‘?’, or ‘?:’ authors/author/lname authors/author/fname authors/author/mname 245:10c by [fname] [mname] [lname]. usually consistent with title page 245:10h [electronic resource] general material designation: electronic resource publication-date 260c yyyy. fields/field/@name=”advisor[1]”/value fields/field/@name=”advisor[2]”/value fields/field/@name=”advisor[3]”/value 500a thesis supervisor: [advisor[1]] thesis supervisor: [advisor[2]] thesis supervisor: [advisor[3]] thesis advisor(s) fields/field[@name=’degree_name’]/value 502a thesis ([degree name]) –university of iowa, 2008. dissertation note abstract 520:3a 520:8a paragraph breaks are removed. local guidelines allow 2000 characters for 520:3a field. remaining characters are entered into 520:8a fields. 538a mode of access: world wide web. system details notes (as recommended by ndltd.) 538a system requirements: adobe reader. fields/field[@name=’advisor1′]/value fields/field[@name=’advisor2′]/value fields/field[@name=’advisor3′]/value 720a [advisor[1]] [advisor[2]] [advisor[3]] added entry – uncontrolled name. (as recommended by ndltd. univ. of iowa libraries uses 700:10, local added entry field, instead.) 720e advisor. label 856:40u http://ir.uiowa.edu/etd/[label] concatenation of ir collection directory path and bepress xml label table 2: bepress schema to marc21xml metadata mapping limitations there are some limitations to our workflow that require manual corrections before and after the batch file is uploaded. the first involves the title field. when students submit a thesis or dissertation, the thesis manual of the graduate college dictates that the title must be formatted in all caps. the proquest metadata inherits this convention, so we generally have all-caps title fields to work with. we chose to reformat them as title-case (only the first letter of each word is capitalized). from a cataloging perspective, the resulting output is not ideal, but since proper nouns are not differentiated in the all-caps of the original metadata, using title-case was a good, scalable compromise. a side-effect of this decision is that any acronyms in the titles will be incorrect (for example, dna is changed to dna). as a final step before uploading, we make manual corrections to the titles in the xml batch file. this manual process can be eased by reviewing a tab-delimited text transformation of the metadata in ms excel (etd-bepressxml2txt.xsl). in a future revision of our style sheet, we would like to address this problem by creating a list of common acronyms and checking the titles against them. any strings in the title that match an acronym on the list would remain capitalized. the source data for publication-date field is a bit problematic, too. we chose to map the publication-date field from the diss_comp_date field. however, this proquest field generally only contains a year (formatted as yyyy).  bepress requires the publication date to include month, day, and year and conform to iso 8601 (yyyy-mm-dd), so our transformation adds “01” for month and date (2008 becomes 2008-01-01). other limitations are due to bepress’ schema. bepress’ batch loading feature only supports the latin-1 character set, so any characters outside of that set need to be corrected after the etds are uploaded. though you can add many custom fields, currently bepress’ schema cannot be extended to include the discipline in which an etd should be categorized. without this discipline information, the etds will not be visible when users browse by discipline, either on the homepage of the repository or in bepress’ cross-repository researchnow! portal. therefore, each etd must be manually categorized within the administrative interface of the repository. bepress is planning to support discipline mapping in a future revision of their batch-loading schema. in addition, for bepress fields that we have limited by a controlled drop-down list (such as department), any values with special characters such as ampersands (for example, electrical & computer engineering) will not match correctly during the batch-loading process. to prevent this, we asked bepress to replace all ampersands in controlled fields to “and” and we use a function in our transform (etd-proquestxml2bepressxml.xsl) to change the “&” symbols in any controlled value fields to “and.” tools markup language: xml 2.0 xml editing and transformations: <oxygen/> xml editor 10.0 processor (via <oxygen/>): saxon-b 9.1.0.3 combining multiple xml documents: notepad; microsoft office word 2007; microsoft office excel 2007 viewing tab-delimited files: microsoft office excel 2007 generating marc files (.mrc) from marc21xml: marcedit [5] other resources there are several other resources that might be helpful for institutions interested in repurposing etd metadata for their repositories. michael witt and mark newton at purdue university have produced an outstanding tutorial about transforming endnote metadata, “preparing batch deposits for digital commons repositories [8].” on iuscholarworks, randall floyd from indiana university libraries describes a workflow for ingesting proquest/umi metadata and etds into a repository built on dspace [9]. conclusion in the future, proquest may change the way they structure or deliver their metadata. other factors could also change, such as how students are required to submit etds and how bepress wants data structured for import. while at some point we may need to repurpose metadata from other sources or revise our transformations, we have developed a successful workflow for efficiently ingesting etds. in addition, developing our transformations was a great introduction to xsl; we are now applying these skills to target other digital resources for our repository and to repurpose metadata for other digital library applications. links iowa research online (iro) — http://ir.uiowa.edu bepress — http://www.bepress.com/ download the university of iowa libraries’ xslt files bepress document-import schema — http://www.bepress.com/document-import.xsd marcedit — http://oregonstate.edu/~reeset/marcedit/html/index.php ex libris primo — http://www.exlibrisgroup.com/category/primooverview ndltd — http://www.ndltd.org/standards/metadata/etd-ms-v1.00-rev2.html witt, m. & newton, m. (2008). preparing batch deposits for digital commons repositories. purdue e-pubs. library research publications. paper 96. http://docs.lib.purdue.edu/lib_research/96/ automated electronic thesis and dissertations ingest. iuscholarworks. http://wiki.dlib.indiana.edu/confluence/x/01y acknowledgments we would like to thank wendy robertson, digital resources systems librarian, university of iowa libraries, for her contributions to the etd project, her guidance on cataloging practices and local workflows, and her editorial suggestions. about the authors shawn averkamp (shawn-averkamp@uiowa.edu) and joanna lee (joanna-lee@uiowa.edu) are digital projects librarians at the university of iowa libraries digital library services. subscribe to comments: for this article | for all articles 3 responses to "repurposing proquest metadata for batch ingesting etds into an institutional repository" please leave a response below: kate, 2011-04-27 why are you coding theses in the fixed field as “manuscript”? kate, 2011-04-27 also, i notice you are not putting a date in the fixed field. kate, 2011-04-27 also– 008 15-17 is not just for publication, but also for place of production, etc. even if you don’t like to use place of production, is not putting these in an institutional repository publishing them to the web? leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – programming poetry: using a poem printer and web programming to build vandal poem of the day mission editorial committee process and structure code4lib issue 45, 2019-08-09 programming poetry: using a poem printer and web programming to build vandal poem of the day vandal poem of the day (vpod) is a public poetry initiative led by the center for digital inquiry and learning (cdil) at the university of idaho library. for four academic years vpod has published contemporary poems daily in collaboration with award-winning poetry presses and journals. this article details the project’s genesis and history, focusing on two aspects of the project: 1) the customized wordpress site, css, and plugins that enable the layout, publication, and social media promotion of the poetry and 2) the innovative means we have developed for promoting the site using receipt printers. the latter portion includes details and code related to two different physical computing projects that use receipt printers–one using a raspberry pi and the other using a recycled library circulation printer– to print individual vpod poems on demand. by evan williamson and devin becker introduction the vandal poem of the day (vpod) is a public poetry project based at the university of idaho library bringing contemporary poems to the campus community and beyond. the project began with a conversation between poets devin becker and keetje kuipers (both winners of the a. poulin, jr. poetry prize) during the 2015 association for the study of literature and the environment (asle) conference held in moscow, idaho. kuipers established aubie’s poem of the day at auburn university [1] in 2014 and asked if there would be interest in replicating the project at the university of idaho. aubie’s poem of the day used a wordpress site to publish one contemporary poem per day from a prominent poetry press, featuring copper canyon press (2014/15) and boa editions ltd (2015/16) during the project’s two years. kuipers worked with librarian jaena alabi to setup the project using library server space and resources. knowing that becker was a digital librarian in addition to being a poet, she thought the model would be a good fit for the university of idaho. becker worked with colleague alexandra teague, a poet and university of idaho english faculty, to write a proposal for the vpod project, securing grant funding from the idaho humanities council (ihc) and a match from the university’s college of letters, arts, and social sciences (class) [2]. the funding from class and the ihc supported marketing materials for outreach and a visit for a featured poet to read as part of the first year’s events. university of idaho library provided technical support, as becker and librarian evan williamson setup the vpod web site and workflows to facilitate the publication of the poems. many of the initial hurdles for establishing the project were already overcome by kuipers. she had worked with the presses to collect the featured poems for the first two years [3] and to receive permission to reprint the poems on a website [4]. thus, once becker worked with the permissions company, inc. to update the permissions agreement, vpod had a trove of content (poems) and metadata (poet’s name, book in which the poem appeared, etc.) already selected for publication on the new site. wordpress site and collaborative workflow setup poems published as jpg vs. poems published as text aubie’s poem of the day published each poem as a jpg image, stored as media files on their wordpress instance. this has been a common practice for the online publication of poetry during most of the 21st century for at least two reasons: 1) the poems were less easily copied and pasted and as such, plagiarism and copyright infringement were thought to be limited; 2) the image of the poem preserved the font and layout of its original publication. publishing the poems as images also evades the complex and interpretive work necessary to mark up the poems to display correctly in html. additionally, kuipers and alabi were invested in the use of qr codes for promotion and access to the site. they were hopeful that many of their users would be arriving at the site via their phones, and kuipers was particularly invested in maintaining the fidelity of the line breaks of the poems. so she and alabi decided, quite understandably, that delivering the poems as images rather than text would be the best option going forward. figure 1. fc.d. wright’s poem “gift of the book” as featured in aubie’s poem of the day (left) and vandal poem of the day (right) (enlarge) after reviewing the content, becker and williamson decided to break with that convention and publish the poems in a text based workflow. locking the poems in image format presented a variety of limitations that vpod hoped to avoid. first, depending on screen size, the images often appeared pixelated or uncrisp, delivering the poems in a slightly unclear way and reducing the readability (see figure 1). second, the images were not necessarily mobile friendly because some of the aspect ratios did not scale well to the wide variety of screen sizes that modern websites seek to support. third, images of text are not accessible to screen readers and other tools used by people with disabilities unless they contain a full transcript as “alt” attribute. finally, becker and williamson intended to re-use the poems’ text outside of the website itself to explore opportunities for outreach and analysis, thus clean marked up text would enable new possibilities beyond the image files. the poems were provided as pdf files with each filename containing the author name, book title, and poem title, with the poem image as a single page. to create plain text, the image pdfs were ocred using adobe acrobat batch processing and the text was extracted using “pdftotext” from xpdf tools [5]. with the poem’s lines in plain text files, williamson developed a series of batch operations to streamline preparation of the text as far as is possible before loading into wordpress and handing off to editors. to set up the basic html markup that would represent the lines of poetry on the web, a series of find and replace operations enclosed each line of poetry in <p class=”line”></p>, and replaced blank lines with an empty paragraph ( <p></p> ). next, a script extracted the author name and book title from the filename, and wrote a “by line” at the top and a “bookinfo” line at the bottom of each file. this batch of files was then combined into a single xml file following the format used to migrate wordpress sites [6]. each poem was assembled as a draft post, with <title>, and the prepared text data inserted in the <content:encoded> element. this file was then imported into the wordpress site, automatically inserting each poem as a draft post [7]. the basic setup of vpod is that each poem is a wordpress “post”, with the author’s name as a “category”, providing a data set that can be iterated over and sorted in a variety of ways to provide access. wordpress “pages” are used to provide contextual means to enter the data, i.e. via the featured poets. each poem’s post provides links to the author’s bio, other vpod poems, the book from the publisher, and related books in the university of idaho library collection. using wordpress’s role management features, each post/poem was assigned to an editor involved in the project to review and polish. each editor checked the ocr text against the original poem image and carefully added markup to represent the original layout of the lines. editors used the “code view” to work on each poem, as the “visual editor” breaks the manual layout by adding extra markup. since many of the editors had no previous experience with html, some found looking at the poems in this way to be intimidating. however, it became an opportunity to learn markup concepts using a very small subset of tags (just <p> with different classes) while deeply engaging with the text and reflecting about representation of poetry from the print book, to images, to digital screens. wordpress setup and poem layout with the poems transformed into viable text data and loaded into the system, becker worked on setting up the wordpress instance to automate delivering the daily poems while minimizing regular maintenance. first, becker looked for a wordpress theme that could be customized to provide the desired functionality and presentation. vpod needed a minimal look and feel, free of distracting designs so that the poem of the day would be the site’s central feature. becker searched for a theme that would be able to deliver those requirements and discovered the founder theme by ben sibley, creating a child theme from that theme in order to maintain his customizations. vpod also needed to account for linking between the poems, the books that the poems came from, the press’s site, and the biographies of the poets who wrote the poems. this was accomplished by creating a page of biographies with anchors for direct links, creating wordpress “categories” for the authors collecting all poems written by each author, and creating wordpress “tags” for the poems indicating the author’s name and twitter handle (if it exists). the information included in each poem’s tags can be used by plugins to generate further links–for example, in the automatic tweet sent when a new poem is published by the plugin wp to twitter. wordpress’s famous plugin ecosystem provided many out-of-the-box solutions that were easily configured to the vpod’s needs. however, creating the site’s front page required some lower-level php customization. becker tweaked the index page php function to display a single post on the front page that then led, via a “read more poems!” link, to an archive of all the posts that are categorized as “poems.” the archive required an offset that allowed the typical wordpress php functions to be slightly adjusted so that the first poem in the “poems” archive wouldn’t appear (as it would already be on the front page) and the pagination would then be adjusted going forward by one post. without this last adjustment, poems/posts would repeat when users went to the next page of poems/posts in the archive. function special_home_pagesize( $query ) { if ( is_admin() || ! $query->is_main_query() ) return; if ( is_home() ) { // display only 1 post for the original blog archive $query->set( 'posts_per_page', 1 ); return; } if ( is_category( 'poems' ) ) { // don't display current poem of day //first, define your desired offset... $offset = 1; //next, determine how many posts per page you want (we'll use wordpress's settings) $ppp = get_option('posts_per_page'); //next, detect and handle pagination... if ( $query->is_paged ) { //manually determine page query offset (offset + current page (minus one) x posts per page) $page_offset = $offset + ( ($query->query_vars['paged']-1) * $ppp ); //apply adjust page offset $query->set('offset', $page_offset ); } else { //this is the first page. just use the offset... $query->set('offset',$offset); } return; } } add_action( 'pre_get_posts', 'special_home_pagesize' ); this code was taken from one of the supporting documents on wordpress’s site itself, highlighting another benefit of wordpress: the excellent documentation and community forums that proved immensely helpful for understanding how and where to deploy bits of php to make the site function as desired. so many of the world’s websites are published using wordpress that finding code and plugins to incorporate is often just a well-phrased google search away. as opposed to the wordpress instance, the workflows and markup required to properly publish the poems did provide some unique challenges for the project, namely how to get the poems to read well on screen while maintaining some indication regarding the line and stanza structure, and how to incorporate non-technical staff and student workers into the workflow for marking up and uploading individual poems. the former problem took a small amount of css to accomplish. we code all lines of each poem as a <p class=”line”> so that each line follows these style constraints: .line { text-indent: -32px; margin: 0 0 0 32px; padding-right: 16px; line-height: 28px; } the “-32px text indent combined with a left-margin of 32px enable a hanging indent effect when a line breaks due to the size of the screen in which it’s held. this hanging indent method is a traditional approach to formatting poetry within long lines on a page so that a reader can still observe where each line begins and ends. this convention is especially prevalent in great book series and other similar collections that gather a variety of poetry into smaller-paged books. this problem of breaking lines in a way that still communicates the author’s intended breaks is also difficult for contemporary poetry presses and publishers [8]. another challenge with the display of poems is the use of unique and irregular spacing at the beginning and within lines. this space is often manually coded using the html entity “&nbsp;”. to provide a more flexible means of markup, vpod utilizes css classes indicating different widths of padding to the left of the required word or line given in em units [9]. for example, .em1{padding-left:1em;}, .em2{padding-left:2em;}, etc. this provided a simple set of classes that could be easily applied by editors with different levels of experience with html/css to lay out the poems. following the conventions allowed for quick adjustments when eyeballing the layout based on the original poem image, as an editor could try a “em4,” for example, realize that a bit more spacing was needed, and adjust the class to “em5.” here’s an example of a printed poem using theses styles: figure 2. c.d. roger reeves’ poem “against its own ringing, as featured on vandal poem of the day below is the code that creates the effect: <p></p> <p class="line">more than once i’ve asked a body to scatter</p> <p></p> <p class="line em6">and have received that scattering.</p> <p></p> <p class="line">more than once, a scar. more than once a scare-</p> <p></p> <p class="line em6">crow watching the corn of another’s body</p> each break is a closed paragraph element. each line is classed as a “line,” and any indented line is indented with the “em6” class, which assigns it a padding of “6em.” benefits of using wordpress over the four years vpod has been running, we have had undergraduates, graduate students, faculty, and staff work on marking up and updating the poems within the wordpress site. these contributors’ familiarity with html and css has varied considerably. by establishing the conventions for layout and classes noted above, and providing training on the process, we were able to provide learning opportunities to collaborators looking to learn basic concepts of html/css. this was not a benefit we anticipated, but we now often use the site and the tasks of laying out poems within it as an html/css introduction for those unfamiliar with the languages. wordpress’s ability to define users and user roles also enabled this collaborative work, and the cms’s gui interface provided a user-friendly environment for these collaborators to learn these skills, check their edits, and then see the results of their work published online. in addition to allowing a means for collaboration and learning, wordpress’s innate ability via plugins to schedule the posts/poems publication and announce these publications with a tweet were clear benefits of working with the cms. once the poems were fully loaded into the wordpress instance, an administrator would use the drafts scheduler plugin to schedule a poem to be published each weekday morning during the academic year. each day somewhere between 6 and 7 am pacific time, a new poem would be automatically published on the site. at this time, the wp to twitter plugin would automatically send out a tweet announcing the new post. as tweets automatically include the poet’s name and twitter handle (if available), this enabled attention and retweeting through the poets own networks and was one of the more successful promotional endeavors we used. the final benefit of wordpress resulted from an innate feature in all wordpress sites. the two plugins detailed above saved us a significant amount of time, as once the poems were scheduled to be published and tweeted, little other maintenance was required. most years the majority of the work would be complete on the site for the full year by october or november, and the site and twitter presence would essentially run itself. this provided us time to, in the now canonical phrase of stephen ramsey, “mess around” with a variety of other uses for the poems, using wordpress’s built in xml export feature to pull out all the edited and marked up poems and associated content as structured data for use in other tools and features. poetry to the people via kioskslides and poembot printers although the grant outlined specific plans for traditional marketing activities, such as providing print promotional materials, planning events, and using the qr codes on posters across town to promote access to the vpod site, new ideas and unexpected possibilities arose as we worked with the poems and collaborated with each other. at one point multiple members of the team shared an article about a short story printer in grenoble, france [10]. williamson had previous experience using raspberry pi and mentioned that it was possible to create a diy version [11]. suddenly our enthusiasm for the idea bloomed into a new avenue for vpod outreach. by printing the daily poems in an ephemeral form, we could literally get poetry into the hands of our community via a mundane object of everyday life, thus using receipts to make a physical connection between people, poetry, and the vpod website. to implement this idea we worked through two versions of a poetry printer, first as an add on to a library information display called kioskslides, and second as a standalone mini-printer named poembot. kioskslides when vpod began, the university of idaho library’s first floor was closed for renovation and a large television welcomed visitors at the temporary entrance using a powerpoint slide deck on a loop to display updates, news, and marketing. we saw an opportunity to integrate vpod and create a more interactive display featuring a daily poem. however, several physical limitations constrained the project: 1) the display was run from an old computer; 2) the computer was offline; 3) the computer was locked with no admin access allowed, and 4) we had, at that point, no budget. necessity, nevertheless, is the mother of invention (or minimal computing), and the kioskslides project became a simple, pragmatic response to our initial limitations [12]. first, the display’s existing powerpoint slide show was exported as images. each image was added to a template html page that centers and contains it as a background to create a “slide” that is responsive to screen size. this set of html pages share a javascript file that advances the slides on a timer (i.e. loads the next page) following the order given as a variable. williamson then recreated the look of the vpod home page (normally dynamically generated by wordpress) in an offline static version. since the static page can not query the online vpod or a database, each poem is represented by a hidden div with an id of the date it should appear. javascript checks the current date and displays the appropriate poem of the day–or during breaks, displays a random poem from the archive. this html slide deck was loaded on to the display computer and set to run on chrome browser in “kiosk mode” (instructions are available in the repository). this locks chrome into full screen and hides the user interface, thus providing a display much like the original slide show. a mouse was added to the display computer to allow interaction from passers by, a click advances slides or prints a poem. to print poems, an old “point-of-sale” printer originally used for library loan receipts was rescued from the trash. most librarians will be familiar with these printers which make quite a bit of noise–in this case attracting attention to poetry entering the world on paper. since the computer had the correct printer drivers installed and chrome browser was set with options “–kiosk-printing –disable-print-preview” to disable dialog inputs, simply adding onclick=”window.print()” turns the html slide into an interactive poem printer. the print output is handled by a media=”print” stylesheet (simply hiding the decorative elements of the page so the poem can print cleanly), and chrome natively converts the html to a printable format reducing many of the potential difficulties. adding this simple printer to the information display created an incentive for people to interact and pause at the slide show, getting more attention for vpod as well as for general library marketing information. figure 3. photograph of the kioskslides poetry printer with promotional materials. (enlarge) poembot the second poetry printer was the poembot project (https://github.com/evanwill/poembot) which brings together a mini thermal receipt printer, raspberry pi, python, and poetry in a sturdy wooden case to print poems wherever there is need. in addition to creating an interesting artifact to promote vpod, this project was an opportunity to gain experience with physical computing to complement a new “makerspace” opening in the ui library. it delivers printed poems, but also acts as a teaching object starting conversations about microprocessors and programming, and demonstrating physical computing in action. poembot’s basic setup is based on numerous other projects that make use of raspberry pi and a cheap thermal printer [13]. these projects, and the poembot, are enabled by the growing abundance of user-friendly devices, supportive distributors, and online tutorials that lower barriers to implementing custom hardware. poembot started from the parts of an “adafruit iot printer” kit featuring a mini thermal receipt printer, a button with led, and a raspberry pi 2 [14]. although our final assembly differs from the kit, we chose to keep our parts list very close, since the great documentation provided by adafruit and easy availability of parts helps lower the barriers to reuse. figure 4. poembot opened to show raspberry pi 2 with “squid” wiring. (enlarge) to connect the components, most other projects solder the circuit together or use a breadboard. however, since poembot is a teaching object, williamson wanted to streamline assembly, aiming to build in the flexibility to change things up, swap parts out, and experiment [15]. inspired by pi guru simon monk’s “squid” concept [16], williamson soldered jumper wires onto the component leads, so they can be easily plugged and unplugged in a modular fashion. to make connecting the squids to the raspberry pi easier, williamson created a template marking the connection layout that fits on top of the gpio pins, inspired by simon monk’s “raspberry leaf” concept [17]. rather than a laser-cut acrylic case, as found in the adafruit kit, williamson built a wooden case from 1” pine boards that provides a hand-made contrast to the electronic components, in addition to sturdy protection. figure 5. poembot with a freshly printed poem. the wooden box contains the raspberry pi and the receipt printer.(enlarge) poetry data with a printer ready, poembot needed poetry data. luckily, vpod’s clean markup and wordpress’s export feature made accessing poem text possible. the export xml was parsed and transformed using openrefine, reshaping the poem posts into a csv of metadata and text. the html markup of the poem text was stripped away, replacing the various “em” indent styles with an equal number of spaces and breaks with “\n”. the poems were filtered by character count to remove longer poems, since the largest would require several feet of receipt paper to print. this data was exported from openrefine as a csv, then converted to the obscure character encoding required by the printer using a text editor. the poem data is stitched together with the printer and raspberry pi using a python script. luckily, adafruit provides a python library to interact with the thermal printers it sells and excellent documentation of the library’s usage [18]. williamson adapted their example implementation to choose a random poem to print when the button is pressed, wrapping the lines with a hanging indent to mirror the original form of the poem (as much as is possible on a two inch wide piece of paper). each of these actions are simplified by using python standard libraries, as the poem csv is loaded using “csv”: with open('vpod_22l_800char.csv') as csvpoems: allpoems = list(csv.reader(csvpoems, delimiter=',')) selected using “random”: randpoem = random.choice(allpoems) and wrapped with the appropriate hanging indent using “textwrap”: wrappedpoem = "" for line in randpoem[3].splitlines(): wrappedline = textwrap.fill(line, width=32, subsequent_indent=" ") wrappedpoem += wrappedline +"\n" keep in mind that raspberry pi does not have an internal clock, so the system does not know the date unless it connects to the internet. originally, the poembot connected to wifi, however, the complications of campus networks with high security and traveling made this become a barrier to use rather than a benefit. instead, poembot is simplified to function offline. to update the poem selection or add promotional messages to the print outs, williamson removes the raspberry pi’s sd card, plugs it into a linux computer, and directly edits the files–a considerably simpler and more pragmatic solution than using ssh to remotely log into the device. poembot in action poembot proved a successful hook for vpod both on campus and at various conferences. we released the poembot officially at our inaugural reading event, which used funds from our idaho humanities council grant to bring in the poet roger reeves for a reading and consultations with mfa students. for the event, the poembot was programmed to print a random selection from the nine poems featured on the vpod site by reeves. the event was attended by over 80 people, and reeves lived up to his reputation as a phenomenal reader. we were nervous to see what a poet thought about seeing his poems printed on receipts in this way, but these fears were soon calmed by reeves’ enthusiasm. the audience members also appreciated the poembot, with most members coming up to print a poem before or after the reading. several students even had reeves sign one of the poembot-printed poems in lieu of his signing a book [19]. poembot has visited several other readings since that time, starting conversations with poets and audiences. the poembot also travelled, via luggage, to conferences in los angeles (association of writers and writers programs conference, 2016) and milwaukee (digital library federation forum, 2016). the authors were nervous about bringing the machine in their luggage, as the internal wires and raspberry pi bore a strong resemblance (presumably) to a bomb. fears notwithstanding, the actual machine was well received at both conferences. becker brought the printer to a presentation at awp about aubie’s poem of the day, after which it resided at the fugue literary journal table, along with some of the vpod marketing materials. at dlf, becker and williamson presented on the vpod project and the poembot specifically and then set it up in the pfister hotel conference center hallway. conference participants were intrigued by the box and impressed by the quality of the poems [20]. the printing of these poems were often conversation starters about poetry and/or technology, as we were invited to talk about each, leading to discussions about python, raspberry pi, poetry, or a combination of the bunch. since then, poembot has been deployed for a variety of special events on campus. we created simple instructions enabling anyone to set it up and start printing poems. the machine spent time on library circulation desks in honor of national poetry month [21], visited the opening of a new digital humanities center [22], and was repurposed to print contact information in our makerspace [23]. the code and documentation on github has led to connections with people across the country and the project has been adapted and used by students and faculty at other institutions [24]. future data starting from a fairly simple concept of creating a daily poetry website, vpod grew into an opportunity to develop skills, think about poetry as data, and build new connections with people–moving away from marketing into programming/outreach. focusing on curating the poems as clean, consistently marked up text enabled opportunities to interact with the poems in unanticipated ways. from poems displayed on huge screens to printed on tiny receipt paper, from text analysis to machine learning, this text dataset opens new possibilities, representing a rich corpus we are just beginning to explore [25]. furthermore, the project has led to unexpected connections between writers, librarians, students, faculty, and others, creating opportunities for learning, dialog, and reflection. as we end our fourth year of vpod and plan for our fifth, we look forward to further exploring the text data, connecting with more people, and reading great poetry (daily). about the authors evan peter williamson is the digital infrastructure librarian (assistant professor) at the university of idaho library, working with data & digital services to bring cool projects, enlightening workshops, and innovative services to life. despite a background in art history, classical studies, and archives, he always manages to get involved in all things digital. his recent focus has been on data driven, minimal infrastructure web development, currently embodied in the collectionbuilder project. devin becker is the director of the center for digital inquiry and learning (cdil) and the head of data & digital services at the university of idaho library, where he directs and maintains the library’s digital initiatives program. becker is also a writer. his first collection of poetry, shame | shame, was selected by david st. john as the winner of the thirteenth annual a. poulin, jr. poetry prize and was published by boa editions ltd in 2015. his most recent web project, ctrl+shift, provides visualizations and analyses of series of interviews he conducted with prominent poets across the country. references 1 aubie’s poem of the day, http://poetry.auburn.edu/ 2 this project and grant proved extremely important for future collaborations between the library and class as it served as a model for the type of collaboration promoted by its digital humanities center, the center for digital inquiry and learning (cdil), which, now in its third year of official operation, continues to enable collaborative projects between class faculty and students and the library. 3 aubie’s poem of the day shut down after kuipers resigned her position at the university of auburn to pursue other opportunities in the northwest. for years three and four, becker worked with persea books and the journal poetry northwest to supply the poems for the site. happily, the connection with poetry northwest, one of the more prominent poetry journals of the last 60 years, was enabled by kuipers, who had become an editor at the journal after her move. 4 all the poems from the featured presses were previously published in books from the respective presses. 5 xpdf tools is a set of open source command line utilities useful to manipulating pdfs that has significantly faster performance than acrobat, http://www.xpdfreader.com/download.html. it provided more consistent quality output than acrobat’s built in export options. 6 importing content, wordpress codex, https://codex.wordpress.org/importing_content 7 after the initial year, the xml import was replaced with a csv import plugin allowing a spreadsheet provided by the editors of poetry northwest to generate draft posts that included the byline above the poem and links below the poem. the poem’s content, however, was not imported. this led to some confusion within the collaborative workflow and the accidental publication of two poems as “[poem here].” this was a rather embarrassing flub, after which we set up a better means for ensuring the poems were correctly prepared for publication. 8 for example: teicher, craig morgan. “fitting poetry to the screen: how one press is working to solve poetry’s e-book problems.” publishers weekly, 26 mar. 2012, p. 36+. academic onefile, http://ida.lib.uidaho.edu:2294/apps/doc/a284552394/aone?u=mosc00780&sid=aone&xid=6615b38a. accessed 11 jan. 2019. (https://www.publishersweekly.com/pw/by-topic/digital/content-and-e-books/article/51178-fitting-poetry-to-the-screen.html) 9 although now often used in web design, em units follow an older printing convention of assigning indentations and spacing based on with width of the letter “m”. 10 mariella moon, “short story vending machine promises old-school distractions”, engadget, 2015-10-24, https://www.engadget.com/2015/10/24/short-story-vending-machine-france/ (archived at: https://perma.cc/bqd5-dwgg). since that time the short edition short story dispenser has grown into a international business ( the short story dispenser, short édition, https://dispenser.short-edition.com (archived at: https://perma.cc/mvh2-8fz2) ) 11 colleagues at university of british columbia slais had recently created the “readers advisory device” that printed book suggestions, https://github.com/asistubc/rad-device 12 basic code template and concepts available in kioskslides, https://github.com/evanwill/kioskslides 13 for example, the little box of geek, http://geekgurldiaries.blogspot.com/2012/12/little-box-of-geek-project.html (archived: https://perma.cc/6tfb-cbgl ) 14 https://learn.adafruit.com/pi-thermal-printer/overview (archived: https://perma.cc/qd65-vt7n ) 15 in fact, a similar simplified style of wiring the circuit was adopted by newer versions of the adafruit kit in 2017. 16 simon monk, squid, https://github.com/simonmonk/squid 17 http://www.doctormonk.com/2013/02/raspberry-pi-and-breadboard-raspberry.html (archived at: https://perma.cc/95y9-f6rx ) 18 this is one of the ways adafruit lowers barriers to developing projects, since in the past this would involve tracking down obscure technical documentation from manufactures. 19 the books available sold out that night, so this did not prevent any purchasing of the book itself! 20 some who printed out poems at the dlf conference have since remarked about how important printing and holding poems on the day of trump’s election was to their well-being. 21 https://www.poets.org/national-poetry-month/home, poembot is especially helpful to celebrate poem in your pocket day, https://www.poets.org/national-poetry-month/poem-your-pocket-day 22 center for digital inquiry and learning, https://cdil.lib.uidaho.edu/ 23 making, innovating, and learning laboratory @ uidaho library, https://mill.lib.uidaho.edu/ 24 poembot has been used by students and librarians at north carolina state university, the college of wooster, swarthmore, and lewis & clark college. 25 the clean text data we now have after 4 years of operating vpod promises a number of possibilities to promote, have fun (https://evanwill.github.io/_drafts/notes/pi-py-poems.html), and analyze the vpod corpus in new ways, including the potential to analyze the different poetry presses and the style of poetry they publish via textual analysis, some of which we’ve begun here: https://uidaholib.github.io/poemchoice/. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – robustifying links to combat reference rot mission editorial committee process and structure code4lib issue 50, 2021-02-10 robustifying links to combat reference rot links to web resources frequently break, and linked content can change at unpredictable rates. these dynamics of the web are detrimental when references to web resources provide evidence or supporting information. in this paper, we highlight the significance of reference rot, provide an overview of existing techniques and their characteristics to address it, and introduce our robust links approach, including its web service and underlying api. robustifying links offers a proactive, uniform, and machine-actionable way to combat reference rot. in addition, we discuss our reasoning and approach aimed at keeping the approach functional for the long term. to showcase our approach, we have robustified all links in this article. by shawn m. jones (0000-0002-4372-870x), martin klein (0000-0003-0130-2097), and herbert van de sompel (0000-0002-0715-6126) information integrity is impeded by reference rot two characteristics of the dynamic nature of the web are link rot and content drift. link rot occurs when links break over time — a detriment we have all encountered numerous times, and that has been studied and quantified abundantly. content drift happens when content at a given web location changes over time, something we expect when gathering information on the current state of affairs — e.g., weather, news, stock prices. because we often assume that the latest web content is the most desirable, the effects of content drift are not immediately apparent. in past work, we quantified the extent of content drift for links to web pages referenced in scholarly papers. we argued that it significantly undermines the integrity of the web-based scholarly record because it prevents readers from revisiting the exact web content that authors referenced. but the detrimental effect of content drift is not limited to scholarship on the web. figure 1. the internet archive holds archived versions (mementos) of the u.s. national stockpile website from april 3, 2020 (left) and april 4, 2020 (right). significant changes in the textual description of the stockpile can be seen when comparing both mementos. content drift can mask changes in government policy. for example, the united states government has maintained a national stockpile of medical equipment since the 1950s and consolidated it at the strategic national stockpile in 2003. as shown in the left archived web page (“memento“, defined in the next section) of figure 1, the stockpile existed to support “state, local, tribal, and territorial responders” in the case of an emergency. while managing the covid-19 pandemic, senior white house official jared kushner directed that the stockpile was not for use by these municipalities, in contrast to the policy stated on its website. the right memento in figure 1 reflects this change in policy one day after kushner’s statement. journalists from ars technica, politico, the hill, the washington post, and others used mementos, like those in figure 1, to prove that the agency policy existed on the website up until kushner’s comments. if journalists had only been able to rely upon the most recent version of this resource, content drift would have prevented them from proving that the agency changed its policy after kushner’s comments. figure 2 demonstrates this problem generically. at time t0 an author wrote article a, which we formally refer to as a linking page. the article links to eight web pages, referred to as linked resources. at a later time tm, a reader visits the linking page and finds link rot for two and content drift for four of the linked resources. only two linked resources are the same as when the author originally linked to them. thus, a reader visiting the linked resources at time tm will not see the same content to which the author linked at time t0, and, in the case of content drift, may not even be aware of it. figure 2. an abstract example of reference rot over time. the linking page (article a) contains eight linked resources. at time t0, all linked resources match what the author viewed. at time tm, two links are rotten, four linked resources have content drift, and only two links remain that accurately reflect what the author saw at t0. all linked resources are subject to link rot and content drift, a web phenomenon which we collectively refer to as reference rot. investigative pursuits in journalism, as illustrated above, the integrity of the scholarly record, and evidence provided to support united states supreme court opinions, are only some of the examples that highlight the essential need to be able to revisit referenced content as it was at the time the authors of linking pages created links. we are therefore motivated to address the reference rot problem and propose robust links as a solution that we devised with an eye on long term functionality in an ever-changing web environment. other approaches to address reference rot reference rot, and especially the link rot component of the problem, has attracted significant attention. as such, it is not surprising that a variety of approaches exist that attempt to address it. here we cover persistent identifiers, approaches to cite web resources, and techniques that leverage web archives. we show that each has shortcomings when it comes to combating reference rot. to discuss these approaches, we borrow terms from the memento protocol. an original resource is the current, most recent version of a web resource. an original resource is identified by a uri-r. mementos are captures of original resources at specific points in time. a memento of an original resource is identified by a uri-m and has a specific archival datetime. persistent identifiers at first thought, we might conclude that persistent identifiers such as digital object identifiers (dois), purls, and w3ids are viable solutions to the reference rot problem. however, a quick analysis of the way they function reveals they can successfully be used in certain settings but not generally. all types of persistent identifiers use a similar approach to avoid link rot for links to a web resource. they consider the actual url of the resource to be temporary, subject to change. they introduce a permanent url for the resource that authors should use for linking purposes. the persistent identifier registrar manages a central lookup table to express the correspondences between the permanent urls and the respective actual, current urls. when a reader follows the link to the permanent identifier of a web resource, the registrar applies the lookup table to generate an http redirect from that permanent url to the current url of the resource. as time goes by and the resource changes web location, its custodian must update the lookup table and associate the permanent url with the new web location. this approach works successfully when the custodian fundamentally cares about the long term accessibility of its web resources and hence is willing to invest time and energy to keep the lookup table up-to-date. while this may be the case for scholarly publishers, cultural heritage institutions, and authors of web ontologies, for example, we cannot expect such commitment from the middle-of-the-road webmaster. the basic premise of the value proposition of persistent identifiers, namely a custodian’s commitment regarding long term access to its web resources, does not hold for the majority of web resources. for those resources, the author of a link rather than the linked resource’s custodian has to invest in making the link durable. citing web resources approaches for citing web resources have evolved. initially, web resources were infrequently cited, leading citation standard bodies to ignore them. as web resources became more commonly used as references, these same bodies recognized the need to update their standards to include web resources and also to support addressing the reference rot problem. figure 3 shows an example reference to a web page in the apa format. it includes a uri-r so that the reader can visit the current version of the cited resource, assuming it still exists. the reference format also specifies a “retrieved” date, an explicit recognition of the existence of reference rot. for consistency between reference standards, we refer to this date as the author view date. the reader can use this date along with the uri-r to manually locate a memento of the linked resource in a web archive. however, there are no guarantees that such a memento exists or that it was archived on or close to the date specified in the reference. as such, the reference leaves it to the reader to manually solve link rot and provides no solution to content drift. figure 3. an annotated example reference to a web page in apa format. figure 4. an annotated example reference from wikipedia. figure 5. an annotated example reference to a web page in iso/dis 690 format. where possible, wikipedia article citations to linked resources contain information that might be used to combat reference rot. they contain a link to the original resource, the date when the linked resource was retrieved by the author of the wikipedia article (the linking page), a link to a memento of the linked resource, and the archival datetime of that memento as shown in figure 4. with the uri-m, the reader avoids link rot. the archival datetime of the uri-m is not always near the author view date, meaning that this approach does not reliably address content drift. while it is possible for an author to create their own memento and manually insert its url into one of these citations, many of these uri-ms are inserted by scripts that are run after the wikipedia article is updated. the uri-ms added by these scripts often reflect a best effort to find a memento near the author view date. this approach used by wikipedia and others has found its way into the recommendations of the recent iso/dis 690 reference standard, as shown in an example in figure 5. it includes the uri-r and a uri-m for the resource. the reader can use the uri-m to visit a specific version of the linked resource. because the author explicitly endorses this uri-m as a reference, this reference format is, in principle, able to address both link rot and content drift. if the uri-m is inaccessible due to technical problems or other issues, the reader could still use the author view date and the uri-r to manually find another memento that might match the version captured on that date. the standard mentions, however, that the date specified by the “viewed” keyword does not always correspond to the author view date of the original resource. the example in figure 5 shows two dates with the “viewed” keyword. the first one, following the uri-r, is the author view date of the original resource. a reader can use this date in combination with the uri-r to locate a memento in a web archive. the second “viewed” date, following the uri-m, refers to the author view date of the memento. thus, the textual position of dates in iso/dis 690 citations must be carefully considered when consulting referenced web resources. authors can adapt these citation standards into any human-readable file format, like pdf, html, or docx. robust links leverage html and thus are confined to that format. techniques that leverage web archives a variety of infrastructural approaches to address the reference rot problem have emerged in which web archives play an essential role because they hold prior, timestamped versions of web pages. criteria for characterizing the techniques all techniques to combat reference rot that leverage web archives rely on the ability to deliver a memento for a linked resource as a means to counteract the fact that, some time after the link was established, the resource may have vanished from the live web or its content may have drifted. as such, these techniques rely on the existence and hence creation of mementos of linked resources. a number of factors contribute to the extent to which a given memento that is delivered addresses reference rot in an accurate and robust manner. we first discuss these factors and then describe and characterize the various techniques accordingly. temporal accuracy of a memento – consider a linking page with a set of linked resources. to address content drift, the archival datetime of a memento for a linked resource must be as close as possible to the date the author linked to it. if the linking page’s author — or someone on their behalf — creates a memento for each linked resource around the time the author views the linked resource’s content, then both link rot and content drift are addressed. because each such memento matches the specific version of a linked resource the way the author saw it when linking, a reader can review the linked resources in the state intended by the author of the linking page. if a third party, unconnected to the development of the linking page, has created a memento for one or more of the linked resources, then this third party has incidentally addressed link rot for that specific link in the author’s linking page. however, because this third party may not have created this memento at the time the author viewed the linked resource, content drift is not necessarily addressed. distribution of mementos – redundancy of mementos is also important for addressing reference rot in a robust manner. consider an approach to reference rot that creates mementos but only places them in one archive. if we link to one of these mementos, but the web archive in which it resides is – temporarily or permanently – not in service, we again have link rot. thus, an approach that stores mementos of referenced content in multiple archives is more robust. similarly, an approach that only retrieves mementos from one web archive is equally vulnerable and its robustness is much increased by providing access to mementos across multiple archives. unconstrained memento access – consider the approach that addresses reference rot, but only for some readers or some linked resources. such approaches apply constraints on access to mementos. some constraints may be imposed by the tools themselves. specific browsers or browser extensions may only address reference rot for the users that install them. a server running a special extension may only create mementos for the linked resources found in the linking pages it hosts. other constraints regard the original resources for which mementos are accessible. for example, an approach may only provide mementos of linked resources from a specific domain, like gov.uk, or it may only address reference rot for the linked resources that are frequently navigated by the visitors of certain cloudflare customer sites. approaches that provide unconstrained memento access are more robust. datetime for memento access – some approaches take a desired datetime into account when delivering a memento to the reader. this datetime could be the publication date of the linking page, a manually selected date, or information from the link itself such as the dates included with various citation standards shown in the previous section. the inclusion of a datetime can help a reader locate a memento near this date and thus can address content drift. ideally, the date information should be machine-readable so that web crawlers, on-page javascript, and other automation can easily parse information and help readers find mementos. characterization of techniques that leverage web archives perhaps the most straightforward approach to combating reference rot is to use the uri-m as the link target when creating new links. this approach supports unconstrained memento access because it works in all browsers, is not limited to specific domains, and can be used with any web publishing platform. if the author relies upon a memento created by others, for example via web crawling by web archives, then this solution at best addresses link rot. if the author proactively creates the memento, then this approach addresses link rot and content drift. unfortunately its effectiveness is dependent on the continued existence of a single web archive. if the web archive storing the uri-m fails for technical, financial, legal, or other reasons, then we again have link rot because the memento becomes unavailable. if the web archive is not available, we could search for the uri-r in other web archives, but how do we know what the original uri was? uri-ms do not always convey the uri-r. for example, the uri-r https://lanl.gov is apparent at the end of the internet archive uri-m https://web.archive.org/web/20201027204016/lanl.gov but can not be obtained from the perma.cc uri-m https://perma.cc/q938-lpmg. thus, if the uri-m fails, the reader has no way of reconstructing the reference. the recent partnership between the internet archive and cloudflare goes a step further. every few days, cloudflare shares its customers’ popular url paths with the internet archive, which then creates their mementos. if a linked resource served by cloudflare is unavailable because its origin server is down, cloudflare detects this failure and redirects readers to the most recent memento of the page. this solution works only for cloudflare-supported websites and only for those customers who opt-in to this service. similarly, the uk government’s web continuity project proactively preserves uk government web sites’ pages and redirects from broken links in government web sites to these uk government uri-ms. while these solutions address link rot, none address content drift because they lack coordination with a linking page’s author. because they only work with specific linked resource domains, these solutions also only apply to a subset of the web. the brave browser, the wayback machine browser extensions for multiple browsers, and the “no more 404s” browser extension all address link rot. when a browser user clicks on a broken link, each of these approaches aims to detect the broken link and seeks a replacement uri-m in the internet archive, ignoring mementos in other web archives. they do not provide unconstrained memento access for readers because they limit readers to only using specific browsers or extensions. the approaches do not address content drift because they rely upon the incidental preservation of linked resources by a third party, meaning if no memento of a linked resource exists, then link rot persists. they also only search for resources in a single web archive, forgoing the possibility that the linked resource might exist in other web archives. it is worth mentioning that none of these solutions a) know the version that the linking page’s author intended to reference and b) can address content drift. in contrast, the memento time travel chrome extension does allow a user to select a datetime from a calendar picker and right-click on a broken link to find a memento of the linked resource around the specified datetime and across multiple web archives. but it is up to the reader to make an informed guess for the datetime that should be used. for example, the reader could choose the creation date of the page that contains the broken link. unless the author deliberately created a memento around this date, this approach relies upon incidental archiving and thus may not address content drift. harvard university’s berkman center maintains the amber project, which provides plugins for drupal and wordpress that create mementos of linked resources for linking page authors. amber stores these mementos in multiple web archives (e.g., perma.cc and the internet archive) and caches them on the author’s server. if a reader clicks on a linked resource and amber detects that the link is broken, it offers to let the reader view the specific memento of the linked resource associated with the linking page. thus, amber satisfies multiple robustness criteria. it creates mementos near the time of linking, addressing content drift. amber allows readers to reach these specific mementos if the link is broken, addressing link rot. however, amber only works for drupal and wordpress, limiting authors to those tools. it also does not provide machine-readable links, making it difficult to apply other solutions if amber fails or its mementos no longer exist. as mentioned in the previous section, english-language wikipedia often links to both the uri-r and a uri-m. additionally, it provides the date of the archiving, as seen in figure 4. this approach focuses on combating link rot. it only addresses content drift for those linked resources for which the author explicitly created a memento. this approach does work in all modern browsers and allows linked resources from any domain, but it only addresses link rot for wikipedia and potentially some mediawiki installations, limiting authors’ options. even though an author can apply mementos from other web archives to their references, the wikipedia scripts that automatically apply this solution to linked resources were built based on a specific partnership with the internet archive. much like the reference standards mentioned above, a reader can use the uri-r and the archive date to find a suitable replacement uri-m in another web archive if the given uri-m is not available. humans can read these references and understand how the pieces fit together, but machines cannot. any client performing analysis on these linked resources would have to resort to heuristics to distinguish uri-ms from uri-rs. if a uri-m fails, a machine client would have no method of detecting which date on the linking page refers to which linked resource and therefore cannot collect the necessary information to find an alternative uri-m for the linked resource. table 1: different web archive-based reference rot solutions and their capabilities. memento creation memento access reference rot archival date is the same as the linking date memento stored in multiple web archives linked resource from any domain works in all modern browsers approach applicable to any publication platform searches for mementos in multiple archives applies date information from linking page machine -readable links addresses link rot addresses content drift brave browser no no yes no yes no no no yes no wayback machine browser extensions no no yes no yes no no no yes no 404 no more firefox extension no no yes no yes no no no yes no internet archive + cloudflare no no no yes yes no no no yes no uk government web continuity project no no no yes yes no no no yes no memento time travel browser extensions no no yes no yes yes at reader’s discretion no yes yes berkman center’s amber project (drupal/wordpress extension) yes yes yes yes no no no no yes yes wikipedia + internet archive sometimes sometimes yes yes no no yes no yes sometimes robust links yes yes yes yes yes yes yes yes yes yes robustifying links motivated by the severity of the problem and the lack of comprehensive and unconstrained solutions, we devised the robust links approach. robust links leverage existing web archiving infrastructure and html5 extension attributes. robust links is the first approach that actually annotates links with a date and a uri-m, making it novel compared to other infrastructure approaches. the approach is devised for links in html but not tied to any specific authoring platform. robust links’ reliance upon html prevents the approach from being used with other document formats that contain links (eg., pdf, docx) in which reference formats as mentioned above can be used. in addition, robust links are machine-readable, inviting the creation of a variety of tools that leverage them. creating a robust link requires the following steps: the linking page’s author, or someone working on their behalf, archives a linked resource in one or more web archives to improve the odds of survival and ensure readers can visit that specific version in the future. the author enriches the html of the linking page with information about the linked resources’ uri-r, the uri-m of the memento created in the first step, and the datetime of archiving, thus enabling readers and machines to find the desired version in web archives. for example, instead of this brittle html link: <a href="http://lanl.gov">los alamos national laboratory</a> we propose to use this robust link: <a href="http://lanl.gov" data-versionurl="https://perma.cc/q938-lpmg" data-versiondate="2019-10-28">los alamos national laboratory</a> readers clicking on the robust link by default will reach the current version of http://lanl.gov. tools that leverage this robust link can additionally support visiting the uri-m specified in data-versionurl and can combine the uri-r found in the href with the date from data-versiondate to find additional uri-ms. alternatively, if the author wishes to allow readers to click on the uri-m by default, they can use this robust link instead: <a href="https://perma.cc/q938-lpmg" data-originalurl="http://lanl.gov" data-versiondate="2019-10-28">los alamos national laboratory</a> in this case, machine clients can combine the uri-r found via the data-originalurl attribute and the date from the data-versiondate attribute to discover a replacement uri-m should the existing uri-m fail. actionable robust links to make the machine-readable attributes provided in robust links actionable for human readers, we have implemented a javascript library (robustlinks.js) that an author can reference in their pages. the javascript code parses these attributes and provides readers with a drop-down menu with three choices, as shown in figure 6. figure 6. an example of the drop-down menu provided by robustlinks.js and robustlinks.css. the javascript code builds this menu from the robust link’s attributes and formats the menu based on a corresponding css file (robustlinks.css). as displayed in figure 6, the reader has several options: they can visit the current live version of the page, which the javascript reads from the href, they can navigate to the uri-m of the “version archived on 2020-07-23”, as extracted from the data-versionurl, or they can click “version archived near 2020-07-23” to seamlessly connect to the memento timetravel service and find the memento closest to 2020-07-23 from more than 20 memento-compliant web archives around the world. the reader does not need to install a plugin because all of this functionality is provided on the page by javascript linked from the html. our code is merely one example of the type of machine client that can be built upon this markup. consider the possibility that both the uri-m and uri-r are inaccessible. a system (e.g., browser plugin, web crawler) could apply the date and the uri-r to find potential replacement linked resources from the same time period or it could analyze the spread of dates from all linked resources to find other documents on the same topic and in close temporal proximity. when authors create robust links and add this javascript and css code to their articles, they allow readers to access the intended default target of their link or find a memento that matches the date the author intended. once the author robustifies a link, they address link rot by providing a memento and combat content drift by pro-actively generating that memento, as such making sure that the memento’s archival datetime matches the author’s viewing date. from here, readers can obtain the content that helps them verify evidence. the robust links web service to assist authors in creating robust links, we provide a robust links web application. as shown in figure 7, the minimalist entry page accepts two user inputs: the url of the resource and the link text. figure 7. the robust links web application entry page. figure 8. the robust links web application response page. figure 8 shows a screenshot of the web application’s response. it instructs the author to follow two steps: decide which resource they want their readers to reach with a default mouse click, the current version or the memento, and then copy the appropriate html. from the second text area, copy the given html to ensure that the robustlinks.js and robustlinks.css are loaded in the linking page. this is only required once per linking page. for users who would like to robustify many links we provide access to the robust links api used by our web application. with this api, a client can supply a uri-r or a uri-m and link text. the api will respond with json containing information about the link and how the service created it, as shown in this curl example: $ curl -g --data-urlencode "url=https://abcnews.go.com" --data-urlencode "anchor_text=abc news for june 15, 2020" https://robustlinks.mementoweb.org/api/ { "anchor_text": "abc news for june 15, 2020", "api_version": "0.8.1", "data-originalurl": "https://abcnews.go.com", "data-versiondate": "2020-06-15", "data-versionurl": "https://archive.li/wip/hwzdd", "request_url": "https://abcnews.go.com", "request_url_resource_type": "original-resource", "robust_links_html": { "memento_url_as_href": "<a href=\"https://archive.li/wip/hwzdd\"\ndata-originalurl=\"https://abcnews.go.com\"\ndata-versiondate=\"2020-06-15\">abc news for june 15, 2020</a>", "original_url_as_href": "<a href=\"https://abcnews.go.com\"\ndata-versionurl=\"https://archive.li/wip/hwzdd\"\ndata-versiondate=\"2020-06-15\">abc news for june 15, 2020</a>" } } the json response from the api returns the link text and the requested url to the user. inside the robust_links_html key is the preformatted html for each default link target. for users’ convenience to construct their own html, the api provides each attribute for the robust link as its own json key. if the user supplies a uri-r, the robust links service will create a memento via the archivenow library and use the current datetime as the value for data-versiondate. the service utilizes either the internet archive or archive.today to create the memento, and we are exploring the addition of perma.cc in a future release. a client can supply an optional argument to the api to specify which web archive they desire for creating the memento. if they do not indicate a preferred web archive, the service randomly chooses one. if the user provides a uri-m to the api, the robust links service will dereference the uri-m and extract the values for data-originalurl and data-versiondate from the memento’s http response headers as specified in the memento protocol. error codes, source code examples, and more are available on the api’s documentation page. robustifying the robust links browser code we considered advocating that authors include our javascript and css inline on every page. this would include the code by-value. unfortunately, releasing javascript code this way has its problems. copying and pasting code is error-prone and invites tinkering, which may result in robust links that do not function properly. additionally, the hosting page will not receive bug fixes or updates that reflect changes in the environment with which it interacts, including changes to web archives that are covered and services overlaying those web archives, such as memento time travel. we therefore decided that it was better to centrally host the code and recommend that web page authors link to its urls, thus including it by-reference. the advantage of this decision was demonstrated during the writing of this article. we resolved an issue with conflicting css styles to ensure that robust links menus are presented as intended. all users of the robusts links javascript/css library will now benefit from this improvement without having to make changes to their own pages. as web technologies evolve, providing code by-reference will allow us to devise new resolutions for future authors and improve everyone’s robust links experience. unfortunately, this means that our code itself becomes subject to link rot. to keep this solution operational, we need to solve two problems: first, we must select a hosting platform for the source code that is expected to remain available for the foreseeable future. second, in case we have to move the code to another platform, we must provide continued access. we explored several options for addressing both issues. with respect to choosing a suitable hosting platform, we realized that hosting our code at robustlinks.mementoweb.org is not a long-term solution. if robustlinks.mementoweb.org loses funding, the take-down of the site is likely. solutions like hosting the code in a web archive creates a dependency on that specific web archive, something robust links is designed to avoid. the international internet preservation consortium (iipc) is an organization with members from over 45 countries, including national, university and regional libraries and archives. it focuses on providing tools, techniques, and knowledge for preserving web content and providing access to that content. the iipc has graciously provided us with a github repository in which to deposit robustlinks.js and robustlinks.css. through github pages, we instruct github to serve the code with the correct content-type so that web page authors can include it by-reference. with the code hosted in iipc’s github repository, we have improved its odds of surviving for the foreseeable future, but we have not solved the potential issue of changing urls. this scenario is not inconceivable as, for example, github pages could change their url paths, the iipc could move their content to another platform such as gitlab, or even host it themselves. a change in url would cause the robust links menus to fail and readers would no longer be able to benefit from the convenience provided by this code. we know from a 2020 cloudflare report that web page authors rarely update a link to a javascript library link once a page is published. we have no reason to believe that links to robustlinks.js and robustlinks.css will be any different, so we decided to investigate the possibility of minting persistent identifiers for our robustlinks.js and robustlinks.css urls. considering that funding agencies did not fund our project infrastructure indefinitely, running our own persistent identifier server would eventually lead to the same link rot for our browser code that our project is trying to defeat. we considered using dois (and their handle variation), purls, and w3ids. we decided in favor of datacite dois because datacite is an organization with a rapidly growing membership base, values regarding openness that we share, and a strong commitment to the long-term usability of their identifiers. our constructive interactions with datacite staff in the context of this effort gave us further confidence in our choice. what was still needed was a party — a datacite member — to mint the dois assigned to our code and manage them going forward, for example, for updating the correspondence table, in case the code’s location changes. we found an agreement with collaborators from old dominion university (odu) to take on this role. our agreement benefits from the fact that odu holds memberships in both organizations datacite and iipc. thus, to robustify the robust links browser source code, we developed the following solution after much collaboration with third parties: ensure that an entity with longevity hosts our code, in our case the iipc. engage a collaborator to mint and manage a doi so that url changes do not break web page access. in our case odu registered 2 dois provided by datacite pointing directly at our javascript and css resources. use dois in all instructions and other code references so that the underlying urls may change, but authors do not need to update their pages. a journey to combat reference rot reference rot, the union of link rot and content drift, is a function of the dynamic nature of the web and the increasing frequency at which we link to web resources in our publications. its detrimental effects have a negative impact on all web-based inquiries that require accuracy of referenced resources, yet, thus far, we have not seen a comprehensive approach to effectively address it. our robust links approach changes the status quo by utilizing publicly available web archiving infrastructure to proactively create mementos in different archives at the time of linking. robustified links offer several fallback mechanisms in case the original linked resource and its pro-actively created memento become unavailable. our robust links web service provides a simple user interface for authors to create robust links and conveniently obtain the html that embeds the robust links javascript and css in linking pages. the underlying api offers the same functionality to machines, for example, to accommodate batch processing of many links. making robustified links actionable in a browser requires the inclusion of robustlinks.js and robustlinks.css into the web page. with an eye on an ever-changing technological environment, we were motivated to establish a solution that allows for continued modification of the code while also helping to ensure its long-term availability and accessibility. our approach entails hosting the code in the community-supported iipc github repository and minting persistent identifiers (dois) that redirect to the code as served by github pages. this setup enables the by-reference inclusion of the javascript and css code using their dois. robustifying links takes effort. while our web service makes this process more convenient for humans and machines, we can envision more automated workflows. for example, content management systems such as wordpress and mediawiki, word processing platforms such as microsoft word and google docs, or blogging platforms such as blogger and medium could integrate robust links functionality. at the end of the day, however, it still takes humans’ commitment to combat reference rot. as is true for all web archiving based approaches, we work under the assumption that “good enough” mementos of linked resources can be created but we realize that existing archiving and replay technology does not always satisfy. however, that is a challenge beyond the scope of this effort that many talented people are working on. acknowledgments the authors would like to thank everyone who contributed to the robust links concept and service. the robust links work was initiated in the hiberlink project, a collaboration between the university of edinburgh and the los alamos national laboratory, funded by the andrew w. mellon foundation. michael l. nelson from old dominion university contributed to robust links, both during the hiberlink project and the recent effort to improve the solution as described in this paper. we appreciate the assistance of datacite staff helena cousijn, martin fenner, and matt buys for supporting the solution that uses dois to accord a persistent uri to robust links browser code. we also appreciate the service of karen vaughn at old dominion university for managing those dois. thanks to mark graham from the internet archive for answering questions about their collaboration with wikipedia. finally, we also thank the international internet preservation consortium for agreeing to ensure the browser code’s longevity. about the authors shawn jones is a graduate research assistant working for the prototyping team at lanl’s research library. shawn is also a phd student at old dominion university and a member of the web science and digital libraries research group, led by michael nelson. he has contributed tools, data, and analysis to projects and frameworks such as memento, signposting, and robust links. in addition, shawn is the lead of the dark and stormy archives project, an initiative to summarize web archive collections through visualization techniques common in social media. more information about shawn is available at: https://www.shawnmjones.org/ martin klein is a scientist and lead of the prototyping team in lanl’s research library. in this role he focuses on research and development efforts in the realm of web archiving, scholarly communication, digital system interoperability, and data management. he is involved in standards and frameworks such as memento, resourcesync, signposting, and robust links. martin holds a diploma in computer science from the university of applied sciences berlin, germany and a ph.d. in computer science from old dominion university. more information is available at: https://orcid.org/0000-0003-0130-2097 herbert van de sompel is chief innovation officer at data archiving and networked services (dans) in the netherlands. he has played a major role in creating the open archives initiative protocol for metadata harvesting (oai-pmh), the open archives initiative object reuse & exchange specifications (oai-ore), the openurl framework for context-sensitive services (ansi/niso z39.88-2004), the sfx linking server, the bx scholarly recommender service, info uri (rfc 4452), open annotation (w3c community group specification), resourcesync (ansi/niso z39.99-2014), memento “time travel for the web” (rfc 7089), robust links, and signposting. he holds degrees in mathematics, computer science, and communication science from ghent university, belgium. more information at https://hvdsomp.info subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – an introduction to using metrics to assess the health and sustainability of library open source software projects mission editorial committee process and structure code4lib issue 57, 2023-08-29 an introduction to using metrics to assess the health and sustainability of library open source software projects in lyrasis 2021 open source software report: understanding the landscape of open source software support in american libraries (rosen & grogg, 2021), responding libraries indicated the sustainability of oss projects to be an important concern when making decisions about adoption. however, methods libraries might use to gather information about sustainability is not discussed. metrics defined by the linux foundation’s chaoss project (https://chaoss.community/) are designed to measure the health and sustainability of open source software (oss) communities and may be useful for libraries who are making decisions about adopting particular oss applications. i demonstrate the use of cauldron.io as one method to gather and visualize the data for these metrics, and discuss the benefits and limitations of using them for decision-making. by jenn colt introduction the lyrasis open source software report (rosen & grogg, 2021) tells us libraries are concerned about “sustainability” when selecting oss solutions. libraries express concern about software receiving regular updates into the future, both for the implementation of new features and to address security concerns. libraries want to know that a project will be available long-term, will evolve new functionality, and will remain secure. sustainability is not the only consideration when choosing which software to adoptfunctionality, ease of deployment, vendor support, and any number of other concerns factor in as well. however, sustainability might be considered a prerequisite for these other concerns. if community health is an indicator of the sustainability of a project, then it is worth investigating and weighing prior to adoption. doing so will let libraries “consider the risk implied by using the oss as is … and decide whether to help improve the health or choose another option.” (linåker et al., 2022) predicting the long-term sustainability of a project is difficult but measuring the health of its supporting community can help. the linux foundation’s community health analytics in open source software (chaoss) project, which creates metrics for measuring oss community health, defines health as “a project’s ability to continue to produce quality software” (goggins et al., 2021). metrics are useful to many stakeholders, including project sponsors and maintainers, but this article is primarily about reviewing metrics as the adopter of a given oss project. that is, when a library is considering using an existing project and beginning a collaborative relationship with an existing oss community. what the library’s contributions to the oss community will eventually be are not yet defined. in this initial stage, the library is both evaluating the software for adoption and considering what kind of strategic contributions they can make to the community to help sustain the software. contributions are defined broadly and do not consist solely of code contributions. an evaluation of metrics at the start of a library’s involvement with a project can allow them to have a better understanding of how to contribute to a project, and to affirm they have the resources needed to effectively adopt the software and participate in the community. chaoss metrics and models the linux foundation’s chaoss project has developed more than 80 metrics for evaluating the health of oss communities. to make it easier for both projects and adopters to apply these metrics, many of them have been further organized into models. the chaoss models allows evaluators to consider a set of metrics in relationship to each other rather than individually, which can provide a better understanding of the real world systems represented by the metrics. one of the metrics models created by chaoss is the starter project health model [1] which will be described below. this model includes four metrics that relate to the current health of the project mainly by measuring how responsive the project is to change requests, how robust the code contributor base is, and how active the repository is. together these metrics offer basic insights into how healthy the project currently is while also providing information about the potential for sustainability. the model has been created as a place to start for people new to gathering these kinds of metrics. available tools when first experimenting with metrics, library staff need to collect data without a high investment of time and energy. there are a number of ways this can be done. for projects that are publicly hosted on github and use the github community features, github insights offer limited data visualization features that support investigating some metrics. however, github insights lacks the ability to customize or dive deeper into the visualizations provided, lacking filters for most of their metrics. the larger the project, the harder it is to understand the full picture of a project through github insights, other than being able to tell whether the project is active or not. for github hosted projects you can also use the github api to create your own implementations for collecting data and creating visualizations. (qui et al., 2023) the chaoss community collaborates on multiple software projects that implement their metrics. cauldron.io, used in the examples below, is built using components of the project’s grimoirelab software [2]. grimoirelab allows the user to integrate data from a range of sources and to consolidate the identities of actors in the data across data sources (i.e., be able to say that two github users are actually the same user). this more complex data gathering and consolidation of identifiers cannot be done in the free cauldron.io instance of grimoirelab, and requires authorized access to the data. chaoss also produces augur [3], another piece of software for analyzing and visualizing project data. augur is more closely focused on the trace data generated by repositories, rather than the broader sources of community data supported by grimoirelab. both of these projects are of course open source and can be hosted or run locally [4]. cauldron.io provides a pleasant user experience for interacting with these metrics, and is the software demonstrated here. cauldron.io does not allow analysis of user data from github outside of the user’s uuid and email domain name. for projects not hosted on github or projects privately hosted on github, both grimoirelab and augur allow you to interact with repositories and additional systems (like jira) if you host the metrics software yourself or pay for them as a service, provided you are able to secure access to the data. if libraries choose to connect additional data sources to these systems, it should be done so with attention to privacy and a degree of restraint. while forums, mailing lists, and chat services can be ingested into grimoirelab, it is not prudent to do so without a great deal of thought about the ethical and privacy implications. the chaoss project offers their own guide on ethics and privacy in relationship to this data [5]. exploring the starter project health metrics model after choosing a tool, the easiest way to understand these metrics may be to try applying them to a repository and examining the results. below are the metrics from the starter project health metrics model as applied to one library-related oss project, dspace using cauldron.io. dspace’s repositories are publicly hosted on github and the project also uses the built-in github issue tracker, making it a good project with public data for trying out these metrics. each of the metrics provides a starting point for further investigation and understanding of the collaboration a library is entering into when joining an oss community. the figures below come from the cauldron.io tool. prior to reviewing the metrics, it should be clearly stated the purpose of these metrics is to evaluate the state of a project and its community, which are complex systems that can involve many people, institutions, and outside forces. the goal of these metrics is not to evaluate people or their performance in any way, but rather to understand a project and find the most productive ways to participate in it. starter project health model metric #1: time to first response time to first response [6] indicates how quickly the project community responds to activities like issues being opened or pull requests being made (chaoss, n.d.). recalling libraries’ concerns about continued feature development in the lyrasis survey, time to first response might be an indicator of whether the project is willing to discuss feature requests, suggestions, or provide troubleshooting answers. it’s important to keep in mind that not all discussion may happen on the platform that you are reviewing. many projects also maintain email lists for discussion. low levels of interaction on github might indicate a need to investigate what other forums are being used, even if you do not directly collect data from them awareness of those venues is important. reviewing the level of conversational activity around a project prior to adopting can help the library understand what level of support and interaction they should expect from the community prior to adopting, so they can understand the impact that will have on staff who need to seek answers about the software, and perhaps they may also see for themselves an opportunity to contribute something like documentation effort if the project needs help providing support to users. below is the timing overview report panel from cauldron.io. this panel is comprised of a set of visualizations in their elastic kibana instance, which provides more flexible reporting capability than the premade reports available on the cauldron.io web site itself. after creating an account in cauldron.io and generating a report for a repository, clicking the “more details” link will take you to the elastic kibana interface. here we can see that the timing is being measured for two repositories – the dspace api and its angular ui. this ability to combine multiple repositories is one advantage of using a metrics reporting system rather than github insights. from the dashboard we see the timing efficiency of the dspace repositories improving over time. the median and mean for both time measures are available as well. the median is likely more meaningful as it is typical of most projects to have at least some issues that are not well-defined or for some other reason not readily solvable that hang around for a long time, driving up the average time to response/close. figure 1. the cauldron.io timing overview dashboard. starter project health model metric #2: change request closure ratio the change request closure ratio [7] is a measure of how many pull or merge requests are created in a week versus how many were closed (chaoss, n.d.). it is an indicator of how well the community is able to keep up with the requests being made. both change request closure ratio and time to first response may also reflect how well the project is able to keep up with necessary security updates from the project’s dependencies, thus providing measurement related to libraries concerns about maintaining secure software. below is the graph from cauldron.io of the issues closed and issues opened by month for both dspace repositories. in the graph, the number of issues closed and opened mostly move together indicating that the maintainers are likely to be keeping up with the number of issues being opened. figure 2. number of issues closed vs created by month graph from cauldron.io. starter project health model metric #3: bus factor bus factor [8] represents the number of people who make 50% of code contributions to the project (chaoss, n.d.). it is an attempt to predict how likely a project is to be able to sustain itself if one of the main contributors were to leave. the smaller the bus factor, the less sustainable the project might be. for projects supported by institutions, this can be a tricky metric to interpret. important projects may be maintained by few individuals, but those individuals might be backed by organizations with long-term commitments to the project, even in the face of staffing changes. so while the bus factor may be small the level of the commitment from an institution may lower the apparent risk. that said, when one of those active contributors leaves their job at an institution, the loss of knowledge can still be very impactful, and there is no guarantee that new staff will be as interested or skilled in the project as the previous contributor. contributors are not interchangeable just because they work for the same institution. organizational diversity [9] is a metric outside of the starter heath model but related to bus factor. it measures how many organizations are represented in a project. if only one institution is contributing to a project, the organizational diversity is low and might be considered a risk for future sustainability much like bus factor. again, the level of commitment from institutions (and their level of influence on the project [10], yet another metric) can also factor in here. to further complicate matters, any metric counting contributors in github is reliant on the way those contributors identify themselves. contributors often do not use organizational email addresses for their account, and they may have more than one account over time. these factors make the bus factor and organizational diversity estimates, but they are still useful as first indicators of potential issues or questions to investigate. the pie chart below demonstrates the rough bus factor for the dspace backend repository alone (in this case it seemed to make the most sense to analyze the repositories separately). 50% of the chart is made up of eight contributors so the bus factor is 8 based on this chart. we know, however, that users can have more than one github account so you would have to investigate further to get a really firm number. this chart is just for the last two years of commits and bots have been filtered out. this pie chart is not premade by cauldron.io but it can be created using the “my workspace” feature in the elastic kibana interface that allows you to create your own visualizations from the repository data. understanding the current distribution of contributors is much easier in this visualization than on the github insights contributors page which is a long list of contributors over the life of the project. figure 3. custom pie chart of commits by user id, made by using the cauldron.io interface into elastic kibana. starter project health model metric #4: release frequency release frequency [11] is a measure of how often the project creates releases (chaoss, n.d.). “release” can be defined in many different ways and some knowledge of how the project handles releases is necessary to measure this metric. for the sake of simplicity, we will look at releases on the github platform, but meaningful releases can happen in many other places and in other ways. because of the many meanings of release, cauldron.io does not offer a release frequency statistic. but for a project hosted and released on github, release frequency can be observed from the repository’s releases page, for example https://github.com/dspace/dspace/releases. when reviewing release frequency for a project, one thing you hope to see is semantic versioning, which allows you to understand the different types of releases being made – major, minor, and patch. [12] the frequency of major releases may be tied to the frequency of new feature releases. while the other types of release may indicate how often bugs are fixed and security patches released. conclusion metrics are useful for at least two reasons. first, people deeply involved and invested in a project may or may not have an accurate big picture of the project. metrics can help confirm perceptions of the project, or point out areas where measurements are in disagreement with perceptions. second, the people making decisions about how to invest in software projects are often not the people who are involved in hands on implementation. in these cases, metrics can help tell the story of why an open source project needs support, or why implementing an administrator’s chosen project will be difficult. perhaps most importantly, reviewing metrics is a cue to engage in careful and strategic thinking about the relationship between an adopting library and the oss community they are joining. we are long past the era where anyone thinks oss means “free” but we still need ways to talk about the differences in expectations and resources of oss adopters and oss communities, and metrics are one tool for exploring those gaps. joining a community should be done with care and thought, and with the knowledge that oss is an important place of collaboration between libraries. if we wish to continue to produce our own software and to have it be competitive with commercial solutions, we should use all the tools we have at our disposal to make our partnerships as productive as possible. acknowledgments the author gratefully acknowledges the support of paula dempsey and the ala library research round table mentorship program. about the author jenn colt is an academic librarian in cornell university library’s technical services department where she is the head of the automation and metadata management unit. she manages data in the library’s open source library services platform, folio, and also serves on the technical council of the folio project. she has developed her interest in open source project governance and health through different levels of involvement in several library-related oss projects. she is still a newbie in the chaoss project. notes [1] starter project health metrics model: https://chaoss.community/kb/metrics-model-starter-project-health/ [2] grimorelab: https://chaoss.github.io/grimoirelab-tutorial/ [3] augur: https://github.com/chaoss/augur [4] chaoss privacy and ethics guide: https://github.com/chaoss/community/blob/main/data-use-statement.md [5] i spent a significant amount of time trying to install both augur and grimoirelab on my computer but was unable to do so successfully, which contributed to my choice of a hosted solution. that said, i think someone with more devops experience (and maybe not trying to use an m1 mac) would probably have a much easier time. self-hosting would be preferable for privacy purposes, especially for project maintainers who wish to do more in-depth analysis related to contributors. [6] time to first response metric: https://chaoss.community/kb/metric-time-to-first-response/ [7] change request closure ratio metric: https://chaoss.community/kb/metric-change-request-closure-ratio/ [8] bus factor metric: https://chaoss.community/kb/metric-bus-factor/ [9] organizational diversity metric: https://chaoss.community/kb/metric-organizational-diversity/ [10] organizational influence metric: https://chaoss.community/kb/metric-organizational-influence/ [11] release frequency metric: https://chaoss.community/kb/metric-release-frequency/ [12] semantic versioning: https://semver.org/ references metrics model: starter project health. chaoss. [accessed 2023 jun 1]. https://chaoss.community/kb/metrics-model-starter-project-health/. gamalielsson, jonas, and björn lundell. “sustainability of open source software communities beyond a fork: how and why has the libreoffice project evolved?” journal of systems and software 89 (march 2014): 128–45. https://doi.org/10.1016/j.jss.2013.11.1077. goggins s, lumbard k, germonprez m. 2021. open source community health: analytical metrics and their corresponding narratives. in: 2021 ieee/acm 4th international workshop on software health in projects, ecosystems and communities (soheal). p. 25–33. https://doi.org/10.1109/soheal52568.2021.00010. linåker, johan, efi papatheocharous, and thomas olsson. “how to characterize the health of an open source software project? a snowball literature review of an emerging practice.” in the 18th international symposium on open collaboration, 1–12. madrid spain: acm, 2022. https://doi.org/10.1145/3555051.3555067. qiu hs, lieb a, chou j, carneal m, mok j, amspoker e, vasilescu b, dabbish l. 2023. climate coach: a dashboard for open-source maintainers to overview community dynamics. in: proceedings of the 2023 chi conference on human factors in computing systems. new york, ny, usa: association for computing machinery. (chi ’23). p. 1–18. https://dl.acm.org/doi/10.1145/3544548.3581317. rosen h. (orcid: 0000-0001-6804-7073), grogg j. (orcid: 0000-0001-5136-4507). 2021 aug. lyrasis 2021 open source software report: understanding the landscape of open source software support in american libraries. http://hdl.handle.net/20.500.12669/97. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – editorial: musing on learning to be a selfish librarian mission editorial committee process and structure code4lib issue 39, 2018-02-05 editorial: musing on learning to be a selfish librarian one of the perks of being the coordinating editor is you get to write the opening editorial for the issue.  it’s an opportunity to think broadly about the community, the journal…current events.  and if you look back over the past year or so, those that have taken on this role have been more than up […] one of the perks of being the coordinating editor is you get to write the opening editorial for the issue.  it’s an opportunity to think broadly about the community, the journal…current events.  and if you look back over the past year or so, those that have taken on this role have been more than up for the task.  i know, i’ve been reading them the past couple of weeks.  ruth had the unenviable task of shepherding the journal right after the elections, at a time when many wondered what the future would hold for the profession, the country, and their families.  peter talked about the current state of the journal, and some of the successes and long-term challenges that we face as an all-volunteer publication.  it’s a great reminder of why the journal exists and why, as a community, we need to continually be evolving and encouraging new voices and new editors.  and his words had an impact, as we welcomed a number of new members to the editorial committee, which has allowed the journal to increase the number of articles published and produce an even better product.  and finally, in the last issue, carol tackled the issue of economics, and the incredible work of the fiscal continuity interest group to investigate, recommend, and provide some long-term vision around the ongoing health of the annual conference.  and while everyone is looking forward to being in washington, dc for what is likely going to be another great conference, the economics of the conference are hard to ignore.  dc is an expensive city, and this year, the conference registration and logistics are higher than they have ever been; a worrying trend that may start to price out attendees.   more troublesome, as i write this today, there still are no volunteers to take up next year’s annual conference, a trend that has now repeated itself over the past few years.  i’ve started to wonder if we are seeing the last of the annual conferences, at least in their current form.   at the same time, i’m consistently heartened by the increased number of local events that have been extending the reach of the community.  carol wrote that it’s time to continue to evolve…maybe that’s what we are seeing now.  it will be interesting to watch. anyway, that is to say, being the coordinating editor and having the opportunity to give the opening remarks is a privilege, and one that our editors take seriously.  and yet, i’m a little at a loss this time.  i’ve started and stopped this opening more times than i can count, and i think i know why.  while i would like to have written something of substance like my coordinating editors before me, what i’d really like to do is tell everyone a story….one that has been rattling around in my head for a while and has become more insistent as time has gone on but has pushed itself into the forefront recently when i was asked a question by a colleague just starting out.  so, i hope that you won’t mind indulging me. all my professional career, i’ve worked in an academic library.  and i’ve done what academic librarians do.  i publish and present, i travel, i serve in a variety of capacities within a number of communities – while i do these things because they are part of what it means to be successful in an academic environment; if i’m honest, i also do this work because i happen to enjoy what i do and the people that i’ve had the opportunity to work with.  most days, i feel like i get to make a small difference in my little part of the world, and on my best days, i can see it.  but lately, i’ve been starting to notice the passing of time and everything that it takes away.  looking around the profession, i see very few of the same faces that started with me almost 18 years ago.  most are no longer in libraries – they’ve moved outside of the profession because the cost of being in libraries got too high without a significant enough return.  in fact, i just had this conversation this week with someone that has been a long-time mentor.  they’d become tired, wondering if the work they had dedicated their lives was really making a difference at all.  and it got me to wonder, what is the price that we all pay to be in this profession.  what price did i pay to be a librarian? in my case, i’ve paid in time.  like many in the code4lib community, we become invested in our work to the point that it becomes tied up in our professional identities.  it’s a high cost…too high, i would argue.  for many of my friends who have left the profession, the cost was burning out and fleeing the profession.  for others, it can be disorienting, as projects inevitably come to an end.  when your professional identity is so closely tied to a single effort, how do you fill that hole and start over?  maybe some of you are in this space right now and are asking if the project or effort was worth the cost of your professional identity.  i don’t have an answer to that question, but it’s one i wonder myself when i think about it.  but if i’m honest, the highest price i’ve paid for my professional life is in time…time that i realize i can’t have back as i start to count down the days to my oldest son’s graduation.  and it’s time, that i’m finding myself holding onto more tightly as i think about the work that i do and the commitments that i make. and it was a question…a question i get asked often by new librarians looking for advice as they are starting out – how much is too much – that ultimately got me to write.  we have so many talented young professionals in our midst, who are well on their way to being the voice of this community and they are finding themselves stretched thin.  in a profession that places service on a pedestal, that creates positions that ask librarians to be all things to all people and encourages personal sacrifice as the price to professional service.  in that environment, what i offer to those that ask this question, what i offer to you all now, is something simple…learn to be a selfish librarian.  probably one of the most important skills that most often gets learned through years of experience is how to say no in our professional lives.  it took me an embarrassingly long time to feel ok passing up opportunities, or projects, or requests for help.  harder still, was admitting that i couldn’t be/do what everyone wanted/needed from me and having that be ok.  our profession thrives on service work and is built on the foundations of those that have been burnt out from too many commitments.  i’m a librarian – it’s how i identify myself, and it is very much a part of how i see myself.  but it is not all that i am…and i’m ashamed to admit how long it took for me to realize that in my zeal to be good in my professional life, it was placing everything else out of balance.  ironically, it was only in admitting that, and saying no more often, that allowed for deeper engagement in the communities that are important to me. and with that, i’d like to encourage everyone to dig into issue 39 of the journal.  the editorial committee received a number of exceptional proposals, and selected articles covering a wide range of topics.  from analytics (ship it: logistical tracking of ill physical loans and using r and the tidyverse to generate library usage reports) to discovery (microdata in the ir: a low-barrier approach to enhancing discovery of institutional repository materials in google) to processing (approaching the larges ‘api’: extracting information from the internet with python) to a touch of the experimental (getting real in the library: a case study at the university of florida) and much more.  the journal has a little something for everyone this issue.  so, please, on behalf of the code4lib journal editorial committee, we invite you to read, learn, and engage with this issue and the topics presented.   –tr             subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – crafting linked open data for cultural heritage: mapping and curation tools for the linked jazz project mission editorial committee process and structure code4lib issue 21, 2013-07-15 crafting linked open data for cultural heritage: mapping and curation tools for the linked jazz project this paper describes tools and methods developed as part of linked jazz, a project that uses linked open data (lod) to reveal personal and professional relationships among jazz musicians based on interviews from jazz archives. the overarching aim of linked jazz is to explore the possibilities offered by lod to enhance the visibility of cultural heritage materials and enrich the semantics that describe them. while the full linked jazz dataset is still under development, this paper presents two applications that have laid the foundation for the creation of this dataset: the mapping and curator tool, and the transcript analyzer. these applications have served primarily for data preparation, analysis, and curation and are representative of the types of tools and methods needed to craft linked data from digital content available on the web. this paper discusses these two domain-agnostic tools developed to create lod from digital textual documents and offers insight into the process behind the creation of lod in general. by m. cristina pattuelli, matt miller, leanora lange, sean fitzell, and carolyn li-madeo introduction this paper presents methods and tools created as part of linked jazz [1], a project that explores innovative ways to enhance the discovery and interpretation of cultural heritage through the application of linked open data (lod) technology to digital archives of jazz history. with the overarching goal to help make visible the rich and diverse network of personal and professional connections among jazz artists, the project serves as a case study of lod creation practices and contributes to the growing body of research on lod in libraries, archives, and museums (lodlam) [2]. the amount of digital heritage data is growing at an exponential rate, and lod technology has emerged as a promising approach to enhance its discovery, interpretation, and use. lod is a recommended best practice for connecting distributed data across the web (heath & bizer, 2011). developed as a w3c project and spurred on by tim berners-lee, lod has taken off as a new technology for extending the traditional web. the promise of lod lies in its concrete functionality: it provides a unifying and open publishing framework that enables data interlinking and facilitates data interoperability, integration, sharing, and reuse. in the context of cultural heritage, lod has the potential to open unprecedented opportunities for information discovery and new approaches to inquiry. there is a need within the lod community to share the results of prototype tests and lessons learned so that best practices can be refined and the rich opportunities for using lod can be demonstrated. to this end, linked jazz has developed methods and tools for generating a dataset of rdf statements that represents personal and professional relationships among jazz musicians as they are described in interview transcripts from jazz history collections. the project has developed over several stages and required the creation of a series of tools to work toward our objectives. it should be noted that the linked jazz tools, while created in the context of digital archives of jazz history, are domain-independent and can be easily transferred to different application contexts. although a series of different applications were created as part of this project, this paper focuses specifically on two that served as the preparatory foundation upon which a complex dataset will ultimately be built: the mapping and curator tool and the transcript analyzer. the methods and applications examined here deal with some of the key aspects and inherent challenges of lod development: data preparation and analysis as well as data curation. in the crafting of lod, these two tools and the uris they create provide the necessary foundation for building the final dataset. mapping and curator tool linked jazz employs lod technology to reveal the social and professional relationships among the community of jazz musicians as described in interview transcripts available through digital archives. at the core of the project is the creation of an rdf dataset that describes these relationships as lod. publishing lod requires a few essential components, the first of which is that each entity must have a unique resource identifier (uri) (berners-lee, 2006). this first step was accomplished by creating the linked jazz name directory—a directory of personal names of jazz artists paired with uris that serves as the foundation for various applications within the project. the name directory was created by ingesting extract files from dbpedia version 3.8 [3] and bibliographic name authority files into the mapping tool. the resulting dataset can then be refined using the curator tool. the mapping and curation process is comprised of two parts: data processing and human curation via an interface that allows a user to interact with that data. the data processing, described in detail below, is accomplished through a series of python scripts that ingest the lod extracts [4]. the first step toward building the name directory was to extract files from dbpedia. dbpedia flat file n-triple extracts were mined and filtered through article category titles, infobox properties, and rdf:type to create a smaller dataset of individuals in the jazz community. a detailed description of the first phase of the creation of the directory is provided in pattuelli (2012). the resulting linked jazz name directory [5] consists of close to 9,000 individuals represented by literal triples. an example is provided in figure 1. <http://dbpedia.org/resource/mary_lou_williams> <http://xmlns.com/foaf/0.1/name> "mary lou williams" figure 1. example of a literal triple describing the entity name “mary lou williams.” the process used to create the linked jazz name directory reflects the methodology the linked jazz team has employed whenever working with large lod datasets. to enable flexibility in how the data is processed and utilized, we generated smaller, domain-specific datasets from large, unwieldy lod data extracts such as the ones derived from dbpedia. the name directory was generally accurate and extensive for common or preferred names of jazz artists, which tend to be consistently present in dbpedia. variant forms of names (e.g., nicknames, aliases, birth names) were largely missing, however. to enrich the name directory with alternate names, we mapped the uris in our name directory to a large lod name authority, the library of congress name authority file (lc/naf). by mapping to lc/naf, we also connected our name directory to the virtual international authority file (viaf) [6], which contains further alternate names from libraries across the globe. authority files play a key role in lod development as they provide reliable identifiers for entities, which otherwise would be hard to identify and disambiguate. mapping uris between authorities, however, is a difficult task. to make the mapping manageable, we downloaded the lc/naf dataset and then trimmed the large data extract (30gb) to include only simple knowledge organization system (skos) data for personal name entities. this was accomplished through filtering on rdf:type of madsrdf:personalname. next, we examined entities based on personal name, birth date, and death date. two entities—a dbpedia entity and an authority file entity—that matched in all three of these values were considered an exact or perfect match. because our linked jazz name directory is domain specific and only a subset of dbpedia, perfect matches were limited. when a perfect match was not available, the mapping tool took additional steps to find a partial match. to optimize matching, we relied on a whitelist of domain specific terms such as “jazz,” “blues,” and related terms associated with a full lc record, using the datasets represented in both metadata authority description schema (mads) and skos. we did this by dynamically downloading the full record through the library of congress linked data service api [7] for the targeted individual. this method eases the whole process by sensibly reducing computational space and the power required in the mapping process. the full matching process is shown in figure 2. figure 2. flowchart of the backend mapping process used to create and refine the linked jazz name directory. because our dataset was relatively small and domain-specific, we could achieve a fairly high rate of success. we were able to automatically match about 85% of the entities in our name directory to the corresponding authority records. mapping results were automatically grouped into six different levels of confidence: perfect: preferred name, birth, and death dates all match. high: preferred name, birth, or death dates match. medium: preferred name matches and whitelisted terms found. low: preferred name matches, no whitelisted terms found. one to many: preferred name matched to many other names, disambiguation needed. none: no possible matches found. the more assumptions the process needed to make to achieve a successful match, the lower the level of confidence the mapping result was assigned. another fundamental aspect of the development of linked jazz is the use of human intervention to complement algorithm-based methods. while the mapping tool is highly automated, we also provide the ability to manually refine the dataset through the curator component of the tool. the curator is a collaborative, web-based interface, which sits on top of the name directory (see figure 3). this interface enables the curation of the personal names included in the name directory through their mappings to lc/naf and lc/naf’s intrinsic connection with viaf. using this interface a user can disambiguate individuals by connecting them to the correct lc/naf authority record, verify their mapping, or remove names of individuals from the directory entirely. a public demonstration of an early version of the prototype is available at http://linkedjazz.org/public_demo_mapping. figure 3. curator tool showing the mapping of the pepper adams entity name to the library of congress name authority. transcript analyzer after completing the directory of jazz musicians’ names and enhancing it through the mapping and curation processes, the next stage of the project entailed the preparation of the digital text of interview transcripts from jazz history collections. this was carried out through our next tool, the linked jazz transcript analyzer (figure 4). the linked jazz name directory played a key role in preparing these interview texts: it was used for identifying connections among jazz artists through string matching of the names mentioned in the interviews. name citations found in the transcripts provided the basic units for creating a first layer of linkages between musicians. these connections were recorded as rdf triples using the predicate rel:knows_of. the assumption underlying this strategy was that if a musician mentions another musician in an interview, this musician at least knows of the other musician in some way, be it as a friend or acquaintance or just by having knowledge of the other musician. identifying the nature of relationships among the musicians and representing them semantically is the next step in the development of linked jazz. a crowdsource-driven approach was chosen to assist with the analysis and classification of relationships. to enable this, the crowdsourcing application linked jazz 52nd street [8] was developed. this tool provides an interface for users to analyze excerpts of text from interview transcripts and discern the social relationships shared by the musicians mentioned. a few preliminary activities need to be performed on the text of the interview transcripts to allow them to be processed by the crowdsourcing tool: named entities must be identified in interview transcripts and the document must be structured into discrete segments of questions and answers. to support the preparation of the text, the transcript analyzer was developed to automatically identify personal names cited in interview transcripts and map them to the linked jazz name directory and name authority files. figure 4. interface of the transcript analyzer with the names identified in the transcript shown on the left, the text of the transcript on the right, and the backend user tool bar above. at its core, the tool performs automated named-entity recognition enabled through the use of the natural language toolkit (nltk 2.0) platform [9]. along with detecting personal names, the transcript analyzer is also capable of recognizing other types of entities. these entities, such as locations, businesses, and even song and album titles, can be manually marked as “other names” if the automated name recognition did not already do so. one feature of the transcript analyzer critical to the preparation of the transcripts is its capability to identify partial names (e.g., first names or nicknames) and link them to the appropriate entity mentioned within a certain proximity, typically a sentence or two before. for example, if “dizzy” appears a sentence after the full name “dizzy gillespie,” that instance of “dizzy” will be automatically considered a reference to be associated with the entity “dizzy gillespie.” however, partial matches can be reviewed for accuracy, and suspected partial name matches that do not have a full name in proximity are presented to the user for manual matching. there are a variety of instances, however, where a name cannot be detected or correctly identified. for example, the name might be misspelled or a variant form of it might be used. some automated processes are used to correct these errors, such as levenshtein distance comparison, yet most are beyond the scope of automated fixes. in these instances, the system allows for manual intervention, and the incorrect or alternate name can be selected and associated with the appropriate entity in the linked jazz name directory. through the transcript analyzer, a user can add or remove names and correct any spelling errors. the system remembers user input (addition, modification, or deletion of entities) and leverages this information to process future transcripts more efficiently. the transcript analyzer is also able to identify names that are not yet in the linked jazz name directory and therefore have not yet been mapped to an authority. in this case, the transcript analyzer enables the user to locate a uri associated with the name and enter it into the name control “authority” panel (see figure 5). commonly used sources for uris include dbpedia, the library of congress linked data services dataset, or the domain-specific musicbrainz. the user can also search these sources for the name in question directly from the name control panel. when obscure artists are cited in interviews whose names cannot be found in these external sources, a new uri can be minted for that entity that will be hosted on the linked jazz namespace. the name control panel makes it possible to enter the url to the information source where the name in question was identified, such as an online encyclopedia. the user can also add notes if clarification is needed. figure 5. the transcript analyzer’s name control panel with a field to enter an authority uri, mint a new uri, and search authorities. in addition to performing named-entity recognition tasks, the transcript analyzer also breaks interview content down into meaningful question and answer segments, which are later employed in the linked jazz 52nd street crowdsourcing tool. based on regular expression patterns, the transcript analyzer splits the transcripts into a question and answer format, attributing blocks of text either to the interviewer or the interviewee. the tool can be dynamically trained on different document layouts, allowing it to process transcripts with multiple interviewees. the transcript analyzer tags each question and answer section with the names that were mentioned within each text segment. if a name occurs multiple times in the same question and answer segment, all mentions will be attributed to the same entity. this data is stored on the server in a configuration file for each transcript as well as at the global level. this allows for a stateful incremental workflow. project team members can easily interact with the transcript analyzer as a collaborative web-based tool, as no programming skills are required to fulfill basic tasks. once an analysis is complete (i.e. all instances of names have been mapped to authorities, all uris have been minted where necessary, and the text has been broken down into questions and answers), the final step is to publish the transcript, which immediately becomes available on the linked jazz 52nd street website and is viewable by the public. although crowdsourcing users can now begin to work on these transcripts, it is always possible to access published transcripts in the transcript analyzer to modify, update, and reprocess them if necessary. the transcript analyzer demonstrates once again that linked jazz’s method of enhancing automated processes with human curation is effective for developing high quality lod semantics. by combining automated and manual efforts, the transcript analyzer performs several important curatorial tasks within the linked jazz project. the mapping performed through the transcript analyzer enables the user to enrich the linked jazz name directory with accurate personal names that would be missed if only relying on the mapping tool. the edits made by transcript analyzer users are formatted as rules, which the user can edit via the backend user toolbar (see figure 4.1). these rules range from simple text strings flagged for the processer to ignore, to full <owl:sameas> statements informing the system of the authority mapping of a newly coined uri for an individual added to the linked jazz namespace. after a transcript is processed, all rules—with the exception of local “ignore” rules—become global rules that are then applied to the entire project. thus, manual work complements and informs the automated process, making the entire flow of text processing more efficient over time. manual intervention makes it possible to overcome the typical limitations of precision and recall common to traditional named-entity recognition tools. human contribution through the creation of rules enables the transcript analyzer to incrementally improve its capacity to recognize names. over time it fosters the ability to: locate new identities, semantically enhance the dataset through mapping to authorities, and, create reputable new uris. as a result of this combined method of processing transcripts, a more extensive set of names will be available to the crowdsourcing user, which will ultimately enhance the completeness and accuracy of the project’s outcome, the rdf dataset and the representation of the social network of jazz musicians. moreover, the transcript analyzer’s ability to track and apply user-generated rules, both locally and globally across transcripts, facilitates an easy working environment for teams to collaborate on backend processes remotely and independently of each other. this versatile tool could be applied to different domains and be beneficial to a wide range of lod initiatives, provided the documents for analysis are rich in text that is in a readable, digital format. this tool opens up new possibilities for scholars, historians, and students to interact with and utilize open access transcripts. by employing different or multiple vocabularies, the analyzer could be used to create rdf triples representing a variety of entities beyond just personal names, thus offering a tool that can support rich and heterogeneous interlinking. conclusion as the amount of digital cultural heritage data continues to grow at an exponential rate, there is a call for new strategies and applications to enhance their discovery, interpretation, and use. the application of lod technology to cultural heritage content holds enormous potential to answer this call. linked jazz explores and develops methods and tools that can open new pathways for the use of cultural heritage materials in the digital age. with the goal of sharing our experience so far, this paper showcased a set of innovative analytical and curatorial tools that facilitate the creation of sound and rich lod semantics and serve as the basis for building effective lod applications. a key part of our approach to the development of these tools is to complement automated processes with human contributions and curation. while the tools described here were created to support the development of linked jazz, they are domain-agnostic and thus can be transferred and used in a wide range of application contexts. our next step is to conduct performance testing on the tools and to later make them freely available to developers and the general public. as lod technology continues to mature and more stable tools become available, it will be possible to streamline methods and continue to explore the unprecedented opportunities that lod offers for cultural heritage data discovery and interpretation. acknowledgements the linked jazz project was initially funded through an oclc/alise grant. we gratefully acknowledge our former team members chris weller and ben fino-radin for their contributions to the project. notes [1] http://linkedjazz.org/ [2] linked open data in libraries, archives, and museums (lodlam) is a community of information professionals working to bring linked open data into libraries, archives, and museums and make it usable for the larger web community. more information can be found at http://lodlam.net. [3] http://dbpedia.org/downloads38 [4] these scripts and instructions on their use are available at https://github.com/thisismattmiller/linked-jazz-name-directory. [5] http://linkedjazz.org/data/jazz_directory_aug_2012.nt [6] http://viaf.org [7] http://id.loc.gov/techcenter/searching.html [8] http://linkedjazz.org/52ndstreet [9] http://nltk.org/ references berners-lee, t. (2006). linked data. design issues. retrieved 9 april 2013 from http://www.w3.org/designissues/linkeddata.html heath, t. & bizer, c. (2011). linked data: evolving the web into a global data space. synthesis lectures on the semantic web: theory and technology, 1:1, 1-136. morgan & claypool. available from: http://dx.doi.org/10.2200/s00334ed1v01y201102wbe001 pattuelli, m.c. (2012). personal name vocabularies as linked open data. a case study of jazz artist names. journal of information science, 38(6), 558–565. available from: http://dx.doi.org/10.1177/0165551512455989 about the authors m. cristina pattuelli (mcpattuel@pratt.edu) is the project director of linked jazz and an associate professor at the school of information and library science at the pratt institute, new york. her research focuses on information organization and knowledge representation methods and tools applied to information systems. her current area of research is semantic web technologies applied to cultural heritage resources. matthew miller (thisismattmiller.com) is a developer for the nypl labs at the new york public library. he holds a dual master’s degree in library and information science and history of art from the pratt institute. leanora lange (llange@cjh.org) is a processing archivist for the center for jewish history and library associate for teachers college, columbia university. she holds a ms in library and information science from the pratt institute and an ma in german from the university of illinois. sean fitzell (sfitzell@pratt.edu) is currently working towards a master of library and information science at pratt institute with a focus on information architecture, knowledge management, and the creation of platforms for efficient information sharing. he’s also written about jazz, film, history, and other topics for many publications. carolyn li-madeo (cmadeo@pratt.edu) is a mlis candidate at pratt institute. she holds a ba in history, poetry and book arts from hampshire college in amherst, massachusetts. she currently works at a college library in brooklyn, ny and has worked on archival processing projects in academic and private collections. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – hathitrust ingest of locally managed content: a case study from the university of illinois at urbana-champaign mission editorial committee process and structure code4lib issue 25, 2014-07-21 hathitrust ingest of locally managed content: a case study from the university of illinois at urbana-champaign in march 2013, the university of illinois at urbana-champaign library adopted a policy to more closely integrate the hathitrust digital library into its own infrastructure for digital collections. specifically, the library decided that the hathitrust digital library would serve as a trusted repository for many of the library’s digitized book collections, a strategy that favors relying on hathitrust over locally managed access solutions whenever this is feasible. this article details the thinking behind this policy, as well as the challenges of its implementation, focusing primarily on technical solutions for “remediating” hundreds of thousands of image files to bring them in line with hathitrust’s strict specifications for deposit. this involved implementing htfeed, a perl 5 application developed at the university of michigan for packaging content for ingest into hathi trust, and its many helper applications (jhove to detect metadata problems, exiftool to detect metadata issues and repair missing image metadata, and kakadu to create jp2000 files), as well as a file format conversion process using imagemagick. today, illinois has over 1600 locally managed volumes queued for ingest, and has submitted over 2300 publicly available titles to the hathitrust digital library. by kyle r. rimkus & kirk m. hess background the university of illinois at urbana-champaign (uiuc) library has a rich history in digitizing print collections. in 2007, it formed a partnership with the internet archive to scan public domain volumes from its general collections on-site, an ongoing project which has resulted in open online access to over 50,000 items (https://archive.org/details/university_of_illinois_urbana-champaign). at the same time, it began participating in the google books (http://books.google.com) project to digitize additional public domain holdings. in 2008, uiuc became a founding member of hathitrust (http://www.hathitrust.org/), a partner-driven organization established in part to allow libraries to more closely steward digital surrogates of print collections being produced by google books and other large-scale digitization projects. to complement these efforts, the uiuc library also manages its own internal digital production unit (founded as digital services and development and later reconstituted in as digital content creation) that works to digitize materials such as rare books, manuscripts, and special collections which, due to their format, rareness, or condition, do not pass through the internet archive or google books workflows. in addition, the library’s preservation unit has worked with trusted vendors since 2007 to digitally reformat embrittled materials from the circulating general collections. because this locally managed content did not initially have a home with google books, the internet archive, or the hathitrust digital library, the uiuc library developed and later revised its own digitization and file format specifications for monographs, and established a solution comprised of open source tools and in-house scripts to provide web access to them. the primary components of this local solution are a landing page with a persistent handle, downloadable pdf full-text, a page-turner view, and a plain text view of the monograph. this local solution has provided open access to works in the public domain, and restricted access to authenticated uiuc patrons for items digitized for preservation purposes but still protected by copyright restrictions. figure 1: a sample volume in the legacy delivery format (http://brittlebooks.library.illinois.edu/brittlebooks_open/books2009-05/hach0001parhac/hach0001parhacv01900i00003/). once hathitrust made its “htfeed” suite of perl scripts (http://www.hathitrust.org/ingest_tools) available in 2012 to facilitate the ingest of content managed by partner institutions, however, the uiuc library decided to reconsider its own reliance on the solution described above in favor of the hathitrust digital library for certain materials. this policy review was catalyzed by hathitrust’s successfully withstanding a trusted repository audit certification review (http://www.hathitrust.org/trac) administered in 2011 by the center for research libraries, further assuring partner institutions of hathitrust’s ability to reliably steward their digital content in the long-term. with anecdotal evidence that scholars are beginning to strongly favor large-scale, readily accessible services such as the hathitrust digital library over dispersed, difficult-to-locate “boutique” collections of digital materials, the uiuc library decided to contribute as many of its locally digitized volumes to hathitrust as possible. implementation what does it take to submit locally managed content to the hathitrust digital library? the requirements are strict, and the process has multiple steps to trip one up along the way. they are best outlined on the links from the hathitrust’s “getting content into hathitrust” webpage (http://www.hathitrust.org/ingest). these steps will be further elaborated later in this paper. to simplify, however: become a hathitrust partner institution. access to the hathitrust digital library is open to anyone. however, only partner institutions may deposit items. these are generally large academic research libraries with significant print collections. partners must be willing to pay yearly membership fees and provide metadata to hathitrust on print holdings, among other requirements. do your paperwork and get it signed. partner institution fills out a “digital asset submission inventory” or “dasi” form (https://docs.google.com/document/d/10nt7hopw5xvtvdu1tw4yc9gsoswmwae952tpdvb0kbu/edit?authkey=cpodvmgf) detailing how many items it wishes to submit, and what their rights status is. this form must be approved by someone with signatory authority at the partner institution prior to ingest. modify your metadata as needed and send it to california. partner submits marcxml records for all items to be ingested to the california digital library’s zephir (http://www.hathitrust.org/bib_data_submission) service. prior to submission, partner makes any changes necessary to xml records as required by hathitrust’s bibliographic metadata specifications (http://www.hathitrust.org/bib_specifications). make sure your files are compliant and package them. partner submits file packages according to specifications in the “cloud packaging and validation service” overview document (https://docs.google.com/document/d/1oq0skaioh8xi0hvvxg4trybrpuptdv4qa70d8ghrltu/). all page-level files must be numbered with eight digits and padding zeros. file packages include:                      a. image files in jpeg2000 or tiff format                      b. page-level ocr transcript files in utf-8                      c. if available, page-level word coordinate files                      d. a checksum file listing every file in package and its corresponding md5 hash value                      e. a valid meta.yaml file to provide additional administrative or technical metadata the intricacies of these requirements are explained in the cloud packaging and validation service document cited above. of all the requirements, however, the provision of valid image files has proven the most onerous to uiuc. hathitrust enforces very strict technical metadata requirements on image files. the table below, for example, details uiuc’s local jpeg2000 image profile as modified to meet hathitrust’s requirements for embedded technical metadata. embedded metadata tag value jp2 header metadata 1. compressionscheme jpeg-2000 2. format jpeg-2000 3. mimetype image/jp2 4. brand (or “majorbrand”) jp2 5. minorversion 0 6. compatibility (or “compatiblebrands”) jp2 7. xsize (or “imagewidth”) matches xmp header metadata/tiff:imagewidth (see #16 below) 8. ysize (or “imageheight”) matches xmp header metadata/tiff:imageheight (see #17 below) 9. numberoflayers mandatory, but no required value 10. numberdecompositionlevels mandatory, but no required value 11. bitspersample 8 for grayscale, (8,8,8  [24-bit]) for srgb 12. xsamplingfrequency generally between 300/1 and 600/1, matches xmp header metadata/tiff:xresolution (see #23 below) 13. ysamplingfrequency generally between 300/1 and 600/1, matches xmp header metadata/tiff:yresolution (see #24 below) 14. samplingfrequencyunit mandatory, matches xmp header metadata/samplingfrequencyunit (see #25 below) xmp header metadata 15. xpacket field w5m0mpcehihzreszntczkc9d 16. tiff:imagewidth matches jp2/xsize (see #7 above) 17. tiff:imageheight matches jp2/ysize (see #8 above) 18. tiff:bitspersample 8 for grayscale, (8,8,8 [24-bit]) for srgb 19. tiff:compression 34712 (=jpeg2000) 20. tiff:photometricinterpretation 2 for srgg, 1 for grayscale 21. tiff:orientation 1 (horizontal/normal) 22. tiff:samplesperpixel 3 for srgb, 1 for grayscale 23. tiff:xresolution generally between 300/1 and 600/1, matches jp2/xsize (see #12 above) 24. tiff:yresolution generally between 300/1 and 600/1, matches jp2/ysize (see #13 above) 25. samplingfrequencyunit mandatory, matches jp2samplingfrequencyunit (see #14 above) 26. tiff:resolutionunit 2 (inches) 27. dc:source object $id/$filename 28. tiff:datetime formatted yyyy:mm:ddthh:mm:ss, for example 2010:05:24t13:45:30 29. tiff:artist university of illinois at urbana-champaign library 30. tiff:make make of camera/scanner 31. tiff:model model of camera/scanner as sensible as these requirements are, the uiuc library found that none of the files it intended to submit to hathitrust met all of them, and that non-compliance differed from one batch of files to the next, depending on the imaging hardware and workflow that produced them. bringing uiuc’s files into compliance required a level of advanced file surgery (often mistakenly referred to as “massage”) skills not necessarily in the toolkit of most librarians. to aid partners in the pre-ingest process, hathitrust provides a tool called htfeed (download link available at http://bit.ly/1fphfai), a perl 5 application developed at the university of michigan for packaging content for submission. htfeed splits file preparation into stages to isolate and remediate problems, and allows users to code custom stages to address problems unique to their own files. when the uiuc library began working with the tool in 2013, its staff had limited knowledge of the perl programming language, and it hired a student programmer to customize the application. htfeed is configured to run in a linux server environment, and requires a strong knowledge of linux and perl to effectively customize and support. htfeed relies on a number of helper applications, such as jhove (http://sourceforge.net/projects/jhove/) to identify embedded technical metadata problems, exiftool (http://www.sno.phy.queensu.ca/~phil/exiftool/) to repair missing technical metadata in image file headers, and kakadu (http://www.kakadusoftware.com/) to create jpeg2000 files within the htfeed framework. in addition to customizations of the htfeed tool, uiuc also developed a remediation process that uses imagemagick to convert invalid jpeg2000 image files to tagged image file format (tiff) with an enumerated srgb colorspace prior to conversion by kakadu to a hathi-compatible jpeg200.[1] by december 2013, uiuc thought it had remediated and submitted 1400 locally digitized volumes for ingest, but they were rejected by hathitrust for nonconformance to their requirements. troubleshooting these problems proved difficult at first, especially prior to the 2014 introduction of hathitrust’s cloud validation tools for single images (http://babel.hathitrust.org/feed/validate_image.html) and complete object packages (http://bit.ly/1jbomic), both of which provide precise reports on exact problems with files prior to submission. with a better understanding of hathitrust’s true ingest requirements afforded by these tools, the uiuc library began to write its own tools in the .net programming language in january 2014 to further remediate its 1400 volumes that had failed upon ingest. image file metadata, it turned out, was not the only problem. uiuc also had many books without page-level ocr text files in utf-8, and rectified this using the abby fine reader (http://finereader.abbyy.com/) software. why did this process prove so challenging? uiuc’s book digitization programs have gone through a variety of changes over the years in file format standards and the hardware and software used in production workflows, a problem when consistency of technical metadata is important. for example, uiuc’s brittle books program has employed at least two vendors in the seven years of its existence, each of whom used a variety of imaging equipment and produced master files in different formats (tiff and jpeg-2000). in addition, uiuc’s digital content creation unit and its precursor digital services and development have utilized a variety of different workflows over the years tied to different types of imaging equipment (flatbed scanner, copystand, sheet-fed scanner). as representative cases, the three examples below furnish a detailed view of packages that were rejected by hathitrust ingest tools and required a significant amount of invasive remediation. example 1: illinois magazine and yearbook archive – illinois chemist (http://catalog.hathitrust.org/record/100072274) the illinois chemist was a quarterly journal published by the uiuc chemistry department 1915-1925. in 2008, the first 5 volumes were digitized using an i2s digibook suprascanner a0 at the oclc preservation service center for ingest into the now inactive illinois magazine and yearbook archive (imay), originally available in an access system called olive software active magazine. legacy package hathitrust requirement solution image files 300dpi grayscale tiff continuous tone images must be in the jpeg 2000 format image files modified using hathitrust remediate perl scripts. image files numbered according to uiuc convention image files numbered sequentially as eight-digit file names with padding zeros image files renumbered using custom vb.net tool ocr files in prxml format page-level ocr transcript files encoded in utf-8 new ocr files generated using abbyy fine reader 11 no marcxml file marcxml file conformant to hathitrust bibliographic specifications (http://www.hathitrust.org/bib_specifications) marcxml file generated using locally developed  z39.50 service and getmarc no checksum file checksum file listing every file in package and its corresponding md5 hash value package checksummed and zipped using locally developed vb.net utility, hathizipandsubmit. example 2: project unica – mensa regum, or, the table of kings (http://catalog.hathitrust.org/record/100008881) “project unica is an initiative of the rare book & manuscript library of the university of illinois at urbana-champaign to produce high quality digital facsimiles of printed books that exist in only one copy”[2]. over the years, uiuc has used a variety of imaging hardware to serve the project, and named files using a locally developed convention to indicate scanning order, often with a letter to indicate non-textual material such as the spine or cover (e.g. 0000001000000b.jp2) and either a number for textual material or two letters (e.g. 0000430000ar.jp2). uiuc scanned most books in full color, using an embedded 48bit adobe photoshop rgb colorspace and a restricted embedded icc profile, frequently using a nikon d200 camera. no xmp metadata was stored in the files, only jpeg2000 tags. these qualities made project unica files particularly difficult to remediate. legacy package hathitrust requirement solution jpeg2000 image files with adobe photoshop rgb colorspace and a restricted embedded icc profile srgb color space, no embedded icc profile files converted to intermediary tiff using imagemagick, color profiles stripped and replaced with an enumerated 24bit srgb colorspace or 8bit grayscale colorspace, and then converted back to jpeg-2000[3] using kakadu kdu-compress using the recommended settings outlined in the htfeed application image files numbered according to project unica convention image files numbered sequentially as eight-digit file names with padding zeros image files renumbered using local scripts no xmp metadata xmp metadata must be present, and must meet hathitrust requirements missing xmp metadata recreated and added into the intermediate images using hathitrustremediate. some packages had the artist and camera make & model added to yaml file.  no ocr file page-level ocr transcript files encoded in utf-8 new ocr files generated using abbyy fine reader 11 no marcxml file marcxml file conformant to hathitrust bibliographic specifications (http://www.hathitrust.org/bib_specifications) marcxml file generated with getmarc z39.50 service. no checksum file checksum file listing every file in package and its corresponding md5 hash value package checksummed and zipped using hathizipandsubmit. example 3: brittle books – de isocratis paedogogia (http://catalog.hathitrust.org/record/100008484) the brittle books project, managed by uiuc library’s preservation unit, scans and reproduces physically damaged books to provide access to users. legacy files were named with two portions, a display order in the first 6 characters and an alphanumeric code where alpha characters indicated not textual content. files were ocr’d into a latin-1 iso-8859-1 plain text file, as well as an adobe pdf and tei.xml. page images were bitonal, identified as having a 1bit srgb enumerated colorspace, and scanned by an external vendor at 600dpi. no camera make, model, artist or tiff metadata was embedded by the vendor. legacy package hathitrust requirement solution jpeg2000 image files, 1bit srgb enumerated colorspace, and scanned by an external vendor at 600dpi must be 8-bit greyscale or 24-bit srgb jpeg2000 masters decompressed and converted to intermediate tiff images using imagemagick, converted to 8bit grayscale, new jpeg2000 masters created using kakadu kdu_compress, using the recommended settings outlined in the htfeed application image files numbered according to brittle books convention image files numbered sequentially as eight-digit file names with padding zeros image files renumbered using local scripts no xmp metadata xmp metadata must be present, and must meet hathitrust requirements missing xmp metadata recreated and added into the intermediate images using hathitrustremediate. some packages had the artist and camera make & model added to yaml file. ocr data embedded in pdf file page-level ocr transcript files encoded in utf-8 ocr text was extracted from pdf file into a single file using pdftotext (http://en.wikipedia.org/wiki/pdftotext) and converted to utf-8, and split into page files using a locally developed vb.net script to match the correct naming scheme for each page. no checksum file checksum file listing every file in package and its corresponding md5 hash value package checksummed and zipped using hathizipandsubmit. packaging workflow steps [4] as is evident from the examples above, package preparation workflows varied considerably based on the files in question. in general, however, workflow steps invariably include: 1. prepare files: the hathitrust cloud ingest method prefers a single folder per objid, named with the objid w/o the namespace. for example, if using an object’s barcode as its unique identifier with object id uiuc.30112108161222, create a folder called 30112108161222 2. copy and create files: the new folder should contain either 600dpi tiff files for bitonal images, or 300, 400, or 600dpi jp2 files for continuous tone images. the file name should be sequential numbers starting with 00000001.jp2 or 00000001.tif. a objid can have both tif and jp2 files. each image file should have a corresponding utf-8 ocr text file with the same filename with the .txt extension. 3. create metadata: two metadata files are required for the cloud ingest format, marc.xml and meta.yml. the marcxml may come from a library catalog z39.50 server with a client such as the one included with marcedit [5]. the marc.xml file must be valid xml with a datafield 955 added with at least one subfield $b for the objid value. optionally, a $v subfield can be added for volume enumeration and chronological information about the item. three common problems are missing or invalid oclc# in 035, an invalid length string in 008, or missing dates in 008 which may cause  problems with rights determination. <datafield ind1=" " ind2=" " tag="955"> <subfield code="b">30112108161222</subfield> <subfield code="v">v.3: no.1 (1894)</subfield> </datafield> example 955$b and 955$v values the meta.yml file requirements are detailed in hathitrust’s sample file (https://docs.google.com/document/d/1iacgd1zgrvxw3e2enuh6nx_h0qv9we_xogqcdxjduxc). note only the capture date and agent are required. an optional pagedata section can be created to detail information about each page such as a label or page number. capture_date: 2008-03-06t11:50:12-06:00 capture_agent: iu meta.yml with required values only 4. validate files:  for help in validating files, rely on the single-image validator (http://bit.ly/1gvd9q7), full-volume validator (http://bit.ly/1jbomic), and htfeed packaging tools, which includes an image validation utility validate_images.pl (http://bit.ly/1fphfai). additionally, you may be required to manually identify problems using your own tools. exiftool is useful for pinpointing problems, and other tools such as jhove can also be used. 5. imagemagick, kakadu, and exiftool remediation examples ideally, all of the files are valid and this step is not necessary. however, if it is, here are some examples of remediation using imagemagick, kakadu and exiftool. [6]               example 1. imagemagick perl (from remediate.pl), remediating colorspace and profile. my $file; #file to test; my $image = image::magick->new; my $old = $image->read($file ); # colorspace should be srgb or grayscale, and profile should be srgb for color images. $property = $image->getattribute("colorspace"); print("colorspace$property\n"); if ( $property ne "grayscale" ) {   $image->set(colorspace=>'srgb');   $image->strip();   $image->profile( '$config_folder/srgb.icm' );   print "---colorspace/icmprofile changed---\n"; } example 2. imagemagick commands to decompress a compressed file, add a srgb enumerated profile, set the colorspace to srgb, the depth to 8, resample to 300dpi with the units per inch. convert $input +compress -profile ./srgb.icm -colorspace srgb -depth 8 -resample 300x300 -units pixelsperinch $output example 3. hathitrust recommended kakadu commands for jp2 creation kdu_compress -quiet -i 00000001.jp2 -o 30112108161222/00000001.jp2 clevels=7 clayers=8 corder=rlcp cuse_sop=yes cuse_eph=yes "cmodes=reset|restart|causal|erterm|segmark" -no_weights -slope 42988 example 4. exiftool perl example (from compress_tif_jp2.pl), updating tags use image::exiftool; sub update_tags {     my $exiftool = shift;     my $outfile = shift;     my $infile = shift;     my $res;     if(defined $infile) {         $res = $exiftool->writeinfo($infile,$outfile)     } else {               $res = $exiftool->writeinfo($outfile)     }     if ( !$res ) {         print("operationfailed");     } } sub main{     foreach my $infile ($argv[0]) {        die("$infile does not exist") if !-e $infile;        # reset remediator;        my $exiftool = new image::exiftool;        my $newfields = {};        my $oldfields = {};        my $outfile = $infile;        update_tags($exiftool,"$outfile");     } } main(); example 5. exiftool – set the xmp-dc:source value for all jp2 files in the current folder exiftool '-xmp-dc:source<30112108161222/$filename' *.jp2 6. generate checksum: a file named checksum.md5. this file can be generated with md5sum on linux or md5 -r on mac os x. windows does not come with a checkusum utility, one option recommended by hathitrust is md5sum from coreutils for windows: (http://gnuwin32.sourceforge.net/packages/coreutils.htm). md5sum * > ~/checksum.md5 7. zip folder zip -r 1751155v3i1.zip 1751155v3i1 hathizipandsubmit: vb.net code snippets for steps 6 & 7 – sub processfolder(foldername as string) lblstatus.text = string.format("processing folder: {0}  folders processed: {1}", "..." & foldername.substring(txtfolder.text.length), lblstatus.tag) if form1.stopping = true then exit sub dim fldrs = directory.getdirectories(foldername) if fldrs.length > 0 then 'there are sub folders to recurse into them txtstatus.writeline("folder: " & foldername) for each fldr in fldrs processfolder(fldr) next else lblstatus.tag = lblstatus.tag + 1 'no subfolders, so process this folder as if it contains a digitized volume if chkchecksum.checked then makechecksum(foldername) end if zipfolder(foldername) if chkdeletefolder.checked then directory.delete(foldername, true) txtstatus.writeline("deleted " & foldername) end if end if lblstatus.text = string.format("processing folder: {0}  folders processed: {1}", "..." & foldername.substring(txtfolder.text.length), lblstatus.tag) txtstatus.writeline("") application.doevents() end sub private sub zipfolder(foldername as string) dim parent = directory.getparent(foldername).fullname dim basename = path.getfilename(foldername) & ".zip" dim newzipfile = path.combine(parent, basename) using zstrm as new filestream(newzipfile, filemode.create, fileaccess.write) using zarc as ziparchive = new ziparchive(zstrm, ziparchivemode.create, false) addfoldertozip(zarc, foldername) end using zstrm.close() end using txtstatus.writeline("created: " & newzipfile) end sub private sub addfoldertozip(zarc as ziparchive, fldrname as string) dim startingfolderlen = fldrname.length path.getfilename(fldrname).length for each fl as string in directory.getfiles(fldrname) dim zent = zarc.createentry(fl.substring(startingfolderlen).replace( path.directoryseparatorchar, path.altdirectoryseparatorchar)) using fstrm as new filestream(fl, filemode.open, fileaccess.read) using zstrm as stream = zent.open fstrm.copyto(zstrm) zstrm.close() end using fstrm.close() end using next for each fldr as string in directory.getdirectories(fldrname) addfoldertozip(zarc, fldr) next end sub private sub makechecksum(newfolder as string) dim setupstream as boolean = false dim outname as string dim strmwtr as streamwriter = nothing 'delete old checksum file if file.exists(path.combine(newfolder, "checksum.md5")) then file.delete(path.combine(newfolder, "checksum.md5")) end if dim files as fileinfo() = new directoryinfo(newfolder).getfiles() dim hashes = new dictionary(of string, string)   for each filez in files if not setupstream then dim foldername as string = path.getdirectoryname(filez.fullname) outname = path.combine(foldername, "checksum.md5") strmwtr = new streamwriter(outname) setupstream = true end if dim ext as string = path.getextension(filez.fullname).tolower if ext.length > 0 then computemd5(filez, hashes) end if next using strmwtr for each h in hashes strmwtr.writeline(string.format("{0} *{1}", h.value, path.getfilename(h.key))) next strmwtr.flush() end using txtstatus.writeline("created: checksum.md5") end sub private sub computemd5(file as fileinfo, hashes as dictionary(of string, string)) using strm as stream = file.open(filemode.open) using hsh as md5 = md5.create() dim data() as byte = hsh.computehash(strm) dim sb as new stringbuilder for i as integer = 0 to data.length 1 sb.append(data(i).tostring("x2")) next hashes.add(file.fullname, sb.tostring) end using strm.flush() end using end sub 8. submit metadata to zephir: requirements and zephir instructions. 9. submit required paperwork to hathi: feedback@issues.hathitrust.org 10. check ingest logs: http://www.hathitrust.org/ingest_logs lessons learned uiuc came away from this project with several key lessons learned. even the most difficult files can be remediated, but the need for a staff programmer with some knowledge of image formats and the perl programming language will make this challenging for many institutions. in addition, if partner institutions manage files in the jpeg2000 format, they will need to develop strong knowledge of its technical metadata elements. while this paper does not address the “is jpeg2000 a reliable preservation format?” question, the authors’ experience with remediating jpeg2000 files for hathitrust ingest continually underscored their own insufficient knowledge of the format’s idiosyncrasies. as a benefit, they and many of the staff at their institution, as well as partner vendors of digital imaging hardware, software, and services, have all come away from this project with a heightened understanding of the way technical metadata is managed in jpeg2000’s jp2 and xmp headers. hathitrust has certainly simplified its process for partner-submitted content with the introduction of cloud validation tools earlier this year. however, many partner institutions will likely find that the process is still a daunting challenge. the authors of this article hope that sharing information about their experience ingesting locally digitized content into the hathitrust digital library will help colleagues at other institutions who have as yet been unable to do so. references software tools abbyfinereader: http://finereader.abbyy.com/ coreutils for windows: http://gnuwin32.sourceforge.net/packages/coreutils.htm exiftool: http://www.sno.phy.queensu.ca/~phil/exiftool/ hathitrust ingest tools: http://www.hathitrust.org/ingest_tools htfeed image remediation tools: http://bit.ly/1fphfai htfeed modifications for uiuc remediation: https://bitbucket.org/kirkhess/hathitrustremediate jhove: http://sourceforge.net/projects/jhove/ kakadu: http://www.kakadusoftware.com/ marcedit: http://marcedit.reeset.net/ pdftotext: http://en.wikipedia.org/wiki/pdftotext hathitrust documentation bibliographic metadata specifications: http://www.hathitrust.org/bib_specifications bibliographic metadata submission: http://www.hathitrust.org/bib_data_submission cloud validation and packaging service: https://docs.google.com/document/d/1oq0skaioh8xi0hvvxg4trybrpuptdv4qa70d8ghrltu/ digital asset submission inventory (dasi): https://docs.google.com/document/d/10nt7hopw5xvtvdu1tw4yc9gsoswmwae952tpdvb0kbu/edit?authkey=cpodvmgf ingest overview: http://www.hathitrust.org/ingest partnership checklist: http://www.hathitrust.org/partnership_checklist partnership eligibility: http://www.hathitrust.org/eligibility_agreements sample meta.yml file: https://docs.google.com/document/d/1iacgd1zgrvxw3e2enuh6nx_h0qv9we_xogqcdxjduxc single image validator: http://babel.hathitrust.org/feed/validate_image.html footnotes [1]. source code: https://bitbucket.org/kirkhess/hathitrustremediate [2]. from project summary at http://illinoisharvest.grainger.illinois.edu/results.asp?searchtype=collectioncontent&newsearch=1&collid=2797&collname=project%20unica [3]. while uiuc took pains to consolidate its workflows to the jpeg-2000 format, hathitrust staff has since made it clear that they are often able to more flexibly handle the ingest of non-conformant tiff files, which may have simplified the steps outlined above. [4]. a summary of the checklist available at the hathitrust site. [5]. marcedit – http://marcedit.reeset.net/ [6]. it’s possible to batch remediate files with photoshop, but through testing uiuc found the use of kdu_compress consistently produced valid jp2 files which validate in the single image validator. about the authors kyle r. rimkus kyle rimkus is preservation librarian at the university of illinois at urbana-champaign library, where he is responsible for coordinating a comprehensive digital preservation program. rimkus@illinois.edu kirk m. hess kirk hess is digital humanities specialist at the university of illinois at urbana-champaign library, and holds a mlis from the university of illinois at urbana-champaign. kirkhess@illinois.edu http://www.library.illinois.edu/people/bios/kirkhess/   subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – editorial: closer to 100 than to 1 mission editorial committee process and structure code4lib issue 51, 2021-06-14 editorial: closer to 100 than to 1 with the publication of issue 51, the code4lib journal is now closer to issue 100 than we are to issue 1. also, we are developing a name change policy. by edward m. corrado the abstract of the first code4lib journal issue that was published over thirteen years ago read: this mission of the code4lib journal is to cover “the intersection of libraries, technology, and the future.” we hope that this journal can be one more contribution to the developing culture of collaboration around library technology, and we welcome you to join in our experiment [1]. as one of the founding editorial committee members back in 2007 when the code4lib journal experiment was first launched, i believed the journal would fill an important niche in the library technology literature. however, i’m not sure i expected it to still be going at issue 51–closer to 100 than to 1. that doesn’t mean we do everything perfectly; from the beginning we have been “making it up as we go along,” [2] like the rest of the code4lib community. for an all-volunteer committee with basically no funding or direct institutional support i think the editorial committee members throughout the years should be proud of their contributions to our profession. i also would be remiss not to mention the numerous authors and, just as importantly, the readers of code4lib journal because without them the journal would be nothing more than a distant memory. talking about things changing as we go along, the editorial committee is currently working on a name change policy. we don’t have the details completely figured out yet but, as some other publishers have done, we plan to base our policy on the “high level principles for name changes in publishing” described by the committee on publication ethics (cope) [3]. as with other name change policies, such as sage publishing’s policy, it is being created to help “address the needs of transgender (trans) and gender non-conforming people who, if unable to update their name, are at an increased risk of harassment and assault, as well as exposure to professional biases (e.g. relating to citation, tenure, and promotion)” [4]. this policy may also benefit authors who have name changes for any other reason including, but not limited to, divorce, marriage, religion, and indigeneity. if this is something that interests you for any reason, please contact the editorial committee or me directly. jonathan rochkind wrote about the first issue saying “we think we’ve got a great first issue. this is due to the great work of our authors, and of the editorial committee” [5]. i believe the same is true of the fifty-first issue. thank you everyone for joining in on the code4lib journal experiment and making it the great success that it is. about the author edward m. corrado (ecorrado@ecorrado.us) is associate university librarian at the naval postgraduate school located in monterey, california, usa. he is writing in his personal capacity; the views expressed here are those of the author and do not reflect those of the u.s. navy, department of defense, or any office of the u.s. government. orcid: 0000-0001-5561-346x. references [1] rochkind, j. (2007). editorial introduction — issue 1. code4lib journal (1). https://journal.code4lib.org/articles/39 [2] rochkind, j. (2007). editorial introduction — issue 1. code4lib journal (1). https://journal.code4lib.org/articles/39 [3] tanenbaum, t., rettig, i., schwartz, h. m., watson, b. m., goetz, t. g., spiel, k., & hill, m. (2021, january 13). a vision for a more trans-inclusive publishing world: guest article. cope. https://publicationethics.org/news/vision-more-trans-inclusive-publishing-world [4] perkins, s. (2021, may 17). our post-publication name change policy, explained. sage perspectiveshttps://perspectivesblog.sagepub.com/blog/author-services/our-post-publication-name-change-policy-explained [5] rochkind, j. (2007). editorial introduction — issue 1. code4lib journal (1). https://journal.code4lib.org/articles/39 subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – a very small pond: discovery systems that can be used with folio in academic libraries mission editorial committee process and structure code4lib issue 57, 2023-08-29 a very small pond: discovery systems that can be used with folio in academic libraries folio, an open source library services platform, does not have a front end patron interface for searching and using library materials. any library installing folio will need at least one other software to perform those functions. this article evaluates which systems, in a limited marketplace, are available for academic libraries to use with folio. by aaron neslin & jaime taylor folio, an open source “next gen” integrated library system (ils)/library services platform (lsp), does not have a front end patron interface for searching and using library materials. any library installing folio will need at least one other software to perform those functions. currently, libraries are using ebsco discovery service (eds, both the “classic” ui and the “new” ui), blacklight, vufind, and possibly home grown systems. other front end discovery systems exist, such as primo, summon, worldcat discovery, vega, and aspen, but no libraries could be identified that had paired these systems with folio. the aforementioned systems actually represent two different kinds of software, with vital differences. one group are the single search discovery layers, which ingest many kinds of metadata from many sources, including electronic resource subscriptions; this group is limited to eds, primo, summon, and worldcat discovery. they are all proprietary systems and include management of an aggregated index of electronic resource metadata. the other group are opac-like front ends, which will only search and display catalog metadata and other local data sources, such as institutional repositories, loaded into their indexes. they do not have an integrated aggregated index of e-resources. these include blacklight, vufind, and aspen, which are all open source systems, and locate, vega, soutron, enterprise, destiny, and tdnet, which are proprietary systems. blacklight and vufind are common in well-resourced libraries. vufind and aspen have some features that are beginning to incorporate electronic resource subscriptions into a single search experience (though those features are much less complex than in the algorithm-driven single search discovery layers); blacklight could theoretically do so with some extra work on its index and other features; and ebsco has plans to integrate locate with eds. regardless, a library would still need to subscribe to one of the proprietary discovery layers’ aggregated index for those features to function. this may present a financial and workload barrier to libraries wishing to implement open source discovery layers, especially higher education libraries, which heavily subscribe to and use electronic resources. single search discovery layers ebsco discovery service (eds) ebsco discovery service, a proprietary system, was launched in 2010 as the front end for ebsco’s electronic resources, and has been expanded to function as a single search interface that includes catalog records, institutional repositories, etc. ebsco’s folio hosting and support services allow for relatively easy integration between the two, on par with the relationships between a company’s proprietary ils/lsp and opac (online public access catalog) or discovery layer. eds is the only single search discovery layer currently being used with folio. many academic libraries and consortia are using, or plan to use, eds with folio. these include the five college consortium (amherst college, hampshire college, mount holyoke college, smith college, and the university of massachusetts amherst, at which the authors work), michigan state university, and missouri state university. while other institutions, such as university of colorado boulder, are or will be using the updated discovery interface, these three institutions are using the “classic” version of discovery. all of these are small consortia of between three and five institutions, and their folio instances are all hosted by ebsco. each has a single folio instance, with multiple ebsco discovery front ends. the five colleges have five eds instances, one per school. michigan has two eds instances for its three campuses. missouri state has separate eds instances for its four campuses and the state historical society. ebsco runs daily oai-pmh loads of catalog updates from the schools’ folio instance to their eds instances. these consortia all use eds’s native “my account” patron functionality for requests, loans, etc., with openathens authentication to it. the five colleges, michigan, and missouri have all reported significant issues with eds since going live with it and folio. among these are lack of test environments, unreliable oai-pmh catalog updates, confusing user experience and overzealous collection of patron data in my account, lack of configurability, and what one librarian expressed as “you can discover things, but you can’t find things.” these difficulties seem to be systemic, rather than only found in one eds installation or at one library, as each consortia describe very similar experiences. it has resulted in patrons feeling like their libraries are “…sending the impression that [we] don’t know what we’re doing or that we don’t care about the patrons’ needs and expectations.” [1] some of these issues are being addressed by ebsco. for example, in june 2023 they reduced the eds my account patron data retention period from 11 months to 35 days, and will be redesigning eds’s gdpr compliance notification and data collection consent experience. however, many patrons may still be uncomfortable with realizing that a company outside of their library is collecting their library use data at all, especially if that library previously used home grown and locally hosted library systems. in ebsco’s defense, their 35 day data retention period combined with the longstanding ability for users to view and delete their data (gdpr rights to access and erasure) within the eds my account interface provides library patrons with significantly more privacy and security than may be offered by other systems. similar retention period information for ex libris or oclc could not be located; ex libris states that primo patron records can only be deleted by the library’s primo administrator and oclc has startlingly little product-specific information about gdpr compliance or other patron data privacy information available in their documentation. [2] [3] the new eds user interface is still under development, even as some libraries are already using it. active development will continue for some time; hopefully ebsco will address the issues that its existing customers have been experiencing with “classic” eds. in the meantime, these three consortia have all expressed desire to investigate alternative or additional systems to ameliorate their eds issues. michigan has gotten the farthest into this effort, having launched an early iteration of vufind in spring 2023. other institutions, such as simmons university, are already live with an eds/vufind/folio ecosystem, where the two front ends serve different purposes instead of having a purely single search patron experience. voluntary product accessibility statements (vpat) for the classic and new eds user interfaces can be found on ebsco connect. [4] [5] these documents indicate that both interfaces either support or partially support all web content accessibility guidelines (wcag) 2.1 level aa criteria. the interface itself supports these criteria, while content often only partially supports them due to the lack of control ebsco has over other publishers’ content. according to ebsco, they assure accessibility to eds and their other products via: completing annual accessibility audits conducted by a qualified third party testing our platforms with both jaws and nvda conducting usability studies with visuallyand physically-impaired users working directly with institutions and their users to address their concerns providing accessibility guides for products like ebscohost and ebsco ebooks [6] primo no folio libraries using primo could be located. according to clarivate, ex libris’s parent company, primo is no longer being sold to non-alma customers. [7] therefore, primo can not be used with folio. the barrier to using primo with folio is purely a business decision on ex libris and clarivate’s part. [8] there is no technical barrier to pairing the two systems. one of primo’s strengths has always been its ability to ingest metadata from a variety of sources and in a variety of formats. folio can export marc metadata via oai-pmh, which is one of primo’s available data import methods. it is possible that a library could install the entire alma/primo ecosystem and also have a folio instance that exports records to primo, though that would probably not make financial or workflow sense in most cases. summon there are no folio libraries using ex libris’s summon. however, marshall breeding’s 2023 library technology report states that summon “can be used with alma or other proprietary or open source ils products.” [9] ex libris confirms that they would be willing to attempt a folio/summon integration if the records exported from folio met the ingest criteria for summon, and that the summon product manager is “always looking for development partners to do some testing and integration work.” [10] [11] records, from any ils/lsp, are ingested into summon via ftp/sftp, which is compatible with folio’s processes. it may be difficult to make some summon features work with folio, for example real-time availability, which queries opac records (which folio does not have, hence this report) instead of referring directly to the ils/lsp. [12] summon uses the same central index as primo. according to ex libris, “summon is continually designed and developed to meet level aa of the w3c web content accessibility guidelines (wcag 2.1) and section 508 of the us rehabilitation act.” [13] the vpat for summon is available on ex libris’s documentation site. [14] worldcat discovery no folio libraries using worldcat discovery could be located, likely because its use is intertwined with other oclc systems. worldcat discovery can only search and display bibliographic records from worldcat (regardless of whether users are searching local records or all of worldcat), and will not ingest bib records from other systems. timely local item information, such as availability, location, number of copies, local notes, etc. can be pulled from folio or other systems via z39.50 (not restful apis) to display with worldcat records in worldcat discovery. a local ils/lsp would also sync its holdings with worldcat via z39.50. these integrations are built on-demand by oclc, and they expressed willingness to look into integrating with new systems. [15] as with alma/primo, this configuration negates some of the benefits of an open source system (e.g. cost, local control), as a library would still have to subscribe to oclc’s suite of systems and services on top of devoting resources to folio, and folio would be duplicating bibliographic data held in oclc’s systems. it is possible that this systems configuration could make sense for libraries that depend heavily on the entirety of worldcat, their resource sharing groups, and interlibrary loan, but it would probably not be suitable for most academic libraries. according to oclc, they “are dedicated to improving the accessibility of our products, focusing on conforming to level aa of the w3c web content accessibility guidelines.” [16] the vpat for worldcat discovery (and their other products) seems not to be readily available on the web; instead, oclc requires submission of a request form to access their vpats and other accessibility reports. [17] opac-like front ends – open source options blacklight while blacklight does not have its own aggregated index and link resolver for electronic resource subscriptions, it can be connected to eds or alma/primo/summon via apis. [18] [19] therefore, libraries using it need to continue subscribing to one of these services and actively use it for electronic resource back end management. however, even with these integrations, blacklight only searches and displays electronic resources separately from local catalogs. most libraries (example: cornell) seem to use blacklight as the front end for a single data source, such as folio, an institutional repository, or an archival description system, and may run several different blacklight instances, one for each data source. a few institutions have several data sources in one instance of blacklight, such as the philadelphia area archives at upenn, which presents ead finding aids from many institutions in one blacklight instance. blacklight’s development community is largely north american and, while diffuse, is quite robust, as it includes several large, well-resourced institutions. it is built with ruby on rails and uses a solr index. there do not seem to be any commercial hosting and support services available, so a library would have to host blacklight locally. libraries using folio with blacklight have options for their folio hosting (in part because there is no company offering a folio/blacklight hosting package). cornell, for example, hosts folio with ebsco, while stanford hosts locally. cornell has been using blacklight as part of their discovery framework for over a decade. [20] while they host folio with ebsco, cornell hosts blacklight locally, as is necessary given the lack of vendor support for blacklight. [21] their index is populated with bibliographic data from folio using a rest endpoint and processed with a java program they have created, before being pushed to solr. blacklight allows for, but may also require, a large amount of customization to meet institutional needs. cornell employs one dedicated blacklight developer, and there is support by many others who work in web development at the cornell libraries. cornell has built their own “my account” ruby on rails application to integrate with their blacklight discovery layer and folio. [22] this application authenticates users using cornell’s institutional shibboleth. using folio api endpoints, it can pull a patron’s user information, as well as post requests and renewals. stanford, like cornell, is using a self-hosted blacklight discovery layer. however, they are also self-hosting folio on a local kubernetes cluster. self-hosting gives them the ability to populate their blacklight index using direct queries to the folio database. stanford takes a similar approach to cornell to patron empowerment, by creating an additional rails application that allows users to see loans, place requests, and renew items. users are authenticated to their patron account using sso. stanford has a significant team working to support their discovery services and the folio integration, with a team of six full-stack developers who have expertise in solr, rails and python. there are two analysts and subject matter experts included in the work cycle as well. [23] while there is not much readily available information on blacklight’s out of the box accessibility, it seems to be decent and improving, with a number of commits and issues dedicated to aria and wcag 2.0 aa standards, and development now including some automated testing. [24] because many blacklight adopters highly customize their installation, any given site may be more or less accessible than the baseline. vufind vufind is more readily used to present several local data sources in one instance. since, like blacklight, there is no aggregated index, a library using vufind will still need to subscribe to eds or primo/summon. if a library subscribes to an aggregated index, vufind can search it, though it searches that separately from the local collections, and at best results are displayed in separate tabs or columns of results for catalog holdings, databases, etc. in version 9.0 is a new feature called blender, which begins to combine data sources and present them more like a single search environment. [25] [26] to use the blender feature, libraries will still need to subscribe to an aggregated index; searches that pass from vufind to that aggregated index will still be subject to the index’s proprietary search and ranking algorithm. nevertheless, some libraries, like the aforementioned simmon university, use vufind only for searching and displaying catalog records and related patron account functionality (for example, requesting and renewing hard copy books), and maintain the aggregate index’s front end for their electronic resource subscriptions. vufind is primarily developed and maintained by villanova university and, while it remains centralized there, has a global community of users and developers. like folio, it is an open library foundation project. [27] it is built with php and javascript, and uses a solr index. hosting and support is available from index data; support services are also available separately from hosting. hosting and support is also newly available from equinox open library initiative. [28] libraries are using both oai-pmh and sftp to send catalog record updates from folio to vufind. texas a&m, for example, locally hosts both vufind and folio and sends hourly incremental updates from folio’s mod-oai-pmh, with full catalog reloads twice a year. university of chicago, on the other hand, which hosts vufind locally and hosts folio with index data, runs a daily sftp transfer using code built by index data. and while the patron account features are “robust,” according to a texas a&m librarian, libraries may choose to use or not use the native features. texas a&m chose not to, as they already had a “my library” patron account system that had been in use with their previous ils and which they were able to integrate with vufind when they went live with it and folio. university of chicago uses vufind’s native patron account features, connected to their campus sso environment. these two examples of features – catalog updates and patron accounts – display the flexibility of an open source system such as vufind; libraries are able to use or not use various pieces of the software as it suits their changing needs and available resources. even with hosting options and vendor support, libraries find that they have to devote at least a little personpower to customizing vufind. at university of chicago, they have an in-house php developer and another person who works on ux; they have both written custom code for their own use and contributed back to the codebase. [29] texas a&m relates that they were able to get “to about 85%” and needed in-house development for the home stretch. [30] beginning with version 7.0 in 2020, vufind developers have been emphasizing the system’s accessibility. [31] this development aims to “get as close to [wcag level] aaa compliant as possible” and also be in compliance with germany’s bitv standards which are equivalent to wcag level aa; however, a formal vpat for vufind could not be found. the wcag 2.1 level aa compliant equinox theme was launched by equinox open library initiatives in april 2023,and as of june 2023, accessibility work is ongoing, as evidenced by continued discussion in the regular community calls. [32] [33] aside from the above mentioned institutions, examples of libraries using vufind with folio include: lafayette college skidmore college wellesley college aspen bywater solutions manages the codebase for aspen, which is descended from vufind, and provides hosting and support services. hosting and support is also available from equinox open library initiative. aspen is mostly used by public libraries, though there are a few academic libraries using it, primarily community colleges. though some folio libraries have expressed interest in aspen, there are no folio libraries using aspen yet (as of july 2023). bywater says that, “although there is not currently an integration with [folio], we would be happy to discuss building one with any interested libraries.” [34] since bywater will also host folio, a unified hosting service for folio and aspen would be possible, similar to the arrangement with ebsco for folio hosting and eds, or index data for folio and vufind. as with other open source discovery systems, aspen does not have an aggregated index. aspen can connect to eds/ebscohost but not to the primo/summon central index. results can be presented either as a combined catalog and eds search with bento type results (where the user searches once and is presented results from different sources in different columns or blocks, which visually resembles a japanese bento lunch box, instead of mixed in one results lists), or the catalog and eds can be searched separately. other data sources can be included via either apis or sideloading, but at this point it is difficult to determine if these would be robust enough for the needs of a research institution. according to bywater, “aspen discovery is currently compliant with web content accessibility guidelines (wcag) 2.1 aa.” [35] aspen does not have a vpat or similar formal documentation outlining their level of compliance. examples of aspen in academic libraries (non-folio): college of aurora yavapai college sheridan college opac-like front ends – proprietary options locate locate is ebsco’s recently launched front end interface for folio. [36] it is unrelated to eds and not dependent upon it. unlike the above opac-like systems, it is proprietary software and not open source. it will only be available to customers that host folio with ebsco. it will only incorporate records from folio inventory, and can not search and display other data sources. this means it can only provide data on eresources if marc records for the resource have been created and added to folio inventory. unlike eds, locate does not depend on a daily feed of records from oai-pmh or file transfer. the apache kafka module within folio provides constant streaming updates without delay. [37] locate also includes left anchored searching, a feature present in traditional opacs, but less common in modern single-search discovery layers. left anchored searching has been a powerful tool for music discovery in libraries. [38] ebsco plans to integrate locate with their other systems, including eds. grand valley state and drew university launched locate instances in the spring of 2023. according to librarians at grand valley state, they chose to implement locate to address some of the shortcomings of eds. [39] like ebsco discovery service, locate’s vpat indicates that the system mostly meets wcag 2.1 level aa. [40] vega vega is a proprietary system created by innovative interfaces, inc. (iii), which, like ex libris, is a clarivate subsidiary. it is largely used by public libraries, and can only be paired with polaris or sierra ilss. therefore, it cannot be used with folio. soutron soutron discovery is a proprietary system from soutron global, whose primary market is the united kingdom, with focus on special and corporate libraries. no libraries are currently using soutron discovery with folio. however, soutron says that “the product will work with any ils“ via apis, so it is theoretically possible. [41] enterprise sirsidynix’s enterprise discovery system is ils/lsp agnostic, so it can probably be paired with folio, though no library has yet done so. it is largely used by public libraries, and therefore may not be appropriate for an academic library. similar to vufind and aspen, it can present search results from ebsco discovery in a bento format via apis. destiny follett’s destiny discover is intended for school libraries, and is part of their destiny suite. while it does some electronic subscription resource integration, it is not appropriate software for research libraries. it could not be determined from the sales materials whether or not destiny discover can be paired with folio or if it is dependent on the rest of the destiny suite. tdnet tdnet discovery has an aggregate index, content management capabilities, and a link resolver, making it similar to the single search discovery layers, but does not cover nearly the amount and diversity of content that those systems do. tdnet’s focus is on stem (science, technology, engineering, mathematics) materials, which would not be sufficient for a multidisciplinary academic library (hence including it here in the opac-like section), and is “designed for corporate, bio-medical and special libraries.” [42] it likely can be paired with folio, though no library has yet done so. conclusions eds is the only proprietary single search discovery layer currently being used by folio libraries. none of the other three systems are being used in conjunction with folio; of them, ex libris’s summon is the most realistic option for folio libraries, as it can almost certainly ingest marc records from folio and would not come packaged with redundant other software. oclc’s worldcat discovery is available; folio would not, though, be installed in place of, but rather in addition to other oclc subscriptions, since worldcat discovery will not accept bibliographic records from other systems (though item information could be ingested from folio via z39.50). ex libris’s primo is unavailable as a discovery system, since ex libris is unwilling to sell new subscriptions without an alma subscription. therefore, if a library wishes to use a single search discovery layer that integrates electronic resource management, link resolving, searching, and use, that library must use eds or summon unless they are willing to pay for software that duplicates some of folio’s functions, in which case worldcat discovery is also an option. according to ebsco, moving to the new eds ui could fix some of the issues that early adopter libraries have in eds classic, but the authors are not confident that this is true. of the three open source opac-like front ends, blacklight and vufind are already being used with folio, while aspen is not. blacklight has the least integration with electronic resources; it also requires a less common and therefore more expensive skillset and has no commercial hosting and service option. both vufind and aspen can integrate electronic resources more readily and present a search experience that is closer to a single search; it is possible that similar aspects could be built into blacklight. blacklight and vufind can integrate with either ebsco’s or ex libris’s aggregated indexes, while aspen will only connect to ebsco’s. while otherwise very similar, vufind and aspen have important differences, especially in that vufind already has folio libraries using it, as well as a larger academic user and developer community. development of vufind’s blender feature will provide the user experience that is closest to a single search, which is a major selling point of the proprietary discovery layers. adopting aspen, however, could be a groundbreaking project that contributes to widening the library software marketplace. if a library wishes to pursue a combined hosting and/or support service for folio and an open source discovery layer, it can choose between indexdata for vufind or bywater for aspen. proprietary opac-like discovery layers represent a style of library software that is verging on outdated; they also lack both the perks and drawbacks of open source systems. of them, only locate is currently being used with folio; ebsco designed it specifically to be an opac front end to pair with their folio hosting service. of the other five, most (except vega) can probably be paired with folio, though no libraries have done so yet. however, these systems are primarily designed for public (enterprise), school (destiny), and special libraries (soutron, tdnet); they may work with folio in those settings, but would likely not meet the needs of an academic library. locate is therefore the best choice, but can only be obtained in tandem with ebsco’s folio hosting. finally, as with any major systems change, a library wishing to implement a new discovery system must thoroughly investigate the needs, desires, and expectations of its users, including users that are not currently making their needs known through existing feedback mechanisms. with this information, a library can make informed decisions and pursue solutions that truly address the needs of both patrons and staff, regardless of which systems are pursued. this inquiry should draw out both where patrons are dissatisfied and not having needs met by current systems, as well as where patrons are satisfied and needs are being met. libraries will also need to be frank about the budget and personnel they have available for implementing and maintaining a discovery layer software. these three classes of software vary greatly in cost and required skills; the sticker price of proprietary software can be shocking, but open source software often ends up being more expensive than originally planned once infrastructure and staff hours are taken into account. since the above described softwares have major differences, for both staff and patrons, a library should start a rfp process only once these needs and budgets have been well defined. notes [1] conversation with missouri state university libraries staff. 21 november 2022. [2] what you need to know about addressing gdpr data subject rights in primo. 2022. ex libris group. [accessed 14 august 2023]. https://knowledge.exlibrisgroup.com/cross-product/security/gdpr/01addressing_data_subject_rights [3] it may be difficult to compare ebsco’s gdpr and other data privacy compliance with that of ex libris or oclc, as they are possibly defining that data differently, with ex libris and oclc using stricter definitions that include less data under the purview of those laws. for example, oclc allows libraries to add their own data privacy notice, rather than having their own, indicating that the library is responsible for patron data. [4] ebsco information services accessibility conformance report international edition. january 2023. ebsco information services. [accessed 13 july 2023]. https://connect.ebsco.com/s/file-download?param=/sfc/p/1h000000p2ep/a/5a000002gfcs/rb5mijtlcypwfykjuqr0zwjjtgiwmp4yj8x.jlqrgxk [5] ebsco accessibility conformance report international edition. january 2023. ebsco information services. [accessed 13 july 2023]. https://connect.ebsco.com/s/file-download?param=/sfc/p/1h000000p2ep/a/5a000002gfcx/l3f_x1iwfznz7dvojnnx1krgghm6xa.tqrkv59alzbc [6] accessibility. ebsco information services. [accessed 13 july 2023]. https://www.ebsco.com/technology/accessibility#:~:text=questions%20about%20the%20accessibility%20of%20our%20products%3f [7] email from clarivate sales. 19 july 2023. [8] reminder that ex libris is now owned by clarivate. proquest bought ex libris from golden gate capital in 2015. proquest, including ex libris, was in turn acquired by clarivate in 2021. for an extensive genealogy, see marshall breeding’s mergers and acquisitions charts: https://librarytechnology.org/mergers/ [9] 2023 library systems report. 1 may 2023. marshall breeding. american libraries magazine. [accessed 13 july 2023]. https://americanlibrariesmagazine.org/2023/05/01/2023-library-systems-report/ [10] summon: exporting catalog holdings – uploading to summon. 6 december 2016. ex libris knowledge center. [accessed 13 july 2023]. https://knowledge.exlibrisgroup.com/summon/product_documentation/configuring_the_summon_service/working_with_local_collections_in_the_summon_service/getting_local_collections_loaded_into_the_summon_index/summon%3a_exporting_catalog_holdings_-_uploading_to_summon#exporting_your_records [11] email from ex libris. 9 june 2023. [12] summon: real-time availability of items in library catalog. 17 november 2022. ex libris knowledge center. [accessed 13 july 2023]. https://knowledge.exlibrisgroup.com/summon/product_documentation/searching_in_the_summon_service/search_results/summon%3a_real-time_availability_of_items_in_library_catalog [13] summon accessibility statement. 14 february 2023. ex libris knowledge center. [accessed 28 july 2023] https://knowledge.exlibrisgroup.com/summon/product_materials/summon_accessibility_statement [14] clarivate: summon accessibility conformance report international edition. february 2023. ex libris knowledge center. [accessed 28 july 2023] https://pq-static-content.proquest.com/collateral/media2/documents/summon_vpat.pdf [15] conversation with oclc sales rep. 22 june 2023. [16] oclc accessibility statement. 2023. oclc.org. [accessed 28 july 2023] https://policies.oclc.org/en/accessibility.html [17] to request oclc’s vpats, including for worldcat discovery:  https://help.oclc.org/librarian_toolbox/troubleshooting/how_do_i_request_a_vpat_for_an_oclc_product [18] edsapi-ruby [blacklight-eds api connection]. 23 july 2022. ebsco github. [accessed 28 july 2023] https://github.com/ebsco/edsapi-ruby [19] integrating blacklight with alma and an ex libris central index. 18 december 2018. ex libris developer network. [accessed 28 july 2023] https://developers.exlibrisgroup.com/blog/summon-index-and-the-alma-link-resolver/ [20] how does the bento/single search work? 12 june 2014. cul discovery and access blog. [accessed 28 july 2023] https://blogs.cornell.edu/discoveryandaccess/2014/06/12/how-does-the-bento-single-search-work/ [21] conversation with debra howell. 16 september 2022. [22] cul-my-account. 21 march 2023. cul-it github [cornell university]. [accessed 28 july 2023] https://github.com/cul-it/cul-my-account [23] conversation with sarah seestone and darsi rueda. 23 june 2023. [24] repo:projectblacklight/blacklight accessibility. guthub. [accessed 28 july 2023] https://github.com/projectblacklight/blacklight/search?q=accessibility&type=commits [25] blended search. 16 february 2023. vufind. [accessed 28 july 2023] https://vufind.org/wiki/configuration:blender [26] deep dive into the blender search backend. 14 september 2022. ere maijala. 2022 vufind summit. [accessed 28 july 2023] https://www.youtube.com/watch?v=-cwgwnkck_0&list=pl5_8_wt3jpggq9an5ya8gydykkdypsud8&index=6 [27] projects. 2023. open library foundation. [accessed 28 july 2023] https://openlibraryfoundation.org/projects/ [28] products. 2023. equinox open library initiative. [accessed 28 july 2023] https://www.equinoxoli.org/ [29] conversation with tod olson. 2 november 2022. [30] conversation with paula sullenger. 21 september 2022. [31] video discussion: theme accessibility. 7 april 2020. vufind. [accessed 28 july 2023] https://vufind.org/wiki/videos:theme_accessibility [32] equinox announces accessible vufind theme. 21 april 2023. equinox open library initiative. [accessed 28 july 2023] https://www.equinoxoli.org/equinox-announces-accessible-vufind%e2%93%a1-theme/ [33] https://vufind.org/wiki/community:pmc_meetings:minutes20230606 [34] email from jordan fields, bywater solutions. 8 june 2023. [35] email from jordan fields, bywater solutions. 14 june 2023. [36] ebsco locate [locate demo site]. 2022. ebsco. [accessed 28 july 2023] https://demo.locate.ebsco.com/search [37] locate frequently asked questions. 26 july 2023. ebsco connect. [accessed 7 july 2023] https://connect.ebsco.com/s/article/locate-frequently-asked-questions-for-administrators?language=en_us [38] music discovery requirements, version 2. august 2017. music discovery requirements update task force, music library association. [accessed 28 july 2023] https://cdn.ymaws.com/www.musiclibraryassoc.org/resource/resmgr/mdr/musicdiscoveryrequirements2.pdf [39] conversation with grand valley state university librarians. 14 july 2023. [40] ebsco information services accessibility conformance report international edition. may 2023. ebsco connect. [accessed 28 july 2023] https://connect.ebsco.com/s/file-download?param=/sfc/p/1h000000p2ep/a/5a000000mkyx/eshwuef2zsn0falf54ntnartomil4.jslgiu0pxnkcm [41] email from soutron sales rep. 15 december 2022. [42] discovery & delivery. 2021. tdnet. [accessed 28 july 2023] https://www.tdnet.io/services/discovery thanks we are very grateful to the library workers who shared information and their experiences with us. we are also grateful to the vendor reps who answered our questions, and to the other librarians who contacted their vendor reps on our behalf when we couldn’t get in touch with vendors ourselves. of particular note are ruth kitchin tillman, noah brubaker, marian carney, debra howell, darsi rueda, sarah seestone, paula sullenger, and bill kessler. about the authors aaron neslin (aneslin@umass.edu) is the folio coordinator at the five college consortium. jaime taylor (jaimetaylor@umass.edu) is the discovery & resource management systems coordinator at the university of massachusetts amherst, which is part of the five college consortium. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – a practical starter guide on developing accessible websites mission editorial committee process and structure code4lib issue 37, 2017-07-18 a practical starter guide on developing accessible websites there is growing concern about the accessibility of the online content and services provided by libraries and public institutions. while many articles cover legislation, general benefits, and common opportunities to improve web accessibility on the surface (e.g., alt tags), few articles discuss web accessibility in more depth, and when they do, they are typically not specific to library web services. this article is meant to fill in this vacuum and will provide practical best practices and code. by cynthia ng and michael schofield web accessibility is increasingly important. one reason is legal: in the us the federal government recently updated their regulation on accessibility of digital content, section 508, incorporating the w3c’s web content accessibility guidelines 2.0 as well as harmonizing the guidelines with the eu’s information and technology communications rules (united states access board, 2017). while section 508 only explicitly applies to federal departments, based on wording in other sections, such as section 504, any federally funded program can also be held accountable in following the same guidelines. another reason is to ensure the site can be used by as broad a community as possible. web accessibility generally refers to making digital services and tools understandable to assistive technology (at) used primarily by those with disabilities. being web accessible also improves access for older people, mobile users, new users, low bandwidth or older technology users, and people with low literacy or familiarity with the language (henry & arch, 2012). meeting accessibility guidelines does not automatically make a web service usable, useful, or valuable. although there are certain considerations specific to assistive technology users, consider using the universal design approach in making web services usable and accessible to as many people as possible including those with disabilities. figure 1. user experience honeycomb showing the different facets of user experience. this article is meant to go beyond the low hanging fruit of creating accessible web content (as is often covered in accessibility articles) and focuses on the role of developers to make sites, tools, or services accessible. as such, the article assumes a basic knowledge of html, css, and website structure, and will not rehash guidelines generally meant for content creators, such as proper document structure, links, images and alt text, providing text alternatives to media, and document creation. the article will instead cover areas that are controlled at the system or template/theme level, emphasizing semantic markup, skip links, forms, media, fonts, roles and attributes of web accessibility initiative accessible rich internet applications (wai-aria), performance, and use of javascript. the article begins with an explanation of aria as it can be difficult to discuss different aspects of developing for web accessibility without referring to how aria may have an effect, especially in regards to early considerations. aria the wai-aria (web accessibility initiative accessible rich internet applications) specification allows roles to be assigned to specific (html) elements that signal the kind of content, possible interactions, and state of the application to compatible at. aria represents an optional developmental layer that devon persing conceptualized as “the accessibility stack” (2016). figure 2. persing’s accessibility stack. this is a useful way to understand this toolset. well-written html without frills is sufficiently accessible. as barry t. smith (2015) scolds us, websites aren’t broken by default, they are functional, high-performing, and accessible. you break them. css’s various show/hide declarations — display: block; display: none; visibility: hidden; etc. — alter at’s access to dom elements. with care, css alone provides enough of a toolset to create screenreader-only instructions, skip links, and the like. simple websites relying only on html and css might find that aria attributes are overkill. however, when javascript interacts with the dom or responds to user behavior (such as a touch event), at’s ability to communicate these changes is diminished. thus, aria acts as a structural support that anticipates and better responds to change. aria has two features that aid at tools: roles and attributes. aria roles we use the ‘role’ attribute in an html element to essentially communicate its purpose. there are many values for the ‘role’ attribute, but the most common are the eight navigational landmark roles that define regions of the page. they are application banner complementary contentinfo form main navigation search the exception is for role=”application”, which tells at that the page is an application rather than a document. these roles help the assistive device communicate the relationship of one piece of content to another. for instance, <div role=”banner”> describes the header of the page, whereas role=”complementary” designates sections that are complementary to the main content (role=”main”), such as a list of related articles. they can also guide complex interactions, for which in many cases semantic markup alone is not enough. these are typically for building complex, custom tools and applications (such as browser-based document editors), not for typical websites and content (see w3c, 2016b, and fischer, 2010 for more information). sometimes, adding a role attribute to certain elements seems redundant, such as <nav role=”navigation”> or <form role=”form”>, but in the recent past this helped bridge the gap between browsers that understood the implicit semantics of new html5 elements (like <nav>) and those that did not. these can be used to simultaneously help assistive devices understand the various roles of content, while also semantically structuring it. examples are often seen with structural roles like <button role=”button”>, <header role=”header”>, <article role=”article”>, and the like. [1] as the technologies mature, the “document conformance requirements for use of aria attributes in html” (w3c, 2016e) is updated with guidelines to follow on which elements benefit from the role attribute. aria attributes like roles, aria attributes are element attributes that convey relationship or the state of the element. they take the form of aria-*, where “aria” is prefixed to an attribute that contains some value or reference. for instance, <p aria-hidden=”true”> is a paragraph that is hidden to assistive technology; or <input type=”checkbox” aria-checked=”true” checked> communicates the “checkbox is checked” state to screen readers. [2] let semantic html do the heavy lifting aria roles and attributes are powerful but can be complex to understand and implement. the aria specification is a living document. it evolves quickly, adapting to the changing landscape of the web. it can be difficult to fully understand, but it can be summarized with this goal: whenever possible, let the browser maintain the implicit relationships between semantic html elements and their assistive conveyances. the bulk of the developer’s responsibility is in maintaining structural integrity: writing good html. of course, as the site or application becomes more complex, monitoring and maintaining its accessibility becomes a greater part of the developer’s technical debt. the browser accessibility tree to best understand how markup is interpreted by assistive technology, it is important to know that each browser provides information to at through its accessibility tree. figure 3. browser rendering process with accessibility tree the accessibility tree and the dom tree are parallel structures. roughly speaking the accessibility tree is a subset of the dom tree. it includes the user interface objects of the user agent and the objects of the document. accessible objects are created in the accessibility tree for every dom element that should be exposed to an assistive technology, either because it may fire an accessibility event or because it has a property, relationship or feature which needs to be exposed. generally if something can be trimmed out it will be, for reasons of performance and simplicity. for example, a <span> with just a style change and no semantics may not get its own accessible object, but the style change will be exposed by other means. (w3c, 2016f) basically, when a page renders in the browser, there is the document object model (dom) that is the underlying structure of the page that the browser interfaces with. it informs the browser that such-and-such is the title, what markup to render, and so on. it is hierarchically structured kind of like a tree. there is a root and a bunch of branches. at the same time, there is an accessibility tree that is created. browsers make them to give at interpretation information. when we use aria attributes, we are in part giving instructions to the browser about how to render that accessibility tree. there is a catch: not all browsers create accessibility trees in the same way; not all screen readers interpret accessibility trees in the same way; not all screen readers even refer to the accessibility tree, but scrape the dom directly instead — some do both. early considerations browser support before even beginning development of a website, one of the major considerations is what browsers to support. particularly in libraries and other non-profit organizations, the users served are diverse and are often still using older browsers. as well, when considering accessibility, one thing to keep in mind is that screen reader users often diverge from general browser usage due to the setup and interoperability with screen reader software. according to webaim (2015), over 50% of screen reader users use some version of internet explorer as their main browser, about 30% for firefox, and less than 10% for safari or chrome. additionally, over 15% are generally using an older version of internet explorer, many of which do not support some of the newer web accessibility features. in order to be accessible and usable to as many users as possible, progressive enhancement should be used. progressive enhancement is the design concept of having basic content and functionality work in all browsers — assume old browsers, no javascript, slow connections, etc. — with additional or enhanced layout, behavior, and features added using modern techniques. it is recommended that web developers use sites such as caniuse.com to determine what is supported in the oldest browsers in use. while each organization has to make their own decisions on when to stop supporting obsolete versions of browsers, using progressive enhancement generally means that even the oldest browsers can see and interact with a site on a basic level regardless of whether they are officially supported. javascript is (probably) imperative we need to address the increasing ubiquity and utility of javascript (js)-reliant front ends. largely informational sites do not require it, and basic interactivity (such as searches and form submissions) can be accomplished without javascript, but mostly js-reliant front ends are commonly found and will likely continue to increase. recently, ruth hamilton for creative bloq (2017) rounded-up a small panel of front end experts to answer where it is ever “acceptable to build sites that do not work without javascript.” this is a topic of hot drama that has persuasive and passionate arguments for either side of the coin. it is not the role of this article to weigh in, but consider it an important point to mention. using progressive enhancement to develop first for the lowest common denominator is a sound strategy for organizations, particularly libraries and non-profits, whose user bases are so diverse that there is no single “device profile” to accommodate. using this method may also assist developers to use interactive components that do not require javascript. while css is increasingly capable and it is possible to build interactive components without javascript — like fly-out menus, tabs, form validation, modals, tooltips — this potential is easy to conflate with progressive enhancement so that some aspire to go “no-js” all the way. removing javascript as a point of failure can be good for stability and good for performance, but there is a downside. sara soueidan (2017), during her attempt at building a fully-accessible tooltip without javascript, observed that javascript is imperative to make fully-accessible interactive components. sure, you can get away without using it in some cases, but a lot of accessibility requires javascript. some aria roles and attributes are absolutely necessary to make components accessible, and many of those will simply not behave as they need to unless you make them work with javascript. no-js is an interesting approach but to be fully accessible it is probably not the correct one. this does not exclude progressive enhancement, but as sites accrue complexity, they more likely benefit from javascript than not. using javascript can be a useful concession early in the planning process because it allows developers to consider javascript options much earlier and with wider applicability. source order, keyboard focus order, and visual reading order a core concept in web accessibility is that the order in which at traverses a page is not always the order it appears on the screen. scott jehl (2016) described this as ‘disconnected interfaces’. the modern web, especially with new css modules like flexbox and css grid layout, allow developers to adapt layouts to suit the context of the user. it is important that developers keep this discordance in mind. here is the vocabulary and how it is used going forward: source order refers to the order in which elements and their content appear in the markup – more specifically as they appear in the dom. keyboard focus order refers to the order through which users use their keyboard to navigate interactive elements like links, forms, and buttons. this normally follows source order but can be manipulated with the attribute tabindex. visual reading order refers to the intuitive order in which a sighted user reads the page content. as mentioned above, screen readers tend to traverse content in the order it appears in the dom — the source order — or something derived from it, namely the accessibility tree. a page laid out as a single column, such as it might appear on a phone, will typically have a visual reading order that is in sync with its source order, meaning that both the sighted and at user have a comparable user experience. in responsive sites, the relationship between source and visual reading orders is at most tenuous, because as the screen-width varies, the page will rearrange itself into multiple columns. depending on visual design decisions, the intuitive visual reading order may drastically differ from its top-down arrangement in the dom. given that keyboard focus order naturally follows the dom, complex layouts can easily become difficult to traverse by keyboard alone. advanced search interfaces in discovery layers are notoriously problematic. although facets are often left aligned, they are visually diminutive compared against right-aligned search results which are intended to draw the eye (because, of course, search results are what is important after performing a search). however, because these facets often appear first in the dom, users must first navigate this link-rich sidebar before arriving at the search results. it is the developer’s responsibility to keep these disconnected interfaces in mind. nevertheless, this issue should not deter developers from optimizing layout for context, screen size, and the capabilities of the device. element attributes like tabindex — as in <input type=”text” tabindex=”1”> — can force the keyboard focus order explicitly. a quick explanation: tabindex=”1” (or greater) defines an explicit tab order tabindex=”0” will place a normally non-interactive component, such as a <div>, in the logical tab order. in this case “0” does not mean that it comes first, but that it should be made available to the keyboard focus order tabindex=”-1” removes the element from the navigational flow but still allows it to receive programmatic focus we should note that tabindex=”1” or any other positive integer is, as webaim tells us, “almost always a bad idea” (2016). developers may be tempted to use tabindex as a way to make up for unorganized source code with an illogical navigation order, but we do not recommend the use of positive tabindex values. instead, fix the navigation order by restructuring the html. (webaim, 2016) the value can be updated on the fly in response to user behavior. for instance, often when the user activates a modal — a pop-up — javascript will set the tabindex of the containing element to “0” and the tabindex of the <body> (the part the modal is floating above) to “-1”. in this way, users cannot keyboard-navigate out of the modal without intentionally closing the popup. nevertheless, tabindex does not compensate well for a “disconnected” source order. rather, developers in certain cases might consider that rather than relying just on css to rearrange the layout of the page, they might go so far to rearrange the dom. the [accessibility] tree created from the dom (and sometimes css) changes a lot. when the tree changes, the browser sends events to assistive technology saying a portion of the tree has changed or updated. making a change to the dom with javascript that causes a reflow or layout (gecko and blink terms for the same thing, respectively) will most likely create or recreate an accessible object for it. (sutton, 2016) this could be expensive, though. there is no silver bullet. in fact, the options are not all that ideal. scott jehl (2016) explains: it seems this problem is only going to get worse now that advanced css layout is widely supported, and it certainly does not help that we now responsively toggle between many of these advanced layouts depending on viewport size. unfortunately, the tools we have as web developers seem to be inadequate for addressing this issue properly. given the choices of a) dynamically adapting our html source order for every breakpoint, b) sending different html sources to each client, or c) renumbering the tabindex attributes of all focusable elements to match their rendered order, we emphatically choose option d) “nope”. still, developers have tools, and browsers’ implementations of grid and flexbox modules could address this issue by setting focus order in which elements visually appear first, but that might not coalesce. while there is no one solution and developers should be cautious about using advanced css layout while they are still maturing, ensuring that elements in the source order are ordered by importance, and that keyboard order still makes logical sense are good places to start. best practices as development begins, there are some best practices and guidelines to consider. many of these practices apply to a general audience and may seem quite obvious, but are still important to keep in mind and implement. each page should have: valid, semantic markup, a descriptive title, the correct language (lang) attribute, consistent navigation, consistent identification (id) of the different parts of the page, a meaningful order to content, and provide multiple ways to discover content, such as a menu and a search box. these best practices help all users, and not only those that need accessibility considerations. without valid, consistent markup and a meaningful order to the content, browsers and search engines would have a hard time parsing and presenting a site. without consistency or correct coding, users will have difficulty navigating a site. for example, if the menu for a site showed up in different places on different pages, or the menu is not the same across all pages, users would find it difficult to navigate, expecting the menu to be the same and in the same place across all pages. beyond the general guidelines of correctness, consistency, and meaningful order, there are a few areas that should be given special consideration. below are best practices for aspects that are either often overlooked or consistently prove to be pain points. this is, by and large, low-hanging fruit, and attention in any of these areas has a huge impact. the lang attribute we can define the language of the site or app at the root <html> element with the attribute lang. its value is an iso language code (roberts, 2014) such as: <html lang=”en-us”>. the lang attribute helps screen readers and other voice over (vo) software switch language profiles in order to correctly accent and pronounce content. what is more, browsers can utilize this value to offer automatic translation if the client’s default language is something other than what is presented. also, if the font is not otherwise declared in the css, operating systems may actually change the default system font to conform with the font that is standard in the parts of the world where that language is native. the lang attribute is valid in all html elements, so it can actually be used in scoped instances where, for example, the language of a quote is different than that of the rest of the page. by instructing vo to adapt its language profile to that content, it is more likely to read that text in the correct language rather than as haunting robotic babble. <p lang="es">el gato es negro.</p><p lang="fr">le chat est noir.</p> additionally, vo is hardly restricted to at and of course is the core component to conversational, voice user interfaces. semantic markup we can use html in such a way as to reinforce the meaning or purpose of the content within the tags, employing semantic markup. semantic markup is particularly important, especially since it can reduce redundancy and the amount of code necessary to make a website accessible. html5 introduced numerous elements that allow coders to impart information about the role of a certain element, such as <nav>, <header>, <aside>, <footer>, assigning meaning to sections of a website through the use of specific tags. screen readers and other assistive software can easily find the menu by looking for the nav element without a coder assigning the navigation specific role to the tag. skip links skip links describe links that appear on every page to skip repeated parts of a site (such as the header) to directly navigate to the content, search bar, or any other important section. skip links are typically found at the top of a page, early in the source order, and usually hidden from users unless the link has focus. the links should be descriptive, such as “skip menu to search results,” so that the user knows what they are skipping and where to. these type of links are particularly valuable to keyboard only and screen reader users, with over two-thirds using skip links sometimes or more often (webaim, 2015). figure 4. screenshot of “skip to menu” example link with focus generally, skip links are hidden unless the user has focus on the links, then they should show so that sighted users can see where their focus is. the following code is a sample of how skip links might be coded to achieve this effect: html <a href="#results" class="assistive-text">skip menu to search results</a> css .assistive-text { position: absolute !important; clip: rect(1px 1px 1px 1px); /* ie6, ie7 */ clip: rect(1px, 1px, 1px, 1px); } a.assistive-text:active, a.assistive-text:focus { background: #eee; border-bottom: 1px solid #ddd; color: #1982d1; clip: auto !important; font-size: 12px; position: absolute; text-decoration: underline; top: 0; left: 7.6%; } while some sources suggest tying these links to access or keyboard shortcut keys, access keys can be very difficult to implement in such a way that it does not interfere with existing shortcut keys of either the browser or screen reader software (webaim, 2016). another option to make content ordering more meaningful is to have less repetitive markup at the top, and have users skip to that content if necessary. for example, have the navigation or menu near the bottom of the page (usually as part of or just above the footer), and have a skip link to the navigation. visually, the navigation can still appear near the top using css positioning, especially on larger screens, but a menu at the end of the page means that users using assistive devices do not need to use the skip link on every page to get to the content they are looking for. all users, especially on mobile, may also find a bottom menu useful, since when they reach the bottom of a page, they are given the option to go somewhere (wroblewski, 2011). fonts major browsers have a default font size of 16px, typically with the serif font times or times new roman as its default. while there are mixed opinions about the general readability of serif vs. sans-serif, sans-serif increases readability for those with reading disabilities, such as dyslexia (rello & baeza-yates, 2013). certain research suggests a sans-serif font with a minimum of 12px (or equivalent) for body text (bernard et al., 2002) should work for the majority of users. however, the trend in this decade has erred toward larger base font sizes, rarely set below the browser default and commonly above. for example, bootstrap, a widely adopted front-end framework, establishes a body font size at 14px, compared to the 16px chosen by its alternative foundation. those highly trafficked destinations dedicated to reader experiences, such as the new york times, washington post, or the blog service medium.com go beyond with base fonts at 17px, 20px, and 21px respectively. while modern browsers correctly zoom text when pixels are used for font size, a small percentage of users may still be using an older browser (webaim, 2015), wherein zooming actually fails to increase the font size like one would expect. these edge cases require relative sizes. relative sizes refer to how the size of the various elements are coded. for example, the font size for h1 (heading 1) can be set to 24px. however, using px (a “fixed” size unit) requests older browsers to try keeping the element at that precise size. on the other hand, the element can be set using percentage (150% in this case) or em (the multiple of the font size, so 1.5em in this case), which will ask the browser to make the font size relative to its parent. in addition to percentages and em, there is also the rem unit, which is relative to the body element’s font size (or “root” em). see mdn (2016) for a more detailed explanation and bracey (2015) for more on the difference between em and rem. often, for reasons of backward compatibility with older browsers, developers will stack their font-size declarations to take advantage of relative unit sizing with a fall back to pixels in the rare case when relative units are not supported. here is what the new york times does: font-size: 17px; font-size: 1.0625rem; using font-size is only one example as relative sizes can be used for any element where the size can be set and will affect zoom when used for the block elements on any site in older browsers. line height and line length according to w3.org: many people with cognitive disabilities have trouble tracking lines of text when a block of text is single spaced. providing spacing between 1.5 to 2 allows them to start a new line more easily once they have finished the previous one. (2016g) additionally, to meet the wcag 2.0 requirements for visual presentation (1.4.8), they recommend that “line spacing (leading) is at least space-and-a-half within paragraphs, and paragraph spacing is at least 1.5 times larger than the line spacing” (w3c, 2016c). their use of line-spacing directly refers to the css attribute line-spacing that can be set on any element. “space-and-a-half” can translate to a relative unit as mentioned above, as either 150% or 1.5em – or just 1.5 without the unit specified. the prerequisite spacing between paragraphs can be accomplished by applying margin or padding above or below. p { font-size: 17px; font-size: 1.0625rem; line-height: 1.5; margin-bottom: 1em; } although there is no official guideline, one also finds common recommendations that the average horizontal length of a line in a paragraph is most readable between 45-90 characters including spaces. this number choice is somewhat subjective so the choice may be more on the side of good design rather than specifically good for accessibility. chris pearson, the creator of the golden ratio typography calculator, went so far as to figure out the mathematical relationship between font size and characters-per-line, noting that “this relationship will differ slightly from font to font.” each font has a character constant, μ, associated with it that relates the font size to the width of each character…. for instance, if you have a font size of 12px, and the font you’re using has a character constant of 2.3, then 2.3 characters will fit in every 12px increment of width (on average). thanks to this relationship, it’s possible to predict cpl mathematically. (2012) it is worth the read. suffice to say, there are easier ways to design for this. chris coyier created a bookmarklet that highlights the sweet spot of a line of text after the 45th character, which can act as a useful visual for determining the width of the column allocated for that text. see the pen bookmarklet to make the text between 45 and 75 characters turn red. by chris coyier (@chriscoyier) on codepen. forms form guidelines apply to any input boxes including forms that only have one field, such as a typical site search. the most basic and important guideline for forms is to have labels. each form field should have a unique and descriptive label using the label tag. making sure each field has an associated label is especially important for assistive software, because the software cannot tell if there is text visually near a specific form field, without the explicit label. assistive software requires a semantic connection between the id of the form field and the for attribute of the label. having appropriate labels also allows all users to choose field entries previously used in the same browser for fields of the same label, or use autocomplete form features. labels can be simple (such as “name”), and clear labels help users fill in forms properly and quickly. instead of labels, some sites use placeholder text. placeholder text shows up in the field, but disappears once a user clicks inside of a text input. while forms can take up less space, placeholder text can be problematic since it only works with input fields, not radio or checkbox inputs. for example, users may forget what field they are filling out without a label, and placeholder text may not be read by screen readers and other assistive software. while placeholder text is not inherently a bad idea, an explicit label is still required even if it is visually hidden, and in general is more helpful than placeholder text by itself. in addition to labels, it is important to have the correct input type for each field. some input types (such as checkbox) need to be correct, because they change the way the input is displayed and functions. label text will also act differently depending on the input (such as selecting a radio button when clicking on the label text). the latest browsers will also warn users pre-submission if a user’s input does not match the expected pattern based on the input type. html5 introduced a number of new types for the <input> element (such as email, see mdn, 2016a), some of which do not change the display of the input field, but do change how users interact with the input field and improve validation. using the correct and most specific input type helps users fill in information accurately. figure 5. screenshot of example email and url pre-submission validation error display in firefox users should also always be notified of whether or not their form submission succeeded. users also greatly benefit from being told the format of a form field (such as yyyy-mm-dd for a date field), having the format enforced by the form, and being reminded of the format if there is an error. users will not be able to complete a form if there is an error and they are not given guidance on how to fix it. when filling in a form, it is not unusual for a user to use the tab key to move from one field to the next. there is no “standard” tab order, so it mostly depends on how the fields are organized and that the tab order makes logical sense based on the fields available. for example, if a form includes first name, last name, and the various address fields, the tab order should not go from first name to street address, city, back to last name, then province/state, and country. users may get confused or fill in the information in the wrong place. having an unintuitive tab order may also frustrate or confuse users to the point where they simply give up. while some may change the tabindex to solve the order in which elements have focus using the tab button, it is not recommended to explicitly change the tabindex order (webaim, 2016) as discussed earlier in the article. figure 6. screenshot of example captcha. source: hampel, m. (2008). suggestion from re:captcha. https://www.flickr.com/photos/a2community/2408912766 cc-by2.0 if a captcha (the test to make sure a user is human) is necessary, consider having an accessible captcha. commonly, captchas involve a distorted image or audio clip which users have to interpret and enter the correct answer. while having both an image and audio version makes a captcha seem accessible, the distortion is such that many users fail the captcha repeatedly (sauer et al., 2008). many alternative captcha methods exist and have been suggested, such as simple questions (asking for the name of the organization, solving a simple arithmetic question, etc.) with a text field input that is validated against a set answer. users are much less likely to fill in forms with a captcha (robinson, 2015 found a decrease of 73%), so seriously consider whether a captcha is needed in the first place or whether there is another method to achieve the intended goal (such as a spam filter). finally, while timed sessions are rare on most websites, they are common with online public catalogues and kiosk interfaces. ensure that users are warned if they are near the end of their time and allow users to extend the session, with enough time given to users to elect to extend the session. any user would be baffled if they are being timed, but do not know how much time they have left, or get kicked out of a system without any warning. users may also have any number of reasons why they need more time, so allowing them an easy way to extend their time will help them successfully achieve their goal. to summarize, make sure that forms: have labels for each field, have the correct input type for each field, mark errors and provide suggestions for fixing the errors if possible, have a tab order that makes sense, use an accessible captcha (if applicable), and have warnings for a timed session ending and allow users to extend their time. media media encompasses any audio or visual format including images, audio, and video. while providing an accessible alternative to the content itself is important, as many other articles already cover how to deal with media as content, the focus here is on the way media is functionally presented or coded and its effects on accessibility. with the introduction of html5, the new <figure> tag can increase the accessibility of media. the main use of the <figure> element is to provide a caption using the <figcaption> tag, so as to directly relate the caption text to the related media, or even a portion of text. for an image, the code might look something like this: <figure> <img src="https://example.library.org/logo.png" alt="an awesome picture"> <figcaption>fig1. example library logo</figcaption> </figure> regardless of whether the source is within an online system or through a third-party, any media that ‘plays’ or has other interactive aspects must have controls to play and pause the interactions. most audio and video players will have these basic controls built-in already, but the controls must also be accessible by screen readers and keyboards, typically using keyboard shortcuts once the media has focus or by allowing the user to tab through the different buttons. as the importance of controls is so that the user decides when to interact with media, it is important that media do not automatically play (autoplay). while uncommon that audio or video autoplay, image sliders or carousels often autoplay and often do not have any controls to pause the animation. autoplay causes multiple issues including, but not limited to: the user not being able to stop something if their focus is not there when first loading the page, assistive software often telling the user of every new image that is loaded, and depending on how it is coded, the autoplay will automatically put the user’s cursor focus on the media when it changes. figure 7. screenshot from shouldiuseacarousel.com automatically advancing image slides another common issue with media and similar embeds is that they become “keyboard traps”. for example, a website has a twitter feed embedded as a way to advertise how the organization is interacting with its users, but as a user tabs through the site, their focus ends up inside of the twitter embed, and cannot get out because it just keeps going through the twitter feed, which is simply loading more content the farther the user tabs inside of the feed. thus, a keyboard-only or screen reader user’s focus gets “trapped” inside of the twitter feed and cannot get past the feed unless they somehow find a way to reload the page and skip the feed on the next passthrough. there are, however, times when the cursor focus should purposefully be trapped. for example, if using a lightbox or similar modal, when a user clicks on something that opens the modal pop up, then the user’s focus should be trapped inside of the modal with an easy way for them to close it and return to the main page. if the user’s focus is left on the site in the background, they may not be able to access the modal to see what has been opened, or to close the modal. the main things to remember with making media accessible are that the user should be the one in control of how they interact with the media and web content, and that their focus moves logically through as they interact. color contrast content should use high contrast and large enough text size to ensure readability of content. this helps all users, including those with visual impairments, reading and learning disabilities, and with small screens. the recommended contrast ratio between the foreground and background colors is at least 4.5:1, or 3:1 for large text (see w3c, 2016c), where large text is the equivalent of at least 24px or bolded 19px. there are many tools to check color contrast and sites, such as webaim’s color contrast checker, that allow the background and foreground colors to be entered manually. auditing accessibility while full coverage of auditing accessibility is out of the scope of this article, a brief overview is provided here in the hopes that it is enough to provide a starting point. assessing accessibility can be done in many ways with different purposes in mind. many accessibility checkers rely on evaluating whether the code meets specified guidelines and suggested techniques. commonly used code checkers include: html codesniffer (bookmarklet) achecker.ca accesslint.com quailjs.com asqatasun (open source site-wide tool) the caution with these code checkers is that it evaluates the code based on very specific rules and may result in quite a few false positives. for example, an empty alt text tag will be considered an issue, but there are many cases where an alt text is supposed to be empty. emulators provide a way to see what users with specific conditions or using specific types of at see or hear: wave toolbar displays webpages in different ways, such as outline of headings. colorblind web page filter simulates different types of colorblindness. fangs, a firefox plugin, emulates a screen reader in text format. while code validators and checkers will ensure that accessibility guidelines are met, to make sure that a website or digital tool is usable, it needs to be tested with assistive software and devices, preferably with users who regularly use the assistive software or devices. conclusion there is a lot to consider and take into account when developing web services in such a way as to be fully accessible. while it may take some extra time and effort to make something accessible, more often than not, the benefits positively affect at and non-at users, and frequently make web services legally compliant, more usable and desirable. while a single article cannot cover everything, the article is meant to provide an overview on developing with accessibility in mind, with references to further reading and resources. hopefully, this article has provided guidance on how to begin meeting and implementing accessibility guidelines. endnotes [1] the specification does not actually require role indicators for the above examples, and could be flagged as redundant in a validator. as browsers have matured, elements such as <nav> and <footer> consistently convey implicit semantic meaning and role at the same time (see w3c, 2016d) without the need to specify. [2] the last “checked” indicates to the browser that the box is checked, but that is not necessarily communicated through the accessibility tree or may not be picked up by assistive technology. while it seems redundant, there are cases, such as ‘checked’ where the aria property is required. for a list of required and supported properties, please refer to the w3c’s aria in html. references bernard, m., lida, b., riley, s., hackler, t., & janzen, k. (2002). a comparison of popular online fonts: which size and type is best? software usability research lab. http://usabilitynews.org/a-comparison-of-popular-online-fonts-which-size-and-type-is-best/ bracey, k. (2015). comprehensive guide: when to use em vs. rem. evatotuts+ http://webdesign.tutsplus.com/tutorials/comprehensive-guide-when-to-use-em-vs-rem–cms-23984 coyier, chris. (2013). bookmarklet to colorize text between 45 and 75 characters (for line-length testing). https://css-tricks.com/bookmarklet-colorize-text-45-75-characters-line-length-testing/ fischer, d. (2010). the accessibility of wai-aria. a list apart. http://alistapart.com/article/the-accessibility-of-wai-aria hamilton, r. is it okay to build sites that rely on javascript? creative bloq. http://www.creativebloq.com/features/is-it-okay-to-build-sites-that-rely-on-javascript henry, s. l., & arch, a. (2012). social factors in developing a web accessibility business case for your organization. https://www.w3.org/wai/bcase/soc jehl, s. (2016). maintaining accessibility in a responsive world. https://www.filamentgroup.com/lab/accessible-responsive.html mdn (mozilla developer network). (2016). summary. font-size – css. https://developer.mozilla.org/en-us/docs/web/css/font-size mdn (mozilla developer network). (2016a). summary. input – html. https://developer.mozilla.org/en-us/docs/web/html/element/input pearson, c. (2012). how to tune typography based on characters per line. https://pearsonified.com/2012/01/characters-per-line.php persing, d. (2016). the accessibility stack: making a better layer cake. http://simplyaccessible.com/article/the-accessibility-stack/ rello, l., & baeza-yates, r. (2013). good fonts for dyslexia. assets. http://www.taln.upf.edu/system/files/biblio_files/assets2013.pdf roberts, a. (2014). iso 2 letter language codes. sitepoint. https://www.sitepoint.com/web-foundations/iso-2-letter-language-codes/ robinson, g. (2015). is google’s no captcha recaptcha a conversion killer? it’s digital marketing. http://www.itsdigitalmarketing.co.uk/2015/02/05/googles-no-captcha-recaptcha-conversion-killer/ sauer, g., hochheiser, h., feng, j., & lazar, j. (2008). towards a universally usable captcha. proceedings of the 4th symposium on usable privacy and security, 6, p. 1. https://cups.cs.cmu.edu/soups/2008/soaps/sauer.pdf smith, b.t. (2015). this is a motherf***ing website. http://motherfuckingwebsite.com soueidan, s. building a fully-accessible help tooltip. https://sarasoueidan.com/blog/accessible-tooltips/ united states access board. (2017). about the update of the section 508 standards and section 255 guidelines for information and communication technology. about the ict refresh. https://www.access-board.gov/guidelines-and-standards/communications-and-it/about-the-ict-refresh/overview-of-the-final-rule w3c. (2016b). aria in html. https://www.w3.org/tr/html-aria/ w3c. (2016c). contrast (minimum): understanding sc 1.4.3. understanding wcag 2.0. https://www.w3.org/tr/understanding-wcag20/visual-audio-contrast-contrast.htm w3c. (2016d). wai-aria overview. https://www.w3.org/wai/intro/aria w3c. (2016e). document conformance requirements for use of aria attributes in html. https://www.w3.org/tr/html-aria/#docconformance w3c. (2016f). core accessibility api mappings http://w3c.github.io/aria/core-aam/core-aam.html#intro_treetypes w3c. (2016g). c21: specifying line spacing in css. https://www.w3.org/tr/wcag20-techs/c21.html webaim. (2015). screen reader user survey #6 results. http://webaim.org/projects/screenreadersurvey6/ webaim. (2016). keyboard accessibility. http://webaim.org/techniques/keyboard/ wroblewski, l. (2011). organizing mobile. a list apart. http://alistapart.com/article/organizing-mobile about the authors cynthia ng is the manager of technology and technical services at the new westminster public library near vancouver, canada. she has previously worked in various technology and accessibility related positions in academic and special libraries. with a focus on users and a holistic approach to technology, she advocates integrating accessibility into projects and everyday work along with other aspects of development and user experience. michael schofield is a developer and librarian deeply interested in user experience design. he helped start the library user experience co. (libux), hosts the metric user-experience podcast, and is part of the practical service design community leadership team. in 2015, he won an acrl is innovation award for his work making an accessible instructional video platform called librarylearn. this year, he joined springshare. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – never best practices: born-digital audiovisual preservation mission editorial committee process and structure code4lib issue 43, 2019-02-14 never best practices: born-digital audiovisual preservation archivists specializing in time-based born-digital workflows walk through the technical realities of developing workflows for born-digital video. through a series of use cases, they will highlight situations wherein video quality, subject matter, file size and stakeholder expectations decisively impact preservation decisions and considerations of “best practice” often need to be reframed as “good enough.” by julia kim, rebecca fraimow and erica titkemeyer as professionals in the field of archival science, we aim to follow ‘best practice’ procedures, established standards that have been tested and proven to provide access to collections over the long term. however, for archivists aiming to preserve born-digital audiovisual files, ‘best practices’ often boil down to ‘it depends.’ the federal agencies digital guidelines initiative (fadgi) audio-visual working group, for example, recommends that archivists “develop selection criteria based on business needs,” “determine and document criteria for when (if ever) it is appropriate to change the video file’s technical properties,” and “create copies with appropriate technical characteristics to meet expected use cases.”[1] and while numerous reports and papers have assessed and compared standardized formats for long-term video preservation, arguing the benefits of losslessly compressed codecs designed for audiovisual digitization, such as jpeg 2000[2] and ffv1[3], the vast majority of born-digital audiovisual formats that an archivist is likely to encounter in the wild are significantly less well-documented and discussed in the field.[4] ‘best practice’ is hard enough to determine and follow when you have control over the creation of the media, but even more challenging when accessioning files created by donors, producers, and folklorists that never followed ‘better’ practices to begin with. the sheer conditionality of recommendations leaves practitioners mired in a sea of questions as they struggle to set realistically adhered to policies for their institutions. should files be accepted as-is, or transcoded to an open and standardized format? when is transcoding to a preservation file specification worth the extra storage space and staff time? if transcoding, what are the ideal target specifications? when developing policies and workflows for batch transcoding a variety of different formats, each with different technical specifications, how do you make sure that preservation files maintain all the perceptible, let alone “significant” characteristics of the original files? this paper presents case studies from three institutions — a university special collections library, a federal government department, and a public broadcasting station — demonstrating how the factors listed above might lead to ‘tiered’ processing and decision-making around ‘good enough’ practices for the preservation of born-digital a/v files. erica titkemeyer, the university of north carolina at chapel hill project director and audiovisual conservator; julia kim, the digital assets specialist at the american folklife center at the library of congress; and rebecca fraimow, the digital ingest manager at the wgbh media library and archives and the american archive of public broadcasting will each discuss how factors such as institutional mission, stakeholder needs, and available resources play into management of risks such as format obsolescence, file complexity, and preservation backlog.[5] university of north carolina at chapel hill by erica titkemeyer i coordinate large-scale digitization of obsolete audiovisual recordings at wilson library special collections, but i also provide guidance on born digital processing of audiovisual files across collecting units in the library. one of our biggest challenges is adapting born-digital workflows to include audio and video files of mixed formats, with the goal of documenting, processing and ingesting collections as efficiently as possible with the few dedicated staff we have. within wilson library special collections at the university of north carolina at chapel hill (unc), acquisitions of born-digital content have begun to grow in parallel to our large-scale digitization efforts for analog audiovisual and manuscript materials. looking specifically at born-digital audiovisual media, content appears most frequently in the southern folklife collection, which collects, preserves and disseminates vernacular music, art, and culture related to the american south. acquisitions can include a myriad of recordings, from live musical performances and oral histories to raw documentary elements. any given accession can range from a handful of gigabytes to well over a terabyte in some of our larger acquisitions. originally, these files had to remain on the media they arrived on (most often external hard drives and optical disks) due to storage limitations, but as digital storage capabilities have improved over the last decade, long term preservation of these files has become increasingly possible. at the moment, the primary long term digital storage location for unc libraries is the carolina digital repository (cdr), a fedora-based digital archive “for scholarly works, datasets, research materials, records, and audiovisual materials.”[6] ingest into the cdr is typically the final step in our processing workflows, followed only by the creation of mods metadata records by our technical services department, allowing everything to be accessible to unc students, faculty, and scholars. figure 1. carolina digital repository architecture represented below dashed line, and preand postingest steps shown above dashed line. however, as the following case studies will show, we have found that audiovisual media challenges many of our pre-ingest workflows while questioning policies in ways that text-based files have not. this has required staff to review and amend exceptions to policies to introduce feasible alternative actions for audiovisual files before they are ready for ingest. case studies the following examples of born-digital audiovisual acquisitions help to explain the types of hurdles we encounter and the ways in which making exceptions to our standard born-digital preservation workflows help us to produce best possible results considering our limitations at the time. in all instances we have to consider the level of quality of the content and how much storage it warrants, and the amount of time and resources we can dedicate to it at the time of processing. limiting files sizes it’s usually not the case that we don’t have enough digital storage space for a large file, but we have to recognize that if the size of the file can be decreased, it means more space for future priorities down the road. in my environment, the decision of whether or not to compress a file or to alter it in any way has to include input from the curator who acquired the collection where the file resides. the curator is more often than not the individual with the most knowledge about the content and the consequences of these actions to the patrons. the first case involving compression happened with an acquisition by the university archives and records management services department and related to non-professional, unedited footage of a recent graduation ceremony, for which the department already had professional footage. the file was 112 gigabytes and encoded using the apple prores[7] codec. apple prores is considered an intermediate lossy codec (though apple prefers “virtually lossless” when referring to its high-end flavors of the codec). its primary use is in editing environments, and it allows for files to be more easily manipulated and rendered compared to their original raw masters. even though it is small enough to be managed by editing software, the encoding still permits high enough quality that the file can exist as a base for creating derivatives of lower quality to distribute across different platforms. as an experiment, the graduation file was transcoded to meet our digitization specifications (ffv1 in the matroska wrapper[8]). ffv1 is a lossless video codec, meaning it is capable of compressing and uncompressing a file without permanently losing bits (unlike lossy codecs, which shave off bits during compression). in audiovisual digitization, it is ideal for minimizing the size of uncompressed files typically generated in digitization workflows. as we suspected, this solution did not work, as the file did not decrease in size. instead, it doubled in size. knowing that the ultimate solution would likely be to use a lossy codec, the curator was contacted about the necessary deletion of bits from the file itself if we were to expect a decrease in size. since this was not the only footage of the ceremony, and it was already low quality in terms of production value, the curator agreed to the compression. ultimately, we decided on the h.264 codec,[9] as it is common in media player libraries and thus could be played back by users using a variety of software programs. another benefit is that it is a relatively stable format that it is well documented and known by users in fields beyond audiovisual preservation. we used the following command for transcoding the original: ffmpeg -i -c:v libx264 -pix_fmt yuv420p -crf 12 -c:a copy [destination] after transcoding, the resulting file was 53 gigabytes, roughly half the size of the original. the -crf (constant rate factor) option is where we were able to manage control over quality, and consequently, file size. an alternative we could have used was the two-pass average bitrate (abr) option, which accepts a target bitrate to control file size output, but does not give us control over quality. the value of 12 was chosen based on ffmpeg’s documentation[10], which outlines the scale values for the crf option (0-51), where 0 is lossless. output files using values of 1-18 remain visually lossless (but not technically lossless). from testing, the value of 12 was sufficient enough in decreasing file size, while also remaining within the 1-18 target range. is this thing on? video files with no video another situation that i come across with surprising frequency is a video file without actual video. usually the lens cap is left on, as the producer only wishes to have an audio recording, or they wish to keep the speaker anonymous. either way, the presence of video means the file is much larger than it really needs to be if we’re only concerned about the audio stream within it. in all cases where this has taken place, i have used ffmpeg[11] to create a standalone audio file from an audio stream in a video file, which is then considered the preservation file for ingest. depending on the duration of the file and the original video stream encoding, this alteration can greatly decrease the amount of digital storage needed. the command i used came from ffmprovisr[12], a resource for using ffmpeg with example commands for archivists: ffmpeg -i -c:a copy -vn [output file] for these cases, it is important to note that documentation of what has been done to the original will be valuable information to future stewards and researchers, so a note (.txt file) is added to an admin folder, along with additional submission documentation, including technical metadata about the original file. this admin folder is bagged along with the media and ingested into the cdr. the case of the hybrid hard drive in fall 2016 i started working through a backlog of digital media, including a one terabyte hard drive from a local musician that had been included in a larger acquisition. fortunately, the drive wasn’t completely full, but it did contain a few hundred gigabytes of data. unfortunately, the content was far from consistent in file types. in addition to jpgs, pdfs and word documents, there were close to 1600 audiovisual files representing twelve different file formats and fifteen different variations of formats and codecs. the actual identification of these files was the first challenge, as many tools used in born-digital processing do not always identify specific stream codecs, only formats. in addition to using droid[13] to identify extensions (we have now moved onto using brunnhilde[14]), i also employed ffmpeg’s ffprobe feature, which can reliably report out audio stream and video stream codecs. it was at this point that i realized identifying the codec of each file early on would be helpful in determining file quality and the available possibilities for normalization or migration, either pre-ingest or in the future. figure 2. example droid output of mov files without any codec information. the puid, mime_type, format_name and format_version fields were all empty figure 3. default output using command ffprobe [file]. metadata is reported on all four streams, including codecs for audio and video streams. the additional ffmpeg step also allowed me to see if there was any embedded metadata within the files that would need to travel with the ingest package into the repository and that would be at risk of loss if we were to choose to transcode files prior to transfer. fortunately for me (or unfortunately, from a metadata perspective), most of the files found did not contain descriptive information (title, artist, copyright statement, etc). now that i knew as much as i could about the files, it was time to make decisions about what, if anything, to do to them prior to ingest. knowing that transcoding the files to our preservation specifications would increase their size greatly and unnecessarily, i struggled to find a solution, especially for the files that were wrapped or encoded with proprietary and/or obsolete formats and codecs. considering my position priorities to coordinate analog audiovisual digitization, i had to make the difficult choice of putting the work of ingesting such a large drive back in my backlog of work to do (after creating a duplicate copy on our servers). i had, however, succeeded in documenting as much as i could about the files. since then, and under the leadership of jessica venlet (unc, assistant university archivist for records management and digital records) we have developed more robust format policies and a more substantial set of expertise (from full-time staff to graduate student assistants) to enlist for guidance and to carry out born-digital processing tasks. these additional aids will inevitably help in tackling more complex ingests such as the hybrid hard drive, and it will be interesting to see how it eventually finds its way through our workflows. potentially we will ingest the files as is or transcode a certain subset of files to a more stable format. either way, we now have familiarity with the additional steps required for documenting audiovisual file specifications to get us started. american folklife center at the library of congress by julia kim the american folklife center (afc) at the library of congress was created in legislation to honor the united states’ bicentennial in 1976, and charged with a mission to capture all manner of folklife. afc is thought to be the largest folkloric and ethnographic archive in the world, and in recent years, afc’s acquisitions have become primarily digital.[15] as the sole full-time digital staff member in the department, i create workflows for our multi-format digital and digitized collections; train and manage staff; support our databases and applications; and, increasingly, liaise with donors, vendors, and cultural organizations. until recently, afc was not a part of the library services or special collection units. the geographic and information technology systems separation from the culpeper audiovisual facility also impacts our division’s audiovisual workflows. one of our key archival workflow strengths is that we have essentially limitless long-term tape server storage. we currently store approximately 600 tb on long term tape servers, with an additional 30 tb for online web-accessible collection space, and 20 tb on processing server space. this will dramatically increase as we begin systematically digitizing all our audiovisual assets. in addition to serving researchers and visitors in the reading room, afc has pushed more of its multi-format content online, essentially reprocessing many of its digitized and born-digital collections to meet the technical requirements for online access. as the major time-based media special collection division on capitol hill at the library, audiovisual files skew our long term digital repository server space needs to the very high end. this naturally influences our workflows. file format normalization upon ingest is not routine, and migration and normalization of time-based media only occasionally happens when preparing content for online access platforms.[16] afc’s born-digital video collections vary greatly in size and complexity. this means that i have to take a conscientious approach to assessing collections, which i do by giving collections numbers on a 1-4 scale to denote first, the 1) complexity and technical challenges to the collection, and 2) the resources the department is going to allocate to the processing.[17] as our collections increase rapidly in size, i’m more often forced to give most collections minimal processing for the time being. this is noted in the case studies below. like many institutions, our workflows are hampered by strict security restrictions that make common software tools and streamlined workflows difficult to implement. our biggest challenge, however, is that other than myself we have no dedicated digital processing staff. most standard workflows are carried out by part-time temporary staff external to our department. other than that, we have one full-time archivist who dedicates a handful of hours a week to digital processing, dependent on priorities. this can easily mean that many part-time individuals handle the processing of a single large collection over a long disparate period of time. the majority of the collections i handle had been ingested many years ago, but have been earmarked for reprocessing. my processing priorities mostly involve reprocessing work with external vendors, donors, or for publishing collections online. case studies many of our born-digital collections have been created “in the field” in un-ideal environmental conditions or with a mind to the cultural content rather than technical specification. they may also have followed a specification from another geographic area with a now-bypassed broadcast standard, and most of our content creators are not experts on these technologies. cleaning out our closets this recent test case is part of a much larger department initiative to uncover our hidden “fugitive media” and then, subsequently, migrate all the optical media.[18] this particular collection was earmarked early on because of its cultural value and its large size – over 400 optical disks – but it was part of a large-scale batch processing workflow, so for the initial pass, it was processed minimally. work was streamlined as much as possible by migrating all the optical content and files “as is,” while also retaining all disk images and logs.[19] this means that each submission package has unusually complex file and directory structures that go 5-6 layers deep. figure 4. top level directory and file level hierarchy created but there are some ambiguities: the optical dvd are all encoded with the older vcd mpeg1 encoding in the pal (or “phase alternating line”) broadcast standard. their native resolution is very poor; it’s not in current use or widespread.[20] the mdf and mdm disk image formats are also proprietary and obsolete disk image types. the norm, however, for these types of collections is to leave them as-is. this is true for many of the obsolete files uncovered through the media migration project this collection was a part of. the priority for these uncovered collections, many over a decade old, is migrating them from their unstable and deteriorating carriers and, essentially, accessioning and ingesting them. afc is able to initiate this project in large part because the staff primarily undertaking the work are non-afc technicians who rotate through the department. future work will necessarily involve active preservation work on at-risk obsolete files uncovered in this migration project, including quality control of outputted files and improving systems to streamline migration of the many different flavors of optical uncovered through this project. one small step i’ve taken for this collection subsequently was combining the mdf and mdm file into a more generic and singular disk image iso. give a folklorist a camera afc often directly creates or funds many of its collections through supporting fieldwork fellows, public programming, lectures, and concerts that become documented and accessioned as archival collections. these collections are unusually heterogenous and non-standardized, so workflows must be tailored anew each time. these collections were urgently prioritized for online display. while the videos themselves are static, captured from a camera on a tripod focused on the interview subject in nondescript rooms, the unnecessary complexity – as well as the ambiguities of the technical specifications – are a good case study for why we need to have interventions with the content creators earlier. in preparing these videos for publication online in our access system, i noted some odd unintended visuals called “combing,” which is a perceptible error noted in digital environments when the content creator creates interlaced audiovisual files meant to conform to analog technologies.[21] while i could deinterlace for access and maybe preservation, i still had to ask myself: was this intentional? this small collection of 20 born-digital audiovisual files had at least 4 different flavors of files with varying aspect ratios, audio, and moving image codecs, and of course, extensions. it seemed reasonable to assume that these were accidents rather than artistic choices. other than normalizing for online browser display, they’re not earmarked for transcoding.[22] figure 5. technical metadata from “ffmpeg -i” command with information on the component video, audio, and data streams in this collection’s audiovisual files. figure 6 detail of interlacing or “combing” in born-digital video let’s try again… a few more times now unlike the last three collections, this collection is one that i’ve been involved with earlier in its life cycle. i now often engage content creators before accessioning to discuss how their work impacts my preservation work down the line. in this case, the content creator was an ethnographer who had recently switched from still image work to audiovisual work. while the documentarian shot in a proprietary format (hd avchd – mpeg 4 avc/h264 .mt2s), he edited down hundreds of hours of footage in another proprietary software program to overlay separately captured audio interviews. so before i received any files, he had already twice modified the technical characteristics in producing the final films. in this case, we are not accepting the raw footage; all stakeholders agree that this is not warranted. after discussions with the donor, we decided to also receive the uncut audio files. after some research, i discovered that the files sent represented a loss of significant data and resolution from the edited down files (see table below). in addition to more obvious changes in the video codec, the table of excerpted technical metadata below also shows that data was lost and resolution decreased from 1920×1080 pixels to a significantly lower resolution of 1440×1080. another potentially highly perceptible change are the different aspect ratios noted by sar (screen aspect ratio) and dar (display aspect ratio, what most people think of when they say “widescreen), but these different ratios indicate that the pixel aspect ratios will also be different, ultimately influencing the image. this was an inadvertent loss only caught through understanding the entire content workflow chain, pre-acquisition. in this and many other cases, just giving content creators specifications without ensuring they understand their context misses the point. table 1.technical metadata of significant characteristics from a “ffmpeg -i” command to assess file characteristics for the camera raw versus submitted edited video output. file assessed video stream codec color sampling and encoding, resolution, aspect ratios, data rates, frame rates, and more interim video stream #0:0[0x1011]: video: h264 (hdmv / 0x564d4448) yuv420p(top first), 1920×1080 [sar 1:1 dar 16:9], 29.97 fps, 59.94 tbr, 90k tbn, 59.94 tbc final edited output stream #0:0[0x810]: video: mpeg2video ([2][0][0][0] / 0x0002), yuv420p(tv, bt709, top first), 1440×1080 [sar 4:3 dar 16:9], 29.97 fps, 29.97 tbr, 90k tbn, 59.94 tbc the content creator re-exported his finalized files to a compromise codec that kept the most significant characteristics intact, but he was concerned over whether or not his home computing environment could handle the processing. in this case, because of the combined aesthetic and technical value in the content captured, these extra processes were something that both the content creator and i were happy to spend additional time on. unlike the previous collections, these moving image files captured minute technical work and had aesthetic and scenic value. this arrangement is also foundational for our ongoing acquisitions with this donor and has new accessions planned. this new type of workflow, in which there are often months of intermittent conversations with a donor pre-accessioning, is now something i engage with on a regular basis across many collections. wgbh by rebecca fraimow i’m the digital ingest manager for the media library and archives within wgbh, boston’s public television station. as an archivist embedded within a production organization, i’m often making decisions based on significantly different factors than the ones that impact my colleagues at more traditional institutions. the wgbh media library and archives (mla) collects production masters, original media, and supporting and ancillary assets created on behalf of wgbh (including raw footage, original photographs, and promotional materials), as well as other materials deemed valuable by departments on a case-by-case basis.[23] over the past ten years, original media materials delivered to the archives have shifted from almost entirely analog to almost entirely born-digital; within the past two years, the same has become true of finalized master programs as well. the vast majority of this material is audiovisual. one of the key factors that distinguish our use case at wgbh is that the producers who provide the bulk of the mla’s assets are also the primary stakeholders in the archive. the production departments generally want to receive their original materials back exactly as they delivered them to the archive — same format, same folder structure — so that they know exactly how to find what they’re looking for, and can drop the footage back into their existing project files for video editing. (relatedly, productions generally want us to store avid and adobe project files and other edit materials as well, even though this isn’t officially a part of the mla’s mandate.) practically speaking, this means that an archival policy of normalizing everything and then storing only the normalized files is out of the question. however, the archive also serves the needs of other users, including producers and editors from other internal departments who want to reuse wgbh-owned footage rather than pay for external stock footage and external producers who pay for the rights to use wgbh’s original footage and interviews. many of the materials created by wgbh also have long-term historical value, including the full programs and raw interviews around topics of public interest, and unedited footage of significant places, persons and events. when possible, this material is made available for free in low-quality proxy formats through the mla’s open vault website[24] and/or contributed to the american archive of public broadcasting=[25] (a collaboration between wgbh and the library of congress.) researchers and other non-production viewers generally do not require high-quality production formats, but the access formats that will be most valuable for them are likely to change with technological shifts. to meet all these use cases, in an ideal world we’d like to have the camera original, a reliable preservation master, and a contemporary access format for all of our masters and raw production footage. however, as the rate and digital footprint of production increases every year, we currently don’t have the resources to keep up with that amount of processing and transcoding. since we began our current digital preservation program on lto tape in 2015, we’ve generated 1.3 pb of unique materials (2.6 pb when including redundant duplication on offsite ltos); our rate of growth is approximately 500 tb/year and increasing rapidly as more productions shift from hd to 4k. storage is, of course, a consideration, but the bigger limiting factor is time — not just human time, but also processing time on a relatively small number of computer stations and lto decks. right now, we have only four lto computer stations and two staff members who work on digital preservation part-time. running material through our standard ingest process generally takes about 24 hours of machine time (on one computer) per tb. batch transcoding the same amount of material can take 48 hours or more, especially if the transcode is processing-intensive, as lossless transcode generally is. while some of these operations can happen simultaneously, attempting to do too many at once risks overloading the machines and losing hours or days of work in the process. the bright side is that (when everything’s running correctly), this work takes a relative minimum of human time — maybe an hour for setup, and 2-3 hours for post-processing and ingesting metadata into institutional databases. however, individualized assessments of each delivery for problematic file formats increase that time exponentially, so for the most part we don’t do them. we’re racing so fast to make sure everything is backed up for our users that we just don’t have time to tackle much extensive curation the way we would if our use case focused more on the historical record, and less on reuse for production. case studies i’ve talked a lot about our standard workflow, and — unlike my colleagues at unc and the library of congress– we do follow a pretty standard (albeit constantly evolving) workflow for handling born-digital video most of the time. that said, every rule has its exceptions, so first i’ll discuss what we do with most of our regular production drives, and then a project that we did develop a specialized workflow for. everything and the kitchen sink a sample 2tb production hard drive we recently ingested included mp3 files from audio-only interviews; avdn animation files in a quicktime wrapper;[26] uncompressed pcm audio narration files in a .wav wrapper;[27] raw camera original footage in a couple flavors of mpeg-2v codecs, wrapped in quicktime or mxf, depending on camera of origin; a number of .jpg still images;[28] and stock footage in a variety of common formats such as h.264 and .avi. as per our standard practice, we used our standard preservation script to back up all content onto two preservation lto tapes. this ruby script uses standard file utilities to copy the files over to two locations and compare them on the the bit level; the sophos command line tool to run a virus check; and harvard’s file information tool set (fits)[29] to document technical and preservation metadata, including md5 checksums. the script creates a virus scan report and text reports verifying the bit-level comparison for each file tested, as well as a detailed report on the codec and wrapper of each individual file. it’s open-source and can be viewed on github here. we then created low-res h.264 viewing proxies for all video and .mp3 proxies for all audio. this might not have been necessary for the audio interviews, which were already in .mp3 formats, and for some of the stock footage, but it’s easier to run a batch script on the whole drive than try to sort out what does and doesn’t need to be transcoded. our batch script for creating video proxies uses bash scripting to call up an ffmpeg command similar to the ones my colleagues have quoted above; it’s available on github here, and the batch script for audio proxies is available here. the proxies can be viewed by internal producers in our local filemaker database to help them better determine what they might want to use in a production. we’ll also store the original drive, and until it dies we’ll use that as the primary means of return to the original production if they want access to their materials. a number of the formats that were delivered on this drive are proprietary, and as time goes on it may become increasingly challenging to play them back. however, right now we don’t have the human time to determine what would be the best preservation formats for all these different kinds of files, and we also don’t have the computing time or storage space to transcode everything to a standard lossless preservation format, so we’re putting off the decision until we run into a use case that requires transcoding these materials further. quality and quantity for a recent high-profile documentary, one of our production units recorded a large number of interviews which were delivered to the archive on 49 lto tapes — over 100 tb of footage. because of the size of this delivery, and because it was already on lto tape, we simply copied each of these tapes over to a backup lto tape using our standard preservation ruby script and stored both tapes, one onsite and one offsite. we didn’t create viewing proxies for this material at the time of ingest; the time it would have taken to copy each of these files off of lto, create proxies, and then copy back to a second lto would have significantly delayed a project that was already set to take approximately 100 days of machine time. this choice was also made in part because many of these interviews were sensitive and unlikely to be available for access in the near future by anyone outside of the original production unit. however, although the files in this collection are extremely large, there actually aren’t all that manyof them — on average, 4-5 files per lto tape — and they all share the same technical characteristics (avdh, 8-bit, 4k, in a quicktime wrapper.)[30] when we migrate our lto-6 tapes forward to lto-8, we also plan to replicate our footage onto a local server which has more transcoding capabilities than our in-house computers. at that time, not only will we be able to create access copies, but this collection will be a good candidate for a batch migration to a more stable and less proprietary format, since we only have to analyze one type of file and we expect that this material will provide significant value to the historical record in the long term. conclusion while each of our institutions has unique characteristics that have led to different decisions, across the board we have found that enforcement of a strict born-digital audiovisual file format policy is unrealistic. this is not just due to the aforementioned lack of consensus around preservation file format targets, but also a result of the complexity and variation in original file formats received; under current conditions in the field, one size just doesn’t fit all. as you can see in our text, a file’s technical specifications can include not only the extension, but also its codec, resolution, and many other significant characteristics that one would want to maintain across file migrations — information that often can only be fully determined with knowledge and tools that may not be available to all stakeholders. determining and researching a file’s characteristics is a time consuming prerequisite to migrating, and as our use cases demonstrate, practitioners need to make judgment calls about when the effort is worth the time. additionally, more than one of us has noted that our policies and workflows are provisional and are apt to change yet again in the near future due to continuing shifts in audiovisual production and preservation. in other words, this is both a technical issue and a much larger issue involving the administration of limited resources. while these issues have no easy solutions, an important first step is for institution must develop transparent policies and workflows addressing interim steps and potential file migration workflows. these policies, shared among practitioners, can help the field as a whole work towards developing programmatic solutions that can better address the diversity of born-digital specifications. our use cases document difficult compromises that are realistic and part of our actual day to day practices, not our much rarer examples of especially laudatory practices done when resources are unconstrained and there’s a clear and documented path for managing what is in front of us. these use cases were chosen because they have ambiguous answers that, in almost all cases, are only provisional and will need to be revisited by future processors. in each case, we’ve had to make policy priorities and treat collections differently based on its content, its stakeholders, and its future potential uses. however, across the board, we have all made sure to assess the risks against our own capacity and make sure we’re preserving the characteristics that are most significant to our users — even though those characteristics are different for each of us. through the course of our research and work, all of us have explored other potential options and made adjustments. in sharing our workflows with one another, all of us have amended policies. over this period of time, conversation in the field has also gradually begun to turn towards the development of more format-specific recommendations and best practices. recently, crystal sanchez and taylor mcbride, video specialists at the smithsonian institute launched a shared google document for practitioners to aggregate information and develop best practices around the preservation of camera original formats currently in use in the field. the lack of transparency around the multitude of proprietary camera original formats in use will otherwise continue to be a major stumbling block for digital preservation. another encouraging development is the work initiated by the fadgi audio-visual working group to define and address what audiovisual “significant properties” are exactly. these two developments will help the audiovisual field answer still unresolved questions important to understanding seemingly basic questions such what is a file’s characteristics and whether or not we need to preserve them. we’re optimistic that research into format-specific practices will allow for the development of improved policies that reflect the complexity of born-digital audiovisual media, as well as the development of tools that can actually make programmatic decisions based on those salient file characteristics.[31] while ‘best practices’ may become more attainable over time, our hope from exploring our own institutions here is to encourage a frank discussion of the cost-benefit analysis that goes into developing workflows for these materials, helping to demystify the decision-making process around digital video and encouraging institutions to take their first steps towards developing ‘good enough’ practices that are right for them. about the authors rebecca fraimow is the digital ingest manager at the wgbh media library and archives and the american archive of public broadcasting, where she oversees digital archiving workflows and the development of the pbcore metadata standard. rebecca was a founding member of new york city’s xfr collective, a nonprofit organization focused on the preservation of at-risk media, and previously managed the dance heritage coalition nyc digitization hub before joining wgbh as part of the national digital stewardship residency program. rebecca_fraimow@wgbh.org, @rhfraim julia kim is the digital assets specialist at the american folklife center at the library of congress. she manages the division’s born-digital and digitized assets, workflows, and systems. she has published and presented on audiovisual workflows, digital forensics and emulation use cases, researcher access to born-digital collections, and at-scale digital access workflows. she is active with the society for american archivists, the association for moving image archivists, and the international conference on digital preservation. she helped cofound the xfr collective, and is a new york national digital stewardship program alumna. juliakim@loc.gov, @jy_kim29 erica titkemeyer is the co-principal investigator and project director on an andrew w. mellon foundation grant titled extending the reach of southern audiovisual sources: expansion. in this role, they coordinate digitization of audiovisual materials from wilson library special collections at unc, as well as collections held by institutional partners across the state. they also support digital preservation policy development as a member of the unc libraries’ digital preservation stewardship committee, while providing guidance on born-digital processing of audiovisual files in the southern folklife collection. etitkem@email.unc.edu, @ekemeyer acknowledgments the authors would like to thank members of the audiovisual and digital preservation community for their positive feedback and encouragement. of special note are the contributors to ffimproviser, a community supported audiovisual resource mentioned throughout this paper. while there are too many contributors to acknowledge individually, we would like to especially thank dave rice and ashley blewer. we would also like to thank our audiovisual colleagues kate murray and crystal sanchez for their invaluable feedback. finally, we’d like to thank shira peltzman and karen cariani for moderating panels on this topic, as well as the association of moving image archivists and ipres for providing us the opportunity to discuss these challenges at their annual meetings. [1] federal agencies digital initiatives, “creating and archiving born-digital video,” available from: http://www.digitizationguidelines.gov/guidelines/video_borndigital.html?loclr=blogsig [2] jpeg 2000 part 1, core coding system [internet]. library of congress; [cite 2019 jan 9]. available from: https://www.loc.gov/preservation/digital/formats/fdd/fdd000138.shtml. [3] ff video codec 1 [internet]. library of congress; [cite 2019 jan 9]. available from: https://www.loc.gov/preservation/digital/formats/fdd/fdd000341.shtml. [4] for more discussion of the pros and cons of various preservation file formats, see peter bubestinger, hermann lewetz and marion jaks’ “comparing video codecs and containers for archives,” indiana university’s white paper “encoding and wrapper decisions and implementation for video preservation master files,” and emmanuel lorrain’s “a short guide to choosing a digital format for video archiving masters.” [5] fraimow f, kim j, and e titkemeyer, never best practices: born-digital audiovisual case studies from production, university, and federal government environments, discussion panel at the international digital preservation conference, boston,september 26, 2018. portions of this resulting paper were discussedin this panel. [6] carolina digital repository: https://cdr.lib.unc.edu/ [7] apple prores 422 codec family [internet]. library of congress; [cite 2019 jan 9]. available from: https://www.loc.gov/preservation/digital/formats/fdd/fdd000389.shtml. [8] matroska multimedia container [internet]. library of congress; [cite 2019 jan 9]. available from: https://www.loc.gov/preservation/digital/formats/fdd/fdd000342.shtml. [9] mpeg-4, advanced video coding (part 10) (h.264) [internet]. wikipedia; [cite 2019 jan 9]. available from: https://www.loc.gov/preservation/digital/formats/fdd/fdd000081.shtml. [10] ffmpeg, “h.264 video encoding guide”, available from: https://trac.ffmpeg.org/wiki/encode/h.264. [11] ffmpeg is a “leading multimedia framework, able to decode, encode, transcode, mux, demux, stream, filter and play pretty much anything that humans and machines have created.” documentation and more information is available from: https://www.ffmpeg.org/. [12] ffmprovisr, “extract audio fran and av file”, available from: https://amiaopensource.github.io/ffmprovisr/index.html#extract_audio. [13] droid “is a file format identification tool, which can process single or batched digital objects to determine their file type, size, age, and change history. droid can currently identify over 250 file formats, with more formats being added regularly.” more information is available from: http://www.dcc.ac.uk/resources/external/droid. [14] “brunnhilde was written to supplement existing file format identification and characterization tools, with a focus on producing human-readable high-level reports for digital archives and digital preservation practitioners.” more information is available from: https://www.bitarchivist.net/projects/brunnhilde/. [15] library of congress fiscal 2018 budget justification: submitted for the use of the committees of the appropriations [internet].[2017]. the library of congress/ [cited 2018 dec 4]. available from: https://www.loc.gov/about/reports-and-budgets/congressional-budget-justifications/, p.118. [16]for more information on afc’s digital preservation workflows, systems, and environments, see the library of congress, “the signal [internet].[2016]. “minimal digital processing at the american folklife center” available from: https://blogs.loc.gov/thesignal/2016/06/the-workflow-of-the-american-folklife-center-digital-collections/ and the library of congress, “the signal [internet]. [2018]. “the workflows of the american folklife center.” available from: https://blogs.loc.gov/thesignal/2016/06/the-workflow-of-the-american-folklife-center-digital-collections/. [17] my approach is inspired and amended from waugh d, russey roke e, farr e (2016) flexible processing and diverse collections: a tiered approach to delivering born-digital archives, archives and records [internet] [cited 2018 dec 4]; 37:1, 3-19. available from: 10.1080/23257962.2016.1139493 https://www.tandfonline.com/doi/abs/10.1080/23257962.2016.1139493 [18] for some background on the term “fugitive media” and its use in archives, see forstrom, m. “managing electronic records in manuscript collections: a case study from the beinecke rare book and manu­script library,” american archivist, vol. 72 (fall/winter 2009), pp. 460–77. https://www.jstor.org/stable/27802697 [19] this project was presented earlier this year by kim j, “time to clean out our closet: challenges accessioning fugitive media” bitcurator user forum, september 2018. [20] video cd [internet]. wikipedia; [cite 2018 dec 4]. available from:https://en.wikipedia.org/wiki/video_cd [21] interlaced video [internet]. [updated 2018]. wikipedia. available from: https://en.wikipedia.org/wiki/interlaced_video [22] while streaming moving image access targets are, in contrast, relatively stable, even in this case we had to transcode multiple times to meet the constraints of our streaming services. the anomalies meant that, for the first time, we had to ultimately outsource our derivative creation to meet our publication time frame. [23] “what we do” [internet]. wgbh; [cite 2018 dec 7]. available from https://www.wgbh.org/foundation/what-we-do/media-library-and-archives [24] wgbh open vault [internet]. wgbh media library and archives; [cite 2018 december 7]. available from http://openvault.wgbh.org/ [25] american archive of public broadcasting [internet]; [cite 2018 december 7]. available from http://americanarchive.org/ [26] avdn is a proprietary avid flavor of quicktime. wgbh primarily uses avid editing software, so many of our materials come to us as avid proprietary codecs. more information about these codecs is available at the avid knowledge base: http://avid.force.com/pkb/articles/en_us/download/en392959 [27] pcm, pulse code modulated audio [internet]. library of congress; [cite 2019 jan 9]. available from https://www.loc.gov/preservation/digital/formats/fdd/fdd000016.shtml [28] jpeg image encoding family [internet]. library of congress; [cite 2019 jan 9]. available from https://www.loc.gov/preservation/digital/formats/fdd/fdd000017.shtml [29] fits wraps together several file characterization tools, including droid, exiftool, and mediainfo, and generates a standard output that includes all generated technical metadata. more information is available here: https://projects.iq.harvard.edu/fits/home [30] another flavor of proprietary avid codec. [31] the working document is available at https://docs.google.com/spreadsheets/d/1ovzkgkiznnx_nz9ovdokjjivfuimk_7fykc77yhouac/edit?usp=sharing subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – analysis of 2018 international linked data survey for implementers mission editorial committee process and structure code4lib issue 42, 2018-11-08 analysis of 2018 international linked data survey for implementers oclc research conducted an international linked data survey for implementers in 2014 and 2015. curious about what might have changed since the last survey, and eager to learn about new projects or services that format metadata as linked data or make subsequent uses of it, oclc research repeated the survey between 17 april and 25 may 2018. a total of 143 institutions in 23 countries responded to one or more of the surveys. this analysis covers the 104 linked data projects or services described by the 81 institutions which responded to the 2018 survey—those that publish linked data, consume linked data, or both. this article provides an overview of the linked data projects or services institutions have implemented or are implementing; what data they publish and consume; the reasons given for implementing linked data and the barriers encountered; and some advice given by respondents to those considering implementing a linked data project or service. differences with previous survey responses are noted, but as the majority of linked projects and services described are either not yet in production or implemented within the last two years, these differences may reflect new trends rather than changes in implementations. by karen smith-yoshimura introduction the impetus for an “international linked data survey for implementers” was a set of discussions with oclc research library partnership (rlp)[1] metadata managers who were aware of some linked data projects or services but felt there must be more “out there” that they should know about. the survey instrument was designed in consultation with oclc colleagues and a few rlp institutions and beta tested by several linked data implementers. we conducted the initial survey in july – august 2014, distributing the link to the survey on multiple listservs and on twitter. the survey targeted those who had already implemented a linked data project or service, or were in the process of doing so. questions were asked both about publishing linked data and consuming linked data. the results were published in a series of posts on the oclc research blog, hangingtogether.org.[2] while the initial survey results received a generally appreciative response from readers, some noted and regretted the absence of several prominent linked data efforts in europe. to address these gaps, we repeated the survey in june-july 2015. the results were published in the july/august 2016 issue of d-lib magazine.[3] curious about what might have changed since the last survey, and eager to learn about new projects or services that format metadata as linked data or make subsequent uses of it, oclc research repeated the survey between 17 april and 25 may 2018. more institutions responded than in the previous survey (81 institutions, compared to 71 institutions in 2015) but described slightly fewer linked data projects or services (104, compared to 112 in 2015). spreadsheets[4] containing all responses to the 2014, 2015, and 2018 surveys include links to the linked data projects or services in production. overview a total of 143 institutions in 23 countries responded to one or more of the surveys (see the list appended at the end of this article). of the 81 institutions responding to the 2018 survey, 46 (57%) had responded to one or both of the previous surveys. even those who had responded to a previous survey did not always describe the same linked data projects or services. of the 104 linked data projects or services described, only 42 had been described previously. even when the same project or service was described, the respondent sometimes differed from the one who responded previously. some respondents did not answer every question, so the totals for each question may vary. half of the respondents to the 2018 survey that have implemented or are implementing a linked data project or service reported that they plan to implement another linked data project within the next two years, a slight decrease from the 60% who had plans within the next two years in 2015. but two of the 2015 respondents who reported that they had plans did indeed do so when repeating the survey in 2018. respondents from the united states to the 2018 survey accounted for 42%, with 34 institutions, followed by spain (12), the united kingdom (8), and the netherlands (4). we received three responses each from canada, germany, and norway, two responses from italy, and one response each from australia, austria, china, the czech republic, finland, france, hungary, japan, luxembourg, portugal, south africa, and switzerland. we categorized the responding institutions by type. research libraries represented 28% of the 2018 respondents (23), followed by 13 national libraries (16%), 11 research institutions (14%), eight library networks and eight government (10%), six service providers (7%), five public libraries (6%), four museums (5%), and two “others” (a concert venue and a publisher). this breakdown generally follows the types of institutions that responded to the previous survey with one exception—for the first time, we received responses from service providers, which provide linked data services for their customers. figure 1 compares the number of responding institutions to the three surveys by type of institution. figure 1:types of institutions responding to the surveys conducted in 2014, 2015, and 2018. of the 104 linked data projects or services described at various levels of detail by the 2018 survey respondents, three-fourths are in production, of which 23% (18) have been in production for more than two years but less than four, and 40% (31) have been in production for four or more years. three of them are not yet accessible, and three are “private,” for that institution’s use only. most of the linked data projects or services are done entirely in-house (61 or 59% of the implementations), with 23 (22%) part of a multi-institutional implementation, and 20 (19%) provided by an external vendor or service provider. but even those who responded that the work to deliver linked data functionality was all done in-house reported collaboration with external groups or organizations. only 25 (24%) of the implementations involved “only my institution.” in order of frequency named, the external collaborators were: other libraries or archives other universities or research institutions external consultants or developers a systems vendor part of an international collaboration and other consortium members part of a national collaboration a corporation or company part of a discipline-specific collaboration a foundation a scholarly society part of a stateor province-wide initiative. the rankings resemble those in the 2015 survey, with slightly higher rankings for a systems vendor, international collaboration, other consortium members, discipline-specific collaboration, and foundation. staffing: most of the institutions that have implemented or are implementing linked data projects/services have added linked data to the responsibilities of current staff (86); only 15 have not. the most significant change from the 2015 survey is that the number of those who have staff dedicated to linked data increased by 50%. in table 1, the first column shows the number of responses and the second column the percentage of all those who responded to the question. table 1: comparison of staffing in 2018 and 2015 surveys. staffing 2018 2015 added to the responsibilities of current staff. 86 85% 98 92% we have staff dedicated to linked data project(s). 30 30% 20 19% adding/have added new staff with linked data expertise. 10 10% 4 4% adding/have added temporary staff with linked data expertise. 13 13% 13 12% hiring/have hired external consultants with linked data expertise. 12 12% 17 16% respondents to question 101 107 funding: twenty-two of the linked data projects or services received grant funding; 73 (70%) are funded by the library/archive and/or the parent institution. six linked data projects received funding support from partner institutions, five were privately funded, and one received corporate funding. eight have never applied for a grant, but plan to. success assessment: more respondents reported that their linked data project or service was successful or “mostly” successful in 2018 than in 2015: 58 compared to 46. fewer didn’t know yet as their projects were still at an early stage (pre-implementation or early implementation): 31 in 2018 compared to 52 in 2015. comments from respondents whose linked data project or service has been in production for at least four years reveal the following indicators of success. usage: most respondents noted substantial increases in usage over the years, and more contributors. one noted the high search ranking for rare content. others noted the increase in requests for digital services support discovered through their linked data offerings. conference presentations and articles by others referring to their own work were cited as evidence of success. expanded exposure on the internet is “dragging people into library services.” data re-use: several noted that other applications making use of their linked data implementation is a metric of success. another metric is the number of bulk downloads. interoperability: several noted their linked data service provides access to their other resources. one noted the value of aggregating data from resources around the world to their users. user satisfaction: linked data offers users a richer experience that is much more contextualized and inter-related. one pointed to better support of multilingualism by fetching multilingual labels from linked data vocabularies. one noted that their “happy users” are “probably unaware that the service is based on linked data.” influence: the success of a project gains attention and illustrates what’s possible. several noted that their services are well-known in the community and developers in the cultural sector are increasingly aware of their value. their data models have influenced other initiatives and moved the discussion on linked data in the library community. professional development: even absent metrics demonstrating linked data’s value to others, linked data projects still provide professional development for staff. “we learned a great deal that we can build on.” the reasons given by implementers for assessing their project or service as partially or mostly successful centered around: the need to upgrade and expand the service but lacking funding for it lack of tools to measure satisfaction of the users with the linked data offering difficulty in assessing success of data dissemination and in determining how the data is re-used impression that interest isn’t high within the library community impact seems marginal did not result in similar projects, and the project did not grow in size and scale in both the 2018 and 2015 surveys, most projects/services both consume and publish linked data. even fewer of the described projects only publish linked data in the 2018 survey. table 2: survey responses on how linked data is used. how linked data is used 2018 survey 2015 survey consume linked data 34 38 publish linked data 5 10 both consume & publish 65 64 what and why linked data is published given the relatively large representation of libraries among respondents, it is no surprise that descriptive metadata and bibliographic data are the most common types of data published (51 and 47 responses respectively), with authority data a close third (45). other types of data published as linked data reported: data about people (33), ontologies/vocabularies (33), digital collections (27), geographic data (23), datasets (19), data about museum objects (12), organizational data (12), encoded archival descriptions (3), and statistical data (3). other responses included holdings and availability information, data about performance work, data bout monuments and archeology, provenance, and time periods. in the three years since the 2015 survey, the number of linked data datasets with over 1 billion triples increased from three to 11, and three of them are over 5 billion triples: biblioteca de galicia ‘s digital library (6.3 billion triples), europeana (a little over 5 billion), and oclc’s worldcat linked data (over 10 billion triples). but most of the linked data datasets are small. of the 63 responses reporting their datasets’ size, 33 were less than 10 million triples, nine were between 10 and 100 million triples, and 10 were between 100 million and 1 billion triples. comparing the 2018 and 2015 survey results, the key motivations to publish linked data appear unchanged. because some 2015 survey respondents had noted the need to publish linked data in order to consume it as an “other” motivation, we offered it as an option in the 2018 survey; it became the fourth most commonly cited reason. table 3: chief motivations for publishing linked data. chief motivations for publishing linked data (n=92 in both surveys) 2018 2015 expose data to a larger audience on the web. 74% 73% demonstrate what could be done with datasets as linked data. 65% 64% heard about linked data and wanted to try it out by exposing some local data as linked data. 45% 47% needed to publish linked data in order to consume it. 25% explore whether publishing data as linked data will improve search engine optimization (seo) for local resources. 24% 30% administration requested that we expose our data as linked data. 11% 5% the british library noted that its linked data implementations were part of the uk government public sector initiative. other reasons written in included: experiment with linked data outside the library catalog increase interoperability required for a grant-funded project to link together information across different institutions to provide an ontology extension to bibframe[5] develop supporting tools more than half of the responses either did not know the average number of requests the linked data project or service received daily over the previous six months, did not keep or have access to usage statistics, or had no usage yet (61 of 103 responses). the eight most heavily used linked data datasets as measured by the average number of requests a day, with over 100,000 requests per day, are: american numismatic society’s nomisma, a thesaurus of numismatic concepts. in the 2015 survey, it had reported daily usage as between 10,000 and 50,000 requests a day, so its usage has more than doubled over the last three years. bibliothèque nationale de france’s data.bnf.fr, providing access to the bnf’s collections and providing a hub among different resources. in the 2015 survey, it also had reported daily usage as between 10,000 and 50,000 requests a day. europeana, which aggregates metadata for digital objects from museums, archives, and audiovisual archives across europe. it had reported the same daily usage in 2015. library of congress’ linked data service with over 50 vocabularies. although usage fluctuates, it receives 500,000 to a million requests a day. national diet library’s ndl search, providing access to bibliographic data from japanese libraries, archives, museums and academic research institutions. it had reported the same daily usage in 2015. north rhine-westphalian library service center’s linked open data service, providing access to bibliographic resources, libraries and related organizations, and authority data. it had reported the same daily usage in 2015. oclc’s virtual international authority file (viaf), an aggregation of over 40 authority files from different countries and regions. it had reported the same daily usage in 2015. oclc’s worldcat linked data, a catalog of over 400 million bibliographic records made experimentally available in linked data form. it had reported the same daily usage in 2015. another three linked data datasets receive between 50,000 and 100,000 requests a day: british library’s british national bibliography. in the 2015 survey, it had reported daily usage as between 10,000 and 50,000 requests a day, so its usage has doubled over the last three years. national library of finland’s finnish thesaurus and ontology service. (although this service was launched in 2014, it did not respond to the 2015 survey.) oclc’s fast (faceted application of subject headings), a faceted subject heading schema derived from library of congress’ subject headings. in the 2015 survey, it had reported daily usage as between 10,000 and 50,000 requests a day, so its usage has doubled over the last three years. linked data datasets use a variety of rdf vocabularies and ontologies, and most use multiple ones. the percentage of those using simple knowledge organization system (skos) decreased from 60% of respondents in 2015 to 44% in 2018, which was mirrored by an increase in those using schema.org (30% in 2015 vs. 46% in 2018). similar decreases showed up in those using dublin core metadata element set, dcmi metadata terms, and friend of a friend (foaf), as well as a smaller decrease in those using rdf schema. bibframe vocabulary usage increased, from 15% in 2015 to 27% in 2018. the table below shows the top eight in 2018 (used by at least 20% of respondents) compared to the 2015 responses. ninety-five responded to the question in 2018 vs. 99 in 2015. the first column shows the number of responses and the second column the percentage of all those who responded to the question. table 4: top 8 rdf vocabularies/ontologies used in 2018 compared to 2015. rdf vocabularies/ontologies used 2018 2015 schema.org 44 46% 30 30% skos 42 44% 59 60% dublin core terms 39 41% 51 52% foaf 36 38% 55 56% dcmi metadata terms 35 37% 49 49% rdf schema 35 37% 45 45% bibframe 26 27% 15 15% local vocabulary 21 22% 19 19% other rdf vocabularies and ontologies named by the 2018 survey respondents in order of the frequency cited are: resource description and access (rda) europeana data model vocabulary (edm) the bibliographic ontology (bibo) cidoc conceptual reference model (crm) expression of core frbr concepts in rdf (frbr) metadata authority description schema (mads) owl 2 web ontology language (owl 2) isbd elements (isbd) wgs84 geo positioning (geo) the event ontology (event) music ontology (mo) the oai ore terms vocabulary (ore) international standard name identifier (isni) metadata object description schema (mods) vivo core ontology (vivo) dpla metadata application profile (map) biographical ontology (bio) frbr-aligned bibliographic ontology (fabio) organization ontology (org) data catalog vocabulary (dcat) eac-cpf descriptions ontology for linked archival data (eac-cpf)rda group 2 elements (rdag2) british library terms rdf schema (blt) library extension of schema.org [aka purl.org/library] (lib) archival collections ontology (arch) nomisma ontology thirty-seven respondents cited rdf vocabularies/ontologies not listed above. in alphabetical order they are: activity streams, description of a project (doap), exif data description vocabulary, funding, research administration and projects ontology (frapo), gnd ontology, international image interoperability framework (iiif), muninn project ontologies, national diet library dublin core metadata description (dc-ndl), ontology for media resources, product ontology, skos extension for labels (skos-xl), upper mapping and binding exchange layer (umbel), vra core, and web annotation data model. licenses[6]: 32 projects/services do not announce any explicit license; 19 apply cc0 1.0 universal, the most common license used by the 2018 survey respondents. other licenses that respondents use, in order of frequency, are: open data commons attribution (odc-by) public domain dedication and license (ppdl) open data commons open database license (odc-odbl) creative commons attribution-noncommercial-noderivatives (by-nc-nd) other licenses cited by 29 respondents not listed above: creative commons attribution-noncommercial-sharealike (by-nc-sa) creative commons public domain mark 1.0 creative commons attribution 3.0 united states (cc-by 3.0 us) creative commons attribution 4.0 international (cc-by 4.0) french government’s open licence (similar to odc-by) open government license [uk] accessibility: of the 70 projects or services that publish linked data, 19 do not currently make their data accessible outside their institution. those that do offer multiple access formats. web pages are the most common, followed by file dumps, content negotiation, sparql endpoint, sparql editor, embedded markup, and applications. the most common serialization of linked data used is rdf/xml. other serializations that are less often used by order of frequency cited: turtle, json-ld, n-triples, rdf/json, rdfa, n3 rdf triplets, and n-quads. technologies: the technologies used by respondents to publish linked data are diverse, and most used multiple technologies. table 5 lists the technologies used in order of frequency. table 5: technologies used for publishing linked data. no. of projects used technology (order of frequency) more than 20 sparql, java 10 – 20 python, xslt, rdf store, solr, jena applications, virtuoso universal server (provides sparql endpoint), 2 – 9 google refine, apache fuseki, blazegraph, graphdb (formerly owlim by ontotext software), digibib for libraries, 4store, fedora commons, map/reduce, metafacture, django, elasticsearch, allegrograph, drupal7, openrdf, pubby 1 4store seme4 platform, amazon neptune, apache spark, arc2 on php, arches, bib-lod-ui (web app for publishing bibliographic linked open data), bib-rdf-pipeline (converting marc into rdf), blacklight, catamandu, cliopatria, cubicweb, d3 libraries, fast converter, freemarker templates, government site builder [germany], hbase/hadoop, jax-rs, marc report and marc global (from the marc of quality), mongodb, mapping memory mapper (3m), marklogic semantics, orbean xforms, permanent identifiers for the web, rdflib for python, researchspace, ruby on rails, skosmos skosify easyrdf library for php, sparql result visualizer, squebi sparql editor, stardog barriers: the rankings of barriers or challenges in publishing linked data are mostly the same in both the 2018 and 2015 surveys. the top barrier in both surveys was the steep learning curve for staff. so many wrote in “lack of resources” as a response to the 2015 survey that it was added as another choice in the 2018 survey, becoming the fourth-most cited barrier, tied with little documentation. table 6: barriers encountered in publishing linked data. barriers/challenges 2018 2015 steep learning curve for staff 41 51% 40 51% inconsistency in legacy data 38 48% 33 42% selecting appropriate ontologies to represent our data 26 33% 31 39% lack of resources 23 29% little documentation or advice on how to build the systems 23 29% 21 27% establishing the links 22 28% 27 34% lack of tools 18 23% 15 19% immature software 17 21% 11 14% ascertaining who owns the data 4 5% 10 13% other 19 24% 21 27% respondents to question 80 79 several respondents also noted as additional barriers the complexity of data transformations, lack of best practices, lack of tool integration, scaling triplestores, security and privacy issues, data sets too large to publish as a whole (and difficult for others to consume), and insufficient institutional support. what and why linked data is consumed in the 2018 survey, a total of 69 projects described consumed linked data (compared to 68 projects in 2015). table 7 shows the top 10 linked data sources consumed by the 2018 survey respondents (used by over 12 projects or services) compared to their usage in the 2015 survey. the first columns under 2018 and 2015 are the number of projects reporting that they consumed the resource, and the second columns are the percentage the resource is used of all projects that consumed linked data described in that year. asterisks denote resources from institutions that responded to the 2018 survey. percentage differences greater than 10% are denoted with a ^. the biggest change: the surge in usage of wikidata, ranking #5 among all the linked data sources by the 2018 survey respondents (compared to #15 in the 2015 survey), now tying worldcat.org in usage. isni also rose to be in the “top 10,” while “resources we convert to linked data ourselves” fell from five in 2015 to 10 in 2018. table 7: top 10 linked data sources consumed. top 10 linked data sources consumed 2018 2015 id.loc.gov* 39 57% 35 51% viaf (virtual international authority file)* 36 51% 41 60% dbpedia 30 43% 36 53% geonames 29 42% 35 51% wikidata 28 41%^ 6 9% worldcat.org* 28 41%^ 15 22% getty vocabularies 23 33% 16 24% fast (faceted application of subject terminology)* 17 25% 15 22% isni (international standard name identifier) 17 25%^ 8 12% resources we convert to linked data ourselves 13 19% 17 25% these could be considered successful publishers of linked data by the degree to which others consume the data provided. although library-related linked data projects have tended to consume linked data from sources in the library domain, the emergence of wikidata as a top resource, along with dbpedia and geonames, indicates more experimentation with expanding implementations’ scope to non-library sources. just under half of the linked data projects and services that now consume wikidata as a linked data source (13) have been in production for over four years. other linked data sources consumed by at least four projects or services in order of frequency cited: europeana*, deutsche national bibliothek linked data services*, lexvo, worldcat.org works, data.bnf.fr *, orcid (open researcher and contributor id), dpla* (digital public library of america), and hispana. the primary reasons for institutions to consume linked data had the same top rankings in both the 2018 and 2015 surveys, as shown in table 8. decreases of 10% or more appear in aspirations to improve seo for local resources (from 28% in 2015 to 10% in 2018); achieve more effective internal metadata management (from 47% in 2015 to 30% in 2018); and to provide greater accuracy and scope in local search results (from 40% in 2015 to 28% in 2018). table 8: chief motivations for consuming linked data. chief motivations for consuming linked data 2018 2015 provide local users with a richer experience. 78% 75% enhance local data by consuming linked data from other sources. 71% 74% heard about linked data and wanted to try it out by using linked data resources. 33% 25% more effective internal metadata management. 30% 47% experiment with combining different types of data into a single triple store. 29% 25% greater accuracy and scope in local search results. 28% 40% explore whether consuming linked data from external sources will improve search engine optimization (seo) for local resources. 10% 28% respondents to question 69 68 barriers: the top barrier in both the 2018 and 2015 surveys was matching, disambiguating, and aligning source data and linked data sources. the biggest difference was an uptick in the number of responses pointing to unstable endpoints and service reliability, as shown in table 9. table 9: barriers encountered in consuming linked data. barriers/challenges 2018 2015 matching, disambiguating and aligning source data and the linked data resources 28 48% 23 39% what is published to the internet as linked data is not always reusable or lacks uris 18 31% 16 27% size of rdf dumps 16 28% 12 20% unstable endpoints 16 28% 10 17% service reliability 15 26% 9 15% mapping of vocabulary 15 26% 17 29% understanding how the data is structured before using it. 14 24% 12 20% lack of needed off-the-shelf tools 14 24% 10 17% datasets not being updated 13 22% 14 24% lack of authority control 11 19% 15 25% volatility of data formats of dumps 10 17% 11 19% disambiguation of terms across different languages is difficult 8 14% 6 10% it’s difficult to get other institutions to do their own harmonization between objects and concepts. 7 12% 9 15% other 13 22% 15 25% respondents to question 58 59 other barriers and challenges written in included: ill-formed rdf serializations; automation processes to link the data are undeveloped; triplestores are not ideal for interactive workflows; complexity of current modeling; and the amount of data cleanup that is required. advice asked what they would do differently if they started their linked data project or service again, respondents wrote about integrating linked data into existing services, remediating legacy metadata, and adjusting expectations, such as: linked data is now more mature; we would therefore have a wider frame of reference. we would seek to develop a system more integrated with our core services. embed it better in our existing infrastructure. take a more holistic perspective and try to incorporate the project more into existing procedures. do more data enhancement at time of conversion. the clean-up of data sets would have benefitted from wider organizational support. the check and correction of the catalog’s data need more staff dedicated in it. the group of data on which the project would be carried out would be more consistent and we would try to give more visibility to the results. maybe we would try other reconciliation services, i.e., beyond openrefine. i would set realistic expectations and make sure that people didn’t only focus on google search results, but also understand how the data is used directly. we are still figuring out the role of bibliographic linked data within our organization and its various processes. however, creating this service has been a major milestone and helped refocus the discussion by making it plain and clear what the possibilities (and limitations) are. respondents offered advice for others considering a linked data project. some recurring themes included: take advantage of the many more exemplars of good practice and information related to linked data; read as widely as possible, including w3c recommendations, and consult with community experts. focus on what you want to achieve, develop internal and external use cases, match the use cases and user needs with what the data can provide, and be prepared to work with multiple data models. define the scope and requirements before starting any work. develop a long-term plan for integration support and ongoing maintenance. consider joining up with others rather than doing everything yourself. integrate work on linked data projects into your daily workflows. never underestimate the amount of data cleanup that will be required. think about acceptable levels of quality before starting. analyze your legacy data to determine what should be converted to linked data. avoid minting your own identifiers for everything, but instead use those already created by authoritative linked data sources whenever possible. use existing ontologies whenever possible rather than creating your own. create your own only to fill gaps. select the most productive linked data sources by both the content and number of links to other linked data sources. make sure that linked data publishers are committed to ensuring their data is persistent. plan on how you will measure impact as early as you can; line up some early users who would benefit from the work. expect that you will start to see benefits only after you have reached scale. be prepared to iterate based on feedback from your users. communicate, communicate, communicate! publicize your results and how your target audiences will benefit. conclusion the responses to the 2018 survey may be considered another partial snapshot of the linked data environment still evolving. this view is limited by which institutions respond to the survey and who responded, as responses even from linked data implementations described in earlier surveys may differ because of different individual perspectives. responses from libraries (individual libraries, research libraries, national libraries, library networks) still predominate. even with an overlapping but different respondent pool, responses to both the 2015 and 2018 surveys are similar in many aspects. the chief motivations for publishing and consuming linked data, and the primary barriers encountered are the same. growth of activity and usage among the more mature linked data services are encouraging. the emergence of service providers may lead to fewer individual institutions launching their own linked data projects. among the 2018 survey respondents, 37% relied at least partially on a system vendor, corporation, or external consultants or developers to implement their linked data project or service, and several institutions were clients of providers who also responded to the survey. commercial providers did not see themselves in some of the survey questions, which are oriented toward cultural heritage institutions, and especially libraries. forty-two percent of the “new” responses to the survey (not described in previous surveys) were outside the library domain. besides the new category of service providers noted earlier, we see more linked data initiatives from research institutions and cultural heritage organizations, such as australian national university, carnegie hall, the historic environment scotland, and the rijksmuseum. alas, springer nature remains the only publisher to respond to these surveys. but the growing diversity of linked data implementations reflected in the survey responses is suggested by the wide range of ontologies and technologies used. a few noticeable differences from the 2015 survey responses: even fewer projects or services that only publish linked data, with a mirrored increase of those that both publish and consume linked data. more staffing dedicated to linked data. an increase in publishing linked data in schema.org and bibframe, mirrored by a decrease in skos. the rise of wikidata as a linked data source. noted the national library of finland, “wikidata is becoming more and more significant for cultural heritage institutions, including our library.” as the majority of linked projects and services described are either not yet in production or implemented within the last two years, these differences may reflect new trends rather than changes in implementations. most linked data projects and services remain experimental or educational in nature. observes the oslo public library, “as far as i can see, oslo public library is still the first and only library with its production catalogue and original cataloguing workflows done directly with linked data.” most implementers encourage more linked data initiatives. one advocate wrote in: “this is the future of data for libraries and the longer we wait the further behind we’re going to fall.” appendix: institutions responding to international linked data survey for implementers table 10: institutions responding to international linked data survey for implementers. responding institution country 2018 survey 2015 survey 2014 survey agence bibliographique d’lenseignement supérieur (abes) france x agencia española de cooperación internacional para el desarrollo (aecid) spain x x american antiquarian society usa x american numismatic society usa x x x anythink libraries usa x x arapahoe library district usa x archaeology data service (uk) uk x x australian national university australia x australian war memorial australia x* bavarian state library germany x biblioteca de castilla y león spain x* biblioteca de galicia spain x biblioteca della camera dei deputati (italy) italy x x biblioteca. real academia nacional de medicina spain x biblioteca valenciana nicolau primitiu spain x x biblioteca virtual de derecho aragonés spain x x bibliotheque nationale de france france x x bibliothèque nationale de luxembourg luxembourg x bibsys ntnu (norwegian university of science and technology) norway x x x big data institute canada x british library uk x x x british museum uk x* x x britain memorial library uk x* campus condorcet france x* carleton college usa x x carleton university canada x* carnegie hall usa x casalini libri (share-vde group) italy x charles university in prague czech republic x x chemical heritage foundation usa x colorado college usa x x x colorado state university usa x x columbia university usa x x* consejería de educación, cultura y deportes gobierno de castilla-la mancha, españaa spain x consorci de serveis universitaris de catalunya spain x* x coordinamento delle biblioteche speciali e specialistiche di torino (cobis) italy x cornell university usa x x x credo reference usa x* cultural heritage agency of the netherlands the netherlands x dartmouth college usa x data archiving and networked services, royal netherlands academy of arts and sciences the netherlands x defense language institute foreign language center (dliflc) aiso library usa x* digital public library of america usa x x x diputación de málaga. cultura y deportes. biblioteca cánovas del castillo (biblioteca virtual de la provincia de málaga) spain x x drexel university libraries usa x east china normal university library china x europeana foundation the netherlands x x x evansville vanderburgh public library usa x frankfurt am main university library germany x* free university of amsterdam the netherlands x* fundacción ignacio larramendi (spain) spain x x x george mason university, roy rosenzweig center for history and new media usa x* george washington university usa x german national library (deutsche nationalbibliothek) germany x x goldsmiths’ college uk x haute école de gestion de genève (swissbib) switzerland x historic environment scotland uk x international institute of social history the netherlands x* j. paul getty trust (getty research institute) usa x x johns hopkins university usa x* koninklijke bibliotheek the netherlands x korea national library of medicine south korea x* laurentian university canada x library link network usa x library of congress usa x x x lund university sweden x* memorial university of newfoundland canada x ministry of defense (spain) (ministerio de defensa ) spain x x minnesota historical society usa x x missoula public library usa x x mt. lebanon public library usa x* national diet library japan x x national library board (nlb) of singapore singapore x national library of finland finland x national library of malaysia malaysia x national library of medicine usa x x x national library of portugal portugal x x national library of scotland uk x national library of spain (biblioteca nacional de españa) spain x x national library of sweden sweden x national library of wales uk x* x x* national széchényi library hungary x x new york public library usa x new york university usa x x north carolina state university libraries usa x* x x north rhine-westphalian library service center (hbz) germany x x ntnu (norwegian university of science and technology) university library norway x x oclc usa x x x ohio state university usa x* oregon state university usa x* oslo public library norway x x x pratt institute usa x x prueba spain x public record office, victoria australia x queen’s university library australia x rero – library network of western switzerland switzerland x* x research libraries uk uk x rhodes university south africa x* rijksmuseum amsterdam the netherlands x royal commission on the ancient and historical monuments of scotland uk x* seme4 uk x singapore integrated library automation services (silas) singapore x* smithsonian usa x x x spanish office of library cooperation spain x* springer usa x x x stanford university usa x x x stichting bibliotheek.nl the netherlands x swiss national library switzerland x* the european library the netherlands x x thematix usa x* tresoar (leeuwarden – the netherlands) the netherlands x università degli studi roma tre italy x university college dublin ireland x university college london (ucl) uk x university of alberta libraries canada x x x university of applied sciences st. poelten austria x* x university of arkansas usa x* university of bergen library norway x* x* x university of british columbia canada x university of california-irvine usa x x university of california-los angeles usa x university of chicago usa x university of colorado boulder usa x* university of florida usa x* university of illinois at urbana-champaign usa x x university of limerick ireland x* university of liverpool uk x university of maryland usa x* university of nevada, las vegas usa x x university of north texas usa x university of oklahoma libraries usa x university of oxford uk x x university of pennsylvania libraries usa x x x university of south florida, st. petersburg usa x university of tennessee, knoxville usa x university of texas at austin usa x x university of washington usa x* university of wisconsin – madison usa x villanova university usa x wellcome library uk x western michigan university usa x x woods hole oceanic institute (mblwhoi) usa x* yale center for british art usa x zeitschriftendatenbank germany x* *institutions reporting linked data projects but described none references [1] the oclc research library partnership allies like-minded research libraries in twelve countries, providing a venue to undertake cooperative actions that benefit scholars and researchers everywhere. see: https://www.oclc.org/research/partnership.html [2] the results of the 2014 survey were posted on hangingtogether.org between 28 august 2014 and 8 september 2014: linked data survey results 1—who’s doing it linked data survey results 2—examples in production linked data survey results 3—why and what institutions are consuming linked data survey results 4—why and what institutions are publishing linked data survey results 5—technical details linked data survey results 6—advice from the implementers [3] smith-yoshimura. 2017. “analysis of international linked data survey for implementers.” d-lib magazine, 22(7/8), 141–167. http://doi.org/10.1045/july2016-smith-yoshimura. [4] spreadsheets with the complete responses to the 2014, 2015 and 2018 international linked data survey for implementers (without the contact information which we promised we’d keep confidential) are publicly available at: http://www.oclc.org/content/dam/research/activities/linkeddata/oclc-research-linked-data-implementers-survey-2014.xlsx. [5] bibframe is the bibliographic framework initiative launched by the library of congress to provide a foundation for the future of bibliographic description in the broader networked world. for details, see: https://www.loc.gov/bibframe/ [6] the w3c provides an overview of licensing for linked open data at https://www.w3.org/wiki/taskforces/communityprojects/linkingopendata/datalicensing. for open data commons (odc) licenses see https://creativecommons.org/licenses/.; for creative commons (cc) licenses see https://creativecommons.org/licenses/. about the author karen smith-yoshimura is a senior program officer working with research institutions affiliated with the trans-national oclc research library partnership. she focuses on issues related to metadata needed to describe and provide access to the multilingual resources managed by libraries, archives, museums, and other cultural heritage organizations. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – how to hack it as a working parent mission editorial committee process and structure code4lib issue 28, 2015-04-15 how to hack it as a working parent the problems faced by working parents in technical fields in libraries are not unique or particularly unusual. however, the cross-section of work-life balance and gender disparity problems found in academia and technology can be particularly troublesome, especially for mothers and single parents. attracting and retaining diverse talent in work environments that are highly structured or with high expectations of unstated off-the-clock work may be impossible long term. (indeed, it is not only parents that experience these work-life balance problems but anyone with caregiver responsibilities such as elder or disabled care.) those who have the energy and time to devote to technical projects for work and fun in their off-work hours tend to get ahead. those tied up with other responsibilities or who enjoy non-technical hobbies do not get the same respect or opportunities for advancement. such problems mirror the experiences of women on the tenure track in academia, particularly women working in libraries, and they provide a useful corollary for this discussion. we present some practical solutions for those in technical positions in libraries. such solutions involve strategic use of technical tools, and lightweight project management applications. technical workarounds are not the only answer; real and lasting change will involve a change in individual priorities and departmental culture such as sophisticated and ruthless time management, reviewing workloads, cross-training personnel, hiring contract replacements, and creative divisions of labor. ultimately, a flexible environment that reflects the needs of parents will help create a better workplace culture for everyone, kids or no kids. by jaclyn bedoya, margaret heller, christina salazar, and may yan background north american culture gives lip service to an idealized version of the nuclear family while our systems and society consistently fail to support parents and extended families in meaningful ways. one might expect that academia (and libraries) with its stereotypically liberal environment would be more progressive in this area. yet the tenure-track process and maternity leave policies highlight that many colleges and universities are as — or even more — unfriendly to families as the corporate world. we were inspired to write this article to prepare those thinking of becoming parents or trying to cope as new parents. we want to shed light on some of the gender inequalities that come with being a working mother in a technical environment and spur a dialog that can bring about necessary social change. this situation does not just negatively impact parents, but all workers. we all want to be able to pursue meaningful careers without having to give the best hours and years of our lives to careers that take precedence over relationships. how disheartened i’ve been at all the times i’ve heard “i learned this programming language” or “i installed that operating system” in the wee hours of the night when we’re officially off the clock and hopefully in bed. as a mom i no longer have the luxury of spare time. — christina, single mother of 4 year old as working mothers, we are especially aware of how social and cultural expectations affect women, and indeed research bears out that working mothers have special challenges. women (particularly low-earning women) on average make less after having a child, whereas men on average make more money after becoming fathers (budig & hodges, 2010). much of the research on working parents focuses on women, but we believe that a technical working culture based on expectations for constant productivity without regard for personal life is damaging to everyone with caretaking responsibilities, and indeed to all workers (kantor, 2014). thus we will use the word “caretaker” when we are discussing these general problems, and “mother” or “parent” when those words are appropriate. all workers have interconnected personal and professional lives, and life events such as taking care of a sick pet or driving aging parents to medical appointments are a part of life for many without children. such events may prevent caretakers from participating in the professional world to a degree that can seriously hurt or end their careers. the library technology community becomes richer by including everyone, regardless of the events and circumstances in their lives. allowing flexibility by respecting different levels of ability to participate means that everyone — kids or no kids — can remain active participants throughout their professional lives. we looked to the literature in a few separate domains to understand this problem including articles on the role and interests of women in technology as it intersects with their personal lives, sexism in academia, parenting in academia (particularly in academic libraries), as well as advice about productivity from a theoretical and practical view. what follows is particularly relevant to librarians working in technical positions in academic libraries but draws inspiration from across, and outside of, the academy. i started my year-long maternity leave just six months after getting my permanent position. so when a service opportunity came up as an association division president, i went for it even though i knew i was going to be taking on a lot of responsibilities during the leave. i was naive and thought i was going to have quiet time and more help with my baby. reality hit when i was struggling to plan a conference while juggling a very active infant with no help from our families. luckily my partner was a very involved parent, so i did my association work after he came home from work, and late at night whenever i wasn’t up tending to the baby. — may, mother of 23 month old tenure productivity expectations have been historically set to measure against the ideal worker: a man with a supporting wife taking care of his home, allowing him to devote time to his research at all hours of the day (wyatt-nichol, cardona, & drake, 2012; sotirin, 2008). the tenure schedule for the typical academic also coincides with the times many wish to start families or struggle with the challenges of parenting young children. women working in academia are generally disadvantaged compared to male counterparts because of obligations at home as a primary caregiver and the demands of a tenure-track career. many women find themselves unable to stay on the tenure track or even start such a job in the first place (wolf-wendel & ward, 2006; ward & wolf-wendel, 2012; mason, 2013; mason, wolfinger, & goulden, 2013). it is not just women who feel the pressures of parenting in academia; research shows that men, as they take on more caregiving responsibilities in the home, struggle to find balance between work and family while navigating tenure (sallee, 2014; philipsen & bostic, 2010). tenure track academic librarians have an additional pressure of a different working environment compared to their teaching faculty counterparts. the place-based nature of library work and the service environment of the library means many librarians are working full time in the library all year without the summer break afforded teaching faculty to pursue research and professional development. these librarians have to move research and service work out of regular office hours and into their own personal time in order to get tenure or qualify for promotions (templer, 2010; spires, 2007; zemon & bahr, 2005). considering these challenges, it is not surprising that many librarians choose to opt out of pursuing tenure-track positions in libraries or are forced out after starting. but in many libraries, even non-tenure track librarians have to work with promotion and advancement expectations that depend on service, research, and professional development (templer, 2010; graves, xiong, & park, 2008). caregivers are at a significant disadvantage. in addition to the quotidian struggles with work-life balance, how can they work on research and service during their personal time, when that has been significantly reduced or is completely nonexistent due to their caregiving responsibilities? those working in technical areas have an additional need to learn new technology skills to keep pace with developments in the field. if this time is not built into the work day, how are workers to find the time to do this outside of sacrificing sleep — assuming children do not need them during those hours when parents should normally sleep? many librarians struggle with work schedules that do not support their roles as new parents. it is not easy to maintain breastfeeding a baby (and pumping at work) if one is scheduled to be on the reference desk for several hours a day. for technology workers trying to minimize impact on patrons, scheduling system downtime for off-peak hours means juggling to arrange childcare because daycare and schools are closed. for single parents with limited support, it may mean being forced to choose between family and work on a regular basis. because libraries are open to patrons for extended hours, staff have to make sure systems are functioning, which may also include supervising staff in person on evenings and weekends. the “always on” nature of digital systems and electronic resources sets up expectations from patrons that the materials will be available at all times. technical library workers feel pressure to maintain a level of service to patrons past regular business hours. the same technologies that allow catching up on email during the commute to pick up a child are the same ones that impose pressure to answer patron requests for help when parents should be helping toddlers brush their teeth. to increase their productivity despite having fewer hours available, successful faculty parents have learned to be more efficient with the time they spend at work (philipsen & bostic, 2008; wolf-wendel & ward, 2006, sallee, 2014; o’reilly & hallstein, 2012). technical caregivers can learn to be more focused and productive at work. this will help maximize the time when they have childcare and not sacrifice personal time. giving up even as little as eight weeks away from my professional commitments seemed impossible, so i worked all through maternity leave using my phone and tablet to stay connected even while holding my baby. i returned to active professional life so early thanks to a dedicated husband who went against american cultural norms and took an equal length of paternity leave after my maternity leave ended. — margaret, mother of 15 month old son proposed solutions the following practical solutions to some of the caregiving problems highlighted above model a better culture that we as a society should work towards. some of the solutions can be taken up at the personal or departmental level, while others require changes at the administrative level. personal productivity solutions time management and tools david allen’s getting things done (2001) and cal newport’s blog study hacks address productivity techniques particularly relevant to technical work. both methods suggest that one can identify all the “things” one needs to get done as a series of projects, though they differ on the ideal way to plan those projects. getting things done suggests dividing up each project into a series of tasks called “actions”, which are captured in a trusted system (whether digital or paper), organized by context (i.e. phone, computer, office), and updated weekly. by arranging work in this way it is easier to identify appropriate times to accomplish work, and to keep focus on the current project by knowing that all the other work is accounted for. ubiquitous computing means that for people who work on computers there always seems to be something to do, which leads to answering emails or other busy work at all hours. newport suggests setting a fixed schedule with strict limits on the amount of time to work in a week — for instance, 40 hours — and careful planning to ensure that the most important projects happen that week (2009). by dedicating long blocks of uninterrupted time to projects, it is easier to accomplish “deep work” (solving problems that can’t be subdivided into smaller tasks), into which much programming and other technical work falls. success requires strict time management, but even more importantly, it requires managing expectations of patrons and co-workers about availability. lightweight project management tools such as asana or trello can keep project work out of email, reduce the need for meetings, and clearly indicate what team members are accomplishing even if they are not physically present. using such tools to track personal and work projects helps prevent individuals from taking on more than they can reasonably handle. keeping work out of lengthy email discussions is best, since these can lead to misunderstandings, as well as create expectations for answering emails outside of work hours. administrative solutions reviewing workloads and division of labor after having children, accomplishing extra work outside of normal working hours may no longer be possible. individuals should reset expectations of what is a reasonable workload for a given work day and set out to meet those new goals, with support from their supervisors. this means reevaluating priorities within departments. projects must be planned realistically without relying on absorbing failures by increasing workload without warning for staff. balancing work and the rest of life requires managing expectations of others and one’s own project load. some of this is through personal choices, some of it institutional. in academic institutions where service is a part of promotion and tenure, administrators who might be tempted to ask women or members of underrepresented groups to serve on committees should ensure that they are not overburdening these people with service commitments. in fact, too many service appointments may be the cause for the gap between women and men in achieving tenure (mason, wolfinger, & goulden, 2013). individuals should ensure that they are not taking on too many commitments as well by culling projects and committees. cross-training personnel and creating backups take time to identify all mission-critical systems and tasks, and ensure there are backup personnel or contingency policies in place. cross-training ensures critical library functions are protected by having multiple people versed in all work functions. this makes the institution more resilient at times when key personnel leave or a position is cut. in many situations, it is advisable to train both within and outside of a department. learning new skills and working across departments create opportunities to seed new ideas and may lead to better collaborations in the library. day to day, this means that employees are able to take sick time and vacation time without the work of the library coming to a standstill. sharing ownership of responsibilities helps alleviate scheduling problems should they arise. managers can shift staff and librarian responsibilities when someone takes a planned or unplanned extended leave of a few weeks or a few months. policy solutions many of the following policy recommendations taken from research into supporting academic faculty in work-life balance will also make a big difference in the library (templer, 2010; philipsen & bostic, 2010, ward & wolf-wendel, 2012). making changes in policy will require the support of leaders in institutions. union activism may help push things forward in an organized labour environment. elsewhere, working with shared governance or other structures may be required, as long as policies set for teaching faculty are equally available to faculty librarians. parental leaves library administrators must offer paid leaves for workers who are becoming new parents, and must encourage the use of these leaves by creating a culture where this is supported. unlike its counterparts in europe and the americas, the united states is alone among industrialized nations in having no paid maternity leave (world policy forum, n.d.). many us employees plan for the caring of their infants by coordinating their pregnancy to meet the requirements allotted by the family and medical leave act for unpaid leave, but rely on any provisions provided by their workplace for funding. many parents simply cannot afford to take unpaid leave. pregnancy and having children are a natural part of life and should not be stigmatized by looking upon it as a disability. as such, parental leaves should be separated from sick and disability leaves (philipsen & bostic, 2008, 2010; templer, 2010; ward & wolf-wendel, 2012). employees returning to work from parental leaves still need to have sick days for when they or their children become ill. hire replacement workers when employees take leave so others do not feel burdened by having to take on additional responsibilities, and employees will be willing to take leave knowing they will not impose upon their colleagues. the profession can benefit from creating a pipeline of qualified workers when librarians train in high learning curve technological skills by taking on these contract positions. the private sector is starting to see the real benefits of paid and longer maternity leaves on employee retention (wojcicki, 2015; mcgregor, 2015; carrns, 2013; florentine, 2014). administrators that wish to retain talented workers (particularly those with specialized technical skills) should do the same in order to remain competitive. on-site childcare/after-school care/emergency childcare providing onsite childcare and afterschool care is ideal. however, even in organizations that provide it, there is often much more demand than spaces available. there will always be demand for more, and so offering assistance in locating or affording childcare may be an acceptable alternative. practically, for most parents, what often is most needed, but missing, is emergency childcare. when regular care providers fall through because of weather or illness, being able to make alternate arrangements makes it possible to meet work commitments without having to take time off. this can be accomplished by subscribing institutions to a service that provides emergency care, allowing employees to make use of the service to make arrangements as needed. this system has been put in place at several american universities (mason, 2013). additionally, allowing extra “family-friendly” days, outside of regular vacation and sick time, leaves room for unexpected absences. telecommuting the same technology that makes it possible to keep working at all hours should imply that work can be done from anywhere. telecommuting might not be suitable or possible for all positions, but it might be ideal for caregivers working in mainly technical roles. a couple of days per week for parents with young children, or a few months while someone needs to take care of an ailing parent in another city, are viable options for both the library and employee to help navigate difficult periods (luce & hartman, 1984; duncan, 2008). unfortunately for many librarians working in a service environment, there can be resistance to telecommuting, even for positions that can be done off-site. think carefully about how critical that “necessary” face time really is — can technical staff get enough idea of work culture by being in the office a few days per week? consider if someone locked in her office all day is really accomplishing more in-person when compared to telecommuting? the flexibility to telecommute can improve work satisfaction and help libraries retain employees during times in their career when they are experiencing time pressures or other caregiving stresses. job sharing/part-time work additionally, job sharing or part-time work solutions should be made available to employees, at least on a temporary basis. this traditionally might be offered only for someone who has a role where they must be physically present but cannot commit to the same hours they previously had (brustman & via, 1988; laynor, 1988; stennett, 1993). why not for technology positions as well? since technical staff are in such high demand, administrators may find that it will help them recruit high quality workers by offering non-traditional hours. that said, while academia is moving toward more part-time workers in the form of adjuncts (national center for education statistics, 2014), those positions usually do not come with benefits or long-term stability. library administrators must be mindful of offering flexibility without exploiting workers. scheduling flexibility/flextime for many caregivers, those responsibilities place external time demands on us that cannot be changed. daycares, pet boarders, and schools have set hours of operation. medical appointments are almost always scheduled for the middle of the work day. flexible and thoughtful schedules alleviate the stress of having to be in two places at the same time. create a culture of support by setting core hours to reflect time for parents to drop off and pick up children from school, and for all workers to be able to leave at a reasonable time. discourage meetings from running late, and do not schedule them late in the day without plenty of notice. allow departments with public service expectations to shift schedules so that some staff members come early and others stay late as works for the individual. our library used google analytics to track usage of online services and identified that our lightest traffic is on friday, which does not correspond to our in person counts. with data to back us up, we’ve begun to schedule server and software updates for fridays – which also works best for the programmer, as both of us have small children and find it difficult to do them on weekends. — jaclyn, mother of 2 change the picture before having children, the authors used to easily allow work to bleed into personal time. we are now more deliberate about respecting these boundaries. work-life balance is something that we were immediately forced to juggle when we returned to work as new parents. we looked for technologies and strategies to try to handle just as much as we did in our pre-children working lives, but the reality is that perhaps, it is just not physically or mentally possible, even desirable. the ideal white collar worker works constantly and puts in “face time”, which is a norm celebrated and set as an example of good work ethic (davis, 2014). nowhere is this mythical ideal worker image more prevalent than in the technology sector. endless hours of coding and working all night on projects is a huge part of the image of a successful technology worker. the culture of “always on” may in fact not lead to greater or more useful productivity. hours worked does not correspond to wisdom gained or progress made (norton, 2014). upholding this cultural lifestyle hurts our profession. the culture of endless hours of code and working at all hours hinders current technologists from career advancement, and forces the caregivers among us to have to choose between career or family. further, both research and anecdotal evidence have shown that young women will opt out of or not pursue technology careers because they do not identify with this lifestyle (margolis, fisher, & miller, 2000). we as a profession need to change this pattern. the authors know that many of these issues are larger than any one of us can solve—these solutions will require a change in social norms over time. beyond social justice considerations, there are real effects on institutions. one of the authors left a tenure-track position because it was having too negative an effect on her family. we all know people who have had to leave libraries because they could not afford daycare or could not find an amenable schedule (cohany & sok, 2007). a common theme in library technology circles is changing the image. bourg (2015) suggests changing the image of technologists because stereotypes are self-reinforcing. library technologists — who have both borrowed and shielded themselves from the cultures of libraries and silicon valley — have a chance to forge a new culture. instead of “always connected”, “all-night coding sprints”, and “brogrammers” can we instead become a haven of family-friendly, flexible, and well-rounded but still smart and committed professionals? let’s take our liminal positions and use them to create a new vision for the kind of profession that we would like to be. references allen, d. (2001). getting things done: the art of stress-free productivity. new york: penguin. bourg, c. (2015, january 28). never neutral: libraries, technology, and inclusion. retrieved from https://chrisbourg.wordpress.com/2015/01/28/never-neutral-libraries-technology-and-inclusion brustman, m. j., & via, b. j. (1988). employment and status of part-time librarians in u.s. academic libraries. journal of academic librarianship, 14(2), 87–91. budig, m. j., & hodges, m. j. (2010). differences in disadvantage: variation in the motherhood penalty across white women’s earnings distribution. american sociological review, 75(5), 705–728. carrns, a. (2013, february 29). parental leave policies at some big technology firms. retrieved from http://bucks.blogs.nytimes.com/2013/02/25/parental-leave-policies-at-some-big-technology-firms/ cohany, s. r., & sok, e. (2007). trends in labor force participation of married mothers of infants. monthly labor review, 130(2), 11–16. davies, a. (2014). the origins of the ideal worker: the separation of work and home in the united states from the market revolution to 1950. work and occupations, 41(1), 18 – 39. duncan, j. (2008). working from afar: a new trend for librarianship. college & research libraries news, 69(4), 216–219. florentine, s. (2014, october 29). lack of parental leave drives employee turnover. retrieved from http://www.cio.com/article/2840574/staff-management/lack-of-parental-leave-drives-employee-turnover.html graves, s. j., xiong, j. a., & park, j. h. (2008). parenthood, professorship, and librarianship: are they mutually exclusive? the journal of academic librarianship, 34(3), 202–210. http://doi.org/10.1016/j.acalib.2008.03.003 kantor, j. (2014, august 13). working anything but 9 to 5. the new york times. retrieved from http://www.nytimes.com/interactive/2014/08/13/us/starbucks-workers-scheduling-hours.html laynor, b. (1988). librarianship and motherhood: a part-time solution. medical reference services quarterly, 6(4), 15–25. luce, r. e., & hartman, s. (1984). telecommuting to work: using technology to work at home. library hi tech, 2(4), 79–83. http://doi.org/10.1108/eb047576 margolis, j., fisher, a., & miller, f. (2000). caring about connections: gender and computing. technology and society magazine, ieee, 18(4), 13–20. mason, m. a., wolfinger, n. h., & goulden, m. (2013). do babies matter?: gender and family in the ivory tower (1st ed.). piscataway: rutgers university press. mcgregor, j. (2015, march 6). an unusual new policy for working mothers. the washington post. retrieved from http://www.washingtonpost.com/blogs/on-leadership/wp/2015/03/06/an-unusual-new-policy-for-working-mothers/ national center for education statistics. (2014). the condition of education. retrieved from http://nces.ed.gov/programs/coe/indicator_cuf.asp newport, c. (2009, june 9). drastically reduce stress with a work shutdown ritual. retrieved from http://calnewport.com/blog/2009/06/08/drastically-reduce-stress-with-a-work-shutdown-ritual/ norton, q. (2014, november 7). against productivity: this essay took four years to write. retrieved from https://medium.com/message/against-productivity-b19f56b67da6 o’reilly, a., & hallstein, d. l. o. (2012). academic motherhood in a post-second wave context: challenges, strategies, and possibilities. bradford, on: demeter press. philipsen, m. i., & bostic, t. (2008). challenges of the faculty career for women. san francisco, ca: john wiley & sons, inc. philipsen, m. i., & bostic, t. b. (2010). helping faculty find work-life balance. san francisco, ca: john wiley & sons, inc. sallee, m. (2014). faculty fathers: toward a new ideal in the research university. albany: state university of new york press. sotirin, p. (2008). academic momhood: in for the long haul. women’s studies in communications, 31(2), 258–267. http://doi.org/10.1080/07491409.2008.10162541 spires, t. (2007). the busy librarian: prioritizing tenure and dealing with stress for academic library professionals. illinois libraries, 86(4), 101–109. stennett, r. (1993). job sharing in librarianship. library management, 14(6), 13. templer, a. (2010). towards achieving work-life balance: the librarian context (pp. 1–17). retrieved from http://people.umass.edu/misra/joya_misra/work-life.html ward, k., & wolf-wendel, l. (2012). academic motherhood: how faculty manage work and family. new brunswick, nj: rutgers university press. wojcicki, s. (2015, january 17). paid maternity leave is good for business. wall street journal, p. a17. wolf-wendel, l. e., & ward, k. (2006). academic life and motherhood: variations by institutional type. higher education, 52(3), 487–521. world policy forum. is paid leave available to parents of infants? (n.d.). world policy forum. retrieved from http://worldpolicyforum.org/policies/is-paid-leave-available-to-parents-of-infants wyatt-nichol, h., cardona, m. m., & drake, k. s. (2012). balancing work and family in higher education. in a. o’reilly & d. l. o. hallstein (eds.), academic motherhood in a post-second wave context: challenges, strategies and possibilities (pp. 108 – 126). bradford, on: demeter press. zemon, m., & bahr, a. h. (2005). career and/or children: do female academic librarians pay a price for motherhood? college & research libraries, 66(5), 394–406. http://doi.org/10.5860/crl.66.5.394 about the authors jaclyn bedoya has lived and worked on three continents, although currently she’s chasing children instead of working. it turns out that growing up in southern california spoils you, and she’s happiest where there are 300 days of sunshine a year. also disneyland. reach her @spamgirl on twitter or jaclynbedoya@gmail.com. margaret heller is digital services librarian at loyola university chicago. she is active in lita and has previously served on a code4lib conference planning committee. find her on twitter @margaret_heller or at http://www.gloriousgeneralist.com. christina salazar is systems librarian at california state university channel islands, which really is not on an island and has fifteen years experience working as a systems librarian in both academic and not for profit organizations. she has been active in the los angeles chapter of american society for information science and technology (lacasist) and the newly active southern california code4lib. she can be reached via twitter @inphomaniac or e-mail at christina.salazar@csuci.edu. may yan is er discovery and access librarian at ryerson university. she recently finished her term as the president of the ontario library and information technology association. she is currently serving as treasurer in her third year as an executive of council. @mayyan is on twitter, but you can also email her at may.yan@ryerson.ca. she is currently learning to be academic after many formative years being corporate. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – editorial: for pandemic times such as this mission editorial committee process and structure code4lib issue 49, 2020-08-10 editorial: for pandemic times such as this a pandemic changes the world and changes libraries. by peter murray this issue is late to publish, and the code4lib journal editors have decided to skip a quarterly production cycle. for future reference, the world was hit by a new (“novel”) virus in late 2019 that in a very short period of time has upended the lives of many on the planet: schools and business abruptly closed, travel restrictions and the phrase “social distancing” entering the lexicon, and scientific and political uncertainty. in case that wasn’t enough, we added: social uprising, clearer demonstrations of the impacts of climate change, and for about 36 hours in north america—“murder hornets.” [1] as a note to our future selves: we were distracted. for me personally, two points early in the 2020 timeline will be memorable. the first is january 23 at the wolfcon reception at texas a&m university. dr. david carlson, dean of the tamu libraries, interrupted the reception to say that there was a possible case on campus of a student with this new virus. further test results turned out negative, but this thing had quite unexpectedly gotten real. the second point was eight weeks later at the code4lib conference in pittsburgh. the conference space started new cleaning protocols and we were encouraged to wash our hands more often. several institutions put travel restrictions on their employees, and the number of remote presenters and viewers jumped dramatically. at the end of the week, there was also more news about the spread of the virus and meetings that were being canceled. i remember remarking to someone, “this might be the last library technology gathering for a while.” the world has evolved since the journal issued its call for proposals for this issue in late march. it will continue to evolve in ways that we cannot predict when we expect to publish the next issue in february 2021 (our 50th!). how will we use technology to facilitate safe and productive interactions with an airborne virus? as technology becomes a more important facilitator of communication, how do we ensure broad accessibility across dimensions like bandwidth and technical literacy? the last task that delayed the publication of this issue was writing this editorial, and for that i apologize. so many words have already been written and spoken about this time without precedent, and i am overwhelmed by change and uncertainty. i can’t predict what will happen next, but i can offer some general advice. take care of yourself and each other. the good days and bad days probably feel more extreme now, and probably tend towards the bad more than the good. we are social creatures, and as a society we will get through this. have awareness in what you do and tolerance of what you accept of others. [2] there is little slack in society right now; many people and systems were already stretched to the breaking point before the pandemic started. if you can add slack back into the system, do so. remember the past, improve the present. time moves in one direction. we can’t go backward to pre-pandemic times, but we can learn from the past to make our present and future better. what are the tools, techniques, and technologies can we build upon to make our libraries better equipped to serve our patron’s needs. postscript: in the spirit of experimentation that is the hallmark of the code4lib community, this editorial uses robustify your links from the international internet preservation consortium to point to snapshots of hrefs in the article. link attributes were constructed manually using the robustify website. the javascript and css additions are not on this page the css formatting wasn’t quite right, but the experiment continues. adding something like robustify seems like a great addition to any journal publication toolchain. summary of issue 49 nicholas weber, sebastian karcher, and james myers offer their insights into a toolchain for human-assisted curation of datasets. with a combination of open source software and open apis, they describe a system for ingesting, description, and discovery at the qualitative data repository (qdr). péter király on the code4lib editorial board shepherded this article through the publication process. in the first of two articles in this issue using geographic information tools, charlie harper and r. benjamin gorham describe a process extracting place names from articles and placing them on an interactive map. this article describes a combination of named entity recognition using python and geolocating with arcgis online. this article was edited by edward corrado on the code4lib editorial board. the second gis article is from greg sohanchyk and dan briem. they extracted and geolocated data from their integrated library system to combine with census data to understand library card usage in their city. mark swenson helped the authors with this article. nicole wood and scott shumate tackle the challenge of keeping their library’s holdings updated in oclc—a task made even more important by the need to improve the efficiency of interlibrary loan requests. they describe an innovative use of marcedit, excel, and python to make this happen. the editorial board’s mark swenson also worked on this article. one of code4lib journal’s most prolific authors, jim hahn, analyzed alma and share-vde for their ability to create linked data clusters from marc data and enrich that data into a bibframe network. péter király worked on this article. devin becker, evan williamson, and olivia wikle write about how they combine contentdm with a static site generator. the university of idaho library likes the features found in contentdm, they were not satisfied with options for displaying collections to their users. they used data extracted from contentdm and jekyll to create a “skin” over their digital collections. this is one of two articles that brighid gonzales worked on for this issue. katharine frazier improved the efficiency of monthly collection reports using data from gobi. from a list of items with a high number of holds, this process automates the task of gathering holdings, pricing, and format data for collection managers. brighid gonzales also worked on this article. ralf weber describes a process for regularly testing the access methods and availability of open access and commercial material. using codeceptjs, this process mimics the users’ actions in a web browser to test if access is available as expected. sara amato helped get this article into shape for publication in the journal. endnotes [1] for a video recap, see explaining the pandemic to my past self and explaining the pandemic to my past self part 2. a third part is no doubt in the works. [2] postel’s law for pandemic times, one might say. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – editorial mission editorial committee process and structure code4lib issue 46, 2019-11-05 editorial if you build it, i’ll probably come. it has been a great pleasure to be the coordinating editor of c4lj issue 46, and to quote eric’s editorial last issue, “library technology is a wonderful community in that we are encouraged to share our solutions with each other and extend them.” this issue has several articles that use focus on sharing tools built with freely accessible software such as google sheets add-ons and apis, python and r.    the recipes and ideas shared have inspired me to try a few ‘hacks’ of my own, and prompted me to think more about what becomes possible to us as the computing tool set becomes both more powerful and more accessible to us ‘mere mortals’.   combine that with the increasing conglomeration of bibliographic data in shared cloud ils systems, and one starts to wonder what the possibility for ‘libraries collections data at scale’ might look like. worldcat provides some insight into some slices of some of the worlds’ collections, and the increasing popularity of alma has brought many academic library bibliographic records into a shared cloud environment.     how long will it be before the tools exist that allow those of us at the local level to explore these ‘big data’ sets in new tools customized to our local needs? over the past year i’ve been watching the growth of shared print programs such as the rosemont alliance and the emerging partnership for shared book collections, and seen the challenges that a lack of clean data and open tools present to these organizations.      last year c4lj ran an article on machine learning which explored some of the possibilities of applying tools such as tensorflow or keras ( https://journal.code4lib.org/articles/13671) to library data, and this issue features an article on natural language processing to create clean metadata. i’m curious to know who else has tried these types of tools in real world projects.  if so, please consider submitting your work to c4lj. here’s looking forward to learning more from you all. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – improving the presentation of library data using frbr and linked data mission editorial committee process and structure code4lib issue 16, 2012-02-03 improving the presentation of library data using frbr and linked data when a library end-user searches the online catalogue for works by a particular author, he will typically get a long list that contains different translations and editions of all the books by that author, sorted by title or date of issue. as an attempt to make some order in this chaos, the pode project has applied a method of automated frbrizing based on the information contained in marc records. the project has also experimented with rdf representation to demonstrate how an author’s complete production can be presented as a short and lucid list of unique works, which can easily be browsed by their different expressions and manifestations. furthermore, by linking instances in the dataset to matching or corresponding instances in external sets, the presentation has been enriched with additional information about authors and works. by anne-lena westrum, asgeir rekkavik and kim tallerås introduction after years of delay, it seems like oslo is finally getting its new public library. at least there are concrete plans for a new, large and innovative building. the plans, of course, primarily concern the physical space: how can the building contribute to modern information services, and which features of a modern public library should it enable and encourage? in connection to such questions, many interesting discussions are taking place. one of them deals with the traditional axis point of the library: the document collection. while there is a rapid development going on in how we think about library buildings, their means and objectives, we are also witnessing a parallel digital revolution, pushing forward new thoughts and solutions on collection development and distribution. with the expansion of the web, online catalogues have a new context that involves both opportunities and challenges (coyle, 2010). the opportunities are related to the effective infrastructure for sharing and dissemination. among other things, the challenges relate to the existing library standards for document description – metadata. these standards were made in a different technological era. the pode project the plans for a new library building and the discussions about library services affected by these plans have given the oslo public library an indirect opportunity to examine how their metadata can be used in new contexts and in ways that contribute to better services. the independent pode project[1], funded by abm-utvikling[2], but located at the oslo public library, has done exactly that. the project has, during the last few years, been experimenting with descriptive metadata related to mash ups, reference models such as frbr (ifla study group on the functional requirements for bibliographic records, 1998), new generations of opacs and linked data. this work has led to at least one central insight: one cannot create better services, based on already existing metadata, than what the quality of the metadata will support. in this article we describe a subproject of pode dealing with (nor)marc[3] records describing manifestations related to the norwegian authors knut hamsun and per petterson. finding the way through library hit lists knut hamsun (1859-1952) is norway’s most prominent novelist and one of three norwegian nobel laureates in literature. his literary production includes about 30 novels, a few plays and collections of short stories, one collection of poetry and some non-fiction and biographical writings. altogether hamsun’s production counts a total of 40 works; it is a bibliography that a library user should be able to browse easily. however, the image that meets the library user is quite different. in the online catalogue at oslo public library, a qualified search for “hamsun, knut” as author will produce a list of 585 hits (as of november 11th, 2011). this is of course way too many hits to provide for an author who wrote 40 books. notice that this is the result of an advanced qualified search. a more typical simple search, which is what most library users would try, provides an even longer list. the problem is that the online catalogue doesn’t distinguish between an author’s different works and different versions of one work. in our list of 585 hits, as many as 63 correspond to different representations of one novel:  hunger. these would be different editions, different formats and translations into different languages. the users must of course be able to choose whether they want the book, the audio book or the movie, and they must be able to choose what language they want to read the book in, but to most users it is more disturbing than useful when the opac makes them choose between more than 20 different editions of hunger in the norwegian language[4]. library standards the library catalogue has traditionally focused on describing physical objects. each manifestation of a book is represented by a separate record, and there are no functional connections between records that describe manifestations of the same work. a library user, who searches the online catalogue for a particular title might therefore sign up on a waiting list to borrow one particular edition of a classic novel without realizing that numerous other editions of the same book are already available. another user might end up not getting the book at all if he accidentally picked an edition that no longer has available copies. the pode project has based its experiments on a hypothesis that library users are typically interested in finding a particular title, not a particular edition of that title, and that this interest is especially typical for fictional works, where different editions usually have identical content. from the perspective of the library system, the user should ideally be able to make a reservation for a title without having to choose between different editions. of course, those who do care about editions should still have the opportunity to specify this, but why should everyone else be forced to pick? as many have pointed out, the present library standards were developed prior to the web and the present infrastructure for production, distribution and utilization of metadata[5]. in addition, the standards that introduced library data to the electronic sphere were developed years before the invention of entity relationship (er) models and relational databases (thomale, 2010). this presents some challenges with implementing reference models like frbr (that separates editions from works), which is based on er-analysis and relationships not implemented in or between marc records. the marc format embodies technical inscriptions and logic from the card catalogue, which it was developed to automate. metadata in card catalogues were read and interpreted by humans, a feature that is continued in the marc format, which dictates the making of (separate) records, largely consisting of human-readable text strings. this is of course a simplified description of library metadata practices. the process of making a marc record is characterized by a complex interaction with cataloguing rules like aacr2 and isbd, and most of the motives behind the text strings are to be found in such rules. in a web context, we want machines to process the data and interpret them for us. text strings must be absolutely consistent in order for machines to accurately interpret the data and create a useful presentation of that data. in relational database and linked data environments the best practice doctrine is to avoid disambiguation by providing unique identifiers – respectively using primary/foreign keys and uris (berners-lee, 2006; codd, 1970). marc records are lacking such explicit identifiers that would have helped our indexing tools and search engines to separate two authors with the same name, or maybe to merge these authors if they are likely to represent one person on the basis of concurrent relationships to the same uniquely identified works. these kinds of challenges will of course continue when we want our machines to identify relations between documents, authors and different frbr entities. which books represent manifestations of a particular expression or a work? how do we identify works like short stories, when they are only described textually in a marc note field? for most books published after 1970 we have isbn numbers that identify manifestations, and we have names and titles that we can get computers to reason over, but without consistent catalogues and machine-readable identifiers, this is still a tough job. it takes at least a proper cleanup. frbrization of marc records the pode project used a tool developed by the norwegian university of science and technology for automated frbrization of marc records (aalberg, 2006). the tool uses xsl transformations on catalogue exports in the marcxchange format to sort the data within bibliographic records based on which frbr entities the data applies to. library catalogue records are mainly descriptions of manifestations; the individual record describes one particular edition of a published work, with manifestation specific information such as time and place of publishing, physical description, isbn, etc. but the record will also contain information that applies to the expression and work this manifestation is related to. for example, the data that contain the name of the author and the original title of a book are pieces of information that describe the work entity, while information about the document’s language and format say something about the expression. the frbrization tool will identify which fields apply to which frbr entities, and use this to divide each record into a work part, an expression part and a manifestation part, with frbr relations between them. if the tool outputs identical work descriptions for two different marc records, the records are assumed to describe two different embodiments of the same work. the tool will also produce group 2 and 3 entities, such as agents and subjects. initially the project ran the selected corpus of records (908 records describing manifestations of hamsun and petterson) uncleansed through the automated frbrization system. the first attempt gave results that were far from perfect. this was mainly due to missing information in the marc records and inconsistent cataloguing practice. to progress further in the experiment, the project had to clean up a considerable number of records in order to get data that were sufficiently expressive and consistent. the clean up process in brief, the clean up process mainly consisted of identifying and adding original and uniform titles to records where this was missing, adding information about individual works and short stories collected in one volume, and setting indicators to distinguish between significant work titles and non-significant titles. in addition some time was spent correcting typos and errors. altogether approximately 60 hours was spent cleaning up the productions of knut hamsun and per petterson, including the time used to determine the rules for correction. one of the main jobs in the clean up process was to identify and improve records for translated works that either lacked the norwegian original title completely, or only provided the original title in human-readable notes. another job dealt with the adding of uniform titles in cases where the titles of non-translated titles differed from the original title. due to norwegian spelling reforms, several of hamsun’s works have been released with different titles at different times. for example, the short story paa tourné (original title) has also been released with the spellings på tourné and på turné. another task was setting indicators to separate significant real work titles from non-significant titles. non-significant titles are typically found in collections of an author’s work, where the content has been put together and the publication has been entitled by someone else than the author himself. publications such as these should not be listed as works in a frbrized bibliography, although the individual novels or short stories contained in the collections should[6]. examples: a) 245 10 ‡agrowth of the soil ‡cknut hamsun ; translated from the norwegian by w. worster (english translation of an original hamsun work. first indicator set to one means this is a significant work title.) b) 245 00 ‡atales of love and loss ‡cknut hamsun ; translated by robert ferguson (english collection of short stories that were not originally published together. first indicator set to zero means this is a non-significant title.) see the appendix for more information about the cleanup process. outcomes while the first attempt at frbrization identified 149 works by hamsun, the list was further reduced to a number of 84 after cleaning up the marc records. in the case of per petterson, we saw an increase in the number of works from 14 to 41, due to the adding of titles of individual short stories and essays to some catalogue records. the resulting lists of works corresponded almost exactly with the actual bibliographies of the two authors. the only exception being one work by hamsun which was listed by the norwegian national library’s hamsun bibliography, but missing in oslo public library’s collection. the clean up process thereby unintentionally provided us with a method for identifying missing works in the library collection. this is otherwise a tedious procedure when you are dealing with long lists of hundreds of manifestations. rdfization the frbrized datasets were converted to rdf, using xslt[7] and a crosswalk between marc fields and rdf predicates that was developed by the project. the crosswalk mainly used well-known vocabularies and ontologies like dublin core metadata terms, bibliographic ontology, core frbr, foaf and skos. but the project also constructed several more specific sub-properties to express our data more exactly than these vocabularies allow for. later the project discovered that the rda vocabularies[8] contain predicates that cover a lot of the more library specific information that needed to be expressed. in later revisions of the crosswalk some of these have replaced our own predicates. see the example in the table below, or the complete crosswalk at http://www.bibpode.no/blogg/?p=1573. marc rdf field subfields subject type predicate object 001 bibo:document pode:titlenumber xsd:integer 008/35-37 bibo:document dct:language lvont:language 019 a bibo:document dct:audience dct:agentclass 019 b bibo:document dct:format pode:physicalformat 019 d bibo:document pode:literaryformat pode:literaryformat 020 a bibo:document bibo:isbn rdfs:literal once the data were converted to rdf, they were enriched with links to other datasets. works were linked to instances in dbpedia and project gutenberg, while persons were linked to dbpedia and viaf. in order to be able to sort a list of works chronologically, the project added information about date for first edition to the work instances. this information is not easily extracted from a marc record, if at all contained. with this new and enriched dataset, the project was able to develop a web application that allowed an end-user to browse through the library’s complete collection of these authors’ books by choosing from a short list of works instead of searching through a flat list with hundreds of manifestations. furthermore the application could give the end-user relevant information about authors from dbpedia, as well as links to digital full text versions in project gutenberg. a simple web application was developed that allowed end-users to browse this part of the library collection, clustered as frbr entities, with additional information provided from external sources made available through the linking of data[9]. summary the increasing literature on metadata quality deals with criteria and principles that aim to maximize the utilization of metadata. consistency (and coherence) are among such criteria (bruce & hillmann, 2004). flexible characteristics, such as extensibility, modularity, the ability of refinements and multilingualism, are others (duval, hodgins, sutton, & weibel, 2002). some also emphasize machine processability as an independent principle (nilson, 2010). this literature has moreover in common the desire to create interoperability and metadata quality across communities (e.g. chan & zeng, 2006; haslhofer & klas, 2010; hillmann & phipps, 2007). this implies a generalization of the quality concept and discussions on public benefit beyond the circumstances of the certain metadata environments and standards. in light of the development of the web and modern principles of metadata quality, it can be tempting to immediately convert library data to a more linked data-friendly format like rdf. on the basis of what we have – the marc records – it’s not yet that simple. nilsson (2010) presents two models of metadata harmonization. vertical harmonization increases interoperability within a given set of standards. horizontal harmonization contributes to interoperability between various standards not given in advance. the pode projects tells us that a conversion of existing traditions to new standards can be problematic in relation to both models of harmonization. they also tell us that a “vertical clean-up” helps a lot in order to make it easier to achieve horizontal effects. the experiments have been performed on a limited set of data. it is hard to estimate how comprehensive the challenges would be when conducting conversions of larger data sets at the oslo public library. the experiments conducted have shown, however, that a conversion in accordance with a reference model such as frbr and principles for linked data provides some immediate benefits in the form of cleaner and more “browseable” results lists, and in opportunities to utilize external quality data – the metadata quality increases. these are experiences that may be important in new projects, particularly in light of the plans for a new, forward looking and modern library. about the authors anne-lena westrum has a degree in multimedia design and is working as a project manager at oslo public library. she is currently studying interaction design and user behavior at gjøvik university college, while managing two new user centered projects at the library. asgeir rekkavik is a librarian at oslo public library. he has been involved in several rdf and linked data related projects. kim tallerås is a doctoral student at the department of archivistics, library and information science at oslo and akershus university college of applied sciences. he is working with issues related to metadata and semantic interoperability. acknowledgements the pode project would not have been possible without the support of the norwegian archive, library and museum authority and the contribution from oslo university college in cooperation with the norwegian library laboratory. the article summarizes a 3 year long project with work efforts from an interdisciplinary team at oslo public library. the team wishes to extend a special thanks to associate professor trond aalberg at norwegian university of science and technology and magnus enger at libriotech for important contributions making the end result possible. notes [1] for more information about the pode project, visit http://bibpode.no/?q=node/9 [2] abm-utvikling is the former norwegian authority for libraries, archives and museums [3] normarc is the norwegian dialect of marc. web edition: http://www.nb.no/fag/kompetansesenter/kunnskapsorganisering/dnk/normarc [4] pisanski & žumer (2010) indicate that users prefer a more overall orientation into a bibliographic universe. [5] see  http://www.loc.gov/marc/transition/news/framework-103111.html for an interesting and important announcement from library of congress advocating a similar message addressing the marc standard especially. [6] although relationships between marc fields are implicitly expressed through a record, it is not easy to unambiguously and explicitly express relations between fields in cases where a document consists of several works. [7] in later work of this type, we have used ruby scripts with a yaml mapping file instead of xslt. for details, see: https://github.com/bensinober [8] http://rdvocab.info/ [9] http://bibpode.no/linkedauthors/ references aalberg, t. (2006). a tool for converting from marc to frbr. ercim news, (66). retrieved from http://www.ercim.eu/publication/ercim_news/enw66/aalberg.html berners-lee, t. (2006). linked data. retrieved from http://www.w3.org/designissues/linkeddata.html bruce, t. r., & hillmann, d. i. (2004). the continuum of metadata quality: defining, expressing, exploiting. metadata in practice (s. 203-222). chicago: ala editions. chan, l. m., & zeng, m. l. (2006). metadata interoperability and standardization?: a study of methodology: part i. d-lib magazine, 12(6). doi:10.1045/june2006-chan codd, e. f. (1970). a relational model of data for large shared data banks. communications of the acm, 13(6), 377-387. doi:10.1145/362384.362685 coyle, k. (2010). library data in a modern context. library technology reports, 46(1), 5-13. duval, e., hodgins, w., sutton, s., & weibel, s. l. (2002). metadata principles and practicalities. d-lib magazine, 8(4). retrieved from http://www.dlib.org/dlib/april02/weibel/04weibel.html haslhofer, b., & klas, w. (2010). a survey of techniques for achieving metadata interoperability. acm computing surveys (csur), 42(2), 7. doi:10.1145/1667062.1667064 hillmann, d. i., & phipps, j. (2007). application profiles: exposing and enforcing metadata quality. proceedings of the 2007 international conference on dublin core and metadata applications: application profiles: theory and practice (p. 53–62). retrieved from http://portal.acm.org/citation.cfm?id=1344591.1344601 ifla study group on the functional requirements for bibliographic records. (1998). functional requirements for bibliographic records: final report. münchen: k.g. saur. retrieved from http://archive.ifla.org/vii/s13/frbr/frbr.pdf nilson, m. (2010). from interoperability to harmonization in metadata standardization: designing an evolvable framework for metadata harmonization. stockholm: kungliga tekniska högskolan. pisanski, j., & žumer, m. (2010). mental models of the bibliographic universe. part 1: mental models of descriptions. journal of documentation, 66(5), 643-667. doi:10.1108/00220411011066772 thomale, j. (2010). interpreting marc: where’s the bibliographic data? code4lib journal, 2010(11). retrieved from http://journal.code4lib.org/articles/3832 appendix – clean up and corrections the main job was to ensure that all the translated records (both due to foreign languages and norwegian language reforms) contained the original work title in the 240 field. where the 240 was missing, it was added automatically based on information in note field 574, a normarc-specific field containing information about original title. where this note field was missing or the automatically conversion failed, the title was applied manually. another time-consuming part of the job was to identify and determine actual (original) titles of significant works in the 245 and 740/700‡t fields and set the appropriate indicator according to the corrections rules. dealing with the second indicator in the 740 field, the project chose to use the value 2 only when the field contains a significant work title. for all other titles the indicator is set to 0, even if they are analytic entries. even if this practice does not conform the convention of using 700‡a + ‡t for analytical title entries, this was decided upon to be the most efficient approach to detect analytical work published in an unambiguous way. based on the results from the first attempt of frbrization, corrections in the 908 records for hamsun and petterson included: correcting the language code in 008 in 5 records added uniform title (or “original title” in precise accordance with normarc terminology) in 240 fields in 85 records and correcting typos of existing 240 fields in 24 records correcting typos in 245‡a (or wrong isbd syntax) in 6 records correcting the first indicator in 245: before the correction 137 records had indicator 1 = 0 or blank, while indicator 1 = 1 was used in 774 records. after correction the distribution was 263 – 651 correcting the 700 fields. in the original records, we found 948 700‡a and 545 700‡t fields. in the corrected records the numbers are reduced to respectively 917 of 700‡a and 481 of 700‡t. the change is due to a more systematic use of 740 fields in all records that have the same author (which is registered in 100). changing the second indicator in the 700 field in order to clarify whether an entry is a unique work or not. subscribe to comments: for this article | for all articles one response to "improving the presentation of library data using frbr and linked data" please leave a response below: james weinheimer, 2012-05-16 this is a very interesting article for anyone interested in how the public works with library catalogs. one additional aspect that would be especially interesting would be a comparison among patrons for the result sets as foreseen by frbr, and those available through modern indexing practices. these can be seen in worldcat now. to take your example of hamsun’s sult the uniform title is: hamsun, knut, 1859-1952. sult, and searching this in worldcat correctly http://www.worldcat.org/search?q=ti%3a%22sult%22+au%3a%22hamsun%2c+knut%22 gives 156 results that the patron can modify by format, other authors, dates, languages, and so on. this list can also be sorted by author, title, date, etc. other indexes could be created too. to get this kind of result, of course needs the catalogers to add consistently and correctly the uniform titles. but that is part of their job. leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – course views: a scalable approach to providing course-based access to library resources mission editorial committee process and structure code4lib issue 6, 2009-03-30 course views: a scalable approach to providing course-based access to library resources the ncsu libraries’ course views project, along with a locally developed widget web service, improves course-based access to library collections and services by dynamically generating library course pages for all 6000+ courses at ncsu. by automatically generating custom content when possible and showcasing authored content when available, course views is able to achieve full course coverage without significantly increasing staff time to create and manage content. this paper will describe the system and the use of web services to achieve scalable and sustainable delivery of course-related library content. by jason casden, kim duckett, tito sierra and joseph ryan introduction in august 2008, the ncsu libraries launched course views [1], a new online service that provides a custom library web page for every course at north carolina state university. this service, promoted to students under the “library tools” brand [2], has helped achieve two major objectives: to provide course-centric access to library resources on an unprecedented scale, and to integrate library resource access points into campus learning management systems (lmss) at an improved scale and level of customization. course views has been in development for several years. a project team convened in 2005 to begin working on the problem of integrating library resources with lmss. although originally intended to focus on lms work, the team realized that a more profitable approach was to take a broader look at making library resources more accessible and relevant to students. the team decided to bring together the most student-centric “stuff” the library has to offer, combining traditional research-oriented content like collection search tools with more task-oriented information about library services such as citation guides, group study rooms, and device lending. goals a course view for every course the union of collectionand service-oriented resources created an interesting opportunity for the team to create a one-stop resource tailored to the needs of students. additionally, research showed that in comparison to subject guides, “library resources organized or delivered at a course level are more in line with how undergraduate students approach library research” [3]. this research led to the development of the primary goal of the course views project: to develop a system that would provide useful, targeted resources for every course at ncsu. the libraries’ earlier efforts in this area had been far more modest, focusing on the creation of hand-authored course guides. these web pages provided high quality information, closely tailored to each course. unfortunately, the libraries never had the resources to create and maintain custom course guides for more than 3% of the courses offered on campus. it was clear that achieving coverage for the remaining 97% of courses would require a new approach. to achieve this goal, we needed to find a way to customize content to the course that could scale across 6,000 courses in over 150 departments on campus. customized to the course of course, a system that covers all courses must still provide relevant content for each course. the previous hand-authored course guide approach at ncsu provided unlimited content customization options. in practice, however, not all authored course guides produced by subject specialist librarians were completely unique. in the early stages of our project we performed a content analysis of existing course guides. this analysis revealed patterns of library content that appeared across course guides, especially for guides authored by the same librarian. for example, links to the library catalog, citation tools, general purpose article databases, and bibliographic instruction content were frequently included on course guides across disciplines. interspersed with this generic content were links to resources that were often discipline-specific, such as links to chemistry databases for chemistry courses. and in some cases, librarians included content that was only relevant to the specific course, such as resources to support class assignments. in planning for a new system, we sought to maintain at least the same level of content customization options that existed previously, while using a centralized data-driven system to manage the generic and discipline-specific content. doing so would free up staff time to focus on producing content that was truly course specific. system design the team considered two options for this project: customizing an existing open source tool, or developing a homegrown solution, similar to efforts at other universities [4]. at the time of the investigation, we were unable to find any suitable vendor solutions to this problem. initially, the team considered adapting an existing course page authoring tool, such as the courselib module of minnesota’s libdata project [5], or rochester’s libcb project [6]. while these tools facilitate the creation of consistent and attractive course pages, we believed the authored course page approach would inhibit our ability to provide customized library content for all courses at our university. after some deliberation, the team decided to develop a homegrown solution. the team realized that the best means of enabling integration with lmss was to organize the system’s content around a shared campus data structure. course views uses the standard course identifiers (e.g., eng 101) managed by the campus registration and records system. an unanticipated benefit of structuring the system in this way is that it opened the door to automated methods of customizing content within the application, because these course identifiers reveal attributes of the course, such as discipline focus, college affiliation, or graduate vs. undergraduate status. balancing the requirement of a view for every course with the goal of customizing content to the greatest extent possible required creative system design. to support course views content delivery, the team decided to develop a “widget” web service that would provide an infrastructure for producing customized content in a scalable way. additionally, this widget system allows these pieces of content to be easily reused, which enables external integration of widget content. figure 1: an example course views page for eac 580 an individual course view (figure 1) is essentially a collection of loosely coupled content modules. the widget system delivers all of these modules, each representing some aspect of library services or content. as a result, a course view is a course-centric portal to library resources that tailors content based on course identifiers rather than user accounts (figure 2). figure 2: high-level architecture of the course views system technical implementation the course views architecture consists of two components: the course views handler and an underlying widget system, both written in object-oriented php. the system’s uris use the rest architectural style [7]. the uri pattern consists of a curriculum code, a course number, and, optionally, a course section. for example, the course views request uri for the course “adult and higher education (eac) 580” is http://www.lib.ncsu.edu/course/eac/580. the process of producing an individual course view is as follows (steps refer to labels in figure 3): the course views handler receives the request (via apache mod_rewrite) from the user. the course views handler validates the course identifier through an http widget request (steps a through c). the course views processing code individually requests all appropriate widgets, passing the course identifier as parameters. upon receiving a widget request, the widget system handler parses the parameters. the widget system instantiates the appropriate widget, passing any additional parameters. the requested widget collects and formats data from a variety of sources, returning content wrapped in a consistent bit of xhtml. the course views processing code assembles the xhtml results and integrates css, jquery and yui elements. an individual course view is returned to the user. figure 3: a low-level diagram of course views request processing widget system the widget system is a restful web service designed to manage and simplify the development of independent php classes which deliver easily reusable content drawn from diverse sources. the system consists of a dynamic handler, one or more abstract classes for building widgets, and a structured directory, file, and class naming pattern. creating course views that were both maintainable and appropriately specific relied heavily on the encapsulation of content provided by the widget system. apache’s mod_rewrite routes http requests to the widget system handler, which then checks for the existence of the requested widget and passes the uri fields and query string on to the widget for processing. it is relatively easy for a developer to create new widgets in this environment by following a few simple conventions and creating a php class that inherits from a base widget class. once this class is created, it has access to uri parameters that are blindly passed by the widget system handler and parsed by the base widget class. the following example shows a very simple widget that returns librarian recommended content for a particular course or curriculum: <?php require_once("lib/courseviews_widget.php"); class recommended_widget extends courseviewswidget { private $rec_file_root = '../recommended'; private $rec_files = array('menu.html'); /** * the required function for all widgets. returns librarian recommended content. * * @access public * @return string an xhtml formatted version of the course identifier */ public function start(){ // check for the recommended widget files and put the contents into $ret_xhtml $ret_xhtml = ""; foreach ($this->rec_files as $rf){ $crawl_ret = $this->print_file_crawl($this->rec_file_root,"$this->dept/$this->course_bare/$this->section/$rf"); $ret_xhtml .= $crawl_ret[1]; } // ...data cleanup (clipped)... // ... libxml_use_internal_errors(true); // is this well-formed? $valid_checker = simplexml_load_string($ret_xhtml); // display the appropriate title for the level of customization $udept = strtoupper($this->dept); if(count($crawl_ret[0]) >= 3){ $wid_title = "recommended for $udept $this->course_bare"; } elseif (count($crawl_ret[0]) == 2) { $wid_title = "recommended for $udept courses"; } else { $wid_title = "recommended for all courses"; } $this->set_widget_title($wid_title); // wrap the content in conventional xhtml for consistent styling if ($valid_checker){ return $this->wrap_widget_content($ret_xhtml); } else { return $this->wrap_widget_content('<p>malformed content</p>'); } } } figure 4: source code for the librarian recommended widget example output for request http://www.lib.ncsu.edu/widget/recommended/eac/580/001: <div id="recommended" class="widget"> <div class="widgetheader"> <h2 class="widgettitle">recommended for eac courses</h2> </div> <div class="widgetcontent"> <div id="librarianrecommendedcard"> <div id="contents"> <p class="contentitem"> <a href="/distance/delivery" class="itemlink">taking a distance education course?</a><br /> get books delivered to your home or office and articles sent to you on the web. </p> <p class="contentitem"> <a href="/recommended/eac/databases.html" class="itemlink">recommended databases</a><br /> a more expansive set of database recommendations for adult and higher education, human resources, and training. </p> <p class="contentitem"> <a href="/guides/statistics" class="itemlink">finding education statistics</a><br /> a set of websites for finding education statistics. </p> <p class="contentitem"> <a href="/guides/apaplag/" class="itemlink">apa citation style and plagiarism guide</a><br /> tips for using apa style, citing your sources, and avoiding plagiarism. </p> </div> </div> </div> </div> figure 5: xhtml output for the librarian recommended widget the widget source code in figure 4 demonstrates how the object-oriented nature of the widget system makes creating groups of similar widgets, such as course views widgets, much easier. the inheritance hierarchy for this and some of the other widgets is shown in figure 6. all widgets extend the widget abstract class, which parses the fields passed to the widget and provides some helper functions, such as the one that wraps outgoing content in a consistent xhtml stub format (figure 5). for course views widgets, an additional abstract class (courseviewswidget) extends the base widget class to provide some additional functions common to most of the course views widgets, like parsing course curriculum codes and numbers. by taking advantage of inheritance when using the widget system for bigger projects, the developer can write widgets for the project that contain very little redundant code. figure 6: widget system class hierarchy customized content an essential concept in course views is the ability to produce content at different levels of specificity in order to balance scalability with the customization of library content to specific course needs (figure 7). the team noted that certain library resources, tools, and services are applicable across many different courses (e.g. citation tools, catalog search), while other content must be customized for the course or course section (e.g. course reserves, certain librarian recommended content). there are also certain kinds of content that are more discipline-specific and can, therefore, be described at the curriculum level (e.g. article databases, certain librarian recommended content). the end result is that each course views page is made up of a standard set of widgets, which might vary in terms of how tailored the content is to the specified course. leveraging the mix of generic, course section, course, and curriculum level content is the key to balancing scalability while allowing for customization by librarians. independent widgets designed to make smart decisions about which content to display make this balance possible. widget name most specific possible level of customization librarian contact course section librarian recommended course section course reserves couse section course title course article databases curriculum quick article search curriculum catalog search site citation tools site google scholar site librarian chat site project tools site technology lending site figure 7: table of maximum possible level of customization for current widgets the team developed a number of widgets that utilize automatically-generated content, librarian-generated content, or a combination of both. an example of a widget that relies heavily on automatically generated content is the course reserves widget. this widget queries the ncsu libraries’ reservesdirect instance and displays a list of instructors whose sections have reserves listed for the relevant course, linked directly to the reservesdirect page for each section. less complex automatic widgets display categorized static xhtml content, including links to project tools, citation tools, technology lending, and im reference services. figure 8: display of the librarian recommended widget in course views the librarian recommended content widget is the component of a course view that most closely resembles a traditional library course page (figure 8), and is an example of a widget where librarian-generated content plays a major role. this widget enables librarians to create sections of annotated links to relevant resources. the course views system employs a unique cascading content authoring and display scheme, where a librarian could author a set of custom resources that is intended to be displayed for a specific section, a course, an entire curriculum (e.g. adult and higher education), or for all courses. the system searches for the most customized content possible, and then falls back to increasingly general levels until it locates content for the course view. this technique is the key to maximizing the benefits of limited staff time by allowing subject specialists to customize content to a course if desired or to present curriculum-level content if more specific content is not necessary. the use of default content, as well as the availability of curriculum level content for all curricula, enables course views to scale for all courses on campus. the article search widget depends both on pre-existing data and data tailored by subject librarians for the widget. the entire widget is supported by a simple database of associations between subjects from our existing browse subjects tool [8] and campus curriculum codes. subject librarians manually assign these associations, and a query returns the appropriate subjects at runtime. these subjects are displayed in the databases by subject section and also power course views’ basic metasearch tool. an algorithm selects one of the fourteen default search categories based on these curriculum-to-subject associations. a search from the quick article search box is redirected to a collection of databases from a single vendor (e.g. ebscohost, csa). subject specialists select the vendor and package of databases for each subject category. the database options are limited to vendors which support automatically generated search uris. interface design the project team was interested in a design that would identify course views as a resource created especially for students. working with an undergraduate student in the ncsu college of design, the team created a layout and visual identity that featured a task-oriented layout with a fresh look and feel. a key requirement for the interface was that the site display well in browsers in use by the majority of library users. the following list describes the web browsers used by approximately 95% of the visitors to our main library website as of august 2008: windows mac os internet explorer 6 firefox 2.x internet explorer 7 firefox 3.x firefox 2.x safari 3.x firefox 3.x the team chose yui [9] for page layout and jquery [10] for visual effects. course views makes limited use of yui. primarily, the project uses yui’s reset-fonts-grids css file for page layout and browser style resets. the team elected to use yui’s page layout css because it provides several customizable fixed-width, multicolumn layouts that display properly across all supported browsers. the bundled reset css made it even easier to create a consistent user experience across browsers by forcing explicit css declarations for all page elements. one particular challenge that the design needed to solve was the need to present a lot of information in a limited space. the team decided to combine the various search widgets into a single tabbed widget. after some experimentation with yui’s tabs, the team chose jquery ui’s tabs because of their semantic markup and ease of use. course views also uses jquery functions for a host of small changes on the site, including creating a two-column list and moving elements around in the dom. thickbox [11] displays the welcome overlay [2], and cycle [12] powers the technology lending widget. integration learning management systems several different learning technology systems are in use at ncsu: wolfware (a homegrown system), blackboard vista, and moodle. integration into these systems is a good strategy for reaching out to students because 89% of enrolled students have at least one course using an lms. wolfware, which is not a full-featured lms, now displays deep links to course view pages in its global navigation. it provides at least one class page for 75% of students. blackboard vista, formerly webct vista before the companies were merged, is the most popular lms in use on campus, with 69% of students using it for at least one course each semester. the moodle installation is a pilot project run by several colleges and units on campus, and provides access to at least one course for 11% of students. moodle usage is expected to increase over the next few years. it is fairly straightforward to integrate external data into moodle courses by developing moodle “blocks” in php. the course views block, which draws its content from a widget, links directly to the course view for the given course. it is available to all instructors and classes and can be added with just a few clicks. the product team is working to have the moodle block included in all new courses by default. automatic integration with blackboard vista has not been achieved due to a variety of technical and organizational constraints. the blackboard vista powerlinks kit [13] enables developers to integrate content into blackboard vista courses. unfortunately, developing powerlinks requires access to a development instance of blackboard vista as well as access to course data for building the links, neither of which were available at the time of development. as a result, we are not currently automatically integrating course views content into blackboard vista. at this point, instructors can temporarily grant a librarian access to their course sections so that the librarian can add a link to course views. between july 2008 and january 2009, instructors made 940 requests for course views links. fall 2008 requests from instructors provided course views access for approximately 25% of the course sections in vista. ncsu libraries website we also provide deep integration of course views content into the library website. beyond static links that appear on the library homepage and the learning commons site, the team took a look at existing information discovery systems to see if there were additional opportunities to promote the new tool. the first successful integration was with quicksearch, the library website’s integrated search system. searching for a curriculum code and course number (e.g. eng 101 or eng101) generates a page of matches, with links to the relevant course view at the top of the page, marked as a “best bet.” we also embed course-specific course views links into our installation of reservesdirect. this provides students who access the reserves system directly an opportunity to discover a broader collection of services and resources for their course. collaboration the course views project would not have been possible without a high level of collaboration among various groups. the project team worked closely with subject specialists, reserves librarians, and students to ensure that the final product would be a system that met everyone’s needs. subject specialists, in particular, are critical partners for the development team. their expertise in selecting resources for courses, as well as their interest in promoting the system to faculty and students, are crucial for the success of the course views project. throughout every phase of the project, the development team consulted with subject specialists to assure that they understood and valued the project, and had ample opportunity to voice their opinions and concerns. the development team also worked closely with campus colleagues who run learning management systems and train faculty in their use. the successful integration into wolfware, blackboard vista, and moodle would not have been possible without close collaboration with these partners. they also serve as important allies in marketing course views to faculty. the project team involved students in pre-development focus groups centered on concept-testing and gathering feedback on which widgets would be most useful. post-launch student focus groups have also provided invaluable feedback to shape the project’s next steps. faculty have been involved in pilot projects with course views pages and working with subject specialist librarians to tailor content to their courses. their interest in including library resources and e-reserves in their online course environments has also driven the project from its inception. course views is built upon successful collaboration across organizational lines both within the library and across campus. this collaboration resulted in a final product that both met the needs of all stakeholders and generated a wide sense of ownership and involvement that is crucial to the promotion and adoption of course views. usage initial usage data from the fall 2008 semester indicates that course views has been a success in increasing student use of library resources, despite modest promotion. during this time period, course views rendered an average of 1,162 pages per semester week, generating an average 1,073 clicks (on links or buttons) each week. controlling for some internally generated usage data and a normal end-of-semester drop, we have observed a mild downward trend of course views requests throughout the semester (figure 9). the team is investigating possible explanations for this trend, and using it as an opportunity to improve the project’s integration, customization, and promotion. course views uses a custom logging system that anonymously tracks requests to individual course views as well as “clicks,” which are defined as link clicks or form submissions. by implementing a custom solution, we are able to track course views usage at a much finer level than what is possible by analyzing standard web server logs alone. as a result, we are able to discretely measure requests for courses, curricula, and the use of individual widgets. figure 9: fall semester 2008 course views requests and clicks using these logs, the team discovered an interesting anomaly that occurred in week 14 of the semester. in typical usage, there is a roughly 1-to-1 correspondence between page requests and clicks, which is to say that the users typically perform one click transaction for each course view request. in week 14, however, the ratio was closer to 1.6-to-1. these clicks were the result of course views being used to display references for a research assignment in a specific course (chemical engineering 205). this anomaly suggests that deepening collaborative relationships with faculty has the potential to increase course views usage dramatically. figure 10: fall semester 2008 course views clicks by widget the chart in figure 10 is clear evidence that widgets based on dynamic content (the “reserves,” “search,” and “recommended” widgets) account for dramatically higher use (92.2%) than the remaining widgets, which display static data for all courses (7.8%). in particular, the reserves widget has been extremely popular, receiving 48.2% of all clicks. this usage pattern suggests that future development efforts should focus on enriching course views with more dynamically generated content. the usage observed so far suggests the project has been successful in expanding the reach of library content for courses. in the fall semester of 2008, course views served pages for 1368 unique courses, a ten-fold increase over the coverage we previously were able to achieve with authored course guides. future directions the results of our fall 2008 student focus groups, combined with continuing collaboration with subject specialists and campus lms stakeholders, will shape the next steps in the evolution of course views. the team is focusing on making it easier for librarians to author and maintain their content, as well as integrating design improvements. subject specialists have been eager to contribute new ideas for widgets. new mappings between course data and library resources may also make it possible to display content tailored to distance education and graduate-level courses. additionally, the system is being further enhanced to display some information discipline-wide and to associate curricula with a branch library. the course views project has accomplished its goals of providing full course coverage that is customizable to the course. our main focus at this point is to increase the actual usage of the service, through further technical developments, improved outreach, and more finely customized content. whatever form course views takes in the future, it will evolve by the same means with which it developed: close collaboration with a broad range of stakeholders. notes [1] ncsu course views project page http://www.lib.ncsu.edu/dli/projects/courseviews/ [2] ncsu course views frontpage http://www.lib.ncsu.edu/course [3] reeb, brenda, and gibbons, susan. “students, librarians, and subject guides: improving a poor rate of return.” libraries and the academy 4.1 (2004): 123-130. (coins) [4] corrado, edward m., and frederick, kathryn a. “free and open source options for creating database-driven subject guides.” code4lib journal 2 (2008). retrieved from <http://journal.code4lib.org/articles/47>. [5] libdata http://libdata.sourceforge.net/ [6] libcb http://sourceforge.net/projects/libcb/ [7] richardson, leonard, and sam ruby. restful web services. sebastopol, ca: o’reilly media, inc., 2007. (coins) [8] ncsu browse subjects http://www.lib.ncsu.edu/subjects/ [9] the yahoo! user interface library http://developer.yahoo.com/yui/ [10] jquery http://jquery.com/ [11] thickbox http://jquery.com/demo/thickbox/ [12] cycle http://malsup.com/jquery/cycle/ [13] blackboard powerlinks http://www.blackboard.com/communities/partners/powerlinks/what-are-powerlinks.aspx about the authors jason casden is the digital technologies development librarian at ncsu libraries, where he develops and implements scalable digital library applications. kim duckett is the principal librarian for digital technologies and learning at ncsu libraries, where she leads efforts to improve the library’s outreach in technology-enriched learning environments. tito sierra is the associate head for digital library development at ncsu libraries, where he leads a team that develops new digital library applications and services. joseph ryan is the digital projects librarian at ncsu libraries, where he works as a project manager for digital library projects and advocates for good user experience practices. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – ­the geospatial metadata manager’s toolbox: three techniques for maintaining records mission editorial committee process and structure code4lib issue 29, 2015-07-15 ­the geospatial metadata manager’s toolbox: three techniques for maintaining records managing geospatial metadata records requires a range of techniques. at the university of idaho library, we have tens of thousands of records which need to be maintained as well as the addition of new records which need to be normalized and added to the collections. we show a graphical user interface (gui) tool that was developed to make simple modifications, a simple xslt that operates on complex metadata, and a python script with enables parallel processing to make maintenance tasks more efficient. throughout, we compare these techniques and discuss when they may be useful. by bruce godfrey and  jeremy kenyon introduction as academic and research libraries move further into building research data collections, a growing number will become familiar with geospatial metadata formats and standards. a common feature of geospatial metadata formats – as opposed to those such as dublin core or mods – is the granularity of description with which they are encoded. contact information, data distribution services and methods, and a myriad of other details all are specified at the element and attribute levels.  in many cases, geospatial technology is designed to take advantage of this rich description.  however, the cost of being highly granular is the requirement to keep the content of records up to date as people come and go, web services change, and even data is updated (wayne 2005). over the past 15 years, inside idaho at the university of idaho library has amassed an extensive collection of geospatial data. as the collection has grown, maintenance and curation tasks have required us to utilize a variety of approaches to perform batch updates on large numbers of metadata records. we receive metadata authored by a variety of data creators with varying levels of expertise. these records come either through agreements with agencies to submit data periodically, are obtained from federal or private organizations, or come from university research projects. maintaining consistency and accuracy of these metadata records across the collection presents a challenge.  in most cases, there are no formal rules for cataloging in geospatial metadata, with the exception of syntactical rules, so it is not uncommon to see different variations of the same information. below, we describe three approaches that we have utilized to perform batch edits to large numbers of metadata records.  these include a locally developed software application and techniques using both xslt and python.  different methods are appropriate for different skill levels of users and open up possibilities to varied ways of automating workflows. each approach has been used successfully to update records in our repository. common geospatial metadata formats inside idaho supports metadata based on 1) the content standard for geospatial metadata (csdgm), 2) iso 191151 [1], and 3) arcgis metadata format 1.0.   csdgm dates to 1995, when the federal geographic data committee (fgdc) originally adopted it. (executive order 12096 1994) csdgm has undergone significant use and growth over the past 20 years, including the development of multiple “profiles” which extend the standard to cover areas like biology and ecology.  over time, federal agencies, state and local governments, higher education institutions, and other geospatial data creators have worked industriously to document the geospatial data they create and curate, primarily using this standard. today, the default format for most geospatial metadata is csdgm. in recent years, we have seen the adoption of the iso 191** (such as iso 19115) standards as the successor to csdgm.  these include 19110 for cataloging features, 19115-2 for cataloging remotely sensed imagery and gridded data, and 19119 for cataloging web services (against which a user would access the data). (international standards organization 2015)  these also function as the standard international geographic metadata format.  iso standards are more complex than csdgm and are developed with modularity and interoperability in mind.  this means that a given iso 191** series record may contain portions of each of the different standards. a map described in 19115 may refer to a 19110 record in order to describe the features represented on that map.  as a result of the modularity, iso is very powerful, particularly for supporting the use of web services or for aggregating a series of associated metadata records. (federal geospatial data committee 2011) arcgis metadata format 1.0 has also become a major standard due to the popularity and widespread use of esri’s arcgis product suite.  this format is used internally within the arcgis suite of products, and for many users, is the primary tool for generating standards-compliant metadata.  however, for many repositories, a best practice would be to transform this format into one of the other two for long-term curation and maintenance. a gui approach – the batch metadata modifier (bmm) tool solutions that employ a graphical user interface can be easier for some users than ones that require some form of coding or scripting.  the need to avoid repetitious script development led us to create the bmm[2].  it is a .net application, which requires windows (our primary workstation environment).  it can be used either as a standalone application or as a plug-in for arcgis, most recently configured for use in arcgis 10.2/10.3. (godfrey et al. 2015)  this tool is used most often where metadata records that have identical structures require simple updates.  for many organizations, simple gui tools can be very cost-effective and efficient. figure 1. screenshot from the batch metadata modifier tool. the image shows some features used to make changes to a single element across many files. one example of using this tool is standardizing keywords across multiple records. if we needed to make all representations of a term standardized, the bmm allows you to open all records in a set, select (as in figure 1) the specific keyword element, enter the correct value for that element, and then apply the change to all the files. alternatively, if you need to add missing information, you can load a template that represents the xml structure you need, and import the relevant elements – and their values – as you see fit (see figure 2).  one set of our csdgm records (approximately 75) contained the common names of insect and plant species as keywords.  we sought to also add the scientific names of the same species, in order to enhance discoverability and indexing.  the bmm tool allowed us to add those terms and a reference to the thesaurus across all records.  in other use cases, you can add elements required in order to validate schema or even replace elements wholesale, if they contain difficult modifications, such as altering the element attributes. figure 2. screenshot from the batch metadata modifier tool. the image shows some features used to make changes to a single element across many files. the bmm does have some limitations.  first, it is only available for the windows platform, and is for use with metadata in fgdc csdgm, iso 19115, and arcgis format 1.0.  so, if you are working in a linux environment with ecological metadata language (eml) records this is not the tool for you.  secondly, it does not have the ability to recognize conditionality or interdependency present between elements within a schema. case in point: if you need to update the contents of a contact position element based on a specific contact person’s name you will need to use the xslt or python approaches described below.  lastly, it is advisable to use this tool on files that have identical metadata structures.  users need to be careful when modifying elements in multiple files that don’t have identical structures.  in many use cases, the bmm’s features are very useful, but as metadata formats become more complex and semantically rich, the tool will need further development. an xslt approach complex metadata formats, such as iso 19115, contain elements and attributes which are dependent or conditional on other aspects of the schema.  for example, one of the significant differences between csdgm and iso 19115 is the introduction of schema-specific codelists.  codelists add attributes that semantically modify elements.  the element can be used to represent different types of work associated with a dataset by using the element, including custodian, owner, user, distributor, originator, publisher, principal investigator, and more.  figure 3 shows an example of this: <gmd:ci_responsibleparty> . . . <gmd:role> <gmd:ci_rolecode codelist="http://www.isotc211.org/2005/resources/codelist/gmxcodelists.xml#ci_rolecode" codelistvalue="principalinvestigator">principal investigator</gmd:ci_rolecode> </gmd:role> . . . </gmd:ci_responsibleparty> figure 3.  example of a codelist in use in iso 19115-2. this approach means that we can have multiple instances of in order to list the various parties involved in the creation, distribution, and management of a dataset, but in management of the record, we may only want to change the value of an element that relates to a specific role. recently, we were presented with a task where the bmm tool was insufficient without significant modification.  we received hundreds of arcgis format 1.0 metadata records from the bureau of land management (blm), idaho state office which were created using esri’s internal metadata schema.  across the records, we found that the organization’s name was written in many different forms.  since the records were in arcgis format, we verified that there were multiple variations of the publisher name by examining the content of the field across all records using esri’s arccatalog software.  at least three variations of the publishing organizations name existed in the records: department of interior (doi), bureau of land management (blm), idaho state office united states department of interior, bureau of land management u.s. department of interior (doi), bureau of land management (blm), idaho state office the source of the inconsistency was that more than one author over time had created these records and few rules were in place governing metadata creation. our locally-developed indexing tool ran against this field, and if the names were inconsistent, it would reject any that failed to match a designated string. normalization of the organization name was necessary for enabling indexing, discovery by users searching the catalog, and so that the inconsistencies didn’t propagate to other catalogs like data.gov.  we decided that the role code of publisher needed to be normalized to: u.s. department of the interior (doi), bureau of land management (blm), idaho state office it was apparent that creating an xslt stylesheet to handle the updates would be a preferable and workable solution.  xslt is expressly designed for manipulation of xml and is commonly used in many different domain communities. (nogueras-iso et al. 2004)  first, we created a template in xslt to normalize the responsible party organization name if it already existed and had the role code of publisher. <!- process the metadata using the templates below --> <xsl:template match="/"> <xsl:apply-templates select="node() | @*" /> </xsl:template> <!- copy all metadata content --> <xsl:template match="node() | @*" priority="0"> <xsl:copy> <xsl:apply-templates select="node() | @*" /> </xsl:copy> </xsl:template> figure 4.  section of xslt for creating the base template. next, in some records a citation existed, but the citation responsible party element (<citrespparty>) did not exist at all. we needed to generate a template to create the element and populate it with the correct responsible party organization text along with the role code value.  as part of this template, we counted the number of times the blm was somehow listed as publisher, by checking for instances in which the role code had a value of ‘010’ (meaning ‘publisher’) and if the ‘bureau of land management’ text was listed in the organization name.  if the value of the count was zero, we had to add the responsible party organization name and role code.  lastly, if none of those earlier cases applied, we left everything alone.  the xslt stylesheet (see figure 5) performed the necessary edits. <xsl:variable name="normalize_rporgname">u.s. department of the interior (doi), bureau of land management (blm), idaho state office</xsl:variable> <xsl:template match="dataidinfo/idcitation/citrespparty[role/rolecd[@value = '010']]/rporgname[contains(.,'bureau of land management')]" priority="1" > <rporgname> <xsl:value-of select="$normalize_rporgname" /> </rporgname> </xsl:template> <xsl:template match="dataidinfo/idcitation" priority="1"> <xsl:choose> <xsl:when test="not(citrespparty)"> <xsl:copy> <xsl:apply-templates select="node() | @*" /> <citrespparty> <rporgname>u.s. department of the interior (doi), bureau of land management (blm), idaho state office</rporgname> <role> <rolecd value="010"/> </role> </citrespparty> </xsl:copy> </xsl:when> <!-now, count the number of times blm is listed as a publisher. if it is zero then we need to add it. --> <xsl:when test="count(citrespparty[role/rolecd[@value = '010']]/rporgname[contains(.,'bureau of land management')]) = 0"> <xsl:copy> <xsl:apply-templates select="node() | @*" /> <citrespparty> <rporgname>u.s. department of the interior (doi), bureau of land management (blm), idaho state office</rporgname> <role> <rolecd value="010"/> </role> </citrespparty> </xsl:copy> </xsl:when> figure 5. section of xslt for normalizing the organization name. xslt allowed us to resolve an issue that the bmm tool could not.  also, xslt is portable between many software environments.  we can execute xslt from arccatalog, oxygen, python, .net and other environments as needed.  even though it took us a bit of time to become familiar with the syntax of this approach we feel certain that it was time well-spent and that we will take advantage of the knowledge we gained moving forward. a python approach python offers a lot of flexibility to address geospatial metadata maintenance.  an advantage to python is the tremendous freedom a user has to incorporate not only xml document modifications, but also to make use of its other libraries to automate and process metadata in highly variable ways.  we found the multiprocessing module an opportunity to run parallel processes on large numbers of files across a network, reducing the time required to make the modifications. for example, we recently had the need to update a position description as an employee received a new job title.  nearly all of our metadata records contained the employee’s name as the primary metadata contact and needed to be changed.  the bmm tool can be cumbersome to use on large numbers of files and once again the need to update one element based on the specific contents of another element was an issue.  our python script utilizes the elementtree module to parse the xml schema and find the relevant elements.  conditional expressions are used to check for null values and match the text and then the script will make a modification from “gis specialist” to “gis librarian” and record the number of times it made the change in a that record (figure 6). # define a function to perform work def update_metadata(metadatafile): count = 0 try: # parse the xml tree = et.parse(metadatafile) # find all occurrences of contact information (all the parents of contact position) # for all the individual contact information elements find the contact person and contact position # ".//" selects all sub-elements on all levels beneath contact information # "/.." gets the parent element (contact information) # helpful information: http://elmpowered.skawaii.net/?p=74 parents = tree.findall(".//cntpos/..") for parent in parents: if parent.find("cntorgp/cntper") is not none and parent.find("cntpos") is not none: if parent.find("cntorgp/cntper").text == "bruce godfrey" and parent.find( "cntpos").text == "gis specialist": parent.find("cntpos").text = "gis librarian" count += 1 elif parent.find("cntperp/cntper") is not none and parent.find("cntpos") is not none: if parent.find("cntperp/cntper").text == "bruce godfrey" and parent.find( "cntpos").text == "gis specialist": parent.find("cntpos").text = "gis librarian" count += 1 else: continue result_array = metadatafile, str(count) + " elements updated" #update the original file if needed if count > 0: tree.write(metadatafile) count = 0 except: result_array = metadatafile, "xml was not parsed correctly" return result_array figure 6. section of python for modifying the records. the standard output from the script documents which files were examined and the number of times changes were made within the file, including zero.  elementtree is a very useful library for modifying xml especially since it is included with the python install.  however, other options exist such as lxml.  you will need to install the lxml module but it offers additional functionality such as xpath and xslt classes. the advantage to using python is not just the varied modules that can be used to deal with xml, it is also the added functionality that other python modules introduce.  we tested the script using approximately 9000 records stored on an internal (local) hard drive and found that the processing occurred in approximately 1.7 seconds without multiple processes across the cpu’s threads. when using the multiprocessing module, the same script ran in 1.6 seconds.  while the difference is minor – especially with a simple activity like changing the text of a string – we noticed a decrease in time.  next, we tested the approach across 76,000 files across our network.  as a single process, it took 45.5 minutes to run.  as a multi-process, it took 33.2 minutes to accomplish the same task.  again, our modification in this instance was small, but larger tasks, such as running an xslt schema transformation from within python, would require more effort by the system, and multiprocessing could significantly reduce the time this takes. this process is accomplished in a fairly straightforward manner.  first, we set up the xml matching and modification function update_metadata within a process pool.  depending on how many cpus you have available, the system will process each consecutive group of files simultaneously until your cpu reaches its maximum threshold. in no case did we detect any negative reactions from the cpu from this processing burden (figure 7). def main(): print "start" starttime = time.clock() file_count = 0 # create a list for all the files my_list = [] # loop through all the files (recursive) for root, dirs, files in os.walk(path): for filename in files: if filename.endswith(".xml") and not filename.endswith(".aux.xml"): metadatafile = os.path.join(root, filename) file_count += 1 my_list.append(metadatafile) # create a pool class and run the jobs. pool = multiprocessing.pool() result_arrays = pool.map(update_metadata, my_list) for item in result_arrays: print str(item) + '\n' # synchronize the main process with the job processes to ensure proper cleanup. pool.close() pool.join() print "finish" stoptime = time.clock() ttime = ((stoptime starttime) / 60) print str(ttime) + " minutes to process " + str(file_count) + " files" if __name__ == "__main__": main() figure 7. section of python for running the update function as a multiprocess. python presents some opportunities for establishing automated, highly efficient workflows that reduce the time necessary to make required updates to your records.  in fact, beyond just metadata maintenance, using the multiprocessing module may introduce more efficient means of doing many different programmatic tasks throughout the library, including processing records for analysis and visualization. conclusion the best technique for managing metadata is the one that works for the personnel who are required to perform the function for any repository environment.  those with more programming-oriented skillsets may benefit from using python or other interpreted languages.  many metadata librarians and others who work with this information have experience in xslt, which is itself a powerful scripting language.  the bmm tool represents an approach that non-programmers can use to make changes across records, provided they have some familiarity with xml. we have found that a toolbox approach to metadata management is the only one that affords us flexibility, creativity, and responsiveness to the metadata we receive.  just as an auto mechanic needs different tools for different tasks in repairing a vehicle, and some of those tools will be replaced by others over time, so too must geospatial metadata managers maintain a collection of different tools.  keeping that toolbox up-to-date is always a challenge, but as libraries progress further in their work with research data collections, managing metadata using different approaches will be the best way to keep up.  moving forward we will certainly be looking for tools in addition to the ones described here to help us do our jobs more effectively and efficiently. references executive order no. 12906, 3 cfr, 882 (1995). retrieved from: http://www.archives.gov/federal-register/executive-orders/pdf/12906.pdf federal geospatial data committee. 2011. geospatial metadata. [internet] [cited 2015 apr 10] (july) available from: https://www.fgdc.gov/library/factsheets/documents/geospatialmetadata-july2011.pdf godfrey b, kenyon j, kyrios a, spinosa d.  2015.  updating your metadata just got easier. esri arcnews [internet] [cited 2015 apr 10] 18: 56-58. available from: http://www.esri.com/esri-news/arcuser/winter-2015/updating-your-metadata-just-got-a-little-easier international standards organization. 2015. standards catalogue: iso/tc 211 – geographic information/geomatics. [internet] [cited 2015 may 26] available from: http://www.iso.org/iso/home/store/catalogue_tc/catalogue_tc_browse.htm?commid=54904 nogueras-iso, j, zarazaga-soria, fj, lacasta, j, béjar r, muro-medrano, pr. 2004. metadata standard interoperability: application in the geographic information domain. computers, environment, and urban systems [internet]. [cited 2015 apr 10]: 28(6): 611-634. available from: http://dx.doi.org/10.1016/j.compenvurbsys.2003.12.004 wayne, l. 2005. metadata in action: expanding the utility of geospatial metadata. gisplanet [internet] [cited 2015 apr 10]: 1-6. available from: http://gep.frec.vt.edu/pdffiles/metadata_pdf’s/2.2metadatainaction.pdf notes 1. these standards are listed in the iso catalogue: http://www.iso.org/iso/home/store/catalogue_tc/catalogue_tc_browse.htm?commid=54904.  however, the national oceanographic and atmospheric administration (noaa) has created some training material that helps in learning more about them, including a workbook at: http://service.ncddc.noaa.gov/rdn/www/metadata-standards/documents/md-metadata.pdf 2. the batch metadata modifier is available for download on the inside idaho website at: http://insideidaho.org/helpdocs/batch_metadata_modifier_tool.html about the authors bruce godfrey is a gis librarian at the university of idaho. he previously served as its gis specialist from 2003-2013. he holds a m.l.s from the university of north texas, an m.s. from the university of idaho, and a b.a. from the university of virginia. jeremy kenyon is a research librarian at the university of idaho. he has been the subject specialist supporting the college of natural resources since 2010. he holds an m.l.s and an m.a. from indiana university, and a b.a. from the university of oregon. appendix: complete code xslt <?xml version="1.0" encoding="utf-8"?> <xsl:stylesheet version="2.0" xmlns:xsl="http://www.w3.org/1999/xsl/transform"> <xsl:output method="xml" version="1.0" encoding="utf-8" indent="yes" omit-xml-declaration="no" /> <!- process the metadata using the templates below --> <xsl:template match="/"> <xsl:apply-templates select="node() | @*" /> </xsl:template> <!- copy all metadata content --> <xsl:template match="node() | @*" priority="0"> <xsl:copy> <xsl:apply-templates select="node() | @*" /> </xsl:copy> </xsl:template> <!- all metadata xslt stylesheets used to update metadata should be identical to this example up to this point --> <!- add templates you'll use to update the metadata below --> <!- desired result: consistent responsible party organization name for the publisher role. there will be only one publisher --> <!- begin by normalizing the bureau of land management (blm) citation publisher responsible party organization name if it exists already. the following 3 variations are present in the metadata: 1. department of interior (doi), bureau of land management (blm), idaho state office 2. united states department of interior, bureau of land management 3. u.s. department of interior (doi), bureau of land management (blm), idaho state office make them all: u.s. department of the interior (doi), bureau of land management (blm), idaho state office--> <xsl:variable name="normalize_rporgname">u.s. department of the interior (doi), bureau of land management (blm), idaho state office</xsl:variable> <xsl:template match="dataidinfo/idcitation/citrespparty[role/rolecd[@value = '010']]/rporgname[contains(.,'bureau of land management')]" priority="1" > <rporgname> <xsl:value-of select="$normalize_rporgname" /> </rporgname> </xsl:template> <!- now if dataidinfo/idcitation/citrespparty does not exist, insert it and add blm as the responsible party organization name with the publisher role code (010). we need this template if blm has simply synchronized and hasn't added a responsible party. we assume dataidinfo/idcitation exists. this is a safe assumption in this case since dataidinfo/idcitation gets created when arcgis synchronizes the metadata and all of these blm data have been viewed in arccatalog--> <xsl:template match="dataidinfo/idcitation" priority="1"> <xsl:choose> <xsl:when test="not(citrespparty)"> <xsl:copy> <xsl:apply-templates select="node() | @*" /> <citrespparty> <rporgname>u.s. department of the interior (doi), bureau of land management (blm), idaho state office</rporgname> <role> <rolecd value="010"/> </role> </citrespparty> </xsl:copy> </xsl:when> <!-now, count the number of times blm is listed as a publisher. if it is zero then we need to add it. --> <xsl:when test="count(citrespparty[role/rolecd[@value = '010']]/rporgname[contains(.,'bureau of land management')]) = 0"> <xsl:copy> <xsl:apply-templates select="node() | @*" /> <citrespparty> <rporgname>u.s. department of the interior (doi), bureau of land management (blm), idaho state office</rporgname> <role> <rolecd value="010"/> </role> </citrespparty> </xsl:copy> </xsl:when> <!-otherwise, leave everything alone --> <xsl:otherwise> <xsl:copy> <xsl:apply-templates select="node() | @*" /> </xsl:copy> </xsl:otherwise> </xsl:choose> </xsl:template> </xsl:stylesheet> python # this script updates the position name for bruce godfrey from gis # specialist to gis librarian for csdgm records import os import multiprocessing import time import xml.etree.celementtree as et # set path to directory containing files path = (r"c:\temp\metadatamultiprocessing\data\csdgm\authenticatedmp") # define a function to perform work def update_metadata(metadatafile): count = 0 try: # parse the xml tree = et.parse(metadatafile) # find all occurrences of contact information (all the parents of contact position) # for all the individual contact information elements find the contact person and contact position # ".//" selects all subelements on all levels beneath contact information # "/.." gets the parent element (contact information) # helpful information: http://elmpowered.skawaii.net/?p=74 parents = tree.findall(".//cntpos/..") for parent in parents: if parent.find("cntorgp/cntper") is not none and parent.find("cntpos") is not none: if parent.find("cntorgp/cntper").text == "bruce godfrey" and parent.find( "cntpos").text == "gis specialist": parent.find("cntpos").text = "gis librarian" count += 1 elif parent.find("cntperp/cntper") is not none and parent.find("cntpos") is not none: if parent.find("cntperp/cntper").text == "bruce godfrey" and parent.find( "cntpos").text == "gis specialist": parent.find("cntpos").text = "gis librarian" count += 1 else: continue result_array = metadatafile, str(count) + " elements updated" #update the original file if needed if count > 0: tree.write(metadatafile) count = 0 except: result_array = metadatafile, "xml was not parsed correctly" return result_array def main(): print "start" starttime = time.clock() file_count = 0 # create a list for all the files my_list = [] # loop through all the files (recursive) for root, dirs, files in os.walk(path): for filename in files: if filename.endswith(".xml") and not filename.endswith(".aux.xml"): metadatafile = os.path.join(root, filename) file_count += 1 my_list.append(metadatafile) # create a pool class and run the jobs. pool = multiprocessing.pool() result_arrays = pool.map(update_metadata, my_list) for item in result_arrays: print str(item) + '\n' # synchronize the main process with the job processes to ensure proper cleanup. pool.close() pool.join() print "finish" stoptime = time.clock() ttime = ((stoptime starttime) / 60) print str(ttime) + " minutes to process " + str(file_count) + " files" if __name__ == "__main__": main() subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – the viability of using an open source locally hosted ai for creating metadata in digital image collections mission editorial committee process and structure code4lib issue 56, 2023-04-21 the viability of using an open source locally hosted ai for creating metadata in digital image collections artificial intelligence (ai) can support metadata creation for images by generating descriptions, titles, and keywords for digital collections in libraries. many ai options are available, ranging from cloud-based corporate software solutions, including microsoft azure custom vision and google cloud vision, to open-source locally hosted software packages. this case study examines the feasibility of deploying the open-source, locally hosted ai software, sheeko, and the accuracy of the descriptions generated for images using two of the pre-trained models. the study aims to ascertain if sheeko’s ai would be a viable solution for producing metadata in the form of descriptions, or titles for digital collections in libraries and cultural resources at the university of calgary. by ingrid reiche introduction one application of artificial intelligence (ai) in libraries is its potential to assist in metadata creation for images by generating descriptions, titles, or keywords for digital collections. many ai options are available, ranging from cloud-based corporate software solutions, including microsoft azure custom vision and computer vision, and google cloud vision, to open-source locally hosted software packages like opencv and sheeko. this case study examines the feasibility of deploying the open-source, locally hosted ai software, sheeko (https://sheeko.org/), and the accuracy of the descriptions generated for images using two of the pre-trained models. the study aims to ascertain if sheeko’s ai would be a viable solution for producing metadata in the form of descriptions or titles for digital collections within the libraries and cultural resources (lcr) at the university of calgary.  setting as with many institutions, the university of calgary’s libraries and cultural resources has a large number of digital collections; many of which have insufficient descriptive metadata in the form of accurate titles and descriptions. much of lcr’s digital collections are archival in nature, and have donor agreements that would require us to seek permission to process them through third-party ai services. during covid-19, lcr was in the process of migrating our digital collections from contentdm to a new digital assets management (dam) system by orange logic (https://www.orangelogic.com/products/digital-asset-management-system). while orange logic’s dam offers third party ai services from google and microsoft for keywords generation, facial recognition, and flagging offensive content, but it does not offer image captioning as part of its ai services. as we looked to ai to help improve and potentially automate some aspects of metadata creation for our digital collections, project sheeko offered a potential solution in the form of a local open-source software for producing captions that we hoped could be used to create descriptions or titles for images; particularly within some of our larger collections.   potential corporate service solutions for ai images in processing  before delving into the specifics of this case study of testing sheeko’s locally hosted ai, it is useful to do a brief environmental scan of the solutions that were now available to the university of  calgary, and weigh some of the strengths and weaknesses in relation to metadata creation. microsoft’s azure custom vision (https://learn.microsoft.com/en-us/azure/cognitive-services/custom-vision-service/overview) “is an image recognition service that lets you build, deploy, and improve your own image identifier models” (farley et al. 2022). there is one major difference between  microsoft computer vision and custom vision services. custom vision gives users the ability to specify labels to be used on objects within images and train models to detect those objects, whereas computer vision offers users a fixed “ontology of more than 10,000 concepts and objects” that can be applied to images (computer vision . . . c2023). computer vision also offers a suite of other services including optical character recognition (ocr), and facial recognition services that are also useful for the discovery of images in digital collections but not addressed in this article. while azure’s computer vision (https://learn.microsoft.com/en-us/azure/cognitive-services/computer-vision/overview) does offer a service for captioning as part of the cognitive services suite, this product is not included in orange logic third party services and would require further customization. google vision (https://cloud.google.com/vision) allows users to “assign labels to images and quickly classify them into millions of predefined categories. detect objects, read printed and handwritten text, and build valuable metadata into your image catalog” (vision ai … n.d.). lcr does plan on testing facial recognition technology, and the auto tagging of images with keywords in the future, but neither were useful on their own in deriving standard descriptions or titles we presently wanted. both microsoft and google services require the images to be sent via an api to their services, then processed and the results returned, which takes time and processing offsite. were this processing to be done on site with a local machine, image data could be processed at a higher volume with improved speed. additionally, both google’s and microsoft’s services have a cost associated, though the costs for these services are not substantial enough to be a deterrent from using these services [1][2]. the biggest strength of these third party services offered via orange logic is their ease of use, their integration with the digital collection platform, and their ability to recognize discrete objects or people and then auto-tag images. however, neither of these services offered the more robust solution of generating multiple words used in captions that are needed for the description of images, or their titles. on the other hand, sheeko did potentially offer a captioning solution. open-source local solution for image ai sheeko  sheeko is a locally-hosted ai software environment developed by a team from the university of utah’s j. willard marriott library, consisting of harish maringanti (associate dean for it & digital library  services), dhanushka samarakoon (assistant head, software development), and bohan zhum (software developer). sheeko applies machine learning models to image collections to detect  objects, and generate captions. sheeko is built on an open-source software environment whose components are listed or available for download via github. sheeko provides potential users three options of how to use the environment, as described in the “training a model” and “sheeko pretrained models resource” sections of the readme in the github repository (samarakoon and zhu 2021). sheeko users can train their own model (using inception v3 model), fine tune an existing model (using the im2txt model), or use one of seven pre-trained models available for download on their website (maringanti et al. n.d). however, before embarking on this process, users must install the package either in virtualbox or by downloading the code package and additional prerequisite software.   the intent of the testing of sheeko at the university of calgary was to ascertain if training a model, or deploying a pre-trained model of sheeko was a feasible answer to the need for description and title metadata required for image collections that were currently undescribed.  the test at the university of calgary was carried out in 2020 during covid-19, while most staff, myself included, were working from home. an initial attempt to install the virtualbox was unsuccessful, and i asked our technology services staff for assistance.  because the virtual box environment is supported using vagrant and requires that users enable bios mode, and this work was to be performed on a university managed laptop currently in my home, with remote assistance from technology services, the use of the virtual box was not an option. therefore, a local install of the code package was the chosen method. to deploy sheeko locally, there are specific hardware configurations required, and a dozen code package dependencies as listed in the catalyst white paper and in the getting started section of their github (maringant et al. 2019; samarakoon and zhu 2021). of the many system perquisites required there are specific gpu and driver installations, and a knowledge of python scripting at more than an introductory level. these tasks took a number of weeks for technology services staff to work through. once the software was installed we decided to test two pre-trained models on a set of 114 images. all of the pre-trained models available for download from project sheeko use tensorflow’s “show and tell” model as a base model which is refined to different specifications in each of the seven models available. tensorflow is an “end-to-end open source platform for machine learning” (tensorflow n.d.). the “show and tell model is a deep neural network that learns to describe the content of images” (kim et al. 2020). this case study used two models:  the ms coco model (ptm-im2txt-incv3-mscoco-3m) that was trained on images from  microsoft common objects in context; and marriott model (ptm-im2txt-incv3-mlib-cleaned-3m) that was trained on historical photographs from the marriott library’s digital collection. the main goal for the testing was to see if either of these models would be suitable for any of a variety of collections ranging from historical to more contemporary photographs. in the vein of more scientific experiments, we wanted to keep as many variables as possible constant across the two models. both the selected pre-trained models were previously trained with the show and tell model, with a starting point of inception v3 and 3 million steps. the two major differences between the models were the images used for the training, and that the marriott model went through natural language processing and cleaning to remove proper nouns.  image sample selection to ascertain the strengths and weaknesses of the two sheeko models selected, and to see which model performed best for images with certain qualities or objects, a cross section of 114 images from large digital collections that were lacking descriptive titles was selected. this included images from the glenbow library and archives (https://digitalcollections.ucalgary.ca/package/2r340826n9xm), featuring largely historical images about western canada and alberta’s provincial history and settler life, dating from the late 1800s onward. images from our mountain studies collection (https://digitalcollections.ucalgary.ca/package/2r340822nl8r), were of the national parks in the canadian rockies and featured mountains, landscapes, and wildlife. images from the calgary stampede collection (https://digitalcollections.ucalgary.ca/package/2r3bf1gbvdj), were again western themed but more contemporary and with a focus on horses, cowboys and fairgrounds, dating from the 1960s onward, though the collection itself contains images from 1908 onward. images from emi music canada archive (https://digitalcollections.ucalgary.ca/package/2r340822nozp), were of music objects including tapes, records, cds and other carrier cases. images from the nickle galleries were of a few art pieces, and currently not available to the public. images from the university of calgary photographs collection (https://digitalcollections.ucalgary.ca/package/2r3bf1sqxf7jw) featured university buildings from 1950s onward. the remaining images were taken from the alberta airphotos collection (https://digitalcollections.ucalgary.ca/package/2r34082235ho), the arctic institute of north america photographic archives collection (https://digitalcollections.ucalgary.ca/package/2r3bf1ss8hhxx) and the winnipeg general strike collection (https://digitalcollections.ucalgary.ca/package/2r340822num1). ranking system for sheeko captions when sheeko runs a model to generate captions for images, each model will output a json file that contains the all file names of the images processed, the three captions generated for each file name, and a confidence rating for each caption. a higher confidence rating did not necessarily correspond to the captions chosen as the best fit.  while the json output is useful, it does mean that users must view the files separately from the captions, which is not user friendly or helpful when one is trying to assess the best captions for an image. one of the library technology support analysts, karynn martin-li, at libraries and cultural resources had the brilliant idea of building a viewer that would display the image and the three captions from each model tested. the custom viewer made the process of selecting the appropriate caption from each model much more user friendly. this was custom built by martin-li and not included in sheeko, but this created an interface and made it much easier to use. {"file_name": "glenbowid21521.jpg", "captions": [{"caption_text": "a black and white cow standing in a field.", "score": 0.0012332111498308891}, {"caption_text": "a black and white photo of a cow standing in the snow.", "score": 0.0004844283302840979}, {"caption_text": "a black and white cow standing in a field", "score": 0.00048442775066202847}]}, {"file_name": "glenbowid21745.jpg", "captions": [{"caption_text": "a black and white photo of a city street.", "score": 0.0016210542383307202}, {"caption_text": "a black and white photo of a city street", "score": 0.000650292656660278}, {"caption_text": "a black and white photo of a city bus", "score": 0.0004501958654109137}]}, {"file_name": "glenbowid21841.jpg", "captions": [{"caption_text": "a black and white photo of a city street.", "score": 0.001934344178677972}, {"caption_text": "a black and white photo of a city street", "score": 0.0014626184147026346}, {"caption_text": "a black and white photo of a building", "score": 0.0007475796404555153}]}, for all 114 images processed, one of the three captions for each image was selected as the best fit from both the ms coco model and the marriott model. the selected captions from each of these models were ranked on a scale of zero to five (zero being the lowest quality caption, and five being the highest quality caption). ranking the captions was based on three variables: the identification of the subjects or objects present in the image (0-2 points), the background or setting of the image (0-2 points), and the image type (0-1 points). a total rank of three or above is something that would be considered usable as a base for a description. examining a few captions from each of the models will illustrate how the rank system worked. observations of sheeko’s captions the photograph titled “’canso’ at head of coronation fiord” in the university of calgary arctic institute of north america collection was given rank of five for ms coco model’s caption “a black and white photo of a plane in the water” (fig.1). the ms coco caption has appropriately described the type of photo as black and white, and the object of an airplane, and the background setting of water. the marriott model’s caption for this image was assigned a rank of one for its caption “photo shows a helicopter being used like a crane to transport a tower during construction of a lift”. this caption accurately describes the type of image as a photo, but does not correctly identify the object, or background in the image. figure 1. the image titled “cattalo, male, named ‘king’ in buffalo national park near wainwright, alberta.” in the glenbow library and archives digital collection received a rank of 4 for the ms coco model’s caption “a black and white photo of a cow standing in the snow” (fig.2). this caption correctly identifies the type of photo as black and white, but description of the objects is only partially  accurate. the animal in this image is a cattalo, a cross between a cow and a buffalo, granted it was anticipated that no ai model has been trained on cattalos at this point. but nonetheless the animal does appear much like a buffalo, and so the caption is half correct when it categorizes the subject as a cow. the background is also only partially correct as the animal is standing in the snow, but there is no mention of a field, pasture, or trees present in the background. the marriott caption for this image was “photo show a cow grazing” ranked at a score of 2, as the type of image being  described as photo is correct, but lacking the descriptors of black and white. the subject of a cow is half correct, and the animal is not grazing, and no background is mentioned. figure 2. for the postcard titled “dog team in heavy” by photographer byron harmon, in the mountain studies digital collection, both models produced captions with a rank of 1 (fig.3). the ms coco model generated the caption “picture of a vase of flowers on a table”, correctly identifying only the image type as a picture, and being incorrect about the dogs and man in the postcard, as well as the setting of a snow covered forest. the marriott model produced the caption “artwork a person” and received a rank of 1, as the image is an artwork but the artwork is a photograph so there were no points given for the type of image. the model did correctly identify one of the subjects as a person, but does not mention the dogs, nor does it describe the background at all.   figure 3. the overall results were that the ms coco pre-trained model performed significantly better, with 53% of the images having a score of 3 or above. the marriott library pre-trained model had only 15% of the images with a score of 3 or above. a full breakdown of the results can be seen in table 1. table 1. rank ms coco count of rank marriott count of rank 0 22 (19 %) 58 (50 %) 1 10 (9 %) 25 (21%) 2 21 (18 %) 16 (14 %) 3 29 (25 %) 13 (11 %) 4 24 (21 %) 1 (< 1 %) 5 8 (7 %) 1 (< 1 %) total images processed 114 114 of the images processed, both models performed poorly on images from emi music canada archive, the nickle galleries, and the university of calgary photographs collections, which contain particular types of objects such as cassettes, records, artwork, and portraits. from these results it was determined that sheeko would not be a good fit for images that require the identification of a particular location or images requiring individuals to be named. the models performed the best on images from calgary stampede collection and glenbow collections where it was describing more general objects such as groups of people, buildings, animals, or landscapes. were further testing to be done using sheeko, the ms coco model would be used as a basis for further training.   in the white paper report written by sheeko’s creators, “machine learning meets library archives: image analysis to generate descriptive metadata” they state ms coco models and other existing models “were developed with a focus on born-digital photographs of everyday objects and scenarios (maringanti et al. 2019). these models contained objects that were not present in historical photographs” (maringanti et al. 2019). the misidentification of certain subjects or objects was supported in this case study. it was clear that both models were trained on certain types of object and subject categories, for example horses and cows, but not bison, buffalo or cattalo. when the models see a cattalo it will be described as a horse or a cow, and not a bison. similarly, neither model produced useful captions for cassettes which the ms coco model identified as a nintendo wii game controller in one case, and the marriott model described as an artwork. to train a model to identify a particular object category not currently included in that model, such as a bison or a cassette, 10,000 images of that object need to be processed from various angles. training the ai models does not require a large time investment in processing, but would require a significant time investment in the form collection analysis and image identification to identify the subject and object categories required for training the models, and gathering the images to train a model. conclusion the goal of this case study was to test two of sheeko’s pre-trained machine learning models to ascertain if they would produce captions that could be used as descriptions or titles, and thereby reduce the amount of time and labor required by staff to describe images in the university of calgary’s digital collections. sheeko’s results show that it does produce captions that could be used as a basis for descriptions, with moderate human intervention. the captions would not be suitable for title creation without significant human intervention. were sheeko to be used for description generation, it is unclear if the amount of time required to select the most appropriate caption from the three choices generated and then create additional information, or modify existing description with information from the sheeko generated captions, would be less time than it would take for a person to create description metadata from scratch. at the end of this case study it was determined that sheeko would not be pursued as a potential ai solution for creating descriptive metadata at this time. the stand alone results from the ms coco model, though significantly better than those from the marriott model, only yielded usable captions for 53% of the images. the potential time commitment and resources required from personnel to train models to identify the type of objects and subjects in lcr’s digital collections without descriptions, or with minimal descriptions, were disproportionate to the quality of the results in this case study. the fact that deploying sheeko, and any further training of the models in sheeko would have leaned heavily on the skills of staff in lcr’s it, and required time to create models and source the images needed for modeling were all deterring factors in pursuing sheeko. this case study was initially designed as a research project for myself, the digital metadata librarian; and technology support were to enable this initiative where and when necessary. the reality was that this project was in many ways led by lcr’s it, to whom i am most grateful.  this case study took place while our digital services department was in the process of migrating from our old digital collections software contentdm to our new digital asset management system by orange logic. now that we have completed the migration, and are more familiar with orange logic, it is time to test the capabilities of the built-in third-party ai services, and explore other captioning solutions. sheeko presented a potential solution for describing a large amount of content locally, without the use of third-party providers, their services fees, or the processing time  associated with them. sheeko and local open-source environments also eliminate the need to pursue seeking permissions and re-writing donor agreements as the images are not processed outside of the home institution, also mitigating any potential data sovereignty issues. a local software solution similar to sheeko may still be useful in filling a gap in ai image description present in our new digital asset management system. the cost of sheeko was not a monetary services fee, but rather the cost of wages and valued staff time of our technology services staff to do the manual set-up of  the software environment needed to run sheeko, and build additional customization in the form of a viewer to facilitate ease of use. ultimately sheeko’s cost was great enough that it has been  eliminated as a potential solution for large-scale image captioning at present. notes [1] the price list for azure cognitive services, including computer vision is available at: https://azure.microsoft.com/en-us/pricing/details/cognitive-services/. the describe service is listed at $1.50/1,000 transactions when processing 0-1 million transactions. after 1 million transactions it is $0.60/1,000 transactions. if libraries and cultural resources were to process all 2,241,793 image in our collection it would cost $2,245. [2] the price list for google vision services is available at: https://cloud.google.com/vision/pricing/ the price for the majority of services, with the exception of web detection and object localization is $1.50/1,000 transactions when processing 0-5 million transactions.  if libraries and cultural resources were to process all 2,241,793 images in our collection, using any one of google vision’s services it would cost $3,362. however, google vision does not offer captioning. about the author ingrid reiche is the digital metadata librarian with libraries and cultural resources at the university of calgary. ingrid works largely with digital collections and is hopeful that ai can supplement human metadata creation to help describe the growing digital content in academic libraries. bibliography maringanti h, samarakoon d, zhu b. (2019). machine learning meets library archives: image analysis to generate descriptive metadata. lyrasis research publications. https://doi.org/10.48609/pt6w-p810  maringanti h, samarakoon d, zhu b. n.d. sheeko: a computational helper [internet]. atlanta (ga): lyrasis; [cited 2023 feb 17]. available from: https://sheeko.org/    samarakoon d, zhu b. [updated 2021 feb 17]. sheeko-vagrant. san francisco (ca): github; [cited 2023 feb 17]. available from: https://github.com/marriott-library/sheeko-vagrant farley p,  buck a, macgregor h, mcsharry c, downer r, coulter d, contributors. 2022 nov 22. what is custom vision? [internet]. redmond (wa): microsoft; [cited 2023 feb 17]. available from: https://learn.microsoft.com/en-us/azure/cognitive-services/custom-vision-service/overview  computer vision [internet]. c2023. redmond (wa): microsoft; [cited 2023 feb 17]. available from: https://azure.microsoft.com/en-us/products/cognitive-services/computer-vision vision ai [internet]. n.d. mountain view (ca): google; [cited 2023 feb 17]. available from: https://cloud.google.com/vision#section-3catalog tensorflow [internet]. n.d. [cited 2023 feb 17]. available from: https://www.tensorflow.org/overview kim j, murdopo a, friedman j, wu n, contributors. [updated 2020, apr 12]. show and tell: a neural image caption generator. san francisco (ca): github; [cited 2023 feb 17]. available from: https://github.com/tensorflow/models/blob/archive/research/im2txt/readme.md  subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – generating geographic terms for streaming videos using python: a comparative analysis mission editorial committee process and structure code4lib issue 45, 2019-08-09 generating geographic terms for streaming videos using python: a comparative analysis in libraries, the relationship between textual descriptions of audiovisual material and access to that material is a primary concern, as users expect to have access to all the library’s resources—which increasingly include audiovisual content—through a simple and effective web interface. at uw-oshkosh, library staff developed a unique site for its streaming video collection that would allow users to search for videos and browse collections on particular topics across each of the three vendors. in order to create more meaningful and topical collections, various programming tools and techniques were employed to identify geographical locations in vendor-supplied marc records. this article describes three different methods for generating geographic terms for streaming videos using different python libraries and evaluates them based on the number of terms generated, overlap in terms generated between the three methods, and the amount of cleanup needed to generate useful geographic terms. by patrick harrington introduction as librarians at james madison university pointed out when describing their library’s process of building a streaming video collection, “streaming media” is similar to other electronic formats (e.g., e-journals and e-books) in that it “requires a substantial number of additional steps to acquire and manage than tangible formats” [1]. while libraries can choose to host streaming content on their own servers, vendors providing streaming video content also make their content available through streaming video sites that the vendors host themselves. this reduces the cost of maintenance and storage for the libraries but libraries still have to give users the means of accessing the video content. libraries create metadata for video contents in a variety of ways depending on their library systems and the resources at their disposal. one common method involves adding bibliographic records, typically provided by the vendor, to their catalog so that users can find them in the online public access catalog (opac). in recent years libraries have made use of discovery services such as primo that enable users to perform keyword searches of the opac, article databases, and streaming video simultaneously. in some cases, libraries can provide access to electronic resources without adding records because their discovery service can index the collection and retrieve results through access points maintained by the vendor. they can also develop a dedicated portal for accessing streaming videos. whether a library chooses to provide access through the opac or through a separate system for streaming video (or through a combination of the two), it is likely that the library will utilize marc records at some point in its workflow. marc records are the primary format for organizing information about library resources regardless whether those records are created through original or copy cataloging done by the library or provided by library vendors. the marc standard allows for the storage of a large amount of detailed information about a resource but can be intimidating to work with, especially for non-catalogers. dublin core was developed to describe a variety of resource types online. in their book creating a streaming video collection for your library, duncan and peterson maintain that “by using dublin core as its base schema, a system ensures interoperability, flexibility, simplicity, and consistency” and reduces the workload of trained catalogers since creating dublin core records is “easy for staff to learn and implement and quick to create.” [2] dublin core is not without drawbacks. one significant disadvantage is the lack of granularity compared to marc. this means that dublin core records often do not “provide enough information about complex resources with issues such as copyright or access information” [2]. while most vendors of streaming video for libraries primarily offer metadata in marc format, cultural heritage institutions often utilize versions of dublin core when creating applications for the web. a 2009 paper from the international conference on dublin core and metadata applications describes the relationship between dublin core and access to streaming video resources for the european digital library europeana: “the widely adopted dublin core metadata element set have provided archives with the tools necessary to build meaningful services that provide unified access to archive videos online” [3]. in building the tool discussed in their paper, the creators of video active utilize a semi-automatic process whereby participating archives create a mapping of their own metadata to the dublin core elements [3]. oomen et al. converted the xml files into rdf triples in order to allow for more sophisticated querying. while the dublin core metadata standard dictates what kind of information should be stored about a particular resource, rdf allows for machines to interpret logical relationships between pieces of information in order to provide more sophisticated querying. oomen et al. provide an example of the kind of querying that is possible using this technology. in their example, they suggest a user query looking for items dealing with germany: this query is formulated in serql “bring me items that has ‘geographical coverage’ the term ‘germany’ and also bring me items of the narrow terms of ‘germany.'” the field ‘geographical coverage’ gets values form[sic] the video active thesaurus. in this way, instead of getting only the items that have been explicitly defined that have geographical coverage the term ‘germany,’ we also retrieve the items from all the narrow terms of germany (i.e. hamburg, thuerlinger, saxony etc.) [3]. this kind of intelligent querying represents the promise of semantic web technology. while wider implementation of the semantic web points to more robust access systems than current methods, finding a way to enrich and utilize vendor-supplied metadata for keyword searching offers a way of enhancing access to this material in the near term. creating the site polk library currently sources its streaming videos from three vendors. in 2016, library staff developed a separate website that would allow users to browse all the streaming videos available from the three different vendors and view thumbnails of videos as well as create personalized lists for future viewing. the streaming video site originally contained only a curated set of videos on topics which were thought to be of interest to students and faculty. the site was re-launched in 2017 with the aim of including all streaming videos available to library patrons. the relaunched site would also allow site administrators to create customized topical collections in response to current events or at the request of faculty and staff on campus by grouping together all videos with particular subject headings, geography terms, individuals, or series names as opposed to individually adding each title, as was the case with the previous version. a screenshot of the streaming video site is shown below:   figure 1. a screengrab from the streaming video site at polk library.   the site runs on the content management system plone and is updated based on the addition of new titles or the release of new marc records from vendors. when library staff are notified that new records or titles are available, the marc records are downloaded and then a python script extracts information from specified marc fields to produce a comma-separated value (csv) file which is then uploaded to the site. once a server-side script captures a thumbnail for each video, it is made available to end users. code for the streaming video site itself can be found here: https://github.com/polklibrary/polklibrary.content.viewer/tree/master/polklibrary.content.viewer. this project had two main goals. one was to increase the visibility of this part of the library’s collection and give patrons a new way to explore the streaming videos available. the other more technically-specific goal was to integrate videos from three different vendors into one system so that videos about similar topics or geographical areas could be grouped into collections regardless of which vendor provided them. enabling this kind of cross-vendor browsing required substantial cleanup and enrichment discussed in further detail in the following section. creating metadata for the streaming video site because the marc standard was developed for use in library catalogs, the data had to be transformed to be effective for retrieval in the site itself. the dublin core metadata element set provided the basis for the fields which would ultimately appear in the site. not all elements from this set were used in the final crosswalk, such as the relation, language, and publisher elements. these were excluded because they were deemed not relevant to user interests or access compared to elements like description, title, contributor, and subject. while transforming marc records into dublin core records is a familiar task in libraries, differences in the extent and quality of the cataloging done across the various vendors meant that additional cleanup and enrichment was needed before the records could be added to the site if all the goals of the project were to be achieved. the figure below shows the subject heading fields provided by two different vendors for the same video, in this case a television documentary called “a blooming business.”   figure 2. subject heading for “a blooming business” provided by two different vendors.   while the genre term “documentary films” and the “human rights” heading appear in both records, one vendor applied geographic subdivisions in subfield z while the other vendor did not. additionally, one vendor seemed to emphasize the specific industry discussed in the documentary, in this case cut flowers, while the other focused on geopolitical issues like globalization. these differences are to be expected considering two catalogers will not always catalog the same thing in the same way, but some reclamation and enhancement is necessary if users are to use subject terms to find resources similar to ones they have already found without having to account for cataloging differences across vendors. the python library pymarc is used to read in the vendor-supplied marc file, extract the relevant fields, and output a csv file suitable for upload to the streaming video site. a table showing the fields used in the streaming video site at polk library along with the corresponding marc field (or fields) from which the fields are populated can be found below.   table 1. fields in streaming video site along with their source marc field. streaming video site field marc field(s) filmid 001 creator 100 title 245 date 260 subfield c runtime 300 series 490 summary 520 associated entities 600,610,700 geographical terms 651 subfield a,650 subfield z subjects 650 genre 655 url 856 subfield u   initially the subject terms were consolidated into a list of shared terms which were then applied to any video which contained them in its title, summary or subject headings. this process was an attempt to simplify the process of making collections by reducing the number of possible subject terms in the site and minimize the effort needed to create collections on a particular topic that could overcome the discrepancies in cataloging between the three streaming video vendors at polk library. however, this workflow was abandoned because the resulting set of possible subject terms was too narrow to be useful when building collections on the site. instead, all subject terms in the marc record are now included on the streaming video site. while most videos had at least one subject heading in the marc record, other topical information, such as particular geographical areas covered in a video, was considered useful for users of the site but not always present in the vendor-supplied metadata. obtaining this information was of interest particularly for collection building, as it would allow administrators to create collections for particular geographical areas or to limit topical collections to videos about that area (e.g., politics in russia or government in the united kingdom). this was the most challenging aspect of the metadata creation and cleanup process. since the initial upload consisted of around 65,000 records and new videos are added regularly, a programmatic means of identifying geographic terms was the only viable option. while most of the videos had subject terms in the 650 field, the 651 field, which is meant for geographic information, was rarely used by any of the three vendors. in order to generate this geographical information, a number of methods, including a few python libraries as well as parsing of existing subject headings, were used to try and generate geographic terms for the streaming videos. all three methods generated geographic terms for at least some streaming videos, but each method approached the problem in a different way. in one method, the natural language toolkit (nltk), a natural language processing library, was used to parse descriptions of videos in order to identify geographical place names based on sentence structure. another method relied on regular expressions to identify place names in text. the third and final method explored in this paper leverages the structure of library of congress subject headings so that geographic subdivisions in subject headings are used to populate a standalone metadata field in the final site. each of these methods will be discussed in more detail, followed by the results of a systematic comparison of each method. generating geographic terms with the nltk according to its developers, the nltk is “a leading platform for building python programs to work with human language data,” supporting a number of natural language processing tasks including named entity recognition [4]. named entity recognition is a subset of natural language processing that grew out of the realization that “it is essential to recognize information units like names, including person, organization, and location names” [5]. one well-known classifier for named entity recognition is the stanford named entity recognizer (ner), which analyzes text with the purpose of identifying persons, organizations, and locations within that text [4]. together the nltk and stanford ner rely on heuristic modeling to capture the structure of a sentence by identifying various parts of the sentence, including but not limited to named entities. depending on the size and complexity of the sentences, this can be a very computationally intensive process. one reason this approach may be preferable is that it does not require an exhaustive list of all potential locations in order to search for locations in a string of text. because of this feature, it was thought that using the nltk and stanford ner would be an adequate, if time-consuming, solution to the problem of generating geographic tags for each video. in order to use the nltk and stanford ner, a python script takes a set of records and then parses the summary of each video. while the stanford ner attempted to extract all named entities, only terms identified as locations were used to populate the geographic field for each video. by design, the ner identifies more than just geographical terms, so the python script would first have the ner analyze the full summary and then look for all the terms tagged as being locations so that they could be added as geographic terms for that particular video. below is a snippet showing how the nltk was used to identify the geographic terms within the summary of each film:   figure 3. code snippet showing implementation of nltk.   generating geographic tags with regular expressions in contrast to the nltk/ner method, the python library geotext relies on regular expressions to search for references to geographic locations based on comparing input text to a predefined list of geographic locations which are built into the library [6]. regular expressions consist of query sequences which are used to identify particular characters within a chunk of text. while the nltk is concerned with parsing sentences based on modeling their linguistic structure, regular expressions ignore grammatical context and instead focus on matching a specific character sequence. below is a snippet of code showing how the geotext library was used to identify the geographic terms for each film:   figure 4. code snippet showing geotext implementation.   one significant benefit of the regular expression approach compared to the nltk is that regular expressions are much faster. however, this speed comes at a price, as it simply looks for matches without taking into account the structure of the sentence or the context in which a particular term is used. the geotext library presented some unique challenges because there were a number of very common english words or proper nouns that the library identified as geographic locations (e.g. “along”, “police”, “university”). for example, without cleanup, video titles or descriptions which included the word “along” would appear to users as having to do with the city in india, thus undoing much of the benefit of this metadata enrichment. because the library is able to identify so many geographic terms, it was only by manually evaluating the results on large files that all these potentially erroneous terms could be identified and removed. for this project, a file consisting of 264 common english words and proper names which were identified as locations by geotext were removed if detected, though they were preserved for the purposes of testing in the trials outlined below. at times, the geotext library correctly identifies a place but can generate false positives by associating a city name with a different country than one might expect. for example, the script will correctly identify the string “barcelona” as a place name, but when both country code and city name are included in the geographic field it becomes clear that the library has identified the city of barcelona, venezuela even though most appearances of the city name were in reference to the spanish city, not the venezuelan. by removing the country but keeping the city name, these can be disambiguated at least in practice, but manual evaluation is needed in order to perform this cleanup if site administrators want to make a collection about venezuela. in addition to removing terms which were homographs of locations according to geotext, the library generates two letter country codes when either the name of a country is detected or the name of a city within that country is detected. thus, a separate function in the python script would swap the two letter country code for the full name of the country using a dictionary. this also speaks to another limitation of the method: any important geographical locations that were not cities or countries (e.g. multi-national regions or geological formations) would not be included as geographic terms if only geotext is used to generate geographic terms. using library of congress subject headings according to the marc standard, subject added entries which contain a geographic name are to be recorded in the 651 field. in theory, simply extracting values from this field could yield useful geographic terms for each video. unfortunately, this field is not found very often in the marc records provided by the three streaming video vendors. subject added entries which are topical terms, found in the 650 field, provide another opportunity to extract this metadata, at least in some cases. according to the marc standard, subfield a could include a topical term while subfield z is where geographic subdivisions for a subject heading are encoded. subfield z is exclusively meant for geographic subdivisions. some videos may not have any geographic subdivisions in their subject headings, but this method would leverage the existing metadata for each video rather than require additional metadata be generated and then cleaned, as would be the case using either of the other methods listed above. in this particular case, both the 651 and 650 subfield z for each record were checked for geographic terms as part of the same script, with any terms found there automatically extracted. comparison and evaluation a systematic comparison of the three methods explored for the purposes of creating this streaming video site lays out the strengths and weaknesses of each method. first, python was used to create 10 sample files each composed of 200 randomly selected records from each of the three vendors, extracting data from the marc fields based on the mapping shown above. then a single python script applied the geotext library to the title and summary of each video and the stanford named entity recognizer to the summary of each video. in addition to these two methods, pymarc was used to extract the contents of 651 subfield a and 650 subfield z in the vendor-supplied marc record. the results of these two were then compared to the geographic terms held in the 651 and 650 subfield z fields if either field was present. the most noticeable difference between these two methods is the time required. every test of the geotext method took under a second to run, while all but one test of the nltk method required over an hour and a half to run. while there were significant differences in time needed for each method, the number of videos for which geographic terms could be generated based on existing metadata were quite similar for each, as shown in the table below. these numbers only measure whether terms could be generated for a particular sample of videos, not the quality or utility of the terms generated.  they are given along with the “tagging percentage,” which reflects the average percentage of videos for which geographic terms were generated by each of the three methods tested. these values are averages across the 10 samples, each consisting of 600 total records.   table 2 comparison of tagging percentage for the three enrichment methods tested. videos with lc terms lc tagging percentage videos with nltk terms nltk tagging percentage videos with geotext terms geotext tagging percentage average 158 26% 286 48% 294 49% median 161 27% 289 48% 295 49% max 171 29% 301 50% 318 53%   in addition to determining whether any geographic terms could be generated, the results of each method were also compared with each other to identify any terms shared by two of the three methods or, in some cases, all three methods. the table below shows the results of these comparisons, again averaged across 10 sample files of 600 records each.   table 3 comparison of common terms between each of the three enrichment methods tested.   nltk/geotext common terms nltk/marc common terms geotext/marc common terms nltk/geotext/marc common terms average 176 15 30 10 median 175 15 30 11 max 194 18 36 13   as shown in the table above, both the nltk method and geotext method were more likely to have generated the same geographic term for a particular video than they were to match any geographic terms provided in the marc record. while a majority of the videos had few terms, looking at those for which both methods were able to generate many terms may offer guidance to those looking to enrich or create geographic terms for streaming videos based on textual metadata. one video identified during testing generated a large number of terms in both methods and while it was an outlier in the dataset, it is worth discussing in greater detail because it contained many of the problems inherent in trying to generate this kind of metadata. in this case, the original marc record provided by the vendor includes a brief summary of the plot of the video along with a listing of film festivals where the video won an award, including the city and country where the film festival took place. these names and places make up a significant number of the geographic terms generated by both methods. the video in question is a documentary about the logging industry in papua new guinea. both the nltk/ner method and the geotext library identified papua new guinea as a geographic term in the record while also including 13 other geographic terms which are merely festivals where the film was screened. while both of these methods correctly identified papua new guinea as a relevant geographic term because it was found in the summary, the term did not appear in either the 651 subfield a or 650 subfield z. the list of shared erroneous terms would have been longer if the geotext library had not failed to recognize the erroneous term “quebec” in the record because it will only recognize the term if it includes the diacritic (i.e., “québec”). this is of course just one example, but contained in this one record is a decent summary of the key challenges in enhancing metadata for streaming video based on existing textual records, particularly when working with many records at a time. the implicit assumption guiding the attempt to generate this kind of metadata was that geographic location is a meaningful concept from an access perspective, meaning both end users and site administrators would like to browse or group videos together by the geographic locations discussed in the video. while advancements in internet infrastructure have made streaming video over the web more technologically feasible, users are still dependent on text-based descriptions of those videos if they are to be found either by browsing particular subject and geographic headings or through keyword searching. a closer look at videos for which the ntlk and geotext identified at least one identical term reveals the challenges of working with either method. below is a table of the terms generated by the nltk and geotext libraries for five videos which had at least one term in common between all three methods. the values are presented as they appear with no cleanup or deduplication, with the exception of adding the full country name in place of the two-letter country codes that are provided by the geotext library.   table 4 sample of terms generated before cleanup for each of the three enrichment methods tested. 651/650$z terms nltk terms geotext terms ‘brazil’, ‘peru’, ‘argentina’, ‘brazil’ ‘brazil’, ‘peru’, ‘argentina’ ‘australia’, ‘australia’ ‘warren’, ‘australia’, ‘united states’ ‘united’, ‘states’, ‘alaska’, ‘st.’, ‘patrick’, “‘s”, ‘cathedral’, ‘new’, ‘york’, ‘jfk’, ‘library’, ‘denver’   ‘new york’, ‘denver’, ‘united states’ ‘england’, ‘france’, ‘virginia’, ‘antietam’ ‘most’, ‘virginia’, ‘union’, ‘lincoln’, ‘united states’, ‘france’, ‘czech republic’, ‘south africa’ australia. ‘earth’, ‘australia’, ‘gondwana’, ‘earth’ ‘australia’   while both libraries identify cities and countries, the nltk also included specific locations within those cities, such as st. patrick’s cathedral in new york city, while the geotext library included seemingly erroneous countries such as the czech republic, which was included because the description contained the word “most” (the name of a town and district in the czech republic). of the five videos shown here, only the last video had any geographic terms in the marc record. relying solely on geographic terms in the marc records might make for less cleanup, but there will also be fewer geographic terms. conversely, either method of generating these terms from textual descriptions will require significant cleanup if the metadata is to be both comprehensible and useful to end users. a table summarizing the benefits and drawbacks of each method can be found below:   table 5 benefits and drawbacks of each of the three methods tested. method benefits drawbacks nltk identifies locations beyond cities and countries does not require exhaustive list of all locations requires more time and greater computing resources additional cleanup needed to properly format results geotext very quick wide geographic coverage generates many false positives additional cleanup needed no non-city or country terms available extraction from marc fields no cleanup on geography terms needed many videos lack geography terms dependent on vendor-supplied metadata   as more audiovisual media becomes available, packaging the media with adequate and accurate metadata about the audiovisual object will be crucial to making the material accessible to users. the challenges faced in creating the site are indicative of how difficult it is to extract this information in batches if it is not provided from the beginning. though there is nothing particularly cutting edge about the techniques used here from a programming perspective, since much of the searching was done with publicly available libraries, it would have been completely unnecessary if vendors had made full use of the marc standard, which has fields for recording geographic coverage information, in creating the catalog records that were to be shared by their customers.   table 6 comparison of tagging percentage for three vendors tested. videos with terms in marc record videos with nltk terms videos with geotext terms vendor 1 10% 55% 56% vendor 2 40% 38% 41% vendor 3 30% 51% 50%   the table above gives the percentage of videos which had geographic locations in either the 651 or 650 subfield z fields, averaged across the 10 random samples along with the number of videos for which the nltk or geotext library were able to identify geographic terms, also averaged across the 10 samples. assuming that geographic information is an important access point for users, there are clear disparities in how different vendors choose to utilize the marc standard, which in turn will impact which of the three methods above would be most effective should libraries endeavor to build a similar streaming video site as the one at polk library. extreme machine learning technologies suggest the possibility of automatic classification of audiovisual material [7]. for now, the nature of current discovery services demands that high quality textual metadata accompany audiovisual material if that material is to be accessible. despite the deficiencies of language to adequately reflect all dimensions of audiovisual material, textual description remains the primary access point for most web services. even if digital technologies for processing and analyzing audiovisual material continue to improve, high-quality metadata and the technologies needed to process and enrich them at scale will remain an essential part of delivering audiovisual content to users. bibliography [1] duncan c, peterson, e. 2012. you ought to be in pictures: bringing streaming video to your library. in:  bernhardt b, hinds l, strauch k, editors. something’s gotta give: charleson conference proceedings; 2011 nov 2–5; charleston. west lafayette (in): purdue university press. p. 453. [2] duncan cj,  peterson ed. 2014. creating a streaming video collection for your library. lanham (md): rowman & littlefield. p. 41. [3] oomen j, christaki a, tzouvaras v. 2009. television heritage and the semantic web: video active and euscreen. in: dublin core metadata initiative international conference on dublin core and metadata applications; 2009 oct 12-15; seoul. p. 97–105, 98. [4] bird s, klein e, loper e. 2009. natural language processing with python. boston (ma): o’reilly media inc. http://nltk.org/book. [5] nadeau d, sekine s. a survey of named entity recognition and classification. 2009. in: sekine s, ranchhod e, editors. named entities: recognition, classification and use. amsterdam: john benjamins publishing company. p. 3. [6] palenzuela ym. 2014. geotext [internet].[cited 2019 jun 20]. available from: https://geotext.readthedocs.io/en/latest/. [7] lu b, wang g, yuan y, han d. 2013. semantic concept detection for video based on extreme learning machine. neurocomputing 102:176-183. about the author patrick harrington is a metadata specialist at the university of minnesota and previously the metadata librarian at the university of wisconsin-oshkosh. he holds an mlis from the ischool at the university of illinois. acknowledgments thank you to david hietpas for his work in developing the streaming video site as well as assisting with the redesign of the metadata workflow. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – using open source tools to create a mobile optimized, crowdsourced translation tool mission editorial committee process and structure code4lib issue 24, 2014-04-16 using open source tools to create a mobile optimized, crowdsourced translation tool in late 2012, osu libraries and press partnered with maria’s libraries, an ngo in rural kenya, to provide users the ability to crowdsource translations of folk tales and existing children’s books into a variety of african languages, sub-languages, and dialects. together, these two organizations have been creating a mobile optimized platform using open source libraries such as wink toolkit (a library which provides mobile-friendly interaction from a website) and globalize3 to allow for multiple translations of database entries in a ruby on rails application. research regarding successes of similar tools has been utilized in providing a consistent user interface. the osu libraries & press team delivered a proof-of-concept tool that has the opportunity to promote technology exploration, improve early childhood literacy, change the way we approach foreign language learning, and to provide opportunities for cost-effective, multi-language publishing. by evviva weinraub lajoie, trey terrell, susan mcevoy, eva kaplan, ariel schwartz, and esther ajambo introduction over the last 10 years, oregon state university libraries & press (osulp) has launched a number of open source programs, including libraryfind and library à la carte. in early 2009, osulp began developing mobile tools and became one of the first libraries with a mobile site optimized for both smartphones and feature phones. osulp has focused on mobile website development to avoid developing for a specific platform or device. as a land-grant institution, we felt optimizing for mobile was a better solution because it promised fewer mobile device compatibility issues. we feel this approach promises optimal openness and availability of our resources. maria’s libraries is a non-governmental organization (ngo) based in new york and working in rural kenya, which, in conjunction with kenya national library services and a network of community groups, has been testing demand-driven approaches to literacy engagement and exploring leapfrogging technologies through a network of libraries in resource-poor settings. maria’s libraries’ work includes building a modern, fully-equipped library in busia, kenya. they test applications with users from a network of libraries called kitabu kenya, where they are working to bring together local level information needs and innovative practices from around the world to develop cutting edge information services through libraries (maria’s libraries [updated 2014]). in 2012, at the american library association (ala) annual meeting in anaheim, weinraub lajoie gave a presentation outlining the work osulp was doing in exploring alternative publishing paradigms. at the conclusion of the presentation, kaplan, a maria’s libraries co-director, approached weinraub lajoie with an idea for a collaboration that would harness osulp expertise in mobile technologies and publishing. kaplan proposed that osulp create a mobile website platform that would use crowdsourcing techniques to enable maria’s libraries’ user base to translate folk tales and existing children’s books into a variety of languages and dialects. what are we looking to accomplish the isolation of even neighboring villages in rural kenya contributes to linguistic drift, such that groups that speak the same root language often use different words to describe the same item, making it difficult and cost prohibitive to print texts in local languages. there have been repeated attempts to make publishing physical books in local languages feasible, but with over forty local languages in kenya alone there is no business model that provides a cost-effective solution (dekutsey, 1995). by moving to an electronic format, and creating a translation tool like this, we are hoping to build a cost-effective mechanism for local language publishing with both educational and cultural preservation objectives in mind. most of the projects the osulp team participates in are either based on large open source communities like hydra, one-off projects for our press, or projects we work on directly with a faculty member. working with maria’s libraries was different in that we were building a new tool, with untested technologies, in support of users well outside of osu. this particular project, however, fell into a few key areas outlined in our strategic plan. in osulp’s strategic goal three, strategy 3, we are tasked to “develop knowledge creation and dissemination opportunities.” our mission statement asks us to “…build gateways to unique resources…” and our vision statement inspires us to serve “as a model of innovation for academic libraries and university presses” (osulp strategic plan 2012 -2017[updated 2012]). requirements as part of our partnership agreement we were tasked with creating a mobile-optimized, crowd-sourced translation platform. this proof-of-concept would allow any user who created a username and password the ability to translate pieces of text into their local dialect and to vote for which translations were best. because we needed the tool to work for groups who spoke different dialects of the same language, we had to allow for the ability to accept multiple sub-dialects of a single language as well as identify the most accurate translation of new dialectic versions. because our partners at maria’s libraries had provided tablet computers to the libraries in busia, building the tool optimized for use on a mobile device was a primary goal. there is ample evidence that due to infrastructure issues, people are skipping over desktops and moving directly to mobile technologies (bridges, rempel & griggs, 2010). we have seen this kind of leap before throughout the developing world. the original requirements document received from our partners included everything they hoped a completed project would do, including a workflow to recommend new root languages and the addition of interactive and cultural content in audio, video, and image format. for the first iteration the requirements were pruned to basic functionality: a system to vote on and view the results of translations from multiple users along with the ability to use it without an active network. content, or what are we translating? libraries and educators are well aware of the benefits of early childhood education on lifelong learning. children with early exposure to books and reading are far more likely to succeed in school than those who do not (snow, et al. 1998). rural kenya finds this effect compounded because the language spoken at home is not always represented in classroom teaching or in available storybooks. children are therefore taught to read later in life and are often taught to read in a language they do not yet speak. for example, in busia, children will initially be taught in their local language of swahili until the equivalent of the third grade, at which point they will be taught in english. this abrupt shift and lack of preparation has significant life-long effects on literacy and educational outcomes (kaplan, et al. 2012). the children who will gain the most from these books are aged 2 – 5, though older children with previous reading exposure will also benefit. these children have not yet started school, and are unlikely to have seen a book or a computer before. there are likely no books in their local language, and any available books in swahili are not geared toward their age group. from the technical perspective, having a small number of words and an obvious delineation between ideas makes the children’s stories ideal for a mobile environment. additionally, since the mama mtoto (mother-child) story time program, which uses mother-child story time to introduce books and a love of reading to young mothers and their children, had already translated texts into local languages, it gave us existing content to work with. in-house resources osulp’s development team consists of 2.5 fte and 4 student employees residing within the emerging technologies & services (ets) department. in 2013, ets voted on the implementation of an open source development policy, which was approved unanimously by library administration. the policy outlines the reasons why we feel that participation in open source development is at the core of our work as an academic and a land grant library. new projects are created in github and are covered under a gpl license by default. this particular project was developed in a local instance of gitlab, open only to our staff. we chose to develop this project within gitlab, a self-hosted private solution, instead of github because we are currently seeking outside funding, and want to keep our options open in regard to future development and fundraising. we expect that we’ll eventually release the code freely on github once we’ve explored all possible fundraising options. for the initial proof-of-concept, two programmers were tasked with spending 50% of their time over a two-month period for development. after its unveiling, one of our staff members spent 50% of their time over an additional month making the interface a bit more user friendly. creating a mobile website osu libraries & press has made a commitment to create mobile websites rather than native applications. we’ve taken this stance largely because of difficulties maintaining native applications across the wide variety of mobile devices that are currently in use. however, this decision presented a few challenges for a project like this. offline features one of the features requested was to be able to use the application without an active network connection. while html5 has a variety of options for offline functionality available, they’re currently limited – primarily by space. for instance, html5’s localstorage allows you to store data associated with a key and access it offline. however, in many browsers that storage is limited to 5 mb, which doesn’t allow for a lot of local edits. other offline storage options in browsers often allow for greater storage (websql), but not all browsers implement them. fortunately, static assets are simple to store offline – html5 has a cache manifest file which can define which assets should be cached. an example of the cache manifest can be seen below: figure 1. cache manifest enabling the cache is as simple as adding <html manifest=”maria.appcache”> to the very top of the html. it’s important to note that cached assets will not be refreshed for clients unless the cache manifest file is updated. screen limitations designing for mobile devices greatly restricts the available real estate for things like descriptive strings. the solution is often to use immediately recognizable icons, which, while not using much space, offer a simple way to access functions. however, in this particular application there are a variety of actions which don’t have a globally understood icon, such as “begin translating this.” further, representations we chose might not have the same meaning for the audience in africa we’re trying to reach. this consideration led to choices such as an open padlock for logging in and a pencil icon for “begin translating.” while these may not always be obvious, they should be easy to remember. we have done our best to select appropriate icons, but further user testing will have to be done to determine their effectiveness. device access unfortunately creating a mobile website rather than a native application means that developers are unable to access much of the device’s internal functionality. for instance, at one point there was a request to be able to attach photos to a book’s translation to offer greater insight into the culture behind that passage. unfortunately, at the time there was no cross-compatible way to allow for attaching images from a device’s internal storage or to access its camera. however, the apis to access this information are quickly improving. for instance, ios 6 and above now allows basic file inputs to attach files from internal storage. what it takes to crowdsource three major points were identified as requirements of a crowdsourced system: a simple interface to navigate, read translations, and add dialects the ability to quickly navigate and vote upon “sections” of a translation. a system that takes the highest voted sections and combines them for a complete translated text. voting on sections a crowdsourced document is only as good as the opinions of its crowd. however, those opinions need to be facilitated by offering relevant content and its appropriate context to vote on. therefore, we identified the need for both the separation of a document into sections (“pages”) and each section into sub-sections (“sentences.”) in this case, fortunately, we had clear breakpoints for both requirements. each page of the children’s stories’ traditional printed book consisted of relatively few sentences and had a clear idea presented along with a picture. therefore, we defined each section as a page of the original book, whose total translation would be voted on, and each sub-section as a sentence on that page which could be individually translated. a simple interface building the mobile interface utilized existing expertise within osulp as well as research into common solutions for interactivity. we felt that as the success of the system relied upon participation it was our responsibility to make the process as seamless as possible. in the end we came up with the following workflow: as the user enters the site they are offered a choice of languages to select. figure 2. language selection screen upon selecting a language, the user is presented a carousel of book covers to choose from. each of these books has been marked for translation in the requested language, but may or may not have been completely translated. this carousel was implemented using wink toolkit’s carousel plugin. figure 3. book selection screen the current page of the book is indicated by a blue dot on the right hand side of the document. swiping up or down allows navigation between pages. figure 4. page view if the user finds that the translation of the page they are reading is inaccurate they may swipe to the left or right to switch to a different translation of the text. translations are ordered per page by number of votes, making the next translation the next most likely to be correct. when logged in the user can see a star on each page translation – selecting the star applies their vote to that user’s page. the highest voted page becomes the default view when selecting that book for a translation. figure 5. alternative translation view should the user find that there’s no adequate translation, the user may log in, click the edit button, and click individual sentences to translate, while still being able to view the original translation for comparison. from there they can save their translation and it becomes available for others to vote on. figure 6. page edit view using votes to build a book the strength of the system lies in the trust of its users to only vote up those pages with the most accurate translations. a new visitor will see only those pages that have the most votes. should they decide the translation is inaccurate they can navigate to other translations and vote for those; the pages they’ve voted on will become the default for them the next time they visit, assuming they’ve logged in. this provides an incentive for users to vote: they create the best experience for themselves while simultaneously improving the experience for others. technology the application was written within the ruby on rails framework, primarily because of existing experience within osulp as well as due to the availability of a variety of plugins (“gems”) to speed development. below are the technical requirements for the project, and the solutions we implemented to meet those goals. implement a registration system devise (https://github.com/plataformatec/devise) offers a standard and feature-laden solution for registration and authentication in ruby on rails applications. allow multiple users to provide multiple translations of a single sentence and easily recall them within the application. the globalize3 gem (https://github.com/globalize/globalize) provided the ability to assign and recall translations of a database object based on the user’s chosen locale. using this in combination with user-specific locales (e.g., en-dickens or sw-antoinette) we could simply interact with models in a translation-aware way. figure 7. changing local translation display carousels and other navigation methods common to mobile applications wink toolkit (http://www.winktoolkit.org/) provided us a small, easy to use, and customizable library that allowed for smooth animations in both the mobile and desktop environments. their carousel and slider plugins in particular proved to be the most efficient amongst the potential solutions we found. other alternatives, such as jquery mobile, twitter bootstrap, and sencha touch, either proved to be too complicated, have too much overhead, or resulted in jittery transitions. what’s missing? so far only the first phase of development has been completed for this project, which leaves a few crucial elements to be finished. administration there’s no way for an administrator to import stories, including both text and images, through the interface. further, there is no automatic separation of books into portions of translatable text with related images – in the current iteration it is done largely through a manual process. there is also no way for administrators to override the will of the crowd, leaving it open to multiple accounts poisoning the data. further crowdsourcing users are unable to recommend new languages and dialects for a book to be translated into. in the final iteration there would be the ability to vote on potential languages and make public those that have been suitably translated. offline functionality the offline functionality of the application needs to better enable the use of the site from remote locations. currently it largely depends on the system’s local cache – the use of html5 localstorage as well as offline manifests could greatly improve its functionality when a user attempts to use the application without an active network. conclusion initially, our intention in creating this platform was based solely around the idea of helping literacy in africa, but we’ve come to realize that the tool itself could have impact well outside the originally intended role. language studies programs, and individuals working with dead or dying languages were excited about the tool and how they might be able to integrate it with their teaching. we are very excited about the possibilities of the usefulness of the platform as a way of publishing books in lesser-known languages or in regions where dialectic publishing is cost prohibitive. it’s our hope that, if we get further funding to expand on the tool, we can release it through github and others can take what we have started and fork it to meet their needs. references maria’s libraries about [internet]. [updated 2014]. brooklyn (ny): maria’s libraries, about; [cited 2014 feb 4]. available from: http://mariaslibraries.org/about/ back dekutsey, woeli, 1995. the story of apnet, harare: african publishers network, commissioned by unesco. back bridges, laurie. hannah gascho rempel, kimberly griggs. 2010 making the case for a fully mobile library web site: from floor maps to the catalog. reference services review [internet]. [cited 2014 feb 4]; 38(2): 309 – 320. available from: http://hdl.handle.net/1957/16437 back oregon state university libraries & press [internet]. [updated 2012]. corvallis (or): oregon state university libraries & press, strategic plan; [cited 2014 feb 4]. available from: http://osulibrary.oregonstate.edu/flipbook/strategicplan/ back snow, c. e., burns, m., & griffin, p. 1998. preventing reading difficulties in young children. washington (d.c): national academy press, p. 4. back kaplan, eva, schwartz, ariel. 2012. mama mtoto story time-use cases. brooklyn (ny); maria’s libraries report. back acknowledgements we’d like to thank suzanne dikker, maria wafula, richard atuti, jimmy cater, and many community members in busia, elangata wuas, garissa, and lamu, especially their librarians. further, partners at africa soma, particularly caroline archambault and jennie glassco, provided critical inputs to this project. all community work related to the mama mtoto story time project was made possible by support from beyond access, global giving, and many private individuals. about the authors evviva weinraub lajoie is the director of emerging technologies & services at oregon state university libraries & press where she works on digital publishing, mobile, and open source tools development. trey terrell is an analyst programmer in the emerging technologies & services department at osulp. he works on research and programming for web-based projects. susan mcevoy is an analyst programmer in the emerging technologies & services department at osulp. she does development and design and is responsible for guidance of web-based projects. eva kaplan is the director and co-founder of maria’s libraries. she has worked in the field of international development for over seven years and is the managing editor of the journal of globalization and development. ariel schwartz is the director and co-founder of maria’s libraries. she is pursuing her doctorate, as well as a portfolio in applied statistical modeling, at the university of texas at austin. her interdisciplinary research sits at the intersection of development studies, information science, and organizational science. esther ajambo has been the head librarian at the busia community library for over four years, managing all day-to-day operations of the library including all literacy and information programs implemented at the busia community library. subscribe to comments: for this article | for all articles one response to "using open source tools to create a mobile optimized, crowdsourced translation tool" please leave a response below: chris markman, 2014-12-22 have you guys talked to the folks running amara? http://amara.org/en/ very similar goals, but for video. leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – using r and the tidyverse to generate library usage reports mission editorial committee process and structure code4lib issue 39, 2018-02-05 using r and the tidyverse to generate library usage reports gathering, analyzing, and communicating library usage data provides a foundation for thoughtful assessment. however, the amount of time and expertise required creates a barrier to actually using this data. by using the statistical programming language r and the tools and approach of the tidyverse, the process of gathering, analyzing, and communicating data can be automated in ways that reduce the amount of time and energy required. at the same time, this approach increases staff capacity for other data science projects and creates a shareable model and framework for other libraries. this article focuses on electronic resource usage reports – especially counter db1 reports – but this approach could be extended to other data sources and needs. by andy meyer introduction this article describes a project that used the statistical programming language r to generate reports about electronic usage and spending. this project had three main goals. first, to automate and systematize the process of gathering, compiling, and communicating data about electronic resource usage. second, to increase library staff capacity for data science. third, to provide a shareable framework and model for other libraries to use when dealing with data intensive projects. this project focused on electronic resource usage information in the form of counter reports but can be extended to manage and communicate data from a variety of other library systems or other data sources. background with library budgets shrinking and the cost of online resources growing, many libraries must critically examine electronic resource usage and related costs to make responsible collection development decisions. yet gathering this information, transforming the raw data into something usable, and communicating relevant information is a difficult and labor intensive process. the goal for this particular project is to provide electronic resource usage information to other library staff to aid collection development decisions. however, this approach could be extended to deal with other data and communicate those results to a wide variety of stakeholders. r and the tidyverse r is an open source programming language freely available under the gnu general public license. r excels in statistical computing and is growing in popularity across many disciplines. r enjoys a large and active user community as well as a number of packages that extend the basic functions of r. the growing popularity of r, the strong and active user community, the ability to do reproducible and rigorous data analysis, and the open nature of r make this an excellent option for many library applications. furthermore, developing staff expertise in r may position the library to take on additional roles in data management and create new connections on campus. the tidyverse is a set of r packages that “share a common philosophy of data and r programming and are designed to work together naturally.”[1] the tidyverse is a great place to begin learning r because these packages are widely adopted and relatively intuitive. there is also an abundance of high quality documentation that is freely available online. counter reports counter is a non-profit organization that provides libraries, publishers, and vendors a code of practice and set of definitions that facilitates the standard and consistent way to look at electronic resource usage.[2] the project described in this paper uses the database 1 report. these reports include the number of regular searches, result clicks, record views, federated/automated searches by month and by database. this is a standard report that most electronic resource vendors provide; details and definitions are available on the counter website.[3] this project uses revision 4 reports but the general approach outlined here could easily be extended to include other revisions and/or other counter-complaint reports. code structure and style this project approaches data science following the model proposed by garrett grolemund and hadley wickham in “r for data science”. the code is divided and structured into three discrete stages: import and tidy transform, visualize, and model communicate figure 1. model of the tidyverse approach to data science from from grolemund, garrett, and hadley wickham. “r for data science” http://r4ds.had.co.nz/introduction.html.[4] following another tenet of the tidyverse, this project has adopted a functional approach to r. this approach limits redundant code and makes the meaning of the code more transparent. in terms of style, the code is written for clarity and simplicity; it is not written to optimize performance. the hope is that the variable and function names as well as the comments allow others to easily use and adapt this code to their local institutional needs. lastly, this project makes extensive use of the pipe function in r. as a core member of the tidyverse, this function makes r code more readable and beautiful by combining complex operations without requiring nested functions. the book “r for data science” offers a very clear explanation of the pipe function.[5] tidy data before moving on, a note about terminology. in the context of the tidyverse and throughout this paper, the word “tidy” carries a specific and technical definition. the goal of tidy data is to create an underlying structure that makes various types of transformation easier. hadley wickham offers this definition of tidy data: “tidy data is a standard way of mapping the meaning of a dataset to its structure. a dataset is messy or tidy depending on how rows, columns and tables are matched up with observations, variables and types. in tidy data: each variable forms a column. each observation forms a row. each type of observational unit forms a table. messy data is any other arrangement of the data.”[6] according to this definition, counter-compliant reports, although standardized, are messy because the structure of the data – the column and row arrangement – does not correspond with the observations and variables. the process of tidying data begins by identifying the variables. for counter-compliant database report 1, the variables are: database publisher platform user activity date (expressed as month-year) usage looking at data from a sample report, we can see that while the report is standardized the data is messy because each row contains many observations. database report 1 (r4) total searches, result clicks and record views by month and database period covered by report: 2015-01-01 to 2015-12-31 date run: database publisher platform user activity reporting period total jan2015 feb2015 mar-2015 apr-2015 may2015 jun2015 database a publisher x platform z regular searches 13811 758 1884 1686 1951 601 186 database a publisher x platform z searches – federated and automated 988 49 163 108 145 60 16 database a publisher x platform z result clicks 10719 611 1562 1277 1609 531 181 database a publisher x platform z record views 1032 80 223 173 177 50 31 conveying this data in a tidy dataframe would keep the database, publisher, platform, and user activity columns and add columns for date and usage. database publisher platform user_activity date usage database a publisher x platform z regular searches jan-15 758 database a publisher x platform z searches – federated and automated jan-15 49 database a publisher x platform z result clicks jan-15 611 database a publisher x platform z record views jan-15 80 database a publisher x platform z regular searches feb-15 1884 database a publisher x platform z searches – federated and automated feb-15 163 database a publisher x platform z result clicks feb-15 1562 database a publisher x platform z record views feb-15 223 database a publisher x platform z regular searches mar-15 1686 database a publisher x platform z searches – federated and automated mar-15 108 database a publisher x platform z result clicks mar-15 1277 database a publisher x platform z record views mar-15 173 database a publisher x platform z regular searches apr-15 1951 database a publisher x platform z searches – federated and automated apr-15 145 database a publisher x platform z result clicks apr-15 1609 database a publisher x platform z record views apr-15 177 database a publisher x platform z regular searches may-15 601 database a publisher x platform z searches – federated and automated may-15 60 database a publisher x platform z result clicks may-15 531 database a publisher x platform z record views may-15 50 database a publisher x platform z regular searches jun-15 186 database a publisher x platform z searches – federated and automated jun-15 16 database a publisher x platform z result clicks jun-15 181 database a publisher x platform z record views jun-15 31 the data in this table is tidy because all of the variables are in columns and each observation is a row. code overview all of the code for this project lives on github: https://github.com/ameyer24/learningr/tree/master/electronic%20resources setup as a preliminary step, this project includes a file that sets up the rest of the process by loading the appropriate packages and by setting up the folder structure. ############################################################################### # loading packages ____________________________________________________________ ############################################################################### library(tidyverse) library(readxl) library(lubridate) library(mosaic) library(zoo) library(scales) library(knitr) ############################################################################### # setting up working directory _______________________________________________ ############################################################################### getwd() db1_folder <"./db1_reports" output_folder <"./outputs" the library commands load the appropriate packages that are required in this project. if the packages are not installed, you will first need to install the packages. the “setting up working directory” section sets relative file paths to the inputs folder as well as output folder. this definitions could be changed to explicit file paths to meet local needs. import and tidy the first step of this project is to import the data from all the files from the folder defined above and create a single tidy dataframe in r. this process might be complicated by a variety of file formats and inconsistent data ranges. to address these problems, the functions in this project can handle counter-compliant reports as both csv and excel files while providing a generalizable framework for other file types. the functions can also de-duplicate the usage data so that reports spanning overlapping date ranges will not cause problems. an area of improvement would be to improve handling of missing data. these functions are designed to work with database report 1 data but could easily be adapted to import and tidy other counter-compliant reports. tidy the data from counter reports # transforms data from the standard counter format to a tidy dataframe. tidy_reports <function(df) { df %>% select(-c(5)) %>% gather(date, usage, -c(1:4)) %>% mutate(date = as.yearmon(date, "%b-%y")) %>% mutate(usage = as.numeric(usage)) %>% rename("user_activity" = "user activity") } this function transform data from a counter-compliant format to a tidy dataframe. looking at each line in detail: select(-c(5)) – this line deletes the 5th column. the reporting period total is not useful within the “tidyverse” and is therefore removed. gather(date, usage, -c(1:4)) – the gather function is the primary way of moving data from a “wide” format into a tidy “long” format. this step keeps the data in columns 1-4 by collapsing all the other columns in key-value pairs. this function creates new variables for the key and value; for counter-compliant reports, the key is the “date” variable and the value is the “usage” variable. mutate(date = as.yearmon(date, "%b-%y")) the mutate function adds a new variable from existing variables; here we update the “date” variable by converting that data from a simple character string to the year-mon data type. mutate(usage = as.numeric(usage)) – similarly, this function converts the usage data from a character to numeric. rename("user_activity" = "user activity") – lastly, this function renames the “user activity” column to avoid the space in the column name; this is probably optional but made life easier while working with the data. import the data from the files the function above transforms a dataframe within r; these functions load the data from a specific file and then apply the tidy_reports function defined above. # this function imports data from csv files and makes that data tidy. import_csv <function(file) { file %>% read_csv(skip=7, col_names = true) %>% tidy_reports() } # this function imports data from excel files and makes that data tidy. import_excel <function(file) { file %>% read_excel(skip=7, col_names = true) %>% tidy_reports() } these functions accept a single file as a parameter and import that data by calling the appropriate reading function. both processes skip the first 7 lines because those lines do not contain relevant information for this analysis and then tidy the data using the function defined above. it is somewhat inelegant to have two functions that do essentially the same thing but keeping the two function separate improves the readability of the code and is effective, conceptually simple, and easily extendable. a generalized import function would also be an improvement. loading and tidying this phase of the project now has functions that can load a file and tidy the data; we now need a function that can apply these functions to all the files in a given folder. # these functions load db1 reports from a given folder. load_csv <function(path) { csv_files <dir(path, pattern = "*.(csv|csv)", full.names = true) tables <lapply(csv_files, import_csv) do.call(rbind, tables) } load_excel <function(path) { excel_files <dir(path, pattern = "*.(xl*|xl*)", full.names = true) tables <lapply(excel_files, import_excel) do.call(rbind, tables) } these functions accept a path to a folder as a parameter and then create a list of file names for files that match a particular pattern. the second line applies the appropriate import function to the list of files. lastly, the do.call(rbind, tables) line binds together these tables by row. the tidy dataframe at last, we have the functions needed to import data from every csv and excel file in a specified folder and create a single tidy dataframe in r. db1_data_csv <load_csv(db1_folder) db1_data_excel <load_excel(db1_folder) db1 <unique(rbind(db1_data_csv,db1_data_excel)) the first two steps create two separate dataframes call db1_data_csv and db1_data_excel that contain the data from the respective file formats. the final line combines the data into in a single dataframe called db1. the unique operation deletes any duplicate observations that may be present in the original usage reports. transform and visualize after importing and tidying the usage data, the next step in wickham’s model is to transform, visualize, and model this data. right now, this project performs basic transformations such as filtering, summarizing, and graphing the data. however, given the tidy structure of the data and the powerful tools that r provides for data transformation, there is an opportunity to build functions that radically transform the data and that allow for new insights. additionally, given the shared data structures and tools, there are opportunities to share these transformation and visualizations freely and create new sets of best practices. like the import and tidy process, this project uses a functional approach to handle transformations and visualizations. the majority of these functions perform transformations on the tidy dataframe created earlier without changing or updating that original dataframe. rather than exhaustively surveying all possible transformations, this paper will highlight two functions. the first function transformation summarizes the data based on academic terms; the second function uses that summarized data to create a simple barplot. transform summarize_usage_academic_term <function(databasename, startyear, endyear, action = all_actions){ db1 %>% filter(database %in% databasename) %>% filter(date >= startyear, date <= endyear) %>% filter(user_activity %in% action) %>% mutate(year = year(date), month=month(date)) %>% mutate(academic_term = derivedfactor( "spring" = (month==1 | month==2 | month==3 | month==4), "summer" = (month==5 | month==6 | month==7 | month==8), "fall" = (month==9 | month==10 | month==11 | month==12) )) %>% group_by(database,user_activity,academic_term,year) %>% summarize(usage=sum(usage)) %>% rename("user activity" = "user_activity") } this function accepts four parameters: database name, start year, end year, and action. it uses these parameters as filters to the tidy dataframe by returning only observations that match the parameters. the action parameter is optional and defaults to all actions through a variable defined earlier. as noted earlier, the mutate function creates new variables from existing variables. in this case, the mutate function creates three new variables from the existing date information: year, month, and academic term. strictly speaking it was not necessary to create a new variable for month – the academic term could be derived directly from the date – but it has been included in the hopes that it makes the code more accessible. the mutate function that creates the academic term variable uses the derivedfactor function from the mosaic package. the last transformation of the data was to group the data and then summarize the usage data based on academic term. when this function is applied to our sample data, we get this: database user activity academic_term year usage database a record views spring 2015 653 database a record views spring 2016 147 database a record views summer 2015 103 database a record views summer 2016 53 database a record views fall 2015 276 database a record views fall 2016 174 database a regular searches spring 2015 6279 database a regular searches spring 2016 4896 database a regular searches summer 2015 1125 database a regular searches summer 2016 1802 database a regular searches fall 2015 6407 database a regular searches fall 2016 5474 database a result clicks spring 2015 5059 database a result clicks spring 2016 3879 database a result clicks summer 2015 933 database a result clicks summer 2016 1481 database a result clicks fall 2015 4727 database a result clicks fall 2016 4091 database a searches – federated and automated spring 2015 465 database a searches – federated and automated spring 2016 338 database a searches – federated and automated summer 2015 88 database a searches – federated and automated summer 2016 89 database a searches – federated and automated fall 2015 435 database a searches – federated and automated fall 2016 122 visualize this function transforms the summarized usage data into a barplot. ggplot(data = summarized_usage_academic_term, aes(x = year, y = usage)) + geom_bar(aes(fill=factor(year)),stat="identity") + facet_grid(. ~ academic_term) + labs(y = "usage", fill="year") + ggtitle("database usage by academic term") + theme(axis.title.x = element_blank(), axis.text.x = element_blank(), axis.ticks.x = element_blank()) the ggplot function comes from the ggplot2 package, a core member of the tidyverse, that is able to create publication quality graphics from a robust set of options and layers. the graphing function creates a base layer with the summarized usage data and sets the aesthetic properties for the entire graph with year on the x axis and the usage data on the y axis. the next layer creates a bar graph; the color of the bars will be determined by the year and the statistic to graph is the identity. we also want a faceted graph; one bar graph for each academic term. the last few lines – started with labs – add labels, titles, and some theming to the barplot. these lines are optional and only scratch the surface of the customization options available within ggplot. figure 2. bar graph created from sample data. in a few short lines, these functions have transformed the tidy usage data into a high quality graph that shows database usage by academic term. creating a comparable graph in excel is possible but the advantages of r are clear. this function exists apart from the actual data; that means that the usage data can be updated and the function can simply be re-run on the updated data set creating consistent graphs over time. similarly, this same function can be used to updated to examine a different database or different date range. lastly, this approach allows for libraries to develop and share transformations and create best practices for domain-specific usage data. communicate the final stage in the model and the last step in this project is to communicate the results. for this part of the process, this project uses the r markdown package. r markdown can combine text, r code, and the results of r code such as graphs and tables and output the results in a variety of file formats including word, pdf, and html. r markdown also offers formatting options and customized themes that can create professional and polished reports. this project's r markdown file begins by calling all the previous files as a source. this allows the r markdown document to use the underlying data and all of the functions defined in earlier files. next, because this project hopes to generate a standard report for many different databases, the document specifies a database and data ranges at the beginning of the document. with these definitions in place, the rest of the r markdown document can call and execute r code without the need for repetitive data entry; updating the database variable at the beginning of the report updates the entire report. this makes generating the same report on a regular basis simple; add new data in the specified folder and re-run the report to include the new data. conclusion using r and the tidyverse to generate library usage reports has many clear advantages relative to alternative methods. automating the reporting process for electronic resource usage has the potential to save hours of staff time and create standardized reports that allow for better collection development decisions. transforming and visualizing this data has the potential to create new insights and raise new questions. beyond improving this particular process, using r and the tidyverse provides the library with many other benefits and new opportunities. learning and using r for data projects builds capacity for other data projects and collaborations across campus. using a free, open source language allows allows libraries to build and share data structures, transformations, and visualizations. the programming language r is powerful enough to recreate any data transformation and is much more shareable than data manipulation done in spreadsheets or in proprietary software. lastly, r has the ability to combine data from a variety of sources. data from circulation transactions, gate counts, computer usage, and head counts could be combined to get a more comprehensive sense of library usage. r is a free and open language that is growing in popularity. this article shares one library's limited experience using r to automate electronic resource usage reports but also argues that the opportunities and advantages for the library community are strong and clear. about the author andy meyer is the head of electronic resources and interlibrary loan at north park university in chicago. he holds a masters degree in library and information science from the university of illinois urbana-champaign where he earned a specialization in data curation. he is interested in developing and sharing resources that help libraries of all sizes manage and use data. orcid: 0000-0001-9198-7100. references [1] grolemund, garrett, and hadley wickham. “r for data science”. this title is available in print or online: http://r4ds.had.co.nz/introduction.html [2] the project counter website provides more information: https://www.projectcounter.org/ [3] https://www.projectcounter.org/code-of-practice-sections/usage-reports/ [4] grolemund and wickham. “r for data science” – http://r4ds.had.co.nz/introduction.html. this approach is also discussed in a lecture hadley wichham gave at reed college entitled “data science with r.” https://www.youtube.com/watch?v=k-ss_ag2k9e. [5] grolemund and wickham. “r for data science” – http://r4ds.had.co.nz/pipes.html. the vignette for the magrittr package also provides a helpful guide: https://cran.r-project.org/web/packages/magrittr/vignettes/magrittr.html [6] grolemund and wickham. “r for data science” – http://r4ds.had.co.nz/tidy-data.html. for a more detailed treatment, see: hadley wickham “tidy data.” the journal of statistical software, vol. 59, 2014 – https://www.jstatsoft.org/article/view/v059i10 subscribe to comments: for this article | for all articles one response to "using r and the tidyverse to generate library usage reports" please leave a response below: terry reese, 2018-02-06 thanks for the heads up tom. we’ve taken care of updating the link in the article. best, –tr leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – editorial: introspection as activism, or, getting our houses in order mission editorial committee process and structure code4lib issue 35, 2017-01-30 editorial: introspection as activism, or, getting our houses in order those of us in libraries like to trace our history to alexandria or to the french governmental system of record-keeping, but the construction of the modern glam world is far more recent, almost as new as coding. it has evolved almost as rapidly. and its future is on us, whether we choose to passively accept a status quo others build or to act and grow and develop ourselves and our workplaces. by ruth kitchin tillman this editorial could not have been written without stacie williams, angela galvan, andromeda yelton, april hathcock, rachel m. fleming, nina de jesus, eira tansey, cecily walker, sam winn, chris bourg, kate deibel, stephanie sendaula, alison macrina, allana mayer, tara robertson, anna-sophia zingarelli-sweet, jarrett drake, bergis jules, hillel arnold, jacob s. berg, and others who elide my memory as i sift through names of all my influences. through keynotes and blog posts and conference presentations and tweets, over coffee and continental breakfasts, you have shaped the parts of me which wrote the better parts of this editorial. i write it also with gratitude for the many others who have shaped the skills i may use to follow through on it and those kindred souls with whom i stand in mutual support while doing so. it is difficult to prepare an issue of the code4lib journal in light of the daily floods of news in the united states. those taking power threaten to eliminate programs which enable art and culture. we fear losing fundamental access to health care. we see doors closing and walls rising. we fear further destruction of the environment, the risks to clean water and clean air, or how rising sea levels and temperatures put towns and their associated cultural heritage at risk. we worry over how fragmentation in the eu will affect our professional collaboration or our livelihoods. we see the effects of years of disabling cuts to the british public library system. we don’t know what the shape of things will be. and in the middle of such uncertainty, how does one stop to focus long enough to edit an article about archival integrations or thesauri or oral history solution packs? how can we sit down at the keyboard and care about our projects and processes against such relentless and monumental risks? but i see in this last year and these last few weeks the material for a call which i intend to make here, a call for action by every library worker and in more countries than the united states. the action i am asking for is not taking to the streets, although that is a worthy effort. it is not taking to the phones, although that is a worthy action. it is something any library worker of any political persuasion can do, or at least can try. it is getting our own houses in order. it is assessing how well we live up to the fundamental principles of our profession. i will start from the close, the familiar, the code or at least code-adjacent. but while this is a code4lib journal editorial, our work is situated in the broader context of libraries, archives, museums, and galleries—in glam and in the world. the records we keep library workers espouse values of privacy, access, and respect and, in the best of times, we live up to these ideals. we have just cause to be proud of zoia horn, the first librarian to be jailed by the us government for refusing to testify about the activities of her patrons. when the fbi tried to recruit snitches in new york city libraries for their “library awareness program” (possibly also called “development of counterintelligence among librarians”), the library workers turned right around and told the new york library association, who told ala, who issued guidance not to collaborate. the attempts were exposed to the public and the people demanded hearings. when the patriot act sent federal agents again into libraries, looking for records, the workers fought back, even under the chilling effect of national security letters which left them unable to call for the profession’s support. these are the high points in our legacy. but between the highs, there are the lows i will address in “ways we treat each other,” and there are far more periods of mundane activity. there is administrivia and there is assessment. library workers love data. we love metadata (and data about our use of metadata), usage data, research data, big data… when we collect data about our patrons, it’s often with the best intentions. we want to design a better website. we want to compete with commercial services in offering relevant suggestions. we want to provide convenience. we want to do our jobs better and to prove our worth. angela galvan recently spoke about a practice she encountered at her institution where the marc 970 (local) field of a book requested for purchase contained the name of the person who had made the request. an administrative practice. a local field. yet, because that field existed, it could be used to track people who requested books on particular research topics—books the library willingly purchased, but books which a hostile administration might consider seditious. and, unlike borrower records which require a subpoena, the marc display[1] exposed it to anyone who knew enough about the system and cared enough to look. the very existence and retention of such data opened the institution to risk of subpoena for a comprehensive data dump. when angela discovered this, she didn’t shrug it off as currently not causing harm. she brought it to her administration, who approved immediate removal of the field from all records. and do borrower records require a subpoena? andromeda yelton’s 2016 libtechconf keynote walks the audience through wireshark, insecure http, and the ways in which patron data may get exposed through self-check machines, whether transmitting over wifi to a physically-hosted ils or across the cloud. hackers, and even hackers working to advance particular governments’ agendas are not a new phenomenon, but when our low expectations of vendors lead to bad security practices, it is on us if that data is harvested and used in any way to harm our patrons. dorothea salo has written on the topic of collecting the data in the first place at length and better than i will do here. anyone who has not read her on the subject should start with this quote: i want academic librarianship to feel uncomfortable about accumulating patron information behavior data, even anonymized, even in aggregate. i want that discomfort to cause us not to collect patron information behavior data at all without a clear need for it, to collect the scantiest data possible when it is needed, to guard that data well, and to throw it away like a hot potato as quickly as feasible to keep ourselves and others from the temptation to abuse it. and then read the entire linked article. it is hard. it challenges. but data collected with the best of intentions can be used to perpetrate great harm. and so we must ask ourselves as we do our work, whether we really need that cool map which uses ip addresses to show downloads occurring in real time and historically. times may come when we must ask whether we should even build the system. times may come when we need to undertake the act of destroying records, because not to do so would bring even greater harm. for more on privacy in libraries and how library workers may act to educate ourselves and our patrons, i strongly suggest visiting the library freedom project our partners and our role models the crux of the piece by angela galvan cited above wasn’t actually the marc field that she found. much of that story played out over twitter, although she references it in her talk. the talk, however, focused on issues of authority in libraries’ complex relationships with vendors. she calls on us to assess and reflect how much we give to vendors and how little we ask in return. when library workers speak of vendors, it’s as often with a sense of resignation and helplessness as it is of the utility of the service. why should libraries have to accept a self-check system which sends unencrypted patron data over the web? why do we allow vendors to retain patron data we discard? angela presents a series for steps we can take toward reclaiming our agency. there are the external partners we choose: prospective donors, organizational sources of funding, and the government who either directly funds our institution or may fund our work through grant-based initiatives. we may be tempted to focus on the alignment of mutual goals for positive actions while ignoring where their actions directly contradict our own ethical standards. when the ala published a press release about libraries’ overlap with the new administration’s stated agenda items, it set off a backlash from members who pointed out the ways in which other stated goals of the administration were in direct opposition to ala’s code of ethics. whether it’s choosing not to partner with a multinational corporation whose practices do not align with our values or risking government reprisals, library workers may experience negative consequences, from inability to fund a great program to the loss of jobs. it is on us to accept that these repercussions do not outweigh the injustice of collaboration. and then there are those whom we seek to emulate. in her libtechconf 2016 keynote, safiya noble addresses algorithms and ethics. too often, we desire to emulate google’s dominance of the search market. we want that same ease and relevance in our systems. yet what are we to make of the radicalization of dylan roof? should we implement summon’s topic explorer when a result for “rape in united states” brings up the wikipedia article “hearsay in united states law”? or “mental illness” “the myth of mental illness”? (matthew reidsma explores this at further length) will we fix our own imperfection with the same google imperfection that, more humorously, proudly displays “traffic collision” when i search for “death of the author barthes”? when we choose our role models, we must choose carefully and we must ask what we are not willing to accept. the ways we treat each other in the section on privacy, i said library workers have just cause to be proud. the bad news is that arguments from history also locate the library in institutional oppression, colonialism, and the destruction of indigenous ways of knowledge. for a fair number of us in libraries, jim crow policies existed within our or our parents’ lifetimes. to unveil how they affected the archives sector, alex h. poole traced researcher access to archives in the jim crow south and documented ways in which white archivists and professional organizations treated black researchers (and archives workers). such policies are within the recent and familial memory of our patrons, too. as he accepted a national book award, john lewis spoke of being denied access to the local library as a child, in the mid-1950s. we may dismiss such instances as part of our past, yet what of the common practice of including private police forces or off-duty officers in our libraries in the interest of “security”? in kansas city, library workers experienced pushback from their security about the content of a display, then off-duty officers arresting a patron against library worker wishes (and charged their director of public relations with interfering in an arrest). we must consider allegations that a bad $10 check for library fines led to a patron’s arrest. in other systems, fines over $10 or $20 go to collections agencies after a period, not necessarily exposing the patron to police violence but to harassing phone calls for money which they may not have. is this in any of our best interests? librarians (a word i choose here as differentiation from “library workers” throughout) are still overwhelmingly white, female, and middle class. librarians undertake diversity initiatives to support bringing people of color into the profession but these professionals become exhausted by the profession’s expectation that they will “perform whiteness,” downplaying difference and brushing off microaggressions in order to “fit in.”” these initiatives focus on bringing in those who can act enough “like us” to get through the door without addressing retention, mentoring, and the workplace shifts that will come as true diversity occurs. we white librarians have much work to do. it begins by listening to what people are willing to share about their experiences in libraries, but it requires intentionality and effort that is informed but not carried by the people who already fight for inclusion. we must confront our practices in creating precarious employment through grant funding. we see digital project positions posted for 2 years at 20 hours/week, or 1 year, or 1 year at 15 hours/week. young professionals are expected to leave behind support systems and fund cross-country moves chasing temporary jobs in the hopes they’ll finally find a permanent position. besides being a byproduct of institutional funding cuts, this practice reflects our desires to jump on the new and “shiny” instead of focusing our resources and respect on the critical maintenance work necessary for a healthy long haul. although the site’s last post was in 2014, we are still eating our young. we must exercise care when creating new projects, and particularly temporary positions, to ask what we are expecting of the person we intend to hire, how we intend to commit to that person’s growth as an individual and a professional. there is also the devaluation of the “non-professional” library worker, the other-than-librarian. emily drabinski addresses the concept of “professionalism,” how it unites and isolates, both within librarianship and among library workers as a whole. in order for some people to be professionals, other people must be nonprofessionals and excluded from the circle of privilege that professionalization affords. in the world of libtech, the distinctions between librarians, library-coders, and librarian-coders, are known sources of stress and disrespect. such strains also exist between librarians and paraprofessional library workers, including paraprofessionally-classed workers who hold mlses but, because of job markets, remain in or can only find non-“professional” positions. how to ensure respect for the experience and value a person without an mls brings to a library? how to show sufficient respect for a person’s real skills and need to earn a living while also wanting that person to have a job which requires the skills of their mls? respect and care for our coworkers should be at the foundation, but solid answers on what this looks like in hiring practices and day-to-day hierarchies still often elude us. in talking about how we treat each other, the keynote stacie williams delivered at the digital library federation forum 2016 provides us with our grounding and also our path forward. all labor is local, she reminds us, and centers us in the care-based labor which our society so regularly devalues. whatever it is that we will get through, we will get through it by centering ourselves in our communities, beyond our own doors. by caring for each other and for those we serve. by choosing to value our patrons’ privacy as a way of valuing their humanity and their inquiry. by valuing coworkers’ contributions even if we’d have done it another way. by listening to each other and building the code for the people, rather than forcing the people into the code.[2] affirming our humanity in code and beyond and i will say a word for the archival integrations and thesauri and oral history solution packs. because besides providing us with direction for our daily labor and stimulation for our problem-solving brains, they all support the good which we can accomplish in the glam sector. consider oral histories. visit storycorps and listen to people from over the united states sharing experiences that mattered to them. start with the highlighted stories from the civil rights era. then listen to stories collected in the righting the record oral history project. these reflect a variety of experiences and identities which are traditionally not centered or lifted up. they make explicit that the struggle for black liberation continues today. for some, this is not news. for others, especially those of us who are white, these oral histories open a window into a world we have never known and experienced, a challenge for action as well as a reminder of others’ fundamental humanity and how some attempt to deny it. at their best, glam institutions offer space for stories, personal and fictional and the mix of personal-fictional in which so many of us live. they offer a place for rest, escape, and respite for the weary activist, the student whose school reflects institutionalized racism, the veteran experiencing homelessness. they offer art—whether hanging on the wall or tucked away in books or digitized and “up” on the web in a gallery. they offer histories, of people who have prospered and of people who have experienced suffering. they offer us the opportunity to see the overlap and the places where we may seek inspiration from the past or resolve not to repeat mistakes. they offer the chance to create and discover—whether art, science, mathematics, literature—at their best, they give people the support they need for what they want to make in this world. we like to trace our history to alexandria or to the french governmental system of recordkeeping, but the construction of the modern glam world is far more recent, almost as new as coding. it has evolved almost as rapidly. and its future is on us, whether we choose to passively accept a status quo others build or to act and grow and develop ourselves and our workplaces.[3] just as we discover and develop new code bases alongside others, we discover and develop new and better ways to be alongside others in our institutions. as we go into our workplaces, we must remain mindful of the value and humanity of our coworkers, of the nature of the systems we create and their effects on the lives of those who use them, of how the data we collect represents real humans, and of the ways that we retain or lose agency in the contracts that we sign. we must strive to make our work a place in which we may grow in community, whether with those who walk in the door or those on slack five hundred miles away. we will get it wrong. and when we get it right, we will not always succeed in our bids to administration to effect the change. but we must do it anyway. if, after reading this, you want it cosmically darker and deeper, i recommend bethany nowviskie’s remarkable keynote “digital humanities in the anthropocene.” acknowledgements i’m deeply grateful to kate crowe, eira tansey, hillel arnold, christy tomecek, and sam winn for reading through drafts of this text in that order and advising me on ways i could make it better and to carol bean for proofing the final draft. i would also like to particularly thank kate and share the advice she gave me earlier this year: that we can make use of every commitment we make as part of advancing our careers to bring about the changes we wish to see in our profession and our world. notes [1] a personal note, please don’t ever get rid of the function of the public marc display as it is so valuable to those of us who’ve had reasons to use it. [2] this had already gotten very long before i started on accessibility. i have so much to say on the subject of accessibility that i finally had to redact it. there is still much work to do, particularly for us code4lib folks. i will say—listen to kate deibel when it comes to accessibility. and listen to disabled people. [3] on this subject, i am acutely aware of how precarity and workplace climate force some to be silent as an act of self-protection. a staffer in a precarious situation may risk harm when speaking up to an administration and see no actions taken. at a previous place of employment, i saw more senior coworkers physically harmed by policies about which we had objected to the director and all i could do was be there with them in that time. but where you have the agency, use it. and where you have a spark of light, share it. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – adaptation: the continuing evolution of the new york public library’s digital design system mission editorial committee process and structure code4lib issue 41, 2018-08-09 adaptation: the continuing evolution of the new york public library’s digital design system a design system is crucial for sustaining both the continuity and the advancement of a website’s design. but it’s hard to create such a system when content, technology, and staff are constantly changing. this is the situation faced by the digital team at the new york public library. when those are the conditions of the problem, the design system needs to be modular, distributed, and standardized, so that it can withstand constant change and provide a reliable foundation. nypl’s design system has gone through three major iterations, each a step towards the best way to manage design principles across an abundance of heterogeneous content and many contributors who brought different skills to the team and department at different times. starting from an abstracted framework that provided a template for future systems, then a specific component system for a new project, and finally a system of interoperable components and layouts, nypl’s digital team continues to grow and adapt its digital design resource. by jay l. anderson & edwin guzman introduction the new york public library’s (nypl) design website system, nypl design toolkit, was built by its in-house team over the past two years, but is only the latest in a much longer collaborative process. the development and implementation of nypl’s current digital style guide is the result of many elements, many changes, and many contributors who brought different skills to the team and department at different times. sometimes those elements came from a team member’s experimentation; sometimes they were inspired by existing frameworks. ultimately this project’s value is due to continuous collaboration. growth and change: a content problem nypl.org is a complex resource which serves multiple purposes and multiple audiences. nypl patrons include a wide range of identities and personas, including high school students, parents of beginning readers, professional researchers, job seekers, all reading in a variety of languages. 91 branches are served by nypl.org, including the 4 research libraries. patrons visit the site for information from local branch hours and catalog access to original content and library events. in order to manage the complexity of the site, nypl.org underwent a major overhaul in 2009 that included migrating existing content into a modern content management system (cms). we adopted drupal as our new cms because we wanted to give our staff a tool to control their own content, as well as the ability to blog for the first time. with this new emphasis on staff expertise and exposure of the library’s offerings, the amount of content produced grew quickly. allowing the content to grow organically encouraged content creators to discover new avenues of interest, but it also allowed the site’s visual treatment to become inconsistent. this was confusing to the user and detrimental to the dissemination of the library’s updated branding, which was launched in 2009 by our communications & marketing department. the organization had no ux design documentation at this time; in fact, the term “ux” was new in the web design industry as well as at the library. as a design team, we were just beginning to have conversations that would evolve into standards and best practices. we had a set of guidelines for using the new logo, and some rules regarding usage of our new color palette, but we didn’t have anything that governed or documented the website look and feel. growth and change: a people problem the team that was responsible for the design and code of nypl.org has changed continuously since the 2009 drupal migration. while some of this change was enacted to respond to evolving needs (both of the site’s content and the technology needed to create it), some changes created challenges. we had a multitude of projects and a small, shifting team. the formation of the nypl labs team in 2011 contributed more capacity for the development of digital properties, and the teams came together and diverged as needed to cover the breadth of those properties. however, the most successful projects, such as the building inspector and nyc space/time directory, were those whose scope supported a short timeline and a singular purpose. design concepts that exceeded a certain scope were not able to mature to their full potential; this included the creation of a unified design system. this kind of change may seem very familiar to other institutions; the shifting of resources and priorities is inevitable in a large multi-audience non-profit organization. add to that the nature of digital growth, such as the evolution of available technologies, and an institution’s digital creative team is hard pressed to maintain consistency. when those are the conditions of the problem, the solution needs to be modular, distributed, and standardized, so that it can withstand constant change and provide a reliable foundation. evolving the design system nypl’s current design system, the design toolkit, has been built to support not only modularity, but a potentially wide distribution of ownership. its modular design means the different user interface components can be added and removed as needed, and its distributed structure means that multiple developers can own and affect each piece. this code-based design system is built on web standards and open source principles, which makes the modular setup easier to sustain. the development of this resource is a story of trial and error, in which the team learned how to put the right people in the right roles, how to make decisions that became best practices, and how to spend as much time building useful code as possible. this process showed our team, over several years, how to build a design system that not only produces design but adapts to change. the first generation: nyplbase design toolkit evolved from other design system projects starting as far back as 2010. the first design system, called nyplbase, was created by the team’s first ux designer, jay anderson. initially, anderson experimented with twitter bootstrap‘s pre-built framework to create a style guide and pattern library for nypl.org. ultimately, inuitcss, an architecture-only framework by harry roberts, was chosen for the underlying foundation of nyplbase, and anderson wrote the sass that contained the styling for the design patterns and branding elements. because of the amount of customization required by a multi-functional site like nypl.org, it was more efficient to write the styling from scratch than to overwrite default styling choices made by a full use interface (ui) framework like bootstrap. nyplbase was used to build an updated locations section of the site, as well as research divisions, a filterable listing of scholarly collections across the four research libraries, and staff profiles, a list of prominent nypl subject matter experts. these projects provided the first test of our new design system against real ux situations. usage of the applications over time revealed the need for significant updates, including better accessibility practices. because anderson, the original designer, was no longer with the team, updates proved difficult to incorporate. while the influence of a single developer had kept the designs from becoming inconsistent, the design system needed to be distributed among teammates in order to grow. a second start: ethyl the next design system was a components guide built with the sc5 style guide generator and called ethyl, built by ricardo galvez. ethyl was created as the style guide that would house the new user interface designs and patterns that were built in collaboration with an outside firm. (this collaboration was initially set up for a redesign of the homepage, with the plans for the rest of the website later in the planning phase, but did not continue after the initial release). ethyl contained the branding elements and additional code implementation documentation for the ui components. supplying code snippets for ui components is a practice that we kept from the nyplbase structure. code snippets help the developers on the team examine how components should be written in the dom with proper class names, which in turn helps them quickly build components from those samples. this was the pattern used when developers began to build ui components in react, the main javascript library we used for our projects. figure 1. sub menu code snippet found in ethyl http://ethyl.nypl.org/section/4.4 (enlarge) the sub menu element, for example, had a corresponding component built in react and used across a few projects. since the dom was decided and defined in ethyl, the developers were able to confidently use one component in multiple places and know that the design pattern was being used correctly. this practice of building applications with more modular components became more common for the team. in order to extend the modularity for these react components, we began publishing those components on the node package manager (npm). for example, ethyl included original versions of our svg icon set. we modularized those icons by creating an equivalent react svg icon set, dgx-svg-icons, which developers could import into their applications through the npm. we detached discrete pieces of code (such as the svg icons and the javascript libraries) wherever possible, and we allowed multiple developers to update those pieces. this approach also allowed developers to run `npm install –save @nypl/dgx-svg-icons` on the command line and know that they were getting the latest version. today’s solution: design toolkit in august 2016, the team began work on the shared collection catalog project. this discovery system allows users to find materials from the recap shared collection, an initiative from the new york public library, columbia university, and princeton university. the basic wireframes and mockups built for this project were collected into a resource called discovery designs (originally created by former team member brian foo), which evolved into nypl design toolkit. the design toolkit was created by mauricio giraldo, former designer and developer at nypl, and was initially implemented for the shared collection catalog. ethyl had been created to support a single project, and giraldo noticed that the design toolkit, though also created to support a specific project, would need a structure that would better support reuse by designers and developers alike. the structure of the design toolkit, similar to nyplbase, provides pre-styled, pre-coded ui components for creating applications. developers import the design toolkit into projects, write the dom structure with the appropriate css classes, and produce components and patterns that adhere to nypl design standards. component-based design the first section of the design toolkit documentation describes the modular pieces that make up the entirety of the project. the documentation for each ui element (such as forms, buttons, and icons) is abstracted to dom element patterns, and includes a description on how to use them. modularity is built into the css structure at the broadest and narrowest definition of each component; there are high level grid classes to construct the layout, and lower level classes for granular html elements. this approach helps the developers construct forms, for example, that can have any amount of input fields but all have the same theme regardless of the type of input. for example, the `nypl-name-field` class wraps the label and input elements and cascades styling to both elements. by abstracting the styles away from the dom, ui developers can focus on composing the application rather than designing components. figure 2. name field class (https://nypl.github.io/design-toolkit/sections/forms.html#namefields) cascades styling to child elements (enlarge) interoperable is accessible nypl projects are built on dozens of platforms, including drupal, react, angularjs, and ruby on rails. for this design system to be useful beyond its application in a single project, it had to be completely platform-agnostic. the only tool constraint built into design toolkit is the use of sass as its css preprocessor. otherwise design toolkit uses standard html for accessibility. some of the design patterns do use javascript for functionality, but that usage is equally modular. for example, the collapsible field sets elements depend on css classes and javascript for functionality. for ease of use across different projects, jquery was chosen to implement this component in the design toolkit. this does not mean, however, that this or any other component needs to be implemented in jquery. figure 3. collapsible field set (https://nypl.github.io/design-toolkit/sections/forms.html#collapsiblefieldsets) : display and code (enlarge) in the shared collection catalog, components’ functionality is handled by reactjs. adding and removing classes through react is simple and allows us to update styles for components dynamically. for example, we use a react prop called `style` to dictate the class name for a form element. regardless of whether react, jquery, or regular javascript is used, it is a helper but not a requirement. figure 4. form element with javascript helper (enlarge) benefits and challenges: components vs. layouts design toolkit’s inclusion of example layouts is a practice that started during the development of the discovery designs, and continued as the team developed the shared collection catalog workflow. giraldo, as the designer on the project, designed components as they were needed by copying output code from the project in development, and refining the component designs in his own environment. when a component was ready, he passed it back to the developer for further refinement, and also contributed it to the design toolkit. this close collaboration between designer and developer, fueled by solving project-based problems, allowed the contributors to learn along the way. full mockups were also created using ui components found in the design toolkit to quickly build proofs of concept for stakeholders and managers. these mockups also gave developers an idea of the user experience expected from their finished components and complete product. figure 5. full page mockup of the discovery search results page (enlarge) this approach worked well except when small components and full pages were built side-by-side. small component development can be lost in the holistic thinking required by full page design. essentially, we had created a dependency in our workflow. building the design and the implementation at the same time (done due to time and resource constraints) meant that we needed to carefully coordinate moving new css rules from the production implementation to the design toolkit, instead of the other way around. for example, a set of components would be built into a full page design. when it was implemented in react, that implementation would need to be broken down by a developer into smaller ui components and style rules. later, refactoring was needed for the full-page design to conform to the modularity of the developer’s components, for use in other projects. benefits and challenges: delivery systems another dilemma we encountered was how to best import the design toolkit into projects. we could have imported it through a content delivery network (cdn) like nyplbase, but we hesitated to add an external dependency and an http request. we decided to host it on nypl’s npm account as a package. for our nodejs and webpack projects, developers can run `npm install –save @nypl/design-toolkit` in the command line at the project’s root folder and get the latest version of the design toolkit. (we also created design toolkit as a gem for use in ruby-based projects.) but in the angularjs applications, such as locations which uses the original nyplbase patterns, we had to import a static file from an nypl cdn endpoint. this meant that developers had to keep close track of the current working version of nyplbase and if there were active changes and updates in nyplbase. the design toolkit is currently nypl’s centralized location for ui components, full page designs, development and accessibility documentation, and other resources and tools. it documents the extensive accessibility knowledge the team gained through development experience. the work on design toolkit’s color accessibility table, for example, helped us fully recognize why some color combinations in our palette are not accessible. other resources are more generalized, such as the validation tools that are used in the qa check workflow. this makes the design toolkit not only a design pattern resource but also a use case for building with accessibility testing. conclusion: the way forward since the inception of nyplbase in 2010 through to nypl design toolkit today, our team has learned that to build a reliable design system, you need to adapt to changes, and to distribute ownership as much as possible. lessons we learned included making sure the roles and responsibilities of the team members are assigned correctly. it was a ux designer that started the first nypl.org design system effort, because her connection to both content and design allowed her to see the full picture of what was needed, even if she couldn’t code every piece. but it was the later expansion of the project to include both designers and developers that allowed the project to grow. with this in mind, a team wishing to build their own design system would do well to assess the skills of the team members and use them to their greatest potential. developers will likely do most of the coding, since they will be most familiar with the nuances of your institution’s tech stack. but team members with less advanced code skills, like certain ux roles or content strategists, are crucial at evaluating pattern usage, design completion and flexibility, and branding. find out what each team member does naturally: that is the work they will do most readily. it’s also best to plan the building of your design system so that it fits into your regular project workflow as much as possible. if a contributor has to stop and remember a different process in order to commit a change to the design system, it is less likely those changes will get updated efficiently. also, posting a “cheat sheet” for the workflow someplace easy to access will save time and resources getting a new contributor up to speed or refreshing the memory of a team member. finally, testing real projects also tests the design system. the development of the design toolkit was tied to a real project, which helped it grow as the more abstract approach of nyplbase did not. real world usage and regular user testing will reveal gaps in the design system’s flexibility, while testing the code will reveal opportunities to streamline or refactor. the value of this effort wasn’t that our team got it right the first time or invented something new; it was that we learned, adapted best practices, and helped each other to a common goal, even across shifts in personnel in the department. as the department underwent changes, so too did the workflow and the technology. the team members, past and present, who worked to create style guides were willing to try new approaches and toss out what wasn’t working. this willingness to adapt, along with a structure that allows the interoperability of separate pieces, allowed the team to build a reliable resource that can still grow. about the authors jay l. anderson is a senior ux designer at the college board. jay was formerly the senior ux designer for the new york public library. http://memorabilist.com/ edwin guzman is a senior applications developer at the new york public library. http://www.edwinguzman.com subscribe to comments: for this article | for all articles one response to "adaptation: the continuing evolution of the new york public library’s digital design system" please leave a response below: andrew darby, 2018-08-17 note: this article was modified after publication to include the name of the ethyl developer. leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – quick lookup laptops in the library: leveraging linux with a slax livecd mission editorial committee process and structure code4lib issue 2, 2008-03-24 quick lookup laptops in the library: leveraging linux with a slax livecd this article discusses the experiences over the past year at laurentian university in deploying a linux live cd solution to create customized kiosk browsers on ‘quick lookup’ laptops. this provides security, reliability, ease of use, and improved response time compared to other options. we explain why we chose this solution, reveal techniques we used to achieve some of our goals, and reflect on possible future iterations of this project. by dan scott and kevin beswick background laurentian university is a bilingual university with over 9,000 students located in sudbury, ontario. complementing the learning commons located within its walls, the j.n. desmarais library provides two additional classes of computers for its patrons: a set of 16 research workstations at traditional tables located near the reference desk for easy access to reference assistance; and a set of 20 laptops mounted at standing height throughout the stacks to provide quick lookup functions including call number retrieval for known items, reserve item access, and account access. the laptops had been purchased in 2003 in conjunction with a new library system to replace the vax terminals used with the previous library system. as the old vax terminals did not use cat5 cable, the decision was made to rely on the laptops’ wireless connectivity rather than running 20 new cat5 connections that would have cost approximately $10,000 cdn. problems the laptops handled the job for years, but by 2006 some problems were beginning to surface. the first problem was that the actual user experience made a mockery of the term “quick lookup laptops”, because the hardware was overly taxed by the configuration of the operating system and software that were installed on it. although the laptops had only 256 mb of ram, they were running windows xp, an antivirus program, windows defender anti-spyware, and a groupwise client. additional programs that had been added to the laptops since their initial deployment, such as microsoft office and scifinder scholar, were virtually unusable due to the taxing memory requirements that resulted in continual paging to virtual memory on disk. a second problem was that the laptop images required a significant investment of ongoing maintenance effort. at first, the new systems librarian tried to maintain the individual laptop images where they were mounted: applying manual upgrades where automated upgrades were not available or were not working; tweaking operating system security settings and profile settings; and running hard drive checks on the aging laptop hard drives. despite limiting users’ privileges, it was not uncommon to find that a game had been installed on the laptop by simply unzipping a set of files onto the desktop. the manual approach to administration was quickly abandoned in favour of a reimaging process relying on the use of the open-source partimage imaging utility. a third problem was that the laptop images did little to protect the privacy of the individual user. by default, the laptops were logged into a local windows xp account. if a user saved files to the hard drive, subsequent users could read those files. if browser sessions were left open (which was typical, because starting a new browser took so long), subsequent users could peruse the browser history and access cached pages. users had the ability to log into their own campus accounts via these laptops, but when they chose this option they also frequently forgot to log out after finding the resource they needed. for historical reasons, the maintenance of these laptops was the direct responsibility of the library rather than the campus information technology (it) department. the library has one systems librarian responsible for all library hardware and software, including the ils, and until the fall of 2007 no additional resources were available to assist the systems librarian, so the solution to these problems had to be relatively inexpensive in terms of human resources. proposal over the course of the summer, the systems librarian investigated several possible means of alleviating the problems experienced with the quick lookup laptops. purchasing more ram was considered and quickly rejected, as the age of the laptops meant that the capacity was limited to a total of 512 mb of ram and the cost would be prohibitive for nearly obsolete technology. continuing the existing approach of reimaging the laptops with new windows xp images was ruled out as an option because the performance problems suffered by the patrons would not be resolved and, at a cost of approximately 20 hours to roll out a new image, there were more pressing demands for the systems librarian’s time. asking the campus it department to assume maintenance of the laptop images was rejected as the it department was already months behind in two other library projects on behalf of the library and also faced a shortage of skilled human resources. therefore, in the fall of 2006, the systems librarian brought forward the following proposal to the library: we would deliver a more secure quick lookup laptop with better performance for our users under the existing memory constraints. to achieve this goal, we would replace windows xp with a simple linux desktop, with an estimated development and deployment investment of 30 hours. the systems librarian was given the go-ahead and challenged to deploy the initial solution by the end of 2006. implementation phase i: evaluating alternative image delivery approaches the laptops had originally been deployed with the batteries and cd-rom drives removed. when we retrieved the drives from storage, we discovered that over the years a number of the cd-rom drives had been lost (“inventory shrinkage”), so we were forced to pursue two approaches to loading linux on the laptops. we looked upon this as a good opportunity to compare approaches and learn from the experience. for those laptops that had cd-rom drives, we opted to take a linux live cd and customize it for deployment as a kiosk browser. a linux live cd is a linux distribution that runs directly from a cd-rom (or other media) rather than having to be installed on a hard drive. after investigating a number of candidate live cds, we decided on the slax popcorn edition for the ease with which it could be customized, its minimalist xfce desktop, and its support for our laptop hardware. we found an excellent tutorial for customizing slax live cds that we were able to rely heavily on for our initial customization efforts. recognizing that we would be going through a number of iterations of the image, we burned the images on cd-rw discs to be environmentally friendly. for those laptops without cd-rom drives, we built a “linux kiosk” image based on ubuntu 6.10. we installed additional xfce packages so that we could reproduce a similar desktop look and feel as the live cd images. these images were installed directly on the laptop hard drives using the now-familiar partimage. quality slax live cd ubuntu installed image advantage performance as only a minimal number of processes run in a live cd, most system resources are devoted to the display and the browser session, resulting in excellent performance for a five-year old laptop. services like updatedb were disabled, so despite a few additional processes over and above what was required for the live cd, the performance was largely comparable to the live cd. equal privacy in a worst-case scenario, rebooting the laptop means that all data would be wiped clean. logging out of a session shut down the laptop, forcing a wipe of all data. files written to disk survive reboots, posing a threat to privacy. we developed a shell script that used rsync with the –delete option to restore the user’s home directory to a known set of files on login as part of the profile. edge goes to slax security when package upgrades are available, they can simply be copied to to the live cd and a new set of cd-rws burned and deployed. however, while popular packages like firefox were upgraded quickly, many core system packages did not receive upgrades. once the image was installed, ubuntu’s apt-get update and apt-get upgrade can be run as a cron job to deliver security updates for all packages. ubuntu speed of deployment rolling out an updated image takes approximately 5 minutes to burn each disc at 12x write speed. we can burn a stack of discs in less than two hours, then simply swap the discs in the laptops out on the floor and reboot to deploy the image. rolling out an updated image took approximately 1 hour per laptop, and as we rely on wireless for connectivity, we had to bring each laptop physically into the office for re-imaging rather than being able to rely on bootp to start a remote re-imaging session over slow wireless. slax robustness in the event of a hard reboot (for example, the power cord being unplugged so that a student can provide power to their own laptop), the system always recovers gracefully. in the event of a hard reboot, the system runs a filesystem check. we found that the ext3 filesystem seemed prone to errors that eventually required human intervention (possibly due to the age of the hard drives). in this respect, the old windows xp images had been much more robust than the ubuntu images. slax we were able to complete the work required to build both images and deploy the images onto the 20 quick lookup laptops in approximately 40 hours in december, 2006. this was slightly more than the estimated time, but was considered reasonable given the unexpected requirement to build two images. phase ii: evaluation during january, 2007, the systems librarian evaluated the success of the newly deployed linux images on the laptops by soliciting feedback in the library newsletter that was emailed to all university students, staff, and faculty; observing users directly; and approaching users to help those having difficulty. the most common complaint with the live cd images was that when a person shut down a session, it shut down the computer rather than automatically starting a new session. the most common complaint with the installed images was that the system was unusable due to the filesystem errors preventing a reboot. a secondary complaint was that many users accustomed to the internet explorer icon could not figure out how to start a browser session despite the desktop consisting of only two widgets: a clock, and a firefox icon. no users complained about the loss of the additional functionality that had been offered in the windows xp images, supporting our belief that most users did not have the patience to try to use those applications in such a resource-constrained environment. due to the instability of the installed images, we opted to purchase additional cd-rom drives for the remaining laptops and to develop an improved live cd that better met users’ requirements. the improvements consisted of: a cron job that automatically starts a new browser session if it detects that no browser session is current running, thereby avoiding the need for a user to figure out how to start a browser session changing the system “shutdown” command to a “reboot” command so that if a user does inadvertently exit a session, within a few minutes the laptop will automatically be ready for the next user rather than requiring the user to power on the system and wait for the live cd to load a customized firefox profile that sets the default home page to the library web site, includes a libx toolbar, paranoid privacy settings so that all cookies, cached files, memorized passwords, and the like are deleted when the browser session ends, and a browser cache reduced to 1mb to avoid using too much system memory adding an ntp module to ensure that the desktop clock was accurate these modifications were trivially easy to accomplish with slax. by placing files in the /rootcopy/ directory of the slax image, or any subdirectory mirroring the normal file system hierarchy, the distribution automatically copies those files over the corresponding file (if any) in the real file system hierarchy at boot time. customized /etc/x11/xinit/xinitrc for our preferred desktop most of the customization reflects a paring-down of the defaults to only include those widgets that we require; to set up our firefox browser profile and launch the browser; and to reboot the system rather than shut down so that it is ready for the next user. we extract the firefox profile from a tarball rather than copying each file from rootcopy for the sake of simplicity. # set up the desktop to our liking cd /root # restores xfce desktop menu & configuration tar xzf /etc/xfce4_config.tar.gz # this is only necessary when running w/o xfce4-session xsetroot -solid black -cursor_name watch # or use old-fashioned startup script otherwise xfce-mcs-manager xfwm4 --daemon xftaskbar4 & xfdesktop & # restore firefox profile settings tar xzf /etc/firefox_profile.tar.gz # install libx extension /usr/bin/firefox --install-global-extension /etc/libx-lul.xpi # start firefox /usr/bin/firefox -fullscreen http://www.laurentian.ca/laurentian/home/departments/library/library+home.htm & # customizing /etc/rc.d/rc.slax to reboot rather than shutdown if [ "$autocmd" != "" ]; then echo "starting autoexec command: su --login -c \"$autocmd\"" su --login -c "$autocmd" # telinit 0 # we want to reboot (init level 6) rather than shutdown (init level 0) telinit 6 fi perl script in /root/check_browser for starting a new firefox session if none exists in retrospect, perl is probably overkill for this script – but it was easy to write and the system demands do not seem to be too onerous for a script that runs every minute. >#!/bin/perl use strict; use warnings; my @procs = `ps wax`;< my $ffox = 0; foreach my $process (@procs) { next unless $process =~ m/firefox/;< $ffox = 1; } if ($ffox == 0) { system('firefox &'); } cron job in /var/spool/cron/crontabs/root when the slax rootcopy process overwrites system files, you have to ensure that the permissions and ownership of the files in the /rootcopy/ directory match your destination. for example, the following cron job file must be owned by root. # check to see if firefox is running every 60 seconds * * * * * display=:0.0 /usr/bin/perl /root/check_browser 1> /dev/null 2> /dev/null screen shot of the laurentian library linux live cd in action phase iii: refine offering for 2007-2008 academic year as laurentian university is an officially bilingual (english and french) institution, users need to be able to search for resources in both languages. the initial deployment of the laptops had set the keyboards to a plain english (united states) layout and provided no means of entering french characters. this problem had to be addressed for the 2007-2008 academic year. we began by investigating methods that could be used to switch the keyboard layout in slax. we had the option of setting the language at boot time, but this method was deemed unacceptable because the user would have to wait while the laptop was rebooted to use the different keyboard layout – if they were even capable of figuring out that the language could be switched by rebooting. we subsequently discovered the standard x command – setxkbmap – which changes the keyboard to a selected layout, and recognized that this could be the basis for our solution. for our interface, we wanted to use affordances that would be familiar to the users. the research workstations in the library, which run windows xp, use a language switching icon in the taskbar to toggle the keyboard between french and english layouts. the icon is stateful and shows the language that is currently being used (en for english, fr for french). when the icon is clicked, the language switches to the next configured language. we wanted to offer a similar, familiar mechanism on the laptops to our users. fortunately, as part of its standard desktop icons, xfce offers a launcher widget called a “two-state launcher” with an ‘on’ and an ‘off’ state. after consulting some documentation, and the forums found on the slax website, we determined that the launcher is controlled by a shell script that returns a 1 for the ‘on’ state, and 0 for ‘off’. we wrote a script that, when the icon is clicked, checks which keyboard layout is currently being used, switches the keyboard layout to the opposite language using the setxkbmap command, and then returns the opposite state depending on which language is currently in use. this script, paired up with the double state launcher, provided a perfect language switching function for the laptops. script for toggling keyboard layouts – keyboardswitch.sh: #!/bin/bash #shell script to toggle between keyboard languages in # xfce using the two-state launcher state=`/usr/x11r6/bin/setxkbmap -v | grep -o \+us` if [ "$1" = "0" ]; then /usr/x11r6/bin/setxkbmap -layout ca; echo "1" elif [ "$1" = "1" ]; then /usr/x11r6/bin/setxkbmap -layout us; echo "0" else if [ "$state" = "+us" ]; then echo "0" else echo "1" fi fi we made a pair of en and fr icons in .png format using the gimp and set these icons to appear on the two-state launcher widget. the script and buttons can easily be modified to offer any keyboard layout. one simply needs to change the keyboard layouts that the script passes as arguments to the setxkbmap command, and to create corresponding 100 pixel square icons to represent the language names. the other modification we made to the lookup laptops in the fall of 2007 was to associate mime types to their respective slax applications to extend the use of the laptops slightly beyond quick catalogue searches. for example, we observed that users often search for an article in the library databases and want to preview it quickly to see if it meets their needs. we added the xpdf application and associated the application/x-pdf mime type with xpdf so that users can view the pdf on the lookup laptops. similarly, we associated microsoft word documents with the abiword application. although we initially attempted to set these preferences in the firefox global preferences files, we found that we instead had to edit the settings in the firefox browser, then create a tarball of the firefox profile to include in the slax rootcopy directory to be extracted at boot time. reflections the use of a customized linux live cd for the quick lookup laptops at laurentian university has been a success. we have achieved our goals of increased performance, security, privacy, and robustness over the previous solution without compromising the ease of use, despite changing from windows to linux. the project has been both a financial success, in that we have been able to defer the purchase of new laptops for at least the 2007-2008 academic year, and an environmental success, as we avoided having to dispose prematurely of the old laptops. however, there are some risks and concerns that we will have to address as the project continues to mature. one risk in depending heavily on a small linux distribution is that it can stop development or disappear relatively quickly. in november, 2007, the project home for slax 5 was replaced by a blog announcing that slax 5 was dead and discussing the slow progress towards the next versions of slax. fortunately, a slax 5 site mirror continues to be available and to publish updated modules, so new security vulnerabilities found in packages like firefox can continue to be addressed until the new versions of slax are released. for maximum hardware compatibility, slax live cd runs x in a generic vga framebuffer, rather than using the native driver for the video card. the downside to this approach is that power management for the display is not supported through the standard “xset dpms 300 600 900” command, so our laptops currently use more power than is required. initial attempts to address this problem with slax 5 have been unsuccessful, but we are optimistic that hardware support in future versions of slax, or in alternative live cd distributions, will improve this situation. as our laptops physically deteriorate, we will purchase newer machines that, even at a third of the cost of the original laptops, are more powerful than the research workstations that we deployed just a year ago. we will have to decide whether to maintain the minimalist profile of the laptops to satisfy a few specific use cases despite their increased power, or whether we shall expand the functionality of the hardware, thus increasing their potential utility but also increasing the potential confusion of the interface. we may opt to purchase more conventional desktop workstations with separate monitors and keyboards; while laptops in 2003 offered a net savings in purchase price over a desktop + monitor + wireless card, the maintenance cost is substantially higher once the hardware is past warranty. for example, repairing a single missing key on our current laptops requires a replacement of the entire keyboard, at a cost of approximately $150. the challenge to deploying conventional desktops becomes a problem of physical security, as the cd drive would be exposed and malicious users could conceivably replace the standard cd image with their own customized cd. if we add wired ethernet ports, we could entertain the possibility of running thin clients that connect to a linux terminal server rather than running independent copies of the operating system. however, as the estimated cost for 20 ports remains at $10,000, this possibility seems unlikely. it is not clear that there would be a sufficient return on investment in learning and deploying a new thin client system. about the authors dan scott. dan has been the systems librarian for laurentian university since 2006, and has been a library geek even longer than he has been an open source geek. you can reach dan via email at dscott@laurentian.ca and follow his blog at http://coffeecode.net. kevin beswick. kevin is a third year computer science student at laurentian university. he enjoys computer programming and playing various musical instruments. you can reach kevin via email at kx_beswick@laurentian.ca. subscribe to comments: for this article | for all articles 2 responses to "quick lookup laptops in the library: leveraging linux with a slax livecd" please leave a response below: richad vizor, 2013-06-27 hallo. i just read your article and out of interest would like to know if you are still using slax. version 7 is now out and is being maintained by thomas m with backing from ttwo gerrman sponsors. kind regards richard vizor united kingdom dan scott, 2013-06-27 hi richard: i’m glad to hear that slax 7 is out! at this time, however, we are no longer running linux-based laptops in the library. as the older laptops were eventually refreshed, there was a concurrent desire to delegate the maintenance effort to our campus it services so that we could focus our own resources on more library-specific problems. the laptops in our library are now running windows xp with deep freeze. leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – are games a viable solution to crowdsourcing improvements to faulty ocr? – the purposeful gaming and bhl experience mission editorial committee process and structure code4lib issue 33, 2016-07-19 are games a viable solution to crowdsourcing improvements to faulty ocr? – the purposeful gaming and bhl experience the missouri botanical garden and partners from dartmouth, harvard, the new york botanical garden, and cornell recently wrapped up a project funded by imls called purposeful gaming and bhl: engaging the public in improving and enhancing access to digital texts (http://biodivlib.wikispaces.com/purposeful+gaming). the goals of the project were to significantly improve access to digital texts through the applicability of purposeful gaming for the completion of data enhancement tasks needed for content found within the biodiversity heritage library (bhl). this article will share our approach in terms of game design choices and the use of algorithms for verifying the quality of inputs from players as well as challenges related to transcriptions and marketing. we will conclude by giving an answer to the question of whether games are a successful tool for analyzing and improving digital outputs from ocr and whether we recommend their uptake by libraries and other cultural heritage institutions. by max j. seidman, dr. mary flanagan, trish rose-sandler, and mike lichtenberg introduction purposeful gaming and bhl: engaging the public in improving and enhancing access to digital texts was a project which ran from december 1, 2013 through november 30, 2015.  funded by the institute of library and museum services (imls) and led by staff at the missouri botanical garden (mbg) along with partners from harvard, cornell, the new york botanical garden (nybg) and dartmouth, its aims were to significantly improve access to digital texts through the applicability of purposeful gaming for the completion of data enhancement tasks needed for content found within the biodiversity heritage library (bhl). this project tackled a major challenge for digital libraries: full-text searching of texts is significantly hampered by poor output from optical character recognition (ocr) software. historic literature has proven to be particularly problematic because of its tendency to have varying fonts, typesetting, and layouts that make it difficult to accurately render. the bhl is an international consortium of the world’s leading natural history libraries, that have collaborated to digitize the public domain literature documenting the world’s biological diversity. this has resulted in the single largest, open-licensed source of biodiversity literature made available both through the internet archive and through a customized portal at http://www.biodiversitylibrary.org/. bhl is a perfect testbed for investigating alternate solutions to the generation of digital outputs both because it is a significantly large corpus (48 million pages of scanned texts accompanied by 48 million ocr outputs) and because most of its content is historic literature (primarily published between 1450s-1900s). purposeful gaming and bhl set out to demonstrate whether or not digital games could be a successful tool for analyzing and improving digital outputs from ocr and transcription activities because large numbers of users can be harnessed quickly and efficiently to focus on the review and correction of particularly problematic words by being presented the task as a game. project design the basic design approach for purposeful gaming and bhl was to take two digital outputs for a text page (from two different sources), identify differences, push only those differences through a game for further verification by players, and extract and interpret the game output. preparation of data for the games the data preparation phase is as follows: before any data can be sent to the games, it first has to be prepared and run through a workflow to determine if it would be a good candidate.  first the team generates two ocr outputs from two different ocr applications. ocr applications, by design, must identify the coordinates of the words on the page to determine the ocr output. they automatically create a bounding box around each recognized word as part of the process.  in order to be sent to the games, the coordinate systems and units of measure used by the different ocr application must match.  bounding boxes around words found by each ocr application must also match (or be very close).  for example, if the first application locates a word in a document with a bounding box identified by the coordinates (100, 100), (150, 100), (150, 75), (100, 75) and the second application locates the same word at the coordinates (99, 100), (152, 100), (152, 74), (99, 74) that is acceptable.  the bounding boxes are nearly identical.  however, if the second application locates the word at (700, 650), (780, 650), (780, 690), (700, 690) then clearly the ocr applications are using different coordinate systems to locate the words in the document. thus, two ocr applications cannot be used together unless the coordinates match, or can be converted to the same coordinate systems. for the purposeful gaming project, the team uses the two ocr applications abbyy (which produces ocr output in the djvu format) and tesseract (which produces output in the hocr format). ocr quality once two ocr outputs are generated on the individual document scanned, the quality of each ocr output is checked twice to determine the suitability of the document for correction by the game. initially, quality is determined by matches to real words. each ocr output document is compared to dictionaries of known words. the ratio of known words to the total number of words in the document produces a quality score for that particular ocr output.  if each output’s score is high enough, then further processing of that document can proceed.  (the “high enough” threshold is configurable.  for the bhl/tiltfactor collaboration it was set to 0.7; that is, 70% of the words in the document identified by ocr are recognizable words in a dictionary.) the next step is to compare the two ocr outputs with one another and assign an accuracy score that provides a measure of the differences between the ‘quality’ scored outputs.  the accuracy score is the sum of 1) the percent difference in total word counts between the two ocr outputs, 2) the percentage of words on which the two ocr applications disagreed, and 3) the percentage of words for which one of the ocr applications produced no output (a blank). if the accuracy score is low enough, then the page can be corrected by the games.  (the threshold for the accuracy score is configurable.  the bhl/tiltfactor collaboration set it at 0.15).  if the quality score of either ocr output is not high enough, or if the accuracy score from the comparison of the two ocr outputs is too high, then the document is not a candidate for correction via the games.  in this case, it needs to be manually corrected or transcribed. the complete workflow for sending data through the game is: create a single ocr output for a document. check the quality of the ocr by comparing it against known dictionaries.  assign an ocr quality score.  the quality score provides a measure of how good a job the ocr technology did at identifying words in the document. if the quality score is high enough, create a second ocr output for the document with software that uses the same bounding box coordinates. check the quality of the second ocr output and assign an ocr quality score. if both ocr quality scores are high enough, compare the ocr outputs. generate an accuracy score.  the accuracy score provides a measure of how well the two ocr outputs match. keep track of the words on which the two ocr technologies did not agree.  be sure to track the coordinates of those words. if the accuracy score is low enough, create game inputs and a template for the final corrected ocr. game inputs are files containing the words to be corrected by the game, along with the bounding boxes of the words.  the template should include placeholders for the words to be corrected by the games. send the game inputs to the games. play the games. extract the corrected words from the games and update the final ocr template with the corrected words. digitization of seed and nursery catalogs while bhl’s traditional literature presents its own challenges for ocr software, the project team also wanted to test other types of texts that would present different types of ocr challenges.  seed and nursery catalogs were perfect candidates.  “since the mid-19th century, seed and nursery catalogs have reflected the agricultural and horticultural landscape of the united states. these catalogs—which began as guides to medicinal herbs, and are still printed today—often contain lists of plant varieties and gardening advice. while seed catalogs are used primarily by commercial growers and home gardeners, they also represent an invaluable resource to historians, artists, and researchers of all kinds” (randall 2014).  their varying column structures, images, tables, and whimsical font designs often confuse ocr software as to the order in which to read the text.  because bhl only contained a small fraction of all the published seed and nursery catalogs in existence, team members decided to digitize them as part of the project activities.  bhl partners at cornell, nybg and mbg digitized approximately 1600 catalogs equaling over 100k pages. transcription of field notebooks, diaries and catalogs field notebooks are another type of text for which ocr software is largely ineffective.  these hand-written notes are recorded by scientists during or after an observation event.  william brewster was a renowned american amateur ornithologist (1851-1919) who bequeathed his bird specimen collection to harvard university along with his journals and diaries which are a “gold mine of scientific observations  and a delightful account of years spent exploring the woods, fields, lakes, and rivers of new england” (paul 2014).  brewster’s notebooks and diaries had already been digitized and made available in the bhl portal so they only needed to be transcribed for this project.  bhl partners at harvard took the lead on selecting and setting up the transcription software as well as enlisting volunteers on two different transcription sites – fromthepage  and digivol.  two tools were chosen because the team wanted to test the effectiveness of each for the transcription task and also to generate two digital outputs, just as the team did for the published texts. the games the project team hired and collaborated with tiltfactor laboratory at dartmouth college to design and create the games. carefully choosing target audiences and target devices are crucial steps in the game design process to creating successful game experiences that players want to play.  in prior work, tiltfactor identified two distinct audience motivations for playing crowdsourcing games: the altruistic motivation, in which players are playing primarily to contribute knowledge, and the gameplay motivation, in which players are playing primarily to enjoy the game experience.  for our project these audiences turned out to be distinct groups; the altruism-motivated audience tended to be older and have some investment with the biodiversity heritage library (being interested in plants or biodiversity, or being a part of the library sciences).  the game-motivated audience skewed younger with no connection to bhl aims or collections.  instead of choosing to craft a single game for both audiences, or choosing to build a game for one audience over the other, tiltfactor created two games. smorball was targeted to the game-motivated audience and beanstalk was targeted to the altruism-motivated audience. initial investigations into these audiences revealed a preference for both mobile games and apps.  while this was our preferred direction for the games, early game design prototype tests showed that due to on-screen keyboard and small screen sizes, mobile devices did not provide enough screen real estate for players to type words.  as a result the games were optimized to be played on desktop computers through web-based browsers. smorball smorball is a challenging, time based browser game. in smorball, players take on the role of the coach of the eugene melonballers, a fictitious team that competes in the goofy, fictional sport of “smorball.”  through shrewd tactics and excellent typing ability, players must coach/direct their athletes to victory.  the player’s team plays defense in every game of smorball, and must prevent the opposing players from getting across the field to score in their endzone.  as the coach, the player must shout (type) commands to their athletes to tell them when to tackle their opponents.  the fewer points the opposing team scores in each match, the better for the player (and their accumulated winnings). the smorball title screen. smorball is designed for players who play games between casually and regularly, and players who are game-motivated (rather than altruism-motivated): players who don’t care or don’t know about the game’s data correction purpose.  the game is intended to be played over multiple sessions, and saves progress locally on the user’s computer. gameplay and game mechanics smorball is a highly leveled game experience; each level represents one sports match in a national series, leading up to the final match of the season: the “smor bowl.”  over the course of the first 3 levels players learn the basics of gameplay, and over the following levels they are challenged to apply the game mechanics more expertly to succeed.  the levels are intended to give the game-motivated players a sense of direction and to retain players at least through the course of all of the levels once, with the potential for replaying each of those levels multiple times.  at the end of each level, players are scored on how well they did, and this score directly contributes to the currency they accumulate to spend on upgrades.  this mechanism lends the game replayability, as players can revisit earlier levels to complete them with a better score. the smorball map/level selection screen. during each level, smorball’s player (the coach) encounters the opposing team’s robots, whom she must stop from scoring by typing commands to her own athletes.  smorball field has three ‘lanes’ down which athletes may run.  when opponents come down the bottom lane, for example, the player must type the word shown in the bottom row.  if she gets the word correct, her athlete in the bottom row runs forward, tackling the opponent. an example smorball level during gameplay. over the course of the game several types of equipment are introduced, which can be picked up off of the field during levels and used by the player at opportune moments to empower their athletes and more efficiently stop the opposing robots. when to use equipment is an important tactical decision that players need to practice in order to become more adept. at the end of each level, players are given winnings based on how well they performed. each level has a maximum payoff, and the winnings are reduced for each point the opposing team scores against the player. players can then use these winnings to purchase upgrades for their team including: the ability to start each level with one unit of each type of the aforementioned equipment, more frequent equipment spawning on the field, increased knockback, increased damage on long words, shorter cooldowns on mistakes, and more. the smorball shop where players can spend their winnings to upgrade their teams. smorball uses a hybrid automatic leveling system, as well a as manual system to adjust for players’ differing typing abilities. during the first level, the game measures players’ typing and gameplay proficiency and sets a recommended difficulty: easy, normal, or hard. this difficulty can then be changed manually by the player via the menu screen. the game relies on time pressure, scoring motivation, and incremental challenges to encourage players to persist in the game, which in the end decodes more words. a typical smorball game level presents 30-60 words for players to correct. beanstalk beanstalk is a non-time based web browser game in which players tend to a budding fictional bean plant that must be “spoken to” in order to grow.  as the player types words correctly to “speak,” an eclectic assortment of leaves and flowers sprout from the beanstalk, and the beanstalk grows higher and higher into the atmosphere and sky.  should the player make a mistake while typing a word, the beanstalk shrinks a small amount.  the plant starts off as a seedling, growing among blades of grass.  when the player reaches the stars, her beanstalk drops another seed, and she can start a second stalk. the beanstalk title graphic. beanstalk is designed for altruism-motivated players who are playing in order to contribute to the biodiversity heritage library’s collections, and whose game experience ranges between ‘zero experience’ and ‘some experience in the past, but no recent experience.’ the game is intended to be played over multiple sessions, and will save progress to the user’s account (if signed in), or locally on the user’s computer (if not logged in). beanstalk gameplay screens showing a small initial beanstalk after very few words have been typed (left) and a larger beanstalk after more words have been typed (right) game mechanics beanstalk is an extremely easy game with no time constraints. the reward for players are changes in the background and growth of the plant, with some hidden animations that change each round. because of the lack of time constraints and compact screen size, beanstalk can be played on phone and tablet browsers as well as desktop browsers. for players who are so inclined, beanstalk features leaderboards to compare the height of their plants (and thus the amount of data they have corrected) with other players’ successes. game design approach design goals in order to create compelling game experiences, games need to be able to immediately reward players for positive decisions, and either withhold rewards from or penalize players for negative decisions (fullerton 2014). this poses a clear problem for crowdsourcing games since, by definition, crowdsourcing is typically applied in cases where one cannot computationally identify the correct answers to the problems they are trying to solve. because the task of correcting ocr outputs is most effective and accurate if corrected through the act of typing out words, we adopted the approach used by many typing training games. to approximate the feel of a commercial typing game that would be familiar to players, the tiltfactor team tested several approaches to estimating the correctness of players’ inputs, so that the games could give players appropriate game feedback (such as gaining points when correct, and playing a buzzer noise when incorrect). the end goal of each of these playtests was to maximize true positives (marking the player ‘correct’ when typing the word correctly) and true negatives (marking the player ‘incorrect’ when making a typo), while minimizing false positives (marking the player ‘correct’ when in reality, the player made a typo), and false negatives (marking the player ‘incorrect’ when, in fact, the word was typed correctly). the goal is to help the players feel as though the game knows when their input is correct and when it is incorrect. research conducted by tiltfactor has shown that games intended to change players’ attitudes are found to be more engaging when the players aren’t aware of the games’ attitude change goals (kaufman and flanagan 2015). because of this research, during the design of smorball and beanstalk tiltfactor operated under the assumption that general gamers with no connection to the biodiversity heritage library or crowdsourcing would find crowdsourcing games less fun than traditional commercial games. to this end, one of the major design goals of smorball was to make the crowdsourcing purpose of the game invisible to players; smorball was intended to appear to players as a commercial typing game, such as typing of the dead. past solutions and limitations comparison to previous players crowdsourcing games traditionally judge ‘correctness’ of player inputs by comparing them to previous players’ inputs for the given word; if someone (or multiple someones) previously submitted the same input, then the current player is marked as ‘correct.’ otherwise, the current player is marked as ‘incorrect’ (chrons and sundell 2011). this approach can solve the problem of providing correct feedback to players, with two rather noticeable exceptions: when new databases of words are uploaded, since no players have played them yet, these words will have a 100% false negative rate–everyone who plays them will be marked incorrect when they type the word correctly. on the other end of the spectrum, once words have been played many times, many different typos and other incorrect inputs will have been logged. if a player types one of these many typos, she will be marked as correct; in other words, the false positive rate will be very high. problems with the “comparison to previous players” method of judging player accuracy in crowdsourcing games. comparison to current players some crowdsourcing games have attempted to solve the shortcomings of the comparison to previous players approach by instead pairing current players together synchronously, and comparing them to one another (von ahn 2006). while this solution works in theory, in practice it is dependent upon a consistent player base which is a challenge for most games. when a player can be found to match against, both the start of project and late in project problems are alleviated. in addition, games that implement this solution can experience subversion problems if players can communicate with one another and collude to match each others’ inputs without those inputs being actually correct. comparison to golden set a second solution to the shortcomings of the comparison to previous players approach is to temper it by intermittently giving the player words with known solutions, and using the player’s performance on the known ‘golden set’ words to infer their performance on the actual crowdsourcing work. while not often used in games, this approach has been used very successfully for recaptcha (von ahn et al. 2008), software designed to test for human users instead of bots. recaptcha gives users images of two words: one known to the system, and one unknown to the system; the software then infers the user’s input correctness on the unknown words based on their accuracy on the known words. with a large enough golden set, this approach solves the problems many projects face when starting in that there is no previous player data against which to refer. unfortunately, this method is not well suited for use in games for two reasons. first, recaptcha software was relying on many users each doing only one data correction. games typically aim to attract fewer players in favor of more time (and thus more corrections) per player. even with a golden set that grows from user input, at least a few users typing multiple words would begin to see the golden set items repeated, and the game would be ripe for subversion (players could identify the golden set item since they had seen it before, and then willfully mislead the game on the unknown words). second, while this method minimizes false negatives by having a golden set with a high confidence solution (presumably typed by a trustworthy source), it provides a very high rate of false positives. when a player makes a typo, there’s a roughly 50% chance that the typo will occur in the unknown word, and will be accepted by the game as ‘correct.’ smorball and beanstalk solutions smorball and beanstalk have a leg up on other crowdsourcing transcription games and tools in that they are not truly transcription games; they are ocr data correction games. because every word to be typed was guaranteed to have two ocr readings associated with it, the game development team chose to use these ocr readings to determine player accuracy, thus resolving the start of project problems that tools and games using the comparison to previous players approach fell prey to. initial attempts initially, the research and game development teams worked under the assumption that of the two ocr readings for each word, one would be correct and one would be incorrect. this would allow for a very simple matching algorithm: if the player types one of the two ocr readings, they are marked correct and that reading is recorded in the database as correct. this would provide a low rate of false positives, as any typo would have to coincide with the wrong ocr reading to be wrongfully marked as correct, and as long as one of the two readings was always correct, there would be zero false negatives. working with a sample set of over 500 words, it quickly became clear that while some words had one correct ocr reading, many did not, as shown below. this obviously posed a problem: while better than nothing, direct comparison to ocr for these pages would result in false negatives roughly 50% of the time. the percentage of words in sample data provided by the bhl which have one correct ocr reading. over the long term, this problem would be solved by the game team’s choice to combine the ocr comparison approach with the comparison to previous players approach.  this way, only one player would ever need to feel the frustration of typing the word correctly and being marked ‘wrong.’  subsequent players would type that same word correctly, and be matched to that first player’s input, and thus marked correct.  over the short term, however, the game team deemed this problem untenable, as initial players of the games would surely be driven away when half of their inputs were marked incorrect.  it became clear that the games would need a more sophisticated algorithm to infer player accuracy. longest common substring the game team developed a modified longest common substring (lcs) algorithm for this purpose.  simply put, this algorithm compares the two ocr readings to find character sequences upon which they agree, and requires the player’s input to contain those sequences in order to mark the player as correct.  in the diagram below, the arrow character pointing to the right refers to the start of a word, and the arrow character pointing to the left to the end of a word. illustration and example of the modified longest common substring algorithm used by beanstalk and smorball. applying this algorithm to the sample data set proved its use; while direct comparison to ocr would yield false negatives 52% of the time, adding in the lcs algorithm resulted in false negatives for only 3% of the words. these false negatives occurred only when both ocr readings agreed on an incorrect interpretation of a letter. user testing showed that at this frequency, when a false negative did occur players would often assume that they themselves had made an unnoticed typo rather than attributing the error to the game. since there is an unknown portion of every word, every word does have the potential to result in a false positive if the player’s typo happens to fall in the unknown portion. however, because the unknown portion of each word is usually rather small, this means that the chance to throw a false positive is equally small. this is much improved over using a “recaptcha-style” method where false positives occur at minimum 50% of the time. the percentage of words in sample data provided by the bhl that can have their correct answers identified algorithmically by either the lcs algorithm or having one ocr reading correct. all together now combining the lcs algorithm, comparison to other players’ inputs, and direct comparison to ocr readings into a three tiered approach, smorball and beanstalk are able to, with great success, infer players’ data correction accuracy for any given word and provide player feedback with minimal false negatives and false positives. upon a player’s submission of a word, the games’ first check is to compare that word to both ocr readings, and any previously accepted player submissions. if the player’s word matches any of those, that entry in the database has its score increased by 1, which will eventually be used to select the most accurate answer. this check serves two purposes. first, it reduces tax on the game, as in approximately 50% of all ocr disagreements, one of the ocr readings is correct. second, this check begins to mediate the few cases in which the lcs algorithm marks correctly typed words as incorrect. since previous players’ submissions are also possibilities for this check, even in cases when neither ocr is correct, and the lcs algorithm fails, players will still often have their correct answers marked as correct. in essence, in the 3% of cases where neither ocr option is correct and the lcs algorithm fails, a single player will be frustrated by being marked ‘incorrect’ when she should be marked ‘correct’ for that word once, and all future players will be rewarded correctly. logical flow for saving player inputs to the database in beanstalk and smorball if the player’s word matches neither the ocr readings nor previous submissions, the game system runs the word through the lcs algorithm. this is particularly relevant in the approximately 48% of cases where neither ocr output is correct, for ocr disagreements that haven’t been played before. in these cases, if the player’s word passes the lcs algorithm, it is inserted into the database as a new option for future players to input, and the player is marked ‘correct.’ if the player’s word matches none of the above, it is checked against a weaker lcs algorithm, which allows for a specific number of character discrepancies from the lcs. if it passes this algorithm, then one of two things has happened: either the player made a small typo, or the original lcs algorithm failed. because it might be a typo, the game marks the player as ‘incorrect.’ however, because it’s possible they typed the right word and the lcs algorithm failed, their submission is still entered into the database to see if future players agree with it. if their word fails the weak lcs algorithm, however, it bears no relation to the ocr output and can be safely discarded as a garbage word. this prevents players from being able to ‘seed’ the database with garbage words to attempt to ensure future matches. project results both games were released to the public on june 9, 2015. once deployed there was a significant amount of activities involving publicity and promotion of the games, which harvard and all project partners did through multiple channels. these included both print media (regional newspapers, academic journals, university publications) and social media (blogs, facebook, twitter) as well as presentations at both national and international meetings,workshops, and seminars. there were two on-site gaming sprints (one held at the boston festival of indie games and another at the new york botanical garden’s mertz library). an online sprint called “data dash” (bhl data dash 2015) was held at the end of the project to encourage one final data collection push. both games drew over five thousand participants in just under six months[1]. there were over 140,000 words typed and at its peak there were 8000 words per day and 1000 sessions per month. smorball was the clear favorite of the two, generating 61% of participation and even garnering the “best serious game” award at the boston festival of indie games (bfig). the award itself was presented by cambridge city councilor leland cheung, who spoke about how playing smorball is “more than just playing a game – it’s a form of civic engagement”. clearly, gaming with a purpose has earned a level of respect on par with gaming as entertainment in the field of game design and is being valued for its contributions to society. both altruism-motivated and game-motivated players reacted positively to the games, and competition amongst them often became heated. reviewers at bfig had this to say about smorball: “the world of the sports game isn’t innovative or unique, but it’s an original made-up game where humans get to smash robots, which makes it fun to watch and play.” “i do feel that this is a complete game, very polished, and addictive.” these sentiments were echoed by players, and are representative of the game’s reception in general: players enjoy smorball on its own, even without knowing its prosocial goals.  often, players do not notice that it is anything more than just a web game. project challenges despite the critical success of the games, there were some challenges along the way that diminished some of their overall impact for the bhl. these include being unable to automatically generate coordinates for hand-written texts and catalogs and being unable to collection enough data from the games to be able to apply the corrections to the full bhl corpus. lack of coordinate information for hand-written outputs in order to include a word in the game, coordinate data is needed to be able to link the text output of a word to an image of the word on the page. this data is then used to generate a picture of the word for display to players in the game. ocr software automatically generates coordinates for these words, but transcription tools for dealing with hand-written text do not. none of the six transcription tools that the project team reviewed could provide this functionality [2]. the team identified a few other tools[3] that attempted to provide the text to image linking services, the most viable of which is called tilt2 , and is being developed at the university of queensland australia. this tool, while a likely candidate for preparing hand-written texts for the games, was not fully developed enough to be used within the timeline of our project. without the coordinates, we were unable to include words from brewster’s handwritten documents or the horticultural catalogs through the games for further verification. while disappointing, we still had the two outputs generated from the transcription tools for these documents which were used to replace the poor quality ocr in the bhl portal. unable to reach a critical mass of data another challenge was our inability to generate a critical mass of data corrections needed to be able to apply them to the larger bhl corpus of 48 million pages. while over 140,000 words have been typed since the games were released, this has resulted in only 7,821 words and 330 pages entirely completed. we believe there were three primary factors that contributed to this outcome: 1) lack of a robust marketing strategy; 2) a limited data collection period; and 3) too high of a player agreement threshold in the game design. marketing despite employing a wide variety of marketing strategies we did not draw the number of players we had hoped. although several media outlets covered the games, none of them had a high enough readership profile to draw a significant number of players. the project needed a large media outlet such the nytimes or a celebrity endorsement to be able to garner more attention and none of the marketing strategists at the partner institutions had these contacts. one of the lessons learned is to have a more robust marketing plan with significant time and resources devoted to it because it is critical in getting uptake of your games. data collection period the data collection period was only six months from the time the games were released until the project end date.  considerable time was spent at the beginning of the project developing and broadcasting the request for proposals, as well as selecting and hiring a game design team. in hindsight, the project team should have had a game designer selected before the project began. this would have allowed for game development to begin earlier in the project and allowed for an earlier release date, thereby resulting in more time for data collection. player agreement threshold an additional factor had to do with the games’ level of player agreement needed for a word to be considered “complete” and thus be removed from the system. when using untrusted users for crowdsourcing work, “agreement” scores are used to minimize the likelihood that any single mistake or intentional subversion will be included in the final data.  instead of marking a word “complete” as soon as one player has typed it, the games wait for multiple players to type the same word, and mark it “complete” if there is a sufficient level of agreement between what the players type.  clearly, the determination of this score results in a trade-off between accuracy and efficiency; a high required agreement score would result in many players having to type any word, but a low agreement score could result in accidentally including incorrect data in the final dataset.  when the games were released the agreement score was set to 4 (i.e. a minimum of 4 people had to agree on a word before it was considered “complete”). this could require anywhere from 4-12 or more plays of a word before agreement is reached. it wasn’t until about five months after the games were released the team decided to lower this threshold, which significantly reduced the time needed to complete words and pages. whether the lowered score negatively impacted the quality of the data corrections has yet to be determined. conclusions the result of not being able to collect a critical mass of data corrections means that the benefits to the bhl corpus are limited for now. our original plan was to use the corrections to create a robust error matrix that would automate further cleanup of the entire corpus, thereby enabling results from the game to become much more scalable. instead we had to opt for a “proof of concept” matrix to illustrate how it could be applied in the future once the data corrections reach a certain tipping point. these challenges, however, do not diminish the overall impact of the project’s benefits to the larger digital library community. the primary aim was to demonstrate whether or not digital games could be a successful tool for analyzing and improving digital outputs from ocr and transcription activities. we believe the games developed for this project successfully proved how human verification of texts could succeed where machines had failed; most people are not willing to sit down and transcribe an entire page from a manuscript. by combining the efficiency of crowdsourcing with the enjoyment and motivation of gameplay, smorball and beanstalk elevated a normally tedious task into an activity full of wonder, reward, and action. we believe the project’s games will become a model for intersecting gaming with crowdsourcing in the cultural heritage sector. when we wrote our original proposal at the beginning of 2013 we noted that despite the uptake of games and crowdsourcing in the museum communities, games have been underexploited in the library community and we would argue little has changed since then. a review of the 2015 new media consortium’s horizon reports for both libraries (johnson 2015) and museums ( johnson 2015) confirms that museums continue to take the lead on use of gamification and crowdsourcing in their digital strategies. in addition to the games, the architecture we developed for managing game input/output and the data preparation workflows provide a roadmap for how other libraries can embark down a similar path. we hope our strategies and outcomes provide the confidence needed for other libraries to engage in similar projects. in this spirit, we have tried to document, in as much detail as possible, our activities on our project website, and in this article. notes [1] exact count was 5,327 players and 7,122 sessions between the two games (composed of smorball 4365 sessions/3,326 players and beanstalk 2,757 sessions/1,911 players) [2] transcription tools reviewed included: smithsonian transcription tool (https://transcription.si.edu/); fromthepage (http://fromthepage.com/); transcribe bentham (http://www.transcribe-bentham.da.ulcc.ac.uk/td/transcribe_bentham); digivol (http://volunteer.ala.org.au/); transcribe http://www.archives.gov/citizen-archivist/transcribe/; t-pen (http://t-pen.org/tpen/) [3] other text to image linking tools revewed included: textgrid (https://wiki.de.dariah.eu/display/textgrid/main+page); tile (http://mith.umd.edu/tile/); alethia (http://www.primaresearch.org/tools) references bhl data dash. [internet]. 2015.[cited 2016 apr 2]. available from: http://blog.biodiversitylibrary.org/2015/12/bhl-data-dash-dec-7th-9th-2015.html chrons o, sundell s. 2011. digitalkoot: making old archives accessible using crowdsourcing. human computation [internet]. [cited 2016 feb 08]; 20-25. available from: http://cdn3.microtask.com/assets/download/chrons-sundell.pdf fullerton t. 2014. game design workshop. boca raton (fl): crc press. p. 99-100 (coins) johnson, l., adams becker, s., estrada, v., and freeman, a. 2015. the nmc horizon report: 2015 museum edition [internet] austin, texas: the new media consortium. [cited 2016 apr 22]. available from:  http://cdn.nmc.org/media/2015-nmc-horizon-report-museum-en.pdf johnson, l., adams becker, s., estrada, v., and freeman, a. 2015. the nmc horizon report: 2015 library edition [internet] austin, texas: the new media consortium. [cited 2016 apr 22]. available from:  http://cdn.nmc.org/media/2015-nmc-horizon-report-library-en.pdf kaufman g, flanagan m. 2015. a psychologically “embedded” approach to designing games for prosocial causes. cyberpsychology: journal of psychosocial research on cyberspace [internet]. [cited 2016 feb 08]; 9(3). available from: http://cyberpsychology.eu/view.php?cisloclanku=2015091601 paul, e. 2014. step back into ornithological history. [internet]. [cited 2016 apr 22]; available from:  http://ornithologyexchange.org/articles/_/community/step-back-into-ornithological-history-r182. randall, p. 2014. the stories seeds tell [internet][cited 2016 apr 22];. available from:  http://blog.biodiversitylibrary.org/2014/11/the-stories-seeds-tell.html von ahn l. 2006. games with a purpose. computer [internet]. [cited 2016 feb 08]; 39(6): 92-94. available from: http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=1642623 von ahn l, maurer b, mcmillen c, abraham d, blum m. 2008. recaptcha: human-based character recognition via web security measures. science [internet]. [cited 2016 feb 08]; 321(5895): 1465-1468. available from: http://science.sciencemag.org/content/321/5895/1465 about the authors max seidman is a game designer at tiltfactor, the game design and research lab at dartmouth college, where he designs impactful digital and tabletop games for health, equality, and knowledge. his formal training lies in engineering and human centered design. mary flanagan is the founding director of tiltfactor and the sherman fairchild distinguished professor in digital humanities at dartmouth college. trish rose-sandler is the data projects coordinator for the center for biodiversity informatics at the missouri botanical garden. she served as principal investigator for purposeful gaming and the bhl project. mike lichtenberg is a senior consultant with sbs creatix. in that role, he has worked at the missouri botanical garden since 2007 as the developer for the biodiversity heritage library. prior to entering the library non-profit world, he earned experience in varied industries including retail, travel, insurance, engineering, agriculture, healthcare, education, and professional sports. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – actions speak louder than words: analyzing large-scale query logs to improve the research experience mission editorial committee process and structure code4lib issue 21, 2013-07-15 actions speak louder than words: analyzing large-scale query logs to improve the research experience analyzing anonymized query and click-through logs leads to a better understanding of user behaviors and intentions, and provides opportunities to create an improved search experience. as a large-scale provider of saas services that returns search results against a single unified index, serials solutions is uniquely positioned to learn from the dataset of queries issued to its summon® service by millions of users at hundreds of libraries around the world. in this paper, we describe the relevance metrics framework that we use to analyze our query logs and provide examples of insights we have gained during development and implementation. we also highlight the ways our analysis is inspiring changes to the summon® service to improve the academic research experience. by ted diamond, susan price, and raman chandrasekar 1.     introduction search engines are commonly used today, ranging from general purpose web search engines such as google and bing, to specialized academic search products such as dialog (dialog 2013) or the summon® web-scale discovery service (the summon® service 2013). in the early stages of information retrieval (ir) systems, studies were typically conducted using a test collection consisting of a static set of documents, a static set of queries, and a set of “gold standard” relevance judgments generated by a human expert.  this approach to studying ir is often referred to as the cranfield paradigm after the studies performed at the college of aeronautics at cranfield by cyril cleverdon (cleverdon 1967).  the series of annual text retrieval conference (trec 2013) events follow a similar methodology on a somewhat larger scale.  such studies emphasized finding ways to improve document-ranking algorithms based on document properties, such as term frequency, or collection properties, such as inverse document frequency and page rank (brin and page 1998).  returned documents were considered either relevant or not, and the user was largely left out of the picture.  the cranfield approach stands in contrast to surveys and user-oriented studies that focus explicitly on the user experience and users’ feedback.  ingwersen and järvelin (ingwersen and järvelin 2005) provide a detailed discussion of the interplay between lab-oriented and user-oriented studies in ir.  more details about document ranking algorithms can be found in classic textbooks such as those by salton (salton and mcgill 1986) or manning (manning et al. 2008). with the proliferation of web-based search systems, it is now possible to aggregate information from a large number of user queries and actions.  for example, studies have reported on the average query length and the most common terms and topics in queries (e.g. jansen et al. 2000; jansen and spink 2005; kamvar et al. 2009).  log data enables us to study ir in the context of users while allowing us to analyze user behavior across a much larger number of queries and a much larger number of users than would be possible in a controlled experiment for which users are recruited, observed, and possibly assigned search tasks. log data is often a more reliable signal than data produced by either simulations of users (submitting batch queries in a test system) or experiments that observe users in a lab because log data is about what people actually do rather than what we think they do or what they say they do. with a significant volume of logs, we can garner useful quantitative insights into our search systems.  furthermore, we can learn from this data.  we can change search systems in response to our findings, and measure the effects of our interventions. this paper is precisely about this – a description of how we use query logs and click-through logs to gain insights about users’ (actual) search behavior, in order to help us provide a continually improving search experience for our users. section 2 outlines the context of this work, to ground the discussion that follows. however, it is important to note that while the work is described in terms of our implementation on a web-scale discovery system, the discussions and conclusions are not specific to our environment. much of what we describe applies to any similar large-scale search system with significant query and click traffic in which all users search the same body of content and the data being searched shares a common schema. section 3 describes the relevance metrics framework  (rmf) that we developed to aggregate and learn from logs.  we start by listing our goals for this framework, as well as two concepts that are at the foundation of this framework. we then outline the dataflow in the framework, and the metrics and statistics we compute. in section 4, we describe some insights that we have gained from the rmf that have helped us improve the summon user experience, basing our changes on real user data. there is, however, more work to be done to improve the framework, and section 5 outlines some challenges and assumptions in the current system, and how we can obviate some of these issues. we conclude in section 6 with a look at work ahead. 2.    web-scale discovery systems in this paper, we describe a framework to analyze logs from a large-scale search engine. much of what we describe here, and the conclusions we reach, would apply to any such search system with significant query and click traffic. however, in order to keep the discussion grounded, and to provide real examples that can be interpreted, we describe the specific search engine that we are using, the summon® discovery service. the serials solutions summon® discovery service provides a hosted software as a service (saas) solution for searching all of the resources in a library’s collection and, optionally, resources beyond the individual library’s holdings.  behind the scenes, summon uses a single unified index that is created after matching and merging records that originate from multiple sources.  the summon index includes well over a billion items from a wide variety of publishers and vendors. figure 1 shows a sample summon result page. from a single search box, users can search over both metadata and full text records, find resources from multiple publishers, and receive a single unified set of search results.  in addition to supplying a simple query consisting of one or more terms, a user can apply a variety of filters and facets to the search, or use advanced functionality to search particular fields or apply boolean or other specialized operators.  the final list of search results is compiled based on a sophisticated set of algorithms that calculate the relevance of various documents to each specific query. figure 1. sample summon result page (current version) the summon® discovery service serves a growing list of over 600 customers and supports native searching in 17 languages. clients may either use our hosted service to retrieve results and to render these results, or retrieve results from the summon api and render the results themselves. because it is a hosted service, and because queries are centrally processed, summon is able to track all queries. in addition, clicks on results can be logged from pages rendered by summon.  thus, unlike systems that federate queries to different engines, summon is able to acquire and aggregate logs, and use these aggregated logs to continuously improve the user experience. the discussions and conclusions in this paper will apply to any similar engine and environment. 3.    the relevance metrics framework we constantly strive to improve the quality of our users’ research experience.  as part of this effort, we built the relevance metrics framework (rmf) as a tool to help us understand how our customers use summon and enable us to assess the quality of the search results we return.  we have three main goals for rmf. 3.1. rmf goals with the rmf, we wanted to define and develop a framework that would permit us to: observe and log user actions. we collect data about each query submitted to summon. for the rmf, we define a query as a single request from the user to the summon system.  a query consists of the search terms plus any filters or facets used to limit the results and any advanced syntax or operators used to define the request.  we do not consider a request for additional pages of results for the same request to be a new query, although we log the total number of results and pages that are displayed to the user. we log the query terms entered, filters applied, facets chosen, and advanced syntax or operators used.  we also collect data about clicks on search results.  our focus with the rmf is relevance, not the user interface (ui), so we do not instrument ui elements or measure ui feature usage. figure 2 has some sample queries submitted to summon. note that in the sample we have queries in different languages, some with filters (green fixed-width font), and some with facet restrictions (blue italicized font).   for example, the query publicationtitle:(bbc monitoring former soviet union) s.fvf=contenttype,journal article,f, s.rf=publicationdate,2007:2007 represents a request for documents with the following terms in the title: bbc monitoring former soviet union and limited to the facet journal articles and filtered to those published in 2007. the data in the rmf is anonymized.  we know when a query was submitted, and to which customer-library’s interface, but we do not store information that could identify the end-user who submitted the query. compute the quality of search results. we compute various metrics to assess the quality of our search results, based on user interaction with the results returned.  we discuss the metrics, and underlying assumptions, in section 3.4 below. analyze data to improve the summon product. we analyze the data we collect to improve search results and, more broadly, to enhance the user’s research experience.   by observing the queries that users submit, we can glean information about query styles (e.g. length and specificity of queries; use of natural language vs. terse phrasing; or inclusion of terms that would appear in different metadata fields, such as title plus author name or pasting an entire citation from a reference list into the search box) and user intent.  measuring relevance helps us identify the types of queries that tend to result in good results and the types of queries that return results that are not as good; this helps us identify areas for intervention and improvement. records of user behavior during day-to-day use provide concrete evidence regarding how a product is used, and real-life feedback regarding how well the product meets users’ needs. we obtain enough data to draw inferences about typical performance on classes of queries. anniversary 9/11 kidney stone moore vs mack moore vs mack trucks armee deutschland rolle in einheit body art in the workplace the moplah rebellion of 1921 john j. banning ????? 9780140027631 “boundary dispute” title:(india) subjectterms: “???” titlecombined: (analytical biochemistry) s.fvf(contenttype,book review,t) hawthorne, mark. “the tale of tattoos: the history and culture of body art in india and abroad.” figure 2. sample summon queries 3.2. useful underpinnings two concepts have proved enormously helpful for interpreting user behavior in the rmf: the notion of search sessions, and the use of clicks as an indicator of relevance. 3.2.1. search sessions when a user interacts with a search engine, whether it is a library discovery tool such as summon, or a web search engine such as google or bing, it is not unusual to enter multiple queries to satisfy a particular information need.  these iterations of the user’s request, each one involving adding, subtracting, or changing terms, filters, or advanced operators in the query, are not independent.  a search session that ends in a satisfied need is a success, even if the search session requires multiple queries. in general, though, satisfying the need with one query is better than requiring multiple queries.  we find the notion of a search session to be a useful abstraction for describing the use of multiple queries to find information for a single information need. by analyzing the data for both individual queries and search sessions, we get a broader perspective on user behavior and the ability of the system to meet users’ needs.   however, it is not easy to automatically identify search sessions in query logs. we heuristically define a search session as a set of one or more queries with no more than a 90-minute gap between queries, and lasting no more than eight hours in total. search sessions are also useful for limiting the effect of noise and query spam on relevance metrics.  as noted later in the paper, we see some query patterns that clearly do not reflect the efforts of serious end-users trying to perform research or other information gathering tasks.  in some cases we have seen thousands of queries being issued over a small period of time, most likely through a script.  grouping these queries into a single session mitigates the effect such queries might have on our metrics, and makes the rmf more robust in the presence of noisy data. 3.2.2. clicks as a proxy for relevance in classical information retrieval experiments, relevance is measured using a known set of queries against a known set of documents for which relevance judgments have been made available.  these relevance judgments are not easy to garner for large-scale search engines, and any such judgments may not necessarily reflect whether a document satisfied the particular information need underlying an individual user’s query.  ideally each user would tell us which document(s) satisfied their need and which document(s) do not, but of course such information is not available. instead, we have to make inferences based on user behavior, and the most readily available signal we have is click data.  not every document a user clicks on is relevant. not every potentially useful document in the search results gets clicked-on, even if it is returned on the first page.  however, documents that are not clicked on are generally not useful to the user (with occasional exceptions, such as when the metadata or a snippet contains an answer to a specific question). so we can consider click data to be an upper bound on relevance.  if there were no clicks, the search was probably unsuccessful.  if there were n clicks, then up to n documents were useful (relevant).   over a very large set of data, clicks provide a useful, if imperfect, proxy for relevance judgments. 3.3. dataflow in rmf data in the rmf comes from two sets of logs: (1) the logs from the search servers, which produce the search results that are returned to users in response to queries, and (2) click logs that record data about result links to documents that are clicked-on by the user.  the search logs, which contain a browser session id, a timestamp, the url corresponding to the query, and other information about the query and the results list are fed into a query summarizer component.  the query summarizer extracts the data of interest for each query, such as time of submission, the customer library receiving the query, search terms, filters, number of results found, and number of results displayed (i.e. number of pages viewed multiplied by number of results displayed per page) into a query summary object.  data from the click logs is associated with individual queries using the browser session id and timestamps. query summary objects are grouped into sessions by first using the browser session id and then applying our own heuristics to segment browser sessions into multiple search sessions when appropriate. the session summarizer component extracts data about each session into a session summary. the query summaries provide data input to a query metrics calculator component, which calculates various query-level metrics.  the metrics are input to the query statistics calculator, which uses the query metrics to calculate statistics based on aggregated query data.  similarly, the session metrics calculator calculates session-level metrics, which are used by the session statistics calculator to produce statistics based on aggregated session data.  query data, query metrics data, session data, and session metrics data are loaded into a relational database for storage and easy retrieval. 3.4. metrics computed in rmf we calculate a number of metrics at the session and query level.  for example, at the session level we track abandonment (defined below), and the number of queries to the first click or to abandonment.  at the query level, we track abandonment and measures of how well relevant documents are ranked.  we describe each of these metrics in more detail below. abandonment when a user submits a query but does not click on any of the search results, we refer to that behavior as query abandonment.  abandonment usually indicates a failed search. there are some rare exceptions to this, e.g. where the user’s information need can be satisfied by the results display itself. for example, the question “who wrote the book the mythical man month?” can be answered by entering the query “mythical man month” and seeing “by frederick p brooks” in the display for the first result, without having to click on the item. when a user submits one or more queries in the same search session and does not click on any search results, we refer to that behavior as session abandonment.  we calculate both average query abandonment and average session abandonment.  low values of abandonment suggest (but are not proof of) better user satisfaction whereas high values of abandonment almost certainly indicate user dissatisfaction. number of queries to first click consider a session where the user enters one or more queries, and scans the results to see if there are any results that look sufficiently relevant to warrant clicking on the result to view the document. the value of number of queries to first click indicates how much effort (in terms of the number of queries issued) was required before the likely-to-be-relevant item was selected with a click.  while not every clicked document is relevant, over a large number of queries, having a lower value for the number of queries to the first click generally indicates a better search system than a higher value does.  mean reciprocal rank (mrr) we want a search engine to return the most relevant results at the highest rank, i.e. at the top of the list of search results.  mrr considers only the rank of the “best” (lowest ranked) clicked result.  mrr is computed as 1/(rank of the top-ranked clicked result).  if the first result is clicked, mrr = 1/1 = 1. if the first clicked result is at result rank 5, mrr = 1/5 = 0.2.  in general, higher mrr is better.  if no results are clicked we define mrr to be 0. discounted cumulative gain (dcg) users frequently click on multiple search results.  sometimes that is because the first result turns out not to be what the user was looking for. for an informational query, many documents may be relevant to the search, and the user may want to find multiple resources.  dcg is a metric that provides a single number to describe the goodness of a search result with potentially multiple relevant results.  the intuition is that the right (relevant) results should be ranked in the right order.  dcg awards a ‘gain’ (score) to each relevant document, but the highest gain is given to documents ranked at the top of the results list. gain is progressively discounted, as results appear lower in the results list. the degree of discounting can be adjusted by changing a parameter in the calculation.  gain is cumulated across all relevant documents for a query. hence the term discounted cumulative gain. in general, a higher dcg indicates a better result set. appendix a has an explanation with examples about how dcg is calculated. 4.    gains from rmf we have seen in the sections above how we acquire log data, and the metrics and statistics we compute from the logged data. in this section, we provide a few examples of our analyses and our use of data from the rmf. broadly, we use the rmf both as a data source for tools that improve the search experience, and as a source of insights about user behavior that inspire new features. 4.1. data source for tools we use query data from the rmf to automatically generate suggested completions of the query as the user starts to type in the search box, as illustrated in figure 3.  once the query has been submitted, we not only return search results, but we also display suggestions for related queries. these suggestions are also derived from query data from the rmf. figure 3. query autocomplete, based on users’ queries 4.2. user behavior and insights 4.2.1 patterns of search use in general, whereas the number of query sessions varies regularly with the day of the week (more query sessions during the week and fewer query sessions on the weekends), the average number of queries per session is quite stable.  figure 4 illustrates the pattern over a six-week period that is near the beginning of the fall term for many academic institutions.  the x-axis shows the number of days since the start of this sample. the y-axis denotes the number of sessions per day, as well the average number of queries per session (on a different scale). we have omitted the numbers on the y-axis.  however, the y-axes cross the x-axis where y=0 so the shapes of the curves indicate the trends and their relative magnitudes. the search volume is highest in the middle of the week and lowest on weekends while the average number of queries per session does not vary much by day of week.  the growth of overall search volume is probably due to a combination of the progression of the academic term and overall growth of summon usage. figure 4. patterns of search usage 4.2.2. optimizing default results per page when we display a larger number of results on each page, the user needs fewer page navigation clicks to scan a large a number of results from his search.  however, if the number of results displayed per page is larger, more data must be transferred to the user, and hence there is a longer latency between query submission and the display of the results page.  many users may prefer shorter latency, but some users will want to see a larger number of results on a page.  summon offers the flexibility to return 50, 25, or 10 results per page (rpp), depending on the preference of the user or library.  the rmf provided insights for setting the ideal default rpp.  from an analysis of rmf data we determined that most clicked documents appear in the top 10 results.  that data gave us confidence that, for most users, the benefit of improved performance would outweigh the cost of reducing the rpp from its original default of 25 to a new default of 10. after we made the change, we monitored several metrics, but we did not see a decline in relevance metrics.  it is still possible for users to change the default results per page (and some do), but we have found that most users do not change the rpp, suggesting they are satisfied with the new default. 4.2.3 the nature of user searches we have found that users typically start with simple searches, and only occasionally make use of advanced search features.  summon offers both a basic search interface (simple text box, similar to that offered by google and bing) and an advanced search interface that allows the user to specify that particular terms should appear in specific fields, such as author, title, or isbn.  advanced syntax may also be used directly in the simple search text box.  for example, advanced search operators are available, such as boolean operators (and, or, not), wildcard expansion (child* matches child, children, and childs) and sloppy phrases (terms in a phrase may be separated by a specified number of positions). not surprisingly, we find that most queries use only the simple search and no advanced syntax.  less than 6% of queries specify one or more search fields, with the most commonly searched fields being author and title.  less than 4% of the sessions start with a query using search fields. so we believe most users start with a simple search, and only add fielded search if they cannot find what they are looking for.  even fewer searches make use of advanced syntax features, with only a little over 2% of searches using one or more advanced operators. on the other hand, filter usage is quite common.  various types of filters are available that allow a user to easily restrict the results of his query to make the query more precise.  for example, summon offers filters for over 90 different content types (e.g. books, journal articles, newspaper articles).  examples of other filter types are filters to restrict the results to particular publication date ranges, or to documents indexed with particular subject terms.  over 40% of queries use at least one filter, and a little over a third of sessions start with a query using at least one filter.  we know that some of our clients set the default search to include one or more filters, so not all the filters are applied directly by the end user.  the most commonly used category of filters is contenttype, with full text and limit to scholarly publications being the next-most common filters. we are using this information to improve our ui and our default settings, to ensure that our users have the best search experience. 4.2.4 query length and abandonment we also investigated the relationship between query length (the number of terms in the query) and query abandonment.  as shown in figure 5, users are much more likely to abandon queries after issuing a short query.  short queries are often vague, not fully representing the real information need of the user, and such queries often return many results that may or may not be useful.  while very unusual words may result in fairly specific and useful results, we see that some of our most common queries are single words, such as leadership, psychology, or marketing.  suggestions for related queries can help a user specify in more detail what aspect of a topic is of interest. figure 5. query abandonment as a function of the number of terms in the query sessions that begin with short queries are also more likely to end in abandonment than sessions that begin with longer queries.  figure 6 shows session abandonment for sessions beginning with queries of various lengths.  unlike query abandonment, which continues to decline as queries get longer, session abandonment rates are fairly stable if the first query consists of four or more terms.  as in figure 4, we have omitted the scale from figures 5 and 6, but because the y-axis crosses the x-axis at zero, the relative size of the difference in abandonment rates with change in number of search terms is evident. figure 6. session abandonment as a function of the number of terms in the first query of the session in addition to the information discussed above, we know that a significant proportion of our queries are three words or less in length. to further improve the experience for shorter queries, the summon® service will be introducing a topic explorer pane along with a redesigned user interface in june 2013.  the topic explorer identifies relevant topics from over 50,000 encyclopedia entries, guides users through the research process and exposes them to pertinent reference content in context of their query.  the summon® topic explorer pane also provides recommendations to relevant research guides and subject librarians. figure 7 provides a screenshot of an early prototype of the redesigned summon user interface with the topic pane. figure 7. the redesigned summon® user interface with topic pane 5.    improving rmf the last section identified many uses of the rmf. there is still work to be done to improve the system. specifically, we have faced challenges in two main categories: data quality and the implications of simplifying assumptions on data interpretation. 5.1. data quality we log data from all queries submitted to summon®, but not all data represents real user queries, and not all user queries are of equal interest.  some queries are submitted automatically to monitor connectivity.  for example, if we examine the logs and see that the same query is submitted to the same library client four times an hour, day after day, we can surmise that the query is being used to monitor connectivity and does not represent a library user’s information need. occasionally we see evidence of what appears to be a hacking attempt.  for example, the following does not look like a legitimate user query: ..%2f..%2f..%2f..%2f..%2f8ud0y%2f..%2f..%2f..%2f..%2f..%2fboot.ini%00.htm.  further analysis revealed that the session in which that query occurred contained 5906 queries, almost all of which contained the pattern 8ud0y and many contained path symbols, such as ../.  another interesting pattern was a set of sessions over a few days that each consisted of over a thousand queries where each query consisted of title: <title of a document> and was probably generated by a script. we have also identified attempts to scrape our content by issuing thousands of queries. such scraping, if left uncaught, reduces the utility of the rmf. our main concern about such data quality issues is that they could distort the relevance metrics and lead to unwarranted conclusions about user behavior.  for example, thousands of monitoring queries, none of which result in clicks, could skew our abandonment rate upward.  a robot that clicks on every search result could skew our metrics such that our relevance appears better than it really is.  exhaustively identifying all bad, or suspicious, data is tedious and error prone.  once we identify the undesirable data, we have several choices for dealing with it.  we could filter out suspect queries, but that will only be successful to the extent that we can proactively identify them.  we know our success would only be partial and we would risk filtering out legitimate user query data. alternatively, we could ignore the problem and hope that it is a small enough fraction of the data such that the effects on metrics are negligible.  we could also retrospectively identify problem queries and sessions and tag them, so that we don’t lose data but we can omit the problematic data from our analyses. we take a pragmatic approach that is a combination of the options described.  we filter out certain queries early in the data processing pipeline.  retrospectively, we examine various data characteristics to identify other “suspicious” queries. we tag such queries and sessions in the database that we believe do not represent real user queries or that represent query “spam.”  we can then filter such data with our database queries.  it will not be possible to identify all such queries and sessions, and clearly, time spent finding and tagging them has diminishing returns.  we try to identify the most impactful instances and assume that the instances not found represent a small fraction of the untagged queries.  for the data from 2012, we tagged less than three quarters of one percent of sessions and about a quarter of one percent of queries as “suspect.” 5.2. the implications of simplifying assumptions conceptually, a query session represents the activity of a single user pursuing information related to a single information need.   practically, we do not have enough information to segment a stream of queries into sessions accurately, and so our sessions are only approximate.  in a public place, such as a library, multiple users may send queries from the same machine and browser. so a session could represent queries from multiple users.  again, a single user may search for information about multiple topics, and we do not try to analyze the content of searches to identify changes in topic.  instead, we take a very pragmatic approach to try to associate the queries one user might issue in a coherent searching session and place boundaries to ensure that sessions do not become overly long. if each user were to tell us which documents we return are relevant to his query, we could accurately measure the relevance of our search results.  in practice, of course, we do not get feedback with that degree of detail.  we use clicks as a proxy for relevance, recognizing that users often click on documents and find that they are not relevant. in addition, relevant documents may be returned in the results set, even on the first page, that are never clicked.  finally, users’ information needs may sometimes be sated with information in the snippet that is returned with the result, so that the user does not have to click on any result. we think of clicks as providing an upper bound on relevance and believe that, over a large number of queries, the proportion of truly relevant documents among all documents that are clicked is relatively stable so that trends in click data are meaningful over time. as we work with this data, we hope to come up with algorithms and approaches to improving components such as our session identification and relevance identification. for now, we have to be careful to understand that any conclusions we make must be considered in the context of the assumptions we have made. 6.    conclusions in summary, we developed the relevance metrics framework for capturing, processing, and analyzing query and click log data.  the rmf enables us to understand how summon is being used and monitor the quality of our search results.  it provides a data source to enable various tools that can improve the user experience and serves as a source of insight and ideas for future product enhancements.  we have provided a sample of information and insights that we have garnered from the rmf and listed some ways we are already using the data, such as the autocomplete feature and the topic explorer, which exemplify the data-driven product design that we have been practicing.  we hope that as we explore the data in rmf further, we will be able to continually improve our systems to directly benefit all our customers and users. references dialog, llc. [internet].  proquest; [cited may 15, 2013].  available from: http://www.dialog.com. the summon® service. [internet].  proquest: [cited may 15, 2013].  available from http://www.serialssolutions.com/en/services/summon/. cleverdon c.,  1967. the cranfield tests on index language devices. aslib proceedings, 19, pp. 173-192, reprinted in jones k and willett p, editors.  1997. readings in information retrieval. morgan kaufmann publishers, inc., san francisco, ca, pp. 47-59. national institute of standards and technology (nist). text retrieval conference (trec) home page. (cited may 15, 2013). http: //trec.nist.gov/. brin s, page l., 1998. the anatomy of a large-scale hypertextual web search engine. in: proceedings of the seventh international conference on world wide web 7,  pp. 107-117, brisbane, australia. ingwersen p, järvelin, k., 2005. the turn: integration of information seeking and retrieval in context.  dordrecht, the netherlands: springer salton g, mcgill m., 1986. introduction to modern information retrieval. mcgraw-hill, inc., new york, ny, usa. manning c, raghavan p, schütze h., 2008. introduction to information retrieval. cambridge university press, new york, ny, usa. jansen b, spink a, saracevic t., 2000.  real life, real users, and real needs: a study and analysis of user queries on the web.  information processing and management 36: 207-227. jansen b, spink a. 2006.,  how are we searching the world wide web? a comparison of nine search engine transaction logs. information processing and management 42: 248–263. kamvar m, kellar m, patel r, xu y.  2009., computers and iphones and mobile phones, oh my! a logs-based comparison of search users on different devices. www 2009, april 20-24, 2009, madrid, spain. järvelin k, kekäläinen j. 2000., ir evaluation methods for retrieving highly relevant documents. in proceedings of the 23rd annual international acm sigir conference on research and development in information retrieval (sigir ’00). acm, new york, ny, usa, 41-48. järvelin k, kekäläinen j. 2002., cumulated gain-based evaluation of ir techniques. acm trans. inf. syst. 20, 4 (october 2002), 422-446. about the authors ted diamond (ted.diamond@serialssolutions.com) is a senior developer on the summon development team at serialssolutions. he designed and created the first version of the relevance metrics framework and has led the development of the new topic explorer feature in summon. susan price (susan.price@serialssolutions.com) is a search analyst on the summon development team at serialssolutions where she works on collecting, analyzing, and using data to improve the summon experience. susan’s research for her ph.d. in computer science entailed creating and evaluating a model for improving the relevance of search results in domain-specific document collections. raman chandrasekar (raman.chandrasekar@serialssolutions.com) is the software architect lead for the summon development team at serialssolutions. chandra has spent many years researching and working on web search systems, including work on relevance on an early version of bing, and on query-log analysis, domain-specific search and human computation as a researcher at microsoft research. appendix a:  calculating  discounted cumulative gain (dcg) a typical formula for calculating discounted cumulative gain (dcg) for a set of search results (järvelin & kekäläinen, 2000; järvelin & kekäläinen 2002) is: dcgp = r1 + ?j=2..p(rj/log2(j)) here dcgp is the discounted gain cumulated to rank p. rj is the relevance of the result at rank j.  this formula sums up the discounted gains for the relevance at all result positions, where the discount at position 1 is 1, and at other positions is (1/ log(j)). to compute dcg, we need to know for each result, whether is it relevant or not. if we use a binary scale, we can score relevance as 1 if the document is clicked, and otherwise 0. alternatively we could use a graded (likert) scale corresponding to relevance levels such as not relevant, slightly relevant, fairly relevant, and highly relevant. in this case, we could use scores ranging from 0 to 3 for not relevant to highly relevant. since we are using clicks as a proxy for relevance judgments, our relevance values are binary. search a document rank (j) relevance (relj) discounted gain (relj/log2(j)) cumulated gain (at rank p) 1 0 0 0 2 0 0 0 3 1 0.63 0.63 4 0 0 0.63 5 1 0.43 1.06 6 1 0.39 1.45 search b document rank (j) relevance (relj) discounted gain (relj/log2(j)) cumulated gain (at rank p) 1 1 1 1 2 0 0 1 3 0 0 1 4 1 0.5 1.5 5 0 0 1.5 6 0 0 1.5 figure a1. dcg for search a and search b to illustrate dcg, we provide a simple example in figure a1.  suppose a user performs two searches that we will call search a and search b. for search a, the user enters a query and clicks on documents at ranks 3, 5, and 6.  the gain at rank 3 is 1, but the discount is 1/log2(3), which is 0.63. similarly we calculate the discounted gain at each rank and keep a running total to get the dcg for search a at rank 6 as 1.45 . for search b, the user clicks on documents at ranks 1 and 4. the dcg for search b is 1.5. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – natural language processing in the humanities: a case study in automated metadata enhancement mission editorial committee process and structure code4lib issue 46, 2019-11-05 natural language processing in the humanities: a case study in automated metadata enhancement the black book interactive project at the university of kansas (ku) is developing an expanded corpus of novels by african american authors, with an emphasis on lesser known writers and a goal of expanding research in this field. using a custom metadata schema with an emphasis on race-related elements, each novel is analyzed for a variety of elements such as literary style, targeted content analysis, historical context, and other areas. librarians at ku have worked to develop a variety of computational text analysis processes designed to assist with specific aspects of this metadata collection, including text mining and natural language processing, automated subject extraction based on word sense disambiguation, harvesting data from wikidata, and other actions. by erin wolfe introduction the black book interactive project (bbip) is a grant-funded collaborative research project conducted by the history of black writing program (hbw) at the university of kansas (ku) [1]. hbw has its origins in 1983 as an effort to compile a comprehensive bibliography of novels by african american authors, and it has continued to evolve into the present [2]. since its inception, the main objective of hbw has been to increase the visibility and accessibility of these literary works for use in research, teaching, and other scholarly uses. bbip’s focus is on computational access to works by “writers from underrepresented communities, most [of whom] are unknown and understudied” for use in digital humanities and digital scholarship projects [3]. in an attempt to increase the number of african american writings being studied and taught as part of the traditional african american canon [4], and specifically, the types of digital humanities research being done using these, hbw has been working since 2012 on developing a digital library of lesser known works by african american authors. the university of chicago has been a primary partner in this, providing digitization services, ocr cleanup, and an interactive web interface using their existing philologic software to provide access and cross-text searching capabilities [5]. this ongoing partnership has so far produced more than 1,500 digitized novels, with cleaned ocr for over 900 texts. the primary goal of bbip is to make these texts available for scholarly use, increasing their visibility by juxtaposing them against more well-known writings. however, more than simply providing access to the texts, bbip wants to actively create avenues for discovery – “to engage in quantifiable analysis from a qualitative interdisciplinary perspective, identifying the specific properties of fictional data, authorship and epitextual elements that make the african american novel a distinctive cultural, linguistic and aesthetic practice.” [6] through detailed data about content and context, bbip hopes to bring forgotten and overlooked texts to light. by making this additional information available online, users will be able to use precise searching and filtering techniques, as well as some cursory data mining tools to uncover works relevant to their area of interest. metadata much of the information that bbip aims to capture is not easily mapped to currently accepted metadata standards for bibliographic information and is generally outside of the purview of traditional cataloging. some of these features are explicitly race-related, such as the ethnicity of the protagonist, or whether the text is set against socio-racial themes. others are traditionally important in the study of african american literature, such as the setting of the novel, or primary themes in the text. still other features serve as important historical context and avenues for further study, such as the literary predecessors and successors of the novel, or how the novel was received when it was published. to this end, hbw received a grant from the national endowment for the humanities in 2014 to facilitate the development of a custom metadata schema that considers race as a central thematic characteristic and to complete a pilot project to create records for a sample of these novels. with the help of partners including consultants from the ku libraries and ku’s institute for digital research in the humanities, and african american scholars from a variety of institutions, hbw developed a metadata template containing more than 50 elements that provides a means to capture these details in a systematic and structured way. hbw has since received additional grants to facilitate the creation of additional records and to fund scholars’ research using the materials [7]. the process of metadata generation is largely a manual one. bbip staff members (primarily a mix of doctoral, graduate, and undergraduate students) might employ any of the following methods: skim reading or close reading sections of the text (e.g. to determine narrative voice and thematic elements), keyword searching (e.g. to identify specific subjects, locations, etc.), general web searches (e.g. to find biographical information about the author), targeted online research (e.g. to find published book reviews in contemporaneous newspapers), among others. the thoroughness of this approach is the only realistic way to achieve the level of specificity desired by bbip for this material. however, the manual nature does not lend itself well to scalability. first of all, the process is resource intensive (staff could spend as many as 4-6 hours on a single book). furthermore, the application of the steps involved are not always applied in a systematic way, varying on factors such as inconsistencies in keyword search terms, online venues searched, or even the reviewer’s perception of the text [8]. code/scripting in an effort to see how computational tools might help bbip staff with aspects of the metadata collection, members of the ku libraries have developed several scripts, primarily using python libraries, that can perform some targeted tasks, including natural language processing (nlp), text analysis, and harvesting open source data. extratextual information from wikidata to gather extratextual information about the author, we developed a script to query wikidata for author entries and harvest relevant data points when possible. wikidata serves as a structured linked open data source for the wikimedia sites and is comprised of primary entries containing properties (represented by a ‘p’ number) and an associated value (represented by a uri based on either (a) a ‘q’ number that identifies a wikidata entry, or (b) a external linked data resource), returned as json from the api [9]. property value p106 (occupation) poet (https://www.wikidata.org/wiki/q49757) novelist (https://www.wikidata.org/wiki/q6625963) civil rights activist (https://www.wikidata.org/wiki/q1021386) p172 (ethnic group) african americans (https://www.wikidata.org/wiki/property:p172) p244 (library of congress authority id) https://id.loc.gov/authorities/n79076619 p4823 (american national biography id) https://doi.org/10.1093/anb/9780198606697.article.1600069 table 1. sample properties for author james baldwin (q273210) beginning with a list of author/title combinations, we query the wikidata api for author name matches. to disambiguate the query results, we filter the returned values by checking the description property for keywords, such as “writer,” “novelist,” or “author.” from the filtered results, we use the pywikibot python package [10] to parse the potentially complex json to look for property p214 (virtual international authority file (viaf) id). we then query the viaf api and retrieve the associated list of works by that author and compare it against our initial entry. if the title of our initial inquiry is in the viaf bibliography, then we can assume with a high degree of certainty that we have an accurate match and can then use pywikibot to harvest other relevant fields from the wikidata entry (11 fields in all, including birth place, dates of birth and death, occupation, education, and more) and write them into the local metadata for that item. additionally, we can use the title list from viaf to expand a bibliography of related works that may be used for further digitization possibilities, reference for researchers, or other future needs. the wikidata process is imperfect, as it requires the author to have entries in wikidata and viaf that meet the criteria above; however, we have so far achieved about 47% total successful matches (32% successful for unique authors). improved results could likely be achieved through further data cleanup or enhancement, such as adding birthdates from external sources or manual review to check for possible formatting differences. in addition, identifying gaps in wikidata sources can provide an opportunity for bbip to contribute missing information back to wikidata as we discover it. identification of specific topics one area targeted for textual analysis by bbip is the identification of specific topics within a text. for example: does violence play a role in the text? how is it depicted? to test with improving efficiency and scope of the manual review process, ku libraries developed a script that evaluates the entirety of the text for the presence of a given topic and outputs a web page with relevant details for review. this process utilizes nltk’s wordnet corpus – a widely-used lexical database that semantically relates words based on their meaning, or “sense.” from an input word, wordnet returns one or more possible senses of that word, called synsets (figure 1). wordnet consists of over 117,000 synsets that are enhanced with additional information, such as definitions, example usage, lemmas, part of speech, and more. these senses are interlinked through a number of conceptual relationships, such as synonymy or hyponymy (general-to-specific meanings), among others [11]. figure 1. wordnet’s definitions of the term “music” (enlarge) beginning with a manually generated list of keywords related to a specific topic, we filter all of the wordnet-returned synset/definition pairs to ensure that only the senses that we want are included. any senses that do not fit the desired topic are excluded, leaving only the synsets that match the topic for which we are evaluating. this review step is a manual process that only needs to be completed once but can be refined at any time. for example, see wordnet’s available senses for the term “music” (figure 1): to exclude the definition “punishment for one’s actions” (e.g. “face the music”), the synset identifier “music.n.05” is simply excluded from the final list [12]. from this list of initial synsets, we use wordnet’s relationships to build an active thesaurus of related senses. this is primarily done via the hyponym function (i.e., recursively collecting increasingly specific terms, such as ‘animal’ -> ‘domestic_animal’ -> ‘dog’), as this provides more precise sense matches. base term initial keyword list wordnet hyponym synsets music 46 1618 violence 53 1879 religion 58 1814 table 2. sample of expanded sense vocabulary using wordnet with this expanded list of synsets as our base, we pass every word in the text through an algorithm in wordnet called lesk, which determines a likely meaning for the word given the context in which it appears [13]. wordnet returns the lesk-generated synset, which is checked against our list of topic synsets, with positive matches recorded for later review. using this sense-based method, we can evaluate each individual word for meaning, rather than relying on keyword matches. at the end of the script, all positive matches are converted to an ordered html table, which can then be manually reviewed by bbip staff to make a determination on what should be recorded in the metadata record. figure 2. snippet of html output of sense matching, from checkmate and deathmate, by martin ashley (1973) (enlarge) additional functions identification of locations african american scholars have long identified setting and location as important elements of african american literature, with deep potential for research and for understanding the cultural context of african american literature [14]. named entity recognition (ner) is a common task in nlp which extracts specific names and concepts from the text – commonly persons, locations, organizations, etc. using a combination of booknlp (a pipeline of nlp tasks designed to work with long texts) [15] and litbank (an annotated dataset and neural network model for tagging entities in literature) [16], we can extract location information in a few different ways to allow the researcher a focused lens through which to view these elements of the text. first, we can group the cities, states, countries, and other locations mentioned in the text. figure 3. sample of locations grouped by type, from long old road, by horace cayton (1965) (enlarge) second, we can group locations according to the 2005 automatic content extract (ace) guidelines [17], which categorizes locations by conceptual type – specifically, (1) geo-political entity (a place with a population, physical or political boundaries, etc.), (2) location (a physical place without a political entity), and (3) facility (a man-made structure or place with a functional use). one of the features that makes litbank an excellent tool for this type of material is that it is trained to recognize literary language and can capture not only short entities like “egypt” and “the palace,” but also longer phrases like “the ancient cities of egypt” and “the illustrious house of the pharaohs.” as before, all of these grouping methods are output to an html file for review by bbip staff or researchers. figure 4. sample of locations grouped by ace guidelines, from a waif–a prince; or, a mother’s triumph, by w.t. andrews (1895) (enlarge) automated keyword extraction there are a number of nlp methods to automatically extract keywords from a text, ranging in approach and effectiveness depending on the tool used and the type of text in question. when dealing with a novel, this process can become difficult. keyword extraction in short texts, such as news articles or user reviews, is often reliant on specific word usage; however, in a novel, key themes often develop over the course of the work without ever being explicitly named. some nlp tools give less common words a heavier weight in determining their importance, resulting in proper names dominating the results of early testing. topic modeling can be a useful approach when working with groups of texts with distinctive topics referencing distinctive words. however, our experiences with topic modeling failed to produce any notably accurate or useful results. again, this may be due to the lack of explicitly stated keywords. better results for our purposes were achieved using the rake (rapid automatic keyword extraction) libraries [18], which tends to capture potentially interesting textual aspects of the work without necessarily identifying thematic elements. for this project, these results are grouped and output to an html file for review. further refinement could be done using a similar wordnet/lesk approach as described above, and then mapping identified synsets to a lexical type (or lexname). figure 5. sample keyword extraction using rake, from long old road, by horace cayton (1965) (enlarge) matching with hathitrust holdings another prospect being explored by ku libraries is the identification of titles in the bbip digital collection that are also held by hathitrust, a process which could allow researchers to make use of the hathitrust research center’s computational analysis tools to examine bbip texts [19]. this is done through matching on author/title combinations using the hathitrust bibliographic metadata set and retrieving the hathitrust identifier [20]. this process involves significant data cleanup (due to formatting variations between the sources) and then comparing results to find very close matches [21]. since the hathitrust data file is very large (nearly 17 million items), precise matching can be labor intensive and requires iterative adjustments to the input strings. current mapping efforts are achieving about 50 percent success, and efforts to improve this are ongoing. conclusion there have been a number of challenges using computational text analysis tools with the bbip texts. this was not unexpected, as many accessible nlp tools are not currently well-designed for the specific types of information that bbip is looking at. in addition, the bbip novels themselves contain characteristics that are relatively unique, such as the inclusion of regional dialect in a long text format. although vernacular has been a subject of some nlp exploration, particularly in modern day aave (african american vernacular english) and social media usage, we have not yet been able to find a reliable method for handling dialect from an nlp perspective as applied to 19th and 20th century novels [22]. however, we have been successful in our primary goal, which is to provide methods to help bbip staff or researchers extract specific types of information from a text, as well as harvesting open data from external sources, and to provide this data in an easy-to-access format. neural networks and transformers are currently the state-of-the-art approach for many nlp applications, and we are testing a few different models (including word2vec, bert, and xlnet) for existing and new tasks, including genre grouping, expanding the topic identification method, dialect identification, and others. the fields of natural language processing and machine learning applied to text are growing quickly, with new applications and methods being introduced at a rapid pace, and we plan for our approaches in computational analysis of these texts to keep evolving as well. about the author erin wolfe (edw@ku.edu) is the metadata librarian at the university of kansas libraries. he is active with the black book interactive project and is interested in text mining, computational humanities, machine learning, and related areas. references [1] black book interactive project homepage: https://bbip.ku.edu/ ; history of black writing homepage https://hbw.ku.edu/ [2] for a full timeline of hbw, see https://hbw.ku.edu/timeline [3] bbip statement of humanities significance: https://bbip.drupal.ku.edu/humanities-significance [4] according to dr. maryemma graham, head of the hbw, fewer than eight percent of known african american texts are included in the traditional canon of african american literature. [5] history of black writing novel corpus at the university of chicago, public/beta version: https://textual-optics-lab.uchicago.edu/black_writing_corpus [6] bbip statement of innovation: https://bbip.drupal.ku.edu/statement-innovation [7] bbip received the neh grant in 2016. in 2018, bbip applied for and received an additional grant from the american council of learned societies (acls), which allowed for continued progress in refining the metadata schema, collecting data for additional works, and funding a program to allow a group of scholars to access and use these materials in their research. see https://bbip.ku.edu/project-updates [8] although outside the scope of this case study, it should also be noted that, since some of the metadata is collected manually from historical sources, the availability of this information relies on those sources themselves being findable and accessible. several invaluable resources for finding contemporary book reviews or related information have been found, but the amount of resources that are not available electronically is unknown. [9] wikidata:introduction: https://www.wikidata.org/wiki/wikidata:introduction . there are a number of online gui tools to help identify which of the over 6,500 wikidata properties might be relevant to a specific data set, such as wikidata propbrowse – https://tools.wmflabs.org/hay/propbrowse/ [10] pywikibot usage with wikidata: https://www.wikidata.org/wiki/wikidata:pywikibot_-_python_3_tutorial/data_harvest [11] for more about wordnet, see https://wordnet.princeton.edu/. for usage of wordnet in nltk, see https://www.nltk.org/_modules/nltk/corpus/reader/wordnet.html. [12] this manual process is generally effective but not particularly flexible or efficient. current tests are underway to examine the feasibility of improving this process using word2vec or neural networks. several recent articles describe current efforts using neural networks that show promising developments in this area. c.f., y. levine, b. yoav, et al. “sensebert: driving some sense into bert” (august 2019): https://arxiv.org/abs/1908.05646v1 ; c. hadiwinoto, h. t. ng, & w. c. gan. “improved word sense disambiguation: using pre-trained contextualized word representations” (october 2019): https://arxiv.org/abs/1910.00194v1 [13] the lesk algorithm is useful but limited, relying on trained word order and co-occurrence and using the wordnet definitions to determine usage and meaning. in practice, this produces a number of false positive and false negative results, requiring human review to obtain fully trustworthy results. see previous note for future directions. [14] c.f., nunlye, v. l. “from the harbor to da academic hood: hush harbors and an african american rhetorical tradition” (2004) in richardson, e.b. & jackson, r.l. (ed.) african american rhetoric(s): interdisciplinary perspectives. carbondale: southern illinois university press. [15] david bamman, ted underwood and noah smith, “a bayesian mixed effects model of literary character,” acl 2014. [16] david bamman, sejal popat and sheng shen, “an annotated dataset of literary entities,” naacl 2019. [17] c.f., linguistic data consortium “ace (automatic content extraction) english annotation guidelines for entities,” 2008. https://www.ldc.upenn.edu/sites/www.ldc.upenn.edu/files/english-entities-guidelines-v6.6.pdf [18] c.f., rake (rapid automatic keyword extraction) library in python: https://github.com/vgrabovets/multi_rake [19] hathitrust research center: https://analytics.hathitrust.org/ [20] data files are updated regularly and can be downloaded from ht: https://www.hathitrust.org/hathifiles [21] currently, best results are achieved by measuring the jaro-winkler distance between strings using the jellyfish library. [22] c.f., a. jørgensen, d. hovy, & a. søgaard. “learning a pos tagger for aave-like language,” proceedings of the 2016 conference of the north american chapter of the association for computational linguistics: human language technologies, p. 1115-1120. (2016) https://www.aclweb.org/anthology/n16-1130.pdf subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – reviving digital projects mission editorial committee process and structure code4lib issue 5, 2008-12-15 reviving digital projects what do you do when you are in charge of assessing and reviving an abandoned digital project you had no part in creating or implementing? this article will talk about the unique challenges and issues involved in such a project, drawing from a specific example at the university of michigan library. we contended with unfamiliar software, limited technical documentation, proprietary file formats and platform migration, and will discuss how we approached each of these specific technical issues. after reviving our project and reflecting on our process, we put together a list of guidelines that we feel will help assist others who may find themselves in similar situations. by dianne dietrich, jennifer doty, jen green and nicole scholtz introduction librarians and professionals working in libraries are inheriting digital projects that they had no part in creating or implementing, including projects that have long since been abandoned. given the unique challenges of digital information and data, including hardware and software obsolescence, a lack of transparency, and incomplete or insufficient metadata, reviving digital projects is a complicated and complex task that will require new technology-specific approaches and methods. how do libraries address the assessment and revival of digital projects? what significant issues do they face throughout the process? often, when digital projects are abandoned, very little documentation survives for those who will ultimately take over the project. the original creators may have left the institution, and it can be difficult to find individuals with the specific technical expertise needed to revive the project. when librarians and information professionals encounter a project that has long been abandoned, how should they determine the immediate next steps for the project? how do stakeholders determine if a project is worth saving, and how are soon-to-be revived projects technically evaluated? if significant changes in computing trends suggest new technologies better suited to the type of project, how do libraries handle this shift? we will explore these issues and concerns, drawing from an example of a successfully revived digital project at the university of michigan library. in the winter of 2008, the arcims service that ran the online atlas of michigan plants was migrated from an aging sun server to a new server. the original database was established a few years prior, and its creator was no longer at the university of michigan. additionally, technical documentation for the arcims software was limited at the outset. a team of librarians and interns evaluated the soon-to-be decommissioned database, determined the best methods for transferring the existing data, and finally migrated an improved version of the database to a new installation of arcims on a new server. in reviving our own digital project, we considered ways libraries and librarians can better restructure the lifecycle of digital projects to make it easier for their successors to continue established projects. this includes strategies for documentation that specifically address the complexity of digital data, methodology for determining maintenance needs for newly established projects, and finally how to effectively delegate responsibility for the care and feeding of these projects, so that they can persist and add value to the services libraries provide. reviving the database history and background the online atlas of michigan plants[1] originated on an installation of arcims that ran on a sun server housed at the graduate library at the university of michigan. the online service provided a web-based interface for users to explore the geographic distribution of plant species throughout the state of michigan. its audience included researchers, students, and others interested in botany. in late 2007, the sun server was slated to be decommissioned, so it was crucial to determine the fate of the online atlas of michigan plants. there was still demand for the service, as spatial and numeric data services at the university of michigan continued to receive emails with questions about it, especially if the site experienced downtime. since there was a demonstrated need for the service, it was clear that it should not be discontinued when the sun server was taken offline. reviving the project, however, posed some considerable challenges. its creator, rachel simpson, was no longer affiliated with the university of michigan. while simpson did provide extensive documentation related to gathering and digitizing the spatial data included in the service[2] , there was a lack of specific technical information pertaining to the setup of the arcims software. the librarians in spatial and numeric data services, while skilled in the use of a number of gis applications, were not familiar with the arcims software. it was also necessary to locate a suitable new home for the service, but it was unlikely that another sun server would be available. migrating the service would definitely mean migration to a new platform. given these challenges, it became necessary to formulate a strategy to migrate the online atlas of michigan plants to a new home. there were several arcims services running on a windows server at the center for statistical consultation and research (cscar) at the university of michigan. this, then, was the ideal choice for the service’s new home. once the new location of the service was established, our group convened in early 2008 to discuss the immediate next steps for the migration.  our plan was to have the service migrated to the cscar server by the end of the term, in april 2008. we scheduled a weekly meeting time to keep all members apprised of the status of the project, and established a group blog to post questions, notes, and future plans to share with the team. understanding the original service, from the outside in our first objective was to determine as much as possible about the original service while it was still active on the sun server. this posed a significant challenge, as none of us had direct access to the sun server. in order to capture as much information about the service while it was still online, we opted to use techsmith’s camtasia screen capturing software to record a video as we explored the service’s web interface. we believe that video provided a significant advantage over static screen captures of the database interface, as it allowed us to record the sequence of events as we queried different parts of the database. we also received a collection of files from the original server, though we were not sure which files belonged to our service and which did not, as we had no documentation or metadata available for them. we knew, from our experience using other esri products, how to identify several of the proprietary file formats, and how to view their content. we viewed as many of the files as possible in a plain text editor in an attempt to make sense of their contents. to keep track of all of the files, we kept a list with the name of the file, a summary description of its contents, plus notes about our own assumptions as to how the files might be related to the michigan plants service. as we learned more about the arcims software, we became better able to understand how these files fit into the existing service. next, we faced an important question: was arcims the best choice for this service, given that online mapping software and web-based interfaces had changed dramatically since the service’s inception? arcgis server (also developed by esri) and the google maps api were two of the newer, more popular options for providing web-based access to geographical data. based on the functionality that we observed on the original service, the needs of the community that used the service, and the existence of other arcims services on the cscar windows server, we decided that, despite the newer options, arcims still remained the best option for this migration. keeping with the same software would allow us to provide continuity to our users, ensuring that the functionality they needed would not be lost in an unfamiliar interface. learning about arcims in order to gain more experience using arcims in its future server environment, we set up a virtual machine using vmware that duplicated the environment of the cscar windows machine. by using vmware, we were able to easily restore the virtual machine to a pre-install state, so that we were able to practice installing and configuring each of the required components for arcims (including apache and servletexec). once the virtual server was set up to our standards, we were able to create videos that captured every screen throughout the process. we also created detailed documentation describing each of the steps necessary to configure the arcims environment. our group blog served as an important record of our progress, as we noted any difficulties, frustrations or issues we encountered while working on the arcims service individually. it also served to provide the entire team with detailed explanations of how we resolved errors during installation and configuration of the service. we incorporated some of this information into our reference documentation for future maintainers of the service. figure 1: digital archeology can be tricky, and the blog enabled us to record our progress in deciphering the old service. [view full image] we were able to locate training and online tutorials for arcims, and we each completed the esri “virtual campus” series for the software. by completing the training modules, we developed a much better sense of how arcims functioned and, most importantly, we learned which files were used to create a complete, functioning arcims service. we were then able to review our original list of the files from the sun server, and determine precisely which ones we needed to re-institute our service. putting it all together once we had completed the online training for arcims, we had a good sense of what we needed to do in order to get the service up and running in its new environment. first, we created a test service using our virtual machine, comparing its functionality to the video we created of the original service on the sun server. when we were certain that we had captured everything from the first incarnation of the database, we easily transferred all of the map and configuration files to the new server and alerted the appropriate administrators to ensure that the service was visible on the web. we sent an email to a select number of users of the system, asking them to evaluate the possible changes and improvements we were considering for the service, and to provide us with any additional feedback. figure 2: we communicated what we learned about arcims that would be helpful to the rest of the team. [view full image] figure 3: we organized our ideas for potential improvements to the service. [view full image] future-proofing the project as the project came to a close, our last task was to provide documentation so that the future maintainers of the project could continue to keep the database alive. our group blog already provided us with a wealth of information regarding the installation and configuration of arcims, the structure of the michigan flora data, and suggestions for future improvements, but we knew we needed documentation that was streamlined and directly addressed the needs of the future maintainers. we wrote three separate documents that addressed how to install the software and configure it specifically for the online atlas of michigan plants, and how to create new services for additional map layers of data, as well as steps to follow to make modifications to existing and future instances. we created camtasia videos with narration that stepped through the process of installing arcims and configuring and customizing the software. we included a comprehensive list of all of the files necessary for the service, including the directory structure, and a short statement of their purpose. these artifacts provided all of the information one would need to reestablish the service. figure 4: we originally discussed the idea of using video documentation on the blog. we archived all of our blog entries, so that they could be consulted as a reference in the future. generalizable issues when reviving digital projects though abandoned digital projects can take many forms, by reflecting on our experience, we can generalize some of the issues we faced in reviving our digital project and provide a list of important questions to consider when tasked with reviving an abandoned digital project. evaluation when considering whether to revive a particular digital project, an important question is how to evaluate the current service provided by that project. evaluation includes determining if there is still a need for the service. if the project is located on the web, is it possible to obtain site traffic statistics to get a clearer sense of who is still visiting the page? is anyone still sending queries about the service to the posted contact person? who was the primary intended audience? are they likely to still use the service? (if the primary audience was a class, perhaps there is very little current use for the service.) another critical component of evaluation includes identifying limitations to the current service that must be changed if it is to be migrated. it might be necessary to initially pass over minor changes to the interface or usability issues, but errors that significantly affect the functioning of the current service must be resolved if the service is to be revived. it is important that updates and changes be prioritized, so that the migration and revival process goes more smoothly. technology another important issue facing professionals tasked with reviving a digital project includes determining if a new technology should be used. often, when a project has been abandoned for an extended period of time, the technology used to originally create it has become obsolete. many projects created from third-party applications soon become obsolete as technology changes. the software vendor may have discontinued the software, or the number of updates since the version used in the project causes a complete incompatibility between files and data. in this case, it is critical to determine how using an outdated piece of software may affect the future maintainers of the project. if the vendor is no longer providing support, is there still a user community present for assistance? (this user community may exist at one’s home institution, or may be located on the web.) if it eventually becomes necessary to find a new piece of software to house the project, how easy is it to extract the needed files and convert them to the formats required by a new piece of software? (some configuration files are written in plain text, or in xml, making them more portable than configuration files in binary file formats.) for certain digital projects, significant changes in technology trends suggest new technologies better suited to a particular project. a regularly updated page, previously maintained by client-side html editing software, may be better served by blog software installed on a web server. in this case, it is critical to have tools in order to batch process multiple files for migration. it becomes important, when facing projects such as this, to survey the technological landscape, and examine the multiple new ways communities provide similar services. which are closest to the spirit of the original digital project? what new features are available now, that were not available when the service was created? is it possible to incorporate these new features with the original service? issues related to programming languages and platforms become important here as well. if the project consists of old code, what language is it in? is there still expertise (and anticipated continuing expertise) in that language at one’s institution? since the project’s inception, has there been policy enacted that standardizes the programming languages used in digital projects? how much time and effort would it take to convert a project from one programming language to another? can the project continue in its current environment or is it on a particular piece of hardware or software that is now outdated? if the project is to be revived, will there be compatibility issues between the old environment and the new environment that will complicate migration? will it be possible to find a new home for the service? migration when individuals inherit abandoned digital projects, often they have limited access to the original project. one of the most difficult tasks in the revival process is capturing the functionality of the older service without having direct and full access to it. it may be necessary to reverse engineer the service, by testing out each piece of it. it may be helpful to invest in screen capture software to record the results of interacting with the service. this may be even more helpful than notes, as it doesn’t rely on interpretation, and what was cryptic in the past may become clearer as one gets a more complete understanding of the project. is it possible to obtain all of the files used for the project? it may be necessary to examine, with a text editor, every file that can be read in that way. diagrams of directory structures and notes for each file will prove invaluable later in the documentation stage. for binary files of unknown type, it may be possible to use utilities (such as “file” or “strings” in linux) to ascertain the identity and content of any files that do not contain plain text. lists of filetypes, including older and obsolete extensions, are readily available on the web. restructuring the process how can libraries and librarians better structure the life-cycle of digital projects to make it easier for successors to continue established projects? reflecting on the information we needed to revive the online atlas of michigan plants, we can offer a few suggestions for those implementing new digital services. it is critical to provide extensive documentation. while one of life’s more onerous tasks, without any documentation, it becomes impossible for a digital service to be maintained long-term. it is important to know how the service was installed, what files are required for its functioning, how to interact and use the service, and how to troubleshoot common errors. this documentation need not be solely written. we found it helpful (and far quicker) to provide video and audio documentation, especially when describing installation and configuration processes. if it is not possible to create documentation as the project is being built, consider setting up a blog to allow for the opportunity to document along the way. any files that are required for the project should, if possible, be readable in plain text. written description should accompany any non-text, or binary, or proprietary file. if a specific directory structure is important, then this should be diagrammed as best as possible. consider creating documentation (both written and audio-visual) that takes a new user through the process of reassembling the service from its components on a new machine. this will provide nearly all of the critical information needed to maintain a project long-term. if the project is coded, realize that, at some point, it may be obsolete. are any standards put in place at the institution informing the creation of maintainable code? if not, create some general guidelines, and document these for future use. any non programming language specific notes or diagrams, if preserved, will provide future maintainers with a way to conceptualize the original vision for the project, without necessarily having to understand all of the technical details. finally, it is important to identify individuals who are responsible for specific maintenance tasks. simply stating that one individual is responsible for maintenance is insufficient. distinct and identifiable tasks should be enumerated, and individuals should be assigned to each of them. an example of a vague task is “updates software.” a better crafted, more specific task can be, “monitors security warnings and updates and applies necessary patches.” specific statements make it easier to identify who is responsible for all of the various maintenance tasks needed to keep the service up and running. conclusion it is important that digital projects persist, to add value to the services that libraries provide in order to connect users with information. no matter how complete the original planning was, some projects may end up in an abandoned state and will require revival months or years later. it is our hope that this document provides help and guidance for those tasked with bringing an aging digital service back to life. additionally, our insights may be able to help those currently building new digital projects determine how best to provide all of the necessary information to the future guardians of these projects. notes online atlas of michigan plants. (2004). reznicek, a.a., voss e.g., & simpson, r. a. university of michigan. http://herbarium.lsa.umich.edu/website/michflora/. accessed september 16, 2008. simpson, r.a. (2007). mapping herbarium specimens: a case study using locality information from the university of michigan herbarium’s michigan flora database. the michigan botanist, 46, 1-13. (coins) about the authors dianne dietrich (dd388@cornell.edu) is the research data & metadata librarian at cornell university. jennifer doty (jendoty@unc.edu) is the city and regional planning librarian at the university of north carolina at chapel hill. jen green (greenjen@umich.edu) is the spatial and numeric data services librarian at the university of michigan. nicole scholtz (nscholtz@umich.edu) is the social sciences and spatial data librarian at the university of michigan. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – special issue on diversity in library technology guest editorial committee mission editorial committee process and structure code4lib issue 28, 2015-04-15 special issue on diversity in library technology guest editorial committee the guest editorial committee for code4lib journal’s special issue on diversity in library technology (issue 28) was developed in order to include new voices and perspectives on the journal’s practices and how they support inclusivity. the committee is comprised of eight guest editors and two regular editorial committee members. more information on the development of […] the guest editorial committee for code4lib journal’s special issue on diversity in library technology (issue 28) was developed in order to include new voices and perspectives on the journal’s practices and how they support inclusivity. the committee is comprised of eight guest editors and two regular editorial committee members. more information on the development of the guest editorial committee can be found here. shawn averkamp, manager of metadata services, nypl labs at new york public library, oversees metadata production and directs the development of metadata infrastructure for nypl’s unique digital resources. prior to joining nypl, she supported metadata management, digital humanities, and data curation at the university of iowa libraries as data services librarian and interim head of digital research & publishing and as a metadata librarian at the university of alabama libraries. she earned her mlis from the university of iowa and holds a ba in music from luther college. laurie bridges, associate professor and instruction & emerging technologies librarian at oregon state university, received her mlis in 2006 from the university of washington and an ms in college student services administration with a minor in women studies in 1999 from oregon state university.  she has co-authored several articles about mobile technologies and libraries, including a 2009 article in code4lib. scott carlson, metadata coordinator at rice university’s fondren library, received his mlis in 2007 from dominican university in river forest, illinois, and recently completed an archives certificate in digital stewardship from simmons college in boston. prior to his position at rice, scott was the cataloging and metadata librarian at the american university of sharjah, an accredited, multicultural institution in the united arab emirates. tara das is the government information librarian at columbia university. she previously was a director in the nyc department of health & mental hygiene where she managed electronic records, operations, and outreach for vital records. she has a joint phd in anthropology and political science, and an mph in quantitative methods. she is completing an mslis at pratt institute. heidi dowding, issue 28 coordinating editor, has worked as a librarian and researcher around the world.  her most recent work at the royal dutch academy of arts and science’s huygens institute for dutch history has involved researching long-term digital preservation interventions for digital scholarly editions.  previously she worked as a national digital stewardship resident at the library of congress and reference and digital services librarian at nazarbayev university. jane foo, digital systems librarian at seneca college, credits an undergraduate course on environmental psychology with sparking a lifelong interest in human-computer interaction and the impact of technology and visual design. her experiences include testing, user-centred design and usability testing, adult teaching and training, and a large variety of systems implementation and management projects. lately, she’s gone back to her undergraduate statistics roots and is pursuing a mooc based data science specialization. shirley lew is the interim director, library and learning centre at vancouver community college. her usual position is coordinator, library systems and technical services. she has taught courses in library automation and is also active in the local literary arts community. yuan li is the scholarly communications librarian at princeton university, where she manages the princeton university library’s efforts to support scholarly publication innovations and reforms, and supervises and coordinates activities related to the princeton open access policy and the princeton institutional repository. prior to joining princeton, she served as the scholarly communication librarian at syracuse university, the digital initiatives librarian at university of rhode island, and the digital repository librarian at university of massachusetts amherst. yuan has a master degree in library and information science from the university of rhode island, a master of engineering degree in applied computer science from the national computer system engineering research institute of china and a bachelor of engineering degree in computer science and technology from yanshan university (china). renée mcbride, head, special formats & metadata section, resource & description management department at unc-chapel hill, received her ma’s in music theory and library & information science from the university of iowa. at unc-chapel hill she supervises catalogers responsible for audiovisual, cartographic and music materials, geospatial data and statistical datasets, and resources for the carolina digital repository. prior to her current position, she worked at hollins university, ucla and the university of oklahoma. renée’s passion for diversity in the lis world spans two decades, from ucla to the music library association to unc-chapel hill, where she is currently co-chair of the library diversity programming and education committee. ethan pullman is a humanities librarian & coordinator of library instruction at carnegie mellon university, where he is also pursuing his phd in rhetoric.  he held various positions as a public services librarian at the university of pittsburgh and adjunct instructor of arabic and library and information sciences special topics in the humanities.  he received two awards for excellence in teaching, support, and providing significant impact and guidance to students’ academic and extracurricular lives from students and staff at both universities. more information about his professional involvement is available here.   why diversity in library technology? shawn averkamp: i got involved in library technology by fluke. the library school i planned to attend had received an imls grant to train new digital librarians. i had little experience in technology, but i needed the funding, so i applied. i was incredibly fortunate to have a diverse string of mentors who not only helped me navigate my early career in libraries, but created and fostered safe spaces to explore technology, ask questions, and fail without negative consequence. my interest in diversity in library technology comes from my gratitude to mentors and a sense of responsibility to pay it forward by supporting and creating welcoming spaces for all voices in this field. as a regular member of the code4lib journal editorial committee, i was also interested in learning from our guest committee how to push our limits of the scope of this journal as well as how to adjust and improve our processes to make the journal a more welcoming venue for potential authors and editors from all backgrounds. laurie bridges: during my childhood, dinner conversations between my parents usually revolved around computers. my dad was an air force programmer who worked with arpanet beginning in the 1970s and he eventually obtained a bs in management information systems in 1987; my mom was also a programmer who got her bs in computer and information systems in 1983. in addition to talking about computers my parents routinely discussed my mom’s experiences as a woman in programming. while my dad’s career in programming and eventual management spanned decades (he retired only two years ago) my mom’s was short-lived and lasted approximately 15 years; she eventually opted out of the male-dominated profession. my mom is a brilliant woman, but she left the field of programming disillusioned as a direct result of the negative workplace gender-dynamics she experienced every day. serving on this editorial committee is a tribute to my mom, and it is my hope that this issue of code4lib can be a step toward revealing the accomplishments and struggles of marginalized persons and groups within (library) technology. scott carlson: my interest on this topic stems from a career where i have been tasked with providing as much of a common background as possible to the people i’ve worked with. in the united arab emirates, i provided technical service guidance and training to a team of paraprofessionals with a wide array of library experience and cultural backgrounds. nowadays, my position at rice affords me the opportunity to liaise with all of my fellow library departments — as well as the rest of the university — on issues related to metadata, not all of whom may be aware of what it is (or how it might be put to work for them). what all of this has taught me is that an array of backgrounds enriches the work that we do and the ways in which we do it. i am eager to participate in a larger discussion on diversity in library technology. tara das:  i come to the libraries after having worked in public health research and technologies. in my work, i have always considered how technologies will be experienced and acted upon by users and user groups. i have many interests in library tech, but the underlying themes have been examining technology trends (and their implications for different groups) and openness to diverse uses, interpretations, and needs.  my own research currently focuses on critical examination of trends in government use of social media and open data.  having worked in new york city government, and now in an nyc academic library, i am comfortable working with diverse people and needs where available technologies can be used and/or re-purposed in unanticipated ways. people may not view technologies in the same way, and it will strengthen our programs and services if we enjoy and engage with these differences. heidi dowding: on a personal level, i am a first generation college student and have a great interest in helping similar students as an academic librarian.  as a graduate student assistant on the reference desk at wayne state university in detroit, i worked with underserved communities and saw how even basic technical skills can open up an entirely new world in terms of employment and education.  as one of the newest members of the code4lib journal editorial committee, i have been thrilled by how supportive the committee is of new ideas within our small group.  in pushing for this special issue, i wanted to see how that could be expanded to support the discussions that have been happening in the code4lib community and beyond. jane foo: the ultimate goal of talking about diversity in library technology is to eliminate the need for this very topic. while i believe that the library technology community is diverse, it is obvious that the public side of library it (publications, presentations, and leadership) lacks variety. it is time to shape the existing knowledge sharing processes and policies within this community into a more inclusive and supportive set of practices that encourages new ideas and opinions from the less vocal and visible members, and i look forward to aiding code4lib on the special issue on diversity. shirley lew: i have held technical positions in libraries for the majority of my professional career. i have worked in environments and engaged in technology communities where i am in the minority in terms of race and gender. i am interested in understanding the reasons for this, the immeasurable loss due to conscious and unconscious systems of exclusion, and how we go beyond diversity metrics to meaningfully transform the environments in which we work and innovate together. i hope this special issue will deepen our collective understanding and appreciation of diversity in the code4lib community. yuan li: as an asian american woman, i always pay a close attention to the diversity issues of the organization where i work(ed).  it’s not only because i am the one who will likely get impacted by the issue, but also because of the importance and benefits of diversity to an organization.  i had been a software engineer and a web developer before. and i am now a librarian. my experience allows me to observe the diversity issues across the professions, to learn how the diversity can be improved, and to witness the benefits of the improved diversity to an organization. in many library tech situations, there is still gender imbalance, far more males than females. i am very interested in helping improve the situation. as roy tennant said in his article “fostering female technology leadership in libraries”, “it doesn’t need to change for (simply) equity reasons, but because we need women in library technology. diversity in your workforce is a good thing. diversity of perspectives, skills, experiences, and ways of working strengthen any organization.” renée mcbride: i was first struck by the positive impact of openness to diversity in libraries in 1994 when i interviewed at ucla, where i was asked how i might contribute to diversity in the library. it was a joy to interview as the comfortable, open lesbian i was. eventually i served on the ucla library committee on diversity and as a diversity representative on a wide range of library search committees. diversity plays a role in my service to the music library association (mla), for whom i have served as publicity & outreach officer, which involved focused efforts at broadening our outreach, and on the arl/mla diversity scholarship publicity task force, which dealt with a scholarship awarded by mla and the association of research libraries (arl) under the arl/mla diversity & inclusion initiative. in my current position at unc-chapel hill, i serve as co-chair of the library diversity programming and education committee, through which i have been delighted to see diversity become an increasingly vital part of our library’s strategic planning. i am honored to continue my work in support of diversity in the lis world as a guest editor for this issue ofcode4lib. in case any of us wonder if there is indeed a need for more diversity in the technology community, i want to share that just last week an asian friend had to send her regular, quarterly email to her it department to remind them to correct her name, which is incorrectly displayed at the beginning of every quarter at her institution. as my friend noted, this is “a systemic problem within the field of programming that continues to insist on standardizing information systems to certain cultural norms.” ethan pullman: i have been an early promoter of online learning systems such as blackboard and have used it extensively in my teaching.  my expertise lies in online/distance learning pedagogy and best practices, including assessment in online environments.  my interest in diversity is demonstrated in creating and coordinating non-english library instruction sessions taught by a team of librarians at the university of pittsburgh,  working in public and academic libraries, including a semester as head librarian of a ship library (semester at sea), and teaching at carnegie mellon university’s satellite campus in doha, qatar. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – from text to map: combing named entity recognition and geographic information systems mission editorial committee process and structure code4lib issue 49, 2020-08-10 from text to map: combing named entity recognition and geographic information systems this tutorial shows readers how to leverage the power of named entity recognition (ner) and geographic information systems (gis) to extract place names from text, geocode them, and create a public-facing map. this process is highly useful across disciplines. for example, it can be used to generate maps from historical primary sources, works of literature set in the real world, and corpora of academic scholarship. in order to lead the reader through this process, the authors work with a 500 article sample of the covid-19 open research dataset challenge (cord-19) dataset. as of the date of writing, cord-19 includes 45,000 full-text articles with metadata. using this sample, the authors demonstrate how to extract locations from the full-text with the spacy library in python, highlight methods to clean up the extracted data with the pandas library, and finally teach the reader how to create an interactive map of the places using arcgis online. the processes and code are described in a manner that is reusable for any corpus of text by charlie harper and r. benjamin gorham using spacy to extract place names what are named entities? a named entity is a “lexical unit referring to a real-world entity in certain specific domains, notably the human, social, political, economic and geographic domains, and which have a name (typically a proper noun or an acronym)” (novel, erhman, and rosset 2016, appendix 5). in practice, named entities include traditional proper nouns, such as person and place names, as well as currencies, dates, named events, and numeric expressions. the definition also extends to field-specific contexts. for example, in medicine, named entities might include genes, pharmaceuticals, chemical compounds, or pathogen names. the extraction of named entities from a text, typically called named entity recognition (ner), can be accomplished with a variety of methods, including regular expressions, statistical models, or neural networks. numerous python libraries with pre-built ner capabilities exist. in this article, we employ the spacy library for ner (https://spacy.io/usage/spacy-101). the spacy library is free and open source, processes text quickly since it is built into cython, boasts strong model accuracy, and has a low barrier to entry. getting started with spacy after installing spacy using your package manager of choice (typically with pip install spacy or conda install spacy), you will need to download a language model before you can use the library. as of writing, spacy has three english language models and additional models for dutch, german, greek, french, italian, lithuanian, norwegian, portuguese, and spanish, as well as a multilingual model designed to work with mixed texts. the english models come in a small, medium, and large variety, with increasing size, features, and accuracy. we will work with the small english model, which is very lightweight at 11 mb. in contrast, the large model is 746 mb because of added features such as word vectors, which are high-dimensional numerical representations of words. the difference between models, however, is negligible for ner purposes since the ner f-scores, which estimate the model’s accuracy in correctly identifying named entities, are comparable, measuring 85.43 and 86.40, respectively. python -m spacy download en_core_web_sm to begin using spacy in python now requires only two lines of code. import spacy nlp = spacy.load('en_core_web_sm') if you’d like to work with other languages, you can download additional models (see https://spacy.io/models) by substituting the model name for en_core_web_sm in the above download and load syntax. processing a text by default, a spacy language model uses a pre-established text processing pipeline (figure 1) which tokenizes the text (i.e. breaks the text into discrete sentences, words, and punctuation), lemmatizes words (i.e. takes their written form and transforms them to their dictionary entry, such as changing both “writes” and “write” to “write”), tags parts-of-speech, parses syntax dependencies (i.e. how one word connects to another, such as marking an adjective as modifying a particular noun), and labels named entities. figure 1. the default spacy processing pipeline (from https://spacy.io/usage/processing-pipelines) although you can remove parts of this pipeline or add additional components as you see fit, there is immense value in this default behavior; to generate all of this data for a single text, you only need a single call to the spacy object we created when loading the language model! import spacy nlp = spacy.load('en_core_web_sm') doc = nlp('this is an example of some text to be processed by spacy.') the variable doc now holds a spacy doc object (https://spacy.io/api/doc) that acts as a container for individual token objects. the first and easiest way to interact with the processed text is by looping through the document’s tokens and looking at the properties of each. for example, the following will print out each token’s lemma and part-of-speech. note that the ‘_’ appended to the property names returns a unicode string; without ‘_’ appended to the end of each property, integer values are returned. each token object has numerous other useful properties and the reader is encouraged to explore these further (https://spacy.io/api/token). import spacy nlp = spacy.load('en_core_web_sm') doc = nlp('this is an example of some text to be processed by spacy.') for token in doc: print(token.lemma_, token.pos_) output: this det be aux an det example noun of adp some det text noun to part be aux process verb by adp spacy noun . punct extracting place names since we are specifically looking at ner, the token property ent_type_ is most relevant. if the token is a named entity recognized by spacy, this property will store the named entity type. default language models in spacy currently recognize 18 types (https://spacy.io/api/annotation), two of which refer to spatial entities: gpe and loc. gpe are geopolitical entities, such as cities, states, and countries. loc are locations that are not geopolitical entities, such as named bodies of water or forests. one way to gather all spatial named entities from the doc object is to loop over each token, check its ent_type_, and then build a list of these tokens. luckily, the spacy doc object provides a shortcut that speeds this process up. rather than looping over every token (many of which will not be named entities), we can use the doc object’s ent property, which is a list of all named entities found in the doc. each of these named entities provides the text, the entity type, and the start and end token number (an named entity can be multiple tokens long, such as ‘united states of america’). import spacy nlp = spacy.load('en_core_web_sm') doc = nlp('ben and i moved to the cleveland area after school.') for ent in doc.ents: print(ent.text, ent.label_, ent.start, ent.end) output: ben person 0 1 cleveland gpe 6 7 using a list comprehension, we can streamline this to succinctly build a list of all places (gpe and loc entities) found in a text. import spacy nlp = spacy.load('en_core_web_sm') doc = nlp('ben and i moved to the cleveland area after school.') places = [ent for ent in doc.ents if ent.label_ in ['gpe', 'loc']] print(places) output: [cleveland] building a .csv of place names in a corpus to apply spacy’s ner to a corpus of texts—in this case, the 500 randomly sampled articles from the cord-19 dataset—we need to add a bit of code to loop over our texts and then store the results in a single .csv. import spacy import pandas as pd import os txt_dir = 'text' nlp = spacy.load('en_core_web_sm') places = [] for fn in os.listdir(txt_dir): print(f'processing {fn}') with open(f'{txt_dir}/{fn}') as f: doc = nlp(f.read()) places.extend([[fn, ent.text, ent.start, ent.end] for ent in doc.ents if ent.label_ in ['gpe', 'loc']]) df = pd.dataframe(places, columns=['file', 'place', 'start','end']) df.to_csv('places.csv', index=false) this code block is extremely useful. by filling a directory with .txt files, you can quickly generate a .csv of places listed in those files. moreover, by changing the list of entity labels that the list comprehension uses (currently [‘gpe’, ‘loc’]), you can actually generate a .csv of any combination of spacy’s recognized named entities! cleaning place names and preparing for geocoding if we open ‘places.csv’ and peruse the results, we can see that not everything spacy has captured is actually a place name, such as “fatigue”, “appendix s1”, or “d?pois(?t”. like all approaches, spacy’s ner is not perfect and the results require some level of cleaning. there is no foolproof way to remove spurious named entities, but by examining the data we can infer that we might apply a few rules to remove bad places. namely, those that are all lowercase or contain non-alphabetic characters should be dropped. import pandas as pd df = pd.read_csv('places.csv') def to_drop(place): if not all(c.isalpha() or c.isspace() for c in place) or place.islower(): return true else: return false df['to_drop'] = df.place.map(to_drop) df_cleaned_places = df[df.to_drop==false] df_cleaned_places.to_csv('cleaned_places.csv', index=false) unique_places = df_cleaned_places.place.unique() df_unique_places = pd.dataframe(unique_places) df_unique_places.to_csv('unique_places.csv', index=false) two .csvs are saved in this code block. after dropping those records that are all lowercase or not alphabetic, a .csv of all cleaned places is saved. next, to prepare for geocoding the place names, a second .csv of only unique names is saved. this .csv will act as the input for geocoding places and ensures that the same place is not processed and mapped multiple times. mapping place names with gis once the data has been extracted and sufficiently cleaned, you can then begin the process of transforming it into a geospatial format from a .csv format. in this tutorial, we use arcgis software and arcgis online tools in order to generate geospatial data using a process known as geocoding. what is geocoding? geocoding is a workflow by which cells in a tabular data file are compared to entities with known geospatial locations via a tool called an address locator. the address locator reads the string of each cell in a user-defined selection and scans for matches in a reference dataset. if a match is found, the location data is appended to the input entry and a match score assigned based on the level of confidence that the data in the cell indicates the entity in the address locator. for example, an input cell with the string “123 main st.” would return a higher match score than a cell with “123 main” when matching against a reference data entry that reads “123 main street.” the geocoding process utilized in the current study employed pre-built tools available in the arcmap and arcgis online platform, and the steps by which the results were achieved are outlined below. it is important to note that geocoding a table of addresses with arcgis online, arcmap, or arcgis pro is not a free service, and consumes credits that can be purchased ad-hoc or as part of an institutional subscription. pricing is variable by plan, but regardless of how they are obtained, geocoding 100 addresses consumes 4 credits. a full description of credits can be found here: https://doc.arcgis.com/en/arcgis-online/administer/credits.htm when geocoding a table of locations or addresses via this process, users rely on an address locator to assign geospatial matches to entries in the input table. address locators are available in a variety of styles, including single field, dual range, city state country, and many more. figure 2. partial list of address locator styles, adapted from https://desktop.arcgis.com/en/arcmap/10.3/guide-books/geocoding/commonly-used-address-locator-styles.htm regardless of the style of address locator, the functional principle of its operation is the same. a collection of reference data forms the structure of the locator, providing a set of features to be matched against. reference data may consist of point features representing cities, line features representing road segments, or polygons representing administrative boundaries such as census tracts. each of these features includes string values describing or defining the feature, so a point feature describing cleveland, oh, may include the fields and values “city or town: cleveland,” “state or province: ohio,” and “country: united states of america,” among others to fully describe the feature. the reference data needs to cover the full geospatial extent of the study data. if locations within the input dataset fall outside the scope of the reference data (for example, the reference data consists of line segments representing streets within the continental united states, but the .csv file to be geocoded contains addresses in canada), the address locator will fail to find matches and will result in gaps in the output. for the present tutorial, we use the arcgis world geocoding service, a pre-built service that employs a robust address locator appropriate for finding locations all over the world. the data scraping and cleaning process described above results in a .csv file with a single field that describes the location of each entity. such a format is ideal for single field address locator styles, but can also be used in other styles that expect (but do not require) multiple field inputs. for many address locators, more precise locational data will be returned if the .csv has multiple fields including street address, street name, city, state or province, country, etc. the address locator will match the string values from these input fields against the full range of entries in its reference dataset and can return multiple matches for each entry. for example, a string value of “100 main st., springfield” with no postal, state, or country identifiers will produce multiple potential matches that the user can select from after the geocoding has completed. the geocoding process in the present tutorial was performed using the arcgis online interface. two workflows to achieve the same end result will be described below, one using a desktop installation of arcmap 10.8 and one using the arcgis online interface. option 1: geocoding in arcmap to geocode a table of addresses or locations via this method, the user will need the following: a working installation of arcmap 10.6 or higher. an active internet connection. an arcgis online account with the ability to spend credits. first, open arcmap, open a blank map document, and sign into an arcgis online account with access to credits. this can be done via the file menu once the map document has been opened and the machine is connected to the internet. you may wish to add the geocoding toolbar to your arcmap workspace to facilitate access to the geocoding tool. the tool can also be found via the search window (search: geocoding) or via the arctoolbox interface (geocoding tools). the toolbar can be added to the workspace by navigating to customize>toolbars>geocoding. figure 3. in the main menu toolbar, navigating to customize>toolbars will show a full list of available toolbars that can be activated for your mapping session. on the geocoding toolbar that you have now activated, select the mailbox icon (geocode table of addresses). the tool prompts the user to select an address locator to use, and supplies two default options, the arcgis world geocoding service or the military grid reference system. you may also add other address locators to the list if they have either created them from reference data or obtained them from other sources, but that is outside the scope of this article. for thorough documentation on building an address locator, see https://desktop.arcgis.com/en/arcmap/latest/manage-data/geocoding/creating-an-address-locator.htm figure 4. clicking on the mailbox opens the address locator selection, and after selecting your address locator you can browse to a locally saved .csv to geocode. the geocoding interface for the chosen address locator (arcgis world geocoding service) appears, and the next step is to input the .csv dataset of addresses that is ready to be geocoded. navigate to the correct file in the “address table” input field and choose “add.” depending on how the data has been formatted, you can now choose to geocode using a single field or multiple fields. the single field option will read string data from a single column of cells and match each entry against the strings in the corresponding fields of the address locator’s reference data. single field data can be as simple as the name of a country or city, or it can take the form of full address information with comma-separated values distinguishing between city, country, zip, etc. choosing multiple field geocoding suggests that the dataset has multiple columns of string values that represent different components of an address. one field stores street addresses, the next stores the name of the city or town, another stores the state or province, etc. figure 5. the right column of the geocoder window allows you to specify which field names in you .csv you want to be checked against fields in the address locator. if your .csv does not have a corresponding field for every field in the address locator, those can be left blank. for single field geocoding, click the dropdown menu and ensure that the location field is selected. for multiple field geocoding, map each of the address locator fields to their corresponding columns in the input .csv file, and leave any unused fields on the “<none>” value. the geocoding process reads the inputs from the .csv and matches them against the reference data to produce an output shapefile of point features. there are two options for how this output shapefile will be saved: a static file with the .csv data appended to the corresponding geospatial features or a dynamic feature class that includes a “relate” to the data in the .csv file. relates may always be added later, so we recommend the first option for the sake of simplicity. choose your preference, specify the name and location of this output shapefile, and run the tool by clicking “ok.” the tool will display the matching process and record the total number of matched, tied, and unmatched addresses. tied addresses indicate the address locator found multiple possible matches with the same score, such as “123 main st” and “123 main street.” when the process is complete, you have the option of “rematching” the data to manually correct any unmatched address or clarify any tied addresses if necessary. the resulting output consists of a point shapefile with each feature corresponding to a location from the input .csv file that has been matched with coordinates from the reference dataset of the address locator. you can inspect the feature data by right-clicking the layer in the table of contents and selecting “attribute table” to confirm that the locations represented are correct. figure 6. screenshot of the arcmap window displaying the geocoded addresses as points on the map. the attribute table for the geocoded list of addresses is accessible by right-clicking the layer. at this point, the map document can be saved and uploaded to arcgis online as a web map. to create a webmap, you should first save the map document locally as a .mxd file, the default filetype for arcmap documents. within the saved document, search for the “mxd to web map” tool in the server>publishing toolbox. this tool publishes a web map to the user’s arcgis online content with the layers and symbology as it is currently saved in the local arcmap document. the “mxd to web map” tool allows users to specify the name of the output web map (which can differ from the name of the locally-saved .mxd if you want) and requires at least one tag to be attached to assist in categorizing the web map. a brief summary is also required to describe the web map. if the saved .mxd has more than one layer, you can specify which layers you want to appear in the web map, and then click “ok” to publish the web map to your arcgis online content. figure 7. the search window and arctoolbox are accessible from the standard toolbar, and will open windows allowing you to explore or search for the full set of tools available in arcmap. pictured is the mxd to web map tool window. this process results in a web map saved in your arcgis.com content. the next section describes an alternate method to achieve the same result. option 2: geocoding with arcgis online another, far less complex method of geocoding a table of addresses is to do so directly on the arcgis online platform. whereas the previous method involved local installations of software and searching for tools to use, this method simply requires the user to upload the .csv file to an open web map. to geocode directly in arcgis online, the following are required: an active internet connection. an arcgis online account with the ability to spend credits. the first step is to sign into your account on arcgis.com. from your arcgis.com home page, click “map” from the menu at the top to open a blank web map to which the data can be added and geocoded. a blank map will open displaying a default basemap of the world. content, including the table of addresses to be geocoded, can be added to the map by clicking the “add” button and selecting “add layer from file.” in the “add layer from file” dialog, choose “browse” and navigate to where the .csv file has been saved, and choose to import the layer. figure 8. the map interface on arcgis.com, showing a blank map canvas with default basemap. data can be added using the add button at the top left. the following dialog presents an option of how the .csv file should be imported; as a raw table or as data to be geocoded. depending on the geographic range of your input data, choose the appropriate coverage in the drop-down menu: we chose “world” to ensure that the input data will be compared to reference data from the entire world. to geocode the locations in the table, choose the “locate features by” option that corresponds to the format of the locations. if the table uses latitude and longitude, select “coordinates,” and if the table uses place names or addresses, choose “addresses or places.” it is essential to then confirm that the field names in the input .csv file are being correctly mapped to the fields in the reference data. depending on the format of the data, select each line in the “location fields” column and ensure that it matches the field from the input dataset. figure 9. the fields from the locally saved .csv are visible on the left, and the fields they will be compared against are visible on the right. select a matching field for each of the input fields you wish to use during the geocoding process. when the fields have been appropriately mapped, click “add layer” to run the geocoding process. the entries in the .csv file will be checked against the reference dataset by the address locator based on the selections described in the above paragraph, and an output layer will be generated that displays the geospatial locations of each matched entry. you may now modify the style or symbology of the features in the layer to your preferences; such changes may be made at any point in the future as well. the feature layer also retains the data from the .csv file in its own attribute table, which is viewable and editable in the content pane of the web map by mousing over the layer and clicking “view attribute table.” linking the article data to the map now that the unique places have been geocoded, we can link data on the articles that mention them. in this case, we will upload our cleaned_places.csv and link the article filename to each place. you can go further to link any type of data you’d like, including more detailed metadata on each article. to upload the cleaned_places.csv, in the top left home dropdown, click content. under the loaded content page, the “add item” button will let us upload our .csv from our computer. we’ll title this layer “cord-19 articles” and under the “locate features by” heading, we’ll select “none” since this file has no coordinate data stored within it. once this upload finishes, we can return to the map and then click “add -> search for layers.” the “cord-19 articles” layer will appear and can be added to the map. figure 10. the upload screen to add our cleaned_places.csv to arcgis online. to link this .csv with filenames to places, use the analysis button along the top toolbar of the map. this opens the analysis pane, with a variety of options. under “summarize data” click “join features.” our target layer will be the point layer uniqueplaces and the layer to join will be the newly added “cord-19 articles.” we’ll choose fields to match and set both fields to place. use a descriptive title to name the layer. here, we used “places to articles”. the join operation will be one-to-many since one mapped place can join many separated articles. then click run analysis to create the join. figure 11. join features allows you to link data by place name and visualize this relationship in the map. with this layer created, all of the data of interest is stored in a single place, and we can remove the previous uniqueplaces and “cord-19 articles” layers. clicking on any point in the map now shows a pop up that lists the number of mentions for that place and provides individual information on the file in which it is mentioned as well as the starting and ending token number. this is also where additional metadata on an article would be displayed if you choose to join it. iceland, for example, has five appearances in the texts and if this were an area of interest, we could now quickly find out which articles to look in. figure 12. clicking on a place in the map now shows how many times the place appears in our corpus and the file it appears in. finishing up the map by navigating to the content page, users can view and edit the details of the webmap. the item page for the web map allows the uploader to augment the metadata associated with the file, and provide an in-depth description, tags, credits, and attribution data. including such metadata will improve the discoverability and citation metrics of the web map, should you choose to make it public. on the web map’s item page, choosing to “open in map viewer” will enable editing the map’s symbology, attribute table, and information pop-ups. each of these can be adjusted by mousing over the layer in the web map’s table of contents and selecting the corresponding item. you should now have a functional, customized web map showing the geospatial locations of features from your original .csv file that can display information from its attribute table as a popup. you can edit and save this map to your content on arcgis.com and choose its level of sharing and privacy. you can continue to add data and modify the web map as you see fit and use it to generate web apps from the item description page in your content section of arcgis.com. bibliography nouvel, d., m. erhmann, and s. rosset. 2016. named entities for computational linguistics. wiley: hoboken, nj. about the authors charlie harper is the digital scholarship specialist at case western reserve university, where he collaborates on and manages a variety of digital projects. much of his work centers on the interdisciplinary applications of machine learning, text analysis, and gis. r. benjamin gorham is the research data and gis specialist at kelvin smith library, case western reserve university. with a degree in classical archaeology, ben specializes in the digital humanities with a focus on gis, photogrammetry, and network analysis and serves as geospatial supervisor for a number of archaeological projects in the mediterranean. at case western, he works with researchers across the university to integrate gis workflows and technologies towards the advancement of research goals. subscribe to comments: for this article | for all articles one response to "from text to map: combing named entity recognition and geographic information systems" please leave a response below: dirk, 2023-09-20 cool stuff! some named entities have multiple locations. e.g. london, england and london, canada. let’s say that you also had bristol as a ner. how would you ensure that london, england is picked for gis rather than london, canada? do you have an algorithm for that? leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – strategies for preserving digital scholarship / humanities projects mission editorial committee process and structure code4lib issue 53, 2022-05-09 strategies for preserving digital scholarship / humanities projects the digital scholarship unit (dsu) at the university of toronto scarborough library frequently partners with faculty for the creation of digital scholarship (ds) projects. however, managing completed projects can be challenging when it is no longer under active development by the original project team, and resources allocated to its ongoing maintenance are scarce. maintaining inactive projects on the live web bloats staff workloads or is not possible due to limited staff capacity. as technical obsolescence meets a lack of staff capacity, the gradual disappearance of digital scholarship projects forms a gap in the scholarly record. this article discusses the library dsu’s experimentations with using web archiving technologies to capture and describe digital scholarship projects, with the goal of accessioning the resulting web archives into the library’s digital collections. in addition to comparing some common technologies used for crawling and replay of archives, this article describes aspects of the technical infrastructure the dsu is building with the goal of making web archives discoverable and playable through the library’s digital collections interface. by kirsta stapelfeldt, sukhvir khera, natkeeran ledchumykanthan, lara gomez, erin liu, and sonia dhaliwal introduction the digital scholarship unit (dsu) at the university of toronto scarborough library provides utsc-specific support for digital scholarship in the areas of data, the digital humanities, digital collections, digital preservation, and scholarly communications. frequently, the dsu partners with faculty and researchers for the creation of public digital scholarship and humanities projects (here referred to as ds projects). the dsu is not unique in recognizing the substantial barriers to the preservation of digital scholarship projects over time, and the ways in which orphaning ds projects is at odds with the library’s overall commitment to preservation and dissemination of the scholarly record (ridley et al., 2015). several articles have identified the ways that institutional funding (insufficient to the costs of maintaining ds projects over time) and research funding models (limited in time, and focused on immediate outputs) inhibit adequate maintenance and long-term access to projects (barats et al., 2020; drucker, 2021; edmond and morselli, 2020; smithies et al., 2019). funding agencies may require a research data management plan, but many dimensions of ds work can be lost if these plans do not consider sustainability in the context of the modern web, and the multiple details that must be considered in application development to sustain the project or make it accessible long-term (smithies et al., 2019). ds projects often focus on access and engagement (often via interactive web application features), and are hosted and developed using personal accounts on third-party services that are easily abandoned. rationale for why sustainability is overlooked in ds is manifold and well described: beyond the endemic issue of resourcing, there are challenges with resourcing ds teams with the technical skills required within the confines of a given project (narlock et al., 2021), as well as with the different perspectives of researchers in the humanities/interpretive social sciences and software developers (barats et al., 2020; edmond and morselli, 2020; poole et al., 2020; smithies et al., 2019). broad solutions to these problems have been proposed, including collaborative partnerships between librarians and researchers (king, 2018; miller, 2017) and advocating for policy framework changes to enable sustaining digital initiatives (ridley et al., 2015). on a more technical level, there are various approaches to preserving the legacy of ds projects. freezing a project is one way to maintain the closest original user experience, however, the drawbacks of this are threefold; lack of maintenance and security, incompatibility over time, and external dependencies (bingert and buddenbohm, 2016; oltmanns et al., 2019). the solution that the dsu is interested in exploring is the use of web archiving and accession of the resulting project files into the institutional digital collections. institutional context the university of toronto scarborough library is one of the university of toronto libraries, and benefits from services managed and resourced for the entirety of the system of 40+ libraries. these resources include an institutional subscription to archive-it, the internet archive’s service, for “collecting and accessing cultural heritage on the web.” in december 2020, the dsu drafted a proposal to gain special access to the university’s archive-it subscription to experiment with the archiving of select digital scholarship projects. the dsu worked on this project in partnership with emerging and new professionals in library and computer science to provide meaningful learning experiences as well as opportunities to collaborate in interdisciplinary research teams. once approved, the dsu began piloting the archive-it service as well as supplementary tools for crawling/recording, playback, and description of web archives for digital scholarship projects. technical characteristics of digital scholarship projects in the dsu’s experience, the context in which web-based ds projects emerge results in a tendency toward certain technical characteristics. ds web applications are typically not developed by a full web development team in permanent roles, but rather with a high level of turnover in project staff. moreover, the iterative nature of the work usually leads to a lack of clear specifications at the onset of development while the limited time and resources in these projects usually leaves little time for code refactoring. large portions of a project may be maintained by post-docs or other temporary research staff that leave and take project knowledge with them, making it challenging to work with the specific technologies they leave behind. web-based ds projects often focus on user interactivity and engagement (usually javascript powered), and often use third-party services to achieve mapping/geospatial visualizations and streaming videos, which poses particular challenges for recording and replay. annotation is another common characteristic in ds projects, with scholars providing additional information to digital collection images, which are sometimes served via an external image server, and in our case, utilizing zoomable images powered by openseadragon (barnes et al., 2017). web archiving (referring here to single-time crawls of at-risk ds projects) is a broad and diverse field of policy and praxis that the dsu seeks to apply to this set of challenges in the areas of crawling, replay, indexing and description for web archives. when a website is ‘crawled’ the target is usually a warc (web archive) file, which is a format supporting the concatenation of the data that comprises a website into a single file, and is an established international standard. recently, the draft wacz (web archived collection zipped) format specification offers additional tools to extend the richness and functions of web-archives and other configurations of web-based recorded data. there are other formats for web archiving, but the dsu work has focused on warc as the established international standard. the specific technologies utilized for the creation of these files can be limited in the content and interactivity they can capture. for example, unindexed or so-called “deep” content is not easily captured (sharma and sharma, 2018), and neither is content that relies on user interactivity (via things like mouse clicks) to expose (brunelle et al., 2016). robots.txt and other technologies declare the preferences of site builders for crawling, indexing, and the timing of requests. in combination with the structures of a site, this can affect the ability to crawl effectively (archive-it team [updated 2021]). content with html that references external services or that requires external api calls will continue to rely on the existence of these services. this can make it difficult to verify the existence of all the files required to manage successful replay of a website. in replay, if these services change so that the html no longer can call the service, this element of the web application will no longer be available in the replayed warc. players also require that specific technologies remain available in the browser, if an object is to be ‘played’ back in a browser. the recent end of life for the adobe flash player, for example, and the slow removal of flash support from browser technology has provided new challenges for replaying content on the web that utilized this technology. in short, it can be difficult to determine what has actually been ‘preserved’ and what remains reliant on external services that may disappear in future, and what has been captured may not remain playable in the long term if browser support for the crawled files disappears. these factors combine to reflect the “archivability” of the site, which stanford libraries defines as “the ease with which the content, structure, functionality, and front-end presentation(s) of a website can be preserved and later re-presented, using contemporary web archiving tools” (archivability, n.d.). finally, there are metadata challenges for the resulting web archives. while most crawlers generate substantive automatic technical and administrative metadata, descriptive metadata must be produced by a human being with sufficient context of the original project to allow for the discovery and citation of the archive. the lack of consistent application and development of descriptive metadata standards is well recognized in the web archiving community (dooley and bowers, 2018) and there have been attempts to address this via standards development (dooley and bowers, 2018). crawling and recording digital scholarship projects the international internet preservation consortium provides an entrance into the numerous tools used for acquisition (crawling/recording) of websites. a separate initiative (data together) maintains a substantial list of web archiving software that compares major features, although this appears to have been last updated in 2018. in the early attempts to archive ds projects, the dsu utilized conifer and webrecorder desktop (which also has a browser extension), and archiveweb.page,  in addition to the technologies provided by archive-it: heritrix, umbra, and brozzler. an overview and comparison of the major characteristics is provided in table 1.   table 1. table of tested crawling technologies. crawling technology notes conifer online service based on webrecorder, relies on a user manually playing all interactions for recording. webrecorder desktop relies on a user manually playing all interactions, which reveals content to the crawler for capture. archiveweb.page the archiveweb.page chrome extension is part of the webrecorder project, which develops multiple open-source tools for web recording and replay. archive-it’s “standard” (heritrix and umbra)  archive-it’s standard crawling technology is a combination of heritrix (used for 20 years for web archiving), extended by umbra, a technology that allows for dynamically rendered sites (increasingly the internet standard) to be viewed and crawled by heritrix. archive-it’s brozzler crawler archive-it’s most modern crawling technology, brozzler does not follow hyperlinks, but rather records interactions between a browser and website and allows for the download of things like linked video content, zooming images and game elements of a website. overall, archive-it’s crawlers provided more automated in-depth crawls (in which more web pages were captured with less work required from an end user) than conifer, webrecorder desktop, and archiveweb.page, and archive-it has a substantial user community and tools for quality assurance. however, all of the non archive-it tools were useful in generating complete web archiving files. archiveweb.page was the most successful (more successful than archive-it technologies) in capturing and storing embedded media, resulting in a truer site archive. replaying digital scholarship projects the dsu has also explored replaying web archives for ds projects. the archive-it service’s wayback machine is provided by default with an archive-it service subscription, and plays back warc files that are produced through the service or that are uploaded into the system. however, the addition of code for the wayback player can occasionally block the functionality of a page on replay. to circumvent this, a user can place an id flag to load the archive without the additional wayback code. to supplement access to web archives provided by the archive-it subscription, the dsu also seeks to keep recorded files in the local digital collections and make them available as part of the digital collections (an earlier version of islandora software supported storage of warcs via the web archive solution pack, but had no native playback functionality). to support replay in our new system (also based on islandora) we are targeting an integration with the webrecorder project, which provides several open source tools and packages for recording and replay, including the replayweb.page viewer, a serverless (client-side) web archive replay tool viewer supporting additional formats beyond warcs. when looking for existing integration projects with drupal, lara gomez found an example of viewer integration in archipelago’s “strawberry field” modules. archipelago, developed under the auspices of non-profit service provider “metro” is also an open-source digital asset management project that, like islandora, uses the content management system drupal. based on the work in archipelago, lara was able to create a drupal module providing this integration more generally for drupal users. the resulting replayweb.page player module provides a field formatter that can take a drupal media field storing a compliant file format (i.e wacz, warc) and play it using replayweb.page within the drupal/islandora site (see figure 1 for how this looks in an interface and figure 2 for an edit page that shows the options available for site editors). replayweb.page relies on a service worker, and requires an ssl (https) certificate to work. the main advantage of the replayweb.page is that it does not require a server side application such as webrecorder pywb to replay the web archive. figure 1. storymap wacz file replay in dupal/islandora integrated with replayweb.page. wacz files can be created from warc files using py-wacz. further, archiveweb.page supports the downloading of archives it creates as wacz. in the dsu’s experience, the wacz file format provides a much faster replay experience.  for example, a 100 mb warc file can take 2 – 3 minutes to load, while the same size wacz file takes only a few seconds to begin to play. the wacz format allows web archive collections to be loaded incrementally as the user replays in the browser, substantially improving the experience for end users. figure 2. end users can specify which url in the web capture should be loaded by default in the viewer based on an uploaded file (here, netpreserve-twitter_0.warc). metadata and discovery the dsu provides access to its completed captures immediately through archive-it’s public portal. collection-specific portals provide a way of pointing users to a specific set of web archives that have been grouped together through archive-it’s partner interface (see figure 3). the wayback machine’s calendar-based interface supports multiple crawls of a single domain, with long-term continuous web preservation in mind. figure 3. captured warc files accessible through the archive-it public portal. metadata can be added at the “seed” level and can be bulk uploaded. metadata fields are based on dublin core, with the ability to add custom metadata fields (see figure 4). files can also be downloaded from archive-it for offsite storage or, in the case of the dsu, supplementary preservation workflows and alternative display and discovery. the dsu has been exploring downloading these files and loading them into test digital collections. the dsu uses standard and extended metadata profiles based on mods version 3.4 in compliance with the dlf/aquifer implementation guidelines for shareable mods records. by downloading and accessioning the warcs into our local system, dsu staff take advantage of the familiar collections management approaches, metadata and discovery, with which other collections are authored and maintained, with advantages such as extended metadata support and customizable themes. figure 4. archive-it’s partner interface. users can add metadata to captured sites. sukhvir khera provided a mapping for both our local metadata profile and archive-it’s metadata profile to the “descriptive metadata for web archiving” recommendations developed by the oclc research library partnership’s web archiving metadata working group (wam). the goal would be to cover the recommendations in both systems, with the ability to add additional information and link resources in our local collections, preserving digital projects developed in collaboration with utsc students and faculty. in islandora 7.x, stored warcs could be fully indexed using the web archive solution pack, which provides solr 4+ integration, and this could be a useful approach for providing better content indexing for these file types in our local digital collections. digital scholarship project preservation: use cases the following section outlines specific cases of ds projects and the extent to which playable warcs could be generated, based on the specific technical characteristics of the application and the applied toolset. preserving gaming functionality a virtual version of the marlin perkins board game was developed in 2014-2015 by students in the dsu as an entryway into a digitized collection of zoo ephemera preserved at the library. the (largely javascript) application provided dynamic, user-driven gameplay. the browser-based game also contained flash content. during gameplay, end users would interact in order to produce 25 static html web pages. this type of web application, presenting mixed content requiring human interaction is a recognized challenge for web archiving (lohndorf [updated 2022]; frequently asked questions, n.d.; archivability, n.d.). crawlers compared for the capture of the “zoo parade” game were conifer, webrecorder desktop, and archive-it’s brozzler crawler. conifer was unable to capture the flash elements of the gameplay page, and captured only the preliminary static html pages (see figure 5). in comparison, the capture made with webrecorder desktop captured a screenshot of the gameplay page, as well as the preliminary html pages. archive-it’s brozzler crawler, using the ‘ignore robots.txt’ rule and seeds for all 25 html pages produced a reasonable facsimile of the original game (see figure 5 and 6). figure 5. comparison of captures: brozzler (left) and conifer (right). figure 6. comparison of live web page (left) and brozzler capture (right). end users are not able to play the game through the capture, but they are able to view the static (html) resources to get an understanding of gameplay and can still play the game on the live site. preserving web mapping projects the mapping “scarborough chinatown” project mapped popular 20th century malls as a way for users to browse images of historical scarborough restaurants and supplementary research materials. the project used drupal’s leaflet plugin. mapping software like leaflet and arcgis contain large amounts of custom javascript, making it difficult not only to capture maps with various crawling technology, but also to replay in wayback (rollason-cass [updated 2021]; using the wayback machine [updated 2021]). conifer, standard (heritrix and umbra) and brozzler were compared for the capture and wayback replay of the leaflet map. all these crawling technologies had difficulty capturing the content inside the pop-up bubbles in the embedded map, requiring moderate modifications to the page so that urls could be more easily captured. archive-it’s “standard” crawlers captured some static elements of the site consisting of text files and a few image files, while brozzler captured the page more comprehensively, including javascript files. by using developer tools to review captures in wayback, sukhvir determined that while archive-it’s crawlers were able to capture different elements of the site they were unable to capture and replay the interactive leaflet map (see figure 7). figure 7. thumbnail image captured by archive-it’s “standard” crawler. in contrast, conifer was successful in capturing many elements that make up the leaflet map (see figure 8). figure 8. leaflet map captured by conifer and uploaded to archive-it for playback in wayback. although it is difficult to measure how complete the conifer capture of the leaflet map is, sukhvir was able to determine that many of its elements (i.e. small images that make up the ‘internal zoom on click’ functionality of the map, icon markers, symbols and thumbnail images) were captured. it can be difficult to determine what file(s) are powering replay in archive-it, which uses an index called the wayback cdx/ci to provide the best matched capture of a given url and timestamp across multiple warcs. when a conifer crawl was uploaded into archive-it, it was indexed and a more complete capture was playable via the archive-it web interface (see figure 9). figure 9. wayback replay of the interactive leaflet map. preserving externally hosted media an esri storymap was created in 2021 about artist doris mccarthy (1910-2010) and embedded into the web-based repository containing her digitized papers and artwork. in addition to challenges with the underlying map (as discussed above), the storymap also used embedded images drawn directly from the underlying repository (islandora) as well as embedded content from a third-party video streaming service (in this case, vimeo). there were also a number of structural issues in the storymap that could have resulted in infinite crawls of non-relevant content (crawler traps), and time-limits and redirects in the site architecture that complicated crawling. archive-it’s standard (heretrix) crawler, their brozzler crawler, and archiveweb.page were all tested for capturing the site. to assess the completeness of the crawl, the individual captures were compared in the wayback player and the replayweb.page player. in the wayback player, wayback url flags (see figure 10) revealed that the video was not captured by archive-it. figure 10. comparison of video replay in wayback: identity flag (left) and iframe flag (right). the lack of archived video content in the capture was further verified by the crawl’s “file type report,” a native tool in archive-it provided to assess the completeness of crawls (figure 11). figure 11. storymap crawl’s file type report included no media. the file type report provides a breakdown listing all of the content (documents or urls) captured by file types, which can be reviewed in-depth by clicking on each hyper-linked file type (see figure 12). figure 12. storymap crawl’s file type report allows each image file to be viewed individually. there are no video file types listed in the report, confirming that the vimeo video has not been captured as part of the recording, and so, if the file is removed from vimeo, the content will be lost. this was also obvious when the warc file from archive-it was loaded into the replayweb.page viewer. the archiveweb.page crawl was assessed in the replayweb.page viewer using firefox’s web developer tools. the network tab of developer tools in most modern browsers will reveal the sources of image and video files on the web page (see figure 13). figure 13. developer tools: network tab review of wayback replay of the storymap crawl. using this tool for firefox, it is possible to see that the mp4 file hosted on vimeo was captured in the archiveweb.page recording, making this a much stronger capture that does not rely on the continued hosting of the file by vimeo. the resulting file played well in the drupal integration with replayweb.page (see figure 14). figure 14. storymap wacz file includes embedded vimeo video, playable in firefox browser. therefore, while brozzler’s capture was an improvement over archive-it’s standard crawler, the most complete archive was achieved by archiveweb.page. one remaining limitation is that the embedded vimeo video is only playable in the firefox browser, which permits a broader range of mime types than chrome and safari at the time of writing. a note on quality assurance when crawling dynamic javascript content the challenges of preserving javascript have been recognized for the past decade (brunelle et al., 2016; leetaru, 2017; boss and broussard, 2017; using the wayback machine [updated 2021]; lohndorf [updated 2022]). wayback replay software adds another layer of complexity to this challenge as it provides playback of captures that interact with the live web, providing the sense that material has been archived when it is still reliant on live web resources. by relying on wayback, it can be difficult for a user to decipher what content has been captured, how complete the capture is, and what embedded media files might be loading from the live web (see figure 15). figure 15. storymap dynamic image playback comparisons: identity flag (left) and iframe flag (right). both show an image that has been loaded from the open web. this same image fails to load in replayweb.page. archive-it’s crawl reports are a more reliable source to review captures for their completeness. particularly, the “file type” reports and developer tools have been key to verifying what content has been captured. for example, to determine whether dynamic images with internal zoom on click functionality were captured for the storymap crawl, a review of the file type report revealed that only the smaller images were captured. webrecorder’s replayweb.page replay technology has been useful for reviewing quality assurance as well. when downloading warc files for testing in the replayweb.page player, the difference between captured data and data that rely on the open web becomes more obvious (see figure 16). figure 16. storymap dynamic image playback in replayweb.page: successfully playing data that has been captured (text file on the right) and failing to load an image from the open web (image file on the left). preserving images served via an image server the harlot’s progress in context (hpic) was created in 2015 to support student-assignments about the work of william hogarth (1697-1764). images were displayed in the open seadragon viewer for islandora, using the islandora web annotations solution pack (barnes et al., 2017). the zooming functionality of the openseadragon viewer and the javascript-powered pop-up annotations were both captured and playable using archive-it’s standard (heritrix) crawler (see figures 17 and 18 for depictions of the captured, annotated images). figure 17. hpic crawl’s file types report shows that images at multiple sizes have been captured (extra-large images captured are highlighted in yellow. medium-sized images captured are underlined in red.) archive-it’s standard (heritrix) crawler alone captured this site in a one-time test crawl. the successful capture and replay of this crawl reflect a high level of archivability in this site, making this design one that should be replicated in future projects. in the case of the hpic project, the images (both extra-large and medium sized versions), and annotations were all captured with relative ease (see figure 17). furthermore, presenting the annotations as static text in the page, as well as through on-click pop-ups, they were more easily captured for replay (see figure 18). in addition, the type of zooming functionality provided by open seadragon (click-to-enlarge), facilitated the capture of the larger version of images by directing the heritrix crawler to these specific images through the provision of standard links in html format (guidelines for preservable websites, n.d.). figure 18. annotated image beside text panel of annotations playable in wayback. conclusions and future directions archive-it provides a comprehensive suite for the capture and replay of webpages, including ds projects. the system is designed for archivists to capture repeating crawls as web pages change, rather than single crawls for capture of an at-risk ds project, and so one-time crawls may represent something of an ‘off-brand’ use of the service. crawls taken through archive-it can also be downloaded for use in local digital collection systems. external recorders such as conifer, webrecorder, and archiveweb.page also perform well, and have been successful at mitigating gaps in archive-it’s crawling capabilities (particularly archiveweb.page, which exceeded the performance of all other crawlers when capturing externally hosted media). despite the occasionally daunting complexity of web archiving, existing technologies do facilitate recording and preserving elements of ds projects in risk of being lost to technological obsolescence. however, the pilot revealed ongoing challenges with the long-term preservation of embedded and dynamic content that remain unresolved at time of writing. despite ongoing challenges, the dsu concludes that preserving a playable warc provides benefits over maintaining a full application stack with better durability. we also concluded that archiving ds projects is potentially very resource intensive. producing a capture requires manual playing of site functionality and checking for quality, and while richer descriptive metadata is always preferred, its production takes time and attention. as with all digital objects, warcs and wacz will also require ongoing maintenance to ensure preservation and access. the comparative costs in time and accessibility for warcs vis a vis other digital objects for the dsu remains to be seen. this web crawling pilot has inspired the dsu to consider archivability earlier in ds projects, following a trend identifiable in literature about web archiving (leetaru, 2017; brunelle et al., 2016; archivability, n.d.; creating preservable websites, n.d.). in future, the dsu will explore how archivability intersects with compliance to the web content accessibility guidelines (wcag) and responsive web development (for low bandwidth and mobile first populations) in service of universal design. bibliography archivability [internet]. n.d. san francisco (ca): stanford libraries; [cited 2022 feb 17]. available from: https://library.stanford.edu/projects/web-archiving/archivability archive-it team. [updated 2021 sep 8]. a new wayback: improving web archive replay  [internet]. san francisco (ca): archive-it blog. [cited 2022 feb 23]. available from: https://archive-it.org/blog/post/archive-it-wayback-release/ archive-it team. [updated 2022 feb 17]. archive-it system status [internet]. san francisco (ca): archive-it support center; [cited 2022 feb 23]. available from: https://support.archive-it.org/hc/en-us/articles/360019259671-archive-it-system-status barats c, shafer v, fickers a. 2020. fading away … the challenge of sustainability in digital studies. digital humanities quarterly [internet]. [cited 2022 feb 23]; 14(3):1-17. available from: http://www.digitalhumanities.org/dhq/vol/14/3/000484/000484.html barnes me, ledchumykanthan n, pham k, stapelfeldt k. 2017. annotation-based enrichment of digital objects using open-source frameworks. code4lib journal [internet]. [cited 2022 mar 4]; (37). available from: https://journal.code4lib.org/articles/12582 bingert s, buddenbohm s. 2016. research data centre services for complex software environments in the humanities. information services & use [internet]. [cited 2022 feb 23]; 36(3-4):189-202. proceedings of the 20th international conference on electronic publishing positioning and power in academic publishing: players, agents and agendas; 2016 jun 7-9; göttingen. available from: https://doi.org/10.3233/isu-160817 blumenthal k-r. [updated 2020]. archive-it 7.0 release notes [internet]. san francisco (ca): archive-it help center; [cited 2022 feb 23]. available from: https://support.archive-it.org/hc/en-us/articles/360039653811-archive-it-7-0-release-notes#wayback blumenthal k-r. [updated 2021 may]. archive-it apis and integrations [internet]. san francisco (ca): archive-it support center; [cited 2022 feb 23]. available from: https://support.archive-it.org/hc/en-us/articles/360001231286-archive-it-apis-and-integrations blumenthal k-r. [updated 2022 jan]. quality assurance overview [internet]. san francisco (ca): archive-it support center; [cited 2022 feb 23]. available from: https://support.archive-it.org/hc/en-us/articles/208333833-quality-assurance-overview boss k, broussard m. 2017. challenges of archiving and preserving born-digital news applications. ifla journal [internet]. [cited 2022 feb 23]; 43(2):150-157. available from: https://doi.org/10.1177/0340035216686355 brunelle jf, kelly m, weigle mc, nelson ml. 2016. the impact of javascript on archivability. international journal on digital libraries [internet]. [cited 2022 feb 23]; 17:95-117. available from: https://doi.org/10.1007/s00799-015-0140-8 cobourn ab. 2017. case study: washington and lee’s first year using archive-it. journal of western archives [internet]. [cited 2022 feb 23]; 8(2):1-13. available from: https://doi.org/10.26077/f2bf-4185 creating preservable websites [internet]. n.d. washington (dc): library of congress. [cited 2022 feb 17]. available from: https://www.loc.gov/programs/web-archiving/for-site-owners/creating-preservable-websites/ data together research: web archiving [internet]. [updated 2018 sep 17]. data together; [cited 2022 feb 23]. available from: https://github.com/datatogether/research dooley j, bowers k. descriptive metadata for web archiving [internet]. dublin (oh): oclc research; 2018 [cited 2022 feb 23]. available from: https://www.oclc.org/research/publications/2018/oclcresearch-descriptive-metadata.html drucker j. 2021. sustainability and complexity: knowledge and authority in the digital humanities. digital scholarship in the humanities [internet]. [cited 2022 feb 23]; 36(supplement_2):ii86-ii94. available from: https://academic.oup.com/dsh/article/36/supplement_2/ii86/6205948?login=true edmond j, morselli f. 2020. sustainability of digital humanities projects as a publication and documentation challenge. journal of documentation [internet]. [cited 2022 feb 23]; 76(5):1019-1031. available from: https://doi.org/10.1108/jd-12-2019-0232 frequently asked questions [internet]. n.d. san francisco (ca): stanford libraries; [cited 2022 feb 17]. available from: https://library.stanford.edu/projects/web-archiving/frequently-asked-questions guidelines for preservable websites [internet]. n.d. new york (ny): columbia university libraries; [cited 2022 feb 17]. available from: https://library.columbia.edu/collections/web-archives/guidelines.html king m. 2018. digital scholarship librarian: what skills and competences are needed to be a collaborative librarian. international information & library review [internet]. [cited 2022 feb 23]; 50(1):40-46. available from https://doi.org/10.1080/10572317.2017.1422898 leetaru k. 2017. are web archives failing the modern web: video, social media, dynamic pages and the mobile web [internet]. new york (ny): forbes magazine; [cited 2022 feb 22]. available from https://www.forbes.com/sites/kalevleetaru/2017/02/24/are-web-archives-failing-the-modern-web-video-social-media-dynamic-pages-and-the-mobile-web/?sh=3b2d652b45b1 lewis v, spiro l, wang x, cawthorne je. building expertise to support digital scholarship: a global perspective [internet]. washington (dc): council on library and information resources; 2015 [cited 2022 feb 23]. available from: https://www.clir.org/pubs/reports/pub168/ lohndorf j. [updated 2021]. archive-it crawling technology [internet]. san francisco (ca): archive-it support center; [cited 2022 feb 23]. available from: https://support.archive-it.org/hc/en-us/articles/115001081186-archive-it-crawling-technology lohndorf j. [updated 2022 feb 17]. known web archiving challenges [internet]. san francisco (ca): archive-it support center; [cited 2022 feb 23]. available from: https://support.archive-it.org/hc/en-us/articles/209637043-known-web-archiving-challenges maemura e. 2018. what’s cached is prologue: reviewing recent web archives research towards supporting scholarly use. in: freund l., editors. proceedings of the association for information science and technology. hoboken (nj): wiley. p. 327-336. available from: https://doi.org/10.1002/pra2.2018.14505501036 miller a. 2017. a case study in institutional repository content curation: a collaborative partner approach to preserving and sustaining digital scholarship. digital library perspectives [internet]. [cited 2022 feb 23]; 33(1): 63-76. available from https://doi.org/10.1108/dlp-07-2016-0026 narlock m, johnson d, vecchio j. 2021. digital preservation services at digital scholarship centers. journal of academic librarianship [internet]. [cited 2022 feb 23]; 47(3):102334. available from https://doi.org/10.1016/j.acalib.2021.102334 oltmanns e, hasler t, peters-kottig w, kuper h-g. 2019. different preservation levels: the case of scholarly digital editions. data science journal [internet]. [cited 2022 feb 23]; 18(1):51. available from http://doi.org/10.5334/dsj-2019-051 poole ah, garwood da. 2020. digging into data management in public?funded, international research in digital humanities. journal of the association for information science and technology [internet]. [cited 2022 feb 23]; 71(1):84-97. available from: https://doi.org/10.1002/asi.24213 ridley m, appavoo c, pagotto s. 2015. seeing the forest and the trees: the integrated digital scholarship ecosystem (idse) project of the canadian research knowledge network (crkn). proceedings of the acrl 2015 conference; 2015 mar 25-28; portland: american library association. p. 676-682. available from: https://www.ala.org/acrl/sites/ala.org.acrl/files/content/conferences/confsandpreconfs/2015/ridley_appavoo_pagotto.pdf rollason-cass s. [updated 2021]. archiving arcgis [internet]. san francisco (ca): archive-it support center; [cited 2022 feb 23]. available from: https://support.archive-it.org/hc/en-us/articles/360051604431-archiving-arcgis sharma dk, sharma ak. 2018. deep web information retrieval process: a technical survey. in: the dark web: breakthroughs in research and practice [internet]. [cited 2022 feb 23]. available from: https://doi-org.myaccess.library.utoronto.ca/10.4018/978-1-5225-3163-0 smithies j, westling c, sichani a-m, mellen p, ciula a. 2019. managing 100 digital humanities projects: digital scholarship and archiving in king’s digital lab. digital humanities quarterly [internet]. [cited 2022 feb 23]; 13(1):1-16. available from: http://www.digitalhumanities.org/dhq/vol/13/1/000411/000411.html usability & web accessibility [internet]. n.d. new haven (ct): yale university; [cited 2022 mar 3]. available from: https://usability.yale.edu/web-accessibility/articles/images using the wayback machine [internet]. [updated 2021]. san francisco (ca): internet archive; [cited 2022 feb 23]. available from: https://help.archive.org/hc/en-us/articles/360004651732-using-the-wayback-machine about the authors kirsta stapelfeldt is a librarian and the head of the digital scholarship unit at the university of toronto scarborough library. sukhvir khera is an emerging professional in digital scholarship in the digital scholarship unit at the university of toronto scarborough library. natkeeran ledchumykanthan is an application programmer of the digital scholarship unit at the university of toronto scarborough library. lara gomez is a current computer science student at the university of toronto scarborough in the software engineering stream, and has completed two terms with the library as an emerging professional in application development. erin liu is an assistant archivist at the university of arts london archives and special collections centre. she was an emerging professional in digital scholarship at the university of toronto scarborough library digital scholarship unit between 2020 and 2021. sonia dhaliwal is the data & digital collections librarian in the digital scholarship unit at the university of toronto scarborough library. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – customizing android tablets for a shared environment mission editorial committee process and structure code4lib issue 24, 2014-04-16 customizing android tablets for a shared environment the valley library at oregon state university libraries & press supports access to technology by lending laptops and e-readers. as a newcomer to tablet lending, the valley library chose to implement its service using google nexus tablets and an open source custom firmware solution, cyanogenmod, a free, community-built android distribution. they created a custom build of cyanogenmod featuring wireless updates, website shortcuts, and the ability to quickly and easily wipe devices between patron uses. this article shares code that simplifies android tablet maintenance and addresses android application licensing issues for shared devices. by jane nichols, uta hussong-christian and ryan ordway introduction many academic libraries offer students and faculty access to popular and current technology by lending laptops, e-readers and tablets. the valley library at oregon state university libraries & press supports this type of access by lending laptops and e-readers. however, the valley library is a newcomer to tablet lending. while the literature on tablet lending in higher education focuses almost exclusively on ipad lending programs (anderson and weatherbee 2012; buzzard and teetor 2011; capdarest-arest 2013; massis 2013; obst 2010; thompson 2011), we chose google’s nexus 7 tablets primarily due to the significant cost savings (at least $100 less than any apple ipad); we later added nexus 10 tablets to the service. the dearth of literature on android device lending programs in academic libraries meant developing some processes from the ground up. this article fills a gap in the tablet lending literature by addressing the lending of nexus 7/nexus 10 android tablets, by sharing code to simplify their maintenance, and by discussing how we resolved licensing hurdles encountered with android applications (apps) on shared devices. coding for lending a significant benefit to choosing the nexus tablet is its open source android operating system because we can customize various aspects of the operating system to meet lending needs. we chose cyanogenmod (http://www.cyanogenmod.org/) (cm) as the replacement android-based operating system for the nexus tablets. cm is a community-supported modification (mod) of the freely available android open source project code (https://source.android.com/). the most important consideration for us was cm’s solution for two issues related to resetting the tablets between patrons. using cm lets us reset the tablets without losing any build customizations; it also allows resetting without having to connect tablets to a computer hosting the stock android backup and restore software (a requirement of the factory android operating system). in addition, with cm we can deploy a wireless update system simplifying individual device updates so circulation, rather than it staff, can easily install them. each of these solutions is addressed in more detail below. tablet reset we decided early on to give patrons full privileges to download and install apps and other personal information such as photos onto the nexus tablets. this would give patrons the most authentic user experience, but it also meant that between circulations the tablets would need to be reset in order to preserve patron privacy and clear all patron data such as credit card information for app or in-app purchases and login details stored in the main device memory. the reset would also need to clear any changes that patrons had made such as downloaded apps, reorganized home screen icons, and photos taken and stored on the tablet’s sd memory card. the default factory reset process for the nexus tablets required tethering each tablet to a computer on which the backup and restore software was downloaded. this placed limits on how many tablets circulation staff could restore at the same time due to limited usb port availability. testing revealed that the factory backup and restore process did not restore system preferences like current language or wi-fi settings, requiring manual reconfiguration from circulation or it staff; it also did not remove any other user data like google or social media account information. furthermore, the factory restore did not clear personal data (e.g. photos) from the tablet’s sd memory card; clearing this memory required separate steps from the factory reset. finally, the backup and restore process restored the tablets to factory settings, which was not ideal. the build we envisioned would include custom website shortcut icons such as osu libraries, library2go (oregon’s statewide public library implementation of overdrive’s e-book service), and an online patron survey along with additional apps not included in the factory build. using cm we could “bake” these customizations into the operating system, creating a unique valley library build which is not lost when tablets are reset. to simplify the full reset we modified the device recovery program to wipe data from both the main and sd storage and to restore the custom build. using android’s device administration api we coded a new custom app called tabletnuke to trigger and complete the full reset process from the tablet itself. we added this custom app to the valley library build just like other third party apps. the full tabletnuke code is available on github (https://github.com/osulp/tabletnuke), and the specific section of the app code that triggers the factory reset is provided below. case r.id.button_reset_device: if (!isactive) { toast.maketext(this, "reset failed, please enable deviceadmin.", toast.length_long).show(); log.d(tag, "tried to reset without toggling admin policy"); return; } // we reset the device this will erase entire /data partition, including virtual sd card! toast.maketext(this, "factory resetting device...", toast.length_long).show(); log.d(tag, "resetting device now system will be restored to factory settings"); devicepolicymanager.wipedata(activation_request); break; circulation staff begin the simple reset process by using a home screen icon named wipe data to access the tabletnuke app. upon enabling the device admin setting, they are able to complete the reset process by pressing the big, red button shown in figure 1. because the wipe data icon is visible on the home screen, it is possible patrons will reset the tablets. while we do not expect that many will launch tabletnuke (via the wipe data icon), we make no attempt to restrict access since patrons would only erase their own personal data. figure 1. tabletnuke’s big red reset button tablet build updates we make ongoing changes to the tablet build as needed or requested. to avoid removing each tablet from circulation to upgrade to the newest build, we developed a way to wirelessly push build updates to each device. this was accomplished by utilizing getcm (http://get.cm) which stores cm update packages and provides a web interface for interacting with them. we customized getcm to run locally on a library server and modified the cmupdater app that is integrated into cm to query our local instance of getcm. now when circulation staff open the tablet settings and look for updates, the most recent version of our custom build is served up by the local instance of getcm. the patch below modifies the url that the cmupdater app uses to find the local getcm instance. project packages/apps/cmupdater/ diff --git a/res/values/config.xml b/res/values/config.xml index fbb3a43..39edde9 100644 --a/res/values/config.xml +++ b/res/values/config.xml @@ -8,7 +8,7 @@ --> <resources xmlns:xliff="urn:oasis:names:tc:xliff:document:1.2"> <!-cmupdate config strings --> <string name="conf_update_server_url_def" translatable="false">http://get.cm/api</string> + <string name="conf_update_server_url_def" translatable="false">http://getcm.library.oregonstate.edu:6543/api</string> <string name="conf_changelog_url" translatable="false">http://localhost/changelog.xml</string> <bool name="alternateisinternal">false</bool> </resources> the valley library custom build along with the stock apps for both productivity and entertainment on the nexus tablet, the valley library customized build includes website shortcut icons on the tablet home screen and additional pre-installed apps. other academic libraries with tablet lending programs have also enhanced their stock build by adding apps and website shortcut icons (capdarest-arest 2013; massis 2013; obst 2010; thompson 2011). the decision to add website shortcut icons for destinations like the library website, a library-created tablet guide, the local public library e-book service, and a short patron survey was straightforward as they give quick access to library services. choosing which apps to pre-install was a more complex decision. the factory build already provided basic functionality for reading e-books, playing videos, and browsing the web. as the tablets were intended to give patrons (e.g. students) a way to experiment with, learn about, and try out tablet technology, we rationalized that part of that learning experience was personalizing the technology by investigating and downloading apps that met specific needs. others share this perspective (capdarest-arest 2013; massis 2013; thompson 2011), and we moved forward with the optimistic perception that patrons would feel comfortable personalizing the devices, just as they do with their personal mobile technology. even though some patrons added apps to meet their specific needs, our optimism may have been a bit misplaced based on consistent requests to pre-install a variety of apps. even though we modified the custom build multiple times to add requested apps, there are reasons to limit the number of pre-loaded apps. reasons center on device and server storage space as well as our evolving understanding of app licensing and app store terms of service. the storage issue was easy to resolve, but the licensing was not, as discussed later. one concern about storage was the amount of server resources needed to create the tablet build. the growing number of library-selected apps in the build required that we move the build system to a new server with more ram and update the build scripts to allow them to use up to 8gb of ram instead of 2gb. the build system cryptographically signs all of the apps included in the build at once, which requires increasing amounts of ram as the number of apps increases. another issue has been the amount of storage space available on the tablets, in particular in the /system partition. because the same build is loaded onto both the nexus 7 and 10, storage space on the smaller tablet is the limiting factor. currently only 12% of the nexus 7 /system partition is free for new app storage, necessitating a very selective approach for adding any new apps. we may soon also need to re-evaluate the suite of currently available pre-loaded apps [1]. navigating licensing tablets, and the apps created for them, were designed for individual consumer use rather than for the shared environment that libraries offer. as we explored which apps to include in the build and whether to get the apps from nexus’ partner app store, google play, or a third party provider like amazon appstore, slideme or cyanogenmod, we discovered that terms of service (tos) agreements differ among app providers regarding app use for sharing services. we learned that when using digital content on google play, the tos prohibit the use of google ‘products’, which include apps, as part of a sharing or lending service (google 2013). it should be noted that the tos does not extend to the android operating system itself. this google play tos restriction prevented us from including their apps, like gmail and google drive, in our custom build. instead, we used third party apps which had no tos lending restriction. however, over the year patron feedback showed that users did not use generic apps like email (email client) and browser (android web browser), which provide functionality equivalent to google’s apps. they wanted google products like gmail and google drive pre-installed. such patron requests meant looking at possible ways to obtain google apps without going against google play’s tos. historically google apps were part of cm. however, this changed after google sent cm a cease and desist letter in 2009, forcing them to stop bundling the apps (kondik 2013). google followed up with a blog post clarifying that while the android operating system source code is open source, the apps it has developed for use on the android platform are not open source, and, therefore, not included in the android source code repository (morrill 2009). in fall 2013 we began extracting the google apps “gapps” packages from the factory build (they are included for device recovery purposes) and folding them into the custom build. because we are not pulling apps from the google play store, distributing the apps for download, or posting our build images for download by others, we feel we have successfully navigated google play licensing restrictions, essentially taking the licensing issue out of the picture, while still making google apps available for our users. lending details we launched the tablet lending service with thirteen google nexus 7 tablets. the nexus 7’s initially circulated for one week. in response to patron interest, fourteen nexus 10s with keyboards were purchased to circulate in fall 2013. the nexus 7 loan length was changed to six hours, and the nexus 10 was set to one week. our thinking was that patrons would use the nexus 7 for short bursts of activity and to fill time in their schedule whereas the nexus 10s with keyboards would be used for more purposeful, productivity-related activities. the tablets have had reasonable use. the nexus 7s averaged 7.8 circulations per week during the first 6 months; this increased to just over 15 circulations per week after the loan length was changed. the nexus 10s circulate 9 times per week on average. see table 1 for circulation totals and loan periods. table 1: loan lengths and circulations nexus 7 nexus 7 nexus 10 time period 2/2013-8/2013 9/1/2013-3/7/2014 9/1/2013-3/7/2014 loan period 1 week circ 6 hour circ 1 week circ number of circs 235 413 249 evaluating tablet lending as we hoped, students used the tablets for academic purposes. for example, student feedback revealed that they did their course readings (59%), research or an assignment for a course (58%), or worked on written assignments (28%). in addition to academics, students checked email for classes, campus activity or work and unsurprisingly, used them for entertainment purposes (37%), such as playing games or watching videos. feedback also showed that when students tried to read articles in pdf they ran into problems. the tablets have a pdf reader, aldiko, but students seemed to not recognize that this can be used to display their course readings rendered in pdf; they may not know how to launch aldiko, or they may simply prefer to use the more familiar adobe acrobat reader. to address this issue we decided to install the acrobat reader app. as we could not download this app from google play due to the licensing restriction noted earlier, we contacted adobe directly for permission to use the acrobat reader app in our tablet build. this permission was granted upon the completion of a distribution agreement and the reader will be added to a forthcoming updated build. discussion by offering nexus 7 and nexus 10 tablets for checkout, we expanded device offerings beyond laptops and e-readers, equipped patrons with technology they might not otherwise have had access to, and contributed to patron skill development. evaluation data show that students use tablets for both entertainment and academic needs and that they would like tablets to be pre-loaded with apps needed to accomplish productivity tasks. because of the device storage space limitations noted earlier, libraries will want to consider how much space to devote to pre-loaded apps and how much to leave for patron downloads. libraries interested in this approach to android tablet lending will want to weigh the amount and type of programming support needed. programming included time to develop the custom build while learning; the programmer spent approximately 10% of his time on this project over three months creating the build, and continues to spend about 5% of his time updating the build. he already knew linux and used that knowledge to customize cm. it should also be noted that using an open source model fits with the valley library’s preference for supporting and implementing open source solutions. another approach to look at is using mobile device management software like cisco’s free meraki system manager. we tried meraki with our tablets but learned they would have to be re-registered after each reset, an impractical step. we are planning a meeting with meraki to further explore how it could address our needs. even with our custom solutions in place, it is good to remember that tablets were created for individual consumers. as goodwin, quoting hasic, says, “when you get frustrated with how the ipads work in a shared environment, remember that you are shoe horning a device that’s designed for a single user into a shared” setting (hasic, 2011, p. 8 as quoted in goodwin 2012). fortunately the cm version of the android os enabled us to “shoe horn” the nexus tablet into a successful lending program. conclusion this initial foray into tablet lending has, on the whole, been positive. it is a well-used service that helped us fill a technology gap and gives the osu community an opportunity to build their skills with tablets. by adding the nexus tablets to our suite of devices, our patrons have access to a wider range of contemporary technology. libraries looking for an affordable way to lend tablets may take advantage of and build off of the provided code to implement a similar service. notes [1] http://guides.library.oregonstate.edu/content.php?pid=478508&sid=3919989 references anderson s, weatherbee s. 2012. growing a technology equipment service in an academic library. comput lib [internet]. [cited 2013 dec 10]; 32(6):6-8. available from: http://www.infotoday.com/cilmag/jul12/anderson-weatherbee–growing-a-technology-equipment-service-in-an-academic-library.shtml back buzzard p, teetor t. 2011. best practices for a university laptop lending program. code4lib [internet]. [cited 2014 mar 14]; (15). available from: http://journal.code4lib.org/articles/5876 capdarest-arest, na. 2013. implementing a tablet circulation program on a shoestring. jama [internet]. [cited 2013 dec 16]; 101(3):220-224. available from: http://www.ncbi.nlm.nih.gov/pmc/articles/pmc3738084/pdf/mlab-101-03-220.pdf goodwin k. 2012. use of tablet technology in the classroom [internet]. [cited 2014 jan 28]. strathfield nsw, australia: state of new south wales, department of education and communities. available from: http://www.tale.edu.au/tale/live/teachers/shared/next_practice/ipad_evaluation_sydney_region.pdf back google play terms of service [internet]. [updated 2013 nov 20]. mountain view (ca): google, inc. [cited 2013 nov 20]. available from: https://play.google.com/intl/en-us_us/about/play-terms.html back kondik, s. 2013 sep 18. a new chapter [blog post]. cyanogenmod: blog [internet]. [cited 2014 feb 8]. available from: http://www.cyanogenmod.org/blog/a_new_chapter back massis be. 2013. from ipads to fishing rods: checking out library materials. new lib world [internet]. [cited 2013 nov 7]; 114(1/2):80-83. available from: http://www.emeraldinsight.com/journals.htm?issn=0307-4803&volume=114&issue=1/2&articleid=17073238&show=html doi:10.1108/03074801311291983 morrill d. 2009 sep 25. a note on google apps for android [blog post]. android developers blog [internet]. [cited 2014 feb 8]. available from: http://android-developers.blogspot.com/2009/09/note-on-google-apps-for-android.html back obst o. 2010. ipad lending project: first results. j eur assoc health inf lib [internet]. [cited 2013 nov 7]; 6(4):39-41. available from: http://jeahil.wordpress.com/2010/10/14/ipad-lending-project-first-results/ thompson sq. 2011. setting up a library ipad program: guidelines for success. coll res lib news [internet]. [cited 2013 jun 14]; 72(4):212-236. available from: http://crln.acrl.org/content/72/4/212.short about the authors jane nichols is collection development librarian at o.s.u. libraries and press. uta hussong-christian is instruction and science librarian at o.s.u. libraries and press. ryan ordway is unix systems administrator at o.s.u. libraries and press. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – microdata in the ir: a low-barrier approach to enhancing discovery of institutional repository materials in google mission editorial committee process and structure code4lib issue 39, 2018-02-05 microdata in the ir: a low-barrier approach to enhancing discovery of institutional repository materials in google georgetown university library curates a multitude of open access resources in its institutional repository and digital collections portal, digitalgeorgetown. over the last several years, the library has experimented with methods for making these items increasingly visible in search engine search results. this article describes the library’s low-barrier approach to applying schema.org vocabulary to its dspace institutional repository using microdata, as well as the challenges with and strategies used for assessing this work. the effects of the application of schema.org microdata to digitalgeorgetown on google search results were tracked over time using three different metrics, providing new insights about its impact. by shayna pekala introduction discoverability is a crucial component of open access, yet it is one that is often overlooked. while web search engines serve as the primary vehicle for the discovery of open access content, much of that content, particularly within institutional repositories, is not fully optimized for display in google search results. search engine crawlers look for schema.org vocabulary represented as structured data to understand and ultimately display web page content; however, this is not presently built into any widely-used institutional repository solutions. schema.org is a shared vocabulary created by search engines to make web content understandable to web crawlers and other machines. it is composed of 589 hierarchical types and 860 associated properties [1], which were created to be intentionally broad to facilitate mass adoption (ronallo 2012). schema.org is applied to a web page by embedding it into the html in one of three formats: microdata, json-ld, or rdfa. in an effort to enhance discovery of institutional repository materials in the open web, some academic libraries have incorporated schema.org into their institutional repositories with varying degrees of complexity and customization. background over the last two years, georgetown university library has engaged in efforts to integrate schema.org vocabularies with microdata in its institutional repository and digital collections portal, digitalgeorgetown [2]. digitalgeorgetown was launched in 2009 and is built on the dspace repository platform. it’s managed by the library’s digital services unit in the library information technology (lit) department, and currently houses over 535,000 items. a significant portion of the records in digitalgeorgetown are citation-only (69%), and of the remaining 165,925 records, 67% are open access, and 33% are restricted access. in 2015, lit began exploring ways to make the items in digitalgeorgetown more discoverable in search engine search results and became interested in integrating schema.org into the repository. the goal was to do this in the simplest way possible, using only the core schema.org vocabulary (as opposed to creating any new types or properties). the microdata format was selected because, at the time, it was the format recommended by google [3]. after testing the application of schema.org in the form of microdata to a single community in dspace, lit decided to extend its use in the repository. in fall 2016, schema.org microdata was applied to all communities and collections in digitalgeorgetown (with the exception of collections that exclusively had citation-only records and those that had a majority of restricted access items), and in spring 2017, a formal assessment was conducted to measure the impact of the implementation on google search results. this paper documents the workflows used for applying schema.org microdata to digitalgeorgetown, the strategies used for measuring impact, and the results. literature review a handful of other academic libraries have also explored the use of schema.org with their institutional repositories. hilliker et al. (2013) at columbia university describe how they added schema.org markup using microdata to their fedoraand blacklight-based digital repository (columbia university academic commons … [updated 2017]) as one of four approaches to making their repository interoperable. developers mapped existing mods metadata fields from the repository to corresponding properties in the schema.org creativework type (they eventually expanded to include other types) and updated the blacklight code to insert the appropriate attributes into the html in the repository. they anecdotally observed that the microdata didn’t seem to provide any additional end user functionality, but no formal assessment was conducted. mixter et al. (2014) at montana state university library detail how they created their own schema.org extension vocabulary for theses and dissertations in their institutional repository because they found the existing classes did not offer enough specificity. they used the new vocabulary to convert all of their 1909 theses and dissertations into rdf using openrefine, and published it as rdfa in their institutional repository. in the future, they planned to begin tracking the amount of structured data that is harvested by search engines. sexton and aery (2013) at duke university library presented at the coalition for networked information (cni) spring 2013 membership meeting on their approach to adding schema.org to their digital collections portal using rdfa as part of larger project to use a customized google site search interface for their digital collections. they used schema.org properties to create their own rich snippets within their google site search. they observed that since they began using schema.org, most objects didn’t look any different in big google searches, but did not detail how they tracked this. walli et al. (2017) at europeana explain how they implemented schema.org in their digital collections portal using json-ld. their work focused largely on mapping the european data model (edm) metadata from their digital collections to schema.org. they also proposed strategies for gathering analytics to assess the effects of adding schema.org to the sites but have not yet implemented them. implementation content selection in fall 2016, schema.org was applied to all digitalgeorgetown communities and collections using microdata, excluding those containing primarily citation-only records and/or restricted access records. this decision was made in order to create the widest possible access to materials in the repository, while respecting the wishes of the collection managers of restricted access collections. in order to decide which schema.org types to use, a content assessment of the repository was conducted. the content in the repository was reviewed to determine which types of items existed in the repository and which of those had corresponding schema.org types. based on this assessment, six schema.org types were selected for use: videoobject, book, visualartwork, photograph, scholarlyarticle, and creativework. types were implemented at the highest level possible in dspace (typically at the community or sub-community level, but occasionally at the collection level) based on the contents (table 1). this enabled the schema.org data to “trickle down” to the sub-communities and sub-collections, when applicable. communities and collections made up of multiple item types were often mapped to the broader creativework type. table 1. assignment of types dspace community/collection name schema.org type woodstock theological library collections, dean peter krogh foreign affairs digital archives transcripts, university archives, manuscripts collections, law library creativework institutional repository scholarlyarticle georgetown university publications, rare book collections, special collections catalog book art collections visualartwork initiative on technology enhanced learning (itel) digital archive, dean peter krogh foreign affairs digital archives videos videoobject envision church photograph in addition to using types based on content, two types based on page structure were also selected: website and breadcrumb. lit was especially interested in employing the breadcrumb type because google uses this type to display a breadcrumb trail in search results (figure 1). the breadcrumb trail is what google uses to indicate in the search results where a page falls in the website’s hierarchy (breadcrumbs … [updated 2017]), and helps the user quickly understand the context of the search result in the larger website. figure 1. breadcrumb display in google search results metadata mapping because all of the selected types are sub-types of creativework, many properties were inherited and therefore shared among types. lit was able to map the dublin core metadata used in dspace to using many of the same properties (table 2). a handful of dublin core fields mapped to properties only used for certain types (table 3). one property associated with the videoobject type, thumbnailurl, was added even though it is not used in digitalgeorgetown’s dublin core metadata because it is required by google. table 2. dublin core to schema.org mapping (all types) dublin core field(s) schema.org property dc.creator, dc.contributor author author dc.contributor.other contributor dc.publisher publisher dc.title name dc.identifier url dc.subject about dc.description, dc.description.abstract description dc.relation.ispartof, dc.relation.ispartofseries ispartof dc.coverage.spatial contentlocation dc.description.version version dc.language inlanguage dc.date.created datecreated dc.date.issued datepublished table 3. dublin core to schema.org mapping (types with unique properties) dublin core field schema.org property schema.org type dc.date.accessioned uploaddate videoobject dc.format.extent duration videoobject (none) thumbnailurl videoobject dc.identifier.isbn isbn book dc.format.medium artmedium visualartwork dc.type artform visualartwork code updates and re-indexing once the metadata mapping was complete, one of lit’s developers updated the dspace code to automatically generate the schema.org microdata and insert it into the html of both simple and full item record pages. this process took less than a week, and the code is available on github at https://github.com/georgetown-university-libraries/dspacesolutions#microtagging-code. following the release of the new code, an updated sitemap was submitted to google to request re-indexing. testing and viewing results to test the content-based microdata in the repository, the easiest method was to use google’s structured data testing tool [4]. when a user enters a url into the tool, google displays the microdata it detects on the page along with any known errors. for instance, if a required property is missing, the tool flags where the expected property should be. lit discovered that the tool shows an error for a property that is missing even if it is only required for the display of rich cards. rich cards are information boxes with images that appear in a carousel at the top of search results for certain item types, but only when certain properties are present. since the content in digitalgeorgetown doesn’t have all of the required metadata for rich cards, lit did not want or try to enable this feature; therefore, the errors shown in the structured data testing tool were irrelevant. it’s important to examine the errors from the structured data testing tool thoroughly and recognize that not all errors require action. google also shows the microdata it detects in the structured data report [5] in the google search console [6]. however, this report only displays microdata detected within the last month, and lit observed that the data it displayed was unreliable. another way to view the microdata is through changes that may be visible in the search results themselves (for example, the text from the “about” property may appear as the summary text in the search results), but this is not guaranteed. google explains: using structured data enables a feature to be present, but does not guarantee that it will be present. the google algorithm tailors search results to create what it thinks is the best search experience for a user, depending on many variables, including search history, location, and device type. in some cases it may determine that one feature is more appropriate than another, or even that a plain blue link is best (introduction to structured data … [updated 2017]). assessment measuring the impact of adding schema.org to digitalgeorgetown with microdata proved more challenging than expected. while the presence of microdata on a webpage can be verified by using the structured data testing tool, there is no single way to measure its effect on search results. this is largely due to the fact that google search result rankings are dependant on over 200 factors and are constantly in flux. according to google, “when a user enters a query, our machines search the index for matching pages and return the results we believe are the most relevant to the user.” (how google search works … 2017). the search results a user sees are personalized based on search history, location, device type, and what google knows about the user’s preferences (introduction to structured data … [updated 2017]). additionally, an item’s search result ranking may shift depending on how other websites around it change, and items with generic titles that encompass broad subject matter (i.e. “parrot”) have to compete with many more websites and may not show up in the first few pages of results. keeping these caveats in mind, lit developed three questions about the impact of microdata in digitalgeorgetown and identified a metric for each. these questions were: did items rank higher [7] in google search results after implementation than they did before? did items display a breadcrumb trail in search results after implementation? did referrals from google to digitalgeorgetown increase after implementation? search results rankings to determine if items ranked higher in google after implementation, lit sampled thirty randomly selected items from representative collections and schema.org types and measured the ranking of each before and six months after implementation. two searches were conducted for each item, the first using the item’s title for the search terms and the second using the author’s last name and a partial title for the search terms. all searches were conducted in google chrome browser (logged out of any google accounts) in incognito mode. if the item was not found in the first five pages of results, it was not counted. lit conducted a separate sampling for images and videos in google image search using ten randomly selected items of the photograph and visualartwork types and in google video search using five randomly selected items of the videoobject type. for google web search, 26.67% of search results rankings increased, 18.33% decreased, and 55% showed no change. for google image search, 10% of search results rankings increased, 30% decreased, and 60% showed no change. for google video search, none of the items showed up within the first five pages of results both before and after the implementation. breadcrumb trail to determine if a breadcrumb trail displayed in the search results after implementation, lit used the same items and search strategies from the search results ranking assessment and observed whether a breadcrumb trail appeared in search results six months after implementation. lit found that 39.39% of items that showed up in search results had a breadcrumb trail displaying. google referrals to determine if google referrals to digitalgeorgetown increased after implementation, we used google analytics to compare the total number of google referrals to digitalgeorgetown from the six months following the implementation (august 2016-february 2016) with the number of referrals from the same six month period over last three fiscal years. we found that there was substantially higher growth rate in 2016 (63.30%), compared to average growth rate from the last three fiscal years (23.27%). discussion examining three different metrics helped to provide a fuller picture of the microdata’s impact on google search results. while the search results ranking data demonstrated an overall positive impact for google web search, over 50% of results still showed no change in ranking. interestingly, the search results ranking data for google image search demonstrated an overall negative change, which is likely due to factors outside of lit’s control. additionally, the search results ranking data for google video search showed that the microdata had no impact at all, as none of the items showed up in results before or after the implementation. google video search requires items to have a “thumbnailurl” property in order to be included in results, yet none of the videos in digitalgeorgetown have thumbnail images, so that property was left blank. in addition, the videos don’t actually live in digitalgeorgetown, but are embedded from sharestream. the combination of these factors could have easily led to the videos absence in google video search results. the relatively low number of breadcrumb trails appearing in search results (39.39%) was disappointing, but not surprising. as mentioned earlier, google emphasizes that just because a feature (such as breadcrumbs) is enabled, this does not guarantee that it will be present in search results (introduction to structured data … [updated 2017]). the significant increase in google referrals was the most encouraging metric. presumably, the reason for this growth is that items are now more visible in google search results, so they are clicked on more frequently, resulting in a higher referral rate. overall, the results of our assessment show a small, yet visible impact on search results. while the presentation of results in google search was inconsistent across items, there were certain observable changes, including fluctuation in search result ranking, appearance of a breadcrumb trail, and increased google referrals. libraries taking on a similar project should be aware that google may take a long time to recognize the presence of microdata in a large repository like digitalgeorgetown, so it is important to set expectations accordingly and to ensure enough time passes before comparing the before and after of search results. conclusion this project demonstrates a low-barrier approach to integrating schema.org vocabularies into an institutional repository using microdata. one of the most time-consuming parts of the process was selecting which types and properties to use because there are so many options within schema.org and because they will necessarily be unique to each institutional repository depending on which metadata fields are already in use. demonstrating impact was also difficult because there are so many factors that influence how search results display in google and are beyond the library’s control. additionally, changes in search results aren’t immediately visible or consistent; therefore, assessing multiple metrics to paint a larger picture of the overall impact is key. while the use case described in this paper is specific to dspace, similar workflows and assessment strategies could be applied to any open source repository. overall, the work completed had small, but positive effects on the visibility of digitalgeorgetown items in search engine results, helping to improve the discoverability of the library’s open access resources. notes [1] http://schema.org/docs/schemas.html [2] http://www.library.georgetown.edu/digitalgeorgetown [3] google now recommends using json-ld according to https://developers.google.com/search/docs/guides/intro-structured-data. [4] https://search.google.com/structured-data/testing-tool [5] https://www.google.com/webmasters/tools/structured-data [6] https://www.google.com/webmasters/tools [7] “higher” meaning that the result appears closer to the top of the page, as opposed to the number of the result. for example, a search result ranking of three is higher than a search result ranking of ten. references ronallo, j. 2012. html5 microdata and schema.org. code4lib [internet]. [cited 2017 jul 31]; 16. available from: http://journal.code4lib.org/articles/6400 columbia university academic commons [internet]. [updated 2017 jul 28]. nottingham (uk): university of nottingham; [cited 2017 jul 31]. available from: http://opendoar.org/id/1317/ hilliker, r., wacker, m., nurnberger, a.l. 2013. improving discovery of and access to digital repository contents using semantic web standards: columbia university’s academic commons. journal of library metadata [internet]. [cited 2017 jul 31]; 13(2-3): 80-94. available from: http://dx.doi.org/10.1080/19386389.2013.826036 mixter, j., o’brien, p., arlitsch, k. 2014. describing theses and dissertations using schema.org. in: moen, w., rushing, a., editors. metadata intersections: bridging the archipelago of cultural memory. proceedings of the international conference on dublin core and metadata applications; 2014 oct 8-11; austin. dublin core metadata initiative. p. 138-146. available from: http://dcpapers.dublincore.org/pubs/article/view/3715 sexton, w., aery, s. 2013. video: using schema.org & google site search with with library digital collections [internet]. washington, dc: cni; [cited 2017 jul 31]. available from: https://www.cni.org/news/video-using-schema-org-google-site-search-with-library-digital-collections walli, r., isaac, a., charles, v., manguinhas, h. 2017. recommendations for the application of schema.org to aggregated cultural heritage metadata to increase relevance and visibility to search engines: the case of europeana. code4lib [internet]. [cited 2017 jul 31]; 36. available from: http://journal.code4lib.org/articles/12330 breadcrumbs [internet]. [updated 2017 jan 26]; mountain view, ca: google; [cited 2017 jul 31]. available from: https://developers.google.com/search/docs/data-types/breadcrumbs introduction to structured data [internet]. [updated 2017 jul 10]. mountain view, ca: google; [cited 2017 jul 31]. available from: https://developers.google.com/search/docs/guides/intro-structured-data how google search works [internet]. 2017. mountain view, ca: google; [cited 2017 jul 31]. available from: https://support.google.com/webmasters/answer/70897?hl=en about the author shayna pekala (shayna.pekala@georgetown.edu) is the discovery services librarian at georgetown university library, where she administers and supports the library’s discovery systems, including the integrated library system, catalog, and discovery layer. her research interests include discovery tools, user experience, and privacy. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – alternative solutions for off-campus authentication mission editorial committee process and structure code4lib issue 3, 2008-06-23 alternative solutions for off-campus authentication the ohio state university libraries created scripts to overcome the local limitations of the proxy server and to offer resource connections at the point of need. all libraries struggle to provide seamless authentication for access to paid resources, such as research databases and electronic journals. in order to obtain access to this content, the libraries must sign contracts promising to limit access to these resources to their user community. the challenge then comes in balancing the patron’s need for easy access to these rich data sources from any computer and the vendors’ desire to protect their assets. by rebekah kilzer, elizabeth l. black and james muir background since the increase in the need for distributed access to library resources, libraries have been struggling with methods to provide seamless, intuitive access for people who are not at the physical library. users expect to be able to do their work from anywhere, including being able to access library resources such as paid databases. some libraries offer remote access via internet protocol (ip) authentication, but that alone does not address the needs of the people who are out of the organization's ip range (blansit, 2007). the ohio state university libraries, like many libraries, have addressed this need by using a proxy server to act as an intermediary between the user and the database. webster (2002) offers a clear definition of a proxy server, pointing out that a proxy server makes it appear as though users are working from valid ip addresses associated with the university, instead of off-site. the literature exposes a variety of issues involved with offering off-site/off-campus access to paid resources via a proxy server. mikesell (2004) offered a discussion on some of the weaknesses of proxy servers, which is useful in framing the various options. in order to authenticate users through the proxy server, libraries often employ one of a few methods. one method is to require users to sign into the proxy server before beginning their search, which is the ohio state university libraries’ current method. some libraries create an "off-campus users" portal through which all off-campus users are required to enter before using paid library resources (hudock, 2003) but this does not address the important goal of offering tools to the users at the point of need (ferguson & bunge, 1997). another method is to include the proxy string in the resource url, so the user is prompted to sign in when they click on the resource url. this approach offers the most seamless access at the point of need, but can be an obstacle when there are thousands of resources that need to be sorted out, and access levels assigned, as was the case at osu. the ohio state university libraries subscribe to 450 databases and have purchased several thousand additional resources. because osu is a public university, there is a mandate to offer certain resources without requiring affiliation with the university, adding an additional layer of complexity. in addition, the size of the library catalog database would have made the updating of records a protracted and labor intensive exercise, even with the automated tools available for use. solution the it team began brainstorming and tested a promising idea of adding code to the catalog to act on urls only when they are selected. after several iterations to account for the inability to put php into the library catalog’s local customization code and the existence of multiple urls on a single catalog page, the final successful scripts combined javascript and php. the javascript code was placed in both the catalog and the web site templates. when a user clicks a url, the javascript calls a php file, which checks the ip of the user against a campus ip list. if the ip is on-campus, the user is passed to the url selected. if the user is off-campus, the php code checks the marc 945 field in the catalog record for an exception code. if the exception code is present, the user is passed on; if the code is not present, the user is sent to ezproxy and forwarded to shibboleth to authenticate. shibboleth is the authentication and authorization framework used at the ohio state university. it is a suite of open source, standards-based solutions that enable the exchange of information about users in a secure manner (internet2, 2008). the exception code was a necessary element to the political success of this idea. osu is a public, land grant institution and the libraries house many unique materials of interest to the community beyond the university. therefore, it was essential that we have a method of keeping access to the finding aids and enhanced descriptions linked from the catalog records open to those who would not have the required credentials to authenticate. exception codes were added to items in special collections, the institutional repository and government documents selected for their likelihood to be accessed by non-osu users. due to the unique challenges of identifying all government documents in the catalog, we did not attempt to retroactively add the exception code for all records but the code will be added to records for all government documents acquired in the future. we also added a feedback feature to the login page so users can report issues, including items that should be made exceptions. if a user has javascript turned off, the url still works, but the logic of the ip check cannot be run. therefore, if off-campus, the user must login to the proxy first before clicking links to paid resources or will be presented with a vendor login page. we decided to leave the off-campus sign-on link in the web site global navigation to accommodate these users and those who were comfortable with the previous method. offering access to users at the point of need is essential in today’s library services environment. implementing the combination of javascript and php code to support this access, while still accommodating the libraries’ standards, has proven to be a solid asset for the libraries. code there are four files available. download zip (5.36 k). note: these files were designed for use with an innovative catalog and the library web site, separately, but could be adapted for other library systems. offcampuscheck.js — this file should be included in the catalog template; it is recommended that it be included in the footer. it refers to urls appearing in the catalog record. offcampuscheck.php — this is the php script that runs the check in the catalog pages and is referenced from the javascript above. generic_offcampuscheck.js — this file should be included in non-catalog web pages. footer_code.php — insert this code in the bottom of your pages (we added it to our global footer file). references blansit, b. d. (2007). beyond password protection: methods for remote patron authentication. journal of electronic resources in medical libraries. 4 (1/2), 185-194. (coins) ferguson, c. d., & bunge, c. a. (1997). the shape of services to come: values-based reference service for the largely digital library. college and research libraries. 58 (3), 252-266. (coins) internet2, "about shibboleth." http://shibboleth.internet2.edu/about.html (accessed may 13, 2008). mikesell, b. l. (2003). anything, anytime, anywhere: proxy servers, shibboleth, and the dream of the digital library. journal of library administration. 41 (1/2), 315-326. (coins) webster, p. (2002). remote patron validation: posting a proxy server at the digital doorway. computers in libraries, 22 (8), 18-23. (coins) acknowledgments the authors wish to acknowledge jason thompson, the member of the it team who had the original idea and without whom this project would never have happened. about the authors rebekah kilzer is systems librarian and assistant professor at the ohio state university libraries. she is involved in several projects to enhance the discovery and delivery of library resources to patrons. her email is kilzer.2@osu.edu. elizabeth l. “beth” black, is an assistant professor and systems librarian for the osu libraries. she leads the system’s web implementation team, and is responsible for the libraries’ web site, web applications and the university’s digital repository, knowledge bank. she can be reached by email at black.367@osu.edu. james muir is a systems developer/engineer for the ohio state university libraries. he is involved with creating and maintaining many web applications for osu libraries. he can be reached by email at muir.29@osu.edu subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – building a better book widget: using alma analytics to automate new book discovery mission editorial committee process and structure code4lib issue 44, 2019-05-06 building a better book widget: using alma analytics to automate new book discovery are we doing enough to market newly acquired book titles? libraries purchase and subscribe to many new book titles each year, both print and electronic. however, we rely on the expectation that users will periodically search our systems to discover newly acquired titles. static lists and displays have been traditional marketing methods for libraries, but require tedious time and effort to maintain. without a practical solution for an academic library, east tennessee state university developed an automated process to generate book widgets utilizing data from alma analytics. these widgets are now deployed in our subject guides, website, and on our digital displays. this article outlines the development and implementation of these widgets. we also discuss the challenges we encountered, such as finding image covers and custom subject tagging. by travis clamon introduction libraries have traditionally promoted new titles utilizing a shelf or display in the library to encourage discovery. technology in the past decade has enabled the adoption of electronic books as well as growth in online education. while the idea of new title discovery is still important, the traditional method relies on an outdated expectation that users solely rely on the physical presence of the library. this method of delivery and promotion needs to adapt digitally and incorporate electronic and physical books as a whole. while discovery systems handle virtual searching and browsing of books, they do not enable new title discovery out of the box. some librarians often resort to creating static lists of curated titles. however, current e-book ownership models have complicated matters of keeping displays and title lists current. the number of additions and deletions often reaches the thousands as ebook vendors release updates throughout each semester. to have reliable data, we need to be able to retrieve automated data to keep titles accurate. at east tennessee state university, we have developed a program that enables automated new title discovery utilizing alma, our current library management system. specifically, we used alma analytics, the reporting component of alma. these reports, which contain new title data, refresh each day when alma runs indexing. this article will outline the development and implementation of these widgets. project background and goals in the past two years, we have made multiple efforts in addressing the goal of new title discovery. the initial idea of automated book sliders occurred when the library was redesigning the website in 2017. one primary task of the redesign was to create uniform subject guides for all of our academic programs. one initiative of the subject guides was to make users aware of the new books in their subject or program. our efforts of creating a book slider for each subject guide relied on separate alma analytics reports and scripts for automation. shortly after the website went live, the library also implemented a new print book widget on our digital display. we soon realized the workflow of each widget was becoming redundant and could not scale to include multiple subjects, formats or templates. implementing new data columns or changes required editing numerous analytic reports and scripts. as the number of individual book widgets and interest from librarians increased, we decided in late 2018 to rewrite our code to introduce new features and eliminate redundancy. the development of the project happened as time allowed by the electronic resources librarian. a total time is unavailable, but we estimate that 50 hours went into developing the new version. before the rewrite began, we identified technical and user requirements that would satisfy existing workflow issues: implement widgets on subject guides, webpages, and digital displays. multiple templates to choose from such as slider widgets and generated table lists. new templates are added without additional configuration. the ability to pull data on new title information (print and electronic) from a minimal number of alma analytic reports. create a widget based on a particular subject, format, or physical location. ease of use; limited technical expertise needed to create a widget high-level overview figure 1. high-level overview we divided development into three distinct processes: alma analytics: reports have been created that warehouse the new book data. we then retrieve the data using the alma analytics api. data transformation and storage: we transform various data fields and store them into an array file. selection and display: accepts the array in step 2 along with additional parameters to create the widget. a selection menu allows librarians to preview their widget and receive an embed code for display. section 1: alma analytics the first step in the process is creating an alma analytics report. when creating a new report, a user will first select a subject area. since our goal is to include both print and electronic books, we created an analytics report for each format in order to capture all necessary data fields. the print report comes from the “physical items” subject area, while the electronic report comes from the “e-inventory” subject area. report fields an important next step of the process is identifying the data fields needed for the project. this is dependent upon the intended uses of the widgets and the information you want to provide. based on our needs and previous experiences, we were able to identify the following fields in each format: data field physical items report e-inventory report title bibliographic details -> title author bibliographic details -> author isbn bibliographic details -> isbn (normalized) publication date bibliographic details -> publication date upload date physical item details -> creation date portfolio creation date -> portfolio creation date subjects bibliographic details -> subjects lc classification group lc classifications -> group1 lc classifications -> group2 lc classifications -> group3 lc classifications -> group4 call number holding details -> permanent call number n/a location code location -> location code n/a location name location -> location name n/a collection n/a electronic collection -> electronic collection public name mms_id bibliographic details -> mms id table 1: alma analytics report filters for physical and electronic items report filters report filters will define the overall target collection of books. at this step, we had to define what our library considered “new” for electronic and print formats. book purchasing and subscription changes will vary in each library. we also imposed a publication date range to ensure that we do not feature older materials. we tested various year ranges to get an idea of the amount of results we wanted to store in a dataset. east tennessee state university defined the following criteria for each report: publication date must be between the past three years up until the future year. isbn is not null. the isbn is used for linking and cover image purposes. portfolio or item creation date must be during the past 180 days. item must be available or activated for online use. specification of item locations and/or electronic collections are optional. we excluded government document books using this filter since they were not relevant. if the location contains more than books, a material type filter is required. retrieving report data from alma analytics api once the report was created and we felt satisfied with the selection criteria, we used the ex libris alma retrieve analytics report api (https://developers.exlibrisgroup.com/alma/apis/docs/analytics/) to receive and store the data from both analytics reports. the api requires a key along with the path to the analytics report. if more than 1000 results are returned from an individual report, a resumption token allows you to receive the remaining results. one issue we encountered when receiving the data is that the column names are generic (column1, column2, etc.). the column numbers returned did not match the column order in the alma analytics report display either. also, we noticed fields such as the lc classification groups had different column numbers between the print and electronic responses. fortunately, these columns remain the same after repeat api calls. we resolved this matter by creating an array for each report in our script that cross-matches the response column number to a more appropriate variable name. section 2: data transformation and storage once we have the data from alma analytics, some additional transformation is required to get the data into a usable format. from the api response, we convert the data to an xml element and process each column using a foreach statement. most of the fields were a direct copy over into the final array, but others needed some additional work: title: we noticed some of our book titles had all upper case letters, mostly due to one particular vendor’s marc records. in order to make a consistent appearance, we used the php mb_convert_case function and specified a mb_case_title conversion. slashes were also common at the end of each title. we used regex to remove the slashes. publication date: this field varied by record and sometimes included copyright symbols, periods, and brackets. we used regex again to remove the invalid characters and leave only digits remaining. this was required so we could sort the publication date correctly. isbn the isbn is used for cover image and linking purposes. since most records contain multiple isbn numbers, we needed a way to determine which isbn would return a cover image from our discovery system, ex libris primo. we first started using the first isbn provided, but quickly realized that it did not always return a valid image. we resolved the issue by breaking up the isbn’s into an array. each isbn assembles as an url and then a curl call checks the file size of the url. the primo service returns a small 1×1 white pixel if a cover image is not available. we determined the white pixel returned by primo was consistently 86 bytes. once the first isbn came across that had a file size other than 86 bytes, the loop would end and the isbn would be become the preferred value. at this point, we analyzed the data again and determined that around 1% of print books and 5% of electronic books were unable to retrieve a cover image from primo. we queried these isbn’s with google books and were able to retrieve a few more cover images. this resulted in us modifying the code to do an additional call using the google books api (https://developers.google.com/books/) only in cases when a cover image is unavailable from primo. we quickly noticed that adding the full array of isbn’s along with the google books api made the import process much longer and would sometimes timeout. in order to reduce the amount of repeat curl calls every time the import process occurs, we started storing the cover image url, current date, and the corresponding mms_id (alma record id) in a separate array file. we scan this array before initiating a curl call. the array retains data between imports. as time continues and the array grows, we will clean out older records by filtering the date stamp. figure 2. cover image retrieval process using primo and google books api (see code) custom subject tagging another relevant section was finding a way to assign custom subject tags to books. these tags correlate to the subject guides available on the library website. due to the variations of vendor-supplied marc records, we determined keyword matching requires the lc group classification, title, and subjects columns. a case statement contains all fifty-seven custom subject terms for the university. we made matches for each subject tag to the corresponding library of congress classification terms. if a match does not occur by lc group, we attempt to do a keyword match using the subjects and title columns from the report. in a similar fashion, custom keywords are also matched using a case statement. these case statement files are stored separately as include files in the subjects folder and are available to reference from the gitlab repository (https://gitlab.com/clamon/alma-book-widget). links subject guide widgets have links attached to book covers, along with titles in a list view. instead of linking directly to an ebook or a detail page for a physical book, we decided to send all requests to our “get it @ etsu” link resolver. we decided on this approach for statistical and maintenance reasons. the primary isbn is the parameter the link resolver uses to locate the title. the other parameter we used is the service id “sid” parameter. we assigned a default value of books_widget. this value allows us to retrieve usage statistics using the link resolver object in alma analytics. the link resolver can also provide the most current availability information on a book, rather than the 24-hour index window in alma analytics. figure 3.  sample url linking to the primo services page using isbn and source id parameters. figure 4. sample alma analytics link resolver report of widget click usage. data storage and automation once all the data processes, the final output consists of a multidimensional array stored as a file inside the export folder. the array size varies, but it typically contains between 1,000 to 5,000 book titles. we chose the array file over a database due to low maintenance and its compatibility with the datatables plugin. this process runs daily using a cron job on our server. to confirm that the call and import were successful, we inserted a slack webhook (https://api.slack.com/incoming-webhooks) to notify us of the status each day. section 3: selection and display we developed the selection and display component separately from the transformation part. this component accepts the multidimensional array file created after transformation. when generating a widget, three parameters are required: filter type, filter query, and template. these parameters are imported from php $_get variables in the url. the filter type correlates to an individual data column inside the multidimensional array. with our current setup, we can filter data on types such as title, author, publication year, location code, subject, or type (physical/electronic). the filter query is the search keyword. we created a function that accepts these values, searches the array, and calls upon the required template to generate the widget html. function createslider($array_source, $filter, $query, $template) { $results_array = array(); //find results foreach ($array_source as $key =&gt; $val){ if(is_array($val[$filter]) === true) { foreach($val[$filter] as $term) { if(stripos($term, $query)!== false){ $results_array[] = $val; }//end if }//end for } elseif(stripos($val[$filter], $query)!== false){ $results_array[] = $val; }//end else if }//end foreach //pass results_array to the designated template require "templates/" . $template . ".php"; }//end createslider for ease of use, we developed an interface that assists with creating a widget. we scan the original array to show all possible combinations of data for filter types as well as templates. both title and author allow free-text input, while the other fields present as options (radio buttons). once all selections are made, a widget preview frame appears, and embed code is generated. example widget selections: new books located on the second floor (location code). new books published in 2019. new books subject-tagged in nursing, computing, etc. new books in research methods (free-text keyword). figure 5. internal interface that assists with book selection and widget implementation display templates a template folder retains multiple different widget display options. we had three formats we wanted to begin with: a small sliding widget for subject guides, a full-screen slider widget for our digital display, and a list view for informational needs. from a technical perspective, display.php processes the widget query, and a new json array is created with the results. at this point, we use the template parameter specified from the url to include the corresponding php file (“template”) that is being used. each template file will then process the results array as needed to create the intended widget. since most of the development time focused on building the data array, we wanted to utilize open source components for our templates. these plugins are stored in a separate folder for easy updating. for the sliding widgets, we found a carousel package called slick developed by ken wheeler (http://kenwheeler.github.io/slick/). this package allows for single and multi-item sliders that are responsive to mobile devices. for the subject guides, we used a multi-item carousel and specified three different responsive configurations. the setup allowed us to show seven book covers at a time for desktop displays while offering a mobile and tablet friendly display of two to five sliding book covers. html title tags were added containing book titles for accessibility purposes. the slider automatically rotates every few seconds and the arrow buttons allow for manual control. figure 6. slider widget embedded on the history subject guide (https://libraries.etsu.edu/research/guides/history/books) we encountered some initial difficulty importing this widget onto our subject guides. we tried first using the remote script option, and while the slider would initially appear, any screen resizing would cause the widget to disappear. we determined there was a jquery conflict between libguides cms and the slick package. instead of using the remote script function to pull in the widget, we ultimately decided to use an iframe. since html code is necessary, instructions and code are provided at the selection menu along with a copy to clipboard function. for the digital display, we utilized the single item carousel and customized the css to fit the display appropriately. in regards to visual design, we chose a dark background with white text that conveys title, author, and location. since the display constantly rotates throughout the day, we configured brightsign (our digital display hardware) to refresh the webpage every 15 minutes. when it refreshes, the script will randomly choose a new order of titles to give a new and fresh appearance. figure 7. slider widget embedded on our digital display inside the main entrance. the list view is useful when providing an overall list of new books. we were able implement the datatables plugin (https://datatables.net/), which enhances html tables with little effort. datatables accepts our converted json array and displays the appropriate fields using a custom display configuration. the subject and type (electronic /physical) were configured as dropdown sorting menus. we had to do some additional work to get the subject dropdown to work and sort alphabetically by breaking up the array of tagged subjects. to reduce loading time, we set deferrender equal to true so data is only loaded for the current page of results and not the entire book array. this is important when you are displaying book covers so the browser does not request thousands of images at once. we also implemented button plugins to allow for excel data exports. this feature is useful for end users who want an offline copy, but as well for librarians who want to send a copy to interested faculty. we also explored pdf exports using the pdfmake plugin, but we encountered issues with book url’s splitting into multiple lines on the pdf. adobe reader would only recognize the first line of the url when clicked, and therefore would result in a bad request. this list is available on our main books page and linked from our display widgets on our subject guides. figure 8. list widget utilizing datatables to display the entire array of new books (https://libraries.etsu.edu/research/books/new) limitations as designed, a widget only supports one filter parameter. there may be a future requirement to support multiple filters, primarily if a large repository of books exists in the alma analytics report. we plan to wait and see how the widgets are used before deciding to rewrite the code. subject tagging is another challenging part of the project. it is important that appropriate subjects matching occur. this ensures our users are seeing all new relevant titles in their subject area. we plan to keep expanding the keyword list and look into the alma configuration to see why specific titles are not inheriting lc group classifications in alma analytics. for electronic books, we determine whether an item is new or not by the portfolio creation date. in some cases, when we have to reload an entire electronic collection in alma, this will cause all of those titles to appear as new again. so far, this has not been an issue, but we currently do not have a workaround devised. during the cover image retrieval process, it should be noted that the google books api has a daily quota limit. we were assigned a limit of 1,000 calls per day. we have made every effort to only call the google api when needed, but if a large load of new books comes through, it may exceed the limit. you can request an higher quota, but will have to provide more information about your project to google. conclusion the development of this widget tool has been a worthwhile project for the library. the digital display widget on our main lobby has captured a lot of attention by both patrons and librarians. our feedback for the display has been positive overall, and we have made minor adjustments such as extending the timer between title changes to allow for longer recognition of the item location and call number (if physical). we updated subject guides to include the new widgets and were able to eliminate over 50 old alma analytic reports. performing results filtering in the script rather than in individual alma analytics reports allows for easier modifications and keyword updates. to capture eye attention to the sliding widgets, we positioned them at the top of the template and enabled auto scrolling. an example of our subject guide widget is available at https://libraries.etsu.edu/research/guides/history/books. while the intended purpose of the project is to highlight new books, there is potential for other uses such as highlighting a special collection. all that would be required is modifying the filter criteria in alma analytics. as funds allow, we are hopeful that more digital displays are installed throughout the library, and we can focus on marketing books by collection or floor. a goal within the next year is to interview students and faculty and receive additional feedback formally. there may also be potential to expand these displays outside the library, such as in our student center and academic buildings. the code for this project can be found on the author’s gitlab page (https://gitlab.com/clamon/alma-book-widget). acknowledgements the author would like to thank co-workers mark-hero agbenu and destin hebner for their invaluable assistance and support with datatables. about the author travis clamon (clamon@etsu.edu) is the electronic resources librarian at east tennessee state university in johnson city, tn. he holds an m.s. in information science from the university of tennessee and a b.s. in computer science at east tennessee state university. his professional interests include discovery, user experience, electronic resource management, and data analytics. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – copyright and access restrictions–providing access to the digital collections of leiden university libraries with conditional access rights mission editorial committee process and structure code4lib issue 41, 2018-08-09 copyright and access restrictions–providing access to the digital collections of leiden university libraries with conditional access rights to provide access to the digitized collections without breaking any copyright laws, leiden university library built a copyright module for their islandora-based repository. the project was not just about building a technical solution, but also addressed policy, metadata, and workflows. a fine-grained system of access rights was set up, distinguishing conditions based on metadata, ip address, authentication and user role. by saskia van bergen and lucas van schaik introduction leiden university has a long tradition of research and education dedicated to non-western cultures, specifically asian and middle eastern cultures. this focus was in large part driven by the historical ties between the netherlands and the countries formerly under colonial rule. as a result leiden university libraries (ubl) owns a significant number of non-western heritage collections. in 2013 the library of the royal tropical institute (kit) was closed and the part of the collection that dealt with the former dutch colonies was transferred to leiden university libraries, a vast collection which included unique historical materials about the former colonies of the netherlands. in 2014 leiden’s collections on southeast asia, the pacific and the caribbean grew through the acquisition of the heritage collection of the royal netherlands institute of southeast asian and caribbean studies (kitlv). the many non-western objects held by leiden university libraries can be of great value both to researchers worldwide and to the general public in the regions in which these objects originated. however, because of the costs of travel it is often difficult for them to consult these objects physically in the library’s reading room. for this reason ubl decided to invest considerably in improving the digital access to these collections. as an important step in this process the library redesigned its repository infrastructure and replaced all commercial, home-built and (partly outdated) open source applications for ingest, storage and presentation with one open source, multi-functional environment based on islandora. in the course of 2018, the new online interface for leiden university libraries digital collections will contain all digitized heritage collections of leiden university libraries. these include manuscripts, early printed books, maps, atlases, prints, drawings, and photographs; all together around 3 million scans and 50 tb. the new infrastructure offers a service that is both manageable and trustworthy. it focuses on the long-term preservation of objects and their metadata. researchers and students can easily view and download digital objects through the online interface, and specific groups of objects can be made available as a separate collection by creating a home page for each of them. the objects and metadata can be requested by external parties through standardized exchange protocols, apis and persistent identifiers. in recent years, european cultural heritage institutions have invested heavily in the digitization and online availability of their collections. in accordance with the intellectual property rights, permission from copyright holders is required for both reproduction and online dissemination. copyright in the netherlands is laid down in the auteurswet (dutch copyright act). the copyright is valid for up to 70 years after the creator’s death. this means that during this period creators (or their heirs) have the exclusive right to authorize and prohibit the use of their works. dutch heritage organizations have for a long time devoted little attention to the follow-up of copyright restrictions. this was based on the idea that they are non-commercial organizations and work for research and educational purposes. thus not all legacy applications maintained by the ubl were equipped with sufficient functionality to comply with dutch copyright law. over the past few years however, several heritage organizations have been brought before court for copyright infringement and as a result they have begun to respond to these requirements. the same goes for our library. collections that had been made available online in the past had to be reconsidered, and if any copyright infringement had occurred, measures had to be taken. fortunately, the islandora infrastructure does enable us to make the availability of digital objects dependent on copyright regulations in a flexible way. in this article we will focus on the implementation of a copyright module for islandora. altogether this project was not just about building a technical solution, but also considered policy, metadata, and workflows. policies and workflows policy most collections at ubl were digitized for preservation purposes, mainly as part of the dutch metamorfoze programme. by digitizing them the originals can be protected from handling, while their content is still available to our users. at the start of the project many digital objects were made available even though they were under copyright protection. with copyright compliance now a priority, this had to be corrected. we also needed to make it unambiguous on a general level what our users can and cannot do with our data. are they allowed to download and reuse our metadata? and can they share, reuse and modify our jpeg files only, or the tiff archival object as well? because our library did not yet have a clear policy for the digital collections, this was the first step necessary for setting up the workflows. we decided that all data are made as openly available as possible, in as many places as possible. if copyright permits, users can download: jpg files of individual scans pdf files of complete works ocr in txt and xml formats (if available) mp3 files of audiovisual materials exceptions can be made for collections or items, depending on the requirements of copyright owners. collections subject to copyright restrictions are made available on a dedicated terminal only accessible in the main building of the library. metadata and copyright clearance all items in our collections are catalogued in our main library management system alma, including archival materials, prints, maps and photographs. alma is the source for the metadata in our discovery system primo and also for islandora. for this reason, we have created a module that uses an oai-pmh enabled source to synchronize the metadata to islandora. synchronization can be performed for an individual item (manage, synchronize) or for a batch. for the physical and digital objects we make two separate records, both with their own metadata and services: the metadata for the physical object contains a request button, while the digital object record contains a handle persistent identifier leading users to the digitized content in islandora. access to all copyright-protected works that were still digitally accessible had to be restricted. a major problem was that the metadata was not always of sufficient quality to determine whether an object was free of copyright (for example, a photographer’s date of death not recorded). over the past few years we have worked hard to improve this metadata. to be able to clear the copyrights, we had to retrieve the contact details of the copyright owners. in the netherlands various agencies offer their assistance in this, including the dutch knowledge center for digital heritage and culture (den) and kennisland. if after careful research the right holders are not known or cannot be traced, one can register the work in the european database for orphaned works and make it publicly available. the national office for orphaned works in the hague offers assistance in determining whether a work can be registered as unidentified. in practice copyright clearance is an intensive and costly process. for this reason, more and more dutch organizations decide to save themselves the effort and switch to the services of a copyright collecting agency. however, the digital asian collections were the main bottleneck for ubl. the clearing and registration of these collections is hampered by a variety of problems, such as language and spelling variations and difficulties in determining years of life and contact details of copyright owners. and although more and more copyright collecting agencies and companies are becoming active in asia, the number still lags far behind that of europe. this meant that – despite our efforts – parts of these collections can only be made available within the walls of the library. rights workflow to be able to make our digital collections available without infringement of any law, we had to create separate workflows for each possible copyright situation. each workflow starts with an analysis, the result is documented in the metadata and this leads to specific access actions. to analyze the status and potential risks of copyright infringement for our digital collections we made use of the licenses granted by both creative commons and rightsstatements, established by europeana and dpla. we distinguished four main categories: works with copyright restrictions works in the public domain works for which the copyright is not evaluated (most asian collections) orphaned works for works with an unknown author or creator, we decided that textual materials made or published before 1900 and maps published more than 70 years ago would be made available free of restrictions. the risk of legal action is very low for these materials. the next step was to create separate workflows for: collections which were not evaluated yet at the item level collections and items within copyright collections and items in the public domain a separate workflow was created for collections which are in use internally by research groups of leiden university and cannot be made available (yet) to other users, because of copyright and/or privacy issues. for each collection the rights and the online availability were registered with one or more of the following permission scopes: access with ulcn account services login, the single sign on environment for students, researchers and guests of leiden university access within the walls of the library building only, on dedicated desktop terminals viewing and streaming rights only: for collections with possible copyright issues, but limited risks access and download rights for everyone, anywhere, anytime this could be one both on item and collection level. the process consists of two steps. first, the access rights for each digital object of a collection are recorded in alma. 506 restrictions on access note (r) $a streaming access (assign first indicator 1) $a download provided (assign first indicator 0) 540 terms of use and reproduction note (r) $a copyright not evaluated $u http://rightsstatements.org/vocab/cne/1.0/ $a this resource is protected by copyright $u http://rightsstatements.org/vocab/inc/1.0/ figure 1. examples of copyright information coded in marc21 this results in a mods file in islandora containing two separate fields for access condition : <accesscondition type="restriction on access">download provided</accesscondition> <accesscondition xmlns:xlink="http://www.w3.org/1999/xlink" type="use and reproduction" xlink:href="https://creativecommons.org/licenses/by/4.0/"> use of this resource is governed by the terms and conditions of the creative commons cc-by license</accesscondition> figure 2. example of copyright information coded in mods to make it clear to users what the access rights for each object are texts, links and images are displayed both below the metadata and inside a download menu (figure 3). the download menu lists all of the available datastreams and indicates whether they are available for download (see figure 4). items that are part of a compound object, like pages of a book and sheets of a map, can have their own access actions as well. if there is no data available (for example for a page in a book), the data of the compound object (the book itself) is used. figure 3. example of public record display figure 4. example context menu islandora conditional access rights module to handle the access rights and display the copyright statements, we built a module called islandora conditional access rights, a drupal module in php. the module pulls the access rights and the displayed copyright statements from the metadata of the object in islandora, synchronized from alma. the module defines conditions and access actions it uses to determine user access. the conditions are evaluated when a user attempts to view an object. based on the outcome of the conditions, certain access actions are performed. an access action is a single effect the user perceives when accessing the object; for example denying access to the object or displaying a download menu. islandora hooks islandora conditional access rights uses islandora hooks to modify the access restrictions of objects and datastreams. islandora hooks are an implementation of drupal hooks which allows modules to change the behavior and data of other hook modules. because islandora itself is partly implemented as drupal modules, it also defines its own hooks. object access hook the islandora object access hook is used to establish the access rights at the object level. if the “deny access to object” access action’s value is true for an object, access to this object is restricted. if the value is false or the value is absent, users have access to this object, unless another module prevents access. if the object subordinates to another object, such as a page in a book, and the child object does not have its own access action, the parent object is checked to determine access rights. datastream access hook the islandora datastream access hook is used to establish the access rights at the datastream level. there are two access actions that determine if the user has access: deny access to datastream if the datastream is present in this access action but not in the “allow access to datastream” access action, the access to the datastream is not allowed. allow access to datastream if the datastream is present in this access action but not in the “deny access to datastream” access action, the access to the datastream is allowed. if the access rights are not handled by these two access actions (because the datastream is not defined or it is defined in both access actions), users have access to this datastream, unless another module restricts access. just as is the case with the object access option, the parent is checked as well for compound objects. conditional statements islandora conditional access rights uses the following information to determine access rights for an object: copyright statement contained in the metadata ip address of the user user authentication user role figure 5 shows the relationship between conditions and access actions. the configuration of the module allows any number of conditions and access actions for a single object. green indicates the condition is met and the access action is taken. figure 5. conditional relationship and access action model a single condition can only be true or false; either the condition is met or not. if a condition is met, the related access action is performed. the module defines six condition types to determine access actions. metadata condition type this condition type compares a value in a specific metadata field to a given value. the metadata field is determined by the datastream id and an xpath in the module. in islandora it is possible to have multiple metadata xml files per object, so the datastream id identifies the xml file to use and the xpath identifies the metadata value to use for the access action. the object’s metadata field is referenced by the metadata key field “restriction_on_access”, which is defined in the module’s configuration [metadata:restriction_on_access] and then used in the condition (metadata = restriction_on_access). location condition type this is used to restrict access to terminals within the library building. this condition is met when the ip address of the machine of the user lies within a specific range or ranges. the ip address of the machine of the user is used when no reverse proxy is employed. when it is behind a reverse proxy the value of the x-forwarded-for header field is used by the module. ipv6 is supported by the module. authorization level condition type this condition is met when the user is authenticated and has a specific role in islandora. this can be used to differentiate between the access rights of employees and end users. and, or, and not conditions three conditions used by the module determine access actions by comparing other conditions. these either negate, conjunct, or disjunct those conditions. negating condition type this condition is fulfilled when another condition is not met. this can be used with a location condition for users outside an ip range, or in case a metadata value is not found. conjunctive condition type this condition is met when all related conditions are met. this can be used when the user has a specific role and is within a specific ip range. disjunctive condition this condition is met when any of the related conditions are met. this can be used when the user has a specific role or is within a specific ip range. access actions which access action is taken depends on which conditions are met. it is possible that multiple, possibly conflicting access actions are confirmed for one object. if this is the case, the order of the access actions in the configuration determines which access action is taken. for example, when the condition “user is administrator” is met and there is an access action associated with this condition that allows access to the object, access is allowed. subsequent access actions are ignored, because access was already allowed. this applies to each access action met for each object. for example let’s assume the following configuration: [access:user_is_administrator] deny_viewing = false [access:internal] deny_viewing = false [access:external] deny_viewing = true for an administrator from inside the ip range, the following applies: access condition access action result user is administrator allow access condition met, action performed: access is allowed intern allow access action is ignored, access was already allowed extern deny access action is ignored, access was already allowed but for an anonymous user from inside the ip range, the following applies: access condition access action result user is administrator allow access condition not met, action not performed internal allow access condition met, action performed: access is allowed external deny access action is ignored, access was already allowed and for an anonymous user from outside the ip range, the following applies: access condition access action result user is administrator allow access condition not met, action not performed internal allow access condition not met, action not performed external deny access condition met, action performed: access is denied the following access actions are available: deny access to object: user receives an http 403 forbidden page with explanation of access restrictions deny access to datastream: user can view object page, but requests to access datastreams are denied with http 403 forbidden page allow access to datastream: all datastreams are accessible to the user but not available for download. can be specific, such as “deny access to all datastreams, except thumbnails” or “allow access to all datastreams, except archival objects.” provide download of datastream: download menu is visible to the user for all datastreams we’ve provided an example of one possible configuration for the islandora condition access rights module: configuration file ; --------------; metadata values ; -------------- [metadata:restriction_on_access] datastream = mods xpath = "/mods:mods/mods:accesscondition[@type='restriction on access']" namespace[] = "mods http://www.loc.gov/mods/v3" [metadata:use_and_reproduction] datastream = mods xpath = "/mods:mods/mods:accesscondition[@type='use and reproduction']" namespace[] = "mods http://www.loc.gov/mods/v3" [metadata:embargo_date] datastream = mods xpath = "/mods:mods/mods:accesscondition[@type='embargo date']" namespace[] = "mods http://www.loc.gov/mods/v3" ; ---------; conditions ; --------- [condition:manager] user_role[] = "administrator" user_role[] = "editor" [condition:intern] ip_range[] = "192.0.2.0-192.0.2.24" ip_range[] = "198.51.100.0-198.51.100.24" [condition:extern] condition[] = intern operator = negate [condition:fullaccess] metadata = restriction_on_access comparator = "lowercaselettermatch" value = "full access" [condition:withinlibrary] metadata = restriction_on_access comparator = "lowercaselettermatch" value = "access within the library premises" [condition:embargo] metadata = embargo_date comparator = "beforedateiso8601" value = "today" [condition:cczero1.0] metadata = use_and_reproduction comparator = "equals" value = "https://creativecommons.org/publicdomain/zero/1.0/" [condition:rsinc1.0] metadata = use_and_reproduction comparator = "equals" value = "http://rightsstatements.org/vocab/inc/1.0/" [condition:intern_withinlibrary] condition[] = intern condition[] = withinlibrary operator = and [condition:extern_withinlibrary] condition[] = extern condition[] = withinlibrary operator = and ; -------------; access actions ; ------------- [access:manager] deny_viewing = false deny_access_to_dsid[] = "none" allow_access_to_dsid[] = "all" provide_download_of_dsid[] = "all" access_text = "manager access." [access:embargo] deny_viewing = true deny_access_to_dsid[] = "all" [access:fullaccess] deny_access_to_dsid[] = "none" allow_access_to_dsid[] = "all" provide_download_of_dsid[] = "all" access_text = "full access." [access:intern_withinlibrary] deny_access_to_dsid[] = "obj" allow_access_to_dsid[] = "none" provide_download_of_dsid[] = "none" access_text = "access within the library premises." [access:extern_withinlibrary] deny_access_to_dsid[] = "all" allow_access_to_dsid[] = "none" provide_download_of_dsid[] = "none" access_text = "access within the library premises." [access:cczero1.0] access_usetext = "use of this public-domain resource is unrestricted" access_link = "https://creativecommons.org/publicdomain/zero/1.0/" access_image = "https://licensebuttons.net/p/zero/1.0/88x31.png" [access:rsinc1.0] access_usetext = "this resource is protected by copyright" access_link = "http://rightsstatements.org/vocab/inc/1.0/" access_image = "http://rightsstatements.org/files/buttons/inc.dark-white-interior.png" figure 6. example configuration in this configuration three metadata values, 10 conditions, and seven access actions are defined. note that not all conditions have an access action, but every access action shares the same name of the condition that grants it. below are three example scenarios for one object and the resulting access rights when the configuration above is used. example object: <?xml version="1.0"?> <mods xmlns="http://www.loc.gov/mods/v3"> <titleinfo><title>full access</title><subtitle></subtitle></titleinfo> <accesscondition type="restriction on access">access within the library premises.</accesscondition> <accesscondition type="use and reproduction">http://rightsstatements.org/vocab/inc/1.0//</accesscondition> </mods> scenario: user is not in the library user is not logged in object is restricted to in library access this user has access to the object but not to any of its datastreams. user sees “access within library premises.” and “this resource is protected by copyright“ messages and cannot download any datastreams. condition result access action condition:manager not met, user does not have right role not performed, condition not met condition:intern not met, user is not within ip range n/a condition:extern met, because intern was not met n/a condition:fullaccess not met, because value is not in metadata not performed, condition not met condition:withinlibrary met, because value is in metadata n/a condition:embargo not met, because value is not in metadata not performed, condition not met condition:cczero1.0 not met, because value is not in metadata not performed, condition not met condition:rsinc1.0 met, because value is in metadata performed condition:intern_withinlibrary not met, because condition:intern is not met (condition:withinlibrary is met, but this is a conjunctive condition type, so both conditions must be met) not performed, condition not met condition:extern_withinlibrary met, because condition:extern and condition:withinlibrary are met performed scenario: user is in the library user is logged in object is restricted to in library access the logged-in user in the library has access to the object and can view the datastreams (except for the obj datastream). the user sees the “access within library premises.” and “this resource is protected by copyright“ messages and cannot download any datastreams. condition result access action condition:manager not met, user does not have right role not performed, condition not met condition:intern met, user is within ip range of the library n/a condition:extern not met, because intern was met n/a condition:fullaccess not met, because value is not in metadata not performed, condition not met condition:withinlibrary met, because value is in metadata n/a condition:embargo not met, because value is not in metadata not performed, condition not met condition:cczero1.0 not met, because value is not in metadata not performed, condition not met condition:rsinc1.0 met, because value is in metadata performed condition:intern_withinlibrary met, because condition:intern and condition:withinlibrary are met performed condition:extern_withinlibrary not met, because condition:extern is not met (condition:withinlibrary is met, but this is a conjunctive condition type, so both conditions must be met) not performed, condition not met scenario: user is in the library user is an administrator and logged in object is restricted to in library access the logged-in admin user at the library has access to the object and can view all datastreams. the user sees the “manager access” and “this resource is protected by copyright“ messages, but can download all datastreams. condition result access action condition:manager met, user does have administrator role performed, condition met condition:intern met, user is within ip range of the library n/a condition:extern not met, because intern was met n/a condition:fullaccess not met, because value is not in metadata not performed, condition not met condition:withinlibrary met, because value is in metadata n/a condition:embargo not met, because value is not in metadata not performed, condition not met condition:cczero1.0 not met, because value is not in metadata not performed, condition not met condition:rsinc1.0 met, because value is in metadata performed, condition met. this access action has other actions then access:manager, so these will be performed. condition:intern_withinlibrary met, because condition:intern and condition:withinlibrary are met not performed, although the condition is met, the access actions from access:manager are already performed earlier and overrule these all access actions condition:extern_withinlibrary not met, because condition:extern is not met (condition:withinlibrary is met, but this is a conjunctive condition type, so both conditions must be met) not performed, condition not met more example scenarios are available for further explanation. detail tools block the details tool drupal block, included in the module, makes the download options visible for users. a drupal block defines part of a web page. in drupal a page consists of multiple blocks, each in its own region of the page. the access rights of the current object and the available datastreams determine the downloadable datastreams. which datastreams are made available depends on the “provide download of datastream” access action. some datastreams, including the relational datastreams (these define the internal and external relations of the object) and the policy datastreams (used for xacml access rights) are never available for download. caches determining the access rights of an object can be time consuming. to prevent the configuration of the module from being read with each request, the entire configuration is sent as an object. this is cached for at least a day. the metadata values that determine the access rights are cached per object as well. if the metadata of an object is changed, the metadata values cache of that object is flushed. this is done by comparing the last modification dates of the metadata with those in cache. this avoids that the cache will persist multiple hours or days, depending on the configuration of islandora. both caches can be flushed in the usual (drupal) manner. caching is also done on page level, for the calculated access actions. this is only done during the load of a single page, but is needed because the access actions are used multiple times during a page load. calculating them once and using them many times during a page load increases performance. however, they cannot be cached for longer than a single page load, because the access actions are dependent on the conditions of the page load: ip of the requester, the user role and/or the time. islandora versus fedora access islandora is an open source digital repository system based on fedora commons, drupal and a host of additional applications. both islandora and fedora commons have their own systems to allow or restrict access to objects or datastreams. islandora conditional access rights works only on the islandora side and can be used alongside other access rights management systems built into islandora and/or fedora commons. fedora commons works with the extensible access control markup language (xacml) to restrict access privileges. xacml defines a “declarative fine-grained, attribute-based access control policy language, an architecture, and a processing model describing how to evaluate access requests according to the rules defined in policies”. xacml is partly implemented in islandora, so it is possible to manage the rights on a higher level and there is no need to make a xacml document yourself. an earlier version of islandora conditional access rights included support for xacml, but the current implementation of islandora xacml didn’t work with all of our requirements , such as ip range checking, time-based (embargo) and inherited (pages of a book). for this reason we decided not to use xacml support for the time being. the downside of using only islandora’s own access control mechanism instead of xacml is that access rights are only controlled on the islandora level. this means that if direct access to fedora commons is allowed, the access rights are not maintained there. because at leiden university libraries all access is via islandora, this doesn’t cause any problems. conclusion and further development with the islandora conditional access rights module we have created a fine-grained system including all levels of access rights necessary for end users; from access on one terminal within the library to full access everywhere and anytime. this makes it possible to make all digitized items and collections available online. the main advantage to this system is that all metadata are maintained in our main cataloguing system, which also makes it possible for curators and other colleagues to adjust the metadata themselves if needed. it is the library’s policy to manage all data within the same environment. for this reason, the next project will be the migration of our scholarly repository from dspace to islandora as well. further development of the access rights module will then be needed, e.g. to provide access to collections with embargo restrictions. about the authors saskia van bergen works as a project manager for leiden university libraries and is currently responsible for the implementation of islandora and the migration of the digital collections. lucas van schaik works as a developer/consultant for leiden university libraries and is the lead developer for the islandora infrastructure of the library. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – conference report: code4libcon 2008 mission editorial committee process and structure code4lib issue 2, 2008-03-24 conference report: code4libcon 2008 this article contains three reports relating to code4libcon 2008 — the third annual code4lib conference in portland, oregon (february 25-28, 2008). the first describes the birth of the conference and introduces the 2008 osu / code4lib scholarship recipients. the latter two parts, written by the scholarship recipients, outline the things they saw and learned at the conference. by carol bean, ranti junus, and deborah mouw (if you have a conference report in the form of a blog posting, web page, etc., then please consider linking it from the comments of this article. –ed.) report from code4libcon 2008 by carol bean in fall of 2005, the possibility of having a code4lib conference was raised on the code4lib mailing list. this had never been done before, but many on the list were inspired by the access hackfest conferences and pushed forward with organizing the first code4lib conference, held a few months later in february 2006. there were 85 attendees and 15 speakers. in 2007, the registrations increased to 140, with a waiting list, and a pre-conference workshop was added. for the 2008 conference, there were 220 attendees (and a waiting list again), with 22 presentations, 3 pre-conference workshops, and an all-day koha camp. in the true spirit of code4lib, the 2008 conference also included an ad-hoc un-pre-conference for early arrivals who were not attending one of the pre-conference programs. while the conference has grown, the basic format has not. the schedule itself is one-track, with times left open for breakout sessions and for five minute “lightning talks” by anyone at the conference on any topic at all. the conference planning has been through a separate mailing list, code4libcon, which anyone can join. this list is where proposals for hosting the conference and proposals for presentations are submitted and where discussions take place about the conference itself. issues are decided by consensus within the mailing list, but proposals are voted on by everyone registered at the code4lib.org site. in 2007, oregon state university libraries sponsored two scholarships to the conference: the 2007 code4lib conference scholarship for women and the 2007 code4lib conference minority scholarship. the osu libraries sponsored two scholarships again this year: the code4lib diversity scholarship, to be awarded to a member of a principal minority group, and the code4lib gender diversity scholarship, to be awarded to a woman. this year’s recipient of the diversity scholarship was ranti junus, the systems librarian for web services and electronic resources at michigan state university libraries. deborah mouw, the electronic resources specialist at western michigan university libraries, was the recipient of the gender diversity scholarship. while there are many blog postings from and about this year’s conference, as well as online versions of the presentations and lightning talks, the code4lib journal asked the two scholarship recipients to submit a summary of their experiences at the conference. these are of particular interest since it was the first code4lib conference for each of them. here are their reports. code4lib conference report by ranti junus un-conference the conference began with an “un-conference” – an open-ended discussion of issues, raised by the attendees themselves. this event was held for the benefit of attendees who came early but were not necessarily participating in the pre-conferences. people could come and go as they wished; there was no need to stay for the whole time. many of the attendees shared information about a project they are working on, a tool or solution that they use, or an application or project that might be useful for their library. a topic could continue into lengthy discussion about technical issues or other ideas that are related to it. one laptop per child computer during the unconference (photo by dan chudnov) this un-conference event, in a way, was quite similar to the lightning talks sessions, except people don’t have to register first and we can have discussions about the topics. the smaller group size at the un-conference also made the conversations and discussions easier to manage. during this session, i learned about a couple of projects that i’d like to try my hands on: oai-pmh in xquery (an open source project from princeton university library) and the drupal based miami university library (oh) experimental catalog. a list of the topics discussed (including related websites) is publicly available on google docs.by coincidence, my conference roommate and i both own the xo laptop from the one laptop per child project and chumby, an open-source, wireless-based gadget. we decided to bring them both with us to the conference and i had a chance to show them off during the un-conference. preconferences the code4lib preconferences included: evergreen – an open source library automated system developed by the georgia public library service and its pines (public information network for electronic services) program zotero – a browser tool for note management developed at george mason university libraryfind – an open source meta search developed by the oregon state university libraries i attended the libraryfind preconference because this is an application that i’d like to work on back at my library to serve our electronic resources collections. with the number of digital collections and electronic resources we have, including course and subject guides, we need a tool like this to help get our search results with proper categorization in one user interface. keynotes the code4lib program committee probably didn’t do this intentionally, but i found the sequence of keynote speakers complemented one another. we started with brewster kahle from the internet archive discussing collections and content, followed by karen coyle on day 2 with collections’ metadata and its issues, and finally, john udell from the microsoft on day 3 speaking on library roles and the need for library expertise in helping curate the data being published on the web.brewster kahle from the internet archive discussed the internet archive background, its collection development activities, and how it has grown to include not only web pages, but also moving images and audio files. he also discussed their new initiative, the open library, and its operation. i am interested in the mission of this initiative and started writing a proposal to our library administration to share our special collections records with the open library. we’ll see.karen coyle discussed the rda in rdf: its background, status, issues, and what’s next. this is something that i still need to learn more about. jon udell discussed how libraries could also be the center of information production as well as the center of information consumption. some attendees at code4libcon 2008 (photo by ray schwartz) scheduled sessions one thing that strikes me about code4lib is the way the sessions are organized. unlike the typical conferences i’ve attended, code4lib sessions are presented in one continuous thread instead of dividing into several concurrent sessions on different tracks. this model allows everybody to hear the same sessions without missing any of them. what makes this method unique is that the community voted for the topics they’d like to hear a few weeks before the conference opened. a complete conference schedule is available at the code4lib web site. breakout sessions breakout sessions were done on day 1 and day 2, and attendees could pick and choose which topic they would like to participate in. day 1 on the first day, i participated in the linked data session. the first part of the discussion talked about the definition of linked data (wikipedia: “a term used to describe a recommended best practice for exposing, sharing, and connecting pieces of data on the semantic web.”) while i am familiar with the concept of exposing, sharing and connecting pieces of data, this is the first time i heard the term “linked data”.the main discussion then centered around the world digital library, a library of congress project that involves many rare, high-value, high-quality records from several national libraries around the world. a question arose about whether it’s worthwhile to open up the data so others could use it by writing their own applications, add tagging, and publish them. there are some issues that make this quite challenging: translation, controlled vocabularies, subject headings, copyright and licensing, reference metadata, uri of the records, and what they can make public. day 2 i went to the drupal breakout session on day 2. there were quite a number of participants in this session. we talked about libraries that implemented drupal for their website, as well as drupal-related projects, useful modules, and themes. a lengthy discussion focused on training for content providers. besides in-house training, other suggestions included utilizing wikis and subscribing to the drupal4lib mailing list. one attendee also suggested training content providers on how to think like the “drupal model.” basically, content providers would need to know how drupal organizes the content in order to understand how the content would be presented on the website.users management (authentication and authorization) was also discussed. drupal is capable of creating groups of users, where each module can be assigned to certain users. this also requires deciding whether or not users can access/edit static and/or dynamic content. for authorization, it is preferable to make it as simple as possible. resources mentioned during this breakout included: drupal4lib – a mailing list drupalib – “a place for library drupallers to hang out” groups.drupal – a library-specific drupal group drupal modules such as cck, imce (image/file uploaders), fck editor (a wysiwyg editor), interwiki (to create links to wiki websites), taxonomy, views (to control the presentation of the content), and shibboleth (to facilitate single sign-on; it’s not compatible yet with drupal v.6 as of the meeting time) i also managed to participate in the future of code4lib session for the last 15 minutes of the meeting. so far, the general consensus from the participants is to still limit the number of attendees and keep the current conference format. lightning talks the lightning talks sessions were one of my favorites. each of the 5-minutes talks gave me just enough information to encouraged me to explore them further. now my head is brimming with ideas. the ones that caught my attention attention the most were: jester, a javascript api for restapis that uses rails faceted backup opac jangle, a restful web framework for building web services for library systems bringing sheet music to life, by andrew bullen (illinois state library), where he used music ocr software to digitize sheet music and create audio files from them. andrew bullen (photo by rob styles) what i got from the conference it has been my experience that at most library conferences web developers are a rather small minority. there were few opportunities at the other library conferences to discuss technical aspects of projects or development topics. the code4lib conference provides a venue where developers can swap experiences. this gave me exactly what i needed: more technical information and discussion of information management projects. conference report by deborah mouw overview brewster kahle, the founder of the internet archive, kicked off the 2008 code4lib conference. kahle’s charge was for all those gathered at the conference to work together to build a digital library together now. this keynote set the tone for the rest of the conference and the presentations that followed did not disappoint.the next 3 days were full of fascinating presentations on topics ranging from the latest database technology to the best way to manage an open source project. excitement was infused in the highly flexible and participatory schedule, featuring many twenty minute scheduled presentations as well as the now legendary “lightning talks.” the long days left conference attendees tired but excited and often discussions lasted late into the night at various restaurants and brew-pubs around downtown portland.pre-conferences for code4lib started on february 25th and the conference proper ran from february 26 through february 28. the pre-conferences included workshops on evergreen, libraryfind and zotero, as well as koha camp and an “un-pre-conference” where all those who didn’t get a chance to sign up for a pre-conference workshop could use the time to meet and discuss projects and issues in an informal manner. i was able to attend the “un-pre-conference” for a short time and get a glimpse of such projects as dbwiz, bibliocommons and the open library project.one of the major themes of the conference this year was how do we leverage what we already have (ils’s, marc data) to create something new and useful for our users. the tension between limited resources and increasing expectations from users has forced library technologists to think creatively of how to make the most out of their own technology as well as how to spread development costs as far as possible. one presentation where this tension was discussed was the presentation by emily lynema and terry reese entitled “dlf ils discovery interface task force api recommendation.” the digital library federation created a task force to investigate creating standards for ils’s to make their data available via an api. this topic was further discussed in a break-out session where discussion centered around the incredible need for such an api and how to get vendors to buy in to this recommendation. emily lynema (photo by rob styles) this year’s conference was also filled with presentations on newly developed systems for various library related tasks. some of these projects included two mods editors using xforms developed by winona salesky at the university of vermont and michael park at brown university, a content management system to assist with the creation of course-specific library guides developed at oregon state university, a metadata registry being developed at cornell university and a tool to publish digital collections and exhibitions (omeka) developed at the center for history and new media at george mason university. there were also many presentations on the progress of older projects, including the wayback machine and zotero.at the end of the third day of the conference, all those who spoke either in a scheduled presentation or at a lightning talk were asked to raise their hands. the participatory nature of the code4lib community and conference was evident when more than half of those in the room had a hand raised. i hope that this participatory nature continues throughout the future of code4lib and that those new to the community (such as myself) can become active members in this exciting group of people. keynotes in the first day’s keynote address, brewster kahle unveiled his “big idea” to create a digital library free to all. kahle outlined four steps to creating this digital library: scanning, selecting, building the catalog and, finally, allowing access to the books. kahle also talked about some of the initiatives started at the internet archive, including the internet archive bookmobile, a moveable machine that downloads, prints and binds a book on demand. brewster kahle (photo by rob styles) the second day’s keynote was done by karen coyle, a library consultant, on rda in rdf and the work of the dublin core metadata initiative to help make rda (resource description and access) useful and usable. karen pointed out some of the major flaws of rda and charged the room of conference attendees to get involved in the discussions of the dcmi/rda task group, as the technical voice was one that was needed in that conversation. karen coyle (photo by rob styles) the third and final day began with a keynote by jon udell, an evangelist at microsoft, who believes that librarians of the future would become “guild navigators for infospace.” this means that librarians will be critical in helping users create, organize and find information on the web. he had a charge especially to those in public libraries to move beyond helping patrons use computers and the internet to find information, to also help them upload, organize, tag and otherwise make accessible their own information creations. jon udell (photo by ray schwartz) about the authors carol bean is a member of the code4lib editorial committee. ranti junus is the systems librarian for web services and electronic resources at michigan state university libraries. she is the 2008 recipient of the osu / code4lib diversity scholarship. deborah mouw, the electronic resources specialist at western michigan university libraries, is the 2008 recipient of the osu / code4lib gender diversity scholarship. tags: code4libcon 2008 subscribe to comments: for this article | for all articles one response to "conference report: code4libcon 2008" please leave a response below: taking diversity to the next level – library hat, 2017-12-18 […] mouw, “conference report: code4libcon 2008,” the code4lib journal, no. 2 (march 24, 2008), http://journal.code4lib.org/articles/72. […] leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – checking the identity of entities by machine algorithms: the next step to the hungarian national namespace mission editorial committee process and structure code4lib issue 33, 2016-07-19 checking the identity of entities by machine algorithms: the next step to the hungarian national namespace the redundancy of entities coming from different sources caused problems during the building of the personal name authorities for the petőfi museum of literature. it was a top priority to cleanse and unite classificatory records which have different data content but pertain to the same person without losing any data. as a first step in 2013, we found identities in approximately 80,000 name records so we merged the data content of these records. in the second phase a much more complicated algorithm had to be applied to show these identities. we cleansed the database by uniting approximately 36,000 records. the workflow for automatic detection of authority data tries to follow human intelligence. the database scripts normalize and examine about 20 kinds of data elements according to information about dates, localities, occupation and name variations. the result of creating pairs from the database authority records, as potential redundant elements, was a graph, which was condensed to a tree, by human efforts of the curators of the museum. with this, the limit of technological identification was reached. for further data cleansing human intelligence that can be assisted by computerized regular monitoring is needed, based upon the developed algorithm. as a result, the service containing about 620,000 authority name records will be an indispensable foundation to the establishment of the national name authorities. this article shows the work process of unification. by zsolt bánki (petőfi literary museum), tibor mészáros (petőfi literary museum), márton németh (monguz ltd.), andrás simon (monguz ltd.) background the main topic of this article focuses on the semantic web related namespace building activities of the petőfi literary museum. however, first we would like to give context to these efforts with a brief introduction to the first major semantic web related project in hungary, at the national széchényi library. the national széchényi library (nszl) published its entire opac and digital library, and the corresponding authority data, as linked open data in 2010 as one of the first public collections in europe. the tools used were rdfdc (rdf dublin core) – marc xml to rdf/xml conversion with xslt for opac bibliographic data, foaf for names, and skos for subject terms and geographic names. nszl uses cooluris. every resource has both rdf and html representations. the rdfdc, foaf and skos statements are linked together. the name authority dataset is matched with the dbpedia (semantic version of wikipedia) name files. nszl also supports html link auto-discovery. all of the available linked data resources (names, subject authority, catalogue records) can be searched and retrieved to external resources in the semantic web via a sparql endpoint with specific browser tools (that are useful in machine-to machine communication). (horváth, 2011b) there was no specific project related to the semantic field within the library, although small developments pointed in the same direction. three members of the directorate of informatics developed the semantic web based services when time permitted. in 2009 they realized that they had almost everything to be able to publish linked data on the semantic web: they converted the library thesaurus to skos format; via the libriurl tools the opac records became accessible via url (here is an example: http://link.oszk.hu/libriurl.php?ln=en&db=any&sry=an&sre=2616972); and the url-based search in the nszl integrated system became available via the sru protocol with the yaz proxy tool. in addition, they could use the experiences of the libris semantic web implementation project. the nszl main goals were: (1) library datasets need to be open (to get data out), need to be linkable, and also need to provide links, and (2) datasets must be part of the network, cannot be an end in themselves, and the system must allow hackability. the major advantages of the rdf-based semantic model are the following: rdf clients can look up every uri in the rdf graph over the web to retrieve additional information information from different sources merges naturally rdf links between data from different sources can be set information expressed in different schema can be represented in a single model (horváth, 2011a) the project of nszl, together with the efforts of the petőfi literary museum, has become the major factor in planning the hungarian national namespace. in the following we will offer an overview of the related activities from the petőfi literary museum. we will also offer a brief introduction into the concept of the hungarian national namespace. a brief introduction to petőfi literary museum petőfi literary museum is the national museum for literary heritage. it is one of the most respected museums in hungary. the history of the institute goes back to the beginning of the 20th century; it collects the tangible and intangible heritage of hungarian writers. the museum is open to new trends and methods in the contemporary cultural heritage field and accordingly, it technology has been present in a wide range of our activities since the 1980’s. we have been working on the development of our collection databases for decades, and now service the entire research environment. our integrated collection management system meets all contemporary standard international requirements and we use it for describing our collection items and to develop significant namespaces. the most important part of this system is the personal name database, which contains approximately 600,000 authority records: it is the largest such dataset in hungary. this namespace contains controlled and certified data stemming mostly from printed resources, and it is published through the opac of the museum (opac.pim.hu). the personal name space serves as the primary source for person name identification in the hungarian library and museum system. our namespace includes different database genres (biography, bibliography, and genealogy). we developed several thematic databases for researchers which are unique in hungary. our colleagues working on the project seek to use only certified resources and have registered every difference among them, so the data are extremely useful for scholarly research and serve as a reliable reference. most of the databases based on lexicographical authority records are biographical, which presents information concerning the life and career of the person in question. for this project’s method, a high percentage of the namespace does not have bibliographic records in the museum cms system, yet they are valuable as a basis for processing collection items. this also means that the proportion of authority records connected to bibliographic items is constantly growing. (kómár, lengyel, & simon, 2008) the databases which we built in the namespace hungarian biographic index this is the largest biographical database of our museum, one of the most important online sources for factographic information in hungary used by librarians and researchers making bibliographies and bio-bibliographies. the records are not lexicon entries but consist of a series of data which makes identification of a person possible: date and place of birth, death, occupation, name variants. an important function of the hungarian biographic index is that it compares and corrects the different data in its printed sources (lexica, documents, etc.) the index defines certified data and, in case of birth and death datasets, it records the different data of the sources used. lexicon of hungarian immigrant authors the term immigrant refers to authors and writers who left hungary for a period of time, living and working abroad. the entries refer to this period of their career. datasets include information on the higher education of the person in question, the date of departure from hungary, the name of the destination country/countries, the states of the scholarly and/or public career, prizes and awards received, the institutes/societies in which s/he held membership, and the publication sources and the alias names used in them. awards and awarded personae the only hungarian database of phaleristics consists of the names of awards and prizes of hungarian relevance, made after 1945, as well as the names of the holders of these. the data collection is not confined to cultural awards but includes awards of other domains as well. the entries also contain the awarding entity, the place and date of the establishment of the award, and its history. our person name authority records do not solely consist of the requested normalized data elements, but further metadata such as occupation, and exact date and place of birth and death. our rich datasets also include references to the data-source and, where applicable, the list of awarded prizes. we also indicate the relatives of the person in question. this data is available to be published on the semantic web. to show the complexity of our namespace, we created a dataset of 30,000 authority personal name records which have 300,000 bibliographic record connections in the library catalogue of pim. model of pim authority records of person names our records follow this architecture: 001 recordid 005 20160309083045.0 090 __ a database 100 1_ a personal name heading, surname j forename b regal numeration g title of nobility c noble rank/titles and other words associated with a name d birth date/ death date m name suffix 400 __ a name variant(s) d name variant(s) date o name variant explanation(pseudonym) t note 500 1_ a relationship: personal name heading, surname j forename b regnal number g title of nobility c noble rank/titles and other words associated with a name d birth date/ death date m name suffix o relationship (mother, daughter) 667 __ a occupation 668 __ b occupation note 678 __ a biography note 856 __ f picture u url 900 __ a exact birth of date 901 __ a date of baptism 902 __ a place of birth 9 place of birth note 904 __ a county of birth 905 __ a exact death of date 906 __ a place of death 9 place of death note 908 __ a country of death 9 country of death note 909 __ a age of death 910 __ a day of the funeral 911 __ a cementery 912 __ a parcel 921 __ a home 922 __ a gender 923 __ a religion 924 __ a walk of life date (marriage, degree, divorce) 925 __ a university education (years) 926 __ a faculty 927 __ a academic degree 928 __ a membership 930 __ a publication 931 __ a translation 940 __ a data source 942 __ a birth data source 943 __ a death data source 960 __ 4 prize/title type a prize/title d prize/title year collection management system in the petőfi literary museum the qulto integrated collection management system was originally written for library automation, especially for medium size special libraries. research institutes, museums, and high schools have bought it for their libraries. museums have begun to use it also as an automated inventory system. so its role has been to be not only a computerized catalogue system but to record and contain factographic information and additional data as well. libraries of hungarian speaking people and in the areas of central europe where the hungarian language is spoken and used (it is a bit bigger than the territory of the hungarian republic) collect mostly hungarian books, so hungarian librarians are used to preparing form entries themselves, especially in special libraries. therefore local cataloguing and indexing still has a significant role in hungarian libraries nowadays. hungarian library automation systems are completely compatible with systems of other countries, including the capacity for data exchange via z39.50 or oai clients. (bánki, lengyel, & tóth, 2007) the qulto icms is based on a relational database that uses marc as its internal format, and works with plenty of authority and quasi authority records, such as personal, corporate, and geographical names. however, copyright holders, suppliers, classification numbers, manufacturers, and prizes can also be authority data types. a couple of the fields are supported by dictionaries, so the main philosophy of qulto is to help the librarian with additional information to prepare the most exact records and lessen mistakes as much as possible. the marc structure with added quasi marc fields is very segmented in the database and helps the user to tag the information. from these controlled data units all kinds of printed or electronic outputs or exchange formats can be easily prepared. data merge theory and practice name authority: first step, basic concept of parsing migrating the bibliographic and authority data from the former separately built databases to qulto icms, we realized that several authority records were duplicates with different metadata content. the 620,000 name authority units held fewer records of living persons. so the “basic” personal record almost always duplicates data from the database of graves of hungarian authors or from the database of the obituaries and death-notices as well. so in 2008, the first step of the authority name merge began. the primary record the basic name authority record is called “primary record”, which is one of the database units among the records describing the same person (human being). doing the authority data merge, the name and date of birth (100$a, 100$j, 100$d) were considered the basic data fields, and the other authority or bibliographic information in the 6xx or 9xx fields were considered the additional data. the records coming from different databases had different contents and all of them should have been inserted into the primary records, so all the information about the same persons and all the bibliographic records linked to them had to be merged into one fully enriched authority record, without any data loss. the merge of the first group of the records had been planned to be done automatically, without any manual intervention except for preparing the algorithm with which the identity of the authority records would be examined. therefore very restricted conditions were set for the automatic identification. we wanted to allow only two records to be merged, if their identity was without any doubt. priority of databases, defining primary records according to the richness of metadata, a priority order of databases was formed in order to choose the primary record from the redundant pairs. the primary record had already been enriched during the first phase of the authority process. records had merged with metadata elements coming from the non-primary items. after needless records were deleted, the redundancy of the database decreased, and it became more exact and consistent, making it more suitable to fulfill the namespace function of including only one record for one person. by the first phase, 80,000 name authority entities were found redundant and had been deleted after the data merge. (lill, 2012) name authority: second step after the first step it was clear that going forward we needed a new and more sophisticated parsing algorithm, which would have to be supported by human knowledge. following the data merging process new pairs had been created and in some groups of records anomalies were detected. our aim was to automatically fix as many pairs as possible. by fixing the rest of the pairs as duplicates manually, lists and reports were created to help the librarians. through this step those records were also reviewed that had not contained enough information for automatic identification. evaluation of the parsing for automatic fixing, an evaluation method had become necessary to rank the pairs. concerning metadata richness and relevancy, pairs were evaluated with scores. two authority records became a pair if their names were identical and their birth and death data weren’t different. using a strict score system, about 11,000 pairs returned as identical. a list was made from the pairs that had lower scores. the criteria for scoring the pairs were the following: year of birth is filled and identical, 1 point year of death is filled and identical, 1 point both year of birth and death is filled and identical, additional 2 points date of birth is identical, 2 points location (village, town, city) of birth is identical, 2 points county of birth is identical, 1 point date of death is identical, 2 points location (village, town, city) of death is identical, 2 points date of birth is not identical, -2 points location (village, town, city) of birth is not identical, -2 points county of birth is not identical, -1 point date of death is not identical, -2 points location (village, town, city) of death is not identical, -2 points if one of the members of the pairs was the member of another pair where anomaly (minus point) was, it was marked as a suspicious item. all the 900 data pairs fixed as suspicious received, -2 more points if some birth or death data were missing, the pair received more negative points if christian name was cut into more elements, the pair received more negative points 36,000 pairs were examined and 33,000 of them were fixed as identifying the same person. common names (tóth, kovács, similar to smith or miller) existing in the database several times caused problems, especially if they were written in different ways (e.g. kovács, kováts, kovách). there was a problem also with the christian and family names having more than one unit, or having other name elements, titles, etc. operational problems there were pairs which contained more than one personal link to other records as husband, wife, father or mother, or had more than one birth or death date. these cases were checked manually. as a part of the data merging process, additional data checking had become necessary and resulted in a more standardised form of the dataset. geographical names existed in different forms or languages, or were written in different ways. the occupations also had to be unified, and subsequently a couple of pairs had to be fixed as identical. some algorithms were defined in order to run automatic data merging processes to avoid future redundancy. using experiences from the past, records could be listed where the person name was different but the other metadata elements were the same. if, from the four metadata indexes representing the year and place of death and birth, at least three are the same, the pair of authority records could be set as identical. sometimes authority records having too few metadata elements could be identified by the bibliographic records linked to them. the aim was to prepare full name authority records, including all the necessary information (school, prizes, family connections, death-reports, etc.), concerning a person in a database record. the aim of the parsing algorithm was to fix duplicated authority records concerning the same person. however, these records could take parts of other pairs as well. if we set a record as primary, it could take part in another pair as a non-primary element as well. we had to take care to avoid creating networks of pairs that resulted in a loop, where each of them appears once as primary and once as non-primary item. this had been avoided by the way we precisely ranked the databases, and set the records as primary using a strict and uniform algorithm. in cases where the two items of a pair came from different databases the one from a higher-ranked database got the primary rank. in other cases the primary status was chosen according to the record id. the richness of metadata wasn’t checked in this step, because all metadata elements had been merged. if a primary record in a pair had parsed in another case as a non-primary one, its primary pair had been placed into the pair in which it existed as primary, so every pair was completed with a super primary unit. super primary members of groups could be only those records which had never been labelled as a non-primary item. another critical element of the parsing method was the problem of “forks”: networks of pairs which have more than one super primary element. these pairs were always checked manually. this problem could occur in cases of missing metadata elements. in most cases the date of death was missing. through checking the result of the data merge we received useful information about the database as a whole. in some cases, we could correct the database records which hadn’t been part of the parsing process. after the data merge all of the work was checked by randomly selecting items. technical elements of the steps of parsing the first step was the selection of the authority records from the database to be part the merging process. the valid record had to be in non-deleted status, and had to contain some data elements, for example, christian name and year of birth. the selected names were normalized, so that the variations of some letters, such as “y” and “i” or “ts” and “cs”, were unified, and all the punctuation was eliminated. from christian names having more than one unit, control strings were created containing all the combinations of the family name and other name segments for the christian name. for example: from this string “mezőhegyesi szilveszter aladár”, three results were generated: mezőhegyesi szilveszter aladár mezőhegyesi szilveszter mezőhegyesi aladár from the index data of family and christian names, year of birth and death, a key string was generated on which parsing of authority records was based. in cases of control subfields (exact date of birth and death, location of birth and death, occupation), additional name elements were added to the key strings. by doing this, the fully segmented qulto marc based relational database records were built into strings for parsing and control. through a detailed analysis of the pairs, we found anomalies (differences according to names and locations). we marked the pairs with numbers, to help the librarian to determine the identity of the primary and non-primary record. the correct items of information increased, and all the anomalies decreased during this process. through an analysis of this list we could decide which pair joined to two identical records, defining the same person in this way. we could determine also if the members of the pairs were identical without manual intervention. merging was done in several steps. at first those pairs were merged which were without any doubt the same. secondly, from the dubious pairs a stack was chosen to merge without human intervention. by the third step, the rest of the list was analysed one by one by the staff of the museum. merging process step by step database backup creating the key string of the record for the comparison normalizing creating the pairs form the data heap creating graphs as a network of pairs (one record could be a member of several pairs) erasing loops and forks creating the pairs that really could be identical erase the duplicates of pairs (these duplicates were created through the splitting of christian names; some records could be involved in the parsing process several times) adding super primary records to the pairs control of the pairs, removing those with a deleted element (the whole operation had been made in a dynamically changing catalogue environment, and for this reason it needed fairly significant time) revision of parsing by manually checking hundreds of pairs preparing the list of checked pairs through the data merge as a process, the first step is the creation of a view dataset for data-checking on the icms interface. the next step is the real data merge, by moving the data subfields from the non-primary to the primary record. if the non-primary record had a new data field, the new content was copied into the primary record. all the other data elements of the non-primary record remained on the view dataset. deleting the duplicates of data subfields and authority records further plans with the pim namespacethe conception of the hungarian national namespace the hungarian national digital archive, national széchényi library, hungarian national archive, and petőfi literary museum started a broad collaboration project in 2012 with the national namespace concept. the idea of the concept is to focus on the experiences of the process of building large collaborative namespaces as professional service frameworks created by international collaboration (getty, viaf, iconclass). it has become clear for the members of the project that the focus of future professional activities will be on the usage of namespaces in the semantic web environment. the target groups are both the institutions with their professional needs and the users of the public collections in the broadest sense. professionals from different institutions are agreed that the institutional namespaces and other high-quality semantic-web compatible metadata collections will form the basis of the common hungarian namespace. following this conception, cleaning the dataset of names, including the identification and data clearance of entities, has been a major contribution by the petőfi literary museum towards the creation of the hungarian national namespace. the creation process of the national namespace is still in progress. however, the concept is already available. the collections from the different institutions’ collections can be fully connected and are interoperable with each other. using joint namespaces seems to be the best way to represent the hungarian cultural heritage as a whole, describing all of the available contexts and connections through the different segments of this broad area. all the cultural databases should be available through a common interface regardless of the traditional institutional barriers among the archive, library and museum sectors. collections and datasets being stored by the individual institutions can be represented in their own local contexts. on the other hand, the public can have access to the digital cultural heritage as a whole. the content of the local collections could be represented through a framework of an integrated system (based on namespaces with common standards on the semantic web). the primary user group of the future national namespace will be the general public. through the first step of the implementation process, however, the collaborating institutions must focus on their own institutional contexts and needs in order to create their own high quality datasets, service workflows, and infrastructure. the second step in the future should focus on the implementation of all the services and functionalities focusing on the general public. another strategic goal of the hungarian national namespace can be a data authority function. the national namespace must serve qualified, controlled, and clear data on request to anyone, whether a public collection and other public institutions, or individuals. conclusion through this article our aim was to offer a brief overview of institutional efforts from the hungarian cultural heritage sphere in the field of semantic namespace building. our primary example was the namespace project of the petőfi literary museum, with a brief description of efforts in the hungarian national library as well. the recently acquired experiences, skills and competencies through local efforts have contributed to making the hungarian national namespace. our main hope is that the project can perhaps be realized to its full extent in the near future. the local institutional projects could be amalgamated into a successful national service model. following its completion, we hope that it will also be recognized by the general public in hungary by representing the hungarian cultural heritage in a complete way that has never been available before. references bánki, z., lengyel, m., & tóth, k. (2007). “ablak” a múzeumokra: a petőfi irodalmi múzeum speciális adattárai a huntékában. networkshop 2007. eger: niif. retrieved 26 may, 2016, from https://nws.niif.hu/ncd2007/docs/aen/095.pdf horváth, á. (2011a). linked data at nszl. retrieved 26 may, 2016, from http://nektar.oszk.hu/w/images/0/04/linkeddataatnszl_06.pdf horváth, á. (2011b). national széchényi library semantic web wiki. retrieved 26 may, 2016, from http://nektar.oszk.hu/wiki/semantic_web kómár, é., lengyel, m., & simon, a. (2008). a petőfi irodalmi múzeum személynév állományának egységesítése és szűrése: hozzájárulás az nda névtér projektjéhez. dunaújváros: niif. retrieved 26 may, 2016, from https://nws.niif.hu/ncd2008/docs/aen/046.htm lill, j. m. (2012). kontrolliertes vokabular. wieso? weshalb? warum? vortragsfolien vom 13. musis-nutzertreffen am 18. juni 2012 in mannheim. retrieved 26 may 2016, from https://swop.bsz-bw.de/frontdoor/index/index/docid/850 about the authors zsolt bánki received a college degree in library science and a degree qualifying as a school teacher in hungarian language and literature from dániel berzsenyi teacher training college, szombathely, hungary. he graduated as a teacher of roman catholic religion at péter pázmány catholic university, budapest, and obtained a master’s level degree in library and information science from loránd eötvös university, budapest. he started his library career in the national library of foreign literature, then worked for the national széchényi library, budapest. his professional interest is focused on systems of museum informatics and professional digital museology standards, theoretical issues on digitization, and planning and implementing national and international databases. he has been the director of informatics in petőfi literary museum, budapest, since 2006. he has been involved in several national and international projects and is a leading member of several hungarian professional organisations. tibor mészáros received a college degree in library science and a degree qualifying as a school teacher in hungarian language and literature from dániel berzsenyi teacher training college, szombathely, hungary. he graduated as a teacher of roman catholic religion at péter pázmány catholic university, budapest, hungary, and has been a librarian at the petőfi literary museum since 1988. he has been the editor of the hungarian biography index since 2010. a primary focus during his career has been on the literary life of the famous hungarian writer, sándor márai. he is an author of several books, editor of bibliographies, curator of exhibitions, and also a professional lecturer related the work of this renowned writer. márton németh graduated from the master’s-level university program in history at the faculty of arts, university of szeged, with high-school teacher-training specialization. he subsequently completed his master’s degree in library and information science at szeged. he also holds a master of social science degree in european studies from aalborg university, denmark, and recently graduated from the international master of social science in digital library learning offered jointly by oslo university college, parma university, and tallinn university. he has worked as an it librarian in the directorate of informatics, national széchényi library, and is now a digital information specialist at monguz ltd. in budapest. he is currently working on his phd at the doctoral school of informatics, university of debrecen, hungary, focusing on the interrelation of public collections and the semantic web. simon andrás graduated from eőtvős loránd tudományegyetem budapest faculty of arts history – librarianship, and did postgraduaate work at the economic university of budapest as postgraduate economist of faculty of marketing. he worked as system librarian in the economic university of budapest, and at the magyar országos kőzős katalógus (hungarian national shared cataloguing system) as system manager. he has been on the mta sztaki itak library automation team, and monguz ltd., working on library and museum automation, taking part in projects, planning, developing, preparing adaptations, educating, selling and supporting content management systems. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – strategies for digital library migration mission editorial committee process and structure code4lib issue 56, 2023-04-21 strategies for digital library migration a migration of the datastore and data model for stanford digital repository’s digital object metadata was recently completed. this paper describes the motivations for this work and some of the strategies used to accomplish the migration. strategies include: adopting a validatable data model, abstracting the datastore behind an api, separating concerns, testing metadata mappings against real digital objects, using reports to understand the data, templating unit tests, performing a rolling migration, and incorporating the migration into ongoing project work. these strategies may be useful to other repository or digital library application migrations. by justin littman, mike giarlo, peter mangiafico, laura wrubel, naomi dushay, aaron collier, arcadia falcone introduction for the past 4 years, a team at stanford university libraries (sul) has been working on and off towards migrating to a new datastore and data model for the stanford digital repository’s (sdr) digital object metadata. [1] the goal of this paper is to describe the motivations for this work and some of the strategies used to accomplish this migration. these strategies may be useful to other repository or digital library application migrations. about the stanford digital repository in support of research, teaching and learning, the stanford digital repository is an ecosystem of applications, systems, and services that house the digital collections of stanford university libraries. collections housed by sdr include: google-scanned books stanford dissertations and theses university archives allen ginsberg papers buckminster fuller papers parker library fugitive u.s. agencies web archive at the time of writing, sdr has almost 5 million digital objects composed of more than 530 million content files. [2] sdr is extremely heterogeneous along several facets, including content types (e.g., books, images, web archives, gis datasets) and file types (e.g., xml, tiff, warc, mp4). repository ecosystem & data flows there are multiple avenues for users to deposit digital objects (metadata and content files) in sdr. these include, but are not limited to: a web based self-deposit interface for single item deposit by researchers (stanford faculty, students, and staff) a web based self-deposit interface for theses and dissertations by stanford graduate and undergraduate students a web based interface for administrative library staff that allows for bulk deposit api based deposit (currently internal library staff use only) customized third-party software optimized for managing digitization of physical materials by stanford university libraries staff sdr interacts with other digital library applications at sul. in particular, it deposits digital objects into the preservation system [3] and publishes digital objects to the access system. sdr also retrieves descriptive metadata from symphony, sul’s integrated library system. [4] the preservation and access systems are not discussed here, though briefly: the preservation system provides redundant onsite and offsite backup, with validity and audit checks. the access system provides web based search, discovery, viewing (e.g., via iiif), and download. throughout this ecosystem, varying levels of discovery, access, and embargo restrictions can be set and are enforced. metadata management store sdr has been in operation since 2006. for at least the last 11 years [5], it relied on fedora 3 as a datastore for its digital object metadata. sdr’s digital object metadata includes description, identification, access / rights, structure, and other administrative metadata. the digital object’s content files are not stored in fedora. while fedora 3 has been the sdr’s “workhorse” over the years, it suffers from a number of critical shortcomings: fedora 3 is no longer supported and was last released in 2015. fedora 3 relies on java 8, which has been “end-of-life” since 2019. the basic units of metadata storage in fedora 3 are xml files stored on disk (“datastreams” in fedora parlance). these datastreams are not validated and schema-less, by design. ruby libraries supporting fedora 3 (activefedora 8.x and rubydora) are no longer under active development, going moribund in 2018, which constrained our ability to update to the latest versions of the programming language (ruby) and web application framework (rails) used throughout sdr. fedora 3 does not support transactions, meaning that problems like network blips can result in digital objects with incomplete or otherwise broken metadata. fedora 3 does not support constraints (e.g., uniqueness), allowing for problematic digital object metadata to be included in the repository. by itself, fedora 3 does not support advanced querying of digital object metadata. in sdr, querying is provided by an instance of the solr full text search service, which must be kept in sync with the underlying digital object metadata datastore. fedora 3 is not designed for load balancing and thus becomes a single point of failure under heavy load, even if other parts of the system are load balanced to handle higher load. we have observed this behavior in sdr where fedora often “falls over” under load. in addition, the sdr application and data ecosystem suffered from critical shortcomings: sdr applications were tightly bound to fedora, interacting directly with fedora to update xml datastreams. many sdr applications took advantage of the flexibility of fedora’s datastreams to store application-specific metadata. without the constraints of a schema / validation, the xml metadata was inconsistent due to: each application directly manipulating xml. users being empowered to manually edit xml datastreams in sdr’s management application as the endorsed way to make certain types of functional changes to objects. abandoned metadata approaches and legacy applications left unremediated and divergent. low prioritization of auditing and quality check tools for entire corpus of sdr xml the totality of these shortcomings motivated the migration to a new metadata management store and a rich, validatable data model for sdr’s digital object metadata. [6] in the rest of this paper, we will describe some of the strategies used to accomplish this migration. to this end, we will focus on the motivation and logic of these strategies, rather than the code or implementation details. strategy #1: adopt a validatable data model one of the crucial lessons from sdr is that the data model for the digital object metadata should be validatable, meaning that an instance of digital object metadata can be checked to ensure that it conforms to the data model. the primary means of making the data model validatable was to specify a schema. in addition to providing validatability, the schema serves as documentation for the data model. we [7] developed a home-grown data model for digital object metadata known as “cocina”. cocina digital object metadata is serialized as json and the schema is represented as an openapi specification. [8] openapi was selected for the schema because: it is widely adopted and has excellent tooling in multiple programming languages. the same openapi specification could be used as a specification for the api applications that “speak” cocina. openapi provides a balance between ease of use and powerful validation. critical to integrating cocina across our application was having a library to provide ruby classes for cocina. thus, cocina digital object metadata is manipulated in code as ruby objects, not as json. the cocina classes are created by a code generator [9] that we authored to transform a subset of openapi into ruby. as changes are made to the cocina model, the openapi specification is modified and new cocina ruby classes are generated. the crux of this approach, however, is that when a new cocina object is instantiated, it is automatically validated against the openapi schema. [10] thus, digital object metadata is validated at the code level and at the api level. and, in both cases, the validation is performed using the same mechanism, viz., the openapi specification. there is a small subset of validations that cannot easily be represented in openapi. for example, validating that the access rights at the object level are consistent with the access rights at the file level. for these, validation is performed in code that is also executed whenever a new cocina object is instantiated. together with the openapi validation, this gives us high confidence in the consistency of cocina digital objects. [11] it is worth noting that there is one part of the cocina data model where this is not strictly true. the cocina data model for descriptive metadata was architected by arcadia falcone (sdr’s metadata specialist) to allow for mapping to multiple metadata formats (e.g., mods and datacite). as such, it was structured to have a great deal more flexibility than other parts of the cocina data model. while the basic structure of the descriptive model is validated via openapi, additional semantic validation is defined elsewhere in the ruby software library via a yaml file. [12] this file enumerates lists of terms (e.g., valid note types) against which the values of selected properties may be checked at cocina object creation time. these lists are more frequently changed than the model as a whole, so storing them separately allows the openapi specification to remain more stable. in addition, the yaml file is used as the basis for documentation aimed at metadata creators who do not need the full model documentation. strategy #2: abstract the datastore behind an api to address the tight binding between applications and the digital object metadata datastore (i.e., fedora 3), the datastore has been abstracted behind an api. this api, known as dor [13] services application (dsa) allows applications to create, retrieve, and update cocina digital objects, as well as a number of other repository functions such as initiating accessioning, managing versions, and recording events. probably the greatest effort in the migration process has been to rewrite or refactor the existing applications to use cocina objects instead of fedora internally and to interact with dsa. this has been an iterative process, as we built out parts of the cocina model and dsa functionality as needed for each sdr application being worked on (see strategy #8). prior to migration, dsa internally performed a real-time two-way mapping between cocina objects and fedora objects [14]. thus, the basic activities of dsa prior to migration were: when an application requested a cocina object from dsa: a fedora object was retrieved from the fedora datastore. the fedora object was mapped to a cocina object. the cocina object was returned to the application. when an application provided a cocina object to be created: a new fedora object was instantiated. the fedora object was updated based on the cocina object. the fedora object was saved to the fedora datastore. when an application provided a cocina object to be updated: an existing fedora object was retrieved from the fedora datastore. the fedora object was updated based on the cocina object. the fedora object was saved to the fedora datastore. given the complexity of the fedora and cocina data models and the substantial heterogeneity of the existing digital objects in sdr, this mapping code was of significant complexity; strategies for managing this are described below. with all of our applications decoupled from fedora, dsa became the sole application interacting directly with fedora; at this point, dsa was itself tightly coupled with fedora. one of the final phases of the migration was to encapsulate all fedora interaction code within dsa to a dedicated service class known as the cocina object store. all other services in dsa manipulated only cocina objects; the mapping between cocina objects and fedora objects happened within the cocina object store; this allowed a rolling migration as detailed in strategy #7. strategy #3: separate concerns one way of reducing complexity in the digital object data model and making the migration more manageable was to promote the separation of concerns, where services with distinct responsibilities were implemented in distinct applications. also part of the separation of concerns was using distinct datastores for individual application specific services, rather than relying on storing metadata centrally in fedora datastreams. one such example is technical metadata. previously, technical metadata was generated by jhove and stored as xml in its own fedora datastream. this approach had a number of shortcomings [15]: technical metadata was tightly bound to fedora. as the syntax of the technical metadata created by jhove changed over time, the metadata grew inconsistent. there was no efficient way to query the technical metadata. this approach was replaced by a dedicated technical metadata service. implemented as a rails application with an api, the technical metadata service utilizes its own relational datastore for storing and querying technical metadata. it also obviated the need to include technical metadata (which itself is very complex and verbose) in the cocina data model. this approach was applied to various other services (e.g., events and workflow) that previously were tightly bound to fedora in a single monolithic application. another benefit of this approach is that if and when these specific services or technologies change, this can be done in isolation as long as the api is maintained. strategy #4: test mappings against real (cached) digital objects as mentioned previously, the two-way mapping between the cocina data model and the fedora data model was developed iteratively. our initial approach was to rely on unit tests to ensure the accuracy of the mappings. almost immediately, however, we encountered problems. despite the unit tests and testing in a staging environment, each new release of mapping code brought a cavalcade of bugs, some breaking entire sdr applications, others affecting individual digital objects. [16] a bug might be an actual unhandled software error or a mismapping. while in part due to the complexity of the mapping, the core problem was the heterogeneity of our digital object metadata. (the reasons for this heterogeneity are articulated above.) in essence, we did not know all of the cases that the mapping had to handle or the unit tests had to cover. to address this, the approach that we took was to test each mapping change against a large set of actual digital object metadata (for a typical code change, the metadata for 100,000 digital objects). testing against a large set of actual digital object metadata had a significant barrier: repeatedly retrieving digital object metadata from fedora at this scale took too long and testing against actual digital object metadata meant impacting the production fedora instance. to overcome this, a read-only cache was created where the datastreams for each digital object were stored in a zip file. the cache was kept updated by a process (called “cache-o-matic”) that queried solr for recently changed digital objects and regenerated the cache files for those objects. the cache was much faster than using fedora (in part because the cache used high speed storage) and had a smaller impact on production. to verify a change, a developer would run “before” and “after” tests. this test would allow the developer to verify that a proposed mapping change would improve roundtrip mapping (or at least not make it any worse) and not introduce an unhandled software error. for each digital object in the test: a fedora object is instantiated from the cache. the fedora object is mapped to a cocina object. the cocina object is run through object creation resulting in a new fedora object. the datastream in the original fedora object are normalized and compared against the datastreams in the new fedora object. the new fedora object is mapped to a new cocina object. the new cocina object is compared against the original cocina object. the normalization in step 4 begs explanation. there are a number of reasons that normalization might be necessary, but in general it is to account for the heterogeneity of the data. for example, in early sdr, labels in the content metadata datastream were expressed as: <attr type=”label”>my label</attr> later this changed to: <label>my label</label> the fedora to cocina mapping knew how to handle both cases. however, when mapping from cocina to fedora to generate content metadata, a label was always mapped to <label />. thus, to allow comparing the original content metadata datastream and the new content metadata datastream, the <attr type=”label”> form had to be normalized to the <label /> form. [17] extensive tooling was developed around round-trip testing. key features included: parallelizing testing for decreased testing times. summary statistics for test runs. logging of results for individual digital objects, including diffs of datastreams and cocina json. as migration progressed, roundtrip testing was also used to identify areas of the fedora data model that were unmapped, aspects of the mapping or normalization that needed additional work (because roundtripping was unsuccessful), or digital object metadata that needed remediation. roundtrip testing was run against increasingly larger samples of digital objects, up to and including all digital objects, and the results analyzed. also, the summary statistics were tracked over time, allowing us to measure our progress. roundtrip testing proved very successful, minimizing the impact of mapping development on users and allowing us to measure the completeness of our mappings. these statistics were particularly helpful in communicating with stakeholders about the project’s progress. strategy #5: use reports to understand the data one by-product of having the cache of fedora objects was allowing us to develop reporting tooling. this reporting made it very easy to answer questions about the digital objects which were crucial for the mapping development and the overall migration process. examples of some of the questions answered include: which objects are missing a mime type for file content? which objects are missing height and width for image content? what are all the values used for abstract types in descriptive metadata? which objects use edtf dates and what are those values? which objects are missing a use statement from their access rights? strategy #6: template unit tests the complexity of the mapping required a massive number of unit tests. using a feature in the rspec testing library called “shared examples,” mapping unit tests were structured so that a test consisted of providing a section of cocina json and the corresponding xml for a fedora datastream. [18] when the unit test was run, numerous tests would be performed on each set of this input json/xml automatically, including: mapping from cocina to fedora. mapping from fedora to cocina. mapping from the new cocina back to fedora, including handling normalization. comparing the starting and ending cocina comparing the starting and ending fedora boilerplate test code was significantly reduced, while increasing the completeness of testing. this approach was particularly effective for descriptive metadata as it allowed arcadia to write unit tests directly. those unit tests would be commented out until the mapping was implemented by a developer. (this was much more effective than the earlier approach in which arcadia maintained mapping examples separately which had to be kept in sync with the developer unit tests.) strategy #7: perform a rolling migration one of the earliest business requirements that emerged from planning for the migration was the need to have no “flag days.” this meant that migration could not involve taking sdr offline; it had to be available to users at all times. to meet this requirement, a rolling migration strategy was used. as described previously, all interaction with the datastore was abstracted behind the cocina object store within dsa. initially, that datastore was fedora. as part of migration, postgres was added as a datastore so that both datastores were in use at the same time. in postgres, cocina objects are stored as json in the cocina data model (i.e., there is no need to map between data models). [19] during migration when retrieving a cocina object from the cocina object store: if the cocina object was already saved to the postgres datastore, it was retrieved from the postgres datastore. otherwise, a fedora object was retrieved from the fedora datastore and mapped to a cocina object. that is, retrieving from postgres was preferred, with fallback to fedora. when creating a cocina digital object with the cocina object store: the cocina object was saved to the postgres datastore. the cocina object was mapped to a fedora object and saved to the fedora datastore. that is, the cocina object was saved to both datastores. when updating a cocina digital object with the cocina object store: the cocina object was saved to the postgres datastore (overwriting the existing cocina object). the existing fedora object was retrieved from the fedora datastore, updated based on the cocina object, and saved to the fedora datastore. that is, the cocina object was updated to both datastores. the other part of the migration process was a script that iterated over each of the digital objects stored in fedora. if the digital object metadata did not already exist in postgres, it mapped the fedora digital object metadata to cocina digital object metadata and saved it to postgres. this ensured that even infrequently accessed digital objects were migrated. for increased performance, the script was parallelized. when migration was completed, i.e., every fedora object was saved as a cocina object, the fedora datastore was removed, along with the last of the fedora-related code, all of the mapping code [20], and the entire caching/reporting/testing framework. [21] all digital object metadata is retrieved from and saved to the postgres datastore. this approach has a number of virtues: the migration could be performed over time without impact to users or downtime. if at any time during migration problems were encountered, the migration could be rolled back by deleting the cocina digital object metadata from postgres; the cocina object store would default to retrieving them from fedora. strategy #8: incorporate migration into project work the prior strategies considered together required a significant dedication of labor and time, and much of this work was invisible to end-users, resulting in negligible user-facing impact. as the team devised migration strategies, none of our project work was going away and would need continuous attention throughout the migration; this was especially important for work supported by external funding which already had set, rigid timelines. it was clear to us that alongside the technical components of the migration, we needed a strategy to plan and resource migration work alongside, and within, our ongoing project work. the team does not unilaterally, or in a vacuum, determine its portfolio priorities or work plans. key to this strategy was ensuring a clear, shared understanding between the team and department leadership. a set of conversations led to a series of guiding principles for what we called “sdr evolution,” a plan to continue chipping away at migration work over a multi-year period. these guiding principles included: deliver user-facing value as sdr evolves, tying evolution work to user requirements. evolve, rather than replace, sdr components (“no flag days!”). inject seams into components to make sdr more apiand service-oriented. focus on point improvements and not “changing the (sdr) world.” with these guiding principles in hand, atop a platform of shared understanding between the team and its stakeholders, we set out to map our known portfolio gaps—the ones most likely to be prioritized for our upcoming work plan—to sdr migration needs. as an example, we knew that replacing our self-deposit application was long-overdue work that our users deeply desired. [22] we also knew that the self-deposit replacement work was an opportunity to make advancements to sdr migration by, e.g., placing an intermediate api between the deposit application and the underlying fedora repository (strategy #2), or moving related workflow definitions out of the repository and into the workflow service (strategy #3). the onus was on the team and its manager to negotiate with product owners about how many of these needs could be included in the scope of our work cycles. fortunately, given leadership and stakeholders were already bullish on this strategy, product owners understood the value of including both their most beloved features along with migration work into our work plans. in the words of tom cramer, our department head: the focus of this work “is the evolution of sdr, updating key components methodically, leading to the long term replacement of the existing environment with a new system that is more modern, robust, and feature filled.” the migration away from one repository back-end to another would have been hampered, at worst, and greatly delayed, at best, without an opportunistic strategy to use sdr-focused and sdr-adjacent work cycles to subsidize migration. this way, the team was able to make progress on sdr migration while new projects and new features continued to be prioritized. conclusion in may 2022, the migration process commenced. the only challenge during migration was that the logs on the fedora server were filling up disk space faster than the logs were being rotated. to address this, the log disk space was expanded, but this required temporarily stopping fedora and restarting it. in a final act of defiance fedora refused to come back up. the team reached deep into its well of experience, mined ancient slack logs, and consulted accrued wisdom of the ages (google) to solve the problem. ultimately, over 4 million objects were migrated in under a week, problematic objects were remediated, the migration was audited to ensure that every object in fedora was in postgres, and lastly, fedora was removed. while the overall digital repository migration did deliver some benefits for sdr users (increased performance and stability, rewrite or refresh of some long neglected applications), incredible patience has been required. the team looks forward to a future in which the potential of sdr is not tethered to the limitations of fedora and we can work with sdr’s users to advance the scholarly mission of the university. acknowledgements the authors appreciate the contributions of current team members (vivian wong, ed summers, john martin, astrid usong), former team members (justin coyne, jeremy nelson, ben albritton, christina harlow, lynn mcrae, and numerous others over the lifetime of sdr) and sul colleagues (including but hardly limited to andrew berger, amy hodge, cathy aster, hannah frost, tom cramer). notes [1] only part of the infrastructure team’s time is dedicated to sdr; we also support other applications such as the sinopia linked data editor, know systemic racism, and the digital library of the middle east. [2] for storage efficiency, the files of the largest collection, google books (2.4 million digital objects) are stored in zipped files, where each zip includes an average of 800 files. in addition, sdr includes 46 terabytes of warc files, each which contains many files. thus, the actual number of files is significantly larger. [3] for more on the preservation system, see https://journal.code4lib.org/articles/8482 and https://2018.code4lib.org/talks/airing-our-dirty-laundry-digital-preservation-gaps-and-how-were-fixing-them. [4] for historical context on sdr, see https://www.dlib.org/dlib/september10/cramer/09cramer.html. note that the sdr nomenclature has evolved over time; this paper reflects the most current usage. [5] the historical record is incomplete. [6] the content files for the digital objects did not need to be migrated. [7] and by “we”, we mean chiefly christina harlow, whose work as sdr architect was crucial to development of the data model. [8] https://github.com/sul-dlss/cocina-models/blob/main/openapi.yml as yaml or rendered at https://sul-dlss.github.io/cocina-models/. [9] https://github.com/sul-dlss/cocina-models [10] hereafter, an instance of a cocina digital object that is instantiated as a ruby class will be referred to as a “cocina object.” [11] post-migration, we have continued to remediate data and add additional validations. for example, we recently added validation of dates. [12] https://github.com/sul-dlss/cocina-models/blob/main/description_types.yml [13] dor stands for “digital object registry,” vestiges of old naming conventions in sdr. [14] just as a cocina object is an instance of a cocina digital object that is instantiated as a ruby class, a fedora object is an instance of a fedora digital object that is instantiated as an activefedora ruby class. the fedora object, in our usage in sdr, is essentially a wrapper around the xml-based metadata datastreams. [15] using jhove had a number of shortcomings for us; however, those will not be addressed in this paper. [16] crucial to identifying these bugs as they occurred was honeybadger exception monitoring. honeybadger is a third party exception reporting service. honeybadger often allowed us to know about bugs before they were reported by users or to better identify the context for a bug reported by a user. [17] with hindsight, it may have been possible to simplify the mapping code a bit by performing normalization before mapping (i.e., step 2). [18] this description is simplified; the approach was actually much more flexible. [19] postgres was selected because of its ability to store and query json, it integrates well with rails, and is well supported by the dlss operations team. [20] actually, some of the mapping code will be retained to allow mapping to/from mods, as it is a preferred format for descriptive metadata, and to the xml that is consumed by the access system. [21] the reporting code was bespoke to fedora; new reports are accomplished with json queries into the postgres datastore. [22] this was one of the few exceptions in which we replaced an sdr application rather than evolve it. about the authors for the past 5 years, justin littman has been a software developer on the infrastructure team at stanford libraries’ digital library systems and services. previously, he worked for george washington university libraries and the library of congress. mike giarlo is a software engineer at stanford libraries, working on a team designing and developing software for long-term access to stanford university’s research and cultural assets. his focus is on development and maintenance of the stanford digital repository and collaboration with open source communities. he has held similar positions at penn state university, the library of congress, princeton university, the university of washington, and rutgers university. peter mangiafico is a software engineer and product manager at stanford university libraries. he previously worked as an engineer, educator, teacher, manager, and research scientist in private companies, for nasa and for other universities. laura wrubel is a software developer at stanford libraries and was formerly a software development librarian at george washington university. naomi dushay is a software developer at stanford university libraries for 15 years. she worked previously at colorado state university library, at cornell university in the digital libraries research group, and at cornell university libraries. for the past 6 years, aaron collier has been a software developer on the infrastructure team at stanford libraries’ digital library systems and services. previously, he worked for california state university libraries as the digital repository services manager. arcadia falcone is the metadata coordinator for the stanford digital repository and heads the stanford libraries metadata design unit. she was previously the discovery metadata librarian at yale university and has worked on metadata at the harry ransom center, the bancroft library, and st. mary’s college of california. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – review of digitalsignage.com mission editorial committee process and structure code4lib issue 24, 2014-04-16 review of digitalsignage.com digital signage has been used in the commercial sector for decades. as display and networking technologies become more advanced and less expensive, it is surprisingly easy to implement a digital signage program at a minimal cost. in the fall of 2011, the university of florida (uf), health sciences center library (hscl) initiated the use of digital signage inside and outside its gainesville, florida facility. this article details uf hscl’s use and evaluation of digitalsignage.com signage software to organize and display its digital content. by clifford richmond and matthew daley introduction digital signage has been used in the commercial sector for decades for the purposes of entertainment, advertising products and services, and displaying information such as menus, flights and messages. as display and networking technologies become more advanced and less expensive, the ability to experiment with digital signage is now easier than ever. the university of florida, health sciences center libraries has been using digitalsignage.com signage software for over two years in its gainesville, florida facility. this article examines some of the processes involved in using digitalsignage.com signage software to create a digital signage campaign, and examines the software’s strengths and weaknesses. project goals there were many goals in mind when the health science center library (hscl) decided to test the use of digital signage in its gainesville, florida facility. digital signage was anticipated to be a more efficient and effective alternative to the conventional practice of taping or tacking hard copy material to walls and bulletin boards. by reducing hard copies as a means of communicating library information and services to local patrons, the library hoped to reduce wear and tear on walls, increase the aesthetic appeal of library space and reduce paper costs and the paper waste stream. by virtue of the digital nature of the content, where movement from source to display takes place via a computer network, the library anticipated a reduction in the time required to create, post, review and remove content. criteria and considerations budget: although a budget cap was not established for this project, cost was by far the single biggest consideration in determining digital signage software and hardware options. it was determined from the outset that the library would use existing hardware. networking: for the purposes of updating, patching and network accessibility, it was preferred that any workstations involved in digital signage be able to function when joined to the uf active directory (ad) [1] domain. content flexibility: the hscl desired the ability to display various types of still, motion and sound content. existing digital signage at uf the hscl investigative efforts concentrated on speed of initial implementation rather than an exhaustive evaluation of current efforts at the university. of the five digital signage implementations researched, the hscl focused on the two programs that were located in the health science center (hsc). the hscl believed that these two projects would provide a close representation of the specific vetting process and timelines to expect. of the two projects currently in place in the hsc, one involved an initial cost of $15,000 and the other one used free software (digitalsignage.com) and existing equipment. digital signage solution in addition to the fact that the software is free, the number of content types that can be displayed and the flexibility of control over the visual layout were the reasons digitalsignage.com software was chosen for this project. in order to start using the software, the user must create a free account at www.digitalsignage.com. after creation of an account, the user may use web versions of the software or download desktop versions. the hscl chose to use the desktop versions, signage studio and signage player. content is first organized using the signage studio adobe air application. then signage player devices are configured to play an existing campaign of content using the signage player adobe air application. adobe air is required to be installed on any computer that is used to organize content and on any device that will coordinate the display of content. the free edition uses a cloud-based [2] approach to content storage, with a limit of 1 gb of storage per account. however, it is not a requirement that content be uploaded to the cloud in order to be displayed on the signage players. in order to organize digital presentations, www.digitalsignage.com defines the following terms and concepts: resource: a fundamental unit of digital content, such as a .png, .jpg, .mov, or .ppt file. components: prebuilt flash-based widgets that allow the inclusion of such items as text and media rss feeds, video, webpages, stock tickers, clocks, video capture inputs and charts in a timeline. the user can typically modify both the data source, as well as the visual elements of these widgets using different skins [3]. screens (outputs): screens are created to characterize the content and layout that a specific signage player will display. a digital signage campaign may use more than one screen. screen divisions: the subdivision of a logical digital display screen into different areas. these areas may represent subdivisions in both x (horizontal) and y (vertical) space, or z space (representing multiple layers). channels: the term and concept used to represent, in the timeline, what content is associated with each screen subdivision. timeline: a chronological configuration of components, scenes and resources. the media rss feed timeline appears in the top portion of figure 3. as time increases from left to right, the content indicated plays in the different channels. sequencer: the timing mechanism used to display content whereby timelines play in an infinite loop. scheduler: the timing mechanism used to display content whereby timelines are set to play at specific dates or times. scenes: scenes are saved user instances of one or more widget components. the same scene can be used in multiple timelines and multiple campaigns. campaign: a single configuration that includes a collection of timelines, resources, screen divisions, components or scenes. each signage player device is associated with only one campaign at a time. signage studio the general workflow in signage studio is: upload content (resource) > create a campaign targeting a screen (output) > create a timeline figure 1 shows the information supplied to the campaign creation wizard to create the vpha campaign. the vpha campaign targets the screen output named “vpha monitor,” with a resolution of 1920×1080. the default “sequence” mode will be used so that timelines will loop after completion. only the campaign and output information are required for this step. figure 1. vpha campaign and screen settings next, using the timeline wizard, the user must supply a name, duration and screen layout to be used. figure 2 shows the timeline details supplied for the media rss feed timeline. figure 2. media rss feed timeline settings the (previously created) screen division template divides the 1920 x 1080 screen into five (5) areas. these areas were designed to display the day, time, date, a multimedia rss feed and a separate text rss feed. the newly created timeline is aware that the designated screen layout provides for five different channels of content. examination of figure 3 reveals that the five content channels (outlined in red) will play in the correspondingly named screen divisions (indicated by the arrows) and will play for five minutes and then repeat. figure 3. screen divisions, channels and content relationship in the media rss feed timeline signage player in order to play the media rss feed timeline, the signage player air application must be installed on a device with a display. the hscl uses large screen displays connected to workstations running windows 7. during the installation, the user assigns a name to the signage player device and designates the screen (output) and content campaign to run, as indicated in figure 4. after clicking the “next >>” button, the user can start running the campaign by clicking the “start” button (figure 5). figure 4. the signage player name, content campaign and screen (output) must be designated figure 5. the user clicks the start button to run the designated campaign content during the signage player installation, the file “signagecontroller.exe” is copied to the startup folder for all users. upon subsequent logins, the signage player software will automatically start and begin downloading content to a local cache directory. an internet connection is required for the initial caching of content to the local signage player hard drive from the cloud. after the current campaign’s content has been cached, an internet connection is only necessary when content organization has changed, or if the content includes web pages from the internet. after content has been modified in signage studio, the signage player software can be restarted from within signage studio. hardware configuration the hardware used for digital signage configuration inside the library is detailed below. the specific equipment used is included for informational purposes, and is not an endorsement of a particular model or manufacturer. the displays located outside the library could not be directly connected to a computer, therefore, the video and audio signals were transmitted over cat5 network cable. hardware configuration inside the library computer: dell optiplex 990, windows 7 32-bit software: signage player, adobe air, powerpoint viewer, vlc player ram: 4 gb cpu: intel® core ™ i7-2600 cpu @ 3.40 ghz/3.40 ghz graphics card: ati radeon hd 3400 series video port: dell display port nic: intel® 82579lm gigabit network connection sound card: onboard realtek alc269 high definition (hd) audio device external speakers: logicitech 2.1 (stereo with separate subwoofer) display: make/model: samsung (led) model un55es6900 viewable area: 55” resolution: 1920 x 1080p through pc input or hdmi video input: dvi, vga and hdmi inputs are available audio input: 3.5 mm stereo trs connector speakers: built-in stereo video signal: computer display port-to-dvi adapter > dvi-to-hdmi cable > hdmi input of monitor workstation and active directory modifications in order to allow the workstations to recover from a reboot and start playing content without any human interaction, the following modifications were necessary: 1an ad user account was created for the purpose of logging into the signage player workstations. 2the “interactive logon: do not require ctrl+alt+del” policy was “enabled” using a group policy object (gpo). this setting is located under: computer configuration > policies > windows settings > security settings > local policies > security options. 3automatic login (autoadminlogin) was enabled in the registry, and the above user account credentials supplied. these settings are shown in figure 6. in windows 7, these registry settings are located under: hkey_local_machine\software\microsoft\windows nt\currentversion\winlogin figure 6. automatic login (autoadminlogin) registry settings 4in order to automatically start the signage player software when (only) the above user logs in, the “signagecontroller.exe” file was moved from the “all users” startup directory to the above user’s startup folder. in windows 7, a user’s startup folder is located at: c:\users\username\appdata\roaming\microsoft\windows\start menu\programs\startup the results figure 7 shows the media rss feed timeline of the vpha campaign. figure 7. media rss feed timeline as appears on signage monitor figure 8 shows a weather timeline which uses a custom skinned weather component (widget). the screen layout consists of a full-screen background layer (clouds) with the weather widget in the foreground. figure 8. weather timeline as appears on monitor figure 9 is a still photo of a full-screen .mp4 animation produced in adobe after effects. signage studio was instructed to use an external player (vlc player) located on the signage player computer to play the file, which was also located on the local signage player computer’s hard drive. figure 9. still image of harry potter exhibit after effects animation conclusions digitalsignage.com pros: the software is free. with the ability to use external players, there is no limit to the type of content one can display. the free edition grants access to hundreds of shared resources, samples and scenes. there are (software) signage players available that allow content display on android, windows and mac devices, ipads, and from inside a browser. a nearly infinite number of x, y and z screen layout configurations is possible. the software supports a logical screen resolution of 4096 x 4096, so one computer with a multi-video display port can coordinate content on several display devices. conversely, one could combine several physical displays to create a video wall. technical help response by email, telephone and online chat has been immediate. the web site has many instructional videos for learning the various aspects of the software, and the videos are clear and concise. digitalsignage.com cons: the free version does not support the assignment of roles and permissions, which would allow for segmentation of duties in an enterprise environment. the free version does not provide access to the api [4] and sdk [5]. consequently, one cannot activate an emergency notification timeline (for example) based on a trigger event outside the software. overall assessment: the hscl is very satisfied with the digitalsignage.com software. all project goals have been realized. the elimination of a hard-copy phase for campaigns has reduced the use of paper, toner, spray adhesive and foam board, reduced the wear on printers and reduced the staff time required to print, cut, mount and transport poster-sized displays. the time required to upload content, insert it into a timeline and deliver to the signage player displays routinely takes five minutes. the information to square-feet of display area has been greatly increased. the number of individual pieces of content posted on each of the three (current) displays varies over time, but at this writing thirty-eight (38) different timelines are in rotation, each comprised of one or more pieces of content. the library has also received positive feedback from students, faculty and staff. reference and notes [1]. although active directory is a term used to refer specifically to the type of directory service used by microsoft corporation, the term is widely used by it professionals to refer more generally to a/the microsoft networking framework. [2]. ‘cloud based’ is the practice of storing regularly used computer data on multiple servers that can be accessed through the internet. merriam-webster.com. cloud computing [internet]. merriam-webster.com; [2014 feb , cited 2014 feb 05]. available from: http://www.merriam-webster.com/dictionary/cloud%20computing [3]. in computing, a skin is a custom graphical appearance achieved by the use of a graphical user interface that can be applied to specific software and websites to suit the purpose, topic, or tastes of different users. wikipedia contributors. skin (computing) [internet]. wikipedia, the free encyclopedia; [2013 nov 12, cited 2014 feb 05]. available from: http://en.wikipedia.org/wiki/skin_(computing) [4]. revealing the application programming interface (api) of a program allows others to create code that can make use of the internal workings of the software program. [5]. a software development kit (sdk) provides additional tools, in addition to apis, that allow development for a platform. about the authors clifford richmond is the it director for the health sciences center libraries at the university of florida. his interests include 3-d modeling, visualization, simulation, training and powershell scripting. matthew daley is the webmaster and graphic designer for the university of florida health science center libraries. he enjoys exploring the design tropes of different decades and is a fan of outsider art, kosmische musik, dub, capoeira and tottenham hotspur football club. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – using amazon s3 in digital preservation in a mid sized academic library: a case study of ccsu eris digital archive system mission editorial committee process and structure code4lib issue 12, 2010-12-21 using amazon s3 in digital preservation in a mid sized academic library: a case study of ccsu eris digital archive system with the increasing numbers of born digital and digitized objects in academic libraries from sources such as digital collections and institutional repositories many academic libraries need to seriously consider implementing some form of digital preservation system. in 2009 the central connecticut state university library decided to use amazon s3 for digital preservation storage despite some drawbacks. the library has developed a system, eris digital archive, to manage all digital preservation processes and to make the system as compliant with the oais model and “trustworthy digital repositories” as possible. by edward iglesias and wittawat meesangnil introduction burritt library became involved with digital preservation out of necessity. electronic resources were increasing and upon beginning a new digitization project we took the opportunity to create a concrete preservation plan. in early 2009 we realized the need for a digital preservation system after disc space on our network share ran out due to an abundance of uncompressed tiff files. we temporarily used an external hard drive for backup and started looking for a system. exploring the digital preservation landscape when burritt library started looking at digital preservation, we began with an environmental scan to become acquainted with the products and solutions available. an emphasis was placed on systems using premis preservation metadata and the oais model. the first system we looked into was oclc’s digital archive (http://www.oclc.org/digitalarchive/) since it was designed mainly to work with contentdm, the digital collection management system we already used. although it had many of the features we wanted, digital archive did not suit us due to the high costs involved in storing terabytes of data. we were also concerned about preserving digital objects not contained in contentdm. the second solution we looked at was lockss (http://lockss.stanford.edu/lockss/home), a distributed digital preservation system. it was originally created to preserve electronic journals that the library purchased, with the permission of publisher. lockss can also be used for preserving other kinds of digital objects on a smaller scale. private lockss networks (plns) is a solution where six institutions can set up their own private lockss network to preserve any digital object desired (private lockss networks, 2008). this solution was desirable since it had little cost to start and maintain. it would also expand our staff’s skill greatly. unfortunately, we were unable to persuade other institutions in our consortia to participate. the idea of implementing an open source distributed preservation system without any help from a vendor created a high entrance barrier for this solution. at this point we realized the need to implement some sort of in-house solution. the first issue to address was storage. we were quite familiar with traditional archive storage for digital objects such as tape backup, preservation dvds, and raid hard drives but the problem remained that we needed to have two copies of our archival object in at least two different geographic locations. the first two solutions we looked at addressed this issue. oclc digital archives would let us mail in dvds, or hard drives that would then be processed in their data warehouse. our solution would have to be similarly flexible in its input. we started to look at cloud computing as an option after peter murray wrote an article that reviewed amazon s3, one of the most popular cloud storage services, as a possible choice for digital preservation on his blog. (murray, 2008) using amazon s3 as dark archive seemed to be a good idea, since the cost was much lower than oclc digital archive and we would also have instant access to our content, unlike traditional off site storage services. we did an annual cost comparison between s3 and oclc digital archive based on a ball park figure of 5tb of data storage. cost for 5tb oclc amazon s3 notes set up $3,000 $0* *not including staff hours on design / setup ingest $2,400 $0   annual storage costs $26,850 $8,610   monthly fee $0 $716   interface provided by oclc we would design   hardware costs $0 $1,350 hp windows home server @ $600; 5 2tb drives = $750 ($150 * 5) hardware costs none extra $800 using current staff 4 days / year = $800 ($200 / day) total costs $32,250 $9,362           table 1. cost comparison between oclc digital archive and amazon s3. although many people had proposed s3, at the time no one had yet implemented a version of it that we could learn from. we needed to develop a system based on what we had learned from the oais model. we also had a very modest budget and no extra personnel. we needed to look at the oais model and digital preservation best practices and implement the most important part of it if this project was going to be feasible. an ideal system would have the following characteristics: all of the technology would be open source and transparent it would follow the oais framework (lavoie, 2004) it would comply with trusted digital repository standards (crl, oclc, 2007) it would need to be as automated as possible given those lofty aspirations, the main goals of our system was to keep the data safe while maintaining costs low. maintenance was also an issue since this system would have to run on its own with no dedicated programmer or full time administrator. we identified, to focus on, these core functionalities: data integrity and providing future accessibility through preservation metadata. network file transfer verification by verifying data integrity through the transfer process to cloud storage. management of the data so that it can be located at any point in the process. design: core functionalities data integrity we decided to address the problem of data integrity through the use of file format verification tools. in early 2009, there were a handful of tools available for file integrity checking, format verification, and creating preservation metadata. we chose statistics new zealand prototype premis creation tool, a tool set available from the premis website (tools for preservation metadata implementation, 2009). it employs these three tools to extract technical metadata from digital objects. jhove – jstor/harvard object validation environment, provides functions to perform format-specific identification, validation, and characterization of digital objects. (jhove, 2009) droid digital record object identification developed by uk national archives based on pronom registry of file signatures specific to file types. (droid, 2006) new zealand metadata extractor, developed by the national library of new zealand to programmatically extract preservation metadata from a range of file formats. its output is used in providing creating software and executable environment information. (national library of new zealand, n.d.) all three tools above generate output in xml format. xslt is then used with the included xsl style sheet to transform the three outputs into premis preservation metadata (premis editorial committee, 2008). network file transfer verification since we decided to use cloud storage as archival storage, a major issue in our system was how to be certain that data transferred over the internet to be archived is not lost or corrupted during transmission. amazon s3 provides a mechanism to detect data lost or corrupted by providing the ability to specify an md5 checksum for the data being sent to s3. amazon s3 will check the object against the provided md5 value. if they do not match, amazon s3 will return an error so the client can try retransmitting. (amazon web services, 2006) many backup software applications already utilized this function [1]. in our system we use couldberry backup for windows home server which allows us to schedule backups automatically with a nice gui.(http://www.cloudberrylab.com/) we also provide another layer of verification by using bagit, a network file transfer tool developed by the library of congress. it provides a way of packaging up arbitrary digital content, so that it can be reliably transported both via physical media as well as network transfers. (library of congress – transfer tools, n.d.) management a mysql database was developed to keep track of the archival object as it goes through the system. we provide a web interface for the depositor to enter their deposit information and also to search for items in the database: figure 1. erisda mysql web interface to perform the digital preservation ingestion process, we automated some of it with batch files and automated backup to amazon s3. design: system system architecture the system was designed to have one server running locally. this server is responsible for processing files upon ingestion, storing processed archival objects in its raid1 hard drive and then backing up archival objects to amazon s3 at night as part of an automated process. despite a preference for open source systems we decided to use an off the shelf hp mediasmart server running windows home server as the operating system [2]. the reasons for going this route had mostly to do with cost. we looked at pre-built linux servers with raid controllers and 4 tb of storage and the cost was significantly more than the hp home server. while reliability was undoubtedly an issue this was mitigated due to the cloud backup. another factor that played into our decision was the practicality of the system. although our it department houses several linux boxes they like them “over there” in the data center. this was not practical since one of our goals was to reduce network traffic. for what was essentially a nas running in the library, windows integrated nicely with the existing network. users can access the server simply by typing \\servername in their windows explorer address bar while staff can interact with the system via windows remote desktop, which has clients available in multiple platforms. preservation process the eris digital archive workflow can be separated into 3 steps: deposit: users deposit digital objects to be archived to the server ingest: system administrator processes the deposited digital objects (creates preservation metadata, puts digital objects in bagit format for network transfer) and updates the mysql database archive: system administrator puts digital object along with preservation metadata into erisda server and makes a second backup copy to amazon s3 cloud storage. also manages long-term maintenance, such as file format refreshing, migration. deposit to deposit digital objects, users log on to a mysql interface to enter information and receive a deposit id. users then package content (directory structure, file naming convention), according to documentation, into a deposit package. at that point the package is transferred to a “deposit” directory via network file transfer or an external hard drive (if data is too large for network transfer). users can also check their past deposit status to find out what items they’ve already deposited via a mysql interface. figure 2. erisda mysql web interface: check deposit status ingest to perform data integrity checks and create preservation metadata we use the new zealand prototype premis creation tool [3]. this is used in combination with dos batch files to automatically run the three extractor tools and use xslt to transform outputs to preservation metadata. this batch file (below) can be created quite easily, or converted into a shell script for a linux environment. @echo off rem !!—premisrunner.bat, will be called by premis.bat--! rem !!-param %1 pass from premis.bat is each sub directory --! echo running jhove c: cd c:\jhove\bin echo jhove java -jar jhoveapp.jar -c ..\conf\jhove.conf -k -o c:\temp\jhoveoutputfile.xml -h xml -krs %1 echo running escape characters filter cscript escapefilter1.vbs cd c:\temp del jhoveoutputfile.xml ren filteredjhoveoutputfile.xml jhove_out.xml echo jhove done echo running droid cd c:\droid java -jar droid.jar -l%1 -sdroid_signaturefile_v29.xml copy output.xml c:\temp\droid_out.xml echo droid done echo running nlnz metadata extraction tools cd c:\metadata-extractor extract.bat extract "nlnz data dictionary" default complex nlnzoutputfile 2010 recurse "%1 copy nlnzoutputfile.xml c:\temp\nlnz_out.xml echo nlnz done echo running xslt rem !!—just run xslt on jhove output, the other two outputs are indentified in xsl--! cd c:\premis msxsl c:\temp\jhove_out.xml erisda_premis.xsl -o premis.xml echo transformed md %1\"preservation metadata" move premis.xml %1\"preservation metadata"\premis.xml" to make this batch file more automated we created another batch file to call the main batch file recursively to extract data for each sub-directory. echo this will create preservation metadata for each sub-directory of the directory you enter set /p input=[enter full path of directory you want to extract] rem cd %input% for /d %%a in (%input%\*.*) do call c:\premisrunner.bat "%%a" after we validate every object and create preservation metadata, we need to prepare them for network transfer. bagit is used to put the object entity into a bagit bag. this process can also be automated with a batch file. we also run anti-virus on the home server and scan contents in this step. bagit then puts files into a directory and creates checksums “payload-oxum” for each bag, which enables us to verify data integrity if we need to retrieve content from the amazon s3 cloud. content of bag-info.txt payload-oxum: 20721997.49 bagging-date: 2010-02-24 bag-size: 19.8 mb batch file to create a bagit bag. @echo off rem !!—bagitrunner.bat, will be called by bagit.bat--! rem !!-param %1 pass from bagit.bat is each sub directory --! cd c:\bagit-3.6\bin bag.bat baginplace %1 echo bagit done the bag is at %1\ pause after the files are processed and put in a bag ready for transfer, we update the mysql database to reflect data in the system. currently we record information such as bag name, bag location, and bag checksum. we again use multiple dos batch files to get information about the bag name then read content of bag-info.txt to get the bag checksum and then insert the information into the mysql database. @echo off rem !!—get input from user, get name of directory--! set /p input=[enter directory you want to update mysql] set drive=%drive:~0,2% set /p path=[enter path this batch of bags reside on the server] set /p depositid=[enter deposit id] set /p pubid=[enter publication id, enter null if don't know] %drive% cd %input% for /d %%a in (*.*) do call c:\readfrmfilebat.bat "%%a" %path% %depositid% %drive% %pubid%   @echo off rem !!—get patload-oxum from bag-info.txt--! cd %1 for /f "tokens=2" %%a in (bag-info.txt) do ( cd.. rem echo %%a call c:\mysqlrunner.bat %1 %2 %3 %%a %4 %5 exit /b   @echo off rem !!—insert info to mysql--! :: var passed from readfrmfile.bat is %1bagname %2serverpath %depositid %4payloadchksum %5drive letter %6 dir.size %7 pub id set name=%1 ::remove quote for /f "useback tokens=*" %%a in ('%name%') do set name=%%~a c: cd c:\program files\mysql\bin mysql -h library.ccsu.edu -u user -ppassword -e "set @bagname:='%name%'; set @location:='%2'; set @depositid:='%3'; set @chksum:='%4'; set @size:='%6'; set @pubid:='%7'; source sqlrun.sql;" erisda %5 once the ingestion process is finished the deposited data is packaged in a bag along with the preservation metadata. archive after ingestion, we copy processed data from the working directory to the archive directory. windows home server uses drive extender technology to enable interaction with multiple hard drives as one. to backup data in raid mode, we simply check “duplication” for each directory in whs consoles and data in that directory will be copied into two hard drives. data in the archive directory is backed up to amazon s3 using software called “cloudberry whs edition” which is configured to automatically backup selected directories. figure 3. cloudberry backup gui evaluation / lessons learned usability / ease of use network share the system we developed has proven to be quite usable. staff needed a minimum of training since they were used to storing items on a network share. mysql the process for updating the internal database also works well. while at first we were using a php/html solution we found that mysql clients worked better and were more secure for simply updating tables. we did keep one page up that simply displays the holdings so that staff may track their items without having to log on to mysql. metadata creation the process for creating metadata using premis and bagit works well enough — although it is not as automatic as hoped. we are currently researching other solutions to see if these scripts might be improved. amazon s3 the decision to go with amazon’s s3 storage has been a very good one. the cost is very low and we have had no downtime in a year of use. hardware our windows home server keeps chugging along. it has not suffered any hardware or software failures in a year of use. data transportation one area we did have a loss was in the portable hard drives we use for transporting large amounts of data. the first one we bought died unexpectedly taking quite a bit of work with it. there would not have been any lost data if the staff member using it had followed proper protocol, but at the time she was using the external drive as her only storage area. lessons have been learned from this and we have made the policy of only using the external drive as a transport mechanism clearer. further development our current research is focused on greater automation of processes. additionally we are investigating ways of testing disaster recovery options. because of the size of the records it is non-trivial to download more than a terabyte just to do a checksum. we are also looking closely at the development of archivematica (http://archivematica.org), an open source comprehensive digital preservation system developed by artefactual systems. its alpha version was released in may 2010. works cited amazon web services. (2006). amazon simple storage service api reference, api version 2006-03-01. retrieved october 1, 2010, from http://awsdocs.s3.amazonaws.com/s3/20060301/s3-api-20060301.pdf. (return to citation). crl, oclc. (2007, february). trustworthy repositories audit & certification (trac) : criteria and checklist. retrieved october 1, 2010, from http://www.crl.edu/sites/default/files/attachments/pages/trac_0.pdf. (return to citation). droid. (2006). droid: project web hosting – open source software. retrieved october 1, 2010, from http://droid.sourceforge.net/. (return to citation). jhove. (2009, february 25). retrieved october 1, 2010, from jhove – jstor/harvard object validation environment: http://hul.harvard.edu/jhove/. (return to citation). lavoie, b. f. (2004, january). the open archival information system reference model: introductory guide. retrieved october 1, 2010, from dpc technology watch series report 04-01: http://www.dpconline.org/docs/lavoie_oais.pdf. (return to citation). library of congress – transfer tools. (n.d.). retrieved from http://sourceforge.net: http://sourceforge.net/projects/loc-xferutils/. (return to citation). murray, p. (2008, may 16). long-term preservation storage: oclc digital archive versus amazon s3. retrieved october 1, 2010, from disruptive library technology jester: http://dltj.org/article/oclc-digital-archive-vs-amazon-s3/. (return to citation). national library of new zealand. (n.d.). metadata extraction tool. retrieved october 1, 2010, from http://meta-extractor.sourceforge.net/. (return to citation). premis editorial committee. (2008, march). premis data dictionary for preservation metadata version 2.0. retrieved october 1, 2010, from http://www.loc.gov/standards/premis/v2/premis-2-0.pdf. (return to citation). private lockss networks. (2008). retrieved october 1, 2010, from lockss: http://lockss.stanford.edu/lockss/private_lockss_networks. (return to citation). tools for preservation metadata implementation. (2009, july 22). retrieved october 1, 2010, from premis: preservation metadata maintenance activity (library of congress): http://www.loc.gov/standards/premis/tools.html. (return to citation). endnotes [1] for linux, see s3sync: http://www.s3sync.net/wiki [2] http://www.hp.com/united-states/campaigns/mediasmart-server/ [3] we’re planning to switch to the file information tool set (fits) (http://code.google.com/p/fits/), a more comprehensive tool set for file validation. xsl style sheets to transform fits output to premis are also available from its website. subscribe to comments: for this article | for all articles 2 responses to "using amazon s3 in digital preservation in a mid sized academic library: a case study of ccsu eris digital archive system" please leave a response below: jodi schneider, 2010-12-25 the cost comparison is particularly useful – thanks for that! options in storage for digital preservation | disruptive library technology jester, 2013-10-21 […] iglesias, edward and wittawat meesangnil (2010). using amazon s3 in digital preservation in a mid sized academic library: a case study of ccsu eris digital archive system. the code4lib journal, issue 12, retrieved 5-jan-2011 from http://journal.code4lib.org/articles/4468 […] leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – workflow tools for digital curation mission editorial committee process and structure code4lib issue 20, 2013-04-17 workflow tools for digital curation maintaining usable and sustainable digital collections requires a complex set of actions that address the many challenges at various stages of the digital object lifecycle. digital curation activities enhance access and retrieval, maintain quality, add value, and facilitate use and re-use over time. digital resource lifecycle management is becoming an increasingly important topic as digital curators actively explore software tools that perform metadata curation and file management tasks. accordingly, the university of north texas (unt) libraries develop tools and workflows that streamline production and quality assurance activities. this article demonstrates two open source software tools, autohotkey and selenium ide, which the unt digital libraries division has adopted for use during the pre-ingest and post-ingest stages of the digital resource lifecycle. by andrew james weidner and daniel gelaw alemneh introduction digital curation is the continuous activity of managing and enhancing the use of digital resources over their life-cycle and over time. digital curation starts when an item is created (born-digital) or selected for digitization (analog) and continues through image processing, metadata capture, derivative creation, and preservation for long-term access (alemneh 2010).  high quality metadata is necessary for implementing reliable, usable, and sustainable digital collections (sumner & custard, 2005).  recognizing the important role of standardized metadata in digital repositories, the university of north texas (unt) libraries actively promote metadata-based digital resource lifecycle management. the unt digital libraries division manages content for the portal to texas history and the unt digital library.  the portal to texas history provides access to cultural heritage materials related to the history of texas.  the unt digital library showcases the scholarly and creative output of the university and highlights some of the libraries’ research holdings.  in managing these repositories, the digital libraries division utilizes various tools and mechanisms to enhance metadata consistency and precision across all digital resources.  before ingesting digital objects, web-based metadata creation templates draw terms from locally controlled vocabularies to ensure standardized data entry values.  after objects have been published online, the metadata records are analyzed with python scripts and command line tools for quality review (phillips, 2013). this article describes specialized tools and workflows developed by the unt digital libraries division that use autohotkey and selenium ide open source software to manage files and create and edit metadata.  autohotkey is especially useful for pre-ingest activities such as file management, data entry, and digital object quality control.  post-ingest metadata enhancements automated with selenium ide facilitate the use, reuse, and preservation of digital objects.  autohotkey and selenium ide provide quick and accurate digital resource management capabilities with minimal human intervention. automated file management:  autohotkey autohotkey is free, open source software for the windows operating system which allows users to create automation scripts.  users write scripts that send multiple keystrokes to the operating system with a single key combination, or hotkey.  the autohotkey scripting language supports programming constructs (e.g., variables, loops, conditionals), dynamic guis, and direct interaction with the windows api.  while autohotkey provides a convenient platform for quickly developing tools to assist with digital curation tasks, the portability of tools created with autohotkey may vary; scripts often require modifications to work properly on multiple machines.  if implemented in a production environment with multiple computers, workstation settings and software should be standardized as much as possible to minimize differences between systems. autohotkey is easier to learn and implement than most scripting languages.  it is an ideal automation solution for digital curators who work primarily in the windows gui and have limited experience with computer programming.  simple autohotkey scripts are useful for large, one-off projects that require repetitive file management actions.  for example, a digital curator may write a script that creates a new folder, enters a portion of the folder name, and readies the cursor for further data entry (figure 1).  a handful of such hotkeys, each designed for one part of the workflow, can speed up large projects and improve metadata accuracy by automating all or a portion of the file management and data entry tasks.  hotkeys should be selected with care to avoid conflicts with key commands reserved for the operating system and other applications. ; define the hotkey: win + n #n:: ; create a new folder via the file menu send, {altdown}f{altup} sleep, 200 send, w sleep, 200 send, f sleep, 400 ; enter the custom folder name text ; for example, the year of a yyyymmdd date send, 2013 figure 1: autohotkey script to create a new folder more complex scripts improve efficiency for those who are unfamiliar with command line equivalents.  with autohotkey, digital curators can perform batch file management tasks within the operating system’s gui.  for example, a digital curator may develop a suite of custom looping scripts to use when renaming all of the folders in a directory (figure 2). pause::pause ; the pause key pauses and restarts the loop ; hotkey: ctrl + r ^r:: ; loop renames folders from mm-dd-yyyy to yyyymmdd format loop { ; activate rename for selected folder send, {f2} sleep, 200 ; move cursor to start of string send, {home} sleep, 200 ; delete the hyphens send, {right 2}{delete} send, {right 2}{delete} sleep, 50 ; cut the yyyy value send, {shiftdown}{end}{shiftup} send, {ctrldown}x{ctrlup} sleep, 50 ; move cursor to start of string send, {home} sleep, 50 ; paste the yyyy value send, {ctrldown}v{ctrlup} sleep, 50 send, {enter} sleep, 50 ; move to the next folder send, {down} sleep, 500 } figure 2: autohotkey loop to rename all folders in a directory autohotkey can also automate actions for multiple applications in one script.  such scripts minimize mouse usage and allow digital curators to focus on content.  for example, the script in figure 3 opens a folder selected in windows explorer, opens the first file with notepad++, and locates the first <mods:number> tag in the file. ; hotkey: alt + o !o:: ; open the selected folder send, {enter} sleep 200 ; select the first file send, {down} send, {up} sleep, 50 ; open the file with notepad++ send, {appskey} sleep, 100 send, n sleep, 100 send, {enter} ; wait for the file to load settitlematchmode 2 winwaitactive, notepad++ sleep, 200 ; activate the find window send {ctrldown}f{ctrlup} ; wait for the find window to appear winwaitactive, find sleep, 100 ; locate the first tag sendinput, sleep, 100 send, {enter} figure 3: automate actions in more than one application. autohotkey applications for metadata creation autohotkey provides a convenient platform for developing digital curation micro-applications that bundle frequently used scripts in a gui.  the standard autohotkey installation includes a compiler for creating executable files that run on any windows computer, even if autohotkey is not installed.  the digital newspaper unit (dnu), a sub-group of the unt digital libraries division, has constructed a set of tools that assist with the creation of digital objects and their associated metadata.  for example, newspapernotes.exe (figure 4) automatically types commonly used metadata notes according to a local data entry standard (figure 5).  the application reduces typographical errors and increases productivity for the dnu’s student employees. figure 4:  the newspapernotes.exe gui. figure 5: newspapernotes.exe controlled vocabulary values. the newspapernotes.exe gui displays keystroke counters for visual interest and short references to the standard hotkey values.  the application also includes three user-defined hotkeys for non-standard notes that may occur on a regular basis in a particular set of newspapers.  users define the wildcard hotkeys via the edit menu (figure 6) and can display the values at any time with the file menu (figure 7). figure 6: newspapernotes.exe edit menu and input box. figure 7: newspapernotes.exe file menu and user-defined values display. in addition to sending keystrokes to the operating system, autohotkey has the ability to interact directly with the windows api.  newspapernotes.exe uses this feature to create and open text files that contain newspaper metadata and easily save and close them after data entry.  figure 8 shows the code that creates a three line text file called  metadata.txt  with the  fileappend  command and opens it for edting with the  run  command.  prior to executing these commands, the script copies the directory path from the windows explorer address bar to the clipboard with f4, ctrl + a, and ctrl + c. fileappend, volume:`nissue:`nnote:, %clipboard%\metadata.txt sleep, 2000 ; wait two seconds for the file to be created run, metadata.txt, %clipboard% figure 8: create and open a text file with the  fileappend  and  run  commands. after entering the volume number, issue number, and any notes in the  metadata.txt  file, newspapernotes.exe users can quickly save the file with another hotkey.  figure 9 illustrates the code that accesses notepad’s file menu with the  winmenuselectitem  command to save and close the file while ignoring any documents open in notepad++. winmenuselectitem, notepad,, file, save,,,,,, notepad++, winmenuselectitem, notepad,, file, exit,,,,,, notepad++, figure 9: select options from notepad’s file menu with the  winmenuselectitem  command. autohotkey applications for metadata editing autohotkey applications are also useful for editing existing metadata published in an online repository.  the unt digital libraries division has developed an application, dashboardtools.exe, that bundles search scripts for its online editing platform and public repository interface.  the application also contains hotkeys that quickly switch between the public and administrative views for individual metadata records.  with this tool, digital curators for the portal to texas history and the unt digital library can search for and edit published records more efficiently.  for example, the script in figure 10 searches the title field in the portal to texas history public interface, no matter what the currently active application happens to be. ; hotkey: alt + 3 !3:: inputbox, searchstring, portal title,,, 250, 100,,,,, ; create a 250x100 pixel input box if errorlevel ; cancel button exits the script return else ; ok button runs the custom url in firefox run, "%firefoxpath%\firefox.exe" "http://texashistory.unt.edu/search/?q=%searchstring%&t=dc_title" return figure 10: portal to texas history title search script. when the user triggers the script, an input box appears that accepts a value for the  searchstring  variable.  after the user clicks the ok button, the script loads the desired search results page in firefox as long as the  firefoxpath  variable is set to the correct directory.  if the user wishes to edit a particular record, the script in figure 11 quickly opens the dashboard editing interface by inserting  edit.  at the beginning of the record’s url in the firefox address bar.  a similar script removes  edit.  to return to the public display after the user has made and published changes to the metadata. ; hotkey: alt + z !z:: ; focus on the address bar send, {ctrldown}l{ctrlup} sleep, 50 ; move cursor to start of url send, {home} sleep, 50 ; add "edit." to url send, edit. sleep, 50 ; load record in dashboard editing interface send, {enter} return ; exit script figure 11: autohotkey script to activate the portal to texas history dashboard editing interface. automated metadata enhancement:  selenium ide selenium ide is a free and open source add-on for the firefox web browser.  it is primarily used by the web development community to perform automated testing of web applications.  selenium ide provides an integrated development environment in which to create, debug and run custom scripts that automate actions in the browser window.  users write or record scripts in the selenium ide window and use standard play controls to run single scripts, called test cases, or groups of scripts, called test suites.  the selenese syntax, encoded as an html table, sends commands to the browser that act on specified web page elements in sequence.  see table 1 for a list of common selenium ide commands. table 1: common selenium ide commands. command function usage assertvalue tests an element’s value. verify that a page element has a specific value. accepts regular expressions. click performs a mouse click. click links, buttons, and check boxes. clickandwait performs a mouse click and waits for the page to load. execute the next command in the script after the new page loads. close closes the current tab or browser window. place at the end of a script to enable automated batch editing with a test suite. keypress sends a key press. delete or add characters in a text input element.use with setcursorposition. select selects a value. select a choice from a drop-down menu. setcursorposition places cursor at a specific location in a text input field. 0 = beginning of text string-1 = end of text string store assigns a value to a variable. use the variable with the type command to paste the assigned value in a text input field. storevalue stores existing text in a variable. use the variable with the type command to paste the stored value in a text input field. type enters specified text. populate a text input field.  overwrites existing text. workflow for post-ingest automated metadata normalization selenium ide is an important part of the digital curation toolkit for the unt digital libraries division.  it has proven to be essential in the ongoing processes of improving and maintaining metadata quality for large collections of digital objects.  naturally, as careful repository stewards, we attempt to publish accurate and complete metadata when we upload items to our repository.  we have a number of tools at our disposal that facilitate normalized metadata creation and eliminate mistakes before items are uploaded. sometimes, for a variety of reasons that are usually out of our control, we end up with incorrect or sub-standard metadata for published digital objects.  after objects have been published, our content management system only provides one method for editing metadata:  the object record.  someone must open an object record in a web browser and manually change the information with the editing interface.  when large sets of records contain metadata that must be normalized in order to improve retrieval or meet our repository’s data standards, the single object method is undesirable. the single object paradigm means that editing large sets of records requires shifting staff time away from more important production activities.  however, if the metadata that must be changed is standard across the entire set, we use selenium ide to automate the editing process.  in the best case scenario, a selenium ide operator creates a test suite that automates the editing process for multiple object records.  if a test suite is not feasible, an operator implements a test case that streamlines the editing process for individual object records. a typical metadata editing workflow begins by identifying a set of objects that require normalization.  if the set is large and the required changes are the same for all records, an operator creates a selenium ide script that performs the changes, publishes the new metadata, and closes the browser tab.  after testing and debugging to ensure that the script performs correctly, the operator creates a test suite.  using the content management system’s search interface, the operator opens multiple object records as web browser tabs.  finally, the operator runs the test suite.  each script in the test suite works on a tab in the browser window until there are either no more scripts in the suite or no more tabs in the browser.  the operator repeats the process until the entire record set is normalized. selenium ide use case:  newspaper titles on the portal to texas history the unt libraries (untl) metadata guidelines specify that the main title of an object is the title as it is printed on the front page.  we use a template tool to create standard metadata for large groups of similar objects, such as newspaper issues, during ingest.  in some cases it is difficult to notice minor changes in a newspaper title during the quality control process.  after the objects have been uploaded to the portal to texas history, however, we can easily view changes in a newspaper title by browsing through the thumbnail images created during the ingest process. if we identify any title changes that are not reflected in the template-produced metadata, we use selenium ide to quickly bring our metadata in line with the untl standard.  what follows is a step-by-step html code breakdown of a newspaper title script which adds the word “the” to the beginning of a newspaper title: test that the original title is what we expect it to be with the  assertvalue  command.  adding tests to our scripts ensures that we avoid making inadvertent changes to the metadata; the script will stop if the value does not match.  the field contains information that is unique to each record, such as volume and issue numbers, so we use a regular expression to check the existing value from the beginning of the string. <tr>     <td>assertvalue</td>     <td>//div[@id=’main’]/div/div[2]/div/input</td>     <td>regexp:^north texas daily</td> </tr> because the original title contains unique information that we wish to retain, we use two variables to combine the existing text with the new text.  create the first variable called  newtext  with the  store  command. <tr>     <td>store</td>     <td>the</td>     <td>newtext</td> </tr> create the second variable called  originaltitle  for the existing text with the  storevalue  command. <tr>    <td>storevalue</td>    <td>//div[@id=’main’]/div/div[2]/div/input</td>    <td>originaltitle</td> </tr> paste the stored text from both variables, with a single space between them, into the input field with the  type  command. <tr>    <td>type</td>    <td>//div[@id=’main’]/div/div[2]/div/input</td>    <td>${newtext} ${originaltitle}</td> </tr> save the new metadata by clicking the publish button with the  clickandwait  command, which waits for the new page to load before executing the next command. <tr>    <td>clickandwait</td>    <td>name=publish</td>    <td></td> </tr> close the tab with the  close  command to allow the next script in the automation suite to work on the next tab in the browser window. <tr>    <td>close</td>    <td></td>    <td></td> </tr> figure 12:  newspaper title script test suite. multiple instances of a newspaper title script combined in a test suite (figure 12) automatically add the word “the” to the beginning of the main title field for the corresponding number of digital object records loaded in browser tabs.  in this manner we can quickly edit large sets of records and avoid the inevitable errors introduced during manual data entry. the dspace demo repository and flickr the dspace demo repository (http://demo.dspace.org/xmlui/) provides a good sandbox for exploring the possibilities of automated metadata editing with selenium ide.  log in as an administrator and use selenium ide’s record feature to create scripts.  for example, the script in figure 13 adds two subject keywords to any item record. figure 13:  dspace demo repository script to add subject keywords. flickr is another resource for experimenting with selenium ide.  the script in figure 14 copies the existing text in the title field, pastes it in the description field, and adds a new title.  this script is useful for quickly adding descriptive titles to photographs uploaded with a non-descriptive, camera-assigned title (e.g., photo-0123.jpg).  as long as the description field is empty, the script will add a new title and retain the original title without deleting any information.  the script can easily be modified to stop if it encounters existing text in the description field.  simply add an  assertvalue  command with a blank value targeted at the description field (//div[@id=’meta’]/div/textarea) before the  type  commands. figure 14:  flickr title script in selenium ide’s table view. considerations for automated digital curation autohotkey and selenium ide are powerful tools for working with digital files and descriptive metadata.  these and other automation tools assist digital curators with their daily workflows so that they can produce better digital collections.  when considering whether or not to implement automation tools, it is important to be mindful of the role of metadata and the importance of digital objects for the scholarly and cultural heritage communities.  high quality descriptive metadata fosters trust among a digital repository’s users, and digital collections often serve as preservation repositories for unique artifacts.  accordingly, there are a number of pitfalls to recognize and avoid while working with automation tools for digital curation. producing and editing large amounts of descriptive metadata is a time-consuming task that requires careful attention to detail.  technologies such as autohotkey and selenium ide can facilitate metadata capture and editing activities by automating repetitive workflows.  however, they should be used as complements to, rather than substitutes for, careful evaluation of information resources and the metadata that represents them by skilled employees.  furthermore, if used improperly, automation tools can introduce metadata errors just as quickly as they can correct them and also have the potential to produce unexpected results while working with files.  when making the decision to implement automation tools, digital curators should proceed with caution, maintain backups of important files, and test tools thoroughly before using them in production. conclusion large digital collections present challenges when producing and maintaining descriptive metadata and managing files for digital objects.  successful digital curation strategies involve mechanisms for both preand post-ingest metadata normalization and digital object quality control. automated workflows with autohotkey and selenium ide improve efficiency and accuracy during the timeand labor-intensive data entry and file management processes. with autohotkey, digital curators can create custom tools suited to their unique needs.  simple autohotkey scripts make large, one-off projects much easier, and complex scripts provide an alternative to the command line for batch operations.  autohotkey micro-applications bundle scripts for frequently-performed file management and data entry tasks in order to streamline digital curation workflows.  the autohotkey scripting language is relatively easy to learn, and it provides advanced options for experienced programmers. from institutional repository platforms such as dspace to commercial content hosting sites such as flickr, selenium ide can be used to enhance metadata in any content management system with a web-based editor.  selenium ide scripts provide intuitive batch metadata editing capabilities with significant run time oversight because each object record must be loaded as a tab in a web browser.  selenium ide is a highly recommended addition to the digital curation toolkit for any institution that serves and maintains a large amount of metadata content. references alemneh, d. (2010). metadata quality assessment: a phased approach to ensuring long-term access to digital resource. proceedings of the american society for information science and technology 46(1). retrieved from http://onlinelibrary.wiley.com/doi/10.1002/meet.2009.1450460380/pdf autohotkey documentation. (2012). retrieved from http://www.autohotkey.com/docs/ phillips, m. (2013). metadata analysis at the command line. code4lib journal (19). retrieved from http://journal.code4lib.org/articles/7818 selenium ide documentation. (2012). retrieved from http://docs.seleniumhq.org/docs/02_selenium_ide.jsp sumner, t & custard, m. (2005). using machine learning to support quality judgments. d-lib magazine 11(10). retrieved from http://www.dlib.org/dlib/october05/custard/10custard.html the portal to texas history (2013). retrieved from http://texashistory.unt.edu/ unt digital library (2013). retrieved from http://digital.library.unt.edu/ unt libraries’ input guidelines for descriptive metadata (2013). retrieved from http://www.library.unt.edu/digital-projects-unit/input-guidelines-descriptive-metadata weidner, a. (2013). dashboardtools.ahk. retrieved from https://github.com/drewhop/autohotkey/blob/master/dashboardtools.ahk weidner, a. (2013). newspapernotes.ahk. retrieved from https://github.com/drewhop/autohotkey/blob/master/newspapernotes.ahk about the authors andrew james weidner (andrew.weidner@unt.edu) is the project coordinator for new mexico historical newspapers at the university of north texas libraries, where he manages digitization activities for the university of new mexico’s national digital newspaper program grant. daniel gelaw alemneh (daniel.alemneh@unt.edu) is the digital curation coordinator for digital libraries at the university of north texas libraries and an adjunct faculty at unt’s college of information where he teaches indexing, abstracting, and information retrieval. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – editorial introduction: prioritizing the future, collaborating in the present, and archiving the past mission editorial committee process and structure code4lib issue 14, 2011-07-25 editorial introduction: prioritizing the future, collaborating in the present, and archiving the past this is an exciting time for libraries and technology, and libraries have the opportunity now to build strong collaborations that both preserves our rich history and prioritizes our future. this issue shines a light on a unique blending of priorities old and new, detailed analysis of our past, and creative solutions that enhance the mission of libraries. libraries big and small have the chance to impact our communities for the better. come along, it’s going to be a great ride. by tim mcgeary this is an exciting time in the library and the technology arenas.  it seems that every week there are more and more innovative applications and projects screaming for our attention. many of them are worth evaluating to discern the value added to our own libraries.  but the technical and user requirements for applications change so rapidly, application analysis is just as much prioritizing for the future as it is to answer a present problem.  yet we must not lose sight that the future of libraries also includes archiving the past, a unique blending of priorities old and new, which only shines a brighter light on the need of daily collaborations. a perfect large-scale example is google+, which recently opened invitations to a wider user base.  for the first time in my tenure here at lehigh university, the release of an innovative application spawned an immediate, and widespread, energy of collaboration.  within two weeks (which in the world of academia is equivalent to the speed of light), i was sitting in the bookstore cafe with faculty members, program administrators, members of university relations, a member from the athletics department, and an alumnus, all who see potential, and effective, uses of google+ on campus.  the discussion was not centered around whether google+ would overtake facebook, but rather a brainstorming session of the numerous, yet convergent, opportunities we could use google+ at lehigh. the unique challenges each of these groups on campus deal with regularly can be daunting in attempting to find common uses of technology, and with the continuing uncertainty of the economy, it would be reasonable and natural to expect everyone to focus only on their essential tasks, minimizing risk and conservatively waiting out this storm.  but success rarely rewards the status quo, and, as evidenced in this issue of the code4lib journal, these authors believe in the value of their endeavors. web 2.0 and social media/networks have lots of advantages, but attempting to aggregate or simply remember all the various resources we use daily is difficult, if not overwhelming.  and users are moving to mobile devices faster than we can say ‘smart phone’.  two articles in this issue are beginning to tackle pieces of these two vast environments.  in a novel method for creating a distributed, collaborative commenting environment for bibliographic items, rurik thomas greenall discusses a unique approach to adding user comments for bibliographic information to existing platforms, such as library catalogs.  the strength of the described approach is that it increases the number of comments available for display in any local catalog by consolidating comments from multiple sites and by clustering comments at the frbr work level.  the adoption of this service is worth keeping an eye on. mike beccaria describes how paul smith’s college implemented an information service via sms messenging in how to provide live library information via sms using twilio. this small academic library is leading the way in using a new class of ‘cloud-based‘ sms vendors to make simple sms-based services efficient and cost-effective to implement, and have many possible applications in the library environment. as libraries focus on how to continue prioritizing their future, while continuing their mission of making archives and valuable collections accessible, these three articles provide useful examples.  online viewing of high-resolution document images is problematic at best, and downright perplexing to a user at worst.  diva.js: a continuous document viewing interface could be the solution you’ve been looking for.  andrew hankinson, wendy liu, laurent pugin, and ichiro fujinaga discuss diva.js, a multi-page browser-based document viewer designed to present high-resolution digitized document images as a continuous, scrollable item.  to enable a more effective use of scanned text documents, specifically those digitized from microfiche, doreva belfiore presents a perl scripting application to use the open-source module perlmagick to automatically adjust the brightness levels of digitized images in using imagemagick to automatically increase legibility of scanned text documents.  nathan mealey reviews three titles published by “a book apart”, the book-publishing arm of the website a list apart.  mealey details the most important elements of these books, which discuss html5, css3, and content strategy.  these books will become a staple in building effective web sites in the future environment demands of users on desktop, tablet, and mobile devices. no matter how libraries deliver content and services, if our systems don’t store and disseminate data and collections effectively, our users will become frustrated fast.  it is mandatory, then, that we examine how our collections and services fit into the new, and changing, landscapes we all work in.  karen coyle thoroughly details marc21 as data, looking at the forty-five year-old data format with deep analysis, and presents a first attempt to inventory marc21 so it can be transformed into a more effective data format in our growing web-based environments.  demian katz, ralph levan, and ya’aqov ziso’s using authority data in vufind to show that authority data can still be combined with modern discovery in useful ways. this article examines several mechanisms in which the vufind environment provides information to its users that can enhance discovery.  in a similar vein, geographical authority data can be powerful in the right interface.  rick bennett, edward t. o’neill, kerre kammerer, and jd shipengrover discuss a google maps mashup application that allows users to see surrounding locations that are also fast subjects.  mapfast: a fast geographic authorities mashup with google maps details a mapping interface which allows for simple selection of a location, with links to enter it directly as a search into either worldcat.org or google books.  often users will discover materials not readily available in their home library.  in joining an open source community: creating a symphony connector for the xc ncip toolkit, michelle suranofsky discusses lehigh university’s decision to adopt the extensible catalog ncip toolkit for its ncip service for their intra-consortium borrowing system, including the technical details about building a connector for the sirsidynix symphony ils. finally, gary browne discusses web-based software integration for dissemination of archival images for the frontiers of science illustrated comic strip collection at the university of sydney.  the frontiers of science illustrated comic strip of ‘science fact’ ran from 1961 to 1982, syndicated worldwide by over 600 newspapers.  their aim was to create a website that could disseminate these comic strips to scholars, enthusiasts and the general public, while enabling users to search and browse through the images simply and effectively, with an intuitive and novel viewing platform.  the unique subject of the collection notwithstanding, a most fascinating aspect of this project was the rapid development cycle of six weeks to deliver the site on time, as well as provide the university of sydney with a framework for similar projects. it is indeed an exciting time for libraries and technology.  with excellent examples of innovation, analysis, and design, libraries have the opportunity now to be on the cutting edge of technology and service.  we have the perfect landscape to look at the big picture in the communities we serve.  whether it is preserving our rich history or creatively pointing users to discovery paths made rich through excellent data and collaboration, our future is bright.  come along with with us, and help us build even stronger collaborations with our communities.  we don’t have to wait for google to do it first.  we can join together and make it even better. subscribe to comments: for this article | for all articles one response to "editorial introduction: prioritizing the future, collaborating in the present, and archiving the past" please leave a response below: code4lib journal, n° 14 « pintiniblog, 2011-07-26 […] editorial introduction: prioritizing the future, collaborating in the present, and archiving the pas… […] leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – supporting open access, integrating distributed research platforms, and building a research information management platform mission editorial committee process and structure code4lib issue 53, 2022-05-09 supporting open access, integrating distributed research platforms, and building a research information management platform academic libraries are often called upon by their university communities to collect, manage, and curate information about the research activity produced at their campuses. proper research information management (rim) can be leveraged for multiple institutional contexts, including networking, reporting activities, building faculty profiles, and supporting the reputation management of the institution. in the last ten to fifteen years the adoption and implementation of rim infrastructure has become widespread throughout the academic world. approaches to developing and implementing this infrastructure have varied, from commercial and open-source options to locally developed instances. each piece of infrastructure has its own functionality, features, and metadata sources. there is no single application or data source to meet all the needs of these varying pieces of research information, many of these systems together create an ecosystem to provide for the diverse set of needs and contexts. this paper examines the systems at pennsylvania state university that contribute to our rim ecosystem; how and why we developed another piece of supporting infrastructure for our open access policy and the successes and challenges of this work. by daniel m. coughlin, and cynthia hudson vitale introduction according to the registry of open access (oa) repository mandates and policies, within the united states there are over 80 institutionally based open access mandates and policies [1]. when pennsylvania state university passed their oa policy in january 2020, it was critical to build upon past experiences and support the penn state community in policy compliance with streamlined infrastructure and services. penn state’s goals in passing and supporting the oa policy were multi-faceted. they wanted to: 1) increase the amount of open access content available worldwide; and 2) ensure compliance with the policy was as simple as possible for penn state authors. like many campus-based oa policies, the penn state oa policy gives penn state the non-exclusive right to make an electronic copy of the author’s accepted manuscript available to the public [3]. authors are expected to provide this copy to the penn state university libraries no later than the date of publication . to note, authors do have the ability to obtain a waiver from this policy. in order to obtain a waiver a researcher can fill out a form that contains information about the publication and submit. upon submission all requested waivers are granted. around the same time that penn state passed its oa policy, the penn state information technology and marketing & communications departments began requesting author bibliometric data in order to populate faculty profiles, leveraging existing databases and information available through research information systems. the need to make the implementation of an open access policy with little overhead to our researchers, and the desire of other units on campus to access biographical and scholarly information from our faculty to populate these profiles had some commonality. in both use cases we were providing access to data from similar existing databases (most prominently digital measures, and pure), there was frustration to access multiple apis (application programming interfaces) from teams on campus, and neither system had a complete picture of the scholarly output at penn state. this scenario led us to develop our own system to mitigate these frustrations and solve these use cases. research information systems are not new or novel to the higher education ecosystem. oclc defines rim as, “the aggregation, curation, and utilization of metadata about research activities” in which “rim systems collect and store metadata on research activities and outputs such as researchers and their affiliations; publications, datasets, and patents; grants and projects; academic service and honors; media reports; and statements of impact.” [3] perhaps surprisingly, the term used to describe rims have evolved over the years or can mean different things in different geographic locations. these varying terms include university knowledge base, current research information systems (cris) [4], institutional information system (iis), and research information management systems (rim), among others. while there may be slight differences between each of these, the basic premise to aggregate research and make connections among entities remains consistent. within higher education university libraries have often been the leaders in developing and implementing these types of research information tools [5]. the 2018 oclc research and eurocris report practices and patterns in research information management conducted a global survey on research information management practices – and found libraries to be key partners in rims especially in terms of “support for open access, metadata validation, training, and research data management”. in fact the survey found that over 91% of respondents listed “supporting institutional compliance for open science policies” as an extremely important, important, or somewhat important reason for developing a rim system. this survey also found that institutions frequently used several systems to support research information workflows, both open-source, locally-developed, and commercial in nature. local landscape of rims not unlike many institutions of higher education, the pennsylvania state university (penn state) has purchased or locally developed many solutions and tools to find and aggregate research information about faculty on their campus. penn state has a vendor product for faculty activity reporting (watermark’s digital measures), searching for experts (elsevier’s pure), and has purchased bibliographic data from clarivate, and has an api subscription for reading and writing content to faculty orcid profiles. in addition, the university has a number of locally developed solutions, leveraging open access frameworks, for our data and open access repository (scholarsphere), and our graduate school electronic theses and dissertations to name a couple. penn state’s researcher metadata database (rmd) integrates the data from these systems, and others, in order to power the workflow providing researchers an easy way to comply with the university’s open access policy. penn state has used watermark’s digital measures to aggregate faculty activity for over a decade with various academic units. information gathered in this system is used for the promotion and tenure process and creates a university dossier. for example, digital measures has publications, grants, courses taught, service, presentations, etc. that represent the scholarly activity of a faculty member at penn state. over the years we have onboarded nearly all the colleges and campuses to use this system for faculty annual reviews and university dossiers. this system is largely a manual one for data entry among faculty and support staff, and where possible we’ve created integration points for automating this data entry. as a system of record for faculty promotion and tenure, the data is considered to be high quality and annually reviewed and updated. however, this data is not complete. not all faculty members within a department may use digital measures, because they may have already completed their career promotion and tenure process, or their department may not require annual reviews. though the data is of high quality, and we have good coverage of our faculty, the system is not comprehensive. elsevier’s pure was purchased as a solution to solve the problems related to searching for researchers at penn state based on areas of expertise. the data in pure is essentially backed from another elsevier product, scopus, and allows a search front end to sit on top of this data for faceted search results. the data from scopus that powers pure, is classified into four broad subject areas: life sciences – 15.4% of titles, physical sciences – 28% of titles, health sciences – 30.4% of titles, and social sciences & humanities – 26.2% of titles [6]. while neither digital measures or pure was a complete picture of all faculty activity, they both provided a valuable source of information on the output of faculty at penn state. the combination of the oa policy and a clear need for a mechanism to aggregate penn state author research demanded a common database of this type of information. specifically, penn state developed an in-house solution that leveraged existing databases and infrastructure to build a local research information management system (rims), named the researcher metadata database. our goals in this project were to: make this process repeatable so we could continue to populate our database over time. provide access to the data for individual colleges, and campuses via a restful api to populate faculty profile and department web pages. have a dashboard that is able to run reports for common data requests. power our open access workflow to increase the footprint of openly accessible research at penn state. open access workflow building an oa policy compliance workflow that placed minimal burden on penn state authors was critical. ultimately, the team developed four mechanisms to penn state oa policy compliance: uploading an open access document as part of the annual process in digital measures to support the open access policy. uploading it to an open access repository and having that discovered via open access button following our workflow from rmd that asks a researcher to upload an open access copy of an article that previously was not openly accessible publication a researcher, unsolicited, uploads their article to scholarsphere and keeps the permissions to make it openly accessible. considerable discussions and workflows were examined to ensure an implementation with minimal complexity to the researcher, while also being a sustainable method for support staff. for example, allowing researchers to reply to an automated email with an attached document would be extremely simple; however, if there are multiple publications it could make sorting out which paper belongs to which citation unnecessarily difficult. ultimately, the simplest workflow was to email researchers a listing of publications that are applicable to the open access policy, which then linked to our researcher metadata database. researchers may submit a paper to our institutional repository, provide an existing open access url for publication, or submit a waiver to the policy. now that the rmd has this updated information it is able to determine that researchers should not be asked about these papers again as we remove it from our list of publications that is not openly accessible. figure 1 displays the workflow from a number of the systems discussed, in and out of rmd as well as the attributes or metadata fields that are being sent to the system. if a publication does not have an openly accessible url, an automated email is sent to the researcher requesting a copy that will then be deposited into our institutional repository, scholarsphere. figure 1. data flow of penn state’s researcher metadata database. to date, there are over 300k publications records in the researcher metadata database (rmd), and over 65% of those have a digital object identifier (doi). open access button provides access to their database of millions of open access articles by way of doi query. the initial work in may of 2020 found that nearly 20% of penn state research is openly accessible, since then that number has grown to 34.5% as of july 2021. open access button provides the rmd system the insight to know what publications have openly accessible urls based on penn state’s policy and the publication date of the work and if the work does or does not have an openly accessible link. orcid api subscription another mechanism or tool used to reduce faculty burden in meeting the penn state oa policy was through the use of the orcid api. orcid provides a persistent digital identifier (an orcid id) that an author owns and controls, and that distinguishes them from every other researcher. an author can connect their id with their professional information — affiliations, grants, publications, peer review, and more. additionally, the id may be used to share information with other systems, ensuring they get recognition for all their contributions, saving time and hassle, and reducing the risk of errors. one of the first ways in which penn state leveraged the orcid api was in connection with a user’s local, penn state access account. if the user does not have an orcid id, then the application will create an orcid id and the subsequent link to the penn state access account. additionally, when linking the accounts, the user gives access to penn state to update their orcid records with works, grants, employment, etc. more recently, we connected digital measures to the orcid api. in this connection, digital measures can read content from orcid and import that content, but it cannot write content that is stored in digital measures. this limitation makes it useful for a smaller fraction of our faculty that are actively updating their orcid record. our difficulty here was again in creating an easy way for faculty to get a large amount of their content into orcid. orcid integration as we worked with other units on campus to discuss rmd, the office of the senior vice president for research approached us to find out if we could help with the pending nsf grant requirements, specifically the mandate for sciencv, a researcher profile system for those who seek federal grants, on bio sketches and current and pending grants. national science foundation (nsf) grant requirements now allow faculty to fill out a downloadable pdf for their bio sketch or they can import content that exists in their orcid profile. however, many faculty do not have content in their orcid profile, and either do not have an orcid id or have not linked their orcid id with their penn state access account. rmd leveraged the orcid api to push data from rmd to orcid at the request of the researcher. now, on the same page that provides an penn state author the ability to upload content to the institutional repository, scholarsphere, the researcher can also push their citation to orcid (figure 2). additionally, from a separate screen, a faculty member can push their organization to orcid as well. through rmd a faculty member can grant permission to rmd to update their orcid profile, and once that authorization is complete, they can click a button to add these records to their orcid profile. once these works are in their orcid profile, they can be imported into sciencv and linked into their nsf bio sketch. figure 2. researcher metadata database orcid id integration to push works to an orcid profile. the selected fields (researchers, publications, and organization history) are relevant to the nsf grant requirements, and although it is not possible to directly integrate with sciencv, this option allows a researcher the capability to push the buttons to upload citations rather than manually enter them into orcid or a pdf. additionally, if other large granting agencies such as national institute of health (nih) follow a similar pattern then the application will be in a good position to provide a solution. providing access an additional goal of the rmd was to provide access to our data. there are many other colleges and campuses at penn state that would like to access this information for the purpose of faculty and department web pages (figure 3). there is a perpetual challenge faced by the marketing and communications departments of these units to keep information up to date. given that one of the sources for our data is from digital measures, our faculty use this system for the process of annual review and promotion and tenure, so the content in there is likely to be high quality and up to date. integrating this data with content from other systems, such as open access button, the rmd has been able to not only provide citations, but also provide links to the actual content of a citation that is not behind a paywall. this increases the exposure of our researcher’s scholarship and their affiliated department, institute, lab, etc. figure 3. penn state’s food science department using data from the researcher metadata api to publish citations and links to their open access copies. challenges throughout this work two of the largest challenges have been the duplication of data and disambiguation of researchers. because content comes from so many different sources, there is bound to be a subset of that data that has already been entered into many of these systems. in fact, a large source of frustration for researchers is wondering why they may have to enter data into one system if it already exists in another. in many of the more recent published academic journal articles in rmd there are identifiers for the papers (digital object identifiers, doi) that can be automatically scanned and matched for sameness. however, if the information is manually entered into a system such as digital measures, and a user does not enter a doi it is more difficult to programmatically determine if the paper is a duplication of another. this scenario has created a two-part process to de-duplicate papers. first, the system applies a matching algorithm to do an initial pass of similar or like metadata for the article’s titles, authors, and dois to merge duplicates. then, the remaining papers are manually reviewed and de-duped. our data indicates that between 2015-2019, there was a rise in papers with dois from 63% to 78%; however, this means 20+% of papers still do not have a doi. at eight and ten thousand papers being produced at penn state in a year, this amounts to potentially thousands of duplicates annually. publications that we have imported via digital measures and pure are linked to a penn state researcher. this helps disambiguate these publications because we know they belong to a particular penn state researcher. however, in the case of clarivate, the data doesn’t have a link back to the penn state author. the system can search for matches that may exist in the database and assume based on matching up both sets of data. in the case where the system is unable to determine the penn state author with certainty, the system will guess and make the publication visible to only the penn state author. the researcher can either confirm or deny that paper within the same interface that allows them to push content to orcid and scholarsphere. these two issues reflect the importance of identifiers in accurately identifying the research output for an institution. it’s important to have a universal identifier for the researcher, such as orcid, as well as an identifier for the output, such as a doi. a lack of these identifiers creates a tremendous amount of manual de-duplication. the initial de-duplication process required nearly 40,000 citations to de-duplicate. once this problem is solved, future updates to the data will not produce anything more than a couple hundred at most. the problem is ongoing, but after an initial challenge, it is much more manageable. future developments there are several areas to (1) enhance rmd, (2) further reduce penn state author burden, and (3) support the compliance mechanism for the penn state oa policy moving forward. as previously mentioned, it was integral to provide as many ways for a penn state author to make their work openly accessible, while also automating these modes of compliance, where possible. for example, if a penn state author uploads their openly accessible article into digital measures, it is a manual process for the compliance team to deposit a copy into scholarsphere and set it to open access. future developments will focus on automating that workflow. alternatively, the process to upload a publication to scholarsphere via rmd now takes place in a more seamless way and eliminates the need for support staff to update urls manually. another future development will be to create an oai-pmh endpoint that will allow open access button to download content from scholarsphere. and finally, rmd will be able to write links to oa articles back into digital measures and have the university dossier not only print researcher’s citations, but also links to the content. ultimately, the researcher metadata database has a process that is repeatable so that information can be updated over time. the system provides access to the data via a standardized api and can run reports for common requests on data for internal/university stakeholders. there is a workflow that will increase the footprint of openly accessible research at penn state and has provided a starting point for knowing what is already open and sharing that content via faculty and department websites. all of this has enabled the libraries to be in a position to help reduce faculty burden in other areas, such as integrating with orcid to publish information that can be used for nsf grant proposals. acknowledgements: nicole gampe, alex kiessling, laura deantonio, kevin fraccalvieri, justin patterson, adam wead, brandy karl, ana enriquez, seth erickson, briana ezray, eric durante, natalie nash, chet swalina, scott woods about the authors daniel coughlin (dmc186@psu.edu) has worked in information technology for almost 20 years and nearly half of that in libraries. he has spent many years as a software developer, primarily working on web applications. currently, dan is the department head for libraries strategic technologies at university libraries, penn state and in his free time is typically trying to chase or find his dog, millie, in rothrock state forrest. cynthia hudson vitale (cvitale@arl.org) is the director of scholars and scholarship at the association of research libraries. in this role she collaborates with key partners, allies, and joint ventures to promote arl’s broad mission of open, equitable scholarly communication, information stewardship, and publishing. prior to joining arl she worked for over 15 years in research libraries building and managing services for computational research and open publishing. notes [1] see: http://roarmap.eprints.org/cgi/search/archive/advanced?screen=search&dataset=archive&country=840&policymaker_type=research_org&policymaker_name_merge=all&policymaker_name=&policy_adoption=&policy_effecive=&mandate_content_types_merge=any&apc_fun_url_merge=all&apc_fun_url=&satisfyall=all&order=policymaker_name&_action_search=search [2] see: https://policy.psu.edu/policies/ac02 [3] bryant, rebecca, anna clements, carol feltes, david groenewegen, simon huggard, holly mercer, roxanne missingham, maliaca oxnam, anne rauh and john wright. 2017. research information management: defining rim and the library’s role. dublin, oh: oclc research. https://doi.org/10.25333/c3nk88. [4] sivertsen g. (2019) developing current research information systems (cris) as data sources for studies of research. in: glänzel w., moed h.f., schmoch u., thelwall m. (eds) springer handbook of science and technology indicators. springer handbooks. springer, cham. https://doi.org/10.1007/978-3-030-02511-3_25 [5] dempsey, lorcan. “research information management systems: a new service category”, lorcandempsey.net, october 26, 2014. https://www.lorcandempsey.net/orweblog/research-information-management-systems-a-new-service-category/ [6] scopus content coverage guide. https://www.elsevier.com/__data/assets/pdf_file/0007/69451/scopus_contentcoverage_guide_web.pdf. accessed july 31, 2021. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – editorial introduction: people mission editorial committee process and structure code4lib issue 32, 2016-04-25 editorial introduction: people by meghan finch two issues ago, coordinating editor carol bean identified a focus on data, in our profession and in the issue 30 articles, and also recognized that as information professionals, it goes beyond just the data to the conventions and standards necessary for working with data. [1] i’d like to offer a similar sentiment […] by meghan finch two issues ago, coordinating editor carol bean identified a focus on data, in our profession and in the issue 30 articles, and also recognized that as information professionals, it goes beyond just the data to the conventions and standards necessary for working with data. [1] i’d like to offer a similar sentiment for issue 32: it’s all about the technology, but it’s also about the people involved with the technology. while all of the articles in this issue focus on technology and innovation, often through the development of tools or processes, several also discuss the personal experiences of those working through these efforts. in how we went from worst practices to good practices, and became happier in the process, french, kayiwa, lawrence, gilbertson, and lohrey address the personal satisfaction of their team members following the implementation of new practices by inviting each to share their impressions. jimmy ghaphery, emily owens, donna coghill et. al share the reactions of their team to an exploration of a rocky discovery layer implementation and their work toward improvement, both of the tool and the regard the team had for it, in building bridges with logs: collaborative conversations about discovery across library departments. dianne dietrich, julia kim, morgan mckeehan, and alison rhonemus’s how to party like it’s 1999: emulation for everyone strikes the perfect balance of personal experiences and step-by-step exploration of emulation. over half of the articles in this issue have four or more authors, and individual voices are given the opportunity to share as part of a larger experience. articles like nick ruest and ian milligan’s an open-source strategy for documenting events: the case study of the 42nd canadian federal election on twitter explore the people involved in technology more broadly, capturing the voices of a nation (on twitter) during an election, all while balancing the preservation of those stories with the privacy of participants. alex caro and chris markman’s measuring library vendor cyber security: seven easy questions every librarian can ask similarly addresses privacy, developing a guide for libraries of all sizes to review their technology infrastructure. even when we aren’t explicitly talking about the people, people are always present in our innovations. ian lamb and catherine larson’s shining a light on scientific data: building a data catalog to foster data sharing and reuse supports the current efforts of scholars to share data and reproduce previous experiments, while jonathan bradley, neal henshaw, liz mcvoy et al.’s creation of a library tour application for mobile equipment using ibeacon technology helps introduce students to the library while improving the maintenance process of the application for the library team involved. french, kayiwa, lawrence et. al refer to the personal narratives they present as “‘feel good’ testimony”[2], but that does not diminish how significant it is that many of the articles that we share in this issue examine not only hard quantitative data and processes, but also look to find the happiness in those working on those projects. it’s a good reminder of why we keep innovating. for all of us. issue 32 overview an open-source strategy for documenting events: the case study of the 42nd canadian federal election on twitter by nick ruest and ian milligan how to party like it’s 1999: emulation for everyone – by dianne dietrich, julia kim, morgan mckeehan, and alison rhonemus how we went from worst practices to good practices, and became happier in the process by amanda french, francis kayiwa, anne lawrence, keith gilbertson and melissa lohrey shining a light on scientific data: building a data catalog to foster data sharing and reuse by ian lamb and catherine larson creation of a library tour application for mobile equipment using ibeacon technology by jonathan bradley, neal henshaw, liz mcvoy, amanda french, keith gilbertson, lisa becksford, and elisabeth givens. measuring library vendor cyber security: seven easy questions every librarian can ask by alex caro and chris markman building bridges with logs: collaborative conversations about discovery across library departments by jimmy ghaphery, emily owens, donna coghill, laura gariepy, megan hodge, thomas mcnulty, and erin white references [1] bean, carol. editorial introduction: it’s all about data, except when it’s not. http://journal.code4lib.org/articles/11072 [2] french, amanda, kayiwa, francis, lawrence, anne, gilbertson, keith and melissa lohrey. how we went from worst practices to good practices, and became happier in the process. http://journal.code4lib.org/?p=11398 subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – the devil’s shoehorn: a case study of ead to archivesspace migration at a large university mission editorial committee process and structure code4lib issue 35, 2017-01-30 the devil’s shoehorn: a case study of ead to archivesspace migration at a large university a band of archivists and it professionals at harvard took on a project to convert nearly two million descriptions of archival collection components from marked-up text into the archivesspace archival metadata management system.  starting in the mid-1990s, harvard was an alpha implementer of ead, an sgml (later xml) text markup language for electronic inventories, indexes, and finding aids that archivists use to wend their way through the sometimes quirky filing systems that bureaucracies establish for their records or the utter chaos in which some individuals keep their personal archives.  these pathfinder documents, designed to cope with messy reality, can themselves be difficult to classify.  portions of them are rigorously structured, while other parts are narrative.  early documents predate the establishment of the standard; many feature idiosyncratic encoding that had been through several machine conversions, while others were freshly encoded and fairly consistent.  in this paper, we will cover the practical and technical challenges involved in preparing a large (900mib) corpus of xml for ingest into an open-source archival information system (archivesspace). this case study will give an overview of the project, discuss problem discovery and problem solving, and address the technical challenges, analysis, solutions, and decisions and provide information on the tools produced and lessons learned.  the authors of this piece are kate bowers, collections services archivist for metadata, systems, and standards at the harvard university archives, and dave mayo, a digital library software engineer for harvard’s library and technology services.  kate was heavily involved in both metadata analysis and later problem solving, while dave was the sole full-time developer assigned to the migration project. by dave mayo and kate bowers introduction what might seem like a straightforward but sizable data migration can become far less than straightforward when large amounts of legacy data are involved. both human solutions – like communication and data-driven but fundamentally subjective weighing of possible outcomes – and technical solutions were needed to meet the challenges posed by extreme variations in the data under migration. in order to understand some of the discussion in this article, a bit of context is necessary. the data we are talking about is metadata in the form of inventories of archival collections. these are also known as “finding aids.” finding aids are pathfinder documents containing collection-level information and, usually, hierarchically nested descriptions of the subordinate components of an archival collection. each inventory may have zero to thousands of these nested components, and the nesting is theoretically infinite, although the deepest nesting at harvard was seven levels. some elements of finding aids are prose-heavy, such as lengthy family histories, and some are more suitable to being parsed and atomized, such as physical measurements and counts. in harvard’s case, the challenge was to move more than 6000 xml-encoded archival finding aids containing two million hierarchically nested descriptions of archival collection components from a native xml database to archivesspace, which is a metadata management system run against mysql. harvard university adopted archivesspace (aspace) in 2015, and decided to move its enormous trove of finding aids from a home-grown backend solution to aspace. this home-grown backend was called the online archival search information system (oasis), and its platform was a tamino xml server. originally designed in 1995 (oasis steering committee, 1996), the last significant changes were made in 2002 (oasis steering committee, 2003). key to the replacement of oasis with aspace was the migration of the descriptive metadata in the finding aids. aspace has an xml importer for the encoded archival description (ead) markup that harvard archivists use when encoding these inventories. this may give the appearance that ingest of the metadata would be fully supported by the new system. however, issues the migration project encountered include scale, variety of the legacy data and the fact that aspace is far more constrained than ead, and surprises from the importer. scale and variety of legacy data to understand the nature of harvard’s legacy data, it will be necessary to place yourself in the shoes of an archivist in the year 1995. things are looking up. you are feeling grateful to have escaped from an elevator just before celine dion started singing for the maybe 700th time that day, and you are excited by the possibilities of your newly-installed browser, netscape, and the miraculous search engine alta vista. you envision replacing a little-used gopher-browsable list of online finding aids with a full-text-searchable body of these heretofore unyielding texts. as an archivist, you are particularly interested in providing one-stop, full-text searchability to finding aids dispersed in about 40 different archives and libraries all around your campus. right now, most of these are consulted in hard copy in 40 different locations around campus. the big problem seems to be just getting all that text online and searchable as fast as possible. you meet with colleagues to focus on a question: do you quickly provide a high volume of text to search, or does that niggling sense that at some point in the future you would regret putting mere html or word-processed documents online convince you to do more? some of your colleagues advocate the former, since this would be low-cost effort requiring only retyping exactly what is available in hard copy. as a bonus, recently-created inventories are probably still hanging around on diskettes somewhere, so that text is shovel-ready. others argue that the group should, at a minimum, use marked-up text, like tei. in fact, an sgml dtd for archival inventories is right now under development. like many committee decisions, this one chooses a middle ground. the online finding aids will be marked up text, but there will be little central control over how the sgml standard should be implemented by individual libraries and archives. this leaves them free to contribute even if the markup is not ideal. in the end, the digital finding aids project at harvard began using the beta version of an sgml dtd called encoded archival description (ead), with the goal of making contribution as easy as possible by making encoding requirements very minimal. the minimal requirements for online finding aids at harvard never changed substantially over many years. these requirements were 1) validity against the ead dtd, 2) prohibitions against a few formatting elements in ead such as tables and line breaks, 3) prescribed frontmatter, and 4) presence of a marc record in the harvard integrated library system with exactly the same author-title content in the ead and the marc. adherence to additional guidelines was recommended, but optional. (oasis steering committee, 1999 and 2011) the scale of harvard’s data determined that any programmatic changes would be done across-the-board. while individual repositories had made their own encoding choices over the years and patterns could be discerned, the pattern of choices were nonetheless far too numerous to implement different programmatic transformations based on patterns that were not discernable across the entire 6000+ corpus. the harvard university archives, for example, had nearly 500 finding aids encoded between 1995 and 2016. staff involved in the encoding ranged from highly experienced archivists to external contractors to interns. over the years practices and templates had changed. thus, even within a single repository the interpretation of the standard and encoding decisions were not predictable. multiplying this situation across all the repositories produces even greater variation. thus, programmatic transformations that focused on a subset of the corpus of ead were rejected as impractical, and instead all transformation scenarios had to be applicable to the entire corpus. problem statement and overall goals archivesspace constraints compared to ead the internal data model of archivesspace varies from and is far more constrained than ead. these differences presented issues of either “missing” data or data in ead that had no home in aspace. to be successful, the migration would need to leave no essential data behind. the first and definitive goal was to make sure that each of the finding aids would ingest, and a close second was to make sure that all the data in the finding aids successfully crossed into the new system. in addition, there was a strong desire to leave no meaning behind, thus a third goal was to ensure that metadata went from its ead element into the analogous field in aspace or at least a good second choice. while data migration was the goal of the project, by no means was it the only goal. success was also to be measured by the satisfaction of the archivists who made, maintained, and used the data. to this end, archivists were needed to analyze the existing data and determine how completely and how well the migration had met its goals, and to specify desirable outcomes for particular data elements. technical support for this analysis and specification work was necessary, including help querying the existing xml data and analyzing the resources that would need to be allocated to various programmatic fixes as well as developing an online web “ead checker” so that archivists could understand how well individual ead finding aids would fare in the constraints of aspace. because harvard’s ead had been growing over twenty years with little prescription or proscription, the scale of the variety was remarkable. another cause contributing to this variety was that ead itself made few specific demands. for example, ead requires that a description contain only one of twelve possible pieces of information: an abstract, or a container, or a link to digital object, or a heading, or the language of the material, or a note, or a creator, or a physical description, or a physical location, or a date, or an identifier, or a title (ead tag library, 2002). by contrast, aspace requires a title and date at the highest level in the hierarchy and either a title or a date at all subordinate levels. in addition, there are data elements in ead that have no home in aspace. for instance, alternate or parallel titles can be provided in ead by repeating the unittitle element with varying @type attribute values, but in archivesspace, there is no current provision for alternate titles, and no conversion in place in the ead importer for such markup. because of these and other differences, the majority of harvard’s finding aids would be rejected by the aspace importer unless they were modified. in all, archivists analyzed seventy-three issues that resulted in three classes of problems: 1) failure to ingest, 2) ingest with data loss, or 3) failure to place data in the optimal field in aspace. the project resulted in the following outcomes: fifty issues were resolved by an ead-to-ead transformation; additionally the issue was reported to repositories for optional modification of their ead data by hand eight issues were resolved by reporting them to the repositories for mandatory modification of their ead data by hand to facilitate ingest to aspace eight issues were resolved by reporting the issue to repositories for optional modification of their ead data six issues were resolved by reporting the issue to aspace which resulted in modifications to the importer one issue was an export issue and was referred to the another group for resolution what determined these outcomes was a combination of the value of the data and our ability to modify it either programmatically or by hand. data that was a requirement for the ead importer or that prevented data loss was of the highest priority, while data parsing and transformation that placed it in the optimal field in aspace was of great but necessarily secondary value. (a full list of issues is in appendix a.) some transformations were relatively simple. because ead had both a generic <note> element and an <odd> element (“other”), but aspace accepts only <odd>, the decision to transform <note> to <odd> prior to ingest was rational. a number of twists were due to the fact that <note> was valid in more places than <odd>, but archivists quickly identified valid locations for <odd> in which the content of former <note> elements would be reasonably placed. the most difficult transformations were our attempts to take free-form textual content and parse it into firm data values. an example of data that might be deemed “missing” by the aspace importer is collection-level physical extent. while harvard’s finding aids might have a very detailed and explicit information, it may not have been tagged in a manner that aspace expected. in fact, aspace recognized only about 50% of our expressions of extent, for example: <physdesc> <extent> approximately 25 cubic feet (75 document boxes; 10,000 photographs) </extent> </physdesc> a data transformation that prepended <extent>1 collection</extent> before the <extent> tags for such values was determined to be the best solution. rejected options included trying to parse these values. difficulties in parsing included conditions where the numeric value was preceded rather than followed by the measurement (e.g. “boxes: 6”) or the unreliability of punctuation as a determinant of count and unit of measurement pairs (e.g. “25 cubic feet (75 boxes (25 half-document boxes; 50 flat boxes)). the measurement “1 collection,” while not really useful, is not misleading. better yet, it is recognizable by archivists as a product of automated processing. in some cases, transformation was impossible and the only resolution was to send reports to repositories and require that they change their ead by hand prior to ingest. such cases included collections lacking titles, for example. a great effort was made to report out on all transformations so that repositories could take the option of changing their encoding prior to migration. one such case were collection descriptions that lacked dates. often these were mere failures to provide tagging around dates which had instead been included as part of titles. however, each of these required a fallback transformation scenario. in the case of dates, the group determined that each collection lacking a date would be given inclusive dates from the year 0 to the year 9999. container management features[1] were introduced to aspace towards the end of harvard’s migration project, so there was little time for a lengthy analysis. in addition, container management is a particularly hands-on and idiosyncratic issue in archives, where storage facilities, storage locations, and shelving varies widely. while we made no programmatic changes to container data, this issue was referred back to repositories to make modifications if they chose, and most indeed elected to do so after hands-on experience with the container management features in an experimental aspace instance. in rare cases, there was simply no solution to offer. a number of repositories included notes explaining the meaning of “controlled access terms,” for example. in this case, there is no home for this content anywhere in aspace and archivists determined it was not essential, so no action was taken and the data was allowed to be dropped during migration. surprises from the importer perhaps the greatest service we did for the community of aspace users was report a number of issues to the aspace team. among these were the failure of the importer to respect the ead standard’s default behavior regarding finding aid content that had no @audience attribute value. for content marked for neither an “internal” nor an “external” audience, ead expects that this content is to be shared, but the aspace importer marked all elements that lacked any @audience value as “internal” (i.e. “not published”). in this case, we were pleased that the project went on long enough for the archivesspace team to provide and to incorporate code contributions that provided alterations to the importer, so that we did not end up with millions of pieces of hidden data that had previously been accessible. these reports were passed to the aspace project team via their issue tracking system, which is accessible to member institutions, and via the issue queue attached to the source code, which they maintain in github. code contributions made by harvard were submitted by dave via github pull requests. technical goals provide support to metadata analyst and archival staff in analyzing our finding aids dave came onto the project after a substantial amount of analysis had been completed, but our corpus was too large, practice too varied, and too few hands comfortable with metadata work were available for it to be exhaustive. additionally, there’s not a rigorous external definition for what a valid ead is in context of archivesspace’s importer. ead is a very permissive standard; many valid ead structures fail to import, or import with omissions or outright errors. the only machine-actionable criteria for an ead file’s correctness with respect to archivesspace is the archivesspace importer’s code itself, so there’s not an easy way to measure success in more than binary terms, i.e. “did it import?” a significant part of development time was spent on aggregating and summarizing errors from archivesspace, as well as on writing xquery and xpath expressions to run over the corpus (pre and post processing). dave had enough library metadata experience to pursue some of this analysis on his own, and also did substantial work in support of analysis by the harvard library’s archivists and metadata librarian. improve ingest process although archivesspace did have support for batch imports via the interface, it was (and is still at time of writing) flawed in several significant respects, one of which was considered a critical flaw that blocked ingest of the corpus. the critical flaw, which several other institutions have encountered, is that archivesspace batch imports halt on the first error encountered in the batch, rather than skipping to the next file in the batch. while this hasn’t prevented use of the tool for small collections, and the batch importer continues to be a part of our workflow for files too large to import via the aspace api, it is clearly not usable with a collection of over 6000 files. enable reporting on progress there was a strong desire from senior and local management to be able to measure progress, even if only approximately. before implementing the tools, the only measure we really had was “does this file ingest successfully without throwing an error, and does it look correct?” since any single error would cause a finding aid to be rejected, and many errors were present through large parts or the entirety of the corpus, this produced a flat graph indicating stasis, even as many individual errors were found and repaired. to produce something like an honest proxy for our progress, a focus on producing an auditable record of problems found and processing actions taken over time was essential. this had a nice synergy with our support and analysis goals; in practice, support and analysis drove more development decisions, and much of the reporting functionality was deferred to manual generation of reports by the developer and archivists. technical decisions error detection the first requirement for migration was a reliable method for detecting errors in the ead files. all of the finding aids started out as valid ead, so this wasn’t an issue of validation; what was needed was a method to describe and report on particular structural or content patterns within the ead files. schematron is a rules-based xml language for making assertions about patterns in xml documents. any pattern describable by xpath can be reported on: for example, here’s the pattern we use to detect extents with non-numeric values: <pattern id="extent-nonnumeric-manual"> <rule context="//*:extent"> <!-'extent' elements --> <assert test="matches(., '^\s*[0123456789.]')" diagnostics="enm-1"> 'extent' element content should not start with non-numeric character except '.'. </assert> </rule> </pattern> it was initially chosen as the mechanism for detecting errors partly on account of the previous work done, and the potential to use it as a “friendlier” error-checking mechanism in the importer. neither of those reasons ended up being particularly relevant, but schematron still ended up being a useful tool for checking errors. preprocessor when it came to actually incorporating our work into the migration process, we had roughly three methods available. we could alter code and submit it to archivesspace for inclusion into the core project, we could create a plugin with the changes, or we could transform the ead files before ingest in ways that avoided causing problems with aspace’s existing importer. while we eventually ended up pursuing all three of these methods, we decided early on to focus on preprocessing our ead files. this decision was based partially on a somewhat negative early evaluation of the archivesspace plugin system and ead converter, and partially in response to broader decisions related to local processes. the cost of coordination with the archivesspace project was also a concern, and continued to be a concern throughout the project. there are a number of attributes that led us to initially decide to avoid focusing effort on the archivesspace converter code. on a surface level, the code is, as far as we were able to tell, sparsely documented, and where documented, is documented in terms of its specific handling of an element or elements. the importer is implemented as a fairly minimal domain specific language over nokogiri::xml::reader and the custom jsonmodel serialization layer defined by archivesspace. most existing plugins implementing converter customizations involved small rewrites of specific element-handling code, without nearly approaching the scale of the customizations desired by harvard. in addition, there were concerns related to the lack of isolation in the design of the archivesspace plugin mechanism, particularly as several system upgrades were expected over the course of the work. the final decision was essentially to do as much as possible in the external preprocessor, and to reserve work on the importer for changes that were not resolvable by altering the ingested ead, or for cases where the converter’s behavior was clearly and unambiguously a risk of data loss or corruption. an external preprocessor was also considered valuable because it allowed local progress to be decoupled to some degree from the archivesspace system. while “what does it do when ingested” was still the final arbiter of correctness, many issues were able to be diagnosed and analyzed strictly from looking at the finding aids as represented in ead. the preprocessor initially allowed for turnaround time of roughly five hours to process the whole collection. while this eventually expanded to twelve hours, as more classes of problems were added to the preprocessor, this was still much, much preferable to the three days that a full ingest of the collection into archivesspace required. as a side-benefit, having the external processor embedded in a web application made it an ideal place to provide self service access to finding aids and reporting. reporting was repeatedly de-prioritized, and thus was never really implemented, but self-service download of finding aids was very well used and had a positive impact throughout the migration process. these functions are directly responsible for the choice to develop the preprocessor in the rails web framework. tools produced for this project schematronium the first step undertaken was to develop an automated way to check all of the finding aids for errors in a reasonable amount of time (no greater than overnight), with solid reference to point of error (we ended up getting xpath location and line number in file, which was considered optimal). our initial survey of the ruby ecosystem revealed a few existing ruby libraries for running schematron – unfortunately, most of them were unsuitable for running xslt 2.0, which (due to regex support in xpath and a few other concerns) was considered preferable to xslt 1.0. the one library that did use an xslt 2.0 processor did so by shelling out to saxon’s command line processor; this was a non-starter because it essentially meant that each file’s processing was preceded by a cold jvm start up, which (in almost all cases) dwarfed the actual execution time of the schematron processing. with over 6000 finding aids, this led to runtimes well outside what was considered reasonable. at this point, dave started hunting for a general purpose xslt-running library, and found matt patterson’s saxon-xslt, which directly used saxon’s java api, and which was well-documented enough to make contributing fixes and improvements reasonable. at this point, dave sank a fair amount of time into working on improvements to saxon-xslt. specifically, he added support for configuring the underlying parser (needed for line number support) as well as a number of convenience methods related to running xpath. he also wrote a small ruby library, schematronium, for running schematron over a file. this library was reused in all succeeding stages of the project, although to date it hasn’t been taken up by archivesspace itself. eadchecker at this point, theoretically, work was ready to begin on the preprocessor. at this point, however, dave proposed and ultimately built a small, single-form web service, the ead checker, to allow archivists and members of the working group to run individual ead files through the schematron on demand. actually writing the service was less than a day of initial development effort, and a few more days throughout the lifetime of the project. the service consists of: a simple form, with a file input for the user’s ead, and two radio selects for setting various options. a help page, whose content is taken automatically from the schematron file. on form submission, the user gets back a list of errors found, in xml or csv formats the ead checker is, in our opinion, one of the better decisions made in the course of our ead migration. for the cost of three days of effort, we substantially broadened the class of people able to do ead testing via the current version of our schematron. additionally, it served as a communication tool. because it used the same schematron as our other tooling, the ead checker help page served as a consistently up-to-date reference to the current state of progress, at least in terms of what problems we could detect and fix in our eads. since the problems in our schematron were primarily general (i.e. corrections required for any ingest into aspace), we’ve exposed the service to the internet at large, at https://eadchecker.lib.harvard.edu. this required additional work to secure the service against various xml parse vulnerabilities. many people are unaware, but most xml parsers, if not all, are extremely vulnerable in their default configuration. for more information, please consult owasp’s xxe page and other security resources. at least twice as much work went into securing the tool against xml parser vulnerabilities as every other part of its construction combined. code for the ead checker is located here. preprocessor despite being primarily a command-line driven xml processor, the preprocessor is implemented as a rails web application, running on a medium amazon ecs instance with 30gib of additional disk space. both information privacy concerns and the development cost of authentication and authorization led us to make the application available only to harvard subnets affiliated with the harvard library. the first step to building the preprocessor was to decide how it was going to manage and identify the finding aids it worked over. our finding aids are uniquely identified by eadid, but it was quickly realized that if we wanted to be able to reliably audit the effects of the preprocessor, we needed to be able to distinguish between different versions of the ead files, so that we didn’t attribute to the tool changes made by staff to the input files, and vice versa. it was also considered necessary to distinguish between different versions of the schematron file, to prevent attributing changes in error counts to the ead improving/regressing which were actually due to changes in the definition of errors. in the preprocessor, these two kinds of files are managed by classes that serve as file registries, schematronfile and findingaidfile. on ingesting a file into the preprocessor, the contents are hashed via the sha1 algorithm, which generates a unique string based on their contents. they are then stored in a directory on the filesystem, named after their sha1 value with `.xml` appended. database records (represented in the system by the classes schematron and findingaidversion) are associated with each file, keyed by sha1, and findingaidversions with the same eadid are connected by a database record and class representing all versions of that particular finding aid. this scheme gives a guarantee that any change to a finding aid or schematron will result in a new object, preserving our ability to audit the schematron’s accuracy over multiple versions of finding aids. as a side benefit, this (somewhat) mitigates the disk space costs associated with storing multiple copies of our large xml corpus, as files that remain unchanged between runs are only stored once on the input side. the schematron file is used directly via xslt for doing the actual checking, but it is also processed on ingest by the system and each individual assertion is represented as an issue, identified by a unique identifier pulled from the @diagnostic attribute. each issue represents one known problem that exists across the ead corpus. there are subsidiary database objects for storing concrete information about individual cases of a problem in individual finding aids, represented as a concreteissue class. it was decided, for reasons of reducing programmer effort, not to implement such a registry for output xml. the output from each run is stored in a subdirectory named for the database id of the run. the application also has a fixes class, which represents the set of repairs that can be applied to fixes. these fixes are read in from a directory on application start up, and are written in a compact ruby dsl developed for this application. as an example, here’s one of the smaller fixes: # add @type='unspecified' to untyped containers fix_for 'ca-1', depends_on: ['noempty-1'] do @xml.xpath("//container[not(@type)]").each do |el| el['type'] = 'unspecified' end end the dsl is fully documented here. the most complicated part of the fixes is that, because they can be order dependent, a small dependency resolver was necessary to ensure that they ran in the correct order. the core functionality of the application is located here, and is essentially a single processing pipeline: input ead is run through the schematron via schematronium, with results stored in the db. these results are used to determine what problems exist in the eads, and each xml file is then run through the fixes associated with each problem. a record of each fix and its status (succeeded/failed) is stored in the db. finally, the altered xml is saved to disk in an output folder. the lion’s share of the effort put into the preprocessor went into four specific areas: the file registries for managing schematron and ead files on disk the dependency management for ordering fixes the methods which actually run the processing over the ead files automated testing of individual fixes in order to make the preprocessor generally useful outside of harvard, an effort has been made to make parts of the preprocessor that are likely to be customized by other institutions pluggable. the schematron, fixes, and tests for fixes are maintained in a separate repository and loaded into the preprocessor on initialization. the preprocessor is available here, and the schematron and fixes developed and used by harvard are located here. ingest scripts like all other institutions surveyed who migrated from an ead-based finding aid collection, we wrote an ingest script using the api, rather than use the built-in batch import functionality in archivesspace. the primary driver of this is the batch importer’s behavior around import error. it fails on the first finding aid with errors in it, and, if the error it hits is a ruby exception rather than a jsonmodel validation error, it fails on the first error in the file, which masks all other errors that occur later in processing that file. there isn’t, as far as we could determine, actually a single “ingest” api endpoint that would handle ead directly. the endpoint that allows resource creation expects data already in the json format understood by archivesspace (and implemented solely in archivesspace). however, mark cooper at lyrasis has written a plugin that accepts an ead file, runs it through the importer, and returns the resulting json representation, or json representing the validation or processing error if the conversion failed. so, we developed an ingest script, originally as a set of bash scripts, which were later rewritten in ruby. the ingest script is located here. it essentially takes each ead file, posts it to the conversion endpoint defined by the plugin, then posted the resulting json to the resource creation api. errors and their causes were recorded in a set of logs, which were then processed by another script. while our ingest script does provide the key required benefit (ingest of batch continues despite failure of individual ead files), the time required for ingest via this method is considerable, averaging between two and three days per complete ingest. much of the code in our ingest script is the result of an attempt to speed up this process; since a good deal of the time of ingest is spent in network transfer, it seemed like a good candidate for parallel processing, where more than one ead file could be uploaded at a time. unfortunately, a race condition in archivesspace (described here) caused failures when more than one resource shared a subsidiary object (agents or subjects, for example) and were uploaded at the same time, and this work ended up producing no real value to the migration project. custom importer plugin while we chose to focus on our preprocessor, as stated above, certain issues with import were not solvable by manipulating the imported ead. in particular, there was no valid ead structure that would fix archivesspace’s handling of: @audience attributes <daogrp> elements <indexentry> elements by and large, when writing code for the import plugin, we also packaged it up as a pull request and submitted it to the archivesspace development team; currently, most of the work in our importer is en route to being included in archivesspace 1.5.2. credit is due to the bentley historical library at umich, whose work our <indexentry> fix is based on. our import plugin is located here. description of our repeated workflow our workflow over the course of the past year, once all the pieces in place, has been constructed with repeatability as a primary goal. working from regular dumps of our eads as they exist in our production system, we run the preprocessor over the entire corpus. archivists would then inspect the post-processed ead to make sure the fixes were transforming the markup as intended. then, using our utility scripts, we ingested these files into our test archivesspace instance, generally over the weekend, due to the time involved. after the ingest had completed, dave and the archivists examined the ingest logs and the records produced by ingest. findings from these sources were used to develop hypotheses as to the cause for classes of failures, and (with cross-checking against the pre-ingest corpus) were used to inform further development. in summary: action work done by regular xml dump automated run ead files through preprocessor developer analyze preprocessor db and ouput xml developer, metadata analyst, archival staff ingest xml to archivesspace developer analyze log from ingest automated, additional work by developer analyze corpus (before and after) developer, metadata analyst, archival staff for xml dumps, we made use of standard unix utilities; cron for scheduling, rsync for copying the files to the preprocessor’s host. the full corpus was copied weekly, with incremental updates copied over daily. throughout most of the work, we worked directly from the weekly dumps, but as we got closer to the final date, we folded in incrementals (manually, using cp over the command line). the preprocessor and ingest script were run directly via command-line, generally in a screen session to avoid interruptions by terminal timeout or disconnect. this isn’t necessarily recommended practice, or even something that will reliably keep processes alive on some systems; anyone replicating our workflow should make sure to investigate the workable methods of managing long-running scripts on their system. individuals involved in the work used a variety of tools and methods for analyzing the xml files; this was less a repeatable, unified process than just people applying what they individually know to the task at hand. special mention is due to several freely available xml databases, exist-db in particular. being able to run xpath and xquery expressions over the whole corpus was utterly invaluable throughout the work. it was especially useful in finding examples of particular structures, in determining the scope of problems throughout the collection, and in verifying the results of fixes. additionally, both the official loc ead tag library and https://eadiva.com/2/ were invaluable resources for the developer, who hadn’t looked at ead since grad school. retrospective good decisions schematron ended up being a very good fit for the project. it ended up being flexible enough to describe each of the problems we had. the ead checker proved useful locally, was used outside harvard, and just generally provided a high reward for relatively low effort and resource cost. it also seems likely to be useful going forward, as an easy way to check archivesspace ead output, or for institutions who continue to maintain their finding aids in ead. additionally, it would be trivial to adapt for other xml metadata formats; all it would take is swapping out the existing schematron for one describing the desired patterns. pursuing this process in an iterative fashion by repeatedly running the whole process over the entire corpus on a test system proved to be invaluable. in addition to the intended goals of auditability and reporting, we also got something else useful; a developer with seven months solid practice in cramming ead into archivesspace. a large number of hard to resolve edge cases came up over the course of our work, that would have been disastrous if we had first encountered them near or during the production load. while it has the unfortunate result of making the fruits of our effort less accessible to the wider archivesspace community, in retrospect, moving the bulk of our work into a preprocessor and decoupling our progress from that of archivesspace was a good decision. this is not to say that we haven’t benefitted greatly from work done by the archivesspace team and community, but fundamentally, avoiding the need for close coordination with the core team allowed us to keep moving forward during periods of time that the archivesspace team were unable to be fully responsive. decisions which in hindsight were not so good because the database component of the preprocessor essentially is a glorified log, high quality logging of errors within the preprocessor was left till fairly late in the development timeline. there were several “mysterious” errors which could have been solved much earlier on if internal logging in the preprocessor had been included from the beginning. while selecting schematron was overall a positive choice, two decisions made early on in implementation had negative implications for the project. the first, and more minor, was using the xslt 2.0 version of schematron, which means it supports xpath 2.0 expressions. xpath 2.0 is significantly more capable, in ways that made describing problems substantially easier at first blush. key amongst these advantages is its support for regular expressions. but the preprocessor’s actual xml processing code is written using nokogiri, which is based on libxml2, and only supports xpath 1.0; choosing the xslt 1.0 version of schematron would have made reuse of expression possible between the schematron code and the processing code. the second, and more serious bad decision made was in the structure of the schematron document itself. schematron has two basic reporting constructions, the <assert> and the <report>. dave made an early decision to use only one of these constructs, to simplify processing of the schematron output. this in itself was probably the right decision; however, he chose the wrong one. namely, all issues are described via the schematron <assert> construction, which evaluates its xpath in a negated context, i.e. creating output only if the xpath returns false. this, in addition to being arguably more awkward, is the exact opposite of what needs to be done in processing code, where xpath needs to be written to find and return the elements to be acted upon. almost all bugs written which involved wrong xpath, which made up a substantial percentage of all bugs, resulted from copying and pasting xpath without remembering to negate it, or writing positive expressions rather than negated expressions. while it didn’t produce problems locally, due to dave having substantial prior rails experience, building on rails probably cost more in complexity for potential adopters outside the university than it provided value in reporting/self-service. the work spent parallelizing ingest scripts ended up being a complete waste of time, because of the aforementioned race condition in archivesspace. at time of print, the issue is solved in the most recent version of archivesspace, but too late for us to benefit. every successful ingest completed, including our final production ingest, has been done one file at a time. about the authors dave mayo (dave_mayo@harvard.edu) is a senior digital library software engineer at library technology services, a division of huit at harvard university, where he has worked for the past three years. he is currently working on behalf of the office of scholarly communications at harvard, and previously worked as primary developer on the migration efforts described in this article. dave holds an mlis from simmons college. kate bowers (kate_bowers@harvard.edu) is the collections services archivist for metadata, systems, and standards in the harvard university archives. she has held a variety of positions in the harvard university archives for the past 21 years, all revolving around metadata. before joining the harvard university archives, she worked at tufts university, the library of congress motion picture division, and harvard law school. she has also been adjunct faculty at simmons college since 2009. her interest is in realizing the potential of metadata and technology to allow people to do what humans are good at–exploring, learning, thinking, teaching, and sharing their ideas. notes [1] container management is, briefly, a collection of features relating to associating the physical location and containment of materials with their records in aspace. see http://guides.library.yale.edu/archivesspace/aspacecontainermanagement and http://campuspress.yale.edu/yalearchivesspace/2014/11/20/managing-content-managing-containers-managing-access/ for more information on the rationale for and development of container management in archivesspace. references encoded archival description tag library, version 2002: ead elements “<did> descriptive identification” (washington, dc : library of congress, 2002) https://www.loc.gov/ead/tglib/elements/did.html accessed 2016-12-01. harvard university. oasis steering committee. digital finding aids at harvard: history and project report march 1996 (cambridge, massachusetts : harvard university, 1996) https://web.archive.org/web/20000305224811/http://hul.harvard.edu/hul/dfap/projdescription.html accessed 2016-12-01. harvard university. oasis steering committee. minimal-level ead v. 1 coding requirements april 1999 (cambridge, massachusetts : harvard university, 1999) https://web.archive.org/web/20000309220757/http://hul.harvard.edu/hul/dfap/dfap_required_tags.html harvard university. oasis steering committee. oasis annual report, 2002-2003 (cambridge, massachusetts : harvard university, 2003) hul.harvard.edu/cmtes/ulc/aac/oasis/2003_oasis_annual_report.pdf accessed 2016-12-01. harvard university. oasis steering committee. harvard ead guidelines, revised march 30, 2011. (cambridge, massachusetts : harvard university, 2011) https://web.archive.org/web/20150909021833/http://hul.harvard.edu/ois/systems/mat/eadguide.pdf appendix a harvard ead normalization for ingest to archivesspace issues list issue description resolution 1. field limits add check to schematron to report elements that exceed aspace database field length. ref to repos 2. /> empty elements should be removed in pre-processing so that their presence does not interfere with other tests. preprocess 3. <– comments are not ingested by aspace importer. do nothing 4. @audience audience (internal vs. external) import does not adhere to ead standard of no value = external fix importer 5. arrangement <arrangement> embedded in <scopecontent> exports invalid ead on roundtrip. <arrangement> imported twice: once as mixed content within scopecontent and again as a note with type=arrangement. the note is mangled, only including items from the last, lowest level of a nested list. preprocess to extract <arrangement> from <scopecontent> and place after. preprocess 6. bioghist on import to/export from aspace, if there is a <persname> within (or adjacent?) to a <bioghist><p>, all the <p> tags are removed upon ingest. it is also making it invalid on ead export from aspace, since <bioghist> needs subordinate elements to be valid. even if it were valid, it would be one inscrutable block of text. preprocess 7. bioghist if there is a <p audience=”internal”> there is no closing </p> tag, thus the ead is invalid upon export. preprocess 8. c <c> with no @level import with @level=otherlevel but no corresponding @otherlevel value. supply @level=otherlevel @otherlevel=”unspecified” in pre-processing. preprocess 9. change if there are multiple changes in our finding aid, multiple revision statements should be created upon ingest. currently aspace seems to create a single change statement for all changes. importer fixed 10. change during import, add a <revisiondesc><change> to indicate preprocessor altered finding aid. preprocess 11. chronlist <chronlist> embedded in <bioghist> and <scopecontent> export twice, once embedded in note and second as standalone element. the standalone element should be omitted. preprocess 12. container analyze uses of container; parse text (e.g., “box”) from data (number) where applicable. wg decided not to normalize this data. ref to repositories 13. container aspace containers require a type attribute, but less than 6% of harvard containers include a type now. absence of this attribute blocks aspace importer. add “unspecified” to controlled list in aspace and add the value during pre-processing. preprocess 14. container set @label=”unspecified” if @label is not already present. preprocess 15. container relationship between container and unitid? moot unless we parse containers, which we aren’t. do nothing 16. controlaccess investigate omitting archdesc/controlaccess during migration and instead pull 6xx and/or 7xx from marc records. do nothing 17. dao harvard <dao> elements lack xlink:title attribute required by aspace. for each <dao>, look for one of the following elements in the immediate parent <did>, in this order: <unittitle>, <unitid>, <container>, or <unitdate>. repeat the text in that element as the <dao> xlink:title attribute. preprocess 18. dao random id is being created when <dao> is imported; may need to import existing id. preprocess 19. dao multiple daos in a <did> or <c> load as separate d.o.s, export as separate daos in the did, but lose their position relative to other elements in the <did>. they are also indistinguishable on export from daos derived from daogrp/daolocs. ref to repository 20. daodesc the <daodesc> is put into a general note and is not exported in the ead. preprocess 21. daogrp aspace imports portions of the component <daoloc>s from an incoming <daogrp> as related file versions. however, it omits xlink actuate attribute “onload” and xlink show attribute “embed” from the associated <arc> element. preprocess <daogrp> to convert <daoloc>s to <dao>s, take xlink:actuate and xlink:show attributes from related <arc> and add them to the appropriate <dao>; remove <daogrp> wrapper. when exported, two <dao>s in a <did> should be converted back to <daoloc>s with <arc>s in a <daogrp>. preprocess 22. descgrp descgrp is not supported in archivesspace, and needs special handling to insure that its constituent elements are imported. preprocess 23. did absence of unitid at the collection level causes import to fail. supply <unitid> content from marc record id. preprocess 24. did absence of unittitle at the collection level causes import to fail. ref to repository 25. did absence of unitdate at the collection level causes import to fail. supply based on earliest and latest nnnn values in component-level unitdates. if no unitdates, supply 0000-9999. preprocess 26. did absence of both unittitle and unitdate at a subordinate level causes import to fail. provide unittitle or unitdate from parent component’s <did>. preprocess 27. did where <did> lacks both unittitle and unitdate, but includes <note>, further analysis needed to identify pattern for conversion. preprocess 28. dimensions dimension element is loaded twice, once as a note w/type=physical description and again as a note with type=dimensions. should be loaded in dimensions in aspace extent when parent <physdesc> includes a simple <extent>. see: https://archivesspace.atlassian.net/browse/ar-1134 fix importer 29. extent extent is required at the collection level, but is missing in some finding aids. supply extent of “1 collection.” preprocess 30. extent separate multiple extents are sometimes encoded in a single extent element. demote these to <phydesc> which will ingest as physical description type notes by the aspace importer. preprocess 31. extent extent needs to be parsed to correctly import whole vs partial extent expressions. wg decided not to parse extents to this degree. do nothing 32. extent assess each <extent> field for parsability. if parsable, determine conformance canonical pattern and measurement term. convert alternate terms to canonical term. preprocess 33. extent multiple extent elements in a single physdesc are loaded with one as a single extent with the others in a container summary. where extents are truly parallel, can/should they be loaded as separate extents in aspace? do nothing 34. extent extents that begin with a statement of approximation (i.e., approximately, circa, ca or ca.) will not load as <extent>. extract statement of approximation. process remainder of extent according to rules for extent processing (see above). add new, additional <physdesc>extent is approximate.</physdesc> preprocess 35. extref extref (links) may not import, but the text of the extref will import. do nothing 36. index in some finding aids, indexes are nested, which is not supported in aspace. in some indexes, the key for implied data within the index entries is provided, e.g., “key: no symbol = writer * = writer and recipient = recipient” if these indexes are retained, this may need to be used in conjunction with the index entries to make the data explicit. preprocess 37. indexentry an index entry with a ref@target and ref@text imports as two separate items. it should import as one item with associated reference and reference text. preprocess 38. indexentry nested <indexentry> elements import in order, but the nesting is lost. repositories have been notified about what will happen if they do not modify ead. ref to repository 39. language the importer appears to be including use-for forms of the language in addition to the preferred form. preprocess 40. language language is not repeatable in aspace and should be. this may require changes to archivesspace for resources and archival objects. do nothing 41. language multiple languages in a single language element need to be parsed into separate element occurrences. preprocess 42. list elements with lists are mangled during import. preprocess 43. list nested lists (i.e., list/item/list) drop data during import. solution is to flatten them. preprocess 44. list <list> inside <bibliography> is imported as separate <odd> element, but with titles stripped out. bibliography note created in aspace, but doesn’t contain any data. to put it another way: <bibliography> with <head> and a <list> of <items> becomes two notes (general with list items; and bibliography with head). drop the <list> element and convert its <item>s to <bibref> preprocess 45. name the importer does not match the documentation: testing for nesting with origination and controlaccess is not occurring, and creation of name_person and agent.type and agent.role is not occurring. retest after qa version is updated. report error 46. name @encodinganalog is not used to set indexentry (aka item) type. convert <name> to different element based on encodinganalog values. preprocess 47. namegrp aspace data structure does not support <namegrp> (a structured string akin to a precomposed subject heading). the elements within <namegrp> import as independent index entries, which is inaccurate. preprocess 48. note 1. note elements are stripped from components during ingest by the aspace importer 2. archivesspace does not support the <note> tag; importer strips them or converts them to <odd> depending on the context. preprocess 49. note <notes> within <controlaccess> are dropped during import. do nothing 50. note <notes> within <archdesc> are dropped during import. convert <note> to <odd> prior to load. preprocess 51. note convert <note> to <odd> when <note> is a child of <c> preprocess 52. note when <note> appears as child of <did> convert to <odd> and place outside and immediately following the <did> in which it occurred. preprocess 53. note <note><p> occurs instead of proper element, e.g. <scopecontent> or <bioghist>. ref to repository 54. p initially there was import/export mangling of <p> resulting in invalid ead on export. importer fixed, but sibling elements that are not paragraphs may be invalid on export. ref to ead export group 55. persname @role=”donor (dnr)” is imported as role=subject. preprocess 56. physfacet same as dimensions. fix importer 57. ptrgrp ref value + target structure is not repeatable in indexentry in aspace. preprocess 58. ref <unitid> loses @id attribute on load. this makes ref targets that reference unitid @ids invalid. preprocess 59. relatedmaterial see <list> and <extref>, above. extref works, lists depends on resolution of list issue. preprocess 60. @startyear/endyear startyear and endyear attributes in <date> and <unitdate> elements are harvard only, invalid in canonical ead used by aspace. transform to @normal. preprocess 61. @startyear/endyear when date values represented in startyear/endyear (or normal) attributes are irrational (the start is later than the end), delete the attributes. preprocess 62. table only 3 harvard finding aids contain tables. they are not supported by aspace and should be manually removed prior to conversion. ref to repository 63. title <title> within <controlaccess> is dropped during import. should be manually changed to subject, corpname, or other choice. ref to repository 64. unitdate parse descriptions in and around <unitdate> to correctly tag bulk and inclusive dates. preprocess 65. unitdate sometimes the words “bulk” or “inclusive” inside <unittitle> but not inside <unitdate>. preprocess 66. unitdate when <unitdate>contains both “bulk” and “inclusive”, it needs to be parsed into two separate unitdates in aspace. preprocess 67. unitdate unitdate includes two date ranges, first with no designation, second with “bulk” before or after date range preprocess 68. unitdate if <unitdate> contains “circa”, “ca.”, “ca”, “approximately”, “approx.” (all case insensitive), set @certainty=”approximate” preprocess 69. unitdate <unitdate> lacks @startyear, @endyear and @normal. with no action, these will be loaded as date expression, with no normalized, searchable dates. where <unitdate> lacks all these attributes, would it be desirable to examine each nnnn inside <unitdate>, keeping the earliest and the latest and creating a @normal=”[earliest/latest]” preprocess 70. unitdate because aspace has separate fields for unittitle and unitdate, it takes rest-of-date information that follows the ead <unitdate> and includes it in the unittitle, i.e., puts it before <unitdate>, in non-dacs fashion, with an added comma. preprocess 71. unitid importer converts “.” to “_” in <unitid> importer fixed 72. unitid for multiple unitids, archivesspace retains the last one and discards the rest. concatenate with a double dagger separator. preprocess 73. unittitle multiple <unittitle>s do not import. e.g., one unittitle in english, one in chinese–only the latter/last unittitle imports. concatenate with double dagger separator. preprocess subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – from isis to couchdb: databases and data models for bibliographic records mission editorial committee process and structure code4lib issue 13, 2011-04-11 from isis to couchdb: databases and data models for bibliographic records for decades bibliographic data has been stored in non-relational databases, and thousands of libraries in developing countries still use isis databases to run their opacs. fast forward to 2010 and the nosql movement has shown that non-relational databases are good enough for google, amazon.com and facebook. meanwhile, several open source nosql systems have appeared. this paper discusses the data model of one class of nosql products, semistructured, document-oriented databases exemplified by apache couchdb and mongodb, and why they are well-suited to collective cataloging applications. also shown are the methods, tools, and scripts used to convert, from isis to couchdb, bibliographic records of lilacs, a key latin american and caribbean health sciences index operated by the pan-american health organization. by luciano g. ramalho 1. introduction the relational database model is well grounded in theory and well supported by the software industry. relational database management systems (rdbms) replaced legacy hierarchical and network database products in nearly every enterprise and became the mainstream. object-oriented dbms have not taken the leading role from relational systems, even after object-oriented programming became the norm, and in spite of the so-called “impedance mismatch” perceived when one needs to store nested objects into a collection of flat tables. after resisting the object-oriented wave, relational databases became so dominant that they seemed like the only game in town, or at least the only game in most shops. but in the last five years, google published a paper on bigtable (chang, 2006), the proprietary, distributed non-relational database behind many of its online properties. then amazon.com revealed dynamo, the non-relational database that powers some of their aws cloud-based services (decandia, 2007). facebook open-sourced cassandra, the distributed non-relational database created to enable its crucial in-box search feature. cassandra is now also used by twitter, digg, rackspace and cisco, among many others, and has such a vibrant community that it became a top-level project of the apache foundation in 2010, less than three years after its initial public release (apache, 2011a). these developments are part of a trend that became known, in 2009, as the “nosql” movement. nosql stands for “no sql” or “not only sql”, depending on who you ask; but everyone agrees it’s not really about the sql language, but about seeking alternative answers to problems that are not amenable to relational solutions. meanwhile, thousands of small to medium libraries in developing countries have been oblivious to all this and continue using their isis databases for daily operations. isis is a family of non-relational database systems with a history that goes back to the 1970’s. it was developed by unesco – the united nations educational, scientific and cultural organization – specifically for bibliographic data, and it is still officially distributed and widely used (hopkinson, 2005). isis had a very positive impact, allowing libraries with limited resources to computerize their catalogs on simple, stand-alone pcs. some isis products became web-enabled, but after that, progress has been very slow in the last 10 years. meanwhile, a couple of new open source nosql databases present a possible migration path for isis users, a path that will not require wholesale normalization of decades of bibliographic data. this paper describes how bireme/paho/who, a digital library that is part of the pan american health organization, and a large-scale user of isis databases, has studied the viability of migrating one of its main databases to apache couchdb, a modern document database. the rest of the paper is structured as follows. section 2 discusses data models for bibliographic records, explains why it is tempting for librarians to look beyond the relational model and compares the structure of marc records to the semistructured data model formalized in the 1990’s. section 3 briefly presents isis, then compares two recent products that support semistructured records, couchdb and mongodb, and shows our criteria for choosing among them. section 4 discusses different formats for representing isis records in couchdb, then describes the tool we developed for converting records and the process used to load them into couchdb. section 5 talks about how indexing works in isis and couchdb, and exemplifies queries on the latter. finally, section 6 summarizes what we have done, our conclusions and ongoing work. 2. data models for bibliographic records one of the defining characteristics of the relational data model, the first normal form (1nf), dictates that attributes must be atomic. in other words, in the industry-standard flat relational model1, fields cannot be structured into subfields nor contain multiple values. on the other hand, multivalued fields are useful and common for representing bibliographic data. books often have more than one author and cover multiple subjects. a publisher, while logically a single attribute of a book record, has attributes of its own, like a name and a place. it is therefore useful to split a publisher field into parts. that is why marc supports subfields and multivalued fields through repeating tags (similar to xml). the marc structure, or “empty container”2 carried over to the iso-2709 standard, which describes a generic data interchange format allowing applications to attach any meaning to the tags and subfield markers. here we will refer to the “iso-2709 data model” as a generalization of the data model implied by the marc record structure. of course, library data is often stored in relational databases, but normalization means that the description of a single item is spread over multiple records in several tables. that may work well within the context of one library, but when exchanging data, it is more practical to have just one record per item. that is one reason why marc is still with us, and why xml became so important in a world dominated by relational databases. if the iso-2709 data model does not adhere to the flat relational model, then what is the theory behind it? at first there was none, but in the mid-1990s, while xml was still under development, researchers from upenn, stanford, at&t labs and the inria french r&d agency conceived the semistructured3 data model (abiteboul, 1999). their work supports reasoning about marc and iso-2709 records but also about the richer data models of xml and json4 documents. a useful definition by dan suciu, a pioneer of semistructured data research, appears in the “encyclopedia of database systems” (suciu, 2009): the semi-structured data model is designed as an evolution of the relational data model that allows the representation of data with a flexible structure. some items may have missing attributes, others may have extra attributes, some items may have two or more occurrences of the same attribute. the type of an attribute is also flexible: it may be an atomic value or it may be another record or collection. moreover, collections may be heterogeneous, i.e., they may contain items with different structures. the semi-structured data model is a self-describing data model, in which the data values and the schema components co-exist. thanks to the semistructured data model, we don’t need to feel ashamed of our denormalized bibliographic records any more. even better, we have a framework to evaluate databases of the current nosql crop with regard to our needs as caretakers of denormalized datasets. the book “data on the web: from relations to semistructured data and xml” (abiteboul, 1999), introduces a notation for representing semistructured data. it is called “ssd-expression” and it looks like this: {name: {first: "lewis", last: "carroll"}, tel: 5553457, email: "cld@ox.ac.uk", email: "lcarroll@pobox.com" } except for the repetition of the email key, it’s very similar to json (crockford 2006a). the presence of keys describing fields like name, first etc., and the storage of those keys alongside the data, is what the encyclopedia entry means by coexisting data values and schema components (suciu, 2009). it is a characteristic shared by iso-2709, xml, and json (with the limitation that iso-2709 fields are identified by tags composed of three alphanumeric ascii characters, though most applications only use numeric tags). the repeated email tag could be represented similarly in xml and in iso-2709, but in json the keys must be unique within an object structure, so multivalued fields are often represented as lists of values: {"name": {"first": "lewis", "last": "carroll"}, "tel": 5553457, "email": ["cld@ox.ac.uk", "lcarroll@pobox.com"] } the iso-2709 record format influenced the record structure of the isis family of databases, introduced by unesco in the 1970s. isis records follow the structure of the iso-2709 standard, and most isis systems import and export to that format. here is the same record of the examples above, as seen in a common isis display format: 10 «^flewis ^lcarroll» 20 «5553457» 30 «cld@ox.ac.uk» 30 «lcarroll@pobox.com» in an isis record the field names are replaced by numeric tags. the «» delimiters are not part of the content, but a feature of the display format. here there are two occurrences of tag #30, the e-mail field. field #10 is split into two subfields (“f” and “l”) by special markers. instead of the subfield delimiter control character used in marc systems, isis uses the circumflex accent: ^ (ascii 94). a subfield identifier is case-insensitive and consists of just one ascii letter or digit. the isis data model is simpler than that of iso-2709 because it does not have field indicators5 and subfields are not repeatable6. clearly the isis data model is not as flexible as the general semistructured data model. given the syntax of subfield markers, isis fields are limited to one level of nesting only. in other words, borrowing from the xml jargon: subfields cannot have child elements, just character data. also, an isis field may have mixed content7: some unmarked text may appear before the first subfield marker. for example, in this field: 18 «lords of finance^bthe bankers who broke the world» the main title, “lords of finance” is not preceded by a subfield marker. the semistructured data model, as described by abiteboul et. al., has no concept of mixed content like xml has. so we will call that part “the main subfield” and pretend it is preceded by an implied “^_” (underscore) subfield marker whenever we need to refer to it in code. with this simple arrangement, any isis record can be represented as an ssd-expression or a json record. 3. semistructured database systems 3.1. the isis family in 1985, giampaolo del bigio ported unesco’s cds/isis mainframe database system to pc/dos (lopes, 2010). the result, called microisis, was distributed free of charge and became the de facto standard for computerized library catalogs in developing countries. its windows version, winisis is still distributed by unesco and remains widely used in small to medium libraries (hopkinson, 2005). beyond local opacs, isis is also used by two large regional bibliographic databases in latin america and the caribbean: scielo (scientific electronic library online) and lilacs (latin american and caribbean health sciences index), both built by bireme/paho/who, which is a specialized center of the pan american health organization located in são paulo, brazil. together, scielo and lilacs routinely handle millions of bibliographic records using cisis, a custom version of isis developed by bireme/paho/who. microisis, winisis, cisis and the more recent j-isis from unesco are all interoperable and form the isis family of database systems. the legacy isis codebases are showing their age, and after years of closed-source development, none of the derivations have managed to become successful open source projects. meanwhile, the nosql movement has highlighted some open source non-relational databases implementing variations of the semistructured data model. this has motivated bireme/paho/who to look for alternatives. 3.2 couchdb and mongodb among the recent crop of nosql systems, two products stand out for implementing the semistructured data model in a way that matches the operational needs of bireme/paho/who: apache couchdb and mongodb (apache, 2011b, mongodb.org, 2011). their data model is not as flat as the key-value stores (optimized for retrieving blobs given a primary key), nor as deep as graph databases (designed to allow general queries over paths of nested objects). what they offer is somewhere in between: json-like records allowing nested structures, and expressive query languages to index and retrieve those records. those rich records are called “documents”. couchdb and mongodb call themselves “document databases”. in the case of couchdb, the document format is json. mongodb uses bson, a binary format inspired by json but offering more data types, such as int32, datetime, and even a type called “javascript code w/ scope” (dirolf, 2010). json can be trivially converted to bson, but the reverse may not be so easy, depending on the data. while the difference between those formats can be considered an implementation detail, they reflect different priorities. mongodb is optimized for fast updating, by aggressively caching writes and by overwriting updated records in place whenever possible: the bson structure is designed to allow updating specific fields of an existing document. the flip-side of this optimization is that a software crash can cause data loss, and a minimum deployment of two servers in master-slave configuration is recommended if durability is required8. in contrast, couchdb is fault-tolerant: it only appends to the database file on disk, and that is an atomic operation in modern operating systems. as a result, the database file is always consistent in the event of a software crash, and backups can be made while the system is running. the drawbacks of the append-only design are the need for periodic database compaction – a time-consuming batch operation – and slower updates. couchdb was influenced by lotus notes, a networked, collaborative application suite designed to support users who are often off-line. so, couchdb allows master-master replication, that is, synchronization between peer nodes which have received inserts and updates independently. mongodb supports only master-slave replication: only one node receives updates and inserts, and replicates to the slaves. while couchdb can be used in large clusters, it is also well-suited as a database for small desktop apps. for that reason it comes pre-installed on ubuntu linux since 2009. it even runs on android mobile devices (a free, one-click installer can be found in the android marketplace). because couchdb supports http and json natively, and can run javascript procedures on the server, applications can be developed without any middleware: browsers communicate directly with the database, and all the logic is written in javascript. the informative post “comparing mongo db and couch db” was written by dwight merriman, ceo of 10gen, the company behind mongodb (merriman, 2010). given the characteristics of these systems, we decided to use couchdb for our lilacs experiments for these reasons: easier deployment in a durable configuration. support for master-master replication, useful for distributed cooperative cataloging applications. direct support for json over http, enabling web services and ajax applications without middleware. however, we do envision scenarios in which we would like to use both mongodb and couchdb. for example, the canonical store of bibliographic data could be couchdb, while a cluster of mongodb instances could be used for user-provided content, tracking, recommendations and other features demanding faster database writes. 4. loading isis records into couchdb 4.1. alternative representations of isis records as json documents the isis data model, a subset of the iso-2709 model, is not as expressive as the json data model. there are several ways to represent the same isis record in json. consider this abridged example of a lilacs record: 1 «br1.1» 2 «538886» 4 «lilacs» 4 «llxpedt» 5 «s» 6 «as» 8 «internet^ihttp://…/imagebank/pdf/v3n3a04.pdf?aid2=168&…» 10 «kanda, paulo afonso de medeiros^1university of são paulo ^2school of medicine^3cognitive disorders of clinicas hospital reference center^pbrasil ^csão paulo^rorg» 10 «anghinah, renato^1university of são paulo^2school of medicine ^3cognitive disorders of clinicas hospital reference center ^pbrasil^csão paulo^rorg» 12 «the clinical use of quantitative eeg in cognitive disorders ^ien» 12 «a utilização clínica do eeg quantitativo nos transtornos cognitivos^ipt» 30 «dement. neuropsychol» 31 «3» 32 «3» 35 «1980-5764» tag #2 is the lilacs identifier, a non-repeating, numeric field, and tag #10 is an author field in an analytic (article-level), record. the lilacs data dictionary (bireme, 2008) defines #10 as a repeating field. in addition, it is composed of the following subfields: subfield description sample content main author name kanda, paulo afonso de medeiros ^1 affiliation level 1 university of são paulo ^2 affiliation level 2 school of medicine ^3 affiliation level 3 cognitive disorders of clinicas hospital reference center ^p country brasil ^c city são paulo ^r degree of responsibility org (organizer) one way to represent such a record in json is shown below (only fields #10 and #2 are shown; some contents shortened for clarity): { "10": [ { "_": "kanda, paulo afonso", "1": "university of são paulo", "2": "school of medicine", "3": "cognitive disorders of … reference center", "c": "são paulo", "p": "brasil", "r": "org" }, { "_": "smidth, magali taino", "1": "university of são paulo", "2": "school of medicine", "3": "cognitive disorders of … reference center", "c": "são paulo", "p": "brasil", "r": "org" } ], "2": [ { "_": "538886" } ] } in the representation above, the record is a json mapping between tags (eg. “10”) and lists of values (or occurrences in isis jargon). the use of lists allows any field to be repeatable, therefore any possible isis record fits this scheme. also, within the list of occurrences, each one is represented by a mapping of subfield keys and values. the special key “_” (underscore) is used to denote the main subfield, and when there are no subfields that is the only key. this creates unnecessary nesting, as demonstrated by field #2 which is the lilacs identifier field, non-repeating and devoid of subfields as defined by the lilacs data dictionary. but it does make the structure homogeneous, an important feature when dealing with legacy semistructured datasets where not all records adhere to the current schema. during our research we called the representation above isis-json type 3 (bireme, 2010). the main drawback of isis-json type 3 is that, by definition, json mappings are unordered9. therefore, neither the tag order within the record nor the subfield order within a field occurrence are preserved. in the lilacs database, tag order within a record is only relevant when there are repeating tags, such as #10 (author, analytical level). fortunately, keeping repeated occurrences in a list does preserve their relative ordering, even if the overall field order is not kept. for example, in the original isis record under analysis field #2 appeared before all occurrences of field #10, while in the json representation the key “10” precedes key “2”. this is irrelevant in practice. but, crucially, the order of the authors is the same in both formats: first “kanda, paulo afonso”, then “smidth, magali taino”. regarding subfield ordering within a field, the lilacs data dictionary does establish a canonical ordering. for example, in the case of field #10 the order is _, 1, 2, 3, p, c, r (the main subfield is always first, by definition). however, as jason thomale points out in “interpreting marc: where’s the bibliographic data?” (thomale, 2010), subfield markers should be interpreted as textual markup, and their ordering is significant. particularly when cleaning up or converting legacy data, maintaining the order of subfields from the original record may be important to ascertain the original cataloger’s intention, even if lilacs subfields do have a prescribed ordering. to preserve subfield ordering, two alternative json representations have been tried at bireme/paho/who. the first is isis-json type 1 (line breaks added within long strings for clarity): { "10": [ "kanda, paulo afonso de medeiros^1university of são paulo ^2school of medicine^3cognitive disorders of clinicas hospital reference center^pbrasil ^csão paulo^rorg", "smidth, magali taino^1university of são paulo ^2school of medicine^3cognitive disorders of clinicas hospital reference center^pbrasil ^csão paulo^rorg" ], "2": ["538886"] } in this format, each field occurrence is a single string, with the subfield delimiters embedded. this essentially “punts” on the issue of how to represent subfields, leaving the parsing to be done by the database. because couchdb uses javascript as its default language for creating indexes, it is a viable approach. another possibility is isis-json type 2: { "10": [ [ ["_", "kanda, paulo afonso"], ["1", "university of são paulo"], ["2", "school of medicine"], ["3", "cognitive disorders of … reference center"], ["p", "brasil"], ["c", "são paulo"], ["r", "org"] ], [ ["_", "smidth, magali taino"], ["1", "university of são paulo"], ["2", "school of medicine"], ["3", "cognitive disorders of … reference center"], ["p", "brasil"], ["c", "são paulo"], ["r", "org"] ] ], "2": [ [ ["_", "538886"] ] ] } here each field occurrence is an association list10, that is, a mapping represented as a list of lists, where each inner list contains a key-value pair. the advantage of an associative list over a json object mapping is that the order of items is preserved, but at the cost of a linear search to locate a key within a field occurrence. because of how indexing works in couchdb, this is not as costly as it may seem, as will be seen in the next section. for marc-style records, the association list has the added advantage of allowing repeated mappings to represent repeating subfields. 4.2. the isis2json conversion tool to convert isis records into json structures, we developed a python script called isis2json.py (bireme, 2010b). it can be executed with both the python and jython interpreters, versions 2.5 through 2.7. when running under python it can read only iso-2709 files, but as a jython script it leverages the bruma java library (barbieri, 2011) and can also read binary isis files in .mst format directly. several options control the structure of the json output. for example, the command line below generates output suitable for batch importing to couchdb: $ ./isis2json.py cds.iso -c -t 3 -q 100 > cds1.json the arguments used in the above example are: -c to generate records within a “docs” list, as required by the couchdb _bulk_docs api; -t 3 for isis-json type 3 export (fields as dictionaries of subfields); -q 100 to output only 100 records; here is the help screen of isis2json.py: $ ./isis2json.py -h usage: isis2json.py [-h] [-o output.json] [-c] [-m] [-t isis_json_type] [-q qty] [-s skip] [-i tag_number] [-u] [-p prefix] [-n] [-k tag:value] input.(mst|iso) convert an isis .mst or .iso file to a json array positional arguments: input.(mst|iso) .mst or .iso file to read optional arguments: -h, --help show this help message and exit -o output.json, --out output.json the file where the json output should be written (default: write to stdout) -c, --couch output array within a "docs" item in a json document for bulk insert to couchdb via post to db/_bulk_docs -m, --mongo output individual records as separate json dictionaries, one per line for bulk insert to mongodb via mongoimport utility -t isis_json_type, --type isis_json_type isis-json type, sets field structure: 1=string, 2=alist, 3=dict -q qty, --qty qty maximum quantity of records to read (default=all) -s skip, --skip skip records to skip from start of .mst (default=0) -i tag_number, --id tag_number generate an "_id" from the given unique tag field number for each record -u, --uuid generate an "_id" with a random uuid for each record -p prefix, --prefix prefix concatenate prefix to every numeric field tag (ex. 99 becomes "v99") -n, --mfn generate an "_id" from the mfn of each record (available only for .mst input) -k tag:value, --constant tag:value include a constant tag:value in every record (ex. -k type:as) 4.3. interacting with couchdb http is the main protocol for interacting with couchdb11. most database administration is done via futon, its built-in web interface. figure 1. couchdb futon web interface showing a lilacs record in isis-json type-2 format. nearly every task which can be done via futon can also be automated using the restful api of couchdb via any http client. in fact, interacting with couchdb is a great way to learn, in practice, what rest and restful interfaces are all about. here we use curl, a command-line http client available for many systems. for example, a database is created with a simple put request: $ curl -x put http://admin_user:password@127.0.0.1:5984/lilacs one operation that cannot be done via futon is uploading records in bulk. continuing the previous example, we could convert an iso-2709 file to json, then upload it to couchdb using curl in two steps: $ ./isis2json.py cds.iso -c > cds1.json $ curl -d cds1.json -h"content-type: application/json" \ -x post http://127.0.0.1:5984/lilacs/_bulk_docs when used together, the -q and the -s (skip) options allow splitting the output into several batches, which results in faster loading for large datasets on multi-core servers. for example, loading 20,000 full lilacs records, totaling 64mb, took 147 seconds using this shell command: $ isis2json.py -c -p v -i 2 -q 20000 lilacs100k.iso | \ curl -d @-h"content-type: application/json" \ -x post http://127.0.0.1:5984/lilacs/_bulk_docs loading the same data in two 10,000-record batches took 100 seconds when taking advantage of two cores with a simple shell script like this one: #!/bin/bash isis2json.py -c -p v -i 2 -q 10000 lilacs100k.iso | \ curl -d @-h"content-type: application/json" \ -x post http://127.0.0.1:5984/lilacs/_bulk_docs & isis2json.py -c -p v -i 2 -q 10000 -s 10000 lilacs100k.iso | \ curl -d @-h"content-type: application/json" \ -x post http://127.0.0.1:5984/lilacs/_bulk_docs if an _id attribute is not present in an inserted document, couchdb provides one with a uuid (universally unique identifier) like "ead3af23a4459b2d7a1aef05cb0012a9". it is highly recommended that an _id is given when adding documents to prevent inadvertent duplication of records if a bulk loading process is interrupted and restarted. therefore isis2json.py provides the -i option, used in the examples above, to fetch the value of one field in the isis input and use it as the _id attribute. so, for example, using the -i 2 option this (partial) isis structure: 1 «br1.1» 2 «538886» 4 «lilacs» 4 «llxpedt» can be converted into this json document record, ready to import into couchdb: { "_id": "538886", "1": [ [ ["_", "br1.1"] ] ], "2": [ [ ["_", "538886"] ] ], "4": [ [ ["_", "lilacs"] ], [ ["_", "llxpedt"] ] ] } note how the content of the first occurrence of the #2 tag was used as _id. 5. indexing and querying 5.1. building indexes for efficient retrieval, records must be indexed. it is useful to distinguish between a primary index and secondary indexes. a primary index allows retrieval via the primary key. every nosql database uses a primary index to allow queries on the primary key. in fact, “key-value” databases often provide only this means of retrieval. document databases also support secondary indexes, allowing fast access via arbitrary attributes of the records. in order to index those attributes, some form of data extraction must be performed on the records. by default, javascript is the language used for data extraction in couchdb12. in isis databases, a simple but terse “isis formatting language” is used to generate secondary indexes, also called inverted files. index definition in isis is done in a special file called the “field select table” (fst). here are just a few lines from the large fst used to generate indexes for the lilacs database: 98 0 v1,"-"v2 10 8 ,mpl,'|au_|'(v10^*|%|/), 38 0 if s(mpu,v38^a,mpl):'cd' then 'sp_cd-rom' fi, without going into all the details, here is what those lines do (the numbering is not relevant to this discussion13): line 98 indexes the contents of tags #1 (cooperating center code) and #2 (record id) concatenated with an hyphen; line 10 indexes each word of the main subfield of field #10 (author), concatenated with an 'au_' prefix; line 38 indexes the string ‘sp_cd-rom’ if field 38, subfield ^a contains the string ‘cd’; to generate an inverted file, an isis system applies each line of the fst to each record in the database, generating one or more index entries (or postings) per record. documents can also be indexed incrementally, but bireme/paho/who does a lot of batch processing and usually does “full inversion”, that is, regenerating inverted files from scratch. the overall process is very similar in couchdb. instead of an fst, couchdb allows us to define “views”, which are the result of running javascript functions to generate indexes. the simplest view contains only one function, the map function, which receives a document as an argument, extracts some data from it, and calls a special emit function to add entries to the index. the emit function takes two arguments: key and value. the key argument is the one actually indexed; queries will be made on it. it is usually unstructured, like a string, but sometimes it is useful to have an array as a key, for instance [country, province], to allow multi-level grouping of results. the value may be any json structure, and it is used to display the results of a query. for example, in an opac application we may have an index where each entry has an isbn as a key, and the book title and year of publication as the value. if you know some sql, you may find it easier to think in terms of a simple select statement: select title, year from books where isbn='9781594201820' here we have a books table with title, year and isbn fields. to support a query like the one above we would need an index on isbn. the title and year fields have no influence on the search, but they are mentioned in the query because we want to display them in the search result. in couchdb terms, to support a similar query we would need a view with a map function emitting the isbn as key, and the value would be a structure like {"title":"lords of finance", "year":2009}. the map function for that view would look like this: function(doc) { if (doc.type == "book") { emit(doc.isbn, {title: doc.title, year: doc.year}); } } because couchdb has no concept of tables like sql databases have, it is common to have different types of records mixed in the same database. therefore, map functions often have an if statement to select which records to index. in the example above, we index only documents of type "book". for each book, the key will be the isbn and the value will be an object with the title and year attributes. in addition to a map function, couchdb views may have a reduce function, which is used to aggregate results. views with reduce functions will be discussed later. indexing isis-json type 2 records: first approach being able to use a powerful language like javascript to create map functions gives a lot of flexibility when creating views. we can dig as deep as necessary into the structure of our documents, and massage the data we find in sophisticated ways. for one report, we created an index of all the lilacs records containing fields with repeating subfield markers. that would not be viable in an isis fst. however, one important limitation of map functions is that we have no access to data that is not part of the document being indexed. it is impossible, for example, to look up some value in another record, like a join operation in an rdbms (isis also allows this, via the l/lookup function of the isis formatting language). to start with a simple example, let us create a view to index lilacs records by tag #1 (cooperating center code), the id of the cataloging institution which created the record. from the lilacs data dictionary we know that tag #1 is non-repeating and has no subfields, which means all the content of that field resides in the main subfield (the first one) and only the first occurrence matters (because there are no repetitions). here is a view function that would create an index with field #1 as key and field #12 (title, analytic) as value: function(doc) { emit(doc["1"][0][0][1], doc["12"][0][0][1]); } here we begin to see the price we pay for the generality of the isis-json type 2 structure. the lilacs field #1 is non-repeating and has no subfields, so it could be represented like this: { "1" : "uy6.1", ... } and then we could reach it within a map function with the simple expression doc["1"]. even better, we could add an alpha prefix to the field name (the --prefix option of isis2json.py does that). then we could use dot notation (doc.v1) to access tag v1 in the following: { "v1" : "uy6.1", ... } however, to deal with isis records of any kind in the absence of schema information, we must assume that every field may have more than one occurrence, therefore its value cannot be just a string, but an array of occurrences. in addition, we must assume that every field may have subfields, therefore each occurrence must be structured as an associative list (isis-json type 2) or a dictionary (isis-json type 3), unless we want to parse the subfields every time when indexing. in an isis-json type 2 record, field #1 is represented as: { "1": [ [ [ "_", "uy6.1" ] ] ], ... } and to access its value we must write doc["1"][0][0][1]. here is why: doc -> {"1": [[["_", "uy6.1"]]]} # entire document doc["1"] -> [[["_", "uy6.1"]]] # list of occurrences of field #1 doc["1"][0] -> [["_", "uy6.1"]] # list of subfields of occurrence 0 of #1 doc["1"][0][0] -> ["_", "uy6.1"] # subfield 0 of occurrence 0 of #1 (key "_") doc["1"][0][0][1] -> "uy6.1" # value of subfield 0 of occurrence 0 of #1 this quickly becomes tedious, and very burdensome in more complex cases, for instance, to retrieve a specific subfield aside from the main one we would need to iterate over the subfield keys. we will soon show a library to make it easier to handle isis-json type 2 records, but first let’s return to our simple map function: function(doc) { emit(doc["1"][0][0][1], doc["12"][0][0][1]); } this function is part of a view in a “design document” in couchdb. a design document is written in json, stored in couchdb as other documents are, but its identifier starts with _design/. it may contain javascript functions for creating views as well as formatting output, validating document inserts/updates, etc. each couchdb database may have several design documents, and each design document may have several views. when trying out couchdb initially, the easiest way to create a view is by using the “temporary view…” option of the view dropdown in the top right area of the futon interface. then you can run the map/reduce functions and quickly see their results (if your dataset is not too large). to save your work to a permanent view, you will be prompted to provide a design document name and a view name. for any serious work, the couchapp tool is highly recommended (anderson, 2009). with it you can develop your design documents in your local filesystem, using your favorite editor and version control system, then push your code to a local or remote couchdb instance with a command. all of the views for this article were developed in this manner. the code is in bitbucket (ramalho, 2011). figure 2. editing and running a temporary view in couchdb futon. when a view is first visited via http, couchdb indexes all of the documents by applying the map function to each of them. couchdb also incrementally updates the indexes if documents are inserted or updated. the indexing is only done on demand, when a view is actually requested, and not when documents are created or changed. to install the map function above, we created a design document called lilacs and within it a view called center (for “cooperating center”). here is part of the result of requesting that view with the curl utility: $ curl -s http://ramalho.couchone.com/lilcouch/_design/lilacs/_view/center | head -5 {"total_rows":926,"offset":0,"rows":[ {"id":"559682","key":"ar1.1","value":"hemorragia digestiva baja: [revisi\u00f3n]"}, {"id":"559683","key":"ar1.1","value":"la radiolog\u00eda del colon por enema en el siglo xxi"}, {"id":"559684","key":"ar1.1","value":"el s\u00edndrome an\u00e9mico en las neoplasias del colon derecho"}, {"id":"559685","key":"ar1.1","value":"met\u00e1stasis cerebrales del carcinoma epidermoide del ano: comunicaci\u00f3n de un caso"}, note that the result comes as a json object with three properties: total_rows, offset and rows, the latter being an array of map function results. besides the key and value generated by the emit function, each result row also has an id attribute, which carries the _id property of the corresponding indexed document. by default, the result is sorted in ascending key order. here we used the head shell command to crop the displayed results; we will soon see a way of limiting the results actually sent by couchdb. by the way, the url shown here is public and you should be able to access it to try the examples by editing the url: http://ramalho.couchone.com/lilcouch/_design/lilacs/_view/center however, please note that the lilcouch database there contains only a sample of 1000 records, not the full lilacs database. 5.2. querying a view as we have just seen, couchdb queries are executed by making http requests on views. if no arguments are passed, all rows of the result are returned. however, there are several arguments that can be used to filter the results. for example, the following query uses the descending and limit arguments. note the use of single quotes around the url, necessary because “&” is a shell operator and we are using the command line. in this case the head shell command was not used. the total_rows property still counts 926, but only the last three rows are returned because of the limit=3 option. $ curl -s 'http://ramalho.couchone.com/lilcouch/_design/lilacs/_view/center?descending=true&limit=3' {"total_rows":926,"offset":0,"rows":[ {"id":"560003","key":"uy6.1","value":"vacunaci\u00f3n antigripal en personal de salud del hospital pedi\u00e1trico del centro hospitalario pereira rossell: evoluci\u00f3n de la cobertura del a\u00f1o 2006 al 2008"}, {"id":"560002","key":"uy6.1","value":"contin\u00faa descendiendo la mortalidadpor asma en uruguay: per\u00edodo 1984-2008"}, {"id":"560001","key":"uy6.1","value":"localizaci\u00f3n de lesiones mamarias subcl\u00ednicas con marcador met\u00e1lico (arponaje): an\u00e1lisis de los m\u00e1rgenes quir\u00fargicos"} ]} another possibility is to filter by key. the next query returns only the documents created by the cooperating center with code “co113”: $ curl -s 'http://ramalho.couchone.com/lilcouch/_design/lilacs/_view/center?key="co113"' {"total_rows":926,"offset":744,"rows":[ {"id":"559986","key":"co113","value":"de serendipia al tratamiento quir\u00fargico de la diabetes mellitus tipo ii"}, {"id":"559987","key":"co113","value":"la academia nacional de medicina se pronuncia sobre la emergencia social, la medicina y la salud"}, {"id":"559988","key":"co113","value":"presentaci\u00f3n inicial de las pacientes con diagn\u00f3stico de c\u00e1ncer de seno en el centro javeriano de oncolog\u00eda, hospital universitario san ignacio"}, {"id":"559989","key":"co113","value":"\u00bfes la diabetes mellitus tipo 2 una enfermedad de tratamiento quir\u00fargico?"}, {"id":"559990","key":"co113","value":"p\u00f3lipos de la ves\u00edcula"}, {"id":"559991","key":"co113","value":"ruptura espl\u00e9nica espont\u00e1nea asociada a linfoma perif\u00e9rico de c\u00e9lulas t, presentaci\u00f3n de un caso y revisi\u00f3n de la literatura"}, {"id":"559992","key":"co113","value":"diverticulosis del yeyuno: complicaciones y manejo; reporte de caso y revisi\u00f3n de la literatura"}, {"id":"559993","key":"co113","value":"actinomicosis abdominal y p\u00e9lvica: reto diagn\u00f3stico y quir\u00fargico para el cirujano general"} ]} it is also possible to limit by starting and ending keys. for example, this query returns the documents created by centers starting with the “co” prefix (for colombia), up to and including the “co149” center. note that the offset property tells us that 744 rows were skipped to get to the first one with key="co": $ curl -s 'http://ramalho.couchone.com/lilcouch/_design/lilacs/_view/center?startkey="co"&endkey="co149"' {"total_rows":926,"offset":744,"rows":[ {"id":"559986","key":"co113","value":"de serendipia al tratamiento quir\u00fargico de la diabetes mellitus tipo ii"}, {"id":"559987","key":"co113","value":"la academia nacional de medicina se pronuncia sobre la emergencia social, la medicina y la salud"}, [... several rows omitted ...] {"id":"560469","key":"co149","value":"diagn\u00f3stico sanitario de diversos zoocriaderos helic\u00edcolas en colombia: determinaci\u00f3n de los principales agentes pat\u00f3genos que afectan el caracol helix aspersa (o.f. muller, 1774) en cada etapa de ciclo biol\u00f3gico"}, {"id":"560470","key":"co149","value":"la salud animal y la globalizaci\u00f3n: el desaf\u00edo de pol\u00edticas sostenibles y equitativas en el contexto de los pa\u00edses en desarrollo"} ]} 5.3. indexing isis-json type 2 records: using an api now we will develop a view called au_countries to list the countries of origin of the authors cited in the sample. this entails fetching the value of the “^p” subfield of field #10. a first stab at the map function came out like this: function(doc) { var occur, subf; if (doc.hasownproperty('10')) { for (occur=0; occur&lt;doc['10'].length; occur++) { for (subf=0; subf&lt;doc['10'][occur].length; subf++) { if (doc['10'][occur][subf][0] === 'p') { emit(doc['10'][occur][subf][1], 1); break; } } } } else { emit(null, 1); } } yes, that is quite ugly. much of the ugliness comes from the fact that isis-json type 2 arranges subfields in a associative list, where a key can only be found through a linear search. so here is the next solution, much simpler thanks to the use of a custom library: function(doc) { // !code vendor/isisdm_t2/_attachments/isisdm.js var i, occurs = isis.getallsub(doc, 10, "p"); for (i=0; i&lt;occurs.length; i++) { if (occurs[i] !== undefined) { emit(occurs[i], 1); } } } the first line of the function is a special !code comment that works as an include, and is processed by the couchapp utility. this is actually a workaround because at this time couchdb does not support the use of the commonjs require function to import modules in map or reduce functions (require is supported elsewhere in couchdb, in show and list functions for example). the isisdm.js module used here is part of the isis data model apis we are developing at bireme/paho/who to handle isis records in javascript and python. currently isisdm.js defines four main functions: get(record, tag, missing) returns first occurrence of tag in record; if tag is not found, returns the value of the last argument or undefined if that is not given; getall(record, tag) returns an array with all occurrences of tag in record or an empty array if tag is not found; getsub(record, tag, key, missing) returns subfield identified by key in the first occurrence of tag in record; if tag or key are not found, returns the value of the last argument or undefined if that is not given; getallsub(record, tag, key, missing) returns an array with contents of subfield identified by key in all occurrences of tag in record; we are using qunit (jquery, 2011) to perform unit tests on the isisdm.js module. here is the the test page: figure 3. qunit test results page, showing that all tests are passing. the tests for the getallsub function are expanded. this page is live at http://ramalho.couchone.com/lilcouch/_design/lilacs/tests.html now, back to the map function, note that the call to emit uses the subfield occurrence as key and the value is just a number 1: emit(occurs[i], 1);. this is because the intent of this view is to produce an aggregate count of each different key. to achieve this, we need a reduce function to sum the values emitted, like this: function(keys, values, rereduce) { return sum(values); } now that our au_countries view has both a map and reduce function, we can query it: $ curl -s 'http://ramalho.couchone.com/lilcouch/_design/lilacs/_view/au_countries' {"rows":[ {"key":null,"value":2368} ]} the simplest, no-argument request returns null as the key and 2368 as the value. this is the result of the reduce function: 2368 is the sum of all the 1’s emitted by the map function. in other words, there are 2368 occurrences of the #10 tag with a ^p subfield. the reduce function can be disabled if the reduce=false option is given in the request. below are the first 5 and the last 5 lines of querying the au_countries view without applying the reduce. what you see is just the result of the map function: $ curl -s 'http://ramalho.couchone.com/lilcouch/_design/lilacs/_view/au_countries?reduce=false' | head -5 {"total_rows":2368,"offset":0,"rows":[ {"id":"560162","key":"\u00c1frica do sul","value":1}, {"id":"560162","key":"\u00c1frica do sul","value":1}, {"id":"560013","key":"alemania","value":1}, {"id":"559596","key":"algeria","value":1}, $ curl -s 'http://ramalho.couchone.com/lilcouch/_design/lilacs/_view/au_countries?reduce=false' | tail -5 {"id":"560034","key":"usa","value":1}, {"id":"560325","key":"usa","value":1}, {"id":"559604","key":"venezuela","value":1}, {"id":"559957","key":"venezuela","value":1} ]} a more interesting result can be obtained if we pass the group=true option: $ curl -s 'http://ramalho.couchone.com/lilcouch/_design/lilacs/_view/au_countries?group=true&limit=10' {"rows":[ {"key":"\u00c1frica do sul","value":2}, {"key":"alemania","value":1}, {"key":"algeria","value":1}, {"key":"argentina","value":289}, {"key":"austr\u00e1lia","value":2}, {"key":"bolivia","value":1}, {"key":"brasi\u00e7","value":1}, {"key":"brasil","value":1185}, {"key":"brazil","value":26}, {"key":"brsil","value":1} ]} the result set above, limited to 10 rows, shows the number of occurrences of each key. obviously there are four entries for brazil, with alternate and incorrect spellings. this would have to be dealt with elsewhere. the key point here is that the reduce function and the group option work together to produce aggregate results, similar to the ones we can produce with the sql group by clause in a relational dbms. 6. results and conclusion 6.1. results we have identified in couchdb and mongodb two modern, open source database systems which are suitable for semistructured records like those defined by the iso-2709 standard and the isis family of systems, serving the needs of marc and lilacs datasets. furthermore, we created a tool to convert isis records from the iso-2709 format to json documents suitable for loading into couchdb (or mongodb, though we have not shown that in this paper). we considered a number of alternative representations for isis data in json format. in this paper we used the type 2 representation which, though somewhat awkward to work with, preserves subfield ordering and allows for repeated subfields, a feature of marc records. isis fields do not have indicators, so we have not discussed how to represent them in json. one possibility would be to use special subfield markers, like “_1” and “_2”, attached to field occurrences where the indicators are present. finally, we developed a javascript library to make it easy to handle isis-json type 2 records when creating couchdb views. 6.2. conclusion these experiments and developments have shown that it is easy to import iso-2709 data into a document database like couchdb or mongodb. after doing so, it becomes almost trivial to create web services to publish the data, using just javascript, particularly in couchdb thanks to its native support for json over http. while the semistructured data model was only formalized in the mid 1990’s, the iso-2709 and isis record formats have always been concrete, albeit limited, examples of it. research into that model includes results such as algorithms to extract a formal schema from actual datasets, methods for dealing with shared or duplicate data, and a normal form adapted to semistructured schemas (tok, 2005). we have much to learn and apply from semistructured data research into our daily work with bibliographic records. at bireme/paho/who we continue investigating the challenges and opportunities of converting from the isis legacy systems to modern open source document databases. meanwhile, we are also developing new applications – not limited to the isis data model and legacy data – using couchdb, javascript, python and the pyramid framework. pyramid is a new web framework resulting from the merger of the pylons and repoze projects, and it includes deform, a form generation and validation library with strong support for repeating fields and nested forms, important features when building interfaces for semistructured data entry (pylons, 2011). these new developments allow us to think about how we want the lilacs bibliographic database to operate in the year 2015, when its 30th anniversary will be celebrated. notes 1 the qualification “flat relational model” is used here because there is actually a controversy about the meaning of the first normal form. c. j. date states that the atomicity requirement of the 1nf is pointless, and that since “types can be anything” it follows that “all relations are in first normal form by definition” (date, 2005, p. 37). but e. f. codd did define normalization as the process of removing non-simple domains from relations (codd, 1970), and current database textbooks define the 1nf as codd did (elmasri, 2006; silberschatz, 2006). leaving aside the theory, although some rdbms – like postgresql and oracle – support array values and even user-defined composite field values, modern database-independent access methods, such as the ruby on rails activerecord orm, often target the lowest common denominator, and that is the flat relational model. 2 “the structure, or ’empty container’, the content designators (tags, indicators, and subfield codes) used to explicitly identify or additionally characterize the data elements, and the content, the data itself (author’s name, titles, etc.) are the three components of the [marc ii] format.” (avram, 1975) 3 the term “semistructured” often appears as “semi-structured” in the literature. searching for both spellings is recommended because google returns more results for the hyphenated spelling, but some key proponents of the model use just one word (abiteboul, 1999; tok, 2005; suciu, 2001). here i use one word, but kept the source spelling in quotations. 4 json is javascript object notation, a data exchange format described as a light-weight alternative do xml (crockford, 2006b). since 2006 json is an internet standard, formalized by rfc4627 (crockford, 2006a). 5 in marc records, indicators are two single-digit positions located between the tag number and the field content, which have different uses depending on the tag number. 6 isis datasets sometimes contain repeated subfields due to typos and processing errors. however, the isis formatting language, used to generate indexes and displays, is only capable of extracting the first occurrence of a subfield. 7 from the xml standard (w3c, 1998): “an element type has mixed content when elements of that type may contain character data, optionally interspersed with child elements.” mixed content also exists in html and in sgml, their common ancestor. 8 here we mean “durability” as in the “d” in the acid database properties, meaning that once an insert or update is committed, it is written to disk. for a discussion on why the designers of mongodb initially compromised on single-server durability, see “what about durability?“ (mongodb, 2010). update: as of march 16, 2011, mongodb 1.8 is released and includes write-ahead journaling. “with journaling enabled, crash recovery is fast and safe” (mongodb, 2011). 9 “an object is an unordered collection of zero or more name/value pairs” (crockford, 2006a). 10 association lists should not to be confused with associative arrays, such as those in php. a php associative array is like a hash in perl or ruby, a python dictionary or a javascript or json object. in contrast, an association list, or alist, is not a primitive type in those languages, but can be built as an array or list of key-value pairs, where each pair is also an array or list. in python, a more readable and convenient representation would be a list of tuples: [(key1, value1), (key2, value2), ...]. association lists originated in lisp and are common in scheme (mit, 2010). 11 as far as we know, apart from http, the only other way to interact with couchdb is via hovercraft, an erlang library which provides direct access to the database. 12 couchdb indexes are generated by “view servers” which can be written in any language. an erlang view server is bundled with recent versions of couchdb, but is not enabled by default. python can be easily configured for that purpose, and java is also known to be used. a list of view server implementations can be found at http://wiki.apache.org/couchdb/view_server. 13 the numbers in the first column (98, 10, 38 in the example) identify a specific index, usually coinciding with the indexed tag number, but not necessarily. some versions of isis support advanced querying using those numbers, others do not. references abiteboul, serge; buneman, peter; suciu, dan. data on the web: from relations to semistructured data and xml. san francisco: morgan kaufmann, 1999. anderson, c.; chesneau, b. couchapp: standalone couchdb application development made simple, 2009 [cited 2011 mar. 26]. available from: https://github.com/couchapp/couchapp/. apache foundation. the apache cassandra project [cited 2011 mar. 26]. available from: http://cassandra.apache.org/. apache foundation. the apache couchdb project [cited 2011 mar. 26]. available from: http://couchdb.apache.org/. avram, henriette d. marc: its history and implications. washington, dc: u.s. government printing office, 1975. barbieri, heitor. bruma (java library source code). são paulo: bireme/opas/oms, 2010 [cited 2011 mar. 26]. available from: http://reddes.bvsalud.org/projects/isisnbp/browser/bruma. bireme/opas/oms. diccionario de datos del modelo lilacs versión 1.6a. são paulo: bireme/opas/oms, 2008 [cited 2011 mar. 26]. available from: http://bvsmodelo.bvs.br/download/lilacs/lilacs-5-dicionariodados-es.pdf bireme/opas/oms. isis-json types. são paulo: bireme/opas/oms, 2010 [cited 2011 mar. 21]. available from: http://reddes.bvsalud.org/projects/isisnbp/wiki/isis-json_types. bireme/opas/oms. isis-dm: the isis data model api. são paulo: bireme/opas/oms, 2010 [cited 2011 mar. 29]. available from: http://github.com/bireme/isisdm. chang, fay; dean, jeffrey; ghemewat, sanjay; et al. bigtable: a distributed storage system for structured data. in: osdi’06: seventh symposium on operating system design and implementation, seattle, wa, 2006 [cited 2011 mar. 21]. available from: http://labs.google.com/papers/bigtable.html. codd, e. f. a relational model of data for large shared data banks. communications of the acm, v. 13, n. 6, p. 377-387, jun. 1970 [cited 2011 mar. 26]. available from: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.98.5286&rep=rep1&type=pdf. crockford, d. rfc4627: the application/json media type for javascript object notation (json). the internet society, 2006 [cited 2011 mar. 21]. available from: http://tools.ietf.org/html/rfc4627. crockford, d. json: the fat-free alternative to xml. json.org, 2006 [cited 2011 mar. 21]. available from: http://www.json.org/fatfree.html. date, c. j. database in depth: relational theory for practitioners. sebastopol: o’reilly media, 2005. decandia, giuseppe; hastorun, deniz; jampani, madan; et al. dynamo: amazon’s highly available key-value store. in: proceedings of the 21st acm symposium on operating systems principles, stevenson, wa, 2007. dirolf, m. bin­ary json. bsonspec.org, 2010 [cited 2011 mar. 21]. available from: http://bsonspec.org/#/specification. elmasri, r; navathe, s. b. fundamentals of database systems. 5th ed. reading, ma: addison-wesley, 2006. hopkinson, alan. cds/isis: the second decade. information development. february 2005 vol. 21 no. 1 p.31-37 [cited 2011 mar. 26]. available from: http://eprints.mdx.ac.uk/2700/3/isisseconddecade.pdf. jquery project. qunit, 2011. [cited 2011 mar. 21]. available from: http://docs.jquery.com/qunit lopes, francisco. historia_isis.doc (internal memo). são paulo: bireme/opas/oms, 2010. merriman, dwight. comparing mongo db and couch db. mongodb.org. [cited 2011 mar. 21]. available from: http://www.mongodb.org/display/docs/comparing+mongo+db+and+couch+db”>. mit. mit/gnu scheme – association lists. cambridge, ma: massachusetts institute of technology, 2010 [cited 2011 mar. 22]. available from: http://web.mit.edu/scheme_v9.0.1/doc/mit-scheme-ref/association-lists.html. mongodb.org. what about durability? mongodb blog, 2010 [cited 2011 mar. 22]. available from: http://blog.mongodb.org/post/381927266/what-about-durability. mongodb.org. mongodb 1.8 released. mongodb blog, 2011 [cited 2011 mar. 22]. available from: http://blog.mongodb.org/post/3903149313/mongodb-1-8-released. pylons project. deform. agendaless consulting, 2011 [cited 2011 mar. 22]. available from: http://docs.pylonsproject.org/projects/deform/dev/. ramalho, l. lilacs on couchdb, 2011 [cited 2011 mar. 26]. available from: https://bitbucket.org/ramalho/lilcouch silberschatz, a.; korth, h.; sudarshan, s. database system concepts, 5th ed. new york: mcgraw-hill, 2006. suciu, dan. managing xml and semistructured data – lecture series (digital slides). seattle: department of computer science & engineering, university of washington, 2001 [cited 2011 mar. 21]. available from: http://www.cs.washington.edu/homes/suciu/courses/590ds/ suciu, dan. semi-structured data model. in: liu, l.; özsu, m. t. encyclopedia of database systems: springer, 2009. p. 2601-2605. thomale, jason. interpreting marc: where’s the bibliographic data? code4lib journal, issue 11, 2010-09-21 [cited 2011 mar. 26]. available from: http://journal.code4lib.org/articles/3832. tok, wang ling; lee, mong li; dobbie, gillian. semistructured database design. boston: springer science, 2005. w3c. extensible markup language (xml) 1.0 (fifth edition), 2008. [cited 2011 mar. 26]. available from: http://www.w3.org/tr/rec-xml/#sec-mixed-content about the author luciano ramalho was designing large-scale web publishing systems before the netscape ipo and the first release of internet explorer. he has a b.s. in library sciences from the university of são paulo, and is a software development supervisor at bireme/paho/who, a digital library that is part of the knowledge management and communication area of the pan american health organization. his english-language blog is at http://standupprogrammer.blogspot.com. subscribe to comments: for this article | for all articles 20 responses to "from isis to couchdb: databases and data models for bibliographic records" please leave a response below: jrochkind, 2011-04-12 it’s interesting to compare the json format you have arrived at to an attempt from ross singer to standardize a marc-in-json format: http://dilettantes.code4lib.org/blog/2010/09/a-proposal-to-serialize-marc-in-json/ i have personally not yet compared them to see how they differ. since the isis format you are working with is based on iso-2709, like marc, it is a similar domain, although different in some ways. it might be nice to try and arrive at a standard that meets both needs. miles fidelman, 2011-04-12 would not z39.50, or dublin core be a better starting point for mapping bibliographic records into json – particularly for human readability? ross singer, 2011-04-12 to follow up on jonathan’s comment, there are a few proposals floating about to serialize marc in json, all of which have their own relative merits. bill dueber came up with marc-hash: http://robotlibrarian.billdueber.com/new-interest-in-marc-hash-json/ which is extremely similar to isis-json type 2. the rationale was that serialization/parsing would be extremely fast and it was 100% roundtrippable (which is a big deal in marc circles). andrew houghton created marc-json: http://www.oclc.org/developer/content/marc-json-draft-2010-03-11 which is very heavily inspired by marcxml. it is also very well documented and 100% roundtrippable (ok, all of the current proposals are). both of these proposals have their advantages, but neither lend themselves terribly well to jsonpath type queries (which is useful for the json-based document stores), which is why i drafted marc-in-json: http://dilettantes.code4lib.org/blog/2010/09/a-proposal-to-serialize-marc-in-json/. it’s considerably more verbose than bill’s a little less (maybe) than andrew’s. i’m not sure andrew’s is supported in any openly available marc parsers; marc-hash and marc-in-json are both available in file_marc (php) and ruby-marc. i definitely second jonathan’s statement that, since the formats are so similar, it would be really nice to find some common ground between the two formats in json. that said, there didn’t seem to be much interest in formalizing a marc-json serialization in this (extremely long) code4lib mailing list thread: http://www.mail-archive.com/code4lib@listserv.nd.edu/msg08077.html luciano ramalho, 2011-04-12 @jrochkind and @ross: thanks for the pointers. i found out about marc-json only after i had submitted the paper. it is highly relevant, i will study the proposals and i will present a virtual lightning talk http://wiki.code4lib.org/index.php/virtual_lightning_talks about handling one of the marc-json variants in couchdb. luciano ramalho, 2011-04-12 @miles: thanks for your comment. we hate numeric field tags as much as anyone else. but our goal was not to map bibliographic records in general to json, but to convert a legacy database with a well documented schema and 25 years of records in it. we have quite a few databases like that one. the idea is to map as directly as possible the original records to json, so that we can import it all into couchdb and then have a much richer set of tools to work with the data and eventually convert it to some new schema which will not have numeric field tags. like i mentioned in the last paragraph, we are also developing new projects that do not depend on supporting legacy schemas, and in those cases we are using schemas that are much more human readable as you can see here: http://reddes.bvsalud.org/projects/scielo-books/wiki/datamodel miles fidelman, 2011-04-13 @luciano: point taken. on the other hand, there are fairly direct mappings between numeric marc fields and various human-readable dublin core attributes – it seems like there would be a lot of utility in adding in the human-readable attributes when importing into couchdb. it would make the resulting database a lot more useful – e.g., if you did it right, adding an sru/cql query interface would be pretty straightforward in couchdb. mapping bibliographic record subfields to json « jakoblog — das weblog von jakob voß, 2011-04-13 […] current issue of code4lib journal contains an article about mapping a bibliographic record format to json by luciano ramalho. luciano describes two approaches to express the cds/isis format in a json […] jakob voss, 2011-04-13 thanks for the interesting article! i compared several methods to encode ordered subfields in a short blog article. ross singer, 2011-04-13 miles, i’m not sure the advantage that storing human readable attribute keys in the data store brings: to use dublin core, you’d be flattening the model, changing semantics and (as a result) losing round-tripability. since (i think) the intention is to still be able to share isis records amongst organizations, this would be lost by mapping to some other metadata format. you can’t “search” couchdb (without a map/reduce function), anyway, so you’d need some abstraction layer to translate sru/cql to something usable in the first place. at that point you can translate human readable terms into whatever your local representation uses without introducing any lossiness into your data model. gabriel farrell, 2011-04-13 stepping away from marcjson formats for a sec (though i will say marc-in-json has my vote — well done ross), my plan for marc imports into couchdb is to create each document with dublin-corish fields as required (“title”, “author”, etc.) and attach the original marc record to the document as a binary. i figure that way i don’t have to care about round-tripability — if i ever need another piece of metadata i can re-parse the original. miles fidelman, 2011-04-13 @gabriel – that’s sort of what i was thinking. @ross – maybe it’s my background in protocol engineering – one of the primary tenets of early protocol design was to make everything both human and machine readable – makes for good error messages and diagnostics (e.g., you can telnet to a web site and, at least to an unencrypted port, and exchange commands and read the protocol responses) – the same principle is implicit in restful apis as opposed to soa-style web service interfaces – generally simplifies things all around as to searching couchdb – i was thinking that if the database is pre-processed, once, then writing map-reduce functions (e.g., for sru/cql queries) becomes pretty trivial though, in thinking it all through, i’d think that something like cassandra would be a better way to handle metadata luciano ramalho, 2011-04-15 @miles: implementing sru/cql in couchdb would be a very interesting project. it would require some (a lot) of preprocessing of the database, but then again any indexed dataset is preprocessed by definition. as for cassandra, it is an order of magnitude mode complex than couchdb, and it would require lots of additional coding and processing to make it support flexible queries as well. the best thing about couchdb is that it is very simple, but it does what it is designed to do very well. miles fidelman, 2011-04-16 @luciano: point taken re. the complexity of cassandra. couchdb certainly has all a nice feature set, and it’s easier to write restful applications on top of it. i was thinking that a key-value datastore might scale better for larger datasets. if not cassandra, maybe riak (restful interface, fairly straightforward to deploy). mysipisis pro with nosql database « bayu-beit, 2011-05-13 […] kutipan salah satu artikel yang mendiskusikan nosql bakal cocok untuk digunakan sebagai pangkalan data bibliografis […] de la base de datos clásica al nosql | tramullas.com, 2011-05-17 […] que ofrece una nosql. recientemente, ramalho ha publicado en la recomendable code4lib journal un trabajo en el cual han exportado registros bibliográficos desde isis a couchdb, con buenos resultados. en un entorno cada vez con mayor cantidad de información etiquetada en […] mysipisis » mysipisis pro with nosql database, 2011-05-31 […] kutipan salah satu artikel yang mendiskusikan nosql bakal cocok untuk digunakan sebagai pangkalan data bibliografis […] mysipisis pro with nosql database | sistem perpustakaan digital, 2011-06-05 […] kutipan salah satu artikel yang mendiskusikan nosql bakal cocok untuk digunakan sebagai pangkalan data bibliografis […] luciano ramalho, 2011-07-16 i just moved the isis2json utility to a new repository, with all needed dependencies, to make it easier to install and use: https://github.com/bireme/isis2json mace, 2011-09-20 thanks, this is the coolest article about nosql for librarians i know of, thanks mate :) helped me grasp the concepts very well. ismael k. isikel, 2013-10-03 thank you luciano g. ramalho for this article.it is a great help in learning marc that am reading from the library of congress. the title of the article from the library of congress website is “what is a marc record, and why is it important?” cheers leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – a fast and full-text search engine for educational lecture archives mission editorial committee process and structure code4lib issue 55, 2023-1-20 a fast and full-text search engine for educational lecture archives e-lecturing and online learning are more common and convenient than offline teaching and classroom learning in the academic community after the covid-19 pandemic. universities and research institutions are recording the lecture videos delivered by the faculty members and archiving them internally. most of the lecture videos are hosted on popular video-sharing platforms creating private channels. the students access published lecture videos independent of time and location. searching becomes difficult from large video repositories for students as search is restricted on metadata. we presented a design and developed an open-source application to build an education lecture archive with fast and full-text search within the video content. by arun f. adrakatti and k.r. mulla introduction e-lecturing has become increasingly popular over the past decade. there is an exponential increase in the amount of lecture video data being posted on the internet.  many universities and research institutions are recording their lectures and publishing them online for students to access independently of time and location. in addition, there are numerous massive open online courses (moocs) that are popular across the globe for their ability to provide online lectures in a wide variety of fields. the availability of online courses and ease of access has made them a popular learning tool. lecture videos and moocs are hosted on cloud platforms to make them available to registered users. most of these resources are for the benefit of the public. the majority of lecture videos are available on popular video-sharing platforms such as youtube, vimeo, and dailymotion. due to a shortage of storage servers and internet bandwidth, academic and research institutions are having difficulty maintaining video repositories. there are a great number of lecture videos uploaded to online platforms that are annotated only with a few keywords, which results in the search engine returning incomplete results. there are only a limited number of keywords available to access lectures and the search can be conducted primarily using their occurrences or tags. videos are retrieved solely based on their metadata, such as their title, author information, and annotations, or by user navigation from generic to specific topics. users can fetch the materials only based on limited options. irrelevant and random annotations, navigation, and manual annotations made without considering the video contents are the major stumbling blocks to retrieving lecture videos. existing approaches and proposed models the fast development of video data has made efficient video indexing and retrieval technologies one of the most essential concerns in multimedia management.[1] bolettieri et al. (2007) proposed a system based on milos, a general-purpose multimedia content management system that was developed to aid in the design and implementation of digital library applications.[2] the goal is to show how digital information, such as video documents or powerpoint presentations, may be reused by utilizing existing technologies for automatic metadata extraction, including ocr, speech recognition, cut detection, and mpeg-7 visual video retrieval (cbvr). yang et al. (2011) provided a method for automated lecture video indexing based on video ocr technology, built a new video segmenter for automated slide video structure analysis and implemented a new algorithm for slide structure analysis and extraction based on the geometrical information of identified text lines.[3] an approach for automated video indexing and video search in huge lecture video archives has been developed.[4] by using video optical character recognition (ocr) technology on key-frames and automatic speech recognition (asr) on lecture audio files, automatic video segmentation and key-frame identification can provide a visual guideline for video content navigation and textual metadata. for keyword extraction, both videoand segment-level keywords are recovered for content-based video browsing and search using ocr and asr transcripts as well as identified slide text. the authors developed and proposed text to speech extraction from videos and text extracted from ocr from the slide used while delivering the lecture videos.  most of these proposed ideas and developed concepts are in commercial platforms which the academic and research institutions are not in a position to be able to afford. motivation for the project in the indian scenario, the vast majority of lecture video repositories and moocs are hosted on the youtube platform and the web links are embedded in content management systems (cms) and e-learning applications. typically, these applications only search for metadata and human annotations associated with specific videos. some applications lack search engines, and the swayam (study webs of active-learning for young aspiring minds) platform is the best example of this. the user needs to browse the videos according to the subject on this platform. a commercial video lecture application focuses only on recording and organizing video lectures based on their subject and topics. the retrieval of videos from repositories is the least concern and applications neglect user concerns. the users do not find the desired information on the lecture video repositories and invest lots of time in browsing and listening to the videos to gain information. the popular open source institutional repositories systems are limited to maintaining only document and image files. the retrieval is restricted to metadata and human annotations. these limitations and restrictions led to the design and development of the educational lecture archive with more focus on the retrieval of the videos based on the content from these videos. development of content-based video retrieval system the capture of e-lecturing has become more popular and the amount of lecture video data on the web is growing rapidly. universities and research institutions are recording their lectures and publishing them online for students to access independent of time and location. on the other side, users find it difficult to search for the desired information from these video repositories. to overcome this issue, the application is designed and developed to organize the education lecture videos through the practical approach of searching videos based on the entire speech contained in the video. the application is named cbmir (content-based multimedia information retrieval). the scope of cbmir covers only educational lecture video repositories maintained and hosted by universities, r&d, and nonprofit organizations of india and is limited to speech extraction and automatic text indexing of lecture videos available in the english language. figure 1. overview of the cbmir application. technical requirement linux based os – ubuntu django – content management system: python-based web framework. python – programming language anaconda – python distribution whoosh library: whoosh is a fast, full text, python search engine library. the cbmir application has three major modules: the administrator, information processing & retrieval, and user platform. figure 2 shows a detailed workflow process for an administrator uploading video into repositories to a user retrieving the desired videos from repositories. flow chart figure 2. flow chart of the content-based multimedia information retrieval. modules of cbmir applications administrator module the administrator module allows the admin to upload the external video using hosted weblinks, and to upload the video file from a personal device or external storage device.  figure 3 shows the dashboard of the administrator module. the authorized admin will have access to the application to manage and upload the data. the admin can view / edit / delete the videos uploaded using the option to manage the content library. the admin can add a new user and assign a limited admin role to distribute the workload of uploading the content in cases when they are creating large educational repositories. the admin was given the privilege to add / edit / delete the text of transcript uploaded into the database. figure 3. dashboard of administrator module. information retrieval processing module the following are the step-by-step processes for the information processing and storage of videos with the cbmir application. figure 4 shows the status bar of the work process after uploading the external video link into the cbmir application. uploading videos download the video from external source: video will be downloaded to the application server from an external server or from the device using youtube api to access the youtube data and google api for accessing the youtube data for login credentials. download the videos from the device: video uploaded from personal devices or external storage devices using python library. video to audio: video file will be converted to audio using the ffmpeg python library audio to text: audio file will be converted to a text file with speech recognition using pocket-spinx python library with an acoustic model text segmentation: whole text will be broken into multiple segments based on the timings. automatic indexing and searching: the text file will be auto-indexed using the whoosh library figure 4. the status bar of processing of information and storage. user module the user module acts as a search engine, where the page contains a search box. it shows a dashboard below the search box that displays the total number of videos and subject, or topic-wise, collections. the user is allowed to search desired information using keyword terms. search results are displayed based on the word occurrence. the whoosh library converts all audio files into a text file, auto indexes these files, and makes a fast searchable format. the keyword term will be searched in the database and a list of videos will be displayed. the video starts playing on a single click on a specific video. figure 5. display of search result of the keyword speech. the whole text will be broken into multiple segments based on the speech timing and auto indexed into the database. with the advantage of segmentation, the video will start playing on the specific keyword or phrase searched by the user. figure 6. display of search results based on the time segmentation. unique features of applications: open source application: the cbmir application is designed and developed based on the open source web application and databases. the source code of the application will be shared on a popular source code distribution platform for further development from the community. fast and full-text search engine: searching through a full-text database or text document is referred to as a full-text search. a search engine is embedded within cbmir that analyses all the words contained within a document and compares them to the search criteria specified by the user during the search process. searching a huge converted text file using a full-text search ensures fast retrieval of results for large numbers of documents. the search engine provides efficient search results that are accurate and precise in all fields. domain based lecture video repository: the cbmir application allows subject-based lecture video archiving for academic institutions. this repository provides consolidated subject-specific lectures and helps users to spend less time searching on popular video sharing platforms for lectures. no video advertisements on the application: the video on the cbmir application will not play advertisements. popular video sharing platforms typically include advertisements at the beginning and middle of their videos. lightweight, less storage space, and unlimited video upload: the cbmir application is very lightweight and it can extract speech, convert it into audio, and save text files stored into databases; usually it requires less storage space. there is no limitation in uploading video into the application. text segmentation for retrieval of video content: the application divides converted text from audio into sets of words along with video timestamp. the video is played from the specific timestamp depending on the user desired search term. conclusion: due to covid-19, e-lecturing became a part of the academic community for all levels of education. academic institutions are finding it difficult to archive lecture video on internal servers and hosting on popular video sharing platforms for future use. users find it difficult to search the desired video information from the larger video repositories. the search function is restricted only to metadata, human annotation, and tags of a particular video. to overcome this problem, the cbmir application has been developed using open source technology to build educational videos repositories based on the relevant subject, focusing on fast and full-text search of the video content. the current cmbir application is limited to converting the speech to text in the english language only. the further development of the application includes converting speech to text for indian regional languages, translating converted transcripts into indian regional languages, including voice search options, text extraction using ocr technology, finding objects from the videos, creating a dashboard on the user module, etc. the source code of the cbmir application is being submitted to the authors’ university in order to fulfill requirements for a degree and will be distributed on github under a creative commons cc by-nc license after completion of the degree. reference [1] saoudi, e. m., & jai-andaloussi, s. (2021). a distributed content-based video retrieval system for large datasets. journal of big data, 8(1). https://doi.org/10.1186/s40537-021-00479-x [2] bolettieri, p., falchi, f., gennaro, c., & rabitti, f. (2007). automatic metadata extraction and indexing for reusing e-learning multimedia objects. proceedings of the acm international multimedia conference and exhibition. https://doi.org/10.1145/1290067.1290072 [3] yang, h., siebert, m., lühne, p., sack, h., & meinel, c. (2011). lecture video indexing and analysis using video ocr technology. proceedings – 7th international conference on signal image technology and internet-based systems, sitis 2011. https://doi.org/10.1109/sitis.2011.20 [4] yang, h., & meinel, c. (2014). content based lecture video retrieval using speech and video text information. ieee transactions on learning technologies, 7(2). https://doi.org/10.1109/tlt.2014.2307305 about the authors arun f. adrakatti (arun@iiserbpr.ac.in) is assistant librarian at the indian institute of science education research (iiser) in berhampur, odisha, india. k.r. mulla (krmulla@vtu.ac.in) is a librarian at visvesvaraya technological university in belagavi, karnataka, india. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – but then you have to make it happen mission editorial committee process and structure code4lib issue 28, 2015-04-15 but then you have to make it happen librarianship as a profession has a strong commitment to diversity and tends to attract professionals ethically inclined to champion inclusion. the authors, both from historically underrepresented populations in library information technology, have a half-century of combined experience in the field and have held positions ranging from technician, systems librarian, instructional technologist, head of circulation, and digital scholarship and services librarian to associate dean in an academic library. the authors share their experiences and discuss how diversity and inclusion must be embraced at the individual level in order to develop a culture of diversity within an organization and to attract and retain diverse technology teams. internal commitments to supporting a diverse environment are ultimately critical to recognizing, assessing, and fulfilling the needs of patrons. the authors identify and detail individual and grassroots efforts that have led to library technology programming for underserved populations, including programs involving outreach to diverse student and prospective student communities over the course of their careers. they reflect on strategies to create and retain a diverse technology group within the library and to advance and support diversity within the day-to-day work environment. they posit that a mix of experiences is necessary to advocate for access to underrepresented patron populations and to negotiate and implement a truly diverse environment with regard to ethnicity, gender, age, and socioeconomic background. by james williams iii and jolanda-pieta van arnhem introduction this article is for administrators, supervisors and managers who are seeking multiple perspectives on creating a more diverse staff or reviewing their current diversity plan and activities, as well as anyone interested in diversity in librarianship. the authors place in context the broader issues and challenges faced in creating a diverse workforce and articulate from personal experience both the rationale and the benefits of having a diverse staff. innovation, whether in technology or the services enhanced by technology, requires diverse perspectives and perceptions. issues of representation and agency, starting with basic assessment of user needs, must be examined from all angles. diversity brings unique perspectives based on personal experiences that help organizations and institutions consider a wider worldview and develop a culture of inclusiveness and active questioning. learning from personal experience the authors, both from historically underrepresented populations in library information technology, have a half-century of combined experience in the field and have held positions ranging from technician, systems librarian, instructional technologist, head of circulation, and digital scholarship and services librarian to associate dean in an academic library. williams, an african-american, has served in libraries in varying capacities involving circulation and stack maintenance since elementary school, never thinking in those early years that he would make it a career. after graduating from his undergraduate studies in 1995 with a degree in philosophy, williams worked as a library technical assistant (lta) in the library’s circulation department. williams wanted to apply what he learned in computer science courses he had taken as an undergraduate student, but library administration expressed concerns over his ability to negotiate and implement complex library systems without a degree in computer science as they were transforming into networked services dependent on campus information technology (lynch 2000). one forward-thinking department head saw williams’ talents and desire for lifelong learning and was convinced that he could make an impact by helping others learn how to use and implement rapidly changing technologies. williams sought to broaden the access of library information and services as the world wide web continued to transform and inform the role of the information professional as well as the organization (rice-livey and racine 1997). williams was given the opportunity to move to an entry-level position in the technical services department where he was able to gain hands-on experience using, maintaining and implementing library automation systems. his continued interest in technology led him to obtain his mlis and become a systems librarian working on network administration. technical success with project implementations led him to leadership opportunities and in 2000, williams was appointed the head of the circulation department. after working for years in library automation, he was convinced that the library would continue to experience major technological changes in all departments (lynch 2000) and was concerned by the lack of access to technology, professional development and growth opportunities available for circulation staff. with equipment and training, staff quickly learned how to word-process, manage data with spreadsheets, create informational materials, and manage course materials through an online electronic reserve system. today the circulation department works closely with faculty to add course reserves to the campus learning management system, helps administer digital signage solutions, and provides assistance with copyright and accessibility of library resources. williams now serves as the associate dean of public services and continues to empower library staff by supplying them with the tools and training opportunities they need to keep them viable, productive and marketable. williams notes the importance of support he received from coworkers at every step of the way in his career, from student assistant to associate dean and has used these experiences to influence his management style, strategic planning, hiring, research and implementation of library initiatives throughout his career. van arnhem has been involved in the technology field since 1989, beginning her tech career as a contract employee for nasa’s jet propulsion laboratory. she took a hiatus from the field, primarily due to nasa budget cuts during the clinton administration (conway [date unknown]). facing downsizing, she was unable to keep up with the cost of staying current with emerging technologies and found herself frustrated with the limited opportunities for growth in the field for women in the early to mid 90s. having worked in a variety of demanding fields, she often found the tech sector unwelcoming, demanding that women work harder for less money than male counterparts. she further found it necessary to temper her accomplishments with a high degree of femininity and humility, and, like many women, had to choose between a balanced lifestyle, home and family obligations (margolis and fisher 2004). most of van arnhem’s technology skills have been self-taught and learned from experience. in an effort to acquire the technology skills she needed to stay competitive in the workforce and have access to the equipment and software required, she took a position as a temporary office manager in an academic computing department in 1998, moved to a field technician position at the university, became an instructional technologist, and ultimately took a librarian position in the digital scholarship and services department at the university. during this time she also completed her bs in education, mfa in visual arts and finally an mlis as a first generation “non-traditional” student, all while working full-time. the arduous work and school load van arnhem faced made her sympathetic to other women, non-traditional and minority students she often sees facing similar circumstances. she is happy to see an increasing awareness for the need to inspire and provide educational opportunities for girls from organizations like girls who code (about … [updated 2015]) and code savvy (organizations … [date unknown]), as well as improved access for adult learners to computer science instruction at an affordable cost from programs such as the georgia institute of technology, udacity and at&t’s online accredited master of science in computer science mooc (presidential double-down … [updated 2015]). the authors’ interest in diversity is personal and stems from the acknowledgment of the intuitive need for a conscious effort to act and operate in a way that promotes diversity. the need for more outreach to women and underrepresented populations in the field of computer science is noted by the national center for women and information technology, citing that in 2013 only twenty-six percent of the computing workforce were women, three percent were african-american, five percent were asian, and two percent were hispanic (by the numbers … [updated 2014]). the american library association’s study of gender, race and age in the field showed a small increase in the percentage of minorities working in the library profession from eleven percent in 2000 to twelve percent in 2009-2010, but notes that the field “remain[s] predominantly female and white” (diversity counts … [updated 2002]). statistical data for members of the lgbtqqaaip community appear to be non-existent in mainstream surveys, and the authors hope that organizations will provide more inclusive options in the future. these numbers, coupled with the recent disclosures by google and other companies in the tech sector such as apple, linkedin and facebook, all show that they are “struggling to recruit and retain women and minorities” (getting to work … [updated 2014]). the similarity between the tech sector reports and the numbers of women and minorities in library technology leads one to question how, with the high number of females in the field, the numbers for both women and minorities are not higher in library technology (peterson 2014). it is important to move from the idea that sexism and racism was a problem “back then” and that everyone has the same access to knowledge and resources (accardi 2013, p 1). innovation needs diversity, and diversity needs support. in “the data on diversity” nelson discusses the benefits and challenges of supporting diversity and argues, “diversity has been shown to create a cognitive and social environment that is a positive indicator for innovation.” based on the analysis, nelson also provides the caveat that “there is not a clear causal relationship shown between diversity and success,” and cites numerous challenges faced when creating effective diverse teams, including “unconscious bias, stereotype threat, exclusion from critical social networks, lack of role models and unaware managers.” nelson posits, “these dynamics may have real-world consequences” (nelson 2014, p 88-89). this same sentiment is echoed in president barack obama’s state of the union address, where he states, “[t]oday, women make up about half our workforce. but they still make 77 cents for every dollar a man earns. that is wrong, and in 2014, it’s an embarrassment. a woman deserves equal pay for equal work” (president barack obama’s … [updated 2014]). the same can be said about most minority populations when equal access to education and resources and barriers to workforce entry are taken into consideration (weise and guynn 2014). google’s own report on improving diversity within the field found that encouragement, self perception, academic exposure and career perception are actionable factors that can increase participation (women who choose … [updated 2014]). obama’s call for payroll equity and google’s report on diversity echo john dozier, chief diversity officer for the university of south carolina’s notion that “diversity is a verb,” underscoring the need for active, rather than passive, approaches to institutional inequalities. author, mit instructor, and macarthur fellow junot diaz summed it up in his exaltation to male authors writing female characters: “i think the average guy thinks they’re pro-woman, just because they think they’re a nice guy and someone has told them that they’re awesome. but the truth is far from it. unless you are actively, consciously working against the gravitational pull of the culture, you will predictably, thematically, create these sort of [expletive] representations” (qtd. in fassler 2012). diaz could just as easily have been writing about any group with unrecognized privilege not understanding the issues facing people outside the group (fassler 2012). further discussion of this topic is found in the work of henrich, heine and norenzayan, who found that “[b]ehavioral scientists routinely publish broad claims about human psychology and behavior in the world’s top journals based on samples drawn entirely from western, educated, industrialized, rich, and democratic (weird) societies” (henrich et al. 2010, p 76). this tendency towards marginalization of needs not only hurts the primary victims; it blinds the very organizations that engage in practices of poor reflection and forethought, engendering institutions that fail to know themselves. heinrich et al. note the use and subsequent overrepresentation of white, western undergraduate students in university psychological studies. “researchers – often implicitly – assume that either there is little variation across human populations, or that these “standard subjects” are as representative of the species as any other population” (henrich et al. 2010, p 61). this is the same problem any large technology project that has failed to examine real use cases and do solid user testing encounters. subsequently, users whose needs are surely already met by existing systems are made up, instead of examining actual users. “gravitation pull” is a wonderfully apt way to frame problems of cultural and institutional scale. active, daily attention is needed to combat inequalities. diversity is a verb williams and van arnhem thus realize the need for a conscious effort to act and operate in a way that promotes diversity in library technology. laura weidman, founder of code2040, whose organization “aims to close the achievement, wealth, and skills gap for blacks and latinos” sums up how important it is to embody an internal commitment to support a diverse environment and promote change, stating that it requires “stay[ing] uncomfortable, and hav[ing] an impact in addressing uncomfortable issues” (gilpin 2014). people want to see people like themselves: if you want to attract a diverse community, you have to be one. in 2012, the college of charleston adopted its first diversity strategic plan, which is providing campus leadership, promoting a more diverse campus environment, and providing an open climate for dialogue about our “campus diversity realities and challenges” (diversity strategic plan … [updated 2012]). there is still a lot of work to do, and developing a culture of diversity and inclusion in the workplace and on campus over the past decade often required grassroots activities and leading by example. the authors have found that every interaction matters and have diligently worked to identify and promote efforts that have led to library technology programming for underserved populations, including programs involving outreach to diverse student and prospective student communities over the course of their careers. they have adopted a “feminist lens” in order to reduce gender, race and other bias in the workplace (accardi 2013, p 15). every campus is different and has different needs. williams and van arnhem hope that by sharing their collective experiences at their institution, readers will identify strategies and efforts that they can employ on their own campus to promote diversity. fostering diversity and inclusion in the workplace diversity leadership at the institutional level matters. a culture of diversity must be promoted within an organization to attract and retain diverse technology teams. in “comparing status and statistical theories of gender discrimination,” correll and benard note: “[e]ven when the direct effect of a single act of discrimination is small (and sometimes such effects are large), the consequences of employers’ decisions accrue over a life course. when contributions of individual women are systematically downplayed, overlooked and misattributed, women as a group disproportionately see their applications disregarded, their promotions fail to materialize and their raises are meager. the resulting gap in material rewards fuels perceptions that women are less interested in such rewards or less capable of achieving them, shoring up cultural beliefs that contribute to persistent disadvantages” (correll and benard 2006, p 112-113). wyche has similarly noted a “cement ceiling” for african-americans and other minorities (wyche 2008, p 166), and wilson subsequently underscored the importance of mentorship and social responsibility in helping those affected with skills maintenance (wilson 2014). as staff is replaced through attrition, the library must constantly be aware of potential age bias as there is a natural tendency to focus on newly minted professionals in the field. one major reason why diversity matters is that individuals want to see someone who resembles themselves participating in all aspects of the organization, but especially in technology. the assumption that many individuals are “comfortable with technology” may not be true. they may want and need someone who they feel is approachable and with whom they can identify, ask questions, and learn from. representation is important in reaching all members of a community and can also lead to interest in joining the library and information science profession. if people don’t see diversity in a field, they infer that there are limited or no opportunities for them. a recent study by cheryan, plaut, handron, and hudson sheds an interesting light on how commonly accepted computer science stereotypes (for example the white, geeky, male with a fascination for science fiction) and past and current media representations may discourage individuals, particularly females, from developing an interest in computer science or stem fields (cheryan et al. 2013, p 61). strategies for supporting diversity within the field must include acknowledgment that library pay is not only unequal with regard to gender but also often quite low in the best circumstances. library journal’s recent salary survey notes that academic librarians in the south had the lowest annual salaries in 2014, and survey respondents commented that the profession suffers from gender bias. in regards to gender and pay, girmsheid and schwarts also note that in the “female dominated field, women librarians make roughly 89% of what their male counterparts earn” (girmscheid and schwartz 2014). with many of these barriers in mind, williams and van arnhem actively advocate for the library to promote individual growth and provide staff with the skills they need to be current and marketable. when pay cannot be addressed directly, williams looks for other areas of coworkers’ day-to-day lives that can be made better. this includes simple recognition of achievements or performance, such as certificates and small monetary tokens of appreciation and recognition of outstanding achievements. williams also advocates for release time from routine activities for library staff and faculty to focus on specific projects of interest. providing exemption from desk hours or other tasks to pursue professional activities helps to affirm the library’s commitment to staff for professional development, research and lifelong learning. providing library staff and faculty with professional leave, flextime and opportunities to work from home are other ways of supporting professional development and growth. commitment to providing ongoing access to emerging technology and learning opportunities is an essential component of promoting diversity and inclusion in the library. it is important to constantly look for creative ways to enhance job skills and access to emerging technologies, whether through webinars, campus training programs, or other offerings in the community that can yield professional development. williams and van arnhem continuously strive to find opportunities to present library staff and faculty with learning opportunities that they may not receive otherwise. van arnhem has worked one-on-one to design learning programs tailored to specific departments’ or staff’s expressed goals. professional development topics have included video editing, podcasting, captioning and accessibility, social media, technology classroom equipment, and digital scholarship tools. willams continuously strives to seek funding for and provide access to new technology in all departments and has been instrumental in creating a staff training committee with a small annual budget. the committee is responsible for bringing in speakers and providing webinars and workshops. staff are encouraged to express their interests and needs. supervisors are encouraged to promote activities and facilitate attendance and participation. in addition to providing professional development opportunities and access to emerging technology within the library, it is important for library faculty and staff to participate in the field by attending and presenting at professional conferences. though professional development resources are limited, library staff who show an interest in the field of librarianship or wish to know more about it have been sponsored to attend the american library association annual conference and report back what they learned from the experience at library wide staff meetings. staff are also encouraged to submit travel requests for state and regional library conferences and pursue staff development grants. the library has also made a concerted effort to offer tuition assistance to help staff begin or continue their education in the library field as funds are available. bartlett notes that formal and informal mentoring for library professionals cannot be underestimated and argues that mentoring activities create a positive organizational culture, contribute to professional development, and may aid in filling library leadership positions as the baby boomer generation begins to retire and leave the workforce (bartlet 2013). williams advocated for a pilot staff mentorship program in 2012. interested staff provided an area of interest and described their current and future goals. five staff members were selected and paired with senior staff members who worked towards outlining projects and activities that increased the mentee’s knowledge in a specific area. participants agreed to share their experiences and provide feedback on the pilot program. due to numerous position searches, including a new dean of libraries, and major library renovations, the pilot is still in its infancy and is scheduled to fully resume in 2015. initial feedback from participants has been positive. williams also advocated for a small group of senior faculty to provide guidance and answer tenure and promotion questions from junior faculty in order to help alleviate some of the anxiety associated with the tenure process. junior faculty are more productive and feel supported knowing that the library is behind them, supporting collaboration instead of competition. all library faculty and staff serve on library committees, which helps to promote a culture of diversity and inclusion. committee work can provide a vehicle for library staff and faculty to learn from and help each other, share expertise, and enable members to reach individual professional goals. as noted by nelson, inclusion of all library staff on committees is one way to provide access to social networks and role models and can lead to innovative projects and services (nelson 2014). a few recent notable outcomes from committee service by library faculty and staff at the college of charleston libraries include a banned books week advocacy program, library-wide student worker orientation events, and an ad-hoc social media and digital signage committee, which includes representatives from all libraries and departments. williams and van arnhem note that the key ingredients to successful committee outcomes are providing support, including stipends, equipment, release time, and training for additional skills development. although many of these efforts may seem to be minor interventions, they play a vital part in developing, as simmons-welburn notes, a team-based organization. williams and van arnhem agree that many of these interventions fall within the realm of “workplace civility,” recommended by simmons-welburn to help build community, and that doing so provides positive benefits for the library and the institution (simmons-welburn 2004). championing participation and awareness in students accardi states that “[s]ystems of cultural and societal oppression are frequently replicated and perpetuated in the educational setting” and promotes active engagement with academic feminism in order to bring value to personal experience and raise consciousness about oppression (accardi 2013, p 30). the authors agree and argue that it is important for the library and its staff to be seen participating in promoting diversity education, outreach, and initiatives. an important role of the library is to provide students, regardless of age, race, gender, or socioeconomic background, with encouragement and academic exposure to successful strategies, proven instruction, and tools that they can employ throughout their education and professional lives. with this in mind, the library promotes outreach not only to campus groups but also to high schools in the local community. from an institutional perspective, community outreach initiatives and partnerships can aid in diversifying student recruitment efforts. the us news diversity index rating for the college of charleston, taken from the student body during the 2013-2014 school year, is 0.29, with 1 indicating a more diverse student population and 0 representing complete homogeneity (us news… [updated 2015]). reaching out to local high schools is one way to build partnerships, connect with the community, and show students that the college is interested in creating a diverse student body. williams, with the help of a library donor, worked to provide local high school students from an underperforming and underfunded nearby high school with access to laptops with reading skills, sat, act and ged prep software. this was done in conjunction with library instruction on how to conduct online research following research guides set up by the librarians to highlight local resources accessible to high school students, while ensuring that students are questioning the relevance, authority and timeliness of these sources in accordance with principles of information literacy. williams, van arnhem and others participated on panels assisting in the critique of senior portfolios for another, predominantly african-american, area high school, which included a critique of their public speaking skills and use of presentation software. librarians also provided professional development workshops for high school teachers on information literacy topics delivered in a tailored summer program. unfortunately, these initiatives were not as successful as williams had hoped due to logistical issues. the high school students had difficulty securing transportation to the library, the senior portfolios presentations were moved to a time that was difficult for library staff to attend, and donor support for professional development activities was not ongoing. however, in all cases, feedback about the library’s interactions and outreach efforts were positive. since the formal adoption of the college’s diversity strategic plan, campus outreach efforts have had a more directed focus, and university administration is making a concerted effort to increase its diversity recruitment efforts and face the challenges of bringing more minority students to campus (knich 2015). in addition to creating a welcoming atmosphere for prospective students, it is also important to improve and support student academic success at the university. this can include a number of facets, from library instruction based around critical pedagogy to working with programs and departments, promoting access, drawing attention to needs of different student populations, and giving students a voice. williams and van arnhem actively seek to promote library partnerships with campus constituents. the authors routinely work with campus multicultural student programs and services, including trio programs and the speedy consolidation and transition program (spectra) to promote equal opportunity and access to emerging technologies, resources, services and support. workshops and programs focusing on information literacy and digital fluency are designed and developed in order to provide a firm grounding in the processes, tools, and digital scholarship skills required to successfully complete academic projects. librarians also collaborate with first year experience, the realizing educational and career hopes (reach) program, and various campus student support services departments. williams encourages all library staff to volunteer and participate in diversity training opportunities and service activities in order to help provide a safe and comfortable atmosphere at the library. as a result, the library has numerous safe zone allies, participates in student mentoring programs, and organizes and participates in local community outreach projects. a few notable projects that have occurred as a result are the library’s annual teacher’s supply closet, which provides donated school supplies to needy families, and food for fines, where library patrons can pay library fines with food that is donated to the local food bank. access to technology resources is a primary concern of williams and van arnhem, particularly for underrepresented student populations on campus. the university does not have a computing requirement, making accessibility to library computing resources key. limited access to computing and digital scholarship tools can limit exposure to stem/steam activities and hinder entry to or advancement in these fields. to that end, the authors have written and submitted grants and communicated their concerns to library and university administration. one recent outcome of these efforts is an ipad lending program, which helps to provide access to a technologically rich environment that encourages technology literacy, information literacy, and collaborative learning. providing mobile devices for checkout, as well as tutorials and instruction, has proven beneficial to underserved populations. in order to provide students with a voice at the library, williams advocated for the creation of a student library advisory board to provide input on library technology, resources and services. now in its third year, students from the advisory board have contributed both a unique and vital perspective on issues such as collaborative learning, desired study atmosphere, technological needs, uses of current technology, and recycling. the advisory board has also contributed to the creation and development of a student video collection, advocated for longer lending periods for graduate students, successfully funded and installed a water bottle refilling station in the library, and provided recommendations for library innovations. in the future, the advisory committee will serve as a focus group and provide user testing for the library website redesign project. conclusion writing this article has provided the authors with the opportunity to evaluate previous efforts and consider new directions. admittedly the approach of seeking opportunities for improvement wherever they may arise has the drawback of being difficult to assess from a broader perspective. there is still a lot of work to do individually and institutionally; however, the college administration’s leadership is providing new opportunities to widen the discussion about campus diversity needs and challenges. moving forward, williams and van arnhem will continue to lead by example by actively participating in diversity efforts and volunteering service to diverse groups on campus and within the community. the authors will continue to assess library diversity efforts and dedicate time for research in order to learn new strategies to facilitate an open climate. williams and van arnhem will also continue to reach out to staff, facilitate dialogue, and advocate for diversity efforts at their institution. for those who agree that promoting diversity is “everyone’s work” and are interested in continuing the discussion about how to support efforts for diversity, equity and inclusion, the authors recommend joining the american library association’s diversity member initiated group (young 2015). many of the experiential strategies to raise consciousness of diversity issues that are discussed here appear straightforward and at times small in scale; however, they all share the commonality of being actionable. a simple strategy like adopting feminist pedagogy and using the search term “women in computing” can provide entry for numerous points of discussion and provide the juxtaposition of ideas, perceptions, and information to counter the replication of patriarchal systems in the education system (accardi 2013, p 37). aurora and accardi note that similar patterns of erasure appear in the literature of librarianship and “advocate for a conscious improvement of the diversity in the voices we are listening to” (peterson 2014; accardi 2013). in order to promote diversity in the library technology field, all voices must be considered, whether in the educational experience, the workplace, or the literature of the field. diversity and inclusion must be supported and championed to remove barriers and encourage participation. the smallest efforts can affect an individual’s trajectory. references accardi, m. 2013. feminist pedagogy. sacramento (ca): library juice press. 148 p. about [internet]. [updated 2015]. new york (ny): girls who code; [cited 2015 feb 11]. available from: http://girlswhocode.com/about-us/ bartlet j. 2013. paying it forward, giving it back: the dynamics of mentoring. library leadership & management 27(4):1-6. by the numbers [internet]. [updated 2014 feb 28]. boulder (co): national center for women & information technology, university of colorado; [cited 2015 feb 10]. available from: http://www.ncwit.org/resources/numbers cheryan, s., plaut, v., handron, c., & hudson, l. 2013. the stereotypical computer scientist: gendered media representations as a barrier to inclusion for women. sex roles, 69(1/2), 58-71. conway e. [date unknown]. jpl history. in: 1991 present [internet][pasadena (ca)]: jet propulsion laboratory, california institute of technology; [cited 2015 feb 12]. available from: http://www.jpl.nasa.gov/jplhistory/the90/ correll s, benard s. 2006. biased estimators? comparing status and statistical theories of gender discrimination. in: thye s, lawler e, editors. social psychology of the workplace. advances in group processes volume 23. kidlington (oxford): jai press. p. 89-113. diversity counts [internet]. [updated 2012]. arlington (va): ala office for research & statistics; [cited 2015 feb 4]. available from: http://www.ala.org/offices/diversity/diversitycounts/divcounts diversity strategic plan [internet]. [updated 2012 april 20]. charleston (sc): college of charleston; [cited 2015 feb 12]. available from: http://pcdaei.cofc.edu/diversity-strategic-plan/diversitystrategicplan2012final.pdf fassler j. 2012. how junot diaz wrote a sexist character, but not a sexist book [internet]. [updated 2012 sept 11]. [cited 2015 jan 31]. available from: http://www.theatlantic.com/entertainment/archive/2012/09/how-junot-diaz-wrote-a-sexist-character-but-not-a-sexist-book/262169/ getting to work on diversity at google [internet]. [updated 2014 may 28]. santa clara (ca): google; [cited 2015 feb 10]. available from: http://googleblog.blogspot.com/2014/05/getting-to-work-on-diversity-at-google.html gilpin g. 2014. laura weidman powers: code2040 founder. minority advocate. discomforter. [internet]. [cited 2015 feb 10]. available from: http://www.techrepublic.com/article/laura-weidman-powers-code2040-founder-minority-advocate-discomforter/ girmscheid, l. and schwartz, m. 2014. payday | lj salary survey 2014. [internet]. [updated 2014 jul 3]. [cited 2015 feb 10]. available from: http://lj.libraryjournal.com/2014/07/careers/payday-lj-salary-survey-2014/#_ henrich j, heine s, norenzayan a. 2010. the weirdest people in the world? behavioral and brain sciences 33(2-3): 61-83. knich d. 2014. increasing diversity a challenge at college of charleston. the post and courier [internet]. [cited 2015 march 9]. available from: http://www.postandcourier.com/article/20140207/pc16/140209461 lynch, c. 2000. from automation to transformation: 40 years of libraries and information technology in higher education [internet]. [cited 2015 mar 8]; jan/feb(2000):60-68. available from: https://net.educause.edu/ir/library/pdf/erm0018.pdf margolis, j., fisher, a. 2002. unlocking the clubhouse: women in computing. cambridge (ma): the mit press. 172 p. nelson, b. 2014. the data on diversity. communications of the acm [internet]. [cited 2015 mar 4]; 57(11): 86-95. available from: http://dl.acm.org/citation.cfm?id=2684442.2597886 organizations [internet]. [date unknown]. code savvy: [cited 2015 feb 10]. available from: http://www.codesavvy.org/p/about.html peterson, r. 2014. seeking a diversity of voices. code{4}lib journal [internet]. [cited 2015 january 15]. available from: http://journal.code4lib.org/articles/9345 president barack obama’s state of the union address [internet]. [updated 2014 jan 28]. washington (dc): the white house, office of the press secretary; [cited 2015 feb 14]. available from: http://www.whitehouse.gov/the-press-office/2014/01/28/president-barack-obamas-state-union-address presidential double-down: obama praises oms cs for 2nd time [internet]. [updated 2015 mar 11]. atlanta (ga): georgia tech, college of computing; [cited 2015 mar 13]. available from: http://www.cc.gatech.edu/news/presidential-double-down-obama-praises-oms-cs-2nd-time rice-livey, ml, racine, jd. 1997. the role of academic librarians in the era of information technology. journal of academic librarianship 23(1):31-41. simmons-welburn j. 2004. creating and sustaining a diverse workplace. in: simmons-welburn j, mcneil b, editors. human resource management in today’s academic library. westport (ct): libraries unlimited. p. 9-17. us news & world report education [internet]. [updated 2015]. washington (dc): campus ethnic diversity regional universities (south); [cited 2015 mar 6]. available from: http://colleges.usnews.rankingsandreviews.com/best-colleges/rankings/regional-universities-south/campus-ethnic-diversity/page+4 weise, e. & guynn, j. 2014. tech jobs: minorities have degrees, but don’t get hired. [internet]. [cited 2015 feb 7]. available from: http://www.usatoday.com/story/tech/2014/10/12/silicon-valley-diversity-tech-hiring-computer-science-graduates-african-american-hispanic/14684211/ wilson e. 2014. diversity, culture and the glass ceiling. journal of cultural diversity, 21(3), 83-89. women who choose computer science – what really matters [internet]. [updated 2014 may 26]. santa clara (ca): google; [cited 2015 jan 31]. available from: http://static.googleusercontent.com/media/www.google.com/en/us/edu/pdf/women-who-choose-what-really.pdf wyche k. r. 2008. good is not good enough and other unwritten rules for minority professionals. new york (ny): penguin. 243 p. young c. everyone’s work [internet] [updated 2015 january 15]. chicago (il): american libraries; [cited 2015 march 15]. available from: http://americanlibrariesmagazine.org/2015/01/15/everyones-work/ about the authors james williams is the associate dean, public services at the college of charleston libraries in charleston, south carolina. he oversees library internal operations, ensures high quality instruction, programming and services are delivered, and supervises the library computer technicians. jolanda-pieta van arnhem is the instructional design librarian at the college of charleston libraries in charleston, south carolina. she provides instruction for digital scholarship tools for research and classroom use in the arts and humanities. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – bringing sheet music to life: my experiences with omr mission editorial committee process and structure code4lib issue 3, 2008-06-23 bringing sheet music to life: my experiences with omr this article describes the process of digitizing sheet music celebrating pullman porters and rail travel from the 1870s-1920s. the process involves 1) digitizing sheet music, 2) running the digitized sheet music through an optical musical recognition (omr) software package, 3)cleaning up the resulting file, 4) converting it into an .mp3/midi file, and 5) tweaking it to use the voices/instruments of a music editing software program. the pros and cons of some popular omr programs are discussed. by andrew h. bullen optical music recognition i believe that music has a unique ability to give an almost visceral clarification of historical trends and events. as an (admittedly amateur) local historian, it has been frustrating to be presented with spectacular examples of sheet music that give shape and depth to history yet be totally inept at playing the tunes on a piano or other musical instrument. happily, as it turns out, through a combination of optical music recognition (omr) and music composing software, i can scan the music, "read" it to detect the notes, time signature, etc., and tweak its playback to get just the right sound i want. the process of turning sheet music into digitized files is neither straightforward nor error-free, but i believe well worth the effort. the skill set itself involved in a sheet music digitizing project is fairly basic: one needs to have some understanding of music theory, be able to read music, and be able to scan documents into an imaging software program such as photoshop. however, the software that actually does the conversion, optical music recognition software, is relatively new and the software and recognition algorithms are less than ideal. indeed, as stated in gerd castan’s page about omr, doing omr is hard. very hard. if you follow the omr science links, you will find much more theses about omr than there are public[ly] available programs. [1] as can be seen in the chart from donald byrd [2] , different omr software packages vary widely in cost and technique. none of them comes close to achieving a success rate comparable to the best optical character recognition software. bringing pullman music to life my first attempt at using omr software to bring music to life was with the pullman state historic site's pullman virtual museum [3]. the pullman state historic site has a growing collection of sheet music related to pullman travel and pullman porters. at the turn of the 20th century, pullman– and particularly the phenomenon of the pullman porter– caught the public’s imagination. a journey on a pullman car summoned up images of romance and adventure, and became a powerful symbol of new beginnings and accomplishment. an example of the pullman car as metaphor for life’s journey can be seen in the 1873 farce-comedy tourists in a pullman car written by william a. mestayer and james barton in which, as the train rambles along, the characters have adventures and romances [4]. the pullman company was also one of the largest employers of african-americans in the rigidly segregated society of 19th and early 20th century america [5]. pullman porters were generally the only african-americans that much of white society ever casually interacted with [6]. as a result, pullman porters were also used in the era’s minstrel shows; many later minstrel productions cruelly featured the porter as a main character, portraying him as a cheerfully amoral ignoramus with a wife in every city [7]. to give dimension to this aspect of lost america, we decided to digitize the sheet music in the collection in order to make it available through the pullman virtual museum. the process began by first scanning each page of the sheet music into a tiff file (no compression) at 400 dpi. i then tried to defox [8] and otherwise clean up the scanned image and clarify some of the staff lines and notes, extremal-boardwhich frequently bled and lost sharp definition during the scanning process. the next step was to use visiv’s sharpeye (http://www.visiv.co.uk/) music scanning software to actually batch the individual tiff images together into one “tune” and omr them. the sharpeye omr engine [9] is used by other omr programs and can be counted on to give fairly good results. as i stated above, however, the error rate can be fairly high. as an example, the 1880 tune tourists in a pullman car by george bowron has 280 measures (5 pages with 14 measures of music, each consisting of 4 parts). because of the condition of the paper, the conversion process resulted in 223 errors. each error might consist of missing notes, improper time signatures, not enough/too many notes per measure, etc. all had to be corrected by hand, using sharpeye’s editing features. i wanted more control of the instrumentation and playback of the final product, so i used a second program, myriad software’s harmony assistant (http://www.myriad-online.com/en/products/harmony.htm). i like to use harmony assistant because of its powerful instrumentation and playback features. i was able to “tune” the output file of my pullman tunes to sound like they were being played on the piano in the lobby of the hotel florence in the town of pullman. image 1: the hotel florence, ca. 1885 image 2: the hotel florence’s lobby, 1892. notice the minton tile floors. the above image of the lobby in the hotel florence is taken from very near the location of the actual hotel piano [10]. i can well imagine what music played in the hotel must have sounded like, played on a ever so slightly out-of -tune (again, as i imagine it) upright piano echoing through the lobby, the sound made “live” by the minton tile floor, iron columns and plaster walls. harmony assistant easily allowed me to export the tunes as .mp3 files. i have also converted them into realmedia (.rm) streaming media files. users of the pullman state historic site have the option of streaming the music or actually downloading the midi file. an added benefit to all of this digitizing is that we have been able to produce a cd for sale in the museum’s gift shop. the cost of producing the cd and information sheet is negligible, and it can be retailed for $15.00. it took approximately 40 hours to digitize the 12 tunes in the collection. a case study: using omr to bring history to life omr-ing sheet music is a very useful and worthwhile endeavor for librarians and museum staff alike, despite its time-consuming nature. it can provide a very powerful enhancement to an online exhibit, for example, or allow a library patron to download or purchase an aural “souvenir” of their visit, increasing library and museum visibility. music is an ideal tool to connect users to information, because few phenomena evoke stronger emotional responses than music [11]. as an example of how a library might use omr to bring an added dimension to a historical event, we can examine the process of digitizing sheet music using a tune that was indirectly responsible for a terrible tragedy. the music in question is the 1903 tune "in the pale moonlight," a part of the musical comedy mr. bluebeard. on december 30, 1903, 1,900 chicagoans flocked to the newly opened iroquois theater to see the latest broadway hit, mr. bluebeard [12]. a lighthearted, big budget musical spectacular like mr. bluebeard was just what the city needed. december, 1903 in chicago had been a very cold month and, with several bitter ongoing strikes, had also been a tense and acrimonious time despite the holiday season [13]. holiday theatergoers packed the opulent theater for a wednesday matinã©e. in the middle of the second act, the theater was bathed in pale blue light for the musical number “in the pale moonlight.” at about 3:15 p.m., a curtain brushed against one of the blue lights, setting the curtain immediately on fire. the fire spread rapidly out of control to the scenery flats above the stage. the almost non-existent firefighting equipment on hand was totally ineffective, and the supposedly impregnable fireproof curtain failed to lower completely. as actors and stagehands panicked and fled out the rear stage doors, the sudden gust of frigid december air from the opened doors created a huge fireball, filling the balconies with flame and superheated air. escape for much of the audience was impossible. many of the fire doors were locked, as the theater management wanted to prevent balcony patrons from sneaking down to the lower levels for better seats; in addition, what fire exits did exist were unmarked and hidden behind curtains with doors that opened inward. in a short time, 602 people had died, mostly women and children who had come to the afternoon matinã©e [14]. to this day, the iroquois theater fire remains the worst single-building fire disaster in u.s. history [15]. among the 4 cast members who perished was the up and coming star aerialist nellie reed, playing a fairy, suspended on a trapeze 100 feet above the main floor when the fireball hit. she was to have gradually descended to the stage while sprinkling rose petals on the audience as she went by. in the ensuing panic, ms. reed was forgotten. her trapeze was almost directly in the path of the fireball [16]. the last tune that she and 601 other men, women and children would ever hear sounded like this: launch in external player as a tribute to her, i have inserted a 2 beat pause towards the end of the music, right before the final dance number, where ms. reed was supposed to arrive on stage. to produce the finished digital file is a multi-step process. the first step, of course, is to secure a copy of the music. fortunately, there are several online collections of sheet music. the african-american sheet music collection at brown university (http://dl.lib.brown.edu/sheetmusic/afam/) as well as the sheet music consortium’s wonderful oai-pmh harvested collection at http://digital.library.ucla.edu/sheetmusic (harvesting the collections of library of congress, duke university, indiana university, ucla, johns hopkins, the maine music box, and the national library of australia) are great places to search. many of the collections available at the sheet music consortium’s site use pdfs; while there is omr software that will read pdfs, i have found it better to download the pdfs files and save the pages as individual tiffs. the next step is to defox the paper, cleaning up the staves, etc. in photoshop. be careful to save the tiff file with no compression (acrobat converts files to tiffs with lzw compression when you do a “save as” to tiff). image 3: digitized sheet music. one could also, of course, scan the actual sheet music. sheet music was printed on poor quality paper, so it is frequently brittle and discolored through foxing; however, scanning may be your best or only alternative.after an initial cleanup, the tiff images are ready for the first processing in sharpeye. the four pages of the sheet music are loaded into sharpeye as a “batch.” after the “read” command is activated, the batch is processed and the notes are read onto an editable music form. the errors appeared flagged by small blue triangles. image 4: omr processing results in sharpeye, before any corrections. despite the clean up of the image files, we still have 42 errors. the error number is deceptive; there will inevitably be “false positives,” notes that were read incorrectly and not flagged as errors. each measure must be carefully compared to the original. a typical error can be seen in the above screen shot. the music is written in a, yet the omr software has read the key signature with one line in the key of d and one in f. image 5: correcting the key signatures. everything in a score can be edited. by clicking on the musical object, an appropriate submenu pops up with alternative choices.after editing the music score, the file can be saved in a wide variety of formats. i saved the file as a midi file, since i wanted to open it in a second program (harmony assistant) to refine its playback characteristics.harmony assistant was developed in france, and the help menus and context-sensitive help text reflect a rather gallic approach to the english language. regardless, this is a very powerful and complex program, allowing a user to create all kinds of effects using a wide variety of different instruments. there are an eye-popping variety of additional midi instruments available for purchase at http://www.myriad-online.com/en/products/demogold.htm. it also has a virtual singer, should one wish to try out the music and lyrics [17], and an add-on called omer which acts as an omr program. image 6: changing the instrumentation in harmony assistant. after setting up the instrumentation to my liking (which i imagined would be cleanest if it sounded like a piano onstage, playing for rehearsals and gently echoing in the “deader” space of a theater), i then saved the file as an .mp3. since the resulting .mp3 file is a bit bigger than 5.1 megabytes, i also converted it to a streaming media .rm file and loaded it onto a streaming media server. other omr software programs of course, this is not the only way of using omr. there are other omr programs. a highly recommended program is smartscore x pro (http://www.musitek.com/smartscre.html). i did not purchase the software for this article (it runs to $399 usd), but the demo version seemed to have a superb recognition algorithm as can be seen below. editing seems particularly easy. image 7: initial pass through smartscore x pro editing a note or other object is simple– click on the object to be edited, select an appropriate value from a side menu, and the object is transformed. error rates with smartscore are comparable to the sharpeye omr engine, liszt. smartscore boasts better playback features– more voices and balance options– and is has a simpler and more intuitive editor. it is also more flexible in actually acquiring images to omr. it can read tiff images with lzw compression, and it seems to have a faster and more reliable scanning interface. it is more expensive ($200 more) than sharpeye, so if cost is a factor, this package might prove to be too expensive.i also looked at the omer add-on from harmony assistant. image 8: initial pass through omer the recognition engine was good, as can be seen in this illustration above; however, i did not like the editing controls. i found them to be clumsy and difficult to read to my rapidly aging eyes. the mini-icons are also very small and similar in design and layout, so navigating between the appropriate sub-menus was very confusing. of the three omr packages i looked at, this was by far the cheapest, and, since i highly recommend harmony assistant, this could well be a viable omr program once you get the hang of its navigation.i urge you to experiment with omr; all of these software packages have at least a demo version where you can experiment with bringing sheet music to life. the process, regardless of how you do it, is time-consuming, but i think well worth it. much of 19th and 20th century history is dramatically illustrated by sheet music, which transmitted music before the widespread adoption of radio. i believe that these powerful tools allow us to create a window to a lost world. notes [1] see music notation’s omp web page at http://www.music-notation.info/en/compmus/omr.html.[2] see donald byrd’s omr (optical music recognition) systems web page at http://mypage.iu.edu/%7edonbyrd/omrsystemstable.html. [3] see the sheet music in celebration of pullman web site at: http://www.pullman-museum.org/exhibits/music/. [4] mentions of this farce-comedy can be found in the new york times, news from the theaters, jan. 24, 1884, p. 3; new york times, theresa vaughan is dead, oct. 5, 1903, p. 7; new york times, theatrical notes, jan. 4, 1880, p. 7. more information can be found about the farce-comedy theatrical genre and tourists in a pullman car in specific in smith, cecil michener. musical comedy in america. new york: theatre arts books, 1950, p. 61. [5] sadly, the beginnings of legalized “separate but equal” segregation occurred because of the pivotal case of plessy v. ferguson in april, 1896, which was a dispute over whether homer plessy, 1/8 black, would be allowed to travel in a whites-only pullman car through the state of louisiana. for more information on pullman porters and the pullman company’s hiring practices of african-americans, see bates, beth tompkin. pullman porters and the rise of protest politics in black america, 1925-1945. raleigh: univ. of north carolina press, 2001 (especially chapter 1). a good prã©cis of pullman porters’ work experiences can be found at http://www.jmu.edu/writeon/documents/2006/mccray.pdf, mccray, kim. pullman porters: the best job in the community, the worst job on the train. there are many, many works on the pullman porters’ experiences, listings of which are beyond the scope of this article, but a fascinating glimpse into a vanished america. [6] lott, eric. love and theft: blackface minstrelsy and the american working class. new york: oxford university press, 1993. bates, demographic stratification described throughout chapter 10. [7] kaser, arthur leroy. kaser’s complete minstrel guide: a collection of minstrel material for every occasion. chicago: the dramatic pub. co., [1934]. inside the minstrel mask: readings in nineteenth-century blackface minstrelsy. edited by annemarie bean, james v. hatch, and brooks mcnamara. hanover, nh: wesleyan univ. press, 1996. and, of course, a good example of a dramatic use of the porter as a tragic buffoon can be found in the character of brutus jones in eugene o’neill’s the emperor jones. [8] according to the collectors’ book auction site (< a href=”http://www.collectorsbookmarket.com”>http://www.collectorsbookmarket.com/), foxing is defined as “the brown age spots thought to be caused by impurities in paper(e.g.: acid, exposure to humidity, etc.).” [9] the sharpeye omr engine is called liszt, and is thoroughly described at http://www.visiv.co.uk/tech-mro.htm. [10] photo from koopman, h.r. pullman: the city of brick. roseland (chicago), il: h.r. koopman (privately published), 1893. [11] indeed, this is the whole basis behind the field of music therapy. this has long been a recognized phenomena; see, for example, burton, robert. the anatomy of melancholy. london: 1621. (project gutenberg edition at http://www.gutenberg.org/etext/10800). [12] it is interesting to read the opening night review of the play in the new york times, “mr. blue beard” on broadway, jan. 22, 1903, p. 9. [13] there are many works and web sites on the tragedy. see, for example, brandt, nat. chicago death trap: the iroquois theatre fire of 1903. carbondale, il: southern illinois university press, 2003. [14] it is heart-rending to read the contemporary chicago tribune accounts of the tragedy. a newspaper column headlines from dec. 31, 1903, p. 4, says it all: “bodies removed by wagonloads. police vehicles and ambulances prove inadequate for the dead. patrolmen forced to fight for passage through frenzied throngs.” [15] the national fire prevention association has a chart of these tragedies at http://www.nfpa.org/itemdetail.asp?categoryid=954&itemid=23343&url=research%20&%20reports/fire%20statistics/deadliest/large-loss%20fires&cookie%5ftest=1. the most recent large fire occurred in rhode island, at the station, a nightclub, on feb. 20, 2003. the loss of life in this fire, 100, ranks it at 17th place. [16] brandt, chicago death trap, p. 85. ms. reed apparently made quite a living appearing in theatrical performances. i can find her first appearance in the new york times, 4 new plays at the theatre this week, nov. 3, 1901, p. 17. [17] the virtual singer, in my very humble opinion, is approaching the spooky side of technology. listen to how the virtual singer interprets la vie en rose: http://www.myriad-online.com/resources/demosvs/rose.mp3. again, just my taste, but i have never used this particular add-on feature. author information: andrew bullen is the information technology coordinator for the illinois state library, and a proud resident of the pullman community. he can be reached at abullen at ameritech dot net. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – collectionbuilder-contentdm: developing a static web ‘skin’ for contentdm-based digital collections mission editorial committee process and structure code4lib issue 49, 2020-08-10 collectionbuilder-contentdm: developing a static web ‘skin’ for contentdm-based digital collections unsatisfied with customization options for contentdm, librarians at university of idaho library have been using a modern static web approach to creating digital exhibit websites that sit in front of the digital repository. this “skin” is designed to provide users with new pathways to discover and explore collection content and context. this article describes the concepts behind the approach and how it has developed into an open source, data-driven tool called collectionbuilider-contentdm. the authors outline the design decisions and principles guiding the development of collectionbuilder, and detail how a version is used at the university of idaho library to collaboratively build digital collections and digital scholarship projects. by devin becker, evan williamson, and olivia wikle context unsatisfied with the limited options for customizing collections hosted in their digital asset management (dam) system, contentdm, librarians at the university of idaho developed collectionbuilder-contentdm, an open source approach that uses contentdm’s api to build customized, discovery-focused web pages and visualizations on top of a collection’s metadata. acting as skins, these overlaying web pages provide an alternative interface for users to explore collection content and context, without directly interacting with contentdm’s website.[1] the digital collection websites built with this tool are generated from csvs using a static web generator, and provide users with total access to and control over all code and data. this article briefly details the background of the “skin” approach, provides an in-depth look at the code making up the collectionbuilder-contentdm template, and lays out the workflows used at the university of idaho to build and maintain the university of idaho digital collections. why a skin? while contentdm facilitates certain features that are difficult to replicate via other tools or services—namely full-text and cross-collection searching, media object storage and retrieval, and established workflows for ingesting/exporting/exposing metadata and objects—we developed skins for our collections for three main reasons: we don’t believe the default contentdm interface is particularly user-friendly, and we don’t find the customization options effective or usable enough to accommodate the discovery-focused designs we aim for with our digital collections. we believe our collections (and archival collections in general) are truly ‘special’ and as such demand and reward customized treatment to relate their extraordinary nature to our users. items in our collections are not disembodied database objects, and we want to ensure context is communicated to users. the seo practices and machine-readable markup, or lack thereof, of contentdm do not promote the discovery of individual collection items via external search engines. the ‘skin’ approach allows us to utilize the search and storage features of contentdm that we find difficult to reproduce, while allowing us to curate a user experience for the collection that matches the quality and interest of the content we are charged with stewarding. historical development we began using this skinned approach with our digital collections in earnest around 2012. starting by hand-coding websites using basic design features, we quickly progressed to exporting metadata from contentdm as xml and using xslt worksheets to generate the overlaying html pages for each customized collection. some of these xml/xslt techniques were described in a 2014 code4lib article detailing the development of the latah county oral history collection.[2] template elements, such as headers, footers, and analytics, were added using php includes to simplify maintaining consistency. links to individual items pointed to the contentdm item page. this workflow was effective in many ways but required the maintenance of many different xslt files that were then run against metadata xml files. since each transformation had to be run individually, and the outputs had to be moved to a development server to test the code, viewing results from iterative development was slow. the system was also highly idiosyncratic, making it difficult to collaborate with other developers. in 2015, we started to experiment with employing modern static web generators to build our collections and found that this method of development resolved the issues of version control and collaboration that hindered our xml/xslt workflow, allowing us to consolidate our code and create collections that are at once more uniform in terms of layout and individualized in the way of content. the concepts behind our original skins were not abandoned: the visualizations, designs, and patterns of use have inspired and informed many of the components we have developed for collectionbuilder. we detail these components below, including brief discussions of how collectionbuilder’s current code has been influenced by past tools and methods as our new approach has matured into a project capable of efficiently generating our new collection skins. if you’d like to fully understand how the system works, and its various requirements, please peruse the documentation available on the collectionbuilder website. code collectionbuilder-contentdm uses the static web generator jekyll together with the templating language liquid to build the layouts and features of the generated websites. three main components are provided or edited by a user to build an individual digital collection website: a csv file containing the collection’s metadata; two main configuration files (_config.yml and _data/theme.yml); and a series of page-specific config files (in csv format) that determine what content appears on certain pages. during generation, jekyll exposes the metadata csv from the project’s “_data” directory, making it available for use in pre-built templates throughout the project to build out visualizations. despite much of the code being pre-built, each page can also be flexibly modified using the user-generated configuration files. this enables the tool to accommodate a wide variety of collection types and content without a user needing to edit the base code. it also allows us to develop the entirety of our digital collections site via one git repository with a branch for each collection. collection as data (and more data and more data …) collectionbuilder’s visualizations, features, and item pages demonstrate an expression of the “collections as data” ideal in that they are fundamentally metadataand data-driven. using liquid templates, the code for each visualization creates specialized derivatives of metadata that are consumed to generate the web page. this approach puts the focus on metadata as data, rather than loosely structured description, and rewards high quality metadata by creating intricate browsing pathways for users. jekyll and liquid are particularly suited to this sort of data transformation, able to generate new formats that can be reused in other applications. rather than hide this data, we made the decision to expose it, making a variety of publicly re-usable derivatives easily downloadable from the site. for example, below is a list of the outputs (with links) that are listed at the bottom of the home page for the archival idaho collection: metadata csv – all metadata fields in csv format, configurable by the “metadata-export-fields” variable in theme.yml metadata json – all metadata fields in javascript object notation (json) format subjects json – unique subject terms and their counts in json subjects csv – unique subject terms and their counts in csv geodata json – a geojson file with geographic coordinates and associated item metadata locations csv – unique location terms and their counts in csv locations json – unique location terms and their counts in json timeline json – a json file formatted for use with timelinejs facets json – unique terms and counts for all metadata fields listed in the “metadata-facets-fields” variable in theme.yml these metadata downloads and a table representation are further highlighted on the data page. the web table is powered by datatables with the js code optimized to handle thousands of items; it allows users to filter and export subsets of the metadata records. of course, all of these data formats are not necessary for every collection, so the tool will only build each data file if the related web page is also included in the navigation configuration file (_data/config-nav.csv), which controls which pages display as headers for the site. this means, for instance, that if one were to remove the map page from their navigation configuration file, the system would not expose a geojson file for the collection. in designing the tool, we took the “collections as data” mantra to heart. we hope this explicit acknowledgment of the data underlying the digital collection will encourage others to creatively reuse and rethink our repository. we also see this feature as an acknowledgment of the reality of migration: no platform is future proof–instead keeping the focus on investing in quality machineand human-readable data in a variety of formats from the start ensures the collections we steward are ready for migration, preservation, and reproducibility.[3] home page, a bento box of options first impressions, as we all know, are important. most digital collections, however, are introduced poorly or not at all. out-of-the-box contentdm collections, for instance, feature a “landing page” that provides a space for a general textual description of the collection, that with some work can be customized with a few visual features (e.g. 2016 example of our higgins collection). alternatively, many more complex systems simply drop a user into a catalog-like presentation of the collection, eschewing contextual information altogether. we believe our collections—and, for that matter, the collections held by libraries and other institutions like them across the us—deserve better introductions than they currently receive. the design of our various skins throughout the years reflect that belief, as does our current iteration. our early skins often included a feature such as a carousel[4], map[5], timeline, or interactive display[6] to draw users into the collection. collectionbuilder sites go a bit further, providing both an eye-catching featured image and contextual information about the collection. the tool uses a collection’s metadata to generate an overview of the collection’s era, location, subject matter, size, and content, and provides entry links from the overview information to encourage exploration. in some ways this is influenced by the popular concept of “dashboards,” but rather than forefronting administrative and use data, we are summarizing metadata to create a simple visualization of overall context for the collection. figure 1. the collectionbuilder-generated home page for our barnard-stockbridge photograph collection. the home page is customizable via the configuration of the theme file (_data/theme.yml). the theme allows a user to choose the featured image, determine the content of the carousel, and adjust the number and/or content of the featured subjects and locations. this allows for a great deal of customization and impact without modifying the base code. choosing a featured image, for instance, is an important act of curation for the collection, as derivatives are used to represent the collection in meta tag markup (which will be displayed on social media). the theme also allows one to adjust the size and placement of that image, which has a significant impact on the look and feel of the home page. below the featured image, the infographic layout is made up of a series of modular cards presented in a bento-box style populated by automatically generated content. for example, “carousel” provides a bootstrap carousel with a randomly selected group of images with direct links to the items. the ‘time span’ card calculates the date range of the collection and links to the timeline visualization page. the code in our “top subjects,” “top locations,” and “objects” cards calculates the unique values in specific metadata fields and provides links to the browse page which will sort to that grouping. these cards can be swapped out depending on which work best with the collection’s metadata by replacing jekyll _include commands — {% include index/[filename].html %} — on the home-infographic-layout.html file, which is contained in the _layouts folder. this enables the page to be quickly rearranged using liquid includes and bootstrap columns to provide variety. browse page as discovery engine early versions of the skin often featured a “browse all” page with all collection images represented on an (almost) infinitely scrolling page. for some collections, all items were further represented as cards with image and basic metadata displayed. these cards were created using xslt from the xml metadata, and arranged on the page using the jquery plugin isotope with masonry layout. isotope allowed for randomizing the items and filtering using a simple search box. these early pages allowed users to seamlessly browse through all items at a larger scale as an alternative to clicking through page after page of thumbnails in the contentdm interface “browse.” building on these earlier designs, we developed a browse page that provides cards for every item in the collection. the metadata featured on each card is fully configurable (via the config-browse.csv in the _data folder), and field information can appear as textual or be represented as a button that, when clicked, instantly filters the page (see figure 2 for an example of the result). this filtering feature is central to collectionbuilder sites, as links on almost every page lead back to filtered versions of the browse page. this allows users to quickly explore groupings and subjects that they discover while browsing items across the site. the page tracks filtering via url hashes, making any filter shareable by copying the link, and allowing other pages to link into the filtered view. figure 2. a portion of the browse page for the idaho cities & towns collection, filtered to those items located in sun valley. to make this visualization scaleable to thousands of objects[7], the creation of item cards and filtering is handled by javascript. following options set in the config-browse.csv file, liquid is used to create a subset of metadata in the browse-js include as a js variable that then drives the visualization. the config file also controls the metadata fields which are displayed on the item card, using liquid to manipulate the javascript function that creates each card. this function is kept fairly simplistic to enable easy modification of the card contents in cases where the automatic configuration is not flexible enough for the needs of the collection. the aim here, and throughout the codebase, is to keep the javascript simple and well commented enough that a typical digital librarian can figure it out, even if they are not expert developers. word cloud (still a really good visualization) word clouds are a common, yet useful way to quickly visualize word frequency in text. applied to a metadata field, they provide a simple representation of the unique facets of a collection, allowing users to get an overall sense of the content, while also surfacing the unexpected. some of our early skin sites featured word clouds of subject terms inspired by the tagcrowd.com implementation.[8] each subject term in those early word clouds linked to a contentdm search, thus sending users out of the skin to the database view. building on this concept, collectionbuilder contains a flexible cloud layout, designed as a template that can generate a word cloud from any relevant metadata field, such as subjects, locations, or creators. each unique term is rendered in the cloud as a button (highlighting its “clickable-ness”), with font size scaled to its relative frequency, and given a hyperlink to the browse page which will display the related group of objects. despite the final result being a fairly simple visualization, this is one of the more complex pages to build, given the need to calculate unique terms and counts from the metadata fields which may contain thousands of individual values. our first implementation was done entirely in liquid, requiring several “for” loops within loops and a hacky data structure (as liquid only supports a basic form of an array). a version of this liquid routine is used in the lightweight version of collectionbuilder, collectionbuilder-gh, which is designed to run on github pages where jekyll plugins are not allowed. we include it here, as it demonstrates a complex use of liquid: --# find and count unique subjects used in the metadata --{%if site.data.theme.subjects-page == true -%} {%assign cloud-fields = site.data.theme.subjects-fields | split: ";" -%} {% comment %} capture all cloud terms {% endcomment %} {%assign raw-terms = "" -%} {%for c in cloud-fields -%} {% assign new = site.data[site.metadata] | map: c | compact | join: ";" %} {% assign raw-terms = raw-terms | append: ";" | append: new %} {%endfor -%} {%assign raw-terms = raw-terms | downcase | split: ";" -%} {% comment %} clean up raw terms {% endcomment %} {%capture terms -%}{%for t in raw-terms -%}{%if t != "" and t != " " -%}{{ t | strip }};{%endif -%}{%endfor -%}{%endcapture -%} {%assign terms = terms | split: ";" | sort -%} {%assign uniqueterms = terms | uniq | sort -%} { "subjects": [ {% for u in uniqueterms %}{% assign count = terms | where_exp: 'item', 'item == u' | size %}{ "subject": {{ u | jsonify }}, "count": {{ count }}, "link": {{ u | url_param_escape | prepend: '/browse/#' | absolute_url | jsonify }} }{% unless forloop.last %}, {% endunless %}{% endfor %} ] }{%endif -%} liquid iteration tags like those used above, however, are fairly slow, which can lead to unreasonably long build times as the size of projects or data increase. in our initial iterations this slowness was compounded, as code throughout the repository calculated unique terms for several data outputs in addition to the cloud page. as we began redesigning larger digital collections, the speed of liquid became a limitation, making iterative development cumbersome as build times soared. at first, we would use smaller subsets of the metadata and turn off calculating clouds during development so that rapid iteration was still possible. one of the advantages of working with collectionbuilder, however, is the way it exposes issues (and interest) in the metadata, so working with subsets is a suboptimal solution. this led us to explore methods to optimize the liquid code, such as replacing “for” and “if” statements with equivalents using “where” or “where_exp,” which are significantly faster and minimize iterations. even with optimization, using liquid for calculations within jekyll slows build time significantly—a much more efficient solution is to use a jekyll plugin. jekyll’s ruby-based plugin system provides a means to add custom functionality injected into the generator engine at build time. a large ecosystem of formally packaged plugins exist, or any new plugin can be written in ruby and added to the project “_plugins” directory. any calculations completed directly in ruby will be exponentially faster than complicated liquid routines. after much experimentation, we developed a plugin “array_count_uniq” that adds a new liquid filter to the project environment. to use it, we first gather the desired data as a liquid array, then apply the new filter: {{ myarray | array_count_uniq }} the filter will return a hash of the unique values and their frequency counts which can then be iterated over like any other array to use the calculated values in the page template. following contentdm conventions, we use semicolons to denote multivalued fields. for example, an average value in the subject column might look like “idaho; potatoes; mountains,” which is obviously three separate subject terms, not a single value. thus, to prepare the array, the cloud layout uses “map” to extract values from the desired field(s), joins them with a semicolon, then splits on semicolon to create a unified array of all terms which can then be passed to the “array_count_uniq” filter. using the filter on a single metadata field (“example_field”) would look like: {% assign uniquehash = site.data.metadata | map: "example_field" | join: ";" | downcase | split: ";" | array_count_uniq %} the implementation in the cloud-js include is more complicated because we support combining multiple metadata fields into a single visualization. this plugin filter reduces build times exponentially, completely removing the old bottleneck of calculating unique counts. since the plugin is fairly simple and idiosyncratic to the needs of the collectionbuilder project, we haven’t independently packaged it—we consider it part of the template and it can be found in the _plugins folder. we see the creation of additional plugins as a way forward for making collectionbuilder more efficient, but hope to balance it against our aim to keep the overall project complexity low. seeing space: from fusion tables to leaflet when our early php-based skins were developed, creating your own interactive map layer was a very challenging undertaking. so when google fusion tables launched in 2009,[9] the web service was embraced by librarians looking to build map visualizations using only a spreadsheet and bit of configuration on the platform. we implemented fusion tables extensively for our digital collections, customizing google api javascript examples to load fusion tables-based maps on our collection pages. like so many other google products, however, fusion tables was terminated in december of 2019, and all our old map features are now obsolete. luckily, already growing cautious of 3rd party services, we had begun moving our map applications to self-contained javascript using the open source library leaflet.js in 2017. leaflet is efficient, well documented, and has a robust plugin ecosystem, making it relatively easy to implement custom map features on static pages using openly available tile layers. to provide data to the map, we use a liquid template to generate geojson features for the items that contain latitude and longitude metadata values. other descriptive metadata configured in “config-map.csv” is included on popups from the map pins or used in searching. a geojson file with more complete metadata is also generated in the data exports, which can be easily used in other projects and analysis. using liquid templates with the metadata is a practical way to carry out these types of data transformations, as can be seen in the code to generate geojson: --# generate geojson data for collection items with lat-longs --{%assign items = site.data[site.metadata] | where_exp: 'item','item.latitude != nil and item.longitude != nil' -%} {%assign fields = site.data.theme.metadata-export-fields | split: "," -%} { "type": "featurecollection", "features": [ {% for item in items %} { "type":"feature", "geometry":{ "type":"point", "coordinates":[{{ item.longitude }},{{ item.latitude }}] }, "properties":{ {% for f in fields %}{% if item[f] %}{{ f | jsonify }}: {{ item[f] | jsonify }},{% endif %} {% endfor %} "reference_url": {{ '/items/' | absolute_url | append: item.objectid | append: '.html' | jsonify }} } }{% unless forloop.last %}, {% endunless %}{% endfor %} ] } three leaflet plugins add further functionality to the map visualization: 1) leaflet-fusesearch which provides basic search functionality for the items displayed on the map; 2) leaflet.markercluster, which clusters individual items on the map, cleaning up the display and significantly improving performance and useability for large collections; 3) leaflet.markercluster.freezable, which allows clustering to work alongside search by temporarily unclustering to display query results. to make working with leaflet easier, we expose the basic map configurations and plugin options as values in the theme.yml file, allowing you to quickly test different settings for each collection. to ensure missing configuration options won’t break the visualization, we set sane defaults using liquid’s “default” filter. figure 3. an example collectionbuilder map page from our archival idaho photograph collection. leaflet plugins allow clustered map items to be searched. traveling time: from simile timeline to simple table (+ timelinejs) it seems dated now, but simile’s javascript timeline visualization was all the rage about 10 years ago. we still believe it’s a fairly good visualization, even if it isn’t that mobile friendly or up to today’s design standards. we used to implement simile timelines for all our collections, generating the required xml file from the collection’s metadata and customizing the timeline code to work with our collection’s time spans. we still proudly use one on our main digital collection website to this day. however, with collectionbuilder, we realized it was probably time to say goodbye to simile timeline. at first, we thought we might need a sophisticated tool that allowed for unique viewing methods such as horizontal scroll. yet as we examined other visualizations, we realized a simple table with the capacity to jump to certain years provided the best use case for our collections. figure 4. an example collectionbuilder timeline page from our psychiana digital collection. a simple table format has proven to be an effective way to organize items by year. the liquid code that generates the timeline operates by mapping all the dates in a collection and finding unique years. it then builds a row for each year that drops a thumbnail or text-based card for each item that has a date field containing that year. we have found, somewhat to our surprise, that this is often the best visualization for us to get a good overview of our collection, and we believe it to be a very effective timeline visualization, possibly even better than simile’s. we also recognize that there is often a need for curated timelines that look a bit fancier and are more enticing to the user. timelinejs is a well-known visualization built and maintained by the knightlab. we had previous experience building timelinejs visualizations by connecting published google sheets with metadata, but with collectionbuilder we decided to generate a json file that could be consumed by timelinejs to build a timeline. this is currently a feature that can be included on a collection’s index in the home-infographic layout or separately as its own page. we use this timeline, for example, on the front page of our dworshak dam collection to provide a historical overview of the politics and media surrounding the debate around building a northern idaho dam in the 1960s. the include to add the timelinejs code to any page looks like this: {% include index/timelinejs.html %}. the feature is driven by the timelinejs.json file, which is generated automatically using liquid.[10] figure 5. an example of the timelinejs include on the dworshak dam collection’s index page. item pages – built to be found our earliest skins provided the browsing features described above, but when it came to displaying individual object pages, links would lead directly to the contentdm item page. this was unsatisfying in that it kicked users out of the skin website and into a separate ecosystem without meaningful links back into the discovery features on the collection’s skin. this also meant that the representation of the object on contentdm must be the definitive one, featuring the most up-to-date and complete metadata. given that individual contentdm metadata is notoriously difficult to update in bulk, this can lead to a large amount of work or technical debt when updating aging collections. the earliest proto-collectionbuilder site was a project to redesign the idaho waters digital library. a small grant provided resources to employ a graduate student with disciplinary expertise to enhance existing metadata, as well as add new documents to the collection. this overhaul of the metadata was too difficult to bulk re-ingest into contentdm, so we decided to provide static item pages featuring the enhanced metadata, rather than forcing it back into contentdm. as we further explored this approach, we realized that because static item pages provide the opportunity to: 1) embed rich markup driven directly by a collection’s metadata csv, 2) include links back into the skin visualizations, and 3) customize the presentation of objects, they were a better option for all of our collections. to generate individual html pages directly from the metadata csv, we turned to the “jekyll data pages generator” plugin by adolfo villafiorita, which was slightly modified for the needs of our project. configured in _config.yml, this plugin injects a page object for each line of a data file into the jekyll engine on build.[11] for collectionbuilder, this means that each object in the metadata will generate an html page and all field values will be available to call into the template using liquid, allowing the metadata to drive the creation of detailed, machine-readable markup and full item representations. creating the object pages is handled by the “item” layout. the look and feel of these pages originally mirrored the contentdm item pages, while cleaning up and simplifying the interface with a two column approach (collapsing to one column on mobile). the left side features an image representation and object download buttons that are logically selected based on the format field in the metadata. these use the contentdm apis to load images and download objects. the right column features the metadata displayed in an easily readable format that can also be copied by the user. figure 6. an example contentdm item page from our barnard stockbridge collection (top) and the same item’s page in our collectionbuilder site (bottom). the metadata fields displayed and meta markup on the item page is configured by the config-metadata.csv, allowing for easy customization of what information is exposed on the page. this config file allows individual fields to be designated as browse links, which transforms values into hyperlinks that lead back to the browse page, which is filtered on the term. items with latitude and longitude will feature the button “view on map,” linking to the item’s location on the map page, and likewise those with a date have a “view on timeline” button, linking to the item’s location on the timeline page. these linkages drive the interactive pathways for the user to explore each collection through visualizations. although the item pages may look similar to contentdm’s object display, underneath things are quite a bit different. contentdm pages are generated using javascript, which loads data from the database, but fails to add any semantic markup standards to the base or rendered page. figure 7. an example of contentdm item page markup. in contrast, collectionbuilder exposes rich, machine-readable markup driven by the full collection metadata. the item layout calls a special meta markup in the head element (_includes/head/item-meta.html). this adds three semantic markup standards in addition to standard html meta tags. figure 8. collectionbuilder item page, rich meta markup in a variety of standards. first, this item-meta code adds meta tags containing dublin core terms schema based on mappings set up in the config-metadata file. this implementation is based on the output of dspace repositories (e.g. view the source of this wsu research exchange item page). this is not a commonly used form of markup, but since it is used by other repository systems, there may be harvesters designed to crawl it. second, it adds open graph meta tags. open graph protocol was originally designed by facebook to create a standard for assigning metadata to represent webpages on the social media site. other platforms, such as twitter, maintain separate markup standards, but can also read open graph as a default, making it a good generic choice to provide this functionality. the open graph prefix is declared on the head element: <head prefix="og: http://ogp.me/ns#"> the schema can then be used in meta tags following the standard, with values added from the item metadata using liquid, e.g. <meta property="og:title" content="{{ page.title | escape }}" /> this structured data will be used by social media platforms to create the familiar card representations when links are entered into posts, providing an official preview of the content. thus, if you tweet out a collectionbuilder item page, the object image or thumb will appear along with the actual item title and description. third, the item-meta include creates schema.org markup in json-ld format. the data is contained in a script element of type “application/ld+json.” a liquid template adds appropriate metadata fields, image links, and establishes an “ispartof” relationship to the full digital collection. additionally, breadcrumbs are provided in the schema standard, further reinforcing the contextual relationship of the object. this structured data is used by search engine crawlers to learn more about the page and its relationships, semantic markup that can help with seo and with the representation displayed to users in search results. how we use collectionbuilder we started using collectionbuilder in earnest to develop our digital collections at the end of 2018, using an agile ‘sprint’ in the data and digital services department at the university of idaho library to jump start the creation of the necessary data and design files. although this early version of collectionbuilder was not fully developed, our team was able to prepare the metadata, configuration files, and textual content. since this data (literally the collection as data) is independent of the collectionbuilder template, the implementation of visualizations and infrastructure could continue to develop and evolve separately, much like a wordpress theme. due to staffing issues, we did not return to applying collectionbuilder to our own collections in earnest again until the summer of 2019, during which we solidified our workflow for building and updating all of our digital collections. currently, we use collectionbuilder via a git/github focused workflow to develop our digital collections in an open, collaborative way with librarians and staff at the university of idaho library. the code for all collections is contained in a single github repository with the master branch representing the generic template. each individual collection is developed in a new branch within the repository, allowing us to create unique sites while maintaining a central codebase. this approach helps us to keep the underlying code up-to-date with improvements we often make while retaining each collection’s individual customizations and commit history that details the changes that were made. the basic workflow moves through seven areas of activity: repository branch creation (github) in our collectionbuilder github repository, we create a branch for the collection, using a branch name that corresponds to that collection’s short url name. for example, the collection https://www.lib.uidaho.edu/digital/barstock/ is developed on a branch named ‘barstock.’ metadata extraction/preparation/revision we extract a tsv file of the collection’s metadata from contentdm using the export feature on contentdm’s project manager tool. we upload this data into google sheets, where we check the metadata for errors, correct any field names that need to change (subjects becomes subject, etc.), and add a unique objectid field. finally, we download the csv of the document and add it to the _data folder of our branch. config file editing we configure the various files that drive the generation of the site. these include: /_config.yml, where we record which collection and url we’re building; /_data/theme.yml, which controls the basic configurations for all of our visualizations (map, timeline, word clouds, etc.), data files, and our front page; and a series of “config-(…).csv” files (config-map.csv, config-browse.csv, config-metadata.csv, etc.) that further configure what information shows up on which pages. development server review we serve up the newly configured site on our computer with the jekyll serve command (‘jekyll s’). we then examine the website generated for obvious errors and additional visualization or browsing feature possibilities. we often look at the facets.json data file generated by the system (and configured by the theme.yml file) to see if there might be other fields to visualize in a word cloud format, or even a modified timeline. build static web files we’ve added a rake command to build our website for production. the command ‘rake deploy’    builds the site for production by running the jekyll command `jekyll_env=production jekyll build.` the production environment triggers the addition of features left out during development, such as full meta markup and analytics snippets. this keeps our analytics account clean from server hits, while allowing us to have the code pre-built with an analytics variable in our _config.yml file. deploy web files to production server we then copy the files and folders contained in the _site folder and add those files to the correct directory on our production web server. push changes to git branch of the repository if more complicated revisions or additions are being made, we try to push changes several times as we revise the site running on the development server. if the changes are small, however, we just push these changes at the end of the process. when substantial development changes are made to the master branch code, these are pulled into all of the collection branches. these collections are then rebuilt, deployed, and their updated code is pushed to the repository (steps 5 through 7). this framework facilitates the rapid development of custom visualizations and features, enabling us to generate unique interfaces for more collections. low priority collections use the standard template, reaping the benefits of the skin interface without a large investment of time. meanwhile, higher priority collections can be more quickly developed into unique sites, while still maintaining elements of a common theme and branding. for collections requiring extensive customization, the templates act as recipes that can be quickly adapted to new purposes, without getting too far from the original code base, making maintenance easier. figure 9. detail of the branches for our digital collections, as seen listed in a github desktop platform. besides being an excellent way to keep our collections’ code up to date, creating and managing our collections on github has also expanded collaborative opportunities, allowing more staff to participate in creating collections and giving them an opportunity to learn from the updates and alterations their colleagues make. more people working on collections makes it easier for us to keep our collections up to date, and any mistakes that might be made by those relatively new to working with our digital collections can be easily reverted by collectionbuilder’s developers using git. conclusion: next steps throughout our development of collectionbuilder-contentdm, we have been encouraged by the value that this tool can bring to other contexts, especially situations involving teaching digital collections and operating without a dam system altogether. this has led us to develop other versions of the tool which function independently of the apis our skin version relies on. two versions, collectionbuilder-gh (github pages) and collectionbuilder-sa (stand alone), utilize lunr.js to power searching within collections as fully static sites. gh is designed to be set up entirely using the github web interface and hosted on github pages. this “lite” version is optimal for pedagogical environments because it does not require users to generate derivatives of their digital objects or install any software, which frees up more time for teaching metadata and web skills. sa combines the optimized code base of collectionbuilder-contentdm with the independence of gh, allowing users to create a more robust collection not tethered to a dam system or apis. this version requires that the user generate thumbnails and small image representations of their full size objects, optimizing the performance and accessibility of the site. once ready to deploy, these digital objects are hosted alongside the site’s static pages on a basic web server of the user’s choice. collectionbuilder-es (elastic) is currently the least developed, but combines the code of the sa version with powerful new search functions that will make large-scale use of collectionbuilder a possibility. by connecting individual digital collections in a modular fashion, our elastic version of collectionbuilder will enable cross-collection searching, and allow for pulling together curated exhibits of items from a variety of collections across different institutions. this could provide a viable alternative to traditional database-driven dams systems for individual libraries and organizations, and open unique possibilities to cross institutional boundaries. to accomplish this goal we are working with a developer to integrate elasticsearch functionality and investigate object storage solutions as well as a variety of deployment options that would enable the tool to follow the jamstack model. ultimately, even as these versions become more mature they will retain their focus on leveraging the collection as data in order to present the collection as a collection, i.e. a grouping that comes to mean more than the sum of its items. through our continued investment in the metadata driven visualizations that inspired our initial skin version and new features such as our recent efforts to help users more easily compose engaging “about” pages, we endeavor to move beyond the catalog-based view offered by most dams platforms. to this end, collectionbuilder opens up possibilities for the librarians, library staff, archivists, and other glam professionals that use it to provide contextually based and data-driven features for the collections they steward. this in turn allows our users to discover and engage with digital collections as collections rather than a series of disembodied items stored in systems we did not design, do not control, and which do not sufficiently communicate the value of the collection as such. for additional information, please see the collectionbuilder website, as it is regularly updated with latest features, a variety of documentation, and examples. about the authors: devin becker is the director of the center for digital inquiry and learning (cdil) and the head of data & digital services at the university of idaho library. becker is also a writer. his most recent (static!) web project, ctrl+shift, provides visualizations and analyses of interviews he conducted with prominent poets across the country. olivia wikle is the digital initiatives librarian at the university of idaho, where she coordinates the digitization of the university’s archival material and builds digital collections that disseminate historical resources. she also works closely with humanities faculty to create digital scholarship projects and teach digital literacy skills to students. evan peter williamson is the digital infrastructure librarian at the university of idaho library, working with data & digital services to bring cool projects, enlightening workshops, and innovative services to life. despite a background in art history, classical studies, and archives, his recent focus has been on data-driven, minimal infrastructure web development, currently embodied in the collectionbuilder project. endnotes [1] this setup could be more colloquially described as a “party in the front, contentdm in the back” approach to development. [2] devin becker and erin passehl-stoddart, “connecting historical and digital frontiers: enhancing access to the latah county oral history collection utilizing ohms (oral history metadata synchronizer) and isotope”, code4lib 29, 2015-07-15, https://journal.code4lib.org/articles/10643 [3] the human-readable aspect of this has come as a revelation, as the facets.json file has now become the first thing we look at when designing a new collection. we use it to quickly evaluate which fields have data within them that would reward a word cloud (or visually faceted) expression, as well as to familiarize ourselves with a quick overview of the collection. [4] e.g. 2016 capture of the stonebraker collection [5] e.g. 2019 capture of campus photograph collection [6] e.g. argonaut newspaper headlines [7] having thousands of images on the page obviously also requires “lazy loading” to avoid page load issues. to intelligently defer image load, we use lazysizes, a simple to use, up-to-date lazyload library that requires no initialization, and will simply load all images if browser support is missing. images are given the class=”lazyload”, and “src” is replaced by “data-src”. [8] e.g. 2016 stonebraker subject cloud [9] https://en.wikipedia.org/wiki/google_fusion_tables [10] we’ve found that it’s best to edit this file down significantly for the best performance, which we do either manually or by creating a truncated csv of the collection’s full metadata csv file. we then edit the code generating the json to refer to that data file, adjusting the “{%assign items = site.data[site.metadata] -%}” line at the top to reference the truncated csv, i.e. {%assign items = site.data.psychiana-select -%}. [11] building item pages using our modified version of the data page generator plugin is efficient, since the iteration through the metadata is done in ruby. however, the sheer number of items becomes a major factor in the build time for the site. since the jekyll build process involves writing out so many files on disk, the speed is not limited by your computer’s cpu, but by the write speed of your hard drive. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – refactoring alma: simplifying circulation settings in the alma integrated library system (ils) mission editorial committee process and structure code4lib issue 60, 2025-04-14 refactoring alma: simplifying circulation settings in the alma integrated library system (ils) refactoring is the process of restructuring existing code, in order to make the code easier to maintain, without changing the behavior of the software. georgia southern university is the product of a consolidation of two separate universities in 2017. before consolidation, each predecessor university had its own cataloging practices and software settings in the integrated library system (ils) / library services platform (lsp). while the machine-readable cataloging (marc) standard focuses on discovery, and descriptive search blended well to support discovery, settings related to circulation were in discord following the merger. three busy checkout desks each had different localized behaviors and requested additional behaviors to be built out without centrally standardizing. complexity stemming from non-unified metadata and settings plus customizations implemented over time for multiple checkout desks had ballooned to make for circulation settings which were overly baroque, difficult to meaningfully edit when a change to circulation practices was needed, and which were layered and complex to such a degree that local standards could not be explained to employees creating and editing library metadata. this resulted in frequent frustration with how circulation worked, difficulty knowing what was or wasn’t a software bug, and inability to quickly fix problems once problems were identified or to make requested changes. during 2024, the georgia southern university libraries (university libraries) undertook a comprehensive settings clean up in alma centered around software settings related to circulation. this article describes step-by-step how the university libraries streamlined and simplified software settings in the alma ils, in order to make the software explainable and easier to manage, and all without impacting three busy checkout desks during the change process. through refactoring, the university libraries achieved more easily maintainable and explainable software settings, with minimal disruption to day-to-day operations along the way. by wilhelmina randtke history of georgia and georgia southern university libraries cataloging, circulation, and the ils/lsp 2017 merger of two universities, and migration from voyager to alma in 2017, two separate events, each requiring a “big lift” occurred simultaneously. (1) in late may 2017, the georgia library learning online (galileo) interconnected libraries (gil) consortium, of which the university libraries is a member, went live fully on alma following an approximately two year statewide migration from voyager to alma(gil 2017[1]; gil 2015[2]). (2) during 2017 the current georgia southern university and the university libraries were formed by a merger of two predecessor institutions: armstrong state university, located in savannah and hinesville, and the former georgia southern university, located in statesboro. a january 2017 decision by the georgia legislature merged georgia southern university and armstrong state university, with merger occurring on short notice and completing fully in about 18 months (i.e. merger to fully complete in summer 2018) (coleman 2017[3]). ils/lsp migration is a resource intensive effort. the voyager to alma migration was a two year effort (lee and frost 2017[4]). in january 2017, when the legislature required the merger of the two predecessor institutions, both predecessors were in the final stretch of migrating records and local workflows into alma. planning new unified workflows was not possible before alma go-live. ils/lsp migration tends to require additional intense configuration time, data cleaning, and communication during the six months to a year after go-live, as nuances arise in the real world (ganz and slavin 2024[5]; nicholson and tokoro 2021[6]; rinna and swierenga 2020[7]). organizational mergers tend to be emotional and demoralizing following the merger for employees involved and productivity tends to drop in part because employees are distracted by the merger (heidari-robinson and heywood 2016[8]). merger within an ils/lsp involves integrating workflows, shared interface design decisions for the end user discovery interface, and metadata integration (flvc 2014[9]). the ils/lsp merger was not fully planned and implemented during 2017 nor 2018. all records from predecessor institutions were moved into the same instance of alma, but cohesive workflows and metadata standards were not set up. essentially each effort – migration and merger – competed for resources and employee bandwidth. additionally, unifying settings, for example, doing things like standardizing checkout times across campuses, was untenable mid-semester on short notice due to the necessity of communicating changes in checkout times to patrons ahead of time. in january 2017, when the merger was announced, each library had different checkout times for various kinds of materials. at the time of the may 2017 alma go live, it was practically a necessity to maintain separate metadata practices and separate workflows within the same ils/lsp in order to provide continuity and a good experience to patrons. during the merger period for the campuses, the organizational structure of the university libraries also merged. the university libraries is one organization, reporting to one director. while some library systems might have independent libraries using the same alma instance, for example, a law library and a main campus library where the accreditation standards require multiple libraries on campus to operate with administrative separation, no such arrangement exists at the university libraries. this is important, because refactoring at the university libraries involved eliminating settings within alma which had been built out at the library level and moving those settings to the institutional level. for an alma instance serving libraries with a different organizational structure, keeping settings at the library level within the ils/lsp might make sense. the near-universal marc standard for discovery oriented metadata aids cohesive interface design, but does not support cohesive workflows nor standardized item level metadata the marc standard is well defined and detailed with regard to creating records at the bibliographic level (library of congress 2024[10]). additionally, copy cataloging of bibliographic records ensures consistent quality and very similar or even identical bibliographic records will exist at different institutions (chan and salaba 2023 p. 41[11]). in the galileo consortium, during the 2015 to 2017 voyager to alma migration, the consortium had engaged in a process to attach items at all 28 member institutions of gil to the same bibliographic record when copies of that item were held at multiple member institutions (lee and frost 2017[12]). that process meant that the vast majority of bibliographic level records at the two predecessor institutions had already been standardized through state-level activity before the legislature decided on merger. at the item level, the marc standard has far less detail. the marc standard has extensive specifications (specs) for the bibliographic, authority, and holdings level records, but has no corresponding spec for item level records (library of congress 2023[13]). only a few fields of the marc standards, standards 876-878 describing item information stored in holdings records, describe item level metadata (library of congress 2008[14]). marc also was designed with discovery in mind. at the time when the marc standard was developed, marc was intended to allow someone to remotely search collections at other libraries, in order to plan a visit or to efficiently place an interlibrary loan request. it was not designed with inventory or workflow as the primary focus, but rather designed with end user discovery as the primary focus (mccallum 2002[15]). interface design decisions for discovery were fundamentally supported by the marc standard, and by statewide consortial activity, even before the merger. the vast majority of academic libraries use the marc standard (moehrle 2012[16]) and employees working in cataloging share a descriptive metadata standard for bibliographic and holdings records. in contrast, cohesive item level metadata is not defined by any near universal standard nor training, and item level metadata would tend to vary significantly from library-to-library. the georgia consortial environment includes specialized features focused on discovery interface, but does not include centralized shared workflows (although other consortia do use centralized shared workflows) georgia southern university is part of the university system of georgia (usg). in georgia, all institutions of public higher education use the same ils/lsp software which is procured statewide through the gil consortium. some configuration decisions are made at the statewide level through a gil committee structure. specialized features in a consortial ils/lsp might support unmediated borrowing services between members and discovery and delivery of shared electronic resource purchases (falsc 2018 p. 17[17]), which relate directly to end user discovery. leading up to and after alma go-live, the gil opac/discovery committee had made collective decisions regarding many of the facets and behavior in the discovery layer. primo at time of go-live, then later primo ve, was the discovery tool bundled with alma in the statewide contract for ils/lsp purchase, and primo / primo ve settings were determined by the statewide committee. all discovery tool interfaces for all members had to be compatible in that all member discovery interfaces included an option to both search just that institution or all member institutions at once (see figure 1). figure 1. drop down menu on the university libraries’ discovery tool interface and public search. a user to can search the holdings of the entire usg. for the past 10 years, the gil opac/discovery committee has consistently made statewide configuration decisions regarding interface design for end users. additional specialized features in a consortial ils/lsp which relate to shared workflows might include identification of the last copy of a print book, and centralized coordination of library employee user account permissions (falsc 2018[18]). in particular, a consortium might have a shared manual for consortially acquired software and might have preconfigured user roles in the software to allow employees to follow the shared workflows. the gil consortium did not operate in this way during the 2015-2017 period and historically has not operated this way, and so circulation workflows and local metadata practices were highly variable from member library to member library when predecessor institutions merged to form the university libraries. how circulation behavior works in the alma ils physical collections vs electronic acquisitions this article covers refactoring alma settings in the context of circulation logic rules for physical items. while digital materials have become an increasingly important part of library resources relative to print in recent years (goek 2024[19]), physical items are key to the university libraries operations and mission. physical items include special collections materials such as personal archives, letters, local history items, and unique items. the university libraries may be the only place someone can access a special collections item at all, and those materials tend to support research into institutional and regional history. equipment such as graphing calculators and laptops experience extremely high usage, with laptops continuously checked in and out throughout the year. additionally, laptop and equipment checkout supports student success in academics by bridging gaps in technology access for students waiting to finance a replacement for a broken personal device, or who may have left a personal device at home on a mostly commuter campus (gonzales et al. 2020[20]; wilmoth 2015[21]; bell 2022[22]). while acquisitions and collections as a whole are moving to digital with a reduced percentage of collections as physical items, the remaining physical items are of vital importance. additionally, while electronic resource packages may come with records to be batch loaded (i.e. somewhat standardized), item level records for physical items are likely to be created and managed fully in-house. with metadata at the item level created and managed fully in-house, workflows for physical items are not streamlined by the wider library ecosystem. because of this, this article focuses on refactoring settings for physical items and circulation behavior. physical item circulation in alma in alma, each physical item has a location. locations are grouped into fulfillment units (fus), and a location can be part of one and only one fu. circulation logic rules are built out in the software for each fu, and circulation logic rules within one fu do not impact circulation logic rules within any other fu. each circulation logic rule points at a terms of use, and a terms of use sets the detailed circulation behavior for that loan. the circulation logic rules can look at the user group of the patron seeking to checkout the item plus any of the following fields on an item: location, material type[footnote 1], or item policy. fu, material type, item policy, and terms of use are settings defined purely within the alma software and not part of the marc standard. alma includes a predefined list of material types and is not fully customizable. for item policy and terms of use, alma is fully customizable. a library using alma can edit a table in the software and create any number of item policies or terms of use. essentially, alma determines circulation behavior based on logic rules which can attach to: patron’s user group. item’s location. item’s material type. item’s item policy. reducing complexity in the circulation rules can be done in three ways: (1) by configuring logic rules to not consider certain fields, for example, to not consider material type, (2) by reducing the number of values in use for any of the fields, for example, by reducing the number of user groups patrons might be assigned to, and (3) by reducing the number of terms of use referenced by circulation logic rules. in contrast to alma, the predecessor system in use in gil and at the university libraries, voyager, had based circulation on item type, a voyager field which was split to two fields in alma – material type and item policy (exlibris 2024[23]). alma allows more complexity, regarding metadata fields which can be made to impact circulation behavior, than did the predecessor ils/lsp in use in georgia. the need for refactoring: unexplainable software settings in 2017, the year of both the voyager to alma migration and the merger of the two predecessor institutions, essentially each effort – merger and migration – distracted from the other, and as a result merger within the ils/lsp was not accomplished regarding item metadata and workflows. in 2017, mismatched metadata and circulation rules were put into place within a shared ils/lsp, with former georgia southern university locations having a different set of locations, material types, and item policies and different circulation logic rules within alma compared to those in use at former armstrong state university locations. in general, within the ils/lsp, settings in use exclusively at former armstrong state university locations had names with “_ll” appended to the end of the setting name. and, in general, a setting without “_ll” appended to the end of the setting name, would work only at former pre-merger georgia southern university locations. (lane library with initials “ll” is the former armstrong state university main library, located in savannah.) the locations belonging to predecessor institutions were fully separated into different fus. alma allows settings to be built out at a library-only level. the university libraries sometimes built settings out at the institutional level and sometimes at the library level. balance between and workflows for coordinating the institutional settings and library settings were encountered late in the migration and close to go-live on the new ils/lsp, with such a short notice for merger. merger, like migration, is a multiyear and stressful process in terms of truly merging to a new organization beyond the org chart (heidari-robinson and heywood 2016[24]). within the university libraries, cataloging was not merged until 2020, ils/lsp systems administration was not brought primarily into one department within the university libraries until 2021, and circulation desks are still separate departments within the university libraries. this meant that as employees came and went, a new employee had no comprehensive explanation of software settings. problems with numerous undocumented settings grew over time, as new employees, particularly in reserves, came on board, did not have comprehensive documentation, and requested new behaviors be built out within alma which duplicated existing but undocumented behavior. meanwhile, making changes to circulation behavior, for example, changing the checkout times or requestability of a specific kind of item or specific collection, required either analyzing increasingly baroque metadata and logic rules settings, or building out a new setting. in general, complexity of logic rules had ballooned such that building out a new and very specific logic rule was the only practical way to accomplish a change in a reasonable amount of time, which meant that even small changes to workflows were likely to cause additional bloat to settings. between the 2017 go-live on alma and 2024, circulation behavior for physical items had gradually been brought into alignment. this process generally had been accomplished by layering logic rules for narrow carve outs. it was common for circulation logic rules to reference very specific combinations of location and material type. this tended to cause problems for items assigned less-used material types, which had not yet been accounted for. when a problem was identified, it might be addressed using yet another circulation logic rule with a narrow carve out, because comprehensive behavior was too baroque to address without extensive time analyzing existing circulation logic rules. going into 2024 there were around 17 fus in use for active library collections, each with dramatically different circulation logic rules, such that comprehensively implementing a change to circulation behavior required analyzing circulation logic rules for all 17 fus. as a result, changes to checkout behavior could not be implemented in a reasonable timeframe. time which should have been spent on the “handshake” of communication between libraries systems and technologies (systems), cataloging, and checkout desks was instead spent on systems and cataloging looking at overly complicated settings and on employees at checkout desks reporting problems. at the start of 2024, settings across the 17 active fus in the university libraries’ alma instance had been built out to include: 33 user groups x 135 locations x 75 material types x 121 item policies = 40,429,125 possible permutations of metadata settings which could impact circulation behavior. meanwhile, the university libraries had a little over 1,000,000 physical items including both traditional library media like books and microform, as well as reserves and equipment! with 135 locations x 75 material types x 121 item policies = 1,225,125 permutations of metadata settings on an item, there were more possible combinations of settings than there were items, and it would have been faster to assess and assign desired behavior one-item-at-a-time than one-permutation-at-a-time. at the start of 2024, there were approximately 1,200 permutations of material type, item policy, and location actually occurring on items in the university libraries’ instance of alma x 33 user groups in use = 39,000 permutations of settings affecting circulation behavior actually in use within alma. at that time, circulation logic rules used all four metadata fields. the 1,200 permutations of item metadata actually in use in alma was a good number for having internal conversations about a shared goal of software explainability. 1,200 is a number people can visualize. 1,200 was also still too many combinations for university libraries employees to meet about and decide how circulation should work for each. going into 2024, employees across the university libraries were frustrated with the state of circulation rules. bloated item level metadata settings and logic rules for circulation had led to regular encounters with apparently chaotic behavior in the ils/lsp. cataloging, which created item records on a daily basis, had centralized notes about settings for item level materials in open stacks, but generally had no explanation of how reserves and equipment items worked. it was common for employees at a checkout desk adding a new reserves item to change settings on the item several times, and then to open a ticket with either cataloging or systems to “fix” the item because they had not been able to achieve a desired checkout behavior. systems felt obligated to provide instructions on how to use the software, while simultaneously realizing that the settings were not explainable. early in the merger, there had been real differences in intended circulation behavior for various kinds of library materials between the two campuses. by 2024, the university libraries had generally brought circulation behavior into alignment by layering logic rules onto non-unified metadata. as a result circulation logic rules and metadata tended to be highly variable from campus to campus. close to merger in 2017, the issue at hand had been the need to meet and make shared decisions to determine shared behavior was for circulation. by 2024, the barrier truly was the software settings. steps in refactoring alma step 1: infrastructure for versioning changes implementing basic infrastructure: naming conventions to allow versioning of circulation logic rules within alma modern software tends to be versioned. using git or subversion (svn) is nearly universal for versioning even simple software. when software behavior is built out through settings in a content management system, the wider coding community has tools for versioning settings. yaml is a language for representing configuration settings which allows importing and exporting configuration settings. yaml can be combined with version control, such as git, to allow versioning of configuration settings (red hat 2023[25]). alma doesn’t have a comprehensive way for customers to export settings. typically, within alma, when looking at a specific kind of configuration setting, there will be an export button in the top right of various settings menus allowing an export, and when there is such an export button, that same export may also be available by calls through the alma application programming interface (api). however not all configuration settings can be exported, and an export allowed from alma may be incomplete in that it may omit important information. in the case of circulation rules within alma, the export includes the letter code, description, and name of the outcome (i.e. name of the terms of use) of each logic rule. the export does not include the logic to get to that outcome. instead, getting that information requires clicking into each logic rule. figure 2. screenshot of the alma page where a circulation logic rules overview is displayed for a fu. this screenshot was taken after implementing a naming convention for the output (terms of use) and for the description of each logic rule. those are labels only. the software behavior is built out by clicking into each logic rule. figure 3. screenshot of the graphical interface for building out a single circulation logic rule from figure 2. each circulation logic rule requires clicking into the rule in order to see settings. the description field is manually written by systems employees, not used by alma, and nothing in the software keeps it in sync with the actual software settings. figure 4. screenshot of an export from alma to microsoft excel. the export includes exact values shown in the overview menu. the university libraries used a standardized naming convention to represent the logic of each rule. in order to do a comprehensive clean up, the university libraries needed to have a reliable way to version settings. to achieve versioning, the university libraries developed a naming convention for the description field of the alma circulation rules. the naming convention for the description field of each circulation logic rule is shown in screenshots above. the description field was edited to include: (1) indication of which item metadata fields were used by that role, followed by (2) pseudo code representation of the actual logic rules. first, at the very start of the description, in brackets, is the initial of any item level metadata fields used by the logic rule. if that item metadata field is used, then the initials are listed. for example, “[mt]” would be the start of a description for a logic rule which references material type in the logic. for example, “[ip]” indicates the logic rule references item policy. for example, ‘“[loc]” indicates the logic rule references location. for example, “[ip,mt]” indicates the logic rule references both material type and item policy. if no item metadata fields are referenced, then the start of the description for that logic rule would read “[null]”. during refactoring, the university libraries phased out use of material type and location in logic rules. because of this, noting what item metadata fields were used prominently and upfront in the naming convention assisted in identifying settings to be phased out, versus settings to be retained and refined. depending on what spec / big picture design a library is working towards, a library might wish to emphasize different fields. second, the remaining description field was filled with text reading “[rules: ” followed by a list of each parameter of that logic rule. user groups was listed first as “user=” and then the list of user groups that rule applied to. (after behavior was streamlined to treat bundles of user groups similarly, names of user groups were phased out, and only the letter of each bundle was listed.) if no user group was listed in a logic rule, then the “user=” part was omitted. following that came “ip=” followed by any item policies referenced in that logic rule. following that came “loc=” followed by any locations referenced in that logic rule. following that came “mt=” followed by any material types referenced in the logic rules. after the naming convention was fully implemented in all fus, the university libraries used the excel export within alma to comprehensively export rule names and descriptions, copy paste all logic rules for all fus into a single large spreadsheet, and then have a record of what the logic rules had been at a specific time. figure 5. screenshot of the spreadsheet showing the june 2024 state of all fus and their circulation logic rules. the spreadsheet continues vertically offscreen and includes a section for each fu including the loan, booking, and request rules for each. with the spreadsheet of comprehensive rules on file, the university libraries could later export a current report of settings for any fu and then used diff to compare the rows for that setting against the most recent comprehensive export. websites like https://www.diffchecker.com/ allow quickly comparing two text files and will highlight additions and deletions. during the refactoring process, the university libraries pulled a fresh comprehensive spreadsheet approximately monthly, and kept all comprehensive spreadsheets in a shared folder in google drive. this allowed later retracing steps, if needed, and allowed answering questions. during the refactoring process, employees at checkout desks were told to look out for odd behavior and report problems. during the early weeks of refactoring, it was routine for employees at checkout desks to report preexisting odd behavior. the spreadsheets of past circulation logic rules were helpful in confirming times when systems had not introduced new problems, and to isolate and quickly correct any problems inadvertently caused during the refactoring process. for reference, initially, with approximately 17 fus in use at the start of the refactoring, exporting logic rules from all fus and putting them into a single comprehensive spreadsheet took around an hour. figure 6. diffchecker report comparing the 2024 june vs the 2024 august spreadsheet with comprehensive reports of all fu circulation logic rules. diff is a common data comparison tool, and many different software packages and operating systems can be used to diff two text files. the naming convention had two immediate benefits. first rules could be versioned. versioning allowed rolling back to a previous state, if needed, which helped to prevent downtime in the case of accidentally introducing problems. second, versioning led to improved communication. versioning is a passive form of communication between computer programmers, and with a versioning process established and agreed on, multiple employees could edit the circulation logic rules and keep shared notes. this allowed cross training within systems. the naming convention enabled employees in technical roles to quickly see and understand the circulation logic rules and have a conversation on the fly oriented around the spreadsheet and without extensive preparation before a meeting to familiarize with a specific fu’s logic rules. this enabled smooth on-demand conversations within systems and between systems, cataloging, and employees working with checkout. naming convention for terms of use each terms of use is built out within the alma software by the customer and is highly customizable, containing the exact outcome behavior of a checkout (whether an item is requestable, how long a loan is, what happens when a loan due time falls during a library closure, etc. (exlibris 2025)). terms of use can exist at the institutional level or the library level within alma. another nuance which can make terms of use harder to document is that a terms of use will apply to either loans, booking, or request. multiple terms of use can have identical names. before refactoring, there were multiple terms of use with identical names. when building out a circulation logic rule, and using the drop down menu to select the terms of use, the only setting which shows is the name of the terms of use. to quickly differentiate terms of use, the university libraries labeled the name of each terms of use with a bracketed note at the end stating whether the rule existed at the library vs institutional level, and whether it applied to loans vs booking vs request. for an institutional level loan terms of use, “[inst., loan]” was appended to the terms of use name. for a library level terms of use, “inst.” was replaced with the name of the library. for example, a terms of use in existence at the lane library level within alma and applicable to request had “[lane, req]” appended to the name of the terms of use. this naming convention had three benefits: (1) it completed the naming convention for the circulation logic rules described above, and allowed versioning (see output column in figures 2 and 5); (2) a goal in refactoring was to eliminate settings built out at the library level, and this assisted in identifying which terms of use existed at the library level, and (3) in the early stages of refactoring standardized naming assisted in locating the details of each terms of use within alma’s configuration area. data dictionary for later reference logic rules for circulation behavior in alma can reference patron’s user group, or item’s material type, location, or item policy in order to determine software behavior for circulation. before refactoring, the university libraries anticipated dramatic changes to those fields. a data dictionary is a “centralized repository containing information of the data in the database such that the meaning, relationship, source of the data, where it will be used and the format is clearly mentioned or specified” (mcdaniel 1994[26]). because of anticipated dramatic changes to those four fields, before changing settings, the university libraries documented existing constraints on settings for each of these fields with a data dictionary for those fields. in general, alma’s configuration area has a table for seeing what user groups have been created, what material types are enabled, what locations exist at each physical library, and what item policies have been created. when viewing each of these tables in the configuration area of alma, it’s possible to click a button in the top right corner to export a spreadsheet of that table. there are also instructions for exporting these settings by alma api posted to https://developers.exlibrisgroup.com/alma/apis/conf/ . the university libraries exported each table representing all settings, and copy pasted all tables together in a single google doc file. the university libraries labeled each table, and an introduction to the data dictionary explained that the purpose of the data dictionary was to document settings before streamlining them, in case of having to retrace or having to reference the information later. exporting the settings was a one time process for the pre-refactoring data dictionary and exporting through the user interface was the most straightforward way to get a one-time export. naming conventions were implemented before assembling the data dictionary. the most important information is not inherently included in the export, but rather has to be added with a labeling convention or other way of recording setting-by-setting notes. regarding terms of use, the university libraries exported the detailed outcomes of each terms of use which was in use anywhere on the alma site, and added each as a separate table in the data dictionary. this allowed retracing steps as terms of use were consolidated and less-used or library-level terms of use were eliminated. because alma allows multiple terms of use to have identical names, the naming convention was implemented and names were made to be unique before exporting each to the data dictionary. only terms of use which were actually referenced by circulation logic rules were exported to the data dictionary. in addition to allowing retracing steps during refactoring, the data dictionary allows later inspection of alma analytics reports, for example, association of college and research libraries (acrl) academic library trends and statistics survey survey reports and integrated postsecondary education data system (ipeds) reports. after refactoring and dramatic changes to settings, if a setting which had been used in an alma analytics report is found to be impacted or broken by changes, it is possible to refer back to the data dictionary and see what the change was which had broken that report. in some cases, pulling a report after clean up might require temporarily recreating a setting within alma, and the data dictionary allows recreating the setting if needed. while eliminating settings, systems looked at whether deleting the setting within alma would alter a statistical report. this included looking at alma analytics, and at panorama, a statistical program which can accept exports of alma data and in which the university libraries’ historic alma data will be stored in long term. deleting settings from the configuration area of alma does not tend to cause trouble with day-to-day behavior within alma, but can cause problems with statistical reports. step 2: writing a written spec for software behavior in spring 2023, no comprehensive standard for how circulation should work in alma existed. a public webpage explained checkout behavior to patrons by giving checkout times for books in the stacks, audiobooks in the stacks (a genre which had been almost fully weeded with a shift to streaming content for audiobooks), and for dvds, for a handful of demographics of patrons. patrons were instructed to contact each checkout desk separately for information about checkout times and availability of other materials. a reason which necessitates refactoring might be that software might tend to initially be designed, and then over time to have small edits made to implement new features, and those edits be made without the big picture design in mind. refactoring can bring the overall software back to the big picture design. design patterns provide targets for refactorings and refactoring can bring software back into alignment with the overall software design (fowler and beck 2018[27]). in order to bring software into alignment with overall design, the university libraries first documented the overall design. technical framework at the university libraries, systems planned to structure circulation behavior on three metadata fields: patron’s user group item’s location item’s item policy material type would not be used, because it acted as a description of the item, and so was felt to have descriptive truth as to what the item was. alma’s controlled vocabulary for item material types includes a code and description for each material type. descriptions are terms like “book”, “cd (audiobook)”, “blu-ray disc”, and “film 16mm”. values were not configurable in that the description could be changed but the code could not, and the list of material types could not be added to. the descriptive nature was hard coded into alma. location would be used to designate specific locations as noncirculating. then within circulating locations, item policy would determine circulation behavior for each user group. before refactoring, noncirculating locations were generally already grouped into the same noncirculating fus within alma. no clean up of metadata was needed to account for noncirculating locations, and conceptually, according to the spec, any location in alma would have one of two behaviors for purposes of circulation behavior: either noncirculating or circulating. regarding user groups, the university libraries eliminated the least used of these and grouped the remaining user groups into bundles. user groups with very few members or with many members all of who were expired were eliminated. members were migrated into the closest user group. for example, multiple community colleges in the region had user groups in the university libraries’ alma, with accounts created ad hoc and just a handful of accounts in each. patrons from all community college user groups were moved to the “community” user groups for public patrons. once empty, user groups were not deleted in alma, because this would impact ability to pull statistics by obscuring user group information in statistics. posted circulation times on a public facing website addressed open stacks materials availability to current undergraduate students, current graduate students, faculty and staff, and community members. additionally, the gil consortium operates gil express, shared print borrowing, with standardized loan times for all materials (gil 2025), representing a fifth clearly defined user group. within alma, these user groups tended to be accounted for and referenced within existing logic rules for circulation, while other user groups might be haphazardly referenced, or not referenced at all. because of this, those five groups were taken as the starting point for narrowing user groups into bundles, and each bundle was labeled with a letter. the core user groups for defining behavior were: (a) gasou students (b) gasou graduate student (c) gasou faculty/staff (d) community (e) usg gil express patron for other user groups which were to be retained long term, systems made a recommendation as to which letter to bundle that user group along with, and then presented the recommendation to the university libraries’ dean’s advisory council, consisting of university libraries department heads, and to the collection development committee (cdc). based on this, user groups to be retained were bundled into five bundles, and each user group was renamed with a letter for that bundle. for example, the (a) user groups are: (a) gasou students (a) ghp fac & staff (a) east georgia students (a) medical college of ga students individual user groups were retained long term. batch load processes for adding users to existing user group were kept in place. pulling up a patron record and seeing the user group tells something about that patron and what their relationship to the university libraries is. patrons were not moved all into only five user groups, but rather, each user group was renamed to start with (a), (b), (c), (d), or (e), and all user groups starting with (a) were to have identical circulation privileges as one another. ultimately, 13 user groups were retained, and were grouped together into five bundles each with identical circulation behavior. bundling user groups and lettering them both aided conversations about how circulation worked by limiting the complexity of checkout behavior decisions to a smaller number of conceptual groups of patrons, and aided in comprehensively accounting for all user groups within alma. alma circulation logic rules are built out in a graphical user interface which sorts user groups alphabetically. bundling and lettering the user groups makes it fast to quickly add the correct user groups when editing a logic rule, because all user groups starting with “(a)” will display together when building out a rule (see figure 7). figure 7. screenshot of the graphical interface for building out a circulation logic rule. when adding user groups as a parameter, alma displays an alphabetical list of user groups. after implementing the naming convention, all user groups starting with “(a)” sort together, and it is fast and easy to pull all up and add all at the same time. it is also obvious if one user group in that lettered bundle of user groups has been omitted. within alma, item policy is fully configurable. an editable table within alma defines the code and the description. an item policy can have a code or description which is descriptive of time frames, such as “1 week, renewable”, something abstract, such as “cotton candy”, or something descriptive of the item, such as “book”. before refactoring, within the university libraries’ alma, the item policy descriptions which show in the drop down menu while editing an item tended to either describe the item (i.e. “book”) or to give a timeframe (i.e. “4 hours”). as purely a software construct and configurable in such a way that it does not have to be descriptive, item policy is a better fit for software behavior than are the descriptive metadata fields. additionally, when software problems were reported to systems, if an item policy stated a timeframe, such as “1 week”, “2 hours”, “4 hours”, etc. and checkout time was different, the discrepancy tended to get reported as a problem. employees across the university libraries had difficulty understanding that location or material type could impact circulation behavior, and tended to want item policy to control the behavior if they saw a label stating a timeframe. because of this tendency of university libraries employees to orient conversations about problem behavior around the human readable item policy description, moving circulation behavior to work off of item policy for circulating locations was in line with getting the software to work in the way that people already expected it to work. motivating employees across the libraries going into 2024, there were 121 item policies built out in the item policy table in alma. this was too many to define behavior for, and too many to summarize concisely. this was also a source of daily aggravation for employees working with item level metadata. when editing an item, item policy is added and edited using a drop down menu. 121 options was too many to easily view all at one time in a drop down menu. aggravation with the drop down menu and a shared goal of shortening it was a motivator early on in refactoring. at the start of refactoring, systems ran reports in alma analytics to identify less used item policies. several item policies were applied only to items in withdrawn locations. (items which are withdrawn at the university libraries are cataloged to a withdrawn location in the software which allows for later auditing regarding disposal of state property, and other recordkeeping.) the university libraries initially created a new item policy, “withdrawn”, batch changed the item policy on all item records in withdrawn locations to be “withdrawn”, and then deleted item policies which were no longer assigned to any item from the alma item policy table. this reduced the item policy drop down to around 80 options. this had a motivational effect, as employees working with item level metadata noticed a shorter drop down, and as all removed options were options had been applied to long gone items and were not in use in anyone’s workflows. systems then analyzed remaining item policies. a trend, which had occurred in reserves was that with employee turn over, a new employee would not have notes as to how settings worked and would request a new behavior be created in the software. systems didn’t have notes as to how settings worked and would implement by building out new settings, rather than editing existing settings. as a result, many item policies appeared to have duplicative intent. there were several settings like “overnightll”, “1 day reserve”, “next day” which all seemed to indicate the same intent of a one day checkout. there were settings for “1 week reserve”, “1 week – no renewals”, and “7-night reserve_ll” which all seemed to indicate the same intent. similarly, the item policies ending with “_ll” tended to each have a corresponding item policy without “_ll”. numerous item policies matched with a smaller number of desired outcomes for checkout behavior. existing software settings for item policy were repetitive and chaotic. because of this, existing software settings were not included in documents shown to focus groups when writing the spec about how alma should work. figure 8. view of the item policy drop down menu when editing an item in alma taken before refactoring. note the scroll bar along the right hand side of the drop down menu. going into 2024, and before refactoring, there were 121 options in the drop down menu – too many options for a drop down and better suited to an autofill field. while the screenshot only shows settings beginning with numbers, which tend to be time periods, numerous other settings for item policy were labeled things like “dvd” and “book_ll” (i.e. abstract words inviting dramatically different interpretation for behavior). in order to define a set of intended behaviors for the spec, a series of focus groups were held. rather than orienting conversation around software settings, systems oriented focus groups around a human readable description of how long the checkout period was supposed to be for each combination of user group letter bundle and each kind of material circulated. focus group documents were largely platform agnostic, and could have been used to establish goals for any ils/lsp, not limited to alma. most time was spent on checkout time. while circulation behavior is more complex than just checkout time, checkout time was simple enough to have a comprehensive conversation across kinds of items and patrons, and checkout time might tend to be the most important core behavior for a loan. for stacks materials, systems prepared a table of kinds of materials (i.e. books, periodicals, dvd movies, government documents), and how long each appeared to currently circulate to each bundle of user groups (see figure 9). systems presented this to the cdc, and got any corrections there. figure 9. screenshot of the document shown to the university libraries cdc for approval and corrections as to big picture intent for behavior. an introduction to this focus group document went over how user groups were bundled. while getting feedback, systems might raise specific user groups as appropriate. regarding items kept behind a checkout desk, systems held a series of focus groups with the department heads over each of the three checkout desks. the first focus group was open ended brainstorming to identify every kind of circulating material (i.e. whiteboard markers, clickers, calculators, projectors, reserves options offered to professors placing items on course reserves). each kind of thing circulated, equipment, reserves, etc. was added as a row to the bottom of the table in figure 9. then two focus groups were held to determine how long each kind of item was intended to circulate to each of the five bundles of user groups. as desired checkout times for each item were finalized in discussion, that row was highlighted in green to note that all relevant stakeholders had agreed on the desired outcome. when all desired checkout times had been agreed on for each kind of circulating material, systems grouped desired outcomes. thirteen groupings of checkout behaviors emerged organically. with a goal of moving all circulation behavior to work off item policy, this meant 13 item policies would be defined in the spec. a goal had been to reduce to under 25 item policies, in order to meaningfully fit within the drop down menu displayed when editing an item in alma, but there was no set number to reduce to. if more combinations had emerged, then more would have been included in the spec. in reducing to 13 item policies, systems matched to existing item policies with preference given toward retaining item policies which tended to be assigned to more records. when an existing item policy had an abstract description within alma, such as “standard”, “dvd”, or “camera”, systems renamed the description within alma to include both the abstract term and the amount of time it was checkoutable to undergraduate students (the most numerous patron group). in this way, all circulating item policies now appear in the alma interface with a label saying how long that item will circulate to undergraduate students. in the future, the abstract portion of the item policy description will be removed. hence, “dvd” was renamed to “dvd (1 week, renewable)” during refactoring, and in the future will be renamed to “1 week, renewable”. the purpose of retaining existing item policies was to avoid adding additional options to an already too-long dropdown menu. reducing the number of options in the dropdown menu was a shared vision everyone working within the alma software could understand, and throughout the refactoring, the gradually shrinking drop down menu and the reduced number of clicks in day-to-day work was a motivator for collective action. incrementally changing names of item policies to be retained was to allow continuity in workflows and notes for anyone using those specific settings. after matching outcomes to item policies, systems added the item policy for each kind of circulating thing to the original focus group document (see figure 10). figure 10. finalized table after completing focus groups showing goal big picture circulation behavior at the university libraries for each kind of circulating item, and showing item policy to be applied in the rightmost column. after refactoring, this same chart could be used to go from real world what-the-item-is to a suggested item policy to assign in alma. systems then conducted a focus group with the heads of each checkout desk to discuss detailed behavior for each item policy. additional circulation behavior beyond checkout time and requestability included: whether a loan shortens or lengthens if the due date falls during a closure; grace periods; whether an item can be recalled while on loan; any limits on reloaning that same item to the same patron; time on the hold shelf before reshelving for requestable items; time overdue for the item to age to lost; and time after a return for checkout history to age off. this more detailed behavior was added at the end of the google doc containing the table in figure 10, and is not reproduced in this article. the more detailed spec was rarely used as a communication tool for messaging about change, but rather was used by systems to quickly know without additional conversation what was and wasn’t a bug if changes in nuanced behavior were reported during refactoring. simplicity in the spec was achieved by limiting the number of options, in the form of fewer defined sets of behaviors, but not in the form of failing to account for settings for each set of behaviors. a small number of options were each defined comprehensively. essentially, all items sharing the same checkout times for the same bundles of user groups would also all have identical behavior in every other way. for example, all “1 week, no renewal” checkouts with a due date falling at a time when the library was closed would shorten to the closing time closest to closure. complexity in software might inevitably be a result of complexity in the workflows which that software supports and complexity in the real world (brooks 1987). streamlining and unifying real world workflows in support of explainability was a key step in a unified software design to support streamlined, maintainable, and explainable software. the written spec to work towards consisted of: 2 location behaviors x 5 chunks of user groups x 13 item policies = 130 permutations of settings to account for. 130 permutations can be explained concisely. at this time, after completing the detailed spec, a libguide page with a concise explanation of checkout behavior in alma was posted on the university libraries intranet, with clear labeling stating that behavior was not yet set up. that is attached as appendix a. table 1. chronological list of steps and focus group topics to write the software spec. systems analyzed user groups. systems coordinated with library administration, including checkout desk supervisors, to discontinue several less used user groups. systems made recommended bundles of the remaining user groups. systems analyzed stacks materials (non-reserves and non-equipment items) and made a table (see figure 9) listing kinds of stacks materials and desired behavior for each. systems met with the cdc, reviewed the table in figure 9, made corrections based on feedback, and answered questions. the cdc also approved the bundles of user groups. this was approximately 30 minutes conducted as an agenda item during a standing meeting of the cdc. systems conducted an hour long focus group with all three checkout desk supervisors to make a comprehensive list of kinds of circulating equipment and reserves items. systems added each kind of circulating reserves or equipment item to the table in figure 9, looked at how that kind of item tended to behave in alma before refactoring, and prefilled the table. prefilled information included discrepancies in patron experience from building-to-building or item-to-item, and in case of discrepancies included a recommendation for checkout time. systems held two focus groups each an hour long with all three checkout desk supervisors to go row by row through kinds of items in the table in figure 9 and define intended checkout time for each. as desired checkout time was finalized, systems highlighted that row green to indicate consensus. systems analyzed intended behaviors for checkout, and identified discrete options. for each outcome, systems selected an existing item policy which had similar behavior in at least some fus, and added this to the first column of the table. systems added the item policy to be assigned to that type of item (see figure 10). systems conducted an hour long focus group with all three checkout desk supervisors to define detailed behaviors for each item policy defined in the spec. this information was added to the end of the table in figure 10 in a separate area with detailed unified behavior for each item policy. the detailed behavior represented the completed spec. systems posted a concise “one-pager” instruction sheet on the university libraries intranet (see appendix a), with notations that settings were not yet implemented. early in the process dissatisfaction with alma and the promise of streamlined settings (i.e. shorter drop down menu when editing) motivated participation. later, after writing the spec, while implementing settings changes, software explainability, as manifested in the “one-pager” was a motivational goal. communications oriented around the simpler spec representing explainability. step 3: decide on a test procedure for testing changes the university libraries did not comprehensively test changes during refactoring, but rather looked at a threat model of which software bugs were acceptable to occur and fix later vs which were unacceptable. systems tested only for unacceptable problems. refactoring is a process of many small changes (fowler and beck 2018[28]). meanwhile, comprehensively testing all aspects of software behavior is overly time consuming for a small change. prior to refactoring, the circulation rules were so baroque that any change tended to have unintended consequences. in developing a test procedure for testing changes, systems considered a threat model of what problems could occur which would cause severe harm. this narrowed down to two potential severe problems. first, special collections materials becoming requestable within the alma software was a serious threat. the vast majority of special collections materials at the university libraries are stored in a three storey high automated storage and retrieval system (as/rs) high density storage facility, referred to as the automated retrieval collection (arc). for materials in the as/rs / arc which are requestable, patrons can request those directly through the public catalog. when a patron places a request, a computer at the circulation desk prints a slip with the item information on it, and a robot crane fetches the bin with that item in it and places it on a counter for a checkout desk employee to retrieve. items are regularly retrieved by student workers fully under the supervision of the henderson library checkout desk and not under the supervision of special collections. while, at the time of install for the as/rs / arc, it would have been possible to physically separate access to arc storage bins by putting in multiple counters on physically separated floors and tieing dedicated storage bins to specific counters, that was not done at the time of installation. retrofitting for physical separation would be a massive construction project, and is not under consideration. as a result, software, specifically the request feature in the ils/lsp, is the only barrier to prevent physical access to special collections materials. due to apparent chaotic circulation behavior within alma, going into 2024, student workers at the checkout desk serving the as/rs / arc all had permissions within alma to override blocks on checkout. if a special collections item were to become requestable within alma, and a patron were to place a request, a student worker at the checkout desk would be the only person in a position to catch the error. the second serious harm which might occur is for a short term loan item, such as a highly in-demand laptop or graphing calculator or a reserves textbook to become checkoutable for either the standard 4 weeks with two automatic renewals for undergraduate students, or the semester long checkout with two automatic renewals for employees. in that case, the item could become unavailable to other patrons for an extended period of time. regarding special collections items in the as/rs / arc system, all were in the same three locations within alma. all locations were in a fu within alma for noncirculating items, and so did not circulate and were not requestable based on location. before changing settings within alma, systems met with the special collections librarian, showed how to confirm whether or not an item was requestable within alma, and identified a special collections item within each alma location. systems explained the refactoring process and timeframe, then gave updates along the way. systems built an alma analytics report to track checkouts of special collections items, and checked this periodically during and after the refactoring process. systems also provided the special collections librarian permissions within alma and a written instruction sheet to allow her to confirm at any time that no special collections items were currently checked out. regarding short term loan items, when long term loan behavior was rolled out to reserves and equipment locations within alma, systems did additional metadata reports regarding items in locations with mostly short term loans (i.e. reserves and equipment locations). if an item had a problematic setting (i.e. an item policy destined for long term loan behavior), systems aggressively pursued getting the desired checkout time from the appropriate checkout desk and coordinated with cataloging to ensure metadata was corrected before changing software settings. throughout the process, systems examined each settings change for potential to impact requestability of special collections items, and potential for having a short term loan item go out on a long term checkout. step 4: rolling out changes the core of refactoring is making many small changes to code/settings, without changing the behavior of the software. problems in software caused by refactoring should be minor and quickly fixable, due to the nature of changes as small and incremental (fowler and beck 2018[29]). the university libraries operates nearly daily throughout the year, and the longest closure is a typically less than two week closure for winter break. implementing dramatic change in one big sweep was not feasible nor compatible with daily year-round operations. closing for six months to change logic rules, do metadata cleanup to match changes, and test all changes was not an option. instead, changes went out weekly during summer and fall 2024. implementing changes in small steps over a longer time period was beneficial in that problems tended to be easier to connect back to a specific change and to fix. institutional will to complete the process was important, so that problems would be reported and collaboratively fixed, rather than treated as a reason to stop. throughout the process, systems communicated that checkout behavior was likely to break at time, and that the goal was for any problems to quickly be reported to systems and fixed. the dean of the university libraries also got updates about the progress of settings cleanup and refactoring throughout 2024. during roll out for settings changes, minor problems occurred regularly and were quickly resolved. the kinds of problems that might occur were things like a 4 hour checkout item with a due time falling during overnight closure had the checkout time automatically extended to closing time the next day, rather than shortened to closing time the same day, as a result of changing away from a library level terms of use to the closest institutional level terms of use. when reported, that problem was investigated and fixed libraries-wide by referring to the detailed written spec which stakeholders had already approved before refactoring began. versioning internally, the university libraries have an alma listserv. any changes to alma configuration settings are announced on the internal listserv, with the idea that this allows later retracing steps. exlibris provides an alma test server and alma production server to each customer. data and settings from production are copied to test every six months, and so generally the test server includes a full copy of production data. at the university libraries, refactoring involved settings changes, and metadata cleanup. all settings changes were made on the test server, announced on the internal listserv, and then rolled out to the production server the next week. initially, changes were made on production on friday mornings, since friday is the slowest day. early in the refactoring project, this changed to a practice of moving settings changes to production on mondays, on the theory that early in the week, more people in technical roles were at work and able to quickly route and fix any problems promptly. metadata cleanup was done directly on the production server. before beginning refactoring, the naming scheme for circulation logic rules was rolled out to alma test then production. all circulation logic rules, with a description which now fully described the logic within the rule, were exported to a spreadsheet. approximately monthly throughout the project, the same process was repeated to allow viewing a relatively recent version of rules in a spreadsheet. as each logic rule or setting was changed, the email notification to the internal listserv included a screenshot of the before and after of the logic rule. by looking at the most recent spreadsheet of settings and then the incremental changes messaged out to the internal listserv, it was possible to know the state of the circulation logic rules on any given date. streamlining user groups during spring 2024, the systems rolled out changes to user groups. changes to user groups settings consisted of changes the descriptive names of user groups to add the letter designation (i.e. adding “(a)”, “(b)”, “(c)”, “(d)”, and “(e)” to each user group name), and then working through existing logic rules for checkout and adding less used groups side by side with the more commonly used user groups. for example, “(c) gasou faculty/staff” was almost universally set up to be able to check items out. meanwhile, “(c) emeriti faculty” was referenced in only a few places. that was despite faculty handbook provisions guaranteeing emeritus faculty equal library privileges to current employees. because there were relatively few emeritus faculty compared to current employees and to community members, problems with checkout occurred far more rarely, tended to have been fixed with an override rather than reported, and so checkout had never been comprehensively configured. bundling the more heavily populated user group together with the less populated user groups together helps to ensure that any problems with configurations will be reported. meanwhile, data cleaning was straightforward. few patrons were moved to another group, but rather groups were renamed to include the letter designation. some of the less populated user groups were emptied, and members moved into the user groups which were to be retained. when a user group was empty of patrons, the description of the user group in alma was edited to indicate that the user group was empty and no longer in use. user groups were not deleted out of alma. each user group has a code and a description within alma. by leaving the user groups in the code table, the connection between code and description remained in place in case of looking at older statistical data. streamlining circulation logic rules for open stacks materials during summer 2024, open stacks materials were addressed. regarding open stacks materials, at lane library, many had legacy settings for the item policy including “_ll” in the name. prior to refactoring, circulation logic rules at all relevant fus rarely referenced item policies in determining checkout times of materials, but rather referenced combinations of location and material type. over summer 2024, cataloging did batch changes to eliminate the numerous “_ll” item policies in the open stacks and arc, and to replace them with item policies which would be retained. after batch changes, systems changed circulation logic rules for circulating stacks and arc fus to now work off item policy. because this was a massive amount of changes to item records, changes were planned in such a way as to not be time sensitive. systems checked over circulation logic rules, opened a ticket with cataloging to do a specific batch change, cataloging could make the change time allowing (turnaround time might be promptly or could be months later), and then systems finalized circulation logic rules and deleted the now unused item policy out of the table in alma. during fall 2024, systems comprehensively rolled out item policies to all circulating locations. with 13 item policies to roll out, comprehensively rolling out item policies took around 14 weeks. each item policy was built out first on a single fu. generally that was a fu which included locations which had many items with that item policy applied. that was done ahead of time by several weeks. then during the fall 2024 semester, one item policy per week was rolled out comprehensively to all circulating fus at the university libraries. roll out meant that that item policy was built out identically on all circulating fus using the exact same circulation logic rules and pointing to the exact same terms of use. alma reads circulation logic rules top to bottom for each fu, and applies the first matching rule to a loan. each newly launched logic rule was sorted early in the list of existing logic rules, such that existing logic rules remained in place as a fall back during the refactoring process. during fall 2024, as each item policy was rolled out comprehensively, reserves and equipment materials were affected. each week, the department head over each circulation desk got a concise human-readable summary of what the intended impacts were from a change and what problems to look out for. each department head also got a spreadsheet report of all items, if any, with the newly rolled out item policy assigned and was able to review it and ask any questions or address any problems. after the 13 item policies were comprehensively rolled out in alma, each checkout desk got a report of all items with item policies not to be retained long term, and was able to use the written spec to select a new item policy for each. systems met with the head of each checkout desk on screensharing to look at the report and was able to add additional columns to assist in identifying items and sorting similar items together. following comprehensive roll out of settings and metadata cleanup, all circulating fus behaved similarly. locations were consolidated into a smaller number of fus, and the fus which had been emptied of locations were deleted. the alma install went from 17 fus to six fus. lastly, the older circulation logic rules predating refactoring were deleted from remaining fus. figure 11. example email announcement of an upcoming settings change announced to an internal listserv. when the change was later rolled out to production, a confirmation went out to the same listserv. figure 12. a circulating fu at the end of the refactoring process. item policies defined in the spec were comprehensively set up in all circulating fus. as metadata clean up completed on legacy item policies which would be discontinued, fus were consolidated. streamlining terms of use at the start of fall 2024, highly varied terms of use were also addressed. going into 2024, numerous terms of use had been built out in the alma software, and were not referenced by any circulation logic rule at all. this included test settings, and deprecated settings, such as circulation behaviors which had been in use during covid closures. of the terms of use which were referenced, names indicated multiple terms of use accomplished similar intended behavior. for example, four separate terms of use defined checkout for four weeks with two renewals. within alma, a terms of use can be deleted only if it is not referenced by any circulation logic rule. before beginning refactoring, the university libraries deleted all terms of use which could be deleted (i.e. all terms of use which were not referenced by any circulation logic rule). this reduced the total number of library and institutional level terms of use from 143 to 61. unreferenced and deleted terms of use were not documented. regarding the 61 remaining terms of use, the detailed settings for each was copy pasted into the data dictionary. as terms of use were later eliminated, each was marked “retired” in the data dictionary. as terms of use detailed settings were edited, the edit was noted in the data dictionary with a redline of the change, and a note as to what date the change was made on alma test and alma production. during the first four weeks of the fall 2024 semester, systems consolidated the existing terms of use. any library level terms of use were eliminated. to do this, systems used diff to compare the detailed terms of use behavior settings within alma for the terms of use at the library level against terms of use at the institutional level which had names indicating a similar behavior. systems identified the institutional terms of use with the most similar (or identical) behavior. systems then scheduled a meeting with the department head over the checkout desk at each building. the meeting agenda included a summary of changes to terms of use which affected locations served by that desk, and notes about any apparent differences between the terms of use to be deprecated and the new target terms of use. during the meeting, systems changed the circulation logic rules for the building that that person’s checkout desk was located in away from referencing library level terms of use and to reference the most similar institutional level terms of use for each. by meeting on screensharing and clicking through and explaining the change during the meeting, heads of checkout desks could know that a change had been made, and could look out for subtle changes to behavior. meetings included a maximum of three functional changes (i.e. consolidating all the week-long terms of use was one functional change, consolidating all the 4 day terms of use was another functional change), with the idea that that way the department head had a limited number of areas to look for problems in over the following week. eliminating library level terms of use and instead referencing only settings at the institutional level further reduced the number of terms of use to 36 early in refactoring. additional consolidation of terms of use with names indicating a similar intended behavior reduced the number of terms of use at the library and institutional level to 28 terms of use. impacts of refactoring rolling out the streamlined settings and doing metadata cleanup had minimal impacts on day-to-day operations. by rolling out changes a little at a time, problems were minor and could be quickly resolved when reported. the effort was largely time spent in analysis of settings by systems, and metadata cleanup by cataloging. the process was not disruptive to checkout desks, not labor intensive, and involved keeping an eye out for unusual circulation behavior and reporting problems to systems and cataloging, which is something checkout desk already do on a day-to-day basis. minimal disruption is by design. the essence of refactoring is small incremental changes, in order to avoid dramatic disruption (fowler and beck, 2018[30]). also, key to not disrupting operations was the idea of streamlining existing behavior. during the refactoring process, settings and metadata were extensively edited, with cataloging changing item policies for around 20% of the physical items in the university libraries. the actual circulation behavior rarely changed for circulating items. instead, during refactoring, a previous configuration in which dramatically different logic rules and metadata came to a similar outcome at different locations was replaced with the current configuration in which standardized logic rules and metadata come to an identical outcome at all locations. while it may seem like a lot of analysis and effort to refactor the software, the time was quickly gained with more efficient overall workflows. before refactoring, simple changes in checkout times for specific kinds of items took an inordinately large amount of analysis. not having an explanation of how the software settings worked was a constant frustration for employees in varied roles. the benefits of refactoring are that problems with checkout behavior of any individual item can now be promptly solved by referencing a concise explanation of settings (appendix a). cataloging, checkout desks, and systems can reference the shared documentation to quickly route and fix the problem, and to understand what options are already available for circulation behavior when cataloging a new item or launching a new collection. time spent refactoring is recaptured with the day-to-day efficiencies of software that works smoothly. objective proof of improvements due to refactoring may be apparent in the rate of overrides on checkouts. in fall 2023, approximately 16% of checkouts at henderson library, the busiest checkout desk serving the building with the largest physical collections, were accomplished by overriding a block on checkout, with numerous blocks being “item is not loanable” indicating that combination of metadata settings was not configured for checkout within alma. in december 2024, that number fell to just over 2% of checkouts accomplished with an override. midway through the fall 2024 semester and midway through the refactoring process, permission for student workers to override blocks on checkout was comprehensively removed without ill effects. conclusion refactoring is the process of restructuring existing code, in order to make the code easier to maintain, and without changing the behavior of the software. while software might over time fall away from a cohesive design, as new features are implemented one-at-a-time without explicit awareness of overall design, refactoring is an opportunity to bring a complex piece of software back into compliance with a comprehensive design. the university libraries comprehensively refactored circulation behavior in the alma ils during 2024. refactoring involved establishing basic infrastructure in the form of workflows for versioning software settings in alma, writing a spec for software behavior, and then making incremental changes to streamline software settings. by developing and vetting a written spec with involvement from all stakeholders, and then making incremental changes in support of the design, overall software function improved. improvement is shown tangibly by ability to quickly troubleshoot and resolve problems, which previously could only be resolved with long turnaround time and high effort, and by a dramatically reduced rate of overrides when checking items out to patrons. acknowledgements the following people performed essential substantive work related to refactoring the university libraries’ alma instance: rebecca hunnicutt, collection management librarian and assistant professor, georgia southern university libraries; justin barnett, network services coordinator, georgia southern university libraries; greg vaughan, metadata quality manager, georgia southern university libraries. footnotes [1]within alma, the alma interface label two settings “material type”. one material type setting refers to alma’s labeling of a controlled vocabulary defined within the bibliographic level leader 06 and 07 and 008 fields, and is not configurable. the other material type setting refers to a configurable item level metadata setting within alma, and is not part of the marc standard. within this article, all references to “material type” within alma refer only to the item level metadata setting. references [22]bell s. moving to mobile: space as a service in the academic library. educause rev. 2022 apr 15. https://er.educause.edu/articles/2022/4/moving-to-mobile-space-as-a-service-in-the-academic-library brooks f. no silver bullet: essence and accidents of software engineering. ieee comput. 1987;20(4):10-19. https://dl.acm.org/doi/pdf/10.1145/130844.130856 [11]chan lm, salaba a. 2023. cataloging and classification : an introduction. 5th ed. lanham (md): rowman & littlefield 773 p. https://doi.org/10.1080/01639374.2024.2406770 [3]coleman d. growth promised as armstrong, georgia southern merger made official. savannah morning news. 2017 jan 20. https://www.savannahnow.com/story/news/2017/01/10/growth-promised-armstrong-georgia-southern-merger-made-official/13899055007/ [23]exlibris. voyager to alma migration guide. [accessed 2024 dec 30]. https://knowledge.exlibrisgroup.com/alma/implementation_and_migration/migration_guides_and_tutorials/voyager_to_alma_migration_guide exlibris. configuring physical fulfillment [internet]. [accessed 2025 jan 2]. https://knowledge.exlibrisgroup.com/alma/product_documentation/010alma_online_help_(english)/030fulfillment/080configuring_fulfillment/020fulfillment_infrastructure/configuring_physical_fulfillment [17][18]florida academic library services cooperative (falsc). 18itn-06aj invitation to negotiate next generation integrated library system (ils). 2018 dec 22. on file with the author. [9]florida virtual campus (flvc). florida virtual campus annual report 2013-2014. florida virtual campus; 2014. [27][28][29][30]fowler m, beck k. refactoring: improving the design of existing code. addison-wesley professional 2nd edition; 2018. [1]galileo interconnected libraries (gil). exlibris press release: university system of georgia selects ex libris alma as the system’s next-generation library management solution [internet]. athens (ga): university system of georgia; 2015 jul 30. https://gil.usg.edu/alma_implementation/announcements/exlibris-press-release [2]galileo interconnected libraries (gil). institutions live with alma and primo (opac) [internet]. athens (ga): university system of georgia; 2017 jun 27. https://gil.usg.edu/alma_implementation/announcements/institutions-live-with-alma-and-primo-opac galileo interconnected libraries (gil). gil express: general information [internet]. athens (ga): university system of georgia; [accessed 2025 jan 1]. https://gil.usg.edu/gil_express/information/general-information [5]ganz m, slavin l. 2024. system migrations: best practices and lessons learned. the southeast. libr., 72(3), article 4: 5-10. https://digitalcommons.kennesaw.edu/cgi/viewcontent.cgi?article=2098&context=seln [19]goek s. the state of u.s. academic libraries: findings from the acrl 2023 annual survey. chicago (il): association of college and research libraries; 2024. https://www.ala.org/sites/default/files/2024-10/2023%20state%20of%20academic%20libraries%20report.pdf [20]gonzales al, calarco jm, lynch t. technology problems and student achievement gaps: a validation extension of the technology maintenance construct. communication res. 2020;47(5):750-770. https://doi.org/10.1177/0093650218796366 [8][24]heidari-robinson s, heywood s. getting reorgs right. harv. bus. rev. 2016 nov. [4][12]lee j, frost g. manipulating data and moving forward: transitioning to a shared cataloging environment. collab. libr. 2017;9(3 article 3):215-217. https://digitalcommons.du.edu/cgi/viewcontent.cgi?article=1351&context=collaborativelibrarianship [14]the library of congress. 876-878 – item information-general information [internet]. washington (dc): the library of congress; 2008 mar 11. https://www.loc.gov/marc/holdings/hd876878.html [13]the library of congress. marc standards [internet]. washington (dc): the library of congress; 2023 jun 15. https://www.loc.gov/marc/ [10]the library of congress. marc 21 format for bibliographic data: 1999 edition update no. 1 (october 2000) through update no. 39 (december 2024) [internet]. washington (dc): the library of congress; [accessed 2024 dec 31]. https://www.loc.gov/marc/bibliographic/ [15]mccallum sh. marc: keystone for library automation. ieee annals of the hist. of comput. 2002;24(2): 34-49. https://doi.org/10.1109/mahc.2002.1010068 [26]mcdaniel g. ibm dictionary of computing. tenth ed. mcgraw-hill, inc.; 1994. [16]moehrle d. marc of the future. pnla q. 2012;76(4):81-87. [6]nicholson j, tokoro s. cloud hopping: one library’s experience migrating from one lsp to another. tech. serv. q. 2021;38(4):377–94. https://doi.org/10.1080/07317131.2021.1973796 [25]red hat. what is yaml? [internet]. red hat, inc.; 2023 may 3. https://www.redhat.com/en/topics/automation/what-is-yaml [7]rinna g, swierenga m. migration as a catalyst for organizational change in technical services. tech. serv. q. 2020;37(4):355–75. https://doi.org/10.1080/07317131.2020.1810439 [21]wilmoth ws. circulating laptops in a two-year academic library: a formative assessment. ga. libr. q. 2015;52(4):1-11. https://digitalcommons.kennesaw.edu/cgi/viewcontent.cgi?article=1846&context=glq about the author wilhelmina randtke has a background in law and technology. her past roles include legal research, technology oversight, and product manager for cloud based publishing software. she is currently head of libraries technologies and systems at the georgia southern university libraries overseeing in-building technology and online presences. georgia southern university libraries is the largest student printing lab on campus, providing approximately 1,500,000 prints to students each year. the libraries provide in-building desktop computers with specialized software for academic projects, a fleet of approximately 190 checkout laptops for students, and 3d printing and scanning services. appendix a: software spec posted to all employees on the university libraries intranet before refactoring software settings. item policies and behavior for each (goals; not yet implemented as of summer 2024) how item policies work (white = not yet set up across circulating locations / green = set up across circulating locations) item policy code how it works noncirculating special collections 001 noncirc_seccoll not requestable students: not allowed to check out grad students: not allowed to check out faculty: not allowed to check out community: not allowed to check out not available to gil express withdrawn 002 withdrawn not requestable students: not allowed to check out grad students: not allowed to check out faculty: not allowed to check out community: not allowed to check out not available to gil express non circulating 003 noncirc not requestable students: not allowed to check out grad students: not allowed to check out faculty: not allowed to check out community: not allowed to check out not available to gil express noncirculating arc materials 004 noncirc_arc requestable students: 2 hours, 2 renewals (shorten to close) grad students: 2 hours, 2 renewals (shorten to close) faculty: 2 hours, 2 renewals (shorten to close) community: 2 hours, 2 renewals (shorten to close) not available to gil express standard (4 weeks) 005 book requestable students: 4 week checkout, renewable twice grad students: one semester checkout, renewable twice faculty: one semester checkout, renewable twice community: 4 week checkout, renewable twice available to gil express dvd (1 week, renewable) 006a dvd requestable students: 1 week, 2 renewals grad students: 1 week, 2 renewals faculty: 1 week, 2 renewals community: 1 week, 2 renewals not available to gil express 1 week, no renewals, not requestable 006b 1wknorenew not requestable students: 1 week, no renewals grad students: 1 week, no renewals faculty: 1 week, no renewals community: not allowed to check out. not available to gil express game (1 week, requestable, no renewals) 006c game requestable students: 1 week, no renewals grad students: 1 week, no renewals faculty: 1 week, no renewals community: not allowed to check out not available to gil express digital cameras and camera equipment (4 day) 007 camera not requestable students: 4 days, no renewals grad students: 4 days, no renewals faculty: 4 days, no renewals community: not allowed to check out not available to gil express 1 day reserve (26 hours) 008 1dayres not requestable students: 26 hours, no renewals grad students: 26 hours, no renewals faculty: 26 hours, no renewals community: not allowed to check out. not available to gil express 8 hours, no renewal, no community 009a 8hrnorenew not requestable students: 8 hour checkout, no renewals (shorten to close) grad students: 8 hour checkout, no renewals (shorten to close) faculty: 8 hour checkout, no renewals (shorten to close) community: not allowed to checkout not available to gil express 8 hours, no renewal, yes community 009b 8hrcommunity not requestable students: 8 hour checkout, no renewals (shorten to close) grad students: 8 hour checkout, no renewals (shorten to close) faculty: 8 hour checkout, no renewals (shorten to close) community: 8 hour checkout, no renewals (shorten to close) not available to gil express 4 hour reserve 010 4hrres not requestable students: 4 hour checkout, no renewals (shorten to close) grad students: 4 hour checkout, no renewals (shorten to close) faculty: 4 hour checkout, no renewals (shorten to close) community: not allowed to check out. not available to gil express 2 hour reserve 011 2hrres not requestable students: 2 hour checkout, no renewals (shorten to close) grad students: 2 hour checkout, no renewals (shorten to close) faculty: 2 hour checkout, no renewals (shorten to close) community: not allowed to check out not available to gil express governor’s honors item 012 ghp_item not requestable (not using standard user groups, and instead only allows checkout by the ghp students and ghp employees groups, each of which gets it until the last day of ghp which is set in the terms of use for the item policy and manually changed each year.) governor’s honors program students: checkout until last day of the governor’s honors program georgia southern students: not allowed to check out. grad students: not allowed to check out faculty: not allowed to check out community: not allowed to check out not available to gil express graduate room study key 013 gradroom not requestable students: not allowed to check out grad students: 4 hour checkout (shorten to close) faculty: not allowed to check out community: not allowed to check out not available to gil express noncirculating special collections locations (location overrides item policy) locations included: henderson library special collections oversize_zsh (code=speccolov) special collections reading room_zsh (code=speccoll) lane library special collections minis room_ll (code=minis) item policy: all items in all special collections locations should have an item policy of “noncirculating special collections”. all items in these locations should be not checkoutable, no matter what the item policy is. additionally, for these locations, the location should also prevent requesting to where the arc robots never grab any item automatically. noncirculating locations (location overrides item policy) locations with no canonical item policy: because these are temporary / processing locations, the entire location is noncirculating regardless of item policy: henderson library available soon_zsh (code = availsoon ) bindery (code = bindery ) collection services_ zsh (code = crsprocess ) csdpay_zsh (code = crspay ) docs us map cases_zsh (code = docsmaps ) docs us reference_zsh (code = docsref ) foy media (code = foymedia ) arc media_ll (code = arcmediall ) locations with a specific item policy which should be used for that location: henderson library: herbarium (code = herbarium ) item policy: all items in “herbarium” should have an item policy of “non circulating”. microform area_zsh (code = microform1 ) item policy: all items in “microform area_zsh” should have an item policy of “non circulating”. microform periodicals/newspapers_zsh (code = microfilm2 ) item policy: all items in “microform periodicals/newspapers_zsh” should have an item policy of “non circulating”. ptrc-henderson (code = zsh_ptrc ) item policy: all items in “ptrc-henderson” should have an item policy of “non circulating”. reference 1st floor_zsh (code = reference ) item policy: all items in “reference 1st floor” should have an item policy of “non circulating”. reference atlas_zsh (code = refatlas ) item policy: all items in “reference atlas_zsh” should have an item policy of “non circulating”. withdrawn learning commons (code = wdlcomm ) item policy: all items in “withdrawn learning commons” should have an item policy of “withdrawn”. withdrawn _zsh (code = withdrawn ) item policy: all items in “withdrawn _zsh” should have an item policy of “withdrawn”. withdrawn: zsh surplus equipment (code = zshsurplus ) item policy: all items in “withdrawn: zsh surplus equipment” should have an item policy of “withdrawn”. lane library: periodicals_ll (code = periodical) item policy: all items in “periodicals_ll” should have an item policy of “non circulating”. reference_ll (code = ref) item policy: all items in “reference_ll” should have an item policy of “non circulating”. withdrawn _ll (code = wd) item policy: all items in “withdrawn _ll” should have an item policy of “withdrawn”. circulating locations: stacks locations with a mix of circulating and non circulating items (item policy determines behavior) circulating stacks locations (will have a mix of circulating and non circulating items): item policy: all item policies in the table are allowed in these locations, except for not “withdrawn”. locations included: henderson arc books_ll (code = arcbookll) arc books_zsh (code = arcbook) arc docs ga_zsh (code = arcgovga) arc docs us_zsh (code = arcgovus) authors’ lounge zsh (code = aut_lo_zsh) brain booth_zsh (code = bb_zsh) docs ga cd/dvd cases_zsh (code = docsgacd) docs ga oversize_zsh (code = docsgaovr) docs ga stacks_zsh (code = docsga) docs us cd/dvd cases_zsh (code = docscd) docs us stacks_zsh (code = docsstacks) foy oversize (code = foyover) foy stacks (code = foystacks) graphic novels_zsh (code = graphiczsh) lorimer reading room_zsh (code = lorimer) popular reading_zsh (code = popularzsh) stacks 3rd floor_zsh (code = stacks3) stacks 4th floor_zsh (code = stacks4) stacks oversize 3rd floor_zsh (code = stacksovr3) stacks oversize 4th floor_zsh (code = stacksovr4) lane: authors’ lounge – lane (code = aut_lo_ll) brain booth_ll (code = bb_ll) compact stacks_ll (code = compact) compact stacks oversize_ll (code = compov) juvenile_ll (code = juv) ll displays (code = display) microform area_ll (code = microf) stacks 2nd floor_ll (code = mainstacks) stacks oversize_ll (code = folio) graphic novels ll (code = graphicll) popular reading_ll (code = popreading) circulating locations: reserves and behind desks locations (item policy determines behavior) reserves and behind desks locations. item policies: all item policies in the table will work in these locations. never apply the following item policies to items behind a desk. if it can’t be checked out, and it’s behind a desk, then no one can use it. everything behind a desk should circulate to someone: “non circulating” “withdrawn” “noncirculating arc materials (2 hour in building)” “noncirculating special collections (noncirculating)” be very cautious applying the “standard (4 weeks)” item policy to items behind a desk. with a long checkout period, an item can be checked out for a long while, and be gone for the semester. locations included: henderson: foy reserves (code = foyreserve) zsh course reserves (code = courseres) zsh reserves (code = zsh_reserv) zsh equipment (code = equipment) zsh repairs (code = zshrepairs) lane: media reserves_ll (code = avreserve) reserve_ll (code = reserve) learning commons: learning commons desk (code = lcomm) subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – wattjournals: towards an economic and lightweight search tool alternative for libraries to help their students and researchers keep up-to-date mission editorial committee process and structure code4lib issue 12, 2010-12-21 wattjournals: towards an economic and lightweight search tool alternative for libraries to help their students and researchers keep up-to-date learn how heriot-watt university library’s wattjournals could be just the search tool your patrons need to efficiently find the content that your library subscribes to. built on top of a restful search api created by the jisc-sponsored journaltocs project, wattjournals is a toolkit for connecting fulltext articles to the people who need them. this article provides a technical overview of the system, showing how it uses citation data pulled from the journaltocs table of contents awareness service to provide access to just your library’s subscriptions. by santiago chumbe and roddy macleod introduction wattjournals [1] is an economic, simple-to-implement and easy-to-use search toolkit developed by the journaltocs project [2] for the library of heriot-watt university. its main purpose is to enable library patrons to find new articles as soon as they have been published, limited to those in journals to which the library subscribes. it thus ensures that the full-text of articles found will always be available. what makes this mashup interesting is that it is a real-low cost alternative to expensive library management and database search systems. it does not involve buying a new system, creating a new database or developing any software. the library concerned needs only to install a software toolkit that enables the combination of relevant library web applications and the reuse of data exposed by the api provided by the journaltocs project. although wattjournals is a customised version of journaltocs, designed specifically for heriot-watt university, it can also be easily implemented by other libraries. as libraries are faced with financial cutbacks, many may be forced to cancel licenses for expensive database search services. in this article, we argue that a low-cost option similar to wattjournals may be a real alternative for academic libraries. background journaltocs, hosted at heriot-watt university, was a project funded by jisc [3] to prototype a current awareness api by making use of article metadata that was already completely freely available through journal table of contents (toc) rss feeds. the initial aim of the project was to help institutional repository (ir) managers to ensure that their repository content was complete and kept up-to-date. it has become the largest, free and searchable collection of scholarly journal tables of contents in the world and now contains tocs for over 14,700 journals collected from more than 600 publishers. journaltocs has taken special care to include all the highest rated journals in their fields, with the caveat that it omits tocs from journals that do not provide toc rss feeds. in december 2009, journaltocs produced a lightweight api to handle restful search requests from external web applications. the simple rss standard is the core technology of this api. journaltocs uses rss to daily aggregate new content from the publishers’ web sites, and can also return search results in rss format to queries received from other web applications. the rss 1.0 feeds produced by the journaltocs api incorporates the dublin core [4], content [5] and prism [6] modules to normalize and extend the functionality of the original toc rss feeds. the base url for the journaltocs api is http://www.journaltocs.ac.uk/api it supports four types of calls: search for journals: http://www.journaltocs.ac.uk/api/journals search for articles: http://www.journaltocs.ac.uk/api/articles list your favourite journal tocs (requires user account): http://www.journaltocs.ac.uk/api/user/your@email.address customised searches (requires previous setup of the journaltocs toolkit): http://www.journaltocs.ac.uk/api/institution/institutionid for example, http://www.journaltocs.ac.uk/api/articles/”university of warwick” will return the latest articles published by authors from the university of warwick (notice the double quotes around university of warwick for phrase searching). http://www.journaltocs.ac.uk/api/journals/0267-5730?output=articles returns the latest toc of the journal with issn 0267-5730. figure 1. journaltocs data sources, database, api and end-users. figure 1 shows the database and the api created by the journaltocs project, as well as the potential end users for the api. the data is collected directly from the publishers using their own journal opml files, if available, and individual toc rss feeds where opml files are not available. current research information systems (cris) are among the potential end users of the api. institutions, particularly european universities, use cris systems to manage, analyse and access information on their research activity. for example, cris systems could use the api, alongside other data sources such as web of science, scopus and biomed central, to provide cataloguers with a source of bibliographic references. the api could also be used as a mechanism to alert institutional repository (ir) managers when new articles written by their academics were published in the toc rss feeds or when items submitted to irs are subsequently published in scholarly journals. the api is also able to search against the email address of a registered user and to return a list of journal tocs that a user has added to his previously selected mytocs folder. other use cases for the api were identified by librarians from universities and colleges. in particular, librarians showed interest in using the capabilities of the “customised searches” call offered by the api: http://www.journaltocs.ac.uk/api/institution/institutionid in this type of call, institutionid is a unique identifier assigned by journaltocs to an entity wanting to have customised search results. for example, techxtra [7] is a subject-based service that wanted to search tocs only for articles published in engineering, mathematics and computing. thus, techxtra registered the techxtra unique id with journaltocs to be able to filter search results to those appearing in the three subjects. for example: http://www.journaltocs.ac.uk/api/techxtra/geospatial%20technology/from/1/to/40 searches the journaltocs database for tocs that include the keywords geospatial and technology in their items; filters the items that are not part of the engineering, mathematics or computing subjects and finally; returns only the first 40 filtered items. the results are returned in rss format, and techxtra uses a simple web application to handle the results retuned by the journaltocs api and to display the final results on its web interface: http://www.techxtra.ac.uk/techtocs/index.html?query=geospatial technology the journaltocs developers realized that the api’s “customised searches” call could be used to limit search results to only those journals to which a library subscribed. this means that users from that institution will always be able to access the full-text of the articles returned in the search results. heriot-watt university library was the first library to implement this capability. the library required that the new search service be able to interoperate with its a-z journal lists, its openurl link resolvers and with off-campus access control mechanisms such as ezproxy. in addition, the library wanted to give its users the option to save searches for later use, and export citations to citation management software, such as endnote. technical implementation journaltocs toolkit is a php script running against a mysql table that contains the journal subscription list. the minimum information stored in this table is the issn numbers and the titles of the journals. the table is indexed by issn number. the library needs to set up this table on its own database server. because the journaltocs api institutional call provides access to filtered search results for an institution’s own journals, the first step is to register a unique institutional id with the journaltocs service. this is an easy process and is more fully described at http://www.journaltocs.ac.uk/docs/index.php?subaction=institution. secondly, the library needs to purchase a licence and set up the toolkit on its own server if it plans to search in more than 100 journals. the library then needs to either develop its own toolkit to query the journaltocs api or customise a toolkit sample provided by journaltocs. at heriot-watt university library, the toolkit was additionally configured to register the library openurl resolver and ezproxy. the toolkit also provides a basic html template to display search results on the web. these templates can be customised or replaced by the library staff. the toolkit is designed to act as a lightweight broker between journaltocs, the library software applications and the user web interface. its main function is to pass “messages” between those applications. below, we present a high-level outline of the key components of the toolkit algorithm. establishes a connection with the library database system. receives and handles user queries. sends http get search requests to the journaltocs api. the request could be as simple as executing the stream_get_contents() function in php. receives lists of matched journals, if any, and their corresponding query ids. queries the library system database to remove journals that are not subscriptions. sends back (http post) filtered lists of journals using two key-value pairs (i.e. queryid=123456 and filteredjournals=issn1, issn2, issn3, …). requests, receives and decodes final filtered search results. renders search results in html format. figure 2 illustrates in more detail the interactions that happen between the applications handled by the toolkit, in response to a search request received from the web. figure 2. journaltocs toolkit interactions. the journaltocs toolkit is identified with the letter t in figure 2. the toolkit is at the core of the implementation and handles all requests, communications and responses made by the rest of applications. the process starts when a web user enters a search query in the online form provided by the “html handler.” the query is immediately passed to the toolkit, which in turn sends the query to journaltocs via the journaltocs api (j). journaltocs executes the search and finds (1) issns for the journals contained in the search results (“found journals”) and (2) the article-level citation information returned by the search query. j caches the search results, assigns a unique id to the query and returns the “found journals” list to t, which in turn sends the list to the subscriptions filter (sf). sf can be a function of t or a separated web application. the sf is normally a simple piece of code provided by journaltocs in the toolkit sample code and customised by the library, with the support of journaltocs if necessary. in the case of wattjournals, the sf is simply a small api that tells whether a journal is in the subscriptions database or not. t uses sf to remove from the j “found journals” list the journals that are not subscriptions. then, t sends the reminder journals back to j. this list of “filtered journals” is used by j to remove the articles that correspond to journals not found in the returned list. the final filtered search results are sent to t, which uses the “html handler” to render the results on the user’s web browser. j destroys the cached search results. to speed up the exchange of data between j and t, a customised results output format was implemented by journaltocs for wattjournals. this format was a simple list of values separated by the | character; where each value contained the metadata for journals or articles and was interpreted as an array element by t. because the communication between j and t requires a fast in-between lookup mechanism, it was felt that it would be “overkill” to use xml (rss) to receive and parse results from j for every lookup made by t. however, other options, such as json, are available. the full implementation of a service similar to wattjournals should take a few weeks. wattjournals is not a new system requiring continuous maintenance, but a combination of existing web applications or apis, all converging together to produce a “new” service at a rapid pace. these combinations, or mashups, are becoming common nowadays, and include apis, rss, folksonomies, and social web tools, giving it managers a new way to approach hard problems with surprisingly effective results. figure 3. journaltocs implementation for wattjournals. user interface from the start, we agreed that the user interface and key features for wattjournals needed to be as simple as possible. we assumed that our typical user was an undergraduate student familiar with rapid ways of looking for information on the web using search engines such as google. when unveiling wattjournals in the library blog [8], gill mcdonald, acting university librarian, said “this new service will be particularly useful to students and researchers who want to get some up-to-date articles and research papers quickly and easily. we know that some students find the whole process of navigating several databases and e-journal sites quite confusing at first, and to do a search and then find that the library doesn’t have a subscription to the journal you’ve found is really frustrating. wattjournals can be a gentle introduction to searching, with the added benefit that all the articles found will be available – no more dead ends!”. figure 4. wattjournal search results screen. the end product is a customised journal search service with a simple, easy-to-use local interface. it searches only journals to which a library (in this particular case, heriot-watt university library) subscribes. wattjournals only provides access to the content of those heriot-watt university subscribed journals which have table of contents (toc) rss feeds. wattjournals also enables searches to be saved for later use, and the export of citations to endnote. initial results and user feedback between 1st and 11th june 2010, the journaltocs project team conducted an online user survey for students and academic staff from heriot-watt university to help determine the usefulness of wattjournals and to receive user feedback. all the participants in the survey found wattjournals to be useful. in particular, students liked its direct access to the full-text. one student summarised: “it’s very useful to be able to search only articles to which i can gain access, so that i don’t have to scour through pages of articles which i can’t read.” although wattjournals, by default, searches recently published articles, it also offers an option for searching past works. based on the survey results, we are now investigating the possibility of merging recent with past issues into a single search target for wattjournals, but at the same time keeping its current awareness feature. aside from the survey, we also received interesting comments and questions by email. behind the scenes, there are several improvements that we would like to make. currently, wattjournals only searches 65% of the journals subscribed by heriot-watt university because not all the subscribed journals produce toc rss feeds and, therefore, are not contained in the journaltocs database. it is becoming clear, after the user survey and initial feedback received by email, that the service would be improved if wattjournals could somehow include these omitted journals. a possible alternative that the team is investigating is adding commercial external apis or other web applications to the mashup. conclusions since its pre-launch in june 2010, wattjournals has quickly proven to be an innovative and economic alternative to large and sometimes complex e-journal search systems. the speed and ease with which wattjournals was built impressed the heriot-watt university library staff and got them interested in the potential of building other mashups for the library. since the pre-launch, interest from academic libraries wanting to implement similar services on their own web sites has been overwhelming. besides enquiring on costs, librarians asked us for specific information on the range of content they could obtain from the journaltocs database via a wattjournals-type application. we stressed the fact that wattjournals-type applications are not intended to replace the large database systems to which libraries normally subscribe. they have also been made aware that, currently, wattjournals cannot cover all the journals to which the library subscribes, as it is only able to search those which provide an rss feed of their tables of contents. nevertheless, the degree of interest was very high and confirmed to us that the benefits of wattjournals-type services are relevant for both librarians and researchers. publishers would also reap benefits from having their new papers more readily available for end users. references [1] wattjournals. university library. heriot-watt university. http://www.hw.ac.uk/library/wattjournals [2] journaltocs. institute for computer based learning. heriot-watt university. http://www.journaltocs.ac.uk [3] jisc. http://www.jisc.ac.uk [4] rdf site summary 1.0 modules: dublin core. http://purl.org/rss/1.0/modules/dc/ [5] rdf site summary 1.0 modules: content. http://purl.org/rss/1.0/modules/content/ [6] rdf site summary 1.0 modules: prism. http://purl.org/rss/1.0/modules/prism/ [7] techxtra. heriot-watt university. http://www.techxtra.ac.uk/ [8] “wattjournals has arrived!” spineless. university library. heriot-watt university. http://hwlibrary.wordpress.com/2010/06/02/wattjournals-has-arrived about the authors dr. santiago chumbe works at the school of mathematical and computer sciences, heriot-watt university, edinburgh, uk. he is the principal investigator of journaltocs and the manager of techxtra services. he has recently been the project manager of the wattnames and journaltocs-api projects funded by jisc. he is the main editor of the journaltocs blog and has more than 30 publications and conference presentations. he is a member of the europeana network and the eurocris organisation. roddy macleod is a retired information professional. he previously worked at heriot-watt university library, the university of malawi library, and the university of botswana library. he has published widely in various scholarly and professional journals and was editor of the internet resources newsletter. he was manager of various jisc funded projects such as eevl and perx, and contributed to other projects such as tictocs and gold dust. he was the winner of the information world review information professional of the year 2000. he now writes a personal blog at http://roddymacleod.wordpress.com/. subscribe to comments: for this article | for all articles one response to "wattjournals: towards an economic and lightweight search tool alternative for libraries to help their students and researchers keep up-to-date" please leave a response below: aberdeen university library | roddy macleod's blog, 2013-09-16 […] journal is open access.  but wait – when i clicked on the details tab, there’s a view full text link, and it takes you to the full […] leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – how to provide live library information via sms using twilio mission editorial committee process and structure code4lib issue 14, 2011-07-25 how to provide live library information via sms using twilio paul smith’s college provides library hours and workstation availability using sms text messages. the service was implemented using an easy and affordable web-based api for sms sending and receiving, from twilio.com. a new class of ‘cloud-based‘ sms vendors make simple sms-based services efficient and cost-effective to implement, and have many possible applications in the library environment. a simple php example is provided which supplies workstation availability over sms based on a database of computer availability from a previous code4lib journal article. by mike beccaria the problem paul smith’s college is a very small (~1,000 fte) 4-year private college nestled in the adirondack park in northern new york. this past year we were fortunate to have record breaking enrollment numbers, but with the added number of students there was added demand. subjectively, the library staff observed that computers were being used a lot more often and in greater numbers than previous years. small surveys and feedback gave indications that students were having trouble finding computers at peak usage times and our usage data largely confirmed these findings. to start to tackle this problem during this budget cycle, we developed a live computer availability map on our website using kim griggs’ code4lib article “how to build a computer availability map” as a template (griggs, 2010) with long term plans to add a few more computer terminals to the library, as well as an information kiosk with the computer availability map on it. we decided to supplement the computer availability map with a text messaging service that would allow students to receive computer availability information on their mobile devices. the solution: sms cloud services sms, or simple message service, is the means by which common text messages are sent via mobile devices. for our text message solution, we decided to use a 3rd party provider that bridges the gap between web services and telecom. we had several choices. twilio, tropo, and smsified, as well as others who provide cloud-based web service api’s that enable users to build applications to leverage voice and sms communications. these companies specifically provide customers with a phone number and a web based api that allows developers to control telecommunication programmatically. for example, using these services makes it possible to create an automated text reply service so that if someone sends a text message or phone call to a phone number that i lease from twilio (the provider we used), i can programmatically reply to the text with customized information. before cloud-based service providers like twilio and tropo arrived, developing applications that use sms and voice communications were both cost and technologically prohibitive, requiring users to learn a telecom programming language or set up their own pbx service. in contrast, cloud-based vendors like twilio and tropo use common and easy to program web apis.  both services charge users a small monthly fee for a dedicated phone number and an affordable pay-per-use pricing model that most people could pay for using the money they spent on a coffee and bagel for breakfast. twilio, for example, currently charges $1/month for a phone number and between 1 and 2 cents per text message or minute of phone voice time (http://www.twilio.com/pricing/). what we did to complement our computer availability map and help students find available computers via text messaging, we created library sms services using twilio. when researching companies for a solution, we found that many of them were about the same price and offered similar functionality, and we ultimately chose to use twilio because that was the company that we signed up for and tested first. students are now able to send a text message to our dedicated sms phone number that we lease from twilio and receive a text message back with useful information (see figure 1). we expanded our services beyond computer availability and provided the means to get other kinds of information. students can send “pc” for computer availability, “laptop” for lending laptop availability and “hours” for the library hours. modeled after google’s sms search application, if they send a blank message or a message that isn’t a command, they will receive a text message with instructions on how to use the service. figure 1: using text messaging to get computer availability in the paul smith’s college library. how to make your own computer availability sms service to create the service, we started with the database structure that we had in place for our computer availability map (see griggs, 2010). once we had the back-end architecture in place and a database that had live computer availability data, we were ready to create a php script to leverage twilio services. when our twilio phone number receives a text message, twilio sends an http requests with post variables about the text message or call information (caller id, geography, the body of the text message, etc.) to our server (see twilio’s api documentation http://www.twilio.com/docs/index). when our server receives the http request, we process the variables and, depending on the data found in them, send an xml response back to twilio with our text message response data. while twilio also provides many other voice and text services including text-to-speech, voicemail transcriptions, automated calling, teleconferencing, and business platform phone services, we are only leveraging the sms text services for our solution. down the road we may wish to add functionality to our services, but for now we are keeping it simple. our solution is straightforward and easy. we placed a single php file on our server that is designed to process our text message requests from twilio. when a request comes in, we determine what we want to send back to the patron. for example, if “hours” or “help” is texted to us, we simply output xml that twilio can read to send a response back. here is a slightly modified portion of our file that processes the “hours” text message: <?php if (strtolower(substr($_request&#91;'body'&#93;,0,4)) == 'hour' ){ ?> <response> <sms><?php echo "psc library hours:\r\nfall and spring semester:\r\nm-th 7:30am-10pm\r\nf 7:30am-6pm\r\nsa 10am-6pm\r\nsun 1pm-10pm\r\nsummer:\r\nm-th 8am-7pm\r\nf 8am-4:30pm\r\nsa 10am-3pm"?> </sms> </response> the if statement checks if the first four characters of the text body equals “hour” in upper or lower case. if that is true, it outputs an xml response that looks like this: <response> <sms>psc library hours: fall and spring semester: m-th 7:30am-10pm f 7:30am-6pm sa 10am-6pm sun 1pm-10pm summer: m-th 8am-7pm f 8am-4:30pm sa 10am-3pm</sms> </response> the xml used in this example is a standard developed by twilio (what they call “twiml”) and it tells twilio to send back a text message response to the patron. the “<response>” xml element is the root element for twiml. under the root “<response>” element, twilio then accepts “verb” and “noun” elements that allow the developer to let twilio know what to do. as is the case in our example, the “<sms>” verb element lets twilio know to send an sms response back to the phone that sent the message. other verbs (mostly used in voice based solutions) include “<say>” which reads text to the caller, “<play>” which plays an audio file, “<record>” which records a call, “<gather>” which is used to get the digits a caller presses, and “<dial>” which calls a number for a conference call. notice that php was used to echo the response for the library hours. this was done so that carriage returns could be added to the text message (in the code as “\r\n” which stands for carriage return newline in php). this example solution is very similar for our two other services (computer availability and laptop availability) except in those solutions we use php to gather computer and laptop availability data and send that information to patron. pasted below is a modified version of our sms_services.php script which determines what the body of the incoming text message from the patron says, and then sends back pc availability, hours, or help depending on the body of the text message sent by the patron. you will have to alter the code for your particular solution depending on how you store computer availability, but the key is to send out twiml (twilio xml) to let twilio know what to send back to the patron. <?php header("content-type: text/xml"); echo "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n"; #if the student texts "pc", get the pc availability of the library computers and respond back if (strtolower(substr($_request['body'],0,2)) == 'pc'){ #database user credentials $user = ""; $password = ""; $database = ""; $host = ""; #connect to the database $db = mysql_connect($host, $user, $password) or die("can't connect to database."); @mysql_select_db($database) or die("unable to select database"); #create the queries to gather computer availability both upstairs and downstairs. $total_pc_upstairs_results = mysql_query("select * from compstatus where (location_group = 'upstairs') and (computer_type='pc') and (status = '1' or status = '0') "); $avail_pc_upstairs_results = mysql_query("select * from compstatus where status='0' and computer_type='pc' and location_group = 'upstairs'"); $total_pc_downstairs_results = mysql_query("select * from compstatus where (location_group = 'downstairs') and (computer_type='pc') and (status = '1' or status = '0') "); $avail_pc_downstairs_results = mysql_query("select * from compstatus where status='0' and computer_type='pc' and location_group = 'downstairs'"); #run the queries and create the output strings $upstairs_pcs = mysql_num_rows($avail_pc_upstairs_results) . ' of ' . mysql_num_rows($total_pc_upstairs_results) . " pc's available upstairs."; $downstairs_pcs = mysql_num_rows($avail_pc_downstairs_results) . ' of ' . mysql_num_rows($total_pc_downstairs_results) . " pc's available downstairs."; mysql_close($db); #send the response ?> <response> <sms><?php echo "psc library computers\r\n" . $downstairs_pcs . " \r\n" . $upstairs_pcs; ?></sms> </response> <?php #if the student texts "hour" send back library hours } elseif (strtolower(substr($_request&#91;'body'&#93;,0,4)) == 'hour' ){ ?> <response> <sms><?php echo "psc library hours:\r\nfall and spring semester:\r\nm-th 7:30am-10pm\r\nf 7:30am-6pm\r\nsa 10am-6pm\r\nsun 1pm-10pm\r\nsummer:\r\nm-th 8am-7pm\r\nf 8am-4:30pm\r\nsa 10am-3pm"?></sms> </response> <?php } else { #if it doesn't recognize a message, send directions. ?> <response> <sms><?php echo "psc text services:\r\ntext \"pc\" for library pc availability\r\n\"laptop\" for library laptop availability\r\n\"hours\" for our hours\r\n";?> </sms> </response> <?php }?> early findings overall we have been very happy with the use and acceptance of this service. after creating the service and doing some simple tests, we decided to give it a soft opening by posting the service to our library facebook page (see http://www.facebook.com/psclibrary) and library news page. once the bugs and tweaks were ironed out, we followed up with a campus wide email. our soft opening on facebook ran for about 25 days between march 3rd and march 27th. during that time we received 31 text requests. on march 28th i sent out a mass email to the campus (see figure 2). from march 28th through the end of the semester, may 15th (48 days) we received 496 text requests, an average of 10 requests per day. for a college of our size (~1,000 fte) we were very pleased with these numbers. when we started the service, twilio was charging a very affordable $0.02 per text message each way. since the inception of the service in late february we spent a meager $25 this semester. in may they dropped their sms service to $0.01 each so the service is now less expensive (see twilio pricing at http://www.twilio.com/pricing/). figure 2: email message to campus community future possibilities with the advent of telecommunication services like twilio, there are many innovative service opportunities that libraries can create easily and inexpensively. i consider our solution to be a drop in the bucket compared to the possibilities that exist. libraries can create customized reference chat services or create an sms search application that searches the library catalog or voice/text applications that allow patrons to renew books or pay fines. at paul smith’s college, we will continue to monitor usage and demand to determine if future applications using mobile sms and voice services will be beneficial to our patrons and whether or not we have the resources to create those applications. conclusion it has been said that “if we don’t care for our customers, someone else will.” libraries need to continue to reach out to patrons in places where they gather, whether in be in physical or virtual spaces, or our patron’s attention and loyalty will diminish and the valuable resources we provide will be overshadowed by others who have taken to time to care for our customers better than us. libraries have a history of changing with the technological tide to remain relevant to our patrons and using 3rd party telecommunications services like twilio aren’t necessarily game changers in our collection of services, but they’re a piece in the puzzle towards showing our patrons that we care. references griggs, k. (2010 dec 21). how to build a computer availability map. code4lib journal [internet]. [cited 2011 may 24]; 12. available from http://journal.code4lib.org/articles/4067 additional links google sms applications: http://www.google.com/mobile/sms/ mosio: http://www.mosio.com twilio: http://www.twilio.com/ tropo: http://www.tropo.com/ smsified: http://www.smsified.com/ about the author mike beccaria is the systems librarian and head of digital initiatives at paul smith’s college, a small baccalaureate institution located in the adirondack park in northern new york. he can be contacted at mbeccaria at paulsmiths edu. subscribe to comments: for this article | for all articles one response to "how to provide live library information via sms using twilio" please leave a response below: at&t’s smart sms feature | bibliographic wilderness, 2012-11-30 […] out-of-the-box ils/lms systems probably have this feature;  but services like twilio (there was a code4lib journal article on twilio) would, i think, make it feasible and affordable for library developers to hack one together on […] leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – the format registry problem mission editorial committee process and structure code4lib issue 19, 2013-01-15 the format registry problem file format identification is an important issue in digital preservation. several noteworthy attempts, including pronom, gdfr, and udfr, have been made at creating a comprehensive repository of format information. the sheer amount of information to cover and the constant introduction of new formats and format versions has limited their success. alternative approaches, such as linked data and offering limited per-format information with identifiers that can be used elsewhere, may lead to greater success. by gary mcgath introduction digital documents come in thousands of formats, with tens of thousands of subvarieties. this makes the life of the digital preservationist difficult. is a valid file one which can easily be opened and will stay that way in the reasonable future, or are special measures needed to ensure its readability? exactly what software is needed to open it? just knowing the format’s generic name isn’t enough. if you have a “microsoft word” file, that doesn’t tell you whether it’s a version from the early eighties, a recent document in microsoft’s proprietary format, or an office open xml document. the three have practically nothing in common but the name. before format registries in the earliest days of computers, each system had its own formats; you couldn’t even copy character codes from one computer to another. as they became more widespread and people started thinking of them as document processing systems rather than just huge calculators, formats started to become an issue. the computer magazines of the seventies were full of ads by startups offering software for all kinds of exciting purposes. these startups might not still be around the next month. every application had its own document format, so preserving your work when you upgraded your s-100 machine meant making a printed copy. as the number of incompatible operating systems declined, long-term retention of files became a possibility. this made it important to recognize a file’s format. the unix file command relied (and still does) on a “magic number” approach to file identification, with the value of characteristic bytes indicating what format a file most likely is. its database isn’t human-readable, but it could be considered one of the first attempts at a comprehensive format repository. libraries and archives have been particularly aware of the need to keep information about formats after the software which creates them goes out of use. an paper on the subject, “risk management of digital information: a file format investigation,”[1] raised the issue in 2000: the most difficult aspect of this project was the acquisition of complete and reliable file format specifications. throughout the project, format-specific information was difficult to acquire from a single source. the authors go on to suggest: if we measured the risk associated with public domain archives on the internet, we would assess all these sites as high-risk operations. sites such as wotsit’s represent the public service efforts of individuals. they lack any vision or plan to sustain the information. this limitation, combined with the unreliable nature of the information contained within these sites, make it unlikely that these sites will contribute meaningfully to digital preservation efforts. there is a pressing need to establish reliable, sustained repositories of file format specifications, documentation, and related software. we recommend the establishment of such depositories for format-specific materials related to migration as a preservation strategy. the quest for format identification has led to a two-pronged strategy, with format registries and format identification software. the efforts to create software that recognizes a file’s format have produced some useful software, such as unix’s file, the uk national archives’ droid, jhove and fits from harvard, jhove2 from the california digital library and others, ffident from a group of open source developers, and so on. these restrict the domain of the problem they address in one way or another. jhove looks at a small number of formats in considerable detail. droid and file can identify a broad range of formats but look only at their characteristic patterns or “signatures.” to be useful, registries need to provide a significant amount of information about many different formats, and this has proven to be a difficult task. pronom the first large-scale attempt, and still the most successful, at “the establishment of such repositories” was pronom (http://www.nationalarchives.gov.uk/pronom/), from the uk national archives. part of the above quotation appeared at the start of jeffrey darlington’s 2003 article explaining the project.[2] by its own description, “pronom is an on-line information system about data file formats and their supporting software products. originally developed to support the accession and long-term preservation of electronic records held by the national archives, pronom is now being made available as a resource for anyone requiring access to this type of information.” droid relies on the pronom database. pronom has lots of useful information. if you search for the extension “tif”, you get tagged image file format, as you’d expect, as well as nine specialized formats that are built on the tiff standard. click on “tagged image file format” and you get several tabs worth of information. figure 1. pronom entry for tagged image file format. look deeper, though, and you’ll find the information is spotty. you can look up software by company; enter “adobe” and you’ll get seven pages of information. however, the most recent version of acrobat listed is 5.0, which dates from 2001. you can look for migration paths for “tagged image file format” (you won’t find any if you look for “tiff”). you get a few options for converting tiff to png, leaving a vast number of possibilities unmentioned. the task of keeping track of the ongoing development of file formats is a huge one, and it isn’t surprising that even a large organization such as the national archives has trouble keeping up. gdfr since maintaining format information is such a huge task, an obvious remedy is to get multiple institutions in on the job. this led to the proposal for the global digital format registry (http://www.gdfr.info/), intended as a “distributed and replicated registry of format information populated and vetted by experts and enthusiasts world-wide.”[3] the harvard university library (hul) received a grant from the andrew w. mellon foundation for a two year project leading to the deployment of gdfr. the development work was done in conjunction with oclc. work began with meetings sponsored by the dlf in 2003, which established an ad hoc group to work on a plan. in 2005 stephen abrams and dale flecker of hul published a paper formally proposing gdfr.[4] the most significant architectural aspect of the gdfr is its distributed nature. the aim of the project is not to build a single centralized registry, but rather to define a common network protocol by which multiple independent, but cooperating, registries can communicate with each other and synchronize their holdings of format representation information. the design was complex in both its data model and its networking features, and the xml database was unwieldy to work with. developers at harvard and oclc collaborated on the project, but personnel and communication issues led to problems. the project ended in 2008 or so without producing either usable software or a practical repository. udfr the failure of gdfr was followed by an effort to pick up the pieces and put together the best features of gdfr and pronom. this project was called the unified digital format registry (udfr, http://udfr.org/). the california digital library took the lead, with funding from the library of congress. the distributed architecture was dropped, but open contribution would be encouraged, with minimal barriers to participation. the project had ongoing problems, as described in the udfr final report.[5] the schedule was extremely tight and there were project management issues. nonetheless, it reached completion and there is a working site. the problem is that it’s a dead one as far as building information is concerned. the final report says that “the initial level of community involvement was quite high,” but this wasn’t sustained after the final deployment of the software. there was a general sense of exhaustion at the end, leaving no one to rally the community to continue updating it. wikipedia? if the problem is too big for any institution, what about the world’s best-known source of crowd-sourced information, wikipedia? it has many articles on file formats. much of the information there is in a semi-structured form, in “infoboxes.” for example, look at the page on “jpeg” and you’ll find a box with items labelled “filename extension,” “internet media type,” “type code,” “uniform type identifier,” “magic number,” “developed by,” “type of format,” and “standard(s).” and, of course, a picture of a cat. better yet, the information in these boxes is available for machine queries, using the linked data approach. the dbpedia project (http://dbpedia.org/) makes the information from wikipedia infoboxes as rdf data accessible with the sparql query language. the downside is that wikipedia is directed at the general reader and mostly lacks the detailed technical information that’s needed for preservation efforts. it often has links to that kind of information, though, including specifications. “just solve the problem” as i’m writing this in november 2012, another kind of crowdsourcing effort is underway. the “just solve the problem” project (http://justsolve.archiveteam.org/wiki/) is bringing together the efforts of many contributors to gather format information. according to its statement of project: over 30 days (and left to run afterwards), this wiki will provide a central source for information on all manner of file formats, self-encapsulated information sets that suffer (over time) from falling into obscurity, losing documentation, and otherwise fading while still containing many works out in the world that might need recovery. by providing an institution-neutral, public-domain, easy to navigate site containing this information, the “problem” can be addressed both by users of the wiki and the many, many related attempts to achieve this goal, all of which can pull this wiki’s information back under their roof.[6] the project has already put together a huge amount of information. it’s heavy on contributions relative to oversight, so the quality, completeness, and reliability vary, but in sheer breadth of information it could be a valuable resource. all contributions are declared public domain under a creative commons cc0 license, so they can be copied and reworked freely. unification by linked data dbpedia isn’t the only source of format information organized as rdf and delivered through sparql. udfr and pronom have endpoints. this raises an interesting possibility. the idea of linked data is to unify diverse data sources on a machine-processable level, just as the world wide web does on a human-readable level. could a single reader query multiple endpoints and provide useful aggregated information? a proposal by bill roberts for the open planets foundation suggests steps in this direction. it stresses the need for “a distributed approach that allows collaboration between different organisations”[7] and suggests linked data as a component of this approach. to try the experiment, i created a java application that would use the jena open source libraries to query different sources of linked data on formats. the sources used were dbpedia, pronom, and udfr. the project is available on github (https://github.com/gmcgath/format-reg-browser). a quick review of the rdf essentials may be appropriate here. rdf data is stored in “triples,” which consists of a subject, a predicate, and an object. a subject is something: a concept, an object, a collection, etc. the predicate identifies some relationship that the subject could have to something else. the object identifies a “something else” for which the relationship holds. for instance, (“united states,” “has state,” “new hampshire”) could be an rdf triple identifying a geographical fact. predicates are the glue of rdf. for a particular information set, the set of predicates has to be known and restricted, so that the user knows what queries are possible. this is done by assigning specific names to predicates and grouping them in a namespace associated with a uri. in the triples, a prefix is associated with the uri to avoid being verbose. for example, udfr has a predicate namespace associated with the uri . in a particular application, any prefix can stand for it, but well-known namespaces have commonly used prefixes, in this case “udfrs.” the value of the subject of a triple is generally a uri identifying a resource. the object can be a data value like a string, or it can be a uri which identifies another resource, possibly at a different site. this is where the “linked” aspect of linked data comes from. an rdf data set can have predicates from one or more namespaces. the aggregate of these is its “ontology.” there is no common ontology for file format information. udfr and pronom each have their own. dbpedia has a number of specialized ontologies for particular topics, but so far none for file formats; it does have a certain amount of informal agreement. this meant that my experimental application had to create different queries for each registry. the udfr ontology is complex and often requires two-level queries. fortunately, richard may had already done some work in the area[8], which was valuable in devising my own queries. dbpedia’s is relatively flat, not as rich but easier for a novice sparql coder to work with. each has types of information that don’t have exact information in the other. dbpedia has “extension” for many formats; udfr has “external signature,” which may or may not be the same thing. i assumed in my code that it is. the application puts up a form that lets the user fill in some fields. these match as case-independent substrings of field values. figure 2. registry browser input form. a search is done on the registries selected by the configuration file, and the results are grouped in a single window. figure 3. sample results of a registry search. the application could easily be expanded in its search fields and the information it returns. finding queries that are equivalent between repositories is more difficult, as is comparing the results. udfr has a lot of different entries with the name “tagged image file format,” which are differentiated by other fields. dbpedia has much less granularity in its information. could a tool of this type, sufficiently expanded, be a serious help in spanning multiple format registries? that isn’t clear as yet. the inconsistencies in dbpedia make it difficult to search and extract high-quality information. a deep understanding of udfr’s complex ontology would be helpful in finding equivalents to other repositories. conclusions the format registry problem is an ongoing one. new formats develop and old ones change both officially and unofficially. no institution has been capable of amassing a complete registry of format information. linked data may provide a way to integrate multiple sources of format information. its potential is currently limited by the lack of a generally accepted ontology. if an institution with sufficient visibility devises an ontology which is simple enough, or has a simple enough subset, to be widely adopted, it could make these efforts more practical. andy jackson has suggested a minimalist solution: “[m]y current thinking on the data model is that the format registry design should only be concerned with minting identifiers for formats, and collecting the minimal representation information require to support this. i think that if we can get this right, then we can use those identifiers to describe the other entities, like software tools and digital object characteristics, but without forcing them all into the same system.”[9] in this view there could still be a central registry, but its role would be something like bowker’s in issuing isbns. no one is going to “just solve the problem” once and for all. ongoing efforts by many different people working independently will be necessary to keep up with the growing variety of formats. references [1] gregory w. lawrence, william r. kehoe, oya y. rieger, william h. walters, anne r. kenney: “risk management of digital information: a file format investigation,” council on library and information resources, washington, dc, june 2000. http://www.clir.org/pubs/abstract//reports/pub93 [2] jeffrey darlington, “pronom—a practical online compendium of file formats,” rlg diginews, volume 7, no. 5, october 15, 2003. [3] “global digital format registry (gdfr) information site.” available at http://www.gdfr.info/. [4] stephen abrams, dale flecker, “a proposal for a global digital format repository,” harvard university library, september 29, 2005. [5] “unified digital format registry (udfr) final report,” uc curation center, california digital library, july 2, 2012. [6] jason scott, “statement of project,” solve the file format problem, october 28, 2012. (http://fileformats.archiveteam.org/wiki/statement_of_project) [7] bill roberts, “a new registry for digital preservation,” open planets foundation, september 2010. (http://www.openplanetsfoundation.org/node/596) [8] richard may, “file identification using fido and the udfr registry,” open planets foundation blog, september 10, 2012. (http://www.openplanetsfoundation.org/blogs/2012-09-10-file-identification-using-fido-and-udfr-registry) [9] andy jackson, “breaking down the format registry,” open planets foundation blog, december 8, 2012. (http://www.openplanetsfoundation.org/blogs/2010-12-08-breaking-down-format-registry) about the authors gary mcgath [email: developer@mcgath.com] is an independent professional software developer. previously he was a digital library software engineer with the harvard library. subscribe to comments: for this article | for all articles 2 responses to "the format registry problem" please leave a response below: richard wright, 2013-01-17 what a shame that the linked data version of pronom wasn’t mentioned: http://labs.nationalarchives.gov.uk/wordpress/index.php/2011/01/linked-data-and-pronom/ i know the author knows about this work, because he has left comments on the above-mentioned page ! edward m. corrado, 2013-04-06 the author might have mentioned pronom’s linked data abilities in a bit more detail, but he does mention it. in the “unification by linked data” section of the paper, gary mcgath discusses a java application that he wrote that queries “different sources of linked data on formats. the sources used were dbpedia, pronom, and udfr.” leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – the automagic of the lii’s ecfr mission editorial committee process and structure code4lib issue 39, 2018-02-05 the automagic of the lii’s ecfr the legal information institute (lii) began providing access to federal legal materials in 1992. this article discusses their work expanding and improving free public access to federal legal resources in the u.s., particularly developing their ecfr product for the code of federal regulations, and plans to integrate docketwrench. by charlotte schneider and sylvia kwakye background cornell law school’s legal information institute (lii) has been publishing law online since 1992. the mission of the lii was, and remains, to publish law online for free, to create materials that help people understand the law, and to explore new technologies that make it easier for people to find the law. the united states code (usc) was the first full collection that the lii made available. the lii’s usc was innovative in that they used hyperlinks to allow researchers to easily navigate between sections referenced in the text. the lii also made the table of popular names, a supplement to the usc, available so that researchers could navigate from the popular name of the statute to the statutory section within the usc. in the years following the availability of the usc, the lii expanded their offerings to include united states supreme court opinions, the united states constitution, secondary content like wex, their legal encyclopedia and dictionary, and more. these collections, in particular, all work together. some collections, like the supreme court opinions, are fully automated so that when new content is available from the supreme court, that information is automatically processed and made available on the lii site. hyperlinks to the lii’s usc and constitution collections are also automatically created when the text is processed. the usc collection went further by incorporating another collection that is independent of the lii: they programmed a feature to automatically create hyperlinks for the statutes at large and public law citations in a statute’s source credits that linked to the full-text of the public law from gpo’s online repository, thomas, the fdsys predecessor. as a result of the lii’s success with the usc, in february 2010, the gpo announced a partnership with the lii to convert the code of federal regulations into xml, which enabled them to publish the first html version of the code of federal regulations, in the same fashion as the usc with respect to its interoperability with other collections. the code of federal regulations the code of federal regulations (cfr) presents the official language of current federal regulations in force. the authority for rulemaking can come from several different directives. one such directive is legislation, in a statutory grant of authority. when agency rulemaking begins, the agencies publish a notice of proposed rulemaking (nprm), with the proposed language of the rule. this proposed rule publication opens up the period for public comments to allow interested persons or organizations, and the general public, to submit comments for informing the rulemaking process. when the agency publishes the final rule, it also publishes a summary of the comments received. the nprm and the final rule are published in the federal register, the daily publication of agency rulemaking activities. the final rules, as published in the federal register, are compiled and organized by subject into the cfr by the office of the federal register (ofr). the cfr is updated only once a year, which can present challenges when doing regulatory research. companion publications and finding aids to the cfr include the parallel table of authorities (ptoa) and the list of sections affected (lsa). the ptoa presents a list of the statutory–or otherwise–grants of authority for rulemaking. this allows researchers to navigate between the cfr and the usc, making connections between statutes granting rulemaking authority and the regulations promulgated in furtherance of those grants. likewise, researchers reading a regulation can easily find out what legislation authorized that rule. the lsa is a compilation of all of the sections that are affected by agency rulemaking activities during the year. each nprm and final rule published in the federal register notes which cfr sections are affected by the rulemaking. the lsa is a compilation of all of the sections that are affected by agency rulemaking activities during the year. each nprm and final rule published in the federal register notes which cfr sections are affected by the rulemaking. researching regulations typically begins with the most recent publication of the cfr and finding the appropriate part or section. to see if that section has been updated since the cfr was published, researchers check the most recent monthly compilation of the lsa. since the lsa is only a monthly publication, researchers still need to check the most recent issues of the federal register. each issue’s table of contents contains a short list of cfr sections affected within that issue. if the section is one that has been affected, researchers then need to use individual, referenced issues of the federal register to find the new final rule, or the nprm that will affect that rule. the cfr is available in print in most law libraries as a collection of over 200 volumes. the cfr is also available online from the gpo, as pdf documents. the ptoa and the lsa are also available in print and online from the gpo. lii’s cfr in february 2010, the gpo announced a pilot project, partnering with the cornell law library and the lii to create “a high-value version of the cfr.”[1] the lii had experience from their early work with the usc in processing government typesetting into xml format, and lent that expertise to the gpo. this gave way to the gpo making available to the public for download the bulk xml source data for the cfr. from that bulk xml source data, the lii built an early, full-text version of their cfr.[2] given their experience with similar government information, the lii applied some of the technologies employed in their other collections to this cfr collection, such as their formatting, with nested and indented paragraphs, hyperlinked cross-references, the breadcrumb trail.[3] other features that enhance the regulatory research experience include: table of contents and individual cfr sections each having a dedicated page and url; each paragraph’s alpha-numeric prefix is formatted in bold so that they stand out when looking for a specific paragraph; federal register citations in the source credits are hyperlinked to the correct issue and page where the final rule is published in gpo’s federal register collection; each page features a breadcrumb trail for context and navigation; and pages contain tabs for organizing relevant information.[4] one of these tabs, “authorities (u.s. code)”, links back to the statutory rulemaking authority for that part or section. in some cases, where the grant of authority comes from something other than a statute, like an executive order, this information will be listed, but not hyperlinked.[5] the information for these authorities pages is programmed to automatically populate relevant information taken from the lii’s ptoa. while just using these online collections with these technologies makes researching regulations easier, sometimes researchers want the current language of the regulation. this can be done by either waiting for the updated publication of the relevant cfr volume, or by using the lsa and individual issues of the federal register, but these options can be time consuming. to answer this problem, the gpo offered an unofficial version of the cfr that was updated almost daily.[6] the lii linked to the gpo’s ecfr from the toolbox on each of their cfr pages in case researchers wanted the current language for a regulation that had been updated. however, since the gpo’s xml files used for the lii’s cfr were derived from print volumes, the files were updated infrequently enough that the content for the end user was dated. the lii addressed this currency issue by creating a “rulemaking” tab. this page is programmed to find issues of the federal register that contain notices, proposed rules, and final rules affecting that section, and then extract the relevant data for display on the page.[7] the cfr project gave the lii a chance to build systems for the way that researchers work. likewise, it gave the lii a chance to work with some new technologies to better connect the general public with the laws they were looking for. to do this, they looked at linked data and semantic web technologies that would enable, for example, someone researching federal regulations to get to the regulation about acetaminophen from a search for the more common name for the drug, tylenol. the lii hosts approximately 30 million unique visits each year. the cfr collection sees approximately one-quarter of that traffic. there’s been an uptick in demand for regulations, but not as many free sources are available for researching regulations as those available for researching legislation. to that end, the lii continues to improve this collection to better help people find and understand the law. lii’s ecfr in august 2015, the gpo announced that they would offer for download the xml files of the ecfr for developers. this gave the lii a chance to scale, re-work, and implement existing technologies on this new collection. it also gave the lii a chance to try out new technologies.[8] the first step in the process was to just get the text of the ecfr up, from the xml files. the new xml was cleaner, but different enough from the book-derived-xml that lii programmer, sylvia kwakye, could not simply reuse what was in place for processing the book-cfr-xml. a new build was required, and was successfully executed. from there, kwakye added all of the lii’s formatting features mentioned above, improving on how those features worked in the old cfr collection.[9] after that, the lii added back the information tabs, including the rulemaking tab and also the authorities tab that integrates their ptoa. another innovation was made for the definitions. key terms can be defined at as high a level as to be applicable to the whole title, or as granular as to be applicable to only that subchapter or even part. to bring clarity to what certain terms mean, especially terms that are seemingly straightforward, the lii created a definitions feature for their ecfr that highlights those defined, legal terms. this feature displays as off-colored, or highlighted words with a dotted-underline. when the highlighted term is clicked, researchers are presented with a pop-up box of useful information which includes the definition of the term, the source for where that term is defined with a hyperlink to that section, and the scoping language which explains, from the source section, where within that title the definition is applicable. the pop-up feature means that the user will not lose their place on the cfr page they are researching. another feature new to the ecfr is the “what cites me” tab. this page is programmed to scan every page of the cfr to find references to that section—i.e. citations to that section found in other cfr sections—and then list them on that page. this is helpful for finding scoping language from a section that might apply to the immediate section being researched. the hyperlinked source credits also got an update. the federal register citations in the source credits of each section in the ecfr now link to the lii’s rio (read it online) database, which brings researchers to a page that provides links to that document from a variety of open and free sources and subscription services from the single, in this case, federal register citation. ecfr pipeline in august, 2015, the gpo finally made available the xml files for their ecfr. today, the lii’s cfr content retrieved for the end user is as up-to-date as the most recent issue of the federal register. when a specific cfr page is requested, i.e., visited, a series of rules within layers of automation retrieve the most up-to-date information available. while the information is current, the pages of content displayed for the end users are static cached pages. lii’s ecfr processing starts with a simple web scraper script that visits the gpo’s bulk download site. it locates the table that lists information about each title. it then extracts each title’s metadata, including the “last updated” date, and compares it to the “last updated” date that we have stored in a file locally. if the date has changed, the program automatically downloads the new xml file and updates the “last updated” date for the title in the local file. after all changed titles have been retrieved, the script writes the list of changed titles to a file named title.changed.text and exits. a non-empty, updated title.changed.text file automatically kicks off the main processing pipeline. this pipeline is a bash script that runs a title file sequentially through a number of programs, each one of which produces a richer xml document until it is finally ingested into an xml database. the processing pipeline is divided into roughly 3 phases: preprocessing, features, and storage. in the preprocessing phase, lii checks (and repairs, if necessary) that all the xml files are utf-8 encoded and well-formed. in addition, lii removes the need to deal with mixed case items by lowercasing the names of all elements and attributes. next, lii adds a variety of identifier attributes for use internally to uniquely identify every division in the document hierarchy. the simplest and most heavily used of these is the “extid,” which compactly encodes the division type and enumerator. with the identifiers in place, lii extracts all sections and appendices into individual xml files. they also generate an xml table of contents document for each title. the table of contents (toc) structure is then mapped to an rdf graph and uploaded to a central graphdb metadata repository. snippets for a section, part, subchapter, and chapter from title 9’s structure rdf are shown below. listing 1: representations of section, part, subchapter and chapter in the rdf graph for cfr title 9. <https://liicornell.org/liiecfr/9_cfr_1.1> a liitop:section ; dc:title "§ 1.1 definitions." ; liiecfr:extid "lii:ecfr:t:9:ch:i:sch:a:pt:1:s:1.1" ; liiecfr:extidds "lii:cfr:2017:9:0:-:i:a:1:-:1.1" ; liiecfr:hasenumerator "1.1" ; liiecfr:hasheading "definitions." ; liiecfr:haswebpage <https://www.law.cornell.edu/cfr/text/9/1.1> ; liiecfr:mayciteas "9 cfr 1.1" ; liitop:belongsto <https://liicornell.org/liiecfr/9_cfr_part_1> ; liitop:haslevelname "section" ; liitop:haslevelnumber 8 . <https://liicornell.org/liiecfr/9_cfr_part_1> a liitop:supersection ; dc:title "part 1 definition of terms" ; liiecfr:extid "lii:ecfr:t:9:ch:i:sch:a:pt:1" ; liiecfr:extidds "lii:cfr:2017:9:0:-:i:a:1" ; liiecfr:hasenumerator "1" ; liiecfr:hasheading "definition of terms" ; liiecfr:haswebpage <https://www.law.cornell.edu/cfr/text/9/part-1> ; liiecfr:mayciteas "9 cfr part 1" ; liitop:belongsto <https://liicornell.org/liiecfr/9_cfr_chapter_i_subchapter_a> ; liitop:haslevelname "part" ; liitop:haslevelnumber 5 . <https://liicornell.org/liiecfr/9_cfr_chapter_i_subchapter_a> a liitop:supersection ; dc:title "subchapter a animal welfare" ; liiecfr:extid "lii:ecfr:t:9:ch:i:sch:a" ; liiecfr:extidds "lii:cfr:2017:9:0:-:i:a" ; liiecfr:hasenumerator "a" ; liiecfr:hasheading "animal welfare" ; liiecfr:haswebpage <https://www.law.cornell.edu/cfr/text/9/chapter-i/subchapter-a> ; liitop:belongsto <https://liicornell.org/liiecfr/9_cfr_chapter_i> ; liitop:haslevelname "subchapter" ; liitop:haslevelnumber 4 . <https://liicornell.org/liiecfr/9_cfr_chapter_i> a liitop:supersection ; dc:title "chapter i animal and plant health inspection service, department of agriculture" ; liiecfr:extid "lii:ecfr:t:9:ch:i" ; liiecfr:extidds "lii:cfr:2017:9:0:-:i" ; liiecfr:hasenumerator "i" ; liiecfr:hasheading "animal and plant health inspection service, department of agriculture" ; liiecfr:haswebpage <https://www.law.cornell.edu/cfr/text/9/chapter-i> ; liiecfr:mayciteas "9 cfr chapter i" ; liitop:belongsto <https://liicornell.org/liiecfr/9_cfr> ; liitop:haslevelname "chapter" ; liitop:haslevelnumber 3 . the pre-processing phase is also where extra content tags and attributes are added to the xml. one of the reasons for the lii’s popularity is the readability of their content: lii developers pay particular attention to the way the information is displayed, for easier consumption by the end user. some of these content tags distinguish a section title from the section text, identify nesting paragraphs, and identify and add hyperlinks to cross-reference citations. listing 2: xml snippet from 5 cfr 151.101 before any enrichment. <p>(c) <i>federal agency</i> means an executive agency or other agency of the united states, but does not include a member bank of the federal reserve system; </p> <p>(d) <i>state or local officer or employee</i> means an individual employed by a state or local agency whose principal employment is in connection with an activity which is financed in whole or in part by loans or grants made by the united states or a federal agency but does not include </p> <p>(1) an individual who exercises no functions in connection with that activity. </p> <p>(2) an individual employed by an educational or research institution, establishment, agency, or system which is supported in whole or in part by </p> <p>(i) a state or political subdivision thereof; </p> <p>(ii) the district of columbia; or </p> <p>(iii) a recognized religious, philanthropic, or cultural organization. </p> <p>(e) <i>political party</i> means a national political party, a state political party, and an affiliated organization. </p> image 1: a screenshot of how the above snippet is presented on gpo’s website listing 3: lii’s enriched xml after identifiers and nested paragraph markup. the class and lev attributes can be used to style the text to improve styles to improve readability. note also the addition of the id attribute which enables the construction of direct links to the content. <p class="psection-1"><enumxml id="c" lev="1">(c)</enumxml> <i>federal agency</i> means an executive agency or other agency of the united states, but does not include a member bank of the federal reserve system; </p> <p class="psection-1"><enumxml id="d" lev="1">(d)</enumxml> <i>state or local officer or employee</i> means an individual employed by a state or local agency whose principal employment is in connection with an activity which is financed in whole or in part by loans or grants made by the united states or a federal agency but does not include </p> <p class="psection-2"><enumxml id="d_1" lev="2">(1)</enumxml> an individual who exercises no functions in connection with that activity. </p> <p class="psection-2"><enumxml id="d_2" lev="2">(2)</enumxml> an individual employed by an educational or research institution, establishment, agency, or system which is supported in whole or in part by </p> <p class="psection-3"><enumxml id="d_2_i" lev="3">(i)</enumxml> a state or political subdivision thereof; </p> <p class="psection-3"><enumxml id="d_2_ii" lev="3">(ii)</enumxml> the district of columbia; or </p> <p class="psection-3"><enumxml id="d_2_iii" lev="3">(iii)</enumxml> a recognized religious, philanthropic, or cultural organization. </p> <p class="psection-1"><enumxml id="e" lev="1">(e)</enumxml> <i>political party</i> means a national political party, a state political party, and an affiliated organization. </p> image 2: a screenshot of how the snippet of 5 cfr 151.101 is presented by the lii. the pre-enrichment xml as used by the gpo and shown in image 1 shows the basic text in paragraph form. since the subsections are all left-justified, it is up to the user to figure out the proper nesting of subsections, whereas the lii’s enriched xml adds styles to the text to improve the readability. specifically, subsection enumerations and sometimes the title for that paragraph are formatted in bold so that pertinent information stands out, and the subsections are properly indented for easier consumption using the class and lev attributes. it is not just the xml files that get enriched when features are added. the rdf is also updated as more metadata becomes available, enabling lii to develop features useful to their users. an example of this is the “what cites me” feature, which, for the end-user, displays as a tab for cfr sections that are referenced in other parts of the cfr. this feature is made possible by adding references to the structure-toc graph. the following snippet of code adds reference information, listing 4: code snippet to update rdf graph with references. if update_toc: toc_graph.add((uriref(my_uri), top["hasxref"], (uriref(target_uri)))) updates the metadata for section 1.1 in title 9 so that it now looks like: listing 5: updated section the rdf graph <https://liicornell.org/liiecfr/9_cfr_1.1> a liitop:section ; dc:title "§ 1.1 definitions." ; liiecfr:extid "lii:ecfr:t:9:ch:i:sch:a:pt:1:s:1.1" ; liiecfr:extidds "lii:cfr:2017:9:0:-:i:a:1:-:1.1" ; liiecfr:hasenumerator "1.1" ; liiecfr:hasheading "definitions." ; liiecfr:haswebpage <https://www.law.cornell.edu/cfr/text/9/1.1> ; liiecfr:mayciteas "9 cfr 1.1" ; liitop:belongsto <https://liicornell.org/liiecfr/9_cfr_part_1> ; liitop:haslevelname "section" ; liitop:haslevelnumber 8 ; liitop:hasxref <https://liicornell.org/liiecfr/9_cfr_1.1>, <https://liicornell.org/liiecfr/9_cfr_2.1>, <https://liicornell.org/liiecfr/9_cfr_part_2>, <https://liicornell.org/liiecfr/9_cfr_part_3>, <https://liicornell.org/liiecfr/9_cfr_subpart_c> . lii built their first “what cites me” feature for a cfr section by querying the central ecfr repository for the list of sections that reference that section. they have since developed a more extensive graph model for references in general, and are moving on to new referencing features, in addition to migrating the “what cites me” feature to other corpura. all of this activity is logged so that lii developers can later compile reports on various metrics and quickly find problems when something goes wrong. the example below shows the progression of the pipeline with some of the messages it logs as it processes a single title. bolding and extra spaces are added to the output to make it easier to pick up important events. listing 6: sample processing log ~ input array of titles to process: ['9'] num_of_titles to pre-process: 1 no pooling, only one title. pre-processing factory-ecfr/stages/gposource/ecfr-title9.xml working in process #: 9733 --cleaning up and making all element and attribute tags lower case --writing new file to to factory-ecfr/stages/lowercase_tags/ecfr-title9.xml --adding identifiers --add_lii_ids took 3.3294479847 seconds. --generating clean toc (for definitions) ----generating toc with misc data elements (eg. hed, notes etc.) --title 9 before merging 2 volumes subtitles: 0 | subjgrps: 150 | appendices: 0 |sections: 2517 | chapters: 3 | subchapter: 19 | parts: 153 | subparts: 145 factory-ecfr/stages/toc/ecfr-title9-datatoc.xml after merging volumes . . . subtitles: 0 | subjgrps: 150 | appendices: 0 | sections: 2517 | chapters: 3 | subchapters: 19 | parts: 153 | subparts: 145 --generating structure rdf from toc -factory-ecfr/metadata/toc_structure/title9 factory-ecfr/metadata/toc_structure/title9.nt --chunking titles: extracting sections/appendices --chunking up the title title 9 done chunking! wrote a total of 2517 files to factory-ecfr/stages/section_chunks/title9/ 2517 section files 0 appendix files chunk_up_title took 3.85553884506 seconds. --creating nested paragraphs --in nested paragraph detector: 2517 files found in factory-ecfr/stages/section_chunks/title9/ .. found 740 new paragraphs. .. found and tagged 22 additional paragraphs. tagged output written to factory-ecfr/stages/nested_paragraphs/title9/ --adding cross reference markup (to other places in the title) -- .. found and tagged 1461 paragraph, 820 section, and 595 supersection cross references within a total of 2518 files. --adding aref markup (external-other cfr titles, uscode etc.) --output for arefs (sections) factory-ecfr/stages/arefs/title9/ ....... finished adding arefs ....... ....... getting xref rdf ....... factory-ecfr/metadata/xrefs/title9 total num of cfr cross-references found (in sections and appendices) in title 9: 4334 ('writing xref rdf to', 'factory-ecfr/metadata/xrefs/title9.nt') --updating lii_ecfr_changelist.json and lii_ecfr_meta.xml metadata files for processed titles which supersections changed in title 9? checking toc ... which sections and appendices changed in title 9? checking file contents ... summary changed = 57 removed = 0 added = 0 checked: 2018-jan-09 14:54:23 ... updating title 9 in /stages/lii_ecfr_urls_changelist.json ... updating title 9 in /stages/lii_ecfr_extids_changelist.json ... updating title 9 in /stages/lii_ecfr_files_changelist.json update_title_changelist_meta took 1.71637392044 seconds. .. meta file found and updated at factory-ecfr/stages/lii_ecfr_meta.xml <title> <num>9</num> <head>title 9 animals and animal products</head> <extid>lii:ecfr:t:9</extid> <old_extid>lii:cfr:2016:9:0</old_extid> <subtitles>0</subtitles> <chapters>3</chapters> <subchapters>19</subchapters> <parts>153</parts> <subparts>145</subparts> <subjgrps>150</subjgrps> <sections>2517</sections> <appendices>0</appendices> <volumes> <total>2</total> <volume amd_date="oct. 12, 2017" node_prefix="9:1." num_sections="1216"> sections 1.1 to 167.10 </volume> <volume amd_date="jan. 1, 2018" node_prefix="9:2." num_sections="1301"> sections 201.1 to 592.650 </volume> </volumes> <published>05-jan-2018 04:02</published> <last_injested>tue jan 9 14:54:24 2018</last_injested> <toc_year>2018</toc_year> <content_year>2018</content_year> </title> --generating redirect files for legacy urls. updating graphdb toc and references repositories for titles 9 python factory-ecfr/semweb/graphdb_toc_load.py 9 ecfr_structure loading into context file://title9.nt response 204 python factory-ecfr/semweb/loader.py 9 ecfr_structure xrefs found rdf file: factory-ecfr/metadata/xrefs/title9.nt response 204 triplestore update after preprocessing ecfr titles 9 complete querying for title 9 reference uris response 200 968 reference uris retrieved. starting what-cites-me with 4 threads what_cites_me took 17.6925630569 seconds. making whatcitesme.json files, complete adding definitions to titles 9 generating char files for titles 9 python factory-ecfr/definitions/getcharfiles.py factory-ecfr/stages/arefs/title9 working [2517 files total] ... done with char files, starting up definitions extraction and markup 9 java -jar factory-ecfr/definitions/bin/ecfrdefinitions-1.5.01-snapshot.jar 9 factory-ecfr/stages factory-ecfr/metadata . new word added to map: fmd-susceptible count: 1 new word added to map: deboning count: 2 new word added to map: strain-to-strain count: 5 new word added to map: heath count: 11 . title 9 is processed in 592 seconds adding definitions to ecfr, titles 9, complete python factory-ecfr/preprocess/htmlizer.py -src defs 9 starting the process of converting xml to xhtml for ecfr title 9 in htmlizer: 2518 files found in factory-ecfr/stages/definitions/title9/ output written to factory-ecfr/stages/html/title9/ finished processing xml files from factory-ecfr/stages/definitions/title9/ it took 3.53 seconds total. this subset of the log shows progression of the pipeline from preprocessing through to the extraction and markup of definitions, and ends when the xml is transformed to xhtml. the definitions feature is currently the most computationally intensive process and takes the longest to complete. title 9, animal and animal products, is in the middle of the pack in terms of file size. it takes about 10 minutes when all sections and appendices are processed for definitions. title 26, internal revenue, the second largest title, takes about 12 hours to complete processing. a full discussion of how the definitions feature is implemented is beyond the scope of this article. in short, lii uses a dictionary of well-defined rules to find candidate definitions. then, they use entity extraction and basic disambiguation techniques to locate defined terms in context and map them to their correct definitions. given all the intermediate stages that the data goes through, it is very important for lii developers to check and make sure that the text content of the regulations remain unchanged. this check is done with a sequence matcher. this is illustrated by the following small section of code from the reference identification and markup script: listing 7: reference identification and markup if len(reflist) > 0: old_textchunk = etree.tostring(elem, encoding='utf-8', method='text') reflist.sort(key=itemgetter(0)) n = markup_refs(elem, reflist, textchunk, title_number) num_crossrefs += n # sequence matching to check for changes in the text # note -there should be none, ideally similarity should be 100% new_textchunk = etree.tostring(elem, encoding='utf-8', method='text') diff, opcodes = sanity_check(new_textchunk, old_textchunk) if diff == 100: seqmatch['same'] += 1 else: seqmatch['different'] += 1 diffnums.append(diff) msg = sanity_check_msg(old_textchunk, new_textchunk, diff, opcodes) print msg as the code snippet shows, after references in a paragraph are marked up, the text is run through the aptly named “sanity_check” utility function (see below). it uses the sequencematcher from the python standard library to compare text from the original paragraph to the newly marked up one. listing 8: sanity check def sanity_check(text1, text2): seq = difflib.sequencematcher() seq.set_seqs(text1, text2) diff = seq.ratio()*100 op = seq.get_opcodes() return diff, op def sanity_check_msg(text1, text2, diff, op): msg = ''' --------------------------------------------------- % similarity = {} text chunk 1: {} text chunk 2: {} opcodes show spots where the text matches and what to do to make text chunk 2 match text chunk 1. {} --------------------------------------------------- '''.format(diff, text1, text2, op) return msg if the texts are not identical, a message will be printed showing exactly where the mismatches occurred. these kinds of messages are one of the things lii developers check for and correct before serving the content. in the storage phase of the pipeline (not shown in the log), the final html, the xml from the last feature implemented, and toc files of the titles are uploaded to an exist database. the json files that support features like definitions and “what cites me” are moved into the appropriate directories within the web tree of the lii site. the original xml files from gpo, the final xml file, log files, and all the metadata files generated during processing are also uploaded to an amazon s3 bucket. finally, the driver script generates an alldone.out file which is also uploaded to the ecfr storage bucket. the ecfr factory is actually an ec2 instance. writing the alldone.out file to s3 triggers a couple of functions. one sends a message to the developers and the second shuts down the instance. drupal is the content management system that the lii uses to serve the content on its website. drupal allows dynamic content boxes to be dragged around by admins to build different views on any given page that will be seen by the end user. examples of content boxes on an ecfr page include the “body” of the page where the text of the table of contents or section is shown, the toolbox on the right-side of each page, and the breadcrumb trail above each page’s body section. each box is programmed by the lii to display different blocks of text or other features that the lii is interested in showing. for instance, the contents of each toolbox change from collection to collection; for example, only the ecfr toolbox will contain a link to the gpo’s ecfr. content is not retrieved from the exist database every time someone makes a request for a page on the lii website. instead, drupal is programmed to retrieve data from exist, with an xquery, the first time someone requests it. it caches and serves this saved version of the page on subsequent requests. this and other optimizations put in place by lii’s systems administrator, nic ceynowa, has worked not only to decrease the load time for these pages, but also to support thousands of simultaneous requests for the same page. [10] listing 9: example of the xquery used to retrieve the table of contents of a title. xquery version "3.0"; let $req_title := request:get-parameter("title", "") (: locate the correct toc document :) let $doc := doc(concat("/db/ecfr/lii/title-", $req_title, "/ecfr-title", $req_title, "-datatoc.xml")) let $base_url := concat("/cfr/text/", $req_title) let $this_title := $doc//.[lower-case(@type) = "title"][./num = $req_title] let $outbuf := if ($doc) then <supcontent> <div class="toc"> <ul>{ (: div2|div3|div4|div5 == subtitle|chapter|subchapter|part :) for $item in $this_title/(div2 | div3 | div4 | div5) let $this_catchline := $item/head let $myextid := $item/@extid let $myid := $item/idnums let $myid_tokens := tokenize($myid, ":") let $mytype := lower-case($myid_tokens[last() 1]) let $myhead := $item/head//text() let $mynum := tokenize($myextid, ":")[last()] let $mypath := concat($mytype, "-", $mynum) return <li class="tocitem"> { if (contains($this_catchline, "[reserved]") or contains($this_catchline, "[reserved]")) then <span>{$this_catchline}</span> else <a title="{$myhead}" href="{$base_url}/{$mypath}">{$this_catchline}</a> } </li> } </ul> </div> </supcontent> else (<supcontent/>) return $outbuf updating drupal’s cache after running the processing pipeline is not yet fully integrated into the automation sequence. first, a developer reviews the logs to make sure there are no errors that need to be resolved. then a visual check of a selection of pages is done in a staging environment. if all is well, they manually kick off another script that cycles through all the updated pages on the lii website. it deletes each one and immediately requests the page so that drupal can cache it. automating this last part of the cfr pipeline is another project that is on-going. the key to doing this successfully is designing meaningful tests for both the processing and presentational aspects. future: docket wrench integration in the last year, the team of programmers at the lii have been working to revive an application called docket wrench. the software was originally built in 2011 by the sunlight foundation, a government transparency organization, to track how organizations and influencers were engaging in the notice-and-comment rulemaking process over time. the application was not maintained and when sunlight foundation discontinued its software development program, lii picked up the project. the software is programmed to pull in rulemaking activity from regulations.gov, as well as individual agency websites who are not using regulations.gov, to create complete and comprehensive rulemaking dockets. where regulations.gov does not compile related materials, docket wrench offers all related information on a single page, in an easily understandable interface. each docket is presented as a timeline, showing the duration of the rulemaking process, at which point in the process the comments were received, and all of the comments and documents collected during the comment period. the software analyzes the comments, allowing researchers to fully understand the major influences with visual representations of when comments were submitted and determining the weight of support based on the number of comments submitted by or on behalf of particular organizations or persons. the challenges in reviving the application and building it out into a full-fledged research tool for the rulemaking process are significant. since docket wrench was originally developed, new classes of data stores have matured, more government rulemakings are now retrievable via api, and technologies such as apache spark have made large scale data processing far easier. refactoring the obsolete parts to use these modern technologies is an ongoing adventure for the developers and students. during the re-introduction of this software, sara frug, associate director for technology at cornell’s lii, said that it “fills a niche adjacent to a popular [lii] offering.”[10] it is our hope that the information compiled and analyzed by the docket wrench software will, in the future, be incorporated into the lii’s ecfr collection. notes [1] quote of tom bruce, lii co-founder and director. gpo and cornell university pilot open government initiative, see https://www.fdlp.gov/news-and-events/733-gpo-cornell-pilot [2] for a more detailed accounting of this process, see sara s. frug. ground-up law: open access, source quality, and the cfr, available from http://www.hklii.hk/conference/paper/2b3.pdf [3] for a complete list of the original features, please see the lii’s youtube playlist, why our cfr doesn’t suck, available from https://www.youtube.com/watch?v=ck_jgmr1uki&list=plelct7ye-lvosm4qroq17dcfh72zbr905 [4] for a detailed accounting of the process and challenges in building the cfr, see frug, supra note 2. [5] when the lii first published their ptoa, there was not a static, linkable destination to these documents. since then, these documents have become readily available and the lii team is in the process of creating a hyperlink application enhancement for these and all other recognizable citations. [6] this unofficial cfr was updated as each new issue of the federal register was published. this means that when a final rule was published in the federal register, that new or amended language was readily available in the gpo’s cfr. [7] the programming for this is legacy code that is soon to be phased out due to the currency of the cfr text. essentially, new federal register content is checked in a similar fashion to that of how the cfr titles are checked for updates, see supra ecfr pipeline, with a major difference being that the federal register document is not downloaded, and only the metadata from the document is stored in a database for display when a user queries a section of the cfr that is in the process of being amended. [8] see lii: under the hood for a more detailed accounting of their ecfr collection, available from https://blog.law.cornell.edu/tech [9] see https://blog.law.cornell.edu/tech/2015/10/23/a-look-at-indentation/ [10] see https://blog.law.cornell.edu/tech/2015/10/30/a-day-in-the-life-traffic-spike/ about the authors charlotte d. schneider, https://orcid.org/0000-0003-3248-1569, is the government documents librarian at rutgers law school in camden, nj. she also works closely with the legal information institute at cornell university. sylvia kwakye is a developer at the legal information institute at cornell university. sylvia uses data science technologies to continuously enhance the experience of users who visit the lii site. she also mentors masters of engineering students, managing their semester-long projects. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – editorial introduction: the code4lib journal isn’t just for coders mission editorial committee process and structure code4lib issue 12, 2010-12-21 editorial introduction: the code4lib journal isn’t just for coders although the primary goal of the code4lib journal is to provide practical solutions for technologists working in libraries, it has a lot to offer non-technologists. technology affects all of the work that our libraries are doing and will define what the future of libraries will look like. by ron peterson coordinating editor, issue 12 most of us spend the bulk of our days in front of a computer, even if we don’t consider ourselves technologists, because the processes that run the library are increasingly driven by technology. every year technology becomes a more pervasive presence in our libraries. every job is affected by technology, and the future of libraries is being shaped by the application of technology. additionally, our parent institutions increasingly want to see data regarding how well we are serving our users. using technology to generate or analyze this data will also become more important. the future of libraries is going to be decided by how, and how well, technology is applied to the needs of the library. if you are interested in libraries, you should be interested in the code4lib journal. whether you are a technologist or not, the code4lib journal has something to offer you. no matter what type of library you work in, the problems you face will be similar, and the technological solutions found at another library can help you solve a problem in your own. if you need to find a way to connect students with available computing resources, the solution presented by kim griggs in “how to build a computer availability map” may work for your library. if you are looking for a better way to manage your web site, but find drupal too complex, “creating library websites with joomla: not too big, not too small, just right” may have an alternative that will work for you. if maintaining your subject guides has become too time intensive, “subject guides & more: creatively transforming an open source management system” presents a solution for libraries with limited resources. these articles will give anyone working in libraries important insights into solutions for your problems. if you aren’t a technologist, reading the code4lib journal will also help you communicate with your it staff. reading about how another library implemented a solution will make you better prepared to discuss solutions in your own library. you will gain an understanding of what is involved when undertaking some of these technology projects, including the pitfalls that your library may face, as well as the advantages it may gain. most importantly, you will get a sense of what challenges your it staff face when they try to implement these solutions. the code4lib journal focuses on articles written in accessible language that offer practical solutions to the problem faced by libraries. being a regular reader of the code4lib journal will help you familiarize yourself with the technology used in libraries. even if you don’t work with technology yourself, you have to work with people who do. the better you understand the technology, the better you will be able to communicate with your it staff. the article, “practical ways to promote and support data analysis programs,” is a great example of collaboration between technologists and library professionals. “it shows that a better understanding of the work others do in the library can benefit everyone. another example of a typical area where non-technical staff have to interact with technical staff is the area of e-resources. “electronic resources security: a look at unauthorised users” can help all library staff members get a better understanding of tracking unauthorized use of licensed resources. also, the article on wattjournals shows how you can use technology to make finding articles that your users can access through your library more efficient. the code4lib journal provides an accessible way to look behind the curtain and see how your it staff are thinking and the environment they are working in. technology is defining the future of libraries. there is no way to avoid it. everyone working in libraries needs to be aware of the ways that technology is impacting our jobs. keeping up with the code4lib journal will help you stay aware of what other libraries are doing and how that may influence how your job will be done in the future. for example, understanding agile development as it is applied to the creation of a mobile web site, “using an agile-based approach to develop a library mobile website,” will help you get a handle on developing your mobile web site and give you some background about how technology projects are managed. reading about the application of cloud-computing to digital preservation in “using amazon s3 in digital preservation in a mid sized academic library: case study of ccsu eris digital archive system” may inspire your own ideas for applying cloud computing or preserving digital objects. for more about cloud computing or to see the application of new cataloging technologies/methodologies to improve user experiences, you can check out “frbrizing an e-library : migrating from dublin core to frbr and mods.” monitoring technology journals is a great way to keep yourself prepared for the future. however, by the time new ideas and new solutions are published in traditional library journals, they are no longer new. the code4lib journal moves quickly from proposal to published article in order to get your colleagues’ ideas to you while they are still fresh. the code4lib journal is an excellent resource for looking into the future of libraries. the code4lib journal is a great way for technologists and coders to share the work they are doing for libraries and to learn about what their colleagues are doing. but you don’t need to be a technologist to read the code4lib journal. the code4lib journal offers a timely, accessible, and open way to learn about how people are using technology in their libraries and to find ways to apply it in your library. by learning about how technology is used in libraries, we can work together, technologists and non-technologists, to determine the future of the library. subscribe to comments: for this article | for all articles one response to "editorial introduction: the code4lib journal isn’t just for coders" please leave a response below: eric lease morgan, 2010-12-21 very nice editorial. i like how you appeal to people who are not computer programmers, and i especially like \reading the code4lib journal will also help you communicate with your it staff.\ well put. we all have more in common that difference. thank you. –elm leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – editorial: on foss in libraries mission editorial committee process and structure code4lib issue 54, 2022-08-29 editorial: on foss in libraries some thoughts on the state of free and open source software in libraries. by andrew darby i’ve been thinking recently about open source software in libraries, partially because i was wondering how i could squeeze into this editorial a plug for subjectsplus [1], but also because i’ve noticed, in an unscientific way, that libraries seem more closed to open source. when i started in the library biz, over twenty years ago, we talked about open source software a lot. it had a feel-good glow around it: it was righteous, it was collaborative, it was free! it was accessible to the self-credentialed and the tinkerers. installing free and open source software (foss) was a way for people in libraries to concretely skill up: a cataloger or a reference librarian might muck around with setting up the lamp stack, encounter a few install errors, and eventually move on to detailed bug reports and then pull requests. their career might veer off in an interesting new direction. learners everywhere–intentional or not–need a concrete project to work on, and foss applications can fill that role. foss used by libraries consists of both library-specific projects (access to memory, blacklight, coral, dspace, evergreen, islandora, koha, omeka, pkp project, samvera, subjectsplus, suma, vivo, vufind, etc.) and more general audience applications that are frequently used as a general purpose cms (drupal, wordpress) or hidden in the stack (apache, elasticsearch, fedora, solr, etc.). there are some library projects that are clearly multi-institutional, like samvera, but the majority have a de-facto host institution. so why, when interviewing candidates for a library technology position, do i sometimes ask the question: what are the pros and cons of open source software? cons? what? * * * the answer, partially, is that some of those tinkerers got credentialed and/or became managers. they started to think about budgets, and the cost of staff time, and competing priorities. they got pushback from campus it on supporting unfamiliar applications. some decided, well, let’s just write a check. for each of the library foss projects i listed above there is at least one paid alternative, and it is generally in the ascendant. [2] meanwhile, over the years the coding ecosystem has gotten much more complicated. at the university of miami we are currently in the final stages of rebuilding subjectsplus from the ground up. what was once basically vanilla php, sql, html, css and js, is now a pyramid of frameworks: symfony instead of php; doctrine instead of sql; twig instead of html; sass instead of css; react instead of javascript; and for good measure api-platform instead of a bespoke api. some space on a server has become a dockerized web app in azure. not only do the learning curves of all these frameworks make it harder for the curious to explore and tinker with a foss project, it also means you can never fully know your own application; there are just too many dependencies downstream. that can seem like a security nightmare, unless you have a measure of trust in the countless people out there who have their own passion projects, who leave their code open for inspection, who plug holes when they encounter them. the visibility of open source code is also one of its strengths, from a security perspective: it’s not a black box. if one can use search trends as a metric of interest or use, “open source” seems to be trending downwards. the first issue of the journal was published on december 17th, 2007, and you can see that during its lifespan there has been a pronounced decline in (googling) “open source”: figure 1. google trends for “open source,” december 2017 – present [3] but what about in the journal itself? you could say that open source is in the code4lib journal’s dna: we are an open access journal (not quite the same thing) which publishes our articles under an open source license (us cc-by) and recommends that any included code be given an open source license of the author’s choosing. when i search the backend of the journal (which is the open source wordpress cms), i get 254 hits for “open source,” which means that almost half of our 516 published posts have this phrase in them [4]. in fact, each of our 54 issues has at least one mention of open source in it. issues 32 and 40 had only a single hit, while issue 12 had an impressive 10. this editorial alone has hundreds of hits! so let’s bow to the numbers: open source is alive and well in libraries. at least with the people who write and read about it. notes [1] https://github.com/subjectsplus/subjectsplus/ i had hoped to be doing a plug for the new version of the software, but with the departure of a key developer, we’re doing calisthenics just short of the finish line. version 5 coming soon-ish! [2] my own project is no exception; we have gone down from over 75 libraries worldwide to, well, i don’t want to count. this could be a factor in my impression that libraries are souring on open source. or it could be that folks are just waiting for version 5 of subjectsplus to switch back. [3] https://trends.google.com/trends/explore?date=2007-12-17%202022-08-19&q=%22open%20source%22 [4] and if one were to broaden your search to include articles that utilize open source applications (but without using the phrase “open source”) or that share code inline or in an external repository, the number would be considerably higher. about the author andrew darby is the head of web & application development at the university of miami libraries, a member of the editorial committee of the code4lib journal and, you guessed it, the founder of the open source library cms subjectsplus. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – the rutgers workflow management system: migrating a digital object management utility to open source mission editorial committee process and structure code4lib issue 1, 2007-12-17 the rutgers workflow management system: migrating a digital object management utility to open source this article examines the development, architecture, and future plans for the workflow management system, software developed by rutgers university libraries (rul) to create and catalog digital objects for repository ingest and access. the workflow management system (wms) was created as a front-end utility for the fedora open source repository platform and a vehicle for a flexible, extensible metadata architecture, to serve the information needs of a large university and its collaborators. the next phase of development for the wms shifted to a re-engineering of the wms as an open source application. this paper discusses the design and architecture of the wms, its re-engineering for open source release, remaining issues to be addressed before application release, and future development plans for the wms. by grace agnew & yang yu introduction ingest of digital objects is a core service of a repository architecture. most repository services, from preservation and storage to discovery and retrieval, are dependent on the information collected about the digital object at ingest. the scalability of the repository, particularly to support the simultaneous ingest of many digital collections, is also dependent on this critical service. the workflow management system (wms) was originally developed to meet rul’s need for a flexible, extensible web-based service to support repository development. the wms includes a sophisticated metadata architecture that was designed to support any digital collection, from any contributor, whether a faculty member depositing the research products of a large scientific experiment or a small historical society participating in a collaborative cultural heritage portal. this article describes the development of an object ingest and metadata creation application that began as a front-end service for rul’s fedora repository. as we presented our wms to our peers at conferences [1], we were approached about sharing the application. before sharing wms with libraries, archives, and other organizations, we needed to significantly retool the software to remove rutgers-specific dependencies and to support customization for each institution’s unique circumstances for information management and delivery. this paper describes the background and rationale for developing the wms; its design and functionality, particularly to provide a sophisticated event-based data model and metadata architecture within a mets (metadata encoding & transmission standard) framework; and the re-engineering decisions that were required to create a robust open source application. the article closes with the policy and procedural issues that must be addressed before the wms is released in the open source community, as well as the next steps in wms development. background in 2002, the rutgers university libraries began exploring open source repository platforms to serve as the basis for a comprehensive cyberinfrastructure that would manage the preservation, access and use of the intellectual property of a large research university. the fedora repository architecture [2] was selected for the sophistication of its service-oriented design and the simplicity of its approach to resource management. within fedora, anything can be an information object. no assumptions are made about the nature of the information to be managed or its intended use. core services for preservation and management are provided for digital objects, but most services beyond basic preservation and access must be locally developed and layered upon the core architecture. our survey of rutgers research, based on an analysis of grant-funded projects and other research products hosted on rutgers websites, revealed an enormous breadth of digital content and a wide range of approaches for managing that content, from large scientific databases maintained on multiple excelâ® spreadsheets to a complete portal with metadata, digital objects, and a suite of user services. we realized that we would need a flexible, extensible metadata architecture to encompass the wealth of information available on campus and to support the metadata decisions that many faculty had already made to support discovery and access by colleagues in their disciplines. in addition to developing an institutional repository to support faculty research and publications, rutgers had received a grant from the institute of museum and library services to develop a statewide cultural heritage repository, the new jersey digital highway [3]. discussions with archivists, museum curators and librarians around the state identified a need to provide not just discovery and access but management of the state’s cultural heritage resources. as one of the original 13 u.s. colonies, new jersey has a rich historic heritage, with many artifacts housed in small historical societies and museums around the state. the cultural heritage community was very interested in an information architecture that would support ongoing management of the analog source materials-photographs, papers, artifacts, etc., as well as the digital surrogates deposited in the new jersey digital highway repository. the rutgers university libraries needed an information architecture that would support several critical needs: to enable the libraries to integrate and support the heterogeneous research products and publications of rutgers university faculty; to support management, preservation and access to historic and cultural source materials, particularly the new jersey digital highway collection and rul’s special collections and to integrate with the fedora core architecture which supports both dublin core and the fedora native xml schema, foxml. the rutgers information architecture rutgers decided to use the metadata encoding & transmission standard, a metadata architecture supported as an international standard by the library of congress [4]. fedora was initially designed with a mets data architecture. fedora migrated its data architecture to foxml, which maps to mets and has been described as “mets lite,” so the choice to use mets made sense from an architectural standpoint. in addition, mets provides all the categories of information needed to manage and provide access to a resource. mets concatenates different types of metadata with one or more versions of the object, as well as structural information and behaviors for relating mets components, navigating complex objects, such as the pages of a book and for displaying and using the digital object. the mets envelope provides a standardized xml wrapper for organizing, storing, managing and transporting all the mets components as a single object. the mets document is a standardized transmission package that can be shared across mets-compliant repositories. there are seven component metadata documents within a mets document: header, which provides basic information about the creation of the mets object descriptive metadata, supporting discovery and access to resources administrative metadata, providing metadata about the creation, provenance and use of resources. administrative metadata includes four subtypes: technical, source, rights and digital provenance file section, which groups together related files, such as the different digital manifestations of a resource-the tiff digital master file, the jpeg access copy, etc. structural map, which provides the hierarchical structure of a complex resource, and links the elements of the structure to relevant content files and metadata structural links, which enable hyperlinks between nodes in the structural map behaviors are procedures or applications that can be executed upon content contained within the mets package. [4] figure 1. mets information package mets accommodates all the physical manifestations of an information entity. in rutgers’ implementation of mets, the source object is the first generation of information under the control of the organization. for example, rul may own a photograph of the venus de milo. the famous statue itself is not owned or managed by the library. instead the source object, which is the first generation of information under the control of the organization, is the photograph of the venus de milo. source information objects are generally information that can pass the “hurricane test,” in other words it is the information you will save if your archive is in the path of a hurricane and you must evacuate the premises. rul documents information about the provenance and condition of analog source materials in source metadata. rul documents the characteristics of born digital source objects and digital master files in technical metadata. rul places an equal focus on access to resources and long-term availability. this requires capturing information at ingest sufficient to manage objects and to safeguard and document intellectual property rights for rights holders. the rutgers mets implementation includes a complete rights metadata implementation, with the ability to link rights documentation (deeds of gift, permission request letters, privacy releases, etc.) to the rights events that secure for rul the ability to make copyright-protected resources available for web dissemination. one important way to ensure the long-term usefulness of information is to provide durable context for resources, so that a user in the future knows what they are accessing, how it is created, and what rights they have to its use. since copyright currently extends 70 years beyond the death of the creator, durable provenance is important both to ensure authenticity of information and to ensure its legal availability over time. we address provenance by supporting the ability to add lifecycle and use information via event metadata continuously throughout the life of a digital resource. the rul data model is primarily an event-based data model, intended to document the lifecycle of each digital resource incrementally, over time. figure 2: rutgers university libraries data model rul adds preservation and condition events, provenance events, rights events, and descriptive events, which document the cultural, pedagogical and research usefulness and impact of resources over time to the core metadata we create at ingest within the different mets documents. we chose the “event”, what happens to a resource at a specific time and place, within the context of mets metadata categories (descriptive, source rights, and digital provenance), as a standard conceptual data model that will work with all information domains. an event can have associated entities, such as a granting agency, an awards agency, a preservation service provider, a rights holder, or an exhibit curator. an event can have associated objects, such as a deed of gift or website. a descriptive event will also allow us to use social networking technologies, to capture critiques, recommendations, and use patterns by colleagues in a standardized way to provide nuanced access to resources. such tracking is particularly critical for information products, such as experiment documentation and multimedia files, that may not have peer review status. the event data model addressed our need to create rich “living” data objects that add the context necessary for provenance, discovery and use, and the mets document provides a framework for the contextual events we wanted to capture. examples of provenance events include acquisition, donation, etc. examples of preservation events include repair, reformatting, etc. examples of rights events include license or permission, rights transfer, etc. events provide meaningful context and document the entire lifecycle of an object. figure 3 provides an example of a descriptive event-the exhibition to which an object belongs. events enable us to document associated entities and objects, such as the curator of an exhibit and the exhibit catalog. figure 3: example of a descriptive event in the wms exhibition to which the resource belongs another important event that can be captured in the wms is the rights event. whenever possible, we document the deed of gift or permission received from the rights holder that enables the repository to make a copyright-protected resource available for users. figure 4: rul rights event -deed of gift the workflow management system once we had a conceptual data model, a metadata architecture and a data element registry, we needed to incorporate the complex metadata architecture into a web-based object ingest and metadata capture tool for the rul fedora repository architecture. this tool was the workflow management system, which at that point consisted of a skeletal “placeholder” metadata implementation and a pipeline application for ingesting digital resources into the repository, creating access copies in multiple formats, and creating ocr files for textual materials or transcripts accompanying media files, which could be searched via full-text. the workflow management system needed to support a range of large and small libraries, museums and archives that would add resources and create metadata independently of the rutgers university libraries, as well as rutgers faculty, who would deposit publications and research products in the rucore, the rutgers community repository [5]. for most new jersey digital highway and rucore participants, the workflow management system would be their only exposure to the inner workings of the repository. the wms needed to be robust and intuitive while supporting a very sophisticated data architecture that used concepts and terminology that were unfamiliar even to experienced catalogers, who have worked primarily with marc, dublin core and mods (metadata object description schema) metadata standards. the workflow management system that we designed over several years to support our sophisticated data architecture is now a core enabling technology for the rul cyberinfrastructure. for two years, the wms was extensively tested through its use for njdh by participating museums, libraries and historical societies around the state of new jersey. three important issues emerged that had a great impact on the continuing development of the wms. to begin with, most participants had already created some level of metadata for their collections, even if just a simple spreadsheet or word processing file, and they were understandably reluctant to re-create this information, even through cutting and pasting. they were willing to iteratively add to their metadata once it was ingested, particularly if they felt the incremental event metadata added value, but they were unwilling to recreate existing information from scratch. a flexible mapping utility that could accept and map data elements from any metadata schema, from the complex to the rudimentary, became a critical component for enabling museums and libraries to participate in the new jersey digital highway. the mapping utility received a further test recently when rul assisted the virginia tech library in developing a commemorative repository for the april 16 shooting tragedy. rul was able to successfully map the spreadsheets that inventoried the thousands of banners, cards, and other tributes that the university received in the aftermath of the tragedy into useful metadata to enable virginia tech to quickly create a permanent digital archive. a second issue was the need to ingest large amounts of source objects in bulk rather than uploading each digital object one at a time. the wms initially required that each object be individually loaded. this proved to be very time-consuming, particularly for large digital files. requiring that users individually load each digital object is very inefficient and definitely not scalable. it became important to develop a mass-ingest capability that supported unattended bulk loading of objects to address this issue. a final requirement was to support some level of customization for participants. we were not able to anticipate the vocabulary needs of all participants for populating metadata elements. we also needed to support local decision-making about data elements to display to their catalogers. we added a template capability to enable participants to select mandatory and recommended data elements for display and to add default values for data elements. default values for technical metadata are particularly useful, since resources are either digitized to standard technical specifications or created as born digital objects in a standard digital format. the template capability allows users to customize the look and feel of the metadata input to suit their needs. given the complexity of the ru metadata implementation, this was a critical feature. the template enables organizations to use the complex data architecture iteratively-adding from the large array of available data elements over time, as their expertise grows or their information management needs change. the workflow management system was initially developed beginning in 2003 to provide a robust and intuitive user interface for the fedora repository system. its initial design reflected its dual purpose for supporting the fedora architecture and the needs of a diverse group of libraries, museums and archives with cultural heritage resources. as the wms developers began to speak about the wms in a wider environment, interest among other organizations in using the wms intensified. in 2006, the library of congress motion picture, broadcasting and recorded sound division contracted with rul to develop a bibliographic utility to support the moving image archives community, as part of its mic (moving image collections) project. [6] in the course of wms development, rul began thinking about adding modularity and additional customization features to the wms so that it could be used as a stand-alone application or integrated with other repository architectures by a wide range of organizations. the features and functionalities associated with wms have been constantly growing, and wms is moving towards becoming a generic integrated digital object workflow management system to be released to public as open source package in early 2008. the obligations imposed by open source required that the design of wms must be flexible to meet different needs of diverse users; the functional components must be highly modular for software developers to easily add or remove; and the software must be readily maintainable. at rutgers, the wms currently serves as a front end for several user applications, the new jersey digital highway, the rutgers community repository (rucore), the faculty deposits module, the open journal platform and the electronic theses and dissertations (etd) module. the wms is a cornerstone application for the rucore cyberinfrastructure. figure 5: role of the wms in the rucore cyberinfrastructure capabilities of the workflow management system the workflow management system provides a complete object ingest and metadata creation system, with services to ingest objects and metadata and to export these objects and metadata, individually and in bulk. the wms consists of object handling, which includes object ingest, object structuring for any format (image, text, multimedia), and object reformatting. the pipestream application that creates multiple formats from the ingested digital master file is very rul specific, since we currently use a pdf server application as the “middleware” for reformatting multiple access formats for text and images. this capability has been modularized and abstracted from the wms so that organizations can integrate their own application, which may be jpeg2000-based, for example. object handling includes the ability to ingest transcripts and to provide ocr for text and transcript files. the wms provides a full mets metadata architecture that supports access, discovery and management of an information resource in all its manifestations, from analog or born digital source objects to digital technical masters and access copies. local customization includes authentication and authorization for managers and metadata creators; the ability to customize and add vocabularies to data elements; and the capability to customize the look and feel of the metadata input and to add default values to data elements. objects are organized within collections and sub-collections. hierarchical organization provides intrinsic relationships among objects (such as articles within the issue of a journal) and enables objects to “inherit” collection level information via templates, such as a rights event for the collection level deed of gift. this ensures that any collection level context is always readily available at the individual object level. the architecture of the workflow management system architecturally, the wms is a container that includes many functional modules. all wms modules follow the three-tiered design pattern that separates the user interface and persistent data storage from the business logic. figure 6: architecture of the wms as figure 6 demonstrates, the wms can be conceptually divided into three layers – the context-driven user interface layer, workflow policy/business logic handling layer, and persistent data storage layer. wms database design the internal persistent data storage includes a relational database and file system. this layer does not directly deal with workflow logic, but its design, especially the schema design for the relational database, affects the data integrity, the system performance, and the overall ability for the system to function. figure 7 wms database design the wms relational database schema is built upon the functionality class of the data. as figure 7 demonstrates, there are 5 major functional blocks in the schema: wms collection management, digital file handling, metadata and template storage, metadata schema builder, as well as controlled vocabulary management. in addition to the standard relational database design considerations, 2 important decisions were made for the wms database structure design. one is that digital object metadata is treated as an xml document as a whole instead of as a hierarchical data set, with a separate searching table for selected indexed terms. this approach ensures that the wms database structure does not depend on any specific metadata schema; increases the overall system performance; and makes it much easier to port existing data to xml-enabled database systems to utilize their xml searching capabilities. the second decision concerned the schema builder. the schema builder block stores metadata input form layout information as well as the associated xml structure for each metadata element. this method significantly reduces the amount of work needed for web form maintenance, and provides the project manager with some capability to build and edit the metadata schema from a web input form. workflow modules the workflow logic layer is where all the workflow policy and business logic handling take place. this layer can contain as many functional plug-in modules as needed. currently, the following functional modules have been developed: authentication/authorization collection/project/user management digital resource handling metadata cataloging utility metadata schema mapping utility digital object batch import/export utility external digital object access module each module operates independently from one another, but they share the following common design patterns: object-oriented software design. though the wms is written in php, the design of the software uses an object-oriented approach. object/relational mapping is used as the solution for data access. database driven application. a relational database structure forms the foundation for all wms modules. many of the modules, especially the metadata cataloging utility, heavily depend on information retrieved from the database for data processing and form rendering. xml as both data model and data carrier. the wms uses xml to model internal data as well as exchange data with external resources. php session handling. the session matrix is used for all modules for keeping track of state information. the metadata cataloging utility is one of the core modules and is the most heavily used module in the wms. currently the metadata entry forms use a mets compliant element structure. the descriptive metadata is built upon mods and the technical metadata utilizes data elements from diverse schemas such as premis [7], mix [8], and the forthcoming aes-x098 metadata standard for audio resources. [9] the wms can be easily changed to fit other metadata schemas through the use of the schema builder configuration utility. the wms was intentionally designed to be schema independent and thus to support all metadata schemas. this is important because metadata is still an emerging area and schemas to support different communities and types of resources continue to proliferate. metadata entered are stored using the internal xml data model, and conversions between different metadata schemas are done through a custom-built xslt script. this design pattern makes the cataloging utility flexible and efficient for both utility users and software developers. the mapping architecture for both data import and data export is key to supporting interoperability with other communities, through data sharing using the oai-pmh (open archives initiative-protocol for metadata harvesting) [10] or federated searching. digital resource handling deals mainly with digital file formats and conversion between them. wms was initially tied to the rul digital file handling policies, which require tiff as the archival master for images, and djvu, pdf, and jpeg for presentation formats, for example. djvu, which is a very functional but proprietary and not widely used digital file format, served as the intermediary format for the conversion of digital master files to multiple presentation formats, particularly jpeg and pdf. rul is in transition for image presentation formats as we integrate adobe pdf server into file handling, particularly to support faculty deposits and etds, and as we evaluate migrating from tiff to jpeg2000 as the rul master format for digital images. the goal now is to abstract this file transcoding capability and enable the individual institution to implement file handling capabilities specific to its needs. the metadata schema mapping/import utility allows batch import of metadata and digital resources into the wms system from an existing external database or files. conversion between schemas is achieved with xslt scripts custom developed for each specific schema. the import utility currently supports mapping and conversion from plain text bibliographic records to mods, dublin core, and the rul metadata schema used in wms. pbcore and mpeg-7 are currently in development, particularly to support the mic bibliographic utility. there are no limits to the number of mappings that can be supported, other than the limits on the cataloger time and effort needed to develop the mappings. a metadata mapping facility developed for the mic project enables any participant in the mic union catalog to input the data elements from the institution’s unique schema, map data elements to mic data elements, test the mapping through a sample record load and then batch load their metadata. the mapping is stored for the organization’s future use. a marc mapping exists as a standard map. this mapping functionality was critical for mic because many moving image archives utilize their own custom metadata schema. this mapping utility will migrate to a future version of wms to enable the wms owner to map metadata that doesn’t conform to a schema recognized by the wms. this is particularly critical for supporting university faculty, who often create their own metadata implementations to provide access to their research. the export module converts the metadata and digital resources in the wms into a user specified format, then stores as files or exports to an external digital object management system, such as fedora. currently the wms exports in mets and fedora object xml (foxml) format. marc is under development. for the software developer, the key to maintaining the flexibility of the wms is to follow the wms modular design pattern. using the common module libraries provided with the wms core release, additional modules could be easily plugged into the system, for example, the fedora object editing module developed for the rutgers university rucore repository, which provides for metadata or object editing after the object and metadata have been ingested into fedora. user interface the aim for the wms user interface design is to provide a simple and logical workflow pattern that a cataloger or someone uploading digital objects can easily follow. at the same time, the interface code needs to be easily maintained and modified, particularly since requests for changes in the wms have mostly involved the user interface. several measures have been taken to fulfill this goal: wms interface is dynamic and context driven. a user can configure the application for what he/she needs. depending on the user’s choice during the working process, the wms only displays sufficient information for him/her to accomplish the task. template for metadata cataloging. the template can be easily created, edited, enabled, or disabled, at the collection, project, or personal level. context driven help is provided for metadata cataloging. wms forms are designed to be clean and simple. distracting elements, such as over-decorating,audio, flash video, etc., are avoided. wms user interface html code is dynamically generated using a handful of code templates, based on the business logic provided to the utility. wms system requirements the wms is written in php, and it depends on a relational dbms to function. the wms runs on php 4 and above, and theoretically any rdbms, commercial or open source, can be used with wms. however, considering the needs of open source software users, we focused on testing the open source rdbms only. currently it has been tested to run without problems on the 2 most popular open source rdbms, mysql (4.0 or above) and postgresql (6.0 or above). wms can be run on unix, linux, or microsoft windows system with any web server, as long as it is configured to support php. re-engineering the wms to support open source open source or free software? the two terms are sometimes used interchangeably, but the difference between the two concepts is significant: for both open source and free software, you can download, install it on your system, and run it without a fee. however, for open source software, you can also read the source code and modify it to suit your needs. you can also build upon the code to create new applications, as long as you abide by any licensing or copyright restrictions. from the software development point of view, providing open source software to the public implies fundamental changes in design decisions compared to creating proprietary institutional utilities, simply because the user base and their needs become much larger and unpredictable. we can no longer expect detailed requirement specifications and quick, convenient interactions with the potential users. things that seemed natural to one institution or organization, such as institutional policies that usually are embedded in the proprietary software, become problems to others. the commercial software that would have naturally been used in a proprietary utility for one institution may not be available to others. the operating systems and supporting software for the utility that worked fine with one institution could be unavailable or out-of-date for others. open source means developers all over the world can modify or contribute to the code. this will exponentially increase the code maintenance and documentation issues and tasks. therefore, moving from a proprietary institutional utility to open source software doesn’t mean simply making the utility available for anyone to download. it is a whole new concept and usually means the software needs total redesign and development. wms was a typical proprietary institutional utility when the project was started. in order to convert the utility to open source software, wms has undergone a total redesign since the beginning of 2006. the design philosophy is that the wms needs to be flexible enough so that it: does not depend on any specific external digital object management system (fedora, dspace, etc.) does not depend on the policies of any specific institution does not depend on any commercial software product the design philosophy must also ensure that it is easy for individual institution to add functional modules and to customize for their needs, while being easy to maintain and support. the redesigned wms is no longer a utility tightly coupled with rutgers university libraries, it is now an open source digital object workflow management framework. under this framework, the wms development team provides core modules for wms to perform basic functionalities and code libraries for partners to use for extending the capabilities of the wms. the core modules conform closely to this design philosophy. they are highly modularized and can be reassembled as needed. the tasks for which each institution or organization will most likely have its own policies, or want to have its own implementations, are moved from hard-coded software implementation to software-supported configuration decisions. the handling of digital resource files gives a good example of this scenario. some institutions want tiff to be the archival master file format for images while others might prefer jpeg2000. the wms no longer forces people to use tiff but instead hands over the decision to each individual institution, allowing each institution to decide what format to support and what software to use to create each format. with respect to metadata, the metadata schema builder module is an attempt to decouple the utility from any specific metadata standard. creating a one-for-all metadata converter is very difficult, and while we are working toward this capability, we haven’t yet reached it. however, creating a utility that enables users to significantly modify the existing schema and associated html form input fields has proven to be totally feasible. documentation is a big issue. there are 2 kinds of documentation related to a piece of software: documentation for the user of the software (e.g., a user manual), and documentation for the developer or potential developer/supporter of the software. for open source software, both kinds of documentation are very important. rul’s current user manual includes a full data dictionary for every data element in the mets metadata architecture and a tour of the wms, demonstrating both navigation and the purposes and use of each mets metadata document. the user manual reflects a much earlier version of the metadata, which predates the rights metadata schema and the use of descriptive events in the descriptive metadata. the user manual will need to be completely rewritten before the wms software is released. the rul metadata architecture is based both on mets and an event structure in each mets document to capture the lifecycle of the resource in many dimensions. these are somewhat novel concepts for catalogers, who have mostly focused on the descriptive information captured in marc or mods. our experience has been that the training and support for initial metadata users within the new jersey digital highway has been significant, involving hands-on training and considerable remote support. training tools that encompass the user manual and beyond need to be produced before the wms is offered as open source. this is not an easy task and is the subject of much discussion at present. the documentation for the software architectural and coding details needs equal attention because of the need for developers to understand the code to add modules or customize options for each organization. this task inevitably falls exclusively on the software architect and developers. the wms open source process the rutgers university libraries are moving cautiously into the open source environment with the workflow management system, primarily due to the sophistication and complexity of the metadata architecture and the modular, customizable design. changes to the wms are incorporated in rucore versions, which are specified and implemented at least twice and sometimes three times per year. while versions are well documented for programmers and stringently tested, user documentation lags significantly and is generally several versions out-of-date. this has not been a real problem to date because the rutgers staff using the software generally specified, approved and tested the modifications to the wms. in the future, when the number and composition of users is unknown, user documentation will be very important. rul is discussing how to handle documentation requirements when no position with a specific assignment to provide documentation currently exists. rul also currently develops version specifications based on the needs of rutgers users, as articulated by library users, or dictated by grant requirements, and as authorized by the rul cyberinfrastructure steering committee. rul utilizes a modification request process that identifies bugs; either during testing or during routine use, and incorporates fixes for any bugs that don’t actively interfere with standard use into future wms versions. rul is discussing how bugs identified by open source users, as well as recommendations for enhancements (or enhancements created by open source users), can be effectively incorporated into the wms development and testing workflow. initially, rul is exploring the development of a small user community of institutions with similar needs, particularly institutions currently using the fedora repository architecture, who can provide testing and collaborative development for the wms. currently, rul is discussing an open source development collaboration for the wms with northwestern and penn state, all of whom are currently testing both the installation and the functionalities of the open source bibliographic utility based on the wms that will be provided to the library of congress in 2008. future developments for the wms one major development for fy08 is the integration of encoded archival description (ead) finding aids into the wms. this integration will support both ingest and export of metadata as an ead finding aid with associated digital objects. ead support is critical because the special collections and archives of the rutgers university libraries base their description and access practices exclusively on finding aids. in addition, a document management capability will be added to enable users to automatically store and associate relevant pdf documents with events, such as a deed of gift with a rights event or a bill of sale with an accession event, within an administrative documents section of the repository. rutgers university libraries are also recipients, with william paterson university and njedge, the statewide internet2 utility, of an institute of museum and library services grant to build a statewide digital video network, njvid. rul will be extending the wms mets structure map to enable faculty and k12 educators to easily segment and annotate videos via a simple web form. rul will also use xacml to place constraints on object use, again based on a simple web form completed by faculty, to enable faculty to reserve video course lectures to users within a course. njvid will also implement a statewide shibboleth facility that provides centralized shibboleth services to all participating organizations, regardless of technical readiness for shibboleth support. digital video functionalities will appear over the course of the grant, from fy08-fy10. conclusion there are a number of open source bibliographic utilities that support the creation and organization of digital information for libraries and archives. we feel that the wms makes several unique contributions, including the event-based data model; a complete mets implementation with fully functional mets documents for description, source, technical and rights metadata; and xml-based customization capabilities that enable users to tailor metadata to their needs while still supporting metadata standards and interoperability. rul has also extended and enhanced technical and source metadata for digital multimedia, to support the needs of the moving image archives community. we feel the wms will be a strong option particularly for organizations that want to document both analog source objects as well as digital surrogates and that want to document and manage resources with copyright and other digital rights issues. we also feel it is a strong utility for organizations wanting to ensure interoperability with other initiatives, even as they customize metadata to meet local needs. we are addressing the remaining issues, that are primarily organizational and policy driven, to enable release of the wms in open source in early 2008. notes [1] “presentations given about rucore” development of rucore. accessed 1 november 2007. http://rucore.libraries.rutgers.edu/collab/index.php [2] fedora commons. accessed 14 october 2007. http://www.fedora-commons.org/ [3] new jersey digital highway. last updated 24 september 2007. accessed 13 october 2007. http://www.njdigitalhighway.org/ [4] library of congress. mets-an overview and tutorial. 13 september 2006. accessed i november 2007. http://www.loc.gov/standards/mets/metsoverview.v2.html [5] rucore-rutgers community repository. accessed 1 november 2007. http://rucore.libraries.rutgers.edu/ [6] library of congress. mic-moving image collections. last updated 25 september 2007. accessed 13 october 2007. http://mic.loc.gov/ [7] library of congress. premis-preservation metadata maintenance activity: official web site. last updated 31 july 2007. accessed 13 october 2007. http://www.loc.gov/standards/premis/ [8] library of congress. mix-niso metadata for images in xml schema: official web site. last updated 17 august 2007. accessed 13 october 2007. http://www.loc.gov/standards/mix/ [9] audio engineering society standards committee. october 2003 meeting of sc-03-06. c2007. accessed 13 october 2007. http://www.aes.org/standards/b_reports/b_meeting-reports/aes115-sc-03-06-report.cfm [10] open archives initiative. open archives initiative protocol for metadata harvesting: protocol version 2.0 of 2002-06-14. accessed 13 october 2007. http://www.openarchives.org/oai/openarchivesprotocol.html about the authors grace agnew, associate university librarian for digital library systems and wms metadata designer, rutgers university libraries ( gagnew at rci dot rutgers dot edu ) yang yu, database architect and wms architect, rutgers university libraries ( yangyu at rci dot rutgers dot edu ) subscribe to comments: for this article | for all articles one response to "the rutgers workflow management system: migrating a digital object management utility to open source" please leave a response below: the rutgers workflow management system: migrating a digital object management utility to open source | library workflow exchange, 2015-06-16 […] http://journal.code4lib.org/articles/25 […] leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – spinning communication to get people excited about technological change mission editorial committee process and structure code4lib issue 41, 2018-08-09 spinning communication to get people excited about technological change many organizations struggle with technological change. often, the challenges faced are due to fear of change from stakeholders within the institution. users grow accustomed to certain user interfaces, to processes associated with a specific system, and they can be frustrated when they have to revisit how they interact with a system, especially one that they use on a daily basis. this article will discuss how to acknowledge the fears associated with technological change and will suggest communication tactics and strategies to ease transitions. specific scenarios and examples from the author’s experiences will be included. by suzanna conrad you’ve probably faced people within your organization who are fearful of change, especially technological change. it can be difficult to convince techno-leery individuals that technological change can be exciting rather than scary. change can take the form of a web redesign, a new integrated library system, virtual servers, morphing roles of library vs. organization it, etc. convincing library stakeholders that change needs to happen can be a daunting task as a systems administrator, web developer, technology consultant, or technology manager. this article will discuss fears associated with technological change from a change management perspective. technology projects will also, within this article, be framed as complex sales or “long sales.” lastly, this article will provide practical examples for modifying communication patterns to get people excited, or at least grudgingly accepting, of change. fear of change fear tends to precede change, regardless of the rationality of that fear. bob sacks calls this “the fear that comes along with change from what was once known and comfortable to what is unknown” and that despite “changing and adapting for a millennium…the pundits of the day try to resist the inevitable” (sacks 2011, 50). christopher durney and richard donnelly acknowledge the challenges of keeping up with change, as “the pace of technological change often outruns and undermines the best project management planning efforts” (2015, 642). durney and donnelly define six strategies that managers can employ to address fear and technological change: (1) identifying risks, (2) defining staff that will be responsible for handling change, (3) using adaptable management styles, (4) focusing on the core business, (5) thinking about how to be resilient, and (6) supporting innovation within the business culture (durney and donnelly 2015, 664). todd jick presents three phases of change management that are important to address: drafting the vision for change, implementing the actual change, and addressing reactions to change (1993). annie bartoli and phillippe hemel discuss the various barriers to change, especially in it management, as separating into strategic, structural, cultural, and behavioral barriers (2004). resistance to change in organizations, especially technology innovation, is something oft discussed in management literature. felicia su wong attributes fear of technology to the fear of missing out: “the organizational and personal experience of technological adoption and use often has been driven by a fear of being left behind in this very process of progress” (2003). yongchuan bao asserts that improving communication within an organization can address organizational resistance to technological innovation (2009). robert reardon finds that learning culture plays a large part in resistance to or acceptance of technological change; reardon defines a strong learning culture as one in which “creating a climate of learning should speed the smooth transition to the new technology” (2010). for an organization to have a strong learning culture, the organization needs to be committed to fostering systematic thinking, be willing to experiment with change, able to learn from past experiences and the experiences of peers, and have the capability of communicating this knowledge throughout the organization (garvin, 1993). without an established learning culture, change is difficult; in reardon’s study of blue-collar workers at manufacturing sites, he found a major correlation between lack of learning culture and disgruntlement (2010). libraries are not anomalies; many of our colleagues may be leery of technological change. they may distrust technology and technologists; and because of these fears they may block innovation or even necessary improvements. but we can frame these projects differently: convincing people to overcome their fear of change can be a gradual, iterative process. change as a complex sale complex sales are sales that require input from multiple decision-makers, often at various tiers of an organization. they are considered to be complex because of the time and monetary investment required to close the deal. think of the library’s business of purchasing ebooks: a simple sale is made when a subject librarian or selector chooses ebooks to purchase in a specific discipline; one book can be purchased without involving others in the final decision making process. on the other hand, selection of an electronic resources management system for managing ebooks requires buy-in from all levels of the organization to implement successfully. input on the system would be provided by personnel in acquisitions, cataloging, systems, management, as well as the subject selectors. selling an electronic resources management system to a library would constitute a complex sale. jeff thull focuses on the idea of “value clarity” in mastering the complex sale. in order to sell a complex service to a company, a salesperson has to be able to connect customers’ needs and potential use cases to the benefits of purchasing a service. in business, it is frequently difficult to close sales due to delayed decisions based on customer uncertainty (thull, 2010). pushing through technological change can be similarly challenging: if customers’ needs are not considered, or the communication is not framed in such a way that they can perceive value, projects can stall to the point that the technology is already outdated. changing the narrative managers of technology projects, or even stakeholders involved in the process, may want to assess how they approach conflict. are you defensive when questioned about a change or do you negotiate with stakeholders to try to find compromises? thomas and kilmann’s conflict mode instrument defines five areas where one can fall on a scale of assertiveness compared to cooperativeness including competing, avoiding, accommodating, collaborating, and compromising (thomas and kilmann 1974). competing is the most assertive of the five areas; a competing person would concentrate on getting their own way with little room for negotiation. accommodating is at the other side of the spectrum; someone who is accommodating allows others to dictate in which direction the project is headed and accommodates others’ wishes. avoiding is the least assertive and cooperative of the five areas; avoiding the conflict means that it is never discussed or broached. collaborating falls at the most assertive and cooperative level; collaborating involves working through the conflict or disagreement to dissect what others’ concerns are in the hopes of finding a new solution that is satisfactory. compromising falls in the center of assertiveness and cooperativeness; it does not involve diving into the details as much as collaborating, but finding wins for each of the parties. collaboration, though it takes the longest, may be the best approach for technologists dealing with stakeholders. by collaborating, both parties can leave with the feeling of accomplishment and involvement in the process, as the solution should theoretically fulfill the needs of everyone. compromising may also be a desirable alternative as the original solution may still be possible after working together to find gains for both parties. in a previous position, i was involved in a major website redesign, which was led by central it. in the beginning of the process, the library and campus it had very different expectations of what headers would look like on the library’s homepage. campus it had designed a header that included a global search box, which would search all campus pages. it was not a simple process to ask campus it to change the template; we had to dig deeper and provide feedback about the role of search on the library homepage. specifically, we presented them with search log analysis, usability testing results, and web statistics to make a case for a library-centric search box. by the end of the process, they had a greater understanding of the library’s role and mission. this was a great example of compromising; it took multiple meetings, data collection, and cordial discussions for campus it to understand why we were so adamantly against prominent positioning of the campus’ search box on the library webpages. similarly, throughout this same process, we made concessions. we received a special template for the library, but agreed to incorporate the campus footer as well as certain elements of the campus header. thinking about how you approach conflict is just one piece of changing how you initiate innovation. positive communication is at the core of changing the narrative. there are tactics that one can use to try to minimize fear, impatience, or distrust. four examples include: (1) delivering regular updates; (2) soliciting feedback; (3) remaining honest; and (4) communicating bad news with potential wins. first, delivering regular updates can make your stakeholders feel informed and included; when someone is not receiving communication regularly enough, it can lead to frustration, anger, and can endanger the success of the project. sometimes, it’s even worth checking in when there is no update to let stakeholders know why nothing new has developed and give estimated timelines for when more news can be expected. secondly, soliciting feedback is important. few people will willingly accept what is perceived as difficult change if they do not feel that they have been consulted. many technologists in libraries work in shared governance environments; you especially have to leave room for consultation in these situations. sometimes you can’t make the changes that someone wants or deliver the product they would prefer, however, if they were consulted and informed, it at least starts a positive dialogue and signals that you care about their input. also, be willing to take and accept the feedback that is given to you. your stakeholders have input based on their daily interactions with a system or with the public. you, as the technologist, will not always have all the details. feedback from stakeholders and users can help with success during implementation even if it does not align with what your expectations may be. honesty is important throughout technological change, or any change. perhaps a change is happening because you have a mandate, you want to improve a service, or because you have more (or less) funding. change can be spun to be positive, especially if you are transparent and forthcoming. exercising empathy can help conversations; if you think about what that particular stakeholder is frustrated about, you might be able to work out alternatives, better solutions, or help with training. lastly, sometimes there is bad news. an interface may be changing for the worse, or a task might be harder to do in a new system. in the case that these changes are inevitable and include bad news, drawing out potential wins to deliver along with the bad news can help soften the impact. for instance, you’re moving to office365 cloud services and people are upset that the webmail interface will change to something they do not like. telling them about increased storage space, then the bad news that the interface will change, and then letting them know they can download the clients on up to five machines may help ease the discomfort with a new user interface. general communication tactics when people feel as though their feedback is being respected, they are more likely to welcome change. certain communication tactics can help you to ease technology-leery individuals into embracing or at least accepting change. these are simple, logical tactics, which might take practice to implement, but will help in guiding you through difficult conversations. first, be careful about how you are reacting to discussions. listen without any disruptions. do not interrupt as someone is speaking – give them a chance to air their concerns. also, is your body language positive? if you’re shaking your head “no” as someone is speaking, rolling your eyes, or pursing your lips, the discussion may escalate into conflict. if you have devices, such as a cell phone or computer, do not glance at your devices unless you are referencing something related to the discussion. the words you use are important, especially when dealing with a situation that is or could become a conflict. using “i” statements instead of “you” statements is good practice. the statements “you’re not listening to me” vs. “i feel as though i’m not being heard” can be perceived very differently. if you avoid using the word “you” in conversations, you may find that the language you’re using is less confrontational. email communication is especially tricky as the written word is easy to misinterpret. the shorter and sweeter your messages in potential conflict situations, the less likely the situation will escalate. three tips that can help with reducing conflict that could start via email are: (1) be brief; (2) state facts and provide resolutions; (3) avoid emotional language. brevity means that less is more; try to respond to an email with two sentences if possible. the longer the email, the more likely you are to write something that could be misinterpreted. stating facts and resolutions allows you to acknowledge a problem and provide a solution rather than to delve into the why or how. lastly, avoid emotional language. third person phrasing without the word “you” may help as mentioned in the above paragraph. avoid the use of adverbs as they add emotion. these can include simple words such as “very” and “just” and most any word ending with an “ly,” such as “easily” or “unbelievably.” here are some examples of written text that could be perceived as emotional by the reader: “i’m sorry you’re having difficulties with the system.” while this sentence may attempt to employ empathy, it could be perceived as judgmental. this is accentuated by the use of the word “you.” using the word “you” could be read as passing blame to the person you are writing to, especially when used in close proximity to the word “difficulties.” a better phrasing might be: “i apologize for the issues the system is experiencing.” here the only blame that could be attributed is to the “system.” often, it is possible to pull the “you” out of a sentence by framing it as an “i” question as in this example. “the system was only down for ten minutes.” in this sentence, “only” implies that ten minutes is not a long time. if a reference librarian is answering a patron’s question and that patron has limited time, ten minutes can be a long time. taking the “only” out of the sentence turns the sentence into a statement of fact: “the system was down for ten minutes.” “i’m just trying to resolve the issue.” the word “just” in this sentence is unnecessary and it gives it emotional weight that could be misinterpreted as frustration or anger. simply stating “i’m working to resolve the issue” leaves less room for emotional interpretation. it is easy to think your email is not emotional, but it could be perceived as such regardless. if you are not sure whether your email sounds emotional, let someone else read it and see how they interpret it. or, if the the content is confidential or sensitive, read it aloud to yourself – you may hear the tone differently when you read out loud. scenarios scenario 1: campus or city website redesign you receive news that the campus or city will be redesigning their website and they expect the library to take the global template. additionally, you are informed that you will have to use the campus or city’s content management system, which you have heard is clunky and requires extra steps for publishing. frequently, libraries’ parent institutions group the work that we do into the chunks similar to the general business of the organization. for instance, an academic library may be perceived as an academic college. arguably, the two organizations do very different work. an academic college’s website is more informational and geared toward sharing details about departments, degree programs, featuring faculty, listing available courses, etc. the search function is less crucial than it is to an academic library’s website. all is not lost though. a message such as the scenario presented here could be an opportunity to rethink the website and how it fits into the mission of the organization. a communication channel can also be opened in which control issues and concerns can be expressed with it or whichever group is initiating the change. additionally, opening a communication channel is also an opportunity to open discussions with it about web services, which could benefit the library in other ways. as previously mentioned, i participated in this scenario in a campus environment and initially had to improve information sharing to facilitate the success of the project. after initial hurdles with sharing library perspectives, our relationship with campus it improved as the web redesign progressed. campus it was migrating most of the websites for the campus and once they realized we had a capable web developer, they gave us some additional permissions in the new content management system to allow us to do our own migration for the most part. more advanced template or technical issues had to be escalated to their staff, however, we developed direct working relationships with contacts we had not had in the past. after the migration was completed, our interactions with this department continued. if we had ideas for new web-based services , such as a faculty profile system or an improved group study reservation system, we often would check in with the web services department first, as they took the time to look at our ideas and could provide internal pathways to other it departments that would help with implementation. scenario 2: consortial system implementation you are a part of a larger consortium of libraries. you hear that the consortium is planning on centralizing the library management system. it is unlikely that it will end up being the system you currently use. you are concerned about losing control and having to learn everything new again. consortial systems certainly offer benefits in cost and functionality that you might not experience with a standalone instance. moving these systems to consortial models is also becoming more and more common. shared systems may lead to staff time savings, shared integration of 3rd party software, monetary savings, etc. there are simple ways to talk about the bigger goals with this kind of move. perhaps certain tasks will be easier in the new system. maybe the system will help reduce mundane tasks and give you the chance to work on more exciting projects. lastly, and perhaps most importantly, you may be able to develop partnerships with institutions and individuals you have never spoken to before. you will have a network of individuals with whom you can interact to make your use of the system more skillful and efficient. in fall 2015, the california state university’s libraries embarked upon the unified library management system (ulms) project to consolidate all library management and discovery systems across the 23 csu campuses into one shared instance. discussions about consolidating had begun over two years prior in spring 2013. two csu libraries were already using the chosen software, alma and primo from ex libris, however, many campuses were using integrated library systems from other vendors. i worked at two csu campuses during the migration period; first, at one that was migrating from innovative’s sierra and then at one already on alma and primo. as these discussions were developing at a higher level, staff at the campus using sierra were panicking. we had only recently migrated to sierra with some hiccups in the process. staff were concerned about major changes to workflow, changes in functionality of the system, and the long-term viability of their jobs as the system was marketed as a way to reduce redundant work across the consortium. because of these fears, our library organized general vendor meetings with all of the vendors who were intending on submitting requests for proposals. within these meetings, it became clear that certain functions would be much easier in alma and highly desired features, especially with analytics, would be possible. a few staff began to see that it could benefit them to move to the new system. internally, we continued to discuss the project as a positive one; the attitudes of the people communicating the change can make all the difference in the success of a project. i moved to another csu campus midway through the migration process. this institution was already on alma and primo. while we did not have as much to prepare for migration as a campus on sierra or voyager, our first migration had not gone as well as we had hoped, so we planned many clean up projects to make the second migration more successful. there was less anxiety about the functionality of the system, although we were more curious about the impact of some of the consortial gains as we had been using the system for a few years before moving to the shared instance. many of the subsequent wins from the second migration were first realized well after migration; gains in improved processes in technical services have been allowing staff to work on other pending projects. the landscape of resource sharing has changed as we share among our institutions. not everything is functioning perfectly in the new shared instance, nevertheless, communication amongst all campuses has improved. we have listservs and slack channels. staff who did not know one another before are connecting and sharing best practices. our institution has had an opportunity to share our best practices, thereby featuring the work and successes of our staff. scenario 3: it is centralizing you receive a message that it is centralizing and will be claiming many of the it staff in the library as well as servers housed in the library. everyone is panicking about how support will suffer and that the library’s priorities will now be listed with everyone else’s priorities. for many of us with local it staff, such as desktop support, web developers, server administrators, and programmers, it can be difficult when staff are pulled away from the library to manage systems and projects for the whole organization. however, centralizing services may help improve service or security in some cases. for instance, recruiting a good server administrator can be difficult; if every department in an organization would need to hire their own server administrator, they might not find enough staff locally. you might also need to hire a server administrator who would handle additional tasks such as web development, application management, etc. a centrally located server administrator in the city or the campus staff could, with proper communication channels, be more effective than dispersed individuals across the institution as efforts could be concentrated on this one area. furthermore, this message that it is centralizing is an opportunity to open communication with it so that they understand the needs and requirements of the library. if communication is initiated, negotiation might even be possible. perhaps there are ways to keep some of the staffing or services depending on expertise levels in the library. or it is possible that with centralization, you have access to a larger network of it support. i have not been part of a centralization where staff are moved out of the library into it, however, i have joined a library shortly after this happened. at my previous institution, there had been four it staff members in the library, two of whom managed desktop computers. the other two staff managed the integrated library system and the library website. when centralization occurred, the library lost both staff members assigned to desktop support, but kept the web developer and ils manager. the library was able to make a case that those two functions were core to library operation. one of the staff members who was moved was still assigned to the library as our main desktop support person for the first year or two, so the transition was easier, as we were familiar with him. after two or so years, it assigned a team of desktop support technicians to the library to manage our labs, classrooms, and staff desktop support. while the transition was not always smooth, we established new connections within this department of campus it. it did, at times, take longer to get a response, but generally we had a network of individuals to contact, so if someone was out of the office, a knowledgeable replacement could be sent quickly. the director of the program was involved in a number of technology spending projects on campus, and due to our new integrated model, he confirmed that usage in the library was higher than other areas of campus. therefore, computer refreshes in the library building were prioritized very quickly. we also had discussions with this director about purchasing computer usage statistics software that would allow us to build availability maps and similar tools for student convenience. scenario 4: moving to a new discovery layer your library has decided to move to a new discovery layer. many are displeased with the news, as they have been using the existing catalog or discovery layer for a number of years. instructional guides and tutorials will need to be changed. users will have to be re-educated about the change. normally, there is a reason for moving to a new discovery layer. one reason may be evolving user needs; many users expect a comprehensive search and are not interested in searching in multiple silos of content. there may be a cost savings to adopting a new discovery layer that could allow for spending in other areas. if the discovery layer is being adopted because of changing user expectations, discussions with stakeholders about ease of use for the patrons is an initial topic to broach. if you have usage data, usability studies, or anything to support these arguments, this would be helpful. transparency in cost savings, if possible, may help to illustrate good reasons for moving to a different discovery layer. if there are hesitations due to the limitations of the new discovery layer, presenting better features not available in the existing system may be a starting point. it may also help to discuss the vendors’ enhancement timelines, to see when and if certain features will be usable. within a few months of getting my first tenure-track librarian job, the consortia i was working in purchased a new discovery layer and our campus opted in for purchase. in efforts to prepare to move to the discovery layer, i served on a local task force investigating the change. i offered to do a survey of student users to gauge their reactions to the new discovery layer. i had nearly 600 responses to the survey with 99% of the responses coming from undergraduate and graduate students. the response to the discovery layer was overwhelmingly positive. we had been separating search results between the library catalog results in a millennium opac and a metalib search of articles. the new discovery layer combined these indices. i summarized the results of the survey, showed a demo to librarians, and we scheduled a launch. in this instance, this could have been handled more effectively. i launched at a less than optimal time during the semester, as i was not familiar with the teaching schedule of many of our librarians. i also was too green to understand the importance of delivering a message multiple times and letting some discussion or contemplation occur. many of our librarians were dissatisfied with the implementation of the discovery layer. over time, many adapted and began to use the discovery layer more consistently, but this situation taught me to take the time to consult with stakeholders and to provide adequate feedback loops. sharing of data and decisions did not need to happen within a short period of time, but should have been spaced to allow all to feel consulted and included. scenario 5: launching a new website your systems department is working diligently on a website relaunch, as the template you are using is many years old. content has become unmanageable and many potentially unnecessary webpages exist within the site structure. your staff is hesitant to make this change as printouts, instructional materials, and other printed materials will have to be updated. furthermore, they are wary about learning the new interface as they are comfortable with the existing site’s functions and content. many libraries struggle with implementing a new library website. staff and patrons are often attached to user interfaces they are familiar with and push back when changes occur. it may be necessary to move to a new website because of a dated infrastructure for the website, or perhaps your library wants to have a more modern look and feel. involving stakeholders in the feedback process when you are designing a website can forewarn them to the change and help ease the transition. this may entail organizing surveys, focus groups, or sharing plans at various points of the process. planning for support and training can also reduce anxiety about change. i have been involved in a number of website relaunches, both in library and corporate settings. the implementations that have been the most effective were those in which feedback loops were built into the project timeline. most recently, i participated in a complete relaunch of a library website where all prior content was discarded and new content was built from scratch. a consultant was hired to administer focus groups, provide a human centered design report, and build templates based on the data. we held focus groups with both staff and patrons. we made transcripts and recordings of these sessions available to library staff for review. analytics reports were reviewed to discuss the information architecture of the site. a website update team was formed including representatives from all departments. after feedback was collected, the consultant compiled results into a report, began creating mockups, and defined the information architecture for the site. we built feedback loops into all stages of the process: whenever a new template or new deliverable was received from the consultant, it was reviewed by the website update team and library administration. all departments crafted text for the webpages affecting their areas; the website update team reviewed these texts to try to align the voice across the website. a library-wide meeting was scheduled once templates for the homepage and subsequent pages were developed; the consultant presented the template and our team talked about launch dates and implementation plans. we left time for questions and also fielded questions via email or phone after the presentation. in the weeks leading up to the launch, we publicized the upcoming change across campus and kept library faculty and staff informed about progress. overall, the launch went successfully. we had some parts of the website that were not ready for production and were launched subsequently. as analytics were collected over the following year, we shared out the results of the changes. i believe that this project ran more smoothly because all of the opportunities for feedback that were included in this process. conclusion approaching change with an attitude of fear or trepidation will almost always make the change more difficult. as a technologist, you can prepare others for change with good communication skills and reassurances. it is rare that a technology will only have negative implications; normally the silver lining can be represented in some way. thinking of change as a “complex sale” that has to be chipped away at for success is a first starting point to implementing change. how you communicate that change, whether verbally or written, can also define and determine the ultimate success of adoption. thinking through the perspective of the user by using empathy can also help your project succeed. if you understand and acknowledge the insecurities of those you are supporting with the new technology, it may make it easier to communicate with them and get them on board. references yongchuan, b. 2009. organizational resistance to performance-enhancing technological innovations: a motivation-threat-ability framework. journal of business & industrial marketing. 24(2): 119-130. bartoli, a., hemel, p. 2004. managing change and innovation in it implementation. journal of manufacturing technology management. 15(5): 416-425. durney, c.p., donnelly, r.g. 2015. managing the effects of rapid technological change on complex information technology projects. journal of the knowledge economy. 6(4): 641-664. garvin, d.a. 1993. building a learning organization. harvard business review. 71(4):78-91. jick, t.d. 1993. managing change: case and concept. columbus (oh): the mcgraw-hill companies. reardon, r.f. 2010. the impact of learning culture on worker response to new technology. journal of workplace learning. 22(4): 201-211. sacks, r.m. 2011. fear and survival. publishing executive. 26(2):50. song, f.w. 2003. being left behind: the discourse of fear in technological change. hedgehog review. 5(3): 26-42. thomas, k.w, kilmann, r.h. 1974. the thomas-kilmann conflict mode instrument. tuxedo (ny): xicom, inc. thull, j. 2010. mastering the complex sale: how to compete and win when the stakes are high. 2nd ed. hoboken (nj): john wiley & sons. acknowledgements thanks to dean yaukey of phantom entertainment for the helpful tips regarding communication brevity and emotional language when writing emails. thanks to carolyn thomas, former library director at sierra madre public library for her tips about providing facts and resolutions in conflict situations. about the author suzanna conrad is associate dean for digital technologies and resource management in the university library at california state university, sacramento. her research interests include human-computer interaction, scholarly communication and ethics in librarianship. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – building and maintaining metadata aggregation workflows using apache airflow mission editorial committee process and structure code4lib issue 52, 2021-09-22 building and maintaining metadata aggregation workflows using apache airflow pa digital is a pennsylvania network that serves as the state’s service hub for the digital public library of america (dpla). the group developed a homegrown aggregation system in 2014, used to harvest digital collection records from contributing institutions, validate and transform their metadata, and deliver aggregated records to the dpla. since our initial launch, pa digital has expanded significantly, harvesting from an increasing number of contributors with a variety of repository systems. with each new system, our highly customized aggregator software became more complex and difficult to maintain. by 2018, pa digital staff had determined that a new solution was needed. from 2019 to 2021, a cross-functional team implemented a more flexible and scalable approach to metadata aggregation for pa digital, using apache airflow for workflow management and solr/blacklight for internal metadata review. in this article, we will outline how we use this group of applications and the new workflows adopted, which afford our metadata specialists more autonomy to contribute directly to the ongoing development of the aggregator. we will discuss how this work fits into our broader sustainability planning as a network and how the team leveraged shared expertise to build a more stable approach to maintenance. by leanne finnigan and emily toner acknowledgements while acknowledgements often appear as a footnote, the work discussed in this article was highly collaborative, and we as co-authors would not have this infrastructure to describe without the invaluable contributions of coworkers, past and present, at temple university libraries and pa digital. we feel it is imperative to begin first by elevating and acknowledging their labor. this includes their involvement in the multiple phases of project planning and management, the design and implementation of technical infrastructure, the iterative development of automated workflows and validations/transformations, and ongoing maintenance. between 2018 and 2021, the following people contributed directly to this work: jennifer anton, rachel appel, timothy bieniosek, leanne finnigan, christina harlow, david kinzer, chad nelson, steven ng, stefanie ramsay, holly tomren, and emily toner. introduction launched in 2016, pa digital is a network that serves as pennsylvania’s service hub for the digital public library of america (dpla). over the last seven years, the development and operations of pa digital have been funded through library services and technology act (lsta) grants from the institute of museum and library services as administered by the pennsylvania department of education through the office of commonwealth libraries and the commonwealth of pennsylvania. with these grants, pa digital contracted temple university libraries staff to support the initiative by providing project management, technical and metadata expertise, and digital infrastructure. one of pa digital’s primary services is harvesting and aggregating metadata records from the digital collections of contributing institutions throughout pennsylvania, including libraries, historical societies, museums, and other regional partners. as part of this work, we offer ongoing metadata support, both transforming harvested records for greater consistency according to our metadata guidelines and thoroughly reviewing metadata to provide feedback and guidance to our contributors. once processed and finalized, aggregated records are then delivered to the dpla on a quarterly schedule. when the project first began, the pa digital team developed a prototype to support oai-pmh aggregation workflows using a customized hydra-based solution (dplah). from there, pa digital expanded rapidly. our total number of contributors increased from 19 at its launch to 84 by 2019 and, over time, required processes for a growing range of digital repositories, including bepress, contentdm, islandora, omeka, and samvera. the team also needed the ability to import csv files for contributors with technical constraints. modifications to these workflows and general troubleshooting proved labor-intensive because of inconsistent legacy code. metadata specialists, working to provide metadata support to our contributors, had little to no flexibility over the processing of incoming metadata and generally relied on the team’s developers to make any local adjustments. assessing our needs through the creation of user personas and scenarios, the pa digital team began to plan in 2018-2019 for the implementation of a new aggregation solution. the goals were to build a more flexible metadata aggregator that would afford more autonomy to our metadata specialists and reduce the developer support required to maintain the system. evaluating our options, we looked at existing dpla aggregation applications, including combine from wayne state university (part of the michigan dpla service hub). through extensive trial-and-error, the pa digital team determined that the best direction for our aggregation workflows was to leverage our existing systems and expertise rather than adopting something new, knowing that doing so would likely lead to many of the same maintenance issues as before. in late 2019, christina harlow proposed the idea to develop new processes in apache airflow, a robust application that temple university libraries already used for other projects. the team decided to pursue this plan, which developed iteratively from 2019 to 2021. apache airflow is an open-source workflow management tool that can be used to manage and monitor pipelines. timothy bieniosek and chad nelson at temple university libraries first introduced airflow into its technology portfolio for smaller projects, including the aggregation of small usage datasets. in 2018, our use of airflow expanded to include processes for harvesting and indexing alma catalog records into our custom-built discovery platform, library search, and other automated workflows for our library’s website in 2019. below we will outline how the pa digital team adopted airflow for metadata aggregation and how this new approach supports related metadata reviews, transformations, and validations. pa digital aggregation workflows in airflow one of the core concepts in airflow is a directed acyclic graph, or dag. in airflow’s architecture, a dag defines a group of tasks along with their relationships and dependencies, which can make up a data pipeline. dags are built with python, with the files loaded and then run via airflow. a more extensive overview of airflow dags can be found in apache’s documentation of its architecture and workloads. these dags can be managed and monitored through airflow’s user interface. each instance of an executed dag is known as a dag run. figure 1. snapshot of airflow dashboard, with individual pa digital dags. in order to manage the variety of external data sources aggregated, the pa digital team developed a template to generate a dag for each contributing institution. the tasks within each dag and the functions they execute are largely standardized based on this template. in order to tailor these workflows individually, the project team utilizes the variables configuration in airflow, matching each dag with a set of custom values specific to the contributing institution. these variables are stored in a json file and regularly updated within airflow. each pa digital dag executes a workflow that harvests, validates, transforms, and publishes metadata into solr. this process connects to temple university libraries’ instance of apache solrcloud, a distributed approach to solr indexing. the set of tasks for each pa digital dag include: figure 2. graph view of a pa digital dag in airflow. set collection name (set_collection_name): this first task establishes the name for a solrcloud collection based on the dag identifier. at the end of this dag, the output will include this solrcloud collection. this label is also applied to related files, directories, and logs for each dag run. harvest oai (harvest_oai) or harvest csv (harvest_aggregator_data): by default, the pa digital dag template uses an oai-pmh harvester task. for each dag, we supply oai-pmh parameters as variables, including the url for the specific oai-pmh endpoint, the preferred metadata prefix, and the collection setspec identifier(s), which are used in making requests. individual dags can also be tagged for the alternative harvester task, which imports csv-formatted data from a private github repository instead. in all cases, the records are sent to our cloud storage once harvested. validation tasks for the harvested records, including: harvest filter (harvest_filter): this post-harvest task filters out any invalid records based on a schematron file defined in the variables and generates a csv file with the list of filtered records. harvest schematron report (harvest_schematron_report): this post-harvest task generates an additional csv report, based on a schematron file defined in the variables. harvest field count report (harvest_field_count_report): this post-harvest task generates a report that includes a count of each metadata field and a percentage of records with that field, in order to assess the completeness of the records. this is adapted from a command-line tool developed by mark phillips (phillips 2013). xsl transform (xsl_transform): using a saxon xslt engine, this task transforms the processed records with an xslt file defined in the variables. the records are sent to our cloud storage once transformed. validation tasks for transformed records, including: xsl transform filter (xsl_transform_filter): this post-transform task filters out any invalid records based on a schematron file defined in the variables and generates a csv file with the list of filtered records. xsl transform schematron report (xsl_transform_schematron_report): this post-transform task generates an additional csv report, based on a schematron file defined in the variables. xsl transform field count report (xsl_transform_field_count_report): this post-transform task generates a report that includes a count of each metadata field present across records and a percentage of records with that field, in order to assess the completeness of the records. this works the same way as the harvest field count report task above. refresh solrcloud collection for alias (refresh_sc_collection_for_alias): after the completion of this task, the transformed metadata is indexed in a solrcloud collection. this step deletes any existing solrcloud collection for this dag, creates a new one, and adds it to a solrcloud alias. an alias provides an alternative way for the querying of one or multiple solrcloud collections. depending on the target alias defined in the variables, the solrcloud collection for each dag run will be listed under a “dev” or “prod” alias, both of which include any pa digital collections indexed in other dags. publish (publish): this task indexes the transformed records to the solrcloud collection, which is included under the “dev” or “prod” alias. the raw xml for each record is stored in a single solr field. here is sample of the json variables for an individual dag: "aps_harvest_config": { "endpoint": "http://diglib.amphilsoc.org/oai2", "md_prefix": "oai_dc", "all_sets": false, "excluded_sets": [], "included_sets": [ "graphics_mss.b.p.31.15d", "graphics_mss.b.p165", "graphics_mss.ms.coll.76.17", "text_mss.365.p381p", "text_mss.b.f826", "islandora_graphics_collection" ], "schematron_filter": "validations/ingest_oai_validation.sch", "schematron_report": "validations/padigital_missing_thumbnailurl.sch", "schematron_xsl_filter": "validations/padigital_reqd_fields.sch", "schematron_xsl_report": "validations/padigital_missing_thumbnailurl.sch", "xsl_branch": "main", "xsl_filename": "transforms/islandora_aps.xsl", "xsl_repository": "tulibraries/aggregator_mdx" }, "aps_target_alias_env": "dev", the github repository for the pa digital airflow dags and templates is https://github.com/tulibraries/funcake_dags. in many cases, the tasks utilized for pa digital aggregation are not unique to this initiative and are used across other projects in temple university libraries. for example, the harvest oai task is also used in harvesting and indexing incremental updates of our libraries’ catalog records for discovery. this centralized code can be found at https://github.com/tulibraries/tulflow. in airflow, each dag run is tracked and logged, making it easier to monitor and troubleshoot issues with these workflows and harvested metadata. individual tasks and dependencies within a dag run can be cleared and/or rerun as needed. for example, we can redo the xsl transform and subsequent tasks without having to reharvest from the contributing institution again. this proves extremely useful when we add solrcloud collections to our “prod” alias, which is done by updating the variables and rerunning the “refresh solrcloud collection for alias” task rather than reindexing. blacklight and solrcloud integrations as described above, we use solrcloud to index the transformed metadata and aggregate solrcloud collections together through the use of aliases. once indexed, these collections can be accessed via a dev or prod oai-pmh endpoint, depending on the target alias supplied for the airflow dag. both of these oai-pmh endpoints are set up and configured through the oai provider plugin for blacklight applications. blacklight is an open-source ruby on rails discovery framework with solr searching, maintained through cross-institutional collaboration primarily within the libraries/archives/museums profession. like airflow, temple university libraries uses this open source project in other contexts, most notably for our main discovery platform library search. the pa digital team had prior experience configuring these applications and therefore could easily incorporate this approach into our broader infrastructure and maintenance support. in addition to the validations and reporting built into our airflow tasks, we needed to set up a method for the manual review of transformed records, an important service as part of broader metadata support provided to our contributors. with established expertise using the framework, the pa digital team created funnel cake (named in honor of the pennsylvania dutch delicacy), a lightweight blacklight site used to search and preview pa digital aggregated metadata internally. the github repository for funnel cake is https://github.com/tulibraries/funnel_cake.  the pa digital team also developed a simple dag to index the metadata in the dev alias into a single solrcloud collection of aggregated metadata. this enables the querying, filtering, and displaying of qualified dublin core fields from the aggregated collections in funnel cake. so through funnel cake, two sets of solrcloud collections can be accessed: a solrcloud collection via the blacklight user interface itself and sets of solrcloud collections through the blacklight oai-pmh plugin. once all the relevant collections are added in the prod alias and made available through the production oai-pmh feed, dpla harvests the pa digital aggregated data. metadata transformations and validations, including git integration several transformation and validation scripts are used throughout the tasks above. prior to the implementation of this new infrastructure, metadata processing was hard-coded into our aggregator software, and update permissions were limited only to the development team. during onboarding of new contributing institutions and related metadata review, metadata specialists on the pa digital team were often faced with a choice when metadata did not meet internal requirements (records lacking a title, rights statement, etc.), either: ask the developers to update the code to make the metadata passable (“some of these records are missing rights statements, and the institution, run by volunteers, cannot add them locally. can you update the code so we can add a provided statement to all of the incoming records?”) ask the contributing institution to make changes locally (“the relation field is filled with html tags and broken links. can you unmap it from your oai-pmh feed?”) or, if neither were an option, decide that a collection was too cumbersome to include. eventually, all of the piecemeal code modifications made our original aggregator software too difficult to maintain. despite our best efforts and ongoing testing, an update here would affect a previous update there. the software became a house of cards and felt increasingly fragile. we saw in airflow a more sustainable path forward that better integrated metadata specialists into the development of aggregation workflows and provided a way for them to maintain and incorporate transformations themselves without affecting the rest of the infrastructure. to do this, we opted to maintain a separate github repository, aggregator_mdx, where metadata specialists had full reign over the creation, testing, and maintenance of transformation and validation scripts. once added to github, the paths to these files are defined as airflow variables and incorporated into individual dag workflows as described above. the repository contains four types of xml files: schematron scripts for validations, xslt scripts for metadata transformations, and sample xml metadata and xspec scripts for unit testing that support the integrity of the repository in a continuous integration workflow. christina harlow was instrumental in building this careful infrastructure and providing initial training. prior to this project, our metadata specialists had varying degrees of comfort with xslt and little, if any, direct experience with github and ci/cd workflows. through ongoing applied learning, they now actively contribute to the repository’s development. these scripts, and their use for various tasks in airflow, are described below. the repository’s documentation provides more detail on their implementation and use, including required dependencies and folder structure. validations the following airflow tasks use schematron files for metadata validation: harvest_filter, harvest_schematron_report, xsl_transform_filter, and xsl_transform _schematron_report. schematron uses xpath to test for different elements and patterns within the metadata. the files we use can be found in the validations folder on github. the harvest_filter and harvest_schematron_report tasks run concurrently on harvested metadata before they are transformed. for the harvest_filter task, the schematron files check for the following: the presence of a title element in any namespace the absence of a stopword (‘pdcp_noharvest’) anywhere in the metadata – our old aggregator checked for a stopword, which was added by the data provider to prevent the ingestion of records they wanted us to skip. this was replicated in the harvest filter to accommodate institutions who used that workaround. the absence of the value ‘collection’ in a type field – we do not accept collection-level metadata or finding aids, and we found that filtering records with a ‘collection’ type reduced the likelihood that these would be harvested. any invalid records as defined by the script called by this harvest_filter task do not proceed to the next task. along with a human-readable error note such as, “there must be a title,” invalid records are listed in a csv that posts to our cloud storage where it can easily be passed on to data providers for review and remediation. the schematron scripts called by the harvest_schematron_report task behave similarly, except invalid records do continue on to the next task. this allows us to create a list of records that match certain xpath selections for assessment purposes. for example, if an institution wants to know which of their records lack a thumbnail url, we can code that into the schematron file and provide them with the resulting csv report without preventing records from moving to the next task. the xsl_transform_filter and xsl_transform_schematron_report tasks run concurrently on metadata records after they are transformed and allow us to assess whether the transforms behave as expected. the schematron script associated with the xsl_transform_filter task, similar to harvest_filter, prevents invalid records from being passed to the next task. it checks for the following, which reflect internal and/or dpla requirements: the presence of <dcterms:title> with a non-null value the presence of <dcterms:rights> for textual rights statements or <edm:rights> for rights uris with non-null values the presence of <dcterms:isshownat> with a valid url to the object in the provider’s repository the presence of <edm:dataprovider> with a non-null value the presence of an identifier in <dcterms:identifier> the schematron script associated with the xsl_transform_schematron_report task, similar to  harvest_schematron_report, allows us to assess transformed metadata and create a report of validation errors while allowing such records to proceed to the next task. transformations the xslt scripts used in the xsl_transform task are written to process xml metadata harvested via oai-pmh as well as static xml or csv files harvested from our private github repository. to limit the number of scripts we need to maintain and update, many are linked, making generous use of <xsl:include> to avoid duplicate transform tasks as much as possible while allowing for specialty mappings to support specific repositories, institutions, or collections. the three main xslt script types are: base crosswalk xslt (e.g., transforms/oai_base_crosswalk.xsl): the most general, this crosswalk type is included in all transforms in our repository that process metadata, and define only those mappings we want applied to every record processed unless defined elsewhere. our oai-pmh base crosswalk contains, for example, the mapping of incoming <dc:title> to pa digital’s <dcterms:title>, <dc:creator> to <dcterms:creator>, etc. it also removes extraneous whitespace and delimits on semicolons for all but the following fields where semicolons are often used as punctuation: description, title, publisher, extent, relation, and rights. separate base crosswalks process csv and non-oai xml stored in pa digital’s private data repository (csv_generic_semic.xsl, which delimits on semicolons and csv_generic_pipe.xsl, which delimits on vertical bars). base crosswalks are not modified unless a change is desired for all processed records. if an override of the base crosswalk is needed, it is done in an institution-, repository-, or collection-specific xslt as described below. all base crosswalks import remediation xslts, also described below. institutionrepository-, or collection-specific xslt (e.g., transforms/cdm_temple.xsl): these are the files that run against harvested metadata; their paths are given as variables in airflow. they include mappings to crosswalk metadata whose location or pattern varies depending on repository and institution. by defining substring variables of a particular metadata element, such as the unique url to a digital object found in <dc:identifier>, we can generate other needed metadata not otherwise found in the record. these may include thumbnail preview urls; iiif urls; data provider, intermediate provider, and collection names; and locally generated identifiers. as stated above, if an override of the included base crosswalk is needed, it is done here so as not to affect other records that use the same base crosswalk (see dcterms:ispartof mapping in cdm_temple.xsl for an example of a crosswalk override; the priority attribute should be used to avoid ambiguous rule matching). if necessary, numerous xslt files can be included in the one run against ingested metadata to avoid duplicating transform templates. the pennsylvania state university (penn state), for example, contributes metadata from contentdm and most of their mappings are consistent with other institutions that use that repository. however, penn state maps an object’s original creation date to <dcterms:created>. others use this field for the date of digitization and <dc:date> for the original creation date. to accommodate penn state’s mapping without negatively affecting other records that use the shared cdm_generic.xsl transform, we added another xslt, cdm_pennstate.xsl, that includes the repository-specific transform. since penn state’s dag is the only one that calls cdm_pennstate.xsl, only their records get the additional mapping defined in that transform. remediation-specific xslt (transforms/remediations/*.xsl): these define lookup parameters used by templates in the above xslt types to normalize string values against certain vocabularies and generate metadata not found natively in the incoming xml. this includes the translation of oai-pmh setspec values into human-readable collection names, and digital object urls into a data provider name. for instance, we can define the base url of an object in the field below by selecting a substring, the pattern of which is found consistently in contentdm records (substring-before(.,”cdm/ref/”)): <dc:identifier><a href="http://digital.libraries.psu.edu/cdm/ref/collection/wpamaps/id/3314">http://digital.libraries.psu.edu/cdm/ref/collection/wpamaps/id/3314</a></dc:identifier></ by creating a parameter that identifies the expected base url, we are able to generate normalized data provider names and consistent identifiers while accommodating other institutions that use contentdm. for example, the following element in remediations/lookup.xsl defines penn state’s expected base url as well as attributes for their institution name and code: <padig:url string="pennsylvania state university" code="pennstate">http://digital.libraries.psu.edu/</padig:url> variables defined in the repository xslt are compared to these lookup parameters and, when there is a match, result in normalized values such as those found in the following generated elements for penn state: <edm:dataprovider>pennsylvania state university</edm:dataprovider> <dcterms:identifier>padig:pennstate-wpamaps-3314</dcterms:identifier> testing several features are embedded within aggregator_mdx to support the integrity of our overall infrastructure and metadata quality. the github repository requires that commits receive at least one review and approval, usually by other metadata specialists, before they are merged to the main branch. circleci enables automated unit testing within the repository itself as well as into the merge process. unit tests use xspec to define expected behavior when a specific transform or validation is run against sample xml metadata. we use actual source metadata from our data providers to ensure the utility of the tests. as is seen in tests/xslt/samvera_shi.xspec, the transform is defined as an attribute within the <x:description> element. scenarios include a human readable label describing the test, a path to the sample xml metadata to be tested (<x:context>), and expected results when the transform is applied (<x:expect>). when this test suite is run locally via the command line, all tests in the directory run and the results are available as formatted html reports. when viewed in a browser, scenarios or tests that pass are highlighted in green, while those that did not are in red along with a visual demonstration of the expected vs. actual result. the scenarios, or the transform or validation scripts they test, can then be modified to produce the expected results. there is an xspec add-on for oxygen that is useful to quickly run individual tests, but since a single transform often runs against multiple sample xml files, it is best to run the suite against the whole directory as described in this repository’s documentation. migration, maintenance, and sustainability when we first began using airflow for pa digital in 2019, there were 394,774 metadata records from 926 collections in our old aggregator. rather than attempting an immediate migration that involved reprocessing all the metadata in airflow at once, the team opted instead to do so iteratively over a longer period of time. this work began in december 2019 and concluded in march 2021. we grouped institutions by repository since transformations and validations could more likely be shared, and incrementally reprocessed collections from these groups as airflow dags. this had the added benefit of allowing time for additional quality assessment of metadata as we went. during this interim period, the pa digital team also harvested and indexed static data from our old aggregator endpoint using airflow and created schematron files to filter records from institutions that had been reprocessed. every time an institution was added to the schematron filter, we watched their records disappear from the old endpoint until reprocessing was complete. as of july 2021, the pa digital team has phased out the old aggregator software, an important and cathartic milestone in our project. the old aggregator caused endless headaches because of its inflexibility. the new aggregation infrastructure and workflows adopted by the pa digital team are still quite complex and require significant technical expertise. however, the knowledge and support of these systems is now much more distributed. developers, metadata specialists, and project managers on the pa digital team all actively participate in the development, maintenance, and monitoring of the aggregation workflows at different levels. with airflow’s flexibility, the team can also modify and adapt our processes more easily. likewise, with the use of transformation and validation scripts in github, the metadata specialists on the project can now independently build and maintain customized workflows for individual contributing institutions. this allows us to provide better metadata support to our contributors and partners as well. the set of applications selected (airflow, solrcloud, and blacklight) also align with what is already maintained by the development team at temple university libraries. this choice will help ensure continued technical support of the pa digital workflows as long as that infrastructure is maintained at temple. the completion of the airflow implementation and migration project came at a critical point for pa digital. in 2020-21, pa digital as a network began a planning process to shape the future vision and direction of the initiative beyond our initial grant funding. these planning efforts focus on the long-term sustainability of the initiative, both organizationally and financially. the operational changes made to pa digital as part of the migration to airflow align with those efforts, allowing our team at temple to better maintain the aggregation workflows for the project, and thus improving the technical sustainability of the project for the years ahead. references phillips, mark. 2013. metadata analysis at the command-line. code4lib journal [internet]. [cited 2021 aug 12]; 19. available from: https://journal.code4lib.org/articles/7818. about the authors leanne finnigan is database management librarian at temple university libraries. she was the co-project manager when pa digital launched and now serves as its metadata coordinator. emily toner is technology projects librarian at temple university libraries and currently a co-project manager for pa digital. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – the road to responsive: university of toronto libraries’ journey to a new library catalogue interface mission editorial committee process and structure code4lib issue 23, 2014-01-17 the road to responsive: university of toronto libraries’ journey to a new library catalogue interface with the recent surge in the mobile device market and an ever expanding patron base with increasingly divergent levels of technical ability, the university of toronto libraries embarked on the development of a new catalogue discovery layer to fit the needs of its diverse users. the result: a mobile-friendly, flexible and intuitive web application that brings the full power of a faceted library catalogue to users without compromising quality or performance, employing responsive web design principles. by lisa gayhart, bilal khalid, gordon belray figure 1. sample responsive layouts (enlarge) introduction with the recent surge in the mobile device market and an ever expanding patron base with increasingly divergent levels of technical ability, the university of toronto libraries embarked on the development of a new catalogue discovery layer in early 2012. the goal: provide an app-like experience for our users through modern and familiar web design patterns that is information-centric and brings our content to our users. the result: a mobile-friendly, flexible and intuitive web application that brings the full power of a faceted library catalogue to users without compromising quality or performance, employing responsive web design (rwd) principles. responsive web design and designing for the user experience “it is the pervading law of all things organic and inorganic, of all things physical and metaphysical, of all things human and all things superhuman, of all true manifestations of the head, of the heart, of the soul, that the life is recognizable in its expression, that form ever follows function. this is the law.” [1] the results of good design are barely noticed or considered, nor should they be. design should be clear, intuitive, and responsive, ultimately guided by the same laws of “form follows function” or as don norman of apple would coin more concisely as “user experience design (ux).” [2] figure 2. ux design layers (enlarge) for us, ux design is a non-linear multi-tiered approach to software development. we weren’t developing the catalogue from scratch: we leveraged our experience, knowledge, and history of users’ interactions of the existing catalogue (which we also developed) as a foundation for improvement. while a primary focus was making the catalogue responsive, a redesign also allowed us to enhance the overall user experience. the mobile first strategy as outlined by luke wroblewski [3], although compelling, wasn’t practical in our case as the majority of users (93% as of the writing of this paper and 2 months into the launch of the rwd catalogue) come to us from a desktop environment. our design challenge: take the current desktop environment and scale it to mobile screens and everything in-between. stats and specs the university of toronto library catalogue is a collection of almost 7.5 million bibiographic records, 2.7 million authority records, and 12 million item records. there are, on average, between 10,000-15,000 transactions per day (record updates and item checkouts/renewals/returns) and the catalogue web application receives over 250,000 visitors per month, responsible for an average of 1.2 million pageviews. a brief background of the web application and its various front-end and back-end components: the front-end of the web application is written using the freemarker template engine, with jquery and css3 providing the interaction and styling to the rendered html. the back-end is comprised of java objects and servlets organized in a model-view-controller pattern. the primary data source for the catalogue is endeca enterprise search and index, which is accessed via their proprietary java api. supplementary data is stored in mysql and the interface for connectivity is the mybatis persistence framework. data is processed into endeca indexes (called “dgraphs”) via various shell and python scripts, which pull the content from the library system’s sirsidynix unicorn ils. the sirsidynix api and oracle sql queries are used to export data from the ils. figure 3. device usage tracked in google analytics. (enlarge) front-end design continuity for users was an important consideration when redesigning the library catalogue. web users in general tend to appreciate consistency and predictability: accordingly, we aimed to minimize any confusion from the switch to a new design, especially for long-time users of the service. as a result, the site experience has remained largely unchanged for the desktop: the search box remains at the top of the page; search results retain a two-column layout (for facets and result items); and record details are contained in a dedicated space [4]. on smaller devices, the facets column is tucked away into an off-canvas layout that can be accessed via a button and the search results column expands to fill the width of the device. the layout is also “orientation-friendly,” meaning that any interactions in the portrait orientation are preserved in landscape orientation. figure 4. device parameters and media query breakpoints. (enlarge) rwd best practice recommends em based media queries to allow for user initiated zooming on browsers, so simply divide your screen resolution by 16 to convert pixel widths to ems. device testing while developing, it was essential to test on as wide an array of devices and browsers as possible, matching the wide user base we support. beyond the standard desktop environments (e.g. various browsers in windows, mac os and linux), we also tested on a variety of ios, android, blackberry, and windows phone devices. emulating real world situations meant our testing had to account for the latest smartphones (e.g. iphone 5) as well as some of the older models (e.g. iphone 3), and speeds ranging from lte to 2g connections. for ease of layout development, we found a number of responsive bookmarklets and made good use of them, including the rwd bookmarklet and viewport resizer. our metrics indicate a need for continued support of older versions of internet explorer with approximately 8% of users still using ie8. therefore our testing also expanded to these older browsers. figure 5. search results by device. (enlarge) as a result of this testing, we made a number of css and javascript based enhancements within our code. we decided to use polyfills [5] to enable backwards compatibility: respond.js allows older, non css3-compatible browsers to support media queries, which in turn uses the matchmedia.js polyfill to execute media queries in javascript. looking back, deeper testing with our users would have exposed many issues we dealt with post-launch. although we held many sessions with users, they often listened and provided verbal feedback rather than performing hands on tasks. in the future, a more structured testing framework will be used to surface issues earlier in the development. three component pages the catalogue interface is broken down into three main component pages: search results, record detail, and marked items. a landing search page exists but the majority of users come to the catalogue via the search engine on the library’s home page. the search results page offers faceted refinement, sorting and viewing options and less cluttered interface for catalogue items. figure 6. search results component, desktop and mobile. (enlarge) the detail page has a persistent “sticky” menu with options to jump to content and includes permitted uses, subject listings, enhanced data where it exists (summaries, reviews, table of contents, author information, etc.), and marc view. users can navigate through their search results with previous/next buttons. figure 7. record detail view, desktop and mobile. (enlarge) the marked record page contains items saved from the user’s session and has tools for emailing, importing into refworks, and exporting as csv. figure 8. marked record view, tablet and mobile. (enlarge) figure 9. off canvas layout for faceted searching. (enlarge) an ongoing design challenge is determining an economy of content, making the most critical information highly visible while hiding secondary content behind icons and detail pages. for example, we use a simple ‘in/out’ with red/green signifiers [6] for item status, and the holding information has been re-ordered according to importance: status, library, location, and call number. figure 10. holdings example. (enlarge) each item’s toolset is hidden and includes: detail view, request item, permanent link, refworks, cite, email, and my.library. a sharing icon accesses these options using jquery and a hidden div. figure11. individual record’s toolset on the search results page. (enlarge) economizing screen space is an integral part of designing for mobile devices. as some item titles are very long, a special truncation feature was implemented in javascript that trims long titles and paragraphs. a button becomes available to all the user to view the full text when necessary. the snippetting occurs on page load and takes the viewport’s dimensions into account, meaning devices with larger screens that can comfortably display the content don’t trigger snippetting. figure 12. title trimming option for long titles. (enlarge) navigation navigation items across the search results and details screens are also responsive. individual navigation actions are visible on-screen when sufficient real-estate is available and collapse into dropdown menus otherwise. similarly, cover jackets for records are responsive. no native responsive image solution currently exists for every web browser, so we have made use of the picturefill.js library [7] to request images of different dimensions using a combination of media queries and javascript. along with screen space considerations, we have also updated our interface to use more modern ways of navigation in the now touch-friendly interface. for example, instead of a paging mechanism for search results, we have switched to lazy-loading results as the user scrolls to the bottom of the page. the ease of scrolling with touch devices makes this a more natural experience than a full page reload. we have also introduced alternative layouts; there is a “text-only” display option available for search results along with the standard display, and a “tiled” display is scheduled to be released soon. figure 13. full record and text only views. (enlarge) headers sticky headers were implemented in both the search results and details screens, making it easier for users to jump to various parts of the page without requiring excessive scrolling. figure 14. sticky header search results example. (enlarge) in the previous iteration of the library catalogue, grouped records entries [8] were not as visible and our visitors often missed important records. the search results page now indicates that items are “grouped” and there are two visual representations indicating multiple items. additionally, we developed the ability to ungroup records, the release of which is pending approval. figure 15.grouped records. (enlarge) icons and book jackets approximately 30% of catalogue items have publisher-provided covers. to create a more balanced row-item display for records without covers, we generate format icons through an in-house application that writes the record’s data onto custom graphics. format icons offer users a visual aesthetic and continuity similar to popular sites such as amazon and provide an immediate indication of what type of material is contained in the item record. we also offer a text only list of search results, which was developed in response to several requests from librarians. figure 16. examples of custom format icons. (enlarge) back-end design improving search functionality, performance, and data-liveness along with developing a responsive front-end interface, we completely overhauled the application and serving infrastructure with four goals: improved site performance; decreased page load time; high availability; and data currency. this work included restructuring the back-end web application, designing the server infrastructure for better performance and redundancy, and overhauling the scripts that fetch the data from the ils. application infrastructure designing the back-end to scale with user demand was a critical requirement. the library catalogue is one of the most heavily used web applications at the university of toronto. for example, over the month of october 2013 we averaged over 250,000 unique pageviews per week, resulting in over a million requests to the endeca dgraph indexes per week. the new architecture uses the open source haproxy load balancer to process requests to both the web application and the endeca dgraph indexes.  when an http request arrives at http://search.library.utoronto.ca, haproxy acts as a sticky load balancer in front of a cluster of virtualized web servers (currently seven) hosting the catalogue application. each web server in turn communicates with a cluster of dgraphs (currently three), using haproxy to load balance the requests.  although normally powered by ten different servers, the entire application can run, if necessary, on only one web server and one dgraph server. figure 17. request – response loop. (enlarge) running the application on multiple identical web servers not only improves performance under load but allows for adding more front-end servers on demand during peak times, such as end of school term.  the site uses several performance enhancing techniques such as using google’s mod_pagespeed apache module to optimize and cache css and javascript files, an internal, heavily cached cdn for serving static content, and delivering cover images from a separate web server to increase the number of simultaneous page element downloads. endeca dgraph performance is optimized by using servers with 128 gb of ram to cache the entire index as well as millions of queries, and by configuring the dgraph processes to make as efficient use as possible of multi-core cpus.  we have consistently achieved catalogue query result page load times of less than 2 seconds, very much in line with user expectations of modern websites. [9] responsive design, responsive data one of the classic challenges in developing a catalogue that has been decoupled from the underlying ils is ensuring that the data has been searched and delivered to users is current and accurate. given that the catalogue data is stored in endeca, and not directly being served from the ils, we use a full and partial update model. a full update syncs the entire bibliographic, item, authority and holding dataset from unicorn to endeca, whereas a partial update syncs data that has been updated in a specific time interval. in our previous model, we used to handle bibliographic and item updates separately. for bibliographic data: full updates once a week and partial updates once a day. for items: full updates once a day, and partial updates every 20 minutes. this arrangement became fairly cumbersome and complex to maintain, especially since the partial updates didn’t capture all changes, and the full updates caused load issues on our endeca index servers. in the new model, we have combined all of our various data sources (bibliographic, item, authority etc), streamlined data retrieval into partial updates every 10 minutes, and more or less eliminated the need for full updates [10]. now, adds, deletes, and updates to all types of data is indexed and available to the web application within 10 minutes. a focus on our users information technology services (its) at the university of toronto libraries is actively working towards improving communications with our diverse user base of students, faculty, and staff. with a major change in service looming – the complete redesign of the library catalogue – we capitalized on the opportunity to manage this change with a structured communications plan. the plan focused on increasing project and staff visibility, obtaining and incorporating user feedback, and preparing users for the pending change in service. the communications plan detailed key messages, goals and objectives, target audiences, available communications channels, stakeholders, budget, collateral to be created, and key performance indicators. a timeline with specific deadlines was attached. the catalogue team worked consistently to increase visibility of the project and its staff members. the low levels of communication around past projects in its were acknowledged and the team committed to transparency around the development of the new library catalogue. messaging around project status, opportunities for feedback, upcoming events, and next steps were disseminated in a consistent manner through a variety of communications channels [see appendix]. internal communications were also increased to ensure that departmental staff were also aware of the project and timelines. accessibility utl is committed to providing all users with equitable access to university of toronto libraries’ digital spaces, such as the library catalogue, central libraries’ websites, and digital collections. we strive to ensure our web spaces conform to level aa standards, as per wcag 2.0 and aoda (accessibility for ontarians with disabilities act) standards and timelines. this work includes making sites compatible with assistive technologies, such as screen readers, as well as ensuring content is clear and understandable to a wide variety of users. the catalogue team received training on developing with accessibility in mind, including attending the university of guelph’s accessibility conference in may 2013. additionally, we were in contact with the university of toronto’s accessibility office for advice on outreach and testing; reached out to resources at scholars’ portal and ocul; and participated in the development of the accessible content e-portal (ace) pilot project.  this information gathering and training allowed our team to run in-person and online usability tests with users with vision impairments and circulate a beta site link to contacts who represented users with disabilities. the team worked with our web coordinator librarian, who leads accessibility work for the library, to test the site at various stages of development and provide recommendations for changes. tools used for testing included colour contrast checker, wave, jaws, chromevox, and html_codesniffer. as users spend more time with the catalogue, we continue to gather feedback and subsequently improve accessibility features. managing change managing the impending change in service required targeted communications to users who were identified as highly affected by the change: faculty, librarians, and library staff.  since the catalogue was scheduled to transition at the beginning of the fall semester (late august 2013), these users required appropriate lead time to get acquainted with the new design and update teaching materials, libguides, and help videos. demonstrations, feedback sessions, and training opportunities worked to give these users the tools they needed to prepare. reaching the majority of these users was a difficult process: many were away during the summer months and not present to receive our attempts at outreach. in the future, it would be useful to avoid the summer months when communicating with these specific groups. promotion and feedback opening and maintaining a two-way dialogue between the project team and our users was integral to the communications effort. users were encouraged to submit comments and questions throughout the entire project and this message was included in all outgoing communications. feedback was received in many ways: dedicated project email address, online feedback form embedded in the catalogue, paper comment forms, usability testing, focus groups, and through conversation with project staff (verbal comments and meeting discussion notes were entered manually into our tracking system). to help users transition to the new library catalogue and encourage users to submit feedback, a link to the beta site was distributed early in the development of the new interface. the beta site ran publically for three months with online feedback open for the duration. the beta site was released in phases: first to early adopters identified by project staff, secondly to departments who work closely with its, and finally to all users. in-person events, such as drop in feedback sessions gave users an opportunity to meet the project’s staff members and engage in discussion.  these events also worked to give a “face” to the project, make personal connections, and create a venue for impromptu brainstorming with users. all feedback was managed through jira, an issue tracking and management web application. using jira allowed us to create tickets for all feedback items, manage conversations, save users’ contact information, and export data for analysis. a central system also promoted better prioritization and planning within the team itself. all solicited feedback was reviewed and incorporated into the development of the catalogue at various stages, where applicable. changes were then communicated back to users. the launch day of the new library catalogue was also widely promoted. staff were available to support users via phone and email. help pages and documentation were limited and users requested more detailed information. new pages are being developed and include video tutorials, created in coordination with reference services, to provide visual instruction on various catalogue features and functions. these videos will be repurposed and used to improve awareness of the rwd catalogue through the main library website, digital signage, and posters. overall, the targeted and consistent communications around this large change was well received. a short survey of staff members indicate that they were well aware of the project, were kept apprised of timelines, and felt that the project was well communicated. anecdotal evidence from library users suggests that most people were aware of the change in service and were prepared to make the transition.  based on this success, this approach to communications planning will be used as a model for deploying new products in the future. future directions we plan on migrating a few remaining features from the previous iteration of the catalogue interface to the current one, such as virtual shelf browse, item location maps, and print on demand. long term plans include developing an authentication layer that would allow integration with other library services such as usage history, loan information, persistent marked records, item recommendations, and more. we would also like to facilitate interaction with social media by way of adding sharing options, as well as improving search engine optimization by introducing linked data into the catalogue. although the catalogue has been launched to our users, the project continues to evolve and grow. we continue to receive feedback from our users and make improvements and changes based on these suggestions. as always, we look forward to discussion with the code4lib community through the journal, listserv, and in-person events. references and notes [1] sullivan, louis h. (1896). “the tall office building artistically considered”. lippincott’s magazine (march 1896): 403–409. accessible via internet archive: https://archive.org/details/tallofficebuildi00sull. [2] “i invented the term because i thought human interface and usability were too narrow: i wanted to cover all aspects of the person’s experience with a system, including industrial design, graphics, the interface, the physical interaction, and the manual.” http://uxdesign.com/ux-defined [3] http://www.lukew.com/ff/entry.asp?933 [4] one key difference with record details: in our previous iteration the details loaded in a popup window atop the search results. we have now dedicated a full page to record details to give the content more breathing room. [5] polyfills are javascript libraries that emulate modern css3 and native browser functionality for older browsers. [6] replaces ‘in/n/a.’ the red/green with white text has a aaa accessibility pass for colour contrast and offers a text alternative for users with difficulties discerning colour [7] https://github.com/scottjehl/picturefill [8] grouped records employ an approximation of frbr (functional requirements for bibliographic records), a model developed by ifla which allows users to define relationships between the entities (eg. books) that provide links to navigate through the relationships, creating new search paths [9] see: http://www.webpagetest.org/result/131115_wq_t73/ [10] unless required for application changes or bug-fixes. appendix: communications channels a summary of outreach and communications channels used: hosted in-person demonstrations and feedback sessions visited the scarborough and mississauga campuses for in-person events facilitated focus groups with staff and students performed individual usability tests created a dedicated email account for direct feedback distributed consistent staff email updates updated staff at university committee meetings updated at staff meetings in various library departments deployed digital signage in various library buildings distributed multiple news items in the university of toronto bulletin (an all-staff and alumni bi-monthly e-newsletter) included articles in departmental newsletters targeting faculty and students contacted faculty and departments through faculty liaison librarians created and distributed a new library catalogue brochure used the active social media accounts at all three campuses and the central university of toronto accounts distributed messages to the code4lib listserv asking for feedback and announcing the product launch created and distributed news items and website banner images (featured on the main library website and two campus websites) created a project-specific webpage detailing the project’s purpose, progress, and next steps distributed rolling messaging via social media accounts and enews with student life (undergraduate focus), graduate student union, the ischool, and the faculty of arts & science performed outreach to librarians and library staff with teaching duties to assist in updating instructional videos and online q/a created an online help page explaining new functions and icons participated in various professional activities, such as conference presentations, poster sessions, and articles about the authors lisa gayhart is the digital communications services librarian in information technology services at the university of toronto libraries. as the dedicated communications team member in its, her number one priority is to move library it from the sidelines to the spotlight through structured and actionable communications. bilal khalid is a senior application developer at university of toronto libraries. he is responsible for the core development for the library catalogue, as well as other web development projects at the library. his interests include: emerging web technology trends, cross-platform development, and improving software development workflows. gordon belray is an information architect at university of toronto libraries who specializes in user experience (ux) design and server application development (fadis and mymedia). his current focus is on responsive web design (rwd) and accessible design. he is a graduate of the department of art, university of toronto. acknowledgement this article expands upon “your library, anywhere: a responsive library catalogue at university of toronto libraries” as published in the winter 2014 20(1) issue of ola access. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – editorial introduction – share your ideas mission editorial committee process and structure code4lib issue 18, 2012-10-03 editorial introduction – share your ideas the code4lib journal’s mission is to foster community and share information. it is my hope that reading the articles in this issue will help you develop your own ideas and solutions. and that you will share your ideas with the community. by ron peterson this december (2012) will mark the fifth anniversary of the publication of the first issue of the code4lib journal. i have had the honor and pleasure of being involved with the journal from the very beginning. it has been very rewarding to see the journal survive, grow, and flourish. i feel a little pride whenever i see an article from the journal cited in another article, especially if i was the one who worked with the author of the article to get it published. similarly, when i see an article from the journal referenced on an email list in answer to someone else’s question, i know that the hard work that goes into publishing each issue of the journal is well worth it. the most satisfying part of seeing people reference an article from the code4lib journal is that i can see that the articles are making a difference and helping someone. they are helping people build and develop their own ideas and solutions to the problems facing libraries. they are then sharing their own ideas and solutions with other people who will build on them further. so i hope that as you read the articles in this issue you think about how you can apply the ideas presented in these articles in your own library. and i hope that when you find a solution to a problem your library is facing, you share your ideas with your colleagues. this issue of the code4lib journal has many articles that will get you thinking about things you can do at your library. if your library uses drupal to manage its web site, maybe matt weaver’s “libalerts: an author-level subscription system” will spark an idea for engaging your users through a service that updates your users about new books from their favorite authors. or it may be that you are looking to leverage content from your drupal-driven web site into a new mobile site, like junior tidal describes in “using php to parse ebook resources from drupal 6 to populate a mobile web page.” if you are looking to go beyond creating a mobile web site and start thinking of ways to develop mobile applications, hearing about “modular mobile application design” from jim hahn and nathaniel ryckman may inspire you to help out with their minrva application for android or create a mobile application of your own. jim and nathaniel also discuss their rapid prototyping process they used to gather user feedback, an idea that is expanded upon by shaun ellis and maureen callahan in “prototyping as a process for improved user experience with library and archives websites.” shaun and maureen share how they used a subset of agile practices, such as measurable goals and iterative prototypes, to inform the design and improve the user experience of their web site. in “hacking 360 link: a hybrid approach,” john durno takes us through his experiences using javascript to integrate functionality developed by the library into the serials solutions 360link interface. the article talks about the challenges of moving locally-developed functionality from a locally-hosted service to a vendor-hosted one. another example of using javascript to extend a vendor product is presented by steven jay bernstein in “patron-driven expedited cataloging enhancement to webpac pro”. steven applies the model of patron-driven acquisitions to the cataloging backlog to create a process where patrons can request items in the cataloging backlog from the libraries’ opac and trigger expedited cataloging of the item. libraries working with shibboleth for authentication may find some ideas for improving their single sign-on solution in alexander jerabek and minh-quang nguyen’s “a hybrid solution for improving single sign-on to a proxy service with squid and ezproxy through shibboleth and exlibris’ aleph x-server.” the hybrid solution developed by the authors was designed to support existing workflows while preparing for using shibboleth for authenticating users. james r.w. macdonald and daniel yule describe how the jarrow software that they developed supports the workflows of users submitting electronic theses and dissertations in the article, “jarrow, electronic thesis, and dissertation software.” jarrow provides customizable workflows for libraries managing the process of submitting electronic thesis and dissertations. it is my sincere hope that the articles in this issue of the code4lib journal will inspire you to think of ways to solve problems facing your library and when you have solved those problems, that you come back to the code4lib and share your experiences. libraries need your creative input. they need to hear from you. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – testing remote access to e-resource with codeceptjs mission editorial committee process and structure code4lib issue 49, 2020-08-10 testing remote access to e-resource with codeceptjs at the badische landesbibliothek karlsruhe (blb) we offer a variety of e-resources with different access requirements. on the one hand, there is free access to open access material, no matter where you are. on the other hand, there are e-resources that you can only access when you are in the rooms of the blb. we also offer e-resources that you can access from anywhere, but you must have a library account for authentication to gain access. to test the functionality of these access methods, we have created a project to automatically test the entire process from searching our catalogue, selecting a hit, logging in to the provider’s site and checking the results. for this we use the end 2 end testing framework codeceptjs. by ralf weber introduction and motivation the badische landesbibliothek karlsruhe (blb) offers their users a variety of e-resources with different access possibilities. some of these are freely accessible, but many require a login with username and password to ensure that you are a user of the library. in the blb we use shibboleth to login. the procedure is always the same: the user usually selects the library as an institution and identifies himself with his user number and password. however, the implementation on the individual portals is not always obvious and often there are functional problems; sometimes access to a source that is actually licensed is denied. from time to time we get feedback from users about problems, but many don’t report them because they suspect the problem to be their own. therefore we thought about the possibility of automatically testing selected sources for their correct operation by mapping the complete workflow of information retrieval. for instance: search for a specific term in the opac expecting to find a specific match login as user access to the full text at the provider by automating these processes, we hope to discover problems or errors in access to the full text in order to react promptly and correct the error as quickly as possible, either on our side or by contacting the provider. what is codeceptjs? why did we choose this framework? codeceptjs is a modern end to end testing framework with a special behavior-driven development (bdd) style syntax for nodejs. the tests are written as a linear scenario of the user’s action on a site. it abstracts browser interaction to simple steps that are written from a user perspective.  a simple test that verifies that “welcome” text is present on a main page of a site will look like this: getting started with codeceptjs feature('codeceptjs demo'); scenario('check welcome page on site', (i) => i.see('welcome'); }); codeceptjs passes execution commands to helpers. depending on what you intend to test, you choose one of the helpers. in our case, we do not need cross-browser support, we are simply interested in faster testing and therefore chose the chrome-based puppeteer helper. writing the tests for our scenarios can be done by librarians with little to no knowledge of programming. codeceptjs is simple to use and tests are written from a user’s perspective. there is an actor (represented as i) which contains actions taken from helpers. a test is written as a sequence of actions performed by an actor. here’s an example: i.amonpage('https://github.com'); i.click('sign in', '//html/body/div[1]/header'); i.see('sign in to github', 'h1'); i.fillfield('username or email address', 'something@totest.com'); i.fillfield('password', '123456'); i.click('sign in'); i.see('incorrect username or password.', '.flash-error'); it’s readable and simple and works using the puppeteer api. since our tests are always more or less similar, we decided to create a template that can be used as a basis for each query. codeceptjs allows us to store often used interactions like common locators and methods in “page objects.” this way the code can be made even more simple and readable. for example, we have created a module to login with shibboleth in which the access data of a test user is stored, saved as shibboleth.js: /** * modul für das shibboleth login */ const { i } = inject(); module.exports = { login () { i.fillfield("#username", "xxxxxxxxxx"); i.fillfield("#password", "xxxxxxxxxx"); i.click(".bbtn"); }, loginintab () { i.switchtonexttab(); i.fillfield("#username", "xxxxxxxxxx"); i.fillfield("#password", "xxxxxxxxxx"); i.click(".bbtn"); i.wait(10); } }; so for the whole login process, in the test file you only have to enter shibbolethpage.login(); or, if it is necessary to switch to the next tab in the browser shibbolethpage.loginintab(); we have created a second module, search.js, which contains all further interactions regarding the search for a hit and the access to the full text at the provider. this is best explained using an example. hands on – a practical example a simple example if there are no special features necessary, a simple test proceeds as follows: load start page of the opac log in via shibboleth search a chosen title by id check if title matches expected title click on link to full text log in again test for a text you expect to see at the provider you can try this live: go to https://rds-blb.ibs-bw.de/opac/ search for the id 1668499266 we expect in our test the title “die angezählten” to be there. now we click on the link to the full text (“zum dokument”) now, as you are not able to login, your journey ends on the shibboleth login page but in our test, we automatically login and then expect to see the text ‘lesen’ on the page from the provider this is a screenshot what the page from the provider looks like after successful login you can see the text “lesen”. the test itself looks like this: feature('content-select test'); scenario('mit shibboleth login @content-select', (i, shibbolethpage, searchpage) => { searchpage.loadbuecherundmehrpage(); shibbolethpage.login(); searchpage.searchfor('1668499266'); searchpage.hopetosee('die angezählten'); searchpage.clicktodocumentinbuecherundmehr('a.btn:nth-child(1)'); shibbolethpage.loginintab(); searchpage.hopetoseefromprovider('lesen', 'div.actions:nth-child(3) > a:nth-child(1)'); }).retry(3); each line corresponds to one of the instructions listed above. as the identifiers (css/xpath) for the search string and the title in the result list are always the same, you only have to specify their values, so no further css/xpath rules are necessary. in the search.js file these functions look like this: searchfor(searchvalue) { i.fillfield("#searchform_lookfor", searchvalue); i.click("button.btn"); i.wait(5); }, hopetosee(whattosee) { i.see( whattosee, "#result0 > div:nth-child(4) > div:nth-child(2) > div:nth-child(1) > div:nth-child(1) > a:nth-child(1)" ); }, in summary, a librarian writing a simple test only needs to take this template and adjust three things: the values for the id to be searched the value of the expected title in the result list the text that can only be seen on the provider’s site after successful login, including the (css/xpath) identifier that he gets with firefox/chrome devtools. the test gets called by the following command: npx codeceptjs run --steps –colors filename_test.js the output looks like this (green when everything is fine, red on errors) troubleshooting and limitations normally the tests run as shown in the example above, with no problems. we have found that it makes sense to take a “break” between the individual steps. otherwise the test may fail because the page is not yet completely loaded and access to the dom will not succeed. codeceptjs offers the command wait(seconds) for this. this ensures that the page is loaded and the complete dom is available before the next test starts. we use this extensively. the test runs longer, but that doesn’t matter. we also wanted to avoid recording these test accesses in statistics. codeceptjs offers mockrequest for this. this helper allows you to use mock requests while running tests, so you can block calls to 3rd-party services like matomo, google analytics etc. i.mockrequest('get', '/matomo/*', 200); these calls are also integrated into the functions. some sites expect the consent of the privacy policy, which appears in a pop-up window. access to this was not possible with codeceptjs via the dom. but we could solve this by setting the required cookies manually. codeceptjs also provides a simple method for this: i.clearcookie(); i.setcookie({ name: "accesssessionsignature", value: "2df0a0adc0a705630e5f62be9f2efdf4b87835094e077dc76e6799dcbdbae627" }); i.setcookie({ name: "accesssessiontimedsignature", value: "b2a0050b0df65bab1b16bad39f08507302c0218db8a0c18a55dbd48f8744984c" }); fortunately, these cookies do not have to be created each time but can be loaded with fixed values directly from the test file. therefore no further work or preparation is necessary and these tests can be started by cronjob without problems too. on some pages you have to select the institution from a dropdown list, sometimes this happens dynamically via ajax. codeceptjs is well equipped for this also as you can see here on these examples: filling fields i.selectoption('choose plan', 'monthly'); // select by label i.selectoption('subscription', 'monthly'); // match option by text i.selectoption('subscription', '0'); // or by value i.selectoption('//form/select[@name=account]','premium'); i.selectoption('form select[name=account]', 'premium'); i.selectoption({css: 'form select[name=account]'}, 'premium'); to make sure that there are no temporary network problems or other technical failures, we optionally run each test three times. codeceptjs offers the possibility of retrysteps. you simply append the function retry(amount) to the scenario. scenario('really complex', (i) => { // test goes here }).retry(2); the execution is only repeated if the test has failed, very nice! there is one thing we have not yet been able to solve. there’s a provider who requests a recaptcha in addition to the login. maybe we can think of a solution for this in the future. conclusion meanwhile our tests are running productively and automated. we only run them from off campus, because if they work there, they also work inside the rooms of the library. for a better organization we have divided them into three groups: database, e-book and e-journals. at the time of writing, we have 83 tests and more will be added. we have set up a cronjob for each group which calls up the test files once a month, with the result is logged and sent by e-mail to the responsible employees. additionally, we have activated the allure plugin codeceptjs.cthis is a reporting tool that shows a representation of what has been tested in a web report form. this allows details of the tests to be tracked, and in case of an error there is a screenshot, which often makes troubleshooting extremely easy. now that we have been in the production phase for a few months, we can say that we have achieved what we wanted. we can automatically control whether external access to full text works or not, we notice long response times on the part of the providers and can ask them to improve. we can better understand the problems of the users to reach the full text, because now we have to write the whole process as a test. here we can see how complex the user guidance is. and we notice changes to the user interface of the website on the part of the provider, because then our tests fail, which fortunately is not often. about the author ralf weber is the information technologies librarian at the badische landesbibliothek karlsruhe. he cares and develops tools for automation of library workflows and for new services within the context of the library. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – editorial: reflecting on the success and risks to the code4lib journal mission editorial committee process and structure code4lib issue 36, 2017-04-20 editorial: reflecting on the success and risks to the code4lib journal at the code4lib 2017 conference, i gave a short lightning talk about the code4lib journal and in the process realized that we will soon be closing out the 10th year of "foster[ing] community and share[ing] information among those interested in the intersection of libraries, technology, and the future." that quote comes from the code4lib journal […] at the code4lib 2017 conference, i gave a short lightning talk about the code4lib journal and in the process realized that we will soon be closing out the 10th year of "foster[ing] community and share[ing] information among those interested in the intersection of libraries, technology, and the future." that quote comes from the code4lib journal mission page. the word cloud at the top of this article contains the words of the 333 articles that have been published in issues 1 through 35 (except for one non-stopword — can you guess the word that would have dwarfed all other words in the image?). since november 1, 2007, the journal's website has seen nearly 1.5 million page views from locations primarily in north america with the long tail of access spanning the world. over the past 10 years, 39 people have donated their time and energy to solicit and edit articles, coordinate the publication of issues, and run the technical infrastructure of the journal. the code4lib journal is a unique collaboration of authors and editors working within a modestly open process to publish articles that further the journal's mission. the editors work openly with authors to offer different perspectives, to widen the applicability of topics, and to polish the text and tone of articles. if you see one of these editors, thank them for furthering the journal's mission. i'm on that list, having served on the editorial board for about half of the journal's existence, and i too care about providing this forum for new and established authors to foster community and share information. in that light, i outlined three areas of concern for the journal during the code4lib 2017 talk. hosting and technical support: in keeping with the all-volunteer theme of the editors, the hosting provider space and the technical support for the journal site are also provided without compensation. the journal uses a wordpress site hosted on ibiblio's infrastructure. the system administrator, software installer, and custom plugin developer have all been volunteers on the editorial committee. this is a sustainable situation only in as far as these volunteer resources remain available. this state of affairs is but a small microcosm of what is happening in the code4lib community, where the main www.code4lib.org website, the wiki.code4lib.org website, the slack administrator, the irc helpers, the mailing list hosting/support, and the conference activities are all done by volunteers. there was a breakout session during the code4lib 2017 conference where these general needs were discussed. what we have now is working, but it is not without the inherent risk of volunteers losing interest or leaving the community causing the structure to collapse or the catastrophic failure of an organization's infrastructure of which the journal is seen as minor collateral damage. that is related to the second concern, archiving and preservation. archiving and preservation: there is good and useful content in the journal — 1.5 million page views across 10 years is not an accident. yet the journal lacks a thoughtful preservation strategy for that content. in the early days of the journal, i remember someone saying that their library printed, bound and shelved the journal. even if that were more than a misremembered email message, would that library still be doing it now? even if that library were doing it now, how does that effectively capture the code snippets and other ephemera associated with articles? i've had some preliminary conversations with a representative from portico, and that could be a solution. at the lowest level of membership fees based on annual revenue (which is zero, of course, for the journal), preservation in portico would cost $250/year — and it isn't clear yet whether such preservation would capture more than what a pdf print of the article could hold. and the financial commitment is related to the third concern, zero cash flow. zero cash flow: technology infrastructure and editor time donated. authors neither pay for nor are paid for publishing articles. there are no advertisements on the website. the only revenue is a few dollars that the journal could receive from ebsco for being included in their searchable indexes, but the journal has no mechanism to receive that revenue (nor would it come close to balancing out the technology infrastructure and/or preservation costs). in some respects i get that this is a point of pride: no money changes hands in the publication of this journal. still, this lack of cash flow mechanisms is a risk that exacerbates the first two concerns mentioned above. kudos to the code4lib fiscal continuity special interest group for their report earlier this year, but there doesn't seem to be a willingness in the community to address this concern/risk. although i've used twice as many words to describe concerns for the journal than i did to describe its strengths, that is not an indication that the journal is in trouble. it isn't. the journal is an institution that is valuable and is valued, and the concerns are described out of a desire to address risks that might bring the journal to an end. and with that strength and love for the journal in mind, i am pleased and honored to present the 36th issue of the code4lib journal. issue 36 summary article proposals come to the journal at random, so it is interesting that this issue has a number of articles on institutional repositories and related systems. in autoload: a pipeline for expanding the holdings of an institutional repository enabled by resourcesync shepherded by editor péter király, the authors describe a system for finding archival copies of publications by authors at los alamos national laboratory and ingesting them into their institutional repository. editor junior tidal assisted with an interactive map for showcasing repository impacts, an article that ties together google analytics and google maps through a ruby-on-rails application to demonstrate where institutional repository resources are being used. in medici 2: a scalable content management system for cultural heritage datasets the authors describe a series of tools in used to store, retrieve, and provide depth (in some cases literally!) to digital surrogates of archeology objects; it was edited by carol bean. in outside the box: building a digital asset management ecosystem for preservation and access, journal editor meghan finch worked with authors at the university of houston to describe the combination of hydra-in-a-box, archivematica, and archivesspace that they use to provide access and preservation support for their collections. there are also two articles related to publishing reusable metadata on the web. in recommendations for the application of schema.org to aggregated cultural heritage metadata to increase relevance and visibility to search engines: the case of europeana the authors describe experiments and make recommendations for transforming their rich and varied metadata into a form more easily understood by web crawlers using schema.org tags; it was edited by ruth tillman (who is also the journal's technical support!). and lastly, andrew darby shepherded linked data is people: building a knowledge graph to reshape the library staff directory through the editing process; it is a fascinating look at how one can turn the simple staff directory into a search-engine-friendly source of data about the expertise in the library. thank you to the editors and the authors for your contributions to the code4lib journal issue 36. call for editors if you are reading this in april 2017 and you are interested in helping the journal to continue, please see the call for editors. we’ve extended the due date and are announcing the call to a broader community of readers. peter e. murray coordinating editor, code4lib journal issue #36 subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – transforming knowledge creation: an action framework for library technology diversity mission editorial committee process and structure code4lib issue 28, 2015-04-15 transforming knowledge creation: an action framework for library technology diversity this paper will articulate an action framework for library technology diversity consisting of five dimensions and based on the vision for knowledge creation, the academic library’s fundamental vision. the framework focuses on increasing diversity for library technology efforts based on the desire for transformation and inclusiveness within and across the dimensions. the dimensions are people, content and pedagogy, embeddedness and the global perspective, leadership, and the 5th dimension – bringing it all together. by barbara i. dewey introduction why is an action framework appropriate for advancing library technology diversity? developing a framework and its accompanying dimensions requires a focus and provides a context for a more comprehensive way to advance diversity. typically frameworks are developed as conceptual. the notion of an action framework includes the conceptual but underscores the importance of visible and concrete action. i was inspired to consider and develop the action framework for library technology diversity by the framework to foster diversity at penn state: 2010-2015 project. the framework consists of seven strategic challenges for building a truly diverse, inclusive, and equitable institution. they are: challenge 1: developing a shared and inclusive understanding of diversity challenge 2: creating a welcoming campus climate challenge 3: recruiting and retaining a diverse student body challenge 4: recruiting and retaining a diverse workforce challenge 5: developing a curriculum that fosters united states and cultural competencies challenge 6: diversifying university leadership and management challenge 7: coordinating organizational change to support our diversity goals [1] naturally these challenges are broad and institutional in scope but provide a way of thinking about advancing diversity in more specific and measurable ways. transforming knowledge creation: an action framework for library technology diversity the specific action framework for technology diversity, comprised of five dimensions, is based on the core mission and vision of the library: mission: the university libraries inspire intellectual discovery and learning, offer robust information resources and academic collaborations in teaching and research, and connect the penn state community and residents of pennsylvania to the world of knowledge and new ideas. vision: the university libraries will be a world-class academic library with a global reach; a welcoming and inclusive environment for learning, collaboration, and knowledge creation; a partner in research and education; a leader in delivery and preservation of library collections; and a place that uses technology and rewards innovation. why focus on knowledge creation? i believe knowledge creation is at the heart of libraries’ existence and remains so in perpetuity. the framework for library technology diversity can be developed through the lens of knowledge creation. in the context of diversity, the process of knowledge creation needs be inclusive and expansive if its purpose is to advance understanding, solve global problems, and advance the human condition. embracing concepts of intellectual freedom which are fundamental to our values in libraries and in the academy is woven into the mission and vision. everything we do with library technology diversity relates to these concepts. thus, the following comprise framework dimensions for advancing technology diversity, not only in the library, but throughout the university. figure 1. transforming knowledge creation: an action framework for library technology diversity – the 5th dimension weaves it all together dimension 1: people library workers and technologists come from a variety of disciplines but are clustered in library and information sciences (lis) and computer science/engineering. there is still a lack of diversity and gender equity in the computer science and computer engineering fields; however, these inequities are not as great in information degrees and programs offered by library and information science schools (lis) and “i-schools” such as information studies, information sciences, informatics, and information technology (mcinerney 2015). [2] also, an nsf report, women, minorities, and persons with disabilities in science and engineering: 2013, noted that family was a predominant reason why people of diverse backgrounds did not work full or part-time in science and engineering. recruiting students from diverse backgrounds is an important component to increasing the pipeline. lord and cohoon (2007) [3] provided the following strategies: have prospective students meet with faculty/staff who understand diversity arrange for prospects to meet with students from diverse backgrounds cultivate and publicize the inclusive aspects of department culture utilize diversity training to increase awareness of effective actions and ways to avoid bias these i-school and lis programs can be the entry point for technology-rich positions especially with the growth of new position configurations such as: data curation specialists rich media specialists visualization specialists science data librarians data learning librarians discovery and access specialists web specialists accessibility specialists user experience specialists interface developers learning design specialists online learning specialists electronic publishing specialists digital learning specialists project managers recruitment strategies for incoming students to then take it-related positions in our libraries and it departments will take time. a more immediate strategy is to launch programs focused to broaden the diversity of applicant pools. one such program is penn state’s library diversity residency program. this program attracts recent library and information sciences graduates from historically underrepresented groups for a two-year experience in academic librarianship. the purposes of this program are to: increase diversity among penn state libraries’ faculty increase diversity in the field of academic librarianship invigorate the organization with fresh ideas, skills and enthusiasm prepare library leaders for the future enhance penn state’s reputation as an institution that supports, trains, and mentors diverse librarians the proposed program grew out of the university libraries’ longstanding commitment to diversity. it supports the university’s strategic plan, particularly strategy 2.1 “faculty recruitment and retention for excellence”, which includes the university’s commitment to increase diversity among faculty, and strategy 4.3 “build on the framework to foster diversity”. within the framework to foster diversity, challenge 4 “recruit and retain a diverse workforce” and challenge 6 “diversify university leadership and management” both directly benefit from a diversity residency program. the libraries’ diversity residency program is a two-year faculty appointment. in the first year, participants experience assignments in a variety of areas of the libraries on a rotational basis. all of the assignments include a technology component and relate to the university libraries areas of priority: digital initiatives emerging technologies instructional and research services repository and data curation services residents participate actively in university and libraries committees, councils and task forces. they develop collegial relationships with penn state faculty members and provide support in a variety of ways for students. residents also contribute to national and regional professional organizations. upon completion of the diversity residency program, participants are eligible for continued employment by penn state, subject to the availability of appropriate positions and opportunities, and are well prepared for opportunities in the field of academic librarianship, including positions at other institutions. residents bring rich and diverse knowledge and perspectives to the library resulting in new programs and initiatives. dimension 2: content and pedagogy libraries have always focused on content to support knowledge creation, and academic libraries have the reputation of being comprehensive. a recent presentation at penn state by chris bourg (2013) confirmed huge content gaps in even the largest academic libraries and universities. she noted that they are shockingly lacking in depth when it comes to actually representing the world’s knowledge. this is because of the way we choose and build content and curriculum. books and journals from many parts of the world are not collected, are not part of the interlibrary loan world, are not digitized, and are not embedded or a part of the “western” scholarly record. likewise, scholarship coming from diverse voices is not always collected and is slow to become part of the canon, and therefore, unavailable to researchers for knowledge creation. societal issues related to power, discrimination, and disenfranchisement stem, in part, from the distortion in the historical record upon which knowledge is created. systems for content access are also limiting in terms of completeness and attention to diversity. sanford berman tackled the misrepresentation of knowledge in his 1971 seminal work, prejudices and antipathies: a tract on the lc subject headings concerning people. [4] librarians schooled in the 1970s, a period of great world unrest, will vividly recall his contributions. unfortunately his work is largely forgotten today. he focused on the bias, bigotry, and shear ignorance of the library of congress subject headings, a large part of the library mapping schema. he writes (1971: xi): knowledge and scholarship are, after all, universal. and a subject-schema should, ideally, manage to encompass all the facets of what has been printed and subsequently collected in libraries to the satisfaction of the worldwide reading community…but in the realm of headings that deal with people and cultures—in short, with humanity—the lc list can only “satisfy” parochial, jingoistic europeans and north americans, white-hued, at least nominally christian (and preferably protestant) in faith, comfortably situated in the middle-and higher-income brackets, largely domiciled in suburbia, fundamentally loyal to the established order, and heavily imbued with the transcendent, incomparable glory of western civilization. his work, thought by some to be overly strident and radical at the time, eventually moved the library of congress and the library profession to change, albeit slowly, the most obvious of the biased lc subject headings and to acknowledge the flawed nature of the schema. this schema is still based on published works held in libraries and does not adequately address knowledge found in many other forms. today we can employ new metadata schemas and technologies to increase access to a wider array of resources for rich knowledge creation more representative of the world’s collective minds. these deficiencies and gaps in knowledge discovery hold serious consequences for expansive knowledge creation. academic libraries are embracing 21st century approaches to support indigenous knowledge stewardship and creation. for example, at penn state the university libraries collaborates with the interinstitutional consortium for indigenous knowledge (icik), which is a global network of more than 20 indigenous knowledge resource centers across the world. icik is a network that promotes communication between those who share an interest in diverse local knowledge systems. these local knowledge systems are an integral part of current and future knowledge creation. the icik initiatives highlight the growing awareness of ways of knowing through a broad definition of ways of knowing in our global society. a center piece for icik is the center for indigenous knowledge and rural development collection, a unique collection comprised of 11 subject areas related to indigenous knowledge, originally collected by the late dr. michael warren, a renowned anthropologist from iowa state university. [5] an online bibliography is available, and plans are underway to digitize at least part of the collection. icik, its collections, and its programming have inspired partnerships to reveal different ways of knowing and knowledge creation. the university of texas libraries human rights documentation initiative (hrdi) is an important example of collecting radically different but critically important content and the importance of their use. [6] in 2007 the hrdi was formed to preserve the most vulnerable records of human rights struggles worldwide. in most cases these are born digital records. texas has made important strides locating these fragile resources, preserving them as evidence and memorial, and making them available to conflict survivors, scholars, activities, and students of human rights. diversity initiatives in the way we build and provide access to content is an important dimension of the framework. penn state university libraries, along with other libraries, have embodied the imperative to advance alternative ways of knowing through proactive programming and innovative approaches to collection development. through this process we are learning more about the knowledge, expertise, and literacy components needed to support the incorporation of different ways of knowing in all aspects of library work and consequently in knowledge creation. these efforts require creative and expansive approaches to technological solutions for discovery, digitization, different models of ownership, and preservation. content combined with pedagogy is an important dimension of our diversity framework for library technology. gregory and higgins explore, through essays in the book information literacy and social justice, critical theoretical approaches to librarians’ work. in their words, “we consider the historical, cultural, social, economic, political and other forces that affect information so that we may explore ways to critique our understandings of reality and disrupt the commonplace; interrogate multiple points of view, to identify the status quo and marginalized voices; and focus on sociopolitical issues that shape and suppress information in order to take informed action in the world.” [7] judy chicago’s the dinner party, now housed permanently in the brooklyn museum was conceived as a way to expose famous women throughout history using techniques that were traditionally not considered true art, such as ceramics painting and needlework. in her words, “i had been trying to establish a respect for women and women’s art; to forge a new kind of art expressing women’s experience; and to find a way to make that art accessible to a large audience.” [8] this alternative way of knowing about women’s culture through the ages also needs to be preserved and curated. judy chicago found a way, a schema, and a mapping strategy with the dinner party to present knowledge and ethos in a new way. the dinner party provided the basis for the judy chicago art education archives. archivists and art education faculty members are using the collection and the chicago “living” curriculum pedagogy to provide students with a transformative way of learning how to teach art. the students in this process are also made aware of a new paradigm and way of knowing from a feminist perspective. [9] judy chicago’s recent work, institutional time: a critique of studio art education, provides a basis for dialogue and a case study for transforming curriculum, not just in studio art education, but with other curriculums guiding research and creativity. [10] chicago instituted a pedagogy particularly responsive to the concerns of women artists on what she terms “circle-based pedagogy,” enabling all to participate. she talks about a “living curriculum” based on values and beliefs, framed around encounters, creating new ways to think about a variety of topics, and, themselves, create a living curriculum. the development of the judy chicago art education portal to discuss issues of pedagogy is another example of interactive and living archives which provide an alternative means of documenting ideas and advancing scholarship and teaching. the portal is being launched in four stages and has already garnered international attention: an invitation from judy chicago – which features a video of chicago discussing major tenets of her pedagogy and a multimedia presentation by penn state art ed faculty member karen keifer-boyd. difference in studio art teaching: applying judy chicago’s pedagogical principles highlights projects utilizing judy chicago’s art education archives and the application of her teaching pedagogy in courses taught at penn state. what about men addresses the often contentious subject of men in a feminist environment and takes up the challenges of making institutional changes in terms of curriculum. transforming curriculum will showcase sculptor bill catling who discusses how to achieve a radical transformation in arts education in both policy and curriculum. [11] the portal is a great example of how technology, itself, can support alternative ways of dialogue and understanding in addition to focusing on content. penn state hosts the sister joan d. chittister archive in partnership with the benedictine sisters of erie and mercyhurst college which emphasizes different ways of knowing and global peace and justice initiatives. sr. joan, internationally known writer and lecturer and one of the most articulate social analysts and influential religious leaders of this age, has established an archival collaboration for the preservation of her accumulated works. her emphasis on social justice provides alternative ways of knowing about the world, peace, and faith, especially from a radical benedictine nun’s point of view. sr. joan saw the dinner party when it was on display in cleveland, ohio in 1981. she noted that many of the women in the installation were religious figures such as sophia from the old testament, judith from saint bridget, and hildegarde of bingen, to name a few. chicago’s work had a profound impact on her, so much so, that she subsequently arranged for three buses to take all of the erie, pennsylvania benedictine nuns to cleveland to see the dinner party. she next worked with a committee to encourage each chapter of the federation of saint scholastica to create a version of the dinner party symbolizing women in their religious communities. in 1982 at the federation’s meeting the sisters set up their installation and discussed the failure of the church to recognize women’s contributions through time. another gap in the religious history curriculum was filled that day through alternative ways of viewing and creating knowledge and it is interesting for students and researchers to see the connection between these two amazing women. dimension 3: embeddedness and the global perspective the presence of embeddedness and the global perspective form a third dimension emphasizing “knowing” from within groups, cultures, regions, and perspectives. examining the framework from a global perspective acknowledges the broad context of scholarship as well as the imperative for diverse perspectives and connections. effective support for and engagement with research, teaching, and learning depends on connections, collaborations, and partnerships at levels never seen before. the notion of embeddedness is key to understanding and engaging with different perspectives and, in itself, is a key component of the framework. in 2004 i wrote: the concept of embedding implies a more comprehensive integration of one group with another to the extent that the group seeking to integrate is experiencing and observing, as nearly as possible, the daily life of the primary group. embedding requires more direct and purposeful interaction than acting in parallel with another person, group, or activity. [12] purposeful and overt engagement with diverse populations, cultures, and activities is central to advancing global perspectives. the inclusion of global perspectives acknowledges the broad context of scholarship and its rapidly changing formats. libraries throughout the world are, more and more, managing abundance rather than scarcity of resources. this kind of management requires acute comprehension of relevant physical and virtual environments in a global context. library technologists and technologies they employ must adapt to the global landscape. a core part of this adaption is deep learning about the networked global information environment through the eyes of the 21st global scholar. dimension 4: leadership librarians and technologists are well positioned to play leadership roles in advancing diversity within the library and throughout the institution if we surface our inherent commitment to embracing people and the multitude of ideas reflecting the breadth and depth of the human experience. furthermore, a comparison can be made of themes in institutional time and in the 2013 book by maria accardi, feminist pedagogy for library instruction. [13] both works talk about the benefits as well as the resistance to a caring and collaborative learning environment. while both are written from feminist perspectives they underscore the issue of caring and empathy which are now being recognized as fundamental to leadership success. leadership traits in dimension four are empathy, strategic vision, and commitment to collaboration. leadership at all levels is committed to technology as a public service and keenly understands its potential to transform teaching, research, and learning endeavors, forge new knowledge and preserve it for continuing access. leadership is flexible and adaptable embracing and encouraging innovation and experimentation. flexibility and adaptability are key. passionate and compassionate leadership is the new normal for advancing, not only diversity, but the mission of the university. the framework embraces selflessness and eschews self-centeredness. the new leadership is also: selfless and user centered (not selfish and ego-centric) respectful (not disrespectful) collaborative (not solitary) flexible (not rigid) outwardly focused (not inwardly focused) proactive (not reactive) this new kind of leadership is based on these core values and traits that epitomize the ability to work across and respect all cultures and points of view. it mirrors the fact that we support everyone regardless of discipline, location, or status. we can lead the way to advance global scholarship and understanding. the fourth dimension of leadership also places major emphasis on what i have coined as “flipped” leadership. just as with the flipped classroom concept, flipped leadership provides the opportunity for librarians and technologists at all levels and from all backgrounds to engage in meaningful leadership roles in the library and throughout the campus. flipped leadership increases the depth and breadth of library presence at more tables bringing deep expertise to the initiatives at hand. the 5th dimension: bring the dimensions together into the framework for library technology diversity people, content and pedagogy, embeddedness and global perspectives, and leadership combine to form the framework for library technology diversity – the 5th dimension. the end result is a dynamic approach to advance knowledge creation which is truly reflective of the diversity of thought, practice, and cultural perspectives. the framework leverages expertise, technology, and leadership to ultimately advance the human condition and provide creative solutions to global challenges through its dimensions and combinations of dimensions. the mission of the university and the library in a multidimensional world is to create knowledge. the 5th dimension fulfils this mission and “lets the sunshine in.” [14] conclusion the framework for library technology diversity and its five dimensions providess a way to look across areas of the library, the university, and the world to advance diversity and increase humankind’s ability to solve problems and create new knowledge for the future. library technology workers are ideally suited to recognize and support different ways of knowing in all of the dimensions of the framework. we need to set them free with the proper support to lead this work. notes [1] the framework to foster diversity at penn state: 2010-2015, http://equity.psu.edu/framework/, accessed february 13, 2015. http://equity.psu.edu/framework/ [2] claire mcinerney, “what can lis offer men and women in the pipeline for technology careers?” panel on re-examining issues of gender and sexuality in lis teaching, research, and service delivery presented at alise ’15 – mirrors & windows: reflections on social justice and re-imagining lis education, january 29, 2015. [3] h. lord and j.m. cohoon, “recruiting and retaining women graduate students in computer science and engineering. report on research findings and workshop recommendations,” washington, dc: cra (october 8, 2006). [4] sanford berman (1971). prejudices and antipathies: a tract on the lc subject heads concerning people.metuchen, nj: scarecrow press: xi [5] penn state icik and penn state cikard, accessed february 13, 2015. http://icik.psu.edu/psul/icik.html, http://icik.psu.edu/psul/icik/cikard.html [6] university of texas libraries human rights documentation initiative, http://www.lib.utexas.edu/hrdi/, accessed february 13, 2015. [7] lua gregory and shana higgins, information literacy and social justice: radical professional praxis, sacramento, ca: library juice press (2013). [8] judy chicago, the dinner party: a symbol of our heritage. garden city new york: anchor books (1979): 12. [9] penn state university libraries. the judy chicago art education archives, accessed february 13, 2015 http://judychicago.arted.psu.edu/ [10] judy chicago, institutional time: a critique of studio art education. the monacelli press (2014). [11] the judy chicago art education dialogue portal. accessed february 16, 2015. http://judychicago.arted.psu.edu/dialogue/ [12] barbara i. dewey, “the embedded librarian: strategic campus collaborations,” in libraries within their institutions: creative collaborations edited by william miller and rita m. pellen. new york: haworth press (2004): 6. [13] maria t. accardi, feminist pedagogy for library instruction. sacramento, ca: library juice press (2013). [14] referencing the hit single aquarius/let the sunshine in (1967) by james rado, gerome ragni, and galt macdermot originally written for hair and later released by the 5th dimension. about the author barbara i. dewey is dean, university libraries and scholarly communications, at the pennsylvania state university. she was dean of libraries, university of tennessee, knoxville from 2000-2010. previously, she held several administrative positions at the university of iowa libraries including interim university librarian. prior to her work at iowa she held positions at indiana university’s school of library and information science, northwestern university libraries, and minnesota valley regional library in mankato, minnesota. she is the author/editor of six books, and has published articles and presented papers on research library topics including digital libraries, technology, user education, fundraising, diversity, organizational change, and human resources. she holds the m.a. in library science, the b.a. in sociology/anthropology from the university of minnesota, the public management certificate from indiana university. she is on the sparc steering committee and arl’s statistics & assessment committee. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – editorial introduction mission editorial committee process and structure code4lib issue 15, 2011-10-31 editorial introduction this hallowe’en finds our contributors working away like (benign) mad scientists, assembling and deploying their creations to bring services and information in novel ways to their patrons and staff, approaching their work with a vital sprit of invention and discovery. by tod a. olson welcome to the 15th issue of code4lib journal. as our hallowe’en publication date draws near, our contributors are busily providing treats to their users. they have been working away like (benign) mad scientists, assembling and deploying their creations to bring services and information in novel ways to their patrons and staff, approaching their work with a vital sprit of invention and discovery. it is this spirit that pushes the bounds of convention and expectation, expanding the role of library it. once we were confined to catalog databases, straining against disk space and fixed record lengths. now we find that digitization, online journals, real-time dynamic information, and arguably ubiquitous computing are all ours. we have claimed them. what next shall we add to our domain? to help users select controlled vocabulary terms for web pages, shun nagaya, yutaka hayashi, shuhei otani and keizo itabashi offer up “controlled terms or free terms? a javascript library to utilize subject headings and thesauri on the web.” sparql queries are executed against the user’s choice of controlled vocabularies to return possible suggestions. after years of experience, pamela c. buzzard and travis s. teetor treat us to “best practices for a university laptop lending program,” where they detail their practices of storing and circulating hundreds of equipment items. the authors explain how they deal with physical damage and purge unwanted bits from disk while patrons still have unfettered administrative access to devices. when digitizing hand-drafted paper documents such as the frederick douglass diary, there remains no good way to automatically extract the text. this leads andrew s.i.d. lang and joshua rio-ross down the path of “using amazon mechanical turk to transcribe historical handwritten documents.” wherein unseen (human) agents are enlisted to transcribe the documents and even perform quality control, with a brief exploration of ethical concerns. in “lessons in public touchscreen development,” andreas k. orphanides guides us through developing and deploying a touchscreen kiosk. in this masquerade, a humble browser is transformed to lure in patrons with the promise of real-time library information. but how to protect the desktop from access by probing fingers? our chronicler then risks his sanity poring over logs and heat maps hoping to perfect his creation. you will be familiar with the sense of panic and dread that looms when technology goes awry, the technology staff are all out on call, and the victim is left alone, in the dark and without assistance. think back to that last system upgrade that was afflicted by gremlins, or perhaps more dire demonic influences. keith kelley, karlis kaugars and scott garrison conjure a talisman against such evil in the form of “an android/lamp mobile in/out board based on wi-fi fingerprinting.” they provide introduction to the arcana of wifi fingerprinting and show how to exorcise those phantoms which misdirect and try to obscure it staff during a crisis. budget horrors recently lead the colorado association of libraries to the near demise of their quarterly journal. in “open access publishing with drupal,” our protagonist nina mchale employs a collection of drupal modules to effect re-animation—the unsung e-journals module acting as the journal’s beating heart. and finally, please join me in welcoming ed summers, a well-respected member of the code4lib community, to the journal. ed joins the editorial committee in courting madness to bring you another issue of … code4lib journal. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – hacking summon mission editorial committee process and structure code4lib issue 11, 2010-09-21 hacking summon when the oregon state university libraries selected serials solutions’ summon as its discovery tool, the implementation team realized that they had an opportunity to implement a set of “hacks” that would improve the overall user experience. this article will explore the space between summon’s out-of-the-box user interface and full developer api, providing practical advice on tweaking configuration information and catalog exports to take full advantage of summon’s indexing and faceting features. the article then describes the creation of osul’s home-grown open source availability service which replaced and enhanced the availability information that summon would normally pull directly from the catalog. by michael b. klein history in late 2009, the oregon state university libraries (osul) discovery service task force selected summon from serials solutions to serve as its next generation discovery interface. [1] implementation took place between january and march 2010. as the implementation progressed, the osul summon implementation team discovered several aspects of the interface they wanted to customize. instead of requesting changes from the vendor, osul developed a number of “hacks” – elegant solutions that worked within the framework of summon’s existing functionality – to accomplish its desired goals. what is summon? serials solutions’ summon unified discovery service promises to deliver “instant access to the breadth of authoritative content that’s the hallmark of great libraries – digital and print, audio and video, single articles to entire e-journals, and every format in between.” [2] since its introduction in early 2009, the installed base has expanded to “95 summon customers – mostly 4-year academic institutions,” [3] serving a potential population of hundreds of thousands of library patrons. yes, but what is summon? from the perspective of an institutional implementation team, it’s helpful to look at summon as a combination of three separate components. summon is an index the heart of summon is its index. the index is based on the apache solr search platform, which in turn is a frontend to the lucene indexing engine. [4] summon’s index consists of e-book and article metadata from participating publishers, catalog records from implementing institutions, and other information harvested from institutional repositories and other digital collections via the open archives initiative protocol for metadata harvesting (oai-pmh). as of this writing, serials solutions claims to have over 500,000,000 items indexed from over 94,000 journal and periodical titles. [5] by combining information in the index with serials solutions’ knowledge of subscribers’ electronic holdings, summon is able to determine which items are part of a given institution’s print collections, which are available online as full text, which are available as online citations, and which would require inter-library or other extra-institutional requests to access. summon is an api the summon application programming interface is a rest-based web service that exposes summon’s discovery functionality. calls to the api can be used to query the index, refine existing searches, retrieve database recommendations based on a specific set of search terms, request item availability information from the local catalog, and even discover the fields and facets exposed by the index. with virtually all of summon’s functionality exposed through the api, it is possible to integrate summon with existing web sites, content management systems, courseware, or even to create a complete discovery interface from the ground up. an api tutorial is beyond the scope of this article, but since the hacks discussed below will affect results retrieved via the api as well as results that appear in the default user interface, it’s important to understand its place in the summon ecosystem. summon is a web application the most visible part of summon, and the part most users’ experiences are filtered through, is the web application that serials solutions has built on top of the api. the application provides a rich user interface with a clean search results layout, dynamic control of facets, and a “saved record” folder. since the application is hosted by the vendor, libraries are limited as to the customizations they can make to the user interface. the rest of this article will focus on leveraging the architecture of the summon web application in order to gain greater control over how certain results are displayed. it’s worth noting that the hacks described don’t intercept, change, rewrite, or redirect a single line of code within the summon application itself. instead, they involve changing the data summon sees. as a result, the reader may choose to see this less as “hacking summon” and more as “taking control of closed-source vendor applications through creative data munging.” for convenience, from this point forward, the default web application and its user interface will be referred to as “summon.” any references to the index or the api will be made explicitly by name. hacking location codes one of the first pieces of information serials solutions asks for when setting up a new summon instance is a mapping from the library location codes used in the marc records to displayable location names. the easy and obvious thing to do is to export that list from the ils and send it along. when osu libraries did just that, the result was a list of almost 110 codes, some of which represented a relatively small number of items. while the level of specificity represented by each code is helpful in locating the item, it’s much less helpful to have a list of values that long in a single facet. when narrowing a search by location, the patron is more likely concerned with what building the items are in, not which stacks on which floor. osu libraries’ solution was to ask serials solutions to re-index all of their catalog content, using a consolidated list of location codes. the 58 variations of “valley library” (including “valley docs,” “valley docs oversize,” “valley maps,” “valley maps oversize,” etc.) were reduced to just three values: “valley library,” “valley archives,” and “valley special collections.” the 15 “guin/hmsc” variants were combined under “guin library, hatfield marine science center.” this smaller list of locations leads to more efficient discovery of resources, while the specific location string – which is still displayed with the individual item record – aids in retrieval. figure 1. library location facet and item location display. hacking marc records summon’s ability to discover items in a library’s physical collection depends on importing the institution’s marc records into the summon index. during the implementation phase, a complete export of all marc records in the ils is sent to serials solutions, where the fields are mapped to the appropriate fields in summon’s index, and the entire corpus is indexed. it is up to the library to provide updates via ftp at whatever interval they choose. osu libraries provides updates once per day. the fact that updates are sent as marc files provides an opportunity to hack those records on the way out. for example, osu libraries has a large number of government documents shelved by sudoc number rather than library of congress call number. summon displays whichever number is present, or both if the record contains both. this proved to be confusing for users who would make note of the lc call number, only to find that it didn’t lead them anywhere once they arrived in the right section. the solution: strip the lc call number (fields 050 and 090) from any record that had a gov. docs location code and a sudoc (field 086). [6] hacking availability information summon’s ability to display real-time availability information for items in the local collection depends on a series of interactions between the web application, the summon api’s availability service, and (typically) the institution’s opac. a typical request/response cycle looks something like this: the client submits a search form to the summon application. the server queries the index via the summon api. the server returns results to the client as html. the client sends an ajax request to the summon availability api requesting real-time availabilities for physical items. while waiting for a response, the client displays the results, allowing the user to start evaluating them or further refine the search. the summon availability api acquires availability information from the library’s opac. to do so, it must retrieve the html page for each item one at a time, scrape the availability and location information out of the page, and assemble the data into a response. availability data is returned to the client as json, parsed, and displayed at the appropriate location(s) within the page. for print monographs, this approach works well. but for items with links in 856 fields, journals available both in print and electronically, or items like serials or government documents consisting of a large number of dated volumes, the summon view can actually be less informative than the native opac view. for example, the default screen-scraper view of osu libraries’ library journal holdings looks like this: figure 2. default availability display. several volumes of print holdings are displayed and listed as available, but there’s no indication of the true extent of the print collection, how many other volumes are available, or whether the journal is available online in any way. to provide greater control over the display of availability information, osul developed a fast, lightweight item availability service that sits between the summon availability api and the opac. summon now sends one or more requests to the osul availability service, which in turn scrapes the opac for the required information, and returns an xml response. using the new, mediated service allows osul’s summon instance to provide a more useful view of holdings and availability: figure 3. item record with enhanced availability information. serial holdings are summarized, the full extent of the collection is visible, and online access options are prominently displayed with live links. there are a number of other advantages to this approach: the osul availability service can look up multiple items in a single request, allowing for fewer round-trip requests between the summon server and the opac. since the osul availability service and the opac are on the same local network, the internet bandwidth and latency delays involved in a given availability request are greatly reduced. if the opac template changes, any changes necessary to support summon can be made quickly in house, without requiring any action on the part of the vendor. other applications can benefit from real-time, client-friendly item availability information with very little effort. the osul availability service uses a response format based on the summon availability api response format. [7] the decision to use summon’s response format was made not because of any specific technical requirement – serials solutions was prepared to work with any well-formed xml response – but because it was well-suited to the purpose. the decision provided one side benefit as well – by using a local proxy server on the development workstation, the developer was able to re-route calls bound for the summon availability api to the development instance of the local availability service. since the wire protocols are functionally identical, this allowed for real-time, in-browser testing of the availability service’s output without disrupting service to patrons. examples a simple availability request/response between summon and osul’s availability service looks like this: get /availability?s.id=b20600380&s.id=b2584345x&s.id=b25012393&s.id=b23795438&s.id=b23896188 accept: application/xml <?xml version="1.0"?> <response xmlns:xlink="http://www.w3.org/1999/xlink" totalrequesttime="0" version="0.1.8" scraperclass="oasisscraper"> <availabilityitems> <availabilities bib="b2060038" expires="2010-08-27t10:15:08-07:00" id="b20600380"> <availability displaystring="available, valley, qa76.6 .h857 2000" statusmessage="available" callnumber="qa76.6 .h857 2000" locationstring="valley" status="available"/> <availability displaystring="available, cocc second floor, qa76.6 .h857 2000" statusmessage="available" callnumber="qa76.6 .h857 2000" locationstring="cocc second floor" status="available"/> </availabilities> <availabilities bib="b2584345" expires="2010-08-27t10:12:24-07:00" id="b2584345x"> <availability displaystring="available, valley, qa76.64 .p47 2010" statusmessage="available" callnumber="qa76.64 .p47 2010" locationstring="valley" status="available"/> </availabilities> <availabilities bib="b2501239" expires="2010-08-27t10:15:08-07:00" id="b25012393"> <availability displaystring="available, valley, g70.212 .s54 2008" statusmessage="available" callnumber="g70.212 .s54 2008" locationstring="valley" status="available"/> </availabilities> <availabilities bib="b2379543" expires="2010-08-27t10:12:24-07:00" id="b23795438"> <availability displaystring="available, valley, qa63 .w55 2005" statusmessage="available" callnumber="qa63 .w55 2005" locationstring="valley" status="available"/> </availabilities> <availabilities bib="b2389618" expires="2010-08-27t10:12:24-07:00" id="b23896188"> <availability displaystring="available, valley, qa76.76.d47 r53 2005" statusmessage="available" callnumber="qa76.76.d47 r53 2005" locationstring="valley" status="available"/> </availabilities> </availabilityitems> </response> the library journal example depicted in figure 3, with entity-encoded embedded html links [8], is a bit more complex: <?xml version="1.0"?> <response xmlns:xlink="http://www.w3.org/1999/xlink" totalrequesttime="0" version="0.1.8" scraperclass="oasisscraper"> <availabilityitems> <availabilities bib="b1903187" expires="2010-08-27t10:52:09-07:00" id="b1903187"> <availability displaystring="available undefinedapr. 01, 2001-) via &amp;amp;lt;a href=&amp;amp;quot;http://proxy.library.oregonstate.edu/login?url=http://www.lexisnexis.com/us/lnacademic/api/version1/sf?shr=t&amp;amp;amp;sfi=ac00nbgensrch&amp;amp;amp;csi=242161&amp;amp;quot; target=&amp;amp;quot;_new&amp;amp;quot;&amp;amp;gt;lexisnexis:academic&amp;amp;lt;/a&amp;amp;gt;" statusmessage="available undefinedapr. 01, 2001-)" xlink:href="http://proxy.library.oregonstate.edu/login?url=http://www.lexisnexis.com/us/lnacademic/api/version1/sf?shr=t&amp;amp;amp;sfi=ac00nbgensrch&amp;amp;amp;csi=242161" xlink:type="locator" locationstring="&amp;amp;lt;a href=&amp;amp;quot;http://proxy.library.oregonstate.edu/login?url=http://www.lexisnexis.com/us/lnacademic/api/version1/sf?shr=t&amp;amp;amp;sfi=ac00nbgensrch&amp;amp;amp;csi=242161&amp;amp;quot; target=&amp;amp;quot;_new&amp;amp;quot;&amp;amp;gt;lexisnexis:academic&amp;amp;lt;/a&amp;amp;gt;" date_start="2001-04-01" status="available" link_title="lexisnexis:academic"/> <!-11 additional online availabilities deleted for brevity --> <availability displaystring="library owns v.101 no.9 undefinedmay 1976)/ valley z671 .l7" statusmessage="library owns v.101 no.9 undefinedmay 1976)-" locationstring="valley z671 .l7" status="available"/> <availability displaystring="library owns v.125, no.1undefinedjan. 2000)-present on paper. / cocc periodicals" statusmessage="library owns v.125, no.1undefinedjan. 2000)-present on paper." locationstring="cocc periodicals" status="available"/> </availabilities> </availabilityitems> </response> like summon’s availability api, osul’s availability service provides response data in xml (application/xml), json (application/json), or streaming, newline-delimited json (application/json-stream) based on the accept: header provided by the client. for example: get /availability?s.id=b1903187 accept: application/json { "totalrequesttime": 1, "version": "0.1.8", "scraperclass": "oasisscraper" "availabilityitems": [ { "expires": "tue aug 31 14:30:14 -0700 2010", "bib": "b1903187", "id": "b1903187", "availabilities": [ { "href": "http://proxy.library.oregonstate.edu/login?url=http://www.lexisnexis.com/us/lnacademic/api/version1/sf?shr=t&amp;amp;sfi=ac00nbgensrch&amp;amp;csi=242161", "statusmessage": "available (apr. 01, 2001-)", "displaystring": "available (apr. 01, 2001-) via <a href=\"http://proxy.library.oregonstate.edu/login?url=http://www.lexisnexis.com/us/lnacademic/api/version1/sf?shr=t&amp;amp;sfi=ac00nbgensrch&amp;amp;csi=242161\" target=\"_new\">lexisnexis:academic</a>", "locationstring": "<a href=\"http://proxy.library.oregonstate.edu/login?url=http://www.lexisnexis.com/us/lnacademic/api/version1/sf?shr=t&amp;amp;sfi=ac00nbgensrch&amp;amp;csi=242161\" target=\"_new\">lexisnexis:academic</a>", "date_start": "2001-04-01", "status": "available", "link_title": "lexisnexis:academic" }, { "href": "http://proxy.library.oregonstate.edu/login?url=http://search.ebscohost.com/direct.asp?db=buh&amp;amp;jid=lij&amp;amp;scope=site", "statusmessage": "available (may 01, 1976-)", "displaystring": "available (may 01, 1976-) via <a href=\"http://proxy.library.oregonstate.edu/login?url=http://search.ebscohost.com/direct.asp?db=buh&amp;amp;jid=lij&amp;amp;scope=site\" target=\"_new\">business source premier</a>", "locationstring": "<a href=\"http://proxy.library.oregonstate.edu/login?url=http://search.ebscohost.com/direct.asp?db=buh&amp;amp;jid=lij&amp;amp;scope=site\" target=\"_new\">business source premier</a>", "date_start": "1976-05-01", "status": "available", "link_title": "business source premier" }, /* 11 additional online availabilities deleted for brevity. again. */ { "statusmessage": "library owns v.101 no.9 (may 1976)-", "displaystring": "library owns v.101 no.9 (may 1976)/ valley z671 .l7", "locationstring": "valley z671 .l7", "status": "available" }, { "statusmessage": "library owns v.125, no.1(jan. 2000)-present on paper.", "displaystring": "library owns v.125, no.1(jan. 2000)-present on paper. / cocc periodicals", "locationstring": "cocc periodicals", "status": "available" } ], } ], } the availability service scrapes all holdings and availability information out of the html view of osu’s iii millennium opac in real-time. [9] holdings are prioritized in the following order: (1) full-text online resources; (2) summary holdings (if present); (3) individual holdings (if no summary present); (4) summary online resources (such as tocs, abstracts, and archival finding aids). code osu libraries’ availability service is written in ruby using the sinatra web micro-framework. [10] the source code is available under an mit license at http://github.com/mbklein/availability-server. conclusion summon has a vast index, as well as a comprehensive api that encourages active development. most institutions, however, will likely focus their deployment strategy on the vendor’s existing web application. by learning to finesse the data that summon indexes, and writing software that lives in the gaps between summon’s index and the library’s opac, customers can customize the user experience and presentation of results in a way that works best for their patrons. more importantly, there’s nothing summon-specific about any of the techniques presented in this article; any application that relies on customer-provided data can be hacked to a certain degree. while summon’s ajax architecture provides a convenient insertion point for custom live availability data, the other two hacks involve simple manipulations of exported marc records. the trick to hacking closed-source applications lies in learning how to hack the data they consume. notes [1] chadwell faye, et al. 2009. discovery services task force recommendation to university librarian [internet]. scholarsarchive at oregon state university; [cited 2010 sept 10]. available from: http://hdl.handle.net/1957/13817 [2] summon [internet]. serials solutions; [cited 2010 sept 10]. available from: http://www.serialssolutions.com/summon/ [3] nagy, andrew. personal communication, 2010 july 23. [4] welcome to solr [internet]. apache software foundation; [cited 2010 sept 10]. available from: http://lucene.apache.org/solr/ [5] summon content & coverage [internet]. serials solutions; [cited 2010 sept 10]. available from: http://www.serialssolutions.com/summon-content-and-coverage/ [6] the downside to this approach is that the lc call number is no longer indexed, so the affected items can no longer be found by searching on the lc call number. while it’s true that the same effect could have been accomplished without this compromise by asking serials solutions to change the display options for certain location codes, the osul implementation team decided the tradeoff was worth it to keep control over these kinds of display issues in house. [7] availability api [internet]. serials solutions; [cited 2010 sept 10]. available from: http://api.summon.serialssolutions.com/help/api/availability [8] embedding html in the displaystring and locationstring attributes isn’t ideal, as it limits the portability of the data to non-html systems, but it’s the only way to get summon to pass those links through to the client. as a compromise, the raw link title, status, location, and date range information are passed through in other attributes. [9] actually, for performance reasons, results are cached – 5 minutes for circulating materials, and 30 minutes for non-circulating and electronic holdings. [10] sinatra [internet]; [cited 2010 sept 10]. available from: http://www.sinatrarb.com/ about the author michael b. klein is the digital applications librarian at oregon state university. his previous experience includes a stint as the digital initiatives technology librarian for the boston public library, where he worked on digitization workflow and digital repository projects, and several years as a contract software developer for the u.s. census bureau. he enjoys elegant programming solutions almost as much as he loves clever hacks. most days, he can be found in the #code4lib irc channel on freenode trying to teach the old channel bot new tricks. subscribe to comments: for this article | for all articles 2 responses to "hacking summon" please leave a response below: extending the availability messages in summon – "self-plagiarism is style", 2011-07-03 […] a code4lib journal article — "hacking summon" — michael b. klein talks about enhancing an availability api to include extra info and […] jason, 2011-07-25 excellent article with some very accomplished solutions. its just such shame that summon is so limited in the fiirst place (compared with other discovery layers)> leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – openroom: making room reservation easy for students and faculty mission editorial committee process and structure code4lib issue 10, 2010-06-22 openroom: making room reservation easy for students and faculty scheduling and booking space is a problem facing many academic and public libraries. systems staff at the ball state university libraries addressed this problem by developing a user friendly room management system, openroom. the new room management application was developed using an open source model with easy installation and management in mind and is now publicly available. by bradley d. faust, arthur w. hafner, and robert l. seaton introduction over the past several years, the ball state university libraries has changed its image from serving the campus community as a repository of knowledge to a destination for research, learning, and friends. this was accomplished by facilities renovation and new space configuration, organizational development, and the infusion of technology in almost all aspects of the libraries’ operation. this new environment has resulted in the libraries becoming the most popular learning space on campus. the size of bracken library’s usable space over its five floors is equivalent to 6.6 football fields and includes stacks, information technology, and individual study spaces. bracken library also includes over twenty group study and meeting spaces that students, faculty, and even local community members can reserve and use for group project meetings, class study sessions, presentation review, and many other purposes. our challenge was to make it easier and more convenient for students and faculty to request and plan use of these group spaces. jumping through hoops to reserve a room for many years, the university libraries used a paper calendar, telephone calls, forms, and in-person requests to reserve and manage bracken library’s group study rooms. the master paper calendar was located at the periodical reserves microform service counter. a student had to call that service desk or stop by and complete a room request form to get a reservation on the calendar. then library users had to remember what room at what time and on which day they arranged their meeting reservation. of course, this was the best method available to control and manage use of these meeting spaces when first implemented in 1976. this legacy approach for scheduling, however, was inconvenient for users when planning and reserving space, checking on a room’s availability, or confirming reservations. the rigid process was time-bound because bracken library had to be open to book a room and place-bound because the user had to visit or call a specific service point. email requests were also used for space requests but suffered some of the same process hurdles. as the popularity and hours for bracken library increased in the latter half of the decade, we knew a web application could improve the service level and reduce staff time required for managing reservations for the group study rooms. buying vs building – why did we build it? in july 2007, we conducted an environmental scan to learn what web-based scheduling applications were available. we found that many already existed, and they ranged from very high-end, complex, expensive enterprise solutions to single room/single user desktop systems. to help explain why we decided to build our own, let’s look at some of the original functional requirements our staff handed us. empower any ball state community member to initiate an online request to use a group study room in bracken library. adhere to the university libraries’ room use guidelines on size, use purpose, requestor’s credentials, use frequency, and length of reservation. limit room reservations to hours when bracken library is open and rooms are accessible. communicate by email with a user when a request is generated. minimize the amount of work a user is required to do when completing the request form. guide users to spaces that are appropriate in size and technology for their meetings. use windows server and iis technologies. represent the entire suite of group study rooms in a single web interface. as we scanned the environment, we had a difficult time finding any scheduling software that met most of our needs. instead, we found systems that cost too much, were too simple, were too complex, ran on unsupported platforms, or used unsupported technologies. for these reasons, we decided to build the application we needed to manage our group study rooms. room scheduler version 1.0 – transfer of control to the user in january 2008, the university libraries released the online group study room reservation system (room scheduler 1.0). the web application immediately became popular with students, faculty, and others for reserving spaces in bracken library. a link prominently displayed on the libraries’ homepage (http://www.bsu.edu/library) and on the student virtual library webpage (http://www.bsu.edu/library/svl/) provided easy access to room scheduler 1.0. university community members with ball state university computer usernames could authenticate, check room availability, book a room, and go on their way. after reserving a room, the application sent the requestor an email confirmation that included essential information about the event, such as time, date, room number, and purpose. from the room scheduler’s landing page (http://www.bsu.edu/libraries/studyrooms) a user unfamiliar with bracken library spaces could preview a photo of each room, learn capacity information, and find out other physical details such as the availability of a whiteboard, flat panel monitor, or projector screen. (figure 1) figure 1. room scheduler version 1.0 landing page [view full size image] during spring session 2008, word about the room scheduler web application spread and the university libraries received several inquiries about using the system to manage spaces in other units on campus, as well as at other universities and colleges. since the application worked well, others were interested in duplicating its functionality at their location. however, although the room scheduler worked well for bracken library spaces, it was integrated tightly with the systems technology on the university libraries’ departmental web server and could not be ported easily to other environments. from room scheduler 1.0 to openroom to meet our needs and yours during the first year the room scheduler application was online, we developed a list of features that we wanted to add to it, including enhancements to improve the user experience and administrative tools for system managers. based on external interest in using the room scheduler in other environments, we also wanted to add portability to the application. for these reasons, we decided to pursue a second round of development to build a more powerful, sleeker, focused, open source version of the system. on the surface, the room scheduler application was elegant and easy to use. under its hood, however, it was not coded for easy sharing or functional expansion. to start, the application needed a more user-friendly administration module in order to be viable as an open source tool. in room scheduler, many of the controls, settings, variables, and web links needed to be changed in the code, requiring a developer to drill down into the system to make adjustments. we wanted to improve the application with a staff interface to make configuration changes easier. customization of the room reservation form was also difficult because it required php code changes. because database connection information was also hardcoded throughout the application, we needed to centralize it in a single configuration file. we viewed this as essential to make the application portable, more secure, and easier for other developers to install and customize for their environment. an unintended performance issue also developed as the schedule filled up. the time necessary to display the room availability page listing all rooms took longer and longer to generate. it was common for a user to wait 10-15 seconds for the page to display. we needed this page load rate to be more like 1-2 seconds. in addition, the room scheduler application’s authentication module used windows authentication on our local server, relying on folder permissions that had to be set manually. because this is risky and requires a server system administrator to make changes, we wanted the new version to feature a user account management module. a final obstacle to sharing this application was the interface design. in our first version, the ball state university libraries’ web page branding was integrated tightly into the system. this resulted in code that could not be changed easily to allow another unit on campus or institution to easily insert their brand into the application interface. this shortcoming itself was a showstopper for sharing the application with interested external parties. in april 2009, we began to look seriously at how to address these and other issues. during the following months, we developed what is now known as openroom 1.0. openroom 1.0 functional specifications in january 2010, the university libraries introduced openroom 1.0 to replace the room scheduler system. the openroom 1.0 software includes many improvements and upgrades: the user interface (ui) was updated to focus solely and simply on the room reservation functionality and to be more aesthetically pleasing. hard-coded references to ball state university libraries’ branding were removed. (figure 2) the day view page in openroom that shows availability of all rooms loads quickly in all instances so that page load time is only 2-3 seconds. (figure 3) as the mouse moves over a room/time cell, the details about that room are displayed in a rollover on the left side of the screen. this description follows the user up and down the display when scrolling so that one can always view details about the spaces. (figure 4) the room reservation form was improved by using a pop-up window to enter event details. (figure 5) a new administrative module enables managers to easily make system parameter changes and local customizations, like adding a room or changing room availability information, without having a developer dig into the code. (figure 6) administrators also enjoy enhanced reporting options to review and analyze room usage patterns. (figure 6) an application level ‘reminder message’ can be added by administrators and the message will appear at the top of every page in the system. (figure 6) administrators have the flexibility to create multiple reservations at once. figure 2. generic login page for openroom with no hard-coded institutional branding [view full size image] figure 3. openroom day view page showing availability of all rooms [view full size image] figure 4. details about each room are displayed in a dynamic rollover figure 5. room reservation form opens in a pop-up window figure 6. a new administrative module provides features for application managers [view full size image] for interested parties, an operational demo version of openroom 1.0 is available at https://www.bsu.edu/libraries/openroomdemo/ with username/password: admin/bracken99. openroom 1.0 technical specifications openroom 1.0 requires a web server (iis or apache), php version 5 or higher, and mysql version 5 or higher. in addition, to use the ssl or ldap optional features, an ssl certificate and a directory server, such as microsoft active directory, is required. the user authentication mode must be determined at installation, and it cannot be changed later. a modular programming structure was used to make application updates and program add-ons easier to develop and implement. any number of new modules can be added  anywhere on the page, providing other developers more control over their own openroom installation. a theme control file manages the system design and the interaction and layout of the modules. each module operates independently of the others. the best comparison is to the wordpress or blogspot theme control. examples of modules include the calendar day navigation element, the room request form, and the room legend. these modules can also be easily removed. if, for example, for some reason you do not need the room detail window (figure 4) to display, that module can be removed. at installation, a single control file holds a number of the system variables that are used to customize the user experience and system settings. these data points are added to the system database and then managed using the administrator module after setup. the installation process for the openroom web application is documented at http://www.bsu.edu/libraries/openroom/installationguide.pdf. installation and setup is similar to the process used to setup an application like wordpress. the most difficult part of the migration from the room scheduler application to the openroom application was restructuring the way the data was stored in and retrieved from the database and then making that data useful and available  to outside sources via xml. this involved a complete overhaul of the database structure. for better portability, the authentication methods in the system were standardized and new internal credential system and access control methods were developed. openroom licensing decision from the outset of development of openroom 1.0, we kept in mind that the system needed to be portable and that we wanted to share it with other units on campus as well as other colleges, universities, and organizations. we anticipated it being open source software since we had requests from both onand off-campus constituents to use the application. on campus today, a second instance of the system is used by students and faculty to reserve dance and theater rehearsal space in the college of fine arts. as we investigated open source software licenses, we discovered that there are 2 major categories of open source/free software licenses: gpl compatible and gpl incompatible. within each of these categories, there are more than twenty individual license options from which to choose. a recent ieee software journal article noted that the open source initiative certifies fifty different licenses [1]. we discovered some institutions had released locally developed software under the gnu general public license (gpl). gpl protects the developer, allows others to download and customize the program for non-commercial use, and allows us to customize the program for our purposes on our schedule [2]. we determined that the mrbs meeting room booking system (one of the free resource booking systems we evaluated) is available under gpl.  we reviewed how it is made available on source forge, and really like that scenario as a model. for these reasons, we have decided to share openroom, including source code, with a gnu gpl license. openroom is currently available from the ball state university libraries web site as a zip file at http://www.bsu.edu/libraries/getopenroom/. conclusion the ball state university libraries have found that openroom 1.0 is a great resource for students and faculty to find and reserve the popular group study and meeting spaces that are available in bracken library. since its release at the beginning of january 2010, openroom has processed more than 8,500 room use requests. like its predecessor, it empowers students and faculty with the flexibility to plan meetings, request space, and notify colleagues from any internet enabled computer, anytime. compared to phone calls, paper forms, random email, and service point visits, this convenient access to valuable study and meeting spaces for students and faculty helps make the university libraries a destination for research, learning, and friends. the application also performs many repetitive tasks that staff and student assistants had done manually in the past, freeing their time to perform other more important duties. the ball state university libraries’ library information technology services unit exists to provide the hardware and software solutions needed for the university libraries to operate and serve the academic community efficiently and successfully. sometimes building a solution is the best course of action to accomplish this mission. the popularity of the openroom system with staff managing other spaces on campus demonstrates the value outside the libraries for the work we have done. we will definitely consider application sharing and an open source solution on future projects. references [1] engelfriet, a. (2010). choosing an open source license. ieee software, 27(1), 48-9. retrieved april 8, 2010, from applied science & tech index database. doi: 10.1109/ms.2010.5. (coins) [2] arch, x. (2010). under the hood – buy, build or borrow. against the grain, 22(2). 13-14. (coins) about the authors bradley d. faust is assistant dean for library information technology services at ball state university. he has served as cio in the university libraries since 1992. email: bfaust@bsu.edu arthur w. hafner, ph.d. is dean of university libraries at ball state university. previously the dean of university libraries at seton hall university, dr. hafner has been at ball state university since 2002. email: ahafner@bsu.edu robert l. seaton is web development analyst in the university libraries at ball state university. robert has served in this position since 2006. rlseaton@bsu.edu subscribe to comments: for this article | for all articles 5 responses to "openroom: making room reservation easy for students and faculty" please leave a response below: shaun ellis, 2010-06-22 congrats on what seems like a great tool. we developed something similar at rutgers university libraries, called bookroom. unfortunately, we did not make the source available. here’s a link to the “public” side of the system: http://libserv4.rutgers.edu/scheduler/ brad faust, 2011-06-17 the openroom web app from ball state university libraries is now a project over at sourceforge.net. if you are interested in contributing to the development of openroom, join the project. pi (weekly) « pi en second life, 2012-08-25 […] the code4lib journal – openroom: making room reservation easy for students and faculty […] camela kinslow, 2014-08-13 are the rooms left unlocked and preference given to reservations thank you. carmela kinslow kresge library notre dame law school 574 631-5990 book a nook — metalab (at) harvard, 2016-07-12 […] / bibliography openroom: making room reservation easy for students and faculty, code{4}lib exposing library services with angularjs, […] leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – supporting oral histories in islandora mission editorial committee process and structure code4lib issue 35, 2017-01-30 supporting oral histories in islandora since 2014, the university of toronto scarborough library’s digital scholarship unit (dsu) has been working on an islandora-based solution for creating and stewarding oral histories (the oral histories solution pack). although regular updates regarding the status of this work have been presented at open repositories conferences, this is the first article to describe the goals and features associated with this codebase, as well as the roadmap for development. an islandora-based approach is appropriate for addressing the challenges of oral history, an interdisciplinary methodology with complex notions of authorship and audience that both brings a corresponding complexity of use cases and roots oral histories projects in the ever-emergent technical and preservation challenges associated with multimedia and born digital assets. by leveraging islandora, those embarking on oral histories projects benefit from existing community-supported code. by writing and maintaining the oral histories solution pack, the library seeks to build on common ground for those supporting oral histories projects and encourage a sustainable solution and feature set. introduction: institutional context the digital scholarship unit (dsu) at the university of toronto scarborough (utsc) library works with campus it to design stable best-practices infrastructures that can support the research conducted by and with faculty on campus as well as digitized and born-digital special collections. the dsu also works with liaison librarians to provide consultation and development services for experimentation and co-curricular pedagogy. activities undertaken by the unit include helping stakeholders preserve and manage data, teaching digital scholarship tools, providing training on issues relating to scholarly publishing and open access, coordinating access to repository and data services developed by the central university of toronto libraries system, and giving advice on metadata development and management. utsc is 50 years old, and prides itself on a relationship with multimedia innovation, having originally been built with a television studio to develop and transmit closed-circuit lectures (utsc timeline). today, several courses at utsc incorporate the creation of audiovisual materials into the assignments completed by undergraduate students and researchers seek the means to develop and analyze unique collections of video and audio materials under the banner of oral history. in 2014, the library embarked on an expansion of the services of the digital scholarship unit, and several faculty approached the library seeking support for oral history projects. scoping oral history support – requirements oral history is defined more as process than as product, “a field of study and a method of gathering, preserving and interpreting the voices and memories of people, communities, and participants in past events” [1]. the breadth of this definition can challenge those tasked with developing infrastructure, software, and workflows to make possible this “gathering, preserving and interpreting” work. teams working on oral histories commonly request abilities to do the following: relate born-digital and surrogate materials drawn from an archive together in a non-exclusive fashion (graph-type relationships) manage materials and administrative processes as required by research ethics boards and granting agencies. materially this means granular permissions, long-term stewardship, and reliable access and discovery of materials describe, transcribe, and translate audio and video files it is also important to note that these features need to be flexible enough to address the unique research and/or collection building goals of each project. relevant materials can be in multiple formats and contain any number of speakers, and research questions can require, for example, that annotation rather than transcription information be appended to objects and indexed for discovery. as projects evolve, it’s also possible that new data sources will be included, enriched and otherwise integrated into pre-existing data sources. in the end, the processes of oral history appear to defy any consistent structure or approach. as a result, a software platform that must address the needs of multiple groups needed to be highly flexible and configurable. the utsc library found that islandora provided ways of addressing most of these requirements. relating born-digital and surrogate materials together oral history project teams often ask for the ability to link audio and video files with other born-digital and surrogate materials for both administrative and historical context. for example, a recording of an interview might be associated with consent forms agreeing to preservation and publication of all or parts of a recording, or an interview may be associated with other files contextualizing the content of the interview for an audience. recordings might also be linked together to form an exhibition or collection of interviews on a particular topic. these connections may be non-exclusive – a single article or image may, for example, be relevant to two recorded interviews. using islandora solution packs, it is possible to support ingest and display of a number of content types. islandora solution packs are drupal modules that offer content models, workflows, forms, and integration with other solution packs or third party software projects to serve the needs of particular types of content or a certain user base. solution packs vary quite widely in their scope and application[2] and often leverage other solution packs or contributed drupal modules as dependencies. moreover, the underlying fedora 3 architecture supports complex and graph type relationships between objects (these functions are further enhanced and extended in fedora 4) and there are interface mechanisms for linking objects together in compounds and collections, and sharing objects in multiple relationships. the ecosystem of solution packs allowed us to utilize a great deal of existing functionality and extend it for our purposes in storing, displaying, and interrelating multimedia content. managing materials and administrative processes a common request from utsc researchers is that oral histories be available to the source community in which they were collected, an audience that may be located internationally and may have permissions to some materials and not to others. an academic audience may seek and use a collection of oral histories differently from members of the general public, and these different modes of access require granular permissions for access. in islandora, fedora’s xacml-based security is highly configurable, as is the role-based drupal permissions system. managing not just access, but also authorship and administration, islandora allows for fairly complex use cases to address the (often fraught) notions of authorship inherent in oral history work. the oral history materials themselves, often containing video and audio files, can be difficult to preserve and serve reliably. codecs change and these files tend to be large in size. it is widely recognized that the problem of preserving multimedia materials will continue to grow, as the availability of recording equipment makes generating these materials ever easier. the islandora video and audio solution packs produce several formats, an access copy, and can be configured alongside utility modules to create and store technical, administrative, and preservation metadata. describing, transcribing, and translating audio and video files we recognized that our need to flexibly describe and index oral histories could be addressed using islandora’s form building modules (xml forms) and search (solr) functions. islandora’s form building modules (xml forms) [3] allow for the creation of form templates that can be used to author xml datastreams in a fedora object. a user can create a new form, associate it with a content model in fedora, and it will become available to users creating a new instance of an object using the content model. once the xml datastream has been created, the gsearch application leverages an xsl stylesheet to transform the new xml into solr documents, and make the information contained available to islandora search functions [4]. indexing new xml elements often requires an extension to the xsl stylesheet to address any new elements not covered in the default xsl packaged with solution packs. while the forms and search infrastructure provided a lot of functionality, there was no islandora mechanism available for researchers who wish to describe, transcribe, or translate interview content and associate it with time codes. beyond the research applications, utsc is also bound by the accessibility for ontarians with disabilities act. this act requires compliance with web content accessibility guidelines 2.0, and includes a requirement for text representations of non-text content. given islandora’s coverage of many of the other requirements identified as essential in an oral histories infrastructure, the group determined that it would be best to develop an additional solution pack that provided these additional transcription, translation, and description features. they were bundled (with some hubris) by the unit as the “oral history solution pack.” the islandora oral history solution pack the oral history solution pack (ohsp) has existed in beta form since 2015, and was recently accepted into the islandora foundation labs repository. the intention is to continue the work to bring the module into the foundation and release it as part of a regular islandora release. those familiar with islandora will be familiar with the workflow by which one or more source files are transcoded and ingested as islandora datastreams in an islandora object: for interface ingestion, users are presented with the option of creating an “oral history” object, and asked to provide a file and fill out descriptive metadata. the object is uploaded and triggers a series of processes that transcode the source file to one or more derivatives, creates relevant datastreams, and provides feedback to the user once the object has been created. the ohsp follows the same pattern, making available an oral history specific content model to collection policies, and leveraging islandora’s video and audio solution packs as dependencies to generate derivatives. the module is designed to operate with the rest of islandora’s released solution packs and utility modules, and the intention is to provide substantial flexibility in implementation. a major contributor to the concept for implementation is edward garrett of pinedrop, whose transcript drupal modules were discovered early in the research phase for this project. transcript ui leverages the concept of “tiers” (also used in transcription tools such as elan) to encode various layers of information about an audio or video file (including transcription, translation, and description). the viewer edward garrett was contracted by the university to abstract the viewer functions from his module into the transcript ui module, which is a dependency and sub-module for the oral history solution pack. garrett’s viewer design makes possible the display of information tiers. in figure 1, it is possible to see how an annotation and transcription tier are both available from the drop-down menu. the names and number of tiers are highly customizable. in figure 2, it is possible to see how individual tier “cues” are highlighted as a video plays, in figure 3, it is possible to see user options for the closed captioning functions of the viewer, out-of-the-box elements of the video.js player used at the heart of the viewer and fed using islandora data. note that users not using a bootstrap theme need to include a custom bootstrap build targeting the oral history elements in order for the viewer to display properly (an example is packaged with the module). figure 1. viewer configuration by user figure 2. highlighting and cc on video figure 3. available cc options in player the viewer displays captions in the video.js player using a simple webvtt file timed track element; transcript cues are parsed and synchronised via html5 video events and api. a webvtt file is either provided at ingest, or generated via a source xml file. when installing and configuring the ohsp, administrators have the option to turn off the creation of webvtt files, to enable cc for display, to enable the transcript display, and to display the media and transcript side-by-side (rather than the default, which is to display the transcript underneath). administrators can also configure the speakers and tiers in source files for display as you can see in figure 4. at the top, you can see that two custom tiers were used in the source xml for the transcript. here in the interface, an administrator can define the human-readable name that will be associated in the viewer drop-down menu with the tier provided in the source xml. similarly, speaker name elements can be glossed with more human readable names using a typical drupal convention of machine name|human-readable name. this configuration panel is designed to enable users to flexibly define the nature of their source transcript files. figure 4. configuring transcript tiers and speaker names for display. ingest formats and workflow to serve the functions of the viewer, the ohsp supports the addition of enriched source material and produces additional derivatives on ingest. in addition to a source video or audio file, users ingest time-coded content files in either a) a custom xml format or b) a webvtt format. the xml format is required to support tiers. an xsd schema is provided with the module so that people creating xml files can ensure they are producing a valid format.[5] the custom xml format has a <cues> root element containing a number of nested <cue> elements. each <cue> consists of a <start> and <end> element indicating the timecode for each cue, and an optional <speaker> element. each cue can contain multiple customizable tier elements (such as transcription, translation, and annotation). if tiers are used, they must be used in every cue. for instance, if a <translation> element exists in one cue, it needs to exist in every other cue even if it is empty. in addition, the cue tiers need to appear in the same order in each cue. the xsd schema does not account for these tier customizations. tiers are identified via the use of tags when authoring an input xml file. see an example of the required xml format in figure 5. further discussion of methods for creating source xml files is provided in the implementation section. webvtt is a w3c standard that provides a very simple time-encoded text format to display captions or subtitles. it was developed to support the html5 <track> element for timed text tracks for closed captions and subtitles in the <video> element. currently, modern browsers and popular html5 video players such as video.js only parse simple webvtt format, which is suitable for simple use cases like closed captioning/subtitles to meet web accessibility requirements. this means that although the webvtt specification technically allows for the inclusion of structured json or xml data snippets, a feature which could have been exploited to capture multiple data tiers within a single webvtt file, the decision was made to continue with a custom xml format and limit use of webvtt to closed captions and subtitles in the viewer. <cue> <speaker>dorjee lhatoo</speaker> <start>36.0</start> <end>45.0</end> <transcript> poor company with me. she would pour me a glass, one sip and it goes right through, you see -</transcript> </cue> <cue> <speaker>interviewer2</speaker> <start>46.0</start> <end>49.0</end> <transcript> [indiscernible] on fire. she sounds like great fun.</transcript> </cue> <cue> <speaker>dorjee lhatoo</speaker> <start>49.0</start> <end>52.0</end> <transcript> and then she used to bidi</transcript> </cue> <cue> <speaker>interviewer2</speaker> <start>52.0</start> <end>53.0</end> <transcript> oh!</transcript> </cue> figure 5. sample snippet of xml when a source xml file is ingested, the ohsp assigns the transcript datastream id, and crosswalks the content to an additional mediatrack datastream in webvtt format. see a sample of webvtt content in figure 6. 00:13.480 --> 00:17.017 anna: it can also be used to carry supplies such as cargo 00:17.017 -->00:25.926 anna: food, materials or anything that can be carried to the international space station and to work up there. figure 6. sample webvtt format a flowchart depicting the ingest process is provided in figure 7. figure 7. ingest workflow support for ingesting webvtt support for ingesting webvtt files natively is a relatively new feature. if a single webvtt is ingested, it will be used to generate a transcript display. multiple webvtt files can be ingested and will appear in the closed captioning portion of the viewer, but only a single webvtt file can appear in the transcript viewer itself. there are no current plans to expand this feature based on local use cases, though this may be a useful feature. all webvtt files ingested with the source media file are stored with the mediatrack datastream id (dsid). in the case of multiple webvtt files, a language code (such as _fr) is appended to the end. supporting webvtt ingest was a community requirement, but it substantially expands the use of the module for those who may be coming from other systems and who already have a variety of source files containing time-encoded transcripts. for example, webvtt files can be crosswalked from other popular formats for captioning such as srt (subrip). one shortcoming of the native webvtt ingest has to do with the inability of solr gsearch (used by islandora) to index this file format. left with the option to cast webvtt as plain text or xml, the decision was made to create an additional xml derivative datastream for indexing. this datastream has a dsid of indexmediatrack_languagecode, with the language code being drawn from the source webvtt file. additional development may be required on this front in order to more fully flesh out this feature and ensure consistency between oral histories using source webvtt or xml files. our unit is committed to continuing to refine these functions and making every attempt to ensure backwards compatibility for both utsc’s local installation and the installations of others. existing implementations reveal the flexibility of the software as it is currently being deployed by the library and in the community. implementation details despite the fact that the module has yet to be included in a formal islandora release, a recent islandora listserv discussion revealed active installations at michigan state university (us); hamilton college (us) (the digital humanities initiative (dhi)); italy’s national research council (consiglio nazionale delle ricerche) and the hagley museum and library (us) [6]. this suggests that the initial goal, to develop a tool that was generalized and flexible in implementation, is at least partially successful. feedback has included a desire to streamline the creation and ingestion of transcript files in xml and webvtt format. currently, many desktop tools make it possible to create these files. at utsc, inqscribe and audacity are common tools, though there has been some exploration of how to use youtube to create these files. the format required for transcripts is simple enough to use any common transcript editor. a good amount of user documentation has been created to guide people through the process, and is available in github.[7][8] the utsc community is currently using inqscribe[9] or audacity[10] to manually create tracks for ingest. michigan state reports using audiogrep[11] to automatically create transcripts, and the hagley museum is creating their timed transcripts in ohms[12]. at utsc the workflow is to take the outputs of inqscribe and audacity as .csv or text files (depending on the source application) and use openrefine’s[13] template export feature to create the appropriate xml file. more detailed documentation about using these options to create timed transcripts is available in the module documentation. the size of source media files has also posed an issue for utsc’s local installation, and a php script was developed to provide an alternative model for the batch ingest of large media files. this script is also available in github[14]. when using the script, oral histories are placed in directories and ingested directly into fedora using a foxml file template and curl. arguments are passed using the command line (example figure 8) php fedora_ingest.php user=fedoraadmin pass=fedorapassword url=http://localhost:8080/fedoraurl ns=demo cmodel=islandora:sp_videocmodel collection=demo:collection target=/absolute/path/to/ingest/directory log=/absolute/path/to/ingest/directory email=admin@example.com figure 8. sample command line call to php script for alternative ingest path. at utsc, two digital scholarship projects and one course are leveraging the solution pack, serving scholars and students in history, women’s studies, and sociology. for classroom use, students are submitting packages that are further prepared and ingested by the course instructor or ta. typically, the creation of the source xml or webvtt files is the largest barrier to entry for users of the software. for some time, project members have been requesting the ability to author and publish media transcription, description, and annotation directly within the web interface. initial work sought to develop these features directly in the oral history solution pack, but this work has recently been abstracted to form the base of a new utility module that aims to facilitate annotation across islandora repositories. future directions – web-based annotation supporting web-based annotation is a priority feature identified by scholars at utsc, including scholars that are not actively engaged in oral history projects. after discussion and research, it was determined that a common approach to annotation across islandora-hosted content types would present the best option for adhering to the w3c web annotation data model. the web annotation data model[15] is the recommendation produced by the w3c web annotation working group that provides a standard way of dealing with annotations online. in this model, “an annotation is a rooted, directed graph that represents a relationship between resources” [web annotation data model]. this model leverages a concept of bodies, targets, and motivations. below is an example of an annotation resource, serialized in the preferred json-ld format: "@context": "http://www.w3.org/ns/anno.jsonld", "id": "http://example.org/anno5", "type":"annotation", "body": { "type" : "textualbody", "value" : "<p>j'adore !</p>", "format" : "text/html", "language" : "fr" }, "target": "http://example.org/photo1" } figure 9. sample annotation resource in json-ld the specification builds upon rest principles and the linked-data platform recommendation, meaning it will natively work with fedora 4, which is being utilized in the next generation islandora (islandora claw). in the current fedora 3 framework, islandora objects will be created with relationships to the source object. however, the web annotation solution pack is conceptualized as a claw-facing project, in that development assumes a migration of content and features to drupal 8 and fedora 4 as the claw project becomes production ready. the unit is open to collaboration with others interested in a utility module to serve islandora viewers and adhere to the emerging web annotation data model. conclusion open scholarship is best served by generalized software tools that can be configured to serve diverse needs while being collaboratively maintained. the ohsp is an attempt to further and facilitate a conversation about how institutions will support oral histories projects. the module furthers utsc library’s goal of aiding in the creation of oral histories and the stewardship of newer forms of scholarship, as well as addressing the realities of the ways that events are witnessed and recorded and how these affect the professional commitments of libraries to preserve a shared cultural heritage. acknowledgements lingling jiang and edward garrett authored much of the original code for the oral histories module and jiang was an author of the or2015 presentation from which this paper drew some of its material. about the authors marcus emmanuel barnes is a software developer in the university of toronto scarborough library’s digital scholarship unit. with over eight years of web programming and application design experience, marcus specializes in creating tools that enhance academic libraries for faculty, staff and students. natkeeran ledchumykanthan is a software developer in the university of toronto scarborough library’s digital scholarship unit. nat has more than eight years of full stack development experience (lamp, php, java, javascript, xml technologies) and is an active volunteer for community digital preservation/library projects. kim pham is the digital projects and technologies librarian in the university of toronto scarborough library’s digital scholarship unit and a liaison librarian for physics. she works on a bunch of great projects, rock climbs, and loves meticulous metadata review as well as code testing. she also has an awesome dog named merle, and she is definitely going to see you at the conference. kirsta stapelfeldt is the coordinator of the university of toronto scarborough (utsc) library’s digital scholarship unit and liaison librarian for computer science. she has been working with islandora and managing digital scholarship/collections projects since 2010. previously at the university of prince edward island, kirsta moved to utsc in 2014 and maintains active connections with islandora through participation on the roadmap committees and islandora board. notes [1] oral history: defined. new york (ny): oral history association; [cited 2016, nov 30]. available from: http://www.oralhistory.org/about/do-oral-history/ [2] solution packs. charlottetown (pei): islandora foundation; [cited 2017, january 2] available from: https://wiki.duraspace.org/display/islandora/solution+packs [3] xml forms. charlottetown (pei): islandora foundation; [cited 2017, january 2] available from: https://github.com/islandora/islandora_xml_forms [4] islandora solr search. charlottetown (pei): islandora foundation; [cited 2017, january 2] available from: https://github.com/islandora/islandora_xml_forms [5] xml validators. toronto (on): digital scholarship unit; [cited 2017, january 2] 6] oct 21, 2016; subject: oral histories solution pack as an islandora component https://groups.google.com/forum/#!topic/islandora/erzu08ndja0 [7] generating transcripts. toronto (on): digital scholarship unit; [cited 2017, january 2] https://github.com/digitalutsc/islandora_solution_pack_oralhistories/blob/master/docs/administrator/01-creatingtranscripts-transcriptionsoftware.md [8] introduction for administrators. toronto (on): digital scholarship unit; [cited 2017, january 2] [9] inqscribe https://www.inqscribe.com/ [10] audacity http://www.audacityteam.org/ [11] audiogrep https://github.com/antiboredom/audiogrep [12] oral history metadata synchronizer http://www.oralhistoryonline.org/ [13] openrefine http://openrefine.org/download.html [14] fedora video ingesting script https://github.com/digitalutsc/fedora_video_ingesting [15] https://www.w3.org/tr/annotation-model/ references a vocabulary and associated apis for html and xhtml: media events. w3c. [cited 2016, nov 30]. available from: http://www.w3.org/tr/html5/embedded-content-0.html#mediaevents. how to make websites accessible. government of ontario. ottawa (on) [cited 2016, nov 30]. available from: https://www.ontario.ca/page/how-make-websites-accessible oral history: defined. new york (ny): oral history association; [cited 2016, nov 30]. available from: http://www.oralhistory.org/about/do-oral-history/ utsc timeline. toronto (on): university of toronto scarborough; [cited 2016, nov 30]. available from: http://www.utsc.utoronto.ca/aboutus/historical-timeline solution packs. charlottetown (pei): islandora foundation; [cited 2017, january 2] available from: https://wiki.duraspace.org/display/islandora/solution+packs web annotation data model: w3c candidate recommendation 22 november 2016.. w3c. [cited 2016, nov 30]. available from: https://www.w3.org/tr/annotation-model/ webvtt: the web video text tracks format: draft community group report 31 january 2015. w3c. [cited 2016, nov 30]. available from: http://dev.w3.org/html5/webvtt/. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – diva.js: a continuous document viewing interface mission editorial committee process and structure code4lib issue 14, 2011-07-25 diva.js: a continuous document viewing interface diva.js is a multi-page browser-based document viewer designed to present high-resolution digitized document images as a continuous, scrollable item. this article examines the current state of the art in online document display technologies, and presents a list of functional requirements the authors used to guide the creation of this new online document viewer. the authors then discuss the image processing infrastructure necessary for deploying the diva.js viewer, and present a brief discussion of how the viewer is currently deployed in their organization. by andrew hankinson, wendy liu, laurent pugin, and ichiro fujinaga introduction diva (document image viewer with ajax) is a multi-page document image viewer, designed for digital libraries to present documents as a single, continuous item, rather than the traditional method of presenting page images one at a time. using “lazy loading” methods for only loading those parts of a document that are currently being viewed, it presents a quick and efficient way of displaying hundreds of page images from digitized books and other documents on a single web page. the diva project centers around diva.js, a unique jquery plugin that provides a user interface for viewing document images. the design of this component was informed by a survey of existing document image viewing interfaces, and from this survey a number of functional requirements for how document images should be displayed in a browser were formulated. while the core of the diva project is the jquery plugin, other image processing infrastructure is required to serve the images in an efficient manner. we discuss how the components of this infrastructure work together in the “how it works” section. as an open-source project, we welcome feedback and code contributions. we are currently using diva as part of an ongoing research project in digital music documents, but are interested in how it can be used for other document digitization projects in libraries and archives. history and motivation diva began as part of a project by the swiss working group for the répertoire international des sources musicale (rism) project [1]. rism is an international initiative, founded in 1952, whose purpose is to identify and catalogue musical sources held in libraries and archives around the world. the rism database contains extensive information about each source in its catalogue, and is an important tool for discovery and source verification for music researchers. in 2008, the swiss working group for the rism project was granted funding for an exploratory project in digitizing musical sources (prints and manuscripts) from swiss composers held in libraries and monasteries across switzerland. the goal of this project was to combine the extensive, research-quality metadata gathered over decades with high-quality images of the source itself, and publish these online in a freely accessible database. this would allow researchers to access the sources globally, and provides an important component for the preservation and promotion of swiss national musical heritage [2]. at the beginning of this exploratory project, a partnership between the swiss rism and the distributed digital music libraries laboratory (ddmal) at the schulich school of music of mcgill university was tasked with identifying the tools and interfaces that would be used for displaying the source images in a web browser. to identify the current “state of the art” we conducted a survey of 24 digital libraries. based on this survey, we identified a list of functional requirements that we wanted in a document image viewer that was specifically designed for displaying musical sources. this survey is discussed extensively by hankinson, pugin, and fuginaga (2009) [3], but we present a brief summary here. survey we formulated five functional requirements from our survey in response to what we considered to be the best and worst features of current digital document image displays. while they were developed in a musical context, we think that they are broadly generalizable over all document types. 1. preserve document integrity one of the most common methods for displaying document images online is in an “image gallery” format, where users are required to click “next” and “back” hyperlinks to navigate page by page through the document. some interfaces provide users with the ability to navigate to a specific page, either using a drop-down selection, a text box, or a series of hyperlinked page numbers. this method of display does little to preserve the original document as a cohesive entity. there are several alternatives to this approach. the google books project [4], perhaps the largest and most well-known book digitization project, presents items as a single scrollable entity embedded within the webpage. this allows users to very quickly scroll and navigate through the entire work. the internet archive [5] method uses a book metaphor where users are presented with images of facing pages and can navigate through the book by “turning” the pages. this method presents some usability challenges for quickly navigating through a source, but it does provide a very accurate representation of the original source layout. 2. allow side-by-side comparison of items music, especially in older sources, is often published as “parts,” where each musician’s score contains only the music for their instrument or voice. these parts are bound in separate physical books or leaflets. as such, reconstructing the entire musical piece requires viewing multiple books simultaneously. providing the ability to view two or more sources simultaneously, therefore, is important for comparing sources or co-reading a source along with its criticism or annotations. 3. provide multiple page resolutions for older materials, especially manuscripts, the ability to view details such as faint pencil markings or different ink colours is necessary. high-resolution images allow the user to identify small details, which is especially important in tasks such as reading cursive script or identifying faded or obscured portions of the image. on the other hand, smaller low-resolution images give users the ability to very quickly get an overview of the entire document. the ability to move between highand low-resolution images is a feature of many existing digital library systems, with many choosing to display three image resolutions: “thumbnail” for overviews of the entire document, “browser safe” for displaying individual pages that fit on a users’ screen, and a “high resolution” view for detailed image study. the state of the art for this type of document viewing can be seen in systems like the world digital library [6] [figure 1], where the user selects from a list of thumbnails provided in the left column to zoom in to see details in the full-page display. figure 1. world digital library. 4. optimize document loading presenting large page images, especially at higher resolutions, makes it difficult for users to quickly leaf through a document. two common approaches are to allow users to download the high-resolution images one at a time, or have them download the entire document encapsulated within a container format like pdf. for ultra-high resolution documents, however, this download may involve hundreds of megabytes, and the user must wait until the entire document is downloaded before viewing it. the google books and internet archives interfaces use optimization techniques that load page images only on demand. as a user scrolls through the document, the browser sends an asynchronous http request (“ajax”) to the server to deliver the images for the pages that the user is currently viewing. this can be further optimized for images at very high resolutions by borrowing a technique from mapping sites where high-resolution satellite photos are broken into smaller tiles, which may then be served to the user in parallel. 5. present item and metadata simultaneously while not part of the document viewer itself, the ability to view the item and its catalogue metadata simultaneously is a highly desirable feature. this requirement takes on a more significant role when integrating the viewer into a larger context, and points to the need for a document viewing system that can be easily integrated into existing digital library environments, including an integrated set of controls or callbacks for manipulating page zooming and scrolling. existing implementations although we compiled our list of functional requirements by examining features from existing implementations, we could find no single existing implementation where all features were present. as part of our survey, we examined some standalone document viewing frameworks, including zoomify [7], the iipmooviewer (and variants) [8], and the seadragon ajax viewer [9]. we determined that these viewers were designed as generic high-resolution image viewers, and as such did not present a document as a complete cohesive entity. google books has a promising document image viewer, but it is currently not available as an open-source project. at the time of our survey, the internet archive used the djvu format and a page-turning book interface that ran in the browser as a java applet. they have since developed a new bookreader [10] interface, currently the only other open-source implementation known to encapsulate many of our functional requirements. their interface offers many interesting features, such as contact sheet and facing-page views. however, their implementation does not use a tiling system for downloading page content, meaning that at higher resolutions the entire page is downloaded to the browser as a single image. towards diva.js our first implementation was written as a component to a larger extjs interface for managing digital images, part of an in-house image system for the swiss rism. it was not released publicly, but it has been adopted by a number of other projects related to the rism. while the extjs component worked very well, it required loading the entire extjs library, including superfluous components, which added significantly to the loading time for pages. the decision was made to port the existing system to jquery, a widely used javascript framework that is smaller and more modular in design, with faster load and execution time than extjs. development on the jquery plugin began in february 2011, using the same design requirements as the original extjs component. in the next section we describe the components of a complete diva.js installation, including image processing, server infrastructure, and the features of the jquery plugin. how it works the most visible and unique component of this project is the javascript-based document viewer itself, but this must be backed by efficient server software to process and deliver multi-resolution images to the user. a brief overview of how these components fit together is given in figure 2. figure 2. the components of the diva.js infrastructure. image processing to switch between multiple sizes of page images, diva uses a multi-resolution, or “pyramid” image format [figure 3], where the original high-resolution image files are pre-processed into several sizes of smaller resolution images that are themselves sub-divided into an array of tiles of a given size. we have found tiles of 256×256 pixels work best. these images can be created by several open-source image-processing utilities. we have found that the vips [11] image processor is the most efficient and provides the most reliable results, but imagemagick [12] could also be used. figure 3. multi-resolution, or “pyramid” image structure. there are two popular image formats that provide multi-resolution image capabilities, tiff and jpeg2000. while jpeg2000 is more modern and arguably provides more fine-grained control over the number of image resolutions that can be served, there are no open-source libraries for jpeg2000 image processing and serving. the kakadu [13] library, currently the most efficient and best supported jpeg2000 library, requires a licensing fee for implementers. for version 1.0 of the diva project we chose to support only multi-resolution tiff images, but jpeg2000 support is scheduled as a desired feature for the next version. in the multi-resolution tiff format, the number of image resolutions available for a given image is determined by the maximum resolution of the original file and the size of the individual tiles. the formula for calculating the number of resolutions (n) available for a given image size is given in equation 1. equation 1. calculating the number of resolutions based on image and tile sizes. the largest dimension, d, of the image (width or height) and the tile size, ts, are both given as pixels. for example, an image with the largest dimension of 4120 pixels and a tile size of 256 pixels would be processed into six zoom levels. a python script, process.py, is available in the diva repository that will process a directory of images into multi-resolution tiff files using the vips python module. tile image server the tile image server takes a request for a segment of the image at a given resolution and returns that portion of the image as a tile. it runs as a fastcgi process and can be integrated into most popular web servers. an example request to the tile server might look like: http://example.com/iipsrv.fcgi?fif=/tmp/image-75.tif&jtl=2,0 the jtl parameter in the request specifies (resolution,tile) which, in this case, is the first tile from the third resolution (resolution numbers are zero-indexed) for image-75.tif. although there are several potential image tile servers, we currently only support the iip image server [14]. while the iip image server presents a very fast method of processing and serving image regions, we found that a single fastcgi process could not provide an adequate response time when dealing with the highest resolution images. our current setup uses the nginx web server as a front-end and load balancer, providing round-robin balancing across five fastcgi processes. the iip image server also has the ability to utilize a memcached database for caching tiles. our current implementation uses five memcached stores. we have found that this setup provides acceptable performance, even on slower connections. data server the data server is responsible for sending information about the entire document to the browser component. this information is used by the browser to construct the container html elements required to display the entire document and its constituent pages. this data server has access to a number of stored properties about the document, and can then construct a json response to send to the browser containing information about the total document height at a given zoom level (i.e., the total height of all images in the collection), the average and maximum heights and widths of the images (for the inter-page margins and ensuring the document is centred in the browser) and the dimensions of each page at a given zoom level. the data server may be implemented in any server-side language. we are currently using a django web application with a sqlite database for storing image information. however, we are also bundling a basic php script, divaserve.php, for providing filesystem-based image serving. this php script caches the required information for each image as a plain text file, which is then read and served when the user requests a document. this has the advantages of running on any php-equipped web server, and not requiring a complete database setup for serving small collections of image files. jquery plugin the diva.js jquery plugin consists of a small number of javascript, css, and image files used for displaying the images in the browser. the core jquery and the jquery ui libraries are both prerequisites for the plugin, and we bundle the dragscrollable [15] plugin with diva.js. the following code block shows a typical instantiation of the diva.js component. <script type="text/javascript" charset="utf-8"> var dv; $(document).ready(function() { $('#diva-wrapper').diva({ adaptivepadding: 0.5, enablezoomslider: true, enablegotopage: true, iipserverbaseurl: "http://example.com/iipsrv.fcgi?fif=/path/to/imgs/", backendserver: "http://example.com/divaserve.php?d=imgs", }); dv = $("#diva-wrapper").data('diva'); }); </script> ... <body> <div id="diva-wrapper"> </div> </body> ... at instantiation, the iipserverbaseurl and backendserver parameters point to the location of the tile image server and the data server, respectively. other options, like a visible zoom slider and a text box for entering specific page numbers, may be enabled here. when a page is loaded, the diva.js plugin uses the data server to gather information about a document and display it in a continuous, scrolling <div> element on the page. the layout is calculated from the information returned by the data server, and the appropriate tile numbers are then requested from the tile image server. as the user scrolls through the document and zooms in and out from the page images, the javascript component continuously sends off requests for new pages of the document from the tile image server. some pages at higher resolutions do not fit into the users’ browser, and diva will optimize the page loading by only requesting the tiles for the portion of the image that the user is currently viewing. the diva.js component also provides a number of convenience methods for navigating through a document. it supports double-click to zoom in, control-click to zoom out, and click-and-drag to scroll the document. also present is a full-screen view that will expand the document viewport to take up the entire width and height of the users’ browser window. diva.js has been tested to work with most modern browsers. we are also targeting mobile safari on the ipad, iphone and ipod touch, and provide platform-specific navigation like single-finger scrolling, and pinch-to-zoom gestures. we will likely support more mobile browsers in future releases. we believe that this setup provides a sufficiently modular and self-contained infrastructure for implementing a continuous scrolling, multi-resolution document image viewer into new and existing digital document initiatives. applications although the core functionality of the diva project is drawn from a variety of existing document and image viewing technologies, we believe diva represents a unique method of displaying document images to users in a seamless interface. diva presents users with a number of unique features, including continuous scrolling through large documents, high-resolution image viewing, and an optimized method for asynchronously loading page images using an image tile approach. we are currently using the diva infrastructure as part of the optical music recognition for plainchant project [16]. this project is exploring methods of using optical music recognition (i.e., optical character recognition for printed music) for automatically transcribing text and music from books with printed square note notation and making it searchable in an online web application. as a pilot study for this project, we are transcribing the liber usualis, a 2340-page service book used primarily by the catholic church. we are using the diva infrastructure to serve all pages in five resolutions, representing a total book size of 3.2 gb. one further advantage of using a tile image approach is that libraries and archives can make their high-resolution document image collections available for viewing online without providing a readily-accessible means of downloading the image. this may be important for situations where organizations want to publish their collections online, but want to maintain copyright control over their images. although it is not impossible to reconstruct an entire high-resolution document from the component tiles, it is not as simple as downloading the page images. we have set up a demonstration site [17] to illustrate how diva.js may be used. the combined size of the images for the manuscript featured in this demo totals 4.7 gb. figure 4 shows two screen captures from this manuscript illustrating the lowest and highest resolutions available. figure 4. lowest (left) and highest (right) resolutions for a sample page. feedback and contributing the diva project is part of a larger research program in digital music archives and libraries being conducted at mcgill university. individuals or projects that find this work valuable are encouraged to let us know where diva is being used. every effort was made to ensure that each component in the diva infrastructure was available under an open-source license. the vips system and the iip image server are both licensed under the gpl, and the data server component may be implemented in any freely available server-side language. all components of the diva project, including the image processing script, the javascript plugin and the divaserve php script are available under the mit license. we welcome participation from interested individuals. the project code, documentation, and issue tracker are hosted on github [18]. we also encourage developers to fork the project and contribute any modifications or features back to the core repository. conclusion in this article we presented the diva project, an efficient continuous document image viewer designed to present a unified view of a document while simultaneously providing users with the ability to view high-resolution versions of each image. we believe that diva represents a unique approach to document image representation. although it is inspired by features available in many other document image viewing interfaces, diva provides a novel combination of these features. it is our hope that others will find this work valuable, and will be interested in helping us grow and extend diva. acknowledgements the authors would like to thank the swiss national science foundation and the social sciences and humanities research council of canada for their generous support of this project. references [1] répertoire international des sources musicales. http://www.rism.info [2] pugin, l., a. hankinson, and i. fujinaga. 2011. digital preservation and access strategies for musical heritage: the swiss rism experience. oclc systems & services. forthcoming [3] hankinson, a., l. pugin, and i. fuginaga. 2009. interfaces for document representation in digital music libraries. in: proceedings of the conference of the international society for music information retrieval (ismir), kobe, japan. http://ismir2009.ismir.net/proceedings/os1-3.pdf [4] google books. http://books.google.com [5] internet archive. http://archive.org [6] world digital library. http://wdl.org [7] zoomify. http://www.zoomify.com [8] iipmooviewer. http://iipimage.sourceforge.net/documentation/iipmooviewer [9] seadragon ajax. http://gallery.expression.microsoft.com/seadragonajax [10] internet archive bookreader. http://openlibrary.org/dev/docs/bookreader [11] vips image processing system. http://www.vips.ecs.soton.ac.uk [12] imagemagick. http://www.imagemagick.org [13] kakadu software. http://www.kakadusoftware.com [14] iipimage. http://iipimage.sourceforge.net [15] dragscrollable jquery plugin. http://plugins.jquery.com/project/dragscrollable [16] optical music recognition for plainchant. http://ddmal.music.mcgill.ca/wiki/optical_music_recognition_for_plainchant [17] diva.js demonstration. http://ddmal.music.mcgill.ca/diva/demo/ [18] diva.js code repository. http://ddmal.music.mcgill.ca/diva/ about the authors andrew hankinson is a phd candidate in the department of music research at the schulich school of music, mcgill university. he holds a bachelor’s degree in music theory/history from acadia university and a masters in library and information studies from mcgill university. wendy liu is an undergraduate computer science and biology major in the faculty of science at mcgill university. laurent pugin is co-director of the swiss office of the répertoire international des sources musicales (rism) in bern, switzerland. he has bachelor’s degrees in music (viola), in musicology, and in computer science, and a master’s degree and a ph.d in in musicology from the university of geneva, switzerland. ichiro fujinaga is an associate professor and the chair of the music technology area at the schulich school of music at mcgill university. he has bachelor’s degrees in music/percussion and mathematics from university of alberta, and a master’s degree in music theory, and a ph.d. in music technology from mcgill university. subscribe to comments: for this article | for all articles 3 responses to "diva.js: a continuous document viewing interface" please leave a response below: asteko laburpena — zeruak eta urak, 2011-08-29 […] shared diva.js: a continuous document viewing interface – the code4lib journal. […] r hariharan, 2012-04-08 hi, i am quite impressed with the diva technology for viewing images seamlessly. i am looking at this diva without having tightly coupled with iip image server and meta data server. please help me to understand, if this is possible so. thanks r hariharan (hari) andrew hankinson, 2012-09-03 hi hari, currently it is not possible to do this. diva requires the iip image server to serve the image files, and the metadata server for performing measurements of the entire document image collection so that the javascript can lay the document out properly. leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – building an archival collections portal mission editorial committee process and structure code4lib issue 3, 2008-06-23 building an archival collections portal columbia university libraries has developed the archival collections portal, a unified search system helping users discover archival resources in a streamlined way. we combined the power of lucene and solr to search xml, parse json objects, create ead-compliant documents, and deliver results in an easy-to-use interface. by reusing marc records and employing new search engine features and techniques, we are able to bring important and hard-to-find collections to researchers and archivists. the canonical home page of the portal is http://www.columbia.edu/library/archival/. by terry catapano, joanna dipasquale, and stuart marquis introduction columbia university libraries (cul), like many institutions, has a huge backlog of archival collections about which our patrons may not know. these collections are variously presented in finding aids (paper or electronic), collection overviews, or online exhibits, or some combination thereof. although most collections are cataloged in our opac, clio, they did not have a special or easily-linkable presence on the libraries’ public website. searching for a collection was not transparent. in many cases, it was not clear that related or similar-interest items may reside in a different repository on campus. in other cases, the sheer number of results — including books, audio, and archives — overwhelmed the user. additionally, the web presence of these collections varied greatly. while some had associated scanned paper or electronic finding aids, others had collection overviews, pictures, or occasionally more fully developed online exhibits. the result was the libraries had a large amount of archival collections and no uniform way to find them, effectively hiding some of our most significant resources. in spring 2007, the libraries presented a challenge to the authors of this article; design and implement a single web space allowing users to seamlessly access our varied archival collections. we were tasked with finding new technology, developing a website, and establishing a workflow to solve some aspects of this problem. our goals were to make the search and retrieval of archival collections completely transparent, trustworthy, uniform, and feature-rich. we also needed a solution that used the campus architecture of solaris and apache, worked with the libraries’ marc data, and generated standards-compliant information for display, harvesting, etc. the solution needed to be scalable, adding new repositories as needed (we have six libraries contributing their collections), and flexible, offering each repository its own areas of customization. finally, we knew we could make a search engine and web pages, but we also wanted to “do better”. we knew we could present our nicely-structured data in a better format for our users through key new technologies. we believe our work to combine already-existing tools, such as marc, with newer software like solr and lucene resulted in a method and product of interest to the library information technology community. this article provides our rationale for technology choices, discusses our basic framework for providing a search interface, and addresses metadata issues for the creation of search engine records and collection-level pages. rationale and design the first challenge in the archival collections project was to assemble and relate all of the descriptive data the libraries’ archival collections. columbia university libraries includes several units which hold archival and manuscript material. the distribution of collections across repositories is uneven as are the methods and degree of intellectual control over them. ultimately, we aimed to convert the data to encoded archival description (ead) for internal maintenance and external exchange. full scale conversion to ead would be difficult, however. ead had not been previously well-supported at the libraries and cataloging methods as well as stylistic conventions for existing finding aids varied across, and even within, repositories. the greatest variety of practice involved the “container lists,” an information layer used to describe the arrangement of the constituent components (e.g., series, subseries, folders, etc.) of the archival collections. this made it very difficult to convert these items to ead at the level of the “description of subordinate components,” or <dsc>, element. however, there was a great amount of highly consistent description at the collection level. cataloging of archival collections in marc has been underway at columbia for over a decade. to illustrate, the repository with the largest number of archival and manuscript collections, the rare book and manuscript library (rbml), had roughly 90% of its collections represented in marc. for this reason it was decided we would initially focus on description at the collection level. further, as the largest, finest-grained, and most-uniform collection data available was in marc, we quickly established our opac as the best data source for collection information. while it was felt ead provided the most appropriate schema for the portal, marc and the significant resources and infrastructure deployed around it offered greater opportunities for development of a workflow. rather than developing an ead document management system from scratch, we could simply leverage existing systems and skills already deployed in our opac. collection records would be created and updated in marc. these could be extracted as marc xml and converted to ead (at the collection level at least) using xslt. the ead files could serve as the source for further xslt conversions to produce the data and outputs needed for the portal. also, the collection-level ead records could serve as the basis for full ead finding aids when the cul could devote sufficient time and resources to the development and support of a comprehensive ead based program for archival control. our next challenge was to work with existing infrastructure to provide a “portal” into our archival collections, develop uniform pages for each item, and smoothly process the data. as we searched for software to build the archival collections portal, we wanted to fit any possible solutions into columbia university libraries’ current setup. to this end, we knew lucene would be our search engine’s foundation. [1] lucene’s extensive set of apis had proven useful for many other cul projects, and was flexible, speedy, and well-established. however, it required additional java class development and did not contain particular features, such as faceted searching, right out of the box. as we investigated how to incorporate new features into our current lucene setup, we discovered solr, a lightweight framework sitting atop lucene, which provided an administrative interface, an easy configuration, and — excitingly — a faceted search feature. [2] these features would make development of the portal much easier, because it shifted the production focus from back-end to front-end development, allowing us to focus on customization. most importantly, it produced result object sets and metadata that were widely interpreted; it provided outputs in javascript object notation (json), python, xml, and ruby. we quickly decided to install and test solr. at the time, columbia university’s servers did not support python or ruby, so we needed to rely upon xml/xslt or json/javascript to write our front-end search interface. while we found solr’s xml output handy, parsing large result sets was slow. in contrast, json parsing was quick, flexible, and effective. json output became our output of choice. meanwhile, the new developments in mit’s exhibit software piqued our curiosity. [3] exhibit’s available api parses a json object and creates beautiful and feature-rich websites from the data. we decided to experiment with solr’s functions and these ready-made javascript classes. we immediately ran into a stumbling block; exhibit requires a “ready” json object to parse, which meant we needed to create a wrapper function to serve up the search results. without a readily available scripting language like php on our tomcat server, we would need to rely on jsp. we created a basic jsp page to start. it worked fine, but created an additional layer of complexity, which made us unhappy. additionally, exhibit proved less readily customizable than we had hoped. its myriad features were more than we needed, while its css classes proved somewhat difficult to integrate into our own web documents. we realized we wanted to change enough about exhibit to meet our archivists’ and our data needs that we decided to write our own front-end code. with these decisions in place, we began our implementation phase. implementation back-end processing and metadata workflow the portal data compilation begins with an extraction from the opac of marcxml versions of collection records, written to directories for each repository. the script performing the extraction selects all records cataloged at the collection level (i.e., with a “c” character at leader position 07). six files of marc records, one for each repository are written to a directory on our unix server. a perl script controlling the processing of the data for the portal likewise runs daily, after the export of marc data from the opac. the procedure is illustrated in figure 1 and runs as follows: split each “repository” marc file into individual marc records, one for each collection, then write to the corresponding directory for the collection’s holding repository. for each new or updated collection marc file, run an xslt stylesheet and output an ead file. for each new or updated ead file, run an xslt stylesheet to create a static html “collection page” and post to a directory on the cul web server. for each ead file, run an xslt stylesheet which outputs a single file containing all the data for the lucene index. http post the file containing the data to solr. commit the changes and optimize the index. figure 1 – flow chart [view full image] this script can be run in batch mode or via the command line. in batch mode, it searches for new records or updates and initiates the processes in figure 1. run on the command line, it gives us a significant set of functions to fine-tune our application. for example, this method allows us to build a collection on one or more server instances (development, test, or production), only run specific xsl transformations, reindex solr, and re-process individual collection pages. we can trigger processing on one item or a range of items, one or many repositories, etc. this additional set of tools provides us the ability to isolate items for troubleshooting and make modifications without harming other areas of the archival collections portal. metadata issues and remediation during the development of the portal workflow, several issues involving the quality and adequacy of available marc data arose. not surprisingly, the marc cataloging in our opac was not perfect. many records, for example, were incorrectly cataloged as collection records (i.e. ‘c’ leader position 07). most frequently the culprits were records for individual manuscripts. likewise, some thought was given to using the leader position 08 (type of control) to determine if a record was of archival material. combined with leader position 07, this would have made for a simple method to select only archival collection records. in practice, however, only 388 of 7490 archival collection records employed the “a” value (archival) in leader 08. selection then would depend on a combination of location codes and the leader position 07. cataloging practices and opac functionality also proved inhibiting in the handling of hyperlinked content. there appears to be a widespread misunderstanding that all hyperlinking from a marc record must be from an 856 field. if and when this is not a misunderstanding, limited opac functionality enforces the practice. in columbia’s case, only 856 fields were configured in our voyager catalog to generate hyperlinks when displayed. as often is the case, practice followed function and catalogers placed all hyperlinks in the 856 field, regardless of the role of the linked content. in fact, marc permits hyperlinks in several fields. particularly relevant to our concerns are the 555/cumulative index-finding aids note and 540/terms governing use and reproduction note fields. as part of an earlier initiative by the libraries digital program division, most of the hard copy finding aids from the rare book and manuscript library had been scanned and converted to pdf. other projects resulted in the creation of ead, html, and ms word finding aids which were posted on the libraries website. the portal aimed to bring together these digital finding aids with the collection data in the marc records. additionally, in some cases archival material had been digitized and this content too was to be integrated with other collection information in the portal. relying on the 856 field alone to contain hyperlinks semantically overloaded the field. for example, it was difficult to simply create the proper link text for a url (“online finding aid” or “related digital content”) on the html collection page. using the 555 field for links to finding aids, the 540 field for linked html permissions policy statements, and the 856 for links to related content would not only be perhaps more bibliographically proper, but would also greatly simplify processing. another problem had to do with strict adherence to cataloging rules. most of the marc collection records followed the rules set forth in the archives, personal papers, and manuscripts (appm) cataloging manual. section 1.1b4 of appm recommends the use of such terms as “papers,” “records,” and “collection” for the title proper of an archival collection. combined with a main entry in a traditional catalog display typical of an opac, this might produce a rough hewn, somewhat understandable heading such as “smith, john 1822-1907. papers, 1814-1902.” however, isolated from its record context, or repurposed in even the most simple way (to, say, generate a heading for a web page), the appm rule falters. the recently released describing archives: a content standard (dacs) provided an alternative to the appm style titles. [4] the dacs rules allow greater flexibility, permitting the creation of more readable and independently meaningful titles for archival materials. initially the solution for adding or overriding data in the marc records was to use a separately maintained table containing the additional metadata for the collections. the table was maintained as a spreadsheet and was converted to xml for use as a lookup table by the marc to ead xslt stylesheet. rather than wait for all the catalog records to be updated, a runtime lookup could add the additional data. the spreadsheet’s primary purpose was to map each marc record id (i.e., the 001 field) to the filename of the corresponding digital version of the collection’s finding aid. since the scanning of the hard copy finding aids for the rare book and manuscript library took place after and aside from the creation of collection records, some means to link the two was required. other options were to have staff at rbml manually enter the url for each pdf to each marc record, or to have the library systems office develop some automated procedure to load the data in batch to the opac. as these involved personnel working in other departments, with their own priorities and commitments. the less disruptive and resource intensive spreadsheet/lookup table emerged as the solution. as the lookup would be performed anyway, the spreadsheet/table was used as the solution to the ungainly appm title proper issue. in addition to the url for the linked finding aid, the table contained “preferred” titles for each collection provided by rbml staff. (see an example of a collections page. [5]) due to the leveraging of the opac as the data repository and add/update system for the portal, remediation of the repositories’ marc records began soon after the portal’s initial development. the marc to ead stylesheet had to be adjusted to handle both updated and unchanged records. for example, the stylesheet used the following logic for handling links to finding aids, use policies, and related content : when (record contains 555$u): then: ead /otherfindaid/p/extref = 555$u value otherwise: when (marc 856$u starts with “finding aid”): then: ead /otherfindaid/p/extref = 856$u otherwise: look up record of collection in table ead /otherfindaid/p/extref = value of finding aid url in collections table it perhaps would have been preferable to write back the results of the transformation’s disambiguation to the source data. however, the process we designed was not bidirectional. enabling modification to the marc records would have involved coordination with the libraries systems office and likely more commitment of time and resources than were available. the remediation of the source marc records went quickly, and increasingly the need for the lookup stylesheet waned. our workflow now entirely draws from the opac records. (see appendix a: mapping between marc, ead, and lucene index.) solr development and front-end functionality with the data issues resolved, we had content available to use within our indexing tool and front-end search-and-display mechanisms with solr. solr’s indexing features provide a set of tools helping us process and display our data. we can select fields to tokenize, control data replication, provide stemming, and enable sort orders through xml configuration files. this easily modified set of files allowed us to add facets, for example, on fields that we did not initially think we needed to facet. since we could control the search engine’s power independent of the front-end application, we had another area where we could isolate problems, change functionality, and build new features. once we indexed our metadata, we needed to provide a search interface. solr allows us to search over standard http and specify a specific format for our result set. at the time, the most useful result format for us was json. because json is easily evaluated to be object-oriented, it was easy to obtain and parse the resulting objects and provide a quick interface. the code to do this is fairly straightforward; solr runs as a web service, so we simply needed to “talk” to it via http: open an ajax query, receive the response, and parse the data. our basic code took a url from a get request, performed some error-checking and query cleaning, did a solr lookup, and called a parsing function, abbreviated here: *********************************************************************** * function xmlhttppost * * takes a url, opens an xmlhttprequest, receives data, and sends it back. * * parameters: * strurl the concatenated url that asks for a json object * ************************************************************************/ function xmlhttppost(strurl) { // set graphic document.getelementbyid("working").innerhtml = '<img src="http://www.columbia.edu/cu/lweb/images/icons/searching.gif" alt="searching..."/>'; var xmlhttpreq = false; var self = this; if (window.xmlhttprequest) { // mozilla/safari self.xmlhttpreq = new xmlhttprequest(); } // if mozilla else if (window.activexobject) { // ie self.xmlhttpreq = new activexobject("microsoft.xmlhttp"); } // if msie try { var querycheck = checkquerystring(strurl); var querystring = '/findingaids/results.html?' + querycheck; self.xmlhttpreq.open("get", strurl, true); } // end try catch (e) { alert(e); } // end catch self.xmlhttpreq.setrequestheader('content-type', 'application/x-www-form-urlencoded'); self.xmlhttpreq.onreadystatechange = function() { if (self.xmlhttpreq.readystate == 4) { // call function to update html page updatepage(self.xmlhttpreq.responsetext); } // if 4 / ready } // onreadystatechange // clear self.xmlhttpreq.send(null); } // end function xmlhttppost an abbreviated returned json object looks like this [6]: "response":{ "numfound":36, "start":0, "docs":[ { "id":"ldpd_5466921_ead", "filename":"ldpd_5466921_ead.xml", "repository_code":"nnc-m", "audience":"external", "unittitle":"henry w. bellows letters,", "sort_title":"henry w. bellows letters,", "origination":"bellows, henry w. (henry whitney), 1814-1882.", "unitdate":"1861-1863.", "begindate":"1861", "enddate":"1863", "bioghist":"unitarian minister; president, united states sanitary commission during the civil war.\n", "scopecontent":"four autograph letters signed by bellows. three, dated july 12 and 31, 1861 and april 1, 1863 are to samuel b. ruggles. the 1861 letters concern the affairs of the united states sanitary commissions...\n", "subject_t":[ "united states--history--civil war, 1861-1865--medical care.", "united states sanitary commission."], "corpname":["columbia university,health sciences library, archives & special collections."], "corpname_t":[ "columbia university,health sciences library, archives & special collections."], "persname":["bellows, henry w. (henry whitney), 1814-1882.", "bellows, henry w. (henry whitney), 1814-1882.", "opdyke, george, 1805-1880.","rossiter, thomas prichard, 1818-1871.", "ruggles, samuel b. (samuel bulkley), 1800-1881.", "strong, george templeton, 1820-1875.", "opdyke, george, 1805-1880."], "bioghist_t":[ "unitarian minister; president, united states sanitary commission during the civil war.\n"], "unittitle_t":[ "henry w. bellows letters,"] } // etc. with a results set of structured data in hand, we could then parse the data, update an html page display, provide faceting, and allow users to paginate through the results. there are two usable methods to parse these json objects: first, the standard javascript eval() function will parse a json output into an object-oriented format, and second, a function called parsejson(). [7] both are recursive, and both are easily implemented. once parsed, the json object above can be referenced and used on a web page. for example, to reference the biographical history (bioghist) field in the object above, we can simply say: document.write(responses.response.docs[0].bioghist); // prints "unitarian minister; president, united states // sanitary commission during the civil war."</code> </pre> <p>we can also update an area of an html page with this data:</p> <pre><code>document.getelementsbyid("biographyarea").innerhtml = "<p>" + responses.response.docs[0].bioghist + "</p>"; by iterating through our result set and selecting particular fields to display, we can present a basic search results screen. however, the tools to move beyond this presentation are inherent in the solr framework, and we took advantage of many of its features. solr’s api supports pagination, highlighting, sorting, and faceting, making our user interface very rich in functionality. [8] faceting, in particular, provided helpful end-user and internal features. for example, through our subject facet, users can perform a search and learn about the types of subjects in their result set. columbia archivists found this feature very useful as well for internal quality assurance, using the facets to see where subjects were inconsistent, needed de-duplication, or needed augmentation. future plans incorporation of full ead instances the archival collections portal and its associated workflow currently makes light use of ead. the collections data is created and maintained in the opac, and the ead produced is at the collection level. in 2008 cul will begin development of a supported and standardized workflow for the rare book and manuscript library to produce full ead for new and newly revised finding aids. these will for the time being continue to be treated as further collection information linked from the 555 field of the corresponding marc record. we will, however, begin to consider how best to incorporate the full ead into the search application itself. for the intermediate term, at least, the portal will contain data for many collections only at the collection level, while an increasing number of others will have data at lower levels. the incongruity of size among collection descriptions will certainly have consequences on overall retrieval recall and precision. we will have to determine with archives staff what would be the most acceptable search behavior. the relationship between the marc records and full ead instances will also complicate further development. in principle the marc collection record should be based on the ead finding aid which would be the primary, most authoritative record for a collection. as mentioned earlier, the current workflow, however, operates in one direction from the opac and marc. for the time being, the maintenance of authoritative data will likely be cumbersome. if we were to enable the capability, certain ead files could take primacy, with revisions written to derived marc records for inclusion in the opac. meanwhile, descriptions of the many other collections lacking full ead instances will continue to be maintained in marc. replacing javascript with php after we released the archival collections portal, our central it department provided us with space on their lamp servers. around the same time, solr developers created a response writer that parsed php and serialized php. we now had php at our disposal, which we used for our next solr projects. while the general steps to search and display solr data are the same (send parameters to the solr engine, and parse what comes back), the advantages of php over javascript are clear. first, response times are much, much faster: while it takes the same amount of time to fetch the data (assuming both data sets are equal), the power of server-side scripting allows more data to be written to the screen in significantly less time. second, because parsejson() and eval() are recursive, large sets of data cause severe browser lag times, and in some cases, outright browser crashes. in contrast, php’s curl functions and unserialize() (for serialized data sets) quickly retrieve and make sense of result sets. solr developers recently released a php for solr module that works well for php 5.2+. moreover, the nature of javascript with ajax, in an educational setting, also causes problems. the browser’s “back” button does not work as expected in an ajax application, because each followed link is really a function call on the same page rather than a page reload. this is an extraordinarily useful technique in many cases, but did not necessarily suit our overall purposes. as a result, we spent some development time ensuring unique, “bookmarkable” urls existed in the archival collections portal, not only as a feature but as a “best practices” for our institution. accessibility compliance is also problematic, as on-the-fly redrawing of a screen and dynamically updated html pages do not render a page that is readable by jaws or other screen-reading software. screen readers will have no actual content to read when the page loads, for there is nothing on the page to be “read” until the response is written out client-side. this is a large barrier to implementation. now that php is available on columbia university servers, we will most likely rewrite this javascript application in our lamp environment to provide better accessibility compliance and faster, easier searching. test exposure of data one of our goals for the archival collections portal was the creation of a site map, generated by perl or xslt, that was indexable by the major search engines (as well as columbia’s own search engine). once we created the portal, we created a google sitemap for this application from one of the flat-file by-products of our perl script. we also wanted to be sure these search engines received the most recent data. we chose to point google to a file that receives continually updated information through our workflow. because we had a readily-available data set to list, it was easy to make it conform to google’s specifications and point its search engine to us. we now have a file that can be easily modified for harvest in other search engines too. this will allow us to expose our site to the greatest number of constituents and will become much more important as we fold in more finding aids and make that data searchable as well. as we create subject lists, we will be able to apply the same intelligence to series and subseries of the finding aids to create a robust, indexable list of data. we have also begun experimenting with incorporating rdfa into the html collection pages. [9] the rdfa metadata would add some semantics to the otherwise generic html encoding. for example, the collection page for the hart crane papers collection could contain code such as: <div xmlns:dc="http://purl.org/dc/elements/1.1/" style="padding:5px;" about="http://www.columbia.edu/cu/libraries/inside/dev/archival/collections/ldpd_4079683"> <h2 property="dc:title">hart crane papers ca.1909-1937.</h2> ... <span property="dc:creator">crane, hart, 1899-1932.</span>... this enables interoperability with rdfa aware applications. interestingly, any attempts to have the rdfa enhanced collection pages to serve as grddl (gleaning resource descriptions from dialects of languages) source documents proved difficult. for example, the w3c grddl service failed to extract the rdf from our pages. [10] the service will successfully process only well-formed xml. while the html code produced by the ead to html transformation is well formed, the resulting pages reference server-side includes containing not well-formed xml. this highlights the benefits exposing well-formed xml, but also the naivete and fragility of the existing w3c service in having low tolerance for sloppy html code. this and any such application intended to operate on existing, “in the wild” html would do well to employ a tool such as the tag soup parser as a preprocessor to output wellfrom ill-formed html. [11] the use of rdfa represents a “pull-based” approach to interoperability and exposure. another is the addition of records to the cul oai repository, for which there are plans. we could also (although we have no firm plans to do this now) take “push” approaches as well. it would not be difficult to post descriptions and links on services like del.icio.us so that the collections could be discoverable in other environments and contexts. summary as we developed the optimal setup, programming, and workflow techniques for our libraries’ digital data, we realized to what extent the combination of apache tomcat, solr, marc, and xml led to a rapid development of stable, ready-to-use services. from the first indexing of our data, to the final interface prototype, our development took approximately six weeks. (this, of course, does not include front-end design tweaks, committee work, and final site launch.) solr, in particular, generated the most time savings: sitting atop a search engine we knew well (lucene), solr allowed us to shift the mechanisms of search and retrieval to front-end user-interface development rather than back-end programming. concepts such as highlights, facets, and even results lists did not have to be customized and interpreted through java classes; basic changes did not require recompilation or reindexing. thus we were able to separate layers of data, design, and development. we also found that our workflow “seeded” the development for other initiatives: retroconversion to full ead finding aids, repository-specific xslt libraries from marcxml to mods, and oai harvesting are reusable in many other digital library projects. finally, for those interested in development and use of open-source programs in conjunction with robust library standards, we found that the combination of solr, json, and marc, residing in an apache tomcat environment, proved robust, scalable, inexpensive, and quick. while there are many libraries beginning — or fully implementing — initiatives to use marc, xml, and solr with their opacs and data sets, we believe that these tools are extraordinarily helpful in the digital library realm as well. together, we have found that they create a useful platform upon which to build many individual projects. appendix a: mapping between marc, ead, and lucene index this table outlines how marc fields were mapped to ead elements and ultimately to fields in the lucene index. some of the fields are not indexed. marc field ead lucene fields 008/07-10 unitdate@normal chars 1-4: begindate; chars 5-8: enddate 100 origination/persname origination 110 origination/corpname origination 130 origination/title origination 245$a$k unittitle unittitle 245$f unitdate unidate 300 physdesc/extent 852 repository 041 langmaterial/language 001 unitid@type=”clio” id 090 unitid@type=”call_num” * unitid@repositorycode repository_code 520 scopecontent scopecontent 545 bioghist bioghist 351 arrangement 600 controlacess/persname persname 610 controlacess/corpname corpname 630 controlacess/title tit 650 controlacess/subject subject 651 controlacess/geogname geogname 655 controlacess/genreform genreform 656 controlacess/occupation 657 controlacess/function 700 controlacess/persname persname 710 controlacess/corpname corpname 730 controlacess/title 506 accessrestrict 524 prefercite 530 altformavail 540 userestrict 541 acqinfo 544 relatedmaterial 555 otherfindaid 561 custodhist 856 dao * eadid filename links [1] lucene – http://lucene.apache.org/ [2] solr – http://lucene.apache.org/solr/ [3] exhibit – http://simile.mit.edu/exhibit/ [4] dacs (describing archives: a content standard) – http://www.archivists.org/catalog/pubdetail.asp?objectid=1279 [5] sample collections page – http://www.columbia.edu/cu/lweb/archival/collections/ldpd_4079136/ [6] full json result object – http://app.cul.columbia.edu:8080/findingaids/ select?q=railroads and repository_code:nnc-*;origination asc,sort_title asc&wt=json&indent=true&facet=true&facet.field=subject&facet.field= repository_code&facet.field=persname&fl.audience=external&facet.mincount= 1&hl=true&hl.fl=all_text&hl.fragsize=50&hl.snippets=5&hl.formatter=simple& hl.simple.pre=%3cspan class=highlight%3e&hl.simple.post=%3c/span%3e&rows=20 [7] parsejson() function – http://www.json.org/js.html [8] example search – http://app.cul.columbia.edu:8080/findingaids/results.html? q=railroads and repository_code:nnc-*;origination asc,sort_title asc&wt=json&indent=true&facet=true&facet.field=subject&facet.field= repository_code&facet.field=persname&fl.audience=external&facet.mincount= 1&hl=true&hl.fl=all_text&hl.fragsize=50&hl.snippets=5&hl.formatter=simple& hl.simple.pre=%3cspan class=highlight%3e&hl.simple.post=%3c/span%3e&rows=20 [9] rdfa – http://www.w3.org/tr/xhtml-rdfa-primer/ [10] grddl – http://www.w3.org/2007/08/grddl/ [11] tag soup parser – http://home.ccil.org/~cowan/xml/tagsoup/ about the authors terry catapano (thc4@columbia.edu) is a librarian in columbia university libraries' digital program division. joanna dipasquale (jd2481@columbia.edu) is a web developer in columbia university's libraries digital program division. stuart marquis (marquis@columbia.edu) is a programmer in columbia university's libraries digital program division. tags: archives, ead, json, marc, solr subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – how to party like it’s 1999: emulation for everyone mission editorial committee process and structure code4lib issue 32, 2016-04-25 how to party like it’s 1999: emulation for everyone emulated access of complex media has long been discussed, but there are very few instances in which complex, interactive, born-digital emulations are available to researchers. new york public library has made 1980-90’s era video games from 5.25″ floppy disks in the timothy leary papers accessible via a dosbox emulator. these games appear in various stages of development and display the work of at least four of leary’s collaborators on the games. 56 disk images from the leary papers are currently emulated in the reading room. new york university has made late 1990s-mid 2000’s era photoshop files from the jeremy blake papers accessible to researchers. the blake papers include over 300 pieces of media. cornell university library was awarded a grant from the neh to analyze approximately 100 born-digital artworks created for cd-rom from the rose goldsen archive of new media art to develop preservation workflows, access strategies, and metadata frameworks. rhizome has undertaken a number of emulation projects as a major part of its preservation strategy for born-digital artworks. in cooperation with the university of freiburg in germany, rhizome recently restored several digital artworks for public access using a cloud-based emulation framework. this framework (bwfla) has been designed to facilitate the reenactments of software on a large scale, for internal use or public access. this paper will guide readers through how to implement emulation. each of the institutions weigh in on oddities and idiosyncrasies they encountered throughout the process — from accession to access. by dianne dietrich, julia kim, morgan mckeehan, and alison rhonemus background imagine this: you’re at your desk, triumphant, because you’ve finally imaged a collection of obsolete media. maybe you had a box filled with 3.5″ floppy disks that have been lingering around the office for at least a decade now, or maybe somebody pointed you towards a spindle of cd-roms that needed imaging. whatever the case, the hard part’s over now, right? you found the right hardware and software to transfer bit-perfect copies of the original storage medium and now you’ve got everything on a modern hard drive, with a file naming convention that would make even a cataloger weep with joy. but the hard part isn’t over, is it? these are really old files, and some of them are completely unintelligible on your modern operating system. you double-click, and you’re confronted with discouraging error messages about incompatibility or, even worse, your modern operating system literally has no idea what to do with a file extension you’ve handed it. what now? maybe you’ve heard something about emulation, about how to run an older computer in a newer computer. how does that work? can you make that kind of magic happen? yes, it’s possible: you can, in fact, party like it’s 1999. emulation has long been discussed, but how does it happen? [1] it may feel like many in the library and archives communities now support emulation, but that wasn’t always the case. in previous years, emulation and migration were pitted against one another as strategies (then largely hypothetical) for preserving and creating access to complex born-digital material. enter “migration vs. emulation” and “digital preservation” in a search box today and you will still find traces of the false dichotomy from the early to mid 2000s.[2] until emory university’s successes with the salmon rushdie archives in the 2010s, there were no well known and publicized use cases of emulation.[3] while it is now widely accepted as an access method, there are still few examples of its implementation. perhaps as discouraging, in most known cases, skilled teams of archivists and technologists had to invest enormous amounts of time, effort, and resources to work with emulation. the archival field has taken great strides to begin dealing with born-digital collections and, with the increase in born-digital content, there is a recognition for studying emulation for datasets, computer generated artworks, and video games; or “complex media.”[4] the cases detailed below offer examples from four institutions with different project scopes and institutional levels of support. in 2012, when cornell university library was writing a grant proposal, “preservation and access framework for digital art objects,” (pafdao) they noted that emulation wasn’t a viable strategy for preservation or access and planned to not investigate it as such.[5][6] by 2013, however, this sentiment had already changed. one of our authors had just started a two-year fellowship to be digital forensic analyst on the project, and was urged by the pafdao advisory board to consider emulation. this was during rhizome‘s successes emulating mid-90s macintosh systems. “compile the emulator from source,” they advised her. she nodded and scribbled this down in the lab notebook. the journey had only begun. the timothy leary papers, acquired by the new york public library in 2011, included over 350 removable media objects. processing of removable media did not take place until 2012, when the project was coming close to completion. upon imaging, it was discovered that over 50 floppy disks contained in-progress versions of computer games leary had been involved in developing. forensic tools were able to identify the games as dos executables, but viewing the games was a different story. the forensic toolkit (ftk) file viewer displays text, images, and some video, but does not run executables. for some amiga and apple disk images, the forensic tools did not provide any more information than the dreaded “unrecognized filesystem.” ultimately, appraising the computer games and prospective game artwork required the use of three emulators. processing the born-digital content in the leary papers included three staff members: the project archivist, the digital archivist and the digital archives assistant. using the three emulators was time consuming in terms of research and implementation, even before they were ready for appraisal. setup and appraisal took approximately four months for one part-time staff member. similarly, as late as fall 2015, new york university’s national digital stewardship fellow was tasked with exploring access methods for the 1990’s era jeremy blake “papers.” the collection’s value was with in-progress artwork created in obsolete computing environments.[7] the fellowship required researching and writing a report exploring emulation as a possible strategy, but the grant funded project did not specify implementation. like others, the fellow had read and reached out to individuals involved with the few known successful case studies at the time. bolstered by these recent successes, and with encouragement from nyu libraries, she decided to explore emulation as a means to authentic access to the complex artworks. she was also encouraged by the latitude afforded from scholarly work embracing a spectrum of authenticity within the inherently variable genre of computer art.[8] setting up and testing the emulation involved 2 staff members, the nds fellow and the digital archivist, working over the course of 1.5 months, using approximately 50% of the fellow’s time and 10% of the digital archivist’s. also in 2015, building on their earlier successes with emulation, rhizome launched a project that brought wide visibility to a new approach to software emulation accessed online. in collaboration with the german bwfla project (baden-wurttemberg functional long-term archiving and access), rhizome presented a popular online exhibition of three videogames/artworks created by theresa duncan as cd-roms in the 1990s.[9] the theresa duncan exhibition built on bwfla’s work with providing access to artworks held in the transmediale cd-rom art archive, and extended bwfla’s remotely-hosted emulation as a service framework to a cloud-based implementation.[10] the ability to share the url for a remote emulation session, and therefore to embed access to a specific emulation environment as an iframe on virtually any website, allowed widespread access to the games, and embedded instantiations of the emulation site proliferated across social media.[11] running the eaas framework via cloud hosting allows for flexibility of server capacity in response to user demand for the games, by increasing or decreasing the number of virtual machines active at any given time as necessary. during the initial launch of the theresa duncan games on rhizome’s emulators site, rhizome reserved 750 virtual cpus to accommodate the spike in user demand prompted by social media announcements of access to the games.[12] following the initial peak demand period, rhizome was able to reduce the virtual cpu allotment number to 16, and accommodate additional requests by queuing other users. for ongoing access to the games, 4 virtual cpus are assigned to the project.[13] how to emulate now that you have a sense of the authors’ initial forays into emulation, you may be thinking about how you might use it to access some of the materials in your own collection. while the road ahead can be winding and complex, it is ultimately quite rewarding to revive complex digital materials that have long been dormant. drawn from the authors’ own experiences, what follows is a step by step guide to getting started with emulation. step 1. figuring out what runs your stuff it may feel overwhelming to try and determine how to breathe life into old files, but here are some starting points that might help you. before you dive into emulators and virtual machines, it is good to get a handle on what type of environment would be compatible with the digital material in front of you. here, it can be helpful to think of files not just as discrete entities, but as objects whose existence is contingent on a complex network of dependencies, both hardware and software. the more you can determine about that environment, the better you can figure out what emulation strategy will work best for your materials. this can be a complicated process, so we’ve organized the overall strategy into manageable steps. use existing system requirements system requirements, if you can find them, can be tremendously helpful. at cornell, the testbed collection for the pafdao project came from cd-roms in the rose goldsen archive of new media art. fortunately for the project team, (nearly) all of the items in the testbed had been individually cataloged, and this included transcriptions of all of the system requirements printed on the jewel boxes and cd cases. in a lot of cases, this information was thorough, and gave the team a clear view of both the hardware and software requirements needed to properly render the files.[14] what happens, however, when the system requirements are not so clear or you don’t have any such documentation? there are a few ways to proceed from here. knowing the approximate age of the digital files can also help you narrow down a compatible software environment. consider what operating systems were in current use [15] at the time of the files’ creation.[16][17] the pafdao team at cornell found it helpful to create a reference table that provided a starting point for possible compatible environments given an approximate age of a cd-rom.[18] examine the disk image for file system clues forensics tools can also provide clues in helping you piece together the puzzle, especially if there is uncertainty about compatible hardware. knowing what file system is present on a disk image will often point you to the hardware environment that you will need to emulate. the linux utility disktype in the bitcurator environment will identify what file system(s) exist on a disk image file.[19] for example, hfs and hfs+ are apple’s proprietary file systems, while ntfs is microsoft’s proprietary file system. if you have cd-roms, like the cornell team, you might find that you can have more than one file system represented on a single disk image, indicating multiple compatible hardware environments. if you are comfortable on the command line, disktype is tremendously helpful, but there are also graphical forensics tools that you can use. ftk imager and isobuster will both display the file system(s) present on a disk image.[20][21] the following examples reveal how file system information from disk images was used to uncover context for digital material. through investigating the corpus of hard drive and optical disk images in the jeremy blake papers (at nyu), it was easy to come up with an accurate timespan of his output (1998 and later) by using file system metadata that indicated he worked in hfs+ file systems.[22] for the leary floppy disks (at nypl) it was the process of selecting the correct sector format to interpret the disk images using the kryoflux that took the most trial and error.[23][24] this often involved trying different formats until the kryoflux gui displayed green sectors to indicate a good result. the kryoflux hardware consists of a control board that connects to a floppy disk drive and from the control board to a modern computer through usb. the kryoflux software can output a reading of the magnetic flux transitions on a disk and interpret those flux transitions as a sector format that could be read by the correct operating system.[25] once this initial format information had been obtained, choosing an emulator was more straightforward. disk images with an amiga filesystem require an amiga emulator like winuae. to use apple formatted disk images, one combines sheepshaver and an os the approximate age of the disk. forensic tools were helpful in determining the date of the dos executables and some mac files. figure 1. kryoflux graphical user interface displaying a sector match for apple dos perform file level analysis what happens, though, when you don’t have a disk image to work from? file type analysis works when you have individual files. one of the most common tools for determining a file type is the command-line utility file, which can be directed at much more than just traditional files.[26] figure 2. using the file command on a directory of files one way to start is by looking at the program executables first to see if that points you in the right direction. here are some examples to guide your thinking here. let’s say file outputs the following for an executable file: “ms-dos executable, ne for ms windows 3.x.” this tells you that you’re likely looking at a file that will need a pc environment running an old version of windows. if you don’t have an executable (or software) file to analyze, don’t fear: running file on even “ordinary” files may still give you valuable information. consider the following output for a word processing document: “rich text format data, version 1, apple macintosh.” this output indicates that a macintosh environment may be needed here, which may be more than you knew before.[27] in the case of the jeremy blake papers, it was difficult to narrow down the range of adobe photoshop software versions he had used; they were not identified with standard file identification tools. instead, this required a deeper look at the hex behind each of the files. hexadecimal is a “human-readable” translation and condensing of the underlying bitstream of any file and, when standard tools don’t help, can provide useful clues.[28] any number of freely available tools allow you to switch from examining the rendered file, or the jpeg and psd file as you were meant to experience it, to the underlying hex (ftk imager, for example). in the case of identifying adobe photoshop software versions used to create different adobe photoshop files, looking at the hex and ascii could be used to corroborate or complicate other file identification tools. the key was to locate the “magic numbers,” or the bit of hex that was constant across different files of the same file type.[29] magic numbers are used in many file-level tools out there. while contemporary adobe photoshop psd files were all identified with 8bps, files created in other versions needed to be identified with other “magic numbers.” through comparing hex with adobe photoshop documentation online, it was possible to identify what software versions of adobe photoshop were needed for access. this was balanced with the limitations in the operating systems emulated by sheepshaver. adobe photoshop cs, for example, requires an mac operating system 10, a system that sheepshaver did not itself support. figure 3. hex and ascii of files from the jeremy blake papers if you’re using a graphical hex editor program, obtaining the data you see here can be accomplished by loading the files in question into the editor. hex editor tools will show the hex values in one column and the more human-readable strings in an adjacent column. many support copying and pasting output as you see here. in nyu’s case, the use of file and standard file identification tools was limited, with example output: “adobe photoshop image, 2925 x 1650, rgb, 3x 8-bit channels.” standard file identification tools like the national archive’s droid (the national archives n.d.) which draws on the pronom file format registry (available as gui, too) also outputted “adobe photoshop” without version specificity.[30] the magic number database drives many of the file identification tools available. limitations like these encouraged the use of multiple tools and resources to verify information against. figure 4. software version identification with droid in the case of matching content to system environments in eaas, the modular architecture allows a streamlined approach to decision-making, while also enabling further customization for specific use cases. the project’s goal in providing pre-configured emulation environments suitable for accessing legacy cd-roms is to take advantage of the original context for this medium; at the height of their popularity, cd-roms were for the most part created as mass-market objects, made for wide access across common computers of the time period.[31] this aspect makes cd-roms a good match for bwfla’s focus on developing eaas as an adaptable architecture, accommodating broad commonalities across different kinds of works; by offering a range of emulation environments based on common computers during the time period in which the cd-roms were released, the framework can provide an access situation for the contemporary user which is comparable to the original situation at the time of a given cd-rom’s creation.[32] for the theresa duncan project, rhizome’s conservator identified four different operating systems which could run the games, and noted macintosh system 7.5 as the best option for quick startup time and system stability.[33] step 2. get the emulator running, but which emulator? and how? from these investigations, you will have a sense of the timespan of your files and, after a little digging with a variety of tools, you may also have good idea of the filesystem. now it’s time to start thinking about emulators. there are many kinds of computer system emulators out there.[34] here is a list of the emulators each of the authors worked with over the course of their projects. in addition, the pafdao project made notes of additional emulators for future reference.[35] figure 5. emulators used (see also this spreadsheet of emulators) after you have figured out which emulator you might need, the next step is to figure out how to get that emulator up and running. one should not downplay the significance of this part, as it can range from fairly easy to maddeningly frustrating. the process for each of these emulators is slightly different. dosbox and qemu may be the most straightforward to install, as they both can be downloaded from their respective project’s websites or, if you’re using linux, easily installed through your distribution’s package manager. if you’re using dosbox, once you’ve downloaded the installer, you’re almost done: the dos environment is already set up and ready for you to work with. in this environment, standalone executables should be able to run. dosbox provides a very simple configuration file to choose a default directory path and mount a directory as the c: drive. additional dos utilities can be added for a more versatile system. qemu requires a little more cajoling to function as a proper virtual machine running an old version of windows, but first, let’s talk about how to get the macintosh emulators started. while there are pre-compiled binaries for basilisk ii and sheepshaver that have been created by the community, the pafdao team and nyu found that compiling from source made them less prone to spontaneous and frequent crashing. this wasn’t straightforward, though, because, at the time (2013-2014) the codebase was fairly old. compiling the source on bitcurator (which was built on ubuntu 12.04lts, and, starting in may 2014, 14.04lts) resulted in a non-functioning program. through some determined sleuthing, the pafdao team at cornell discovered that using an older version of the compiler (i.e., gcc 4.4) was the only way to stably run the emulator on their systems.[36] the authors stress that this shouldn’t deter you — there is still an active community for both basilisk ii and sheepshaver and there are likely more functional pre-compiled binaries now than when we started. at nyu, the fellow initially tried pre-built emulators and also experimented with using a variety of host computers, from contemporary windows and mac environments, before ultimately turning to using older laptop computers installed with ubuntu 10.0 operating system. in the initial attempts, accessing photoshop psd files in adobe photoshop 3 versus adobe photoshop 6 software versions resulted in very differently rendered images. images were often glitchy and, from comparing across platforms and software systems, clearly not anywhere approaching what the artist intended. the bwfla-demo site was invaluable as an additional comparative emulator to test identical files against. where was the fault? why did one version work, but not the other in the same environment? these errors initiated several weeks of investigation; they also made it clear how many different dependencies were in play. even small changes to emulation settings factored into the stability of the system. while the bitstream of the original file might have been exactly preserved, the many supporting systems mimicked are inexact. ultimately, the nds fellow was tipped to forgo using the optical media images she created in ftk imager, and instead use the linux utility dd to create an iso file. loading these new images via usb onto the emulator solved one of the problems. the host computer was an additional unexpected major variable. the digital archivist and fellow tested a variety of operating systems on different host laptops. each test configuration took approximately 6 hours set-up to test if the emulator loaded properly and rendered the adobe photoshop file in multiple software versions correctly. at some points, it seemed that a viable solution did not exist, but ultimately they were able to configure an older non-contemporary laptop installed with ubuntu 10.0 to accept the photoshop files. figure 6. glitchy rendering of emulated adobe photoshop files (jeremy blake papers) once you have qemu, basilisk ii, and/or sheepshaver started, you’ve made significant progress in your journey, but you’re not totally there yet. you’ll need two (or three) more ingredients to keep going. first, you will need an original installation disk from (or a disk image of) a compatible operating system. for example, if you wanted to use qemu to emulate a windows 2000 environment, you will need qemu and a copy of the windows 2000 installer. similarly, if you wanted to use sheepshaver to emulate a mac running os 9, you will need a retail copy of os 9.[37] second, you will need to create a file that essentially represents the hard drive of the emulated system. the process by which you do this in qemu, basilisk ii, and sheepshaver varies slightly. in qemu, there is a command line utility to create the disk image file or you can use any of the graphical front ends to do this.[38] [39] for basilisk ii and sheepshaver, you use the sheepshaver graphical interface to do this. when considering what size to make this disk image file, remember that hard drive sizes have increased dramatically since the 1990s. that is to say, it’s unlikely that for most of the older operating systems, you’ll need anything more than a 2gb file. if your target is a macintosh or amiga system, there are a few more steps you will need to take. in addition to the emulator and the operating system installer, you also need a read only memory (rom) file, which is the firmware for the hardware required for these systems. in the apple and amiga communities, early users of emulation had to extract the firmware using actual hardware, but you can benefit from their hard work and download the files you need fairly easily online.[40] [41] in the basilisk ii, sheepshaver, and winuae interfaces, on the “memory/misc” tab there is a place to enter in the location of the rom file so you can get things moving. the main difference in the preference interface of the amiga emulator winuae is that the file name of the rom is kickstart.rom. in addition to a rom file, it is useful to have unmodified workbench disk images. workbench is the desktop interface of the amiga. at nypl the leary papers did come with workbench disks that were imaged, but since these disks had been written to, they were loaded as secondary disk images rather than as the boot disk image. figure 7. workbench: amiga desktop winuae, timothy leary papers, neuromancer, development disks, nypl figure 8. grace jones as molly, timothy leary papers, neuromancer, development disks, nypl although dosbox was simple to install, and able to run the leary game executables (at nypl) fairly consistently, game play is not entirely smooth. almost all of the disk images contain in-progress works. the games may end abruptly or crash, or there could be display issues that make some text illegible. the disk images also contain correspondence between leary and the other collaborators working on the games alongside the executables. this text can be viewed in dosbox but additional dos utilities would make viewing the documents easier and could help with other display issues. additional utilities were tested but were not installed in the reading room. the correspondence included with the executables is also available in more standard electronics records components of text documents, but it does take some digging to find which document belongs to which disk. figure 9. neuromancer in dosbox, timothy leary papers, neuromancer, development disks, nypl you may be thinking about alternatives to all of this custom configuration and setup, and we think you have good reason. all of the above work is complex, and those of us who slogged our way through it did often dream of a more automated way to pull everything together. while we think there’s nothing quite like knowing it’s possible to have a handle on all of the knobs and levers to tweak every aspect of your emulated environment (perhaps to support access for a highly specialized collection), there are other options out there for you. in october 2015, bwfla developed a range of options that enable interested users to download, install, and run the emulation as a service platform via docker containers, thereby making it quite straightforward to run a variety of preconfigured emulation environments locally.[42] the contained version includes workflows for ingest, customization, and testing that mirror the functionality of the framework provided at the bwfla demo site, which allows the user to select the emulator and operating system they would like to use from drop-down menus in a graphic user interface. availability of the entire framework packaged within these containers greatly simplifies installation of the framework as a local implementation, allowing a wider range of users to try out eaas. the contained implementation also allows for running the framework in a variety of ways, including as a self-contained system, or from a live usb system, which allows one to boot a computer directly into an emulation environment.[43] it’s important to note that this range of options also allows users to experiment with creating their own customizations of the framework, saved on their local machine. two posts on the bwfla project blog detail the structure of the docker setup, as well as system prerequisites, and instructions for installation and operation.[44] these posts also provide links to the usb image for download, and to bwfla’s docker containers and scripts. step 3. at long last, access user interfaces have changed dramatically in the last twenty years. once you’ve got your emulated system up and running — and we feel like a broken record for saying this, but it’s true — there may still be more hurdles for you to clear. do you remember how to use dos? one of our authors appreciates the os 9 training she received as an undergrad employed by her university’s helpdesk, which is the only reason she was able to quickly locate the system settings on that operating system. the question facing you once you’ve set everything up is, how do users and researchers react to emulated environments? [45] what can you do to make the translation of old technology on newer machines as seamless as possible for your users? another concern in many institutions is how will archival staff use and support researchers coming to access emulated collections? at nyu, the fellow met with reference archivists to plan for this step through walking through the basics of what to do to, for example, restart a crashed emulator. in meetings with archivists across the institution, the oft-cited example of, “if the collection is in another language, it’s the researchers’ responsibility to know that language (not the archivist),” did a lot to define how far staff could and should support patrons’ access to emulated collections.[46] it’s easy to forget that, for older digital material, readme files and documentation may not be optional. to give one example, the pafdao team was working with a researcher who noticed that a certain artwork was too responsive to mouse input and was moving far too fast. at first, it was easy to dismiss this as a typical artifact of the emulation process — many have documented this exact behavior when using emulation — but after closer inspection, it actually wasn’t the case. the documentation from the artist indicated that the user should navigate the artwork using the keyboard and not the mouse. once the user made that change, the artwork responded more in line with the user’s expectations than before. heed our advice here: it may not be the emulator’s fault. it may be something much more subtle. as part of assessing born-digital access, the nyu fellow invited experienced archival researchers to explore emulated collections. in multiple tests, the searchers loaded the same image on older computers as well as on the emulator. it reassured and reminded both archivist and researcher of their own modern expectations. in the 1990s computers were so comparatively slow it was hard to comprehend that something was not broken. the pafdao team at cornell contended with the fact that they were working with artworks that, in the words of one of the curators, “invoked the need to authentically preserve their digital context” [47] ; that is, the hardware and software environments required to run them. this sometimes meant the muted, reddish tones of a crt monitor, and not the bright blue hues on a new widescreen lcd monitor. the team found some ways to mediate changes like this, such as making use of utilities that reduce the brightness of lcd screens to alleviate eye strain or manually setting the color balance on monitors. none of these strategies were perfect, however, and every adjustment was often at the expense of some other aspect of the experience. at nypl, guides to using the emulators are available in the reading room in versions for researchers and library staff. researchers do need guides in order to be able to view this material. user interfaces have changed and those used to 21st century computing may find older systems counter-intuitive and frustrating. while there are guides to using emulators, there are often no guides to playing the games. the development status of the material means the help sections of the programs are mostly useless. when game instructions do exist they may be located in an electronic record component or even in paper records. emulation for access would work best for researchers who are familiar with older computer systems, appreciate the context provided by an age appropriate computing environment and are ready for a challenge. this is a rare combination. the audience for the theory of leary’s games is present but often researchers writing on leary’s theory and intent for the games have little interest playing the games. a popular frame of reference for the games is that leary saw them as the new psychedelics yet researchers comfortable writing about mind altering experiences were still not up to play the games. the researchers that came to study jeremy blake’s papers all appreciated the emulation, especially as they came to better understand the processes in place to create it. surprisingly, they preferred to use contemporary computers to the emulator. ultimately, stability, speed, and familiarity were much more important to researchers than authenticity and fidelity. if they were going to work more on the technical specifications or were writing a book-length work on his work processes, the researchers would explore the emulator more fully, but for ease of access contemporary researchers want contemporary machines.[48] stability and speed were also primary criteria in evaluating the emulated performances of theresa duncan’s artworks in rhizome’s exhibition. in the case of providing access to interactive works via emulators running remotely, reducing network latency is important for minimizing the potential for sluggish response to user input, which would obviously have a negative effect on the quality of a user’s experience. as described above, the interactive nature of cd-rom artworks requires seamless delivery of multimedia content such as a responsive mouse pointer or constantly changing narrative and pictorial elements. for this reason, it’s necessary to evaluate and if possible reduce lag time between the remote emulation and the user’s computer to provide adequately quick response for the context of a particular game. in the case of the theresa duncan games, the user’s perusal of a detailed fictional landscape, and the voice-over narration of printed text on the screen encourages a relaxed pace. for this use case, streaming was able to deliver sufficient response time, but the streaming model could present problems for faster-paced skill-based interactive games requiring instantaneous reaction time.[49] streaming also presents a significant challenge for providing continuity of audio content. since the user’s interaction with the emulated software unfolds in real time, buffering content is not possible, but any gaps or stuttering in music or sounds meant to be continuous will be very distracting and disruptive of the overall rendering quality. evaluating sound quality of performances of the same artwork via different access points, for example on the bwfla project’s demo server located in germany, at rhizome’s cloud-hosted emulators site, which attempts to optimize network latency by allocating server resources close to the user’s location, and on a local installation demonstrates the importance of this criteria for remote emulation’s viability as an access strategy. to address these issues, in selecting an environment for presenting duncan’s games, performance was evaluated for overall clarity of representation in graphic and sound elements, and the framework was configured accordingly; project documentation notes that for rhizome’s emulators site, eaas was configured to use lossless (compressed) graphics instead of video codecs to eliminate visual artifacts, and ogg/vorbis audio compression to prioritize continuity of sound.[50] conclusions the examples discussed in this paper, and the increasing body of supporting research cited here, demonstrate the viability of emulation as a method for accessing complex digital objects that would otherwise be lost to incompatibilities and technological obsolescence. for example, the model employed by eaas provides significant reduction of many of the complexities experienced by projects described in this paper, by providing emulation environments constructed as modular components. ongoing development of eaas demonstrates further improvements in interoperability, such as more flexible support for multiple disk image formats across emulators.[51] additionally, a number of unexpected observations within these projects indicate that as viability paves the way for an accumulation of usage instances, applying emulation as a strategy across a broader range of legacy materials may provide additional affordances, such as enabling detailed and accurate analysis of complex digital objects. for example, faced with an artwork containing files that seemed out of place on the particular file system (e.g., readme files for running a work in windows on a macintosh-formatted disk), pafdao team members discovered that emulation helped clarify what was happening and allowed for better user documentation. in the course of examining another set of interrelated digital material, the pafdao team also combined file-level analysis with emulation to clarify how close to the original an emulated version actually was, or had the potential to be.[52] in the nyu case, emulation exposed major errors in the imaging process of the jeremy blake files, as well as incompatibilities with the closed and proprietary software versions used. given the value of the insights provided in this instance, it is particularly important to emphasize that this project found it necessary to rely on older host machines for running the emulators. this challenge, and the essential role played by emulation for accomplishing the broader project objectives, underlines the need for ongoing awareness and collaborative work, to update existing emulation software builds and to ensure sustained interoperability with contemporary host computers in the future.[53] while archives are just beginning to address backlogs of born-digital content for preservation and access, as more archival collections continue to collect contemporary (born-digital) collections, understanding the problems learned from emulating yesterday’s obsolete collections now will lay the groundwork for the future. about the authors dianne dietrich is the digital projects librarian at cornell university library. from 2013-2015, she was digital forensic analyst and digital scholarship and preservation services fellow working on the library’s neh-funded project, preservation and access frameworks for digital art objects. dd388@cornell.edu @smallandmath julia kim is the digital assets manager at the american folklife center at the library of congress. she was previously a national digital stewardship resident at new york university libraries where she completed the work covered in this article. juliakim@loc.gov @jy_kim29 morgan mckeehan is a national digital stewardship resident in the 2015-16 ndsr-nyc cohort. her ndsr host site is rhizome, an internet-based contemporary art organization dedicated to promoting richer and more critical digital cultures. morgan.mckeehan@rhizome.org @anyformation alison rhonemus is the digital archives assistant at the new york public library. she has been working on born-digital collection material there since a previous internship focused on digital forensics. alisonrhonemus@nypl.org acknowledgements the authors would like to thank lisa darms, dragan espenschied, thomas liebetraut, susan malsbury, donald mennerich, klaus rechert, jennifer ulrich and the bwfla project. this work was done with the support of many organizations, including the ndsr in new york program, which is funded by imls, and implemented by the metropolitan new york library council in partnership with brooklyn historical society, and the neh-funded preservation and access framework for digital art objects project at cornell university library. finally thanks to the researchers that were willing to come in and give emulation a shot! endnotes [1] rothenburg, j. 1999. avoiding technological quicksand: finding a viable technical foundation for digital preservation. council on library and information resources [internet]. [cited 2016 feb 19]; 77. available from: http://www.clir.org/pubs/reports/rothenberg/pub77.pdf [2] bearman, d. 1999. realities and chimeras in digital preservation. d-lib magazine [internet]. [cited 2016 feb 17]; 5(4). available from: http://www.dlib.org/dlib/april99/bearman/04bearman.html [3] carrol l, farr e, hornsby p, ranker b. 2011. a comprehensive approach to born-digital archives. archivaria [internet] [cited 2016 feb 17]; 72: 61-92. available from: http://pid.emory.edu/ark:/25593/cksgv/ [4] for this paper, we are using “emulation” to mean any strategy where a hardware and software environment are rendered in a host machine, and “emulator” to mean any program that lets you do this. [5] preservation and access framework for digital art objects was a two-year research and development grant at cornell university library funded by the national endowment for the humanities. for more information on the grant, see (rieger et. al 2015) and the site’s wiki for more information about the project scope, activities, and advisory board: https://confluence.cornell.edu/display/pafdao/ [6] casad m, rieger o. y., alexander d. 2015. enduring access to rich media content: understanding use and usability requirements. d-lib magazine(21) 9. doi: 10.1045/september2015-casad [7] kim j, mennerich d. forthcoming. jeremy blake’s time-based paintings: a case study. electronic media group, volume 4. [8] espenschied, d. 2012. one terabyte of kilobyte age: digging through the geocities torrent. [cited 2015 feb 17. available from: http://blog.geocities.institute/archives/3214/ [9] rhizome to restore and present theresa duncan cd-roms [internet].[updated 2014 nov 18]. rhizome; [cited 2016 feb 19]. available from: http://rhizome.org/editorial/2014/nov/18/announcing-theresa-duncan/ [10] rhizome’s emulators are hosted on google compute. [11] robertson, a. 2015. the girl game archival project that’s rewriting geek history. the verge [internet]. [cited on 2016 feb 19]. available from: http://www.theverge.com/2015/4/17/8436439/theresa-duncan-chop-suey-cd-rom-preservation/ [12] d. espenschied, personal communication, april 1, 2015 [13] espenschied d, valizada i, stobbe o, liebetraut t, rechert k. 2015. (re-)publication of preserved, interactive content – theresa duncan cd-roms: visionary videogames for girls. in ipres 2015 annual meeting, 2015 nov 2-6; chapel hill, nc [14] for an example of system requirements on an artwork, see: https://newcatalog.library.cornell.edu/catalog/6099417/ [15] see also https://twitter.com/euanc/status/698265036997992448/ for an excellent graph of operating system usage over time. [16] timeline of operating systems [internet].[updated 2016 feb 18]. wikipedia; [cite 2016 feb 19]. available from: https://en.wikipedia.org/wiki/timeline_of_operating_systems/ [17] timeline of operating systems [internet].[updated 2016 feb 18]. wikipedia; [cite 2016 feb 19]. available from: https://en.wikipedia.org/wiki/timeline_of_operating_systems [18] dietrich d. and adelstein f. 2015. archival science, digital forensics, and new media art. digital investigation(14) supplement 1. doi: http://dx.doi.org/10.1016/j.diin.2015.05.004 [19] bitcurator [internet].[updated 2016 jan 18]. bitcurator.net; [cited 2016 feb 19]. available from: http://www.bitcurator.net/ [20] ftk imager [internet].[updated 2010 dec 17]. forensics wiki; [cited 2016 feb 19]. available from: http://www.forensicswiki.org/wiki/ftk_imager/ [21] isobuster [internet].[updated 2016]. smart projects; [cited 2016 feb 19]. available from: http://www.isobuster.com/ [22] hfs+ was introduced in 1998 by apple computer. [23] beech, william a.. 2015. floppy disk formats, standards and geometry [internet]. [cited 2016 feb 19]. available from: http://www.nj7p.org/computers/disk%20subsystems/floppies.html [24] the kryoflux is a hardware and software system used to create disk images of floppy disks. http://www.kryoflux.com/ [25] software preservation society, glossary, retrieved from http://www.softpres.org/glossary:encoding/ [26] see the wikipedia entry for file for example usage: https://en.wikipedia.org/wiki/file_%28command%29#examples [27] even though it’s tempting, the “version 1” part doesn’t help you out that much: https://en.wikipedia.org/wiki/rich_text_format#history [28] if you’re feeling adventurous, you can also use the output of the command line utility strings to find more deeply hidden clues as to what to do next. see http://www.thegeekstuff.com/2010/11/strings-command-examples/. [29] magic number (programming) [internet].[updated 2016 jan 10]. wikipedia; [cited 2016 feb 19]. available from: https://en.wikipedia.org/wiki/magic_number_%28programming%29 [30] the national archives. n.d. download droid: file format identification tool [internet]. [cited 2016 feb 19]. available from: http://www.nationalarchives.gov.uk/information-management/manage-information/preserving-digital-records/droid/ [31] this conceptual approach is similar to the commonalities noted in the reference table provided by the pafdao project’s documentation, mentioned above. the bwfla project is also working to automate the process of determining suitable emulation environments for a given set of files (rechert et al. 2015) ipres. [32] rechert k, liebetraut t, stobbe o, valizada i, and steinke t. 2015. characterization of cd-roms for emulation-based access. in ipres 2015. [33] espenschied d, valizada i, stobbe o, liebetraut t, rechert k. 2015. (re-)publication of preserved, interactive content – theresa duncan cd-roms: visionary videogames for girls. in ipres 2015 annual meeting, 2015 nov 2-6; chapel hill, nc [34] list of computer system emulators [internet]. wikipedia, the free encyclopedia; 2016 jan 10 [cited 2016 feb 17]. available from: https://en.wikipedia.org/wiki/list_of_computer_system_emulators/ [35] dietrich d. emulation and classification documentation for pafdao project. available from: http://hdl.handle.net/1813/41453 [36] dietrich d. emulation and classification documentation for pafdao project. available from: http://hdl.handle.net/1813/41453 [37] setting up sheepshaver for mac os x [internet].[updated 2014 feb 2]. emaculation.com; [cited 2016 feb 19]. available from: http://www.emaculation.com/doku.php/sheepshaver_mac_os_x_setup/ [38] qemu: creating a hard disk image [internet].[updated 2016 feb 15]. [cited 2016 feb 19]. available from: https://wiki.archlinux.org/index.php/qemu#creating_a_hard_disk_image [39] http://wiki.qemu.org/links#gui_front_ends [40] capturing a mac rom image [internet].[updated 2010 mar 20]. emaculation.com; [cited 2016 feb 19]. available from: http://www.emaculation.com/doku.php/capturing_rom/ [41] see http://www.redundantrobot.com/sheepshaver-tutorial/ and https://archive.org/details/mac_rom_archive_-_as_of_8-19-2011/ for macintosh rom files that you can use and http://fs-uae.net/kickstarts for amiga kickstart roms. [42] rechert, k. 2015a. emulation as a service as a docker (beta) [internet]. [cited 2016 feb 19]. available from: http://bw-fla.uni-freiburg.de/wordpress/?p=817 [43] rechert, k. 2015b. boot to emulation — eaas as a local option (beta) [internet]. [cited 2016 feb 19]. available from: http://bw-fla.uni-freiburg.de/wordpress/?p=844 [44] rechert, k. 2015a. emulation as a service as a docker (beta) [internet]. [cited 2016 feb 19]. available from: http://bw-fla.uni-freiburg.de/wordpress/?p=817; and rechert, k. 2015b. boot to emulation — eaas as a local option (beta) [internet]. [cited 2016 feb 19]. available from: http://bw-fla.uni-freiburg.de/wordpress/?p=844 [45] kim, julia. “researcher interactions with born-digital: out of the frying pan and into the reading room.” saa ers blog. society of american archivists. blog. [cited 2016 march 23]. available from: https://saaers.wordpress.com/ [46] aims work group. 2012. aims born-digital collections: an inter-institutional model for stewardship. [cited 2016 feb 17], p.37. available from: http://www2.lib.virginia.edu/aims/whitepaper/aims_final.pdf [47] personal correspondence with madeleine casad, 2015. [48] hedstrom m, lee c, olson j, lampe c. 2006. “the old version flickers more”: digital preservation from the user’s perspective. the american archivist [internet]. [cited 2016 feb 19]; 69(1):159-187. available from: http://www.ils.unc.edu/callee/dig-pres_users-perspective.pdf [49] espenschied d, valizada i, stobbe o, liebetraut t, rechert k. 2015. (re-)publication of preserved, interactive content – theresa duncan cd-roms: visionary videogames for girls. in ipres 2015 annual meeting, 2015 nov 2-6; chapel hill, nc [50] espenschied d, valizada i, stobbe o, liebetraut t, rechert k. 2015. (re-)publication of preserved, interactive content – theresa duncan cd-roms: visionary videogames for girls. in ipres 2015 annual meeting, 2015 nov 2-6; chapel hill, nc [51] liebtraut, t. unifying access to virtual disk images – the eaas way[internet]. [cited 2016 march 23]. available from: http://bw-fla.uni-freiburg.de/wordpress/?p=914 [52] for more detail on how the pafdao team used emulation for analysis, see (dietrich and adelstein 2015). [53] there are several recent reports highlighting other challenges to implementing emulation. see legal challenges: http://blogs.loc.gov/digitalpreservation/2016/01/intellectual-property-rights-issues-for-software-emulation-an-interview-with-euan-cochrane-zach-vowell-and-jessica-meyerson/ and for an assessment of the current state here: https://mellon.org/media/filer_public/0c/3e/0c3eee7d-4166-4ba6-a767-6b42e6a1c2a7/rosenthal-emulation-2015.pdf bibliography aims work group. 2012. aims born-digital collections: an inter-institutional model for stewardship. [cited 2016 feb 17], p.37. available from: http://www2.lib.virginia.edu/aims/whitepaper/aims_final.pdf bearman, d. 1999. realities and chimeras in digital preservation. d-lib magazine [internet]. [cited 2016 feb 17]; 5(4). available from: http://www.dlib.org/dlib/april99/bearman/04bearman.html beech, william a.. 2015. floppy disk formats, standards and geometry [internet]. [cited 2016 feb 19]. available from: http://www.nj7p.org/computers/disk%20subsystems/floppies.html bitcurator [internet].[updated 2016 jan 18]. bitcurator.net; [cited 2016 feb 19]. available from: http://www.bitcurator.net/ carrol l, farr e, hornsby p, ranker b. 2011. a comprehensive approach to born-digital archives. archivaria [internet] [cited 2016 feb 17]; 72: 61-92. available from: http://pid.emory.edu/ark:/25593/cksgv capturing a mac rom image [internet].[updated 2010 mar 20]. emaculation.com; [cited 2016 feb 19]. available from: http://www.emaculation.com/doku.php/capturing_rom casad m, rieger o. y., alexander d. 2015. enduring access to rich media content: understanding use and usability requirements. d-lib magazine(21) 9. doi: 10.1045/september2015-casad dietrich d. emulation and classification documentation for pafdao project. available from: http://hdl.handle.net/1813/41453 dietrich d. and adelstein f. 2015. archival science, digital forensics, and new media art. digital investigation(14) supplement 1. doi: http://dx.doi.org/10.1016/j.diin.2015.05.004 encoding [internet].[updated 2006 mar 07]. software preservation society glossary; [cited 2016 feb 19]. available from: http://www.softpres.org/glossary:encoding ftk imager [internet].[updated 2010 dec 17]. forensics wiki; [cited 2016 feb 19]. available from: http://www.forensicswiki.org/wiki/ftk_imager guttenbrunner m, von suchodoletz d, rechert k. 2012. towards practical emulation tools and strategies [white paper]. in: ipres 2012 annual meeting, 2012 oct 1-4; toronto. https://emulationws2012.files.wordpress.com/2012/10/emu-ws-intro-paper.pdf hedstrom m, lee c, olson j, lampe c. 2006. “the old version flickers more”: digital preservation from the user’s perspective. the american archivist [internet]. [cited 2016 feb 19]; 69(1):159-187. available from: http://www.ils.unc.edu/callee/dig-pres_users-perspective.pdf isobuster [internet].[updated 2016]. smart projects; [cited 2016 feb 19]. available from: http://www.isobuster.com/ jeremy blake papers; mss 217; fales library and special collections, new york university libraries kay r. 2009. emulation or virtualization? computerworld. [cited 2016 feb 18]. available from: http://www.computerworld.com/article/2551154/virtualization/emulation-or-virtualization-.html kim, julia. “researcher interactions with born-digital: out of the frying pan and into the reading room.” saa ers blog. society of american archivists. blog. [cited 2016 march 23]. available from: https://saaers.wordpress.com/ kim j, mennerich d. forthcoming. jeremy blake’s time-based paintings: a case study. electronic media group, volume 4. konstantelos http://radar.gsa.ac.uk/2806/1/pocos_vol_2_final_release%5b1%5d.pdf list of computer system emulators [internet]. wikipedia, the free encyclopedia; 2016 jan 10 [cited 2016 feb 17]. available from: https://en.wikipedia.org/wiki/list_of_computer_system_emulators magic number (programming) [internet].[updated 2016 jan 10]. wikipedia; [cited 2016 feb 19]. available from: https://en.wikipedia.org/wiki/magic_number_%28programming%29 mckeehan m. 2016. intellectual property rights issues for software emulation: an interview with euan cochrane, zach vowell, and jessica meyerson. the signal: digital preservation [internet]. [cited 2016 feb 19]. available from: http://blogs.loc.gov/digitalpreservation/2016/01/intellectual-property-rights-issues-for-software-emulation-an-interview-with-euan-cochrane-zach-vowell-and-jessica-meyerson/ mennerich d, kim j. 2015. preserving jeremy blake’s digital art and archives. imaging science & technology [internet]. [cited 2016 feb 18]; vol 1: 49-50. available from: http://ist.publisher.ingentaconnect.com/content/ist/ac/2015/00002015/00000001 the national archives. n.d. download droid: file format identification tool [internet]. [cited 2016 feb 19]. available from: qemu: creating a hard disk image [internet].[updated 2016 feb 15]. [cited 2016 feb 19]. available from: https://wiki.archlinux.org/index.php/qemu#creating_a_hard_disk_image rieger o, murray t, casad m, alexander d, dietrich d, kovari j, muller l, paolillo m, mericle dk. 2015. preserving and emulating digital art objects [internet]. [cited 2016 mar 23]. available from: http://hdl.handle.net/1813/41368 rhizome to restore and present theresa duncan cd-roms [internet].[updated 2014 nov 18]. rhizome; [cited 2016 feb 19]. available from: http://rhizome.org/editorial/2014/nov/18/announcing-theresa-duncan/ rechert k, liebetraut t, stobbe o, valizada i, and steinke t. 2015. characterization of cd-roms for emulation-based access. in ipres 2015. rechert, k. 2015a. emulation as a service as a docker (beta) [internet]. [cited 2016 feb 19]. available from: http://bw-fla.uni-freiburg.de/wordpress/?p=817 rechert, k. 2015b. boot to emulation — eaas as a local option (beta) [internet]. [cited 2016 feb 19]. available from: http://bw-fla.uni-freiburg.de/wordpress/?p=844 rechert k, valizada i, von suchodoletz d, latocha j, “bwfla – a functional approach to digital preservation”, pik, 35 (4), de gruyter. s. 259-267. doi: 10.1515/pik-2012-0044 robertson, a. 2015. the girl game archival project that’s rewriting geek history. the verge [internet]. [cited on 2016 feb 19]. available from: http://www.theverge.com/2015/4/17/8436439/theresa-duncan-chop-suey-cd-rom-preservation rothenburg, j. 1999. avoiding technological quicksand: finding a viable technical foundation for digital preservation. council on library and information resources [internet]. [cited 2016 feb 19]; 77. available from: http://www.clir.org/pubs/reports/rothenberg/pub77.pdf rosenthal d s.h. emulation and virtualization as preservation strategies [internet]. [cited 2016 feb 19]: available from: http://mellon.org/rosenthal-emulation-2015/ setting up sheepshaver for mac os x [internet].[updated 2014 feb 2]. emaculation.com; [cited 2016 feb 19]. available from: http://www.emaculation.com/doku.php/sheepshaver_mac_os_x_setup software preservation society, glossary, retrieved from http://www.softpres.org/glossary:encoding timothy leary papers, manuscripts and archives division, the new york public library subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – use of cue sheets in audio digitization mission editorial committee process and structure code4lib issue 23, 2014-01-17 use of cue sheets in audio digitization audio digitization is becoming essential to many libraries. as more and more audio files are being digitally preserved, the workflows for handling those digital objects need to be examined to ensure efficiency. in some instances, files are being manually manipulated when it would be more efficient to manipulate them programmatically. this article describes a time-saving solution to the problem of how to split master audio files into sub-item tracks. by austin dixon if done by hand, the digitization process for audio can be slow, tedious, and error prone. thankfully, the process is benefited by the use of custom scripts that utilize open source tools. i will describe three scripts that i wrote for our department and how they have greatly improved our workflow. traditionally, the workflow for audio digitization involves manipulating the master .wav files manually using the flowing steps: step 1. digitize the original reel tapes into master. wav files. step 2. digitally optimize each file (level the volume and remove extraneous silence) using a digital audio editing suite such as sound forge [1]. step 3. split master .wav files by hand into sub-item tracks using the audio editing suite. this is done by visually selecting a portion of the audio file’s wave form, copying it, and then saving that copy as a new file. step 4. convert sub-item tracks from .wav to .mp3 format using the audio editing suite. the master .wav files are then archived on a server and the .mp3 tracks are uploaded to an online database. after initially performing this process manually for several items, i realized that the process was unnecessarily problematic. specifically, splitting the master .wav file in step 2 was not only a very tedious task which took an ample amount of time to complete, but it introduced too much room for human error. it is difficult to visually select accurate start and end times, and this difficulty increases with the number of tracks that must be split. when done incorrectly, tracks begin too soon or end too early or unacceptable overlaps are introduced. so i began researching how we could complete this task programmatically, thus speeding up the process and removing the possibility of human error. i knew that in order to split an audio file programmatically, i would need a reference file to identify the location of begin and end times for each sub-item track, and this reference file would need to adhere to a strict schema so my scripts could easily process it. my research revealed that there were not many readily available schema options. i considered using adl (audio decision list) files, which is a form of metadata that many librarians use for this exact purpose, but adl files were larger and more complicated then i needed for such a simple task. they also are not designed to be easily readable by humans, which becomes important if any sort of troubleshooting is ever required. i eventually decided to use cue sheets [2]. the cue sheet schema was first developed to be used with cdrwin (a cd/dvd burning software for microsoft windows), and can be used by media players to determine where tracks begin and end. while not typically used by libraries for digitization purposes, using cue sheets to split our tracks appealed to me over other methods for three reasons: unlike other options, cue sheets can be used by many open source tools such as shntool, a .wav processing utility, to manipulate files [3]. using these open source tools would save me a considerable amount of programming time. cue sheets are easy for humans to read. cue sheets are easy to build via a script because they follow a very simple format. after studying up on cue sheet structure, i created a script, cuemaker [4] (using python 3.2), that will take a text file i supply with item identifiers and track times, and use it to generate a .cue file for each item. this input text file is easy to create; in our workflow a metadata librarian is responsible for supplying me with item level metadata for each track, which includes the item identifiers and track times. i simply cut and paste this information into a text file in the following manner: filename1 begin time filename2 begin time only each track’s begin time is required, as each begin time is also the previous track’s end time. the first track’s begin time is always 00:00. figure 1. sample cue sheet. figure 2. understanding the parts of a cue sheet. here is a sample of the python code i used to build the. cue file: def main(): filename = getfile() tracktitle, tracktime = timecode(filename) makecue(filename, tracktitle, tracktime) def getfile(): f = open(&quot;cue.txt&quot;,&quot;r&quot;) filename = f.readlines() f.close() return filename def timecode(filename): tracktime = [] tracktitle = [] num = 1 while (num &lt; len(filename)): s = filename[num].split(' ') time = s[1] title = s[0] tracktime.append(time) tracktitle.append(title) num = num + 1 return tracktitle, tracktime def makecue(filename, tracktitle, tracktime): t = 1 num = 0 q = '&quot;' header = 'file ' + q + filename[0].rstrip() + q + ' wave' file = open(&quot;file.cue&quot;,&quot;w&quot;) file.write(header) file.write('\n\n') while (num &lt; len(tracktime)): if (t &lt; 10): trackname = &quot;title &quot; + q + tracktitle[num] + q ts = &quot;0&quot; + str(t) track = &quot;track &quot; + ts + &quot; audio&quot; index = &quot;index 01 &quot; + tracktime[num].rstrip() file.write(track) file.write('\n') file.write(index) file.write('\n') file.write(trackname) file.write('\n\n') t = t + 1 num = num + 1 else: trackname = &quot;title &quot; + q + tracktitle[num] + q track = &quot;track &quot; + str(t) + &quot; audio&quot; index = &quot;index 01 &quot; + tracktime[num].rstrip() file.write(track) file.write('\n') file.write(index) file.write('\n') file.write(trackname) file.write('\n\n') t = t + 1 num = num + 1 file.close() main() once the cuemaker script was functional, i created another script, cuesplitter [5](this time using perl 5.16.1), that sends a command line argument to the system evoking shntool. shntool is an open-source wave data processing utility. i choose to use it in my script because it handles .wav files very well, and can perform complicated tasks with one-line commands. i can also use it in conjunction with lame, an open-source software codec used to encode/compress audio into .mp3 file format [6]. the command i use in my script makes shntool read the generated .cue file and use it to split the master .wav file into sub-item tracks. once split, the sub-item tracks are converted from .wav to .mp3 format using lame. here is the command: shntool split -f cue_file -t%t -o “cust ext=mp3 lame --quiet %f” -d output_file wav_file the script also names the files correctly by using the item identifier (title) in the .cue file. then i created a graphic user interface, cue gui [7] (using perl 5.16.1 and tkx, a perl module that “provides yet another tk interface for perl” [8]), that could launch both scripts with the push of a button. figure 3. screenshot of the graphic user interface. so our new workflow now consists of: step 1. digitize the original reel tapes into master .wav files. step 2. digitally optimize each file (level the volume and remove extraneous silence) using a digital audio editing suite. step 3. run cuemaker and cuespliter scripts by using cue gui. then archive the master .wav files to our server and upload the resulting .mp3s to our online database. as you can see, steps one and two have not changed. i am working on a script that will level the audio automatically. when it is complete, i will build it into the cuesplitter script. unfortunately, the extraneous silence removal will (probably) always require human intervention. to test these new steps, we used them to process 18 new items, consisting of 207 audio tracks. the time and effort it took to upload these tracks using the scripts instead of our old workflow was significantly lower since the most labor intensive tasks are now completed instantly with one button press. we have gauged that this process saves us approximately one hour per item. other libraries that are digitizing audio in a manner that results in a concatenated master file should be able to use the method i have outlined here to help them split and convert the audio tracks quickly and easily. i have made the code for my cue scripts open source so other libraries can use them in their workflow as needed. the scripts can also be easily customized by anyone with some basic programming knowledge. references [1] sound forge product family overview [internet]. middleton (wi): sony creative software; [cited 2014 jan 13]. available from: http://www.sonycreativesoftware.com/soundforgesoftware. [2] wikipedia contributors. cue sheet (computing) [internet]. wikipedia, the free encyclopedia; 2014 jan 7, 0:13 utc [cited 2014 jan 13]. available from: http://en.wikipedia.org/wiki/cue_sheet_(computing). [3] jason jordan. shntool [internet]. etree.org; [cited 2014 jan 13]. available from: http://www.etree.org/shnutils/shntool/. [4] cuemaker [internet]. university of alabama libraries digital services; 2013 jun 11, 14:10 edt [cited 2014 jan 13]. available from: http://www.lib.ua.edu/wiki/digcoll/index.php/cuemaker. [5] cuesplitter [internet]. university of alabama libraries digital services; 2013 jun 11, 14:09 edt [cited 2014 jan 14]. available from: http://www.lib.ua.edu/wiki/digcoll/index.php/cuesplitter. [6] roberto amorio and sebastian mares. lame mp3 encoder [internet]. the lame project; [cited 2014 jan 14]. available from: http://lame.sourceforge.net/index.php. [7] cue gui [internet]. university of alabama libraries digital services; 2013 jun 11, 13:58 edt [cited 2014 jan 14]. available from: http://www.lib.ua.edu/wiki/digcoll/index.php/cue_gui. [8] tkx [internet]. cpan; [cited 2014 jan 14]. available from: http://search.cpan.org/~gaas/tkx-1.09/lib/tkx.pm. about the author austin dixon (ardixon1@ua.edu) is the digitization technologist for the university of alabama libraries’ digital services department. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – customizing alma and primo for home & locker delivery mission editorial committee process and structure code4lib issue 50, 2021-02-10 customizing alma and primo for home & locker delivery like many ex libris libraries in fall 2020, our library at california state university, northridge (csun) was not physically open to the public during the 2020-2021 academic year, but we wanted to continue to support the research and study needs of our over 38,000 university students and 4,000 faculty and staff. this article will explain our alma and primo implementation to allow for home mail delivery of physical items, including policy decisions, workflow changes, customization of request forms through labels and delivery skins, customization of alma letters, a python solution to add the “home” address type to patron addresses to make it all work, and will include relevant code samples in python, xsl, css, xml, and json. in spring 2021, we will add the on-site locker delivery option in addition to home delivery, and this article will include new system changes made for that option. by christina l. hennessey introduction the california state university, northridge (csun) library serves a campus of 38,310 students and over 4,000 faculty and staff (fall 2020) [1]. a physical library that is normally open 7 days a week (an average of 90 hours a week with extended hours for finals), with over 1.3 million in-person visits per year and over 71,000 physical material circulations a year (2018-2019 statistics) [2], was suddenly closed to the public in march 2020 due to covid-19 concerns. a library and campus closure of what was predicted to be a few weeks, turned into a closure for for the rest of 2020 and then of an entire academic year, and we needed to find safe ways to continue to provide physical items for our patrons and their study and research needs. using our ex libris alma and primo systems, we implemented home delivery of physical items in time for the fall 2020 semester (august 24, 2020). in addition to the home delivery option, in spring 2021, we will start delivering library materials to an outdoor locker where patrons can pick up their library items in a touchless and safe manner. neither the home delivery nor the locker delivery were automatically available in the existing system without several configuration changes, which will be covered here. csun is located in the county of los angeles in california, one of the strictest counties in the united states for covid-19 closures and lockdowns. our library opening decisions along the way had to accommodate the decisions of local and state governments, and at the california state university level and the campus level, in addition to our own library administration. our decisions to implement home & locker delivery were also influenced by the fact that although our physical library was centrally-located on campus, there was no way to quickly drive up to the building or park your vehicle nearby, so curbside pickup of library items was not an option. our faculty, staff, and students live an average of 11 miles from campus (2019) [3] so home delivery times through mail could be reasonable, and a visit to a local library locker on campus would not be unreasonable. even though our physical library has been closed since mid-march 2020, the library and library staff continue to provide library services through libchat, libanswers, text and phone, email, and zoom appointments. the staff working these services needed to be familiar with the home and locker services we provided so that they can support the questions of our patrons. csun migrated to the alma and primo systems in 2017, as part of a shared system between 23 california state university campuses and a central chancellor’s office. decisions and systems changes made locally at csun needed to match up and complement the decisions made at the chancellor’s level. home delivery implementation as it became clear in july 2020 that the library was not going to re-open physically for the first day of classes for fall 2020 (august 24, 2020), we needed to determine how to implement home delivery in our alma and primo systems in a way that was publicand staff-friendly, and easy to use. alma and primo have a configurable “resource request” function [4] with online forms that can be customized and can pass the bibliographic & user information directly to the alma request queue for staff processing. an alternative option would be for patrons to just fill out a webform, or send an email requesting materials for mail delivery, but we wanted to make the process as easy as possible when the patron had already found the material in our library discovery system (http://bit.ly/csunprimo) and they had already logged in with their library user account. with students having to move back home or find other places to live in town due to not needing to come to campus this semester, we also wanted to give the option for requesters to input a different mailing address than the one the library already had in its database. in addition to our own local patrons requesting our local materials for home delivery, we wanted to allow our patrons to do their regular resource sharing requesting through this implementation as well. with the charge of “make home delivery happen by fall 2020,” there were many decisions to make and discuss, which was very difficult during a time when all our meetings had to be virtual and making sure that all those affected and had knowledge and opinions were included in the analysis. discussions included: what user groups can requestoriginally, we were going to allow home delivery requests for all of our 25 alma user groups, the same user groups that could circulate physical materials when we were open. however, concerns about the possible popularity of the service led us to first review recent circulation statistics from each user group to determine what the impact would be for supporting all user groups, and what groups we could leave out of the service. in the end, we only allowed requests for students, faculty, and staff, and not other campus-affiliated user groups like community members, high school students, and alumni. there was the concern on messaging and the goodwill that might be lost with excluding some of these groups (e.g. local high school students that might end up attending csun because of their positive csun library experience) but this has turned out to not be a huge issue – it seems that these other groups respect that this is a temporary circulation restriction due to the pandemic, and they will be able to access and request our library materials soon again in the future. what material types and locations can be requestablebeing a relatively new alma & primo implementation (only three years at the time of home delivery implementation), we had not done much with the resource request features in the past. for home delivery implementation, we needed to make decisions on each one of our 103 locations and 101 material types, and what could be requestable and what could not. some of the decisions were easy (we were not going to circulate laptops as the entire inventory was already checked out when we abruptly closed in march 2020) and some were more difficult, such as some of our more unique materials from our national center on deafness (ncod) collections. we also needed to revisit what locations and material types were requestable by patrons from other libraries for resource sharing, and if we wanted those to circulate to other libraries during this time. even for some of our normally non-circulating collections, these were revisited as something marked as “non-circulating” may have just meant “this really is allowed to circulate but not automatically as we want to manually approve it.” for example, some of our ncod collections were normally non-circulating to cut down on use of rare materials but could circulate with special permission granted at public services desks. with no staff to manually approve these requests in person, we needed to allow circulation and requesting on the item first through home delivery, and then the staff processing the request could make that decision instead to circulate the item or cancel the request with an explanation note. limits on how many items could be requested at a timeas with the limits on non-circulating locations with manual override allowed, we did not previously have limits on the number of items that could be checked out at a time for some of our patron groups. even though there were no limitations on the number of checkouts, a library staff member would notice if someone brought 200 books to the desk to check out, and could enforce a manual limit if necessary. however, with all the requests and checkouts being virtual, we needed to set limits on how many requests could be done at a time to lessen the impact on staff processing time and the home delivery costs. we decided that five items could be requested at once based on the average number of books checked out at a time when we were physically open, and concerns that one or two users might request a large number of books at a time and strain our mailing budget. five requests at a time seems like a low number until you realize that as soon as a request was processed (as either a checkout or cancellation), then more requests could be made, so patrons could still have many books checked out at once for research or study. this limitation of five requests could also be overridden by a staff member from a phone call or email if necessary. we only experienced a few times in fall 2020 where a patron requested five items at a time, and no one ever called or emailed to request more, so this was not as big of an issue as we expected. how often requests will be processeddue to csun limitations on the number of people that should be in an area or office on campus during this time, only one person in our guest services department was assigned each weekday to come to campus to pick up and delivery the library’s mail from campus mail (which was no longer delivering to or picking up from the library during campus closure), pick up items from the 10 library bookdrops across campus, and then process the requests that came in since the last processing of the queue, including retrieving items from the stacks and cancelling requests for items that were not found. so if a patron made a request for home delivery on a friday afternoon, the request might not be processed until monday morning. we tried to communicate this possible delay to our patrons but in a time of “amazon delivery time expectations,” there were still attempts at multiple duplicate requests for patrons that had assumed their request did not go through since there was no action for a few days. handling/quarantine of retrieved itemsthere was also much discussion on how the retrieval and processing of items from the stacks would work with scientific unknowns of how long the covid-19 virus may live on materials. we kept a close eye on the testing and recommendations from the realm project[5]. with one person in the stacks each day, over 2 million items in the stacks across four floors, a map collection, and an automatic storage and retrieval system (as/rs) in a separate part of the library building, the likelihood that a staff member would handle books in the same area often was low. we decided that the home delivery books would not be quarantined after retrieval from the stacks and would be immediately mailed. any virus living on the items would be gone by the time the item arrived at the patron’s home address. basics of system setup and the patron experience there are many ways that one can set up home delivery inside of alma and primo, but the following explanation is how csun chose to do it. we implemented our local home delivery requests through the hold request form in primo, which is customizable for which fields you wish to share in your form. when the patron is logged into their library account in primo, and they are part of a user group that allows requesting on an item material type and location that allows requesting, the request link will appear in the “get it” display in primo. you don’t have to name this link “request”, you could have “request for home delivery” or any text you want there, but “request” is the text default. figure 1. request link in primo. the “request” link takes you to this next screen. figure 2 is the look of the out-of-the-box request form, and figure 3 is what the request form can look like after customization. figure 2. request form with no customization. figure 3. request form with customization. customizations of this form can be done through alma’s discovery interface logic labels and the hold request form customization [6], and through delivery skins in alma and primo back office. the labels can include html to include links and formatting. we changed the name of the default “comment” field to “mailing address” so the patron could add their preferred mailing address, instead of us assuming they wanted the item mailed to the mailing address we have on file for them. this will pass the mailing address through to the request queue in the “note” field for each request. the hold request form customization will need to have the “comment” field turned on to show mailing address, and the “generaluselabel” field turned on for all the text that is below mailing address on this form example. during our home delivery-only phase, we did not want to confuse the patron with a pickup location choice, since there was no choice to make, so we removed this selection and field from the form. to do this, we used delivery skins [7] and updated the mashup_new.css file with the text in figure 4 to remove that particular field. for institutions using primo back office, you will also need to change some labels in the mapping tables – delivery subsystem – templates to point to the new delivery skin name, and then deploy the changes. /* hide pickup locations on hold request form clh 08/19/20 */ .holdrequest #pickuplocationh { display: none; } .holdrequest #pickuplocationv { display: none; } /* end hide pickup locations on hold request form clh 08/19/20 */ figure 4. the pickup location is hidden on the hold request form. once the patron has completed their request, it will show up in the alma request queue, which staff can find at the menu option monitor requests & item queue in alma. all the items requested for home delivery will show up with pickup location = home delivery. figure 5. home delivery item in the request queue. staff will take the list of requested items to the stacks to retrieve each one. for all items not found, the request is cancelled with a note to the patron that they can try requesting it through other csu libraries or interlibrary loan. we chose to not automatically convert these requests to resource sharing requests as the patron might not want it if it is arriving in 3-4 weeks instead of 1-2 weeks and we did not want to involve the cost of interlibrary loan for an unwanted request. for all items found, the items will be checked out to a local circulation desk. this will send an automatic email that the item is on the way. in alma, this is the loan receipt letter. figure 6. loan receipt letter in alma. alma letters are very configurable using labels in each letter and through the xsl template for each letter. depending on what library and department the item was from, we had conditional text in the letter so the contact information at the bottom of the letter is different, and so we don’t recommend that library technology equipment is returned to the bookdrops. figure 7. xsl conditional text in alma loan receipt letter. after all items are checked out to patrons, they are packed into envelopes pre-printed with the patron’s chosen mailing address and mailed that day from campus mail. within a week, patrons will receive the items in the mail. python script needed to make home delivery work in addition to the system changes for hold request forms and letters, other changes must be made in alma to support what ex libris calls “personal delivery,” [8] which can be used to deliver to home or office addresses. the request terms of use (tou) for all requestable locations must have the following fields changed to support personal delivery: is requestable = requestable (if it wasn’t already) pickup locations = pick up only in owning library (only if you removed the pickup location from the form. if you kept it, use the setting “anywhere”) on shelf request policy = allow for pickup anywhere regardless of availability personal delivery = home (or “office”, or “all” for home and office) personal delivery fee = none (you can set a fee for personal delivery costs, we did not) the circulation desk configuration for each library in your alma system that will be supporting home delivery will need to have the option checked to “support personal delivery”. if you would like to inform your patrons that the item is on the way via the loan receipt letter email, you will need to turn on “send loan receipts” for that circulation desk. one of the limitations of the ex libris personal delivery option is that the requesting patron must have an alma user record with at least one mailing address type of “home” or “work” in order for the request option to appear in primo [9]. options for mailing address types in alma are: alternative, home, work (the “work” type also referred to as “office” in some of the documentation), and school. all of csun’s faculty & staff user records have a preferred mailing address type of “home,” but all our student records only had an address marked as “school” so they could not request home delivery unless all their records were updated. through testing, we determined that an address can have multiple address types assigned to it, so if we just add “home” to any address that doesn’t already have that type, that will solve the development problem and home delivery will show up for these users. depending on where your user records originate from, who controls the configuration of that data and the transfer to the alma database, and how often this update occurs, you may be able to solve this problem with a full update of all user records, adding the “home” address type to each address in each record. at csun, our user database is updated weekdays at 4am from our campus’ student information system (sis) database, picking up any changes in enrollment, new users, and new addresses for users each day. we chose to not request an update to the centralized records but to control our updates locally in our alma database. however, this decision would require us to check and update our user records daily for records missing the “home” address type. the first problem is to determine which alma user records are missing the “home” address in the user database. to save time, you can limit the set of user records you want to update to active, non-expired users, and if you have determined ahead of time what user groups will be able to do requesting, you only need to update records for those user groups instead of the whole database. in july 2020, i determined that out of our 48,000 active, non-expired user records, only around 5,000 of them had a “home” address type. there was no way i could update these user records one-by-one in time for the fall 2020 semester, so i created a python program that uses the alma user rest api [10] to get an alma user record, determine if the preferred address had a home address type, and if not, to put it back into the user record. since only one address needs to have the home address type for personal delivery to work, i chose to only check and update the preferred address for a home type, instead of the unlimited number of addresses a user record can have. you can run the program on your entire database of primary ids, or a quicker process is to just run it on the ids that you have already determined are missing a home address type. the program works on both internal and external address records. the program is available at github, as either a windows executable or a python program that can be modified for your own needs. the program requires the following modules: tkinter, requests, and xmltodict. here is an example of an alma user record that needs the update. figure 8. a user record that has a preferred address with only the type “work.” the python program accepts a filename as input: figure 9. python program asks for a filename that contains primary ids. a file of primary ids looks like this: figure 10. input file of primary ids. after the program runs, the updated address in alma looks like this: figure 11. user record that has a preferred address with both the type “work” and “home.” here is the format of the address part of an alma user record in xml. “home” is added as an “address_type” to the preferred address record by the program. figure 12. sample of xml format for an alma user record, address section. once i had updated the whole database of alma with active, non-expired user records, the daily maintenance was as follows: checking the 4am weekday sis alma job update for the new and updated user ids and run the program on a list of those ids. an alma analytics scheduled report [11] emailed every day at 6am with records missing a preferred, home address. if there were any in this list, i would run the program on those ids as well. both steps are necessary to keep all user records updated. any records updated at 4am would suddenly leave those users unable to request home delivery until the re-update of their records. the analytics query was necessary to catch any changes made by staff during the previous day where a new record without “home” was created, but analytics was last updated at 4pm the previous day so the results were slightly out of date. the updates are not necessarily required each day – near the beginning of each semester when there were lots of user record updates due to new student registrations, we needed to run it every day, but at the end of each semester we only ran it every few days. we have no plans to remove the “home” address type from alma user records that already received this update, even if the home delivery implementation is eventually fixed by ex libris. having multiple address types on alma user records doesn’t seem to affect the record or degrade performance, and it may be difficult to retroactively determine which records had a real “home” address type and those that only had one to make home delivery work. plans for locker delivery although the home delivery service worked well for fall 2020, the time, staff, and material costs for home delivery were expensive at a time budgets are tight, and the home mail delivery was slow due to the impact of covid-19 on postal delivery times. the purchase of an outside library locker could be paid for with cares act funding. this limits the impact on the library’s budget and the patron could get the requested item faster instead of depending on postal mail. after receiving quotes and demonstrations from different locker options, csun chose the vendor luxerone for an outdoor locker [12]. the locker sizes and number of lockers is configurable, and csun ended up with 63 lockers of varying sizes in 3 units total. the locker is controlled by an ipad housed inside one of the locker units, which is connected to the internet and a power supply. luxerone has been in the locker business since 2005, starting with delivery of laundry to lockers, and with apartment and business package delivery since 2014 [13], but is relatively new to the library delivery space. in fall 2020, a team of california state university library staff and luxerone staff developed a webhook integration [14] from the alma user database to the luxerone locker user database. this webhook pulls appropriate user data from the alma database and shares it with the luxerone database to make that same user information accessible at the locker. the webhook determines that a request has been made in the alma system (there are many types of requests) and will move a small part of that user data into the luxerone database. this webhook to only load information about each requesting user is preferable to sharing or downloading the full active alma user database of over 48,000 active user records. webhook configuration in alma is through the integration profiles and looks like this, with a “secret key” to talk between alma and luxerone: figure 13. luxerone webhook configuration in integration profiles. here is part of the request information that is sent from alma to luxerone through the webhook, in json format: figure 14. sample request information sent from alma to luxerone in json format. in addition to the webhook implementation, in order to offer both locker and home delivery at the same time for patrons, a new library has to be created inside of alma with a new circulation desk, as you cannot configure multiple pickup locations for a particular library. for our locker deliveries, we check the items out to the new “24/7 lockers” library and circulation desk, instead of our regular main library circulation desk. this has the added advantage of allowing us to track our locker deliveries separately from our other circulations, and we can customize the alma letters based on whether it is home or locker delivery. another configuration change to make this work is that the “relationships” table will need to be configured to allow circulation between the 24/7 lockers circulation desk and the other circulation desks. changes also need to be made again to request tous to support multiple pickup locations (pickup locations = anywhere), the changes in delivery skin mashnew.css need to be commented out so the pickup location shows on the form, the “generaluselabel” on the form will need to be changed to reflect the locker pickup option and instructions. other configurations needed for the locker include the emails and texts that will remind the patron that there is an item to be picked up. these communications come from the luxerone system, not alma, and need to be configured through luxerone staff instead of having direct access to configure the emails. we set csun’s email reminders at 2 days, 4 days, 5 days, and then on the 6th day, the item will be returned to the shelves for use by other patrons and to free up that locker. the item can be picked up touchlessly if the patron has installed the luxerone app on their phone. when they are near the locker, they can press a button on the app that will open their particular locker so they can retrieve their items. there is also a touchscreen where they can input their access code or scan a qr code, opening the locker. the patron experience for locker delivery will involve choosing a pickup location at the request form (see figure 15). figure 15. pickup location 24/7 lockers is chosen. figure 16. close-up of drop-down for pickup location changes in staff workflow are that the items for locker delivery show up in the request queue with “pickup location = 24/7 lockers”, instead of “pickup location = home delivery” (see figure 17), and that items are checked out at the “24/7 lockers” circulation desk instead of the main circulation desks. figure 17. a request for locker delivery. future plans as of this writing (january 2021), csun plans to be back on campus in fall 2021 and the library may physically open a few months before that. however, we still think there will be interest in the home delivery service, especially for our faculty & staff that are continuing to work remotely. our lockers will exist permanently outside of our library and we predict the 24/7 availability of them will be popular with our patrons as they can pick up their items at times that the library is closed. we currently only plan to put our own library and interlibrary loan items in there for pickup, but we may eventually configure the locker to allow faculty, staff, or couriers to leave items for us to pick up from the lockers. the script for updating addresses in alma user records works well for a site that has someone assigned to update the records each day, but future plans would be to automate these updates each day, possibly through a webhook. also, for libraries that don’t want to do anything with code, i want to explore making a cloud app [15] that other libraries can download into their alma to do these same tasks. ex libris is aware of the limitations of home and locker delivery in their system and we hope to see updates to their software over the next year that address the limitations of requiring a “home” address and of requiring a new library for the locker checkouts. bibliography [1] about csun [internet]. csun office of institutional research; [cited 15 december 2020]. available from: https://www.csun.edu/about-csun. [2] library statistics, 2018-2019 [internet]. csun oviatt library; [cited 14 december 2020]. available from: https://library.csun.edu/about/librarystatistics. [3] commuting behavior and transportation preferences of the csun community [internet]. [march 27, 2019]. [cited 15 december 2020]. available from: https://www.csun.edu/sites/default/files/commutingreport_2018_final_0.pdf [4] requests in alma [internet]. ex libris; [cited 14 december 2020]. available from: https://knowledge.exlibrisgroup.com/alma/product_documentation/010alma_online_help_(english)/030fulfillment/010introduction_to_fulfillment/requests_in_alma [5] realm project: happening now [internet]. oclc; [cited 15 december 2020]. available from: https://www.oclc.org/realm/happening-now.html [6] discovery interface display logic [internet]. ex libris; [cited 15 december 2020]. available from: https://knowledge.exlibrisgroup.com/alma/product_documentation/010alma_online_help_(english)/030fulfillment/080configuring_fulfillment/100discovery_interface_display_logic [7] branding the delivery tabs [internet]. ex libris; [cited 15 december 2020]. available from: https://knowledge.exlibrisgroup.com/alma/product_documentation/010alma_online_help_(english)/060alma-primo_integration/060configuring_alma_delivery_system/090branding_the_delivery_tabs [8] supporting personal delivery for requests [internet]. ex libris; [cited 15 december 2020]. available from: https://knowledge.exlibrisgroup.com/alma/product_documentation/010alma_online_help_(english)/030fulfillment/020circulation_desk_operations/020creating_a_request_from_the_institution#supporting_personal_delivery_for_requests [9] home and office delivery [internet]. [updated 11 feb 2014] ex libris; [cited 15 december 2020]. available from: https://knowledge.exlibrisgroup.com/alma/training/03_what%27s_new_videos_2014/11_february_2014_release/home_and_office_delivery [10] users and fulfillment [internet]. ex libris developer network; [cited 16 december 2020]. available from: https://developers.exlibrisgroup.com/alma/apis/users/ [11] scheduling and subscribing to alma analytics reports [internet]. ex libris; [cited 16 december 2020]. available from: https://knowledge.exlibrisgroup.com/alma/product_documentation/010alma_online_help_(english)/080analytics/040scheduling [12] luxer lockers [internet]. luxer one; [cited 15 december 2020]. available from: https://www.luxerone.com/product/luxer-lockers/ [13] interview with arik levy: the entrepreneurial journey to luxer one [internet]. [updated 9 august 2018]. luxer one; [cited 15 december 2020]. available from: https://blog.luxerone.com/interview-with-arik-levy-the-entrepreneurial-journey-to-luxer-one [14] webhooks [internet]. ex libris developer network; [cited 15 december 2020]. available from: https://developers.exlibrisgroup.com/alma/integrations/webhooks/ [15] ex libris cloud apps [internet]. ex libris developer network; [cited 16 december 2020]. available from: https://developers.exlibrisgroup.com/cloudapps/. about the author christina l. hennessey has worked in libraries for twenty years in various capacities and has been the systems librarian at california state university, northridge (csun) since 2018. before becoming a librarian, she worked in the software industry for 10 years as a programmer, and is happy that she still gets to write code on a daily basis as a librarian to solve library systems problems. subscribe to comments: for this article | for all articles 2 responses to "customizing alma and primo for home & locker delivery" please leave a response below: eric m, 2021-10-25 thanks for this very useful article, christina. in figure 3, which label did you use for the “delivery note”? we are testing your solution in our system now but can’t figure out which label to use. thanks for any info you can share. anji mertens, 2023-01-19 hi i was wondering if you have found a way to configure the request form customization in primove? i can’t seem to find comparable functions to the ‘skin’ feature in ve. does this have to be done via css now? thanks! leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – communicat: the next generation catalog that almost was… mission editorial committee process and structure code4lib issue 1, 2007-12-17 communicat: the next generation catalog that almost was… georgia tech libraries broke ground and made considerable headway on “the communicat”: a content management system designed to improve access to the library’s collections (including records from the catalog, institutional repository and other sources) as well as allow individuals and groups to create their own localized libraries (similar to a social bookmarking service), that in turn helps build and grow the main collection. by ross singer as the pursuit of the next generation catalog (ngc) gains momentum, librarians and libraries are frequently looking outside of their traditional integrated library systems to achieve the kinds of functionality needed. whether they be alternate search and discovery interfaces, such as endeca, the university of virginia’s blacklight or ex libris’ primo; social software applications such as penntags or sopac; or combinations of the two such as scriblio and vufind, it becomes apparent that most libraries have a different definition of “next generation” and that few believe that “next generation catalog” has to be a component of the historical backbone of the modern library: the opac. georgia tech’s library chose to take on many web 2.0 concepts in its vision of the ngc: tagging, annotations, recommendations, faceted search, etc. the main focus of the project was to allow users and groups to collect their own personal libraries and use that community-based collection as the source of the central catalog: the communicat. as users aggregate, annotate and share the resources they find and use, this in turn would expand and inform the larger georgia tech collection. the library and librarians would continue to add value, enhancing the metadata and content around the resources gathered that fit the scope of the institute’s mission. envisioning a communicat roughly two years ago, the library formed a committee to work on this project. it was an informal group, with membership from systems, reference, cataloging and digital initiatives, and had no clear objective or mandate. the main goal was to brainstorm ideas of what an ideal “catalog” would look like. a recurring theme was the rigidity of the incumbent catalog (ex libris’ voyager) and the absence of any semblance of community control or input to what the library defined as its “collection”. the integrated library system had no capacity to evolve as new sources of information appeared about existing items or to build connections between items in the collection or with objects on the web. while the group did not actually produce anything substantive, it did manage to document many concepts and desires collectively via whiteboards and helped set the priority for how to proceed with such a project. through the brainstorming, the essential services and their project names were conceived. the major points uncovered were: the “catalog” does not begin to address the free web in any serious capacity. the de-emphasis of traditional cataloging (and subsequent de-emphasis of the cataloging department) requires a decentralized approach to metadata aggregation, composition, and maintenance. relationships between items can be inferred by how they are used. if two resources are used for a particular project, there is a much stronger chance they have something in common. explicit relationships between items should be set when appropriate: a movie version of this book; the soundtrack to this movie; the following proceedings appear in this conference. there was no intention to merge these records, but merely to make the connections between them clear. the scope of the “collection” is strongly dependent on the context in which the user is searching. if the user is a physics graduate student, weighting the resources that are defined as physics or sciences over, say, sociology would be desirable. there are times that searches would start “context-neutral”, but the ability to focus on particular subject areas, especially as the collection grows, could offset large result sets. as these ideas began to take shape, some requirements for what such a system would need to be able to do became clearer. versioning was essential; if authorship of metadata were to be distributed, a mechanism to roll back to a previous, stable snapshot would be necessary. access control would also be important. for the integrity of consistency and discoverability in the “main” collection, some elements (such as the original marc, dublin core or onix record metadata) would need to be controlled, but other aspects of the record would still be editable. there also needed to be a separation between the machine-readable and highly structured elements and the public display interface. while the marc records, et. al., would help define the display record, the public display would not merely be a literal representation of marc (or onix or any other standard). the metadata records could then be preserved, without danger of being adulterated; yet the discoverability and readability of the resources could be altered as data and needs surrounding an asset changed. such a project also required a sense of users, groups and permissions. not only would groups (or people) likely prefer that non-members not be able to manipulate the groups’ items and content, that sort of control would probably be necessary within the groups themselves as well, especially larger groups. this necessitated the concept of role on top of user and group. the downside of such granularity is that it generally makes interfaces to maintain resources more complicated, but the alternative, in the committee’s opinion, was too limiting. the brainstorming work also provided a handful of desired services that could be broken out into individual projects that, together, would make up the communicat. first, there needed to be a means to aggregate references to resources to add to the collection. social bookmarking services, such as del.icio.us and especially connotea and cite-u-like, were looked at as models to draw from. given the need for more robust metadata, however, it was decided that utilizing the openurl link resolver to gather data from appropriate sources would be easiest way to meet this requirement. obviously, a system to store, index and manage these resources was also needed. it had to be able to deal with a variety of metadata schemes and flexible enough to accommodate data that it might not understand or be able to use. this data store also needed to be accessed through several different interfaces and act upon or display its contents accordingly. this repository was the largest and most important decision, since it influenced much of the direction of the project. the other major service the group envisioned was the means to be able to organize one’s resources into various lists or categories. this would open the door for faculty to create their own reserves lists; librarians to create subject and course guides; and users to be able to make web-accessible bibliographies and reference lists. a possible outcome of this service was a concept the committee called research trails in which, as a user compiled and annotated resources for a given assignment or project or article, he or she created a bucket of information of all the resources discovered while researching their topic. while only a subset of these items would appear in the bibliography, the resources that led to the discovery of the actual materials cited would still be associated with the final product. the researcher in essence would, through the course of their work, make associations between our resources for us. the goal was for any service to be independent of any other. the library could enable the personal library space without the community collection piece or the link resolver or any combination thereof. if the repository became too demanding to implement, it could be replaced with a simpler incarnation that met the revised needs of the users. putting the vision into place prioritizing the pieces of a project like this was non-trivial. there was only one developer, me, that could work on this project. it was difficult to commit significant resources towards it given its highly experimental nature, size, complexity and, frankly, its potential for failure. to devote time and people to a project that ultimately might not work or be largely ignored by our users runs counter to the traditional risk-averse culture of libraries. also, like so many other libraries, georgia tech has a backlog of needs that also must be addressed, further straining the commitment that the committee could make on the communicat. with these conditions in mind, it was decided that it would make the most sense to begin with a piece of the project that solved other library needs outside of the requirements of the communicat. the link resolver enhancements, known internally as the ã¼beresolver, were targeted as a good starting point, since they also addressed problems we had resolving print materials and, possibly even more importantly, solved a major issue with conference proceedings. if this service was successful and obvious benefits could be seen in its development, then its success could be used to garner the political capital needed to prioritize development of the rest of the project. putting the 㜠in openurl the link resolver seemed the logical place for users to enter their data into the communicat. the preferred variety of resources submitted to the system were scholarly (at least for the sake of the larger goals of the project), and theoretically, the openurl link resolver (at georgia tech, ex libris’ sfx) should be a nexus for citations as the researcher searches in the library’s licensed databases. quite a bit of information can be gathered from the resolving process: where the user came from, what the user wants (eliminating the need to enter detailed metadata by hand or via screen scraping) and where they went. in practice, however, openurls’ metadata can be rather spotty, so one of the goals of this service was to improve that as well as enable discovery of open access materials in repositories on the web. what tech developed was the ãœmlaut (the name ã¼beresolver was deemed too self-aggrandizing): an openurl middleware layer written in ruby on rails. the goal of the ãœmlaut was not to replace sfx, since it handled the task of linking to full text resources quite well, but to enhance it, accessing sfx via its xml interface. the ãœmlaut queries crossref, pubmed, georgia tech’s catalog, the state union catalog, amazon, yahoo, google and a host of social bookmarking sites for each incoming openurl to gather the most complete metadata snapshot of a given citation. through google’s and yahoo’s web search apis, it also analyzes the links to determine if any point to free manifestations in open access archives and present its findings along with the vended full text links. figure 1. diagram of ãœmlaut workflow from the catalogs, it pulls mods records (if the item was found), which offer much richer metadata than what is generally available in the openurl context object. by leveraging oclc’s xisbn service, other manifestations of a particular work could also be found rather than relying on the exact isbn sent in the openurl request. more sophisticated searches can be performed for different objects, further increasing the success rate for items such as conference proceedings. since the ãœmlaut was to have fit into the broader context of the communicat, machine-readable xml and json interfaces were created and exposed via unapi. this allowed the ãœmlaut’s services to be consumed by the opac or other applications. when the time came to develop the bookmarking service, resources could be enriched in the background without user intervention. development of the application proceeded quickly, going from proof-of-concept to production service in fewer than five months. outside of the expected fits and starts of launching a new, heavily used resource, the ãœmlaut was generally considered a success and the value it added over sfx offset its somewhat sluggish performance. the few months after its august 2006 launch were largely consumed with enhancements and bug fixes, and by january of the next year, planning had already begun on the next version, dubbed ãœ2. the successful implementation and subsequent national attention of the ãœmlaut (it was the winning entry in the 2006 oclc software contest) gave us the credibility with the rest of the library needed to proceed with the next phase of the project. in search of a repository now that we had a means to collect metadata, it seemed appropriate to find a place to store it. the challenge was to locate an existing system that was both extensible, versatile and, ultimately, replaceable if the needs or direction of the communicat were to shift dramatically. as a last resort, the metadata store could be built from scratch, but we felt this was less desirable given the time it would take to design and develop and the fact that it would simply be easier to modify an established project and possibly alter our expectations than make something new. early in the process, before the more inspired goals from the whiteboard were drafted, the library considered using unalog, a python-based social bookmarking engine based on the quixote web framework. while unalog’s current implementation was rather modest in scope, dan chudnov, the project’s creator, had a desire to redesign its data model to utilize more detailed metadata, such as mods. while appealing, the scalability of such an endeavor was unclear and, ultimately, the necessary refactoring never was made. the nascent design ideas through this point had an assumption that the data would be stored as mods, since it was more detailed than dublin core, but considered easier to work with than marc. the evergreen ils was also briefly considered for the data store; tech was interested in it as an ils and it used mods internally as its data model. the shortcomings of evergreen (lack of acquisitions module or serials support) were a non-factor, since it was only going to be used as a bibliographic database. it, too, was eventually considered inappropriate for the expanding requirements of the project and was abandoned. through continued discussion, the expectation of using mods was dropped and focus turned to more flexible repository frameworks. georgia tech’s digital initiatives department had considerable experience with mit’s dspace, but its architecture was deemed too monolithic to apply to all of our use cases. fedora looked more attractive, as did the jsr-170 based repositories, such as apache’s jackrabbit. either of these solutions would have required a considerable amount of development time to implement. in the end, we finally settled on outerthought’s daisy content management system. while a cms may seem an odd choice at first for such an application, daisy has a unique design that suits itself well for alternative uses outside of traditional web page publishing. it actually is two separate cocoon-based services: a backend xml document repository using mysql and an optional front-end wiki interface. the document repository can be used entirely on its own and has java, javascript and http apis to access and manage its collection. the documents themselves are defined by schemas which are made up of various internally defined types: parts, where content is defined and stored, and fields, which are basically triples defined as strings, booleans, dates, long integers, etc. these are combined with document metadata, such as title and identifier to define the schema. schemas can be changed after data has been entered using them, but fields cannot be deleted if any document is using them (although they can be marked deprecated). daisy maintains every version of a document, making it possible to roll back to a previous iteration. it also has the concept of branching a document, which is intended for having multiple editable versions of the same document if, say, the content of a document needs to be different based on the context in which it is being viewed. coupled with yet another axis that daisy documents can branch on, language, this can make for quite a complicated model. for our purposes, however, simple versioning would be sufficient. pushing daisy now that we had identified our repository backend, it was time to build a framework to interact with it and model our data to make it work effectively within said framework. since we wanted a layer of abstraction between the communicat and its underlying data, we created another ruby on rails application named cortex to interact with daisy’s restful http api and manage the documents for indexing and display. cortex was also tasked with manipulating marc data to enter into daisy, harvesting and ingesting dublin core records from oai-pmh servers as well as normalizing and storing data from other sources (i.e. onix from the publishers). this made for a rather complicated metadata model. the daisy document schemas were divided into two groups: component documents, which stored the original metadata records; and page documents, containing the publicly viewed data as well as the association of the viewable record to its components. page documents also defined the relationships between themselves and other page documents. component documents held the original metadata, whether they were marc21, dublin core, onix, openurl or whatever was available. for data for which no component schema had been created, there was a genericcomponent so the resource could be preserved until it could be identified. to aid with indexing and retrieval, important metadata – such as author, subject, issn/isbn etc. – was stored in fields in the component document. the page document held more high-level data about the resource, such as what its item type might be (serial, book, etc.) or whether or not it was a conference or a government publication. all pages stored the title, this being a requirement of daisy. the relationship to the component documents was kept in a repeatable link field. in many ways, the page documents were very similar in philosophy to rdf, making connections between documents through uris. this architecture meant that for each item in the catalog, there had to be at least two daisy documents. phase ii would require at least one more document per record as authors were to be split from the item records as identity records. communicat user records would also be identity documents, so users could be associated with any works they may have created. figure 2: schematic of daisy table design. the relevant documents surrounding a particular resource (a public page document, its requisite component documents and identity records) were merged and indexed in solr. the individual components were also indexed in solr, so besides powering the main catalog search, solr could also be used for administrative and maintenance tasks. managing the relationship between solr documents and daisy documents was another responsibility of cortex. a fourth planned document type was the bookmarkpage, essentially a page to a page document. these documents would be associated with users (identities) and contain annotations, tags and comments. through bookmarkpages, users would be able to organize and share the item level page documents. public bookmark annotations would be searchable alongside the standard bibliographic data, allowing dialog about resources to aid in their discoverability. in an effort to allow users to control the scope of their searches, communicat documents are assigned one of three realms: core, community or world. the realms break down rather simplistically. core resources are purchased or created by georgia tech. these would be items in the catalog; in dspace, the institutional repository; databases; journals; or other sources across campus that serve the university’s mission. community resources are those added by users actively enrolled or employed by georgia tech. this way a user’s search could include materials that their peers are vetting and commenting upon. lastly, documents in the world realm are anything created by users outside the georgia institute of technology. searching these resources is optional, since they may or may not have any relevance to tech users. it was determined early that for maximum effectiveness, gather (as the social bookmarking/citation management service was called internally) would need to be open to people outside of georgia tech. researchers frequently work with colleagues from other institutions, making this a necessity. this philosophy of openness also prevailed in the institution’s sakai implementation, with each site receiving 50 guest logins for membership outside campus. after a bit of development, some limitations in daisy became apparent. daisy had never been intended to scale to the numbers of documents we were proposing (which would be in the hundreds of thousands or much more, depending on popularity). the first flaw we came across concerned the retrieval of large numbers of documents. daisy has a very sql-like query language that allows for retrieval of documents by structured queries. however, because of the access control lists, which must be determined post-query, daisy returns all matching documents before sorting them by relevance and applying any query limits. for queries that could produce an extremely large number of documents (such as “select the id for every catalogrecord page document”, which would be required to delete them), the load on jetty, mysql and cocoon overwhelms daisy and the only way to keep all system resources from being consumed is to kill the daisy repository server. an alternative to this method would be necessary since large maintenance tasks would be inevitable, especially during the development process. it also was discovered that there was no capability to create or modify document schemas via the http api. for these sorts of administrative tasks, the daisy wiki would need to be run in order to access the administrator interface. fortunately, this would only be necessary in very specific cases. due to a shift of institutional priorities at georgia tech (largely due to my position being assigned to support the campus sakai initiative and a massive reorganization within the library), development had to be stopped shortly after the capability to parse and insert records into daisy and solr was written. it is unclear whether or not this project will be completed. gather: bringing it all together the last component of the communicat project, gather, was to be the public interface. this was the means that people would add items to their personal or group libraries and it was the way they would search within them. the concept is modeled strongly after social bookmarking sites such as del.icio.us or connotea but with the capability of much richer metadata. it would also serve as a very crude citation manager, but merely for storage and organization of references. for formatted bibliographies, gather would export to applications more suited for that purpose, like endnote, zotero or bibtex. the preferred path to add citations would be through the ãœmlaut, which would have a link to the user’s preferred bookmarking service. if it is a service other than gather, the user will have the choice of adding it to both places, including a syndication feed in gather to harvest their public links or bypass gather altogether. we felt it important that the functionality did not interfere with the users’ normal workflows, so efforts would be made to allow them to retain the status quo. since it is likely that users will desire to add resources that would not fall in the typical openurl resolving chain, bookmarklets or browser extensions would need to be generated to jumpstart the process. it would also be desirable for gather to work with the libx and zotero extensions. as there was no desire to maintain a suite of screenscapers to pull metadata from the html pages, piggybacking on the screenscrapers used by cite-u-like, connotea, or zotero would lessen the burden, as would coins (context objects in spans) and unapi support. users would be able to organize their bookmarks any way they wish. they may annotate them, share them and open them for comments. since users are members of groups, they also can comment on other’s entries. the bookmarks are associated with groups, it becomes possible to weight the solr queries: resources saved by a group, or similar groups, would be ranked more highly than other materials. as such, the groups become a focused portal into the larger community collection. the future of communicat sadly, with my departure from georgia tech, it is unlikely that the library will complete this project. there are not the resources to enable such a project. with the proposed launch of vufind in the spring 2008 term, the library will be addressing its next generation catalog needs. now that i am currently employed by talis, it is possible that the communicat could become a talis platform application. since the platform is an rdf triple store, the data model would only need minor tweaking to apply. it was this sort of scenario, the daisy component being replaced, that inspired development of cortex, so architecturally, the design would not need to change. it would most likely have to become much simpler to be marketable from a vendor standpoint, however. observations while several next generation catalog projects attempt to address some or part of the communicat’s goals, none seemed to be quite as ambitious as georgia tech’s vision; perhaps pragmatically so. this was an extremely large project with each individual piece itself a noteworthy endeavor. institutional support was imperative, but the goals of the initiative were difficult to comprehend. this, in turn, made it difficult to prioritize. daisy, despite its limitations, was a strong choice. it would have provided a flexible backend for a host of future services. since it was primarily being used as a data store, however, possible alternatives could be a native rdf database or a document-centric database, such as couchdb. daisy has a very strong user community, though, and its natural capability to strongly define documents and create links between them is functionality that would need to be duplicated. a project such as this is difficult to manage within a committee. when the concept is abstract and the end-result could change dramatically from week to week, an agile process is necessary. many library cultures have a difficult time subscribing to that philosophy. it might be better for a core team of developers and implementers to design a proof-of-concept before a larger committee is formed. the ideals of the communicat, an organic organization of resources, collected by the library’s users and expanded and enhanced by librarians from every department, and not just cataloging are quite revolutionary and require considerable commitment from an organization. the benefits, however, are enormous: a catalog that reflects the actual research performed by those in the constituent community with current and topical added value layered on top by professionals within the library. instead of shoehorning these services into our existing data silos or export the data from our silos verbatim into a third party search and discovery interface, wholesale revision of how data acts and interacts is necessary. georgia tech was unable to see this vision through completely, but perhaps a few of its goals can see their way into other library’s projects. about the author ross singer is interoperability and open standards champion at talis. he may be reached by email at rossfsinger@gmail.com. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – building a library search infrastructure with elasticsearch mission editorial committee process and structure code4lib issue 48, 2020-05-11 building a library search infrastructure with elasticsearch this article discusses our implementation of an elastic cluster to address our search, search administration and indexing needs, how it integrates in our technology infrastructure, and finally takes a close look at the way that we built a reusable, dynamic search engine that powers our digital repository search. we cover the lessons learned with our early implementations and how to address them to lay the groundwork for a scalable, networked search environment that can also be applied to alternative search engines such as solr. by kim pham, fernando reyes, and jeff rynhart introduction the elastic search platform [1], as its name implies, can consist of many different applications. its distributed nature is used to solve a multitude of use cases, including application search, security analytics, metrics and logging. elastic’s most popular application is elasticsearch [2], known for its ability to provide extremely fast full-text search functionality for many of our applications. its searching and analytics functionality relies on the open-source information retrieval software library apache lucene [3]. elastic also supports other applications such as kibana, which serves as the interface to the data stored in elasticsearch. it allows us to add data to the cluster, build visualizations, and test new or existing queries. logstash is a data processing pipeline that allows us to filter and manipulate data. it provides the ability to output the data to various products such as elasticsearch, redis, file systems and databases. beats is one way to retrieve and ship data to logstash and ultimately into the elasticsearch cluster. our elastic stack elasticsearch is a distributed solution that is designed to be scaled horizontally. in other words, we can add elasticsearch instances, called nodes, across multiple machines to handle our searching and analytics workloads [4]. a collection of nodes in the elasticsearch ecosystem is called a cluster. the data in an index is divided and stored across a cluster in shards. a shard in each node contains only some of the data that makes up the overall index. sharding, the process of dividing data among shards, is how elasticsearch is able to distribute the data over a cluster. every elasticsearch index is configured by default to have five shards although the number of shards per node can be changed to meet your specific needs. json documents consisting of multiple fields are the basic units of information stored in elasticsearch. this increases the speed and efficiency of searching, protects data loss and adds redundancy if a server becomes unavailable. [5] for example, our elastic cluster consists of 3 node servers (see figure 1). we find it is best to use three shards for our system because we are working with a relatively small index (less than 1gb, our most complex mapping has about 100 fields). if we configure an index to have 3 shards and 2 replica shards, each node will contain 1 shard, and there will be 2 replica shards of each shard, and each replica shard will be placed on a different node than its shard. the resulting cluster contains 3 nodes, each containing one primary shard and 2 unrelated replica shards. as mentioned, our cluster contains relatively small indices from multiple applications. we generally try to have a separate index for our production and for our test instance. for our repository, we have two production indices (one public and one for internal library use where you can view unpublished metadata) and multiple test indices for development. figure 1. elasticsearch cluster. our cluster implementation consists of three nodes (1 node = 1 elasticsearch instance). each node is a red hat 7 virtual machine. our indexes are configured to make use of three shards and two replicas. logging centralized logging is another benefit of the elastic stack. our elastic cluster is maintained by our it department, and all of our applications that leverage elastic utilize this cluster. this enables our it department to continuously log the elastic requests and responses from our applications, and also store logs that are produced by the applications themselves. aside from this simple logging of requests and responses, our it department can capture the various logs sent out from our applications on a live, automated system, configured to capture application logs on a set schedule. this system utilizes filebeats to continuously check the output of application logging modules, such as log4js. we simply include log4js in our applications and the central elastic stack can be easily configured to connect to each application’s log output. central digital repository index our elasticsearch cluster serves as our central indexer for our digital collections ecosystem. the digital collections ecosystem contains four main systems (see figure 2) – archivesspace (archival description), archivesdirect (preservation microservices and storage), kaltura (media management and streaming server), and our custom digital repository. our digital repository (digitaldu) is comprised of a backend [6] to manage digital objects and a frontend [7] for public access and viewing. figure 2. university of denver digital collections ecosystem, an integrated system architecture used to manage, access, and preserve special collections and archives digital objects. data from these systems are mapped into elasticsearch, where it can be accessed across the entire ecosystem in areas where it’s needed (see figure 3). figure 3. centralized index for repository search. data from various sources is sent to elasticsearch – logstash processes and indexes on fields mapped to elastic fields. for instance, when an audio object is ingested into our digital repository through the digitaldu backend, as the object passes through the ingest workflow elasticsearch is constructing its document record. from digitaldu it indexes viewing information such as its parent collection identifier and access controls, from archivesdirect it uses the archivematica and duracloud apis to index the object’s filepaths and technical information, from kaltura it indexes the derivative identifier, and using a custom archivesspace plugin it indexes archival descriptive metadata (see figure 4). we view the data in elasticsearch as an index to the persistent stores in each various system – it is not the authoritative data source. it is where data from multiple systems is consolidated that can be re-indexed by sending a request to the apis and connectors that are associated with each system. this index powers our digital repository, both the digitaldu backend and digital frontend, including generating iiif manifests for viewing content, and is especially important in developing the search module for both these applications. figure 4. digital collections ecosystem application search elasticsearch is used to integrate search into our applications and has improved and simplified our design and implementation of application search functionality. search is built as a reusable component where the same code can be used across our applications and only requires reconfiguring settings and integrating the search module. elasticsearch has a powerful range of directives and options that enabled us to develop high quality search applications containing modular and scalable code. searches are performed by posting query data to the elasticsearch rest api in a flexible and powerful format; this is one of the characteristics that makes elastic search queries so easy to use and customize. this format consists of directives from the elasticsearch domain specific language (dsl [8]) in a json object structure, which is constructed dynamically using the user submitted search data along with application search configuration settings data. the “elasticity” of the structure of the elasticsearch query object has simplified the development and integration of search functionality into our applications. building the elasticsearch query object when a user performs and submits a search in an application, the data from the search form is used to construct the elasticsearch query object. behind the scenes, a search query object can be dynamically created based on full-text queries [9], such as user-selected search parameters, control characters in the search string, and settings stored in the app’s configuration. these data are used to construct a query object for each search, which is posted to the elasticsearch server to return the matching results (see figure 5). figure 5. flow to construct an elasticsearch query object. the code checks the query string, also if any facets or advanced search parameters exist and their conditions. the query object is sent to the elasticsearch cluster to get back a json response of search results. due to its simple structure, the query object can be constructed and posted to elasticsearch very quickly, permitting a wide range of searches without requiring a specific function or query object for each type of search. from search query to query object a search request includes the field(s) in the index in which are to be searched, a query string containing search terms, search options (such as to target a specific index field or all available fields, or ignore a field), and facet filter data (such as a specific author). this data is inserted into the query object in the required format to define what types of documents will be considered a “match” [10] to the search query. the match query type is a standard, full-text query type that allows for fuzzy or proximity matching. the query object may contain different “subqueries” that work together. for example, if a user opts to search in “all fields,” a subquery will be created for each available search field (as set in the app configuration). they will be placed in the dsl “should” array in the query object. this means if one of the queries is a match on a document, the document will be a match for the search so a keyword match in just one field will be sufficient. if the user has selected a filter for the author “john smith,” a query will be created to match “john smith,” and will be placed in the dsl “filter” array; so then, one term match must occur in the specified fields, and the filter query must match as well. note: the query object can be dynamically created by a programming language, such as javascript, python, or ruby. the following snippets are based off of javascript code from our digital repository. // a dsl subquery as it appears in the elastic query object. // the “query type” here is “match” { “match”: { “title”: “great expectations” } } typically, each query will use the dsl “match” directive. this will return results that match terms in the tokenized query string. this does not have to be an exact match on the string, but document fields that contain more matching terms from the querystring are given a higher score in the results. facet queries will use the “match_phrase” directive [11], meaning they must be an exact match in the document. different directives may be used if the querystring contains certain control characters, to request a specific type of query for the terms. control characters in the querystring: quotation marks words grouped by quotation marks must appear in a document in the exact order to result in a search hit. other combinations of the words will not result in a hit. if(query.includes(‘“‘) { querytype = "match_phrase"; } for example, a query string may contain quotation marks that group some or all of the search terms together, or a wildcard character. each query string is parsed, the control characters or terms are detected, and the appropriate directives are added to the query data object. control characters in the querystring: wildcard character the presence of a wildcard character in the string will create an elastic “wildcard” query, which will return results that contain any characters in place of the wildcard character. this dynamic creation of the elastic query based on characters and terms present in the query string allows the utilization of common query based search techniques by the user. if( searchterms.indexof('*') >= 0 ) { querytype = "wildcard"; } additional search parameters: array of fields – search all search parameters set within the search form are another way the user can customize a search. a user may specify a single field to search, or may select a group of fields, such as “search all” (see figure 6). available groups of search fields are defined in the app configuration; so if a specific group is specified in the search, the fields are retrieved from the configuration and added to the elastic query object, specified using the “multi_match” query [12]. any field that contains the search terms will generate a hit. individual fields can also be targeted in the search using the “fields” query, such as in “search title,” where a field called “title” will be examined but the others ignored. the multi_match or multiple field search is the same as a regular search except that you can search in multiple fields (the same as regular search except that you can combine queries). figure 6. the digitaldu-frontend implementation of search all. if the user selects “search all fields,” an array of fields is passed in and all of these fields are added to the query as a “multi_match” query: for(var field of selectedsearchfields) { let query = {}, temp = {}; query[ field ] = query; temp[ “multi_match” ] = query; // will be added to the booleanquery matchfields.push( temp ); } additional search parameters: single field – search single value if the user wants to search in a particular field, it will not use “multi_match” and it will just search in the specified field. in our system, we provide the user the option to search in specific fields, such as “title,” “creator,” or “description” (see figure 7): let query = {}, tempobj = {}; query[ singlesearchfield ] = query; temp[ “match” ] = query; // will be added to the booleanquery matchfields.push( temp ); figure 7. the digitaldu-frontend implementation of search by a single field. add restrictions to the query data elastic’s “must not” [13] query is used to restrict results. for example we don’t want our collection type objects to be returned: // add the restricted query restrictions.push({ "match": { "object_type": "collection" } }); facets filter the results if facet fields are included in the request, they are combined with the query string search with an elastic “must” query, so the results must contain the facet data and match the facet query exactly. facet queries can be executed with or without an accompanying query string. the elastic query object is built to either apply the facet fields to the main query to limit its results (query string search limited by the facet query), or to search the index using the facet data with no search query string (facet term query not limited by a query string). facet searches always use the “match_phrase” query in a boolean and relation to the queries already present in the current search. only an exact match on the facet string, combined with any existing queries, will result in a hit. this allows faceting to limit the search results. add facets to the query data if facets are selected, loop through the facets by name and retrieve the requested facet value to add to the search. for example, the facet name is “creator” and the value to limit the results to is “john smith.” the facet object is built by looking up the facet field in the configuration by -facet name. an example facet name might be “creator” and an example facet field in the index might be “names.namepart.” a facet query is added for each facet field included in the search request. searchfacets = { "creator": "john smith" } configuration.facets = { "creator": "name.namepart" } // create facet query if( searchfacets ) { for(var facetname of searchfacets) { facetvalue = searchfacets[ facetname ] facetfield = configuration.facets[ facetname ]; // query => {“names.namepart” : “john smith”} query[ facetfield ] = facetvalue; facetquery.push({ "match_phrase": query }); } } query object is nested in the boolean query to return search results, documents are brought back from the index that matches a query. a query is typically broken down into sub-queries that contain a set of built-in conditions. the conditions contained within the query object wrapper are called the boolean query object [14] (see figure 8). this object consists of a set of arrays, each containing a set of individual queries. each individual array imposes a specific condition on the queries within to determine if they will match a document: “should” array: if one query in this array matches a document, the array will be interpreted as “true” by the bool query. “must” array: if all of the queries in this array match a document, the array will be interpreted as “true” by the bool query. otherwise, it is false. “must_not” array: if none of the queries in this array match a document, the array will be interpreted as “true” by the bool query. “filter” array: all queries must match the document. the difference between this array and the “must” array is that the documents that match in that array are given weights to assign priority within the search result list. filter does not weight the results; it simply blacklists any results that do not match all of the queries within the array. this is useful for limiting results with facet queries. if all of the above arrays are interpreted as true for a document, the document is a match and is returned as a search result. queryobj = { "bool": { "should": booleanquery, "filter": facetquery, "must_not": restrictions } } three subquery arrays are available to be added to the main query object. each array can contain multiple data objects to control the main query. should: the booleanquery array contains the field/keyword queries. if a search field has been specified, this will contain one field. for a search of all available fields, the array will contain one data object for each field. any object with at least one matching field will result in a hit. filter: the facetquery array contains the currently selected facet search fields. an object must match all facet queries here to result in a hit. must not: the restrictions array contains data to omit from search results, such as objects that are part of a compound object. any object that matches any query in this array will not result in a hit. figure 8. the elastic boolean query serves as the wrapper for the main query object. advanced search in our advanced search (see figure 9), the user can add multiple queries combined by boolean logic (and, or, not) and can specify contains or not contains for each search field (such as title) or group of fields (see figure 10). this utilizes elastic’s native boolean query must/must not. in addition to this, the user can target a wider range of search fields to fine tune the search. it is possible to add additional search parameters as well, such as contains, does not contain. figure 9. the digitaldu-frontend advanced search interface. to construct the advanced search query, another boolean query object is assigned to the main query object “should” field, replacing the array of queries that is used in a simple search. the “must_not” and “filter” arrays of the main query object contain the same data as in a simple search. if there are multiple queries present, these queries are added to the new bool object per the user’s selection of boolean logic for their combination. the bool object has 3 arrays or “buckets” that hold the individual queries and define their boolean relationships: “should” (or combination), “must” (and combination), and “must_not” (not combination). the first query in an advanced search is always assigned to the “must” array. for subsequent queries, any query inserted as an or will be assigned to the “should” array, queries inserted as and will be assigned to the “must” array, and queries inserted as not will be added to the “must_not” array. therefore, an advanced search query object will contain the fields “must_not” (the restrictions), and “filter” (the facets), and “should”, which contains another nested bool object with the “should,” “must,” and “must_not” buckets containing the queries included in the advanced search. figure 10. advanced search, still wrapped inside the boolean query object but itself is made up of boolean query objects. to construct the advanced search, multiple queries are created and added to the should, must not, and must array. where the user selects ‘and’ as the operator for search fields, the field gets pushed to the must array. if(bool == "and") { booleanquery.bool.must.push(boolobj); } where the user selects not as the operator for search fields, the field gets pushed to the must not array. else if(bool == "not") { booleanquery.bool.must_not.push(boolobj); } where the user selects ‘should’ as the operator for search fields, the field gets pushed to the must array. else { booleanquery.bool.should.push(boolobj); } adding the query object to the elastic data object the data object contains index and type settings, the query object, fields in the index to aggregate in the search, search results size and pagination data. this object is converted to json and posted to the elastic server. /* build the data object to post to the elastic server: get parameters from the configuration and add the query object add facet aggregations to the search, such as title, creator, etc. */ var data = { index: config.elasticsearchindex, type: config.searchindexname, body: { from : (pagenum 1) * config.maxresultsperpage, size : config.maxresultsperpage, query: queryobj, aggregations: facetaggregations } } these are some examples of how the elastic query object’s versatility permits a relatively simple search module to be created to leverage the power of elastic; as controlled by application search settings, selected search parameters, and control characters within the user’s query. future directions as a tool for logging, as a core component to integrate the various systems in our digital collections ecosystem, and as a powerful search engine, we are able to use elasticsearch in multiple ways that proves the platform’s flexibility. most recently we developed a repository api using elasticsearch which is currently in beta, but provides programmatic access to data based on rest queries. internally we are discussing other potential projects which include improving our central indexer to collect and update data at a more frequent timecycle, application performance monitoring, which includes generating periodic reports and notifications about applications using log data, consolidating our elasticsearch instances into a single cluster, and improving our application search by incorporating some of elasticsearch’s built-in capabilities like autosuggestion, predictive text, highlighting and fuzzy searching. references [1] elastic stack overview, introduction https://www.elastic.co/guide/en/elastic-stack-overview/current/introduction.html [2] elasticsearch reference, elasticsearch introduction https://www.elastic.co/guide/en/elasticsearch/reference/current/elasticsearch-intro.html [3] apache lucene core, https://lucene.apache.org/core/index.html [4] pranav, shukla, sharath kumar. learning elastic stack 6.0, seminar presented at the elasticsearch engineer i training, december 2017 [5] comparing elasticsearch with solr – https://thishosting.rocks/comparing-elasticsearch-with-solr/ [6] digitaldu-backend github repository. https://github.com/dulibrarytech/digitaldu-backend [7] digitaldu-frontend github repository. https://github.com/dulibrarytech/digitaldu-frontend [8] elasticsearch reference, query dsl. https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl.html [9] elasticsearch reference, full text queries. https://www.elastic.co/guide/en/elasticsearch/reference/current/full-text-queries.html [10] elasticsearch reference, match query. https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-match-query.html [11] elasticsearch reference, match phrase query. https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-match-query-phrase.html [12] elasticsearch reference, multi-match query. https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-multi-match-query.html [13] [14] elasticsearch reference, boolean query. https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-bool-query.html about the authors jeff rynhart is a software developer at the university of denver, specializing in full-stack node.js applications. he is the lead developer and architect of the frontend discovery application for the library’s new digital repository. he is also a part-time web developer. kim pham is the information technologies librarian at the university of denver. she is the coordinator for the library technology services department and oversees the development of the digital repository ecosystem. on twitter she’s @tolloid. fernando reyes is a full-stack software developer at the university of denver. he is the lead developer and architect of the backend application for the digital repository and the archival system integrations for the digital collections ecosystem. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – hacking 360 link: a hybrid approach mission editorial committee process and structure code4lib issue 18, 2012-10-03 hacking 360 link: a hybrid approach when the university of victoria libraries switched from a locally-hosted link resolver (sfx) to a vendor-hosted link resolver (360link), new strategies were required to effectively integrate the vendor-hosted link resolver with the libraries’ other systems and services. custom javascript is used to add links to the 360link page; these links then point at local php code running on uvic servers, which can then redirect to appropriate local service or display a form directly. an open source php openurl parser class is announced. consideration is given to the importance of maintaining open protocols and standards in the transition to vendor-hosted services. by john durno introduction in the fall of 2011, the university of victoria (uvic) libraries opted to transition from ex libris’ sfx link resolver to serials solutions’ 360 link. this did not reflect dissatisfaction with ex libris’ technology, but rather the need to consolidate knowledgebase maintenance in our technical services unit following the implementation of the summon discovery service in 2010. because sfx was locally hosted and built with a modular architecture, it had been relatively easy to implement a number of customizations to better integrate it with our voyager ils (integrated library system), relais ill (inter-library loan) management system, and various other service and support workflows. it was deemed desirable to replicate many of these customizations in 360 link. 360 link provides more limited support for functional enhancements than does sfx, at least in its native interface. while arguably it can be made to do most of what we wanted out of the box, in fact its built-in abilities to integrate with external systems are more limited than what we had been used to with a locally hosted system. we knew from experience that optimizing the linking behaviour of the system would necessitate having the ability to iteratively tweak its behaviours in ways that were simply not possible if we did not have direct access to the code. 360 link does provide a fairly rich api (talsky, 2008), which would make it possible to considerably extend its feature set, assuming a willingness to build and/or support a locally hosted interface [1]. that approach was considered but eventually rejected, primarily because of the high degree of interdependence between summon and our link resolver. at the time, summon was generating more traffic for our link resolver than all our other research databases combined. it therefore made sense to host them on the same infrastructure. hosting the interface locally would have meant living with an additional and arguably unnecessary point of failure in the technical architecture of one of our core services. furthermore, we realized that because of the increased traffic generated by summon, the relative importance of our link resolver as a service delivery point had escalated. in our previous environment, a link resolver outage would have inconvenienced our users, but because they would have been using native database interfaces they would still have been able to access most online content directly. if, as seemed likely, use of native search interfaces were to decline following our summon deployment, a link resolver outage would in fact be much more problematic, because it would render our major discovery service all but useless. consequently, our existing single-server link resolver implementation was not adequate to ensure continuity of service now that link resolving had been promoted to ‘essential’ status. this constituted a supporting argument for not hosting our own interface. while building and maintaining a robust, redundant infrastructure was not beyond our capability, it would involve more effort and expense than rolling out a single server. and finally, there was the argument that application hosting was one of the services we were paying for. simply put, it seemed reasonable to expect that moving to a vendor-hosted service would reduce our local hosting requirements. having rejected a locally hosted interface as a solution, we then investigated a hybrid approach whereby the core functionality would be delivered from the hosted interface, but the supporting functionality would be delivered by a set of modules developed and hosted by the library. “core functionality” was defined as linking out to full text online content; “supporting functionality” was defined as linking out to other library resources and services such as the catalogue, ill management system, and so forth, which would typically only be of interest to users when no online content was found. because the availability of these supporting services would not affect the operation of the discovery service, it was deemed acceptable to host them on local infrastructure. and because traffic for these services was expected to be relatively light, we could host them on an existing server. a quick demo before launching into a detailed description of our implementation, it may be useful to view the following screencast demo of the user-facing aspect of our link resolver enhancements. the user interface of our production system is also publicly viewable, with the caveat that over time it will likely evolve away from the system described in this article as features are added and dropped in response to changes in our technical environment. (video screencast hosted at http://archive.org/details/screencastforhacking360link) passing metadata the key to the hybrid approach lay in the ability to pass metadata via openurl from 360 link to several locally hosted modules. fortunately, link resolvers are by definition highly interoperable with external systems, and they are that way by means of an niso standard (openurl, z39.88-2004) for transmitting metadata over http. a hybrid, loosely-coupled approach appeared feasible largely thanks to the existence of openurl, which defines both a finite set of possible metadata elements and the means of their transmission (niso, 2004). in principle, passing metadata to our local modules should have been straightforward. 360 link receives and parses openurl metadata as part of its normal operation, and it has the ability to link out to external targets. however, it turned out that 360 link does not have the innate ability to pass the openurl metadata it receives directly to external systems. its outbound links are based on templates which, although customizable, could only be counted on to pass a subset of the information contained in the original openurl to the external target, since it is not realistically possible to define templates sophisticated enough to handle all of the permutations of metadata that could theoretically be found in an openurl. consequently it was necessary to develop a workaround, since we wanted to ensure all the available metadata was available to the library-hosted modules. javascript is of course the tool of choice when it comes to making web interfaces do things their creators never intended. like many other hosted applications, 360 link can accommodate custom html in its header and footer blocks, which can in turn include externally hosted javascript code. this enabled us to create and embed a simple function that extracts the openurl query string from within the browser itself, appends it to the url pointing to the service requested by the user, and opens the resulting output in a popup window when the user clicks the appropriate link. for example, if the user wanted to check the library catalogue, they would click on the following link: <a href="javascript:popitup('catalogue')">check the uvic libraries catalogue</a> this specifies the service they wish to access by sending the ‘catalogue’ parameter to the javascript function below. the function then uses the built-in javascript location.search.substring function to obtain the openurl string from the users’ browser, constructs the appropriate link, and forwards them to their destination. function popitup(service) { /* location of our locally hosted modules */ var url = 'http://library.uvic.ca/extfiles/360link'; /* get the openurl string from the user's browser */ var qstring = location.search.substring(1); /* default dimensions for the popup window */ var height = 800; var width = 800; /* determine where the service specified by the user lives and append it to the url*/ switch(service) { case 'catalogue': url = url + '/voyager/index.php?'; width=900; break; case 'illo': url = url + '/relais/index.php?'; break; case 'problem': url = url + '/problem/index.php?'; height = 600; break; /* other services go here */ default: return false; } /* open a new window and forward the user to the service they requested appending the openurl string obtained earlier */ newwindow=window.open(url+qstring,'name','height='+height+',width='+width+',resizable=1,scrollbars=1,menubar=1,location=1,toolbar=1'); if (window.focus) { newwindow.focus(); } } on receiving the ‘catalogue’ parameter, this function would open a popup window containing the output of a php script located at: http://library.uvic.ca/extfiles/360link/voyager/index.php?$openurl. (the token $openurl is a placeholder for the contents of location.search.substring(1), which in this context is expected to be a valid kev openurl.) parsing metadata once the user has been forwarded to the correct service, the openurl metadata must be parsed before anything useful can be done with it. as nearly as we could determine, there appeared to be no pre-existing openurl parser class available for us to use in the scripting language of our choice (php), so we set out to create one. the code for the parser class is freely available under the gnu gpl [2], so a detailed description of how it works is unnecessary; those interested should just refer to the code. a few high-level considerations are probably worth noting, however. we determined early on that the class would only represent a partial implementation of the niso openurl standard. twenty-seven metadata formats have been defined for openurl, but in our experience only a small number are used. we immediately eliminated the xml formats, as we had never (with one exception) encountered an xml openurl in production; all known sources appear to use the key/encoded-value (kev) formats. we also focussed on the most commonly-encountered content/metadata types, those being journal, book, dublin core, dissertation, and patent. it was necessary to support version 0.1 of the standard as well as version 1.0, because a number of important sources (google scholar, for example) still use it. version 0.1 was the development implementation, superceded by version 1.0 in 2004 when openurl became a formalized standard. the class builds what is known in openurl parlance as a “context object”, which is just a fancy way of referring to openurl metadata when it has been bundled into a convenient package. the class is currently able to package 72 different metadata elements defined in the standard. it also supplies a number of utility methods to, for example, provide human-readable labels for the metadata elements, and determine which of the 72 possible elements are actually present in the context object. finally, it attempts to compensate for bad data in a variety of ways. perhaps not surprisingly, there are a number of what one might charitably refer to as imperfect implementations of openurl out in the wild. missing format identifiers, confusion of genres and formats, the use of incompatible metadata elements, and failure to include key metadata elements are just some of the evils visited upon those who would attempt to parse openurls. in practice it’s not that bad, particularly if one does not insist on enforcing a rigorous application of the standard. most problematic are missing format identifiers, given that different formats often need to be processed in different ways. in such cases it becomes necessary to guess the format based upon other clues within the openurl string, such as the genre or the presence of an rft.jtitle or rft.btitle element (journal and book title, respectively). there is a limit to how much of this is possible, however. since rolling out the parser in production we have made a number of minor tweaks and one major enhancement. the major enhancement involved supplementing the metadata in the openurl with data from the crossref doi api in cases where little metadata beyond the doi is supplied by the openurl. [3] as noted above, parsing the openurl query string was intended to make as much metadata available to supporting services as possible. while that has generally worked as intended, there are instances in which less metadata is available than would have been the case had we used 360 link’s native linking capability. those are instances in which the metadata in the openurl is incomplete in some way (eg. missing or truncated book or journal titles). 360 link is sometimes able to fill in in the missing metadata from its internal knowledgebase when it finds a match. we are therefore considering enhancing the parser class yet further with a callback to the serials solutions 360 xml api in cases where key data elements appear to be missing. building services once we were able to parse the metadata, there were three kinds of services that we wanted to build: web forms enabling the user to forward the citation information to various library service points via email, without re-keying it. these service points included our law library interlibrary loan service, our distance education service, and our problem reporting service. services to transmit the metadata to other applications. these included our relais interlibrary loan management system, our voyager library catalogue, and google scholar. services to present additional information about the item. currently only one service exists in this category; a ‘persistent link’ pointing back to the item in 360 link. building the webforms was a relatively straightforward exercise. in all cases, once the metadata had been packaged as a context object it was trivial to embed it in a web form. additional fields were added to the form to allow the user to specify their name, email address, and other information relevant to the action being performed. on submission, the form is processed by a script that creates an html email message, sends it to the appropriate recipient(s), and provides an acknowledgement to the user. [4] forwarding metadata to other applications, particularly the catalogue, was a somewhat more involved process, and we are still fine tuning our algorithms when we come up against interesting edge cases. the standard algorithm for linking from link resolvers to library catalogues goes something like this: if it’s a journal and it has an issn, search on that, otherwise do a title search. if it’s a book, do a title search. while not perfect, this is probably the most effective algorithm possible in the majority of cases, given the known limitations of searching against isbns. it is possible to achieve a baseline implementation of this algorithm using 360 link’s native linking capabilities; however we wanted more control in case there were ways we could improve the algorithm by tweaking it somewhat. in practice, our tweaks have not been major. we still follow the logic of the standard algorithm; however we have made some modifications to increase the likelihood of a successful match. for example, our voyager ils requires the dash to be present in an issn or eissn search; we can ensure that there is one by pre-processing the incoming data. for book searches, we can throw in an author keyword if a title search alone looks like it might retrieve too many hits, but only in cases where we are not attempting to match against a chapter citation. (in the case of books, openurl provides no way to determine whether an author element refers to the author of a section or the author of the entire work; we assume the former if the ‘atitle’ element is present in the openurl, indicating the citation is for a chapter.) we also break up titles into main and secondary strings based on the presence of a colon to accommodate an idiosyncracy in how voyager parses title phrase searches, and remove certain punctuation characters that are known to cause problems. finally, in cases where the openurl contains an isbn but no title element, we run a search against the worldcat basic search api to see if we can augment the metadata. once the voyager query string has been assembled, the user is forwarded to voyager via a header redirect. [5] relais accepts an openurl-like syntax which is not entirely standard, so our relais parser simply translates the incoming openurl into relais-compatible syntax before forwarding the user on via a header redirect. there were no major challenges as the relais syntax is fairly well documented; the major difficulties were posed by certain idiosyncracies in how some data elements needed to be formatted in order to parse effectively into relais. a bit of trial and error was needed to get it right, but our relais parser has been stable for several months now. [6] for the google scholar target parser, our general impression has been the less precision the better. we do specify fields using google scholar parameters for elements like journal title, book title, and author, but too much precision can exclude useful matches. better to send a few keywords and let the ranking algorithm bring the useful results to the top. [7] additional enhancements in addition to the services above, we discovered shortly before our go-live date that 360 link does not have the ability to prepend our proxy server prefix to the refworks export function. this was particularly problematic because we use an instance of refworks hosted by the ontario scholars portal in order to comply with a canadian hosting requirement mandated by provincial privacy legislation. on-campus users were being redirected appropriately from refworks’ main site to the ontario site; but off-campus users would hit a dead end on the main site, as refworks would have no way of knowing they should be redirected. once again, we were able to hack the interface using javascript, this time to create a custom submit button that inserts the proxy prefix when the refworks export was chosen. [8] integrating 360 link with metalib we originally acquired sfx as part of a package that also included metalib, ex libris’ federated search tool. although the two products operate independently from a technical standpoint, metalib relies on sfx for some of its functionality at the interface level. as we did not wish to decommission metalib or continue to maintain sfx following our 360 implementation, we were curious to see whether we could use 360 link to replace sfx in this context. as it turned out, it was possible. earlier in this article i mentioned that with one exception we had never encountered an xml openurl. metalib was the exception; it uses xml formatted openurls to communicate with sfx. unfortunately not only does our homegrown parser not parse xml openurls; 360 link does not parse them either. [9] our original plan was to create middleware to translate the xml openurls into their kev equivalents, however this proved to be unnecessary when we realized it is possible to tell metalib to use openurl v.01 rather than 1.0. since the xml formats did not appear until version 1.0, activating the earlier version of the standard caused metalib to revert to kev. things we couldn’t do we were unable to replicate two of our sfx enhancements in 360 link. because sfx was hosted locally and we had complete access to the application, we were able to embed pretty much anything we wanted to in the interface. this included services which required callbacks to external applications, namely: an ulrich’s lookup to indicate whether the item was contained within a peer-reviewed journal. the results of the catalogue lookup, obviating the need for users to click on a catalogue link to determine whether or not we have the item in print. in both cases, the best way to approximate the service was to link out to the external resource, which regrettably makes the process less efficient for the user. we were unable to reconstruct these enhancements because serials solutions was could not assign a ‘uvic.ca’ hostname to our 360 link instance. [10] consequently ajax callbacks to library-hosted middleware would not work. for security reasons, browsers are designed to prevent javascript calls to hosts on domains other than the one the browser is connecting to (the “same origin policy”), preventing us from calling out to library middleware from the 360 link results screen. this limitation regarding hostname configuration has other undesirable consequences as well, discussed below. transitioning the service after many hacks and interface tweaks, it was time to move 360 link into production. cutting over a link resolver can be a time-consuming business, as links to the old resolver must be updated in every external resource that points to it. in practice, that often means updates to dozens or even hundreds of database interfaces, which is why it is not uncommon for sites transitioning between link resolvers to run the new one and the old one in tandem for a while to give their technical services staff time to make the necessary updates. we did not want to run sfx and 360 link in production simultaneously, for several reasons: the version of sfx we were running was going end of life at the end of 2011. we did not wish to upgrade a service we were about to phase out. keeping both in production would have meant our technical services staff would have had to continue updating two knowledgebases even after the new service went live. resource contraints in our technical services department meant that it was going to take several months to update all the third-party databases linking to our resolver. linking to similar but different link resolvers might have confused some of our users. in order to cut over cleanly, we changed the configuration of the webserver on host ‘sfx.uvic.ca’ to forward all openurl requests to our 360 link instance at ‘lg5jh7pa3n.search.serialssolutions.com’ using apache’s built-in mod-rewrite. the result was seamless to users, and allowed us to immediately decommission sfx (the software, not the server) when 360 link went into production. hostname issues of course, it would have been even easier to transition the service had we originally chosen a more generic hostname for our link resolver (for example, ‘resolver.library.uvic.ca’), and been able to transfer that hostname to the new service. neither condition applied in this case, however. by calling our sfx server ‘sfx.uvic.ca’ we guaranteed that our technical services staff would have to do more work than would otherwise have been necessary during a linkresolver transition. however as it turned out serials solutions was not able to assign an institutional hostname to a hosted instance of its link resolver, so even if we had the foresight to choose a generic hostname we would not have been able to apply it to our new service. this was somewhat unexpected, as it is not a limitation of hosted services in general. typically, the customer can set up a cname alias in their dns, while the external host updates their webserver configuration to associate the alias with the customer’s instance on their server. unfortunately the best we could have done in this case would have been to set up a reverse proxy, a solution that would have had many of the disadvantages of hosting our own interface discussed in the introduction. apart from giving us the ability to more readily transition to a new service, a local hostname would have enabled us to avoid the cross-site scripting restrictions mentioned above. conclusions i had three reasons for documenting our link resolver transition: first, on the chance that our experience will be useful to other sites contemplating a similar transition; second, to share the code that we created; and third, to lay out some background that might be useful for sites (including our own) contemplating the transition to vendor-hosted technologies more generally. while the last is quite a large topic, i believe it is possible to generalize some points from our 360 link implementation that have relevance to that broader scope: standards (even flawed ones) remain critical to achieving interoperability. no matter how feature-rich our hosted systems become, they will never do everything we want them to do. this means they will need to interoperate with a range of external systems, some of which have yet to be envisioned, let alone developed. standards are critical for this. as we have seen above, systems developed around the openurl standard enabled us to develop supporting services, integrate with external systems, and efficiently transition from one service to another. where possible, develop to standards rather than proprietary apis. because most of our supporting services were built around the openurl standard, they should be portable in future. developing extensively on top of a proprietary api can be considered a form of lock-in, as the decision to transition services means either jettisoning or extensively rewriting locally developed code. (note that this applies just as much, if not more, to extensive customizations made to locally hosted systems). consider your exit strategy. looking forward, it is fairly obvious that we lost ground in this respect. it will not be possible to replicate our clean cut-over from sfx to 360 link should we ever need to transition to another link resolver, because we do not ‘own’ the hostname this time around, nor do we have the ability to rewrite 360 link urls. assign a local hostname, if possible. as a corollary to the preceeding point, there is an obvious administrative advantage to having the ability to assign a local hostname to a remotely-hosted service. there are also advantages from a development perspective, as it is possible to override some of the javascript ‘same origin’ restrictions for different hosts on the same domain. as customers, we need to ensure our vendors provide this capability, particularly as more of our services migrate to a hosted environment. for libraries with technical staff, the option of working through vendor technical support is not always optimal. achieving interoperability with local systems is often a matter of making many small tweaks after observing and testing the behaviour of applications in the wild. even with the most responsive vendor support, this kind of iterative tinkering is always more efficient if it can be done directly. libraries typically need to enhance the functionality of vendor-supplied systems to meet local requirements, whether these be the addition of features or integration with the library’s other systems and workflows. when locally hosted systems migrate offsite, it is important to retain as much of this capability as possible, while still maintaining the advantages of a hosted installation. delivering the core service through the hosted interface while supporting the bulk of our enhancements locally is a functional split that has been working well, though not perfectly, in this context. notes [1]. for example, umlaut https://github.com/team-umlaut/umlaut/wiki [2]. all of the code referenced in this article is available from the uviclinkresolver github repository: https://github.com/jdurno/uviclinkresolver. the php openurl parser class is available within the repository at: https://github.com/jdurno/uviclinkresolver/blob/master/modules/includes/contextobject.inc.php [3]. crossref/doi class: https://github.com/jdurno/uviclinkresolver/blob/master/modules/includes/crossref.inc.php [4]. for example, see the code for the problem report form at: https://github.com/jdurno/uviclinkresolver/tree/master/modules/problem [5]. openurl -> voyager module: https://github.com/jdurno/uviclinkresolver/blob/master/modules/voyager/index.php [6]. openurl -> relais module: https://github.com/jdurno/uviclinkresolver/blob/master/modules/relais/index.php [7]. openurl -> google scholar module: https://github.com/jdurno/uviclinkresolver/blob/master/modules/google/index.php [8]. see the anonymous function at the top of: https://github.com/jdurno/uviclinkresolver/blob/master/modules/popitup.js [9]. serials solutions technical support, august 15, 2012 [10]. serials solutions technical support, august 8, 2012 references niso.(2004) registry for the openurl framework – ansi/niso z39.88-2004. retrieved from: http://alcme.oclc.org/openurl/servlet/oaihandler?verb=identify talsky, d. (2008) auto-populating an ill form with the serial solutions link resolver api. code{4}lib journal (4) retrieved from: http://journal.code4lib.org/articles/108 acknowledgements much of the technical work described above was done by the uvic libraries web developer, ben sheaff, and senior unix administrator sandy gordon. the project benefitted enormously from their efforts and expertise. about the author john durno (jdurno <at/> uvic <dot/> ca ) manages systems at the university of victoria libraries on vancouver island, british columbia. subscribe to comments: for this article | for all articles one response to "hacking 360 link: a hybrid approach" please leave a response below: mang sun, 2013-01-03 excellent! as to the ajax same origin issue, two apache directives proxypass and proxypassreverse can be used as a workaround. leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – controlled terms or free terms? a javascript library to utilize subject headings and thesauri on the web mission editorial committee process and structure code4lib issue 15, 2011-10-31 controlled terms or free terms? a javascript library to utilize subject headings and thesauri on the web there are two types of keywords used as metadata: controlled terms and free terms. free terms have the advantage that metadata creators can freely select keywords, but there also exists a disadvantage that the information retrieval recall ratio might be reduced. the recall ratio can be improved by using controlled terms. but creating and maintaining controlled vocabularies has an enormous cost. in addition, many existing controlled vocabularies are published in formats less suitable for programming. we introduce a javascript library called “covo.js” that enables us to make use of controlled vocabularies as metadata for the organization of web pages. by shun nagaya, yutaka hayashi, shuhei otani and keizo itabashi introduction metadata is a powerful tool for organizing information resources, and its importance has increased on the web. in particular, a method of information organization called “social tagging” has spread rapidly on sites like delicious, youtube, and twitter. users of these web services freely create and manage metadata called “tags” that enable clustering and improved searching of information resources. these metadata are called “free terms”. free term tagging is very simple and easy to use for the internet user; but there is also a disadvantage in that the information retrieval recall ratio can be reduced. the recall ratio can be improved by using “controlled terms.” if controlled tags, or terms, were used in web services, the recall ratio would improve. however, there is an enormous cost and it takes special knowledge to create, maintain and apply controlled vocabularies. many existing controlled vocabularies are published in formats less suitable for programming, such as paper, pdf, and html documents. machine readable controlled vocabularies could improve the recall ratio and open up access to valuable resources, and we are starting to see examples of web-based controlled vocabularies appearing. thus, we propose to use controlled terms as tags in web services, social tagging services and information retrieval services. for this purpose, we developed a javascript library called covo.js that allows web service users to input controlled terms. in this article, we will introduce covo.js and the ways to use controlled vocabularies of various formats in client-side javascript libraries. covo.js covo.js is a javascript library that makes it easy to use controlled terms as a tool for information organization. the features of covo.js are controlled vocabularies in drop-down lists that are selectable by inputting a trigger word [1], an incremental search from controlled terms, and some customizable function configurations, as shown in figure 1. in the present version of covo.js (as of september 2011), we do not yet make use of the hierarchical structures of controlled vocabularies. figure 1: how to input controlled terms covo.js supports ten controlled vocabularies that fall into three categories, or patterns, that are listed in parentheses and will be discussed below: web ndl authorities (beta) (1) faceted application of subject terminology (2) form and genre terms for fiction and drama (2) library of congress annotated card program subject headings (2) library of congress subject headings (2) medical subject headings (2) tgm i, subject terms: thesaurus for graphic materials (2) tgm ii, genre and physical characteristic terms: thesaurus for graphic materials (2) inis multilingual thesaurus (3) wikipedia thesaurus (3) getting started with covo.js including the covo.js library on your web service makes it easy to utilize thesauri and subject headings. covo.js can be introduced into social tagging or information retrieval services. to try them out you will need to download covo.js and some necessary components, put them on your server, and write some configuration code. covo.js requires a few components to be installed: jquery ui and the jquery autocomplete plugin. if you want to customize covo.js, please refer to the online documentation. add covo.js to the html <head> element: <script src="covo.js" type="text/javascript"></script> and create the covo.js widget in your <body> element: <div class="ui-widget"> <label for="covo">let's covo.js ! : </label> <input id="covo" /> </div> why did we choose javascript? this library is coded with javascript. these are two reasons why we choose javascript rather than server-side scripts like perl, php, or python. the first reason is that covo.js runs on the client side and does not depend on the server-side environment. however, some behaviors of javascript depend on the client environment. this is called the cross-browser issue, and we selected the jquery ui to solve this problem. the second reason is that web service users can embed covo.js in web browsers using greasemonkey or firefox add-ons. currently, we can use covo.js only if a web service provider has set it up on the server. there are planned future enhancements of covo.js that will allow users to choose whether they use it or not. basic structure of covo.js $(function() { $("#covo").autocomplete({ source: function(request, response) { $.ajax({ url: http://api.scraperwiki.com/api/1.0/datastore/sqlite?format=jsondict&name=inis-thesaurus&query=select * from swdata where keyword like '%25" + request.term + "%25' limit 10", datatype: "jsonp", success: function(data , datatype) { response($.map(data, function(item) { return item.keyword; }) }) }) } }) }); when a javascript library cannot access thesauri and subject headings, we need to use server-side programs or web services. how to use thesauri and subject headings on the web one can add other controlled vocabularies and extend covo.js. the problem with extending covo.js is how to use controlled vocabularies in javascript libraries. through the development of covo.js, we found the following four patterns for using web-based controlled vocabularies. pattern 1: web api with jsonp pattern 2: web api without jsonp pattern 3: without web api pattern 4: not on the web in this chapter, we explain data-handling methods for various controlled vocabularies on the web. figure 2 provides an overview of the organization of this chapter. pattern 1: web api with jsonp recently, many web services have begun supporting web api, and some of them return values in jsonp (javascript object notation with padding) that are directly loadable by javascript. thus, if the web api provides jsonp data, they can be used in client-side javascript. this pattern turns out to be the simplest among the four to support. in this section we introduce the case of web ndl authorities (beta) as an example. web ndl authorities (beta) has been produced by the national diet library of japan and enables us to use all of ndl’s authority files: personal names, family names, corporate names, geographic names, etc. it accepts sparql, one of the rdf query languages, and returns authority data in jsonp format. sample sparql query prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> select ?x ?label where { ?x rdfs:label ?label. filter regex(?label, "edible") } endpoint of web ndl authorities(beta) http://id.ndl.go.jp/auth/ndla?query={{sparql}}&output=json&callback=ndl sample response ndl({ "head": { "vars": [ "x", "label" ] }, "results": { "bindings": [ { "x": { "type": "uri", "value": "http://id.ndl.go.jp/auth/ndlna/00966476" }, "label": { "type": "literal", "value": "edible books" } } ] } }) pattern 2: web api without jsonp when web services provide web apis that do not support jsonp, the web api cannot be accessed directly from javascript. this is due to the same origin policy of web browsers, which prohibits cross-domain requests [2]. the simplest technique for this pattern is to make a “proxy” on the same server as covo.js that converts other data formats to json. in this section, we introduce the case of terminology services provided by oclc. this service supports seven vocabulary resources, and it is equipped with the sru interface. this interface covers multiple representations including html, marc xml, skos, and zthes. thus, we have to prepare a server-side program to convert xml or html into jsonp. the following is a converter program. sample converter php program <?php $query = strip_tags($_get&#91;"q"&#93;); $subject = strip_tags($_get&#91;"subject"&#93;); $callback = strip_tags($_get&#91;"callback"&#93;); $url = 'http://tspilot.oclc.org/'. $subject . '/?query=oclcts.terms+any+%22'. $query . '%22&version=1.1&operation=searchretrieve&recordschema=info%3asrw%2fschema%2f1%2fmarcxml-v1.1&maximumrecords=10&startrecord=1&recordpacking=xml&sortkeys='; $source = file_get_contents($url); $xml = new simplexmlelement($source); $xml->registerxpathnamespace('ns2', 'http://www.loc.gov/marc21/slim'); $source = $xml->xpath('//ns2:datafield[@tag="150"]/ns2:subfield[@code="a"]'); echo $callback ."(["; foreach($source as $value){ echo '{"keyword":"'. $value .'"},'; } echo "])"; ?> pattern 3: without web api when web services are not equipped with web api, we need to harvest and scrape html with certain web services or server-side programs. this section explains the two cases of the inis multilingual thesaurus and the wikipedia thesaurus. the key difference between these two services is that the inis multilingual thesaurus consists of static web pages, while the result pages of the wikipedia thesaurus can vary dynamically based on the user’s query. the inis multilingual thesaurus is a set of subject vocabularies on nuclear science and technology published online in html and pdf formats. now, we need not only to prepare a proxy, as explained above, but also to scrape data from such web pages. here, our approach is to use another web service, scraperwiki. we simply write some php code on “screen scraper,” the online editor of scraperwiki, and then it scrapes only the necessary data from the web pages and returns them in jsonp. sample php proxy program <?php require 'scraperwiki/simple_html_dom.php'; $html = new simple_html_dom(); $html = file_get_html("https://nkp.iaea.org/inismlthesaurus/en/ind.html"); foreach($html->find('body h2') as $data1){ $targeturl = $data1->find('a',0)->href; $targeturl = "https://nkp.iaea.org/inismlthesaurus/en/". $targeturl; $htmltree = new simple_html_dom(); $htmltree = file_get_html($targeturl); foreach($htmltree->find('body a') as $data2){ $keyword = $data2->find('p',0)->plaintext; $record = array('keyword' => $keyword); scraperwiki::save(array('keyword'), $record); } } ?> wikipedia thesaurus is a product of special interest group on wikipedia mining. for this service, one needs to query the database with a http get request, get the search results in html, and scrape them using regular expressions. the following is a sample proxy script for wikipedia thesaurus. a url to get http://dev.sigwp.org/wikipediathesaurusv3/search.aspx?k=nuclear&t=0&l=english part of the response <a id="dsearchresults_ctl00_attributelist_ctl01_hrelatedterm" href="search.aspx?k=plutonium&t=0&l=english">plutonium</a> sample php proxy program <?php $query = strip_tags($_get&#91;"q"&#93;); $lang = strip_tags($_get&#91;"la"&#93;); $callback = strip_tags($_get&#91;"callback"&#93;); if (strlen($query) == mb_strlen($query, 'utf8')) { $lang = "english"; } else { $lang = "japanese"; } $url = 'http://dev.sigwp.org/wikipediathesaurusv3/search.aspx?k=' . urlencode($query) .'&t=0&l='. $lang; $source = file_get_contents($url); preg_match_all('/<a id=\"dsearchresults_ctl&#91;0-9&#93;&#91;0-9&#93;_htitle.*<\/a>/', $source, $result, preg_set_order); echo $callback ."(["; foreach($result as $value){ $value = preg_replace('/<a.*">|<\/a>/','',$value); echo '{"keyword":"'. $value[0] .'"},'; } echo "])"; ?> pattern 4: not on the web when controlled vocabularies are not provided in any machine-readable formats, they are challenging to use. unfortunately, we have not yet met this challenge. if the controlled vocabularies you are interested in are only available offline, you might ask the suppliers to publish their resources online (and implement web api for their services if possible). the final possibility is digitizing them yourself using a document scanner and ocr technology. for all these patterns, we are required to pay attention to the copyright law on using controlled vocabulary services. conclusion and future work covo.js provides support for user input of controlled vocabularies. the number of controlled vocabularies has gradually increased. not all web services support access via a web api. however, with some javascript or server-side code, controlled vocabularies without web api can be made available to covo.js. the methods to accomplish this are divided into four patterns, as indicated in this article. furthermore, two problems should be solved to improve the functionality of covo.js. first, controlled vocabularies have rich structures of links among terms, such as bt (broader terms), rt (related terms), or uf (used for) links, but covo.js does not make good use of this. in the present version of covo.js, each keyword can only be displayed in a linear mode. second, up to now, our proposed library has only used controlled vocabularies on the web. it is true that controlled terms can improve the information retrieval recall ratio, but free terms are widely used in actual services, such as metadata used in social tagging or search engine queries (or indexes). thus, in the future we will include such free terms as well as controlled terms. for this purpose, it is necessary to use a mechanism to implement free terms as a controlled vocabulary. notes [1] the “trigger word” can be customized; it has been set to “-“(hyphen) in this article. [2] in this article we do not deal with solutions to the same origin policy other than using jsonp. however, we want to note one of them here, cross origin resource sharing (cors: http://www.w3.org/tr/cors/). web browsers that implement xmlhttprequest (xhr) level 2 allow cross-domain requests from any client if the web servers are configured to include `access-control-allow-origin: *` in the headers of the http responses. many modern browsers, including internet explorer 8+, have been supporting xhr level 2. about the authors shun nagaya (haseharu@gmail.com) is a librarian at the japan atomic energy agency. he edits jaea reports and jaea r&d review and manages the grants for paper submission and conference presentations. he holds a bachelor’s degree in engineering from the university of electro-communications and a master’s degree in library and information science from university of tsukuba. yutaka hayashi (hayashiyutaka@gmail.com) is a research librarian at the national diet library, japan, and is engaged in writing blogs and articles for libraries and library science on the current awareness portal (http://current.ndl.go.jp/). he has a master’s degree in mathematics from nagoya university. he formerly worked at kyoto university until march 2011. shuhei otani (otani0083@gmail.com) is a reference librarian at the university of ryukyus and is in charge of information literacy. he has a bachelor’s degree in korean history from kyushu university. nagaya, hayashi, and otani are library geeks and formed a group called “li:d tech” in january 2011. keizo itabashi is a deputy director of the intellectual resources department of the japan atomic energy agency. subscribe to comments: for this article | for all articles 5 responses to "controlled terms or free terms? a javascript library to utilize subject headings and thesauri on the web" please leave a response below: diego ferreyra, 2011-10-31 very nice work!! i will try to develop interface from tematres vocabulary server. there are more than 100 vocabularies on tematres. i do some test with jquery autocomplete… (http://vocabularyserver.com/tools/autocomplete/) buy i think your script is more configurable and elegant. jonathan rochkind, 2011-11-01 this seems useful not just for assigning vocab, but for suggesting search terms to searchers, from controlled vocab, in a database that uses it. i wish more providers of vocab would provide it in a jsonp form, so a proxy server was not neccesary. your js code seems to be using proxy server for everything; have you found any controlled vocabs whose maintainers provide jsonp access in the first place, allowing cross-domain jsonp access from javascript? shun nagaya(author), 2011-11-05 hi diego ferreyra, thank you for your comments ! i think tematres is very cool project about controlled vocaburary. mr. hayashi (author) wrote news-report about tematres published in current awareness portal ( http://current.ndl.go.jp/node/18780 ). please try covo.js, if you possible ! shun nagaya(author), 2011-11-05 hi jonathan rochkind, thank you for your comments and question ! we hope all data providers provide jsonp form or cors. covo.js (ver 0.1) is supporting web ndl authorities, and this service provided jsonp data. these detail are described at “pattern 1: web-api with jsonp”. change or be irrelevant @ commonplace.net, 2012-10-10 […] and information literacy must support each other” and shun nagaya and keizo itabashi – “covo.js : a javascript library to utilize subject headings and thesauri on the web”. this doesn’t mean that the other talks were bad, i just didn’t manage to talk to people […] leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – digitization decisions: comparing ocr software for librarian and archivist use mission editorial committee process and structure code4lib issue 52, 2021-09-22 digitization decisions: comparing ocr software for librarian and archivist use this paper is intended to help librarians and archivists who are involved in digitization work choose optical character recognition (ocr) software. the paper provides an introduction to ocr software for digitization projects, and shares the method we developed for easily evaluating the effectiveness of ocr software on resources we are digitizing. we tested three major ocr programs (adobe acrobat, abbyy finereader, tesseract) for accuracy on three different digitized texts from our archives and special collections at the university of western ontario. our test was divided into two parts: a word accuracy test (to determine how searchable the final documents were), and a test with a screen reader (to determine how accessible the final documents were). we share our findings from the tests and make recommendations for ocr work on digitized documents from archives and special collections. by leanne olson and veronica berry introduction this paper grew out of a need we experienced at the university of western ontario. when we began digitizing material from our special collections and archives, we found that there were several popular software options for performing optical character recognition (ocr) recommended by our colleagues, but evaluation of their effectiveness for historical documents was anecdotal. we wanted to make an informed choice and understand which program would best fit our needs. we started with a search for evaluations of ocr software. information about ocr in the literature tended towards quick blog posts (e.g. gringel 2020; han 2019) or in-depth articles designed for digital scholarship projects, often with a heavy dose of computer science (e.g. springmann et al. 2016; clausner et al. 2020). we realized that relying on the results from previously published evaluations wouldn’t be helpful, as software continues to evolve: a study from five years ago may no longer be applicable. could we recreate a test used elsewhere? many of the evaluative tests of ocr processes for digital humanities projects are time-consuming (for example, creating a ground truth document) and require significant expertise. we wanted to develop a reasonably simple method to evaluate software in a way that made sense for our digitization projects. we wanted to be able to reproduce this method when we had a new type of material to digitize, and to re-run the test a few years in the future to determine if the “best” software had changed. we considered our digitization project needs. our archives and special collections team is interested in creating searchable pdfs from scanned, typed documents such as journals, newspapers, and books. these pdfs are accessed by our users through our digital repositories such as omeka, open journal systems, and our institutional repository that runs on digital commons. some of our items are scanned in-house, and some scanning is outsourced. we are also currently scanning handwritten material, but that is outside the scope of this paper — our archivists are piloting a user transcription project for handwritten documents such as diaries. for this project, we are not looking at the mets/alto xml standards for digitization projects, though we are hoping to explore this in the future. for this study, we determined that we wanted to know how ocr programs facilitate searching and accessibility of documents in a repository or database, replicating how our users would interact with the documents online. we examined how well these programs recognize individual words and how well they handle page segmentation. we tested three (and a half) popular ocr programs: abbyy finereader (and its hot folders option), adobe acrobat pro, and tesseract. we also looked briefly at two options from google. we chose documents from recent digitization projects in our archives and special collections, each with different potential problems that could affect the accuracy of the ocr. we tested each program on the same section of each of three documents. background optical character recognition and historical documents optical character recognition software is not perfect, particularly when dealing with historical text (carrasco 2014). problems with ocr in widely-used academic resources are well documented. for example, thompson’s study found ocr errors rates of up to 30% in digitized historical issues of the british medical journal (thompson, 2016, 1). kichuk (2015) describes problems with poor ocr in major e-book providers such as project gutenberg, the internet archive, and hathitrust. carracso (2014) provides a variety of reasons for why ocr of historical documents is more difficult: the quality of the original document might be poor, with stains or smudging, older fonts may be less easily recognized by software, the documents may have complex layouts, and older vocabulary may be spelled differently and less easily recognized. han (2019) states that “in most cases if you need a complete, accurate transcription you’ll have to do additional review and correction.” low quality ocr can hurt the effectiveness of searching over documents. if words are misrecognized or not recognized at all, a keyword search in a database or repository will not return the document, leaving out relevant material for researchers (thompson 2015). from the library or archive’s perspective, if the documents aren’t findable, the work, time, and money we’ve put into digitizing them is wasted. pletschacher argues that knowing the accuracy of ocr helps us “manage researcher expectations and understand what we as a library are offering” (2015, 39). for example, thompson notes that “more sophisticated search methods, e.g., those based on text mining…techniques, can be even more adversely affected” (thompson 2015, 1). if we know the approximate accuracy of the ocr on a digitization project, we can advise a researcher whether the resource is searchable, whether it’s accurate enough for text mining, and whether additional cleanup work will be needed. evaluating the accuracy of ocr for a project in advance also helps to estimate the time needed to correct errors after ocr (clausner 2020). understanding the accuracy of the content we’re providing means that we as librarians and archivists can decide whether the ocr output from software is good enough for a specific project, or whether we’ll need to set aside staff time after running ocr to continue cleanup. for example, at the university of western ontario, most of our projects only require searchability, but on one project we are partnering with our department of philosophy to perform accurate cleanup of individual characters in the journal locke studies to prepare it for a text analysis project. methods for testing ocr accuracy as part of planning this project, we looked at the literature for methods used to test ocr. we discuss how we adapted these methods below in our “testing method” section. to evaluate ocr software, we have access to “[t]wo sources of information: the output of the ocr engine, and the ground truth content of the image” (carrasco, 2014, 182). the output of the ocr engine refers to the accuracy of the individual characters: how well the ocr software recognizes each character. the ground truth content refers to where the characters are located on the page. when testing with characters, alkhateeb (2017) recommends looking at “the number of inserted, deleted, replaced (substituted) symbols in the recognized text” (769). in the literature, there are a variety of similar formulas used for testing the error rate of individual words. where wer is the word error rate, iw is the number of words inserted, sw the number of words substituted, dw the number of words deleted, and nw the total number of words in the document. essentially, each of these authors is recommending a similar method for calculating character/word accuracy. generally, when testing ocr software for specific projects, testing characters is more onerous and less useful (pletschacher, 2015) than testing words, and “word error rates are usually higher than character error rates” (carrasco 2014, 180). testing the layout of the document while ocr software might excel at recognizing individual characters, for some projects this is not enough. pletschacher explains that “[i]n addition to textual results, page reading systems are also expected to recognise layout and structure of a scanned document page. this comprises segmentation (location and shape of distinct regions on the page), classification (type of the regions defined by the segmentation; e.g. text, table, image, etc.), and reading order (sequence/grouping of text regions in which they are intended to be read)” (pletschacher, 2015, 39). with a simple text document this is straightforward. however, ocr software can have trouble recognizing different document layouts, such as newspaper columns, headlines, photo captions, and tables. sentences and paragraphs can blend together, with the software reading across the entire page from left to right without recognizing breaks between columns, articles, or tables. page contents can be scrambled, with sentences running together, paragraphs in the wrong order, or tables reduced to gibberish. a thorough way to test page layout recognition is to manually create a ground truth. this is a transcription of the original document, replicating the exact location of each character (pletschacher, 2015; carrasco 2014). carrasco (2014) developed a tool to compare a ground truth transcription to an ocr output, pletschacher (2015) developed ground truths for a project analyzing historical newspapers, and springmann (2016) looked at automatic ways to compare ocr accuracy using ground truth documents. creating the ground truth is a very time-consuming, detailed process, and the literature also includes a few cautions about understanding what information it provides. clausner notes that “if the reading order is wrong the character accuracy measure can be very low, even if the actual recognition of each individual character is perfect”, and that sometimes, reading order does not even matter – in a newspaper, “there is no prescribed order in which to read the different articles on a newspaper page” (clausner, 2020, 391). when we designed our study, we thought about what made sense to us for testing: was it important to look at the accuracy of individual characters, or full words? did the exact placement of a character on a page in a transcription matter to us, or would our users primarily interact with scanned documents by searching them? is it feasible to take the time to create a ground truth document each time we want to test the accuracy of software for a new project? we get into our own decisions below. our study resources selected for testing we began planning our study by thinking of typical text digitization projects seen in academic library archives and special collections: digitization of historical newspapers, older reports published by the university, journal backfiles for open access projects, and special collections books. we looked only at printed english text in the latin alphabet, excluding handwriting, music notation, other languages, and other alphabets, all of which require more sophisticated ocr work. for now, we’re interested in beginner-level projects. we selected three digitized resources to test with different ocr software: the 1928-29 annual report of the president from the university of western ontario, the january 16, 1975 issue of western news from the university of western ontario, and the first issue from 1865 of the voice of the bondsman newspaper from stratford, ontario. we selected these resources for several reasons: they were from a variety of different time periods (1856, 1929, and 1975) they were typical of the textual material our archives and special collections team digitizes they had different formats and some of the common issues affecting ocr quality, such as columns, tables, varying font types and sizes, noise and smudging on the original paper, images and captions, older or unusual spelling, and skewing the scans were high quality, and the items were scanned at 300 dpi or above the items had already been digitized, since we did not have regular access to our scanning equipment during the covid-19 pandemic the chart below (figure 1: digitized resources tested in this study) summarizes the resources we tested. the university of western ontario provides open access to each of these resources, and they can be viewed at the links below the chart. figure 1. digitized resources tested in this study – annual reports of the president, western news, voice of the bondsman software selected for testing we intentionally chose software that is commonly used, that is affordable (under $300/year for an individual license) and that has a modest learning curve. on our archives and special collections team, we don’t have regular staff involved in text-based digitization (we do have an archives assistant who specializes in photographs, but she does not have extra time in her schedule for other digitization projects). we have one librarian who manages our in-house digitization (one of the authors, leanne olson), who often needs to find help from part time students or staff from other departments who have extra time to spare while working on a service desk. the lack of consistency in staffing means we are not looking for software that takes a long time to set up or is difficult to use. at western, staff are currently using abbyy finereader and adobe acrobat pro dc for different projects. we were also curious about trying an open source software program to compare its effectiveness for our purposes, and had heard tesseract recommended by colleagues at other institutions. the chart below (figure 2: software tested in this study) provides a glimpse at the software we selected for this study: figure 2. software tested in this study – abbyy finereader, adobe acrobat pro, tesseract abbyy finereader pdf allows users to edit pdfs, and it has features that allow the user to “draw” columns and other text breaks onto the digital document to allow for more precise ocr. we tested the standard desktop version, as well as abbyy’s hot folder feature. this allows administrators to “[s]et up a watched folder on a local or network drive, ftp server, or mailbox, and any files placed in it will be converted automatically with the predefined settings, immediately or according to a schedule.” (https://pdf.abbyy.com/how-to/) we were curious to see if this process worked as well as running ocr in the individual desktop program. the desktop version we tested was an individual abbyy finereader pdf license that one of the authors purchased for a $260 cad yearly fee. the hot folders feature was provided by our library’s it services for all staff to use. adobe calls adobe acrobat pro “the complete pdf solution” (https://acrobat.adobe.com/ca/en/acrobat.html), and it allows users to create, edit, convert, share, and sign pdfs. we tested acrobat pro as part of an individual adobe creative cloud license that had already been provided by our library’s it services to one of the authors. to purchase adobe acrobat pro individually, the cost is $270 cad per year. tesseract is an open source ocr engine. users can interact with tesseract directly from the command line (which is what we did for our testing), by using an api, or by using third-party add-on graphic user interfaces (https://tesseract-ocr.github.io/tessdoc/). similar to abbyy finereader, tesseract allows for more refined document definition, though different options on the command line. we tested all software on windows 10 enterprise version 2004 with the exception of adobe acrobat pro, which we tested on macos mojave version 10.14.6. we installed the standard version of each program, without custom modifications. we looked into two other options from google, but did not test them fully as a part of this study. google drive has an ocr function as part of its cloud storage service, and allows users to freely convert image files and pdfs to text in a google document. (https://support.google.com/drive/answer/176692). google also offers the google cloud vision api which is a paid product with an ocr function (https://cloud.google.com/vision/docs/ocr). vision api focuses on images stored in the google cloud, using machine learning to understand and classify images, build metadata, detect objects and faces in the images, read handwriting, and extract text from images. (https://cloud.google.com/vision) method when developing our testing method, we thought about the purpose of performing optical character recognition on these digitized documents. what would our users care about, and what are their needs? generally, we’re interested in making sure the document is searchable and findable in databases or repositories. we’re also interested in ensuring the document is accessible to users with visual or other disabilities. occasionally, we might have users interested in extracting transcriptions from the documents or running digital humanities projects – however, from what we’ve learned in our literature review, we knew that at western we didn’t have the staff time available to handle post-ocr corrections to the point where the text is perfect or close enough to perfect for these boutique projects. therefore we are primarily interested in searchability and accessibility. preparing the documents we started with the documents in pdf form, since our goal is to create searchable pdfs. the voice of the bondsman newspaper and president’s reports bound volumes were originally scanned as tiffs (with the archival tiffs retained for digital preservation purposes), edited as needed, and then converted to pdfs for access. the western news digitization project was outsourced, and we only had access to the pdfs (the vendor did not provide image files). beginning with the pdfs for each project allowed us to test the scans using the same process. in the case of tesseract, we did need to convert the pdfs to image files to run the software. we did this using an open source program imagemagick (https://imagemagick.org/) that can be called from the command line in the same way tesseract is. in some cases, the documents had already been edited as part of our post-digitization process (outlined above in figure 1). we ran the ocr software on each document. on our first pass, we did not give the software any instruction on what to do with columns, tables, or other document formatting. we were curious to see how the software would handle these issues out of the box. determining ocr accuracy in deciding how to evaluate the ocr quality for each program, we focused on two things: how well the document could be searched in a database or repository (word accuracy using counting and percentages of correct words) and how well the document could be accessed by someone visually impaired (page segment accuracy using a screen reader). we will call these two tests “word accuracy test” and “screen reader test.” word accuracy test we chose to test the accuracy of words to determine how well the documents could be searched in an online database. we eliminated the approach of counting characters based on pletschacher’s suggestion that “due to the nature of the algorithm, calculating the character accuracy is too resource intensive for long texts (such as found on newspaper pages). moreover, character accuracy is typically only interesting to developers of ocr systems and not normally used to assess the suitability of recognized documents for specific use scenarios” (2015, 41). in our case, we are looking at a specific use scenario: searching the document. we worked one page at a time. to determine the error rate for individual words, we manually counted the number of words on the page in the original pdf, and recorded this. then, we selected all text in the ocred page and pasted it as unformatted text into a text editor (microsoft word). this was our text document to compare to the original pdf. we compared the text document to the original pdf file by manually counting the correct words and the errors. at this stage, we focused on accuracy of the individual words: we did not worry about the order of paragraphs and articles, or about sentences that ran together from the ocr engine misreading columns. before counting errors, we took a quick look at the resulting text to see what types of errors had been introduced, in order to make consistent decisions on how to treat them. we counted a word as incorrect if it had any of the following problems: a missing character an extra character (including a space breaking up a word) an incorrect character (where a character is replaced by an incorrect on) swapped characters (where all the characters are present, but in the wrong order) an entire word missing each of these errors breaks the word and keeps it from being found in a full text search. we compared our initial decisions to equations offered in the literature and found they corresponded well with equations already developed. this is what we used: in this case, “∑ total words” refers to the total of all words on the page in the original document; we counted these by hand. “∑ incorrect words” is the sum of all words with one of the problems listed above (1-5). we chose not to count differences in formatting such as italics, underlining, font face, font size, or punctuation that did not break the word (for example, if a comma was inserted in the middle of the word, it would affect searching, but if a semicolon after a word was replaced by a comma, it would not). we might have different considerations if we were digitizing material for digital humanities projects, which often involve writing algorithms to recognize patterns in a large corpus of text. errors in punctuation, case, font size, font face, underlines, italics, etc. become more important with this type of project. screen reader test to determine the effectiveness of the ocr engine at preserving segments in the case of documents with columns, tables, or articles, we first thought about creating a ground truth as in several of the papers we read (e.g. carrasco 2014, clausner 2020). we realized that this didn’t make sense for our study: we didn’t have the time or staff resources to create the ground truth documents, and we were also asking a different question than those researchers. we were interested in how the quality of the ocr would affect users of the pdf interacting with it in an online database. we had recent experience learning about how our users interact with pdfs from digitization projects. in the fall of 2020, the university of western ontario ran an accessibility study of our digitized collections site in omeka. we watched users interact with the site and saw visually impaired students using screen readers to listen to pdfs read out loud. we realized that was what we cared about in our digitization projects: would any problems with ocr that affected columns, articles, and tables, actually inhibit the ability of users to interact with our documents we briefly tested copying text from each pdf and pasting it into a text editor (notepad). we found that the text created by each ocr program had errors in ordering columns, articles, and tables. we thought of counting the number of segments (e.g., a newspaper article, a column of text or numbers) that were out of order in the resulting text document. however, this still doesn’t replicate the user experience we’re interested in, since our users would not be interacting with plain text versions of the documents. we wanted to test the ocred text using the pdf itself. we thought back to our accessibility testing and used the free screen reader nvda version 2020.4 (https://www.nvaccess.org) to test whether the document’s ocr could be read correctly by a screen reader. for this test, we went back to the original ocred pdfs. the nvda screen reader, once installed and open on the user’s computer, reads the text out loud as the user hovers over different sections of the document. we opened a pdf and hovered over the text to see how well it was read, making notes on the effectiveness of the screen reading for each of the documents. we focused on two aspects of use: whether the text was recognized well enough to be understandable by ear, even if some words were slightly garbled whether the screen reader could read columns, tables, and articles correctly, or whether it blended sentences together between page sections, making it incomprehensible to the user. by nature this test was qualitative: the screen reader gave us a good sense of how well the ocr had worked, without forcing us to spend time creating a ground truth to get specific percentages of correct and incorrect text positions.we developed these basic rankings for the screen reader test: excellent there are few to zero errors in the text and no errors in the page segmentation, so the document is read smoothly good words are understandable even with errors in the text; there are few or small errors in the page segmentation, such as in headings or captions fair the words are difficult to understand, and some words may be skipped, but sentences are generally in order and a user can listen to the text, even if the experience is frustrating unusable either the text recognition or the page segmentation is poor enough that a user cannot understand the document by listening to it figure 3. screen reader evaluation categories findings below, we share our findings for each of the programs on each of the documents. figures 4 and 5 summarize our recorded word error percentages and our notes and rankings from the screen reader experiment. then, we discuss our general notes about each program. figure 4. word accuracy findings. figure 5. screen reader observations. general observations on software performance the word accuracy test showed us that in general, the two versions of abbyy and tesseract were all fairly similar when it came to recognizing text. both handled the article and table content well in western news and the president’s report, and struggled with larger or different fonts used in headers, article titles, and photo captions. tesseract was weaker on voice of the bondsman: it had more difficulty filtering out noise and smudges on the newspaper and occasionally inserted symbols (representing the noise) that broke individual words. tesseract also inserted extra “words” with western news when it interpreted dotted borders between articles as characters. tesseract handled the screen reader test well with voice of the bondsman and the president’s reports, but failed at understanding column breaks in western news. abbyy finereader was the strongest at recognizing breaks between articles and tables. we also found the accuracy of both versions of abbyy (desktop and hot folders) to be similar. on all three resources, adobe acrobat pro performed worse than the other programs, and with the oldest text, voice of the bondsman, the ocr quality was not acceptable. adobe in particular had difficulty with word accuracy: although the individual characters were often correct, many of the errors were extra spaces in the middle of words, no spaces between the words, or punctuation inserted in the middle of words – all of these issues render the words unfindable in a search. we were asked by the code4lib editors to investigate two google products as well. the first was google’s cloud vision api. the best way to understand how the cloud vision api works is to drop an image in their “try the api” function: https://cloud.google.com/vision. we tested this and it was able to quickly determine whether there was a person or animal in the photo, recognize the emotion on a person’s face, recognize objects, extract text from book titles in the photo, and assign metadata labels (keywords) based on what the photo contained. the vision api seems most useful to create metadata and identify objects and text in large collections of digitized photos, which is how the new york times has used the product (https://cloud.google.com/blog/products/ai-machine-learning/how-the-new-york-times-is-using-google-cloud-to-find-untold-stories-in-millions-of-archived-photos). the api does include a document text detection feature that is potentially applicable to text digitization projects. we found the setup process to be quite involved and that it wouldn’t work for our purposes: our study’s purpose was to find software that is easy to set up and can be used by librarians or archivists new to digitization and ocr work. the cloud vision api is intended for users significantly more advanced than we were, and its primary purpose is not creating searchable pdfs or screen readable pdfs, but extracting text from images. this is a fascinating and useful tool, but not one that made sense to use for our text digitization projects. google drive has an ocr function as part of its cloud storage service, and allows users to freely convert image files and pdfs to text in a google doc. (https://support.google.com/drive/answer/176692). we found that google drive worked fairly well at recognizing text in the president’s reports, but was unable to recognize any text in western news and was not able to open voice of the bondsman. google drive is likely not useful for archival and special collections materials, but may work as a no-cost solution for very clear, modern texts. recommendations and conclusion pre-ocr workflow before running optical character recognition on scanned documents, we recommend putting thought into the scanning and image editing process. we generally try to choose the cleanest copies we can find, scan documents with at least 300dpi, and perform batch image editing after scanning and before converting the images into pdfs. we recommend using an image editor such as adobe lightroom or gimp to batch deskew and rotate your images, adjust contrast, darken or lighten text if needed, crop out black borders from scanning, or fix any other obvious problems with the scans that make it more difficult to read the text. we used this study to refine our workflow. our process now is to scan items as high-resolution tiffs, batch edit the images in adobe lightroom, export them as tiffs or jpgs (depending on the quality we need), then import them into adobe acrobat pro and combine them into a pdf. we then switch to abbyy finereader for running ocr. selecting software these recommendations are based on our testing and the type of digitization projects undertaken by staff in library archives or special collections. in your own library or archive, it’s worth doing some quick tests to determine which software works best for our project. the accuracy counts we’ve recorded in our test are a snapshot in time: the software will likely continue to improve in the future. also, the more time you are willing to spend with the software up front (learning settings in tesseract or abbyy finereader, in particular), the more you can improve the accuracy of your ocr results. we found abbyy finereader to be the most useful when the goal is to ocr a document that will be uploaded into a repository or database, where you want the text searchable and a high accuracy for the ocred words. we recommend the “hot folders” feature if you have multiple people needing to run ocr on many documents. it allows users with little knowledge of ocr or digitization to drop documents into a folder on a network drive and have the ocr run automatically, producing high accuracy for documents with basic page layouts (for example, a book or a journal article). it will also provide high word accuracy for more complex layouts, and ensure these documents are searchable. for a small and complex project, we suggest trying the desktop version of abbyy finereader to individually ocr scans with difficult page segmentation content such as columns, tables, or newspaper articles: the digitizer can spend extra time up front to “draw” the page segments, ensuring that abbyy will ocr the document correctly. for more basic page layouts, this isn’t necessary. we recommend using tesseract when you want to extract text and get it into a text file for a digital humanities project, or to add a transcription or full text metadata into a repository. with tesseract, we used the standard text output to do our testing (saving the step of copying and pasting text from the pdf so we could count errors). we used pdf in the output format to output a searchable pdf, which is what we want for our experiment with the screen reader. tesseract handled the tables extremely well compared to the other programs — everything was in order. we would recommend tesseract for if you want to extract text from a pdf and maintain tabular order. tesseract is also an excellent free, open source alternative to abbyy and fairly easy to learn if a staff member is comfortable with the command line interface. both the abbyy findreader desktop version and tesseract include additional tools and settings for recognizing columns and tables. with abbyy, the user can “draw” and select columns before running ocr, to show the software where to segment it. tesseract has different settings that can be called when running commands on multiple pages. for a small project similar to voice of the bondsman, where the newspaper is older and deals with a lot of noise, it may be worth testing out abbyy’s manual column selecting. for larger projects such as western news, this isn’t feasible from a workload perspective. abbyy handled the columns fairly well with standard settings; if using tesseract we would recommend doing some experimentations with the different commands first. we started with the manual and examples on the tesseract github site: https://tesseract-ocr.github.io/tessdoc/command-line-usage.html google drive works as a free tool if you need to ocr a very simple document, such as a report that’s been scanned. we don’t recommend it for archival material. adobe acrobat is very useful for editing pdfs, organizing pages, rotating pages, etc., but based on our tests in this study, we now know adobe acrobat is less useful for ocr of digitization projects common to archives and special collections, as we found the accuracy to be quite low with the historical documents tested. further investigations: at western, we plan to continue testing our chosen software (abbyy finereader and tesseract) following the method in this study, as we embark on new digitization projects. as we get a sense of which software works best for different types of projects, and how much post-ocr correction is needed, we’re developing internal best practices documents that can apply to general classes of projects: for example, large-scale newspaper digitization, or small scale rare book digitization. we’re also working on manual post-ocr correction for a digital humanities project that requires perfect or near-perfect ocr at an accuracy the software can’t reach on its own, and we plan to continue fine-tuning our methodology. references alkhateeb f, abu doush i, albsoul a. 2017. arabic optical character recognition software: a review. pattern recognition and image analytics [internet]. [cited 2020 may 21];27(4):763–776. available from: https://link.springer.com/article/10.1134/s105466181704006x carrasco rc. 2014. an open-source ocr evaluation tool. in: proceedings of the first international conference on digital access to textual cultural heritage; 2014; madrid: acm press. p. 179–184 [internet]. [cited 2020 may 26]. available from: http://dl.acm.org/citation.cfm?doid=2595188.2595221. clausner c, hayes j, antonacopoulos a, pletschacher s. 2017. creating a complete workflow for digitising historical census documents: considerations and evaluation. in: proceedings of the 4th international workshop on historical document imaging and processing; 2017; kyoto: acm press. p. 83–88 [internet]. [cited 2020 jun 16]. available from: http://dl.acm.org/citation.cfm?doid=3151509.3151525. clausner c, pletschacher s, antonacopoulos a. 2020. flexible character accuracy measure for reading-order-independent evaluation. pattern recognition letters [internet]. [cited 2020 may 20];131:390–397. available from: https://www.sciencedirect.com/science/article/pii/s0167865520300416?via%3dihub gringel f. (2020). comparison of ocr tools: how to choose the best tool for your project. medium [internet]. [updated 2020 jan 19]. [cited 2020 jun 15]. available from: https://medium.com/dida-machine-learning/comparison-of-ocr-tools-how-to-choose-the-best-tool-for-your-project-bd21fb9dce6b. kichuk d. 2015. loose, falling characters and sentences: the persistence of the ocr problem in digital repository e-books. portal: libraries and the academy [internet]. [cited 2020 may 18];15(1):59–91. available from: https://muse.jhu.edu/article/566423 han t, hickman a. 2019. our search for the best ocr tool, and what we found. source [internet]. [updated 2019 feb 19]. [accessed 2020 jun 16]. available from: https://source.opennews.org/articles/so-many-ocr-options/. pletschacher s, clausner c, antonacopoulos a. 2015. europeana newspapers ocr workflow evaluation. in: proceedings of the 3rd international workshop on historical document imaging and processing; 2015; gammarth, tunisia: acm press. p. 39–46 [internet]. [cited 2020 may 22]. available from: http://dl.acm.org/citation.cfm?doid=2809544.2809554. springmann u, fink f, schulz ku. 2016. automatic quality evaluation and (semi-) automatic improvement of ocr models for historical printings. arxiv [internet]. [cited 2020 may 21]. available from: http://arxiv.org/abs/1606.05157. thompson p, mcnaught j, ananiadou s. 2015. customised ocr correction for historical medical text. in: 2015 digital heritage. vol. 1. p. 35–42. available from: https://ieeexplore.ieee.org/document/7413829 about the authors leanne olson is the digitization and digital preservation librarian in archives and special collections at western university in london, ontario, canada. previously, leanne worked as a metadata management librarian, also at western. lolson3@uwo.ca https://works.bepress.com/leanne-olson/ veronica berry is the records management & document control specialist at hts engineering. during her mlis at western university she participated in an eight-month co-op at western libraries where she worked on projects relating to digital preservation and digital collections. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – automated metadata formatting for cornell’s print-on-demand books mission editorial committee process and structure code4lib issue 8, 2009-11-23 automated metadata formatting for cornell’s print-on-demand books cornell university library has made print-on demand (pod) books available for many of its digitized out-of-copyright books. the printer must be supplied with metadata from the marc bibliographic record in order to produce book covers. although the names of authors are present in marc records, they are given in an inverted order suitable for alphabetical filing rather than the natural order that is desirable for book covers. this article discusses a process for parsing and manipulating the marc author strings to identify their various component parts and to create natural order strings. in particular, the article focuses on processing non-name information in author strings, such as titles that were commonly used in older works, e.g., baron or earl, and suffixes appended to names, e.g., “of bolsena.” relevant patterns are identified and a python script is used to manipulate the author name strings. by dianne dietrich background cornell university library has made 90,000 of its digitized books available as print-on-demand books through both its internal bookstore and amazon.com [1]. other libraries have also offered, or are preparing to offer, their digitized public domain works as print-on-demand books. a print-on-demand book is, essentially, a physical object created from multiple digital sources–the digitized version of the original physical object and the metadata from the online catalog. the final new physical book should maintain the logical consistency of the original physical book: all of the pages should be in order, and the author’s name on the front cover should look like the author’s actual name. this is reasonably easy to accomplish. our print-on-demand book candidates are all out of copyright and in the public domain, so most are quite old. some of these authors have titles attached to their names that aren’t commonly seen anymore, such as count, earl, and duchess. these authors are all in our catalog in an order that makes complete sense for cataloging our holdings, but not for displaying on the front cover of a book. in order to prepare our print-on-demand books, we had to create a comprehensive list of bibliographic identifiers, titles, and authors. we needed to identify the following fields for each author string–title, first name, middle name, last name, suffix, and contributor. it was not always immediately apparent from our catalog metadata which parts of the author string were associated with which of these fields. to begin the processes of formatting the metadata, we first extracted bibliographic information from our catalog. for each work we were making available, we pulled the necessary identifiers, title information, and the author string. some author names were in the form “surname, forename,” which made identifying their component parts straightforward. other names were less straightforward. a sampling of the more complex author strings (as they were stored in the appropriate column of our spreadsheet) is listed below: christina, of bolsena, saint liljencrants, johan, baron lennox, william pitt, lord stanhope, charles stanhope, earl collingwood, cuthbert collingwood, baron campbell, john campbell, baron iddesleigh, stafford henry northcote, earl of sandwich, edward george henry montagu, 8th earl of brown, william, of montreal george, william, of the st. paul bar the players, new york nathan, nathaniel, sir carriel, mary turner, mrs. hanson, john wesley, jr. university of california, berkeley. clarke, john, lecturer in university of aberdeen. monkswell, robert alfred hardcastle collier, 3d baron moulton, john fletcher moulton, 1st baron many of these strings have three component parts to them, so it was reasonable to assume that they were of the form “surname, forename, title.” the author strings for baron johan liljencrants and lord william pitt lenox (lines 2 and 3) follow this pattern. when we split these strings on the commas, we found ourselves with the following substrings. liljencrants johan baron lennox william pitt lord it was then simple to identify the component parts and reorder these substrings appropriately. the third substring was the title, and should be positioned before the forename and the surname. we ran into problems very quickly, though. many of the author strings with three component parts included a suffix, rather than a title. we could no longer assume that the third substring should be placed in front of the forename and surname. the author string for john wesley hanson, jr. (line 14) is an example of this. we soon realized that our task was much more complicated than compiling a list of suffixes and titles, assigning the third substring to the appropriate field, and positioning it accordingly. many of the author strings that had three component parts were unexpectedly complicated and required different parsing rules altogether. the author string for earl charles stanhope (line 4) was one example of an author string (with three component parts) that required extra attention. the three component parts of the string were not in the order we were expecting. stanhope charles stanhope earl when we assumed that this string followed a “surname, forename, title” pattern and tried to reorder it, the result was “earl charles stanhope stanhope.” in this case, the first “stanhope” was actually part of the title and is not a surname, as it was probably meant to read something like, “earl stanhope, charles stanhope.” saint names were even more complicated to reconstruct. one example is the author string for saint christina of bolsena (line 1). when we split the author string on commas, we had three distinct parts to the name. christina of bolsena saint again, when we assumed a “surname, forename, title” pattern, and then tried to reorder the string, we ended up with “saint of bolsena christina.” we clearly needed different rules for parsing this string, because in this case, while “saint” was a title, “of bolsena” was a suffix. manual reconfiguration of all of these author names was impractical, as we were dealing with tens of thousands of records. we decided to create a script capable of parsing as many author string variations as possible. we first identified the types of reconfigurations we were likely to encounter. next, we worked on developing the script to automatically handle string reconfiguration. as we inspected our results, we refined and adjusted the script to suit our needs. to start to understand the data we were working with, we isolated all of the author strings, and placed them in a separate file. (each author string was comma delimited, as in the list above.) we then printed out the contents of the third position of the string, as it was the position most likely to contain titles, suffixes, contributors, or other related words that would require special handling. we sorted our output and extracted the unique values in the resulting list. we accomplished this fairly quickly with a shell command. awk -f "," ' { print $3 } ' full_author_list.txt | sort | uniq [2] this list gave us a good sense of the variations we could encounter in our data, which we used as the foundation to build cases for parsing and formatting the author strings. as we continued to fine-tune the script, we viewed its output in excel, taking advantage of the filter options to identify outliers and special cases. figure 1: probable titles, suffixes, or other non-name words requiring special handling (as much as we might like, “librarian” probably shouldn’t be listed as a proper title, with “prince,” “rev.” and others.) the following sections describe the types of author string parsing we performed on our metadata. corporate names at the outset, it was important to distinguish between personal names and corporate names. since the spreadsheet we worked with did not capture full marc metadata, we had to develop a set of heuristics to use to identify corporate names. as we worked with the data, we identified a number of cases where it was highly likely that an author string was not a personal name. it included a geographic location, like a u.s. state name or abbreviation, or country name. it included words associated with government, like “government,” “congress,” or “senate.” it was an institution name, and included words like, “college” or “university.” the string in the “forename” field started with numbers, or resembled a date range. these heuristics weren’t foolproof, but they did help us identify many of the cases where we did not want to do any parsing of the author string at all. lords and ladies it was a relatively simple case when the author string followed the “surname, forename, title” pattern. consider the following two author strings (lines 2 and 3): liljencrants, johan, baron lennox, william pitt, lord each string was split on commas and stored in a python list as the variable authorstring. we wanted to correctly identify “baron” and “lord” as titles, while correctly identifying forenames and surnames and putting them in the correct order. to accomplish this, we used the following regular expression [3]: isbaron = re.compile('(.* )(saint|baron|lord|countess|baroness)(\.*)$', re.i).search(authorstring[2]) we then had the following three captured groups for each of the aforementioned author strings: 1 2 3 none baron none none lord none the three captured groups formed the title, and so we simply reorganized the string as “title forename surname.” the third captured group was for optional punctuation after the title. we captured any text preceding the title – in the first captured group – separately because, in some cases, it was crucial to mark the distinction between the first baron from the third, or the fifth. the following author strings for 3d baron robert alfred hardcastle collier and 1st baron john fletcher moulton (lines 17 and 18) are examples of this. monkswell, robert alfred hardcastle collier, 3d baron moulton, john fletcher moulton, 1st baron after running the above regular expression on these strings, we ended up with the following captured groups: 1 2 3 3d baron none 1st baron none when the script detected a value in the first captured group, it concatenated this value – usually an ordinal number – with the rest of the captured groups to form the full version of the title. we then capitalized only the first character of the second captured group. you might have noticed that if we transformed 1st baron john fletcher moulton’s author string as we described above, we would end up with “1st baron john fletcher moulton moulton.” in fact, we found that when we applied this rule to our entire list of author strings, many of the resulting strings had this exact error, a “duplicate” surname. last name hiccups as we saw earlier, some of the author strings we were working with were stored in such a way that, when we applied our formatting rules, the resulting string had one word duplicated. the author strings for earl charles stanhope and barons john campbell and cuthbert (lines 4, 5, and 6) provide us with examples of this case. stanhope, charles stanhope, earl collingwood, cuthbert collingwood, baron campbell, john campbell, baron when we used the formatting rule we described earlier, we ended up with the following author strings. earl charles stanhope stanhope baron cuthbert collingwood collingwood baron john campbell campbell these would look somewhat strange on the front cover of a book. instead of following the pattern “surname, forename, title,” these cases followed the pattern “word associated with title, forename surname, title,” where the surname was the same word as the “word associated with title.” to account for these cases, we set up a regular expression to match the value in the first position of the authorstring list. if it matched, we removed the duplicate surname from the forename [4]. tmp = r'(%s)' isduplicate = re.compile(tmp % authorstring[0]).search(authorstring[1].lstrip(' ')) if isduplicate: title = "%s%s" % (isbaron.group(1).lstrip(' '), isbaron.group(2).capitalize()) firstname = re.compile(isduplicate.group(1)).sub('',authorstring[1].lstrip(' ')).rstrip(' ') lastname = authorstring[0] we could then reorder the author string and display the names properly. dukes of earl some authors were countess or baron of a particular place. the author strings for stafford henry northcote and edward george henry montagu (lines 7 and 8) were examples of this case. (in montagu’s case, there is a number and place associated with his “earlness.”) iddesleigh, stafford henry northcote, earl of sandwich, edward george henry montagu, 8th earl of the pattern here was the same, “words associated with title, forename surname, title.” the difference between this case and the one that came before was that there was no redundant information–no “duplicate” surnames. the “words associated with title” needed to be placed with the title, to create the complete author name. to reformat these author strings, we used the following regular expression and code. isbaronof = re.compile('(.* )(baron|lord|countess|baroness) (of|de|von|de|d\'|di)$', re.i).search(authorstring[2]) if isbaronof: firstname = authorstring[1].lstrip(' ') lastname = '%s%s %s %s' % (isbaronof.group(1).lstrip(' '), isbaronof.group(2).capitalize(), isbaronof.group(3), authorstring[0]) this associated the component parts of the author strings with the proper fields, and the strings appeared in the right order. stafford henry northcote, earl of iddesleigh edward george henry montagu, 8th earl of sandwich place names what about the authors who weren’t earls or barons of some place, but were simply, “of” that place? the author strings for william brown and william george (lines 9 and 10) both included place information in addition to the personal name. brown, william, of montreal george, william, of the st. paul bar in the previous cases, we saw that when the string in the third position of the author string ended in some form of the word “of,” it was often part of a title. in this case, when the string in the third position started with “of,” these strings were suffixes, and were placed after the forename and surname. the following code identified this case and parsed the author name accordingly. isof = re.compile('of (.*)$', re.i).match(authorstring[2].lstrip(' ')) elif isof: firstname = authorstring[1].lstrip(' ') lastname = authorstring[0] suffix = isof.group(0) miscellaneous saints finally, in some cases, we didn’t have a “surname” at all. the author string for saint christina of bolsena was an example of this. christina, of bolsena, saint in this case, this string matched the isbaron regular expression, because it resembled the pattern “surname, forename, title.” we added the following code to check if the string that typically denotes the “forename” started with some form of the word “of,” indicating it was not a true forename, but rather a suffix. any substring that matched this pattern was reassigned to the suffix field so that the full author string could be reorganized appropriately. if authorstring[1].lstrip(' ').startswith('de') or authorstring[1].lstrip(' ').startswith('of'): title = "%s%s" % (isbaron.group(1).lstrip(' '), isbaron.group(2).capitalize()) firstname = authorstring[0] lastname = authorstring[1].lstrip(' ') re-imagining our process when we originally exported all of our bibliographic metadata, we did not include marc fields, subfields, or delimiters. after we examined our cases, we realized that if we had the marc metadata, some of our processing would have been simpler. identifying corporate author names would have been far simpler if we pulled the marc field out of our catalog. it is rather easy to flag the corporate author strings in this list: 100 1_ |a brown, william, |c of montreal. 100 1_ |a george, william, |c of the st. paul bar. 110 2_ |a the players, new york. 111 2_ |a continental congress for economic reconstruction |d (1933 : |c washington, d.c.) fields 110 and 111 denote “corporate name” and “meeting name,” respectively. our script would not have to use our “best guess” for determining corporate names if we had this information at the outset. we were also hesitant to strip out dates from all author strings because we did not have a foolproof way of distinguishing between corporate and personal names. if we had the marc fields, we could have easily removed the subfield $d from the personal names only. with marc metadata, we would have been better able to identify those author strings that don’t follow the “surname, forename” pattern. to correctly parse “saint christina of bolsena,” remember that we had to test what we thought was the “forename” to see if it started with the word “of” (or “de”). that was our only clue that we had a name that wasn’t in inverted order. if we had the marc first indicator along with the author string, we would have known for sure that we were encountering a special case. 100 0_ |a christina, |c of bolsena, saint, including full marc metadata would not have eliminated the need for a custom script, but it is clear to see how it may have helped us handle certain cases. script the complete python script is available at amazon pod processing script. a file of the sample author names discussed in this article and designed to accompany the code is also available. conclusion our goal was to create a script that would automatically process the majority of predictably formatted author strings so that only a small percentage would have to be corrected manually. we still conducted a human review of the metadata before we sent it out, and we estimated that we needed to hand-correct no more than 5% of the metadata after running the script. thank you fiona patrick, bronwyn mohlke, and liz muller provided invaluable help deciphering marc metadata, identifying cases, and refining the results of this script. notes [1] bill steele. “cornell and amazon.com join to resurrect 90,000 rare books via print-on-demand.” cornell chronicle online (february 26, 2009) http://www.news.cornell.edu/stories/feb09/amazonpod.ws.html [2] it is also possible to get all of the unique values by using the “text to columns” feature in excel and then filtering the results on the third column. [3] for the sake of brevity and clarity, only a few of the strings we used to match this case will be in the code snippet. see the full script and the appendix for the full list of strings. [4] it seems overkill to do the surname like this, but the first naive matching inexplicably chopped off too much of certain authors’ names (see “sir nathaniel nathan,” line 12), so a more rigorous solution was needed. [5] place names that are also common personal names have been deliberately excluded from this list. about the author dianne dietrich (dd388 at cornell dot edu) is the research data & metadata librarian at cornell university. appendix the following tables include lists of the strings that we used in our script to match the various cases. these lists were generated as we developed our script using our own metadata, and will, most likely, be expanded when necessary. corporate names strings we used to test for the presence of a corporate (not personal) author string [5] state abbreviations association atlanta boston college confederate congress democratic england florida great britain halifax massachusetts missouri new york ontario pennsylvania republican rhode island united states university washington d.c. ala. ark. ariz. calif. colo. conn. del. fla. ga. ill. ind. kan. ky. la. me. mass. md. mich. minn. mo. miss. mont. n.c. n.d. neb. n.h. n.j. n.m. nev. n.y. okla. ore. pa. r.i. s.c. s.d. tenn. vt. va. wash. wis. w.va. wyo. lords and ladies titles (generally found without “of” attached) abbe abp. baron barone baroness bp. cardinal chaplain colonel compte comtesse conte contessa count countess dame duc earl father freiherr graf hon hrabe kniaz lady lord mme. mrs. prince princess rev. saint sir viscount viscountess dukes of earl titles (generally found with “of” attached) variations of “of” apb. baron bishop bp. compte comte conte conti count countess duc duchess duchesse duke earl freifrau freiherr furst graf grafin lady marchioness marquess marquis marquise prince vicompte vicomte viscount d’ de de di of von subscribe to comments: for this article | for all articles 2 responses to "automated metadata formatting for cornell’s print-on-demand books" please leave a response below: dorothea salo, 2009-11-23 why do regex matching instead of string matching for the duplicate-surname situation? dianne dietrich, 2009-11-23 an earlier version of the script handled the duplicate-surname situation using string matching. when we reviewed the results, there were cases where that code inadvertently garbled certain author strings, which is why i decided to try using a regex. it worked — that is, it didn’t garble strings the way my first attempt did — and so i kept the code as it was. that said, i’m sure it’s possible to accomplish the same thing using only python string operations. leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – creating filtered, translated newsfeeds mission editorial committee process and structure code4lib issue 10, 2010-06-22 creating filtered, translated newsfeeds google translate’s api creates the possibility to leverage machine translation to both filter global newsfeeds for content regarding a specific topic, and to aggregate filtered feed items as a newsfeed. filtered items can be translated so that the resulting newsfeed can provide basic information about topic-specific news articles from around the globe in the desired language of the consumer. this article explores a possible solution for inputting alternate words and phrases in the user’s native language, aggregating and filtering newsfeeds progammatically, managing filter terms, and using google translate’s api. by james e. powell, linn marks collins, mark l. b. martinez; knowledge systems and human factors team, los alamos national laboratory research library introduction the first decade of the 21st century presented many challenges, including the threat of a deadly global pandemic. in late 2003, sars suddenly appeared in china and southeast asia, later spreading as far as canada, claiming over 800 lives, and significantly impacting the financial wellbeing of the region. next, bird flu re-emerged, simmering in numerous hotspots around the globe, but as of yet, failing to erupt into a global killer. just as health authorities around the globe had finished updating their pandemic flu plans to address a feared bird flu pandemic, yet another highly communicable and sometimes deadly respiratory virus emerged from mexico. initially dubbed swine flu, the disease eventually did make its way around the globe, but failed to fulfill the anticipated apocalyptic vision of a deadly killer on par with the spanish flu of 1918. nonetheless, many of our worst fears were confirmed regarding the rapidity with which a new pathogen can spread among geographically separated regions of the world. in the 21st century, the world is a much smaller place indeed, at least for an infectious human respiratory virus. it was also a decade that saw many news organizations, both large and small, move more of their presence online. today it is rare for any news outlet not to have a presence on the web, via twitter, facebook or various other sites enabled by the increasing ubiquity of network technology. this information overload has fueled the popularity of content formats that encourage aggregation and rapid consumption, such as rss (really simple syndication). rss and its close cousin, atom, are structured information syndication formats. they are often used to aggregate news stories from individual content providers such as magazines or newspapers. a newsfeed can be thought of as a temporal scrolling window that provides a view of a collection of news – older articles disappear from the window as newer headlines replace them. the feed includes both collection-level metadata (what publication is the source of the headlines), syndication information (frequency of update, a timestamp for this instance of the feed), and item-level information, including an article title, description and link. a newsreader application can be configured by a user to maintain a set of feeds of interest and refresh that list as needed based on user preferences and syndication information included within the feed. feeds can also be embedded in portals and collaboration environments, to be shared by groups of users. in summer, 2009, the los alamos national laboratory research library undertook an effort to develop a prototype information aggregator for content related to the h1n1 flu [1]. various news sources were identified and integrated using the drupal collaboration framework [2]. in some cases, there was content that we wanted to integrate with the framework which was not available in a format which could be easily integrated with drupal. since our preferred mechanism for integrating content was to use drupal’s news aggregation capabilities, we developed rss wrappers for a couple of sources, including an internal source – our own bibliographic metadata catalog, oppie, which houses over 95 million peer-reviewed science and technology articles, spanning over a century. part 1: generating a newsfeed programmatically with rome as one might expect, there are plenty of programming libraries for generating and working with news feed content. for java developers, one of these tools is rome. rome is described by its developers as a “set of parsers and generators for the various flavors of syndication feeds, as well as converters to convert from one format to another.”[3] it supports eight different variants of rss, as well as two versions of the newer atom format. rome enables rapid development of java applications that can work with, or produce newsfeeds. the following use of rome illustrates one of the libraries’ core capabilities – conversion between feed formats. the following code takes the uri of a newsfeed as input, and given a format parameter that indicates the output format, produces a comparable feed which uses a different structure. in this example, the input feed (stored in a file on the filesystem) is in rss 2.0 format, and the output is in atom 1.0 format: string outputtype = “atom_1.0”; string feedfilename = “samplefeed.xml”; file feedfromfile = new file(feedfilename); syndfeedinput input = new syndfeedinput(); xmlreader feedreader = new xmlreader(feedfromfile); syndfeed feed = input.build(feedreader); feed.setfeedtype(outputtype); syndfeedoutput newfeed = new syndfeedoutput(); newfeed.output(feed,new printwriter(system.out)); line 5 loads the feed contained in a file called samplefeed.xml, and line 6 builds a syndfeed object from the contents of the file. the feed object is constructed from an xml representation of the retrieved feed. syndfeed provides a java bean interface for exploring, manipulating, and generating a news feed independent of the output format. once you’ve loaded a feed into a syndfeed object, you can request access to specific feed-level metadata (such as the feed title, author, or description) or iterate through the entries in a feed, which are stored in a java list object called entries, which means that a simple request such as feed.gettitle() is all that’s needed to retrieve a feed title once it has been loaded and parsed. the following code iterates through a set of feed entries and outputs a simple html table of the items: iterator entryiterator = feed.getentries().iterator(); while (entryiterator.hasnext()) { string anentry = ((syndentryimpl) entryiterator.next()).gettitle(); system.out.println(anentry); } the utility of a development framework like this became more apparent when we were faced with the task of generating an rss feed from search results. the oppie system relies exclusively on the apache solr search engine [4]. bibliographic metadata is indexed using solr and the underlying lucene libraries, and user queries are handled by solr as well. so it was a simple matter to convert a search result set to a newsfeed, since the two share common characteristics. we developed java code to retrieve a results set, parse, and truncate it, then used rome to generate a newsfeed version of the content. the following request returns 122 results, encoded in solr’s xml result set format: /solr/select/?rows=500&q=((neutrino%20detector)%20 and%20(los%20alamos%20national%20laboratory)) an excerpt from the result set follows: <?xml version="1.0" encoding="utf-8"?> <response> <lst> <int>0</int> <int>862</int> <lst> <str>((neutrino detector) and (los alamos national laboratory))</str> <str>500</str> </lst> </lst> <result numfound="122" start="0"> <doc> <str>zimmerman, e.d.</str> <str>nuclear physics b, proceedings supplements (july 2003) vol.123, p.267-71</str> <str>boone has begun</str> <str>info:lanl-repo/inspec/7814871</str> <date>2008-09-02t11:43:03.742z</date> </doc> <doc> <str>dalton, c.</str> <str>report no. la-8988-ms, 1981-09-01</str> <str>electronics for the lampf neutrino experiment`s veto counter system</str> <str>info:lanl-repo/ecd/6001286</str> <date>2008-09-11t00:42:58.843z</date> </doc> ... </response> the first <str> element, which occurs before the <result> element, and has an attribute name with the value q, provides the original query string used to generate this result set. the next useful piece of information is the numfound attribute for the <result> element. in our test application, we set a threshold of 25 results, meaning that we only map the first 25 items in the result set to a newsfeed representation. this threshold value later became a user definable parameter for a servlet implementation of the newsfeed wrapper application. following that, we iterate through the child <doc> elements of the result element, to retrieve information for each entry in the generated newsfeed. the application uses xpath statements to retrieve the value of various elements in the result set, which was retrieved using the apache httpclient library [5]. these xpath statements incorporate a replaceable character string which serves as a stand-in for the specific element within the result set we were attempting to retrieve: private string solrnumbresultsxpath = "/response/result/@numfound"; private string titlesxpath = "/response/result/doc[_q_]/str[@name='displaytitle']"; private string authorsxpath = "/response/result/doc[_q_]/str[@name='displayname']"; private string descriptionsxpath = "/response/result/doc[_q_]/str[@name='displaysource']"; private string idsxpath = "/response/result/doc[_q_]/str[@name='recid']"; private string datesxpath = "/response/result/doc[_q_]/date"; as the application processed the result set, the string “_q_” was replaced with an integer value from a counter as we worked our way through the doc element and child elements in the result set, until we hit our threshold value: for (int count=1; count<=25; count++) { string xpathindexpattern = "_q_"; pattern titleregexppat = pattern.compile(xpathindexpattern); matcher titlematcher = titleregexppat.matcher(titlesxpath); string actualtitlexpath = titlematcher.replaceall(integer.tostring(count)); string title = domprocessing.executexpath(solrresults, realtitlexpath);&#91;/sourcecode&#93; the external class domprocessing has a method called executexpath, which accepts two arguments – a string containing the solr result set, and an xpath that points at the value of a specific element within that result set, returning the value of that element – in this case, the title of one of the items in the result set, as a string. finally, we build instances of the entry class and add them to a list of entries, which become part of the syndfeed object. the last step is trivial and merely requests that the syndfeed class output a serialization of its contents in one of the available newsfeed formats, e.g. rss 2.0. the following code excerpts highlight the steps required to generate and output a newsfeed for the result set: &#91;sourcecode language='java'&#93; syndfeedimpl solrresultsfeed = new syndfeedimpl(); ... list resultentries = new arraylist(); solrresultsfeed.settitle(solrquerystring); solrresultsfeed.setdescription(solrquerydescr); solrresultsfeed.setlink(solrresultsfeedlink); syndentryimpl aresult = new syndentryimpl(); for (int count=1; count<=25; count++) { ... aresult.settitle(title); resultentries.add(aresult); aresult = new syndentryimpl(); } solrresultsfeed.setentries(resultentries); solrresultsfeed.setfeedtype("rss_2.0"); ... syndfeedoutput solrresultsfeedoutput = new syndfeedoutput(); solrresultsfeedoutput.output(solrresultsfeed,new printwriter(system.out));&#91;/sourcecode&#93; when all this code is merged and executed, the results are an rss 2.0 newsfeed containing up to 25 entries from the original solr result set. once we tested the stand-alone tool for mapping solr results to rss, we converted it to a parameterized servlet which accepted a query string, threshold value, and a newsfeed format type string as input parameters, enabling any query to be retrieved, trimmed, and mapped to the specified newsfeed format via a url request, e.g., <pre> /rsswrapper/solr_res_cvt?q=((neutrino%20detector)%20 and%20(los%20alamos%20national%20laboratory)) &amp;numresults=15&amp;feedformat=atom_0.3</pre> <h2>part 2: using google translate to generate filtered, translated newsfeeds</h2> while the h1n1 news aggregator and the oppie newsfeed wrapper were under development, library staff were tasked with identifying other potential sources of information for the service. by that time, h1n1 had already made its presence felt on several continents, and news outlets in europe and asia included articles documenting the latest suspected cases of swine flu, as well as rumors about the progress towards developing a vaccine. some staff were able to identify articles at non-english news sites which were related to the new flu strain. once candidate articles were located, google translate [<a id="ref6" href="#note6">6</a>] could be used to translate the contents of the article. as suspected, these articles often provided local details about the spread of the disease in their respective source countries, and usually included details that were not reported in english language outlets in the united states. foreign news sources represented a potentially rich source of additional information about the spread of h1n1, if users could read them. the utility of the google translate service led to an exploration of google's translation api, to enable automatic translation of newsfeeds, and to the development of a tool for selective filtering of this content as well. the rome api makes it easy to import and transform an existing newsfeed, as illustrated in the previous section. it enables an application to iterate through the individual entries contained in a feed, and to locate the various components of each entry. in this case the entry titles were of particular interest, since we felt that if we could translate these and generate a feed in the preferred language of h1n1 news aggregator users, then it would enable us to identify sources that frequently reported important regional information about the spread of the swine flu. in fact, if our version of the wrapper application that used google translate service proved to be effective at handling newsfeed content, then the next step would be to filter the feeds so that only the entries of interest would be displayed to users. the resulting application did turn out to have some utility, but we discovered some unexpected challenges along the way. google translate is based on a statistical machine translation model. statistical machine translation relies upon the existence of a sort of rosetta stone of matching corpora in different languages. google relies upon a collection of united nations documents, which have been meticulously translated into a number of the world's languages. exact details are not published for google translate, but in general terms, statistical machine translation relies upon matching passages and co-occurrences of words between respective corpora. despite the fact that such a method results in no real comprehension of the content by the translation software, both anecdotal reports from google, and industry accepted measures for machine translation (e.g. ibm bleu [<a id="ref7" href="#note7">7</a>]) have shown that statistical machine translations can yield excellent, readily comprehensible results for human users when translating between supported languages. indeed we often found that google translate, when used manually via the translate.google.com web page, provided an adequate translation of newsfeed titles. so this led us to explore google's ajax language api, which includes a rest-based web service that can be used to translate strings of text between languages [<a id="ref8" href="#note8">8</a>]. this toolkit uses the same machine translation service used by google translate. it is specifically designed to support javascript-based machine translation of web content, but it includes services which can be called by applications written in other languages, such as java. the prototype feed translation wrapper has much in common with the oppie solr feed utility. it uses the apache httpclient library to retrieve the contents of a specified news feed, accepts the url of a feed as a parameter, and uses rome to generate the output feed in a format specified using another request parameter. what sets it apart is that it makes external calls to the google language api to translate each feed entry title and description. we also decided to implement a simple string filter that would be used to determine whether a particular item ought to be included in the resulting feed, so that a custom feed containing only entries that match a particular word or phrase would be generated. so the translation wrapper first requests a translation of the filter term, and then requests translations of the entries, makes a comparison, and adds translated title and descriptions for matching entries to the entry list object which becomes part of the generated feed. the following is an example request to the feed translation service: <pre>/rsswrapper/rssctr?feed=http://www.lemonde.fr/rss/sequence/ 0,2-3244,1-0,0.xml&amp;amp;amp;amp;lang=fr&amp;amp;amp;amp; q=flu&amp;amp;amp;amp;qtrans=y&amp;amp;amp;amp;format=rss_1.0</pre> the feed parameter is the url of the target input feed. the lang parameter is the identifier for the language of the input feed. the q parameter is the filter term that will be used to determine whether a particular entry should be included in the translated feed. the qtrans parameter was added after some initial experiments showed that the translation api did not always yield good results for filter term translation, at least in the case of queries associated with swine flu. the phrase “swine flu” or even the word “flu” is not always translated in such a way that it matches the actual word or phrase used by target news sources. for example, in japanese, flu is translated to the katakana-borrowed word “infurenza” by google's software, and yet the abbreviated katakana word “infuru” was almost exclusively used in the news sources we tested. in other cases, google's translation for the filter term yielded decent results. so we modified our tool to optionally allow a knowledgeable user to include a translated filter term, and turn off filter term translation via the qtrans parameter flag, enabling optimal per-feed configurability. finally, our tool also allows for the target feed to be of a different format than the source feed. the following example illustrates how one entry in a french news feed was identified, matched against the filter term, and translated using the google api: original feed entry title: vaccins anti-grippe a : l'etat débourserait 48 millions d'euros original feed entry description: les négociations engagées par le gouvernement avec les trois laboratoires chargés initialement de fournir à la france 94 millions de doses de vaccin anti-grippe a sont en passe d'être conclues.clues. title translation request: <pre>http://ajax.googleapis.com/ajax/services/language/translate? v=1.0&amp;amp;amp;amp;langpair=fr%7cen&amp;amp;amp;amp ;q=vaccins+anti-grippe+a+%3a+l%27etat+%c3%a9bourserait+48+millions+d%27euros</pre> translation response: <pre> {"responsedata": {"translatedtext":"a flu vaccine: the state would pay 48 million euros"}, "responsedetails": null, "responsestatus": 200}</pre> description translation request: <pre>http://ajax.googleapis.com/ajax/services/language/translate? v=1.0&amp;amp;amp;amp;amp;langpair= fr%7cen&amp;amp;amp;amp;amp;q= les+n%c3%a9gociations+engag%c3%a9es+par +le+gouvernement+avec+les+trois+laboratoires +charg%c3%a9s+initialement+de+fournir+%c3%a0 +la+france+94+millions+de+doses+de+vaccin+anti-grippe +a+sont++passe+d%26%2339%3b%c3%aatre+conclues.</pre> translation response: <pre> {&quot;responsedata&quot;: {&quot;translatedtext&quot;:&quot;negotiations with the government by the three laboratories initially provide france with 94 million doses of flu vaccine are going to be concluded.&quot;}, &quot;responsedetails&quot;: null, &quot;responsestatus&quot;: 200}</pre> as you can see, generating an appropriate request to the google ajax language api is relatively straightforward. full capabilities of the service are documented online [&lt;a id=&quot;ref8&quot; href=&quot;#note8&quot;&gt;8&lt;/a&gt;]. there are only three parameters for a basic translation request to their rest-based translate service: the version parameter (v), the language pair parameter (langpair) which uses two character language codes separated by a url encoded vertical bar character (%7c), and a query parameter (q), which is assigned the url encoded value of the string to be translated. google translate returns its response using the json format, which is widely used in javascript applications, but trivial to parse using java string manipulation methods or java json parsing libraries. the only additional required code is that which determines whether or not an entry title or description matches the filter term, which we implemented using java's indexof method: [sourcecode language="java"] if ((origtitle.indexof(translatedqueryterm)>-1) || (origdescription.indexof(translatedqueryterm)>-1)) { the following channel content is an example of the type of news feed that would be generated by our translation service: <rdf:rdf xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns="http://purl.org/rss/1.0/" xmlns:sy="http://purl.org/rss/1.0/modules/syndication/" xmlns:dc="http://purl.org/dc/elements/1.1/"> <channel rdf:about="http%3a%2f%2fwww.lemonde.fr%2frss%2fsequence%2f0%2c2-3244%2c1-0%2c0.xml"> <title>planète le monde.fr (planet the monde.fr)</title> <link><!&#91;cdata&#91;http://www.lemonde.fr/rss/sequence/0,2-3244,1-0,0.xml&#93;&#93;></link> <description><!&#91;cdata&#91;translation of: "http://www.lemonde.fr/rss/sequence/0,2-3244,1-0,0.xml"&#93;&#93;></description> <copyright>research library, los alamos national laboratory</copyright> <sy:updateperiod>weekly</sy:updateperiod> <sy:updatefrequency>1</sy:updatefrequency> <sy:updatebase>2010-04-23 13:22:40</sy:updatebase> <items> <rdf:seq> <rdf:li resource="http://www.lemonde.fr/planete/article/2010/03/23/vaccins-anti-grippe-a-l-etat-debourserait-48-millions-d-euros_1323036_3244.html#xtor=rss-3244"/> </rdf:seq> </items> </channel> <item rdf:about="http://www.lemonde.fr/planete/article/2010/03/23/vaccins-anti-grippe-a-l-etat-debourserait-48-millions-d-euros_1323036_3244.html#xtor=rss-3244"> <title><!&#91;cdata&#91;a flu vaccine: the state would pay 48 million euros (french)&#93;&#93;></title> <link><!&#91;cdata&#91;http://www.lemonde.fr/planete/article/2010/03/23/vaccins-anti-grippe-a-l-etat-debourserait-48-millions-d-euros_1323036_3244.html#xtor=rss-3244&#93;&#93;></link> <description><!&#91;cdata&#91;negotiations with the government by the three laboratories initially provide france with 94 million doses of flu vaccine are going to be concluded.&#93;&#93;></description> <dc:date>2010-04-23 13:22:40</dc:date> </item> </rdf:rdf> conclusion in an increasingly multilingual world where comprehending information in a language you do not speak may be a matter of life or death, machine translation services can be an essential tool in maintaining comprehensive awareness of news provided by outlets around the world. our prototype feed translation application enables us to generate custom filtered newsfeeds programmatically, with a modest amount of effort on the part of information professionals required to identify appropriate news sources and tune the service parameters to ensure that news entries are properly identified and mapped over to the generated feed. machine translation services are just starting to mature and since we developed this tool, google has incorporated their machine translation services into their chrome browser [9]. users can now set a preferred language for browsing, and have content transparently and automatically translated as they surf the web with chrome. our solution offers a similar level of automation and transparency to consumers of newsfeeds, and proved to be very straightforward to implement using the rome java newsfeed api and google’s ajax language api. we believe that tools like this fill an essential role in support of critical information awareness. about the authors james powell and mark martinez are members of the knowledge systems and human factors team (http://www.lanl.gov/ks-hci/) at the research library at los alamos national laboratory (lanl), los alamos, new mexico, us. linn collins is the team leader. the team works with scientists and engineers at lanl to leverage advances in information science and technology in support of scientific research and global security. notes [1] linn collins, mark martinez, james powell, jorge roman, and michelle garcia. semantic components of e:sos: emergency situation overview and synthesis. poster presented at department of homeland security science and technology summit, south central region, february 9, 2010. [2] drupal: http://drupal.org/ [3] project rome: https://rome.dev.java.net/ [4] apache solr: http://lucene.apache.org/solr/ [5] apache httpclient: http://hc.apache.org/ [6] google translate: http://translate.google.com/# [7] bleu: a method for automatic evaluation of machine translation: http://domino.watson.ibm.com/library/cyberdig.nsf/1e4115aea78b6e7c85256b360066f0d4/5c651a88cb24938185256acb0055e548?opendocument [8] google ajax language api: http://code.google.com/apis/ajaxlanguage/ [9] google chrome: http://www.google.com/chrome subscribe to comments: for this article | for all articles one response to "creating filtered, translated newsfeeds" please leave a response below: jodi schneider, 2010-06-23 another approach to filtering feeds–without the emphasis on machine translation, but with attention to 4 facets (topics, people, sources, time)–may be of interest: feedwinnower, chi 2010: http://dx.doi.org/10.1145/1753326.1753466 blog post about it: http://asc-parc.blogspot.com/2010/04/information-stream-overload.html leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – what’s new? deploying a library new titles page with minimal programming mission editorial committee process and structure code4lib issue 35, 2017-01-30 what’s new? deploying a library new titles page with minimal programming with a new titles web page, a library has a place to show faculty, students, and staff the items they are purchasing for their community. however, many times heavy programing knowledge and/or a lamp stack (linux, apache, mysql, php) or apis separate a library’s data from making a new titles web page a reality. without it staff, a new titles page can become nearly impossible or not worth the effort. here we will demonstrate how a small liberal arts college took its acquisition data and combined it with a google sheet, html, and a little javascript to create a new titles web page that was dynamic and engaging to its users. by john meyerhofer a new titles list is a great way for libraries to show off their new items, to demonstrate to faculty their requests are being purchased, and provide a tool for students to find new items at the library. in the physical library, a common tool for highlighting new titles is a book display while in the virtual world, a web page can achieve a similar result. however, many times lack of programming knowledge often prohibits libraries and librarians from implementing one. intimidating descriptions of lamp stacks, with php scripting and mysql databases can often scare even the most technical librarian from attempting to build their own deployment. currently most opac vendor systems don’t include viable options for deploying a new titles list either. often times utilizing these programming tools means being proficient with extracting data from a catalog, importing that data into a database, and finally developing a web application to read, display, and maintain that data. surprisingly this doesn’t have to be the case. at macalester college, we were able to develop a relatively robust new titles web page with only some basic opac data, a google sheet, html, and a little javascript. in order to arrive at our end result, it’s important to look at the history of the previous new titles page and how it was structured and developed. we’ll also examine the reasons for moving to a simpler version. finally, we’ll walk through the development of the new process and detail a typical procedure to update the new title page with new title data. through these steps, we hope that many libraries will see this as a starting point to developing their own new titles list or re-examining their existing one. history at macalester college, over the previous few years we had developed a new titles web page that used php to access new titles data in a mysql database. this page was developed in-house and utilized a library server which was outside the traditional realm of the institutional servers and thus was lightly supported by macalester’s information technology services (its). this support meant there were backups of the server but no updates to the operating system or any of its applications. the existing process for updating the new item data involved first exporting data from oclc’s worldshare using the wms report designer. next, we took this data, which was in a tab-delimited file, and imported it into a mysql database using a php script. this script did some text conversion as well as retrieved the new titles’ cover image. this process had worked well for the library during a one to two year timeframe with minor changes and enhancements. historically the dewitt wallace library doesn’t purchase a large amount of new materials. typically we see about two hundred and fifty new titles in a month with an average of three to four thousand in a year. the number of titles each week fluctuates during the academic year with the spring having more than the other seasons. this fluctuation is of course a result of many factors but like many academic libraries, static budgets and falling circulation have affected purchasing. with these kind of numbers throughout the year, updates to the new title page traditionally occur every other week. finally, new item records are given a two to three day window for processing to ensure items on the web page are actually in the library. reasons for change although the existing new titles page was working and satisfied our needs, we began to re-assess it due to several factors. first, and most important, was that its had informed us our server’s operating system was no longer going to be supported and thus was slated for retirement. without support, we could no longer depend on the web pages and applications that were on that server. besides the operating system, individual software programs were extremely old, too. in particular, php and mysql, main components in the existing new title list, were several versions behind current acceptable ones. without having updates to the operating system and main software running the new titles page, the library and the college were exposed to security risks and vulnerabilities. with the impending retirement of our server, we needed a new home for the new titles page. initially, there were many ideas entertained, including looking at another lamp stack deployment or even a cloud based solution. however, to determine a more practical solution, we focused our criteria for our new titles page. first, ideally we wanted the new titles page to be integrated with our campus content management system (cms). the existing new titles page, which lived outside our cms, would have required extensive rework to make it match the campus template and be mobile ready. there are many benefits to integrating with our cms (dotcms). these include gaining the benefit of standard styling of the web page as well as instantly having the same look and feel of our existing library web page. being a part of the cms also gave us the security of backups, file versioning, and being under the campus’ umbrella of supported technologies and environments. we were also lucky in that our cms allowed us to deploy pages with javascript. our second criteria was a need for the new titles page to dynamically display data with pagination and filtering. these are features our users were accustomed to with the existing page. third, we needed a place to store our new titles data that would allow us to quickly update and add items when new titles were purchased. in the past this was a mysql database, but without this option, we wondered if there were other technologies that might be utilized that could satisfy, or at least mimic, a database’s functionality. our final criteria was a need for the page to be fast, automatically scale with the addition of new item data, and be easy to maintain. speed was paramount as most users won’t wait even a few seconds for a page to load. scaling with the addition of data would be a benefit and ultimately help make the page easy to maintain. as with most projects that have any amount of system data, some manual work might be required, but keeping it to a minimum would help create a solution that was simple and easy to sustain. with all of these criteria combined, it was hard to imagine a solution that didn’t involve programming. awesome table our first attempt, but ultimately a failed one, was to utilize the awesome table google sheet template. this was a fairly robust solution where data was housed in a google sheet and then included within a web page via an html iframe tag. the tool came with filtering, searching, and pagination and seemed like a perfect fit, except that during our testing typical load times for the page averaged 12 seconds. figure 1. awesome table progress bar while loading (enlarge) while loading the data from the google sheet, the tool did display a small blue progress bar to show that it was processing, however, the 12-second delay was too long and we decided to look elsewhere for a solution. a solution presents itself after working with the awesome table product in which a google sheet was used to store data, housing the new titles data in a google sheet seemed like a solution to many of our proposed criteria. housed in a spreadsheet, the data could easily be updated and new rows added, therefore addressing our criteria of maintainability. however, as we discovered with awesome table (or even researching other tools like tabletop.js), the google sheet api, as others have pointed out, may not be production worthy. there is concern that google could put rate limits on access, may be adding extra load to the display, and with a publicly accessible google sheet, an ill-intentioned person could traceback the api calls to access the data. the epiphany in finding a solution came when realizing that the google sheet housing our data could be used as a tool to create the html display as well. if we create html code that could then be copied into a cms-housed page, although not automated, it could be a simple process to update the new item data and thus populate our new items page. to achieve our vision, we needed a client-side tool that would create the pagination and filtering. a javascript tool seemed like it might be the best fit for our needs. running locally meant it would likely be faster, and implementation would likely be easier as our cms allowed us to implement javascript code. this line of thought led us to examine the list.js tool. list.js is a javascript file that can add searching, filtering, and sorting to an html item list or html table. combined with the pagination plugin, list.js ultimately satisfied our criteria of integration with our cms, pagination, and filtering for our new titles page. after deciding to use the list.js tool, the development path became fairly straightforward. first we would need to create an html page with the list.js and pagination plugin. next, we would get each item’s book cover image, which was not part of our data extract. third, we would need to concatenate all of the data into an html title list that could be part of an include file. finally, we would include that file into the html page to dynamically generate our new titles list. in the end we would have a google sheet that houses all of the data, an html file that lives in our cms which displays the data, and an include file that contains the html new titles list which is dynamically included into the main html file. in this way we would compartmentalize the data, display, and include file. create the html file to start, we built an html page that formatted a basic set of test data which allowed us to fine tune the display. utilizing our campus pattern library, we quickly created an html item list of eight items that had book cover images, each to the left of the tile and author. next, we integrated the list.js and its pagination plug-in into the page, allowing us to test the filtering by item type (i.e. book, dvd, etc.) and purchase department (i.e. chemistry, english, etc.). the filtering was accomplished by including these data values in the html with hidden tags. even when hidden, list.js is able to use this data for filtering based on user choices. final html file <script src="list.min.js"></script> <script src="list.pagination.min.js"></script> <style> .pagination li { display:inline-block; padding:5px; } .active a { font-weight: bold; text-decoration: none; pointer-events: none; cursor: default; color: #ea6506; } </style> <div id="new-arrival-list"> <p> <form class="inline-form"> <!-label class="control-label" for="select_element_h">search for a title or author:</label> <input type="text" onfocus="clearform();" style="width:50%;" placeholder="search for a title, author, or department..." class="search" /> <br / --> <label class="control-label" for="select_element_h">select a department:</label> <select name="fund" class="fund" id="fund" onfocus="clearform();" onchange="listobj.search(this.value, ['dept']);"> <option name="all" value="all">select a department...</option> <option name="bame" value="american studies">american studies</option> <option name="bant" value="anthropology">anthropology</option> <option name="bart" value="art">art</option> <option name="balc" value="asian languages and cultures">asian languages and cultures</option> <!-option name="bbio" value="biology">biology</option --> <option name="bche" value="chemistry">chemistry</option> <option name="bcla" value="classics">classics</option> <option name="bdan" value="dance">dance</option> <!-option name="beco" value="economics">economics</option --> <option name="bedu" value="education">education</option> <option name="beng" value="english">english</option> <option name="benv" value="environmental studies">environmental studies</option> <option name="bfre" value="french">french</option> <option name="bgeg" value="geography">geography</option> <!-option name="bgel" value="geology">geology</option --> <option name="bger" value="german">german</option> <option name="bspa" value="hispanic studies">hispanic studies</option> <option name="bhis" value="history">history</option> <option name="bint" value="international studies">international studies</option> <!-option name="blat" value="latin american studies">latin american studies</option --> <option name="blin" value="linguistics">linguistics</option> <option name="bmat" value="mathematics and computer science">mathematics and computer science</option> <option name="bhum" value="media and cultural studies">media and cultural studies</option> <option name="bmus" value="music">music</option> <!-option name="bneu" value="neuroscience">neuroscience</option --> <option name="bphi" value="philosophy">philosophy</option> <option name="bphy" value="physics">physics</option> <option name="bpol" value="political science">political science</option> <!-option name="bpsy" value="psychology">psychology</option --> <option name="brel" value="religious studies">religious studies</option> <option name="brus" value="russian">russian</option> <option name="bsoc" value="sociology">sociology</option> <option name="bthe" value="theater">theater</option> <option name="bwom" value="woman/gender studies">woman/gender studies</option> </select> <br/><label class="control-label" for="select_element_h">or select a format:</label> <button class="button" type="reset" onclick="formatsearch('video_dvd');">dvd</button>&nbsp;&nbsp;<button class="button" type="reset" onclick="formatsearch('book');">book</button>&nbsp;&nbsp;<button class="button" type="reset" onclick="formatsearch('book_digital');">ebook</button>&nbsp;&nbsp;<button class="button" type="reset" onclick="formatsearch('music_cd');">cd</button> &nbsp;&nbsp;&nbsp;&nbsp; <button class="button orange-button icon-to-right icon-refresh" type="reset" onclick="clearform();">reset</button> </form> </p> <hr> <ul class="newslisting list clearfix"> #dotparse("/library/new/new_arrival_list.vtl") </ul> <h3><ul class="pagination"></ul></h3> </div> <script> var paginationoptions = { name: "pagination", paginationclass: "pagination", innerwindow: 2, outerwindow: 2, left: 1, right: 1 }; var listoptions = { valuenames: [ 'name', 'dept', 'type' ], page: 8, plugins: [ listpagination(paginationoptions) ] }; var listobj = new list('new-arrival-list', listoptions); //added for url parm passing of department //ex: http://www.macalester.edu/library/new/index.html?dept=american%20studies //ex: http://www.macalester.edu/library/new/index.html?dept=chemistry var urldept = getqueryvariable("dept") if (urldept != "") { listobj.search(urldept, ['dept']); $('select').val(urldept); } function clearform() { listobj.search(); $('select').val('all'); $('input:text').val(''); } function formatsearch(p1) { clearform(); listobj.search(p1, ['type']); } function getqueryvariable(variable) { var query = window.location.search.substring(1) var vars = query.split("?"); for (var i=0;i<vars.length;i++) { var pair = vars[i].split("=") if(pair[0] == variable){ if(pair[1].indexof('%20') != -1){ console.log(pair[1].indexof('%20')) var fullname = pair[1].split('%20') console.log(fullname) return fullname[0] + ' ' + fullname[1] } else { return pair[1]; } } } return(false) } </script> <p style="font-size:small;">the new arrival page displays the new items from <a href="http://www.macalester.edu/">macalester’s</a> <a href="http://www.macalester.edu/library/">dewitt wallace library</a> for the 2016-17 academic year. powered by <a href="http://www.listjs.com/">list.js</a> with images provided by <a href="https://www.oclc.org">oclc</a>. </p>? one thing to note is the “#dotparse(“/library/new/new_arrival_list.vtl”)” line of code which acts to include the separate html list items file. with this implementation, going forward to update the page with new items, we would only need to add new html list items to that file. here is an example of a final html item list code for one new item: <li> <img alt='cover' style='height:96px; width:72px;' class='boxed-imagesm left' src='http://coverart.oclc.org/imagewebsvc/oclc/+-+4626438_70.jpg'> <a class='name' target='_blank' href='https://macalester.on.worldcat.org/oclc/3891719'> the mongol world empire, 1206-1370<br/> by john andrew </a> <p class='dept' style='display: none;'>geography</p> <p class='type' style='display: none;'>book</p> </li> in our google sheet, using string functions, we would concatenate the data from each item with the required html tags to get an html item list code. the html file with the list.js and pagination code, and the file with the html item list code would both be housed in our cms. item image before we discuss how the new items cover image is retrieved, it should be mentioned that the dewitt wallace library at macalester college was able to work with oclc to secure approval to use their images in our new titles web page. the method we use is inline linking, or hotlinking. from our web page, we reference the url of oclc’s image. an image appears to be part of the new titles web page, but only virtually, in the sense that the image file is not physically present on a macalester server. the actual location of the image file is at oclc and no downloading or copying of any images occurs using this method. besides the cover image, most of the required information needed for the new items page is within our exported data. that exported data is kept in a google sheet and over time new item data is added to it. however, to get the item’s image we need to perform a different kind of search in a separate google sheet. first, we copy just the new items’ oclc numbers from the master google sheet into a column in a working google sheet. because the importxml function we use to get the image will re-request the information when a google sheet is opened, we perform the image search separately from our master google sheet. this ensures that as the number of new items increase in the master file, the google sheet isn’t calling that function again and again. =importxml("https://macalester.on.worldcat.org/oclc/" &a1, "//img[@alt='cover art image']/@ng-src") we use the importxml function to extract the image url from each item’s web page. importxml is essentially opening the new item’s oclc web page and using the xpath query language to search for the url of the item’s cover art in the html. performing this action for several oclc numbers can take a while and some rows will be stuck in the “loading…” phase. we found one to two hundred oclc numbers may take up to an hour to populate. figure 2. waiting for the google sheet to find an item’s cover image with patience all the image url’s will be populated and can be copied over to the master google sheet. in this way, the image url value, not the importxml function, is available to be included in our new title list. figure 3. cover image urls populated (enlarge) creating the html list item lucky for us, the new process uses the same data export process with the wms report designer in oclc’s worldshare. with our data exported, it is a simple process to copy the new rows into the master google sheet, adding them to the new items’ data. this file builds upon itself so new rows are just added to the end. figure 4. google sheet with item data (enlarge) most of the data itself is self-explanatory, but the “fund code” is a field that we use to designate which fund was used to purchase the new item and coincidentally it also represents the academic department. utilizing a namedrange in the google sheet, we are able to translate the fund code: for example, changing “bhis” to history. with our image location and department, we can combine the data with the necessary html tags to generate the html item list. our basic html display code consists of an item image, a title, an author, item type, fund code, and a link to our catalog. an individual item’s html code will look like this: <li><img alt='cover' style='height:96px; width:72px;' class='boxed-imagesm left' src='url to image'><a class='name' target='_blank' href='https://macalester.on.worldcat.org/oclc/xxxxxxxxxx'>item title<br/>by item author</a><p class='dept' style='display: none;'>department</p><p class='type' style='display: none;'>item type</p></li> next, to create the item link to our catalog, we utilize the oclc number (ex: https://macalester.on.worldcat.org/oclc/10694032). finally with all of the necessary values populated, we can concatenate the data into one string. =if(e95<>"",concat("by ",e95),"") resulting in html item list code for each item. <li><img alt='cover' style='height:96px; width:72px;' class='boxed-imagesm left' src='http://coverart.oclc.org/imagewebsvc/oclc/+-+4626438_70.jpg'><a class='name' target='_blank' href='https://macalester.on.worldcat.org/oclc/3891719'>the mongol world empire, 1206-1370<br/>by john andrew</a><p class='dept' style='display: none;'>geography</p><p class='type' style='display: none;'>book</p></li> when each row’s data is combined and then included with the html file containing the list.js logic, the new titles page displays with the appropriate styling and logic. to finish, we simply select the entire column with the html list item code, then copy it into the include file. this results in a finished new titles web page. example html item list code figure 5. going forward, to update the new titles page with new data, we will only need to copy the new rows from the html items list column in the google sheet to the include file. final thoughts during a typical update process, which usually happens every other week, it might take 15-20 minutes to update the new items page. this isn’t continuous, as the image cover process can sometimes take an hour to retrieve two hundred cover images. however, the image url retrieval doesn’t need to be monitored and can occur simultaneously to other work. also, prior to the addition of new titles, we also create a backup of the master google sheet that holds the data. this helps to ensure that if any problems occur during the update, we can revert back to a previous version. with only four months using the new version, it’s hard to make any statements regarding increased or decreased use of the new titles page. the actual page-views of the new page are slightly higher than the previous system over a similar time frame, but more time is needed to see if this is a bigger trend. in all honesty, the move to the new system wasn’t to garner more views, although that would be nice too, but, as mentioned elsewhere, the benefits were to eliminate the security risks of the old server, improve the update process, and integrate the new titles page into our cms. finally, as a proof of concept, in our testing we duplicated the number of rows to match a typical year’s worth of new items, about four thousand, and list.js performed wonderfully with only a 2-3 second delay, depending on the browser, in generating the web page. figure 6. the macalester new arrivals web page (enlarge) in conclusion, using list.js, google sheets, html, and a little javascript, we now have a new titles page that satisfies many of our criteria. although not optimal, in that some steps require manual intervention, the time vs. cost benefit seems to be justified and appropriate. as with most website projects, the future will allow for changes to the current process, possibly automating some parts that are currently manual in nature. again, we hope for those libraries without access to it staff or without advanced programming knowledge, this article provides a viable option to implementing a new titles list. about the author john meyerhofer is the digital scholarship librarian for the dewitt wallace library at macalester college in st. paul minnesota. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – hacking summon 2.0 the elegant way mission editorial committee process and structure code4lib issue 26, 2014-10-21 hacking summon 2.0 the elegant way libraries have long been adding content and customizations to vendor-provided web-based search interfaces, including discovery systems such as proquest’s summon(™). unlike solutions based on using an api, these approaches augment the vendor-designed user interface using library-provided javascript code. recently, vendors have been implementing such user interfaces using client-centric model-view-controller (mvc) frameworks such as angularjs, which are characterized by the use of modern software engineering techniques such as domain-specific markup, data binding, encapsulation, and dependency injection. consequently, traditional approaches such as reverse-engineering the document model (dom) have become more difficult or even impossible to use because the dom is highly dynamic, the templates used are difficult to discern, the vendor-provided javascript code is both encapsulated and partially obfuscated, and the data binding mechanisms impose a strict separation of model and view that discourages direct dom manipulation. in fact, practitioners have started to complain that angularjs-based websites such as summon 2.0 are very difficult to enhance with custom content in a robust and efficient manner. in this article, we show how to reverse-engineer the angularjs-based summon 2.0 interface to discover the modules, directives, controllers, and services it uses, and we explain how we can use angularjs’s built-in mechanisms to create new directives and controllers that integrate with and augment the vendor-provided ones to add desired customization and interactions. we have implemented several features that demonstrate our approach, such as a click-recording script, coins and facet customization, and the integration of ebook public notes. our explanation and code should be of direct use for adoption or as examples for other summon 2.0 customers, but they may also be useful to anyone faced with the need to add enhancements to other vendor-controlled mvc-based sites. by godmar back and annette bailey background in mid-2013, proquest released the first preview of summon 2.0 to customers. soon after, libraries who had made customizations to their old summon 1.0 interface discovered that those customizations did not port easily to summon 2.0 because of the underlying technology upon which summon 2.0 was built. summon 2.0 uses angularjs, a modern model-view-controller (mvc) framework, which has drastically changed the way in which such web applications are built, and thus how they must be expanded and customized. to customize summon 2.0, the vendor allows customers to include a javascript (.js) file in the web application that is being run. this facility was offered in response to customers who exploited an html sanitization bug in summon 1.0’s configuration interface to inject their own javascript into the summon 1.0 user interface. because the customer-provided javascript code executes with the same privileges as the original vendor-provided javascript code, it has access to all parts of the user interface and can make arbitrary changes. thus, this approach provides the (theoretical) ability to control and override any feature of the existing user interface, but doing so in practice in the summon 2.0 interface turns out to be difficult. angularjs to help readers understand the techniques we have developed to augment the angularjs-based summon 2.0 user interface, we first discuss essential features of angularjs. angularjs is an mvc framework that supports the creation of the client side portion of a web application[1]. it has seen tremendous growth in the past few years. because describing all aspects of angularjs would exceed the scope of this article, the interested reader is referred to the official documentation[2]. angularjs uses html syntax to express the components of a web application, but it allows html to be extended with custom directives[3]. the directives can provide additional elements or additional attributes to elements. for instance, <div ng-show=”visible”>…</div> applies the built-in angularjs directive ng-show to a <div> element, controlling its visibility based on the truth value of the variable visible. angularjs provides many directives out of the box[4], but complex applications such as summon define many more. once defined, a directive can be reused many times – directives are thus similar to subroutines in ordinary programming. angularjs enforces relatively strict model/view separation. in angularjs, the model state is kept in pure javascript objects, attached as properties to an object referred to as “scope.” whenever the values of these variables change, angularjs automatically updates any dependent views without the programmer needing to code the behavior explicitly. using the example above, the programmer would simply set visible = true and the framework would change the visibility of the element in question. the use of pure javascript objects – as opposed to special objects provided by the framework – makes this automatic updating a bit more expensive for the framework since it needs to perform “dirty-checking” during so-called digest cycles whenever the values of model variables might have changed. dirty-checking compares the last known value of a variable to its current value and triggers updates if it has changed. the big advantage to the programmer is that they do not need to change their javascript code to be aware of angularjs. for instance, javascript objects obtained from parsing json delivered by an external service can be attached directly to scopes. the ‘view’ in an angularjs application is an html description of the user interface, controlled by model variables. these variables appear in angularjs expressions and control aspects such as visibility, conditionals, repetition/loops, and others. angularjs’s expression language is very similar to javascript in terms of its syntax and operators. to further simplify mvc-based application development, angularjs supports two-way data bindings using the ng-model directive. for instance, <textarea ng-model=“message”></textarea> binds the variable message to the content of a textarea element displayed on the screen. this binding not only ensures that whatever the user enters into the textarea can be accessed via the current scope’s message property, but it also ensures that setting the scope’s message property from within angularjs expressions, or from within javascript code, will update what is being displayed in the textarea. angularjs’s “controller” feature allows the addition of javascript code to further control the interaction between model and view. unlike in other mvc frameworks, a controller is not strictly necessary since angularjs’s plumbing is often sufficient to keep model and view in sync. angularjs controllers are used to implement additional logic that cannot be expressed in angularjs expressions or to interact with outside sources such as apis. the result is often a single-page application, like summon 2.0, where resources are loaded dynamically from the server and client-side code renders changes to the dom on the fly. compared to a traditional web app or website where the controlling logic is on the server side, like summon 1.0, this type of application is faster to render and navigate. lastly, angularjs enforces the use of a number of modern software engineering ideas that help with structuring the code in a modular way while providing strong encapsulation of individual modules and components. angularjs uses “dependency injection”[5] – components such as controllers must explicitly declare which other components they intend to use in a given context. at runtime, those dependencies are then passed (“injected”) so that the code can run. unlike the import feature in the module systems used in languages such as python, this dependency injection allows control over which instances are injected at runtime. consequently, a mock instance of a particular dependency can be injected during testing whereas the actual instance of a dependency is used at deployment time. reverse engineering summon 2.0 summon 2.0 currently defines about 84 directives, 71 services (of which 28 are mock services for testing), 21 controllers and about 10 filters. the components are defined in 38 modules (28 of which are solely for providing mock components during testing). we identified the names of these components by grepping the obfuscated javascript source code for statements such as this.angular.module("summonapp.directives").directive("doilink", doilink) this statement defines a directive doi-link inside the summonapp.directives module [6]. the majority of the user interface is expressed as individual directives. angularjs provides multiple ways of defining directives, but summon 2.0 mostly defines so-called template expanding directives, which primarily consist of html templates. for instance, doilink = function() { return { replace: true, template: '<a ng-href="http://dx.doi.org/{{doi}}" target="_blank">{{doi}}</a>', scope: { doi: "@" } }; defines a simple directive which, when invoked, will replace the element in which it occurs with a link to the doi registry, based on the value passed to a doi attribute. it can be used by other directives, like “detail-row,” here: <li detail-row row-value="document.dois[0]" … > <a … doi-link doi="{{rowvalue}}"></a> </li> the camel-cased rowvalue refers to the value of the attribute row-value, which in turn is tied directly to the first doi (digital object identifier) occurring in the api response to which the document variable refers. when the directive is applied, an <a> element that links to dx.doi.org is created. since embedding large amounts of html in javascript code would be awkward, most summon 2.0 directives are instead defined in small, separate files. these so-called partial templates contain just the html for one directive. each such partial is referred to using a templateurl attribute inside the directive in which it is used. for instance, the directive “detail-row” mentioned above is defined using templateurl: “/assets/detailrow.html” summon currently uses 92 partial templates. to avoid having to fetch many small files during deployment, the summon 2.0 developers combine all partials as string literals[7] into a single javascript file (e.g., /assets/templates-f1fad2416306a297561379336e1f5644.js). when this file is loaded, the templates so defined are placed into angularjs’s template cache and thus can be used without fetching the individual small files. we exploit this design to reverse engineer summon 2.0’s templates by including the templates into a separate angularjs-based webpage, which then uses the highlight.js[8] package to display the templates in syntax-highlighted form[9], as shown below: figure 1. the figure shows the html code for the document-summary directive which displays information about individual documents returned in response to a user’s search. using this method, we were able to obtain a general understanding of the summon 2.0 user interface and could zoom in on areas of interest. to inspect the model state that is accessible in a particular directive, we used a trick: we provided a second definition for the directive. in the angularjs world, such a redefinition is also referred to as “stacking” directives. for instance, the code below provides a second definition for the document-summary directive: angular.module('summonapp.directives').directive("documentsummary", function() { return { link: function (scope) { console.log("scope object in documentsummary directive:"); console.dir(scope); } } }); the link method of the object returned by the function passed to the .directive method refers to what is called the  postlink()[10] function. invoking the postlink() function of a directive is the last step in angularjs’s compilation and rendering process. in the postlink() function, a directive writer has the ability to add event handlers to the dom to achieve desired functionality or to provide the facilities that will update the view if the model state changes. in this example, we simply log a snapshot of the scope object to the console. angularjs applies both the original and the added (stacked) directive definition whenever it encounters uses of document-summary in any template it processes. since all directives applied to an element share the same scope, we can inspect the scope object that is used by the original definition, as shown below using the chrome javascript console: figure 2. in the example, we see that the scope within a document-summary directive contains a document property that carries the metadata for the summon document being displayed. this object was returned from the summon api[11] using json. this approach to understanding the model state works most successfully if the directive in question either does not create a separate scope or creates a new child scope that prototypically inherits from the parent. prototypal inheritance[12] is a feature unique to javascript in which scope objects inherit properties from a parent in a prototype chain but are able to override them upon assignment. to support stricter encapsulation, angularjs also allows directives to create so-called isolate scopes which do not prototypically inherit from their parent scopes. instead, the directive writer has to specify which properties should be passed into the directive’s scope. the majority of summon 2.0 directives uses isolated scopes. when these are redefined, the new directive will not have access to the properties defined inside the original directive’s isolated scope. rather, it will have access only to the properties defined in the scope where it is used. this detail is important to keep in mind, as trying to access properties that occur in an isolated scope is often tempting. moreover, if we redefine a directive that is itself used inside a directive that uses an isolated scope, the redefined directive will have access only to those properties that are included in its parent, rather than all properties that exist in ancestor scopes. to find and access such properties, it is necessary to follow the $parent chain of scopes, both when performing manual inspection with the chrome debugger and when accessing those properties programmatically. example hacks it is worth discussing why the traditional approach to adding user interface improvements while ignoring summon’s use of angularjs will not work. in static web pages like summon 1.0, it was possible to trigger some code after the dom of the page had been built, and for that code to examine the dom which was then more or less stable until the page was reloaded. this approach fails in angularjs because the dom of the page is constantly being updated in response to changes in the model state. moreover, the dom is difficult to reverse engineer by inspection since much of it is generated by built-in or custom directives. it is possible to exploit angularjs’s scope $watch (or $watchcollection) facility to obtain notifications when the model state changes, for example, after the results of the search api have arrived. the watch listener fires also when the user engages the infinite scroll feature and more results are fetched from the api. however, there is no way[13] to know when angularjs has completed updating the dom in response to this change in model state. a timeout could be used as a work-around, but this is brittle and subject to flickering since the browser might make the changed dom visible to the user before the timeout is executed. instead, the modifications need to make use of angularjs’s facilities. technique 1: changing templates on the fly it is possible to change the templates that are placed in the template cache before they are being used. as an example, consider a modification that places coins[14] inside an invisible container to suppress its display. coins embed metadata that is processed both by metadata extractors such as zotero and by accessibility tools such as libx. libx processes those coins and adds a hyperlinked icon (a cue) that links to the openurl resolver. this functionality is generally redundant on the summon page; we wish to suppress it. to achieve this, we need to change the portion of the document-summary template that places the coins, which is shown below <span class="z3988" title="{{doc.open_url}}"><!-coins --></span> such that it is wrapped into an outer, invisible span like so <span style="display: none"> <span class="z3988" title="{{doc.open_url}}"><!-coins --></span> </span> the following code accomplishes this via string replacement: angular.module('summonapp').run([ '$templatecache', function (templatecache) { var docsummary = "/assets/documentsummary.html"; var v = templatecache.get(docsummary); v = v.replace(/(<span.*z3988.*coins --><\/span>)/, "<span style='display: none'>$1</span>"); templatecache.put(docsummary, v); }]); this code is placed in the existing summonapp module as a run block, which is executed after the module has been initialized. the example shows how angularjs implements dependency injection: access to the angularjs templatecache[15] service is provided by listing the name $templatecache inside an array whose last element is the function into which the service is passed as an argument.[16] technique 2: direct dom manipulation in our second example, we wish to record when a user clicks on any of the links summon 2.0 provides that lead to a discovered item. summon 2.0 provides a fair number of such links, some of which are marked in red in the screenshot shown below: figure 3. we needed to identify those links reliably and add event handlers that are invoked when they are clicked. this required special casing for each location where such links occur. below we show the directive that handles the primary links occurring in the main and the preview panel. function recordclicksonprimarylinks() { return { link: function (scope, ielement, iattrs, controller, transcludefn) { ielement.find("h1.customprimarylinkcontainer").on("click", function (event) { // find document id var doc = scope.document || (scope.preview && scope.preview.doc); var id = doc && doc.id; if (id !== undefined) recordclick(id); }); } } } angular.module('summonapp.directives') .directive("documentsummary", recordclicksonprimarylinks) .directive("preview", recordclicksonprimarylinks); we use the same code in both cases and redefine the document-summary and the preview directives. in this case, the original, template-based document-summary and preview directives have already added the <h1><a class=“customprimarylink” … ></a></h1> element when the redefined directive’s postlink function is called, so an onclick event handler can be attached directly to that element. in the event handler, we retrieve the id of the document in question from the scope. if used inside the document summary, this id will be contained in the property document; for the preview pane, it will be in the preview.doc property. we found those by inspecting the scope object using the chrome debugger. we used the same technique for the other links displayed on the results page. the technique of using direct dom manipulation inside a postlink() method comes with a big caveat, which is that the elements being manipulated must exist when the postlink() method is called. to see that this is not always true, consider the following failed attempt at implementing the coins-hiding hack discussed earlier using direct dom manipulation in a redefined document-summary definition: angular.module('summonapp.directives').directive("documentsummary", function() { return { link: function (scope, ielement, iattrs, controller, transcludefn) { // does not work: ielement’s dom is incomplete var $invisible = $("<span></span>").css("display", "none"); ielement.find(".z3988").wrap($invisible); } } }); even though the original, template-expanding document-summary directive had already been applied, the processing of its template was not yet complete. specifically, an ng-repeat clause (<div ng-repeat=“doc in availabilitydocs”) in which the coins spans were nested had not yet been expanded. since ng-repeat must re-render whenever the collection (here: availabilitydocs) changes, its rendering occurs inside a watcher function that is attached via $watchcollection. despite the fact that the availabilitydocs field is already initialized when our directive’s postlink function is called, the accompanying dom elements that result from the expansion of the ng-repeat directive do not yet exist (we discovered this by printing ielement.html()). there also is no way in angularjs to receive a notification when the rendering of a particular directive is complete. technique 3: pure model interaction in our third example, we consider how to implement localized logic for the facets summon displays, as shown in figure 4. figure 4. when summon 2.0 performs a search, facets such as library location are displayed in collapsed form, leaving some users confused about whether they can restrict their search by library location. one library desired a different behavior in which this facet is, by default, expanded. once the user learns how to interact with those facets by collapsing or expanding them, it is desirable for the user interface to remember the user’s choice, which requires persisting the user’s choice across page reloads. the code below implements this functionality. angular.module('summonapp.directives') .service("facetstatesaverservice", ["storeservice", function (storeservice) { this.collapsed = storeservice.bucket("collapsedfacets"); }]) .directive("facetfield", ["facetsservice", "facetstatesaverservice", function(facetservice, facetstate) { return { link: function (scope) { var facet = scope.facet; var shouldbecollapsed = facetstate.collapsed.get(facet.label); scope.$watch("facet.collapsed", function (nval, oval) { if (nval == oval) return; facetstate.collapsed.set(facet.label, nval); }); if (shouldbecollapsed != null) { if (shouldbecollapsed != facet.collapsed) facetservice.togglecollapsed(facet); } else { // library wishes to expand library facet by default if (facet.label == "library" && facet.collapsed) facetservice.togglecollapsed(facet); } } } }]); we define a new service facetstatesaverservice, which in turns imports a service summon 2.0 provides (storeservice). this service provides a simple facility to save and restore objects using the html5 localstorage[17] facility; we use it here for convenience and to demonstrate how to inject existing summon services into local customizations. we redefine the directive facet-field, which is used to render each facet. inside this directive we interact with the angularjs model state. the scope refers to a facet object whose attribute collapsed denotes whether the facet is currently collapsed. we observe changes of that attribute via scope.$watch, persisting any changes. if the user collapsed or expanded  the facet at least once, its last state is restored when the facet-field directive is rendered. otherwise, if the facet refers to the library location, we expand the facet. when restoring a facet’s collapse state, we do not simply set facet.collapsed, because we want to avoid collapsing facets that are currently applied. we use summon’s provided facetservice for this purpose. we inject the name and invoke its togglecollapse() method to that end. the methods provided by this service, as well as others, are undocumented and were obtained by reverse-engineering proquest’s code. as such, our code may break if their implementation changes. any such failure would be localized and result only in the loss of this particular feature. technique 4: injecting your own directives our last example inserts public notes into summon’s interface. this feature allows librarians to notify the public in a timely manner when a ebook resource goes down, or it allows the display of license restrictions related to accessing an ebook. in the image below, the message framed in green was inserted by our service. figure 5. we implemented a new directive for this feature, availability-additional-info, shown below: angular.module('summonapp.directives') .directive("availabilityadditionalinfo", ["$http", function ($http) { return { template: "<p ng-repeat='resource in resources'>" +" <b>{{resource.name}}:</b>" +" <span ng-bind-html='resource.msg'></span>" +"</p>", link: function (scope, ielement, iattrs, controller, transcludefn) { scope.$watch("type", function (type) { switch (type) { case "rta": case "fulltext": var doc = scope.avdoc; if ('isbn' in doc) $http.jsonp("http://libx.lib.vt.edu:4000/isbn/" + doc.isbn + "?callback=json_callback") .success(function (data) { scope.resources = data.resources; }); } break; } }); } } }]) this availability-additional-info directive uses an html template and angularjs directives ($http) in the same way as if we had full control of the summon 2.0 front-end. it contacts a json-p[18] service we provide and inserts the resources property of the returned json object into the scope so that the corresponding display can be rendered[19]. this directive is added to the availability directive’s template using the template cache technique described earlier: var availability = "/assets/availability.html"; var v = templatecache.get(availability); v = v.replace(/(<\/div>)\n$/, "<span availability-additional-info></span>$1"); templatecache.put(availability, v); since this directive will be used inside the availability directive, it must be adapted in two ways: the document whose isbn should be used must be extracted from the avdoc property of the surrounding scope. in addition, we wish to display this message only if the type of the resource is fulltext. since the type property is computed inside the postlink function of the surrounding availability directive, we perform this action inside a $watch callback once type becomes available. risks by now the curious reader may be wondering: what could possibly go wrong with writing code that directly interfaces with vendor code as we have presented? a moderate number of things, it turns out. first, hosting an external script requires that the server hosting the script is at least as available as the summon service. since its javascript is executed synchronously (and must be for our techniques to work!), an unavailable server will result in lengthy delays since the client will time out before executing the remaining javascript in the summon page. second, any syntax error in the script, or any runtime error thrown in a config or run block that was added to a module[20], will prevent the initialization of the module, causing the entire ui to fail and the user to see a white page. as a precaution, we recommend wrapping such critical blocks in try/catch clauses to prevent errors from propagating. to avoid syntax errors, we are using a technique where experimental versions of our scripts are seen only by debugging users, which we identify using a cookie whose value determines the version of the script served (via an apache mod_rewrite[21] directive). a special value empty instructs apache to serve an empty script, turning off all modifications. third, we have encountered rare, but extremely difficult to debug problems in two instances. when we initially deployed summon 2.0, librarians noticed that the legacy, get-based summon 1.0 search boxes had ceased to work. summon 2.0 implements compatibility with these search forms by examining the url in its window.location variable and executing a search if necessary. instead of using the recommended $location[22] service, summon’s implementors directly accessed window.location. in doing so, they introduced a subtle dependency on when the $location service was first instantiated[23]. when we injected $location into an entirely unrelated run block, we subsequently caused the page not to execute the search. the second instance concerned when a summon directive’s template uses both the reload: true option (which says that this directive’s template should replace the element to which it is applied) and if it contains an ng-if directive that evaluates to true (see issue #8748) [24]. in this case, stacked directives are not executed, causing those techniques that rely upon redefinition of directives not to work. summary we have presented four techniques we used to extend an existing angularjs application (summon 2.0) that was not originally designed for such extensions. based on limited reverse engineering of the application’s structure, in particular its use of template-expanding directives, we were able to devise a number of ways in which the behavior of this application can be changed and improved in a seamless manner. like all code that interacts with web pages whose implementation is controlled by another party, our code is subject to potential failure if their implementation changes. however, any such failure is likely to be local, affecting only features that depend on a particular template or service. because our techniques rely only on angularjs features whose behavior is documented, they are likely to work in future versions of angularjs. our “hacks” are fully integrated into angularjs’s processing, specifically its dirty-checking process that provides the plumbing that keeps model state and the rendered view in sync. as a result, flickering effects are minimized and there is no potential for timing dependencies or race conditions. related work the idea to mash enhancing content or customizations into vendor-provided interfaces dates back several years (schmidt, 2013). for instance, many libraries use tools from librarything to enrich bibliographic records displayed in their opac with information that ranges from tables of contents to suggestions of reading based on the item the user is currently viewing (librarything). syndetics, a company later acquired by proquest, formed a business around including book covers and other information, now offered as proquest’s syndetics solutions enrichment elements (proquest). librarians have also used this technique: for instance, oldham’s work demonstrates how to add user interface improvements to the university of guelph’s instance of primo, a discovery product that competes with summon (oldham, 2014). klein’s article “hacking summon” explains how oregon state university was able to quickly roll “hacks” to improve the summon 1.0 user experience (klein, 2010). reidsma developed open source code that allows customers to gather click information from the summon 1.0 interface (reidsma, 2012a-b). all of these examples mentioned were implemented using traditional techniques, such as dom manipulation. electronic resource librarians benefit from the ability to modify vendor interfaces not only because it allows them to cosmetically improve their user’s experience, but also to improve collections management tasks and statistics. for instance, reidsma’s code was used by pattern at the university of huddersfield to collect and analyze usage patterns and statistics from summon users (pattern, 2012). the ability to manipulate a vendor user interface also facilitates the roll-out of entirely new services. for instance, prior to our work, jing embedded links in summon 1.0 to a licensing database that she created (jing, 2014). this feature allowed users to find detailed license information about the resource being linked to. lastly, librarians actively study the features provided by modern discovery systems. as an example, fyn et al. studied students’ acceptance of and use of the database recommender feature in summon (fyn, 2013), which promotes a database from a predefined list at the top of the display of search results. these examples show that there is a great need for, and tremendous potential benefit from, being able to improve vendor user interfaces to collect data and statistics, implement new services, and facilitate user studies that contribute to a better understanding and improvement of the services librarians provide to their community. acknowledgements we are indebted to the summon user community for ideas and discussion regarding the hacks we presented. we would also like to thank sara amato for her feedback and help in shepherding the paper, and jason thomale for his excellent editing. notes [1] officially, google refers to angularjs as a mvw (“model-view-whatever”) framework to sidestep the largely philosophical discussion of whether it is closer to a traditional mvc (“model-view-controller”), or a mvvm (“model-view-viewmodel”) or mvp (“model-view-presenter”) approach. [2] https://docs.angularjs.org/api [3] https://docs.angularjs.org/guide/directive [4] https://docs.angularjs.org/api/ng/directive [5] http://en.wikipedia.org/wiki/dependency_injection [6] by convention, the camelcased name doilink in the definition corresponds to the hyphenated directive doi-link when invoked. [7] https://developer.mozilla.org/en-us/docs/web/javascript/guide/values,_variables,_and_literals#string_literals [8] https://github.com/pc035860/angular-highlightjs [9] http://libx.lib.vt.edu/services/summonvis/summon2.0/listtemplates.html [10] https://docs.angularjs.org/api/ng/service/$compile [11] http://api.summon.serialssolutions.com/ [12] http://en.wikipedia.org/wiki/prototype-based_programming [13] https://github.com/angular/angular.js/issues/1306 [14] http://ocoins.info/ [15] https://docs.angularjs.org/api/ng/service/$templatecache [16] angularjs provides multiple methods of achieving the same goal, which can be confusing. they are discussed here (https://docs.angularjs.org/guide/di). our example uses the “inline array annotation” method. [17] http://diveintohtml5.info/storage.html [18] http://en.wikipedia.org/wiki/jsonp [19] we would have liked to use angularjs’s $resource service, but it is not included in summon 2.0. [20] https://docs.angularjs.org/guide/module [21] http://httpd.apache.org/docs/2.2/rewrite/intro.html#rewritecond [22] https://docs.angularjs.org/guide/$location [23] https://docs.angularjs.org/guide/services [24] https://github.com/angular/angular.js/issues/8748 references fyn af, vera lux and robert js. 2013. reflections on teaching and tweaking a discovery layer. reference services review 41(1):113-124. hevery m. 2009. building web apps with angular. oopsla. orlando, fl: association for computing machinery.  http://www.oopsla.org/oopsla2009/program/demonstrations/254-building-web-apps-with-angular-2-of-2 hevery m, abrons a. 2009. declarative web-applications without server: demonstration of how a fully functional web-application can be built in an hour with only html, css & javascript library. proceedings of the 24th acm sigplan conference companion on object oriented programming systems languages and applications. orlando, florida, usa: acm. jing j. 2014. adding copyright/license information to different library systems. ala. las vegas, nv. [cited 2014 september 09]. available from: http://www.slideshare.net/happyrainb/ala2014 klein mb. 2010. hacking summon. the code4lib journal 11. [cited 2014 september 09]. available from: http://journal.code4lib.org/articles/3655 librarything. librarything for libraries [internet]. [cited 2014 september 29]. available from: https://www.librarything.com/forlibraries oldham r. 2014. hacking your discovery layer. computers in libraries. washington, dc. [cited 2014 september 09]. available from: http://www.infotoday.com/cil2014/session.asp?id=a105 pattern d. 2012. relevancy rules. self-plagiarism is style. [cited 2014 september 09]. available from: http://www.daveyp.com/2012/05/06/relevancy-rules/ proquest. syndetics solutions, enrichment elements [internet]. [cited 2014 september 29]. available from: http://proquest.syndetics.com/marketing/detail/enrichmentelement reidsma m. 2012a. guerrilla analytics. matthew reidsma. [cited 2014 september 09]. available from: http://matthew.reidsrow.com/articles/24 reidsma m. 2012b. summon-stats. [cited 2014 september 09]. available from: https://github.com/mreidsma/summon-stats schmidt k, elguindi ac. 2013. discovery systems, layers and tools, and the role of the electronic resources librarian. electronic resource management: practical perspectives in a new technical services model. chandos publishing. p. 109-139. about the authors godmar back is associate professor of computer science at virginia tech, where he has been doing research and teaching in computer science since 2004. dr. back obtained his phd from the university of utah and worked as a post-doctoral scholar at stanford university. his research interests are diverse, including operating systems, virtualization, web technology, scientific computing, programming languages, and library information systems. he is an active collaborator with librarians in the area of advancing library technology to ensure that modern technology can find its way into the library sphere. since 2007, he has been involved in the libx project, providing technical supervision and input.  recently, he has been leading the libfx project (libfx.lib.vt.edu) which creates online visualizations of real-time library usage. annette bailey is currently the assistant director for electronic resources and emerging technology services for the jean russell quible department of collections and technical services at virginia tech. bailey serves on the program planning committee for the er&l conference. she co-developed the open source libx plug-in, for which she received the 2007 lita brett butler entrepreneurship award.  she won a national leadership grant in 2006 for libx and in 2008 for libx 2.0. her role in collections technical services is to ensure best practices are implemented as workflows and data change for electronic resources. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – unix commands and batch processing for the reluctant librarian or archivist mission editorial committee process and structure code4lib issue 23, 2014-01-17 unix commands and batch processing for the reluctant librarian or archivist the unix environment offers librarians and archivists high-quality tools for quickly transforming born-digital and digitized assets, such as resizing videos, creating access copies of digitized photos, and making fair-use reproductions of audio recordings. these tools, such as ffmpeg, lame, sox, and imagemagick, can apply one or more manipulations to digital assets without the need to manually process individual items, which can be error prone, time consuming, and tedious. this article will provide information on getting started in using the unix environment to take advantage of these tools for batch processing. by anthony cocciolo introduction as a teacher of digital archives and digital libraries at pratt institute’s school of information and library science, i often try to give students a taste of the unix environment. when presenting on it, i sometimes hear a small gasp when i bring up the black terminal window with white text, making it clear to me that students think that this seemingly anachronistic computing interface is somewhat daunting if not outright terrifying. my advice to them is get to love unix. one major reason the unix environment ought to be of interest to librarians and archivists is it enables the quick and inexpensive batch processing of born digital and digitized assets. this means that if you are looking to perform one or more transformations on a batch of assets (e.g., resizing videos, creating access copies of digitized photos, create mp3s from wav files), you can do it very quickly, and without the need to manually process individual items, which can be error prone, time consuming, and tedious. unix is not the only way to do batch processing. many programs with graphic user interfaces have incorporated batch-processing capabilities, and batches can be performed with other operating systems like windows’ command prompt (although with some less functionality). despite these other options, a persistent advantage of the unix environment is the wide array of open source software that can be used for batch processing. some of these programs include ffmpeg, lame, sox, and imagemagick. the field of library and information science has begun to recognize the value that the unix environment can bring to lis professional practice. botticelli et al. (2011) noted that digital curators “need a balance between technical and analytical skills,” including “hands-on practice with syntax, whether in typing unix commands or applying valid xml tags” (p. 154). heising, lewis, moen and salter (2013) confirm this need and note that of the 208 digital curator and manager job postings they analyzed, 28 required some knowledge of linux or unix. to address this need, they have developed a one-week unix fundamentals course at the university of north texas. some of the best tools librarians and archivists may need for batch processing will be discussed here, as well as some of the most basic and often used unix commands. your mac as unix environment if you are a macintosh user, you already have access to a unix environment since osx is built on top of unix. you can access the command-line unix environment by going to your macintosh hd, selecting applications, utilities, and then terminal. you can easily navigate between the text-based terminal and the graphical finder. for example, if you type the following in the terminal prompt, it will open your current directory within the finder: $ open . the “.” in unix means the current directory. if instead you typed “open ..”, you would open the parent of the current directory, since “..” refers to the parent directory. the parent directory is the directory that contains the current directory. if you are not a macintosh user, consult the unix for windows users section of this article, or use unix on a remote host, as described below. connecting to a remote host as often is the case, you will be connecting to a unix computer from a personal computer. for example, this could be a webserver where you are hosting files for your end users. if you do not know what unix remote servers are available at your institution, or what your username and password may be, be sure to contact your information technology department. many webhosts allow users to connect via secure shell (also known as ssh), which effectively provides the unix command line of a remote server. on a mac, to connect to a remove server, open the terminal app, and issue the following command: $ ssh -l my_user_name my_remote_host.edu this command instructs ssh to remote connect to my_remote_host.edu, with the username my_user_name. it will prompt you for a password. once authenticated, you will have access to the unix command prompt. if you want to connect to a remote host via a windows pc, you can use the freeware program putty.[1] in addition to being able to connect to the unix shell, it is also very useful to know how to transfer files between a unix host and your computer. to do this, you can use a secure ftp program. for mac, a program for doing this is cyberduck.[2] for windows, a program for doing this is winscp.[3] in these programs, you specify your host, your username, and password, and once connected you can drag and drop files between your remote unix host and your personal computer. unix file system the unix file system is similar to most operating system file systems in that it is a hierarchy of directories with files in them. for example, a fairly common top-level organization of the unix file system includes: / /users /etc /usr /bin /var typically, most user home directories are contained within “/users” or “/home.” the “etc” directory generally contains system settings. the other directories typically include programs and utilities. to find out what directory you are in, you can use the “pwd” command, which will output your current directory. you can see a listing of the files contained within a current directory by typing “ls” (or list). for example, typing the following will show the contents of my home directory: $ ls /users/acocciolo similarly, you can type ls . or ls .. to list the contents of the current directory or the parent directory, respectively. you can copy a file by issuing the “cp” command. for example: $ cp anthony.txt anthony2.txt this will copy a file called anthony.txt to a file called anthony2.txt. similarly, if you want to move a file or rename it, use the “mv” (or move) command. for example, the following command will rename anthony2.txt to anthony3.txt: $ mv anthony2.txt anthony3.txt you can change your current directory by issuing the “cd” command. for example, if i’m currently in my home directory, the following command changes the current directory from my home directory to the root directory: $ cd / i could then return back to my home directory by issuing either of the following commands: $ cd /users/acocciolo $ cd ~ you should note that the “~” is shortcut for your home directory. you can create a new directory by issuing the “mkdir” command. the following command will make a new directory within the current directory: $ mkdir my_new_directory similarly, you can use the “rmdir” command to remove a directory, and the “rm” command to remove a file. wildcards the power of unix starts to become apparent when we start using wildcards. wildcards allow you to perform some operation on a set of files that match the wildcard. for example, imagine that in my current directory, i have a set of text files that i want to output to the screen. the following command would output each file that has a .txt extension: $ for i in *.txt; do cat "${i}"; done this command loops through every .txt file in the current directory, and runs the “cat” command on it, which just outputs the contents of the file to the screen. this is not an incredibly useful function, but the use of wildcards in conjunction with loops becomes more apparent with the following examples. commands for photo collections although products such as adobe photoshop have incorporated batch features, performing batch operations on born-digital or digitized photos through unix can be quicker and more cost effective since the software is open source and does not require an expensive software license. for example, imagine that you have digitized a large quantity of images and produced them as high-resolution tiff files. you would like to produce a high-quality jpg version for distribution in the reading room, as well as a lower quality version with a watermark for distribution over the web. these operations can be performed using the command line tool imagemagick. to download imagemagick on a mac, the easiest way is to install the package manager macports[4], and then to run the following command from the terminal application: $ sudo port install imagemagick note that ‘sudo’ runs the command as an administrator on the machine, and may prompt for a password to do so. once installed, you can run the following command from the directory with your tiff files in it to create your reading-room ready photos: $ for i in *.tif; do convert "$i" -density 72 "$i.jpg"; done this command converts the tiff files to jpg, and changes the image resolution to 72dpi, which is the resolution amenable to most computer monitors. it does not change the size of the image. the web accessible versions with the thumbnail can be created using the following command: $ for i in *.jpg; do convert "$i" -resize 500 "web_$i"; composite -dissolve 40% -gravity southeast pratt.png "web_$i" "web_$i"; done the above command takes all the newly created jpgs and resizes them to a maximum width of 500 pixels using imagemagick’s “convert” command. in doing so, it creates a new file with the prefix “web.” after it creates the resized image, it adds a watermark by using imagemagick’s “composite” command. the logo is 100 pixels in width and contained within pratt.png. the “gravity” parameter indicates the watermark should go in the southeast part of the image, which is also the lower right. the image is made partially transparent with the dissolve option set to 40%. a sample resulting image is shown in figure 1. figure 1: resized image with watermark. photo by anthony cocciolo many options are available for adding watermarks using imagemagick, as well as changing the properties of images. additional commands can be found on their website.[5] commands for pdfs as someone who has worked on several outsourced digitization projects, i am always surprised to find vendors who charge per page to convert multipage tiffs into pdfs. fortunately, this is easily and quickly accomplished by using imagemagick once again. the following command will convert a directory full of single or multi-page tiffs into pdfs. $ for i in *.tif; do convert "$i" "$i.pdf"; done commands for sound collections the unix environment also offers many benefits for transforming audio collections. for example, imagine that you have a large collection of digitized audio that originated on audio cassette. you have digitized the files as wav files and would like to make them available online. however, you were unable to secure rights to put them online, so you have decided to make a small portion of the audio available under fair-use conditions. from the digitization process, you know that your recordings are roughly 60 minutes in length, and you feel that you can confidently put 10% of the recordings online (or 6 minutes). however, the process of trimming the files in an audio editor such as audacity[6] and then exporting to a format that is amenable to internet access such as mp3 could be a time consuming process. an alternative is to perform the work as a batch. the following command will use the open-source program sox to create 6 minute wav files from the original 60 minute wavs: $ for i in *.wav; do soxi "6min_$i" "$i" trim 2 360; done before you can run this command, you will need to download sox, which is an open-source audio editing program. you can download sox from their website[7] and install using the instructions provided. the command will create a new file with the prefix “6min_” given the input file “i.” it will also start two seconds into the recording (since recordings on audio cassette often have leaders with no audio) and trim it to 360 seconds (or 6 minutes). this will create our trimmed wav files, yet these are still not amenable to access. the following command will convert all the wav files into mp3s using the open-source mp3 encoder lame: $ for i in *.wav; do lame "${i}" "${i}.mp3"; done like sox, lame also needs to be downloaded and installed.[8] for the mac, it can be downloaded from c-net.[9] commands for video collections imagine that you have spent the last several weeks digitizing a collection of vhs tapes. the files you have created are 640 by 480 pixels, but you want to make them smaller so they will play quickly over the internet without the use of a streaming service such as youtube. you can use command line tools like ffmpeg to apply a transformation across this entire batch of files rather than having to pull each file into a video editing program, applying the transformation, and exporting the file. although ffmpeg is not included with all unix installations, it can be downloaded from their website.[10] a version that is easy to install from macintosh users is also available.[11] within the current directory of mp4 files (which are your digitized video), you can issue the following command: $ for i in *.mp4; do ffmpeg -i "${i}" -vcodec h264 -s 320x240 -movflags faststart "320_${i}"; done this will loop through every mp4 file in the directory and run the ffmpeg command on it. the “-i” option specifies the input file. the “-vcodec” parameter specifies that the output video should be encoded using the h264 encoder, which is a very popular video format used for delivering video over the internet. the “-s” command instructs ffmpeg to change the size dimensions of the video. the “-movflags faststart” command instructs ffmpeg to place the video metadata into the front of the file so the video can start playing as soon as enough information has been downloaded (rather than having to wait for the entire video file to download before it starts playing). the final parameter says that the output file should be named with the prefix “320_” to indicate that this is our new, resized file. you can modify the above script to perform a wide variety of transformations on video files. for example, the website “19 ffmpeg commands for all needs” includes a list of frequent commands you can use with ffmpeg.[12] commands for web archiving there are several methods for creating web archives, such as producing warc files, which is an iso standard format for web archives.[13] however, one limitation of this approach is that you will need a program to read the warc files. a less complex solution is using the wget program to download the webpages into a directory (fino-radin, 2012). the wget program is able to crawl an entire website and download all related html, image files, and other media that might be linked to the page. to install wget on a macintosh, the easiest method is to install it using macports by running the following command: $ sudo port install wget once it is installed, you can run commands like: $ wget -mpk http://www.thinkingprojects.org the above command will download my entire website. the “-mpk” parameter is a series of options to wget. the “m” specifies that all contents stemming from the homepage should be downloaded. the “p” command causes wget to download all the files that are necessary to properly display a given html page. the “k” option converts links within the document to make them amenable to local viewing. a listing of all wget options is available on their website.[14] other useful commands there are hundreds of commands available in the unix environment, and many of them you can find out about by doing a google search (e.g., by searching for “basic unix commands”). some that i have found particularly useful over the years include: $ df -hk this command will output the amount of disk space used and free space available to your unix computer. this is especially useful for librarians and archivists, who often deal with many large files that can quickly fill-up a hard disk. $ find . -name '*.tif ' -print the find command searches across the specified directory and all its sub-directories for a file name matching the wildcard. the example above will look for all files with the extension .tif. you can also search an entire unix file system for tif files using the related command: $ find / -name '*.tif ' –print the last useful utility that will be discussed here is grep. grep allows you to search for text occurring within files. for example, the following command will search the current directory and all sub-directories for occurrence of the word “curation” with all the files. $ grep –r "curation" * whereas the find command will search for filenames, grep actually searches within files. unix for windows users unlike macintosh os x, microsoft windows is not built upon unix. however, there are several options for accessing unix and unix-like functionality on your windows machine. the simplest option is to remote connect to a unix host using ssh, as described earlier in this paper. however, if you do not have access to a remote unix host, you can use one of the following options. please note for all of these options, you would need administrative rights on your computer to install the requisite software. the options include: option 1: install cygwin, which is a unix-like environment for windows, that allows you to run many unix programs and can be downloaded for free from their website.[15] it includes a package manager with a graphical user interface that allows you to install unix programs into the environment. for example, if you wanted to install sox, you could choose to install it via the package manager interface (see figure 2). you could then run all of the commands included here within the cygwin terminal window. files can be made available outside of the cygwin environment using standard windows file copy routines (e.g., copying files to the c:\cygwin directory using windows explorer). figure 2: cygwin allows you to install packages such as sox using a graphic user interface. option 2: install a unix virtual machine on your windows computer. this option allows for an entire unix operating system to be running within a windows environment much like any other windows program. an easy and inexpensive way to accomplish this is to use oracle’s product virtualbox, which is free virtual machine software that can be easily downloaded and installed from their website.[16] after installing virtualbox, you can then download a linux (unix) environment that has been pre-packaged for use within virutalbox. for example, the bitcurator [17] team has assembled an ubuntu linux, a popular flavor of unix, environment that can be easily downloaded and run within virtualbox.[18] this linux installation is pre-loaded with tools that are useful for processing born-digital materials. once inside the ubuntu linux environment, you can use a package manager with a graphical user interface to install additional packages that may not be included with the default installation. the downside of this approach is that running an entire os within another os can be disk-space and processor intensive, so you would want only to attempt this on a well-equipped modern computer. for example, the bitcurator virtual machine environment requires a 2.6 gb download from their website. option 3: the final option to run the commands included in this article is to sidestep the use of unix altogether and instead use windows’ command prompt environment, and install windows versions of the programs described here. programs described here such as ffmpeg, sox, and imagemagick are available as windows programs, and instructions are available for installing them for windows from their respective websites. the windows command prompt has a similar look to the unix environment, and some commands function very similarly. for example, you can run the “cd” command in much the same ways in both windows and unix environments. however, the windows command line environment does not offer as rich a set of functionality as unix, and the syntax for commands is slightly different. for example, the command to convert a batch of wav files to mp3 using lame in unix is the following: $ for i in *.wav; do lame "${i}" "${i}.mp3"; done however, in windows, you would write the command as the following: c:\>for %i in (*.wav) do d:\lame\lame.exe %i %i.mp3 in this example, d:\lame\lame.exe refers to the location of the lame executable program, which would be different depending on where you installed that program. note the use of the “%” as a way to denote a filename variable within the windows environment that is absent from the unix environment. conclusion this article offered ways that librarians and archivists can take advantage of the unix environment for the expedient processing of digitized and born-digital assets. these scripts can be modified in many ways to suit a variety of purposes. google searches for scripts, which point to many useful resources such as stackoverflow,[19] are very helpful in modifying scripts to meet local needs. also, the novice batch processor should note that it is fairly common that scripts don’t work right away and may require refining to produce the desired result. however, depending on the size of the collection you are looking to process, the investment in producing a script can save time in the long run, and can eliminate human error that occurs when processing items one at a time. notes [1] http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html [2] http://cyberduck.en.softonic.com/mac [3] http://winscp.net/eng/download.php [4] http://www.macports.org/install.php [5] http://www.imagemagick.org/usage/annotating/ [6] http://audacity.sourceforge.net/ [7] http://sox.sourceforge.net/ [8] http://lame.sourceforge.net/download.php [9] http://m.download.cnet.com/lame-audio-encoder/3000-2140_4-51042.html [10] http://ffmpeg.org/download.html [11] http://ffmpegmac.net/ [12] http://www.catswhocode.com/blog/19-ffmpeg-commands-for-all-needs [13] http://www.iso.org/iso/catalogue_detail.htm?csnumber=44717 [14] http://www.gnu.org/software/wget/manual/wget.html [15] http://www.cygwin.com/ [16] https://www.virtualbox.org/wiki/downloads [17] http://www.bitcurator.net/ [18] http://wiki.bitcurator.net/index.php?title=software [19] http://stackoverflow.com/ references botticelli p, fulton b, pearce-moses r, szuter c, watters p. 2011. educating digital curators: challenges and opportunities. international journal of digital curation 6(2):146-164. fino-radin b. 2012. wget cheat-sheet. available from: http://notepad.benfinoradin.info/tag/wget/. heising j, lewis p, moen we, salter j. 2013. scaffolding for digital curation education: a one week unix fundamentals course. alise annual meeting 2013, january 22-25, seattle, wa. available from: http://digital.library.unt.edu/ark:/67531/metadc139456/ about the author anthony cocciolo, ed.d. (acocciol at pratt.edu) is an assistant professor at pratt institute school of information and library science. his research and teaching are in the areas of digital archives, digital libraries, educational technology and moving image and sound archiving. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – usability analysis of the big ten academic alliance geoportal: findings and recommendations for improvement of the user experience mission editorial committee process and structure code4lib issue 38, 2017-10-18 usability analysis of the big ten academic alliance geoportal: findings and recommendations for improvement of the user experience the big ten academic alliance (btaa) geospatial data project is a collaboration between twelve member institutions of the consortium and works towards providing discoverability and access to geospatial data, scanned maps, and web mapping services. usability tests and heuristic evaluations were chosen as methods of evaluation, as they have had a long standing in measuring and managing website engagement and are essential in the process of iterative design. the btaa project hopes to give back to the community by publishing the results of our usability findings with the hope that it will benefit other portals built with geoblacklight. by mara blake, karen majewicz, amanda tickner, jason lam introduction and overview of group activities the big ten academic alliance (btaa) geospatial data project is a collaboration between twelve member institutions of the consortium and works towards providing discoverability and access to geospatial data, scanned maps, and web mapping services. (see appendix i for a full list of participating institutions.) the project officially began in june 2015 with the goal of creating a centralized geoportal populated by a collection of harmonized geospatial metadata records. the participating institutions divided work up geographically, with each institution collecting and editing metadata for openly available collections of geospatial data in their region. working with the project’s metadata coordinator, the institutions created iso 19139 records for geospatial data and dublin core records for scanned maps. in the summer of 2016, the project launched the btaa geoportal, built with geoblacklight, an open source geospatial discovery application. an interface steering group convened in fall 2016 with the charge to assess the interface and usability of the btaa geoportal. the team included six members from five institutions of the task force and a usability student assistant. they focused on determining perceived usefulness and perceived ease of use, which are important factors in determining acceptance of new technologies (saade and bahl 2005; adams, nelson and todd 1992; baroudi, olson and ives 1992). usability tests and heuristic evaluations were chosen as methods of evaluation, as they have had a long standing in measuring and managing website engagement and are essential in the process of iterative design (letnikova 2003; campbell 2001). giving back to the geoblacklight community provided another motivation behind conducting the usability study of the btaa geoportal. as open source software, geoblacklight relies on a robust community of developers to contribute to the code base. because the btaa geospatial data project does not include a full-time developer, our project has not been able to contribute very much technical development to the software. our project hopes to give back to the community by publishing the results of our usability findings with the hope that it will benefit other portals built with geoblacklight. the group began with an important question: who is/are the target audience(s) of the geoportal? addressing the needs of diverse users, in part, motivated the task force to select geoblacklight in the first place. this is because geoblacklight distinguished itself among geoportal technology as being designed for an audience beyond advanced gis users (hardy and durante 2014; stanford 2014). the project’s decision to include records for scanned historical maps meant that we anticipated serving users who may not be familiar with gis conventions and terminology. the group discussed this issue at length and decided to see if usability evaluation of the site identified differences in the experience of users with dissimilar backgrounds. usability studies of geoportals the literature on geoportal usability includes articles that lay out the criteria that defines what a geoportal is and how it should function (aditya and kraak 2005; wyttenbach et al. 2008; henzen and bernard 2014). these articles are mainly from europe, which may be a reflection of the inspire directive that requires all states in the european union to harmonize and share specified spatial datasets (i.n.s.p.i.r.e. 2007). this directive resulted in different countries looking to create geospatial discovery applications, and a need arose to document and standardize geoportal functionality. the literature also suggests that there has been limited formal user testing on geoportals. most available studies have been confined to large scale national spatial data infrastructures, including sweden (he et al. 2012), australia (barton et al. 2015), switzerland (kellenberger et al. 2016), and brazil (de araujo et al. n.d.). resch and zimmer (2013) employed a survey method to interview users of multiple geoportals across the european union. an exception to this pattern was a test performed at the university of new hampshire library on the open source application, opengeoportal, instead of an established geoportal (wolff and parker 2014). in each of these studies, gis professionals were the test subjects. many were even already familiar with the specific geoportal applications being tested (resch and zimmer 2013; he et al. 2012). although these studies fully documented their testing methodology, the problems they uncovered were mainly confined to minor interface design issues and did not affect the primary features of a geoportal, namely the combination of graphical spatial, categorical, and text searches. heuristic evaluation heuristic evaluation provides a way to assess a user interface against established principles of usability. the members of the interface steering group completed a form designed to guide us through some of these standard usability issues and evaluate our interface. the group produced an overall score of 82/100. despite a relatively high score, the heuristic evaluation identified some usability issues. the keyword search proved the most problematic in the heuristic evaluation as searches often did not return expected results. conducting the heuristic evaluation also sensitized the members of the group to the user experience with the geoportal and helped us prepare for user testing. (see appendix ii for the full heuristic analysis.) developing and administering the user tests the interface steering group completed a total of 16 user tests, including users with and without gis experience, and representing the span from undergraduate student to advanced researcher. three institutions completed the user testing on site: university of michigan, university of minnesota, and michigan state university. task development for the tests proved challenging. we wanted to assess various features of the geoportal, but were aware of the likely knowledge gap between different users. we found writing tasks that provided generous background information without making assumptions about someone’s experience with gis impossible, because suitable user tasks should provide all of the context that the user might need to complete it without giving unnecessary hints. therefore, users were split into two groups: those without gis experience and those with gis experience. levels of expertise with gis varied. we developed two very similar, but not identical, sets of tasks for each group with the intention of enabling a high probability of task completion while still identifying potential differences between them. (see appendix iii and iv for the user testing tasks.) users’ tests took approximately one hour to complete. most tests were conducted by two people: one serving as moderator and the other as note taker. in some cases, only one person was available to administer the test and served both functions. the moderator asked a few introductory questions to determine the correct set of tasks. the user then completed the tasks without intervention from the moderator or notetaker. if users got stuck, the moderator prompted them to do what they would do at home or in their office. users were asked to indicate when they had completed a task; the moderator often asked the user to rank their confidence that they completed the task correctly on a scale of 1 to 5. after completing the tasks, the moderator debriefed the user, followed up on issues related to tasks, and asked the user to provide feedback on each of the three main sections of the geoportal: home page, results page, and item page. during this debrief period, the moderator and/or note taker answered any questions from the user about the interface or the larger project. user test results and observations we collaboratively analyzed our results, first at our local institutions and then collectively. an exercise using post-it style notes to organize findings provides a common way to identify trends in results of user testing and serves as a debriefing mechanism for the researchers. because other members of the group performing the tests work in distributed geographical locations, a physical session would not work. we instead held a virtual meeting and used trello, an online project management tool, to analyze and combine findings for user tests. we created entries in four categories: observations, insights, questions, and recommendations. trello offered helpful features for organizing and prioritizing the findings of the user tests. figure 1: screenshot of trello project board site structure users expressed unanimous praise for the visual design, color, and aesthetics of the site. several individuals commented that the site was straightforward to use and that the simplicity of options made it easy to learn. an ignored design feature was the use of icons for institution, access, and data type. users did not know what these icons are supposed to represent, and none of the users discovered their explanatory hover labels. users navigated the site by performing searches and occasionally clicking the site logo to return to the homepage. no one clicked on the main toolbar to view the about, help, or history pages. many users also did not notice the navigation options for “back to search” and “start over.” figure 2: screenshot of the homepage during testing figure 3: screenshot of search results page during testing figure 4: screenshot of item page during testing search strategies the geoportal offers three main search functions: a text box, a map inset, and categorized facets. although all three options are available from the homepage, every participant chose the text box as their first and primary search strategy. in many instances, this strategy returned limited results. the most common cause of a failed text search was after typing in a place name or phrase that did not exist in any of the metadata records. when this occurred, users were often stymied and did not know how to proceed. for example, one user searched for “ann arbor, mi.” the state abbreviation “mi” did not appear in any metadata records, causing surprise on the user’s part that the geoportal did not seem to include items for the city of ann arbor. on their own, users did not initially identify the map inset as a search tool. after the map was pointed out and its functionality explained, users did take advantage of it to browse for items. only one user independently identified that the map inset offered another way to search on the results page, although they initially did not see it on the homepage. the user said that the text “search when i move the map” next to a checkbox that appears on the results page looked “just like yelp.” this text allowed that particular user to learn that the map search on the homepage provided the same search functionality. at the time of testing, the geoportal listed ten facets stacked upon each other on the results page in this order: institution, collection, subject, author, place, publisher, year, access, data type, and format. the first three facets were expanded by default, and users rarely clicked on any others. some of the labels on the facets were not intuitive to users, particularly in the cases of “institution” and “collection.” users often selected values from these facets anticipating that they would filter the results to a location, but these values represent administrative metadata that did not provide meaningful information to the users. one user even said, “the institution and collection part were not helpful.” users found the third facet, subject, helpful for exploring the overall available content. however, these keywords had not been standardized for capitalization, which negatively affected scrolling through the full subject list, since the same or similar keywords appeared multiple times. although the year facet appeared farther down the list, several users did access it. they struggled with its default sort, which lists the most commonly occurring dates on top. users expected the sort to appear in chronological order, and felt frustrated by its lack of range setting. lastly, many users did not notice that they can combine facets to narrow down the items returned. as a result, they also did not realize that as they clicked on facets, the system “remembered” selections, and this behavior could cause subsequent searches to fail. item page evaluation when a user selects a record, the item page view appears with three sections: a list of metadata elements and values, a tool box with links for external metadata files, web services, and/or direct dataset downloads, and a map preview that shows bounding boxes, web service previews, or a scanned map image. some users encountered challenges understanding these options, especially when any of the tools or map previews were missing. users were especially confused by records that did not provide a direct download link and expressed annoyance when they realized that they needed to visit an external site to obtain the data. the btaa geoportal combines both gis datasets and scanned maps in the same application, bringing up another point of confusion for users. several users misinterpreted the map preview feature and incorrectly concluded that it was the actual resource, instead of a basemap layer. as a result, some users could not discern if they had found a shapefile or a scanned map, nor did they realize that there is a distinction. metadata view users generally found the available metadata to be inadequate. they wanted to see more robust descriptions, as well as a feature catalog listing the attributes. without this information readily available, several users expressed suspicion of the quality and provenance of the data. none of the users found the “metadata” link useful, because it displays a preview of the metadata in xml format. one user, upon seeing the metadata as xml, said “i know this is important, but i can’t make sense of it. i wish that there was something else here.” usage analytics the interface steering group also examined google analytics reports to check for conflicts with their observations of the user tests. although the site is still fairly new, many trends shown by usage statistics complement and reinforce what we learned from observing users. for example, google analytics shows that the vast majority of text searches are place name keywords. we can also see that users most frequently select only the top three facets on the results page and ignore the lower facets. lastly, an analysis of the visited page urls indicates that users are not taking advantage of the map to filter their results, a behavior we also observed during testing. recommendations for site improvements user testing generated specific recommendations on the three main areas of the geoportal: homepage, search results page, and item page; as well as changes to metadata. each recommendation includes how much effort would be entailed in implementation (easy, medium, difficult); the priority of implementation (high, medium, low); and the group responsible for the task (the btaa group or geoblacklight developers). metadata 1) normalize element values for keywords, place names, and authors/publishers. while geoblacklight is our geoportal interface, the actual metadata records are stored in an index built with solr, a standalone enterprise search server. with our initial solr configuration, “inland waters” and “inland waters” would not be grouped together, nor would “philadelphia, pa” and “philadelphia, pennsylvania.” rather than adding numerous “rules” to solr, we determined that the best practice would be to normalize spellings and punctuation before ingest. this practice also improves interoperability with other catalogs that might not have configured solr to group these terms. effort: medium priority: high task owner: btaa 2) add synonym files to solr to facilitate discovery. solr offers the option to incorporate a defined synonym file. this would expand the search results to include similar or equivalent terms. for example, a user search for “bike” would also return results with the word “bicycle.” effort: medium priority: medium task owner: btaa 3) encourage contributors to submit robust entries into the description field. many users prefer to glean additional information from a condensed description field written in a narrative format, rather than sorting through elements. effort: medium priority: low task owner: btaa homepage 1) remove problematic facets and replace them with place, data type, and subject on the homepage users struggled to understand the categories institution and collection. we recommended replacing them with place and data type, because they were of the highest interest to users and were also the facets targeted for metadata standardization. effort: easy priority: high task owner: btaa 2) put more suggestions on a failed search page when a user typed a term into the search box that yielded no results, they did not know what to try next and might have given up. a message giving suggestions for alternate spellings or to browse certain facets could keep a user engaged to continue searching. effort: medium priority: low task owner: btaa 3) make the map search more prominent and more intuitive. the tests made it clear that the map search was not useful, but we were challenged to determine the best solution to this problem. the full page width map was located under the fold during the tests, so it made sense that users did not notice it. however, moving it higher up on the page would require it to share real estate with the text search and facets, potentially rendering it too small for a positive user experience. we also wanted to address the challenge of navigating the map with just pan and zoom controls. using these features only works if you already know how to navigate to the place you are searching for, but test users weren’t necessarily familiar with the location of a small city like ypsilanti, mi. we considered adding a text box within the map that would just zoom to locations, but multiple search boxes have tested poorly in other usability studies, as users have difficulty differentiating between them (nivala et al. 2008; wolff and parker 2014). finally, we realized that because the map does not provide a visualization of the search results, it does not convey much meaningful information, especially for users without strong prior knowledge of a particular geographical region. we decided to refer this problem to the geoblacklight developer community to glean more insight with further research and experimentation. effort: high priority: high task owner: geoblacklight developers search results page 1) improve the temporal faceting options. users found the year facet difficult, because it displayed by numeric count instead of chronological order. however, we were challenged by solr to display this facet chronologically without affecting the other facets. we investigated inserting a time slider widget, but did not find one that was suitably intuitive. we settled on recommending a time period facet that groups items into selectable centuries, decades, or other predefined eras. effort: medium priority: high task owner: btaa 2) reorder and expand the facets on the left sidebar menu, making the first four: place; data type; subject; time period. users only clicked on the the expanded facets listed first and ignored the folded ones lower down. similar to observations from the homepage, users found institution and collection confusing. we selected the top four most useful facets, place, data type, subject, and time period, and set them to display as expanded by default. effort: easy priority: high task owner: btaa 3) make search options clearer on the top of the page and buttonize the text “start over.” sites built with solr can offer powerful search filtering options by allowing the user to combine multiple facets. however, the selected facets values need to be prominently displayed for this functionality to be recognizable. in our tests, many users did not notice that their previously selected facet values were being “remembered” in the interface. more emphasis on the “start over” button and the facet values list should make the search function clearer. effort: easy priority: medium task owner: btaa 4) increase set results page default from ten to twenty. many users expressed a desire for more results per page in order to minimize having to load another set. twenty offers more results without distorting the map inset on the search results page. effort: easy priority: medium task owner: btaa item page 1) display human readable metadata. many users asked for metadata that was more easily understood than an xml format. more understandable metadata would assist the user’s selection of appropriate data, as well as alleviate some suspicion of geospatial data. effort: medium priority: high task owner: geoblacklight developers 2) have a consistent download button/or substitute for download button. once users learned where to find the download button on the right side of the item page, the records missing a direct download button confused them. we should strive to have a download option whenever possible. when providing this is not possible, a substitute in the same place in the interface would help users. effort: hard priority: low task owner: geoblacklight developers 3) add text that indicates what is being shown in the preview box. many users did not understand what the map preview with a blue bounding box indicated, particularly those less experienced with gis. effort: hard priority: low task owner: geoblacklight developers 4) include thumbnail for scanned map if international image interoperability framework (iiif) is not available. many institutions do not have iiif, and their map records have only a bounding box in the preview window. given the confusion about the bounding box previews, adding support to use a thumbnail preview for scanned maps could help distinguish them from geospatial data. effort: medium priority: high task owner: geoblacklight developers 5) add functionality for supporting documentation, including attribute table information, a codebook, data dictionary, or other supporting files. many users wanted to find this information directly in the geoportal. while the primary purpose of the btaa geoportal is discovery and access, we also want to facilitate data use whenever possible. users expressed the desire to see this information and indicated that it would help them identify that a resource contained the data that they need. this is another step that may help alleviate suspicion of geospatial data. effort: medium priority: low task owner: geoblacklight developers reporting and implementing recommendations taking the next steps of reporting and implementing our recommendations required an assessment of the extent of each of the changes and who will make them. some changes would be implemented only to the btaa geoportal, while some would require development on the geoblacklight codebase. local changes local changes were those that fell under the category of customizations, and included adjusting the facet order on the homepage and search results page, normalizing the metadata for place and subject keywords, adding a solr synonym file for place names, highlighting search options, increasing the default number of search results per page, and inserting a new time period facet. these changes made searching by place and time easier and more prominent in the interface. these issues were tracked within our own project’s public github repository, where they can optionally be adopted by other geoblacklight users. homepage facets during testing and new options after metadata cleanup figure 5: before figure 6: after search results page facets during testing and new options after metadata cleanup figure 7: before figure 8: after start over button during testing and after enhancement figure 9: before figure 10: after codebase enhancements geoblacklight is open source software shared on github, and the main developers welcome collaborative development. our recommendations that would require modifications to the codebase were reported as potential “enhancements” to the geoblacklight github issues tracker. these issues were reviewed by the geoblacklight developer community during a code sprint during the summer of 2017 and selected for inclusion based upon shared need and achievability. this process resulted in improvements in metadata views, more comprehensive recommendations for the metadata profile, and an investigation into the layout and functionality of the map search. conclusions, remaining questions, and future research observing the search strategies across a variety of users revealed that the level of gis experience appeared to have little to no effect on item discovery. instead, success could be predicted based upon the level of experience in research, libraries, or even online shopping. users who were already sophisticated searchers were able to creatively expand their strategies to complete the tasks, even when exposed to unfamiliar gis terminology. in contrast, less experienced searchers found using the geoportal difficult, and might find any type of catalog application challenging, regardless of the discipline. one instance where having a discipline knowledge of geospatial resources was helpful occurred when interpreting the map preview box, which, depending upon the resource type, will display either bounding boxes, web service overlays, or scanned images. although we identified and addressed many usability issues, a number of questions remain, particularly regarding metadata quality and the map search interface. many users repeatedly said that they would only feel confident about the data if they could download and explore it in a gis application. this leaves us with the question of how the btaa geoportal, and geoportals in general, can alleviate this suspicion of geospatial data. while our recommendations, such as providing more readable metadata and a codebook, should help, this poses an interesting topic for future research into the user behavior of geospatial data users. this was one area where the institution facet helped, as some users saw this affiliation as being a sign that the data was vetted by the institution and this made them feel more comfortable with the data. overall, this suspicion was likely compounded by the fact that a significant portion of the available public data in the geoportal was not fully documented in the first place by the source agencies. our metadata remediation efforts can only partially address this inadequacy by supplying basic administrative information and normalization improvements. to more fully mitigate this challenge, the geospatial community would need stronger incentives for more comprehensive metadata authoring practices across the united states beyond the federal level. our study challenged the convention of the map search as it is typically designed for geoportals. most general web maps that augment digital library searching use point clustering (digital public library of america) or choropleth shading (the portal to texas history) to display search results on a map. this approach allows users to browse and click through items on a map to access them. in contrast, geoportals feature a blank base map that needs to be adjusted to define an area of interest. panning or zooming a geoportal map returns a separated text list of items within that spatial extent, but the map itself displays little information about the defined scope of the collection. hovering over a search result within the list will trigger a partially transparent bounding box of that item on the map inset, but the density of available items by place will not be obvious. depending upon the geoportal technology, a user may be able to preview one or more datasets as an overlay on the base map only after an individual item is selected. although starting with a blank map might be the expected behavior for regular geoportal users (kellenberger et al. 2016>), users in our study did not intuitively understand this. further, it conflicts with how they have come to expect how to interact with map interfaces. a key difference fueling this departure is that items in a geoportal are spatially referenced by their bounding box extents, while most web maps representing collections use a single point or polygon centroid to illustrate location. points can be easily aggregated and selected, but displaying multiple bounding boxes on a single map presents a design challenge. a comparison of map interfaces on digital library collections websites figure 11: digital public library of america. map. https://dp.la/map figure 12: portal to texas history. explore locations. https://texashistory.unt.edu/explore/locations/ figure 13: btaa geoportal search results page. the btaa geoportal will continue to grow as the project adds more metadata records. we conducted our usability testing before adding many of these records. notably, after completing the usability testing, the project added over a thousand metadata records for scanned maps from participating institutions’ library collections. the project also plans to add records for licensed geospatial data at some point in the future, which would require authentication for authorized users. these additions will likely impact the user experience of the btaa geoportal. to ascertain how implemented recommendations impact the usability of the interface, as well as to evaluate the inclusion of additional types of records, we plan to conduct another round of user testing in the future. as noted in our literature review, we found a dearth of studies on usability of geoportals, particularly in the united states. we hope to see future articles addressing geoportal usability, as well as broader research into the user behavior of geospatial data users. while our study focused on the usability of our implementation of geoblacklight, it revealed broader topics to explore further related to the integrated map search and user suspicion of geospatial data quality, accuracy and provenance. references adams, d. a., nelson, r. r., & todd, p. a. (1992). perceived usefulness, ease of use, and usage of information technology: a replication. mis quarterly, 16(2), 227. doi:10.2307/249577. aditya, t., & kraak, m. j. (2005, june). reengineering the geoportal: applying hci and geovisualization disciplines. in 11th ec gi & gis workshop esdi: setting the framework sardinia, june 2005 (p. 92). baroudi, j. j., olson, m. h., & ives, b. (1986). an empirical study of the impact of user involvement on system usage and information satisfaction. communications of the acm,29(3), 232-238. doi:10.1145/5666.5669. barton, j. e., goldie, x. h., & pettit, c. j. (2015). introducing a usability framework to support urban information discovery and analytics. journal of spatial science, 60(2), 311-327. big ten academic alliance geoportal. https://geo.btaa.org/. campbell, n., & american library association. (2001). usability assessment of library-related web sites: methods and case studies. chicago, ill: american library association. de araújo, v. o. h., de carvalho augusto, m. j., da silva py, h., & e oliveira, r. a. a. c. the usability of the national spatial data infrastructure (inde) geoportal. proceedings of the 27th of international cartographic conference. http://www.icc2015.org/abstract,855.html. digital public library of america. map. https://dp.la/map. directive, i. n. s. p. i. r. e. (2007). directive 2007/2/ec of the european parliament and of the council of 14 march 2007 establishing an infrastructure for spatial information in the european community (inspire). published in the official journal on the 25th april. http://eur-lex.europa.eu/lexuriserv/lexuriserv.do?uri=oj:l:2007:108:0001:0014:en:pdf fernández wyttenbach, a., moya honduvilla, j., poveda, b., & angel, m. (2008). first approaches to the usability of digital map libraries. e-perimetron, 3(2), 63-76. stanford university libraries. (may 29 2014) geoblacklight concept design v0.3.0. http://geoblacklight.org/documents/geoblacklight%20concept%20design%20v0.3.3.pdf. hardy, d. & durante, k. (july 01, 2014). a metadata schema for geospatial resource discovery use cases. code4lib journal, 25. he, x., persson, h., & östman, a. (2012). geoportal usability evaluation. international journal, 7, 88-106. kellenberger, b., iosifescu enescu, i., nicola, r., iosifescu enescu, c. m., panchaud, n. h., walt, r., … & hurni, l. (2016). the wheel of design: assessing and refining the usability of geoportals. international journal of cartography, 2(1), 95-112. letnikova, g. (2004). usability testing of academic library web sites. internet reference services quarterly,8(4), 53-68. doi:10.1300/j136v08n04_04. nielsen, j. (1994a). enhancing the explanatory power of usability heuristics. proc. acm chi’94 conf. (boston, ma, april 24-28), 152-158. nivala, a. m., brewster, s., & sarjakoski, t. l. (2008). usability evaluation of web mapping sites. the cartographic journal, 45(2), 129-138. portal to texas history. explore locations. https://texashistory.unt.edu/explore/locations/ resch, b., & zimmer, b. (2013). user experience design in professional map-based geo-portals. isprs international journal of geo-information, 2(4), 1015-1037. saadé, r., & bahli, b. (2005). the impact of cognitive absorption on perceived usefulness and perceived ease of use in on-line learning: an extension of the technology acceptance model. information & management,42(2), 317-327. doi:10.1016/j.im.2003.12.013 wolff, r., & parker, k. (2014). usability testing: open geoportal 2.0. place project. 9. http://scholars.unh.edu/place/9 appendix i: list of participating institutions (year joined) university of chicago (2017) university of illinois at urbana­-champaign (2015) indiana university bloomington (2016) university of iowa (2015) university of maryland (2015) university of michigan (2015) michigan state university (2015) university of minnesota (2015 – host institution) ohio state university (2017) pennsylvania state university (2015) purdue university (2015) university of wisconsin-­madison (2015) appendix ii: heuristic evaluation summary for the btaa geoportal heuristic evaluations were completed by 6 internal members using a standard template based on nielsen’s (1994) usability heuristics for user interface design. minor issues defined as usability ratings above 70% major issue defined as usability rating between 61% to 75% catastrophic issue defined as usability ratings less than 60% features & functionality average usability rating features and functionality meet common user goals and objectives. 4.83/5 97% features and functionality support users desired workflows. 4.67/5 93% frequently-used tasks are readily available (e.g., easily accessible from the homepage) and well supported (e.g., shortcuts are available). 2.93/4 73% users are adequately supported according to their level of expertise (e.g., short cuts for expert users, help and instructions for novice users). 1.90/3 63% call to actions (e.g., register, add to basket, submit) are clear, well labelled and appear clickable. 2.30/3 77% homepage / starting page the homepage / starting page provides a clear snapshot and overview of the content, features and functionality available. 2.60/3 87% the home page / starting page is effective in orienting and directing users to their desired information and tasks. 2.80/4 70% the homepage / starting page layout is clear and uncluttered with sufficient ‘white space’. 2.80/3 93% navigation users can easily access the site or application (e.g., the url is predictable and is returned by search engines). 1.52/2 76% the navigational scheme (e.g., menu) is easy to find, intuitive and consistent. 3.60/4 90% the navigation has sufficient flexibility to allow users to navigate by their desired means (e.g., searching, browse by type, browse by name, most recent, etc…). 2.20/3 73% the site or application structure is clear, easily understood and addresses common user goals. 4.00/5 80% links are clear, descriptive and well labelled. 2.60/3 87% browser standard functions (e.g., ‘back’, ‘forward’, ‘bookmark’) are supported. 3.87/4 97% the current location is clearly indicated (e.g., breadcrumb, highlighted menu item). 1.67/2 83% users can easily get back to the homepage or a relevant start point. 2.00/2 100% a clear and well structure site map or index is provided (where necessary). 0.33/1 33% search a consistent, easy to find and easy to use search function is available throughout (where desirable). 2.88/4 72% the search interface is appropriate to meet user goals (e.g., multi-parameter, prioritised results, filtering search results). 2.93/4 73% the search facility deals well with common searches (e.g., showing most popular results), misspellings and abbreviations. 1.07/2 53% search results are relevant, comprehensive, precise, and well displayed. 2.53/4 63% content & text content available (e.g., text, images, video) is appropriate and sufficiently relevant, and detailed to meet user goals. 5.00/5 100% links to other useful and relevant content (e.g., related pages or external websites) are available and shown in context. 2.00/2 100% language, terminology and tone used is appropriate and readily understood by the target audience. 3.33/4 83% terms, language and tone used are consistent (e.g., the same term is used throughout). 2.40/3 80% text and content is legible and scannable, with good typography and visual contrast. 2.50/3 83% help online help is provided and is suitable for the user base (e.g., is written in easy to understand language and only uses recognized terms). where appropriate, contextual help is provided. 3.04/4 76% online help is concise, easy to read and written in easy to understand language. 2.85/3 95% accessing online help does not impede users (i.e., they can can resume work where they left off after accessing help). 2.25/3 75% users can easily get further help (e.g., telephone or email address). 83% performance site or application performance doesn’t inhibit the user experience (e.g., slow page downloads, long delays). 3.20/4 80% errors and reliability issues don’t inhibit the user experience. 3.20/4 80% possible user configurations (e.g. browsers, resolutions, computer specs) are supported. 1.92/3 64% overall usability score 81.50/100 82% appendix iii: test tasks for users with gis experience task 1: find and download a map of ann arbor from the 1800s. task 2: you are researching the lakes within minnesota. find the approximate number of records that include information about lakes in minnesota that are available. task 3: you are writing an article about wisconsin governor scott walker and want to include maps of who voted for him in the 2010 election. find where to download this information. task 4: you want to make a simple map of all of the counties in pennsylvania. find where to download this information. task 5: you just moved to the city of ypsilanti in washtenaw county and you want to learn more about the area. find the approximate number of records that are available. task 6: you are working on a project where you want to overlay roads with aerial imagery in hennepin county, mn. find the aerial imagery and how to access it, then find the roads data and where to download it. task 7: you are researching the concept of environmental justice and plan to analyze census population demographics and access to public parks for the states of indiana. find where to download this information. appendix iv: test tasks for users without gis experience task 1: find and download a map of ann arbor from the 1800s. task 2: you are researching the lakes within minnesota. find the approximate number of records that include information about lakes in minnesota that are available. task 3: your grandpa, henry jennings, family is selling their farm in sperry township, which is in clayton county, iowa. you think that they bought the property in 1910. can you find a map of sperry township? task 4: you want to make a simple map of all of the counties in pennsylvania. find where to download this information. task 5: you just moved to the city of ypsilanti in washtenaw county and you want to learn more about the area. find the approximate number of records that are available. about the authors mara blake is the data services manager at the sheridan libraries and museums at johns hopkins university. in this position, blake leads the data management services and geographic information systems (gis) services for the library, including the jhu data archive. previously, blake worked on the big ten academic alliance geospatial data project as a spatial and numeric data librarian at the university of michigan. mara is committed to developing user-centered library data services. karen majewicz is the geospatial project metadata coordinator and project manager for the big ten academic alliance geospatial data project. she works at the john r. borchert map library at the university of minnesota – twin cities. amanda tickner is a gis/makerspace/urban planning librarian at michigan state university and previously worked at the research hub in davis library at the university of north carolina chapel hill. she currently serves as a representative of msu to the big ten academic alliance geospatial data project. jason lam is a graduate student at the university of michigan who specializes in human computer interaction and mental health. he works as a research assistant at the school of information where he is engaged in developing new technologies to make information more accessible. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – free and open source options for creating database-driven subject guides mission editorial committee process and structure code4lib issue 2, 2008-03-24 free and open source options for creating database-driven subject guides this article reviews available cost-effective options libraries have for updating and maintaining pathfinders such as subject guides and course pages. the paper discusses many of the available options, from the standpoint of a mid-sized academic library which is evaluating alternatives to static-html subject guides. static html guides, while useful, have proven difficult and time-consuming to maintain. the article includes a discussion of open source database-driven solutions (such as subjectsplus, libdata, research guide, and library course builder), wikis, and social tagging sites like del.icio.us. this article discusses both the functionality and the relative strengths and weaknessess of each of these options. by edward m. corrado and kathryn a. frederick introduction librarians have a long-standing relationship with subject guides. since the introduction of subject-specific research guides in the 1950s (vileno, 2007), the work of defining subjects, selecting resources that will be included, and creating the guides has been time-consuming and often unappreciated. the addition of electronic resources, which frequently change, has made creating and reviewing subject guides even more challenging. while web-based subject guides have significantly improved access, static web pages limit who can update guides and do not lessen the amount of time spent creating and updating. considerable effort has been made since the late 1990s to move away from static, labor-intensive html pages, and toward searchable, database-driven subject guides. in her march 2007 review of literature on pathfinders, luigina vileno (2007) reports that while the initial implementation of a database-driven solution may consume more time and resources, “the streamlined process greatly enhanced productivity thus cutting down on the cost in the end” (p. 448). productivity is enhanced primarily by improving the process of updating style and content. for example, a popular electronic resource may appear on many subject pages, meaning that updating links would involve multiple edits. in a database-driven world, where content is pulled from a central location in real time, that same update is made once. graphical user interfaces allow librarians with little or no knowledge of html to add resources, thus encouraging increased staff participation. opportunities for participation and collaboration can also be made available to students and faculty through the use of weblogs, wikis, and social bookmarking software. though, as chad f. boeninger (2006) discovered when he successfully used web 2.0 tools to present business resources at the ohio university libraries, you can lead a horse to water but you can't necessarily make him drink: even when librarians demonstrate subject guides to students, students do not necessarily use them. librarians must demonstrate the guides’ value to students. in addition to improved workflow and collaboration, database-driven subject guides may give us a way to provide the personalized features and customizable content that users have come to expect. thanks to work done by libraries to create open-source subject guide applications, the availability of open-source course management systems, and the proliferation of free web 2.0 tools, the option to create cost-effective, database-driven subject guides is available to all libraries. open source subject guide applications there are a number of open source subject guides applications available. these web-based applications, while performing some similar tasks, can vary in functionality and underlying complexity. most of these applications require a scripting language such as perl or php and a relational database such as mysql. they also require a web server such as apache. below, some of the more popular applications are discussed. mylibrary mylibrary (http://mylibrary.library.nd.edu/) was one of the first tools developed by a library that could maintain web-based subject guides. development on the mylibrary@ncsu portal began as a concept of the digital library initiatives department (dli) of the north carolina state university (ncsu) libraries. according to eric lease morgan, one of the members of the original project team, the goal was to make it possible for ncsu students, faculty, and staff to “create a personalized profile based on specific academic disciplines.” (morgan, 2006). mylibrary@ncsu was named one of the top technology trends in 1999 by the library and information technology association (lita) and received wide interest in the library world. subsequently it was released as open source software in february 2000. mylibrary, at the time, was more focused on user personalization than on being a pure subject guide tool. however, it was designed in such a way that by selecting subjects, users would have access to materials that were selected by subject specialists for that discipline. the lead developer of mylibrary, eric lease morgan, brought the application with him to the university of notre dame and has since rewritten it. today, mylibrary is now much more than simply a subject guide portal. while creating dynamic web pages, such as subject guides, is still one of the major features of mylibrary, it is now a full-fledged digital library framework and toolbox. librarians have various options to import content into the system. aside from accepting manual data-entry, mylibrary can import marc data and oasi-accessible data. content from the system can be syndicated into institutional-wide portals and other systems, and users can also personalize pages in “my library” that are akin to myyahoo, igoogle, etc. mylibrary is well documented, as evidenced by a 200+ page, two-volume manual. mylibrary is written in perl and requires mysql. it is licensed under the gnu general public license (gpl) version 2. figure 1: mathematics subject guide created with mylibrary: http://www.library.nd.edu/subjects/mathematics subjectsplus subjectsplus (http://www.ithacalibrary.com/subsplus/) is a database-driven subject guide maintained by andrew darby of ithaca college. subjectsplus is designed to dynamically manage three aspects of a library web site. it allows librarians to maintain subject guides, a-z database lists, and a staff list that is sortable by department and subject specialist. subjectsplus makes it possible for librarians to easily add resources of all kinds (even including subject specialists) to subject guides. while not as full-featured as mylibrary, subjectsplus’ simplicity may be welcomed by some institutions. subjectsplus is built using php and a mysql backend. when darby was first looking for a database-driven subject guide application he discovered a program developed at east carolina university's joyner library known as piratesource (http://web.lib.ecu.edu/piratesource/). he liked the core program, but wanted to make a number of improvements and updates to the program. he contacted east carolina university and found out that while they were still using the piratesource application, it apparently had not been maintained since 2003 and did not have any software license attached to it. east carolina university granted darby permission to take the code, improve upon it, and re-license his derivative under the gnu gpl license. like its successor, piratesource requires php and a mysql backend. piratesource is still available for download from east carolina university but since it is not currently being maintained, we encourage librarians who are interested in piratesource to look at darby's recent adaptation. figure 2: communications subject guide created with subjectsplus: http://www.ithacalibrary.com/subjects/comm libdata libdata (http://libdata.sourceforge.net/) is a library web page authoring environment built on top of open source tools. it was developed and first put into production at the university of minnesota for the 2003 fall semester. libdata provides an authoring environment for three different types of web pages, although a library does not need to use all of them. the three different pages are subject guide pages (research quickstart), course related pages (courselib) and general purpose web pages (pagescribe). like many other open source applications, libdata is designed to be very flexible and allows for great local variation. libdata contains approximately forty different database tables and appears to be more complicated architecturally than piratesource or subjectsplus. this means that libraries will most likely require greater local expertise to install and maintain libdata and a successful installation will require more cooperation between librarians and systems staff. the payoff for this complexity is “a rich authoring environment with a great deal of possibilities.” (bramscher, 2005). like mylibrary, one of the strengths of libdata is its documentation. available documentation includes a 32-page introduction, installation notes, and an e-r (entity-relationship) diagram. the most recent version of libdata (2.30) was released in 2005 and includes features such as rss feeds and robust statistics collecting features. figure 3: higher education subject guide created with libdata: http://caddis.rider.edu:8080/libdata/rqs.phtml?subject_id=56 researchguide the university of michigan released its subject guide application, researchguide (http://researchguide.sourceforge.net/), under the mit open source license. researchguide provides a web environment for librarians to use in order to create subject guides and information pages about themselves. researchguide is somewhat less complicated architecturally than libdata, having fewer than ten database tables. although the database table structure is not overly complicated, some librarians have expressed concern about being able to implement the authentication method at their institution (farkas, 2007). some of the benefits of researchguide, like many of these tools, include a “standard look and feel of guides and centralized management of content” (libner, faq). researchguide requires that php and mysql be installed. the creators of researchguide also recommend that systems administrators have some programming expertise in php and mysql as well as a readiness to provide some instruction to librarians and others using the system. the current downloadable version is 0.5, released january 29, 2002. although researchguide has not been updated in more than six years, it is being used in at least six universities in three countries (libner, 2006). figure 4: geography subject guide created using researchguide: http://www.library.cqu.edu.au/resources/databases/index.php?dbbysub=1&subid=11 library course builder the university of rochester libraries has released library course builder (libcb) (http://sourceforge.net/projects/libcb/) as an open source program available under either the gnu library or lesser general public license (lgpl). unlike the subject guide systems previously reviewed in this article libcb requires the use of some proprietary software. while libcb itself is available free of charge, libraries wishing to implement it will need to have a cold fusion server and a microsoft sql backend. if a library has expertise, and licensed versions of cold fusion and microsoft sql available, libcb may prove to be a good choice. libcb, as the name suggests, was created more for building course web pages than subject guides. libcb is designed for academic libraries to use to pull together paper and digital library resources to support all the courses offered on campus. the university of rochester's implementation of libcb is called course resources system. librarians only need to fill out simple web forms to add resources into the system for their course pages. they do not need to know any html or other specialized software. some of the benefits the university of rochester obtained with the course page approach are increased involvement by teaching faculty and pages that “have proven to be an excellent tool for bibliographic instruction sessions.” (gibbons, 2003) comparison of subject guide applications application programming language database utilized current version date of last update software license mylibrary perl mysql 3.0.4 june 2007 gpl v2 subjectsplus php mysql 0.6 oct. 2007 gpl piratesource php mysql unknown july 2003 unknown libdata php mysql 2.3 dec. 2005 gpl researchguide php mysql 0.5 jan. 2002 mit libcb cold fusion microsoft sql 1.1 mar. 2003 gpl v2 social bookmarking sites some librarians have taken to using social bookmarking sites such as del.icio.us to maintain web links for course and subject guides. some libraries simply provide links to the web page automatically created by del.cio.us (http://del.icio.us/) for a user's tag. others get a little fancier and embed the tags using del.icio.us's rss feeds for a tag combined with javascript code designed to convert rss to html. if there is no one familiar with javascript on staff to create a custom rss to html program, an rss–to-html service such as feed2js.org can be used. other social bookmarking sites such as unalog, furl, simpy, and yahoo's my web can be utilized in a similar fashion. figure 5: french language subject guide built using del.icio.us: http://www.tcnj.edu/~library/moulaison/frenchstudies.htm an easy way provided by del.icio.us to include a set of links is to use their linkrolls tool that will create a small snippet of code that can be inserted into an html document. another way to automatically embed items bookmarked using del.icio.us is to take advantage of the json functionality provided by del.icio.us. this is the approach the college of new jersey libraries took with their modern languages subject guides. the display of the links is further customized using cascading style sheets. json is a “lightweight data-interchange format [that] is easy for humans to read and write [and] for machines to parse and generate” (“introducing json“). javascript can be embedded into the html of a web page to dynamically display data that is made available in the json format. because del.icio.us provides the base javascript code and makes the data available in json format, it is not necessary to be knowledgeable about either of these technologies to dynamically include bookmarks tagged in del.icio.us on a web page. however, a good knowledge of cascading style sheets (css) would be beneficial since using css will allow the library to customize the display of the bookmarked links. one downside to this approach is that as the number of different feeds per subject guide increases, page loading tends to slow down. however, one way to help alleviate this issue is to cache the content on a local server. this can be done by creating a small script that automatically downloads the json feed every x minutes. another concern about using del.icio.us that has been raised by meredith farkas (2007) and some other librarians is the inherent danger of relying on a free, third-party service for core functionality of a web site. while it is unlikely that del.icio.us will disappear anytime soon, the risks associated “with del.icio.us being a third-party service can be mitigated by exporting your data – with the api, the rss & json feeds, and in standard browser-bookmarks-format – to make a backup copy of your guides on your website” (gustafson, 2007). blogs many librarians use blogs for both personal and professional communication. some, including chad f. boeninger, a librarian at ohio university who is a business subject specialist, have created subject-specific blogs. boeninger (2006) believed that if he made answers to “popular questions available via the web, then that might help save the students time in their research endeavors [and] in the process of saving them time, the blog might also help to free up some of [his] time, allowing [him] to focus on more advanced research questions.” while this is a relatively common use of blogs in librarianship, using blogs to create and maintain subject guides is considered less often. one library that is using blogging software for subject guides is plymouth state university in new hampshire. plymouth state uses scriblio, which is a customized version of the wordpress blogging software for their subject guides as well as their library opac (bisson, 2007). more information about scriblio can be found at http://about.scriblio.net/. it is possible to do this with traditional blogging software as well since most blogging software can be easily customized and edited without special knowledge. however, if one wants a custom look and feel to match the rest of the library web site, someone with knowledge about how to create and maintain blog themes will be required. themes allow blog maintainers to quickly change the layout and design of their blog. figure 6: graphic design subject guide built using scriblio: http://library.plymouth.edu/by-subject/graphic-design another library that has started working with blogging software to create subject guides is george mason university. they are using a combination of the open soure wordpress blogging platform and the open source collection workflow integration system (cwis) to create research portals. cwis is is software that is designed to “assemble, organize, and share collections of data about resources, like yahoo! or google directory but conforming to international and academic standards for metadata” (cwis overview 2008). george mason's bioinformatics research portal (http://phobos.gmu.edu/melange/) is quite impressive, but as wally grotophorst (2008) points out “a librarian serving as curator for one of these research portals will be doing quite a bit of work–identifying new resources, writing blog entries, chatting with clients, performing maintenance on links and more.” therefore, this approach should be considered in environments where research portal maintainers believe that such an effort is justified. wikis popular wiki platforms (for example, mediawiki, seedwiki, twiki, and pbwiki, among others) have been used by many libraries to create internal resources, and are quickly spreading to the public side of library services. meredith farkas, in her may 2007 american libraries article, points out that wikis can resolve three major problems of static subject guides by “making a subject guide easier to update, searchable, and collaborative” (farkas 2007, p. 33). mediawiki, best known as the software that runs wikipedia, has been used to create subject guides at florida state university libraries, brock university's gibson library, and by the ohio university library's chad boeninger. mediawiki's major advantage is its association with wikipedia, which ensures a strong user base, frequent software updates, a degree of longevity, and a familiar look and navigation system for users. florida state university libraries also liked that mediawiki is based on mysql, which they felt would ease transfer of content between programs. in addition to mysql, php and a web server are required to install mediawiki, which, according to tim ribaric, “can be installed in under an hour” (2007, p. 26). of all the database-driven options discussed here, because of their ease of use, familiarity, and collaborative nature, wikis are perhaps the best option for collaboration among library staff, faculty, and students, though libraries have utilized that collaborative component to varying degrees. figure 7: early childhood education subject guide built using a wiki: http://www.lib.fsu.edu/wiki/index.php/early_childhood_education course management systems beyond the options previously discussed here, another option for some libraries may be to utilize their campus course management system. the supported learning centre at city lit, the largest adult education college in europe, has created subject guides using the city lit's installation of moodle. moodle is an open source course management system that “can scale from a single-teacher site to a university with 200,000 students” (moodle, 2008). city lit's subject guides can be viewed by anyone from http://moodle.citylit.ac.uk/moodle/mod/resource/view.php?id=123. librarians at the pennsylvania state university have created generic subject guides within their angel course management system that provide links to electronic books and library databases (lovett, 2004) and guides are automatically available to students through the course management system. in order to allow librarians to create and edit these guides, a customized and flexible template was created that includes default information and resources that the library can use or modify (snavely and smith, 2003). while angel is proprietary software, there is no reason to believe that similar projects could not be implemented using either moodle or sakai, another open source course management system. in fact, the sakai's sakaibrary project (http://www.dlib.indiana.edu/projects/sakai/) is currently developing a working prototype of a subject research guide component. the sakaibrary project's goal is to free library resources from silos within the library and make them available within the sakai course management system. the subject research guides will aim to meet the needs of three relatively distinct user groups: librarians, instructors, and students. the goal of the research portion of the project “is to design an application to easily produce (and later customize) research guides” (dueber, 2007). conclusion librarians have approached the creation of web-based subject guides in various ways. they have used static html, database-driven subject guides, course management systems, and web 2.0 technologies such as blogs and social bookmarking web sites. each of these approaches have merit and individual libraries may find some approaches better suited to them than others. librarians who are going to be creating and maintaining these subject guides will need to work closely with systems staff that are responsible for installing and maintaining the software to see which package is the right fit for their organization. the decision about which software package to implement should be made after evaluating local infrastructure, technical expertise, the goals of the project, and the functionality desired. while it is true that this is not a one-size fits all solution, there are a few things that seem true when reviewing the literature and while investigating the various options out there. it is important that whichever solution is chosen, that subject guides are maintained and updated on a regular basis. this can include the use of link checkers and other automated methods. however nothing is better than having a person (either a librarian or even a student worker) check the pages manually. if not, they will become stale. database-driven and social software-based subject guides offer non-tech savvy librarians an easy way to edit and keep the subject guides that they are responsible for current. database-driven guides, because of their nature and design offer additional benefits as well. their design typically enforces a standard look-and-feel for the guides. likewise, a resource can be updated in one place and that change will appear throughout the system. this makes maintenance of these guides less time-consuming and more accurate than a traditional static html web page. while all of the guides reviewed offer benefits, when investigating these guides we were impressed with the ongoing development of subjectsplus and its rather straightforward and easy to maintain design (for someone with some technical expertise). however, if a library has a need for a more full-featured solution, they may want to look into a system like mylibrary that is more than a subject guide portal and offers a full-fledged digital library framework. this is not to say the other solutions reviewed don't have technical merits, but the lack of recent updates may cause some concern for libraries without programmers on staff. social software based solutions, such as blogs, wikis, and social bookmarking sites may also be a good solution for smaller libraries that have limited number of staff creating and maintaining guides. they are relatively easy to use and with many free or inexpensive hosting options available, it is not necessary to have a systems administrator on staff. however, these approaches may not offer the uniformity and the “edit-a-resource-once, update everywhere” functionality that solutions like mylibrary, subjectsplus, libdata and researchguide offer. references bisson, casey (2007, sep 6). launch! maisonbisson blog. retrieved february 9, 2008 from <http://maisonbisson.com/blog/post/11923/lamson-library-website-based-on-scriblio-launched>. blackburn, jonathan d., jackson, millie, & mcdonald, robert h. (2007). mediawiki open-source software as infrastructure for electronic resources outreach. the reference librarian, 48(1), pp. 19-36. (coins) boeninger, chad f. (2006). blogs, wikis, and im: communication tools for subject specialists. retrieved january 12, 2007 from <http://www.higheredblogcon.com/index.php/ blogs-wikis-and-im-communication-tools-for-subject-specialists>. bramscher, paul f. (2005, november 18). libdata introduction & functional description. retrieved december 14, 2007 from <http://libdata.sourceforge.net/docs/libdata_intro.rtf>. cwis overview (2008). retrieved march 19, 2008 from <http://scout.wisc.edu/projects/cwis/>. darby, andrew (2006). implementing an open source application in a college library: ecu’s pirate source. college & undergraduate libraries, 13(1), 41-52. dueber, bill (2007, feb 22). research guides and sakaibrary. retrieved janaury 27, 2008 from <http://confluence.sakaiproject.org/confluence/display/slib/research+guides+and+sakaibrary>. farkas, meredith (2007). subject guide 2.0. american libraries, 38(5), 33-33. farkas, meredith, (2007, october 24). the long road towards subject guide 2.0. information wants to be free blog. retrieved november 8, 2007 from <http://meredith.wolfwater.com/wordpress/index.php/ 2007/10/24/the-long-road-towards-subject-guide-20/>. gibbons, susan. (2005). library course pages. library technology reports, 41(3), 33-43. gibbons, susan (2003, march). building upon the mylibrary concept to better meet the information needs of college students. dlib magazine, 9 (3). retrieved from <http://www.dlib.org/dlib/march03/gibbons/03gibbons.html>. gustafson, britta (2007, nov 20). “i think some…” weblog comment. retrieved december 15, 2007 from <http://meredith.wolfwater.com/wordpress/index.php/ 2007/10/24/the-long-road-towards-subject-guide-20/#comment-182988> herzog, brian (2007, july 28). library subject guides using del.icio.us. herzogbr.net blog: a hitchhiker’s guide to fear and loathing at a public library reference desk. retrieved december 9, 2007 from <http://www.herzogbr.net/blog/?p=164>. introducing json (2007). retrieved august 24, 2007 from <http://json.org>. libner, kelsey. frequently asked questions. retrieved december 12, 2007 from <http://researchguide.sourceforge.net/faq.html>. libner, kelsey (2006, february 12). update on researchguide implementations. retrieved december 12, 2007 from <http://sourceforge.net/forum/forum.php?forum_id=539358>. lovett, deborah g, (2004). library involvement in the implementation of a course management system. medical reference services quarterly 23(1), 1-11. (coins) moodle: a free, open source course management system for online learning (2008). retrieved january 27, 2008 from <http://moodle.org/>. morgan, keith (2006). pioneering portals: a history of mylibrary@ncstate. in eric lease morgan, et al., designing, implementing, and maintaining digital library services and collections with mylibrary. retrieved december 9, 2007 from <http://mylibrary.library.nd.edu/documentation/ch/ch28.html>. ribaric, t. (2007). it’s time to use a wiki as part of your web site. computers in libraries, 27(10), pp. 24-29. (coins) snavely, loanne & smith, helen (2003). bringing the library to students: linking customized library resources through a course-management system. paper presented at the association of college and research libraries 11th national conference, april 10-13, 2003. retrieved january 27, 2008 from <http://www.ala.org/ala/acrl/acrlevents/snavelysmith.pdf>. vileno, luigina (2007). from paper to electronic, the evolution of pathfinders: a review of the literature. reference services review, 35 (3), pp. 434-451. (coins) about the authors edward m. corrado is systems librarian at the college of new jersey (tcnj) located in ewing, nj, usa. corrado earned his masters of library service from rutgers university and has been working with library technology for over ten years. his scholarly interests include the use of social networking, emerging technologies, and open source software in libraries. he is also on the editorial committee of the code4lib journal. if you would like to contact him, his web site is http://ecorrado.us/. kathryn a. frederick is the access and electronic services librarian at elmira college (elmira, ny). ms. frederick views libraries as vital social institutions and ponders their changing role in society. she is interested in expanding the scope and quality of library services by, among other things, integrating web 2.0 and open source tools. ms. frederick is active in the new york library association (nyla), currently serving as president of the nyla smart section. visit www.kathrynafrederick.org for more information. subscribe to comments: for this article | for all articles 12 responses to "free and open source options for creating database-driven subject guides" please leave a response below: jane nichols, 2008-04-02 thanks for your article! very helpful & informative. i especially liked your analysis about which solution to consider based on an institution’s staffing (and other) resources. kudos! philip king, 2008-04-02 this is a very interesting article and very timley for me personally. in your opinion where do tools such as the following list fall? fedora http://www.fedora-commons.org/ dspace http://www.dspace.org/ greenstone http://www.greenstone.org/ ncore http://wiki.nsdl.org/index.php/ncore edward m. corrado, 2008-05-07 i think digital library/repository software like the programs you mention can play an interesting role with subject guides. however, i’m not really familiar with anyone using them as a subject guide. i have seen some sites that appear to use digital library software for electronic reserves though. i think the main issue with using one of these programs as a subject guide tool is copyright. they are typically designed to store information instead of linking to it. as we have seen by the recent lawsuit at georgia state university, this method would probably not be appreciated by publishers (and besides, this approach would not be able to deal well with constantly updating information). all that said, maybe with some code modifications someone could make one of these tools into a pretty good subject guide tool. hazman aziz, 2008-07-04 have you guys explore gregarius (http://gregarius.net). i can’t get hold of feed library by blogdrive. so, i used gregarius as an alternative feeds directory to provide users the listing of my database feeds. ben mccallum, 2009-10-10 thanks, a very interesting article. similar in many respects to the blog post i have written. you have covered a huge range of resources available to librarians and this could come in very handy for an assignment i am soon to complete for university. look forward to sharing valuable resources with you. -ben mccallum eve isk, 2010-06-02 i think the main issue with using one of these programs as a subject guide tool is copyright. they are typically designed to store information instead of linking to it. as we have seen by the recent lawsuit at georgia state university, this method would probably not be appreciated by publishers (and besides, this approach would not be able to deal well with constantly updating information). all that said, maybe with some code modifications someone could make one of these tools into a pretty good subject guide tool. ha social book marking | library 2.0, 2011-10-08 […] libraries can use bookmarking as a type of cataloging method for web sites. the article titled free and open source options for creating database-driven subject guides discusses the use of social bookmarking in libraries and provides a link to the college of new […] angel luis calle, 2011-10-25 una vez mas jesús gracias por tu amabilidad de actualizar contenidos de software esenciales para el trabajo cotidiano. es un excelente contribución en el día del bibliotecario abrazos ángel luis calle delicious social bookmarking, libr246-13bell | library 2.0, 2011-10-25 […] libraries can use bookmarking as a type of cataloging method for web sites. the article titled free and open source options for creating database-driven subject guides discusses the use of social bookmarking in libraries and provides a link to the college of new […] regeniaec, 2012-05-22 this will surely work for any library in work, to be organize that much so no one will be lost and take their time when searching for a certain book. free and open source subject guide software | bibliographic sketches, 2012-11-28 […] free and open source subject guide software posted on november 28, 2012 by zamiel free and open source options for creating database-driven subject guides […] any other resource guides out there? – randy's bean bag, 2017-03-13 […] in searching for articles that might help me in my search for similar resource guide platforms, i found only a few sources.  the first article was somewhat dated, but it’s incredibly informative and helped me track down some of the efforts to create new platforms.  here is the link to the article: free and open source options for creating subject guides […] leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – teaching ai when to care about gender mission editorial committee process and structure code4lib issue 54, 2022-08-29 teaching ai when to care about gender natural language processing (nlp) is a branch of artificial intelligence (ai) concerned with solving language tasks by modeling large amounts of textual data. some nlp techniques use word embeddings which are semantic models where machine learning (ml) is used to learn to cluster semantically related words by learning about word co-occurrences in the original training text. unfortunately, these models tend to reflect or even exaggerate biases that are present in the training corpus. here we describe the word embedding navigator (wen), which is a tool for exploring word embedding models. we examine a specific potential use case for this tool: interactive discovery and neutralization of gender bias in word embedding models, and compare this human-in-the-loop approach to reducing bias in word embeddings with a debiasing post-processing technique. by james powell (0000-0002-3517-7485), kari sentz (0000-0002-1530-1952), elizabeth moyer (0000-0003-1604-6049), martin klein (0000-0003-0130-2097) introduction in the 1800s, the author mary ann evans wrote under the pseudonym george eliot due to gender bias and widely embraced stereotypes about female authors. due to the recent pandemic, online video conferencing replaced face-to-face interactions. in this isolated and impersonal space, many chose to overtly assert gender identity by specifying their preferred personal pronouns. today we are now much more inclined to embrace our gender identity, both personally and professionally. but the passage of time does not erase the past, for which we have a vast digitized written record. so we face a new dilemma, as we become more dependent upon machine learning in our daily lives, we run the risk of unseen bias having unforeseen consequences. so we need to neutralize biases, such as those associated with gender that are perpetuated in machine learning models. to address this issue, the emerging consensus suggests we need to inspect and adjust for bias that has made its way into ml models. in other words, we ought to adjust our models, rather than our stories. what are word embeddings? word embedding models encode semantic information about words that they learn based on the contexts of those words in unlabeled (raw) text. there are various ways to create word embeddings, but the results are typically a word matrix of 50-300 numeric values per word that encode information about the context of each word in the corpus. these values represent the location and magnitude of a vector describing each word, which can be used to measure proximity and distance among terms in the word embedding space. the two most common approaches for generating word embeddings are word2vec (mikolov et al. 2013) and global vectors for word representation (glove) (pennington et al. 2014). glove generates word vectors from a large matrix that contains counts of word co-occurrences across the entire training corpus. word2vec generates a model of co-occurrence patterns in text which is built up as it slides a fixed length word “window” over the sequence of text from the corpus. this creates a model that can predict a word or a sequence of words given another word, or words, as input. the model is encoded as a list of words in the corpus, and a sequence of numbers (a vector) that is an approximate numeric representation of word co-occurrences found in the corpus. vectors that represent similar words will have endpoints that are closer to one another (figure 1). these vectors can be evaluated to determine just how similar two words are by using a metric such as cosine distance, which is a simple geometric measurement of the angle between two word vectors. when this angle is small, it means that two words are semantically (or in some cases, syntactically) similar.                       figure 1. principal component plots for the terms art, science, and religion showing their nearest neighbors in the new york times corpus at left, in googlenews at right. word embeddings have characteristics which make them suitable for both intrinsic tasks such as measuring word similarity, and extrinsic tasks by serving as input to machine learning (ml) pipelines designed to solve nlp problems (whitaker et al. 2019). extrinsic nlp tasks include predicting whether a text segment is positive or negative (sentiment analysis), assigning a category to a document (such as a subject heading), or determining whether one sentence logically follows another (entailment). one way to solve these kinds of problems is via supervised ml. supervised ml requires labeled training data, that is, explicit examples of data that represent learnable patterns. labeled training data for automated sentiment analysis might consist of movie review texts labeled as positive or negative. given this training data, an ml algorithm can learn the patterns that constitute positive and negative sentiment (a good or bad movie review). one type of ml algorithm is a neural network. neural networks are written to learn patterns in data, rather than to solve a specific problem. given large amounts of training data, a neural network can learn a model that can make predictions about new, previously unseen data. word embeddings can be used to speed up training of neural networks when the training data contains text, because word embedding models already contain useful information about text, such as which words have similar meanings. using a pretrained model to support training of another model is referred to as transfer learning, because it enables re-use of an existing ml model to facilitate training of another model for a different task. transfer learning also speeds up training of new predictive models since the neural network no longer needs to relearn everything from scratch. for many deep learning/nlp tasks, it is standard practice to download pre-trained models such as googlenews, common crawl, and wikipedia word embeddings models and use them for deep learning tasks. since pretrained models are trained on extremely large corpora, they often contain superior word co-occurrence vectors to those that would be found in a smaller model. large word embedding models work well if there is enough overlap between the vocabulary of the model and the vocabulary of the text of the local training corpus. if there is a significant mismatch, where there are a lot of vocabulary words not found in the embedding model, then large word embedding models are less beneficial. in those cases, researchers will train local word embedding models. this might be done for example in cases where an embedding model for the native language of the text does not exist, when the vocabulary used in the corpus is highly specialized, such as is the case with genre specific text or scientific publications, or if they want to work directly with a particular word embedding model or set of models, as is the case with diachronic text analysis. diachronic text analysis involves using word embeddings that include a time dimension, allowing for measurement of change among words over time. the dukweb project (tsakalidis et al. 2021) is a large-scale effort to generate pre-trained diachronic word embeddings for the contents of websites in the .uk subdomain retrieved from the internet archive, allowing researchers to study the changes in relationships and meaning among words in this corpus spanning 1996-2013.   when word embeddings go wrong the common crawl [1] word embedding model is trained on one of the largest corpora that is publicly available. the corpus consists of 2.96 billion active and defunct web pages which have been collected since 2008. a 2022 study entitled “based on billions of words on the internet, people = men” (bailey et al. 2022), found that in a word embedding model trained on common crawl, the cosine distance between word embedding vectors for “men” and “people” was smaller than the cosine distance for embedding vectors for “women” and “people.” validation of this hypothesis inspired a second line of inquiry based on a follow-on hypothesis that, given the model’s conflation of “people” with “men”, gender stereotype associations in this model would be asymmetric. they also found evidence to support this. their results showed that women were more likely to be associated with gender stereotypical language than men. in 2016, a group of researchers wrote a landmark paper (bolukbasi et al. 2016) about the prevalence of gender stereotypes in large word embedding models, and proposed techniques for addressing it. they first demonstrated this problem via analogies mapped to simple vector math, showing that many common gender stereotypes were present in word embeddings trained on the google news corpus. they enlisted crowdsourcing to identify two sets of 100 pairs of words that could be used to identify definitional (e.g. she, grandmother) and stereotypical (she, secretary) gender associations. next, they used ten of these gender pairs to identify what is in essence a direction for gender. their concept that gender might be associated with a direction in a word embedding model inspired them to propose several debiasing techniques for word embeddings. in one approach, they use gender-specific words to find a vector representing a gender direction common to the terms. using this “gender” vector, they identify a second vector that is orthogonal to it. they then perform debiasing by projecting (move) gendered words onto this orthogonal vector. this eliminates the gender direction from the terms, which they suggest results in the neutralization of gender in these word embedding vectors. this causes words that previously had a strong association with one gender to be equidistant from gender words. for example after the adjustment, “secretary” would be the same distance from “he” as it is from “she”. their second approach aims to preserve gender associations while reducing bias. for example, this approach ensures that “grandmother” retains a relationship to “female” gender, and “grandfather” to male gender. it then adjusts terms that might have a gender association learned from bias in the corpus. this might cause the terms “hiking” and “baking” to be adjusted so that they are equidistant from “grandmother” and “grandfather.” this is a much simplified description of their work, and this paper is definitely worth a read in its entirety. it is important to note the central role of human judgment in the research described above. the authors are careful to note that human judgment plays an important role in their approach to identifying bias and debiasing word embedding models. they broadly observe that “gender associations vary by culture and person.” they also discuss at length the process of manually creating gender word pairs suggesting that “the choice of words is subjective and ideally should be customized to the application at hand.” we will revisit these issues shortly. gender bias has historically had a significant impact on many aspects of life, so the identification of gender bias in data that is so central to many machine learning algorithms, and the idea that this bias could possibly be neutralized was an extremely important step for the field. this opened the door for the possibility that other forms of social bias (garg et al. 2018) in this data could be identified and mitigated. but social biases are not the only problems that can manifest in word embeddings. fixed word embedding models trained on large corpora will inevitably encounter words that have multiple meanings, or senses. this results in a diffuse representation that does not represent any sense of the word very well. for example, the word “bank” has several distinct uses, including “financial institution” and “the land adjacent to a river.” if a training corpus includes references to both senses of “bank”, then the resulting embedding vector will represent the merger of what are effectively two semantically distinct words represented by the same string of characters, which have different word neighbors and usages. in other cases, the quality of some word embedding vectors may suffer due to a paucity of examples in the text. sometimes an embedding model may encode cultural biases, or be heavily influenced by false or misleading information sources, or it may skew toward particular opinions, political positions, or scientific disciplines. word embeddings may also have problems learning consistently meaningful representations for distinct scientific disciplines, particularly for terms that span multiple disciplines. a scientific corpus skewed towards physics and other hard sciences will generate better representations for terminology in those fields, but will offer little to no meaningful semantic representations for other fields such as sociology or linguistics. nor would it be useful for identifying important co-occurrences that might be indicative of multidisciplinary efforts. this problem, perhaps obviously, is more likely to occur for word embeddings generated for a scientifically focused corpus, and it exemplifies the tradeoff between capturing a good representation for a technical vocabulary over modeling science more broadly or depending on a large-scale word embedding model. digital libraries provide wide ranging access to intellectual and cultural artifacts. machine learning models can be used to support many features of digital libraries. they can be used to suggest topics for documents in a corpus, recommend or summarize content, perform automated text classification, and support query formulation and offer search term suggestions, to name a few. if any of these features depends on a model that has problems such as encoded bias, incomplete semantic representations, or term meanings affected by skewed points of view; it could potentially produce unpredictable or misleading or even offensive results. this could be particularly harmful if, for example, some type of bias affects the perception or distribution of historical or culturally sensitive content, or if it results in the inadvertent censoring of certain ideas due to current political sensibilities. it is also important to note that digital libraries can be the source of corpora used to produce word embedding models for various tasks, treating library collections as data (padilla et al. 2019). collection curators are in a particularly good position to identify and mitigate problems with word embeddings trained on their corpora if they can be empowered to do so. the word embeddings navigator we believe that one of the best ways for a human to “explain” bias to a word embedding model is to enable them to inspect that model and give them the ability to iteratively perform targeted adjustments to vectors in that model. the goal of explanation “is to fill in the gap between his audience’s knowledge or beliefs about some phenomena and what [they take] to be the actual state of affairs” [2]. with that goal in mind, we introduce the word embeddings navigator (wen), an interactive web application for exploring and modifying word embedding models. wen facilitates human-in-the-loop interaction with word embedding models to enable iterative query and adjustment of term vectors. although there are other word embedding visualization tools, including conceptvector (park et al. 2017), embedding projector (smilkov et al. 2016), vec2graph (katricheva et al. 2019), and wordbias, a tool which was specifically designed to facilitate bias detection through visualizations (ghai et al. 2021); we believe this is the first tool that combines visual word embedding exploration with the ability for a user to interactively adjust individual word vectors. in the following sections we will illustrate how this works, describe some of the technical details of the system, and evaluate how it performs in comparison to post-processing debiasing. wen was originally conceived of as a tool for exploring and adjusting temporally aligned word embedding models. for example, figure 2 shows a wen-generated heatmap visualization of terms near the word “mars” in temporal snapshots of a scientific corpus. wen was a product of an internally funded research project (sentz et al. 2019). a broader goal of the project was to support discovery of latent knowledge stored in an embedding model (tshitoyan et al. 2019). latent knowledge exists in all corpora but for scientific corpora that span significant time periods within a particular field, there are often hints of discoveries that have yet to be formally declared, and these can be represented by similar contexts that occur at different points in time. the ability to change word embedding vectors was added when it became apparent that small, locally trained models failed to capture relationships that were immediately obvious to subject matter experts, but were not reflected in the source training data. adding a mechanism to change individual embedding vectors fit nicely into the application since those changes would be made to the embedding model in memory, and thus manifested in subsequent user queries. we envisioned that users would make these adjustments based on specialized knowledge such as that possessed by a subject matter expert. how wen works wen is designed to allow for interactive exploration and iterative adjustment of word2vec word embedding models or other embedding models, such as glove embeddings, that can be converted to word2vec format. it emphasizes a four step approach: “search – navigate – visualize – adjust”. all four phases are in service of the goal of surfacing quality issues in word embeddings and making it possible to address these issues directly and immediately within the model. we implemented a prototype of this concept, at the core of which is a capability we refer to as interactive refitting. the prototype consists of the following components: a search interface for providing one or more search terms a textual result interface that presents most similar terms from the word embedding model links to explore these results via three web visualization tools a mechanism allowing users to identify and specify target for adjustment via refitting a web services wrapper for our implementations of the refitting objective an update function that modifies selected word2vec embedding vectors and stores them in the live model the embedding navigator tool makes extensive use of the gensim [3] python library for natural language processing. an instance of the navigator can be configured to load an arbitrary number of related or unrelated word embeddings in word2vec binary or text format, only limited by system memory. at launch time, navigator loads a configuration file, and as a flask application, it then iterates through file system references to the embedding files and loads them into memory. since this tool allows users to update individual word embedding vectors, we load the full word embedding model, rather than just a dictionary of words and their vectors, which would have a smaller memory footprint but would not allow model updates. at run time, wen parses its configuration file to discover which models to load, and there is additional information provided to support the user interface as illustrated by this json metadata excerpt: { 'name': 'nyt 1987-1991', 'filename': 'nyt-1987-1991.model', 'label': 'nyt-87-91' } the name is the full corpus name to be presented to the user, filename is the primary word2vec binary model file to be loaded at runtime, and the model label is used in the ui to provide short labels for interface form elements and various visualizations. a serialized model is loaded from the filesystem using a call to gensim’s word2vec, e.g.: this_model = word2vec.load(models['filename']) where models[‘filename’] corresponds to a file containing word2vec embeddings. metadata about the loaded models is utilized throughout the interface. flask templates ensure that the user can query any or all of the embedding models depending on what aspect of the embeddings they are interested in. since wen was initially conceived of as a tool to explore temporal slices of word embeddings, there are some facilities for visualizing the behavior of a group of words associated with a query across a set of embeddings, usually a temporally aligned collection of models. a sankey diagram illustrates the changes in similarity to the target word in each instance of the loaded embeddings. this works for temporally aligned and unaligned word embeddings as well. it can also be used to view non-temporal shifts among a set of word embeddings. figure 2. wen-generated heatmap for terms most similar to “people” across new york times word embedding models spanning 1987-2006. wen use cases the search interface allows for several types of queries (figure 3). the basic query is for a word or phrase in a single user selectable word embedding model. the query itself is performed using the gensim “most_similar” method of the word embedding model. it computes the cosine similarity of the query word’s vector to all other word vectors in the target embedding space, and returns a list of entries that are the most similar to the query term. somewhat counter intuitively, the most_similar method can also be used to perform a query that involves subtracting the vector of one term from another and then finding the most similar entries to the resulting vector. we implement this as the difference query. figure 3. search interface for an instance of the word embeddings navigator exposing four temporally adjacent word embedding models for the new york times annotated corpus (1987-1991, 1992-1996, 1997-2001, 2002-2006). embedding navigator supports several other methods for interactively querying an embedding model. the analogy search corresponds to a common means of evaluating word embedding models: x is to y, as a is to b. the user provides three of the four elements of the analogy query. in its simplest form this sort of match would be performed using very basic vector arithmetic where the most similar left out term is the term closest to the vector y-x +a. an important characteristic of the gensim library is that it includes implementations of many of the latest state of the art approaches to solving natural language processing tasks. for analogies, gensim’s word2vec class provides an improved method for the analogy task in the form of a method called most_similar_cosmul. this method uses a normalized multiplicative objective called 3cosmul when calculating best matches for the left out analogy parameter. the exact method call looks like this: this_model.w2v.most_similar_cosmul( positive=[term1, term2], negative=[term3], topn=10 ) where x and a are the positive terms, y is the negative term, and topn indicates the number of matches to return. when compared to simple vector math for solving analogy questions with word embeddings, 3cosmul provides a 20% improvement (levy et al. 2014) in correctly identifying analogy relationships. the most_similar_cosmul method is also used to identify terms that are most similar to a pair of terms. although the actual calculation is a bit more complex, in essence, the best matches are those which are closest to the normalized average of the two query terms provided by the user. this is indicated by the inclusion of a positive parameter, and omission of a negative parameter passed to the method. figure 4. a sankey diagram for the term “spacecrafts” from the 1987-1991 instance of the nyt corpus. these four query options form the core of the embedding navigator search functionality. results are available in several forms and the user can specify the number of results they wish to see at query time. the basic results form is a sorted list of most similar terms together with their respective cosine similarity scores. from this listing users can perform several tasks. the list provides click to query access for each term in the results set. there are also three visualization options provided: a t-sne plot of the most similar terms, a sankey flow diagram of first and second order results, and a graph, or network visualization, which also presents first and second order results. since two of the visualizations are graph based and it may not be immediately obvious how this would be achieved, we explain it here. first order results are those which are in close proximity to the original query term. second order similar terms are the set of results associated with each first order search result. thus the farthest nodes in the network visualizations would generally be two degrees from the initial query term. the advantage of providing a network view is that it can reveal instances where a term has multiple connections among nodes in the first or second order result sets. figure 5. a force-directed graph of the term “art” together with its most similar terms in the embedding model. t-sne (van der maaten et al. 2008) is a technique for visualizing high dimensional data which preserves a reasonable degree of variation during dimensionality reduction. the true value of t-sne is realized when visualizing both high dimensional data and a large number of data points. we use a javascript implementation of a t-sne visualization tool which performs the dimensionality reduction in the browser. for small results sets, performance is adequate and we find that the plots it generates accurately reflect the relationships of the embedding vectors. while it is a tall order to expect good performance from browser-based dimensionality reduction of large amounts of high dimensional data, we include this capability because it provides a different perspective on a cluster of word embedding vectors. t-sne remains a state of the art technique for visualizing high dimensional data. as alluded to above, the other two visualizations provided are both based on network visualizations. the sankey diagram (figure 4) shows the query term and two levels of results. the line connecting each result indicates how similar the term is to the term it is connected to. it is implemented using the javascript version of the plotly library [4], so it provides connection information when the user hovers over a result node, which is particularly useful when there is only a small amount of variation in the similarity scores. from the sankey diagram, the user can change the number of search results for the current query to see additional words in relation to the initial search term. the user can also elect to view the result set as a graph (figure 5). the graph implementation is a basic network diagram implemented in d3 [5]. the graph does not utilize similarity scores so edge width is uniform. this graph is intended to provide a comprehensive view of a term in relation to its nearest (most similar) neighbors in the embedding space. the graph uses a force directed layout, which is a layout model that tries to minimize overlap of nodes. the user can interact with nodes in the graph to further reduce overlap in a given region of the graph by selecting and dragging a node. a major advantage of the graph view is that recurring nodes are connected wherever they occur in the result set so that the user can see connections such as cycles and clusters that occur when the same term occurs in multiple results sets. these are relationships which are not depicted in the sankey view. so to summarize, the t-sne view provides an approximate 2-d representation of term proximity, the sankey view links related terms and conveys similarity via node and edge width, while the graph view provides a more complex contextual perspective where first and second order results are visible. adjusting word embeddings with interactive refitting human interaction with word embeddings allows for the injection of human judgment into the model. humans are better at detecting bias and can use their expert knowledge to improve or correct other relationships in the embedding space. we are good at organizing terms according to more nuanced relationships, for example by emphasizing particular word groupings, flagging certain words as orthogonal to other words, reducing emphasis on syntactic relationships, and adjusting for the effects of bias. interaction can occur after embeddings are generated via an unsupervised method. user provided modifications to the embedding space can be directly applied to word vectors or recorded and analyzed for consistency and resolution if re-fitting adjustments from different users target the same word or set of words. there’s an important caveat to the benefits of having a human identify and minimize bias: we all have implicit biases. implicit biases are unconscious stereotypical associations we learn over the course of our life. no matter how hard we try, most of us bring implicit biases to situations and circumstances we encounter, even efforts to address bias. this includes interactively mitigating bias in word embeddings. if you lack awareness of your implicit biases, you might inadvertently substitute one kind of stereotype for another, thereby exacerbating bias rather than reducing it. there are online tests such as those provided by “project implicit” [6] and “learning for justice” [7], which allow individuals to assess their own implicit biases. both tests are based on the “implicit association test” (bertrand et al, 2005) which is discussed in more detail below. it is good practice for individuals engaged in evaluating and mitigating bias in word embeddings to know their own biases beforehand. it is also recommended to incorporate input from multiple individuals when possible. with that in mind, we can now look at how wen enables users to interactively address bias in word embeddings. in their paper “retrofitting word vectors to semantic lexicons” (faruqui et al. 2014) , the authors considered how to improve word embeddings by injecting additional semantic information into previously trained word embedding models. their goal was to improve word embeddings by using relationships specified in a curated lexical dictionary such as wordnet [8] to adjust words that were identified as synonyms, so that their embedding vectors were more similar. they developed a light-weight post-processing approach that searches for embedding model terms in a lexical dictionary such as wordnet, identifies and locates the vectors of any synonyms, and moves the synonym vectors closer to the target term. mathematically, their post-processing reduces the euclidean distance of each synonym vector to the target term vector. they were able to demonstrate that this approach improved representational semantics of the embedding vectors. inspired by this work, we explored replacing data from a lexical dictionary with real-time human judgment. instead of affecting the embedding model by moving synonyms, we implemented a mechanism whereby the user could interactively select words to be moved. our project enables users to trigger one of two types of refitting of user-selected word vectors in the word embedding model using wen (figure 6). one moves a chosen term or list terms closer to a target word where the target word vector remains unchanged, while the other allows the user to indicate that the vectors for a set of terms should be moved closer to one another. we refer to this technique as refitting (powell et al. 2020). refitting enables a wen user to make adjustments to the embedding vectors of selected words and immediately see the results of these adjustments in subsequent queries and visualizations. further adjustments can be made in an iterative fashion until the desired changes are achieved. users can use whatever criteria they wish in order to make refitting adjustments. users are presented with two primary methods for modifying the embedding space: to move a single word closer to a list of related terms (figure 7), or to move a set of words closer to one another (figure 8). these affordances allow users to improve (decrease the distance) between embedding vectors. each user interaction affects the vectors for the selected words. the recalculated word vectors are stored in a separate modified word embeddings table, along with a unique identifier, so that they can be used, analyzed, reclustered, and reconciled as needed. in addition to offering users the ability to refine relationships among words, users can take actions that compensate for bias in word embeddings, for example by moving gender specific pronouns closer to gender neutral pronouns. figure 6. example of the refitting interface which a user has configured to adjust a set of selected terms. as noted above, the original retrofit objective is at the heart of the refitting strategy. we adapted the retrofit algorithm to support interactive word embedding adjustments and exposed it as a web service used by wen. parameters include a set of words and optionally a target word. if a target is not provided, then each provided term will be adjusted using its original embedding vectors and the refitting objective so as to move it closer to all other terms provided. otherwise, only the target embedding will be adjusted. the web service returns the preand post-refitted vectors, as well as cosine distance scores among the terms that were refitted. the target word embedding vectors are loaded into a dictionary from the word2vec model: for idx, key in enumerate(this_model.wv.vocab): wordvecs[key] = this_model.wv[key] the refitting function iterates over the entries in wordvecs from this_model. the new embedding vector will be temporarily stored in castvector castvector = [float(i) for i in wordvecs[key]] once the word embedding vector is adjusted per user input, the refitted version is stored in the target word embedding model: this_model.wv.syn0[this_model.wv.vocab[key].index] = castvector the old embedding vector for the word represented by the variable “key” is replaced with a new refitted embedding vector. this changes the in-memory version of the target word2vec model. to permanently serialize the updated version of the model to the filesystem, we call the save method of the target embedding model: this_model.save(model_filename) when a refitting task is completed, a radar visualization is presented to show how the selected terms were moved in relation to one another, based on old and new cosine distances. figure 7. a before-and-after radar plot illustrating the effects of adjusting the word embedding vector for one term (“mexico”) so that it is closer to a list of other words (names of other countries). figure 8. a before-and-after radar plot illustrating the effects of refitting an entire set of terms. each time an embedding vector is adjusted, wen stores the previous and updated vector in a mysql database, together with the identifier for the target embedding model, an action label, and a timestamp. although this is currently just a logging feature, the plan for this data in the future is to add an interface that would allow a user to “roll back” a selected vector to a previous state. evaluating wen refitting we would be remiss if we were to discuss gender bias issues without acknowledging the various persistent historical biases including gender bias, that have been perpetuated in language for centuries, sometimes causing great harm. references to binary gender identities and to gender stereotypes are based on historical data and are referenced in order to provide clear examples for the purpose of potential bias mitigation techniques applied to data consumed by machine learning algorithms. it is not our intent to perpetuate or endorse any form of bias, or to make any claim that any historical social or cultural norms are superior to current norms. furthermore, we note that the corpus and the examples used in our evaluation are all based on english language terms and publications, including news articles from the new york times [9] spanning 1987-2006. because of this, many of our examples may not be directly relevant to non-western, non-english speaking audiences working with non-english corpora who are considering non-western cultures, historical narratives, stereotypes, or biases. considerable effort has been invested in recent years in characterizing bias, especially gender bias, in word embeddings, for example (brunet et al. 2019) and (garg et al. 2018), and the impact on machine learning tasks such as machine translation (savoli et al. 2021). from a technical perspective, two theories in particular have dominated the investigation of how gender bias is represented in word embeddings: the role of word frequency and the potential existence of a gender-specific component in word embeddings. word frequency has also been found to impede post-processing debiasing. we believe the value of a tool such as wen is that it is agnostic with respect to the way in which bias was represented in a training corpus, or how it was propagated to a word embedding model. it likely would not scale as a production tool for mitigating bias in large embedding models but is intended to be an exploratory platform in which one may perform queries and adjustments in an iterative fashion, enabling users to perform experiments based on theory and intuition. our data set consists of word embeddings constructed from four five year snapshots of the annotated new york times annotated corpus, which contains over 1.8 million articles from the new york times spanning january 1987 until june 2007. we generated local word2vec word embeddings for each five year collection (1987-1991, 1992-1996, 1997-2001, and 2002-2006) after extracting the articles from the original xml source and performing some rudimentary text preprocessing which normalized all text to lowercase and removed most punctuation, except for punctuation which marked sentence boundaries. we used the gensim word2vec library with the cbow (continuous bag of words) model, eliminating any words that occurred fewer than four times in the corpus, trained for 20 epochs resulting in word vectors containing 100 dimensions per term. the sliding window for evaluating co-occurrences was left at its default value of 5, as were other parameters not explicitly listed here: model = word2vec(sentences=sentences, size=100, workers=4, min_count=4, sample=0.05, sg = 0, iter=20, hs = 0) some details about the source corpora and resulting word embedding models, as well as some initial bias scores for the models, are provided in table 1. nyt article corpus year span word count embedding size weat: career and family weat: math and arts weat: science and arts 1987-1991 490380 240723430 436398 1.48 0.84 0.77 1992-1996 391855 178927558 351059 1.64 0.95 1.03 1997-2001 448514 219950197 416824 1.68 1.1 0.94 2002-2006 447227 219969632 426894 1.68 1.04 0.74 table 1: corpus and embedding models overview using wen and refitting, we conducted several experiments to determine if we could in fact reduce gender bias as measured by the word-embedding association test (weat) (caliskan et al. 2017). weat was inspired by the implicit association test (iat), which measured the response time of individuals who were asked to perform a word pairing test: pairing two words they found similar and two words they found different. weat uses the cosine distance between pairs of words “as analogous to reaction time in iat” (caliskan et al. 2017). word pair lists for weat can be defined by the user, but standard sets of words are provided with the toolkit for measuring several types of bias. in the first two tests, a single adjustment was made to a set of terms and the resulting model was evaluated. in the third experiment, we combined the two strategies before measuring the effects. we applied identical strategies to all four models. the resulting weat scores were compared to the hard debiasing strategy described in (bolukbasi et al. 2016). weat scores  corpus 1. pronouns 2. personal names a combination of 1 and 2 hard debiasing using debiaswe 1987-1991 0.68 0.46 >0.49 0.48 1992-1996 0.94 0.89 0.76 0.57 1997-2001 0.86 0.79 0.71 0.6 2002-2006 0.65 0.34 0.51 0.5 table 2: wen refitting strategies versus application of hard debiasing. best results are in bold. the first two experiments were inspired by research into the causes of gender bias and the occurrence of gender stereotypes in word embeddings: the effect of personal pronouns on gender bias (atir et al. 2018), and the role played by the common usage of personal names (johns et al. 2019), specifically first names with respect to gender and science. experiment 1: the “personal pronouns” strategy is based on the notion that personal pronouns may play a significant role in gender bias that is learned by word embeddings. we perform a search for science, and then perform a refitting with “she, her, hers.” we performed the same task for each of the four embedding models and then computed the weat “science and art” bias score for each. experiment 2: the “personal names” strategy. this entailed first performing a search for female personal names in each embedding model. we searched for “mary” which is the most common female name from the last 100 years per the us social security administration [10]. we then selected all entries in this result set and refitted them with the terms “science” and “scientist” and again evaluated the results using weat. in experiment 3 we combined strategies 1 and 2 in order on each embedding model, and then measured the results. table 2 summarizes the results of the experiments. for comparison, we used the word embeddings fairness framework (wefe) [11], for debiasing and gender bias evaluation. the wefe python library is to word embeddings and bias, what gensim is to nlp. it implements a variety of common debiasing strategies and metrics proposed in literature, thus collecting many state of the art approaches to these problems under a single umbrella. we selected wefe’s hard debiasing strategy for comparison to the results of our refitting experiments. using it is straightforward: we first load each of the nyt embedding models as gensim keyedvectors and then apply the hard debiasing strategy as illustrated by this code excerpt: debiaswe_wordsets = fetch_debiaswe() definitional_pairs = debiaswe_wordsets["definitional_pairs"] equalize_pairs = debiaswe_wordsets["equalize_pairs"] gender_specific = debiaswe_wordsets["gender_specific"] hd = harddebias(verbose=false, criterion_name="gender").fit( word2vec_model, definitional_pairs=definitional_pairs, equalize_pairs=equalize_pairs, ) gender_debiased_model = hd.transform( word2vec_model, ignore=gender_specific, copy=true ) three data sets are used by debiaswe hard debiasing to reduce gender bias. the definitional pairs data set includes pairs of words such as “woman” and “man” and “girl” and “boy.” the equalized pairs set is more role and career oriented, and includes pairs such as “king” and “queen” and “brother” and “sister.” the gender specific list appears to be learned from the original corpus and includes many additional social roles, biological gender specific topics such as “obstetrics” and “prostate cancer,” and many personal names. the first set is used to identify what the authors refer to as a “bias subspace.” this is used to determine how to neutralize bias. words found in the gender specific list are omitted from bias neutralization. the remaining terms are adjusted by removing the bias direction identified from the bias subspace. finally, equalized pairs are adjusted again so that they are equidistant from the vector representing the bias direction. results and conclusion figure 9. plot which comparing the results of hard debiasing and several strategies applied using refitting in wen on four temporal instances of the nyt corpus. prior to any debiasing efforts, table 1 illustrates the gender bias scores for several areas of bias commonly found in text corpora: career and family, math and arts, and science and arts. the bias score from the weat metric can range from 2.0 to -2.0, where positive values indicate male gender bias, and negative values represent bias toward women. a score of zero suggests that gender bias does not exist per the test suite, or has been neutralized. we specifically focused on test cases related to “science” and “arts”, as the two topics seem distinct and queries for the two terms showed there was no overlap between them among their 200 nearest terms. strategy 1 yielded only modest improvements, in the presence of gender bias as measured by the weat metric, moving the proverbially gender needle slightly away from male gender bias by between 8 and 12%. the personal names strategy was more successful, but varied dramatically from model to model, with a 13.5% shift for the 1992-1996 model, but a much more substantial 54% reduction in male gender bias for 2002-2006. combining the two refitting strategies yielded consistently good results scoring which were consistently closer to the hard debiasing results than any other approach. meanwhile, hard debiasing achieved better results for two models, and the improvements it made to the models were in general more consistent. figure 9 illustrates the before and after impact of each strategy. future work there are several areas ripe for future exploration with wen. a follow up paper inspired by the concept of retrofitting entitled “counter-fitting word vectors to linguistic constraints” (mrkšić et al. 2016) proposed a post-processing technique to introduce other information from lexical dictionaries. notably this technique would identify antonym relationships and adjust word vectors accordingly. this could be readily adapted to wen. it would be interesting to see how users would apply counterfeiting in a word embedding model. would they focus strictly on antonym relationships or would they take advantage of the feature to reduce the proximity of terms for other reasons? a full implementation of the capabilities enabled by refit logging would be another area of exploration. questions such as how to select refitting actions for rollback, reversing a sequence of refitting tasks, and investigating the impact of selective rollbacks would be possible. related to this would be expanding the somewhat neglected temporal capabilities of wen. would refitting across temporally aligned models (diachronic refitting) be a desirable feature? more generally, wen would benefit from additional methods of visualizing word embedding vectors. for example, the plotly python library supports a large number of visualizations and its performance is quite good. related to this, it would be interesting to further explore what kinds of visualizations would best facilitate diachronic word analysis and latent knowledge discovery across temporally aligned word embeddings. although exploration of temporal word embeddings inspired wen, it lacks sufficient features to really explore words along a temporal dimension. finally, integrating bias metrics would be a highly desirable feature. but as with most debiasing-related approaches, these are designed to be run as post-processing tasks. thus performance would likely be an issue, as would determining which metric(s) to implement. recent research raises questions about the effectiveness of weat in mitigating bias, for example. some debiasing strategies fail to actually remove bias, even though they result in improved bias scores. this raises several questions, such as does wen do better or worse at debiasing? are there other bias metrics that more accurately measure bias? are bias metrics suitable for integration with an interactive application? these are questions that could be explored in a future iteration of wen. wen is primarily intended to be used with smaller, locally trained embedding models such as those based on the contents in an institutional repository, documents from a specific genre, or temporal embeddings where distinct models represent individual time slices of the original corpus. it allows users to test out different potential strategies to make adjustments to word embedding models, whether to neutralize bias, or to otherwise compensate for limitations in the source data upon which the model was trained. locally trained models may have unique forms of bias that may not be effectively addressed by post-processing tools such as wefe‘s hard debiasing. we believe that wen fills a niche in nlp / machine learning pipelines not adequately addressed by any existing tools. references mikolov, t., sutskever, i., chen, k., corrado, g.s. and dean, j., 2013. distributed representations of words and phrases and their compositionality. advances in neural information processing systems, 26. pennington, j., socher, r. and manning, c.d., 2014, october. glove: global vectors for word representation. in proceedings of the 2014 conference on empirical methods in natural language processing (emnlp) (pp. 1532-1543). whitaker, b., newman-griffis, d., haldar, a., ferhatosmanoglu, h. and fosler-lussier, e., 2019. characterizing the impact of geometric properties of word embeddings on task performance. arxiv preprint arxiv:1904.04866. tsakalidis, a., basile, p., bazzi, m., cucuringu, m. and mcgillivray, b., 2021. dukweb, diachronic word representations from the uk web archive corpus. scientific data, 8(1), pp.1-12. bailey, a.h., williams, a. and cimpian, a., 2022. based on billions of words on the internet, people= men. science advances, 8(13), p.eabm2463. bolukbasi, t., chang, k.w., zou, j.y., saligrama, v. and kalai, a.t., 2016. man is to computer programmer as woman is to homemaker? debiasing word embeddings. advances in neural information processing systems, 29. brunet, m.e., alkalay-houlihan, c., anderson, a. and zemel, r., 2019, may. understanding the origins of bias in word embeddings. in international conference on machine learning (pp. 803-811). pmlr. savoldi, b., gaido, m., bentivogli, l., negri, m. and turchi, m., 2021. gender bias in machine translation. transactions of the association for computational linguistics, 9, pp.845-874. garg, n., schiebinger, l., jurafsky, d. and zou, j., 2018. word embeddings quantify 100 years of gender and ethnic stereotypes. proceedings of the national academy of sciences, 115(16), pp.e3635-e3644. padilla, t., allen, l., frost, h., potvin, s., russey roke, e. and varner, s., 2019. always already computational: collections as data. texas digital library doi:10.5281/zenodo.3152934 park, d., kim, s., lee, j., choo, j., diakopoulos, n. and elmqvist, n., 2017. conceptvector: text visual analytics via interactive lexicon building using word embedding. ieee transactions on visualization and computer graphics, 24(1), pp.361-370. smilkov, d., thorat, n., nicholson, c., reif, e., viégas, f.b. and wattenberg, m., 2016. embedding projector: interactive visualization and interpretation of embeddings. arxiv preprint arxiv:1611.05469. katricheva, n., yaskevich, a., lisitsina, a., zhordaniya, t., kutuzov, a. and kuzmenko, e., 2019, july. vec2graph: a python library for visualizing word embeddings as graphs. in international conference on analysis of images, social networks and texts (pp. 190-198). springer, cham. ghai, b., hoque, m.n. and mueller, k., 2021, may. wordbias: an interactive visual tool for discovering intersectional biases encoded in word embeddings. in extended abstracts of the 2021 chi conference on human factors in computing systems (pp. 1-7). sentz, k., powell, j., skurikhin, a., porter, r. 2019. searching for context: microtasking to solve computationally unsolvable problems. ldrd reserve final report la-ur-19-30118. tshitoyan, v., dagdelen, j., weston, l., dunn, a., rong, z., kononova, o., persson, k.a., ceder, g. and jain, a., 2019. unsupervised word embeddings capture latent knowledge from materials science literature. nature, 571(7763), pp.95-98. levy, o. and goldberg, y., 2014, june. linguistic regularities in sparse and explicit word representations. in proceedings of the eighteenth conference on computational natural language learning (pp. 171-180). van der maaten, l. and hinton, g., 2008. visualizing data using t-sne. journal of machine learning research, 9(11). bertrand, m., chugh, d. and mullainathan, s., 2005. implicit discrimination. american economic review, 95(2), pp.94-98. faruqui, m., dodge, j., jauhar, s.k., dyer, c., hovy, e. and smith, n.a., 2014. retrofitting word vectors to semantic lexicons. arxiv preprint arxiv:1411.4166. powell, j. and sentz, k., 2020. interactive re-fitting as a technique for improving word embeddings. arxiv preprint arxiv:2010.00121. caliskan, a., bryson, j.j. and narayanan, a., 2017. semantics derived automatically from language corpora contain human-like biases. science, 356(6334), pp.183-186. greenwald, a.g., mcghee, d.e. and schwartz, j.l., 1998. measuring individual differences in implicit cognition: the implicit association test. journal of personality and social psychology, 74(6), p.1464. atir, s. and ferguson, m.j., 2018. how gender determines the way we speak about professionals. proceedings of the national academy of sciences, 115(28), pp.7278-7283. johns, b.t. and dye, m., 2019. gender bias at scale: evidence from the usage of personal names. behavior research methods, 51(4), pp.1601-1618. mrkšić, n., séaghdha, d.o., thomson, b., gaši?, m., rojas-barahona, l., su, p.h., vandyke, d., wen, t.h. and young, s., 2016. counter-fitting word vectors to linguistic constraints. arxiv preprint arxiv:1603.00892. jo, h. and choi, s.j., 2018. extrofitting: enriching word representation and its vector space with semantic lexicons. arxiv preprint arxiv:1804.07946. gonen, h. and goldberg, y., 2019. lipstick on a pig: debiasing methods cover up systematic gender biases in word embeddings but do not remove them. arxiv preprint arxiv:1903.03862. end notes [1] common crawl. see https://commoncrawl.org/ [2] “teaching method: explanation” a discussion of explanation as a teaching method. see https://en.wikipedia.org/wiki/teaching_method#explanation [3] gensim “topic modeling for humans” software library. see https://radimrehurek.com/gensim/ [4] plot.ly visualization library. see https://plotly.com/ [5] d3 network graph gallery. see https://d3-graph-gallery.com/network.html [6] “project implicit.” see https://implicit.harvard.edu/implicit/takeatest.html [7] “test yourself for hidden bias.” see https://www.learningforjustice.org/professional-development/test-yourself-for-hidden-bias [8] wordnet. see https://wordnet.princeton.edu/ [9] the new york times annotated corpus. see https://doi.org/10.35111/77ba-9×74 [10] “top names over the last 100 years” from the us social security administration. see https://www.ssa.gov/oact/babynames/decades/century.html [11] documentation for “the word embeddings fairness evaluation framework.” see https://wefe.readthedocs.io/en/latest/index.html about the authors james powell (jepowell@lanl.gov) is a research engineer and a member of the digital library research and prototyping team at the research library at los alamos national laboratory, where he has worked for 17 years. previously he worked at the university libraries at virginia tech. his latest book is a librarian’s guide to graphs, data and the semantic web. chandos information professional series. waltham, ma: chandos, 2015. kari sentz (ksentz@lanl.gov) is currently a scientist in the information sciences group (ccs-3) at los alamos national laboratory. she has a ph. d. is in systems science from binghamton university and a master’s in linguistics from the university of virginia. sentz has worked at both los alamos and sandia national laboratories since 2000 researching generalized probability, text mining, risk analysis, probabilistic graphical modeling, and data and information fusion. elizabeth moyer (emoyer@lanl.gov) is a librarian and member of the reference and research services team at the los alamos national laboratory research library. in this role, she helps support researchers by providing research support and is the physics liaison. she has an interest in the social bias of artificial intelligence through directed research conducted while she was a student at the university of british columbia school of information under the direction of dr. richard arias-hernández. she has continued interest and supports research in this domain for researchers at lanl through the development of a resource guide. elizabeth holds a masters of library and information studies (mlis) from the university of british columbia school of information in vancouver, british columbia, canada. martin klein (mklein@lanl.gov), los alamos national laboratory, is a scientist and lead of the prototyping team in lanl’s research library. in this role, he focuses on research and development efforts in the realm of web archiving, scholarly communication, digital system interoperability, and data management. he is involved in standards and frameworks such as memento, resourcesync, signposting, and robust links. martin holds a diploma in computer science from the university of applied sciences berlin, germany, and a ph.d. in computer science from old dominion university. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – connecting the real to the representational: historical demographic data in the town of pullman, 1880-1940 mission editorial committee process and structure code4lib issue 1, 2007-12-17 connecting the real to the representational: historical demographic data in the town of pullman, 1880-1940 the pullman house history project is a part of the pullman state historic site’s virtual museum and web site (http://www.pullman-museum.org/) which links together census, city directory, and telephone directory information to describe the people who lived in the town of pullman, illinois between 1881 and 1940. this demographic data is linked through a database/xml record system to online maps and perl programs that allow the data to be represented in various useful combinations. this article describes the structure of the database and xml records, as well as the methods and code used to link the parts together and display the data. by andrew h. bullen the pullman house history project the pullman house history project, a part of the pullman state historic site’s virtual museum and web site (http://www.pullman-museum.org/) and henceforth referred to as the phhp, links together census, city directory, and telephone directory information to describe the people who lived in the town of pullman, illinois between 1881 and 1940. this demographic data is linked through a database/xml record system to online maps and perl programs that allow the data to be represented in various useful combinations. demographic data collection is particularly important for research into the pullman factory and community because both were a major destination for immigrants throughout the 110 years of the company’s existence, allowing scholars to study 19th and 20th century migration and ethnographic trends. it has also revealed hidden facets of the pullman story, allowing us to discover, for instance, the existence of a local militia extant between 1882 and 1890. a pullman family, mid 1890s the history of the town of pullman in the mid-1870s, george mortimer pullman decided to expand his very successful rail passenger car service, as he had outgrown his detroit shops and needed to dramatically expand his production facilities if he wanted to capitalize on the explosive demand for railroad car sales and service. he chose chicago as a location for his new factory complex. pullman had a fondness for chicago as he had made an earlier fortune there in 1858. he also respected the wide open, entrepreneurial, winner-take-all aspect of the city. pullman began acquiring land on the far south side of chicago in 1880, setting up in secret the pullman land association which was based in massachusetts to avoid land speculation. pullman purchased 3,600 acres of prime industrial property, located between the rock island railroad and illinois central railroad lines and near the natural harbor of lake calumet. he commissioned a young architect named solon beman, 26, to design his factory complex and surrounding town. the pullman area was much more than a rail car manufacturing facility. george pullman wanted to create what he envisioned as a workers’ paradise, charging beman to design and build what was eventually to become 32 blocks of row houses laid out in neat neighborhoods directly north and south of the factory complex. almost unique in their day, each row house was equipped with running water, an indoor toilet, a spacious back yard, and lots of natural light and ventilation. the pullman company provided power, fixed the houses as needed, picked up garbage, and maintained the carefully designed landscaping. pullman also provided shop spaces in a market hall and an indoor shopping arcade; private entrepreneurs could rent from the company and set up grocery stores, department stores, etc. in these spaces. george pullman accepted numerous accolades for his “most perfect town” and made it a chicago show piece during the 1892-1893 columbian exposition. houses on cottage grove under construction, 1881 as a response to cancelled orders and business downturns during the 1894 economic panic, pullman laid off many workers and slashed the wages of his remaining employees. he did not, however, similarly slash rents, waive fees, or in any way attempt to mitigate the dire economic circumstances faced by his employees. the effect of this disastrous decision was the greatest of the 19th century labor struggles, the pullman strike of 1894. the strike, spread nationwide by eugene debs and his attempt to organize all railroad workers into one all-encompassing union, lasted throughout the summer. it was eventually broken with help from federal troops dispatched by president grover cleveland which were sent in to ensure that the mail trains would be kept running. pullman viewed the strike as a personal affront and never forgave the striking workers. he died shortly thereafter in 1897. as a result of a 1907 u.s. supreme court decision, the company was forced to sell the town to private owners, and the great experiment ended. the city of chicago annexed the town of pullman, renumbering the properties and renaming the streets. today, pullman still has 98 percent of its original housing stock. it is a thriving chicago neighborhood, home to an economically, socially, and racially diverse mix of people who deeply love the community. it has been my honor and privilege to attempt to represent the rich history of this fascinating community. challenges in representing pullman history as of this writing, we have identified three main sources of primary historical material that describe the town of pullman: contemporary newspaper articles, images, and census and other demographic data. melding together these disparate data sources into a useful research tool has proven to be challenging. my first step was to cast about for a common thread that could tie these sources together and discovered that they could all be categorized in one or more of three different ways: as people, places, or events. this core assumption provided the main design philosophy for the whole project, and guides the path of future developments. i prefer to program in perl (for no other reason than that is what i am most familiar with) and i am very comfortable working with databases. my approach to developing an online pullman history research tool was, therefore, to use a mysql database to hold the data, using the paradigm of “people/places/events” as the core design guideline, and perl to access the data tables and dynamically build web pages from the results set. information that is unique to a particular data set is housed in external (to the database) xml files, referred to by mysql records and displayed dynamically using perl programs. three classes or groups of tables for the pullman database have emerged: images, demographic data, and newspaper clippings. this article will describe in detail the three demographic data tables, which together form the core of the pullman house history project, and describe how they relate to the image and (coming in the future) newspaper clippings tables. part i: representing the people of the town in order to best represent pullman demographic data in the phhp, i created a database-based/xml record system containing the data from the 1883 and 1889 city directories, a 1916 telephone directory, and the small portion of the 1900 u.s. census that we have entered. eventually, the 1900 census will be completed, as well as adding in data from the 1910, 1920, 1930 and 1940 censuses. the 1890 census was, of course, lost in a fire in new york city; the data in the 1883 and 1889 city directories has proven to be an adequate substitute, allowing us to at least see who lived in which house and what they did for a living. a database/xml record system is an ideal way to maintain disparate and unique data sources and yet still be able to manipulate them in a useful manner. for instance, demographic data in the phhp database consists of records from censuses, city directories and telephone directories, each of which has unique data representation requirements. in addition, each decennial census asked questions unique to that census. gathering together such a seemingly unruly and disparate herd of data at first glance seemed to me to be an impossible chore. upon reflection, i realized that all of these data sources share a few common elements such as names and addresses. a simple mysql table, shown below, became the structure that ties together the common elements. xml files, pointed to from within the individual records, can be used to contain the unique data elements of the different data sources. the main table itself is laid out in the following manner: mysql> describe census; +------------+------------------+------+-----+---------+----------------+ | field | type | null | key | default | extra | +------------+------------------+------+-----+---------+----------------+ | address | varchar(100) | yes | | null | | | aptnumber | varchar(50) | yes | | null | | | street | varchar(50) | yes | | null | | | lastname | varchar(100) | yes | | null | | | firstname | varchar(50) | yes | | null | | | occupation | varchar(255) | yes | | null | | | ownorboard | char(1) | yes | | null | | | source | varchar(50) | yes | | null | | | uid | int(10) unsigned | | pri | null | auto_increment | +------------+------------------+------+-----+---------+----------------+ this mysql database also consists of several image metadata management tables. like the census table shown above they contain brief records that point to more extensive and descriptive corresponding xml files describing scanned images of pullman. our scanned images and their metadata may be viewed at http://www.pullman-museum.org/cgi-bin/pvm/objectsviewrecords.pl?letter=a. description of these tables is, alas, a subject for another article. most of the fields are self-explanatory. the ownorboard field lets me distinguish between renters and sub-letters. the source field describes the source of the data. uid, present as a unique key in all tables, is the unique identifier for the record, which may also be used to point to any corresponding xml record(s). when a record is retrieved, its uid is then used to return the corresponding xml record from a file held on the server as 1900census.uid.xml. as i mentioned, the data set from the 1900 census (and subsequent censuses) is much more complex, and must also be represented by a corresponding xml record. the dtd for the census information is: <!element housedocumentation (housedescription, physicaldescription)> <!element census1900 (houseinformation, personalinformation, ancestry, immigrationstatus, educationandoccupation, housingstatus, censusinformation)> <!element houseinformation empty> <!attlist houseinformation recordid cdata #required address cdata #required aptnumber cdata #implied street cdata #required dwellingnumber cdata #required> <!element personalinformation empty> <!attlist personalinformation familiesnumber cdata #implied lastname cdata #required firstname cdata #required relation cdata #implied race cdata #implied sex cdata #implied dobmonth cdata #implied dobyear cdata #implied age cdata #implied maritalstatus cdata #implied yearsmarried cdata #implied motherof cdata #implied childrenliving cdata #implied> <!element ancestry empty> <!attlist ancestry placeofbirth cdata #implied placeofbirthfather cdata #implied placeofbirthmother cdata #implied> <!element immigrationstatus empty> <!attlist immigrationstatus immigrated cdata #implied yearsinus cdata #implied naturalized cdata #implied> <!element educationandoccupation empty> <!attlist educationandoccupation occupation cdata #required monthsunemployed cdata #implied monthsatschool cdata #implied read cdata #implied write cdata #implied english cdata #implied> <!element housingstatus empty> <!attlist housingstatus ownorrent cdata #implied ownormortgage cdata #implied> <!element censusinformation empty> <!attlist censusinformation pagenumber cdata #required linenumber cdata #required reelnumber cdata #implied notes cdata #implied> the xml record itself: <?xml version="1.0" ?> <census1900> <houseinformation recordid="1405" address="100" aptnumber = "" street="stephenson" dwellingnumber="69" /> <personalinformation familiesnumber = "124" lastname="jeffery" firstname="william" relation="head" race="white" sex="male" dobmonth="jan" dobyear="1841" age="59" maritalstatus="married" yearsmarried="36" motherof="" childrenliving="" /> <ancestry placeofbirth="england" placeofbirthfather="england" placeofbirthmother="england" /> <immigrationstatus immigrated="1871" yearsinus="29" naturalized="naturalized" /> <educationandoccupation occupation="police officer" monthsunemployed="0" monthsatschool="" read="yes" write="yes" english="yes" /> <housingstatus ownorrent="r" ownormortgage="unknown" /> <censusinformation pagenumber="111b7" linenumber="72" reelnumber="290" notes="" /> </census1900> this approach-using tables to hold brief records of disparate data sources that share a common thread and “offloading” the unique part– has the added advantage of being able to collect any data from any source, as long as it can be tied back to a pullman address and combine it with other disparate but related data. as long as a record contains the bare minimum of a numerical street number, street name, and data source type, the basic data can reside in the census table. the uid field can then point to an external xml record containing the additional information. for instance, we will eventually need to index newspaper articles that describe events that occurred in specific houses. by creating a census table record that contains the street address in question and by adding the value of article to the source field, i can successfully index the article (at least as it relates to one or more specific addresses) and can retrieve a more detailed external xml file when i retrieve any record which matches the specified address. the data collected also varies among the different censuses and can also be dealt with in this manner. each federal census is slightly different; while all of them record respondents’ names, addresses, etc., each decennial census asks questions unique to it. representing this data in a database can be a challenge. one could, for instance, create a separate table for each census year (“1900″,”1910”, etc.) and record the data from each decennial census into the appropriate table. searching across the various tables would involve a fairly complex sql join operation, resulting in a lot of unnecessary wear and tear on the sql server. i have chosen, instead, to place the common information that all the censuses share in one table (creatively called census), offloading the unique information in the individual censuses into external xml files. as i have described, this approach can accommodate a wide variety of disparate data. in addition, it allows the collection to be exposed to oai harvesters. while i have not yet implemented the software necessary to expose the image and demographic xml records, it is being developed. such a step will allow other institutions to make use of our data and-frankly-help us in securing grants, since we can quickly point to this facility as an indication of commitment to resource sharing. because the town was annexed and renumbered in 1907, i have had to create an “authority” table that translates between the old and new addresses, which leads us to the next demographic data table. the table is laid out as: mysql> describe addressmaster; +-----------------+-----------------+-------+-----+---------+----------------+ | field | type | null | key | default | extra | +-----------------+------------------+------+-----+---------+----------------+ | oldstreetnumber | varchar(50) | yes | | null | | | newstreetnumber | varchar(50) | yes | | null | | | oldaddress | varchar(50) | yes | | null | | | newaddress | varchar(50) | yes | | null | | | uid | int(10) unsigned | | pri | null | auto_increment | +-----------------+------------------+------+-----+---------+----------------+ here is an example record, which indicates that the modern address of 100 stephenson is 11114 s. champlain ave.: mysql> select * from addressmaster where oldstreetnumber="100" and oldaddress="stephenson"; +-----------------+-----------------+------------+------------+------|--------+------+ |oldstreetnumber | newstreetnumber | oldaddress | newaddress | uid | +-----------------+-----------------+------------+------------+------|--------+------+ | 100 | 11114 | stephenson | champlain | 3046 | +-----------------+-----------------+------------+------------|------|--------+------+ this particular record tells me that the modern address of 11114 s. champlain ave. was known before 1907 as 100 stephenson ave. the perl programs that make up the phhp use this table to perform, essentially, an authority check on post-1907 addresses. i have decided to declare the pre-1907 address the “official” one simply because i had to pick one of the two to ensure data precision in my select statements. when a perl program needs to find information in the census table, it first sends a select command to the addressmaster table requesting the old street number and street name of the address to ensure that all of the records preand post1907 are retrieved. the last demographic information table in the database contains a housing survey, created in 2000 by art institute of chicago students working in the historic preservation department. the students, as part of a massive year long project, collected data about each private structure in the neighborhood. we were then presented a copy of the resulting excel spreadsheet data and accompanying photographs, which i promptly massaged into a table form. like the census information, most of the data is resident in external xml files: <?xml version="1.0" ?> <!doctype housedocumentation system "/xml/dtd/housedocumentation.dtd"> <housedocumentation> <houseinformation recordid="94" imagefilename="11114champlain.jpg" address="11114" street="champlain" ownername="balderama / stigler" nonpullmanaddress="both owneroccupied as of 02/04" /> <housedescription housetype="workers cottage" housetypeother="" buildingtype="rowhouse" buildingtypeother="" dateofconstruction="1880" dateofsurvey="2002-04-28" /> <physicaldescription rooftype="hipped" rooftypeother="" roofmaterial="asphalt" roofmaterialother="" porchtype="wooden" porchtypeother="" railings="wooden" railingsother="" mortar="brown" mortarother="" trimmaterial="wood" trimmaterialother="" chimney="corbelled" chimneyother="" contdetails="dormers" contdetailsother="" windows="wood" windowsother="" windowlights="" storms="aluminum" stormsother="" windowsills="limestone" gutters="other" guttersother="" maindoor="modern wood" maindoorother="" screendoor="metal" screendoorother="" othersalientfeatures="replacement windows and doors, replacement porch, painted brick lg. center dormer flanked by two smaller front gable dormers w/ decorative rafter tails brick corbeling below cove and between 1st. &amp; 2nd. stories." brick="original" brickother="" basementwindows="" basementwindowsother="" /> </housedocumentation> for various internal rights and permissions issues, i chose to place the data in its own separate table. the table is fairly simple, like the census table: mysql> describe housedocumentation; +---------------+------------------+------+-----+---------+----------------+ | field | type | null | key | default | extra | +---------------+------------------+------+-----+---------+----------------+ | filename | varchar(100) | yes | | null | | | address | int(10) unsigned | yes | | null | | | street | varchar(50) | yes | | null | | | mapreference | varchar(100) | yes | | null | | | vistarecordid | int(10) unsigned | yes | | null | | | uid | int(10) unsigned | | pri | null | auto_increment | +---------------+------------------+------+-----+---------+----------------+ the filename field points to an external jpeg house photograph file. address and street, of course, refer to the (post-1907) address of the property, which, as described above, is pointed to by the addressmaster table. uid, like its brethren in the census table, is used to point to an external xml file as housedocumentation.uid.xml the mapreference field is not currently used, but could be developed as a link to the online maps (see below). vistarecordid is also not being used, but is meant to point to a historical streetscape photograph if one exists. should i develop this feature, i would create another table whose records pointed to image files of streetscapes and landscape vistas. i could then view a streetscape image and click on a link in that image to view in more detail the individual houses that make up that streetscape. i have not developed this feature at all except for putting in place the linkages that would make it possible; such a feature would have to be a future development. part ii: joining the real and the representational how do all of these tables and xml records work together? my first course of action was to create perl programs that would connect to the database through the excellent dbi::dbd library. i created a perl script that presented the names, addresses, and professions of the people represented in the main table in alphabetical name order. clicking on the more… link above for any of the 1900 census entries actually brings up that specific census record. the perl code that actually does the work behind the scenes is in the census.pl file (comments designated by octothorpes [#]). a snippet is shown here: # pulls in the $str variable to fill in the limiting letter. this select statement calls the records themselves from the census table. $query = "select * from census $str order by lastname, firstname"; my $sth = $dbh->prepare($query); $sth->execute(); while (@address = $sth->fetchrow()) { $address = $address[0]; $aptnumber = $address[1]; $dispstreet = $street = $address[2]; $lastname = $address[3]; $firstname = $address[4]; $occupation = $address[5]; $ownorboard = $address[6]; $recordid = $address[7]; $source = $address[8]; $uid = $address[8]; } if ($firstname ne "") { $nameofperson = $lastname . ", " . $firstname } else { $nameofperson = $lastname } $count++; $dispstreet =~ s/th//g; if ($ownorboard eq "r") { $ownorboard = "primary renter." } elsif ($ownorboard eq "b") { $ownorboard = "boarder." } elsif ($ownorboard eq "u") { $ownorboard = "" } if ($source eq "1900 census") { $recordid = "<a href=\"/cgi-bin/pvm/househistorydisplay1900census.pl?filename=1900census." . $recordid . ".xml\">more...</a>" } else { $recordid = "" } 1910 sanborn maps, useful in allowing a more visual, map-based approach to discovering the demographic data in the phhp, connect to the demographic tables as well. these maps were created as part of this project. they have been hand colored using appropriate pullman colors, and, by using html image mapping (ismap), mapped to individual address records to create clickable online maps. there is also a fairly extensive set of pullman maps online at http://www.pullman-museum.org/phhp/maps/. to me, maps help users relate demographic data to the actual site being described, so i have scanned in and enhanced a number of maps. clicking on one of the icons takes one to the actual linked map, such as the example below: notice that this 1910 map conveniently has both addresses on it for each property, reflecting the 1907 address changeover. let us examine one property in detail, 100 stephenson/11114 champlain avenue. here is an 1882 photograph of this block of stephenson, found as one of many images at the pullman state historic site’s site: 100 stephenson is tinted blue and has a blue dot above it. [i have, of course, modified a copy of the image especially for this article, which leads me to an intriguing side discussion. a possible future development for the phhp would be to make images of the town interactive, connecting them to the demographic tables in the much the same way as the sanborn maps are connected. as i say, this remains a future development.] the house history record for this property is retrieved using the perl script househistorygetaddress.pl. seen here is the snippet of code that actually retrieves the records (comments again designated by octothorpes): if ($range eq "old") { $addressfield = "oldaddresssort"; $streetfield = "oldaddress"; } else { $addressfield = "newaddresssort"; $streetfield = "newaddress"; } # allows the user to switch back and forth between old address (pre-1907) and new address (post-address) $queryaddress = "select * from `addressmaster` where $addressfield = '$address' and $streetfield = '$street'"; my $sth = $dbh->prepare($queryaddress); $sth->execute(); ($oldstreetnumber,$newstreetnumber,$oldaddress,$newaddress,$oldaddresssort,$newaddresssort,$uida) = $sth->fetchrow(); # the above code simply finds the old and new street numbers and names for the indicated property if ($oldstreetnumber eq "") { $addresstext = "no records found for this property."; $oldaddress = $street; $oldstreetnumber = $address; } else { $addresstext = "before 1907, this property was at $oldstreetnumber $oldaddress. it is now $newstreetnumber $newaddress." } $imagefilename = $newaddresssort . $newaddress; $imagefilename =~ tr/ //d; $imagefilename = $imagefilename . ".jpg"; $imagefilename =~ tr/a-z/a-z/; # if we have a photo of the property from the 2000 aic project, this will determine its name for subsequent display $query = "select recordid from housedocumentation where address='$newstreetnumber' and street='$newaddress'"; my $sth = $dbh->prepare($query); $sth->execute(); $xmlfile = $sth->fetchrow(); $sth->finish(); $housingsurveystr = ""; if ($xmlfile) { $housingsurveystr = "<a class=\"centeredobject\" href=\"/cgi-bin/pvm/housingsurveymenuviews2.pl?address=$newstreetnumber&street=$newaddress\">2000 survey</a>" } # if we have a 2000 aic survey for the property, then build a link to a cgi program that will display it # i am skipping some code here, which simply builds up the headers to create a dynamic web page # now iterate through the 4 data sources for demographic data (1883 and 1889 city directories, 1900 census, and 1916 phone book <h3 class="displaybanner"> 1883 city directory </h3> <p /> <table border="1"> eof # i will repeat this code as many times as necessary to retrieve all the names of the people who lived or worked at this address. $querya = "select * from `census` where street = '$oldaddress' and address = '$oldstreetnumber' and source = '1883 city directory' order by lastname, firstname"; my $sth = $dbh->prepare($querya); $sth->execute(); while (@address = $sth->fetchrow()) { &getvalues } print <<eof; </table> <p /> <h3 class="displaybanner"> 1889 city directory </h3> <p /> <table border="1"> eof $querya = "select * from `census` where street = '$oldaddress' and address = '$oldstreetnumber' and source = '1889 city directory' order by lastname, firstname"; my $sth = $dbh->prepare($querya); $sth->execute(); while (@address = $sth->fetchrow()) { &getvalues } print <<eof; </table> <p /> <h3 class="displaybanner"> 1900 census </h3> <p /> <table border="1"> eof $querya = "select * from `census` where street = '$oldaddress' and address = '$oldstreetnumber' and source = '1900 census' order by lastname, firstname"; my $sth = $dbh->prepare($querya); $sth->execute(); while (@address = $sth->fetchrow()) { &getvalues } print <<eof; </table> <p /> <h3 class="displaybanner"> 1916 phone book </h3> <p /> <table border="1"> eof $querya = "select * from `census` where street = '$oldaddress' and address = '$oldstreetnumber' and source = '1916 phone book' order by lastname, firstname"; my $sth = $dbh->prepare($querya); $sth->execute(); while (@address = $sth->fetchrow()) { &getvalues } print <<eof; <p /> </table> eof getfooter(); print "</body>\n</html>\n"; $dbh->disconnect; # the routine getvalues retrieves individual records from the sql select statement above and formats them into html sub getvalues { $address = $address[0]; $aptnumber = $address[1]; $street = $address[2]; $lastname = $address[3]; $firstname = $address[4]; $occupation = $address[5]; $ownorboard = $address[6]; $recordid = $address[7]; $source = $address[8]; $uid = $address[8]; if (($address ne "") and ($street ne $oldaddress)) { $dispstreet = $street; $street =~ s/th//g; $street =~ s/ /%20/g; $address = "<br />lived at: <a href=\"/cgi-bin/pvm/househistorygetaddress.pl?range=o ld&street=" . $street . "&address=" . $address . "\">$address $dispstreet</a>"; } else { $address = "" } if ($firstname ne "") { $nameofperson = $lastname . ", " . $firstname } else { $nameofperson = $lastname } $aptnumber =~ s/room//g; if ($aptnumber ne "") { $nameofperson = $nameofperson . " room " . $aptnumber } if ($ownorboard eq "r") { $ownorboard = "primary renter." } elsif ($ownorboard eq "b") { $ownorboard = "boarder." } elsif ($ownorboard eq "u") { $ownorboard = "" } if ($source eq "1900 census") { $recordid = "<a href=\"/cgi-bin/pvm/househistorydisplay1900c ensus.pl?filename=1900census." . $recordid . ".xml\">more...</a>" } # builds a moreã¢â‚¬â¦ link that points to an external xml file, 1900census.$recordid.xml. else { $recordid = "" } print <<wombat; <tr> <td width="20%"><span class="nametext">$nameofperson</span></td><td width="4 0%"><span class="italictext">$occupation</span></td><td class="40%"><span class="rentorboardtext">$r ecordid $address</span></td> </tr> wombat } officer william jeffrey’s 1900 census record from the example above is retrieved using the perl script househistorydisplay1900census.pl (snippet displayed further down): # i am using the perl module xml::simple to reformat the xml file that contains the 1900 census data. the programs knows which file to retrieve because it was passed from the more… link above, and was determined by the getdefaults subroutine above. use xml::simple; my $xmlref; my $xmlref = xmlin("/home3/pullman/sites/pullman-museum.org/html/xml/$xmlfile"); # is the actual file name and path ##### houseinformation ##### $aptnumber = $xmlref->{houseinformation}->{aptnumber}; $address = $xmlref->{houseinformation}->{address}; $street = $xmlref->{houseinformation}->{street}; $dwellingnumber = $xmlref->{houseinformation}->{dwellingnumber}; ##### personalinformation ##### $familiesnumber = $xmlref->{personalinformation}->{familiesnumber}; $lastname = $xmlref->{personalinformation}->{lastname}; $firstname = $xmlref->{personalinformation}->{firstname}; $relation = $xmlref->{personalinformation}->{relation}; $race = $xmlref->{personalinformation}->{race}; $sex = $xmlref->{personalinformation}->{sex}; $dobmonth = $xmlref->{personalinformation}->{dobmonth}; $dobyear = $xmlref->{personalinformation}->{dobyear}; $dob = $dobmonth . " " . $dobyear; $age = $xmlref->{personalinformation}->{age}; $maritalstatus = $xmlref->{personalinformation}->{maritalstatus}; $yearsmarried = $xmlref->{personalinformation}->{yearsmarried}; $motherof = $xmlref->{personalinformation}->{motherof}; $childrenliving = $xmlref->{personalinformation}->{childrenliving}; challenges ahead in the future, we hope to make better use of sources that give the census entries life and breadth. the records representing people need to be expanded to have any meaning and context. consider the case of malcolm mcqueen and his widow: i have found two references to mcqueen in the chicago tribune. malcolm mcqueen, foreman of the repair shops at pullman, went into the cellar of his residence at no. 124 watt avenue about 4 o’clock yesterday morning and hanged himself. in his pockets were found a note saying that his head pained him and a message to his wife bidding her goodbye. chicago tribune, sept. 23, 1888 and robert mcqueen is a guest at the home of his sister in law, mrs. malcolm mcqueen on watt avenue… the funeral services of the late malcolm mcqueen were held at the presbyterian church monday afternoon under the charge of the masons. before 2:30, the hour set for the funeral, the church was filled to its utmost capacity and it was with difficulty that the ushers kept the crowd from filling the aisles. chicago tribune, sept. 30, 1888 mr. mcqueen’s story is poignant and deserves to be highlighted. these accounts should be connected to mcqueen’s demographic information, tying his name to the text of the article. the primary source for newspaper articles like these is the pullman company scrapbooks. the scrapbooks, held at the newberry library in chicago, were kept by the pullman company from 1870 to 1925. they represent a priceless treasure of clippings concerning the pullman town and company. there are over 100,000 individual clippings in the scrapbooks, of which a few example pages can seen at http://www.pullman-museum.org/scans/. we are currently working on a grant that will allow us to digitize these articles as ocr’d pdf files which can then be keyword indexed. we use swish-e as our search engine and harvester, and we have been very happy with it. i can easily connect the names contained in the census table and these newspaper clippings by writing a brute force program that searches for each name as selected in a swish-e-based clippings index. this is, however, a crude and imprecise solution. a better solution would be to employ a natural language processing program that will create a table more precisely linking the names in the census table and the records in the keyword index. the ideal solution, of course, would be to create item level metadata records for each article, a very expensive and time-consuming proposition. how we are going to properly handle the clippings records is a matter undergoing discussion. another major hurdle is simply completing the census data inputting. i have, to this point, relied on volunteer efforts to completely input the 1883 and 1889 city directories and the 1916 phone directory. volunteer efforts, however, have only been able to enter 5% of the 1900 census. clearly, volunteers will not be able to get the rest of the census data entered, so we are currently working on yet another grant that will allow us to engage a commercial firm to enter the rest of the 31,750 census entries. in conclusion the pullman state historic site’s virtual museum records an average of 212,854 hits a month. the use of the pullman virtual museum increases on a monthly basis, and we look forward to expanding it to ever greater functionality by adding in the newspaper clippings and finally digitizing the remainder of the census records. it has been, so far, a rare opportunity to work on such an enjoyable and unique project. please come visit us at http://www.pullman-museum.org/ and explore the world of pullman. author information: andrew bullen is the information technology coordinator for the illinois state library, and a proud resident of the pullman community. he can be reached at abullen at ameritech dot net. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – adopting a distributed model for data services mission editorial committee process and structure code4lib issue 35, 2017-01-30 adopting a distributed model for data services this article describes how the saint edward’s university library implemented a distributed model for the institutional repository. based on cloud based platforms and apis, the library has created an institutional repository that is scaleable and modular, considerably lowering its implementation and maintenance costs, while lowering its technical complexity. by casey gibbs, marcos hernandez, pongracz sennyey http://ir.stedwards.edu at the saint edward’s university[1] library, we have implemented a simple, modular, inexpensive, and effective model for providing digital services in a resource constrained environment. until about two years ago the library did not offer institutional repository or digital data management services, and given the lack of demand from the campus, we were skeptical about its need. in 2014, however, the new dean of natural sciences approached the library to request support for making his atmospheric research data more easily managed and shared on the internet. to accommodate this request, the library assembled a set of digital data management tools connected via apis that enable us to provide an expanding portfolio of services without locking ourselves into complex vertically integrated platforms. this model has arbitrary storage, presentation layers, data modelling and network connections. in this paper we will describe our implementation process and discuss our distributed model for data management services. the evolution of the concept: figshare implementation at first we selected figshare[2] as our foundational stack, and built the data management services around it. we selected figshare for a set of appealing characteristics: it is a cloud product, thus lessening its demands on local resources; it has a powerful set of apis, which appealed to our need to address multiple very different types of projects; it offers full functionality directly to end users without library staff intervention, and it purported to be simple to configure, simple to use, and flexible in terms of metadata, file formats, and storage quotas. once selected, the figshare implementation team was great to work with and it did not take long to get the basic framework set up to our specifications. but figshare offers very restricted options for branding the front-end and for user interface design. we needed a much more flexible ui with more embedded functionality and interactivity, but figshare did not offer those features out of the box. instead, it allowed us to design our own dedicated website connected to figshare via apis. having done so we asked ourselves whether it would be possible to fold in other api connected services as well.   the dean’s atmospheric data project was a good test case for our new model. his project consists of collecting data from an ongoing series of weather balloon launches. for each launch, a raw data file is produced, and from that data several static data visualization files are generated. his data archive was originally hosted in a complex, static website that he manually updated multiple times per year[3]. fig 1: original site. this page lists the launches and links to its datafiles and graphs. the service request was for us to simply replicate the project’s original website. in the planning phase, however, we uncovered two manual processes we could automate by migrating his data into figshare: data from each weather balloon launch had to be uploaded manually. with figshare, we use api calls to automate uploads from the dean’s computer to his figshare account, for each launch, the website had to be edited manually. figshare’s api makes it possible to dynamically render our front end by querying the dean’s figshare account, thus eliminating manual edits. see figure 2 for the javascript queries we use to render the links and the menu structure in our interface: fig 2. javascript/jquery call to figshare api. a for loop is used to parse the returned data. next we were able to further extend the dean’s original project by supplementing its static, file-based .pdf and .jpg visualizations with dynamic svg plots generated and displayed upon page load. in order to do this for a given launch, first we use javascript to parse the raw data file.   fig 3: raw data is parsed into javascript objects to be rendered in d3.js. the next step is to use the d3.js library (a javascript library for producing dynamic, interactive data visualizations in web browsers – see https://d3js.org) to generate visualizations.   figure 4: left: pdf of wind direction data created by custom software. right: the same data dynamically generated by d3 in the browser. similarly, we were able to use the original .kml file for each launch with a call to google maps’ api and dynamically generate an interactive map of each launch, rather than using a static image of a map. fig 5. new site: displays google map generated from the .kml though these dynamic renders marginally increase page load time, we accrued several substantial benefits over the previous model: identical visualizations are now rendered for all launches. edits to the visualizations can now be made without re-rendering previous launches. our new visualizations now provide interactivity, such as the ability to mouse over individual data points to get their numeric values. we are now able to display visualizations that were not possible before, e.g., the same data points visualized across several launches. lessons learned our api based system solved a number of problems and made for a simpler site for our users. but figshare’s architecture is optimized for end users with stable content. as it happens the dean’s project is actually not a perfect fit for figshare’s contraints. learning to use figshare’s api made us aware that we could match future projects with other api enabled platforms as needed. we have subsequently built projects on s3[4], wordpress[5], flickr[6] and omeka. we are able to take advantage of the strengths of these platforms while still consolidating access to their content in our custom front end[7]. the result is a series of projects built upon a platform agnostic architecture – a distributed model – that retains a clean branded interface. our modular system allows the library to incorporate new technologies to meet our clients’ disparate needs. the distributed model has proven to be: lightweight and modular: our system consists of an interchangeable set of cloud-based saas platforms. each performs one or more specific functions (such as storage, apis, data visualization, metrics, etc.), resulting in an adaptable and modular architecture. these modules are made available through a simple, custom-built, html front-end. we are not restricted to any one product, we have no vendor lock-in, and we can easily replace one module with another by migrating our data and updating our api calls. for example we could easily replace amazon s3 with google’s cloud storage. low maintenance: our only upkeep involves monitoring current systems for api changes. simple for the user: although any number of systems may be in use at a given time, the end user’s only access point is our own website. simple to develop: as systems evolve and innovative new products become available, bringing them into our architecture will be a matter of learning their apis. for example, project-specific metadata functionality could be added anywhere in our stack (on our server, or on a cloud-based database engine) without the need to worry about extending current systems by, e.g., developing plugins. custom code (such as visualizations, playback controls, and analytics) can be written for the server or the front-end. end-user access: our use of figshare makes it possible for end users to publish work and have it automatically appear on the website, with the library administering issues such as storage quotas and access privileges. this means that users can deposit materials with no intervention from the staff, and subsequently make versioned edits. while figshare provides end user access very well, our modular architecture allows alternative solutions if deemed preferable. easier to navigate: having a front end that is automatically populated with our content means less editing and fewer errors. maintaining our own entirely custom front-end also means we can use common interface conventions across our web properties. we have found the distributed model to suit our needs well. these include class projects that involve faculty-student collaborations, special collections projects initiated from the library itself, and more traditional ir functions such as the storage of documents, e.g., yearbooks, campus publications and assorted manuscripts. challenges ahead: the rollout and implementation of this service has raised a number of challenges which we have  confronted along the way, and a few we see on the horizon. among them the most important are: although we recognize that metadata is a critical aspect of data management services, we have not yet addressed it systematically. instead we have created ad hoc solutions to each of our projects. for example in the atmospheric data project we are relying on file-level metadata as well as simple collection-level metadata functionality provided by figshare. conversely an on-going photo collection project (http://ir.stedwards.edu/natural-sciences/camera) relies on file-level metadata alone.  although we aim to implement, or develop, a metadata application, the challenge is that it should be compatible with our distributed, modular, architecture. we have not yet found an ideal solution. similarly to the challenge presented by metadata, we have not yet systematically addressed the issues related to trustworthy long-term preservation of digital content. however, we consider it an administrative challenge, that includes protocols for managing the life cycle of digital materials, dedicating staffing and materials resources etc., more than a technical one, for which our infrastructure is well equipped to accommodate evolving practices in this area. identity management has been a recurring bureaucratic challenge, inasmuch as it has required  us to synchronize efforts with other units on campus (it) that are operating on a different set of priorities and timeframes. implementing identity management tools, however, is an often requested feature of collaborative projects. for example, the campus it department already provides instances of managed web applications, such as wordpress, that are preconfigured with single sign on for campus accounts. assisting our clients with instances on those platforms is simple, but developing instances of our own custom applications requires us to work with it to use their single sign-on server. the problem is compounded by the fact that we must address this with every such application we develop. since we do not run our own sso server, this remains an unsolved bureaucratic challenge. conclusions: in retrospect, it is striking how much our previous experience with integrated systems limited our thinking about the services we can provide. by adopting a platform agnostic architecture we are able to customize our toolset for each project. as a result, no one platform is a limiting factor, and we can prioritize the needs of our patrons. in effect, data services is not built on one platform, such as figshare, but rather a series of platforms, which are transparent to the patron, but embedded in a common front end. in our experience digital scholarship is evolving rapidly. once data is digital and online, scholars develop new modes of inquiry, new questions and new methodologies. to keep pace with these innovations, service providers must be able to iterate and experiment nimbly. we have come to the conclusion that no one monolithic platform can satisfy the evolving requirements of the landscape of digital scholarship. the resulting complexity and cost of such a system would be prohibitive. while there is much more to be done, and as the university community discovers its potential, demand is starting to increase. this new model solves a number of challenges that were previously beyond our reach, and has opened the way for the creation of a new library service that supports both research and pedagogical innovations at st. edward’s university. references [1] saint edward’s is a liberal arts institution with about 5,500 students, located in austin, texas. [2] figshare’s homepage at https://figshare.com/, the saint edward’s instance can be found at https://stedwards.figshare.com/ [3] see http://physics.valpo.edu/ozone/ [4] http://ir.stedwards.edu/natural-sciences/camera [5] http://sites.stedwards.edu/wereyouthere/ [6] https://www.flickr.com/photos/135420022@n04/sets/72157660077085270 [7] http://ir.stedwards.edu/ authors: casey gibbs: library digital projects manager at saint edwards university, austin texas. (rgibbs@stedwards.edu) marcos hernandez: library front end designer at saint edwards university, austin texas.(mherna14@stedwards.edu) pongracz sennyey: library director at saint edwards university, austin texas. (pongracz@stedwards.edu) subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – web-based software integration for dissemination of archival images: the frontiers of science website mission editorial committee process and structure code4lib issue 14, 2011-07-25 web-based software integration for dissemination of archival images: the frontiers of science website the frontiers of science illustrated comic strip of ‘science fact’ ran from 1961 to 1982, syndicated worldwide through over 600 newspapers. the rare books and special collections library at the university of sydney, in association with sydney escholarship, digitized all 939 strips. we aimed to create a website that could disseminate these comic strips to scholars, enthusiasts and the general public. we wanted to enable users to search and browse through the images simply and effectively, with an intuitive and novel viewing platform. time and resource constraints dictated the use of (mostly open source) code modules wherever possible and the integration and customisation of a range of web-based applications, code snippets and technologies (dspace, extensible text framework (xtf), omniformat, jquery tools, thickbox and zoomify), stylistically pulled together using css. this approach allowed for a rapid development cycle (6 weeks) to deliver the site on time as well as provide us with a framework for similar projects. by gary browne introduction the rare books and special collections library at the university of sydney was bequeathed the rights to the original strips, or “pulls”, of the frontiers of science comic strips which were published in over 600 newspapers worldwide from 1961 to 1982. the series was co-written and produced by professor stuart butler from the school of physics at the university of sydney and journalist and film-maker bob raymond. the early art work in the series was by andrea bresciani, continued later by david emerson, with the aim of disseminating information about current topics in science in a novel and entertaining way. having been given permission to digitize the original strips, sydney escholarship at the university of sydney sought to make the frontiers of science strips available once again, this time online, for wider access by scholars and enthusiasts. our goal was to offer simple, robust, flexible discovery tools including search, faceted browse and a tag cloud, combined with a novel display mechanism. there was an upcoming exhibition of frontiers paraphernalia at the australian broadcasting commission (abc) in ultimo, sydney, which was to coincide with science week. we wanted the frontiers of science website to be launched to coincide with this, meaning the deadline at the time was 6 weeks away. many tasks needed prioritization and consideration within the short time frame. these included but were not limited to: importation of the digitized pulls to the institutional repository, metadata tagging of the items, processing of images (for thumbnails and for use with the image viewer), processing of metadata, indexing of image metadata for browsing and searching functionality, creating the browse and search front-end, tag cloud generation, deciding upon and implementing an image viewer, presentation and layout design (within university style constraints), and developing the site, including associated historical and descriptive content and complying with (as far as practicable) w3c standards. to expedite the process, it was decided to rely heavily on technologies that we were already familiar with or that could be quickly integrated into existing (open source) architecture. preparing the images and metadata scanning of comic strips the original frontiers of science material was bequeathed to the rare books and special collections library of the university of sydney by angela raymond and miriam butler. here, they were scanned in black and white on a bookeye scanner (be3-scl-r1) at 600dpi by nicholas heath in january of 2009. each resulting archival tiff file was about 400kb in size. content archiving the sydney escholarship repository, the university’s institutional research repository, was used to archive the original scanned tiff images. the repository employs a customized instance of the dspace open source digital library software, which provides by default a facility for tagging items with qualified dublin core metadata. most collections in the repository are tagged using the dublin core schema, so we decided to use this metadata schema to describe the frontiers images. tagging of the comic strips within the sydney escholarship repository was completed by professor peter harrowell from the school of chemistry, university of sydney. due to time constraints, no controlled vocabulary was applied, so the metadata was less than ideal for data interchange purposes, but sufficient for our requirements of browsing and searching. initially, about 200 pulls (each with one week’s worth of comic strips) were tagged with keywords, with the aim of adding more tagged strips over time, following the launch of the site. metadata processing the search and browse system was developed using the extensible text framework (xtf) [1], which recognizes dublin core metadata in a particular format. to accommodate this, we designed a java class for dspace which: created the dublin core metadata xml files with the necessary metadata fields formatted the xml files for use with the default xtf metadata “prefilter” named the xml files based on dc.identifier.other we decided that it would be simpler and more efficient to do all three tasks in one java class, rather than creating and naming the files this way and then customizing the xtf prefilter to suit the xml file format. the main writemetadata method of this class (see code listing 1 below) sets up the xml file for xtf by defining the root element as <dc>, the xtf default. within the <dc> tags, we tested only for the following metadata elements: title, date, identifier and subject. code listing 1. the writemetadata method to setup xtf-ready dublin core files // output the item's dublin core into the item directory private static void writemetadata(context c, string schema, item i, file destdir) throws exception { // id to rename file with string id=""; string filename; if (schema.equals(metadataschema.dc_schema)) { filename = "dc.xml"; } else { filename = "metadata_" + schema + ".xml"; } file outfile = new file(destdir, filename); system.out.println("attempting to create file " + outfile); if (outfile.createnewfile()) { bufferedoutputstream out = new bufferedoutputstream( new fileoutputstream(outfile)); dcvalue[] dcorevalues = i.getmetadata(schema, item.any, item.any, item.any); // xml preamble byte[] utf8 = "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n" .getbytes("utf-8"); out.write(utf8, 0, utf8.length); // "dc" instead of "dublin_core" string dctag = "<dc>\n"; utf8 = dctag.getbytes("utf-8"); out.write(utf8, 0, utf8.length); for (int j = 0; j < dcorevalues.length; j++) { dcvalue dcv = dcorevalues[j]; string qualifier = dcv.qualifier; // get element string element = dcv.element; // test only for elements we want if(element.equals("title")){ utf8 = (" <title>" + utils.addentities(dcv.value) + "</title>\n").getbytes("utf-8"); out.write(utf8, 0, utf8.length); } else if(element.equals("date") && qualifier.equals("issued")){ utf8 = (" <date>" + utils.addentities(dcv.value) + "</date>\n").getbytes("utf-8"); out.write(utf8, 0, utf8.length); } else if(element.equals("identifier") && qualifier.equals("other")){ utf8 = (" <identifier>" + utils.addentities(dcv.value) + "</identifier>\n").getbytes("utf-8"); out.write(utf8, 0, utf8.length); id = dcv.value; system.out.println("wrote id: " + id); } else if(element.equals("subject")){ utf8 = (" <subject>" + utils.addentities(dcv.value) + "</subject>\n").getbytes("utf-8"); out.write(utf8, 0, utf8.length); } else{ system.out.println("no matching element found"); } } // close "dc" tag utf8 = "</dc>\n".getbytes("utf-8"); out.write(utf8, 0, utf8.length); out.close(); // rename file prepend identifier file idoutfile = new file(destdir, id + ".dc.xml"); outfile.renameto(idoutfile); } else { throw new exception("cannot create dublin_core.xml in " + destdir); } } the output of this program was one directory for each image. each directory contained the xtf-formatted dublin core metadata file (named $dc.identifier.other.dc.xml where $dc.identifier.other is the issue week) and the “pull” image plus other accessory files. this directory structure is the standard dspace submission information package (sip) configuration. the “.dc.xml” extension is recognized by xtf as a dublin core metadata file. when loaded into dspace, the tiff file names were disparate. some were simply numeric (“064.tif”) and others had varying prefixes (“fos” or “fos_”). to rectify this, another java program was written to standardize the naming of the tiff files in line with the metadata files i.e. $dc.identifier.other.tif, where $dc.identifier.other is the issue week. the main method (see code listing 2 below) which retrieves the identifier from the dublin core xml file uses the java api for xml processing (jaxp) [2]. code listing 2. main method to retrieve identifier from dublin core xml files /** * reads the frontiers of science identifier from an image's dublin * core metadata file (exported from dspace using itemexporter) * * @param file the file from which to read * @return fosid the frontiers of science image id */ public string readdcfilefosidentifiervalue(file file){ // the fos identifier value string fosid=""; try { documentbuilderfactory domfactory = documentbuilderfactory.newinstance(); domfactory.setnamespaceaware(true); // never forget this! documentbuilder builder = domfactory.newdocumentbuilder(); document doc = builder.parse(file); xpathfactory factory = xpathfactory.newinstance(); xpath xpath = factory.newxpath(); xpathexpression expr = xpath.compile("//dcvalue[@element='identifier' and @qualifier='other']/text()"); object result = expr.evaluate(doc, xpathconstants.string); fosid = result.tostring(); } catch (saxparseexception err) { system.out.println ("** parsing error" + ", line " + err.getlinenumber () + ", uri " + err.getsystemid ()); system.out.println(" " + err.getmessage ()); }catch (saxexception e) { exception x = e.getexception (); ((x == null) ? e : x).printstacktrace (); }catch (throwable t) { t.printstacktrace (); } return fosid; } the format of a typical dc.xml file is shown below: <?xml version="1.0" encoding="utf-8"?> <dc> <date>1963-03-18</date> <identifier>79</identifier> <subject>physics</subject> <subject>space</subject> <subject>lunar surface</subject> <subject>tom gold</subject> <subject>powder oceans</subject> <subject>tycho crater</subject> <subject>moon landing</subject> <title>the surface hazard</title> </dc> this naming convention and metadata file format standardization would assist in further processing and dynamic rendering of the images, as well as indexing for the search and browse interface. image processing creating scrollable preview images in keeping with the format of the comic strips, we aimed to make the style of the website very visual and high contrast. the faceted browse interface and search results would allow users to scroll through preview images in an image slider. these previews were created by processing the images output from the dspace export using the omniformat (free) document conversion utility. jpegs at 200px x 400px were created for scrolling in the image slider (see figure 1 below) and we were very happy with the image quality. figure 1. scrollable preview images for browsing creating tiles for zoomify design zoomify is commercial software enabling zooming and panning of high-resolution images. the decision to purchase a commercial licence for zoomifydesign4-win rather than code an image zoomer and panner in, for example, jquery, was made due to time and resource constraints. technically, this worked well in terms of development time. however, due to zoomify being flash-based, it meant that the site would not work on ipads or iphones. this was an oversight at the time of development and the change over to jquery for zooming could be a possible future site enhancement. zoomify requires the creation of image “tiles”, which were created by dragging and dropping images onto the zoomify converter.exe program icon in windows. this processing produced one directory corresponding to each tiff image (as shown below), consisting of subdirectories of “tile groups”, collections of small jpegs which are small portions of the original image. 1 |-imageproperties.xml |-tilegroup0 | |-0-0-0.jpg | |-1-0-0.jpg | |-1-0-1.jpg . . . |-tilegroup1 | |-5-0-10.jpg | |-5-0-11.jpg | |-5-0-12.jpg . . . 2 |-imageproperties.xml |-tilegroup0 | |-0-0-0.jpg | |-1-0-0.jpg | |-1-0-1.jpg | |-2-0-0.jpg | |-2-0-1.jpg these tile group directories were then ready to be presented using the zoomify viewer, a shockwave flash file which is bundled with zoomify design. building the site the search and browse platform at the university of sydney library we have used the extensible text framework (xtf) for several projects over the last few years and have found it to be flexible enough to meet our needs for developing customized search and faceted browse functionality for a variety of content types and metadata schemas. having some expertise in creating xtf sites, we decided to use this open source platform for the search and browse functions of the frontiers of science site. given the time and resource constraints, we wanted to keep things simple, so we pared down the full-featured xtf advanced search engine to allow searching only on keyword or title. we used the term “keyword” since the subject metadata values were not controlled. figure 2 shows the simple keyword/title search forms page. clear examples of available boolean operations were provided. figure 2. keyword and title search page showing sample searches xtf comes with a highly configurable text indexing facility which we used to generate a lucene index from the dublin core files. the lucene index is read for searching by the “crossquery” servlet. this is also highly configurable through xsl files and templates. the search engine produces a results page which is powered by xtf but which integrates the use of the jquery tools’ scrollable module. figure 3 shows the salient features of this search results page and the scrollable module. firstly, the search itself is displayed on the right hand sidebar and the number of matches is shown. the results can be sorted by title or date. through xtf’s faceted browsing feature, users are able to narrow their search results further. once again, we decided to keep it simple, only defining “date” and “keywords” as faceted browse fields. figure 3. main features of the search results page. the display of the actual results is visual, showing the jpeg files in a scrollable component which may be navigated either by clicking the arrows beside the component, or the circles above (thus, users can jump to any part of the list of results, which is particularly useful when there are a large number of matches). each jpeg has a title and publication date associated with it. this display uses the jquery tools library, included in the document <head>: <script src="script/jquery/jquery.tools.min.js" type="text/javascript"/> also included are two css files for the styling of the module: <link rel="stylesheet" type="text/css" href="css/default/scroll.css"/> <link rel="stylesheet" type="text/css" href="css/default/scrollable-navig.css"/> xtf uses xsl stylesheets to control and configure the java classes which drive it. the main scrollable module is defined within an xsl template in $xtf_home/style/crossquery/resultformatter/default/resultformatter.xsl, which is called on the condition that a search returns at least one match (“dochit” in code listing 3). code listing 3. the test for search term matches. <xsl:if test="dochit"> <!-scrolling images with zoomable image overlaid onclick --> <xsl:call-template name="scrollable"> <xsl:with-param name="browseorsearch" select="dochit"/> </xsl:call-template> . . . </xsl:if> the “scrollable” xtf xsl template defines container classes and ids such as “scrollable”, “thumbs”, “thumb” and “meta”, the styles of which are set in the two css files, “scroll.css” and “scrollable-navig.css”. the collection of pulls included some that had not been previously published and, therefore, had no publication date. a workaround was put in place for this – if the pull did not have a date greater than zero, instead of displaying a formatted date, the text “unpublished” was displayed (see “unpublished material hack” in code listing 4). code listing 4. the “scrollable” xsl template <xsl:template name="scrollable" exclude-result-prefixes="#all"> <xsl:param name="browseorsearch"/> <!-navigator --> <div class="navi"></div> <!-prev link --> <a class="prevpage"></a> <!-root element --> <div class="scrollable"> <!-container for items --> <div id="thumbs"> <!-loop through browse/search results, add each to scroller --> <xsl:for-each select="$browseorsearch"> <xsl:variable name="id" select="meta/identifier[1]"/> <xsl:variable name="title" select="meta/title[1]"/> <!-unpublished material hack --> <xsl:variable name="published" select="if(number(substring(meta/date[1],1,4)) > 0) then format-date(meta/date[1], '[d1o] [mnn,*-3],[y]') else 'unpublished'"/> <!-setup the scrollable "thumbnails" --> <div> <div id="thumb"> <a class="thickbox" href="#tb_inline?height=700& width=1000&inlineid={$id}"> <img src="img/jpgs/{$id}.jpg" alt="{$title}" title="{$title}" /> </a> </div> <!-end thumb div --> <div id="meta"> <a class="thickbox" href="#tb_inline?height=700px& width=1000&inlineid={$id}"> <h4><xsl:value-of select="$title"/></h4> </a> <p><xsl:value-of select="$published"/></p> </div> <xsl:call-template name="overlay"> <xsl:with-param name="id" select="$id"/> </xsl:call-template> </div> </xsl:for-each> </div><!- end thumbs div --> </div><!- end scrollable div --> <!-next link --> <a class="nextpage"></a> <script type="text/javascript"> // only execute scripts when dom is ready $(function() { // initialize scrollable $("div.scrollable").scrollable({ size: 3, items: '#thumbs', hoverclass: 'hover' }); }); </script> </xsl:template> within each “thumb” and “meta” <div> a hyperlink is set up which has a class of “thickbox” and passes some parameters in the url, most notably the $id of the pull to show, as follows: <a class="thickbox" href="#tb_inline?height=700&width=1000&inlineid={$id}"> <img src="img/jpgs/{$id}.jpg" alt="{$title}" title="{$title}" /></a> the $id is, in all cases, standardized and represents: the issue number, or publication week the prefix of the *.dc.xml metadata file the prefix of the *.jpg thumbnail file the prefix of the zoomify image tile directory once a user wishes to view a particular pull in detail, they may click on the jpeg (or its title). an overlay with zoom and pan controls then allows the user to view each week’s comic strip in detail. the viewing platform rather than click through to a standalone page for viewing the pulls in zoomify, we decided to integrate zoomify with xtf through the use of an overlay. we chose thickbox for creating the overlays, primarily for its ease of use (both from a user and developer perspective). the following javascript and css files were required for the thickbox component: <!-thickbox --> <script type="text/javascript" src="script/thickbox/jquery.js"></script> <script type="text/javascript" src="script/thickbox/thickbox.js"></script> <link rel="stylesheet" href="css/default/thickbox.css" type="text/css" media="screen" /> <link rel="stylesheet" href="css/default/overlay.css" type="text/css" media="screen" /> when the pull thumbnail or title is clicked in the scrollable component (from browse or search results), the xtf overlay xsl template (see code listing 5 below) is called to display the zoomable and pannable pull. code listing 5. the “overlay” xsl template <!-=============================================================== --> <!-overlay template --> <!-=============================================================== --> <xsl:template name="overlay"> <xsl:param name="id"/> <div class="overlay" id="{$id}"> <p> <object classid="clsid:d27cdb6e-ae6d-11cf-96b8-444553540000" codebase="http://download.macromedia.com/pub/shockwave/cabs/flash/swflash.cab#version=6,0,40,0" width="1000" height="750" id="themovie"> <!-this works for ie --> <param name="flashvars" value="zoomifytoolbarskinxmlpath={$zoomifytoolbarskinxmlpath}&zoomifyimagepath={$zoompath}{$id}&zoomifynavigatorvisible=1&zoomifyinitialzoom=10&zoomifysplashscreen=0&zoomifytoolbarlogo=0&zoomifytoolbartooltips=1&zoomifynavigatorheight=230&zoomifyinitialx=0&zoomifyinitialy=0&zoomifynavigatorx=870&zoomifynavigatory=460" /> <param name="menu" value="false" /> <param name="src" value="{$zoomifyviewerpath}" /> <!-this works for firefox --> <embed flashvars="zoomifytoolbarskinxmlpath={$zoomifytoolbarskinxmlpath}&zoomifyimagepath={$zoompath}{$id}&zoomifyzoom=180&zoomifynavigatorvisible=1&zoomifyinitialzoom=10&zoomifysplashscreen=0&zoomifytoolbarlogo=0&zoomifynavigatorheight=230&zoomifyinitialx=0&zoomifyinitialy=0&zoomifynavigatorx=870&zoomifynavigatory=500" src="{$zoomifyviewerpath}" menu="false" pluginspage="http://www.macromedia.com/shockwave/download/index.cgi?p1_prod_version=shockwaveflash" width="1000" height="750" name="themovie"></embed> </object> </p> </div> </xsl:template> figure 4 shows the zoomed pull. the zoom and pan controls are shown at the bottom centre (green border), and the navigation window is shown at the bottom right of the overlay (yellow border). figure 4. the zoomify overlay showing zoom and pan controls, and the navigation window. the zoomify window provides a convenient method of navigating through and reading the pulls and gives the user the opportunity to look in detail at the high resolution (tiled) images. clicking on the “close” link, using the “esc” key, or clicking anywhere outside of the overlay window closes the zoomify overlay window. the home page the home page features five selected strips presented in a vertical tabbed format (shown in figure 5). figure 5. vertical accordion tabs on the home page. we used jquery tools accordion tabs (see code listing 6 below), with a “fade” effect, rather than sliding. clicking on a strip title fades in that particular strip’s “thumbnail”. clicking on the thumbnail itself employs the same methods as the search/browse functionality to display a zoomable and pannable pull. code listing 6. jquery tools accordion tabs <link rel="stylesheet" type="text/css" href="css/default/tabs-accordion.css"> <link rel="stylesheet" href="css/default/thickbox.css" type="text/css" media="screen"> <script src="script/jquery/jquery.tools.min.js" type="text/javascript"> </script> <script type="text/javascript" src="script/thickbox/thickbox.js"></script> . . . <script type="text/javascript"> $(function() { $("#accordion").tabs("#accordion div.pane", { tabs: 'h2', effect: 'fade' }); }); </script> the tag cloud as an alternative method for browsing the archive, it was suggested that a tag cloud be developed. i like steven york’s [3] approach to this, so i used a slightly modified version of his “accessible tag cloud in php and css (with mysql)”. our “tags” were embedded in the $dc.identifier.other.dc.xml files, so a java class (tagcloudfromdc.java) was written to extract the list of tag values from them, once again using the jaxp. the program outputs a flat text file with one tag per line, by traversing a directory of the dc.xml files. steven york’s php code was then modified to generate the html for the tag cloud from this text file (see figure 6). figure 6. the tag cloud the navigation menu highlighting the current page’s menu item was accomplished using the technique described at http://www.456bereastreet.com [4] among others. a unique id or class is added to the body element of the page. then css rules are written to match each body id or class with the appropriate menu item anchor tag: /* highlight current page’s corresponding navigation menu item */ body#home a#homenav, body#tagcloud a#tagcloudnav, body#about a#aboutnav, body#help a#helpnav, body#contact a#contactnav, body#search a#browseallnav, body#browseall a#browseallnav, body#browsetitle a#browsetitlenav, body#searchform a#searchnav, body#creators a#creatorsnav, body#permissions a#permissionsnav{ font-weight: bold; background: #ff3300; color: #fff; } presentation and copy styles aside from its functionality, the website needed to be attractive in a way that reflected the history of the strips. hence, the fonts chosen were those that were in keeping with the style of the original comic strips. images for the site headers and page background were actual cropped portions of the original pulls. the colors used were exciting and bright to denote the entertaining aspect of the content. the copy itself was also written in a fun, lighthearted, but informative sense to match the mood of the strips. the general styles and layout were loosely based on the university of sydney stylesheets, so there were some constraints on the graphic design. serving the site: system architecture part of what made it possible to create this site within a limited time was our established infrastructure. at the university of sydney library, we have a base xtf server setup using vmware. this virtual machine comes pre-packaged with all the components required to run an xtf website (see table 1 below), including xtf itself. we can create an xtf clone, implement name changes, firewall rules and dns setup, usually within 3 hours. table 1: university of sydney library it services standard server setup component version comments operating system redhat enterprise linux 5 java sun java 5 web server apache httpd v2.2 handles incoming requests, passing them to tomcat (using mod_proxy_ajp) servlet container tomcat v5.5 serves the xtf servlets proxying mod_proxy_ajp pushes port 80 requests to port 8080 (tomcat) the mod_proxy_ajp setup is contained within the apache virtual host for frontiers: proxypass / ajp://localhost:8009/ proxypassreverse / ajp://localhost:8009/ in this manner, we can serve the home page as: http://frontiers.library.usyd.edu.au instead of: http://frontiers.library.usyd.edu.au:8080 conclusions outcomes referral data from google analytics and our in-house web usage statistics indicate that the sources of incoming traffic to the site include: direct traffic (primarily as a result of the launch) the sydney university website the sydney university library website wikipedia the sydney morning herald site (article) the california digital library xtf website (featured application) google search facebook we have had a lot of qualitative positive user feedback, with comments such as: “…the design is innovative and fresh…”, martin haye, california digital library “…it is beautifully done and really showcases a great way of displaying and granting access to such an ephemeral art form as the newspaper comic strip”, bernard caleo, comic book maker and publisher “…i’m delighted to discover that frontiers of science is being preserved for posterity!”, andrew collier cameron, professor of astronomy at the university of st andrews we have also seen traffic peaks during times of media interest, such as the frontiers of science exhibition (and comic drawing competition) at the university of sydney scitech library in february 2011. this points to the value of marketing and media used creatively in conjunction with the website. of course, there are budgetary constraints on these activities. issues following the launch of the site, there were several issues that came to our attention. firstly, to get the site done quickly, we used zoomify. this worked well, but constrains the use of the site to flash-compatible devices. therefore, zooming and panning the pulls does not work on iphones and ipads. ideally, a javascript-based mechanism would be employed for the zooming and panning functionality. the site was w3c validated as far as practicable for html 4.01 transitional or xhtml 1.0 transitional. validating proved difficult primarily due to the embed code for zoomify. changing from a flash-based script to javascript would circumvent these issues. in some cases, css code for the layout and styling proved difficult and was less than ideal. for example, aligning the search/limit results boxes with the jquery tools scrollable component could be better. also the arrows for scrolling were not clear to some users, but changing them would have taken some time. these issues may have been avoided had the project received more funding for technical staff (i was the sole programmer available to work on the site, along with my other existing projects). future work one of the major enhancements planned for the site is using ocr to scan the original pulls and obtain full text for indexing. full text searching would then be a possibility. digitization and tagging of additional pulls is also yet to be done, to expand the current database of material available. in addition, we are currently in the process of migrating our server java environments from sun jdk to open jdk. pending funding, there is also the possibility of video interviews and other multimedia additions to the site. social networking and blogging would also be valuable tools for the future of the site, allowing greater user interaction with the site and with the general and academic community. acknowledgements judy campbell and professor bob hewitt – initial ideas. julie price and nicholas heath, rare books & special collections library, university of sydney – site planning and scanning of pulls. susan murray-smith, sydney escholarship, university of sydney – finalizing contracts. alison muir, science foundation for physics – grant to enable digitization. abbie thomas, abc science – frontiers exhibition. digital and print media, university of sydney – advice on university of sydney style guidelines. miriam butler and angela raymond – rights and licensing. professor peter harrowell – for tagging of the pulls. references [1] the extensible text framework, http://xtf.cdlib.org/ [2] the java xpath api, http://www.ibm.com/developerworks/library/x-javaxpathapi.html [3] creating an accessible tag cloud in php and css (with mysql), http://stevenyork.com/tutorial/creating_accessible_tag_cloud_in_php_css_mysql [4] highlighting the current page’s menu item using css, http://www.456bereastreet.com/archive/200503/setting_the_current_menu_state_with_css/ about the author gary browne (gary.browne@sydney.edu.au) is a development programmer for library it services and sydney escholarship at the university of sydney library. he enjoys the challenge of cobbling together bits of code to produce things of simple, functional beauty. gary is interested in the application of web technologies to improving the end-user experience. subscribe to comments: for this article | for all articles 10 responses to "web-based software integration for dissemination of archival images: the frontiers of science website" please leave a response below: code4lib journal – n° 14 « pintiniblog, 2011-07-27 […] web-based software integration for dissemination of archival images: the frontiers of science websit… […] anon, 2011-07-29 oh, and funny, it does not work without javascript. if you had have just had a simple click on the thumbnail to view the larger image, you wouldnt need flash or javascript. indeed, you acknowledge the problems of using flash, not usable by some useragents. javascript in and of itself (without flash) is also unusable on by some useragents. the correct way to design a website such as this, is to make it usable with just html, then make it nice looking with css, and then add javascript to add all that nice stuff like lightbox or whatever. gary browne, 2011-07-31 @anon, thanks for your feedback. granted, the site is not ideal in this respect. this stems from the difficulty of balancing several projects at once, tight deadlines and conflicting user requirements by which one is bound. i have attempted to acknowledge this in the article. thanks also for your development tips, they are much appreciated. i hope i do get some time in the future to improve the site further in terms of accessibility. kind regards, gary tesla generator, 2011-08-19 i like this because it’s highly functional and extremely useful for the user. i’m going to have to go over the information a little bit longer, but at least it’s all here. thanks for taking the time to write this up! gary browne, 2011-08-21 @tesla generator, glad this information is useful to you. kind regards, gary serge zenin, 2012-06-23 wow, this is amazing. i never knew you could make scrollable previews like that. just looking at all the code to produce this, sorta makes my head explode, but i’m definitely bookmarking this for reference! so cool, how much time did you spend on this? gary browne, 2012-06-24 @serge, thanks, i hope your head is ok now :) the site was done in 6 weeks, not including digitisation of the strips. the jquery tools library is great to work with, i used it for the scrollable thumbnails. i combined that with the search capabilities of the extensible text framework (xtf). cheers, gary mike, 2012-07-02 awesome, just checked out the live site. i love the zoom effect when you click on the comic strips. gary browne, 2012-08-01 @mike, thanks for the feedback. zoomify is pretty easy to work with, though next time i’d be more likely to use or write a jquery plugin. cheers, gary xtf, visualizar colecciones « guilleten, 2013-01-26 […] de imágenes en el escenario descrito?. punto crítico sobretodo en el intercambio de metadatos. browne (2011), por ejemplo, propone la creación de una clase java que permita la extracción de metadatos cuando […] leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – shining a light on scientific data: building a data catalog to foster data sharing and reuse mission editorial committee process and structure code4lib issue 32, 2016-04-25 shining a light on scientific data: building a data catalog to foster data sharing and reuse the scientific community’s growing eagerness to make research data available to the public provides libraries — with our expertise in metadata and discovery — an interesting new opportunity. this paper details the in-house creation of a “data catalog” which describes datasets ranging from population-level studies like the us census to small, specialized datasets created by researchers at our own institution. based on symfony2 and solr, the data catalog provides a powerful search interface to help researchers locate the data that can help them, and an administrative interface so librarians can add, edit, and manage metadata elements at will. this paper will outline the successes, failures, and total redos that culminated in the current manifestation of our data catalog. by ian lamb and catherine larson in september 2015, two researchers with the federal reserve board in washington, d.c. released a study that shook the field of economics [1]. after attempting to replicate the findings of 67 papers published in 13 different economics journals, andrew c. chang and phillip li were only able to successfully reproduce the results of 33% of the papers. even after directly contacting the original authors for help, this number only rose to 49%, meaning that more than half of the sample was irreproducible. chang and li’s paper was unflattering to the field, but it was just one of a growing number of similar studies in other sciences. just a few months before, in august 2015, science published a damning study proving that only 36 out of 100 psychology studies were reproducible, even when using the original study materials [2]. these papers made headlines, but what made them possible is a growing movement to encourage the sharing of research data for reuse or reproduction. publications such as plos [3], and government bodies such as the white house’s office of science and technology policy now require that authors share data or provide a data sharing plan [4]. and libraries are particularly well placed to foster this sharing, with our expertise in resource classification, metadata, and discoverability, as well as our close relationship with our research communities. at nyu langone medical center, the department of population health (dph) informed their library liaisons that researchers were having difficulty locating datasets relevant to their work. even when they decided on a dataset to use, actually getting access to it could be a labyrinthine process. many large public health datasets are split into multiple sections that are hosted on different websites, making it difficult to know which one to use. they can be subject to confusing licensing agreements, and often even the data itself is confusing or lacks a thorough data dictionary. for these reasons, researchers often try to work with an individual who has used the dataset before and can serve as a guide of sorts, but they would have to rely on word-of-mouth to locate these individuals. a project began to take shape that would marry the classification, discoverability, and web development abilities of the health sciences library with the research experiences of faculty at the dph. with this project, it was hoped that we could help researchers locate useful data for their studies as well as provide a way for them to share their own data with the world [5]. a metadata schema would have to be developed that could accommodate these public datasets as well as data produced in-house by nyu’s own researchers. once developed, a “data catalog” website would be created that combined a full-featured, faceted search with this rich metadata schema, and provided an administrative interface so non-technical staff could perform basic crud functions on these datasets without any assistance. given the dph’s focus on large, population-level datasets such as the us census and the american time use survey, it was decided that we would not store datasets locally. this avoided several practical problems including the cost of indefinitely storing petabytes of data. it also allows us to catalog data that researchers may be reluctant to share just yet, or data that contains personal information that has not yet been de-identified. groundwork the first step was for librarians to begin work on a metadata schema using examples from existing data catalogs. nature’s journal scientific data, which began publishing in 2014, is a wonderful example of data sharing and contains detailed, peer-reviewed descriptions of scientific datasets. however, we found their “data descriptor” article type to be too detailed, and too limited in scope for our purposes. we also looked at existing tools such as dryad (http://wiki.datadryad.org/metadata_profile ), datacite (https://schema.datacite.org/meta/kernel-3.1/index.html ) and the w3c data catalog vocabulary (https://www.w3.org/tr/vocab-dcat). in fact, one of our librarians looked at 45 nih data sharing repository metadata schemas to identify their most common metadata elements [6]. his research informed both the nih’s biocaddie initiative [7] – which attempts to define a standard data descriptor format for all disciplines – and our own catalog. we believe strongly that using standard or at least similar metadata elements will be an advantage in the future and will increase the compatibility of our catalog with others. simultaneously, developers were searching for existing data catalogs and trying to learn from them. we found one such example made using drupal, which was convenient because several of our own websites run on drupal, and as a full-featured cms it would include the crud (create, read, update, and delete) functionality and administrative interface we desired. we were excited by this possibility and got to work building a prototype. before long, however, it became clear this was not a workable solution for us. though our metadata at this time was not complex, making drupal aware of it and searching it was not easy. within drupal, a content type was created that could accommodate the metadata, but building a search interface that worked anything like we wanted proved difficult. developers felt they were wrestling with drupal (perhaps a common feeling among drupal developers) and this strategy was dropped. more promising was the example set by the national snow and ice data center (nsidc) (http://nsidc.org/data/search/), affiliated with the university of colorado – boulder. their search page is a slick javascript-based page, written mostly with the backbone framework and using modern tools like underscore.js and require.js. their search worked well by our standards, and the results even included helpful maps showing the geographic areas covered by the dataset. a whole new world seeing the success that nsidc had using backbone – and because our developers were interested in learning this new tool – we set out to see what could be done. it was also decided at this time that to get the full-featured and flexible search we needed, apache solr would be the only reasonable solution. we had some limited experience with solr after using it on another project, but it was our first time using it for a purely search-focused project. and yet, after surprisingly little time, a working prototype of the data catalog was up and running that used solr, backbone.js, jquery and underscore.js almost exclusively. some things about backbone proved to be incredibly useful, almost game-changing compared to the traditional server-side world of php and python that our developers were used to. one such piece was the built-in publish/subscribe event system, whereby any object in your application can listen for events emitted by any other object, and trigger their own. this is a common design pattern, but in backbone, it could not be simpler to set up. by simply declaring a top-level variable to hold the events object (var megaphone = _.extend({},backbone.events);), any view (or other object) in your application can communicate with any other without directly referencing it. so if an object does something noteworthy – like, say, receives new results from solr – it can publish this fact easily by using our “megaphone” system: megaphone.trigger("solr:newresults",data); any other object in your application can listen for this event as easily, and also specify a method to be called when it happens (in this case showresultsfound()) var resultsview = backbone.view.extend({ initialize: function() { this.listento(megaphone, "solr:newresults", this.showresultsfound); } }); using this paradigm, views can function almost as independent actors, listening for whatever is relevant to them and then reacting to that data according to their own sets of rules. but the real beauty stems from the fact that in backbone, these views are attached directly to dom elements on your page. this becomes a very fun way to think about your site, and it’s one of the chief advantages of a javascript-based front-end framework: a webpage that would be more or less static after being served up by a server-side framework now becomes malleable and vibrant, almost alive, with each dom node able to update itself and to influence any other nodes to do the same, each playing by their own rules. with some creative animations this effect can really be amplified. we liked that backbone, though it is “minimalist,” did enforce a few good practices. the model-view-controller paradigm and the use of restful url structures are some of the few rules that backbone has, and they are good ones. while your javascript files will grow greatly in size, at least the functionality is compartmentalized in a way that makes sense and will continue to make sense indefinitely. a third appealing aspect of backbone is a direct result of that minimalism: its much-touted flexibility. coming from drupal, a project which is approaching 15 years old and which has its own way of doing things to the point of having its own ecosystem, it was a great relief for developers to be able to come up with an idea and immediately know how to implement it, and to be able to use relatively basic javascript and html to do so. we did not have that sense of “wrestling” with the framework that we had with drupal. rather, we had a ton of freedom with how the page looked and how it interacted with other systems and with itself. and yet, despite these and other advantages, life with backbone was not perfect. in fact, as our requirements grew and work began on the authentication and content management functionality, we saw some very stark limitations. it became clear that, while backbone is commonly referred to as a framework, its name tells the real story: it is really just a skeletal add-on to jquery that provides a few useful features. and while backbone is often praised for being unopinionated, after a while one gets the impression that it is actually quite opinionated when it comes to the things it doesn’t do — which is most things. functionality that those coming from other mvc frameworks might take for granted will have to be crafted from scratch in backbone, which means you cannot “stand on the shoulders of experts.” for instance, authentication and sessions will have to be handled manually, leaving room for error. all decisions about application structure and memory management fall to the developer, and there are some known pitfalls that cause memory leaks. to address these shortcomings, a plethora of additional javascript libraries have been created to give more structure and functionality to backbone. these include marionette (http://marionettejs.com/), backbone boilerplate (https://github.com/tbranyen/backbone-boilerplate), chaplin (http://chaplinjs.org/), epoxy.js (http://epoxyjs.org/), backbone session (https://www.npmjs.com/package/backbone-session), and others. but adding more libraries would seem to defeat the purpose of using such a lightweight framework in the first place. most damning for our purposes, however, was backbone’s naïve assumption – a strong opinion, if you will – that it will always be interacting with a perfectly crafted restful backend that communicates in json. basically, if the server you are communicating with does not strictly follow the restful url structure and use the http methods as prescribed, you will spend significant time overriding basic backbone functions in order to communicate with it. solr does speak json, but like most other search software, it has its own url structure and relies on numerous url parameters – many with their own unique syntax – to craft the search. so, we had to create a php script that essentially translates between backbone and solr. while it wasn’t such a big deal at first, as the search functionality grew increasingly complex, this translation became more and more difficult to manage. it dawned on us that for our application to be of any use, we would either have to use numerous additional javascript libraries, or rely on more of these server-side scripts to handle basic functionality. whether approaching it from a theoretical, systems design standpoint, or a practical, long-term maintenance standpoint, neither solution was appealing. once more unto the breach simultaneously, librarians were cataloging a growing list of datasets, and their need for a full-fledged administrative interface to manage their metadata was growing. this would require the addition of ldap authentication with our institution’s active directory, session management, detailed input forms with validation, and much more – all those things that backbone doesn’t do. as planning for this next phase started, and our heads buzzed with a dozen add-on javascript libraries that could help us, a painful decision was eventually made to put the backbone version on the backburner and move forward with a more traditional server-side framework: symfony (https://symfony.com/). while starting again was not an easy choice to make, the advantages were immediately obvious. symfony has an authentication system that meshed relatively easily with our institution’s active directory system and automatically generates csrf tokens. session management is robust, flexible and built-in, and memory management is almost never a concern. the form system is a bit inelegant and has a steep learning curve, but form validation is a dream, performed automatically based on php annotations in the source file itself – the same annotations used by the doctrine orm tool to construct the database automagically. before we look at how easy it is to set up the database, let’s take a quick look at the data model we ended up with. as mentioned previously, librarians drew heavily on existing metadata schemas. we had our own unique needs, mostly stemming from our desire to include both “external” datasets (such as the us census) and “internal” ones (those produced in-house by our researchers). for example, while it’s fair to assume that external datasets are catalogued elsewhere, for internal datasets, our own catalog is likely to be the authoritative reference. to adequately describe both types of data would require a somewhat complex schema. for example, external datasets have a “local expert” field, which denotes an individual at our institution who has agreed to help fellow researchers who wish to use one of these datasets, which can be enormous and quite confusing. those who wish to use internal datasets, however, are encouraged to contact the author directly, as he or she is the expert and is usually the only one who can provide access to the data. there is also the challenge of being able to describe datasets from different fields. at this point we’ve been working closely with the department of population health, but we will eventually need to accommodate data from our clinical and basic science departments, so we included fields where we can describe equipment such as microscopes and imaging devices. another source of complexity was translating this all to a normalized relational database. many of these metadata fields represent items that are entities unto themselves and need their own tables, relating back to the dataset via a many-to-many relationship. for example, a publisher has an existence apart from any dataset it may be associated with. by creating a separate publisher entity (in symfony and in the database), any information about a publisher can be managed independently and changes are reflected immediately in all related dataset records. this is all pretty standard, but developers were concerned with the sheer number of entities required to faithfully represent this flexible model. figure 1 is a partial representation of our database, showing most of the complexity. in total, our model required 24 separate entities and 54 database tables. to display a full dataset record requires at least 20-30 database calls, as reported by symfony’s built-in debugging tools. while the database was conceptually pure, we began to wonder if it would perform well enough in the real world to be of any use. perhaps some entities should be reevaluated for possible merging back into the main dataset entity. figure 1. despite our concerns about performance, in the end there were no issues whatsoever. even the most complex dataset records load almost instantaneously. the doctrine object-relational mapping (orm) tool that ships with symfony (http://www.doctrine-project.org/) handles most of the database communication behind the scenes, so it’s hard to be sure what’s happening without detailed knowledge of how it works. it does seem to avoid complicated joins in favor of separate queries, which probably helps with responsiveness. but we credit symfony’s multiple layers of caching for most of the boost. at the database itself, there is of course the built-in mysql query cache, which speeds up a repeat query by more than half in our case. interestingly, the doctrine orm also maintains its own query cache on the web server, as well as a results cache and metadata cache. additionally, symfony provides two options for storing a class map to help it find your source files. there is a built-in option that uses composer (a php dependency manager – https://getcomposer.org) to build a giant array of the paths to all your class files. the other option – and the one we chose – is to use the apc byte code cache to store this class map for ultra-fast retrieval. we also went further and used apc to store a bootstrap cache, which is a big file containing the full definitions of all our classes so that in most cases, symfony doesn’t have to load the source files at all. most of this caching was enabled by default except the apc caches, which we enabled simply by uncommenting a few lines in symfony’s configuration files. even without the performance issues, we would have had some real struggles constructing this network of relationships by hand, so it was great that we didn’t have to. the process is done almost entirely automatically by use of symfony’s built-in command line tool. while defining classes and properties, the developer can use php annotations to describe what they want the database to look like (http://doctrine-common.readthedocs.org/en/latest/reference/annotations.html). the command-line tool then reads all these annotations and translates them to sql commands to create the database. we loved this system because it enabled us to define the database structure in the same place where we are creating the object. for example, the following code not only declares a few object properties, it also describes the table structure for doctrine, and sets constraints on the object which are then enforced by mysql as well as by symfony’s form validation service. use doctrine\orm\mapping as orm; use symfony\bridge\doctrine\validator\constraints\uniqueentity; /** * @orm\entity * @orm\table(name="publishers") * @uniqueentity("publisher_name") */ class publisher { /** * @orm\column(type="integer",name="publisher_id") * @orm\id * @orm\generatedvalue(strategy="auto") */ protected $id; /** * @orm\column(type="string",length=255,nullable=false) */ protected $publisher_name; } in the annotations on the class, we’ve told doctrine that this is a new entity represented by the table “publishers.” we’ve also told the form validator that the “publisher_name” field needs to be unique. furthermore, using annotations on the “publisher_name” field, we’ve told doctrine that the database must limit this string field to 255 characters and allow null values. though we didn’t explicitly tell the form validator about these two constraints, it will honor both of them as part of its “field options guessing.” now, we just run a simple command and doctrine sets up the database just the way we asked: php app/console doctrine:schema:update --force annotations are also used to set up join tables, foreign keys, and pretty much anything else you want out of a database. while it’s nice to have everything in one place, these declarations can get very verbose and represent almost a pseudo-language that must be learned. look at this annotation declaring the relationship between the dataset entity and the related equipment entity: /** * @orm\manytomany( * targetentity="relatedequipment", * cascade={"persist"}, * inversedby="datasets" * ) * @orm\jointable( * name="datasets_related_equipment", * schema="datasets_related_equipment", * joincolumns={ * @orm\joincolumn( * name="dataset_uid", * referencedcolumnname="dataset_uid" * ) * }, * inversejoincolumns={ * @orm\joincolumn( * name="related_equipment_id", * referencedcolumnname="related_equipment_id" * ) * } * ) * @orm\orderby({"related_equipment"="asc"}) */ protected $related_equipment; much of this can supposedly be done automatically, but there are some quirks and we found that explicitly defining our relationships and join tables in this way prevented symfony from misinterpreting us. integration with solr, while imperfect, was much easier than in backbone. we used symfony’s concept of service containers to create an object that is globally accessible in the application and would provide a simple interface whereby any object could communicate with solr. we also created an object “searchstate” to hold the current state of the user’s search – what facets are currently activated, what keywords have been entered, what page they are viewing, etc. the http request is fed to the searchstate object, which reads the url parameters and sets its properties accordingly. it then gets passed to the solrsearchr service, which painstakingly constructs a url that solr will accept, and then fetches the results. the results are fed into a “searchresults” object that is eventually rendered through a template and presented to the user. the process looks like this in our controller: // reading the search state $currentsearch = new searchstate($request); // calling the service $solr = $this->get('solrsearchr'); // setting search parameters in the service $solr->setusersearch($currentsearch); $resultsfromsolr = $solr->fetchfromsolr(); $results = new searchresults($resultsfromsolr); we are pleased with this elegant solution, but we did not use the word “painstaking” lightly. while symfony was happy to communicate with solr outside the constraints of a prescribed url structure (unlike backbone), it turns out that solr is extremely particular in its own ways. for example, solr has incredible support for faceted searches, but the url syntax required to set it up is somewhat verbose, and the date facets (e.g. limiting the search to between 1975 and 1984) use a different syntax than non-date facets. and the facets are just one of seven distinct url components that are necessary to represent our search. just as an example, a search query where the user has specified a keyword and activated one date facet and one non-date facet looks like this: http://yoursolrserver.com/solr/data_catalog/select? q=text%3a%22health%22+or+dataset_title%3a%22health%22 &facet=true&amp;facet.mincount=1 &facet.field=subject_domain_fq &facet.field=subject_geographic_area_fq &facet.field=access_restrictions_fq &fq=subject_domain_fq%3a%22quality+of+health+care%22 &fq=dataset_years%3a%5b1975-01-01t00%3a00%3a00z+to+1984-12-29t12%3a59%3a59z%5d &facet.range=dataset_years &f.dataset_years.facet.range.start=now/year-40year &f.dataset_years.facet.range.end=now/year &f.dataset_years.facet.range.gap=%2b10year &f.dataset_years.facet.range.hardend=true &f.dataset_years.facet.range.other=before &sort=score+desc &start=0 &rows=10 &fl=id,dataset_title,dataset_alt_title,description,subject_domain,access_restrictions,local_experts,subject_geographic_area,dataset_start_date,dataset_end_date &wt=json not apocalyptic, but it does take some care to dynamically produce a search url like this that is valid every time. then, when the facets are returned as part of the json response, the facet counts (e.g. how many items the search would return if limited by that facet) are in an array separate from the rest of the results, with a flat structure that must be walked through: ["delivery of health care",4,"health care costs",2,"population characteristics",1] we find solr to be easy to use overall and very powerful. while there is endless customization available to advanced users, a totally stock installation of solr after maybe five minutes of setup time is already better than any other search tool we’ve seen. formal complaints the last piece of the puzzle was the administrative interface. we’ve alluded to the form system before, and dealing with the forms was probably the most challenging part of this project. with such a complex data model to enter, as well as the different metadata elements required for internal vs. external datasets, and the management of 16 of our related entities (publishers, local experts, grants/awards, etc.) we ran up against some limitations of symfony’s form system but, in the end, are happy with the results. symfony does have a very easy option to render an entire form. basically, you would pass a form object to your template in your controller, and then in the template (twig, in this case) simply write: {{ form(form) }} this will render the form tags, all the fields, as well as the submit button and csrf tokens. however, this is not very useful in practice. you can’t control the order the fields appear in, there’s no control of css tags, and in a very long form (like ours) there’s no way to break the fields into sections to improve usability. in the end, most symfony developers will be declaring each form field manually in a special class file, for each entity that needs a form. it is a painstaking and repetitive process that is somehow more tedious than writing the entire form in raw html. for instance, the configuration of one of our form fields looks like this: $builder->add('publishers', 'entity', array( 'class' => 'appbundle:publisher', 'property'=> 'publisher_name', 'required' => false, 'query_builder'=> function(entityrepository $er) { return $er->createquerybuilder('u')->orderby('u.publisher_name','asc'); }, 'attr'=>array('style'=>'width:100%'), 'multiple' => true, 'by_reference'=>false, 'label' => 'publishers', )); that’s 12 lines to configure one form field, which is actually a pretty basic field with little customization. much like those complicated database annotations, this markup is a little too verbose and idiosyncratic for our taste. also, our form has 41 fields. in the template it is much easier, and most fields can be rendered like so: {{ form_row(form.publishers) }} but that’s not the whole story. imagine that you have 100 publishers in the database. to prevent spelling errors and duplicate entries, and to improve usability, it would be prudent to include autosuggest in this form field. it would also be very useful to be able to add a new publisher record directly from the main data entry form, without having to leave the form and come back after creating that publisher. librarians requested both of these features, a totally reasonable request, but one which required quite a bit of work to integrate with symfony’s form system. we found symfony’s documentation on some aspects of its form system to be sorely lacking. the select2.js library includes a number of great features to really spruce up a <select> element, including autosuggest. this part was relatively easy. the difficult part was being able to select a brand new entity that was just added to the database (using a modal box containing that entity’s form). the main entry form must be made aware of the entity that was added in the modal so it can be added to the <select> box. we ended up using html data-* attributes to pass the name of the entity back and forth between the main form and the modal popup. we also needed to do some string manipulation to match the type of the entity with symfony’s generated id tags in order to find the proper <select> box. we eventually got this all sorted out — but the form still threw errors on every attempt to submit. it turns out that symfony uses the actual primary key value from the database as the “value” for each option in a <select> field. this means that we have to calculate the key for the newly-added entity just as the database would so we can use it as the “value” in the options list. in the end, the javascript needed to accomplish all that looks like this: // on submitting the form from the modal $('#addentityformmodal').on('submit','form', function(e) { e.preventdefault(); $('#loading-indicator').show(); var form = $(this); postform(form ,function(response) { $('#addentityformmodalcontent').html(response); // get the name of the entity type (e.g. publisher), and the name of the item that was just added var displayname = $('#addentityformmodalcontent #entity-display-name').attr('data-displayname'); var addedentityname = $('#addentityformmodalcontent #added-entity-name').text().trim(); // figure out the id generated by symfony for the relevant form element var selectboxid = 'select#dataset_' + displayname.replace(/ /g,'_') +'s'; // get the max value of the options, increment like the database would, add new item to end var optionslist = $(selectboxid + ' option').map(function() { return $(this).attr("value");}).get(); var maxvalue = math.max.apply(math, optionslist); var nextoption = maxvalue + 10; // if your database ids increment by 10 $(selectboxid).append(''+addedentityname+''); // get the list of already-selected items, if any, and add ours to the end var currentvals = $(selectboxid).val(); if (currentvals) { currentvals[currentvals.length] = nextoption; $(selectboxid).val(currentvals).trigger("change"); } else { currentvals = [nextoption]; $(selectboxid).val(currentvals).trigger("change"); } }); return false; }); the javascript isn’t that complicated, but due to a lack of official documentation, and a very limited number of blog posts from people who had attempted a similar thing, it took some time and frustration to arrive at a working solution. furthermore, from a conceptual standpoint, we don’t love how tightly coupled this form is with the database, nor how much it relies on finicky string manipulation and data-* elements. however, the form works excellently now, and librarians and developers alike find it easy to use. in fact it is hard to imagine a form of this complexity being very usable at all without these features. despite the length of the form, we decided to keep it all on one page rather than going with a tabbed or multi-page setup. since this form would be used both for the initial data entry and for editing metadata in existing entries, we wanted to make sure it would be easy to scan the whole form, reference other sections, and jump around as needed. these goals are very difficult to meet with a tabbed or multi-page form. for example, it’s easy to lose your place in a large multi-page form, and browsers tend not to handle the back button well when dealing with forms. since most of the fields in a multi-page or tabbed form are hidden at any given time, it’s difficult to refer back to what you entered somewhere else. compounding that difficulty is that it’s often not clear which fields live on which page or tab. as a result, you must click out of your current tab/page in order to search for the field on another (e.g. “did we put the ‘local expert’ field on the contact information page, or the access information page?”). our decision to individually render each form field, while tedious, allowed us to split it up into sections with plenty of white space, which we find quite manageable. section headers are prominent, making it easy to jump to your desired field, and no more than a few fields scroll into view at a time, keeping it from overwhelming the user: figure 2. conclusion the data catalog went live last year. we began collecting statistics shortly after launch, and we’ve had over 9,100 unique page views, with almost 200 users clicking through to access a dataset. we are working with the department of population health and others in our institution to encourage more researchers to take this opportunity to make their data discoverable to the world and improve the quality of scientific research wherever possible. this year, we also plan to release our source code on github so that other institutions can set up a catalog of their own with much less development time. our application provides a json output, as well as a linked open data output in the form of json-ld embedded into every page, making it theoretically possible for future installations of the catalog to link up and for it to become a kind of metasearch across institutions. of course, this is a long way away, and we are content with the progress we have made so far. we invite you to visit our catalog at: http://datacatalog.med.nyu.edu and feel free to drop us a line with any questions or comments. figure 3. references [1] board, f. r., chang, a. c., & li, p. (2015). is economics research replicable?? sixty published papers from thirteen journals say “ usually not .” [2] open science collaboration (2015). estimating the reproducibility of psychological science. science, 349(6251), aac4716–aac4716. http://doi.org/10.1126/science.aac4716 [3] one, p. (2014). sharing of data, materials, and software. retrieved from http://journals.plos.org/plosone/s/data-availability#loc-acceptable-data-sharing-methods [4] holdren, john p. (2013). increasing access to the results of federally funded scientific research [memorandum]. washington, dc: office of science and technology policy. retrieved from https://www.whitehouse.gov/sites/default/files/microsites/ostp/ostp_public_access_memo_2013.pdf [5] read, k., athens, j., lamb, i., nicholson, j., chin, s., xu, j., … surkis, a. (2015). promoting data reuse and collaboration at an academic medical center. international journal of digital curation, 10(1), 260–267. http://doi.org/10.2218/ijdc.v10i1.366 [6] read, k. (2015). common metadata elements for cataloging biomedical datasets. figshare. http://doi.org/10.6084/m9.figshare.1496573 [7] members, w. (2015). wg3-metadataspecifications: nih bd2k biocaddie data discovery index wg3 metadata specification v1. zenodo. retrieved from https://zenodo.org/record/28019#.vsdmmpkrjhe about the authors ian lamb (ian.lamb@med.nyu.edu) is a full-stack web developer at the new york university health sciences library, part of the nyu school of medicine. he focuses on building friendly and usable systems to advance the institution’s clinical, educational and research goals. catherine larson (catherine.larson@med.nyu.edu) is the head of the systems & technology department and systems librarian at the new york university health sciences library. her work centers on providing technical support and usability assessment for the library’s systems with emphasis on design and user experience. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – modular mobile application design mission editorial committee process and structure code4lib issue 18, 2012-10-03 modular mobile application design this article describes the development of the minrva library app for android phones. the decisions to build a native application with java and use a modular design are discussed. the application includes five modules: catalog search, in-building navigation, a barcode scanning feature, and up to date notifications of circulating technology availability. a sixth module, amazon recommendations, that is not included in the version of the app that was released is also discussed. the article also reports on the findings of two rounds of usability testing and the plans for future development of the app. by jim hahn and nathaniel ryckman minrva modules and future-proofing mobile apps minrva is designed as a modular mobile software application. this conceptualization is novel to the field of mobile computing in library settings, which generally has developed one-offs that run the risk of obsolescence over time. native app development supports a module system that would not be a one-off but rather would grow in usefulness and services over the life of the project. the minrva app we are developing is a test case in the modular design approach where each function of the application is a module that can be refreshed or added to as needed and in a collaborative manner. in this paper we suggest advantages to the native developer environment, particularly java modules, while reporting through qualitative user study results how native components of mobile apps are useful in library settings. figure 1 minrva conceptual model the app is comprised of modules that are configurable to local library user needs. features of the app are not possible with traditional desktop computing. these services include data integration among the library wayfinding (gps-like functionality), location-based recommendation, camera data and library opac data. the 1.0 release of minrva, available on the google play website (http://goo.gl/r7pyy) includes five interactive modules. there is a sixth module (the suggest module) that is not available on google play at this time. the functions we studied include a barcode scanner module, a suggest module, a home module (main state), a search module, a wayfinding module, and a technology equipment availability module. scanner the barcode scanner allows a user to scan a book with her phone and get a book summary. the scanner can also load book summaries into the home module. suggest the suggest module sends a request to amazon for a list of isbns for items in the collection that are relevant to the scanned library item. the suggest module incorporates amazon recommendation data into the user’s stacks location and subject interest. home the home module is the main state of the app, and lists the title, author, and call number location of the item. the home module will also present an image of the book cover with book summary. the home module will show the user current availability and display what departmental library the item is held within. search the search module supports searching by title, subject, or author in the library catalog, and supports a list view of search results. selecting a search result item will take the user back to the home module where they can get the detailed book item information. wayfinding the wayfinding module presents the user with a map of the undergraduate library and also draws a red dot over the section of the library stacks where the desired item is located. technology equipment availability the technology equipment availability module presents a list of the circulating technology equipment that is currently available for checkout in the library. the technology equipment availability module is organized by popularity (based on circulation history) and then lists the number of remaining items for checkout. figure 2 technology equipment availability module consider the following use case in any library setting. the user searches for an item and is then guided to the book using the way-finding module, which has displayed a map that shows the location of the item in the library book stacks. once in the library book stacks, at her desired item, the student can access more information about nearby books, such as a book summary by taking a picture of the book’s library barcode. the student can also learn the current availability for checkout of any circulating technology equipment by tapping the technology availability tab. design of library apps there are three commonly used approaches to design of mobile apps: native, web based, or a hybrid approach of the previous two. the native approach includes developing in compiled languages like java in the android platform, and objective c in the ios platform. web based mobile apps make use of html5 and css and javascript to achieve an app-like interface and experience. jquery mobile is a popular framework for developing web based mobile apps http://jquerymobile.com/. hybrid approaches rely on using mobile frameworks like phonegap http://phonegap.com/. we developed minrva as a native app to future-proof the design. we chose to develop in java in order to make use of java’s modular feature set. each of the components and functions of the minrva app was designed as its own module. these modules are designed for customization across other library settings, meaning that other library systems outside of the university of illinois could use the architecture and web services for their environment. while such an initiative is not without local implementation work, we designed for sharing, and as such systems librarians and other professionals working in library systems should be able to adapt the framework for local needs. each of the android modules is lightweight. they receive all data through a wi-fi, or cellular data connection such as 3g. modules request data from restful web services and receive json feeds in response. a typical restful request to one of the services that we have set up will look like this: http://localhost:8080/minrvaservices/loanabletech the json response returned from the request is as follows (abbreviated): [ { “bibid” : “6399958”, “count” : “2”, “name” : “macbook pro “, “rank” : “1” }, { “bibid” : “6426672”, “count” : “4”, “name” : “macbook charger “, “rank” : “2” }, { “bibid” : “5241959”, “count” : “12”, “name” : “laptop “, “rank” : “3” } … in order to provide the json feeds, we have set up a web application called minrva services. minrva services is a restful web app written in java that runs on apache tomcat 7 (see: huang, wu, guo, 2009 for an example of a project using these tools [1]). minrva services is able to map http requests to java code and automatically serializes its response to a variety of formats by utilizing a dependency called jersey (http://jersey.java.net). the format that the data is serialized to depends on the request header; thus, minrva services is an api that can be used to provide data to multiple types of software including iphone apps, android apps, and websites. each minrva module receives data from a corresponding minvra web service. each web service is composed of a data access object, a data model, and a resource class. the resource class is responsible for mapping restful requests to the proper data access object. the data access object is responsible for retrieving relevant data from a variety of sources and packaging the retrieved data into data model classes. the data model classes are responsible for serializing the web app’s response into a variety of formats including: html, xml, and json. minrva services format responses into json when responding to the minrva app requests because minrva app modules request json in their request headers. the data model used by minrva services to serialize the technology equipment availability module’s data to json (abbreviated): package edu.illinois.ugl.minrva.model.loanabletech; import javax.xml.bind.annotation.xmlrootelement; @xmlrootelement public class model { private string bibid; private string thumbnail; private string name; private string count; private string rank; public model() { super(); this.bibid = ""; this.thumbnail =  ""; this.name =  ""; this.count =  ""; this.rank =  ""; } … with the aid of a java library called gson, the very same data model can be used by the android minrva app in order to parse the json feed created by minrva services. the data model used by the minrva app to parse the technology equipment availability module’s data feed: package edu.illinois.ugl.minrva.loanabletech; public class model { private string bibid; private string thumbnail; private string name; private string count; private string rank; public model() { super(); this.bibid = ""; this.thumbnail =  ""; this.name =  ""; this.count =  ""; this.rank =  ""; } … note from the preceding two examples of code that the java data models used in the android code are the same models used in the java servlet running in tomcat 7.  each of the modules in the minrva application was designed to gather or provide unique information about an item in the collection (book, media, circulating loanable technology) by using a unique identification number (e.g. internal bibliographic identification number). only one unique identification number is loaded into the minrva app at any given time. when the module runs, it will send the unique identification number to the restful web service. the restful web service will then send back information to the module that will then be displayed on the phone. in order to avoid dependencies between individual modules, all of the modules communicate with one another indirectly through the unique identification number. in addition, the data retrieval process is completely independent of the android app because the restful web service is responsible for retrieving the data. as an instructive primer on restful services see richardson & ruby, 2007 [2]. methods for user study rapid prototyping studies are shown to be an effective method to gather user preferences throughout the design phase such that products can be delivered in a shorter development time as well as encompassing desired functionality of end users. the rapid prototyping method described and tested by jones & richey (2000) is the basis for the use study results presented here.[3] this study encompasses qualitative data collection that used detailed interview and observation. this methodology allowed us to understand areas of use and fail points for each module, and to iterate designs before the initial 1.0 release of the app. the use study took place in the university of illinois library, in both the main building and the undergraduate library building. the study had a total of 10 test users. see the appendix for interview questions and observation log. test users were given a $20 gift card for pizza if they agreed to participate in the user study. we observed the following actions of all users: students as they scanned an item in the collection, searched for an item with the search module, located an item in the stacks using the wayfinding module, received amazon suggestions from the suggest module, and discovered what circulating technology equipment was available in the library from the technology equipment availability module. thematic use results thematic use results are the aggregate usability trends over all test users of the app. themes of use represent preferences for desired functionality and overall minrva app performance. these include: intuitive interactions among the modules, library-wide functionality, a deeper integration and emulation of the library catalog. these themes can be applied across all modules since each participant made use of each of the functions of the minrva app. intuitive interactions users of the minrva module made many comments that showed us the need for intuitive interactions. this manifests in two modules especially. the scanner and wayfinding modules require further information to make them easier to understand. since the map may not display exactly in the position the user is facing, it requires a compass-like function  to help the user get to the location in the stacks. secondly, if the scanner is unfamiliar to the user, it will require helpful graphics which illustrate a proper scan and an example library barcode (see figure 3). users expected seamless integration of functionality among the modules. students expected that tapping an element in one module, like the search module, should have the effect of loading this item (book data) into the home module. figure 3 scanner module with usage indicator library-wide functionality library wide functionality was added to the minrva app based on the use studies. we added the ability to search library collections within the disparate library units at the university of illinois. the users we spoke with also desired a consortia search. before we uploaded the minrva app to google play app store, we incorporated a consortia search (to search multiple departmental holdings) into the vufind module – this necessitated a reworking of the unique identifier we used to pass data among the application modules. we re-worked the data architecture while still maintaining adherence to the conceptualization of one unique identifier passed among the component modules of the application (see elmasri & navathe, 2006, p. 704, regarding the importance of unique object identity)[4]. figure 4, below, shows the home module with a library location outside of the undergrad book stacks, which uses the re-designed consortia search. figure 4 home module another library-wide functionality request was the ability to use maps for multiple library locations, not just the undergrad library. directional help in other highly circulating collections such as the grainger engineering library, main stacks, the music and performing arts library and others is planned for integration into the wayfinding module. opac integration the thematic use result of deeper integration and emulation of the online library catalog is indicative of the desire for the minrva app to have a similar element set to that of the catalog. this includes showing item availability, call number, and library location. user preference for deeper emulation of the online catalog is manifested in the desire among students to be able to login to accounts, renew items, as well as request items from the library catalog. test users want everything the catalog can do, plus all of the features that a mobile device is capable of – e.g., indoor positioning systems akin to gps functionality. there were also requests for the amazon suggestions to then be pushed into the catalog search such that the catalog would filter out any amazon recommendations that were not in the library collection. while a rapid prototyping methodology is not statistically significant in a quantitative sense, we were able to collect a set of rich qualitative use data on the modules’ functionality and interaction among the features studied. these use results informed the 1.0 release now available on the google play app store. during our study we also uncovered new modules to develop. future design iteration the use studies informed the design iterations planned for the next release of minrva. based on user feedback we will design a patron login feature so that students are able to renew items and view their library account. test users also requested a favorites module; they asked that we be able to retain their favorites in the app and work on implementing a wish list of items inside of the mobile app. at the time of this article our staff has investigated posting to a user’s facebook timeline from the app, much like the popular goodreads (goodreads.com) mobile application. this feature may be implemented in the 1.1 release of minrva. students asked that we design a module that could function similarly to the technology equipment availability module, where the minrva app can show the user how many group study rooms are immediately available in the library. students also made a feature request to generate citations of the book data from the home module, and then send these citations to their student email account. further, students would also like to have additional library information included in the minrva app, like library hours. our campus office of technology management did not allow the minrva app to be uploaded with the suggestions module.the suggestion module was retrieving information from amazon’s website and re-using this information inside of the app, a use that currently is not an allowed according to the campus interpretation of amazon’s copyright claim. we are exploring the possibility of getting written consent from amazon to make use of their website for isbn referrals and this may be incorporated again into a future version of minrva. conclusion the value of native application development is shown by our thematic use results, which indicate that users do find that a modularly designed native application delivers helpful and useful assistance at point of need in library settings. our plans to integrate desired functionality demonstrate the value of using this modular concept to future-proof the app. our design approach can accommodate students’ ideas as we build out additional modules to the minrva mobile app. we hope other library staff (coders like you) would make use of the existing architecture to implement any of the core minrva services in their local setting. we designed for this purpose. it is our additional hope that library staff might contribute new modules that they have the resources and time to implement. in sharing these modules with the minrva project they may be implemented in any library using the minrva app. this is the key advantage of the app concept: modular collaborative design can pool collective advances from the community of minrva developers. the modules are available at github: https://github.com/minrva at the university of illinois, through a variety of internal and national grants we are actively researching the following features as project next steps: port the minrva app to ios systems implement optical character recognition and augmented reality services for “augmented book stacks” browsing in the scanner module implement an indoor positioning system with wifi data in the wayfinding module incorporate article search into the search modules, along with catalog requests and a favorites feature that posts to a facebook timeline make available additional campus library maps in the wayfinding module design a group room availability and scheduler as a new module design a due date reminder push notification feature we anticipate securing additional grant funding so that we can place minrva services into public and academic libraries across the united states at no cost to any individual library. we are especially interested in working with traditionally under-served areas such that we could place next generation technologies into communities that may not traditionally have been able to design library mobile apps. a pew internet and american life report, smartphone adoption and usage [5] underscores the point that smartphone adoption is highest among non-whites, noting that “44% of blacks and latinos are smartphone users,” indicating that mobile first initiatives can make inroads to serving traditionally under-served populations. we believe that as a special advantage to mobile apps, these tools may increase library use among user groups that have not been engaged with or interested in traditional library services. to move this initiative forward we now require the collaborative efforts of library coders. the minrva project’s github space supports collaborative coding. while such a call to action for librarians has antecedents in open source projects, the library world has yet to see a collaboratively maintained and enhanced mobile software application. the true value for this project is that all libraries and library users devoting time to this effort will see their work power an app that can become ever more useful and feature rich as the project grows. acknowledgements the authors wish to acknowledge the research and publication committee of the university of illinois at urbana-champaign library, which provided support for the completion of this research. notes [1] huang, y.m., wu, d.f., guo, q. (2009), “build a restful web service using jersey and apache tomcat,” http://www.ibm.com/developerworks/web/library/wa-aj-tomcat/. [2] richardson, l. & ruby, s. (2007), restful web services, o’reilly, sebastopol, ca. [3] jones, t. & richey, r. (2000), “rapid prototyping methodology in action: a developmental study,” educational technology research and development 48, 63-80. [4] elmasri, r. & navathe, s. (2006), fundamentals of database systems, addison wesley, boston, ma [5] smith, a. (2011), “smartphone adoption and usage,” pew internet and american life, http://pewinternet.org/reports/2011/smartphones.aspx see also the minrva project: library technology prototyping initiative http://minrvaproject.org/ appendix investigator log please describe any previous experience finding items in the undergraduate library? scanner module does the scanner work in the undergrad library bookstacks? do students scan the library barcode? how do students react to the scanning functionality? do students select any of the elements displayed after scanning? what unexpected things occur? search module how do student use the search module? do they type in any free text for searching? what unexpected things occur? suggestions module how do students use the suggestions? do students select any of the elements displayed? wayfinder module how do students react when the wayfinding application does not work as they expect? what unexpected things occur? how do students make use of the recommendation features? are students able to locate the items that are on the shelf? note any additional observations of student use of the minrva software: interview questions how easy is the application to use? what would make it easier to use? what was hard to do with the application? what was confusing? what was surprising? what do you wish you could have done with the application while you were using it? how useful do you find the application? what would make it more useful? would you recommend it to friends? what do you actually want from a library app? is there something else that should be here that is not here? is this application a worthwhile tool for the library to develop? about the authors jim hahn (jimhahn@illinois.edu) is orientation services and environments librarian, undergraduate library, university of illinois. nathaniel ryckman (nathanielryckman@gmail.com) is visiting research programmer, undergraduate library, university of illinois. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – archive this moment d.c.: a case study of participatory collecting during covid-19 mission editorial committee process and structure code4lib issue 50, 2021-02-10 archive this moment d.c.: a case study of participatory collecting during covid-19 when the covid-19 pandemic brought life in washington, d.c. to a standstill in march 2020, staff at dc public library began looking for ways to document how this historic event was affecting everyday life. recognizing the value of first-person accounts for historical research, staff launched archive this moment d.c. to preserve the story of daily life in the district during the stay-at-home order. materials were collected from public instagram and twitter posts submitted through the hashtag #archivethismomentdc. in addition to social media, creators also submitted materials using an airtable webform set up for the project and through email. over 2,000 digital files were collected. this article will discuss the planning, professional collaboration, promotion, selection, access, and lessons learned from the project; as well as the technical setup, collection strategies, and metadata requirements. in particular, this article will include a discussion of the evolving collection scope of the project and the need for clear ethical guidelines surrounding privacy when collecting materials in real-time. by julie burns, laura farley, siobhan c. hagan, paul kelly, and lisa warwick introduction on friday, march 13, 2020, the executive office of the mayor muriel bowser sent an email to all government employees announcing dc public schools and dc public library locations would be closed until wednesday, april 1 in response to the increasing spread of covid-19 within the region. on march 30, with new york city under a strict lockdown, mayor bowser announced a stay-at-home order for the district of columbia. the opening date for dc public library was ultimately pushed back to may 29, when select library branches started providing curbside services. the people’s archive is the local history center for the dc public library. the first week working from home, the people’s archive manager, kerrie cotten williams, asked how the department could document this moment in history. the same way we were looking back on the pandemic of 1918, future generations would be asking what it was like to be alive during this time. we felt a responsibility to collect from and connect with our community. our first step was to gather information on what other organizations had already started in this vein. the dc history center encouraged people to start diaries with the intention of donation [1] . the arlington public library collected homemade zines[2]. there were also national projects collecting firsthand accounts. the people’s archive’s mission is: “to connect you to unique resources that illustrate the district of columbia’s local history and culture. we are a home of discovery, where diverse stories – past and present – are preserved and amplified.” our department had been rebranding over the course of the last year with increasing emphasis on documenting the everyday experience of life in the district; not just the story of those in power. along with informing the public, creating community and connection was also considered. our department wanted a participatory campaign that would be: easy to collect and share; create a sense of community; and serve as a vehicle to talk about history in daily life. after creating a rough outline of our concept, we reached out to our staff to form a committee to work on the project. having a small group allowed us to be nimble and make quick decisions. this is also the group that made selection decisions and created metadata later in the project. our team included: julie burns, library associate; laura farley, digital curation librarian; siobhan hagan, memory lab network project manager; paul kelly, digital initiatives coordinator; maya thompson, library associate, and lisa warwick, reference coordinator. with our team assembled, we began the work of building archive this moment d.c. (atmdc). staff chose to collect via social media using the general hashtag #archivethismomentdc so the project could be ongoing, capturing future historic moments in d.c. outside of the pandemic. this hashtag clearly stated the project goal without being specific to any event. we did not use covid-19 or coronavirus in the hashtag or any marketing materials for this project. in an effort to limit misinformation, the district of columbia’s joint information center had to review any media from the d.c. government related to coronavirus (covid-19). dc public library’s communications department and the district joint information center advised using the phrase: “in this time of social distancing” in our social media and marketing materials. another consideration was making contributing content to the project as equitable as possible. for those without social media, we created an airtable form that could be filled out for submissions [3]. airtable is a free, collaborative, online database and spreadsheet tool. the public was able to fill out a simple form, view and agree to usage terms, and upload files that were fed into a spreadsheet that staff could access.some of the more varied and creative submissions came through the airtable form or were sent over email. it has to be emphasized that we did not know or predict that we would still be in what d.c. calls phase 2 (limited restaurant seating, no in-person public schools, limited public transportation) nearly one year later. the urgency of the initial two, and later six week, stay-at-home order is partially what led us to collect images, video, and text files instead of daily diaries or other ephemera, as well as using social media as the main collecting point. outreach and engagement dc public library took several interdepartmental actions in order to engage the community and to encourage participation in the atmdc project. first, the people’s archive consulted with the library’s communications department. the main outreach goal was then created to target social media users. a webpage for the project was created on the dc public library website with all the relevant information about the initiative [4]. the communications department promoted atmdc in the weekly enewsletter in mid-april that reaches more than 200,000 people, many of whom are journalists [5] . in addition, the deputy mayor for education included details on the program in their newsletter to members of the d.c. education cluster. the people’s archive promoted information on the project through our departmental enewsletter, the intelligencer, and our departmental social media accounts [6]. dc public library’s main social media accounts also promoted the project. the inclusion of social media as a collection option using #archivethismomentdc was planned as not only a simple way to include more d.c. residents and meet them where they were on social media, but also as an outreach tool: as people posted their submissions to the hashtag, they were promoting our initiative to their followers as well, while engaging the d.c. community in participatory archiving and the telling of their own stories. we sent out all-staff emails about the project asking colleagues to submit content and to share, and in addition we reached out directly to staff who we knew were on instagram and/or twitter and asked them to contribute to model the type of content that fit into our call for submissions. in all of our outreach and anytime someone submitted an entry through the airtable form, we implored the reader to share the initiative with their d.c.-area friends, families, and neighbors. all this coordinated outreach led to an april 16, 2020 article explaining the project in detail and asking for submissions from district residents in washingtonian, a digital and print magazine that reaches more than a million unique readers every month [7]. when researching previous and current similar participatory archiving projects, dcpl connected with the maryland historical society, now renamed the maryland center for history and culture (mchc). initially, this connection was for resources and information sharing, and then led to mchc planning a webinar on this topic since there were so many folks reaching out to them with questions about participatory archiving and to share lessons learned. staff presented on the mchc organized virtual panel, “collecting in crisis: responsive collecting in a digital age” on may 14, 2020 [8]. this covid-19-focused webinar aimed to provide a rough guide for cultural institutions in collecting crowd-sourced born-digital materials. the mchc, virginia museum of history & culture, salisbury university, and dc public library shared their approaches to developing, building infrastructure, and conducting outreach to create different types of responsive collecting initiatives. as submissions started coming in, we retargeted our outreach to d.c. neighborhoods and areas where we had received little or no submissions. we sent emails to d.c. advisory neighborhood commissions, neighborhood associations, and specifically reached out to library branches in neighborhoods with zero or minimal numbers of submissions. while this did assist in engaging these regions more, our submissions were still heavily skewed towards the northwest quadrant of the district. an unexpected result of collecting through social media was that most submissions had a lighthearted tone that is typical to these platforms. while reviewing the submissions against a backdrop of horrible news of rising rates, we became starkly aware that we were not truly capturing the whole picture of daily life. although some posts do show the struggles of this time, most of the submissions could be around the theme of “make the best of it” – distanced wine drinking with neighbors, porch photo shoots, walks in the park, so many cherry blossom images. (note: the tidal basin was eventually closed because of too many visitors to the area.) in september of 2020, eighty-eight photos were chosen from the instagram submissions to be the first items ingested and published into dc public library’s official archive this moment d.c. collection [9]. the library’s communications department published a press release for the launch of the collection to the public (please see appendix i). this garnered three local news articles on the project: one on dcist [10], an article in the washingtonian [11], and another on the local npr affiliate station website [12]. technical workflow collecting born-digital materials from the internet is not new to dc public library; indeed, when the atmdc project began, we intended to utilize tools and techniques that were already at our disposal. in short, atmdc was initially conceived as a web archiving project that would utilize tools such as internet archive’s archive-it platform and rhizome’s (at that time) webrecorder. while these tools can be powerful when applied to certain situations, we found our results targeting them to crawl specific social media hashtags to be dissatisfying not only in terms of data collection, but also in terms of usability. it was during this period of experimentation that we began to crystalize exactly what we hoped this project would be, at least in terms of user-experience. we concluded that, even if our crawls were perfect (which they absolutely were not), the step of replaying warcs through, say, the wayback machine was simply one too many – we wanted a sense of familiarity and immediacy in the end product more in line with a traditional digital collection. we therefore turned to the twitter and instagram apis, and to george washington university’s social feed manager (sfm) [13] twitter module and arc298’s instagram scraper [14]. this approach eliminated our frustrations with incomplete web crawls because we were no longer reproducing websites (we were now harvesting images and extensive descriptive and technical metadata directly), but it was not without its challenges. while sfm utilizes a relatively intuitive browser-based interface once installed, reaching that point, which requires not only deploying docker containers but also acquiring api credentials and adding them to the program, is not exactly straightforward for new users. these api credentials were essential to performing the work, and we were lucky to already have them (indeed, we requested an additional set from twitter when the project began and as of the date of publication the request remains unfulfilled). instagram scraper conversely requires no such credentials due to the more open nature of the instagram api. in this sense, it’s easier to begin using, but is more difficult in others (it must be installed via pip, and its operation is command-line only). the digital curation librarians installed the selected programs on their personal computers, since they were on leave when the stay-at-home order took effect, rendering their work-supplied laptops inaccessible. this was a less than ideal scenario, but options at that time were frankly limited. sfm was configured to harvest both media and metadata based on a twitter search for #archivethismomentdc once every 24 hours; instagram scraper was configured as a scheduled task where a command would be similarly executed every 24 hours, also harvesting media and metadata based on a search for the same hashtag. due to harvesting problems that were never fully reckoned with, sfm was only able to collect tweet datasets, but not the associated media. therefore, results of those harvests were retained in docker, exported as json once per week, and the twarc python library was utilized to collect any media missed by sfm. all metadata and media collected was backed up on a work-supplied external hard drive, and in a dc.gov google drive that was shared with all project team members to facilitate appraisal. appraisal and selection initially, our methods of selection were set out loosely in our call to participate (see appendix ii). we required submissions to be sent to us via certain channels, and we allowed all types of digital media. we then began to refine our selection criteria to just the geographic area of the district of columbia and offered tips on what type of content to submit. after receiving several submissions, we began to formulate a more solid selection criteria we called our first pass. (see appendix ii). during the collection period we received over 9,000 pieces of media. we began the selection process by splitting up the submissions amongst the group and used the criteria to come up with “yes” or “no” answers to submissions. the “maybe” submissions would be discussed by the group. in our spreadsheet of submissions, we instructed everyone to put in the notes field reasoning for a “maybe” designation. media were typically flagged “maybe” for content including: protests; faces of children; personally identifiable information like phone numbers, addresses, or health information; or where consent was in question. the group decided it was best to confirm consent with the creator before posting content with any of these issues. it is important to note that collecting from a social media hashtag leaves more of a chance that the donor does not fully understand or know about the extent of our project, whereas with the airtable form we created, there was a box that had to be checked to affirm consent from the donor. perhaps an instagram user saw their friend use the hashtag and just wanted to use it themselves, without realizing that it was part of a participatory archiving project? we regularly collect unknowingly from publicly available social media accounts in our regular web archiving collecting workflows, but we felt with this project we needed to ethically double check on consent with certain subject matter. by the beginning of june, people began using #archivethismomentdc to document protests and marches that swept through the entire nation in response to the murder of george floyd. since protests were not within our initial collecting scope of atmdc, we had to pause the project and reconsider. we began to discuss as a group if we would allow protest content to be included with our criteria–after all, this was what was happening in d.c. at the moment. initially, we discussed potentially including it with selection criteria ensuring that we did not collect any media with people’s faces, but even this didn’t seem to be enough. our reticence stemmed from two issues: the fact that we had specifically set out to collect submissions about life in the district under required social distancing; and to protect the privacy and safety of protesters and activists. we had difficult discussions within our department about the desire to collect materials in real-time to preserve this moment; versus the responsibility to respect the privacy and activism of those living through history. to help inform our thinking we turned to the work of the blacktivists [15] and witness [16]. we also consulted the society of american archivists’ code of ethics which states: “archivists recognize that privacy is an inherent fundamental right and sanctioned by law. they establish procedures and policies to protect the interests of the donors, individuals, groups, and organizations whose public and private lives and activities are documented in archival holdings” [17]. we take this very seriously at dcpl. what if activists uploaded to our hashtag, we archived the content, and then they deleted it from instagram, worried about their privacy, and they forgot that they submitted it using our hashtag? also, we received media of crowds that included many people’s faces (most were masked, but still, there are other identifying characteristics): while the person who created the media and submitted them to our hashtag is consenting to themselves being archived, there is no way we can get the consent from all the other people. it could be argued that the media was captured at a public event so there’s no need to eliminate these submissions, but it just didn’t feel like it fit within the code of ethics as we understood it. were we getting the most informed consent we could in this submission format and with the resources we had with a limited staff? we asked ourselves, what about embargoing access to the collection for 20 years as an answer to our issues? the initial goal for the project was to engage our community virtually immediately since we could not do so safely in person, so this went against the initial project’s aim. secondly, we felt strongly that we were limited in our ability to establish policies to protect the interests of our community, even and perhaps especially in the case of embargoed protest material. these are current events and activists who were arrested (or could still be) would be undergoing active trials–we have no precedence for collecting from active investigations and trials, and felt uncomfortable doing so. we also felt like opening up our selection criteria to include this content would be a massively time-consuming addition to our already extensive project for our small staff, and the fact that we still couldn’t come up with answers where we all felt ethically comfortable left us with our decision. ultimately we chose to focus on the stay-at-home order, which lasted from march 16 to may 28, and not include protest content. we felt this kept the project to a manageable amount of work for our small department and decided to move some focus to supporting local activists not through collecting their content, but through doing what libraries do best: sharing information — specifically on personal archiving. siobhan organized and held a memory lab network webinar entitled, “supporting community archivists: preserving and sharing content ethically” in august with yvonne ng from witness and representatives from the xfr collective [18]. siobhan and laura farley gave a free one hour “digital preservation 101” virtual class in october to the public. we also regularly shared personal archiving resources on social media and our electronic newsletter, the intelligencer. by helping activists to preserve their personal archives safely, these records that document this historical moment will live on within the community. perhaps we will then collect this media in 20 years for our archive, perhaps not. to best archive this moment in d.c., we feel that it wasn’t about collecting activists’ content at all: it was about sharing information with our community to assist them in preserving themselves. due to the large number of submissions and the unfamiliarity with the workflow, we decided to chunk the submissions into phases. the first phase began with instagram hashtag submissions only and that had been selected as answering “absolute yes” to all of the questions in our selection criteria. we did not include twitter submissions using our hashtag due to the low number of entries. metadata once we finished selecting which objects to include in the collection, we needed to describe them in order to provide context and make them accessible to users. dc public library utilizes the dublin core schema to describe objects. collecting content via social media presented certain opportunities and challenges in terms of metadata. in addition to the usual fields such as title, date, and description, we wanted to be able to capture information unique to this type of project, such as hashtags and original captions. the first step in completing metadata for the objects to be included in the online collection was to transform the json file into a csv file for tracking and reference. after mixed results with using microsoft excel and python to open, parse, and write json files into csv files, the “convert json to csv” tool provided by data designs group, inc. was utilized to convert the harvested data [19]. preservation copies of the json files were retained. the real challenge began with divvying up objects between staff for metadata creation. multiple issues made what seemed at the onset to be a simple task of dividing work into a massively time intensive project. first, the file names generated by instagram were not sequential, meaning that posts that included multiple objects would not automatically group together. this complication with the files names was not immediately obvious and as a result we began the selection process, without the json file, one file at a time in sequential order. some objects alone were not worth keeping individually, but were worth keeping as part of a larger post when in context with its original media. in hindsight, consulting the metadata during file selection would have been best; however due to staffing time constraints and a desire to keep the project moving forward we learned this lesson the hard way. ultimately, once the json file was converted into a master spreadsheet, it was heavily consulted in the selection process. additionally, a column was added to each cataloger’s metadata spreadsheets indicating if an object was part of a set. this labor intensive process required sorting the master metadata spreadsheet alphabetically by the “original caption” field to find posts with multiple objects. this process was not foolproof, as many posts had identical or similar “original captions.” “user id” and “original date” fields were also checked to ensure objects belonged together. a second challenge with the json metadata for instagram was that only the user id for a post was provided, not the handle. this makes sense because the user id remains the same if a user changes their handle; however, we wanted to display the handle in the metadata in addition to the user id because people are more comfortable searching for and referencing handles than user ids. it is possible to harvest a json file through the instagram api with handles and user ids, however this process requires an access token granted through instagram and although both the digital curation librarian and the digital initiative coordinator applied, neither was granted one. as of publication they are, in fact, both still waiting for access tokens. a simple, though time consuming, solution was to enter each user id into the comment picker “find instagram username” search [20] and record the handle in a new column on the master spreadsheet. a third challenge with the json metadata for instagram was that the timestamp was recorded in unix time, a format that is not human readable. a date converter formula was applied to the column containing the unix time column before splitting off the hours:minutes:seconds portion of the timestamp and removing it from the spreadsheet. the result was a column of dates in iso 8601 date and time format, yyyy-mm-dd (see appendix iii ). [21] after resolving these challenges with the json metadata, each cataloger received a google sheet to record object metadata including: original date, original caption, hashtags, user id, handle, and if the object was part of a set. catalogers manually completed metadata for title, subject, neighborhood, address, and description for each object. the library of congress thesaurus for graphic materials [22] served as our source for subject headings of images and the library of congress subject headings [23] for original captions. a local controlled vocabulary was utilized for d.c. neighborhoods and address information, if known. this project also prompted a staff discussion on the use of gendered pronouns in description fields. to avoid making assumptions about members of the public, we ultimately decided to utilize gender-neutral pronouns for atmdc, unless the subject’s gender was otherwise known or documented. (as an aside, we plan to continue this policy for all digital projects going forward.) access atmdc is available in dig dc, dc public library’s home for digital special collections, partially funded through the institute of museum and library services and supported by the washington research library consortium. dig dc utilizes the open-source software islandora, which provides a framework for digital asset management and access. islandora is flexible in objects that can be ingested and highly customizable, built upon a base of drupal, an open-source web content management framework; fedora commons, the underlying structure to a digital repository; and solr, an open-source search platform. when combined these platforms and frameworks provide highly customizable, searchable, and ordered access to digital objects and some long-term preservation protections. as previously discussed, metadata for dig dc is created using the dublin core schema. after staff completes metadata for each individual digital file, dublin core is migrated into mods to transform the metadata into a machine-readable xml schema. these metadata files are ingested together with digital files to create a digital object within a collection. dig dc is made up of objects, which are digital files with accompanying metadata within a collection. drupal offers modular islandora solution packs that standardize the ingestion of objects and metadata for given formats such as newspapers, documents, or images. these solution packs may be used individually or combined within a collection depending on the format of digital files. all solution packs require a digital object and a mods record. a high resolution digital file is ingested into a solution pack which then generates an access copy to speed up front end access and, if permissions are set to allow it, allows the user to download a copy of the object. [24] these objects may be entire archival collections selected for digitization, portions of archival collections selected for digitization, or born digital materials. in dig dc, a collection is a grouping of objects that all come from the same archival collection; or a grouping of objects that come from different archival collections but share a common theme. all collections include brief description essays to provide context to the grouping of objects. when considering how to build the atmdc collection in dig dc, the structure of the original social media posts were recreated as closely as possible. to achieve this grouping the compound object solution pack was utilized, which functions as a mini-collection within a larger collection, meaning objects of different formats may be combined within a compound object to create a package of objects. for example, an instagram post with four images with the hashtag appears in dig dc with all four images in one container, not as listed as individual objects (see appendix iv). an additional consideration was how to best display captions, hashtags, and emojis from social media posts with the limitations of mods. normally this information would be included in a “notes” field in the mods records, however the special characters and emojis could not be preserved in that format. instead every caption was saved as a separate pdf document, regardless of whether the caption contained special characters and emoji or not. these pdf documents were then added to the compound object (see appendix v). the resulting compound object, while not as user friendly as the original social media post, pairs the visual component of the original post with the textual component. there are no access restrictions and few use restrictions on the atmdc collection in dig dc. every collection and object in dig dc includes a rights statement provided by rightsstatments.org to guide users on copyright and reuse guidelines. all objects in the atmdc collection are in copyright, with the creator retaining rights. for objects submitted through instagram the rights holder is displayed as the username. users are able to download content from the collection but it is their responsibility to assess proper use of the objects. conclusion and next steps participatory collecting requires strong relationships with members of the community. we were able to engage some quadrants of the district more than others, despite efforts to reach out to community organizations and leaders. there is no catch-all solution to collecting social media materials from the internet. web archiving has its advantages, as does harvesting via api, and archivists must be adept at iterating their approach when results do not meet expectations. selection is a process that requires always expecting the unexpected. you can try to plan for it, but then you need to be ready to adapt to circumstances. just because you can collect something, doesn’t mean you should. people’s safety in the present is more important than preserving material for the future. when it comes to creating metadata for objects harvested from social media, what seems like a straight-forward task may become complicated and time consuming due to unforeseen data formats. our team learned the hard way to consult metadata in-tandem with object selection to ensure we were getting the entire context of an object. in providing access to the atmcd collection in dig dc, social media posts were recreated as faithfully as possible and original captions were saved as pdfs to capture special characters and emojis. while collecting for atmdc may have concluded, our work on this project is ongoing. short-term steps include completing metadata on selected instagram objects and ingesting them into dig dc; and beginning the selection process of objects submitted through airtable and email. mid-term steps include revisiting the “maybe” media from instagram and confirming consent with creators when minors are involved; and continued outreach on personal digital preservation and awareness-raising social media campaigns on atmdc. long-term steps include programming tailored around drawing connections between current collections and personal collecting. bibliography [1] dc history center in real time: collecting current events webpage: http://www.dchistory.org/in-real-time/ [2] arlington public library quaranzine webpage: https://library.arlingtonva.us/2020/03/30/quaranzine-call-for-submissions/ [3] archive this moment d.c. airtable form: https://airtable.com/shrawingyo42fnygs [4] dc public library archive this moment d.c. website: https://www.dclibrary.org/archivethismomentdc [5] archive this moment d.c. announcement. beyond words [internet].[cited 2020 jan 08]. available from: https://us2.campaign-archive.com/?u=ced1a87cc829e2107d5bc2dfc&id=61ecb4dbcc [6] archive this moment d.c. announcement. the intelligencer [internet]. [cited 2020 jan 08]. available from: https://preview.mailerlite.com/i3n5q2 [7] cartagena, rosa. 2020. help dc public library “archive this moment” with your zoom screenshots, quarantine diaries, and more. the washingtonian [internet]. [cited 2020 jan 08]. available from: https://www.washingtonian.com/2020/04/16/dc-public-library-archive-this-moment-zoom-screenshots-quarantine-diaries/ [8] maryland center for history & culture, collecting in crisis: responsive collecting in a digital age recording: https://vimeo.com/420284489 [9] archive this moment d.c. collection in dig dc: https://digdc.dclibrary.org/islandora/object/dcplislandora%3a259406 [10] blitz, matt. 2020. a new dcpl collection shows what d.c.’s first two months of covid-19 lockdown looked like. dcist [internet]. [cited 2020 jan 08]. available from: https://dcist.com/story/20/09/30/dc-public-library-coronavirus-pandemic-archive-this-moment/ [11] byck, daniella. 2020. do these photos perfectly capture life during covid?. the washingtonian [internet]. [cited 2020 jan 08]. available from: https://www.washingtonian.com/2020/09/25/archive-this-moment-dc-public-library-photos-capture-life-during-covid-19/ [12] blitz, matt. 2020 d.c. public library premieres photo exhibit showcasing first two months of the pandemic. wamu [internet]. [cited 2020 jan 08]. available from: https://www.npr.org/local/305/2020/10/01/919072121/d-c-public-library-premieres-photo-exhibit-showcasing-early-months-of-the-pandemic [13] george washington university social feed manager (sfm): https://gwu-libraries.github.io/sfm-ui/ [14] arc298 instagram scraper: https://github.com/arc298/instagram-scraper [15] the blackivists documenting protests resources: https://www.theblackivists.com [16] witness activist video archiving resources: https://archiving.witness.org [17] saa core values statement and code of ethics [internet]. [updated 2020 aug 06] .society of american archivists; [cited 2020 dec 11]. available from: https://www2.archivists.org/statements/saa-core-values-statement-and-code-of-ethics [18] memory lab network, supporting community archivists: preserving and sharing content ethically webinar: https://www.youtube.com/watch?v=_nx1ci-dszc&t=81s [19] data design group, inc. convert json to csv tool: http://convertcsv.com/json-to-csv.htm [20] comment picker find instagram username tool: https://commentpicker.com/instagram-username.php [21] iso 8601 date and time format [internet]. [updated 2020 mar 11]. iso – international organization for standards, popular standards; [cited 2020 dec 08]. available from: https://www.iso.org/iso-8601-date-and-time-format.html [22] library of congress thesaurus for graphic materials: https://id.loc.gov/vocabulary/graphicmaterials.html [23] library of congress subject headings: https://id.loc.gov/authorities/subjects.html [24] islandora solution packs [internet]. [updated 2017 jul 13] .lyrasis wiki; [cited 2020 dec 08]. available from: https://wiki.lyrasis.org/display/islandora/solution+packs#:~:text=islandora%20solution%20packs%20are%20drupal,as%20books%20or%20audio%20files appendix i press release life in washington during covid-19 captured in library’s “archive this moment d.c.” collection 88 photos submitted by district residents in dc public library’s first release of collection showing daily activities and sights during city’s stay-at-home order (washington, d.c.) – as the district shut down for the covid-19 stay-at-home order in april, the marquee on the anthem music venue says, ‘we’ll get thru this.” at the world central kitchen in the u street corridor, dreaming out loud program alumni donned face masks while preparing meals for seniors and homebound residents. these are two of the 88 photos in the initial release of dc public library’s “archive this moment d.c.,” a collection created by washingtonians capturing what people were thinking, feeling and doing during the early part of the covid-19 pandemic. the photos were submitted by people posting on instagram using the hashtag #archivethismomentdc. from march 25 to may 28, the library collected more than 2,000 images from social media, through email and a web form. the majority of the items were collected in april, followed by may and march. telling stories of covid-19 from the viewpoint of the people who lived through it, makes the archive this moment d.c. collection an invaluable resource for the district, said richard reyes-gavilan, executive director of the dc public library. “some of the most important accounts of history are first-person. this collection will help future researchers understand how residents reacted to the pandemic, and how quickly it changed the way we all lived.” the collection includes selfies and portraits of people in masks; handmade signs of support of essential works and the community; people navigating the city during the pandemic; and signs posted in business windows about closures, restrictions, and new business protocols. when complete, the “archive this moment” collection will consist of images, audio recordings, videos, and text. all four quadrants of the district are represented. this initial release includes images from the atlas district, barney circle, bloomingdale, capitol hill, carver langston, cleveland park, downtown, dupont circle, eckington, georgetown, kingman park, logan circle, manor park, mount pleasant, southwest, u street corridor, west end and woodley park. since the library began collecting images, librarians have been working behind the scenes creating complete records of the materials. every item added to the collection receives a unique identifier, title, description, and a library of congress subject heading so that people can find the items easily. all original captions from instagram and twitter are saved as pdf files to preserve emojis and other special characters. as more items are cataloged, the collection will increase. the archive this moment d.c. collection is housed on dig dc, the library’s web portal for digitized and born-digital special collections items. to view the collection, visit https://digdc.dclibrary.org/. ### appendix ii first pass selection criteria if you answer no to any of these questions about the submissions, the submission does not pass the first pass of selection criteria: was it created within the official boundaries of washington, d.c. and/or does it document the people, places, and ways of life within the official boundaries of washington, d.c.? was it created after march 1, 2020? was it created before may 28, 2020? was it created by the submitter? if submitted through social media, was it posted by a public account? absolute yes criteria (apply to content after asking the above questions) tells the story of daily life in the district during this period of social distancing obviously documents what has changed – recently posted rules on social distancing, face masks, empty streets, related street art no: protests, faces of kids, or private info like phone numbers, addresses, doctor’s names, or other personal or health information maybe: anything you feel like should be looked over by the group: put in the notes field why you weren’t certain protests, faces of kids, or private info like phone numbers, addresses, doctor’s names, or other personal or health information contact creator to inquire if they would like this to be added to our collection if they answer “no” or we do not receive a response, these will not be selected 2nd pass – final selection criteria no to: protest or activist content; any personally identifying information date created march 25-may 28, 2020 appendix iii steps used to instagram convert unix time metadata into iso 8601 in google sheets add a new date field column to the master spreadsheet apply the formal =((a1-21600000)/86400000)+25569 to the appropriate column to convert the timestamp from unix time to iso 8601 change the format of the new date column using format>number>date time remove the hours:minutes:seconds portion of the timestamp using data>split text to columns>separator: space and delete the extra split column. appendix iv figure 1. example of archive this moment digital object with files grouped together to emulate the original instagram post. https://digdc.dclibrary.org/islandora/object/dcplislandora%3a267598< appendix v figure 2. example of archive this moment d.c. digital object with original instagram caption captured as a pdf. https://digdc.dclibrary.org/islandora/object/dcplislandora%3a267888 figure 3. example of archive this moment d.c. digital object with original instagram caption captured as a pdf. https://digdc.dclibrary.org/islandora/object/dcplislandora%3a267888 appendix vi figure 4. archive this moment d.c. project workflow. about the authors julie burns (julie.burns@dc.gov) is a library associate for the people’s archive at dc public library. she holds an m.a.t. in museum education from the george washington university and a b.a. in english from the university of mary washington. laura farley (laura.farley@dc.gov) is the digital curation librarian for the people’s archive at the dc public library. she holds her m.a. in library and information studies from the university of wisconsin-madison and a b.a. in history from the university of iowa. laura’s passion is shining a light on our shared histories through accessibility and outreach. siobhan c. hagan (siobhan.hagan@dc.gov) is the project manager of the dc public library memory lab network. siobhan is also the founder & ceo of the mid-atlantic regional moving image archive (marmia) and holds her m.a. in moving image archiving and preservation from nyu’s tisch school of the arts. paul kelly (paul.kelly2@dc.gov) is digital initiatives coordinator for people’s archive at dc public library. he holds his m.s. in library and information science from the catholic university of america and m.a. in english literature/film and television studies from the university of glasgow. his professional interests include collections as data, web archiving, and digital preservation. lisa warwick (lisa.warwick@dc.gov) is the reference coordinator for the people’s archive at dc public library. she holds her m.a. in information science from the university of maryland and a b.a. in film studies from the university of pittsburgh. her passion is using history to connect individuals to community. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – experimenting with a machine generated annotations pipeline mission editorial committee process and structure code4lib issue 48, 2020-05-11 experimenting with a machine generated annotations pipeline the ucla library reorganized its software developers into focused subteams with one, the labs team, dedicated to conducting experiments. in this article we describe our first attempt at conducting a software development experiment, in which we attempted to improve our digital library’s search results with metadata from cloud-based image tagging services. we explore the findings and discuss the lessons learned from our first attempt at running an experiment. by joshua gomez, kristian allen, mark matney, tinuola awopetu, and sharon shafer i. introduction the labs team in the fall of 2018, the ucla library began an effort to modernize our software development processes by adopting a devops culture. see (gomez, 2020) for a discussion of that process. to that end, a focused subteam was formed with the sole purpose of conducting experiments. this team was named “the labs team” and this paper describes our first attempt at running a software development experiment. we expect that many, if not all, of the experiments that this team conducts will somehow be related to the application of artificial intelligence and machine learning (ai/ml) technologies to library resources and workflows. however, as developer time is a precious resource of its own, the experiments we choose to conduct should be of strategic value to the larger team and to the library. thus, we will also attempt to devise experiments that can guide the wider software development team when making decisions about future development opportunities, architectural designs, and tool selections. ii. our first experiment – mgap overview pre-trained machine learning models have become more widely available for commercial use over the past decade. regarding models specifically for computer vision, many of the big players in the tech industry now offer services that automatically tag images with descriptive labels. we wanted to know whether it would be worthwhile to apply these tags to our collections in the new digital collections repository. once we stated this as our question, we were faced with another one: how would we measure improvement of metadata? we are a group of software developers, so we are familiar with user testing as a means of evaluating systems. we assert that if the users show increased satisfaction, then the system has been improved. the machine learning models underlying these commercial tagging services are trained with large corpuses of generic images. while large training sets generally improve the accuracy of a model, the scope and quality of the training data also affects accuracy and appropriateness for different use cases. thus, we suspected that the tags we would get would be too generic for scholarly research purposes but could potentially be useful to casual users looking for interesting images. question – can commercial image tagging services improve the digital library’s metadata? assertion – we will evaluate improvement via user testing. hypothesis – the tags will be of no use to scholars conducting research on specific topics but will be more useful to casual users looking for interesting images. strategic motivations in addition to answering the stated question, we had a few other motivations for conducting this experiment. these were all related to trying out tools and methods that could potentially be of use to the other development subteams, thus making this a strategic experiment. another outcome of the reorganization of the software development team was the creation of a user experience research & design (ux) team. this team developed a strategic plan (gomez 2019) to improve our ux maturity over the next three years. this experiment offered the opportunity for us to gain more experience conducting usability tests and to try out an unmoderated user testing tool: usertesting.com. our new digital library makes use of iiif [1] protocols for image delivery. we designed the technical setup of this experiment in such a way as to make use of our new iiif services and to experiment with using web annotations [2], a sort of sister protocol to iiif, to apply the tags to the images. one of our higher level goals was to move towards evolutionary architectures, therefore we also designed the experiment in such a way as to use event-driven systems to coordinate the tasks of harvesting, tagging, annotating, and indexing. iii. setup context in early 2018, prior to the new leadership and reorganization of the software development team, the ucla library began development on a new digital library repository and public interface [3a] by adopting the open source samvera platform. the following year, with the new leadership and reorganization came our commitment to move towards a more service-oriented architecture and to reduce the responsibilities of monoliths. as a first step, our services subteam implemented a suite of services to support image transformation, presentation, and delivery, with the two latter services conforming to iiif protocols. iiif is a suite of api specifications that facilitate shared usage of image-based collections across multiple institutions. at the time in which the mgap experiment was conducted (spring and summer 2019) only three collections had been migrated to the new repository, which was still in alpha mode and would not be publicly released until december of 2019. these three collections were: the la daily news negatives (ladnn) – 5,172 images [3b] the will connell papers (connell) – 502 images [3c] walter e. bennett photographic collection (bennett) – 79 images [3d] all three of these were journalistic photograph collections dating from the 1920’s to the 1960’s. each of these were “simple” image collections, meaning that they did not contain paginated materials or other media formats. the simplicity of these collections was certainly an advantage in running the experiment. however, the small size of the data set would prove to be a limitation. these collections were already described by staff in the library’s cataloging and metadata unit. tools the following tools, platforms, and services were used in this experiment. search interface & index – blacklight & solr [4] iiif image server – cantaloupe [5] annotation server – elucidate [6] application containerization – docker [7] tasks and queue – python (celery) [8] + rabbitmq [9] image tagging – aws rekognition [10], clarafai [11], google vision [12], microsoft azure cv* [13] unmoderated user testing – usertesting.com [14] *note: midway through the experiment we reached the limit of azure’s free tier, and therefore dropped its tags from the experiment. architecture the diagram below illustrates the architecture of the system and the flow of data within it. we began by taking the existing solr index of the new repository and replicating it five times: three to be modified by individual tagging services, one to be modified by the combination of all of three tagging services, and one to be left untouched as a control. each of these indexes also had its own copy of the digital library frontend (blacklight). we wrapped these five instances of the digital library in docker containers to make them easier to deploy and tear down on both local developer machines as well as in our cloud hosting environment (aws). once the infrastructure was set up the essential process went as follows: the images were harvested from the image server and put into a job queue. as those jobs were executed, images were sent to each of the three image tagging services. the returned tags were transformed into annotations and posted to the annotation server. the tags/annotations were then used to update each image’s document in the solr index. note: the modified indexes still contained the library’s existing metadata. this test was not a direct comparison between the library’s metadata and the tags. the purpose of the test was to determine if the tags could enhance the existing metadata, not to determine if they could replace it. figure 1. architecture diagram iv. preliminary analysis prior to user testing we did some cursory analysis of the tags and found some interesting differences between the three services. tag quantities on average, google vision applied a smaller number of tags to each image and had a narrower range, whereas aws rekognition had a much wider range and a higher average. this reveals a potential difference in the confidence thresholds that determine whether to apply a given tag or not, with google having a more conservative default setting (i.e. a higher threshold). the default threshold for aws was 55%, whereas google did not specify the threshold. clarafai applied exactly 20 tags to every image, regardless of confidence level, which is the default if you do not specify a minimum value for the confidence threshold or a maximum number of concepts. tag count per image (average) amazon clarafai google bennett 18.6 20 4.8 connell 12.8 20 2.8 ladnn 16.3 20 4.3   tag count per image (min & max) amazon clarafai google bennett 5 34 20 20 2 9 connell 2 36 20 20 1 14 ladnn 1 47 20 20 1 16 tag qualities to get a sense of what kinds of tags were actually being applied, we took a look at the top 10 tags that each service applied to the la daily news negatives collection. as illustrated in the charts below, the most common tags from each service were very generic terms. one might be inclined to infer our hypothesis (that the tags would only be useful to casual users and not to scholars) was on strong ground after seeing these generic terms. that would be a bad inference. it makes sense that the top 10 most common terms would be the most generic terms. what is striking, however, in these results is the precipitous decline in term frequency from google compared to the other two. this is in line with the tag counts discussed above, which suggest that google’s service is more conservative. figure 2. “ladnn” according to amazon rekognition: 10 most common tags figure 3. “ladnn” according to clarifai: 10 most common tags figure 4. “ladnn” according to google vision: 10 most common tags v. moderated user testing once the images had all been tagged and the indexes updated, we conducted user tests to determine if the changes in search results proved to be of greater utility to the end users than the unmodified digital library’s search results. methods our user testing began with in-person, think-aloud protocol tests moderated by members of the ux team. we conducted these tests in early june, which unfortunately coincided with finals week. we were unable to recruit as many students as we had hoped for and were forced to rely mostly on library staff. in total, we conducted nine in-person tests. in these tests, the user was presented with the control interface and one of the four modified interfaces. we did not want to overwhelm them by having each user look at all five instances. due to the limited size and scope of the three collections, we created a series of three tests that initially introduced the users to the kinds of materials in the collection with search terms we provided, and then incrementally gave them more freedom to perform their own searches. test 1: provided search terms the participants were given exact terms (e.g. “sunglasses”, “theater”, “chinatown”, “cat”) to enter into the search bars of both user interfaces. we asked them to rate the relevancy of both sets of search results independently (i.e. we did not ask them to compare or rank the two results). test 2: provided scenario the participants were presented with a scenario in which they were costume designers working on a historical film set in 1940’s los angeles. they were asked to look for inspiring images by using their own search terms. again, we asked them to rate the relevancy of both sets of search results independently. test 3: free form search at this point, we hoped the participants had gotten a feel for what kinds of materials were in the collections. we therefore asked them to perform searches of their own and asked them to rate relevancy of both sets of results independently. results in this round of testing, we encountered many limitations to the experiment. as mentioned above, the reliance on library staff could lead to skewed results as they are not representative of the typical user base and may be biased due to their training and expertise. more importantly, the image set was just not very large (less than 6000 images), and the images were limited to a narrow scope of three historical collections. the limited collection of resources made evaluation of the relevance of the search results very difficult. most users, including the library staff, just didn’t know what to look for since they were not familiar with the collections. we tried to mitigate that problem by graduating from spoon-fed terms to free-form searches. the hope was that we would expose the users to what the collections were about in the first two tasks so that the free-form test would feel a little more natural to the participants. whether we achieved that goal is debatable. perhaps the biggest problem was in the actual test itself. in our test we asked users to rate the relevance of each result set independently. our goal was to compile the results and determine which service scored highest on average. unfortunately, the ratings across the interfaces were not significantly different. a better approach may have been to just ask the users to compare the two sets of search results and tell us which they thought performed better. some other challenges we faced included recruiting, as mentioned above. but even when we found willing participants, scheduling was also an impediment. finding time slots during which both a team member and a participant were both available proved quite challenging. based on these limitations, we decided to revise our methods and perform a second round of user testing, this time all online. vi. unmoderated user testing methods as mentioned above, we recognized several limitations of our first round of tests. we could not address them all, in particular we could not change the collections, as only three had been migrated into the new digital library platform. but there were two areas in which we could improve: the participant pool and a forced ranking of the search results instead of independent scoring. this second round of testing was conducted entirely online using the saas testing platform usertesting.com, thanks to a free trial they offered us. this service provided three key benefits. first and foremost, they have a panel of 1.5 million users they pay to participate in usability tests. this meant no more recruiting woes for us. second, the tests are completely unmoderated. we filled out a form with our test instructions and questions. the service then overlaid our site with those instructions and questions as the users progressed through the tests. this freed us from the other large burden of scheduling. finally, the service recorded the participants’ screens and audio responses for us to analyze later. we conducted 21 unmoderated sessions. the first was a dry-run that helped us refine the instructions and questions. we then conducted the remaining 20 sessions with five participants for each of the four modified indexes. tests 1, 2, and 3 revised we performed the same three tests as before: provided search terms, provided scenario, and free form search. the key difference was that instead of asking the users to rate the results independently, we asked them to compare the two search result sets (the control set and one modified set) and determine which was better. they also had options to state that neither set was better than the other or that neither set was relevant at all. test 4: evaluate a single image’s tags we added one extra test in the second round. we asked the participants to look at the machine generated tags for a single image in all four modified interfaces. we then asked the participants to rate (on scales from 1 to 5) the accuracy and the usefulness of the tags for each set independently (i.e. not to rank or compare them). this was the only task in which the participants were exposed to all four sets of modified tags. results test 1 the charts below illustrate the users’ preferences of the search results when asked to search for specific words (“sunglasses” and “theater”). in both cases the search result set from the control (i.e. the library’s existing metadata) was rated higher than or equal to the set modified with tags from aws rekognition by every user, whereas the sets modified with google vision were voted higher than the control sets by some users. the users determined that the sets modified with tags from clarafai were neither better or worse, while the set combining tags from all three services was rated worse than the control in very similar proportions as aws. figure 5. task 1a – “sunglasses” comparisions figure 6. task 1b – “theater” comparisions test 2 in the wardrobe designer scenario, we saw a slight variation of the results from test 1. once again, the results modified with tags from google vision received votes ranking it higher than the control set, and once again aws received no votes. clarafai received as many votes as google vision. interestingly, the control received almost no votes (just one, when rated against aws). the majority of users across all four groups found the result sets to be equal. figure 7. task 2 – wardrobe designer test 3 when performing free-form searches, the users never rated the control better than the modified search results sets. clarafai received a pair of votes, and google vision one, while aws still did not receive a single vote. what is interesting here is the significant increase in the number of users who voted that neither the control or the modified sets were relevant. figure 8. task 3 – free form search test 4 in this test we asked the users to look at the tags from all four sets for the image below of lyndon johnson riding a horse. the tags from google vision received the highest average votes for both accuracy and usefulness. aws received the lowest average for accuracy and clarafai received the lowest votes for usefulness. figure 9. task 4 – jlb riding a horse figure 10. photo: jlb riding a horse discussion across all four tests, google vision performed the best, while aws rekognition performed the worst, including the direct comparison in test 4. this is likely a result of google’s more conservative application of tags which we observed in the preliminary analysis. the tags from aws may be more “noisy” and therefore less accurate and useful, leading to poorer search results as determined by end users. in most of the tests the combined set of tags performed nearly as poorly as aws. therefore, based on just the results presented here, if we decide to incorporate machine generated tags into our digital library’s metadata, it appears that we should use a single image tagging service and that google vision would be the best choice to provide that service. however, it is not clear from these results that incorporating these extra tags would be worthwhile. across tests 1 and 2, comparing the library’s existing metadata to the enhanced sets, there were many more votes for the control or for “equal” than there were for any of the tagged sets. this is fairly in line with our hypothesis that the tags would not be useful to scholars looking for something specific in the digital library. the second half of our hypothesis stated that the tags would be beneficial to casual users looking for interesting images. therefore, we expected to see the tagged sets perform better in test 3 (free-form search). however, the results show that the users found neither the control or the tagged result sets relevant. thus, the machine generated tags do not appear to provide much benefit to casual users either. other factors to consider when deciding to incorporate machine generated tags include cost and level of effort. the prices for these services are all very reasonable — on the order of a few dollars per 1,000 images. we currently host around 3 million images across all of our legacy digital library platforms. as we migrate these collections to the new repository over the next year, the cost to tag them all would only cost a few thousand dollars. the level of effort to operationalize this process is more significant than the cost of the service, but is still not prohibitive. our event-handling infrastructure performed well for our needs and would likely be sufficient for the mass migration of content (i.e. no need for enterprise level streaming data solutions like kafka or airflow). but operationalizing this setup would incur a fair amount of developer time, especially as the knowledge would need to be transferred from the labs team to the services team, which is responsible for building and maintaining our backend services. so, while the total cost of adoption is reasonably low, it still outweighs the meagre potential benefits. we therefore decided not to prioritize adoption of machine generated tags in our 3-year software development roadmap, though we will likely revisit this topic again in the future. limitations one limitation in our study was the variation in settings and scope across the cv models we used. we did not ensure that each service conformed to the same number of returned labels or, more importantly, to the same confidence threshold. without this calibration, our findings are bound to be skewed in favor of the one that is most conservative by default, as the others will have more “noise.” furthermore, the google service applied labels only using object detection, which is concrete, whereas clarafai and amazon also apply more abstract labels for things like moods, concepts, and themes. a significant limitation of this experiment was the framing of the question and the method of measurement. relevance is a difficult attribute to measure. without a precise definition, any determination of relevance will be subjective. scholars in the field of information retrieval address this issue with the construction of a corpus for events like trec (text retrieval conference) [15], in which the relevant results are known ahead of time, and the search index is evaluated on its ability to return all of those results and no others, with no recourse to end user judgement of the actual result sets. we wanted to test the tags on our own collections, not on a generic, standard data set like trec. but creating our own corpus was too high a barrier for our less formal experiment. to avoid that high barrier we chose instead to embrace the inherent subjectivity in determining relevance by employing usability testing as our method of measurement. this seemed appropriate as our goal was to determine if the tags would help our users, not to determine their technical accuracy and precision. unfortunately, due to timing we were unable to conduct the usability tests on our actual users. we relied on staff for in-person testing and completely unaffiliated users for the unmoderated testing. therefore, our results tell us nothing about our actual users. regardless of the source of participants, our results still suffered from small numbers of both the users and the images available in the user interface. even if all test results indicated our hypothesis was correct, the small sample sizes would prevent confirming the hypothesis with confidence. this is an inherent attribute of usability testing, which is by its nature a qualitative method. vii. conclusion lessons learned when conducting experiments, we have learned to make sure our question is not overly broad and that all the concepts are well defined. however, we should add that we do not intend to tighten our focus as narrowly as a true scientist might, as we intend our experiments to produce immediately actionable insights, not minute increments of factual knowledge. we have also learned to make sure that the instrument of measurement is appropriate. using a qualitative method to gather quantitative data is bound to be problematic in both execution and results. goals achieved and bonus findings despite the limitations and the negative results, the project was highly valuable for our team. we achieved all of our strategically motivated goals, including gaining experience with tools for web annotations, event-driven systems, and unmoderated user testing. perhaps the greatest benefit of the experiment was the discovery, during all those tests with users, of several usability issues with the actual interface. these discoveries were shared with the applications team, which was tasked with building out the new digital repository’s public interface, and they were all addressed before the december 2019 public release. future work ai/ml while we have decided not to apply machine generated tags to our image collections, we may try using similar ail/ml tools on some of our video collections. as of october 2019, the ucla film and television archive is now under the wing of the ucla library. the archive is the second largest in the country, behind only the library of congress. the archive’s collections do not have the same level of descriptive metadata as the library digital image collections. in fact, many of the items have no descriptive metadata. thus, machine generated tags would make a vast improvement on the discoverability of the materials. we will begin exploring these options in the spring of 2020. usability testing as for unmoderated user testing, we intend to explore other service providers, such as loop11 [16], which offer similar features but with a “bring your own users” business model. this will put the burden of recruiting back on our shoulders, but it will still relieve us from the difficulty of scheduling, and the prices are significantly lower. iiif & web annotations to support the large amount of video coming our way from the film & television archive, the services team will begin upgrading our iiif services to conform to the new version of the iiif presentation api to support time-based media. we would like to continue working with web annotations and to enable our users to translate and transcribe materials, as well as to build their own virtual collections which they can annotate with notes and discussions. however, this is not currently on our 18-month development roadmap, which is likely to be dominated by a website redesign and cms migration, as well as the library’s migration to a new ils. bibliography ford n, parsons r, kua p. 2017. building evolutionary architectures: support constant change. sebastopol: o’reilly media. gomez j. 2020. modernizing a software development team [internet]. [modified 2020-05-02]. [cited 2020-05-02]. available from https://misc.dev/posts/modernizing-a-dev-team/ gomez j, awopetu t, prigge a, shafer s. 2019. ux strategic plan 2020 – 2022 [internet]. available from: https://ucla.box.com/s/mldgoduuf33hsr3ysgrahzczrgni7kuo kim g, debois p, willis j, humble j, allspaw j. 2016. the devops handbook : how to create world-class agility, reliability, and security in technology organizations. portland (or): it revolution press. rasmusson j. 2012. the agile samurai: how agile masters deliver great software. dallas (tx): the pragmatic bookshelf. notes [1] https://iiif.io [2] https://www.w3.org/annotation/ [3a] https://digital.library.ucla.edu/ [3b] https://digital.library.ucla.edu/catalog/8zn49200zz-89112 [3c] https://digital.library.ucla.edu/catalog/tpf2h200zz-89112 [3d] https://digital.library.ucla.edu/catalog/m8f11000zz-89112 [4] http://projectblacklight.org/ [5] https://cantaloupe-project.github.io/ [6] https://github.com/dlcs/elucidate-server [7] https://www.docker.com/ [8] http://www.celeryproject.org/ [9] https://www.rabbitmq.com/ [10] https://aws.amazon.com/rekognition/ [11] https://www.clarifai.com/ [12] https://cloud.google.com/vision/ [13] https://azure.microsoft.com/en-us/services/cognitive-services/computer-vision/ [14] https://www.usertesting.com/ [15] https://trec.nist.gov/ [16] https://www.loop11.com/ about the authors joshua gomez is the head of software development and library systems at the ucla library kristian allen is a software developer at the ucla library mark matney is a software developer at the ucla library tinuola awopetu is a ux developer at the ucla library sharon shafer is the search & assessment librarian at the ucla library subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – leveraging a custom python script to scrape subject headings for journals mission editorial committee process and structure code4lib issue 52, 2021-09-22 leveraging a custom python script to scrape subject headings for journals in our current library fiscal climate with yearly inflationary cost increases of 2-6+% for many journals and journal package subscriptions, it is imperative that libraries strive to make our budgets go further to expand our suite of resources. as a result, most academic libraries annually undertake some form of electronic journal review, employing factors such as cost per use to inform budgetary decisions. in this paper we detail some tech savvy processes we created to leverage a python script to automate journal subject heading generation within the oclc’s worldcat catalog, the mobius (a missouri library consortium) catalog, and the vufind library catalog, a now retired catalog for the carli (consortium for academic and research libraries in illinois). we also describe the rationale for the inception of this project, the methodology we utilized, the current limitations, and details of our future work in automating our annual analysis of journal subject headings by use of an oclc api. by shelly r. mcdavid, eric mcdavid, and neil e. das introduction in the past, renewal of journal and journal package subscriptions at lovejoy library at southern illinois university edwardsville (siue) had been piecemeal, with each decision referencing metrics specific to that purchase alone in isolation from others. the two principal factors considered were percentage price increase over the previous year and cost per use. if the percentage price increase could be kept below roughly 4-5% and cost per use per journal article kept below the rate of ordering articles through delivery services such as illiad and get it now, then the journal or journal package was renewed. regarding the process, the renewal price and statistical analysis were provided by the electronic resources librarian to individual subject librarians who made the final decisions on renewals. in this system, cross-disciplinary analysis of journal subscriptions occurred only to ensure that a resource was not being paid for more than once. there was no more granular analysis, especially analysis based on the subject content of journals. our project originated in the spring of 2019, with the goal of generating subject heading data for the journals we subscribed to at lovejoy library at siue. the desired outcome of the project was detailed analysis and education of library liaisons and campus faculty about the journal’s subscriptions in their disciplines, including usage data, cost per use, and potential alternate open educational resources (oer) or open access (oa) resources or less costly comparable product replacements. principal author shelly mcdavid had seen a similar model in action in her previous library and wanted to apply it to collection development and annual portfolio reviews of collections by lis for campus disciplines at siue as well. background at lis at siue, we had not been in the practice of identifying subject headings of electronic resources to which we subscribed, nor did we perform any type of discipline specific journal usage analysis. these were the goals of this project, with an eye to such analysis serving as the basis of annual portfolio reviews to present to campus faculty and administrators. in fall 2020, the lis lovejoy library collections analysis & strategy committee was formed to work toward these goals. subject scraper methodology the principal problem addressed in this project was to collect specific metadata for a group of issn numbers in an efficient, systematic way across different library catalogs. the solution was to create a python (v3.7.1) script which makes an https request to the catalog search engine for the issn number and parses the html search results to gather the relevant metadata for that issn. the script was designed to be used from the command prompt and allows for a single issn search or a batch search by passing in a csv file. one of the required positional arguments is a catalog name: worldcat, carli_i-share, or mobius. figure 1. output from running “python subjectscraper.py -h”. catalog parameters one feature of the script was to allow for flexibility in choosing the catalog for which the search is to be conducted, as only one catalog can be searched at a time. since each library web catalog is formatted in different ways with different html structures and css styles, a python dictionary data structure was created so the script could operate in a systematic way regardless which catalog was chosen. this dictionary was defined for each catalog and included the following parameters: base_url – the beginning part of the catalog search url, starting from https:// and ending before the first / e.g., “https://www.worldcat.org”. search_url – this is the rest of the catalog search url without the base_url, this is parameterized with a “{0}” for the issn to search for e.g., “/search?qt=worldcat_org_all&q={0}”. search_title – css selector for the parent html element that contains the issn title on the search results page. bib_record – css selector for the html element that contains the record metadata on the catalog item’s page. bib_title – css selector for parent html anchor element containing the item title on the catalog item’s page. bib_subjects – the html table element where the text begins with “topics” or “subject” in the catalog item’s page in context of bib_record. the script as written supported worldcat, mobius library catalog, and carli’s vufind library catalog, but more catalogs could also be searched by defining the above parameters for a specific library catalog. the parameters do require extensive knowledge of how html elements and css styles work and function. process regardless of whether the script is running in single or batch mode, the process of searching for an issn is the same and involves two url requests for a single issn: search the catalog for the issn and get the url for the catalog item. if the catalog item url is valid, search the catalog item’s page for relevant metadata. output or write to disk the metadata and search criteria. the script does not work for every issn and will give informative errors for any failure involved in the above steps. one complication when sending url requests to web servers is that there must be a time delay between requests, otherwise the server will assume that it is being spammed and deny the request. so, every url request is preceded by a time delay of half a second. output if the script is running in single issn mode, the output is sent to the screen. this output includes the issn that was searched for, the catalog item url, the item title, and the list of subjects that are in the record for the item. figure 2. output from running subjectscraper in single mode. in batch issn mode, a csv output file is created in the current working directory with the chosen catalog name appended to the file name i.e., “batch_output_worldcat.csv”. this output file will contain the same output from above but for all issns listed in the input file. some issns will produce errors when searching, and for those the url, title, or subjects may be blank. the script will also show a progress bar when in batch mode to inform the user about the approximate time to completion. external libraries the script uses the following external python libraries: urllib.request.urlopen – this is how the url request is made. bs4.beautifulsoup – this is an html parser that operates on the results from the url request. tqdm.tqdm – this is the progress bar when running in batch mode. source code – subjectscraper.py from urllib.request import urlopen from bs4 import beautifulsoup from tqdm import tqdm import re, argparse, sys, csv, time catalogs = { #'catalog name' : { # 'base_url' : beginning part of url from 'http://' to before first '/', # 'search_url' : url for online catalog search without base url including '/'; # make sure that '{0}' is in the proper place for the query of issn, # 'search_title' : css selector for parent element of anchor # containing the journal title on search results in html, # 'bib_record' : css selector for record metadata on catalog item's html page, # 'bib_title' : css selector for parent element of anchor containing the journal title, # 'bib_subjects' : html selector for specific table element where text begins with # "topics", "subject" in catalog item's html page in context of bib_record 'worldcat' : { 'base_url' : "https://www.worldcat.org", 'search_url' : "/search?qt=worldcat_org_all&q={0}", 'search_title' : ".result.details .name", 'bib_record' : "div#bibdata", 'bib_title' : "div#bibdata h1.title", 'bib_subjects' : "th" }, 'carli_i-share' : { 'base_url' : "https://vufind.carli.illinois.edu", 'search_url' : "/all/vf-sie/search/home?lookfor={0}&type=isn&start_over=0&submit=find&search=new", 'search_title' : ".result .resultitem", 'bib_record' : ".record table.citation", 'bib_title' : ".record h1", 'bib_subjects' : "th" }, 'mobius' : { 'base_url' : 'https://searchmobius.org', 'search_url' : "/iii/encore/search/c__s{0}%20__orightresult__u?lang=eng&suite=cobalt", 'search_title' : ".dpbibtitle .title", 'bib_record' : "table#bibinfodetails", 'bib_title' : "div#bibtitle", 'bib_subjects' : "td" } } # obtain the right parameters for specific catalog systems # input: catalog name: 'worldcat', 'carli i-share', 'mobius' # output: dictionary of catalog parameters def get_catalog_params(catalog_key): try: return catalogs[catalog_key] except: print('error unknown catalog %s' % catalog_key) # search catalog for item by issn # input: issn, catalog parameters # output: full url for catalog item def search_catalog (issn, p = catalogs['carli_i-share']): title_url = none # catalog url for searching by issn url = p['base_url'] + p['search_url'].format(issn) u = urlopen (url) try: html = u.read().decode('utf-8') finally: u.close() try: soup = beautifulsoup (html, features="html.parser") title = soup.select(p['search_title'])[0] title_url = title.find("a")['href'] except: print('error unable to search catalog by issn') return title_url return p['base_url'] + title_url # scrape catalog item url for metadata # input: full url, catalog parameters # output: dictionary of catalog item metadata, # including title and subjects def scrape_catalog_item(url, p = catalogs['carli_i-share']): result = {'title':none, 'subjects':none} u = urlopen (url) try: html = u.read().decode('utf-8') finally: u.close() try: soup = beautifulsoup (html, features="html.parser") # title try: title = soup.select_one(p['bib_title']).contents[0].strip() # save title to result dictionary result["title"] = title except: print('error unable to scrape title from url') # subjects try: record = soup.select_one(p['bib_record']) subject = record.find_all(p['bib_subjects'], string=re.compile("(subjects*|topics*)"))[0] subject_header_row = subject.parent subject_anchors = subject_header_row.find_all("a") subjects = [] for anchor in subject_anchors: subjects.append(anchor.string.strip()) # save subjects to result dictionary result["subjects"] = subjects except: print('error unable to scrape subjects from url') except: print('error unable to scrape url') return result # search for catalog item and process metadata from item's html page # input: issn, catalog paramters # output: dictionary of values: issn, catalog url, title, subjects def get_issn_data(issn, p = catalogs['carli_i-share']): results = {'issn':issn, 'url':none, 'title':none, 'subjects':none} time.sleep(time_delay) url = search_catalog(issn, params) results['url'] = url if url: # only parse metadata for valid url time.sleep(time_delay) item_data = scrape_catalog_item(url, params) results['title'] = item_data['title'] if item_data['subjects'] is not none: results['subjects'] = ','.join(item_data['subjects']).replace(', -', ' ') return results # main loop to parse all journals time_delay = 0.5 # time delay in seconds to prevent denial of service (dos) try: # setup arguments for command line args = sys.argv[1:] parser = argparse.argumentparser(description='scrape out metadata from online catalogs for an issn') parser.add_argument('catalog', type=str, choices=('worldcat', 'carli_i-share', 'mobius'), help='catalog name') parser.add_argument('-b', '--batch', nargs=1, metavar=('input csv'), help='run in batch mode processing multiple issns') parser.add_argument('-s', '--single', nargs=1, metavar=('issn'), help='run for single issn') args = parser.parse_args() params = get_catalog_params(args.catalog) # catalog parameters # single issn if args.single is not none: issn = args.single[0] r = get_issn_data(issn, params) print('issn: {0}\r\nurl: {1}\r\ntitle: {2}\r\nsubjects: {3}'.format(r['issn'], r['url'], r['title'], r['subjects'])) # multiple issns elif args.batch is not none: input_filename = args.batch[0] output_filename = 'batch_output_{0}.csv'.format(args.catalog) # put name of catalog at end of output file with open(input_filename, mode='r') as csv_input, open(output_filename, mode='w', newline='', encoding='utf-8') as csv_output: read_in = csv.reader(csv_input, delimiter=',') write_out = csv.writer(csv_output, delimiter=',', quotechar='"', quoting=csv.quote_minimal) write_out.writerow(['issn', 'url', 'title', 'subjects']) # write out headers to output file total_rows = sum(1 for row in read_in) # read all rows to get total csv_input.seek(0) # move back to beginning of file read_in = csv.reader(csv_input, delimiter=',') # reload csv reader object for row in tqdm(read_in, total=total_rows): # tqdm is progress bar # each row is an issn issn = row[0] r = get_issn_data(issn, params) write_out.writerow([r['issn'], r['url'], r['title'], r['subjects']]) except exception as e: print(e) limitations while there are many limitations to this project, one glaring limitation is that typically subject headings are generated and assigned by a human, and this equates to the possibility of human error. we were hopeful to norm our subject heading data by generating and comparing data from three sources. however, the vufind catalog has been retired requiring us to update the subject scraper script for the new carli i-share catalog, something we have not done at this time. another factor impacting the task of scraping subject headings from library catalogs is that many servers, like the one for oclc’s worldcat, can only be pinged 999 times in a 24-hour period from the same ip address. this is a common practice for servers. therefore, when needing to scrape over a few thousand subject headings this work must be parsed out over a series of days. future work at the time we completed this project, lis at siue did not have the structure in place to apply the goals of this project into real world action. however, in the summer of 2020, a new dean took over leadership at lis and established the collection analysis and strategy committee that fall. a stated yearly goal of this committee is to create “a data driven approach to collection development with analysis of annual usage data, cost per use data, and, when applicable, journal impact factor data.” and additionally, “the library presents its objective view of potential cuts and engages the academic department faculty in a conversation about departmental needs, related to curriculum and research agendas.”[1] however, our future work of analyzing subject headings in journal collections will not rely on the custom developed python web scraper code that we describe in this article. oclc has since developed an experimental api (application programming interface) called fast api. “fast is an acronym for faceted application of subject terminologies for subject headings data” (fast api [updated 2021]).[2] now we will leverage this api to generate subject headings to aid in our electronic journal usage analysis by subject for collection development and to create annual portfolio reviews for campus departments. one specific area of possible future work is the analysis of our large journal packages from major publishers. this aspirational goal is inspired by the ideas of nabe and fowler in “leaving the ‘big deal’… five years later” (nabe and fowler 2015).[3] understanding the subject compositions of these massive databases would be a key tool in assessing whether we might replace these costly subscriptions with more targeted options. this would certainly aid in achieving the goals of maximizing roi on journal expenditures and creating departmental portfolio reviews conclusions this project was valuable in two major ways, both of which set the stage for the processes that will allow lis at siue to get the most out of our journal and database subscriptions going forward. first, the idea to analyze journal collections by subject headings in addition to cost per use, itself, anticipated the work that has been taken up by the collection analysis and strategy committee. this will allow for far more granular analysis of our renewals and is a foundational concept in the creation of discipline-based portfolio reviews for university departments. second, examining the architecture of the three library catalogs, writing the python script, and testing and correcting errors provided an understanding of the quality and extent of the subject heading metadata of academic journals. this knowledge will be invaluable as we now employ and assess the usefulness of the api from oclc in accomplishing the same tasks as our python script. however, we are hopeful that the script and processes described in this article may provide a low-cost pathway for others to analyze their own journal collections, if only as an experiment or as a first step in the journey toward more robust journal usage and subject analysis. code availability – github https://github.com/smcdavi/subjectscraper.py/blob/master/subjectscraper.py references [1] mcdavid, s. [2020]. library and information services collection committee proposal. edwardsville (il): southern illinois university edwardsville; [cited 2021 july 31]. [2] fast api [internet]. [updated 2021]. dublin (oh): oclc. [cited 2021 jul 15]. available from https://platform.worldcat.org/api-explorer/apis/fastapi [3] nabe j, fowler, d. [2015]. leaving the “big deal” … five years later. the serials librarian [internet]. [cited 2021 jul 15];69(1): 20-28. available from https://www.tandfonline.com/doi/full/10.1080/0361526x.2015.1048037 about the authors shelly r. mcdavid has been stem librarian (since february 2019) and area coordinator for access & library spaces (since january 2021) at southern illinois university edwardsville in edwardsville, illinois. her research interests include evidence-based and data-driven decision making for assessment of library collections, data, instruction, operations, resource sharing, services, and spaces in libraries, institutional repositories, scholarly communications, and open educational resources. author email: smcdavi@siue.edu eric mcdavid is a programmer/analyst who works with the center for health policy at the university of missouri. author email: mcdavide@missouri.edu neil e. das has been electronic resources librarian at southern illinois university edwardsville in edwardsville, illinois since january 2019. his research interests include student and faculty research behavior in the use of library discovery layers, library provided databases, and google scholar. author email: nedas@siue.edu subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – viafbot and the integration of library data on wikipedia mission editorial committee process and structure code4lib issue 22, 2013-10-14 viafbot and the integration of library data on wikipedia this article presents a case study of a project, led by wikipedians in residence at oclc and the british library, to integrate authority data from the virtual international authority file (viaf) with biographical wikipedia articles. this linking of data represents an opportunity for libraries to present their traditionally siloed data, such as catalog and authority records, in more openly accessible web platforms. the project successfully added authority data to hundreds of thousands of articles on the english wikipedia, and is poised to do so on the hundreds of other wikipedias in other languages. furthermore, the advent of wikidata has created opportunities for further analysis and comparison of data from libraries and wikipedia alike. this project, for example, has already led to insights into gender imbalance both on wikipedia and in library authority work. we explore the possibility of similar efforts to link other library data, such as classification schemes, in wikipedia. by maximilian klein and alex kyrios introduction libraries wanting to increase the value and discoverability of their metadata should explore options for making that data available and useful outside of the data silo of the library world. with library users increasingly turning to general search tools like google before library sources, libraries risk losing relevance by taking a passive role and expecting users to come to them. as profit-driven entities, google and amazon don’t exhibit such passivity. fortunately, wikipedia is proving a powerful partner for libraries. some librarians have traditionally regarded wikipedia with the same sort of skepticism they do google, as a generalist source that users turn to for “satisficing” in the place of high-quality library information sources. indeed, wikipedia has some limitations that make a certain amount of skepticism healthy. but libraries looking to keep their relevance in a digital age need partners, and wikipedia is well suited to be such a partner (kyrios 2013). unlike google or amazon, wikipedia is run by a nonprofit organization, the wikimedia foundation. as of september 2012, the wikimedia foundation employed 142 people, compared to over 50,000 for google and over 91,000 for amazon. in many ways, the wikimedia foundation thinks like a library. it wants to make lots of information available to lots of people, at no cost and for no profit. these are values many librarians share. there have been many efforts in which libraries have partnered with wikipedia, generally benefiting both parties. we will focus on a recent effort to bring authority control, a function librarians have long excelled at, to wikipedia. library authority work, consisting of creating authorized forms of personal names (as well as names of other entities, such as corporate bodies) helps distinguish individuals with similar names. wikipedia has its own practices, such as disambiguation pages and redirects, to deal with articles on individuals with the same name. spearheaded by wikipedians in residence at oclc and the british library, this project matched virtual international authority file (viaf) identifiers to hundreds of thousands of biographical wikipedia articles, using a matching algorithm and viafbot, an automated wikipedia account (a “bot”). viaf first proposed in 1998, viaf was created in 2003 as a joint project of oclc, the library of congress, and the german national library (deutsche nationalbibliothek). in 2007, the national library of france (bibliothèque nationale de france) joined, and in 2012, viaf became an openly accessible oclc service. viaf works with institutions to create master authority files. it assigns unique identifiers to each of its records, and also links these records to files maintained by its partner institutions (32 agencies as of june 2013, mostly national libraries) (murphy 2012). beginning in summer 2012, viaf files have also begun to be linked to wikipedia articles, and vice versa. wikipedia and wikidata wikipedia was launched in january 2001, though it hasn’t always enjoyed the prominence it does today. arguably, 2003 was the year the project hit the mainstream, marking its hundred thousandth article, the creation of the wikimedia foundation, the first real-life meetup of wikipedia users, and the creation of the globe logo that the site still uses ten years later. as of june 2013, there are wikipedias in 285 different languages, though the english wikipedia is the oldest and largest. this article discusses the english wikipedia unless otherwise specified. readers of wikipedia will be familiar with the site’s encyclopedic articles, but content in these articles is also affected by thousands of behind-the-scenes pages. this project in particular took advantage of such pages known as templates. templates serve a variety of purposes, but this project mostly used a few templates of structured data which apply to articles about persons. the infobox person template and its derivatives create the boxes of metadata visible towards the top of many personal articles. the persondata template serves a similar function, though it is invisible and has only basic parameters, similar to the fundamentals covered by a marc authority record. the defaultsort template specifies how an article will be alphabetized. personal articles typically have a defaultsort specifying a surname, given name order. finally, the authority control template, visible at the bottom of certain personal articles, already existed to link from wikipedia articles to authority files, though over the course of this project, this template was refined and came into use on hundreds of thousands of more articles. the project also benefited from the creation of wikidata, a wikimedia project that serves as centralized data storage for all wikipedias. savvy wikipedia users may know about wikimedia commons, a project which collects media for use across various wikipedias. wikidata plays a similar role for data. proposal (rfc) the idea of the viafbot project was first proposed on wikipedia in june 2012 by wikipedians in residence at oclc and the british library. a description of the project basics received mostly positive reception at the village pump, a discussion forum on wikipedia where new ideas are often proposed. subsequently, the full proposal was put to a request for comment (rfc), a more formalized procedure allowing wikipedia users to express their support for, or opposition to, the project. community input soundly endorsed the proposal, with thirty-eight editors in support and only two against (the editors voting against did not oppose authority control itself, but rather the addition of viaf identifiers at the bottom of articles, which they considered to be cluttered spaces). about a month after the initial proposal, the rfc was closed, with “clear consensus” to proceed with the viaf integration. the open, collaborative nature of wikipedia made it an ideal partner for this project. while the wikimedia foundation has some structured leadership, such as its board of trustees, these individuals typically have no more editorial authority at wikipedia than any other user. thus, a library attempting a similar project need not petition foundation leadership; an informal consensus among wikipedia users generally suffices. the process this project was made possible by algorithmic matching between viaf and english wikipedia conducted by the viaf engineering team. once the viaf matching algorithm had determined links between the two datasets, the work for viafbot became an elaborate copy and paste job. the viaf matching algorithm is designed to cluster authority file entities. these have properties like name, birth date, death date, and publications. owing to semi-structured data that occurs on english wikipedia through the use of templates, authority file-compatible schemas could be created from wikipedia dumps. in particular the templates persondata and defaultsort were most useful. it should be mentioned that matching occurred with one wikipedia to simplify the matching problem, and english was used in this case for its corpus size, as the largest wikipedia. however, such matching could be performed on any wikipedia, or even multiple languages. data from authority files matches by name and date, and publications were computed into viaf clusters. the efficiency of viaf clusters is improved with the number of contributing files because of the nature of this matching. there can be situations where a matches b and b matches c, but c doesn’t match a, yet a,b,c are a cluster. an example of such a case is the viaf cluster for bengali author taslima narsin. owing to differing romanizations of her name, many spelling variants exist. the wikipedia data is close enough to match the entity from bilbliotheque nationale de france, but not the union catalogue of polish research libraries. yet because the polish and french entities are close enough to match, viaf accepts the transitive inference to connect wikipedia and the polish entity. more data means better clusters in the viaf world. the first viafbot operated on english wikipedia and utilized the pywikipediabot python-wikipedia framework. its starting points were the clusters of the viaf database that contained an associated english wikipedia article. there were 269,494 viaf clusters which had a wikipedia link. each of these links was loaded, and the page was scanned to “sanity check” that the page was about an individual. sanity was accepted if the page had a template that was typically used on articles about people such as persondata, which was the case for 254,678 articles. of this group 9,034 had preexisting authority control templates. for each “sane” english article the bot followed the german interlanguage link, where possible, to attempt to find any german metadata from previous authority work. (interlanguage links are links to articles on the same subject in other wikipedias. they appear on the left hand side of many articles. the spain article, for example, includes a link to españa on the spanish wikipedia.) this interlanguage checking led to 109,087 german articles, of which 92,253 had the normdaten template, the german equivalent of authority control. of this subset, 74,864 had a viaf id (figure 1). comparing three possible data sources—english wikipedia, german wikipedia, and viaf—the bot noted how often each disagreed with another. the error rates varied between 10.5% and 15.9% (figure 3). in the case of any of these conflicts, the article was added to a conflict log and nothing was written to the wikipedia article. where there were matches, viafbot added a link to the relevant viaf file in that article’s authority control template (and added the template if the article did not already have it). figure 1: viafbot decision tree figure 2: viafbot statistics by wikipedia language as of november 2012 figure 3: disagreements among english wikipedia (authority control), german wikipedia (normdaten), and viaf.org as of november 2012 since november 2012, an ongoing period of human correction has followed the bot’s initial efforts. as of june 2013, 217 community-led handmade replacements have been made. after this process was complete, there was interest from other language wikipedias who wanted viafbot to run on their wikipedias. users at the italian wikipedia took matters into their own hands and migrated what viaf ids they could find with interwiki links. in that process they copied about 40,000 identifiers. however, the problem of how to spread this data to all 285 wikipedias still remained. in times past, the only option would be to write bots to shuffle and translate the templates for each individual wikipedia. even then, linked articles would not be guaranteed to have synchronized data—an error could be fixed in a german normdaten but remain in the corresponding english authority control. fortunately, wikidata was created to solve such problems, and started providing support in february 2013, while the viafbot project was ongoing. as a central datastore of semantic data for groups of pages all related by interlanguage links, its creation in relation to this project was well timed. wikidata did not emerge as a live product until after the italian copy had occurred. with the advent of wikidata, it was then possible to leverage this multilingual semantic data connector to expand viaf matching to new wikipedias. the next iteration of viafbot utilized a version of pywikipediabot with only alpha wikidata support. while the wikidata version of viafbot was custom written, the design pattern has since been incorporated into the script harvest_template.py. at this juncture, merging the data existing in all of english’s authority control templates along with its equivalents in german, french, and italian was attempted. these languages were selected because they had the highest incidences of authority control templates on their respective wikipedias. in order to get the freshest possible data for this operation, the bot actively retrieved live wikipedia pages rather than working off of a dump. while this was useful because it ensured up-to-date data, the total runtime of the bot extended past three weeks. the starting points for this bot were the lists made by using the special what links here bidirectional link function of wikipedia. when what links here is called on a template page, it returns all the pages that transclude (dynamically include) that template, so checking what links to common templates such as infobox person garner very useful, if unwieldy, data sources. working off this list of transclusions, each page was transformed using mwparserfromhell, a module which converts “wikitext,” wikipedia’s markup language, into convenient python objects representing data fields. each identifier in the authority control template is jotted down along with a source statement about the language wikipedia in which it was found. then the wikidata item number was queried from the wikipedia article. taking into account any preexisting authority control on wikidata, the new authority control is either added, or if it matches an existing claim, only the source is added. wikidata can display seemingly conflicting data as it tries to model provable statements—not truth, reflecting wikipedia’s standard of verifiability, not truth. in total, this process added close to a million statements of authority control from seven sources (table 1). table 1: authority file statements added to wikidata (gray and klein 2013) total number of items with any ac 417,915 authority file amount in wikidata virtual international authority file (viaf) 388,763 gemeinsame normdatei (gnd) 218,084 international standard name identifier (isni) 185,711 library of congress control number (lccn) 157,082 bibliothèque nationale de france (bnf) 15,665 système universitaire de documentation (sudoc) 12,950 istituto centrale per il catalogo unico (iccu) 1,540 addressing errors a sample of 100 random viaf ids from wikidata revealed 98 correct and only two incorrect identifiers. fortunately, wikidata was founded with the same open, collaborative model as wikipedia, so accuracy should improve naturally over time. however, while wikidata can keep data synchronized across wikipedias, it cannot do so outside of wikimedia projects. so when a when a user changes a link from wikidata into viaf, the link from viaf into wikipedia will not change, leading to a disagreement. to address this issue, it is planned that viaf will read the inbound links from wikidata and heal the link discrepancy. healing will happen more often in wikidata than in english wikipedia because it can be healed by a user speaking any language. there are approximately 85,000 active wikipedia editors of any language, compared to 30,000 on the english wikipedia alone. web traffic impact since viafbot launched on wikipedia, viaf.org has seen a threefold increase in traffic. figure 4: viaf.org traffic statistics notable specifically is the increase in traffic coming from english wikipedia, where the bot first operated. the wikidata version of viafbot started in march and has yet to show much impact, as it is not fully finished. however, the increase in referral traffic is greater than the sum of increase in wikipedia traffic, so there is also some knock-on effect from secondhand referrals, not from any one particular other referrer. for some perspective, not all traffic to viaf is referral traffic; from september 2012 until june 2013, viaf received 577,000 visitors from google searching, 303,000 from direct visits, and 207,000 from wikimedia projects. analysis case study: sex data after the wikidata import, viafbot performed a proof-of-concept auxiliary import that used the connections it had laid. the property for the sex of a human is one of the most used properties on wikidata. it is also information that is stored in viaf, and viaf’s linked data partners’ databases. wikidata was queried using pywikipedia bot for all the pages that have viaf as a property. the bot used code such as the following: viaf_property_page = pywikibot.itempage(wikidata, 'property:p214') #214 is viaf; stored numerically to avoid language ties. pages_with_viaf = viaf_property_page.getreferences() #which returns a generator, that we can cycle through for page in pages_with_viaf: #but these instances of class page are just promises, #until we use a method to get the data page_parts = page.get() #and when we do get the page it's returned as dict claims_list = page_parts['claims'] each item that contains a viaf id as a claim can possibly also have a sex as a claim. the viaf id string was used to build the uri of the record at viaf.org. the viaf.org record provides viaf’s “opinion” on the sex of the entity. that opinion is a result of a behind-the-scenes merge of all the national library files’ data. unfortunately, in this merge not all the data and its provenance is preserved. but because we live in a linked data era, it’s possible to follow links out of viaf into the online databases of the contributing libraries. the library of congress system was a good source for this effort because their data model for dealing with complex sex cases is nuanced. library of congress will record multiple sexes with applicable dates if they exist, which is a step in the right direction compared to the problematic binary (or trinary, counting those classified as intersex) wikidata model. now data could be compared between the library of congress and viaf. specific information from library of congress trumped the merged information on viaf. then the single ruling library opinion from that comparison was held to wikidata. if the opinions matched, the wikidata claim gained a source where no wikidata opinion previously existed. in the cases where wikidata and viaf disagreed with one another, a list was made for human correction. figure 5: comparison of sex data between viaf and wikidata of the entire eligible data set of 388,000 wikidata items with the viaf property, a subset of 131,650 has both wikidata sex data and viaf sex data. reassuringly, only .2% of this subset disagree. for each of those 311 items in that .2%, human research can right the discrepancy. for instance the deutsche nationalbibliothek thinks nadine warmuth is male, but wikidata thinks female (wikidata is correct). on the other hand, wikidata thinks that nguyen thi binh is female, but viaf is correct to suggest otherwise (klein 2013b). both wikidata and library sources make mistakes. there are also instances without even two sources to conflict with each other. there were 125,781 times when wikidata had sex data that viaf did not. this is a case where libraries could glean information from wikidata. conversely, wikidata was pleased to be informed of the 44,526 scenarios where viaf or loc had sex information but wikidata did not. lastly, and for perspective, each time sex information was handled, its content was tracked. of the 257,431 wikidata items with sex data, 14.7% were female, 85.3% were male, and 0.002% were intersex. the sex data that came from viaf showed a very similar story at 14.6%, 85.4%, and .006%, respectively, even at a lower sample size of 176,187 (see figure 6). the closeness between these two measures speaks to shared ingrained biases. wikipedia, which has been criticized for male bias (lam et al. 2011), hosts a thoughtful essay on its own systemic bias. this review of sex data suggests that libraries, frequently perceived as progressive institutions, may have some of the same issues. alternatively, perhaps the sort of systemic bias found on wikipedia simply reflects the bias of publishing in general. figure 6: composition of wikidata and viaf by sex this very brief case study of sex data is just one example of the types of research that will be enabled by the connection of the two datasets. future implications we believe the success of viafbot can be replicated with similar sets of structured library data. integrating authority data with wikipedia has been called “a milestone in the way library authority data are repurposed for non-traditional uses” (lovins 2006).this initial project with personal names is just the beginning. viaf contains other authority files, such as those for corporate names, that could be matched to wikipedia articles with a similar process. and of course, a huge portion of wikipedia articles describe neither a person nor a corporate entity. many of these topical articles could be matched with subject headings from controlled vocabularies such as fast, lcsh, or mesh. mesh terms, in fact, can already be manually linked with a template. as of june 2013, over 6000 medical articles already link to their respective mesh entries. articles could also be linked to classification numbers, so wikipedia readers could follow a link to worldcat to find library resources for further reading (or especially savvy users could take that number straight to the stacks and browse!). one possible technique is to use the subject classifications of citations of the article to determine the subject classification of the article itself (klein 2013a). the release of the dewey decimal classification (ddc) system as linked data might allow such a project (mitchell and panzer 2013), as might wikipedia’s own articles listing dewey and lc classes, which link to subject articles (ayers et al. 2008). concordances between wikipedia’s classification and universal decimal classification (udc) have also been explored (salah et al. 2012). additionally, mapping any one of ddc, lcsh, or lcc to wikipedia articles could aid in mapping the others. through the use of existing crosswalks, such as those included in the library of congress’s classification web, correlations among these systems could potentially make it easier to link to additional ones. this approach would require refinement, however, or human review, as ddc, lcsh, and lcc do not always match in neat one-to-one relationships. this would advance an interest among library of congress personnel in integrating lc vocabularies into linked data environments (tillett et al. 2011). at least one author has explored the use of wikipedia data to create a new classification scheme (yelton 2011). dbpedia, a project to extract structured data from wikipedia that was founded in 2007, is another potential partner in such efforts (morsey et al. 2012). conclusion the viafbot initiative has connected library authority data with hundreds of thousands of pages on one of the world’s most popular websites, increasing the visibility and availability of that data and, by extension, libraries as an institution. the positive reception to the project at wikipedia affirms the strength of libraries in performing authority control and proves the utility of this work in the era of linked data. the project offers a blueprint for similar efforts to integrate library data with wikipedia and, perhaps even more importantly, has built good will for the library community with wikipedia. at a time when many libraries worry about keeping up with information in a digital world, collaborations like this one offer an exciting glimpse of what libraries can still do to help connection their users with high-quality information resources. references ayers p, matthews c, yates b. how wikipedia works: and how you can be a part of it [internet]. san francisco (ca): no starch press; 2008. available from: http://archive.org/details/howwikipediaworks gray a, klein m. 2013. wikipedia in the library. refer 29(2):6-10. klein m. 2013a. wikipedia analytics engine [internet]. hanging together. available from: http://hangingtogether.org/?p=2452 *klein m. 2013b. sex ratios in wikidata, wikipedias, and viaf part 2 [internet]. hanging together. available from: http://hangingtogether.org/?p=2986 kyrios a. 2013. time for libraries to take a fresh look at wikipedia [internet]. the idaho librarian 63(1). available from: http://theidaholibrarian.wordpress.com/2013/05/17/time-for-libraries-to-take-a-fresh-look-at-wikipedia/ lam sk, uduwage a, dong z, sen s, musicant dr, terveen l, riedl j. 2011. wp:clubhouse?: an exploration of wikipedia’s gender imbalance. in wikisym ’11: proceedings of the 7th international symposium on wikis and open collaboration; 2011 oct 3-5; mountain view (ca). new york (ny): association for computing machinery. p. 1-10. doi: 10.1145/2038558.2038560 lovins d. 2006. cataloging news. cataloging & classification quarterly 42(1):139-147. doi: 10.1300/j104v42n01_10 mitchell js, panzer m. 2013. dewey linked data: making connections with old friends and new acquaintances. italian journal of library and information science 4(1):177-199. doi: 10.4403/jlis.it-5467 morsey m, lehmann j, auer s, stadler c, hellmann s. 2012. dbpedia and the live extraction of structured data from wikipedia. program: electronic library and information systems 46(2):157-181. doi: 10.1108/00330331211221828 murphy b. 2012. virtual international authority file service transitions to oclc; contributing institutions continue to shape direction through viaf council [internet]. 2012 april 4. dublin (oh): oclc. available from: http://www.oclc.org/news/releases/2012/201224.en.html salah aa, gao c, suchecki k, scharnhorst a. 2012. need to categorize: a comparative look at the categories of universal decimal classification system and wikipedia. leonardo 45(1):84-85. doi: 10.1162/leon_a_00344 tillett b, dechman l, mclean l. linking to lcsh and lcc: controlled subject headings and classification systems through the web [internet]. 2011. available from: http://conference.ifla.org/past/ifla77/149-tillett-en.pdf yelton a. 2011. a simple scheme for book classification using wikipedia [internet]. information technology and libraries 30(1): 7-15. available from: http://ejournals.bc.edu/ojs/index.php/ital/article/view/3040 about the authors maximilian klein (kleinm@oclc.org) is the wikipedian in residence at oclc. alex kyrios (akyrios@uidaho.edu) is the metadata and catalog librarian at the university of idaho and a wikipedia enthusiast. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – editorial introduction: ride on, mighty warriors mission editorial committee process and structure code4lib issue 13, 2011-04-11 editorial introduction: ride on, mighty warriors i joined the fellowship of the journal only one year past. in the time since i have helped to caress into sonnets the grunts and mutterings of paladins near and far. but forgive my crude tongue, for i do not wish to offend. you who share your drunken yarns are the font, the wellspring, of all that you see here. by gabriel farrell coordinating editor, issue 13 dark forces lurk in the fields of librarianship. the stagnant ponds of outdated systems surround us. we battle the dragons of inconsistent metadata, ghastly beasts like bureaucracy and inertia, the ruthless warlords who call themselves vendors in a niche market, and the arcane magic of ancient formats. at times, our swords and shields may seem no match for the challenges of this bleak world. but i beg of you, do not despair, fearless venturer! for herein lies records of others who likewise explore the uncharted lands betwixt the kingdoms of data and technology. read these words and rejoice in the camaraderie. sip at this stream of knowledge and receive sustenance. in our first conte, “groupfinder: a hyper-local group study coordination system,” brave souls joe ryan and josh boyer deliver a stirring account that could only have come from those accustomed to the travails of that baffling dimensional plane known as the academic library. may we all achieve the vigor with which they renewed their campaign after each focus-group fracas they fomented with the scholars of our age. a number of our parables relate quests for rare gems and runes to enhance that most powerful object born of alchemical sorcery: the mobile device. the renowned graham mccarthy and sally wilson reveal their familiarity with the infernal gadgets when they unleash a stunning spider’s web of technologies in “isbn and qr barcode scanning mobile app for libraries.” narrators of code nonpareil, denis galvin and mang sun also carry us deep into the fray with “using web services for a mobile opac.” if, in addition, you hope to harness the puissance of mystical tablets for your oracular companions, immerse yourself in the good james macdonald and kealin mccabe’s “iroam: leveraging mobile technology to provide innovative point of need reference services.” “implementing time travel for the web” paints a landscape never dreamt of until now. it recounts the twisting road taken and the uphill skirmishes fought by robert sanderson, harihar shankar, scott ainsworth, frank mccown, and sam adams, a truly formidable band. their lofty and audacious goal, to rend the very fabric of time, may unravel the lives of peasant and noble alike. renée mcbride seeks lost treasure among antediluvian ballads in “look what we got! how inherited data drives decision-making: unc-chapel hill’s 19th-century american sheet music collection.” let her regale you with her epic struggle to transform the moldy texts using every weapon in her pack. in a similar vein, “from isis to couchdb: databases and data models for bibliographic records” depicts luciano g. ramalho as he faces isis, the sister-demon of marc, in his crusade to free the lore of old from its esoteric prison. his target, the incalculable majesty of semistructured records, is magnanimous indeed. our own timothy m. of the clan mcgeary plunges us into the heat of the great game with “applying lessons from 8 things we hate about it to libraries.” his vision of susan cramm’s eloquent saga is bespoke for our library world, where a single misinterpreted word can lead to tumultuous uproar within the empire. another interpretive recitation among our generous offerings is “book review: html5: up and running,” in which mark cyzyk channels mark pilgrim with tidings of new armaments in the browser melees. learn to wield them before they are wielded against you. have you forgotten, my dear ladies and lords, what blessed and dreadful events unfolded at this winter’s bawdy festival? or mayhap an engagement with a fang-toothed kobold chieftain kept you from attending? if either be the case, or even if neither be, feast your eyes on the excellent annals penned by bohyun kim and elias tzoc at “conference reports: code4lib 2011,” our final breath in this spring missive. before i go i feel i must implore you once more. please, brave champion, press your steed onward into the long night. follow your destiny down the mysterious passages in the labyrinths of books and code, then return to tell us all that you have seen and heard. we will dutifully journey from village to village spreading the tales of your heroic deeds. subscribe to comments: for this article | for all articles 5 responses to "editorial introduction: ride on, mighty warriors" please leave a response below: mike giarlo, 2011-04-11 i hereby dub thee nerd champion of the realm. carry on, fair hero. jodi schneider, 2011-04-12 gsf++ john mark ockerbloom, 2011-04-12 truly an editorial for the ages. but where’s the stew? gabriel farrell, 2011-04-12 @ockerbloom you’ve uncovered a serious oversight. let it be known that the chaos at february’s gathering toppled many a silver tankard of stew. and now i must read the name of the wind. anna headley, 2011-05-11 i feel so encouraged right now! also feel certain that you need to lead a d&d session at next year’s conference. leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – simplifying ark id management for persistent access to digital objects mission editorial committee process and structure code4lib issue 54, 2022-08-29 simplifying ark id management for persistent access to digital objects this article will provide a brief overview of considerations made by the utsc library in selecting a persistent identifier scheme for digital collections in a mid-sized canadian library.  arks were selected for their early support of digital object management, the low-cost, decentralized capabilities of the ark system, and the usefulness of ark urls during system migration projects.  in the absence of a subscription to a centralized resolver service for arks, the utsc library digital scholarship unit built an open source php-based application for minting, binding, managing, and tracking ark ids. this article will introduce the application’s architecture and affordances, which may be useful to others in the library community with similar use cases, as well as the approach to using arks planned for an islandora 2.x system. by kyle huynh, natkeeran ledchumykanthan, kirsta stapelfeldt, irfan rahman editorial note: this post was updated on 14 november 2022 to include changes requested by the author. please examine the wayback-computed difference between the original article and this version to see the changes. introduction “persistence is a complex and expensive undertaking, eased only here and there by technology.” (kunze, j., 2003) as an increasing number of digital objects enter library collections, the challenge of persistence becomes ever-more germane for memory institutions. most institutions with repositories are making some kind of promise of continual access, and one manifestation of this promise is the stable, opaque, ‘clickable,’ identifier — a persistent url— based on an ark, purl, urn, handle, doi, wikidata qid, or orcid id (to name but a few). these identifiers promise both continual access and disambiguation on an increasingly complex information stage and facilitate librarians’ professional commitment to act as stewards of the distinctive collections/datasets of their local communities. the university of toronto scarborough’s digital scholarship unit (dsu) provides access to complex, multimedia digital objects across several web applications (maintained in the open-source digital management system islandora). recently, the unit has received increased requests for citation features so digital objects can be referenced in other contexts. there is also increasing interest in providing and using machine-readable endpoints. furthermore, the dsu understands that oai-pmh feeds and iiif manifests both benefit from greater reliance on a single, authoritative url. finally, as the unit has been developing a system based on islandora 2x and preparing for a migration, the timing seemed appropriate to revisit the potential of having a pid (persistent identifier) url system. this article outlines the logic for adopting the archival resource key (ark) system and introduces the dsu’s application for minting, binding, resolving and tracking arks. selecting a persistent identifier there are many excellent descriptions of the multiple systems provided for minting and maintaining pids. leggott et al (2016) provides an overview of the major types and functions of persistent identifiers, and barsky (2020) makes specific recommendations in the canadian context. koster (2020) provided a similar comparative overview of pids, and the heritage pids project maintains a detailed chart of comparative features of pids (kotarski, 2021). dois, which use the handle system, are readily recognizable in the academic context, particularly for published literature. according to doi.org, eleven registries currently provide doi services. despite representing what could be argued as an industry standard, the fee structures for these services exceed both the dsu’s budget and mandate for unique identifiers. (see datacite as an example of the fee structure of one such service.) the underlying handle system is appealing with a nominal annual fee to maintain prefixes and register with a resolving agency. however, in the end, several features of the ark system made it a better fit for the dsu. introducing arks arks are similar to other systems, such as dois, in that they give users the ability to author and maintain a persistent url for long-term access to an object. arks, introduced in 2001, appear to be associated most strongly with digitized collections, although the specification supports digital, physical, and conceptual objects of many types. an ark url consists of a domain name, the ark label, and a name assigning authority number (naan), which can be registered for free. under a naan, institutions can assign names, sub-parts and other variants to the ark (see figure 1). there are no fees for creating arks. figure 1. “a peek at ark anatomy” from https://arks.org/about/. although centralized services for resolving arks are available (via https://n2t.net/), they are not required. the ark alliance (the community maintaining arks)  is supported by over 40 organizations and is a member of the national digital stewardship alliance (arks.org, community). the actively updated naan registry table currently contains over 900 institutions globally that have registered to assign arks. for the dsu, perhaps the most attractive features of the ark system relate to metadata management. arks and metadata management “arks should be created at object birth, or even before. we sometimes name our babies before they’re born, and we name and refer to objects in the conception stages, sometimes long before they bear fruit.” (ark implementation best practices) of particular interest in the ark system is the support for ‘early object creation.’ every digital object that is gestated in the digital scholarship unit spends some time period in development and enrichment stages, often among many staff and stakeholder groups, and generally involving spreadsheet-based workflows that are well supported by arks. arks are minted and then can be appended to a metadata record at any stage of the object development process. in the islandora 2x system, the application islandora workbench supports direct, spreadsheet-based metadata ingest and update. drupal’s views and other serialization tools then allow for flexible exporting of object information, making roundtrips for metadata straightforward. as objects go through stages of description, reformatting, quality assurance, and publication, they bring their ark with them. in the past, the dsu has minted local identifiers or used identifiers from associated systems (such as identifiers from archives) to serve this purpose. however, these systems are inconsistent, and the resulting identifier rarely serves a clear purpose as part of the final metadata record. by forming part of a url that can be directed and redirected to other online locations, the flexibility of arks supports collaborative work on object creation and maintenance. to implement arks with the metadata manager in mind, the dsu developed the arks-service application. arks-service application introduction and setup the arks service is a php standalone application with a user interface based on the noid4php by daniel berthereau. this package is a php solution for the popular perl-based noid project written by john kunze. furthermore, this application adapts the work of akio sensei who has integrated a mysql database into noid4php, improving its performance. running the playbook will set up a web application with a public website that simplifies the process of minting, binding, and resolving ark urls by providing both a ui and templates for common ark tasks. the locations to which arks point can be changed in the application, and the number of redirects performed by the application is tracked in the application database. the dsu believes this advances a potential turnkey solution for ark management, particularly for smaller software development teams. this application supports both the older and emerging ark standards. the application provides a public website and a back-end interface (in the dsu environment, the back end is protected by a firewall). the public site contains the resolver that will direct an ark url user to the digital object referenced. (see figure 2). figure 2. systems overview of arks-service deployment. after installing this application successfully, users will be  prompted with the initial step of setting up an organization’s name and url as well as system administration credentials (see figure 3). figure 3. launch page after installation. creating (“minting”) arks after setup, a user can log in using these credentials in a browser and create a database to hold a ‘collection’ of arks (see figure 4). after naming a collection, a user is asked to select a template. these templates define the way that arks are generated by the system. for example, if a user would like to mint arks that have a “sequential 4-mixed-digit numbers with constant prefix of ‘sdd’” they would select the template called “sdd.sdede.” if a user would like the arks to be a “random 3-digit numbers, stopping after 1000th,” they would select the template .rddd. more information about the various available templates and their characteristics are available in the noid documentation. the dsu is using a single sequential series of arks for most of its collections; the ”.zd” template is used because it provides sequential numbers without limit. the “prefix” is a substring for the id  that must be unique if the user intends to host multiple ark databases. figure 4. creating a collection of arks. once the ark collection has been created, new affordances are available for a spreadsheet-based process of minting, binding, and updating ark ids. as depicted in figure 5, end users can choose to create or “mint” a number of ark ids, based on the schema selected on setup. the user enters a number of arks (for example 100) and then these are created in the database. they can then be downloaded in .csv format, which can be opened by excel and other spreadsheet programs. figure 5. minting arks. metadata and url (“binding”) arks once arks have been minted, they can be associated with their library-managed location (i.e., the url of the object in the digital repository), as well as any additional metadata using another spreadsheet (.csv file). the application provides a template to download, where additional columns can be added for specific fields. only the url field is strictly mandatory, although other fields are recommended. any number of additional metadata fields can be added to the ark. figure 6 shows the two screens available for bulk binding, although binding can be done at the level of a single object via the interface. figure 6a. bulk binding arks screen 1. figure 6b. bulk binding arks screen 2. these bindings and all associated metadata can also be updated in bulk or individually (which is useful for content migrations, as described below).  once a url is bound, the ark url will take a user directly to the digital object (as depicted above in figure 2). appending single and double question marks to the ark url will also display a metadata panel with all of the previously bound metadata fields (see figure 7, which shows a minimal metadata record brought back by appending a question mark to the end of the ark url). figure 7. bringing back bound metadata using the ark url. note that the metadata here is not sufficient for the ark standard, but is provided here as an example of this affordance in the application. metadata enrichment will be part of the work undertaken in the dsu upcoming data migration, particularly developing a persistence statement (a vocabulary is suggested in kunze, 2017) and adjusting the application to provide this on request. reporting features for ark ids at any point, a list of arks and their corresponding data can be exported from the system for review. an overview of the screen for this is provided in figure 8 – reporting for arks. an end user can search for individual records using the metadata and export the existing mappings. the system also records the number of times an ark redirect url gets used, enabling a user to see which arks are in use (in column ‘number of redirects’). figure 8. reporting for arks. the full suite of interface features and rest api documentation, as well as explanatory videos, is available in the github wiki for the project. edits and pull requests are welcome. persistent identifiers in islandora in islandora legacy (islandora 7), fedora 3x provides a persistent identifier (pid) under a specific namespace, accessible as a specific part of the object url. in the dsu system, where multiple individual drupal sites are serviced by the same underlying fedora, this often leads to varied urls. for example, the following two urls lead to the same object. https://gundagunde.digital.utsc.utoronto.ca/islandora/object/gundagunde:17492 https://collections.digital.utsc.utoronto.ca/islandora/object/gundagunde:17492 this approach also requires maintaining multiple subdomains (both gundagunde and collections), when either might change. the islandora provided pid (in the example above, gundagunde:17492) could uniquely identify the object in both cases but was not guaranteed to be universally unique. the unit also encountered challenges with the level of semantic meaning associated with the first part of the pid (the pid namespace, in this case, “gundagunde”) which stakeholders would request to change. adopting a more opaque naming strategy going forward is a priority in the unit. in islandora 2x, the ark id, stored in the metadata of the object (a drupal entity), can form the backbone of a url alias authored automatically by the drupal system (see: pathauto). node aliases that use ark ids mask the underlying drupal “node id” with something more persistent and unique. associating the ark id with the digital object via a drupal entity allows the system to use it in other system contexts. figure 9 depicts a sample of the new ‘permalink’ in the islandora 2x system, printed under the object but also available in features like citation generation. figure 9. permalink in the new islandora interface. conclusions there are multiple excellent systems and technical strategies for providing persistent identifiers and urls for digital objects. in the utsc library digital scholarship unit, the ark system was found to be the most flexible, least resource-intensive method for minting, binding, and resolving persistent urls. although the migration to islandora 2x is not yet complete, the ark system is already being used in multiple contexts in the existing islandora legacy system. persistent urls provide a measure of confidence to end users when citing objects in other contexts and will provide continuity during the system migration. the dsu’s arks-service application builds on well-developed open-source software already existing in the community to provide additional flexibility and user interface affordances for authoring and managing ark identifiers. the dsu hopes that this work will advance the ecosystem of lightweight, affordable, flexible open source tools for persistent identifier creation and management. works cited alliance, the ark. (2021, january 26). ark implementation best practices. ark alliance. retrieved june 28, 2022, from https://arks.org/about/best-practices/ alliance, the ark. (2022, february 23). community. ark alliance. retrieved june 28, 2022, from https://arks.org/community/ barsky, e. (2020, december 3). persistent identifiers in canada?: white paper [r]. doi:http://dx.doi.org/10.14288/1.0395014 jordan, mark. islandora workbench (2022), github repository, https://github.com/mjordan/islandora_workbench koster, lukas (2020) issue 47 code4lib journal 20202-02-17 retrieved from https://journal.code4lib.org/articles/14978 kotarski, rachael, kirby, jack, madden, frances, mitchell, lorna, padfield, joseph, page, roderic, palmer, richard, & woodburn, matt. (2021). developing identifiers for heritage collections (v2.0). zenodo. https://doi.org/10.5281/zenodo.5205757 kunze, j. (2003). towards electronic persistence using ark identifiers. retrieved from https://n2t.net/e/towards_electronic_persistence_using_ark_identifiers.pdf kunze, j., & rodgers, r. (2008). the ark identifier scheme. uc office of the president: california digital library. retrieved from https://escholarship.org/uc/item/9p9863nc kunze, j., calvert, s., debarry, j.d., hanlon, m., janée, g. and sweat, s., 2017. persistence statements: describing digital stickiness. data science journal, 16, p.39. doi: http://doi.org/10.5334/dsj-2017-039 leggott, mark, shearer, kathleen, ridsdale, chantel, barsky, eugene, & baker, david. (2016). unique identifiers: current landscape and future trends. zenodo. https://doi.org/10.5281/zenodo.557106 pathauto. drupal.org. (2022, april 24). retrieved june 28, 2022, from https://www.drupal.org/project/pathauto team, d. c. (n.d.). fee model 2022. datacite. retrieved june 28, 2022, from https://datacite.org/feemodel.htm acknowledgments the authors would like to acknowledge john kunze’s help in editing this post-print version of the article. note that further updates to documentation will be maintained in our github organization. about the authors kyle huynh is a software engineer at the university of toronto scarborough library’s digital scholarship unit. natkeeran ledchumykanthan is a software engineer at the university of toronto scarborough library’s digital scholarship unit and a committer with the islandora 2.x project. kirsta stapelfeldt is a librarian and the head of the digital scholarship unit at the university of toronto scarborough library irfan rahman is a systems administrator at the university of toronto scarborough library’s digital scholarship unit subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – editorial introduction mission editorial committee process and structure code4lib issue 17, 2012-06-01 editorial introduction coordinating editor tim lepczyk salutes change in this issue, welcoming new editors to the journal and announcing his departure. by tim lepczyk i joined the editorial committee of the code4lib journal over two years ago.  i had jumped at the chance to combine my professional interests in library technology with my personal interests in writing. now, in 2012, change is coming to both my personal and professional life. this summer, i’m getting married, relocating to arkansas, and starting a new job. with all of these changes in mind, it is with regret that i announce my departure from the editorial committee.  however, change never happens in isolation. as editors step aside, new people have joined the editorial committee in order to continue the journal’s success. please welcome peter murray, terry reese, elisa graydon, sara amato, and shawn averkamp as the committee’s newest members. it takes some time to find the balance between the journal, work, and one’s personal life, but i’m confident these five people will find it. continuing the theme of change, it’s time to turn our attention to the articles published in this issue and read how the contributors dealt with change or created change in their institutions through the use of technology. kirk hess’ article “discovering digital library user behavior with google analytics” delves into google analytics and demonstrates how libraries can track events and analyze data in order to better understand user behaviour. libraries often face issues regarding tight budgets, limited staff, and a large amount of processing. in the article “the martha berry digital archive project: a case study in experimental pedagogy” stephanie a. schlitz and garrick s. bodine write about the problems their team faced and how they overcame these challenges. some challenges libraries face are broader than what comes across in a case study. for instance, if your library is interested in using user-generated content for bibliographic records, how might the problem be approached? one answer is in “using semantic web technologies to collaboratively collect and share user-generated content to enrich the presentation of bibliographical records–development of a prototype based on rdf, d2rq, jena, sparql and worldcat’s frbrization web service” written by ragnhild holgersen, michael preminger, and david massey. “glimir: manifestation and content clustering within worldcat” by janifer gatenby, richard o. greene, w. michael oskins, and gail thornburg discusses a project to mine worldcat’s vast store of bibliographic data to create record clusters and identifiers that will help bring together disparate versions of metadata for the same thing at multiple levels and provide new tools to connect library data with other applications. automation can be a great tool for change. doreva belfiore shows how perl and cgi scripts streamline their workflow in “case study: using perl and cgi scripts to automate a quality control workflow for scanned congressional documents.” finally, when planning for change or moving across the country, maps are wonderful tools to use. in “from the catalog to the book on the shelf: building a mapping application for vufind” by kathleen bauer, michael friscia, and scott matheson you will learn how the yale university library created a mapping application for mobile devices. we hope you find the issue informative and inspiring. as a community, code4lib encourages experimentation and collaborative problem solving. hopefully, the articles above will help you envision ways you can improve services and change things for the better at your institution. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – using google calendar to manage library website hours mission editorial committee process and structure code4lib issue 2, 2008-03-24 using google calendar to manage library website hours the management and display of hours of operation on a library website can be needlessly complicated. one relatively simple solution is to manage the library hours in google calendar, and then use the google api to extract this data for use on the public website. this article outlines how the ithaca college library used google calendar, php and mysql to manage and report against our library's operational hours. example code is included. by andrew darby introduction the hours of operation are perhaps not the sexiest aspect of a library website, but they are nonetheless a “killer app,” providing a small, crucial bit of information for your users. library hours tend to be quite dynamic, varying from day to day, semester to semester, and year to year. from the user’s perspective, this changeability means it’s the sort of information you will want to quickly look up before lacing your shoes, or planning some last minute research. from the web developer’s point of view, dealing with this changeability just makes your work more difficult. developing a back end to manage your website hours–or better, to allow someone else to manage the hours–becomes complicated quickly. after working on such a management tool for a while, and getting caught up in the complexities of dealing with the competing academic and gregorian calendars, i decided to outsource the most complicated part of the process–handling the addition and removal of dates–to google calendar, and then use the gdata api to power my pages. a related problem was the management of the reference desk calendar. it was in an excel file on a shared network folder, but we started using google calendar for this, too. as a result, using the google api, we have access to information such as the current librarian on the desk (in order to show their picture) and upcoming reference desk shifts for a given librarian. [1] what follows is an explanation of how to get your library hours running using google, php and mysql. [2] there is a list of resources at the end, as well as downloadable code for the back-end and front-end pages. figure 1: library hours page [view full-size image] requirements the one absolute requirement to set up an hours management system using google calendar is a google account, but for the process i’m going to outline, you’ll also need php >= 5.1.4 in order to use the time-saving zend gdata library, and a database to store your library hours. [3] setting up google calendar you may or may not be familiar with google calendar already, but as you will be interacting with the calendar api, you need to make sure the information you enter into the calendar can be retrieved again. essentially, you (or someone) will need to enter the dates for every day of the semester or term. this sounds quite time-consuming, but since you can set events–in this case, the hours of operation for a given day–as repeating, it can be done quite quickly. using google calendar is outside of the purpose of this article, but let me share two tips which might save you some time: if you choose the “repeats weekly” option when adding an event (see figure 2, below), you get a nice set of tick boxes for each day of the week, and you can set a ending date. i found this the easiest way to handle a repeating event. if the library is closed on a given day, you might add an event with hours like 7:30 – 8:00 am, and then use the word “closed” in the title. the “hours.php” script searches for the word “closed” in the title field, and if found, sets a closed flag in the database. the problem is that if you leave a day blank, the feed from google does not include that date, so you need something in every day. this example does not use the data from the what, where or description fields of google calendar, but keep in mind that all of these should be accessible through the api if you wanted to use them. figure 2: calendar — select weekly [view full-size image] setting up the database in mysql you will need to create the table libhours in one of your mysql databases. the following sql may be used to generate libhours. create table libhours ( libhours_id integer unsigned not null auto_increment primary key, ymd varchar(10) default null, # (yyyy-mm-dd) one entry for each day dow varchar(10) default null, # day of the week opening varchar(10) default null, # opening time closing varchar(10) default null, # closing time is_closed integer(1) default 0 # 0 (open) or 1 (closed) ); interacting with the google api the google api developer’s guide (resources, below) provides tutorials for python, php, .net, java and javascript. as noted, you will be using php. the php page supplies detailed instructions, which seem to be updated with some regularity, but here are my own slightly less detailed instructions to get you up and running quickly. 1. download the zend google data client library while it is possible to interact with the google api directly, using this library makes everything much simpler. go to the client library download page (resources, below), download, unzip, and place in your php includes folder. or, put it in any folder and make sure to use the php set_include_path to point to this other location. 2. edit the configuration file the file config.php (code, below) contains some configuration variables you will need to set, as well as a basic function to connect to mysql. open config.php in a text editor and enter your gmail credentials (for the calendar you wish to connect to), your mysql database name, your mysql username and password, your hostname (if it’s not localhost), and the path to get to your zend gdata library (if it’s not in your system-defined php includes folder, or you haven’t added an entry to php.ini). you can also set the number of months into the future that the script should check. 3. create connection/update file assuming you have at least one item in your google calendar, the mysql table libhours, and the zend client library, you can now create a page, gdata_connect.php (code, below), to connect to google calendar, return your calendar data, and update your database. when you make a change to your google calendar, you will need to run this file again to update. if your hours get modified frequently, you could run this page with a cron job; in the more likely case that the hours get set once a semester or term, and perhaps tweaked a couple of times thereafter, you could manually run the page after changes are made. this page: calls config.php to get database connection and configuration information. includes the zend/loader.php file and loads the zend classes. sets google calendar authentication. this example uses the clientauth authentication type, but there are two other options: authsub and magiccookie. clientauth is considered the least secure–because you include your username and password directly in the file–but it is also the simplest. [4] loops month by month, each time adding the information for each day to a sql query. (i ran into issues sending, say, a six month range, so to be safe it is set to fetch one month at a time.) inside the loop, the function outputcalendarbydaterange is called. connects to the database. deletes all entries in the database for the date the script is run on into the future, and then uses the sql generated in the loop to insert the new values into the database. all entries before the day the script is run are retained. in the function outputcalendarbydaterange, which is based on the “retrieving events for a specified date range” example in the php developer’s guide [5], we use zend_gdata_calendar_eventquery to get back information from our calendar in the form of an atom feed. [6] a number of properties can be set here, but none need to be modified right now. they are mostly self-explanatory: setuser will set the user whose calendar you are connecting to. “default” is only available for “private” feeds, and, as you might expect, will display the default calendar based on the login credentials. setvisibility has two options: public and private. public feeds may be accessed by anyone, while private feeds are … private. [7] setprojection gives you some options for what data you want the feed to contain. a list of options may be found in the api reference guide. [8] setorderby allows you to either order by “starttime” or “lastmodified.” setstartmin and setstartmax are set in the loop that calls this function. note that dates are returned as rfc 3339 formatted strings, i.e., 2008-01-26t10:00:00.000-05:00. 4. run gdata_connect.php visit gdata_connect.php with your browser, and it will run the above script. it should populate your libhours table with the correct data. note that you seem to need to log out of gmail/google calendar after you make an update (and before this file is run), or else you will not get the most recent changes. 5. create a public html page you will now need to provide a front-end to show your library hours on your website. since your hours are in a database, you have some flexibility in how you can display them. you could display today’s hours on the home page, and then link to a fuller calendar on its own page (which is what we do). the included hours.php (code, below) is a script for a page to display the hours in a monthly calendar, using the data generated by gdata_connect.php. [9] conclusion with any luck, you now have a working library hours application. if you wanted to show a picture of the reference librarian on duty, use the same procedure, but with live queries to google calendar. [10] there is a great deal more one can do with the google calendar api. you can create, update and delete events in a calendar; find out who is subscribed to a calendar; retrieve “secondary” calendars (which might be useful if you need to manage multiple sets of library hours, although i have not explored this); create entirely new calendars; and so on. there is even the (very simple) option to embed a google calendar directly in your page, if you’re willing to hand over the look and feel to google. resources the following resources are helpful for getting a handle on using the google api and php. note that google is regularly adding features and documentation to their pages, so information and urls might change. google calendar google calendar api php developer’s guide (for google calendar) zend gdata client library download zend api guide zend programmer’s reference guide (ch 14, zend_gdata) code there are three files available. download zip (6.5 k) config.php — set a couple of configuration variables here. gdata_connect.php — the file to connect to a google calendar, extract the next x months of data, and insert into a database. hours.php — the file to display the library’s hours in a month-by-month format. notes [1] faculty and students often pop by looking for a specific librarian, and i thought these might be useful bits of information. plus, it’s cool. live examples of these pages are available: ithaca college library hours and ithaca college reference desk hours. [2] of course, you could use other scripting languages. the api developer's guide has tutorials for python, .net, java and javascript in addition to php. [3] each query sent to the google calendar api can take a couple of seconds, depending on network connectivity, so some form of caching of results is desirable for a page like the library hours, which is both reasonably static across a semester/term, and relatively high traffic. i like the flexibility of having the information in a database, but you could also print out a set of static pages for upcoming months, cache live queries at an appropriate interval, or just query anew each time a user hits the page. on our reference hours page, we do use real-time queries to display a picture of the librarian on the desk at the moment. this is because: a) this is a low-traffic page, b) the lookup is done asynchronously, so there is no delay for the main contents of the page to load, and c) the information changes quite regularly and unpredictably (humans are like that). [4] see http://framework.zend.com/manual/en/zend.gdata.calendar.html #zend.gdata.calendar.connecting for examples of each. [5] see http://code.google.com/apis/calendar/developers_guide_php.html #retrievingdaterange. [6] looking at the feed directly is not so helpful, because each repeating item will appear as a single entry in the feed, with a description (inside the <content> tags) that looks something like this: <content type="text"> recurring event first start: 2007-11-06 08:00:00 est duration: 64800 where: in the library event status: confirmed event description: this is a repeating event </content> there is a singleevents parameter which might make this more readable, however. see http://code.google.com/support/bin/answer.py?answer=55834&topic=10360. [7] a feed’s visibility is set within a specific google calendar’s “calendar settings” area: click the settings link on the upper right of your google calendar page, select the calendar tab, and then click the link with the name of your calendar on it–which will magically create a new “calendar details” tab, where you can scroll down and find public and private addresses. [8] see http://code.google.com/apis/calendar/reference.html#projection. [9] this builds upon a script found at http://www.plus2net.com/php_tutorial/php_calendar.php — the first google hit at the time i was first working on this. [10] the reference librarians at ithaca college keep a google calendar of the reference desk staffing, and make sure to always use the person’s initials in the title field, i.e. “ad on desk.” the reference hours page queries the google calendar (through an ajax request, so the rest of the page loads first) for only that hour’s entry, and then matches the initials (ad) to the correct picture, and displays it. about the author andrew darby is the web services librarian at ithaca college in ithaca, ny. he maintains subjectsplus, a free and open-source subject guide management tool. subscribe to comments: for this article | for all articles 14 responses to "using google calendar to manage library website hours" please leave a response below: tomkeays, 2008-07-09 i came across this article on ibm developerworks today, “integrate your php application with google calendar”, that nicely extends andrew’s piece. http://www.ibm.com/developerworks/web/library/x-googleclndr/ adi, 2008-09-22 hello my task to extract data from mysql database, which contains events, dates and authors and put the events automatically to google calendar. so any help would do. if you can provide me any help, i would be very tahnkful. thanks adi darren, 2008-12-16 nice article – thanks darby :) how do you display today’s hours on ithaca library’s mainpage? andrew darby, 2008-12-18 @darren: we use the method noted above, and a cron job queries the db every night and saves that day’s hours to a file, which is then read by our home page. michelle, 2008-12-18 interesting way to do that, andrew, and less taxing on the server i would think. i implemented this in a barebones form for haverford’s library website, but didn’t think to do that, so every time the page is loaded it’s querying the db. plus we just got php 5, so i haven’t been able to implement the google calendar portion of this yet. i’m going to try that out in january. michelle, 2008-12-18 oops, i lied! i did come up with something so it doesn’t have to query the db every time the page loads, but not as nice as what you described. darren, 2008-12-22 thanks again darby – nice code dung le, 2010-06-23 thank you so much for such a well written a useful article. dung. patty, 2010-09-11 this is pretty useful..i think i might find a couple other uses for this code. ray voelker, 2011-03-24 i haven’t looked very deep into this, but has anyone had trouble with 24 hour events not showing up in the feed? i tried, and they just won’t import into the local db. thing 8 – i dream of google calendars « librarian in a nutshell, 2011-08-05 […] pages, since currently i manually update html tables in my capacity as the library webmaster.  the code4lib article and blog post about how libraries are using google calendar (both provided in the original thing 8 […] web designer, 2012-01-04 i can’t seem to find the ability to show multiple calendars. is it possible or i have to separate the queries? thing eight: google calendar « charlie's 23 things for professional development, 2012-08-10 […] see some examples of how libraries use google calendar and thought ithaca college library’s use of the google api to push google calender data to their library homepage was particularly ingenious. particularly the […] amy, 2014-12-19 hi all – just an fyi, google changed their calendar api in november (see https://developers.google.com/google-apps/calendar/) so the code from this will need some tweaks to get it to work. i’m hoping to have time for this in the new year. thanks, amy simmons college library leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – standing up vendor-provided web hosting services at florida state university libraries: a case study mission editorial committee process and structure code4lib issue 58, 2023-12-04 standing up vendor-provided web hosting services at florida state university libraries: a case study createfsu is florida state university libraries’ branded reclaim hosting, domain of one’s own web-hosting service. createfsu provides current fsu faculty, staff, and some students web domains and over 150 popular open-source content management systems including wordpress, drupal, scalar, and omeka. since the launch of the service in september 2021, the libraries have negotiated the demands of providing such a service with various administrative stakeholders across campus, expanded the target audience, provided support and refined our workflows and documentation to make the service fit campus needs. using this service, members of the fsu community showcase the fruits of their research to a broad audience in ways that are highly accessible and engaging. more work needs to be done to promote createfsu to the fsu community and identify opportunities to integrate the service into existing research and learning workflows. to expand the service to meet new use cases and ensure its scalability, the libraries hope to convince campus partners to consider its utility to their missions and contribute funding. this article lays out our experiences in launching and hosting this service over its first two years and proposes steps for future development and growth. by matthew e. hunter, devin soper, and sarah stanley background & need florida state university (fsu) libraries’ office of digital research and scholarship (drs) has been partnering with fsu faculty and students on the development of digital humanities projects since 2015. through these partnerships, drs librarians quickly identified web hosting for digital projects as an important gap in fsu’s academic computing services. for example, many of the researchers who participated in drs’s first project enhancement network and incubator (pen & inc.) program in 2016 expressed web hosting needs that the libraries were not able to meet with our it infrastructure at the time, and that also did not meet the criteria for hosting through fsu information technology services (its) website content management system or webdav site hosting services. similar needs were expressed by the faculty who participated in the demos project for studies in the data humanities (est. 2018) and by faculty and students affiliated with the program in interdisciplinary humanities’ digital humanities ma program. in each of these cases, faculty and students expressed a need for flexible web hosting that would enable them to easily install a variety of different content management systems, from hyper-accessible options like wordpress to more specialized options like omeka and scalar. they also expressed a need to have some level of control over the development and administration of these sites, including the ability to create accounts for student workers tasked with creating and publishing content on the domains as well as the ability to make updates to the content as needed. in thinking through these needs, the drs team realized that the requested web hosting service could also help to round out existing campus infrastructure related to digital pedagogy, particularly for instructors wishing to produce web-based pedagogical materials or assignments outside of canvas, the learning management system environment employed by fsu. specifically, we recognized that a flexible web hosting service could provide fsu instructors and instructional support offices such as the office of distance learning and the center for the advancement of teaching with an additional tool to enable the exploration of innovative pedagogical frameworks (e.g., open pedagogy and public writing) and to continue to meet the needs of distance learners in new and innovative ways. in evaluating options to meet these needs, there was no better candidate than reclaim hosting’s domain of one’s own (dooo) service. born out of collaborations between professors, instructional designers, and educational technologists, reclaim hosting is an llc that provides flexible, scalable web hosting that is marketed primarily to the education sector. reclaim hosting supports one-click installation of over 100 popular content management systems, including wordpress, drupal, omeka, and scalar. the dooo service is designed specifically for academic institutions, and has been adopted by institutions across the us to meet a variety of use cases related to research and learning. although most institutions use the dooo service to support teaching and learning, there are also institutions that use it to support research specifically (e.g., university of kentucky and university of south carolina). because the drs team is focused predominantly on research support services, the use cases that we planned to accommodate with the service (and that we will address in this report) are mostly research-related; however, we recognize the utility of the service in meeting a variety of teaching and learning use cases, and we hope that we can eventually secure funding to better meet these needs in the coming years. dooo implementations are typically branded to appeal to the users at an institution. at fsu libraries, we chose the name createfsu to brand the service. figure 1. screenshot of createfsu landing page. literature review most case studies on reclaim hosting’s domain of one’s own service pertain to student-focused implementations that aim to support blended learning and advance students’ digital literacy skills (e.g., kehoe & goudzwaard, 2015; o’byrne & pytash, 2017; taub-pervizpour & clarke, 2017; haimes-korn, greene, & rorabaugh, 2021). winchester & freeman (2021) discuss the use of dooo to support digital scholarship, but they discuss it only in a cursory way as part of a broader suite of services. hootman’s presentations at recent coalition of networked information meetings (2020, 2022) provide the only in-depth case studies on the implementation of dooo as a service in academic libraries. given this relative paucity of published case studies, we believe that this case study about our implementation of dooo at fsu libraries fills a gap in the literature on the topic and could be especially helpful to readers of code4lib journal, particularly given the efficiencies that dooo web-hosting can provide relative to the alternative of developing bespoke web-hosting services. scope and use cases the drs team proposed the createfsu service to the libraries’ senior leadership team in november 2020 as a three-year pilot project. the service is supported primarily by two librarians from the drs team and, as mentioned above, is intended to support a variety of use cases related to the research enterprise at fsu. createfsu is available to any fsu faculty members, graduate students, or undergraduate honors students who want a web domain to host content related to their research. more specifically, we envisioned supporting the following use cases as types of projects that could fit within the createfsu portfolio: a faculty member interested in creating a digital supplement to a book that they authored, showcasing archival photos and other media that the publisher was not able to include in the print version. a graduate student interested in creating a digital appendix to their dissertation with datasets and exhibits that further support their argument which could not otherwise be included within the confines of the written manuscript. an interdisciplinary research team interested in crafting a website to showcase their team members’ expertise and collaborative projects in order to bolster their grant proposals to research funders. an instructor interested in expanding their pedagogical impact by creating a place for her students to build a digital exhibit where they find materials related to the course, share them with the class, and provide contextual information explaining how the materials relate to core curricular concepts. a faculty member interested in developing a website that stands as a digital-native research output in itself, such as an ongoing crowdsourced space to showcase pop culture references to her area of research. an instructor teaching a graduate course on digital rhetoric and first-year composition who would like to have a collaborative blog in which students can write and then comment on each other’s work. a undergraduate research opportunity program (urop) project director interested in creating a site for all of their research assistants to add to, which can survive beyond the yearly required poster, and which project alumni can reference as part of future portfolios. an undergraduate honors in the major student who would like to craft a digital project website to go along with their honors thesis project. a faculty member or graduate student is interested in showcasing notable publications, grants, and awards, as well as other salient information about their research agenda more generally while associating with the broader florida state university web environment. to keep support of the project in-scope and sustainable for the pilot period, our team originally planned to restrict the following use cases as out-of-scope: departmental websites (including research centers and institutes) student assignments [1] websites that have any vendor/marketplace component websites that have paywalls for access to content in some cases, these limitations were implemented to clarify lines between the createfsu service and university its web hosting. in others, the restrictions were based on concerns about workload for librarians, university policy, and security. lab websites for faculty were originally considered out-of-scope, but this restriction was relaxed after conversations with fsu’s central its. members of the createfsu team and its determined that labs websites that fell into the category of official research centers and institutes would be administered by its, but uncharted lab websites (i.e. a faculty member and their research team) would be eligible for the services offered by createfsu. implementation after securing approval and funding for the three-year pilot from the libraries’ senior leadership team, our first step toward implementation involved discussing the proposed pilot with fsu its leadership, particularly with respect to questions of it governance and data security and privacy. its representatives recognized the needs that we intended to meet with the service were under-addressed. therefore its was open to the libraries undertaking the pilot project and gave their support. data security and privacy were the principal concerns for its, with a close second being any institutional liability that might result from users’ mis-using the service to cause data security or breaches. however, these concerns were alleviated once we explained the nature of the use cases that we intended to support and offered to include language in the terms of service prohibiting the creation of websites that contain personally identifiable information (as defined by hipaa and related privacy legislation). its also asked us to include language in the terms of service related to intellectual property, relevant university codes of conduct, and the right of the university to monitor content on the platform and remove it at its discretion. after the terms of service were approved by its, the next steps involved working with fsu procurement to complete relevant software vendor checklists and review and draft an addendum to the dooo contract to accommodate various requirements that are specific to our university. following this work with its and procurement, the next steps involved configuring our instance of dooo to reflect the new createfsu brand, the scope of the service and allowable use cases, and wider university branding guidelines. createfsu administrators at fsu were given administrative or root access to three main accounts: the createfsu wordpress, webhost manager (or whm), and the client-management tool “webhost manager complete solution” – whmcs. the createfsu wordpress gives administrators the ability to manage user accounts, create and modify request forms, and manage informational content on the createfsu website. whm allows administrators to control storage quotas, modify available applications, and set default content for new application installations. whmcs serves as the client management system, which allows administrators to email users, switch accounts between users, and terminate accounts when a user leaves the university. all of these systems work together seamlessly. for example, when an fsu user logs into the createfsu wordpress, an account is created for them in whmcs. if they go on to create a domain, that domain is automatically available through whm. the createfsu team also worked to further synchronize services by connecting support and new account request forms in the createfsu wordpress to our request tracker system for managing tickets. we also created a handful of sandbox testing domains in order to teach ourselves how to use the various back-end, administrative systems that control the creation and configuration of domains. these testing environments also allowed our team to get better acquainted with the content management systems that we expected to be most popular with our target audiences. in the course of putting these aspects of the platform through their paces, we identified a number of technical issues that needed to be resolved before we could officially launch the service. for instance, we realized that automated emails sent from createfsu domains (e.g., for user account creation) were being blocked by spam protections put in place by our central its office, which required extensive troubleshooting with reclaim and its stakeholders in order to find a resolution. the conversations with its continued after the initial setup phase. there were several highly specific policies and compliance topics that were not identified during the initial implementation that needed to be resolved. for example, wordpress accounts accessible through fsu.edu domains would need to have their default login url hidden for security purposes. the createfsu team needed to work with reclaim hosting to systematically hide these login locations. additionally, the team needed to have conversations with its and university communications about site branding and ensuring that each site was labeled as a createfsu site (as opposed to a university-sponsored site). these topics were not immediately apparent during the setup conversations and they necessitated ongoing communication with central university partners to create and implement solutions. ultimately, both wordpress login configuration and createfsu labeling for sites were resolved through template installations, a feature provided through reclaim hosting’s services. we installed a wordpress instance in the createfsu environment and added and configured required security plugins. we also edited the footer of this wordpress instance to link out to createfsu. we were able to configure this wordpress instance to be the default content and configuration for all new wordpress installations through the service. this solution has satisfied the requirements of both its and university communications. marketing the createfsu service was, in large part, primarily a response to expressed need from faculty and students who had previously collaborated with drs staff. initial outreach for the service was therefore initially focused on notifying these individuals about the launch of the createfsu pilot. however, to move beyond this initial group and address the broader need for campus-wide web-hosting, it was necessary to develop a plan for targeted marketing and outreach. the first, easiest, and fastest way to advertise the new service was a series of blast-announcements in shared notification pathways such as those on the weekly internal libraries newsletters (to generally inform subject liaison librarians and others with potential contacts across the university) and in the campus-wide “important announcements” newsletter. additionally, a press release was published and linked to on the libraries’ homepage for several weeks. these pathways generated light but nontrivial engagement and resulted in several initial site provision requests. since the service is still a pilot and is also restricted in both user count and storage limit, there were no initial plans to send university-wide emails beyond the general newsletter and press releases. beyond this, outreach needed to focus a bit more on those users who could reasonably be expected to have a pertinent use-case or would be interested in developing one based on their research or teaching area. historically, drs has had great success with word-of-mouth advertising of our services and programs, and we hoped to leverage that with a strategic focus on pitching the service to select departments, researchers, or communities we could rely on to advocate for the service in their own networks. with this in mind, our task in identifying who to initially reach out to was made easier by the initial scope of the createfsu use case prescriptions developed at the outset of the project. our primary audiences for an initial round of outreach were scholars with ties to drs and the digital humanities research network on campus that we felt could utilize a website to showcase their work. these projects were often text-based or encoded textual scholarship that could utilize web technologies to publish and display their work. the first accounts created were for public-facing scholarship that engaged with legal and educational topics that were broadly pre-written and intended to be ported to the new service rather than written from scratch. their quick completion also meant they would be great to have as initial examples we could then showcase to future potential users whose content was not already created (and administration who would be interested to see the impact of the new service). after initial adoption by partners in the colleges of education and law, we became interested in targeting our services to include more public-facing scholarship, and hence have focused our outreach into areas of research on campus that often engage with public writing and digital pedagogy centered on web-writing. to this end, we have focused our outreach efforts towards centralized locations on campus that specialize in instructional support, rather than bombarding individual users directly through additional mass emails. fsu’s office of distance learning and our center for the advancement of teaching were chief among these locations, as they are connected with faculty attempting to bring digital research tools and data literacy methods into the classroom. by partnering with these two units, we will be able to more easily reach faculty who are working on pedagogical projects that may benefit from the createfsu service concurrent with our efforts in fostering word-of-mouth advertising to research-focused use-cases. by far, the most impactful form of marketing and outreach for the service was a digital project incubator program that provided several forms of dedicated hands-on support from createfsu staff. in its current iteration, the program is run by the digital scholarship and digital humanities librarians, who also serve as the primary administrators of the createfsu service. this incubator program, dubbed the “project enhancement network and incubator” (backronymed from the punny “pen & inc” abbreviation) provided participants dedicated consultation time, account provision in the createfsu service, and a moderate financial incentive. the incubator’s target audience was faculty and students interested in generating scholarship utilizing digital methods or work intended to be shared primarily through digital publication, with the understanding that they would be willing to serve as beta-testers for the hosting service as we worked out kinks. the pen & inc program has been run twice with createfsu as its foundation. as a chief win, the incubator program allowed the libraries to garner interest in the service broadly through departmental communications, two recognition events, and through the advertisement of participants’ sites. beyond this, it provided an opportunity to refine policies and workflows and “load test” our own capacity for customized support with a group of invested and motivated researchers. the first cohort had eight applicants and eight participants. the program drastically increased in popularity for the second iteration, with 26 applicants and 15 participants. this spike in interest demonstrates the need for web hosting services and support on campus. partially in response to the visibility the incubator program brought to the service, our team was able to make inroads with new departments on campus that had not previously shown much interest in digital scholarship or library publishing work. one in particular was the undergraduate honors college, which is now exploring digital portfolio capabilities for undergraduates using the service. usage there are currently 106 total domains across 99 active users. “active users” are defined here as users who have access to one or more domains. of these users, a majority are faculty at the university. we also have several graduate students and undergraduate honors in the major students participating in the program. our hope is to expand the number of users in the student category (especially those who may want a web presence for theses and dissertation projects). accounts labeled “admin” are createfsu accounts run by librarians to manage createfsu systems, test out functionality and features, and promote createfsu-related programming. figure 2. count of users by role at fsu. the breakdown of users by college shows a more even distribution compared to the users by role. a majority of users are from the college of arts and sciences (the largest college at fsu) and the university libraries, but there is also representation from the college of fine arts, college of music, and college of education. the remaining users (listed as “other”) are from departments and colleges including the college of human sciences, college of law, and other administrative units of the university. figure 3. count of users by college. the most popular content management system (cms) used by the createfsu community is far and away wordpress, with 83 total installations. omeka and omeka s taken together comprise 13 total installations. scalar has 6 installations across createfsu, and there are four other miscellaneous application installations. during the pilot phase of this project, each account is provided an initial 1gb of storage, with the ability to increase this storage for select power-users upon request and at the discretion of our team. at present, accounts are averaging about 423mbs of disk usage, with ten accounts exceeding 1gb of space. about 22% of accounts (23 total) are utilizing under 100mbs of space. in cases where there is low space usage, it usually indicates that the user has not begun building the site beyond perhaps installing a content management system. we had a high number of new account requests (16 in total) at the beginning of our academic year’s fall semester (late august and early september) when these most recent statistics were pulled, so we anticipate that the frequency of low-usage accounts will decrease as these most recent users begin working on their sites in earnest throughout the year. in addition to active users who are currently building sites, 381 members of the university community have logged on to the createfsu site to learn more about the service. as an account is generated by all signed-in fsu community users regardless of intent, we are uncertain if these users originally intended to apply for an account and were unsure of how to proceed or if they just logged in out of curiosity about the service. one of our strategies for increasing usage will be to reach out to these users and remind them about the service and how to use it. impact at the start of this project, we identified several quantitative and qualitative measures to evaluate the impact of the service, with a primary interest being to understand how the service was successfully meeting the needs of our campus partners. one of the reasons for demonstrating researcher uptake was to demonstrate the value of continuing the service to administration. many of the quantitative measures – the number of domains, number and breakdown of content management systems installed, number of pages created on each domain, number of user accounts, and usage statistics for completed websites, etc. provide a snapshot of awareness of the service across campus and have helped us to understand unmet needs across campus over the past few years. data demonstrating these metrics track with previous anecdotal feedback from collaborators about needing easy-to-use web hosting that originally spurred the launch of the hosting service in the first place. since launching the program in 2021, we have found that the more effective measures in demonstrating need and success, however, have been not the quantitative counts of accounts or pages, but rather qualitative feedback received from users of the service which explain the benefit of the libraries’ provision of easy-to-use web-hosting for the benefit of their research endeavors more broadly. as there are no other humanities computing services on campus, and the only other web-hosting provided by central university it services requires web editing software such as dreamweaver (and the knowledge and time to manage file uploads within that service), many users have reported liking the ease of use of our service in communicating about their research to new users. this is particularly the case when dealing with users that want to provide a simple web presence for a much larger and more complex research project. as with many r1 institutions, fsu is interested in strategically expanding research accomplishment across the university, and our service seems to offer a new way of connecting with public audiences (helpful for “broader impacts” in many grant applications) and to grow the broad awareness of ongoing projects on campus. testimonials from many of the researchers who participated in our incubator program illustrate how the service has impacted their work and enabled them to accomplish goals that would have been difficult to attain without it. for instance, a participant in a createfsu-focused digital project incubator hosted by drs remarked that the hosting service was “essential to the next phase” of a “longstanding digital project” which compiled social media and digital-native content. their companion website, digital exhibit, and web-publication which createfsu served to host were paramount to the continued success of the decade-long project. in this case, the createfsu service filled a gap in the researcher’s scholarly support needs and allowed for centering fsu’s role in a well-respected cross-institutional digital project. another researcher was able to extend the reach and scope of a physical exhibit that he created in collaboration with fsu’s museum of fine arts by using the createfsu service to lay the groundwork for a companion digital exhibition page that has a national reach. this researcher also highlighted how createfsu’s support infrastructure and mechanisms for guidance (through tickets, built-in support documentation, and community) made the project possible for himself and a team of new undergraduate and graduate collaborators in a way that was otherwise not supported by other mechanisms at our institution. these support systems do not always exist when faculty create sites on their own, and providing them institutionally through createfsu helps ensure that projects are feasible for users new to web publishing. challenges & opportunities one of the first challenges that we encountered during this process was the issues of funding. although the base cost of a dooo contract ($6,000 at time of writing) is very reasonable compared to the cost of many of the e-resources packages that libraries routinely license, the idea of libraries paying a vendor to provide web hosting as a service was new to most of our colleagues and was originally met with some trepidation. given that our libraries – like most academic libraries – have been experiencing flat or declining budgets for decades, it was understandable that advocating for the diversion of limited resources away from existing priorities in order to try something new would not be immediately popular. however, we have thankfully been able to document a long history of expressed need on campus which made the case for funding stronger and ultimately successful. even more than that, the rapid success of the program was such that, after the initial launch, positive user feedback and steady demand helped the team to successfully secure an additional 3-year renewal after the initial 3-year trial period. funding was just the first of many challenges, however. the next-largest challenge was related to staffing capacity. there is an immense amount of staff time required to implement and launch a service like this, including not just all of the work outlined above, but also the ongoing administration and maintenance. this entails responding to domain requests, onboard users into the system, ensuring that they have access to training and support documentation relevant to their chosen content management systems, and helping them troubleshoot issues that invariably come up and require outside assistance. in our case, we were fortunate to have three librarians who were able to devote a portion of their time to getting the service up and running – two of whom are responsible for ongoing management of the service and have been able to handle the demand to date. however, covering this new suite of service offerings meant pulling capacity from their other duties – something that had to be negotiated with a wide variety of stakeholders including academic departments and internal committees. we were also fortunate that the two librarians tasked with ongoing management of the service already possessed advanced technical skills that put them in a good position to learn about the intricacies of the platform and troubleshoot issues as they arose. that said, should the service become significantly more popular than it is currently – which is likely if we expand the allowable use cases – it may outstrip our current capacity. the time commitment required of users to claim and populate their domains with content is another notable challenge. in the case of faculty, this challenge is well documented in the literature on labor in the digital humanities (vinopal and mccormick 2013). faculty across the disciplines are faced with a multitude of competing priorities that demand their time and attention, and, as a result, it is typically very difficult for these users to find additional time to take on work related to digital content creation. this problem is exacerbated by the nature of reward systems in higher education, and particularly guidelines and expectations around review, promotion, and tenure, which tend to value work that produces more traditional research outputs such as journal articles, books, and grant proposals. the incubator program discussed above is one example of a strategy that can help to mitigate this challenge by providing faculty with structured support and some measure of recognition for their efforts, but the challenge remains a redoubtable one for most faculty. time commitment is also a challenge for students, particularly in cases where their work on a domain and related content is not integrated with curricular or academic program requirements. in such cases, students need to carve extra time out of their already busy schedules to devote to crafting additional content, and this can be exceedingly difficult for students who are already enrolled in a large number of courses and/or have significant competing demands on their time related to employment and familial duties. and though this set of challenges is real and significant, it most directly impacts our provision of this service through the fact that our success or failure can be seen in the number and quality of sites published using the service. when that metric is out of library control, and to an extent even beyond the personal control of our service partners (being as it is a structural and systemic issue), determining success based on content creation is a challenge. one promising opportunity for growth that manages to address some of the challenges above is the provision of web-hosting for curricular student projects. there are several units on campus that either currently use or are exploring the use of eportfolios for undergraduate work. we believe that createfsu could be useful for some or all of these eportfolio use cases. eportfolios are a particularly good fit for the service because they are relatively short-term projects based on a relatively small number of individual works comprising a thematically recognizable thematic project. the students are also extrinsically motivated to complete these projects, which helps drive completion. however, the current size of these programs and the large number of individual students that participate currently prohibit them from being included in createfsu based on our current level of support. one unit alone (such as the editing, writing, and media program housed in the department of english) could single handedly fill the 500 account limit within two years. in order to manage resources such as this account limit, we need to find ways of testing eportfolio use-cases at a smaller scale without excluding other students in the cohort. testing like this is necessary for determining if createfsu would serve this need at scale, but remains a promising glimpse at what the future of the service could look like at expanded levels. conclusion and looking ahead although the first two years of the createfsu pilot was successful, there is much work that remains to be done. specifically, our team would like to do more to promote the service to the fsu community and enlist the support of campus partners whose missions could benefit from broader use of the service. one ongoing goal that we have is to build more robust support resources for createfsu users. we are currently in the process of enhancing support documentation and resources to better answer users’ needs and questions without relying on one-on-one consultation time with our team. we are also exploring one-to-many support options such as workshops and group training sessions, templates for popular use cases, and short video tutorials that guide users through the process of installing applications on their domains and using their chosen applications to achieve their goals. based on interest in and use of the service thus far, we would also like to more sustainably broaden the allowable use cases for the service to include lab websites, funded projects, and e-portfolios, to better meet needs across campus. we believe that allowing these use cases would significantly increase engagement with the service and, with a bit of conscientious planning, are confident that we have the capacity to support the additional projects. our team is also seeking additional resources to support the service including funding from campus partners. this additional support will be extremely useful in the future if we wish to accommodate the new use cases mentioned above, providing not only some overhead for supporting the labor demands, but also financially allowing us to expand the service beyond the 500 accounts that are included in the initial rollout. marketing is another ongoing priority for a service like this which we would like to expand. like all universities, fsu welcomes well over a hundred new academic staff and thousands of new students each year, and the only way to inform these new members of our community about the service is through ongoing, targeted marketing and promotion. finally, our team is developing a survey to gauge overall satisfaction with the service, the technical support that we provide, and the selection of applications available through the platform. we will also use this survey to assess how many outputs are finished vs. unfinished, active vs. inactive, and faculty-created vs. student-created, as well as the distribution of content management systems selected by users. although we still have many goals to accomplish, we believe that the first two years of this project have been a success and have made a positive impact in the lives of many creators at our university. references haimes-korn k, greene j, rorabaugh p. 2022. cultivating a grassroots approach to digital literacies via domain of one’s own. in: multimodal composition faculty development programs and institutional change. [place unknown]: routledge; p. 241–258. hootman j. 2020. createuk: opportunities for digital pedagogy, projects, and collaborative infrastructure [internet]. in: [place unknown]; [accessed 2023 sep 11]. https://works.bepress.com/jlhootman/27/ hootman j. 2022. createuk: a review and recommendations for growing and enhancing campus collaborations [internet]. in: [place unknown]; [accessed 2023 sep 11]. https://works.bepress.com/jlhootman/34/ kehoe a, goudzwaard m. 2015. eportfolios, badges, and the whole digital self: how evidence-based learning pedagogies and technologies can support integrative learning and identity development. theory into practice. 54(4):343–351. https://doi.org/10.1080/00405841.2015.1077628 o’byrne wi, pytash ke. 2017. becoming literate digitally in a digitally literate environment of their own. journal of adolescent & adult literacy. 60(5):499–504. https://doi.org/10.1002/jaal.595 taub-pervizpour l, clarke t. 2017. cultivating open blended learning with domain of one’s own [internet]. [accessed 2023 jun 14]. https://repository.brynmawr.edu/blended_learning/2017/2017/10 vinopal j, mccormick m. 2013. supporting digital scholarship in research libraries: scalability and sustainability. journal of library administration. 53(1):27–42. https://doi.org/10.1080/01930826.2013.756689 winchester s, freeman a, boyd k. 2021. from the ground up: building a digital scholarship program at the university of south carolina. south carolina libraries [internet]. 5(1). https://doi.org/10.51221/suc.scl.2021.5.1.5 notes [1] instructors may create one domain per course and enable their students to register for author accounts on that domain. this use case is suited to assignments for which students engage in public writing (e.g., course blog), collaborative curation projects (e.g., omeka projects), or other digital pedagogy assignments. about the authors matthew e. hunter (orcid: 0000-0003-4858-6947) is the digital scholarship librarian at florida state university libraries. devin soper (orcid: 0000-0002-2667-4594) is the director of the office of digital research & scholarship at florida state university libraries. sarah stanley (orcid: 0000-0001-6089-3910) is the former digital humanities librarian at florida state university. she is currently pursuing an master of arts in teaching at moravian university. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – between the sheets: a library-wide inventory with google mission editorial committee process and structure code4lib issue 38, 2017-10-18 between the sheets: a library-wide inventory with google when it comes to taking an inventory of physical items, libraries often rely on their traditional integrated library system’s (ils) à la carte add ons; outside vendors; or other possibly outdated, complex, and often expensive methods. for libraries with shrinking budgets and other limited resources, high costs can put these methods out of reach. at the university of dayton libraries, we set out to develop an inexpensive and reasonably easy-to-use method for conducting a library-wide physical item inventory. in this article, we explain a custom built google sheets-based library inventory system, along with some code for the implementation of a restful api (written in php) that interacts with our ils. we will also explain our use of google apps scripts in our google sheet, which are crucial to our systems. although this method used a specific ils (innovative interfaces’ sierra product) and custom-built restful apis, it may be possible to use similar approaches with other ils software. additional notes include areas for improvement and recommendations for interoperability with other ils systems. by craig boman and ray voelker conducting a library-wide inventory with google apps in the summer of 2015, the university of dayton roesch library needed to scan, reshelve, reorder, or restore item statuses for all of the books in our circulating library collection. the previous library-wide inventory was conducted the summer of 2008 or “that summer when the library air conditioning went out.” at that time, our predecessors used a portable percon barcode scanner and barcode data was imported through the ils inventory module (then iii’s millennium, now sierra). even when the shelves were scanned in the right order and the percon battery did not die, finding a computer with a serial port to get the data off of it was not easy. although there are certainly better upgrades available to older percon scanners, in 2015 the library did not have the budget to buy lots of devices for an inventory. in our department (information systems and digital access), we did, however, have spare usb barcode scanners, some spare laptops, and a handful of student workers we could hire. to be entirely honest, we started our 2015 inventory with the question, “what if we use google sheets for our library inventory?” discussions of why and how you should do a physical library inventory (sung, whisler, & sung, 2009) and how you should avoid doing a physical library inventory (loesch, 2011) all served as evidence which encouraged us to further explore alternatives (like sheets) for our new inventory. we made a decision to explore a google sheets-based inventory based on the flexibility it would provide to create better inventory data than most vendor-supplied inventory products; products which we, and many other libraries, cannot afford. this article will not be a roi study but more of a case study of the tools and methods we used to conduct our own physical book inventory. it also should be clear that we are not advocating this as the best way to conduct a library inventory; this is simply a method that we chose to use at the time, and one that worked reasonably well for our defined goals. identifying project needs and requirements when we began thinking about doing a library-wide inventory of the university of dayton libraries’ public collections we naturally started examining the tools that were available to us in the integrated library system (ils) in use at our institution. at the time of writing this article, that system was sierra ils. thanks to the 2008 inventory, we had some documentation from that process to work with. as we examined the documentation, we were quickly able to identify several problem areas with the methods and tools used for that process. while we were both with the library during the previous inventory, we were in different roles at the time and therefore not involved with the inventory planning process, so we were excited to take an ownership role in this 2015 inventory. we wanted to get started with the process of streamlining the inventory to make it more efficient right away. the first thing that we noted from our previous inventory was that the method was slow, laborious, and prone to scanning errors and data loss. the percon barcode scanners used had to be of a very specific type, and were, for all intents and purposes, pretty dumb devices. first, they had no way of telling the user the last series of barcodes scanned. if the person scanning got distracted and lost their place scanning, they had no way of knowing where to pick back up with scanning on the shelf. another downside was the small amount of memory built in for storing the scanned barcodes. this meant that the data had to be “off-loaded” from the device frequently, often while scanning the middle of a range or some other inopportune time. the task of moving the inventory data from the device to the ils system had to be done by a staff member—one with full permissions to edit and update records—and had to be done at a workstation that had the ils system client installed. this meant that capable staff had to be available at the time any scanning was taking place to perform these importing and updating functions. student workers had to sit idly by and wait for these tasks to be completed before they could get back to their inventory scanning work. on top of these severe limitations, the scanners were unreliable; on several occasions—either through some user error or malfunction of the scanner itself—entire scanning sessions had to be repeated, wasting time and effort. we wanted something better, and we were determined to develop the tools and methods to make it so. after examining the previous methods used for conducting an inventory, we concluded that our needs could be broken down into the following three categories : accuracy/reliability: if what is on the shelf at the time of scanning is not accurately and consistently captured and recorded, then what was the point of even doing an inventory? we also wanted to minimize any chances of data loss and corruption. additionally, we wanted to be able to quickly identify any problems with the scanning process—in real time—if at all possible. speed: time is a precious commodity, and we wanted to perform this inventory as quickly as we possibly could. since this was an academic institution, carrying out our work over the course of a summer and finishing up before the start of the fall term was ideal. ease of use: we wanted to un-complicate the process as much as we possibly could for everyone involved. for the individuals doing the scanning, it should be easy to do the scanning, see what was scanned, and know what still needs to be scanned. also, the individuals doing the scanning should be able to do so with almost zero training. from the perspective of the library employees managing the inventory, it should be easy to assign work, examine the accuracy of that work, and then finally act upon that inventory data. in order to successfully capture, examine, and act upon the inventory data, we felt that we needed to answer two very important questions: first, how could we connect to the sierra ils to get synchronous data out during our scanning process, and secondly, what was to be captured from the ils for each scanned barcode? the how in answering the first question, how, we knew that the sierra ils conveniently provides a method of connecting to the back-end database by way of a feature called, “sierra direct sql access”. this feature provides read-only access to the database tables by way of database table “views”, which expose the live catalog data running on sierra’s postgresql database. while this data is read-only, it still provides us with a great way to extract what we need from bibliographic and item record data associated with a barcode in real-time. the real-time aspect is important if we are to consider the inventory process to be a “snapshot” of the state of the shelf at the moment in time that we examined or inventoried it. since the database server runs the popular postgresql object-relational database management system, many different software libraries for connecting to this type of database exist for different programming languages. this gave us a great deal of flexibility in selecting a programming language to script and process the queries we needed to perform on the database. we picked the php language as we had a great deal of background with the language, having done other development projects it. the second step after getting the data out of the database, is getting the data into the spreadsheet. as mentioned previously, google sheets can be augmented with google apps scripts. this makes it possible to create “triggered” events that can call additional custom functions to make external api calls; our custom php restful api is one such api that, when called, queries the database and returns formatted results that can be processed by google apps scripts. we will discuss more specifics on these two scripts and how they work together a little bit further into the article. the what as for the second question, what, we determined that the following data about the item would be essential and should be recorded when each item was scanned: item barcode normalized call number item location code item status code check out status (date due) with that data, we are able to identify and address several problems rather quickly. first, by checking item location code we were able to identify books that did not belong in the section, or did belong but had incorrect information in the catalog. some of the item record data was fixed as a result of this discovery. with the status code we were able to flag items that have any status code other than “-”, which in sierra indicates that there is some other status other than available. items on the shelf should all have an available status. after a short period of testing, we later determined that some of the books on the shelves were still checked out; checking the check out status proved to be essential as part of this process as well. with normalized call number, we are able to sort the sheet either ascending or descending to determine the correct shelf order of the items scanned and to pull items that did not belong in the range. it is worth mentioning that using unformatted (or non-normalized) library of congress (lc) call numbers in a spreadsheet is problematic because they do not follow normal lexicographical sorting methods used in spreadsheet applications like google sheets. in order to make lc call numbers alphabetically sortable, the call numbers must be normalized. normally, sierra performs such a normalization—storing the value in the database for easy retrieval—but there are certain circumstances where this may not be the case—when retrieving call numbers directly from marc fields for example. a possible solution to this problem is to incorporate a function that filters the call number data, normalizes it, and then returns the normalized form to our application. our normalization function is available for those in need of such a feature. the following additional data about items were also identified as being helpful to the inventory process: volume number item record number title shelf position date scanned sheet name (or range number) volume, item record number, and title come from the database. shelf position, date scanned, and sheet name are produced by the spreadsheet itself. the entire query we developed to return all this data can be found here: https://github.com/rayvoelker/2015roeschlibraryinventory/blob/master/sql/inventory_barcode_query.sql implementation and scripts server considerations we decided to write the script to fetch the data from the database using php and that the script should act as a restful api endpoint. this decision made it much easier to write the google apps script as it has simple functions built-in for interacting with restful api endpoints. and since this api is restful, that meant that we needed a web server to host and execute the php. for that purpose, we chose linux (ubuntu server), the apache http server, and of course php, installed as an apache server module. (if you have a domain certificate in place, ports 80, or 443 will need to be open to the internet so that google services can interact with the endpoint. google has more information on this requirement.) restful api endpoint we picked a restful architecture for our method of developing the script for a number of reasons. first, the architecture makes it easy to test and use the script. secondly, this is ultimately a web application, and as such, certain components of the application (google apps scripts for example), are designed to easily interact with restful endpoints. as a quick example to demonstrate how this restful api functions, and especially for those not familiar with the method: if we were to scan the barcode of an item on the shelf, we would send the following http get request to the api. since the request is a standard get request with no authentication, it is easy to test this in a web browser. http://127.0.0.1:8080/?barcode=35054031024744 figure 1. an example of a restful api http get request the output of this get request is in the json format, and would look similar to figure 2. { "barcode":"35054031024744", "item_record_num":"i9233953a", "item_record_id": 450976172208, "call_number_norm": "pz 7 g1273 gr 2008", "call_number_050": "pz 7 g1273 gr 2008", "volume": null, "location_code": "mdju", "item_status_code": "-", "best_title": "the graveyard book", "due_gmt": null, "inventory_gmt": null } figure 2. an example of the json response produced from the php-based restful api http get request a few things to note that are important about that script before we move on: first, we want to “sanitize” the barcode input to be searched based on the type of barcode that is used at the institution. this is as simple as limiting the length of input as well as the type of characters accepted as the “barcode” argument in the http get request. if implementing this for another institution, we advise modifying the code to accept barcodes that fit the description of those in use. the practice of input sanitization is much more important when making updates—making modifications to a database or inserting new items for example—but is still a good habit to maintain. additionally, you may want to change the query for the non-normalized call number (call_number_050 in the sql query, and json results) to reflect local cataloging practices or to get the non-indexed call number. some sierra ils sites have reported that the call number may not appear in the “call_number_norm” field; a support call to the ils vendor may be required to remedy that. google apps the majority of the inventory was centered around google sheets and conveniently, our institution had recently transitioned to using g suite for education (formerly called google apps for education), which included this easy-to-use spreadsheet tool. using google sheets offered several benefits, including access for every student, faculty and staff member from their university account and easy sharing options. this made assigning spreadsheets to student workers a quick and easy task for the inventory coordinator. lastly—and perhaps most importantly—google apps scripts effectively allowed us to construct our inventory application almost entirely from within the google sheets tool—pretty powerful stuff! as an added bonus, google sheets works well in “offline” mode. this is important, since taking a laptop and usb scanner into the stacks and then losing wifi access still means that you will be able to complete the task of scanning barcodes into the google sheet. data placed into the spreadsheet are automatically synced to the cloud when network access is restored. unfortunately, this requires an additional manual step of having to run a command from the google sheet that fetches the additional information from the ils after re-establishing a network connection. luckily, that task is a relatively simple one, especially when compared to the re-scanning that the previous inventory method often required. the “cloud-based” nature of google sheets is also a positive as it helps to reduce the possibility of data loss as well as data corruption. google apps script google apps script, with its various custom functions, is the method that “triggers” the import of bibliographic and item data on each barcode scanned into google sheets. below we have crafted a simplified example of the “onedit()” trigger method that is assigned to a google sheet to demonstrate how we handle importing data. function onedit(e) { try { var value = e.range.getvalue(); var url = 'http://domain.edu/inventory_api/v1.0/barcode.php?barcode=' + value; // populate column b with the api call e.range.offset(0,1).setvalue('=\"' + url + '\"'); // make the api call, and then parse the results into a json object var result = urlfetchapp.fetch(url); var json_data = json.parse(result.getcontenttext()); // populate column c with the title (replace double-quotes with escaped versions) e.range.offset(0,2).setvalue('=\"' + json_data.best_title.replace(/"/g, '""') + '\"'); } // end try catch(e) { // todo: // something more to catch errors } // end catch } figure 3. google apps script for the google sheets onedit trigger in this simplified example, the google apps script reads the value from the cell when data is placed there, forms the url string (appending that cell value as the argument), then passes it to the built-in apps script service urlfetchapp. this service then interfaces with our php restful api operating on our web server which produces the json response. the data is then parsed into the “json_data” variable where part of it (“json_data.best_title”) is then finally output back to the spreadsheet. when scanning a single barcode (r008818288 in the example) into the sheet (with the above trigger enabled for the sheet), we would see the two cells (under column b and column c) populated with data as demonstrated in figure 4. figure 4. an example of data output to a row in google sheets figure 5. example of scanning process 2015 roesch library inventory : scanning in addition to the onedit method discussed above, we also used the google apps script to perform a handful of other important inventory functions as well. these functions included a method of “fixing” missing column data (for when the application was operating in “offline mode” and not collecting additional bibliographic and item data from the external api). this method was often used by the staff member after the student submitted the finished sheet of scanned barcodes, by selecting this function from a special inventory menu from within the sheet itself, as illustrated in figure 6. figure 6. an example of the menu produced by the custom google apps script a second custom function, called “produce reshelve sheet” (seen as a menu option in figure 6) would produce an additional sheet in this spreadsheet based on two sets of data contained in separate sheets: the inventory “snapshot” and the inventory “shelflist”. the “snapshot” consists of our scanned item information, and the “shelflist” consists of a list of items we expect to be on the shelf. that is to say, given the item location code and the start and end normalized call numbers we produce an sql query (https://github.com/rayvoelker/2015roeschlibraryinventory/blob/master/sql/create_shelflist.sql), importing the results into the spreadsheet. once that data is imported into the sheet and scanning on the main inventory sheet is completed, running the “produce reshelve sheet” function would take these two sets of data and perform something similar to an sql join, matching the items from “shelflist” (left) with the items from the “snapshot” (right). this will tell us what items we found in our shelflist or items we expect to be on the shelf, as well as which items we did not find on our scanned shelf. figure 7 should help to further illustrate the concept. the overlap in orange in the second image represents the items appearing in both lists and therefore “found”. figure 7. an example of the left join performed by the script to help users determine what items should and should not be on the shelf figure 8. creating and sorting the reshelf sheet lastly, there is an additional custom function that can be run from google sheets called “check sort order”. the intended use of this function is to create visual indications on the “reshelve” sheet when items are on the shelf in incorrect shelf order based upon the item’s assigned call number (a demonstration of this can be found in the same video in figure 8, starting at the 30 second mark. the full google app script that we used in this project is included at the end of this article. for more information on using google apps script and setting up triggers please see the google apps script documentation. extending google sheets and getting started with the web server set up, the types of data to pull identified, and the google apps script ready, we were on our way to getting this inventory started. soon there would be large amounts of inventory data (this is sort of the point though, is it not?). again, google sheets shines in its ability to organize; we were able to maximize features available to us through google sheets to help us extend the use of the application to organize all of our scanned inventory data. google sheets has seemingly endless methods of transporting inventory spreadsheet data between different sheets. every google sheet has a sheet id in the url string, evident in figure 9 between the final set of forward slashes: https://docs.google.com/spreadsheets/d/1njdq06zfv0jaz5bjwylzumrra44w4ildnqp6orlplm8/ figure 9. an example of the google sheet id, and the method of linking to it using this sheet id, we built a google spreadsheet-based inventory data dashboard which aggregates metadata from all of the google sheet ranges currently being scanned into one spreadsheet, creating a simple user interface in the process. in our inventory dashboard, we were able to assign ranges to students in one column, and note when the range scanning started and ended in another column. additionally, using sheet ids, we could view the total number of books we anticipated in a range (based on books between the start and end call number range pulled from our ils), and compare that against the actual number of scanned books on the range. while the ranges were scanned, we also counted item statuses that were anything other than available (“-” in our case). the moment the range had been completely scanned, we had a count or report of exactly how many books needed attention, and the specific types of statuses encountered in that range. early versions of our inventory dashboard were simple. through various iterations, our dashboard developed into what it is today: a comprehensive and practical at-a-glance workflow, management, and reporting tool (figure 10). figure 10. google sheets-based dashboard for location “rc6” in figure 10, you can see one of the formulas which is using the sheet id stored in cell t2 to count the total rows of data in column n from the sheet named “inventory”. at a glance, we can see that the total number of scanned books in range rc6-01 has 3436 books, but we were expecting 3571 books. in some cases you may end up with more books in the range than you expected, for various reasons. continuing into the spreadsheet (figure 11), we can see more specific information about the range rc6-01 starting in column j: figure 11. google sheets-based inventory dashboard for location “rc6”; additional columns shown in figure 11, we can see from column l and the formula displayed at the top of the image we are again using our sheet ids stored in column t in the dashboard to count the total number of books with a status of missing or “m” in column e of the sheet called “inventory”. similarly, all columns on this dashboard between j-q are using some combination of a spreadsheet function, countif(importrange()), to create reports about our inventory data while we are scanning, without delay or manual intervention. we can then use these data to justify to our stakeholders the value of our inventory by showing that we pulled a specific number of books from each range for various reasons including: items had bad status codes, were not in correct locations, etc. google sheets limitations using our inventory dashboard to manage inventory workflows and create post-inventory reports is not without some limitations. to allow one google spreadsheet to import data, or in any way access data from another google spreadsheet, google requires spreadsheet users to grant permission to access data stored in another google spreadsheet. there may be a way to script this permission-granting using google app scripting, but at the time of our inventory we were manually interacting with a row which contains an importrange() formula to permit it to access another spreadsheet’s data. for other cynical thoughts about google’s use of this permission feature, see also: any google spreadsheet support forum. additionally, when we conducted our inventory, we manually created every google spreadsheet for the project. we had 150 ranges (3000–5000 books each) for which we needed to create a spreadsheet for each range, and put that sheet id into our dashboard to pull in data from those inventory spreadsheets. in our case, with a brief bit of instruction, our fantastic student employees were quickly trained to make a copy of a previous range’s spreadsheet they had just finished and use it to continue scanning the next range. a solution to consider for next time would be to create a google apps script that would create a new spreadsheet and set it up automatically. getting student workers involved in the course of a library inventory, it may be important to augment the actions of library staff with help from library student employees. when training library students it is important to anticipate the minimum level of knowledge necessary to successfully scan and reshelve books. below are some recommendations for training students, in case it is helpful: basic call number training–without this training, books may be pulled and reshelved again in similarly incorrect locations, making a follow-up inventory necessary. in our case, we had the advantage of being able to use circulation students already familiar with re-shelving shelf orientation introduction– even after training students, within the first hour we observed a student scanning books in the wrong shelf-order into the spreadsheet. quickly, we were able to take corrective action, which potentially saved us large amounts of additional work. inventory progress running a library wide inventory is a major project. at our peak, we managed four student employees over three months working thirty-five hours each on average, or 140 total hours per week. coordinators of inventories should not underestimate the amount of work that will go into an inventory or the variability of student worker progress speeds. the transparency of google sheets may help mitigate this variability by encouraging slower students to emulate the pace of their peers as spreadsheets are populated. post-inventory resolving bad statuses just because your library is done scanning books in a range, does not mean you are done with your library inventory. after a range of books has been scanned, some information in our ils had to be updated starting with the incorrect item record statuses. any item found to have a bad status—one without a status of “-”—was routed to the correct staff in the library who was best suited to resolve the issue. in some cases, no record was found in our ils because we had deleted the bibliographic or item record many years prior, requiring additional cataloging. other status issues included incorrectly shelved consortium books and books requiring reshelving in the correct order. getting data back into the sierra ils another large part of our post-inventory processes included importing the scanned barcodes into our ils and applying an inventory note. this inventory note–applied as a global update–included where the item was scanned last (which was also the range’s google spreadsheet title). we also globally updated the inventory date on our item records, so we know when the book was last inventoried. an added benefit of the inventory note field is that if, for whatever reason, the book was not pulled from the shelf during our inventory, all library staff had an idea of where the book was last inventoried and could reference the google sheet to find the location in the range where the book was scanned. as an innovative sierra library, getting our inventory data into sierra was as simple as importing a list of item record numbers as a review file. we created a php-based web tool to convert our list of scanned barcodes, range by range, into item record numbers. this web application can be found in our git repository. the web tool queries sierra ils and returns the list of item record numbers. in future iterations, we will pull in item record numbers and corresponding bibliographic record numbers for each book scanned in a spreadsheet. there has been some discussion of sierra allowing for the importing of barcodes directly as a review file in future release updates, but this is not possible in sierra v3.1, which is the latest version of the software at the time this article was published. communication concerns similar to the conclusions reached by kohl, bénaud, and bordeianu (2017), communication will always be an issue in library inventory projects. in our case, we did notice that although we were communicating well with staff in other areas, the impact of the inventory was not always communicated well to student employees who were responsible for shelving books. more information could have been communicated by the inventory coordinator to student employees directly, informing them that their reshelving work would increase dramatically during the inventory. additionally, a primary goal of our inventory was to communicate our progress and our findings to stakeholders before, during, and after the inventory. even though this should be obvious, we needed to collaborate closely with our library technical services and access services (circulation) to process changes to our books. overall, communicating our progress to our library community helped the overall project go more smoothly, and avoided stepping on any metaphorical toes. conclusions although our method worked reasonably well for our staff and the needs of the university of dayton libraries, the needs of other libraries may be different. having said that, we have presented a method which should be easily transported between libraries. we hope to encourage other libraries, regardless of ils vendor, to consider the benefits described when using google sheets for a library inventory. as libraries continue to share methods like these, we can all benefit from local solutions moving into collaborative, open source projects. if you would like to collaborate on improving our processes, please reach out to us. as promised, the scripts used in this project, as well as a short presentation that was prepared for oh-iug in 2015 can be found in the following github repository. videos demonstrating this process can also be found in the internet archive. thanks and acknowledgements we would like to thank the university of dayton roesch library and our supervisor at the time of this inventory, fran rice, for allowing us the opportunity to develop this method, and carry it out. ray voelker would like to thank everyone at roesch library who had to put up with him during the development of this project. also we would like to thank all of the student workers whose patience with us helped make this project possible; they did all the heavy lifting too. about the authors craig boman (bomanca@miamioh.edu) is an assistant librarian and discovery services librarian at miami university libraries. previously he was at the university of dayton libraries for 7 years, most recently as the application support specialist. additionally he is a ph.d student in higher education leadership at the university of dayton. read more at his website: https://craigboman.github.io/ ray voelker (ray.voelker@gmail.com) is currently the ils developer and system administrator for the public library of cincinnati and hamilton county. he was previously with the university of dayton roesch library for nearly 17 years, where he more recently served a role as application developer. https://github.com/rayvoelker references kohl, l., bénaud, c., & bordeianu, s. (2017). finding shelf space in an academic library: a multifaceted approach. technical services quarterly, 34(3), 268-282. doi:10.1080/07317131.2017.1321378 loesch, m. f. (2011). inventory redux: a twenty-first century adaptation. technical services quarterly, 28(3), 301-311. doi:10.1080/07317131.2011.571636 sung, j. s., whisler, j. a., & sung, n. (2009). a cost-benefit analysis of a collections inventory project: a statistical analysis of inventory data from a medium-sized academic library. the journal of academic librarianship, 35(4), 314–323. https://doi.org/10.1016/j.acalib.2009.04.002 subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – sierradna – demonstrating the usefulness of direct ils database access mission editorial committee process and structure code4lib issue 30, 2015-10-15 sierradna – demonstrating the usefulness of direct ils database access innovative interface’s sierra(™) integrated library system (ils) brings with it a database navigator application (sierradna) – in layman’s terms sierradna gives sierra sites read access to their ils database. unlike the closed use cases produced by vendor supplied apis, which restrict libraries to limited development opportunities, sierradna enables sites to publish their own apis and scripts based upon custom sql code to meet their own needs and those of their users and processes. in this article we give examples showing how sierradna can be utilized to improve library services. we highlight three example use cases which have benefited our users, enhanced online security and improved our back office processes. in the first use case we employ user access data from our electronic resources proxy server (wam) to detect hacked user accounts. three scripts are used in conjunction to flag user accounts which are being hijacked to systematically steal content from our electronic resource provider’s websites. in the second we utilize the reading histories of our users to augment our search experience with an amazon style “people who borrowed this book also borrowed…these books” feature. two scripts are used together to determine which other items were borrowed by borrowers of the item currently of interest. and lastly, we use item holds data to improve our acquisitions workflow through an automated demand based ordering process. our explanation and sql code should be of direct use for adoption or as examples for other sierra customers willing to exploit their ils data in similar ways, but the principles may also be useful to non-sierra sites that also wish to enhancement security, user services or improve back office processes. by james padgett and jonathan hooper background in 2012, innovative released a major upgrade to their long established millennium library management system (lms). the sierra integrated library system (ils) [1] comes with a feature named sierra database navigator application (sierradna)[2] – giving sierra customers direct read-only access to their ils data through a postgresql database. postgresql[3] is a modern object-relational database management system allowing easy access to data via the structured query language (sql)[4]. whether directly via a postgresql client such as pgadmin[5] or as we have done, through a php postgresql connector, custom sql can be executed against virtually the entire gamut of ils data. from acquisitions to circulation; reading histories to electronic resource access attempts – library data is available to be exploited. our own use of sierradna focuses on the creation of restful apis based on the tonic php framework[6] and php scripts executed through the cron time-based job scheduler[7]. thus we highlight our experiences with sierradna through three of our favorite use cases. sierradna sierradna is actually a simple concept represented by two capabilities – (1) it provides a mechanism to connect a postgresql client to the sierra database to gain read-only access to the data and (2) the provision of 349 sql views so that queries may be run safely without fear of overwriting your data. the views represent most aspects of the ils system, from bibliographic[8] and item level data to circulation transactions to reading history. each view is constructed from one or more tables and can be queried using sql as if it were a table, through postgresql client software. sierradna is not an api, or a dbms layer – it is a direct connection to a structured set of sql views of the sierra ils database. detecting compromised user accounts academic libraries face the same intrusion threats as any other sector; their users vulnerable to the same phishing attacks[9]. the purpose of such attacks is to capture username / password pairs which can be subsequently used to exploit corporate it systems. in the case of the university of leeds, compromised accounts are commonly used to circumvent the libraries’ proxy to gain unauthorized access to 3rd party eresources. at leeds, the first indication the library had of this activity was a notification the eresources coordinator received from an eresource publisher. they highlighted, with examples, several episodes of suspicious activity originating from our proxy server. upon examination of the library’s proxy server web access management (wam) access logs it was clear this activity was systematic and it had been going on for some time. in a typical attack (measured over a 24 hour period) multiple ip addresses (35+), on several continents were being used to machine read content from over 60000 urls across more than 155 publishers. having identified the problem, the library turned to sierradna to help with a solution. the wamreport view within sierradna records each wam access attempt, whether successful or not, capturing the following information: wamreport ("id", "logged_gmt" "requesting_ip" "requesting_port", "requesting_iii_name", "dest_port", "dest_code", "response_category_code_num", "patron_record_metadata_id", "ptype_code_num", "pcode1", "pcode2", "pcode3_code_num", "pcode4_code_num", "rejection_reason_code_num") this view combined with three sql queries have been created to highlight potentially suspicious activity. the first counts the number of ip addresses used by a user within a 24 hour window. the second performs a similar task for the number of different resource providers accessed. the last counts the number of individual url access attempts made over the same period. the first query (script 1) consists of an inner (line 3 – 8 inclusive) and an outer query and generates a list of users who accessed e-resources from more than 2 ip addresses in the previous 24 hours. line 3 of the inner query selects the patron number of the user and the ip address the request originated from and joins the patron_view table – line 4 specifies the columns on which the join is performed. on line 5 we limit our search of requests to those within the last day – to standardize the query timeframe – but in theory this could be shorter if one wanted to generate reports more frequently. the group by clause on line 6 allows our script to count the connection attempts by patron number / ip pairing. on line 7 we specify that we want to list all connections attempts; ordering by patron number (line 8). the outer query groups the results from the inner query by patron number, returning all those who connected using more than 2 ips. 1 select w.record_num, count(*) from 2 ( 3 select pv.record_num,wr.requesting_ip, count(*) from sierra_view.wamreport wr, sierra_view.patron_view pv 4 where wr.patron_record_metadata_id = pv.id 5 and wr.logged_gmt::date >= now()::date 1 6 group by pv.record_num,wr.requesting_ip 7 having count(*) >= 1 8 order by pv.record_num asc 9 ) w 10 group by w.record_num 11 having count(*) > 2 12 order by count(*) desc; script 1 the second query (script 2) consists of an inner (line 3 – 8 inclusive) and an outer query and generates a list of users who accessed more than 8 eresource publishers. line 3 of the inner query selects the patron number of the user and the publisher the request is accessing and joins the patron_view table – line 4 specifies the columns on which the join is performed. on line 5 we limit our search of requests to those within the last 24 hours – to standardize the query timeframe. the group by clause on line 6 allows our script to count the access attempts by user / publisher pairing. on line 7 we specify that we want to list all access attempts; ordering by patron number (line 8). the outer query groups the results from the inner query by patron number, returning all those who accessed more than 8 publishers. 1 select w.record_num, count(*) from 2 ( 3 select pv.record_num, wr.dest_code, count(*) from sierra_view.wamreport wr, sierra_view.patron_view pv 4 where wr.patron_record_metadata_id = pv.id 5 and wr.logged_gmt::date >= now()::date 1 6 group by pv.record_num, wr.dest_code 7 having count(*) >= 1 8 order by pv.record_num asc 9 ) w 10 group by w.record_num 11 having count(*) > 8 12 order by count(*) desc; script 2 the third query (script 3) lists users who accessed more than 500 e-resource urls. 1 select pv.record_num,count(*) from sierra_view.wamreport wr, sierra_view.patron_view pv 2 where wr.patron_record_metadata_id = pv.id 3 and wr.logged_gmt::date >= now()::date 1 4 group by pv.record_num 5 having count(*) > 400 6 order by count desc; script 3 line 1 selects the patron number and a count of the number of eresource urls accessed and joins the patron_view table – line 2 specifies the columns on which the join is performed. as in the previous two scripts we limit our search to the last 24 hours (line 3). the group by clause allows a count of url access attempts to be made. each of the 3 queries produces a list of users identified by their patron number. the lists are combined into a daily report – users appearing in more than one list are highlighted for our eresource team, who receive the report each morning. the daily report has taught us that a typical compromised account will be represented in each list; an account being used to systematically steal eresource content will have a considerable count in the second and third scripts; an account used to machine read content will often have used a large number of ip addresses. since its introduction, the report has often been the first indication that a university account has been compromised. in all cases we have identified the compromised account before receiving notification from the eresource publisher. indeed, we proactively report suspicious activity to the publishers, explaining the actions taken to prevent further theft. people who borrowed… index based discovery systems have in recent years allowed for an explosion of searchable content (breeding 2015). relevancy-based retrieval is ever more important to users who want best match items on the first page. however, what if there was a different way of searching; a more serendipitous way, one in which you can stumble upon that key item you may have otherwise overlooked. sierradna has allowed us to offer a “people who borrowed this, also borrowed…” feature – presented as a carousel at the bottom of our record display screens on our opac – figure 1. the feature relies on the collection of reading history from the circulation activity of our users. sierra can be configured to collect reading history in a number of ways; our library uses the opt-out setting – allowing reading history to be collected automatically unless a user opts out of the feature (an option which is available to users through their library account). the collection of reading history data in this manner adheres to the university of leeds code of practice on data protection [10]. figure 1   reading histories are collected within the sierra database in the reading history table – the following information is captured. reading_history("id", "bib_record_metadata_id", "item_record_metadata_id", "patron_record_metadata_id", "checkout_gmt") each row has a primary key id, used to uniquely identify the loan, along with a bib_record_metadata_id identifying the bib record of the item borrowed, item_record_metadata_id identifying the actual item borrowed, patron_record_metadata_id identifying the user to which the loan was made and checkout_gmt identifying when the item was checked-out. the first script (script 4) generates a list of users who borrowed the item currently being viewed – in figure 1 this is a monograph with an entrepreneurship library of congress subject heading. on line 1 we select the distinct patron record number and join with the bib_view table – line 2 specifies the columns on which the join is performed. on line 3 we pass the bib record number of the item being viewed. 1 select distinct(rh.patron_record_metadata_id) from sierra_view.reading_history rh, sierra_view.bib_view bv 2 where rh.bib_record_metadata_id = bv.id 3 and bv.record_num = $bib_record_num; script 4 the second script (script 5) consists of an inner (line 3 – 14 inclusive) and an outer query and uses the list of users obtained in the first script to generate a list of items also borrowed amongst that group of users – excluding the item currently being viewed (line 5). line 3 of the inner query selects the bib title, isbn, bib record number, and a count representing the number of times the item has been borrowed amongst the group of users. we use a window function to rank the isbn number for the item – which is used in combination with line 16 from the outer query to return the highest value. bib records may contain multiple isbn fields and this approach guarantees we return only one. the isbn is needed only to provide a book jacket image from syndetics[11]. 1 select * from 2 ( 3 select count(bv.title) as count, bv.title, s.content as isbn, rank() over (partition by bv.record_num order by s.content desc), bv.record_num as bib from sierra_view.bib_view bv, sierra_view.reading_history rh, sierra_view.subfield_view s 4 where bv.bcode3 = '-' 5 and bv.record_num <> $bib_id 6 and bv.record_num = s.record_num 7 and rh.checkout_gmt::date > now()::date 365 8 and rh.patron_record_metadata_id in ($patron_ids) 9 and rh.bib_record_metadata_id = bv.id 10 and s.record_type_code = 'b' 11 and s.field_type_code like 'i' 12 and s.tag = 'a' 13 group by bv.title, bv.record_num, s.content 14 order by count desc 15 ) sub_query 16 where rank = 1 17 limit $limit; script 5 additionally, on line 3 we join the reading_history table with the bib_view table and another table subfield_view, which contains the isbns associated with the bib record. lines 6 (subfield_view) and 9 (reading_history) specify the columns on which the joins are performed. on line 4 we place a restriction to include only bibliographic records that are available for circulation. on line 7 we limit our search of checkouts to those within the last year – mainly to improve query performance. on line 8 we pass the list of users generated by the first script as a list of comma separated patron ids. lines 10, 11 and 12 allow us to retrieve the isbns of the items. line 5 prevents the item currently being viewed from being returned. we group the results by the bib title, record number and isbn (line 13) – since title is in essence a controlled vocabulary we can be confident this will be meaningful. we order the results of the inner query by the count representing the number of times the item has been borrowed amongst the user group (line 14). we can also limit the maximum number of returned results by passing a limit parameter to the outer query on line 17. the key part of the outer query is line 16 which limits the isbn returned to the one ranked the highest. this technique is sufficient for our needs and appears to work well with syndetics book jacket image api[12]. demand driven acquisitions patron driven acquisitions are ever more popular in a world of shrinking resource budgets. although usually driven through an initial ebook offering, our library wanted a process which could identify which items from our physical collection were being heavily used. the original aim of script 6 was to mimic the purchase alerts report in sierra, reduce staff time needed to process the report and to allow the resource acquisition team (rats) to respond more quickly to changes in demand. whereas purchase alerts is run weekly – due to the processing overhead, script 6 is run nightly. in the long term, it is hoped the script will allow the library to gradually reduce the size of its main collection, since it will be able to respond to changes in demand quicker and with less staff involvement. the report generated by script 6 lists monographs with long hold queues; in essence it is a decision support tool for the ordering process. the query looks for records with more than three bib-level holds, or with more than three item-level holds on any single attached item. it outputs: hold type (“bib” or “item”), bib record number, book title, if any item is currently available in the library, the maximum holds on any one item, the total holds on all items, the number of items attached to the bib, the date of the latest hold. an implicit measure of urgency is manifested in the sorting of the report, with those most urgent being presented at the top. an example report is presented in figure 2 in html format – although a csv version is also produced. figure 2 the script (script 6) consists of two parts; the first (line 1 to 12) returns bib holds and the second (line 14 to 36) bibs with item holds – the two are combined with the union operator on line 14. the first part of the script contains an inner and outer query. the inner query of the first part of the script selects the bib id, record number, title, if the item is available, a hold count on the bib and the date of the latest hold on the item. we join the query with the hold table on line 6 and group by the bib id and record number on line 7. on line 8 we restrict to rows with an item hold count of 3 or more. the outer script select on line 1 specifies these rows to be bib holds and carries through the bib record number, title, if the item is available, the total hold count and the date of the latest hold. it also introduces the total number of items attached to the bib, which becomes possible following the join on line 11 with the bib_record_item_record_link table – which contains details of bib / item record links. line 2 introduces a hold queue index – which is the measure of urgency used to sort the report. script 6 the second part of the script starts on line 14 and consists of a series of nested sub-queries to build up the rows on line 15 of the outer query. they match identically the ones seen in the first part of the query on line 1, except they relate to bibs with item holds. the inner bih query starts on line 21 and selects holds based on item records by joining with the bib_record_item_record_link and bib_view tables. the bh query on line 19 uses this item hold information and groups by bib id and record number and a series of aggregate functions to determine a max hold count, a total number of holds and a date for the latest hold from each of the attached items. the bhc query uses this information and groups by bib id, record number and max hold count to restrict the results to those bibs that have a max hold count of 3 or more. a join is again made with the bib_record_item_record_link table to enable a count to be made of the number of items attached to the bib. the hold queue index is calculated as an average of the max hold count and the number of holds per item. this is to ensure that the hold queue index is less sensitive to single items with a high number of holds, for example an item which may be in our high demand collection. indeed, we recognize this as an area for improvement as currently we do not distinguish item types. the outer most query on line 15 orders the results by hold queue index and the hold date. in combining item and bib type holds into a single report rats can also determine if one or more of our library sites is experiencing abnormal demand. rats can respond to demand fluctuations where they had previously been unable – allowing the library to make more targeted use of its resource budget. summary we have presented our favorite uses cases in demonstration of sierradna – a mechanism for read-only access to a sierra ils database. with considered reading of the sierradna documentation we have been able to demonstrate its usefulness improving services for users as well as the back-office. it should be noted, our scripts have not been through any serious performance optimization, and we do not see this as a priority as most are run daily. perhaps the “people who borrowed…” scripts would benefit further from such development, given they must load in real-time onto record displays within our opac. however, the content is loaded asynchronously using ajax[13] and rendered at the foot of the webpage; on occasions the script takes longer to execute both features help to disguise any poor performance. our code is also subject to the changing structure of the sierra ils database. the fact queries using sierradna execute against a set of views, insulate our scripts from changes to the underlying database. although the examples presented relate to sierradna, we hope non-sierra libraries can use the principles discussed to good effect with their particular flavor of ils. acknowledgements thanks must go to the staff of the resource acquisitions (rats) and eresource teams within the university of leeds library. without their help we would not have been able to gather the requirements to create the scripts in the first place. their continued feedback and patience during development contributed considerably towards a successful outcome. notes [1] sierra. available from: https://www.iii.com/products/sierra. [2] sierradna. available from: https://www.iii.com/news/it/inntouchsierradna.pdf. [3] postgresql. available from: http://www.postgresql.org/. [4] sql. available from: https://en.wikipedia.org/wiki/sql. [5] pgadmin. available from: http://www.pgadmin.org/. [6] tonic – the restful web app php micro-framework. available from: http://www.peej.co.uk/tonic/. [7] cron. available from: https://en.wikipedia.org/wiki/cron. [8] marc. available from: http://www.loc.gov/marc/. [9] phishing. available from: http://www.actionfraud.police.uk/fraud-az-phishing. [10] data protection – code of practice. available from: http://www.leeds.ac.uk/dpa. [11] syndetics. available from: http://proquest.syndetics.com/. [12] syndetics api. available from: https://developers.exlibrisgroup.com/resources/voyager/code_contributions/syndeticsstarterdocument.pdf. [13] ajax. available from: http://api.jquery.com/jquery.ajax/. references breeding, m. the future of library resource discovery, national information standards organisation [internet]. [cited 2015 aug 10]. available from: http://www.niso.org/apps/group_public/download.php/14487/future_library_resource_discovery.pdf about the authors james padgett is a systems librarian in the library systems team at the university of leeds, where he has been since 2009. dr padgett holds a phd in distributed and high performance computing from the university of leeds. he currently has a support and development role across a range of library services including the sierra ils and dedocr, an in-house online course readings management tool – which he developed. jonathan hooper is also a systems librarian in the library systems team at the university of leeds, where he has been since 1999. his current role sees him supporting many of the library’s back-office functions, including development against the sierra ils, for which he is secondary support contact. hooper has created a number of in-house services, including a reading list system and an online payments system. subscribe to comments: for this article | for all articles issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – easing gently into opensrf, part 2 mission editorial committee process and structure code4lib issue 10, 2010-06-22 easing gently into opensrf, part 2 the open service request framework (or opensrf, pronounced “open surf”) is an inter-application message passing architecture built on xmpp (aka “jabber”). the evergreen open source library system is built on an opensrf architecture to support loosely coupled individual components communicating over an opensrf messaging bus. this article introduces opensrf, demonstrates how to build opensrf services through simple code examples, explains the technical foundations on which opensrf is built, and evaluates opensrf’s value in the context of evergreen. part 2 of a 2 part article in this issue. by dan scott part 2 of a 2 part article. see part one at http://journal.code4lib.org/articles/3284. returning streaming results in the previous implementation of the opensrf.simple-text.split method, we returned a reference to the complete array of results. for small values being delivered over the network, this is perfectly acceptable, but for large sets of values this can pose a number of problems for the requesting client. consider a service that returns a set of bibliographic records in response to a query like “all records edited in the past month”; if the underlying database is relatively active, that could result in thousands of records being returned as a single network request. the client would be forced to block until all of the results are returned, likely resulting in a significant delay, and depending on the implementation, correspondingly large amounts of memory might be consumed as all of the results are read from the network in a single block. opensrf offers a solution to this problem. if the method returns results that can be divided into separate meaningful units, you can register the opensrf method as a streaming method and enable the client to loop over the results one unit at a time until the method returns no further results. in addition to registering the method with the provided name, opensrf also registers an additional method with .atomic appended to the method name. the .atomic variant gathers all of the results into a single block to return to the client, giving the caller the ability to choose either streaming or atomic results from a single method definition. in the following example, the text splitting method has been reimplemented to support streaming; very few changes are required: text splitting method – streaming mode sub text_split { my $self = shift; my $conn = shift; my $text = shift; my $delimiter = shift || ' '; my @split_text = split $delimiter, $text; foreach my $string (@split_text) { # <1> $conn->respond($string); } return undef; } __package__->register_method( method => 'text_split', api_name => 'opensrf.simple-text.split', stream => 1 # <2> ); rather than returning a reference to the array, a streaming method loops over the contents of the array and invokes the respond() method of the connection object on each element of the array. registering the method as a streaming method instructs opensrf to also register an atomic variant (opensrf.simple-text.split.atomic). error! warning! info! debug! as hard as it may be to believe, it is true: applications sometimes do not behave in the expected manner, particularly when they are still under development. the service language bindings for opensrf include integrated support for logging messages at the levels of error, warning, info, debug, and the extremely verbose internal to either a local file or to a syslogger service. the destination of the log files, and the level of verbosity to be logged, is set in the opensrf_core.xml configuration file. to add logging to our perl example, we just have to add the opensrf::utils::logger package to our list of used perl modules, then invoke the logger at the desired logging level. you can include many calls to the opensrf logger; only those that are higher than your configured logging level will actually hit the log. the following example exercises all of the available logging levels in opensrf: use opensrf::utils::logger; my $logger = opensrf::utils::logger; # some code in some function { $logger->error("hmm, something bad definitely happened!"); $logger->warn("hmm, something bad might have happened."); $logger->info("something happened."); $logger->debug("something happened; here are some more details."); $logger->internal("something happened; here are all the gory details.") } if you call the mythical opensrf method containing the preceding opensrf logger statements on a system running at the default logging level of info, you will only see the info, warn, and err messages, as follows: results of logging calls at the default level of info [2010-03-17 22:27:30] opensrf.simple-text [err :5681:simpletext.pm:277:] hmm, something bad definitely happened! [2010-03-17 22:27:30] opensrf.simple-text [warn:5681:simpletext.pm:278:] hmm, something bad might have happened. [2010-03-17 22:27:30] opensrf.simple-text [info:5681:simpletext.pm:279:] something happened. if you then increase the the logging level to internal (5), the logs will contain much more information, as follows: results of logging calls at the default level of internal [2010-03-17 22:48:11] opensrf.simple-text [err :5934:simpletext.pm:277:] hmm, something bad definitely happened! [2010-03-17 22:48:11] opensrf.simple-text [warn:5934:simpletext.pm:278:] hmm, something bad might have happened. [2010-03-17 22:48:11] opensrf.simple-text [info:5934:simpletext.pm:279:] something happened. [2010-03-17 22:48:11] opensrf.simple-text [debg:5934:simpletext.pm:280:] something happened; here are some more details. [2010-03-17 22:48:11] opensrf.simple-text [intl:5934:simpletext.pm:281:] something happened; here are all the gory details. [2010-03-17 22:48:11] opensrf.simple-text [err :5934:simpletext.pm:283:] resolver did not find a cache hit [2010-03-17 22:48:21] opensrf.simple-text [intl:5934:cache.pm:125:] stored opensrf.simple-text.test_cache.masaa => "here" in memcached server [2010-03-17 22:48:21] opensrf.simple-text [debg:5934:application.pm:579:] coderef for [opensrf::application::demo::simpletext::test_cache] has been run [2010-03-17 22:48:21] opensrf.simple-text [debg:5934:application.pm:586:] a top level request object is responding de nada [2010-03-17 22:48:21] opensrf.simple-text [debg:5934:application.pm:190:] method duration for [opensrf.simple-text.test_cache]: 10.005 [2010-03-17 22:48:21] opensrf.simple-text [intl:5934:appsession.pm:780:] calling queue_wait(0) [2010-03-17 22:48:21] opensrf.simple-text [intl:5934:appsession.pm:769:] resending...0 [2010-03-17 22:48:21] opensrf.simple-text [intl:5934:appsession.pm:450:] in send [2010-03-17 22:48:21] opensrf.simple-text [debg:5934:appsession.pm:506:] appsession sending result to opensrf@private.localhost/_dan-karmic-liblap_1268880489.752154_5943 with threadtrace [1] [2010-03-17 22:48:21] opensrf.simple-text [debg:5934:appsession.pm:506:] appsession sending status to opensrf@private.localhost/_dan-karmic-liblap_1268880489.752154_5943 with threadtrace [1] ... to see everything that is happening in opensrf, try leaving your logging level set to internal for a few minutes – just ensure that you have a lot of free disk space available if you have a moderately busy system! caching results: one secret of scalability if you have ever used an application that depends on a remote web service outside of your control — say, if you need to retrieve results from a microblogging service — you know the pain of latency and dependability (or the lack thereof). to improve the response time for opensrf services, you can take advantage of the support offered by the opensrf::utils::cache module for communicating with a local instance or cluster of memcache daemons to store and retrieve persistent values. the following example demonstrates caching by sleeping for 10 seconds the first time it receives a given cache key and cannot retrieve a corresponding value from the cache: simple caching opensrf service use opensrf::utils::cache; # <1> sub test_cache { my $self = shift; my $conn = shift; my $test_key = shift; my $cache = opensrf::utils::cache->new('global'); # <2> my $cache_key = "opensrf.simple-text.test_cache.$test_key"; # <3> my $result = $cache->get_cache($cache_key) || undef; # <4> if ($result) { $logger->info("resolver found a cache hit"); return $result; } sleep 10; # <5> my $cache_timeout = 300; # <6> $cache->put_cache($cache_key, "here", $cache_timeout); # <7> return "there was no cache hit."; } the opensrf::utils::cache module provides access to the built-in caching support in opensrf. the constructor for the cache object accepts a single argument to define the cache type for the object. each cache type can use a separate memcache server to keep the caches separated. most evergreen services use the global cache, while the anon cache is used for web sessions. the cache key is simply a string that uniquely identifies the value you want to store or retrieve. this line creates a cache key based on the opensrf method name and request input value. the get_cache() method checks to see if the cache key already exists. if a matching key is found, the service immediately returns the stored value. if the cache key does not exist, the code sleeps for 10 seconds to simulate a call to a slow remote web service or an intensive process. the $cache_timeout variable represents a value for the lifetime of the cache key in seconds. after the code retrieves its value (or, in the case of this example, finishes sleeping), it creates the cache entry by calling the put_cache() method. the method accepts three arguments: the cache key, the value to be stored (“here”), and the timeout value in seconds to ensure that we do not return stale data on subsequent calls. initializing the service and its children: child labour when an opensrf service is started, it looks for a procedure called initialize() to set up any global variables shared by all of the children of the service. the initialize() procedure is typically used to retrieve configuration settings from the opensrf.xml file. an opensrf service spawns one or more children to actually do the work requested by callers of the service. for every child process an opensrf service spawns, the child process clones the parent environment and then each child process runs the child_init() process (if any) defined in the opensrf service to initialize any child-specific settings. when the opensrf service kills a child process, it invokes the child_exit() procedure (if any) to clean up any resources associated with the child process. similarly, when the opensrf service is stopped, it calls the destroy() procedure to clean up any remaining resources. retrieving configuration settings the settings for opensrf services are maintained in the opensrf.xml xml configuration file. the structure of the xml document consists of a root element <opensrf> containing two child elements: the <default> element contains an <apps> element describing all opensrf services running on this system — see [serviceregistration] –, as well as any other arbitrary xml descriptions required for global configuration purposes. for example, evergreen uses this section for email notification and inter-library patron privacy settings. the <hosts> element contains one element per host that participates in this opensrf system. each host element must include an <activeapps> element that lists all of the services to start on this host when the system starts up. each host element can optionally override any of the default settings. opensrf includes a service named opensrf.settings to provide distributed cached access to the configuration settings with a simple api: opensrf.settings.default_config.get accepts zero arguments and returns the complete set of default settings as a json document. opensrf.settings.host_config.get accepts one argument (hostname) and returns the complete set of settings, as customized for that hostname, as a json document. opensrf.settings.xpath.get accepts one argument (an xpath expression) and returns the portion of the configuration file that matches the expression as a json document. for example, to determine whether an evergreen system uses the opt-in support for sharing patron information between libraries, you could either invoke the opensrf.settings.default_config.get method and parse the json document to determine the value, or invoke the opensrf.settings.xpath.get method with the xpath /opensrf/default/share/user/opt_in argument to retrieve the value directly. in practice, opensrf includes convenience libraries in all of its client language bindings to simplify access to configuration values. c offers osrfconfig.c, perl offers opensrf::utils::settingsclient, java offers org.opensrf.util.settingsclient, and python offers osrf.set. these libraries locally cache the configuration file to avoid network roundtrips for every request and enable the developer to request specific values without having to manually construct xpath expressions. getting under the covers with opensrfget on the messaging bus – safely one of the core innovations of opensrf was to use the extensible messaging and presence protocol (xmpp, more colloquially known as jabber) as the messaging bus that ties opensrf services together across servers. xmpp is an “xml protocol for near-real-time messaging, presence, and request-response services” (http://www.ietf.org/rfc/rfc3920.txt) that opensrf relies on to handle most of the complexity of networked communications. opensrf requres an xmpp server that supports multiple domains such as ejabberd. multiple domain support means that a single server can support xmpp virtual hosts with separate sets of users and access privileges per domain. by routing communications through separate public and private xmpp domains, opensrf services gain an additional layer of security. the opensrf installation documentation instructs you to create two separate hostnames (private.localhost and public.localhost) to use as xmpp domains. opensrf can control access to its services based on the domain of the client and whether a given service allows access from clients on the public domain. when you start opensrf, the first xmpp clients that connect to the xmpp server are the opensrf public and private routers. opensrf routers maintain a list of available services and connect clients to available services. when an opensrf service starts, it establishes a connection to the xmpp server and registers itself with the private router. the opensrf configuration contains a list of public opensrf services, each of which must also register with the public router. opensrf communication flows over xmpp in a minimal opensrf deployment, two xmpp users named “router” connect to the xmpp server, with one connected to the private xmpp domain and one connected to the public xmpp domain. similarly, two xmpp users named “opensrf” connect to the xmpp server via the private and public xmpp domains. when an opensrf service is started, it uses the “opensrf” xmpp user to advertise its availability with the corresponding router on that xmpp domain; the xmpp server automatically assigns a jabber id (jid) based on the client hostname to each service’s listener process and each connected drone process waiting to carry out requests. when an opensrf router receives a request to invoke a method on a given service, it connects the requester to the next available listener in the list of registered listeners for that service. services and clients connect to the xmpp server using a single set of xmpp client credentials (for example, opensrf@private.localhost), but use xmpp resource identifiers to differentiate themselves in the jid for each connection. for example, the jid for a copy of the opensrf.simple-text service with process id 6285 that has connected to the private.localhost domain using the opensrf xmpp client credentials could be opensrf@private.localhost/opensrf.simple-text_drone_at_localhost_6285. by convention, the user name for opensrf clients is opensrf, and the user name for opensrf routers is router, so the xmpp server for opensrf will have four separate users registered: opensrf@private.localhost is an opensrf client that connects with these credentials and which can access any opensrf service. opensrf@public.localhost is an opensrf client that connects with these credentials and which can only access opensrf services that have registered with the public router. router@private.localhost is the private opensrf router with which all services register. router@public.localhost is the public opensrf router with which only services that must be publicly accessible register. all opensrf services automatically register themselves with the private xmpp domain, but only those services that register themselves with the public xmpp domain can be invoked from public opensrf clients. the opensrf client and router user names, passwords, and domain names, along with the list of services that should be public, are contained in the opensrf_core.xml configuration file. opensrf communication flows over http in some contexts, access to a full xmpp client is not a practical option. for example, while xmpp clients have been implemented in javascript, you might be concerned about browser compatibility and processing overhead – or you might want to issue opensrf requests from the command line with curl. fortunately, any opensrf service registered with the public router is accessible via the opensrf http translator. the opensrf http translator implements the opensrf-over-http proposed specification as an apache module that translates http requests into opensrf requests and returns opensrf results as http results to the initiating http client. issuing an http post request to an opensrf method via the opensrf http translator # curl request broken up over multiple lines for legibility curl -h "x-opensrf-service: opensrf.simple-text" \ # <1> --data 'osrf-msg=[ \ # <2> {"__c":"osrfmessage","__p":{"threadtrace":0,"locale":"en-ca", \ # <3> "type":"request","payload": {"__c":"osrfmethod","__p": \ {"method":"opensrf.simple-text.reverse","params":["foobar"]} \ }} \ }]' \ http://localhost/osrf-http-translator \ # <4> the x-opensrf-service header identifies the opensrf service of interest. the post request consists of a single parameter, the osrf-msg value, which contains a json array. the first object is an opensrf message ("__c":"osrfmessage") with a set of parameters ("__p":{}). the identifier for the request ("threadtrace":0); this value is echoed back in the result. the message type ("type":"request"). the locale for the message; if the opensrf method is locale-sensitive, it can check the locale for each opensrf request and return different information depending on the locale. the payload of the message ("payload":{}) containing the opensrf method request ("__c":"osrfmethod") and its parameters ("__p:"{}). the method name for the request ("method":"opensrf.simple-text.reverse"). a set of json parameters to pass to the method ("params":["foobar"]); in this case, a single string "foobar". the url on which the opensrf http translator is listening, /osrf-http-translator is the default location in the apache example configuration files shipped with the opensrf source, but this is configurable. results from an http post request to an opensrf method via the opensrf http translator # http response broken up over multiple lines for legibility [{"__c":"osrfmessage","__p": \ # <1> {"threadtrace":0, "payload": \ # <2> {"__c":"osrfresult","__p": \ # <3> {"status":"ok","content":"raboof","statuscode":200} \ # <4> },"type":"result","locale":"en-ca" \ # <5> } }, {"__c":"osrfmessage","__p": \ # <6> {"threadtrace":0,"payload": \ # <7> {"__c":"osrfconnectstatus","__p": \ # <8> {"status":"request complete","statuscode":205} \ # <9> },"type":"status","locale":"en-ca" \ # <10> } }] the opensrf http translator returns an array of json objects in its response. each object in the response is an opensrf message ("__c":"osrfmessage") with a collection of response parameters ("__p":). the opensrf message identifier ("threadtrace":0) confirms that this message is in response to the request matching the same identifier. the message includes a payload json object ("payload":) with an opensrf result for the request ("__c":"osrfresult"). the result includes a status indicator string ("status":"ok"), the content of the result response – in this case, a single string “raboof” ("content":"raboof") – and an integer status code for the request ("statuscode":200). the message also includes the message type ("type":"result") and the message locale ("locale":"en-ca"). the second message in the set of results from the response. again, the message identifier confirms that this message is in response to a particular request. the payload of the message denotes that this message is an opensrf connection status message ("__c":"osrfconnectstatus"), with some information about the particular opensrf connection that was used for this request. the response parameters for an opensrf connection status message include a verbose status ("status":"request complete") and an integer status code for the connection status (`”statuscode”:205). the message also includes the message type ("type":"result") and the message locale ("locale":"en-ca"). tip before adding a new public opensrf service, ensure that it does not introduce privilege escalation or unchecked access to data. for example, the evergreen open-ils.cstore private service is an object-relational mapper that provides read and write access to the entire evergreen database, so it would be catastrophic to expose that service publicly. in comparison, the evergreen open-ils.pcrud public service offers the same functionality as open-ils.cstore to any connected http client or opensrf client, but the additional authentication and authorization layer in open-ils.pcrud prevents unchecked access to evergreen’s data. stateless and stateful connections opensrf supports both stateless and stateful connections. when an opensrf client issues a request message in a stateless connection, the router forwards the request to the next available service and the service returns the result directly to the client. when an opensrf client issues a connect message to create a stateful conection, the router returns the jabber id of the next available service to the client so that the client can issue one or more request message directly to that particular service and the service will return corresponding result messages directly to the client. until the client issues a disconnect message, that particular service is only available to the requesting client. stateful connections are useful for clients that need to make many requests from a particular service, as it avoids the intermediary step of contacting the router for each request, as well as for operations that require a controlled sequence of commands, such as a set of database insert, update, and delete statements within a transaction. message body format opensrf was an early adopter of javascript object notation (json). while xmpp is an xml protocol, the evergreen developers recognized that the compactness of the json format offered a significant reduction in bandwidth for the volume of messages that would be generated in an application of that size. in addition, the ability of languages such as javascript, perl, and python to generate native objects with minimal parsing offered an attractive advantage over invoking an xml parser for every message. instead, the body of the xmpp message is a simple json structure. for a simple request, like the following example that simply reverses a string, it looks like a significant overhead: but we get the advantages of locale support and tracing the request from the requester through the listener and responder (drone). a request for opensrf.simple-text.reverse(“foobar”): <message from='router@private.localhost/opensrf.simple-text' to='opensrf@private.localhost/opensrf.simple-text_listener_at_localhost_6275' router_from='opensrf@private.localhost/_karmic_126678.3719_6288' router_to='' router_class='' router_command='' osrf_xid='' > <thread>1266781414.366573.12667814146288</thread> <body> [ {"__c":"osrfmessage","__p": {"threadtrace":"1","locale":"en-us","type":"request","payload": {"__c":"osrfmethod","__p": {"method":"opensrf.simple-text.reverse","params":["foobar"]} } } } ] </body> </message> a response from opensrf.simple-text.reverse(“foobar”) <message from='opensrf@private.localhost/opensrf.simple-text_drone_at_localhost_6285' to='opensrf@private.localhost/_karmic_126678.3719_6288' router_command='' router_class='' osrf_xid='' > <thread>1266781414.366573.12667814146288</thread> <body> [ {"__c":"osrfmessage","__p": {"threadtrace":"1","payload": {"__c":"osrfresult","__p": {"status":"ok","content":"raboof","statuscode":200} } ,"type":"result","locale":"en-us"} }, {"__c":"osrfmessage","__p": {"threadtrace":"1","payload": {"__c":"osrfconnectstatus","__p": {"status":"request complete","statuscode":205} },"type":"status","locale":"en-us"} } ] </body> </message> the content of the <body> element of the opensrf request and result should look familiar; they match the structure of the opensrf over http examples that we previously dissected. registering opensrf methods in depth let’s explore the call to __package__->register_method(); most of the members of the hash are optional, and for the sake of brevity we omitted them in the previous example. as we have seen in the results of the introspection call, a verbose registration method call is recommended to better enable the internal documentation. here is the complete set of members that you should pass to __package__->register_method(): the method member specifies the name of the procedure in this module that is being registered as an opensrf method. the api_name member specifies the invocable name of the opensrf method; by convention, the opensrf service name is used as the prefix. the optional api_level member can be used for versioning the methods to allow the use of a deprecated api, but in practical use is always 1. the optional argc member specifies the minimal number of arguments that the method expects. the optional stream member, if set to any value, specifies that the method supports returning multiple values from a single call to subsequent requests. opensrf automatically creates a corresponding method with “.atomic” appended to its name that returns the complete set of results in a single request. streaming methods are useful if you are returning hundreds of records and want to act on the results as they return. the optional signature member is a hash that describes the method’s purpose, arguments, and return value. the desc member of the signature hash describes the method’s purpose. the params member of the signature hash is an array of hashes in which each array element describes the corresponding method argument in order. the name member of the argument hash specifies the name of the argument. the desc member of the argument hash describes the argument’s purpose. the type member of the argument hash specifies the data type of the argument: for example, string, integer, boolean, number, array, or hash. the return member of the signature hash is a hash that describes the return value of the method. the desc member of the return hash describes the return value. the type member of the return hash specifies the data type of the return value: for example, string, integer, boolean, number, array, or hash. evergreen-specific opensrf services evergreen is currently the primary showcase for the use of opensrf as an application architecture. evergreen 1.6.1 includes the following set of opensrf services: the open-ils.actor service supports common tasks for working with user accounts and libraries. the open-ils.auth service supports authentication of evergreen users. the open-ils.booking service supports the management of reservations for bookable items. the open-ils.cat service supports common cataloging tasks, such as creating, modifying, and merging bibliographic and authority records. the open-ils.circ service supports circulation tasks such as checking out items and calculating due dates. the open-ils.collections service supports tasks that assist collections agencies in contacting users with outstanding fines above a certain threshold. the open-ils.cstore private service supports unrestricted access to evergreen fieldmapper objects. the open-ils.ingest private service supports tasks for importing data such as bibliographic and authority records. the open-ils.pcrud service supports permission-based access to evergreen fieldmapper objects. the open-ils.penalty penalty service supports the calculation of penalties for users, such as being blocked from further borrowing, for conditions such as having too many items checked out or too many unpaid fines. the open-ils.reporter service supports the creation and scheduling of reports. the open-ils.reporter-store private service supports access to evergreen fieldmapper objects for the reporting service. the open-ils.search service supports searching across bibliographic records, authority records, serial records, z39.50 sources, and zip codes. the open-ils.storage private service supports a deprecated method of providing access to evergreen fieldmapper objects. implemented in perl, this service has largely been replaced by the much faster c-based open-ils.cstore service. the open-ils.supercat service supports transforms of marc records into other formats, such as mods, as well as providing atom and rss feeds and sru access. the open-ils.trigger private service supports event-based triggers for actions such as overdue and holds available notification emails. the open-ils.vandelay service supports the import and export of batches of bibliographic and authority records. of some interest is that the open-ils.reporter-store and open-ils.cstore services have identical implementations. surfacing them as separate services enables a deployer of evergreen to ensure that the reporting service does not interfere with the performance-critical open-ils.cstore service. one can also direct the reporting service to a read-only database replica to, again, avoid interference with open-ils.cstore which must write to the master database. there are only a few significant services that are not built on opensrf in evergreen 1.6.0, such as the sip and z39.50 servers. these services implement different protocols and build on existing daemon architectures (simple2zoom for z39.50), but still rely on the other opensrf services to provide access to the evergreen data. the non-opensrf services are reasonably self-contained and can be deployed on different servers to deliver the same sort of deployment flexibility as opensrf services, but have the disadvantage of not being integrated into the same configuration and control infrastructure as the opensrf services. evergreen after one year: reflections on opensrf project conifer has been live on evergreen for just over a year now, and as one of the primary technologists i have had to work closely with the opensrf infrastructure during that time. as such, i am in a position to identify some of the strengths and weaknesses of opensrf based on our experiences. weaknesses the primary weakness of opensrf is the lack of either formal or informal documentation for opensrf. there are many frequently asked questions on the evergreen mailing lists and irc channel that indicate that some of the people running evergreen or trying to run evergreen have not been able to find documentation to help them understand, even at a high level, how the opensrf router and services work with xmpp and the apache web server to provide a working evergreen system. also, over the past few years several developers have indicated an interest in developing ruby and php bindings for opensrf, but the efforts so far have resulted in no working code. the lack of a formal specification, clearly annotated examples, and portable unit tests for the major opensrf communication use cases is a significant hurdle for a developer seeking to fulfill a base set of expectations for a working binding. as a result, evergreen integration efforts with popular frameworks like drupal, blacklight, and vufind result in the best practical option for a developer with limited time — database-level integration — which has the unfortunate side effect of being much more likely to break after an upgrade. in conjunction with the lack of documentation that makes it hard to get started with the framework, a disincentive for new developers to contribute to opensrf itself is the lack of integrated unit tests. for a developer to contribute a significant, non-obvious patch to opensrf, they need to manually run through various (undocumented, again) use cases to try and ensure that the patch introduced no unanticipated side effects. the same problems hold for evergreen itself, although the constrictor stress-testing framework offers a way of performing some automated system testing and performance testing. these weaknesses could be relatively easily overcome with the effort through contributions from people with the right skill sets. this article arguably offers a small set of clear examples at both the networking and application layer of opensrf. a technical writer who understands opensrf could contribute a formal specification to the project. with a formal specification at their disposal, a quality assurance expert could create an automated test harness and a basic set of unit tests that could be incrementally extended to provide more coverage over time. if one or more continual integration environments are set up to track the various opensrf branches of interest, then the opensrf community would have immediate feedback on build quality. once a unit testing framework is in place, more developers might be willing to develop and contribute patches as they could sanity check their own code without an intense effort before exposing it to their peers. strengths of opensrf as a service infrastructure, opensrf has been remarkably reliable. we initially deployed evergreen on an unreleased version of both opensrf and evergreen due to our requirements for some functionality that had not been delivered in a stable release at that point in time, and despite this risky move we suffered very little unplanned downtime in the opening months. on july 27, 2009 we moved to a newer (but still unreleased) version of the opensrf and evergreen code, and began formally tracking our downtime. since then, we have achieved more than 99.9% availability – including scheduled downtime for maintenance. this compares quite favourably to the maximum of 75% availability that we were capable of achieving on our previous library system due to the nightly downtime that was required for our backup process. the opensrf “maximum request” configuration parameter for each service that kills off drone processes after they have served a given number of requests provides a nice failsafe for processes that might otherwise suffer from a memory leak or hung process. it also helps that when we need to apply an update to a perl service that is running on multiple servers, we can apply the updated code, then restart the service on one server at a time to avoid any downtime. as promised by the opensrf infrastructure, we have also been able to tune our cluster of servers to provide better performance. for example, we were able to change the number of maximum concurrent processes for our database services when we saw a performance bottleneck with database access. to go live with a configuration change, we restart the opensrf.setting service to pick up the change, then restart the affected service on each of our servers. we were also able to turn off some of the less-used opensrf services, such as open-ils.collections, on one of our servers to devote more resources on that server to the more frequently used services and other performance-critical processes such as apache. the support for logging and caching that is built into opensrf has been particularly helpful with the development of a custom service for sfx holdings integration into our catalogue. once i understood how opensrf works, most of the effort required to build that sfx integration service was spent on figuring out how to properly invoke the sfx api to display human-readable holdings. adding a new opensrf service and registering several new methods for the service was relatively easy. the support for directing log messages to syslog in opensrf has also been a boon for both development and debugging when problems arise in a cluster of five servers; we direct all of our log messages to a single server where we can inspect the complete set of messages for the entire cluster in context, rather than trying to piece them together across servers. appendices: code examples opensrf_core.xml complete example: this complete example of opensrf_core.xml from a working test system is included for your convenience. opensrf.xml complete example: this complete example of opensrf.xml from a this complete example of opensrf.xml from a working test system is included for your convenience. python client: a python client that makes the same opensrf calls as the perl client. about the author dan scott is the systems librarian for laurentian university and a committer for the the evergreen and opensrf projects. you can reach him by email at dan@coffeecode.net or follow his occasional ramblings at http://coffeecode.net. subscribe to comments: for this article | for all articles one response to "easing gently into opensrf, part 2" please leave a response below: leonard parker, 2011-08-01 i was just starting to read the two articles mentioned on the evergreen site. i think the links found on that site are reversed when one goes to fetch the content of part 1 and part 2 of the topics. should you have questions, feel free to drop me a line. leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – editorial: forget the ai, we have live editors mission editorial committee process and structure code4lib issue 56, 2023-04-21 editorial: forget the ai, we have live editors welcoming new editors to the code4lib journal i’m sure by now, you, like me, have explored chatgpt for coding, writing articles, and and just about everything else you can think of. and, like me, you probably are both excited and concerned about its possibilities. i couldn’t resist trying it with a prompt of “write an editorial about chatgpt for the code4lib journal”, and it did a good job that i probably could have passed off, especially if i had asked it to do it my writing style. i also asked it to write an editorial of its impact on libraries, which was also quite impressive, and, truth be told, would probably be accepted into this journal if submitted. this is the first issue which includes some code written by chatgpt, and i’m sure we’ll be seeing more. for fun, i asked it to convert some old perl code to python (on my to-do list with lots of old scripts i have floating around) and while it didn’t make an attempt, it did give me ‘guidance’ on the variations in syntax between the two languages, fwiw. i also asked it to write me a perl script that would identify marc records with invalid marc tags, which spit out a reasonable little bit of code, though it only checked that a tag was numeric, and not for length, and only took one record at a time for input. still, it seems like an easy way to get the outlines of small tasks written. i’m sure many of you will show us some more powerful ways to use this tool and others. as ai technology progresses and permeates our libraries, i’m sure we’ll be talking about it in many areas from systems to reference to bibliographic instruction and information literacy (good luck with that!). for example this issue also features an article exploring locally hosted ai for image metadata creation. but for us here at c4lj, the real excitement is not about ai and chatbots, but about the 4 new live editors who joined our editorial board this month. we are happy to welcome them and trust adding them to our ranks will help alleviate some of the bottlenecks we’ve run into over the past few years. please join me in welcoming: mark eaton – associate professor/reader services librarian at kingsborough community college kirsta stapelfeldt – head, digital scholarship unit at university of toronto christina l. hennessey – systems librarian, california state university at northridge university library herbert van de sompel research fellow at dans (netherlands) and guest professor at ghent university (belgium) and with that, here are the articles in issue 56: apples to oranges: using python and the pymarc library to match bookstore isbns to locally held ebook isbns the brooklyn health map: reflections on a health data dashboard for brooklyn, ny building a large-scale digital library search interface using the libraries online catalog premis events through an event-sourced lens strategies for digital library migration to everything there is a session a time to listen, a time to read multi-session cds utilizing r and python for institutional repository daily jobs the viability of using an open source locally hosted ai for creating metadata in digital image collections subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – citation needed: adding citations to contentdm records mission editorial committee process and structure code4lib issue 53, 2022-05-09 citation needed: adding citations to contentdm records the tennessee state library and archives and the illinois state library identified a need to add citation information to individual image records in oclc’s contentdm (https://www.oclc.org/en/contentdm.html). experience with digital archives at both institutions showed that citation information was one of the most requested features. unfortunately, contentdm does not natively display citation information about image records; to add this functionality, custom javascript had to be written that would interact with the underlying react environment and parse out or retrieve the appropriate metadata to dynamically build record citations. detailed code and a description of methods for building two different models of citation generators are presented. by jenn randles & andrew bullen introduction the tennessee state library and archives and the illinois state library identified a need to add citation information to individual image records in oclc’s contentdm (https://www.oclc.org/en/contentdm.html). experience with digital archives at both institutions showed that citation information was one of the most requested features. unfortunately, contentdm does not natively display citation information about image records; to add this functionality, custom javascript had to be written that would interact with the underlying react environment and parse out or retrieve the appropriate metadata to dynamically build record citations. jenn randles, then digital materials librarian at the tennessee state library & archives, and andrew bullen, information technology coordinator for the illinois state library, both expressed a willingness to work on this problem via the contentdm user community forum site. shane huddleston, senior developer at contentdm, volunteered to assist randles and bullen. this article delves into each approach, demonstrates the code of each solution, and displays the end results. the problem both institutions’ statewide repositories for digital collections contain a large number of digital objects from a wide variety of contributors. the tennessee virtual archive [teva] (https://teva.contentdm.oclc.org/) comprises 67 collections. the illinois digital archives [ida] (https://www.idaillinois.org/) consists of more than 200 collections. metadata in teva is uniform across the varied collections since metadata is submitted by one organization. ida collections, however, have widely varied metadata schemas since they are contributed by many institutions. using the iiif api works well when metadata fields are consistent across collections but isn’t applicable otherwise. therefore, the contributors had to develop their own approaches. despite working on two solutions, randles and bullen worked together to develop javascript code that would work for each environment. they consulted as needed with shane huddleston for expert guidance on programming javascript in contentdm’s react environment. since there had to be two approaches to the same problem, randles and bullen have each written and documented their separate, successful code solutions. solution 1. jenn randles, tennessee virtual archive. tl;dr. because teva’s metadata is uniform, jenn parsed the iiif json manifest for the record to dynamically build the citation. link to script: https://teva.contentdm.oclc.org/customizations/global/pages/js/tsla-cite-global.js for teva, the most pressing reason to provide citations alongside a digital collection item is simple: users asked for it. over the five years i held this position, one of the most requested features was citation on each item. to start my work, i reviewed other institutions to see how they accomplished this task. other organizations added citations manually, but with over 30,000 records already in the system, we did not have the staff to undertake such a project. we do have the addthis widget (https://www.addthis.com/) on every item page on teva, but it only includes one citation manager: citethis. since many of our users among the general public do not use that service or any citation manager at all, i wanted to go with another option. my end goal was not to provide academic-style citations but give users like bloggers and genealogists a quick way to record sources without having to make many decisions. i made the decision to use contentdm’s international image interoperability framework (iiif) api to dynamically build record citations. contentdm enabled the iiif api (https://help.oclc.org/metadata_services/contentdm/advanced_website_customization/api_reference/iiif_api_reference) on all hosted sites several years ago. the iiif framework provides a way to access digital assets and their metadata from a variety of platforms like mirador and leaflet. knowing that the metadata i needed existed in the iiif item manifest that was already a part of contentdm, this seemed like the perfect opportunity to try it out. since all teva’s metadata is in fields with the same labels across collections, i knew i could use the api to return the same data from each item and print it to the screen in a specific order. data needed first, i determined what data i needed from the api: contentdm collection alias contentdm item id title creator date of item collection name storage location (box and folder in the collection) digital object id (essential for locating digital surrogates) owning institution (we house both tennessee state library and archives and tennessee historical society collections) repository (teva) getting collection alias and item id the first step is to find two variables: the contentdm collection alias and item id. both can be grabbed from the url of the item’s page. the format of urls in contentdm is https://teva.contentdm.oclc.org/digital/collection/[collection alias]/id/[id], such as https://teva.contentdm.oclc.org/digital/collection/highlander/id/1032. the contentdm item id is different than our digital object id. we need both, since the itemid is part of the iiif manifest url. to get these i split the window pathname, called those parts by index number, and assigned them to variables. i had to include the global var at the front, since an upgrade to the backend broke my previous code. i need to declare this at the beginning; otherwise, it throws an error and no citation shows on the page. var global = global || window; patharray = window.location.pathname.split("/"); collalias = patharray[3]; itemid = patharray[5]; mapping manifest fields i now have the necessary information to build a manifest link. i can map contentdm fields based on api labels in the manifest, such as in https://teva.contentdm.oclc.org/iiif/2/highlander:1032/manifest.json. an iiif json manifest is an array of key-value pairs that can appear in a different order based on what metadata exists in the record. simply calling these by index will not work due to this variable order; however, since our labels are consistent across collections, i can use them instead. the script calls data by label later on, but first i will put these fields in order of the final citation, so they show up in the order i want. const mappings = { title: "itemtitle", order: 0, creator: "itemcreator", order: 1, date: "itemdate", order: 2, "collection name": "itemcollection", order: 3, "storage location": "itemstorage", order: 4, "id#": "itemid", order: 5, "owning institution": "itemowning", order: 6, repository: "itemrepository", order: 7, }; check for parent record id i now need to check and see if the script was not returning manifests due to items being part of a compound object. compound objects are multi-page items stored in a hierarchy on contentdm. in teva, we always have citation fields on the item level, but often not on the page level. additionally, each page of an item and the main item-level record itself have different contentdm ids. we need to check whether a record has a parent and return that parent record information to locate the manifest. otherwise, the script will not return anything since it cannot find the manifest. i got this code from shane huddleston at oclcthanks! function getparent(item, collection) { return ( fetch( `/digital/bl/dmwebservices/index.php?q=getparent/${collection}/${item}/json` ) .then((response) => response.json()) // make getparent api call and return as json .then((json) => { let parent = false; // parse json for 'parent' value; -1 indicates parent id is the same as item id if (json.parent === -1) { parent = item; } else { parent = json.parent; } return parent; }) .then((parent) => { // once parent is known, check if iiif pres manifest exists return fetch(`/iiif/info/${collection}/${parent}/manifest.json`) .then((response) => { if (response.status != 200) { console.log("no iiif manifest exists for this record."); parent = false; // if no manifest exists, return is 'false' so that iiif button is not inserted return parent; } else { // check if manifest is for single-item pdf return fetch( `/digital/api/collections/${collection}/items/${parent}/false` ) .then((response) => response.json()) .then((json) => { if (json.filename.split(".").pop() === "pdf") { // if item format is pdf return is false so that iiif button is not inserted console.log("pdf?", json.filename.split(".").pop()); parent = false; return parent; } else { return parent; } }) .catch((error) => console.log("item api request failed.", error) ); } }) .catch((error) => { console.log("manifest request failed.", error); parent = false; return parent; }); }) .catch(function (error) { console.log("getparent request failed.", error); parent = false; return parent; }) ); } api call, assigning data via labels the next step is an api call, pointing to the manifest using collalias and itemid, then mapping label/value pairs to the variables in the const mappings. async function buildmetadataobject(collalias, itemid) { let response = await fetch( "https://teva.contentdm.oclc.org/iiif/info/" + global.collalias + "/" + global.itemid + "/manifest.json" ); let data = await response.json(); return data.metadata.reduce( (acc, { label, value }) => ({ ...acc, [mappings[label]]: value }), {} ); } creating the citation the final section takes the pieces specified and puts them in order using the following steps: grab today’s date for date accessed. create reference url from collalias and itemid. add itemtitle to blacklist because we need to add quotes to it in the builder. reduce to ignore undefined values, since not every field has data in it. it was printing “undefined” before this addition. create const itemcite using mappings order and join those. i decided to go with a local citation format so we don’t need to update often and it won’t clutter the page. the format works for the public who just needs something quick and not a specific format. this gets us a citation that works for many users, without much upkeep on our side or accounting for multiple formats. add link to our citation info that has a guide to citations for digitized archival materials, then put the entire thing in a div at the end of the metadata section after a line. i grabbed this location idea from my co-author andrew bullenthanks! function insertcitation(data) { var ttitle = data.itemtitle; var url = "https://teva.contentdm.oclc.org/digital/collection/" + global.collalias + "/id/" + global.itemid; var datetoday = new date().toisostring().slice(0, 10); const fieldblacklist = ["itemtitle"]; const itemcite = `<hr /><strong>citation:</strong> \n "${ data.itemtitle }," ${object.values(mappings) .reduce((acc, cur) => { if (fieldblacklist.includes(cur)) return acc; const value = data[cur]; return value ? [...acc, value] : acc; }, []) .join( ", " )}, ${url}, accessed ${datetoday}. <br>note: this citation is auto-generated and may be incomplete. check your preferred style guide or the <a href="https://teva.contentdm.oclc.org/customizations/global/pages/about.html">about teva page</a> for more information.`; const citationcontainer = document.createelement("div"); citationcontainer.id = "citation"; citationcontainer.innerhtml = itemcite; if (ttitle) { document .queryselector(".itemview-itemviewcontainer") .appendchild(citationcontainer); } } printing to page this section prints the citation after the page loads or on the page update. (async () => { document.addeventlistener("cdm-item-page:ready", async function (e) { const citationdata = await buildmetadataobject(); insertcitation(citationdata); }); document.addeventlistener("cdm-item-page:update", async function (e) { document.getelementbyid("citation").remove(); const citationdata = await buildmetadataobject(); insertcitation(citationdata); }); })(); link to the whole script: https://teva.contentdm.oclc.org/customizations/global/pages/js/tsla-cite-global.js the end result figure 1. metadata from a teva collection item. iiif json manifest for this item: https://teva.contentdm.oclc.org/iiif/2/highlander:1032/manifest.json figure 2. dynamically-created citation for the item in fig. 1 solution 1: conclusion if i were to expand this, i would work on a multi-format citation generator so users could choose what they needed. i would still display the local citation on the page but would have other options available with a few clicks. solution 2. andrew bullen, illinois digital archives tl;dr. because ida’s metadata varies widely across the collections, andrew created a collection of collections that contains a field listing the collection’s field names. he then used contentdm’s apis to retrieve the data from the record itself, which could then be used to assemble the proper record citation. link to the whole script: https://www.idaillinois.org/customizations/global/js/citation.js a collection of collections we have found a useful addition to our contentdm instance is a management collection, which contains administrative information about the other collections. our so-called collection of collections can be viewed at https://www.idaillinois.org/digital/collection/p16614coll32/search/. among the information contained about each collection is the creator of the collection, which contains the agency and its url of the contributor; controlnumber, which is useful in connecting the agency and collection to an agency administration system; and, since we have a number of local history collections, city and county, which will someday be used to map the collections and geolocate them. we added a field, citationfield, that contains the field names of the elements that make up image citations in most formats. we chose what we determined were the most popular: mla (version 8), chicago/turabian, apa (version 6), and ieee. the seven field names are held in a string separated by the pipe symbol (|). field names and nicknames in contentdm since our contributing institutions contribute and create their own metadata records, we cannot count on uniform field names or even the existence of all of the fields that make up a citation. a problem that we encountered from the first was that contentdm stores fields using a nickname, which can vary from collection to collection even if the label used is the same. for instance, one institution might have a field of creator, with a nickname of creato, and another might have the same field name stored as creator. i created a simple php program to display the field names and their nicknames of a given collection. it is invoked from a shell prompt as php findcitation.php <contentdm collection alias>. <?php $alias = $argv[1]; $fielddata = file_get_contents("https://server16614.contentdm.oclc.org/dmwebservices/index.php?q=dmgetcollectionfieldinfo/$alias/xml"); // of course, change the line above to the url for your server instance… $fielddata = simplexml_load_string($fielddata); foreach ($fielddata->xpath('//field') as $record) { $resulta[] = array( 'nick' => (string) $record->nick, 'name' => (string) $record->name ); } $resultacount = count($resulta) 1; for ($ia=0;$ia<=$resultacount;$ia++) { $nick = $resulta[$ia]["nick"]; $name = $resulta[$ia]["name"]; echo "$name: $nick\n"; } ?> in order for our method to work, we need to assemble author, title, date, type, collection name, institution name, and description from each collection. for many collections, we can assemble reasonable surrogates; for some, fields are not available and so are simply marked as “n.a.” or “n.d.”. the information is then added to the citationfield field in the record, as the nicknamed aliases for creator | title | original date | type | collection name | publisher | description. figure 3. screen shot of a collection of collections record contentdm’s apis and the javascript program two listeners initiate the program. they check, respectively, for a brand new page and for a next or a previous page change. document.addeventlistener('cdm-item-page:ready', async function (e) { executescript(e.detail.collectionid,e.detail.itemid); }); document.addeventlistener('cdm-item-page:update', async function (e) { document.getelementbyid('citation').remove(); executescript(e.detail.collectionid,e.detail.itemid); }); executescript begins the program action itself. we need to access the information contained in our collection of collections in two steps. in the first step, we need to find out what the seven-element citationfield value is as well as retrieve the alias of the collection. function executescript(collectionname,recordnumber) { var cdmstr = "/digital/bl/dmwebservices/index.php?q=dmquery/p16614coll32/identi^" + collectionname + "^all^and/citati!identi/citati/1/1/0/0/0/0/json"; const request = new xmlhttprequest(); request.open('get', cdmstr); request.responsetype = 'json'; request.send(); request.onload = function() { var obj = request.response; generatediv(obj, recordnumber); } } executescript takes two values, the contentdm alias of the collection and the record number of the current record. it then makes an api call, dmquery (https://help.oclc.org/metadata_services/contentdm/advanced_website_customization/api_reference/contentdm_api/contentdm_server_api_functions_-_dmwebservices), querying the collection of collections (p16614coll32) to return the seven-element citation string and the collection alias. it then calls the next function, generatediv, to actually build the citation entries. generatediv is called with two values, the json results of the first dmquery api call and the record number of the current record. it then takes the citation field and splits it into its elemental parts. the resulting variables contain the nicknames of the relevant fields in the main record. var apirecord = obj['records']; var citation = apirecord[0]['citati']; var collectionid = apirecord[0]['identi']; if (citation) { var recorddata = citation.split("|"); var creatorlabel = recorddata[0]; var titlelabel = recorddata[1]; var datelabel = recorddata[2]; var typelabel = recorddata[3]; var collectionlabel = recorddata[4]; var institutionlabel = recorddata[5]; var descriptionlabel = recorddata[6]; generatediv assembles a new contentdm api call, dmgetiteminfo, to access the data of the record in question. var mainrecordstr = "/digital/bl/dmwebservices/index.php?q=dmgetiteminfo/" + collectionid + "/" + recordnumber + "/json"; it creates a container, a <div>, so that the newly assembled citation data has a home: const citationcontainer = document.createelement('div'); citationcontainer.id = 'citation'; …and actually makes the async query, resulting in a json record being returned: var xmlhttp = new xmlhttprequest(); xmlhttp.onreadystatechange = function() { if (this.readystate == 4 && this.status == 200) { var myobj = json.parse(this.responsetext); since we know the nicknames for the fields in question, we can now load variables with proper values: var author = myobj[creatorlabel]; if (author.length === undefined) { author = "n.a."; } var title = myobj[titlelabel]; var dateoriginal = myobj[datelabel]; if (dateoriginal.length === undefined) { dateoriginal = "n.d."; } var type = myobj[typelabel]; if (type.length === undefined) { type = "n.a."; } var collectionname = myobj[collectionlabel]; var institution = myobj[institutionlabel]; var description = myobj[descriptionlabel]; we will need to know what day today is, as well as what the address of the record is: var datetoday = new date().toisostring().slice(0, 10); var url = "http://www.idaillinois.org/digital/collection/" + collectionid + "/id/" + recordnumber; and we can then assemble the four citation formats (mla, chicago, apa, and ieee): var mla = "<hr />\n<h3 style=\"color:#aa0000;\">cite this image:</h3>\n<p><strong>mla (v.8)</strong>: " + author + ". \"" + title + ".\" " + institution + ": " + collectionname + " (illinois digital archives)," + dateoriginal + ", " + url + ". " + datetoday + ".</p>"; mla = mla + "<p><strong>chicago/turabian</strong>: " + author + ", <em>" + title + "</em>, " + collectionname + " (illinois digital archives), " + datetoday + ", " + url + ".</p>"; mla = mla + "<p><strong>apa (v.6)</strong>: " + author + ". (" + dateoriginal + "). " + title + " [" + description + "]. retrieved " + datetoday + ", from " + url + "</p>"; mla = mla + "<p><strong>ieee</strong>: [x] " + author + ", \"" + title + "\", " + institution + ": " + collectionname + " (illinois digital archives)," + dateoriginal + ". [online]. available: " + url + ". [accessed: " + datetoday + "].</p>"; citationcontainer.innerhtml = mla; document.queryselector('.itemview-itemviewcontainer').appendchild(citationcontainer); incidentally, it took some detective work on my part to find the different regions that make up a contentdm item record in order to place the citation <div>. i did a view source on an item record and found a likely css file (https://www.idaillinois.org/digital/f45fcea/app.css). i then experimented with the line document.queryselector('.itemview-itemviewcontainer').appendchild(citationcontainer); to find the proper region of the record for my citation <div>, using likely candidates until i hit upon the proper container. link to the whole script: https://www.idaillinois.org/customizations/global/js/citation.js the end result figure 4. end result of item metadata and citations below loading custom javascript programming into contentdm   because contentdm is now based on the react js library and environment, a second program is needed to properly load custom javascript programs. shane huddleston has provided a loader script, loader-1.1.js (https://cdmdemo.contentdm.oclc.org/digital/custom/recipedownloads) that will check the ready state of the page being loaded and then execute the listed js programs. here is an example of his loader program, which loads an iiif metadata viewer and then a citation generator: loader.js 'use strict'; // helper function to load js file and insert into dom // @param {string} src link to a js file // @returns promise function loadscript(src) { return new promise(function(resolve, reject) { const script = document.createelement('script'); /* script.crossorigin = 'anonymous'; */ script.src = src; script.onload = resolve; script.onerror = reject; document.head.appendchild(script); }); } // helper function to assemble full url of each js file function filepath(file) { return filedirectory + file; } // path corresponding to directory where js scripts are uploaded const filedirectory = '/customizations/global/js/'; // array containing file names of each js file const scriptfilestoload = [ 'field-insert-iiif.js', 'citationgenerator.js' ]; (function() { const allscripts = scriptfilestoload.map(filepath); allscripts.foreach(loadscript); document.addeventlistener('cdm-custom-page:ready', function(e) { if (e.detail.filename === 'field-insert-iiif') { loadscript('/customizations/global/js/field-insert-iiif.js'); } else if (e.detail.filename === 'citation') { loadscript('/customizations/global/js/citationgenerator.js'); } }); })();   loading your programs into contentdm in your server administration, go to collections->website->(login)->custom->custom scripts. browse to the location of your new js programs on your local computer. upload the citation.js program. press the save button, and then the publish button. the very last program to load is the loader.js program. save it, publish it, and it should work. please note that the programs must be loaded, saved, and published in this order. conclusion fortunately, we both were able to leverage different aspects of contentdm’s system to provide an effective way to generate citations on the fly for digital collections content. if your collections have the needed citation information in the same fields across collections, the first approach using the iiif manifest may be best for you. if it varies, try the second approach, and see what works for you. we encourage other institutions using contentdm to try our solutions or develop their own and share it with the wider contentdm user community. we hope that our solutions may lead to a citation recipe in the contentdm cookbook or ideally, even a new feature in a future release. about the authors jenn randles (jenn.randles@tn.gov) is enterprise data librarian at the state of tennessee center for enterprise data and analytics, formerly digital materials librarian at the tennessee state library & archives. jenn enjoys user experience design, building and maintaining digital collections and data catalogs, and finds satisfaction in making systems accessible for users at any level of tech knowledge. andrew bullen (abullen@ilsos.gov) is the information technology coordinator for the illinois state library. he works with the preservation and presentation of born-digital state documents, digitized cultural resources, and other digital humanities opportunities. he also works with optical musical recognition and crafting narratives from digital artifacts. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – opening the door: a first look at the oclc worldcat metadata api mission editorial committee process and structure code4lib issue 25, 2014-07-21 opening the door: a first look at the oclc worldcat metadata api libraries have long relied on oclc’s worldcat database as a way to cooperatively share bibliographic data and declare library holdings to support interlibrary loan services. as curator, oclc has traditionally mediated all interactions with the worldcat database through their various cataloging clients to control access to the information. as more and more libraries look for new ways to interact with their data and streamline metadata operations and workflows, these clients have become bottlenecks and an inhibitor of library innovation. to address some of these concerns, in early 2013 oclc announced the release of a set of application programming interfaces (apis) supporting read and write access to the worldcat database. these apis offer libraries their first opportunity to develop new services and workflows that directly interact with the worldcat database, and provide opportunities for catalogers to begin redefining how they work with oclc and their data. by terry reese introduction the past decade has represented a kind of “gold rush” for libraries and library developers as the need for data interoperability and access has been pushed to the forefront. as libraries and librarians work with researchers interested in doing large scale data mining, or find themselves pursuing their own research agendas around comprehensive discovery or service integration, software and services providing robust mechanisms for accessing and interacting with data have very quickly become the norm. and it is out of this environment that libraries are finding vendors and developers seeking to become partners in creating the scaffolding necessary to support future innovation and research within libraries[1][2], in addition to developing a number of conceptual services that begin to probe some of the ways in which library data can be remixed and repurposed for other uses. however, while many in libraries have benefited from the growing expectations around data access and availability, integrated library system (ils) and marc cataloging and metadata communities have largely been left out. unlike many other content services, the library metadata community largely relies on a single data service to provide cooperative bibliographic cataloging and cooperatively manage an organization’s holdings information to support global resource sharing – oclc’s worldcat. and while oclc has developed a number of tools and services built around worldcat[3][4][5], the ability to create, read, and update bibliographic and holdings data within worldcat has remained tightly controlled and coupled to the client software oclc has provided to the cataloging community. the result has been a lack of innovation and integration of workflows, as the need to work with worldcat hamstrung those efforts. however, in early 2013, oclc surprised the library community by releasing documentation to the worldcat metadata api[6]. the api provided, for the first time, the ability to read, create, and update bibliographic and holdings data within the worldcat database. for catalogers and the metadata community, it represented an opportunity to rethink how they interact with worldcat and an opportunity to rethink aging workflows and processes. working with members of the cataloging community, i released a number of libraries and components developed as a part of a joint research project investigating how marcedit could be used to provide direct integration with worldcat to simplify holdings management workflows[7] giving me an opportunity to closely examine where the strengths and weaknesses are of the current metadata api service. this paper will look at the current worldcat metadata api; its capabilities, limitations, and many quirks. getting started the worldcat metadata api is one part of a much larger portfolio of resources currently being made available as part of oclc’s developer network[8]. the developer’s network is oclc’s portal for information and documentation about oclc’s production and experimental development services. this includes access to information regarding apis, as well as customized services like viaf[9], classify[10], and services specific to oclc’s worldshare management services. in addition to information about services and apis provided by oclc, the developer’s network portal also provides access to an api explorer[11], a web application that allows potential developers an opportunity to demo specific apis as a way to supplement the available documentation. before getting started, a user interested in working with the metadata api will need to request a web services key. keys are assigned to specific users, are associated with one or more specific api services, and are requested through the portal [https://platform.worldcat.org/wskey]. the metadata api is somewhat unique, however, in that the service key only provides access to the api and authenticates the requests made against the server. to obtain access to the metadata api, one needs to have two accounts; 1) a worldcat.org account to request the key, and 2) a worldshare user account with cataloging/metadata permissions which are generally managed at the local level. without the correct cataloging permissions, users with valid service keys will be able to make requests against the server, but the requested actions will not be processed due to insufficient permissions to work with the worldcat database itself. authentication for long-time oclc users who may have utilized previous api offerings, the metadata api utilizes a very different authentication approach. oclc’s previous api services, like the worldcat search api, used a key that was appended to the url as part of the service request. for example, a request for hamlet would look like: http://www.worldcat.org/webservices/catalog/search/sru?query=srw.ti%3d%22hamlet%22&wskey=[mykey] in the case of many of oclc’s older services, the key was provided as part of the request. however, for the metadata api, oclc has introduced a new authentication strategy. rather than providing the key as part of the request, the metadata api requires authentication to utilize an hmac signature embedded within the authentication header of the request. hmac, or hash-based message authentication code[12], is an encrypted signature used as a mechanism to authenticate the data request made against the service. oclc’s implementation, based on oauth 1.0 specification[13], requires a very specific set of arguments, generated in the correct order, hashed utilizing an sha256 encryption algorithm and then encoded in base64 when embedded into the authorization request header. to generate the hmac signature, the following data must be present within the request: the wskey a unix posix timestamp calculated at the time of the request a nonce value, or a random set of numbers, unique for the specific request the request entity-body hash. at this point, this is always a blank line. the http request method. this value must be in uppercase. the string: www.oclc.org the string: 443 the string: /wskey the query component of the request uri. an empty line   every request made against the metadata api service must generate a new hmac signature for the specific request, otherwise the request will fail. each request made against the metadata api service must have a unique nonce value and a unix posix timestamp that corresponds to one and only one request, or the request will fail. finally, each line of the signature must include a new line (\n) or the request will fail. once the signature is calculated, that is provided as one value of the authorization header. the full header includes the following hash: schema url clientid timestamp nonce signature principalid principleidns   so, assuming the parameters: wskey: [your wskey] timestamp: 1399498027 nonce value: 898581797066788588888373908665858579788865896874726973898675 method: get query string: ?holdingcode=ore&classificationscheme=libraryofcongress&holdinglibrarycode=main principleid: [your id] principleidns: [your value] secret key: [your secret key]   the process used to generate the request looks like the following: prehashed key: [your wskey] 1399498027 898581797066788588888373908665858579788865896874726973898675 get www.oclc.org 443 /wskey holdingcode=ore&classificationscheme=libraryofcongress&holdinglibrarycode=main   this pre-hashed key would then be hashed via an sha256 algorithm utilizing your secret key and the pre-hashed values as a string. this is the generated signature. the signature would then be embedded into the authorization header, which would look like the following: http://www.worldcat.org/wskey/v2/hmac/v1 clientid=”[your wskey]”, timestamp=”1399498027″, nonce=”898581797066788588888373908665858579788865896874726973898675″, signature=”yuubag9gsjl3dizliom2g3m48sx3en5vbczbxjra+lq=”, principalid=”[your principleid]”, principalidns=”[your principleidns]” fortunately, code exists to simplify oclc’s hmac authentication process. for use in marcedit, a public domain library, found at: https://github.com/reeset/oclc_api provides a c# implementation of oclc’s api services, encapsulating authentication. likewise, oclc has released a ruby gem at: https://github.com/oclc-developer-network/oclc-auth-ruby that provides a ruby wrapper around oclc’s authentication and a code library available at: https://github.com/reeset/wc_metadata_api, provides a ruby wrapper around the various metadata api functions. while i will not provide code examples in languages other than ruby and c#, oclc has also released authentication code libraries for php (https://github.com/oclc-developer-network/oclc-auth-php) and python (https://github.com/oclc-developer-network/oclc-auth-python). c#: authenticating string srecord = ""; string principleid = "[your id]"; string principleldns = "[your dns]"; string wskey = "[your key]"; string secret = "[your secret]"; string holdingsid = "ocpsb"; string instsymbol = "ocpsb"; string schema = "libraryofcongress"; oclc_api.oclc_api_metadata obj_om = new oclc_api.oclc_api_metadata(wskey, secret, principleid, principleldns);   ruby: authenticating require 'net/http' require 'oclc/auth' wskey = oclc::auth::wskey.new('api-key', 'api-key-secret') url = 'https://worldcat.org/bib/data/823520553?classificationscheme=libraryofcongress&amp;holdinglibrarycode=main' uri = uri.parse(url) request = net::http::get.new(uri.request_uri) request['authorization'] = wskey.hmac_signature('get', url, :principal_id =&gt; 'principal-id', :principal_idns =&gt; 'principal-idns') worldcat metadata api functionality the worldcat metadata api provides the ability to create, edit, and sometimes delete very specific classes of data stored within the worldcat database. the api makes logical use of the standard http methods (put, get, delete, and post actions) to define specific actions performed by the api calls and utilize a specific base uri corresponding to each functional group. additionally, the api utilizes http status codes to indicate the outcome of each request call. functionality can be broken into four primary areas: actions against master bibliographic records, actions against local bibliographic data records, actions against institutional holdings, and organizational information. as noted earlier, the ability to perform each of these specific actions requires users not only to have the keys necessary to authenticate against the oclc worldcat metadata api, but the username to which the api key is attached must also have the necessary cataloging permissions within the worldcat database to make the requested changes. a description of each functionality group can be found below. working with master bibliographic records the metadata api provides the ability to create, read, and update master bibliographic records in the worldcat database. missing from the feature set is the ability to delete bibliographic records– an option generally reserved for oclc’s quality control staff. in looking at the api documentation, one of the most impressive parts is the wide range of request and response formats that the api supports[14]. the api has the ability to provide various flavors of xml and json formatted results. request formats are more controlled, with the api expecting data submitted into the api to be in either marcxml or oclc’s cdf (common data format) serialization in either xml or json format. read records: baseurl: https://worldcat.org/bib/data request type: application/vnd.oclc.marc21+xml c#: read record #823520553 string srecord = ""; string oclcnumber = "823520553"; string principleid = "[your id]"; string principleldns = "[your dns]"; string wskey = "[your key]"; string secret = "[your secret]"; string holdingsid = "ocpsb"; string instsymbol = "ocpsb"; string schema = "libraryofcongress"; oclc_api.oclc_api_metadata obj_om = new oclc_api.oclc_api_metadata(wskey, secret, principleid, principleldns); obj_om.worldcat_service_uri = @"https://worldcat.org/bib/data/"; obj_om.instsymbol = instsymbol; string url = "https://worldcat.org/bib/data/823520553?holdingcode=ore&amp;classificationscheme=libraryofcongress&amp;holdinglibrarycode=main"; bool results = obj_om.worldcatgetrecord(schema, holdingsid, instsymbol, oclcnumber, out srecord); system.windows.forms.messagebox.show(srecord);   ruby: read record #823520553 require 'rubygems' require 'wc_metadata_api' key = '[your key]' secret = '[your secret]' principalid = '[your principal_id]' principaldns = '[your principal_idns]' schema = 'libraryofcongress' holdinglibrarycode='[your holding code]' instsymbol = '[your oclc symbol]' client = wc_metadata_api::client.new(:wskey =&gt; key, :secret =&gt; secret, :principalid =&gt; principalid, :principaldns =&gt; principaldns, :debug =&gt;false) response = client.worldcatgetbibrecord(:oclcnumber =&gt; '823520553', :holdinglibrarycode =&gt; holdinglibrarycode, :schema =&gt; schema, :instsymbol =&gt; instsymbol) puts response   marcxml response format: <?xml version="1.0" encoding="utf-8"?> <entry xmlns="http://www.w3.org/2005/atom"> <content type="application/xml"> <response xmlns="http://worldcat.org/rb" mimetype="application/vnd.oclc.marc21+xml"> <record xmlns="http://www.loc.gov/marc21/slim"> <leader>00000cam a2200000ia 4500</leader> <controlfield tag="001">ocn823520553</controlfield> <controlfield tag="003">ocolc</controlfield> <controlfield tag="005">20140207112204.5</controlfield> <controlfield tag="008">120827s2012 nyua 000 0 eng d</controlfield> <datafield tag="040" ind1=" " ind2=" "> <subfield code="a">ocpsb</subfield> <subfield code="c">ocpsb</subfield> <subfield code="d">ocwms</subfield> <subfield code="d">ocpsb</subfield> </datafield> <datafield tag="035" ind1=" " ind2=" "> <subfield code="a">(ocolc)823520553</subfield> </datafield> <datafield tag="049" ind1=" " ind2=" "> <subfield code="a">ocpsb</subfield> </datafield> <datafield tag="100" ind1="0" ind2=" "> <subfield code="a">oclc recordbuilder</subfield> </datafield> <datafield tag="245" ind1="0" ind2="0"> <subfield code="a">record builder added this test record on 03/04/13 08:36:54</subfield> </datafield> <datafield tag="500" ind1=" " ind2=" "> <subfield code="a">field added by tr.</subfield> </datafield> <datafield tag="500" ind1=" " ind2=" "> <subfield code="a">for developers house</subfield> </datafield> <datafield tag="500" ind1=" " ind2=" "> <subfield code="a">test record -do not use</subfield> </datafield> <datafield tag="500" ind1=" " ind2=" "> <subfield code="a">note field added by tr 10/18/2013</subfield> </datafield> <datafield tag="500" ind1=" " ind2=" "> <subfield code="a">second note field added by tr 10/19/2013</subfield> </datafield> <datafield tag="500" ind1=" " ind2=" "> <subfield code="a">third note field added by tr 12/6/2013.</subfield> </datafield> </record> </response> </content> <id>http://worldcat.org/oclc/823520553</id> <link href="http://worldcat.org/oclc/823520553"></link> </entry>   create record: c#: create new master record string principleid = "[your principleid]"; string principleldns = "[your principledns]"; string wskey = "[your key]"; string secret = "[your secret]"; string holdingsid = "main"; //4 letter holding code string instsymbol = "ocpsb"; string schema = "libraryofcongress"; oclc_api.oclc_api_metadata obj_om = new oclc_api.oclc_api_metadata(wskey, secret, principleid, principleldns); obj_om.worldcat_service_uri = @"https://worldcat.org/bib/data/"; obj_om.instsymbol = instsymbol; string srecord = "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n" + "<record xmlns=\"http://www.loc.gov/marc21/slim\">\n" + "<leader>00000nam a2200000ia 4500</leader>\n" + "<controlfield tag=\"008\">120827s2012 nyua 000 0 eng d</controlfield>\n" + "<datafield tag=\"040\" ind1=\" \" ind2=\" \">\n" + "<subfield code=\"a\">ocpsb</subfield>\n" + "<subfield code=\"c\">ocpsb</subfield>\n" + "</datafield>\n" + "<datafield tag=\"100\" ind1=\"0\" ind2=\" \">\n" + "<subfield code=\"a\">reese, terry</subfield>\n" + "</datafield>\n" + "<datafield tag=\"245\" ind1=\"0\" ind2=\"0\">\n" + "<subfield code=\"a\">record builder added this test record on 05/08/2014</subfield>\n" + "</datafield>\n" + "<datafield tag=\"500\" ind1=\" \" ind2=\" \">\n" + "<subfield code=\"a\">original record has one field.</subfield>\n" + "</datafield>\n" + "</record>\n"; bool results = obj_om.worldcataddrecord(srecord, instsymbol, holdingsid, schema, holdingsid); system.windows.forms.messagebox.show(obj_om.lastresponsecode);   ruby: create new master record require 'rubygems' require 'wc_metadata_api' key = '[your key]' secret = '[your secret]' principalid = '[your principal_id]' principaldns = '[your principal_idns]' schema = 'libraryofcongress' holdinglibrarycode='[your holding code]' instsymbol = '[your oclc symbol]' rec = "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n" + "<record xmlns=\"http://www.loc.gov/marc21/slim\">\n" + "<leader>00000nam a2200000ia 4500</leader>\n" + "<controlfield tag=\"008\">120827s2012 nyua 000 0 eng d</controlfield>\n" + "<datafield tag=\"040\" ind1=\" \" ind2=\" \">\n" + "<subfield code=\"a\">ocpsb</subfield>\n" + "<subfield code=\"c\">ocpsb</subfield>\n" + "</datafield>\n" + "<datafield tag=\"100\" ind1=\"0\" ind2=\" \">\n" + "<subfield code=\"a\">reese, terry</subfield>\n" + "</datafield>\n" + "<datafield tag=\"245\" ind1=\"0\" ind2=\"0\">\n" + "<subfield code=\"a\">record builder added this test record on 05/14/2014</subfield>\n" + "</datafield>\n" + "<datafield tag=\"500\" ind1=\" \" ind2=\" \">\n" + "<subfield code=\"a\">original record has one field.</subfield>\n" + "</datafield>\n" + "</record>\n" client = wc_metadata_api::client.new(:wskey => key, :secret => secret, :principalid => principalid, :principaldns => principaldns, :debug =>false) response = client.worldcataddbibrecord(:holdinglibrarycode => holdinglibrarycode, :schema => schema, :instsymbol => instsymbol, :xrecord => rec) puts client.lastresponsecode.body   response message: <?xml version="1.0" encoding="utf-8"?> <entry xmlns="http://www.w3.org/2005/atom"> <content type="application/xml"> <response xmlns="http://worldcat.org/rb" mimetype="application/vnd.oclc.marc21+xml"> <record xmlns="http://www.loc.gov/marc21/slim"> <leader>00000nam a2200000ia 4500</leader> <controlfield tag="001">ocn879376100</controlfield> <controlfield tag="003">ocolc</controlfield> <controlfield tag="005">20140508111006.3</controlfield> <controlfield tag="008">120827s2012 nyua 000 0 eng d</controlfield> <datafield tag="040" ind1=" " ind2=" "> <subfield code="a">ocpsb</subfield> <subfield code="c">ocpsb</subfield> </datafield> <datafield tag="035" ind1=" " ind2=" "> <subfield code="a">(ocolc)879376100</subfield> </datafield> <datafield tag="049" ind1=" " ind2=" "> <subfield code="a">main</subfield> </datafield> <datafield tag="100" ind1="0" ind2=" "> <subfield code="a">reese, terry</subfield> </datafield> <datafield tag="245" ind1="0" ind2="0"> <subfield code="a">record builder added this test record on 05/08/2014 recordbuilder add original test: 2014-05-08 11:10:06.093</subfield> </datafield> <datafield tag="500" ind1=" " ind2=" "> <subfield code="a">original record has one field.</subfield> </datafield> <datafield tag="994" ind1=" " ind2=" "> <subfield code="a">10</subfield> </datafield> </record> </response> </content> <id>http://worldcat.org/oclc/879376100</id> <link href="http://worldcat.org/oclc/879376100"></link> </entry>   when successful, the <id> and <link> will point to the new oclc record that has been created. when an error occurs, those values will be replaced by: <oclc:error>; providing information regarding the specific errors encountered while performing the operation. for example: <oclc:error> <oclc:code type="application">cat-other</oclc:code> <oclc:message>new record required</oclc:message> </oclc:error>   updating records: c#: updating master bib record #879376100 string principleid = "[your principleid]"; string principleldns = "[your principledns]"; string wskey = "[your key]"; string secret = "[your secret]"; string instsymbol = "ocpsb"; string holdingsid = "main"; string schema = "libraryofcongress"; oclc_api.oclc_api_metadata obj_om = new oclc_api.oclc_api_metadata(wskey, secret, principleid, principleldns); obj_om.worldcat_service_uri = @"https://worldcat.org/bib/data/"; obj_om.instsymbol = instsymbol; string srecord = "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n" + "<record xmlns=\"http://www.loc.gov/marc21/slim\">\n" + "<leader>00551cam a2200169ia 4500</leader>\n" + "<controlfield tag=\"001\">879376100</controlfield>\n" + "<controlfield tag=\"003\">ocolc</controlfield>\n" + "<controlfield tag=\"005\">20140508114136.1</controlfield>\n" + "<controlfield tag=\"008\">120827s2012 nyua 000 0 eng d</controlfield>\n" + "<datafield tag=\"040\" ind1=\" \" ind2=\" \">\n" + "<subfield code=\"a\">ocpsb</subfield>\n" + "<subfield code=\"c\">ocpsb</subfield>\n" + "</datafield>\n" + "<datafield tag=\"035\" ind1=\" \" ind2=\" \">\n" + "<subfield code=\"a\">(ocolc)879376100</subfield>\n" + "</datafield>\n" + "<datafield tag=\"049\" ind1=\" \" ind2=\" \">\n" + "<subfield code=\"a\">main</subfield>\n" + "</datafield>\n" + "<datafield tag=\"100\" ind1=\"0\" ind2=\" \">\n" + "<subfield code=\"a\">reese, terry</subfield>\n" + "</datafield>\n" + "<datafield tag=\"245\" ind1=\"0\" ind2=\"0\">\n" + "<subfield code=\"a\">record builder added this test record on 05/08/2014 recordbuilder add original test: 2014-05-08 11:10:06.093</subfield>\n" + "</datafield>\n" + "<datafield tag=\"500\" ind1=\" \" ind2=\" \">\n" + "<subfield code=\"a\">original record has one field.</subfield>\n" + "</datafield>\n" + "<datafield tag=\"500\" ind1=\" \" ind2=\" \">\n" + "<subfield code=\"a\">updated on 5/8/2014 at 11:29 am est. field added.</subfield>\n" + "</datafield>\n" + "<datafield tag=\"500\" ind1=\" \" ind2=\" \">\n" + "<subfield code=\"a\">updated on 5/8/2014 @ 11:53 am est.</subfield>\n" + "</datafield>\n" + "</record>\n"; bool results = obj_om.worldcatupdaterecord(srecord, instsymbol, schema, holdingsid); system.windows.forms.messagebox.show(obj_om.lastresponsecode);   ruby example: require 'rubygems' require 'wc_metadata_api' key = '[your key]' secret = '[your secret]' principalid = '[your principal_id]' principaldns = '[your principal_idns]' schema = 'libraryofcongress' holdinglibrarycode='[your holding code]' instsymbol = '[your oclc symbol]' rec = "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n" + "<record xmlns=\"http://www.loc.gov/marc21/slim\">\n" + "<leader>00000nam a2200000ia 4500</leader>\n" + "<controlfield tag=\"001\">879649505</controlfield>\n" + "<controlfield tag=\"003\">ocolc</controlfield>\n" + "<controlfield tag=\"005\">20140514152217.2</controlfield>\n" + "<controlfield tag=\"008\">120827s2012 nyua 000 0 eng d</controlfield>\n" + "<datafield tag=\"040\" ind1=\" \" ind2=\" \">\n" + "<subfield code=\"a\">ocpsb</subfield>\n" + "<subfield code=\"c\">ocpsb</subfield>\n" + "</datafield>\n" + "<datafield tag=\"100\" ind1=\"0\" ind2=\" \">\n" + "<subfield code=\"a\">reese, terry</subfield>\n" + "</datafield>\n" + "<datafield tag=\"245\" ind1=\"0\" ind2=\"0\">\n" + "<subfield code=\"a\">record builder added this test record on 05/14/2014</subfield>\n" + "</datafield>\n" + "<datafield tag=\"500\" ind1=\" \" ind2=\" \">\n" + "<subfield code=\"a\">original record has one field.</subfield>\n" + "</datafield>\n" + "<datafield tag=\"500\" ind1=\" \" ind2=\" \">\n" + "<subfield code=\"a\">second field added</subfield>\n" + "</datafield>\n" + "</record>\n" client = wc_metadata_api::client.new(:wskey => key, :secret => secret, :principalid => principalid, :principaldns => principaldns, :debug =>false) response = client.worldcatupdatebibrecord(:holdinglibrarycode => holdinglibrarycode, :schema => schema, :instsymbol => instsymbol, :xrecord => rec) puts client.lastresponsecode.body   the response format when updating records is identical to the response format received when creating new records. the primary difference between the update record and new record functions is the requirements around certain data elements within the bibliographic record being submitted. for example, when updating records, the oclc control number must be present, without prefixes, in the 001 field. likewise, the last transaction date, found in the 005, must be identical to the data found in worldcat. these two values are specific to record updating and must be present for any request to begin the update process. it’s also worth noting that records submitted via the metadata api are validated using oclc’s validation service. unfortunately, this service is not provided as part of the metadata api as published, so when utilizing the api, there is no method to determine, prior to submitting data to oclc, if the provided metadata is valid. and in the case of the api, validation occurs both in respect to the format of the data being submitted (is it well-formed?), as well as in relationship to the record’s adherence to marc21 rules. working with local bibliographic data records local bibliographic data records provide a way for organizations that use worldcat for cataloging to record locally-maintained information related to a master bibliographic record. local bibliographic data records are only visible to the particular institution with which they are associated. local bibliographic data records are somewhat different from master bibliographic records in that they have their own oclc record numbers, but they are linked to a master bibliographic number via marc field 004. local bibliographic data records are used for a number of purposes within the worldcat database, though the most common use is by libraries utilizing oclc’s worldshare platform. local bibliographic data records are used to store local notes relating to access, processing, etc. as well as local access points, like organization-specific urls for access. while oclc has set forth a set of best practices around local bibliographic data record usage, practical use of these records has demonstrated that nearly any data, including local subjects, added main entries, notes, etc., can be stored within a local bibliographic record. oclc’s metadata api provides the ability to interact with an institution’s local bibliographic data record. like the functions that support the creation, reading, and editing of master bibliographic data, api users can create, read, and edit local bibliographic data records as well. the one distinct difference is that the api also provides users the ability to delete local bibliographic data records. when working with local bibliographic data records via the api, the response and request formats are identical to the functions used when editing master bibliographic data. the real differences between the two functional groups are around the base url of the service and the type of data being sent through the service. as noted above, local bibliographic data records have some distinct differences from master bibliographic records because these records are linked to an existing bibliographic record. because of this, these records require three specific data fields: 004 – which links to an existing master oclc bibliographic record 935 – a locally generated system number (can be a timestamp, but must be unique) 940 – field that identifies the oclc institution generating the record for users generating new local bibliographic data records, the following template would always be used: <?xml version="1.0" encoding="utf-8" ?> <record xmlns="http://www.loc.gov/marc21/slim"> <leader>00000n a2200000 4500</leader> <controlfield tag="004">[master oclc number]</controlfield> <datafield tag="935" ind1=" " ind2=" "> <subfield code="a">[your local generated identifier – can be a timestamp]</subfield> </datafield> <datafield tag="940" ind1=" " ind2=" "> <subfield code="a">[your oclc symbol]</subfield> </datafield> </record> using this template, users can include a wide range of local information for submission back into the worldcat database. and like the api functions that interact with the master bibliographic data record, a local bibliographic data record is passed between the client and oclc utilizing either marcxml or oclc’s cdf in xml or json format. likewise, when editing an existing local bibliographic data record, the same caveats relating to the 005, the inclusion of the 001, and data validation that applied when working with master bibliographic data records, apply to local bibliographic data records as well. local bibliographic data function list: c# headers: add a local bibliographic record: bool worldcataddlocalbibrecord(string xrecord, string inst, string schema, string holdingcode) delete a local bibliographic record: bool worldcatdeletelocalbibrecord(string xrecord, string schema, string holdingcode, string instsymbol) read a local bibliographic record: bool worldcatreadlocalbibrecord(string schema, string holdingcode, string oclcnumber, out string srecord) update a local bibliographic record: bool worldcatupdatelocalbibrecord(string xrecord, string inst, string schema, string holdingcode) ruby headers: wc_metadata_api::client.worldcataddlocalbibrecord(:holdinglibrarycode, :schema, :instsymbol, :xrecord) wc_metadata_api::client.worldcatdeletelocalbibrecord(:holdinglibrarycode, :schema, :instsymbol, :xrecord) wc_metadata_api::client.worldcatgetlocalbibrecord(:oclcnumber, :holdinglibrarycode, :schema, :instsymbol) wc_metadata_api::client.worldcatupdatelocalbibrecord(:holdinglibrarycode, :schema, :instsymbol, :xrecord) working with institutional holdings data worldcat’s central role in facilitating interlibrary loan between member libraries highlights the importance that accurate holdings information plays for libraries. the challenge for libraries has always been that library collections are not static entities. libraries regularly need to add and withdraw large batches of items, and the processes necessary for maintaining accurate holdings information during these large processing events have never been well supported by the tools provided by oclc. for many libraries, large withdrawal processes require significant effort to manually remove holdings from individual bibliographic records, and holdings for e-versions of content may never be attached within oclc simply because the holdings for these materials are characteristically ephemeral. however, issues related to holdings management only multiply when dealing with non-monographic materials and the need to publish patterns and holdings information regarding specific issues and volumes, materials currently handled through the creation of local holdings records. oclc’s metadata api provides limited support for holdings data manipulation, restricting holdings management to institutional holdings attached to a bibliographic record. at this time, the ability to manage local holding records (lhrs) is not supported as part of this service. and yet, even with its limited functionality, the metadata api opens up the possibility of rethinking workflows around large weeding projects, as the api supports the ability to unset large numbers of institutional holdings attached to bibliographic records. the metadata api supports two actions: 1) add/set holdings on a bibliographic record and 2) delete/unset holdings on a bibliographic record. the ability to query a record for holdings status is not a part of the current api functionality. add/set holdings c# example on oclc record #: 823520553 string oclcnumber = "823520553"; string principleid = "[your principleid]"; string principleldns = "[your princpledns]"; string wskey = "[your key]"; string secret = "[your secret]"; string instsymbol = "[your oclc symbol]"; string holdingsid = "[your holdings code]"; string schema = "libraryofcongress"; oclc_api.oclc_api_metadata obj_om = new oclc_api.oclc_api_metadata(wskey, secret, principleid, principleldns); obj_om.worldcat_service_uri = @"https://worldcat.org/ih/data"; obj_om.instsymbol = instsymbol; bool results = obj_om.worldcataddholding(schema, holdingsid, oclcnumber); system.windows.forms.messagebox.show(obj_om.debug_info);   ruby example on oclc record #: 823520553 require 'rubygems' require 'wc_metadata_api' key = '[your key]' secret = '[your secret]' principalid = '[your principal_id]' principaldns = '[your principal_idns]' schema = 'libraryofcongress' holdinglibrarycode='[your holding code]' instsymbol = '[your oclc symbol]' client = wc_metadata_api::client.new(:wskey => key, :secret => secret, :principalid => principalid, :principaldns => principaldns, :debug =>false) response = client.worldcataddholdings(:instsymbol => instsymbol, :holdinglibrarycode => holdinglibrarycode, :schema => schema, :oclcnumber => '823520553') puts client.lastresponsecode.body   delete/unset holdings c# example on oclc record #: 823520553 string oclcnumber = "823520553"; string principleid = "[your principleid]"; string principleldns = "[your princpledns]"; string wskey = "[your key]"; string secret = "[your secret]"; string instsymbol = "[your oclc symbol]"; string holdingsid = "[your holdings code]"; string schema = "libraryofcongress"; oclc_api.oclc_api_metadata obj_om = new oclc_api.oclc_api_metadata(wskey, secret, principleid, principleldns); obj_om.worldcat_service_uri = @"https://worldcat.org/ih/data"; obj_om.instsymbol = instsymbol; bool results = obj_om.worldcatdeleteholding(schema, holdingsid, oclcnumber); system.windows.forms.messagebox.show(obj_om.debug_info);   ruby example on oclc record #: 823520553 require 'rubygems' require 'wc_metadata_api' key = '[your key]' secret = '[your secret]' principalid = '[your principal_id]' principaldns = '[your principal_idns]' schema = 'libraryofcongress' holdinglibrarycode='[your holding code]' instsymbol = '[your oclc symbol]' client = wc_metadata_api::client.new(:wskey => key, :secret => secret, :principalid => principalid, :principaldns => principaldns, :debug =>false) response = client.worldcatdeleteholdings(:instsymbol => instsymbol, :holdinglibrarycode => holdinglibrarycode, :schema => schema, :oclcnumber => '823520553') puts client.lastresponsecode.body   when querying the holdings functions, the response format is represented through http status codes. if the operation completes successfully, the operation will return an http status 200 with no additional information provided. however, if an error occurs, the http status code will be set to reflect the operation failure, and will be accompanied by an xml package embedding the response code from the server. for example, attempting to set holdings on a bibliographic record where the organization’s holdings are already present will result in a http status code of 409, and the accompanying error message will be returned in the response body:   <?xml version="1.0" encoding="utf-8" standalone="yes"?> <error xmlns="http://worldcat.org/xmlschemas/response"> <code type="application">ws-409</code> <message>trying to set hold while holding already exists</message> </error>   the ability to retrieve an institution’s holdings codes appears to be the first of what i hope will be a wider selection of api functions provided to gather information about a specific user or the organization that a user happens to be attached to, enabling software to simplify many of the configuration challenges around using the api. the institutional holding code is one value that is a part of almost all the metadata api calls. these are fourletter, alphanumeric codes that represent the physical locations with an organization. within worldcat, each code is typically used to identify a location, collection type, or category of materials. within marc, this data shows up in the 049, or local holdings field (https://oclc.org/bibformats/en/0xx/049.html). this information is used within worldcat and worldcat local to display where an item is physically stored at a particular institution. since most catalogers and developers may not know all of an organization’s holding codes, the ability to retrieve these values for use when making other api calls is important. this api provides a single value that returns an xml response encapsulating all of the corresponding holding codes for the institution. c# example: string wskey = "[your wskey]"; string wskey_secret = "[your secret key]"; string principalid = "[your principalid]"; string principalidns = "[your principaldns]"; string keyvalues = ""; oclc_api.oclc_api_metadata obj_om = new oclc_api.oclc_api_metadata(); obj_om.worldcat_service_uri = oclc_api.oclc_api_metadata.holdings_codes_data_uri; // "https://worldcat.org/bib/holdinglibraries"; obj_om.principleid = principalid; obj_om.principleidns = principalidns; obj_om.wskey = wskey; obj_om.secret_key = wskey_secret; string[] m = obj_om.worldcatretrieveholdingcodes("[your oclc symbol]");   ruby example: require 'rubygems' require 'wc_metadata_api' key = '[your key]' secret = '[your secret]' principalid = '[your principal_id]' principaldns = '[your principal_idns]' schema = 'libraryofcongress' holdinglibrarycode='[your holding code]' instsymbol = '[your oclc symbol]' client = wc_metadata_api::client.new(:wskey => key, :secret => secret, :principalid => principalid, :principaldns => principaldns, :debug =>false) response = client.worldcatretrieveholdingcodes(:instsymbol => instsymbol) puts response   gaps in the implementation oclc’s implementation of a metadata api against the worldcat database has the potential to unlock a wide range of innovations around technical services and metadata processing. at the same time, while the first release of the metadata api includes a number of useful and interesting features, there are a number of gaps in the service that will limit some of that potential. in working with the metadata api, i believe that the significant gaps affecting the service can be broken into four broad categories: 1) scalability of the service, 2) service level validation, 3) service testing and sandboxing, and 4) documentation. scalability concerns after working with metadata api, it is readily apparent that the service was not developed with batch metadata editing in mind. the apis interact on a record by record basis, meaning that requests against the api must be made at the individual record level. while developers can, and have, written batch processing applications utilizing these api, the current process requires a great deal of back and forth between oclc and the client, with each individual record needing to be submitted, validated, and then a response providing information around the success or failure of a particular submission returned back to the client for parsing. in a batch processing environment, this type of interaction costs a great deal of time, and complicates the processing and reporting on the status of a batch upload as validation of records occurs within real time. this means that the client must make a decision either stop the processing of an entire batch on any error, removing any successfully processed items prior to restarting the update, or provide a report at the end that requires manual intervention to isolate just the data that needs further processing. validation concerns one of the most glaring omissions around the oclc metadata api services is around validation. currently, record validation occurs when metadata is sent through the api to update or create a new record. in an environment where the api is used to edit one record at a time, this type of validation makes sense. however, within a production environment where multiple records may need to be passed through the api services, the inability to pre-validate records prior to submitting bibliographic data through the api artificially complicates the submission and pre-validation process. additionally, catalogers familiar with editing records via oclc’s cataloging tools have the ability to “lock” bibliographic data for editing. locking records freezes the bibliographic record within worldcat, allowing the cataloger the ability to edit the record without worrying that the information may be changed by someone else. users utilizing the metadata api lack this ability to “lock” metadata records, meaning that run they risk of making substantial changes to sets of records, and then being unable to upload that data back to worldcat because a record has been changed. these validation issues are significant problems that place huge limits on the usability of the api, as well as raising confidence barriers to users of the api that the changes made via the api will be processed. aside from data validation, there remain large gaps around user validation. presently, oclc offers no service to validate that a user has a specific set of permissions to perform specific actions within the worldcat database. this raises interesting challenges, since a user may have all the permissions necessary to utilize the api, but none of the permissions necessary to perform the requested actions within the worldcat database. unfortunately, there is no way for a user or application to determine the permissions accessible to a user until the application attempts to perform a specified action and an error is generated. service testing and sandboxing one of the curious choices that oclc has made in the development and implementation of their api is the lack of a true sandboxed environment for doing code development[15]. when making changes or adding records via the metadata api, these changes happen in real-time, within the production worldcat database. the practical impact of this is that user testing and development must be done in a carefully controlled environment because depending on the user editing permissions that a particular worldcat service key is attached to, the testing software could have the capacity to make substantial and harmful changes to master records or an institution’s holdings. in my own experience while developing integrations between marcedit and oclc’s metadata api, i found that the development process needed to be significantly restrained because each new test record that was added, each new local bibliographic record added or edited, would eventually need to be removed from the system. and in the case of new master bibliographic records, those new records could only be removed by oclc. the practical outcome of this restrained development was that the integration between the two services was much less “baked” than i would have normally preferred, because the usual testing processing where millions of records are processed through new features simply couldn’t be accommodated. documentation like many initial projects and services, the lack of documentation associated with the metadata api raises the learning curve for using the service. while oclc does provide some brief, high-level documentation around the service, a number of assumptions and required values simply aren’t documented. in many cases, i found it easier to send invalid requests to the services and parse the error messages to determine allowed values, simply because that information didn’t exist in the documentation. fortunately, this is slowly changing as more time passes since the metadata api release and as oclc continues to add information based on feedback that they receive. likewise, because the metadata api assumes some level of knowledge about worldcat’s cataloging rules and validation practices, non-catalogers will likely have a difficult time navigating the unwritten rules around necessary fields, placements, and simply the language necessary to ask the questions of why a seemingly valid request might be failing. fortunately, oclc has been endeavoring to reduce the difficulty in getting started. some of these efforts have been around the release of code libraries on the oclc developer network’s github account[16] to provide working code libraries around authentication. others have been around the release of tools like the oclc api explorer which provides an opportunity to test drive specific functionality prior to working with it. using marcedit as a proof of concept one of the most exciting aspects of the release of the worldcat metadata api is the potential to integrate a wide range of local libraries’ services directly with oclc’s worldcat. for example, the release of the api provides a clear pathway for metadata creation into worldcat from institutional or content repositories. makers of repositories like dspace, contentdm, islandora, or any number of hydra-based platforms could utilize the metadata api not only to simplify the flow of information into oclc’s worldcat, but also as a mechanism for utilizing worldcat to populate metadata within the repositories themselves. likewise, these apis could be used by local technical services departments interested in created customized workflows around specific material processing, or for use by the libraries’ ils community to provide deeper integration between the local library system community and worldcat database. in examining how the current worldcat metadata api’s could be exploited to help augment existing technical services workflow, i investigated how these services could be integrated into marcedit to facilitate a handful of batch processing tasks related to holdings management and record upload. utilizing the c# library referenced earlier in this article, i created two new tools: 1) batch bibliographic records tool for creating, updating, or deleting both master and local bibliographic data and 2) batch holdings management tool, for adding or deleting organizational holdings on bibliographic records. these tools provide real-life examples of how existing metadata workflows can be modified or enhanced utilizing the worldcat metadata api to interact intimately between various systems. creating/editing bibliographic records marcedit’s oclc bibliographic records uploader has the ability to process both local and master bibliographic record sets. the functionality was implemented as a stand-alone tool (figure 1) and as part of a set of integrated workflows within the marceditor for streamlining real-time data edits. the tool provides users with a simple-to-use mechanism for batch processing bibliographic records, utilizing the metadata api as the mechanism to communicate between marcedit and worldcat. of course, there were some challenges in developing this service, the most notable being that the api services don’t work well within a batch records environment. in creating this service, marcedit must break down the batch of marc records to their individual record components, convert the data into utf8 (if necessary) and transform the data into a usable format for interacting with the api (in marcedit’s case, the format chosen was marcxml). the application must then send each record one-by-one to oclc, and await a response to determine the success or failure of the operation. because this is a batch operation and there is a potential that a user may want to process tens of thousands of records, marcedit creates a thread queue, initiating multiple requests at a time, tracking and reordering responses so that a report can be generated and presented to the user on completion. figure 1: master and local bibliographic records batch processing tool batch processing worldcat holdings data while bibliographic metadata present a number of challenges, the processing of holdings data is much more straightforward. however, again, the process is complicated by the api’s current single record design and the inability to determine current holding status of an item. the lack of these two options necessitated certain trade-offs, which are apparent on the tool’s binary user interface (figure 2). the tool has no way besides attempting to set a local holdings and trap for an error to determine the holding status of a record. it would be more effective if it could modify an existing holdings record based on its present status to off or on, or allow for the creation of reports to query holdings status of specific records. again, like the bibliographic updating tool, the batch holdings processing tool creates a multithreaded processing queue to initiate multiple requests against the oclc system. however, because the api doesn’t provide the ability to determine holdings status, the current interaction of the tool provides a set of binary actions, where all items processed are processed as holdings additions or holdings deletions. the ability to query holdings status, rather than waiting for an error code, would enable the ability to provide conditional holdings processing. figure 2: batch worldcat holdings management conclusion oclc’s worldcat database has long been a database of contradictions. on the one hand, the worldcat database has represented the deep collaborative nature of libraries and librarians. on the other hand, the worldcat database has long been one of the most impenetrable datasets within the library community, with access to the resource carefully guarded by its oclc gatekeepers. the development of the metadata api, along with a handful of other services specifically targeted at making worldcat data more accessible, represent a welcome shift in how libraries are able to interact with their data, and a set of opportunities to develop new collaborations and workflows around the library community’s metadata operations. references [1] fiander, david. 2013. comparing the librarything, oclc, and open library isbn apis. http://journal.code4lib.org/articles/8715 [2] rochkind, jonathan. 2013. a comparison of article search apis via blinded experiment and developer review. http://journal.code4lib.org/articles/7738 [3] oclc connexion client: http://www.oclc.org/connexion/interface/client.en.html [4] oclc connexion browser: http://connexion.oclc.org/ [5] oclc z39.50 cataloging service: http://www.oclc.org/z3950.en.html [6] oclc metadata api documentation: http://www.oclc.org/developer/develop/web-services/worldcat-metadata-api.en.html [7] worldcat metadata api powers new enhancements in latest marcedit release. http://www.oclc.org/en-us/news/announcements/2013/marcedit.html [8] oclc developer network. http://www.oclc.org/developer/home.en.html [9] virtual international authority file. http://www.oclc.org/content/developer/worldwide/en_us/develop/web-services/virtual-international-authority-file-viaf.html [10] classify. http://www.oclc.org/content/developer/worldwide/en_us/develop/web-services/classify.html [11] oclc api explorer. https://platform.worldcat.org/api-explorer/ [12] hash-based authentication code. http://en.wikipedia.org/wiki/hmac [13] oauth 1.0 specification. http://tools.ietf.org/html/rfc5849 [14] oclc metadata api bibliographic resource request/response types. http://www.oclc.org/developer/develop/web-services/worldcat-metadata-api/bibliographic-resource.en.html [15] worldcat metadata api sandbox testing: http://www.oclc.org/developer/develop/web-services/worldcat-metadata-api/sandbox-testing.en.html [16] oclc developer network github: https://github.com/oclc-developer-network about the author: terry reese works at the ohio state university libraries as the head of digital initiatives. his primary research interests tend to focus around the development of services and infrastructure to support large scale data projects and preservation. he’s been a participant and author of a number of software applications, both within and outside of the library community; though is probably most well known as the author and creator of marcedit, a metadata processing suite geared towards librarians and technicians working with library metadata. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – create efficient, platform-neutral, web-based augmented reality content in the library mission editorial committee process and structure code4lib issue 45, 2019-08-09 create efficient, platform-neutral, web-based augmented reality content in the library augmented reality (ar) is an interactive experience of viewing computed-generated objects onto your view of the real world. since the pokemon go craze in 2016, many libraries have tested the waters with ar programs. some went on to the next step of developing their own ar content to enhance library services and marketing. while there are many ar applications that libraries can use for this purpose, it usually thwarts customers that they must install various ar mobile apps in order to enjoy these experiences on their own devices. this becomes the major hurdle of making ar more enjoyable and accessible at the library. what's more, libraries cannot share home-grown ar content across different platforms easily because of the technical barriers in various ar platforms. in this article, i would like to introduce a completely open source ar developing tool that allows library staff to create fast and efficient ar content with pure web solutions. it is standard and works on mobile devices with no installation required. i have created a basic ar experience with the tool for a regional pacific library partnership conference and it proved to be a success in improving the accessibility and shareability of ar content. by dan lou introduction last fall, palo alto city library was planning to host a regional conference[1] with the goal of celebrating the achievements of the past pacific library partnership (plp) innovation and technology grant recipients. at the planning stage, we brainstormed on how to make the event more inspiring and mind-opening via integrating nascent technology elements into it and the idea of creating a badge with some ar magic emerged. none of the team members had any previous ar development experience, so someone had to figure out how feasible the project was. that was when and where we started the journey to explore various ar technologies and to identify the most affordable one for the project. a comparison of popular ar tools we began by researching major players in the ar tools space. wikipedia has a pretty comprehensive page listing a wide range of augmented reality software.[2] we also searched the web for “tools to create augmented reality content”. a few pages with titles like “top x augmented reality sdk for mobile apps” caught my eye. combining this information, the library created a chart to compare ar tools. table 1. ar tool comparison chart ar tool development environments supported platforms open source? features end product free? arkit[3] ios mainly selected apple phones no – persistent ar experiences – shared ar experiences – object detection and tracking a mobile app pay to become a ios developer arcore[4] java, unity, unreal, ios mainly selected android phones yes – motion tracking – environmental understanding – light estimation a mobile app yes vuforia[5] ios, android, universal, windows platform, unity ios, android no – supports a variety of 2d and 3d targets – localized occlusion detection – create and reconfigure target sets programmatically at runtime. a mobile app free trial available artoolkit[6] mac os x, pc, linux, android, ios mac os x, pc, linux, android, ios yes – motion tracking – camera calibration support – simultaneous tracking – full unity3d and openscenegraph support a desktop app or a mobile app yes ar.js[7] combined with a-frame[8] mac os x, pc, linux, android, ios mac os x, pc, linux, android, ios yes – motion tracking – camera calibration support – simultaneous tracking a webpage yes the four questions we asked when evaluating these tools were: is it open-source? does it work across multiple platforms? how easy is it to use for end users? are users required to download an app? how steep is the learning curve for an absolute beginner to develop a basic ar scene? as you can see from the chart, the majority of the ar tools generate an end product in the form of a mobile app. the advantage of a mobile app is obvious. a developer can write more complicated code and add more sophisticated features in the mobile app development environment with the native access to smartphone hardware. because of this, these tools tend to have a better support for various ar targets. an ar target is an object used to anchor digital content in the real world. in order for ar to look realistic to the viewer, it is important that the positioning of the virtual object in the real world matches the viewer’s perspective. to accomplish this, a target is used to determine the perspective. the target is made known to the ar application during the development. when a camera spots the target, the ar application analyzes and recognizes the target’s position in the real world. it is then able to place the virtual content accordingly in the real world view. in the early days of ar, long before pokemon go, black and white markers with irregular patterns were used as ar tracking targets. in contemporary applications, the targets can be many different things thanks to machine learning and artificial intelligence technologies. images, text, shapes, 3d objects, spacial maps, geolocation can be all used as ar targets.[9] some of the advanced tracking technology seems to produce an ar experience without any ar trackers in the view at all. for example, apple’s arkit contains a component called “world tracking” that allows you to put an ar object anywhere you would like in the camera view. this is made possible because arkit can track the realtime motion sensing data returned by the ios device to determine the real-time motion of the device in the real world, hence, the motion of an ar object in the real world. it also analyzes the video stream from the camera in real time to track and determine the position of the device and the objects in the real world, and consequently, the position of an ar object in the real world. other advanced ar tools like arcore and vuforia all provide similar support for various ar targets. nevertheless, these advanced tools need a mobile app to accomplish the sophisticated computation. the ar experience based on a mobile app faces two challenges: accessibility and the ability to share across platforms. generally speaking, if it is required to make a mobile app available on different platforms, it is necessary to develop different versions of the mobile app in different environments. for example, if an ios app is developed using arkit, its android equivalent should be created with arcore. although there are platforms that support building a mobile ar app that can be cross-compiled for multiple environments[10], this method is still cost prohibitive and time consuming in terms of development. the mobile app needs to be maintained and updated all the time. for the badge magic we were trying to pull off, it was overkill. additionally, even if apps for both android and ios were developed, they would only cover the majority of smartphones and tablets but not computers and laptops. that is why we chose ar.js and a-frame, a technology that supports basic ar effects. in terms of ar targets, it only supports pattern markers and customized image markers, but it was sufficient for our project. we decided to trade off some fancy features and novelty targets for a more straightforward and accessible solution. ar.js and a-frame is the only option on the above chart that does not result in a mobile app. it is also free, open source, and quick to develop. a-frame is an open source web framework for building virtual reality experiences which work on vive, rift, desktop and mobile platforms. ar.js is a web-based ar solution that is standard and works with any device with webgl[11], webrtc[12], and a camera. that meant once we developed an ar experience, anyone could access and enjoy it from anywhere, regardless of the device type. create your own ar experience once the right tool was identified, the next step was to test creating customized ar content. through trial and error, we learned that it is more efficient to begin a design with a vr scene first by using a-frame. while developing ar requires the consideration of associating the digital objects with the marker and the real world view, developing vr is a process that focuses on only the digital objects. therefore, starting from vr can save on the development time. in a-frame, it is pretty easy to develop a vr scene from a plain html file without any additional software. it is only necessary to provide a reference to the a-frame javascript library from inside the head of the html file. <!doctype html> <html> <head> <script src="https://aframe.io/releases/0.9.1/aframe.min.js"></script> </head> <body style='margin: 0px; overflow: hidden;'> </body> </html> adding the following code inside the body will generate a cube in the vr scene. <a-scene> <a-box position="0 0 -2" rotation="0 45 45" scale="1 1 1" color="blue"> </a-box> </a-scene> you can preview vr scene by opening the html file in a browser , and using the mouse to drag the cube around. i started all the html coding in the online code editor codepen and i could view the result right away.[13] once a vr scene has been created, it is time to add the ar component. we experimented with the default ar tracker [14] first. in the head of the html file, we created a reference to the ar.js javascript library. <<head> <script src="https://aframe.io/releases/0.9.1/aframe.min.js"></script> <script src="https://cdn.rawgit.com/jeromeetienne/ar.js/1.7.0/aframe/build/aframe-ar.js"></script> </head> it is then necessary to make changes inside the body to enable the web page to use a device’s camera and recognize the default ar tracker as a trigger for the vr object. <a-scene embedded arjs> <a-marker preset='hiro'> <a-box position="0 0 -2" rotation="0 45 45" scale="1 1 1" color="blue"> </a-box> </a-marker> <a-entity camera></a-entity> </a-scene> if this modified web page is opened in a browser on a device and the camera is pointed at the default ar tracker, a virtual blue box will appear where the tracker is on the screen [13]. after creating this simple proof of concept test, we moved on to design the real content for the conference badge. the augmented reality display can handle several kinds of vr content. among others, we tested 3d text, a 3d drone model and a 3d cube covered with images. in the two final versions of the app used for the conference, a 3d conference logo and an r2-d2 were displayed. the beauty of a-frame is that not only can 3d objects be prototyped quickly with a few lines of the markup language, but also pre-designed 3d models can be easily loaded into it. for the conference badge, we decided to design a conference logo using the 3d design software sketchup.[15] the logo was composed of the palo alto city 3d logo, shaped like an oak leaf, and name of the conference in 3d text. the model was saved as a gltf object, loaded into the a-frame code and some animation effects were added. the default behavior for the logo was to rotate back and forth at a slow speed. <a-assets> <a-asset-item id="logo" src="./leaf/leaf_plp.gltf" ></a-asset-item> </a-assets> <a-entity gltf-model="#logo" scale="3 3 3" position="0 0 0" rotation="0 0 0"> <a-animation direction="alternate" easing="linear" attribute="rotation" repeat="indefinite" dur="5000" from="60 0 0" to="-60 0 0"></a-animation> </a-entity> for the next step, we needed to consider the ar tracker to be used. to do that we needed to decide on what image or text to use as the tracker and the obvious choice for us was to use a symbol closely associated with our library. our successful robot coding program, sponsored by plp, was selected and recognized as a california state library lsta copycat grant program in 2018 [16] and was going to be celebrated at the conference. that is why we decided to train the image of our robot as the custom tracker on the ar.js marker training webpage.[17] within a week, this ar magic was live and working. we ended up not using the ar magic the way we originally proposed. in order to go green and save paper, the library decided not to print out badges. instead, we emailed the tracker image with the confirmation information to all attendees in advance. during the conference, we turned the ar magic into an activity. about twenty ar trackers were printed out and put on each table in the conference room. our director, monique le conge ziesenhenne, gave instructions to all attendees on how to play the ar magic. adventurous librarians then took out their phones, pointed them at a nearby ar tracker and were pleasantly surprised. figure 1. ar created for plp conference. tracker printed on paper and ar viewed from chrome browser on an android phone. figure 2. animation of ar created for plp conference. the future of web-based ar in libraries besides conference badges, web-based ar has a lot of potential in libraries. i would like to take the chance to share some of my ideas with you. an ar tracker is an image that can be printed out on different materials physically or kept as a digital image opening on a device. for example, i quickly made a code4lib ar scene for my lightning talk at code4lib 2019.[18] in the demonstration, the associated tracker, trained with the conference logo, was opened on a laptop. the designated ar web page was opened in the chrome browser on a smartphone. the ar scene appeared when the associated tracker was detected by the smartphone. supposing the tracker were instead printed on t-shirts that distributed to all attendees, this could turn into a fun conference activity where people would share the ar experience by scanning the trackers on each other’s t-shirts with their smartphones. similarly, we can use web-based ar scenes to enhance promotion and marketing materials at the library. for big programs like the summer reading program, libraries can turn the 2d flyers into mixed reality scenes by using this technology. figure 3. ar created for the lightning talk at code4lib 2019. tracker open on a laptop, and ar viewed from chrome browser on an android phone. another idea is to incorporate the ar technology into 3d design workshops and tutorials. many libraries now offer 3d design help to the customers. web-based ar can take customers’ 3d designs another step forward. we can help customers make their own customized ar business card for instance. a third idea is to enhance the library’s collection. for example, libraries can attach interesting ar to specific children’s books or bookmarks. parents can then enjoy a magic moment with their children. combine ar and distributed web one issue remains if a library wants to adopt any of the above ideas: how and where to store the ar content? a library might need to purchase local servers or a cloud server like amazon web services(aws) to keep the growing ar content persistent, but this comes along with an ongoing cost and maintenance effort that a library might find difficult to afford. that is why we have also explored storing the ar content in the distributed web. i added part of the original ar project folders to the distributed web called internet planetary file system (ipfs).[19] the folder, as well as each file and the 3d model inside it, now has a persistent and unique cryptographic hash in ipfs. they become part of the distributed web permanently. people can obtain or access the files via the peer-to-peer ipfs protocol or via the ipfs gateway, which is an old-fashioned http web protocol. [20] this might be the solution for file storage and sharing in the future. as a matter of fact, our library just started to launch a new round of innovative programs by using both a-frame and ipfs. we hosted a workshop called “virtual reality hackfest” for absolute beginners on jun 22, 2019. [21] in the two-hour workshop, around twenty attendees learned how to create simple vr scenes using a-frame code. these vr scenes then were published in ipfs. at the end of the workshop, every attendee walked away with a qr code that contained the ipfs hash to the vr scene he/she published. when the qr code is scanned by a smartphone, the associated vr scene is displayed. source code and examples all the scripts and examples are stored in a github repository. [22] everyone is welcome to try out the examples. i hope more librarians will make creative use of the ar tool described in this article in library programs and activities. about the author dan lou is a senior librarian at palo alto city library, where she works on web content management and develops pioneering projects on robotics, coding, ar and distributed web. her particular interests include python, machine learning and data analytics. previously, she worked as a technical project manager at wolfram alpha, a knowlegebase queried by apple siri and amazon alexa. before that, she was a systems librarian at innovative interfaces. references [1] positioning libraries for the virtual future. 2018 aug 31. [accessed 2019 may 15]. http://plpinfo.org/event/positioning-libraries-for-the-virtual-future/ [2] list of augmented reality software. wikipedia; https://en.wikipedia.org/wiki/list_of_augmented_reality_software [3] arkit. apple inc; https://developer.apple.com/arkit/ [4] arcore. google; https://developers.google.com/ar/ [5] vuforia. ptc; https://www.ptc.com/en/products/augmented-reality [6] artoolkit. github; https://github.com/artoolkit [7] ar.js. github; https://github.com/jeromeetienne/ar.js [8] a-frame. https://aframe.io/ [9] kumaraswamy a. types of augmented reality targets. 2018 apr 8. packt; https://hub.packtpub.com/types-augmented-reality-targets/ [10] aseem wangoo. flutter and ar. 2019 apr 15. flutter; https://flatteredwithflutter.com/how-to-create-ar-in-flutter/ [11] webgl overview. the khronos group; https://www.khronos.org/webgl/ [12] webrtc. https://webrtc.org/ [13] lou d. a-frame blue box. https://codepen.io/loudan/pen/qrmoyx [14] ar.js hiro maker. github; https://jeromeetienne.github.io/ar.js/three.js/examples/marker-training/examples/generator.html [15] sketchup. https://www.sketchup.com/ [16] copycat grant toolkits – coding with the robot – california state library. 2019. ca.gov; https://www.library.ca.gov/services/to-libraries/copycat-grants/the-journey-begins/. [17] ar.js marker training. github; https://jeromeetienne.github.io/ar.js/data/images/hiro.jpg [18] lou d. web based ar. beautiful.ai; https://www.beautiful.ai/deck/-lzir85yqslxohlupw7s/arcode4lib2019 [19] ipfs. https://ipfs.io/ [20] lou d. ar / mr in the web: available now in the distributed web. github; https://github.com/fishbb/ar#available-now-in-the-distributed-web [21] vr hackfest. 2019. cityofpaloalto.org; https://library.cityofpaloalto.org/news/vr-hackfest/. [22] lou d. ar / mr in the web. github; https://github.com/fishbb/ar subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – editorial introduction: changes on the editorial board mission editorial committee process and structure code4lib issue 29, 2015-07-15 editorial introduction: changes on the editorial board the publication of the 29th issue of the journal brings with it several changes to the editorial board. the publication of the 29th issue of the journal brings with it several changes to the editorial board. following on the heels of the special diversity edition, we are walking the walk and welcoming four new editors: meghan finch digital assets librarian, oakland university péter király,  cultural heritage/digital  humanities developer, gesellschaft für wissenschaftliche datenverarbeitung mbh göttingen (gwdg) junior tidal,  web services & multimedia librarian,  new york city college of technology, cuny ruth tillman, metadata librarian for nasa goddard (cadence group) being an all volunteer committee, we are ever grateful when talented individuals are willing to give of their time and expertise to make the code4lib journal a reality, and are exceedingly happy to have them on board.  we also grateful to ruth tillman for taking on the role technical editor, keeping this site up and running. this issue we also sadly say goodbye to two gifted editors: shawn averkamp, new york public library kelley mcgrath, university of oregon we will miss their expertise and wish them will as they move on to focus on other priorities. issue 29 contains a broad range of articles, which can be loosely grouped as follows: front end features: implementing a bento style search in libguides v2 building a better book in the browser (using semantic web technologies and html5) connecting historical and digital frontiers: enhancing access to the latah county oral history collection utilizing ohms (oral history metadata synchronizer) and isotope 3d adaptive virtual exhibit for the university of denver digital collections making user rights clear: adding e-resource license information in library systems security considerations: exploring information security and shared encrypted spaces in libraries a novel open source approach to monitor ezproxy users’ activities metadata improvements: improving access to archival collections with automated entity extraction ­the geospatial metadata manager’s toolbox: three techniques for maintaining records review and update of previous work: barriers of contribution to open source software projects   subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – approaching the largest ‘api’: extracting information from the internet with python mission editorial committee process and structure code4lib issue 39, 2018-02-05 approaching the largest ‘api’: extracting information from the internet with python this article explores the need for libraries to algorithmically access and manipulate the world’s largest api: the internet. the billions of pages on the ‘internet api’ (http, html, css, xpath, dom, etc.) are easily accessible and manipulable. libraries can assist in creating meaning through the datafication of information on the world wide web. because most information is created for human consumption, some programming is required for automated extraction. python is an easy-to-learn programming language with extensive packages and community support for web page automation. four packages (urllib, selenium, beautifulsoup, scrapy) in python can automate almost any web page for all sized projects. an example warrant data project is explained to illustrate how well python packages can manipulate web pages to create meaning through assembling custom datasets. by jonathan e. germann introduction and background the library role of gatekeeper and gateway to information has been expanding in a networked environment. valuable digital information inside and outside paid database subscriptions has been expanding as well. many information needs require raw or custom datasets for direct manipulation. for example, library patrons will often find themselves in need of compiled source data from a website in a spreadsheet like program to analyze. patrons may need to combine data from various sources or track source information over time. unfortunately, much, if not most, of this information is not created or optimized for digital manipulation but for direct human consumption. a staggering amount of information is available online to meet these needs. for example, google, microsoft, amazon, and facebook are estimated to hold 1,200 petabytes of data alone. open access common crawl is downloading and indexing approximately three billion open web pages quarterly for future public consumption and manipulation. as of october 2017, it is estimated that 47 billion web pages exist on the internet. of these 47 billion, it is estimated that only five billion are indexed. most web pages are designed and optimized for direct human interaction — information marked with html to be interpreted and visually displayed in a web browser. when examining network traffic, however, most activity on the internet is attributed to automated spiders/bots. these bots are downloading, indexing, and recompiling information embedded in web pages to create new tools, meaning, and insight. regrettably, only those who understand the complexities of retrieving and extracting networked information can gain additional insight from these billions of information sources. libraries have historically filled unmet information gaps. there is a need for datafied information and custom datasets. access can only be created with bots to pull custom information out of web pages. libraries can fill this current growing information gap by focusing on the data residing in the billions of internet pages. internet data has a high volume, variety, and velocity. this information is beyond library curation abilities — there are too many websites to curate and too many diverse information needs. libraries can instead focus on curating tools to wrangle data for patron use. specifically, focus on library sponsored services and educational outreach assisting users in locating and extracting relevant information. librarians must get proximate to the millions of websites holding valuable information by assisting in the creation of custom bots and tools. browsing the modern web web pages are transferred via hypertext transfer protocol (http: which mainly utilizes get, post, put, and delete requests) to be rendered for human consumption by a browser. the process is fairly straightforward: content is requested via http, the response is interpreted and rendered by a browser (html, xml, css, xpath), and content is manipulated dynamically using javascript and the dom api. the process of rendering a web page over http is simplified into four steps, beginning when a url is typed into a browser. when a user types an url into a browser/clicks a link, a number of actions take place. first, the domain name, or the initial part of the url, is used to lookup the appropriate web server. second, the remaining end of the url, in conjunction with other information like cookies and browser version, is sent as a request to the server by the browser. third, the server replies (hopefully!) by sending an html file. fourth, the html is interpreted by the browser and translated into a document object model (dom, a tree representation of the page). the dom representation is rendered in the browser and users can interact with the information. modern web page rendering is largely driven by structured data (databases, xml, json) and scripts (javascript/ajax/jquery) formatted according to w3c standards. web pages are designed to be delivered to browsers that parse the html/css into standardized structures (the document object model). these structures enable dynamic content creation and monitoring through the dom api. the dom api can be accessed by scripts (javascript/jquery/ajax) to dynamically change content in real time. the complexities of this process are normally handled by the web browser with a human interacting with the information to drive further requests or dynamic page changes. these processes together form a well-structured web api — an api normally designed and optimized for a web browser and human team. manipulating the web api is fairly straightforward when needing to create custom datasets. the complexities arise when it is necessary to automate the human part of the team. web scraping in python there are a number generic web crawlers for obtaining bulk website data (apache nutch, wget, curl). however useful in crawling through entire websites, they lack functionality in obtaining targeted data. for targeted web scraping tasks, python excels. python is a user-friendly programing language that has a vibrant web scraping community. the below inexhaustive list of python packages will assist in obtaining targeted information from almost any web page on the internet. they are used to request appropriate information and then parse the results. urllib is a package for directly working with urls. modules facilitate requests for opening and reading urls, error handling for parsing exceptions raised, and parsing for navigating urls and robots.txt files. urllib is good for the direct requesting of responses from a server (get, post, put, delete). it does, however, need to be used in conjunction with an html/xml parser to extract data. for more information, see: https://docs.python.org/3/library/urllib.html selenium is a package that automates web browsers. although originally designed for testing purposes, it is also currently valuable when attempting to navigate modern web pages that run on javascript. it excels at manipulating web pages through xpath (xml path language for querying web page information) and removing most programming complexities by utilizing the browser for complex user-agent checking, cookies, and add-ons. selenium can be resource intensive when using headed (visible) browsers. for more information, see: http://www.seleniumhq.org/ beautifulsoup is a package to navigate, search, modify and extract information from web formatted documents. it converts incoming web documents to unicode and parses pages into navigable trees for easy data manipulation and extraction. named after a poem in alice’s adventures in wonderland, beautifulsoup attempts to make sense of web pages. for more information, see: https://www.crummy.com/software/beautifulsoup/ scrapy is a fully open source application framework built in python for extracting and navigating web pages. it has developed a robust and efficient framework that is useful in handling concurrent requests for large or ongoing projects. scrapy has a learning curve and in most cases must be used in conjunction with other tools (beautifulsoup, headless browsers). for more information, see: https://scrapy.org/ moral, legal, and practical considerations before performing automated data harvesting on a remote web server, it is prudent to consider the possible moral and legal issues. a website’s robots.txt file will indicate which part(s) of the website the owner does not want bots to crawl and/or index. subscribed databases and password protected websites are usually governed by user or license agreements restricting automation. servers also have physical restrictions. irresponsibly fast scraping can slow the server to a crawl or act like a denial-of-service (dos) attack. in all cases, it is a good idea to obtain the data without leaving a noticeable footprint. servers have safeguards, set intentionally or by default, that will block ip addresses or ban registered users at certain thresholds. scraping slowly or during low traffic periods will keep scripts running smoothly and website owners unaffected. case study: obtaining warrant data a legal scholar had an information need in obtaining bulk warrant data to examine demographic disparities. the necessary raw information did not exist in either paid or free websites. the information did exist on government websites but needed high-volume, repetitive downloading and parsing before the information would be useful or meaningful. austin was one government website that had the desired data in an unusable single-serving format (located at https://www.ci.austin.tx.us/police/warrants/warrantsearch.cfm). the data was available by searching by full name (two characters needed in both first and last fields) or birthdate. figure 1. name search form for warrant data the process for obtaining this data was 1) examining the website’s http requests/responses, 2) creating a plan to loop over every relevant web page (here date of births (dobs) in a database), 3) using a python package to obtain raw web pages, and 4) parsing results to extract relevant content to a csv file. this approach also allowed for reexamining the data for accuracy verification. 1. initial website inspection initial website inspection was performed by accessing the developer tools in google’s chrome web browser. chrome developer tools allowed for inspection of web page elements and network traffic. if simple http requests were performed during search, the best way to acquire the data would be to insert query terms directly into the http request string. if data was being dynamically generated behind the scenes through dom api access, a simulated browser would need to be used to navigate the web page. figure 2. captured http headers here, test searches initially performed on austin’s warrant search page revealed a http post request being utilized to return results. austin’s website also used cross-site request forgery (csrf) tokens as added security. csrf tokens could be handled using python’s requests package, scrapy, or a web browser driven by selenium. since low computer overhead or fast scraping speeds were not needed, the most programmatically friendly approach to extraction was using selenium to drive searches while taking care of the csrf/header/cookie complexities. first collecting the raw html results driving a headless chrome instance and then parsing the raw html for desired warrant data with beautifulsoup. 2. loading dates of birth to sqlite to collect possible results, the first step was to populate a database with a range of possible birthdates (relevant dob searching being about 50,000 requests versus name searching at about 457,000 requests). python’s datetime package was used to populate a sqlite database. from datetime import timedelta, date #to fill database with dates import sqlite3 #the database def createdb(): """creates a sqlite table. date field for 100+ years of dates. success field for indicating if the date was successfully scraped. warrant field for indicating if any warrant was found on that date. html field for storing the raw html web page responses""" conn = sqlite3.connect('austin.db') conn.execute('''create table data (success text, warrant text, html text, date text);''') conn.close() def daterange(start_date, end_date): '''creates and yields dates using python’s datetime package.''' for n in range(int ((end_date start_date).days)): yield start_date + timedelta(n) def loaddates(): '''connects to sqlite and loads each date into the database by iterating through the "daterange" function.''' conn = sqlite3.connect('austin.db') conn.text_factory = str start_date = date(1900, 01, 01) end_date = date(2017, 12, 31) for single_date in daterange(start_date, end_date): getdate = single_date.strftime("%m/%d/%y") print getdate conn.execute('''insert into data(date, success) values (?, ?)''', (getdate, "n/a")); conn.commit() conn.close() 3. driving chrome to collect html responses a browser needed to be initiated, driven by selenium, each possible date of birth injected into the search box, and resulting html saved to the sqlite database. selenium was used to drive a chrome browser with custom options. to save computer resources, no visible browser was rendered. to enable the clicking of the submit button without elements overlapping, chrome was opened with a virtual window size of 1000 x 1000. import sqlite3 #to drive each search by date of birth. from selenium import webdriver #browser control with options/waits/exceptions. from selenium.webdriver.chrome.options import options from selenium.webdriver.support.ui import webdriverwait from selenium.webdriver.support import expected_conditions as ec from selenium.webdriver.common.by import by from selenium.common.exceptions import timeoutexception import time #for slowing down the scraping process. while error < 20: """ browser setup. wrapping scraping loop in a condition: if the server temporarily goes down or we run into errors -we want to pause the scrape. """ chrome_options = options() chrome_options.add_argument("--headless") chrome_options.add_argument("--disable-gpu") chrome_options.add_argument("--window-size=1000,1000") """opening chrome in headless mode (does not render a visible browser), with no gpu usage, with an imaginary window size of 1000 x 1000 (so we can click on web links/buttons without them overlapping).""" browser = (webdriver.chrome(chrome_options=chrome_options, executable_path= r'c:\python27\selenium\chromedriver.exe')) #loops through html and calls "gethtml" function to extract table data. drivebrowserbydb() browser.quit() print "errors are " + str(error) the selenium webdriver can interact with a web page much like a human. selenium can use the formatted document structure (html, css, xpath) to locate and interact with page elements (by id, name, xpath, etc.). this information can be found by right-clicking and selecting inspect in chrome. after examining these elements, each necessary ‘human’ step was added. def gethtml(date): """ a scraping function driven by looping over dates. this function takes a date, performs a search on the austin web page, and returns the raw html response or error.""" global error try: """loads the austin search page and waits up to 10 seconds for the warrant date search form to appear.""" browser.get("http://www.ci.austin.tx.us/police/warrants/warrantsearch.cfm") element = (webdriverwait(browser, 10) .until(ec.presence_of_element_located((by.id, 'warrant_date')))) """clears the date box, then inserts the search date into the box. waits for the submit button to appear before clicking it.""" element.clear() element.send_keys(date) element = (webdriverwait(browser, 10) .until(ec.presence_of_element_located((by.name, 'submit')))) element.click() """waits for the container box to load, waits for a prescribed time (to slowly and politely request information from the server) and returns the raw html to be stored in the sqlite database.""" element = (webdriverwait(browser, 10) .until(ec.presence_of_element_located((by.id, 'container')))) time.sleep(waittime) html_source = browser.page_source.encode("utf-8") return html_source #if any of the above fails, an error is written to the sqlite database. except exception as e: print e.__doc__ print e.message error = error + 1 return false 4. parsing the raw html to obtain needed warrant data after ~50,000 dob searches and corresponding html requests were obtained, the next step was to extract the warrant data into a spreadsheet for further analysis. the beautifulsoup package was used for parsing html into a navigable tree (the tree can be navigated/selected by tag name or by parent/child relationships). the below is an example snippet of a beautifulsoup tree for the austin warrant data results. <tbody> <tr class="odd" role="row" valign="top"> <td class="sorting_1"> moon </td> <td> moon, jane </td> <td> </td> <td> 04/27/81 </td> <td> b </td> <td> m </td> <td> bond: $ 125.00 the beautifulsoup object had a clear table with two classes: odd and even rows. therefore, the code to parse the object merely needed to pull information out of these two class labels. first, a beautifulsoup object was created and searched for the table containing the warrant data. then each cell containing data was parsed and formatted. from bs4 import beautifulsoup #to create navigatable object. import re #to remove whitespaces. import sqlite3 #to pass previously scraped html to beautifulsoup. def getthings(rawhtml): """raw html is being passed to this function. the function will pull any available warrant rows and write them to an opened csv file.""" try: #creates a beautifulsoup object from raw html using a html parser. soup = beautifulsoup(rawhtml, 'html.parser') #used to print the beautifulsoup tree structure for analysis. print soup.prettify().encode('utf-8') #creates a beautifulsoup object containing both odd/even rows. warrant = soup.findall("tr", { "class" : ["odd", "even"] }) """loops over the object (over rows, then cells) to extract warrant data. if no data exists, fills in blank. writes to file after each row.""" for row in warrant: writerow = '~' for cell in row: try: item = cell.text.replace(" ", "") writerow = writerow + str(item) + "~" except: writerow = writerow + "blank" + "~" output.write(str(writerow)) output.write('\n') except exception as e: print e.__doc__ print e.message return false return true 5. final output the data was written to csv for end-user manipulation. thousands of austin warrants were parsed and saved to csv to be manipulated and analyzed for demographic disparities. figure 3. sample csv formatted into table conclusion networked environments allow for advanced information exploration. the modern web runs on a handful of protocols and standards that together make an internet api. querying the api and compiling different datasets creates insight and meaning desired by patrons. libraries are uniquely positioned to facilitate access to the billions of web pages on the internet. librarians have access to subscribed databases/websites and have an in-depth understanding of the availability and reliability of resources on the open web. by learning simple python and related packages, librarians can open these data sources. for every patron their dataset. for every dataset its patron. bibliography common crawl: get started[internet]. [accessed 2017 dec. 1] available from http://commoncrawl.org/the-data/get-started/ imperva incapsula’s bot traffic report [internet]. [accessed 2017 dec. 1] available from https://www.incapsula.com/blog/bot-traffic-report-2016.html. kouzis-loukas, d. 2016. learning scrapy. packet publishing.(coins) mitchell, r. 2015. web scraping with python: collecting data from the modern web. o’reilly media.(coins) panchenko, alexander et al. 2017. building a web-scale dependency-parsed corpus from commoncrawl. available from https://arxiv.org/abs/1710.01779 taddeo, m., floridi, l. 2017. the responsibilities of online service providers. springer. retrieved from http://books.google.com about the author jonathan e. germann (jd, mlis) is the digital services librarian at the georgia state college of law. jonathangermannlaw@gmail.com subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – factor analysis for librarians in r mission editorial committee process and structure code4lib issue 46, 2019-11-05 factor analysis for librarians in r this paper offers a primer in the programming language r for library staff members to perform factor analysis. it presents a brief overview of factor analysis and walks users through the process from downloading the software (r studio) to performing the actual analysis. it includes limitations and cautions against improper use. by michael carlozzi background my public library recently performed a needs assessment as part of its five-year planning process. we distributed a fairly standard patron survey, which asked respondents to evaluate their preferences for and satisfaction with several library-related services. we had also asked survey respondents whether they were interested in joining focus groups. i had conducted needs assessments before and wanted to do something different this time. so, i performed exploratory factor analysis (efa) on this survey, as i had speculated that certain “types” of patrons might exist, such as those who preferred quiet study, and that i could identify these types to help improve services. further, i wanted to recruit patrons who “loaded” on (that is, were similar to) certain factors for our focus groups in an attempt to diversify the candidate pool. in the end, some interesting factors were revealed for future study and analysis, i refined my survey instruments, and we had an engaging focus group which included participants of varied perspectives. efa helps researchers to identify latent constructs. observed data, such as survey responses, are analyzed with the aim of uncovering “factors” that cannot be directly measured. it is a technique which holds broad appeal, used in virtually all disciplines, including library science (e.g., kwon & gregory, 2007). and, although factor analysis is mathematically intensive, modern computing means that any practitioner, regardless of mathematical aptitude, can perform it. this paper is a primer for librarians in the programming language r on using efa. it hopes to guide interested library workers in conducting such analyses responsibly. while factor analysis is not as inaccessible as it once was, it is still complicated enough that a primer cannot hope to cover all of its nuances and pitfalls. disagreement exists for virtually all steps of a factor analysis, and due consideration of these options would quickly bog this paper down in technical details. i, therefore, make recommendations based on the work of established researcher jason osborne. i would direct readers interested in learning more about efa to consult osborne and anna costello’s article (2005) and, for a more extensive treatment, osborne’s book (2014). efa simplified; and its uses factor analysis is a mathematically complex process which computer programs now fortunately manage. the challenge for human researchers is to perform the analysis responsibly, notably to extract and interpret factors appropriately. essentially, researchers distribute any item-based questionnaire and then use efa to determine whether underlying constructs (“factors”) may exist. the payoff is in gaining a richer understanding of phenomenon as well as simplicity; rather than needing twenty questions, for example, to explain something, two factors might instead suffice. efa, as the name suggests, is exploratory. its primary benefit is to help you navigate data in a (hopefully) clear manner based on your preconceptions. that data can take many forms but it is often some type of questionnaire such as a survey. in our case here, you might enter the analysis expecting to find certain “patron profiles,” such as patrons who use the public internet workstations and not much else. you would hope, then, that an efa would reveal a factor of those distinct respondents—patrons who showed no real (or even a negative) preference for services outside of public internet workstations. but because you are exploring data, firm conclusions should not be drawn. results should be used to further pursue lines of inquiry or research; other methods, such as confirmatory factor analysis, can analyze structures identified by efa. indeed, your goal may not even be to identify the “true” number of factors. in my case, for example, i was more interested in identifying patterns among my users. for example, across four, five, and even six factor models, i was able to notice similar patterns of patron preferences and usage. if one of these models helped clarify these patterns better, then i might choose to settle on that model. regardless of which model i chose, i was receiving interesting and potentially actionable information about my patron base. so long as you are flexible, do not become married to any particular factor structure, and remain open to the exploration of data/analysis, you should be fine. data integrity fundamentally efa analyzes item scales. in other words, it asks how hypothetical factors might “load” onto questions. we can only have faith in analysis if we trust the data on which it has been based. i would recommend, then, using results to refine surveys. the better your survey tools, the clearer your analysis should be. for example, efa revealed that my patron survey had several faults. one part of my survey asked patrons to rate their interest on a scale of 1-5 in a particular service. for example, a statement might be “borrowing physical materials from the library.” it seemed straightforward enough. but i suspect that the survey suffered from a pronounced “halo effect.” in a library setting, the halo effect might occur when survey respondents indicate that they like all aspects of a library’s services because they like the library itself; they therefore have difficulty separating the “parts” (specific services) from the whole. you do not want responses clouded by a halo effect because they will muddle your results. indeed, library patrons who fill out surveys usually like the library; in my case, 96% responded that library services overall qualified as “good” or “excellent.” it stands to reason, then, that they would recommend or show interest in services which they did not personally value. in fact, some respondents wrote that, while they did not value a certain service, they rated it highly because they felt it was important to others. i felt obligated to remove those responses, but i have no idea how many users had engaged in similar behavior. we also had a major problem with respondents who almost exclusively used our english as a second language instruction. despite affirming that they only used the esol classes, these patrons almost universally rated highly all library services. as i found this likely the result of a halo effect, i felt forced to remove their responses from the analysis. based on these concerns, i suspect that my statement wording was problematic. for future surveys, i plan to design similar questions/statements that instead are more substantive, such as “i wish the library had more dedicated quiet study or reading space.” that kind of user-centered question asks users to rate how valuable quiet study space is to them in a way that (hopefully) distances the response from the library itself, making it (again hopefully) less likely that users will feel a low rating reflects badly on the library. so, one immediate benefit of an efa is that it will help you to refine your survey/tests. this, of course, is not limited to efa, as any comprehensive analysis will achieve the same goal, but factor structures help to reveal readily problematic items/questions. now, after you have distributed an excellent survey, you should make sure that your data are structured to facilitate efa. for best results, collect a lot of responses, analyze only items on the same measurement scale (e.g., all responses range from 1-7 or 1-5), and drop responses with missing data (or fill in those responses consistently; e.g., someone who does not answer a response asking whether he/she liked a service could reasonably be interpreted as not liking that service). efa needs a large sample. osborne (2014) recommends a ratio of at least 20 respondents per item; for instance, a survey of 10 items would require at least 200 respondents. therefore, ensure that whatever data you collect, survey or otherwise, is promoted vigorously and long enough for an adequate sample to be collected. further, test items which do not yield much discriminating information should probably also be removed. for example, i had one question that asked, on a scale of 1-5, how “good” library services were. as 96% of respondents answered “good” or “excellent” (not terribly surprising in a survey of active library users), this question failed to discriminate among respondents, so i dropped it. installing the necessities r is a programming language that (almost) trivializes factor analysis. this paper is not a tutorial on r, but the commands presented are simple enough that even total beginners should be able to follow along. first, you must install the r language. i recommend installing the free and open-source r studio (www.rstudio.com) ide/gui. one of r’s strengths is the extensibility of its packaging system. r packages contain prebuilt functions that facilitate ease of use, meaning that you do not have to write your own functions. only one package needs to be installed for this tutorial, “psych.” so, once r and r studio have been installed, run the following command in r studio to install the psych package. commands should be run in the console section, which by default is in the lower-left quadrant of the screen. install.packages("psych") after installation, the package must then be loaded. type the following command: library(psych) this is not an important warning message; it is not uncommon in r to receive such messages, and in many cases they can be safely ignored. loading your data your data should be saved as a .csv file as a wide-format spreadsheet (rows for responses; columns for each question on a survey). the r command getwd() will inform you of the current working directory. your files must be in the current working directory for r to access them. it is easier to just move your .csv file to that directory than to change the working directory through r’s interface. so, for example, if r tells you that the current working directory is c:/users/michael/documents, move your .csv file to c:/users/michael/documents. it can now be read by r. after your file sits in the current working directory, you must create a variable of it in r so the program can manipulate it. the following screenshot shows several lines of code which will be explained in turn: library(psych) factor_data <read.csv("factor_cleaned.csv") head(factor_data) kmo(factor_data) five_factors <fa(factor_data, nfactors=5, fm="pa", rotate="oblimin", cor="poly") the second line creates a variable called factor_data from the file “factor_cleaned.csv”.  “factor_data” is the name of the r variable and “factor_cleaned.csv” is the name of the spreadsheet with my survey’s responses. in this paper, i have only analyzed data from one part of my survey, the part which assesses patron’s interest (the “i_” of each category) in a service (1 = not interested; 5 = very interested). each variable is a separate question that the patrons were asked to answer. it is not important to know what all of these variables are now; when necessary, i will explain them. the command head() returns the first several rows of a data structure. this is a useful command for inspecting how your file’s data have translated to r. if the data look correctly structured, i.e. mirroring your spreadsheet, then you are ready to proceed. kmo (case sensitive) runs the kaiser-meyer-olkin test. kmo is a quick “sanity check” to ensure that you are not applying factor analysis to inappropriately structured data. put simply, higher values suggest that the data are amenable to factor analysis. in general, an overall msa value greater than 0.70 is acceptable. lower values do not mean factor analysis cannot be done, only that the results should be understood as having wider bands of uncertainty. individual items with low msa values may be dropped if, later on, you suspect they are causing problems. i then run a factor analysis on the final line, storing the results in a variable called five_factors. in this case, i started with five possible factors. the arguments in the command are: nfactors = the number of factors to extract, in this case 5. it is important to realize that you choose (and experiment with) the number of factors. indeed, choosing the “right” number of factors is in many ways the most challenging part of efa. some techniques exist for helping you choose, which will be covered later. fm = “pa” is the factoring method, in this case “principal axis factor.” i choose “pa” over maximum likelihood (“ml”) because the data are not multivariate normal (osborne, 2014, p. 5). to test multivariate normality, install the package mvn (it will also install many dependencies) and refer to the syntax in its help file, (https://cran.r-project.org/web/packages/mvn/vignettes/mvn.pdf). alternatively, just assume that your survey data are not multivariate normal and default to “pa.” readers familiar with some research on factor analysis may wonder whether they can perform principal component analysis (pca) instead of principal axis factoring. pca is a popular technique that is often called factor analysis. however, i do not recommend pca per osborne (2014), as it is not true factor analysis and is merely a data reduction technique. rotate = “oblimin” refers to the factor rotation method, which in this case is oblique. factor rotation helps to “clarify” data structures. there are two primary ways to rotate factors: orthogonal and oblique. oblique rotations are generally superior and preferred when factors are correlated, which in this case they are (osborne, 2014). you can probably get away with always using “oblimin” instead of other oblique options (e.g., “promax”) but you can also experiment with other rotation methods to see how they affect the underlying structure. cor = “poly” is the kind of correlation used, in this case polychoric. this is recommended by van der eijk and rose (2015) when analyzing ordinal data, which survey data often are. the next screenshot depicts output from the factor analysis. print(five_factors$loadings, digits=2, cutoff=.3) the print command presents the factor loadings (correlations between each factor and survey question) for each factor, generically labeled by the software as pa3, pa1, pa4, pa2, and pa5. those labels refer to the possible factor chosen. i chose to print only two digits and, arbitrarily, to show only loadings that cutoff at .30. loadings below .30 were not strong enough to merit my attention. what essentially happened is that i told the software: please (always pays to be polite) analyze all of the survey responses in a .csv file and build five factors to explain them. near 0 loadings/correlations represent no relationship and those with greater values in either direction represent stronger relationships. note that, when performing oblique rotations, it is possible (but potentially problematic) to have relationships greater than 1.0 (as is the case here with pa4). it is important to remember that loadings close to zero do not represent a negative relationship between the factor and the response, only that the factor is not strongly associated with that response. so, for example, factor pa3 is not strongly associated with interest in borrowing physical items or picking up holds. this does not mean participants who load highly on factor pa3 dislike circulating physical books. that would be the case if they loaded negatively. it means that they are, as a group, no more or less likely to prefer that activity. on the other hand, patrons in pa3 load very highly (0.94) on interest in computing; they are, as a group, far more likely to prefer that activity as well as using the library to study (0.68) and to use the library’s office supply services (0.68). you’re looking for factors, then, that have clearly defined relationships, that make logical sense (you don’t want, for example, interest in contradictory items), and that do not cross load. cross loadings mean that items load highly on more than one factor. this is not ideal; in a perfect world, your factors have very little crossover to reflect that they are measuring completely discrete constructs. so, for example, “interest in taking a class” is a bit problematic, because it loads on two factors (both 0.31). one solution may be to remove the item, but since that would omit data it should be a last resort. it makes sense here that an interest in classes would correspond to a factor about general learning (pa1) as well as a factor about one-on-one instruction and ereaders (pa4). given this logic, i chose to leave the item, planning in future surveys to be more specific about what kinds of classes are being offered. if you do have lots of cross loadings or a completely unclear structure, then your data may not be suitable for factor analysis. try writing item questions that more clearly examine different concepts and/or collect more responses. you should also try to adjust the number of factors, as we discuss next. wait. how many factors? computer programs cannot churn out “the right answers.” it is up to the subject matter expert, here the library staff, to evaluate whether factor structures make theoretical sense. in this way, factor analysis becomes more art than science. there is no magic formula to build, no miracle command to type, which will tell you how many factors are “really” underneath the data. in efa, you are (predictably) exploring. i do not recommend, then, following dogmatic prescriptions on selecting the number of factors. two common but limited approaches recommend choosing factors based on 1) their eigenvalues, the so-called kaiser criterion, and 2) scree plots. as these are not really recommended methods, i will not detail them. instead, i prefer to experiment with various numbers of factors and rotation methods to identify something clear, logical, and likely to replicate. if you would like a more mathematical/computational method, then you probably should use parallel analysis, and, indeed, parallel analysis makes a good starting point. performing parallel analysis is very easy in r: fa.parallel(factor_data, fm="pa", fa="fa", n.iter=20, cor="poly") the command fa.parallel takes an r variable (in this case factor_data) as well as several arguments. the arguments resemble what has been entered before; fm (factoring method) = principal axis factoring, fa = show values for factor analysis, and n.iter = number of iterations to perform, because parallel analysis simulates data. the result of this parallel analysis is why i began with five factors. however, we should continue exploring to ensure that this solution is the most appropriate. i now generate a three_factor model, using the same command as earlier to except changing nfactors to 3. three_factors <fa(factor_data, nfactors=3, fm="pa", rotate="oblimin", cor="poly") print(three_factors$loadings, digits=2, cutoff=.3) here, three factors do not really seem sufficient since pa1 loads most questions onto it and we have now two cross loadings. this is not to say the factors are unintelligible. the explanation might be, under this structure, that pa1 represents a default-like patron who likes many library services, that pa3 represents patrons who privilege technology and learning new skills, and that pa2 represents those who really only want to check out items and pickup holds. four_factors <fa(factor_data,nfactors=4, fm="pa", rotate="oblimin", cor="poly") print(four_factors$loadings, digits=2, cutoff=.3) a four factor model, on the other hand, may be a more sensible factor structure; i know of many public library patrons who are mostly interested in isolated and quiet study (pa3). they enter the library without interacting much with staff or other patrons, and consequently show little interest in learning skills from staff. the three factor structured implied that they would have had a modest interest in one-on-one instruction (0.32), which might not be the case. five factors, if we recall, also feels plausible. in that structure, we have preserved the isolated users, and we have added a separate group that is interested in learning from library staff (pa4). that one-on-one instruction (1.02) is high alongside ereaders (0.43) suggests that these patrons may enjoy one-on-one ereader instruction. pa5 is now a new factor which represents patrons who are largely interested in ezone, rhode island’s electronic book collection (0.62), and advanced technology (0.90). these may be patrons who do not physically come to the library much; indeed, when filtering those respondents who load highly on this factor (see below for factor scores), i find that they are significantly less frequent visitors to the library. we could consider six factors: six_factors <fa(factor_data, nfactors=6, fm="pa", rotate="oblimin", cor="poly") print(six_factors$loadings, digits=2, cutoff=.3) this structure also makes sense, but it suggests that users who like ezone also might want to take classes. this may be true but would merit further investigation. the five factor model, recommended by parallel analysis, makes a lot of sense, but it has a factor which loads on only two questions; generally you want factors to load on at least three items. on the other hand, the three factor model does not appear to make as much sense. and the others all seem to have positives and negatives. so, which structure is “correct”? unfortunately, you can’t tell. one possible way might be to try and replicate certain factor structures across many different environments with confirmatory factor analysis. the structure which replicates best would more likely be the “true structure.” another way is to attempt to use replication analytics on the current dataset. for example, provided the dataset’s large enough, you might halve it and compare the factors generated in each sample. osborne (2014) recommends this technique and beaujean (2014) presents the applicable r code. yet all of the factor structures reveal similar service-related patterns. we can fairly confidently assert that some patrons show (group-level) interest in quiet study and solitary space. this pattern emerged in all factor structures. we probably also have patrons who really only seem to care about core borrowing services (holds, physical circulation). this is not to say i will not perform future analyses (i will), but i will proceed as if these patterns do exist. in other words, regardless of which structure is “true,” i have learned something of interest regarding my users. what’s in a name? however many factors you settle on, you should name them. recall our five factor model, where factor pa1 represented patrons who wanted to attend programs (0.69) and learn new skills (0.90). pa1 is not a particularly illuminating name. we might instead label that factor something like the li(brary)-curious. i would call pa3 the invisible patrons, a term i have used to describe those who come in and use our services but, because they keep to themselves, do not appear on many of our measures; they are, effectively, invisible. factor scores finally, you can choose to generate factor scores and append those to individual respondents. this is how i was able to recruit specific focus group respondents. osborne (2014) distrusts factor scores as they are not known for strong replication. he instead recommends more rigorous methods like structural equation modeling. i think, though, that when used with caution and care regression factor scores are ok. if you would like to generate such scores, then you attach them to the original dataset like so: factor_data <cbind(factor_data, five_factors$scores) write.csv(factor_data, file="new_factordata.csv") these commands first add factor scores to the original file (factor_data) as another column and then exports that file into a new file (here called new_factordata.csv). below is a screenshot (edited for clarity) of the new csv file. the scores represent how much someone “loads” on a particular factor. participants who load negatively or positively on all factors should probably be ignored. ignoring them makes it more likely that you can find someone with clear and definite opinions. for example, respondent 10 loaded (very) negatively on all factors. reviewing this person’s survey responses, i see that he/she only indicated interest in picking holds up at the library. this also helps me reflect on my survey design, as it suggests he/she did not really understand the statements presented; he/she should have indicated a high interest in physical borrowing as well but did not. in contrast, respondent 5 loaded highly on pa3 (1.15) and modestly on pa5 (0.59). these factors represent invisible patrons (quiet study, computer usage) as well as attending programs and learning new things. such a respondent appears to want to stick mostly to him/herself but also wants to participate in programs. what kind of programs, for instance, might interest this person? limitations of factor analysis; a note on humble expectations credible and generalizable research is hard. in this context, one would have to perform an efa analysis and then perform confirmatory factor analyses, all the while trying to improve both the measurement tools and the library’s services. you shouldn’t expect to receive revelations based on one efa. as osborne (2014) emphasizes, “many researchers use confirmatory language and draw confirmatory conclusions after performing exploratory analyses. this is not appropriate. efa is a fun and important technique, but it is what it is” (p. 119). therefore, results should be interpreted cautiously as they are not necessarily generalizable to other libraries. for instance, readers should not believe that, because my survey had identified what i call invisible patrons, that they necessarily exist at other libraries, even those of similar structure (they obviously do, but one efa cannot determine that). it is also important to understand that locally-designed survey questions are not necessarily reliable or valid measures. flaws in survey design, many of which i’ve mentioned already in regard to my own survey, introduce uncertainty, clouding results. nor should factor loadings be accepted as gospel. because efa needs considerable sample size, often at least several hundred participants, significant uncertainty exists around these loadings. i would recommend, then, not interpreting, for example, a factor loading of 0.75 as literally 0.75. instead, i would suggest that this is a decent-sized loading and reflective of a potentially strong relationship. it is much harder to generate confidence intervals in factor analysis, so i did not cover them here, but make sure that you approach all point estimates with skepticism. because of these limitations, i do not think that efa based on likert survey data makes for appropriately generalizable research. i think it is best used internally to analyze a particular library’s services. the librarians can refine their analysis as well as their survey instruments to try and identify structures which best approximate their experiences but also offer insights. in that respect, it bears repeating that the thrust of this article is to help librarians use efa in r to improve internal services. it is not to argue that the patron types i had identified in our library’s survey generalize to other libraries, public or otherwise. conclusion despite the aforementioned limitations, i find efa to be a helpful tool for understanding patron profiles and for improving survey instrumentation. efa has helped me to not only refine survey instruments but also to consider potentially insightful “patron profiles.” in our needs assessment specifically, it helped me to recruit opinion-diverse focus group participants. please remember that this is a very basic introduction to using efa in r. it should suit most practitioner purposes. do not hesitate to consult some of the excellent literature on efa for further information, notably osborne (2014). references beaujean, a.a. (2014). r syntax to accompany best practices in exploratory factor analysis (2014) by jason osborne. retrieved from https://www.dropbox.com/s/5cif1thztkodvqf/bestpracticesefa_rcode.pdf?dl=0 costello, a.b., & osborne, j.w. (2005). best practices in exploratory factor analysis: four recommendations for getting the most from your analysis. practical assessment, research & evaluation 10(7). retrieved from https://pareonline.net/getvn.asp?v=17&n=15 kwon, n., & gregory, v.l. (2007). the effect of librarians’ behavioral performance on user satisfaction in chat reference services. reference & user services quarterly 47(2), 137-148. osborne, j.w. (2014). best practices in exploratory factor analysis. createspace independent publishing platform. van der eijk, c., & rose, j. (2015). risky business: factor analysis of survey data — assessing the probability of incorrect dimensionalisation. plos one 10(3). retrieved from https://www.ncbi.nlm.nih.gov/pmc/articles/pmc4366083/ appendix all code in this document: # installs and loads the required package for the fa function install.packages(“psych”) library(psych) # identifies r’s current working directory; must match your # .csv file location getwd() # creates a variable in r of a.csv file factor_data <read.csv(“factor_cleaned.csv”) # displays first several rows of data in variable to make sure # it loaded correctly from the spreadsheet head(factor_data) # performs the kaiser-meyer-olkin test kmo(factor_data) # the command below creates a variable in r for a five factor model # using the arguments factor_data, as the data input, nfactors as the # number of factors, fm as the factoring method, oblimin as the # rotation method, and polychoric as the correlation five_factors <fa(factor_data, nfactors=5, fm="pa", rotate="oblimin", cor="poly") # the command below prints to terminal the factor loadings from the # five_factors variable to two digits, omitting any value below # (absolute value) of .30 print(five_factors$loadings, digits=2, cutoff=.3) # changing the number of factors just means changing the nfactors # argument, as below four_factors <fa(factor_data, nfactors=4, fm="pa", rotate="oblimin", cor="poly") print(four_factors$loadings, digits=2, cutoff=.3) # the command below performs a parallel analysis on the variable # factor_data, using pa as the factoring method, factor analysis # (fa) as the values to show, number of iterations to 20 for the # simulation, and the correlation = polychoric; the parallel analysis # recommends the number of factors in the variable factor_data fa.parallel(factor_data, fm="pa", fa="fa", n.iter=20, cor="poly") # appends factor scores to the factor_data variable factor_data <cbind(factor_data, five_factors$scores) # creates a new .csv file called new_factordata with the # new factor_data file (factor scores) write.csv(factor_data, file="new_factordata.csv") about the author michael lives and works in rhode island. he’s a recovering joycean and a reluctant instrumentalist. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – considered content: a design system for equity, accessibility, and sustainability mission editorial committee process and structure code4lib issue 50, 2021-02-10 considered content: a design system for equity, accessibility, and sustainability the university of minnesota libraries developed and applied a principles-based design system to their health sciences library website. with the design system at its center, the revised site was able to achieve accessible, ethical, inclusive, sustainable, responsible, and universal design. the final site was built with elegantly accessible semantic html-focused code on drupal 8 with highly curated and considered content, meeting and exceeding wcag 2.1 aa guidance and addressing cognitive and learning considerations through the use of plain language, templated pages for consistent page-level organization, and no hidden content. as a result, the site better supports all users regardless of their abilities, attention level, mental status, reading level, and reliability of their internet connection, all of which are especially critical now as an elevated number of people experience crises, anxieties, and depression. by erinn aspinall, amy drayer, gabe ormsby, and jen neveau design systems: an introduction w3c standards call for designing an online experience that is equitable for people with diverse abilities. building on this guidance, the university of minnesota libraries (libraries) developed a technology-agnostic design system and applied it to the health sciences library (hsl) website [1]. the system outlines core principles and strategies for accessible, ethical, inclusive, sustainable, responsible, and universal design across its content, code, and design elements. if design systems are not on your radar, jeremy keith provides this description: the generally-accepted definition of a design system is that it’s the outer circle—it encompasses pattern libraries, style guides, and any other artefacts. but there’s something more. just because you have a collection of design patterns doesn’t mean you have a design system. a system is a framework. it’s a rulebook. it’s what tells you how those patterns work together [2]. this article aims to outline the rationale behind using a design system, and how it was practically implemented to uphold our design system’s core principles. old problems, new solutions the libraries has a distributed development environment. the libraries’ web development team manages the primary websites, while separate departments each have responsibilities for managing the online catalog, digital repositories, springshare products, and online finding aids interface. web content ownership is also distributed; with owning and editing rights provided to all 300 staff members. early efforts to take a more unified approach to web development began in 2015. the web development team produced semver releases (a code versioning method) of an assets repository, which were made available from the main website server. developers and content owners could reliably point at any one version of the library assets to receive some cohesive branding and support [3]. the assets contained images, css, and javascript. the semver releases acted somewhat like a content delivery network (cdn), but they relied on the main website server’s uptime. this model included a brief style guide built on the bootstrap framework and relied primarily on the framework’s code-focused documentation. due to a lack of asset repository adoption, communication, and content documentation, however, the websites managed outside of the web development team quickly became outdated. in 2017, the libraries adopted libguides, and guidelines were established by the implementation team. again, limits of ongoing support for these guidelines led to their inconsistent application. these experiences made it clear that the libraries needed a more established framework to meet its goal of cohesion and clarity with distributed content ownership. during our libguides implementation, the web development team began investigating how web principles might manifest within our work. these principles — accessible, ethical, inclusive, sustainable, responsible, and universal design — aimed to create a user-centered approach to web development that addressed real and urgent community needs. a driving factor for a principles-based approach came from a 2018 university of minnesota student survey which reported that 42% of the participating students were diagnosed with at least one mental health condition in their lifetime, and nearly 20% had a diagnosis within the past year [4]. during covid, these numbers have jumped to a 48% diagnosis rate for students, with 21% diagnosed in the past year [5]. additionally, a 2015 study from the national council on disability reports that approximately 11% of undergraduate students have a disability and that students with disabilities attend college at rates similar to nondisabled students. however, their completion rates are much lower, with only 34% of students with disabilities finishing a four-year degree in eight years [6]. in comparison, the six-year national graduation for all students is 62% [7]. with library websites serving as a common tool to support student success, the need to move towards content that is accessible to all users was clear. as we moved towards a user-centered, principles-based approach, the web development team was able to make accessibility changes to the theme but had little control over the accessibility of the content. the primary intent of the libraries website is to get a person to their information destination, and in this way it acts as a directory. as a directory to other resources, the web principles prompted work to reduce obstacles a person might experience in their journey. we could make optimizations to the code and server setup but wanted to push ourselves to reach every content editor and developer. we heard about design systems through dan mall and brad frost in a video series by superfriendly, and it seemed like a solution to our concerns [8]. after warm reception to a presentation in principled web design to the stakeholders for the hsl web redesign project, the web development team’s user interface developer began a nearly year-long research and development project to write principles using shopify’s polaris design system as a model [9, 10]. the libraries’ design system was born in a google doc. we began by integrating an internal values discussion into principles to guide the work, then developed content guidelines with our content strategist for review and approval by the web content management committee, the libraries steering committee for content on the web. as content was audited and rewritten to new guidelines, code patterns and components were developed, and the live version of our design system was written with the cleanest most semantic accessible code possible (figure 1). by developing the patterns we hoped to put into practice, we had a shared language to move towards our goal to produce a highly user-centered, task-focused, inclusive website. figure 1. libraries design system in progress as we began developing our design system, we learned that the university of minnesota’s university relations team was working on a pattern library, which eventually evolved into a design system (figure 2) [11]. when reviewing the university’s pattern library, we found that we had slightly different goals. our design system incorporated principles and content sections and avoided javascript dependent components, while the university’s pattern library reflected broader user goals and a far more diverse clientele that were out of scope for the work of the libraries. we borrowed liberally from the university’s efforts as well as existing design systems where possible, modifying content and code to meet local needs and priorities. figure 2. folwell design system developed by the university of minnesota’s university relations the libraries’ design system is technology-agnostic, allowing for development from any operating system. as a technology-agnostic tool, the design system required removing complexity and frameworks from the development stack. when it was migrated out of google docs, it was coded into pure html and css published in a github pages repository with some simple gulp commands. this allowed for quick iteration during the communications and development phase. presenting the solutions in html rather than scripting languages provides the universal outcome that colleagues should aim towards rather than dictating the method to generate code as it will present in the dom (i.e., we wrote only html and did not additionally document how to write it in angularjs). with the markup and intent publicly documented in the design system, it is possible to have deeper and more meaningful conversations to progress towards a mature web. setting the culture shift practical implementation of our design system on the hsl site was prompted by the move from drupal 7 to drupal 8. before detailing the technical considerations of this process, we will share how we addressed the known sociocultural complexities that required attention to ensure successful design system application. namely, the use of our design system required a cultural shift towards our shared, user-centered principles to achieve a more unified approach. this cultural shift took time and support as we moved through the stages of separation, transition, and incorporation that are common to any rite of passage [12]. separation shared principles helped us begin to separate from our long history of distributed ownership as they provided an important foundation on which to build a cultural shift. creating separation allowed us to be forward-thinking in our work and move away from a “how we’ve always done things” approach. we started by introducing the project team to a user-centered principles-based approach through a locally-developed presentation (figure 3) [13]. the presentation built awareness around the principles on which our design system was built, and it was instrumental in separating from old ways of working as it developed a shared vocabulary and an understanding of the user-centered goals for web content — ultimately leading to team buy-in. figure 3. slide from a presentation on overcoming self-centered design transition separation was followed by a transition time when we learned about how to best create a user-centered site. with team buy-in in place, the design system took center stage, showing how the principles shared in the presentation could be practically applied to the hsl site. as a result, our design system reinforced a user-centered approach and helped us achieve team readiness. as the project team became familiar with our design system, we cemented our user-centered mindset when we incorporated what we learned and applied best practices across all site elements. and because many viewpoints and priorities were reflected in our project team — knowing that we were all using a unified approach and working towards a common purpose created a safe space to share and test ideas. as a result, we were able to take a highly collaborative approach to the hsl redesign from the very beginning, which proved to be a meaningful step towards a more unified approach. throughout the redesign process, we shared our process, content, and designs broadly with staff at departmental and management level meetings. we also sent updates in our internal newsletter and had broad representation on our project team from across our library staff. at these contact points, we reinforced a user-centered approach and spent time helping colleagues join in the learning process, offering opportunities to understand and question our approach, voice concerns, surface problems, and acclimate to new site features. this allowed our whole organization to join in the transition. incorporation with the design system in place, we were able to consider the user experience as the sum of the experiences encountered by users — from lean code that can impact load time, to accessibility best practices, to content that is readable and thoughtfully organized to support successful task completion. although this culture shift didn’t happen overnight, the shift was made faster through a uniform approach, centered on our design system. we acknowledge that a culture shift takes time. it requires an application of complex skills and an ability to navigate complex relationships. we recommend building in extra time to accommodate this shift, as well as concrete plans to facilitate the process. this way, the project team members feel authority and ownership over the process, and the expectation of authority is understood and accepted within the larger organization. investing time and resources to create a shared purpose and approach has proven to have lasting returns, as conversations could start at the point of shared understanding as we introduced a centralized drupal-based editorial workflow and extended design system standards to non-drupal content that is pulled dynamically to our site (e.g., hours, news, events) so we maintained our design system standards and user-centered goals across all content elements. content first once the culture shift was in motion, we could more effectively apply design system guidelines. this work began with a holistic look at site content, with a review of every page (and every word on every page) to ensure that our content-related guidelines were met. core strategy statement the content review process began by writing a core strategy statement — a tool to guide content creators through evaluating existing content along with how and when to create new content. it does this by identifying the audience and tying their needs to business goals. the strategy statement was developed using a madlibs exercise, modeled on sara wachter-boettcher’s 2018 workshop “coaching content: turn any colleague into a content ally” [14]: the (site) supports (primary audience, and it can’t be everyone!) (primary tasks, limit to 3), which helps us (internal goal). in order to do this, our content must be less (problem area 1), (problem area 2), (problem area 3), and more (aspiration 1), (aspiration 2), (aspiration 3). filling in the placeholders for hsl, our final strategy statement read as follows: the university of minnesota health sciences libraries website, as part of the university library system, seeks to provide low-barrier access to health sciences information in order to support 1) discipline-specific information needs of ahc (academic health center) faculty, students, and researchers; and 2) effective and meaningful engagement with library staff, services, and tools. in order to do this, our content must be less hidden, less complex, and less confusing, while being more task-driven, more accessible, more user-focused, and more contextualized. card sort a card sort was used to review existing content against our content strategy. from a practical standpoint, we placed printed labels on sticky notes for each published page. the sticky notes included the page title and a unique id that matched our spreadsheet records. our strategy group then worked with the web development team to sort the sticky notes on large sheets of paper (figure 4). figure 4. card sort. the combined paper/sticky note approach let the content drive design. as we moved through each page as a physical object, separated from legacy structures, we were able to organically identify existing patterns and start creating and defining the primary content buckets: services, resources, spaces, exhibits. each content bucket was simply defined for easy application to the content. for example, the services bucket was defined as the stuff we do (the things requiring interaction with staff), while the resources bucket was defined as the stuff we have (items in collections, or subscriptions, that users can discover on their own). we also began to markup the card sort with brainstorming ideas, identifying pages to be combined or deleted, as well as those that would need reworking to fit into a new bucket. content review (evaluating for content r.o.t.) after the content buckets were identified, the project team moved into the content review process, where we audited all existing drupal content, both published and unpublished. all of the 167 pages of unpublished content were reviewed and all the pages could be removed permanently. for the 500 pages of published content, each page was reviewed by two team members, who determined if there was content r.o.t., or content that is redundant, outdated, or trivial. those determinations were followed by a recommendation for final action: keep, delete, or update (figure 5). with two reviewers for each page, we could determine if there was consensus or a need for further review and discussion. figure 5. content review spreadsheet. templates simple templates were developed for each content bucket to align with design system standards and create a consistent experience as users moved between pages in the same bucket. the templates were intentionally basic, with the hopes that their simplicity would encourage compliance. the services template, for example, includes a title and a plain language description. it also includes a way to contact someone to engage in that service — a required element of the services template — as well as placeholders for sidebar content, including contact information, and related services and resources (figure 6). the templates were used as the project team members reviewed and re-wrote the pages. they served as shared tools to foster collaboration between the project team and the subject matter experts, and they gave writers an opportunity to completely re-imagine their content in alignment with a uniform approach. by effectively separating the content from its presentation, the templates allowed subject matter experts to do what they do best — know stuff — and content and user experience experts to deliver that content in the best way to users. figure 6. service content template example. writing the design system brought a set of core principles and their practical application to the writing process. it contained specific guidelines for writing inclusive and accessible content with a focus on plain language, as well as a cohesive editorial style (presented both in full, as well as in a brief checklist format). it served as a training tool for the project team to learn and apply content standards and best practices. all of this led to a shared sense of the “why” and the “how” behind user-centered content that set the stage for the shift towards a more unified approach. despite these efforts, content writing proved to be the hardest and lengthiest part of the project due to perceived and real challenges around content ownership and authority. as a result, the writing process took three times longer than projected. ultimately, we were able to come together with a cohesive approach, but our experience with the writing process reiterated the importance of supporting a cultural shift within an organization when technical changes disrupt established norms. ongoing quality assurance through workflow drupal’s workflow system provided the tools to ensure an ongoing review of content changes. the workflow system established “states” for content revisions — new draft, for editorial review, published, archived — and delegated different “transitions” to different “roles” (authors could “create new draft” and “request editorial review”; site editors could “publish” and “archive.”) this helped ensure that the standards for content and design so painstakingly developed as the site was built could remain in place through a more uniform approach to content oversight. theoretical code to practical application as content discussions progressed, the web development team began discussing the technical implementation of drupal’s content editing interface. taking the agnostic html-based patterns from our design system and incorporating them into drupal required work on several different levels: the authoring system (drupal’s content create and edit interfaces) needed to present the design system patterns in a way that content editors could easily select and use without concerning themselves with the intricacies of html code; drupal and our drupal theme, which rely on an elaborate nesting of templates using the twig templating system, had to be set up to output this staff-created content consistent with our design system; site content generated by query builders (views), feed harvesting (e.g. from google calendar and wordpress), or administratively-configured content (blocks) also needed to use the design system components. presenting patterns to content editors out of the box, drupal presents a content editor with the “body” field, a large, freeform field for a web page’s main content. typically, this field offers a wysiwyg editor with a wide range of formatting and media options. our goal was to “empower through constraint” by reducing the overwhelming number of options presented and better aligning these options with the components outlined in our design system. we found a solution in the paragraphs module. paragraphs allows drupal site administrators to set up a number of paragraph types, each of which is a collection of fields presented as a modular package. the previously-mentioned freeform “body” field may then be reconfigured to present a list of paragraph types that may then be inserted, arranged, and rearranged within the body area (figure 7). as each paragraph pattern was developed, we ensured the authoring interface also maintained the design system’s accessibility standards. figure 7. within a content section, editors may insert certain types of paragraphs. here, a card deck is already added (with sub-elements). the dropdown menu shows other allowed paragraph types suitable for adding to the section. the content section, shown, is in turn a sub-element within the page’s “body” field. we developed a suite of paragraph types that mirrored many of our design system components. thus, we could provide, for example, an “informational image” paragraph type that included fields both for displayed content (the image, the caption, and the image credit) and for display options available to content editors (such as alignment and relative width (figure 8). the user interface developer and drupal developer spoke frequently about the nuances of the theoretical code and how to apply the options in an authoring interface. how much control should the author have, what were the options built within each component, and how were these decisions best communicated to the people who would use them? figure 8. after choosing to add an informational image, the options for the informational image paragraph type are available. conforming drupal template output to design system goals drupal’s model for rendering web content is to use multiple nested template files, written in the twig templating language. broadly, the template files provide the html markup, and drupal sends content variables into the template to provide the content within that markup. by design, a system of precedence based on template file names allows individual drupal themes and modules to override default templates at nearly any level, from the html wrapper (<head> and <body>) to individual content fields. the paragraphs module, tailored to match our design system, also provides these template suggestions, allowing us to customize output for each paragraph type. we went one step further, however. to make the relationship of drupal templates with our design system component more clear, we made the mapping explicit. to do this, we used another drupal module called component libraries. this simple module allows the declaration of a new theme template namespace, “components,” that can be referenced by twig’s “embed” functionality. with this module in place, we could map files named for standard drupal template suggestions (“sources”) to files in our “components” directory within our theme (“targets”). this also allowed us to make the component templates cleaner and abstract (figure 9), while using the source files to transform content values coming out of drupal into values that fit the more abstract “target” implementation (figure 10). figure 9. the full “components/image/informational-image.html.twig” file. this implements our design system pattern for an informational image. it has clear expectations of what it is receiving for data, and logic is limited to testing whether a rendered value exists or not, where that affects markup. figure 10. an excerpt of “templates/paragraphs/paragraph–informational-image.html.twig”. we turn user-editable “content” values for relative width and alignment into values better suited for use as html class names (lines 41-45), which is what our abstract design component expects. the “embed” code block (lines 46-52) specifies which component template to use and maps variables sent into the “paragraph–informational-image.html.twig” template to those it will send forward to the design component template. aligning different drupal content sources not all drupal content comes from paragraphs created by content editors, however. content may also be sent to a page by views (drupal’s query builder for listing site content), by blocks (pre-configured content not bound to a single page), or by drupal modules, such as those that harvest events or news feed data. any of these sources for data may be passing in content suitable for various design system components. fortunately, our use of the component libraries module to send data from a specific drupal source template to a target component template provides a solution: any number of drupal instance templates can route to a single component template. each specific instance template will likely have different inbound data. for example, an html class selection may originate as a “yes”/”no” checkbox value in a content field in one source or as a hard-coded value in a php array in another source. the instance templates can manipulate these different kinds of incoming data to fit the expectations of the target template. as the abstract design system components evolve, we need only update the corresponding drupal files in one place: the templates under our ‘components’ namespace. user response launched in january 2020, the new hsl site takes a bold and user-driven approach that incorporates our design system’s core principles of accessible, ethical, inclusive, sustainable, responsible, and universal design. significant changes include a “menu-less” design (navigation sections link to the home page, with additional no drop-down links), no hidden content, links to form actions from the homepage, the search box on every page, and information in context (e.g., the makerspace page includes makerspace workshops, news, events, staff). we love that the site presents our entire library to users at-a-glance, which helps them have a better understanding of the breadth and depth of our work. the usability study and other feedback post-launch suggest that users love it too — even though 80% of our web pages were removed as part of the redesign process. we had very little feedback from hsl staff (making the old catalog interface available, allowing color photos for staff directory), very little feedback from the broader libraries staff (one comment related to aesthetics), and very little feedback from users (one comment about a database that was removed from the homepage). additionally, the post-launch usability study showed good use of the site’s search capabilities, good task completion, and successful navigation through the site (figure 11). the usability study also showed that clear content buckets added context to users for ambiguous terminology (e.g., services and resources). figure 11. usability study results (excerpt). conclusion when implemented, the principles-based design system proved to be critical in our bold and user-centered approach across our code, content, and design. the new site demonstrates the highest levels of usability, accessibility, inclusiveness, and sustainability. it meets and exceeds wcag 2.1 aa guidance and also cognitive and learning considerations. analytics from webpage test and website carbon calculator demonstrate that our site is [15,16]: faster (fully loading in 1.37 seconds), more sustainable (nearly carbon neutral, could offset with 1-2 trees planted / year); and lean (the size of our new homepage is less than 15% of the previous homepage). additional performance statistics show that our new site: moved from 62 http requests to 14 over http/2; reduced the total homepage download from 1,478kb to under 203kb or less; reduced the number of triggers for dom repainting; reduced the number of pages that load javascript (one bit of inline javascript is used for the chat link on every page); and passed the google lighthouse performance audit with 100%. these data points demonstrate that the application of our design system to the hsl site has aligned the site with our core principles. as a result, it better supports all users regardless of their abilities, attention level, mental status, reading level, and strength of their internet connection, all of which are especially critical now as an elevated number of people experience crises, anxieties, and depression. notes [1] health sciences library. [internet]. c2020. minneapolis (mn): regents of the university of minnesota; [date unknown] [cited 2020 dec 8]. available from: https://hsl.lib.umn.edu [2] keith, j. design systems. [internet]. 2018. adactio; [cited 2021 jan 21]. available from https://adactio.com/journal/13844 [3] preston-werner, t. semantic versioning 2.0.0 [internet]. [date unknown] [cited 2020 dec 8]. available from: https://semver.org [4] boynton health [internet]. 2018. health and health-related behaviors university of minnesota – twin cities students. regents of the university of minnesota. [cited 2020 dec 8]. available from: https://boynton.umn.edu/sites/boynton.umn.edu/files/2019-09/cshs-2018-umn-twin-cities.pdf [5] boynton health [internet]. 2020. results from the modified college student health /covid-19 survey. [cited 2021 jan 21]. available from: https://boynton.umn.edu/sites/boynton.umn.edu/files/2020-11/cshs-covid-19-survey-report.pdf [6] national council on disability. 2015. briefing paper: reauthorization of the higher education act (hea): the implications for increasing the employment of people with disabilities. [cited 2021 jan 21]. available from: https://ncd.gov/publications/2015/05192015 [7] u.s. department of education, national center for education statistics. 2020. fast facts: graduation rates. [cited 2021 jan 21]. available from: https://nces.ed.gov/fastfacts/display.asp?id=40 [8] superfriendly co. classes: make design systems people want to use [internet]. c2012-2020. superfriendly co; [date unknown] [cited 2020 dec 8]. available from: https://superfriendlydesign.systems/classes [9] drayer, a. principled web design [internet]. 2018. [cited 2020 dec 8]. available from: https://z.umn.edu/code4lib-issue50-artifact1 [10] shopify. polaris [internet]. [date unknown] [cited 2020 dec 8]. available from: https://polaris.shopify.com [11] university of minnesota. folwell [internet]. c2019. minneapolis (mn): regents of the university of minnesota; [date unknown] [cited 2020 dec 8]. available from: https://folwell.umn.edu [12] van gennep a. 2001. the rites of passage, trans. monika b. vizedom and gabrielle l. caffee. chicago: university of chicago press. [13] drayer a, neveau j. 2019. through the looking glass: overcoming self-centered content. university of minnesota libraries. minneapolis, mn. [14] wachter-boettcher. 2018. coaching content strategy: how to turn any colleague into a content ally. confab. minneapolis, mn. [15] catchpoint. [date unknown]. webpagetest: test a website’s performance. [cited 2020 dec 8]. available from: https://www.webpagetest.org [16] wholegrain digital. [date unknown]. website carbon calculator [internet]. strand (london): scamper ltd. [cited 2020 dec 8]. available from: https://www.websitecarbon.com about the authors erinn aspinall (aspin005@umn.edu) is an associate director at the university of minnesota health sciences libraries where she works to ensure that users have a consistent experience across the library’s full range of services. erinn holds an msi from the university of michigan school of information, and has interests in organizational development and continuous process improvement. amy drayer (adrayer@umn.edu) is a user interface developer at the university of minnesota libraries where her work focuses on developing web interfaces utilizing inclusive design principles, usability, accessibility, and optimization tools to deliver user-oriented experiences. amy has a masters degree in library and information studies from the university of wisconsin-madison. gabe ormsby (gormsby@umn.edu) is a web developer at the university of minnesota libraries, where his focus is on maintaining the libraries’ drupal-based websites and integrating them with other library tools and services. he holds a bachelor of arts from gustavus adolphus college. jen neveau (jneveau@umn.edu) is the web content strategist for the university of minnesota libraries. she’s been working on the libraries websites since 1996, in a variety of capacities, including design, development, & usability testing. jen serves on the board of the university of minnesota communicator’s forum, is co-chair of the libraries dei leadership committee, and holds an advanced certificate from the university’s office of equity & diversity. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – digital archaeology and/or forensics: working with floppy disks from the 1980s mission editorial committee process and structure code4lib issue 34, 2016-10-25 digital archaeology and/or forensics: working with floppy disks from the 1980s while software originating from the domain of digital forensics has demonstrated utility for data recovery from contemporary storage media, it is not as effective for working with floppy disks from the 1980s. this paper details alternative strategies for recovering data from floppy disks employing software originating from the software preservation and retro computing communities. imaging hardware, storage formats and processing workflows are also discussed. john durno, university of victoria libraries introduction in this paper i will discuss tools and techniques i have employed to retrieve content on floppy disks from various 8and 16-bit computer systems dating from the 1980s. in the context of several projects completed over the past two years i have recovered data from approximately 500 floppy disks. disks from different sources exhibited many unique characteristics and required specialized processing to extract the content in a usable form. specifically, this paper describes the tools and techniques i have used to recover content saved to 5.25″ and 3.5″ floppy media from the following systems: ibm pc ms-dos kaypro ii cp/m mac os 7 apple ii prodos apple ii p-system & ibm pc p-system atari 130 xe floppy disks were sourced by way of donations to our archives, from materials in the library collection, and from researchers seeking to access materials on early computing media from their own collections. the different requirements of these use cases also affected the processes employed to retrieve information. the researchers’ interests were best served by simple file recovery, sometimes involving format conversion, while materials destined for archival preservation required a more structured and consistent process. while my main reason for writing this paper is the hope that my methods may prove useful to others confronted with similar problems, these case studies are also intended to support a larger argument. as evidenced by disk imaging toolkits such as those found in archivematica and bitcurator much of our current practice reflects the assumption that tools derived from the domain of digital forensics can productively be applied to all types of digital media, from 1980s floppy disks through contemporary hard drives. i would argue instead that floppy disks from the 1980s are a separate problem domain requiring specialized tools from outside the digital forensics community. relative to earlier eras, today we enjoy a high level of standardization in our computing environments and routinely expect digital content to be intelligible across multiple platforms. the major challenge is managing the sheer volume of data that is stored on modern systems. in the 1980s however these conditions were reversed. while the volumes of data were often miniscule by contemporary standards, a wide range of computing systems developed by competing vendors implemented proprietary technologies up and down the stack, from disk encodings to file systems to applications and their associated file formats. far more than in our own time, digital information from the 1980s was deeply enmeshed in the particular platforms, operating systems and applications that created it. we should not therefore expect to approach the very small amounts of boutique data found on floppy disks with the same tools we use to wrangle the vast amounts of content on newer hard drives.[1] the observation “forensic techniques and tools will not eliminate the problems presented by older media, but they can make certain parts of the preservation process more efficient and more secure”[2] points in the direction my argument will take, but i would argue it does not sufficiently stress the distinction between forensic tools and forensic techniques. while forensic techniques are still, broadly speaking, applicable, in most cases it does not make sense to apply tools developed by the digital forensics community to the problem of data retrieval from floppy disks. often it is not even possible. tools developed by the software preservation and retro-computing communities are generally far more useful as the cases described below will illustrate. disk imaging the first step in any kind of data recovery from floppy media is to create a disk image, a sector by sector copy of the entire disk, including the parts of the disk not normally seen by users (boot sector, file table, sector numbering, data marked deleted but not yet overwritten). any further manipulation of the disk contents (such as content extraction) is done from a copy of the disk image. this has the benefit of minimizing the handling of the original floppy media, and provides a baseline known good state to control for inadvertent changes that could be introduced by subsequent manipulation of the contents. write blocking is critical in the imaging process in order to prevent inadvertent alterations to the source media. in this my approach differs not at all from standard digital forensics methodology. in most cases, the right tools make imaging disks relatively straightforward. in addition to drives of the requisite size (typically 3.5″ and 5.25″), one also needs a controller (circuitry to control the drive) and appropriate software to run the imaging process. historically, floppy drive circuitry was often built into computer motherboards but most modern motherboards have limited capability in this regard. devices like the kryoflux and fc5025 supply external usb-attached drive controllers, with sophisticated circuitry enabling them to bypass the limitations of older disk controllers (for example, they can read disks with both gcr and mfm encodings). at the university of victoria we have three devices capable of imaging floppy disks, each with their own strengths and weaknesses. figure 1. kryoflux controller connected to 5.25′ floppy drive and laptop (via usb) 1. kryoflux. without a doubt, the kryoflux is the most versatile device we have for imaging floppy disks. it consists of a controller, cabling, and software. with appropriate drives it is capable of imaging both 3.5″ and 5.25″ disks and can handle a wide range of encodings and sector formats. it has hardware-level write-blocking. the software has both gui and command line modes (the command line is harder to use but more flexible). the kryoflux is also the most capable of our three devices for recovering data from damaged disks, and is the only device we have that is capable (with a modified drive) of recovering data from ‘flippy’ disks (single-sided floppies with data written on both sides). however it is easy to damage and can only be handled by experienced technical staff. 2. windows 7 workstation. this is a purpose built workstation with a deviceside data fc 5025 usb floppy controller (for 5.25″ disks), a motherboard-attached 3.5″ floppy drive, and standard optical media drive. the fc 5025 controller has proven to be surprisingly capable given its low cost (relative to the kryoflux) and can handle a wide range of disk formats. the 3.5″ floppy drive has been under-utilized but will work with any windows imaging software; for example, ftk imager. the fc5025 unit employs hardware level write blocking, while write blocking for other devices is enabled at the level of the windows registry. this approach is also useful for acquiring content from non-floppy devices, such as external hard drives, cds and thumb drives. 3. industrial motherboard workstation. another purpose-built workstation, this one has a 32 bit industrial motherboard and can boot into dos, windows xp, and linux (centos). 3.5″, 5.25″ and optical drives are all directly attached to the motherboard. of our three workstations we have used it the least for imaging, but it has come in handy for edge cases (see the section on pc p-system disks below). imaging software includes standard open source tools like guymager and dd on the linux partition, the versatile omniflop[3] on the windows xp partition, and dave dunfield’s imagedisk[4] in dos. write blocking is implemented at the application level and by using write protection features on the disks themselves.[5] given its versatility, the kryoflux is the generally preferred option for floppy disk imaging. however, the other two devices have occasionally provided better results in accessing the contents of copy-protected disks and disks with unusual sector formats. and one should always be conscious that floppy drives can silently go out of alignment, resulting in disk images that cannot be read, or worse still, contain corrupted files. having multiple drives and devices available makes it possible to periodically run sanity checks on your imaging output.[6] most imaging software comes with built-in settings for known disk types. however it is not always easy – particularly in the case of posthumous donations of archival materials – to determine what particular combination of hardware and operating system was used to write to the disk in the first place. in some cases the relevant information is noted on the disk label, but in others a certain amount of trial and error may be required. image formats tools developed by the digital forensics community typically save disk images as either aff (advanced forensics format) or ewf (encase/expert witness format). the main advantages of these formats are seen to be that they store metadata, including checksums, within the disk image file itself. that provides some assurance that the metadata is accurate and the image has not subsequently been tampered with. unfortunately, aff up to version 3 has been deprecated by its creator[7] , and ewf is proprietary (albeit well-documented and with some open source support)[8] . for these reasons, and because neither format is well supported outside the forensics community, there is some cause to doubt their suitability as long-term preservation formats. sector image formats would appear to be more suitable for floppy disk preservation, for the following reasons: they are the standard output of imaging tools such as the kryoflux and fc5025. they are widely supported by a range of open source content extraction utilities, their contents are accessible under emulation, and they have been in existence longer than most preservation formats, which bodes well for their continued longevity. the forensics community designs tools for criminal investigators, not for digital archaeologists. tools that support aff and ewf (guymager, fiwalk and the sleuth kit for example), while undoubtedly valuable in their problem domain, were not designed to recover and preserve content from computer media from the 1980s. most of the tools that can recover content from 1980s floppy disks either output or expect to work with standard sector image formats. sector images vs. stream files stream files are a proprietary but publicly-documented format output by the kryoflux controller. they contain a great deal of information and are the easiest to acquire: they contain all the low-level information the kryoflux generates while reading flux transitions, and can be made without knowing any technical details (encodings, sector geometries) about the source media beforehand. however, stream files are quite large in comparison to sector images (roughly 100 times larger, so a stream file for a 360k disk will come out to around 40 mb), and need to be converted to sector images in order to access their data in human-parsable format. the kryoflux software, dtc, can be used to create images from stream files. they are not a preservation format, as noted in the documentation: ‘stream files are hardware specific (to the kryoflux device) and therefore are not intended for long term preservation.’[9] it is worth noting that acquiring only stream files would be a workable strategy for obtaining floppy disk images in situations where it is necessary to use semi-skilled labour to do the imaging work. in such cases the work of converting streams to sector images could be deferred until later. sector images invert the characteristics of stream files: they are typically the same size as the original disk, accessible to emulators and content extraction tools, non-proprietary in most cases, and are not a complete recording of all the low level data the controller was able to read from the disk, such as flux reversals and index blocks. processing images for information storage and access while creating disk images is relatively straightforward, making their contents intelligible to humans can be more challenging. fortunately a wide range of readily available tools exist to facilitate the process. i will be describing some of them below. except where otherwise noted, all of the work was done on a modern lenovo thinkpad 450s running ubuntu linux 15.10. most open source content extraction tools and emulators work well in ubuntu. many of these tools have a command line interface, which makes it possible to script the repetitive parts of the job. also, working in linux provides added security against computer viruses. floppies were an attack vector for viruses in their day and many of the viruses from 25 years ago are still perfectly capable of infecting a modern windows computer. for this reason, files recovered from floppy disk images should be immediately scanned for viruses prior to further handling. i use the open source clamav[10] for this purpose. the images themselves should also be scanned to check for boot sector viruses. while one could preserve disk images only and not extract their contents until such time as they were required, running the extraction process up front facilitates an examination of the image contents in order to ensure the imaging process was successful. structured output as noted above, the desired end product of the content extraction process varies depending on the nature of the project. for digital preservation, one reasonable option would be creating a structured directory containing the disk image, the extracted contents of the disk image, and metadata files: figure 2. an example data structure for the image and output of a floppy disk non-exhaustively, the meta folder may contain: index.txt: a recursive directory listing of the contents of the disk image checksum.txt: an md5 checksum of the disk image, generated immediately after the image was acquired image.log: a log of the image creation process generated by the kryoflux or other imaging software, noting the condition of disk sectors meta.xml: metadata recorded in accordance with archival policy, for example control numbers, date the image was made, text from the disk label, disk image format, tools used to extract the files and so on. av.txt: output from the antivirus scanner time and resources permitting, photos of the disks could also be included in the meta folder. floppy disk labels often contain useful metadata. case study 1: pc/ms-dos floppy disks (3.5″/5.25″) floppy disks created on pc/ms-dos systems were not as prevalent in our processing queues as we originally anticipated. due to their ubiquity in other contexts, these are the easiest kinds of disks to work with, and the only one of the case studies in this paper that digital forensics tools like guymager, the sleuth kit, and fiwalk can handle. disk imaging: all standard imaging tools come with settings for pc/ms-dos disks. because the kryoflux works at the level of the disk encoding rather than the file system, the mfm setting (i4) is appropriate here. for double density disks, the step setting -k2 (indicating 40 cylinders) should be used, for example: dtc -fimage.img -k2 -i4 file extraction: again, multiple options exist for copying individual files from the disk image to the local file system. i typically use the set of utilities collectively known as mtools, which enable unix systems to work with ms-dos file systems. figure 3. partial output of mdir command the following commands produce the desired output, for a disk image with the name “image.img”: recursive directory listing including hidden files, output to the file “index.txt”: mdir -/ -a -i image.img > index.txt recursive content extraction output to subfolder “content”, preserving modification times and file attributes: mcopy -p -s -m -i image.img ::* content in a small number of cases where mdir and mcopy failed to read a disk image one may mount the image in the linux file system and extract the contents that way. use: sudo mount -t msdos -o ro,loop image.img mountpoint/ … where “mountpoint/” is the name of the empty directory where you want the image to mount. standard unix utilities (ls, cp) can be used to work with the files once the image has been mounted. case study 2: kaypro ii cp/m (5.25″) disk imaging: the kryoflux mfm setting (i4) works for kaypro ii cp/m disks. the fc5025 floppy controller has a setting specifically for kaypro ii cp/m disks. file extraction: i used michael haardt’s cpmtools package[11] , which functions similarly to mtools, but for cp/m file systems. one significant difference is that the disk format must be specified, because the cp/m operating system ran on a variety of systems, not just pc-compatibles. in this example, the command line switch -f kpii (for kaypro ii) is added. disk formats are defined in a disk definitions file which, on linux systems is usually located in /usr/local/share/diskdefs. the file contains human-readable text specifying floppy disk geometries (sector lengths, number of tracks, block sizes, and so on) employed by the many different different systems that ran cp/m. a comment at the beginning of most stanzas identifies the system to which it applies. for example, the disk definition for the kaypro ii is contained in the following stanza: #kaypro ii diskdef kpii seclen 512 tracks 40 sectrk 10 blocksize 1024 maxdir 64 skew 0 boottrk 1 os 2.2 end most versions of cp/m did not implement directories for file storage, so recursive output is not necessary. instead, cp/m structured file storage around what were called user spaces, numbered areas where different users could store their files. as most cp/m home computers were single user systems usually there will only be one user area, numbered 0 indicating that files were accessible to all users. figure 4. output of cpmls command for a directory listing a number of different output options are available. the “-l” flag produces unix style output, with permissions, file modification times (not always trustworthy, as illustrated above) and file sizes. cpmls -l -i -c -f kpii image.dsk > index.txt this command extracts all the files on “image.dsk” in user area 0 to the subdirectory “content”, preserving original time stamps. cpmcp -p -f kpii image.dsk 0:* content/ case study 3: mac os, 3.5″ double sided, double density (800k) mac floppy disks from the mid-1980s through the mid-1990s differed from pc disks in two significant ways: the file system, hfs (hierarchical file system) the disk encoding scheme, apple gcr there are a couple of exceptions: the earliest mac 400k floppies had a different file system (mfs, for “macintosh file system”) and the later high density floppies used mfm encoding, not gcr, for pc compatibility. another aspect which may cause confusion is apple’s successor to hfs, developed for os x, called hfs plus (or sometimes, incorrectly, “hfs extended”) and sometimes referred to as hfs. software like the sleuth kit that supports the later variant of hfs do not necessarily support the earlier one.[12] these complications notwithstanding, if you need to work with double density 3.5″ mac floppy disks you will require tools capable of handling gcr encoding and hfs file systems.[13] disk imaging the kryoflux can image 3.5″ double density disks using the “apple dos 400/800k sector image” setting. as pc disk controllers cannot read gcr encodings and the fc5025 can only handle 5.25″ disks, the kryoflux is the only option for imaging mac 3.5″ disks up to 800k unless you have a vintage mac with a working floppy drive. because the later mac hd (1.44k) floppies were mfm-encoded, there is a much broader range of options for obtaining mac hd floppy disk images. most pc floppy disk controllers can handle them, and most pc-compatible disk imaging software (eg. omniflop, ftkimager, or guymager in linux) includes the appropriate settings. file extraction my toolkit of choice for working with hfs file systems is hfsutils, a collection of utilities for unix systems that has been in development since 1996[14] . it works similarly to mtools and cpmtools, except that the disk image needs to be mounted before it can be read. the following command mounts an hfs disk image named “image.dmg”: hmount image.dmg if the command was successful, hmount will return some information about the disk including the volume name, time created and last modified, and number of bytes free. one can capture that output by modifying the above command to: hmount image.dmg > hmount.txt to recursively list directory contents, including hidden files and catalog ids, in long format, output to the file “index.txt”, use: hls -i -a -l -r > index.txt note that the disk image file does not need to be specified, because it was mounted in the previous step. the hfs file system includes resource forks and data forks. every file in the file system may have both. data forks contain unstructured data, while resource forks include structured data relating to the file, including icon bitmaps, positioning of windows, and so on. while much of this information is irrelevant to viewing standard file types on modern systems, data in the resource fork sometimes extends beyond system-specific metadata including, for example, the embedded images in a word processing document.[15] while it is therefore important to preserve resource forks, it is debatable whether resource forks need to be included among the files extracted from the disk image as a routine procedure. if the goal is to read the extracted files on non-mac systems, then resource forks are unnecessary as other systems cannot use them. if the goal is to read the files on a mac, the best approach is probably to mount the disk image and access the files that way. to extract files without the resource fork, to a subdirectory named ‘content’, use: hcopy -r :* content/ to extract files with resource forks, use: hcopy -m :* content/ after working with a disk image, one must unmount it before mounting another one. the following command unmounts the disk: humount case study 4: apple ii the apple ii series was in production from 1977 through 1993, during which time between five and six million were sold.[16] apple ii hardware and software evolved significantly during that period, and even models of the same era could boot into different operating systems. no single approach works for all variants when it comes to processing apple ii disks. in this section i discuss two variants that i have come across.[17] apple ii prodos (5.25″) prodos was first introduced in late 1983 to overcome a number of shortcomings associated with the previous apple dos version, 3.3. it was eventually renamed ‘prodos 8’ after a 16 bit version was released. disk imaging apple dos 3.3 and prodos use the same 16 sector format for 140k disks, so the kryoflux setting for “dsk, dos 3.3 interleave” was appropriate here. the fc5025 has an “apple prodos” setting. the kryoflux command line requires the following parameters for an optimal read of apple dos 3.3 disks: dtc -f<imagename> -x0 -i8 -l8 -dd1p file extraction as hfs was specific to macs, hfsutils cannot be used to extract content from apple ii disks, which had their own file system. initially i mounted a few sample disk images in the kegs apple iigs emulator to examine their contents, which as it turned out consisted primarily of files written in an early version of appleworks.[18] while it was possible to read the appleworks files in the emulator, the process was awkward and slow. the sheer volume of content (hundreds of files on more than 90 floppy disk images) would have rendered that approach problematic for anyone wishing to read more than a handful of documents. figure 5. viewing files in the kegs apple iigs emulator instead i used applecommander to extract and convert the contents of the disk images[19] . applecommander is an open source java program with both gui and command line interfaces. in addition to prodos it can handle a range of common apple ii disk image formats, versions of dos 3.3, apple pascal, and cp/m among them. figure 6. applecommander gui to write a directory listing into ‘index.txt’, use: java -jar ac.jar -ll disk.dsk > index.txt … where “ac.jar” is the name of the apple commander jar file and “disk.dsk” is the name of the disk image you want the directory listing from. the switch “-ll” outputs a long file listing with file name, creation and modification dates, and file sizes. the current stable release version of applecommander (1.3.5) does not have the ability to extract all the files from a disk image via the command line interface, but the interim release candidate (1.3.5.13) does, using the command: java -jar ac.jar -x disk.dsk content this command extracts all the files on the disk image to the subdirectory “content.” it automatically applies a conversion filter to certain types of documents; for example appleworks documents are extracted as text files. while text conversion was acceptable for our project, it would be problematic in cases where it was important to retain formatting. in a curious reversal of the norm, the applecommander gui provides a greater range of bulk extraction options than the command line, including text, rtf, html, and no conversion (binary extraction). case study 5: apple ii p-system & ibm pc p-system 5.25″ the p-system os was developed in the late 1970s at the university of california, san diego. like cp/m, was designed to be portable across different varieties of hardware, including the apple ii, the pdp 11, and later the ibm pc. two variants of p-system disks (apple ii and ibm pc) were donated to our archives, corresponding to two phases of a project undertaken by our computer science department collaborating with the victoria-based artist glenn howarth in the first half of the 1980s. disk imaging p-system disks used the encoding scheme appropriate for the hardware on which they were running, mfm on pc and gcr on apple. for imaging the apple disks, the built-in “dsk, dos 3.3 interleave” kryoflux setting produced disk images readable by the file extraction utilities described below. imaging the pc disks proved to be more of a challenge. the pc version of the p-system os was a rarity in its day. as a consequence of its relative obscurity, there are no built-in settings for it in any of my imaging tools. for example, it is not included in the list of ~150 formats supported by omniflop, nor is it a known format that is supported by the fc5025 and kryoflux controllers. the kryoflux was, of course, able to obtain stream files but my attempts to produce sector images resulted in files that were oddly twice the size they should have been (720k images for a 360k floppy). further analysis of these disks using a program called anadisk indicated the sector numbering was unusual. following a recommendation on the kryoflux forums i reimaged the disks using a dos utility called imagedisk and converted the resulting imd files to img.[20] these worked somewhat better, allowing me to extract some, but not all, of the files on the disks. i believe the remaining problems have to do with how the ucsd p-system handled sector interleaving, but have not yet confirmed this. file extraction as with other test cases, specialized tools were required to process the p-system disks. in this case, the same tools and settings could be used for both the apple ii and ibm pc disk images. i used peter miller’s ucsd-psystem-fs (http://ucsd-psystem-fs.sourceforge.net/), a set of utilities for manipulating ucsd p-system disk images. for both pc and apple disk images, the following command outputs a directory listing from the disk image “disk.img” into the text file “index.txt”: ucsdpsys_disk -f disk.img -l > index.txt the following command extracts the files on the disk image “disk.img” into the subfolder “contents”: ucsdpsys_disk -f disk.img -g contents/ case study 6: atari 130xe 5.25″ atari manufactured their line of 8-bit home computers from 1979 through 1992. the mass-market 130xe model dates from the latter half of that period. atari systems were popular for gaming but were also used for office applications. in this case the disks came from the personal collection of a researcher, not from our archives. the researcher was able to tell us both the model of computer and the software (atariwriter) that was used to create the files on the disks. disk imaging early atari floppy disks were typically fm encoded, single sided, and had 40 tracks with 18 sectors each. later disk drives used mfm encoding and additional tracks per sector, permitting increased storage density. the kryoflux has settings for both types. the “mfm xfd, atari 8-bit” gui setting worked for these disks. file extraction i was unable to locate command line tools to work with atari disk images. instead, we configured an instance of the atari800 emulator, and used atariwriter to read the files from the disk images mounted as virtual floppies.[21] it proved possible to send files as postscript from atariwriter to a print queue on the host system. in this case the emulator was installed on the researcher’s os x laptop and configured to print files to pdf. this would not be an optimal technique for converting hundreds of files as it involves touching each file individually, but it addressed our requirements in this particular case. figure 7. atariwriter document rendered in atari800 emulator automating the process the work of imaging and processing floppy disks typically falls into two distinct phases. in the first phase, the disks are examined in order to determine the optimal imaging and content extraction strategies. once the technical analysis phase is over, however, the work quickly becomes rote: issuing the same commands over and over again as the individual disks are processed. further analysis and decision making is typically only required in the case of damaged disks or in the event anomalous disks are encountered. it is therefore advantageous to script as much of the work as possible. to automate the workflow described above, i have written scripts that reduce the effort required per disk to two brief commands, one to create the image and associated log file; and the other to create the directory structure (as shown in ‘structured output’ above), move the disk image and log to their correct locations, checksum the disk image, extract the contents and directory listing, and run the antivirus scan and log its output to the correct location. the sections above outline the basic building blocks of these scripts. examples of my cp/m and dos content extraction scripts are found in github[22] . they are not in any way sophisticated examples of the art of programming; the good news is they don’t need to be. rendering files on modern systems the topic of rendering obsolete file formats on modern systems is vast and the challenges are many. the following are general observations from my own experience. speaking very broadly, there are two ways to go about accessing files with formats that are not supported on modern systems. one may convert the obsolete formats to their contemporary equivalents, or else recreate a period software environment either on old hardware or more usually via emulation. these approaches are not mutually exclusive; in practice they may be combined as in the atari example above.[23] one of the common characteristics i have observed across multiple sets of floppies is that disks from the same source typically exhibit little variety in terms of file formats. unlike the computers of our own time with their plethora of different applications and associated file formats, personal computers of the 1980s were typically used for a much smaller range of tasks and ran a much smaller range of productivity software. in my experience, it is not uncommon for dozens of disks from the same source to contain nothing but files from a single word processing application. here again, the tools developed for enumerating file types in the context of a forensic investigation are of little use. they typically have problems identifying file types from 1980s systems, even common formats like wordstar and appleworks. there is also less utility in enumerating file types in a set of floppy disks that contain only one or two types of files. figure 8. converting wordstar to pdf via wordperfect 12 running on windows 2000 in virtualbox bulk format migration tools are also problematic. in many cases they don’t exist for the formats in question or are expensive and proprietary. this is particularly troublesome in the case of wordstar files, since wordstar was widely used on multiple computing platforms in the first half of the 80s. our current strategy for converting wordstar documents to pdf involves opening them in wordperfect 12 installed in a virtualized (virtual box) instance of windows 2000 and rendering them as pdf files via adobe acrobat professional 8. wordperfect 12 has wordstar conversion capabilities back to version 3.3. this approach respects the formatting of the original document but is time-consuming as documents must be converted individually by hand. it is far simpler and faster to script a bulk conversion from wordstar to plain text (8-bit ascii) if that will suffice.[24] damaged/problem disks in general, the 5.25′ floppy disks i have encountered have held up well over time, calling into question some of the more alarmist claims about the rate of deterioration of data on floppy media. i have seen numerous examples of disks stored in very average conditions that were readable 30+ years after they were first created. the exceptions have largely been disks containing application files, presumably because these were used heavily, whereas disks meant primarily for saving data would have been used less frequently. the relative proportion of damaged 3.5′ floppies has been higher, despite being on average 10 years newer than their 5.25′ predecessors. the disposition of damaged disks requires careful consideration, not least because disks may be damaged in multiple ways: the disk itself may be damaged beyond the point where it can be imaged the protective case may be damaged or bent, but the disk inside might still be viable the disk can be imaged, but with errors indicating bad sectors, leaving the resulting image in one of two states: it is possible to extract files from the disk image, but some of the contents might be corrupted it is not possible to extract files using the standard tools the disk might be copy-protected, have an unknown sector format (as with the pc p-system disks mentioned above), or be a hard sector disk with enough effort many of these problems can be overcome or at least mitigated. however doing so requires expertise that might not be readily at hand, and time, which is always in short supply. for example, if the protective case is damaged one may try cutting it open and carefully transferring the disk inside to an undamaged case. if the disk is copy protected or has an unknown sector format, there are tools that can be used to analyze its idiosyncrasies and help determine a solution. but, if the data on the disks is of uncertain value, it may not be worthwhile to expend the time necessary to address these problems, and in some cases problems will persist no matter how much effort is expended. here it becomes a case of preserving what one can. if a disk is truly unreadable, the only question is whether to preserve the physical media against the day when another attempt might be made. if the disk can be imaged but with bad sectors, then it seems reasonable to preserve that image along with a log file indicating where damage was encountered. if it is possible to extract files, that should also be done, however the possibility of damage should be clearly indicated in the metadata accompanying the image and extracted files. conclusion: forensics or archaeology? this paper was originally envisioned as a technical note exploring the disk image ingestion capabilities of our library’s digital preservation storage system, archivematica. however during the early stages of research it became apparent that archivematica could only perform a bit-level ingestion of the floppy disk images i had been acquiring in the course of my data recovery projects. also, even had it been able to extract individual files, archivematica did not recognize common 1980s formats like wordstar and appleworks, so was limited in its capacity to migrate their contents to archival formats.[25] archivematica relies heavily on open source tools developed by the digital forensics community. as noted above, these tools (for example, the sleuth kit and fiwalk) are not designed to handle the peculiarities of 1980s floppy disk image formats. the comparative recency of the emergence of digital forensics as a professional discipline seems a likely explanation. most digital forensics software in common use dates from the late-1990s onward and reflects a disciplinary focus on retrieving data from contemporary storage media, not media long considered obsolete. archivematica’s disk image ingestion capabilities appear to have been optimized to work with images created by bitcurator[26] , or more accurately with its forensic imaging software guymager. guymager is limited to imaging within the constraints of the disk controller on the physical hardware of the underlying system, which in a typical configuration would be a pc controller. this means it lacks a number of the specialized abilities of the kryoflux mentioned in the section “disk imaging” above, making it a less than optimal choice for floppy disk imaging. it should be noted that in addition to the standard suite of forensics tools, bitcurator includes a number of tools that originated outside the digital forensics community. specifically, it includes mtools and hfsutils, both mentioned above, and hfs explorer for accessing the contents of older mac floppies. it would be straightforward to further expand bitcurator to encompass most of the other software discussed in this paper, and this could in fact be done by the end user. building similar content handling capabilities into archivematica would be considerably more effort given the more tightly coupled nature of its tool chain, and it is possible to question whether it would be worth the effort. in my experience, recovering content from floppy disks involves enough variables to sufficiently confound any attempt to fully automate the process. as the case studies above illustrate, employing a variety of specialized tools represents a more viable approach to content recovery from 1980s floppy disks than relying on the software found in archival toolkits. given the long involvement of the retro-computing and software preservation communities with this problem domain, it is not surprising that the most effective tools originate from those communities as distinct from the realm of digital forensics. about the author john durno is head of library systems at the university of victoria (uvic), where he manages the team responsible for building and maintaining the libraries’ computing environment. for the past two years he been working with uvic archives and special collections to develop preservation strategies for at risk digital materials on volatile media. prior to joining uvic in 2006 john coordinated province-wide library technology projects for the british columbia electronic library network. footnotes [1] limitations of digital forensic software have been noted within the forensics community itself. see beebe, n. digital forensic research: the good, the bad and the unaddressed. advances in digital forensics v, ifip aict 306. 2009. doi:10.1007/978-3-642-04155-6_2: ‘one of the successes identified in the previous section was the collective ability to archaeologically identify, excavate and examine digital artifacts. the problem, however, is that knowledge and expertise are heavily biased toward windows, and to a lesser extent, standard linux distributions. the fat12/16/32, ntfs and ext2/3 file systems, the operating systems that implement them (windows 9x/me/nt/xp/vista and various linux distributions), and common user applications installed on them (microsoft internet explorer and outlook, mozilla firefox and thunderbird, etc.) have been well studied. researchers have paid insufficient attention to other operating systems, file systems and user applications, especially in light of current market trends.’ [2] in kirschenbaum m, ovenden r, redwine g. 2010. digital forensics and born-digital content in cultural heritage collections. available from https://www.clir.org/pubs/reports/pub149/pub149.pdf [3] omniflop disk imaging tool available from: http://www.shlock.co.uk/utils/omniflop/ [4] dunfield, d. disk image/software image archive. available from: http://www.classiccmp.org/dunfield/img/index.htm [5] for a full description of this workstation and its construction, see durno, j, & trofimchuk, j. 2015. digital forensics on a shoestring: a case study from the university of victoria. code{4}lib journal 27. available from: http://journal.code4lib.org/articles/10279 [6] see, for example, formatted hd dos floppies reading as unformatted. 2016. kryoflux support forums. available from: http://forum.kryoflux.com/viewtopic.php?f=3&t=1166 [7] aff format deprecated. available from: https://sourceforge.net/p/guymager/wiki/aff%20format%20deprecated/ [8] forensics wiki. encase image file format. available from: http://www.forensicswiki.org/wiki/encase_image_file_format [9] louis-guérin, j. 2013. kryoflux stream file documentation. revision 1.1. available from: http://www.kryoflux.com/download/kryoflux_stream_protocol_rev1.1.pdf [10] clamav. available from https://www.clamav.net [11] haardt, m. cpmtools 2.0. available from: http://www.moria.de/~michael/cpmtools/ [12] sleuthkitwiki. hfs. available from: http://wiki.sleuthkit.org/index.php?title=hfs [13] for further discussion of the challenges of working with early mac floppies, see purity, s. working with macintosh floppy disks in the new millenium. available from: http://siber-sonic.com/mac/newmillfloppy.html [14] leslie, r. hfs utilities. available from: http://www.mars.org/home/rob/proj/hfs/ [15] wikipedia. resource fork. available from https://en.wikipedia.org/wiki/resource_fork [16] ‘apple ii’ series, available from https://en.wikipedia.org/wiki/apple_ii_series [17] information about the evolution of apple ii operating systems is widely available, see for example ‘dos 3.3, prodos & beyond’ available from http://apple2history.org/history/ah15/ [18] kegs apple iigs emulator. available from: http://kegs.sourceforge.net [19] green, r. applecommander. available from http://applecommander.sourceforge.net/ [20] for more information on this topic, see ucsd p-system disks (pc version). 2016. kryoflux support forums. available from http://forum.kryoflux.com/viewtopic.php?f=3&t=1153 [21] atari800. available from http://atari800.sourceforge.net/ [22] sample cp/m and dos ingestion scripts, see https://github.com/jdurno/floppy-utils [23] for an excellent discussion of the challenges and opportunities afforded by emulation as an access strategy, see dietrich d, kim j, mckeehan m, & rhonemus a. 2016. how to party like it’s 1999: emulation for everyone. code{4}lib journal 32. available from: http://journal.code4lib.org/articles/11386 [24] the basic priniciple is described in: just solve the file format problem. wordstar. available from http://fileformats.archiveteam.org/wiki/wordstar. . also see my implementation of the principle at: https://github.com/jdurno/floppy-utils/blob/master/wsconv [25] while wordstar is listed in the pronom registry that archivematica relies on for file format identification, its ability to recognize the wordstar format appears to be based on the file extension. unfortunately, appending file extensions to identify file types was not as common in the 1980s: while i have encountered hundreds of wordstar files in my work to date i have never seen one with a .ws, .ws3 or equivalent extension. [26] reference “wherever possible, use bitcurator packages for forensics tools”. archivematica wiki. https://wiki.archivematica.org/digital_forensics_image_ingest works cited atari800. available from http://atari800.sourceforge.net/ beebe, n. 2009. digital forensic research: the good, the bad and the unaddressed. advances in digital forensics v, ifip aict 306. doi:10.1007/978-3-642-04155-6_2 chessman s. 1996. dd. linux journal 32. available from: http://www.linuxjournal.com/article/1320 diamond e. 1994. the archivist as forensic scientist – seeing ourselves in a different way. archivaria 38. available from: http://journals.sfu.ca/archivar/index.php/archivaria/article/view/12031/1300 dietrich d, kim j, mckeehan m, & rhonemus a. 2016. how to party like it’s 1999: emulation for everyone. code{4}lib journal 32. available from: http://journal.code4lib.org/articles/11386 durno j. 2016. floppy-utils. available from: https://github.com/jdurno/floppy-utils durno j, trofimchuk j. 2015. digital forensics on a shoestring: a case study from the university of victoria. code{4}lib journal 27. available from: http://journal.code4lib.org/articles/10279 forensics wiki. encase image file format. available from: http://www.forensicswiki.org/wiki/encase_image_file_format green, r. applecommander. available from http://applecommander.sourceforge.net/ guymager wiki. aff format deprecated. available from: https://sourceforge.net/p/guymager/wiki/aff%20format%20deprecated/ haardt, m. cpmtools 2.0. available from: http://www.moria.de/~michael/cpmtools/ just solve the file format problem. wordstar. available from http://fileformats.archiveteam.org/wiki/wordstar kirschenbaum m, ovenden r, redwine g. 2010. digital forensics and born-digital content in cultural heritage collections. available from https://www.clir.org/pubs/reports/pub149/pub149.pdf kegs apple iigs emulator. available from: http://kegs.sourceforge.net kryoflux support forums. 2016. formatted hd dos floppies reading as unformatted. available from: http://forum.kryoflux.com/viewtopic.php?f=3&t=1166 kryoflux support forums. 2016. ucsd p-system disks (pc version). available from: http://forum.kryoflux.com/viewtopic.php?f=3&t=1153 leslie, r. hfs utilities. available from: http://www.mars.org/home/rob/proj/hfs/ louis-guérin, j. 2013. kryoflux stream file documentation. revision 1.1. available from: http://www.kryoflux.com/download/kryoflux_stream_protocol_rev1.1.pdf pollitt m. 2010. a history of digital forensics. advances in digital forensics vi, ifip aict 337 purity, s. working with macintosh floppy disks in the new millenium. available from: http://siber-sonic.com/mac/newmillfloppy.html ross s, gow a. 1999. digital archaeology: rescuing neglected and damaged data resources. available from http://www.ukoln.ac.uk/services/elib/papers/supporting/pdf/p2.pdf sleuthkitwiki. hfs. available from: http://wiki.sleuthkit.org/index.php?title=hfs weyhrich, steven. dos 3.3, prodos & beyond. available from: http://apple2history.org/history/ah15/ wikipedia. resource fork. available from https://en.wikipedia.org/wiki/resource_fork wolverton, m. 2016. digital forensics in the library. nature. vol 534: 139-140. available from: http://www.nature.com/polopoly_fs/1.19998!/menu/main/topcolumns/topleftcolumn/pdf/534139a.pdf subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – geocoding lcsh in the biodiversity heritage library mission editorial committee process and structure code4lib issue 2, 2008-03-24 geocoding lcsh in the biodiversity heritage library reusing metadata generated through years of cataloging practice is a natural and pragmatic way of leveraging an institution’s investment in describing its resources. using library of congress subject headings (lcsh), the biodiversity heritage library generates new interfaces for browsing and navigating books in a digital library. lcsh are grouped into tag clouds and plotted on interactive maps using methods available within the google maps application programming interface (api). code examples are included, and issues related to these interfaces and the underlying lcsh data are examined. by chris freeland, martin kalfatovic, jay paige, and marc crozier introduction the biodiversity heritage library (bhl)1 is a consortium of ten international natural history museum libraries, botanical garden libraries, and associated technology partners, formed with a mission to digitize the corpus of historic biodiversity literature. the partner libraries include american museum of natural history (new york, ny); the field museum (chicago, il); harvard university botany libraries (cambridge, ma); harvard university, ernst mayr library of the museum of comparative zoology (cambridge, ma); marine biological laboratory / woods hole oceanographic institution (woods hole, ma); missouri botanical garden (st. louis, mo); natural history museum (london, uk); the new york botanical garden (new york, ny); royal botanic gardens, kew (richmond, uk); and the smithsonian institution libraries (washington, dc). the university library of the university of illinois at urbana-champaign (urbana, il), a lead member of the open content alliance, is also supporting the work of bhl as a contributing member. the collections held by bhl partners total upwards of two million volumes and contain core scientific literature dating from the 15th century. these collections have been curated for more than 200 years to support the research activities of scientists and students describing the living world. because of shared interest with smaller scientific societies, bhl collections offer many unique titles unavailable even in larger university collections. from a scholarly perspective, these collections are of exceptional value because the domain of systematic biology depends upon references to original publication of species descriptions contained in historic literature.2 however, this wealth of knowledge is readily available to only a small subset of the entire community who need access to the materials, as these collections are primarily held in large libraries in north america and europe. scientists must spend considerable time and funds to travel to these libraries for their research. further, while in non-digital form, this corpus of literature is unavailable to data mining endeavors within taxonomic study, biodiversity conservation, protected area management, disease control, and associated educational outreach. in addition to works in the public domain, bhl is committed to working with partners in the scholarly publishing community to obtain permission to digitize materials of scientific interest and make it available in an open access environment. this “opt in” model for digitization will allow for the broadest representation of taxonomic literature while obtaining buy-in from the scholarly publishing world. digitization and metadata creation in 2005, concurrent with early bhl planning sessions, the missouri botanical garden (mobot) received funding from the w.m. keck foundation to model a robust software platform to connect biodiversity databases with digital library content, including interfaces, both user-centered and programmatic, that addressed the specific needs of the taxonomic research community.3 the platform, named botanicus, was demonstrated using materials scanned in-house at the missouri botanical garden and is online at www.botanicus.org. bhl project planning began in earnest during 2006. the internet archive (ia)4 was invited to join bhl as a technical partner because of its efficient, high volume scribe scanning workstations and commitment to open access distribution of public domain materials through its lead participation in the open content alliance.5 in june of 2007 bhl received funding from the john d. and catherine t. macarthur foundation and the alfred p. sloan foundation to scan upwards of 60 million pages of scientific literature held across bhl partner collections. book digitization began in late summer of 2007 at scanning centers in london, boston, new york, washington dc, and at the university of illinois urbana-champaign. botanicus was adopted as the model for bhl, with slight modifications, and deployed at www.biodiversitylibrary.org. at the time of scanning, the marc record for the monograph or serial being scanned is retrieved from the contributing library’s ils using a yaz z39.50 fetch6 and transformed to marcxml. the marc and marcxml records are deposited alongside the raw camera images in a directory for each scanned book on ia clusters. additional structural and administrative metadata, created at the time of scanning, is also deposited in the book’s directory. automated processes run against the raw files to produce a series of derivatives, including jpeg2000 images, color pdf, black and white pdf, and ocr text.7 to facilitate the advanced features and functionality modeled for the taxonomic community through development of the bhl portal, the files for a scanned monograph or serial volume are ingested from ia clusters and transformed into the bhl data model. in this way, books and bibliographic metadata from multiple libraries are delivered through a common interface with advanced functionality specifically developed for the communities served by bhl partner libraries. metadata reuse bhl partners were particularly keen to reuse the vast amounts of specialized descriptive metadata created by professional catalogers contained in the bibliographic records. partners were also interested in evaluating new visualization techniques for displaying these data in a modern, service-oriented interface. library of congress subject headings (lcsh) were selected as the first dataset for experimentation as they offer a controlled vocabulary that has been routinely applied within bhl libraries, and which offer an important access point for users looking for materials on a given topic. because the assignment of lcsh is regimented, these data provided an interesting counterpoint to the popular application of user-generated tags, also known as a folksonomy.8 collections of user-generated tags are often displayed in tag clouds, which give visual weight and emphasis to tags representing more content and deemphasize tags representing less content, as demonstrated in figure 1 from the photo sharing portal flickr. figure 1: “all time most popular tags” from flickr, 19 jan 2008. http://www.flickr.com/photos/tags/. [view full-size image] another example is the internet archive, which displays a tag cloud for its digitized content, including its children’s book collection.9 to build a similar interface from lcsh, the marcxml for each scanned title is parsed into corresponding fields in the bhl database, currently deployed in sql server 2005 on servers at the missouri botanical garden. working under the guidance of librarians at bhl partner libraries, all subject headings (marc 6xx fields)10 available in a given record are selected to form the tag cloud. this activity commits each subdivision of a lcsh string to its own database record associated with the title from which it was assigned. these records are then grouped according to their number of occurrences and displayed using a tag cloud interface. for a digital library the size of bhl, currently expected to include more than 60 million scanned pages by 2012, techniques are required to selectively view subject tags, as viewing the entire cloud would require several seconds to draw and would include hundreds of results with only a single associated title. figure 2 illustrates the approach taken with the bhl portal. by default, users are presented with the top 100 subjects and given links to other quantities, as well as a link to view all tags. this approach allows the tag cloud and surrounding page to draw quickly, while still providing interfaces to those who want to view more details. figure 2: tag cloud from bhl, 25 feb 2008. http://www.biodiversitylibrary.org/browse/subject. [view full-size image] geocoding lcsh to then further extend use of lcsh, each term recorded in subfield $z (geographic subdivision) is geocoded using classes and methods within the google maps application programming interface (api).11 geocoding is the process by which structured geospatial information is determined for map-based data, such as coordinates for a street addresses or, in this example, coordinates for a geographic place name.12 the resulting interface is displayed within the bhl portal at http://www.biodiversitylibrary.org/browse/map and in figure 3. figure 3: google map with geocoded lcsh from the biodiversity heritage library. to begin using the google maps api for geocoding, organizations must first sign up for an api key that is submitted along with each api call.13 the google maps api geocoder is accessible via several protocols and can return results in a variety of formats, including csv, xml, google kml, and json.14 in current practice, bhl submits strings for geocoding via http from server code and receives responses in csv, as these methods are the easiest to form and least bandwidth-intensive. here is a sample http geocode request for the tag “brazil” (with parameters on separate lines for clarity): http://maps.google.com/maps/geo?output=csv &key=[developer api key] &q=brazil this request returns the following result: 200,1,-14.235004,-51.925280 “200” is the http status code (which indicates success) “1” indicates accuracy (in this case, “country” level accuracy) “-14.235004” is the latitude “-51.925280” is the longitude as a comparison, here is the same result when formatted in kml, google’s xml-based language for geospatial data: <?xml version="1.0" encoding="utf-8"?> <kml xmlns="http://earth.google.com/kml/2.0"> <response> <name>brazil</name> <status> <code>200</code> <request>geocode</request> </status> <placemark id="p1"> <address>brazil</address> <addressdetails accuracy="1" xmlns="urn:oasis:names:tc:ciq:xsdschema:xal:2.0"> <country> <countrynamecode>br</countrynamecode> </country> </addressdetails> <point> <coordinates>-51.925280,-14.235004,0</coordinates> </point> </placemark> </response> </kml> regardless of response format, the values returned for latitude and longitude are stored in the bhl database. if geocoding was unsuccessful, those tags are marked as a failure in the database with a timestamp to re-attempt the geocode after 30 days. this way each failed tag is resubmitted at regular intervals, allowing for the possibility that google may be able to find it in the future should their list of place names or their geocoding algorithms change. when a user browses to the page at http://www.biodiversitylibrary.org/browse/map, server-side code first looks to see if there are any new tags that need to be geocoded, since new tags are added to bhl as new materials are ingested. after submitting new tags for geocoding, the tags that have previously failed are submitted again if it is 30 days after their last attempt. this additional lookup is negligible and does not affect response times or user experience. finally, all tags with latitude and longitude are sent to the browser and are drawn on the map. in this way values can be stored and cached for increased performance, while still allowing updates to occur for new tags and tags that have previously failed. another approach would be to schedule this lookup as a cron job or other scheduled task on the server, but bhl developers favored including the lookup directly in the web application itself in order to minimize the number of dependent applications. bhl is not alone in visualizing lcsh through tag clouds or google maps. “the maritime history of the great lakes” digital collection uses similar techniques to display lcsh in a tag cloud,15 as well as geocoding lcsh and displaying a map for individual records.16 this presentation and functionality has been well-received by our user community, who have described the interface as “a system to quickly and easily find the information you want with minimal distractions.”17 complications and limitations although successful, this method of geocoding and presenting data gleaned from lcsh is not without problems. some of the issues are in the data themselves and some have to do with limitations of the google maps api and the methods by which it geocodes and displays placemarks. the main issue concerning lcsh values is that they are only as complete as each bhl partner’s cataloging practices. bhl partners are not explicitly applying or improving lcsh before scanning, so if there are no values present in the original marc record there are no data to geocode, leading to visual gaps in the display. for example, a bhl partner might have digitized a volume about costa rica, but if that assertion has not been recorded in lcsh there will not be a placemark on costa rica in the map display. other limitations in the display relate to how google maps geocodes values and how it then displays those placemarks. by definition a placemark is a single point on a map, which works well with traditional uses of google maps, such as displaying points for street addresses. however, geocoding returns a somewhat different result for less granular place names like “missouri.” the point associated with “missouri” is the centroid, or center point, of the polygon defined by the boundaries of the state of missouri. while this is not a major barrier to viewing results, it does pose some ambiguity when there are placemarks for both a state and a city within the state, such as “missouri” and “st. louis”, as demonstrated in figure 4. figure 4: google map showing placemarks for “missouri” and “st. louis”. even as place names describe larger geographic areas, such as continents, a single placemark is plotted on the centroid. figure 5 shows how the place name “south america” is plotted in southern bolivia, as that point represents the center of the geographic boundaries of the continent. as demonstrated by these two examples, a single placemark for a regional place name might produce ambiguous results when plotted alongside more specific place names. figure 5: placemark for “south america” as indicated by pointer. a further complication with the single placemark paradigm relates to the inability to visually represent areas of density within google maps. rather than viewing single placemarks at each country’s centroid, a more compelling display would be to view a map with shading to represent countries associated with more digitized content, similar in display to a population density map. it is possible to cluster placemarks such that multiple points are represented by a single placemark when zoomed out on the map,18 allowing developers to streamline maps with many points in close proximity to one another. however, clustering still doesn’t allow for visual ranking or weighting of results. unfortunately the only methods within google maps to define polygons (such as state or continental boundaries) and color-code for visual weight are cumbersome to use and difficult to display without considerable delays in performance. to accurately draw the polygon for south america each point along the continent’s coastline would have to be defined, requiring thousands of points and therefore significant delays in delivering the map to users (upwards of 20 seconds or longer each time the map is drawn). an illustration of this point is available by viewing voting results from the 2008 new hampshire primaries.19 future work in addition to investigating techniques for displaying polygons and relative density, bhl developers plan to evaluate other mapping services. alternatives to google maps, such as yahoo! maps web services and microsoft virtual earth, have grown in functionality and community use. these services, as well as other efforts underway within the opengis community, bear further evaluation. work is also needed to perform a more thorough study of user experience and user behavior to determine efficacy of browsing a digital library through a map-based interface. another area of further investigation is to geocode and map marc values outside of lcsh, such as the marc code list for geographic areas, commonly represented within marc field 043.20 the same complications with incomplete data exist within this field, similar to the problem previously stated with lcsh. conclusion reusing metadata generated through years of cataloging practice is a natural and pragmatic way of leveraging an institution’s investment in describing its resources. unfortunately, many digitization projects disassociate metadata about the original object from the metadata about its digital surrogate when publishing that surrogate online. through reuse of lcsh in forming tag clouds and interactive maps, bhl has demonstrated how easily that original cataloging work can be brought into the web space and partnered with emerging technology to provide advanced user interfaces and functionality. while the methods described in this paper result in access points that are not comprehensive, they do provide new methods of resource discovery and new opportunities for evaluating visualization techniques. it is of utmost importance that libraries continually evaluate how best to reuse legacy data in light of emerging technologies and to determine how best to remix those data into new and innovative interfaces. notes 1. http://www.biodiversitylibrary.org 2. krell, frank-thorsten. 2000. “impact factors aren’t relevant to taxonomy.” nature. 405: 507-508. doi:10.1038/35014664. coins 3. http://www.botanicus.org/about.aspx 4. http://www.archive.org 5. http://www.opencontentalliance.org/ 6. http://www.indexdata.com/yaz/ 7. http://www.archive.org/download/freshwaterfishes00eige/ 8. http://en.wikipedia.org/wiki/folksonomy 9. http://www.archive.org/browse.php?field=/metadata/subject&collection=iacl&view=cloud 10. http://www.loc.gov/marc/bibliographic/ecbdsubj.html 11. http://code.google.com/apis/maps/documentation/reference.html 12. http://en.wikipedia.org/wiki/geocoding 13. http://code.google.com/apis/maps/signup.html 14. http://code.google.com/apis/maps/documentation/services.html#geocoding_direct 15. http://www.hhpl.on.ca/greatlakes/glimages/browsebysubject.asp 16. http://www.hhpl.on.ca/greatlakes/glimages/details.asp?id=9 17. http://moultriecreek.us/family/2007/11/03/is-this-any-way-to-run-an-archive/trackback/ 18. http://googlemapsapi.110mb.com/clustermarker/index.htm 19. http://maps.google.com/maps/mpl?moduleurl= http:%2f%2fwww.google.com%2fmapfiles%2fmapplets%2fnhprimary%2fnhprimary.xml &ie=utf8&om=0&ll=43.975,-71.628&spn=1.585195,3.691406&z=8 20. http://www.loc.gov/marc/geoareas/gacs_code.pdf about the authors chris freeland is the technical director of the biodiversity heritage library and manager of the bioinformatics department at the missouri botanical garden. martin kalfatovic is head of the new media office and preservation services at smithsonian institution libraries. jay paige is a senior programmer in the missouri botanical garden’s bioinformatics department. marc crozier is a programming consultant in the missouri botanical garden’s bioinformatics department. subscribe to comments: for this article | for all articles 4 responses to "geocoding lcsh in the biodiversity heritage library" please leave a response below: chris freeland, 2008-03-26 if you are interested in learning more about the process for ingesting and parsing content from internet archive, read the bhl blog entry on the topic. jonathan rochkind, 2008-05-20 i just discovered this open access source of place name to lat/long mapping, essentially an oa gazeteer. http://www.geonames.org/ super nice! yes, it has a web service, as well as downloadable data. seems worth considering for any projects that need to translate from existing controlled place vocabulary without lat/long (like lcsh!) to lat/long. thoughts on “geocoding in the lcsh biodiversity library” « digital mappings: readings & reflections, 2012-05-16 […] geocoding in the lcsh biodiversity library, the authors detail how they have taken marc record locations and ingested them through the google […] regeniaec, 2012-05-22 a fantastic view of point for google maps. never miss a thing when you need it most of the time to find a location that your not familiar with. leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – lessons in public touchscreen development mission editorial committee process and structure code4lib issue 15, 2011-10-31 lessons in public touchscreen development in october 2010, the ncsu libraries debuted its first public touchscreen information kiosk, designed to provide on-demand access to useful and commonly consulted real-time displays of library information. this article presents a description of the hardware and software development process, as well as the rationale behind a variety of design and implementation decisions. this article also provides an analysis of usage of the touchscreen since its debut, including a numerical analysis of most popular content areas, and a heatmap-based analysis of user interaction patterns with the kiosk's interface components. by andreas k. orphanides introduction a common challenge in libraries is to present service information of immediate interest and need to users in an unmediated way. at the ncsu libraries, we have in the past attempted to fulfill this need through the use of e-boards with rotating displays of such information (morgan and mori 2008). while this e-board initiative has been successful, a notable shortcoming is that users cannot call up the desired information on demand. in october 2010, the ncsu libraries debuted its first public touchscreen information kiosk, in order to provide on-demand access to these commonly consulted real-time displays of library information. the kiosk is available during all hours of library operation (a total of 146 hours per week during the school year [1]), and remains powered on at all times. in this paper i report our process for designing and implementing the web-based touch interface used to drive this kiosk, provide an analysis of the development process and the kiosk’s usage, and draw conclusions to inform future development of public touch-based interfaces. i first recount the process for creating and adapting content, selecting hardware and software, designing the interface, and deploying the kiosk, and provide a rationale for our choices at each stage of development. i also identify challenges we encountered during the development process and shortcomings in our implementation that have revealed themselves since project inception. i then explore how users have interacted with the kiosk since its deployment by analyzing the most frequently consulted content areas and by rendering heatmaps to examine user touch patterns. i conclude with lessons learned in the process of creating a public touch kiosk, including recommendations for developers, guidelines for interface design, and avenues for future research. motivations the inspiration for development of a touchscreen kiosk in the d. h. hill library resulted from the confluence of several ongoing endeavors at the ncsu libraries: a perceived need for improved user access to unmediated wayfinding information. the ncsu libraries has long recognized a need for a publicly accessible, interactive wayfinding solution in the d. h. hill library. an initiative to showcase and test out novel user interfaces, including touch-enabled devices, in the d. h. hill library, in preparation for the use of such interfaces at the james b. hunt jr. library, currently under construction. a desire to provide single-point access to a growing collection of dynamic, time-dependent, user-facing information tools describing library and campus services. as part of the above-mentioned novel interfaces initiative, the research and information services department at ncsu libraries received a 42” touchscreen e-board display for use as an interactive kiosk in the learning commons of the d. h. hill library. the task of selecting and developing content, identifying and configuring additional hardware, and designing and implementing an interface, fell to me, along with my colleagues keith morgan, principal librarian for digital media [2], and jason walsh, technology support specialist [3]. a primary goal for this project was rapid development and deployment — the project began in early summer of 2010 with the goal of rolling out the final product early in fall semester of the same year. this goal drove many of the major implementation decisions: in order to minimize delays due to hardware acquisition and configuration, we elected to use the macintosh mini as the system cpu. mac minis are our preferred hardware for other e-board applications; as such, the mac mini platform is familiar, well supported, and readily available. to limit potential difficulties associated with learning new technologies, to maximize opportunities to reuse existing web content, and to ensure flexibility in design, we elected to develop the interface in html and display it through a web browser. to minimize costs, we chose to use common, freely available web browsing software and plugins to configure the system to be suitable as a public kiosk. content selection at the time of project inception, the libraries already had a well-established e-board initiative (morgan and mori, 2008), with numerous flat-panel e-boards in the 40” range distributed throughout the library and displaying libraryand campus-oriented information on availability of services, advertisements for events, and staff contact information. an early goal of this project was to identify the most “useful” information displayed on the e-boards and consolidate that information into the kiosk system so that users could view the information on-demand. we recognized this need from casual observations at the reference desk: at the d. h. hill library, several e-boards are mounted prominently near the reference desk, displaying various information on a rotating schedule. a common interaction pattern that librarians (including myself) observed during reference desk shift was as follows: user approaches reference desk, but keeps his/her eyes fixed on a nearby e-board. user halts and continues to watch e-board. after a few moments, user leaves reference desk area. observations from the user side of the reference desk confirmed our suspicion that users exhibiting this behavior were waiting for a particular piece of information to appear in the rotation. although all our observations were anecdotal in nature (i.e., no statistics on these observations were maintained), it appeared that the most common displays consulted in this manner were the campus bus system's real-time map of bus locations (presumably to check for the proximity of a particular bus to the library bus stop) and the computer availability map of the learning commons (presumably to identify the location of an available computer). the common thread in these two activities was that users were seeking to fulfill an immediate need, related to a resource with time-dependent variations in availability. this suggested that a public kiosk would fulfill user needs best if it exposed such time-dependent information to users on demand. such a kiosk would also spare users having to wait for their desired information to appear in the rotation. with this in mind, we selected the following services to include on the kiosk: computer availability [4] (see sierra 2009) real-time bus location map for the campus bus service [5], centered on the library (referred to below as the "campus bus map" for brevity) group study room availability groupfinder service [6] (see ryan and boyer 2011) library hours [1] in addition, as a response to the perceived need for interactive wayfinding as mentioned above, we elected to include a wayfinding feature to direct users to common destinations in the library. hardware and software configuration one of our implementation goals was to build a touch interface that appeared to be completely dedicated and self-contained: we did not want it to be apparent to the user that the interface had been created with and was being driven by commodity components. to this end, we wanted to minimize user awareness of operating system components, software and hardware infrastructure, etc. this goal, along with the three implementation decisions stemming from our short development timeline (as mentioned in the “motivations” section above), drove many of our system configuration decisions. the touchscreen hardware, an elo touchsystems 4220l 42” single-touch display [7], was provided to the research and information services department with the express purpose of creating an interactive kiosk. the display behaves as a normal large-format flat-panel display, with the addition of a touch-responsive overlay that acts as a single-touch input device. the cpu that drives the display is a macintosh mini (mid-2007 edition) with a 1.83ghz intel core 2 duo processor and 1 gigabyte of ram [8]. this cpu was repurposed from another e-board display setup elsewhere in the library. the touchscreen’s touch-sensitive overlay connects to the cpu via a usb connection, and with the appropriate drivers the cpu receives input from the touchscreen as though it were a mouse. the touchscreen uses the updd touch driver from touch-base ltd. [9], which provides calibration software as well as several options to configure how touch interactions are interpreted. of the options available, we faced a choice between “click and drag,” which allows the user to perform a mouse-drag action by touching the screen, dragging while still in contact with the screen, and releasing the touch; and “point and click” which only allows click actions on the screen — any drag action performed by the user is ignored. the selection of “point and click” to disallow drag actions has many benefits: click events on interactive elements are interpreted more clearly and reliably by the application, since users may accidentally drag their finger slightly while touching the screen. in “click and drag,” the slight drag action will register as a drag event in the application and may not respond in the expected manner. users are prevented from accidentally highlighting on-screen text in the web browser. accidental highlighting may be disruptive or confusing to the user; on a more aesthetic level, the ability of the user to highlight text reveals the fact that a web browser is being used to render the display, a fact that we would prefer to disguise. likewise, disabling "click and drag" prevents the user from accidentally performing a drag action on displayed images, which in modern browsers causes a “ghost” duplicate of the image to appear to support drag interactions with the remote image file. this ghosting action again would have the potential to be disruptive and confusing to the user. in addition, the driver software can be configured to permit or disallow double-clicks. the primary advantage of disallowing double clicks is in a failure case: should the browser driving the kiosk display crash, or should the machine be forced to reboot, the computer’s desktop (in this case, a typical macos x desktop) would become visible and available to users of the touchscreen. disallowing double clicks prevents users from opening applications and filesystem locations in cases where the system desktop becomes available. despite the benefits of “point and click” and disabling double-clicks, we opted to place the driver software in “click and drag” mode and to permit double-clicking. our reasoning stemmed from the inclusion of the campus bus map, which is driven by google maps, as one of our content areas. in order to provide as interactive an experience as possible, we wanted to allow users to pan and zoom in the bus map application, which requires dragging to support panning and double-clicking to support zooming (no zoom slider is provided in the campus bus map application by default, and we were unable to discover a method to enable it programmatically). we felt that this use case was sufficiently strong to enable both “click and drag” and double-clicking, at least provisionally. in order to create a touch interface that feels dedicated and self-contained, we felt it was important not to display a cursor on the screen at any time, lest the illusion of a system independent of standard computer software be broken. we were surprised to find that the updd console software did not offer an option to hide the cursor, as we felt as though a “kiosk-like” implementation was a common use case for touchscreen drivers. we discovered a program for macos x called cursorceror [10] that would hide the cursor on demand; installing this software allowed us to keep the cursor continuously hidden once the system was booted and the software launched. due to its standards compliance, familiarity, and breadth of available plugins, we chose to display the web page serving as the kiosk interface in firefox [11], although we had some reservations about its stability over uptimes measuring in the days and weeks. since the screen would be in a location with easy staff monitoring and access, we felt the potential stability risks were outweighed by the utility of the firefox plugin system and our facility with developing web sites that rendered well in that browser. to make firefox suitable for kiosk use, we needed to hide the browser controls themselves; although firefox’s full-screen mode (accessible with the f11 key) did a passable job of hiding its controls, especially considering that users would not have access to a mouse or keyboard, we had concerns that the users might still be able to reveal the browser controls somehow with the touch driver in “click and drag” mode. to keep the browser more thoroughly locked down, we installed the r-kiosk plugin [12], which completely hides all browser controls while it is enabled, causing the browser to display only the viewpane. although our design for the kiosk interface did not include links to external websites, we found that the campus bus map application’s google maps window includes a link to google maps’ terms of use, which opens in a new browser tab. this presented two problems: first, this link provides access to web content outside the kiosk interface, an unacceptable condition; second, although r-kiosk hides the browser's tab bar, a new tab is opened when the link is touched. to prevent a user from accessing other web content, we installed the procon latte web filtering plugin [13] for firefox. this plugin allows whitelisting or blacklisting specific websites, and also permits redirecting blocked traffic to a specified web address. to partially solve the “terms of use” link problem, we whitelisted only the kiosk interface’s url. all other sites were blocked and set to redirect to the kiosk interface. although we were not able to find a solution to prevent a new tab from opening if a user happened to click on the “terms of use” link, the procon latte configuration simply redirected the new tab to the kiosk interface’s url; the existence of multiple tabs remained hidden to the user. this solution presented two problems, which we considered acceptably minor: first, the action of opening a new tab presented the user with a blank gray screen for a moment as the website reloaded; second, there was the potential for unlimited multiple tabs to be open in the background if this event occurred repeatedly, consuming computer resources and potentially causing browser instability. the former problem we simply accepted; the second problem could be solved with periodic monitoring of the kiosk’s performance and by restarting the browser if necessary. although it was not a problem at the time the project was initiated, it should be noted that both r-kiosk and proconlatte are supported only through firefox 3.6.x [14]. a search for acceptable alternatives to these plugins for the current version of firefox (at the time of this writing, version 6.x) was not successful. it should also be noted that opera software’s opera browser [15] is nominally capable of white/blacklisting websites, suppressing the opening of new tabs and windows, and running in a locked-down kiosk mode without the need for additional plugins. we discovered this only after successful configuration and testing of the firefox solution mentioned above, and near the deadline for rolling out the touchscreen; because of the timing we did not have an opportunity to test an opera-only solution to a locked-down browser. interface design our goal for an interface was to develop something simple and self-evident, with the minimum explicit instruction necessary to effectively guide the user. the rough outlines of the interface were hewn in prototyping: early development used yui grids templates [16] to split the screen into a ⅓ width column on the left for main navigational controls and a ⅔ width column on the right for content (see figure 1), a layout that was preserved through production. the interface concept was to maintain a static set of controls to call the available content areas on the left, updating the right pane with the corresponding content when the user selected a control. figure 1. basic kiosk layout and screen division an early design hurdle was determining how to indicate that the kiosk was interactive and touch-enabled, a challenge exacerbated by the fact that the touch kiosk would be replacing a non-interactive e-board of the same general size in roughly the same location. because of this, we wanted to be explicit; at the same time, we wanted to include the minimum amount of instructional clutter necessary. the main navigational controls were designed in a “button” style to reinforce to the user that they are interactive. we chose to include a simple legend reading “touch a button below” immediately above the main controls, trusting that the labels on the buttons could be made to be sufficiently descriptive for the user to predict (or quickly learn) how the screen would respond to a touch. we hoped that the “touch” language and the button-style controls, coupled with the lack of other input devices at the kiosk, would be enough to suggest a touch-enabled interface. we designed the buttons carefully to make it as clear as possible that they were control elements. the lack of a cursor deprives the user of cursor-contextual affordances that indicate controls (e.g., the cursor turning into a “pointer-finger” icon or the background color of a control element changing on mouseover), so we felt it was important to strengthen the visual affordances provided by the controls by giving them a familiar, button-like appearance. at the same time, we wanted to disguise the interface’s web-page nature. we found the options for html button styling to be too limiting, so we created image files for simple two-state “lozenge”-style buttons with a heavy border and a subtle 3d effect (see figure 2). the 3d styling and prominent border recall both real-world and html buttons — inviting the user’s interaction — while having a distinctive appearance. the buttons were made oversize, 275 pixels wide by 50 tall (measuring approximately 7 ¼ x 1 ¼ inches as rendered on-screen) to increase both their prominence and the readability of their labels. the same button styling motif was repeated on other controls to cue the user to interactive elements throughout the interface. figure 1. two-state button graphic. the upper part of the graphic is the normal button state; the lower part the "mousedown" state integrating content into the interface for every content area except wayfinding, there was already a web-based version of the content available, originally created for display either on a website or on an e-board. for those content areas owned by the library, i.e., computer availability, library hours, group study room availability and groupfinder, we were able to modify existing displays or create new displays based on the same data with only minor modifications to styling or rendered size. to support smartphone users, for each content area where a mobile web address existed to provide access to the same content (groupfinder, campus bus map, computer availability, group study room availability), we included a qr code pointing to the corresponding mobile url in the lower right of the content area. library hours the library hours display used in the kiosk interface was created especially for the kiosk, using the data service that drives hours displays on the libraries' website. the data service provides current and future operating hours for a large number of library services and all campus libraries; we thought carefully about the right extent of information to present in order to balance utility of display with ease of browsing for a particular entry. we settled on displaying hours for a three-day time span — the current day and the next two days — to allow a user to see building and service hours through the weekend if they were viewing the kiosk on a friday. we chose to provide both building and service hours for the d. h. hill library, where the kiosk is located, as well as the building hours for the branch libraries. in addition, in the kiosk interface's header bar, which is visible to the user at all times, we provided the building's hours for the current day. campus bus map integrating the campus bus map presented multiple challenges. the campus bus map is provided by the university’s transportation department and is operated by a third-party developer, transloc [17]. a display generator [18] is provided by transloc to create campus bus maps for web applications, which offers options for map area to display, alert sidebars, branding, etc. the display generated from this tool is a full-page flash application running at a transloc url with query parameters specifying the display options; the only method we were able to identify for integrating the map into another website was to use an html iframe tag. the combination of iframe and flash raised concerns about browser stability and responsiveness, but inclusion of the campus bus map was a high priority so we chose to set aside these concerns. as mentioned above, we wanted to allow users to pan and zoom the bus map to provide a maximally interactive experience and increase the utility of this feature. since it would potentially be difficult or impossible for the user to return the map to its original view after extensive panning and zooming (recall that no zoom slider is present on the campus bus map application (see figure 3), so double-click zooming cannot be undone), we included a "map reset" button in the campus bus map content area, which reloaded the iframe containing the map, returning the map to its original viewport. figure 3. campus bus map application as configured for kiosk. note the absence of a zoom slider wayfinding the wayfinding content area required generating the majority of its content from scratch. we chose a simple text-and-picture interface to provide a menu of destinations, with each text/picture block serving as a control to bring up a wayfinding map guiding the user from the touchscreen location to the destination. sixteen common destinations on the first and second floor of the d. h. hill library were selected based on our anecdotal experience in providing directional service at the reference desk. we chose sixteen as the number of destinations based on the amount of information that comfortably fit in the space available: a 4×4 matrix of images with text provided adequate space for the images to be discernible while allowing sufficient space between each image (see figure 4). we were unable to develop a satisfactory design for these controls that mirrored the button styling described above — failing to achieve the goal of keeping all control elements consistently styled. to partially mitigate this shortcoming, the wayfinding screen states explicitly that the user should touch his or her desired destination. figure 4. wayfinding display, destination selection when the user selects a destination, a graphic of the main level of the d. h. hill library is displayed in a "lightbox"-style overlay rendered by the jquery tools plugin [19], which overlays the majority of the interface; a star indicates the user’s current location with a red path showing the user to his or her destination, or to the stairs/elevator for destinations on the second floor (see figure 5). a button is provided to toggle the view to the second floor of the library; if the user's destination is on the second floor, a continuation of the route is shown on the second floor map. another button allows the user to close the lightbox to return to the main interface. figure 5. wayfinding display, map overlay the map graphic and path are implemented as separate png files of the same dimensions in pixels, positioned absolutely, with the path atop the map graphic. the background of the path image (i.e., all parts of the image that are not path) is transparent, allowing the map to show despite the fact that it is overlaid with another image. this arrangement allows us to keep the map images logically separate from the path images; thus, if the map must be updated or if a path is changed, only one file needs to be altered or replaced. idle behavior in addition to selecting content for the kiosk and developing an interaction model for that content, we determined that some sort of idle mode would be necessary to call attention to the kiosk when it was not currently in use. our first idea was to display a rotation of slides, similar to other (non-interactive) e-boards in the library, advertising the kiosk’s functionality and possibly other library services. we decided against this approach, feeling that simply stating that the screen was touch-responsive and listing the available content areas would not be compelling enough to elicit user interaction. instead, we decided to have the kiosk cycle through displaying the available content areas during idle periods, showing the same content that users would see if they manually selected a content area. during idle times, the kiosk sequentially displays each content area for 10 seconds, cycling through them in the same order as listed in the main navigational controls, a state that we refer to as “attract mode.” a user exits attract mode by touching anywhere on the screen; if the user touches a main navigation button, the display switches to the corresponding content; otherwise, the currently displayed content is maintained. once attract mode is exited, the kiosk will respond normally to user input (“interactive mode”) until it receives no input for a period of 180 seconds, at which time it returns to attract mode. a visual cue is added to the appropriate main navigation button when the screen is in interactive mode; this cue is not present in attract mode (see figures 6 and 7). this allows the user to know at a glance whether the screen is displaying its idle cycle or is awaiting user input in interactive mode. figure 6. group study room availability display in attract mode figure 7. group study room availability display in interactive mode. note the visual cues on the content selector button to indicate the selected content area and the fact that the display is in interactive mode the interactive campus bus map requires special handling in attract mode due to interactions between the iframe and flash under firefox: there is a slight lag between displaying the iframe and initiation of the flash application every time the iframe is displayed. this lag proved unacceptable during the content area's brief window of visibility in attract mode. our solution was to display a static screen capture of the map interface, with a legend reading “touch to activate map” (see figure 8), during attract mode. when a user exits attract mode to display the campus bus map, the static image is hidden and the active map application is shown. there is still a slight delay while the map initiates, but this delay is unavoidable and acceptably short in this context. figure 8. campus bus map display in attract mode implementing interactivity and logging in considering how to coordinate and control the different content areas we chose to include, we wished to ensure a responsive environment that would betray little to no lag between the user’s request and the presentation of content. for this reason, we decided to have all content rendered in a single html page, performing all navigation and content switching through javascript and performing updates to time-dependent data through background ajax updates. the javascript library of choice at the ncsu libraries is jquery [20], and most display manipulation and ajax requests were implemented using the appropriate jquery methods. although the interface is rendered as a single html page, some of the content is maintained in external files in several locations, as content such as groupfinder and room availability are more easily configured and maintained in separate documents (in some cases, in separate directories). these external files are loaded into the main page via ajax either at page load or when the content is first displayed [21]. javascript/jquery is used to manage all content display and manipulation, entering and exiting attract mode, and logging. the primary javascript components may be grouped as follows: timer components, which manage cycling content display during attract mode, ajax content updates, and countdown to re-enter attract mode. attract mode is initiated on page load and terminates on any user input; to re-enter attract mode, a javascript page refresh is forced after 180 seconds with no user input. content updates for library hours, computer availability, groupfinder and group study room availability occur every ten seconds. button evaluation routines, which trigger display of the appropriate content area and associated visual elements. logger, which sends an ajax request describing an event (typically a user interaction or an idle reset) to an external logging script. touch-responsive components, which perform tasks such as resetting the idle timer, triggering the button evaluator to display content, and triggering the logger, in response to user input. the logger records every touch interaction with the kiosk as a separate event, as well as idle resets and page loads. events that cannot be captured by local javascript event handles, in particular touches to the iframe element containing the campus bus map, are not recorded. the information logged includes the following: ip address of client computer. this allows us to filter out any interactions that result from remote tests of the kiosk page. x and y coordinates of user touch, relative to top-left of the display. if the logged event is a non-touch event, such as an idle reset, the coordinates are recorded as (-1,-1). actions such as “exited attract mode by touching control element,” “exited attract mode by touching non-control element,” “touched a control element in interactive mode,” “idle reset,” etc. “origin” and “destination” screens: if a user moves between two content areas by touching one of the main controls, both the origin and the destination are recorded. control element touched. each control element is provided with a unique id, which is passed to the logger if that element is touched. design, development and deployment challenges in the time since this project was initiated, a number of problems and challenges have been revealed, primarily stemming from the short development timeline for the project and the ad hoc implementation and integration of content areas. design a number of design compromises and arbitrary decisions which arose during protoyping were never properly reconsidered later in the project. for instance, the ⅓ – ⅔ screen split between controls and content was not re-evaluated until a point had been reached where an extensive display redesign would have been impractical. we had originally selected the ⅓ – ⅔ split for two reasons: first, it was one of the available column layouts in the yui grids framework used for prototyping; second, such a split provided the content area with an aspect ratio close to a traditional (non-widescreen) presentation. however, once the content was selected, it became clear that the content was well-suited to a widescreen layout, due in particular to the layout of the library floorplan, the east-west layout of ncsu's central campus and bus routes on the campus bus map, and the dimensions of the table required for group study room availability. in a full redesign of the interface, a better location for the main controls might be along the bottom of the screen to provide a widescreen presentation to the content area, but this is infeasible without extensive modifications to the current implementation. even given the current general presentation with a vertical button layout on the left, a ¼ – ¾ screen split with more efficient use of space in the main navigation area would provide more room for the content display. development the ad hoc approach to content selection led to coding decisions that have resulted in more difficult backend management. existing scripts from other projects in other locations on the server were modified only to the extent needed to integrate into the kiosk display. because these projects were created by other individuals, we were not familiar with the architectures used, and the limited timeline did not afford developing a deep understanding of the external projects’ architecture. thus our adaptations consisted largely of duplicating and modifying existing scripts, keeping them in the same location on the server as the originals, so that we would not have to interpret and alter relative file paths in the duplicated scripts. the result is that file management for the kiosk is spread across no fewer than four directories on the server. in addition, since this project integrates components from so many other library projects and extensively reuses code from other files that are maintained separately, updates to our own code are required any time one of the referenced projects is updated, which increases the frequency of required maintenance. because this was our first attempt to implement a touch-based interface, and because our understanding of interface requirements and interaction patterns grew only as we built the interface and integrated content, the javascript driving the interactive components of the interface developed in a very organic fashion. this led to redundancies and inefficiencies in coding, as multiple event handlers were attached to certain elements to capture different aspects of interaction, multiple timers were set up to manage different timed events, etc. during javascript development, many unexpected side effects were discovered as different features were implemented, due to the lack of a centralized approach to handling user interactions. a frequent problem was mismanagement of onclick events: for instance, to capture a user’s touch on (say) button1, we might attach the following event handler to button1 (this example uses jquery syntax): $("#button1").click(function() { executemethoda(); executemethodb(); executemethodc(); }); later, we might develop a timer which, among other things, programmatically triggers a click event on button1: setinterval(function() { … // some code $("#button1").click(); … // more code }, timerinterval); if executemethodc()is meant only to be executed on an actual user touch on button1, the timer code triggers a spurious execution of executemethodc(). similarly, we often found we had written click event handlers connected to both an element and its ancestor, sometimes serving partially redundant purposes. in an attempt to forestall unintended consequences from cascading click events, we would set the descendant's event handler to return false. we would then find that we had been unwittingly relying on the cascading click event to trigger another ancestor's click event handler to produce some effect. in our efforts to manage these issues, the javascript code evolved into a complex of event handlers with overlapping functionality, methods with passed callback functions, methods with flag variables to trigger or suppress behaviors, and global state variables that might charitably be described as a "morass." because of this, code management time and effort is increased; in addition, the code is not configured as a practical platform for other touchscreen endeavors. this limits the utility of the code for other projects. the quality of the javascript, in my own estimation, is the weakest point in this project's infrastructure. deployment deployment of the kiosk in the public library space has not been without problems. our original concerns about browser stability and slowdown have proven to be correct: occasionally the browser will slow down significantly, to the point where javascript stops responding or the page does not completely load after a triggered refresh. to remedy this problem, we periodically inspect the kiosk and manually restart the browser when its responsiveness seems lacking. it should be noted that during browser restarts, we do sometimes see that multiple tabs were open during the previous session; this suggests that users are, in fact, touching the "terms of use" link in the campus bus map application, triggering the opening of a new window. we have also had problems with software updates or browser crashes revealing the underlying system components of the kiosk setup. although we disabled automatic updates in firefox, some software that was already installed on the machine, such as adobe acrobat and apple itunes, had automatic updates enabled. when these update processes were triggered, it led to the display of a dialog box appearing over the browser window, breaking the illusion of a dedicated kiosk system and interfering with the usability of the kiosk application. power outages and software crashes are also problematic. in one instance, upon returning from a long weekend, i found that the kiosk was displaying its computer’s macos desktop rather than the kiosk interface, and nearly every application on the machine had been opened, presumably by a curious or malicious user. this circumstance was apparently the result of either firefox crashing or the system rebooting to the desktop. disabling double-clicks on the touchscreen would have helped to mitigate the miscreant behavior, but would not have prevented the exposed desktop. it is possible that some of the issues resulting from browser instability and/or system crashes could be mitigated by scheduled system reboots along with a boot script that automatically launches firefox, but we have yet to explore this solution. analysis as mentioned above, every user interaction and idle refresh is logged. logging was implemented on november 4, 2010, providing over 9 months of log data comprising over 500,000 log entries. of these entries, nearly 80,000 represent user touch interactions. an obvious question is which of the content areas is most popular, though the definition of "popular" is not necessarily clear: is it the content area that users select the most frequently? the content area that receives the most user touches? the content area that spends the most time on-screen while in interactive mode? one approach is to examine what content users are interested in when they approach the kiosk, on the premise that the first content area the user selects is the one most relevant to their needs. if we assume that the kiosk is in attract mode when the user approaches (which, granted, will not capture every instance where a user approaches the kiosk), we can measure this notion of popularity by examining what content the user selects to begin interactive mode. the screen exits attract mode when any part of the screen is touched; if the action that exits attract mode is a touch to one of the main navigational controls, then an "attract mode exit" is credited to the selected destination. for a touch anywhere else on the screen, the current content display remains active and the screen simply switches to interactive mode. in this case, an "attract mode exit" is credited to the currently-displaying content. the number and percentage of attract-mode exits for each content area between november 4, 2010 and july 21, 2011 is shown in table 1. content area attract mode exits percent of exits computer availability 3515 26.70% campus bus map 2915 22.14% groupfinder 2382 18.09% study room availability 2327 17.68% wayfinding 1146 8.70% library hours 880 6.68% total 13165 100.00% table 1. content areas by attract mode exits, nov. 4 2010-jul. 21, 2011 these results are more or less in alignment with our expectations, based on our anecdotal understanding of user interaction with fixed e-boards in the library, with computer availability and campus bus map dominating. however, it is possible that this data is skewed towards the campus bus map — during attract mode, the campus bus map displays the kiosk's largest explicit instruction to the user to touch the screen. it may be that the campus bus map may be capturing a higher percentage of attract mode exits than it would otherwise receive. we can consider another approach to identifying popular content: simply count the number of times a user touches one of the main navigation buttons. such touches represent a deliberate act on the part of a user to view a content area; while some acts of viewing particular content might not be captured by counting button presses, we can be reasonably certain that each button press represents specific user intent. the number and percentage of main navigation button touches for each content area between november 4, 2010 and july 21, 2011 is shown in table 2. content area button touches percent of touches campus bus map 10532 26.55% computer availability 10084 25.42% groupfinder 5900 14.87% study room availability 5621 14.17% wayfinding 4144 10.45% library hours 3392 8.55% total 39673 100.00% table 2. content areas by main navigation button touches, nov. 4 2010-jul. 21, 2011 during this time period, there were nearly 40,000 touches on the main navigation buttons, which includes attract mode exits that result from a touch to these buttons (about 84.5% of attract mode exits). since each touch on such a control corresponds to a specific user intent to view that content, and since the space of such touches captures the vast majority of attract mode exits, this analysis should provide us with a more reliable measure of user interest in the content areas. note that, contrary to our hypothesis that the campus bus map was receiving more than its due credit for attract mode exits, in the button-touched analysis the percentage of campus bus map selection is higher than in the attract mode exit analysis. otherwise, the relative ranking of each content area is the same. it is interesting to note that despite our having identified a need for publicly accessible, non-intermediated wayfinding assistance, the wayfinding feature of the kiosk sees relatively little use. it is unclear whether this suggests that our perception of a need is incorrect; it may be that a need does indeed exist but some particular aspect of the wayfinding implementation is not meeting the existing need [22]. the apparent lack of usage of the library hours function may indicate that users' need for library hours information is largely met by the static display of today's building hours that appears in the header bar of the kiosk interface, or that detailed service hours are of limited utility in a building that is open 24 hours a day. heatmap analysis recording the location of each touch interaction allows us to examine users' interaction patterns with on-screen content [23]. a standard approach for this sort of analysis is the heatmap. a heatmap visually represents clustering of two-dimensional data points: regions with many tightly clustered data points have a high value, whereas regions with no points have a zero value. value is typically represented by intensity of color. by feeding the (x,y) coordinates of touch interactions into heatmap generating software, we can create a heatmap that correlates frequency of touch interactions to the region of the interface where those touches occurred. clickheat [24] by labsmedia is an open source click-tracking and heatmap generation package for web servers, intended to track user clicks on websites and generate corresponding heatmaps. it consists of javascript components for click tracking and a php heatmap class for generation of heatmaps from the logs recorded by the tracking component. the heatmap package is designed to work directly with the logging methods provided by the click-tracking component, but a minor modification allows the generation of heatmaps from arbitrary data provided as a php array. the modified code, based on the file heatmapfromdatabase.class.php in clickheat 1.1, is available as an appendix to this article. the modification was straightforward: the standard version of heatmapfromdatabase.class.php reads coordinate data from a database connection in chunks, and renders it as a two-dimensional php array. we modified the class to remove the database connections and associated chunking, and added a public property $inputarray to the class so that an array could be generated externally and passed to the class via the property. we then modified the heatmap generation method to read $inputarray rather than read from the excised database connection. we configured the heatmap package to generate heatmaps using the default parameters (with the exception of suppressing a default "copyleft" logo on outputted images) and at the same dimensions as the resolution of the kiosk display. the source data for the heatmaps consisted of the (x,y) coordinates of user touches as derived from the kiosk logfile. in the gimp photo manipulation program [25], we transformed each image's color data to rgb format and then used gimp's "color to alpha" feature to generate a transparent heatmap image that could be manually overlaid onto screen captures of the kiosk interface. the images below were generated in this manner. for this heatmap analysis, we will examine all touches between january 20, 2011 and july 21, 2011. this is a shorter log period than the numerical analysis above; due to a minor interface redesign that included moving some control elements that occurred on january 20, log entries prior to that date do not reflect the current layout of the interface and cannot be applied to this analysis. a natural first investigation in our analysis is to examine all touches that occured during the log period. figure 9 below represents the heatmap of all touches overlaid on a "typical" kiosk display (in this case, the computer availability display). figure 9. heatmap of all touches between jan. 20, 2011 and july 21, 2011 as expected, there are many areas of high touch activity on the main navigational controls on the left. however, there is evidence of touch activity across the interface, which can lead to analysis difficulties. one approach is to concentrate only on touches that occurred on control elements, which includes both the main navigational elements and controls associated with specific content areas, which are not visible on all content screens. in figure 10, which includes all touches on control elements during the log date range, we see a concentration of touches on the left corresponding to the main navigational elements, as well as several clusters of touches elsewhere that are associated with specific content areas (such as the cluster in the upper right, which corresponds to a control that switches between two computer availability views). figure 10. heatmap of all control-element touches between jan. 20, 2011 and july 21, 2011 the most notable behavior in this view is the portion of the heatmap associated with the main navigational controls. we see a majority of touches on the first and fifth button, which correspond to computer availability and the campus bus map, respectively. this is informative, but this knowledge can be gleaned more precisely from numerical analysis. a more interesting approach is to examine only the touches not on control elements, which we will refer to as "random" touches. such touches can reveal information about user expectations, error patterns, and more. this analysis is most telling if we consider only the touches that occur during the display of particular content areas. for instance, figure 11 is a heatmap of all random touches during the log period that occured while group study room availability was displaying. figure 11. heatmap of random touches to group study room availability display between jan. 20, 2011 and july 21, 2011 user expectations and discovery the telling pattern in figure 11 above is the array of touches on room labels and on the room availability grid itself. it is clear that users are expecting that (or hoping that, or curious whether) some sort of action will occur when they touch the availability matrix; perhaps they expect some contextual method to make a reservation or to see what is booked at a given time. a similar pattern of behavior can be seen in the random touches on the computer availability display (see figure 12): figure 12. heatmap of random touches to computer availability display between jan. 20, 2011 and july 21, 2011 here, we see patterns indicating that users are touching the dots that represent computer locations. it is unclear to me what kind of actions they may be expecting as a result. it seems that, despite efforts to make control elements have a uniform appearance, the distinction between control and non-control is not clear to users. fault may in part lie with our own design, as there are some control elements, particularly in the wayfinding content area, that do not have the "button" styling we created for other controls. it may also be the case that users simply don't understand the distinction between controls and non-controls regardless of how deliberately the controls are designed; since there is no cursor and none of its associated affordances, users experiment with the interface to discover which features act as controls and which elements are inert. in this sense, touches to both inactive and active elements serve as discovery tools for understanding the environment of the kiosk. implementation errors referring back to figure 12, there are a number of touches recorded on the control in the upper right. these touches are a cause for concern, as this button is considered a control element and this heatmap is meant to capture only touches to non-control elements. this suggests the possibility of a coding error — a control element not correctly identified as such when touches on it are logged — or an error in input capturing, where touches are logged but do not correctly trigger an event on the control. in either case, this is a phenomenon worthy of closer investigation. that the heatmap analysis reveals this information is of interest, since it is unlikely that the existence of these errors would have been otherwise revealed. design errors heatmap analysis can also reveal design flaws. consider the heatmap of random touches on the campus bus map (figure 13). note that touches on the "touch screen" instruction are counted as random touches, and that touches on the active campus bus map are not captured by the logger. figure 13. heatmap of random touches to campus bus map display between jan. 20, 2011 and july 21, 2011 notice the map reset button in the upper right. it is surrounded by a halo of touches, particularly on the upper and lower edges, that only just fail to hit the button. it is clear from the touch patterns that the button is too small. this fundamental design error's significance is amplified by three factors that are endemic to touchscreen inputs: the "fat fingers" problem (leung 2005, 25; meredith 2008): a user's finger is typically considerably larger than a single logical location on a touch input device. the touch input must use some form of averaging to reduce the contact area to correspond to a single location in hardware (see, e.g., ball et al. 1980 [26]); since this introduces ambiguity in the correspondence between the location of physical touch and the location of the touch as interpreted by the interface, controls in a touchscreen context should be made oversize to account for this ambiguity. parallax (leung 2005, 25): in practice, the touch-sensitive layer of a touchscreen and the image produced by the display are at different distances from the user. the resulting parallax effects can lead to user touches that are not in alignment with the intended target, despite appearing aligned to the user. it should be noted that users of different heights or standing before the screen in different left-right orientations will experience different parallax effects. manual calibration between touch-sensitive space and pixel space (leung 2005, 26): touchscreens, including the one used in this project, typically require calibration software that uses sample touches to map the touch input's touch-sensitive region to the pixel space on the system's display. calibration is typically performed only infrequently. combined with the above effects, the process of calibration can introduce systemic error into the mapping between touch interface locations and corresponding screen locations. the conclusion to be drawn is that this button — and controls on touch interfaces in general — should be made as large as feasible. this conclusion is supported by fitts' law (heim 2007, 260), which suggests that the larger the control, the more quickly a user will be able to successfully interact with it. one possible solution is to make the active area of the control slightly larger than the control's image (st. amant 1999, 339). this provides the control with a "buffer" or margin that allows for slight alignment failures in any direction, which can help to alleviate some of the additional challenges associated with fat fingers, parallax and calibration errors, all of which come down to a lack of alignment between the touch as actuated by the user and that touch as interpreted by the device. curiosity consider figure 14, a heatmap of all random touches in the logging period. figure 14. heatmap of all random touches between jan. 20, 2011 and july 21, 2011 note the touch activity on the qr code in the lower right. this suggests that users are curious about the qr code, perhaps expecting to receive additional information if they touch it. users familiar with qr codes may be interested in learning where the code points; unfamiliar users may simply want to learn what it is. the density of touches on the qr code may also tell us something about our user population: although we can't know what information users are expecting to learn when they touch the code, it is a reasonable assumption that at least some of the touches arise from certain users' complete lack of familiarity with the object. this provides some anecdotal evidence to suggest that our user population may not be as savvy to mobile technologies as we sometimes assume. in general, touches on non-control elements, such as qr codes, the room availability matrix, and the computer availability map, reveal valuable information: users are interested in interacting with these elements, and may expect to be able to gain more information by interacting with them. this realization can help to guide future development efforts: if a user touches a qr code, we may want to provide a popup describing qr code technology and the code destination. we may wish to provide some detailed information on a particular room's availability, or instructions on making a room reservation, when users touch the room availability matrix. there may be information we can provide to users in the computer availability context that users would find informative. although determining the exact nature of these improvements is probably best left to user studies, the very presence of touches on non-control elements reveals an interest among users in engaging in more in-depth interaction with these components. conclusions development of a public, touchscreen kiosk is possible at low cost, using standard, nonspecialty hardware and free software, and requiring only standard web development skills. such a kiosk can be robust enough for continuous operation with 24-hour user access, though regular manual inspection and occasional software resetting may be necessary to ensure responsiveness. interface design for a touch kiosk interface must take into account the unique context. design considerations must include an awareness of the limitations of the touch-interactive environment, including the relative imprecision of user interactions and the reduced contextual feedback available to users. users adapt to the lack of contextual feedback by experimenting with the interface through touch, the only input available to them. traditional web development practices are adequate to create an effective user interface for touch kiosk applications. this project has seen some difficulties resulting from a short project timeline and a prototype implementation that was not sufficiently revisited before being put into production; more rigorous software engineering practices would have produced more versatile and maintainable code, which would have in turn made it easier to update the interface in response to usage data. numerical analysis can provide a basic level of usage information for a touch-based kiosk, including exposing usage patterns and user demand for available content areas. in the context of touch interfaces, heatmap analysis has great potential value: as user input is limited to touch, touch logging is able to capture the entire body of user input to the device. heatmaps associated with this input, particularly heatmaps focused on non-control components of the interface, can reveal user expectations, design and coding errors, and interface components in need of added functionality. there is much room for improvement of this project, and for exploration of touchscreen kiosk design in general. heatmap analysis has revealed several avenues for investigation and improvement of the kiosk interface. developing a more robust and flexible version of the software driving the kiosk interface is a priority. we hope to identify a set of browser plugins, or an alternative to mozilla firefox, that will permit a secure kiosk mode on a current-generation web browser. on a larger scale, task testing on a large-format, wall-mounted touchscreen is in order to help identify the ideal size and layout for user interface controls in a public kiosk context. we feel that there is more to be learned from heatmap analysis of touchscreens, possibly through examining other slices of data. the techniques used in heatmap analysis in this investigation are easily adaptable to touch-sensitive displays of any size, and we are curious to see what discoveries will be made using this technique to analyze mobile-scale touch applications. acknowledgements my thanks go to colleagues and project collaborators keith morgan and jason walsh, without whom this project would not have been possible. keith, our resident expert on e-boards, brought extensive knowledge of e-board hardware management to the project and is largely responsible for making hardware selection and configuration go as smoothly as it did. jason maintains and wrangles many of the library data services that were integrated into this project, and is responsible for adapting most of the previously existing content displays for use in the kiosk interface. the three of us were equal partners in determining the concept, direction, and design of the kiosk, and without jason and keith's extensive contributions and many hours of collaboration, this project would not have been a success. thanks to jason casden, digital technologies development librarian, and troy hurteau, technology support specialist, for their invaluable advice and assistance in javascript, jquery, and php development. thanks to emily struthers for her contributions to the wayfinding components of the interface. thanks to bohyun kim, jonathan rochkind, and others who provided feedback and encouragement in response to my code4lib 2011 lightning talk on heatmap analysis. thanks to kim duckett, gillian speace, markus wust, carolyn argentati and anne burke for their feedback and editing assistance. thanks to andrew darby and the code4lib journal editorial staff for their careful attention and extensive editorial input. about the author andreas k. orphanides is librarian for digital technologies and learning in the research and information services department at the ncsu libraries. his work focuses on the intersections between technology, instruction, and public services. he holds a ba in mathematics from oberlin college and an msls from the school of information and library science at the university of north carolina at chapel hill appendix a: code due to technical limitations, the file linked below is named “heatmapfromarray.phps” on the server. after saving, rename it to “heatmapfromarray.class.php” to ensure proper execution. heatmapfromarray.class.php notes [1] hours: ncsu libraries http://www.lib.ncsu.edu/hours [2] keith morgan: ncsu libraries http://www.lib.ncsu.edu/staff/kamorgan/index.php [3] jason walsh: ncsu libraries http://www.lib.ncsu.edu/staff/jdwalsh/index.php [4] computer availability display: dli: ncsu libraries http://www.lib.ncsu.edu/dli/projects/compavailability/ [5] the ncsu wolfline – transit visualization system http://ncsu.transloc.com/ [6] groupfinder: dli: ncsu libraries http://www.lib.ncsu.edu/dli/projects/groupfinder/ [7] 4420l 42" desktop/wall-mount touchmonitor http://www.elotouch.com/products/lcds/4220l/ [8] mac mini (mid 2007) – technical specifications https://support.apple.com/kb/sp7 [9] touch-base ltd – universal touch screen and controller drivers http://www.touch-base.com/ [10] doomlaser » » cursorcerer: hide your cursor at will http://doomlaser.com/cursorcerer-hide-your-cursor-at-will/ [11] get more from your firefox — mobile, add-ons & other stuff http://www.mozilla.org/en-us/firefox/new/ [12] r-kiosk :: add-ons for firefox https://addons.mozilla.org/en-us/firefox/addon/r-kiosk/ [13] procon latte content filter :: add-ons for firefox https://addons.mozilla.org/en-us/firefox/addon/procon-latte/ [14] mozilla firefox 3.6 http://www.mozilla.org/en-us/firefox/all-older.html [15] opera browser | faster & safer internet | free download http://www.opera.com/ [16] yui 2: grids css http://developer.yahoo.com/yui/grids/ [17] transloc inc: transit visualization http://transloc.com/ [18] livedisplay configuration http://ncsu.transloc.com/livedisplay/livedisplay_configuration.php [19] jquery tools – the missing ui library for the web http://flowplayer.org/tools/ [20] jquery: the write less, do more, javascript library http://jquery.com/ [21] originally we had used a combination of ssi and php includes to bring this content into the main php file, but we found that database requests in external php files sometimes behaved in an unpredictable manner if they were included via the ssi “include” directive. because of this, we preferred php include statements over ssi directives early in the development lifecycle. eventually, when we recognized the need for regular and frequent ajax updates of time-dependent data, we elected to eliminate in-script include statements, preferring dynamic ajax content loads to pull in the external data even on initial display. [22] code4lib editor tod olson points out that low usage of the wayfinding feature does not necessarily imply failure of the feature. he remarks that “…specific wayfinding tasks tend to be easily learned, [thus] even when the wayfinding aid is successful the person will rarely come back for the same task,” as they will not need to refer to the wayfinding information again. by contrast, the ever-changing nature of bus locations means that the campus bus map will be revisited again and again by those who consult it. [23] the ability to analyze user interactions "remotely" is of particular value here; since the kiosk is unique in the library and is located in a very high-traffic area, direct user observation and usability testing is impractical. [24] labsmedia's clickheat – clicks' heatmap http://www.labsmedia.com/clickheat/index.html [25] gimp – the gnu image manipulation program http://www.gimp.org/ [26] ball et al. 1980, 204, referring to an early trackpad: "a rigid epoxy glass base has a series of 128 conductors printed on it and a flexible membrane with 128 orthogonal conductors is separated by about half a millimetre from the base by a slight air pressure. a small downward pressure by the finger causes a number of the flexible conductors to make contact with those on the base…. a scanning process, to calculate the centroid of the contact patch, is controlled by an intel 8080 microprocessor." references ball, r.g., r.s. newton, and d. whitfield. (1980). development of an off-display, high resolution, direct touch input device: the rsre touchpad. displays 1(4), 203-207. doi:10.1016/0141-9382(80)90137-7 (coins) heim, s. (2007). the resonant interface: hci foundations for interaction design. boston: pearson addison wesley. leung, c.k. (2005). handling ambiguous user input on touchscreen kiosks [thesis]. cambridge, ma: massachusetts institute of technology. 94p. <http://hdl.handle.net/1721.1/34461> meredith, w. (2008 may 16) [cited 2011 sept 1]. best practices of touch screen interface design [blog post]. voltage creative: 360angles. <http://voltagecreative.com/articles/best-practices-of-touch-screen-interface-design/> morgan, k. and t. mori. (2008). digital signage in the library. 11th annual lita national forum, cinncinatti, ohio, october 16-19, 2008. <http://www.ala.org/ala/mgrps/divs/lita/conferences/forum/2008/21_digital_displays.pdf> ryan, j. and j. boyer. (2011). groupfinder: a hyper-local group study coordination system. the code4lib journal 13. <http://journal.code4lib.org/articles/5001> sierra, t. (2009). patterns in academic library computer use. triangle research libraries network (trln) annual meeting 2009, chapel hill, nc, july 24, 2009. <http://www.lib.ncsu.edu/dli/projects/compavailability/trln-annual-2009.ppt> st. amant, r. (1999). user interface affordances in a planning representation. human-computer interaction 14(3): 317-354. doi:10.1207/s15327051hci1403_3 (coins) subscribe to comments: for this article | for all articles 7 responses to "lessons in public touchscreen development" please leave a response below: marius, 2011-10-31 hi andreas, interesting article, congratulations! i was wondering whether you (and the team) considered flash as a potential platform to develop the entire interface, without using a browser. and if so, what were the pro/cons? thanks, marius andreas orphanides (author), 2011-11-01 hi marius, thanks for commenting, and for the compliments! when we were choosing a development environment, the question we asked ourselves was really more like, “do we want to use html/javascript for this project, or something else?” choosing html was an easy way to leverage our existing strengths while keeping the project unknowns bounded — which was important given the relatively short turnaround time for the project. setting aside the above, i am not sure whether we would have taken a flash approach. there’s not a strong domain knowledge of flash in general at our institution, so implementing the application in flash would have incurred a significant cost in terms of developer training. in addition, since almost all of the content in the application already existed as php applications rendering html, keeping the main interface html-based allowed us to do all of our work in a single (very familiar) development environment. developing the main application in flash would involve either rebuilding existing content in flash, or a lot of negotiating between the flash components and the html components to get everything looking and behaving correctly. i think that if we already had significant flash expertise and we weren’t repurposing existing html content, there would be an argument for building a freestanding flash application — but in our context the choice of html/javascript was clear. -andreas andreas orphanides (author), 2012-03-27 here’s a quick update since the article came out. although we’ve been working on a new interface design that addresses some of the problems i uncovered in this article, we haven’t implemented any major changes to the interface yet. i’ll probably spend some time this summer working towards an all-new interface with an improved codebase. the most significant change that we’ve recently implemented is at the back-end: we switched from firefox (which somehow forced itself to upgrade to ff10 even though i’d deliberately installed a legacy version and thought i’d disabled auto-updates) to opera. once firefox was on version 10, the entire interface became phenomenally unresponsive to the point of being almost useless, so the change was quite warranted. switching to opera was pretty easy: there were almost no changes to page rendering other than a couple of acceptable alignment issues, and the whole interface was more responsive and stable than even firefox 3.6 was. opera also has really robust built-in kiosk and url whitelist functionality, as i alluded to in the article. however, now that we’ve spent a few months on opera, we’ve uncovered a few issues. after opera has been running for a long time (read: a couple of weeks), the flash application that runs the campus bus map starts loading very slowly, and sometimes stops loading altogether. in addition, shutting down and restarting the browser seems to take longer and longer. this came to a head last week when the browser simply wouldn’t restart. a bit of log analysis and some online research suggested a problem with the cache; and lo, when i manually deleted the opera cache i found that it contained over 5.5 million files. this is despite default caching policies in opera limiting cache size to 20 megabytes (by my calculations, just the index tables for 5.5 million files is somewhere in the gigabytes in hpfs). deleting the cache files fixed the issue and brought back opera’s zippiness. i also set opera’s cache limits a little lower, though i don’t think this is the problem. i suspect that what’s happening is related to the flash application running the campus bus map: my guess is that it uses opera’s cache directory but maintains its own caching policy; it probably also isn’t so smart about reusing cache files with the same content. i can just imagine an entire new image of all the flash application’s contents being cached every time someone uses the bus map application. after a few hundred renderings, the cache would be ridiculously bloated. i haven’t put a permanent fix in place yet, but the solution is pretty simple: regularly delete opera’s cache manually. i should actually be able to integrate this into the browser restart procedure, since i’m using a script to ensure that opera starts in kiosk mode. i’ll just prepend a couple of carefully crafted “rm -rf *” commands, and then every time opera starts it’ll have a squeaky clean cache. if anyone has questions about opera kiosk mode or the configuration files i’m using, let me know and i’ll be happy to share them. gslis jobs and opportunities blog » systems librarian internship, paul smith’s college, paul smiths, ny, 2012-06-27 […] – implement a touch screen interface for a public kiosk (similar to ncsu – for context, see here: http://journal.code4lib.org/articles/5832) – develop a room reservation system for group study rooms that interact with hardware screens – […] gslis jobs and opportunities blog » systems librarian internship, paul smith’s college, brighton, ny, 2012-07-09 […] – implement a touch screen interface for a public kiosk (similar to ncsu – for context, see here: http://journal.code4lib.org/articles/5832) – develop a room reservation system for group study rooms that interact with hardware screens – […] miriam, 2015-10-29 hello! thank you for this insightful walk through your process. it is magnanimous of you to put out this document for the rest of us to learn from. i’m curious: did the question of universal design or accessibility come up? thank you for your time. andreas orphanides (author), 2015-11-20 ooh, great question. accessibility was a definite concern, though it was also a point of failure for the prototype. the main failure from my perspective of the prototype is: even though the actual touchscreen is installed at an accessible height, not all of the controls are within side-reach height. this is something i remedied in the second version of the touchscreen, where i deliberately put all the interactive controls beneath the side-reach height. other accessibility concerns i attempted to address in the second version were things like contrast ratios and color selection (also potential points of failure on the prototype). for instance, study room availability used to just be gray (occupied) vs green (available), but in the second version the occupied gray has a diagonal pinstripe effect to give it more textural variation from the available blocks and make the difference more apparent to colorblind users. one really nice thing about this particular touchscreen is that the touch sensing is piezo-acoustic and not capacitative. that means that you can touch it with a pen or a cane or a prosthetic limb and the touches will register. i think this is a really important consideration when choosing a touch technology for a public kiosk that often gets overlooked. i’m not sure how much more i could realistically do to make things better for blind/low-vision users. the text could be larger in some parts (though there’s the inevitable balance to be struck between text size and information density). the system is about due for another redesign, so i’ll be thinking about how to make the interface even more universal in the future. one grace (read: cop-out) is that all of the content available on the touchscreen is also available on our website. of course, this just pushes the accessibility question from me to the web team, which isn’t really fair, and it’s also not really in the spirit of universal design. if someone happens to know of a good set of accessibility practices specifically for a setting like this, i’d love to see it. leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – conference report: code4lib 2010 mission editorial committee process and structure code4lib issue 9, 2010-03-22 conference report: code4lib 2010 conference reports from the 5th code4lib conference, held in asheville, nc, from february 22 to 25, 2010. the code4lib conference is a collective volunteer effort of the code4lib community of library technologists. included are three brief reports on the conference from the recipients of conference scholarships. by birong ho, banurekha lakshminarayanan, and vanessa meireles the 5th code4lib conference was held in asheville, nc, from february 22 to 25, 2010. as in years past, the code4lib community was able to offer scholarships focused on gender diversity and minority representation, this year sponsored by oregon state university and brown university. following are conference reports from the scholarship recipients. for more notes, many informal, by other code4lib community members, find content on the internet tagged “c4l10″ or “#c4l10″, a label attendees were encouraged to use to aid collocation. for example: http://delicious.com/tag/c4l10 photo 1 the conference t-shirt: code4lib pacman, tagged c4l10 on flickr photo credit: ranti junus from birong ho: solr power my favorite theme in code4lib 2010 was solr. i found these sessions extremely helpful, especially since i am directly involved in implementing vufind as the primary interface for our opac here at western michigan university. [1] solr is an open source enterprise search server based on the lucene search library, with xml/http and json apis, hit highlighting, faceted search, caching, replication, and a web administration interface. at the pre-conference, there were solr white belt and black belt sessions. in solr black belt, eric hatcher, from lucid imagination, gave detailed explanations of the features of solr version 1.4. basic searching for both the standard lucene query parser and the dismax query parser were explained. eric also introduced solr fundamentals such as caching, replication, faceting, and distributed search. in general dismaxqparser is good for situations when you want to pass raw input strings from novice users directly to solr. by default, query params use luceneqparser, but you can use prefix notation to override that. he gave a detailed explanation of dismaxqparser about what parameters control which fields are searched, how significant each field is, how many words must match, and how to allow for additional options to artificially influence the score. advanced features such as solr apis, deployment architecture, and the solr index were also explained. processes such as the indexing of different documents (e.g. xml, csv) were demonstrated. he then went over the following tools, which can be configured in schema.xml: analyzers (process text before tokenizing), tokenizers (control how your text is tokenized), and tokenfilters (mutate and manipulate the stream of tokens). finally, searching performance was discussed. for more information on solr, see the bibliography below. the conference itself had numerous talks on solr-related topics and projects, including 3 blacklight-related talks and a breakout session, a talk on call number browse, and 3 sessions on advanced search. blacklight solr 1.4 is the backbone of the blacklight project, which uses solr, rather than a traditional relational database management system (e.g. oracle or mysql), as the index. uw forward (lightning talk): blacklight is the framework behind this project at the university of wisconsin libraries system. the challenges they have include multiple campuses and deduping marc data to be included in the master solr index. hydra (lightning talk): blacklight + activefedora + rails at stanford university brings a ruby on rails front-end to a fedora-based repository. media, blacklight, and viewers like you: chris beer from wgbh interactive media archives talked about their open vault project. in the open vault project, blacklight and solr are the major frameworks used. media material is the major component in their solr index. call number browse how to implement a virtual bookshelf with solr: naomi dushay and jessie keck of stanford implemented an interface for the browsing of call numbers. they applied a lexical sort on call numbers, found them very messy to deal with, but still managed to produce a sorted list with 8 million entries. advanced search complicated searching algorithms implemented in advanced searching menus were explored in various manners during the conference. they included: catalog auto-suggest using solr (lightning talk): jill sexton from the university of north carolina talked about using already extracted authority data to create a solr index. this module uses author, title and subject authority files to auto-suggest search terms for users when they input the string in the search box. “do you mean” is used instead of “did you mean.” kill the search button: michael nielsen and jørn thøgersen presented their project from afar. they make the search experience easier for users by updating search results at keypress. jquery is used for their front-end, backed by a solr index with eight million marc records. a better advanced search: naomi dushay, from stanford university, shared her definition of advanced search and explained its solr 1.4 implementation. she also explained the challenges, which include multi-select facets in the advanced search, the integration with the current user interface, and actionable facets in the search results. conference formats included 20 minute sessions and 5 minute lightning talks. all speakers came well prepared, and even the speakers at the lightning talks were well-organized, many with powerpoints. the host universities put enormous efforts into this conference including moving people to and from the airport. and it was a well-received conference by all attendees. i will definitely go again next year and will recommend this conference to system librarians, software developers and server administrators. notes i gave a lightning talk about western michigan university’s vufind. further reading on solr lucidworks for solr certified distribution reference guide published by lucid imagination. smiley, d., & pugh, e. (2009). solr 1.4 enterprise search server. packt publishing. coins from banurekha lakshminarayanan as a software programmer who has worked mostly in retail and automotive companies, i had limited knowledge and exposure to the revolutionary things that can be accomplished in library systems. with one month of experience at the hesburgh library in university of notre dame, i headed to the code4lib conference not knowing what to expect, but looking forward to learning and interacting with my peers from across the country. pre-conference & conference talks i liked the way pre-conferences were arranged. they were mostly workshops and we could choose the one that we were interested in. i liked getting hands-on knowledge of digital repository technologies such as solr, fedora and blacklight. i attended the blacklight workshop and got some guidance on how to get started with it. in my opinion it would have been more useful if this workshop had dedicated more time towards the installation and basic configuration of blacklight, since many of us were from diverse fields and new to this technology. another feature that was nice about the conference was the way presentations were organized. every effort was taken so that everyone could participate. i also liked the presentation by bess sadler, “vampires vs. werewolves: ending the war between developers and sysadmins with puppet.” bess discussed the most common issues that every developer and system administrator face in day-to-day life and also discussed some solutions to resolve them, and made us all chuckle during the process. breakouts & lightning talks the conference facilitated an amazing amount of participation/contribution with its breakout sessions and lightning talks. the breakout sessions allowed attendees to informally discuss current projects, issues, and recommendations. the lightning talks covered a variety of topics (applications, specifications, protocols, etc.) and introduced me to new ideas, resources and projects for further investigation. i found “hydra: blacklight + activefedora + rails” very useful since we at notre dame are trying to build a digital repository using the same architecture. the hydra development team is trying to build a pretty nice framework for digital repositories for the entire community, and i am very excited about exploring this application. i liked the way lightning and breakout sessions were arranged, and the lightning talks were very diverse. some of them were extended into breakout sessions as well. i felt the breakout sessions helped me to get better acquainted with people who were trying to achieve similar goals in the library. a few of the lightning sessions were difficult to follow, as they were presented in a rapid manner. i understood that this was the nature of such sessions, but thought that if the time for each session was increased and the total number of sessions decreased, it would have helped us to better understand the topics. ask anything finally, i believe the ask anything session deserves its own praise. the idea was to ask a roomful of people for help, advice, references, etc. on any topic, and i think it worked very well. this session proved the vibrancy and strength of code4lib: largely unstructured, social, and focused on mutual aid. i will spare you any comparisons to rainbow gatherings, but i’ll say this much: i’m proud to be part of code4lib, and i’d like to encourage anyone who is interested in library technology to attend this conference. this is one of the best conferences i have attended. i saw people who were willing to help each other and willing to contribute a lot to the community. i gained a lot of knowledge and i look forward to my next code4lib conference. from vanessa meireles as a software engineer with many years of it industry experience, the code4lib 2010 conference was a very different and educational experience for me. this experience began with the preconference, where i attended the web services and widgets session. it was a very informative session with great innovative ideas for providing more services to users, as well as demonstrating how some traditional ils systems limitations have been addressed. these are common frustrations to library programmers and system librarians. that day continued with the “newbie dinner” where my coworker and i met some other conference attendees; i think this was a great idea and opportunity. i do think that the organization of finding our restaurant group was chaotic and that eight people is too large of a group for everyone to talk and get to know the other individuals during dinner. a smaller group of people, either five or six, would be easier to interact with and seat at restaurants. i enjoyed all sessions during the conference. i found karen a. coombs’ session concerning the “7 ways to enhance library interfaces with oclc web services” very straightforward, easy to follow and extremely helpful. her seven possible ways to improve the local opac are great. i also found both sessions on thursday concerning “…developing a mobile catalog” and “mobile web app design” very thorough, well researched, informative and with many tangible examples. an important aspect that i did find refreshing was the collaborative spirit and use of open source software and platforms wherever possible, as well as an overall willingness to share ideas, experiences and to listen to other ideas and challenges our peers are facing. this conference was a great networking experience, a chance to meet colleagues from other universities and see some of the cutting-edge projects they are working on. i was able to gather all this information through the conference topics, the lightning talks, break-out sessions and side conversations in the irc chat channel. at times it was very distracting to follow the irc chat channel and the presentations simultaneously. the irc chat channel consisted of informal side conversations and inside jokes as well as occasional data relevant to the presenter’s topic. code4lib is very community oriented. the community votes months ahead of time to choose presentations on the topics they want to hear discussed. i liked the sequential presentation arrangement for the conference. it ensured main conference topics did not occur simultaneously and, therefore, i didn’t feel that i missed out on anything or was rushed from one room to another. photo 2 our seating arrangement where everyone sits together for the main conference topics. this seating arrangement is perfect in fostering a community feeling. photo credit: thomas mcmillan grant bennett the lightning talks allowed for a wide coverage of topics in a short period of time, which enabled conference attendees to be exposed to more topics. sometimes these lightning talks had enough interest to create breakout sessions. these breakout sessions are informal topic-centered discussions held in different rooms. coming from a more structured computer science industry approach, i found these to be similar to brainstorming meetings and some breakout sessions had lively heated debates. for example, one such debate concerned metadata and vocabularies. discussion touched on whether there is universal benefit in incorporating locally produced metadata terminology into existing and new controlled vocabularies or whether locally produced metadata is irrelevant to register because it is specific to local scenarios. photo 3 the hotel lobby where we had nice afternoon cookie and caffeine breaks, essential for all technical people, and another opportunity to mingle and network. photo credit: vanessa meireles i feel that the code4lib 2010 conference was a great learning experience and that i was able to bring back many new and innovative ideas, along with solutions for the projects our library is working on. it was also a wonderful networking experience. personally, i think the conference might not be beneficial to non-technical individuals who have very limited to no programming experience and who are inexperienced with library technologies. i would also not recommend the conference to technical software engineers whose job roles do not deal with libraries directly or encompass similar needs to the library industry, since the conference is very industry-specific. i was very pleased with the level of in-depth technical talks and the relevance to the library industry and my current job role, hence i think the conference name of code4lib suits it perfectly. about the authors birong a. ho, recipient of a 2010 code4lib gender diversity scholarship, is a systems librarian at western michigan university libraries. birong’s library experience includes work in cataloging, metadata, data conversion, and science reference. she has an ms in electronics and computer control systems from wayne state university, an mlis and an ma in english from the university of wisconsin-madison, and a ba in western languages and literature from national chengchi university in taipei, taiwan. banu lakshminarayanan, recipient of a 2010 code4lib minority scholarship, is a java/web developer with over five years of experience. she is new to library technology and recently joined hesburgh library in the university of notre dame. she holds a bachelor of engineering in computers from madurai kamaraj university, india. vanessa meireles, recipient of a 2010 code4lib gender diversity scholarship, joined the university of miami libraries in february 2009 as their main programmer. before working in libraries, she spent 14 years in the it industry, including 6 years at ibm. vanessa holds a bs in compuer science from the florida atlantic university and is pursuing a masters of science in management, technology & economics from the swiss federal institute of technology in zurich. tags: c4l10, code4lib subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – for video streaming/delivery: is html5 the real fix? mission editorial committee process and structure code4lib issue 22, 2013-10-14 for video streaming/delivery: is html5 the real fix? the general movement towards streaming or playing videos on the web has grown exponentially in the last decade. the combination of new streaming technologies and faster internet connections continue to provide enhanced and robust user experience for video content. for many organizations, adding videos on their websites has transitioned from a “cool” feature to a mission critical service. some of the benefits in putting videos online include: to engage and convert visitors, to raise awareness or drive interest, to share inspirational stories or recent unique events, etc. along with the growth in the use and need for video content on the web; delivering videos online also remains a messy activity for developers and web teams. examples of existing challenges include creating more accessible videos with captions and delivering content (using adaptive streaming) for the diverse range of mobile and tablet devices. in this article, we report on the decision-making and early results in using the kaltura video platform in two popular library platforms: contentdm and dspace. by elías tzoc and john millard introduction the video market share on the web is such that some tech companies such as cisco are now predicting that online video will surpass social media in popularity by 2017.  for universities, creating and delivering video content has also gained attention in the last few years, especially for those moving into hybrid (online and face-to-face) types of classes and e-learning initiatives.  as for library content, the digitization of collections originally stored on reel-to-reel, vhs or dvd along with the creation of new video content like oral histories, has created new opportunities to deliver video content on the web. flash was once the preferred video container file format used over the internet; however, the new version of html5 offers new possibilities for managing and delivering video content.  web developers can create customized video players and create more accessible captioned videos, end-users can stop installing third-party browser plugins including flash.  flash is particularly problematic due to generally high cpu usage for video playback and no support for ios devices.  a recent study on video performance analysis between flash video and html 5 video suggests that a combination of both technologies may still work better [1].  in the next sections, we report on the decision-making and early results in using the kaltura video platform, the workflow and tools used in migrating and re-formatting several video collections, the integration of a flash with html5 fallback in popular library platforms, and an early evaluation of the time/cost of creating videos with captions. evaluation of video delivery options in  late 2012, miami university libraries faced a perfect storm of technical developments that forced the re-evaluation of our delivery of media-based digital collections.  on the one hand, we were forced to migrate our content due to hardware failures locally and decommissioning of a centrally managed statewide repository.  on the other, we had the opportunity arising from tremendous gains in network bandwidth both on campus and in “last mile” providers and of course the continued transition to mobile devices in a post-pc landscape.  our previous solution for delivering media involved a local quicktime streaming server for some content and a flash-based progressive download system delivering other content from the statewide repository running dspace. as we began the process of evaluating options, we developed a short list of required capabilities that addressed perceived shortcomings in our previous configuration. standards compliant and non-proprietary:  whatever method we chose had to provide high quality video and audio and be standards compliant; something based on open source technologies would be a highly sought after plus. not flashy: the selected solution needed to not rely on flash or installation of a plugin.  flash excludes ios devices and plugins would most likely be platform specific. keep it simple: be simple to implement and maintain; like many organizations, our staff is small and we have multiple competing obligations; thus, the solution needed to “just work” for both novice and experienced staff. embeddable: ability to embed content into multiple existing or future digital delivery platforms; right now we use contentdm, dspace , and omeka, but that may change tomorrow. in general, media files can be delivered in one of three ways, via streaming, progressive download, or adaptive bitrate streaming.  each has its purpose and for the most part, regular users will not notice the difference. streaming involves delivering the media to the client via a server process using specific streaming protocols (such as rtmp).  video playing begins almost immediately, especially if the video file was encoded at a data rate similar to the effective bandwidth of the target viewer.  streaming video is also often not cached by the client so a local copy of the video is not held in its entirety on the client machine.  while it is not impossible for an enterprising person to capture and hold a copy of the stream, it takes more effort than the casual viewer may be willing to take on.  to adapt for the slowest common denominator in regard to end-user bandwidth, streaming videos are often encoded at lower quality and data rates. progressive download simply delivers a media file via traditional webserver technologies.  the file begins playing on the client as soon as enough data has been buffered to provide a smooth uninterrupted viewing experience.  progressive downloaded files are easier to capture since an entire copy of the file is downloaded to the local machine.  also, the quality of the file can be higher simply because a user on a slower connection will just have to wait longer for the viewing to begin.  progressive download, at least in our experience, is also easier to graft into systems that don’t natively support video and audio such as dspace and contentdm. adaptive bitrate streaming is a kind of best of both worlds.  as the name implies, adaptive bitrate is a streaming technology and generally requires a dedicated streaming server.  in this case, media files are transcoded into multiple bitrates with the appropriate streaming being delivered to the user based on their available bandwidth.  adaptive streaming servers can also dynamically change the bitrate as network conditions dictate. early in our evaluation, we considered adopting whatever platform was being supported by our local it organization for e-learning and copyright compliant streaming.  however, until recently, that service had been tailored for restricted access viewing of commercial content and as such was limited to on-campus use by authenticated university users.  the libraries needed a platform that allowed open, unregistered access to copyright free materials.  the environment changed this year with the adoption by the university of the hosted kaltura video platform service. the kaltura hosted service is built on the kaltura open source online video platform.  as a service, kaltura met all our requirements.  it is an adaptive bitrate solution that automatically transcodes uploaded video to optimized output formats.  the video player provides automatic device detection and delivers either an html5 or flash player depending on capabilities.  kaltura claims support for android, blackberry, and ios devices.  published pricing starts at $750/month so it’s definitely in the range of enterprise level as far as cost.  however, the method we describe works with similar services like vimeo which can be hosted more cheaply for smaller scale projects.  in our case, we piggy-backed on an institutional subscription. migrating and re-formatting video collections much of our open access video content was stored on analog media.  one of the biggest challenges was creating digital media from obsolete and rapidly disappearing technologies like reel-to-reel tape, betamax, vhs, digital audio tape, audio-cassette, and even some radio nab cartridges.  with help from a now defunct local radio station and other sources, we’ve acquired machines to play back most of the media we encounter, including broadcast reel-to-reel, dat, sony minidisc and betamax. we have assembled a rube goldberg-esque system of analog to digital convertors and media adapters to connect our analog devices to a mac pro for processing.  video devices connect through a firewire da converter or a digital minidv deck.  audio devices are connected through a roland usb audio interface.  we use the usb device to filter processor and hard drive noise and to allow more connection types.  the roland, for example, can accommodate 3pin xlr and ¼ inch inputs commonly found on professional audio equipment.  we use audacity for audio capture and editing and final cut pro x for video capture and editing.  we also employ handbrake and mpeg streamclip to perform format conversions and transcoding, although final cut’s compressor can also be effective, especially for large batch processes.  adopting kaltura has allowed us to eliminate a time intensive step in our workflow, mainly the transcoding of video for streaming.  kaltura takes whatever format we have (within reason) and does the transcoding on their cloud platform. we have also recently begun capturing 16mm sound movies using a custom built unit from the texas-based moviestuff llc.  their sniper hd telecine is composed of a modified 16mm film projector with  optical sound. video is captured by an hd camera and sent along with analog audio to a blackmagic capture card in a custom built windows pc.  the resulting video quality is quite good considering that most of the films we have are approaching 50 years old. integration of video players in library platforms as mentioned in section two in this article, one of the four requirements identified as important for our new video solution was the ability to embed content into multiple digital platforms with automatic device detection and fallback.  one of the key features we found with kaltura was the ability to support both flash and html5 in the video player, satisfying our need to provide video to devices that do not support flash.  although we did not use it, kaltura also provides an html5 video library and a video player api to allow for the development of custom players. we were satisfied with the included player and used the embed settings available on the kaltura control panel.  figure 1 indicates the options we selected.  once confirmed, kaltura generates the appropriate embed code. figure 1. kaltura embedding options. a typical kaltura embed generated code will look something like the next following lines: <div id="kaltura_player_1376588661" style="width: 400px; height: 333px;" itemprop="video" itemscope itemtype="http://schema.org/videoobject"> <span itemprop="name" content="{$video-title;}"></span> </div> <script src="http://cdnapi.kaltura.com/p/886232/sp/88623200/embediframejs/uiconf_id/16115322/partner_id/886232?autoembed=true&entry_id={$kaltura-video-id;}&playerid=kaltura_player_1376588661&cache_st=1376588661&width=400&height=333&flashvars[streamertype]=auto"> </script> our three target systems were dspace, contentdm, and omeka.  each of these provides the ability to serve up video files, but do not provide native streaming. in the ohiolink solution, developers were successful in simulating a streaming service by embedding a flash video player that served up mp4 video bitstreams as progressive downloads.  we considered re-implementing that player in our local install of dspace and perhaps upgrading it to an html5 player.  a similar technique could have been adapted to work with other platforms as well.    the problem we saw was that we would have had to continue to contend with the other limitations of progressive video with that solution.  we wanted something more robust that could, among other things, provide an optimized viewing experience at multiple target bandwidths.  kaltura provides this functionality and a relatively straightforward method to embed video in a foreign system. examples of how this code was integrated in our three digital platforms are below: contentdm provides a way to add records with a url as the primary item; this alternative allows the system to display webpages or links to streaming media servers.  in this case, we just needed to add urls for every video item with a valid kaltura video id.  then, the next step is to add a few lines of php and html code in the loader.php file or basic_view file located at: /content6/public_html/ui/cdm/default/collection/default/viewers/showlink /content4/docs/collection_alias/includes/basic_view.php <?php //echo retrieving kaltura video id if (strpos($parent->itemlinkurl,'http://www.kaltura.com/') !== false) { $kaltura-video-id = $parent->itemlinkurl; $kaltura-video-id = substr($kaltura-video-id, 23, 10); ?> <!-begin kaltura video code --> <div id="kaltura_player_1376588661" ... itemtype="http://schema.org/videoobject"> <span itemprop="name" content="<?php echo $video-title;?>"></span> </div> <script src="http://cdnapi.kaltura.com/ ... &entry_id=<?php echo $kaltura-video-id;?> ... "> </script> <!-end kaltura video code --> example: the freedom summer digital collection available at: http://digital.lib.muohio.edu/fs-videos/ dspace can handle video files as a primary bitstream within the digital object and simply sends the bitstream to the user as a download.  ohiolink’s video player provides an inline viewer by locating the bitstream’s physical location in the dspace assetstore and feeding that asset back to the flash player.  that approach requires a fair amount of hacking with the mets metadata that dspace keeps internally.  alternatively, integrating the kaltura hosted video was much simpler, requiring  the addition of  an extra metadata field in dspace with the kaltura video id and using that as a reference to embed the kaltura player.  the final step was to add some xsl and html code in the theme.xsl file located at: /webapps/xmlui/themes/your-theme/lib/xsl/aspect/artifactbrowser/ <!- adding mp4 call in itemsummaryview-dim --> <xsl:template name="itemsummaryview-dim"> <xsl:apply-templates select="./mets:filesec/mets:filegrp[@use='content']/mets:file[@mimetype='video/mp4']" mode="flashplay" /> </xsl:template> <!-begin kaltura video code --> <xsl:template match="mets:filegrp[@use='content']/mets:file[@mimetype='video/mp4']" mode="flashplay" > <xsl:variable name="v1" select="(dim:field[@element='relation' and @qualifier='isversionof'])"/> <div id="kaltura_player_1376588661" ... itemtype="http://schema.org/videoobject"> <span itemprop="name" content="{$video-title;}"></span> </div> <script src="http://cdnapi.kaltura.com/ ... &entry_id={$kaltura-video-id} ... "> </script> <!-end kaltura video code --> example: the cawc lecture series digital archive available at: http://digital.lib.muohio.edu/cawc/ omeka can also support items with urls.  the trick is to create items and add a url in one of the fields.  we chose the identifier field but other fields will work as well, especially if your metadata profile requires a different use of identifier such as for a doi or accession number. with the kaltura video id in the identifier field,  we then added the viewer code in the show.php file located at: /themes/your-theme/items <?php // get kaltura video id $kaltura-video-id = metadata('item', array('dublin core', 'identifier')); ?> <!-begin kaltura video code --> <div id="kaltura_player_1376588661" ... itemtype="http://schema.org/videoobject"> <span itemprop="name" content="<?php echo $video-title;?>"></span> </div> <script src="http://cdnapi.kaltura.com/ ... &entry_id=<?php echo $kaltura-video-id;?> ... "> </script> <!-end kaltura video code --> our current example will be launched later this year. one of the greatest benefits of using the video player sdk is that we only needed to embed 4-5 lines of code in our local library platforms.  using the html5 video library self-hosted option would require an explicit list and access for all the files available for every video; an example for two different bitrates and two ios devices is below: <video title="{$video-title}" data-entryid="{$kaltura-video-id}" data-durationhint="3271" poster="http://cdnsecakmi.kaltura.com/p/886232/sp/88623200/thumbnail/entry_id/{$kaltura-video-id}/version/100000/width/640" controls="controls" style="width: 640px;"> <source data-bitrate="2832" data-width="640" data-height="480" src="http://cdnbakmi.kaltura.com/p/886232/sp/88623200/playmanifest/entryid/{$kaltura-video-id}/flavorid/1_hfavjjt9/format/url/protocol/http/a.mp4" data-flavorid="ipad" type="video/h264"> <source data-bitrate="2408" data-width="480" data-height="360" src="http://cdnbakmi.kaltura.com/p/886232/sp/88623200/playmanifest/entryid/{$kaltura-video-id}/flavorid/1_rkjqsj99/format/url/protocol/http/a.mp4" data-flavorid="iphone" type="video/h264"> <source data-flavorid="ipadnew" type="application/vnd.apple.mpegurl" src="http://cdnbakmi.kaltura.com/p/886232/sp/88623200/playmanifest/entryid/{$kaltura-video-id}/flavorids/1_hfavjjt9/format/applehttp/protocol/http/a.m3u8"> <source data-flavorid="iphonenew" type="application/vnd.apple.mpegurl" src="http://cdnbakmi.kaltura.com/p/886232/sp/88623200/playmanifest/entryid/{$kaltura-video-id}/flavorids/1_rkjqsj99/format/applehttp/protocol/http/a.m3u8"> </video> cost of creating videos with captioning a great introduction to this topic along with some quick facts and stats on the importance of video captioning was given by the moderator of a recent panel “strategies for deploying accessible video captioning” that was part of the 2013 streaming media east conference.  creating captioned videos can be time-consuming but there are legitimate reasons to do so. four reasons and benefits include: a) complying with organizational or legislated accessibility requirements b) to provide a good alternative to videos with poor quality audio, c) to make videos text searchable to search engines, and d) to increase comprehension.  from a pragmatic perspective, our campus recently mandated that all educational video be provided with accessible captions, especially in online learning contexts.  in response, in summer 2013, we began an effort to investigate and benchmark captioning for our video content. the two main activities in creating captions are: transcribing the video and synchronizing the text and video. the transcription process remains the most labor intensive activity.  although there are now some software tools that claim to auto-transcribe speech to text, the results from these software tools are still not that accurate. therefore, outsourcing the work to video transcription services can be a viable solution; organizations such as verbal ink, speechpad, castingwords, or speakertext  charge between $1 and $2.50 per minute with a delivery service in 24-48 hours and with options for time-stamped files. of course, another solution for transcribing videos is to do it in-house.  in our case, back in 2008-2009, a team of two librarians and student assistants led the video transcription work for two video collections.  the average time for transcription was between 5-7 times the runtime of each video. one of the challenges we found with our solution is that the workflow did not include the synchronization of text and video. users have been able to view and download the transcript in pdf files, but with the new demand for captioned videos, we would like to add captions to as many videos as possible.  we are currently testing the moviecaptioner software, which allows us to import the transcripts as text files; the export options provides a set of file formats including srt, scc, sub, smil, tt, and many others. kaltura supports srt/dfxp files for video captioning. another example of a university creating captioned videos is the stanford captioning project; they outsource the transcription process and the captioning team does the synchronization part. as for the actual labor and cost of creating captioned videos, our current estimate indicates that for 1 hour of video, we would need an average of 6 hours for transcribing and an additional 3 hours for synchronizing; in other words, for every minute of video, we would need to pay 9 minutes of work.  the actual in-house cost would depend on how much of the work is done by whom -e.g. undergraduate students, graduate students, library staff, or professional librarians.  but assuming that 75% of the work is done by a student and 25% of quality control and editing is done by a librarian, it should be fair to assume that every hour of work would cost a library about $12; therefore, creating video captions for a 1 hour video -which would require about 9 hours of workcan cost about $108.  for comparison,  one of the transcription services quoted a cost for captioning a 1 hour video to be about $120 with no guaranteed turnaround time.  rush options such as “next day” or “2-3 business days” were more expensive.  the two prices are close enough that it is hard to suggest or choose one over the other. lessons learned when dealing with videos, delivering the best possible video experience is the ultimate goal for every web team.  we all have seen good evidence that several html5 compliant video players (videojs, html5video, jw player, etc.) are becoming excellent alternatives when a fallback for non-flash support is required.  perhaps the answer for the title/question of this article “is html5 the real fix for videos?” remains unanswered; but the hybrid solutions (flash and html5) can be good examples of how to deliver video content, regardless of which device or browser users are using. after all, users do not necessarily care about how their videos are being delivered or processed.  one surprising observation we made as a result of this work was that there was a relative dearth of documented work on the topic of video streaming in the library community.  as for the kaltura platform, it was definitely good to learn that it is also currently used by the internet archive to manage its digital video library to seamlessly support html5 and flash.  last but not least, for those interested in learning and contributing to the kaltura community supported project (hey code4libers), check the free community-supported site.   if you are looking to meet other open source video developers and share ideas, then kaltura.org may be the place for you. endnotes [1] ilias, i. s. h. c., munisamy, s. b., & rahman, n. a. a. (2013). a study of video performance analysis between flash video and html 5 video. in proceedings of the 7th international conference on ubiquitous information management and communication (p. 30). acm. available at: http://dl.acm.org/citation.cfm?id=2448586 accessed on september 15, 2013. about the authors elías tzoc is the digital initiatives librarian at miami university libraries. he assists the head of the center for digital scholarship to provide miami university scholars with the facilities, services, and expertise to support the creation and use of digital scholarships. his recent work includes: co-developing a framework for creating and publishing open access e-books; prototyping and implementing web interfaces for digital projects using contentdm, dspace, ojs, and omeka; researching and publishing on technical issues and open source applications for libraries; researching for new access points and mobile apps for digital library programs using the jquery mobile framework; and developing/publishing web plug-ins using php, html5, xslt, css, and jquery. john millard is the head of the miami university libraries center for digital scholarship. a graduate of the university of illinois at urbana-champaign with a master of science in library and information science. his research interests and activities focus on developing digital library systems to serve distinct user communities. he is a former nasa and usgs funded investigator working for the ohioview project, a congressionally sponsored project to further the use of u.s. civilian satellite data by the public. at miami university, he also serves as an adjunct faculty member in the interactive media studies program, teaching a course on information studies in the digital age and on the faculty of the learning technologies summer institute. subscribe to comments: for this article | for all articles 2 responses to "for video streaming/delivery: is html5 the real fix?" please leave a response below: sravani sundeep, 2020-08-28 is there any updated guide on this for textual content? what if we want to have a system that shares all our content and streams it on user device as per the request? peter murray, 2020-08-28 sravani: you might try reaching out to the authors, elías and john. they are both still at miami university: https://www.lib.miamioh.edu/about/organization/staff/ leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – using event notifications, solid and orchestration for decentralizing and decoupling scholarly communication mission editorial committee process and structure code4lib issue 58, 2023-12-04 using event notifications, solid and orchestration for decentralizing and decoupling scholarly communication the paper presents the case for a decentralized and decoupled architecture for scholarly communication. an introduction to the event notifications protocol will be provided as being applied in projects such as the international coar notify initiative and the nde-usable program by memory institutions in the netherlands. this paper provides an implementation of event notifications using a solid server. the processing of notifications can be automated using an orchestration service called koreografeye. koreografeye will be applied to a citation extraction and relay experiment to show all these tools fit together. by patrick hochstenbach, ruben verborgh, and herbert van de sompel introduction efforts to decentralize scholarly communication are motivated by concerns regarding the status quo of the web and the web-based research communication system. both have become dominated by central hubs that provide millions of users access to information but simultaneously collect and leverage usage statistics about them (pooley, 2022)(siems, 2022). many books, newspaper articles, and specialized reports investigate this questionable practice, which is known as surveillance capitalism (zuboff, 2015)(zuboff, 2019)(aspesi, 2019). since the early 2000s, concerns regarding the centralization of scholarly infrastructure by an ever-smaller group of commercial publishers have led to calls for countermeasures such as open access, open repositories, and open science (posada and chen, 2018). interoperability – protocols, not platforms – plays a fundamental role in these decentralization efforts. for example, the widely adopted oai-pmh metadata harvesting protocol (lagoze, 2002) enabled the emergence of web services such as connecting repositories (core) [1], directory of open access journals (doaj) [2], open access infrastructure for research in europe (openaire) [3], and the registry of open access repositories (roar) [4] that support the discovery of publications in thousands of repositories worldwide. a more recent interoperability effort, the event notifications in value-adding networks (event notifications) protocol (hochstenbach, 2022) (hochstenbach, 2023), proposes a distributed ecosystem of data nodes and service nodes that exchange information about life-cycle events about scholarly, museum, and archival artifacts. the goal of the event notifications protocol is to assist new decentralization efforts by glam institutions with an interoperability layer for the exchange of push-based notification messages and to allow for the decoupling of services in decentralized networks. decoupling means the implementation of services of scholarly communication (registration, certification, awareness, and archiving) (roosendaal and geurts, 1998) in separate service nodes instead of consolidation in a one-stop shop. the coar notify initiative [5] employs the event notifications protocol, connecting institutional repositories (data nodes) with overlay journals [6] and peer-review services (service nodes). the nde-usable program [7] of the dutch digital heritage network aims to redesign how cultural heritage institutions and applications exchange data (vander sande, 2022). event notifications provides an interoperable communication layer between museum and archival objects and service nodes that provision indexation, annotation, and full-text analysis. by far, the coar notify initiative, generously funded by arcadia [8], has the most advanced implementation of event notifications involving project partners such as biorxiv [9], medrxiv [10], dataverse [11], zenodo [12], pci peer community in [13] and open journal systems [14]. the coar notify use cases depend on human intellectual analysis and, thus, by definition, are dependent on manual steps to fulfill services requested through notifications. nde-usable use cases are more semi-automated and rely on asynchronous processing of data. in both use cases, orchestration can assist the actors in the network and provide the data nodes and service nodes with the correct information at the right time. to illustrate the potential of the event notifications protocol, we provide in this paper a short introduction to the protocol and provide an implementation of event notifications using solid [15], the web decentralization project of berners-lee. the decentralization and decoupling of scholarly services require some form of orchestration to automate interactions between the actors and all other components in the event notifications network. we introduce an orchestrator as an automated agent to assist the network in these tasks. koreografeye is a prototype orchestrator employed in a citation notification experiment about which we will report. event notifications in value-adding networks an event notifications network exposes life-cycle information (see below) about artifacts within its decentralized network. in a scholarly network, artifacts, in general, are web resources that include (but are not limited to) pre-prints, datasets, reviews, published versions, comments, and web archives. in a glam context, artifacts can include museum objects, archival collections, and other digitized materials. life-cycle events that are of interest are those that demonstrate the (scholarly) value of artifacts. in this sense, value is not a monetary value but contextual information that allows one to assess the trustworthiness of an artifact. artifacts with values have a higher chance of being recognized by the scholarly community, vetted for academic honesty, and become part of an official record of a researcher, university, archive, or museum. examples of life-cycle information for artifacts are information about when and where they were made available online, registered in a repository, peer-reviewed, published in a journal, and indexed in a scholarly database. the associated values are claims of precedence of ideas, proof of being part of the institutional output and establishing the validity of a scholarly claim by peers and the wider academic audience. to illustrate the connection between an artifact, life-cycle events, and values, we take the example of the researcher alice, who publishes a preprint on her website. the fact that alice publishes this preprint is already an essential life-cycle event: alice claims precedence of a scholarly finding. when alice registers this preprint to a timestamp service, she can link her artifact to a trusted timestamp. the value of both events is a trusted claim of precedence for her ideas. next, alice decides to register this preprint in the institutional repository of her university. the institutional repository creates a new record that links to her preprint. the lifecycle event is the registration of the preprint in the repository. this event’s value is establishing her preprint as part of the university output. an event notification could inform alice of this fact. next, alice requests a review of the preprint in a decentralized peer service such as prereview [16], which, after some weeks, results in a new life-cycle event: a review of her preprint is published. the linkage between her preprint and the review establishes the validity of her scholarly claim. in the next stage, alice’s preprint gets picked up by bob, another researcher, who cites her artifact in his paper (another artifact). the lifecycle event, in this case, is the citation. alice could be informed of this event through an event notification and create a bi-directional link to bob’s artifact. in the current scholarly system, discovering these scholarly added values requires a lot of heuristics, post-factum processing, mining massive datasets, and screen scraping websites (e.g., cabernac et al. efforts to discover preprint/publication links (cabanac, 2021)). the process of finding these life-cycle events and the communication about these events is the motivation for event notifications. to improve on the status quo, event notifications introduces a messaging protocol as a means to orchestrate life-cycle events or exchange information about them. different actors in an event notifications network have roles depending on the information they make available and the services that they provide: data nodes are network nodes that store research output defined as artifacts. service nodes are network nodes that process or augment artifacts and add a (scholarly) value to them. the resulting resources of this process are called service results. for instance, in the example of alice, her website is the data node, and her preprint is an artifact. service nodes are the timestamp service, institutional repository, and prereview service. service results are the trusted timestamp record, registration record in the institutional repository, and prereview review. data nodes and service nodes in the event notifications network send and receive notifications orchestrating life-cycle events and exchanging information about them. for instance, when alice announces the preprint on her website it triggers a notification. when she requests a timestamp, her data node sends a notification to a timestamp service node. when the institutional repository registers alice’s preprint, it sends a notification to alice’s data node. the data node can use these notifications to start scholarly workflows such as creating bidirectional links, triggering new publication workflows, and creating public shared event logs containing life-cycle information. the event notifications specification [17] describes many possible usage patterns for sending notifications: one-way communication patterns are a one-shot notification that expresses information such as “i did this activity on an artifact.” two-way communication patterns describe a more elaborate communication style regarding providing a service for an artifact. the communication style depends on the use case and the domain implementing the specification. for instance, coar notify [18] uses two-way communication patterns to describe how institutional repositories, peer-review services, and overlay journals can communicate about requesting services and responding with value-added results. harvard university library uses one-way communication patterns to communicate about data-literature links between artifacts in a dataverse data repository and their dspace institutional repository. the event notifications protocol of choice to exchange life-cycle information is linked data notifications (ldn) with activitystreams2 (as2) payloads, which will be introduced in the next section. activitystreams2 notifications in an event notifications network, data nodes and service nodes exchange information about life-cycle events in the form of json-ld notifications using a profile of the activitystreams2 (as2) vocabulary (snell and prodromou, 2017). this format has recently gotten some attention due to the popularity of activitypub (lemmer-webber, 2018) implementations such as mastodon [19] (a decentralized version of x, formerly known as twitter), friendica [20] (a decentralized version of facebook) and peertube [21] (a decentralized version of youtube). activitypub is the decentralized network protocol for delivering notifications. as2 is the data format for the payloads of such notifications. whereas the focus of activitypub notifications is the actions of humans and information about humans, in an event notifications network the notifications are about artifacts or service results about artifacts. event notifications creates a profile of as2 that communicates life-cycle events for artifacts: as2 notifications sent by data nodes focus on requesting a value-adding service for an artifact. as2 notifications sent by service nodes focus on announcing a value-adding service result about an artifact on a data node. an important fact about the event notifications as2 notifications is that they promote linking to information about artifacts and service results rather than providing inline information about them. as such, event notifications as2 notifications are not a metadata transportation mechanism. each as2 notification contains enough information to route a life-cycle event to the correct destination, but not more. other follow-your-nose discovery mechanisms, such as signposting (van de sompel, 2023), can be used to learn more about artifacts and service results. the example below shows a fragment of an event notifications as2 notification in the json-ld format: { "@context": [ "https://www.w3.org/ns/activitystreams", "https://purl.org/coar/notify" ], "id": "urn:uuid:6e5faf88-a7f1-47a4-b087-77345ebff495", "type": [ "announce", "coar-notify:reviewaction" ], "actor": { "id": "https://nortport.edu/profile/bob#me" }, "origin": { "id": "https://nortport.edu/repository" }, "context": "http://riverpool.edu/artifacts/modal-surfaces-in-rdf", "object": { "id": "http://nortport.edu/review/modal-surfaces-in-rdf", "type": "document" }, "target": { "id": "https://riverpool.edu/profile/alice#me" } } listing 1: a truncated example of an event notifications as2 notification in json-ld expressing bob’s announcement of a new review artifact (ar nortport.edu) to alice (at riverpool.edu) the property id contains a unique identifier for a notification message. the property type specifies a life-cycle event (in this case, an announcement of an endorsement that bob sends to alice). the property actor describes the agent that sends the notification, in this case, bob (we don’t show all the sub-properties for brevity). the property target describes the intended audience of the notification, in this case alice. the property origin describes the software agent used to send notifications. the property object describes bob’s artifact. the property context describes alice’s artifact. the next section introduces linked data notifications (ldn) to exchange as2 notifications between data and service nodes. linked data notifications in its simplest form, ldn is an http protocol to send and receive rdf messages (such as the event notifications as2 notifications) using a post request to an ldn receiver (the endpoint that implements ldn) (capadisli and guy, 2017). in event notifications, data and service nodes are ldn receivers of as2 notifications. the network endpoint that receives notifications is called the ldn inbox. implementing an ldn inbox with basic functionality (append-only, without authentication or authenticated read functionalities) is relatively straightforward. any http endpoint receiving a post request with a json-ld body and returning an http 201 created status code with the network location of the accepted notification is already an ldn inbox. an ldn sender is an application that can send rdf messages to an ldn inbox. in its most basic form, an ldn sender should be able to issue an http post request to the location of an ldn inbox. in the example below, we use a curl command to send the json-ld notification stored in the example.jsonld file to an ldn inbox located at https://riverpool.edu/inbox/ curl -h "content-type: application/ld+json" \ --data-binary @example.jsonld \ https://riverpool.edu/inbox/ in this paper, we will create a more advanced implementation of the ldn inbox and the ldn sender to demonstrate the potential of the event notifications protocol. we will implement the ldn inbox with append and read capabilities as defined by the solid protocol. using the solid protocol, we can decouple the logic of scholarly web services from implementing the ldn inbox with the help of automated agents. solid we explained event notifications as2 notifications and the ldn protocol in the previous two sections. this section will introduce solid to implement ldn inboxes with append and read capabilities. in a typical setup, creating an ldn sender and ldn receiver with basic http post functionality would be all that is required to implement the event notifications protocol. internally, an implementation would require some logic to act on incoming as2 notifications and send out new as2 notifications. projects such as coar notify provide guidelines for setting up such systems. many pilot partners are already starting to experiment with ldn inboxes in local instances of institutional and data repositories such as dspace and dataverse and services such as hal [22] (the french research archive), prereview [23], peer community in [24], and many others [25]. in the andrew w. mellon-funded researcherpod project [26], we take a more extreme decentralization approach and investigate what would happen if every researcher employed their own data node instance (or where libraries would provide such services). in the vision of the researcherpod (van de sompel, 2018), every researcher has access to a solid pod (capadisli, 2022) on which they store artifacts. the solid project (“solid”) was started in 2015 by tim berners-lee as a vision of independent data storage and services. the name “solid” can mean many things depending on the context. it is the name of the web decentralization movement, a set of protocols [27], and a bunch of implementations [28]. the protocols proposed by solid are similar to the fedora commons project [29] but are targeted to different use cases. the central use-case of fedora commons is providing a backend service on top of which institutions can build institutional repositories, data repositories, and digital archives. the fedora commons provides out-of-the-box support and much documentation about data models and protocols for glam use cases. solid is targeted to personal storage use cases but with direct access to the web. solid servers do not have out-of-the-box specialized workflows for data archiving purposes. but they do come with a global authentication system. decentralized solid clients, such as single-page applications, provide all interactions with the solid server. web identities [30] provide a global identity, solid-oidc [31] a global authentication mechanism, and web access control [32] a global access mechanism. each solid user can decide who has access to what data and can interact with web identity with solid installations worldwide. the drawback of solid is that this flexibility is not without complexity. creating authenticated sessions with solid-oidc requires specialized software libraries currently only available in javascript and java. also, solid, like fedora commons, doesn’t come with a web front-end, which needs to be created by the community. the advantage of solid is that it can be easily installed and doesn’t require any external dependencies such as database systems or full-text indexes that need to be maintained. in our experiments, we employ a solid server to implement an ldn inbox with the functionality to provide (authenticated) web agents append and read access to notifications in the inbox. a decoupled orchestrator (see next section) implements the logic to process notifications in the ldn inbox. we use the community solid server (css) [33] implementation of solid, which has the advantage that the installation doesn’t require any other dependencies than a relatively recent node.js version with the npm package manager. we can install a css 6.1.0 version with the following command: npm install -g @solid/community-server@6.1.0 note: the command-line examples in this paper were created for a macos/linux system. the syntax should be the same on modern windows systems, starting with windows 10 and server 2019. to start the css instance in a new data_node folder, execute the following commands: mkdir data_node cd data_node community-solid-server -c @css:config/file-no-setup.json the file-no-setup configuration should only be used for testing and demonstration as it exposes the root container for the world with read and write access rights. we refer to the css website for documentation about more secure setup choices [34]. in the new data_node folder, we can create an ldn inbox with the following command: mkdir inbox the curl command of the previous section can now be used to send an event notifications notification to this ldn inbox with network location http://localhost:3000/inbox/. read the contents of the ldn inbox by issuing an http get request against the ldn inbox: curl http://localhost:3000/inbox/ the response will be a turtle document containing the ldn inbox. the turtle document below shows a fragment of the response: <> a ldp:container, ldp:basiccontainer, ldp:resource . <ec8267ac-e4af-4233-9786-9192801a4baf> a ldp:resource . this ldn inbox contains one resource with the relative url ec8267ac-e4af-4233-9786-9192801a4baf. ldp, linked data platform [35], is a specification for reading and writing rdf data using http. ldn is a specialized subset of the ldp protocol. a ldp:resource is a class to describe resources (similar to a file). a ldp:container is a class to describe a collection of resources (similar to a folder). issuing an http get request on this url, we can retrieve the event notifications as2 notification: curl http://localhost:3000/inbox/ec8267ac-e4af-4233-9786-9192801a4baf to get programmatic access to solid servers open source javascript [36] and java [37] solid clients are available. at ghent university, we require a lot of command-line processing of notifications and created the bashlib [38] client to interact with solid servers. install bashlib with the command: npm install -g solid-bashlib with this command, a new sld executable is available that provides many bash command equivalents (such as ls, cp, remove, tree) to interact with an (authenticated) solid server. to list the contents of our ldn inbox issue: sld ls http://localhost:3000/inbox/ to read the contents of a resource, execute the sld cat command: sld cat http://localhost:3000/inbox/ec8267ac-e4af-4233-9786-9192801a4baf orchestration in a decentralized event notifications network, the data and services nodes do not resort to one place but are intentionally distributed. as a result, the actors in the network need to communicate with multiple service nodes to execute a desired scholarly workflow. some of these workflows are near instantaneous, and some can take several days or months to finalize. an orchestrator component automates part of the required interactions to avoid excessive manual work by the data node and service node maintainers to keep up with the received and sent notifications. in many ways, orchestration resembles the function of a rule engine employed by email agents such as outlook rules or mozilla thunderbird filters. in the researchpod project, we decouple, using an orchestrator, the logic of processing notifications from the ldn inbox implementation (the solid pod in our case). the solid protocol provides read and write capabilities for ldn inboxes on which an orchestrator can act. in this way, some valuable characteristics become available for the orchestrator as an automated agent: an orchestrator can communicate with one or more remote ldn inboxes. an orchestrator is an ldn receiver and an ldn server of event notifications as2 notifications. an orchestrator uses a collection of rules to define actions that trigger state changes in the network. in the next section, we introduce a prototype orchestrator, koreografeye, capable of providing the functionalities described in this section. this orchestrator will then be used in an experiment to automatically notify data nodes in an event notifications network about citations to their stored artifacts. koreografeye figure 1. the koreografeye architecture koreografeye is an orchestrator built for the researchpod project to provide a command-line solution for orchestration. we chose a collection of command-line tools to allow for integration with larger-scale projects we did in the past using apache nifi. our solution, however, can run standalone. figure 1 shows the architecture of koreografeye. the top left depicts an ldn inbox that the orchestrator monitors. we use the bashlib command-line client and a cron job to watch the ldn inbox and download notifications to a local input folder. in step (1), the orch component processes incoming as2 notifications with a rule engine. zero or more policy documents describe the actions to follow when receiving an as2 notification. the output (2) of the rule execution is an augmented as2 notification resource that contains enough technical metadata for a pol component to execute (3) the actions defined by policies. policy actions include instructions such as “send an as2 notification”, “send an email”, “create a mastodon ’toot’” or “update a resource on a solid pod.” notation3 notation3 [39] is our first attempt to create a policy language. given the rdf nature of as2 notifications and the declarative way of writing rules in the notation3 language, this seems a reasonable choice. notation3 rules are combinations of if-then statements expressed using an extension of rdf turtle. the example below shows a simple rule: @prefix as: <https://www.w3.org/ns/activitystreams#> . @prefix : <urn:foo-bar:> . { ?id a as:announce . } => { ?id a :coolnotification . } . the first curly brackets { } describes the if-part of the policy: “if the input rdf document has a subject ?id of type as:announce“. the second curly brackets after => describes the then part of the policy: “then, add to the subject ?id the (sub)type :coolnotification“. the ?id binds the matching subject of the if part. in koreografeye, each policy adds in the then part rdf triples that describe an action. for instance, the policy below requests that every as2 notification of type as:announce with subtype ex:citation should result in an email: @prefix as: <https://www.w3.org/ns/activitystreams#> . @prefix : <urn:foo-bar:> . @prefix ex: <http://example.org/> . @prefix pol: <https://www.example.org/ns/policy#> . @prefix fno: <https://w3id.org/function/ontology#> . { ?id a as:announce, ex:citation . } => { ex:myemailpolicy pol:policy [ a fno:execution ; fno:executes ex:sendemail ; ex:to "patrick.hochstenbach@ugent.be" ; ex:subject "you got a citation!" ; ex:body "you got a new citation" ] . }. implementation the implementation of koreografeye was written in typescript (javascript) and runs on node.js (https://nodejs.org). an installation of node.js includes the npm and npx commands we will use in the rest of this paper. to run the koreografeye examples a node.js version v16.20.0 or more recent is advised. to install koreografeye, issue the following command: npm install -g koreografeye in our demonstration, we will use the as2 notification example from the start of this paper as the example.jsonld file in an input folder. an orch run will apply the mypolicy.n3 policy on the example.jsonld as2 notification: npx orch --info --keep --in input --out output mypolicy.n3 the --info option provides verbose information about the orchestration run, the --keep option tells the orchestrator not to delete the input file after processing, the --in input option is the input folder that contains the as2 notifications, the --out option is the location of an output folder, and mypolicy.n3 is the notation3 policy that is applied to the incoming as2 notification. a (truncated) example of the mypolicy.n3 rule is given below [40]: { # we need an announce ?id a as:announce . # we need an object ?id as:object ?object . } => { # send a notification to an indexing service ex:mysendnotificationdemo pol:policy [ a fno:execution ; fno:executes ex:sendnotification ; ex:to <http://indexing.for.us/inbox/> ; ex:notification [ a as:offer ; as:actor <https://repository.edu/profile/card#me> ; as:origin <https://repository.edu/orchestator/card#me> ; as:object ?object ; as:target <http://indexing.for.us/profile/card#me> ] ] . } . listing 2: a truncated example of a policy in notation3, expressing that as2 announce notifications should be forwarded to an indexation service node (indexing.for.us) using an as2 offer notification, listing2 describes a policy rule that instructs the orch that every incoming as2 notification of type announce with an object should be forwarded to a hypothetical indexation service http://indexing.for.us. the output of the previous orch run is an augmented rdf resource in the output directory, which contains the original as2 notification plus technical metadata to instruct the pol policy executor to send out a new as2 notification to the indexation service node. in this case, the triples that describe the ex:mysendnotificationdemo is added to the as2 notification. the output file output/example.ttl contains the original as2 notification with the added triples from the orch process. the pol policy script will execute the results of the previous step: npx pol --info --keep --in output when executed, the pol will result in an error because the http://indexing.for.us/inbox/ is a fake address. had we used a real ldn inbox address, an as2 notification would have been delivered by the orchestrator. implementing in koreografeye a new type of policy execution step, for instance, updating a local institutional repository would require writing a koreografeye plugin [41] in javascript or typescript. the plugin gets as input the as2 notification plus the configured parameters from the notation3 policy and can then execute any required call to the api-s for your repository. citation notification experiment in the remainder of this paper, we combine the above mentioned techniques into a citation notification experiment that ghent university library and imec idlab conducted in the summer of 2023. the experiment investigates how an orchestrator can notify data nodes in an event notifications network about new citations to their stored artifacts. we imagine a repository a that holds artifacts and is the source of citations and repositories b, c, d, etc., that hold artifacts that are the targets of the citations. repository a would like to execute an automated workflow to notify repositories b, c, d, etc., about these citations. in this experiment, we don’t necessarily investigate the feasibility of citation extraction (a research topic on its own) but the potential of using a network of decentralized services and automated agents to inform data nodes about citations. figure 2. the citation network experiment architecture experimental setup figure 2 displays the architecture of the citation notification experiment. on the left side, in blue, institutional repository a (data node) contains the source publication artifacts. we would like to send event notifications as2 notifications to a (demonstration) network of external repositories b, c, d, etc. (data nodes) to inform them that an artifact in repository a has cited artifacts in their repository. to implement this workflow, we introduce: a network of ldn inboxes in the form of solid css instances that operate on behalf of the repositories a, b, c, d, etc. two service nodes: a citation extractor service node that, given an event notifications as2 notification pointing to an artifact, will extract the citations from the artifact, publish the extracted citations as a service result document, and send an event notifications as2 notification with the link to this service result back to the data node that requested the citation extraction. a citation relay service node that, given an event notifications as2 notification pointing to a service result of extracted citations, tries to discover the ldn inboxes of the cited artifacts and sends to each of them an event notifications as2 notification about the discovered citations. an orchestrator that: contains an oai-bridge that transforms the pull-based oai-pmh protocol of repository a into the push-based ldn+as2 protocol of the event notifications network. in this way, every newly added artifact in the institutional repository a will generate a new event notifications as2 notification that triggers the orchestrator. contains policies describing how to interact with the citation extractor and citation relay services to drive the experiment. experimental implementation to explain the experiment, we will provide some details below about the execution steps in fig 2. in step (1), the orchestrator uses an oai-pmh bridge to inform itself about new artifacts in repository a. the oai-pmh bridge is a component that pulls the oai endpoint of repository a and generates push as2 notifications to the orchestrator. for each as notification, the orchestrator consults its policies. one of the policies describes a rule that demands that for every new artifact in repository a an event notifications as2 offer notification should be sent to the citation extractor service. in step (2), the citation extractor receives the event notifications as2 offer and starts a citation extraction process for the offered artifact. we skip the exact details of this process but mention that we are experimenting with citation extraction tools such as cermine [42] and opencitations [43]. after some processing time, this process results in an rdf turtle document containing all the extracted citations. every citation contains an http link to the citing artifact and an http link (such as a doi) to the (remote) cited artifact. in step (3), a link to the service result of step (2) is sent as an event notifications as2 announce notification to the ldn inbox of repository a. the orchestrator is triggered again, consults its policies, and finds a policy that demands that citation service results should be forwarded as an announce notification to a citation relay service. in step (4), the citation relay receives the event notifications as2 announce notification and starts a process to discover the ldn inbox for every citation link in the service result. the citation relay resolves ever http link for each citation in search for http headers that describe the location of an ldn inbox. in our scholarly networks, we can’t expect any implementations of ldn inboxes to be available yet. when no external ldn inbox is found, a local demonstration network of solid css servers is used as an ldn inbox. in step (5), the citation relay service sends an event notifications as2 announce notification to the discovered ldn inboxes, informing them about a relationship between an artifact in repository a and a citation of an artifact in their repository. first results we conducted the citation notification experiment on samples of publications in the biblio – academic bibliography institutional repository of ghent university library [44], which acts as repository a. for this repository, random samples of 1, 10, 32, and 100 artifacts were harvested that contain at least a doi link as metadata (about 40% of recent biblio records contain such doi-s). these artifacts were presented to the orchestrator as as2 notifications and offered to the network of service nodes as described above. for each sample, the total time was measured to trigger the orchestrator, searching citations in the citation extraction service using the opencitations api [45], notifying the biblio ldn inbox about the results, forwarding the service results to the citation relay service, discovering remote ldn inboxes and the delivery of event notifications as2 notifications to the demonstration network of repository data nodes. the total execution time is shown in fig 3, demonstrating a near-linear scalability of the solution. the total time is based on using a single thread of execution in all components. figure 3. total time in seconds of extracting citations (using the opencitations api), discovering ldn inboxes for remote data nodes, and sending event notifications as2 notifications to these inboxes for n=1,10,32 and 100 random sample artifacts with doi-s. we tried different citation extraction strategies, such as using cermine. however, our experiments demonstrated that 80% of the execution time is consumed by dereferencing doi-s while trying to discover ldn inboxes. this was a surprise because we assumed that most of the time would be spent extracting citation data from pdf files. finding obstacles in doi resolution is not new and was already reported by researchers from los alamos national laboratories in 2020 (klein and balakireva, 2020). we worked with a relatively small sample of artifacts, but for this small sample of 100 artifacts (using opencitations doi resolution), we could find 1896 citations that were distributed over 163 data nodes. in other experiments, we used the cermine software to locally extract citation information with mixed results. in an ideal world, these citations shouldn’t have to be extracted from binary formats such as pdfs and directly made available in a machine-readable format (martin and henrich, 2022). the software of this experiment is available in open source as oai-bridge [46], citationextractionservice [47], and citationrelayservice [48]. the resulting event notifications as2 notifications can be downloaded from zenodo at https://zenodo.org/records/10017325. conclusion and future work we covered quite some ground to describe the raison d’être of event notifications and the notification protocols it employs for communication between decentralized nodes that each fulfill decoupled versions of scholarly functions. a possible decoupled implementation of an event notifications inbox was demonstrated using a solid css service, which provides out-of-the-box web-ready authenticated ldn inbox functionalities. the orchestrator was introduced as an active component that uses a rule-based policy system to automate scholarly value chains. the citation notification experiment is one example of how this setup can be used for real-world use cases. the potential of event notifications doesn’t stop here. communicating about value-adding events is one thing, but making value-adding events publicly available and discoverable provides new opportunities for future applications. if all data nodes and service nodes would make public event logs available that describe how scholarly artifacts got registered, certified, published, and archived, it would not only benefit the bibliometrics researcher who gets more transparency in the scholarly process but would also provide the public to better tools to differentiate facts from fiction. a web user who discovers any article online could be provided with an interface to discover the event logs of all value-adding services we collected and made available in the event notifications network and discover the academic values of the information at hand. thinking about scholarly communication nodes in a decentralized and decoupled fashion could create a new open ecosystem involving preprint servers, journals, independent content-vetting initiatives, and curation services that provide more multidimensional signals for papers and avoid the current conflation of trust, quality, and impact (sever, 2023). acknowledgments this work is funded by the andrew w. mellon foundation (grant number: 1903-06675) and supported by solidlab vlaanderen (flemish government, ewi, and rrf project vv023/10). the authors would like to thank the erfgoedpod project (miel vander sande) and imec (ruben dedecker, wout slabbinck, and jos de roo) for joint discussions on the requirements for decentralized orchestration in both of our projects. the authors would also like to thank the coar notify (kathleen shearer, paul walk, martin klein) for involving them from the outset, allowing them to provide input and gain valuable insights in the scholarly communication process. about the authors patrick hochstenbach (webid:https://patrickhochstenbach.net/profile/card#me) is a digital architect in the library automation team of ghent university library, belgium. in ghent, he was involved in the openurl/sfx project and created the first version of the sfx linking server. at los alamos national laboratory, he participated in research projects on federated digital repository architectures. at lund university, sweden, he created large-scale full-text indexation services. and again, at ghent university, he is currently doing his ph.d. research in computer science on applying decentralization techniques for scholarly communication. ruben verborgh (webid:https://ruben.verborgh.org/profile/#me) is a professor of decentralized web technology at idlab of ghent university – imec and a visiting fellow at the oxford martin school within the university of oxford. he is the head of data interoperability at inrupt and an advisor to other companies. from his hybrid academic and industrial perspective, his professional mission is to support solid in inspiring, transforming, and reshaping our data-driven society. herbert van de sompel (webid:https://hvdsomp.info/#i) is a research fellow at data archiving and networked services (dans) in the netherlands and a visiting professor at the internet technology & data science lab of ghent university. he has played a major role in creating the open archives initiative protocol for metadata harvesting (oai-pmh), the open archives initiative object reuse & exchange specifications (oai-ore), the openurl framework for context-sensitive services (ansi/niso z39.88-2004), the sfx linking server, the bx scholarly recommender service, info uri (rfc 4452), open annotation (w3c community group specification), resourcesync (ansi/niso z39.99-2014), memento “time travel for the web” (rfc 7089), robust links, and signposting the scholarly web. references aspesi, claudio, and allen, nicole s., and crow, raym, and daugherty, shawn, and joseph, heather, and mcarthur, joseph t. w., and shockey, nick . “sparc landscape analysis.” sparc, 29 march 2019, https://infrastructure.sparcopen.org/landscape-analysis cabanac, guillaume, and oikonomidi, theodora, and boutron, isabelle. “day-to-day discovery of preprint–publication links.” scientometrics, vol. 126, 2021, pp. 5285–5304, https://doi.org/10.1007/s11192-021-03900-7 capadisli, sarven, and bernes-lee, tim, and verborgh, ruben, and kjernsmo, kjetil. “solid protocol.” solid project, 2022, https://solidproject.org/tr/protocol capadisli, sarven, and amy guy. “linked data notifications.” w3c, 2 may 2017, https://www.w3.org/tr/ldn/ hochstenbach, patrick, and vander sande, miel, and dedecker, ruben, and walk, paul, and klein, martin, and van de sompel, herbert. “event notifications in value-adding networks.” event notifications in value-adding networks, 6 september 2023, https://www.eventnotifications.net hochstenbach, patrick, and van de sompel, herbert, and vander sande, miel, and dedecker, ruben, and verborh, ruben. “event notifications in value-adding networks.” linking theory and practice of digital libraries, vol. 13541, 2022, p. 133, https://doi.org/10.1007/978-3-031-16802-4_11 klein, martin, and lyudmila balakireva. “on the persistence of persistent identifiers of the scholarly web.” digital libraries for open knowledge. tpdl 2020., vol. 12246, 2020, https://doi.org/10.1007/978-3-030-54956-5_8 lagoze, carl, and van de sompel, herbert, and nelson, michael l., and warner, simeon. “protocol for metadata harvesting – v.2.0.” open archives initiative, 2002, http://www.openarchives.org/oai/openarchivesprotocol.html lemmer-webber, christine, and tallon, jessica, and shepherd, erin, and guy, amy, and prodromou, evan. “activitypub.” w3c, 23 january 2018, https://www.w3.org/tr/activitypub/ martin, leon, and andreas henrich. “rdftex: knowledge exchange between latex-based research publications and scientific knowledge graphs.” linking theory and practice of digital libraries. tpdl 2022., vol. 13541, https://doi.org/10.1007/978-3-031-16802-4_3 pooley, jeff. “surveillance publishing.” the journal of electronic publishing, vol. 25, no. 1, 2022, https://journals.publishing.umich.edu/jep/article/id/1874/ posada, alejandro, and george chen. “inequality in knowledge production: the integration of academic infrastructure by big publishers.” elpub 2018, 2018, https://hal.archives-ouvertes.fr/hal-01816707 roosendaal, hans, and peter geurts. “forces and functions in scientific communication: an analysis of their interplay.” cris97, 5 january 1998. https://perma.cc/5hym-bekf sever, richard. “biomedical publishing: past historic, present continuous, future conditional.” plos, 3 october 2023, https://doi.org/10.1371/journal.pbio.3002234 siems, renke. “das lesen der anderen: die auswirkungen von user tracking auf bibliotheken.” o-bib. das offene bibliotheksjournal, vol. 9, no. 1, 2022, https://www.o-bib.de/bib/article/view/5797 snell, james m., and evan prodromou. “activity streams 2.0.” w3c, 23 may 2017, https://www.w3.org/tr/activitystreams-core/ vander sande, miel. “use cases & business processes.” 2022, https://erfgoedpod.github.io/usecases/ van de sompel, herbert. “scholarly communication: deconstruct & decentralize?” youtube, cni: coalition for networked information, 2 january 2018, https://www.youtube.com/watch?v=o4nue-6ln-8 van de sompel, herbert, and klein, martin, and jones, shawn, and nelson, michael l., and warner, simeon, and devaraju, anusuriya, and huber, robert, and steinhoff, wilko, and tykhonov, vyacheslav, and boruta, luc, and meijers, enno, and soiland-reyes, stian, and wilkinson, mark. “fair signposting profile.” signposting the scholarly web, 2023, https://signposting.org/fair/ zuboff, shoshana. the age of surveillance capitalism: the fight for a human future at the new frontier of power. profile books, 2019. https://doi.org/10.1080/17530350.2019.1639068 zuboff, shoshana. “big other: surveillance capitalism and the prospects of an information civilization.” journal of information technology, vol. 30, no. 1, 2015, https://journals.sagepub.com/doi/10.1057/jit.2015.5 endnotes [1] https://core.ac.uk/ [2] https://www.doaj.org [3] https://www.openaire.eu/openaire-portal [4] http://roar.eprints.org [5] https://www.coar-repositories.org/notify/ [6] an online journal composed out of links to self-archived preprints of postprints. see also https://en.wikipedia.org/wiki/overlay_journal [7] https://netwerkdigitaalerfgoed.nl/bruikbaar/ [8] https://www.arcadiafund.org.uk [9] https://www.biorxiv.org [10] https://www.medrxiv.org [11] https://dataverse.org [12] https://zenodo.org [13] https://peercommunityin.org [14] https://pkp.sfu.ca/software/ojs/ [15] https://solidproject.org [16] https://prereview.org [17] https://www.eventnotifications.net [18] https://notify.coar-repositories.org [19] https://joinmastodon.org [20] https://friendi.ca [21] https://joinpeertube.org [22] https://hal.science [23] https://prereview.org [24] https://peercommunityin.org [25] see 1st phase partners at https://www.coar-repositories.org/notify/ [26] https://knows.idlab.ugent.be/projects/mellon/ [27] https://solidproject.org/tr/ [28] https://solidproject.org/developers/tutorials/getting-started [29] https://fedora.lyrasis.org [30] https://www.w3.org/2005/incubator/webid/spec/identity/ [31] https://solidproject.org/tr/oidc [32] https://solidproject.org/tr/wac [33] https://github.com/communitysolidserver/communitysolidserver . in our experiment we use css version 6.1.0. this version requires nodejs version 16.14.0 be installed or better. an easy way to install a nodejs version is by using the node version manager (nvm) script https://github.com/nvm-sh/nvm. [34] see, e.g., the css configuration generator https://communitysolidserver.github.io/configuration-generator/v6/ [35] https://www.w3.org/tr/ldp/ [36] https://docs.inrupt.com/developer-tools/javascript/client-libraries/authentication/ [37] https://github.com/inrupt/solid-client-java [38] https://github.com/solidlabresearch/bashlib [39] https://w3c.github.io/n3/spec/ [40] see https://gist.github.com/phochste/08a3898bd59ce7c2c80bee1ae8a38aa0 for the full example [41] see: https://github.com/eyereasoner/koreografeye-plugin for example plugins [42] http://cermine.ceon.pl/index.html [43] http://opencitations.net [44] https://biblio.ugent.be/ [45] http://opencitations.net/index/api/v1#/references/%7bdoi%7d [46] https://github.com/mellonscholarlycommunication/oai-bridge [47] https://github.com/mellonscholarlycommunication/citationextractorservice [48] https://github.com/mellonscholarlycommunication/citationrelayservice subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – glimir: manifestation and content clustering within worldcat mission editorial committee process and structure code4lib issue 17, 2012-06-01 glimir: manifestation and content clustering within worldcat the glimir project at oclc clusters and assigns an identifier to worldcat records representing the same manifestation. these include parallel records in different languages (e.g., a record with english descriptive notes and subject headings and one for the same book with french equivalents). it also clusters records that probably represent the same manifestation, but which could not be safely merged by oclc’s duplicate detection and resolution (ddr) program for various reasons. as the project progressed, it became clear that it would also be useful to create content-based clusters for groups of manifestations that are generally equivalent from the end user perspective (e.g., the original print text with its microform, ebook and reprint versions, but not new editions). lessons from the glimir project have improved oclc’s duplicate detection program through the introduction of new matching techniques. glimir has also had unexpected benefits for oclc’s frbr algorithm by providing new methods for identifying outliers thus enabling more records to be included in the correct work cluster. by janifer gatenby, richard o. greene, w. michael oskins and gail thornburg introduction the project glimir (global library manifestation identifier) was initiated by stu weibel of oclc research to create a reliable manifestation identifier that could be freely diffused globally for common usage. whilst a series of iso manifestation level identifiers already exist (isbn for text, issn for serials, ismn for music, isrc for sound recordings, v-isan for audio-visuals and doi for electronic texts) studies of worldcat revealed that they were only present in 30 percent of library records. iso identifiers are applied by trade organisations to resources of commercial interest. libraries also collect cultural resources that are outside the scope of iso identifiers. (gatenby 2008). “we need identifiers that are globally scoped, business neutral, usable by all, and managed in either a centralized or federated manner. to the extent that such identifiers are canonical – that is, become the dominant identifier for a given asset, they increase the ‘uri equity’ for library assets and will strengthen the library presence on the web.” (weibel 2008) worldcat, as the largest union catalogue, including more than 256 million records representing the holdings of more than 72,000 libraries world-wide (oclc 2012) was proposed as the best source for creating and, just as important, maintaining a globally valid identifier. however, although worldcat record identifiers once had a nearly one-to-one relationship with manifestations, this is no longer the case. the main reason for this is that since oclc released its open worldcat and worldcat.org platforms in 2003, an increased number of libraries world-wide provided records for loading, resulting in parallel records describing the same resource but in different languages and scripts. before manifestation identifiers could be created, it was necessary to create clusters of records describing the same manifestation of a work. early internal studies examined some work sets with 30 or more records and noted not only multiple records for the same resource in different languages of cataloguing, but also records described in the same language where oclc’s duplicate detection program was not confident to demand de-duplication and merge. it was then noticed that from the end user’s viewpoint, it would also be helpful to cluster records where the content was the same, e.g. original text, microform, electronic reproduction, reprint (but not a new edition) or an original film and a dvd release. this potential higher level cluster serves the user well, as usually users are more concerned with content and less with actual imprints. the project’s focus started to move from identifiers to record clustering. the project was approved in early 2009, tasked with creating the clusters but working within the current database and system architecture. existing system environment of worldcat the worldcat database has at its heart bibliographic records in marc21 format extended with additional data including coding for format, work identifiers and with links to authority and holding records. as new records are added or modified, some updating is done in real time and some in the background or overnight. there are two processes that are relevant to the glimir project: the frbr algorithm that creates work clusters and assigns the oclc work identifier (owi), and duplicate detection and resolution (ddr). the frbr algorithm works in real time and operates via examining author and title data elements of the record and making an author/title key. all records with an equivalent author/title key are clustered and given the same owi. ddr works as an offline process, launching queries to find candidate duplicates, and then comparing records in pairs. a level of certainty is assigned to a match. if the match confidence is below the threshold, the records are left untouched. a resolution program examines the records themselves in multiple match cases to determine the retained record in a merge. matching techniques in ddr are discussed in more detail elsewhere (thornburg and oskins 2007). it was decided to develop glimir starting with ddr, adjusting the matching algorithms and using the resulting matches to drive the indexing which creates clusters and then seals the clusters with identifiers. this was a radically different application of matching from the existing ddr application. glimir processing overview a team at oclc worked for more than 18 months adapting and testing the duplicate detection algorithms and devising procedures for assigning the identifiers and making the content and manifestation clusters. the development was heuristic, that is sample works with hand-crafted expected groupings were compared against actual program test results and the algorithms honed and re-tested until optimal results were achieved. tests were conducted several times a week cumulating in seven major baseline tests. more than 40 sets were hand created, most with around 60 members with a range of characteristics and material types. very large sets and smaller sets were also included. testing proved an intricate process; unlike ddr testing where a single match result is evaluated, not only the match itself but the ensemble of matches for all records in a work set and the resulting clusters are evaluated. records dealing with music posed particular challenges that have been discussed in detail by thornburg and oskins. “describing musical pieces, whether sound recordings, scores, librettos, videos, has always involved cataloger interpretation and judgment. there is considerable variation in records created for exactly the same item. and there is never ‘proof’ that two records which seem to describe the same item actually do” thornburg and oskins (2012) the major achievements of the project included detecting equal content, and defining improvements to ddr and to frbr work-set clustering. detecting equal content involved two major components: defining compatible and incompatible formats from the typical user’s perspective, such as sound recording and text, and coping with inconsistent format encoding within records. maintaining two streams of analysis for the manifestation and content levels in the one process was also an achievement. detecting parallel records at the manifestation level, glimir is clustering records in different description languages and also detecting records that are probably duplicates but could not be safely de-duplicated for various reasons; either the cataloguing style was too different for the records to be merged, or there were too many candidates for merging which is often the case for sparse records, or the software returned a confidence level that was not sufficient for merging. use of the ddr software in a large de-duplication project during 2009 and 2010 removed more than 7 million duplicate records from worldcat, but there are still many records that seem duplicates to the human eye. because glimir is clustering records but not merging and deleting, it can be more aggressive in determining duplicates and bringing them into the manifestation clusters. if it is found that records have been incorrectly clustered, undoing a cluster is considerably easier than undoing a merged record. the project started with the ddr programs and removed the test for language of cataloguing. figure 1: two records in a manifestation cluster with different language of cataloguing the two records above describe the same manifestation; the one on the left is catalogued in english and the one on the right in french. in the process, many improvements were made to duplicate detection itself, particularly in relation to matching records in different languages of cataloguing and created with differing cataloguing rules. examples of some new tests that benefitted ddr directly are: creation and employment of extensive tables of language equivalents of words such as “stories”, “contents”, “volume” etc. these are often used in subtitles in some but not all records, preventing matching. matching titles containing abbreviations using specially developed title comparison methods for german patterns of abbreviation and detection/deletion of nontitle information. figure 2: two records clustered by special title analysis matching publishers with short and long versions and abbreviated versions using tables and other equivalence checks and trigrams inferring language where none was coded in the 008 field by, for example, examining title keywords and publishers. detecting size equivalents, e.g. 8 ° = 22 cm = 8gr. = 8 inches figure 3: two records clustered by detection of equivalence in size expression and by publisher matching detecting banal collation information within titles figure 4: two records clustered by ignoring collation information in the title not all tests were adopted by ddr. glimir was able to be more tolerant, as mentioned earlier. because the matching software running glimir was guided by programmatic metatype, enhancements could be applied generally or restricted to glimir with relative ease. examples of glimir-specific routines: tolerating minor errors in extent, e.g. 288 p. versus 228 p. allowing minor variations in edition statements, especially where the edition statement is non-numeric machine comparison can approximate but never equal human judgment. automatic comparison cannot be on a case by case basis, meaning that each rule employed must reasonably apply to all records in the database. where there is uncertainty, it is not safe to de-duplicate, or sometimes even to cluster. figure 5: two records that have not matched the two records above are a case where glimir has not been able to determine a match. the records disagree on isbn, the one on the right saying 9782703665302 is false, and the one on the left coding it as good and a different one as false. glimir would have accepted the small differences in publisher and coding of the author, but the presence of a strong edition statement in one, coupled with the disagreement on isbn, caused the failure. determining equal content the glimir algorithms detect equal content by a two-pronged approach. firstly the duplicate detection tests are relaxed such that differences in the publisher, publication date and format are tolerated, where they are not for the manifestation level, and thereby making candidates for a content cluster. second, a further set of tests will overrule and deny a match based on cast, language of work and format analysis. cast analysis looks for the major contributors (directors, conductors, soloists and major actors of a musical, film or theatrical performance or readers of a talking book). only the major contributors are sought as the cast detail in records varies greatly. a multi-lingual table for cast terminology was developed and deployed. for language of item, most records have correct codes that can be used, but for many records without such codes, it was necessary to attempt to infer the language using a variety of indicators such as distinguishing common words in the title, isbn prefix and publisher. format analysis prevents incompatible formats from clustering, e.g. a film and a book, talking book and printed text. the glimir content level is aimed at achieving useful clustering of existing records from a user perspective. whilst librarians are traditionally focused on the accurate description of the actual manifestations they own or have access to, users are more typically seeking at the work and content level; if a particular edition is not readily available an alternative with equal content is usually acceptable for all but elite researchers. in terms of the functional requirements for bibliographic records (frbr) model (ifla 1998), the glimir content clusters are achieved by parallel links among manifestations. the glimir content cluster is not the equivalent of the frbr expression layer. however, by clustering content within a work set, the real differences among its manifestations, such as translations, cast differences or editions with different prefaces, are made more apparent. sherry velluci (2007) gives a good discussion of the complexities of the frbr expression level in music. studies have concluded that effectively generating the frbr expression level from existing marc records is not possible. “our experience, which reportedly has been the experience of other groups as well [elag], has led us to concentrate on the identification of works and, to a great extent, to abandon our experiments on identification of expressions for now.” (hickey 2002) coincidentally, the glimir content level is similar to the concept of clusters in the istc (international standard text code). “an istc does not ‘belong’ to a single author/publisher; rather, it ‘belongs’ to the work it identifies. this means that the same istc number should be used to identify the same content even when it is being published by a different publisher and/or in a different publication format.” (istc 2012) figure 6: two records in a content cluster the two records above have the same content but they are not equivalent manifestations, having different imprints. clustering a new resolution process was written for glimir. instead of the ddr resolution process that chooses a record to retain and may incorporate fields from the record being deleted, the process makes clusters of records and assigns an identifier. because of the diversity of detail and cataloguing styles within the records in worldcat, matching among records is not always symmetrical. record a may be determined to have the same content as records b, c and d but record b may in addition have the same content as record e but not have been matched with record d. however, where a matches with b, b will always match with a. the clustering allows for asymmetry by applying the “friends of a friend rule.” the same clustering principles are employed in viaf as is illustrated below. figure 7: viaf clustering using the friend of a friend method in the picture above, the viaf cluster for smith, martin cruz, 1942includes 13 sources. most sources are cross matching, except the israeli source is only matching with the library of congress naco record. (viaf 2012) early in the project it was noticed that special treatment was needed for sparse records to prevent over-clustering. sparse records, for example those without a date or without publisher and imprint information exist in worldcat, most often representing retrospective records converted from paper sources that had been batch loaded to worldcat over the years. performances without cast are also considered sparse. ddr is not able to successfully de-duplicate and merge these records because it is not able to determine with confidence which of two or more records would be the best candidate. as most sparse records only have one holding, glimir places these records into one manifestation cluster and neutralizes their matches so that the “friend of a friend” rule does not take effect for them. glimir clustering can be more aggressive than ddr as correction is an easy process, usually achieved by upgrading a record that has been wrongly clustered. clustering also brings other benefits: relevant holdings can be viewed at any level – work, content, manifestation at each viewing level the metadata displayed can be selected from the most appropriate record, either the most complete record or the most complete record in the relevant description language as determined from information about the viewer, such as ip address range more accurate statistics are available on the numbers of works, manifestations and records within worldcat worldcat will be better structured to support web linking and better analysis of collective global library resources (the so-called “collective collection”) frbr work level improvements from the glimir project improvements to duplicate detection were expected; what was not expected were the considerable improvements that glimir was able to bring to frbr work sets. the improvements are significant, especially to interfaces like worldcat.org where the initial displays of result sets are based on representative records from the existing work groups. it is important that records are not split erroneously among work sets, else users will see duplication, even where ddr and glimir have produced correct results. the frbr algorithms work on examining author, title and uniform title. glimir works on many more elements of the record and can find outliers that should be in the same work but aren’t. example: figure 8: two records in a content cluster in different work sets in this example glimir has determined that these two have equal content, though they are not the same edition. the record on the left is a reprint of the one on the right, indicated by the identical number of pages (both are german translations). the record on the right has coded the original english title in a note where the current frbr algorithm cannot find it. glimir will pull these into the same work set. more than 30 work algorithm improvements have been identified by glimir. some of these include: locating author in separate places within the record (e.g. in 700 or 720 rather than field 100) locating original title of translations (11 coding variations detected) normalising titles by ignoring such things as extraneous collation information using trigrams to tolerate abbreviations ignoring banal qualifications, e.g “a novel” (using multi-lingual table) matching multi-part works coded in different ways, as a title in a series and as a title with a part title defining compatible and incompatible formats and how to detect them in inconsistently coded records glimir matches will also bring together works where it finds a match at either a content or manifestation level, in cases where the frbr algorithm cannot determine a match. an example is where one record for a translation contains an original title and another only contains the translated title. thus glimir and frbr routines are complementing one another. frbr takes a top down view of the records, examining selected fields and generating an author / title key that is used in conjunction with authority records and some hand crafted author/title pairs. glimir comes along in a second, independent pass, selecting candidate records via a series of queries and examining a larger range of fields. as a result, sometimes it is able to identify outliers for a work set. by bringing outliers into the work set, glimir will ensure that there will be records retrieved that were not previously retrieved, depending on the search criteria. all enquiry interfaces will benefit from this. in the case of worldcat.org, it will also mean that the initial search result which displays one record from each work set, will be able to display fewer records as they will be correctly grouped. figure 9: two records in pre-glimir worldcat.org display (each is in a different work set) the figure above shows the results of a search in worldcat.org as it was before glimir for the novel the long walk. two separate non adjacent lines were shown in the search results on different screens because there were 2 different work sets, one with the author as stephen king and another with the author by his, now deprecated, pseudonym richard bachman. each work set has different holdings; the so called “holdings scatter” problem. glimir has found record content pairs from across these work sets (and a couple more) and has caused the work sets to merge. for worldcat.org, this is a double benefit. firstly only one line will show in this display. secondly, when “view all editions and formats” is clicked, it will display all and not just some editions and holdings. figure 10: the search result after glimir processing (condensed work sets) further improvements will come when worldcat.org is able to show a work landing page. the following screen shows a possible work landing page with glimir. figure 11: mock-up of possible work set display with glimir clusters for the first line, the original english edition, there are cataloguing descriptions in english, french and slovenian. behind the scenes, worldcat.org will be able to use the most appropriate record, depending on the ip address of the user. glimir has made 12 content clusters (of which 5 are displaying) within this work and has also condensed several work sets into one. this will enable worldcat.org to initially present the work in approximately 20 lines (12 clusters and 8 or so singles) instead of 81 as that is the number of records in the work set. it should be recognized that machine evaluation will not always be perfect, just as the records being compared have been catalogued in many different styles, converted by many different programs and created by many persons. regardless of the level of sophistication of any machine-matching algorithms, the matching decisions will not always mimic the decisions humans might make when comparing records. as a result, allowance has been made for both forced merging and forced splitting of glimir content clusters and of work sets in the architecture. from extensive testing, glimir merges work sets correctly. the oclc frbr work set algorithm currently does not handle works without authors well at all and glimir is correctly bringing these together, particularly for films and serials. the frbr work set algorithm also consistently misses translations and records catalogued in languages of cataloguing other than english. glimir is bringing serials into the same work set, though, like the frbr work set algorithm, it cannot differentiate and hence cannot solve incorrectly coded latest entry and successive entry records with reliability. so in the case of serials glimir may cause over-consolidation, but this is far better than leaving scores of work sets where there should only be 3 or 4. glimir also does not solve the problem of collective works for frbr. if the frbr algorithm has combined single and collected works into one work set, glimir matches may consolidate outliers into the set for both the single and the collected work. in this case, at least the glimir content clusters will differentiate these within the set. another problematic area was where a frbr work set erroneously contains records with clashing formats; examples were noted where existing work sets contain films and books. the frbr algorithm has since been corrected and erroneous work sets are in the process of being corrected prior to glimir processing while glimir processing is proceeding on sets of records that are unaffected. note that glimir only causes work sets to merge. it does not split them. also, as it is now, glimir cannot replace the frbr algorithm as explained earlier. there is, however, another unexpected gain that glimir brings for existing frbr processing concerning response time. the frbr algorithm works in real time and glimir works in background. there is a system constraint that requires a work id for every record, but it would be ideal if glimir processing could precede frbr processing. for inserted records, the sequence will remain frbr-glimir, but for all changed records the sequence will be reversed and glimir will only refer changed records for frbr re-processing where glimir determines that the content has changed. thus, the load in real time caused by frbr processing will be lessened. the following table shows some of the test work sets. extensive and aggressive testing was also done on random sets, including sets for particular languages, large sets, a set for hathi trust and other specific formats. table 1: some test work sets used in the glimir project work set number of records content clusters work sets consolidated * notes double indemnity 47 7 6 love (morrison) 73 20 6 ravens gate 46 23 1 2 other records in 2 different work sets; glimir didn’t fix remember (bradford) 68 16 5 taking pelham 29 6 8 time and again 41 16 8 zhiznyi 73 27 8 curculio (plautus) 86 26 13 goya caprichos 73 19 22 golden apples 75 16 6 great men of science 59 6 10 long walk 36 11 6 common law (broom) 60 11 1 for kicks 75 35 4 1 other record in different work set; glimir didn’t fix eric 50 16 34 library journal 83 1 or 2 47 science news 56 4 25 should be 3-4 work sets lancet 71 3 53 plus 8 other regional national editions * during glimir processing if glimir determines 2 records in different work sets have equal content, the work sets will be merged. glimir processing started on worldcat in september 2011. it started slowly, so that results could be evaluated and improvements implemented. by december 2011, processing had started to accelerate so that at the end of the month just under 500,000 records had been processed. records coded with language of work german were selected for priority treatment as it was expected that glimir would have a significant impact on them. the results confirmed this expectation: table 2: glimir results december 23 2011 records processed works sets merged % work sets merged content clusters manifestation clusters 475,556 287,066 60.3% 422,208 426,308 deployment of the clusters the first phase of the glimir project generates the clusters and corrects the work sets. a follow-on phase will create internal services to be used by all products that will deliver clustered results with holdings totals at all levels and summaries, e.g. of publishers, formats and dates of publication per cluster. it is envisaged that the manifestation cluster will enable services to display only one record from the cluster, the selection depending on the user. no matter which record is displayed, all holdings of all records in the cluster will be displayed. the manifestation cluster will also give a new, reasonably accurate count of the number of manifestations held by worldcat libraries. the content cluster will enable services to display a summary line for the cluster in initial short list displays which will improve the experience of browsing a short list and making a selection. aslo, in the first phase, glimir identifiers will be included in externally facing web services, xoclcnum and the worldcat api. oclc is considering expanding its beta owi/ocn table service to include content and manifestation identifiers and metadata. currently, just a table containing selected work (owi) and record (ocn) identifiers is provided to two library networks corresponding to their subset of worldcat. diffusion options and formats are also being evaluated, including xml and rdf/linked data, as part of a general evaluation of export services. a separate paper is envisaged to discuss the various business cases for glimir metadata and identifier services. figure 12: diagrammatic schema of oclc’s metadata and identifier structure further work follow on work is bringing ddr, glimir and frbr processing closer together, further improving the work level clustering and improving the clustering and linking of multi-part works, for example by allowing a compendium manifestation to link to multiple work records, namely the compendium work and individual works for the components, and by allowing links among works. research work is also underway to reconsider the data architecture. many existing parallel records were made by copy cataloguing. one typical example is below where the record coded swedish has swedish subject headings (and also english headings not shown), “p” substituted by “s” to represent pagination, which was probably done by a macro, but with a note about bibliographic references in english. figure 13: example of a parallel record made by copy cataloguing only certain parts of a typical marc record will differ when the resource is described in a different language. the description fields (title, imprint, extent, edition) have no significant content that varies with the cataloguing language and what content of this nature that there is can in most cases be generated programmatically using the subfield encoding. for example the statement of responsibility subfield code within the title (marc21 245 $c), which is rendered as a slash in isbd, could be rendered as “by” in english, “par” in french, etc. even the table of contents will not change when the language of cataloguing changes. the parts of a parallel record that can vary by language and cannot be generated by an algorithm are: author – there are linguistic variants in the name form subject headings summary and other non-descriptive notes these are all work or expression level data, but they are currently held at the record level and as such are valuable access points and information that are not easily available to other records in the work. moreover, the author and subject headings can be better managed as multi-lingual authority records. this leaves basically only the summary as having any real value in the parallel record and that too is most often work level information. the rest of the record is redundant storage. the majority of the parallel records do not have these summaries. oclc research is now working on a new project entitled “moving beyond marc: rethinking the architecture of bibliographic description.” one of the objectives is to explore the storage and linkage of bibliographic components to enable the assemblage of a record in a particular language from authority records and from macros working on the subfield coding. it would allow different language versions to be produced for all records, not just those for which parallel records have been loaded and the records would be richer as they could be assembled from all relevant fields in all records in the cluster. relevant fields include language sensitive fields in the relevant language and all those that are not language sensitive. conclusion the initial goal of the program was to create manifestation clusters and identifiers. twice, the scope was broadened, firstly to create content clusters and identifiers and secondly to improve frbr work clusters. the increased scope resulted in considerable challenges such as detecting equal content, defining compatible and incompatible formats and coping with inconsistent format encoding within records. maintaining two streams of analysis in the one program was also a challenge. the initial visible impact of glimir will actually be from improved work level displays. this will improve the perception of duplicates at the work level which is usually users’ first impression. once worldcat has been fully processed, estimated to be in the third quarter of 2012, and services start to use the glimir clusters, the direct benefits of glimir will become apparent and diffusion of the identifiers can commence. references gatenby, janifer (2008). the importance of identifiers. metalogue, 22 august 2008 [cited 2012 march 5]. available from: http://community.oclc.org/metalogue/archives/2008/08/the-importance-of-identifiers.html hickey, thomas b., o’neill, edward t., and toves, jenny (2002). experiments with the ifla functional requirements for bibliographic records (frbr). d-lib magazine, v. 8, no. 9 [cited 2012 march 5]. available from: http://www.dlib.org/dlib/september02/hickey/09hickey.html ifla study group on the functional requirements for bibliographic records (1998). functional requirements for bibliographic records: final report. — münchen: k.g. saur, 1998. — (ubcim publications; new series, vol. 19). — isbn 3-598-11382-x [cited 2012 march 5]. available from: http://www.ifla.org/en/publications/functional-requirements-for-bibliographic-records istc (2012). [cited 2012 march 5]. available from: http://www.istc-international.org/html/about.aspx oclc (2012). [cited 2012 march 5]. available from: http://www.oclc.org/uk/en/worldcat/statistics/default.htm thornburg, gail, and oskins, w. michael (2012). matching music: clustering versus distinguishing records in a large database. oclc systems and services, v. 28, issue 1 [cited 2012 march 5]. available from: http://www.emeraldinsight.com/journals.htm?articleid=17006810 thornburg, gail, and oskins, w. michael (2007). misinformation and bias in metadata processing: matching in large databases. information technology and libraries, v. 26, no. 2, 15-22 [cited 2012 march 5]. available from: https://www.ala.org/ala/mgrps/divs/lita/publications/ital/26/2/thornburg.pdf vellucci, s. l. (2007). frbr and music. in taylor, a.g. (ed.). understanding frbr: what it is and how it will affect our retrieval tools. westport, ct: libraries unlimited, 2007. p. 131-151. (coins) viaf (2012). display for smith, martin cruz, 1942[cited 2012 march 5]. available from: http://viaf.org/viaf/95175916/#smith,_martin_cruz,_1942weibel, stu (2008). a glimir of the future. weibel lines. 12 february 2008 [cited 2012 march 5]. available from: http://weibel-lines.typepad.com/weibelines/2008/02/a-glimir-of-the.html acknowledgments the authors would like to thank the following members of the glimir team for their work and input to this paper: robert bremer, ted fons, ying li, pat schuette, jay weitz and kelly womble. about the authors janifer gatenby (janifer.gatenby_at_oclc.org) is emea program metadata manager at oclc in leiden. she is currently researching a new bibliographic architecture related to the frbr model following on from the glimir project and is working on the implementation of the database and management system for isni (international standard name identifier). richard greene (greener_at_oclc.org) is a senior consulting database specialist at oclc in dublin, ohio. he is currently working on enhancements to bibliographic record matching for those categories of records not currently de-duplicated and on improvements to batch processing. w. michael oskins (w.oskins_at_att.net ) has worked as a developer and researcher at oclc in dublin, ohio for over twenty years. gail thornburg (thornbug_at_oclc.org ) has taught at the university of maryland and elsewhere, and is now a senior-level developer and researcher at oclc in dublin, ohio. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – automated 3d printing in libraries mission editorial committee process and structure code4lib issue 53, 2022-05-09 automated 3d printing in libraries this article highlights the creation of an automated 3d printed system created at a health sciences library at a large research university. as covid-19 limited in-person interaction with 3d printers, a group of library staff came together to code a form that took users’ 3d printed files and connected them to machines automatically. a ticketing system and payment form was also automated via this system. the only in-person interactions are dedicated staff members that unload the prints. this article will describe the journey in getting to an automated system and share code and strategies so others can try it for themselves. by brandon patterson, ben engel, and willis holle how it started in early 2020, hospitals were close to the end of their supply of personal protective equipment (ppe) due to covid-19 and healthcare workers ran the risk of being exposed to the virus without proper protection. the j. willard marriott library and spencer s. eccles health sciences library at the university of utah sprung into action and provided access to several dozen 3d printers that could print face shields, mask ear protector, powered air purifying respirator (papr) hoods, and various lab equipment. before covid-19, these printers were used by students and library workers who used a flashdrive to complete printing projects. in doing repetitive 3d prints during the national lockdown, we decided we could automate this process. a lead developer went to work creating a system where a file could be uploaded to a form, checked for errors or bugs, and trigger the print automatically. we officially launched this service in fall semester 2021 after some debugging and test runs with the code. the automated system was well-received by students, staff and faculty. it also provided additional benefits to the library including all-hours access to printing, decreased dependence on staff, and decreased filament waste and costs. the libraries have a fleet of prusa i3 mk3s+ using 1.75mm pla+ filament (image 1). they are housed in locations where trained staff can unload prints and reset printers for the automated system to begin again. technology has been purchased through student computing fees and staff support is provided through library funds. in addition to the prusa machines performing automated prints, the marriott library also has lulzbot taz 6 and gigabot 3+ machines that are reserved for hands-on printing only. the service has seen hundreds of prints since its inception. image 1: prusa printer automated 3d printer “farm” along with printed anatomy pieces like “yorick the skull”. technical guide figure 1: flowchart representing the steps to getting a 3d object printed automatically a visual overview of our 3d printing process is referenced via the flowchart above (figure 1). it begins when a user creates a 3d model (stl/obj) file, using the prusaslicer with a downloaded configuration file, then the g-code file gets submitted to an intake form we had created. this intake form then opens in a ticketing system (we use jira) with accompanying files and information. the file server then accesses jira via their api and downloads all of the uploaded g-code files. once the files have been downloaded, an octoprint api sends the files to the printer. the system monitors the printers over that octoprint api and updates jira as needed. when a print is downloaded, started, or finished, jira is updated and notifies the user through email of their updated print status. everything related to 3d printing always starts with a model, these 3d files need to be translated via a slicer into something the 3d printer can read and re-create, and our system is no different. we require our users to use a program called prusaslicer. we use prusaslicer because of its easy integration with our system. it lets us distribute config bundles to every user who wants to use our system, these config bundles are very important to the health and safety of our printer farm. we limit use to only pla plastic as the default support material, provide a cost per gram, and most importantly, a start and end g-code is added to every print. these special parts of g-code are critical to the function of this system. a “m0” command is placed just above the last line of the end g-code so that when a print is completed the printer will wait for user input. this will hold the printer out of the ready printers until the previous print has been removed and the knob on the printer clicked by a person to confirm everything was printed correctly. once a user is ready to submit their new g-code file they will go to the intake form and upload their file. this intake form can hold a myriad of information, for example we ask what their university affiliation is, if they have a class key, or what printer they are planning on printing the part on. this intake form can be created in a number of different ways, such as a google form, but we decided to go with a jira ticketing system as that is what our library uses currently to track similar projects. once the user submits the form it creates a ticket in the jira project that was set up for us. if not using jira, this part will be a bit different but the same ideas will hold. now that a file is uploaded to a service that is accessible, an api is used to read the list of open tickets and download the g-code files. in our case we had an issue where jira would only let us accept files up to 25mb which prompted us to allow users to paste a google drive link into the description box and the code allows it to be downloaded just the same. the goal here is to get g-code files into a folder on the machine running our code. in our case we rename the files to the id number of the ticket it is associated with. this lets us do a few ticket updates later on and keeps project anonymity of the user. the server now has a folder full of files, these files need to be put onto an open printer, so via the api, each printer is made to check its printing status. once a print comes back as open, the server will post the g-code file to that printer and tell it to start printing. because we saved the ticket id number in the file name of the g-code, we can read that and update the ticket so that the user is notified that their print has started. the server will continue to do this process for every file it has downloaded. if no printer is stated as open it will try again in a fixed interval until an open printer is found. a printer is marked as open once the m0 command is cleared. m0 is the g-code we placed in the “end g-code” section of the config file. it will wait for user input and then continue the print. if a person does not clear the m0 command by pressing the knob on the printer, octoprint will report a progress of 99.99% complete and the print will not be marked as finished unless the knob is pressed. this process was put in place so as to not start a new print while an old one is still in place on the platform. this is critical to get right regardless of what system is being used. by default, octoprint blocks the m0 command and replaces it with a pause. if a pause is not put in place, it would require a login to the printer’s octoprint instead of a visual confirmation by a person, to mark the print as complete, which is not recommended. when setting up octoprint for each printer, settings must be changed to remove any mention of blocking or re-mapping the m0 command. once the printer gets that last command cleared it will trigger another api call to jira, updating the ticket status to complete, and posting a comment for the user to either pay for their print or give a location for pickup. some additional things have been built into our system that are of note. we have made a number of config files so that everything can be changed and more modular. we used flask, a micro-web framework written in python, to make a web ui for monitoring and editing the printers live using web sockets, and have included docker compose files to get a virtual print farm going in just a few minutes after testing for debugging. a link to our github project can be found here https://github.com/benbeezy/octofarmjira. note that we are using jira and it is heavily integrated into our repository as of right now. the goal is to continue work on this project and make it more platform agnostic. helping users and getting the word out to assist users in utilizing the printer farm, a short guide was created and posted on the intake form (figure 2). this guide runs each user through downloading the prusaslicer software, as well as the specialized config bundle required to use our service. despite being quite short, this tutorial appears to function exactly as desired and has helped to maintain a very low print failure ratio. for users needing a more in depth guide, a short youtube video was created that runs users through each step of using the print service and also serves as a brief introduction to operating within the prusaslicer software. in addition, to help with any questions that users might have, staff are available for one-on-one consultations. figure 2. setup guide for new users a key benefit of this software is the ease of expansion and adoption. given how the printer farm software has been developed, expansion is as simple as adding a new printer ip and an api key to the config file. this ease of expansion has allowed the j. willard marriott library, another library on the university of utah campus, to adopt this software for their 20+ printers. this serves to show both that adopting the software can be done and in a short time frame, as well as proof that printer farms of dramatically different sizes can use the software without any substantive work required. to raise awareness of the capability of the automated 3d printing process on campus, a tv display board was installed within the library advertising the libraries printers as well as a qr code leading to the intake form. in addition, multiple emails were sent to the library’s emailing list with information regarding the print farm. several people have also heard about it through word of mouth. we are currently working with several departments on campus, including the health sciences simulation center, to continually print replacement parts for their equipment. after several months, we’ve increased the usage and awareness of the library’s automated 3d printing farm. conclusion overall, automating the 3d printing process at the library has streamlined our 3d printing process. it has made our work in the library more efficient and cost effective. it is more efficient because anyone can start a print at any day and time and we lessen the chance of failed prints simply because less people touch the printers and potentially break them. it is more cost effective because we don’t need to pay staff to start, stop and check on machines as often. there is a downside to all 3d printing automation though because novices may be apprehensive to go online and start a print themselves in preference of physical assistance. there is also less opportunity for an interested patron to learn about the mechanics of a 3d printing machine. because of this, the library has 3d printers set aside, beside the ones used for automation, for students that would like a more hands-on approach to their learning. to further improve upon our automation process, we’re interested in making it ticket system agnostic so that we don’t need to rely on jira, google forms or other ways to direct it onto the request. we’re currently working on an update status feature that can automate responses with percentage complete, photos, and other useful information for users that have questions about the status of their prints. finally, we have one of our libraries that charges for filament to print and we are trying to automate this to charge a user the weight of the filament based on volume x density x cost per gram. for those looking to start their own 3d printing automation service, here are a few best practices: this service doesn’t need to cost extra, use existing equipment. before deciding if this service is right for your library, get stakeholders together (i.e. library staff, patrons that use 3d printing, it assistance, and others) and have a discussion about whether an automated or part-automated 3d printing service would make sense. it is essential to have a good programmer who knows python. students may be able to serve in this role. the github in this article can be a great resource and we encourage suggestions to make the code better. if your library already has a ticketing system, like jira, ensure that whoever controls that is onboard. you can also create your own, like through google forms, you’ll just need to do a little more coding to sync it properly to talk to the 3d printers. you don’t have to go all in. it’s nice to start small, maybe having only a few printers automated. nevertheless, it’s recommended to always have some spare 3d printers available that aren’t part of the automated “farm” that can be tinkered with (although we found it surprising how many users converted to the automated system after we made it available). acknowledgements special thanks to the 3d printing team at the j. willard marriott library at the university of utah: tj ferrill, matt elliott, erika church and sebastian jenson. about the authors” brandon patterson is the technology engagement librarian at the eccles health sciences library at the university of utah. he connects students, staff, and faculty to digital tools and emerging technologies and creates meaningful experiences using prototyping tools, virtual reality, and online learning platforms. he is an health sciences education liaison and coordinates with faculty to incorporate information literacy instruction and technology into their classrooms. ben engel is the technical lead on many projects associated with 3d printing and vr at the eccles health sciences library. he leads the effort in managing, programming, and navigating equipment and software in these areas. some highlights include leading development teams to build out four (4) different modules for the school of dentistry, department of surgery, ophthalmology, and the national library of medicine. he’s also created a custom automation system to assist with 3d printing. willis holle is the xr student intern at the eccles health sciences library at the university of utah. he assists in 3d printing and xr projects related to health sciences education and is currently studying physics, with an emphasis on condensed matter. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – a novel open source approach to monitor ezproxy users’ activities mission editorial committee process and structure code4lib issue 29, 2015-07-15 a novel open source approach to monitor ezproxy users’ activities this article describes using elasticsearch/logstash/kibana (elk) to monitor and visualize ezproxy logs in real time. by qing zou introduction ezproxy is a popular web proxy server that is widely used in academic libraries for providing off-campus access to their owned or subscribed electronic resources.  although ezproxy provides many mechanisms (e.g. auditing, intrusion detection, server sessions status, and usage limit functions) to ensure secure accesses, malicious system intrusions and illegal downloading have been experienced by many libraries.  these activities may cause temporary blockage of access to certain electronic resources by content providers.  due to the high amount of usage and limited existing functionality of ezproxy for monitoring users’ activities, it is challenging for libraries to be more proactive in detecting users’ illegal activities. approach this paper proposes a novel way to monitor ezproxy in real time by integrating  elasticsearch/logstash/kibana (elk) into the existing ezproxy environment.  in this approach, the log files of ezproxy are considered as valuable dynamic data sources.  unlike normal  traditional approaches[1] to analyzing ezproxy log files, users’ activities through ezproxy are monitored and visualized in real-time. system architecture in this approach, each open source application package of elk plays a different role collaboratively to “make searching and analyzing data easier than ever before” [2].  built on lucene search engine, elasticsearch [3] provides indexing functionality for handling log information. logstash [4] is a tool for managing events and logs . kibana [5] is used to visualize logs and time-stamped data. since a typical ezproxy server may run with limited memory (1 gigabyte random access memory or less) and elk requires more memory to run smoothly, it is not practical to run elk and ezproxy on the same server. fortunately, logstash-forwarder [6] is able to run with limited memory and ship logs out of the ezproxy server without significantly decreasing the level of performance of the existing ezproxy server.  figure 1 shows how elk works with an existing ezproxy server. figure 1. system architecture basically logstash-forwarder works with an ezproxy server and monitors any changes of the log file of the server and sends new log entries to the monitoring server.  on the monitoring server, logstash listens for new messages (log entries) from the logstash-forwarder and indexes them into elasticsearch.  kibana visualizes the indexed logs and presents them to administrators. how it works this approach converts unstructured ezproxy log entries into structured indexes.  ezproxy records any accesses in log files.  the typical entry in an ezproxy log file looks like as follows: xxx.xxx.xxx.xxx rezou1 [01/feb/2015:04:07:44 -0500] get http://apps.webofknowledge.com:80/wos_generalsearch_input.dosid=1 czc4vbptsujje4mdgy&product=wos&search_mode=generalsearch&errorqid= 45 http/1.1 200 81049 the format of log entries is controlled by logformat [7] in the ezproxy configuration file.  the example above is using the following format: logformat %h %l %t %r %s %b field value %h host accessing ezproxy (always ip address). %l remote username obtained by idented (identd is not used, so this always inserts -). %t date/time of request; may also appear as %{format}t to specify a strftime time format. %r complete request (e.g. get http://www.somedb.com http/1.0). %s http numeric status of request (always 200 in ezproxy 1.0 and 1.2). a table of special ezproxy-specific status codes appears below. %b number of bytes transferred. figure 2 logformat fields [7] logstash-forwarder is a log shipper that collects logs locally in preparation for processing elsewhere.  the logstash-forwarder is written in go language [8] developed by google and can be compiled into one executable binary file.  it is rather simple to run: logstash-forwarder -config logstash-forwarder.conf all configuration settings are from logstash-forwarder.conf as the -config option indicates.  the content of the config file is straightforward, similar to the following: { "network": { "servers": [ "logstatsh server:port" ], "ssl certificate": "/etc/pki/tls/certs/logstash-forwarder.crt", "ssl key": "/etc/pki/tls/private/logstash-forwarder.key" }, "files": [ { "paths": [ "/var/log/ezproxy/ezproxy.log" ], "fields": { "type": "apache-access" } } ] } the “network” stanza covers network configuration; “servers” indicates which server is listening for messages from the ezproxy server.  the “ssl certificate” and “ssl key” are used to authenticate server connections so that only authenticated server(s) can receive messages.  the “files” stanza tells logstash-forwarder where the ezproxy.log resides and what type of the file it is. the elk stack is quite easy to install.  elasticsearch and kibana almost work out of the box. the configuration of logstash is similar to the following: input { lumberjack { port => portnumber ssl_certificate => "/opt/logstash/logstash-forwarder.crt" ssl_key => "/opt/logstash/logstash-forwarder.key" type => "apache-access" } } filter { if [type] == "apache-access" { grok{ patterns_dir => "patterns" match => { message => "%{ezproxylog}" } } } } output { elasticsearch { host => localhost } } the configuration consists of three parts: input, filter, and output. the input contains settings for connecting with the logstash-forwarder on the ezproxy server. the “port” is the port specified in the logstash-forwarder “servers”. the “ssl certificate” and “ssl key” are the same files for the logstash-forwarder. the “filter” stanza is for dividing messages into small separate pieces. since the type of our messages from the ezproxy server is set as “apache-access”, the filter matches this type of messages and uses grok [9] to parse log data and to follow the pattern called “ezproxylog” shown as follows: ezproxylog %{ip:client_ip} %{user:ident} \[%{httpdate:timestamp}\] %{word:verb} %{notspace:request} (?:http/%{number:httpversion}) %{number:response} %{number:bytes:int} the message is parsed into the following structure: { "client_ip" => "xxx.xxx.xxx.xxx", "ident" => "rezou1", "timestamp" => "01/feb/2015:04:07:44 -0500", "verb" => "get", "request" => "http://apps.webofknowledge.com:80/wos_generalsearch_input.dosid=1czc4vbptsujje4mdgy&product=wos&search_mode=generalsearch&errorqid=45", "httpversion" => "1.1", "response" => "200", "bytes" => 81049 } the “output” stanza is to tell logstash to spill out the converted log data into elasticsearch. by default, logstash creates one index for the log data each day. for example, the message indexed in elasticsearch is similar to the following json  (javascript object notation) snippets: { "_index": "logstash-2015.02.01", "_type": "apache-access", "_id": "autxcf8a5ax4wyjlyjpr", "_version": 1, "_score": 1, “_source”:{ "message" : "xxx.xxx.xxx.xxx rezou1 [01/feb/2015:04:07:44 -0500] get http://apps.webofknowledge.com:80/wos_generalsearch_input.do?sid=1czc4vbptsujje4md gy&product=wos&search_mode=generalsearch&errorqid=45 http/1.1 200 81049", "@version" : "1", "@timestamp" : "2015-02-04t15:59:32.223z", "type" : "apache-access", "file" : "/var/log/ezproxy/ezproxy.log", "host" : "ezproxy.lakeheadu.ca", "offset" : "4937", "client_ip" : "xxx.xxx.xxx.xxx", "ident" : "rezou1", “timestamp" : "01/feb/2015:04:07:44 -0500", "verb" : "get", "request" : "http://apps.webofknowledge.com:80/wos_generalsearch_input.do?sid=1czc4vbptsujje4mdgy&product=wos&search_mode=generalsearch&errorqid=45", "httpversion" : "1.1", "response" : "200", "bytes" : 81049 } the logstash-forwarder running on the ezproxy server watches for any additions to the ezproxy.log file and sends newly added entries to the logstash. the logstash service receives new entries and indexes the entries in elasticsearch. kibana loads changes in the elasticsearch indexes and visualizes in real time. results visualization can be viewed using any modern browser. kibana provides many ways to configure preferred views. the following bar diagram displays the amount of access from january 29th to february 3rd , 2015 at lakehead university. figure 3. events over time the following is a combined view of top 10 users and top 10 ip addresses. one of the top 10 ip address has been selected, shown in the bottom pie diagram and the users who used this ip address shown in the upper diagram. it is clear enough to spot high volume traffic of a user or one ip address in either top 10 users or ip addresses diagrams. usually, high volume may indicate potential problems. figure 4. a combined view of top 10 users and ip addresses although this approach makes easier to visualize log entries and to spot potential problems, it still requires attention to check continually. fortunately, raw log entries have been converted into structured indexes. more proactive approaches can be developed to detect anomalies or retrieve usage counts during a period of time. for example, the following query retrieves the users associated with  2,000 or more log entries and their corresponding ip addresses: curl –xget ‘http://monitoringserver:9200/_search?pretty’ -d ‘ { "aggs": { "users": { "terms": { "field": "ident", "min_doc_count": 2000 }, "aggs": { "ips": { "terms": { "field": "client_ip" } } } } } similarly, a query for retrieving a specific user’s usage can be developed. other scripts can be developed to collect information and send emails to administrators automatically if there are activities that may indicate potential problems. future work more proactive approaches can be developed by applying data mining, pattern matching, or machine learning techniques for detecting outliers automatically.  elasticsearch provides rich support for geographical information. meanwhile, kibana has map plugins that can display geolocation of ip addresses in a straightforward way. logstash provides a “geoip” filter to associate ip addresses with their geolocation, including latitude, longitude, city, region and country. with geolocation information stored in elasticsearch indexes, it is easier to identify illegal usages (e.g. a user with multiple ip addresses and accesses from different locations at the same time). conclusion the elk approach has been explained and demonstrated in this paper.  this novel approach will simplify the effort to monitor ezproxy users’ activities. it provides a friendly interface via visualization. it makes it possible to develop more proactive approaches to detect outliers and prevent blockages from happening. acknowledgements the author would like to thank the editors and reviewers for their valuable comments and suggestions.  special thanks to moira davidson (head of technical services at lakehead university library) for her suggestions to improve this article and working closely with the author to develop this approach. notes [1] http://acrl.ala.org/techconnect/?p=4684 [2] http://elasticsearch.org [3] http://elasticsearch.org/overview/elasticsearch [4] http://logstash.net [5] http://www.elasticsearch.org/overview/kibana/ [6] https://github.com/elasticsearch/logstash-forwarder [7] https://www.oclc.org/support/services/ezproxy/documentation/cfg/logformat.en.html [8] https://golang.org [9] http://logstash.net/docs/1.4.0/filters/grok about the author qing zou is a systems librarian at lakehead university and a phd candidate at the school of information studies at mcgill university, canada. his research interests are integrated library systems, institutional repositories, metadata, ontology, knowledge organization systems, and digital archives. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – design reusable shacl shapes and implement a linked data validation pipeline mission editorial committee process and structure code4lib issue 45, 2019-08-09 design reusable shacl shapes and implement a linked data validation pipeline in july 2017, w3c published shacl as the standard to validate rdf. since then, data modellers have the possibility to provide validation services based on shacl shapes together with their models, however there are considerations to be taken in account when creating them. this paper aims to list such considerations and shows an example of a validation pipeline to address them. by emidio stani introduction with the increasing demand of data to be provided to citizens, organizations have the challenge to make their processes more efficient so that decisions can be made fast, while maintaining and even increasing confidence in their data. therefore, organizations have the need to structure their data properly as the risk to have poor data quality might incur in high costs [1]. thus, organizations are encouraged to invest into applying metadata that plays a fundamental role beyond classifying data, as data needs to be transformed, integrated, and transmitted. however, like data, metadata needs to be harvested, standardized and validated. in the case of linked data, expressed as rdf, the harvesting phase has already become an important step implemented at large scale, see for example the european data portal (https://www.europeandataportal.eu/en/homepage) that provides access to 81 harvested catalogues for a total of 892,354 datasets. this is possible only by defining a common data model that allow describing catalogues and datasets in a uniform way, as is the case of dcat-ap for the european data portal (https://www.europeandataportal.eu/en/providing-data/goldbook/preparing-data). once a standard is defined, a validation service can built. such service will be based on violation rules, which in turn will depend on the data model. in rdf, it is already possible to validate data by means of querying it via the sparql language but as of july 2017, w3c released the shacl specification (https://www.w3.org/tr/shacl/) by which data modelers can provide shacl shapes together with their data models so that users can reuse them to validate their data. at the european level, the european commission developed a few application profiles such as dcat-ap (https://ec.europa.eu/isa2/solutions/dcat-application-profile-data-portals-europe_en) and cpsv-ap (https://ec.europa.eu/isa2/solutions/core-public-service-vocabulary-application-profile-cpsv-ap_en) (and some core vocabularies such as person, organization, etc.) for which the physical data models have been implemented in rdf. each release of the application profiles is accompanied with the respective shacl shapes. in the past, the european commission developed a sparql validator for dcat-ap (http://dcat-ap.semic.eu/dcat-ap_validator.html) giving a user-friendly way to validate catalogue and dataset descriptions and where the sparql query is the union of several sparql queries checking the presence of properties/relations described in the dcat-ap data model. currently, the european commission is providing a testbed as a service (https://www.itb.ec.europa.eu/shacl/swagger-ui.html) and product that allows users to check conformance of the application profiles by uploading files, provide a url or upload multiple files. at national or local level, we see the case of the flemish government (https://data.vlaanderen.be/) which developed a data model, called oslo2 (https://overheid.vlaanderen.be/oslo-wat-is-oslo2), partially based on the european data models, and aimed to standardize the data in the government of flanders. the flemish government implemented as well a tool chain to generate specifications and documentation from the uml class diagram of oslo2, among them the shacl shapes for each application profile (https://data.vlaanderen.be/shacl/). on top of that, the flemish government released the so-called oslo2 validator that provides a way to test data against the oslo2 shacl shapes (https://data.vlaanderen.be/shacl-validator/) where users can upload a file or just provide the url. both approaches, the european and the flemish government, depend on: the way shacl shapes are defined, which are not modular enough to be reused, forbidden and evaluated; the data provided, but there might be the need to enrich such data before validating by means of inferencing or business rules; the interface which oblige users to have few possibilities to validate data. introduction to shacl shacl (shapes constraint language) is a standard released by w3c in 2017 which aims to validate rdf data by means of so-called shapes. such shapes can be described in rdf itself, therefore allowing rdf store to be able to store them next to the rdf data contrary to sparql which has been defined with its own language. each shape is defined by 2 parts: target, to indicate the target (rdf class, specific rdf nodes, etc.) of the shape; constraints, to be applied to the target (cardinalities, range of values, etc.) in the example below (see listing 1), there is an example of a shape defined as ex:personshape which has as target all the classes (sh:targetclass) of type ex:person. further the shape defines 2 constraints on the property ex:name for which: the cardinality must be minimum 1 (that is a way to define mandatory property); the data type must be a string. ex:personshape a sh:nodeshape ; sh:targetclass ex:person ; # applies to all persons sh:property [ # blank node sh:path ex:name ; # constrains the values of ex:name sh:mincount 1 ; sh:datatype xsd:string ; ] . listing 1. example of shacl shape it is worth to note that the constraints are defined within a blank node (https://en.wikipedia.org/wiki/blank_node). further, shacl defines 2 types of shapes: nodeshapes, which allow to specify constraints about a focus node; propertyshapes, which allow to specify constraints about a property and the values of a path of a node. for example, another way to describe the shape described in listing 1 is to declare the constraint explicit with a propertyshape (ex:personshape-name), which contains the same constraints, and it is linked from the nodeshape as illustrated in listing 2. ex:personshape-name a sh:propertyshape ; sh:path ex:name ; sh:mincount 1 ; sh:datatype xsd:string . ex:personshape a sh:nodeshape ; sh:targetclass ex:person ; sh:property ex:personshape-name . listing 2. example of shacl propertyshape this document invites the reader to use propertyshapes as much as possible as described within the next section. defining reusable shacl shapes as mentioned earlier, shacl shapes are described in rdf itself so they can be managed in the same way as rdf data, however they can be described in different ways affecting their reusability. as an example oslo2 shacl shapes (and dcat-ap shapes) are defined with properties described as blank nodes e.g.: <https://data.vlaanderen.be/shacl/dienstencataloog#publiekedienstverleningshape> a sh:nodeshape ; sh:targetclass <http://purl.org/vocab/cpsv#publicservice> ; sh:property [ # blank node sh:name "heeftparticiperende" ; sh:description "participatie (van een agent) aan de publieke dienstverlening." ; sh:path <http://data.europa.eu/m8g/hasparticipation> ; sh:class <http://data.europa.eu/m8g/participation> ; ] ; sh:property [ # blank node sh:name "beschrijving" ; sh:description "beschrijving vd publieke dienstverlening." ; sh:path <http://purl.org/dc/terms/description> ; sh:datatype <http://www.w3.org/1999/02/22-rdf-syntax-ns#langstring> ; sh:mincount 1 ; sh:maxcount 1 ; ] . listing 3. example of a oslo2 nodeshape with blank nodes the problem of the definition of blank nodes is that other shacl shapes, described in the same data model or in other data models, cannot reuse such properties in contrast with the spirit of semantic web where reusability (by interlinking) is desired. a more concise way to describe such shacl shapes would be declaring explicitly the propertyshapes and referencing to them. listing 4 shows 2 propertyshapes and 2 nodeshapes defined within the cpsv namespace (the reader can find a more detailed example on https://github.com/catalogue-of-services-isa/cpsv-ap/blob/master/releases/2.2.1/cpsv-ap_shacl_shapes%20v2.2.1.ttl): cpsv:publicserviceshape-hasparticipation a sh:propertyshape ; sh:path cv:hasparticipation ; sh:class cv:participation ; sh:severity sh:violation . cpsv:hasmin1max1shape-description a sh:propertyshape ; sh:path dct:description ; sh:mincount 1 ; sh:maxcount 1 ; sh:nodekind sh:literal ; sh:severity sh:violation . cpsv:publicserviceshape a sh:nodeshape ; sh:targetclass cpsv:publicservice ; sh:property cpsv:publicserviceshape-hasparticipation ; sh:property cpsv:hasmin1max1shape-description . # shape reusing cpsv:hasmin1max1shape-description cpsv:participationshape a sh:nodeshape ; sh:targetclass cv:participation ; sh:property cpsv:hasmin1max1shape-description ; sh:property cpsv:hasmin1max1shape-identifier ; sh:property cpsv:participationshape-role . listing 4. example of a cpsv-ap shapes reusing propertyshapes in listing 4, we can see, for example, that the propertyshape hasmin1max1shape-description is reused by the publicserviceshape and the participationshape and requires a description property (dct:description) while the publicserviceshape-hasparticipation is only used by publicserviceshape requiring a participation class (cv:participation). in the case of dcat-ap, shacl shapes are described by reusing the same classes (dcat:catalog) declared in dcat-ap: dcat:catalog rdf:type sh:nodeshape ; sh:name "catalog"@en ; sh:property [ sh:path dct:description ; sh:mincount 1 ; sh:nodekind sh:literal ; sh:severity sh:violation ; ] . listing 5. example of a dcat-ap shape the shape in listing 5 doesn’t use the sh:targetclass (https://www.w3.org/tr/shacl/#targetclass) property, while oslo2 shacl shapes use it, which means that subclasses (of the dcat:catalog in the case) cannot inherit the shapes applied to it and benefit from the validation. third, by having the properties described as blank nodes, third party models which extend the original data models do not have the possibility to forbid (by means of sh:deactivated (https://www.w3.org/tr/shacl/#deactivated) ) some shapes if they do not apply in their domain (imagine that a xyz public service might have 2 descriptions while the cpsv-ap public service allows max 1 description), thus generating more violations than necessary. last, but not least, by explicitly naming shacl properties, validation libraries which return shacl validation reports might use the name of the shape by means of sh:sourceshape (https://www.w3.org/tr/shacl/#results-source-shape) that would be otherwise not accessible as they would be blank nodes. there are two side effects to this way to describe shacl shapes to be taken in account: uris for shapes needs to be defined; as shapes are reused and inherited, there is much more need to evaluate conflicting shapes. for the first effect, naming conventions could be applied; we have seen above an example “hasmin1max1shape-description” which explicit the cardinality and the name of the property but the name could be even opaque as “ps123”. in the second case, care has to be taken when there are several data-models in cascade owned by different organizations which define their own shacl shapes; of course it might happen that some of them conflict (properties could have different cardinalities or types). in this case, sparql itself could be helpful to identify conflicting shapes among each other: prefix sh: <http://www.w3.org/ns/shacl#> prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> select ?shape1 ?shape2 (count(?targetclass2) as ?distance) where { ?shape1 sh:targetclass ?targetclass1 . ?shape2 sh:targetclass ?targetclass3 . ?targetclass1 rdfs:subclassof* ?targetclass2 . ?targetclass2 rdfs:subclassof+ ?targetclass3 . ?shape1 sh:property ?p1 . ?p1 sh:path ?path1 . ?shape2 sh:property ?p2 . ?p2 sh:path ?path2 . ?p1 sh:mincount ?min1 . ?p2 sh:mincount ?min2 . filter (?path1 = ?path2) . filter (?min1 != ?min2) . filter (?targetclass1 != ?targetclass2) } group by ?shape1 ?shape2 order by desc(?distance) limit 100 listing 6. sparql query to find out shacl shapes in conflict on the minimum cardinality the sparql query in listing 6 will take all shapes that are applied to classes which are interlinked via rdfs:subclassof relations and check if the minimum cardinality (indicated by sh:mincount) are different, therefore checking a possible conflict; of course there could be other checks that could be done. applying inferencing and business rules before validating one of the benefits of using linked data is that rdf provides the basis for vocabularies, such as rdfs and owl, to define meaning for relations: rdfs (https://en.wikipedia.org/wiki/rdf_schema), describes concepts like subclass and sub properties; owl (https://en.wikipedia.org/wiki/web_ontology_language), defines new type of relations, such as inverse, transitive and symmetric. thanks to a rdfs or an owl reasoner (https://en.wikipedia.org/wiki/semantic_reasoner), new relations or properties could be generated and added to the original data (e.g if the “friendship” relation is defined as transitive and a is friend of b and b is friend of c, then a is friend of c, thus it would be possible to create a more interconnected graph). in addition, there have been different languages such as spin or swrl that allow defining business rules able to enrich the current data by creating new relations or properties. more recently, shacl allows to generate new relations by means of shacl rules (https://www.w3.org/tr/shacl-af/#rules). in both cases, adding new relations or properties might cause violation of the original data model, thus the need to perform validation after having enriched the data. however, as the enrichment might be an expensive operation in terms of time, it should be an optional task for a validator to perform. having a flexible user interface when performing a validation of rdf, one has to think who the end users are and how they might use such validation. a web interface is definitely good for end users, however, it requires manual validation (by means file upload, etc.) and, considering that rdf should be at first managed by machines, such validation mechanism should be automated as much as possible. thus the need to create a validation mechanism decoupled from the web interface and to which a web frontend could connect to. shacl validation can also run via a command line interface which could be integrated in a script to be run automatically. for example, topquadrant released a shacl library and a command line interface where the user can specify the data file and the shacl file to be checked against (https://github.com/topquadrant/shacl#command-line-usage). a web interface is desirable to manage the validation process which in turn could trigger a data manipulation process. a possible linked data validation pipeline having discussed the above points, i found out in apache nifi, an open source software framework, a valid data flow tool that could be easily put in place by people not having a technical background as: it has a web interface, therefore accessible through a web browser; it provides already many components (processors) that can be arbitrary connected between each other such as reading files in a folder, reading a file from url or provide a web service waiting for data to be sent over; it provides data lineage over the pipeline so the end user can see at any steps if data is valid and how it is enriched; it allows to export the configuration of the data flow to be reused and imported by other users apache nifi could be then used as basis for a linked data validation pipeline that could be connected to a web frontend for user input and reporting activities. for the purpose of this paper, five types of validator processor have been created: sparql, shacl, rdfs, owl, and custom rules. such processors are independent and can be put in any sequence the user needs, so the validation can run through different stages. if the data is valid for one of those processors, the same data is passed to the next processor; if data is not valid, an error is generated and sent to a dedicated processor that handles errors. further, each validator processor can connect to existing processors providing different way to the users to connect to. the sparql validator processor could be run to validate user data but also to run the query previously provided to check if the shacl shapes are in conflict between each other, thus the data modeler could use it. the shacl validator processor can make use of the shacl shapes and validate thanks to the open source topquadrant shacl api (https://github.com/topquadrant/shacl). the rdfs, owl and custom rules validator processors are based on jena reasoners (https://jena.apache.org/documentation/inference/) that enrich the data and check that there are no inconsistencies. in the image below a possible data flow is designed where, at any step, if the validation does not pass a validator processor, highlighted in green, the output is written to file saved to a folder. figure 1. data flow in apache nifi (enlarge) the source code of the validators has been released on github (https://github.com/emidiostani/shacl-processor) which can be compiled and reused directly in apache nifi. conclusions and future directions within this paper, the reader understands what are the implications of writing shacl shapes and the limitations of the current approaches. a possible solution to create linked data validation pipeline is provided. developers could integrate this pipeline within their frontend on one hand and to storage facilities such as triple stores once the data passed the validation, to a reporting mechanism or to trigger a data manipulation process. note [1] estimates differ, but experts think organizations spend between 10-30% of revenue handling data quality issues. ibm estimated the cost of poor quality data in the us in 2016 was $3.1 trillions. – source the dama guide to the data management body of knowledge (dama-dmbok) (https://technicspub.com/dmbok/) about the author emidio is a technical expert in the technology consulting in belgium. he is active in the fields of data management, semantic and technical interoperability as well as software engineering. he supports clients from designing data architecture, assist in the creation of data models and ensure data quality during etl processes. you can reach emidio via emidiostani@gmail.com or https://twitter.com/emidiostani. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – utilizing r and python for institutional repository daily jobs mission editorial committee process and structure code4lib issue 56, 2023-04-21 utilizing r and python for institutional repository daily jobs in recent years, the programming languages r and python have become very popular and are being used by many professions. however, they are not just limited to data scientists or programmers; they can also help librarians to perform many tasks more efficiently and possibly achieve goals that were almost impossible before. r and python are scripting languages, which means they are not very complicated. with minimal programming experience, a librarian can learn how to program in these languages and start to apply them to work. this article provides examples of how to use r and python to clean up metadata, resize images, and match transcripts with scanned images for the colorado state university institutional repository. introduction & background r is a programming language and software environment commonly used for statistical computing, data analysis, and graphical representation. it provides a wide variety of statistical (linear and nonlinear modeling, classical statistical tests, time-series analysis, classification, clustering, …) and graphical techniques, and is highly extensible.[1] python is a computer programming language often used to build websites and software, automate tasks, and conduct data analysis. it is a general-purpose language, meaning it can be used to create a variety of different programs and isn’t specialized for any specific problems. this versatility, along with its beginner-friendliness, has made it one of the most-used programming languages today.[2] in 2021, colorado state university libraries (csul) moved heritage materials from dspace to contentdm. to streamline the process, r and python scripts were used to handle several time-consuming tasks. tasks included resizing images in batch regardless of their orientation, matching transcripts with scanned images, creating pdf files based on file names, and building file structures for batch ingesting compound objects. setup if you’re new to programming, you might find it difficult to set up the compiling environment on your computer. thankfully, there are workshops and classes available that can teach you how to do it effectively and efficiently. for example, colorado state university collaborates with software carpentry (https://software-carpentry.org/) to offer the free workshops “programming with r” and “programming with python” to all employees. coursera has various python classes, including entry-level classes, for instance, “python for everyone” (https://www.coursera.org/specializations/python). these workshops and online courses normally provide detailed instructions on how to set up the environment for running these programs. if you run into problems, there will be someone to help. to run r programs, you need to install r software itself (http://ftp.ussg.iu.edu/cran/) and rstudio desktop (https://www.rstudio.com/products/rstudio/download/#download). to run a python script, you can use the python command in a terminal, which works on all platforms, including mac os, windows, and linux. you may also use anaconda, an open-source python distribution platform. for editing scripts, you can use jupyter notebook, atom, or even notepad. using r for metadata normalizations challenges csul previously hosted its cultural heritage materials, data, and traditional institutional repository materials in dspace, which adapts dublin core standards. for our institutional repository, qualified dublin core is our default metadata schema. for example, dc.contributor with qualifiers can be expressed as: dc.contributor.author dc.contributor.advisor dc.contributor.committeemember at the time of migration, this element had more than 20 qualifiers, which was found to be unsustainable. the migration team decided to adopt the resource description and access (rda) guidelines[3] for descriptive cataloging. for example, in dspace one record’s author field was expressed as: dc.contributor.author bright, donald e. in contentdm this field would be expressed as: dc.creator bright, donald e., author with more than 90,000 records to clean up in two weeks, there was no way this could have been done manually. the complexity of the metadata also meant that out-of-the-box software could not be used. therefore r was chosen for this task because of its data manipulation abilities. solution exported all collections metadata from dspace into csv files. used an r script to read csv files and convert dc.contributor.* to dc.creator. updated collections in dspace using the newly generated csv files. migrated collections from dspace to contentdm (performed by oclc). how the r script works behind the scenes reads an exported collections’ metadata file, creates a copy, and adds the dc.creator column to the end if it doesn’t already exist. scans files row by row. if a dc.contributor.* entry is found, manipulate the entry, place the new data in its dc.creator cell, and then delete data in the original dc.contributor cell. scans for next dc.contributor entry until all dc.contributor.* columns are processed and move to the next record. loops through and processes the next collection’s file in the folder until all csv files have been processed. all names from dc.contributor.(qualifier) columns were modified and moved to the dc.creator columns. some entries had only one author while others could have multiple authors with different qualifiers. the most difficult column to clean is dc.contributor.author. when items are added to dspace, the system automatically attaches universally unique identifiers, uuids to the entry. uuid is used to identify information across a computer system.[4] for example, ff4d344e-a2c5-4bfc-8271-553b416db8f0 is a uuid. the underlined example record in figure 1 has three authors: albertson, maurice l.::ff4d344e-a2c5-4bfc-8271-553b416db8f0::500||birky-kreutzer, pauline::a588f279-d14b-4941-87f6-fc1ed5741ac6::500 and peach corps (u.s.) at the time of migration, we could not find software to clean up entries with uuids efficiently. however, looking closely, we found patterns. uuids begin with “::” followed by either 39 or 41 alphanumeric characters. if there are multiple authors, their names are separated by “||”. a programming script can easily extract names from these entries that contain uuid. additional “dc.contributor” fields will be appended to the end of the entry if present. the new creator filed would be: albertson, maurice l., author|| birky-kreutzer, pauline, author||peace corps(u.s.), institution figure 1. comparison of the original and the newly created metadata files. running r code using rstudio an r script is essentially a text file that can be executed in two ways: through the use of rstudio or via a command prompt. we chose to create and execute r code in rstudio, a free, open-source ide (integrated development environment) for r. its interface is organized so that the user can clearly view graphs, data tables, r code, and output all at the same time.[5] the screenshot below displays two sections. the top section is the code display window, while the bottom section shows the running processes. any errors, warnings, or print statements will be displayed in the bottom section as well. figure 2. rstudio interface sample code r has many modules that come with useful functions. for this project, “sjmisc” and “stringr” modules were imported. library(sjmisc) library(stringr) the “setwd” function is used to set a working directory that will house all original csv files and newly created ones. the “list.files” built-in function retrieves all files from a specified folder. setwd("~/desktop/r/batch1") file_list <list.files() the following commands read in csv files. orgdata <read.csv(file_list[[z]]) newdata <orgdata[,names(orgdata)!=""] this command defines a function to modify author names. add_new_authors <function(authorlist, column, title){ for (c in 1:total_rows){ … } to extract author names, the “regexp” function,[6] is used to find the positions of “::”. positions < gregexpr(pattern='::', org_names, ignore.case=false) the “substr” function is used to extract author names from dc.contributor.author field. authorname <substr(org_names,name_first_char, name_last_char) the “paste0” function is used to attach “, author” to an author name. authorlist[[c]] <paste0(authorname,", author") for a reason we could not determine, dspace sometimes creates columns with “[]”. for example, one spreadsheet might have dc.contributor.author and/or dc.contributor.author[] columns. to check for the existence of these columns, we use the “%in%” operator, which is designed to determine if an element belongs to a vector or dataframe. for this script, each csv file is read in as a dataframe. to check if a column with “[]” exists or not, r uses “..” instead of “[]”. therefore to check if the dc.contributor.author[] column exists or not, “dc.contributor.author..” %in% is used. if these columns exist, it calls the “add_new_authors” function to create an entry for the dc.creator column. if("dc.contributor.author" %in% colnames(newdata)) { clist <add_new_authors(clist, newdata$dc.contributor.author,"author") newdata$dc.contributor.author <"" } if("dc.contributor.author.." %in% colnames(newdata)) { clist <add_new_authors(clist, newdata$dc.contributor.author..,"author") newdata$dc.contributor.author.. <"" } after we had the data for the creator column, we used the “write.csv” function to create a new file. the new file name will have “creators-” prefix. newfilename <paste0("creators-",file_list[[z]]) write.csv(newdata, file = newfilename, row.names = false) improvements this script was the first script the author wrote in r, so it is by no means perfect. there could be many improvements, especially error handling. though, even an imperfect script could process a collection with thousands of entries in several seconds and it helped us to meet the migration deadline. using python to resize images contentdm provides an extension that enables the project client, a platform for adding content to contentdm site, to generate file transcripts by using optical character recognition (ocr).[7] to use the extension, a separate license needs to be purchased and each license only allows 10,000 pages per month. if a scanned image is measured bigger than 8.27 by 11.69 inches, it will be counted as two or more pages depending on its actual size. scanned a4 size documents with extra edges exceed the specification and would be counted as multiple pages. unfortunately, this counting behavior caused us to run out of our monthly license quota quickly, resulting in long wait times for ingesting new items. to avoid this issue, we need to resize these images. while adobe photoshop and bridge have batch resizing functions, they cannot resize based on page orientation or process multiple folders. additionally, adobe lightroom can resize images by long side, but it requires importing images to its library, which can be time-consuming. therefore, we developed a simple python script that can resize images based on page orientation from multiple folders and create new folders to hold the resized images. human-created vs. ai-generated code later, we tried to use chatgpt to generate a script to achieve the same goal. however, we had to modify the input to the ai several times before it produced a script that met our desired functionality and user experience. additionally, we had to relocate one line of code to a different location to improve its efficiency. command for chatgpt write a python script to resize images based on the long side. prompt for a directory with original images to be resized. the original directory can have sub-directories that contain images that need to be resized. prompt for the root folder name which will be used to hold all resized images. the new folder should be in the same root directory as the to-be-resized folder. if the original directory has subdirectories, create sub-directories with the same names. keep directory structure. original images can be .jpg .jpeg .png .tif or.tiff. resized images should be saved as .jpg. prompt for size in inches to be used for resizing. all images will be resized at 300 dpi and the long side length will be the user-provided value. we discovered that even if the initial attempt does not succeed, we can still refer to the ai-generated code and modify it to achieve the desired outcome more efficiently. human generated code import os from pil import image, imageops image.max_image_pixels = none # prompt for inputs org_dir = input('input the orginal folder name.') print(org_dir) resized_dir = input('input a folder name for resized images.') print(resized_dir) os.mkdir(f'{resized_dir}') dir_list = os.listdir(org_dir) input_long_side = input('input long side length in inches. (e.g. 8,9,10 or 11): ') long_side = int(input_long_side) long_side_pixels = long_side * 300 # batch process. will process a directory with sub-directories. for subdir in dir_list: print('now processing folder: ', subdir, 'please wait...') os.mkdir(f'{resized_dir}/{subdir}') og = f'{org_dir}/{subdir}' orgscans = os.listdir(og) for orgimage in orgscans: orgfilename = f'{org_dir}/{subdir}/{orgimage}' newfilename = f'{resized_dir}/{subdir}/{orgimage}' if(orgimage.startswith('.')): print('ignore a hidden file: ', orgfilename, 'still processing files. please wait...') elif(orgimage == 'thumbs.db'): print('ignore file: ', orgfilename, 'still processing files. please wait...') else: newjpg = newfilename.split(".") resizedjpg = newjpg[0] + ".jpg" img = image.open(orgfilename) width,height = img.size area = width * height if (area >= 89478485): print('file: ', orgfilename, 'exceeds pixel limitation and would not be converted. please convert it manually.') else: rgb_image = img.convert('rgb') resizedimage = imageops.contain(rgb_image, (long_side_pixels,long_side_pixels)) resizedimage.save(resizedjpg, dpi=(300, 300)) print('total processed ', len(dir_list), 'folders.') ai-generated code import os from pil import image # prompt for original directory orig_dir = input("enter the path of the directory containing original images to be resized: ") # prompt for root folder name for resized images root_folder_name = input("enter the name of the root folder to hold all resized images: ") root_folder_path = os.path.join(os.path.dirname(orig_dir), root_folder_name) long_side_inches = float(input("enter the size in inches for the long side of a resized image: ")) # create root folder if it doesn't already exist if not os.path.exists(root_folder_path): os.mkdir(root_folder_path) # walk through the original directory and its subdirectories for dirpath, dirnames, filenames in os.walk(orig_dir): # create corresponding subdirectories in the root folder subfolder_path = os.path.join(root_folder_path, os.path.relpath(dirpath, orig_dir)) if not os.path.exists(subfolder_path): os.mkdir(subfolder_path) # loop through image files in current directory for filename in filenames: # check if file is an image if filename.lower().endswith(('.jpg', '.jpeg', '.png', '.tif', '.tiff')): # open image and get dimensions filepath = os.path.join(dirpath, filename) img = image.open(filepath) width, height = img.size # determine new dimensions based on user input and maintain aspect ratio if width >= height: new_width = int(long_side_inches * 300) new_height = int(height * new_width / width) else: new_height = int(long_side_inches * 300) new_width = int(width * new_height / height) # resize image and save as .jpg resized_img = img.resize((new_width, new_height)) new_filepath = os.path.join(subfolder_path, filename) new_filepath = os.path.splitext(new_filepath)[0] + '.jpg' resized_img.save(new_filepath, format='jpeg') using python to process scanned documents with transcript csul’s institutional repository contains many digitized personal diaries and hand-written letters. since these documents are not suitable for optical character recognition (ocr) processes, they were transcribed by volunteers or student employees. these scanned documents and their transcripts files were imported into dspace as pdf files. users must download both files to their computers and use two browser windows to view them side-by-side. one great feature of contentdm is displaying transcripts alongside scanned images, which significantly improves user-friendliness. figure 3. transcript display in contentdm. to ingest the document illustrated in figure 3, we chose the ingest method, a compound object with a directory structure. for each compound object, there is an item-level folder that contains: an image folder that stores all page images. a transcript folder that includes all page-level transcript txt files. an item-level metadata txt file. original files for each entry, we have a pdf file, a transcript txt file, and a collection-level metadata csv file that was exported from dspace. the collection-level metadata file contains metadata not only for this record but also for dozens, or even thousands, of other records. figure 4. original files for an entry with transcript in dspace manual process doing this manually would require the following work for each document: create three folders, one item-level folder, an images folder, and a transcript folder. export a pdf to images and place them in the image folder. copy and paste text from the original transcript file into page-level transcript files. the most time-consuming step. figure 5. creating page-level transcript find corresponding rows for this item. each item has three rows of data in the collection-level metadata file. merge data into one row and use it to create an item-level metadata file which is a tab-delimited txt file. figure 6. metadata migration this manual process would be extremely time-consuming, prone to numerous human errors, and ultimately result in the failure of the ingest process. how the python script works to streamline and process multiple items at once, we utilized a python script to automate these steps. the script takes three parameters. the name of a folder that contains a set of pdfs. the name of a folder that contains corresponding item-level transcript files. these transcripts have not been split into individual transcriptions at page-level. a collection-level metadata csv file. figure 7. python command line to run the script. the script will: create a batch ingest folder that contains compound object folders. export pdfs to images and save them to corresponding “scans” folders. read in master transcript files, split them into page-level txt files, and name them according to the corresponding image names. create an item-level metadata file by extracting data from the collection-level metadata file. figure 8. inputs and outputs of the script. original files script’s output import modules in python, a module can be considered a code library that contains functions to be used in your scripts. for this program, the following libraries were used. import os import csv import pandas as pd from pdf2image import convert_from_path input and new directories the code below takes three parameters, finds all pdf files, and creates a new directory called “batchingest” to store all compound objects. pdir = input('input a directory with original pdfs:') orgtransdir = input('input a directory with original transcript .txt files:') orgmetadata = input('input the master metadata file name in csv format:') dir_list = os.listdir(pdir) subdir = "batchingest" os.mkdir(subdir) a loop is used to create the batch ingest structures. it creates folders based on pdf file names. each pdf file will result in a folder with the same name and inside of this folder will have two subfolders, one is for holding page-level images and another is for page-level transcript files. for file in dir_list: y = file.strip(".pdf") os.mkdir(f'{subdir}/{y}') os.mkdir(f'{subdir}/{y}/scans') os.mkdir(f'{subdir}/{y}/transcripts') convert pdf files to jpg images to convert pdf files to images, you need to install two python modules: pdf2image and poppler. the pdf2image module converts a pdf file to a pil object, while the poppler module allows you to read, render, or modify pdf documents. windows users will have to build or download poppler for windows.[8] go to https://github.com/oschwartz10612/poppler-windows/releases/, download the zip file, and unzip it to any location you like. we placed it at the top level of the c drive. a detailed tutorial on how to install poppler can be found at https://pdf2image.readthedocs.io/en/latest/installation.html. a useful tutorial, convert pdf to image using python, can be found here https://www.geeksforgeeks.org/convert-pdf-to-image-using-python/. we included the following two lines of code in our script. from pdf2image import convert_from_path poppath = r"c:\poppler-22.12.0\library\bin" our “pdftoimage” function converts a pdf file to jpg images using the “convert_from_path” function from pdf2image module. to adhere to the standard naming convention employed at csul, we utilized the “zfill(4)” function to generate four-digit numbers. as a result, instead of producing “_page_15.jpg” the script generates “_page_0015.jpg”. def pdftoimages(pdfdir, pdffile, imgdir): # omitted code images = convert_from_path(pdf_file_name, poppler_path=poppath) for i in range(len(images)): # omitted code images[i].save(f'{imgdir}/{file_name_preceeding}_page_' + str(j).zfill(4)+'.jpg', 'jpeg') generate transcript files each pdf file has a corresponding transcript file. at csul, it was decided that 4 consecutive blank lines would signify the start of a new page. the script reads in the original file line-by-line, appends each line to a new string, and keeps track of empty lines. once four consecutive empty lines are detected, a new txt file is generated. for example, the image wreg03014_page_0001.jpg will have a corresponding transcript file, wreg03014_page_0001.txt. f = open(orgtrans, 'r') newfile = [] for file in imagenames: newfile = file.replace('jpg','txt') … if(len(aline) == 1): # find empty lines linebreakcount += 1 aline = f.readline() if not aline: #reach the end of file. newfile.append(aline) generate metadata file first, the script reads a comma-separated value csv file into a dataframe named “orgmd” using a built-in function, “read_csv”. then it employs a built-in function, “index”, to locate the row number where a corresponding file name is present. next, it finds the row that contains the matching pdf file name by using the “loc” property, and merges data and assigns the resulting value to “resultrow”. finally, it writes this row to a new tab-delimited txt file. the “to_csv” function writes an object to a file in either csv or txt format with comma-separated values. orgmd = pd.read_csv(orgmetadata, encoding= 'unicode_escape', low_memory=false) matchingrowindex = orgmd.index[orgmd['file name'] == file] resultrow = orgmd.loc[[matchingrowindex]] mdrow = int(matchingrowindex[0])+2 resultrow = orgmd.loc[[mdrow],'title':'file name'] csvfilename = f'{file.strip(".pdf")}.txt' tn = f'{csvpath}{csvfilename}' resultrow.to_csv(exportedfilename, mode='a', sep ='\t', index=false, encoding='utf-8') conclusion by utilizing custom r and python scripts, the csul repository librarians have saved many hours of work and encountered fewer human errors. besides practical instances discussed in this article, csul also used r and python scripts for gis data analysis and post-ingest verifications. although commonly used by data scientists, computer and electrical engineers, and geospatial scientists, these languages are also useful for librarians. we highly encourage librarians to take online courses or free workshops to learn these skills, and then apply them to simpler projects. as more experience is gained, they will find many more opportunities to use these programming languages. endnotes [1] “what is r?” the r foundation, accessed december 1, 2022, https://www.r-project.org/about.html. [2] “what is python?” coursera inc., accessed december 12, 2022, https://www.coursera.org/articles/what-is-python-used-for-a-beginners-guide-to-using-python. [3] “rda, resource description & access toolkit”, american library association, accessed december 12, 2022, https://www.rdatoolkit.org/about#:~:text=rda%3a%20resource%20description%20and%20access,user%2dfocused%20linked%20data%20applications. [4] steele, nick. “breaking down uuids”, accessed december 12, 2022, https://duo.com/labs/tech-notes/breaking-down-uuids#:~:text=universally%20unique%20identifiers%2c%20or%20uuids,both%20the%20ietf%20and%20itu. [5] “statistical & qualitative data analysis software: about r and rstudio”, 26, october er, 2022, accessed december 12, 2022, https://libguides.library.kent.edu/statconsulting/r. [6] “gregexpr: extended gregexpr with substring retrieval”, rdocumentation.org, accessed december 12, 2022, https://www.rdocumentation.org/packages/micropan/versions/1.0/topics/gregexpr. [7] “ocr”, oclc, 1 january, 2023, accessed march 23, 2023, https://help.oclc.org/metadata_services/contentdm/project_client/enter_metadata/ocr?sl=en. [8 kumar_satyam. “convert pdf to image using python.” 21 jan, 2021, accessed december 9, 2022, https://www.geeksforgeeks.org/convert-pdf-to-image-using-python/. about the author yongli zhou is the digital repository librarian at colorado state university libraries (csul). she has been involved in several migrations of the institutional repository during her tenure, including a transition from contentdm to digitool, then to dspace, and back to contentdm. she was also involved in integrating the institutional repository into the library’s discovery systems. she is passionate about using new technologies, including ai, to simplify system migration and enhance discoverability for digital collections. subscribe to comments: for this article | for all articles one response to "utilizing r and python for institutional repository daily jobs" please leave a response below: carolyn sullivan, 2023-07-17 hello yongli, thank you for this article and for providing your code. i’m curious though–how did your institution come to select contentdm as a dams over dspace? it surprises me since most libraries over the past decade seem to be adopting dspace, and i’d read a few articles that seemed to suggest folks had previously found contentdm problematic (ie. ‘breaking up with contentdm’, code4lib 2013, gilbert & mobley; ‘contentdm to digital commons’, journal of archival organization 2018, mita, pelli, reamer & ince). has contentdm added any interesting features to their platform or corrected any existing issues? thanks for any info you can provide! leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – open source tools for scaling data curation at qdr mission editorial committee process and structure code4lib issue 49, 2020-08-10 open source tools for scaling data curation at qdr this paper describes the development of services and tools for scaling data curation services at the qualitative data repository (qdr). through a set of open-source tools, semi-automated workflows, and extensions to the dataverse platform, our team has built services for curators to efficiently and effectively publish collections of qualitatively derived data. the contributions we seek to make in this paper are as follows: 1. we describe ‘human-in-the-loop’ curation and the tools that facilitate this model at qdr; 2. we provide an in-depth discussion of the design and implementation of these tools, including applications specific to the dataverse software repository, as well as standalone archiving tools written in r; and 3. we highlight the role of providing a service layer for data discovery and accessibility of qualitative data. keywords: data curation; open-source; qualitative data by nicholas weber, sebastian karcher, and james myers introduction the automation of curation tasks – including portions of the ingest and transformation of submitted data to a repository – is crucial for scaling services to researchers that are managing increasingly complex and large data collections [1]. however, the diverse characteristics of qualitative data often limit the applicability of fully automated tools. moreover, when data include privacy sensitive information, many automated processes require a curator’s intervention to mediate, check quality, and evaluate compliance with responsible practices for collecting and properly handling personally identifiable information (pii) [2]. at the qualitative data repository (qdr), we have started to design curatorial interventions that act as a “human-in-the-loop” to the automation and verification of different components of our data repository infrastructure. practically, this takes the form of tools that handle mundane preparation of data at the point of ingest and provide graphic user interfaces to apis that allow curators to publish different aspects of a data collection unique to qdr. in the following paper, we describe four different initiatives and the way that qdr uses this model of human-in-the-loop curation to efficiently publish qualitative data at scale. the three contributions we seek to make in this paper are as follows: we first describe ‘human-in-the-loop’ curation and the tools that facilitate this model at qdr; we then provide an in-depth discussion of the design and implementation of these tools, including applications specific to the dataverse software repository, as well as standalone archiving tools written in r; and finally, in describing each tool we briefly highlight the role of providing a service layer for data discovery and accessibility of qualitative data at qdr. human-in-the-loop curation in a repository setting, data curation often begins with the transform of a submission information package – including data, metadata, and information documentation – into an archival information package. this transformation (from sip to aip) may include manual or semi-automated tasks performed by a curation staff, such as data normalization, cleaning, and file migration, as well as fully automated workflows that can ingest, perform preservation actions, and prepare an archival information package for reliable long-term storage. reducing the number of manual tasks that need to be carried out by curators is an important and long-standing challenge for repository developers [3]. early innovations with workflow automation, such as rule-based systems for automating preservation tasks [4], were important preemptive steps towards automation in a curation and preservation setting. further, by building assistive technologies that can support practical curation work repositories can reduce the time, effort, and financial costs of publishing data collections. enabling automation of manual tasks also enables curation services to scale with the complexity and size of data that are generated by researchers. despite the promise of large-scale automation, a number of scenarios require the purposeful intervention of a curator. these include consultation with depositors as well as the ingest and publication of privacy sensitive data that contains personally identifiable information about human participants. these scenarios require deliberate decisions that are heavily dependent upon the context and content of data. a key emerging challenge is the development of assistive curation technologies that can take advantage of affordances offered by workflow automation, and enable curation interventions [5]. human-in-the-loop systems attempt to do exactly this – they provide a means for human curators to avoid (or at least significantly reduce) the amount of time spent on repetitive, automatable tasks while keeping the entire curatorial process under human supervision. a crucial component of human-in-the-loop curation is its modularity: individual components of the curation process can be removed from automation should data or context call for a manual approach. in this vein, we see the development of tools that can effectively help curators to manage deposits, transform and package data, and provide reliable means to prepare data for long-term preservation as vital to the future sustainability of large scale data collections. in the following section, we review a set of open-source tools developed to assist curators at the qualitative data repository. for each tool, we describe its development goal, the way in which the tool helps curators perform curation tasks, and the relation to qdr’s overall archiving workflow. overview of tools and approach to facilitating human-in-the-loop curation at qdr in the following section, we review four tools that help qdr to practice human-in-the-loop curation: dvcurator, a tool to automate processing and documentation of data deposits; archivr, a tool that automates the transformation of hyperlinks into persistent web-identifiers; annotation-fetcher, an interface for the dataverse platform that allows web-based annotations to be archived; and, dv-previewers, a set of technologies that allow multimedia files to be viewed within a repository. data ingest: dvcurator [6] one of the rote tasks that data curators at qdr must perform is the copying of submitted data projects to local machines for manual curation tasks (e.g. checking for pii anonymization, reformatting files, changing file names to a common standard, etc) and to systematically document and track those steps within a multi-person curation team. to automate portions of this initial process, we have written a series of r scripts that will simultaneously fetch submitted data from qdr’s dataverse, create local and back-up copies in a cloud storage environment, log a set of general issues on our github repository for tracking curation workflows, and then extract metadata for creating a compliant .tsv to re-upload to qdr’s dataverse at publication. dvcurator’s main function start_curation() initiates the process of curation by creating github issues for standard curation tasks and associating them with a github project for the duration of the data project. the github issues track individual curation tasks using checkboxes to track and issue comments to document curation steps. the project board, a simple kanban-style board with three columns, allows for quick assessment of curation status and access to individual issues (see figure 1). figure 1. github curation project board for a sample data project dvcurator then automatically creates a local curation folder (on the curators desktop) with subfolders for original deposit and qdr prepared versions of the dataset. next, using the dataverse api dvcurator fetches a .zip file for the full data project and populates the original deposit folder and an unzipped version to qdr prepared for further curation. this includes a list of files with associated metadata as a .csv file to the same folder. the tool also implements an entirely separate function, datasets_bydate() that creates an .html file with citations of all datasets published within a specified date range, designed to facilitate reporting for curation staff. collectively these functions allow qdr to automate most of the pre-processing of a data collection and ensure consistent application of steps for beginning a curation protocol for newly deposited data at qdr. data preparation: archivr [7] submission information packages at qdr often contain links and references to diverse web resources. this may include, for example, policy documents that live at a particular web address, or gray literature such as blogs and other informal communications. as an authoritative repository tasked with the long-term accessibility of both data and documentation related to scholarly research qdr requires web-archiving for these resources. archivr is a project developed by qdr to create persistent copies of cited web-resources and transform web-links with data documentation into persistent identifiers. the basic function is archive that takes a list of urls and stores them in the internet archive’s ‘wayback machine’ for long-term archiving. given a url to a web-resource, archivr will return a dataframe in r containing the callback data for the service. the r dataframe consists of, simply, the service used and the corresponding new archived identifier (e.g. in the code snippet below the url ‘www.example.com‘ is archived in the ‘way back machine’ and a dataframe containing the new identifier is returned). arc_df <archiv(list("www.example.com", "notaurl", "www.github.com")) arc_df$way_back_url # wayback_url # 1 http://web.archive.org/web/20190128171132/http://www.example.com # 2 url not found # 3 http://web.archive.org/web/20190128171134/https://github.com/... code snippet 1. an example r dataframe returned by archivr archivr will also archive all urls in a text file (including docx, pdf and plain text formats such as html, xml, and markdown). to allow for pre-processing of urls before archiving, archivr also provides access to the functions used to extract urls from a webpage (extract_urls_from_webpage("url")) and from a file (extract_urls_from_text("filepath")), and includes an except argument as part of archive that excludes a url pattern, defined by a regular expression from archiving (this could be used, for example, to avoid unnecessarily archiving digital object identifiers (dois). archivr also works with perma.cc’s archiving service – and qdr principally uses perma.cc for link archiving. an api key is required for this service[8]. the archivr tool allows qdr to prepare data collections with hyperlinked text and web-resources to be automatically archived using available services (including for projects using hundreds of web sources such as [9] – allowing curators to focus time and attention on other publishing needs, while simultaneously guaranteeing that content in data collections remain persistent and accessible for the long-term. data publication: annotation fetcher over the last two years qdr has been involved in a collaboration with the platform hypothes.is [10] to extend the capability of using web-annotations to facilitate transparent social science research. the collaboration, annotation for transparent inquiry (ati), allows authors to persistently link data to existing or planned digital publications with a web-annotation, and archive related files (datasets, source documents, or even software) in qdr’s dataverse [11]. curating ati projects currently requires data curators at qdr to manually transform comments made in microsoft word or pdf documents into linked annotations (in html) using the hypothesis web client. the annotations, when displayed in a browser, are linked to a rendered web page (see figure 2). figure 2. annotations made to a scholarly article. in the left panel, an html rendering of the text that appears on the web has been annotated using the hypothesis client that appears on the right. annotation content is displayed within the right-hand panel. the html-based web-annotations contain linked documents and other data sources that are stored in qdr’s dataverse. however, the content of the annotations (as we as the anchored text that is being annotated) remains external to qdr as html web pages. to retrieve and store this content we designed ‘annotation fetcher’ – a gui interface tool for dataverse that allows a curator to call the hypothes.is api, retrieve the annotation content and anchors (as a json file), and archive the annotations within a dataverse. since annotations are stored as plain-text (retrieved via the hypothes.is api as json) they can also be indexed, and searched for within qdr’s dataverse as part of a data collection (containing the original dataset, the annotations, and any documentation that is related to the collection) – thus transforming annotations into first-class scholarly research objects that are not only discoverable but also citable through dataverse [12]. figure 3. annotations archived in qdr’s dataverse can be searched for and discovered as part of a data collection. using the dv-previewer, the annotation content and anchor can also be viewed within dataverse. data discovery: dv-previewer [13] until recently, the only way for end-users to access images, audio recordings, videos, and other multi-media in a dataverse-based repository, was to download and open files on a local desktop. this results in an undue burden for end-users when browsing or exploring new data collections. to improve access and discovery qdr developed a set of data previewers that conform to the dataverse external tools interface. dr-previewer tools are lightweight wrappers around standard html5 functionality (e.g. audio, video), or third-party libraries (pdf) or some combination (e.g. standard image displays with a third-party library to allow zooming, simple text/html displays with third-party libraries used to sanitize content to avoid security issues). previewers can be used without individual data repositories needing to download and configure local copies, but by simply running a set of curl commands to register the previewer with a local dataverse instance – and there is just one command per mime-type (i.e. multiple commands to cover different types of images [14].) importantly, to preview restricted content, a user must have permission to view the relevant dataset version and download the relevant file and must have created an api token for themselves in dataverse. at qdr we provide a simple button to generate an appropriately credentialed api token corresponding to the registered user’s access rights. more recently, the dvpreviewers have been adopted by the global dataverse community consortium and are now being used in multiple dataverse installations. in combination, both dv-previewer and annotation fetcher allow curation staff to easily implement and make available unique data sources for viewing and consumption. conclusion through the development of open-source tools to facilitate meaningful curatorial interventions, we remain optimistic that challenges in preserving qualitative data, including privacy and protecting sensitive data can be overcome. this paper has focused in particular on ways in which the development team at qdr is using tools in conjunction with the dataverse repository platform to facilitate “human-in-the-loop” curation tasks that enable efficient, reproducible, and contextually meaningful work with data. by off-loading repetitive tasks, and creating easy access to api methods, qdr’s curation staff can focus time and attention on preparing sensitive data for publication, and preserving valuable data collections over the long-term. notes [1] plale, b. a., dickson, e., kouper, i., liyanage, s. h., ma, y., mcdonald, r. h., … & withana, s. (2019). safe open science for restricted data. data and information management, 3(1), 50-60. [2] foster, i. (2018). research infrastructure for the safe analysis of sensitive data. the annals of the american academy of political and social science, 675(1), 102-120. [3] blanke, t., & hedges, m. (2013). scholarly primitives: building institutional infrastructure for humanities e-science. future generation computer systems, 29(2), 654-661. [4] westerlund, p., andersson, i., päivärinta, t., & nilsson, j. (2019). towards automated pre-ingest workflow for bridging information systems and digital preservation services. records management journal. [5] foster, i. (2018) [6] repository with code and documentation for dvcurator: https://github.com/qualitativedatarepository/dvcurator [7] repository with code and documentation for archivr: https://github.com/qualitativedatarepository/archivr [8] information on how to obtain an api key for perma.cc can be found in the developer documentation: https://perma.cc/docs/developer#api-key [9] wedeen, l. (2019). data for: authoritarian apprehensions: ideology, judgment, and mourning in syria [data set]. qdr main collection. https://doi.org/10.5064/f63776w4 [10] hypothes.is https://web.hypothes.is/ [11] elman, c., & kapiszewski, d. (2018). the qualitative data repository’s annotation for transparent inquiry (ati) initiative. ps: political science & politics, 51(1), 3-6., karcher, s., & weber, n. (2019). annotation for transparent inquiry: transparent data and analysis for qualitative research. iassist quarterly, 43(2), 1-9. [12] as an additional example, the following project contains an annotation file that is discoverable on the qdr dataverse, and is also previewable within a browser: snyder, jack. 2015. "data for: “russia: the politics and psychology of overcommitment,” in: the ideology of the offensive: military decision making and the disasters of 1914". qualitative data repository. https://doi.org/10.5064/f6kw5cxs. qdr main collection. v3 [13] repository with dv previewers code and installation instructions: https://github.com/qualitativedatarepository/dataverse-previewers [14] for specific installations see https://github.com/qualitativedatarepository/dataverse-previewers#installation about the authors nicholas weber (nmweber@uw.edu) is an assistant professor at the university of washington, and the technical director at the qualitative data repository. he holds an mlis and ph.d. in information science from the university of illinois. his research focuses on the design, implementation, and use of data infrastructures, and scientific software sustainability. sebastian karcher (skarcher@syr.edu) is the associate director of the qualitative data repository and a research assistant professor in the department of political science at syracuse university. he holds a ph.d. in political science from northwestern university. his research interests include qualitative data management and the development of scholarly workflows. james myers (qqmyers@hotmail.com) is scientific software researcher and developer, and currently serves as the lead developer at the qualitative data repository. he holds a ph.d. in experimental chemistry from the university of california, berkeley. over the last two decades, jim has worked in applied computer science, creating cutting-edge scientific "cyberinfrastructure" and working to understand the socio-technical issues that contribute to the ultimate success or failure of scientific software efforts. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – leveraging python to improve ebook metadata selection, ingest, and management mission editorial committee process and structure code4lib issue 38, 2017-10-18 leveraging python to improve ebook metadata selection, ingest, and management libraries face many challenges in managing descriptive metadata for ebooks, including quality control, completeness of coverage, and ongoing management. the recent emergence of library management systems that automatically provide descriptive metadata for e-resources activated in system knowledge bases means that ebook management models are moving toward both greater efficiency and more complex implementation and maintenance choices. automated and data-driven processes for ebook management have always been desirable, but in the current environment, they become necessary. in addition to initial selection of a record source, automation can be applied to quality control processes and ongoing maintenance in order to keep manual, eyes-on work to a minimum while providing the best possible discovery and access. in this article, we describe how we are using python scripts to address these challenges. by kelly thompson and stacie traill introduction over the past decade, vendor-packaged ebooks — once a small component of most academic library collections — have become staple resources. this transformation has come with substantial challenges for libraries in effective management of descriptive marc metadata for ebooks. challenges identified in the literature include quality control, completeness of coverage, scalability of workflows, and ongoing management. (mugridge and edmunds, 2012), (traill, 2013). solutions and workflows that address these problems are often both labor-intensive and specific to an individual library. academic libraries ranging from small to very large have shared batch processing workflows for ebooks that vary widely, and largely focus on how to manage vendor-supplied marc metadata. (mugridge and edmunds, 2009), (martin and mundle, 2010), (wu and mitchell, 2010), (hodge, et al., 2013), (sapon-white, 2014), (chen, et al., 2016). partial automation often plays a role in these workflows, especially for the tasks of record normalization, error checking, and access checking. (lybarger, 2013), (spurgin, 2014), (fenichel, [accessed 2017]). the recent emergence of library management systems (lmss) that provide access to descriptive metadata for e-resources activated in system knowledge bases means that ebook management models are moving toward both greater efficiency and more complex implementation and maintenance choices. after many years of relying on vendor-supplied marc records as the sole source of descriptive metadata for many ebook collections, the availability of other metadata supply streams is welcome. records supplied directly by vendors have long been the most popular option for the provision of catalog and discovery access, often because such records offered the only feasible route for library discovery. the development of library services platforms, like alma (ex libris) and worldshare (oclc), that integrate electronic resources management functions with traditional ils functions means that multiple sources for e-resource metadata are now available to many libraries. but these choices come with serious implications for ebook management workflows: both the initial selection of a marc record source and the ongoing management of ebook metadata streams present new challenges for many libraries. the integration of knowledge bases with linked marc records into lmss is intended to increase efficiency by eliminating the need to load (and periodically update or delete) marc record batches for electronic resources. however, in practice, this integration means that libraries often have multiple options for procuring descriptive metadata for ebooks where previously there was only one choice. depending on local standards and practices, metadata managers may need to evaluate multiple record sources for each new collection, because automatically-supplied records may not be the best choice for discovery and access. but there is a cost incurred if a library prioritizes using the best metadata, regardless of source: ongoing management for collections with non-lms metadata sources must then be handled either manually or by processes outside the lms. as ebook collections grow (and libraries’ staff shrink), management processes must also be scalable. automated and data-driven processes for ebook management have always been desirable. in the current environment with its complex implementation choices and demanding maintenance workflows, automation is essential to make these choices and tasks manageable. this paper will discuss the solutions to these problems currently in use at the university of minnesota libraries, where we rely on a combination of python scripting and alma’s batch import capabilities to manage marc metadata for ebooks. we first present a use case in which python scripts aid selection of the best metadata source among record providers offering a marc record set for a given electronic collection. sources evaluated include worldshare collection manager (wcm), the alma community zone (acz), and content providers that directly supply marc records for their collections. using “markers” or “flags” indicating record completeness, we are able to quickly determine aggregate quality of a record batch in order to make data-driven choices about the source of the metadata. we also discuss python scripts developed to assess record quality indicators once an automated workflow has been set up to import those records. our case study involves both new records and record updates delivered from wcm via ftp and automatically imported into alma. we will discuss how we programmatically evaluate record quality indicators and use them to split files into batches for either automated processing or human intervention. this process allows the majority of records to be loaded without any eyes-on work and produces manageable lists of problem records for manual investigation and remediation, allowing staff to spend targeted time on remediation activities with the highest impact on discovery and access. background the implementation of alma at the university of minnesota libraries (umn) in late 2013 was followed less than a year later by oclc’s shift to worldshare collection manager for bulk record delivery as they began the phase-out of previous-generation record delivery services. both of these systems are designed to provide discovery metadata for electronic resources activated in their respective knowledge bases: alma does so by default for resources activated in its community zone, while worldshare collection manager offers opt-in marc record delivery of worldcat bibliographic records linked to active knowledge base entries. by late 2014, it was clear to umn staff that for at least some ebook collections, it would be necessary to choose a metadata source. what was not clear was how best to make those decisions. the simplest solution for alma libraries is to rely on the alma community zone (acz) to handle both link resolution and discovery metadata whenever possible. however, as an early adopter of alma, umn chose not to take this approach, because the quality of ebook marc records in the acz was generally very low in alma’s early days. umn had also been a heavy user of oclc’s worldcat collection sets to acquire ebook metadata, so when oclc eliminated that service in favor of worldshare collection manager (wcm), umn implemented wcm marc record delivery for ebook metadata previously sourced via collection sets. of course, continuing to source marc records directly from vendors is typically also an option (though in some cases vendors now work directly with oclc to provide marc records via wcm and do not provide records themselves). with two or three possible record sources for each new ebook collection, we began routine manual evaluations (facilitated by marcedit) of marc record files from each potential source before selecting one. although this exercise was worthwhile, evaluations conducted via this method were time-consuming and necessarily incomplete, especially for large files containing thousands of records. scripting offered a way to systematically and consistently evaluate every record in a file, and to generate rough quantitative measures that could be used for quick comparisons across files for the same collection from different sources. evaluating record completeness for metadata source selection completeness is determined by evaluating files of marc records from available sources (acz, wcm, vendor) with a python script that checks for the presence or absence of several elements, and assigns weighted numeric values to each element. these values are added to arrive at a “total record score” for each record. the script is intended to be run as needed from the command line. it evaluates either every marc binary file in the current working directory, or a single marc binary file, based on console input. most elements were selected for automated evaluation based on umn local guidelines for pre-import editing of marc record files for ebooks (traill, [updated 2017]). the guidelines were originally created following a project in which record sets from several ebook vendors were closely evaluated, and have been continuously updated. the umn guidelines incorporate three major components: the results of the project described above, documented local discovery needs, and the library of congress’s program for cooperative cataloging (pcc) provider-neutral guidelines (program for cooperative cataloging, 2017). the script does not evaluate every element in umn’s guidelines. script-evaluated elements include most of those given in local guidelines that can affect end user discovery and/or identification of resources in umn’s discovery layer, primo. table 1. completeness evaluation rubric record element marc field/position/subfield how counted isbn 020 1 point for each occurrence of field authors 100, 110, 111 1 point for each occurrence of field(s) alternative titles 246 1 point for each occurrence of field edition 250 1 point for each occurrence of field contributors 700, 710, 711, 720 1 point for each occurrence of field(s) series 440, 490, 800, 810, 830 1 point for each occurrence of field(s) table of contents and abstract 505, 520 2 points if both fields exist; 1 point if either field exists date (marc 008) 008/7-10 1 point if valid coded date exists date (marc 26x) 260$c or 264$c 1 point if 4-digit date exists; 1 point if matches 008 date. lc/nlm classification 050, 060, 090 1 point if any field exists subject headings: library of congress 600, 610, 611, 630, 650, 651 second indicator 0 1 point for each field up to 10 total points subject headings: mesh 600, 610, 611, 630, 650, 651 second indicator 2 1 point for each field up to 10 total points subject headings: fast 600, 610, 611, 630, 650, 651 second indicator 7, $2 fast 1 point for each field up to 10 total points subject headings: other 600, 610, 611, 630, 650, 651, 653 if above criteria are not met 1 point for each field up to 5 total points description 008/23=o and 300$a “online resource” 2 points if both elements exist; 1 point if either exists language of resource 008/35-37 1 point if likely language code exists country of publication code 008/15-17 1 point if likely country code exists language of cataloging 040$b 1 point if either no language is specified, or if english is specified descriptive cataloging standard 040$e 1 point if value is “rda” element weighting is somewhat arbitrary. records containing a large number authority-controlled subject strings (lcsh, mesh, and fast), contributors, and isbns receive high scores under this rubric, which we view as appropriate in light of the high impact these elements have on discoverability. lower-weighted elements affect either retrieval in alma or facets/filtering in our discovery layer, primo, (e.g., 008 dates and language codes), or serve as general markers for record quality (e.g., the presence of “rda” in marc 040 subfield e, and language of cataloging). the code itself relies heavily on pymarc, “a python library for working with bibliographic data encoded in marc21”, and regular expressions to parse and match values extracted from the marc records (summers, [updated 2017]). once every record in a file has been evaluated and assigned a score, the script uses pandas, a python data analysis library, to create a dataframe (a tabular data structure) and calculate the mean score and standard deviation for the file. the script outputs a csv file for each marc file processed (pandas … , [updated 2017]). the csv is meant for human review, and includes mean, standard deviation, and the total score for each record, along with the score for each evaluated record element. figure 1. csv output for a script-evaluated file completeness evaluation results for selected ebook collections umn began using the completeness evaluation script in spring 2017. since then, we have used the script to select a record source for 11 newly-acquired collections. in most cases, three potential sources of marc records were available. for two collections, only acz and wcm were options, while for two other collections, records were available only from either wcm or the content vendor. all of the marc files from each source were evaluated with the completeness script. table 2. marc record availability per collection/package collection vendor acz wcm a yes yes yes b yes yes yes c yes yes yes d yes yes yes e yes yes yes f no yes yes g yes yes yes h yes yes yes i no yes yes j yes no yes k yes no yes we have added additional features to the script to incorporate comparison of many collections across record sources. we use matplotlib, “a python 2d plotting library which produces publication quality figures in a variety of hardcopy formats and interactive environments”, to read data from the pandas dataframes created in the analyses above and plot it in grouped bar charts (matplotlib … , [updated 2017]). these visual representations of our data give us a snapshot view of record quality across sources and collections. the graphs presented here were inspired by forum responses on stack overflow and a blog post by chris albon. (how do i plot…, [updated 2012]), (albon, 2016) figure 2. mean record scores by collection figure 2 shows record mean scores (with error bars representing standard deviation) grouped by collection. this graph is useful in visually comparing record sets on the collection level, in order to select a record source for a particular collection. figure 3. mean record scores by source figure 3 shows record mean scores (with error bars representing standard deviation) grouped by record source. this graph is useful for visually comparing record sources on the record provider level. this helps us to get a sense of how our various record sources compare against each other holistically, beyond a single collection. table 3. overall means of mean record scores by record set source all vendor sets all acz sets all wcm sets mean of all collections’ mean quality scores 19.1 19.8 36.3 mean of all collections’ standard deviation among scores 2.14 6.75 5.19 one insight is that it is important to consider not only the mean quality scores of a record set among different sources, but also the standard deviation within the sets. for example, the mean scores for vendor-provided record sets were comparable to that of alma community zone records (an average of 19.1 points for vendor records and 19.8 points for acz records.) however, the standard deviation of the record scores in the alma community zone sets was on average 6.75 points, compared to 2.14 for the vendor records. this data supports a gut assumption based on our previous hands-on examination of individual record sets that the quality of vendor supplied records tends to be more consistent within title sets than those from the alma community zone (for example, vendor records tend to either all include an element or all lack an element, whereas records in the community zone could be quite different from each other). we surmise this is due to a variety of factors, most notably that vendors tend to use record creation methods that lead to very uniform records, whereas records in the acz are from a variety of sources. some acz sources offer fairly rich descriptive metadata, while some records — mostly those generated automatically from knowledge base metadata — offer very minimal metadata. these results clearly show why we have most often opted to use wcm as a metadata source when multiple sources are available. the wcm records sets consistently had high mean quality scores (here the average was nearly twice that of comparable sets from vendors or the acz, at 36.3 points compared to 19.1 and 19.3), and the standard deviation within the sets was comparable to that in the acz (5.19 and 6.75, respectively), indicating that the wcm set is not more variable due to a few high-scoring outlier records. using wcm with alma in this way, however, means sacrificing some of the efficiency of alma’s built-in e-resource management functions. this is a significant trade-off for the anecdotally positive, but as-yet-unquantified, improvements to user discovery we learn about via feedback from users and public services staff. but with some effort, we have been able to automate large parts of the wcm record management workflow, which means the reduction in efficiency (versus relying on automatic management via the acz) is much less than it would be otherwise. although the initial implementation was challenging, wcm record delivery works well for the most part with alma’s facilities for automating and scheduling marc record import jobs. the next section of this paper will discuss how we use python scripts for intermediate processing between wcm’s automated marc file generation and alma’s automated import jobs. record quality evaluation for wcm delivered records once we have selected wcm as a record source for a collection, we follow our procedures for activating the collection(s) in wcm and creating automatic import profiles in alma to ingest these records. in wcm, settings are configured to automatically add and remove fields in the marc records in order to follow local practices. for record updates, wcm enables the configuration of the record output to include a field stating the reason for the record update. this field is essential to our workflow for splitting updates files before upload, which we discuss further below. daily, weekly, or monthly record delivery is scheduled on the collection level, and we schedule the automatic collection-level import profiles in alma to run on a complementary schedule. while the process for importing and evaluating new record batches and updates are very similar, there are some key differences. new records for new records, we configure the alma import profile to pick up the record batches with file names containing “new” directly from the oclc ftp server without any intervention on our part first. the rationale for this is that it is better to provide access as soon as possible and then clean up any issues that might impact optimal discovery when time permits. updates to records and ocn merges for updates to records (including changes to oclc control numbers(ocns) due to record merges), we use python scripts to first pick up the files from the oclc ftp server with file names containing “updates” or “merges”, assess and split out the files, and then only put files without serious issues back on an ftp server for the alma import profile to pick up. a staff person regularly processes any records that could not be automatically imported. the rationale for this order of operations is because the record is an update, we assume we are already providing access to the resource with the record originally delivered, and that any updates are “bonus”. because some of the updates have caused interruptions to service (multi-volume records with multiple urls are not handled well by the wcm-to-alma ecosystem, for example), in order to “first, do no harm”, we have decided to first process the records, even if that creates a small delay, and then import them into the system. evaluation and output: what, why, and how the goal of the scripts detailed in the first part of this paper were to use markers within marc record sets from different sources with the goal of selecting a record source based on aggregate collection quality. once a record source has been selected, our focus gets a bit more granular, from the forest to the trees themselves. the scripts we run to evaluate records delivered from wcm aim to identify records that might cause problems we know to be common with this record source. the specific issues we look for vary between the new records and record updates, and we’ll discuss those in the next two sections. like our record selection script, these scripts rely heavily on pymarc and pandas libraries, and script outputs are either csv reports or marc record files, as detailed below. new records in evaluating new records, our primary goal is to identify records that may not match our record quality benchmarks. the three questions our scripts answer for each record are: is the record truly for an electronic resource, or does the record describe a print resource? is the record duplicated in the set or does it seem to indicate that it might be a record for a multi-volume set? (in either case someone would want to check to make sure the urls are properly imported without overwriting themselves.) is the language of cataloging for the record english? the script also currently identifies records likely not cataloged in rda as well as records without any kind of subject classification (call number or subject heading), but we are currently not doing any remediation on these records. because we have already selected the collection, we are not focused on overall record quality; we are mainly concerned with critical issues that have caused problems with discovery or access in the past at the new record import stage. as we mentioned above, these records should all have already been imported into alma; these are post-ingest quality measures. corrections or changes are typically made at the global knowledge base level (worldcat and/or wcm), then imported into our local system (either waiting for the update to be delivered, or overlaying manually if the need is critical). the script picks up the marc record files containing the text “new” from the appropriate directory on the ftp server, and based on the three above categories, splits the files into three separate files of marc records. to gain insight into the selected record characteristics, the script uses pymarc functions to read a number of fields in the delivered marc records that it then evaluates against a list of expected or acceptable values. for each characteristic, if the value found in the record matches the expected value we want, a value of 1 is assigned to the attribute. if it does not match the expected value, a value of 0 is assigned. the sum of all scores related to a specific characteristic is calculated. this pseudo-weighted score is compared to a benchmark value so the script can “decide” whether or not the issues with the record are critical enough to be put on a remediation list. data is stored in a pandas dataframe and written to csv as reports. table 4. automated evaluation of wcm-delivered new records record characteristic fields evaluated and scored (assign 1 if value meets expected, otherwise assign 0; sum scores) weighted score needed to trigger reporting for remediation e-resource or print? 006 positions 0, 6, 9 = mod 007 positions 0-1 = cr 008 position 23 = o 300 $a description includes “1 online resource” 337 $a = “computer” 338 $a = “online” <4/6 duplicate or multi-volume? ocn matches another ocn in record batch   300 field contains “vol” or “v.” if repeats, indicate duplicate   if contains, indicate potential multi-volume language of cataloging 040 $b = “eng” =0/1 rda cataloging 337 present ; 338 field present =0/2 classification/subjects call number (one or more of 050, 060, 070, or 090 present) ; subject headings (one or more of 600, 610, 611, 630, 650, 653 present) =0/2 the output of this script is a list of records (oclc numbers) in the batch sorted into csv files based on what kind of human attention is needed. this script currently generates three files: a file of the actual values found in the records, a file that presents all scores and an overall sum score for each record (this file indicates which records are likely to be for print resources or have a language of cataloging that was not english), and a file indicating records that may be duplicates or multi-volume sets. the script output is a report format instead of record files because the records should all have already been automatically loaded into alma via ftp, and the records didn’t need to be sorted again. the reason for grouping the outputs in this way by these characteristics has to do with how the reports are then manually processed. the reason that records flagged as possibly describing print resources or as having a non-english language of cataloging are grouped together in one output is that they would potentially be remediated by making a global edit to the worldshare knowledge base. within most wcm collections, libraries can assign a different oclc control number (ocn) to represent a given resource in a collection. if the record provided does indeed describe a physical resource and there is a record in oclc which describes the electronic version, we will update the wcm knowledge base with the proper ocn for that resource. for records cataloged in a language other than english, often the record simply does not have a $b in the 040 field, so we will update the master record in oclc. if it was truly cataloged in a non-english language, and an english cataloging record exists, we will update the ocn in the wcm knowledge base as appropriate. if the impact on discovery seems critical, the record can be overlaid or imported from oclc immediately; otherwise we just wait for delivery of the updated record. neither of these errors is very common anymore, but when we first set up wcm, it was a more pervasive issue. the potential duplicates and multi-volume sets are grouped together because these are issues that could potentially lead to problems with electronic inventory in alma. any manual remediation related to these records is done directly in alma (for example, by updating urls in alma portfolios). updates for existing records in evaluating updates delivered for records we have already loaded into alma, our primary goal is to identify why a record update was sent so that we can merge the record into alma appropriately. as mentioned above, worldshare collection manager has a feature wherein you can export the “reason for update” data in your record output, making this otherwise impossible task feasible. our record delivery is configured so that this information is placed in a 960 field, so the script evaluates the value of only that specific marc field. the main types of reasons for updates to records that we see are: changes to bibliographic records which are descriptive in nature, not related to an identifier or access url change. ocn updates (this indicates the same record in oclc has a new oclc record number, or has been merged with another record. as of august 2017, these records are actually being delivered in their own files, which are distinguished with the text “merges” in the file name. we process these files with the same script; we just added regular expressions so the script will recognize these files as files to be evaluated along with the “updates” files. also note that these ocn updates are different than changes to the worldshare knowledge base which involve using a different record for the same resource. these changes are typically handled by wcm sending a delete file with the old record and then providing the new preferred record with the new record batch. we talk more about this in the “wcm deletes processing” section at the end of the paper.) knowledge base url changes the script picks up the marc record files containing the text “updates” and “merges” from the ftp server, and based on the three above categories, splits the files into three separate files of marc records. the first category, the descriptive changes category, is the easiest to deal with. those records are considered “ready for ftp”, and are placed back on an ftp server for a scheduled alma import profile to automatically retrieve and ingest. currently these files are manually moved to the server because the scripts are run on a desktop computer and we had security concerns; however, when the scripts are running on our server, we will automate this step. if a library did not have access to a server, they could also manually upload these files for ingest into their lms. the second category of record changes, ocn updates, are also placed in the “ready for ftp” record file along with descriptive changes, but they are also duplicated in a second file so that a staff member can verify that the identifier change was successful. we have yet to find an error in this process, so we are considering eliminating this step; however, we are establishing an active oclc synchronization process to keep our holdings up-to-date, so we try to practice the most effective “oclc number hygiene” that we can while still automating as much maintenance as possible. the third category, url changes, is the most tedious to deal with and requires manual handling. due to the way that alma’s import profiles handle inventory updates, the current software only supports overwriting any portfolios (the inventory unit via which urls are attached to bibliographic records in alma) already existing on the record. because the data from wcm do not indicate if the url change is an update to a url for the same resource, an addition of the next volume in a multi-volume set, or a false update (where the url isn’t a new addition and hasn’t changed from a previous version, but is being sent labeled as a “kb url update”), we can’t sort the file any more granularly than “something with the url(s) might be different”. any remediation of these records is done manually, directly in alma. multiple delivery frequencies, one script because different collections change at different rates (for example, a perpetual access collection might hardly ever have new additions or significant changes, whereas a medical reference resource might change every day), wcm makes it possible to configure your record delivery at different intervals to suit these needs. we currently have various collections set on daily, weekly, and monthly record delivery frequencies. each of these files is then output with a letter coded in the file name indicating if it is a daily, weekly, or monthly delivery. in order to process files on the most efficient basis, delivery frequencies are not handled separately. two total scripts run as part of this process: a script for daily, weekly, and monthly “new” files; and a script for daily, weekly, and monthly ocn “merge” and “update” files. when we were developing this process, we used six separate scripts (one each daily, weekly, and monthly per file type), but have since streamlined this so the scripts just pick up records of any delivery frequency when they are run daily. alma ingest profiles for new records require a more specific schedule frequency (daily for daily record deliveries, weekly for collections with weekly delivery). we run the profiles for monthly deliveries weekly, to ensure that files do not sit unprocessed on the server for a month if the import profile misses it by a day. this specificity is mostly because alma requires a separate import profile for each collection if the profile creates inventory, because the profile must be linked to a single electronic collection in which the inventory items (portfolios) are stored and maintained. for merges and updates, we have two import profiles, one to handle each type of file. these profiles make changes only to existing bibliographic records (matching on ocn) and never create or update inventory, so they do not need to link to any specific inventory collection. separate import profiles for the “updates” and “merges” are needed only because each import profile recognizes actionable files by a regular expression for file name patterns. further automation opportunities the scripts and processes described above have improved ebook metadata management at umn by making it more consistent, data-driven, and efficient, while also improving metadata quality for discovery. however, we see potential for both the expansion of existing automation, and at least partially automating several other tasks we are currently managing manually. currently the python scripts that process wcm update files run on a windows 10 desktop computer. the scripts are scheduled to run on a daily basis using task scheduler, which comes pre-installed on windows computers. this is a low-barrier-to-entry way to automate scripts for those who do not have access to their own server team. however, now that we have refined our process, we are working with our systems administrators to get these scripts running directly on an sftp server, for increased continuity and security. server-side processing will also make it possible to automatically place processed update files on an sftp server where alma’s scheduled import profiles can pick them up. wcm delete processing another future area of investigation is how to manage record delete files generated by wcm. while in many cases, delete records are legitimately for titles deleted from a collection, this is not always true: wcm also supplies a delete record when there is a switch in preferred record for a given resource. in these instances, wcm first sends a delete file for the existing record, and then sends a new or updates record for the new preferred record. while the gaps have grown smaller over time, in the past the lag between deletion and supply of a new record were too large, and caused interruptions in access that were noticeable to library users. because we are reluctant to perform any maintenance activity that we believe has a high chance of interrupting user access, we currently handle deletes manually for those collections to which we have perpetual access rights. we do this by periodically performing a manual audit for a given collection on the records we have in alma, the records available from wcm, and the list of titles to which the vendor says we should have access. this is an incredibly time consuming task, mainly because not every collection has a consistent identifier across marc record data, vendor lists, and knowledge base kbart metadata. this inconsistent data makes automation a tremendous challenge, one that we are continuing to work on as services evolve. conclusion we conclude this paper by stressing that as products under constant development and enhancement, both alma and wcm see frequent changes to both the marc records themselves and the ways those records are handled in their knowledge bases. all of the processes, analyses, and procedures described here are representative of this moment in time. we expect to continuously monitor available record sources and their characteristics, so that we may continue to adapt our workflows as the environment shifts. a good example of this is wcm’s recent (starting august 2017) addition of the “merges” file for ocn merges, as an alternative to including these records in the broader “updates” file. because our script and import profile was already handing ocn changes in the “updates” files, we adapted our existing script to simply pick up these files as well. this was as simple as adding regular expressions for the “merges” file name patterns to the script. before we started receiving these files, however, we had no idea what to expect or how this change would impact our previous workflow. these kinds of vendor-initiated changes are fairly common when dealing with electronic resource metadata provision, and we expect to keep encountering them. also worth noting is that the quality of discovery metadata for ebooks in the alma cz has demonstrably improved over time, and ex libris continues to work with both alma customers and content providers to improve it further. the best eventual outcome for alma libraries would be for metadata completeness and quality in acz records to match that available via wcm, making the processes described in this paper obsolete. that goal is unlikely to be reached anytime soon, but we will continue to work toward it. our goal in developing these scripts was to automate the processes for a) selecting ebook record providers and b) processing records for lms ingest and identifying records that needed a human to look at them. we have established automated workflows for both of these processes, which account for national and local bibliographic standards and practices, informed by the quirks we have observed in the datasets over time. while these workflows have reduced the amount of hands-on analysis needed to provide adequate discovery for these resources on a day-to-day basis, we anticipate that future timeand expertise-intensive work will always be needed to update the processes when vended services or our local needs change. references albon c. group bar plot in matplotlib. [updated 2016 may 1]. available from: https://chrisalbon.com/python/matplotlib_grouped_bar_plot.html chen m, kim m, montgomery d. 2016. ebook record management at the university of texas at dallas. technical services quarterly 33(3): 251-267. available from: http://dx.doi.org/10.1080/07317131.2016.1169781 fenichel, e. [accessed 2017 august 15]. marc_batchloading_checks (github repository). available from: https://github.com/ethandf/marc_batchloading_checks hodge v, manoff m, watson, g. 2013. providing access to e-books and e-book collections: struggles and solutions. the serials librarian 64(1-4): 200-205. http://dx.doi.org/10.1080/0361526x.2013.760411 how do i plot just the positive error bar with pyplot.bar? [updated 2012 november 12]. stackoverflow post. available from: https://stackoverflow.com/questions/13312820/how-do-i-plot-just-the-positive-error-bar-with-pyplot-bar lybarger k. 2013. keeping up with ebooks: automated normalization and access checking with normac. code4lib journal 20. available from: http://journal.code4lib.org/articles/8375 martin ke, mundle k. 2010. cataloging e-books and vendor records: a case study at the university of illinois at chicago. library resources & technical services 54(4): 227–237. availabe from: http://dx.doi.org/10.5860/lrts.54n4.227 matplotlib: python plotting [updated 2017 may 10]. documentation webpage. available from: https://matplotlib.org/ mugridge, rl, edmunds j. 2012. batchloading marc bibliographic records current practices and future challenges in large research libraries. library resources & technical services 56(3): 155–170. available from: http://dx.doi.org/10.5860/lrts.56n3.155 mugridge rl, edmunds j. 2009. using batchloading to improve access to electronic and microform collections. library resources & technical services 53(1): 53–61. available from: http://dx.doi.org/10.5860/lrts.53n1.53 pandas: python data analysis library [updated 2017 july]. available from: http://pandas.pydata.org/index.html program for cooperative cataloging (pcc). 2017. provider-neutral e-resource marc record guide: p-n/rda version. available from: https://www.loc.gov/aba/pcc/scs/documents/pn-rda-combined.docx sapon-white r. 2014. e-book cataloging workflows at oregon state university. library resources & technical services 58(2): 127-136. available from: http://dx.doi.org/10.5860/lrts.58n2.127 spurgin km. 2014. getting what we paid for: a script to verify full access to e-resources. code4lib journal 25. available from: http://journal.code4lib.org/articles/9684 summers e. [updated 2017 july 26]. pymarc (github repository). available from: https://github.com/edsu/pymarc traill s. [updated 2017 may 25]. editing and import guidelines for ebook record sets. available from: https://z.umn.edu/ebookguidelines traill s. 2013. quality issues in vendor-provided e-monograph records. library resources & technical services 57(4): 213-226. available from: http://dx.doi.org/10.5860/lrts.57n4.213 wu a, mitchell am. 2010. mass management of e-book catalog records: approaches, challenges, and solutions. library resources & technical services 54(3): 164–174. available from: http://dx.doi.org/10.5860/lrts.54n3.164 about the authors kelly thompson (kjthomps@umn.edu; http://orcid.org/0000-0002-3812-8910) and stacie traill (trail001@umn.edu; http://orcid.org/0000-0001-9945-2303) are metadata analysts at the university of minnesota libraries in minneapolis. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – outside the box: building a digital asset management ecosystem for preservation and access mission editorial committee process and structure code4lib issue 36, 2017-04-20 outside the box: building a digital asset management ecosystem for preservation and access the university of houston (uh) libraries made an institutional commitment in late 2015 to migrate the data for its digitized cultural heritage collections to open source systems for preservation and access: hydra-in-a-box, archivematica, and archivesspace. this article describes the work that the uh libraries implementation team has completed to date, including open source tools for streamlining digital curation workflows, minting and resolving identifiers, and managing skos vocabularies. these systems, workflows, and tools, collectively known as the bayou city digital asset management system (bcdams), represent a novel effort to solve common issues in the digital curation lifecycle and may serve as a model for other institutions seeking to implement flexible and comprehensive systems for digital preservation and access. by andrew weidner, sean watkins, bethany scott, drew krewer, anne washington, and matthew richardson introduction this article outlines the workflows and tools that the university of houston (uh) libraries have developed to facilitate digital curation activities for hydra-in-a-box (hyku), archivematica, and archivesspace, collectively known as the bayou city digital asset management system (bcdams). bcdams development work began in early 2016 based on the recommendations of a uh libraries task force charged with evaluating, testing, and implementing a new dams (wu et al. 2016) [1]. the recommendation outlined a three phase implementation project, and the bcdams team completed most of its phase one (systems installation) deliverables by the end of 2016, producing workflows and tools to support preservation and access ingest activities. as a whole, these workflows and tools comprise an ecosystem that supports the long term preservation of and access to the digitized cultural heritage materials in the uh libraries unique collections. implementation phase one the project team, consisting of members from digitization, metadata, special collections, and web services, employed an agile development methodology to lay the technical foundation for the uh libraries migration from our current systems to the bcdams [figure 1]. figure 1 bayou city dams implementation phase one. view larger image   over nine development cycles in 2016, each lasting approximately one month, the bcdams team implemented five applications (named for bayous in the houston metropolitan area) that address key digital curation tasks: preparation of files and metadata for preservation ingest (carpenters) preparation of files and metadata for access ingest (brays) persistent identification of digital resources (greens) local management of linked data vocabularies (cedar) representation of archival finding aids (halls) in addition, the project team began the process of descriptive metadata analysis and remediation to prepare for collection migration, resulting in a ruby gem for downloading metadata through the contentdm api (hunting), a framework for crosswalking metadata (buffalo), and a metadata application profile (bcdams-map). each development cycle ended with a formal report documenting the project’s progress that was made available to the uh libraries, and the overall progress was recorded in a public timeline [2]. bayou city dams ecosystem as a whole the bcdams applications represent an ecosystem of modular components, each described in detail in the sections below, that work together to address all aspects of the digital curation lifecycle [figure 2]. the preservation workflow begins in special collections where the finding aid for a collection is created in archivesspace. using the carpenters digitization workflow and preservation ingest application, special collections personnel import the archivesspace finding aid and select the list of items to be digitized by checking boxes next to the folder or item in the collection hierarchy. figure 2 bayou city dams ecosystem architecture. view larger image   as materials are scanned, digitization unit personnel associate files with digital objects in the collection’s intellectual hierarchy within the carpenters interface. after scanning is complete, carpenters exports a submission information package (sip) that reflects the collection hierarchy for preservation with archivematica. during the export, carpenters requests a new identifier for the sip from the greens persistent identifier minter and produces a dissemination information package (dip) that contains the access files and minimal descriptive metadata pulled from archivesspace. the access portion of the workflow begins when metadata unit personnel load the carpenters dip in the brays descriptive metadata editor and create descriptive metadata for all objects. brays suggests controlled vocabulary terms from the cedar linked data vocabulary manager and validates the record against the descriptive metadata specification defined in the bcdams-map. future development will focus on building the tooling necessary for minting digital object identifiers, posting digital object records to archivesspace, and packaging brays output for batch ingest into the hyku repository. carpenters: digitization workflow & preservation ingest carpenters is a cross-platform desktop application built using the electron framework. electron uses web technologies such as html5, css, and javascript to create rich applications without the need to develop for specific platforms such as windows, mac, and linux. because electron uses web technologies it can take advantage of other libraries and frameworks used in web development. as a result, the angular and bootstrap frameworks were used to create carpenters’ user interface and backend processes. together, these frameworks provided quick development time, flexibility between different user groups, and easy deployment. carpenters allows preservation administrators to organize digitized content into hierarchies that preserve the contextual linkages and provenance of the original archival collection. this process begins in special collections, where curators create a finding aid in archivesspace. the provenance collection is then selected in the carpenters selection interface, which queries the archivesspace api to automatically display the nested archival objects contained in the collection. special collections staff select the archival objects to be digitized using the check boxes at each hierarchy level in carpenters, creating a shot list in the files interface that informs the digitization unit which items are included in the project [figure 3]. as materials are scanned, digitization unit staff associate files for each digitized item stored on the local file system with the corresponding archival object in the files interface [figure 4]. through this process, the archivesspace uri for each archival object is linked to its digital surrogate. figure 3 carpenters selection interface with data loaded from an archivesspace finding aid. view larger image figure 4 carpenters file association interface. view larger image   utilizing the arrangement provided by the archivesspace-imported hierarchy structure, carpenters automatically moves the preservation masters on the file system to a set of nested directories in an archivematica-compatible sip [figure 5] that replicates the intellectual arrangement of the original collection. by eliminating the need for manually creating directories or moving files on the file system, the carpenters application streamlines batch ingest preparation for preservation master files, making it ideal for large-scale workflows. the tool integrates with greens to mint an archival resource key (ark) for each preservation package, creating a persistent identifier that connects the preservation master files to the access objects published in hyku. carpenters also outputs a dip of access files and a metadata csv file that is used as input for the brays descriptive metadata application. figure 5 archivematica sip exported from carpenters   brays: descriptive metadata workflow & access ingest brays provides a metadata creation interface [figure 6] and file viewer [figure 7] for metadata specialists working with digital objects in preparation for hyku ingest. it is based on the same tools and frameworks found in carpenters, and it employs a similar look and feel to give a sense of unity and connection between the two products within the bcdams workflow. before brays was developed, metadata unit staff used a combination of spreadsheets, microsoft access databases, and autohotkey scripts to produce tab delimited files for batch ingest. the bcdams implementation team held focus groups with metadata specialists to gather information about their workflow and solicit feature requests for a new descriptive metadata creation tool. the result is an application that promises to greatly improve efficiency in the descriptive metadata creation workflow through a one-stop interface that integrates with other bcdams applications. figure 6 brays metadata creation interface with validation and type-ahead suggestions. view larger image figure 7 brays file viewer. view larger image   brays dynamically reads and writes to a metadata csv file included in the dip generated by the carpenters application described above. color coding in the metadata creation interface indicates to metadata specialists which fields are required, recommended, and optional [3]. additionally, once the record contains all required fields, the object name in the object viewer turns from red to green. problematic records can be flagged and annotated to record contextual information during the metadata creation process. metadata specialists can also use the autofill function to populate values that are repeated in all or a select group of records. entries in the date field are validated against iso/wd 8601-2 [4], the proposed extended date/time format (edtf) extension of iso 8601. inspired by the edtf humanize gem, brays parses edtf dates and translates them into a human readable form in the metadata entry display. in order to ensure metadata consistency across systems, brays retrieves the metadata elements in the bcdams-map with a query to the access metadata specification api [5]. brays applies those requirements to enforce valid metadata entry. for some metadata fields, brays leverages ranges defined in the map to provide metadata specialists with a controlled list of authorized field values. other fields require values from the cedar vocabulary manager (see below), which are presented as suggestions when metadata specialists type in the form field. future plans for brays development include an export mechanism for packaging the access and metadata files into a format suitable for ingest into our hyku repository. greens: persistent identifier minter & resolver integral to the brays and carpenters workflow and ingest applications are tools that constitute the bcdams ecosystem infrastructure: the greens persistent identifier minter and the cedar linked data controlled vocabulary manager (described in the next section). the bcdams team recognized early on that the persistent identification of resources is essential to maintaining the long-term integrity of digital collections. as systems change and data is migrated, the bcdams must be able to provide seamless access to the resources it manages. the team selected the ark specification (kunze and rogers 2008) [6] as the persistent identifier scheme most applicable to digital resources for cultural heritage collections. after investigating the ezid service, the team chose to implement its own identifier minter and resolver to manage identifiers locally. greens is a ruby on rails application that mints opaque ark identifiers with the noid gem and includes the ability to add prefixes. the bcdams team plans to mint identifiers for three types of resources with the following prefixes: pm: preservation master sips do: digital objects au: authority records [figure 8] figure 8 ark identifier and greens erc record for a bcdams authority.   greens resolves arks, passing through any suffixes appended to the identifier, and the api supports minting, updating, and destroying arks managed by the application. two bcdams applications currently depend on greens to mint and update arks: carpenters (preservation package identifiers) and cedar (authority identifiers). cedar: linked data vocabulary manager cedar is a ruby on rails application driven by the iqvoc skos vocabulary management gem. the bcdams team has started to populate a local cedar instance with vocabulary terms already in use across uh libraries systems: the uh digital library, the uh institutional repository, and the finding aids in special collections. the uh libraries local skos vocabulary is based on the context classes of the dpla map. as mentioned above, brays makes use of cedar data [7] through the iqvoc api for authority control during the descriptive metadata creation process. records in cedar are served as html for end users and rdf/xml, n-triples, and turtle [figure 9] machine-readable formats. whenever possible, uh libraries authorities link to external vocabularies such as the library of congress subject headings and the getty thesaurus of geographic names. each record linked to an external authority is organized in a skos collection for that vocabulary, and every external authority link uses the skos closematch predicate. the skos exactmatch predicate is reserved for the authority ark identifier minted in greens. figure 9 turtle representation of a cedar authority record. view larger image   halls: finding aid representation in an effort to envision complementary access systems for digitized archival objects, the bcdams team and special collections recognized the benefits of developing a custom public interface for finding aids managed in archivesspace. early experiments using xslt to transform ead for display on the web [8] helped to gain buy-in for this approach. the decision to pursue local development of an archivesspace public interface was ultimately guided by a desire to ensure that we implement a finding aid access system driven by users’ research needs [9]. this path also presents an opportunity to maximize systems integration and provide a seamless user experience between the bcdams access repository and the finding aid interface. by displaying the context of the archival or provenance collection for content stored in the digital repository [figure 10], halls will play a key role in the access portions of the larger bcdams architecture. currently a proof-of-concept slated for further development, halls uses xslt transformations of ead-encoded finding aids pulled from the archivesspace api to present digital objects in the hyku repository within their archival context. halls will also display search results that clearly convey both the intellectual arrangement (series, sub-series, and file-level titles and descriptions) and the physical instance (box and folder information) associated with an archival object. figure 10 halls finding aid hierarchy with links to the digital object. view larger image   beyond the halls finding aid interface, archivesspace and its api allow for other integrations within the bcdams digital curation ecosystem. the archival arrangement managed in archivesspace informs the hierarchies that are used to structure a digital collection’s preservation sip in carpenters, and archival description provides a starting point for metadata creation in brays. with the goal of leveraging existing archival arrangement and description in the overall workflow for both preservation and access, we are approaching the migration of finding aid data from archon to archivesspace as a data quality enhancement and clean-up project. many legacy finding aids in archon predate standards such as describing archives: a content standard or encoded archival description. as each finding aid is imported into archivesspace, a careful review is necessary to ensure quality and consistency. while this was less of an issue in the previous, siloed ecosystem, standardized arrangement and description–including adherence to clearer input guidelines–is crucial for sharing and reusing archival metadata across the bcdams digital curation ecosystem. next steps phase two of bcdams implementation calls for the migration of digital objects from the uh digital library to the bcdams. as with our collection finding aids, migrating digital objects to a new system presents opportunities for improving metadata quality at many levels. toward that end, the bcdams team developed the hunting ruby gem, which provides convenience methods to access descriptive metadata through the contentdm api. combined with a custom framework for crosswalking metadata called buffalo, hunting allows metadata creators to quickly preview how the mapped data will appear in the new system and produce reports that facilitate data cleanup and controlled vocabulary analysis. the bcdams-map is a central component of the migration effort that is already underway. two ad hoc working groups, the descriptive metadata working group and the rights metadata working group, are using the github wiki and issue tracker to develop the bcdams-map metadata element set and crosswalk. the bcdams-map, a website built with the jekyll static site generator, doubles as a host for the tables used to preview crosswalked metadata [10]. just as the uh libraries metadata dictionary served to guide metadata creation for the uh digital library, the bcdams-map input guidelines will assist metadata specialists as they modify existing legacy data for migration to the bcdams. after the initial release of hyku, the bcdams team will engage in thorough ingest and migration testing of the system with the goal of deploying a fully functional production workflow and demonstration repository that can be used to solicit stakeholder feedback. development activities will continue with the creation of the armand (ingest) and jackson (export) utilities that will take advantage of the api hooks being developed in hyku. upon importing a dip to the repository, armand will mint identifiers for each object and post a corresponding digital object to archivesspace at the appropriate level of description. end users will then be able to view digital objects in their archival context through the halls finding aid interface. jackson will create a preservation sip containing the hyku access objects for archivematica ingest. the bcdams team will also begin investigating and experimenting with the avalon media system as an access solution for audio-visual materials. conclusion with this article, the bcdams team hopes to inform dams migration work underway at many institutions and provide a model for institutions considering migration projects. the tools that the bcdams team has developed address some of the many challenges associated with access, preservation, persistent identification, authority control, and archival representation in the cultural heritage digitization context. by sharing these tools openly and widely, we hope to create opportunities for conversation and collaboration around these common challenges. references [1] wu a, thompson s, vacek r, watkins s, weidner a. 2016. hitting the road towards a greater digital destination: evaluating and testing dams at the university of houston libraries. information technology and libraries. [cited 2017 feb 15]; 35(2). available from:  http://ejournals.bc.edu/ojs/index.php/ital/article/view/9152 [2] bayou city dams project timeline. [cited 2017 feb 15]. available from:  https://www.tomsplanner.com/public/bayou-city-dams [3] future development will include additional visualizations beyond color coding. [4] data elements and interchange formats — information interchange – representation of dates and times — part 2: extensions (iso/wd 8601-2). international organization for standardization. [cited 2017 mar 10]. available from: http://www.loc.gov/standards/datetime/iso-tc154-wg5_n0039_iso_wd_8601-2_2016-02-16.pdf [5] bcdams-map brays api. university of houston libraries. [cited 2017 feb 15]. available from: https://vocab.lib.uh.edu/bcdams-map/api/brays.json [6] kunze j, rogers r. 2008. the ark identifier scheme. california digital library. [cited 2017 feb 15]. available from: https://escholarship.org/uc/item/9p9863nc [7] university of houston libraries vocabularies. [cited 2017 feb 15]. available from: https://vocab.lib.uh.edu [8] ead-xslt. sean watkins. [cited 2017 mar 10]. available from:  https://github.com/seanlw/ead-xslt [9] pshock d. 2017. finding aids user interviews, december 2016. user experience at university of houston libraries. [cited 2017 feb 15]. available from: http://weblogs.lib.uh.edu/ux/finding-aids-interviews [10] bcdams-map migration. university of houston libraries. [cited 2017 feb 15]. available from: https://vocab.lib.uh.edu/bcdams-map/migration about the authors andrew weidner is the metadata services coordinator at the university of houston (uh) libraries where he oversees the metadata unit and is the project manager for the bayou city dams. prior to joining uh, andrew worked as the new mexico state project coordinator for the national digital newspaper program at the university of north texas (unt). he holds masters degrees in library science from unt and history from the university of texas of the permian basin. email: ajweidner@uh.edu sean watkins is the lead repository developer at university of houston libraries. sean earned his degree in computer science at southeast missouri state university in 2002. sean has developed and lead many web application projects, which include system integrations and data mapping to provide better experiences for users. when sean isn’t glued to a computer screen he spends his time playing board games, video games and taking on small home automation projects. he currently serves on the dams task force and is responsible for the system architecture and development of the new digital library. email: slwatkins@uh.edu bethany scott is the coordinator of digital projects at the university of houston libraries where she currently works on projects to implement systems such as archivesspace, archivematica, bitcurator, and omeka. as a representative of uh special collections, she contributes knowledge on digital preservation, born-digital archives, and archival description to the dams task force. she received an ms in information studies from the university of texas at austin and a ba in studio art from the university of st. thomas. email: bscott3@uh.edu drew krewer is the digitization services coordinator for the university of houston libraries, where he also serves as chair of the digital preservation task force. he is currently the chair of the dpn preservation metadata standards committee. he holds both an mfa in poetry and an ma in information resources and library science from the university of arizona and has worked in the past with the getty research institute, the center for creative photography, and the university of arizona poetry center. email: ajkrewer@uh.edu anne washington is the metadata librarian at the university of houston libraries where she manages the metadata unit, which is responsible for creating and improving digital collections metadata. her research interests include linked data initiatives with a focus on description and discovery strategies. she is currently the co-chair of the alcts/lita linked library data interest group. anne earned her mlis from the university of wisconsin-milwaukee and held previous positions at the university of virginia library and the university of wisconsin-milwaukee libraries. email: awashington@uh.edu matthew richardson is program manager in university of houston libraries special collections. there he is responsible for managing archival finding aids, coordinating both physical and born digital accessions, and assisting with reference in the reading room. current finding aid projects include the transition to archivesspace, updating holdings in texas archival resources online (taro), and coordinating description with the bcdams project. he holds an ma in liberal studies from northwestern university and is currently pursuing an ms-ls from the university of north texas. email: rmrichardson@uh.edu subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – finding and supporting new voices: code4lib journal’s issue 28 on diversity in library technology mission editorial committee process and structure code4lib issue 28, 2015-04-15 finding and supporting new voices: code4lib journal’s issue 28 on diversity in library technology welcome to code4lib journal’s special issue on diversity in library technology. as c4lj’s first-ever special issue, 28 brings together a plethora of voices from the library tech world in order to approach the challenge of inclusivity within our field from all directions. over a year of development has gone into this project, which has involved […] welcome to code4lib journal’s special issue on diversity in library technology. as c4lj’s first-ever special issue, 28 brings together a plethora of voices from the library tech world in order to approach the challenge of inclusivity within our field from all directions. over a year of development has gone into this project, which has involved the tireless and unpaid efforts of over thirty individuals, including authors and our outstanding guest editorial committee. development for issue 28 began with the editorial introduction for issue 24 – seeking a diversity of voices – in which ron peterson discussed the gender gap in c4lj’s authors and editors. this was truly eye opening for many members of the editorial committee, and we began discussing proactive ways to seek these missing voices. the idea for a special issue came out of this conversation, and it is truly exciting to see how the final product has evolved from our initial conception. at the beginning of this process, we reached out to people who have written or presented on diversity in library and technology in the past, and received some feedback that was pretty difficult to hear. the regular editorial committee could have written this off as a few people’s opinions, but instead chose to listen and try to amend the journal’s practices. one of the most important lessons i’ve learned from this issue is that, when it comes to developing a more inclusive journal, nothing is unimportant and nothing is sacred. if even one person thinks that the journal’s practices are unfair or exclusionary, then their opinion is absolutely valid and should be thoughtfully considered. conversely, no practice is absolutely necessary if it is inhibiting the inclusion of new ideas or voices. as a direct result of this feedback, we developed a guest editorial committee for the special issue in the hopes that necessary changes could be made if we had new eyes looking at the journal’s established practices. this process began with an open call sent out over our usual listservs, asking applicants to address their background in diversity as well as their potential contribution to the development of the special issue. after receiving a good response, the journal’s regular committee developed a target number of guest editors based on the desired number of articles for inclusion in the special issue (8-10, with one guest editor per article) and selected the final members based on their responses, writing and editorial background, and overall professional experience. while we aimed to represent in the guest editorial committee the diversity we ultimately seek to support, there are some obvious biases (e.g., a large majority are academic librarians and researchers) – a result of the regular committee’s lack of nuanced knowledge of online communities focused on diversity in library technology in putting out the call for applications, the applicant pool, and our publication schedule. the work of the guest editorial committee began in late 2014, so the process required great agility and adaptability on the part of every member. not only were they up to the challenge, but they worked diligently through winter breaks and over vacations to send out our second call for proposals, vote on proposals, and work with authors to develop articles. they have helped the regular editorial committee to lay bare our practices and discussions that go beyond the public code4lib journal-discuss google group and code4lib journal wiki – things that are normally invisible to those outside of our small group. in sharing our development cycle, policies, and ad hoc practices with the guest editorial committee, it has provided a new lens through which to view code4lib journal. i hope that we can continue this momentum and provide further transparency in the journal’s article selection and development processes on a continual basis. we still have great strides to make before the journal, and our field at large, is more supportive of diversity and inclusion, but we have begun the uncomfortable process of change and i hope that we continue to move forward in that. in her essay beyond the measure of men, roxanne gay pushes: “publish more women writers. if women aren’t submitting to your publication or press, ask yourself why, deal with the answers even if those answers make you uncomfortable, and then reach out to women writers. if women don’t respond to your solicitations, go find other women. keep doing that, issue after issue after issue.” we need to keep doing this, but of course not just with women. the biggest challenge in putting together this special issue has been locating and engaging the voices we haven’t heard from yet. while we sought out new listservs and online communities, the landscape is lamentably hard to traverse. those who are already well-known advocates for diversity are inundated with speaking and writing requests, while those who are not are harder to locate and engage. there are myriad reasons why that is, but the bottom line is that we all are responsible for changing that; luckily, there are some great new resources being built to support better development of the community surrounding diversity in library technology. for example, http://whodidimiss.github.io/ provides resources and templates for opening a dialogue about the need for diversity in conferences. attendees are able to provide constructive feedback while also creating a new bridge between their underrepresented community and the conference coordinators. i truly hope that some of you reading this will take the ‘who did i miss?’ template, modify it to address code4lib journal, and send it to journal@code4lib.org. we want to know what we can be doing better. as one of our boundary-pushing articles, code as code, asserts, we need to be questioning the very basis of our push to inclusion: “we suggest that by not probing lis’ definitions of “diversity”, “inequity”, and digital literacies at the most personal level of lived experience, we in the lis and technology professions will remain as well-intentioned, but in-effective in genuine inclusion. we question the editors’ statement “…we strengthen our libraries if we enjoy and engage with these differences.” will we? how do we know? what evidence of success do we have?” this is exactly what we were looking for in developing this special issue – articles that push the journal to be better, to evolve antiquated practices, and to question everything. one way in which we are experimenting with our limitations is by publishing code as code in a new way. because code4lib journal requires authors to write and edit their article within a four-month cycle, our publication schedule does not necessarily support inclusion of more theoretical or heavily researched works. in the case of code as code, we have worked with the authors to publish the first section of their work, with the aim of publishing more in subsequent issues. i am excited to see how this experiment turns out and i hope that we can continue to test the limits of our format in future issues. the question now is, what next? while code4lib journal will always remain, at heart, a practical journal for library techies, we hope that this special issue will be the beginning of a new chapter of the journal’s development, as we continue to solicit articles that push boundaries and feature underrepresented ideas and voices. i hope that you enjoy this special issue, comment on the articles, email us feedback, and consider submitting your own proposal for a future issue. code4lib journal is also actively seeking a few new editors to add to our regular editorial committee; consider applying to join us. whoever you are and wherever you are, you are part of our community and we want to hear from you. -heidi dowding, issue 28 coordinating editor subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – the drawings of the florentine painters: from print catalog to linked open data mission editorial committee process and structure code4lib issue 38, 2017-10-18 the drawings of the florentine painters: from print catalog to linked open data the drawings of the florentine painters project created the first online database of florentine renaissance drawings by applying linked open data (lod) techniques to a foundational text of the same name, first published by bernard berenson in 1903 (revised and expanded editions, 1938 and 1961). the goal was to make berenson’s catalog information—still an essential information resource today—available in a machine-readable format, allowing researchers to access the source content through open data services. this paper provides a technical overview of the methods and processes applied in the conversion of berenson’s catalog to lod using the cidoc-crm ontology; it also discusses the different phases of the project, focusing on the challenges and issues of data transformation and publishing. the project was funded by the samuel h. kress foundation and organized by villa i tatti, the harvard university center for italian renaissance studies. catalog: http://florentinedrawings.itatti.harvard.edu data endpoint: http://data.itatti.harvard.edu by lukas klic, matt miller, jonathan k. nelson, cristina pattuelli, and alexandra provo introduction & data source first published in 1903, the drawings of the florentine painters represents berenson’s most important contribution to art history. though the first edition was published over a century ago, it is still regularly cited and used as an authority, together with two revised and expanded editions of the catalog, in 1938 and, in italian, in 1961. the entries from the 1961 edition describe a total of 4,051 object resources, i.e. drawings by painters active in florence during the renaissance. organized in alphabetical order by artist, the records for each drawing document key elements including city, collection, title, material, dimensions and, for some drawings, bibliography and critical commentary. between editions, some of these elements were modified (e.g., artist attribution or location) or additional information was added; the later editions included more drawings and expanded on berenson’s commentary. although standardized cataloging practices for works of art were not in place at the time bereson was compiling the works, the records in the drawings of the florentine painters are generally consistent in their structure and metadata within each edition. given concerns about space on the printed page, numerous abbreviations are used in the 1938 and 1961 editions. figure 1 shows various examples, some barely comprehensible today, such as “sp.” for the drawing technique “silverpoint”. figure 1: records of drawings by filippino lippi from berenson’s catalog, 1938 edition. catalog records, the information source for our project, were of course first made available on the printed page. the original readers could detect and (usually) understand the abbreviations, as well as various discrepancies, inconsistencies, or omissions. for example, one drawing in the 1961 edition lacks a catalog number; it became “0000000-berenson” in our dataset (figure 2). figure 2: unnumbered record from berenson’s catalog, 1961 edition deviation from the common structure of a drawing entry is also found when multiple drawings are grouped together under a single entry. generally, berenson assigned a drawing entry only to individual drawings, but drawings from a sketchbook or mounted together on the same support received different treatment (figure 3). figure 3: two drawings on a single mount. additionally, berenson was inconsistent about assigning catalog numbers when drawings appeared on both sides of the same sheet: the recto and verso might (or might not) receive their own entries. while these exceptions rarely reduced the overall quality of the data or the readability of the original format, they did pose significant problems for the migration and conversion of the data into a machine-readable or structured format. data preparation as a first step, berenson’s records were transcribed by catalogers to tabular data, including the following elements: artist name berenson-assigned catalogue number location (city, country, and collection) collection inventory number title material (only the mark-making material) commentary not included were metadata about dimensions, material support, and bibliography, as these were outside the scope for the project. data was transcribed in the summer of 2015 in two successive phases. first, transcribers created a separate excel spreadsheet for each artist cataloged. about 61% of the total number of records were transcribed in this phase. in the second phase, transcribers were given one google sheet each in which to work. new columns were added, and transcribers were asked to use a slash [/] between multiple data points in the same cell to facilitate data processing. data processing a considerable amount of cleaning and normalization was needed to get the source data ready for lod conversion. indeed, data cleaning constituted one of the most labor-intensive components of the entire project. not just a mechanical activity intended to address messiness, the notion of data cleaning has deep connotations and significant implications for the scalability of a project. as rawson and muñoz (2016) argue, what is perceived as a mess, should be instead considered as “diversity [that] has seeped or broken into what were designed to be smoothly scaling systems.” data cleaning requires careful consideration of the delicate balance between the need for data normalization that enables project interoperability and the benefit of retaining elements that carry complex meaning and bring diversity of perspectives. this was a tradeoff we faced in a few instances when transposing berenson’s unique catalog data into a standardized form. it provided an occasion for extended discussion between various team members: art historians, data specialists, librarians, technologists, developers, and catalogers. whenever possible, we preserved the uniqueness and richness of berenson’s data. for example, we retained berenson’s spellings of artists’ names rather than replace them with labels from external vocabularies (while allowing one to search for either spelling). however, there were instances in which the normalization of terms through controlled vocabularies was also an opportunity for data enrichment. this was the case with descriptions of techniques; term normalization, through the use of external vocabularies (getty’s aat) helped reconcile the many variants berenson had employed in his entries. the semantic richness of his description of techniques and materials, however, was maintained for those terms that would not find a suitable correspondent in the controlled vocabulary. in a few instances, however, berenson’s terms were contributed to aat, as discussed below. because the data was transcribed in various stages following different guidelines, a new iteration was needed to combine and consolidate the spreadsheets. more specifically, the individual artist spreadsheets from phase one were combined into one spreadsheet, and the two spreadsheets created in phase two were combined into one. in each of these master spreadsheets, columns were split and merged and problem characters were removed from cells. the spreadsheets were saved as separate documents with successive version numbers to ensure that the data could be recovered easily if a transformation failed, needed to be modified, or produced unexpected results. ablebits power tools for google sheets was used to manipulate the data. the following list details the transformation tasks performed in both master spreadsheets: split recto and verso data in the technique, title, and berenson comment columns into separate columns removed “recto” and “verso” indicators from the cells split artist attribution qualifiers (such as “school of”) in the artist columns into a separate column removed square brackets split techniques into separate columns combined berenson numbers and letters columns (for example, berenson catalogue numbers like 180 a had been split into two columns) split sub-collection into a separate column results were manually spot-checked and corrected. the two master spreadsheets were combined into one, and each individual drawing was given a unique identifier.  the identifier was created based on the berenson catalogue number from the 1961 edition. the suffix (berenson) and padding zeros were added automatically via a python script (for example, 000567a-berenson). an index match function =index(sheet3!b:b,match(b3614,sheet3!a:a,0)) was then used in google sheets to place the identifier in the correct row. the resulting column of identifiers was copied and the values were pasted in order to replace the function with the resulting values. not all the elements processed in the original spreadsheets were included in the master combined spreadsheet. in the case of the page numbers and plate numbers from the 1903 edition, the vlookup function was used to align them with urls for jpeg scans of the 1903 edition, since multiple entries were recorded on each page. data enrichment with the identifiers in place, the data was enriched by reconciling various fields against external vocabularies. these vocabularies provide persistent identifiers and address issues of disambiguation, homographs, spelling variants, etc. links were created to lod controlled vocabularies including geonames, ulan and aat to assign controlled values to locations (cities and countries), names of artists and artistic techniques. table 1 outlines the elements targeted for enrichment and the methods used: table 1 data element vocabulary used enrichment method city geonames python script to call geonames api and compare with source data. the english spelling was used. country geonames python script to call geonames api and compare with source data. the english spelling was used. collection viaf openrefine facet feature to normalize spelling and combine terms. openrefine reconcile feature with viaf reconciliation service. the spelling used in the main heading from the viaf record was used. artist ulan python script to call getty union list of artist names api and compare with source data. only the uri was taken from ulan. technique aat openrefine facet feature to normalize spelling and combine terms, but not to fetch uris. a subject expert was consulted about terminology; one result was the use of the term metalpoint instead of silverpoint. list of getty art & architecture thesaurus uris hand-compiled and inserted into the master combined spreadsheet using an index match function. reconciliation work is a complex process that often requires fine tuning of a query to obtain an optimal balance of precision and recall. although these specific methods were employed for the enrichment process, one could enrich source data with external vocabularies in many different ways. for example, one could construct a sparql query to run against ulan in openrefine. the wikidata plugin for openrefine could also be used to retrieve the wikidata uri, which can subsequently be used to retrieve the ulan or geonames uri. reconciliation was completed on a separate spreadsheet, and results were then incorporated into the combined master spreadsheet by matching on the unique drawing identifier using an index match function. unmatched names (e.g., due to misspelling or ambiguity) had to be matched manually. where uris were not available from the source vocabularies, project-specific uris were created (e.g., for collection and artist names). in a few instances, terms were contributed to a source vocabulary (the getty aat), including the terms pastel (material) and metalpoint (technique). significantly, name reconciliation occasionally conflicted with the need to convey the historical attribution as recorded in berenson’s catalog. for example, berenson treated raffaellino del garbo and raffaellino di carli as two separate artists in all three editions of his catalog, but historians have since determined that the two painters should be considered the same person. thus, ulan contains one entry for “raffaellino del garbo” with “raffaellino dei carli” listed as an alternate name. to preserve the two identities’ distinctness and convey the philological value of the metadata, we opted for assigning two different uris in our dataset.  it was not possible to use the term id uri that ulan provides for the term label (i.e., the non-preferred name raffaellino dei carli) as it would conflict with the range of the property used to represent the relationship between the production of a resource object and the creator (cidoc crm property carried out by with range entity crm:person versus getty term id corresponding to entity xl:label). interlinking interlinking is the mechanism of connecting resources to external datasets. a key tenet of lod principles, and a condition to fully meet the requirements of the 5-star rating systems introduced by tim berners-lee (2006), is integrating a dataset with links to external resources; this enables the discovery of related data. in addition to the enrichment described above, we associated our project records with those owned by the british museum. this was achieved by querying the british museum sparql endpoint through openrefine, matching the museum inventory number from both datasets. we began with a sparql query on one specific inventory number: select ?objectid where { ?id rdfs:label "1895,0915.454". ?objectid ecrm:p1_is_identified_by ?id } after running this query, we found that many of the inventory numbers (created in the late 19th and early 20th centuries) had multiple objects assigned to them. we were able to further refine our query by adding a qualifier of object type:drawing select ?objectid ?objecttype { ?id rdfs:label "1952,0405.7". ?objectid ecrm:p1_is_identified_by ?id. ?objectid bmo:px_object_type ?objecttype. ?objecttype skos:preflabel "drawing" } that query can be represented with a url that can be retrieved from the sparql endpoint. adding an option to output a json file, as well as replacing the inventory number with the openrefine “value” variable (escaped), will pass the value (inventory number) from our data to the sparql query. we ended up with the following query to input into openrefine: 'http://collection.britishmuseum.org/sparql.json?query=prefix+rdfs%3a+%3chttp%3a%2f%2fwww.w3.org%2f2000%2f01%2frdf-schema%23%3e%0d%0aprefix+ecrm%3a+%3chttp%3a%2f%2ferlangen-crm.org%2fcurrent%2f%3e%0d%0aprefix+bmo%3a+%3chttp%3a%2f%2fcollection.britishmuseum.org%2fid%2fontology%2f%3e%0d%0aprefix+skos%3a+%3chttp%3a%2f%2fwww.w3.org%2f2004%2f02%2fskos%2fcore%23%3e%0d%0aselect++%3fobjectid%0d%0a%7b+%3fid+rdfs%3alabel+%22'  + escape(value, 'url') + '%22.%0d%0a+%3fobjectid+ecrm%3ap1_is_identified_by+%3fid.%0d%0a+%3fobjectid+bmo%3apx_object_type+%3fobjecttype.%0d%0a+%3fobjecttype+skos%3apreflabel+%22drawing%22%0d%0a%7d&_implicit=false&implicit=true&_equivalent=false&_form=%2fsparql' we then used the “edit column” ? “add column based on object uri” command in openrefine to retrieve the results in json format, and parsed the value using “edit column” ? “add column based on this column” with the following grel expression: value.parsejson().results.bindings[2].objectid.value the result is a uri from the british museum that is the exact same object as the one from our catalog. the interlinking in our database was then obtained through owl:sameas statements, the most frequent practice to support linked data integration across identified resources and multiple datasets. obtaining the object uri provided another significant advantage: we were able to programmatically retrieve all of the images. after the programmatic interlinking of our catalog to that of the british museum, a cataloger manually enriched the remaining records in our catalog with both urls of the drawing, and images from other museum collection websites. these resources were linked using the predicate rdfs:seealso. had these other collections been made available in lod with sparql endpoints, all of this work could have been performed programmatically—a testament to the value of making data queryable and open. data transformation the process of data transformation is a long process that involves many issues related to ontology selection and modeling. for the purpose of this article, we limit our description to the technical details of the data transformation phase, which was performed using 3m (mapping memory manager). 3m is a web-based application developed by the institute of computer science at forth (foundation for research and technology – hellas) that allows the user to map a source xml file to rdf using ontologies (in our case, primarily the cidoc-crm). researchspace, a mellon-funded project based at the british museum, uses 3m to produce the british museum’s linked open data and conducts workshops on how to use the software. members of our team attended workshops in order to understand how to use the software and learn more about the cidoc-crm. to begin the transformation, the csv file created during the preparation phase was converted to xml format, with column headings becoming xml elements. the resulting xml file was loaded into four separate 3m “projects” — one for each of berenson’s editions, and one for what we deemed “project data.” other projects were created for elements that were transformed at a later time, such as the number of the plate, i.e., a page with only one or two images, the image, and the page number data. in each of these projects, xml elements were mapped to corresponding cidoc crm entities and properties. although there are a few pieces of software that have similar functionality to 3m, such as karma, 3m was built to specifically support data transformation using the cidoc-crm ontology. complex mappings can be defined in 3m, enabling the user to include intermediate nodes and add types to entities. this feature enabled us to specify additional elements, such as a production event, which was not previously defined as a part of our dataset (figure 4) figure 4: 3m mapping showing a production event. 3m generates uris if they are not present in the source xml file. while uris existed in our dataset for elements which had been enriched (e.g., location and artist names), they had to be generated for components (e.g., recto and verso, i.e., the front and back of the sheet), data elements (e.g., title, inventory number, berenson’s catalogue number), as well as for intermediate nodes like production and identifier assignment nodes. an xml file defining uri generation patterns was created so that the 3m application could create these uris according to consistent and semantically meaningful patterns. within the mappings, each entity had to be assigned a uri generation pattern from the pattern xml file. during this process, rdfs labels could also be generated using patterns from the xml file (figure 5). figure 5: uri generator mapping for the berenson catalogues number in 3m. with the mapping and uri generation instructions in place, the data could be run through the 3m’s transformation tab. because our source xml files were large (9.2 mb of text), we ran the transformation engine (called x3ml) locally in an instance separate from the mapping interface. the resulting files were serialized as n-triples (figure 6). figure 6: n-triples generated by the mapping for the 1938 edition. since our project used named graphs to indicate data provenance, those triples were then converted to quads using a simple python script, adding the graph uri to each triple. using a shell command, the various quads files were then combined into a single file. data publication the master quads file obtained from the data transformation phase was then uploaded to blazegraph, the graph database we chose to host our data. as a front-end to our data endpoint, we chose an open-source platform called metaphactory from the company metaphacts. this platform is tightly integrated with the blazegraph database, providing a public sparql endpoint, dereferencing for our uris, and access control functionality, among other features. most importantly, it enabled us to construct a user interface to browse our raw data using a templating engine built with html5 and sparql queries. metaphactory is used as middleware by the british museum to build researchspace, an end-to-end lod system for hosting, publishing, and cataloging in linked data using the cidoc-crm ontology. at the time of data publishing, the researchspace system had not been publicly released. had we used metaphacts to build a platform to browse our catalog, it would have been costly and it would have duplicated the work of researchspace. instead, we relied on the metaphactory platform (figure 7) to build a basic interface that enables data scientists to explore the data. figure 7: user interface for navigating the catalog data http://data.itatti.harvard.edu using project blacklight (figure 8), a popular ruby on rails platform often used for library catalogs and digital collections, we built a separate platform that would provide an intuitive and clean interface for researchers and the general public. project blacklight uses a solr index that does not natively support rdf, so the primary challenge in customizing the platform involved mapping and ingesting our rdf data to the solr index. this was done with a rake task but meant that in order to have updated data, one would need to rerun the task. the rake task queries the sparql endpoint and builds and indexes a comprehensive solr document for each record. this is a somewhat elaborate process as it needs to combine data from as many as four different named graphs. for example, a single record could have multiple titles from different editions or even more than one artist attribution that changed between editions. this process also queries remote data sources such as viaf and getty — pulling in external data based on uris in the record. this enrichment has the potential to add additional data that could aid in search and discovery, indexing alternative name labels from viaf, for example. the resulting solr document has approximately 70 fields and highlights the challenge going from a graph based data model to a flat solr index. this code is available on the project github page: https://github.com/villaitatti/florentine-drawings figure 8: user interface for the drawings of the florentine papers http://florentinedrawing.itatti.harvard.edu user interface and design project blacklight is natively designed to handle the kind of data present in our catalog, providing an intuitive and clean user interface out of the box. the main decisions we had to make addressed layout and theming. faceted and keyword search provide easy access points to the data, while the home page provides additional points of entry for browsing the data. each record also points users to a semantic view on the data endpoint, allowing them to browse the data either through a table of statements or a graph visualization. these features are essential for users such as data scientists who need to explore the data structure before accessing it programmatically. to help us make decisions about the layout and theming of the user interface, we conducted a few informal user tests to study the effectiveness of element labels and generate suggestions for modifications to the interface. users were presented with two tasks (one focused, and one exploratory) and asked to think aloud. these sessions provided insights into how best to rearrange the elements on the screen and raised content issues that we were then able to correct in the dataset. lessons learned and future work the project began in january of 2015 and concluded in may of 2017, although additional improvements are still underway. during this period, a series of technological developments forced us to change course. specifically, the initial beta release of metaphacts in january of 2017 led us to rethink the choice of the data endpoint and uri dereferencing. the possibility of building the public user interface using metaphacts directly might have been desirable in order to maintain all elements within one system. nevetheless, this would have hindered the possibility of incorporating developments from the researchspace initiative, and to possibly add more collections in the future.  building a minimal, but functional interface in the metaphactory software enabled us to provide access geared toward data scientists while remaining open to future developments from the researchspace platform. the data transformation phase required substantial knowledge about both the cidoc-crm data model and the transformation software 3m. this process required formal training, which was not always available locally. two team members were able to attend researchspace workshops on 3m and the cidoc-crm, but the steep learning curve of the software and the complexity of the cidoc-crm, meant that collaborating on this part of the project was challenging. managing data once the project was finished also proved to be more difficult than anticipated. it was fairly straightforward to load an rdf file to a graph database, but developing a strategy for the long-term maintenance of that data was more complex. after publishing the project, we discovered a host of issues with our data that needed to be updated. the update process proved difficult, as we had to decide either to update the rdf files and reload them every time, or perform a sparql update query to make changes. this brought forward new unforeseen issues regarding the “master” copy of the data. later, these issues were partially resolved with the metaphacts platform, which has a “form” feature that allows administrators to update data in a graphical user interface through a form input. this did not, however, address the broader issue of how to deal with data that needs to be changed in many locations across the dataset. this requires a significant time investment building sparql queries. issues surrounding data management remain as tools for managing rdf data are still early in their inception; ideally these would be included in the metaphactory platform. moving forward, villa i tatti will seek to build out the metaphacts platform, integrating the researchspace developments and expanding their digital collections to include digital editions, photographs from the historical photo archive, and documents from the archive, all within a single end-to-end lod platform. any development will be contributed back to metaphacts so that those changes can be incorporated into the open-source version. the blacklight instance will be dockerized and maintained in its current state as long as the current web standards and infrastructure will allow it to exist. most importantly, as more institutions adopt lod as a standard for publishing their data, we hope and expect to see more tools being developed and shared. this will lower the barriers to publishing datasets in lod and allow greater discoverability and interoperability of content across the web of data. the drawings of the florentine painters dataset is made available under a cc0 license references berenson b. 1903. the drawings of the florentine painters: classified, criticized and studied as documents in the history and appreciation of tuscan art, with a copious catalogue raisonné. london: john murray berenson b. 1938. the drawings of the florentine painters. chicago, ill.: the university of chicago press berenson b. 1961. i disegni dei pittori fiorentini. milano: electa editrice. linked data – design issues. [accessed 2017 aug 17]. available from:https://www.w3.org/designissues/linkeddata.html. rawson k, muñoz t. 2016 jul 6. against cleaning. curating menus. available from: http://curatingmenus.org/articles/against-cleaning/ about the authors lukas klic is the manager of information services and digital initiatives at villa i tatti – the harvard university center for italian renaissance studies, where he co-directed the drawings database project matt miller is a librarian and technologist focusing on the intersection linked open data and cultural heritage. he is a visiting assistant professor in the school of information at the pratt institute. jonathan k. nelson is faculty associate at syracuse university in florence. formerly assistant director of academic programs and publications at villa i tatti – the harvard university center for italian renaissance studies, where he co-directed the drawings database project. cristina pattuelli is an associate professor at the school of information at the pratt institute, new york and founder and director of the linked jazz project. her current area of research is semantic web technologies applied to cultural heritage resources. alexandra provo is the digital production editor for the enhanced networked monographs project at nyu. she was the 2015-2016 kress fellow in art librarianship at yale university and has been the project manager for two linked open data projects: florentine renaissance drawings: a linked catalogue for the semantic web and the linked jazz project.   subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – participatory design of websites with web design workshops mission editorial committee process and structure code4lib issue 2, 2008-03-24 participatory design of websites with web design workshops at the university of rochester's river campus libraries we have included users in technology development with great success. "participatory design" entails collaboration among designers, developers, and users from the earliest stages of conception through to implementation of websites and other technology. using participatory methods, a project to redesign the library website began with workshops to identify user needs and preferences. the results of these workshops led to the identification of key tasks for the main page. they also generated a hierarchy of tasks for sub-pages and rich information about how students and faculty members use current websites in their work. in our article, we explain our reasons for running participatory design workshops, describe our methods, review participants and recruitment, and summarize key findings. we also include information about our local implementation and general conclusions about the value of design workshops for website design and development. by nancy fried foster, nora dimmock, and alison bersani introduction: background and reason for the project early in 2007, the university of rochester’s river campus libraries decided to revamp the libraries’ web presence. the library website serves many purposes, the most important being to provide an entry point to the library catalog, subscription databases, and digital repository. the site contains extensive information about other library resources and services, such as special collections, interlibrary loan, electronic reserves, news and events, and library locations and hours. it also aggregates and highlights course content and related resources, providing for the use of databases and other finding aids, contact with librarians, and access to other important library information. the redesign was motivated both by the problems uncovered in ongoing usability testing of the site and by the desire of librarians to incorporate new technologies and features into the libraries’ web presence. although many of the site’s catalog-related usability issues, such as context-sensitive forms or frbr or other faceted results, remain outside the scope of a web redesign (and are currently being addressed by the extensible catalog project), improvements to the site’s architecture and ease of use made the complete redesign a priority. river campus libraries has developed a process for new technology design and development, represented in figure 1. the process begins when the libraries’ dean constitutes a project team to head up the overall development process, and gives that group a mandate, defining the aims and scope of the project. the project team (sometimes also known as a “content group”) has 2 major roles. one is to define the requirements of the new technology and the other is to manage the entire process from research to implementation and continuous improvement. figure 1. technology design process used by river campus libraries, university of rochester “participatory design” is an approach to technology design and development that we have followed for more than 5 years. it entails collaboration among designers, developers and end users, as facilitated by social scientists. the approach derives from work done in scandinavia in the 1970s to include worker input into the way computer technology was introduced into their workplaces (schuler and namioka, 1993). it has been further developed in the u.s. mainly in corporate settings. the hallmark of this approach is providing workers and other end users with ways to provide their expertise to the project, often by documenting and analyzing the way they do their work – their “work practices” – and how they use existing technologies, how they work around technological problems, and how they would like to improve their work. as the image shows, we conduct work-practice studies continuously, often in conjunction with grant-funded projects, but also as part of the libraries’ regular work. we draw from this pool of information whenever we begin a new project to build technology. this process is a departure from the previous process by which the design team created initial designs based on requirements from the project team but without user input. in the case of our current website redesign project, the project team defined key tasks for the main page and prioritized tasks for lower-level pages by drawing extensively on the information gathered in the participatory design workshops. they handed this information off to the design team, which created designs. the usability testing team is currently in the process of testing these designs and working with the project team and the design team to improve them. once the designs are acceptable, they will be handed off with a full set of specifications to the implementation team (programmers) for coding. with stops along the way for additional usability testing on prototypes, the process will run through to completion of the new code, testing of the new site, and implementation. in this article, we recount how the project team drew on existing work-practice studies and commissioned additional research, which was gathered through participatory design workshops, in order to create an informational basis for the identification of key tasks and other requirements for the new site. we explain our methods, describe the participants, and then review our key findings. we also show how our findings have led to important decisions for our local implementation. we conclude with general remarks about the costs and value of this kind of user research in technology development. while this article is not meant as a manual, it will give the reader some idea of the methods and benefits of this approach and whether it may be worthwhile to seek some initial support in trying it out. methods the website redesign project provided the perfect opportunity for the project team to bring the user into the design phase. river campus libraries had recently completed a major project on undergraduates and how they write their research papers (foster and gibbons, 2007). we were able to draw on that research, plus additional research with faculty members that we had done on an imls-funded project related to digital repositories (foster and gibbons, 2005). while 2 participatory design workshops with undergraduates provided the project team with a good pool of research from which to draw, they needed more information about the habits and preferences of faculty members and graduate students. the leaders of the redesign process formed a web redesign user research team – we’ll refer to this group as the “team” in this article – to do just this. the team included katie clark, the director of our science library; nora dimmock, the head of the multimedia center; marc bollmann, library assistant in the art and music library; alison bersani, engineering librarian; and nancy foster, the libraries’ anthropologist. we want to point out that much of this work is possible precisely because our library has an anthropologist on staff to assist in the planning and execution of our research activities and to help make the step from raw to analyzed data and then to concepts and implementation. the team reviewed the results of several participatory design workshops that had taken place in the library, including the 2 workshops in the undergraduate research project (smith and clark, 2007) and another design workshop related to the renovation of a large library space (gibbons and foster, 2007). in one of the undergraduate workshops, students were given a blank piece of paper and asked to design their ideal homepage. in the second undergraduate workshop, students were given a blow-up of the current homepage and asked to mark it up, crossing out elements that they do not use and circling things they value. in the library renovation workshop, students were asked to design the perfect library. the team used all of the drawings – and the insights we gleaned from them – to plan a participatory design workshop specifically for graduate students and faculty members. the workshop had 4 parts: first we asked the participants to fill out a questionnaire (see appendix i). using a laptop, we asked participants to show us the page they go to for library work and we asked them what they did on this page. then, we asked participants to imagine that one library page had disappeared. which page, we asked, would they find most disruptive to lose. we also asked what they did on that particular page. we bookmarked these pages so we could analyze the data later. for the design portion of the workshop, we first conducted an “x and o” activity. we gave participants a blowup of the homepage of the current library website to modify. we asked them to use pens and sticky notes to… circle the items on the page that they like and use the most add any items that are not currently on the page but that they would like to have there “x” out any items on the page that they do not like or use figures 2 and 3 illustrate this step of the workshop. figures 2 and 3. examples of “x and o” workshop activity finally, we conducted a “plain paper” design. we gave participants a blank sheet of paper and asked them to design their ideal library website. to do this, we asked them to imagine the following scenario: “let’s say the library were going to redesign its website and asked you to come up with a website that would be perfect for you. use your marked-up page plus these art materials to create that page.” figures 4 and 5 illustrate this step of the workshop. figures 4 and 5. examples of “plain paper” workshop activity the workshop protocol marked a significant departure from the typical usability testing approach used in the design of our current website in 2002, especially in terms of the role of the user. participatory design and usability testing are complementary methods that come into play at different points in the design, development and implementation cycle. usability testing refers mainly to having some end users try a product out in the course of attempting representative tasks. this is usually done with a prototype or working version of the technology and it provides quick and inexpensive feedback for course correction, improvement and refinement of a product. participatory design, by contrast, begins before a product is even identified and continues throughout conceptualization, design and development. in our experience, usability testing is indispensable and has saved us far more time and money than it has cost. participatory design is more costly and not as commonly done or understood, thus harder to justify. we continue to use this approach because it is our primary way of understanding our users’ needs and, more importantly, their preferred and successful work practices. this ensures that we have an informational basis – as opposed to a set of unchecked assumptions – for building technology that meets our users where they are as opposed to forcing technology upon users. and our results have justified this approach (cf. foster and gibbons, 2007). in the previous design cycle, the usability group conducted a series of mental model tests to gather information about users. graduate and undergraduate students were asked to complete key tasks on one of three academic library sites and the results were translated into key findings for the designers to use in creating initial designs for the website. the task-based nature of the test protocol did not allow the users to give us any information about what they valued about our website, or what features they would like to see added or removed; it only gave us task-specific information, such as “most students do not recognize library jargon like catalog, database, or circulation services.” the key tasks were pre-determined by the project team based on their expertise, and without user input. the participatory design workshops, however, focused solely on the personal experience of the user and were open-ended. workshop activities were designed to elicit a personal narrative, for example, “draw your perfect webpage.” these data were then used to determine what the key tasks ought to be, thus allowing the user to participate in the initial design. participants 10 faculty members participated in the workshop, representing the following disciplines: anthropology, english, history, linguistics, modern languages and cultures, political science, and visual and cultural studies. although we invited science and engineering faculty to the workshop, no one in any of those fields attended. 15 graduate students also attended the workshop, representing such disciplines and programs as chemistry, english, history, ece, mechanical engineering, optics, physics and astronomy, psychology, and visual and cultural studies. we provided coffee and cookies to all participants. additionally, we thanked the graduate students who participated by forgiving their video overdue fines. 13 undergraduates had attended the web design workshops held the previous year. we did not document any demographic information but our observations were that the groups were fairly diverse along the dimensions of age, sex, and major. key findings faculty members are all business when they use the library website, something they do more often than entering the library building. they are frequently looking for a known item; they also browse for literature in their research and teaching areas. they want to be able to get to their search – whether a voyager or database search – with no delay. they use the library web pages to support their own research and to select, designate, and review teaching materials. they sometimes use the site to see what materials other instructors are using in their classes. and they want to request and retrieve interlibrary loan materials and keep track of their loans. faculty members want a library web page that is all about their work. graduate students are heavy users of the library catalog search box, interlibrary loan, and databases. in general, graduate students begin all searches from the homepage rather than going directly (via links or bookmarks) to the pages they use most heavily, although many say that they wish they could go directly to sites within proprietary databases. by their own reports, graduate students use the library website more frequently than faculty members – nearly every day. they also use the physical library more often than faculty members, many of them on a daily or nearly daily basis. they want a library website that supports their research. in other words, graduate students want the same thing that faculty members want: a library page that supports finding articles and books. undergraduates differ from graduate students or faculty members, and they use the library homepage in a completely different way. their academic and social lives are intertwined. they want a web site that reflects that total integration of work and play. undergraduates want a portal that enables them to check their grades, get in touch with a professor, order a pizza, and find a book or article. undergraduates want to include a lot more “outside stuff” in a page or portal. they want links to library resources as well as links to professors’ pages, exam schedules, music, facebook, im, and so on. undergraduates want to personalize their library pages with color, outside links, and other features. this is a very different vision than a faculty or graduate student library page. however, undergraduates do not want to be denied access to any library resources, pages, or portals that are available to their instructors. they want the bells and whistles, but they also want to conduct searches using the library catalog, use other borrowing tools, check course resources, and find articles. they want it all. we want to point out the relatively small number of participants in our design workshops and explain that our ongoing research involves much larger numbers. however, even with small numbers the research is valuable and actionable since we are conducting design research not exhaustive, authoritative studies. in design research, the alternative to small numbers of participants is no participants at all. specific implementation analysis of the results, facilitated initially by the staff anthropologist and conducted subsequently by the project team, led to the identification of key and secondary tasks for four different groups of users: faculty members, graduate students, undergraduates, and library staff. the following table indicates these tasks and their importance to the different user groups. key tasks faculty graduate undergrad library staff find databases x x x x find books x x x x find journals & ejournals x x x x find articles x x x x find dvds     x x find reserves     x x place request (ill, off-site, purchase request) x x x x manage my account (see what i have out) x x x   get help (phone #’s, staff, chat)     x x find library hours   x x x secondary tasks faculty graduate undergrad library staff find subject resources   x x x find out about remote access & wireless x x x   find maps of buildings   x x   find a place to study   x x   access staff directory       x find all ur libraries       x log into staff modules       x go to ur webpage       x find out about library and ur news & events       x manage course reserves x     x we found that different groups want to do different things with the website and that undergraduates alone want non-library content on the library website. these findings have led us to consider creating a customizable, modular website. the design team was able to use the analyzed data to develop preliminary guidelines for design of the new site. the following is a rough, preliminary list of guidelines that were developed by our interaction designer working with members of the project team: website that knows you what's going on in the library intelligent search improved catalog website that helps you website that links you with human contact website that listens to you website that builds a community around its resources the project is currently in the initial design concept stage and the design team is working with the project team to refine the concepts, tasks, and requirements. this will lead to the selection of features and functionality and the creation of screen designs. the process will continue from there, as in figure 1. general conclusions google, amazon, facebook and other heavily-used sites create high expectations that make new demands on libraries. time and again in our workshops, undergraduates demonstrate the expectation that the library will provide a “search engine” with a single search box that searches across all material types and locations. all participants in our workshops indicate that they want to be able to customize the page both with the functionality they want and with their own look and feel. faculty members and graduate students similarly show in their workshop drawings that they want the ability to select modules for their page so that they can go directly to the library services, search tools, databases, and other resources that are most important to them. one of the biggest problems facing our libraries is that few patrons know how to use any but a few of the numerous services and resources already available on the site. for example, we can see from our research that a large number of our patrons need but are not aware of the bibliographic tools that are already available to them. one of the major challenges coming out of the web design workshops, and corroborated by other research that we have done, is to determine how to expose users to the tools already at their disposal. the obvious answer is to make everything on the website “intuitive,” but this can be impossible in the case of proprietary tools. even when the tools are part of the local code, it is hard to educate patrons on their use because there are so many tools, and they are not all on the homepage. we have yet to discover, despite our attention to this problem, a good way to solve it. the workshops have affirmed our belief that user research is a valuable tool in our design and development toolkit. as much as we value usability testing, its emphasis on task-based analysis is not sufficient for conceptualizing and building great websites. the user research described in this article is an affordable and effective way to gather information about what it is that people need to do. we can use this information to come up with tools that support the work that they do and then perform usability testing to be sure the tools are friendly, intuitive, and functional. simply asking people what they are doing and what they need their tools to do is better than making assumptions about how they do their work and what would be “good for them.” the methods briefly described here are a great way to stop assuming and start asking. appendix: questionnaire name: department: when was the last time you used the library via computer? please tell us the date and what you used it for or what you did. when was the last time you used the library in person: when and why? what three websites do you use the most on your work computer? did you use a cell phone today? what web browser do you use? â˜� internet explorer â˜� firefox â˜� safari â˜� other____________________________________________ what word processing program do you use? â˜� microsoft word â˜� wordperfect â˜� claris works â˜� wordstar â˜� tex â˜� other____________________________________________ acknowledgements the authors thank katie clark and vicki burns for their careful reading and helpful comments. we also thank katie clark for providing resources from carlson library to make our workshops possible. references foster, nancy fried and gibbons, susan. (2005). understanding faculty to improve content recruitment for institutional repositories. d-lib magazine, ii (1). http://www.dlib.org/dlib/january05/foster/01foster.html foster, nancy fried and gibbons, susan. (2007). studying students: the undergraduate research project at the university of rochester. chicago: association of college and research libraries. isbn: 978-0-8389-8437-6 http://www.ala.org/ala/acrl/acrlpubs/downloadables/foster-gibbons_cmpd.pdf foster, nancy fried and gibbons, susan. (2007). library design and ethnography. in nancy fried foster, & susan gibbons, studying students: the undergraduate research project at the university of rochester (pp. 20-29). chicago: association of college and research libraries. isbn: 978-0-8389-8437-6 http://www.ala.org/ala/acrl/acrlpubs/downloadables/foster-gibbons_cmpd.pdf schuler, douglas and namioka, aki (eds.) (1993) participatory design: principles and practices. boca raton: crc press. isbn: 0-8058-0951-1 participatory design of websites with web design workshops (coins) smith, jane mccleneghan and clark, katie (2007). dream catcher: capturing student-inspired ideas. in nancy fried foster, & susan gibbons, studying students: the undergraduate research project at the university of rochester (pp. 30-39). chicago: association of college and research libraries. isbn: 978-0-8389-8437-6 http://www.ala.org/ala/acrl/acrlpubs/downloadables/foster-gibbons_cmpd.pdf about the authors: nancy fried foster is lead anthropologist at the university of rochester, river campus libraries. she conducts work-practice studies on the academic practices of faculty members, graduate students, and undergraduates and facilitates the incorporation of findings into the technology design and development process. contact: nancy.foster@rochester.edu nora dimmock is the director of the multimedia center at rush rhees library at the university of rochester. she is the subject specialist for film and media studies and a founding member of the library’s website usability group. alison bersani is the engineering subject librarian at the carlson science and engineering library at the university of rochester. she provides reference assistance for students and faculty and supports engineering collection development for the library. subscribe to comments: for this article | for all articles 3 responses to "participatory design of websites with web design workshops" please leave a response below: participatory design for library websites? — library journal reviews, 2011-11-29 […] participatory design for library websites? november 29, 2011 by cheryl laguardia leave a comment // as i’m gearing up for this week’s clir workshop, i’ve been looking around for materials related to what we’ll be working on, and i came across, “participatory design of websites with web design workshops,” in the code4lib journal, i…. […] got web redesign resources? — library journal reviews, 2012-03-21 […] like to offer up any resources that others have found helpful. i’ve already suggested, participatory design of websites with web design workshops, by the scintillating nancy fried foster, nora dimmock, and alison bersani, but would love to hear […] week 2 reflection | idea9106 design thinking, 2014-06-05 […] article: nancy fried foster, nora dimmock, alison bersani. (2008). participatory design of websites with web design workshop (2). http://journal.code4lib.org/articles/53 […] leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – tree representations: graphics libraries for displaying hierarchical data mission editorial committee process and structure code4lib issue 6, 2009-03-30 tree representations: graphics libraries for displaying hierarchical data tree representations can be useful for presenting hierarchical data on the screen. in this article i’ll briefly describe building trees using the dojo, yahoo user interface, java server faces, and google web toolkit libraries. by mark wilhelm introduction those of us who write programs that operate on library data will sooner or later have to figure out how to present hierarchical data (i.e., thesauri, loc subject headings, etc) on the screen. usually this means some kind of tree representation. in the past, creating a tree for a program’s user interface meant a home grown solution that may or may not have been a great fit. today, however, the work has already been done for us if we know where to find it. in this article i’ll briefly describe building trees using the dojo, yahoo user interface, java server faces, and google web toolkit libraries. dojo tree you may have heard of dojo, the javascript library that enables you to do a lot of heavy lifting very quickly. dojo has, among other things, functions for all kinds of gui widgets as well as for ajax communication. dojo’s tree widget in particular is very useful and easy to set up. (for more information on these libraries, please see the dojo site at http://dojotoolkit.org/.) (i adapted the below example from dojo: the definitive guide from o’reilly media. it’s an excellent primer and i highly recommend it if you plan to fiddle with dojo at all). in dojo, trees are represented by the dojo.dijit.tree object. dojo elements are described using a dotted notation—thus tree, being in the “dijit” library of dojo is listed as “dojo.dijit.tree.” dojo comes with several libraries such as base, core, dijit, and util. dijit is the library that contains many useful user interface widgets including tree. dojo tree elements are, true to the dojo philosophy, easy to use right out of the box. all you need to do is: have a data source. for purposes of this example, we will use a data store in json format, as dojo provides a data store implementation for this format. (you could also use xml or some other data format). for more info on json, see the references at the end of this article. create an instance of dojo.data.itemfilereadstore (this is the dojo data store implementation for json format) and point it at your json data source. note the “dojo.data” reference here – “data” is another of dojo’s libraries, this one having to do with data abstraction-that is, giving you a uniform interface to access your data no matter what the format. (the library called “dojox.data” also provides canned data store implementations for other formats such as xml.) create an instance of dojo.tree.treestoremodel and point it at your itemfilereadstore. treestoremodel is a bit of abstraction – it is a layer in between the tree and the data, created so the tree itself need not care what format the data is in. treestoremodel worries about that. tree itself is concerned with how to display the data it gets from treestoremodel. here is what the sample json file would look like: { identifier : 'name', label : 'name', items : [ { name : 'modern european fine arts styles and movements', children: [ { name : 'cubist', children: [ { name : 'analytical cubist' }, { name : 'synthetic cubist' } ] }, { name : 'dada' }, { name : 'impressionist' }, { name : 'surrealist' }, { name : 'symbolist' } ] } ] } and here is what the sample html would look like: <html> <head> <title>modern european fine arts styles and movements</title> <link rel="stylesheet" type="text/css" href="http://o.aolcdn.com/dojo/1.1/dojo/resources/dojo.css" /> <link rel="stylesheet" type="text/css" href="http://o.aolcdn.com/dojo/1.1/dijit/themes/tundra/tundra.css" /> <script type="text/javascript" src="http://o.aolcdn.com/dojo/1.1/dojo/dojo.xd.js" djconfig="parseonload:true,isdebug:true"> </script> <script type="text/javascript"> dojo.require("dijit.tree"); dojo.require("dojo.data.itemfilereadstore"); dojo.require("dojo.parser"); </script> </head> <body class="tundra"> <div dojotype="dojo.data.itemfilereadstore" jsid="datastore" url="./movements.json"></div> <div dojotype="dijit.tree.treestoremodel" jsid="model" store="datastore" query="{name:'*'}"></div> <div dojotype="dijit.tree" model="model"></div> </body> </html> finally, here is what the tree would look like on the web page: and that’s it! now you have a dojo tree up and running in your web application. if you haven’t taken a look at the other dojo widgets, they are worth exploring. they will make your life as a web developer much easier! yui treeview yahoo also has a set of javascript based web components intended for out-ofthe-box use. this set of components is called the yui (yahoo ui) library. yui’s implementation of a tree component does not distance itself from the format of the data as the dojo tree component does. the yui tree expects to see data in one of two ways: either in html list format or in a formatted list of strings when instantiating the tree in javascript. in some ways this is simpler than instantiating a tree in dojo, as the html list item format is fairly simple and well-known. however, things may get complicated when you try to squeeze whatever data you have into this particular format. yui trees can be implemented in one of three ways. the first method is to instantiate the tree with a dom id of the html element that contains the html lists. the second is to actually list a formal representation of the html lists inline in the constructor. the third option is to use javascript to instantiate a tree from an existing tree. in this example, an html list is used to populate a tree: <div id="markup"> <ul> <li>modern european fine arts styles and movements <ul> <li>cubist <ul> <li>analytical cubist</li> <li>synthetic cubist</li> </ul> </li> <li>dada</li> <li>impressionist</li> <li>surrealist</li> <li>symbolist</li> </ul> </li> </ul> </div> yui javascript code is then invoked to generate the tree. note how the instantiation references the “markup” div so that yui can find the html and use it to build the tree: <script type="text/javascript"> var tree1 = new yahoo.widget.treeview("markup"); tree1.render(); </script> (note that i placed the above script within the “body” tags. the script didn’t work when placed within the “head” tags.) the tree generated by the above markup and code will look like this: java-based tools the following two methods for implementing trees are java-based and therefore assume some familiarity with java programming and concepts like the java bean. if you’re not into learning java just now, you can just skip down to the list of resources and get moving on dojo or yui! jsf tomahawk t:tree2 java server faces (jsf) is one of the many java frameworks available for quickly developing web applications. jsf is described by some as a successor to struts (http://struts.apache.org), another java framework that has gained recent popularity among web developers. as part of the framework for rapid web application development, jsf has libraries of visual components available that (in theory) you can pull in and use. “tomahawk” is one such library of visual components. available on the apache site at myfaces.apache.org, tomahawk is listed under the umbrella of myfaces, which is itself one implementation of sun’s specification for jsf. included with tomahawk is a tree component. since the components in jsf take the form of tags to be used in jsp or xhtml pages, the tree component is called t:tree2. (to avoid possible confusion, i must point out that tomahawk has two tree components: “t:tree2,” and “t:tree.” however, for reasons outside the scope of this article, it was much easier to get an example working with t:tree2 than with t:tree. of course, the readers of this article are free to experiment.) in jsf, in order to get the tree to display on the view side, we need to do some java coding on the server side. we need to instantiate a treenodebase class and then add nodes to it as needed. we do this in a server side java bean. here is an example: public class mybean { ... //create a base node treenode treedata = new treenodebase("parentnode", "modern european fine arts styles and movements", false); //create child nodes treenode childnode1 = new treenodebase("childnode","dada",false); treenode childnode2 = new treenodebase("childnode","surrealist",false); treenode childnode3 = new treenodebase("childnode","symbolist",false); //add the child nodes to the parent node treedata.getchildren().add(childnode1); treedata.getchildren().add(childnode2); treedata.getchildren().add(childnode3); public treemodel gettreemodel { .. } public void settreemodel { .. } } } // end mybean class as you can see, the treenode constructor takes three arguments. the first is a string “type” that matches the string invoked in the client jsp. the second is a “description”—this is actually the string that you want displayed on the tree in your web page. the third argument is a boolean “isleaf” parameter—false if you think this node will have sub nodes, true if it is a “leaf” or final node. in jsf, view side components can be retrieved from the server side (and vice versa) as long as we follow the rule of creating a getter and setter in the bean for that component. so for the above tree we would set it into our bean by calling: this.settreemodel(new treemodelbase(treedata)) next, it is time to set things up on the client side. we need to set up a t:tree2 tag and reference an instance of the treemodel class: <t:tree2 value="#{mybean.treemodel}" var="node" clientsidetoggle="true" > <f:facet name="parentnode"> <h:panelgroup> <h:outputtext value="#{node.description}"/> </h:panelgroup> </f:facet> <f:facet name="childnode"> <h:panelgroup> <h:outputtext value="#{node.description}"/> </h:panelgroup> </f:facet> </t:tree2> the resulting tree will look like this: note that we are using three libraries: jsf component tag libraries represented by the prefixes “f” and the “h” in addition to the tomahawk library represented by the “t.” let’s work through this code from the top. first, let’s look at the attributes of the t:tree2 tag. the “value” attribute points to the name of the treemodel instance in the bean “mybean.” in that bean there will be a “gettreemodel” getter method to retrieve the tree for display on the page. “var” is the local variable name we are giving this instance of treemodel so that we can reference its attributes locally with dotted notation; e.g., “node.description”. (node.description calls the getdescription method on that instance of treenode, returning the string we used to name that node). “clientsidetoggle” determines whether we want to handle toggling of the tree’s nodes locally (and thus keeping all the data local and loading it all at once when we load the tree) or not (in which case the data for a tree’s node would not be loaded until it was clicked). after we declare the tree2 tag itself, we then begin laying out the tree nodes using “facet” and “panelgroup” tags. the facet tag in jsf is used within other “container” components. it is used to create a named section within these containers. the panelgroup tag is used to organize the layout of visual components on the web page. the facet “childnode” will include a list of all the treenodes we labelled as “childnode” in the server side java code. the iteration through that list is handled automatically by jsf. finally, within the panelgroup tags, we output the nodes of the tree using the h:outputtext tag. this is how the tree is ultimately fleshed out. google web toolkit the google web toolkit is written in java, not javascript like dojo or the yui library. however, there is a twist. a developer writes their “ajax front end,” as google refers to it, in the java language. the java code is then compiled by google into cross-browser javascript. the approach to deploying a tree in gwt is similar to jsf in that it must be done through java, and each node of the tree must be populated in a java program. in this sense, it would seem that there is a tad more overhead in getting started with this toolkit than with yui or dojo. however, google makes it very simple to get started – i got the example below working very quickly. one of the ways google makes it easy is by giving you tools that make starting a skeletal application extremely simple. if you are using eclipse, you can generate a project directory easily by doing the following: open a command prompt and navigate to the directory where you installed the gwt download. create a directory that will hold your project files. for this example we’ll call it mygwttree. run the following command: projectcreator -eclipse mygwttree -out mygwtree you should see some files such as a .project file in your directory now. now run the command to finish creating the bare bones gwt project: applicationcreator -eclipse mygwttree -out mygwttree com.google.gwt.gwttree.client.gwttree in the “applicationcreator” command, you specify the name of the class in your new project. next, you can import this project into eclipse (using the “existing projects into workspace” option) and begin adding code. to build a tree in gwt, you need to create some instances of the treeitem class and then put them into an instance of the tree class. it’s actually pretty simple. in the code below, both the class and the “onmoduleload” method stub are auto-generated by gwt. all you need do is fill in the code to create a tree: public class newgwttree implements entrypoint { /** * this is the entry point method. */ public void onmoduleload() { // create a tree with a few items in it. treeitem root = new treeitem("modern european fine arts styles and movements"); treeitem item0 = root.additem("cubist"); root.additem("dada"); root.additem("impressionist"); root.additem("surrealist"); root.additem("symbolist"); item0.additem("analytical cubist"); item0.additem("synthetic cubist"); tree t = new tree(); t.additem(root); // add it to the root panel. rootpanel.get().add(t); } } gwt then compiles the java and generates the resulting javascript. to test your code, run the shell command generated by gwt in your “mygwtree” project directory. you should see something like this: dynamic or “lazy” loading of tree node data of course, there will be situations where a developer does not want to load data for the entire tree at one time but rather “as needed” for a particular node or nodes. some of the above libraries explicitly provide for this need; others need a little tweaking to achieve it. here is a quick summary with links to resources where available. dojo’s dijit.tree does not provide an explicit mechanism for lazy loading. however it is possible to enable lazy loading by manipulating the isitemloaded() method of (dojox.data.datastore). this article on the ibm developerworks web site by scorr johnson (http://www.ibm.com/developerworks/websphere/techjournal/0805_col_johnson/0805_col_johnson.html) describes how. according to the yui documentation, yui treeview “supports dynamic loading of node data (via in-page data or via xmlhttprequest using connection manager) .” the yahoo documentation provides a detailed example of how to implement lazy loading in a yui treeview at http://developer.yahoo.com/yui/examples/treeview/dynamic_tree.html. jsf does not have a built in mechanism for lazy loading out of the box. however some developers have shared their custom solutions to this problem at http://wiki.apache.org/myfaces/tree2. finally, i was not able to find information on whether gwt explicitly supports dynamic loading of data for tree nodes. however, if anyone happens to come across this information, i’d love to hear about it! conclusion as you can see, there is more than one way to build a tree. each of the methods of building trees above has their own strengths and weaknesses. for example, jsf has a steeper learning curve, but this is due to the strong data modelling on the server side. gwt and jsf require knowledge of the java language, while yui and dojo require only familiarity with javascript. dojo’s tree has a nicer separation of data and presenetation than yui’s treeview, but yui allows you to use simple html markup to populate a tree. more resources if you haven’t checked these libraries out, i highly recommend that you do. below are some resources if you want to do some further exploring. finally, a note on the data used for the examples above: to populate the trees i used a sample from the art and architecture thesaurus at the getty center http://www.getty.edu/research/conducting_research/vocabularies/aat/. russell, matthew. (2008). dojo: the definitive guide. sebastopol, ca: o’reilly media. coins geary, d. and horstmann, c. (2007). core javaserver faces. upper saddle river, nj: prentice hall. coins the dojo toolkit. (2009). retrieved january 15, 2009 from http://dojotoolkit.org/. google web toolkit. (2009). retrieved january 15, 2009 from http://code.google.com/intl/en/webtoolkit/. the yahoo! user interface library. (2009). retrieved january 15, 2009 from http://developer.yahoo.com/yui/. the apache myfaces project. (2009). retrieved january 15, 2009 from http://myfaces.apache.org/. introducing json. (2009). retrieved january 15, 2009 from http://json.org/. johnson, scott. “lazily loading your dojo dijit tree widget can improve performance.” retrieved march 1, 2009 from http://www.ibm.com/developerworks/websphere/techjournal/0805_col_johnson/0805_col_johnson.html. about the author mark wilhelm is a software engineer at an electronic information company in philadelphia. you can reach him at markcwil at gmail.com, and be sure to follow his information sciences blog at http://infoscinews.blogspot.com/. subscribe to comments: for this article | for all articles 2 responses to "tree representations: graphics libraries for displaying hierarchical data" please leave a response below: midas, 2009-04-02 cool! thanks. jonathan, 2009-04-23 i’m totally hooked on tech4libraries now… never even crossed my mind. outstanding. leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – leveraging google drive for digital library object storage mission editorial committee process and structure code4lib issue 48, 2020-05-11 leveraging google drive for digital library object storage this article will describe a process at the university of kentucky libraries for utilizing an unlimited google drive for education account for digital library object storage. for a number of recent digital library projects, we have used google drive for both archival file storage and web derivative file storage. as a part of the process, a google drive api script is deployed in order to automate the gathering of of google drive object identifiers. also, a custom omeka plugin was developed to allow for referencing web deliverable files within a web publishing platform via object linking and embedding. for a number of new digital library projects, we have moved toward a small vm approach to digital library management where the vm serves as a web front end but not a storage node. this has necessitated alternative approaches to storing web addressable digital library objects. one option is the use of google drive for storing digital objects. an overview of our approach is included in this article as well as links to open source code we adopted and more open source code we produced. by eric c. weig small virtual machines in my work at the university of kentucky libraries special collections research center, i rely more and more on low-cost, small virtual machines (vm) to manage small to medium sized digital libraries. [1] for example, i manage an oral history digital library that has 14,000 records, and another digital library of digitized photographs that has 110,000 records. in particular, the university of kentucky libraries use reclaim hosting as a vendor for many vms. [2] i have found over the last five years that the small vm approach is a viable solution. however, one consideration with this approach is that these small vms have limited storage space. for larger digital libraries, this makes them a good tool for managing a digital library’s front end search interface and index, but not as useful for storing lots of digital library objects such as images, audio, and video. this has made it necessary for us to explore alternate approaches to storing web addressable digital objects. one avenue we have taken for the storage of both web deliverable and large archival reference copies of digital files, is our unlimited google drive for education account. [3] moving toward this as a solution has has offered both benefits and challenges. identified benefits some instant benefits to utilizing google drive are unlimited storage space without an additional cost for our project.  google drive is also very easy to use. files can be pushed into google drive via command line scripting or through the use of an online gui upload interface.  in this way, files can even be dragged and dropped in. setting file or folder permissions is also quite easy and offers a lot of flexibility as far as controlling access. [4] identified challenges an immediate local challenge with google drive became adjusting to an object vs. file based storage system. [5, 6] with past projects, file name references included within our metadata were used to build links to digitized files held in file based storage systems configured on web servers. metadata for our digital projects includes references to related digitized files including images, video, audio, and pdf documents. since google drive is an object based storage system, these file references were no longer sufficient to build urls to access our digital objects. once our files were uploaded to google drive, the next task at hand was to find an efficient way of gathering a lot of google drive object identifiers. these were needed to build links to the new google drive objects. i knew that what we really needed was an automated way to produce a file manifest that would list file names and object identifiers for files within google drive. [7] the information from this file manifest would then need to be combined with existing metadata and loaded into our web publishing platform, omeka. sample google drive object identifier based link: https://drive.google.com/file/d/1mael63sup6eyvgsxkivxbrqkxcm1twwe/view sample standard file system based link https://somedomain.org/files/series_one/240a.0001.jpg figure 1. sample urls for an image in google drive vs. an image in a file based storage system. lastly, i needed to learn how to utilize google drive objects within our web publishing platform’s public facing item view pages. this involved learning how googlele drive urls worked, and producing custom code for our web publishing platform, omeka, to allow for easy linking to google drive items and/or embedding them within html formatted item views. object embedding, in particular, was a desirable goal. each of these challenges presented unanswered questions. in this way, the project was just like any other i had completed in the past. my approach has always been to simply dig in and figure my way through it one step at a time with hope that no insurmountable roadblocks would present themselves. google drive api scripting google drive has a robust api that allows for javascript coding to interact with stored objects. [8] searching through github and the google drive api online manual, any number of examples for such scripts can be found. i was able to utilize a simple one, with minor modifications, in order to run a process on any given google drive directory in order to generate a file manifest holding metadata describing the files in a given directory. [9] the google drive api offers a large number of descriptive data points which can be gathered. [10] for my purposes, since existing metadata sets included file names, in order to be able to map the google drive identifiers to their corresponding metadata records, i simply needed a manifest listing the file names and their corresponding google drive object identifiers. this was easily done using two established google drive data points. these specific data points are highlighted in the following code sample. var file; var filename; var gdrivelink; var row; while(contents.hasnext()) { file = contents.next(); name = file.getname(); link = file.getid(); sheet.appendrow( [name, link] ); } figure 2. google drive data points for gathering file names and google drive identifiers. setting up the script and running it were simple. [9] the resulting output of the script was a google sheet which held two columns. one column lists a file name and a second column lists the corresponding google drive identifier for the file. figure 3. example api generated manifest file. the manifest, along with additional metadata describing the digital files, was then used to construct a single csv file for batch loading the metadata into omeka. during this part of the process, an important decision needed to be made. google drive offers a number of url flavors for objects. some of these specify how access to the object will work. for example, one flavor establishes a simple hyperlink to a google drive object. another url flavor can be used to embed a google drive object within an html page. sample embed url: https://drive.google.com/file/d/<objectid>/preview sample link url: https://drive.google.com/file/d/<objectid>/view figure 4. example google drive urls for both simple hyperlinks and object embedding. once i decided on whether or not to use hyperlinks or object embedding for our item view pages, i added the url formatting into the manifest file and then merged that file into my csv holding our existing metadata. in order to accomplish this merge, i keyed on file names matched between the manifest and csv files. i was left with a single csv file including the existing metadata and new google drive object urls. luckily, there is an existing omeka plugin for batch loading csv formatted metadata. [11] so, once i had all the necessary metadata in a csv file, including the google drive identifiers, importing into the omeka database was easy. a simple plugin for omeka some bad news was that no omeka plugin for utilizing google drive files existed. i needed to create one that would satisfy the following functional requirements: parse the contents of an existing omeka item type metadata field ‘url’. if the field is not empty, determine whether or not the contents are in the form of a google drive url. if a google drive url is found, determine if the url syntax is for a simple hyperlink or an embedded object. based on information gathered in step 3, format the link to the google drive object accordingly. allow site administrators to establish configuration settings for default hyperlink text, default linking behavior, and default iframe height and width. figure 5. an image of the configuration page for the new gdrivelinks omeka plugin. in the sample plugin code below, the $text variable holds the value of the url field. this field is intended to contain the google drive url. analyzing the url syntax, the code parses the link and detects whether or not it is specifying a google drive simple hyperlink url or a google drive embed url. based on the results of this analysis, the code then formats the link for the html page accordingly. if (strpos($text, '/view') !== false) { if (preg_replace('/[&amp;lt;&amp;gt;]/', '_', filter_var($text, filter_validate_url)) === $text) { return "&amp;lt;a href=\"" . $text . "\" target=\"" . $linktarget . "\"&amp;gt;$linktext&amp;lt;/a&amp;gt;"; } else { return $text; } } elseif (strpos($text, '/preview') !== false) { if (preg_replace('/[&amp;lt;&amp;gt;]/', '_', filter_var($text, filter_validate_url)) === $text) { return "&amp;lt;iframe src=\"" . $text . "\" width=\"" . $embedwidth . "\" height=\"" . $embedheight . "\"&amp;gt;&amp;lt;/iframe&amp;gt;"; } else { return $text; } } figure 6. sample php code from web publishing platform, omeka. controls public field display for google drive urls. putting the pieces together once the new plugin was installed and configured and some of our metadata indexed, google drive objects began to appear within our item view pages. to promote the use of the plugin for others and help them visualize functionality, i created a demonstration site with some sample files. [12] here are some user-facing screen shots of the plugin in action. figure 7. example item view showing an embedded google drive video. figure 8. example item view showing an embedded multipage google drive pdf. figure 9. example item view showing an embedded jpg image in google drive. outcomes now, identified challenges have been overcome and project goals achieved: to create an efficient method for generating google drive file manifests, to match files in google drive with their corresponding metadata in csv format for loading into our web publishing platform, and to code and install a simple omeka plugin for allowing linking/embedding with google drive urls. i now manage two large digital library projects utilizing google drive for digital library object storage. i have made both the google drive api code along with instructions for use, and the omeka plugin available open source on github. [9, 13] the plugin can also be found on the omeka.org plugins page. [14] about the author eric c. weig has been an academic librarian at the university of kentucky libraries since 1998. his current title is digital library architect. he designs, builds and manages digital libraries for the university of kentucky special collections research center. notes [1] https://en.wikipedia.org/wiki/virtual_machine [2] https://reclaimhosting.com/ [3] https://www.teachthought.com/current-events/google-drive-education-now-free-offers-unlimited-storage/ [4] https://www.guidingtech.com/google-drive-sharing-permissions-guide/ [5] https://en.wikipedia.org/wiki/object_storage [6] https://en.wikipedia.org/wiki/file_system [7] https://en.wikipedia.org/wiki/manifest_file [8] https://en.wikipedia.org/wiki/application_programming_interface [9] https://github.com/libmanuk/gdrivemanifest [10] https://developers.google.com/apps-script [11] https://omeka.org/classic/plugins/csvimport/ [12] https://ukylib-exhibit-test.org/gdrive/ [13] https://github.com/libmanuk/gdrivelinks [14] https://omeka.org/classic/plugins/gdrivelinks/ subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – bringing our internet archive collection back home: a case study from the university of mary washington mission editorial committee process and structure code4lib issue 31, 2016-01-28 bringing our internet archive collection back home: a case study from the university of mary washington the internet archive is a great boon to smaller libraries that may not have the resources to host their own digital materials. however, individual items uploaded to the internet archive are hard to treat as a collection. full text searching can only be done within an item. it can be difficult to direct patrons to local resources. since 2010, the university of mary washington has uploaded over two thousand digitized university publications, including the student newspaper and the yearbook, to the internet archive. taken together, these represent almost 100 years of umw history. using apache lucy, we built a search interface, eagle explorer, that treats our internet archive collection as a cohesive whole. patrons can use eagle explorer to full-text search within the collection and to filter by date and publication. this article will describe how we created eagle explorer, the challenges we encountered, and its reception from the campus community. by katherine perdue background in 2010, the university of mary washington participated in the lyrasis mass digitization collaborative (mdc), an initiative partially funded by the alfred p. sloan foundation that helped libraries complete digitization projects that they would not have had the resources to accomplish on their own..  lyrasis partnered with the internet archive to host digitized materials so that they would be accessible to users all over the world [1].  umw digitized five collections through the mdc: academic course catalogs from 1913-2013, the alumni magazine, 1939-2011, the aubade literary magazine, 1971-2010, the battlefield yearbook, 1913-2010, and the bullet student newspaper, 1922-2010.  together they comprise some two thousand documents and a century of umw history.  over the course of the next four years, these collections were added to the internet archive, with the final issues of the bullet uploaded in the spring of 2014. the internet archive does not offer detailed usage statistics, but it does provide number of views per day per item [2].  umw’s content is all held within the university of mary washington collection, so therefore it is possible to see views per month for the entire collection.  usage has increased steadily since january 2011, totaling nearly 350,000 views [3]. it is difficult to say anything definitive about who these users are.  some of the spikes in usage correlate with times the libraries were adding to the collection or doing maintenance, and surely represent library staff use.  what is clear from looking at the libraries’ statistics is that most usage did not originate from the libraries’ website [4].  in a five month period from july 2014-november 2014 (during which there were almost 44,000 views), links from the libraries’ website to one of the internet archive collections were clicked 160 times.  the bullet newspaper was the most popular at 64 times, and the aubade literary magazine, the least, at only 6.  it is possible that our patrons were able to find the collections through other means: through google perhaps or by going directly to archive.org.  however, it is equally possible that users affiliated with umw, students, faculty, and alumni, failed to find the collections at all: collections that should have have been highly relevant to them. moreover, although lyrasis and the internet archive allowed us to digitize and make available collections that we would never have been able to host on our own, they did not do so in a way that allowed our patrons to accomplish what they most wanted from an online archive.  looking back at our records of reference questions, the most frequent request was to find all mentions of a specific person, event, building, or umw tradition: be it alumni looking for articles about themselves or that they wrote when they were on the bullet staff, family members in search of their mother’s yearbook photo, researchers trying to find information about the construction of the amphitheatre, or students looking for pictures of devil-goat day [5] festivities long past.  previously, the only way to find such material was to consult the bullet index, a detailed subject index first on cards and later online as a series of pdfs.  however the index was cumbersome to use and only covered materials through 2004.  there was no index for the other collections.  the way modern users expect to find such information is via full-text search. when a file is added to the internet archive, an ocr’d text is automatically created.  it is easy to search within the text of each item.  it is not possible, however, to search within multiple items: to find, for example, as one student wanted to do, all mentions of apartheid anywhere in any issue of the student newspaper.  each issue would have to be searched separately.  the student in question elected to page through paper copies instead. the libraries do not have vast reserves of server space.  for the present, it isn’t feasible to host all of our digital collections ourselves.  additionally, we have no desire to replace or duplicate functions that the internet archive is already fulfilling.  we are happy with the way that items are presented there: the book reader viewer is a pleasant way to read and users are able to download content in many different formats.  we simply wanted to make sure that our patrons were able to view our collections in a way that was cohesive and coherent and that allowed them to find the information they most wanted.  to accomplish these goals, we built eagle explorer (named for our mascot), an interface and search engine for umw’s internet archive collections [6]. the project we chose to use the apache lucy search engine library to power eagle explorer.  lucy is a “‘loose’ port” of lucene in c with perl bindings [7].  lucy worked well for us because we were already using perl for several other library projects and lucy’s features met our needs for this project.  it allows for boolean search and (of particular interest in this case) range searching, so that users could search within a date range.  it is also fast, easy to set up, and modular, with many options for tweaking results [8]. to build the index, the first step was to obtain the text for all items.  the internet archive uses a predictable url structure: if the identifier for an object is known, it’s possible to deduce the url for the text (https://archive.org/stream/identifier/identifer_djvu.txt) and many other useful views (the book reader viewer, for example, at https://archive.org/stream/identifier, and the cover image at https://archive.org/services/img/identifier).  because of this, it was easy to automate the download using our list of identifiers. once we had the text archive, we chose five fields to index: identifier, title, content (the full text), date, and source (which publication the document was from).  the identifier and source we already had and the content we had just downloaded.  the title and date were trickier to obtain.  because the lyrasis mdc project had occurred over a period of four years, during which there was staff turnover both at lyrasis and umw, no consistent naming convention had been applied to identifiers.  most identifiers contained at least the year and some form of the title, but even within the same publication, the exact formatting varied.  some did not contain a date at all.  for these, we used screen scraping (using web::scraper [9]) to pull the title, which did contain the date, off the internet archive page at the same time we were copying the text. for my $id (@identifiers) { sleep 2; #two second delay between items to be polite my $url &nbsp;= "http://archive.org/stream/$id/" . $id . "_djvu.txt"; #location of full text my $scraper = scraper { process "div.container pre", fulltext =&gt; 'text'; #div containing full text process "div.container h1 a", title =&gt; 'text'; #div containing title }; … } further complicating our efforts, the alumni magazine had changed titles eight times and the course catalog, eleven; not surprising at a school that has changed names almost as many times.  with much parsing and massaging of data, we managed to extract consistent titles and dates that at the very least included the year from all items. a good example of this process is the 1952 spring issue of the alumni magazine. the identifier for this item is “alumnaeassociati00univ”, which gives no clue about the date. the title in the internet archive is “alumnae association – news bulletin, 1952 (spring)”. although date formats were not consistent in titles (they might also appear as “alumnae handbook and the news, 1942-43” or “alumnae news, 1961 (february)”) the format [title],[date] was consistent, so we were able to separate the two and extract only the date. ultimately, it proved short-sighted not to retain the title as well, since this was something we had to deal with again in a later step. at this point, we had a file that contained the full text of the issue and was named “alumnaeassociati00univ,1952 (spring).txt”. a series of unix shell commands converted the various season and month names to numbers. in this case: for filename in *\(spring\)*; do mv "$filename" "${filename/ \(spring\)/-03}"; done leaving us with “alumnaeassociati00univ,1952-03.txt” ready to be indexed. during indexing, we used the identifier to distinguish between eight possible alumni titles; anything beginning with alumnaea was translated to ‘alumnae association’ versus alumnaeb, which would have made it ‘alumnae bulletin’, and so on. lucy uses ‘analyzers’ to filter text before matching it to a query [10].  applying these analyzers is part of the indexing process.  for identifier, title, date, and source, we did not use any analyzers, since they contained exact values and finding near matches would not be useful.  for content, we found that the defaults used in lucy’s easy analyzer module suited our purposes.  easy analyzer combines lucy::analysis::standardtokenizer, which splits strings into words, lucy::analysis::normalizer, which normalizes to unicode, and lucy::analysis::snowballstemmer, which is language specific and reduces words to their roots, so that searching for ‘horses’ also returns results for ‘horse’.  in the future, we may want to investigate whether using different analyzers would improve search results.  in particular, the stemmer is over-aggressive: searching for “john williams” in quotation marks should not return results for “john william”.  however, for now, this approach works well enough. it was important to us that users be able both to search for specific keywords and to browse by date.  the interface for eagle explorer features a timeline at the top [11].  in browse mode, the user selects a year on the timeline, then can move forwards and backwards in time by pressing the arrows on either side or moving the marker on the timeline [figure 1].  matching titles and their covers are displayed below and if a user clicks on one, the internet archive page in book reader view opens in an iframe [figure 2].  if the user enters a search term, then the timeline shows an expandable date range instead of a single year.  the term must occur between the two years indicated.  if the user changes the values on the timeline, a new search is run.  the user may also choose to change the sort order from relevance to date.  at any time, users may also choose to limit the search to one publication source or to search all of them at once. figure 1. eagle explorer timeline selector figure 2. internet archive book reader in iframe all of this requires slightly complex search logic.  fortunately lucy allows multiple search queries to be combined via lucy::search::andquery.  the relevant code follows, but in brief, if there is a search term, it is parsed and the results are then limited by date or date range and by source if set, then ordered by date or relevance.  if there is no query, then all items are treated as a match, then limited down in the same way. my $searcher = lucy::search::indexsearcher-&gt;new( index =&gt; $indexpath, ); my $qparser &nbsp;= lucy::search::queryparser-&gt;new( schema =&gt; $searcher-&gt;get_schema, ); if ($q ne '') { #if there is a query, then parse it $query = $qparser-&gt;parse($q); } else { #if no query, match everything $query = lucy::search::matchallquery-&gt;new; } my $rangequery = lucy::search::rangequery-&gt;new( field &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;=&gt; 'date', lower_term &nbsp;&nbsp;&nbsp;=&gt; $afterdate, upper_term &nbsp;&nbsp;&nbsp;=&gt; $beforedate, include_lower =&gt; 1, include_upper =&gt; 1, ); #limit the results by the date range $query = lucy::search::andquery-&gt;new( children =&gt; [ $query, $rangequery ] ); my $sourcequery = lucy::search::termquery-&gt;new( field =&gt; 'source', term &nbsp;=&gt; $source, ); #if a source is selected, limit the date range query by source if ($source ne "all") { $query = lucy::search::andquery-&gt;new( children =&gt; [ $query, $sourcequery ] ); } my $sortspec = lucy::search::sortspec-&gt;new( rules =&gt; [ lucy::search::sortrule-&gt;new( field =&gt; 'date' ), ], ); #if searching, not browsing, keep the default relevancy sort if ($type eq 'search' &amp;&amp; $sort ne 'date' ) { $hits = $searcher-&gt;hits( query =&gt; $query, offset =&gt; $offset, num_wanted =&gt; $page_size, ); } #otherwise, if browsing or if the user wants to sort by date, sort by date else { $hits = $searcher-&gt;hits( query =&gt; $query, offset =&gt; $offset, num_wanted =&gt; $page_size, sort_spec =&gt; $sortspec, ); } reception eagle explorer launched on july 6th, 2015, at the tail end of summer session.  the libraries posted an announcement on our facebook page.  our director emailed key people on campus who might be interested in using it.  usage was moderate, with 96 sessions the first week.  users were clearly engaged with the content, spending an average of just over nine minutes on the site and viewing an average of sixteen pages per session.  then the alumni association posted a link on their facebook page and eagle explorer was deluged with alumni looking themselves up.   within two days, thanks to the alumni, eagle explorer had received 201 referrals from facebook, more traffic directly through the libraries than the internet archive collection had received in five months the previous year. figure 3. a search for the author’s grandmother yields newspaper articles, yearbook photos, and a graduation announcement usage has continued to be robust.  it is hard to provide statistics that are directly comparable to the data from before eagle explorer because once a user left the libraries’ website, it is impossible to know what they did on the internet archive site.  what can be said is that where we had very little evidence in the past that our users were actually finding the internet archive collections, the evidence now is ample that they are finding them and that they are making meaningful use of them.  eagle explorer accounts for 3.5% of the activity on the libraries’ website.  between july and november 2015, there have been 1527 unique views of items in the collection through eagle explorer (a unique view is unique either because it is the first time that item has been viewed or because it contained a search term; search terms are passed on from eagle explorer to the internet archive book reader viewer so that users can quickly find what they are looking for).  roughly 60% of views included searches.  adding to our conviction that this use has been meaningful, during the same period, links to eagle explorer have been shared to facebook or twitter 27 times, often directly linking to a particular search term, date, or item. feedback from users has been positive.  we have been able to strengthen ties between the libraries and university relations, who have been using eagle explorer to help personalize outreach to alumni.  our reference logs reveal that students and faculty have been using it to find more information about events mentioned in campus histories and to browse particular decades of the yearbook and the bullet. future improvements while we are happy with eagle explorer, there is always room for improvement.  we are in the process of scanning the missing issues of the bullet from 2010-2014, when it changed names and became the blue & gray press.  we also plan to add a collection, the bayonet student handbooks. as noted above, there may be improvements we can make to search results by using different analyzers.  lucy also has different ways of weighting relevance that may provide better results and should be investigated.  one such is the default bias towards shorter documents.  using lucyx::index::longfieldsim should remove this bias, but we have not yet had the chance to see if this improves results [12]. eagle explorer has an as yet undocumented api that we are using to pipe results into a bento-style search for special collections.  a faculty member contacted the libraries about using eagle explorer to set up a twitter bot that could automatically tweet what our students were writing about fifty years ago, for example.  we hope to use the api to help with this project and others that have yet to be thought of. conclusion one of the most important things that sets digital objects apart from physical ones is that physical objects have a set place in the world and a set spatial relationship with other objects.  in libraries of the past, this meant a library’s collections were held physically within the library, placed according to call number, and any alternative ordering had to be done laboriously with cards, which also had a set location and set order.  relationships and ordering are still important for digital objects; digital objects, like anything else, derive much of their meaning from their placement within a collection.  but with digital objects, there are no limitations on order and relationship.  they can be sorted and displayed in any order, anywhere, in whatever manner best meets the needs of whatever community of users might want them.  they can be part of the vast collection that is the internet archive, available to researchers around the world who may want to compare, for example, student newspapers of the 1930s across the united states.  at the same time, without needing extra copies on extra servers and without disrupting other users, we were able to bring our digital objects home with eagle explorer so that our users could find what they wanted in a single, unified mary washington collection. notes [1] http://americanlibrariesmagazine.org/2011/08/10/step-easily-into-the-digital-future/ [2] https://archive.org/about/faqs.php [3] the collection and the usage statistics can be found here: https://archive.org/details/universityofmarywashington&tab=about [4] all usage statistics unless otherwise noted were collected with google analytics. these numbers are approximations: the actual numbers are likely to be higher because some people block analytics cookies and because some clicks don’t register. additionally, google uses sampling techniques that make it difficult to know how accurate the data is. [5] an annual competition between the devils (students graduating in an odd year) and the goats (even year graduates): http://spinningwheel.umwblogs.org/devil-goat-day/ [6] live project site: http://libraries.umw.edu/eagle-explorer [7] http://lucy.apache.org/faq.html [8] all modules are documented here: http://lucy.apache.org/docs/perl/ [9] http://search.cpan.org/~miyagawa/web-scraper-0.38/lib/web/scraper.pm [10] http://lucy.apache.org/docs/perl/lucy/analysis/analyzer.html [11] we used the nouislider, a lightweight javascript slider whose claim to fame is that it does not use jquery ui, to create the timeline. http://refreshless.com/nouislider/ [12] https://lucy.apache.org/docs/perl/lucyx/index/longfieldsim.html references anderson, k, gemmill, l. 2011. step easily into the digital future: lyrasis gets varied collections online quickly, and cheaply, through collaboration. american libraries 42(7/8): 36-39. http://americanlibrariesmagazine.org/2011/08/10/step-easily-into-the-digital-future/ apache lucy. (n.d.). apace lucy. http://lucy.apache.org/ internet archive. (n.d.). internet archive: digital library of free books, movies, music, & the wayback machine. https://archive.org about the author katherine perdue (kperdue@umw.edu) has been the assistant systems librarian at university of mary washington libraries since 2013. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – evaluating htj2k as a drop-in replacement for jpeg2000 with iiif mission editorial committee process and structure code4lib issue 57, 2023-08-29 evaluating htj2k as a drop-in replacement for jpeg2000 with iiif jpeg2000 is a widely adopted open standard for images in cultural heritage, both for delivering access and for creating preservation files that are losslessly compressed. recently, a new extension to jpeg2000 has been developed by the jpeg committee: “high throughput jpeg2000,” better known as htj2k. htj2k promises faster encoding and decoding speeds compared to traditional jpeg2000 part-1, while requiring little or no changes to existing code and infrastructure. the iiif community has completed a project to evaluate htj2k as a drop-in replacement for encoding jpeg2000 and to validate the expected improvements regarding speed and efficiency. the group looked at a number of tools including kakadu, openjpeg, and grok that support htj2k and ran encoding tests comparing the encoding speeds and required disk space for these images. the group also set up decoding speed tests comparing htj2k with tiled pyramid tiff and traditional jpeg2000 using one of the major open source iiif image servers, iipimage. we found that htj2k is significantly faster than traditional jpeg2000, though the results are more nuanced when compared with tiff. by glen robson, stefano cossu, ruven pillay, michael d. smith introduction the international image interoperability framework (iiif) has emerged as an open, standards-based solution for making high-quality images available on the web in a way that is efficient and interoperable. the first api to be released was the iiif image api which provides a way of constructing a url that defines a particular region and size of an image. there are a wide number of implementations of the image api in the form of iiif image servers but the iiif standards deliberately avoid prescribing a single file format. instead, they focus on the server-client interaction and the api. many image servers work with a variety of image formats including jpeg, png, and tiff but naturally some file formats are better suited to satisfying iiif requests, particularly image formats that store the image data in multi-resolution forms such as image pyramids (figure 1) where different resolutions or sizes can be retrieved quickly. figure 1. multi-resolution tiled pyramid structure used in tiled pyramid tiff another feature of the multi-resolution image pyramid above are the tiles that make up each of the different resolution levels. similar to the way online mapping tools like google maps achieve efficient zooming into a map, this tiling allows an iiif viewer to only request the regions of the image (or tile) that are currently being viewed. this means that the user can quickly dive into the detail of an image without downloading the full sized image. in the iiif community, the most popular formats which provide these image pyramids are currently jpeg2000 (jp2) [1][2] or tiff (tiled pyramid). high throughput jpeg 2000 (htj2k) is a new addition to the jpeg2000 family of standards [3][4]. htj2k is a replacement for the slow block coder in jpeg 2000 part-1 and the new standard is designed as a drop-in replacement to work with existing jpeg2000 encoding/decoding tools. in addition to choosing the image format, users of jp2, htj2k, and tiff images need to decide whether to compress the images using lossless or lossy compression. lossless compression means that the original image bitmap can be re-created exactly and that none of the data is lost in the conversion. lossy compression results in much smaller files that aim to be visually close to the original, but the conversion is not reversible. in the iiif context, some institutions have separate preservation copies of images and so can choose to go for lossy compression for their images made available for access through iiif. there are both commercial and open source tools for encoding and decoding jp2 and htj2k images and this paper describes tests undertaken to determine their level of support for htj2k and how performant they are compared to each other. tiff is an open format with a wide range of available tools. in particular, the widely-used libtiff library and image processing tools such as vips or imagemagick. the tools used to encode htj2k image were: kakadu, a commercial sdk and command-line toolkit used to encode and decode several still and moving image formats; openjpeg, a popular open source jpeg toolkit that includes support for jpeg2000 part 1 (traditional jp2) and part 15 (htj2k); grok, another open source toolkit with several optimizations for jp2 and htj2k. support for htj2k at the time of these tests was still emerging and a summary of this support for various libraries is shown below: kakadu grok openjpeg htj2k lossy encoding yes partial* no htj2k lossless encoding yes yes no htj2k lossy decoding yes yes yes htj2k lossless decoding yes yes yes this table was correct at the time of testing in may 2022. * grok clarified that lossy encoding can be done but the bit rate or quality of the compression is currently not configurable [issue tracker]. encoding tests and storage comparison the first part of testing the suitability of htj2k for iiif is to look at the encoding speed. for these tests we used a sample of 1,000 tiff files from the getty museum and getty research institute. the test set included a variety of sizes to reflect what can be found in typical institutional collections. generally, for iiif use, encoding is performed once whereas decoding is done for every iiif request. the encoding often happens soon after an item is digitized and can be part of the digitization workflow or part of the ingestion process into a repository. the following tests were run natively on bare metal using linux (ubuntu 20.04 lts). the server used was a powerful 16-core intel(r) xeon(r) cpu combined with raid 10 disk array storage. at the time of testing, support for htj2k was emerging in the open source libraries openjpeg and grok but was already established in the commercial kakadu library. this led to some of the tests only being supported in some libraries. the encoding tests undertaken were: tiff encoded using vips jpeg2000 part 1 using openjpeg, kakadu and grok htj2k using kakadu and grok (lossless only for grok / openjpeg unable to encode htj2k) the sizes of the files generated are represented in the following graphs (figure 2). figure 2. average compression ratios for each encoder and format for both lossy and lossless compression (lower is better). the vertical axis represents the ratio of the resulting image file size with respect to that of the original uncompressed file. a ratio of 1 would indicate that the compressed file is the same size as the original the tests for each combination of format and codec were carried out 3 times for each of the 1000 images with the times averaged for reach encoder/format combination. the results show: jpeg2000 can achieve significantly better compression than tiff for both lossy (jpeg) or lossless (deflate) htj2k compressed files are slightly larger than for jpeg2000 part 1 different jpeg2000 codecs produce almost identical output results when comparing the encoding time are represented in the following graphs (figure 3): figure 3. average relative encoding time for each encoder and format combination (lower is better). the vertical axis represents relative encoding time with respect to the fastest encoding combination. the fastest (kakadu/htj2k) is, therefore, equal to 1 the encoding times vary significantly between encoders with lossy tiff encoded with vips taking roughly 7 times longer than with kakadu encoded to htj2k. encoding to htj2k is faster than to jp2 for both kakadu and grok. the encoding speed also depends on the amount of data that needs to be encoded and, therefore, on the size of the source image. the relationship between image size and encoding time can be seen in the following graph (figure 4): figure 4. average encoding time vs raw image size. the vertical axis represents the absolute time in milliseconds and the horizontal access, the raw image size in bytes. the results show: there are large differences between codecs kakadu is significantly faster for all types htj2k encoding is faster than jpeg2000 part 1, especially for lossless encoding lossless jpeg2000 encoding is faster than lossy encoding for all codecs and jpeg2000 types there  is a strong linear relationship between image size and encoding speed summary on encoding when evaluating the encoding, the resulting size on disk might be a more useful metric than encoding speed because encoding images for use with iiif is only carried out once whereas decoding is done for every iiif request. as seen in these results, jp2 and htj2k produce significantly smaller images with respect to tiff for both lossy and lossless compression. htj2k images are marginally larger than jp2. if encoding speed is important, then the commercial library kakadu currently has a speed advantage over the open source libraries. with the lack of support for lossy htj2k encoding in the open-source libraries, it is difficult to say definitively that encoding to htj2k is faster than encoding to jp2 for all libraries, but it is clearly faster with kakadu for both lossy and lossless and faster with grok for lossless. decoding tests testing decoding is important, as it occurs for every iiif request made by a client. there are a number of different types of requests a client can make. types of requests include: info.json requests the image api provides an info.json response where a client can request a json file that contains essential metadata about the image, for example: full width and height, or tile information. during normal use, this file would be requested once per image and is usually cached by an image server, as it is often requested. the information in the info.json response is generated by the image server using information obtained from the metadata within the image file itself. results for testing these types of requests have not been included in this article but timings were similar for all types of jp2 and slower for tiff. however, as these files are often cached by the image server the performance difference between formats is unlikely to be noticed by users. tile requests these are the most common requests made by iiif viewers. both jpeg2000 and tiff define the size and number of tiles at different zoom levels which the client will then request. some of these tiles may be cached but it depends if multiple users are looking at the same region of an image. full region requests these are requests for the full image at different sizes. thumbnail-sized images are requested often by clients whereas full-sized images are very rarely requested by iiif viewers but may be requested if users wish to download the image at a particular size. custom region requests in the iiif image api, it is possible to build a url to request an arbitrary region of an image that can be larger than a single tile or not aligned with the boundaries of the existing tile structure. this url can be generated, for example, using an iiif cropping tool. generally, there won’t be many requests that fall into this group, but if the link is shared on social media or in a blog it has the potential to get many requests. if the same link is accessed multiple times, it will likely be cached. results for this type of request have not been included in this report but can be found in the group’s full results. testing decoding libraries the iipimage server supports both the openjpeg and kakadu codecs and can handle both jp2 and htj2k. it also provides precise (microsecond) timing information for each internal process, which allows the extraction of precise timing for the decoding part of the process. there are a number of iiif image servers, but iipimage was chosen for these tests. for the decoding tests, the 50 largest images from the getty test set were chosen and a set of random tile requests was generated. 100 random iiif tile requests were made at different regions and sizes for each image, resulting in 5000 requests in total. each request was made 3 times and the resulting times averaged. the same 5000 iiif requests were used for all decoding tests. grok is not supported by iipimage, so only openjpeg and kakadu were included in this round of testing for jpeg2000. results can be seen in figure 5. figure 5. average relative decoding times for each tile request for each combination of decoder and format (lower is better). the vertical axis is normalized such that the fastest encoder/format combination (lossy tiff in this case) is defined as 1. the results show: tiled pyramid tiff is by far the fastest format for random tile access on large images for jpeg2000, openjpeg remains significantly slower than kakadu: 10x slower for lossy encoded images and up to 15x slower for lossless encoded images htj2k decoding is about 2x faster for lossy jpeg2000 and ~20% faster for lossless for both openjpeg and kakadu lossy encoded images are quicker to decode than lossless the next set of tests used the same 50 largest images from the getty test set but replicate a more typical image server setup with the timings being the full end to end timings that an iiif viewer would receive including the http request handling. the iiif image server, again iipimage, was set up using docker and was run on a 2020 macbook pro alongside the testing scripts. only the kakadu library was used to focus on the differences between jp2 and htj2k rather than the difference in library processing speeds. tiff is included as a comparison. the first set of tests looked at the full region of the image scaled to different sizes. a 50-pixel image would be a small thumbnail whereas a 3000-pixel image would be a large image that a user might want to download. images that are requested often like thumbnails would usually be cached in the iiif infrastructure. figure 6. average response time for a 50px width image request for tiff, jp2 and htj2k for both lossy and lossless compression (lower is better). the results are shown in figure 6 and show: htj2k is slightly faster than jp2 and tiff for very small images. at these very small sizes the difference between lossy and lossless is very small. the next test looks at the other end of the scale where the image is requested with a width of 3000 pixels. figure 7. average response time for a 3000px width image request for tiff, jp2 and htj2k for both lossy and lossless compression (lower is better). the results are shown in figure 7 and show: jp2s are considerably slower for larger images htj2k and tiff are very similar for large images, though tiff is slightly faster these tests were repeated for five different widths: 50 pixels, 500 pixels, 1024 pixels, 3000 pixels and full sized images. the results for both lossy and lossless compression are shown below. figure 8. average response time vs requested image width for lossy tiff, jp2 and htj2k the results (figure 8) show: tiff and htj2k results are similar for most sizes jp2 (jpeg2000 part 1) is significantly slower for 50, 500, 1024 and 3000 pixel images but is similar for full sized full region images. figure 9. average response time vs requested image width for lossless tiff, jp2 and htj2k the results for lossless (figure 9) show the same trends but the difference between jp2 and the other two file formats is much more significant. the next two results use urls generated from opening up the images in two of the common viewers in the iiif community: universal viewer (uv) and mirador. this is intended to give a realistic set of results that could be expected from using one of these viewers. for both viewers, the urls were generated by creating a manifest with all 50 images in and then navigating between all of the images and zooming around the first few images. the test urls include both thumbnail and tile requests. figure 10. average response time for mirador (top row) and universal viewer (bottom row) for lossy and losslessly compressed tiff, jp2 and htj2k both viewers produce very similar results (figure 10) and from these results it can be said that tiff is slightly faster than htj2k but both formats are considerably faster than jp2 decoding conclusions through the majority of tests it can be seen that tiled pyramid tiff is the fastest format for iiif requests both for tile requests and for most of the full region tests. htj2k has performed well and has proven faster than jp2 in all cases and is nearing tiff performance particularly with full region requests. recipes as part of this work and after further investigation, the recommended parameters for creating optimal jpeg2000 files for both htj2k and jp2 with kakadu are as follows: htj2k lossy: kdu_compress -i input.tif -o output.jph \ cmodes=ht \ creversible=no \ qfactor=90 \ cprecincts="{256,256}" \ orggen_plt=yes \ cblk="{64,64}" \ clevels=8 htj2k lossless: kdu_compress -i input.tif -o output.jph \ cmodes=ht \ creversible=yes \ orggen_plt=yes \ cprecincts=”{256,256}” \ cblk="{64,64}" \ clevels=8 jp2 lossy: kdu_compress -i input.tif -o output.jp2 \ creversible=no \ qfactor=90 \ corder=rpcl \ cprecincts="{256,256}" \ orggen_plt=yes \ cblk="{64,64}" \ clevels=8 jp2 lossless: kdu_compress -i input.tif -o output.jp2 \ creversible=yes \ orggen_plt=yes \ corder=rpcl \ cprecincts="{256,256}" \ cblk="{64,64}" \ clevels=8 more details on the recommended recipes for htj2k and jp2 can be found on the iiif guides site. full results, image server docker and encoding scripts can be found on the group’s github repository: https://github.com/iiif/htj2k conclusion the testing clearly shows that tiled multi-resolution pyramid tiff is the fastest format for iiif, but it comes at a cost of significantly more storage space compared to both htj2k and jp2. so the question of which format to choose will depend on the size of the image collection that people are working with. for small collections with high visitor traffic, the storage costs for tiff may potentially be outweighed by the increased performance. for those with larger collections which have many rarely accessed images, the scales might be balanced the other way and reducing storage costs may be more important than the slight performance increase with tiff. generally, decoding lossy images is faster than decoding lossless images. one of the reasons for this is due to lossy files being smaller than the lossless files, and so there is less entropy decoding work to do since there is less entropy coded data to process. for users already using jpeg2000, the results show a clear performance advantage to migrating from jp2 (jpeg2000 part 1) to htj2k (jpeg2000 part 15) for all iiif requests tested. there is a slight increased storage cost for htj2k but the improvement in performance could outweigh the increase. note when migrating to htj2k this requires transcoding to a different file format, which uses .jph as its file extension. it would be useful to do further work on testing the decoding support from the open source jp2 libraries as their support for htj2k matures. notes [1] https://www.itu.int/rec/t-rec-t.800/en [2] https://www.iso.org/standard/78321.html [3] https://www.itu.int/rec/t-rec-t.814/en [4] https://www.iso.org/standard/76621.html about the authors glen robson (orcid: 0000-0003-2008-5243) works as the iiif technical coordinator for the consortium of institutions that support iiif. his main role is to help others implement iiif through training and working with the community of volunteers to extend the specifications and increase the amount of software that supports iiif. stefano cossu (orcid: 0000-0003-2474-1180) is a digital repository architect at harvard university it. previously he worked at the getty trust on designing and building etl pipelines and delivery services for iiif images, and before then he led a team at the art institute of chicago, which introduced iiif-based services for public image and metadata delivery. ruven pillay (orcid: 0000-0001-5398-1365) is a research scientist working on the application of advanced imaging and data analysis techniques to the study of art. he works at the c2rmf, the national research center for art housed within the louvre in paris and has over 25 years of experience working in major art galleries, having also worked at the national gallery, london and the nationalmuseum, stockholm. he is also a software developer and maintains iipimage, an open source image server, widely used within the iiif and scientific imaging communities. michael d. smith (orcid: 0000-0001-6573-9883) has worked as a digital imaging consultant for 20 years. he has worked on projects for many organizations including kakadu r&d pty ltd., warner bros., sony pictures entertainment, dci, noaa, dolby, and dts. he was co-editor of the high-throughput jpeg 2000 image compression standard that was published in 2019. in 2018, he received a screen credit for his color science work on mary poppins returns.  from 2013-2015, he was co-chair of blu-ray disc association’s uhd-tf video subgroup which defined the video-related requirements for the ultra hd blu-ray disc format. in 2012, mr. smith’s work on warner bros. digital end to end (dete) system led to winning a technical emmy.  mr. smith was editor of the book “3d cinema and television technology: the first 100 years” published by smpte in 2011 and achieved smpte fellow in 2019. he received the b.s. and m.s. degrees in electrical engineering from ucla in 2001 and 2004 respectively. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – node-based configuration management architecture for private lockss networks mission editorial committee process and structure code4lib issue 34, 2016-10-25 node-based configuration management architecture for private lockss networks node-based configuration management describes a services architecture for private lockss networks that transfers administrative services onto a peer preservation node in the network. the architecture also describes techniques for enabling full redundancy of data for configuration administration utilizing the preservation protocols in lockss. the goal of node-based configuration management is a horizontal administrative model where any peer node can assume administrative services with complete redundancy of configuration data across all nodes. by tobin m. cataldo lockss lockss (lots of copies keeps stuff safe)[1] is digital preservation software that creates a preservation model by emphasizing geographical distribution of identical replicas across multiple, potentially heterogeneous, storage devices or preservation nodes. the core concept is that systemic failure at any single location does not affect the validity and reliability of data preserved in the other nodes. lockss is a java software suite that runs well on centos.[2] lockss is typically installed via rpm.[3] in most cases, preservation storage is defined on disk partitions over raid 5 or raid 6. lockss is open source and the code repositories are available.[4] lockss supports two generic use cases of the preservation software: post-cancellation access to electronic serials content, or global lockss network (gln), and closed networks of known peers for distribution and preservation of digital data, or private lockss networks (pln).[5] both use models can operate under the same technical and administrative architecture. membership in the lockss alliance is requisite for participation in the gln as well as many extant plns.[6] the alabama digital preservation network (adpnet)[7] is a private lockss network originally modeled under the technical and administrative architecture that characterizes the global lockss network. in the original architecture, lockss controls the single authoritative source for the distributed preservation network. while this model has worked well for many years, adpnet, like any maturing network, experienced both a need for more internal control as well as a need to identify and reduce external vulnerabilities that potentially could affect continuous service. as a result, adpnet researched and instituted preservation node-based configuration management. node-based configuration management is an alternative management model that seeks to flatten a hierarchical administrative model by enabling the preservation node to serve as the network’s administrative source and create a more failsafe network with redundancy.[8] in node-based configuration management, a preservation node is promoted to the configuration services provider and exposes network configuration data through ancillary software. the goal of this model is to create an administrative system that can be transferred to any preservation node without significant impact on continuous preservation services.this administrative model has been in place at adpnet since december 2015. the node is the master private lockss networks are closed networks with established perimeter controls that identify known preservation peers. critical configuration data for the preservation software are referenced by url, and in most instances name-based url routing is utilized. a pln takeover of configuration services could follow two routes: bring new software online on new hardware (not a known preservation peer); or promote a preservation peer to the configuration services provider and enable the other peers to ingest the data necessary for node promotion using the lockss preservation software. as a distributed network, discrete pln infrastructure is typically under direct administration of the hosting institution, and a pln may have little or no centralized administrative control at the firewall or the server. changes in configuration services endpoints that result in ip changes could result in a bottleneck for continuous service while various security and access lists are updated. this is particularly true when configuration services are brought up on new hardware at a member institution. should that hardware fail or network membership changes force a relocation of configuration services, new server infrastructure would need to be in introduced and each network member would need to add firewall and access controls. the second method of modeling configuration services takeover is to utilize a preservation node and promote it to the configuration services provider. in order to minimize preservation network vulnerability and maximize redundancy, it is not enough to simply promote a node. rather, the promoted node should expose the data necessary to promote a peer preservation node with little downtime and without loss of data. typically, this can be accomplished by introducing ancillary software onto the preservation node to expose the configuration data as sources and defining the exposed sources as an archival unit in the title list. preservation nodes must be known to the peers, and leveraging a known network host to provide non-intensive services is the most seamless way to achieve continuous service without additional investment in redundant hardware and infrastructure. the services a promoted node should provide: lockss.xml title list plugin repository signing keystore server configuration data exposed as an archival unit (au)[9] the resources needed to provide these services: storage web server data store (database) processing language there are various methods of delivering the configuration data, and most of the data could be reduced to static files. the lockss.xml file is relatively unchanging as are the plugins. in some scenarios, the title list could also be relatively unchanging. for unchanging data, simply exposing that data over http would suffice. however, a moderate to heavily used pln should incorporate architecture that enables reuse of title list components to enable additional administrative services, such as centralized au status reporting, and more complex storage design, such as data partitioning.[10] dynamically creating the title list also lends itself to limited recovery crawls via host specific title list parameters (i.e. crawl_proxy).[11] host specific title list parameters facilitate peer-to-peer recrawls without impacting network state. moreover, there is less potential for error by maintaining a single definition set for multiple purposes rather than multiple definition sets for a single purpose. for these reasons, the configuration services model at adpnet was based on relational database storage with dynamic assembly and delivery of xml streams over http. node-based configuration services software the goal of node-based configuration is to create a network administrative model that can be transferred to any of the preservation nodes. any software utilized should be stable, common, small, easily configured, and have potential for extensibility. preservation nodes take a long view and stability is a must. by selecting common software, there is a greater likelihood of pre-existing institutional knowledge of software package administration. as configuration services provision is not the primary purpose of a preservation node, software should be small and utilize minimal resources. software should also be easily configured so as to minimize the potential downtime during transference of configuration services. software should also demonstrate capacity for extensibility so new changes can be incorporated without a full stack rework. storage the preservation node storage array (in most cases spinning disks in a raid 5 or raid 6 configuration) provides sufficient storage space for the configuration services without negative effect. in practice, preservation nodes can economically extend to 72 tb or more of raw storage. command prompt example file system information (using # to indicate command prompt) # cat /proc/mdstat … md0 : active raid1 sda1[0] sdb1[1] ... md1: active raid6 sdc1[0] sdd1[1] sde1[2] sdf1[3] … with file system mounts # cat /etc/fstab … /dev/md0      /            ext3 /dev/md1      /cache0       ext3 the os partition typically contains more than enough storage to house the os, lockss software, and the ancillary software for configuration services takeover. according to size as reported by yum info, the impact of the ancillary software on storage space in the os partition amounts to less than 750 megabytes. the promoted node at adpnet reports sufficient operational space. command prompt # df -h / filesystem     size used avail use% mounted on /dev/md0       7.5g 3.2g 4.4g 42% / the software used to provide configuration services was selected, in some part, by its ability to define data directories external to the default install paths. software installed via yum or other package managers typically are installed in the os drive. any software with more than minimal data use should redirect data directories outside of the os drive. command prompt # df -h /cache0 filesystem     size used avail use% mounted on /dev/md1       2.7t 2.0t 564g 79% /cache0 a common storage pattern in lockss may appear like: /cache0/gamma configuration services software data directories could easily be inserted into the storage pattern: /cache0/configuration data store – mysql mysql is common, small, easily configured, and powerful. the configuration file (my.cnf) is transportable and the data directory locations are definable. mysql can be installed and maintained from yum.[12] [13] data in a mysql database is easily transferrable as a static file over http via mysqldump output.[14] command prompt get the repo stub. # wget http://dev.mysql.com/get/mysql57-community-release-el7-7.noarch.rpm # yum localinstall mysql57-community-release-el7-7.noarch.rpm get mysql community server from repo # yum info mysql-community-server.x86_64 --> version 5.7.10 (current, at this time) # yum install mysql-community-server.x86_64 replication of data to peer nodes is also an option but would require mysql to be running on each node. by utilizing static file transfer and staging the data as an au, the lockss software on the distributed peer nodes can crawl, ingest and preserve the configuration data following normal routines. changing the default mysql data directory on selinux enabled systems can pose an initial challenge.[15] command prompt # yum install policycoreutils-python # getenforce --> enforcing # semanage fcontext -a -t mysql_db_t “/cache0/configuration/data(/.*)?” # restorecon -rv /cache0/configuration/data in the adpnet implementation, mysql is not accepting tcp connections and the data directory is defined to reside in /cache0. my.conf file [mysqldump] protocol = socket socket = /cache0/configuration/data/mysql/mysql.sock [mysqld] … datadir=/cache0/configuration/data/mysql socket=/cache0/configuration/data/mysql/mysql.sock log-error=/var/log/mysqld.log pid-file=/var/run/mysqld/mysqld.pid web server – apache apache http server can be installed from yum, however the repository version is not the most current version. plns are closed networks and the needed ability to bring this software up easily weighed against software freshness. current resource requirements of apache http server are significantly less than previous generations. apache supports name-based virtual hosting, proxy passthroughs, and mod_wsgi. the apache configuration files are easily transferrable. command prompt modify selinux check the contexts with # semanage fcontext -l | grep httpd_sys # semanage fcontext -a -t httpd_sys_content_t “/cache0/configuration/data/web(/.*)?” # restorecon -rv /cache0/configuration/data/web # setsebool -p httpd_can_network_connect = 1 the web server software can be made to expose the httpd and mysql configuration files for crawling over http by creating symbolic linking to the files in a web accessible directory. httpd.conf file # (# indicates comments) # alias the /cache0/configuration directory alias /configuration /cache0/configuration/data/web/staging by creating symbolic links to the production configuration files, changes to those files are immediately transparent to the distributed preservation nodes seeking to crawl that content. command prompt # cd /cache0/configuration/data/web/staging/takeover/conf apache # ln -s /etc/httpd/conf/httpd.conf httpd.conf # ln -s /etc/httpd/conf/httpd_peers.conf httpd_peers.conf # ln -s /etc/httpd/conf/configuration.adpn.conf configuration.adpn.conf # ln -s /etc/httpd/conf.d/autoindex.conf autoindex.conf mysql # ln -s /etc/my.cnf my.cnf figure 1. configuration files exposed. with apache http server, exposing sources of executable applications can be accomplished through aliases with alternative directives through each path. the result is local applications that can be executed and exposed as source files from the same file system directory. httpd.conf file # mod_wsgi installed with easy_install loadmodule wsgi_module modules/mod_wsgi-py27.so wsgiscriptalias /titlelist /cache0/configuration/data/web/staging/takeover/titles/application.py # location instead of directory, as alias to directory # via configuration should include indexes <location /titlelist> allowoverride none options none </location> alias /configuration /cache0/configuration/data/web/staging <location /configuration> allowoverride none options +indexes #indexignore . .. manifest.html thumbs.db indexignore . .. thumbs.db indexoptions ignoreclient indexoptions xhtml serversignature off </location> figure 2. title list sources exposed. processing language – python python is common, small, extensible and does not require a separate build environment. moreover, python 2.7 is already installed on the centos servers. python works well with apache http server when connected with mod_wsgi. python libraries are sufficiently available for extensibility. werkzeug works well to handle common http tasks, and suds can be added to support au status lookups via soap requests. connectors and libraries mod_wsgi[16] via easy_install (current) werkzeug[17] via easy_install (current) suds[18] via yum (current) mysql connecter/python[19] via yum (current) command line need extra libraries to build apache modules # yum install httpd-devel.x86_64 # yum install gcc # yum install python-devel.x86_64 # easy_install --dry-run mod_wsgi # easy_install mod_wsgi (version 4.4.21) # cp /usr/lib/python2.7/site-packages/mod_wsgi-4.4.21-py2.7-linux x86_64.egg/mod_wsgi/server/mod_wsgi-py2.7.so /etc/httpd/modules/ # easy_install --dry-run werkzeug # easy_install werkzeug (version 0.11.2) # yum install python-suds.noarch # yum install mysql-connector-python.x86_64 configuration data as an archival unit a critical component of creating configuration services redundancy in the network is to define the exposed configuration services data as an archival unit. in lockss, an archival unit (au) is a set of urls.[20] when the urls of the configuration services data are defined as an au, lockss regularly crawls and preserves (caches) those data in the file system. the current lockss version file system organization of the cached url is dependent on the access url.[21] modifications of the access url of an established au can result in either a uncrawlable au or a new cached url set. by utilizing dns to define a persistent name-based url that can terminate at any of the preservation nodes, the au definition and the preservation of the au data are unaffected by configuration services transfer between nodes. network topography configuration data is exposed as an archival unit and cached in each preservation node. the result is each node has the core data requisite to serve as the configuration provider. figure 4. dns identifies current node acting as the configuration services provider. in the event of a system failure, institutional withdrawal from the network, or other event that affects the ability to provide configuration services, a network peer can be promoted to the configuration server, load the current configuration data and begin offering configuration services quickly. figure 5. configuration services transferred to new node. url persistence with dns by abstracting references to name-based url routing, a number of problems related to fixed ip references can be mitigated. lockss preservation networks depend heavily on ip references for node communication. by utilizing name-based addressing for the configuration services provider the underlying ip for the name-based resolution is transferrable. dns represents the easiest route to modular addressing by updating the answer record as needed. in terms of redundancy and failsafe operations, dependency on an external dns server, potentially administered by a single network member, may introduce unacceptable vulnerability in the preservation network. an alternative is to modify /etc/hosts on each node. by updating the ip for the fully qualified domain name on each node’s /etc/hosts, the need for external dns resolution is removed. while removing external dependencies is an important consideration, the pln should also balance ease of use and maintainability. adpnet utilizes an external dns service under the administration of a network member. storage schema the title list can be provided to network peers in various ways. adpnet has implemented a database storage solution. the title list is normalized into a relational schema and reconstructed on xml streams over http. the resultant title list could either be generic for all nodes, or node-specific by utilizing the org.lockss.titledbs parameter in lockss expert config.[22] figure 6. relational schema for title list and title status. archival unit status reporting the lockss ui reports on node-specific server and au state. while this type of information is useful to system administrators maintaining the local preservation node, it does not communicate value to executive leadership. pivoting from the node-oriented to the au-oriented view of the data allows aggregation of au state across all nodes. this helps answer some questions related to the overall health of an institution’s data in the preservation network. pivoting to the au perspective also allows the publisher to identify problems related to remote node access to the content staging area. reporting preservation network and au status in a consolidated and informational way is something that needs more development. by utilizing a relational database for the title list, adpnet has begun the process of developing collection mechanisms for the specifics of au state as reported by the discrete hosts. lockss user accounts[23] and web services[24] facilitate access and collection of the remote data. the suds python library is a lightweight solution for pulling in data from web services.[25] in the extended entity relationship (eer) diagram, adpn_au_status table is the lockss web service austatus response data.[26] there is no time dimensionality to this data set. the most current state is the reported state. au status lookups across multiple nodes for thousands of archival units can take quite a long time. the response time delay and potential for failure are exacerbated when querying terabyte-sized aus with upwards of 15,000 urls.   providing an out-of-process store, although no-longer real-time, provides adequately current information for reporting. figure 7. example report of publisher aus (local and remote) consolidated. title list application the title list could be provided to the preservation peers in various ways. the most common approach in lockss networks is to make a static xml document available over http. adpnet developed a method of provisioning the title list utilizing a relational database backend that delivers node-specific xml streams over http by examining query string parameters. werkzeug is a small python library capable of handling http request parsing and response modification.[27] mod_wsgi provides a fast interface for python applications in the apache http context.[28] as title lists of several thousand aus can be large, string building optimization may be necessary. without relying on additional python libraries, python lists function reasonably well enough for large string generation.[29] protecting the data the preservation network should only allow traffic to and from known hosts. this is particularly true when configuration data is exposed over http protocols. known hosts could include: preservation network peers lockss staff local system administrators centos 7 firewall configuration could be modified to be more restrictive. command prompt # firewall-cmd --get-default-zone # drop # firewall-cmd --get-active-zones # drop … # internal … # public … # firewall-cmd --zone=public --list-all # … # sources [white-listed ips] # ports [white-listed ports] provided that the appropriate ports are open, the benefit of using a known preservation host is that these access restrictions would require no change when configuration services are transferred to a different preservation network peer. preparation of peers in order to minimize downtime and to assess the ability of any particular node to function as the configuration services provider, the ancillary configuration services software should be pre-installed. additionally, the appropriate firewall settings should be in-place, particularly opening port 80 to the known preservation network peers. the ancillary configuration services software do not need to be running when the node has not been promoted. next steps node-based configuration management transfers the rigors of network administration to the preservation network. a pln wishing to explore this approach should be well equipped to manage the preservation network. the node-based configuration management model has proven to be useful to adpnet. following the introduction of the management model in december 2015, the network reduced delays for new au releases and more than doubled the preserved au count. the model has helped redefine a sense of ownership and control of the network by the participants, and the network has had the opportunity to develop talking points and demonstrate local expertise in digital preservation. more work is needed in title list provisioning and administration, particularly in regards to development of a user-friendly administrative user interface. more work is also needed to develop concise and useful administrative and technical reports. finally, the code for administering and delivering the title list should be developed and ultimately released for public use. notes [1] what is lockss? accessed at https://www.lockss.org/about/what-is-lockss/. [2] lockss: basic concepts – lockss daemon. accessed at https://documents.clockss.org/index.php?title=lockss:_basic_concepts#lockss_daemon. [3] clockss: box operations – centos installation and configuration. accessed at https://documents.clockss.org/index.php?title=clockss:_box_operations#centos_installation_and_configuration [4] lockss-daemon. accessed at https://github.com/lockss/lockss-daemon [5] global and private lockss networks. accessed at https://www.lockss.org/community/networks/. [6] how to join. accessed at https://www.lockss.org/join/. [7] alabama digital preservation network. accessed at http://adpn.org/. [8] cataldo, tobin m. “a new approach to configuration management for private lockss networks.” d-lib magazine. 22, no. 3 / 4, (march/april 2016). http://doi.org/10.1045/march2016-cataldo [9] in lockss, an archival unit (au) is a set of urls. au definitions include a crawl start url and define a boundary crawl specification.the resultant discovered urls comprise the archival unit. more information is available at https://plnwiki.lockss.org/index.php?title=lockss_technical_manual#lockss_content_crawlers. [10] cataldo, tobin m. “data partitioning for private lockss networks.” presentation at the lockss pln community meeting, montgomery, al, october 24-25, 2013. [11] lockss technical manual – data transfer options. accessed at https://plnwiki.lockss.org/index.php?title=lockss_technical_manual#data_transfer_options. [12] installing mysql on linux using rpm packages. accessed at http://dev.mysql.com/doc/refman/5.7/en/linux-installation-rpm.html [13] installing mysql on linux using the mysql yum repository. accessed at https://dev.mysql.com/doc/mysql-repo-excerpt/5.7/en/linux-installation-yum-repo.html [14] mysqldump – a database backup program. accessed at http://dev.mysql.com/doc/refman/5.7/en/mysqldump.html [15] change the default mysql data directory on linux with selinux enabled. crashmag.net accessed at http://crashmag.net/change-the-default-mysql-data-directory-with-selinux-enabled. [16] mod_wsgi. accessed at https://modwsgi.readthedocs.io/en/develop/. [17] werkzeug. accessed at http://werkzeug.pocoo.org/. [18] suds. accessed at https://fedorahosted.org/suds/wiki/documentation [19] mysql connector/python. accessed at https://dev.mysql.com/doc/connector-python/en/. [20] lockss technical manual – lockss content crawlers. accessed at https://plnwiki.lockss.org/index.php?title=lockss_technical_manual#lockss_content_crawlers. [21] lockss software – file system deep dive. accessed at http://www.adpn.org/wiki/lockss_software#file_system_deep_dive [22] lockss configuration parameters. accessed at https://www.lockss.org/lockssdoc/gamma/daemon/paramdoc.html. [23] lockss software – user configuration. accessed at http://www.adpn.org/wiki/lockss_software#user_configuration [24] lockss technical manual – web services. accessed at https://plnwiki.lockss.org/index.php?title=web_services. [25] suds. accessed at https://fedorahosted.org/suds/wiki/documentation [26] austatus – class definition. accessed at https://lockss.org/lockssdoc/gamma/daemon/org/lockss/ws/entities/austatus.html [27] werkzeug. accessed at http://werkzeug.pocoo.org/ [28] mod_wsgi. accessed at https://modwsgi.readthedocs.io/en/develop/. [29] python – data structures. accessed at https://docs.python.org/2/tutorial/datastructures.html about the author tobin m. cataldo (tcataldo@bham.lib.al.us) is the coordinator of collection management at birmingham public library. he has been involved with the alabama digital preservation network since 2011. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – a metadata schema for geospatial resource discovery use cases mission editorial committee process and structure code4lib issue 25, 2014-07-21 a metadata schema for geospatial resource discovery use cases we introduce a metadata schema that focuses on gis discovery use cases for patrons in a research library setting. text search, faceted refinement, and spatial search and relevancy are among geoblacklight’s primary use cases for federated geospatial holdings. the schema supports a variety of gis data types and enables contextual, collection-oriented discovery applications as well as traditional portal applications. one key limitation of gis resource discovery is the general lack of normative metadata practices, which has led to a proliferation of metadata schemas and duplicate records. the iso 19115/19139 and fgdc standards specify metadata formats, but are intricate, lengthy, and not focused on discovery. moreover, they require sophisticated authoring environments and cataloging expertise. geographic metadata standards target preservation and quality measure use cases, but they do not provide for simple inter-institutional sharing of metadata for discovery use cases. to this end, our schema reuses elements from dublin core and georss to leverage their normative semantics, community best practices, open-source software implementations, and extensive examples already deployed in discovery contexts such as web search and mapping. finally, we discuss a solr implementation of the schema using a “geo” extension to mods. by darren hardy and kim durante introduction geoblacklight, as the name implies, builds on top of blacklight [1] to provide discovery services across a federated multi-institutional repository of geospatial resources (figure 1). its modularity as a rails engine lends itself to be deployed in a variety of blacklight applications and contexts. it uses a metadata schema that enables specific geospatial discovery use cases for search, view, and curation functionality. figure 2 shows the integration of the opengeoportal [2] cloud of metadata, the geoblacklight schema, and various tools in the geohydra [3] toolkit for geospatial data management. this effort is an open collaborative project aiming to build off of the successes of the blacklight and the multi-institutional opengeoportal federated metadata sharing communities. the geoblacklight schema reuses elements from dublin core (dublin core metadata initiative 2011) and georss [4] not only to leverage their normative semantics, but also their community best practices, open-source software implementations, and extensive examples already deployed in discovery contexts such as web search and mapping. as a result, the semantic elements specific to geoblacklight are reduced to a few layer elements. geospatial data come in numerous formats and resolutions. geoblacklight makes a distinction between layers and datasets. a layer is a specific unit of data that contains a set of geospatial features, a metadata description, and a feature catalog. for example, a census layer would have the geometries of census tracts and the demographic results within each tract. a dataset is a collection of layers. this distinction is important since distributors of geospatial data sometimes provide the data on fixed media segmented into many layers, yet the metadata catalog might only cover the dataset across all layers. since the geoblacklight schema requires metadata per layer, which may not be explicitly provided by the distributor, the curation workflow may be more labor-intensive. nevertheless, users expect that the discovery use case workflow leads to a specific layer which is visualizable, downloadable, sharable, and citable. our goal is to build a full-service repository for gis data that manages gis data as durable digital library assets, delivers vector, raster, and georectified maps via a spatial data infrastructure, provides discovery across a variety of applications and contexts, and works within the hydra [5] ecosystem. in this paper, we describe how the geoblacklight schema enables discovery use cases for geospatial digital libraries and discuss how we implemented the schema in a geoblacklight prototype [6] using blacklight and apache solr. [7] figure 1. geoblacklight prototype of layer view. figure 2. geoblacklight’s modular architecture. background geoblacklight builds upon prior work on geospatial digital libraries, geoportals, geospatial metadata, and gazetteers. geolibraries a geospatial digital library, or geolibrary, provides an organization and access methods to digital holdings with some geospatial properties. the geospatial properties need not be complex; even point coordinates are useful (bidney 2010). goodchild et al. (1999) define a geolibrary as “a digital library filled with geoinformation and for which the primary search mechanism is place.” in the late-90s and 2000s, researchers conceptualized and developed geospatial digital libraries (or “geolibraries”) with various success (boxall 2003; erwin & sweetkind-singer 2009; goodchild 2004; siegel et al. 2004; smith 1996). in general, these geolibrary use cases and user interfaces, however, were not geared toward current era search and discovery use cases with natural language inputs and sophisticated relevancy ranking algorithms (hearst 2009). instead they were more like electronic catalogs with spatial search as the primary discovery use case (hill et al. 2000). they also used visualization metaphors from gis desktop applications, such a multi-layered maps and extensive interactive toolbars. moreover, these early geolibraries dealt heavily with digital content procurement problems, such as digitization and georectification of paper maps (westington & bridge 2013), particularly scaling these methods to large collections. geoportals, such as opengeoportal, esri geoportal server [8] and geonetwork, [9] are typically “one-stop” shopping applications that focus on discovery of resources within a federated catalog. they are discovery applications that rely on a spatial data infrastructure for delivery and repository services, unlike geolibraries which were more monolithic in nature. their goal is to provide a destination for domain-specific collections of gis data (goodchild et al. 2007). like geolibraries, they usually do not provide flexible faceted refinement and nagivation of search results, a feature of modern search interfaces (hearst 2009). they also use similiar user interfaces akin to gis desktop applications likely due to their underlying assumption of spatially savvy users like gis professionals, and thus limits their ability to incorporate modern search semantics and visualizations. metadata metadata that documents the entire lifecycle of a spatial dataset can be extremely granular and may require information regarding data management, data quality reports, or lineage statements that include original source citations and geo-processing activities. while these metadata are important for long-term preservation and subsequent citations, they are often superfluous to the discoverability of gis resources in a web environment. moreover, metadata authoring is a labor-intensive process, although research into automated geospatial metadata generation is promising (hatcheller et al. 2009). currently, metadata for geospatial data are created using either the content standard for digital geospatial metadata (fgdc) or the iso standards for geographic metadata. [10] while the more recent iso metadata model offers increased potential for standardizing field entries and enables wider adoption among multi-lingual applications, a large store of legacy fgdc metadata exists, documenting several decades’ worth of spatial data production and research (federal geographic data committee 2014). one of the functions of the geoblacklight schema is to reconcile core elements within an aggregation of fgdc and iso records and to distill them into a set of common search indices. fgdc or iso level metadata are considered essential to spatial data infrastructures in order to satisfy the needs of archival and preservation use cases. creation of source metadata that conforms to either spatial standard is widely recognized as a very time-consuming and labor-intensive process. to alleviate some of these complexities, stanford has implemented auto-generation for unique identifier elements, and developed batch-editing techniques that expedite the metadata creation process. additionally, we have created a set of xslt stylesheets that automatically transform the source spatial metadata into other formats for use in a variety of applications. [11] therefore, the integrity of the source spatial metadata plays a major role in the quality of discovery metadata that is auto-generated for use in the geoblacklight application. on an international level, attempts to streamline and standardize metadata creation are being addressed by the opengeoportal metadata working group, a multi-institutional partnership of data and metadata experts, who are seeking to develop a set of common community practices for creating and exchanging geospatial metadata. [12] included in these best practices are recommendations for the use of keyword and name vocabularies, outlining encoding standards for the construction of free-text fields, and methodologies for documenting and relating collection-level metadata to individual layers. this group has been successful in providing guidance for the use and interoperability of the fgdc and iso metadata standards. a major objective of this partnership is to bring about more normalized and automated methods for the exchange of spatial metadata to ease the burden on local institutional repositories. currently, there are several tools available that support the creation of metadata in accordance with either the fgdc or iso standards. many of these tools, however, are lacking in one or more fundamental features and often require human intervention in order to fully document and preserve the data. for instance, many institutions maintain licenses for the esri mapping software, making the arccatalog tool an obvious starting point for metadata creation activity. arccatalog provides support for the creation of iso 19115/19139 metadata, but it does not natively support the creation of iso 19110 feature catalog metadata. the 19110 standard describes data attributes, attribute definitions, and definition sources, metadata that are often considered crucial to using and interpreting data. as a result, stanford has developed an external xslt that builds an iso 19110 record from the arcgis formatted metadata. gazetteers finally, gazetteers are authoritative registries of place names, often with spatial footprints and feature descriptions (hill et al. 1999). in discovery use cases, users wish to search and browse by location, and gazetteers enable categorical use of place names relative to geospatial location. increasingly gazetteer services, such as the open-source, community-editable geonames, [13] provide linked data interfaces into the place name registry. these interfaces enable new types of discovery uses, such as hierarchical spatial browsing, similarity search, and ontological queries (kessler et al. 2010). for digital libraries, freeland et al. (2008) geocoded the library of congress subject headings (lcsh), but little research has been done to use linked data to integrate the lcsh controlled vocabulary with an authoritative gazetteer. in our prototype, we’ve created a mapping for hundreds of placenames to both geonames and lcsh. discovery use cases in geoblacklight, we focus on discovery-oriented use cases, which are divided into search, view, and curation. the search use cases include three modalities of text search, faceted refinement, and spatial search, and two actors: a casual user and a portal user. on the curation side, there are two actors: a curator and an administrator. each schema design element, discussed in the next section, enables one or more of these use cases. search search users either use web search engines or geoportal search directly (figure 3). this is a very important design point as web search is so heavily ingrained into users’ workflows for discovery of resources in general (hearst 2009). thus, the search use cases include both web search and portal-specific search features of text search, faceted refinement, and spatial search. the result sets are brief and return quickly, and are cleanly organized with title, snippet, institution, and data type. furthermore, the result sets contain layers from many different institutions as part of a federated multi-institution repository. when the result set layers are clicked, they have a landing page of the layer view use case (figure 1). figure 3. search and view use cases. use case: web search casual users enter in natural language text into a search box of a web search engine. the result sets include layers from the geoportal as well as other relevant results from elsewhere on the web. [14] the result set should show the appropriate title and snippet for the layers. optionally, the spatial footprint should show on search engines that support spatialized web pages. use case: geoportal text search portal users enter natural language text into a search box of the geoportal. this interaction should mimic web search interfaces as users have become accustomed to these simple but powerful interfaces (hearst 2009). use case: geoportal faceted refinement portal users click on categorical facets to narrow their result sets. the facets adjust dynamically to show how many results belong to each facet, and multiple facets can be engaged simultaneously. use case: geoportal spatial search portal users zoom and pan a “slippy” web map to narrow their result sets to those results that intersect the current map view. the result set list is ranked by a spatial relevancy algorithm that favors layers with footprints entirely within the current map view. the result list dynamically adjusts as the portal user zooms and pans the slippy map. view both casual and portal users see a view landing page on clickthrough from the result sets. these view pages are shareable, persistent, and bookmarkable via a unique identifier which is important for clear data citation (michener et al. 2011). they satisfy the following use cases: use case: view dataset users view a dataset, which is a collection of individual layers that are related. the dataset has its own descriptive metadata, and a result set with all the matching layers within the dataset. use case: view metadata users view the descriptive metadata for either the dataset or an individual layer. the metadata displays with links to appropriate assets such as thumbnail previews or related layers. use case: visualize layer users see a data visualization of the layer within a web map, if available. one issue that affects data visualization with a federated spatial data infrastructure is the latency and reliability of remote data access.[15] use case: download layer users can download the layer data in multiple formats such as geotiff or shapefile or kml, and then import the data into their gis desktop application, such as esri’s arcgis or google earth. curation the curation use cases cover how the geoportal application builds and manages its federated multi-institution repository of holdings (figure 4). these uses cases have two actors: geoportal administrators, and curators (librarians). administrators ingest remote holdings from many partnered institutions on a regular basis, and they also export their repository’s holdings for other geoportals and web crawlers to ingest. curators build collections of layers through multiple source channels, such as faculty/student self-deposit, purchasing, and archives public data. figure 4. curation use cases. use case: export holdings the administrator exports layers from the repository and makes them available to partner institutions to download and ingest into their local geoportal repository. they also export layers to web search engines to enable the web search use case. use case: ingest holdings the administrator downloads layers from geoportals at partner institutions, and then ingests them into the repository for discovery. use case: monitoring operations the administrator monitors the operations of the geoportal application and repository to provide a specific level of quality of service. use case: curate holdings the curator builds collections of layers through multiple source channels, and provides the appropriate metadata for discovery and layer data for visualization. use case: purchasing policies the curator sets purchasing policies such that newly purchased layers or datasets adhere to metadata standards sufficient for the discovery use cases. use case: archiving public layers the curator selects and downloads publicly available layer data, such as government holdings for census data. then, the curator archives the public layers in a repository. use case: faculty/student self-deposit the curator facilitates self-deposit of layers from faculty and student researchers, and archives these layers in a repository for preservation. schema data model the geoblacklight schema focuses on the discovery use cases described above. text search, faceted refinement, and spatial search and relevancy are among the primary features that the schema enables. the schema design supports a variety of discovery applications and gis data types. we especially wanted to support contextual collection-oriented discovery applications as well as traditional portal applications. to generate its spatial metadata, stanford relies on the use of institutional iso 19139 xml metadata templates to standardize inputs and reduce the number of fields that require manual entry. after the template has been applied and local edits are made to an item’s metadata, python scripts are run on each record to assign unique identifiers for the data layer, the metadata file, related collection metadata, and feature catalog identifiers (if applicable). post-scripting, an xslt processes all records and generates any metadata that is redundant within individual records, such as access urls or contact details. in some cases, xslts are also used as a final step to rapidly perform batch edits on large collections of metadata before they are ingested into the repository. to achieve a consistent set of metadata elements, the source geographic metadata (iso or fgdc) are first transformed into the metadata object description schema (mods) using an xslt stylesheet [16] (figure 5). the resulting mods is a product of several conversion routines that attempt to normalize the metadata using hardcoded values for genres and resource types, conditional parameters for assigning formats and geometry types, and strict rules for capturing dates or ranges of dates. the mods schema also supports the use of attributes, including xlink for linking to related concepts or collections, and authority references for topical and geographic subject thesauri (mods editorial committee 2013). the automatic generation of a detailed mods record also meets the objectives of stanford’s overall digital library infrastructure by providing metadata for spatial data in a commonly used standard. this allows for gis objects to be incorporated and searched amongst a comprehensive collection of digital library assets. figure 5. metadata transformation workflow. each layer has a unique identifier stored in the uuid field, noting that each institution may have its own scheme for minting unique identifiers. dublin core metadata initiative (2011) defines the semantic descriptions of all of these fields. we’re using both dc elements (dc) and dc terms (dct). bold fields are required, * fields are multivalued. term description example uuid unique identifier http://purl.stanford.edu/vr593vj7147   http://ark.cdlib.org/ark:/28722/bk0012h535q urn:geodata.tufts.edu:tufts.cambridgegrid100_04 dc:identifier unique identifier often the same as uuid but may be an alternate identifier dc:title title digital map of village boundaries of andhra pradesh, india, 2001 dc:description description village boundaries of andhra pradesh linked to census 2001. includes data for 26041 villages, 237 towns, 23 districts and 1 state. this layer is part of the village map of india which includes socio-demographic and economic census data for 2001 at the village level. dc:rights rights for access restricted, public dct:provenance source institution berkeley, harvard, massgis, mit, stanford, tufts dct:references urls to related services see example 1. uses json-ld syntax (http://www.w3.org/tr/json-ld/) with role values based in part on catinterop (https://github.com/osgeo/cat-interop) link properties. dc:creator* author(s) george washington dc:format file format of layer data geotiff, shapefile dc:language language english dc:publisher publisher ml infomap dc:relation* urls to related resources http://sws.geonames.org/1252881/about.rdf dc:subject* subject census, human settlements dc:type resource type dataset, image, physicalobject dct:spatial* spatial coverage and place names paris, france dct:temporal* years 2010 dct:issued date issued 2/18/2008 dct:ispartof* holding dataset village maps of india for the dublin core elements, we use dc:identifier to implement a unique identifier across a federated multi-institution aggregation of records. we’ve added additional semantics for dct:spatial coverage to enable linked data for place names in gazetteers by reconciling text strings and unique identifiers between the geonames webservice and the library of congress name and subject authorities. dct:temporal provides for temporal faceting by date or a range of dates. similarly, we’ve added additional semantics to dct:references, reusing a controlled vocabulary from the foss4g catinterop project, [17] which relates content specific to gis resources, such as web mapping services, and to the original metadata from which the discovery metadata were derived. we use dc:format with a set of controlled values that are useful for faceting and categorizing existing spatial data types. finally, we use dct:ispartof to relate the resource to a host collection of layers. the schema has a small number of layer-specific elements such as layer.slug that enables permalinks, ensuring all gis resources are directly addressable via a shareable url. we use georss for geospatial features where only a bounding box element is required. note that georss data are in wgs84 (epsg:4326 projection) and use latitude-longitude (y,x) pairs. term description example georss:box bounding box as maximum values for s w n e 12.6 -119.4 19.9 84.8 georss:point point representation for layer as y, x — i.e., centroid 12.6 -119.4 georss:polygon shape of the layer as a polygon in the form s w n w n e s e s w 12.6 -119.4 19.9 -119.4 19.9 84.8 12.6 84.8 12.6 -119.4 the following elements for the discovery application are all layer-specific and all required. these elements have normative semantics specific to the geoblacklight application. term description example layer:id the complete identifier for the wms/wfs/wcs layer druid:vr593vj7147 layer:geom_type geometry type for layer data. point, line, polygon, and raster layer:modified last modification date for the metadata record 2014-04-30t13:48:51z layer:slug unique identifier visible to the user, used for permalinks stanford-vr593vj7147 gazetter the concept of location is fundamental to searching and evaluating spatial data. metadata in the form of place names helps to contextualize data within a set of coordinates and also provides a method of collocating similar resources based on geographic extent. current best practices in metadata creation recommend using an existing vocabulary for the identification and assignment of place keywords (mods editorial committee 2013). for its iso metadata records, stanford uses the geonames ontology for keyword assignment. geonames uniquely identifies geographic entities and provides uris to concepts and feature documents represented as rdf/xml. the conceptual uris are embedded into the iso place keywords xml through the use of the xlink attribute. although geonames identifiers and corresponding uris are unique, it is often the case that the text strings used to express a place name are not universally unique given the recurrence of place names worldwide. therefore, it is necessary to construct place keywords using one level of hierarchy (most often the country name), or by addition of an administrative designation (such as ‘district’ or ‘state’), in order to disambiguate between similar entries in the geonames database. example: patiāla (india) – for the city in india patiāla (india : district) – for the district in india, which also contains the city the addition of qualifiers to place names enhances accuracy in the search and retrieval capabilities of the geonames database, and also provides a human-readable and unique text string for use in metadata records. to manage the list of qualified place names, stanford has developed an open-source gazetteer that establishes links between the geonames ontology and the library of congress name (lcnaf) and subject (lcsh) vocabularies. [18] it is currently created through manual effort. freeland et al. (2008) implement a similar scheme but use the google maps geocoder as an authoritative gazetteer. when parallels exist between the geonames and lc ontologies, the gazetteer maps the text string and the unique identifier of the geonames entry to the corresponding lc text string and identifier. the qualified version of the place name is also added to the gazetteer record making it available for selection and reuse within other metadata. qualified name geonames term geonames id lc term lc identifier patiāla (india) patiāla 1260107 patiāla (india) n88147583 as a final step, the library of congress’ uri is added as a link inside the geonames record, permitting users to search for an entry in the geonames database using its analogous library of congress identifier. the creation of a linked vocabulary gazetteer allows for the dynamic exchange of place keywords by establishing equivalence among concepts from different ontologies. since use of the lcnaf and lcsh vocabularies is customary in digital library metadata, the geonames data are replaced with their lc equivalents when iso records are transformed into mods. implementation our geoblacklight prototype (figures 1 & 2) uses the blacklight 5 and solr 4.7 releases with encodings in both json and xml, and metadata standard support for iso 19115/19139, fgdc, mods, and the opengeoportal solr schema (figure 5). the geoblacklight solr implementation is available online at https://github.com/sul-dlss/geoblacklight-schema. in part, we generate our schema metadata from a mods record with “geo” extension elements. the mods specification is vague in how coordinates should be formatted, and in practice there doesn’t seem to be a standard format. the mods editorial committee (2013) writes: one or more statements may be supplied. if one is supplied, it is a point (i.e., a single location); if two, it is a line; if more than two, it is an n-sided polygon where n=number of coordinates assigned. no three points should be co-linear, and coordinates should be supplied in polygon-traversal order. we have some internal mods guidelines at stanford, but we do not have a normative recommendation for coordinates other than using marc 034 and marc 255, both of which have multiple representations. thus we developed a mods extension with the following goals: include humanand machine-readable encodings for geospatial coordinates for mods without requiring a detailed geometries (e.g., paper maps vs. gis data) clean, simple mods display logic pass xml validation for mods 3.4+ schema support point and bounding box coordinates for geospatial indexing, in multiple projections compatible with marc034 [19] and marc255 [20] formats. for example, in marc034 coordinates are encoded as “$de079.533265 $ee086.216635 $fs012.583377 $gs020.419532”, and in marc255 they are encoded as “(w 151°28’46”–w 78°5’6″/n 69°25’57”–n 26°4’18”)”. the human-readable data are in mods’ subject/cartographics/coordinates element, but the machine-readable data are in a mods geo extension element which embeds existing geospatial standards. using the geography markup language (gml) [21] and rdf we can support arbitrary projections for bounding boxes, and using dublin core we can define spatial facets like the format (e.g., a shapefile) and type (e.g., a dataset with point data), and zero or more associated place names (figure 6 and example 2). figure 6. xml schema for mods “geo” extension. this is an example of a mods “geo” extension: <mods> ... <titleinfo> <title>oil and gas fields in the united states, 2011</title> </titleinfo> ... <subject> <cartographics> <scale>scale not given.</scale> <projection>north american datum 1983 (nad83)</projection> <coordinates>(w 151°28?46?--w 78°5'6"/n 69°25'57"--n 26°4'18")</coordinates> </cartographics> </subject> ... <!-rdf encoding for xpath of /mods/extension[@displaylabel='geo'] --> <extension displaylabel="geo" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"> <rdf:rdf xmlns:gml="http://www.opengis.net/gml/3.2/" xmlns:dc="http://purl.org/dc/elements/1.1/"> <rdf:description rdf:about="http://purl.stanford.edu/cs838pw3418"> <dc:format>application/x-esri-shapefile</dc:format> <dc:type>dataset#point</dc:type> <gml:boundedby> <gml:envelope gml:srsname="epsg:4269"> <gml:lowercorner>-151.479444 26.071745</gml:lowercorner> <gml:uppercorner>-78.085007 69.4325</gml:uppercorner> </gml:envelope> </gml:boundedby> <dc:coverage rdf:resource="http://sws.geonames.org/5332921/about.rdf" dc:language="eng" dc:title="california"/> <dc:coverage rdf:resource="http://sws.geonames.org/6252001/about.rdf" dc:language="eng" dc:title="united states"/> </rdf:description> </rdf:rdf> </extension> </mods> using gml, the bounding box can optionally include a valid time period: <gml:boundedby> <gml:envelopewithtimeperiod gml:srsname="epsg:4269"> <gml:lowercorner>-151.479444 26.071745</gml:lowercorner> <gml:uppercorner>-78.085007 69.4325</gml:uppercorner> <gml:beginposition>2008</gml:beginposition> <gml:endposition>2008</gml:endposition> </gml:envelopewithtimeperiod> </gml:boundedby> for the solr 4 implementation, we derive solr-specific fields solely from other schema properties described in the previous section. field derived from description example solr_pt georss:point point in y,x 12.62,84.76 solr_bbox georss:box bounding box as maximum values for w s e n. 76.76 12.62 84.76 19.91 solr_geom georss:polygon shape of the layer as a envelope wkt [22] using w e n s. envelope(76.76, 84.76, 19.91, 12.62) solr_jts georss:polygon similar to solr:geom, but using jts library and polygon wkt. polygon((76.76 19.91, 84.76 19.91, 84.76 12.62, 76.76 12.62, 76.76 19.91)) solr_ne_pt georss:box north-eastern most point of the bounding box, as (y, x). 83.1,-128.5 solr_sw_pt georss:box south-western most point of the bounding box, as (y, x) 81.2,-130.1 solr_issued_dt dct:issued issued date in solr date format 2001-01-01t00:00:00z solr_year_i dct:temporal year for which layer is valid. 2012 solr_wcs_url dct:references url to wcs service http://example.com/wcs solr_wfs_url dct:references url to wfs service http://example.com/wfs solr_wms_url dct:references url to wms service http://example.com/wms the schema makes heavy use of the dynamicfield feature of solr: suffix solr data type using dynamicfield solr fieldtype class s string solr.strfield sm string, multivalued solr.strfield t text, english solr.textfield i integer solr.trieintfield dt date time solr.triedatefield url url as a non-indexed string solr.strfield bbox spatial bounding box, rectangle as (w, s, e, n) solr.spatialrecursiveprefixtreefieldtype pt spatial point as (y,x) solr.latlontype geom spatial shape as wkt spatial4j version of solr.spatialrecursiveprefixtreefieldtype jts spatial shape as wkt jts [23] version of solr.spatialrecursiveprefixtreefieldtype this is an example of the geoblacklight schema implementation [24] for the spatial types. example 3 provides an example document that follows this schema: <schema name="geoblacklight" version="1.5"> <uniquekey>uuid</uniquekey> <fields> ... <dynamicfield name="*_d" type="double" stored="true" indexed="true"/> ... <dynamicfield name="*_pt" type="location" stored="true" indexed="true"/> <dynamicfield name="*_bbox" type="location_rpt" stored="true" indexed="true"/> <dynamicfield name="*_geom" type="location_rpt" stored="true" indexed="true"/> <dynamicfield name="*_jts" type="location_jts" stored="true" indexed="true"/> </fields> <types> ... <fieldtype name="double" class="solr.triedoublefield" precisionstep="8" positionincrementgap="0"/> ... <fieldtype name="location" class="solr.latlontype" subfieldsuffix="_d"/> <fieldtype name="location_rpt" class="solr.spatialrecursiveprefixtreefieldtype" disterrpct="0.025" maxdisterr="0.000009" units="degrees" /> <fieldtype name="location_jts" class="solr.spatialrecursiveprefixtreefieldtype" spatialcontextfactory="com.spatial4j.core.context.jts.jtsspatialcontextfactory" disterrpct="0.025" maxdisterr="0.000009" units="degrees" /> </types> </schema> the primary query used by geoblacklight for spatial search uses polygon containment for spatial relevancy. for example, we issue a solr query (q) that scores containment by a factor of 10 and uses a text query, then we filter the results (fq) via an intersection spatial predicate: q=solr_bbox:"iswithin(-160 20 -150 30)"^10 railroads fq=solr_bbox:"intersects(-160 20 -150 30)" the default scoring formula in solrconfig.xml [25] where ^n is the relevancy ranking for the term: text^1 dc_description_ti^2 dc_creator_tmi^3 dc_publisher_ti^3 dct_ispartof_tmi^4 dc_subject_tmi^5 dct_spatial_tmi^5 dct_temporal_tmi^5 dc_title_ti^6 dc_rights_ti^7 dct_provenance_ti^8 layer_geom_type_ti^9 layer_slug_ti^10 dc_identifier_ti^10 these fields are all categorical and thus available as facets: dct_ispartof_sm dct_provenance_s dct_spatial_sm dc_format_s dc_language_s dc_publisher_s dc_rights_s dc_subject_sm layer_geom_type_s solr_year_i conclusion we describe a metadata schema that enables sophisticated discovery within a geospatial digital library. the schema is designed to support a federated multi-institution repository and facilitates inter-institutional sharing by reusing normative semantics from well-established schemas, dublin core and georss. to reduce duplicate effort we’re also discussing ways to copy catalog for geospatial metadata across partner institutions. our goal is not only to enable discovery use cases for geospatial resources but also to sizably increase the available resources in the service through easier sharing. we aim to treat gis assets as durable digital library objects with discovery, delivery, and preservation services. acknowledgements we wish to thank jack reed and bess sadler for their expertise, work on geoblacklight, and comments on earlier drafts. we also thank amy hodge and julie sweetkind-singer for their input and discussions about discovery and curation of geospatial resources. we also thank anonymous reviewers for their thoughtful comments. finally, we wish to thank the geoblacklight, opengeoportal, and blacklight open-source communities. notes [1] http://projectblacklight.org [2] http://opengeoportal.org [3] http://github.com/sul-dlss/geohydra [4] http://georss.org [5] http://projecthydra.org [6] http://github.com/sul-dlss/geoblacklight [7] http://lucene.apache.org/solr/ [8] http://github.com/esri/geoportal-server [9] http://geonetwork-opensource.org [10] a series of standards including iso 19115the base standard for geographic metadata description, iso 19139the xml schema implementation, iso 19110the methodology for feature cataloging, and iso 19119for describing web services. [11] https://github.com/sul-dlss/geoblacklight-schema/tree/master/lib/xslt [12] http://opengeoportal.org/working-groups/metadata/ [13] http://geonames.org [14] deridder (2008) provides an example web search integration with a digital library. [15] https://github.com/sul-dlss/geomonitor [16] https://github.com/sul-dlss/geoblacklight-schema/blob/master/lib/xslt/iso2mods.xsl [17] https://github.com/osgeo/cat-interop [18] https://github.com/sul-dlss/geohydra/blob/master/lib/geohydra/gazetteer.csv [19] http://www.loc.gov/marc/bibliographic/concise/bd034.html [20] http://www.loc.gov/marc/bibliographic/bd255.html [21] http://www.opengeospatial.org/standards/gml [22] http://spatial4j.github.io/spatial4j/apidocs/com/spatial4j/core/io/wktshapeparser.html. requires spatial4j version 0.4, and note the inverted ordering of s and n. [23] the query performance of the spatial4j implementation is better than the jts implementation, in general, due to the more simple spatial predicates in spatial4j. [24] https://github.com/sul-dlss/geoblacklight-schema/blob/master/conf/schema.xml [25] https://github.com/sul-dlss/geoblacklight-schema/blob/master/conf/solrconfig.xml references batcheller jk, gittings bm, dunfey ri. 2009. a method for automating geospatial dataset metadata. future internet 1(1):28-46. bidney m. 2010. can geographic coordinates in the catalog record be useful? journal of map & geography libraries 6(2):140-150. boxall j. 2003. geolibraries: geographers, librarians and spatial collaboration. the canadian geographer 47(1):18–27. deridder jl. 2008. googlizing a digital library. code4lib (2) [internet]. available from http://journal.code4lib.org/articles/43. dublin core metadata initiative. 2011. dublin core user guide. available from http://wiki.dublincore.org/index.php/user_guide. erwin t, sweetkind-singer j. 2009. the national geospatial digital archive: a collaborative project to archive geospatial data. journal of map & geography libraries 6(1):6-25. federal geographic data committee. 2014. geospatial metadata standards. available from: http://www.fgdc.gov/metadata/geospatial-metadata-standards#shouldouragencyuseiso. freeland c, kalfatovic m, paige j, crozier m. 2008. geocoding lcsh in the biodiversity heritage library. code4lib (2) [internet]. available from http://journal.code4lib.org/articles/52. goodchild mf, adler ps, buttenfield bp, kahn re, krygiel aj, onsrud hj. 1999. distributed geolibraries: spatial information resources. washington dc: national academy press. goodchild mf, fu p, rich p. 2007. sharing geographic information: an assessment of the geospatial one-stop. annals of the association of american geographers 97(2):250–266. goodchild mf. 2004. the alexandria digital library project: review, assessment, and prospects. d-lib 10(5) [internet]. hearst ma. 2009. search user interfaces. cambridge: cambridge university press. hill ll, frew j, zheng q. 1999. geographic names: the implementation of a gazetteer in a georeferenced digital library. d-lib 5(1) [internet]. available from http://www.dlib.org/dlib/january99/hill/01hill.html. hill ll, carver l, larsgaard m, dolin r, smith tr, frew j, rae ma. 2000. alexandria digital library: user evaluation studies and system design. journal of the american society for information science 51(3):246–259. kessler c, janowicz k, bishr m. 2009. an agenda for the next generation gazetteer: geographic information contribution and retrieval. seatle, wa: proceedings of the 17th acm sigspatial international conference on advances in geographic information systems. michener w, vieglais d, vision t, kunze j, cruse p, janée g. 2011. dataone: data observation network for earth: preserving data and enabling innovation in the biological and environmental sciences. d-lib 17(1-2). mods editorial committee. 2013. mods user guidelines (version 3). washington dc: library of congress. available from http://www.loc.gov/standards/mods/userguide/index.html. siegel d, burns b, strawn t. 2004. hgl: a web-enabled geospatial digital library. san diego, ca: proceedings esri international user conference. available from http://gis.esri.com/library/userconf/proc04/docs/pap1024.pdf. smith tr. 1996. a digital library for geographically referenced materials. ieee computer 29(5):54-60. westington ma, bridge k. 2013. the value of a bounding box: moving historical charts beyond the image browser. journal of map & geography libraries 9(1-2):108-127. examples example 1: dct:references with service urls dct:references = { "http://www.opengis.net/def/servicetype/ogc/wms":"http://geowebservices-restricted.stanford.edu/geoserver/wms", "http://www.opengis.net/def/servicetype/ogc/wfs":"http://geowebservices-restricted.stanford.edu/geoserver/wfs", "http://www.opengis.net/def/servicetype/ogc/wcs":"http://geowebservices-restricted.stanford.edu/geoserver/wcs", "http://www.isotc211.org/schemas/2005/gmd/":"http://purl.stanford.edu/zy658cr1728.iso19139", "http://www.loc.gov/mods/v3":"http://purl.stanford.edu/zy658cr1728.mods", "http://www.opengis.net/cat/csw/csdgm":"http://purl.stanford.edu/zy658cr1728.fgdc", "http://schema.org/url":"http://purl.stanford.edu/zy658cr1728", "http://schema.org/thumbnailurl":"https://stacks.stanford.edu/file/druid:zy658cr1728/preview.jpg", "http://schema.org/downloadaction":"https://stacks.stanford.edu/file/druid:zy658cr1728/data.zip", "http://library.stanford.edu/iiif/image-api/1.1/context.json":"http://purl.stanford.edu/zy658cr1728.iiif" } example 2: mods “geo” extension <extension displaylabel="geo" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"> <rdf:rdf xmlns:gml="http://www.opengis.net/gml/3.2/" xmlns:dc="http://purl.org/dc/elements/1.1/"> <rdf:description rdf:about="http://purl.stanford.edu/dg850pt1796"> <dc:format>application/x-esri-shapefile</dc:format> <dc:type>dataset#polygon</dc:type> <gml:boundedby> <gml:envelope gml:srsname="epsg:4326"> <gml:lowercorner>68.110092 6.755698</gml:lowercorner> <gml:uppercorner>97.409103 37.050301</gml:uppercorner> </gml:envelope> </gml:boundedby> <dc:coverage rdf:resource="http://sws.geonames.org/1252881/about.rdf" dc:language="eng" dc:title="west bengal"/> <dc:coverage rdf:resource="http://sws.geonames.org/1253626/about.rdf" dc:language="eng" dc:title="uttar pradesh"/> <dc:coverage rdf:resource="http://sws.geonames.org/1254169/about.rdf" dc:language="eng" dc:title="tripura"/> <dc:coverage rdf:resource="http://sws.geonames.org/1256312/about.rdf" dc:language="eng" dc:title="sikkim"/> <dc:coverage rdf:resource="http://sws.geonames.org/1258899/about.rdf" dc:language="eng" dc:title="rajasthan"/> <dc:coverage rdf:resource="http://sws.geonames.org/1259223/about.rdf" dc:language="eng" dc:title="punjab"/> <dc:coverage rdf:resource="http://sws.geonames.org/1260108/about.rdf" dc:language="eng" dc:title="patiala"/> <dc:coverage rdf:resource="http://sws.geonames.org/1263706/about.rdf" dc:language="eng" dc:title="manipur"/> <dc:coverage rdf:resource="http://sws.geonames.org/1264542/about.rdf" dc:language="eng" dc:title="madhya pradesh"/> <dc:coverage rdf:resource="http://sws.geonames.org/1268731/about.rdf" dc:language="eng" dc:title="kutch"/> <dc:coverage rdf:resource="http://sws.geonames.org/1269320/about.rdf" dc:language="eng" dc:title="jammu and kashmir"/> <dc:coverage rdf:resource="http://sws.geonames.org/1269750/about.rdf" dc:language="eng" dc:title="india"/> <dc:coverage rdf:resource="http://sws.geonames.org/1270101/about.rdf" dc:language="eng" dc:title="himachal pradesh"/> <dc:coverage rdf:resource="http://sws.geonames.org/1273293/about.rdf" dc:language="eng" dc:title="delhi"/> <dc:coverage rdf:resource="http://sws.geonames.org/1275339/about.rdf" dc:language="eng" dc:title="bombay"/> <dc:coverage rdf:resource="http://sws.geonames.org/1275638/about.rdf" dc:language="eng" dc:title="bilaspur"/> <dc:coverage rdf:resource="http://sws.geonames.org/1275715/about.rdf" dc:language="eng" dc:title="bihar"/> <dc:coverage rdf:resource="http://sws.geonames.org/1278253/about.rdf" dc:language="eng" dc:title="assam"/> <dc:coverage rdf:resource="http://sws.geonames.org/1279160/about.rdf" dc:language="eng" dc:title="ajmer"/> </rdf:description> </rdf:rdf> </extension> example 3: layer metadata in geoblacklight schema <doc> <str name="uuid">http://purl.stanford.edu/zy658cr1728</str> <str name="dc_identifier_s">http://purl.stanford.edu/zy658cr1728</str> <arr name="dct_ispartof_sm"> <str>village map of india</str> </arr> <str name="dc_title_s">andaman and nicobar, india: village socio-demographic and economic census data, 2001</str> <str name="dc_format_s">shapefile</str> <str name="dc_language_s">english</str> <str name="dc_rights_s">restricted</str> <str name="dct_provenance_s">stanford</str> <str name="dc_type_s">dataset</str> <str name="layer_id_s">druid:zy658cr1728</str> <str name="layer_slug_s">stanford-zy658cr1728</str> <date name="layer_modified_dt">2014-04-17t23:02:53.672z</date> <str name="dct_references_s">{ "http://schema.org/url":"http://purl.stanford.edu/zy658cr1728", "http://schema.org/thumbnailurl":"http://purl.stanford.edu/zy658cr1728.jpg", "http://www.loc.gov/mods/v3":"http://purl.stanford.edu/zy658cr1728.mods", "http://www.isotc211.org/schemas/2005/gmd/":"http://purl.stanford.edu/zy658cr1728.iso19139", "http://www.opengis.net/def/servicetype/ogc/wms":"http://geowebservices-restricted.stanford.edu/geoserver/wms", "http://www.opengis.net/def/servicetype/ogc/wfs":"http://geowebservices-restricted.stanford.edu/geoserver/wfs", "http://www.opengis.net/def/servicetype/ogc/wcs":"http://geowebservices-restricted.stanford.edu/geoserver/wcs" }</str> <str name="dct_issued_s">2013</str> <arr name="dct_temporal_sm"> <str>2001</str> </arr> <int name="solr_year_i">2001</int> <str name="layer_geom_type_s">point</str> <str name="dc_publisher_s">ml infomap (firm)</str> <arr name="dc_creator_sm"> <str>ml infomap (firm)</str> </arr> <str name="dc_description_s">this point dataset shows village locations with socio-demographic and economic census data for 2001 for the union territory of andaman and nicobar islands, india linked to the 2001 census. includes village socio-demographic and economic census attribute data such as total population, population by sex, household, literacy and illiteracy rates, and employment by industry. this layer is part of the villagemap dataset which includes socio-demographic and economic census data for 2001 at the village level for all the states of india. this data layer is sourced from secondary government sources, chiefly survey of india, census of india, election commission, etc. this map includes data for 547 villages, 3 towns, 2 districts, and 1 union territory.this dataset is intended for researchers, students, and policy makers for reference and mapping purposes, and may be used for village level demographic analysis within basic applications to support graphical overlays and analysis with other spatial data.</str> <arr name="dc_subject_sm"> <str>human settlements</str> <str>villages</str> <str>census</str> <str>demography</str> <str>population</str> <str>sex ratio</str> <str>housing</str> <str>labor supply</str> <str>caste</str> <str>literacy</str> <str>society</str> <str>location</str> </arr> <arr name="dct_spatial_sm"> <str>andaman and nicobar islands</str> <str>andaman</str> <str>nicobar</str> <str>car nicobar island</str> <str>port blair</str> <str>indira point</str> <str>diglipur</str> <str>nancowry island</str> <str>andaman and nicobar islands (india)</str> <str>andaman islands (india)</str> <str>nicobar islands (india)</str> <str>car nicobar island (india)</str> <str>port blair (india)</str> <str>diglipur (india)</str> </arr> <str name="georss_box_s">6.761581 92.234924 13.637013 94.262535</str> <str name="georss_polygon_s">6.761581 92.234924 13.637013 92.234924 13.637013 94.262535 6.761581 94.262535 6.761581 92.234924</str> <arr name="dc_relation_sm"> <str>http://sws.geonames.org/1278647/about.rdf</str> <str>http://sws.geonames.org/1278648/about.rdf</str> <str>http://sws.geonames.org/1261448/about.rdf</str> <str>http://sws.geonames.org/1274969/about.rdf</str> <str>http://sws.geonames.org/1259385/about.rdf</str> <str>http://sws.geonames.org/1259116/about.rdf</str> <str>http://sws.geonames.org/1272607/about.rdf</str> <str>http://sws.geonames.org/1261993/about.rdf</str> </arr> <str name="solr_geom">envelope(92.234924, 94.262535, 13.637013, 6.761581)</str> <str name="solr_bbox">92.234924 6.761581 94.262535 13.637013</str> <double name="solr_sw_pt_0_d">6.761581</double> <double name="solr_sw_pt_1_d">92.234924</double> <str name="solr_sw_pt">6.761581,92.234924</str> <double name="solr_ne_pt_0_d">13.637013</double> <double name="solr_ne_pt_1_d">94.262535</double> <str name="solr_ne_pt">13.637013,94.262535</str> <str name="solr_wms_url">http://geowebservices-restricted.stanford.edu/geoserver/wms</str> <str name="solr_wfs_url">http://geowebservices-restricted.stanford.edu/geoserver/wfs</str> <str name="solr_wcs_url">http://geowebservices-restricted.stanford.edu/geoserver/wcs</str> <long name="_version_">1465674129654939648</long> <date name="timestamp">2014-04-17t23:02:53.672z</date> <float name="score">1.0</float> </doc> about the authors darren hardy is a gis software engineer at stanford university, where he develops open-source geospatial digital library software and services. he earned ph.d. and masters degrees in environmental science and management from the university of california, santa barbara, and b.s. and m.s. degrees in computer science from the university of colorado at boulder. kim durante is a metadata analyst for geographic and scientific data at stanford university. she holds a m.s. degree in information science from the university of wisconsin at milwaukee, and a m.a. degree in art history from georgia state university. subscribe to comments: for this article | for all articles 2 responses to "a metadata schema for geospatial resource discovery use cases" please leave a response below: stephen richard, 2014-09-11 example 3 includes fields named ‘solr_sw_pt_0_d’, ‘solr_sw_pt_1_d’, ‘solr_ne_pt_0_d’, ‘solr_ne_pt_1_d’, but these are not mentioned in the text. there should be some explanation of what these are (i shouldn’t have to guess) stephen richard, 2014-09-11 the dc_relation_sm array in example 3 contains the uris for geonames instances. this pattern is not described in the text. is it actually your convention for putting place name uris in the solr response. some clearer explanation of how dct_references_s and dc_relation_sm are being used would be a good thing. you might have a look at https://github.com/project-open-data/project-open-data.github.io/issues/291 to see some discussion on the problem of putting links for api based access in metadata. leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – indexing bibliographic database content using mariadb and sphinx search server mission editorial committee process and structure code4lib issue 25, 2014-07-21 indexing bibliographic database content using mariadb and sphinx search server fast retrieval of digital content has become mandatory for library and archive information systems. many software applications have emerged to handle the indexing of digital content, from low-level ones such apache lucene, to more restful and web-services-ready ones such apache solr and elasticsearch. solr’s popularity among library software developers makes it the “de-facto” standard software for indexing digital content. for content (full-text content or bibliographic description) already stored inside a relational dbms such as mariadb (a fork of mysql) or postgresql, sphinx search server (sphinx) is a suitable alternative. this article will cover an introduction on how to use sphinx with mariadb databases to index database content as well as some examples of sphinx api usage. by arie nugraha introduction to mariadb and sphinx mariadb is a drop-in replacement for mysql, a very popular open source database server. what makes mysql popular is its ease of use and deployment compared to other database servers such as postgresql and oracle. the official mysql reference manual states that: “mysql server was originally developed to handle large databases much faster than existing solutions and has been successfully used in highly demanding production environments for several years. although under constant development, mysql server today offers a rich and useful set of functions. its connectivity, speed, and security make mysql server highly suited for accessing databases on the internet.” in 2009, oracle acquired sun microsystems, and with it mysql ab, the company that has developed and maintained mysql from the beginning. mysql’s founder, michael monty, fearing that oracle would close the mysql codebase, created a fork of mysql called mariadb which has the same features and improved performance. integrated library systems (ilses) mostly store their metadata/bibliographic description inside database servers to enable fast storage and retrieval. one of the problems when a database server stores a large number of documents is the speed of retrieval. for document retrieval, mariadb has built-in full-text searching capability, but it comes with its own problem: it could only run on the myisam table type before mariadb version 10.0 and the myisam table type comes with its own problems, because it is easily corrupted under some circumstances (http://dev.mysql.com/doc/refman/5.7/en/corrupted-myisam-tables.html). scalability and performance are additional issues, especially when we deal with a high number of concurrent queries. sphinx search server (sphinx) is a standalone product that provides fast full-text search capability for relational databases such as mariadb, mysql, postgresql and any database that can be connected via odbc such as ms sql server. sphinx supports two query methods that integrate particularly well with mariadb and mysql. the first allows us to query an indexed mariadb or mysql instance directly and retrieve results indexed in sphinx by setting up sphinx’s mysql storage engine (sphinxse). the second lets us query the sphinx server by connecting to it via the mariadb or mysql client library (libmysql version > 4.1 at minimum) and issuing queries using the sphinxql syntax, a sphinx-specific sql dialect. note that the first method requires our indexed database to be mariadb or mysql, while the latter method only requires the client and can be used no matter what kind of database we index. some of sphinx’s features are explained on its official website (http://sphinxsearch.com/docs/2.1.8/features.html) and make it very suitable for indexing large documents: proven scalability up to billions of documents, terabytes of data, and thousands of queries per second; high search speed (up to 150-250 queries/sec per core against 1,000,000 documents, 1.2 gb of data); supports boolean, phrase, word proximity and other types of queries; supports stopwords; supports stemming (stemmers for english, russian, czech and arabic are built-in; and stemmers for french, spanish, portuguese, italian, romanian, german, dutch, swedish, norwegian, danish, finnish, hungarian, are available by building the third party libstemmer library); from this author’s experience, sphinx is easy to set up both as a standalone server and integrated with an existing mariadb database installation, and it is also very fast. this article covers basic usage of sphinx to index mariadb database content. its main focus is integrating sphinx and mariadb via sphinxse, but it also demonstrates searching using sphinxql as well as the sphinx api. for the purpose of this article, the author will set up sphinx, sphinxse (the sphinx storage engine of mariadb) and mariadb with this hardware and software configuration: operating system : centos 6.4 64 bit gnu/linux with latest update database server : mariadb 5.5.37 with sphinx plugin sphinx server : sphinx 2.1.8 processor : intel core i7 3770 memory : 8 gb drive : 30gb ssd sphinx compared to other search engine servers compared to other digital content indexing solutions such as solr or elasticsearch, which are more web services-oriented, sphinx was specially designed to integrate well with sql database servers, and to be easily accessed by scripting languages. as such, sphinx is very suitable for those who already have existing digital content inside database servers such as mariadb, postgresql, ms sql, etc. and want the content to be indexed for fast retrieval without having to convert to another format first. sphinx has very good indexing and retrieval speed; from this author’s experience sphinx can index approximately 86,200 documents in under 20 seconds. mariadb setup setting up mariadb to work with sphinx is straightforward and relatively easy. to use sphinx in mariadb, we should have mariadb version 5.2.2 at the minimum and also sphinxse engine installed in the mariadb server. the author uses the centos operating system. to install mariadb, open the terminal program and issue this command as root: # yum install mariadb-server mariadb-client # service mariadb start before executing the above command we must make sure that mariadb has been added to the yum repository, or else yum won’t install anything. more on how to setup mariadb on centos/redhat/fedora can be found at https://mariadb.com/kb/en/installing-mariadb-with-yum/. setup sphinx and the sphinxse storage engine after completing the mariadb installation process, the next step is to set up sphinx and sphinxse. one of the methods used to integrate mariadb with sphinx is to enable mariadb’s sphinxse storage engine. to enable sphinxse in mariadb, we can issue a one-time sql command: install soname 'ha_sphinx'; show engines; one advantage of using the sphinx engine in mariadb is that we don’t need to use the sphinx api to retrieve records from the sphinx server, all queries to the sphinx server are handled by mariadb using sql syntax. this means that there will be no big changes to ils programming code. download the latest stable release from http://sphinxsearch.com/downloads/release/, choosing the appropriate operating system and operating system architecture (32/64 bit). in this article, the author used the rhel/centos 6.x x86_64 rpm binary and installed it by issuing this command: # yum install postgresql-libs # rpm -uvh sphinx-2.1.8-1.rhel6.x86_64.rpm sphinx is working as a server/daemon, much like the mariadb daemon or any other daemon in gnu/linux. to start the sphinx server we then issue this command: # searchd figure 1. starting sphinx searchd server (enlarge) if we don’t see any errors, that means we are ready to go. by default, the server listens on port 9312, but we can tweak this configuration by editing the sphinx.conf file located in /etc/sphinx. to index our database content, we need to configure sphinx so it knows what kind of records/documents from our database to index. to configure sphinx indexes, we must open sphinx.conf and edit some parameters: source our_documents { type = mariadb sql_host = localhost sql_user = root sql_pass = secret sql_db = test sql_port = 3306 sql_query = select biblio_id, title, series_title, author, topic, \ notes from search_biblio sql_query_info = select * from search_biblio where biblio_id=$id } index idx_our_documents { source = our_documents path = /var/lib/sphinx/idx_our_documents stopwords = /etc/sphinx/stopwords.txt docinfo = extern charset_type = sbcs } indexer { mem_limit = 64m } searchd { listen = 9312 listen = 9306:mysql41 log = /var/log/sphinx/searchd.log query_log = /var/log/sphinx/query.log read_timeout = 5 max_children = 30 pid_file = /var/run/sphinx/searchd.pid max_matches = 10000 seamless_rotate = 1 preopen_indexes = 1 unlink_old = 1 workers = threads # for rt to work binlog_path = /var/lib/sphinx/ } in the above configuration, our main settings are in lines 1-22, and line 40; the “indexer” and “searchd” may not need to be changed at all. our custom settings simply state that we want “our_documents,” taken from our mariadb database with the query “select biblio_id, title, series_title, author, topic, notes from search_biblio from search_biblio,” to be indexed into an index named “idx_our_documents.” note that sphinx will assume that the first column (biblio_id) on the “sql_query” parameter is a document id and this document id must be an integer or bigint type. other columns specified in “sql_query” will be indexed by the sphinx indexer. to improve indexed data, we also can set a stopwords file in the “stopwords” parameter inside our index. the stopwords file is just a plain text file which contains a comma separated list of words that are excluded from the index. in this example, the stopwords file is located at /etc/sphinx/stopwords.txt. the sphinx configuration file allows more than one index to be defined. this is very useful when we wish to index the contents of two or more databases. after the configuration is completed then we must run the sphinx indexer by executing the following command: # indexer idx_our_documents this command tells the sphinx indexer to fetch records from mariadb based on an index named “idx_our_documents”. the result of this will be something like the following: figure 3. indexer result (enlarge) in this example, we have 86,213 library bibliographic records populated from springer’s marc download tool. the table structure for storing bibliographic records is taken from the open source ils senayan library management system (slims), search_biblio table. slims uses this as a redundant table for storing the bibliographic records index to speed up search. the structure of this table can be seen below: column type biblio_id int(11) title text edition varchar(50) isbn_issn varchar(20) author text topic text gmd varchar(30) publisher varchar(100) publish_place varchar(30) language varchar(20) classification varchar(40) spec_detail_info text location text publish_year varchar(20) notes text series_title text items text collection_types text call_number varchar(50) opac_hide smallint(1) promoted smallint(1) labels text collation varchar(100) image varchar(100) input_date datetime last_update datetime figure 4. bibliographic record table structure after we create the index with sphinx, we must next create a table in mariadb which acts as a proxy to the sphinx server. the table structure must be at least like this: create table idx_our_documents ( id bigint unsigned not null, weight integer not null, query varchar(3072) not null, index(query) ) engine=sphinx connection="sphinx://127.0.0.1:9312/idx_our_documents"; the first three fields are mandatory, because they will be used by sphinx server when returning search results. querying records in sphinx sphinx has historically had five document matching modes: “all,” “any,” “phrase,” “boolean,” or “extended.” however, according to the official documentation, the preferred matching mode is now “extended” (sph_match_extended), which has subsumed the others (which are deprecated). the “extended” match mode supports the following syntax: boolean syntax: operator or: information | knowledge operator not: information -retrieval information !system field match syntax: field search operator: @title information @content information retrieval field position limit modifier: @content[50] information multiple-field search operator: @(title,body) information retrieval ignore field search operator (will ignore any matches of ‘information’ from field ‘title’): @!title information all-field search operator: @* information retrieval phrase match syntax: phrase search operator: “database indexing” proximity search operator: “database indexing”~10 quorum matching operator: “using sphinx to index digital content”/3 strict order syntax strict order operator: aaa << bbb << ccc field-start and field-end modifier: ^information retrieval$ there are also other search operators not mentioned in this article, but we think that above examples are the most important operators which will be commonly used for search query. more information about other operators can be found at http://sphinxsearch.com/docs/2.1.8/extended-syntax.html. specific field match before using sphinx, to query our bibliographic records from our database we go straight to the mariadb table that holds bibliographic data like this (assuming that we have already created a full-text index on the columns we are searching): mariadb query select * from `search_biblio` where match(`title`, `notes`) against('+database +indexing +multimedia' in boolean mode) order by `title` asc; in the above query, we want a document whose title and notes fields must contain the terms “database, “indexing”, and “multimedia”. when using sphinx we query the proxy table instead with this equivalent syntax: sphinx query select * from `idx_our_documents` where `query`='@(title, notes) database indexing multimedia;mode=extended;limit=10000;sort=extended:@weight desc'; in this example, both mariadb’s full text search (fts) and sphinx return the same results, 36 unique documents retrieved. the difference is in speed, mariadb’s fts query took 4.5758 seconds to return the result, while sphinx only took 0.0019 seconds. phrase match now we will try to search using a phrase match query. the “notes” field in the “search_biblio” table holds keywords and also a document abstract. with traditional mariadb’s fts query, our query for documents which contain the phrase “information retrieval” and also contain the phrase “natural language” inside the “notes” field will be like this: mariadb query select * from `search_biblio` where match(`notes`) against('"information retrieval"' in boolean mode) and match(`notes`) against('"natural language"' in boolean mode) order by `title` asc; the query returns 64 documents in 0.1041 seconds. with sphinx the query will be like this: sphinx query select * from `idx_our_documents` where `query`='@notes "information retrieval" "natural language"; mode=extended; limit=10000; sort=extended:@weight desc'; sphinx also return 64 documents, but with faster speed, 0.0064 seconds. start end match since mariadb’s fts doesn’t support start and end match operators yet, if when we want to search for documents that start or end with some words in mariadb, we use a non-fts query (which uses an ordinary index, not a full-text one) and also the wildcard operator (the % symbol) in this query: mariadb query select * from `search_biblio` where `title` like 'information%' or `title` like '%management' order by `title`; the query searches for documents which start with “information” or end with “management”. the result returns 571 documents in 0.1045 seconds. sphinx’s equivalent query will be like this: sphinx query select * from `idx_our_documents` where `query`='@title ^information | @title management$; mode=extended; limit=10000; sort=extended:@weight desc'; the interesting thing we found here is sphinx returns more results, with 925 documents in 0.0085 seconds. sphinxql usage another method to query sphinx indexes is through the mariadb or mysql native protocol with sphinxql. with sphinxql, we don’t need to use the sphinxse storage engine and its proxy table, but rather we can query the sphinx server directly using the mysql or mariadb client library. with this method, the sphinx server acts as if it is a mysql or mariadb server, allowing any mariadb/mysql client api to connect to and query it. sphinxql is sphinx’s own subset of sql. it is similar to standard sql, but it has additional features that are not supported in standard sql. one of the advantages of using sphinxql is that the syntax supports real-time updating of sphinx indexes where the traditional api does not. to connect and query our existing index using sphinxql in php we can write code like this: <?php // sphinxql-test.php // define constant define('sphinx_host', '127.0.0.1'); define('sphinx_port', 9306); define('sphinx_index', 'idx_our_documents'); // connect to sphinx server using mysqli $dbs = @new mysqli('127.0.0.1', '', '', '', 9306); // query $query = '@title database indexing'; // query index $query = $dbs->query("select * from ".sphinx_index." where match('$query') limit 0, 10000"); // get record(s) $found = $query->num_rows; if ($found > 0) { echo "found ".$found." record(s) :\n"; while ($data = $query->fetch_object()) { echo "document id : ".$data->id."\n"; } } else { echo "sorry, no match found...\n"; } figure 5. using php to connect & query sphinx server with mariadb and sphinxql if this script is executed in terminal we will see output like this: figure 6. result of php script when executed sphinx api usage besides using sphinxse and sphinxql, sphinx’s developers also provide an api to popular programming languages such as java, ruby, python and php. these apis are already bundled inside the sphinx distribution. if we are using the sphinx api, then we connect straight to the sphinx server and not via mariadb’s sphinxse engine. to connect and query to our index in php, we need at minimum to write code like this: <?php // sphinx-test.php // define constant define('sphinx_host', 'localhost'); define('sphinx_port', 9312); define('our_index', 'idx_our_documents'); // query $query = '@title database indexing'; // main api class inclusion require "sphinxapi.php"; // create sphinx client instance $cl = new sphinxclient(); // set connection params $cl->setserver( sphinx_host, sphinx_port ); $cl->setconnecttimeout( 1 ); $cl->setarrayresult( true ); $cl->setweights( array ( 100, 1 ) ); $cl->setmatchmode( sph_match_extended2 ); $cl->setsortmode( sph_sort_extended, '@weight desc' ); $cl->setselect( 'title, author, topic, notes' ); $cl->setlimits( 0, 100, 10000 ); $cl->setrankingmode( sph_rank_proximity_bm25 ); // execute the query $res = $cl->query( $query, our_index ); // iterate the result if ( $res===false ) { // query is failed print "query failed: " . $cl->getlasterror() . ".\n"; } else { if ( $cl->getlastwarning() ) : print "warning: " . $cl->getlastwarning() . "\n\n"; endif; print "query '$query' retrieved $res[total] of $res[total_found] matches in $res[time] sec.\n"; print "query stats:\n"; if ( is_array($res["words"]) ) : foreach ( $res["words"] as $word => $info ) : print " '$word' found $info[hits] times in $info[docs] documents\n"; endforeach; endif; if (is_array($res["matches"]) ) { $n = 1; print "matches:\n"; foreach ( $res["matches"] as $docinfo ) { print "$n. doc_id=$docinfo[id], weight=$docinfo[weight]"; foreach ( $res["attrs"] as $attrname => $attrtype ) { $value = $docinfo["attrs"][$attrname]; print ", $attrname=$value"; } print "\n"; $n++; } } } figure 7. using php to connect and query the sphinx server using the php sphinx api if the above script is executed in a terminal, we will see output like this: figure 8. result of sphinx php script when executed (enlarge) conclusion sphinx is a good alternative for indexing digital content/documents already stored inside an existing dbms, especially mariadb. it is easily deployed and configured, and therefore presents a lower barrier to entry for system librarians or library/archives software developers. combined with mariadb and its sphinxse engine, sphinx becomes even easier to integrate into existing ilses that use mariadb as their backend. sphinx’s ranking feature makes it easier to rank documents based on their relevance. sphinx’s performance is also good compared to native mariadb indexing system, so it is also a good alternative if we just want to index existing mariadb database records for fast retrieval. bibliography aksyonoff, andrew. 2014. sphinx 2.1.8-release reference manual. sphinx open source search server [internet]. [cited 2014 may 17]. available from: http://sphinxsearch.com/docs/manual-2.1.8.html aksyonoff, andrew. 2011. introduction to search with sphinx: from installation to relevance tuning . sebastopol, ca. o’reilly media. daniel bartholomew et. al, 2014. enabling sphinx in mariadb. mariadb [internet]. [cited 2014 may 17]. available from: https://mariadb.com/kb/en/about-sphinx/ gilfillan i, razzoli f. 2014. fulltext index overview. mariadb [internet].[cited 2014 may 17]. available from: https://mariadb.com/kb/en/fulltext-index-overview/ monty m, bartholomew d. 2012. installing and testing sphinx with mariadb. mariadb [internet]. [cited 2014 may 17]. available from: https://mariadb.com/kb/en/installing-and-testing-sphinx-with-mariadb/ mysql [internet]. corrupted myisam tables. [cited 2014 july 15]. available from: http://dev.mysql.com/doc/refman/5.7/en/corrupted-myisam-tables.html mysql [internet]. what is mysql? [cited 2014 july 15]. available from: http://dev.mysql.com/doc/refman/5.7/en/what-is-mysql.html about the author arie nugraha currently works as a junior lecturer for the department of library and information science, faculty of humanities, university of indonesia. he lectures about library automation, databases and also digital libraries to bachelor degree library science students at the university of indonesia. he is also an active open source developer. he may be reached at dicarve@gmail.com. he blogs at http://dicarve.blogspot.com. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – an android/lamp mobile in/out board based on wi-fi fingerprinting mission editorial committee process and structure code4lib issue 15, 2011-10-31 an android/lamp mobile in/out board based on wi-fi fingerprinting library technology and other professionals with diverse skills must be able to locate each other during the workday, in order to most responsively serve their clients. while staff often carry cellular phones, contact can be especially challenging given the constant, highly mobile nature of library work, especially on larger campuses with variable cellular phone service. western michigan university (wmu) libraries has developed an android/lamp application that library staff may use on their increasingly prevalent wi-fi enabled mobile devices to “check in” at various locations where they do work, so that their colleagues may locate them as needed. the application takes advantage of wmu’s widespread wi-fi network, a set of free platform and software development tools and open standards, and methods from the computer science literature, and overcomes gps and telephony limitations. this article describes the application, which is based on wi-fi fingerprinting, and suggests how other developers could use it and new methods from the computer science literature as starting points to create their own applications. by keith kelley, karlis kaugars and scott garrison introduction and background university library systems departments often face support-intensive periods, such as when multiple campus and/or library systems change at once. changes in one area can break other functions, forcing staff who are already busy to stop and resolve user-reported emergencies. these emergencies can be unpredictable, make staff all the more difficult to find, and slow helpdesk response. the ability for individual staff to find the right person to quickly respond to emergency calls can be critical to maintaining operations and quality of service, as well as easing the strain on both the department’s capacity and the staff members themselves. how to track staff is a key issue. dry erase, magnetic and electronic in/out boards that rely on memory, predictable staff movement patterns, and/or direct user input are not well suited to the dynamic nature of desk side support and emergency calls. hardware solutions that require staff to carry additional devices to communicate via what can be an expensive hardware infrastructure may be unsuitable for staff tracking at a university such as wmu, which has a multimillion-square foot campus including multiple branch libraries and campuses. these solutions are not affordable for most libraries. an automated employee tracking system that detects where people are even when they have been unable to record that information is a better approach. many libraries such as wmu already pay for staff smartphones and it is in a library’s business interest to maximize return on smartphone investment. as of 2011, increasing numbers of students, including those who work in library systems, carry personally owned smartphones connected to campus wi-fi. though smartphones’ telephone function may not work well indoors, a smartphone can be location-aware anywhere with a wireless network. an indoor positioning system (ips) on android with a lamp backend is inexpensive, relatively easy to develop and deploy, and works within complex, unfamiliar and uncontrolled environments by making use of an existing wi-fi infrastructure with no modifications. some have developed location systems accurate to within ten meters, which may be sufficient for a mobile in/out board. it might seem logical to use a smartphone’s combination of built-in location services and social applications to meet helpdesk needs. social apps such as facebook places, foursquare and google latitude can locate smartphones, but tend to offer only public building-level accuracy. gps-based apps may be accurate to six meters, but do not work well inside most buildings. cellular provider network localization works indoors, but is only accurate to between 100 feet and 1000 meters. wi-fi signal strength, however, is primarily available indoors and can be used to estimate location within a range of a few meters. since even redpin, an open source indoor localization system requiring user input, would still require significant customization to use as a mobile in/out board, wmu decided to write its own system. the system consists of an android ips app and lamp backend that use wi-fi fingerprinting and a location determination algorithm to display where smartphone-carrying staff members are within wmu. staff and students alike use a common ips app to locate one another. those lacking permanently issued or owned android phones can use the app on library-owned used phones, in wi-fi only mode. about the system wmu libraries’ mobile in/out board system is effectively an ips that uses symbolic natural language locations such as rooms, floors, and buildings, rather than absolute, relative or physical locations based on coordinates. while originally developed as a self-positioning system, the current production system may be classified as an indirect remote positioning system. the system sends a received signal strength (rss) measurement sample gathered by a mobile device to a remote localization server that computes the device’s location based on a wi-fi fingerprint, or a map of wireless access points (aps) in a given location. wmu’s system uses wi-fi fingerprinting because it is simpler, more accurate, and more reliable than other wireless indoor positioning methods (jekabsons and zuravlyov 2010; ali 2011; navarro, peuker, quan 2010). the system is participatory, in that the users themselves define the fingerprints in the ap map by detecting them through the mobile app. the argument for wi-fi fingerprinting wi-fi localization maps signal strength to location. the closer a mobile device is to a wi-fi ap, the stronger the signal generally is, but there are many factors that influence signal strength. in a perfect world, with no interference, a clear line of sight, a vacuum, and geographic location coordinates (such as latitude and longitude) for every ap, a mobile device could use simple geometry to calculate exactly where it is. this describes one class of methods for wireless indoor positioning known as triangulation (also known as trilateration). triangulation does not work well in places like libraries, which contain books in stacks, moving people, and other signal absorbing, reflecting, and interfering artifacts. wi-fi fingerprinting is a two-stage method of wireless indoor positioning. first, a training/collection (also known as offline or calibration) stage builds a radio map, or fingerprint database of all wireless aps in a given area. second, a positioning/tracking (also known as live or online) stage measures signals and estimates a wireless device’s location by matching the measured signal to the fingerprint. though not perfectly reliable, as ap antenna signal strength is random within a certain range, wi-fi fingerprinting is the primary method of wireless indoor positioning. there are many choices to make when implementing a wi-fi fingerprinting solution. after having experimented with several available algorithms, including nearest neighbor in signal space (nnss; for a detailed explanation of how nnss works, including considerations of euclidian distance versus manhattan distance, see bahl and padmanabhan 1999, 2002, and li et al. 2005) the authors wrote a homegrown location estimation algorithm for this system.[1] given that wmu’s system uses room-level location that is not as precise as some applications require, it uses an average signal strength in a simple deterministic wi-fi fingerprinting with no filtering (honkavirta et al. 2009). building the system this system is built on the popular and open lamp and android platforms. though the same functionality may be achieved and the app distributed on an iphone, private apis would be required, which apple neither encourages nor supports. prerequisites: an ip-accessible server to use as a localization server and output method android devices (not necessarily with a service plan) carried by every participating person a campus with nearly universal wi-fi coverage software components: a lamp/mamp/wamp server a custom android app custom php scripts hardware components (optional): a wall-mounted computer monitor for shared location viewing tools required: a text editor or other tool for server side development the java development kit the android sdk the eclipse development environment mysql tables: only three tables are strictly necessary for a simple implementation. two others are optional (see figure 1). employee – the table of users for display on the in/out board location – the table of symbolic location names signal – the table of individual ap antenna address (bssid) and received signal strength (rss)[2] measurement pairs (optional) scan – though kaemarungsi and krishnamurthy (2004) indicate it should be unnecessary, wmu libraries recommends keeping the signals from one scan together, in case it is desired for future analysis, or it is necessary to delete erroneous contributed scans or a location determination algorithm chosen in the future requires it (optional) logentry – allows users to see the last several places someone visited figure 1. in/out database entity relationship diagram php files only four scripts are necessary for a simple implementation. recognize.php – given the input of a measured scan or set of scans (sent as a post request from a mobile device on a periodic basis), this script compares the observed set to the sets in the radio map, returns a symbolic location, and updates the employee table with that location name.php – the tracking/collection process associates a location name with a certain wi-fi fingerprint. the same script can be used to initially create the location and to add to the location definition if it is determined to be not well enough defined. in either case, this script is called by user interaction on the phone. board.php – a web page that displays the contents of the employee table either to a wall-mounted monitor or department members’ pcs or their own phones so they can find each other in the field selectall.php – for the “correct” button, to provide the entire list of defined locations, containing the location name by which the current observed set of signals should have been recognized how the system works training/collection stage the training/collection stage builds the radio map. a radio map is not a graphical map or floor plan, but a database of locations mapped to signal strengths. the primary method for building a radio map is recording samples of signal strengths at known locations (known as training points). wmu chose to use a participatory approach where users contribute their own fingerprints for the locations they visit, which both removes the initial training/collection burden normally associated with building a radio map and minimizes the database size to only the location definitions used. before users can use the system, they need to define at least one fingerprint (see figure 2). the mobile app has one button that assigns a name to a single scan result set, or fingerprint, and a second that adds several scan results to that named fingerprint. in locations with less overlapping wi-fi coverage, a single scan may suffice. in more central locations with greater overlap, it is better to submit multiple scans in order to build up the fingerprint definition and differentiate the location from its neighbors. if there are several close locations, all of the fingerprints nearby need to be well defined through multiple button presses. a location with a single fingerprint as its definition may yield many false positive identifications (depending on the algorithm), while the user is actually elsewhere. the training/collection stage is not really a distinct stage in a participatory approach, because the users contribute the fingerprint definitions in many small instances (even while actively trying to identify their current location). however, it is useful to think of this stage distinctly, partly because each location must be defined before it can participate in a positioning/tracking stage (this approach also helps readers of the literature better understand ips principles). it can be useful to have a lightweight distinct training/collection stage where an assigned user defines a few fingerprints at commonly visited locations, while still allowing other users to populate the rest of the fingerprint database later. in other words, the database must contain some common locations before the system can track locations. more location scans over time iteratively refine locations, for greater system reliability. figure 2. defining a fingerprint positioning/tracking stage during the positioning/tracking stage, the phone app periodically measures and sends observed wi-fi antenna rss values to a script on the lamp server, which runs a location determination algorithm such as nnss. the app does not process the data, but sends measured scan results which the localization server either uses individually or averages together, to algorithmically determine the smartphone’s location (see figure 3). once the server determines location, the script checks the system’s database for any associated building name override, which has been set by a server administrator. if an override does not exist, the script identifies the building by looking up the strongest ap antenna in the signal table (lookup failure returns a value of “unknown”). during processing, the localization server also compares each ap’s network name (ssid) to the campus ssid, determining whether the mobile device is “on campus.” having been given the employee’s name, return message, and a set of scan results, the localization server calculates the location name, building name, and on campus status. the server updates the employee table with these data and the current time. once the employee table is updated, the calculated values are returned to the mobile app and board.php for display purposes. figure 3. determining a location the phone app a locally developed android app such as this may be deployed to any compatible phone, either through the eclipse development environment (with the correct drivers) or by downloading it from a web page (though some carriers prevent this). none of the app’s functionality requires rooting or jailbreaking the phone, so users do not risk losing carrier support or updates. the only real requirement is to compile the app with support for the sdk level matching the oldest participating phone and to have “usb debugging” or “unknown sources” options enabled on each phone. in order for the app to be most effective: it must launch on boot and not be killed by any task killer personnel must carry their phones at all times the lamp server must be available to the phones at all times, through either local wi-fi or a data plan if the phone has one the wi-fi antenna on each phone must be active at all times, which may be done programmatically each user must have a distinct username in the employee table and that username must be set in each phone’s app settings mobile app components the following are the main components of the system’s android app. main user interface – it is necessary to have a simple interface to populate the fingerprint database and set simple preferences, such as username. this also starts an “alarm” that triggers the periodic actions. on android, this class extends activity and is the main entry point for the app. name button – takes user-selected or user-entered input for a location name and associates it with one or more of the most recent previous scan results. if future apps ever use a location determination algorithm that calculates standard deviation, it is best to have at least two scans per definition. confirm button – adds most recent previous scan results to refine the current location’s fingerprint definition (i.e. if the location has been correctly identified) correct button – adds most recent previous scan results to a chosen location’s fingerprint if it has previously been incorrectly identified (i.e. from the server’s list of all defined locations; see figure 4) menu->settings – allows the user to set values such as user name, location recognition frequency, return message, and match threshold (i.e. a composite percentage of all of the different perceived antennae together) at which a fingerprint is no longer considered as matching according to wmu’s specific algorithm (which strives for greater accuracy than nnss; the authors prefer a location of “unknown” to one defined by weak ap rss detected from nearby) figure 4. correct button fed by selectall.php autostart – in order to be most useful, the app must start on boot. a class that extends broadcastreceiver with an onreceive method that calls startactivity to launch the main interface accomplishes this (if properly registered and has permission for receive_boot_completed in the android manifest file). alarmreceiver – another broadcastreceiver, this receives the alarms at regular intervals, does one or more wi-fi scans over a period of as many seconds, receives the scan results, and uses the httpcon class to send that information to the localization server. a single broadcastreceiver on an alarm proved to be the most reliable and energy-efficient timing method. when the phone is asleep, the alarm wakes up enough to create the receiver, which does its task and destroys itself. httpcon – as there is no remote database support on android, it is common to use an http layer to transfer data from the app to the server as this system does. the httpcon class in this object uses json objects and http post to transfer information between android java and the php on the localization server in the place of remote database connectivity apis. data presentation as stated above, board.php displays current employee locations both on phones running the app and in a web browser. wmu libraries systems removed a dry erase in/out board near their office entrance and mounted a computer monitor on the wall in its place to display employee locations. the monitor continuously displays a full screen web browser using an add-on to automatically refresh location information from the lamp server (see figure 5). the system’s web output is styled after the layout of the board that the wall-mounted monitor replaced. the name column contains the username of each participating cell phone owner (as entered into the app and database). below the name is the user’s cell phone number, entered into the database by the system administrator (i.e. not from the phone, as the user may be using a device without cellular phone service). the second column contains a color-coded circle indicating whether the user was last seen on or off campus (green for on, or red for off) and whether they checked in during the last fifteen minutes (light for yes, or dark for no). the third column contains the name of the building, either from an added column in the location table or simply based on the strongest antenna observed during the training/collection stage. the fourth column, labeled department, contains the user-entered location name (e.g. a department, a room, a desk, or anything else). the fifth column contains a free form text message set by the user, often containing an intended return time or the name of the meeting after which they will be returning. the last column contains a timestamp so that the reader can tell how current the information is, with particularly old information displayed in gray text. figure 5. in/out board web display please see the appendix for a list of detailed technical considerations (in the form of questions and answers) the authors recommend to others wishing to implement a system like wmu libraries’ mobile in/out board. lessons learned as stated above, how a wi-fi fingerprinting application functions depends heavily on the location determination algorithm it uses, and the local environment. readers should refer to the growing body of wi-fi fingerprinting application literature when considering which algorithm to use or adapt. given wmu’s specific requirements, the authors wrote their own algorithm using manhattan distance. in general, users have found the system easy to use, and far preferable to and much more accurate than the previous physical in/out board. one user-suggested modification that has been implemented is to log a user’s last several locations, so that staff may identify and discount the occasional false locations that are recorded. the authors are also considering displaying the past several recently recorded locations through board.php to help those seeking a particular user. however, leaving aside the question of how best to do this, recent locations may not always be useful, because the app allows varying device check-in frequency (ranging from manual and only periodic, to automatically, every twenty to sixty seconds). a typical user checks in every thirty seconds, which the authors consider a best practice. different check-in times (ranging from every twenty seconds, leading to a long list of redundant locations, to once per hour, leading to a much more reasonable list) are appropriate for different users, and the log of recently recorded locations is most useful if the user has chosen appropriately. the mobile in/out board is built to store and display names and phone numbers, but the system also includes support for contacting each phone not using cellular service (e.g. through a voip or text function such as skype). this may be useful given that wi-fi signal strength is typically much more robust than the highly variable cellular signal strength in wmu’s main library, and given the use of some no-service cellular phones. library administrators, who mostly also use android smartphones, also have busy calendars that keep them moving all over campus, sometimes unpredictably. they have expressed interest in implementing a separate mobile in/out board so they may keep track of each other. wmu libraries has also discussed using this system for roving reference librarians, as well as offering the system to other support units on campus. no matter the job, if someone wants to be more available, using smartphones (even personally-owned or without service) on existing wi-fi is a low-cost option. finally, systems like wmu’s are susceptible to local network and other conditions. for example, wmu campus it is currently in the process of replacing the main library’s network switches, wireless network, and wired phones. were an entire wireless network to be replaced at once, this system’s entire database would need to be retrained. given other concurrent campus network projects, campus it is replacing the network in phases (adding a switch, then adding new wireless aps, then replacing phones, and finally removing old switches and aps), one floor at a time over an approximately one-year period, requiring only that staff add to the fingerprints of locations as they visit them. in addition, given differences in performance that seem to coincide with changes in weather and building hvac conditions, the authors suspect that variability in signal strength may be influenced by environmental conditions such as relative humidity (given that water can absorb radio signal). notes [1] wmu’s local custom algorithm is not included. instead, the pseudo-code for nnss, which is the most straightforward starting point can be found in the appendix. readers interested in wmu’s algorithm or source code are welcome to e-mail keith dot kelley at wmich dot edu. [2] it is important when reading this article to understand what an rss value is. a chipset manufacturer actually calculates and reports an arbitrary integer that may be an average of the signal strength it detects over a specific time period that can definitely be as long as a full second. the 802.11 specification places few requirements on received signal strength indicator (rssi). rssi does not necessarily map to distance, power, or the same number on different devices nor is it necessarily calculated in real time. also, rssi is not a constant and interference from multiple access points has a strong negative impact on positioning accuracy (chan et al. 2010). similar systems navarro et al. (2010) describe a senior project that tracks children on a playground. the system discussed by lee et al. (2010) is participatory and room-level. bolliger (2008) describes redpin, a participatory and room-level system, but which also uses knn and x and y coordinates. ching et al. (2010) describe uniwide, which provides room-to-room accuracy at a university via fingerprinting. suggested reading liu et al. (2007) provide a good place to start reading about methods of indoor positioning including wi-fi fingerprinting and symbolic (referred to as semantic) locations. references ali s. 2011. indoor geolocation for wireless networks [dissertation]. cranfield (uk): cranfield university. 271p. retrieved from http://dspace.lib.cranfield.ac.uk/handle/1826/4721 bahl p. and padmanabhan v. 1999. user location and tracking in an in-building radio network. microsoft research. retrieved from http://research.microsoft.com/en-us/um/people/padmanab/papers/msr-tr-99-12.pdf bahl p. and padmanabhan v. 2002. radar: an in-building rf-based user location and tracking system. infocom 2000. nineteenth annual joint conference of the ieee computer and communications societies. ieee. p. 775-84. retrieved from http://research.microsoft.com/en-us/um/people/padmanab/papers/infocom2000.pdf bolliger p. 2008. redpin-adaptive, zero-configuration indoor localization through user collaboration. proceedings of the first acm international workshop on mobile entity localization and tracking in gps-less environments; 19p. retrieved from http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.155.205&rep=rep1&type=pdf chan e, baciu g, and mak s. 2010. effect of channel interference on indoor wireless local area network positioning. ieee 6th international conference on wireless and mobile computing, networking and communications (wimob); p. 239-45. retrieved from http://www.cse.ust.hk/~csclchan/paper/effectofchannelinterference.pdf ching w, teh r, li b, and rizos c. 2010. uniwide wifi based positioning system. 2010 ieee international symposium on technology and society (istas); p. 180-89. retrieved from http://www.gmat.unsw.edu.au/snap/publications/ching_etal2010.pdf liu h, darabi h, banerjee p, liu j. 2007. survey of wireless indoor positioning techniques and systems. ieee transactions on systems, man and cybernetics. part c, applications and reviews 37(6):1067-80. retrieved from http://www.sis.pitt.edu/~dtipper/2011/survey1.pdf dawes b and chin k. 2011. a comparison of deterministic and probabilistic methods for indoor localization. j syst software. 84(3):442-51. (coins) hansen r, wind r, jensen c, thomsen b. 2009. pretty easy pervasive positioning. advances in spatial and temporal databases: 11th international symposium; p. 417-21. retrieved from http://books.google.com/books?id=abv0uh7aieac&printsec=frontcover&source=gbs_ge_summary_r&cad=0#v=onepage&q&f=false honkavirta v, perala t, ali-loytty s, and piche r. 2009. a comparative survey of wlan location fingerprinting methods. ieee 6th workshop on positioning, navigation and communication; p. 243-51. retrieved from http://math.tut.fi/posgroup/honkavirta_et_al_wpnc09a.pdf jekabsons g and zuravlyov v. 2010. refining wi-fi based indoor positioning. proceedings of the international scientific conference applied information and communication technologies; p. 87-95. retrieved from http://www.cs.rtu.lv/jekabsons/files/jek_aict2010.pdf kaemarungsi k and krishnamurthy p. 2004. properties of indoor received signal strength for wlan location fingerprinting. the first annual international conference on mobile and ubiquitous systems: networking and services (mobiquitous); ieee; p. 14-23. retrieved from http://www1.cs.columbia.edu/~andreaf/downloads/01331706.pdf lee m, yang h, han d and yu c. 2010. crowdsourced radiomap for room-level place recognition in urban environment. 8th ieee international conference on pervasive computing and communications workshops (percom workshops); ieee; p. 648-53. retrieved from http://academic.csuohio.edu/yuc/papers/smarte2010_submission_2.pdf li b, wang y, lee h, dempster a and rizos c. 2005. a new method for yielding a database of location fingerprints in wlan. communications, iee proceedings-iet; 580p. retrieved from http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.70.5053&rep=rep1&type=pdf navarro e, peuker b, quan m. 2010. wi-fi localization using rssi fingerprinting [senior project]. san luis obispo (ca): california polytechnic state university. retrieved from http://digitalcommons.calpoly.edu/cgi/viewcontent.cgi?article=1007&context=cpesp&sei-redir=1 tsui a, chuang y, chu h. 2009. unsupervised learning for solving rss hardware variance problem in wifi localization. mobile networks and applications. 14(5):677-91. retrieved from http://140.112.30.160/papers/unsupervisedlearningforsolvingrsshardwarevarianceprobleminwifilocalization.pdf about the authors keith kelley is director of systems at western michigan university libraries and a ph.d. student in computer science at western michigan university. scott garrison is associate dean for public services and technology at western michigan university libraries. karlis kaugars is chief information officer at suny college at oneonta. appendix: decisions implementers need to consider as this article points out, there are many possible ways to implement and configure a wi-fi fingerprinting location system, depending on local preferences and environment, algorithm, and many other factors. this appendix offers a list of questions that those who wish to deploy such a system should consider, and the authors’ suggested answers from their own experience. training locations how many scans, over what period of time, are needed at each department or desk to train/collect a location (define a fingerprint)? the practical answer is that the app can train for as long as the user has patience to wait until fingerprint definition is complete. ten scans over ten seconds on the confirm or correct operation is the most the app should add to a definition at once, but the app can actually start a fingerprint definition with a single scan result. for an isolated location with a fingerprint that has little in common with other fingerprints in the database, one scan result is all that is needed, and it saves space and processing time for all location recognition operations. note that due to high variability of antenna rss values even for a stationary object, no single train/collect session will record every possible value, or the full characteristics of the probability distribution. how many scans over what period of time are needed on the mobile device to match that fingerprint? as noted above, rss values fluctuate considerably and an app will definitely have mixed results if it tries to match on just one measured sample. actual impact on accuracy will depend on choice of location determination algorithm. like wmu’s own testing, honkavirta et al. (2009) showed accuracy improvements as more scan results were sent from a fixed location, but noted that there were no dramatic increases beyond ten seconds. since a mobile device is likely to be moving, scan results over a period of even ten seconds may be from several locations, which will decrease accuracy by supplying bad data to the algorithm. wmu’s live tests have shown that too few scans resulted in false positives from distant locations, and too many in a noticeable lag in location detection. a practical compromise is three  scan results taken over three seconds. no of measurements sent 1 2 3 4 5 6 7 8 9 10 correct identifications 86.7% 90.0% 91.5% 92.2% 92.8% 93.2% 93.4% 93.8% 94.1% 94.3% table 1. effect of number of measurements observed on correct location identifications. initial testing of wmu’s algorithm showed two scans were much better than one, but while there can be improvements from including four or more scan results sets, each additional set only resulted in a correct match less than 1% more often. this is just a guideline; results will be implementation specific. an app will generally need “a lot” of measured signals to build the fingerprints in a radio map in the training stage and “a few” to recognize the location in the positioning stage. how many locations should be trained? one per room with solid walls? one per desk? one every ten feet? the answer to this question also depends on specifics of the implementation and the environment. to avoid the upfront cost in training time, wmu uses a participatory approach, which works well largely because all users have the same model of mobile device (all mobile devices will not see the same rss values at the same location). if the app uses a participatory approach, users will add as many locations as they like based on either their needs or the perceived performance of the system. they will likely want to add many more than are needed. the room-level system in hansen et al. (2009) suggests taking a fingerprint every two to three meters, but wmu’s experience suggests that the answer really depends on desired results and the chosen algorithm’s accuracy. in a tracking system like this, just being a nearest neighbor in signal space may not be near enough, so wmu set a threshold at which a user’s location is considered unknown and leaves this as a configurable option on the mobile app. a higher unknown threshold generally means a smaller radius is considered as part of the location definition and a larger portion of the real world is considered to be an unknown location. it may be beneficial to define one fingerprint facing each cardinal direction at each location and consider them separate locations with the same or similar names. however, the user may not value that benefit highly enough to willingly do such detailed fingerprinting, and instead consider the single orientation results as good enough. likewise, a user can take similarly named fingerprints at different times of day. both are more realistic than trying to have one fingerprint definition that matches under all orientations at all times of day. as a user, this translates into adding, confirming, or correcting fingerprint definitions when the user happens to be at the location, so that the system works the way the user does, with the user contributing more information to the fingerprints of more frequently visited locations. in practice, the user is both the producer and consumer of the location definition, defining and using fingerprints. choosing an algorithm the article very briefly covers what a good fingerprint recognition algorithm is, but what else should implementers consider about the algorithm? not all algorithms are suited to symbolic locations (many require x and y coordinates to function). of the static (unfiltered) algorithms tested in honkavirta et al. (2009), some perform better than others at different levels of training effort. however, considering the small impact on accuracy (0.2m error), the app may as well use straight nnss for the sake of simplicity. symbolic locations generally represent a room or a desk or a cubicle; a change of 0.2m still puts the device in the same location. many articles exist comparing appropriate algorithms. lee et al. (2010), for example, compared a deterministic knn method with k=1 and a probabilistic method, finding slightly less accurate but vastly more efficient results from the latter. this offers license to go either deterministic or probabilistic. dawes and chin (2011) and honkavirta et al. (2009) are recent articles among the many that have compared a large number of different algorithms. all of the ones that do not rely on having x and y coordinates will perform the task, so the implementer can make an informed decision about the benefits and drawbacks of each. something close to nnss with manhattan distance is a good place to start because it is simple, it works well enough, and articles improving nnss use in similar applications appear continually in the literature. using an average of a representative size training set (depending on number of nearby aps and location) can be adequate, while using a single scan result can be next to meaningless in a densely populated radio map. the following is pseudocode for the simplest case location determination algorithm using a fairly straightforward implementation of nnss with manhattan distance. initialize location distances to maxint foreach location in database foreach expected_antenna in fingerprint foreach observed_antenna in scan result set if expected_antenna == observed_antenna location[distance]+= abs(expected_antenna.average_rss observed_antenna.average_rss) return location with shortest distance in the end, the authors wrote their own algorithm (which is somewhat more complicated and more efficient) based on local environment and field tests, but something as simple as the above can be a functional starting point for a location determination algorithm. note that the above method implicitly ignores new and dropped aps, which is not specified in the original algorithm and has some impact in those cases, but none in the ideal case. matching how much of a match is a match? a partial fingerprint match, such as seeing one of seven antennas at -92dbm instead of -33dbm may make a certain location the nearest neighbor, but also may mean the mobile unit is a building away. in these situations, it may be better to return a value of “unknown.” for this reason, the implementer may choose to use a percentage score when configuring the system instead of (or in addition to) a distance in signal space or using a maximum allowable distance in signal space. the final answer depends on the chosen algorithm, and will have to be discovered through usage, but it may be more generous than expected. wmu has users that use thresholds between a 10% and 40% match on score. should the app use a percentage match or just a score or distance units? the location determination algorithm will determine the metric used to judge whether one location is a better match than another, and this may differ widely. it is important to consider that a straight score based on the training set won’t work well with unevenly defined fingerprints (larger fingerprint definitions will always beat better matches). an algorithm based on a percentage match of observed signals against expected signals will have the problem that weakly defined fingerprints can outweigh well-defined ones, which is also a problem. a percentage of expected signals that match over observed signals will not give too much advantage to large (well defined) fingerprints, but will sometimes give too much advantage to small fingerprints (i.e. a 100% match to a single scan result probably shouldn’t be considered a better match than a 90% match to 400 scan results). luckily, for the most part, an implementer can browse the research, pick a location determination algorithm that has a combination of strengths and weaknesses that suit a particular implementation, and plug it in on the localization server without having to update the mobile clients (which is more difficult once users have them in the field). how many aps should i consider when matching? honkavirta et al. (2009) report that the optimal choice is no more than five aps. however, jekabsons and zuravlyov (2010) found that there is benefit from using all available aps, including the weakly sensed ones. in wmu’s tests, it depends on the algorithm used to match the observed set to the expected fingerprint. if the chosen algorithm does not include a recommendation, use every antenna observed. miscellaneous should i use filtering? no, the application does not demand it. many location determination algorithms use filtering to improve accuracy at the cost of simplicity and processing time. filtering is generally a good thing, but adds complexity and isn’t necessarily good in every application. honkavirta et al. (2009) conclude that “all filters improved results significantly over static methods” in their testing, but dawes and chin (2011) noted that in their testing, at one second of observation data (four samples), performance was better with an unfiltered algorithm. the authors estimate that most of the articles published on wi-fi localization are not considering symbolic locations, so their findings do not necessarily apply. what devices should i use? have all system users on the same model of device if possible, or the app will have to adjust for differences or the users will have to live with some fairly poor matching. wmu’s tests show that a fingerprint made by one device may vary more from a fingerprint made by a different device, than from a fingerprint made at a different location. the scatter plots in figure 6 show the mean rss of an antenna as observed from two phone models (an htc aria and a motorola citrus) at the same location over the same period of time. the pattern of clustering shows that while there is some correlation between what two devices perceive it is not a simple task to translate between fingerprints on two different model devices. figure 6. two phones’ observations of the same environment: perceived rss from aria vs citrus phones for antenna 1 (top) and antenna 2 (bottom) tsui et al. (2009) describe one way of addressing the problem that different devices see different rss values in the same circumstance, but the effort, complexity, and uncertainty can be saved if users simply use the same devices. how often should i scan for signals and recognize my location? users will notice significant battery usage if the device uses the wi-fi excessively, so be prudent. while wi-fi does not use as much battery as gps, it can still be the largest cause of battery drain. if users have different movement habits, it makes sense to make this a configurable option, with staff who move less often checking in less often. should i use the gps location when it’s available to use because i can? definitely not. polling gps will drain users’ batteries faster than any other activity, potentially rendering phones dead that are primarily used for keeping in emergency contact. using gps is worse than not having a phone at all, because it gives the user a false sense that they are available (when in reality, the phone is unlikely to last half the workday). can i use wi-fi fingerprinting in multi-floor environments? yes, many implementations are multi-floor, including wmu’s. the more aps that are visible, the more likely the implementation is to work well (up to the point where channel interference becomes a problem; chan et al. 2010). a lateration method would need three antennas to determine a location in two-dimensional space and four for three-dimensional space, so it is reasonable to assume a fingerprinting method will need at least that many. subscribe to comments: for this article | for all articles one response to "an android/lamp mobile in/out board based on wi-fi fingerprinting" please leave a response below: mikey, 2012-04-20 cool thing thx. leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – code as code: speculations on diversity, inequity, and digital women mission editorial committee process and structure code4lib issue 28, 2015-04-15 code as code: speculations on diversity, inequity, and digital women all technologies are social. taking this socio-technological position becomes less a political stance as a necessity when considering the lived experience of digital inequity, divides, and –isms as they are encountered in every-day library work spheres. personal experience as women and women of color in our respective technological and leadership communities provides both foreand background to explore the private-public lines delineating definitions of “diversity”, “inequity”, and digital literacies in library practice. we suggest that by not probing these definitions at the most personal level of lived experience, we in the lis and technology professions will remain well-intentioned, but ineffective, in genuine inclusion. by sharon l. comstock, jerica copeny, and cynthia landrum all technologies are social. no matter the discipline-bound definitions of our most closely allied fields of library and information science (lis), human computer interface design (hci), and computer science (among others), our understandings are grounded in socially constructed experience. this obvious statement bears repeating and requires scrutiny when we are faced with the very practical concerns of a community recognizing educational inequities, secondary and tertiary digital divides (if not primary), and indistinct -isms that may be barriers to solving the very concerns we in libraries are attempting to address in our respective communities. as library colleagues and women—and two of us as women of color—we are discovering core, shared questions about our perceptions of the afore-noted inequities in the community in which we work; and—perhaps more urgently—in our personal, lived experiences of our roles in addressing those inequities. over coffee cups and behind closed office doors, in after-work texts and late-night tweets, we explore complex socio-technical territory. together, we realized we need to enter that public intellectual space where emerging questions can be given more careful attention than those twilight conversations allow. in fact, the very informal channels we have been using serve to marginalize the very lines of inquiry we are co-discovering. lines of inquiry: probing practice our initial questions center around our experiences in our work that we address in our respective spheres: “what responsibility does the library have to work as social justice leader—not merely as traditional participant? how do we facilitate and strategically align considerable community resources to solve identified education and digital literacy inequities along racial lines? what lines of legitimacy do we take: as women, as digital advocates and leaders of color, and librarians: a trifecta that can disempower (and all three which we have faced on personal and professional levels)? “ we are not alone in examining the problem of inclusion and diversity in our chosen profession, library and information science (c.f., the library quarterly: information, community, policy, vol. 85, no. 2, special issue: diversity and library information science education, [april 2015]). however, the discourse of diversity itself needs to be engaged as a topic: where inclusion alone may not be enough to address the enduring barriers to sustained agency and power beyond mere “access” to education and technologies. we are compelled to voice our emerging questions—and in so doing frame a practitioner-based research study grounded in everyday work-life. our positions of understanding and roles are distinct, yet intertwined: a digital services librarian trained in human-computer interactive services and an active digital advocate; an information behavior and informatics literacy researcher, evaluator, and strategist whose role is to inform and assess the library’s role in the community; and an administrator responsible for a robust public services agenda and library leader in both local and national library arenas. legitimizing lived experience: a library study considering the workplace socio-technologically, we take seriously our position within it as parts of a complex system. in fact, to neglect our lived experience would be reductivist at least and falsifying at worst. we are consciously choosing to make visible our experiences; making explicit the public-private lines we cross and re-cross in daily work culture(s) in general, and two projects in particular. adapting auto-ethnographic method—as part of a triangulated, mixed-methods approach as appropriate for a public library applying evidenced-based practice—we hope in the coming months to tease out and directly address what “diversity”, “coding”, and “inequity” mean in situated contexts. necessarily, our past informal “debrief” sessions, will be formalized; intra-coding our work-place field notes in the context of two library projects. specifically, we are examining our public library’s implementation of two technology-based initiatives with stated objectives of addressing digital literacy and education inequities: (1) assessment of library technology classes relative to stated needs to address digital literacies in the village that have implications for learners’ needs for employment, social interaction, and functional technology skills; and (2) the launch of a state of illinois history digitization grant award, “hacking hemingway: cracking the code to the vault” that will digitize ernest hemingway artifacts and allow us to work with local middle-schoolers to build “computational thinking”. in the first of these two, we are six months into what will be an 8-month evaluation of current on-site technology classes, using observation, librarian-teacher reflections and focus groups, and participant focus groups. thus far, we are seeing clear incongruities in at least two areas: learner-participants’ stated perceptions of “learning” relative to observed digital literacy behaviors; and librarian-instructors’ desire to solve what they call a “digital divide” in the absence of a shared definition of “digital divide” or criteria of what successful outcomes may be in addressing it. the second project, hacking hemingway, while only just begun january 2015, has been designed to build over 15 months a community of middle school learners who engage with these digital objects in ways that have meaning for them. the key for us has been how librarians, the public, and teachers use the terms, “encode”, “code”, and “coding” in seemingly distinct ways to signal or signify status. we particularly want to probe what these mean in the very real context of documented achievement gaps between african american and white students in the school district. most recent illinois assessments in the district of grades 6-8 show a sustained achievement gap of 36% (reading) and 46% (mathematics), with african american students alarmingly at the disadvantage. we want to unpack what we are only beginning to see: “coding” as magical—or worse–out of reach. both digital literacy initiatives require a nuanced understanding based on thick data if we hope to address their stated intentions, much less identify outcomes. building practitioner-research into the natural cycle of evaluation-design-reflection-redesign is a formative process that can inform both library accountability and librarian communities of practice. we believe by using these two programmatic areas of strategic emphasis, we may find a way in—and through—the complexities of diversity and inclusion when we consider digital fluencies. we believe by addressing the very personal and real -isms of being women and women of color–where negotiated roles and boundaries of self and other are narrative evidence—we can align ourselves with those in the community we serve. it is a conscious choice to identify with—not separate ourselves at seemingly safe distances from—the community we seek to serve. de-coding “coding” in context we suggest that by not probing lis’ definitions of “diversity”, “inequity”, and digital literacies at the most personal level of lived experience, we in the lis and technology professions will remain as well-intentioned, but ineffective in genuine inclusion. we question the editors’ statement “…we strengthen our libraries if we enjoy and engage with these differences.” will we? how do we know? what evidence of success do we have? in the coming months, we will be challenging legacy beliefs anchored in privilege that may contain, limit, and exclude. by applying formative reflective inquiry in intentional and structured ways, we are seeing the opportunity to bring tacit experience into the forefront as legitimized data. therefore, our perspectives will truly be “in the mix” of mixed-method research in applied environments such as we need in a library. we see data and our writings as multi-purpose: assessment, evaluation, and research. personal narratives provide powerful insights that can inform public action. we hope to bring into question the very underpinnings of how we represent knowledges and legitimize–dare we say, “codify”–our socio-technologies of everyday library practice. readings cole m. 1971. the cultural context of learning and thinking: an exploration in experimental anthropology, new york: basic books. eisenhart m.a. finkel e. 1998. women’s science: learning and succeeding from the margins. chicago: university of chicago press. finamore j. kahn b. 2015. characteristics of the college-educated population and the science and engineering workforce in the united states [internet]. [cited 2015, apr 8]; available from http://www.nsf.gov/statistics/2015/nsf15317/ foucault m. 1980. power/knowledge, new york, ny: pantheon. lave j. 1993. the practice of learning. in: chaiktin s. lave j., editors. understanding practice: perspectives on activity and context. cambridge: cambridge univ. press. p. 3–32. stake r. 2000. case studies. in denzin n. lincoln y., editors. handbook of qualitative research. thousand oaks, ca: sage. p. 435-454. stake r. 2006. multiple case study analysis. new york, ny: guilford press. stake r. 2010. qualitative research: studying how things work. new york, ny: guilford press. sundin o. johannisson j. 2005. pragmatism, neo-pragmatism, and sociocultural theory: communicative participation as a perspective in lis. journal of documentation. 61(1): 23-43. tuominen k. talja s. savolainen r. 2005. information literacy as socio-technical practice. library quarterly 75(3): 329-345. wenger e. 1999. communities of practice: learning, meaning, and identity. cambridge, uk: cambridge. williamson k. 2006. research in constructivist frameworks using ethnographic techniques. library trends 55 (1): 83-101   about the authors sharon comstock earned her ph.d. from the graduate school of library and information science, university of illinois, urbana-champaign, studying everyday life information behaviors of young adults as legitimizing knowledge (comstock, 2012); and her m.a. from northwestern university studying folklore motifs. applying john dewey’s theories of pragmatism, her research addresses informatics in socio-cultural, technological contexts. drawing on her work in evaluation of national science foundation grants, among others, she applies ethnography, qualitative inquiry, and discourse analysis to make visible the tacit ways communities pose—and work to solve—new questions. her email address is scomstock@oppl.org. jerica copeny, m.l.s., m.s. student at depaul university in human computer interaction, chicago, digital services librarian, oak park public library-she is currently on the board of directors for literacy volunteers of western cook county where she is working to incorporate concepts of digital literacy into the strategic plan for the organization. she loves the great intersection, of when technology can be used as tool to learn and learning can be used as a tool to explore technology. she is an avid believer that coding can be an art form of self expression. her email address is jcopeny@oppl.org. cynthia landrum has worked in academic, medical and public libraries in arizona, illinois and pennsylvania. a member of several community and professional associations, cynthia serves on ala council and the spectrum advisory committee. in her current role as assistant director for public service at the oak park (il) public library, cynthia provides strategic direction, and leadership in the areas of community engagement, organizational development and learning, stewardship, and assessment and evaluation. cynthia earned an undergraduate degree from northwestern university, mlis from university of southern mississippi, was awarded the bcala e.j. josey scholarship as a graduate student and has been trained as a public innovator through the ala/harwood libraries transforming communities initiative. she is a doctoral student in the school of library and information science at simmons college in boston with interests in diversity, inclusion, equity and social justice in lis practice and leadership, use of narrative methodologies and critical theory in lis research, spiritual leadership theory and transformative leadership in lis practice, and organizational development in public libraries. her email address is clandrum@oppl.org.   subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – responsive vs. native mobile search: a comparative study of transaction logs mission editorial committee process and structure code4lib issue 44, 2019-05-06 responsive vs. native mobile search: a comparative study of transaction logs the consortium of academic and research libraries in illinois (or carli) is comprised of 130 libraries, a majority of which participate in the union catalog i-share for resource sharing. the consortium implemented vufind 4, a responsive web interface, as their shared union catalog in december 2017. this study compared search transaction logs from a native mobile app that serves the consortium with search transactions in the responsive mobile browser. library professionals in the consortium sought to understand the nature of mobile search features by evaluating the relative popularity of mobile devices used, search terms, and search facets within the two mobile search options. the significance of this research is that it provides comparative data on mobile search features to the library ux community. by jim hahn introduction this paper reports the results of a logfile analysis of users’ experiences with the library catalog search accessed through mobile digital interfaces in the carli academic library consortium. the analysis included three months of search logs from two different versions of the library consortium’s vufind search index. susan farrell, a ux strategist, advocates analyzing 3 to 6 months of search logs for ux insights, writing that “…search engines can produce a log (text file) containing a list of all the questions and terms that users type into the search tool. logs also have useful information about each search query, such as the user’s ip address or other identifier and the time of the request, which means you can often look at a sequence of searches in one person’s session if you sort the list by user identifier and time” (farrell, 2017, p. 1). according to farrell, analyzing logs can help ux practitioners understand “…what your web visitors want, how they look for it, and how well your content strategy meets their needs” (p. 1). in the case of this study, search logs were analyzed to understand the popularity of mobile device types used (e.g., ios and android devices), the nature of search terms in mobile searches (the number of words per query), and vufind search facets used and not used in mobile search of the library catalog. vufind is available both as a responsive mobile web page and through a native mobile search app. each of the two platforms offers specific strengths and benefits (discussed in more detail in the following section). this study analyzed the search logs from the two different platforms to see whether and in what way users’ search behavior for the same service differs between the platforms. because log files contain both the actual searches’ text, as well as the amount of time users spent interacting with the platform, we can learn a great deal by performing some basic data analysis. the questions this research addressed included: do users approach the service differently because of their differing interfaces? if so, in what way? what are the implications for library mobile app design? the design of, and access to, mobile technology has evolved as the web has grown to accommodate more and more mobile devices, and there are a variety of options to increase access via mobile devices. mobile access history and development in libraries is considered in the literature review. literature review this literature review addresses three related topics: explanation of the terminology used to describe native apps and mobile web content; discussion of contemporary library mobile app design, with an emphasis on the use of mobile web sites versus native apps, and finally, other ux studies that investigated specifically the differences between use of native and mobile web apps both in and out of libraries. mobile and native app terminology a mobile website is developed using html which is interpreted by a web browser to render content on a device’s display. in contrast, a native app is written in a compiled language which takes better advantage of a device’s hardware. by taking advantage of mobile phone capabilities, such as gps and bluetooth low energy (ble) connectivity, native apps offer a seamless interface with these sensors. a general low-cost solution to make a website mobile friendly is to use responsive design. responsive mobile design is a content strategy designed “…to make a web page look equally good regardless of the screen size of a device” (kim, 2013, p. 29). table 1 below delineates and summarizes the various mobile design options available to libraries. contemporary frameworks allow for development of hybrid apps that include html compiled to a binary for distribution on an app store, and newer libraries to compile javascript to native apps. table 1. terminology and definitions used to describe mobile access options. term definition example native mobile apps written in compiled languages and delivered to users via app stores. ios, android responsive websites written in html, css, and javascript. html websites use css to respond to mobile screen size. hybrid apps a combination of html, javascript, and compiled languages. may exist in app stores or be hosted on websites. contemporary mobile app design mobile access in libraries has been of interest to ux practitioners since the contemporary wave of smartphones emerged in the form of iphone and android devices. a review detailing mobile implementations made by libraries between 2010 and 2016 (blummer & kenton, 2016) revealed that library studies on mobile apps specifically were underrepresented the most in the literature. however, there were several similarities in the progression of trends among libraries that implement mobile initiatives, which, in general, followed the path of “…obtaining support, collecting data, as well as promoting and assessing the projects. one or more of these strategies remained characteristic of libraries’ mobile initiatives identified in the literature during these years” (blummer & kenton, 2016, pp. 118). a related review of mobile learning tools available noted that the volume of research had doubled from 2012-2016 compared to the literature in 2007-2011 (tu & hwang, 2018). these studies underscore the interest and growing body of work in the area of mobile access, but also the need for research studies that provide user implementation data related to native mobile app use. while studies of mobile app use do exist (hahn, ryckman, & lux, 2015), sustained study of mobile app uses remains comparatively rare in the literature. this paper helps fill this gap by providing mobile app study data. a research paper that compared access of library-hosted video content from an app and a mobile website found an equal distribution of use with no clear preference on the part of student users of the content (wong, 2012). the authors noted that the app use exceeded expectations because they believed that overall mobile access in libraries would be comparatively low (wong, 2012, pp. 108). therefore, it is possible to find information supporting either mobile or native app development in the literature. this could be attributable to the way that each author frames the debate, but as the ux practitioner may well surmise, the choice ultimately will depend on the objectives that the tool would help meet. it may well be the case that among end users, the underlying technology is less important than whether the technology is useful to complete desired tasks quickly and easily (berg, 2010, p. 8). related studies on mobile apps: responsive web design and hybrid approaches a mobile usability study of web presence designed responsively predicted that implementing a responsive site would provide greater consistency across all of the library’s digital channels (tidal, 2017). however, the study found that “…responsive design may not necessarily equate with a singular experience across different devices. smartphone users will continue to have a greatly different experience compared to that of tablet and workstation users” (tidal, 2017, p. 32). with respect to smartphones, tidal noted that facets are hidden from users so that filtering options by format is difficult because it is found at the bottom of the smartphone list view (2017). this study reviews and supports the relative popularity of facets used in mobile vufind and native apps. in a similar study of web services that send data to native and mobile web apps, researchers found that, compared to a native app’s response time “…the web apps initiate more requests and consume more traffic and need longer response time” (liu et al., 2015, p. 343). a comparative study of mobile web and desktop-based searches in a study of fifty users revealed that a library mobile website completed some tasks more efficiently with faster search times (chanlin & hung, 2016). these included searching by title, author, and subject; therefore, mobile interfaces vary from library to library and findings based on task may not have the ability to be generalized to mobile websites across all user studies. researchers have advocated clustering the results of search queries and found it effective in retrieving information by prototyping ways to improve the mobile presentation of information-based search results (carpineto et al., 2009). hybrid approaches to mobile native app development were introduced to help web developers take advantage of native app development and make use of an app store’s reach and popularity. previously, hybrid approaches were viewed as an alternative to creating a full-fledged native app (charland & leroux, 2011). in their review of the relative benefits and drawbacks of each, the authors suggested “…software makers must balance the web-vs.-native debate based on an application’s primary objectives, development and business realities, and the opportunities the web will provide in the not-so-distant future” (p. 53). with respect to summarizing mobile design trends overall, blummer and kenton’s (2016) analysis of over one thousand papers found that, “the literature on mobilizing libraries’ services and resources point to the importance of incorporating responsive web design, conducting an environmental scan, partnering with other entities in developing mobile services, employing commercial products in their creation, and promoting the services to students. monitoring students’ usage of mobile devices, as well as the library’s services and resources following the mobilization of the website is essential to ensuring the library remains relevant…” (p. 129). the data reported in this study helps provide an indicator of overall mobile use of a responsive website and native app. these data are indicators of mobile search popularity in the carli consortium which have not been previously understood. methods the method used was a transaction log analysis. previous studies in this area were illustrative in understanding foundational approaches to analyzing logs and what types of technologies are available that make log analysis possible. peters’ (1993) pioneering work indicated that early research did not attempt to build a methodological science to analyze logs but focused more instead on a specific study’s objectives to understand subject and keyword searches or other retrieval metrics, such as recall and precision. jensen (2006) refined and standardized transaction log methodology and articulated an approach to employ databases that store information about queries at the time of the transaction. these databases proved useful subsequently in understanding trends in user searches, particularly because the queries were organized in a relational model that made querying by researcher interest possible. at present, study logs are analyzed using relational models, but database queries have been replaced instead by using splunk enterprise software (available at: https://www.splunk.com/) to parse unfiltered server logs shown in figure 1, filter them based on browser data, and export them in a more amenable format, such as the comma-separated values (csv) model, a file format for associative data that is easy to read (figure 2). the csv is imported to visualize data and analyze them further with tableau (https://www.tableau.com/). tableau was the visualization software used to derive the trends in figures 5-8 reported in the results section. figure 1. server logs in the splunk enterprise system before exporting transactions as comma-separated values (csv). figure 2. search logs transformed in csv format used in tableau to visualize data. logs of both native and responsive mobile apps can be compared for search terms used, search query length, and relative popularity of each access type across the state, e.g., baseline data about the use of each.  the time frame of the logs analyzed was mid-july to mid-october 2018 and comprised ninety days of users’ searches across a majority of the academic libraries in illinois. because the web servers that display the catalog search results generate the server logs automatically, the server pre-processing steps depend on the data’s source.  in the case of the native mobile app, all log data serve the mobile app, although they could be serving a mobile phone or a tablet as well. in the case of the vufind 4 server, the catalog search results are parsed based on user agent data that the mobile browser sends the server. figure 3 shows an example of a browser agent from the logs. figure 3. sample browser agent from the search transaction logs. irb approval was obtained to analyze the data that users had entered into the search previously. this study was a retrospective human research study in the sense that we did not recruit participants, but analyzed their actions instead with no intervention on our part. filtering for facets in the logs was performed with the use of openrefine (http://openrefine.org/), a data tool that offers many filtering and data reconciliation options, and is used to work with messy datasets on the web. in the case of this research, openrefine was used to filter based on facets and search format types found in the query url path. figure 4. example filter used to report keyword-only searches in the native app. finally, a text analysis of search terms in both search logs was undertaken using voyant text mining tools. these tools can be run on a desktop or accessed from the web (sinclair & rockwell, 2016). the principal advantage of running a voyant app on a desktop is that the data are secure and insights into large datasets can be obtained without the need to transfer data over a web request. depending on the desktop system resources, the text mining results likely will be generated more quickly if they are on a desktop computer rather than the https://voyant-tools.org/ webpage. results of search transaction analysis general access and use access and use data were generated first to understand the makeup of the two server logs. the general access analysis found that during the 90-day study period, there were 7,983 native mobile catalog searches. a delineation by month and a grand total is shown in figure 5 below. figure 5. total native catalog searches by month and overall. a similar analysis of access data was undertaken for vufind mobile website totals. the analysis of these server logs indicated that there were 45,001 searches of the responsive mobile vufind website during the study period (figure 6). therefore, the initial data analysis indicated much higher mobile browser than native app access. a further examination of the percentage of mobile browsers represented in all searches showed that 2,172,451 searches were performed during this period with all browsers, inclusive of mobile access. therefore, those mobile agents that displayed a mobile friendly page represented only 2% of vufind searches during the study period. figure 6. responsive website totals by month and overall. number of ios and android devices among responsive vufind mobiles to understand android and ios devices’ relative popularity as a way to access the responsive vufind page, two popular mobile device types from the vufind webserver were analyzed (figures 7 and 8). a comparison of the totals in the figures shows that the ios devices observed in the search logs were two times more popular than were other mobile device types. figure 7. ios responsive vufind totals. figure 8. android responsive vufind totals. search terms the server logs from each service recorded search phrases that were analyzed with the voyant text mining software. the results indicated that the mobile responsive vufind search corpus had a total of 123,815 words and 14,221 unique word forms. search queries performed in the responsive vufind interface had a mean of 4.2 words. the most frequent words in the responsive corpus included “history” (512); “psychology” (493); “e2” (432); “american” (420), and “art” (412). figure 9. word cloud of responsive search terms voyant text mining generated. results from text mining the native mobile app search corpus found a total of 17,351 words and 2,181 unique word forms in searches.  the native mobile app’s mean number of words per search query was 6.5. the most frequent words in the native app query corpus were: “saundra” (322); “hitchcock” (257); “k.ciccarelli” (256); rochberg (173), and “darkest” (168). figure 10. word cloud of native search terms voyant text mining generated. catalog features used in searches asking what catalog features were used in the mobile searches revealed facets and format trends. with respect to the responsive website, 81% of the time (36,609 of 45,001 total searches), the default keyword/no facets were used. in mobile responsive searches, the next most popular facet was title followed by author. table 2 shows the results from the responsive mobile interface facets. table 2.responsive search facets used.  85% of all native app searches (6,791 of 7,983 total searches) also used the keyword facets. the next most popular facets were author followed by title searches, respectively. the results from the native app interface facets are shown in table 3. table 3. native app search facets used. format types were not used significantly in the mobile responsive web interface. book format searches were the most popular for this interface but represented less than 4% of all searches. table 4 shows the delineation of other format filters used. table 4. responsive search format types. searching by format type was observed more prominently in the native app. book format searches were the most popular for this interface and represented 20% of all searches. another popular format filter for the native app included music scores, which constituted 6% of native app searches, and dvds, which constituted 5% of searches in this interface. the delineation of other format filters used is shown in table 5. table 5. native search format types. discussion a key finding was that format filtering is used more often in the native mobile app. however, with respect to general popularity, it is clear that during this study’s period, the responsive mobile website in vufind was used more overall. the logs showed that users accessed mobile responsive websites more often with ios devices. the analysis also showed that ios devices in the consortium are twice as popular than are other mobile devices. it was not surprising that keyword (default) searches were the prominent search type, as this trend has been observed in other search log studies in library discovery systems. however, when query length was examined, it was somewhat surprising to see that the query length in the native app were longer than were those in the average responsive app. the average responsive app users’ mean query length was 4.2 words, while that of native app users was 6.5 words. it is unclear whether this is because native app users may be more likely to cut and paste words and full citations more often than do responsive web users, but this may be one factor to consider in the longer mean search terminology. one additional finding in this respect was that the data showed no overlap in popularly-occurring phrases or search terms, which underscores the finding that the mobile interface type may play a role in whether and what users search for in the library catalog. search types varied by interface. the native app seemed to make greater use of format filtering than did the respective responsive app format filters. however, after the search interfaces were compared side by side (figure 11), it is clear that the native app has prioritized filtering in the drop-down menu, but the responsive mobile site has filtering by format at the bottom of a results page, thereby precluding its use until the user reaches the bottom of the page. figure 11. side by side comparison of the mobile interfaces. the native app is on the left, the responsive mobile website on the right. placing the format filtering in a more prominent position in the responsive interface is recommended, because, as the native search logs show, library patrons use format filtering when it is displayed prominently. further, from the information retrieval standpoint, format filtering helps users find the exact item for which they are looking when searching the mobile interface. the search log analysis does raise the question of whether the native interface perhaps is used by some segment of the library user population that prefers to search for known items and format types. therefore, the question of what the native app seems to offer to different segments of the user population could be a focus of further qualitative study. conclusion implications for library mobile app design in the consortium include understanding the mobile search features that users find helpful. we can ascertain several areas to improve the responsive interface and understand its significance, as more users access mobile content designed responsively than through native access. it was clear that native access seemed to support more of the catalog features used, such as format filtering. future focus groups that study why some users chose to install the native app, rather than use the mobile responsive vufind page, would help to expand the quantitative data collected in this study with qualitative feedback to better understand several of the new questions introduced by this study. references berg, a. (2010). browser vs. app: which is better? wireless week, december, 2010. blummer, b., & kenton, j. (2016). academic libraries’ mobile initiatives and research from 2010 to the present: identifying themes in the literature. in l. briz-ponce, j. a. juanes-méndez, and f. j. garcía-peñalvo (eds.), handbook of research on mobile devices and applications in higher education settings. hershey, pa: igi global, 118-139. https://doi.org/10.4018/978-1-5225-0256-2.ch006 carpineto, c., mizzaro, s., romano, g., & snidero, m. (2009). mobile information retrieval with search results clustering: prototypes and evaluations. journal of the american society for information science and technology, 60(5), 877-895. https://doi.org/10.1002/asi.21036 chanlin, l-j. & hung, w-s. (2016). usability and evaluation of a library mobile web site. the electronic library, 34(4), 636-650. https://doi.org/10.1108/el-07-2015-0119 charland, a., & leroux, b. (2011). mobile application development: web vs. native. communications of the acm, 54(5), 49-53. https://doi.org/10.1145/1941487.1941504 farrell, s. (2017). search-log analysis: the most overlooked opportunity in web ux research. https://www.nngroup.com/articles/search-log-analysis/ hahn, j., ryckman, b., & lux, m. (2015). topic space: rapid prototyping a mobile augmented reality recommendation app. code4lib journal, 30, http://journal.code4lib.org/articles/10881 jensen, b. j. (2006). search log analysis: what it is, what’s been done, how to do it. library & information science research, 28(3), 407-432. https://doi.org/10.1016/j.lisr.2006.06.005 kim, b. (2013). responsive web design, discoverability, and mobile challenge. the library mobile experience: practices and user expectations, library technology reports, 49(6), 29-39. http://dx.doi.org/10.5860/ltr.49n6 liu, y., liu, x., ma, y., liu, y., zheng, z., huang, g., & blake, b. (2015). characterizing restful web services usage on smartphones: a tale of native apps and web apps. 2015 ieee international conference on web services, new york, ny, 337-344. https://doi.org/10.1109/icws.2015.53 peters, t. a. (1993). the history and development of transaction log analysis. library hi tech, 11(2), 41-66. https://doi.org/10.1108/eb047884 sinclair, s., & rockwell, g. (2016). voyant tools. http://voyant-tools.org/ tidal, j. (2017). one site to rule them all, redux : the second round of usability testing of a responsively designed website. journal of web librarianship, 11(1), 16-34. http://dx.doi.org/10.1080/19322909.2016.1243458 tu, y-f., & hwang, g-j. (2018). trends of library-associated mobile learning based on a review of academic studies published from 2007–2016. the electronic library, 36(5), 875-891. https://doi.org/10.1108/el-06-2017-0138 wong, s. h-k. (2012). which platform do our users prefer: website or mobile app? reference services review, 40(1), 103–115. http://doi.org/10.1108/00907321211203667 about the author jim hahn is an associate professor in the undergraduate library at the university of illinois. his research into technology-enhanced learning has led to many software development projects within library settings and provides unique insights into new student’s expectations and needs and helps inform the work that he does as the orientation services and environments librarian for undergraduate students at the university of illinois. he founded and manages the minrva project (https://minrvaproject.org) and currently serves as project pi for a research & development grant funded through the university of illinois campus research board, entitled, “information and environment: integration of an iot-powered recommender system within the folio open source platform,” the aims of which are to incorporate internet of things functionality into a folio wayfinder application ( https://github.com/minrva/ui-wayfinder) and make the resulting software available in the open source. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – an interactive map for showcasing repository impacts mission editorial committee process and structure code4lib issue 36, 2017-04-20 an interactive map for showcasing repository impacts digital repository managers rely on usage metrics such as the number of downloads to demonstrate research visibility and impacts of the repositories. increasingly, they find that current tools such as spreadsheets and charts are ineffective for revealing important elements of usage, including reader locations, and for attracting the targeted audiences. this article describes the design and development of a readership map that provides an interactive, near-real-time visualization of actual visits to an institutional repository using data from google analytics. the readership map exhibits the global impacts of a repository by displaying the city of every view or download together with the title of the scholarship being read and a hyperlink to its page in the repository. we will discuss project motivation and development issues such as authentication with google api, metadata integration, performance tuning, and data privacy. by hui zhang and camden lopez project motivations scholarsarchive@osu, the institutional repository of oregon state university (osu) built using dspace, is the central space at the university to share and preserve scholarly works including electronic theses and dissertations (etd), faculty publications, and datasets. quantitative measures of impact such as number of downloads by article are provided through scholarsarchive (sa@osu), and university administrators can request from the library a usage report for items deposited in sa@osu. however, librarians working with sa@osu face growing demands to move beyond mere reporting and to showcase research visibility and impact. for instance, authors want to know when and where their articles are being read so they can have a fresh view of their research impact. in addition, university administration would benefit from a visual tool that demonstrates the purpose and impact of the institutional repository (ir) to donors and the general public. after collecting comments from stakeholders of sa@osu, we started looking for existing showcasing tools for dspace and had no luck to find one. one of our colleagues then recommended the readership activity map from bepress digital commons because it was a good match for our requirements. however, after reviewing that application, we decided to develop a new readership map for a few reasons: the readership activity map by bepress is proprietary software and available only to institutions that are paid subscribers of digital commons, an ir platform developed by bepress. we wanted to use either publicly accessible or open access software for the development of the new readership map. we wanted to decouple the new readership map from the underlying ir platform. overall, the development goals were to create a readership map that is feature-rich, open source, and not tied to any specific repository platform. the first step of the map development was to choose the data source and decide how to integrate sa@osu metadata. tracking repository traffic with google analytics elasticsearch vs. google analytics sa@osu uses elasticsearch as the engine to provide usage statistics instead of solr, which is the default engine of dspace. we chose elasticsearch because of its capacity to process large amounts of data quickly. sa@osu is a large ir with traffic of about 7 million legitimate visits (including both views and downloads) to sa@osu in 2016. elasticsearch is shipped with dspace 3.0 as an optional solution for usage statistics. it is turned off by default and available only with an xmlui interface. however, turning on elasticsearch statistics involves editing only two configuration files. once deployed, it can collect information about user requests such as ip address, user agent, and dspace object ids. using the elasticsearch index as the data source for the readership map would have violated our goal of decoupling the map from the underlying repository platform. another problem with the elasticsearch data is that the usage statistics are subject to contamination with web spider and bot traffic. dspace provides a tool for filtering out such traffic, but the tool is available only for solr. sa@osu administrators must clean the elasticsearch statistics manually. the readership map needs clean data to be available immediately because it displays readership in near-real-time. google analytics (ga), the free service offered by google for tracking and reporting web traffic, is an independent platform that is more resistant to spider and bot traffic. because ga uses javascript to collect and transmit tracking data, spiders and bots which do not process javascript will not trigger any tracking info to be recorded by ga. furthermore, ga provides a suite of features for spam and bot filtering at the administrator dashboard. in the end, we chose ga as the data source for the readership map because it can provide consistent and reliable usage statistics. additionally, modern ir platforms such as dspace 4.0 and hydra have adopted ga for recording and reporting usage statistics. next, we will describe how we track and report ir usage with ga and discuss how we address data privacy concerns. tracking article views and downloads a javascript library provided by google, analytics.js (latest version) or ga.js (legacy version), must be included on each page for which usage statistics are to be collected. for dspace-based repositories using the xmlui user interface, the library should be added to page-structure.xsl in dspace-xmlui/src/main/webapp/themes/mirage/lib/xsl/core, where mirage is the theme in use by xmlui, as in the following snippet. <script type="text/javascript"><xsl:text> (function() { var ga = document.createelement('script'); ga.type = 'text/javascript'; ga.async = true; ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/analytics.js'; var s = document.getelementsbytagname('script')[0]; s.parentnode.insertbefore(ga, s); })(); </xsl:text></script> to track page views with the included ga library, we also add the following to page-structure.xsl. var _gaq = _gaq || []; _gaq.push(['_setaccount', 'ua-xxxxxxxx-x']); _gaq.push(['_setdomainname', 'ir.library.oregonstate.edu']); _gaq.push(['_trackpageview']); the argument following ‘_setaccount’ needs to be a google analytics tracking id, and the argument following ‘_setdomainname’ is the domain of the ir in which tracking will take place. tracking file downloads in dspace with ga is more difficult because we have to record the download activity as an event triggered by a user clicking on view/open in a repository item page. the following snippet, added to item-view.xsl in dspace-xmlui/src/main/webapp/themes/mirage/lib/xsl/aspect/artifactbrowser, serves this purpose. <xsl:template name="view-open"> <a> <xsl:attribute name="onclick"> trackdownload(this) </xsl:attribute> <xsl:attribute name="href"> <xsl:value-of select="mets:flocat[@loctype='url']/@xlink:href"/> </xsl:attribute> <i18n:text>xmlui.dri2xhtml.mets-1.0.item-files-viewopen</i18n:text> </a> <script type="text/javascript"> <xsl:text> function trackdownload(link) { _gaq.push(['_trackevent', 'bitstream', 'download', link.href]); settimeout('document.location = "' + link.href + '"', 200); return false; } </xsl:text> </script> </xsl:template> the first part of the snippet, between <a> and </a>, adds the trackdownload method which is triggered when the file is downloaded by a user. the second part of the snippet specifies that the trackdownload method will report the visit to ga as an event with ‘bitstream’ as the event category, ‘download’ as the event action, and the uri for the downloaded file as the event label (see https://developers.google.com/analytics/devguides/collection/analyticsjs/events). data privacy the fact that sa@osu uses ga for usage tracking raises issues of protecting visitor and data privacy. one way to address the privacy concern is to take advantage of a ga option which partially anonymizes the visitor ip addresses stored and used by google. however, according to google, this reduces the accuracy of the geographic data that are calculated from the ip address (“greater choice and transparency for google analytics”, 2010). this is a significant problem for our map because knowing the locations of readers is a major requirement from our repository stakeholders. we prefer to address data privacy concerns by providing transparency about our data collection practice. osu library is drafting a data privacy policy to protect users’ rights of privacy and confidentiality, and part of the policy is telling users what information the library will collect for recording and reporting. for example, the privacy policy will state that the library websites (including sa@osu and the readership activity map) collect the following information from user requests: domain, country, ip address dns/hostname date, time user agent resource (title, uri) in sa@osu that has been used the collected data are exclusively for evaluation of site design to best serve user needs. map implementation the implementation of the readership map will be described in detail, but here is a brief overview. the map itself is implemented in javascript using the google maps javascript api. the map gets readership data from a ruby on rails (rails) application we created to serve as an intermediary between the map and google analytics. readership data initially flows from our repository pages to ga when users visit the repository and view or download items. then, when a user loads the readership map and the map requests data from the rails application, the data flows from ga to the map through the rails application. google analytics data the readership map uses the following data collected by ga for each page view or download on an sa@osu page: page uri and title, hour and minute when the page was visited or a download occurred, and location (city, region if applicable, and country) of the user. the page uri allows us to provide a hyperlink to the repository item page within the map. the page title (specified by the “title” element in the page’s html) happens to be the item title for sa@osu repository items. that is, an item in the sa@osu repository titled “abc” will have a page within the sa@osu web interface with html title “abc”, and that title gets recorded by google analytics as part of the data collected for a page view or download event. so the ga page title data provide us with a convenient way to display a hyperlinked item title in the map. we use the hour and minute to simulate real-time display of the readership data. we found that the most recent ga data became available in batches, approximately every 5 to 10 minutes. repeated queries for the most recent data would yield the same data set for 5 to 10 minutes until a query returned an updated set which had new data for the past few minutes. subsequent queries would return those same data for 5 to 10 minutes until the next update, and so on. if we displayed data as it became available, the resulting display would be uneven over time. as described in the google map section, our solution was to use the hour and minute associated with each view or download to simulate real-time display. the location of a user is determined by ga from the user’s ip address and is only an approximation. we filter out any data for which there isn’t a valid location, indicated by a value of “(not set)” for the city. ga provides latitude and longitude coordinates for the center of the user’s city, and we use those to place city markers on the map. for users inside the united states, the region given by ga is the unabbreviated state name. for readership associated with those users, we use the city and state to label the location on the map. otherwise, we use the city and country. rails application the rails application for the readership map is open source and made available through github. this intermediary application between google analytics and the map was necessary for the ga authentication and authorization process. to access ga data, it is necessary to authenticate one’s identity as a user with authorization to access the requested data. a person can provide authentication when accessing data through a web browser by signing into a google account. software applications must provide authentication through a google service account. a secret key stored in a file provides the essential authentication information. we could not put such a secret key in the readership map code because it could be downloaded and read by anyone who loaded the map. instead, we created a rails application running on an osu server to access the ga account on behalf of the readership map. another reason for using an intermediary application between ga and the readership map is to improve efficiency. since the ga data get updated every 5 to 10 minutes, it would be wasteful to have every instance of the readership map generating its own queries to ga, most of which would be redundant. with the intermediary application, we can limit the number of ga queries to at most one every 5 to 10 minutes, store the data temporarily on the server, and serve the data quickly in pre-processed form to readership map instances. we chose to implement the server application in ruby on rails, a web development framework built on the ruby programming language. rails was ideal for our purpose because it is designed for creating web applications with minimal configuration. additionally, because we are going to renovate sa@osu using a rails-based framework called hyrax, there was already knowledge and resources available at osu library for using rails. the rails application temporarily stores readership and supplemental data in a mysql database. the main table in the database, containing all of the readership data used by the map, is readers. we use another table, locations, for storing location information (city name, region, etc.). the location information has to be downloaded separately from the readership data because ga limits the number of fields (dimensions) one can request in a query to 7, which is not enough to get all readership and location data in one query. the locations table helps match the results of the separate readership and location queries to form complete records of readership. the rails application is fairly simple, with most of the functionality defined in app/controllers/data_controller.rb. when the readership map requests data, if the rails application has retrieved ga data less than 10 minutes ago, then the saved data are simply sent to the map in json format. otherwise, the rails app queries ga for all of the current day’s data (with separate queries for page views, downloads, and location information), does some minor processing of the results, updates the database, and sends data to the map. the following snippet shows how we construct the ga query for readership data. the terms “metrics” and “dimensions” (“dims”) are the google analytics terms for quantitative measurements and non-quantitative attributes, respectively. when we request, for example, the metric “ga:pageviews” and the dimensions “ga:hour”, “ga:minute”, etc., ga finds all unique combinations of the dimensions we requested, and for each combination calculates the total number of pageviews. the results returned by ga have one row per unique combination of the dimensions. the results are calculated for a specified date range, which in the code snippet is given as the first arguments of query (from “today” to “today”). if activity == 'download' metrics = 'ga:totalevents' else metrics = 'ga:pageviews' end dims = 'ga:hour,ga:minute,ga:cityid,ga:pagetitle,ga:hostname,ga:pagepath' ... rows = query('today', 'today', metrics, dims, filters, sort, max) although hour and minute data are available for pageviews and events in ga, it isn’t possible (or at least isn’t easy) to query for a range of times. one can query only for a range of dates. so with each query, we get the whole current day’s data, most of which will have been already downloaded in the last query. this inefficiency is not much of a concern because we query once every 10 minutes at most, but there may be a way to avoid the redundant downloads. when we send data from the rails application to the readership map, on the other hand, we send only the most recent data. selecting for a range of times is straight-forward once the data have been stored in a mysql database with a timestamp as one field. having such convenient database functionality is another advantage of using a server application to go between ga and the map. authentication and authorization for querying google analytics our rails application queries google analytics for sa@osu usage data using the core reporting api. each query requires authentication and authorization via oauth. setting up authentication and authorization takes several steps. in the google developers console, enable the analytics api, create a service account, and obtain a credentials file within the relevant project (readership-map in our case). more detailed instructions are at https://developers.google.com/api-client-library/ruby/auth/service-accounts. note the service account id (an email address ending in “gserviceaccount.com”). in the admin section of the google analytics ui, for the google analytics view containing the data that the readership map will use, add the service account id to the list of users in user management. grant the service account id “read & analyze” permission (see figure 1). put the credentials file for the service account in a secure place where the rails application can access it (on an osu server, in our case). figure 1. granting the readership-map service account access to google analytics data. the googleauth ruby gem provides a convenient method, get_application_default, for loading an oauth credentials file into a rails application. we currently load the credentials a slightly different way using environment variables, but in the following snippet we show the get_application_default method. the google-api-client gem provides a ruby interface with the analytics core reporting api. putting these together, the query function used at the end of the last code snippet acquires authorization and queries ga as follows. require 'googleauth' require 'google/apis/analytics_v3' def query(start_date, end_date, metrics, dimensions, filters, sort, max) service = google::apis::analyticsv3::analyticsservice.new scopes = ['https://www.googleapis.com/auth/analytics.readonly'] service.authorization = google::auth.get_application_default(scopes) service.authorization.fetch_access_token! service.get_ga_data(env['ga_profile_id'], start_date, end_date, metrics, dimensions: dimensions, filters: filters, sort: sort, max_results: max) do |result, err| ... return result.rows end google map the map’s javascript implementation is hosted by the rails app in app/assets/javascripts/rm.js.erb. we instantiate and initialize a google map using the google.maps.map function from the google maps javascript api. then the script retrieves recent readership data in batches from the rails application every 10 minutes, using the jquery.get() function to asynchronously send requests. the data consist of the times, locations, page titles, and uris associated with the batch of page views and downloads, and are transferred in json format. we simulate real-time display of the data by thinking of a virtual clock running 30 minutes behind the actual time (the delay and other timing parameters are configurable). each minute, we update the virtual clock and display pageviews and downloads whose timestamps have been reached or passed by the virtual clock. the 30 minute delay is intended to prevent the virtual clock from skipping over data which ga has taken an unusually long time to process. var currenttime = new date(); virtualtime.setmilliseconds(virtualtime.getmilliseconds() + math.round(currenttime lastdisplaytime)); ... while (datastack.length > 0 && datastack[datastack.length 1].time <= virtualtime) { todisplay.push(datastack.pop()); } when there is more than one page view or download to add to the map in a single display interval, we try to even out the display of the data by dividing the interval into equal parts and adding one page view or download at-a-time. if (displaystack.length > 1) { displayonetimer = setinterval(displayone, mintoms(displayinterval()) / displaystack.length); } displaying a page view or download means creating a marker (using google.maps.marker) on the reader’s city if one doesn’t already exist, then animating the marker for a few seconds and updating the count of page views or downloads associated with the city. the map keeps track of page titles and uris for the most recently viewed and downloaded items for each city, and we use those to fill an info box which opens when a user selects a city marker. users can click on a link to the most recently viewed/downloaded item, opening a new browser tab with that item’s page in sa@osu. figure 2. readership map with the info box for a city opened. next steps we deployed the readership map in early 2017 and embedded the map on the homepage of sa@osu. figure 3. deployment of readership map on sa@osu homepage. we added a section called readership map to the home page with a brief introduction to the application and a clickable screenshot of the map. interested users can click the image and will be redirected to the readership map application. this is accomplished by editing news-xmlui.xml in dspace/config with a new “div” section. <div> <head>readership map</head> <p>the map of readership activity for scholarsarchive@osu exhibits the global impacts of an ir by displaying the city and activity, such as view of download, of every visitor together with title and url link of the scholarship that is being read.</p> <figure source="themes/mirage/images/slides/readershipmap.png" target="http://readershipmap-dev.library.oregonstate.edu/"></figure> </div> as osu libraries are planning to migrate sa@osu from dspace to the hydra framework with a rails-based front-end, the immediate next step is to investigate how to embed the readership map into a rails view so that the live map can be viewed within sa@osu rather than on a separate webpage. because we draw our readership data from ga without interacting directly with the repository, the readership map will require few or no modifications as the repository moves from dspace to hydra. future plans also include a usability study to explore whether and how an interactive visualization like the readership map improves user satisfaction with an institutional repository. references greater choice and transparency for google analytics [internet]. [updated 2010 may 25]. [cited 14 march 2017]. available from: https://analytics.googleblog.com/2010/05/greater-choice-and-transparency-for.html author biographies hui zhang (hui.zhang@oregonstate.edu) is the digital application librarian for oregon state university libraries and press. he works with a team of developers to keep the digital repositories with the latest technologies and create software tools and solutions for the library and its patrons. camden lopez (camden.lopez@oregonstate.edu) is a graduate student in the department of statistics at oregon state university. he worked on the scholarsarchive@osu readership map as a part-time programmer in osu libraries. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – diy doi: leveraging the doi infrastructure to simplify digital preservation and repository management mission editorial committee process and structure code4lib issue 38, 2017-10-18 diy doi: leveraging the doi infrastructure to simplify digital preservation and repository management this article describes methods for how staff with modest technical expertise can leverage the doi (digital object identifier) infrastructure in combination with third party storage and preservation solutions to build safer, more useful, and easier to manage repositories at much lower cost than is normally possible with standalone systems. it also demonstrates how understanding the underlying mechanisms and questioning the assumptions of technology metaphors such as filesystems can lead to seeing and using tools in new and more powerful ways. by kyle banerjee and david forero introduction the needs that digital repositories serve are at least as diverse as the objects they contain, but virtually all libraries must provide: preservation, to store digital items securely and reliably; context, to maintain descriptive, structural, and administrative (fixity, integrity, and provenance) metadata; and continuity, to migrate objects and metadata across systems when technology and needs change without imposing complexity on the user. (this list is adapted from gladney, h. m. (2006). principles for digital preservation. communications of the acm, 49(2), 111-116. doi:10.1145/1113034.1113038) it’s expensive and time-consuming to perform these tasks within individual repositories because methods are particular to each installation. this article describes a simple but powerful method for supporting these functions using the doi system. although the implementation details will vary between individual libraries, the method is simple enough that it can be used in manual as well as heavily automated environments. we describe a way to use the doi system with amazon’s glacier service to protect assets and metadata currently provided to users by our omeka and bepress systems, but it could be used with virtually any system. libraries have used identifiers to improve services for decades. oclc numbers, isbn (international standard book numbers), and issn (international standard serial numbers) simplify ordering, processing, and requesting. as people have come to rely more heavily on electronic information, other identifiers such as pmcids (pubmed central ids) and dois (digital object identifiers) have become more relevant. these identifiers are typically used as unique strings to simplify locating resources and serve as matchpoints to facilitate maintenance. though important, these uses represent only a portion of the value identifiers can deliver. in fact, some identifiers have the potential to deliver functionality that goes well beyond a findable string in a search engine or matchpoint for maintenance tasks. for example, dois can easily track a wide variety of administrative, structural, descriptive, and technical metadata. unlike other types of identifiers, they can connect people directly to resources, track derivative and archival copies, and be easily maintained using automation. combining the power of dois with third party storage solutions, such as aws, allows repository managers to build systems that are relatively cheap and easy to use, maintain, and migrate to new platforms. because of the high level of redundancy these systems provide, better levels of protection are introduced compared to those achieved with local solutions. what are dois and how do they simplify repository administration? nominally, a doi is a string of case insensitive alphanumeric characters that uniquely and permanently identify a digital or physical item. dois have a simple structure designed to allow for independent creation without the risk of overlap. consider the doi 10.6083/m4nz86rr figure 1. the doi structure (enlarge) the portion before the slash is known as the prefix which is composed of a directory indicator and registration agency (ra) which services as namespace so suffixes do not overlap. the shoulder also provides a namespace function and allows the registrant to create as many namespaces as necessary. the length and format of both the shoulder the suffix are arbitrary and can contain any textual characters. this system allows dois to be generated in a distributed environment without any risk of collision. the doi standard is defined by iso 26324 (https://www.iso.org/standard/43506.html). a doi can be assigned to only one object, and each object can have only one doi. however, dois can be issued for any entity that needs to be identified at any level of granularity. for example, a dataset could have its own doi, subsets within that dataset could have their own dois, and individual files with the subsets could also have their own dois. dois differ from other identifiers such as isbns in that dois: are associated with metadata that can be changed as the object changes. are designed to be created and maintained independently in a distributed environment. are intended to be resolvable so they connect people directly to the resource. can be assigned to any digital object rather than a specific type of object, such as books (isbns), journals (issn), and articles (pmcids). can be easily created and maintained with automation. these distinguishing factors make dois much more useful than other identifiers that functionally are little more than unique strings. the ability to store metadata [1] that can be modified using automated processes makes it possible to simplify migrations in the following ways: dois are platform agnostic because they operate and store metadata independently. therefore, developing interfaces between each type of repository and the doi system is all that is necessary, rather than creating tools for each specific migration. changes to any specific type of metadata from a single element or combination of elements can be made without retrieving the object or the entire metadata record. why use third party storage providers? many institutions seek out “cloud” services as they are perceived to be safer and more reliable than locally provided services. however, despite high profile organizations trying to create formal definitions [2] in recent years, from its inception the term “cloud computing” is virtually meaningless from a technical perspective and is really a marketing term [3] — the way many people use the term is to mean a service is provided somewhere else by someone else and delivered over a network. as such, they aren’t inherently better than locally hosted options so some institutions have chosen to maintain their own local clouds. a local cloud might imply one or more clusters of computers that offer some sort of redundancy or fault tolerance. having said that, vendor maintained cloud services on encrypted systems hosted in secure data centers may prove cheaper, easier to use, more secure, and more reliable than what can be provided locally — particularly if a major vendor commits significant resources to develop services that address very specific use cases. for example, amazon, dropbox, google, microsoft, box, and other entities offer robust storage solutions. the capabilities of these and other services differ, so selection should be based on specific needs. for our purposes, we examine amazon’s s3 and glacier services (aws). aws was chosen because glacier offers functionality that is of particular interest to libraries — namely secure offline storage intended for permanent archives, thus guaranteeing file integrity that is cheaper than other alternatives. s3 is an online storage service that can be used to deliver files of any type over the web. s3 and glacier offer the following benefits that are difficult to achieve using local resources alone: robust application programming interface (api) that allows automation of any function the service provides. file integrity guaranteed through cryptographically signed transfers supported by monitoring and repair. excellent access controls and protection against modification and deletion. meets stringent security and privacy requirements. geographically distributed and high availability. scales as needed; only resources consumed require payment. what’s the connection between dois and third party storage providers? in three words: metadata, reliability, and portability. dois allow you to store virtually any kind of metadata. this means you can configure the metadata so the doi directs the user to an access copy of a resource. contains information to retrieve derivative and preservation copies as well as locations on local systems. contains descriptive, structural, and administrative information needed to use and manage the resource. aws is particularly well-suited for this task. although most interfaces to their s3 service look like a filesystem, s3 is an object store where the objects can be given names that look like directory paths. consider the following: figure 2. displaying objects as files and folders in amazon s3 (enlarge) what appears to be two files named photo1.tiff and photo2.tiff are not files and what appears to be a directory named folder1 containing them is not a folder. rather, folder1 does not exist, and the only thing that exists are two objects with the 20 character names “/folder1/photo1.tiff” and “/folder1/photo2.tiff”, respectively. these objects have no relationship to each other as far as s3 is concerned. the software interface makes it appear that folder1 exists and that it contains two files, but this is an abstraction which has the sole purpose of providing a familiar metaphor to users. the distinction is important because it has implications for operations such as renaming directories or files because directories and files are abstractions. commands that appear to rename a directory actually copy all the objects that begin with the same string to new objects and then delete the old ones. on a personal computer, such an operation is not a matter of concern. however, when working with large numbers of objects in a data center designed to meet a different set of needs, this can be an expensive and time-consuming operation. it also means it is impossible to download a “directory.” rather, all objects that begin with the same string must be retrieved individually, which also has performance cost and data structure implications. likewise, all operations that presume relationships derived from filesystem relationships must be synthesized by other software acting through amazon’s api. the lesson here is that it’s risky to think of all systems in terms of familiar metaphors (e.g., object stores are different than filesystems and need to be interacted with differently). relationships between objects are better maintained using metadata which allow any desired relationship to be expressed rather than file hierarchies, which are inflexible and only allow for the expression of specific types of relationships. virtual hierarchies expressed with metadata can be converted to a traditional filesystem structure and can be more readily analyzed, modified, or extended as necessary. even on traditional filesystems, files and directories don’t exist in a literal sense. rather, both are metaphors that make it easier for humans to interact with digital objects. directory paths and file properties (such as the file name) are really metadata about an object that is kept separate from the object itself in a way that makes working with them more user friendly. by storing all object metadata in a single place (e.g., within doi metadata) rather than splitting it between databases and filesystems, libraries are able to manage resources more easily, cheaply, and with more flexibility. for example, within metadata, a “file” can have as many names and belong in as many hierarchies as desired, rather than just exist in a single place like a real paper file. leveraging the complementary functionality of dois and third party storage providers, such as aws, libraries can simplify repository administration and lower costs by: storing descriptive, structural, and administrative information in the doi metadata where any element or combination of elements can be easily maintained and remains unchanged if the library moves to another repository platform. using the doi as a key to derivative copies in s3. different types of derivatives are stored in special purpose s3 buckets. using the doi as a key to a special bucket that connects to glacier where the preservation copy is stored. syncing doi metadata with repository descriptive, structural, and administrative metadata, which protects it, makes it more actionable, and simplifies migrations. be aware that doi registration agencies’ graphical interfaces are designed to capture and display certain types of metadata. if you need metadata supported by the underlying architecture that the interface was not designed to handle, you may find that you can only create, modify, and extract it via an api. a practical example for purposes of using dois as part of a preservation strategy, we have the following expectations about items needing to be preserved: they may or may not have been processed. they may or may not be publicly accessible. they may or may not have metadata associated with them. the metadata associated with them may or may not change after they are archived. they may or may not have dois already assigned to them when they are processed. it may be necessary to process thousands of files at once using automation. they can be of any size or complexity. they must be preserved unmodified. figure 3 demonstrates the process for using the doi system in tandem with a third party storage provider, such as amazon. the basic mechanism is simple and has the following attributes: if an object lacks a doi, one is automatically created for it. objects are placed in zip files where the filename is the doi. an underscore is substituted for the slash in the doi to simplify processing and retrieval on regular filesystems. an alternate id is added to the doi metadata that matches the full file path to the object. if an object lacks a title, the title in the doi metadata is the full file path. figure 3. preparing objects for archiving with dois (enlarge) in figure 3, we assume that an automated or manual process put file1.tiff in the zip archive 10.1234_a1b123.zip and assigned the full file path as an alternate identifier. as such, this object requires no further processing and can be transferred directly to the archival storage system — in this case, amazon glacier. meanwhile, file2.tiff, file3.tiff, and file4.tiff have not been processed, so dois are automatically created for them with the full file path stored both as the alternate identifier and title. if these are processed at some later date, the metadata for the doi will be modified — for example, a real title will replace the file path which serves as a placeholder. these files are automatically zipped using the doi as the archive name. all doi zip files are then transferred to a special amazon s3 bucket where they are later transferred to glacier. the intermediate s3 bucket is used rather than transferring items directly to glacier, because this mechanism allows files to be recovered using dois instead of glacier ids which would need to be managed separately. zipping the objects lowers storage and bandwidth costs while providing a predictable mechanism (namely the doi itself) for accessing them without altering any file attributes. if different types of access copies, thumbnails, etc. are desired, these can be created and referenced through doi fields. any file or combination of files can be recovered from glacier using the doi keys in a gui interface as shown in figure 4 or via api. figure 4. recovering an archive from glacier (enlarge) this scheme lacks the compelling metaphor of a traditional file system. however, both systems are conceptually simple and have been designed with an eye to compatibility into the future. we use the doi system and glacier to protect assets and metadata in our omeka and bepress systems, and a migration to samvera (https://samvera.org) next year will be made easier because we can migrate metadata directly from the doi system rather than developing separate methods to migrate from each repository system. what limitations and risks are there to using this method? administrative limitations to use the doi system, you must work through a registration agency (ra). as of this writing, there are only ten registration agencies, each of which determines its own policies, so methods described here for the datacite ra may require modification when used with other ras. dois can theoretically be assigned to anything, but in practice are assigned to documents, articles, datasets, and other specific types of information resources. this means the metadata schemas supported by the doi ras are optimized to support specific use cases that may be difficult to adapt for significantly different purposes. regarding the datacite metadata schema, it is extensible, but it is heavily influenced by a bibliographic data model and academic citation needs. in practice, data fields can be used for many purposes. for example, an alternate identifier field is the natural location to store file paths that indicate where an item belongs in a structure. multiple options exist for maintaining metadata not explicitly provided for in the schema: metadata may be encapsulated in other fields, such as description, which limits neither content nor length; however, this complicates use, analysis, and maintenance of metadata. storing wildly different kinds and amounts of data in the same field makes it difficult to process metadata efficiently. metadata may be kept in a totally separate file, such as an archival information package (aip), and referenced by another doi. this too complicates use, analysis, and maintenance of metadata. having a separate location and set of files to parse and correlate will introduce new hurdles to overcome for systematic processing. our tests indicate that at least one ra supports arbitrary metadata. this provides extraordinary functionality, ease of use, and maintenance. however, relying on such functionality may be risky as long-term support is in question. security limitations network-based archival solutions present inherent security challenges — objects are not safe if a program or human can destroy them. the safety of archived objects can be ensured by implementing access controls that allow objects to be created but not modified or deleted. destructive credentials and cryptographic keys must be kept in a safe and unconnected space and protected by robust procedures. in the event object deletion is desired as a part of normal workflow, it can be handled through s3 and glacier’s transparent and easily auditable retention schedule functionality. summary we have presented you with a lot of detail and background. this may seem like yet another scheme to manage files, but certain aspects of this scheme make it especially noteworthy. first, since the advent of personal computing, the dominant metaphor for managing digital objects mimics standard paper filing systems where documents are placed in folders. from a technical perspective this is a strict and unique hierarchy of categorization combined with limited strings of text as identifiers. this is a helpful metaphor, but it doesn’t reflect what actually happens on a computer, nor is it robust enough to support some of the more powerful ways computers can handle data. computers operate more like databases than hierarchical file systems despite the dominance of that particular abstraction. databases allow relationships to be defined arbitrarily and non-exclusively. this means that a single digital object can belong to many groups of digital objects. filesystems have attempted to provide this functionality with pointers, symbolic links, shortcuts, and aliases. pointers allow an object to be part of multiple groups, but the relationship tends to be only one way. the pointer lets us find the file but the file does not let us find the pointer. second, separating files from the method by which they are categorized and organized is key to how our scheme works. amazon glacier excels at reliable, simple, and inexpensive storage of data, but it is neither fast nor dynamic. glacier’s simplicity improves longevity by not tethering objects to a complicated and perhaps ephemeral file structure. rather, our method separates storage from the filesystem (we store the “filesystem” along with other critical metadata within dois). by separating where we store files and the data about the files, we leverage the strengths of both systems. at the ohsu library we are currently using this method to retrospectively organize and protect archival, scholarly, and image collections. by storing all our metadata and objects using the same predictable mechanism, we hope to simplify a migration from our omeka and bepress systems to samvera next year as well as position us for future migrations. this scheme lacks the compelling metaphor of a traditional file system. however, both systems are conceptually simple and have been designed with an eye to compatibility into the future. it is very probable that these two technologies will be easy to integrate into whatever future technologies present themselves. notes [1] metadata elements as well as the container used to transmit them depends on the registration agency (ra) and the mechanism for populating the doi system (e.g., object within api, file (xml/json/text) transmitted via http post, etc.) [2] mell, p., & grance, t. (2011, september). the nist definition of cloud computing. doi:10.6028/nist.sp.800-145. [3] regalado, a. (2011, october 31). who coined ‘cloud computing’? – mit technology review. retrieved from https://www.technologyreview.com/s/425970/who-coined-cloud-computing/ about the authors kyle banerjee (banerjek@ohsu.edu) is the digital collections and metadata librarian at ohsu library. david forero (forero@ohsu.edu) is the technology director at ohsu library. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – building strong user experiences in libguides with bootstrapr and reviewr mission editorial committee process and structure code4lib issue 48, 2020-05-11 building strong user experiences in libguides with bootstrapr and reviewr with nearly fifty subject librarians creating libguides, the libguides management team at notre dame needed a way to both empower guide authors to take advantage of the powerful functionality afforded by the bootstrap framework native to libguides, and to ensure new and extant library guides conformed to brand/identity standards and the best practices of user experience (ux) design. to accomplish this, we developed an online handbook to teach processes and enforce styles; a web app to create twitter bootstrap components for use in guides (bootstrapr); and a web app to radically speed the review and remediation of guides, as well as better communicate our changes to guide authors (reviewr). this article describes our use of these three applications to balance empowering guide authors against usefully constraining them to organizational standards for user experience. we offer all of these tools as foss under an mit license so that others may freely adapt them for use in their own organization. by randal sean harrison, ph. d. introduction at notre dame’s hesburgh libraries, one of our subject librarians’ primary roles is to curate and serve research resources to patrons. librarians had used the library-a-la-carte platform for this until it reached end-of-life in 2015. at that point, librarians at notre dame were making limited use of their library-a-la-carte guides during research consultations and information instruction, but found the usability of the platform made it difficult to update their guides with new resources frequently. an informal survey at that time found that most avoided updating their guides altogether. this typically resulted in librarians creating a single subject guide, structured as a too-exhaustive, single-page list, which was made to serve patrons regardless of situational or instructional needs. the adoption of the springshare libguides 2.0 cms (libguides) in 2015 was meant to improve librarians’ ability to serve patrons by offering a platform in which they found it easy to create and powerfully customize a variety of guides to suit a range of research and instructional needs, including general– and specific subject guides, how-to guides, reference and general purpose guides, course guides, and internal guides. soon after the launch of libguides at notre dame, i was hired and asked to chair the libguides management team (lmt) and lead the effort to port all extant guides from the library-a-la-carte platform to libguides. before migration, the lmt’s work began with the creation of a digital guides handbook, in which we outlined possible guide types, processes for guide creation, and library brand and identity standards. the handbook also offered guide authors a primer in best practices when writing for the web and offered an abridged list of design principles from the ux myths website. once the guide was finalized, the lmt used it to lead internal training on authoring in libguides for all internal stakeholders. one of libguides’ most useful features is that it enables librarians to embed direct links to print and electronic resources hosted by the library and affiliates. [1] libguides is also built on the twitter bootstrap 3.x.x mobile-ready frontend framework—a set of html interface elements and css and javascript libraries which offer an extensive set of device agnostic user interface elements. basing the libguides frontend on bootstrap, springshare ensured that libguides authors also had access to these bootstrap interface elements (e.g., accordions, alerts, tables, and tabs and pills, etc.) all of which work seamlessly when embedded directly in the libguides rich text/html box object. however, because most subject librarians at notre dame were not comfortable with writing or even editing the html or css needed for complex bootstrap interface elements, they did not immediately take advantage of the affordances and preattentive attributes afforded by bootstrap. to make it easier for librarians to use these bootstrap elements, we included in the next iteration of the guides handbook approximately 15 digital tools to build the customized code for the bootstrap interface elements we felt might be most useful to guide authors. these included accordions, alerts, buttons, labels, blockquotes, code wells, columns, jumbotrons, responsive google docs, sheets and slides, custom icons, labels, lists and media lists, panels, responsive images and videos, tables, and tabs/pills. in the following iteration of the guides handbook, these tools were removed, improved, and combined with approximately 10 more tools—including carousels, checklists, forms, font awesome icons, glyphicons, modals, popovers, and even whole pages—to create the stand-alone bootstrapr application. as guide authors began to create a wider range of guides and experiment with using bootstrapr to add powerful new interface elements, the conformity of guides to library brand and identity standards—which had existed strongly when we first worked closely with authors to port their guides from library-a-la-carte to libguides—began to suffer. the lmt recognized the need for a way to ensure both new and extant guides conformed to a basic set of standards. we settled on a minimum set of required metadata and used it to develop the reviewr application, which we used to remediate guides directly and communicate those changes to guide authors effectively and efficiently. below i will describe in greater detail the deployment of this three-part web solution which consisted of the guides handbook, and the bootstrapr and reviewr apps. guides handbook because the guides handbook contained not only style and structure requirements but eventually tools for creating added functionality in the form of bootstrap interface elements, the lmt needed to provide access to the guides handbook at authors’ point of need. we tackled this in three ways. first, we placed a permanent alert in the author’s announcements box on the user’s home page. this announcement held a link to the guides handbook, as well as an image of the location of the button to access the handbook from within the guide creation page. (figure 1) figure 1. a permanent alert linking to the guides handbook. the guides handbook offered a checklist for guide creation for each of the guide types. (figure 2) therefore, we also placed an alert on the guide creation page inside a box labeled “instructions.” (figure 3) figure 2. checklist for guide creation. figure 3. guide creation page alert and link. finally, we used the jquery framework [2] that bootstrap is built upon to prepend a guides handbook link-button to the libguides mini command bar [3] on the guide editing page. (figure 4) figure 4. link button on the libguides mini command bar. the jquery required to prepend the button to the menu was embedded at the system level, under the admin > look & feel > custom js/css menu [4]. (figure 5) figure 5. jquery required to prepend the handbook link to the mini command bar. by using all three ways of communicating with guide authors, a link to the guides handbook was made available at every point in the guide creation cycle. for maximum connectivity, a link to the bootstrapr application was also prominently located in the main navigation of the guides handbook. bootstrapr the bootstrapr app was developed to allow guide authors to leverage the improved user experience afforded by the bootstrap framework to better structure and style their content. many guide authors at notre dame typically work in the wysiwyg editor rather than the source code window. even librarians with elementary familiarity with html and css may find it daunting when presented with the level of nesting or the hierarchy of css classes required for sophisticated bootstrap elements such as accordions. (figure 6) figure 6. example of a two-panel bootstrap accordion component. this results in something i informally refer to as the “coding gap,” by which i mean something analogous to the distance between the ability to understand and the ability to speak when learning a foreign language. many students of a new language inhabit an arc wherein their ability to understand in the target language, which linguists term “comprehensible input,” at first outstrips their ability to produce in the target language, which linguists term “comprehensible output” (unc chapel hill). this phenomenon is sometimes based on the rise of an “affective filter” which chametzky describes as “the invisible psychological barrier that raises or lowers depending on a person’s stress level thereby potentially preventing or severely limiting interaction with the task at hand.” even guide authors with some limited understanding of html or css, when presented with code structures complex enough to induce anxiety, may choose to abandon using or experimenting with them if they are required to complete or further customize their code. bootstrapr was designed to address the rise of the affective filter that exacerbates this coding gap by rebalancing authors’ comprehensible output (writing code) against their comprehensible input (reading code). it does this in several ways: first, the interface to any bootstrapr tool, e.g. accordions, is designed to move the user along a clear path to completion from left to right in five simple steps. (table 1 and figure 7) table 1. ordered steps in process of building bootstrap components. step 1 for each tool, authors are offered a unique set of non-jargon / human-readable options which they use to customize the interface element. step 2 authors then follow a critical path to completion downward to the build button, which renders the code for the interface element in the code window at right. step 3 authors may then use the prettify button in the code window to indent the code, radically improving its readability for authors. step 4 the code window also uses the google sunburst theme to highlight the syntax such that the text nodes—the aspect of any bootstrap element that most users need to adjust—are white. for the most part, all other “messy” and overly complex code is colored, and can safely be ignored by authors. after each change to the code in the code window, authors can click the update example button to see the changes reflected in the example window. step 5 once authors have fully customized their element to their liking, they can click the copy button to copy the code with a single click and paste it into any libguides rich text box. figure 7. ordered steps in process of building bootstrap components showing user interface. the bootstapr tool thus allows guide authors to utilize a full range of bootstrap user interface elements, ultimately contributing to a better user experience for patrons. if bootstrapr represents a tool for expansion and experimentation for guide authors, the reviewr app represents a tool for standardizing and norming organizational identity. reviewr the reviewr app is a small app that aims to ensure that new and extant guides conform to proper metadata, brand, and identity standards. there is parity between the rubric in the reviewr app and the guide-creation checklist found in the guides handbook. authors are asked to run their own nascent guides through this checklist before submitting their guide to the lmt for approval to publish (figure 2). however, because the libguides platform allows guide authors to create and publish guides at will, sometimes inadvertently bypassing the lmt quality assurance function altogether, and because, in any case, as changes are made to extant guides over time they tend to move out of compliance with brand and identity standards, the lmt instituted periodic screening of all guides. the reviewr app was built for this purpose. it allows the lmt and its student workers to rapidly remediate new and extant guides by making changes directly to authors’ guides. making direct changes to authors’ guides obviates the problem of maintaining a complex system by which necessary changes are described to authors with the hope that they will find time to make these changes. in practice, this opens an entirely different and additional system of surveillance for guide compliance, whereby the lmt must continually badger guide authors to ensure authors are remediating their own guides. this entire additional machinery of tracking and reprimand is obviated by the approach taken with the reviewr app. reviewr is structured simply as a rubric for recording changes made to libguides, including remediating subjects and tags, ensuring an omnipresent author profile, ensuring proper information architecture and templating, and requiring human-readable urls at the guideand page-level—all of which aims to ensure a richer user experience for patrons and to ensure that changes made to templates will propagate across the libguides platform. the reviewr app also, and perhaps most importantly, allows guide remediators of the lmt to radically increase the speed and standardize the communication channel with which these direct changes are communicated to guide owners. the process is as follows. (table 2 and figure 8) table 2. ordered steps in process of communication guide remediation. step 1 select the guide author from a dropdown. the dropdown is populated from a json list. our json file is static, but such data could be read from any api, e.g., a library personnel directory. selecting the custom author option opens additional guide author name and custom email address fields. step 2 enter the guide name in the guide title field. step 3 toggle on the criterion for each type of change you’ve made to the guide. step 4 to view the email text assigned to any criterion, click the information icon next to its brief descriptor. step 5 to view where in the guide page to edit the criterion, select its numbered circle. step 6 optionally, add any additional comments. step 7 once all changes have been tracked in reviewr, click the email author button to generate an detailed email to the author. figure 8. ordered steps in process of communication guide remediation showing user interface. reviewr then creates a fully formatted email addressed to the guide author. the email thoughtfully and carefully explains each of the changes you made to the author’s guide. importantly, it also surrounds the description of the changes with scaffolding language such as politeness strategies and opening and closing salutations. (figure 9) this is especially important when student workers are using reviewr. before creating reviewr, we had found that when a student unknown to a guide author emailed them to tell them they had made changes to the author’s guide, typically without significant context, this introduced sizeable confusion in the process of remediating guides. reviewr solved this problem in several ways. first, while the email it creates is sent from the student’s email account, the email itself is formally ‘authored’ by the lmt. the reply to field returns any reply to the lmt listserv, and not to the student worker. finally, the lmt listserv is cc’d whenever an email is sent to a guide author. this allows us to track the workflow of any student workers, and provides us with an archive of remediation of guides, and communication with guide authors, over time. figure 9. example email output produced by reviewr application. with reviewr, libguides can be rapidly remediated, keeping consistent focus on agreed upon standards, and obviating the problem of communicating clearly and consistently with authors. reviewr simplifies this process to the point that it can be confidently assigned to student workers with a high expectation of success and a high rate of turnover for remediating guides. and the carefully chosen language educates authors on the changes made in politic and educative language. conclusion we find that this three-part suite of web applications has both empowered guide authors to take advantage of the powerful functionality afforded by the bootstrap framework native to libguides, and yet ensured new and extant library guides conformed to brand / identity standards and the best practices of user experience (ux) design. the bootstrapr and reviewr applications have been released under an open source mit license, allowing for the adoption and adaptation of these apps by any organization which might find similar benefits. specifically, an organization might choose to modify the parameters of the reviewr app and/or the specific properties of any bootstrapr tool to better align with their own brand and identity standards and guide requirements. notes [1] the libguides cms affords direct linking within any guide to books from the library’s catalog, and to databases to which we subscribe. [2] jquery is a relatively light-footprint framework of javascript and css libraries for [3][4] for more information on prepending the button to the mini command bar, see my blog post “add a link to an external libguides handbook.” references unc chapel hill. “learning a second language.” https://learningcenter.unc.edu/tips-and-tools/learning-a-second-language/ barry chametzky, p., 2017. offsetting the affective filter. grounded theory review: an international journal, 16(1), pp.grounded theory review: an international journal, 01 june 2017, vol.16(1). accessed online http://groundedtheoryreview.com/2017/06/21/offsetting-the-affective-filter/ about the author randal sean harrison is the emerging technologies librarian at the hesburgh libraries, university of notre dame. there he assists patrons with the identification, evaluation, and use of emerging technologies, innovative online tools, and related services. a significant part of his work involves the creation of applications to help both librarians and patrons more efficiently and effectively navigate online information spaces. he is always eager to collaborate with other librarians on ways to drive greater user experience in digital tools. you can find information on current projects and how to contact him at randalseanharrison.com. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – creating a custom queueing system for a makerspace using web technologies mission editorial committee process and structure code4lib issue 55, 2023-1-20 creating a custom queueing system for a makerspace using web technologies this article details the changes made to the queueing system used by virginia tech university libraries’ 3d design studio as the space was decommissioned and reabsorbed into the new prototyping studio makerspace. this new service, with its greatly expanded machine and tool offerings, required a revamp of the underlying data structure and was an opportunity to rethink the react and electron app used previously in order to make the queue more maintainable and easier to deploy moving forward. the new prototyping queue application utilizes modular design and auto building forms and queues in order to improve the upgradeability of the app. we also moved away from using react and electron and made a web app that loads from the local filesystem of the computer in the studio and runs on the svelte framework with ibm’s carbon design components to build out functionality with the frontend. the deployment process was also streamlined, now relying on git and windows batch scripts to automate updating the app as changes are committed to the repository. by jonathan bradley the challenges with the old system the 3d design studio at the university libraries at virginia tech for years used a battle-tested queueing system. created by jonathan bradley in 2017, the queueing system was a react app packaged and installed using electron with a backend api created using dreamfactory on a digital ocean vps cloud instance. the queue was composed of a form filled out by student workers with the help of patrons, and all of the data dumped into multiple filterable, sortable, and searchable tables that allowed for updating said queue entries. the system also accommodated multiple queues, one for our standard print requests and a “special request” queue that handled particularly challenging patron requests that required input from the studio manager. the system also had a few bonus comforts, such as automatically sending an email to the user when their print was marked as completed in the system and informing them they could come pick it up, connecting to a small receipt printer in the room which is used to label prints for pickup and organize the physical objects, collecting reference question statistics, and checking user requests to make sure that a particular patron only had one job in the queue at a time. figure 1. a screen capture of the queueing system for the 3d design studio, made using react and electron but the system wasn’t without its difficulties. the electron system itself presented many of these challenges, the main of which was the difficulty of updating. each time a new build needed to be made, a long build process had to be run. the software that builds electron, called electron forge, was constantly changing, and it was quite common for our app to no longer build just from the changes that had happened between updates to the app. this usually meant that what should have been a small and quick update to the interface became hours of work as the build process was reestablished and dependency versions were adjusted and conflicts cleared. this made updating the app onerous, which resulted in fewer updates in general. but even after the build was completed, the install process meant physically loading the installer on a flash drive and running down to the space to install it on the studio computer. even though electron has a squirrel installer option that can be run on a server to provide updates centrally, from our research, such a server could not easily be setup privately for software not intended to be distributed to the public. the react app itself was also a problem. in general, over the years, we’ve found that react just isn’t a good solution to most of our software projects, as it is far too involved for the small projects we are creating, and the framework itself requires more overhead than we really need. and finally, our digital ocean vps was presenting an ongoing problem, as a change in university policy meant the payment method we were using for digital ocean was not an option anymore. with no alternative way to pay them, we needed to move our backend to a different service. when we received confirmation in fall 2020 that our proposal to build the prototyping studio, a large makerspace that would absorb the 3d design studio and greatly expand on its offerings and service model, was moving into construction, we knew this would be the time to fix many of the challenges we were facing. the new service model demanded a change to how we handle patron requests. fixing our mistakes the first major challenge we wanted to take on was the change to the service model and ultimately, our data structure. in the 3d design studio’s queue, the database had a single table called “queue” that contained a field “type” where we stored information about whether the print was intended to be a resin print, special request, or just a standard print. when we made api calls to dreamfactory, a server-side software that generates, documents, and manages rest apis based on a database’s structure, we would filter based on this field in order to generate the individual queues in the system that were displayed to our student workers. and that worked fine for a service that was only offering 3d printers, but with the prototyping studio, in addition to offering all of those types of queues, we would also have people coming in to use our cnc machines, laser cutter, vacuum former, pick and place machine, etc. additionally, patrons might come in to use multiple machines or just have multiple jobs on a single machine that were all contributing to a single project. we wanted to be able to capture how these projects were coming together and the various tools needed to finish them. we changed the primary field in our database from “queue” to “project” and decided that all entries in the system would be a project, and every project would hold the machine jobs needed to complete it. this meant our database now had many tables in it, starting with the “project” table and adding a table for each type of queued machine, including “cnc”, “laser”, “resin”, etc. the database also contains many-to-many join tables for each machine, allowing for each project to contain multiple entries for a machine job, and multiple types of machine job entries, which means now a patron could have a single project with entries for a laser cutting job, 2 resin 3d printing jobs, and a cnc machine job. figure 2. the main screen of the new prototyping studio queue while we kept the individual tabbed queues from the old system, each one is now filtering jobs out of a single “project” api call to our dreamfactory instance. but in order to accommodate this, our form for adding projects needed to grow in complexity over the old system. since a project can contain any number of job types and individual jobs, our form now has checkboxes for selecting the queues needed for the project and templated sub-forms that our student workers can add to any project. figure 3. a sample of the form used to gather project information into the queue with a new data structure in mind, we tackled the problem of our vps, which we were able to solve by moving our dreamfactory instance out of the cloud and back on-premises into a vm owned and secured by the university libraries. by doing this, we were able to also include added security by limiting access to the dreamfactory instance to virginia tech’s internal network, drastically reducing the number of potential attack vectors on the api server. after the backend was sorted, the question became how to handle front-end concerns for the application. we wanted to avoid using electron for the reasons listed above. tauri, a rust-based alternative to electron, and proton native were both considered as well, but tauri was still in early beta and both seemed to present similar concerns to electron. we stepped back and thought about the need in more detail. the app needed to be present on three computers, all within the physical building of the library; all the data would come from api calls; the ancillary functions, like sending emails and printing receipts, were all accomplished via api calls to various endpoints as well; even saving files, an option present in the previous version of the queue but underutilized, was handled via api calls to the dreamfactory server. we didn’t need our students to log in using any federated system because we can find out which student updated the software based on update timestamps and work schedules if need be, and we both didn’t need and didn’t want the public to be able to access the app. at the core of it, the app didn’t need to do anything outside the capabilities of a simple web application, and i had been playing around more and more with local web apps that run within the browser from the computer’s file system instead of via a server as a means of deploying one-off apps that needed to be on kiosks or other controlled points with no access from the greater internet. this sort of deployment had worked well for our patron satisfaction kiosks we have spread throughout the various studios in newman library, and it seemed like it could be a viable option for bypassing some of our greatest hiccups with the previous version of the queue. the next hurdle was choice for front-end development. i had moved away from using react months prior for new projects given that it was simply far more infrastructure than was needed and often made our projects less maintainable, since not many people in our library were working within the react ecosystem. since moving away from react, i was forgoing a front-end framework all together for new projects and coding in vanilla javascript, but i’m a proponent of using the right tool for the job and no more. the prototyping studio queue was going to be far more complicated than a feedback kiosk or a dynamic upcoming events page for digital signage, and the nature of the project, handling large amounts of data that would need to be loaded, displayed, sorted, and filtered frequently and in a user-friendly manner, could actually benefit from some of the features many frameworks offer out-of-the-box, such as two-way data binding, templating, and component-based layouts. in the end, i decided to go with svelte; i had used it on a couple previous projects, and i really appreciated the way it implements data-binding and how little overhead it adds, leaving a project that doesn’t require much additional knowledge beyond standard html/css/js. its handling of app state via the built-in stores implementation was worlds ahead of the react + redux solutions i had needed to use previously in terms of simplicity, and its build and bundle up-front strategy for websites was a benefit for a locally deployed site, since websites loaded using the filesystem instead of a server have some additional hurdles, particularly related to fetching additional files, which are treated as having opaque origins by modern browsers regardless of actual location. the other reason svelte was chosen was that it had a good community that had already built some component libraries for the framework based on various popular design languages, like material design by google or carbon by ibm. i often look to component libraries when developing larger projects, as they can take care of much of the implementation of common web elements like forms with validation, date/time calendar pop-ups, tooltips, etc. and in general take the process of design off the shoulders of the person doing the coding and place it on someone with more explicit training in design, which usually results in a more user-friendly experience that follows best-practices in the field of ux. it also frees up more of the developer’s time to work on the unique components that will be required for their particular app, which is always a bonus. looking through the potential options for a component library, we eventually landed on using the svelte implementation of carbon by ibm for two main reasons. the first reason was that it had a fully-featured data table component, which is the main focal point of a queueing app. their implementation included built-in search components, sortable headers, customizable cell views, collapsible sub-rows, and many other small quality-of-life touches that can make a big difference in an app where a user is primarily interacting with data tables. the second reason was that out of the options available, carbon was closest to virginia tech’s branding style, meaning with only a few tweaks to the css i could get the components to meet branding guidelines for the university. i would note that even though this application, given its niche use and the lack of any public-facing distribution, didn’t actually have to meet our university’s branding guidelines. i always try to meet them regardless because i find that 1) it is good practice to be in the habit of always trying to meet your university or organization’s defined style, 2) in a component-based development process, the things you build can be reused elsewhere in the future, and 3) it lends legitimacy to the things you build, ensuring your student workers and any patrons who see the application view it as a cohesive part of the ecosystem established by your organization and not some random or potentially sketchy software. the structure and maintainability of the app a number of features were implemented in the development of this app from the beginning with the goal of making it a simple task to update. the first was the modularization of the major functions. all the named functions within the app are contained in their own file, which is named after the function, in a folder called “lib,” and the functions are exposed the rest of the app via an index file in the folder that imports and exports each function export { searchtable } from "./searchtable.js"; export { getdata } from "./getdata.js"; export { resetschema } from "./resetschema.js"; export { sendreceipt } from "./sendreceipt.js"; export { sendemail } from "./sendemail.js"; export { filterqueues } from "./filterqueues.js"; export { editentry } from "./editentry.js"; export { buildsubrows } from "./buildsubrows.js"; export { saveentry } from "./saveentry.js"; export { getarchive } from './getarchive.js'; export { addjob } from "./addjob.js"; export { makeactive } from './makeactive.js'; export { closeproject } from './closeproject.js'; export { checkprojectstatus } from './checkprojectstatus.js'; figure 4. code snippet from index.js that imports and exports all of the function in the lib folder this sort of modularization is common in many programming ecosystems and may not seem like much, but i strongly recommend this practice to anyone building apps, especially in javascript. this allows for easy debugging of your functions and makes them reusable not only within the app, but within other projects as well. i’ve written a filtering function for data tables that has made its way into half a dozen different projects, which was made possible by designing all of the functions to be modular. the second concern was the nature of the space this queuing app will serve. given our previous experience with the 3d design studio, we know to expect this service to evolve, which almost assuredly means both new machines in need of their own queue becoming part of the service and changes to the nature of the forms and the information we need to run a job on a given machine. in the past, making substantial changes to the forms or adding a new queue would mean a lot of editing code and customizing the solution for that particular scenario, which was something i wanted to avoid moving forward. this app contains a config.js file that exports a json object to the app with definitions for all of the queues in an array of objects: queues: [ { label: "projects", machine_id: false, headers: [ { key: "project_name", value: "name" }, { key: "email", value: "email" }, { key: "user_name", value: "user" }, { key: "complexity", value: "complexity" }, { key: "timestamp_created", value: "date submitted" }, { key: "timestamp_updated", value: "last updated" }, { key: "machines", value: "machines involved" }, ], }, { label: "extrusion printing", machine_id: "extrusion", form: { active: "extrusion_jobs", definition: extrusiondefinition, buttontext: "extrusion print", }, headers: [ { key: "user_name", value: "user" }, { key: "email", value: "email" }, { key: "activeextrusion.currently_on", value: "currently on", }, { key: "activeextrusion.timestamp_created", value: "date submitted", }, { key: "activeextrusion.filename", value: "filename" }, { key: "activeextrusion.comments", value: "comments/notes", }, { key: "activeextrusion.printer_size", value: "print size", }, { key: "activeextrusion.material", value: "material" }, { key: "activeextrusion.print_weight", value: "estimated filament", }, { key: "activeextrusion.print_time", value: "estimated time", }, ], }, figure 5. snippet of code from config.js illustrating the structure of the array of objects that define two example queues instead of coding a ui for each queue, the system loads this array and loops through it, building the queue ui tabs based on the data provided. this allows for the creation of a new queue by simply adding a definition to this config file instead of coding an interface for it. <tabs> {#each config.queues as tab} <tab key={tab.label} label={tab.label} /> {/each} <tab key="archive" label="archive" /> <main slot="content"> {#each config.queues as tab} <tabcontent key={tab.label}> {#await $alldata} <loading /> {:then} {#key $alldata} <table fulldata={filterqueues($alldata, tab.machine_id)} headers={tab.headers} title={tab.label} id={tab.machine_id} /> {/key} {/await} </tabcontent> {/each} <archive /> </main> </tabs> figure 6. snippet of code from tabbar.svelte that builds the queues similarly, the project contains a folder called forms, which contains numerous definition files, one for each form. these definition files, too, export a json object that contains an array of objects with the required data to build the form, including the type of form question, be it text input, number input, dropdowns, etc., the data the form answer should be bound to, and any restrictions on the input. export default [ { type: "text", id: "filename", label: "filename", placeholder: "should follow naming conventions", bind: "filename", }, { type: "select", id: "status", label: "status", bind: "status", email: true, options: [ { value: "in queue", text: "in queue", }, { value: "completed successfully", text: "completed successfully", }, { value: "failed", text: "failed", }, { value: "remove", text: "remove", }, ], }, { type: "select", id: "currently_on", label: "currently on", bind: "currently_on", receipt: true, options: [ { value: "not manufacturing", text: "not manufacturing", }, { value: "lazer face", text: "lazer face", }, ], }, { type: "select", id: "source", label: "file source", bind: "source", options: [ { value: "downloaded it", text: "downloaded it", }, { value: "made it myself", text: "made it myself", }, { value: "edited a download", text: "edited a download", }, ], }, { type: "select", id: "job_type", label: "job type", bind: "job_type", options: [ { value: "cut", text: "cut", }, { value: "engrave", text: "engrave", }, { value: "both", text: "both", }, ], }, { type: "select", id: "material", label: "material", bind: "material", options: [ { value: "wood", text: "wood", }, { value: "acrylic", text: "acrylic", }, { value: "hardboard", text: "hardboard", }, { value: "other/patron provided", text: "other/patron provided", }, ], }, { type: "number", id: "length", label: "length of material", invalidtext: "this won't fit on the machine.", helpertext: "in inches", bind: "length", min: 1, max: 36, }, { type: "number", id: "width", label: "width of material", invalidtext: "this won't fit on the machine.", helpertext: "in inches", bind: "width", min: 1, max: 24, }, { type: "textarea", id: "comments", label: "comments/notes", placeholder: "enter comments here...", bind: "comments", }, ]; figure 7. code from laserdefintion.js that defines the form for our laser cutter the folder also contains formtemplate.svelte, which takes the data provided in the array and loops over it in order to build each form. this approach means that adding a whole new form means only adding the definition file, and updating an existing form is as simple as changing the values in a json file instead of re-coding an interface. it also means bugs, like one we encountered where dropdowns weren’t resetting after an entry was saved, need only be fixed in a single place for all forms. <div class="queue-container" style={$currentproject[[active]] ? "" : "display:none;"} > {#each formdefinition as question} <div class="question-container"> <formgroup> {#if question.type === "text"} <textinput id={question.id} labeltext={question.label} placeholder={question.placeholder} bind:value={$data[index][[question.bind]]} required={true} /> {:else if question.type === "select"} <select id={question.id} labeltext={question.label} bind:selected={$data[index][[question.bind]]} on:input={(e) => { question.receipt ? receiptcheck(e, index) : question.email ? emailcheck(e, index) : null; }} > {#each question.options as option} <selectitem value={option.value} text={option.text} /> {/each} </select> {:else if question.type === "number"} <numberinput min={question.min} bind:value={$data[index][[question.bind]]} max={question.max} invalidtext={question.invalidtext} helpertext={question.helpertext} label={question.label} /> {:else if question.type === "textarea"} <textarea id={question.id} labeltext={question.label} bind:value={$data[index][[question.bind]]} placeholder={question.placeholder} /> {/if} </formgroup> </div> {/each} </div> figure 8. code snippet from formtemplate.svelte illustrating how the form is generated using the definition file one of the other major concerns to arise concerning maintainability surrounded the global state of the app. using svelte already solved many of the headaches related to store-based global state, primarily because it doesn’t require writing additional control functions for saving and retrieving data and it provides some nice shortcuts in the form of the $ operator before a variable, which allows you to access the store value instead of having to first assign it to a local variable. but the nature of this queue, where every project is potentially composed of multiple sub-queues, and those sub-queues may have multiple entries in them, meant that reconciling this complex series of forms prior to saving updates or creating new entries in the database required multiple custom functions. however, we were able to replace these functions and instead use the derived store functionality of svelte to reconcile them, drastically improving the readability of the code and removing that logic from the inner workings of the app and placing it in the stores.js file where it can be easily found. export const finalproject = derived( [ currentproject, currentextrusion, currentresin, currentlaser, currentmetal, currentmill, currentpcb, ], ([ $currentproject, $currentextrusion, $currentresin, $currentlaser, $currentmetal, $currentmill, $currentpcb, ]) => { let project = { ...$currentproject }; delete project.cells; delete project.timestamp_created; delete project.timestamp_updated; project.extrusion_jobs ? ((project.extrusion = $currentextrusion), delete project.extrusion_jobs) : (delete project.extrusion_jobs, delete project.extrusion); project.resin_jobs ? ((project.resin = $currentresin), delete project.resin_jobs) : (delete project.resin, delete project.resin_jobs); project.laser_jobs ? ((project.laser = $currentlaser), delete project.laser_jobs) : (delete project.laser, delete project.laser_jobs); project.metal_jobs ? ((project.metal = $currentmetal), delete project.metal_jobs) : (delete project.metal, delete project.metal_jobs); project.mill_jobs ? ((project.mill = $currentmill), delete project.mill_jobs) : (delete project.mill, delete project.mill_jobs); project.pcb_jobs ? ((project.pcb = $currentpcb), delete project.pcb_jobs) : (delete project.pcb, delete project.pcb_jobs); // console.log(project); return project; } ); figure 9. code snippet from stores.js of derived store that contains all queues app deployment with the app itself working and ready for use, the question of how to improve the deployment process over the old queue became front and center. three major concerns stood out, the first being the need for the process to be as easy as possible. apps that are easy to update are updated more frequently; that’s just a fact of the development world, and it’s a truth i didn’t want to fight against. the second concern was removing the need for manual deployment to the physical computers where the app would live. carrying flash drives around the building, interrupting service points, running from floor to floor to each computer, all of this amounted to an annoyance that had grown over time. and finally, the third concern was the need for a development version of the application to be available as well. in the past, we’ve had to train student workers using the production queue since it was the only version that our manager had access to, and training using production data is never a good idea. eventually it became clear that git was the solution that would fix concerns one and two. git was already part of the workflow for development of the app, and we had used git pulls as a way of deploying other apps, such as the aforementioned feedback kiosks, so this seemed like a reliable solution here as well. we also decided to simply bundle the development version of the app in with the deployment of the production app, and differentiate the version in both name and visual style. figure 10. the header for the development version of the app figure 11. the header for the production version of the app the eventual workflow began to look like this: the developer makes an update to the app, and once tested and ready for production, uses a custom npm script command called “build-all” that builds two versions of the app, one that is passed a “dev” variable and another that is passed the “prod” variable. these variables are used in the build process of the app to decide which api endpoints and keys to use and which template to use to style the app header. each separate build, after completing, is copied by the script into its own special folder, one called “production” and the other “training.” those two folders are part of their own git repository, separate from the repo that contains the source code, and after the build script is completed, the repo is committed and pushed to virginia tech’s install of gitlab. meanwhile, each computer in the building that should have access to the queue, which includes the manager’s, the service point computer, and the developer’s, all go through a one-time setup. on each computer, a windows batch script is placed in the windows documents folder. this script has the information needed to pull the repository down from gitlab and place it in the current directory. the script is then added to the windows task scheduler to run every night at 2:00am, when it will do a “git pull” and get any updated code from the repository. finally, two shortcuts are placed on the desktop of the computer, one called “prototyping queue” and the other “training queue,” each of which point back to the index.html files in their respective folders. when clicked, the app is opened in the default browser, and the user can go about managing queue functions as normal. this structure means that updates are as simple as running a single bash command to build the app and then committing it to the repository. from there, the app will automatically be updated at 2:00am, or in the case that something isn’t working quite right or the update needs to happen immediately, the user on the computer can simply open their documents folder and double click on the windows batch file, which will immediately instigate a “git pull” against the repository. using this deployment structure, we’ve been able to manage the app in a much more timely manner, including identifying bugs and then fixing and deploying those changes in a matter of minutes as opposed to the far more involved development pipeline that existed for the previous application. looking ahead the changes to our queueing system put into place for the new prototyping studio have proven to be effective and have saved a great deal of developer time and effort, and the overall gains that could accumulate over the lifetime of the application speak to the benefits of this simple deployment structure for applications that don’t require public-facing servers. the biggest challenge another organization might face in producing similar applications is likely the dreamfactory server space for api handling. we are lucky enough to have infrastructure in place to fill that gap in the workflow, but that may not be the case for everyone. in those situations, a local sqlite3 database bundled with the app might be a usable stand-in. it is an alternative i have been considering more frequently as of late, but having a readily available and self-generating api on top of said database is a problem i have not investigated fully. i feel confident that a solution could be put together using deno and its ability to compile scripts to binary executables, but it is possible that an even simpler solution exists as well. as for our own prototyping queue, the issues tracker has no current outstanding bugs reported from our student workers, mostly due to the ability to quickly make updates and deploy them, but we do have some feature enhancements we are looking to implement in the near future. the one currently at the top of our priority list is to enable file uploads from our student workers, which will allow for better tracking of the files needed to complete a project and potentially allow for the collection of “evidence” in the situation in which a patron wishes to add their project to their eportfolio and needs photos of the prototyping process or other materials collected related to their work to upload later. the other major enhancement we are currently strongly considering is the addition of a data visualization tab that would allow for a quick glance at some anonymous real-time statistics about the various projects in the queue. both of these, however, are luxuries, not requirements, that we are able to pursue now that we have a queue redesigned with a focus on simplicity and maintainability instead of just having something that works. about the author jonathan bradley is the assistant director for learning environments and innovative technologies at the university libraries at virginia tech. in this position, he oversees the libraries’ six studio spaces, which focus on technologies ranging from virtual reality to media production to maker tools. he also does web development for the studios, creating various tools to gather feedback from patrons and improve the service point experience. he earned his doctorate from middle tennessee state university in 2013. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – using authority data in vufind mission editorial committee process and structure code4lib issue 14, 2011-07-25 using authority data in vufind the use of keyword-oriented next-generation catalogs in libraries has diminished the perceived value of the structured authority data that played a more crucial role in earlier opacs. however, authority data can still be combined with modern discovery in useful ways. this article examines several ways in which the open source vufind environment provides information to its users, showing how these mechanisms can be combined with authority data to enhance discovery. topics covered include autosuggestion, context-sensitive recommendations, use of apis, and means of harvesting and locally indexing authority data. by demian katz, ralph levan, and ya’aqov ziso background libraries have long relied on authority data to ensure consistent, accurate retrieval of information. by maintaining unique, established terms for names and subjects, it is possible to find very precise sets of associated records, assuming proper metadata maintenance and adequate cross-referencing. in recent years, discovery tools like vufind have begun to replace the traditional browse-oriented opac with an environment built around keyword searching and faceted navigation. while this new keyword orientation often makes the library search environment more accessible to the casual user, it also tends to diminish the use of authority data, and by extension, to reduce the precision of name and subject searching. next-generation discovery and authority data need not be mutually exclusive. it is possible to augment a discovery tool with authority data, allowing users to benefit from librarianship’s data curation without having to learn a whole new skill set. in “querying oclc web services for name, subject, and isbn,” ziso et al. examined some of the ways in which authority data can be accessed programmatically through oclc’s web services and proposed possible uses of this data in a discovery environment. this article follows up on this research, showing exactly how those ideas can be implemented in the popular vufind discovery environment [1]. autosuggestion one of the simplest ways of integrating authority data into a discovery environment is to use it to provide search suggestions in real time while the user enters a query. assuming a well-maintained set of metadata, a search for established forms of name or subject terms should yield high-quality search results from the local catalog, assuming any results exist there. even when no results are found locally, established terms are still useful for searching external services. for example, since vufind includes modules for searching oclc’s worldcat [2] and serials solutions’ summon [3], it would be possible to configure the standard “no results found” message to provide links into these additional resources, preventing dead ends in a user’s search experience. as of release 1.0.1, vufind still had no standard autosuggestion mechanism, although the feature was occasionally requested and discussed. implementing simple autosuggestion is almost trivial, but two factors made autosuggestion in vufind slightly more complex than average. first of all, vufind tries to be javascript library agnostic; depending on the selected theme, the software may use either yui [4] or jquery [5], so a solution needed to be easily compatible with both libraries. additionally, vufind aims for extremely granular configurability – administrators should be able to customize separate autosuggest behavior for each search type (subject, author, title, etc.) and be able to easily create their own autosuggestion plug-ins; a one-size-fits-all suggester would not suffice. ultimately, the design did not prove especially difficult. vufind’s final autosuggestion implementation, released in version 1.1, works like this: vufind’s templates include javascript code that passes the current contents of the search box and the current state of the search type drop-down to an ajax handler and then renders the handler’s response as clickable suggestions. this is easily accomplished with either yui or jquery code, depending on which theme is selected. a configuration file (searches.ini [6]) allows the administrator to specify which autosuggestion classes are loaded in response to which search type requests. the ajax handler loads the configuration file and instantiates an appropriate autosuggester object based on the current search type. it uses this object to obtain suggestions based on the user’s current search terms, and it renders these suggestions in a format that can be easily processed by the calling javascript. the autosuggester classes used by the ajax handler follow a common interface [7], so it is easy to write new ones. the generation of suggestions is completely separated from the implementation details of passing those suggestions back to the browser, allowing each of these problems to be solved independently. with this infrastructure in place, it is straightforward to provide suggestions from a variety of sources, including the local solr index or remote authority services. vufind is packaged with simple default autosuggesters that use its default bibliographic index, but at least two search types (author [8] and subject) can potentially benefit from authority-based suggestions. proof-of-concept name suggestion classes were written for both oclc’s identities autosuggester [9] and a local solr authority index [10] (discussed in more detail below). both solutions performed reasonably well. using the remote service introduces the usual hazards of depending on an external service (network lag time, potential outages) while also offering a lesser degree of control over the data; however, it is very easy to utilize, and it leverages oclc’s huge worldcat database for relevancy ranking. using a local index offers better control over data and behavior while requiring significantly more local resources (memory and disk space) to achieve acceptable performance. figure 1. worldcat identities-based autosuggestions in vufind’s default theme the current proof-of-concept authority-based autosuggesters work on the assumption that the bibliographic data in vufind’s index has headings consistent with the authority source being used. libraries whose integrated library systems support authority control features should have little trouble ensuring that this is the case. when working with authority and bibliographic records exported from the same ils, consistency of headings between records is assured. when using an external api, there is less of a guarantee that the remote headings will correspond to the local records. bibliographic records that have been subjected to regular name and subject authority work have a good chance of consistency with oclc’s identities records or the latest subject headings from the library of congress, but many libraries are not committed to this ongoing maintenance due to the time and money involved. research is currently underway to investigate the possibility of using open data and open source tools to improve consistency between authority records and bibliographic records; this is discussed in more detail below. context-sensitive recommendations the search suggestions offered in an autosuggestion box are inherently limited in their ability to convey information. fortunately, the use of authority data need not be limited to autosuggestions. starting with version 1.0, vufind also provides a mechanism for displaying context-sensitive suggestions as part of the search results screen, either above or beside the list of matching documents. these suggestions are most commonly manifested as a box containing links to alternate searches or resources related to the current query, but the design is flexible enough to allow other creative applications. the “recommendations module” feature is designed like this: like autosuggesters, recommendation modules are built using a standard interface [11]. each recommendation module is a class that obtains information about the user’s current search from vufind, performs its own internal processing and then returns all of the information necessary to render suggestions on the search results screen. since vufind internally represents searches as objects, recommendation modules have access to many details (search query, search type, current filters, etc.) without requiring a lot of parameters in the interface. whenever a search is performed, vufind checks a configuration file (searches.ini, the same file used for autosuggestion settings) to see if any recommendation modules are relevant to the current search type (title, author, etc.). it is possible to set a default recommendation module for use across multiple search types, and it is also possible to load multiple recommendation modules for a single search. all relevant recommendation modules are instantiated and used to provide suggestions. the recommendations interface is not just a handy way to extend vufind; it is also a key element of its core design. for example, vufind’s prominent facet display (used for narrowing searches) is actually a recommendation module. figure 2. context-sensitive author recommendations in vufind in their article, ziso et al. presented demo perl applications (name finder and term finder) which queried oclc’s worldcat terminologies and identities services for related, broader, and narrower terms. this code was easily translated into php recommendation modules [12], allowing author and subject searches to be augmented with context-sensitive suggestions for related avenues of investigation. in practice, the identities suggestions generally seem more effective than the terminologies suggestions, since the identities api offers relevance ranking based on frequency data from the worldcat database, while the terminologies service for library of congress subject headings returns records in arbitrary order. this may improve in the future if improved sort functionality is added to the terminologies lcsh search or if vufind’s recommendations module is adjusted to use the fast index [13] instead of lcsh (since fast now supports a “recordcount” sort key). implementing recommendation functionality based on a local solr index is also reasonably straightforward, offering similar trade-offs to those discussed under autosuggestion above. also as with autosuggestion, the effectiveness of these tools relies on consistency between recommended headings and those actually found in bibliographic records. indexing authority data as previously discussed, it is often useful to have a local index of authority data. in addition to protecting against external service outages, a local index also offers greater control over both content and ranking behavior. for example, suppose it would be helpful to highlight headings that pertain to a particular local collection. as long as the data being indexed contains enough detail to identify which headings apply to which collections, records outside of the target collection could be excluded with filter queries, or else the target collection could be given higher relevancy rankings through boost queries. with access to enough data, it would be possible to store headings for a local library, a consortium, and all of worldcat in a single index, retrieving the most appropriate suggestions for any given scenario through context-sensitive filtering and boosting [14]. this sort of functionality is not available solely through use of oclc’s identities and terminologies services, which currently reflect a single global, collection-neutral pool of headings. vufind is well-equipped to index authority data. it is packaged with the solrmarc [15] indexing tool, which quickly maps marc records into a solr index, and its default multi-core solr configuration includes a separate repository specifically intended for authority data. however, even though vufind’s authority core has existed for quite some time, it was not until release 1.1 that its use began to mature with a more functional solr schema and well-defined import scripts [16]. figure 3. abbreviated sample marc authority record the solr schema used by the latest version of vufind relies on a convenient fact: although the marc authority specification [17] defines a plethora of fields, most of them boil down to the same things: main headings, see also references and see from tracings. within the 1xx, 4xx and 5xx groupings, many tags are defined, but they all have the same basic meanings; the different numbers exist solely to distinguish the types of heading being referenced. rather than defining separate fields for every heading type, vufind’s authority schema instead takes a faceted approach: it has just one field each for “heading,” “use_for” and “see_also,” but it uses “source” and “record_type” fields to facet out where records came from and what types of headings are involved. solrmarc properties files and a handful of custom beanshell indexing scripts [18] are used to get the right data into the right places. once the index is built, it becomes possible to search across all authorities or to narrow in on a particular type of heading from a particular source. it is also important to note that even though the authority index currently focuses on 1xx, 4xx and 5xx fields, the remainder of the record is also used. an “allfields” catch-all indexes keywords from other places in the record that are not directly mapped to their own fields, improving findability [19], and a “fullrecord” field stores the original marc data in full, allowing it to be easily displayed to the end user and offering flexibility for as-yet-unforeseen purposes. harvesting authority data if you have local authority data in your ils, you can easily export it and load it into vufind using solrmarc. however, if you do not maintain local authority data, it is necessary to obtain it from another source. fortunately, a lot of authority data is government-generated and thus public domain, so it is possible to collect it from online sources. as a proof of concept, vufind is packaged with a command-line php script which is capable of incrementally harvesting the entirety of the library of congress name authority file through an sru interface provided by oclc [20]. using the browse by modification date feature of the oclc sru interface, this script collects all of the naf records in order of modification and saves them to a directory on disk in marc-xml format. it can be re-run periodically to harvest new records modified or created since the last run. these harvested records can then be batch-indexed using the same tools that vufind provides for dealing with oai-pmh harvests [21]. in addition to collecting incremental additions to the index, the harvesting tool is also able to detect deleted records, a feature necessary since naf records are sometimes removed when they are discovered to be redundant or obsolete. the tool identifies unwanted records by comparing lccn values on the remote sru server against lccn values in the local solr index. since this requires a full sru index scan and extraction of many terms from the solr index, detecting deletes is a time-consuming and processor-intensive task. if the sru interface is enhanced in the future to provide data on deleted records by date, it may be possible to achieve the same effect more efficiently. for now, at least the harvester serves its purpose: it produces a text file containing a list of records to delete, which can be processed by another of vufind’s standard oai-pmh tools to clean up the local index. oclc’s lcnaf index was chosen as a proof of concept for harvesting because of its convenient api and its use of marc data, which is especially easy for vufind to index. by constructing similar tools, other authority sources could also be loaded into vufind’s index. a particularly good starting point for future research would be the library of congress subject headings (lcsh), which are freely provided in linked data format [22]. the potential application of lcsh to vufind is fairly obvious, but there is also broader potential in developing tools for working with linked data; this may be a first step toward enabling vufind recommendations based on much larger pools of semantic data such as freebase [23]. utilizing authority data in addition to the autosuggestion and recommendation applications mentioned above, vufind’s local authority index currently has two other applications. first of all, vufind includes a simple authority module which allows direct searching and faceted browsing of the authority data, offering direct access to the original marc data. while authority records have traditionally been the domain of library staff rather than library patrons, exposing them through vufind broadens their functionality and usefulness. the tremendous research effort that has gone into collecting the data they contain is thus made accessible to a broader audience. even more significantly, vufind (starting with version 1.1) includes an alphabetical browse module [24] that implements structured heading exploration within vufind, providing a useful alternative to keyword search for power users familiar with traditional early-generation opac functionality. the alphabetical browse module works by loading all titles, authors, and subjects from vufind’s bibliographic index. it does not require an authority index, but can access authority data to incorporate see and see also references for names and subjects into its results. future work as suggested earlier, exploring user-friendly contexts in which locally indexed authority records could be presented (for example, in an “author information” module) may be worthy of further research. in order to justify its continued creation and maintenance, the value of rich authority data should be shared with everyone, not just catalogers. additionally, more powerful applications may be possible with the help of the extensible catalog project’s metadata services toolkit [25], a tool for bulk processing of metadata. starting with vufind 1.1, vufind’s authority and bibliographic indexes are harvestable via oai-pmh [26]. this makes it possible to export records from vufind into the mst and augment them in various ways – flipping obsolete headings in bibliographic records, identifying authority records that are used in a particular collection, etc. research on the optimal workflow for authority processing with the mst and vufind is currently ongoing. once completed, this research will help reduce inconsistencies between authority and bibliographic records in libraries that do not currently have access to expensive authority processing services, expanding access to all of the benefits provided by authority data while reducing the costs of having to outsource work to private authority vendors. conclusion using authority control during cataloging can benefit even a non-authority-aware discovery environment by, for example, ensuring that facets are applied consistently. however, by making the discovery environment directly aware of authority data, it is possible to tap into power beyond these basic passive benefits. so far, this area has not been explored to its full potential, and there is plenty of room for deeper research in the future. in the meantime, the various tools discussed above already demonstrate some of the potential of combining authority data with the discovery environment. notes [1] vufind: http://vufind.org/ [2] worldcat: http://www.worldcat.org/ [3] summon: http://www.serialssolutions.com/summon/ [4] yui: http://developer.yahoo.com/yui/ [5] jquery: http://jquery.com/ [6] searches.ini: https://vufind.svn.sourceforge.net/svnroot/vufind/trunk/web/conf/searches.ini [7] vufind autosuggester documentation: http://vufind.org/wiki/autocomplete [8] what vufind calls an “author” search could be more accurately described as a “contributor” search, since it may also turn up results matching editors and other types of contributors. the label “author” is used by common convention. the label “name” is avoided because it implies a broader scope; a “name” search might reasonably be expected to turn up works about a person as well as those contributed to by that person. throughout this article, the term “author” is used to refer to a contributor-oriented search, while the term “name” is used to refer to authority data that could be used as either contributor or subject headings. [9] identities autosuggester code: https://vufind.svn.sourceforge.net/svnroot/vufind/trunk/web/sys/autocomplete/oclcidentitiesautocomplete.php [10] solr authority index autosuggester code: https://vufind.svn.sourceforge.net/svnroot/vufind/trunk/web/sys/autocomplete/solrauthautocomplete.php and https://vufind.svn.sourceforge.net/svnroot/vufind/trunk/web/sys/autocomplete/solrautocomplete.php [11] recommendation module documentation: http://vufind.org/wiki/building_a_recommendations_module [12] oclc recommendation modules: https://vufind.svn.sourceforge.net/svnroot/vufind/trunk/web/sys/recommend/worldcatidentities.php and https://vufind.svn.sourceforge.net/svnroot/vufind/trunk/web/sys/recommend/worldcatterms.php [13] fast (faceted application of subject terminology) search interface: http://tspilot.oclc.org/fast/?operation=explain&version=1.1 [14] although his work does not deal directly with authority data, david walker’s screencast on his award-winning bridge project (http://library.calstate.edu/bridge/) discusses the value of distinguishing between different layers of availability (not just the simple binary of “local” and “in worldcat”). going into depth on this idea is beyond the scope of the current article, but this idea is important to think about when designing search workflows that incorporate authority data – the user may have several different search contexts, and it is important to make these easy to understand and navigate between. [15] solrmarc: http://code.google.com/p/solrmarc/ [16] vufind authority import documentation: http://vufind.org/wiki/importing_records#importing_authority_records [17] marc authority specification: http://www.loc.gov/marc/authority/ [18] see lcnaf.bsh and get*lccn.bsh in this directory: https://vufind.svn.sourceforge.net/svnroot/vufind/trunk/import/index_scripts/ [19] for example, note how the 360 and 680 fields contain potentially useful keywords in this example subject authority record: http://fred.lcsubjects.org/lccn/sh85070835 [20] lcnaf harvest script: https://vufind.svn.sourceforge.net/svnroot/vufind/trunk/harvest/harvest_naf.php [21] vufind oai-pmh harvesting documentation: http://vufind.org/wiki/importing_records#oai-pmh_harvesting [22] lcsh download: http://id.loc.gov/download/ [23] freebase: http://www.freebase.com/ [24] vufind alphabetical browse documentation: http://vufind.org/wiki/alphabetical_heading_browse [25] xc mst: http://code.google.com/p/xcmetadataservicestoolkit/ [26] vufind oai-pmh server documentation: http://vufind.org/wiki/tracking_record_changes#oai-pmh_server_functionality references ziso y, levan r, morgan el. 2010. querying oclc web services for name, subject, and isbn. code4lib journal [internet]. [cited 2010 december 21]; 9. available from: http://journal.code4lib.org/articles/2481. about the authors demian katz has a b.s. in computer science from west chester university and an m.l.i.s. from the university of pittsburgh. as villanova university’s lead vufind developer, he is greatly enjoying the opportunity to apply all of his education to one job and to collaborate with an enthusiastic open source community. in his copious spare time, he maintains a bibliography of paper-based interactive fiction at http://gamebooks.org (please pardon the lynx-friendly interface). ralph levan is a senior research scientist in oclc research. lately he has been working on identity issues and authority databases in general. he is a member of the sru standards group within oasis and seems to find a way to put an sru server at the bottom of all his technology stacks. ya’aqov ziso has been involved with bibliographic control and maintenance of authority headings for both corporate and academic sectors. currently he directs the digital archiving of the school for designing a society and composes experimental music. subscribe to comments: for this article | for all articles one response to "using authority data in vufind" please leave a response below: dlovins, 2011-07-27 great article. thanks! i was just wondering who’s experimenting with vufind and the extensible catalog mst (mentioned in the “future work” section). leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – prototyping as a process for improved user experience with library and archives websites mission editorial committee process and structure code4lib issue 18, 2012-10-03 prototyping as a process for improved user experience with library and archives websites prototypes can be persuasive tools for proposing changes within an organization through “imagine if” scenarios. they not only show how to enhance the online experience, but can provide a way to improve the overall organizational environment as well. in redesigning the princeton university finding aids site (http://findingaids.princeton.edu), we used a flexible subset of agile practices based around measurable goals, iterative prototypes, meetings with institutional stakeholders, and “discount usability testing” to deliver an innovative and much-improved user experience. this article discusses how integrating relatively untested, but promising new ideas for online finding aids required us to adopt a development process that would allow us to better understand the goals of both general and staff users and in turn foster an environment for innovation that thrives on collaboration, iteration, and managed risk. by shaun ellis and maureen callahan introduction what’s an interface for? library software developers and interface experts are often asked to improve the usability of our websites. yet, all too often, efforts to improve usability don’t go far enough to address the real user experience (ux) challenges libraries must overcome (mathews, 2012).  since our websites are actually a reflection of a whole ecosystem of tools, practices and attitudes within the library, the authority required for changing the underlying practices is beyond the reach of the technology departments alone.  at the same time, senior management is understandably wary of runaway projects, and generally wants to avoid being too disruptive without proof that the risks involved with implementing “paradigm-shifting” ideas are manageable and effective. moreover, both library developers and senior management are far removed from most of their user base.  with such limited interaction, how can developers really know what is right for our users? many claim the agile method of software development avoids delivery-time disappointment and improves user experience by offering a more inclusive and collaborative process with stakeholders. agile was designed to alleviate anxiety and build trust, but as simple as its proponents claim it is, there can be challenges to its adoption and such dramatic shifts in process are not always feasible.   moreover, agile assumes that the “client” is a known entity, a stakeholder that will help guide the software toward everyone’s needs.  in libraries, it’s hard to involve our patrons and researchers as much as they should be, especially when there’s such diversity among their needs and goals. in this paper, we discuss how integrating relatively untested, but promising new ideas for online finding aids required us to adopt a development process that would allow us to better understand the goals of both general and staff users, while reassuring stakeholders that a departure from convention would be worth the investment.  our approach used a flexible subset of agile practices based around measurable goals, iterative prototypes [1], weekly meetings with institutional stakeholders, and “discount usability testing” in an attempt to develop greater empathy with our users, flexibility in our process, and ultimately deliver an innovative and much-improved user experience. [2] background simply following modern web conventions is the solution to the lion’s share of usability problems (alexandre et al, 2012), even for archival access systems.  at a time when users can expect to find all kinds of content on the web, they are understandably frustrated that most archives offer world-wide online access to metadata, but then subject them to confusing processes to be able to get to the archives, register, request materials and look at them. [3] perhaps just as frustrating are non-standard local practices, professional jargon, and long, complicated finding aids that make archives sites difficult for the novice researcher to navigate.  we went to work imagining an archives access system more in concert with what patrons have come to expect from the web. princeton university libraries holds approximately 2200 archival and manuscript collections, nearly all of which are described in ead-encoded finding aids. although these are also described in an abstracted form as collection-level marc records and available in an opac environment, online finding aids are the primary context for user access, and web search is the primary discovery channel. the previous iteration of finding aid data at princeton was a straightforward html rendering of ead xml– it treated structured data as a single, often monograph-length document. for the last few years, archivists and library technologists at princeton have been thinking about what the next generation of archival description might look like, leveraging the potential of structured data. the experiment the idea of making ead components, not collections, the “atomic unit” of our finding aids site to improve findability and performance was the guiding notion of the site redesign. what does this mean, exactly? archives are described hierarchically – for instance, a file of correspondence from a writer’s literary agent may belong to a file group of business correspondence, which belongs to a correspondence series (which may also include personal and fan correspondence), which belongs to the collection of all of the writer’s papers. each level has descriptive metadata, and belongs to what we call a component. when encoding archival description in ead, the component is a wrapper element that helps denote the beginning and end of an archival unit. what was important to us was to no longer privilege any particular level of description but instead return the part of the collection most relevant to the patron’s search, without losing the place of this component in the context of the collection. viewing a single component as a record is a very different experience from the usual practice of viewing it as one line in a long list in a traditional finding aid. but the component-as-record approach is fully acceptable within archival standards. we engineered our rendering of component-level records so that they include all of the descriptive metadata fields necessary to meet single-level minimum requirements as described by dacs (describing archives: a content standard) and each component record has a unique identifier and a stable url.  in other words, we have urls that represent every level of the collection hierarchy. we certainly did not invent these ideas – archivists have been thinking and writing about this idea of breaking up archival components into discrete descriptive records for years. [4] [5] perhaps the reason that we see so few single-level displays today, despite the long-standing call to action, is because it is a major break in convention for online finding aids.  changing web conventions can confuse expert users who learned how to use the old tools despite their drawbacks, so we needed to proceed with caution.  the only way we could know if this approach would make research easier for both basic and expert users was to observe both using it. brief goals document based on an extensive review of the available literature about how users respond to archival description, the finding aids website redesign working group created a list of desirable features for the new finding aids site. in addition to “atomizing” components, we also wanted to allow users to view any available digitized content they describe.  we were already digitizing documents for researchers who couldn’t come to campus at several of the archival divisions and the digital photography studios, and we were interested in making this content available to the general public [6].  links to some of this content were being added to finding aids, but the interface for viewing these scanned materials was poor, and we had very limited infrastructure for managing and storing the scanned files. we hoped that by providing an intuitive interface for browsing digitized content and a standard workflow, we could offer an easier path toward “user-initiated digitization” for divisions that were not yet recycling this content for public access (see figure 1). figure 1. access to digitized content: side-by-side comparison of component-level views of one collection in prototype (left) and final site (right). thumbnails of digitized content can be selected from the “view images” menu for quick scanning of what can sometimes be hundreds of documents (shown on right). lazy loading of thumbnails (aka, “load as you scroll”) improves performance in such situations. right-hand example can be seen at http://findingaids.princeton.edu/collections/ac044/c0002. it also made sense to leverage the eac-cpf (encoded archival context for corporate bodies, persons, and families) encoding standard for archival authorities [7]. a cpf record provides another portal to content in a way that may be more intuitive to the user – it starts with the subject of a research inquiry rather than the materials that might describe that subject. for instance, instead of searching the collections for everything that we may have about woodrow wilson, an eac-cpf record provides information about woodrow wilson, and then lists the resources available in the repository that relate to wilson, providing a “browse experience” as an alternative to search. finally, we wanted to allow for conversations to take place around our collections.  commenting on every page, which means at every level of description, allows users to provide insights, corrections, comments, transcriptions and explanations of our material that while useful, do not necessarily fall under the realm of archival description. and as an institution, we have made a commitment to read, reply to and follow-up on every comment that we receive. we presented a short document listing these features to high-level decision-makers at the library. after securing their approval, this goals document became our touchstone for working through the project. it allowed us to measure how close we were to our initial goals and kept us focused on functionality without restricting the process or possible scope of solutions. the governance and management of this project was limited to seven members [8] — an interface developer, a metadata analyst, and five librarians and archivists, all of whom are responsible for creating and maintaining archival description, and many of whom have daily interactions with the users of archives.  weekly meetings were scheduled among these members throughout the lifecycle of the project, which was essential to success as it provided the developers with the continuous feedback they needed to improve the prototype. the prototype in action user empathy leads to better design one of the main challenges to improving ux for libraries and archives is how to cultivate user empathy in both developers and information professionals. the attitude that through online help, training, or brute force, students and researchers are expected to pay their dues and learn how to interact with these systems is a crutch that belies our commitment to patron satisfaction. programming has traditionally been a rational process where problems are considered “solved” if they meet functional requirements, so developers have little motivation to further improve a system unless they actually witness the frustration and emotional turmoil some users experience with their products. during the course of our work, as we came to important design decisions, members of the group would express preferences based on perceived user needs. many of these decisions had to do with vocabulary and labels, which could be changed easily if we could come to a consensus. but others could require significant code changes, such as debates over search results expectations, navigational solutions, and content browsing. one of the best decisions that we made as a group was to focus our tests around these internal debates, so that we could see how our users responded and ask them about their preferences instead of speaking for them. the right tools for the job in order to get real screens and real data in front of all our users, our developers had to figure out how to get from goals to prototype as fast as possible, and then gather feedback and respond to it just as quickly. it was important that the prototype was not simply a mock-up, but that users were interacting with real data. since we already had experience using exist (http://exist-db.org) as an ead database and mvc framework for the old finding aids site, getting real data into our prototype involved delivering contextual snippets of eads in various formats based on http requests. we added solr to improve overall search performance and quality. and two new tools we adopted helped us enormously when it came to usability testing: bootstrap (http://twitter.github.com/bootstrap) for prototyping, and trello (https://trello.com) for tracking feedback and progress. up until now, prototyping for the web has been hard. but some prominent designers have recently argued in favor of abandoning graphics software and the extra work that comes with them and go straight to the working prototype (fried, 2008). we found that bootstrap, twitter’s open-source set of reusable front end design patterns, gave us the tools we needed to build our pages with grids, elegant defaults, and a wide variety of instantly interactive web conventions. best of all, the design is relatively neutral and elegant “out of the box,” and keeps the feedback focused on functionality, not distractions like color, type, and layout choices. customization and branding decisions can happen later. what bootstrap does not do is offer ready-made solutions to domain-specific design problems. we had to create our own solutions to these problems in our prototypes, test them and gather feedback. building a prototype quickly means nothing however, if gathering, discussing, and acting on feedback is not just as fast. trello caught our eye for managing feedback because it is a free workflow tool that takes cues from agile’s use of index cards [9] to take user stories from idea to finished feature. in trello, we try to state the problem in a single sentence, with a link if necessary, quickly turning feedback into cards. further discussions and elaborations can occur easily through trello’s notifications. the card is “dragged” through a custom-defined workflow until it ends up in the “done” column. assigning someone to the ticket is as simple as dragging an avatar onto the card. moreover, we provide public, read-only access to our trello board [10] so that anyone can see where their suggestions fall in the development priorities and workflow, building trust through transparency and inclusion (see figure 2). figure 2. example trello board showing our “list-based” workflow for bug, feature and issue tracking. discount usability testing strategy once we had our prototype built and were ready to gather feedback, we took a page from steve krug’s chapter in don’t make me think, titled “usability testing on 10 cents a day: keeping it simple so you test enough”. krug notes that after a team has worked on a site for even a few weeks, they can’t see it fresh anymore. the only way you can be sure about how well it works is to test (krug, 2000). some stakeholders grow uneasy when a team brings up the topic of testing. the ghosts of past scholarly research projects can haunt those who have taken part in organizing studies on student populations, no matter how small. while they might be necessary for those who need to publish or perish, the administrative requirements, irb training, and other overhead associated with formal testing place a heavy burden on what should be a simple process for developing user empathy. an alternative testing approach, known as discount usability testing, simplifies the testing process with an emphasis on direct observation of a handful of users (nielsen, 2000). krug also warns that focus groups should not be confused with usability tests. focus groups can be helpful for encouraging interactive discussion among a group on reactions to design, but because they do not allow us to observe users performing goal-oriented, real-life tasks, they are not as effective in evaluating the usefulness of the site. we kept these warnings in mind as we thought about how to gather good feedback from our users. we designed our user testing scenarios based on how much time the group could spare, with the understanding that some testing is better than none, and that we could get important information about our system without formal, extensive testing conditions. and we did it at a time when everyone in the project group had been looking at it long enough to know what it was supposed to accomplish, but too long to approach it with fresh eyes. we wanted to get a quick reality check — do users understand what they’re looking at? are they able to find, identify, select and obtain objects that meet their research needs efficiently? are we allowing for serendipity, for researchers to stumble upon collections that may address their subject but don’t necessarily fit their search terms? our testing approach and observations we divided testers into two groups – basic users (those who may have some knowledge of the subject matter or archival research, but have no knowledge of creating archival description) and staff users (archivists and librarians familiar with archival and bibliographic access systems) — with the objective of creating an optimum experience for both groups. we tested basic users individually and (for the sake of time) asked staff users to do testing in groups; each member took turns walking through the research task, which was projected onto a screen for others to see. we asked each test person to perform five research tasks — two known-item searches, two subject searches, and one search that we hoped would prompt use of an eac-cpf record to locate the information. users were encouraged to speak through their process, explain how they chose their search terms, and talk through their clicks. as they performed these tasks, three of us from the development group kept notes on their process. we recorded areas of fluency and areas of hesitation with the system, and interrupted the process periodically with questions about what kinds of responses they expected from the system. we also asked a series of questions that tried to get a sense of not just how well the user was able to interact with the system, but how it felt to do so. one research prompt illustrated the evolution of our system particularly well — we asked undergraduate students, public services archivists, reference librarians, and a professional researcher whose field of study is the cold war era in the united states (with a particular emphasis on john foster dulles) the following question: “i’m writing a paper about john foster dulles, and how his personal religious convictions may have informed his evolving relationship with ngo dinh diem. what do you have?” in the previous finding aids site, a search for “john dulles diem” would return collection-level information such as the titles of the finding aids, abstracts, creators, and information about how to cite the collection and where it is (see figure 3). however, since the match might occur anywhere in the finding aid, the actual search terms were not necessarily displayed on the results screen and the exact relationship between the search terms and the search results wasn’t always apparent. in this example, there’s no mention of “diem” on the results screen. clicking through, a user looking for information about john foster dulles’s religious beliefs would be forced to search (or control+f) through a fifty-thousand-word finding aid for relevant material. figure 3. search results for keywords ‘john dulles diem’ in the old finding aids site. entire collections are returned, but the relationship with the query is not apparent. in the prototype, our professional researcher first navigated to the john foster dulles papers before unsuccessfully clicking around the finding aid (and finally searching within the collection) for information about ngo dinh diem. along the way, he discussed very sophisticated strategies about how to approach the kind of interpretive analysis that the prompt required — perhaps reading dulles’s diaries for mentions of his religious beliefs, or looking through correspondence about or with diem, or about or with religious groups. this is important to note, because archival description does not necessarily include all the possible keywords that might lead to discovery — the researcher does need to develop a research strategy based on the possibilities of synonymy and aggregate description because an archivist may not have used the exact search term the researcher has in mind to describe materials about that subject. we found in our testing that experienced archives users have a tendency to assume that the best place to find information on a person is within that person’s records. that is, if a person is looking for material about john foster dulles, he typically looks for the john foster dulles papers. and considering the search environments that we have been providing, where it is difficult to find records about a person when that person isn’t the collection creator, this search behavior is rational. the flip side of this behavior is that relevant information from unlikely sources tends to stay hidden. as is common, our library holds more than one collection about a historical figure (in this case, an oral history collection, records of other members of the dulles family, as well as records collected as part of a different institutional context than dulles’s personal papers). a search across the prototype for terms about both dulles and diem clarifies the relevancy of the results because users can immediately see which parts of each collection might be most interesting to them, and the keywords appear in the results to provide some context. an additional search for only ngo dinh diem reveals wondrous possibilities, including political cartoons from an artist’s collection and the records of the council on foreign relations, a public policy group that would probably have an interesting take on what cold warriors thought of the vietnamese leader (see figures 4 and 5). figure 4. search results for “john dulles diem” allow the user to go directly to the relevant part of the finding aid while still providing context for where it is located within the finding aid. though off-screen, relevant components from other collections show up in the search results. figure 5. the component-level result. users can ask a question about this series-level component, which will be sent to the reference archivist, or make a comment for others to see (at bottom of screen, not shown). in this case, the components have children, which can be sorted in the table below. integration with aeon, a special collections circulation tool, allows users to request the components directly from this page. the group is discussing a similar feature for digitization-on-demand. during testing, we were surprised that differences between the prototype and previous site enabled basic users to find and obtain relevant results quickly, with less second-guessing and angst than professional archivists and researchers. we recognize that our staff-users might need to un-learn the strategies that helped them find materials previously, but also that there’s no such thing as a right way to use a system — there are just ways that our users find easy, and ways that they find difficult. user testing forced us to accept user habits as they are, rather than as we would like them to be. some users preferred to interact with the system in ways that we would consider inefficient — or at least they didn’t make use of new functionalities. but by regarding feedback as both a reality and a challenge, we were able to re-think design to highlight features that weren’t immediately apparent to users. and we found that even though the major push behind the new architecture of this site was to make the archival component the atomic unit of description, some users really did prefer to see the finding aid as a long document. we chose to respect these kinds of preferences, understanding that practices would change as users became used to new tools, and made a decision not to remove any existing functionality. we added the option of “single-page” views — html and pdf renderings of the entire finding aid — for users who preferred that experience. use of analytics to discover usage patterns any attempt at improving ux through usability testing should also be supplemented by quantitative data about user behavior. before this project, there was no record of use outside of apache logs, from which it was difficult to extract actionable information. one of our goals was to get a better idea of user behaviors and search strategies with google analytics. component atomization allowed us to track user interest at a more granular level because out-of-the-box google analytics gathers data based on unique urls. however, google analytics defaults did not allow us to track other new functionality we added, such as in-page, javascript-based interactions. to really understand user behaviors, and ultimately how successful our solutions were, we needed to track clicks in a more strategic way to gather more actionable data, something we could never have done with traditional web logging software. how many users were using the collection-level information tabs? how many used the image viewer? were they requesting multiple items at a time, or single items? kirk hess’ article in code4lib journal, discovering digital library user behavior with google analytics, helped us figure out how to implement “event tracking” and gather the data we need (hess k, 2012). prototypes in production there’s no reason why we can’t think of some pieces of our production site as an ongoing prototypes, so long as critical functionality is not at risk. for example, we were not able to really test our commenting system due to the low number of users we recruited for discount usability tests and the type of behaviors we were expecting. we could not easily create testing scenarios where the user knows more about the materials than the archivist, and then capture comments, reminisces, interpretations and disagreements in a natural setting. making users’ interactions with materials a two-way street was a central goal of this project. fortunately our solution required little development effort, allowing us the luxury to test it out mostly in production. building our own commenting system was out of scope of the project, and we did not want users to have to create new logins simply to make a comment. we thought it was more likely that conversations would take place on our users’ own turf, in the social networks where their peers hang out. at the same time, we wanted to maintain a reference to the object of discussion for tracking purposes. to this end, we were able to use intense debate (http://intensedebate.com), a cut-and-paste javascript commenting system. this system allows users to use a twitter, facebook, openid or google credentials, and then share their comments from the finding aid across other systems, increasing visibility of the site. though the jury is still out on how much of an impact the comments feature will have, within days of releasing the beta site, we received feedback about description errors on components that we might never have discovered otherwise. conclusion when it comes to improving the user experience of our information systems, it is no secret that the library and archive community has a lot of catching up to do with our peers in the private technology sector. many attribute this problem to shrinking budgets and slim it staffs. but when most internet startups are short on both money and staff, it seems hard to make the case that these challenges hinder innovation. what has really slowed us down is the enormous weight of our past, whether it is in the form of third-party software lock-in, domain jargon, or outdated organizational processes that stifle innovation. prototypes can be persuasive tools for proposing changes within an organization through “imagine if” scenarios. they give us the freedom to test, fail, and improve along the way — all without damaging a reputation for success that maintains trust. prototypes not only tell us how to enhance the online experience, but can provide a way to improve the overall organizational environment as well. they take proposals out of the abstract and provide a concrete example of how such changes will improve our quality of service, and how exactly we may need to adapt, making big changes much more palatable to our managers. we learn about our users by entering their world and seeing how they solve their own research challenges. this opens up our perspective and allows us to question legacy practices that prevent them from reaching their goals. on the other hand, changing those practices can create flux at all levels of our organizations, and empathy with managers and colleagues is just as important as empathy with our “end users”. if we focus on creating a culture of empathy within the organization, we will be more likely to create an environment for innovation in our libraries and archives that thrives on collaboration, iteration, and managed risk. notes [1] what we consider a prototype is based on eric reis’ notion of a minimum viable product (mvp), which is that version of a new product which allows a team to collect the maximum amount of validated learning about customers with the least effort. http://www.startuplessonslearned.com/2009/08/minimum-viable-product-guide.html [2] we encourage browsing the final site at http://findingaids.princeton.edu to supplement the screen shots we have included in the article. [3] much of our process was informed by thinking developed by the ifla’s functional requirements for bibliographic records report. [4] pitti, daniel v. “technology and the transformation of archival description.” journal of archival organization 3, no. 2–3 (2006): 9–22. (coins) [5] nimer, cory. applying inheritance: single-level displays and repurposable metadata. society of american archivists 2010 research forum. http://www2.archivists.org/sites/all/files/cnfinal.pdf [6] oclc’s thoughtful examination of the potential of scanned material in special collections libraries informed our thinking. schaffner, jennifer, francine snyder, and shannon supple. 2011. “scan and deliver: managing user-initiated digitization in special collections and archives.” dublin, oh: oclc research. http://www.oclc.org/research/publications/library/2011/2011-05.pdf [7] http://eac.staatsbibliothek-berlin.de/ [8] dan santamaria provided skilled and knowledgeable leadership for this project and was a member of our user testing team. john delaney, regine heberlein and don thornbury provided knowledge of archival descriptive practices and user needs; jon stroop, our metadata analyst, was responsible for the system architecture. [9] http://www.agilemodeling.com/artifacts/userstory.htm [10] https://trello.com/board/finding-aid-feedback/4fe4801276ca124c1029e650 references alexandre n. tuch, eva presslaber, markus stoecklin, klaus opwis, javier bargas-avila. (2012). the role of visual complexity and prototypicality regarding first impression of websites: working towards understanding aesthetic judgments. international journal of human-computer studies, vol. 70(11), pp. 794-811 available at: http://research.google.com/pubs/pub38315.html fried, j. (2008). why we skip photoshop. 37signals. available from: http://37signals.com/svn/posts/1061-why-we-skip-photoshop/ hess, k. (2012). discovering digital library user behavior with google analytics. code4lib journal. available from: http://journal.code4lib.org/articles/6942 hess, w. (2012). user experience is not enough. available from: http://whitneyhess.com/blog/2012/04/21/user-experience-is-not-enough/ krug, s. (2000). don’t make me think! a common sense approach to web usability. indianapolis, ind: macmillan usa;. (coins) mathews, b. (2012). think like a startup: a white paper to inspire library entrepreneurialism. vtechworks, virginia tech university library. available from: http://vtechworks.lib.vt.edu/handle/10919/18649 nielsen, j. (2000). why you only need to test with 5 users. jakob nielsen’s alertbox. available from: http://www.useit.com/alertbox/20000319.html about the authors maureen callahan is the public policy papers project archivist at the mudd library at princeton university. shaun ellis is the user interface developer for digital initiatives at princeton university library. subscribe to comments: for this article | for all articles 2 responses to "prototyping as a process for improved user experience with library and archives websites" please leave a response below: emergency mobile mechanic, 2025-07-18 you made a complicated idea feel approachable and kind. that’s something special. feel free to explore my website if you’d like more of that calming, thoughtful tone. skip hire basingstoke, 2025-08-19 this information was very good regarding with ui/ux interfaces, thank you for this one! leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – editorial introduction: a brand new year mission editorial committee process and structure code4lib issue 27, 2015-01-21 editorial introduction: a brand new year by terry reese happy new year!  issue 27 marks the first issue of 2015, a fresh start to a new year.  and an interesting year i think it will be, for both the code4lib community and the journal; especially the journal as we embark on our first special issue with a guest editorial committee and […] by terry reese happy new year!  issue 27 marks the first issue of 2015, a fresh start to a new year.  and an interesting year i think it will be, for both the code4lib community and the journal; especially the journal as we embark on our first special issue with a guest editorial committee and editors.  but i’d like to talk about the community, specifically something that has been so unique to the code4lib experience:  it’s annual meeting.  this year marks the 10th anniversary of the code4lib annual conference, returning back to oregon – a much bigger and more influential event than i think anyone could have imagined when it first started. i remember the first annual meeting in corvallis, or.  the conference was smaller then – with about 80 in attendance.  it started as an experiment of sorts, one in which nearly every attendee in the conference would also be a participant – presenting as part of the main program or in one of the many lightning talks that made up the day.  i can’t remember ever being at an event where so much code was being written live on stage by presenters as they demonstrated projects or took ideas presented earlier in the day and ran with them.  after all this time, i don’t remember the talks, but i remember the atmosphere, the people, and the excitement of that first event.  i also remember thinking how difficult it would be to replicate the magic from that first event and wondering if this type of grassroots experiment in conferencing could actually be sustainable. and yet, here we are…10 years later, going back to oregon, a much larger and stronger community.  has it changed?  oh yes – the conference has grown with the community and it’s become a bit more polished, and i’m sure this will finally be the year that the wifi works throughout the conference week, knock on wood, but at its core, i still see the same magic that made this event so memorable for me back in 2006.  so whether this is your first or tenth annual meeting, find the thing that interests you and just jump in. overview of issue 27 i’m particularly pleased with the articles that made up this issue of the journal, as it includes a little bit of something for everyone. for the diyer’s among us we have three articles: digital forensics on a shoestring: a case study from the university of victoria by john durno and jerry trofimchuck homegrown worldcat reclamation: utilizing oclc’s worldcat metadata api to reconcile your library holdings by sarah johnston using google tag manager and google analytics to track dspace metadata fields as custom dimensions by suzanna conrad. for those interested in workflows and best practices, we have: using semanticscuttle for managing lists of recommended resources on a library website by tomasz neugebauer, pamela carson, and stephen krujelskis training the next generation of open source developers: a case study of the osu libraries & press’ technology training program by evviva weinraub lajoie, trey terrell, and mike eaton. and finally, for those interested in how things just work, we have two articles: communication between devices in the viola document delivery system by theodor tolstoy query translation in europeana by péter király.             subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – consortial-based customizations for new primo ui mission editorial committee process and structure code4lib issue 34, 2016-10-25 consortial-based customizations for new primo ui users interested in customizing their primo installation are required to configure specific settings, files, and code during the view setup process. a consequence of this is that unique customizations are not easily sharable between institutions. with the release of the new primo user interface, ex libris has enabled institutions to manage interface customizations via the package customization manager. in the summer of 2016, an orbis cascade alliance working group investigated the efficacy of the package manager as a means of centrally sharing and deploying orbis cascade alliance primo toolkit customizations. by virtue of passively loading customizations to the central package, each institution could pass custom parameters with local js in order to adapt central customizations to the specific needs of that institution’s users. this article will address both the potential and the limitations of the primo package customization manager. it will also provide best practices for consortia seeking to centrally manage and share primo enhancements and it will identify areas of future development for centrally shared customizations. by dan moore and nathan mealey introduction the orbis cascade alliance is a network of 39 resource sharing institutions in washington, oregon, and idaho that migrated to a shared ils alma/primo installation over multiple years from 2013 to 2016. due to the nature of the shared ils, it became necessary to establish teams and working groups to address particular issues, identify areas for development, and experiment with configurations in various components of both alma and primo. each institution is granted their own primo instance that they may customize to fit the needs of their local users. users interested in customizing their primo 4 installation are required to configure specific settings, files, and code during the view setup process, sometimes requiring users to upload individual css files, html footer and header snippets, javascript, and additional content files independent of one another. a consequence is that customizations are not easily sharable between institutions; similarly, institutions using multiple views of primo for different libraries are required to manually implement each customization into each view of primo. further, customizing the interface requires a relatively high degree of familiarity with primo architecture, markup languages like html, and programming languages like javascript. with the release of the new primo user interface in august 2016[1], ex libris has enabled institutions to manage the new angularjs-based primo ui via the package customization manager [2]. additionally, the package customization manager allows consortia to share customizations via a centrally configured package. using the package customization manager, users are able to organize their customizations with a locally managed repository uploaded as a .zip archive that may then inherit and build off a consortial central package. in the summer of 2016, the primo consortial configuration working group (pccwg), an orbis cascade alliance working group consisting of four individuals from different alliance member institutions, investigated the efficacy of the package customization manager as a means of centrally sharing and deploying orbis cascade alliance primo toolkit customizations. by loading customizations to the central package, each institution could pass custom parameters with local javascript in order to adapt central customizations to the specific needs of that institution’s users. this article will address both the potential and the limitations of the package customization manager. it will also provide best practices for consortia seeking to centrally manage and share primo enhancements and will identify areas of future development for centrally shared customizations. background in 2015, the orbis cascade alliance initiated a project to develop a suite of shared customizations for the primo 4 user interface. motivated by a desire to take advantage of work that had already been completed and tested by several individual member libraries to customize their primo instances, the project’s goal had been to determine how best to share these customizations so that libraries throughout the consortium could selectively take advantage of them. the principal challenge with this effort was the practical reality that different libraries within the consortium have different levels of technology skills available among their staff. while some consortium libraries have a number of technology staff and several developers, others have no dedicated technology staff at all. thus, the solution the project arrived at needed to accommodate and overcome this challenge as best as possible. the resulting solution, the primo toolkit, offered a curated, browseable, and searchable inventory of customizations for the primo 4 interface that had been developed by alliance libraries. the items included had been prioritized by staff at member libraries, and so targeted those functionalities of interest to the greatest number of libraries. overall, the primo toolkit sought to accomplish the goal of sharing a range of primo customizations such that as many libraries could take advantage of them as possible, by ensuring that the documentation and approaches used for the shared items was as consistent as possible. terminology, included elements, and consistent references were standardized across items, and a great deal of effort was put into providing step-by-step instructions that were as clear as possible. ultimately, the hope was that even a novice programmer or website admin could implement the majority of the customizations. relative to the possibilities raised by the customization package manager in the new primo ui, the approach embodied in the primo toolkit is very limited. but the project laid the groundwork for the primo consortial configuration working group by creating a curated list of customizations to target, and firmly establishing the goal of sharing customizations across alliance libraries as effectively as possible. technology figure 1: customization package manager. the customization package manager allows users to manage a .zip archive of css, html, javascript, and image files that are loaded into the new primo ui and override default ex libris styling. it is accessed via the primo back office (above figure). prior to the new primo ui, customizing primo required users to upload individual html and css files that were then selected during the view setup process. with the new primo ui, customizations are collected into one archive that can be more rapidly deployed and shared. each institutional package is associated with a view. ex libris provides users with a template package that contains empty files in which users may add new customizations. this package may act as a model for the view’s package or can be used to restore a view to default configuration. upon unzipping the archive, the user will discover a root folder that is by default named view_code. it is imperative that the name of this folder be identical to the view to be customized. in the example above, the root folder of the archive would need to be named “osu” in order for the system to accept the file. live validation is present in the system so an uploaded package with an incompatible view code will be rejected. users zipping the archive on a mac machine would also need to confirm that any .ds_store metadata files are not present; at the time of this experiment, the primo back office automatically rejects any .zip archive with this file in it. at the time of our testing, packages were organized in the following manner: figure 2: organization of packages. the organization of the package is hierarchical and intuitive. all css rules are stored in custom1.css, all edits to the landing page of the new primo ui are stored in home_en_us.html (although header navigational items are still controlled by the main menu tile in the primo view configuration wizard), custom images are stored in the img folder, and javascript is stored in custom.js. this structure is identical between central and institutional packages. on the customization package manager screen, users configuring packages and uploading for deployment may toggle “inherit from central package” to load the central package into the selected view. participation in the central package is voluntary and is configured on the specific view. this allows greater freedom for individual institutions to have separate views that might not require the central package for local needs or for local testing. for example, if an institution uses shallow client browsing terminals that would not benefit from the heavy customization meant to address users who are logging longer search sessions, it may select not to inherit customizations from the central package. at the time of our testing, additional files in any folder except the img directory were not actionable by any means in the new primo ui. the group tested with additional html and css files and were not able to call or reference those files during page load. as this is an evolving feature of the new primo ui, which itself will undergo another major release update in november 2016, it may be possible in the future to host additional css, html, or javascript files in the package. recommendations for consortial partners the package customization manager, both local and central packages, offer great value to consortia seeking to share enhancements; additionally, individual institutions seeking to take advantage of consortially loaded customizations will benefit from reduced development and implementation time. despite the low early adoption rate of the new primo ui in alliance libraries, we have identified multiple use cases for the central package. these include: deploying material type icons centrally so that resource type graphics are unified across the alliance; publishing css styling to make each institution’s resource sharing request forms similar; loading javascript triggers for hiding unneeded actions or adding new ones based on popular alliance resource sharing workflows. at the time of our testing, however, we discovered that any customizations made to the html template in the institutional package would entirely override any customizations made to the html template in the central package. it is not feasible to use this technology in its present state to deploy static html customizations. the central package may be used to deploy a consistent set of images throughout the consortia as a means of standardizing the display of common resource types as well as other common primo features. we foresee several situations where this will be a virtue. with resource icons standardized, screenshots of search results may be shared openly throughout the consortia; barring significant style choices that might make one institution vastly different from another (such as a different color scheme or font styling), it may be helpful to crowd-source asynchronous training modules that can then be deployed to both local and distance users of the discovery layer. a standard set of material type images will support the alliance’s resource sharing program by reducing barriers for users at any consortial library. furthermore, standardizing resource type icons with a consistent set of images that are curated by the consortial partners itself means future changes to image sets published by ex libris may be safely disregarded. it is important to note that the /img directory within both the central and institutional packages can act as a file store for custom images used throughout the primo interface. early experiments revealed this to be a strong candidate for hosting infographics or other media used within the discovery layer; however, due to the upload times of the central package and timelines set forth for the experiment, the authors did not test the limits of either file size or file types stored in this directory. due to the nature of and need for individual styling at member institutions, we foresee the value in shared css customizations to be less than the value of shared images or custom javascript. however, there is still value, based on resource sharing workflows within consortia, in using the centrally loaded style guide to standardize the display of request forms across all member institutions. the central package is an ideal candidate to deploy any style fixes to the default primo layout that may boost accessibility. it is important to note that the institution package css file is loaded last and will override both style rules in the native new ui as well as rules in the central package. this affords individual institutions greater freedom to break away from the pack, but we recommend this only if the users of that institution will benefit more from those unique customizations than adhering to consortial styling. centrally shared javascript offers exciting promise for the future of the new primo ui. early experiments within the working group have demonstrated that javascript functions stored within the central package can be triggered, modified, and customized by javascript code stored in the institutional package. an early experiment conducted by the working group began by uploading a basic javascript code snippet to the central package’s custom.js file: function writetoconsole(p1) { console.log("triggered from central package by "+p1); } in the local package’s custom.js file, the function was called with additional parameters: writetoconsole("osu"); upon page load, the console records the correct log declaration: figure 3: console log entry. one promise of this feature is that it is plausible for the central package to act as a code repository for consortium-wide javascript customizations. individual institutions can modify their institutional package to enable those customizations and configure how those customizations work. for example, one popular javascript enhancement shared through the orbis cascade alliance primo toolkit is adding a “report a problem” link to the actions menu on the brief record and full record pages.[3] clicking this link redirects the user to a locally-configured drupal (or other cms) form. within the new primo ui and taking advantage of the central package, the javascript code for this enhancement can be loaded and shared natively with all members. members can then modify the central code by supplying different parameters that will redirect users to the appropriate forms. furthermore, additional code in the local package might enable that institution to gather specific data from other angular data modules. further experiments are required to determine the efficacy of using the package customization manager to centrally deploy javascript code for this purpose but early results are promising. in order to make the most of this technology, it would behoove consortia to develop and organize task forces charged with directing development, managing customizations, and training users. specifically, the pccwg identified three primary goals for the fall and winter of 2016: develop training modules to initiate alliance users into angularjs, codify best practices for creating and publishing customizations via the central package, and investigate a greater extent of use cases to explore. depending on the current status of a consortium’s shared implementation of primo, it may be advantageous to commit to developing a central package as part of the consortial agreement. conclusion adopting the customization package manager as the vehicle for sharing customizations within the alliance is a promising venture. we have identified a few areas that require special consideration before widespread adoption can take place. when transitioning from developing for primo 4 to the new primo ui, a primary hurdle is adapting to the angularjs framework. while this will impact future development of new enhancement, it also requires a review of existing customizations published in the primo toolkit. users who are already live with primo and have customized their ui extensively should pay particular attention to the compatibility of jquery and angularjs; it was discovered in our experiments that several popular jquery-based customizations were not compatible with the new primo ui and would have to be rewritten. as discussed above, the virtue of distributed development of customizations is an attractive quality of the package customization manager. developing across the consortium in this manner saves precious development time and resources by avoiding ‘reinventing the wheel.’ this also allows the consortium to share with its smaller institutions, or institutions that lack a dedicated development team or technology specialists, attractive customizations that enrich the primo experience. it would benefit consortia to create internal training modules to acquaint library programmers and primo users to angularjs. while a significant portion of a primo view are still customizable with the views configuration wizard in the primo back office in the new primo ui, the entire front-end has been built using angularjs and makes use of its directives to display information. librarians and programmers accustomed to the current primo 4 interface will need to understand how angularjs works at a fundamental level in order to best write customizations for the new interface. additional experiments are necessary to confirm compatibility and best practices with current widgets and external resources that some members of the global primo community have programmed into their primo 4 ui. the new primo ui does not supplant or replace the existing primo 4 ui. this concurrent release and support on ex libris’ behalf allows institutions individually and consortia as a group to decide when to go live with the new user interface. considering the significant changes the new user interface present as well as the new technologies that underlie it, it may benefit institutions and consortia to maintain the existing primo 4 ui while current development on the new ui is underway. ultimately, the primo consortial configuration working group is optimistic that the primo customization package manager will enrich the shared primo experience throughout the orbis cascade alliance. notes [1] early discussion of the new primo ui referred to the new user interface as “primo 5.” ex libris has stressed that the new user interface is a separate release and the authors will maintain the separation between the new primo ui and primo 5 in this article. [2] documentation may be found here: https://knowledge.exlibrisgroup.com/primo/product_documentation/back_office_guide/090primo_utilities/the_ui_customization_package_manager [3] “add a “report a problem” link in primo actions menu”. https://www.orbiscascade.org/blog/9/?bid=119 about the authors dan moore is discovery services librarian at oregon state university, dan.moore@oregonstate.edu. nathan mealey is manager of library technologies at portland state university, mealey@pdx.edu. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – supporting library consortia website needs: two case studies mission editorial committee process and structure code4lib issue 57, 2023-08-29 supporting library consortia website needs: two case studies louis: the louisiana library network provides library technology infrastructure, electronic resources, affordable learning, and digital literacy support for its 47 academic library members. with this support comes a need to develop web solutions for members, a challenging task as the members have their own websites on a multitude of platforms, and a multitude of library faculty and staff with differing needs. this article details two case studies in developing consortia-specific web design projects. the first summarizes the louis tabbed search box order form, an opportunity for members to “order” a custom-made search box for the various services louis supports that can then be embedded on their library’s website. the second involves the louis community jobs board, a member-driven job listing tool that exists on the louis site, but that members can publish jobs to using a google form. both the search box order form and the jobs board have resulted in increased engagement with and satisfaction from member libraries. this article will include best practices, practical solutions, and sample code for both projects. by elizabeth joan kelly introduction louis: the louisiana library network plays a vital role in supporting the online needs of its 47 academic libraries. through its affordable learning, digital literacy, resource, and technological initiatives, louis ensures that members have the necessary collections, tools, and infrastructure to deliver quality services to their respective communities. member library websites are a key component of making these resources and services available to users, and each member library operates its own website, utilizing different content management systems, design frameworks, and customization options.  however, many academic libraries lack the staff, expertise, and/or time to engage with their websites as fully as they would like. supporting the website needs of consortia members provides unique challenges, since consortia staff don’t usually have editing privileges on member libraries’ websites but still need to provide solutions to ensure the library’s patrons can access the library’s search tools, electronic resources, and more. this article examines two case studies that illustrate the successful implementation of consortia-specific web design projects within the louis network. the first, the louis tabbed search box order form, provides an opportunity for members to “order” a custom-made search box for the various services louis supports that can then be embedded on their library’s website, enhancing the user experience and facilitating access to louis resources. the second, the louis community jobs board, is a member-driven job listing tool that exists on the louis site but that members can publish to instantly and conveniently using a google form. both the search box order form and the jobs board have resulted in increased engagement with and satisfaction from member libraries. this article will include best practices, practical solutions, and sample code for both projects. this will hopefully prove beneficial to other consortia and library networks in developing similar web solutions that cater to the unique needs of their respective members. background louis was founded by the library deans and directors of louisiana’s public and private college and university libraries in 1992 to ensure cost-effective attainment of library technology and resources. member libraries range from research-intensive universities to rural community colleges, to small seminary libraries, and everything in between. louis’s main technology support comes in the form of the library services platform (lsp), comprised of the sirsidynix symphony integrated library system, including enterprise discovery; ebsco suite’s discovery service (eds) and publication finder; worldcat discovery service; and atlas systems’ illiad resource sharing management software. louis also works with members to reduce the cost of instructional materials through the affordable learning louisiana initiative; to address digital inclusion needs, especially digital literacy, via our digital inclusion pilot; to provide professional development opportunities; and to build a community of practitioners across the state. louis’s bylaws outline criteria for membership. accepted members pay annual fees and participate in consortial governance. members may use some or all the previously mentioned services through their louis membership. because of the wide range of needs created by such a diverse set of institutions, student bodies, and faculty and staff, louis’s core staff of 11 provides a wide and ever-evolving range of services to meet the needs of the membership. tabbed search box order form in 2021, louis engaged in a self-assessment to identify the consortium’s functional needs from the lsp, survey satisfaction with the current lsp, and to determine if the consortium should move forward with a request for proposal (rfp) process or pursue a new contract with the existing lsp provider. data used in the assessment includes the results of a member survey, usage data from the lsp, data from louis’s support ticket system, and data from sirsidynix’s training portal to assess on which products and features members desired training. one outcome of the lsp assessment was that members were not satisfied with existing tools provided by the consortia’s discovery systems to create search boxes for their websites. while louis staff sometimes assisted members in such projects, it was at the request of the members, and there had not been any concerted effort to more broadly offer this as a service to all members. in addition, the lsp assessment usage data showed that libraries without a search box on their website, unsurprisingly, had fewer searches in eds and enterprise than libraries with a search box, even when accounting for size of the institution. member websites were coded as to whether they had a search box with a primary tab for searching eds, a search box with a primary tab for searching enterprise, separate search boxes for eds and enterprise, no search box, or summon (one library which subscribes to summon outside of their louis membership). the assessment showed that institutions with an eds search box have greater eds and enterprise search statistics than the other options, and unsurprisingly these institutions have far more eds searches than enterprise searches. institutions with both an eds and an enterprise search box also have more than double the amount of eds searches than enterprise searches on their site. institutions with an enterprise search box and institutions with no search box have greater enterprise searches than eds searches.  a related study found an inverse relationship between database usage and embedding a generic nc live (another statewide library consortia) search box on an academic library website; while there is not a definitive correlation between the two, the authors theorized that this (among other features) “point to a less customized library website, which could mean that these libraries are lacking staff time or technical expertise to help their users get directly to database content, perhaps resulting in overall lower use” (morris and guhde, 2014). figure 1. number of searches by louis member libraries with different search box types on their website in the report to the membership overviewing the results of the lsp assessment, louis staff recommended that member institutions looking to increase discovery of their resources may wish to consider implementing an eds search box on their website. as a recommendation for further action, louis committed to providing training as well as templates for creating and customizing search boxes. initially, the hope was that louis could create a tabbed search box builder, like the ebsco search box builder [1]. while both eds and sirsidynix make tools and code available to create basic search boxes for their products, it was clear from previous member requests that members were not always sure what configuration details they needed to include in their search boxes. and even once the generic search boxes were created, customizing them to visually fit and function correctly within the member’s library website proved challenging, especially for libraries with less staff. thus, the decision was made to instead create a search box order form with guided options for members to fill out. louis would then create a custom-made search box based on their specifications, test it for accessibility, and test it embedded within their website using browser developer tools. the tabbed search box order form walks members through a series of questions about what they would like included in their search box. members are given the option to include two different colors in their search box, a primary color and a secondary color. these colors are checked using webaim’s contrast checker [2] for sufficient contrast against the library’s website homepage as well as for appropriate text color to use. figure 2. louis’s tabbed search box order form members can select to include tabs for eds, enterprise, ebsco’s publication finder, an a-z databases list, or other services they would like featured in the search box (pending louis’s ability to add and configure these). depending on which tabs they select to include, they are asked further questions such as whether they want to use authentication for their searching; whether to include selectors, like keyword, subject, or isbn; and whether to include limiters, such as full text or peer reviewed. there is also space in the form to make notes about other features or concerns. after the member has submitted the form, louis staff open a ticket with them in the ticket management system, clarify any questions, and send a first draft of their search box within a week. the draft includes a working prototype on the louis website which they can use to test functionality and a screenshot of what the search box will look like once embedded in their website, using chrome’s developer tools to visualize the change. the code for the search box is analyzed using both achecker’s web accessibility checker [3] (which can check against wcag 2.0 (level aaa) guidelines) and the wave web accessibility evaluation tool [4] (which uses the wcag 2 checklist) to find and correct accessibility and web content accessibility guideline (wcag) errors before sharing. the search box is also tested using multiple browsers and an online responsive checker [5] for multiple devices. because of the difficulty in anticipating what features or functionality a member website might or might not have and how these might interfere with the search box code, a general template is used to mitigate potential issues. the search box uses only bootstrap, css, and html for the search box code. figure 3. example ticket to a member library with a screenshot of what their new search box will look like when embedded on their homepage for ease of modifying the template, css variables [6] are employed to change out the member’s requested color scheme. for example: :root { --primarybackgroundcolor: #c4d8e2; --primarytextcolor: #00263e; --secondarybackgroundcolor: #ffd700; --secondarytextcolor: #00263e; --tabhoverbackgroundcolor: #003743; --tabhovertextcolor: #ffffff; } the tabbed search box order form was promoted in the louis lately biweekly newsletter, in monthly watercooler meetings for the entire membership, and at the biannual system administrators meeting. since its release in january 2022, ten members have requested search boxes, and eight of those are functioning as the primary search box on the member’s website (two members have not yet implemented theirs). of the ten members who have requested search boxes, only one member did not previously have a search box on their website. three had single use search boxes (either enterprise or eds), and the remainder already had a combination tabbed search box but were interested in redesigning it. there are still ten members in the consortia who do not have search boxes on their websites, so next steps will include direct outreach to system administrators at these libraries to see if they would like to request a search box. community jobs board another recent project is the louis community jobs board [7]. in 2022, after many pandemic-related statewide restrictions on in-person meetings began to be lifted, louis staff “hit the road” at the request of members and paid visits to member libraries around the state. a frequent topic of conversation at these visits was the difficulty members were having hiring open positions, of which there were many. retirements, organizational restructuring, and the great resignation were all contributing to an abundance of available jobs at louis member libraries. louis has long advertised employment opportunities at member libraries in its biweekly newsletter, but the need for a quicker and more easily accessible means for libraries to advertise their positions to a wider audience became clear. the idea for a jobs board that would live on the louis website but that would allow submissions from members evolved over the course of several team meetings and small group consultations. in outlining requirements for the jobs board, the following were identified as priorities: either use louis’s existing website infrastructure or integrate easily into it allow posting by members, ideally without having to create an account or login to a new system automatic takedown of old job postings after a set period of time in addition, the option to subscribe to new job postings was identified as a “nice to have” but not an essential feature. louis’s website [8] uses springshare’s libguides cms [9] as its content management service. ideally, the jobs board could live on the louis website, and members could post new jobs without needing to ask louis to add their account to louis’ libapps. while libguides’ native functionality didn’t seem to include a solution that exactly matched louis’s requirements, the libguides discussion board [10] feature came close. the jobs board would be a public discussion board, and css could be used to modify its appearance to more closely match some aspirational job boards, such as the code4lib jobs board [11]. while users need to have a libguides account on the hosting website’s libapps platform, there is an option to allow self-registration and to limit to approved domain names. using this tool, louis staff could pre-populate the approved domain names with the domains for all of our member institutions. while members would need to create an account to post their job, their registration would be approved immediately so long as they used their institutional email. anyone with either a louis libguides account or a patron account could sign up to be notified of new posts. this would allow louis staff to monitor any possible spam that might come through. it also meant that job hunters could receive notifications for new job postings, though they’d either need to already have an email address from an approved domain, or request for louis staff to manually create a patron account for them. this would need to be a larger discussion, as allowing email registrations for non-louis members would necessitate a change to existing policies. figure 4. louis jobs board draft using libguides’ discussion board the result met most of the functional requirements, and beta testers from member institutions said the registration process was easy and quick. but because the model was trying to retrofit a discussion board format into a job posting format, there were hard limits on how much customization could be made to the styling of the board, and as a result, some of the labeling and functionality were confusing to testers. there wasn’t a way to customize the submission form to require or even encourage the inclusion of certain fields, like salary, location, or contact information. furthermore, most of the visual styling to do simple things like rename the “add new discussion” link to “add new job posting” were achieved using css hacks which might affect the accessibility of the page for users with screen readers (f87: failure of success criterion…, 2016). /* change "add new discussion" to "add new job posting" */ #s-lg-disc-add-button { visibility: hidden; position: relative; } #s-lg-disc-add-button:after { content:' add new job posting'; visibility: visible; position: absolute; top: 0; left: 0; margin-left: 10px; } figure 5. sample code for modifying default discussion board labels with css for a second attempt at a solution, louis staff turned to integrating outside tools with the libguides cms website. instead of using the discussion board feature, members would fill out a google form for job submissions. these submissions would automatically publish to the website using the google sheets api. while this version required quite a bit more setup and has multiple moving parts to make it work, it provided the desired customization flexibility and also removed the need for members to create an account to post. the google form lives in the louis google account. it is customized to include all of the desired fields and to make some fields required. the link to the google form is included in the biweekly louis lately newsletter and is also linked from an instructional article in the louis knowledge base, which louis members can request access to if they don’t already have it. the results of the form are posted to a linked google sheets workbook. the first page of the workbook (titled “jobs”) displays the results of the google form responses exactly as they were submitted, in chronological order. these are copied over to a second sheet using the formula: =sort(jobs!a:k,1,false) to sort the most recent job first. this sheet also includes an additional column, “hyperlink,” with a formula to turn any urls submitted in the “more information” form field (column h) into a hyperlink: =if(h2<>"","<a href='"&h2&"'>more information</a>","") there is a google apps script running on the spreadsheet that finds the current date and deletes any rows with a timestamp in column a older than 60 days. triggers deploy the script once a day at midnight. function dro() { var sh = spreadsheetapp.openbyid("yourid").getsheetbyname("jobs"); const sr = 2; //data starts here // assume first column is date const rg = sh.getrange(sr,1,sh.getlastrow()-sr+1, sh.getlastcolumn()); let v = rg.getvalues(); let d = 0; let dtv = new date().valueof(); v.foreach((r,i) => { if((dtv-new date(r[0]).valueof())/3600000 > 1440) { sh.deleterow(sr+i-d++); } }); } figure 6: google apps script code to delete entries older than 60 days the google sheets table from the sorted sheet is brought into the libguides cms using a slightly modified version of a codepen by duyentnguyen [12] and css from jeff vdovjak [13] on stackoverflow. finally, a simple javascript search box is also included to search the job postings. the scripts and styling are pasted into a media/widget asset [14] in libguides. google forms has an integrated feature that allows for the google account holder to be notified every time the form has been filled out. that has allowed louis staff to monitor submissions to the board and ensure that any spam submissions are taken down (though so far that has not been an issue). if members need to edit their job posting after submitting the form they can do so immediately if they have not yet closed the submission window; if they have, they can contact louis staff, who can edit the google sheet with the responses on it. figure 7. louis community jobs board this new and improved version of the jobs board has proven highly successful since its debut in september 2022. since then, the form has been submitted sixty times, and the jobs board webpage was the 11th most viewed page (out of 1024 published pages) on the louis website from october 2022-june 2023. in a separate project involving a redesign of the louis website, members were surveyed for feedback about the current website as well as desired features on the new website, and asked to work in small groups to identify the pages on the website they used the most. in both activities, users mentioned the jobs board as being a welcome recent addition to the site. the full code for the media/widget, as well as instructions, are in github [15]. conclusion while supporting the web needs of a statewide academic library consortia poses unique challenges, the benefits to doing so make the effort worthwhile. the louis tabbed search box order form and community jobs board both use streamlined processes and templates to make these projects manageable, adaptable, and reusable. both projects have been embraced by the louis membership and resulted in solutions to longstanding consortia needs. next steps for the search box order form include continued outreach to members who do not yet have a search box on their website, and analysis of usage statistics for libraries with their first search box to see if usage increases as theorized. the community jobs board will continually be evaluated for its utility and to ensure that it is updated to meet new accessibility requirements. as louis engages in further web endeavors for the louis membership, both projects can serve as a sustainable, responsive model for louis as well as for peer consortia and library networks. notes [1] https://searchbox.ebsco.com [2] https://webaim.org/resources/contrastchecker/ [3] https://achecks.org/checker/index.php [4] https://wave.webaim.org [5] https://www.websiteplanet.com/webtools/responsive-checker [6] https://www.w3schools.com/css/css3_variables.asp [7] https://louislibraries.org/member-resources/jobs-board [8] https://louislibraries.org [9] https://springshare.com/libguides/cms.html [10] https://ask.springshare.com/springboards/faq/1950 [11] https://jobs.code4lib.org [12] https://codepen.io/duyentnguyen-the-lessful/pen/vwxmoem [13] https://stackoverflow.com/posts/69007434/revisions [14] https://ask.springshare.com/libguides/faq/939 [15] https://github.com/elizabethjoankelly/jobs-board bibliography f87: failure of success criterion 1.3.1 due to inserting non-decorative content by using :before and :after pseudo-elements and the ‘content’ property in css [internet]. 2016. w3c® (mit, ercim, keio, beihang); [cited 2023 june 22]. available from https://www.w3.org/tr/wcag20-techs/f87.html morris, j, guhde, e. 2014. making usage data meaningful. serials review [internet]. [cited 2023 june 22]; 40:3, 161-174. available from https://www.tandfonline.com/doi/full/10.1080/00987913.2014.948146 about the author elizabeth joan kelly is the library web and applications development administrator at louis: the louisiana library network. her primary responsibilities include managing the consortium’s web presence, customizing user interfaces and search tools, analyzing usage data, and creating data visualizations to communicate assessment results and inform evidence-based decision-making. previously, she worked for over a decade managing the digital library, digital preservation, digital scholarship, and library web services at loyola university new orleans. elizabeth is an active music composer and has won awards for her instrumental, vocal, and electronic music. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – peripleo: a tool for exploring heterogeneous data through the dimensions of space and time mission editorial committee process and structure code4lib issue 31, 2016-01-28 peripleo: a tool for exploring heterogeneous data through the dimensions of space and time this article introduces peripleo, a prototype spatiotemporal search and visualization tool. peripleo enables users to explore the geographic, temporal and thematic composition of distributed digital collections in their entirety, and then to progressively filter and drill down to explore individual records. we provide an overview of peripleo’s features, and present the underlying technical architecture. furthermore, we discuss how datasets that differ vastly in terms of size, content type and theme can be made uniformly accessible through a set of lightweight metadata conventions we term “connectivity through common references”. our current demo installation links approximately half a million records from 25 datasets. these datasets originate from a spectrum of sources, ranging from the small personal photo collection with 35 records, to the large institutional database with 134.000 objects. the product of research in the andrew w. mellon-funded pelagios 3 project, peripleo is open source software. by rainer simon, leif isaksen, elton barker, pau de soto cañamares introduction: the pelagios initiative pelagios [1] is a community-driven initiative that facilitates better linkages between online resources documenting the past, based on the places they refer to. our member projects are connected by a shared vision of a world – most eloquently described in tom elliott’s article “digital geography and classics” [2] – in which the geography of the past is every bit as interconnected, interactive and interesting as the present. each project represents a different perspective on our shared history, whether expressed through text, map or archaeological record. but as a group we believe passionately that the combination of all of our contributions is enormously more valuable than the sum of its parts. initially focusing on the classical worlds of greece and rome, pelagios has been working with a growing network of partners towards connecting different types of online resources relevant to the study of the past more generally – the literature of different periods and languages, archaeological data and databases, maps and other images, the results of scholarly research, etc. – so that they become more easily discoverable and seamlessly navigable for users. the key to connectivity in pelagios is the use of shared online gazetteers [3] – directories of places that assign each place a unique, stable identifier in the form of a uniform resource identifier (uri). pelagios advocates the idea that whenever you refer to a place in your data, you should do so using a gazetteer uri. this way, otherwise isolated datasets become implicitly joined up to an interconnected graph, with the gazetteers as their central backbone [4], [5]. pelagios has been working on developing best practices and tools to support its vision: e.g. by establishing conventions for the publication of gazetteer metadata as linked open data [6]; or through the development of the recogito annotation platform, which aids the process of linking digital texts and maps to the places they refer to [7]. in this paper, we introduce another outcome of the pelagios initiative: peripleo, a tool to visualize and navigate the growing pool of interconnected open data, published collectively by the pelagios community. our test installation of peripleo [8] presently holds approx. half a million records from 25 datasets. these datasets encompass gazetteers (7 datasets), as well as content linking to them (18 datasets). the smallest dataset is a personal flickr photostream with 35 records; the largest dataset is provided by the american numismatic society and contains approx. 134.000 numismatic records. how peripleo works peripleo is ancient greek for “to sail (or swim) around” in the sense of exploration or discovery. the notion of being able to freely roam the “sea of open data” brought together by our partners (and discovering the treasures hidden in remote places and ancient times!) was exactly the metaphor we had in mind when starting development. a key design goal was to provide, on the one hand, a familiar google-maps-like interface, with full-text search and auto-completion; while allowing a more free-form mode of exploration on the other. in this “exploration mode”, peripleo conveys a sense of the scope, breadth and structure of the data as a whole, by representing geographic coverage on the map; by showing temporal spread as a histogram; and by graphically illustrating distribution across different thematic facets (such as document language or data source). this way, users can easily gain an overview first, and then filter and drill down according to their own interests. an additional goal was that peripleo should function as a service, and provide an api that enables other sites to re-use pelagios data, either by building their own mashups or by embedding and linking to peripleo search results easily. a short example below will illustrate how peripleo works. figure 1. peripleo search result example. fig. 1 shows the result of a search for the term “tetradrachm” (an ancient greek coin type). the search, in this case, produces a total of 21,576 results. the map shows a distribution of dots that indicate where those results are located. peripleo also shows preview images for some of the results. clicking on the “filters” button reveals additional information about how our result set is organized (fig. 2). for example, we can see that most results come from the american numismatic society collection [9]. a small fraction (hovering the mouse over the bars we can find out it is nine results) originates from the fralin | uva art museum [10]. peripleo also shows how the results are distributed over time, between 545 bc and 280 ad, with a peak of finds dated to around 300 bc. figure 2. peripleo search result example, with open filters panel. by zooming and panning the map, it is possible to explore the distribution of the search results in more detail. the information shown in the panels updates in real time. the total result count, temporal coverage, thumbnail previews, the sizing of the map markers which indicates the relative number of results at a place vs. others on the map – everything reflects the situation in the currently viewed map area. fig. 3, for example, shows that the region around sicily holds a subset of 898 results out of the total of 21,576. it also shows how the local temporal distribution differs from the global distribution, being limited to the time range from 500 to 200 bc (as opposed to 600 bc to 300 ad globally). figure 3. results for the area around sicily, with syracusae as local center (indicated by largest dot). in a similar fashion to how peripleo facilitates the exploration of spatial sub-regions of the result set (by zooming and panning the map), it also enables temporal investigation. using handle controls, a specific range can be selected in the timeline graphic. dragging the selection interval across the timeline will, as before, update all the information in real time: result counts, geographic distribution, thumbnail images, etc. as an example, the sequence of images in fig. 4 shows how the spatial “footprint” of the “tetradrachm” search results exhibits a distinct and changing pattern over the period for which we have evidence of the coins’ find spots. figures 4(a-d). exploring changing search result distribution over time, by using the time range selection tool. peripleo features a range of additional capabilities. for example, it is not necessary that each object corresponds to exactly one point location on the map, as in the previous example. it is perfectly possible for a single object to be linked to many – even thousands of – locations. this is, in fact, a typical case for literary texts, where peripleo can even provide full-text search within the document, and render maps based on the places occurring most closely to the search term (cf. fig.5). furthermore, peripleo is not restricted to representing places as points. place geometries can be rendered as point, line, or polygon features. peripleo makes use of this information for the purposes of not only visualization, but also spatial ranking. this way, when focusing the map on crete, for example, the gazetteer record for the island will be ranked among the top relevant objects; whereas, when zooming out to view the entire mediterranean and near-eastern area, a literary geographic work like herodotus’s histories is much more likely to rank among the top hits, as its geographical coverage is a much better match to the currently viewed map area. figure 5. a search for the term “embalming” returns a single result — the histories by herodotus. the map shows places that appear close to the search term in the text. technical architecture peripleo is open source software, available through our github repository [11]. the technical architecture follows a state of the art 3-tier web application model. it is implemented on a jvm (java virtual machine) technology stack, using the play application framework [12] and the scala programming language as a basis for the “middle tier”, which provides, primarily, a json api. the presentation tier is a client-side javascript/html5 single page application built on top of this api. at the same time, the api is also open to the public, and can be used freely by external developers to implement their own peripleo-like interfaces, or re-use and integrate peripleo data into their own applications. in terms of storage, peripleo implements a hybrid approach: a relational postgresql database holds detailed record information, as well as auxiliary tables to speed up specific kinds of queries or drive special detail visualizations (e.g. network visualizations of places co-occurring in documents). the primary storage backend, however, is a set of indices (based on the open source lucene indexing engine), providing fast access to: i) place information; ii) document and item metadata; iii) document full-text search; and iv) auto-completion hints. peripleo makes particular use of spatial indexing functionality and different types of faceting approaches (taxonomy faceting, temporal faceting, 2d spatial faceting) to enable real-time visualization of the thematic, temporal and spatial composition of large search result sets. integrating peripleo making data that differ vastly in terms of size, content type and theme uniformly accessible under a single user interface requires agreements on some of the basic principles of how we express it. pelagios’s approach to this challenge is simple and pragmatic: rather than getting everyone to agree on how to represent the data, pelagios provides a set of lightweight conventions for how to express links between the data and the things described in it. we refer to this approach as “connectivity through common references”. pelagios uses this approach specifically with regard to places. but the approach as such is, of course, equally applicable to other “ordering dimensions” such as people, time periods, events or classification schemes. in principle, any object that is published online, under a stable uri – and which contains references to places – can be made discoverable through peripleo. what is needed for integration is a dataset summary: a data file that lists all objects, along with basic dublin core metadata (title, description, date, provenance information, etc.), and information about the places each object is related to (and, optionally, how it is related to them). to encode the latter, we have chosen the open annotation data model [13]. the metaphor of annotation is not only appropriate for the act of identifying (or “tagging”) a place reference in arbitrary digital content with a gazetteer uri. it also has the connotation that, in general, the identification (or “tag”) is not to be considered certain fact, but rather that someone (a human editor, an automated geo-parsing script) is making a claim about some kind of relationship between part of the source document and the place. the rdf snippet below provides a “minimum dataset summary” example of a dataset with a single object, expressed in rdf/turtle notation. (full documentation is available at [14].) @prefix cnt: <http://www.w3.org/2011/content#> . @prefix dcterms: <http://purl.org/dc/terms/> . @prefix foaf: <http://xmlns.com/foaf/0.1/> . @prefix oa: <http://www.w3.org/ns/oa#> . @prefix pelagios: <http://pelagios.github.io/vocab/terms#> . @prefix relations: <http://pelagios.github.io/vocab/relations#> . @prefix xsd: <http://www.w3.org/2001/xmlschema> . # an object you want to link to pelagios <http://example.org/pelagios/dump.ttl#items/00l> a pelagios:annotatedthing ; # title and homepage url are mandatory dcterms:title "honorific inscription of ostia" ; foaf:homepage <http://edh-www.adw.uni-heidelberg.de/...> ; # everything else optional (but highly encouraged) dcterms:description "honorific inscription, findspot ostia" ; # use iso 8601 (yyyy[-mm-dd) or time interval (<start>/<end>) for dates dcterms:temporal "366/402" ; # we encourage the use of periodo identifiers to denote time periods dcterms:temporal <http://n2t.net/ark:/99152/p03wskd389m> ; # greco-roman # use rfc 5646 for the object's language (e.g. literature, inscriptions, etc.) dcterms:language "la" ; # feel free to assign 'tags' to your data dcterms:subject "inscription" ; . # objects are 'annotated' with any number of gazetteer references <http://example.org/pelagios/dump.ttl#items/00l/annotations/1> a oa:annotation ; # mandatory: the 'annotation target' is the uri of your object; oa:hastarget <http://example.org/pelagios/dump.ttl#items/00l> ; # mandatory: the 'annotation body' is the gazetteer reference oa:hasbody <http://pleiades.stoa.org/places/422995> ; # optional: extra metadata about the nature of the place reference pelagios:relation relations:foundat ; oa:hasbody [ cnt:chars "point (41.755740099 12.290938199)"; dcterms:format "application/wkt" ] ; oa:annotatedat "2014-11-05t10:18:00z"^^xsd:date ; . along with the dataset summary, peripleo also requires a small rdf file that describes the dataset as a whole (what’s inside, who the publisher is, under what license the metadata is published, etc.) and which acts as machine-readable entrypoint to the — potentially large — summary file. we use the rdf vocabulary of interlinked datasets (void) [15] to encode this information. a minimum example is shown below. @prefix : <http://my-domain.org/my-data/> . @prefix dcterms: <http://purl.org/dc/terms/> . @prefix foaf: <http://xmlns.com/foaf/0.1/> . @prefix void: <http://rdfs.org/ns/void#> . :my-dataset a void:dataset; dcterms:title "my archaeological dataset"; dcterms:publisher "my institution or project"; foaf:homepage <https://my-domain.org/>; dcterms:description "a dataset of archaeological items."; dcterms:license <http://opendatacommons.org/licenses/by/>; # this is very important location of the dataset summary file void:datadump <http://my-domain.org/downloads/pelagios.ttl> ; . depending on the nature of the data, different gazetteers may be suitable as annotation vocabulary. in our current demo installation of peripleo we have, so far, included seven gazetteers from our partners: the pleiades gazetteer of the ancient world [16], the digital atlas of the roman empire [17], vici.org [18], idai.gazetteer [19], the china historical gis [20], nomisma.org [21], and place records from the trismegistos project [22], along with all links to geonames, wikipedia and wikidata that these gazetteers have included in their data exports. importing gazetteers works in a similar fashion to adding data; that is to say, through a lightweight “gazetteer summary” data format. it is therefore perfectly possible to set up an installation of peripleo with a different combination of gazetteers, or a single institutional one. a minimum example for a gazetteer summary with a single place is shown in the rdf snippet below. (full documentation is available at [23].) @prefix cito: <http://purl.org/spar/cito> . @prefix cnt: <http://www.w3.org/2011/content#> . @prefix dcterms: <http://purl.org/dc/terms/> . @prefix foaf: <http://xmlns.com/foaf/0.1/> . @prefix geo: <http://www.w3.org/2003/01/geo/wgs84_pos#> . @prefix geosparql: <http://www.opengis.net/ont/geosparql#> . @prefix gn: <http://www.geonames.org/ontology#> . @prefix lawd: <http://lawd.info/ontology/> . @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> . @prefix skos: <http://www.w3.org/2004/02/skos/core#> . <http://www.mygazetteer.org/place/athens> a lawd:place ; # don't think of label and description in terms of a 'primary name' or # detailed abstract. think of it in terms of ui: what do you want users # to see about your place in a list of search results? rdfs:label "athens"@en ; dcterms:description "a major greek city-state"@en ; # optional: a present-day (iso-3166 alpha2) country code gn:countrycode "gr" ; # don't think of dates in terms of 'how long the place existed'. use dates # to specify the period your gazetteer is concerned with the place and/or # provides attestations for it. # use iso 8601 (yyyy[-mm-dd]) or time interval (<start>/<end>). dcterms:temporal "-750/640" ; # we encourage the use of periodo identifiers for time periods dcterms:temporal <http://n2t.net/ark:/99152/p03wskd389m> ; # greco-roman # use skos:closematch to express 'vague' matches, e.g. to link to a # modern-day town now located there skos:closematch <http://sws.geonames.org/264371/> ; # use skos:exactmatch to express (geographical, temporal, cultural) identity skos:exactmatch <http://pleiades.stoa.org/places/579885> ; # express names for the place using lawd:variantform. for language encoding, # use rfc 5646. lawd:hasname [ lawd:primaryform "athenae" ] ; lawd:hasname [ lawd:primaryform "athens"@en ]; # you can provide attestions (e.g. bibliographic reference) for individual # names as in the example below). lawd:hasname [ lawd:primaryform "?????"@el ; lawd:hasattestation <http://www.mygazetteer.org/att/0001> ] ; # optional: a representative point coordinate geo:location [ geo:lat 5.16 ; geo:long 52.05 ] ; # optional: detail geometry as wkt string. # alternatively, use osgeo:asgeojson for a geojson string geosparql:hasgeometry [ geosparql:aswkt "linestring (5.16 52.05, 5.17 52.05, 5.16 52.06)" ; ] ; foaf:primarytopicof <http://www.mygazetteer.org/place/athens.html> ; # optional: express hierarchical relations, if they exist in your gazetteer dcterms:ispartof <http://www.mygazetteer.org/place/greece> ; . # example: metadata of an attestation <http://www.mygazetteer.org/att/0001> a lawd:attestation ; dcterms:publisher <http://www.mygazetteer.org/> ; cito:citesasevidence <http://www.mygazetteer.org/documents/01234> ; cnt:chars "?????" . dataset and gazetteer summaries can be published separately from the source data. this approach is sometimes referred to as “standoff markup” [24], and helps to avoid data management problems which can arise when annotations have to be natively incorporated into an existing data model (potentially introducing changes to internal metadata schemas and database implementations). in addition, it is possible to publish a summary as a single dump file (i.e. an extra static “file download” placed somewhere on the institutional web server), with the result that no changes to existing web presences or apis are necessary. outlook peripleo is prototypical rather than finished software. but we are convinced that an interface which empowers users to tap into and navigate through highly heterogeneous online collections is crucial in order to demonstrate the utility of lightweight linking approaches, and to make their benefits more tangible to end-users and non-technical specialists. on a technical level, we feel that the loose coupling that exists between peripleo and the resources it makes searchable is very much in the spirit of linked open data, in contrast to more tightly coupled traditional search frontends which are typically bound to a specific repository. as regards future steps, we plan to extend peripleo to make use of other linked open data initiatives as and when they emerge. priority will be given to supporting temporal classification schemes like periodo [25], and person authority schemes such as snap [26], as well as customizable type classifications. another future work item pertains to scalability in terms of the amount of data. we are confident that the present architecture can support several times the number of records we are presently hosting in our demo installation (approx. half a million). but it remains to be tested how considerably larger datasets (coming, for example, from museum collections) would affect query response times and visualization performance. inevitably, the single-index backend that drives the bulk of the peripleo frontend interaction will have to be replaced. to this end a solution that supports distribution across a cluster is being prepared. acknowledgements the authors wish to thank the andrew w. mellon foundation for funding this work. about the authors rainer simon is a senior scientist at the austrian institute of technology (ait). his work focuses on the application of linked open data principles – specifically with regard to geospatial data – in the digital library and digital humanities domains. leif isaksen is a senior lecturer in history the university of lancaster. his interests cover a wide range of digital applications in archaeology and the humanities more generally with a particular emphasis on spatial and web technologies. elton barker is a reader in classical studies at the open university (ou). his research interests specialise in ancient greek literature and thought, with a particular focus on ancient greek agonistics and cultural geography. pau de soto cañamares is a researcher at the institute of catalan studies (iec). his interest lies in the use of digital information to advance our knowledge of the past, one example being the use of network analysis to understand the movement of goods in antiquity and the organization of roman territories. references [1] pelagios project blog. http://pelagios-project.blogspot.co.uk (last accessed december 2015) [2] elliott, t. and gillies, s. 2009. digital geography and classics. in digital humanities quarterly. vol 3. number 1. http://www.digitalhumanities.org/dhq/vol/3/1/000031.html(last accessed december 2015) [3] to excerpt from the introductory chapters of berman, mostern & southall, placing names: enriching and integrating gazetteers, indiana university press (in print): “most simply, a gazetteer is a list of places.” […] “in recent decades, and especially in a computing context, gazetteers have often been defined as simple pairings of place names and coordinates, sometimes including feature types as well.” “[…] gazetteers are the basis for much of the spatial search and visualization that specialists and the public have come to take for granted.” […] “the future of [gazetteers] is online and sharing-centric, following a general development trend that is generally known as linked open data.” “[…] all sharing-centric repositories must be online, and have a way to uniquely identify every data record [by means of a uniform resource identifier (uri)].” [4] isaksen, l., simon, r., barker, e. and de soto cañamares, p. 2014. pelagios and the emerging graph of ancient world data. in proceedings of the 2014 acm conference on web science, pp. 197-201. [5] simon, r., isaksen, l., barker, e. and de soto cañamares, p. 2015. the pleiades gazetteer and the pelagios project. in placing names: enriching and integrating gazetteers. berman, m. l., mostern, r. and southall, h. (eds.) indiana university press (in print). [6] bizer, c., heath, t., berners-lee, t. 2009. linked data – the story so far. in international journal on semantic web and information systems, 5(3): 1-22. [7] simon, r., barker, e., isaksen, l. and de soto cañamares, p. 2015. linking early geospatial documents, one place at a time: annotation of geographic documents with recogito. in e-perimetron. vol.10, no.2 (2015), pp. 49-59. issn 1790-3769. [8] peripleo public demo. http://pelagios.org/peripleo/map(last accessed december 2015) [9] american numismatic society collection database. http://numismatics.org/search(last accessed december 2015) [10] the fralin | uva art museum numismatic collection. http://coins.lib.virginia.edu/(last accessed december 2015) [11] pelagios github repository: peripleo. http://github.com/pelagios/peripleo(last accessed december 2015) [12] play web application framework http://www.playframework.com(last accessed december 2015) [13] sanderson, r., ciccarese, p. and van de sompel, h. (eds.). 2013. open annotation data model. community draft, 08 february 2013. http://www.openannotation.org/spec/core/(last accessed december 2015) [14] joining pelagios — documentation of pelagios dataset and metadata description rdf profile. https://github.com/pelagios/pelagios-cookbook/wiki/joining-pelagios(last accessed december 2015) [15] describing linked datasets with the void vocabulary. w3c interest group note 03 march 2011. http://www.w3.org/tr/void/(last accessed december 2015) [16] pleiades gazetteer of the ancient world. http://pleiades.stoa.org/(last accessed december 2015) [17] digital atlas of the roman empire. http://dare.ht.lu.se/(last accessed december 2015) [18] vici.org. archaeological atlas of antiquity. http://vici.org(last accessed december 2015) [19] idai.gazetteer, german archaeological institute. http://gazetteer.dainst.org(last accessed december 2015) [20] china historical gis http://www.fas.harvard.edu/~chgis/(last accessed december 2015) [21] nomisma.org. http://nomisma.org(last accessed december 2015) [22] trismegistos project. http://www.trismegistos.org(last accessed december 2015) [23] pelagios gazetteer interconnection format. https://github.com/pelagios/pelagios-cookbook/wiki/pelagios-gazetteer-interconnection-format(last accessded december 2015) [24] thompson, h.s. and mckelvie, d. 1997. hyperlink semantics for standoff markup of read-only documents. in proceedings of sgml europe 97. p. 227-229. [25] rabinowitz, a. 2014, it’s about time: historical periodization and linked ancient world data. in isaw papers 7. http://dlib.nyu.edu/awdl/isaw/isaw-papers/7/rabinowitz/(last accessed december 2015) [26] standards for networking ancient prosopographies (snap) project website. http://snapdrgn.net/(last accessed december 2015) subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – data reuse in linked data projects: a comparison of alma and share-vde bibframe networks mission editorial committee process and structure code4lib issue 49, 2020-08-10 data reuse in linked data projects: a comparison of alma and share-vde bibframe networks this article presents an analysis of the enrichment, transformation, and clustering used by vendors casalini libri/@cult and ex libris for their respective conversions of marc data to bibframe. the analysis considers the source marc21 data used by alma then the enrichment and transformation of marc21 data from share-vde partner libraries. the clustering of linked data into a bibframe network is a key outcome of data reuse in linked data projects and fundamental to the improvement of the discovery of library collections on the web and within search systems. by jim hahn introduction contemporary linked data implementation has been bolstered by several novel projects that use transformation and enrichment of marc data to bibframe graphs (xu, hess, & akerman, 2018). the underlying catalyst in the contemporary wave of linked data implementations in libraries was bibframe data model development and subsequent experimentation at the library of congress and the program for cooperative cataloging (naun, 2019). the spread and refinement of the bibframe data model, which takes an inclusive approach to library description on the semantic web, has ushered in a new wave of modern experimentation and implementation for linked data enrichment of traditional cataloging records (jin, hahn, & croll, 2016). alternatively, non-bibframe efforts at linked data implementation used schema.org, a vocabulary first developed by major search engines for discovery of structured data on the web (cole, han, weathers, & joyner, 2013; lampron, mixter, & han, 2016). an excellent schema.org primer, “html5 microdata and schema.org,” appeared in the code4lib journal (ronallo, 2012) along with several exemplar implementation cases (clark & young, 2015, 2017; pekala, 2018; wallis et al., 2017). projects such as the share-vde linked data catalog (samples & bigelow, 2020) and linked data enrichment within the exlibris alma product are two examples of modern linked data implementation. these new implementations offer valuable insights into the road ahead for libraries transitioning to new linked data descriptions to provide enhanced discovery of collections. the purpose of this paper is first to analyze the source marc data transformed and enriched within alma using the university of pennsylvania alma and alma sandbox to present a case study for data analysis; and next, to evaluate the enrichment of source marc data transformed to bibframe graphs in the share-vde linked data catalog. the technical work in share-vde progresses from enrichment and transformation of marc21 data from upenn to clustering of linked data into a graph bibframe network. the enriched, transformed, and clustered data from the share-vde catalog are then made available to upenn libraries in the marc21 format and in bibframe/rdf graphs. the linked data graphs are encoded with bibframe rdf and also provide helpful provenance information using n-quads. labeling an rdf subject, predicate, and object is a particular strength and use of the n-quad standard (w3c, 2014). share-vde provides a linked data catalog to access the clustered works, agents, and subjects. while share-vde comprises over 20 partner libraries, members can skin a linked data catalog. the skin provides a local look and feel to the catalog. and using additional apis, the skin can interface with local requesting and catalog account features such as placing holds or accessing local e-resources. this localization does require additional configuration and depends on the apis available in the implementing library’s integrated library system. a mockup of the university of pennsylvania share-vde skin is shown in image 1. figure 1. prototype screenshot for the bibliographic work description sets by a given bibframe agent. source: filip jakobsen university of pennsylvania libraries: alma bibframe alma is the data source for metadata encoded in marc in the university of pennsylvania libraries. alma provides a bibframe record consisting of work and instance data for each marc metadata record in the system. a grid view of the xml analyzed through oxygen xml editor is shown in image 2. figure 2. grid view of alma bibframe xml illustrating work and instance entities that are associated with an alma marc record. alma enrichment and conversion for bibframe relies on the library of congress marc2bibframe2 code (library of congress, 2020a). the bibframe record is shown under a linked data tab for the bibliographic record, as seen in image 3. figure 3. a screenshot of the linked data tab entitled “bibframe” in the university of pennsylvania’s alma sandbox. source: alma the resulting transformation produces instance level data about the bibliographic record, and within the same record tab, related work data. work descriptions, as presented in the alma interface, are not associated with a superwork bibliographic description such as a bibframe hub or in share-vde, an opus. an example of the emerging lc hub entity, a type of superwork, can be seen in image 4. figure 4. a recent approach to modeling the library of congress hub (or lc:hub), an example of a superwork. in this case the library of congress hub is a type of work entity set. the superwork is defined differently in various vocabularies, in much the same way that no universal work bibliographic description exists. however, “a superwork may contain any number of works as subsets, the members of which while not sharing essentially the same information content are nevertheless similar by virtue of emanating from the same ur-work“ (svenonius, 2000, p. 38). according to alma documentation (exlibris, 2020a) available on the web, “alma groups several bib record into a single work based on the bibliographic information,” with features available on request for clustering marc items descriptions under a shared bibliographic work description; however, “the work grouping logic is not implemented by default. institutions that are interested in this functionality should contact exlibris and ask for it. for institutions that did not ask to implement it, each bib record will be considered as having its own work”(ex libris, 2020a). bibframe records in alma contain uris as linked data that are not present in the source marc stored in alma. the uris in alma bibframe records include: library of congress authorities mesh viaf integrated authority file, the german national library’s gemeinsame normdatei (gnd) geonames local to ex libris uris ex libris work ids ex libris agent ids alma’s faq on standards does indicate record support for bibframe: “the ‘general publishing profile’ in alma includes an option to publish in bibframe format. alma will support the importing of catalog records in bibframe format, allowing bibrame formatted records to be easily and seamlessly made part of the alma catalog, regardless of the cataloging format in which it is managed” (exlibris, 2020b). linked data created externally to alma can be imported into the system. the sinopia linked data editor is the “cloud-based environment for original metadata creation,” the work product from the ld4p2 grant project (ld4p2, 2020). the sinopia code base continues to evolve over several iterations and was initially built upon a hard fork of code that made up the library of congress bibframe editor or bfe, described as “a simple tool that enables input of any bibframe vocabulary element,” (library of congress, 2020b). catalogers can make native rdf bibframe records from this system. the sinopia editor is not integrated into alma. but sinopia bibframe data can be exported as json-ld and includes uris for instance and work record descriptions, among others, which can bring in the relevant data needed to undertake transformation from bibframe into marc. the process is illustrated in image 5. figure 5. the data flows from sinopia rdf through the bibframe2marc code and in the final stage imports marc to alma. the library of congress released bibframe2marc code in may 2020 (library of congress, 2020c), making the transformation from a bibframe record possible. however, to account for the sinopia namespace, developers created the helpful processing code rdf2marc while working under the ld4p2/3 grants at stanford. after conversion to marc21 or marcxml, the record can then be imported to alma. a possible output of the newly launched ld4p3 grant project led by stanford and cornell with funding from mellon may make it possible to connect directly to an ils or export marc from sinopia, thus obviating the need for the transformation shown in image 5. share-vde bibframe graphs the share-vde hosted stardog database provides access to upenn’s linked data. the implementation of share-vde graphs uses n-quads, which can be viewed record by record in a file archive provided to upenn. the query shown in image 6 for the record’s mmsid (unique identifier in alma) will return all associated graph uris for cluster analysis. select * where { graph <http://share-vde.org/sharevde/rdfbibframe/provenance/upenn9977142688903681> { ?s ?p ?o . } } figure 6. stardog studio retrieval from share-vde n-quad query of all graphs related to a bibliographic identifier. source: stardog studio the stardog studio offers a visual node explorer. the example graph exploration in image 7 illustrates work nodes related to a subject “school sports – fiction.” the center in this example graph is where we can see work relationships pointing to this identifier. share-vde clustering technologies aspire to model the deep interconnections among data in the records and among other related records in its database. whether alma clusters linked data among other libraries’ works is unclear and not yet observed from the alma backend that an institution can access. perhaps clusters do exist elsewhere in ex libris graph databases that power other parts of their infrastructure such as summon or a larger index. the uris in share-vde (last update feb 2020): id.loc.gov/vocabulary/geographicareas isni viaf lcnaf library of congress subject headings fast orcid integrated authority file, the german national library’s gemeinsame normdatei (gnd) local share-vde uris; share-vde author id; share-vde publisher entity id; share-vde work id figure 7. stardog visualization functionality. source: stardog studio sample share-vde stardog data as xml graph <?xml version='1.0' encoding='utf-8'?> <sparql xmlns='http://www.w3.org/2005/sparql-results#'> <head> <variable name='s'/> <variable name='p'/> <variable name='o'/> </head> <results> <result> <binding name='p'> <uri>http://www.w3.org/1999/02/22-rdf-syntax-ns#type</uri> </binding> <binding name='s'> <uri>http://share-vde.org/sharevde/rdfbibframe/provenance/upenn9977142688903681</uri> </binding> <binding name='o'> <uri>http://www.w3.org/ns/prov#entity</uri> </binding> </result> <result> <binding name='p'> <uri>http://www.w3.org/1999/02/22-rdf-syntax-ns#type</uri> </binding> <binding name='s'> <uri>http://share-vde.org/sharevde/rdfbibframe/agent/upenn</uri> </binding> <binding name='o'> <uri>http://www.w3.org/ns/prov#agent</uri> </binding> </result> </results> </sparql> importing share-vde enriched marc21 into the penn alma sandbox the uris in share-vde above are also found in the share-vde of the penn marc21 enriched with uris. a sample of the upenn share-vde marc21 enrichment was loaded into the penn alma sandbox. this experiment aimed to understand what, if any, discovery gains may be achieved by reuse of the share-vde enriched marc21. to view marc, marcedit’s mnemonic format was used in displaying human readable marc. for more background on the mnemonic format, see the helpful documentation at the marcedit field guide (reese, 2020). share-vde enriched marc21(w/uris) =ldr 02458ngm a22004693i 4500 =001 9977142688903681 =005 20160817151618.0 =006 m|||||o||c|||||||| =007 cr\|n||||||||a =007 vz\|za|z| =008 160817s2007\\\\cau002\e\\\\\|o\\\v|eng\d =035 \\$a(vaalasp)avonasp3241241/marc =035 \\$a(ocolc)avon957520113 =040 \\$avaalasp$beng$erda$cvaalasp =245 00$adocumentation.$pessentials of nursing documentation.$phealth care plan $h[electronic resource] /$c[produced by medcom, inc.]. =264 \1$a[cypress, california] :$bmedcom,$c[2007]$9http://share-vde.org/sharevde/rdfbibframe/publisher/1144933 =300 \\$a1 online resource (2 minutes) =306 \\$a000143 =336 \\$atwo-dimensional moving image$btdi$2rdacontent$0http://rdaregistry.info/termlist/rdacontenttype/1023 =337 \\$avideo$bv$2rdamedia$0http://rdaregistry.info/termlist/rdamediatype/1008 =337 \\$acomputer$bc$2rdamedia$0http://rdaregistry.info/termlist/rdamediatype/1002 =338 \\$aother$bvz$2rdacarrier =338 \\$aonline resource$bcr$2rdacarrier$0http://rdaregistry.info/termlist/rdacarriertype/1018 =347 \\$adata file$2rda =500 \\$atitle from resource description page (viewed august 19, 2016). =506 \\$arestricted for use by site license. =520 \\$athis instructional video, from the documentation series produced by medcom, inc., is about the documentation of health care plans. =546 \\$ain english. =650 \0$anursing records.$0http://id.loc.gov/authorities/subjects/sh85093417 =650 \0$anursing care plans.$0http://id.loc.gov/authorities/subjects/sh85093387 =650 \0$amedical records$xmanagement.$0http://id.loc.gov/authorities/subjects/sh85083015 =650 \0$acommunication in nursing.$0http://id.loc.gov/authorities/subjects/sh85029081 =655 \7$ainstructional films.$2lcgft =655 \7$afilm excerpts.$2lcgft =710 2\$aalexander street press.$1http://isni.org/isni/0000000122177075 =710 2\$amedcom, inc.,$eproduction company.$1http://viaf.org/viaf/146569313 =740 0\$aacademic video online. =758 \\$4http://rdaregistry.info/elements/m/p30004$1http://worldcat.org/oclc/957520113 =856 40$uhttp://hdl.library.upenn.edu/1017.12/1929840$zconnect to streaming video =996 10$adocumentation. essentials of nursing documentation. health care plan$9http://share-vde.org/sharevde/rdfbibframe/work/11033735-1 =997 10$aupenn =999 8\$bweb $hfrom the enriched marc sample loaded into the penn alma sandbox, it did not appear that alma supported a 1-to-1 import of uris in marc. specifically, for the sample of records the 996-field containing a share-vde work id was not imported into the alma system. note that the 900s are not defined in the marc21 standard, and vendors use them for local purposes. several important uris such as added entries do get added to the bibframe linked data record in alma. associating multiple work identifiers in bibframe is possible, but in the version of alma “merge on match” the share-vde work ids in 996 did not merge with the linked data/bibframe record within alma. without a corresponding discovery load into blacklight, evaluating improvements to discovery gains using alma imports are not possible at upenn. the penn libraries discovery layer to alma does not use primo, but rather blacklight as its front-end to discovery. a separate test blacklight instance that indexed marc entity uris as compared to non-uri enhanced marc could provide a comparative analysis of discovery relative to enrichment. this experiment for indexing the work entity id associated with an mmsid in the upenn blacklight/lucene is in the planning stages. in the early 2000s a set of now classic marc usage evaluations from moen and others demonstrated the “ground truth” metrics (tennant, 2016) of marc utilization. the blacklight testing of work ids takes an inclusive approach to linked data implementation by revisiting and extending marc studies to inform a comparative blacklight study, evaluating retrieval within a blacklight index using “linky”-marc and the same corpus of marc records encoded with “non-linky” marc. the premise of “linky-marc” is the transitional notion of incorporating entity uris into marc records (wallis, 2018). moen and benardino (2003, p. 171) demonstrated that “…less than 5% of available content designation accounts for 80% of occurrences.” this finding underscores the need for intelligent and measured indexing and discovery of marc; namely, that not all fields must be treated with equivalent indexing importance in the process of making descriptions of bibliographic works discoverable. the present study focused on work entity uris containing work ids in “linky”-marc and contrasted the retrieval to a non-enriched marc index of the same corpus. evaluation of share-vde enriched marc for consistency with the pcc guidance for uris in marc the use of subfields $0 and $1 in the enriched marc from share-vde were evaluated for consistency, with recent pcc guidance from the september 12, 2019 report “pcc task group on linked data best practices final report” and the corresponding marc object table (library of congress, 2019). a baseline report of the enriched share-vde marc21, which was generated with marcedit and then plotted with tableau, showed the nature of marc fields, used $0 and $1, and is shown in image 8 and image 9. figure 8. $1 uris from penn share-vde enriched marc. source: author the set of records in table 1 uses $1 uris primarily within 100s (main entry) and the 758 (resource identifier) fields. regarding the added entry fields: 700 (added entry-personal name), 710 (added entry-corporate name), and 758 (resource identifier). pcc guidelines for $1 in these fields recommend that the identifier in $1 must always be a uri. the occurrences in the sample conform. for the 758 fields above, proper guidance for the use of $4 was also observed. the 758 field, according to the pcc guidance, can be used for work or resource identifiers (library of congress, 2019, p. 18). figure 9. $0 uris from penn share-vde enriched marc. source: author the 650 field contains the most usage of the enriched marc $0 subfield. guidelines from the pcc marc object table for the 650 (subject added entry – topical term) allow for the $0 to be used here, however, guidance suggests use of “$0 only if the entire heading is established. do not provide $0 for parts of the heading.” a meta-analysis of 650 uris in image 7 found that this sample conforms with pcc best practice guidelines. evaluation of penn marc alma samples for consistency with the pcc guidance for uris in marc a sample of 50,000 alma marc records available from the penn open data website was used for evaluating alma data. benchmarks were created to understand if and where the uses of $0 and $1 exist in alma. in a sample of 50,000 records, no $1 subfields were in use by penn marc records included in this evaluation. the $0 subfields did have uses in the 600s (subject access fields) and 880s (alternate geographic field) for a total of 693 uses in the sample of 50,000. baseline metrics are delineated in image 10. the preponderance of $0 can be found in 650 (subject added entry – topical term) followed by 655 (index term – genre/form) and 651 (subject added entry – geographic name) fields. figure 10. $0 subfields in a selection of alma marc records at penn. share-vde enrichment and clustering supporting penn discovery on the web: schema.org reusing linked data from share-vde enrichment and clustering processes, it is feasible to add linked data into the html elements of the work page descriptions in blacklight. this may result in additional discovery gains for records searched from google and the world wide web. an example of a json-ld data implementation for html records follows. this example is not from an existing webpage, rather it is an implementation model verified as valid json-ld through google’s structured data testing tool. <script type="application/ld+json"> { "@context":"http://schema.org", "@graph": [{ "@id": "https://share-vde.org/sharevde/docbibframe/work/25071", "@type": "book", "name": "hamlet : text of the play, the actors’ gallery, contexts, criticism, afterlives, resources", "bookedition": "second edition", "numberofpages": "338", "author":{ "@type": "person", "name": ["william shakespeare ; edited by robert s. miola"], "url": "http://viaf.org/viaf/96994048" }, "publisher": { "@type": "organization", "name": "w.w. norton & company", "url": "http://viaf.org/viaf/127676589" }, "url": "https://franklin.library.upenn.edu/catalog/franklin_9977629232903681" }] } </script> book actions (https://developers.google.com/search/docs/data-types/book) are emergent schema.org descriptions for search engines that may provide a more optimal experience for borrowing print and e-books on the web. the book actions will appear through existing knowledge panels on the web. libraries can register their interest to pilot book actions from the data type link above. streamlining alma reports large alma publishing is resource and time intensive. it may be possible to replicate certain alma reports that pose bottlenecks due to alma cloud throttling of large reports. a upenn postgresql database hosted locally may be able to address these bottlenecks by setting up an indexing and query service. in such a circumstance delta imports must be made based on the progression of mmsid integers in the alma system relative to the postgresql enriched json database. the enriched marc21 from share-vde has been converted to json using marcedit. with some configuration to line-by-line json, it is possible to load the enriched data into postgresql using the jsonb, the preferred, and more feature rich indexing data type for additional analysis and evaluation. the json output from marcedit was further transformed using the jq software tool so that the json array would span one line. for more information on jq see: https://formulae.brew.sh/formula/jq flattening a json record array into one line was accomplished with the following command: jq -c '.[]' output-upenn_biblio_01.json | awk '{print > "file_01.json"}' the json can be added to a postgresql table using the jsonb data type, and a modification of the csv importing command to simulate row based importing that csv commands afford. \copy {table-name} from 'file_01.json' csv quote e'\x01' delimiter e'\x02' example table load was found here: http://adpgtech.blogspot.com/2014/09/importing-json-data.html. following data load, a simple index on json fields in postgresql can be processed over the enriched library data: create index idxgintags on allmarc3 using gin ((marc -> 'fields')); next, it is possible to query the postgres index on jsonb data using keyword search. the example query below is establishing a set of records that contain an 880 field. select * from allmarc3 where marc->>'fields' like '%880%'; figure 11. retrieval of jsonb enriched marc records through a postgresql keyword query. source: pgadmin 4 this workflow can provide local data persistence with indexing and retrieval in both the enriched marc, and if desired, the n-quads from share-vde; both can be loaded and indexed in a persistent database under local control. indexing n-quads under a local triple store can better achieve the goals of linked data by maintaining the enriched bibframe graph in a graph database such as blazegraph, used by wikibase. locally hosted wikibase of enriched n-quads from share-vde wikibase docker images (https://github.com/wmde/wikibase-docker) are a promising tool for hosting enriched linked data graphs locally, within the upenn library data infrastructure. oclc used this infrastructure for project passage (godby et al., 2019). the oclc infrastructure for linked data will likely continue to use wikibase for the mellon funded entity management project. an opportunity exists to reuse the wikidata properties for ongoing linked data research at the libraries, offering several advantages, including avoiding vendor lock-in and ensuring local control over penn linked data. the wikibase infrastructure offers the possibility to retain share-vde records locally and to act as a backbone for continued experimentation. a local wikibase instance of share-vde n-quads alleviates concerns of possible vendor lock-in of n-quads in the stardog database hosted by share-vde. finally, there may be ways to reuse the n-quads from share-vde (e.g. superwork to work clustering) for informing clustering of catalog data outside of share-vde. figure 12. share-vde author id property in wikidata. source: wikidata wikibase software can support the increased production of links among upenn scholarship, collections, and the broader web of linked data. recent studies have pointed to the importance of wikimedia content to search engines/discovery on the web writ large (vincent & hecht, 2020). the use of wikibase knowledge graphs in recommender systems development can serve to promote library content (tsuji, 2019). linked data offers some advantages over big data for providing personalization/recommender services (campbell & cowan, 2016); e.g. recommendations are based on structured knowledge not personal data mining. reusing existing knowledge structures is preferable to creating personalized data stores from which to derive a recommendation system, especially in libraries. discussion this section ties together the preceding data reuse activities within a common, overarching long-term outcome: improving discovery of penn library collections. using a logic model framework, table 1 offers a delineation of the above activities that coalesce around the long-term goals focused on increased discoverability. logic models identify the anticipated outcomes based on resources, activities, and outputs of organizations (schryvers, 2020). as a goal like improved discovery is a difficult long-term outcome to assess, it helps to create short-term (julian, jones, & deyo 1995) and mid-term outcomes of discovery improvements in which the libraries can collect measurable metrics from activities and understand if the outputs from resources and outputs are serving the intended effects. table 1. liked data logic model: penn libraries linked data short-term, mid-term, and long-term outcomes resources activities outputs short-term outcomes mid-term outcomes long-term outcomes 1. linked data team 2. sinopia / bibframe editor group 3. discovery team 4. ld4 affinity group participation 5. wikibase/oclc project entity management group participation 1. share-vde catalog development 2. bibframe editor/ sinopia training, original rdf data creation 3. blacklight tests of work entity ids in franklin 4. bibframe editor + data flow improvements (ld4p3 affinity groups) 5. oclc entity management api testing, wikibase pcc project participation 1. share-vde record enrichment 1.a. share-vde linked data penn catalog 2. original rdf linked data from bibframe editor 3. work entities in franklin 4. improved bibframe editors and data flow 5. oclc entities api data 5.a. wikidata pcc project 2: professional staff familiarity with sinopia, linked data, and bibframe. 1, 2, 5: local penn wikibase supporting rdf data persistence of linked data from share-vde enrichment, original rdf creation, oclc entities api data 2: professional staff expertise with bibframe editors, linked data models (lrm, rda), and bibframe. 2, 3, 4: alma load of select marc records transformed from bibframe editor. 1, 3, 5: integrate share-vde interface into library discovery; bento box of share-vde catalog; indexing in blacklight to target work entity uris from share-vde; oclc; and/or bibframe editor work data. 1, 2, 3, 4, 5: enhance discovery of penn collections teams involved in this work represent internal resources along with external linked data projects. the internal penn linked data team continues to evaluate the efficacy of the share-vde linked data catalog and shares membership with a discovery strategic project team whose goals include integration of the share-vde linked data catalog into the libraries’ bento search layout. integration of the linked data catalog into library search results will affect the long-term outcome of linked data implementation. the penn libraries is beginning a mixed methods evaluation incorporating log studies and remote user testing of interviews and observation to understand how semantic search fits into the search process. log studies (jansen, 2006; peters, 1993) are quantitative measures that provide great insights and empirical evidence. quantitative studies (mischo et al., 2012; lown et al., 2013; dougan, 2018) can inform new hypothesis development that qualitative measures of interviews (mckay et al., 2019) and observation are better able to address. the emerging research agenda at penn on semantic interfaces aims to evaluate user tasks as articulated in ifla’s library reference model (lrm). the lrm harmonized several ontologies (e.g. frbr, frad, frsad) with distinct perspectives and aims. an underappreciated section developed in the reference model is the detail devoted to supporting user tasks (riva, le boeuf, & žumer, 2017). user tasks are “…the ends of any bibliographic system” (dousa, 2018). žumer (2017, para 9) wrote that the “…lrm defines five user tasks : find, identify, select, obtain, explore. the tasks are listed in an order that reflects typical user behavior, which does not mean that all tasks need to be performed and that they cannot be repeated. particularly identify and obtain often occur in parallel and in interaction.” previous user research into discovery findings and the lrm user tasks informed semantic search hypothesis development. the hypothesis generation process necessarily involves turning lrm tasks into research questions, and understanding open questions in discovery research—a selection of which is shown in table 2. table 2. selected ifla lrm tasks (riva, le boeuf, & žumer, 2017) selected lrm exemplar tasks find “… all expressions of a work that are written in a given language” (p.97) identify “… a personal name that corresponds to the person sought by the user, even though other people are identified by similar names” (p.98) explore “… relationships in order to understand the structure of a subject domain and its terminology” (p.99) semantic interface hypothesis expressions of a work in each language can be easily ascertained in a semantic interface search result page. name disambiguation is supported in semantic search results. semantic interfaces support relationship exploration in a subject domain (e.g. browsing). conclusion ongoing and sustained reuse of linked data are a fundamental component of improving library discovery operations. outside of the university of pennsylvania libraries, collaborative approaches to developing wikibase-focused, persistent structured data, and future tracking of the newly funded oclc project entity management work will extend penn resources and penn linked data with web-based knowledge graphs. an integrated linked data strategy with discovery and web strategic projects will help to further penn libraries’ overall goals for increasing the visibility of penn resources on the web and within library search systems. about the author jim hahn (https://orcid.org/0000-0001-7924-5294) is the head of metadata research at the university of pennsylvania libraries leading linked data and metadata projects and research for the libraries. working collaboratively across the libraries, his work is developing a vision for the services, technologies and policies to enhance discovery of collections, following international standards and best practices for linked data and metadata. references campbell, d. g., & cowan, s. r. (2016). the paradox of privacy: revisiting a core library value in an age of big data and linked data. library trends, 64(3), 492–511. https://doi.org/10.1353/lib.2016.0006 clark, j. a., & young, s. w. h. (2015). building a better book in the browser (using semantic web technologies and html5). the code4lib journal, 29. https://journal.code4lib.org/articles/10668 clark, j. a., & young, s. w. h. (2017). linked data is people: building a knowledge graph to reshape the library staff directory. the code4lib journal, 36. https://journal.code4lib.org/articles/12320 cole, t. w., han, m. j., weathers, w. f. & joyner, e. (2013). library marc records into linked open data: challenges and opportunities. journal of library metadata, 13(2–3), 163–196. https://doi.org/10.1080/19386389.2013.826074 dougan, k. (2018). the “black box”: how students use a single search box to search for music materials. information technology and libraries, 37(4), 81–106. https://doi.org/10.6017/ital.v37i4.10702 dousa, t. (2018, december 5). the ifla lrm model: a brief introduction [video]. youtube. https://youtu.be/pmgp1khrlkq exlibris (2020a). bibframe. ex libris knowledge center. https://developers.exlibrisgroup.com/alma/integrations/linked_data/bibframe/ exlibris (2020b). standards. ex libris knowledge center. https://knowledge.exlibrisgroup.com/alma/product_materials/050alma_faqs/general/standards godby, j., smith-yoshimura, k., washburn, b., davis, k., detling, k., fernsebner eslao, k., folsom, s., li, x., mcgee, m., miller, k., moody, h., tomren, h., & thomas, c. (2019). creating library linked data with wikibase: lessons learned from project passage. dublin, oh: oclc research. https://doi.org/10.25333/faq3-ax08 jensen, b. j. (2006). search log analysis: what it is, what’s been done, how to do it. library & information science research, 28(3), 407–432. https://doi.org/10.1016/j.lisr.2006.06.005 jin, q., hahn, j., & croll, g. (2016). bibframe transformation for enhanced discovery. library resources and technical services, 60(4), 223–235. https://doi.org/10.5860/lrts.60n4.223 julian, d. a., jones, a., & deyo, d. (1995). open systems evaluation and the logic model: program planning and evaluation tools. evaluation and program planning, 18(4), 333–341. https://doi.org/10.1016/0149-7189(95)00034-8 lampron, p., mixter, j., & han, m. j. (2016). challenges of mapping digital collections metadata to schema.org: working with contentdm. in e. garoufallou, i. s. coll, a. stellato, & j. greenberg (eds.), metadata and semantics research – 10th international conference, mtsr 2016, proceedings (pp. 181–186). switzerland: springer-verlag. https://doi.org/10.1007/978-3-319-49157-8_15 ld4p2 (2020). sinopia user group. linked data for production: pathway to implementation. https://wiki.lyrasis.org/display/ld4p2/sinopia+user+group library of congress (2019). pcc task group on linked data best practices final report. program for cooperative cataloging > task groups. https://www.loc.gov/aba/pcc/taskgroup/linked-data-best-practices-final-report.pdf library of congress (2020a). marc2bibframe2. network development and marc standards office. https://github.com/lcnetdev/marc2bibframe2 library of congress (2020b). using the bibframe editor. bibframe: implementation, tools, and downloads. https://www.loc.gov/bibframe/implementation/bfe-howtouse.html library of congress (2020c). bibframe2marc. network development and marc standards office. https://github.com/lcnetdev/bibframe2marc lown, c., sierra, t., & boyer, j. (2013). how users search the library from a single search box. college & research libraries 74(3), 227-241. https://doi.org/10.5860/crl-321 mckay, d., chang, s., smith, w., & buchanan, g. (2019). the things we talk about when we talk about browsing: an empirical typology of library browsing behavior. journal of the association for information science & technology, 70(12), 1383–1394. https://doi.org/10.1002/asi.24200 mischo, w. h., schlembach, m. c., bishoff, j., & german, e. m. (2012). user search activities within an academic library gateway: implications for web-scale discovery systems. in m. p. popp & d. dallis (eds.), planning and implementing resource discovery tools in academic libraries (pp. 153–173). hershey, pa: igi global. https://doi.org/10.4018/978-1-4666-1821-3.ch010 moen, w. e., & benardino, p. (2003). assessing metadata utilization: an analysis of marc content designation use. dcmi international conference on dublin core and metadata applications, 171–180. https://dcpapers.dublincore.org/pubs/article/view/745 naun, c. (2019). expanding the use of linked data value vocabularies in pcc cataloging. cataloging & classification quarterly, 1–9. https://doi.org/10.1080/01639374.2019.1705953 pekala, s. (2018). microdata in the ir: a low-barrier approach to enhancing discovery of institutional repository materials in google. the code4lib journal, 39. https://journal.code4lib.org/articles/13191 peters, t. a. (1993). the history and development of transaction log analysis. library hi tech, 11(2), 41–66. https://doi.org/10.1108/eb047884 reese, t. (2020). the marcedit field guide. [chapter 4: editing my first set of records] http://marcedit.reeset.net/learning_marcedit/welcome-to-marcedit/chapter-3-editing-my-first-record/2/ riva, p., le boeuf, p., & žumer, m. (2017). ifla library reference model: a conceptual model for bibliographic information. https://www.ifla.org/files/assets/cataloguing/frbr-lrm/ifla-lrm-august-2017_rev201712.pdf ronallo, j. (2012). html5 microdata and schema.org. the code4lib journal, 16. https://journal.code4lib.org/articles/6400 samples, j., & bigelow, i. (2020). marc to bibframe: converting the pcc to linked data. cataloging & classification quarterly, 1–15. https://doi.org/10.1080/01639374.2020.1751764 schryvers, p. (2020). bad data: why we measure the wrong things and often miss the metrics that matter. guilford, cn: prometheus books. svenonius, e. (2000). the intellectual foundation of information organization. cambridge, ma: mit press. tennant, r. (2016). “ground truthing” marc. hanging together.org. https://hangingtogether.org/?p=5648 tsuji, k. (2019). book recommender system for wikipedia article readers in a university library. 2019 8th international congress on advanced applied informatics (iiai-aai), 121–126. https://doi.org/10.1109/iiai-aai.2019.00034 vincent, n., & hecht, b. (2020). a deeper investigation of the importance of wikipedia links to the success of search engines. arxiv:2004.10265 https://arxiv.org/abs/2004.10265 w3c (2014). rdf 1.1 n-quads: a line-based syntax for rdf datasets. w3c recommendation. https://www.w3.org/tr/n-quads/ wallis, r., isaac, a., charles, v., & manguinhas, h. (2017). recommendations for the application of schema.org to aggregated cultural heritage metadata to increase relevance and visibility to search engines: the case of europeana. the code4lib journal, 36. https://journal.code4lib.org/articles/12330 wallis, r. (2018). marc and beyond: our three linked data choices, paper presented at: ifla wlic 2018 – kuala lumpur, malaysia – transform libraries, transform societies in session 113 – information technology, http://library.ifla.org/id/eprint/2124 xu, a., hess, k. & akerman, l. (2018). from marc to bibframe 2.0: crosswalks. cataloging & classification quarterly, 56(2–3), 224–250. https://doi.org/10.1080/01639374.2017.1388326 žumer, m. (2017). ifla library reference model (lrm): harmonisation of the frbr family. isko encyclopedia of knowledge organization. http://www.isko.org/cyclo/lrm subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – editorial: welcome new editors, what we know about who we are, and submission pro tip! mission editorial committee process and structure code4lib issue 37, 2017-07-18 editorial: welcome new editors, what we know about who we are, and submission pro tip! want to see your work in c4lj? here’s a pro tip! by sara amato during this last editorial cycle, c4lj brought on 4 additional editors:  eric hanson (johns hopkins university), rebecca hirsch (yale university), norma palomino (syracuse university), and mark swenson (winnetka-northfield public library district).  this brings us to an all time high of 15 editors. this was done in order to provide some overlap with editors who have indicated their intention to step down, to provide for some capacity for editors to sit out an issue when other priorities arise, and to accommodate issues that have a particularly high volume of quality proposals.  (i’ll confess that with this issue there were some good articles that didn’t make it in due to lack of an editor.)   this seemed like a good time to revisit ron peterson’s editorial “seeking a diversity of voices” http://journal.code4lib.org/articles/9345, and his follow-up last year. as an editorial board, we appear to be mostly white with mostly a technical services, archives or systems background, and mostly from academic institutions, though there is some vendor and consortia representation.   we have members from the united states, canada and germany.    we are pleased that our new editors include representation from a public library, and slightly improve the gender balance (40% identifying as female / 60% as male).   we could do a better job of both ethnic and institutional diversity.  we have no members who identify as working in public services positions, which may not be surprising given the subject matter of the journal, but does perhaps give us some tunnel vision.    our usual recruitment for new editorial board members is through the c4l listserv and shout-outs at the conference, but we may wish to rethink this in the future to target more diverse candidates. as of september, 2016, all submitters to the journal have been asked to fill out an optional demographic questionnaire, which 37% of submitters opted to complete.  only half of those submitted an ethnicity, with all but one self identifying as white, or a subcategory thereof, e.g. ‘white, arab american’.     one could draw a myriad of conclusions from this (lack of submission diversity and unwillingness to identify as other come to mind), though it is hard to know more without more complete data. however, using the gender portion of the demographic questionnaire combined with best guesses at name gender, which is admittedly fallible, it is possible to classify by gender 90% of the submissions over the past 18 months. in peterson’s 2014 editorial he noted that “women make up less than 40% of the authors published in the journal.”   this raised the question of how do acceptance rates compare to submission rates by gender.   in the chart below you can see that while the majority of submissions are by male authors, their acceptance rate is not proportionally higher than for submissions by female authors.  perhaps most notable is if an article is authored by a mixed gender group, it appears more likely to be published.  one can only guess as to the reasons for this, but let us hope that is because when we increase the diversity of voices, we end up with better products. pro tip c4lj submission to publication ratios by gender so here is a pro tip – when working on your next submission to the c4lj, gather a diversity of inputs to better your chances of being published. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – editorial mission editorial committee process and structure code4lib issue 47, 2020-02-17 editorial on diversity and mentoring by péter király last month in the europeanatech mailing list the organizers announced the 2020 iiif conference [1]. in the announcement there was a sentence about diversity: “only panel sessions that have a diverse set of panelists will be considered, with representation along many possible dimensions, including but not limited to race and ethnicity, national origin, gender identity and expression, sexual orientation, disability, age, and socio-economic status.” one of the mailing list recipients expressed his opinion that this sentence gives priority to diversity over the quality of content. the mail triggered a passionate discussion on this question without a real conclusion — i won’t repeat arguments here, if you are interested, please read the mailing list archive. this discussion gives me an apropos to bring up the topic again, which has been already discussed multiple times in previous code4lib editorials, focusing this time on geographic diversity. the majority of code4lib authors are affiliated with north american organizations (however based on their names they might come originally from different parts of the world). fortunately in almost every issue we have an article from europe, and from time to time we have articles from asia and australia. as far as i remember we have never published any articles submitted from african and south american cultural heritage organizations. thinking about the reasons behind it, i think that one of the reasons might be the way we select proposals. our editorial process starts with the evaluation of the submitted abstracts. the result of the evaluation is binary: yes or no. sometimes it happens that we ask questions from the submitter or ask her to refine the abstract, but in the evaluation period we have time constraints (usually there is an overlap between abstract evaluation and the final steps of publishing the current issue, plus all of us edit code4lib in our spare time). all in all, the abstract is expected to be ready for the evaluation. this system does not provide a real feedback loop to authors who do not have experience with writing proposals and articles. to encourage submissions from outside of north america and europe, i’d like to suggest a kind of mentorship process, in which the author could start discussion on her plans before submission. to start it immediately, here are some hints to prospected authors. in the evaluation process we consider different aspects: the article first and foremost should fit with the thematic scope of code4lib, i.e. it should discuss an important it aspect of cultural heritage. we receive very promising proposals which however are out of scope, so we finally don’t accept them. second, it should have a clear, well expressed focus. third, the article should have novelty. it does not mean, that it should talk about a high-tech, avantguard and groundbreaking development. we are looking for case studies, application of well known solutions in different environment. our purpose is to publish articles that would be useful for others. if the approach is applicable in organizations similar to the author’s organization, and the details of that setup has not been published elsewhere, it fits well to our profile. finally proficiency in english is not a requirement. during the editorial process we can help authors to fix language issues. this preliminary discussion is but a small step; it is not equivalent to an academic writing course, or a textbook on the topic, but it is something i could do, and perhaps it will be helpful for some professionals, and in addition might improve the geographic diversity of the code4lib journal. if you are interested, please contact me via kirunews(at)gmail.com. happy reading and happy coding! [1] https://list.ecompass.nl/listserv/cgi-bin/wa?a2=ind2001&l=europeana-tech&d=0&p=965 subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – homegrown worldcat reclamation: utilizing oclc’s worldcat metadata api to reconcile your library’s holdings mission editorial committee process and structure code4lib issue 27, 2015-01-21 homegrown worldcat reclamation: utilizing oclc’s worldcat metadata api to reconcile your library’s holdings oclc’s worldcat metadata api now allows libraries to set and delete their oclc holdings without using connexion or batch services. st. olaf college libraries had used up our “one free turn” at a reclamation several years ago. unfortunately due to the normal inconsistencies that accumulate over time, as well as problems with holdings being set for us by external entities, we had reached a point where we were once again in need of reconciling our holdings with oclc’s. this time we wanted to accomplish the reclamation in a low-cost way that would also allow us to have more local control over the process. using the worldcat search api and metadata api in tandem, we first retrieved all oclc numbers with holdings currently set and deleted these holdings with a fairly simple perl script. we then pulled from our local catalog all the oclc numbers for which we wanted holdings set and updated again using the metadata api. the result, in the words of one of our catalogers, is that “for the first time since i got here, 15 years ago, i feel our holdings finally reflect what we really own.” in this article, i will discuss the issues to consider if you wish to do a similar project with your oclc holdings, share the perl scripts i wrote for processing, and reflect on the pros and cons of the process overall. by sarah johnston background maintaining holdings in oclc over the course of 30 years is bound to lead to inconsistencies between a library’s catalog and the holdings represented in worldcat. the problem is common enough that oclc has long offered a service called reclamation (now grouped with similar services as batchload). the basic process for doing a reclamation project is: create a list for oclc of all the oclc numbers in your local catalog for which you wish to have holdings set. oclc runs this file against the worldcat database and sets all matches with a current timestamp. reports are generated of (a) oclc numbers from the local catalog that had no holdings match in worldcat and (b) oclc numbers for which your institution had worldcat holdings set but no number was included in the file from the local catalog. you tell oclc whether you want to set holdings for report a and/or delete holdings for report b. oclc will do a reclamation project for free one time; subsequently you incur charges for going through this process.  a quote for a reclamation project can be requested through oclc’s online service center. at st. olaf college, we had attempted a reclamation project in 2009-2010. at that time, we ran into problems particularly with report a (although it impacted the report b). because we didn’t subscribe to the bibliographic record notification service, we had many outdated oclc numbers in our catalog (when an oclc record is merged with another, the old number(s) is stored in the 019). we received an “xref” report but the actual processing didn’t update the holdings on these records. we were uncomfortable moving ahead with the project when so many of our records had no matches in worldcat and there were so many potential deletes because of oclc record merges. over the past two years, we noticed more and more interlibrary loan requests coming in for things we did not own. we were also having problems with some of our e-resource vendors setting holdings for us on collections we had not acquired. we knew it was time to tackle the reclamation again, but we had used up our “free turn” and we wanted to have more control over the process. oclc web services came to our rescue. oclc web services before we dive into the specifics of our project, a few words about oclc’s web services platform might be helpful to those that have not used it yet. each api has different requirements for its use, as well as terms of use. the metadata api is available to any institution with a cataloging and firstsearch subscription. the first step is to obtain a wskey, which can be requested through the oclc service configuration interface. you can also request a key specifically for their sandbox, which is a great help in testing and viewing responses. the metadata api also requires attribution to a worldshare user (used in the authentication process as the principal id). when you request access to the metadata api, you will also receive this authorization in your email confirmation. considerations before you start if you plan to remove all your oclc holdings before adding holdings (which is the process i will outline in this article), consider when the optimum time to do this might be, as it will affect patrons and library staff searching worldcat during the project. we had hoped to do our project during the summer when, as an academic library, we have the lowest use of worldcat. for reasons i discuss in more detail below, the project was pushed to september, but because of other timing constraints we chose to move forward while making sure the rest of the staff was well aware of the temporary impact on worldcat. there will also be an impact on interlibrary loan, particularly if you and your primary borrowers use illiad. we made sure our ill staff knew that they should check our local catalog before sending ill requests rather than relying on illiad (which in turn relies on worldcat holdings). finally, it is important that anyone who adds holdings for your institution stop during the project lest their additions and deletions be overwritten. we kept a good record of the “gap” holdings: depending on your local system and practices, this may be as easy as running a query at the end of the project for all items added or deleted during the process, or as difficult as writing down every oclc number for items added or deleted during the project. deleting the holdings because oclc does not offer a way to query when a holding was set, it is necessary to delete all your holdings before starting the resetting process (otherwise you would have no way of finding the holdings that no longer matched records in your local system).  for the deletion phase of our project, we used two apis: metadata and search. we wanted to delete all holdings except items that had an lhr (local holdings record).  we had about 7000 records with lhrs, all serials.  as a trial run, i created a perl script to send an http request (to the metadata api) sequentially for all oclc numbers, starting at 1. we quickly ran into two issues with this approach: it would take forever (months) to run through all billion + oclc numbers, and it overloaded the logs on oclc’s servers. we decided to pause the project at that point and rethink our approach. oclc was very responsive to the project and worked to improve their logging capabilities. after about a month, we received word from oclc that we could proceed with no limit on the number of http requests sent per day. as oclc was working on this, i was considering other approaches. prior to beginning the project, we had asked oclc if it would be possible to get a file of all oclc numbers on which our holdings were currently set. unfortunately that wasn’t possible at the time.  since we completed our project, oclc has developed a way to create this data file using their collection manager product.  this is a new process for which oclc is currently creating documentation, which should be available soon. casting about for another way to get a smaller set of oclc numbers to work from, rather than the enitre universe of oclc numbers, i started playing with the oclc search api. this web service is designed for incorporating worldcat results into a discovery service, but since it returns the oclc number and allows you to limit by holdings symbol, i thought that with a broad enough search i might be able to use it to pull back the majority of our holdings, and from there we would be able to finish the delete processing in connexion. the search api returns 10,000 hits in 100 record increments. my script saved each xml file separately, then used xpath to locate and extract the oclc numbers from each file into a text file. once this was completed for the set of 10,000, the script began sending the numbers to the metadata api, using the http delete method. oclc has excellent documentation on the metadata api, and i won’t go into great detail here about how to set up the http requests. my script should give you a good idea of how we utilized the api with perl [https://github.com/sarahzum/reclamation.git]. note that oclc has recently added a new authentication method to the metadata api. i used the wskey hmac signature.   you can now also use the access token method.  the benefit of this method is that you don’t have to hash every request; the access token is set to expire after 20 minutes, although, if you are considering a similar project, oclc will extend that expiration period for you on a temporary basis to increase efficiency even more. if you are interested in doing this, contact devnet@oclc.org. their only caveat is that you inform them when the project is complete so the wskey can be reset to issue shorter lived tokens. the delete processing took us about 1 week to complete for approximately half a million holdings.  this approach had its drawbacks: because we were using the search api for a purpose it was not designed for, we had to do some “creative” keyword searching to return records with our holdings. you cannot simply send a “limiter” search like oclc holding symbol without including a non-limiter index search [http://www.oclc.org/developer/develop/web-services/worldcat-search-api/bibliographic-resource.en.html]. the search api returns the full record in marcxml or dublin core.  we chose to save and process full marcxml records in order to grab the oclc number for each one, which was time and processor intensive. in the end it did prove impossible to return all our records this way. we had to do the final deletes in connexion, using a combination of our holding symbol and material type (a combination not allowed in the search api–see above). in retrospect, i would have approached this part of the project as follows: exported all numbers in the 001 and 019 fields from our local system (we use iii millennium currently) into a text file. looped through that file sending each as a delete request to the metadata api. used connexion and/or the search api (depending on how many records were left) to catch any remaining records with our holdings set. the metadata api handles old oclc numbers well: it finds the current record and deletes your holdings, so including the old oclc numbers in the 019 might be overkill, but in my opinion it might find some things where you have an incorrect number in the 001 (or wherever you store the current oclc number on your local system) and save you some work later on. figure 1.search results for our holdings symbol in firstsearch when the delete processing was complete restoring the holdings as thrilling as it is to see your oclc holdings dwindle to almost zero, the time comes to start adding those holdings back to the worldcat database. the key intellectual work at this stage is deciding which records in your local system should have holdings set. this is a perfect opportunity to consider discovery in worldcat and anywhere else you may want to draw from worldcat data. what items do you want library patrons to find? just as importantly, are there things you don’t want patrons to know you have, particularly those patrons that would use worldcat to view your collections? at my college, we were concerned that some of the items in our special collections that we did not wish to advertise (since we have very restricted access to them) had holdings set.  we have also decided not to set holdings on electronic resources, primarily because we wanted to be able to track those “mystery” holdings that were popping up from vendors for collections we had not purchased. in the end, we used the following criteria in our local system: oclc number in the 001 not suppressed (publicly viewable in our local catalog) not electronic-only (available in a physical format) we exported the oclc numbers from all records that met this criteria. we then used the holdings_add perl script [https://github.com/sarahzum/reclamation.git] to send an http post request to set the holdings for each number, looping through the text file until the process was complete. this processing went a bit more quickly than the delete process, finishing in just under 1 week. results in raw numbers, we had about 520,000 records with holdings set prior to beginning the project. once the reloading process was complete, we had about 480,000 records with holdings set.  a not insubstantial difference! during both the delete and set phases of the project, i kept logs of each response. i have not had a chance to analyze the logs yet to see if they can tell me anything about the overall accuracy of our holdings preand postproject, but anecdotally we’ve noted some positive trends: we’ve had a few interlibrary loan requests come in for items that we couldn’t find in our local catalog, however they have all been attributable to a recently weeded item (we delete holdings in oclc for weeded materials once per week) or the vagaries of serials in both our local catalog and our lhr statements in worldcat. we have an ongoing project to clean up materials with lhrs so we hope to resolve some of this confusion soon. since we decided not to add holdings for anything we hold solely electronically, we have also been able to monitor electronic resources and make sure that those “phantom” holdings are not popping up again. we hope to repeat this process again next summer, using some of the improvements i’ve noted in the article above.  we’d like to do a more sophisticated analysis of the accuracy of the oclc numbers in our records, perhaps using the xid api to compare other identifiers in our records. we feel much more secure in the links between worldcat and our local catalog. this is a huge benefit to anyone searching the worldcat database of course, but also important to interlibrary loan, collection development (we use oclc’s collection analysis program), and the many future uses we’d like to make of resources like the new worldcat discovery api.  many thanks to anna sylvester, joe abrams, and john chapman at oclc for their technical assistance with our project.  also thanks to karen coombs and shelley hostetler for additional information about oclc’s web services provided for this article. references worldcat search api: http://www.oclc.org/developer/develop/web-services/worldcat-search-api.en.html worldcat metadata api: http://oclc.org/developer/develop/web-services/worldcat-metadata-api.en.html about the author sarah johnston is the systems & web services librarian at st. olaf college in northfield, mn.  she is currently chairing the group that will select the next library management system for st. olaf and carleton college.  she loves data cleanup, but only if it can be automated!   her website is at http://sarahjohnston.info/ . subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – bridging technologies to efficiently arrange and describe digital archives: the bentley historical library’s archivesspace-archivematica-dspace workflow integration project mission editorial committee process and structure code4lib issue 35, 2017-01-30 bridging technologies to efficiently arrange and describe digital archives: the bentley historical library’s archivesspace-archivematica-dspace workflow integration project in recent years, archivesspace and archivematica have emerged as two of the most exciting open source platforms for working with digital archives. the former manages accessions and collections and provides a framework for entering descriptive, administrative, rights, and other metadata. the latter ingests digital content and prepares information packages for long-term preservation and access. in october 2016, the bentley historical library wrapped up a two-year, $355,000 grant from the andrew w. mellon foundation to partner with the university of michigan library on the integration of these two systems in an end-to-end workflow that will include the automated deposit of content into a dspace repository. this article provides context of the project and offers an in-depth exploration of the project’s key development tasks, all of which were provided by artefactual systems, the developers of archivematica (code available at https://github.com/artefactual-labs/appraisal-tab). by max eckard, dallas pillen and mike shallcross project background in april 2014, the university of michigan’s bentley historical library (“the bentley”) received a $355,000 two-year grant from the andrew w. mellon foundation to partner with the main university of michigan library to integrate functionality of archivesspace, archivematica, and dspace in an end-to-end digital archives workflow. the bentley has managed and preserved major collections of born-digital archives since the 1997 accession of a former university president’s personal computer. for many years the library tailored digital preservation strategies to specific collections and procedures were largely manual due to limited staff and technical skills. in 2009, the bentley received a two-year grant from the mellon foundation, “email archiving at the university of michigan,” that added additional storage infrastructure (including the use of deep blue, the university of michigan’s dspace repository) as well as two staff members to enhance and standardize digital preservation policies and procedures.[1] this grant-funded work led michael shallcross to experiment with automating key digital preservation procedures and in 2011, the library implemented a local digital preservation tool, the automatedprocessor (autopro).[2] inspired by the archivematica digital preservation system’s micro-service design and pipeline workflow, this resource automated key portions of the digital preservation workflow and helped standardize the creation of metadata and log files. autopro provided a single interface for more than twenty applications and guided users through a standardized workflow that included steps such as a virus scan, the identification of personally identifiable information, and migration of content to sustainable file formats. however, as a command line tool written in windows cmd.exe shell scripts, autopro was never intended to be a long-term solution given its limited error handling, poor user interface, and support issues (primarily installing and updating the cmd.exe scripts and component utilities on multiple workstations). as acquisitions of digital archives continued to grow and the bentley sought to involve more of its staff in working with such content, it was clear that a more robust and user-friendly solution was required. in reviewing available open-source tools, the bentley realized that archivesspace and archivematica had the potential to be integrated with the local dspace repository in an end-to-end digital archives workflow. each of these systems addressed important aspects of the bentley’s approach to preserving and providing access to born-digital archives: archivesspace (aspace) is an open-source archival management software that combines the best features of archon and archivists’ toolkit. this system permits institutions to track accessions, manage collections, and generate encoded archival description (ead) finding aids and marcxml. development of aspace was funded by the andrew w. mellon foundation (2011-2013) and lyrasis now serves as its institutional home.[3] archivematica is a free and open-source digital preservation system developed by artefactual systems (british columbia). archivematica employs a micro-service design to “provide an integrated suite of software tools that allows users to process digital objects from ingest to access in compliance with the iso-oais functional model” and furthermore employs mets and premis to record and track descriptive, administrative, and rights metadata.[4] dspace is an open-source repository platform that “preserves and enables easy and open access to all types of digital content including text, images, moving images, mpegs and data sets.” initially developed by mit libraries with partial support from a mellon foundation grant, dspace has acquired a growing community of developers and is employed by the university of michigan and approximately 1,400 other academic, nonprofit, and commercial organizations around the world.[5] by integrating functionality and sharing information among these platforms, the bentley’s basic goals for the project were to: facilitate the creation/reuse of descriptive and administrative metadata across preservation and management systems. streamline the ingest and deposit of content in a preservation repository. find solutions that meet the bentley’s local needs, but which are also flexible and scalable for other institutions; modular, so that institutions may adopt some, none, or all of the development features; and based upon open standards so that other tools and/or repository platforms could be integrated. share all code and documentation with the archives and digital preservation communities. as staff at the bentley and the university of michigan library worked with project developers from artefactual systems on user stories and a review of functional requirements, a list of significant development tasks emerged, which included: the introduction of functionality to appraise and review content from within archivematica so that archivists can acquire a fuller understanding of the nature and scope of born-digital collections. allowing archivists to perform this work in the archivematica dashboard would obviate the need to navigate and record information from multiple external tools and systems and greatly facilitate the training of staff. the resulting ‘appraisal and arrangement’ tab in archivematica was designed to allow users to characterize distributions of file formats within acquisitions, identify sensitive data, and preview content and also to apply tags for the purpose of recording information about restrictions, deaccessions, and/or the intellectual arrangement of materials. the integration of archivematica and archivesspace via the introduction of an archivesspace ‘pane’ within the appraisal and arrangement tab. this feature utilizes the archivesspace application programming interface (api) to create and edit descriptive metadata for new or existing archival objects in archivesspace; display resource records in a tree view, depicting the intellectual hierarchy of archival objects; and drag and drop digital content onto archival description to create archivesspace digital object records and launch the ingest of submission information packages (sips) in archivematica. the integration of archivematica and dspace so that fully ingested content (a single archival information package, or “aip”) and associated descriptive metadata will be automatically uploaded as a unique item in dspace. the persistent url (a “handle”) will in turn be written back to archivesspace so that it may serve as a link to the digital content via the aspace public user interface or when archival description is exported to an encoded archival description (ead) finding aid. see below for a representation of this data flow: figure 1. dataflow between archivesspace, archivematica and dspace mapped to sequential actions of the digital curation centre (dcc) curation lifecycle model. the primary use case employed for the planning and development aspects of the project involved a large accession of digital archives from the office of former u.s. senator carl levin, which amounted to 1.2 tb of data and some 217,135 files. this content was selected for its scale, diverse array of content, and variation in structure (from meticulously organized folders to loose aggregations of files), with the acknowledgement that it would inform and be applicable to other use cases. at the same time, the archivesspace-archivematica-dspace workflow integration project is intended to be flexible enough to accommodate a wide variety of local practices and collections. code for the appraisal and arrangement tab will be integrated in version 1.6 of archivematica (to be released in 2017) and is currently available at https://github.com/artefactual-labs/appraisal-tab. in-depth review of key development tasks/project features appraisal and arrangement tab archivists begin by creating records for all accessions, regardless of format, in archivesspace. accession records are then associated with an existing archivesspace resource record (i.e., finding aid) or, if necessary, an archivist may generate a new resource record based on accession record data. any digital content associated with the accession is then transferred into archivematica’s backlog. during this process, archivematica runs through a number of micro-services, assigning uuids to transfers and files, assigning checksums (or verifying them if present), unzipping zipped or otherwise packaged files, scanning for viruses, indexing, performing file format identification and validation, and extracting technical metadata. transfers remain in the backlog with a minimum level of control until they are ready to be retrieved and processed at a later date. the new appraisal tab sits between the current transfer and ingest tabs in archivematica, and provides a number of features that allow archivists to appraise and review content from within archivematica so that archivists can acquire a fuller understanding of the nature and scope of born-digital collections. these features include the ability to characterize transfers, identify sensitive data, preview content, and tag content for restrictions and other purposes as well as create an intellectual arrangement for materials. after performing a search (e.g., by accession number) for one or more transfers in the archivematica backlog, archivists may select an entire transfer, a folder within a transfer, or individual files and perform a number of analyses on this content based on technical and other metadata extracted during archivematica’s initial transfer micro-services. the “objects” analysis, for example, gives a report of file formats which archivists may view as a tabular report or as a visualization. figure 2. tabular report of file formats in a transfer folder figure 3. visualization of file formats in a transfer folder selecting a format or format policy registry (fpr) group from either the tabular report or visualization applies a facet to the transfer selection and populates the transfer file list pane with files of that format. archivists may preview individual files from this pane; file formats that are supported by the archivist’s browser (e.g., pdf and many image file formats) will be displayed in the appraisal tab, while a copy of any file can be downloaded for viewing locally using the utility of the archivist’s choice. figure 4. previewing a pdf in the archivematica dashboard the “examine contents” analysis allows archivists to identify sensitive data by exploring bulk_extractor log content for personally identifiable information (pii) such as social security numbers and credit card numbers. when an individual file is selected, archivists get a tabular view of the content of a hit and its surrounding context. the identification of this type of data can be used to inform appraisal and processing decisions, e.g., after confirming that a hit is indeed an example of pii, an archivist may choose to separate this content or apply a particular access restriction. at this time, this functionality is only informative, as the user is not able to redact the pii from content. figure 5. identifying (valid but fake) social security numbers using bulk_extractor logs finally, archivists may apply ephemeral tags (i.e., they are not maintained as part of the aip) to content in a number of places. intended as a type of virtual post-it note, possible use cases for tags include: tagging everything an archivist plans to arrange to a particular series or sub-series (and later use tags to facet objects in the backlog to facilitate this arrangement). tagging sensitive or restricted content as it is identified. applying tags as a simple aide-memoire during processing. figure 6. tagging in the backlog pane and the file list pane, and viewing a “tags” analysis archivematica-archivesspace integration the integration of archivematica and archivesspace was designed to allow archivists to search for archivesspace resource records, to edit or add to existing archival description, and to create archivesspace digital object instances with which content from the archivematica backlog can be associated from within the archivematica appraisal and arrangement tab. this functionality, which uses the archivesspace api, allows archivists to associate digital content with archival description without navigating between multiple systems and rekeying or copying and pasting metadata, achieving one of the key project goals of facilitating the creation/reuse of descriptive metadata. integration with an archivesspace instance is configured in the archivesspace dip upload section of archivematica’s administration tab. here, an archivematica administrative user may enter the archivesspace instance’s host url, the backend port at which the archivesspace api is running, and the username and password of an archivesspace user with sufficient administrative privileges. in this example, we have created a user with the username “archivematica” in the local archivesspace instance and, within that archivesspace instance, given that user permissions to view, edit, and create archivesspace resources and digital objects. no additional configuration files or changes to settings within archivesspace are necessary to integrate archivematica and archivesspace. figure 7. configuring archivesspace integration within archivematica within the appraisal tab’s archivesspace pane, archivists may search for an existing archivesspace resource by title or identifier and are presented with the full resource trees, including series, subseries, and additional levels of intellectual arrangement, for any matching collections: figure 8. an archivesspace resource record rendered in the appraisal tab by selecting a particular archival object within the resource tree, archivists may edit a minimal amount of existing metadata, including the archival object’s title, level of description, general note, and conditions governing access note: figure 9. “edit metadata” template archivists may also create a new child archival object below the selected component and enter a slightly expanded set of metadata: figure 10. new child record metadata template the metadata fields that may be edited or created within the archivesspace pane reflect both the minimum metadata fields required to create an archivesspace archival object (level of description and a title or a date) in addition to a few select fields–the “general note” (mapped to “general” note in archivesspace and the <odd> other descriptive data tag in ead) and the “conditions governing access note,” for example–that bentley archivists frequently use to describe digital content. more extensive metadata editing or creation, including the creation of additional notes or the reordering of archival objects within a resource tree, will be done directly in archivesspace rather than in the appraisal tab. once an archivesspace archival object with which to associate digital content has been identified or created, an archivesspace digital object instance can be associated with that archival object. relevant content from archivematica’s transfer backlog can then be associated with the digital object by dragging and dropping folders or individual files from the backlog pane onto the digital object. after all relevant content has been associated with a given digital object, archivists will finalize arrangement to initiate archivematica’s ingest workflow and the deposit of content to dspace. figure 11. the “newsletters” folder from the backlog pane dragged and dropped onto an aspace digital object archivematica-dspace integration the integration of archivematica and dspace was designed to streamline the ingest and deposit of content into a preservation and access repository. once processors have finalized appraisal and arrangement, archivematica runs sips through several ingest micro-services, including normalization, and packages them into aips. aips are then automatically deposited into dspace utilizing a number of new features that were developed as part of this project. archivists configure the dspace integration within archivematica’s storage service.[6] a new “dspace via sword v2” archivematica “space” points to a particular instance of dspace and an associated service document (which describes the “contract” between a particular user and a repository), a username and password and a dspace “group” (a kind of entity that can be granted permissions in the authorization system) to be used for restricted metadata. new “locations” on that space (which inherit the same configurations) point archivematica to particular dspace collections using the collection’s handle and a short description (such as the collection title).. figure 12. configuring a “space” for “dspace via sword2 api” within archivematica’s storage service figure 13. configuring a “location” for a dspace collection within archivematica’s storage service during archivematica’s final ingest microservice, “store aip,” archivists select (or preconfigure) the archivematica location corresponding to the appropriate dspace collection to which the aip will be moved. figure 14. selecting a dspace collection during archivematica’s “store aip” micro-service when the “store aip” process begins, archivematica fetches key bits of descriptive metadata about the aip from archivesspace, such as: a display title that concatenates a component’s title and creation date(s). an abstract, mapped from the general note. the creator or author. these bits of information (which are also included in the mets file of the aip) are used to populate metadata fields on the dspace item to facilitate searching and browsing within dspace (and for search engine optimization). other bits of information, such as the date of publication and a copyright statement, are generated on-the-fly or hard-coded. aips in archivematica (which, by default, are packaged in accordance to the library of congress bagit specification and whose data directory consists of the mets file for the aip and three folders: logs, objects and thumbnails) are then repackaged into “objects” and “metadata” packages: the objects package contains the objects folder and includes original and normalized objects as well as some associated metadata and submission documentation. the metadata package contains everything else, e.g., the normalization log, malware scan log and the extraction log. it also includes, importantly, bulk_extractor logs that indicate the precise location of the credit card numbers, social security numbers and other types of sensitive information. in dspace, items and their associated bitstreams inherit the permissions of the collections or communities to which they belong. for the bentley, this means that unless otherwise stated, items and their bitstreams that are deposited using this method are openly accessible to anyone. in most cases, this is desirable behavior for the objects package. when it is deposited, it will be open (i.e., belonging to the “anonymous” group in dspace)[8]. however, in order to ensure that the sensitive data contained in the metadata package does not remain public, archivematica applies the group configured earlier for restricted metadata in the dspace space. in the bentley’s case, this means that only dspace users that belong to a “bentley archivists” group may access the bitstream. figure 15. a dspace item with open and restricted bitstreams data deposit is facilitated by the simple web-service offering repository deposit (sword v2) protocol, an interoperability standard that allows digital repository servers to accept the deposit of content from clients via a standardized protocol.[7] with it, an item is posted to a particular dspace collection with minimal metadata and two bitstreams are posted to that item, one each for the objects and metadata packages. while archivematica goes on to use the dspace api to update the permissions on the metadata bitstream (there are problems with the way that sword v2 handles permissions), using sword v2 for the initial deposit keeps this integration in line with the project’s goal that outcomes are flexible and modular. dspace is only one of a number of sword-compliant repositories (others include arxiv, dataverse, eprints and fedora); theoretically, an institution may replace dspace with any of these other repository platforms. keeping the deposit simple (e.g., with minimal metadata and a predictable structure and outcomes) also ensures that, even with the use of the dspace api, this integration will work with other instances of dspace, not just the university of michigan’s highly customized implementation of dspace, deep blue. finally, archivematica completes the deposit by making use of the archivesspace api to write the handle of the dspace item back to the file uri of the linked digital object in archivesspace. this ensures that archivists can manage the locations of collection components, physical and digital, in archivesspace. conclusion the archivesspace-archivematica-dspace workflow integration project concluded on october 31, 2016 with the successful completion of all development tasks and the code will be included in the release of archivematica version 1.6. to use the new appraisal functionality, interested parties will simply need to install or upgrade archivematica and the archivematica storage service. to use the optional archivesspace and/or dspace integrations, archivists or system administrators will need to configure the integrations by pointing archivematica at the appropriate instances of archivesspace and/or dspace (with users that have adequate authorizations) and enable sword v2 in dspace as well as the rest api, which are disabled by default. the bentley is currently working with artefactual systems on some additional development work to fully integrate the new features into the university of michigan’s local infrastructure and processes. while the project has resulted in fully functional deliverables that may be employed by members of the archives and digital preservation communities, the project team hopes that these outcomes evolve and improve as more institutions employ the appraisal and arrangement tab and adapt it to local needs or integrate new functionality. in recent professional meetings (including the 2016 society of american archivists annual meeting and the inaugural archivematica camp[9]), peers have suggested a variety of exciting potential features and improvements, which include: treemap visualizations of files and folders integration of tim walsh’s brunnehilde, “a python-based reporting tool for born-digital files that builds on richard lehane’s siegfried. brunnhilde runs siegfried against a specified directory, loads the results into a sqlite3 database, and queries the database to generate csv reports to aid in triage, arrangement, and description of digital archives.”[10] introduction of additional analysis tools to work with indexed contents of digital files, such as named entity recognition, topic modeling, natural language processing, network analysis visualizations, etc. we look forward to working with other members of the archives and digital preservation communities on expanding the functionality and applications of this integration project. readers are encouraged to see the project blog (http://archival-integration.blogspot.com/) or to contact the bentley’s grant team (bhl-mellon-grant@umich.edu) for more information or updates on the work with archivematica and the new appraisal and arrangement tab. notes [1] the bentley historical library’s archival community in deep blue is available at https://deepblue.lib.umich.edu/handle/2027.42/65133. [2] for more information on autopro, please see this poster and short paper from the 2012 ipres conference at http://hdl.handle.net/2027.42/95923. [3] for more information on archivesspace, please see http://archivesspace.org/. [4] for more information on archivematica, please see https://www.archivematica.org. [5] for more information on dspace, please see http://www.dspace.org/. [6] for more information on the archivematica storage service, please see https://www.archivematica.org/en/docs/storage-service-0.9/. [7] for more information on the sword protocol, please see http://swordapp.org/sword-v2/. [8] archivematica does not support the automated deposit of items to dspace with objects packages that need access restrictions. alternative, local workflows are being developed for this type of material. [9] for more information on the 2016 archivematica camp, please see https://wiki.archivematica.org/community/camps/umsi2016. [10] for more information on brunnehilde, please see https://github.com/timothyryanwalsh/brunnhilde. author biographies max eckard (eckardm@umich.edu) is the assistant archivist for digital curation with the bentley historical library at the university of michigan. prior to that, he was the metadata and digital curation librarian at grand valley state university. a graduate of the school of library and information sciences at north carolina central university, he is passionate about digital curation, users and user experience, and service. dallas pillen (djpillen@umich.edu) is the assistant archivist for metadata and digital projects at the university of michigan’s bentley historical library, primarily involved with the bentley’s archivesspace-archivematica-dspace workflow integration project, its archivesspace implementation, and its web archiving program. mike shallcross (shallcro@umich.edu) is the assistant director for curation at the bentley historical library and an adjunct instructor at the university of michigan school of information. in addition to his extensive experience developing and implementing digital archives workflows, he oversees the bentley’s digitization program, conservation unit, and processing operations for digital, analog, and physical materials. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – using oai-ore to transform digital repositories into interoperable storage and services applications mission editorial committee process and structure code4lib issue 6, 2009-03-30 using oai-ore to transform digital repositories into interoperable storage and services applications in the digital age libraries are required to manage large numbers of diverse objects. one advantage of digital objects over fixed physical objects is the flexibility of ‘binding’ them into publications or other useful aggregated intellectual entities while retaining the ability to reuse them independently in other contexts. an emerging framework for managing flexible aggregations of digital objects is provided by the open archives initiative (oai) with its work on object reuse and exchange (ore). this paper will show how oai-ore is being used to manage content in digital repositories, in particular institutional repositories, and has the potential ultimately to transform the conception of digital repositories. by david tarrant, ben o’steen, tim brody, steve hitchcock, neil jefferies and leslie carr introduction one of the main decisions to be made when starting a digital repository is which software to use. currently there are a wide range of choices, including open source and proprietary systems. most of these choices are in active development and some have been available for nearly a decade, which represents a high degree of stability in the relatively young history of digital repositories. not every institution creating a repository will make the right initial choice of software, however, and over time the needs of a repository may evolve. not every repository software will survive. the well managed repository will have a preservation strategy, but what happens if the underlying software has to change? while repositories, especially the open source variety, are not designed for data lock-in, different data models mean that lossless data transfer between repositories has not been simple. this is about to change with the emergence of oai-ore. the starting point for our work was an investigation into institutional repository (ir) preservation services by the jisc-funded preserv 2 project [25], which wanted to find a way of effectively replicating a whole ir across any repository platform. the aim was to avoid tying digital objects to the repository software used to manage them. the vision was of a set of low-level resources, which are then managed by a series of services including repository software such as eprints and fedora. the oai-ore specification includes approaches for representing digital objects and facilitates access and ingest of these representations beyond the borders of hosting repositories, enabling a new generation of cross-repository services. in this way oai-ore standardises the description of the relationship between digital objects. this relationship could be between versions of an object, such as might be found in a repository record, or aggregations of objects, such as a web page with images, or a collection of chapters in a book. we also envisaged applying oai-ore to a much larger collection – a complete repository. this paper describes an award-winning [27] rapid development project, entered into a competition [23] sponsored by the common repository interface group (crig) [5] at the open repositories conference 2008 [21]. using oai-ore with a simple extension that enables the import process to operate more quickly, this project was able to replicate not only objects but also all the metadata and history data for those objects across repositories.  the demonstration combined oai-ore with the fedora and eprints repository platforms and culminated in a lossless transfer of two live archives from one software to the other. the implication of this approach is that repositories could become interfaces to remote storage where content is represented and accessed via an oai-ore resource map. rather than repositories storing and organizing objects internally and then revealing this content to other services through oai-ore maps, the repository understands oai-ore maps at the lowest level. further, these maps need not be exclusively managed by the repository itself. for example, we might have resources (documents, pdfs) stored directly in an open storage platform and an oai-ore map binds these resources into collections. in this scenario, objects and resource maps are no longer controlled by, and reliant on, a single repository software. instead, we can envisage the prospect of many repository softwares (eprints, fedora, etc.) running over one set of resources. while this vision is some way off, the application of oai-ore makes this transformation from repositories-as-software to a services-based conception a real possibility. this paper provides a concise introduction to the oai-ore specification, from its roots in web architecture through the development of multiple serialisation formats. we focus on how oai-ore can be used to represent aggregated publication records, expanding this to show how oai-ore can be used to map an entire repository and its content. we explain how oai-ore was used to transport the entire contents of one repository into another, and vice-versa (with the two repositories running seemingly incompatible softwares). since the initial demonstration (done one day before oai-ore was formally announced [18]), some of the interfaces that were developed have been integrated into the respective repository softwares, showing that oai-ore already provides real benefit to the digital repository community. we also look briefly at possibilities for further work in this area. oai-ore is still in its infancy and currently there are not many user-facing applications. we expect that this will change and that oai-ore will play a greater role in binding digital objects both on the desktop and on the web without needing to move those objects into one static location. using oai-ore at a low level has a number of implications for repositories, in particular being able to construct a repository which consists of a set of interchangeable services. to understand how repositories might change, first we need to consider the current purpose and structure of repositories, in particular, of irs. institutional repositories and preservation irs are designed to host and disseminate academic research and institutional output, and they play a key role as a platform for managing digital content. a good ir will be driven by the needs of the institution, and the main benefits will arise from being embedded in institutional processes and cultures. these benefits include growth of content accessible to all web users, integration into institutional web presence, and automatic profile and cv generation for researchers. irs have also been shown to aid academic research evaluation [4]. each repository software solution performs three basic functions: ingest, management and dissemination of content. although the various repository software solutions are solving the same, or similar, set of problems, each platform is constructed with differing low-level architectures, including fundamental differences in how files and metadata are stored and organised. these differences mean that you cannot simply transport the raw contents from one platform to the other. from a high level standpoint, the repository platforms strive to provide compatibility by supporting protocols and specifications which allow easy ingest and discovery of the data they contain. figure 1 models a simplified three-stage repository showing example interfaces, or plug-ins, which can be used at each stage. an ideal repository implementation will take a completely modular approach such that plug-ins and extensions can be developed for each stage of the model. in this paper, the import (get content) and export (serve content) plug-in layers provide the mechanism for interoperability with other systems. figure 1: the 3 stage repository model. while oai-pmh has been fundamental in enabling interoperability between repositories, it is designed as a method of discovery; there is no corresponding import protocol with the same application programming interface (api). until now there has been no standard way of abstracting a complete object such that it can be transported from one repository platform to another without some loss of data. this is where oai-ore comes in. oai-ore specifies import and export interfaces to enable to re-use and exchange of digital objects. from a digital preservation perspective, this enables future migration of objects to a newer platform while preserving the functionality expected from the digital repository. a quick overview of oai-ore ‘open archives initiative object reuse and exchange (oai-ore) defines standards for the description and exchange of aggregations of web resources. these aggregations, sometimes called compound digital objects, may combine distributed resources with multiple media types including text, images, data, and video. the goal of these standards is to expose the rich content in these aggregations to applications that support authoring, deposit, exchange, visualization, reuse, and preservation. although a motivating use case for the work is the changing nature of scholarship and scholarly communication, and the need for cyberinfrastructure to support that scholarship, the intent of the effort is to develop standards that generalize across all web-based information including the increasing popular social networks of “web 2.0”.’ [20] this section looks at the relationship between oai-ore and irs and how oai-ore can be used to bind documents (agreggating metadata, history, content) in a repository as well as to bind the repository itself (aggregating objects and repository configuration). first, we consider the technical model behind oai-ore and how it has been constructed. the oai-ore model oai-ore is based entirely on the architecture of the web (as outlined in [11]) and advocates use of recent developments in the areas of the semantic web, linked data and cool uris [3]. the biggest influence on oai-ore, however, is the rdf model (described best by the w3c rdf primer [16]), which uses the idea of triples to describe things. a triple comprises a subject-predicate-object statement where each part is a uri, with the exception of the object, which can also be a plain text value. figure 2 presents a simple repository-focused example of the rdf model, using the dublin core namespace [6] to provide all of the predicates (and shows the classic use of the rdf model to annotate objects with metadata). figure 2: an rdf representation of publication metadata. oai-ore focuses on objects and the relationships between these objects, leading to a use similar to that shown in figure 3. here our publication object now has two parts, the publication itself (pdf) and the xml record which in this case represents the stored metadata. figure 3: rdf representation of a publication using dublin core. aggregations on top of the rdf model, oai-ore introduces the concept of aggregations and aggregated resources. an aggregation is simply a set of aggregated resources, all of which are represented by uris. figure 4 is adapted from the ore primer [13] and demonstrates how this concept applies to a publication resource in a repository. now that we are using the oai-ore methodology we see the use of the “ore” namespace (shorthanded from “http://www.openarchives.org/ore/terms/” to “ore:”), which is designed specifically for oai-ore. figure 4: example oai-ore aggregation of a publication. while figure 4 shows a single publication record, the abstract concept introduced by oai-ore is much more powerful and allows nesting of aggregations. for example, the highest level aggregation could be the repository and the aggregated resources thus become the publications, which in turn contain their own aggregated resources. there is no limit to the depth with which resources can be aggregated, although it is not recommended to go to too many levels due to the recursive operations that will need to be performed to import these resources. resource maps resource maps represent the highest level of the oai-ore model. a resource map has a uri and is used to describe a single aggregation. note that a resource map can only link to a single aggregation in the oai-ore model. figure 5 (taken from the ore primer [13]) shows the resource map (rem-1) to the left of the aggregation with some example metadata that is expressed using the dublin core namespace. based on rdf, the resource map is able to support any namespace and metadata the user wishes to add. figure 5: complete oai-ore model representations rdf/xml is the obvious and most flexible model for representing oai-ore resource maps, but it is not the only one. another is the popular atom/xml serialisation [10], which is in widespread use in web 2.0 services. atom is supported by many browsers that are able to display an atom document in human readable form. atom is also currently supported by google as a means to update its search engine. table 1 links to an example of each type of serialisation, both of which represent the same publication taken from an eprints archive enabled with oai-ore functionality. rdf/xml serialisation atom/xml serialisation example in rdf/xml example in atom/xml table 1: example oai-ore resource map serialisations of the same publication. this section has covered the basics of oai-ore and outlined the parts which directly affect our work. for a more in depth explanation and additional feature specification we recommend the oai-ore home page [20]. it provides links to further serialisation specifications and the ore primer [13], much of which we have summarised here. more information on the rdf model can be found through the w3c’s rdf primer [16]. combining oai-ore with the repository in this section we demonstrate how oai-ore can be used to transfer resources from one repository software to another, as performed for the award-winning demo at open repositories 2008 [27]. the full demonstration required writing export plug-ins to reveal publication records as oai-ore resource maps and corresponding import plug-ins for each repository software platform. as expected, the export plug-ins were the easiest to create as all of the data exists already and the ir model is a good fit to the oai-ore specification. building the import plug-ins led to the discovery of an issue best solved in the oai-ore specification rather than on a per-repository basis. we discuss this issue and solution further in the “extensions to the oai-ore specification” section. by using eprints and fedora, both well established repository software platforms, as the basis for the development and trial of the oai-ore plug-ins, we had the advantage of two platforms that can be readily extended both in terms of import and export. our aim was to see if oai-ore could indeed enable the re-use and exchange of repository objects. the demo to demonstrate the power and ease of applying oai-ore to a real world scenario, two real world repositories were chosen as data sources. the example fedora repository was the oxford research archive [24]. the data from this archive would be harvested into an empty eprints repository configured with the import plug-in. the chosen eprints repository was that of the open repositories conference itself [22]. the harvesting would also be performed in the opposite direction, transferring objects from the eprints repository to an empty fedora repository. with the four repositories configured (2 originals used for export and 2 temporary ones used to import data from the opposite software), a small script was run on each destination repository pointing it at the resources to import, at which point the process was automatic. a video is available showing the demo and questions from the competition judging panel [17]). building the plug-ins for the purposes of completing the demonstration within the two days of competition we made a series of early decisions about the limitations of the oai-ore export and import plug-ins and a minimal acceptance test – the complete lossless transport of a full set of records from one repository software to another – was specified. the limitations concerning the rest of the implementation are listed below: rdf/xml resource map serialisation only. this not only speeds up the process of producing the resource maps that are exported, but more importantly it reduces the overhead of parsing the document when it is imported. rdf has a simple parser which was already available in both softwares. dublin core metadata is source of primary metadata. again, to reduce overhead and the amount of implementation needed we agreed that the dublin core metadata (conforming to the oai_dc specification [12]) would be sufficient when indexing the records sourced from another system. it was also assumed that both repository softwares had interpreted the meaning and definition of the dublin core terms in the same way.note: this does not imply that non-dublin core metadata was lost, simply that it was not indexed in the destination repository at this time. this could easily be achieved later by implementing or enabling a series of importers which can understand other types of metadata representations, such as fedora object xml (foxml) and eprints xml. the eprints export plug-in files and documents associated with each record/publication are listed using the built-in eprints model which allows us access to each “eprint” object and a list of the files which are part of that “eprint.” these files form the first part of the aggregation. to export the complete publication object we also need the history and provenance data which is revealed by the eprints export plug-in manager. the eprints export plug-in manager reveals all resources that can be exported through the eprints interface including the history, provenance and metadata to be included in the oai-ore resource map. listing all these resources along with the files in the aggregation guarantees the complete transport of each publication. finally the aggregation is encapsulated in an ad-hoc resource map which is generated on demand at a consistent uri. as the resource map of each eprint is generated by the export manager, the resource map may contain a link to itself. it is necessary to handle this possible recursion on import. the eprints import plug-in the eprints oai-ore import plug-in supports importing either a single xml resource map or a list of resource map locations (urls). the imposed limitation was that this resource map had to contain a single aggregation (as per the oai-ore specification) representing a publication. in turn this aggregation had to contain a dublin core record conforming to the oai_dc specification. to import a list of resource maps the plug-in iterates through the input list of urls, retrieving each one and then attempting to import it. each resource map is parsed for a component resource that /conformsto/ the dublin core (dc) metadata schema (see “extensions to the oai-ore specification” section for information about the dc:conformsto predicate). the built-in eprints dc import plug-in is used to convert the provided dc record to an internal eprints record. each aggregated resource in the resource map is then retrieved and added to the eprints record as a document. this includes the dc record xml as well as any other metadata resources provided such as foxml. at this stage in development in the eprints import plug-in, recursive importing was not supported. this would be desirable in future versions such that you could hand the plug-in a top level resource map which contains an aggregation of all the resources in the repository. at this point it would be necessary to determine the type of resource before importing it, as they might not all be publications and each type might require a different import workflow as a result. the eprints import and export plug-ins that were used in this challenge on an eprints 3.0 platform are available online [28]. it is intended that both of these plug-ins will come packaged with future versions of the eprints software. the fedora export plug-in due to the architectural design of fedora, it was easier to create an independent application to produce oai-ore resource maps of the objects held in the repository than to attempt to extend fedora’s code with an export plug-in. fedora’s basic object model fits the oai-ore notion of aggregation, so that a one-to-one mapping between aggregations and fedora objects is a semantically sensible assumption. the application code to create resource maps was added to the python-based web interface powering the oxford research archive repository [24]. the python library rdflib [26] provides the means to construct, hold and serialise rdf graphs, such as an oai-ore resource map. the service works by adding triples to an empty graph according to the oai-ore specifications, directly mapping the predicate ore:aggregates to each fedora object’s items, or ‘datastreams’ [7]. in fedora, a datastream refers to any object which can be represented by a file, including provenance and history data. additional triples were added to the graph to further refine the information held in the resource map for each datastream, such as indicating the mime type of the attachment and what standard the datastream might conform to (adopting the dcterms:hasformat property for this purpose, but initially using it in a technically incorrect manner). a built-in method of rdflib was then used to serialise the resource map into whichever syntax was required, rdf/xml, ntriple [9], n3 [2], or turtle [1] formats. the atom format serialisation was added after the open repositories 2008 conference [21]. the fedora import plug-in to enable importing oai-ore resource maps into fedora, a script was used which first gathered the ids of the items in the open repositories 08 eprints repository and then ran through the list, maintaining a log of progress. the rdf/xml resource map for each eprint object was downloaded and parsed by rdflib. the resulting graph was then queried to find out the urls of all the attachments and metadata files, and to find out which of these conformed most closely to simple dublin core. a fedora object was instantiated for each eprints object, and a process downloaded the associated files from eprints and added them as datastreams to the surrogate fedora object. the metadata file that conformed to simple dublin core as mentioned above was placed in a datastream with the id of ‘dc’, which is the convention for fedora objects. extensions to the oai-ore specification as has already been hinted above, we found it necessary early on to include a small amount of dublin core relating to the aggregated resources themselves within each aggregation. with each publication aggregation containing many metadata records of different types, it is impossible to determine which object conforms to which metadata specification without parsing them individually. this leads to a solution that is computationally more expensive than it needs to be. the key to solving this problem is to extend the oai-ore model by adding the dublin core namespace to the aggregation (it already exists in the resource map), such that the dc.conformsto and dc.hasformat predicates can be used to state explicitly which namespace or specification and document format each object in the aggregation corresponds to. this allows the import plug-ins to quickly determine which object in the aggregation represents dc metadata. figure 6 shows an abstracted version of a resource map for a single publication as used in the demonstration. this example resource is taken directly from the open repositories conference 2008 repository used as one of the demonstration repositories. figure 6: complete oai-ore example resource map and aggregation with dublin core extensions [view full-size image] eprints -> fedora for this demo the live publications archive from the or08 conference (in eprints) was used as the source archive. this was then exported into the fedora repository software in around 20 minutes. figure 7 shows screenshots of the source and destination repositories in each software and displays the subject tree. this was recognised by the fedora software with only minor changes to the software (telling fedora which field to group and index on). as can be seen, not only is the content transported, but also the functionality and usability of the objects. figure 7: screenshots of the or08 repository transport, eprints to fedora. [view full-size image] fedora -> eprints for this demonstration, the live publications archive from the oxford research archive (in fedora) was used as the source. due to the size of the ora it was decided to stop the import after about 35 minutes, which still resulted in the harvesting of a substantial number of records. again screenshots of both source and destination repositories are shown. figure 8 shows the same publication displayed by each repository.  we can see a much more compressed citation in the eprints repository and a greater number of files. these files represent the internal data from the source repository which only fedora understands and can interpret; the same issue exists when moving content from eprints to fedora. to solve this problem a further series of import plug-ins could be developed to interpret this data in a fashion suited to the particular repository software. figure 8: screenshots of the ora repository transport, fedora to eprints. more screenshots can be found here. [view full-size image] it should be noted that the branding of each destination repository was done by hand and is not part of the oai-ore solution. this part of the demonstration was intended to show that a repository need not be stuck forever with its first choice of software. rebranding is likely to be a key part of migrating to new repository software. oai-ore futures following the demonstration in april 2008, the final oai-ore specification [19] was released in november, and we hope that tools that use oai-ore to bind objects start to emerge in the near future. with such tools, oai-ore import support for the repository will become a mechanism for obtaining new resources, not just a tool to enable interoperability. this paper has shown how oai-ore can be used to express resources at a relatively high level within the repository such that they can be transported and re-used elsewhere. this might imply that oai-ore is simply a high-level tool that is used on demand and then discarded. we feel, however, that the oai-ore model can be applied at a much lower level, becoming the basis on which repository software operates. current computing practices relating to low-level storage consist of a set of files on disk which, in the majority of cases, are bound together using an old office paradigm known as the folder. files can only reside in a single folder unless they are copied. in other words, current filing systems contain no way to create custom collections of files without replicating the files themselves. oai-ore provides opportunities to bind low-level objects into multiple collections that can be used by higher level software while avoiding the need to make copies. the same object can be used in many aggregations that are related to many resource maps. oai-ore is likely to gain momentum as different storage technologies, such as cloud storage and open storage [29], are used more widely. these platforms provide a “drop box” type technology where objects are assigned a uid or uri at the point they are stored. the objects in these platforms are all classed as individual resources and no low-level binding exists. using uids and uris for the object identifiers fits perfectly with the oai-ore model, where the uris are required to apply oai-ore. if oai-ore finds use in binding objects below any repository software, then we can envision how this demonstration would be different. instead of using oai-ore to export and re-import the objects, we could simply change the software which sits on top of the objects. a little more work may also open the possibility of two repository softwares simultaneously managing the same set of low-level objects. at this point the definition of a repository as a single software solution starts to become abstracted into many layers involving basic storage and a set of services. this will also help from the point of view of repository services software being able to handle future technologies such as open storage with oai-ore bindings. conclusion the introduction of the oai protocol for metadata harvesting (oai-pmh) was the first step in enabling repository interoperability, but it was not a complete solution. the oai-ore protocol adds new functionality while it is a completely separate standard that ‘neither extends nor replaces’ oai-pmh. oai-pmh has as its basic tenet “a mechanism for harvesting records containing metadata” from one repository for reuse elsewhere [13]. this has so far been adequate and supported the openness of repositories. however, as the uses of repositories and the types of content they contain expand, more comprehensive methods for sharing content and more capability in terms of what is harvested and how it is reused are required. this is where oai-ore comes into the picture. this paper introduced the basic concepts of oai-ore and outlined how it can be used to express a single publication object or to build entire collections of objects. we showed how such representations can be serialised in both rdf/xml and the atom format. implementing export plug-ins to reveal publication records as oai-ore resource maps in both the eprints and fedora platforms enabled us to write the corresponding import plug-ins. this then allowed us to demonstrate the complete transportation of the content from one repository into the other. oai-ore provides another tool to help repository managers tackle the problem of long term preservation, providing a simple model and protocol for expressing objects so they can be exchanged and re-used. in future we hope to see oai-ore being used at the lowest level within a repository, the storage layer. binding objects in this manner would allow the construction of a layered repository where the core is the storage and binding and all other software and services sit on top of this layer. in this scenario, if a repository wanted to change its software, instead of using oai-ore to migrate the objects from one software to another, we could simply swap the software. references beckett, d. (2004). turtle – terse rdf triple language. http://www.dajobe.org/2004/01/turtle/ berners-lee, t. (1998). notation 3 – a readable language for data on the web. http://www.w3.org/designissues/notation3 berners-lee,t. (2006). linked data – design issues. (widely known as “the 4 rules of the web”)   http://www.w3.org/designissues/linkeddata.html carr, l., white, w., miles, s. and mortimer, b. (2008) institutional repository checklist for serving institutional management. in: third international conference on open repositories 2008, 1-4 april 2008, southampton, united kingdom. http://pubs.or08.ecs.soton.ac.uk/138/ common repository interface group (crig), set up to examine the boundaries between repositories and other systems. http://www.ukoln.ac.uk/repositories/digirep/index/crig dublin core metadata terms. http://dublincore.org/documents/dcmi-terms/index.shtml fedora tutorial #1. introduction to fedora. http://www.fedora.info/documentation/2.2.3/userdocs/tutorials/tutorial1.pdf fielding, r. et al. hypertext transfer protocol – http/1.1. rfc 2616, section 10, status code definitions. http://www.w3.org/protocols/rfc2616/rfc2616-sec10.html grant, j. and beckett, d. (2004). rdf test cases – ntriples. w3c recommendation. http://www.w3.org/tr/rdf-testcases/#ntriples gregorio, j., de hora, b. (2007). the atom publishing protocol. ietf rfc 5023. http://tools.ietf.org/html/rfc5023 jacobs, i. and walsh, n., architecture of the world wide web (volume one), w3c recommendation. http://www.w3.org/tr/webarch/ johnston, p., et al (2002). open archives initiative dublin core (oai dc). xml schema. http://www.openarchives.org/oai/2.0/oai_dc.xsd johnston, p., et al (2008). open archives initiative object re-use and exchange. ore user guide. primer. 11 july 2008. http://www.openarchives.org/ore/primer joint information systems committee – supporting eduacation and research. http://www.jisc.ac.uk lagoze, c., van der sompel, h., nelson, m.l., et al, (2008). object re-use and exchange: a resource-centric approach. http://arxiv.org/ftp/arxiv/papers/0804/0804.2273.pdf manola, f. and miller, e., et al, (2004). rdf primer. w3c recommendation. http://www.w3.org/tr/rdf-primer/ mining for ore, dave tarrant, ben o.steen and tim brody. video: http://blip.tv/file/866653 open archives initiative announces u.k. public meeting on april 4, 2008 for european release of object reuse and exchange specifications. http://www.openarchives.org/ore/documents/eukickoffpressrelease.pdf open repositories 2008: oai-ore european rollout meeting. http://or08.ecs.soton.ac.uk/ore.html open archives initiative object re-use and exchange, (2008). ore specification and user guide. http://www.openarchives.org/ore/toc open archives initiative protocol – object exchange and reuse. http://www.openarchives.org/ore/ open repositories 2008 (or08). http://or08.ecs.soton.ac.uk/ open repositories 2008 digital repository. http://pubs.or08.ecs.soton.ac.uk/ open repositories 2008: developer activities – the repository challenge. http://or08.ecs.soton.ac.uk/developers.html oxford university research archive (ora). http://ora.ouls.ox.ac.uk/ preserv: inspiration and provision of preservation services for digital repositories. http://www.preserv.org.uk/ rdflib. a python library for working with rdf. http://rdflib.net/ repochallenge. for details and videos of entries see: http://www.ukoln.ac.uk/repositories/digirep/index/crig_repository_challenge_at_or08 tarrant, d., brody, t. (2008) eprints ore resource map import and export plug-ins. http://files.eprints.org/353/ tarrant, d., o’steen, b., hitchcock, s., jefferies, n. and carr, l. (2008) applying open storage to institutional repositories. in: 2nd european workshop on the use of digital object repository systems in digital libraries (dorsdl2), september 18th 2008, aarhus, denmark. http://eprints.ecs.soton.ac.uk/16679/ about the authors david tarrant is a researcher in the school of electronics and computer science at the university of southampton. his main areas of research are web, web science and their relations to digital scholarly material. recent projects which he has worked on include preserv2, which focuses on digital preservation and eprints, the digital repository platform, of which he is a developer. email: dct05r@ecs.soton.ac.uk ben o’steen is an institutional repository software engineer at oxford university library services. email: benjamin.osteen@ouls.ox.ac.uk tim brody is at the university of southampton, where he develops the following repository services: celestial (oai harvester), citebase (citation-ranked search service covering arxiv and other subject repositories), the registry of open access repositories (roar), and pronom-roar, a joint initiative with the uk national archives and the jisc preserv project to add a format identification service to roar. tim was awarded a phd at southampton in 2006. email: tdb01r@ecs.soton.ac.uk steve hitchcock is project manager for the jisc preserv 2 project. he has a phd (2002) from southampton university, where he now works in the areas of digital publishing, digital preservation, open access and repositories. email: sh94r@ecs.soton.ac.uk neil jefferies is r&d manager at oxford university library services. email: neil.jefferies@ouls.ox.ac.uk les carr is a senior lecturer at the university of southampton, with expertise is in web-based information technologies. les is a fellow of the web science research initiative, course leader for a new msc in web technologies, and technical director of the eprints repository software team. email: lac@ecs.soton.ac.uk tags: oai-ore, repositories, standards subscribe to comments: for this article | for all articles one response to "using oai-ore to transform digital repositories into interoperable storage and services applications" please leave a response below: felipeducaa, 2009-09-22 enhancement of repository services has been furthered by the adoption of rdf together with the oai-ore specification and its manifestation by the preserv2 team’s activities that won them success in the or08 developer’s challenge. in the ore world, the focus moves away from metadata and on to content, especially when dealing with the complexities of digital objects, resulting in better services for users. in addition to its other benefits, implementing oai-ore provides one tool in the repository managers’ toolkit to help tackle the monumental problem of digital preservation, curation and continued access. the preserv2 model of preservation services provides one option for preservation activities to be undertaken in a way that is realistic and removes the prospect of a single gargantuan task to be undertaken at source. coupled with oai-ore, such services can be applied to repository content of mixed complexity where details such as provenance and relationships are retained. being able to invoke automated actions on digital objects in a way that is manageable and scaleable, and by enhancing object mobility so that more copies of those digital items can be stored in distributed locations, offers the increased likelihood that the objects will not only survive impresoras fiscales , but will remain accessible and by extension, usable. once these preservation services and actions are realised, the focus then moves to policy. leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – within limits: mass-digitization from scratch mission editorial committee process and structure code4lib issue 25, 2014-07-21 within limits: mass-digitization from scratch the provincial library of west-vlaanderen (belgium) is digitizing a large part of its iconographic collection. due to various (technical and financial) reasons no specialist software was used. fastscan is a set of vbs-scripts that was developed by the author using off-the-shelf software that was either included in ms windows (xp & 7) or already installed (imagemagick, irfanview, littlecms, exiv2). this scripting package has increased the digitization efforts immensely. the article will show what software was used, the problems that occurred and how they were scripted together. by pieter de praetere introduction fastscan is a set of visual basic scripts written to ease the digitization efforts of the provincial library tolhuis (province of west-vlaanderen, belgium). its main goal was, and to an extent still is, to increase scanning throughput, reduce manual intervention, and safeguard the quality of the resulting digital files. as a side effect, it was also hoped that it would increase morale of the scanning operators by allowing them to do more in less time, making the work less tedious. it was impossible to create an entire scanning application that could do both the scanning and the processing of the items, so we decided to use available applications wherever possible and use a scripting language to bolt them all together. as there is no budget, all applications needed to be freely available. furthermore, it policy forbids the installation of additional software, so they had to be already installed or usable without installation. the first version of the script was written in ms batch and lessened manual intervention and increased throughput by automatically numbering and scanning items, while facilitating cropping. this script, however, proved very unwieldy and difficult to maintain. to address that problem it was rewritten in visual basic script, a scripting language installed by default in microsoft windows computers, which offered additional features. afterwards, more features (such as auto-cropping, colour profiles, etc.) were added to the script, as well as support for more scanners and types of material. while generally successful, the script still has some shortcomings. this article, written by the author of fastscan, attempts to describe the application and why and how it was developed, as well as show some issues still remaining, with possible solutions. digitization process the provincial library “tolhuis” is a relatively small governmental library located in bruges. this institution houses three major collections; a welfare section, a local history section (by far the largest) and an iconographic collection, consisting of local (west-vlaanderen) items. the iconographic collection mostly dates from before the 1950s, with a large amount of items pre-dating wwi. the types of material in this collection are diverse, consisting of postcards, photographs, glass negatives, porcelain cards etc. while being fairly well preserved, it has not been catalogued or described. a couple of years ago, the decision was made to start digitizing the entire collection, both to reduce handling of the items in the interest of preservation and to increase public exposure of the collection. the digitized items were to be properly described and put into an online collection management system (memorix maior, by picturae)[1], with a public component at http://www.beeldbankwest-vlaanderen.be. at the same time, the items are cleaned (duplicates and items that should not be in the collection are removed), catalogued and numbered. so far (2014), only a small part of the entire collection (mostly postcards) has been treated in this manner, but the goal is to digitize the entire collection in this way. the initial project was a fairly low-tech effort, using consumer scanners (one canon canoscan 3200f and one hp photosmart 8200) and gimp software for scanning. every step in the process was manual. recently (2013), a new, more heritage-worthy scanner, a mikrotek scanmaker 9800xl, supporting more options, was added to the mix. however, while the other scanners are consumer devices, they support the basics of heritage scanning, e.g. optical ppi settings of 300 and above and exporting to tiff. before fastscan was created, about 10,000 images were digitized in various projects, though not all of them were cleaned, numbered or described. most of the items that have been digitized have been postcards. the rationale for this is relatively straightforward: a large component of this collection are items concerning the first world war. digitizing this part of the collection allowed the library to participate in world war one remembrance projects. nevertheless, other items (photographs) are routinely digitized, in much the same manner. digitization is carried out by normal staff in addition to their other duties. there is no full time digitization expert or scanner operator, and those who scan did not receive formal training in digitizing heritage materials. some items are being sent to specialized firms for digitization, mostly large items the scanners cannot accommodate. due to the addition of a new scanner last year, transparent source material (negatives, including glass negatives) are now digitized in house. the quality of the digitization is adequate, with sufficient ppi and colour depth. items are stored in tiff format on a central server, with viewing copies in jpg uploaded to the online collection management system. issues the initial project did not comply with international digitization standards [2]. it did however meet several of the criteria, such as a minimal sampling rate of 300 ppi, colour scan (notwithstanding the fact that the source material is mostly black and white), a bit depth of 8bits per channel and the use of tiff (uncompressed baseline tiff v6.0) as the storage format, but total compliance was not achieved. scanning equipment was uncalibrated, as were most computers. colour profiles were not used. while this produced (at first glance) no visible errors or mistakes in the resulting digital images, the potential for corruption of the colour information in the images is quite large. another major issue was the speed at which digitization proceeded. all work was done manually, using image editing software, mostly gimp or the software that came with the scanner (proprietary scanning software). any alterations (especially cropping and rotating) were done afterwards with gimp. to store the image, the operator had to copy the number written on the item into the “save as”-box, taking care not to confuse numbers or to miss any of the zeroes with which the base number was padded. apart from being slow, this also caused repeated errors and mistakes in the file names, which had to be rectified afterwards. the slow digitization speed and the large amount of errors resulted in a huge backlog, with only a limited amount of material digitized. furthermore, staff members executing the digitization became demotivated due to the large amount of work involved, resulting in a fairly small amount of items (20 – 25) digitized each day. the looming commemoration of the first world war contributed to the pressure, as the goal was to digitize the wwi postcard collection before the start of the remembrance year (summer 2014). priorities for improvement before improvements could be made, priorities had to be established. based on “best practices” [3], compliance with international standards should have been the preferred way to go forward. however, the library chose to make an increase in scanning speed the number one priority. this choice was based on several arguments. firstly, while we did not comply with the complete package, some minimal requirements (sample rate, file format, bit depth) were met additionally, there had been no reports of errors due to the lack of calibration or not using colour profiles. this made adherence to the entire standard a much less important issue although compliance certainly is a goal, especially since one of the goals of digitizing the collection is to present the digital item as a surrogate of the physical one wherever possible. it is clear that adherence to international standards is a must for this type of digitization projects. but on the whole, the situation was not as dire as we first thought. improvements in scanning speed were more attractive for a number of reasons. going faster, with automation, would reduce the time staff had to spend on tedious manual tasks, improving the morale and limiting the possibility of human error. an increase in speed would also allow us to scan our entire wwi postcard collection before the summer of 2014, so we could jump on the remembrance train. if we were going to automate the process, it also seemed superfluous to put a lot of effort into calibrating systems and forcing the use of colour profiles while still using the original process, as we would probably be forced to do it all over again when the new system was installed. with a solid automated framework in place, adding the elements that would increase compliance was expected to be easier to install. for those reasons, we chose to address the scanning throughput first and compliance second. possible solutions existing alternatives were evaluated before choosing to create our own system. however, several constraints severely limited the available options. firstly, due to policy reasons it could not require (complicated) installation procedures. the computers to which the scanners are attached are used by different users, and installing and configuring the software for all of them would be prohibitively time-consuming. furthermore, installation could not require administrative privileges, as those would require another layer of bureaucratic involvement. the variety in systems, both operating systems (windows xp and windows 7) and scanners is something that had to be taken into account as well. further constraints were added by the fact that there is no budget for these kinds of applications, so prospective software would have to be free (gratis). however, the most important requirements for the system were (and are) ease of use and ease of customisation. most of the staff operating the system are not “digital natives” and shouldn’t need to be. the system thus had to be easily usable by people that are less technically inclined. the ease of customisation was equally important. no two digitization projects are the same. while most of the basics (e.g., storage file format, optical resolution) will indeed be shared by different projects, other elements (file name structure, folder structure, scanning systems) will usually not be the same; the system had to allow for this level of customisation. to summarise, a prospective system needed to be easily configurable (and preferably not need administrator privileges to install), allow for a diversity in systems and scanners, be easy to use and be easy to customise to the situation of the provincial library. finding a system answering to all those needs is extremely difficult. some projects advocate the use of adobe photoshop [4], but there are some major problems with the use of photoshop. photoshop supports twain technology [5], but has no support for batch scanning. another issue with adobe was the cost; as there was no ,for extra applications. gimp, a free (gpl) image editor with features comparable to those of photoshop, had the same issues as the adobe application. like photoshop, it allows for twain-based scanning, but not for batch image scanning. research also indicated that scripting gimp to allow for batch scanning was also not feasible. another, more basic image editor, irfanview, is pre-installed on all systems in use in the library. this program allows for scanning using twain and has a batch scan mode [6](skiljan, 2010). this batch scan mode is very difficult to use together with a script (one that would, for example, take care of giving the correct name to scans), so in effect only the single-scan mode could be used. also, the gui version was not considered adequate as an image editor, especially when compared to gimp or photoshop. while other, more comprehensive, options were certainly available, budget and time constraints forced the use of several free or pre installed applications that would be tied together using a set of scripts, thereby working around several limitations in the original software. this is not an ideal option, as it requires knowledge of a scripting language and takes a lot of time, but it would result in a script that is created with all of the requirements of the library in mind, something that would be very hard to achieve with off-the-shelf software. developing internally was also a requirement, as there was no budget for outside contractors. fastscan fastscan has three major components; irfanview, imagemagick, and a catalogue number generator. only the last part is home grown; the first two are off-the-shelf systems. irfanview in command line mode is used to acquire images from the scanner using a twain interface. twain is a api that forms a generic interface between scanner drivers and applications [7](twain working group, 2014). while irfanview can be used as an interface between twain and another application (such as this script), it is not meant to do this. it would probably be cheaper to use an application that only functions as a scriptable twain program and not a complete, if basic, image editor. using irfanview has its advantages however. it’s installed on every system in the library, so it does not require intervention. furthermore, due to its wide installation base, users can be expected to be familiar with the system, which is important when things go wrong. irfanview also fulfills all our needs (i.e., a scriptable twain system), so there is no pressing need to change it. in short, while not perfect (also not being free software), irfanview is more than adequate to be used in the system. irfanview is called in batch mode with the /scanhidden, /dpi=(300,300) and /convert=filename options. the /convert option is used to store a scan of the entire surface of the scanner as an uncompressed tiff file. this is considered the “raw” file, which is then later cropped to the size of the object. the /dpi=(300,300) options set the ppi of the scan. this value is for now hard coded, but in a future version this will be specific to the type of object being scanned, so it can be set higher for smaller objects. /scanhidden hides the twain gui, according to the irfanview manual [8](skiljan, 2014), which is essentially the gui of the software bundled with the scanner. this is hidden for two reasons. firstly, it opens up the potential for user-induced errors, as options can be changed, thus potentially corrupting the scan (lowering ppi, only scanning parts of the image etc.). we had to configure every driver before using fastscan, to set the scanned area and ppi to its maximum, because otherwise irfanview would scan only what the bundled software allowed (e.g. only half of the surface at 150ppi). the second reason to hide this is to lessen confusion. fastscan is a cli app with no gui elements. to have this one pop up suddenly, requiring a manual click of the “scan” button, would introduce a unnecessary step. in order to lessen the potential for errors and to decrease the amount of steps, we choose to hide this dialog. the second component is imagemagick, or im for short. im is a free image editing and viewing suite, released under the apache 2.0 licence [9]. im is used in two ways in fastscan; as an image cropper and as an extractor for metadata. the first function is by far the most important, as it removes the need for manual intervention while scanning. irfanview creates an image of the entire surface of the scanner, after which imagemagick cuts the object out. this allows for “blind” scanning, i.e. without intervention from the operator, as long as the background colour contrasts enough with the colour of the object (e.g. a black background with a light scan or vice versa). to do this, im guesses (using blurring techniques and fuzziness) the contours of the object and outputs its location. the fuzz-factor is the most important here, as this determines how im defines item and background. a large fuzz-factor means a lot of the detail is blurred (and thus ignored), potentially causing parts of the object to be considered background. a small fuzz-factor on the contrary, takes more detail into account when defining an image, thus resulting in a larger share of the background being considered part of the object. with a high-contrast background (e.g. black on white), a factor of 15% was found to be adequate and produce only a small amount of images that are too large or too small. another call to imagemagick then crops the entire scan down to the object and stores this as a new file. we keep the original full surface scan around till the end of the program in case something goes wrong. note that because this is an xp system the name of the imagemagick convert.exe program has been changed to im_convert.exe so as not to inadvertedly call the windows application convert.exe [10]. the error rate for this component is thus fairly low, although it could potentially generate a lot of errors, especially when the contrast between the background and the item is not very large. to date however, only a small amount of images has been rejected because im cropped too much. this is mostly due to the fact that the items the library is scanning now have a white or black border around them (postcards, photographs, in memoriam cards), making it easy to select a high contrast background (fully white or black). the fuzziness factor used is quite conservative as well, which resulted in a larger (but still fairly small) group of scans that had to be re-cropped because im cut out an area too large, mostly due to dust or scratches on the scanner surface. this is considered to be less of a problem since it can be rectified easily. using im to crop the image has been a huge time-saver. before the implementation of im, a lot of the time spent on scanning was devoted to cropping the images. now this can be done as part of the process with an error margin in some cases lower than when done by hand. it did require some adaptations from the operators, such as using (and switching) a high-contrast background (in the library we used sheets of thick black paper slightly larger than the surface of the scanner with great success) and making sure the object was straight when laid on the scanner surface the first time around (in order to save time later and lessen the probability of im errors). previously, they could adapt and change the rotation of the image while scanning, as the resulting image appeared on their screen before saving. because implementing im decreased the scanning time, these small adaptations were not seen as a problem by the staff. the third component is a custom catalogue number generator. every object has a number, consisting of two elements. the first element is a prefix, denoting the type of material and consisting of three letters. the second element is a six-character number, with zero-padding. if both sides of an object (front and back) are to be digitized, a letter is added to the number. the letter a denotes the front side, the letter b the back side. the implementation of these catalogue rules created some difficulties in the program. the first problem is the generation of the six-character number. in normal cases, items are digitized in order (e.g. 2 follows 1 which is followed by 3, 4, 5 etc.), so it is possible and even recommended (in order to save time) that the system automatically determines the catalogue number. this is, however, not as easy as it sounds. during the lifetime of the application, the number is stored in memory and auto-incremented. however, this would mean that every time the script is started, the user has to enter the number of the first item. this was considered not very user-friendly, so the system stores this value in a text file. because we do this, it is vital that the application writes the last issued number to this file after every scan. this also prevents faults in the numbering when the application unexpectedly quits after a bug, as long as the fault occurs after scanning. this text file is tied to the user and the computer executing the script, and not centrally stored (e.g. on the shared drive). this was done for two reasons. firstly, this allows more than one user using the program at the same time (on different computers), without the need for complicated locking procedures. secondly, if more than one user is using fastscan, they can continue scanning from where they left without readjusting the number every time, as it only remembers “their” number. nevertheless, this component is not without faults, so the user is prompted before every scan whether the generated number matches the one on the item and to change it accordingly. at present there is no automated manner to prevent duplication between users. however, this is not a major problem, as all objects are numbered before they are scanned. the only thing operators have to take care of is indicating on the items whether they have scanned them or not. in most cases, they simply take a batch of objects out of their storage and return them when finished, taking care to indicate that they have been scanned. the second problem is the prefix generation. having a prefix denoting the type of material is less than ideal, as it invariably leads to drawn-out discussions and confusion about which item belongs in which category. nevertheless, the system is used, so the script must know how to work with it. this proved to be quite easy, but the easy solution had some caveats. at first, we created a vbscript dictionary with all the types, and based on cli input (cscript fastscan-3.0.vbs type_of_material) selected the right prefix and continued. this approach has one huge drawback; to add a new type, someone has to code it in. this is not a tenable solution, as there are currently no other programmers in the library. to rectify this, the dictionary was moved to an xml file, which contained not only the prefix, but also the length of the number after the prefix (zero-padding is still the default, no matter the length) and the increment (default is 1). together with some redefined code, this makes it easy to add material types to the application. <list> <material name="postkaart"> <name>postkaart</name> <key> <prefix>pkt</prefix> <length>6</length> <step>1</step> </key> </material> </list> ' prefix determination prefix = mxml.selectsinglenode ("/list/material[@name='" & wscript.arguments(0) & "']/key/prefix").text ' key length k_length = mxml.selectsinglenode ("/list/material[@name='" & wscript.arguments(0) & "']/key/length").text ' key step k_step = mxml.selectsinglenode ("/list/material[@name='" & wscript.arguments(0) & "']/key/step").text tying of the cli option type_of_material to the correct entry in the xml file is done with xpath. this proved to be easier than expected and has thus far not resulted in major issues. while using an xml file solves some issues, others remain unsolved. some types have customisations in the code (e.g., different scanning or cropping parameters). adding these to the script still requires some programming. at the moment, there are no plans to move those to a configuration file as well, as this would entail a lot of extra work and overhead for relatively small gains. the third (and final) problem faced is the scanning of both sides of an object. this does not happen all the time (roughly 20% of objects have an interesting enough backside to be selected for scanning), but still must be supported. the established procedure is to add a or b to the inventory number to differentiate between the front and the back of an object. this is supported by the application, but only after manual user intervention (there is no way it can guess whether or not the item will have a back side), except for certain types of material (this is one of the customisations hard coded in the application). supporting this required some modification to the flow of the application. normally, every time the application loop restarts (the loop continues to run until the user quits), the inventory number is incremented by k_step. if the user answers yes to the prompt “does this item have a backside?”, this incrementing should not happen the next time the loop runs, otherwise the inventory number would be wrong (front and backside of the item have the same inventory number). to prevent this, after answering yes the flag backside is set to 1. the next time the loop runs, backside will evaluate as 1, so the system knows not to increment the inventory number. if backside <> 1 then ' not a backside ' reset brun brun = 0 ' new number number = last + cint (k_step) ... else ' is a backside brun = 1 ' so we know when to reset backside number = last end if ... if backside = 1 and brun = 0 then ' a side filename = prefix & pad (number, k_length) & "a.tif" elseif backside = 1 and brun = 1 then ' b side filename = prefix & pad (number, k_length) & "b.tif" else ' normal case filename = prefix & pad (number, k_length) & ".tif" end if ... ' reset counters if brun = 1 then backside = 0 end if in order to determine whether the letter should be a or b and when to reset backside to 0, another flag is used; brun. this one is set to 0 when the question “does this item have a backside?” is answered by yes. this signals the application that this is the first run, so the letter a must be used. the next loop, when backside = 1 evaluates to true, brun is set to 1. this means we are now in the second run, so the letter must be b. furthermore, at the end of the application, the flag brun is checked again. if it equals 1, backside is reset to 0, because now both sides have been scanned. afterwards, the application continues normally and will again prompt the user whether this item has a backside or not (this test is skipped for obvious reasons when backside equals 1). while seemingly complicated, this system has not caused any problems. the components are tied together with visual basic script, a scripting language created by microsoft and installed by default on windows xp and windows 7 systems, which are the two operating systems used at the library. an earlier version used the microsoft batch file commands, but this was very unwieldy and hard to maintain, so the entire script was rewritten in vbscript. all things considered, the language is adequate, in some areas even excellent, especially its xml bindings. given the choice, more features could probably be added easier using python or perl, such as the ability to use the twain api from within the script, rather than using some other software [11], but due to the ban on installing software, this is not possible. vbscript is also not very difficult to learn (if harder to master), so other people (with a small amount of training) could take over maintenance, which is important with regard to the future of the system within the library. fastscan has been designed from the ground up to reduce the workload for the operators. most steps are automatic and for those that aren’t a wizard-like interface is used. in ideal cases the operator starts the application and answers the first set of questions to determine the number of the first object. the item is next placed on the scanner and is scanned, cropped and saved. no intervention is necessary. after this step, the cycle repeats. when the application has been used before, or a new cycle is started, the number of the first item (and all following numbers) are determined automatically by incrementing the previous number with 1. when the situation is not ideal, e.g. when the number is wrong, the number can be corrected using the wizard. in short, user intervention is limited to pressing the enter key a few times when everything goes according to plan, and to entering some numbers when there is a problem. added value fastscan as described above is more than adequate to digitize heritage material. however, it still lacks elements that would allow for more compliance with established heritage standards. the first element is the support of embedded colour profiles. colour profiles (icc) are the preferred way of sharing colour information between systems. in the base system, this is not supported. supporting it would entail calibrating scanning equipment, installing colour profiles and finding a way to embed them in the images. as this was not possible with the described tools, new ones had to be found. the second element is metadata support. metadata is here to be understood as technical metadata, not descriptive. the tiff file format supports embedding metadata, thus creating a durable link between data and metadata. as with colour profiles, embedding required the use of extra software, although im could be used to extract most of it. both elements were researched and provisionally implemented, but only the module that adds colour profiles is currently used. adding metadata is possible, but slows down scanning immensely. i will only briefly describe how we added colour profiles; the script for adding metadata will not be described, but is available at github . adding an icc profile is supported in irfanview with the use of plug-ins. however, due to the set-up at the library, it is impossible to install those plug-ins, so we had to find another solution. we stumbled upon the free (gpl) colour management system called argyll cms. it is attractive to us for two reasons. firstly, it can be used to calibrate scanners (note that a calibration target is still required) and create a valid icc colour profile. secondly, it contains software (tifficc) that allows embedding of colour profiles in tiff files. furthermore, this software is easily scriptable, so it could be integrated in fastscan. to use argyll within fastscan, we first had to create icc colour profiles from the scanners and store them in a central location. using an xml file which contained the name of the scanner and the computer it was attached to (to customise certain options of twain), we linked them together and created a script that called tifficc to embed this profile in the image. the resulting images are stored in the edited images sub folder of fastscan. embedding is done after cropping, to prevent the embedding from being stripped by imagemagick. while supporting colour profiles had no immediate impact on the work flow or the (perceivable) quality of the scans, it does mean increased compliance with international standards and safeguards the quality of the items. fastscan issues fastscan as an application is not without issues. three of the most important problems are the dependence on external software and not software libraries; the lack of portability due to the hard coding of some variables and options; and the possible cessation of maintenance and support when the contract of the main author ends. two of those issues are related to the way fastscan works, while the third is a policy problem, for which other solutions may be needed. depending on external software is a necessity because of the programming language. while some libraries for vbscript exist, they are internal and there is no library repository for vbscript such as, for example, there is in cpan for perl. external dependence is also necessary because of the ban on installing software, which forces the use of either pre-installed applications or programs for which no installation is required. fastscan makes liberal use of both of these types. to reduce this dependence, a major rewrite in a different programming language, either a scripting language such as perl or python, or a compiled high-level programming language like c++, seems to be the only solution. this is however not a likely solution for two reasons. firstly, this would take huge amounts of time for (in some cases) only marginal benefits. secondly, while it would be easier to do a perl or python rewrite, this would entail the installation of interpreters on all systems where fastscan is used. it is unlikely that this would be permitted within the library and furthermore, it would add a large amount of extra work, i.e. installing the interpreters on every system. nevertheless, for future use, a rewrite (in a scripting language) remains a distinct possibility. the second problem is the reliance on hard coded variables. this is quite an old problem; in the first versions almost everything was hard coded. in version 3.0, reliance on these variables has been reduced and as much as possible has been moved to either a configuration file (for paths to external applications) or an xml file. this xml file contains “local” information specific to the institution and to the systems where fastscan is used. in the file scanners.xml the name, type, colour profile and computer to which a scanner is attached are stored. this information is used by fastscan to set certain options and embed the right profile. in the second file, material.xml, options for the scanned item types is set, such as prefix, key length and number increment. while this is already an improvement over the previous situation, some extra settings should be stored in those files, such as the last used number for every item and the cropping settings (especially the fuzz factor). doing so would make it easier to add new types without changing the source code, which is important with regard to future maintainability. another consequence of the reliance on hard coded variables and, more generally, the way in which the script is written especially for the library, is the difficulty of porting it to other environments. it is certainly possible, in theory, but in practice a lot of the logic will have to be changed. the settings for the external applications can probably remain the same, but the number generator, including the front side/backside switch, will have to be rewritten to adapt to the new environment. this is possible, as the source code is freely available, but certainly not easy. it might make more sense for institutions to simply create their own script, with their own quirks and settings. other parts of the script present less problems, although reliance on external applications requires the installation of them, which is usually not a problem; but certain components (irfanview) are not free software, so it is not guaranteed to always remain available. to sum up, while porting is possible, it will usually be easier to write your own script, perhaps based on the source code for fastscan. the third issue is the uncertain status of future support and maintenance. while the current author is employed by the library, this contract will soon end and at the moment there is no replacement. using the application does not require specialist knowledge, nor does changing configuration values. solving bugs and adding new options does, however, require knowledge of vbscript and of the options and settings of the various external software. some of these concerns are rectified by the fact that fastscan is open source (gpl 3), so the source code is available for all to see. this might result in another maintainer taking over, or the original author developing further, but this is by no means certain. furthermore, technical documentation of the software is still lacking (a user manual is available, a developer manual is not), although work on this is ongoing. it still remains possible to scan without fastscan, but this would mean returning to the previous manual system. while not impossible; it is certainly would not be an ideal situation. concluding remarks all things considered, fastscan has been a success. basically, the script does nothing more than tie together several open source (or freeware) components to support the workflow in the library. it is not inconceivable that other, more expensive and specialised, scanning software would be capable of doing the same, but due to the limited funds available and the difficulties with installing software, those cannot be easily implemented library-wide. fastscan is thus a “compromise solution”, but one that is tailored to the library, supporting the entire work flow and all its quirks. such a level of customisation would be very difficult to achieve with off-the-shelf software. this is both its major advantage and its major weakness. as long as everything stays the same, fastscan is a great piece of software. when things change (as they inevitably will), it will require someone with vbscript knowledge and knowledge of the applications used to adapt the script. there is a major push/rewrite going on to address some of these issues by relying on configuration files. nevertheless, it is not impossible that in the future fastscan will no longer be used because it cannot be adapted. to assist future maintainers and to prevent a possible “black hole”, the script is released under the gpl v. 3.0 licence and the source code (of the script, not the additional software) is available on github: https://github.com/pieterdp/sobki/tree/master/fastscan. while it is being used, its advantages are clear. items are scanned faster and, arguably, better than before. the most popular feature was the auto-number generator, as this is the most labour intensive and boring part of the job. auto-cropping and auto-scanning further reduce the workload, increasing throughput. some other additions, such as metadata and colour profiles, do not increase the scanning rate, but the quality of the scans. this had not been the original goal, but that does not diminish its importance. with future reuse and durable storage in mind, compliance to international standards is a must. together with the increased throughput, this makes fastscan an important part of the digitization process at the library. references [1] picturae. 2014. “memorix maior. collectiebeheer. picturae’s webbased beheertool.” https://picturae.com/nl/brochures/380-memorix. (back) [2] van dormolen, hans. 2008. “metamorfoze preservation imaging guidelines.” /archiving conference/2008 (1): 162–65. (back) [3] packed. 2013. “over cest – cest – cultureel erfgoed standaarden toolbox | packed vzw expertisecentrum digitaal erfgoed.” http://www.projectcest.be/index.php/over_cest#.c2.a0.c2.a0.c2.a0digitaliseren. (back) [4] umass amherst libraries. 2011. “guidelines for digitization”. amherst: umass amherst libraries. http://www.library.umass.edu/assets/aboutus/attachments/umass-amherst-libraries-best-practice-guidelines-for-digitization-20110523-templated.pdf. (back) [5] adobe. 2014. “twain plug-in | photoshop cs4, cs5, cs6.” accessed may 4. http://helpx.adobe.com/photoshop/kb/twain-plug-photoshop-cs4-cs5.html. (back) [6] skiljan, irfan. 2010. “command line options for irfanview.” december 27. http://www.robvanderwoude.com/files/iviewcli.txt. (back) [7]twain working group. 2014. “twain.” accessed may 6. http://www.twain.org/. (back) [8]skiljan, irfan. 2014. “command line options for irfanview.” accessed may 6. http://www.robvanderwoude.com/files/iviewcli.txt. (back) [9]imagemagick studio. 2014. “imagemagick: convert, edit, or compose bitmap images.” accessed may 6. http://www.imagemagick.org/. (back) [10] hugemann, wolfgang. 2012. “usage under windows — im v6 examples.” march 21. http://www.imagemagick.org/usage/windows/#convert_issue. (back) [11] ouwerkerk, lennert. 2002. “win32::scanner::eztwain – search.cpan.org.” http://search.cpan.org/~lennerto/win32-scanner-eztwain-0.01/eztwain.pm. (back) about the author pieter de praetere is a employee of the provincial library “tolhuis” (west-vlaanderen), where his main occupation is the maintenance and technical support of the provincial image library (http://www.beeldbankwest-vlaanderen.be). apart from this, he is also responsible for the management of the digitization project. he holds a master’s degree in archival sciences and one in history, while also having a great interest in programming and scripting. e-mail: pieter.de.praetere@helptux.be url: http://erfgoeddb.helptux.be subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – visualizing fedora-managed tei and mei documents within islandora mission editorial committee process and structure code4lib issue 44, 2019-05-06 visualizing fedora-managed tei and mei documents within islandora the early modern songscapes (ems) project [1] represents a development partnership between the university of toronto scarborough’s digital scholarship unit (dsu), the university of maryland, and the university of south carolina. developers, librarians and faculty from both institutions have collaborated on an intermedia online platform designed to support the scholarly investigation of early modern english song. the first iteration of the platform, launched at the early modern songscapes conference, held february 8-9, 2019 at the university of toronto’s centre for reformation and renaissance studies, serves fedora-held text encoding initiative (tei) and music encoding initiative (mei) documents through a javascript viewer capable of being embedded within the islandora digital asset management framework. the viewer presents versions of a song’s musical notation and textual underlay followed by the entire song text. this article reviews the status of this technology, and the process of developing an xml framework for tei and mei editions that would serve the requirements of all stakeholder technologies. beyond the applicability of this technology in other digital scholarship contexts, the approach may serve others seeking methods for integrating technologies into islandora or working across institutional development environments. by raffaele viglianti, marcus emmanuel barnes, natkeeran ledchumykanthan, kirsta stapelfeldt introduction early modern songscapes (ems) is an interdisciplinary web project co-developed by the university of toronto scarborough’s digital scholarship unit (dsu), the university of maryland (including the maryland institute for technology in the humanities), and the university of south carolina. ems, “aims to develop an intermedia online platform to support scholarly investigations early modern english song,” by “tracing individual lyrics and musical settings that moved through different textual and performance contexts in sixteenthand seventeenth-century england.” ultimately, the group has goals to build a community and resource for researchers and performers interested in sixteenth and seventeenth century english song. the researchers driving the project are katie larson, from the utsc english department, scott trudell from the university of maryland and sarah williams from the university of south carolina. the first iteration of the project focused on henry lawes’s 1653 ayres and dialogues, highlighting versions of the songs that have been published, co-located with image and sound materials relating to each individual song. [2] ultimately, henry lawes’s work provided a reasonable starting point for study by virtue of scope, availability, and significance of his work. however, the hope is that the corpus will grow (and the openness of the project to collaboration is reflected in the absence of a strict scope for content). at the utsc digital scholarship unit (dsu), developers and librarians collaborate on the creation of a shared software platforms on which projects like ems can be developed. uniquely among the dsu’s past projects, ems incorporated off-site development of a viewer that would display tei/mei delivered from the library’s islandora-based infrastructure. project infrastructure islandora is an open-source software framework that combines fedora’s repository architecture with the flexibility of drupal and apache solr. islandora developers provide drupal modules that, when installed with their dependencies, allow users to manage, create, and access fedora repository items (digital objects comprised of files, or datastreams) in a drupal interface. core indexing functions are provided through islandora-specific modules that provide custom integration with apache solr. the software is often used to support digital collections and preservation, with “utility” modules providing enrichment functions such as annotation. as fedora repository software stores any form of xml inside repository objects, this software has been used in several tei projects and other applications have been built to integrate the tei into islandora (stapelfeldt & moses, 2013). the wysiwyg tei editor “cwrc writer” [3]< has been integrated into islandora through the the canadian writing research collaboratory [4] and projects like the “islandora critical edition solution pack” [5] through the editing modernism in canada project [6] incorporate tei features. the early modern songscapes (ems) project made substantial use of existing utility modules in islandora for storage and display. the islandora xml solution pack, which makes possible the storage of xml files, enabled the project to store tei and mei assets produced by project researchers. [7] the project also used a module called islandora context [8] which itself extends the functionality of the drupal context [9] module. the context module is very useful in this instance as it permits for the execution of code given particular user or environmental conditions. in the case of ems, the “context” is when an xml object is viewed in the repository, and the executed code is a javascript-based “edition viewer,” authored by raffaele viglianti at the maryland institute for technology in the humanities at the university of maryland. digital scholarship unit developer natkeeran ledchumykanthan the provided an additional ems integration module (dsu, 2019) [10]. the ems integration module includes a context for use with the islandora context module, providing a way for the edition viewer javascript bundle to be loaded only when viewing xml objects. xml data structures & academic requirements of the project the text encoding initiative (tei) [11] is an extensible, xml-based markup language for the representation of texts. the music encoding initiative (mei) [12] mirrors this work to create markup language for music, but does not have a formal relationship to tei. these two xml formats both apply descriptive markup principles to text and musical text encoding, which privilege semantic descriptions (i.e. what some text is) over procedural ones (i.e. how some text looks). both tei and mei were required for this project, as the lawes’s book that formed the initial corpus for the ems project is at the nexus of literature and music and pays careful attention to both the literary and musical aspects of the songs. in the digital editions produced by the project, the tei is used for verse and the mei format for notated music. this approach renders both editorial intervention and annotation machine readable. making this information explicit allows for the preservation of editorial scholarship in the fedora repository as well as the development of an interactive publication. combining tei and mei despite the lack of formal relationship between the two initiatives, schemas in tei and mei can be combined, and both formats provide elements that make the connection meaningful; namely the <notatedmusic> element in tei and a number of elements in mei that contain textual data, such as on-staff directives <dir>, poem divisions <verse>, or the more generic <anchoredtext>. combining these two formats is becoming a fairly established practice, see for example the thesaurus musicarum latinarum [13] (menegozzi 2011: 841-842), but is not without challenges as existing tools and workflows are usually focused on either tei or mei. the hierarchical nature of these formats also requires one of the two to contain the other or, in other words, take a primary position. in ems, the data model priorities tei, in part because tei has a well established metadata header in which bibliographical information is stored and this was determined to be desirable. moreover, the single entry point provided by the tei element <notatedmusic> elements makes it possible to embed separate mei documents within the tei using xinclude, [14] a w3c recommendation that simplifies the inclusion of xml from multiple sources. see figure 1 for a schematic representation of the encoding data model for a song, with tei including mei data. the song shown is “when on the altar of my hand.” facsimile from early english books online (first published here). figure 1. a schematic representation of the encoding data model for the song when on the altar of my hand, a facsimile from early english books online, with tei/mei data. the lyrics underlaid under the music notation are encoded both in the mei as syllables under notes and again as a tei-encoded verse. this duplication is intentional: subsequent verses, when present, are added after the music notation and follow editorial patterns of early modern printing practice, which is best encoded with tei. more information is available about the editorial decisions made by the team. [15] re-encoding the first verse (typically laid under the notation) allows project editors to represent their interpretation of how it may appear if separately formatted. because early modern songs are often studied as poetry (i.e. removed, for better or worse, from their musical context), this repetition of the first verse may become useful if the text were to be presented without the music. collating versions of text & music a number of songs in the collection have multiple versions coming from different sources, such as manuscripts or other contemporary printed editions, which can be with or without music notation. for these cases, the ems scholars produced a variorum edition that is a presentation of the text that showcases differences across the sources without privileging one over the other. both tei and mei are well equipped formats for modeling textual variance, but both assume that one text will be the main reading text (often called the ‘base text’) and only variant text will be encoded from other sources. [16] to overcome this apparent limitation, the project presents separate combined tei and mei documents that represent a collation – a document that lists where the differences between the sources of one song are to be located. in practice, this document encodes a set of pointers grouped by variant; each pointer refers to one or more element in the source xml documents by simple id references. <app> <rdggrp> <rdg wit="#l638_1"> <ptr target="tei-l638_1.xml#v4"/> </rdg> <rdg wit="#c709"> <ptr target="tei-c709.xml#v4"/> </rdg> </rdggrp> <rdg wit="#bl_53723"> <ptr target="tei-bl_53723.xml#v4"/> </rdg> </app> listing 1. a tei apparatus entry with pointers to three source documents the example above shows a tei apparatus entry with pointers to three source documents identified by a witness code (e.g. #l638_1). each pointer resolves to text that the editors consider being a variant of each other. note that #l638_1 and #c709 are grouped by the <rdggrp> element, which indicates that they “agree” or carry the same text (they resolve to the word “deserted”); #bl_53723, on the other hand, carries a different readings of the text (it resolves to the text “forsaken”). with this approach, the editors are able to encode, and preserve in fedora, each source separately to the degree of detail that we deem appropriate without worrying about tessellating multiple sources in one place. the applicability of this approach goes beyond early modern song and has been previously applied to other texts by viglianti (2016) and is currently being improved and documented thanks to ems and the ongoing frankenstein variorum edition (beshero-bondar and viglianti 2018). ems however, is the first project to extend this model to music encoding. [17] figure 2. diagram demonstrating how the collation document points to variants. edition viewer while the xml is stored in the fedora datastore, the display and interactivity are handled entirely in the front end. a javascript react/redux [18] application loads the collation file for a specific song and determines which other source files to request based on the pointers contained therein. only one source (such as a specific manuscript or print) is shown at a time, with highlights indicating where the sources diverge. clicking on these highlights will resolve relevant pointers to the other sources and show the different readings. the ‘base’ source can be switched at any time. see figure 3 for an example of the viewer rendering the opening text and musical notation with an expanded musical variant. figure 3. a screenshot of the viewer showing a rendering of the opening text and musical notation of “theseus, o theseus, hark! but yet in vain,” with an expanded musical variant in the bass line of measure 4. tei projects typically perform a lossy transformation into html for display, but in this case we are using ceteicean (cayless and viglianti 2018), a javascript library that registers tei element as html custom elements, thus requiring minimal or no changes to the tei source for display in the browser. for rendering mei we use verovio (pugin 2016) a library that renders mei into svg for display. both of these tools offer a solution for rendering these data directly in a user’s browser, thus reducing the need for server-side infrastructure for tei and mei publications. they also provide isomorphic (that is one-to-one) renderings of the data, which allows to manipulate the rendering as if it were the actual underlying data. given that the ems data model strongly relies on data pointers, these isomorphic representations greatly simplify resolving the pointers: their targets will be easily found also in rendering-ready data. conclusions / future directions the early modern songscapes (ems) project demonstrates a mechanism for rendering multi-edition tei and mei xml data within islandora. future work may include overall accessibility of the project site, adding additional songbooks and related resources to the repository, as well as providing a means to annotate the audio and video objects, for example, by using the islandora web annotation module (dsu, 2018). about the authors dr. raffaele viglianti is a research programmer at the maryland institute for technology in the humanities at the university of maryland. raffaele’s work revolves around digital editions and textual scholarship. he is currently an elected member of the text encoding initiative technical council and an advisor for the music encoding initiative, which produces guidelines for the digital representation of music notation with a focus on scholarly requirements. kirsta stapelfeldt is a librarian and the coordinator of the digital scholarship unit at the university of toronto scarborough library. marcus emmanuel barnes is a software developer in the university of toronto scarborough library’s digital scholarship unit. marcus specializes in creating tools that enhance academic libraries for faculty, staff and students. natkeeran ledchumykanthan is a software developer in the university of toronto scarborough library’s digital scholarship unit. nat has more than nine years of full stack development experience (lamp, php, java, javascript, xml technologies) and is an active volunteer for community digital preservation/library projects. references beshero-bondar, elisa e., and raffaele viglianti. “stand-off bridges in the frankenstein variorum project: interchange and interoperability within tei markup ecosystems.” presented at balisage: the markup conference 2018, washington, dc, july 31 – august 3, 2018. in proceedings of balisage: the markup conference 2018. balisage series on markup technologies, vol. 21 (2018). https://doi.org/10.4242/balisagevol21.beshero-bondar01. cayless, hugh, and raffaele viglianti. “ceteicean: tei in the browser.” presented at balisage: the markup conference 2018, washington, dc, july 31 – august 3, 2018. in proceedings of balisage: the markup conference 2018. balisage series on markup technologies, vol. 21 (2018). https://doi.org/10.4242/balisagevol21.cayless01. kepper, johannes. “das freidi-datenmodell” in freischütz digital online project. (2014) https://freischuetz-digital.de/datamodel.html. pugin, laurent. “interaction with music encoding” in „ei, dem alten herrn zoll’ ich achtung gern’“ festschrift für joachim veit zum 60. geburtstag. eds. kristina richts and peter stadler. (2016): 617-631. allitera verlag mengozzi, stefano. “digital and multimedia scholarship.” journal of the american musicological society 67, no. 3 (2014): 839-47. doi:10.1525/jams.2014.67.3.839. raffaele viglianti. “music and words: reconciling libretto and score editions in the digital medium” in „ei, dem alten herrn zoll’ ich achtung gern’“ festschrift für joachim veit zum 60. geburtstag. eds. kristina richts and peter stadler. (2016): 728-746. allitera verlag stapelfeldt, kirsta, and donald moses. 2013. “islandora and tei: current and emerging applications/approaches.” journal of the text encoding initiative 5. http://jtei.revues.org/790. doi:10.4000/jtei.790. dsu (the digital scholarship unit at the university of toronto scarborough library), march 5, 2019. digitalutsc/ems: initial tagged release (version v1.0.0). zenodo. http://doi.org/10.5281/zenodo.2584156 dsu (the digital scholarship unit at the university of toronto scarborough library), december 2, 2018. digitalutsc/islandora_web_annotations: 7.x-1.12 (version 7.x-1.12). zenodo. http://doi.org/10.5281/zenodo.2483252 notes [1] early modern songcapes; [cited 2019, jan 28]. available from: http://ems.digitalscholarship.utsc.utoronto.ca/ [2] henry lawes (1596-1662) is a 17th century composer that the project pis deemed particularly well suited to explorations of song and text. a more fulsome argument about why lawes’s work was selected as the first contribution to this project is proffered on the project website. http://ems.digitalscholarship.utsc.utoronto.ca/content/why-henry-lawes [3] https://cwrc-writer.cwrc.ca/ [4] https://cwrc.ca/about [5] https://github.com/discoverygarden/islandora_critical_edition [6] http://editingmodernism.ca/ [7] https://github.com/sfulibrary/islandora_solution_pack_xml [8] https://github.com/sfulibrary/islandora_context [9] https://www.drupal.org/project/context [10] early modern songscapes viewer integration islandora module https://github.com/digitalutsc/ems/releases/tag/v1.0.0 [11] https://tei-c.org/ [12] https://music-encoding.org/ [13] http://www.chmtl.indiana.edu/ [14] https://www.w3.org/tr/xinclude/ [15] http://ems.digitalscholarship.utsc.utoronto.ca/content/editorial-statement [16] see chapter 12 of the tei guidelines https://www.tei-c.org/release/doc/tei-p5-doc/en/html/tc.html and chapter 10 of the mei guidelines https://music-encoding.org/guidelines/v4/content/critapp.html. [17] a similar approach has been the subject of experimentation of the freischütz digital project, but with substantial differences. in short, the collation document (called core) carries most of the notation data, while sources import common data and provide additional detail (kepper 2014). while this method may be more economical (reduces duplication of notation common to all sources), it makes the production of data considerably harder to achieve. we argue that our method achieves the same goal more simply. [18] respectively https://reactjs.org/ and https://redux.js.org/. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – interpreting marc: where’s the bibliographic data? mission editorial committee process and structure code4lib issue 11, 2010-09-21 interpreting marc: where’s the bibliographic data? the marc data format was created early in the history of digital computers. in this article, the author entertains the notion that viewing marc from a modern technological perspective leads to interpretive problems such as a confusion of “bibliographic data” with “catalog records.” he explores this idea through examining a specific marc interpretation task that he undertook early in his career and then revisited nearly four years later. revising the code that performed the task confronted him with his own misconceptions about marc that were rooted in his worldview about what he thought “structured data” should be and helped him to place marc in a more appropriate context. by jason thomale introduction the machine readable cataloging (marc) format was conceptualized in the early 1960s and first piloted in 1966 (avram, 1975). it is now well over 40 years old. considering just the advances in computer data representation that have happened since, today’s world is alien to the one in which marc was conceived. in 1966, it would still be three years until dr. edgar f. codd would publish his first paper describing a relational model for data as an ibm research report, and four years until he would revise the paper and publish it more widely in communications for the acm (date, 1998); eight years until donald chamberlin and raymond boyce would first present their work on sequel (sql) (chamberlin & boyce, 1974); and ten years until peter chen would first propose the entity relationship model (chen, 1976). we who work with library technology and systems cannot help but view marc [1] through a lens colored by 44 years of rapid technological change (figure 1). we now look at marc based upon a worldview that is utterly different than the one that gave rise to the format; making anachronistic assumptions about how we might think it would work is all too easy. of course we get poor results when we treat marc data the way we would treat data in a relational database—marc predates the earliest formal expression of relational data modeling concepts by three years! figure 1. timeline comparing creation of marc to major developments in software, networking, and data representation between 1960 and 1980 furthermore, the cataloging task, which generates the data that goes into a marc record, seems enigmatic to most library technologists—and for good reason. whereas a format built upon a modern data model would store data about bibliographic items directly [2], marc actually stores bibliographic data by way of “cataloging record[s] … or the information traditionally shown on a catalog card” (furrie, 2009). from a modern systems design perspective, this seems odd. our systems are designed to facilitate online search and retrieval of bibliographic information. is the catalog card abstraction really the best carrier for that data in that context? but marc was not invented to drive computerized information retrieval systems. its original purpose was to automate the processes and tasks of a 1950s/60s technical services department—i.e., the creation and printing of catalog cards (avram, 1975; coyle, 2005; coyle & hillmann, 2007; tennant, 2002). the same basic rules that determined how bibliographic data was to be stored and displayed on those catalog cards became the same rules that determined how data was to be formatted and stored in marc records. in fact, though they have changed over time, the cataloging rules originated long before the advent of modern computer technology (coyle & hillmann, 2007; taylor, 1999). we can now perhaps more easily understand some of the contention we see within the current library metadata environment, which, on the surface, might seem baffling. library catalogers and programmers clash (often passionately) about what constitutes “good library metadata.” on the one hand, many librarians consider traditional library cataloging not only to be useful, but vital for retrieval of online bibliographic information [3]. on the other, many programmers who try to work with catalog records come to conclude that they hinder online access and retrieval because they are so difficult to interpret algorithmically [4]. indeed, i have witnessed and read conversations in which programmers and catalogers attempt to explain their points-of-view to one another. too often both sides speak vaguely. concrete examples—if offered—are explained in language that only one side or the other understands. very little actual communication seems to happen [5]. as a metadata librarian, i have a foot in both worlds: i must be able to read and interpret marc data, but i must also understand systems, data design, and programming. my background in programming and database administration, however, preceded my training as a librarian. when i learned cataloging, my worldview was already set. despite training and experience, i still find that i make automatic, mistaken assumptions about marc. some recent metadata clean-up work required me to revisit code that i had written much earlier in my career; it afforded me the opportunity to see firsthand some of the assumptions i made as a novice and the concrete results of those assumptions. solving one problem in particular—cleaning up generic title metadata that had been derived from the marc 245 field—helped me consciously to experience certain insights about working programmatically with marc data that would have perhaps seemed counter-intuitive to me when i wrote the original code. examining what led to these insights illustrates how programmers can think differently about marc data to understand it better and develop methods to parse it more effectively. it also reveals evidence demonstrating the problems that traditional library cataloging poses in a modern systems context. the problem obtaining metadata for digitized items from the physical items’ marc records is a common task. three and a half years ago i developed a program that did such a thing for one of my institution’s first digital collections: recorded music that we stream online to support our school of music’s teaching activities. because this was one of our first digital collections and i was still new at working with marc data, the resulting metadata suffered from a number of problems. when i recently undertook a revision of my original code to fix these problems, one issue near the top of my “to-fix” list seemed simple on the surface. title metadata (that is, titles for albums—not individual pieces) sometimes exhibited odd formatting quirks. indeed, over the years i had fixed a number of records manually that simply looked weird, and more recently i had begun to notice titles that were both formatted oddly and seemingly incomplete. looking back at the code that i wrote three and a half years ago, i had to try to remember how i originally intended the title element to look. the data came from the marc record’s 245 field. although i could have simply followed cataloging conventions and used the entire title statement exactly as it appeared in the marc record, the formatting seemed arcane, stilted, and jarring to the untrained eye. it would not serve our users well, i thought. in addition, the 245 tended to contain extraneous information that i did not think users would recognize as belonging to the title, such as the statement of responsibility. removing the extraneous information required reformatting what remained. ultimately, i wanted the metadata to appear as natural, simple, and readable as possible—based more on the rules of english grammar than the cataloging rules. when i wrote the first version of the title-extraction routine, i did not realize that it would have such a high failure rate, but i later noticed that, in roughly 25% of cases, odd formatting errors had crept into the output that thwarted my goal. the original solution the original algorithm was simple. i began with the 245‡a (main part of the title), which was all that was actually required. if the 245 contained a ‡b (remainder of the title—usually used for a subtitle) or a ‡n or ‡p (number/name of a part or section of a work), i added a colon and then the ‡b, ‡n, and/or ‡p, in that order, onto the title string. i decided to ignore the ‡c (statement of responsibility) because it was not actually part of the title, and i decided to ignore the ‡h because it tended just to contain “[sound recording].” other subfields were not commonly used, so i ignored them. further, i recognized that any equals signs (=) in the data indicated words that were translations of what had been transcribed from the item. such translations i thought would be important to keep but that the equal-sign notation would be jarring to end users, so i converted equal signs to commas. i also removed extraneous punctuation from the end of the string (such as forward slashes (/)). below are three examples of titles that did not convert as intended. =245 10‡a3 symphonies‡h[sound recording] ;‡bthe rock = der fels = le rocher /‡cserge rachmaninoff. came out as: 3 symphonies: the rock, der fels, le rocher by sheer coincidence, this formatting makes it look as though the rock, der fels, and le rocher are the titles of the three symphonies. in reality, the rock is its own work separate from 3 symphonies whose title just happens to be translated into german (der fels) and french (le rocher) in the marc record. =245 10‡afantasie in c major, op. 15‡h[sound recording] :‡bwandererfantasie /‡cfranz schubert. fantasie in c major, op. 17 / robert schumann. came out as: fantasie in c major, op. 15: wandererfantasie this time the formatting is not bad, but it completely misses the second work on the album (schumann’s fantasie in c major, op. 17). =245 10‡aconcerto no. 1 for piano & orchestra, op. 15, c major‡h[sound recording] =‡bc-dur = ut majeur ; concerto no. 3 for piano & orchestra, op. 37, c minor = c-moll = ut mineur /‡cludwig van beethoven. came out as: concerto no. 1 for piano & orchestra, op. 15, c major: c-dur, ut majeur ; concerto no. 3 for piano & orchestra, op. 37, c minor, c-moll, ut mineur in this case, the resulting title is at least decipherable, but it does not follow my “natural, simple, and readable” criteria. specifically, the punctuation seems inconsistent—even random. a different approach my revision began with a systematic investigation about what i had done wrong. i noted some of the problem titles and examined those titles’ marc 245s vis-à-vis my original code. i asked myself: what was the problem? had i made assumptions the first time that did not match how certain items’ marc was structured? did those items simply contain cataloging errors? if the mistakes were in the cataloging, were they regular and intelligible enough that i could code around them? if the mistakes were mine, how could i make sure that i was not simply writing revised code based on a new set of bad assumptions about the data? examining the marc proved to be eye opening. the majority of errors appeared not to be in the cataloging, but rather in how i had interpreted it. seeing my misinterpretations so clearly made me realize that i needed to change my entire approach to the problem. the three examples below illustrate my mistaken interpretations. =245 1\ ‡amusique de chambre.‡nvol. ii =‡bchamber music, vol. ii = kammermusik, vol. ii‡h[sound recording] /‡cfauré. =245 10‡afantasien op. 116‡h[sound recording] =‡bfantasias ; intermezzi op. 117 ; klavierstücke op. 118 & 119 = pieces for piano /‡cjohannes brahms. =245 10‡aoverture to candide‡h[sound recording] /‡cleonard bernstein ; [arr.] grundman. george washington bridge / william schuman. an outdoor overture / aaron copland. el salon mexico / aaron copland ; [arr.] hindsley. chester / william schuman. la fiesta mexicana / h. owen reed. in my initial solution, i was treating each subfield as a discrete piece of data. i picked and chose the subfields i thought i needed and simply ignored the others. in many cases, however, the subfields i was ignoring—the ‡h and the ‡c—actually contained important information. in the second example, above, the ‡h contains a translation-indicator (=) that tells me that the next subfield begins with a translation, and is probably not actually a subtitle; in the third example, the ‡c contains a lengthy list of titles of works that appear on the album in addition to what is already listed in the ‡a. my initial solution also disregarded the order in which subfields appeared. it assumed that it could impose a standard order (‡abnp) regardless of what appeared in the data—indicating that i thought either that order does not matter or that subfields always appear in that particular order. in the first example, not only does a ‡n appear between a ‡a and a ‡b, but the ‡n also contains an “=” immediately before the ‡b. taken out of order, the “=” would apply to something other than what is obviously intended—that “chamber music” is a translation. on a more general level, i realized that the contents of these 245s looked and acted less like data from a structured data record and more like textual markup from a document [6]. structured data formats are intended for machine consumption and tend to follow rules that are simple and consistent—simpler rules make data easier to parse. also, the more structured a data record is, the more explicit the semantics tend to be. meaning is clear and encapsulated—the overall context in which data appears within a record is irrelevant because, apart from what might be specified in the data model, context carries no semantic meaning. interpreting a structured data record is no more complex than reading each data element and interpreting it according to that format’s specifications. a document, however, is less straightforward, as the information it contains is meant more for human consumption; even if the document is marked up to aid machine-processing, the underlying structure is based on linguistics rather than a format that was designed to be machine-readable. the information in a marked-up document therefore behaves more like language than data. meaning is not clear and encapsulated. the document as a whole contains semantic meaning beyond the sum of its marked-up data elements. interpreting the document requires reliance on subtle ambiguities and context clues more than it does simple rules and specifications. removing or refactoring data elements, unless done in a way that takes context into account, easily changes the meaning of the document as a whole. looking at it from this perspective, i saw that the marc 245s that i had been studying demonstrate the characteristics of marked-up documents. each 245 field functions as a complete unit. the subfields help make meaning explicit by delineating the parts of that unit, but the context in which the subfields appear also carries meaning: if you remove a subfield without considering how it relates to the whole, or if you change the order of subfields without considering how they relate to one another, you can inadvertently change the meaning of the whole. the 245 has an implicit structure that is not defined by the subfields; rather it is defined by the cataloging construct known as the title statement. the subfields merely help demarcate some of the parts of that title statement. because my original solution to the problem had come from a structured-data perspective, its approach was decidedly rules-based. if i had maintained that approach throughout the revision process, my next step would have been to examine the marc and cataloging documentation more closely to determine the precise rules upon which the contents of the 245s were supposed to be based. then i would have developed methods for handling the subfields that followed these rules. i probably would have ended up with a complex series of loops and if/then/else statements that would have attempted to arrive at appropriate decisions based on which subfields appeared in what order and what punctuation they contained. the code would have become messy quickly. but what if i treated the data more as it was now beginning to look to me: as textual markup? looking at the problem this way revealed new possibilities. i began to think of a solution that would recognize and match the patterns inherent in the data instead of attempting to deconstruct and then reconstruct the data based upon the rules on which it had been built. finally, to ensure that i did not repeat my earlier mistake of coding to a data set that was too limited, i employed a more systematic workflow (figure 2): i created a script that would apply my title extraction routine to each 245 in the entire batch of marc records for the collection and output the results in a text file for me to examine. i also created a script that would apply the same routine to a single title that i defined within the script (for testing and developing). i ran the batch conversion script on the entire marc record store. the script generated a plain text file containing each converted title. i looked through the output to find the next title that did not convert correctly. i looked up the 245 in a human-readable version of the marc record store. i used the new 245 as my test case and rewrote or tweaked my code until that 245 converted to an acceptable-looking title without breaking my previous test cases. i repeated steps 2 through 5 until i could find no more titles in the batch output that looked incorrect. figure 2. title-conversion-routine-revision workflow toward a revised solution when i was ready to write my new algorithm, i began by browsing through my marc record dump to see what consistent patterns i could find in the 245 data. i noticed several. periods and semicolons seemed to delimit major components of the overall title—especially when titles of separate works were contained within the overall album title. forward slashes delimited the beginning of a list of responsible parties, which always ended with a period. equals signs indicated translations. colons and commas indicated minor subdivisions within a component of a single work’s title. brackets indicated information that the cataloger had included but had not transcribed directly from the physical item during cataloging. from these particular patterns, i could divide punctuation appearing in the title into three separate categories: punctuation that served as a major component separator (.;), punctuation that indicated information i wanted to remove (/=[]), and punctuation that served as a minor component separator (,:). i noticed a couple of complicating factors, however. first—and most seriously—periods were used not only as component delimiters but also to indicate abbreviations. second, i noticed that there were actually cases in which bracketed information was desirable to keep in the final output. the first problem was the more challenging. there seemed to be no convenient way to tell which periods were used for abbreviations and which were not. i solved the problem using brute force—i created a separate script that looped through each 245 in the entire marc record batch. it read any words that ended with a period, tallied them up, and generated a sorted list: 721 op 573 no 251 mozart 250 bach 109 beethoven 105 nos 95 vol 94 k 83 haydn 70 mendelssohn 69 vivaldi 67 schubert 65 schumann 65 brahms 43 handel 40 j 35 chopin 34 d 30 shostakovich 30 prokofiev 29 al 28 no 28 debussy 25 music 25 ravel it took little time to eyeball the list, extract what appeared to be abbreviations, and use those abbreviations as exceptions in my pattern-matching routines. i also excepted any initials (i.e., individual alphanumeric characters appearing as words unto themselves or sets of individual alphanumeric characters separated by periods). the second complicating factor—the brackets—was much easier to rectify. in general, i noticed that brackets appearing in ‡h contained data that i wanted removed; brackets appearing in any other subfield contained data that i did not want to remove, though i still wanted to remove the brackets themselves. my new code takes a basic two-step approach. first, it loops through each subfield contained within a 245 field and preprocesses the subfield data—it processes abbreviations; it generates an unambiguous delimiter character (the pipe (|)) to replace the periods that serve as “major component” delimiters; and it preprocesses certain other punctuation marks based upon the subfield in which they fall, such as the brackets in the ‡h. second, it processes the entire title as a single string—it matches patterns in the whole string that it would otherwise miss simply looking at each subfield in isolation; it determines data to be removed and removes it; it consolidates any repeated pipes into a single pipe character; and finally it converts all pipes to semicolons and performs other formatting cleanup. after a few iterations of coding, testing against a single test case, testing against the entire marc dump, and then picking a new test case, i arrived at the final routine. the perl code for the conversion routine is as follows: sub version2_convert { my $field = shift; # $field is a marc::field object my $abbrevs = "op|no|nos|vol|vols|al|etc|nr|st|posth|opp|jr|" ."dr|app|arr|orchestr|orch|ed"; my $title; foreach my $subfield ($field->subfields()) { my $title_part = $subfield->[1]; # convert all . to | $title_part =~ s/(\.)/|/g; # convert | after $abbrevs back to . $title_part =~ s/(^|[^\p{l}^\p{n}]+)($abbrevs)\|/$1$2./ig; # convert | after initials back to . $title_part =~ s/(^|[^\p{l}^\p{n}]+)(\p{l})\|/$1$2./g; # convert . at the end of the string to .| # (periods at the end of a subfield indicate end of a section) $title_part =~ s/(\|)(\p{l})\|/$1$2./g; $title_part =~ s/(\.)(\p{l})\|/$1$2./g; $title_part =~ s/\.$/.|/; # if this is a subfield h if ($subfield->[0] eq "h") { # make sure there's a space before any ending / $title_part =~ s/(\s)(\/)$/$1 $2/; } else { # remove [ and ] $title_part =~ s/[\[\]]//g; } # if this is a subfield c and a / is missing from the title if ($subfield->[0] eq "c" && $title !~ /\/$/) { $title .= " /"; } # if this is a subfield n or p if ($subfield->[0] eq "n" || $subfield->[0] eq "p") { # convert a pipe at the end of the title string so far to a comma $title =~ s/\|$/,/; # convert pipes within the subfield n or p to commas $title_part =~ s/\|/,/g; $title_part = ", " . $title_part; } $title .= $title_part; } # remove ||| (comes from transformation of ...) $title =~ s/\|\|\|//g; # remove everything between = and /;,:| or eos $title =~ s/=(.*?)(\s\/|;|,|:|\||$)/$2/g; # remove everything between / and | $title =~ s/\s+\/.*?(\||$)/$1/g; # remove everything between []s $title =~ s/\[.*?\]//g; # convert ; to | $title =~ s/;/|/g; # remove repeated | $title =~ s/(\s*\|\s*)+/\|/g; # remove repeated , $title =~ s/([:,\|])(\s*,\s*)/$1 /g; # convert | to ; (space insensitive) $title =~ s/\s*\|\s*/; /g; # smoosh :,;. over to the left $title =~ s/\s+([,\.;:])\s*/$1 /g; # add a space after commas that have been smooshed together $title =~ s/(\p{l}|\p{n}),(\p{l}|\p{n})/$1, $2/g; # remove leftover ; and space at the end $title =~ s/(,|;|\s)*$//; return $title; } table 1 shows, for a selection of titles, a comparison between the marc 245, the output of the original algorithm, and the output of the revised algorithm. marc 245 original output revised output =245 1\ ‡a3 symphonies‡h[sound recording] ;‡bthe rock = der fels = le rocher /‡cserge rachmaninoff. 3 symphonies: the rock, der fels, le rocher 3 symphonies; the rock =245 1\ ‡afantasie in c major, op. 15‡h[sound recording] :‡bwandererfantasie /‡cfranz schubert. fantasie in c major, op. 17 / robert schumann. fantasie in c major, op. 15: wandererfantasie fantasie in c major, op. 15: wandererfantasie; fantasie in c major, op. 17 =245 1\ ‡aconcerto no. 1 for piano & orchestra, op. 15, c major‡h[sound recording] =‡bc-dur = ut majeur ; concerto no. 3 for piano & orchestra, op. 37, c minor = c-moll = ut mineur /‡cludwig van beethoven. concerto no. 1 for piano & orchestra, op. 15, c major: cdur, ut majeur ; concerto no. 3 for piano & orchestra, op. 37, c minor, c-moll, ut mineur concerto no. 1 for piano & orchestra, op. 15, c major; concerto no. 3 for piano & orchestra, op. 37, c minor =245 1\ ‡aweihnachtsoratorium‡h[sound recording] : bwv 248.‡nkantate 1-3 =‡bchristmas oratorio /‡c[johann sebastian bach]. weihnachtsoratorium: kantate 1-3, christmas oratorio weihnachtsoratorium: bwv 248, kantate 1-3 =245 1\ ‡afantasien op. 116‡h[sound recording] =‡bfantasias ; intermezzi op. 117 ; klavierstücke op. 118 & 119 = pieces for piano /‡cjohannes brahms. fantasien op. 116: fantasias ; intermezzi op. 117 ; klavierstücke op. 118 & 119, pieces for piano fantasien op. 116; intermezzi op. 117; klavierstücke op. 118 & 119 =245 1\ ‡aoverture to candide‡h[sound recording] /‡cleonard bernstein ; [arr.] grundman. george washington bridge / william schuman. an outdoor overture / aaron copland. el salon mexico / aaron copland ; [arr.] hindsley. chester / william schuman. la fies overture to candide overture to candide; george washington bridge; an outdoor overture; el salon mexico; chester; la fiesta mexicana =245 1\ ‡aviolin concerto no. 2‡h[sound recording] ;‡bsymphony no. 4 : landscape /‡cpaul cooper. let us now praise famous men ; elegy / samuel jones. violin concerto no. 2: symphony no. 4 : landscape violin concerto no. 2; symphony no. 4: landscape; let us now praise famous men; elegy =245 1\ ‡apentimento‡h[sound recording] /‡cezra laderman. symphony no. 3 : the tricentennial / lester trimble. pentimento pentimento; symphony no. 3: the tricentennial =245 14 ‡athe ballad of baby doe‡h[sound recording] :‡b[opera in two acts /‡clibretto by john latouche ; music by douglas moore]. the ballad of baby doe: [opera in two acts the ballad of baby doe: opera in two acts =245 14 ‡athe protecting veil /‡ctavener. third suite for cello, op. 87 / britten. [thrinos :‡bfor solo cello / tavener]‡h[sound recording]. the protecting veil /: for solo cello / tavener] the protecting veil; third suite for cello, op. 87; thrinos: for solo cello =245 1\ ‡a4 orchesterstücke.‡nop. 12 sz51‡h[sound recording] =‡borchestral pieces = pièces pour orchestre ; konzert für orchester, sz116 = concerto for orchestra = concerto pour orchestre /‡cbéla bartok. 4 orchesterstücke.: op. 12 sz51 orchestral pieces, pièces pour orchestre ; konzert für orchester, sz116, concerto for orchestra, concerto pour orchestra 4 orchesterstücke, op. 12 sz51; konzert für orchester, sz116 =245 1\ ‡astring quartets op. 51‡h[sound recording] ;‡bstring quartet op. 67 /‡cjohannes brahms. “american quartet” op. 96 / antonin dvorák. string quartets op. 51: string quartet op. 67 string quartets op. 51; string quartet op. 67; “american quartet” op. 96 =245 1\ ‡apartita no. 1, bwv 825‡h[sound recording] ;‡benglische suite no. 3, bwv 808 = english suite = suite anglaise ; französische suite no. 2, bwv 813 = french suite = suite française /‡cjohann sebastian bach. partita no. 1, bwv 825: englische suite no. 3, bwv 808, english suite, suite anglaise ; französische suite no. 2, bwv 813, french suite, suite française partita no. 1, bwv 825; englische suite no. 3, bwv 808; französische suite no. 2, bwv 813 table 1. comparison between marc 245, output of the original algorithm, and output of the revised algorithm. analysis and conclusion the revision process of my marc title-extracting routines revolved around a significant fact and a significant insight. the fact: marc is, at its heart, a data format built to contain catalog records; bibliographic items are described via the catalog records rather than directly via the structured marc data. understanding and internalizing this leads us to the insight: if we think of those catalog records as structured documents rather than as data records, then marc has as much in common with a textual markup language (such as sgml or html) as it does with what we might consider to be “structured data.” thinking broadly, the fact that the marc format focuses on catalog records rather than bibliographic items lends credibility to the idea that marc has been decreasingly relevant to library systems since the demise of the card catalog. many in the library profession recognize that improving library tools requires the development of data formats that better support system functionality that modern library users need and expect. extensive efforts to rectify the situation have been underway for over a decade, such as creating new models for bibliographic data, updating cataloging rules, and attempting to convert library data to new formats [7]. however, much bibliographic data is still locked away in marc. programmers who must write the code to interpret marc have the daunting task of trying to understand a data format that is inherently alien to them. the aforementioned fact and insight can, perhaps, help make marc seem slightly less alien to the programming mind—specifically upon examining the implications. first, a marc record does contain an explicit structure. it contains fields, subfields, and indicators, and our tools for processing marc give us the ability to extract discrete, granular chunks of data just as we would from, e.g., a database. concrete rules and semantics define the data that goes into a marc record and dictate the use of the fields and subfields. when we must interpret, extract, or otherwise act upon the data, the documented rules and semantics can help guide our interpretation. second, data within a single marc field often behaves like document markup. unlike data occurring in a database, data expressed through document markup gains some degree of meaning based upon its context within the document and its relation to other data within the document. in marc, punctuation that appears in one subfield can subtly change the meaning of data in a different subfield; changing subfield order can also subtly change the data’s meaning. third, extracting bibliographic data from marc is often not a simple matter just of interpreting the explicit structural information. in many cases—such as the case of the 245 field—we must recognize the existence of both an explicit and an implicit structure. marc subfields comprise the explicit structure; cataloging rules define the implicit. the two are intertwined, and both must be interpreted. what appear to programmers as cataloging artifacts—odd patterns of punctuation and formatting within the data that are often dismissed as merely presentational—actually carry semantic meaning that is lost if the patterns are ignored. if we want to pull a book’s title out of the 245, we must understand that doing so requires us to interpret the “title statement” cataloging edifice—otherwise our methods will not work well. this need to decipher data built upon cataloging rules programmatically is perhaps the biggest technical hurdle in the quest to convert library data into more machine-readable formats en masse. cataloging has a long, complex history [8]. multiple cataloging standards—or codes—have been developed throughout the years. standards used concurrently with marc—such as the anglo-american cataloguing rules (aacr & aacr2)—have continued to evolve. because marc contains cataloging but does not enforce the use of any single cataloging code, it may not be possible to pinpoint exactly what set of cataloging rules any given record is based upon [9]. furthermore, the act of cataloging—no matter the set of rules used—relies on human beings to enter data logically and consistently. because modern cataloging rules are complex, any marc record set will naturally contain some errors that could cause a program to misinterpret the data. the “cataloging” in any given set of marc records is therefore a moving target. in my particular example, the records that i used had been cataloged mostly according to some version of the aacr2 standard [10], which somewhat mitigated the effects of this problem. even so, i believe that the approach i have described offers a potential avenue for handling record sets where this is not the case. because the approach is one of reverse engineering based on bottom-up, pattern-matching techniques, it should allow for greater flexibility in accommodating inconsistencies (e.g., when the cataloging does not follow the cataloging rules). it also should require no knowledge of exactly which cataloging standards were used in creating the marc data; in theory, it only requires a programmer to be able to recognize the sets of consistent patterns and variations on those patterns that appear within the data. future work would need to test these hypotheses on record sets that exhibit increasingly more complex and varied cataloging in order to help determine how to refine and adapt the approach. although outside my direct realm of expertise, exploring the use of machine-learning algorithms and data analytics to help with pattern identification would be another logical pathway for developing the approach further. references avram, h. d. (1975). marc; its history and implications. washington, dc: library of congress. british broadcasting corporation. (2008, august 5). the accelerator of the modern age. retrieved from: http://news.bbc.co.uk/2/hi/technology/7541123.stm coyle, k. (2005). catalogs, card—and other anachronisms. the journal of academic librarianship 31(1), 60-62. coyle, k., & hillmann, d. (2007). resource description and access (rda): cataloging rules for the 20th century. d-lib magazine (13)1/2. retrieved from: http://dlib.org/dlib/january07/coyle/01coyle.html chamberlin, d. d., & boyce, r. f. (1974). sequel: a structured english query language [electronic version]. proceedings of the 1974 acm sigfidet workshop on data description, access, and control, 249-264. new york, ny: association for computing machinery. retrieved from: http://www.almaden.ibm.com/cs/people/chamberlin/sequel-1974.pdf chen, p. p. (1976, march). the entity-relationship model—toward a unified view of data [electronic version]. acm transactions on database systems, march 1976, 9-36. new york, ny: association for computing machinery. retrieved from: http://csc.lsu.edu/news/erd.pdf date, c. j. (1998). the birth of the relational model: thirty years of relational. intelligent enterprise magazine, october 1998. retrieved from: http://intelligent-enterprise.informationweek.com/db_area/archives/1998/9810/feat4.jhtml furrie, b., & database development department of the follett software company. (2009). understanding marc bibliographic: machine-readable cataloging (8th ed.). washington, dc: library of congress. retrieved from: http://www.loc.gov/marc/umb/ hauben, r. (1998, june 23). from the arpanet to the internet: a study of the arpanet tcp/ip digest and the role of online communication in the transition from the arpanet to the internet. retrieved from: http://www.columbia.edu/~rh120/other/tcpdigest_paper.txt holmevik, j. r. (1994). compiling simula: a historical study of technological genesis [electronic version]. ieee annals in the history of computing 16(4): 25-37. piscataway, nj: ieee educational activities department. retrieved from: http://www.idi.ntnu.no/grupper/su/publ/simula/holmevik-simula-ieeeannals94.pdf stroustrup, b. (2010, march 7). bjarne stroustrup’s faq. retrieved from: http://public.research.att.com/~bs/bs_faq.html taylor, a. g. (1999). the organization of information. englewood, co: libraries unlimited. tennant, r. (2002, october 15). marc must die. library journal. retrieved from: http://www.libraryjournal.com/article/ca250046.html notes [1] just a quick note of clarification: throughout the paper, i specifically have in mind the marc 21 format for bibliographic data, even though i refer to it only as “marc”. [2] the functional requirements for bibliographic records data model is a perfect example of a “modern data model” for bibliographic data. this model was developed in the late 1990s. see http://www.ifla.org/en/publications/functional-requirements-for-bibliographic-records for more information. [3] this sentiment is so widespread in the library profession that any citations i could provide to back up this statement would be both woefully incomplete and largely unnecessary—any cataloger would validate my assertion. you can, however, view a good illustration of this point—for example—in the comments following the language log blog posting about the “metadata train wreck” that is google books: http://languagelog.ldc.upenn.edu/nll/?p=1701. [4] as with the preceding statement about librarians’ attitudes toward cataloging, i cannot really give citations to back up this claim with 100% certainty and objectivity—as far as i can tell, programmers’ attitudes toward cataloging have never been studied. i can give examples, however. roy tennant’s 2002 piece entitled “marc must die” (referenced elsewhere in this paper) provides a good look at some of marc’s problems from a modern technological viewpoint. also, on occasion, library coder jonathan rochkind documents his problems writing code that must use marc data. see, for example: http://bibwild.wordpress.com/2009/09/24/a-reasonable-display-for-series-data-in-marc/ http://bibwild.wordpress.com/2009/09/28/more-marc-issues-700/ and http://bibwild.wordpress.com/2009/09/30/cataloging-and-citations/. there is also the statement about working with marc data purportedly made by google engineer leonid taycher that “the first thing he had to learn was that the ‘machine readable’ part of the marc acronym was a lie” (from http://go-to-hellman.blogspot.com/2010/01/google-exposes-book-metadata-privates.html). finally, the ngc4lib, or next generation cataloging for libraries, listserv (view archives at http://serials.infomotions.com/ngc4lib/) provides a wealth of documented discussions between programmers and catalogers about the utility of cataloging data. [5] clearly this is my opinion and is based mainly on anecdotal evidence. i again submit that the ngc4lib archives contain numerous examples backing up my point that catalogers and programmers often speak past one another. [6] within the xml community exists an analogous concept—that of “document-centric” versus “datacentric” xml. (for a concise explanation, see http://techessence.info/node/51.) thinking of the document/data division as a continuum rather than a binary either/or (and applying it beyond just the xml format), marc would fall somewhere in the middle. it is fielded and thus behaves like data, but the contents of many of those fields behave more like documents (where the subfields provide mark-up). [7] frbr is the primary example of a “new” model for bibliographic data. the work happening on resource description and access (rda) is the primary example of cataloging rules being updated. there have been a number of recent efforts to publish library cataloging data in linked data formats on the web: for example, http://www.viaf.org and http://id.loc.gov/. [8] chapter three (pages 37-52) of the organization of information by arlene g. taylor contains a succinct chronology of the historical developments that lead to modern cataloging. cataloging as we know it is largely a product of the 19th and early 20th centuries. [9] position 18 of the marc leader contains a code that can help you determine what descriptive cataloging standard was used in the creation of the record. it can indicate aacr2, international standard bibliographic description (isbd), non-isbd, or an unknown cataloging standard. still, knowing you have a “non-isbd” or “unknown” standard is not terribly helpful, and it does not tell you what version of aacr2 or isbd were used. [10] because the records were cataloged according to aacr2, the formatting of the marc 245s (e.g., the punctuation) conformed to isbd. although the approach that i took did not require it, i could have consulted the isbd documentation to help me make sense of the data. about the author jason thomale (j <dot/> thomale <at/> ttu <dot/> edu) is a metadata librarian currently going on his sixth year at texas tech university. subscribe to comments: for this article | for all articles 25 responses to "interpreting marc: where’s the bibliographic data?" please leave a response below: jakob, 2010-09-22 thanks for the practical insight! i’d like to point out that your approach does not only hold for marc, but for almost any kind of data that is not fully atomic and normalized. schemas (sql, xml schema, owl…) only describe what you called “explicit” structure, but as people create data, they add “implicit” structure, based on additional conventions or ad-hoc rules. i bet in 40 years we will complain about some ancient xml or rdf data that contains strange artifacts. of course marc is much worse because it is so old, designed for cataloging records. but try to get information out of other decade-old databases that have been designed for a specific use-case that is not yours, and you will stumble upon similar problems. mary mastraccio, 2010-09-22 jason thomale’s article on interpreting marc is the best article i have read on the issue. it should be required reading for every cataloger and anyone trying to create or map marc records. jrochkind, 2010-09-22 excellent article. i’m hoping that the contortions you had to go through to get the title out in a flexibly displayable format (spending quite a bit of time investigating, arriving at a fairly complex algorithm) help demonstrate why marc is not in fact a very good data format for contemporary needs. but i’m also planning on taking your algorithm and porting it to ruby for my own marc title display purposes, thanks for including the code. mike m., 2010-09-22 very interesting article with many clarifying points. while i appreciate the thrust of the article as an exemplar of the difficulties of marc and the needs of info retrieval i’d like to point out as a practical point that i’d probably prefer to use the 240 uniform title and 700|t as my main title fields. you’d run into many of the same problems but would be starting out a bit cleaner. jason w. dean, 2010-09-22 jason, what a beautiful article. i love the whole to part explanations that you do about metadata in the beginning – and specifically marc’s place within this galaxy of metadata. illuminating – i am sharing this for sure – thanks for writing this!! matthew phillips, 2010-09-23 i enjoyed the article, but i wondered whether you had actually checked aacr2 to investigate the rules regarding punctuation? your section “toward a revised solution” reads as though you were trying to determine the patterns by look at the example data. fair enough, i suppose, as aacr2 is considered indigestible by many people. but often the punctuation preceding the subfield indicator is a greater clue to the purpose of the data than the indicator itself. anyway, your approach works very well, especially when faced with the all-conquering |c subfield after which no more subfields can be specified! it’s just a shame that us-marc won. in the uk we developed a marc variant call uk-marc where the subfield markers were closely related to the data types, and punctuation for display was generated from the subfield markers. sadly this computer-friendly format was abandoned in the name of international standardisation. you would not have had half the problems if you had been dealing with uk-marc: http://www.bl.uk/ukmarc/marc245.html james weinheimer, 2010-09-23 this is a great article, and i am sure i will refer to it repeatedly. but it does show something more about the differences between a cataloger and a programmer: the cataloger knows the rules and looks at the record as a complete entity, whereas the programmer looks at each field separately. so, in the records you show, you must go beyond the 245 field to get a true grasp of the situation. here is an excerpt of a record from lc, taken from one of your examples: ================================== 100 1_ |a bach, johann sebastian, |d 1685-1750. 240 10 |a partitas, |m harpsichord, |n bwv 825, |r b? major; |o arr. 245 10 |a partita no. 1, bwv 825 |h [sound recording] ; |b englische suite no. 3, bwv 808 = english suite = suite anglaise ; franzo?sische suite no. 2, bwv 813 = french suite = suite franc?aise / |c johann sebastian bach. … 700 12 |a bach, johann sebastian, |d 1685-1750. |t englische suiten. |n nr. 3. 700 12 |a bach, johann sebastian, |d 1685-1750. |t franzo?sische suiten, |n nr. 2. ================================== with our cataloging rules, the parsing you mention has always been performed manually, using separate 700 author/title analytic entries, plus the 240 field. the fundamental purpose of the 245 field is not so much for search and retrieval, as to provide a reliable transcription of what appears on the title page, including all of the typos, etc. it was there for description and identification purposes. the access of the item was always through controlled fields. when keyword was introduced, while it added to access in certain ways, it also “threw a spanner in the works”, in other ways, when looking at it from a traditional viewpoint. but yes, your basic point is correct: in many ways, marc records are very akin textual markup language. ted gemberling, 2010-10-01 thanks. that’s a very interesting and enlightening article. i hope lots of people read it. ted jakob, 2010-10-07 beside the book “marc; its history and implications” by avram 1975 there are two articles by sally h. mccallum, worth to mention: “marc: keystone for library automation”. ieee annals of the history of computing 24(2), page 34-49, 2002; and “machine readable cataloging marc: 1975-2007”. encyclopedia of library and information sciences, 3rd edition, page 3530-3539. taylor & francis, 2009. melanie, 2010-10-08 speaking as an extremely interested cataloger, i would love to know: 1) would there be any way to change marc to do the job better? maybe something as simple as creating fields without subfields; for example, 245 could be just the title proper and a 246 could be what is now in the 245 $b and a 247 could be what is now the 245 $n and the 248 could be what is now the 245 $p, etc. would that work at all? 2) is there any software framework that can replace marc yet? one that works better for computer manipulation of data? i’ve heard murmurs of rdf, but i’m not sure what it does yet. i’d be happy to give up marc if a good replacement is offered. jakob, 2010-10-15 @melanie 1) i think you could change marc to do better but then i would not be marc anymore. you would need to ban many current use, so it would be a totally different format. i suppose that just changing fields will not work. 2) just using marc, xml, rdf, or anything else is not enough but you need to keep in mind the framework around these formats and how they are actually be used. the task is to exactly define the format in atomic parts, that can be used independently from each other. as shown in this article, marc was not designed for this task. xml can do (but it does not require) and rdf is more designed to split and merge pieces of data. you can also design unusable records in rdf, just as you could design better usable records in marc, but the framework around rdf and how it is actually used, encourages you to design data in a more usable way. melanie, 2010-10-22 so, if i understand you correctly: 1) you could do that with marc, but it wouldn’t really be a good solution to the problem. it certainly wouldn’t solve the problem of legacy data!, but then i realized that even before i asked the question. the conversion of legacy data is a lead weight that drags the whole process down, i think. and 2) there isn’t yet really a good replacement yet. maybe xml with a really good dtd? matthew phillips, 2010-12-15 i see also that unimarc (the main rival to marc21 outside the english-speaking world) went a similar route to uk-marc, meaning that the punctuation separating fields is generated from the subfield markers. so it looks like it’s just marc21 which is tied to the era of catalogue card production. earl_j, 2011-06-15 great article provides much to think and rethink… earl_j, 2011-06-16 hello jason, tried your direct email without success … you know, in my mls training, we called marc machine-readable code … maybe if we bring back that term, your notion of the markup might make more sense to others. just a thought that zipped through as i was thinking about the article … or perhaps i was rethinking, gee whiz, it is hard to keep track. grin until that time … earl j. embracing lossy: sacrificing metadata to gain agility | american society for information science & technology, 2011-07-08 […] [6] thomale, j. (2010, september 21). interpreting marc: where’s the bibliographic data? code{4}lib journal, 11. retrieved march 24, 2011, from http://journal.code4lib.org/articles/3832 […] linked data and libraries: linked data opac » overdue ideas, 2011-07-14 […] tricky… 245 field in marc may duplicate information from elsewhere got lots of help from http://journal.code4lib.org/articles/3832 (with additional work and […] interpreting marc – article « all things cataloged, 2011-08-19 […] current issue of the code4lib journal features an excellent article by jason thomale, “interpreting marc: where’s the bibliographic data?”. the abstract: […] rules vs. format « all things cataloged, 2011-08-19 […] to rethink the relationship between the rules and the format. in his article for code4lib, “interpreting marc“, jason thomale cogently talks about explicit vs. implicit structure as one of the reasons […] my spring 2014 “digital archives + institutional memory” studio | words in space, 2013-12-06 […] thomale, “interpreting marc: where’s the bibliographic data?”code4lib 11 […] re: interpreting marc: where’s the bibliographic data? | first thus, 2014-06-26 […] to interpreting marc: where’s the bibliographic data? by jason thomale code4lib journal, issue 11, […] re: linked data | first thus, 2014-06-26 […] rochkind wrote:concerning: “one example of this can be found reported in this article: http://journal.code4lib.org/articles/3832&#8220; <snip>okay, what would someone who “knows library metadata” do to get […] re: interpreting marc: where’s the bibliographic data? | my great wordpress blog, 2014-06-26 […] to interpreting marc: where’s the bibliographic data? by jason thomale code4lib journal, issue 11, […] marc, linked data, and human-computer asymmetry | peer to peer review, 2015-02-05 […] the displayed materials down into computer-manipulable data. sometimes it can be done, but only at great cost in time and effort; sometimes it is outright impossible. even when retooling human displays as data is possible, the […] my spring 2014 “digital archives + institutional memory” studio – words in space, 2020-09-08 […] thomale, “interpreting marc: where’s the bibliographic data?”code4lib 11 […] leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – groupfinder: a hyper-local group study coordination system mission editorial committee process and structure code4lib issue 13, 2011-04-11 groupfinder: a hyper-local group study coordination system groupfinder is a system designed to help users working in groups let each other know where they are, what they are working on, and when they started. students can use the groupfinder system to arrange meetings within the library. groupfinder also works with the phpscheduleit room reservation system used to reserve group study rooms at the d.h. hill library at ncsu. information from groupfinder is presented on the groupfinder web site, the mobile web site and on electronic bulletin boards within the library. how groupfinder was developed from the initial concept through the implementation is covered in the article. by joe ryan and josh boyer introduction groupfinder is a system designed to help users working in groups let each other know where they are, what they are working on, and when they started. there are several ways to post to the groupfinder system. users can view groupfinder posts inside of d.h. hill library (the main campus library at north carolina state university) on large electronic displays (hereafter referred to as “eboards”), library kiosk workstations, the library web site, and the mobile web. groupfinder is inspired by bulletin boards, a way many libraries have traditionally facilitated informal student meet-ups. it was designed and developed by ncsu libraries staff from the digital library initiatives and research & information services departments. today, groupfinder is a heavily used, highly visible system in an unusually busy and connected library. tour guides often talk about groupfinder as they showcase the library, and some students use it every week to set up meetings. a few years ago, however, groupfinder was just an idea brought about by a set of observations: that the library is really busy, and that students in the library were heavy facebook users. project history and motivation groupfinder grew out of the creative burst of energy that accompanied the opening of the learning commons in march 2007. like the myriad academic libraries that have opened learning commons (or information commons) in the last decade, we hoped to provide a more technology-rich and student-friendly environment. the learning commons in the d.h. hill library features over 100 desktop computers (pcs and macs); printers; scanners; a poster plotter; laptop lending; other devices available for check-out such as cameras, gps units, and kindles; reference books; comfortable seating; group study rooms; a presentation practice room; rolling whiteboards; video games; and a service desk staffed by reference staff and student employees. library staff have worked hard to devise ways to help patrons find all these new tools and services. an early success in this effort was a set of computer availability maps. these maps display the up-to-the-minute availability of 103 computers in the learning commons on the 1st floor and another 108 computers on floors 2 – 9 (see http://www.lib.ncsu.edu/compavailability/web.php). with the computer availability maps displaying continuously on eboards throughout the library, students who need a desktop computer can quickly find one. what else might students need help finding? library patrons need help finding books, study rooms, specialty rooms like our digital media lab, restrooms, vending machines — you name it, someone is lost in a library looking for it. in the early days of the learning commons, josh boyer, associate head of research and information services, and tito sierra, associate head for digital library development, pondered another problem. in addition to considering what students were looking for in the library, they wondered who they were looking for. patrons come to the library not just for the collections, equipment, and services, but to find each other. d.h. hill library is one of the most heavily used buildings on the ncsu campus. hundreds and sometimes as many as 1,000 students are scattered throughout the library at any given time. our 9 reservable group study rooms are nearly always full. we had observed students standing near the entrance, waiting for each other, trying to find each other using their mobile phones. could we help them find each other? if we found a way to help our patrons connect, we could further the library’s role as academic heart of the campus. the first effort was simple and cheap. we put a bulletin board on the learning commons website. we put it out there to see how students would use it and hoped it would be useful. we wondered if students would use it to find each other and arrange meetings. they did not. the bulletin board gathered useful feedback for a few months (mostly debates pro and con about the noise level in the boisterous and social learning commons), and then grew repetitive and argumentative. we abandoned the bulletin board idea. we were trying to create an online social tool to help students find each other in the library. online… social… college students — facebook!  it seemed so obvious — we needed to build a facebook application. “the d.h. hill activity wall,” as sierra dubbed it, would be a brilliant custom facebook app. we would plug in to the dominant social network of our time. success would surely follow. sierra brought joe ryan, digital projects librarian, into the conversation. by late 2008, sierra and ryan had a prototype facebook app up and running. in early 2009, ryan and kim duckett, principal librarian for digital technologies and learning, took the app to a group of students in a communications class. their feedback was unanimous and clear — good idea, but don’t build it in facebook. when boyer heard this, he was stunned and disappointed. what could possibly be wrong with the simple equation: online + social + college students = facebook? boyer and duckett held a focus group of students to further test the idea of the d.h. hill activity wall idea. same result. good idea, but don’t build it in facebook. the students told us we had correctly identified a need, but our solution was wrong. yes, they told us, finding friends and classmates in the library was difficult, and yes, the library could certainly provide some kind of tool to help, but they could not imagine going to facebook to find out where their group was meeting. “i don’t know,” one student said. “it doesn’t sound practical. it would be an extra step, an extra process.” facebook is a social space for friends and peers. it’s not about studying, the students told us, and interacting with the library on facebook feels uncomfortable and strange. we gave up on the facebook app. going back to the drawing board, we held a series of one-on-one interviews with students in order to explore why they come to the library and how they find friends and study partners once they are here. the students described many contexts for meeting at the library — groups of friends relaxing, students meeting tutors, club meetings, etc. — but the activity they talked about most was group study for a class. students described how they arrange, before or after class, or via email, a time to meet at the library. these study groups often include friends and also acquaintances from class whose phone numbers they do not have. so they meet at the appointed time near the entrance to the library, and there and then the problems begin. the group’s schedule is hostage to its most tardy member. even groups of friends who all have each others’ cell phone numbers wait for each other at the entrance because the cell phone coverage of the d.h. hill library gets worse as you move beyond the entrance. once all members of a study group have assembled, they search the library for a suitable place to study. in our crowded library, this often involves looking one place, then another, and maybe a third spot before settling down to study. this process is why would-be group members have to meet at the entrance in the first place — if they decide on a specific location ahead of time, it will likely turn out to be unavailable. during these interviews, we mostly stuck to listening to students describe how they did group study. we did not spend time pitching our various ideas for the nascent, non-facebook d.h. hill activity wall. when we did hint, however, that any potential tool would probably post information to library’s eboards, students embraced the idea. one student, reflecting on the wandering that groups have to do to find a good place to study, said “yes, you could post ‘we’ve moved to the 4th floor.'” there is an important exception to students’ inability to plan a time and place to meet within d.h. hill library — 9 group study rooms. these rooms are reservable online. the popularity of these rooms is difficult to overstate. during desirable meeting times, the pm hours, they are booked nearly non-stop. we imagined that the d.h. hill activity wall would serve the study groups with unsettled plans — the “we’ve moved to the 4th floor” groups — more than those with reserved study rooms. as you will see below, we were in for a surprise. ryan got to work building a prototype of the new tool. after several design iterations informed by individual and group interactions with students, a prototype was ready for testing in october 2009. ryan put the d.h. hill activity wall on a computer kiosk in the main lobby of the library for in situ usability testing. ryan, boyer, and a usability consultant, abe crystal of morebetterlabs, inc., offered students coffee and asked them to test drive the application. as always, students responded positively to the purpose of the tool, agreeing that it met a real need. “this definitely would be useful,” one said. “it’s pretty cool,” said another. ryan’s prototype was more than halfway there. it worked. it was usable. the tests identified several minor user interface issues but validated the product’s functionality. while ryan was making his last few tweaks, we rethought the name. d.h. hill activity wall no longer captured what we were building. the tool was about a specific kind of activity, group study. its job was to help people find their group. we decided not to over-think it: groupfinder. with final tweaks complete, groupfinder launched on october 26, 2009. would anyone use it? we put a large poster in the lobby next to the new kiosk workstation, and hoped. hours passed. would anyone use it? what if no one did? what if only a few people did? busy with these thoughts and philosophical musings about what “success” might be, we were too busy to notice that just after 1:00 pm, (in fact, at 1:08:37 est), we had our first post. and it wasn’t even a staff member taking the new system for a test drive! it was, however, a student employee. he used groupfinder for its intended purpose and told his group to check the eboards to find out where to go. the next day saw its first official use by a non-library employee. in the days to follow, use trickled in. a post or less per day was the norm. what was happening? boyer ended up in a conversation with a library student employee who made what turned out to be a revolutionary suggestion: what if you could post to groupfinder directly from the online room reservation system? after all, the employee reasoned, when someone reserves a room, they already have to tell us who they are, when they will be here, and where they are–almost the same information groupfinder displays. several weeks later, on november 30, 2009, integration work with the study room reservation system was complete. now, when a patron reserved a room, a single check box labeled “post to groupfinder?” allowed easy access to the system. that same day, seven different students posted activities to groupfinder. use increased steadily for the remaining few weeks of the semester. during that time, ryan developed a mobile web version of groupfinder that allowed both posting and viewing of activities. early the next semester, ryan refined the visual design of the various interfaces so that they shared a common branding and color treatment. since early 2010, the system has been in use in this state. read on to find out how it works. how it works assuming that people are often reluctant to adopt new ways of accomplishing basic tasks like setting up group meetings, the groupfinder team set a goal of having the system be extremely easy to use, both for the organizer of a study group and for the members of the study group who need to know the topic, time, and location of the group. creating a groupfinder post the groupfinder system supports activity creation via three venues: the main groupfinder interface on the libraries’ web site (http://www.lib.ncsu.edu/groupfinder), the online study room reservation system, and the libraries’ mobile web site. figure 1. groupfinder main interface creating a groupfinder post on the main interface (figure 1) takes just three steps. first, the user enters a short description of their study group in the large input box. then, the user clicks, “where are you meeting?”, which displays a list of locations in the building, sorted by floor. figure 2 shows the dialog box and its use of small photographs to help users identify locations whose names may not be familiar. after describing the activity and its location, the user can then click the “post” button. once the user has successfully authenticated to the campus authentication system, the activity is displayed on all groupfinder interfaces. figure 2. list of available locations posting on the mobile web interface (figure 3) is similar to the main interface, except that the locations are represented in dropdown form instead of a dialog box. again, after filling out the form completely, the user must authenticate, and the activity is displayed on all groupfinder interfaces. figure 3. mobile web interface the most popular method of creating groupfinder posts to date, however, has been via its tight integration with the ncsu libraries study room reservation system. the ncsu libraries uses phpscheduleit, an open source reservation system, for scheduling of group study rooms in the d.h. hill library. the groupfinder team modified the source code of phpscheduleit to show an option on the reservation form to allow users to have their reservation information display in groupfinder at the appropriate time (figure 4). users are required to provide a brief description of their activity and can only reserve a room if they can authenticate to the campus authentication system. figure 4. phpscheduleit and groupfinder posting to groupfinder via the study room reservation system, the idea of the student employee mentioned above, has accounted for nearly 94% of posts. this surprised us, because we imagined there would be a great deal more use of groupfinder to announce meetings in ad hoc contexts. because of groupfinder, we have a greater appreciation of students’ need to reserve group study rooms. viewing groupfinder posts figure 5. viewing a post on the main groupfinder web site groupfinder posts are visible in three places: the main groupfinder web site (figure 5) , the mobile web site (figure 3), and on eboards. each interface displays activities in reverse chronological order, with the newest activities at the top of the screen, and the oldest activities at the bottom of the screen . groupfinder displays activities for a set length of time depending on where they were created. for activities posted from the main or mobile web interfaces, the activity displays for two hours, although users can end their own activities early if desired. groupfinder displays activities that were created via the room reservation system starting fifteen minutes before the start of the reservation through the end of the reservation window. moderating groupfinder posts during the project planning process, we were worried that, once undergraduates learned that they could post messages to eboards, abusive or offensive posts might be a problem. to that end, groupfinder has an administrative interface where individual posts can be disabled by an administrator. to aid community moderation, any authenticated user can also report an offensive activity for review by the groupfinder team. once a user reports an activity, the activity is emailed to the team for review. fortunately, there have only been a small number of cases where the team has found it necessary to disable a post. typically, boyer checks the queue of activities that will display in the morning. night staff at the circulation desk also have the ability to disable a post. in the past year, we have disabled fewer than ten activities, most of which were disabled because of expletives in the activity description. under the hood groupfinder is a php application that uses mysql for data storage. visual effects in the user interface are supported by jquery ui. the ncsu libraries has released an open source (mit license) version of groupfinder that includes a web interface and an eboard display template. because the study room reservation system code and the mobile web code depend heavily on other custom code in use at ncsu, they are not included in the open source release. the source code is available on google code at http://code.google.com/p/groupfinder/. usage since its launch on october 26, 2009, groupfinder has displayed information on more than 3,400 group study sessions created by over 1,100 unique users. because library usage patterns closely mirror the semester schedule, we will present numerical use data from the spring 2010 and fall 2010 semesters, and will conclude with an examination of the ways patrons are using groupfinder. for purposes of this analysis, “semester” is defined as starting on the first day of classes and ending on the last day of final examinations. spring 2010 semester usage spring 2010 (january 18 – may 13) was the first semester where groupfinder was available in its present form for the entire semester. groupfinder displayed 1,316 posts by 495 unique users, an average of 2.7 posts per user. on the busiest day, april 20, 2010, groupfinder displayed 30 posts. on that day, groupfinder displayed at least one activity continuously from 10:45 am until 12 midnight on april 21. fall 2010 semester usage the fall 2010 semester (august 18 – december 16) saw the heaviest use of groupfinder to date. there were 1,611 posts by 639 unique users, an average of 2.5 posts per user. this semester’s usage reflects a 29% increase in the number of unique users and a 22% increase in the number of posts. on the busiest day, november 2, 2010, groupfinder displayed 28 posts. on that day, groupfinder displayed at least one activity continuously from 10:00 am until 12 midnight on november 3. types of usage groupfinder was originally envisioned to be a system for students to notify each other about planned and ad hoc study sessions within the library space. and, indeed, this is its most popular use. since its launch, however, groupfinder users have made the system their own by adapting it for a number of alternative uses. below are some rough categories of usage, each with example posts from the system. for privacy, the names of the posters are not reproduced here. class-oriented study groups * eng 261 (october 25, 2010, 6:30 pm – 7:30 pm) * csc302 numerical methods is really frustrating :( (march 9, 2010, 12:30 am – 2:30 am) person-oriented study groups * eric’s tutoring (october 26, 2010, 2:30 pm – 3:30 pm) * alonso’s study room (may 3, 2010, 12:00 pm – 12:30 pm) campus organization meetings * campus farmers market at ncsu (september 27, 2010, 7:00 pm – 9:00 pm) * hosa executive board meeting (april 4, 2010, 8:00 – 8:30 pm) organizationally oriented study groups * interested gentlemen of lambda theta phi (november 5, 2010, 1:30 pm – 3:00 pm) * theta tau study hall (february 2, 2010, 9:00 pm – 11:00 pm) religious use * christians on campus bible study (september 16, 2010, 5:30 pm – 7:30 pm) * tajweed (january 28, 2010, 5:00 pm – 7:00 pm) wildcard use * kittykat’s getting physical (october 27, 2010, 6:00 pm – 8:00 pm) * blink 182 fan club! (september 8, 2010, 8:30 pm – 10:30 pm) * egg fryin’ on a rock (april 5, 2010, 2:00 pm – 4:00 pm) * star warriors (january 30, 2010, 3:00 pm – 5:00 pm) what users are saying on november 29, 2010, we sent out a link to a web-based survey to every single person who had posted to groupfinder since the start of the fall 2010 semester. 77 of the 591 users completed our survey, a response rate of 13%. our motivation for surveying our users was simple: we designed groupfinder to make it easier for library patrons to meet up at the library in order to get group work done. we asked questions about what role in the group the respondent typically plays (organizer or member), their perception of groupfinder’s utility, whether groupfinder prevents them from having to wait in the lobby for latecomers, questions related to mobile phones’ utility to their meet ups, ways they typically organize meetings, and, finally, how we could make groupfinder more useful to them. do our users find groupfinder useful? 46% of respondents rated groupfinder “very useful” in their efforts to set up group meetings. an equal share of respondents rated groupfinder “somewhat useful,” and 9% of respondents rated groupfinder as “not very useful.” did we achieve our goal of reducing the amount of time people have to wait in the lobby for group members who are running late? 73% of respondents said that we did. so far, so good! but of course most of our use comes from posts made via the room reservation system. we asked if respondents had thought about using the standard posting interface? 80% of them said that they hadn’t even thought about it. one respondent commented, “i didn’t know you could use it outside of the reservation system” while another said, “i tried to do this once, but could not figure it out.” the question that yielded the most interesting responses was a free-form question, where we invited respondents to suggest ways to make groupfinder more useful. we received many interesting suggestions, ranging from adding the ability for users to add their own start times on the web interface, to email and/or text message reminders that a meeting was about to start. overall, the survey data suggest a need to promote groupfinder generally to increase awareness and usage. also, as confirmed by use data, there is a need to promote the ability to use groupfinder outside of the study room reservation system. some survey respondents did not seem to be aware that groupfinder was a separate system from the room reservation system, which presents an interesting question: should we work to merge the two systems further and market them jointly as “groupfinder,” or should we work to differentiate the two systems further? conclusion although groupfinder started out as a few casual observations about a busy library, lots of group activity, and the popularity of facebook, it has grown into a multi-faceted and heavily used tool that is helping to change how patrons use library spaces. we consider it our own take on location-based social networking. compared to the location-based services of twitter, facebook, foursquare, gowalla, and others, groupfinder is hyper-local, limited to one building! the success of groupfinder owes a lot to our willingness to listen to our users and to take risks. if you are interested in pursuing a similar project at your library, we urge you to begin with the users and their needs instead of a desired tool or technology. by following this process with groupfinder, we believe that we have built a useful system. about the authors joe ryan (josephdryan@gmail.com) is the humanities research associate at unc chapel hill research computing. he was the digital projects librarian at north carolina state university until february 2011. josh boyer (josh_boyer@ncsu.edu) is the associate head of research & information services at north carolina state university. subscribe to comments: for this article | for all articles 2 responses to "groupfinder: a hyper-local group study coordination system" please leave a response below: the limits of facebook « fail!lab, 2011-04-12 […] i saw some evidence from the good folks at ncsu to back up my gut […] cbs bibliotek blog – innovation & ny viden » blog archive » nyt nummer af the code4lib journal, 2011-04-14 […] groupfinder: a hyper-local group study coordination system […] leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – implementing a collaborative workflow for metadata analysis, quality improvement, and mapping mission editorial committee process and structure code4lib issue 23, 2014-01-17 implementing a collaborative workflow for metadata analysis, quality improvement, and mapping the university of north texas (unt) and the oklahoma historical society (ohs) are collaborating to digitize, process, and make publicly available more than one million photographs from the oklahoma publishing company’s historic photo archive. the project, started in 2013, is expected to span a year and a half and will result in digitized photographs and metadata available through the gateway to oklahoma history. the project team developed the workflow described in this article to meet the specific criterion that all of the metadata work occurs in two locations simultaneously. by mark phillips, hannah tarver, and stacy frakes introduction the university of north texas (unt) and the oklahoma historical society (ohs) are collaborating to digitize, process, and make publicly available more than one million photographs from the oklahoma publishing company’s historic photo archive.  the project, started in 2013, is expected to span a year and a half and will result in digitized photographs and metadata available through the gateway to oklahoma history.  the project team developed the workflow described in this article to meet the specific criterion that all of the metadata work occurs in two locations simultaneously. the gateway to oklahoma history the gateway to oklahoma history [1] was launched in april of 2012 to showcase the unique collections housed at the oklahoma historical society.  the gateway is operated by the university of north texas libraries for ohs and provides access to nearly one million pages of early oklahoma newspapers.  the oklahoma publishing company (opubco) photography collection is the first non-newspaper collection added to the gateway.  all digital content added to the gateway is held in the preservation repository at the unt libraries and uses the content delivery system and digital preservation infrastructure at unt alongside other digital library projects such as the unt digital library [2] and the portal to texas history [3].  ohs expects that a wider range of content will be added to the gateway in the upcoming years including collections of audio, video, maps, and letters, as well as more digitized newspapers and photographs. oklahoma publishing company photography collection the oklahoma publishing company photography collection grew over time as photographs were taken to accompany stories published by oklahoma newspapers.  opubco archived the hard copies which span from the late 1800s to roughly 1994 when the company decided to use digital photography.  most photos are of newsworthy events in oklahoma or noteworthy oklahomans living across the united states or abroad.  in 2011, the gaylord family sold the newspaper company and the new owners had no interest in retaining the collection.  at that time, the family gifted the collection – approximately 1.7 million photographs – to ohs for preservation. the photos are filed alphabetically by subject in 8 ½ inch by 12 inch envelopes, each with a heading that describes the photo or photos inside. the back of each photo is labeled with any known information, such as the photographer’s stamp, the date it was taken, the date it was published (if it was published), and the article that accompanied the image. in addition to photographic prints, the collection contains other items that are significant to newspaper photojournalism, including images on zinc printing plates, pressed paper printing press images, negatives, slides, and glass plates. the items in this collection illustrate the history of newspaper photography and the advances made in photograph technology during the 20th century. ethics in journalism grant early in the process of acquiring the opubco collection, ohs applied for a $300,000 grant from the ethics in journalism foundation. the funds cover two years and are being used to digitize, process, and store the collection for presentation in the gateway to oklahoma history. initially the funds helped hire staff and purchase supplies to move the collection from the opubco headquarters to its current location at the oklahoma history center. now that the project is under way, the funds pay for digitization of the collection at the records conversion division of oklahoma correctional industries; workers scan the original photos and prepare metadata. grant funds also support unt’s processes for ingesting the data into the gateway to oklahoma history. because the grant funding lasts only two years and ohs anticipates the project will extend some years into the future, ohs may need to seek an alternative source of funding when the grant ends. the workflow the workflow for digitizing the opubco photograph collection comprises five distinct steps: image preparation, image and metadata capture, metadata cleaning, metadata mapping, and metadata quality control. during the first step, photo archivists at the oklahoma historical society prepare images for the digitization process by pulling photographs that will not be included in the project.  the remaining photographs are delivered to oklahoma correctional industries where the digitization and metadata extraction occurs.  the images are scanned at 300 ppi and saved in the tiff format; due to the size of the project, the backs of the photographs are not scanned.  as part of the digitization process, the contractors generate descriptive metadata by transcribing information from the photograph file into a spreadsheet (see table 1), with each row representing a single photograph and each spreadsheet representing an entire archival box. field name field description example entry imagefile unique identifier for an individual photograph, matching the filename of the corresponding tiff. 2012.201.b0316.0019.tif envelope text printed on the envelope that contains the photograph. clark, jimmy / scottish / race car driver headline headline for the article in which the photograph appeared. safety queen back in the lime light caption caption that accompanied the published photograph. here’s a dog who has been living up to his good, catchy name. photog name of the photographer who took the photograph. richard green originaldate date that the photograph was taken. 19610428 publishdate date that the photograph was published, often with a code representing the newspaper in which it appeared. o-05-24-1959 captionwriter author(s) of the caption printed in the newspaper to accompany the photograph. vernon b. snell credit newspaper in which the photograph was published. daily oklahoman source newspaper in which the photograph was published. oklahoma times city city where the photograph was taken. tulsa state state in which the photograph was taken (or similar division for locations outside the united states). oklahoma country country in which the photograph was taken. usa table 1. descriptive metadata fields transcribed from the photo files for each image. the vendor delivers the original photographs plus the digitized images and metadata spreadsheets to ohs in batches. there, staff members inventory the photographs and conduct high-level metadata analysis to verify that the digitization vendor is meeting the requirements of their contract.  after performing these checks, staff members group images and metadata into a larger batch of roughly 40,000 images (including metadata) to deliver to the unt libraries. metadata cleaning raw metadata in the spreadsheets needs some initial clean-up before mapping.  the following graphic illustrates the remaining steps for processing the supplied metadata and converting it to the local untl metadata format: figure 1: workflow for opubco photography project. in some cases, values in a particular field or column require normalization to fix typos and other minor inconsistencies in the source metadata.  more complex problems can occur when a row has multiple columns with values that have been switched or fields that have been accidentally skipped.  the project team uses the open-source tool open refine (formerly google refine) and the web-based google fusion tables for metadata cleaning and a locally-developed tool (m2m) at the unt libraries for metadata mapping.  final changes and quality control happen within the gateway’s metadata editing framework. open refine (google refine) open refine is “a free, open source, power tool for working with messy data” [4].  in this workflow, unt libraries staff members use it first to load multiple similarly-formatted spreadsheets simultaneously and concatenate the individual box-level metadata spreadsheets into a larger dataset representing the entire batch of content.  the first two batches processed each contained more than 39,000 photographs with metadata spread across 60-80 separate excel spreadsheets; open refine could load all of these items in a single import step.  once staff members have loaded the data for a batch, they identify and normalize values across rows within specific columns.  columns that have reasonable overlap of values between records are most suitable for this type of editing.  for example, the country column shares many codes among multiple records, while publishdate and caption do not, making country a better candidate for this step. typically, once staff members identify a suitable column for this step, they select the option to create a “text facet” from each unique value in that column.  this lets them sort the entire set of values by the number of times each one occurs in the dataset, or alphabetically by value.   by browsing this list they can identify values to normalize.  two examples in this dataset are the normalization of various forms of “united states” and “unknown” for missing values.  in the country field, disparate values such as united states, u.s.a, us, u.s, and usa, were all normalized to “usa,” chosen to represent the country “united states” in this project.  additionally, staff members for this project made a decision to explicitly identify rows without a known value using a default value of “unknown.”   various versions of this string were present in all columns in the dataset and were normalized using the same faceting process. open refine also identifies the number of cells in each column that have a “blank” value and allows the user to adjust them as needed.  one of the most powerful features of open refine for this type of work is the ability for a user to make mass edits across a number of cells in a column.  for example if there were sixty instances of the string “united states” in the country column, a user could change all of the values to the preferred “usa” with a single change.  this makes the normalization process across the dataset much quicker than approaching it in a row-oriented manner. other features of open refine which are helpful for the normalization process include the ability to apply basic string transformation techniques to each cell in a column.  two of these common transforms include “trim leading and trailing whitespace” and “collapse consecutive whitespace” to remove superfluous spaces.  other features include changing values to title case, uppercase, or lowercase, and the option to unescape html entities.  it is also possible to write custom transformations on the cells within a column by writing short bits of transformation code in one of three languages: clojure, jython, or google refine expression language (grel).  although these tools were not utilized in this workflow directly, they provide endless possibilities for similar kinds of data transformations and normalization of other datasets. open refine allows users to make use of facets for multiple columns as they process data, to make identification of inconsistencies across rows more obvious.  in the opubco dataset there are three columns which contain information related to place: city, state, and country.  these three columns were compared together to identify rows in the dataset where the country was identified as usa or canada, but for which the state column value was not one of the fifty states or thirteen provinces or territories, respectively.  likewise, this provided the ability to identify rows where a known city, such as “oklahoma city,” had missing or mismatched state and country values. the faceting approach also helped users identify cells which were transposed by one or more columns during the initial data entry process.  open refine allows a user to facet on custom functions related to the column cell values.  for example the number of characters in a field can be used as the facet value instead of the actual string present in the field.  this is helpful when there is an expected range for the length of a field, such as a date field which should always have four to eight digits for this dataset.  fields that did not meet the expected length often contained values entered into incorrect columns during the transcription process. a powerful feature of open refine is the ability to “cluster” values of a column and provide a simple interface for merging similar values.  this is especially useful for columns that have a reasonable amount of overlap but sometimes suffer from transcription errors, such as names.  open refine has two kinds of clustering: key collision, and nearest-neighbor.  key collision takes the values of a column and applies a weak hashing function to the input string, then groups all of the values on this newly generated hash or key.  a number of hashing functions are present and described in the tool’s documentation.  the first key collision hashing function is called fingerprint and applies the following transformations to the input string in order to create the key: remove leading and trailing whitespace change all characters to their lowercase representation remove all punctuation and control characters split the string into whitespace-separated tokens sort the tokens and remove duplicates join the tokens back together normalize extended western characters to their ascii representation (for example “gödel” >> “godel”) this is helpful for catching a number of typos and grouping inverted and non-inverted names if they consist of only first and last name.  for example the following values are all assigned the same fingerprint of “a owen y”: a. y. owen owen, a. y. a y owen a. y owen once a collision is identified, open refine chooses the entry with the most instances as the target to be used when merging these values.  a user has the opportunity to modify this suggested value with a more appropriate value to be applied after the merge.  as a user merges various terms, he can save changes to the dataset and rerun the clustering process. in addition to the fingerprint function, open refine uses ngram-fingerprint, and two phonetic fingerprint algorithms: metaphone3 and cologne-phonetic.  the documentation for open refine explains these algorithms and their strengths in further detail [5]. open refine can also create clusters using a nearest-neighbor clustering algorithm (also known as knn).  there are two algorithms available for use in this clustering process, levenstein (or edit distance) and ppm (prediction by partial matching).  for a full explanation of the algorithms and process used for the nearest neighbor clustering functionality, consult the open refine documentation [5]. once staff members have finished column-centric data normalization, they export the entire dataset as a comma-separated-value (csv) file for the next step of the process. google fusion tables google fusion tables [6] is “an experimental data visualization web application to gather, visualize, and share larger data tables” [7].  the project team has found it to be a useful tool for the opubco photograph project for several reasons.  first, the application is available freely through the google drive suite of tools in a web-based context; this means that multiple people across both institutions could interact with one hosted version of the dataset and not have to resort to checking out, modifying, and then checking in files, or even worse, e-mailing versions of the dataset between the various project team members.  second, fusion tables provides a row-based view of the dataset with the ability to edit each row as a “record.” the ability to edit a row as a record makes it possible to edit a field within a row or to copy and paste between fields of a row quickly. third, the application provides the ability to search and facet on one or more fields similarly to open refine to help analyze values across the dataset. a staff member loads the exported spreadsheet from the open refine portion of the workflow into google drive to generate a new fusion table.  once created, other participants in the project are allowed editing access to the table with the “share” feature.  so far there have been two team members at unt and one member at ohs who have participated in data cleanup utilizing one fusion table. each team member focuses on a different aspect of data cleanup. one focuses on place, photographer, and caption writer names; another focuses on dates; and yet another focuses on headlines, captions, and envelope data.  with all three working on specific areas of the dataset, work happens quickly.  row-based changes – such as moving values that had been transposed – are straightforward using fusion tables.  in this tool, focusing on one value opens the entire row/record making it easy to shift values or change multiple field values in a single incorrect row. figure 2: detail of edit row screen in google fusion table. because of data inconsistencies in the first batch of items, roughly 3,500 rows were modified using fusion tables. once all data manipulations are completed in google fusion tables, staff members export the resulting dataset as a csv file for use in the next step of the conversion process. metadata mapping all of the digital objects hosted in the unt system – including those in the gateway to oklahoma history – have the same metadata schema containing twenty-one possible fields.  this unt libraries (untl) format is a locally-qualified dublin-core-based metadata format; semantic guidelines are fully documented on the digital projects unit website [8].  at this stage in the workflow, data from the photos in the first batch needed to be mapped from the spreadsheets to xml files in the untl format; once established, the same mapping will work for all future batches. untl metadata the goal behind mapping the field values included preserving all of the information available (i.e., documented in the spreadsheets) while matching untl formatting to improve usability of information for the general public.  based on the content available for the opubco photographs, staff members conceptually mapped data from the spreadsheets to the most relevant fields in the untl metadata (see table 2). untl field qualifier opubco field(s) title main imagefile added headline creator photographer photog contributor author captionwriter date creation originaldate description content source caption subject keyword source envelope opubco envelope coverage place name city state country date originaldate source newspaper headline source publishdate identifier local control number imagefile note display credit publishdate table 2. relationship of opubco photo fields to untl fields. additionally, some untl fields had default values that did not come from the opubco data, such as resource type, language, and physical description.  some values required formatting changes, e.g., inverting personal names for creators and contributors, normalizing dates to a yyyy-mm-dd format, and changing locations to hierarchical strings. for each untl field, staff members documented which values should come from the spreadsheets, any changes or ordering of data, and how to handle information that was not available.  as an example, the “source” field pulls information from the headline, source, and publishdate fields in the spreadsheet.  initial notes about the mapping included the following: source qualifier: newspaper text: “{{{headline}}},” {{{source}}}, oklahoma city, oklahoma, {{{publishdate}}} x-mm-dd-yy >> month day, year — if headline is unknown, use {{{source}}}, oklahoma city, oklahoma, {{{publishdate}}} — if source is unknown, do not include source — if publish date is unknown, leave it out terms surrounded by three braces represent values pulled from columns in the spreadsheets with those titles.  this list provided the general information to create a script for generating the final records.  so a complete untl source term would look like: “air safety officers,” oklahoma city times, oklahoma city, oklahoma, october 25, 1957.  this documentation serves as pseudo-code utilized for creating the formal mapping between the spreadsheet data and the untl metadata format discussed in the section below. metadata to metadata (m2m) a final mapping step uses a local unt libraries tool to generate untl records from the spreadsheets, following the conceptual mapping.  this tool, locally called m2m which stands for “metadata to metadata,” is based on work described by janée and frew in their paper “a hybrid declarative/procedural metadata mapping language based on python” [9].  the m2m tool works with a spreadsheet in the csv format and applies a set of declarative mapping functions to each row in the dataset to emit a properly-formatted untl xml file for ingest into the digital library system. example 1. sample m2m mappings for title, contributor, and subject #create record instance record = recordclass("mphillips") # mapping of column headline to title with addedtitle qualifier record.map("basic", "title", row["headline"].title(), required=false, qualifier="addedtitle") # mapping of column captionwriter to contributor with aut qualifier, # per for the name type, and "caption writer" as added info" record.map("agent", "contributor", invertname(row["captionwriter"]), qualifier="aut", required=false, agent_type="per", info="caption writer.") # mapping of column envelope to subject with kwd qualifier, split input on # forward slash and then apply a title case function to each piece, then create # an instance of subject in the metadata record for each piece record.map("basic", "subject", row["envelope"], qualifier="kwd", required=false, split=" / ", function=(lambda x: x.title())) example 2. command line script to convert spreadsheet data to xml records python m2m.py -m opubco2untl.py opubco-november.csv because the mappings in the m2m tool are written in python, it is easy to implement modifications to the input data as it is processed, for example: changing the case for a field to title case, inverting personal names, loading external data to use as replacement dictionaries, or creating new metadata fields based on the presence of specific attributes in the supplied data.  these features were used extensively in the full mapping file for the opubco project. example 3. input data formatted in json { "caption": "\" the tail section of a t-33 jet trainer from vance air force base, enid, landed in an open field southeast of tinker afb after colliding with an f-104 starfighter jet thursday afternoon southeast of tinker.\" ", "captionwriter": "mark sarchet", "city": "oklahoma city", "country": "usa", "credit": "daily oklahoman", "envelope": "accidents / airplane / county / 1950-1970", "headline": "unknown", "imagefile": "2012.201.b0051.0221.tif", "originaldate": "19591112", "photog": "cliff king", "publishdate": "11-13-1959 oklahoman", "source": "daily oklahoman", "state": "oklahoma" } example 4. resulting untl metadata.xml file after metadata mapping <?xml version="1.0" encoding="utf-8"?> <metadata> <title qualifier="officialtitle">[photograph 2012.201.b0051.0221]</title> <creator qualifier="pht"> <type>per</type> <name>king, cliff</name> </creator> <contributor qualifier="aut"> <info>caption writer.</info> <type>per</type> <name>sarchet, mark</name> </contributor> <date qualifier="creation">1959-11-12</date> <language>nol</language> <description qualifier="content">photograph used for a story in the daily oklahoman newspaper. caption: "the tail section of a t-33 jet trainer from vance air force base, enid, landed in an open field southeast of tinker afb after colliding with an f-104 starfighter jet thursday afternoon southeast of tinker."</description> <description qualifier="physical">1 photograph: b&w</description> <subject qualifier="untl-bs"> business, economics and finance journalism </subject> <subject qualifier="kwd">accidents</subject> <subject qualifier="kwd">airplane</subject> <subject qualifier="kwd">county</subject> <subject qualifier="kwd">1950-1970</subject> <subject qualifier="kwd">daily oklahoman photographs</subject> <subject qualifier="opubco">accidents / airplane / county / 1950-1970</subject> <primarysource>1</primarysource> <coverage qualifier="placename"> united states oklahoma oklahoma county oklahoma city </coverage> <coverage qualifier="date">1959-11-12</coverage> <coverage qualifier="timeperiod">mod-tim</coverage> <source qualifier="newspaper"> daily oklahoman, oklahoma city, oklahoma, november 13, 1959 </source> <collection>okpcp</collection> <institution>okhs</institution> <resourcetype>image_photo</resourcetype> <format>image</format> <identifier qualifier="local-cont-no">2012.201.b0051.0221</identifier> <note qualifier="display">publishdate: 11-13-1959 oklahoman</note> <note qualifier="display">credit: daily oklahoman</note> <meta qualifier="metadatacreator">mphillips</meta> <meta qualifier="hidden">true</meta> </metadata> the resulting metadata file is coupled with the master image file and a json file containing the original data mapped into the untl file.  the resulting structure is created for each image file before it is loaded into the gateway. example 5. digital folder structure for an object ready for ingest into gateway. 2012.201.b0051.0221 |-01_tif | |-2012.201.b0051.0221.tif |-02_json | |-metadata.json |-metadata.xml quality control the untl records generated from the original spreadsheets are ingested along with the images into the gateway to oklahoma history.  metadata editors at the ohs then use the gateway’s metadata editing interface to finalize the records by adding details and making corrections.  this typically includes generating a title for each photograph, adjusting content descriptions, and adding keywords or personal names.  editors also check the other values to ensure that the information matches the photograph and catch any additional formatting or typographical errors missed during the metadata cleaning stage.  as editors complete this final stage, they make photographs accessible to the public. conclusion the workflow that the oklahoma historical society and the unt libraries use for the opubco photo project is a straightforward, scalable, and repeatable process that others can leverage for their own collaborative projects.  since it uses primarily web-based tools, it allows for flexibility in the number of editors that have access to the datasets and limitations of resources.  combining open refine and google fusion tables allows staff members to edit data using both column-based and row-based tools and techniques to achieve the cleanest data possible given the size of the dataset and the timeframe for editing.  this approach drastically reduces the amount of manual editing post-ingest and helps to prevent consistency errors.  these tools also make it easy to divide the work into a number of parts and share them between the institutions, since multiple project members could edit the metadata without the need to track and merge changes between versions.  finally the utilization of the unt-specific metadata-to-metadata (m2m) mapping process provides another layer of flexibility for helpful modifications to the metadata records just before they are loaded into the production system, which reduces the amount of repetitive manual editing required for the large collection of photographs.  with a project completion date of late 2014, unt libraries and osh expect the workflow outlined in this article to scale adequately to meet the challenge of adding nearly one million new images with metadata to the gateway to oklahoma history. notes [1] the gateway to oklahoma history; http://gateway.okhistory.org/ [2] unt digital library; http://digital.library.unt.edu/ [3] the portal to texas history; http://texashistory.unt.edu/ [4] open refine; http://openrefine.org/ [5] open refine documentation; https://github.com/openrefine/openrefine/wiki/clustering-in-depth#key-collision-methods [6] google fusion tables; http://www.google.com/drive/apps.html#fusiontables [7] about google fusion tables; https://support.google.com/fusiontables/answer/2571232?hl=en [8] unt libraries metadata documentation; http://www.library.unt.edu/digital-projects-unit/metadata/ [9] janée, greg, and james frew. “a hybrid declarative/procedural metadata mapping language based on python.” national geospatial digital archive. n. page. web. 15 jan. 2014.  http://www.ngda.org/docs/pub_janee_ecdl_05.pdf. authors mark phillips is assistant dean for digital libraries at the university of north texas libraries. hannah tarver is head of the digital projects unit at the university of north texas libraries. stacy frakes is the oklahoma publishing company photo archivist at the oklahoma historical society. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – rss feed 2.0: the crux of a social media strategy mission editorial committee process and structure code4lib issue 31, 2016-01-28 rss feed 2.0: the crux of a social media strategy this article explains how the university of nebraska kearney calvin t. ryan library improved their social media strategy by using an rss 2.0 feed to update and sync social media tools and create a slideshow on the library’s home page. an example of how to code a well-formed rss 2.0 feed with xml is given, in addition to php, html, and jquery utilized to automate the library home page slideshow. by michael sutherland according to a recent acrl conference paper, it is clear that a lot of academic libraries are using social media. in the conference paper, “making social media meaningful: connecting mission and policies” johnson and burclaff indicate that 94% of academic libraries have a social media presence, primarily on facebook, twitter, and youtube (johnson et al. 2013). what is not clear is how libraries are coordinating content across those social media channels. using social media can be time consuming and maintaining the libraries’ various social media channels a chore. not only does it take time to learn a specific tool such as facebook: an individual could spend their entire day monitoring, updating, contributing content, and posting responses to each tool. in order to sync content, and coordinate those channels, libraries can turn to enterprise solutions, such as gremln, salesforce marketing cloud, or hootsuite; however, a simpler approach is to utilize an rss (really simply syndication or rich site summary) feed to disseminate content to various social media channels. at the university of nebraska kearney calvin t. ryan library, we started using a wordpress blog for news and events in the library that grew to include items of potential interest to the campus community connected to the library. categories were created to describe posts in more detail, structure content and link related posts, such as “library info” or “e-resources”. items the library wanted to highlight on the home page of the library web site were placed in a selective category titled “home page”. an rss feed was created from the category by simply adding the word “feed” in front of the url like this https://inbriefonline.wordpress.com/category/home page/feed/ the feed was then run through feedburner in order to optimize the way the feed was presented to subscribers. blog posts from the home page were cross-posted on social media channels by following the feed in facebook and twitter. this method worked well for a while, but over time, introduced several problems. first, how do you handle retweets, other institutions’ blog posts, or news stories not created by library staff that might be of potential interest to library users? it is awkward to repost those items through the blog, such as an article the library dean wanted posted from the chronicle of higher education. in this case, what meaningful content can be contributed about a subject that someone else has already thoroughly addressed? secondly, the rss feed did not immediately update through feedburner. on several occasions, feedburner and wordpress had to be forcefully and manually updated and resynced. not only was this not ideal, it was a problem because our feed populates a slideshow on the library home page and front door kiosk; thus the library home page and kiosk were not updating dynamically as intended. further, the calvin t. ryan library did not have a defined social networking strategy, so posting from wordpress to facebook and twitter appeared to lack personality. to solve these issues, the web services librarian chose to build a basic rss 2.0 feed on the library web (lamp) server to replace the wordpress feed and bypass feedburner. maintaining the feed is the responsibility of the web services librarian who manually makes all updates to the feed, while different librarians continue to make entries in wordpress. updates to the feed are not scheduled nor even made daily, but more organically driven by library events, service changes, or by interest pieces pertinent to the library. only those items that the library wanted highlighted on the homepage by the web services librarian, in collaboration with colleagues and the dean, were put into the feed. now links can be directly made to items instead of run through the blog. when items become dated, usually after two weeks, or past the event advertised, they are removed from the feed. simultaneously, facebook was taken over by the project archivist, while twitter was maintained by the web services librarian. those librarians made an effort to follow related institutions and people and make posts so that items from the feed are interspersed with other posts; this helped to make those social media channels more dynamic while giving them some personality. at any given time, the feed contains at most four items. creating an rss feed is simple. for those familiar with html, creating an rss feed should not be a problem. all that is needed to create one is a text editor and some server space. we used vi on our server to create the feed. developers can also utilize software such as dreamweaver to create rss feeds, but if an institution does not have a license for that software, individuals can easily create an rss feed utilizing notepad or any other free text editor. the server can be anywhere and the needed disk space is miniscule. if the server space is off-site, developers will need the ability to ftp the xml file as well. rss is an xml format essentially made up of three parts. the first part is the top level xml declaration. <?xml version=”1.0” encoding=”utf-8”?> although the encoding is optional, it is recommended. the next line is the rss declaration that identifies the document as an rss 2.0 document. <rss version="2.0"> the last part is the <channel> element used to describe the rss feed and contains the other elements in the rss feed. the <channel> element has three required parts – <title>, <link>, and <description>. <title>calvin t. ryan library rss 2.0</title> <link>http://library.unk.edu/rss.xml</link> <description>news and information from the calvin t. ryan library at the university of nebraska kearney</description> the <channel> element may have one or more elements. similarly, each <item> element contains three required parts – <title>, <link>, and <description>. <title>on display at the library</title> <link>https://inbriefonline.wordpress.com/2015/07/09/mona-freda-spaulding/</link> <description>several sketches and painting created by spaulding between 1940 and 1960 make up a collection currently on display at the calvin t. ryan library courtesy of the museum of nebraska art.</description> all elements in xml must have closing tags; thus the last two pieces are closing </channel> and </rss> tags. the following is a complete example of an rss 2.0 feed. <?xml version="1.0" encoding="utf-8"?> <rss version="2.0"> <channel> <title>calvin t. ryan library rss 2.0</title> <link>http://library.unk.edu/rss.xml</link> <description>news and information from the calvin t. ryan library at the university of nebraska kearney</description> <item> <title>on display at the library</title> <link>https://inbriefonline.wordpress.com/2015/07/09/mona-freda-spaulding/</link> <description>several sketches and paintings created by freda spaulding between 1940 and 1960 make up a collection of art currently on display at the calvin t. ryan library courtesy of the museum of nebraska art.</description> <guid>https://inbriefonline.wordpress.com/2015/07/09/mona-freda-spaulding/</guid> <pubdate>thu, 09 jul 2015 16:32:30 gmt</pubdate> </item> </channel> </rss> when coding the rss feed is complete, it should be saved as an .xml file. if not already on a server, ftp it to a server and check for validation in order to ensure it is a well formed rss feed. a number of free validation services exist including the w3c feed validation service, rss advisory board, or feed validator. this is an important step to insure that the rss feed conforms to existing rss specifications and standards. feeds that are well formed will show up better in social media channels such as twitter and facebook. numerous tutorials, videos and instructions for creating rss feeds are available on the web. for novice users the w3c schools introduction to rss http://www.w3schools.com/webservices/rss_intro.asp is a good guide. advanced users will appreciate rss 2.0 at harvard law – http://cyber.law.harvard.edu/rss/rss.html – which gives more information about the rss 2.0 specification including optional tags. only the <title>, <link>, and <description> tags are required. other tags, such as the <pubdate> can be beneficial depending on need. the <pubdate>, for instance, used for indicating the date and time when an item should be published, is also useful for ensuring items in the feed do not linger and become dated. a lot of information can be found about turning your facebook or twitter into an rss feed, but not how to follow a rss feed. unfortunately, following rss feeds in twitter and facebook is not a straightforward task. when an item is created in the rss feed, automating the process of updating separate social media channels will save time. third party tools, such as twitterfeed, ifttt, and dlvr.it are perfect for this kind of automation. we chose dlvr.it because it is free, contains a lot of functionality, and supports both facebook and twitter. for example, dlvr.it has filter options that allow users to filter on items other than content. statistics on popular posts, clicks on posts, etc., are provided as well as support for integration with google analytics. users can schedule posts to update social media on certain times and days and can even embed a dlvr.it widget on the library web site that will show visitors popular posts. however, dlvr.it is not as intuitive to set up as creating a trigger in ifttt, depending on how you have set up your library facebook instance. the calvin t. ryan library facebook is set up as a page and not with a specific library account. individuals assigned to edit or administer facebook have to use their personal facebook instance. as a result, dlvr.it will want to administer all of an individual’s pages, which may not be preferable to some, although dlvr.it will only send updates to the page it is assigned to. figure 1. dlvr.it administration view. in addition to posting on facebook and twitter, the rss feed updates a slideshow on the library home page. thanks to php code from jason clark at montana state university, the rss feed is parsed into an html list. <?php //request items from rss 2.0 feed $rawfeed = 'http://library.unk.edu/rss.xml'; //load feed as array object to be parsed and traversed $xml = simplexml_load_file($rawfeed); //determine number of items to display and set limit $number = count($xml->channel->item); if ($number >= 4) { $number = 4; } //loop through xml data and print as html if (!empty($xml->channel->item)) { foreach ($xml->channel as $feed) { //loop through items for display; number of items determined by $number variable above for ($i=0; $i<$number; $i++) { echo '<li><a href="'.$feed->item[$i]->link.'"><img alt="'.html_entity_decode(substr($feed->item[$i]->title, 0, 85)).'" src="./meta/img/ctr'.($i+1).'.jpg" /></a>'; echo '<div class="slideshow-caption"><p>'.html_entity_decode(substr($feed->item[$i]->title, 0, 85)).'&nbsp;&nbsp;<a href="'.$feed->item[$i]->link.'">read more</a></p></div></li>'."\n"; } } } else { echo '<li><a href="http://inbriefonline.wordpress.com/"><img alt="calvin t. ryan libary exteriorj" src="./meta/img/librarybuilding.jpg"></a>'; echo '<div class="slideshow-caption"><p>please visit the library\'s offical blog <a id="inbrief" href="http://inbriefonline.wordpress.com/">in brief online</a></p></div></li>'."\n"; } ?> the above code essentially produces the following html <noscript><div id="slideshow-wrap"></div></noscript> <ul id="slideshow"> <li><a href="link here"><img alt="" src=" image source " /></a> <div class="slideshow-caption"><p> text for caption <a href=" repeat link ">read more</a></p></div> </li> <li><a href="link here"><img alt="" src=" image source " /></a> <div class="slideshow-caption"><p> text for caption <a href=" repeat link ">read more</a></p></div> </li> <li><a href="link here"><img alt="" src=" image source " /></a> <div class="slideshow-caption"><p> text for caption <a href=" repeat link ">read more</a></p></div> </li> </ul> a jquery script available at github – https://github.com/sutherlandmj/jquery-slideshow – coupled with an image, produces the slideshow effect on the home page. figure 2. library web site slide show. the goal of this project is to publish once, disseminate widely, and provide more flexibility in doing so. the custom rss feed has solved the issues of running everything through our wordpress blog, reduced the amount of time devoted to social media, while at the same time keeping our home page and social media channels updated and in sync. updates occur in real-time as the feed is saved. for our library and web services this is a viable solution. for larger libraries with committees or groups that manage social media, this could be a piece of a larger strategy for managing and contributing to social media channels. using the method described, only those with access to the web server have the ability to update the rss feed. if it is possible, future development would allow multiple contributors to update the feed and allow for better scheduling other than through dlvr.it. another consideration to get more traffic to our site and more widely disseminate the feed is to publish it in feed directories, such as rssfeeds.org, many of which are free. a listing of rss directories can be found at dmoz.org. references johnson, c., & burclaff, n. 2013. making social media meaningful: connecting missions and policies. in imagine, innovate, inspire. indianapolis, in: american library association. p. 399-405. about the author michael is currently the web architect and manager for the university libraries at virginia polytechnic institute and state university and former web services librarian at the university of nebraska kearney calvin t. ryan library. he has been a web developer in academic libraries for over seven years. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – mining data from isi web of science® reports mission editorial committee process and structure code4lib issue 4, 2008-09-22 mining data from isi web of science® reports journal citation data is valuable as a selection tool for adding new journals as well as for discontinuing subscriptions that are no longer cost-effective. this article presents and discusses an example of data extraction from a typical isi web of science report. the strategy was developed following a review of the data relationships and embedded data output format. while perl was used in the example, the method described can be implemented with most programming/scripting languages. the example demonstrates also that citation-based studies and reports can be based on large sets of extracted data rather than the typical, small samples. the value of the data is discussed using a actual decision-making scenario. by alfred kraemer information about the journal articles that an institution’s researchers have cited in their own articles can provide valuable data on that institution’s publication record, on the journal usage by researchers as reflected in their publication, and on trends and changes. the one common denominator in nearly all published citation studies appears to be the difficulty of collecting and evaluating a sufficiently large set of citation data. this article uses an approach that differs from the methods the author found when reviewing a large number of articles on citation analysis. in the majority of articles the data was described as manually compiled from electronic sources [1, 2]. the impetus for the method described in this article was the need to determine the citation use for a specific journal, mechanisms of development. to achieve this, the first step was to develop a large data set of references cited by authors at my institution. it was then possible to mine this data set for cited references specific to the target journal. as a result, the set of data not only answered the initial question about the usage of one journal but formed the basis for other citation-based reports. the example discussed here highlights several common characteristics of ad-hoc data extraction strategies: a good understanding of the quality, scope, and structure of the source data is essential. familiarity with basic strategies of parsing data and selectively extracting nested data. in the example described below regular expressions were not needed, however simple array functions for disassembling the data structure of the exported isi web of science information were critical. a clear plan to make sure specific data relationships are preserved in the data output. extracted data sets should provide for data elements that would allow for quick trouble-shooting of the output data. for that purpose the isi number was included in the output file. subsequent data evaluation and presentation is usually performed more efficiently in common spreadsheet or database programs. the approach presented in this article is not very technically challenging. nevertheless, a review of the many articles on citation analysis did not yield any information on similar methods to selectively extract citation information from the isi citation databases from large exported data sets. impetus for the data evaluation several faculty members at the medical college of wisconsin (mcw) expressed a critical need for the journal mechanisms of development and suggested that the library purchase a subscription. the high price and the fact that there had not been earlier indications of the importance of the journal led the library to explore ways to determine the need to add a subscription as well as alternatives to a subscription. it was felt—and the faculty library committee agreed—that for this basic science journal the use in locally published articles would be a good indicator of whether or not a subscription should be purchased. the isi web of science provided the source data. obtaining and examining the source data the source data for this project were collected using the following 2 steps: performing a search in the isi web of science using a search statement that had been tested previously to ensure the correct set of data would be retrieved. the search criteria in the example include affiliated teaching hospitals. the search criteria use the address field in the isi web of science. the institution name(s) were verified using the institution name look-up option in this database. figure 1: search criteria for the source data [view full-size image] selecting the appropriate output/export option. the perl script used later in this process expects a particular sequence and format of output fields. below are the settings for the ‘output records’ function in the web of science that were chosen. as the number of output records for a project such as the one discussed here will generally exceed the output limit of 500 records, the output procedure will need to be repeated as many times as needed to complete exporting the entire result set. it is not necessary to redo the search; just the export function will have to be run for each successive set of 500 records. this did not lengthen the time for the data collection significantly. figure 2: output options [view full-size image] figure 3: progress window from the isi web of science [view full-size image] the number of records that included the specified institution name(s) varied from year to year between 1,300 to 1,500. from each annual set of exported data about 40,000 cited references were extracted and reported in the script-generated output file. examining the source data the example below shows the nested structure of the export format for a full isi web of science citation. the option to include cited references was selected before retrieving the tab-delimited isi file output format. example: (cited references for this article are in red references to mech develop further highlighted in blue.) j watt, aj; battle, ma; li, jx; duncan, sa gata4 is essential for formation of the proepicardium and regulates cardiogenesis proceedings of the national academy of sciences of the united states of america article med coll wisconsin, dept cell biol neurobiol & anat, milwaukee, wi 53202 usa duncan, sa, med coll wisconsin, dept cell biol neurobiol & anat, milwaukee, wi 53202 usa duncans@mcw.edu bruneau bg, 2002, circ res, v90, p509; chen thp, 2002, dev biol, v250, p198; cripps rm, 2002, dev biol, v246, p14; crispino jd, 2001, gene dev, v15, p839; duncan sa, 1997, development, v124, p279; dunwoodie sl, 1998, mech develop, v72, p27; eisenberg lm, 1995, circ res, v77, p1; garg v, 2003, nature, v424, p443; gassmann m, 1995, nature, v378, p390; gittenbergerdegroot ac, 2000, circ res, v87, p969; grepin c, 1997, development, v124, p2387; jiang ym, 1996, dev biol, v174, p258; kisanuki yy, 2001, dev biol, v230, p230; kubalak sw, 1999, heart dev, p209; kuo ct, 1997, gene dev, v11, p1048; kwee l, 1995, development, v121, p489; lee kf, 1995, nature, v378, p394; lin q, 1997, science, v276, p1404; lough j, 2000, dev dynam, v217, p327; lyons i, 1995, gene dev, v9, p1654; mcfadden dg, 2000, development, v127, p5331; meyer d, 1995, nature, v378, p386; mikawa t, 1999, heart dev, p19; mjaatvedt ch, 1999, heart dev, p159; molkentin jd, 1997, gene dev, v11, p1061; molkentin jd, 2000, j biol chem, v275, p38949; moore aw, 1998, mech develop, v79, p169; nagy a, 1993, gene targeting pract, p147; narita n, 1997, dev biol, v189, p270; narita n, 1997, development, v124, p3755; niederreither k, 2001, development, v128, p1019; parmacek ms, 1999, heart dev, p291; parviz f, 2003, nat genet, v34, p292; perezpomares jm, 2002, dev biol, v247, p307; romeih m, 2003, dev dynam, v228, p697; rossi jm, 2001, gene dev, v15, p1998; sedmera d, 1996, bioessays, v18, p607; sengbusch jk, 2002, j cell biol, v157, p873; sepulveda jl, 2002, j biol chem, v277, p25775; soudais c, 1995, development, v121, p3877; stennard fa, 2003, dev biol, v262, p206; stuckmann i, 2003, dev biol, v255, p334; viragh s, 1973, j ultrastruct res, v42, p1; winnier g, 1995, gene dev, v9, p2105; yang jt, 1995, development, v121, p549 0027-8424 aug 24 2004 101 34 12573 12578 isi:000223596200033 within the above tab-delimited export of isi data for an article, the cited references—in red for emphasis—are delimited by semicolons. each cited reference in turn has elements that are delimited by commas. author, year, journal, volume, page bruneau bg, 2002, circ res, v90, p509; chen thp, 2002, dev biol, v250, p198; cripps rm, 2002, dev biol, v246, p14; crispino jd, 2001, gene dev, v15, p839; duncan sa, 1997, development, v124, p279; dunwoodie sl, 1998, mech develop, v72, p27; eisenberg lm, 1995, circ res, v77, p1; (…) the pattern consists of several layers of nested arrays: a nest of cited reference elements—delimited by commas, inside a nest of cited references—delimited by semicolons, inside a nest of data fields for an individual citation—delimited by tabs, inside a list of exported journal article citations from isi—delimited by end-of line/return characters. extracting nested data elements with a perl script virtually any programming or scripting language includes functions to parse input fields in a systematic manner and has methods to handle arrays. perl was chosen for convenience, and because it is widely used for extracting and transforming data. the script’s runtime for a typical annual set of data—about 1,500 records—was clocked at than 10 seconds on a pc with a 3.0 ghz pentium 4. # process the section below the script (below the __data__ token) # record by record # file handle for output to a file: $outfile="./isioutrefs2004.txt"; open (output, ">$outfile"); while ( <data> ) { # remove trailing record separator if found: chomp; # split line by tab delimiter and put the result in an array: @recordline=split (/\t/,$_); $fullrec=$_ ; chomp (@recordline[7]); while ( <data> ) { # remove trailing record separator if found: chomp; # split line by tab delimiter and put the result in an array: @recordline=split (/\t/,$_); $fullrec=$_ ; # save the isinumber -the 37th element under a separate variable $isinumber=@recordline[36]; $publishedinjournal=@recordline[4]; chomp (@recordline[7]); foreach (@recordline[14]) { $count++; @citedrefs=split /;/, $_; foreach (@citedrefs) { # do for each cited reference $countrefs++; @onecite=split /,/,$_; # add a line with running number, pubyear, #title, and isinumber to the output file print output "$countrefs\t$onecite[1]\t$onecite[2]\t$isinumber\n"; # this line just echoes some fields to the screen #as the script runs. # it may be commented out. print "$countrefs\t$onecite[1]\t$onecite[2]\t$isinumber\n"; } } } } # the next two lines just echo some fields to the screen as the script finishes. # they may be commented out. print "total refs: $countrefs\n"; print "total number of articles authored/co-authored by mcw: $count"; # the data to be processed begins below the __data__ token. __data__ pt au ca ti so se la dt de id ab c1 rp em cr nr tc pu pi pa sn j9 ji pd py vl is pn su si bp ep ar pg sc ga ut j watt, aj; battle, ma; li, jx; duncan, sa gata4 is essential for formation of the proepicardium and regulates cardiogenesis proceedings of the national academy of sciences of the united states of america article med coll wisconsin, dept cell biol neurobiol & anat, milwaukee, wi 53202 usa duncan, sa, med coll wisconsin, dept cell biol neurobiol & anat, milwaukee, wi 53202 usa duncans@mcw.edu bruneau bg, 2002, circ res, v90, p509; chen thp, 2002, dev biol, v250, p198; cripps rm, 2002, dev biol, v246, p14; crispino jd, 2001, gene dev, v15, p839; duncan sa, 1997, development, v124, p279; dunwoodie sl, 1998, mech develop, v72, p27; eisenberg lm, 1995, circ res, v77, p1; garg v, 2003, nature, v424, p443; gassmann m, 1995, nature, v378, p390; gittenbergerdegroot ac, 2000, circ res, v87, p969; grepin c, 1997, development, v124, p2387; jiang ym, 1996, dev biol, v174, p258; kisanuki yy, 2001, dev biol, v230, p230; kubalak sw, 1999, heart dev, p209; kuo ct, 1997, gene dev, v11, p1048; kwee l, 1995, development, v121, p489; lee kf, 1995, nature, v378, p394; lin q, 1997, science, v276, p1404; lough j, 2000, dev dynam, v217, p327; lyons i, 1995, gene dev, v9, p1654; mcfadden dg, 2000, development, v127, p5331; meyer d, 1995, nature, v378, p386; mikawa t, 1999, heart dev, p19; mjaatvedt ch, 1999, heart dev, p159; molkentin jd, 1997, gene dev, v11, p1061; molkentin jd, 2000, j biol chem, v275, p38949; moore aw, 1998, mech develop, v79, p169; nagy a, 1993, gene targeting pract, p147; narita n, 1997, dev biol, v189, p270; narita n, 1997, development, v124, p3755; niederreither k, 2001, development, v128, p1019; parmacek ms, 1999, heart dev, p291; parviz f, 2003, nat genet, v34, p292; perezpomares jm, 2002, dev biol, v247, p307; romeih m, 2003, dev dynam, v228, p697; rossi jm, 2001, gene dev, v15, p1998; sedmera d, 1996, bioessays, v18, p607; sengbusch jk, 2002, j cell biol, v157, p873; sepulveda jl, 2002, j biol chem, v277, p25775; soudais c, 1995, development, v121, p3877; stennard fa, 2003, dev biol, v262, p206; stuckmann i, 2003, dev biol, v255, p334; viragh s, 1973, j ultrastruct res, v42, p1; winnier g, 1995, gene dev, v9, p2105; yang jt, 1995, development, v121, p549 0027-8424 aug 24 2004 101 34 12573 12578 isi:000223596200033 (…) producing reports from the data with ms access the delimited output file created by running the perl script can be directly imported into a database such as ms access. basic manuals and the ‘help’ option ms access provide information about the procedure. figure 4 below shows an excerpt from the table in ms access created after importing the delimited text file that was previously produced by the perl script. figure 4: data imported into ms access from the output of the perl script [view full-size image] to obtain aggregate data, especially counts, the query shown in figure 5 was composed. right-clicking on any column in the query grid shown in figure 5 will pop up the window that shows ‘totals’ at the top of the list of choices. after selecting ‘totals’ the correct option for the ‘total’ option has to be selected for each column, depending whether the column should be a ‘group by’ column or contain a specific aggregate function, e.g. count. please note that it is often useful or required to use some columns twice in an aggregate query. the example below uses the ‘title’ column from the imported table twice: first for displaying the actual journal title in the query output, second for showing the count for the number of times the journal title was found. figure 5: setting query options in ms access [view full-size image] a portion of the resulting query output is shown in figure 6. for the journal, mechanisms of development, seven cited references were found in articles authored or co-authored by mcw authors in 2004. information gathered in the same manner for later years did not show a significant variance in the number from year to year. during the discussion of this result, the question was raised whether there would be a significant difference in the number of articles that cite a specific journal—perhaps multiple times—versus the overall number of cited references. since the isi number was included in the output file of the perl script, this number which references a specific article was used to obtain that count. figure 6: query result for mechanisms of development [view full-size image] for mechanisms of development, it was found that the seven cited references were found in five articles. (see figure 7.) figure 7: query variation: the number of articles authored/co-authored by mcw authors in 2004 that cited mechanisms of development [view full-size image] the answer to the question which journals were cited the most by mcw authors in 2004? was found by sorting the counts. the top of that listing is shown in figure 8. figure 8: top of journal listing sorted by citations in 2004 articles [view full-size image] caveats and cautions the cited reference format in isi is less regular for references to book chapters, etc. this requires some review of sorted output lists. some journal article references, e.g. those for articles in supplements, differ slightly in the title field. example: heck c, 2002, neurology s4, v59, s31. this requires caution when counting/grouping references. the scope of the isi databases may not be a suitable match to capture the publishing activities of an institution. there are many uses for journal articles that are important but do not result in the publishing of new research. outcome / results when the data was presented to the faculty library committee, there was agreement that a subscription would not be cost-effective. as an alternative, the library began its pre-paid article service. during the past two years, the request levels for this journal have never exceeded 20% of the subscription price. the isi data has become a part of the evaluation of requests for new journals. the trends shown in reports from successive years indicate that dramatic changes from year to year are rare for most journals but noticeable for journals that are making a quick impact in their subject area. it has become evident that the data is valuable as a selection tool for adding new journals as well as for discontinuing subscriptions that are no longer cost-effective, helping keep the selection of journals in alignment with patron needs. additional uses for the collected data soon after the immediate goal of the data evaluation was met it was realized that the data collected could also yield other journal use metrics. additional evaluations of the collected data were possible mainly because a comprehensive set of data had been gathered at the start. some of the examples of further uses of the data are: establishing change trends. a comparison of the journal titles cited by mcw authors with current subscriptions/licenses showed that some heavily-used journals were not among the subscribed journals. when the data for several successive years were included in the comparison, several new journals showed a fast increasing trend line. this additional information—together with faculty requests—was used to adapt the journal subscriptions to the increased relevance of those journals. trend information from several successive years is critical for supporting subscription changes. one key consideration in the decision to add a new journal subscription was the range of years of the cited references. if a currently not subscribed journal is cited increasingly over the recent years and the cited references from the journal are also from recent years the likely cost/benefit ratio of a subscription is more advantageous. this pattern was observed for several—but not all—nature reviews journals. lists of journals that affiliated researcher either publish in or use as a source of cited references. a sorted listing of journals that researchers primarily rely on for cited references can be produced easily from the collected data. with a small modification of the perl script, a sorted list of the journals where an institution’s researchers publish can be created. comparison of counts of cited references vs. articles. an initial criticism of the output data from the perl script was that it showed the number of citations from a specific journals used by mcw authors in a particular year but not the number of mcw authored/co-authored articles in which the journal was cited. the concern was that relatively small number of articles could account for a much larger number of cited references to a journal if the articles included many references to articles from the same journal. summary the lack of data extraction and integration methods has impeded the analysis of the scientific communication at a time when the need for a better picture of scientific communication is greater than ever. all participants in the scholarly communication contex—researchers, publishers, academic and other research institutions, etc.—need to better understand measures of quality and effectiveness in scholarly communication. analyzing the components of citations/cited references continues to be a foundation for determining the key journals for specific areas of research and for author/institution impact [3]. the method described for extracting data from large collections of citation data shows that a basic but careful evaluation of the source citation data can point to successful data extraction methods that can be quite simple. since most citation analyses use sources that also provide their data in either a delimited or—even better—in xml format, there is no reason why citation studies have to be limited to small, manually collected samples. the author has not yet reviewed other sources for article citation data such as scopus and google scholar. since the underlying relationships of scholarly communication are the same, the data elements in those sources should be similar. the method and the basic perl script presented in this article may be used and modified as needed. since output formats of the data source—in this instance, the isi web of science—are subject to change, the script may need to be adapted. it is also important to keep in mind that database vendors may restrict specific uses of their data. the license agreement with the database vendor should be consulted. notes 1. glã¤nzel, w. and moed, h. f. 2002. journal impact measures in bibliometric research. scientometrics 53(3): 171-194. coins 2. van raan, a. f. j. 2004. measuring science. in “handbook of quantitative science and technology research. the use of publication and patent statistics in studies of s&t systems” (eds moed, h. f., glã¤nzel, w. and schmoch, u.), kluwer, dordrecht, the netherlands. pp. 19-50. oclc: 56695465. 3. meho l. i. jan. 2007. the rise and rise of citation analysis. physics world. 20(1): 32-36. also: arxiv:physics/0701012v1. coins about the author alfred kraemer is project manager for academic it in the office of research at the medical college of wisconsin in milwaukee. until july 2007 he was assistant library director at the same institution. author email: alfred.kraemer@gmail.com tags: citation analysis, collection development, microsoft access, perl, web of science subscribe to comments: for this article | for all articles 2 responses to "mining data from isi web of science® reports" please leave a response below: smartiepants, 2008-09-22 bravo alfred, bravo. tomkeays, 2010-05-23 isi web of science has changed their output format since the article was published, adding 14 new columns and requiring, therefore, that the perl script be modified slightly. in each recordline array: instances of @recordline[4], which reference the so (journal source) column, need to be changed to @recordline[8]. instances of @recordline[7], which reference the dt (document type) column, need to be changed to @recordline[11]. instances of @recordline[14], which reference the cr (cited reference) column, need to be changed to @recordline[25]. instances of @recordline[36], which reference the ut (isi number) column, need to be changed to @recordline[50]. the full revised script is: # process the section below the script (below the __data__ token) # record by record # file handle for output to a file: $outfile="./isioutrefs2004.txt"; open (output, ">$outfile"); while ( <data> ) { # remove trailing record separator if found: chomp; # split line by tab delimiter and put the result in an array: @recordline=split (/\t/,$_); $fullrec=$_ ; chomp (@recordline[11]); while ( <data> ) { # remove trailing record separator if found: chomp; # split line by tab delimiter and put the result in an array: @recordline=split (/\t/,$_); $fullrec=$_ ; # save the isinumber -the 37th element under a separate variable $isinumber=@recordline[50]; $publishedinjournal=@recordline[8]; chomp (@recordline[11]); foreach (@recordline[25]) { $count++; @citedrefs=split /;/, $_; foreach (@citedrefs) { # do for each cited reference $countrefs++; @onecite=split /,/,$_; # add a line with running number, pubyear, #title, and isinumber to the output file print output "$countrefs\t$onecite[1]\t$onecite[2]\t$isinumber\n"; # this line just echoes some fields to the screen #as the script runs. # it may be commented out. print "$countrefs\t$onecite[1]\t$onecite[2]\t$isinumber\n"; } } } } # the next two lines just echo some fields to the screen as the script finishes. # they may be commented out. print "total refs: $countrefs\n"; print "total number of articles authored/co-authored by mcw: $count"; # the data to be processed begins below the __data__ token. __data__ leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – gamifying information literacy: using unity and github to collaborate on a video game for the library mission editorial committee process and structure code4lib issue 60, 2025-04-14 gamifying information literacy: using unity and github to collaborate on a video game for the library gamification, as a way to engage students in the library, has been a topic explored by librarians for many years. in this article, two librarians at a small rural academic library describe their year-long collaboration with students from a game design program to create a single-player pixel-art video game designed to teach information literacy skills asynchronously. the project was accomplished using the game engine unity and utilizing github for project management. outlined are the project’s inspiration, management, team structure, and outcomes. not only did the project serve to instruct, but it was also meant to test the campus’ appetite for digital scholarship projects. while the project ended with mixed results, it is presented here as an example of how innovation can grow a campus’ digital presence, even in resistant libraries. by halie kerns and leah fitzgerald gamification in libraries across a student’s academic career, library instruction is unique. unlike your average teaching faculty, the majority of the interactions an instructional librarian has with students are one-shot sessions, lasting a single class period. a librarian will focus on basic information literacy skills so a student can get through the research aspect of their paper. this instruction often includes instructions on how to access their library’s catalog online and how to search in a database. in our current media landscape, a large component of this instruction includes lessons on information evaluation and what makes a good source. properly teaching information literacy concepts in a single, hour-long session proves to be difficult due to the minimal time spent with the students [1]. while each session normally concludes with an assessment portion, due to activities needing to be completed in a short time and typically not repeated, students often have low recall of the skills they just learned during the session. libraries and higher education in general, have started implementing game-based learning and gamification into their classes, especially in the post-covid 19 environment due to the increase in remote learning [2][3]. gamification in remote learning motivates students in an environment where they struggle to pay attention and engage with coursework [4]. gamification brings gaming elements to non-gaming concepts, such as information and media literacy [5][6][7]. using gamification during a lecture allows students to participate in active learning in a low-risk environment in the classroom while increasing their engagement due to the motivation provided by the gaming elements [8]. gamification can have a high cost associated with it when developing your own game making the concept less accessible for many librarians [9]. creating a game focused on the known abilities of the students in an institution allows the game to be tailored to what your students need in terms of information literacy skills. it also allows the librarian to make sure the game fits their sessions and that it can be repeated after a session by students to increase their skills and build upon what they have learned so they can succeed during their time in higher education and beyond. institution introduction the development of this video game project occurred while we were employed as suny canton’s access services and instruction and outreach librarians. suny canton is a technical college within new york’s public university system, the state university of new york. located in an economically disadvantaged, rural area of northern new york, outside of ottawa, canada, the college currently has a full-time enrollment of around 3,000 students [10]. half of the student body takes their classes fully online due to many of the majors having the option of being completed 100% remotely [11]. almost all of the majors offered by suny canton are career-driven, with the most popular being nursing, cybersecurity, and engineering. many of the majors are highly selective, including the popular video game design program, which focuses on developing games on all platforms through the use of the most current technologies [12]. during the initial development stage of the design process we, as librarians, had been employed by the college library for less than a year. through our observations during that time, we noted a general lack of ways for students to create, share, and develop their work in a digital format despite having majors that would greatly benefit, such as game design, graphic design, and computer science programs. it was also observed that despite having high-tech majors, the campus itself is very low tech, containing limited resources, staff, and infrastructure to move forward with digital projects. during this time, suny canton was in the middle of promoting the second semester of the new presidential internship initiative, which was experiencing a slow start due to a lack of projects for students to work on [13]. both of these factors allowed us to move forward on developing a video game project and testing to see how digital projects would fare on the campus. the vision and evolution of the project the video game project came to life due to a blend of our professional interests in digital scholarship and information literacy, with a combined goal of bringing them together to benefit the campus. during the start of the project, the instruction and outreach librarian taught one-shot sessions with various classes across campus and a credited first year experience class. the access services librarian was working to streamline the libraries’ system workflows and digital presence. both librarians were also teaching sections of an asynchronous, credited information literacy course. while the library had materials and lesson plans to instruct in-person and synchronous information literacy sessions fairly easily, they were looking to expand the potential for asynchronous or online instruction. in 2021, the suny board of trustees updated the general education (ge) framework, which was implemented for all first-time students in the fall 2023 semester [14]. the new ge requirements included a required core competency for information literacy. students were now required to learn how to use various tools to locate information, to evaluate the information they find properly, and to understand the ethics behind the information literacy cycle. aware of the new requirements and the online student population, we, the librarians, believed the development of a digital project to be used by faculty and students to gain information literacy skills would be a net positive for the campus. the original plan for the development of the video game involved the librarians developing the storyline and focusing on the information literacy skills to include in the game. the presidential internship was initially used to attract two video game design students to work with us to develop the coding for the game. this initiative, funded through the president’s office, allowed different departments on campus to hire students to complete projects. the initiative would be used to pay the students for the work and allow them an extra opportunity to create a working product for them to use in their portfolios. due to the recent launch of the program as a new initiative, there was a lack of projects from the campus faculty and staff. as such we were allowed to add two additional students to our team, bringing us to a team of four for the first semester. over the first semester, we understood the amount of work and time necessary for proper development of a game, especially about worldbuilding storyline, and further evolved our process and team. with the presidential internship funding for the students, we were able to achieve the goal of having a fully functional game to present to the campus by the end of the second semester. due to the success our interns achieved in the first semester, the coordinators of the internship initiative allowed us to request any amount of students we needed to complete the video game. we hired five students for the second semester: three coders, one artist, and one writer. overall, this meant that we received $9,000 worth of funding for the students over the academic year. with a full development team, the interns took over the mechanics of creating the game. the roles were assigned and distributed by the students themselves, with one student becoming the unofficial leader for the students. we all met in person three times over the semester to see updates on the game, offer guidance on data management, and make sure the information literacy elements were playing out correctly during gameplay. by the end of the semester, our team developed a playable, single-play, pixel-art, story-driven video game over the course of two semesters, complete with gamified information literacy concepts and tools where students would be able to play remotely from home. unity and team roles original plans for the game, before recruiting interns, were based on the librarians’ limited coding and gaming experience. an early prototype for the game was built in twine by librarians. however, once the game design students were on board, the sophistication level of the game rose. the choice of platform was left up to the interns who chose to work with unity, which they had prior experience with as the primary engine of the game design program at suny canton. unity is a game engine that supports the creation of various video game formats (including desktop, console, ar/vr, and mobile). it is well known for its accessibility for new developers and popularity in indie game design. the interns worked off unity version 2020.3.38f1. the primary coding language of the game was done in c# and shaderlab which dominate the unity engine and game development in general. since the interns came from the same program, they were all familiar with navigating unity individually and as a group. although the unity line contains many different products, the engine the interns used was a free version that included all the capabilities needed for the game. while unity is not technically open source, it supports an open ecosystem and is easily and freely shared. as long as the files were managed properly, future cohorts of students could easily access and build upon the original files. figure 1. unity ui example. to handle a large and long-term game project, interns were assigned different roles and parts of the game work. these roles reflected industry standards that they had learned in their program. the three roles were game designer, storyteller, and coder. the coders were then further broken down by game mechanics which could be summarized as gameplay and looping, scoring and game communication, and game lore and built in wiki. over the year, we worked with six different interns. the same three coders worked both semesters, so their roles were well-defined and they were the most familiar with the game by the end. they built a prototype of a single “day” of gameplay for the first semester. the sample code below breaks down an example of their work, and how they created the scoring method [figure b]. users select whether to sort information as reliable or not and are scored accordingly for the “day”. by january they were able to build upon that experience to flesh out the rest of the game. the tasks and outcomes within their roles broke down as follows in figure b, including improvements for future work in the game: table 1. table of roles. gameplay and looping scoring and game communication game lore/wiki tasks create the mechanics of the game including how the player moves through the game, the choices they have, and game loop and timeline of play. create the scoring that runs in the background as you gain and lose points during tasks. create all the in-game communications with the player. creating an in-game encyclopedia for players to measure information accuracy against. outcomes created playable “days” that each contained a cumulative focus on a letter in the craap test. each day contained a thematic story and mystery for the player to try to solve with the information supplied. the loop took in account the ability of the player to successfully identify the “reliable” and “unreliable” information through the gameplay. scoring based on whether you were able to identify good or bad information using the craap test. at the end of each “day” within the game, you got a pass or fail score. at the end of the “week” your outcome was based on your cumulative score. communications in the game were created to move the gameplay forward, including different emails, social media posts, and other sources of information. a wiki page was created with information about characters that the player interacts with, places in the world of the game, and information about social aspects of the world. the page can be accessed at any time throughout the game. future fixes make the play more dynamic and less repetitive. create scoring and outcomes that better match a cohesive story of the game. make the wiki more responsive to the gameplay. using system.collections; using system.collections.generic; using unityengine; public class answerprompt : monobehaviour { dictionary<int,int> answers = new dictionary<int,int>(); //<day, correct answer #> // start is called before the first frame update void start() { answers.add(0, 1); //day 1, correct answer: choice 2 answers.add(1, 1); //day 2, correct answer: choice 2 answers.add(2, 1); //day 3, correct answer: choice 2 answers.add(3, 1); //day 4, correct answer: choice 2 } //get the button that is pressed and check whether it's correct. public void getbutton(int num) { if(answers[gamemanager.currentday] == num) //checks if we selected the { gamemanager.playerscore += 100; // add 100 to player score } //correct option, returning true or false findobjectoftype<main>().endday(answers[gamemanager.currentday] == num); gameobject.setactive(false); } } figure 2. example code from the game that answers prompts within the gameplay. our art designer in the first semester did not return, so we recruited a new designer in the second semester. our second art designer created the feel of the game and the aesthetics of the game play including the interface, graphics, fonts, and in-game communications. the game was meant to have a retro-dystopic feel so pixel art was a good match for the design. our art designer used aseprite to create the art for the game, a sprite animator and pixel art tool. the game was from the point view of the player so the interface replicated a computer and various communications for which the designer created art, often based on existing real life aesthetics. the player interacts with characters and learns about them secondhand, for which little animations were created. the art designer also created a font for the game’s text to create a cohesive feel. all of this work was then added to unity to be applied to the gameplay framework of the coders. figure 3. art in unity. finally, our last addition to the team was a storyteller to create the story of the game. this included building the world of the game, the history and social context of the player, scripts of interactions between the player and characters in the game, and the progression of the story as the player moves through the week (whether successfully or unsuccessfully). originally, the librarians leading the project were responsible for this work, but as mentioned above, it ended up being more time-consuming and complicated than expected, and an “expert” intern was added. they were able to get a better grasp on a cohesive story and world for the game. all five of these roles worked very closely with each other and collaborated throughout the entire project. together in unity, they created a playable game that takes a player about thirty minutes to complete. the game has a distinct aesthetic feel and a very specific narrative that fits both the information literacy goals of the game and the culture of suny canton. project management to successfully build the game, the project needed close and committed management. the goals of which can be broken down into four different categories: communications, version control, data management, and administration. the team (five interns and two librarians) worked closely and communicated often to manage the project together and primarily remotely. communication the team met at the beginning and end of the semester in person. once a semester, each intern met with the two librarians face-to-face to discuss individual progress. with the exception of individual meetings, all communication was done remotely over discord. the interns were required by the internship agreement to work 120 hours for the semester, but the majority of the work was done outside of the librarians’ schedule. however, they were able to check progress and questions on their schedule with discord. version control a github repository was used for the five interns to share the work they were doing, update accordingly, and save the work. all were familiar with it through their program. it also allowed the librarians to check what progress was being made and what updates and commits had been made. the use of version control with github was successful, and at no point in the project was any work lost or overwritten. the repository also allowed for the game files to be open source and available in the repository that the students built for future expansion on the game. data management along with github, and their personal desktops, all data was saved to a shared google folder in accordance with good data management practices. a standard read me file was created to guide users through the github and the google drive. since the interns worked on the game throughout the year, the files in the google drive were the finalized version at the end of each semester, and github was where the version history lived. besides the game files, the raw text files of the story and image files were saved and organized accordingly in google drive. administrative the librarians oversaw all the work and kept track of the progress of the internships. the institution had benchmarks to meet for students to receive their stipend outside of the project’s tasks. librarians communicated with stakeholders across campus as well. this included game design instructors who promoted the internships among their students; the president’s office, which hosted and dispersed funds for the internship; the career center, which did the posting and hiring process, and library leadership, which supported the initiative. at the end of each semester, librarians submitted reports about the project. finally, librarians were responsible for the project’s future, which included hosting, promoting, and archiving the work. successes and failures at the end of the academic year, a playable game was created. in the short term, the game was a success, and digital project possibilities from the library were shared with the campus. the game was presented during the annual research days, where librarians and interns did presentations. the interns won the top award in student research for their work and the game demo drummed up new interest in information literacy instruction with the library. however, long-term implementation of the game in actual information literacy classes failed. ultimately, the campus it department refused to host the game on institutional servers. they lacked the resources to do so. as a campus with little to no digital scholarship projects happening, there was no service to use institutional servers for projects, and they did not want to open up the opportunity for others to think they could ask the same. while it could still be saved on local machines, this half-hearted implementation defeated the purpose of remote students or asynchronous gameplay. it was a very practical lesson for the librarians to understand the entire lifespan of a digital project and what other stakeholders need to be involved from the beginning of an idea. while other options for hosting are available for such games, by the time a final decision was delivered by it, enthusiasm for the project had waned due to other pressing matters in the library. even though the game was never implemented, and both librarians left the institution in the year that followed into more technology-based positions, the game had a positive impact on the library as a place for digital work. one of the biggest legacies has been the libraries’ continual use of the internship program to do digital work. because of the relationship created between the internship stakeholders on campus and the libraries, the libraries continue to employ anywhere from 25-50% of the total internships on campus to work on digital projects. one of the biggest projects has been slowly digitizing the libraries’ archives, which had not been possible prior to this project. overall, the game may not have accomplished the original goals, but new goals were discovered along the way. the interns gained work experience, and librarians created a collaborative digital project that created a buzz, resulting in new interest in library services. in the end, the project did bring digital work to a small, resistant campus community with limited resources. although it may seem small, the project had a positive impact on both the participants and the greater campus, reminding everyone of the possibilities of digital innovation. references [5][8] barber, c. s. (2021). when students are players: toward a theory of student-centric edu-gamification systems. the journal of information and systems in education, 32(1), 53–64. https://jise.org/volume32/n1/jise2021v32n1pp53-64.pdf [2][6][9] contreras-espinosa, r. s., & eguia-gomez, j. l. (2023). evaluating video games as tools for education on fake news and misinformation. computers, 12(9), 188. https://doi.org/10.3390/computers12090188 [1][3][7] helbing, r. r., lapka, s., richdale, k., & hatfield, c. l. (2023). in-person and online escape rooms for individual and team-based learning in health professions library instruction. journal of the medical library association jmla, 110(4), 507–512. https://doi.org/10.5195/jmla.2022.1463 [4] khaldi, a., bouzidi, r., & nader, f. (2023). gamification in e-learning in higher education: a systematic literature review. smart learning environments, 10(10), 1-31. https://slejournal.springeropen.com/articles/10.1186/s40561-023-00227-z#sec1 [14] state university of new york (2023). suny general education framework (suny ge). state university of new york. https://system.suny.edu/academic-affairs/acaproplan/general-education/suny-ge/ [10] suny canton. (n.d.). about canton. suny canton https://www.canton.edu/about/college.html [11] suny canton. (n.d.). suny canton quick facts. suny canton. https://www.canton.edu/research/quickfacts.html [12] suny canton. (n.d.).game design & development – b.s. suny canton. https://www.canton.edu/csoet/game/ [13] suny canton. (n.d.). presidential internship program. suny canton. https://www.canton.edu/internship/presidential/ about the authors halie kerns is the digital scholarship librarian for data science and stem at binghamton university. she offers support for data-driven digital tools for research and pedagogy, including data collection, analysis, visualization, and management. leah fitzgerald is the assistant director at the amsterdam free library where she currently oversees programming, collection development and staff. her current focus is developing spaces within her library to provide access to services and resources to underserved populations within her community. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – data preparation for fairseq and machine-learning using a neural network mission editorial committee process and structure code4lib issue 55, 2023-1-20 data preparation for fairseq and machine-learning using a neural network this article aims to demystify data preparation and machine-learning software for sequence-to-sequence models in the field of computational linguistics. the tools, however, may be used in many different applications. in this article we detail what sequence-to-sequence learning looks like using code and results from different projects: predicting pronunciation in esperanto, predicting the placement of stress in russian, and how open data like wikipron (mined pronunciation data from wiktionary) makes projects like these possible. with scraped data, projects can be started in automatic speech recognition, text-to-speech tasks, and computer-assisted language-learning for under-resourced and under-researched languages. we will explain why and how datasets are split into training, development, and test sets. the article will discuss how to add features (i.e. properties of the target word that may or may not help in prediction). by scaffolding the tasks and using code and results from these projects, it’s our hope that the article will demystify some of the technical jargon and methods. by john schriner introduction there are many tools in the field of natural language processing (nlp) and computational linguistics that: help us to understand language better; find patterns that we cannot perceive; find word collocation (i.e. when words are commonly near other words); improve text-to-speech; perform text summarization; perform information extraction; provide sentiment analysis; and perform machine-translation. some of these tools are the user-friendly web-based voyant tools,[1] the python software platform natural language toolkit (nltk),[2] and praat[3] phonetic software for examining sound. the nlp tool linguistic inquiry and word count (liwc)[4] is a psycholinguistic black-box[5] tool that can provide sentiment analysis, language style matching, and many other metrics using over 100 dimensions of text. liwc has been widely used for decades, is dictionary-based, and does not involve machine learning. although we may not see a lot of conspicuous use of machine-learning in libraries at present, any project in library and information science that uses an input sequence to map to an output sequence could be improved with this technology; indeed, our discovery services and search engines embrace techniques identified in 1995 that can “analyze user queries, identify users’ information needs, and suggest alternatives for search” (chen, 1995, p. 1). moving to the present day, in zhu & lei (2022) we see machine-learning being used in classification of research topics in covid-19 research. they extract noun phrases from an experimental corpus of full text articles indexed in web of science. these noun phrases numbered 19,240 with a minimum frequency of 10 per million words. zhu & lei (2022) identify research topics whose subject matter was increasing; these are labeled hot topics and categorized into larger categories such as biochemical terms, public health measures, symptoms and diseases, etc. their methods are robust and they work with six different classification models, finding that a random forest classifier[6] yields the best results. in a similar vein and apropos to information literacy, sanaullah et al. (2022) offers a systematic review of covid-19 misinformation research involving machine-learning and deep learning. in their review they selected 43 research articles and categorized them into misinformation types: fake news, conspiracy theory, rumor, misleading information, and disinformation (deceptive information, as opposed to inaccurate in the case of misinformation) (sanaullah et al., 2022). after a thorough discussion of methods, this survey finds that deep-learning methods are more efficacious than traditional machine-learning methods. with known datasets, or datasets created from scraped web data, we can use modern machine-learning tools for any number of projects in different subfields of linguistics like phonology (the study of linguistic sound), morphology (the study of words and how they are formed and used together), and even historical linguistics (the study of languages over time, including language families). this paper focuses on sequence-to-sequence models, the conversion of a sequence from one domain into a sequence of another domain. this could be, for example, polish words converted to their pronunciation in the international phonetic alphabet (ipa) format: e.g. osłu ‘donkey’ converted to ɔswu. this model would effectively aid in text-to-speech systems. another example of sequence-to-sequence modeling could be to predict the correct inflection and placement of a stress marker given a word and its part of speech: training a model that when given the russian adjective эйфорически ‘euphorically,’ must successfully place the stress on the middle «и́» as in эйфори́чески. the idea is that we will use 80% of the data to train on, 10% for development with which to choose the best parameters and model, and 10% for the test set. it’s easy to feel overwhelmed with these tools and their architectures. the aim of this paper is to help demystify this particular type of machine-learning with a well-prepared dataset and project goals. importance of open data open data is essential for original research and replication studies.  sparc states that “despite its tremendous importance, today, research data remains largely fragmented—isolated across millions of individual computers, blocked by disparate technical, legal and financial restrictions” (“open data,” n.d.).  to combat this fragmentation, a call for open data would require that research data: “(1) is freely available on the internet, (2) permits any user to download, copy, analyze, re-process, pass to software or use for any other purpose; and (3) is without financial, legal, or technical barriers other than those inseparable from gaining access to the internet itself” (“open data,” n.d.).  open data can be found worldwide in glam labs such as the data foundry[7] at the national library of scotland, and linguistics repositories such as the tromsø repository of language and linguistics (trolling).[8]  the registry of research data repositories[9] indexes nearly 3,000 research data repositories that provide databases, corpora, tools, statistical, and audiovisual data.  with open and well-described data alongside open access papers, our research lives on in repositories, waiting to be replicated, rebutted, added to, or improved. projects like the one we describe below rely on data scraped by the wikipron project (lee et al., 2020), providing phonological and morphological datasets coupled with frequency data, all regularly updated and open.  the wikipron project contains 1.7 million pronunciations from 165 languages.  better still, the project released its mining software so that anyone may mine the data themselves so that researchers “no longer depend on ossified snapshots of an ever-growing, ever-changing collaborative resource” (lee et al., 2020, p 4223). under-researched languages like adyghe or urak lawoi’, or an endangered/moribund language like wiyot can benefit from projects that have access to open phonological data for language revitalization efforts or preservation.  the wikipron project even has 452 words from old french (842 ce – ca. 1400 ce) that could be used to track sound change to modern french.  repositories and applications like wikipron provide invaluable data that can be used in countless ways. projects with fairseq each project requires preparing the data in a way that can be used by the software.  in this paper we use fairseq (ott et al., 2019) which is a “facebook ai research sequence-to-sequence toolkit written in python.”[10]  the toolkit requires that characters be separated with a space if that is what we’re trying to sequence.[11]  a wikipron dataset may be downloaded as a tab-separated values (tsv) file. in this article we’ll look at two projects and how we’d manipulate the data for fairseq. esperanto esperanto is a constructed language (conlang) created to be a universal auxiliary/second language to aid in international communication.[12]  from the wikipron project we first download the tsv file for esperanto.[13]  in esperanto, each letter has only one pronunciation, so it should be trivial to convert characters to the ipa pronunciation and our machine should be able to do this with great accuracy.  stress is not marked in the dataset, but in esperanto stress is always placed on the penultimate syllable.  the data is in two tab-separated columns with the grapheme (the written word) in the first column and the phoneme (the ipa representation for pronunciation) in the second column:   table 1. example data from the tsv file from wikipron.   aarono a a r o n o abadono a b a d o n o abateco a b a t e t͡s o abelmanĝulo a b e l m a n d͡ʒ u l o abortitaĵo a b o r t i t a ʒ o     the tsv is shuffled using shuf and then split into three tsv files: an 80% training set, a 10% development set, and a 10% test set using a python script.[14] python3 split.py \ --seed 103 \ --input_path epo.tsv \ --train_path epo_train.tsv \ --dev_path epo_dev.tsv \ --test_path epo_train.tsv   to prepare the data for fairseq, the important part of the code to note is that each of the three tsv files are then split into .g (for grapheme) and .p (for phoneme) files for training, dev, and test: import contextlib import csv # data was shuffled using `shuf` and split 80-10-10 using `split.py` train = "epo_train.tsv" train_g = "train.epo.g" train_p = "train.epo.p" dev = "epo_dev.tsv" dev_g = "dev.epo.g" dev_p = "dev.epo.p" test = "epo_test.tsv" test_g = "test.epo.g" test_p = "test.epo.p" # processes training data. with contextlib.exitstack() as stack: source = csv.reader(stack.enter_context(open(train, "r")), delimiter="\t") g = stack.enter_context(open(train_g, "w")) p = stack.enter_context(open(train_p, "w")) for graphemes, phones in source: print(" ".join(graphemes), file=g) print(phones, file=p) # processes development data. with contextlib.exitstack() as stack: source = csv.reader(stack.enter_context(open(dev, "r")), delimiter="\t") g = stack.enter_context(open(dev_g, "w")) p = stack.enter_context(open(dev_p, "w")) for graphemes, phones in source: print(" ".join(graphemes), file=g) print(phones, file=p) # processes test data. with contextlib.exitstack() as stack: source = csv.reader(stack.enter_context(open(test, "r")), delimiter="\t") g = stack.enter_context(open(test_g, "w")) p = stack.enter_context(open(test_p, "w")) for graphemes, phones in source: print(" ".join(graphemes), file=g) print(phones, file=p)   as shown above in table 1, the second column characters were already spaced correctly, so we needed to add spaces to only the first column. the result is two files for each set with spaced characters:   table 2. example of data ready for fairseq.   train.epo.g train.epo.p s t a c i o s t a t͡s i o o m a ĝ o o m a d͡ʒ o ĉ i r k a ŭ f l a t a d i t͡ʃ i r k a w f l a t a d i     the generated files are now ready for pre-processing in fairseq: fairseq-preprocess \ --source-lang epo.g \ --target-lang epo.p \ --trainpref train \ --validpref dev \ --testpref test \ --tokenizer space \ --thresholdsrc 2 \ --thresholdtgt 2   this pre-processing creates a folder called data-bin with binaries and a log file that provides the number of tokens found.  we can now start the training: fairseq-train \ --data-bin \ --source-lang epo.g \ --target-lang epo.p \ --encoder-bidirectional \ --seed {choose a random whole numeral} \ --arch lstm \ --dropout 0.2 \ --lr .001 \ --max-update 800 \ --no-epoch-checkpoints \ --batch-size 3000 \ --clip-norm 1 \ --label-smoothing .1 \ --optimizer adam \ --clip-norm 1 \ --criterion label_smoothed_cross_entropy \ --encoder-embed-dim 128 \ --decoder-embed-dim 128 \ --encoder-layers 1 \ --decoder-layers 1   with these parameters it took my machine[15] a half-hour to train.  tweaking the max-updates, the number of encoding layers, the architecture, or the optimizer (e.g. transformer instead of adam) will provide different, and perhaps better results.  doubling the encoder and decoder layers, or doubling the encoder and decoder dimensions to 256 slowed the processing time significantly, without improving the model in this case.  the training part of these experiments is meant to help us decide on which parameters we hope will yield the best results from many different options.[16]  we’ll run this training several times with different parameters and choose three models.  the dev part (10%) of the experiment is meant to choose the model that performs the best on the dev set.  lastly, confident on our model, we’ll use that model on the test set, as yet unseen data. to determine how well each model is doing, we use fairseq-generate that provides us an error analysis that details where our model came up short. fairseq-generate \ data-bin \ --source-lang epo.g \ --target-lang epo.p \ --path checkpoints/checkpoint_best.pt \ --gen-subset valid \ --beam 8 \ predictions.txt   the generated error analysis in predictions.txt is quite readable and shows where the expected hypothesis may be different from its target sequence: s-17 i k t i o s a ŭ r o t-17 i k t i o s a w r o h-17 -0.14448021352291107 i k t i o s a w r o d-17 -0.14448021352291107 i k t i o s a w r o s-824 e k s i ĝ o n t a j t-824 e k s i d͡ʒ o n t a j h-824 -0.12416490912437439 e k s i d͡ʒ o n t a j d-824 -0.12416490912437439 e k s i d͡ʒ o n t a j s-1085 k a p t o ŝ n u r o t-1085 k a p t o ʃ n u r o h-1085 -0.15732990205287933 k a p t o ʃ n u r o d-1085 -0.15732990205287933 k a p t o ʃ n u r o   the rows in predictions.txt are source, target, hypothesis (tokenized, meaning any punctuation symbols in a project with sentences would be space-separated), and detokenized (not broken into separate linguistic units).  the number before the hypothesis is the log-probability of this hypothesis.  for our project, if the target matches the hypothesis, the model has predicted correctly. we use a script written by dr. kyle gorman to parse the output of fairseq-generate and provide a word error rate (wer).  using this script, if any character is incorrect, the word error rate is raised.  as there should be no ambiguity in pronunciation and the conversion of a character to a sound, we expected that our model would perform near-perfectly. choosing the model that performed the best, we can now give the model the test data.  predictably, on the test data, the word error rate was 0.00, a perfect score. russian stress to explore how to add features to the model, we can look at experiments in russian stress.  features are properties of the target that may or may not help in prediction.  features could include part of speech, frequency, animacy (whether a noun is sentient or not), or many other characteristics. similar to the esperanto project above, we have columns with data in a tsv file.   table 3. example data from the tsv file from schriner (2022).   ямбам я́мбам 1 ямб n;msc;inan;pl;dat шихтовее шихтове́е 1 шихтовой a;cmpar;pred щелкануть щелкану́ть 0 щелкануть v;perf;inf иноки и́ноки 2 инока n;fem;anim;pl;nom стёсанном стёсанном 2 стесать v;perf;der;der/pstpss;a;neu;anin;sg;loc     the first column of data is the word with no stress markers.  the second column is the word with stress marked.  the third column is a stress code derived from the placement of the stress in the word: reversing the text in place and counting from 0 at the end of the word, each word was given a stress code; this data was added to the tsv as a column.  only vowels in russian may have stress, so deriving the stress code was simply a matter of counting vowels until a stress marker occurred.  «ё» is always stressed, so the script stops and assigns a code when an «ё» is discovered.  the fourth column is the word’s lemma (its root).  the fifth column contains the full morphology of the word including the word’s part of speech, the tense for those that are verbs, animacy, gender, grammatical number (whether a noun is singular or plural) and russian case (e.g. nominative (nom) case for the subject of the sentence, or dative (dat) case for an indirect object of a sentence).  for the adjective (a) in table 3, the word is comparative (cmpar, as in more) and it functions as an adjective predicate (pred), linked to the subject of the sentence. in this paper we will not be processing this with fairseq, but some promising results may be found in schriner (2022).  this project is already significantly different from our esperanto example in that stress in russian has complicated patterns and ambiguous rules that will challenge a machine to place the stress correctly.  incorrectly-stressed words may be unintelligible or prove more difficult to place correctly with the existence of stress homographs such as óрган ‘organ of the body’ and оргáн ‘organ’ (musical instrument) (wade & gillespie, 2011). similar to the esperanto example, we have to format our text for fairseq and sequence-to-sequence modeling.  to do this we’ll again have space-separated characters that we’ll convert to other space-separated characters.  from table 3, the word иноки ‘others’ will be converted to и́ноки so our tsv file should have spaces: и н о к и will convert to и́ н о к и.  we want our machine to learn that given certain features we can expect a certain outcome in training.  the features in table 3 are: stress code, lemma (the root of the word), and the full morphology including part of speech. we can create several experiments from this data including:   given the word and its lemma, predict the stress code: и н о к и инока  ← the feature added to the spaced-characters 2   ← the target will be the stress code, three vowels from the end starting at 0   given the word and its part of speech, predict the stress code: и н о к и noun  ← the feature added to the spaced-characters 2   ← the target will be the stress code, three vowels from the end starting at 0   given the word and all of its morphological properties, predict the stress code: и н о к и n;fem;anim;pl;nom  ← the feature added to the spaced-characters 2   ← the target will be the stress code, three vowels from the end starting at 0   from the first experiment, the data in the tsv would be formatted like so, with the feature added to the end of the data in the first column, itself with no spaces: table 4. formatting the tsv data.   source (column 1) target (column 2) я м б а м ямб 0 ш и х т о в е е шихтовой 1 щ е л к а н у т ь щелкануть 0 и н о к и инока 2 с т ё с а н н о м стесать 2     the same methods used in the esperanto example could be used: we would train the model using fairseq on 80% of the data so the model can learn that words like иноки with the root of инока would have a stress code of 2.  once trained, we choose the model that performs best on the dev set (10%).  then we use that model on completely unseen data in the test set (10%).  by examining and contrasting different experiments, we can see if knowing the word’s root helps in placement of the stress, or if adjectives tend to have stress in particular places, or possibly even that the ambiguity in stress-placement can not be aided with this type of machine-learning.  experiments similar to these were conducted in schriner (2022), showing that knowing the word’s root led to the best predictions and the lowest word error rate, while adding the part of speech feature led to the worst results and the highest word error rate. conclusion preparing for experiments like those above require hypotheses, planning, and formatting the data for the software.  we used fairseq and found that with our wikipron data, the model we chose had no errors in predicting pronunciation in esperanto, even with unseen data.  in the russian stress experiment we looked at how to prepare data in the same way but added features to the model’s training.  the fairseq framework makes it astonishingly easy to toggle and experiment with different parameters from the terminal and work on experiments like those described above.  with continued, collaborative, and open data, we can expect invaluable further research in this area. about the author john schriner is the e-resources and digital initiatives librarian at nyu law school. his research tends to coalesce at the intersection of linguistics, cybersecurity, and librarianship. references chen, h. (1995). machine learning for information retrieval: neural networks, symbolic learning, and genetic algorithms. journal of the american society for information science, 46(3), 194–216. https://doi.org/10.1002/(sici)1097-4571(199504)46:3<194::aid-asi4>3.0.co;2-s lee, j.l., ashby, l., garza, e., lee-sikka, y., miller, s., wong, a., mccarthy, a., and gorman, k. (2020). massively multilingual pronunciation mining with wikipron. in proceedings of the 12th language resources and evaluation conference, pages 4223-4228. open data. (n.d.). sparc. retrieved november 29, 2022, from https://sparcopen.org/open-data/ ott, m., edunov, s., baevski, a., fan, a., gross, s., ng, n,. grangier, d., and auli, m. (2019). fairseq: a fast, extensible toolkit for sequence modeling. in proceedings of the 2019 conference of the north american chapter of the association for computational linguistics (demonstrations), minneapolis, minnesota. association for computational linguistics, (pp. 48-53). sanaullah, a. r., das, a., das, a., kabir, m. a., & shu, k. (2022). applications of machine learning for covid-19 misinformation: a systematic review. social network analysis and mining, 12(1), 94. https://doi.org/10.1007/s13278-022-00921-9 schriner, j. (2022). predicting stress in russian using modern machine-learning tools. https://academicworks.cuny.edu/gc_etds/4974/ wade, t., & gillespie, d. (2011). a comprehensive russian grammar. wiley-blackwell. zhu, h., & lei, l. (2022). a dependency-based machine learning approach to the identification of research topics: a case in covid-19 studies. library hi tech, 40(2), 495–515. https://doi.org/10.1108/lht-01-2021-0051 endnotes [1] https://voyant-tools.org/ [2] https://www.nltk.org/ [3] https://www.fon.hum.uva.nl/praat/ [4] https://www.liwc.app/ [5] meaning simply that the input and output are visible but the inner-workings and source code are closed [6] https://towardsdatascience.com/understanding-random-forest-58381e0602d2 [7] https://data.nls.uk/ [8] https://dataverse.no/dataverse/trolling [9] https://www.re3data.org/ [10] fairseq can be installed via pip from https://pypi.org/project/fairseq/ [11] this is specified in the preprocessing below [12] for a fascinating history of esperanto from its beginnings through the early soviet union, please see brigid o’keeffe’s esperanto and languages of internationalism in revolutionary russia, ‎2021, bloomsbury academic [13] https://github.com/cuny-cl/wikipron/blob/master/data/scrape/tsv/epo_latn_narrow.tsv [14] this script is agnostic to the data-format and is written by kyle gorman and jackson lee.  the script can be found here: https://github.com/cuny-cl/wikipron-modeling/blob/master/scripts/split.py [15] intel core i7-6700 cpu @ 3.40ghz × 8 with 32gb ram [16] for all available parameters for training, please see: https://fairseq.readthedocs.io/en/latest/command_line_tools.html#fairseq-train subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – jarrow, electronic thesis, and dissertation software mission editorial committee process and structure code4lib issue 18, 2012-10-03 jarrow, electronic thesis, and dissertation software collecting and disseminating theses and dissertations electronically is not a new concept. tools and platforms have emerged to handle various components of the submission and distribution process. however, there is not a tool that handles the entirety of the process from the moment the student begins work on their thesis to the dissemination of the final thesis. the authors have created such a tool which they have called jarrow. after reviewing available open-source software for theses submission and open-source institutional repository software this paper discusses why and how jarrow was created and how it works. jarrow can be downloaded and the project followed at http://code.library.unbc.ca. by james r.w. macdonald and daniel yule introduction this article examines the development of an electronic thesis and dissertation (etd) software module, jarrow, built for drupal [1] and the selection of islandora as the digital institutional repository software at the university of northern british columbia (unbc). the article also provides an overview of available software, largely open-source, for managing theses submissions. the university of northern british columbia (unbc) is a small, research-intensive university. over the last 4 years, graduate students at the school have produced an average of 77 masters level theses and three doctoral dissertations each year. the average thesis, about 115 pages, represents a significant amount of work not just from the student in researching and writing, but also from the student’s supervisor, committee and university support staff. once a student submits their thesis for defense, a flurry of activities over several months takes the manuscript through a process of approvals, revisions, descriptions and finally publication and dissemination to the larger academic world. submitting a thesis as a print manuscript at unbc requires as many as 15 paper forms that must be signed and approved by various individuals: the student’s supervisor, the dean of graduate programs, the registrar, the student’s committee, an external examiner and of course, the student. a date and time must be scheduled for the thesis defense and the university community must be notified of public defenses. along the way there are dozens of emails between the various actors involved and therefore multiple opportunities for miscommunication and errors. the misspelt name of a committee member on one form or an improperly formatted title page can lead to a cumbersome and sometimes embarrassing correctional process. submission in print is also costly. graduate programs at unbc spends an estimated $3,500 (cad) each year on shipping approval forms and theses. thesis submission and dissemination is a process crying out for automation and electronic processing. an institutional repository is the final resting place for completed etds. when theses are produced in print, that repository is generally the library’s print collection or the university’s archives. when theses or dissertations are produced electronically, a digital institutional repository is required. at unbc islandora is the software that has been chosen to support its future repository. etds, however, do not magically appear complete with metadata and organized in the repository. some mechanism or process must organize, describe, and evaluate these items, regardless if they are print or digital. project planning following software industry best practices, the first phase of development was requirements gathering. identifying software solutions that can meet the needs of a complicated and policy driven process like thesis submission requires an in-depth understanding of the current process and its evolution. the library managed the development of the etd project but the business logic of etd submissions is the responsibility of graduate programs. the library generally only becomes involved in theses submissions at the very end when the thesis is catalogued and made public. producing graduate level work involves faculty, librarians, staff and, in the case of an electronic submission process, information technology services. therefore, identifying a system that will meet the needs and enhance the process requires input from all of these areas. sharon reeves of library and archives canada (lac) and the networked digital library of theses and dissertations (ndltd) provide a step-by-step guide to establishing an etd submission program. they advocate cross-institutional representation in order to develop a balanced initiative (reeves 2010). therefore, a governing committee and two working teams were created to establish an institutional repository for unbc and convert the print theses submission process to an electronic process. governance the steering team is responsible for the overall governance and direction of the project. the dean of graduate programs and the university librarian co-chair this committee. also on the steering committee are the institution’s chief information officer and vice president, research along with the project manager, who chairs the implementation team and serves as a link between the three teams. the implementation team is comprised of stakeholders from graduate studies, the library and it. this team is responsible for the actual development of the institutional repository and conversion to an electronic thesis submission process. the team researches and implements best-practices and is responsible for communicating their decisions to the steering team for approval. the advisory team is a small group (6) of interested faculty and research staff. this group meets at project milestones with members of the implementation team or steering committee to provide feedback on decisions made or yet to be made. essentially, this team has worked as a standing focus group throughout the development of the project. current processes the following flow chart illustrates the print submission process at unbc as it was in 2012. figure 1. the complete submission process at unbc. from this chart, one can begin to understand the complexity of thesis processing. the student, upon approval from their committee and supervisor, delivers the thesis to the graduate office. the graduate office reviews the submitted document for formatting errors and coordinates with the student in an iterative process on any required changes. an external examiner is selected for the thesis defense, the defense is scheduled, and a room is booked before the process of defense can begin. once a defense is completed the graduate office reviews the thesis again and the student must read and agree to licenses allowing the university, proquest and the national library to distribute their work. after all of this a librarian catalogues the work and it is placed in a repository for both preservation and dissemination to the academic community and public. this brief synopsis neglects issues around publication embargoes, selecting a chair for the thesis defense, scheduling a practice defense and many other details. a graduate office can process, depending on the size of the school, dozens to thousands of theses each year. in many institutions, electronic submission of theses is managed with paper forms and scanned documents. the thesis itself is often delivered to the graduate office over email. presentations by angela mccutcheon and richard pollenz and lisa piazza at the 2011 united states electronic thesis and dissertations association (usetda) conference demonstrate the complexity of managing the etd submission process (mccutcheon 2011; pollenz and piazza 2011). both of these presentations describe methods of organization including file structures, saved email templates, and the design of paper and online forms for managing etds. the etd submission and approval process is the perfect candidate for an automated or guided online software solution. several institutions have designed and implemented dynamic form based websites that guide students and faculty through their process and maintain a greater degree of organization over etd submissions for graduate office staff. for example, in her 2012 usetda conference presentation nathalia bauer highlights how the university of central florida (ucf) created a dynamic and logic driven student website when excel spreadsheets and elaborate file organizations proved insufficient to manage the etd process (bauer 2012). the ucf’s college of graduate studies developed this website after failing to find an existing software solution that was flexible enough to meet their needs.like the university of central florida, unbc also concluded that existing software solutions for managing etd submissions are not sufficiently flexible to accommodate the processes at unbc. existing software ultimately fell short in two areas: 1) the underlying architecture was deprecated or insufficiently maintained, making the software unstable and difficult to work with or 2) the software did not manage the complete etd submission process. unbc was interested in a solution that would manage etd submissions beginning with the first time the graduate office began working with a student. this is typically well before the defense of the thesis and includes the scheduling of the defense. existing software solutions begin after the successful defense of a thesis at the earliest. existing software there are at least 5 open-source software solutions for etd submission management. they include: etd-db, tapir, valet, openetd and vireo. the following chart lists open-source software for the management of etds and indicates the year of their first public release and their most recent update, giving an approximation of their lifespan. figure 2. figure 2 demonstrates that although there have been many attempts to create etd systems, most fail to gain much traction. figure 2 demonstrates that although there have been many attempts to create etd systems, most fail to gain much traction. there are some years in which there were no viable open source options at all, with all of the existing software no longer being maintained (for example, from 2008-2010). this gap clearly motivated the creation of the more modern etd systems (vireo and openetd), but there is still no software that can handle the entirety of the submission process. what follows is an overview of some of the past and present attempts at an etd system along with why unbc ultimately chose not to use them. etd-db virginia polytechnic institute and state university (better known as virginia tech) was one of the first universities to adopt an electronic process for thesis submission, beginning in the nineties (institutions requiring electronic thesis/dissertation submissions 2012). in partnership with the networked digital library of theses and dissertations (ndltd) they released the first open-source software designed to handle etd submissions, etd-db, in 1999. etd-db combines a public user submission interface and web management with a searchable database of past-approved submissions. essentially virginia tech’s software combines the submission tool with a small searchable institutional repository of theses. the software is available free of charge from virginia tech’s website. there is no formal software license applied to the module (mcmillan 2012). unfortunately, etd-db has not had a major release since 2004. while the software is still in use by virginia tech and others, it is dated. etd-db in short “is a series of webpages and perl scripts that interact with a mysql database (etd-db: home 2012).” this means that customization of the software can be difficult. perl is a language that unbc has little institutional expertise in. the application also relies on some older and deprecated code libraries. etd-db was a seminal work in the etd field but the underlying architecture requires a greater degree of technical expertise to maintain and customize than is desirable. tapir the university of edinburgh library developed the theses alive plug-in for institutional repositories or tapir in 2003 (jones 2004). it is available under a 3 clause “modified bsd license.” the software focuses heavily on collaboration tools designed to assist students and faculty with the editing and revision of theses. today, collaboration tools for editing documents are widely available from word’s track changes to adobe reader’s tools for pdf document markup. as such tapir’s innovative focus is less relevant. tapir was written in java and designed as an add-on to dspace; however, it was created to work with dspace 1.2. parts of the dspace code were modified with tapir to deliver some of its core functionality. these changes were not contributed back to the dspace community. tapir is therefore bound to dspace version 1.2 (five versions and eight years old). this alone makes tapir an unlikely candidate for adoption. valet visionary technology in library solutions or vtls inc. developed an open-source etd submission module named valet and publicly released it in 2005 (rogers 2005). valet is available under a mozilla public license 1.1. like etd-db, valet is a perl based module but coupled with a fedora digital repository. the product is best suited when deployed in conjunction with vtls’ proprietary institutional repository, vital (valet for etds 2012). valet appears not to have released any major update since its initial release in 2005, and has little documentation, making installation and use difficult. during the summer of 2008, the australian research repositories online to the world (arrow) project, led by monash university in australia forked valet when they created a java version called squire, which is also open source (arrow_contribution 2010). unfortunately, the java version appears to have no documentation of any type and has not been updated in almost four years, essentially rendering it unusable. openetd openetd was developed by rutgers university in 2010 and is available under the gnu general public license 3.0. this system is repository agnostic. in other words, the software exports theses and dissertations in a mets/xml compliant file for import into a repository of choice. this is a compelling feature for those not willing to be tied to particular institutional repository software. built on linux, mysql and php this etd solution utilizes better web application architecture than its predecessors. with a host of features such as email notifications, automatic margin and page number detection, and a simple user management this system is quite compelling; however, openetd focuses on the post-defense process. the software also offers very limited ability to customize forms or the logic around a submission process. vireo vireo is a java based dspace plugin developed by texas digital libraries, and is available under a 3 part “modified bsd license (vireo etd submission and management 2012).” vireo was released in 2011 and is under active development. the software’s integration with dspace, ease of use and ability to make simple system customizations has led to an active user community that is beginning to grow outside of the texas area. vireo offers many of the same features open-etd does, such as automatic email notifications and templates. although vireo handles more of the process than any other system, beginning immediately after defense and handling corrections in addition to basic ingesting, it does not handle any pre-defense procedures, such as scheduling or initial committee approval. proquest’s umi administrator proquest has been in the business of theses and dissertation publishing since 1938. a large number of academic institutions in north america publish theses and dissertations with proquest. as a result they have developed etd software, umi administrator, and offer it free of charge to institutions wishing to publish with them. umi administrator is not, however, open source. proquest hosts and maintains the software and will customize its interface to reflect the branding of an institution. at the end of the submission process, proquest returns theses and dissertations electronically for ingest into local digital repositories. if an institution chooses to use umi administrator, their students are no longer required to pay proquest’s publishing fees. in cases where an institution pays these fees on behalf of the student the umi administrator is extremely compelling. there are several shortcomings of the umi administrator that do not make it suitable for use at unbc. first, the system only handles the post-defense submission process. some thought was given to developing a system that would cover the pre-defense submission and then have students use umi administrator for the post-defense process. however, this would require students to create a separate account and re-enter all of the meta-data and other details in umi administrator. while this is not an insurmountable barrier, it does greatly reduce the simplicity of any chosen process. if proquest were to create an api that would allow the transfer of user and thesis data from an open-source system to umi administrator, future integration could be possible. the second significant drawback to using umi administrator would be the reliance it creates on a third party for publishing an institution’s theses and dissertations. for many this may not be an issue. however, it is arguable that while proquest provides a valuable service, third party control and profiting from publicly funded research should be avoided. selecting institutional repository software the software selected for storing and disseminating completed theses would have ramifications for nearly any solution to producing electronic theses or dissertations (etds). as a result, the project first had to identify appropriate repository software with an eye to automating a thesis submission process. there are many proprietary software solutions for institutional repositories. for example, oclc’s contentdm, and berkley’s bepress offer compelling software platforms. however, regardless of the software chosen (proprietary or open-source) it was believed that the skills of a web-developer would be necessary for this and other projects. because of financial limitations, unbc could not both purchase and maintain proprietary software and hire a web-developer. unbc chose to hire a web-developer. this decision, to hire a web developer, essentially limited the choice of repository software to an open-source solution. there are many open-source software platforms that could act as an institutional repository (ir). however, the three most popular platforms used by many universities are: dspace, eprints and the up and coming platform, islandora. dspace dspace was developed as a joint project between mit and hewlett-packard in 2002, but has since been spun off into its own company: duraspace. it is a java-based product that ships with a built in repository based on either postgresql or oracle.. the product has wide distribution and is in use at several institutions within british columbia, including simon fraser university and the university of british columbia. dspace is arguably the most widely used institutional repository software by universities, especially in north america. dspace can be challenging to configure and install, with a multi-stage build and install process, involving a fair amount of knowledge. however, once installed, maintaining and configuring the system is straightforward enough, and is backed by extensive documentation and an active support group. dspace offers an interface that can be extensively customized using xml, and a standard organizational scheme (communities are made up of collections are made up of sub-collections are made up of objects). eprints eprints was developed at the university of southampton in 2002. a perl based ir solution, eprints is easy to get up and running, and provides an excellent discovery and submission interface. however, configuring anything beyond the standard is quite challenging. the configuration files themselves are perl scripts. there is decent documentation, but a good understanding of perl is required to administer the system. eprints also uses a custom built repository, but supports the standard simple web-service offering repository deposit or sword interchange protocol, meaning that digital objects can be extracted and shared (lewis 2012). the eprints community is very active, and support for configuring and maintaining eprints repositories is readily available. there are, however, no known eprints installs in western canada making it slightly less compelling than it could be for unbc. islandora islandora is a newcomer on the institutional repository software market. developed at the university of prince edward island in canada and supported by an independent company, discovery garden, islandora brings together two well-established open-source products: drupal and fedora. because islandora is built on top of drupal, all of the necessary content management system (cms) and application framework components are included. for example, any existing drupal theme will automatically work with islandora because it uses standard drupal components. furthermore, by leveraging such a well-established and sophisticated cms like drupal, islandora makes it much easier to organize repository collections for public consumption in ways that may be much more challenging for dspace or eprints. fedora is a well-established open source database for organizing digital objects. since the duraspace foundation runs both dspace and fedora, some software designed for dspace also works with islandora. for example, duracloud, an off-site backup and preservation module for digital objects, can be used with islandora. furthermore, discovery garden has recently released a php toolkit named “tuque” which allows developers to ingest objects into a fedora repository independently of islandora. this means that any module built with tuque could potentially export into a dspace institutional repository, or conceivably an eprints one as well, if the tuque library was expanded. the flexibility is a good fit with unbc’s goal of allowing for change down the road. islandora itself is a particularly good fit for unbc. as this project began, the university just completed an extensive selection process for a new cms to manage its web presence. drupal was the cms chosen. having institutional knowledge and expertise in drupal bodes well for the sustainability of a new ir built on islandora. furthermore, the university had already invested in islandora in partnership with the prince george public library to house a digital collection of local newspapers. unbc has a knowledgable user group to draw on, both internally and with the prince george public library. both eprints and dspace offered compelling and powerful options but in unbc’s context, the answer is islandora. developing an open software solution by far the most important aspect of designing a new software package is picking a good name.  unbc chose the name jarrow for its etd module.  jarrow is the name of a monastery in england that boasted an enormous library and was a beacon of enlightenment in the middle ages (yule 2012); a fitting name for a tool designed to assist with knowledge creation and dissemination. having thoroughly investigated existing options for etd submission, unbc determined that building its own drupal module to integrate with islandora was the only way to satisfy the requirement to handle the entirety of the submission process as well as deal with any varying processes in place. the process for submitting a doctoral dissertation is different from the process for a master’s thesis at unbc. it is likely that many institutions have significant logical differences in submission workflows for type of a degree, department, or the school the degree is awarded from. an etd submission module should be flexible enough to accommodate these differences. in addition to being able to handle multiple workflows, the system must be able to integrate with institutional administration software, such as banner, reducing the possibility for incorrect metadata entered by students. given that the module being built needed to be sufficiently flexible to handle unbc’s multiple workflows, and is intended to be open source software, it was decided to create a module which any institution could adopt and use as their etd submission system. in addition to providing a straightforward gui for creating and managing custom workflows, the software builds on top of drupal’s module system to allow for institutions to create their own modules to further extend the software’s functionality. system architecture the workflow management aspect of the software is central to the overall design, as all other features are designed to integrate with it.  the system works much like survey software. those who have used survey monkey, fluid surveys or any number of other online survey applications will find jarrow both familiar and easy to use.  it is comprised of three components, the data model, the form editor and the process manager. the data model defines all meta-data and related information about a submission.  this includes everything from the title of the thesis to the date of defense.  if there is a piece of information relevant to a thesis submission, it must be defined in the data model. figure 3. a demonstration data model created for unbc. new elements are added by dragging them in from the right and dropping them on the data model in their desired location. the form editor allows the creation of forms to collect the data defined in the data model.  there can be many forms associated with a single data model. figure 4. the form editor. this form is generated from the “thesis information” data element group visible in figure 3. form elements are created by dragging in data elements from the data model (as was done here) or by adding them manually. the process manager provides a system for adding logic to the creation, editing, display and approval of forms. figure 5. the process manager. any forms created in the form editor can be added, along with digital objects to represent supplementary files. here the pre-submission stage has three forms that must be filled out. once an administrator has configured the above, students, faculty and staff members can log into the system and see any submissions that are currently active and with which they are affiliated. for example, a supervisor with multiple graduate students would have access to each of their submissions.  students can create a new submission, at which point either they or the system will select the type of process they will follow. users can edit the information in a form, which is automatically saved as soon as they enter it, although it is not submitted until they specifically choose to submit the form.  the system does not require a particular order for the users to fill out information, any user can fill out any form that they have access to at any time.  the order and number of forms displayed is completely controlled by the administrator through the process manager. figure 6. a submission. the form shown in figure 4 is displayed to the student as dictated by the process in figure 5. once the student and all staff and faculty have completed a workflow by filling out and approving all required forms, the system will deposit the thesis, supplementary files, and meta-data collected along the way into the institutional repository, in unbc’s case islandora. jarrow can be adapted to deposit completed theses in any fedora based repository.  the entire process could take several weeks, as the student will first interact with the system when they begin to ask for approval for scheduling a defense, but jarrow will guide them through the entirety of the process. the flexibility of jarrow was highlighted at the usetda 2012 conference by the authors (macdonald and yule 2012). that presentation complete with screen captures and screen casts is available at http://code.library.unbc.ca. other features in addition to the core workflow management system mentioned above, jarrow has several other features designed to make etd management easier: date scheduling the etd system provides an availability poll component that can be added to forms to allow for students, committee members and chairs to indicate the times they will be available for a defense. graduate programs staff can then choose a date that works for all members. license management institutions typically require students to agree to a non-exclusive license for their thesis or dissertation, but may want to allow students to also apply a creative commons license, or allow students to apply different licenses to the supplementary files associated with a thesis (such as granting or revoking performance rights for supplementary videos). the system provides the ability to manage all licenses an institution may want to use, and comes with the creative commons licenses pre-installed (although they do not have to be used.) email templates emails can be sent as part of the submission process, and the etd system allows for editing email templates including information from a data model in an easy point and click manner. drupal integration since the system is built as a drupal module, it can take advantage of drupal’s rich module ecosystem. authentication (using ldap, cas or shibboleth), extended profiles using profile2, or rich text editors for editing email templates, licenses or other text fields through the wysiwyg module are all available, as well as the extensive theming options provided by the drupal community or through custom themes. the flexibility of the workflow component, jarrow’s open source nature and drupal’s modular nature allow for many more features to be added at any stage of the process, either by adding code to jarrow itself or by writing modules that interact with its many integration points. conclusion unbc ambitiously decided to build an etd submission module from scratch in order to meet its need to handle the entirety of the submission process. by involving stakeholders from across campus, thoroughly investigating existing options and designing a very flexible system, unbc has created an etd system that any institution can easily customize to reflect their existing workflows from beginning to end. the question remains whether jarrow will become another etd software relic. significant development time has been committed to this project. the software will enter a pilot phase in the fall of 2012. the authors hope that the flexibility and openness of the system will encourage its adoption at other institutions and lead to its long-term sustainability. the project can be followed at http://code.library.unbc.ca. references arrow_contribution – valet – vtls’ valet for etds [internet]: google project hosting; c2010 [cited 2012 08/01]. available from: http://code.google.com/p/valet/wiki/arrow_contribution. bauer n. 2012. building an etd administrative system and student services website. in: proceedings of united states electronic thesis and dissertation association (usetda) 2012 conference.; june 13, 2012 – june 15, 2012; boston. available from: https://conferences.tdl.org/usetda/usetda2012/schedconf/presentations. etd-db: home [internet]: virginia tech; c2012 [cited 2012 08/01]. available from:  http://scholar.lib.vt.edu/etd-db/index.shtml. institutions requiring electronic thesis/dissertation submissions [internet]: networked digital library of theses and dissertations; c2012 [cited 2012 08/01]. available from: https://docs.google.com/spreadsheet/ccc?key=0atsglihgwckpdhjvounszuzyrc04uxrua0w3umgtywc&hl=en_us#gid=0. jones r. 2004. thesis alive! at edinburgh university library [internet]: edinburgh university library; c2004 [cited 2012 08/01]. available from:  http://www.thesesalive.ac.uk/dsp_home.shtml. lewis s. 2012. a brief history of sword [internet]; c2012 [cited 2012 08/01]. available from: http://swordapp.org/about/a-brief-history/. macdonald j. and yule d. 2012. developing a revolution in etd workflow management: a software solution from thesis proposal to final dissemination and publication. in: proceedings of united states electronic thesis and dissertation association (usetda) 2012 conference.; june 13, 2012 – june 15, 2012; boston. available from: https://conferences.tdl.org/usetda/usetda2012/schedconf/presentations. mccutcheon a. 2011. how to create a highly functional etd office. in: proceedings of united states electronic thesis and dissertation association (usetda) 2011 conference.; may 18, 2011 – may 20, 2011; orlando. available from: https://conferences.tdl.org/usetda/usetda2011/schedconf/presentations. mcmillan g. 2012. etd-db license. email. pollenz r. s. and piazza l. 2011. a model for management of the electronic thesis/dissertation (etd) process that promotes efficiency and student success. in: proceedings of united states electronic thesis and dissertation association (usetda) 2011 conference.; may 18, 2011 – may 20, 2011; orlando. available from: https://conferences.tdl.org/usetda/usetda2011/schedconf/presentations. reeves s. 2010. how to set up an etd submission program [internet]: networked digital library of theses and dissertations; c2010 [cited 2010 07/30]. available from: http://www.ndltd.org/resources/how-to-set-up-an-etd-submission-program. rogers m. 2005. vtls’s free valet service. library journal. 130(17): 22. vireo etd submission and management [internet]: texas digital library; c2012 [cited 2012 08/01]. available from:  http://www.tdl.org/etds/. valet for etds [internet]: vtls inc.; c2012 [cited 2012 08/01]. available from: http://www.vtls.com/products/valet. yule, d. 2012. jarrow, a history [internet]: university of northern british columbia; c2012 [cited 2012 08/01]. available from:  http://code.library.unbc.ca/jarrow-etd/jarrow-history/. notes [1] jarrow is designed for drupal 7, and should work with any sub-version (7.1-7.15) within that. when released, it will be compatible with whatever the latest version of drupal 7 is at that time. about the authors james r.w. macdonald was the web services librarian at the geoffrey r. weller library, university of northern british columbia at the time this article was written. he is now the web services librarian at the american university of sharjah in the united arab emirates. james’ research interests currently focus on usability and the user experience in digital environments. daniel yule was a web developer for the geoffrey r weller library at the university of northern british columbia. he has since moved on to pursue his ph d in computer science at dalhousie university, focusing on library technology. subscribe to comments: for this article | for all articles 3 responses to "jarrow, electronic thesis, and dissertation software" please leave a response below: glenn bunton, 2012-11-20 a very interesting article. you discuss and compare other open-source products. i’d be interested to know where you see your product as compared to a commercial product such as digital commons? daniel yule, 2012-12-03 hi glenn, thanks for the comment. the reason we evaluate strictly open source products is twofold. firstly, we decided early on that we wanted an open source system, hence we evaluated those to see if they met our needs. secondly, it is significantly more challenging to evaluate commerical products in a similar fashion. for each of the open source products we mention, we had a fully functional server up and running. i suspect this would have been impossible for commercial products. i did some research into various commercial based ir solutions, and did not find anything like what jarrow offers. digital commons, content dm and the like offer some excellent ir features, but are not well optimized for etd submission. we hope that down the road, someone might write an export filter for jarrow that allows for jarrow to be integrated with these systems (this should not be very difficult; jarrow was designed with this in mind). to summarize, we believe that jarrow is a one of a kind product. so far as we know, no other software handles pre-defense and submission. ali mohamed, 2013-11-07 we at the libyan board for research, science and technology have been examining different open source options available for a national etd system. one of our requirements is early registration of thesis to ensure research non duplication or overlap. i found jarrow very interesting on this regard. thanks. leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – editorial: some numbers mission editorial committee process and structure code4lib issue 34, 2016-10-25 editorial: some numbers wherein the journal’s most popular article and other small mysteries are revealed. by andrew darby 34 welcome to issue 34 of the code4lib journal. we have a great issue for you, with articles about digital preservation, digital archaeology, open source application development, customizing proprietary applications, and more; please check them out. but it’s election season in the united states, polls and surveys and dubious statistics are in the air, so let’s talk numbers. author demographics ten issues ago, ron peterson wrote about diversity as it relates to both authors and editors of the code4lib journal (http://journal.code4lib.org/articles/9345), and he checked in again with the editorial for issue 33 (http://journal.code4lib.org/articles/11859). to attempt to get more accurate numbers about our authors, as of issue 35 we are now asking prospective authors to fill out a brief demographic survey. it is not required, but we would like to find out more about their institutional affiliation, gender identification and geographic location. we aren’t yet sure what we are going to do with this data; we just want to catch these numbers before they slip away. 1,382,173 that is the number of hits recorded by google analytics since the code4lib journal opened its doors in december 2007 (through oct 1 of 2016). i think all of us who work with web analytics take these numbers with a grain of salt, but they’re still fun to look at. so, i thought i’d run through some of the reports, and highlight things that struck me as interesting. #1 of course, everyone who has published an article in the journal wants to know–who is the most popular author? the answer is clear: andrew h. bullen, for “bringing sheet music to life: my experiences with omr” (issue 3, http://journal.code4lib.org/articles/84). although graham mccarthy and sally wilson might make a case for “isbn and qr barcode scanning mobile app for libraries,” (issue 13, http://journal.code4lib.org/articles/5014) since they had more unique pageviews, less time on the web, and further the average reader spent almost three minutes longer on their article. and any authors who have published more than once in the journal might want to add up their pageviews to see if they can rise above 27,071 and wrest away the crown. figure 1. top hits click to enlarge #5 if i may raise my own oversized foam finger, i will note that an article i wrote way back in issue 2 takes the fifth spot. i mention this because first, well, hey, #5! but also because every time i am reminded of this article, i’m also reminded that the information in there is completely obsolete. the google calendar api has moved along, the code won’t work anymore, and it seems like there should be a “do not read” banner at the top. i’m not sure how many other c4lj articles suffer a similar malady, or what the solution is except caveat lector. but i think it’s easier to ignore that old timey copy of windows 95 sitting in your office than a google result that appears to be exactly on topic and with the code4lib imprimatur. and as for the proposed “do not read” banner–we are generally unwilling to modify an article after it is published, even though we are occasionally asked to do so. i also think it’s interesting that the phrase “library hours” returns 100 hits on the old listserv archives (https://listserv.nd.edu/cgi-bin/wa?a0=code4lib) as well as a red message that reads “your search produced too many hits and the query was aborted. please narrow your search phrase and try again.” it seems that there are some problems for which there is still no perfect solution, and an easy way of managing library hours–perhaps across multiple branches, collections and service points–is one of them. referrals the referrals graph doesn’t hold a lot of surprises–our biggest referrer is google, but it appears our articles are getting passed around on twitter (t.co) and found via the directory of open access journals (doaj.org). we also are the answer to some questions on stack overflow. figure 2. top referrers click to enlarge social network twitter, as noted, is our top social network referrer, followed by stack overflow and . . . hacker news? the once-popular delicious also makes the cut, which is a reminder that our data stretch back almost a decade, a long time for the internet. figure 3. social networks click to enlarge if we look at some more data, it turns out that our visitors from hacker news didn’t know what they were in for when they clicked that link; their average session duration was only 11 seconds. our friends from blogger were more resolute, staying on average over two minutes. figure 4. social networks (again) click to enlarge but maybe those hacker news readers aren’t so fickle; if we drill down and see what brought them to the journal, over two-thirds (by pageviews) of visitors were lured in by dianne dietrich et al‘s “how to party like it’s 1999: emulation for everyone” (issue 32, http://journal.code4lib.org/articles/11386) from earlier this year. and they appear to have been engaged readers, since they stayed an average four and a half minutes. figure 5. hacker news referrer click to enlarge more interesting nuggets this list of “page title by full referrer” reveals some other interesting things. our stack overflow readers were overwhelmingly coming to see a single article: “isbn and qr barcode scanning mobile app for libraries.” figure 6. stack overflow referrer click to enlarge it is also interesting to note that many referrals for our #1 article, “bringing sheet music to life: my experiences with omr,” come from google’s image search (google.com/imgres, which now is found at google.com/imghp). figure 7. google image search referrer click to enlarge a search for just “sheet music” doesn’t turn up a result from this article anywhere near the top, but try “digitized sheet music” on the other hand and it is the sixth result (for me, on this day). since the caption for one image in the article includes “digitized sheet music,” i guess this speaks to the value of giving your images good descriptive captions. since the average time is 50 seconds, it seems that some of the people who just came for a good picture of some digitized sheet music stayed to find out more about “the process of digitizing sheet music celebrating pullman porters and rail travel from the 1870s-1920s.” good for them. figure 8. google image search for “digitized sheet music” click to enlarge readers as one might expect, the majority of our readers are accessing journal articles from computers in the united states. figure 9. geographic location click to enlarge and the top article for canadians? “bringing sheet music to life” once again, although they do express some national pride by reading “isbn and qr barcode scanning mobile app for libraries” (from ryerson university in toronto), lisa gayhart et al’s “the road to responsive: university of toronto libraries’ journey to a new library catalogue interface” (issue 23, http://journal.code4lib.org/articles/9195) and “on dentographs, a new method of visualizing library collections” (issue 16, http://journal.code4lib.org/articles/6300) from york university librarian william denton. figure 10. top hits, canada click to enlarge the top articles for indian readers is quite different. and if i may raise my foam finger once again, it appears they like that google calendar article so much it appears twice! (remember what i said about a grain of salt?) figure 11. top hits, india click to enlarge rounding out our top ten we have: united kingdom — “bringing sheet music to life: my experiences with omr” see top 10 germany — “bringing sheet music to life: my experiences with omr” see top 10 australia — “how hard can it be? : developing in open source” by joanne ransom et al (issue 7, http://journal.code4lib.org/articles/1638) see top 10 france — “html5 microdata and schema.org” by jason ronallo (issue 16, http://journal.code4lib.org/articles/6400) see top 10 spain — “isbn and qr barcode scanning mobile app for libraries” see top 10 italy — “bringing sheet music to life: my experiences with omr” see top 10 netherlands — “an android/lamp mobile in/out board based on wi-fi fingerprinting” by keith kelly et al (issue 15, http://journal.code4lib.org/articles/5859) see top 10 finally, as a bonus, “controlled terms or free terms? a javascript library to utilize subject headings and thesauri on the web” by shun nagaya et al (issue 15, http://journal.code4lib.org/articles/5994) is big in japan. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – annotation-based enrichment of digital objects using open-source frameworks mission editorial committee process and structure code4lib issue 37, 2017-07-18 annotation-based enrichment of digital objects using open-source frameworks the w3c web annotation data model, protocol, and vocabulary unify approaches to annotations across the web, enabling their aggregation, discovery and persistence over time. in addition, new javascript libraries provide the ability for users to annotate multi-format content. in this paper, we describe how we have leveraged these developments to provide annotation features alongside islandora’s existing preservation, access, and management capabilities. we also discuss our experience developing with the web annotation model as an open web architecture standard, as well as our approach to integrating mature external annotation libraries. the resulting software (the web annotation utility module for islandora) accommodates annotation across multiple formats. this solution can be used in various digital scholarship contexts. by marcus emmanuel barnes, natkeeran ledchumykanthan, kim pham, and kirsta stapelfeldt introduction annotation has been a topic of interest since the advent of digital formats and computer networking, but the conversation moved toward productive solutions at the symposium on “humanities computing: formal methods, experimental practice” sponsored by king’s college, london on may 13, 2000. at this symposium, john unsworth coined the phrase “scholarly primitives” to describe several activities (including annotation) that cross disciplinary boundaries and are essential to most academic endeavors[1]. since his talk, the frequency with which the term “scholarly primitive” has been used[2] is a reflection of the attractiveness of thinking about annotation as something that is disciplinary-agnostic and discrete from other web-based creation and publication activities. since that time, great strides have been made in the ability to provide web-based annotation features. it has become possible to annotate digital files on the web, persist annotations over time, group them over copies of a resource, and discover annotations that are related by subject or content. the ability to think of annotation as a unique activity has arguably been important to these developments. however, in describing what is different about annotation in comparison to other digital objects, we might forget what is the same. annotations are digital objects or entities themselves and can benefit from the same tools and procedures. indeed, at the same moment that unsworth coined the term “digital primitives,” he acknowledged that considering annotation as axiomatic was philosophically problematic[1]. with this in mind, our team at the university of toronto scarborough library leveraged existing work to develop a web annotation utility[3] that operates in the islandora framework. we added features to the existing infrastructure for digital objects to enable the creation of annotations and associations/relations with other digital objects. w3c’s new proposed recommendations for a web annotation model[4], protocol[5], and vocabulary[6] and the maturation of annotation libraries provide new opportunities for web annotation features in a digital repository. a web annotation utility in islandora’s ecosystem islandora is an open-source framework supporting digital repositories and collections. the framework has two major components: drupal (a popular content management system) and fedora (a best-practices digital repository). islandora developers provide drupal modules that, when installed with their dependencies, allow users to manage, create, and access fedora repository items (digital objects comprised of files, or datastreams) in a drupal interface. core indexing functions are provided through islandora-specific modules that provide custom integration with apache solr. outside of islandora’s core indexing and fedora connection functions, modules fall broadly into three categories: solution packs – package custom forms, workflows, and content models. content models are templates for an object of a particular type. solution packs usually service a data format (such as newspapers or large images) utility modules – provide functions to some or all repository objects (such as workflow or the creation of checksums) viewers – provide customization of the presentation layer specific to object content-types. islandora viewer modules are part of the “view” layer, used in islandora’s multi-tier architecture. (these are players for things like video or large image content) the dsu’s web annotation utility module falls into the utility module category. the module installs content models that follow the w3c web annotation data model, protocol, and vocabulary, leverages existing viewers, and implements javascript libraries to form a sustainable base for web annotation of multimedia content. the module allows users to annotate large and basic images. it also provides the ability to annotate video, audio, and oral history objects. all annotations, regardless of what type of content they are written for, adhere to a common data model. on the front end, popular viewer-specific libraries are used to provide users with the ability to create annotations. we hope our approach will facilitate migration to the next major version of fedora (fedora 4) which, as a native linked-data platform, will allow for greater adherence to the w3c web annotation data model, protocol, and vocabulary. we also hope that the web annotation utility module provides an approach that could work for other existing or emerging islandora viewers. for example, this approach could be extended to provide annotations on other types of digital objects, such as pdfs. this is not the first islandora annotation module. an islandora image annotation solution pack[7] was developed for use in islandora by the university of prince edward island’s robertson library. the initial module was based on the open annotation collaboration (oac) standard[8] a precursor to the current w3c web annotation model, protocol, and vocabulary[9]. our web annotation utility module does not adopt the same approach or use the same data model as the islandora image annotation solution pack, which is currently hosted in islandora labs. the content models used by the web annotation utility module initially appeared in an early version of the islandora oral history solution pack[10]. however, we realized that our users sought to perform annotations on other types of objects and decided to repurpose this functionality and create this utility module. our desire for a more agnostic approach (as well as the desire to make annotations separate islandora objects) also prompted us to move away from the image-specific framework of the islandora image annotation solution pack. the w3c recommendations for annotations on january 17, 2017, the w3c web annotation data model, protocol, and vocabulary documents became proposed recommendations. the documents focus on reuse of annotations across systems by describing a simple format (in json-ld) that can be used in multiple contexts. the web annotation data model describes annotations “to be a set of connected resources, typically including a body and target [… that …] is related to the target”[4]. annotations have at least one target, and may also have a body (see figure 1). the annotation and target each contain metadata describing their properties and relationships. the body may be embedded in the annotation or consist of another digital object. the recommendation describes the example of a post made about a particular web page. in this example, the “annotation” would link together the post as the body and the web page as the target. in contrast, a comment on a webpage could be stored more simply as an annotation containing the body of the comment. figure 1. from the web annotation data model document [4], depicts the conceptual separation of annotation from the body and the target of the annotation.   the second recommendation, the web annotation protocol[5], describes the standard for interactions between clients and servers via rest architecture. the protocol prescribes that annotations are to be managed via an annotation container (which conforms to the linked data platform specification[11] of an ldp container). the protocol also describes crud operations for annotation, retrieval and discovery mechanisms via the container (annotationcontainer), and paging mechanisms for breaking down the response (annotationpage). the protocol assumes a linked data platform and does not specify how containers are created or deleted, or how to handle annotationcontainer if an ldp is not being used. in fedora 3, the web annotation utility module implements the annotationcontainer and annotation using separate content models (templates for digital objects, as previously described). the final recommendation, the web annotation vocabulary [6], defines a vocabulary for annotations. our module adheres to this vocabulary in the content of annotationcontainer and annotation objects. these content models are described in more detail in the following section. web annotation utility module content models and workflow the content models installed by the web annotation utility module are the islandora web annotation content model (islandora:wadmcmodel) and the the islandora web annotation container content model (islandora:wadmcontainercmodel). as users create annotations, their annotations and annotation containers conform to these content models. both content models define datastreams, which are files that comprise the digital object. every islandora object must contain a dublin core (dc) and rdf datastream (rels-ext) datastream. additional datastreams contain annotation-specific content. islandora:wadmcmodel defines two annotation-specific datastreams: wadm and wadm_search. an interface view of all the datastreams that make up a web annotation is depicted in figure 2. the wadm datastream, which is in json-ld format, defines the annotation body and target. we hope that storing json-ld will ease migration to the fedora 4 linked-data platform. figure 2. data object datastreams in an islandora (drupal) interface. the wadm and wadm_search datastreams are specific to the web annotation utility module. view larger image   { "@context":[ "http://www.w3.org/ns/anno.jsonld" ], "@id":"http://localhost:8000/islandora/object/annotation:20", "@type":"annotation", "target":{ "source":"http://localhost:8000/islandora/object/islandora%3a615/datastream/medium_size/view", "format":"image", "selector":{ "shapes":[ { "type":"rect", "geometry":{ "x":"0.29", "y":"0.08791208791208792", "width":"0.12", "height":"0.07967032967032966" } } ] } }, "body":{ "type":"textualbody", "bodytext":"an annotation, created by me!", "format":"text/plain" }, "creator":"testannotator", "created":"2017-05-25t18:55:58z" } figure 3. json-ld content of the wadm datastream   the second datastream, wadm_search, contains an xml representation of the json-ld data in the wadm datastream for indexing purposes. by default, islandora’s use of solr leverages an application called gsearch, which only indexes xml and text content. by creating an xml representation, we ensure that data (like the annotation author’s username and the content of the annotation) is properly indexed. this approach also permits users familiar with islandora’s gsearch/solr framework to make modifications to the indexing of annotations. by indexing annotations in solr, users and site builders can index, retrieve and display annotations for either a specific object or repository-wide. users can also use extensions such as islandora solr views module (which permits indexed solr fields to be displayed/visualized or downloaded using drupal’s views framework) [12][13]. figure 4. sample content of the wadm_search datastream   the web annotation utility module also makes modifications to fedora 3’s default rdf datastream (rels-ext) to add rdf that associates the annotation to an annotation container object (islandora:wadmcontainercmodel). figure 5 shows the content of the rdf datastream for an annotation object, which is connected to an annotation container using fedora:ismemberofannotationcontainer. figure 5. example of fedora’s default rdf datastream for an annotation object. the fedora:ismemberofannotationcontainer statement contains the reference to the annotation container.   islandora:wadmcontainercmodel defines one annotation-specific datastream, wadmcontainer. an interface view of all the datastreams that make up a web annotation container is depicted in figure 6. this datastream stores the json-ld representation of the container. figure 6. wadmcontainer datastream specific to the web annotation utility module. view larger image   { "@context": [ "http://www.w3.org/ns/anno.jsonld", "http://www.w3.org/ns/ldp.jsonld" ], "@id": "http://localhost:8000/islandora/object/annotation:19", "@type": [ "basiccontainer", "annotationcollection" ], "total": "1", "label": "annotationcontainer for islandora:7", "first": { "id": "http://localhost:8000/islandora/object/islandora%3a7/annotations/?page=0", "type": "annotationpage", "items": [ "annotation:26", "annotation:27" ] } } figure 7. json-ld content of the wadmcontainer datastream   the web annotation utility module also makes modifications to fedora 3’s default rdf datastream (rels-ext) to add rdf that associates the container to the object being annotated. figure 8 shows the rdf content of this datastream in the container object, which is connected to the object being annotated using fedora:isannotationcontainerof. figure 8. example of fedora’s default rdf datastream for annotation container. the fedora:isannotationcontainerof statement contains the reference to the object being annotated.   rdf statements in the container object connect it to the object being annotated. similarly, rdf statements in the annotation connect it to the annotation container. for each object that is annotated, there will be one annotation container and potentially multiple annotations. annotation creation workflow the diagram in figure 9 provides a high-level overview of the entire annotation creation process. when a user views and creates an annotation, one of two things occur: an annotation is created and linked back to an existing annotation container, or an annotation container is created concurrently with the annotation (when the annotation is the first one made on an object). not only does this structure reflect the specifications of the web annotation data model, it also facilitates better performance when bringing back an annotated object’s related annotations. figure 9. annotation creation workflow. an annotation container and annotation are created and related to one another. subsequent annotations will be related to the annotation container for the object.   drupal/islandora viewers and annotation creation in building the islandora web annotation utility, we tried to reuse as much existing community work as possible. on the front end, all annotation authoring is managed by drupal. the wide adoption and development of drupal is an impetus for institutions installing islandora. drupal has also inspired islandora developers, who sometimes extend existing drupal modules or adopt drupal module approaches, to provide enhanced features for islandora. one example of this type of an approach is the islandora context module[14]. in drupal, the context module[15] allows for site builders to author particular contexts in which rules are executed. each time a page is loaded, the context module seeks relevant conditions and executes rules that have been defined by an administrator. mark jordan, head of library systems at the w.a.c. bennett library at simon fraser university, built the islandora context module to extend the context module and enables rules to be executed on islandora (fedora) specific contexts, such as a keyword in an object datastream, or when the object is of a particular content model. the islandora web annotation utility uses the islandora context module to load javascript and css files and target specific islandora viewers and content models, enabling annotation. work thus far has focused on targeting simple images, large images, book pages, video, audio and oral history content. figure 10 outlines the setup of the module in a drupal environment. site builders or administrators define content-specific contexts, depending on what type of content they wish to make it possible to annotate. users are then given annotating “roles” in drupal to view, create, edit or delete their own annotations and/or the annotations of others. users can then begin annotating repository content. figure 10. a site administrator defines contexts for specific content types and assigns appropriate permissions to annotating users   the islandora context module provided us a way of targeting specific types of repository content. existing annotation libraries could then be leveraged to provide annotation tools depending on the type of content being annotated. local use cases guided our decision to prioritize the ability to annotate objects using the openseadragon viewer (which is a zoomable viewer for large images) and the video.js player, as well as our oral history object player (which is based on, and utilizes, our video.js player). openseadragon and annotorious openseadragon is a open-source javascript viewer for web content that supports high resolution zoomable images. in islandora, the openseadragon module [16] enables the viewer to work with an islandora installation by providing a djatoka [17] tile source for the viewer. administrators can configure the openseadragon viewer to display islandora-managed large image and page content. the islandora web annotation utility module uses the annotorious javascript library, developed “under the leadership of the austrian institute of technology,” [18] to provide users with the ability to annotate objects using the openseadragon viewer. when a user encounters an image object and has been granted permissions to annotate this content model, two new buttons appear alongside openseadragon’s default navigation buttons (figure 11), allowing the user to view and hide annotations and to create new annotations. figure 11. user experience when hovering over an annotation created in the openseadragon viewer. each annotation is identified by a number that corresponds to a legend below the image   when a new annotation is created, it is assigned a number in the interface which corresponds to the annotation text that is displayed beneath the metadata (figure 12). these numbers are a ui guide only and assigned when the user visits the page. hovering over the image reveals the zoom features of the openseadragon viewer, as well as the text of annotations. rich text for image annotations is not currently supported, though this is supported in the web annotation specification and is on our roadmap. figure 12. annotations and their corresponding number in the interface   while the annotation module provides a listing of annotations beneath the viewer as a default, islandora solr views can be configured with contextual filters to provide custom views of indexed annotation content, including rich text elements such as links to users and the management page for the annotation object. an example is described in the documentation for the project. when installing the module, a default drupal view that users can modify is also installed and available for use [19]. the annotorious library also provides the annotation features for the simple image viewer in islandora. video.js and open video annotations video.js is a viewer for islandora that provides the default viewer for video and audio objects, as well as the core of the islandora oral histories solution pack [13]. the front end uses the openvideoannotation javascript library developed by center for hellenic studies to enable users to annotate objects in the video.js viewer. openvideoannotation makes use of annotator.js, which is popular library that underpins the hypothes.is project [20] [21] [22] [23]. when a user has permissions to annotate media assets, new buttons provided by the plugin appear in the video.js toolbar (figure 13). users select portions of the timeline to annotate. rich text is supported for this module. figure 13. video.js viewer, configured for annotation with active green annotation selected by user. three new buttons available. the button highlighted in this screenshot shows the regions of the video that have been annotated.   in addition to creating annotations, users can also see where annotations have been added to the video via yellow bars across the timeline of the media being annotated. users can also view statistics for which portions of the media timeline have been most heavily annotated. as with the openseadragon viewer, annotations are viewable in a block. however, unlike the automatic block provided with the simple and large image viewer, the view is powered by solr indexing of annotation content in fedora (figure 14). figure 14. annotations on right-hand region of interface show start and end time of annotations and their values, sorted by start time. view larger image   in both players, permissions provided by the library are respected by the islandora ecosystem and created objects conform to the same object model. idiosyncrasies in the islandora ecosystem our team is committed to robust testing and documentation for the module, as well as more advanced administration features and rational integration with other islandora-contributed modules. we acknowledge that the module has weaknesses in this regard. strictly speaking, the module bypasses some core islandora permissions, in that only the core “view objects” permission is required to utilize the web annotation specific permissions. this means that a user can create annotations, which are islandora objects, without having the core islandora “create objects” permission. however, we have not encountered any practical setbacks as a result of this situation. the biggest practical weakness we have discovered is that the module does not work as expected with islandora’s simple workflow module[24]. this is because that module requires that objects be created in fedora’s “inactive” state, and our annotation containers need to be “active” objects for the module to work. however, we need to support annotations moving through a workflow and tackling this issue is also of concern to our group for future versions of the software. current implementation and release schedule early local uptake of the module is promising. a utsc english professor (anne milne) is using the module in an upper-level undergraduate classroom, where students annotate the detailed engraved plates created by early eighteenth-century artist william hogarth as a method for both exploring and describing his historical and cultural context. since winter 2015, senior students have curated around 200 annotations on photographs of four “series” of engraved plates on the hogarth website [25]. a local oral history project led by history professor chris berkowitz also aims to use the module to enrich existing histories. chris berkowitz and her colleagues have played a seminal user role in the development of these features. oral histories have been collected by chris berkowitz’s class since 2013, representing a large dataset ripe for enrichment in future years. the current release of the module is available on github [3] or our main website [26]. the currently released version has been tested against the islandora 7.x-1.9 release. details of our testing procedures, including supported browsers and themes, are available in the module repository. we aim to test features after the release of each islandora version (currently twice yearly, spring and fall). we are open to hearing from other potential implementers or collaborators regarding this work. conclusion as web annotation continues to become a refined concept associated with community best practices, it may be possible to further consider how to integrate annotation functions into existing repository and digital library frameworks by leveraging tools and data models that are already in place. while our focus is islandora and fedora-focused, we expect to see similar work in other systems. we also believe that all practical work to implement the specification will contribute to additional uptake and use of the standard across systems. there is great potential for digital library programs to enrich existing collections and facilitate web-based scholarly research and collaboration. for islandora users, we hope this may become an approach to annotation that can be agreed upon and supported by the wider islandora community. acknowledgements we would like to acknowledge mark jordan, head of library systems w.a.c. bennett library at simon fraser university for developing the islandora context module. we would also like to acknowledge lingling jiang, metadata & systems librarian at sheridan college, for her initial work on the web annotation content models (initially part of the oral history solution pack). references [1] unsworth, j. (2000). scholarly primitives: what methods do humanities researchers have in common, and how might our tools reflect this? in symposium on humanities computing: formal methods, experimental practice. king’s college, london (vol. 13, pp. 5–00). retrieved from http://people.brandeis.edu/~unsworth/kings.5-00/primitives.html [2] blanke, t., & hedges, m. (2013). scholarly primitives: building institutional infrastructure for humanities e-science. future generation computer systems, 29(2), 654 – 661. https://doi.org/http://dx.doi.org/10.1016/j.future.2011.06.006 [3] barnes, m. e. b., ledchumykanthan, n., pham, k., & stapelfeldt, k. (2017). islandora web annotation utility module. retrieved from https://github.com/digitalutsc/islandora_web_annotations [4] sanderson, r., ciccarese, p., & young, b. (2017a). web annotation data model. w3c working draft. retrieved from https://www.w3.org/tr/annotation-model/ [5] sanderson, r., trust, j. p. g., & rs. (2017). web annotation protocol. retrieved from https://www.w3.org/tr/annotation-protocol/ [6] sanderson, r., ciccarese, p., & young, b. (2017b). web annotation vocabulary. retrieved from https://www.w3.org/tr/annotation-vocab/ [7] banks, nigel (2017). islandora image annotation. github, retrieved from https://github.com/islandora-labs/islandora_image_annotation [8] haslhofer, b., simon, r., sanderson, r., & sompel, h. v. de. (2011). the open annotation collaboration (oac) model. in proceedings of the 2011 workshop on multimedia on the web (pp. 5–9). washington, dc, usa: ieee computer society. retrieved from https://doi.org/10.1109/mmweb.2011.21 [9] sanderson, r., ciccarese, p., & young, b. (2013). open annotation data model. retrieved from http://www.openannotation.org/spec/core/ [10] barnes, m. e. b., ledchumykanthan, n., pham, k., & stapelfeldt, k. (2017). supporting oral histories in islandora. the code4lib journal, 1(35). retrieved from http://journal.code4lib.org/articles/12176 [11] speicher, s., arwe, j., malhotra, a. (2015). ldp linked data platform 1.0. w3c recommendation, w3c. retrieved from https://www.w3.org/tr/ldp/ [12] islandora foundation (2017). islandora solr views. github, retrieved from https://github.com/islandora/islandora_solr_views [13] drupal (2017). views module. drupal.org, retrieved from https://www.drupal.org/project/views [14] jordan, mark et al (2017). islandora context. github, retrieved from https://github.com/mjordan/islandora_context [15] drupal (2017). context module. drupal.org, retrieved from https://www.drupal.org/project/context [16] islandora foundation (2017). islandora openseadragon. github, retrieved from https://github.com/islandora/islandora_openseadragon [17] chute, r., & van de sompel, h. (2008). introducing djatoka. d-lib magazine, 14(9), 10. [18] annotorious contributors (2015). annotorious: image annotation for the web. github, retrieved from https://annotorious.github.io/ [19] barnes, m. e. b., ledchumykanthan, n., pham, k., & stapelfeldt, k. (2017). building an islandora solr view. retrieved from https://github.com/digitalutsc/islandora_web_annotations/wiki/building-an-islandora-solr-view [20] center for hellenic studies (2017). open video annotation project. github, retrieved from https://github.com/ctrhellenicstudies/openvideoannotation [21] annotator contributors (2015). annotator. retrieved from http://annotatorjs.org/ [22] bonn, m., & mcglone, j. (2014). new feature: article annotation with hypothesis. journal of electronic publishing, 17(2). [23] the hypothesis project (2017). retrieved from https://hypothes.is/ [24] durkart, jordan et al. (2017). islandora simple workflow. github, retrieved from https://github.com/islandora/islandora_simple_workflow [25] university of toronto scarborough library. anne milne. hogarth in context (2017). retrieved from: http://hogarth.digitalscholarship.utsc.utoronto.ca/ [26] university of toronto scarborough library. the utsc digital scholarship unit (2017). retrieved from: http://digitalscholarship.utsc.utoronto.ca/ about the authors marcus emmanuel barnes is a software developer in the university of toronto scarborough library’s digital scholarship unit. with over eight years of web programming and application design experience, marcus specializes in creating tools that enhance academic libraries for faculty, staff and students. natkeeran ledchumykanthan is a software developer in the university of toronto scarborough library’s digital scholarship unit. nat has more than eight years of full stack development experience (lamp, php, java, javascript, xml technologies) and is an active volunteer for community digital preservation/library projects. kim pham is the digital projects and technologies librarian in the university of toronto scarborough library’s digital scholarship unit and a liaison librarian for physics. she works on a bunch of great projects, rock climbs, and loves meticulous metadata review as well as code testing. she also has an awesome dog named merle, and she is definitely going to see you at the conference. kirsta stapelfeldt is the coordinator of the university of toronto scarborough (utsc) library’s digital scholarship unit and liaison librarian for computer science. she has been working with islandora and managing digital scholarship/collections projects since 2010. previously at the university of prince edward island, kirsta moved to utsc in 2014 and maintains active connections with islandora through participation on the roadmap committee and islandora board. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – the wise use of statistics in a library-oriented environment mission editorial committee process and structure code4lib issue 6, 2009-03-30 the wise use of statistics in a library-oriented environment as with most businesses, libraries use statistics to justify expenses, to monitor the library’s expansion and to predict prospective developments. this article describes sql and shell techniques for data retrieval as well as further processing of the data using the open source statistical environment r. the article emphasizes some of the pitfalls and reasoning errors librarians could easily slip into. having an academic background on statistics, the author is appointed to projects and tasks which need mathematical and statistical methods to be successfully accomplished. by mathias weyland 1. basic considerations one of the most important questions when working with statistics is “why are we doing this?” actually, this should be asked before any work is undertaken. in science, there is typically some underlying research answering this question. proximate examples for such answers are “to find out if this new drug works better than the established ones” or “to describe the effect of inter-cropping on plant growth” while ultimate answers are “to improve medical treatment” or “to find appropriate cultivation techniques”. it is easy to lose track of these ultimate reasons for conducting a statistical analysis. losing track of the ultimate answer is a high risk especially for the library-oriented environment, where statistics are often computed on a regular basis. in the worst case, the statistics are complied by an it department and then given back to the people who initially requested them for interpretation. factors including staff change, time and the fact that the people interpreting the results were not party to the methods of calculation can lead to misinterpretation or to discarding the results entirely. it is therefore a good idea to keep track of the following procedure: figure 1: statistical workflow furthermore, evaluation of the periodic statistics should be done on a regular base, checking why the statistics were requested in the first place, what matters for these statistics and if the statistics are still required. 2. understanding stochastic processes in publications of all kinds, one often finds statements such as “in 2007, there were 361 deadly traffic accidents in switzerland, this is an increase of 6% compared to 2006” or “graduate students borrowed 361 items from the library this month, which is an increase of 6% compared to last month’s 340.” such a statement seems very easy to interpret at first glance. either the number of loans rises, which is usually considered good [1] , or it falls, right? – not exactly! let’s have a closer look at the loans example: every single user decides, more or less independently of the others, whether to borrow an item or not. the probability of a loan transaction can be influenced by weather, upcoming exams, holidays, opening hours, a cute librarian at the desk and a vast number of other things, but it still is a probability. let’s assume that all of these influencing factors would not change for two consecutive weeks. by understanding this gedankenexperiment [2] as stochastic process, the weekly loan count would most likely be different even though the environment did not change. in case this puzzles you: roll a die twice. the pip count will most likely be different, even though the environment did not change at all. this is quite a disturbing fact in connection with the above loan statistic, because an increase or decrease is not as explicit as we would like. just to show how meaningless the declaration of just two numbers is, i will briefly work out what kind of statement can be established from such a declaration. the nature of those loans (almost independent, small amount of lots of items are loaned) suggests the use of poisson distributions. it was explained above that the loan rate is influenced by environmental factors. this results in two hypotheses about the unknown, real rate λ: either λ does not change during the two consecutive weeks or it did change. we call the first hypothesis the null hypothesis h0 and the second hypothesis the alternate hypothesis ha. don’t confuse this with a change in the weekly loan count. the loan count can change, but even if it does, this does not necessarily mean that λ has changed. it is, however, possible to construct a confidence interval (ci) [3]. in few words, there is evidence against the h0 if the second loan count is not inside the confidence interval around the first loan count. this test can and should be criticized because it relies on approximations and particularly because λ is set to the first loan count, but it has shaped up as a handy quick-test, especially while reading newspapers or annual reports. the math behind all this is: formula 1: statistical formula thus the second loan count of 361 is within the confidence interval, hence not providing enough evidence against the h0, so it’s not possible to say that the hais true, namely that the rates have changed. [4] 3. on communication in a large library, tasks are usually assigned to different work groups, each may be responsible for end user interaction, fees, acquisition of new items, application maintenance, web design, database interaction and so on. this leads to the generation of statistics by some it-related work group for one of the other work groups, especially when data retrieval is non-trivial, for instance when the data has to be retrieved directly from an sql database. this inevitably results in trouble if the workflows are not accurately defined and maintained. for example, a library may outsource some of its rarely requested items to a depot. this would result in more space for frequently used items, but also in higher transportation costs as well as increased waiting time for the outsourced items. in this scenario there will be a requirement to determine the loan counts of the outsourced items to deduce to determine if an item is loaned often enough to stay at the main library or whether it should be moved to the depot. somebody not familiar with the workflow would just count the loans on all items. he may not know that the items from the depot, once requested, are loaned on special service accounts to track the transportation process and once they arrive in the main library, moved from the service account to the user’s account when he comes to pick up the item. this means that two loans are registered when an item from the depot is loaned. it is obvious that this has a serious impact on the statistics which are supposed to help with the decision which items should be moved back to the main library. the quintessence here is that people should talk about their workflows and ensure that the staff sticks to them. once the modus operandi is clarified, the data can be retrieved with sql. however, a mechanism is needed to distinguish between the service loans and the “real” loan. in aleph 500, a so called event is added to an event table called z35 each time an item is loaned. luckily, the system distinguishes between simple loans (the “real” loan, event type 50) and transfer loans (the service loan, event type 56). it is crucial that the loans to be counted are restricted to a specific time period of interest. this ensures that the statistics describes the recent state and does not boost items that were interesting years ago, and even more important, that rarely loaned items acquired a long time ago don’t get boosted because the few loans sum up. the resulting sql looks like this [5]: select z35_rec_key, z35_item_sequence, count(*) from z35 where z35_sub_library=’e16’ and z35_event_type=50 and z35_event_date between ’20080101’ and ’20081231’ group by z35_rec_key, z35_item_sequence order by 3 ; when statistics are given away, it is always a good idea to spend some time thinking about how the result can be used best by the requester. in this case, the rec key and the item sequence are probably not going to be very helpful. ideally, the request process for statistics of any kind should enforce that the requester declares what information should be on the list. in this case, it may be useful to provide title and author of the item [6]: select z13_author, z13_title, z35_item_sequence, count(*) from z13, z35 where z13_rec_key=z35_rec_key and z35_sub_library=’e16’ and z35_event_type=50 and z35_event_date between ’20080101’ and ’20081231’ group by z13_author, z13_title, z35_item_sequence order by 4 ; 4. exploring user behavior some time ago, we had to investigate the user behavior on our webopac to model a benchmark [7]. to design an appropriate benchmark, three things had to be figured out. first we were interested in how many users were using the system simultaneously at a given time of the day, second it was important to know what functions our users were using and third, we had to know how long they were waiting between the clicks. this is referred to as thinking time in the following and is quite important since it has a huge impact in the resource consumption of the system – a shorter thinking time leads to a higher resource consumption. 4.1 amount of web users the first task was the easiest one, the other two tasks needed slightly more complex actions but could be achieved in similar ways. the number of active users is logged by the web server service each time a new user appears. this is an excerpt of the log file: 1 active users : 000000004 2 www-f : find-c 3 2009-02-05 06:16:20 63 [001] [vrb] [...] unfortunately, the relevant line does not include a time-stamp. however, time stamps are present in lines logging other operations. a small awk script was used to collect these time stamps and output the nearest one together with the number of active users: 1 /^2009-02-05 / { 2 d=$1; t=$2 3 } 4 /^active users :/ { 5 l = d " " t "," $4; 6 if(length(l)==29) print l; 7 } the condition length(l)==29 was introduced because the logfile held some malformed lines [8] such as 1 active users : 2009-02-05 08:41:52 11 [...] 084152 running this small awk script parses the log file such that it can be processed further: 1 $ time awk -f 2 > active.awk < www.log > active.log 3 4 real 0m18.719s 5 user 0m18.677s 6 sys 0m0.036s 7 $ head active.log 8 2009-02-05 06:21:05,000000000 9 2009-02-05 06:24:19,000000000 10 2009-02-05 06:24:25,000000001 11 2009-02-05 06:24:28,000000002 12 2009-02-05 06:24:42,000000003 13 2009-02-05 06:24:49,000000004 14 2009-02-05 06:24:56,000000005 15 2009-02-05 06:25:02,000000006 16 2009-02-05 06:25:02,000000007 17 2009-02-05 06:25:07,000000008 this data can be visualized with a scatter plot for better understanding. numerous tools can be used for this purpose – the following listing reads the file active.log, parses the timestamps and draws the plot with the open source statistical computing environment r [9]: 1> raw<-read.table("active.log",sep=",") 2> names(raw)<-c("ts","users") 3> attach(raw) 4> posix_ts<-as.posixct(ts) 5> plot(users~posix_ts,pch=20) 4.2 further tasks as written above, information about what functions were used and about the thinking time can be found using similar approaches. the use of webopac-functions such as searching, listing records or renewing loans are logged and can therefore be counted using simple shell scripts using awk or grep. the thinking time was investigated by parsing the apache log file where time stamps both session cookies and ip were logged. first, the lines related to the cgi were extracted to remove the noise of images and additional files not relevant for the result. then, the time stamps and session ids were extracted and the resulting list was sorted using the session id as primary sort key and the time stamp as secondary sort key with lower priority. the time stamps were parsed with r as shown above and the time difference (δt) in seconds between two consecutive time stamps of the same session id was computed using a for-loop. statistical analysis was then done on δt using r functions such as mean() (arithmetic mean) and sd() (standard deviation) as well as the robust counterparts median() and mad() (median absolute deviation). 5. conclusions albeit much can be done with statistics, it is very important to keep in mind the reason why something is done and to ask if the result is really going to be accurate enough. it is important in the same way to verify that work flows are followed and to be aware that even the raw data can contain errors or be misleading. notes [1] for the other example involving deadly accidents, an increase would be considered bad. [2] a gedankenexperiment, or thought experiment, according to wikipedia, “is a proposal for an experiment that would test or illuminate a hypothesis or theory. given the structure of the proposed experiment, it may or may not be possible to actually perform the experiment.” [3] providing a full introduction to confidence intervals (cis) would go beyond the scope of this article. more information on cis can be found in any introductory book to statistics. [4] it is very important to realize that it is also not possible to state that is wrong with the above calculation only. [5] e16 is the code for the depot library, the rec key identifies the title and the item sequence together with the rec key identifies the item. [6] aleph stores title data in a table called z13, hence this request needs an inner join to the z13 table. [7] the actual benchmark was carried out after the user behavior was reconstructed using jmeter from the apache jakarta project. [8] probably the result of several threads just logging into the file without coordination. [9] r is available at http://www.r-project.org subscribe to comments: for this article | for all articles one response to "the wise use of statistics in a library-oriented environment" please leave a response below: gobals, 2011-09-01 lot of thanks. . for this information leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – using semanticscuttle for managing lists of recommended resources on a library website mission editorial committee process and structure code4lib issue 27, 2015-01-21 using semanticscuttle for managing lists of recommended resources on a library website concordia university libraries has adopted semanticscuttle, an open source and locally-hosted php/mysql application for social bookmarking, as an alternative to delicious for managing lists of recommended resources on the library’s website. two implementations for displaying feed content from semanticscuttle were developed: (1) using the google feed api and (2) using direct sql access to semanticscuttle’s database. by tomasz neugebauer, pamela carson, and stephen krujelskis introduction an important feature of a library website (and subject guides, in particular) is the listing of recommended resources created by librarians. these resources usually consist of links to a catalogue, database, or website. each resource includes 3 parts: a title, a url, and a brief description/annotation. due to the changing nature of web-based resources, it is an ongoing challenge to maintain these lists, ensuring that links are active and appropriate, and editing, removing or adding resources. the same resources often appear on different webpages on the library’s website, so a centralized method of maintaining this content saves time and reduces errors. delicious, the social bookmarking site, was suggested as way of managing lists of resources centrally, saving time and eliminating the need for duplicating content (corrado 2008; corrado and frederick 2008; darby and gilmour 2009). resources could be added to delicious, tagged with subject terms, and rss feeds could be created with tag combinations. corrado (2008) stated that “using a small amount of javascript, these bookmarked resources can be dynamically included in subject guides and other web-based library resources.” concordia university libraries used this method to populate many of its web pages with lists of recommended resources for a number of years. however, in 2011, delicious was sold by yahoo! to avos (karunamurthy 2011). in september 2011, delicious was upgraded and the libraries lost access to delicious feeds for several weeks. again, in march 2012, feeds were unavailable for several days. this prompted the libraries to search for alternatives to delicious for managing content on its website. method semanticscuttle was identified as the ideal replacement for delicious. locally hosted and open source, this php/mysql social bookmarking application had all of the essential functionality of delicious, including a user-friendly interface, but it also came with the stability and reliability required for use on a website that receives nearly 4 million page views annually. figure 1. semanticscuttle web interface to add an item to semanticscuttle, librarians log into the web application, click add a bookmark, enter the item’s url, fill in the title and description, and tag the item with relevant keywords. the tags form field includes an autocomplete feature, helping librarians maintain tag consistency (see figure 2). figure 2. adding an item to semanticscuttle to preview a feed before adding it to the website, tags can be added to the end of the url, with multiple tags separated by plus signs (“+”), (see figure 3). for example: http://[scuttle host]/rss.php/all/videos+2014 figure 3. previewing a feed on the library website, a javascript function allows the librarian to specify the requested tags, define the sorting order (alphabetical or by date, ascending or descending) and limit the number of resources retrieved: <script type="text/javascript"> scuttlefeed("videos+2014", "date_desc", "");</script> figure 4. feed populating the section on the webpage for “our ten most recent video orders of interest” on http://library.concordia.ca/guides/sociology/newvideos.php?guid=ten the source code for the scuttlefeed() function is included below. the list of resources then displays on the webpage as a bulleted list with the resource title hyperlinked to the url and the description (see figure 4). in addition, a useful feature of semanticscuttle is the ability to export all records from the web interface as an html, xml or csv file, allowing for the links to be submitted to a link checker to identify broken links. displaying semanticscuttle content in moodle semanticscuttle’s use is not limited to populating pages on the library’s website. we also developed a moodle ‘library block’ in php that exposes course and discipline specific resources curated by subject librarians in semanticscuttle (see figure 5). the library block uses the direct sql query method described below. to display resources in the library block, a librarian assigns the appropriate course (e.g. moodle_phil201) or discipline (e.g. moodle_math) tag in semanticscuttle. discipline tags are assigned to resources of broad utility and display on all the courses in that discipline, for example, mathscinet for mathematics. to toggle the display of each resource in the library block in moodle, librarians add an additional tag that determines if the resource is to appear or not.   this allows librarians to collect links for courses and disciplines without necessarily displaying them all. figure 5. moodle “library block” for philosophy 201 course. the placement of the library block is controlled by the course instructor and “your librarian recommends” links are controlled by librarians through semanticscuttle tags implementation the use of semanticscuttle involves a combination of php, mysql and javascript, and is summarized in figure 6. figure 6. use of semanticscuttle on the library website and moodle-based course management system two implementations for displaying feed content from semanticscuttle on the library’s website and moodle were developed: using google feed api: asynchronous javascript call to construct a google feed api formatted html for display based on the rss/xml from semanticscuttle. using direct sql query: asynchronous javascript call to a php script that constructs an sql query to the back end of semanticscuttle’s database and returns formatted html for display.   google feed api the google feed api can be used to display any feed on a website. the main advantage of using this api over other methods is that it simplifies the local implementation while leveraging the robust google servers in delivering content to the page. the existence of the google feed api includehistoricalentries() option confirms that google is using caching on its servers to improve performance. placing the google feed api as an intermediary to external rss feeds (e.g., those from journal publishers) makes particular sense, but the fast performance of the api makes its use practical for locally hosted semanticscuttle feeds as well. the google feed api method uses both jquery and the google api, so the following two scripts need to be loaded in the head of the html document: <script src="//code.jquery.com/jquery-1.8.3.min.js"></script> <script src="https://www.google.com/jsapi"></script> the following javascript function takes the semanticscuttle tags, sort, and count parameters, constructs the appropriate http query string to semanticscuttle, and generates the resulting feed display using google feed api: function scuttlefeed(tags, sort, count){ var arrscripts = document.getelementsbytagname('script'); arrscripts[arrscripts.length 1].id=makeid(); var strscripttagid = arrscripts[arrscripts.length 1].id; if (sort == ""){sort="title_asc";} if (count == ""){count=999;} var container = $('#'+strscripttagid).parent(); var imgid=makeid(); html='<img id="'+imgid+'" src="/images/loading.gif" />'; container.append(html); //scuttle does the sorting var feedurl="http://[scuttle host]/rss.php/all/"+tags+"?sort="+sort; google.load("feeds", "1"); function initialize() {    var feed = new google.feeds.feed(feedurl); if (count != ""){feed.setnumentries(count);}      feed.load(function(result) {        if (!result.error) {          ulid=makeid();          html='<ul id="'+ulid+'">';          container.append(html);          containerul=$('#'+ulid);          for (var i = 0; i < math.min(count,result.feed.entries.length); i++) {            var entry = result.feed.entries[i];            html = '<li><a href="' + entry.link + '">' + entry.title + '</a>';            html += '<br />' + entry.content;            html += '</li>';            containerul.append(html);          }        $('#'+imgid).hide();        }      });    } google.setonloadcallback(initialize); } asynchronous loading of rss content into a page uses jquery to append elements to the dom of the page. a helper makeid javascript function is used to generate ids for added dom components such as a new <ul> tag with the content of the feed. making google feed api a dependency in this implementation carries with it some risk that google may discontinue or begin charging for this service. the google feed api terms of service commit to backwards compatibility with the current state of the api only until april 20, 2015. how the interface and license will change after that is unknown.   to mitigate this risk, we experimented with processing rss feeds locally using php’s xml parser. function scuttlelinks($tags,$sort,$count) { // location of scuttle rss $feedurl = "http://[scuttle host]/rss.php/all/$tags?count=$count&sort=$sort"; $feed_raw = file_get_contents($feedurl); // remove colons from xml tags (i.e. namespaces) that can break simplexml $feed = preg_replace("/(<\/?)(\w+):([^>]*>)/", "$1$2$3", $feed_raw); // parse xml $xml = simplexml_load_string($feed); // start an unordered list echo "<ul>"; // loop through bookmarks in rss feed foreach ($xml->channel->item as $bookmark) {     // get properties     $title = $bookmark->title;     $link = $bookmark->link;     $desc = $bookmark->description;     // other interesting properties include pubdate and category     // there can be multiple category attributes     // write list items     echo "<li><a href=\"$link\">$title</a><br />$desc<br /></li>\n"; }  // end list  echo "</ul>"; } since this is a php function, an asynchronous javascript call to it would ensure that the page load is not held up by the processing of long rss feeds. this would be done with jquery’s ajax function which performs an asynchronous http request to process the feed on the server with php.   the following is the implementation of that function: function scuttlefeed(tags, sort, count){ var arrscripts = document.getelementsbytagname('script'); arrscripts[arrscripts.length 1].id=makeid(); var strscripttagid = arrscripts[arrscripts.length 1].id; $.ajax({     url: '/scuttle-ajax.php',     type: "post",    data: {tags : tags, sort: sort, count:count},     success: function(data, textstatus) {        $('#'+strscripttagid).parent().append(data);       } }); } alternatively, as long as semanticscuttle resides on the same domain as the syndicating site, it should also be possible to implement an xml/rss parser using jquery’s parsexml function.   this implementation option would keep the feed processing on the browser side without the use of the google feed api. direct sql query as mentioned above, semanticscuttle is also used for displaying discipline and course specific links in the university’s moodle course management system. in this context, the google api component was replaced with a sql connection and regular expression based query to the semanticscuttle database. this avoided the need for jquery, which was not being loaded by our moodle system at the time of implementation. the semanticscuttle database describes bookmarks in table sc_bookmarks and relationships between bookmarks and tags in table sc_bookmarks2tags. the bookmark id, bid, is the primary key only in sc_bookmarks, since there can be many tag relationships for one bookmark in sc_bookmarks2tags. for example, the following query retrieves a bookmark with bid 5 from the sc_bookmarks table: select bid, btitle, baddress, bdescription from sc_bookmarks where bid =5 the result of the above query is shown in table 1. bid btitle baddress bdescription 5 fortunate son http://clues.concordia.ca/record=b3017339~s0 autobiographical documentary by the son of greek immigrants living in canada. […] table 1. item from sc_bookmarks table table sc_bookmarks2tags contains relationships between bookmarks and tags. for example, the following is a sample of some bookmark-to-tag relationships in the sc_bookmark2tags table. the tags for the bookmark with bid 5 can be retrieved using the following query: select bid, tag from sc_bookmarks2tags where bid =5 the result of this query is shown in table 2. bid tag 5 religion 5 canada 5 2013 5 family 5 videos 5 bronf 5 culture 5 system:imported table 2. item from sc_bookmarks2tags table the system:imported tag indicates that this link was part of a bulk import from delicious using semanticscuttle’s import function. to retrieve all bookmarks for a given tag, or a set of tags, the two tables above must be joined using the bid, concatenating and sorting all tags into one field. for example, the following is a sample sql query that accomplishes this for a bookmark with bid 5: select sc_bookmarks.bid, btitle, baddress, bdescription, group_concat( tag order by tag ) as tags from sc_bookmarks left join sc_bookmarks2tags on sc_bookmarks.bid = sc_bookmarks2tags.bid where sc_bookmarks.bid =5 the result of the above query is shown in table 3. bid btitle baddress bdescription tags 5 fortunate son http://clues.concordia.ca/record=b3017339~s0 autobiographical documentary by the son of greek immigrants living in canada. […] 2013, bronf, canada, culture, family, religion, system:imported, videos table 3. example of join between sc_bookmarks and sc_bookmarks2tags showing a bookmark with all tags instead of searching for a particular bid, the result of this join can also be searched by tag using a regular expression. for example, to search for bookmarks with tags canada, culture, family and videos, the sql query with the regular expression would be: select bdatetime, btitle, baddress, bdescription, group_concat( tag order by tag ) as tags from sc_bookmarks left join sc_bookmarks2tags on sc_bookmarks.bid = sc_bookmarks2tags.bid group by btitle having lower( tags ) rlike "(^|,)canada($|,.*)culture($|,.*)family($|,.*)videos($|,.*)" the regular expression for a given set of tags is constructed using php as follows: // $tags_in is sent in the function call as a string of format "taga+tagb+tagc" // explode this string on the + signs and normalize the tags to lowercase $tags = explode('+',strtolower($tags_in)); // sort the array of tags sort($tags); $regex = '(^|,)'; // next word is beginning of string or is preceded by a comma foreach ($tags as $tag) { // for each tag, include the tag and some punctuation for what comes next, // which could be either the end of the string, or a comma and another tag   $regex .= $tag . '($|,|,.*,)'; } discussion with support for ldap, delicious bookmark import and a firefox plugin, semanticscuttle is a robust open source (gplv2 license) php/mysql application for the management of bookmarks. semanticscuttle is also user friendly, as librarians were quick to adopt it for managing lists of resources on their subject guides. a combination of a presentation, a one-page “quick guide” and some in-person training was sufficient to get librarians using the system. as of november 2014, the libraries’ semanticscuttle holds over 2500 links, and 41 pages on the libraries’ website display this content. ten librarians are using semanticscuttle feeds on their subject guides. semanticscuttle has proven to be a very useful tool for populating library blocks in the moodle course management system. as of november 2014, there are 62 discipline specific feeds (e.g., general resources for political science) and 166 course specific feeds (e.g., specific resources for a political science course such as poli 201). with the exception of tags for courses and disciplines in moodle, the tags in semanticscuttle currently used by librarians are not controlled through a taxonomy or published vocabulary.   the libraries use more than 700 tags in total to describe items in semanticscuttle. future work could include increased control over tag vocabulary for feeds populating webpages. semanticscuttle’s bdescription field for each url does not support html by default, so the addition of hypertext for this field is a functionality to consider for the future. rss format allows for the possibility of cdata html in the <description> field of each <item>. however, since semanticscuttle content is pulled in for display on the university’s website, security is a key concern, and stripping markup (e.g.: javascript) in semanticscuttle’s database is an important component of that security.   therefore, this feature would require the addition or development of a new sanitization function that would allow the basic html but nevertheless offer some level of protection. conclusion semanticscuttle is a relatively simple and powerful solution for managing web resources with the potential for adoption by other libraries. we have outlined multiple implementations for the syndication of web links in semanticscuttle. an interesting question that emerges from this project is that of the extent to which library web links curated by university libraries are re-usable in the context of open data.   although the content in semanticscuttle is accessible through rss, an investigation into existing schemas and controlled vocabularies that could be applied to these resources would be a prerequisite to meaningful open distribution of this data. acknowledgements the authors thank the editors of code4lib journal for their comments and suggestions. references corrado, em. 2008. delicious subject guides: maintaining subject guides using a social bookmarking site. partnership: the canadian journal of library and information practice and research [internet]. [cited 2014 june 13]; 3(2): 1-19. available from: https://journal.lib.uoguelph.ca/index.php/perj/article/view/328/1370 corrado, em, frederick, ka. 2008. free and open source options for creating database-driven subject guides. code4lib journal [internet]. [cited 2014 june 13]; 2. available from: http://journal.code4lib.org/articles/47 darby, a, gilmour, r. 2009. tutorial: adding delicious data to your library website. information technology and libraries [internet]. [cited 2014 june 13]; 28(2): 100-103. available from: http://www.ala.org/lita/ital/files/28/2/darby.pdf karunamurthy, v. 2011 september 27. the first 20 hours. delicious blog [internet]. [cited 2014 april 28]. available from: http://blog.delicious.com/2011/09/the-first-20-hours/ redden, cs. 2010. social bookmarking in academic libraries: trends and applications. the journal of academic librarianship [internet]. [cited 2014 june 13]; 36(3): 219-227. available from: http://dx.doi.org/10.1016/j.acalib.2010.03.004 about the authors tomasz neugebauer is digital projects & systems development librarian at concordia university libraries. pamela carson is web services librarian at concordia university libraries. stephen krujelskis is systems administrator/analyst at concordia university libraries. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – using python scripts to compare records from vendors with those from ils mission editorial committee process and structure code4lib issue 55, 2023-1-20 using python scripts to compare records from vendors with those from ils an increasing challenge libraries face is how to maintain and synchronize the electronic resource records from vendors with those in the integrated library system (ils). ideally vendors send record updates frequently to the library. however, this is not a perfect solution, and over time a problem with record discrepancies can become severe with thousands of records out of sync. this is what happened when, at a certain point, our acquisitions librarian and our cataloging librarian noticed a big record discrepancy issue. in order to effectively identify the problematic records among tens of thousands of records from both sides, the author of this article developed some solutions to analyze the data using python functions and scripts. this data analysis helps to quickly scale down the issue and reduce the cataloging effort. by dan lou the issue of record discrepancies at a certain point our acquisition librarian noticed there were large discrepancies between the records from vendors (axis360 and overdrive) and those in our sierra system. the vendors usually send us regular record updates, but unfortunately overtime, it still became a challenge to keep the information perfectly synchronized. in order to resolve these discrepancies more efficiently and quickly, i was asked to compose a python script to compare record details and identify the exact records we needed to modify in our catalog. comparing records with isbn as the first step i tried to identify the discrepancies in records by comparing isbn numbers. i chose python pandas (pandas 2018[1]) as the data analysis tool for this task. pandas is an open source python package that is widely used for data science and machine learning tasks. pandas is built on top of another python package named numpy that provides support for multi-dimensional arrays, and it works very well with other python packages like matplotlib that is widely used for data visualization. pandas makes it simple to do many of the time consuming, repetitive tasks associated with working with data, including but not limited to loading and converting data, data scrubbing, data normalization, data fill, data merges and split, data visualization, data inspection etc. you can install pandas and all the relevant packages on your local machine but a better option is to start your coding with google colab(google 2019[2]). google colab is a cloud-based python jupyter notebook service and it provides a free tier. by default, pandas and matplotlib are already pre-installed in google colab. to enable pandas in a jupyter notebook in google colab, we simply import the package and give it an short alias: import pandas as pd we exported the relevant marc records from sierra to a comma-separated values (csv) file, and loaded it into the jupyter notebook as a pandas dataframe object. for example, this is how we read a csv file of the axis360 audiobook records exported from sierra: sierra_axis_audio = pd.read_csv( "/content/drive/my drive/library/axis_overdrive/sierra_axis360_audio.csv") this file typically contains columns like title, author and isbn entries. a record sometimes can contain more than two isbn numbers, and those are stored in the “unnamed” columns towards the end for now: sierra_axis_audio.columns index(['title', 'author', 'isbn1', 'isbn2', 'unnamed: 4', 'unnamed: 5', 'unnamed: 6'], dtype='object') we also needed to further clean up all the isbn numbers in order to compare them with those from the vendor. a typical marc 020 field can contain many subfields besides the subfield a for the isbn number. in our exported file, the isbn field usually contains more information than we needed for the comparison. for example, “9780804148757 (electronic audio bk.)“, “9780062460110 : $62.99” etc. i composed a python function to extract the isolated isbn number from each of the columns after the “author” column, and compile all the isbn numbers into a new dataframe object. one sierra record can contain multiple isbn numbers. in this way, i was able to generate a complete list of all isbn numbers from the marc records exported from sierra: def extract_isbns(df): isbns = pd.dataframe() result = pd.dataframe() for i in range(2,len(df.columns)-1): new_isbns = df.iloc[:,i].astype(str) new_isbns = new_isbns.str.split(' ', 1).str[0].str.strip() isbns = pd.concat([isbns, new_isbns]) isbns = isbns.reset_index() isbns = isbns[isbns[0]!='nan'] return isbns in comparison the vendor’s csv file is simple and straightforward. it contains only one isbn field with the clean isbn number. it also has some other fields like title, author, publication date, data added, format, total quantity and total checkouts. all that is left to do is to import these as a pandas dataframe object. next i composed a python function to compare the isbn numbers in the cleaned up data from both sources. i created a new column, “exist,” in both python dataframe objects. if an isbn number from a vendor record exists in a sierra record, the “exist” column on the sierra record has a value of “1”; if not, the value is “0”. the opposite is also true. if an isbn from the sierra record matches that of a vendor record, the “exist” column on the vendor record will have a value of “1” set; if not, the value is “0”. the function also returns two additional dataframe objects that contain the discrepant records only. def find_mismatch(vendor_df, sierra_df, sierra_isbns_df): vendor_df["exist"] = vendor_df["isbn"].astype(str).isin(sierra_isbns_df[0]) not_in_sierra = vendor_df[vendor_df["exist"]==false | vendor_df["exist"].isna() | vendor_df["exist"].empty] sierra_isbns_df["exist"] = sierra_isbns_df[0].astype(str).isin(vendor_df["isbn"].astype(str)) exists = sierra_isbns_df[sierra_isbns_df['exist']==1].drop_duplicates("index") not_exists = sierra_isbns_df[sierra_isbns_df['exist']==0].drop_duplicates("index") not_exists = not_exists[~not_exists["index"].isin(exists['index'])] isbns = pd.concat([exists, not_exists]) sierra_df = pd.merge(sierra_df, isbns, how="left", left_index=true, right_on="index") not_in_vendor = sierra_df[sierra_df["exist"]==false | sierra_df["exist"].isna() | sierra_df["exist"].empty] return vendor_df, sierra_df, not_in_vendor, not_in_sierra after running this script against our records, i did some data analysis to see how many conflicting records were resolved after comparing isbn numbers. i generated the list of all marc records that need to be added to the ils and another list of records that need to be removed. as shown in the following chart, we were able to identify that the majority of our axis360 records were well positioned, but on the overdrive side, about a third of the records are problematic. in total, we still have over 4,600 records in sierra that are troublesome. figure i. vendor vs. sierra: total mismatches by isbn comparing records with title and author fields to further scale down the issue, i moved on to align the remaining mismatched records by comparing the similarity of the author and title fields. a list of mismatched records is one of the outputs from the previous step after aligning the isbn numbers. first, we needed to clean up and extract the title and author field. i made a function to concatenate the title and author fields for each record from both sources. the following function transforms all text to lower cases, removes extra punctuation, extracts the relevant information, and concatenates the data to create a new column, “titleauthor,” for all the records: def extract_title_author(vendor_df, sierra_df, vendor_title_field, vendor_author_field, sierra_title_field = "title", sierra_author_field ="author"): vendor_df['titleauthor'] = vendor_df[vendor_title_field].astype(str).str.lower().apply(remove_punctuations).astype(str) + ' ' + vendor_df[vendor_author_field].astype(str).str.lower().apply(remove_punctuations).astype(str) sierra_df['titleauthor'] = sierra_df[sierra_title_field].str.lower().str.extract("([^[]*\w*)").astype(str).apply(remove_punctuations).astype(str) + ' ' + sierra_df[sierra_author_field].str.lower().astype(str).apply(remove_punctuations).astype(str).str.extract("([ a-z]*)").astype(str) then i used the python module difflib to compare the similarity of the “titleauthor” columns from both sources. the difflib module provides classes and functions for comparing data differences. here is the function to compare and get the similarity level from two sources: from difflib import sequencematcher def get_close_matches_indexes(word, possibilities, n=3, cutoff=0.6): if not n &gt; 0: raise valueerror("n must be &gt; 0: %r" % (n,)) if not 0.0 &lt;= cutoff &lt;= 1.0: raise valueerror("cutoff must be in [0.0, 1.0]: %r" % (cutoff,)) result = [] s = sequencematcher() s.set_seq2(word) for idx, x in enumerate(possibilities): s.set_seq1(x) if s.real_quick_ratio() &gt;= cutoff and \ s.quick_ratio() &gt;= cutoff and \ s.ratio() &gt;= cutoff: result.append((s.ratio(), idx)) after comparing the similarity of the title and author fields, i was able to generate new spreadsheet files for all of the mismatched records. each row in the spreadsheet contains a sierra record together with the most similar record from the vendor side using title and author. all rows are sorted by similarity level (on a scale of 0 to 1). similarity level 0 means the two records in the row are very different; 1 means they are highly identical. again, i made some charts to see how much it would help to scale down the issue further. figure ii. similarity of title and author for mismatched axis 360 audio records figure iii. similarity of title and author for mismatched axis 360 ebook records figure iv. similarity of title and author for mismatched overdrive audio records figure v. similarity of title and author for mismatched overdrive ebookrecords as shown in the charts, if we take a similarity level of 0.7 as a benchmark, roughly two-thirds of the mismatched records required more cataloging effort. with the python scripts we successfully narrowed down a problem of 30,843 records to that of 3,394 within a few days. only the 3,394 records require further examination by our cataloger. this reduced the cataloging effort eventually needed to fix record discrepancies. figure vi. comparing 30,843 records from sierra and vendors (overdrive and axis360) conclusion it has become an increasing challenge to keep the marc records of online resources up to date between the library’s ils and the vendors’ provided records due to the frequent changes taking place. the solution described in this article is a good remedy to identify and fix record discrepancies accumulated over time. i highly recommend libraries adopt this method and make it a routine task to prevent the issue having a snowball effect. it is better if we can insert in the vendor provided marc records a customized field to identify the valid date range. then we can develop a script to automatically renew or modify the records that are close to or past the due date by retrieving them from the ils system via record exporting or via api. many vendors tend to not generate a 404 error page when a library loses access to an online resource like an ebook, so the script will need to be developed to be versatile enough to detect those links that stop working properly. on the other hand, a huge change has been taking place in recent years as the record system of online resources slowly eases out of the library’s ils. for example, library online service provider bibliocommons has implemented bibliocloudrecords which allows libraries to add certain online resources records via api to the library’s catalog website without needing to add the marc records to the ils. while this improves the resource access for library customers, it means patron data and usage statistics have inevitably shifted from libraries to the vendors. online resources have had a stronger presence in the library’s collection since the pandemic and will become more and more influential in the foreseeable future. it is a good question to ask now how libraries could better position ourselves in this new online resources landscape. references [1] pandas. 2018. python data analysis library — pandas: python data analysis library. pydataorg. https://pandas.pydata.org/. [2] google colaboratory. 2019. googlecom. https://colab.research.google.com/ about the author dan lou is a senior librarian at palo alto city library, where she works on web content management and develops pioneering projects on robotics, coding, ar and distributed web. her particular interests include python, machine learning and data analytics. previously, she worked as a technical project manager at wolfram alpha, a knowledge base queried by apple siri and amazon alexa. before that, she was a systems librarian at innovative interfaces. author email: loudan980@gmail.com subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – wayfinding serendipity: the bkfndr mobile app mission editorial committee process and structure code4lib issue 42, 2018-11-08 wayfinding serendipity: the bkfndr mobile app librarians and staff at st. john’s university libraries created bkfndr, a beacon-enabled mobile wayfinding app designed to help students locate print materials on the shelves at two campus libraries. concept development, technical development, evaluation and ux implications, and financial considerations are presented. by valeda dent, kiichi takeuchi, ben turner, heather ball, caroline fuchs, ann jusino, and shilpa karnik introduction in 2017, members of the st. john’s university libraries’ web and emerging technologies group (wemtech) embarked on a mobile wayfinding project using beacon technology, which would assist students in finding print materials on the library shelves. inspired by the navapp developed at the university of oklahoma libraries (aruba networks, 2015), the st. john’s university libraries’ bkfndr app provides a way for students to navigate the often-confusing library stacks and make their way to relevant materials at the shelf level. the goals of the project were two-fold: to assist students in becoming more familiar with the libraries’ physical collections and layout; and to support the serendipity of browsing via mobile technology. this article details the project implementation at the queens campus library. beacon technology has been around since 2013, when apple introduced its ibeacon technology (newman, 2014). beacons can be found in retail, museum, hospital, airport, and other settings where wayfinding is important (moody, 2015; chawathe, 2008; kajioka, 2014; newman, 2014). beacons are passive devices – they emit a signal that simply identifies where they are physically. the beacons are not aware of the devices that receive or process the signals they emit. beacons do not collect data, are usually platform independent, and although beacons may be referred to as being part of the internet of things (iot), they typically do not connect to the internet without an additional layer of software that can interpret their signals. the beacons used for this project are bluetooth low energy (or ble) beacons. beacons often work with bluetooth technology which allows bluetooth-enabled devices to pick up the signals emitted by beacons once activated. mobile devices can also be used as beacons since they are typically bluetooth-enabled. as devices pick up signals being emitted by the beacons, the devices can identify the distance from the beacon, or the distance between multiple beacons. in this way, devices become location aware or location sensitive (newman, 2014). the st. john’s university libraries’ bkfndr app required the placement of beacons on the library shelves of the circulating collection, and then mapping of each beacon to an exact location. these locations were defined by shelf ranges, thus allowing each beacon to be mapped to a specific shelf content area. the number of beacons on any given shelf is related to the signal range of the beacons, and the degree to which those signals can be kept from interfering with one another. beacons placed too close to one another might give students multiple location readings for the same item, and beacons placed too far apart might not get students close enough to the materials they are looking for. literature review there are a variety of technologies that enhance location awareness for users, including near field communication (nfc), global positioning systems (gps), and quick response code technology. the literature suggests that beacon technology is far more accessible (moody, 2015), mainly because all three of the aforementioned technology types require the user to initiate a pull for information as opposed to having information pushed to their devices. wayfinding and location-aware technologies are useful in a variety of settings, in part because of the potential for integration with local systems already in place and the application of analytics to enhance the user experience (sturari et al., 2016). hospitals and other healthcare settings are particularly interested in improving the user experience, and timely provision of updates as a patient moves through various destinations (such as check-in, radiology, er, surgery) is one way to improve these experiences for both patient and family (dutta & palta, 2014). the application of beacon technology in library and information settings is less common than in healthcare and retail, yet there is great potential for adoption in libraries. “the potential for beacons is limited only by the applications that are developed to work with them, so it will be fascinating to see how libraries use them in the future.” (spina, 2015). application developers also recognize the potential for wayfinding technologies to be used within library settings. capira technologies and bluubeam are two examples of vendors who focus on library and information settings (enis, 2014). capira’s beacons will offer general notifications, and since capiramobile is already designed to communicate with a library’s ils in order to support functions such as self checkout and patron record notifications, its beacons will also be able to send personalized messages to individual patrons when they are in range. (enis, 2014). babu (2016) suggests that the 12 areas where beacons will become more prevalent are hotels/hospitality; retail; airports; schools/universities; hospitals/healthcare; banks; museums; zoos/theme parks; smart cities; events, and libraries. babu (2016) also details the top use cases for beacon technology within libraries: shelf information book circulation notifications event updates navigation support booking study rooms (babu, 2016) babu (2016) presents the example of the bavarian state library in munich, which uses more than 200 beacons in 90 different locations to help with wayfinding in a large complex library building. chia (2014) describes the development of a 3d wayfinding app at the nanyang polytechnic library in singapore, which uses the conveno wayfinding tool and leverages a cloud computing platform. the app directs users to specific items in the stacks as well as to rooms and collections. similarly, librarians at the university of urbana-champaign developed a an app to help users navigate the print collections, using student testing and input to improve the app, which resulted in the addition of features like a search box that allows users to search by book title, and visual enhancements such as display of the book’s cover (hahn & morales, 2011). the fayetteville free library in new york and new jersey’s somerset county library system have both experimented with the use of beacons to enhance library user experiences (enis, 2014). other location-aware library apps serve a marketing function by promoting services and collections to users. the borough of manhattan community college (bmcc) library initiated a beacon project that was designed to fill the spaces where librarians could not always be present. the library placed six beacons in locations such as the stacks, display areas, and other public areas where users might need to be alerted to the presence of services and key collections (eng, 2015). the orange county library system began using bluetooth-enabled beacons in 2014 to promote new video acquisitions and to highlight featured library collections. mount pleasant public library in new york state placed an ils-integrated beacon near its circulation desk which alerts users when their items are due, or when holds are available for pick-up (dempsey, 2016). the navigation app at the university of urbana-champaign also made location-based recommendations to users (hahn, 2011). at delft university of technology in holland, librarians used 15 bluetooth-enabled beacons to create a self-paced tour of the library, partnering with a dutch start-up that developed apps for museum tours (dempsey, 2016). technical development beacon selection the product selected for the st. john’s university libraries’ wayfinding project was the radius network beacon. this product was selected because of the affordable price ($10 per beacon if more than 100 are purchased), the variety of beacon hardware available (configured and without configuration), and the well-supported configuration apps. cost funding for the project was provided by the provost’s initiative fund, a special fund designed to get start-up projects off the ground with the intention of eventually making these new services or products permanently available for the university community. at the time of this writing, st. john’s university libraries had spent approximately $12,865 usd on bkfndr, including the beacons ($2920 for 280 beacons), app developer kits, and software development costs (approximately 153 hours @ $65/per hour). there were also indirect costs related to the initial stages of the project, namely, the amount of time that the wemtech group and the developer spent discussing and planning for the design of the app. the most significant cost was related to the collection maintenance and catalog work that had to be completed so that the location features of the app would accurately reflect materials on the shelves and the catalog. the university libraries had not conducted a thorough shelf reading project in many years, thus this was a major initiative that included many hours of work. approximately 46 students, 4 staff members and 3 faculty on the two campuses worked on the project beginning in the late spring of 2017. to date, student worker costs have amounted to about $43,000 for both the queens and staten island libraries. libraries undertaking a similar project should ensure the accuracy of the catalog and shelf list prior to development. this will save time and money. app design the wemtech group at st. john’s university libraries facilitated design of the app. the group discussed the following basic requirements during the development phase: catalog search capability (fig. 1) interactive library map display (fig. 2) graphically represented user location display graphically represented book location display figure 1. library catalog search screen (enlarge) figure 2. map screen layout (enlarge) architecture figure 3 illustrates the data flow between the client and server for the bkfndr app with the deployed beacons. the libraries at st. john’s university use koha, an open source integrated library system (ils), and the catalog search is the first prompted input required of the user. once the user enters search terms, results are returned by the catalog, and the user selects the item they want from the list (see fig. 1). the next screen provides bookshelf-level location details facilitated by the detection of beacon signals (see fig. 2). as the user moves within the space, the app continuously updates their location in relation to the item on the target bookshelf until they are in the right spot. figure 3. client and server architecture diagram technology stacks in the initial development phase, a sample app was designed based on the ibeacon implementation guide from apple (2014). this first test app was built to help determine the strength of the beacon signals. programming was accomplished using swift 3 (apple, 2018a) and xcode (apple, 2018b). in the production version, the ionic framework (ionic, 2018) was used. ionic is a free, open source, multi-platform software development kit that is used in the creation of mobile applications. this was an important consideration for the st john’s team since students function in both ios and android mobile environments. two other important considerations in using the ionic framework were the previous experience of the project’s developer in using this resource and the availability of the ibeacon plugin within the ionic ecosystem. the latest ionic technology layers are based on the angular framework (angular, 2018), and most of components and business logic are implemented in typescript. the presentation layer was implemented using ionic components with sass. the ionic framework uses webpack as the build system (iconic, 2018), and the debugging environment is on the local web server. unfortunately, when testing with the ibeacon application, the app has to be deployed on a physical device, and deployment is a time-consuming process. to overcome this slow debugging process, a timer-based function was created to emit dummy signals within the web browser, thus eliminating the need for the developer to use the device to test the app over and over again. the preparation of this simple beacon simulation code in early development phase was instrumental in terms of making the rest of development phases more efficient. the server-side modules for the app are also critical to overall function. initially, bookshelf and beacon location data were hardcoded within the app itself. those datasets eventually needed to be published as a representational state transfer (restful) web service because of the dynamic quality of the app. the database size is expected to remain relatively small, with no more than an estimated few hundred items. st. john’s university libraries already had in place a hosted linux server running mysql, and the data was imported into that same server. backend services in php were implemented to publish the records and to make them searchable. the project relied mostly upon open source technology, which significantly reduced the overall development cost. mapping and translating datasets at the beginning of the project, several options for facilitating the mapping features of the app were explored. these included arcgis, cad data, and a few third party indoor gis solutions. in the end, standard web technology (html5 / css3) was selected to render the target marker. this choice kept development costs down. the rendering of the actual bookshelf maps was informed by a process used at wayne state library to facilitate locating materials on the shelves (gallagher, 2010). librarians integrated map coordinates as an overlay on highlighted bookshelf areas featured on the library’s floor map (gallagher, 2010). in the case of st. john’s, the raw images were used to create the graphic designs. the mapping of the library bookshelves at st. john’s involved the acquisition of the original blueprints as a high-resolution image file. the database containing the bookshelf call number ranges was then constructed using mysql. once all the records were imported, an administrative module was created to support the mapping of each bookshelf on the administrator’s mobile device (fig. 4). using the administrative module, the target bookshelf name was selected from the dropdown menu at the top of the screen, the corresponding bookshelf location touched on the screen, and the new position saved. figure 4. administrative screen for beacon management (enlarge) since the screen size of mobile devices can vary greatly, and the app relies heavily on a graphic representation of bookshelves, it was important to display the relative coordinates (fig. 4, top left corner) as a percentage rather than as an absolute pixel length. location algorithm the concept of using mapping technology in a setting like st. john’s university libraries required both an understanding of the way that mapping technology works in traditional navigation environments (such as roadways) and the challenges of using the same conceptual framework indoors. using beacons to navigate users step by step is neither practical nor precise because of the variability in beacon deployment (apple, 2014). the ble beacon used for this project informs the user of the signal strength via its received signal strength indicator (rssi). this signal is strongest at 0 and decreases as it weakens. it is important to note that the rssi is a measure of signal strength only: it is not a measure of the physical distance between the beacon and the device. to orient the user in the library, a blue dot appears on the screen in relationship to the closest beacon; a red dot marks the shelf location of the item selected by the user (see fig. 2). however, initial field tests showed the blue dot bouncing here and there, and not necessarily in relation to the user’s location. the problem was the stability of rssi, causing a user with a handheld device standing between two beacons to receive conflicting information about the location of the nearest beacon. if the signal strength for one beacon is -35 dbm, and the closest beacon to that one has a signal strength of -50 dbm, the app directs the user to the first beacon since it has a stronger signal at that moment. if those signals change in relation to the handheld device being moved slightly in the next moment, the app may then direct the user to the second beacon even though the initial signal reading was accurate. to address the temporary spike in the signals, the development team at st. john’s created a timeline-based algorithm to control the variability of the signal strength. essentially, the algorithm allowed for the weighted total of signals emitted within a certain timespan to be used as an indicator of the beacon likely to be closest to the user. once the algorithm was in place, attention was focused on beacon placement. zou et al. (2016) found 6.5 m (about 21 feet) to be an effective range in terms of beacon signal coverage. beacons were thus deployed at both ends of each library bookshelf, about 15.4 feet (or 4.7 m) apart, well within the range suggested by zou et al. (2016). deployment configuration & placement the circulating and reference collections at the queens campus library consist of four wings – a north and south wing – on the 3rd and 4th floors. these wings consist of shelves where the beacons were installed. as of the writing of this article, 20 beacons had been placed on top of bookshelves within the library’s 3rd floor north wing (fig. 7). upon completion of the north wing, another 200 beacons will be deployed in the remaining three wings at the queens campus library. at the staten island campus library, a total of 60 beacons are in the process of being configured and installed. the beacons are not visible from the ground and are relatively safe from theft or damage. each beacon was set up initially with the lowest strength and frequency in order to conserve energy and allow operation for approximately 6 months before needing a battery change. to facilitate the precision of the positioning and mapping features, each beacon was identified by a universally unique identifier (uuid), a major coordinate, and a minor coordinate. uuid is set by the manufacturer based on the iso/iec 11578:1996 standard (iso, 2018). major and minor are 16-bit identifiers which need to be set by the developer. using this naming convention, a mapping schema was created to document the major and minor settings (fig. 5). bookshelf bookshelf number front (0) / back (1) beacon major minor figure 5. mapping schema in this schema, bookshelf 3n05-f will have a beacon whose coordinates are 5 for the major and 0 for the minor. all bookshelves for this phase of the project have the same prefix of 3n which is the library’s 3rd floor, north wing. the postfix, f, represents the front of bookshelf (fig. 6). this configuration and designation also allowed library staff to locate a beacon if for some reason it got moved or lost. lastly, a general pin was assigned to all of the beacons to provide configuration security. user experience, user testing results & implications overall, the bkfndr app adhered to the most basic principles of ux design for mobile applications: an uncluttered interface; minimization of cognitive load for the user; offloading of certain tasks (e.g. bkfndr helps users fill in the title of the item they are looking for); and minimization of user input (babich, 2018). many elements beyond the control of the developer can impact mobile ux, thus user testing is critical. members of the wemtech team performed preliminary field testing of bkfndr on the 3rd floor north wing of the queens campus library and found that the app performed satisfactorily in directing them to shelves that contained specific titles. in a second round of field testing, wemtech recruited four students, and one alumnus, for a total of five test users. one of these students was a library student worker. participants were each given a different title of a book that resided on the 3rd floor north wing and were instructed to use the app to find the item in the stacks. participants used the “think aloud” protocol (bastien, 2010) to allow greater insight into their thoughts and experiences as they attempted the task. a member of the wemtech team accompanied and observed each participant. according to nielsen (2012), testing with five users can reveal the majority of usability problems, whereas using more than five tends to produce duplicate results and diminishing returns on time and money invested. the app was not available in google play at the time of testing, so only iphone users were included in the initial field tests. once the app becomes available in google play, user testing will also be conducted with android users. in general, the app worked as expected, and all of the test users were successful in entering the book titles and eventually locating the book in the stacks. three test users took advantage of the “autocomplete” feature of the app that saved them from typing the complete title into the search box. the app successfully identified the correct shelf for four of the five books, giving the test users a clear path to finding them. in one case, the app was slightly off, and directed the user to a stack next to the correct one. user feedback was generally positive. one student mentioned that it would be helpful if the red dot marked the location of the book, rather than just the shelf, but still appreciated that the app expedited the process of finding a book in the stacks. another test user who was entering her senior year admitted that although she had regularly used the library’s electronic resources, had only used the physical library on rare occasions, and that she would have had difficulty locating a book using the call number alone. even the student who was initially directed to the wrong stack was able to use the call number provided by the app to direct her to the correct one. a common problem encountered by test users was in orienting themselves in relation to physical space and the map displayed by bkfndr. in three of the five tests, the user had to stop for a few moments to look around the 3rd floor north wing and back at the map to situate themselves. in two of the five cases, this problem was compounded by a delay in the appearance of the blue dot (that identifies the location of the user). in all of the tests, the blue dot tended to lag slightly behind the location of the test user, which could be somewhat disorienting to an individual trying to get his bearings in the library. these are problems that the wemtech will address in upcoming modifications of the bkfndr app. another difficulty test users encountered was in attempting to read the shelf numbers found in the maps. (the shelf numbers are used primarily by library staff for shelf-maintenance functions but are potentially helpful for library users.) three test users attempted to zoom in to the map to read the shelf numbers by spreading their fingers on the screen, but bkfndr does not allow zooming. the members of wemtech will need to consider removing the shelf numbers from the maps altogether, enlarging them to make them more easily legible, or enabling users to zoom in so that they can actually read them. evaluation of the bkfndr app will align with the recommendations made by neilsen (2018) whereby the initial discovery of usability challenges will be addressed until resolved. an iterative approach to testing is also in place so that as new features are added, the testing can focus on these features accordingly (neilsen, 2018). figure 6. common bookshelf/beacon configuration figure 7. location of deployed beacons and call number ranges (enlarge) next steps & future direction bkfndr is currently available in the itunes app store, and completion of the beacon installation at both the queens and staten island libraries is on target for september 2018. after the hardware installation is complete and the software is updated accordingly, additional user testing will be conducted. the app has not yet been promoted outside of the libraries, but the wemtech team has created a marketing campaign consisting of social media, print, and web-based strategies that will be launched once user testing and final updates conclude. the bkfndr app has also been awarded a campus technology impact award for 2018 (https://campustechnology.com/articles/2018/08/07/a-mobile-guide-to-library-resources.aspx?page=1) in the student systems & services category – this will help in the promotion of the app. the novelty of the bkfndr app may be attractive to students, but novelty is fleeting – especially when it comes to mobile apps. to that end, wemtech intends to promote the bkfndr app as part of a larger suite of productivity apps designed to motivate students to use the libraries. these apps include inquery (a research support app, currently available in itunes and google play stores) and the “st. john’s university libraries” app – a general library app that allows users to search the catalog and check out books using their mobile devices. as highlighted by our user testing, there is room for improvement, most notably refinement of the location indicator, and better initial location orientation for the user (possibly by adding a compass or directional arrow on the screen). these challenges are being addressed for the next update of the mobile app. conclusion the bkfndr app is one of the few applications that provides users the opportunity to find materials on library bookshelves using their mobile devices. the app supports students who know what they are looking for and want a quick way to locate the items on the shelves, as well as those who want to browse. facilitating serendipity in an important outcome. students who browse library collections and find materials that are useful are more likely to return. the simple design of the app (which consists of just two screens) hides a sophisticated configuration that makes use of one of the more accessible wayfinding technologies on the market – beacons. references angularjs — superheroic javascript mvw framework. [accessed 2018 apr 13]. https://angularjs.org/. apple, inc. 2014. getting started with ibeacon. [accessed 2014 jun 2]. developer.apple.com/ibeacon. babich, n. a comprehensive guide to mobile app design. [accessed 2018 aug 7]. https://www.smashingmagazine.com/2018/02/comprehensive-guide-to-mobile-app-design/. babu, pavithra. 2016 apr 7. top 12 industries that beacons will disrupt in 2016. blog.beaconstac.com. [accessed 2018 apr 13]. https://blog.beaconstac.com/2016/04/top-12-industries-that-beacons-will-disrupt-in-2016/. bastien, j.m.c. 2010. usability testing: a review of some methodological and technical aspects of the method. international journal of medical informatics 79(4): 18-23. borrego-jaraba f, garcía gc, ruiz il, gómez-nieto má. 2013. an nfc based context-aware solution for access to bibliographic sources in university environments. journal of ambient intelligence and smart environments 5:105–118. bradley j, henshaw n, mcvoy l, french a, gilbertson k, becksford l, givens e. 2016 apr 25. creation of a library tour application for mobile equipment using ibeacon technology. [accessed 2018 apr 13]. https://vtechworks.lib.vt.edu/handle/10919/71832. chawathe ss. 2008. beacon placement for indoor localization using bluetooth. in: 2008 11th international ieee conference on intelligent transportation systems. p. 980–985. chen g, xin y, chen n-s. 2017. informal learning in science museum: development and evaluation of a mobile exhibit label system with ibeacon technology. education tech research dev 65:719–741. doi:10.1007/s11423-016-9506-x. chia wy. 2014 apr. mobile 3d library map: an innovative wayfinding tool for library users on the move. library hi tech news:9–10. doi:10.1108/lhtn-03-2014-0017. chu h-c, hwang g-j, tseng jcr. 2010. an innovative approach for developing and employing electronic libraries to support context?aware ubiquitous learning. the electronic library 28:873–890. doi:10.1108/02640471011093552. dempsey k. 2016. bluetooth beacons are starting to shine in libraries. computers in libraries 36:28–31. dutta a, palta a. 2014. location-awareness based data dissemination for smart hospitals. in: 2014 international conference on medical imaging, m-health and emerging communication systems (medcom). p. 72–75. enis m. 2014 nov 18. “beacon” technology deployed by two library app makers. library journal. [accessed 2018 apr 13]. https://lj.libraryjournal.com/2014/11/marketing/beacon-technology-deployed-by-two-library-app-makers/. enis m. 2015. technology: library app makers deploy beacon tech. library journal 140:20–20. fast facts | st. john’s university. [accessed 2018 apr 13]. https://www.stjohns.edu/about/administrative-offices/provost/institutional-research/fast-facts. gallagher p. 2010 jun 22. map it @ wsu: development of a library mapping system for large academic libraries. library scholarly publications. getting started with ibeacon.pdf. [accessed 2018c apr 13]. https://developer.apple.com/ibeacon/getting-started-with-ibeacon.pdf. hahn j. 2011. location-based recommendation services in library book stacks. reference services review; bradford 39:654–674. doi:http://dx.doi.org/10.1108/00907321111186677. hahn j, morales a. 2011. rapid prototyping a collections-based mobile wayfinding application. journal of academic librarianship 37:416–422. doi:10.1016/j.acalib.2011.06.001. hahn j, zitron l. 2011. how first-year students navigate the stacks: implications for improving wayfinding. reference & user services quarterly 51:28–35. hocking c. 2014. the beacon experiments: low-energy bluetooth devices in action. shine solutions group. [accessed 2018 apr 13]. https://shinesolutions.com/2014/02/17/the-beacon-experiments-low-energy-bluetooth-devices-in-action/. ionic. build amazing native apps and progressive web apps with ionic framework and angular. ionic framework. [accessed 2018 apr 13]. https://ionicframework.com/. kajioka s, mori t, uchiya t, takumi i, matsuo h. 2014. experiment of indoor position presumption based on rssi of bluetooth le beacon. in: 2014 ieee 3rd global conference on consumer electronics (gcce). p. 337–339. moody m. 2015. analysis of promising beacon technology for consumers. elon journal of undergraduate research in communications 6. [accessed 2018 apr 13]. http://www.inquiriesjournal.com/articles/1136/analysis-of-promising-beacon-technology-for-consumers. nielsen, jakob. why you need only test with 5 users. nielsen norman group. [accessed 2018 aug 7]. https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/. nielsen, jakob. how many test users in a usability study? nielsen norman group. [accessed 2018 aug 6]. https://www.nngroup.com/articles/how-many-test-users/. newman n. 2014. apple ibeacon technology briefing. journal of direct, data and digital marketing practice; basingstoke 15:222–225. doi:http://dx.doi.org/10.1057/dddmp.2014.7. rodriguez-sanchez m c., martinez-romo j. 2017. gawa – manager for accessibility wayfinding apps. international journal of information management 37:505–519. doi:10.1016/j.ijinfomgt.2017.05.011. spina c. 2015 may 21. keeping up with… beacons. association of college & research libraries (acrl). [accessed 2018 apr 13]. http://www.ala.org/acrl/publications/keeping_up_with/beacons. sturari m, liciotti d, pierdicca r, frontoni e, mancini a, contigiani m, zingaretti p. 2016b. robust and affordable retail customer profiling by vision and radio beacon sensor fusion. pattern recognition letters 81:30–40. doi:10.1016/j.patrec.2016.02.010. the university of oklahoma libraries launches navapp to provide mobile indoor navigation and content for more than one million visits annually 2015 | aruba networks newsroom. [accessed 2018 apr 13]. http://news.arubanetworks.com/press-release/university-oklahoma-libraries-launches-navapp-provide-mobile-indoor-navigation-and-con. zou h, jiang h, luo y, zhu j, lu x, xie l. 2016. bluedetect: an ibeacon-enabled scheme for accurate and energy-efficient indoor-outdoor detection and seamless location-based service. sensors 16:268. doi:10.3390/s16020268. acknowledgments the authors wish to thank anthony todman, alyse hennig, and judy bland from the st. john’s university libraries and matt cook, emerging technologies librarian, university of oklahoma libraries for their support on this project. author biographies valeda dent, ph.d. is dean and professor, university libraries at st. john’s university in new york. dr. dent has created online and mobile learning tools at long island university, rutgers university, and hunter college, and collaborated with kiichi takeuchi on the development of several library-based mobile apps including researchplus™, inquery, and bkfndr. kiichi takeuchi is a senior web application database developer at long island university (liu) in new york where he also serves as an adjunct professor. takeuchi is also the chief technology officer for objectgraph, a company that focuses on mobile and web application development. takeuchi has created more than 140 mobile apps, including many that employ geospatial technology. takeuchi works closely with academic libraries and has created numerous mobile library apps including research plus™ and inquery – the only mobile apps designed for academic library environments around information literacy standards. both apps are available in both ios and android stores. benjamin turner is associate professor, university libraries at st. john’s university in new york. he holds an m.l.i.s. from the university of british columbia and an m.a. from hunter college, cuny. his research interests include information literacy, library assessment and user experience. presently serves as assessment librarian for instruction and collections at st. john’s university libraries. he has previously published in the global librarian(2013), and information technology and libraries (2015). heather ball is the student success librarian and assistant professor at st. john’s university. she holds an mls with dual certificate in preservation and archives, as well as an mlitt and bachelor’s in medieval studies. she sits on university committees, and has been invited to present her research at several conferences this year. her research interests include qualitative/quantitative data analysis, assessment measures, information literacy instruction, digitization and encoding of historical manuscripts, and twelfth-century britain. caroline fuchs is associate dean and learning design librarian at st. john’s university. she holds an mls, an m.a. in english and an m.a. in history. she is a senior fellow at the vincentian center for church and society, and is a member of the web and emerging technologies team. she teaches courses in the division of library and information science, and serves on the board of directors for association of college and research libraries. ann jusino is an associate professor of the st. john’s university libraries where she has served as the systems administrator for the koha integrated library system since 2010. she is actively involved with many emerging technologies projects, including the development of the libraries’ mobile apps and website redesign. jusino is a long-standing member of the waldo library consortium, serving as secretary of the waldo executive board for the last 5 years. shilpa karnik is the associate director of emerging technologies in st. john’s university libraries in new york. she holds an m.l.s and a b.s. in computer science and mathematics. a member of the libraries’ web and emerging technologies committee, karnik plays a key role in exploring and leveraging emerging technologies in the mobile environment. she likes to be challenged with the complexity and dynamism of connections between users and technology based systems. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – iroam: leveraging mobile technology to provide innovative point of need reference services mission editorial committee process and structure code4lib issue 13, 2011-04-11 iroam: leveraging mobile technology to provide innovative point of need reference services the university of northern british columbia’s geoffrey r. weller library can boast of a healthy and stable reference service. while statistical analysis reveals that patron use of this service is on the decline, this is not unlike current trends experienced by many libraries today. the library averages a total of 6300 reference transactions per year, a significant number for a small, research-intensive university serving 3500 fte. the unanswered question is why are the numbers dropping? one theory is that providing research and reference assistance in a traditional manner is affecting the number of transactions. reference service is traditionally provided in a stationary manner, whereby patrons are required to visit the reference desk of their own volition. recognizing that a stationary librarian cannot reach a stationary patron, unbc library began an innovative roaming reference pilot project in september, 2010. combining the power of wireless networks, tablet computing and chat services, 5 librarians provided point-of-need, face-to-face and virtual reference services during peak reference hours over the fall 2010 semester. this article outlines the project and technologies employed to make it happen (ipad, apps, instant messaging widgets and wireless networks). by james r.w. macdonald and kealin mccabe introduction face-to-face reference transactions at the university of northern british columbia’s geoffrey r. weller library have been declining at an average rate of 9.8% annually; conversely virtual reference (vr) transactions have grown an average of 500% per year. the overarching goal of iroam roaming reference was to regain control of our declining reference statistics. through the promotion and provision of iroam roaming reference, we hoped to increase student awareness of reference services as a whole, while increasing the visibility of librarians and library services at the same time. this article examines the authors’ technical decisions and chosen components in designing a roaming reference service, including: chat service software, the ipad, apps and wireless networking. reference services at the geoffrey r. weller library – a background prior to the launch of iroam in september 2010, the library provided traditional reference services from monday to friday from 11am to 6pm and sundays from 1pm to 5pm. since 2008 the library has also provided its patrons with access to virtual reference (vr) via the provincial chat reference consortium askaway. students contact the askaway service–which is powered by oclc’s question point software–through an html-based widget or “qwidget” on the library’s homepage. just as quickly as they have embraced new technologies, students have embraced vr, so much so that unbc’s use of the service has increased by over 1000% since its implementation. in an effort to gain some insight into why our reference desk statistics were on the decline while virtual reference statistics were skyrocketing, it was determined that we needed to take an in-depth look into where our patrons were when accessing the library’s services. due to privacy restrictions put in place by our consortial chat service, we were unable to analyze patron ip addresses. this information would have provided us with a great deal of data about our patrons’ exact locations on campus, even possibly answer why they increasingly prefer virtual interactions with librarians over face to face. to circumvent this obstacle, a study of the library server logs was undertaken. this study allowed us to see that on average 61% of library page requests over a three year period were placed from somewhere on campus. given these statistics we began to feel safe in the assumption that roaming reference services would help our stationary librarians reach our stationary patrons. what is iroam? fall 2010 marked the launch of iroam: roaming reference using ipads. the library purchased 5 ipads in the summer of 2010 to be used for the purpose of this project. roaming reference services were offered during our peak reference hours in addition to our traditional reference services. through the analysis of reference statistics and library server logs, we were able to determine the hours roaming reference should be provided: monday-wednesday between 1 and 4pm. during roaming shifts, librarians would roam the floors of the library and various campus outlets with the goal of assisting students at their point of need. roaming the library looking for patrons presents another problem. while a stationary librarian may never reach a stationary student, those with the need for a librarian may have quite a difficult time locating a moving target. therefore, instant chat services were explored as a means of “paging” the roaming librarian both by students and the library’s circulation and reference desk staff. by providing a chat widget on the library’s website, students could chat with the ipad-carrying librarian. if the student happened to be on campus they could give the librarian their location and the librarian would come to them. if the student was not on campus the librarian would conduct a chat session with the patron using the ipad. using this model, the service was piloted from september to the end of november, 2010. why the ipad? roaming reference service employing tablet computing is not new, as demonstrated in holly hibner’s 2005 article describing a roaming reference service using a tablet computer. until recently tablet computers have been heavy, lacked sufficient battery life and had slow start up times. when apple announced the launch of its ipad in january 2010 it seemed as though the aforementioned deficiencies would finally be resolved. at 9.7 inches the screen is larger than most netbooks, but the ipad only weighs approximately 1.5 pounds. an impressive battery life of 10 hours means that forgetting to plug in the ipad after a roaming reference shift does not lead to a dead battery halfway through the next (apple 2010). the ipad operates much like a smartphone, instantly going from a sleep state to useable at the push of a button, and by utilizing push notifications it can audibly and visually update the librarian of an incoming chat message much like text messaging on a smartphone. all things considered, the ipad appeared engineered to be the reference librarian’s best friend. roaming reference service could be delivered using any number of tablet computing devices. at the time of this pilot project the ipad answered concerns about weight, battery life, screen size and price. at just over 500 dollars the ipad will arguably be the least expensive tablet computer for some time (chen 2011). choosing a chat service the number of chat service software applications available is nearly overwhelming. among them are the obvious and inexpensive solutions, such as meebo, microsoft messenger, aim and google talk. these types of services are generally free and developed for personal use. as a result, they tend to lack the more robust features found in enterprise software such as chat queues, canned responses, multiple chat operators, chat transcripts and reports. any chat software used in this project needed three main features: a chat widget from which library patrons could communicate with librarians; an ipad application capable of interfacing with the chat software; and the capacity to have multiple librarians simultaneously monitoring incoming chats requests. karen mccoy’s webjunction online conference presentation, implementing reliable instant messaging at your library (2010), ranks chat services into three tiers: bronze, silver and gold. the chat reference software featured in mccoy’s gold level discussion is libraryh3lp. libraryh3lp is a chat service designed by and for librarians and offers many of the features found in enterprise level chat software solutions. libraryh3lp’s greatest value, however, is its software architecture. libraryh3lp was built on the xmpp or jabber standard chat protocol which makes it possible to pick from a variety of chat aggregators using the same protocol. in other words, while libraryh3lp does not have a purpose-built ipad application for sending and receiving chats, there are dozens of chat apps in apple’s app store capable of working with libraryh3lp. selecting the right apps the two essential ipad apps required for this project were a chat application capable of receiving chats over the jabber protocol and a web browser. the chat application ultimately selected for this project was beejive. beejive offers a clean and intuitive interface for handling multiple incoming chats, it easily receives messages from a variety of chat services and employs push notifications to inform the librarian of incoming messages. push notifications appear anytime a patron sends a chat to the librarian when the beejive app is not open. as a result the roaming librarian can respond to queries whether they are in another ipad app or have the ipad in sleep mode. finding the perfect chat application for this project was difficult. beejive met the essential requirements for the project but could be greatly improved upon. for example, using beejive the librarian cannot transfer patrons to other librarians at the end of a shift. it does not have the ability to capitalize on a canned response feature. canned responses include simple messages that can be sent to a patron to answer frequently asked questions, such as: how do i login to my account? or, when is the library open? the addition of these features would greatly improve the librarian’s chatting experience. apple’s app store is a rapidly changing environment (148apps.biz 2011). by the time this article is published there will be thousands of new apps available and possibly improvements to the apps discussed here. at the time of this project there did not appear to be an app available that could take full advantage of all the features of robust chat service software such as libraryh3lp. there are several enterprise chat software services that provide purpose-built ipad applications; both liveperson and bold chat are good examples. their suitability for supporting roaming reference services is worth investigating. the combination of beejive and libraryh3lp provides the minimum feature set necessary to support roaming reference services. finding the perfect combination of chat app and chat software service to support roaming reference clearly requires, short of developing the software in house, further investigation. the second essential ipad app is the web browser. the librarian needs access to the library catalogue and other online resources. there are a variety of web browsers available for the ipad besides apple’s safari browser. at the time of this pilot project the ipad’s safari browser did not render ebsco-hosted databases very well. alternative apps to safari include: icab mobile, life browser, and perfect browser. perfect browser appears to live up to its name. this app offers tabbed browsing, easy access to bookmarked pages and appears to render library resources such as ebsco databases very well. as an added bonus, perfect browser offers vga-out support for displaying webpages through a projector, a necessity if librarians wish to use the ipad in classroom instruction. [1] connecting with students virtually and face-to-face being a member of the askaway provincial virtual reference consortium presented a small problem for the roaming reference pilot project. how would patrons respond to an additional chat widget on the library’s homepage? would there be a general confusion about which chat widget to use? was there enough real estate on the library’s website to house two widgets? to address these concerns it was decided to present only one chat widget to the library patron at a time. therefore, when roaming reference was available the consortium widget was removed from the library’s homepage and the roaming widget was added. as a result, all of the library’s chat reference was handled by local roaming reference librarians during roaming service hours: mondays to wednesdays from 1pm to 4pm. presenting website visitors with the right widget at the right time presented a small technical problem: how to remove and replace the opposing widgets three days a week? the solution was to create an alternate library homepage containing the roaming reference chat widget. the library’s systems analyst simply created a server cron job to replace the default library homepage containing the askaway widget with a page containing the roaming reference widget during the hours the service was available. in this way the library website’s visitors were only ever presented with one chat option but always connected to the most appropriate service. since this roaming reference service was simply a pilot project, changing the library homepage worked very well. it did, however, make it necessary to make any updates to the library’s homepage in two places. this method would also be impractical for including the chat widget in more than one place, such as: research guides, databases or other online resources. the wireless network while presenting the right widget at the right time is crucial, without a stable wireless network the librarian would never reach a virtual patron. at unbc, like many other institutions, there are two options for connecting to a wireless network: through a secure virtual private network (vpn), or through a captive portal to an unencrypted open connection. unfortunately, unbc was not in a position to support a vpn connection from the ipad to its wireless infrastructure when this project began. connecting through a captive portal was not a practical option either. captive portals force a user to login through a web browser before gaining access to the internet. anyone with experience using captive portals at airports, hotels, and restaurants recognizes that this method is often unstable and impractical for anything beyond simple web browsing. however, by adding the devices’ mac addresses to the wireless system’s gateway the ipads could instantly connect to the institution’s unencrypted network. essentially the wireless router recognizes the device when it attempts to connect and automatically allows the device access. since the ipad, like many modern wifi-enabled devices, can remember wireless networks once this relationship is established, the librarian is not required to re-authenticate or take any other special steps to make subsequent connections to the wireless network. while connecting to the network in this manner is extremely convenient it also comes with some security risk. traffic being passed over this type of network is unencrypted if the site does not have a valid security certificate. the library, therefore, has taken steps to obtain security certificates for its catalogue and proxy services. unbc is currently working on deploying an on-campus secure wireless network which bypasses the need for a vpn. regardless of the state of an institution’s wireless infrastructure, communication between the library and those responsible for the network is essential to the success of a roaming librarian. is the ipad the reference librarian’s “best friend?” with all of the hardware and software in place the geoffrey r. weller library was prepared for a highly successful service. unfortunately, the ipad came up lacking in a couple of key areas: multitasking and an app designed to deliver this type of service. when roaming, the ipad was an excellent tool for navigating the library’s catalogue and databases for quick information retrieval. however, extensive searching using a variety of research databases seemed cumbersome on the small touch screen. roaming librarians reported that they oftentimes found themselves using the patron’s computer or the closest opac station if a question required in-depth searching. if the question happened to come through chat, the librarian would be faced with a similar conundrum. unfortunately, using the closest computer would force the librarian to resort to typing links into the ipad or typing out detailed navigational directions, a time consuming and cumbersome process. since the ipad did not offer multitasking or “fast app switching,” moving between the chat conversation and the web browser, notepad or other applications resulted in a slowed conversation and may have led to abandoned chats or frustrated librarians and patrons. by late november, 2010 many of the shortcomings described here were addressed. apple released an update to their ios operating system which brought multitasking capacity and “fast app switching” to the ipad. the ability to quickly navigate between the chat application and other ipad apps should alleviate much of the cumbersome navigation experienced during the pilot. ultimately, a custom-built ipad app supporting canned responses and chat question transfers would go much further in addressing these problems. not every librarian using the ipad took to the device equally. there were two unexpected surprises regarding the adoption level of the ipad. in the case of one librarian, considered to be an early technology adopter in most cases, the ipad proved to be the exception. for this librarian pen and paper is still superior to finger and screen. in the other instance a librarian generally reticent to technological change–she is still running windows xp and office 2003–appears to have fully adopted the ipad into her routine. dropbox is a feature of her desktop, emails from her often include the signature “sent from my ipad,” and one can often find her annotating pdfs and creating or editing documents on the ipad. future directions the ipad is an exciting new technology which has set an initial standard for tablet computing. this being said, there are a variety of tablet computers appearing on the market. the ipad’s light weight, long battery life and instant-on technology make it almost an ideal tool for roaming reference. however, as these technologies are incorporated into ever thinner and lighter laptop computers the laptop may become the device of choice for this type of service. apple’s recent release of the macbook air is a worthy competitor to the ipad in weight, battery life and instant-on technology. a laptop brings the added advantage of a full keyboard and mouse, which may be more conducive to in-depth searching. the least expensive option may be to simply use an ipod touch or a smartphone. a smartphone would add the benefit of cellular data plans, thus bypassing the need for a wireless connection where the wireless infrastructure at an institution is not reliable. this option would work best where the library catalogue and its databases are configured for smartphones [2]. as some of the librarians in this project found that they were gravitating toward the nearest opac station or the patron’s laptop to answer reference queries, an ipod touch or smartphone may be all that is needed. conclusion the technology used in this project proved to be an excellent addition to reference services at the geoffrey r. weller library. roaming reference questions accounted for 8% of all reference questions in the fall 2010 term. this is equivalent to all email and phone reference over the same period. the significance of 8% becomes clear when one realizes that the roaming reference service was only available 9 hours a week compared to 39 hours of regular reference. it has long been argued that a stationary librarian may be missing out on as many as two thirds of library patrons who will not approach the reference desk (swope, 1972). the results of this roaming reference pilot project appear to substantiate that claim. while the information landscape is rapidly changing, the traditional reference desk often appears to be an unmovable institution in many academic libraries. as the authors analyze patron feedback and reference statistics a new model of reference services seems to be emerging for the geoffrey r. weller library. in 1984 an article appeared in collegiate microcomputer arguing that patrons should have the capacity to communicate with librarians through terminals placed throughout the library (smith). nearly 25 years later it appears that all the pieces of technology are finally available to provide such a service in a meaningful way. the authors have plans to continue this project into the winter term. it is hoped that what is learned from this project will help to remodel the way reference services are delivered at the geoffrey r. weller library. notes 1 requires an apple 30pin to vga or dvi connection cable or a 30 pin to hdmi connection cable. 2 higher end ipad models are also available with 3g cellular data plans. references 148apps.biz [internet]. [2011]. count of active applications in the app store; [cited 2011 mar 25]. available from: http://148apps.biz/app-store-metrics/?mpage=appcount apple [internet]. [2010]. markham (on): apple (canada) – ipad – technical specifications and accessories for ipad [apple]; [cited 2011 jan 7]. available from: http://www.apple.com/ca/ipad/specs/ chen, brian x. why nobody can match the ipad’s price [internet]. 2011. [place of publication unknown]: wired, gadget lab; [updated 2011 feb 18; cited 2011 mar 25]. available from: http://www.wired.com/gadgetlab/2011/02/ipad-price/ hibner holly. 2005. the wireless librarian: using tablet pcs for ultimate reference and customer service: a case study. library hi tech news. 22(5) 19-22. (coins) mccoy, karen. technology essentials 2010 online conference: implementing reliable instant messaging at your library [internet]. 2010. [place of publication unknown]: webjunction; [updated 2010 feb 2; accessed 2010 march 2]. available from: http://www.webjunction.org/conference-tech-essentials-2010/-/articles/content/90952660. smith, dana e., hutton, steven m. 1984. back at 8:00 am – microcomputer library reference support programs. collmcmp. 2(4), 289-294. (coins) stpeter. about: jabber.org [internet]. 2009. [place of publication unknown]: jabber.org; [updated 2009 apr 2; cited 2011 jan 7]. available from: http://www.jabber.org/about/. swope, mary jane and katzer, jeffrey. 1972. why don’t they ask questions? ref users serv q. 12 (2) 161-166. (coins) about the authors james r.w. macdonald is the web services librarian at the geoffrey r. weller library, university of northern british columbia. his research interests currently focus on usability and the user experience in digital environments. kealin mccabe is the research and learning services librarian at the geoffrey r. weller library, university of northern british columbia. she is deeply involved in reference and instruction practices throughout the province and is the winner of the british columbia library association (bcla) 2011 academic librarians in public service award for outstanding service. subscribe to comments: for this article | for all articles 10 responses to "iroam: leveraging mobile technology to provide innovative point of need reference services" please leave a response below: terri karlseng, 2011-05-23 i’m very interested in the use of digital technologies in library instruction practices. is there any way i can keep up with your research? james macdonald, 2011-05-26 hi terri and everyone else reading this, we are working on an analysis of the statistics kept during this project and hope to publish it soon. we’ll post the citation here when it is available. july 18 – wait, what?, 2011-07-18 […] macdonald, j. & mccabe, k. (2011). iroam: leveraging mobile technology to provide innovative point of need reference services. the code4lib journal, 13. available at http://journal.code4lib.org/articles/5038. […] ipad??????20??? | nalsi???????iii, 2011-09-27 […] 3?roaming reference?????????ipad??????????????code4lib?13????????iroam: leveraging mobile technology to provide innovative point of need reference services […] james macdonald, 2011-12-19 as promised here is the link to our latest article on this topic. published in the library journal – partnerships: http://journal.lib.uoguelph.ca/index.php/perj/article/view/1496/2263 james macdonald, 2012-01-13 just thought i’d share this wordpress plugin – designed by our web developer – for auto rotation of chat widgets based on a pre-determined schedule. it is fantastic and allows you to place your widgets outside of wordpress too. see here: http://wordpress.org/extend/plugins/timed-widget/ arccentric, 2012-03-05 just out of curiosity; do you plan on ever applying this towards an alternate platform that would be android based? james macdonald, 2012-04-10 @arccentric – we have no plans to deliver the program with android based tech. however, the program is really technology agnostic and i think easily adaptable to android based solutions. ipads for reference « ualr library tech blog, 2012-05-29 […] reinvigorating reference through point of need service. using the ipad for reference services. iroam: leveraging mobile technology to provide innovative point of need reference services horizon report like this:likebe the first to like this post. posted in emerging technologies, […] traci, 2013-09-25 i am considering a proposal for a conference session on the changing needs of reference and reinvigorating reference. do you have any recommendations for speakers? thanks leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – a hybrid solution for improving single sign-on to a proxy service with squid and ezproxy through shibboleth and exlibris’ aleph x-server mission editorial committee process and structure code4lib issue 18, 2012-10-03 a hybrid solution for improving single sign-on to a proxy service with squid and ezproxy through shibboleth and exlibris’ aleph x-server this paper describes an implementation of a hybrid solution for improving the library’s proxy service by integrating shibboleth and exlibris’ aleph’s x-server using a proxy server running both ezproxy and squid applications. we will describe in detail the hybrid solution of a proxy service within the context of our institution and explain how this service improves the user experience. we will explain how we developed and implemented this solution with a minimum of development cost and describe some of the preliminary empirical research undertaken. the main benefit of this solution is that instead of relying on e-resource vendors to become shibboleth-compliant, we are able to prepare and deploy a shibboleth-ready environment while granting our patrons reliable and stable access to e-resources via different types of connections. as of december 2011, the hybrid solution is running in production. by alexander jerabek and minh-quang nguyen definitions proxy service, in this library context, is a service that monitors and facilitates access to e-resources. this service acts as an intermediary between an e-resources vendor and libraries who offer these resources to their users (see http://en.wikipedia.org/wiki/proxy). proxy server is a computer system that has proxy services. squid – open source software released under the gnu general public license – is a proxy caching server for web clients. the squid application allows authenticated users off-campus to access e-resources such as proquest, heinonline, and springerlink. squid provides features such as: filter access to e-resources, validating user input, and reduce redundancy to limit access to sites’ traffic providers (see http://wiki.squid-cache.org, http://en.wikipedia.org/wiki/squid). squid requires that the browser is configured with a proxy-auto-configuration (pac) file that contains a list of domains that need to be proxied. ezproxy is proprietary software that provides the same functionality as squid but unlike squid, ezproxy has a shibboleth module (see http://en.wikipedia.org/wiki/ezproxy). ezproxy does not require any configuration of the browser (e.g. no need to add a pac file) but it needs to add a prefix url to the e-resource’s url to be proxied. for example, where the original e-resource is http://ieee.org, its new link is http://proxy.bibliotheques.uqam.ca/login?url=http://ieee.org. shibboleth is a federated identity management system that uses an authentication method based on the security assertion markup language (saml) standard. if different applications or web services use the same authentication method via shibboleth, they just need to logon once and other applications will benefit from this logon information (i.e. single sign-on) (see http://shibboleth.net and http://www.moodle.uqam.ca/docs/index.php/shibboleth). the shibboleth service is becoming more and more a standard and accepted authentication method for institutions and libraries (see http://www.canarie.ca/en/caf/participants) that provide controlled access to e-resources, and vendors such as ieee, sciencedirect, and proquest have shibboleth implementations as well. [1] in this article, we define the term “shibboletized” to mean an action taken that makes an application provide authentication via shibboleth. exlibris’ x-server aleph is a restricted web service developed by exlibris. this service accesses the aleph system that manages confidential library patron data. background in 2003, the information technology team (itt) of the university of quebec at montreal’s (uqam) library service implemented its first proxy server running with the squid application. there are around 800 requests daily for access to e-resources that are controlled by a proxy. this number can increase to up to 1500 requests at peak sessions. in 2009, the computers and telecommunications service team of uqam (sitel – service de l’informatique et des télécommunications) shibboletized its first application, the e-learning moodle platform (see http://moodle.uqam.ca). in 2010, itt shibboletized its primo discovery tool http://virtuose.uqam.ca and subsequently the library’s website http://bibliotheques.uqam.ca. in 2011, sitel shibboletized the uqam human resources’ website http://www.rhu.uqam.ca. at the end of this year, itt shibboletized the proxy service. challenge there is actually no way to use the shibboleth service as a provider of access control with a squid proxy application unless we develop a new squid-shibboleth application. this represents a substantial development cost for our service. so we needed to find a solution at a lower cost that would at the same time offer better service, improve user experience, and remain transparent. squid has been in use for over eight years at uqam. both ezproxy and squid have their own advantages, and we wanted to see if we could combine these. the advantage of squid is its ease of use when it’s configured correctly. moreover, there is no need to provide an explicit reference to the proxy in the url for squid. in other words, one can click on any url on the web and still get redirected to the proxy. on the other hand, ezproxy can trigger shibboleth functionality; there is no need to configure the browser client. a disadvantage of ezproxy is that the url needs to be rewritten to point to the ezproxy server. after some consideration, we decided to keep squid’s workflow on our server, add ezproxy to it, and allow both services to be used with shibboleth. this decision brings us to a challenge whose technological implementation must satisfy certain constraints: preserving the habits of users, i.e. continue to use squid’s workflow with a pac file enabling shibboleth within ezproxy. avoid the development of complex source code as much as possible. providing portability to facilitate the evolution of the product. be prepared to deploy our single sign on functionality when all vendor e-resources have been shibboletized. the hybrid solution integrates all three applications — shibboleth service provider, ezproxy and squid — on the same server. obviously, to make this solution work, we needed to make squid and ezproxy communicate with each other. important questions raised are: how to achieve this communication between squid and ezproxy at a lower cost? how to activate shibboleth within squid without changing its source code? how to redirect the domain proxied from the pac file to the ezproxy service instead of the squid service without changing the habits of end users? this communication between the two services aims to take the outgoing data of squid, transform it to ezproxy’s format, and then provide it as input for ezproxy. because ezproxy can invoke shibboleth, if the squid information can be redirected to ezproxy, the job will be almost done. the technical solution required will be to translate all those tasks into a computer program; in this case, we’re using php scripting. another challenge is that we have to add the ezproxy prefix link (i.e. http://proxy.bibliotheques.uqam.ca:2048) to all our aleph bibliographic records that contain links to e-resources. we have presently some 98,598 records for e-resources. this is a precise global modification that must be made in a database containing over 1 million records. this addition will allow users who do not want to add the pac file to continue to access e-resources through our primo discovery tool, virtuose http://virtuose.uqam.ca. this work was done in august 2012 at the time we were writing this article. methodology to achieve our solution, we have chosen for the system architecture of the proxy server a linux cent/os with squid, ezproxy, shibboleth service provider (sp), apache 2.2.3 and with php for the programming scripts. as for the security of our servers, we use an approved wildcard ssl certificate. there are three separate physical servers: (1) one shibboleth identity provider (idp), (2) one proxy server for squid, ezproxy, apache, and shibboleth service provider (sp), and (3) one server for exlibris’ aleph x-server service. to create our computer program, we first have to collect the log data (e.g. some 300 urls of suppliers) from squid, ezproxy, shibboleth, and apache to build a corpus of data. it helps us to analyze different ways urls are transmitted from squid to ezproxy. we start by separately running squid first, then ezproxy without shibboleth, and finally ezproxy with shibboleth. we examined the log data file of each service to retrieve the url syntax through each proxy server. the result of the analysis of log data file indicates where to intercept the information for the reprocessing and redirecting from and to the appropriate service. following this analysis, we compare the various features offered by ezproxy and squid then, from these differences, write a scripting code that can exchange information between these two proxy server implementations. after several empirical tests, we found that apache error’s script can be used for the exchange information between these servers. we will see this detail in the following sections. operation with squid proxy service without shibboleth prior to the implementation of the ezproxy/squid combination described in this paper, the information technology team (itt) library service at uqam provided the squid software that is used principally to validate the identity of uqam’s users. the user must first authenticate and be authorized before access is permitted to the requested e-resource. authentication is against ldap – lightweight directory access protocol – while the authorization is provided by exlibris’ aleph x-server, a restricted web service that accesses the confidential library patron data stored in the aleph system. one of the elements of this confidential information is the status of users (i.e undergraduate student, graduate, professor, employee, etc.) and their expiry date. for example, a visiting student may be able to use the library loan services but may not access restricted e-resources. figure 1. squid proxy workflow in production until december 2011. figure 1 describes a workflow of a url request using the squid proxy. to use the squid, the url request should be primarily declared in a pac file. this file is added to the browser’s proxy settings. here is a sample extract of the proxy auto-configuration file format: function findproxyforurl(url, host) { if ( dnsdomainis(host, "ejournals.ebsco.com") || dnsdomainis(host, "ejournals.library.ualberta.ca") || dnsdomainis(host, "search.proquest.com") return "proxy 12.345.67.89:3128"; else return "direct"; } the user enters a url request (1) such as http://search.proquest.com. when the pac file is turned on, the browser executes the findproxyforurl() function (2) that will call the proxy server or go direct to e-resources. if the domain request is not in the pac file, the browser goes directly to e-resources without authentication (3) (i.e. return “direct”) and the e-resource is then displayed. otherwise, the findproxyforurl() function calls squid proxy (i.e. return “proxy 12.345.67.89:3128). the squid proxy invokes ldap (4) for authentication (5 and 5 ‘). the user is prompted to enter his id and password (5”) which is validated against the active directory (6). after the return of authentication (6’), squid calls the x-server (7) to validate the authorization of the user. it is at this point that his status and expiration date will be checked (e.g. a graduate can borrow materials but no longer has the right to access online resources). we also see in this last step that the squid service, after confirming the authorization (7’’), is unable to alert the user if he or she is not authorized to access the resource (e.g. you do not have the right to refer to this resource). this is due to the squid’s script running in the background on the proxy server. this is a significant functionality gap of squid, whereas with ezproxy we can interact with the user by providing informative messages. moreover, we can also customize the web pages and informational messages sent by ezproxy. we’ll see how this is possible in the next section. operation of the proxy server with squid/ezproxy and shibboleth in this hybrid solution, the pac file has been modified so that the squid output data feeds into apache (port 3128 in the previous model is replaced by port 80). function findproxyforurl(url, host) { if ( dnsdomainis(host, "ejournals.ebsco.com") || dnsdomainis(host, "ejournals.library.ualberta.ca") || dnsdomainis(host, "search.proquest.com") return "proxy 12.345.67.89:80"; else return "direct"; } with this modification, squid and apache should be operating on the same port 80 at the same server. in fact, it is impossible to bind two processes on the same port on the same server. we stop the squid process, but still activate and maintain the forwarding to port 80 in the pac file, and then run the apache and ezproxy services. even though the squid service is stopped, the redirection of urls to the proxy server is always assumed by the pac file and in this way is a part of the squid’s workflow showed in figure 1. to this workflow we will add the ezproxy and shibboleth services to achieve the hybrid solution. when an url request arrives on port 80, apache will interpret and replace with the hostname of the proxy server as shown in the example below: original request comes from the user’s browser: http://search.proquest.com/results/1366ab0876f332b6d17/1/$5bquery after apache interprets and replaces: http://proxy.bibliotheques.uqam.ca/results/1366ab0876f332b6d17/1/$5bquery apache replaced the search.proquest.com by proxy.bibliotheques.uqam.ca. this url, however, does not exist on our proxy server. therefore, apache returns an error code 404 (i.e. not found). the key to the solution is here. we must modify apache’s 404 scripting file to correctly rewrite the link that contains the call to ezproxy: http://proxy.bibliotheques.uqam.ca/login?url=http://search.proquest.com/results/1366ab0876f332b6d17/1/$5bquery once ezproxy is called, all this is easily configurable at the ezproxy level without having to implement complex developments. to create an ezproxy session, we used the ‘ticket’ method from among the three existing ezproxy methods: external-cgi method, local method, and ticket method. the choice of the ticket method is justified by its simplicity to implement, without being any less secure, and by its flexibility in being able to integrate customized codes such as calls to the aleph x-server and to shibboleth. figure 2. at present (december 2011) new hybrid proxy solution has been implemented. figure 2 shows a new method for validating users. as in figure 1, when the user enters a url request (1), if the url request is not in the pac file, the browser will be forwarded directly to the e-resource without authentication (3) (i.e. return “direct”). otherwise, this url request is redirected to apache on port 80. apache rewrites the query using the 404.html scripting file (3’) and sends it to ezproxy. the ezproxy service calls the shibboleth service (4), which is on another server. after logging in (5, 5’and 5’’), the ezproxy receives a reply with the validation of the user (6, 6′). if the user is properly authenticated, ezproxy will call the x-server (7) to determine the authorization. after this last step, the e-resource may be permitted or prohibited depending on the result of x-server reply (7’). note that ezproxy can issue informative messages to the user when access is refused (e.g. your account has expired, please contact our loan service). for adding the ezproxy prefix link, we launched a single process in aleph to update all 98,598 records of e-resources that must be proxied and we run a daily process that updates all new e-resources. conclusion with this new solution, we achieved our goal of deploying a shibboleth-ready environment while granting our patrons reliable and stable access to e-resources via different types of connections. future work although we chose ezproxy because it allows integration with shibboleth, interaction with the user via informative feedback messages, and its easily customized interface that requires little development effort, there are other ways to set up a proxy such as the apache reverse proxy module. we also intend to run ezproxy on ports 80 and 443 instead of 2048 and 2443. we will test this possibility in future work. endnotes [1] see http://www.ieee.org/publications_standards/publications/subscriptions/clientservices/info/faq_clientservices.html, http://www.info.sciverse.com/sciencedirect/implementation/quicktips/faq/shibboleth_access, http://www.proquest.com/en-us/aboutus/pressroom/09/20090209.shtml, https://spaces.ais.ucla.edu/download/attachments/28082219/shibtf-final-reportrev9202010.pdf?version=1&modificationdate=1285016932320. about the authors alexander jerabek is a librarian in the information technology services at uqam libraries. he has over 15 years experience in libraries and library systems. he maintains and develops the primo discovery tool, and is partially responsible for aleph, sfx, and metalib. minh-quang nguyen, ph. d is a computer analyst for information technology team of uqam’s library service. he obtained his ph. d. in cognitive computing in 2008 at uqam. researcher in cognitive computing, his research is based, from theory to experiment, on a cognitive approach in the field of spoken human-machine interaction. acknowledgement we would like to express our gratitude to all those people who helped us complete this work. first of all, mr. pierre roberge, the director of the information technology team of uqam’s library service, who always encourages and supports us, and gave us all the time and material necessary to complete this work. mr. pierre nault, our itt systems librarian, provided us with crucial information about the aleph system and we thank him for his brilliant idea for how to optimize the request to the aleph x-server. he also was responsible for injecting the ezproxy prefix link into aleph database. also, many thanks to the members of sitel, especially, a thought in memory for daniel pouliot, computer analyst, who spent almost all of his time for helping us to implement shibboleth; he was the shibboleth guru at uqam. and mme stéphanie lanthier, computer analyst, who has spent much of her precious time setting up and configuring our shibboleth environment. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – managing library it workflow with bugzilla mission editorial committee process and structure code4lib issue 11, 2010-09-21 managing library it workflow with bugzilla prior to september 2008, all technology issues at the university of colorado denver’s auraria library were reported to a dedicated departmental phone line. a variety of staff changes necessitated a more formal means of tracking, delegating, and resolving reported issues, and the department turned to bugzilla, an open source bug tracking application designed by mozilla.org developers. while designed with software development bug tracking in mind, bugzilla can be easily customized and modified to serve as an it ticketing system. twenty-three months and over 2300 trouble tickets later, auraria’s it department workflow is much smoother and more efficient. this article includes two perl template toolkit code samples for customized bugzilla screens for its use in a library environment; readers will be able to easily replicate the project in their own environments. by nina mchale background & introduction the auraria library is the academic library for three institutions of higher learning: the university of colorado denver (ucd), metropolitan state college of denver, and the community college of denver. the combined student fte is 28,000; however, part-time enrollment on campus is very high, which makes for a total student enrollment of nearly 50,000. the campus is also almost entirely a commuter campus, which puts a premium on library space; there are no dorms to which students may retreat in between classes, and the library is quite busy during most days of the academic year. the 169 public computers are always in high demand. network support, including email and wireless networks available in the library, is provided via the ucd information technology services department, the ucd campus-level it unit. in 2008, the recommendations of an external consulting effort aimed at improving library workflows prompted a reorganization of the auraria library systems department, embedding a number of systems staff members into other departments and introducing the use of student assistants for the first time ever. in addition, two employees with over 50 combined years of institutional knowledge retired from the department in this year. these factors emphasized the need to coordinate technology issue reporting and resolution effectively across the library as the greatest concentration of technology skills was no longer centralized in one department. prior to the reorganization, the six-person systems department had managed desktop support in an ad hoc fashion. a dedicated phone line, checked frequently during the library’s open hours, served as the primary means of reporting technology issues. in january of 2009, the network administrator and desktop support technician positions were re-filled and four student workers were hired with the intent that this team would be able to provide most of the library-wide desktop support, freeing other systems staff members from these operational duties. in particular, this would allow the it manager to pursue larger-scale projects that would benefit the library technology environment such as server virtualization and thin clients for public workstations. as the responsibilities of the systems department (re-titled the division of technology strategy and learning spaces) were re-envisioned, its director sought ways to improve efficiency and communication both within the department and beyond to the entire library. this necessitated a more formal means of tracking, delegating, and resolving reported issues. to accommodate these drastic personnel changes and achieve this improved efficiency, the library’s it staff recognized the need to modernize workflow management using a powerful communication and tracking tool. finding the right tool in order to find an appropriate tool, it staff investigated half a dozen or so open source or low-cost options. the most desirable criteria included ease and speed of deployment, low cost—both in terms of money and staff resources, and suitability of the current server environments available to the library it staff. the primary tools evaluated included bugzilla (http://www.bugzilla.org), sugarcrm (http://www.sugarcrm.com/crm/) request tracker (aka rt, http://bestpractical.com/rt/), h2desk (http://www.h2desk.com/), otrs (http://otrs.org/), and eventum (http://freshmeat.net/projects/eventum). bugzilla was the final choice because it was compatible with the local server environment, it was easy to install and customize, and its broad user base ensured future development. it allowed for quick, local customization and rapid launch without extensive programming or specialized knowledge of perl, neither of which were available in the library at the time. extensive community support was also available via a newsgroup, a mailing list, and an irc channel. given the lack of library resources available for managing this type of tool, this level of responsive external community support was crucial [1]. also tipping the balance in bugzilla’s favor was the support of eric sisler, a systems librarian for westminster public library, who provided the auraria staff with support in launching bugzilla. bugzilla configuration first publicly released in 1998, bugzilla is an open source bug tracking system created by the mozilla.org developers. bugzilla is used by many high-profile organizations, including nasa and facebook, to track defects in the software development process [2]. bugzilla is, however, easily modified to accommodate tracking technology issues in a library environment; in fact, “it support queue” is one of the suggested uses for the software [3]. in the spring of 2008, bugzilla was installed on the library’s web server at http://bugzilla.auraria.edu utilizing a mysql backend. the existing lamp environment required a few additions to perl modules, which were performed quickly and easily. (see current bugzilla documentation for specifics on requirements for perl and other software) [4]. the next step was to create user accounts for all library employees. while there is an ldap authentication option within bugzilla, there was no in-library support for using ldap to connect to the library staff accounts in the ucd active directory. since local bugzilla users can reset their own passwords, it was believed that this would suffice for user access management. however, as will be discussed later, this was actually one of the major challenges in using bugzilla library-wide. about 70 user accounts for all library faculty, staff, and student employees were added manually. when employees terminate employment, their accounts may be inactivated, but it is recommended that they not be deleted as their names are associated with existing tickets and deletions can cause database referential integrity issues. since the auraria installation was meant to be an internal reporting system, only administrators were allowed to create new accounts. however, there are options in bugzilla to allow for completely open, user-driven account creation or even open account creation within a specific network. after all users were created, staff began defining the structure of library technology within bugzilla. bugzilla refers to all items as “products,” which are further broken down into “components” (see figure 1 for the list of technology products at auraria library). products and their components are easily added and edited in the web-based administrative interface. descriptions can be added to each product to help end users reporting problems choose the appropriate product and component when creating a ticket. figure 1. library technology products for auraria library [view full-size image] the fact that these products and components are locally defined is in large part what allows for bugzilla to be used as an it troubleshooting queue. for example, one product in the auraria bugzilla customization is “library web sites.” the components of “library web sites” are “archives web site,”  “blogs,” “contribute,” “intranet,” “libguides,” “other,” and “public web site” (see figure 2). figure 2. sample components for “library web sites” [view full-size image] as shown in figure 2, each component has a default assignee; this directs all newly created tickets to the staff member best equipped to resolve the issue. it is also possible to copy multiple people for each component; at least one cc recipient is recommended in case of absence of the primary recipient. additionally, if a team of staff is responsible for certain products, an email alias can be used so that all team members are notified as soon as a new ticket is created for that component. for example, the e-resources team at auraria is comprised of three librarians who address access problems with electronic resources. all three receive emails for tickets related to databases, ebooks, and off-campus access. another auraria workflow example relates to the “computers” product. the helpdesk manager, who directly supervises the student employees, receives all tickets related to staff and public workstations. he then delegates these tickets to students based on their schedules, skills, and availability to ensure quick resolution of the ticket. while care should be taken in the initial creation of products and components, products and components may be added or edited at any time post-launch. as with any tool, customer use of the product will inform tweaks and changes. it is also good practice to include an “other” product and an “other” component for each product so that a ticket may still be created if the person reporting the ticket is unsure as to what product or component to select. additionally, misfiled tickets are easily reassigned to the proper product and/or component within bugzilla. the final step was to customize the default screens so that they were relevant to a library environment. most of the work involved simplifying screens that library staff would use to report technology issues, primarily the bug/ticket creation form. eric sisler’s extensive customization instructions—even though they were written to version 2.18.3—proved  invaluable during this step [5]. sisler’s instructions identify all of the places in which software language and terminology is used, and he provides suggestions for a library-centric install. bugzilla is written in perl template toolkit, a language that is relatively easily modified by anyone with working knowledge of html [6]. a complete list of screens customized for the auraria instance is available in figure 3. figure 3. ten screens customized for auraria library bugzilla the most dramatic changes were made to the index/main page (index.html.tmpl [7]) and to the ticket creation page (/bug/create/create.html.tmpl [8]). within the bugzilla installation, the screens are located in the /template/en/ directory. the default screens are located in /template/en/default/, but rather than editing the default files directly, administrators should copy the files they wish to customize into a /template/en/custom/ directory (as in figure 3). only the files that need to be modified need to be copied; the contents of the entire /default/ directory do not need to be recreated as bugzilla detects and displays the customized screens automatically. creating this separate /custom/ directory allows for quicker bugzilla upgrades, as a clean installation of a newer version will not overwrite the contents of this directory. while the customized screens may require some tweaking after an upgrade, they will not need to be completely recreated. (complete documentation for upgrading bugzilla, including cvs methods, is available at http://www.bugzilla.org/docs.) figure 4. out-of-the-box bugzilla index [view full-size image] figure 5. customized index page for auraria bugzilla install [view full-size image] bugzilla workflow for an end user, creating a ticket begins begins by clicking on the “report a problem/create a ticket” link on the bugzilla main page (see figure 5). the user is taken to the product selection screen (see figure 1), and s/he is then taken to the ticket creation screen (see figure 6). figure 6. auraria bugzilla ticket creation form [view full-size image] while this form has thirteen fields, only four are required: selecting the component and ticket type, adding a quick summary, and adding a description of the problem. ticket types are fully customizable; auraria uses “please fix!,” “request,” and “information.” not surprisingly, the most common types of issues reported are “please fix!” “request” is used when there is a need for non-urgent work, such as a new software installation, rather than an existing problem. “information” is rarely used, but is a way to store research or helpful information in a common space. once the “create ticket” button is clicked, the information is entered into the mysql database, and the ticket can be viewed at a unique url. bugzilla assigns the ticket automatically to the it staff member responsible for the indicated component, and that person receives an email notification. the it staff may accept or reassign the ticket, if appropriate, or even reclassify the product and/or component if another better fits the issue. once work on the ticket begins, notes may be recorded about the steps taken to resolve the issue. notes may be added by anyone, not just the assignee, and whenever a note is added, notification emails are sent to all users who are associated with the ticket, i.e., the reporter, the assignee, and the cc assignee. if the work being completed on a ticket is (or becomes) significant to other staff members, cc emails may be added. when work on the ticket is done, it is marked as “fixed,” and it no longer displays in reports of open tickets (more on reporting below). figure 7. auraria bugzilla sample ticket [view full-size image] as figure 7 shows, there are a number of additional features that have not been described here. these include estimates of work time, deadlines, and blocks. these features seemed useful at the time of setup; however, they were simply not used, even by members of the it staff. the expectation of a 48-hour ticket turnaround time made these more or less obsolete. an out-of-the-box bugzilla installation also includes search plug-ins for ie7+ and firefox 2+ as well as a customizable screen reserved for ticket writing guidelines. the search plug-ins were not widely discovered or used outside of the it department. the ticket writing guidelines, which included suggestions and sample verbiage for successfully reporting it issues, were demo’ed at training sessions, but it is not known how frequently they were consulted beyond that. (see appendix: auraria library ticket writing guidelines). reporting, delegating, and managing the most used and useful reporting feature in bugzilla is the “my tickets” saved search, which is generated simply by clicking on the “my tickets” link that appears on the footer of all of the screens in bugzilla. the “my tickets” search displays at a glance all tickets assigned to the logged in user, which is helpful in prioritizing work. as figure 8 demonstrates, an individual’s open tickets may be sorted by the ticket id number (i.e., chronologically, as tickets are numbered sequentially beginning with 1), severity, priority, operating system, assignee, status, or summary. (no resolutions are shown in this report as these are open tickets.) figure 8. web librarian’s open “my tickets” report [view full-size image] bugzilla’s robust reporting features also allow it managers to delegate incoming work to student assistants, monitoring ticket assignments by viewing all open issues for a given student’s user id. managers can also track individual employee performance by checking the number of tickets each employee has closed over a given period of time. finally, to improve communication beyond the technology strategy and learning spaces division, a link on the library intranet allows any library staff member to view all open tickets. open reporting makes the often invisible work performed by it staff visible to all other library employees. figure 9. sample report for an it student’s open tickets [view full-size image] bugzilla has also became a motivational tool, particularly for the it students. the helpdesk manager has occasionally split the students into teams or challenged them individually in a friendly competition to close the most tickets in a given time period or to close a single challenging ticket. this boosted morale—the prize was often a free lunch for the winner—and increased productivity. at weekly it staff meetings held to review all open tickets, being the owner of the oldest ticket means bearing the brunt of friendly jokes. while these weekly meetings are centered around open tickets, they are a helpful venue for discussing “stuck” projects and finding ways to “unstick” them collaboratively by coordinating work across departments. launch and library-wide use the first bugzilla ticket was created by the it manager from an email message received on september 17th, 2008. bugzilla was soft-launched internally to the staff responsible for the major it systems through the month of october, with technology staff creating tickets from emails and calls from the dedicated phone line. half a dozen faculty and staff members were invited to begin using it to identify any major issues prior to library-wide release, and when no crucial issues were found, bugzilla was officially launched. a brief training session was held for library staff in october, and the helpdesk manager created a short, light-hearted tutorial that demonstrated how to log in and how to create a ticket to report an issue. after a couple of weeks of use, the word “bugzilla,” as in “i’ll bugzilla it,” became a verb. initially, staff seemed to think that bugzilla was only for desktop support; the web librarian still received emails requesting changes or reporting problems with the library’s web sites that could have easily been reported via the “library web sites” product. to manage these frequent emails, the web librarian created a custom email signature in outlook, requesting the reporter use bugzilla to request or report the issue. staff reaction was mixed, as is often the case with the introduction of a new technology in any environment. some jumped in immediately and began using it, while others have yet to log in and create a ticket. the departmental phone line was kept in place, but library staff using this mechanism to report problems were gently encouraged to report the issue in bugzilla themselves. it staff also reminded callers that their issue would be resolved more quickly if they entered the ticket themselves as it would be assigned as soon as it was submitted, as opposed to waiting for an it staff member to listen to the phone message and create a ticket on the reporter’s behalf. the dedicated phone line still exists and is answered whenever possible by the it student employees. staff may also still use the phone line to report a problem with their own workstation or a large-scale emergency, such as an outage that renders most of the public area computers unusable or affects mission-critical products like the ils. lessons learned after nearly two years of use, the primary lessons learned about bugzilla as an it tracking system in a library environment center around ease of use for non-it library staff. the first relates to authentication, or specifically, not enabling ldap authentication so that all library employees could use network credentials. the second most commonly stated dissatisfaction was the look and feel and, to some extent, the branding of the auraria library bugzilla installation. as previously mentioned, the ldap option in bugzilla used to create accounts automatically and manage authentication was not implemented at auraria library as there was no ldap expertise among the library it staff to establish a connection to ucd active directory accounts. while users are allowed to reset their own passwords, the password reset is not particularly user-friendly as it appears on the lower half of the login screen. library staff would click the “reset password” link, and were frustrated when it appeared they were simply being taken back to the login screen. having to use a non-network password was especially an issue for library employees who worked at public services desks. often, with a line of patrons or a classroom of students waiting, staff could not remember and did not have time to reset a password in order to report an issue in a timely manner. a brief experiment with a generic account and password that could be used by all public services staff was attempted; however, if the tickets were not specific enough, it was impossible to trace the individual who reported the ticket for follow-up. this also made it impossible to report the outcome back to the ticket creator, as the email account assigned to the generic account was a dummy account. as a result, ldap-enabled authentication became a requirement for all future web development projects. maintaining a separate set of credentials was, according to anecdotal evidence, the biggest barrier to employee use of bugzilla. another issue for library staff was the unappealing look, feel, and branding of bugzilla. it was described as a “geeky” product and even the it students, when asked what could be done to improve bugzilla, suggested “make it prettier.” for the auraria library installation, the web librarian had opted not to modify the css files in the interest of releasing the system as quickly as possible. however, bugzilla does support creation and use of custom skins, and a handful of skins created by developers may be found on the internet [9]. also, later releases have simplified and improved the main page out-of-the-box with the addition of graphical elements and a more interesting color palette (see figure 10). even the name “bugzilla” sounded ridiculous to some library employees. armed with this feedback, the web librarian installed another instance of bugzilla at http://helpdesk.auraria.edu and modified the out-of the box version 3.4.5 with graphics and fonts/colors that matched the public-facing web site (see figure 11). with support from a technology consultant, ldap was also enabled, and the ticket creation form was pared down to the four most basic fields, keeping all of the other form fields in the 3.0.4 version of the form available on an “advanced fields” screen (see figure 12). however, this revamped instance was never launched due to a decision to upgrade to a more fully-featured it management tool. figure 10. out-of-the-box improvements to bugzilla main page, version 3.4.5 [view full-size image] figure 11. auraria library redesigned bugzilla main page [view full-size image] figure 12. auraria library redesigned bugzilla ticket creation form [view full-size image] conclusion over 2300 tickets and 23 months later, the numbers speak for themselves. each month averaged 100 tickets, approximately 4-5 per business day. bugzilla has proved itself a more than worthy tool for not only managing, but modernizing the library’s it workflow.  however, bugzilla  has become a victim if its own success as the technology strategy and learning spaces division decided in may 2010 to purchase numara’s track-it! (http://www.numarasoftware.com/track-it/help_desk_software.aspx) for it management. with the growing importance placed on technology in the library’s strategic plan and the past success using bugzilla to manage operational support issues, the it manager sought out a more fully-featured software package that included functions such as inventory, equipment checkout, and software license management, in addition to ticket management. bugzilla does not have these features and is unlikely to develop them as they fall outside the original focus of the product. utilizing bugzilla to help shift the focus of day-to-day support issues to student employees allowed the it manager to focus on larger-scale projects and improvements that were aligned with the library’s strategic plan, including deploying thin clients and virtualizing network servers. this creates a better learning environment for auraria’s 50,000 students and simplifies desktop support for the 169 public computers, as well as support of the ils, the library’s web site, e-resources, and all other technology products. additionally, bugzilla’s reporting feature has created a historical knowledgebase that provides data to assist with personnel decisions about necessary levels of training and staffing, in addition to information about the quality of service provided by it staff members. multiple tickets about a specific product can be used to provide budget justification for replacing outdated equipment. bugzilla has also created a work record to document what is often invisible to library administrators and those who work outside of it. finally, the installation and configuration of bugzilla was a successful foray into open source development that opened the door to using mediawiki for a staff intranet and to migrating the library’s public facing sites to drupal. bugzilla was an extended proof of concept that helped the technology staff begin a much-needed modernization of the it department. notes [1] bugzilla, “support,” http://www.bugzilla.org/support/. accessed 9/9/2010. [2] bugzilla, “installation list,” http://www.bugzilla.org/installation-list/. accessed 9/9/2010. [3] bugzilla, “about,” http://www.bugzilla.org/about/. accessed 9/9/2010. [4] bugzilla, “the bugzilla guide, chapter 2.1 installation,” http://www.bugzilla.org/docs/tip/en/html/installation.html. accessed 9/9/2010. [5] sisler, eric, “bugzilla at the westminster public library,” http://westminster.lib.co.us/bugzilla-custom/. accessed 9/9/2010. [6] chamberlain, darren, david cross, and andy wardley. perl template toolkit. o’reilly. sebastopol ca. 2004. (coins) [7] sample code for index.html.tmpl, http://journal.code4lib.org/media/issue11/mchale/index.html.tmpl.txt. local modifications for auraria library thanks to bryan mollenkopf. [8] sample code for create.html.tmpl, http://journal.code4lib.org/media/issue11/mchale/create.html.tmpl.txt. local modifications for auraria library thanks to bryan mollenkopf. [9] bugzilla, “the bugzilla guide, chapter 6.2 custom skins,” http://www.bugzilla.org/docs/tip/en/html/cust-skins.html. accessed 9/9/2010. about the author nina mchale (nina.mchale@ucdenver.edu, http://milehighbrarian.net) is assistant professor, web librarian, at the auraria library, which serves the university of colorado denver, metropolitan state college of denver, and the community college of denver. appendix: auraria library ticket writing guidelines why you should read this: simply put, the more clearly you report your problem, work request, or question, the easier it will be for help desk staff staff to understand, respond to, and resolve it. before reporting an issue to the help desk: if there seems to be a power problem with your equipment, check all cords and cables, including power strips. if the problem seems software-related, save your work, close the software, and reboot. these are the first things that help desk staff would try, so you might save yourself some time by performing these simple checks. creating a new ticket in five minutes or less: if you have searched existing tickets, and no one else appears to have reported your problem, then: under step 3 on the help desk home page, click on “log in and create a new ticket.” your login id is your full email address, i.e., firstname.lastname@ucdenver.edu. your password is unique and set by you. forgotten it? no problem! click the link to reset it. once you’re logged in, select the appropriate product. if you can’t find a product that describes what you need help with, select “none of the above.” and don’t worry about selecting the wrong product; help desk staff can always reassign a ticket and/or create a new product. fill out the four required fields on the form. these fields are: component: this is simply a more specific part of a product. for example, the components of the product “skyline” are “cataloging,” “circ functions,” “opac,” “proxy/wam,” and “serials.” you can click the word “component” to see a description of each component. when you select a component, the “assign to” field is filled in automatically, so you don’t have to worry if you’re sending it to the right person or not ticket type: the three options are “fix me!,” “request,” and “information.” select “fix me!” if you’re reporting a problem. select “request” if this is not a problem, but a new request for work, such as a software installation or upgrade. select “information” if you have a question or would like more information about a technology-related topic or product. summary: in ten words or less, give your ticket a title. a good summary should quickly and uniquely identify a ticket. it should explain the problem, not your suggested diagnosis or solution. description: describe what is happening (or not happening!). again, you don’t need to try to diagnose the problem; just report where you are, what you’re doing and what should be happening. be as specific and precise as you can. here are some examples bad: “my email is down.” good: “when i open outlook, i can only see e-mail messages that are very old, from may 2008.” bad: “there’s a broken link on the hours page.” good: “the link to the adi on the hours page is broken. it tries to go to http://library.cudenver.edu/findit/databasesindexes.php, and it should go to http://library.auraria.edu/apps/databasesindexes/dbaseindex.php” two optional, but potentially very helpful, features on the form are the url field and the “add an attachment” button. url: if there’s a specific url relative to your problem, you can type or cut and paste it in this field. this is not always applicable, but could be very helpful, obviously, for a problem with the library web site or an electronic resource. “add an attachment” button: you can add an attachment of almost any file type. attachments that would be helpful to help desk staff include screen captures of error messages or documentation related to the product that you would like help with. double-check your additions to the form for errors and omissions, then click the “create ticket” button. voilà! your ticket has been entered into the auraria library help desk database and assigned to the appropriate help desk staff member(s). you’ll get an email when any additional information is entered by help desk staff, or when the problem status is changed. subscribe to comments: for this article | for all articles 4 responses to "managing library it workflow with bugzilla" please leave a response below: john harmon, 2011-08-15 even though we are software developers, we have used bugzilla and find it to be very user friendly and like it quite a bit. tom millerd, 2011-08-15 the ability to add attachments of any file type was particularly useful t us in our most recent project, scott long, 2012-01-18 omnistar live http://www.omnistarlive.com is a great customer support software. it is easy to use and makes tracking my support tickets easy. nina mchale, 2012-03-26 have been meaning to update this article with this comment: auraria dumped track-it! and returned to bugzilla. numara’s other tools were useful, but the ticketing system wasn’t as easy to use as bugzilla. the forms were more complicated, the form fields difficult to customize, and most of all, track-it! just didn’t seem to provide the open communication that bugzilla does; track-it! feels like a one-way reporting tool, a black box to get the issue to it. leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – launching an intranet in libguides cms at the georgia southern university libraries mission editorial committee process and structure code4lib issue 59, 2024-10-07 launching an intranet in libguides cms at the georgia southern university libraries during the 2021-22 academic year, the georgia southern university libraries launched an intranet within the libguides cms (libguides) platform. while libguides had been in use at georgia southern for more than 10 years, it was used most heavily by the reference librarians. library staff in other roles tended not to have accounts, nor to have used libguides. meanwhile, the libraries had a need for a structured intranet, and the larger university did not provide enterprise level software intended for intranet use. this paper describes launching an intranet, including determining what software features are necessary and reworking software and user permissions to provide these features, change management by restructuring permissions within an established and heavily used software platform, and training to introduce libraries employees to the intranet. now, more than a year later, the intranet is used within the libraries for important functions, like training, sharing information about resources available to employees, for coordinating events and programming, and to provide structure to a document repository in google shared drive. employees across the libraries use the intranet to more efficiently complete necessary work. this article steps through desired features and software settings in libguides to support use as an intranet. by wilhelmina randtke introduction during the 2022-23 academic year, the georgia southern university libraries (gs libraries) launched an intranet within the libguides cms (libguides) platform. while libguides had been in use at the gs libraries for more than 10 years, it was used most heavily by the faculty librarians. library staff in other roles tended not to have accounts, nor to have previously used libguides. meanwhile, the gs libraries had a need for a structured intranet, and the larger university did not provide enterprise level software intended for intranet use. this paper describes launching an intranet, including determining what software features are necessary to support use as an intranet and reworking software and user permissions to provide these features, change management by restructuring permissions within an established and heavily used software platform, and training to introduce gs libraries’ employees to the intranet. now, approximately two years after launch, the intranet is used within the gs libraries for important functions, like training, sharing information about resources available to employees, for coordinating events and programming, and to provide structure to a document repository in google shared drive. employees across the gs libraries use the intranet to more efficiently complete necessary work. this article steps through desired features and software settings in libguides to support use as an intranet. literature review what an intranet is socially fundamentally, an intranet is a communication tool ( [1] costello and del bosque 2010, 77). benefits of an intranet include: provides the same information to all employees, regardless of how long each has worked at the organization, or other arbitrary traits, and regardless of soft connections. employees can find information without looking in email. posted information can be revised based on feedback. better workplace culture: transparency by allowing any employee to find key information. when employees can find important information, this fosters trust in the organization, and ensures all employees work in alignment with each other. with an intranet, during onboarding, new employees can get all existing information. employees can know workflows in other departments and better plan across departmental lines ( [2] costello and del bosque 2010, 78). to provide all those benefits, the intranet has to have accurate, appropriate, and up-to-date information. and employees have to have buy-in and to use the intranet. an intranet is about people and communication fundamentally, an intranet is a communication tool which supports employees. the most important aspect of communication is people, not technology ( [3] wakimoto 2021, 11; [4] wisniewski 2016, 67). technology is a means to an end, and the focus should be on employees and on employee use and adoption of the intranet. in light of this, the choice of what technology to use and how to configure it depends on the user community of employees. for non technical employees, a what-you-see-is-what-you-get (wysiwyg) interface is essential ( [5] costello & del bosque, 2010, 79; [6] yoose, 2010, 170). people have a tendency to use what is familiar to them, rather than to take time to learn something new ( [7] fichter 2006, 52). change fatigue and resistance to enterprise level change has increased from 2016 to 2023 ( [8] o morain and aykens 2023). this can be a barrier to adopting a new technology tool. if an intranet is built using software that employees are already familiar with, that can aid adoption by removing the entry barrier of having yet another software platform to learn. different organizations will have different core functions, such that which software platform is best and how that tool should be configured will vary from organization to organization. wisniewski ( [9] 2016) advocates for libguides over sharepoint for use as a library intranet, specifically because of this cultural aspect of employee adoption. sharepoint is full featured and allows technical requirements to be met, but is complex. sharepoint dominates the intranet conversation in industry, but also was initially designed for very large organizations with tens of thousands of employees. meanwhile, library employees are already using libguides, so there isn’t a learning curve or new tool to learn when a library implements an intranet in libguides ( [10] wisniewski 2016, 67). employee preferences for technology may also change over time, as the technologies which are commonly used on a day to day basis change ( [11] verčič and špoljarić 2020, 6). in a migration away from a platform which did not allow wysiwyg editing, [12] yoose (2010) emphasized the importance of matching the intranet to the skillset of the majority of the employees so that all can contribute ( [13] yoose 2010, 170). what is needed in an intranet will vary depending on what the organization does and will vary from organization to organization ( [14] earley 2016, 13). it may be the case that the technology with the best features is not the technology employees are most comfortable with and open to adopting. in light of the need for employee buy-in and adoption, the best option may be to use an existing tool already integrated into the library. this may be true even when the tool is not intended for an intranet and lacks specific desirable features. best practices / desired features for an intranet common features desired in an intranet include: being able to access from anywhere, including access from home ( [15] kim 2010, 198). wysiwyg editor and not needing technology training to use the intranet ( [16] costello and del bosque 2010, 80; [17] etches-johnson and baird 2010, 266-67; [18] wakimoto 2015, 97; [19] johnson 2014; [20] yoose 2010, 165). being able to restrict access to employees only ( [21] costello and del bosque 2010, 78; [22] etches-johnson and baird 2010, 267; [23] johnson 2014). easy log-in with existing employee credential ( [24] etches-johnson and baird 2010, 267; [25] yoose 2010, 170. having an edit history and ability to track changes ( [26] costello and del bosque 2010 78; [27] wakimoto 2015 103). search capability and intuitive navigation ( [28] chen 2011, 62; [29] etches-johnson and baird 2010, 272; [30] lawrence 2022, 80; [31] wisniewski 2015, 66; [32] johnson 2014; [33] yoose 2010, 165). best practices for long term intranet success in literature, three traits are repeatedly emphasized as key for intranet success: support from management, including sharing links to the intranet in important announcements, putting important forms and documents on the intranet, and including the intranet in orientation ( [34] fichter 2006, 51; [35] kim 2010, 200; [36] sharma 2020). employee adoption of the intranet ( [37] johnson 2014; [38] yoose 2010, 173). ongoing attention to the intranet over time including ongoing training and maintenance ( [39] wakimoto 2021, 3-4; [40] johnson 2014; [41] yoose 2010, 163; [42] jeffery and dworak 2010, 181; [43] sharma 2020). background: the need for an intranet at the georgia southern university libraries during covid, multiple long term employees retired from the gs libraries with specific key retirements occurring from 2020 through 2022. the need to have institutional memory documented in written workflows in a shared permanent location became more urgent, partly due to key retirements around covid quarantine time. the georgia southern university is a google suite campus. within the gs libraries, individual departments, committees, and units had used google drive to share files for several years. in some cases, google drive was used in a more structured way intended to serve as an intranet for a specific committee or department. google drive is poorly suited to records management in a large organization. almost universally, gs libraries employees used google drive to store key documents and written workflows with documents owned by their individual work account. this was problematic with employee turnover, because when a specific document has an individual employee as the owner within the google drive software platform, then when that employee leaves, the document is deleted and purged on a tight schedule by main campus: either 30 days after departure for staff or 90 days for faculty. additionally, there were problems with permissions settings. even though google drive is easy to click into and edit for someone without special technology skills or training, the permissions are not intuitive. editing is intuitive. permissions are not. editing a single file is intuitive. working with search and organization for a large set of files is tricky. google offers an 8 hour long introductory course about google drive ( [44] google cloud 2024), which suggests that there is a learning curve to being able to truly master the software. google drive presents a problem in that a document must be shared to a specific person in order for that person to be able to search and pull it up within the google drive user interface. even if a document in google drive is set up with permissions open to all employees at georgia southern university, if a specific employee has not previously clicked into that document and the document has not been shared with that employee, then it is not possible for the employee to locate the document by search, or by any other means than a direct incoming link. with long term employees, the employee is more likely to be able to access key documents, because over time, the person will have clicked into incoming links from emails, such that they can later pull up that document by search. in contrast, with newly hired employees, the vast majority of content stored in google drive will be invisible to them. with fragmented ownership of google drive documents within the software, it’s impractical to comprehensively add someone to all important documents at the time of hiring, or when they come into a new role. there are tools for improving use of google drive to share information. specifically, google shared drive (formerly called google team drive) is available to enterprise accounts such as georgia southern university and streamlines folder and file ownership and permissions for a group environment, and using google groups listservs to set up group based permissions within google suite can help to manage permissions in a group setting. however, as of spring 2022, the gs libraries was not using group based permissions, nor google shared drive. choosing software platforms for the intranet in spring 2022, the libraries website committee (lwc) was tasked with launching an intranet in collaboration with the libraries technologies and systems department. in designing an intranet at the gs libraries, the lwc wanted certain features: to be able to restrict access to only current gs libraries employees, including graduate assistants and student workers when needed, and to exclude non-libraries employees. to be able to use the same campus login as for other tools, rather than for employees to have another username and password to juggle. to have standardized and automatic access / sharing to intranet content, such that once an employee is added to the intranet, that employee will see the same things as all other employees and the intranet will work the same for everyone. this is a necessary prerequisite to being able to deliver training, since standardized training requires the underlying software to work in a predictable and standardized way for everyone relying on that training. easy enough to use that employees can find and update information without specialized technical skills (ie. employees do not have to learn html, sed, or other technical information). able to see an edit history or older versions of a page. this allows seeing whether a page has been updated over time (ie. extensive recent updates mean the information is more likely to be current). it also allows checking over edits and touching base with anyone else editing a page. and it allows reverting to a previous version in case of an edit being made in error. to support editing and viewing by all employees, each through their own individual account issued in their name. in choosing software tools, a primary consideration was to use tools that already existed, rather than to select a new tool. without already having content ready to load, or existing uses and activities to justify funding, as might be the case with an intranet migration or revamp, requesting funding for a software tool was less viable than it might be for an established intranet. a new software tool also inherently includes challenges in terms of getting employees familiar with it, getting employees to use yet another thing, and in terms of having dedicated funding to renew the platform each year. libguides was selected to provide the heart of the intranet – the navigation and the most important high level contextual information about what records exist – with google drive / google shared drive holding long documents. using libguides to present and structure information, with longer detailed documents stored within google drive, was a model used within the gs libraries already on a small scale within one department. the collection services department at the gs libraries had a long running intranet consisting of a libguide which linked out to workflow and analysis documents within google drive. this provided a model to assess, tweak, and to bring to the rest of the gs libraries. and it provided an example to show during early stages of the intranet in order to assist decision makers across the gs libraries in understanding how the two tools would interact, to get buy-in, and to share a vision. figure 1. an existing departmental intranet within the gs libraries as of spring 2022 used a single libguide to store a list of links to significant google drive files owned by employees within the department. this example was useful to show to gs libraries employees in order to show the concept of a two pronged approach: libguides for structure, and google drive for longer and existing documents, and to emphasize to employees that they did not need to migrate content (high effort), but rather to identify and link out to important content (low effort). libguides for structure, overview, and an entry point libguides was a readily available tool which was already familiar to a critical mass of gs libraries employees. the gs libraries already had a subscription to libguides cms (referred to throughout this article as libguides), such that no dedicated funding was needed. and all faculty librarians and some staff employees already used libguides regularly or even daily. this meant that some training and familiarity with the software would already be present across the gs libraries as a whole, and that employees would be able to peer mentor one another on the software and to share tips on using the software. libguides is also able to support many, but not all, of the gs libraries’ desired features in an intranet. (lack of edit history is the only desired feature which libguides could not be made to support.) given the critical mass of employees already familiar with and using the software within the gs libraries, libguides might be a best fit in the abstract, even if the gs libraries had had free choice to select any available software platform in the world. employee buy-in and use of a software tool is necessary for that tool to succeed as a communication tool and for the organization to adopt the tool. being already familiar to employees was a strong feature in favor of libguides as the gs libraries’ intranet software. meanwhile, by having a list of features desired in an intranet, it was possible to dig into libguides settings and configure most of the desired traits into the platform. having a list of desired features which was platform agnostic helped us to dig into settings and get the software as close as possible to meeting the gs libraries’ needs regarding software functionality. focusing on desired features also allows periodically rechecking software settings over time, in case updates from springshare make a desired feature available in the future. google drive and google shared drive for file storage google drive was heavily used by and familiar to employees at the gs libraries, as well as to the general public. familiarity with google drive was essentially universal within the gs libraries and not limited to faculty librarians, as libguides familiarity had been. google docs has been available free of charge to members of the public since 2006 ( [45] mchugh-johnson 2021). google docs later expanded to google drive. as of 2023, google drive had 31.19% of the market share of file sharing software globally ( [46] taylor 2024). the georgia southern university campus has used google drive as part of google suite for more than 10 years, and both main campus and the gs libraries routinely use google drive for sharing out documents and for revising documents in a group setting. google drive is complementary to libguides as a place to keep longer files. google drive is able to store lengthy files, such as workflows, by-laws, or meeting notes documents. google drive also allows collaborative editing of files. and, google drive allows users with editor permissions on a document to see a revision history for that document ( [47] google docs 2024), which is a key desired trait for an intranet which is missing from libguides. as of spring 2022, individual departments, units, committees, and employees at the gs libraries were already using google drive to share documents. tapping into that was important. a goal for the intranet launch was to allow intranet adoption with minimal effort. for example, the lwc encouraged linking out to existing documents and meeting notes in google drive which required less effort than migrating content would have. documents already in google drive could stay in google drive. a short description of the file and an incoming link from libguides would make a document in google drive findable for everyone. even if the document is not shared to the person’s account, they can find the incoming link on a libguide, click in, and send a request to view the document. for an employee coming into a project, if there is an up-to-date list of links on a libguide, the employee can click each link and request access to the key google drive document(s). libraries wide adoption of google shared drive was out of scope for launching the libguides intranet at the gs libraries. dramatic changes to file permissions is a high-effort and high-focus task, because of the need to plan for confidential information (ie. a password or work schedule being shared to a much wider group might not be appropriate), and because without incoming links to key documents in google drive, it wasn’t possible to know what files should be moved nor to efficiently plan such a move. because of this, a move to google shared drive was not emphasized during intranet launch, although it was later planned out and is presently underway as a separate records management initiative. building the intranet preparation: standardizing account permissions within a decade old libguides site one of the desired features in an intranet was to have standardized permissions, such that it would work the same way for all employees. in spring 2022, as the intranet was planned, about half of gs libraries employees had a libguides account. as part of the preparation work and before issuing additional accounts, libraries technologies and systems standardized libguides account permissions on existing accounts. in january 2022, the gs libraries had approximately 29 faculty librarians and 78 other employees, excluding student workers. about half of the gs libraries’ 107 employees had accounts. libguides familiarity among non-librarians was highly variable. in order to standardize accounts, libraries technologies and systems first mapped out permissions for each gs libraries employee with an existing libguides account. within the libguides platform, each guide can be assigned to a group, and then users can be added to each group of guides. libraries technologies and systems assessed existing libguides, looked at which groups of guides were most used and how each group was used. based on this analysis, libraries technologies and systems then determined a standard set of permissions for all gs libraries employees. the standard permissions are potentially more permissive than for most libraries. all employees, regardless of job title or duties are technologically able to create a guide and post it live to the gs libraries libguides homepage. the libguides software supports editorial workflows and approval processes for publishing guides, however editorial workflows for libguides are not in use at the gs libraries. establishing a culture of what to post and who can post is currently handled through onboarding training to libguides and through a page of instructions on the gs libguides intranet which was incorporated into the intranet launch and into onboardings going forward and which employees can reference later as needed. the reasoning for this is that having standardized simple permissions, and having a common baseline set of permissions for all employees, allows standardized training, prompt and straightforward tech support, and peer mentoring. when the software works the same for everyone, it’s easier to provide support to all employees and easier for employees to ask one another questions. this plan was uncontroversial, likely because a few years earlier, the gs libraries had begun moving traditional webpage content into libguides and had begun taking the libraries website in wordpress run by main campus in the direction of a one page library website linking out to informational libguides about specific gs libraries services. starting in 2019, the gs libraries began moving subpages on the wordpress site into libguides. for example, before 2019, each department in the gs libraries had a page on the wordpress site while libguides was used almost exclusively for research guides, pathfinders, and course guides. starting in 2019, departmental subpages and other subpages were moved to libguides, and the wordpress site was pared back into a single page of outgoing links with many outgoing links pointing to departmental and functional unit pages on libguides. and employees across the gs libraries, in a variety of roles not limited to traditional reference librarians, routinely maintained content in libguides. as a result, for several years, traditional website content had been presented alongside pathfinders and research materials within libguides, and so there was already an expectation that a broad set of employees would update publicly available content in libguides. figure 2. standard permissions for all non-student gs libraries employees were defined. then libraries technologies and systems began the process of bringing each existing employee account in libguides into alignment with the standard permissions. out of 34 groups of guides in the gs libraries libguides site, four were identified as being of general interest. the specific groups and the way they are used at the gs libraries is not as important as is the process of examining each group and determining which groups are of interest libraries-wide. after determining standardized permissions for all gs libraries employees, libraries technologies and systems then worked through each existing employee account at the gs libraries to ensure each employee had either a regular level account with the gs libraries standard permissions or an admin account. this was accomplished by looking at an org chart, and making a spreadsheet of employees within each department including their current libguides permissions and a summary of changes that libraries technologies and systems desired to make in order to bring each employee’s account in line with the standard permissions. for employees getting additional permissions assigned, this was a straightforward process of communicating what was going on and letting the employee and supervisor know when the change was made. for employees getting permissions removed, this involved a conversation about whether removing permissions would interfere with work. almost all employees getting permissions removed were employees with admin accounts which would be converted to regular accounts. as of spring 2022, the gs libraries had had a subscription to libguides for more than 10 years. over the previous decade, libguides had introduced finer grained user permissions. for example, 15 years ago, libguides had only admin and regular users on the site, as well as the ability to add an account as an editor to one guide only. today, those three roles exist, however, permissions settings have been added to each role to allow, for example, a regular account to edit tags on any guides, or a regular account to edit any guide on the site. guide groups and interactions between guide groups and regular level accounts were also added in the years since libguides launched. so, in the previous decade permissions on the regular account had been developed by springshare to be much more fine grained than was the case when the gs libraries initially set libguides up. within the gs libraries, several long-term employees had admin accounts, and on investigating why each had an admin account, it became clear that some of these admin accounts had been issued before springshare introduced more fine grained permissions for regular level accounts. during permissions clean up, as employee permissions were addressed, the list of gs libraries standard permissions was updated to include basic information about any deviations from standard permissions. for employees with an admin level account, the list of standard settings for accounts includes the person’s name, a note that that person has an admin account, and a short statement with a reason why the person has an admin account. for any extra permissions added to a regular account beyond the standard permissions, the list of standard settings for springshare accounts includes the person’s name, a note describing the extra permission, and a short statement of why the extra permission is added to that person’s account. standardizing account permissions was a months-long process. most time was spent in communication regarding reducing account permissions for specific employees from admin to regular. admins can edit any guide on the site. meanwhile, regular users must be an owner or an editor on a guide in order to edit that guide. for these cases of reducing permissions, it was necessary to identify any guides the person needed to be able edit and when converting from admin to regular, to add them as an editor directly to each guide, then email the guide’s owner regarding that addition. it was also necessary to ensure that each employee regularly using the software who was getting a permissions reduction had an opportunity for a short conversation regarding the change, and knew how to report any problems caused by the change. leading by example: seeding content in the intranet most academic library publishing regarding library intranets discusses moving from one intranet to another. for the gs libraries, a libraries-wide intranet was completely new. it was not replacing any previous intranet. while specific departments within the libraries had sets of files intended as a departmental intranet, this is not the same as a libraries-wide intranet. when a comprehensive strategy is not applied to an intranet, a danger is that units within the organization will each try to create their own solution, leading to fragmented approaches and duplicative content in different locations ( [48] earley 2016, 12). and, an intranet fundamentally should be led with an overarching strategy ( [49] sharma 2020). a structured organization-wide intranet truly is new for the gs libraries and is not a continuation of individual siloed solutions. libraries technologies and systems provides tech support to the gs libraries. the department had existing handouts and instruction sheets which had been shared by email in the past. for example, information about using the campus vpn, information about what software is available for employees to have installed on work computers, and information about equipment available to employees such as microphone equipped headsets and replacement mice or keyboards. as libraries technologies and systems sent out emails in response to tech support requests, commonly requested instructions were posted as libguides intranet pages. this was a fast process to seed the libguides intranet with useful information that employees regularly request. before official launch and training, technical information was posted to the intranet. after official launch and training, libraries technologies and systems has emailed out a link to the intranet page for common questions, rather than directly emailing information in the body of an email or as an attachment. this has the benefit of integrating the intranet into day-to-day operations and of allowing the information to be reviewed and revised regularly by checking over the information and making any necessary revisions each time the link is emailed. a key piece of content which was developed and vetted before the intranet’s launch and before new accounts were issued was instructions about how to use libguides as a whole (not just the intranet group of guides). rather than being technical, these instructions were about social aspects of publishing new guides or updating public content. the lwc requested upper management to define any expectations for review and an approval process. the expectations were that faculty librarians could post content live, and that other employees would need their department head’s approval to publish a new libguide to the public site. these expectations were clearly stated on the intranet page with training about libguides, along with instructions about how to post a libguide to the home page, and an faq of common tasks which the libraries technologies and systems department commonly provided answers to. seeding content meant that there was useful information and something to look at on the intranet when employees later got onboardings. login quirk: making the intranet available to employees only suppresses the central authentication service (cas) login option on intranet pages libguides supports cas login. however, when a user visits a guide in an “internal” group, libguides will not prompt for the cas login, but rather will prompt only for a unique libguides login. meanwhile, “internal” was the only setting compatible with restricting the intranet to only gs libraries employees. within libguides, guides can be added to groups. settings on each group of guides allow the group to be set as public, restricted, or internal. springshare’s description of each, as stated within the software in a message shown when applying the setting, is as follows: public: “public groups make published content available (linked, indexed, etc.) to everyone on the public side of your site.” restricted: “restricted groups link and index all published content on the public side, but restrict access to the content via password, ip restrictions, and/or libauth rules. if you set more than one restriction, users must pass all restrictions to gain access. password and ip restrictions are set on the group edit page. libauth rules must be set at the system level (documentation on adding libauth rules) after libauth has been set up in libapps (springboard on adding/managing libauth configurations).” internal: “internal groups do not make any content available on the public side of your site except to users with accounts in the system with access to the group. if users are logged in and have access to the group, they will see the guides/groups in lists on the public side, designated in italics. published guides are indexed by us and are available in search results to logged in users who have appropriate group access. no content (published or otherwise) is indexed by other search engines.” because the gs libraries have libauth set up to allow logging in to libguides using the main campus cas login, restricted is the best option from a usability perspective. when an employee is logged out of libguides and goes to the url of a guide in a restricted group, the user sees both an option to login directly to libguides using libapps account credentials stored within libguides, or to click to the campus cas login. figure 3. login page a user sees when visiting a guide in a restricted group. meanwhile, internal is bad for usability, because when an employee who is logged out of libguides goes to the url for a libguide within an internal group, the login screen shows only the option to log in with libapps credentials. figure 4. login page a user sees when visiting a guide in an internal group. the gs libraries were unable to set the group as restricted (the better option for a smooth login). the restricted setting can set user access up based on libauth rules. libauth rules in the gs libraries’ libguides install could build on main campus settings as to what group each user account is in within main campus active directory. essentially, libauth rules could distinguish between university employees versus students, because that was a distinction within the main campus authentication system, but couldn’t get much more fine-grained than that. this meant that if the restricted setting were used for guides in the intranet group, then it wasn’t possible to limit access to libraries employees only. this likely will be the case at any academic library, in that main campus will operate the centralized login and a combination of human resources, enrollment, and main campus information technology (it) will determine what groups exist, and the campus library will have to work with those settings coming with the cas. as a result of not having a way to use libauth to pull a list of only library employees, the gs libraries intranet had to be set as an internal group within libguides. in order to have a landing page for the intranet which also would allow employees to have a link to the cas login, a libguide was set up specifically for employees to be able to bookmark and be able to have both a link to the cas login for libguides and to the intranet homepage: figure 5. the login page for the intranet. the “login” link goes to the libapps login which gives both options for logging (with libapps credentials, or with the campus cas login). the employee must login with cas on that page, then must return to the intranet login page libguide and click the link to the intranet home page. login is a clunky two step process. for faculty librarians who tend to be always logged in to libguides, they are able to click directly on “intranet” to get to the intranet home page. for employees who use libguides less frequently, and whose main interest is in visiting the intranet, they have to click through to the login, then return to the “intranet login page”, then click “intranet” while logged in. official rollout with formal training the official rollout was launched with formal training through the lwc. the lwc scripted an intranet onboarding. this was planned for 30 minutes. 15 minutes at the beginning was planned for having each attendee confirm that they had logged into libguides and to use the time to resolve any connection issues during the meeting. then 15 minutes was allotted for a scripted tour of the intranet, with an emphasis on the social expectations of launching new libguides (ie. when to seek approval and who to inform). then the representative from the lwc was available to ask questions. one onboarding was completed for each department, and a separate onboarding was completed with each member of administration. in conducting the onboardings, including 15 minutes to work through any login issues during the meeting was important, because in some departments almost all employees had never previously had a libguides account nor logged in, including long term employees who had earned an mlis while employed at the gs libraries. for faculty librarians creating libguides regularly, this was not an issue. for circulation desk employees, the norm was for the employee never to have been issued any libguides account before the intranet launch. staff in non-circulation roles tended to have libguides accounts, although familiarity with libguides was highly variable with some using it daily and some having not logged on in years. intranet structure the gs libraries intranet is structured into three general areas: (1) a libguide for each committee, team, and department (the functional units within the gs libraries), (2) a section of technology instructions for things like printing and vpn access, and (3) information from gs libraries administration including policies, forms, and gs libraries reporting information. figure 6. the homepage of the gs libraries intranet. almost all content is linked from a single side navigation menu, and a search box allows searching the libguides intranet. after launch building momentum after launch in order to remain vibrant and viable, an intranet requires maintenance. an initial training for each employee is not enough to establish a long term solution. since launch, two important steps have been taken to entrench the intranet into long term ongoing training initiatives at the gs libraries. first, the intranet was included in an onboarding checklist provided to each supervisor at time of hiring a new employee and maintained by gs libraries administration. when the onboarding checklist was revised by management in spring 2023, two important steps were taken to entrench the intranet. first, the onboarding checklist was moved into the google shared drive portion of the intranet. the onboarding checklist had been maintained by a recently retired secretary. her files included several versions of it. during the revision process, the most current version was located and moved to the google shared drive portion of the intranet, ensuring all supervisors can access it electronically. next, during the revision process, two items were added to the checklist: for the supervisor to contact libraries systems and technologies and request accounts be issued, and for the lwc to conduct the intranet onboarding with the new hire. second, the intranet was incorporated into the gs libraries’ committee appointment process. gs libraries committees are appointed annually each july. each member is appointed for a 2 year term and each chair is appointed for a 1 year term. in july 2023, libraries technologies and systems conducted a meeting with each outgoing chair and incoming chair to go over committee electronic accounts. one meeting was conducted with each committee. these meetings relied heavily on the intranet. each incoming chair got a tour of the committee’s libguide on the libguides intranet, was told what standard areas of a committee libguide exist, and ownership of the committee libguide was transferred during the meeting. each incoming chair got a tour of the google shared drive document library, and the outgoing chair was invited to describe what important committee documents exist and how these are stored. chairs were also shown a page on the libguides intranet overviewing basic information about how the google shared drive works, how permissions work compared to individual google drive, and basic steps to migrate content from google drive to google shared drive. for committees that were interested in migrating important documents and meeting notes to the google shared drive, libraries technologies and systems made arrangements for follow up regarding assistance with the migration, including offering to meet up with people owning key documents in google drive and working with them to transfer the documents to google shared drive. as a result of this process, about half of university libraries committees are now using both the libguides intranet and the google shared drive for the university libraries. committees have tended to adopt the intranet eagerly, because the frequent turn over of members means that organization-wide tools are a better fit than are ad hoc tools used by a smaller group. an especially big draw is seamless sharing and permissions to all incoming committee members of important google drive files. using the libguides intranet to communicate out to the entire gs libraries about important committee business such as reports and surveys is a significant benefit. libraries technologies and systems plans to repeat the committees onboarding each year when committee chairs rotate, because reviewing committee accounts with the incoming and outgoing chair streamlined account transitions and proactively addressed support issues. refinements since launch: using additional springshare suite tools to meet specific needs after launching the intranet in libguides, the libraries built out and integrated two additional tools using other products within the springshare suite: an internal planning calendar to “save the date” and better plan events at the gs libraries to avoid having several events one week and few the next week, and an equipment reservation system for employee equipment. internal save-the-date calendar an issue the gs libraries had was uneven distribution of events. for example, in april 2022, during a single week, the gs libraries hosted more than 15 events. and in other weeks during that same month, there were far fewer or no events. within the gs libraries, various committees and departments used google calendars to coordinate specific things. libraries technologies and systems inventoried google calendars in use at the libraries by requesting all employees to volunteer information about google calendars in use. this turned up 31 calendars, and likely is an incomplete inventory. similar to google drive, permissions in google calendars can be tricky. each google calendar is managed separately, and many were created ad hoc to meet a single purpose. when employees are newly hired or change roles, it is not necessarily possible to add an employee to google calendars in a consistent fashion, because each calendar is managed by a different person and so finding a contact is haphazard. similar to google drive files, it is easy to create and use a google calendar quickly, and similar to google drive, managing permissions for google calendars in a large organization is tricky and complex. meanwhile, libcal allows creating calendars. instructions are posted at https://springyu.springshare.com/courses/libcal/calendars. desire to advertise events is the biggest motivator for gs libraries employees to use the calendar. to roll out the calendar, two things were done regarding training. first, instructions were posted and kept up-to-date on the libguides intranet. second, the calendar was built out in communication with the gs libraries communications and marketing committee (cmc). all events geared towards patrons are supposed to be reported early to the cmc and the cmc is responsible for advertising events in a variety of channels. so the cmc had both a strong interest in having a way to skim for upcoming events which had not yet been reported to the cmc, and also had a roster of upcoming events which could be input to the calendar while configuring settings. during a full summer, libraries technologies and systems met weekly with the cmc. cmc made requests or gave feedback during meetings, libraries technologies and systems made the changes in response, and cmc began incorporating the save-the-date calendar into their workflow for advertising all gs libraries events. the calendar initially was launched as an internal save-the-date calendar. categories within libcal were set up to allow events to be assigned to a category for each of the gs libraries locations, and for internal vs tentative vs ready to appear live on the homepage. anyone working at the gs libraries can post an event to the calendar, and can edit and delete all events on the calendar. similar to libguides, the standardized permissions were added to the springshare account for every gs libraries employee, and instructions were posted on the libguides intranet, which include the social expectations for what to post and when to make an event live on the homepage. the process of posting to the gs libraries’ homepage on the main campus led wordpress site involves getting information from libcal’s api in the json format and manipulating it to display in a “news and events” section on the gs libraries’ wordpress homepage. springshare posts public documentation on using the api, which other libraries can use to build out a similar display ( [50] springshare 2024). springshare also makes calendar widgets available, which are less customizable. the api requires coding, while the widgets can be configured using menus and clickable settings. specifics of showing libcal calendar events on a campus website would vary from library to library based on technical architecture and campus security practices. the key elements of the save-the-date calendar which matter most are: first, the user management being controlled by the gs libraries, as opposed to google suite where all google suite admins are with main campus it. and, secondly, the ability to send data out from libcal to another website, such that advertising at the end of the process is a motivator. figure 7. screenshot of the libguides intranet page about the gs libraries event planning calendar. this tab of the guide shows the internal save-the-date calendar for planning gs libraries events. when someone is planning an event, they can post it here. categories on the events allow an event to start as a save-the-date internal event, then later as it is finalized, to be tagged as ready to appear on library homepage. the “home” tab on this guide has instructions on how to use it, so when employees interact with the calendar, they tend to come in through the libguides intranet and to have instructions handy. figure 8. events in the save-the-date calendar which are added to the “ready to appear on library homepage” category publish live to the “news and events” section of the gs libraries’ homepage in wordpress. internal equipment reservation calendar for library programming and events an additional tool integrated into the libguides intranet is an equipment reservation system. the equipment reservation system allows any libraries employee to pull up a list of all equipment available for event programming, and to reserve specific items. equipment includes tablecloths with gs libraries branding, a set of matching folding chairs, tablecloths, folding tables, a pop up sun shade for tabling outside, a portable sound system and more. each item has a photo, so that employees can “shop” items when planning an event, and clicking on the photo brings up details about the item, such as where it is physically stored and any special usage instructions. for example, mobile presentation equipment and the sound system can be reserved and then libraries technologies and systems should be contacted to request set up, and the form for reserving equipment for that item has been customized to require a checkbox be clicked confirming the person emailed and scheduled delivery with libraries technologies and systems. figure 9. screenshot of the equipment reservation system. any libraries employee can see what items are available to support event programming. in order to launch the equipment reservation system, libraries technologies and systems built out the reservation system, and added all equipment. springshare makes available instructions on setting up equipment reservations in libcal at https://ask.springshare.com/libcal/faq/1419, which can be followed to set up library locations and to set up equipment at each location. libraries technologies and systems then posted a page of instructions on how to reserve equipment on the libguides intranet, and set up standard permissions in libcal to ensure that the instructions worked for all employees. (all employees are able to delete or modify reservations through the admin interface of libguides, which requires a libcal account, and instructions are posted on how to do this.) it was initially planned to do an onboarding or training, similar to the launch of the libguides intranet. however, with clear instructions and the incentive of getting fast and easy access to equipment, employees involved in event planning learned the reservation system, and began using it before the lwc had planned or scheduled training. beyond being an additional feature, the equipment reservation system is an example of management buy-in. the reservation system was launched at the request of the dean of libraries, who specifically had each department provide a list of event programming items and move those to a storage room at each campus (the gs libraries operate in both savannah, ga and statesboro, ga). because of the administrative drive to have a strong internal equipment reservation system, a wide variety of equipment was gathered and presented centrally. this proved to be a big draw with employees excited to learn that certain items were available for use, and already owned by the gs libraries. for example, the outreach librarians were excited to learn about the pop up sunshade tent, because they had previously done tabling outside in the sun in georgia’s summer heat. libcal supports equipment reservations, which is geared towards managing checkouts to patrons, since catalog systems tend to be designed for books rather than equipment, board games, or non traditional materials. because of the focus on public patrons, there are a couple of weaknesses in libcal for employees-only equipment, which we’ve addressed through settings in the software. the settings within libcal can be set up to allow equipment reservation to be restricted to logged in users only, however, that changes the user interface to where employees would have to come through the back end only with no friendly url and with a list of item names only but no “menu” of browsable pics of equipment. the change to the interface when restricted to employees only dramatically reduces usability, and so the gs libraries have the reservation calendar set “private” (hidden from the public and not appearing in menus and lists; accessible by friendly url). while it has not happened yet, we anticipate a day when a student will google into the equipment reservation calendar and put in a reservation. in order to prepare for that, we modified the template for displaying equipment to include a message that the equipment is for use by gs libraries employees only. figure 10. this setting is at “admin”, “spaces and equipment”, then under the “settings” tab. benefits and limitations of the intranet since launch, the intranet has proven valuable. in 2023 and 2024, georgia southern university worked with carnegie consulting to streamline the campus-wide websites. part of that process was to have each college at the university, including the gs libraries, inventory all public facing pages and determine whether each was intended for an internal audience versus for prospective students, the public, or other external audience. the gs libraries launched the intranet internally over the course of the 2022-23 academic year. part of that process had been to gather workflows and documents and move them to the intranet. for example, before launching the intranet, a page on the libraries’ main website in wordpress had consisted solely of a list of hard-to-remember urls intended for libraries employees to bookmark and for use in day to day work. this included, for example, the login url for the gs libraries’ back end alma integrated library system login page. at the time when the main campus began streamlining the campus web presence, the gs libraries had already taken down the page of links, posted identical information on the libguides intranet, and worked with all libraries employees to update links. additionally, during the process of streamlining the gs libraries’ primary website, having an intranet available meant that there was a plan in place for handling information intended for gs libraries employees. during the streamlining process, the libraries had the choice available for each public page to retain, delete, or make internal by moving to the intranet, and this made for an efficient process to align with a wider campus initiative and make progress on bigger goals. long term, most intranets fail. reasons for failure are a lack of vision, employees not adopting the intranet, and weak support from management. regarding the gs libraries intranet, long term success depends on continued maintenance of the intranet. meanwhile, maintenance is notoriously difficult to resource compared to new initiatives or growth. in support of long term viability and vibrancy for the intranet, the gs libraries have included intranet trainings in employee onboardings, have systematically onboarded gs libraries committees to the intranet which provides steady flow of current information of interest, and have built out useful tools such as an equipment reservation system and a save-the-date calendar. conclusion we have found that libguides is a viable platform for an intranet at an academic library. while the software does not support all features which might be desired for an intranet, it has an intuitive interface for editing and library employees tend to already be familiar with the software with librarians using it daily. because of this, adoption has a low barrier and it is not another-tool-to-learn. a person-centered approach is more important than technology for launching an intranet that sticks. and by identifying desired features in an intranet and then examining software settings, it is possible to get better performance from any system, including libguides. this article provides a case study of libguides at the gs libraries, including configuring the software to provide common features in an intranet, change management to establish a new organizational use of an established piece of software, and an onboarding program for gs libraries employees. references [28] chen e. 2011. from static html to interactive drupal: redesigning a library intranet that enables collaboration and social interaction. in brick and click libraries symposium proceedings; 2011 nov 4; maryville (mo); northwest missouri state university. pp. 62–69. https://soar.wichita.edu/server/api/core/bitstreams/f4db9e49-9140-4d42-bf38-bbd3858b7e52/content#page=71. [1][2][5][16][21][26] costello k, del bosque d. 2010. from forgotten intranet to successful wiki: best practices for implementing an academic library staff wiki. in brick and click libraries symposium proceedings; 2010 nov 5; maryville (mo): northwest missouri state university. pp. 77–82. https://files.eric.ed.gov/fulltext/ed513812.pdf#page=86. [14][48] earley s. 2016. getting more from sharepoint part 1: using “intranet in a box” preconfigured offerings. kmworld magazine 25(6):12–14. [17][22][24][29] etches-johnson a, baird c. 2010. corralling web 2.0: building an intranet that enables individuals. journal of web librarianship 4(2-3):265–276. [7][34] fichter d. 2006. making your intranet live up to its potential. online 30(1):51–53. [44] google cloud. 03 google drive. google cloud skills boost. [cited 2024 april 28]. available at https://www.cloudskillsboost.google/paths/23/course_templates/199. [47] google docs. find what’s changed in a file. google docs editors help. [cited 2024 april 28]. available from https://support.google.com/docs/answer/190843. [42] jeffery k, dworak e. 2010. who moved my intranet? the human side of introducing collaborative technologies to library staff. journal of web librarianship 4(2/3):177–86. https://www.doi.org/10.1080/19322909.2010.500848. [19][23][32][37][40] johnson cd. 2014. cultivating a grapevine: building an intranet using wordpress. tennessee libraries 64(1): 1. [15][35] kim b. 2010. organizational and social factors in the adoption of intranet 2.0: a case study. journal of web librarianship 4(2/3):187–206. https://www.doi.org/10.1080/19322909.2010.501276. [30] lawrence, valerie j. 2022. “employing a user experience design process to redesign an intranet library website positively affected site usage and user satisfaction.” journal of hospital librarianship 22 (2): 77–84. https://doi.org/10.1080/15323269.2022.2053411, at 80 [45] mchugh-johnson m [internet]. [blog post dated 2021 oct 11]. 15 milestones, moments and more for google docs’ 15th birthday. https://blog.google/products/docs/happy-15-years-google-docs/. [8] o morain c, aykens p. 2023. employees are losing patience with change initiatives. harvard business review. 2023 may 9. https://hbr.org/2023/05/employees-are-losing-patience-with-change-initiatives. [36][43][49] sharma d. 2020. don’t let your company’s intranet become a junk drawer. harvard business review. 2020 may 8. https://hbr.org/2020/05/dont-let-your-companys-intranet-become-a-junk-drawer. [50] springshare. api: overview of the libcal api. springshare help center. [cited 2024 june 13]. available from https://ask.springshare.com/libcal/faq/1407. [46] taylor p. 2024. leading vendors’ share in the file sharing software market worldwide in 2023. statistica. [updated 2024 mar 22]. available from https://www.statista.com/statistics/1328893/global-file-sharing-market-share-by-vendor/. [11] verčič at, špoljarić a. 2020. managing internal communication: how the choice of channels affects internal communication satisfaction. public relations review. 46:1–7. https://www.doi.org/10.1016/j.pubrev.2020.101926. [18][27] wakimoto dk. 2015. using google sites as library intranet. in: smallwood c, editor. the complete guide to using google in libraries: instruction, administration, and staff productivity. lantham (md): rowman & littlefield. p. 97–104. [3][39] wakimoto dk. 2021. exploring internal communication in public libraries: challenges and opportunities for library leaders. library leadership & management 35(2):1–18. https://doi.org/10.5860/llm.v35i2.7466. [4][9][10][31] wisniewski j. 2016. checking in on the intranet. online searcher 40(4):66–68. [6][12][13][20][25][33][38][41] yoose b. 2010. when the new application smell is gone: traditional intranet best practices and existing web 2.0 intranet infrastructures. journal of web librarianship 4(2/3):161–75. https://www.doi.org/10.1080/19322909.2010.502740. about the author wilhelmina randtke has a background in law and technology. her past roles include legal research, technology oversight, and product manager for cloud based publishing software. she is currently head of libraries technologies and systems at the georgia southern university libraries overseeing in-building technology and online presences. georgia southern university libraries is the largest student printing lab on campus, providing approximately 1,500,000 prints to students each year. the libraries provide in-building desktop computers with specialized software for academic projects, a fleet of approximately 190 checkout laptops for students, and 3d printing and scanning services. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – thresholds for discovery: ead tag analysis in archivegrid, and implications for discovery systems mission editorial committee process and structure code4lib issue 22, 2013-10-14 thresholds for discovery: ead tag analysis in archivegrid, and implications for discovery systems the archivegrid discovery system is made up in part of an aggregation of ead (encoded archival description) encoded finding aids from hundreds of contributing institutions. in creating the archivegrid discovery interface, the oclc research project team has long wrestled with what we can reasonably do with the large (120,000+) corpus of ead documents. this paper presents an analysis of the ead documents (the largest analysis of ead documents to date). the analysis is paired with an evaluation of how well the documents support various aspects of online discovery. the paper also establishes a framework for thresholds of completeness and consistency to evaluate the results. we find that, while the ead standard and encoding practices have not offered support for all aspects of online discovery, especially in a large and heterogeneous aggregation of ead documents, current trends suggest that the evolution of the ead standard and the shift from retrospective conversion to new shared tools for improved encoding hold real promise for the future. by m. bron, m. proffitt and b. washburn introduction archivegrid is an aggregation of nearly two million archival material descriptions, including marc records from worldcat and finding aids harvested from the web. it is supported by oclc research as a corpus for experimentation and testing in text mining, data analysis, and discovery system applications and interfaces. archival collections held by thousands of libraries, museums, historical societies, and archives are represented in archivegrid. although roughly 90% of what is in archivegrid are marc records, as of april 2013 oclc research had harvested 124,009 ead encoded finding aids for inclusion in archivegrid[1]. this small segment of archivegrid is important because ead has been embraced by the archival community since it’s inception in the 1990s, and is supported by a range of tools designed specifically for archives, such as archivesspace, archivists’ toolkit, archon, calm, and others. in creating the archivegrid discovery interface, the project team has wrestled with what we can reasonably do with this corpus. for example, it would be useful to be able to sort by size of collection, however, this would require some level of confidence that the <extent> tag is both widely used and that the content of the tag would lends itself to sorting. other examples of desired functionality include providing a means in the interface to limit a search to include only items that are in a certain genre (for example, photographs) or to limit a search by date. again, we would need to have confidence that the metadata we have will actually support these features, and not leave out potentially important collections simply because of the absence of certain tags. specifically, we will consider how the variability of use of elements in finding aids affects discovery considering five different possible dimensions of a discovery system: search, browse, sort, limit, and display. as a warning to the reader: this paper delves deeply into ead elements and attributes and assumes at least a passing knowledge of the encoding standard. for those wishing to learn more about the definitions and structure, we recommend the official ead website or the less official but highly readable and helpful eadiva site[2]. related work the work that is the most closely related to our research was done by katherine m. wisser and jackie dean[1]. in 2010 wisser and dean solicited ead files repositories from institutions in order to ”identify encoding behavior.”[3] in total, 108 repositories submitted up to 15 finding aids for the analysis; 1,136 finding aids comprise the entire sample. the formal results of their analysis will be published in the fall 2013 edition of american archivist. we are grateful to the authors for sharing their early work with us, and note with interest that in many cases, their analysis of ead usage is quite similar. however, in some notable cases, the findings from the two samples diverge dramatically (see for example elements in <archdesc> above the <dsc> in table 9). as noted by wisser and dean some of this variation can be attributed to the many different ways in which ead files can be obtained. wisser and dean invited a limited contribution (12-15 finding aids) from a wide variety of repositories, including significant contributions from institutions outside of the us; even though wisser and dean carefully articulated that results would be anonymized, there is some chance that the results were somewhat skewed by the process of selecting files for inclusion. by contrast, our data set was assembled by harvesting ead documents from institutions directly, see below. contributing institutions have been motivated to contribute to archivegrid primarily to share information about their collections, not their ead practices. additionally, archivegrid is primarily constituted by repositories from the united states, with few institutions from europe or elsewhere represented in the data set. either or both of these key differences may account for divergence in findings between our work and that of wisser and dean. the 2010 report, “implications of marc tag usage on library metadata practices” focused on an analysis of the marc standard as reflected in world-cat [5]. although the emphasis of the the report was, similar to dean and wisser, meant to “inform community practice,” a secondary purpose was to draw conclusions about the suitability of marc data for machine matching and processing, which is similar to our desire to identify functionality (and gaps in functionality) that exist in our current ead corpus. oclc research regularly harvests ead documents from contributing institutions to update their representation in the archivegrid index.  the update cycle is roughly every six weeks.  institutions are contacted to obtain their permission to harvest and use the data in archivegrid, and to identify the target urls and rules for selection.  for some contributors, the harvesting rules are simple: a directory listing or an html page is made available to our crawler, with every link leading to an ead xml file on the contributor’s server.  for other contributors we may make use of a website designed for human visitors, applying custom include and exclude rules to the urls we find to select only links to ead documents.  though oai-pmh repositories and other more specialized harvesting protocols may be available at some contributor sites, we have seen little interest among contributors in their use, and currently we are using only standard http get requests for all the many hundreds of ead document providers.  maintaining the ead harvesting operation continues to be a significant component of the archivegrid support costs covered by oclc research. methods defining thresholds it is difficult to predefine thresholds for the level of usage of an element at which it becomes more or less useful for discovery. is an element that is used 95% of the time still useful but one that is used 94% not? in this paper we consider the thresholds resulting from working with our sample of documents. we will use the terminology documents and finding aids interchangeably throughout the paper. as an indicator for usage of an element we use the percentage of documents that contain the element at least once (% uniq). the nested nature of finding aids, however, influences the usage of elements as the absence of a parent element reduces the possibility of the occurrence of child elements. as an alternative indicator for usage we use the percentage of documents that contain an element in the sample of documents that contain the element’s parent element (% uniq in c). figure 1 shows how often the percentage of usage of an element falls into certain intervals. note that we use relative usage (% uniq in c) here. the distribution of element usage could be roughly divided into 4 groups: (i) usage between 0%-50% or low use; (ii) usage between 51%-80% or medium use; (iii) usage between 81%-95% or high use; (iv) usage between 96%-100% or complete use. although we will use these levels as a reference point in this document, we do so with a recognition that correlating usage with discovery is an artificial construct. in the absence of a more effective approach, we are using these levels as an initial framework for discussion. the absence of an element does not directly lead to a breakdown in a discovery system. it is more like a gradual decay of the effectiveness of a discovery system. but not all elements are created equally – in current archival discovery systems, we see a range of functionality that is offered, both in terms of search and advanced search options, as well as sorting features, and results display. within smaller aggregations, we might very well expect tag usage to be considerably more internally consistent than is the case in the archivegrid aggregation. but in the case of archivegrid and similar large aggregations of finding aids, what functionality can be reasonably supported, given the present state of the data? what functionality can we offer with assurance, if we look only at elements that are in the high or complete categories? figure 1: the distribution of percentage of element usage (% uniq in c). elements are nested and the absence of a parent element influences the occurrence percentage of a child element. for this reason we use the number of element occurrences relative to the occurrences of the parent element (% uniq in c). counting element occurrences finding aids follow the encoded archival description standard, which is a complex xml structure. as an example of the complexity of ead in implementation, we found more than 26,000 paths in our 129,009 document set. to provide a starting point for obtaining element counts we recreated the many (but not all) tables of element, attribute, and value counts as presented in the report by wisser et al. [4]. each table was recreated by performing one or more xpath queries over the corpus of finding aids. in the discussion of our analysis we do not follow the same structure as in wisser et al. [4] as our focus is on implications of element usage on discovery and presentation. where appropriate similarities and differences between element usage in our sample of finding aids and those used in wisser et al. [4] are reported. in the rest of the paper we use the following notation in our tables: (i) n is the total number of occurrences of an element; (ii) n uniq is the number of documents in which the element occurs at least once; (iii)   is the percentage of documents in our sample of ead documents (s= 124009) that contain the element at least once; and (iv)  is the percentage of documents that contain the element in the sample of documents (n=…) that contain a certain element. we will provide the size of each particular sample explicitly. for example, when considering the <eadheader> element that occurs in every document we get , which is the same as . we use  to indicate the percentage of documents that contain the element in the sample of documents that contain a certain element as collected by wisser. in most cases the sample size will be all documents in wisser’s sample, i.e., . finally, we use diff to indicate the percentage point difference between the percentage nuniq and nuniqk, i.e., between wisser’s and our sample. dimensions for analysis our analysis considered the following dimensions: search: all discovery systems have a keyword search function; many also include the ability to search by a particular field or element: examples include name, date, subject. browse: many discovery systems include the ability to browse finding aids: examples include browse by repository, browse by material type. results display: once a user has done a search, the results display will return portions of the finding aid to help with further evaluation: examples include title, dates, collection size. sort: once a user has done a search, they may have the option to reorder the results. examples include: order by date, order by title, order by size. limit by: once a user has done a search, they may have the option to narrow the results to only include results that meet certain criteria. this may be done through presentation of facets: examples include limit by collections with digital material, limit by repository. current discovery interfaces we reviewed a number of different discovery interfaces for finding aids in order to provide an overview of the type of search, browse, sort, limit, and display options that are generally available. interfaces included are: the online archive of california (http://www.oac.cdlib.org/), the northwest digital archive (http://nwda.orbiscascade.org/), texas archival resources online (http://www.lib.utexas.edu/taro/index.html), arizona archives online (http://www.azarchivesonline.org/xtf/search), the five colleges archives and manuscripts collection (http://asteria.fivecolleges.edu/index.html), the rocky mountain online archive (http://rmoa.unm.edu/), the harvard library’s online archival search information system (http://oasis.lib.harvard.edu/oasis/deliver/home?_collection=oasis). the interfaces we surveyed are very traditional in the capabilities they support — this is no doubt in part an outcome of the type of functionality that is supported in ead 2002. in addition to assessing the suitability of the archivegrid corpus for some general archival-specific discovery interfaces, we wanted to cast our net a little wider and speculate on how well ead may meet the needs of emerging nextgen (or nowgen!) approaches to discovery that may not be represented in our interfaces surveyed, or supported by 2002 era ead. emerging discovery apparatus include: support for geo-locating archival locations, subjects of collected materials, and other elements, to server map-based search interfaces.  examples of map-based discovery interfaces include: historypin (http://www.historypin.com/), whatwasthere (http://www.whatwasthere.com/), historvius (http://www.historvius.com/) similarly, we see support for event-based retrieval, using timelines or similar devices, as an area in which discovery systems are evolving.  some examples include: simile, example project timeline for jewish history http://simile.mit.edu/timeline/examples/religions/jewish-history.html, timeline view, philippine archives collection, nara http://www.archives.gov/research/military/ww2/philippine/timeline.html zagora archaeological project http://www.powerhousemuseum.com/zagora/timeline/ analysis details we now take a closer look at which elements might drive each function, how the aggregated data fits this purpose both in terms of meeting our  thresholds, and how well the content of key elements are fit for purpose. with each element, we’ve included a note about how they are used in archivegrid and in other discovery systems. date our analysis shows use of <unitdate> within the high-level <did> as medium (72.64% — see table 7); this makes <unitdate> values less than reliable for functions such as sort and limit by. consider, for example, a scenario where a researcher is interested in material from the second world war. filtering by a date range between 1939-1945 will result in only those documents being presented that have a <unitdate> assigned in that period and may lead to the researcher missing potentially relevant documents. alternatively, only those documents could be excluded that have a date outside of the indicated range. however, with a large amount of eads missing a <unitdate> field this approach defeats the purpose of filtering. investing effort to bring this element closer to high or complete may be warranted; however, to support dimensions beyond just display, the content of the field or contents of the “normal” attribute must be easily parseable. when we look at the content of <unitdate>, we find a wide range of descriptive practices, some of which could pose problems for machine parsing to support use in indexing and retrieval. another issue involved in using the <unitdate> field is that it can be used in several places, e.g., on its own in the top level <did> or as a subelement of <unittitle>. comparing the usage of <unitdate> in our collection of ead documents and that of wisser, we find that it is one of the elements where we see the greatest divergence, i.e., wisser’s sample shows a usage of <unitdate> in the <did> of 97.00%. in archivegrid, dates are used in: search: they are keyword searchable display: with the collection title (when available) in brief displays in other archival discovery systems: search browse sort display extent our analysis shows use of <extent> within the high-level <did> as medium (70.43% — see table 8); as with <unitdate>, the content of <extent> is quite varied and does not easily facilitate sorting, with values ranging from “miscellaneous artifacts” to “2 ceramic heads.” the syntax of the <extent> element (with attributes for @encodinganalog, @type, and @unit) does not currently lend itself to structuring data in a way that can be used for sorting without clear guidelines, tools to enforce appropriate encoding, and rigor on the part of institutions; retrospectively refitting to be utilized in sorting could be a daunting challenge for many institutions. many documents in the archivegrid corpus have multiple <extent> statements, further complicating matters, as the system would need to decide which one to sort, for example. for display, including  <extent> statements in order to help aid researchers in evaluating results seems fit to purpose. in archivegrid, extent is used in: search: extent values are keyword searchable display: presented in brief displays and separately in the display of individual collection descriptions in other archival discovery systems: sort display collection title our analysis shows use of <unittitle> in the high-level <did> as complete (99.93% — see table 7); this would suggest that it is suitable for all uses. however, for sorting and browsing, again, utility depends on the content of the element. if the content of the <unittitle> element is something generic like “records” or “papers” (in cases where perhaps the creator has been recorded separately in the origination element), then all functions may be less than ideal, but particularly sorting by title or creating browse lists. many interfaces either construct browse lists of collections titles, or allow users to sort results by title, or search within titles. not surprisingly, we found that the required <filedesc> element in the <eadheader> to be complete. although our analysis did not include elements below <filedesc>, we can assume that the required <titlestmt> and its required child, <titleproper> will be similarly complete. the fact that <titlestmt> is fully populated is good news for searching and display; however for sorting and constructing browse lists, we would need to have some assurance that the contents of <titleproper> are fit to purpose. this would be an opportunity for further evaluation, although a quick scan of the contents of <titleproper> encouragingly revealed that 42% of archivegrid finding aids have a @type attribute with the value “filing”, which is rather remarkable as there is no specified list of values for type. in archivegrid, collection titles are used in: search: they are keyword searchable display: collection titles appear in brief search results in other archival discovery systems: sort browse display subject our analysis shows use of <controlaccess> as medium (72.89% — see table 9); <controlaccess> is the parent element of both subject as well as other access points (such as <corpname>, <genreform>, <geogname>, and <persname>). our analysis did not include drilling down to use of <controlaccess> subelements. (given differences in library and archival practices, we would expect control of form and genre terms to be relatively high, and control of names and subjects to be relatively low.) in archivegrid, subjects are used in: limit by: we show <controlaccess> values for people, groups, places and topics as result overview facets for limiting a search result in other archival discovery systems: search browse material type researchers may wish to limit to or seek out material in a specific format, and our survey of discovery systems reveal that some systems support this functionality. our analysis did not include the children of <controlaccess>, which includes <genreform>. in archivegrid, material type is used for: search: material types in <genreform> are keyword searchable in other archival discovery systems: search browse limit by names (personal or corporate) names can be found in multiple places — for the the creator of a collection, is most logically found in <origination>, where both <persname> and <corpname> are child elements.  the use of the origination tag is medium (87.78% – see table 7); our analysis did not include evaluation of the use of <persname> and <corpname> in origination. otherwise, personal and corporate names as access points may be found in <controlaccess> (see above). name elements occur ubiquitously in ead version 2002, and our analysis did not include a detailed inventory of <persname> and <corpname> in the many places they can occur. a weakness of the distributed nature of names throughout ead documents is that without detailed annotations and co-references, discovery systems only have a shallow understanding of names and their relationship to the collection and to one another. discovery systems are not always able to differentiate between names when used in a creator context versus those covered in the description, which may show up as access points. in archivegrid, names are used for: search: names are keyword searchable limit by: names for people, groups and places appear in the result overview in other archival discovery systems: used in search used for limiting repository the name of the repository is found in the high-level did in <repository>. use of this element falls into the promising complete category (99.46%: see table 7). however, a variety of practice is in play, with the name of the repository being embellished with <subarea> and <address> tags nested within <repository>. to avoid the difficulties in handling these variations across a range of contributing institutions, archivegrid maintains a separate system to manage the form of the institution name for use in the system. in archivegrid, <repository> is not used as an access point, though archivegrid’s separately administered and controlled form of the repository name is used for search, browse, sort, limit and display. in other archival discovery systems, used in: browse limit by scope note, biographical note, abstract our analysis shows use of <scopecontent> as high (84.41% — see table 9), while <bioghist> (70.42% — see table 9) and <abstract> (79.20% — see table 7) are medium; all three are suitable for search and for display in a results view, although they can be quite lengthy. for search, its worth noting that the semantics of these elements are different, and may result in unexpected and false “relevance” for matches against descriptions in <bioghist> (about the person) and <scopecontent> and <abstract> (which may be more about the collection). in archivegrid, these notes are used in: search: notes are keyword-searchable display: <scopecontent> notes appear (in truncated form if lengthy) in brief search results in other archival discovery systems, used in: search display (in snippets or in their entirety) collections with digital content our analysis did not explore the use of <dao> or <daogrp> elements, which can be used in a variety of places in ead 2002. wisser and dean found that <dao> is used in 7.7% and 9.3% of the documents in their sample, putting both into the low category (see wisser, table 26, <dao> elements). however, with growing interest in digitized materials from archival collections, identifying those materials is of increasing importance. in archivegrid, we provide no mechanism for searching or identifying collections with digital content. in other archival discovery systems: limiting results to those with digital content flagging collections with digital content future work in order to make ead-encoded finding aids more well suited for use in discovery systems, the population of key elements will need to be moved closer to high or (ideally) complete. however, it is not only a matter of populating the elements, but ensuring that the data will reliably power key aspects of discovery systems. this will take concerted effort and tools, both on the part of individual institutions and groups. in the analysis of “nextgen” discovery services, we noted the use of geolocation-based discovery. although we would need to do further analysis in <controlaccess> to assess the usage for <geogname> in our document set, the current structure of the <geogname> element does not support geolocation functionality. however, as part of the redesign for ead3, ead is becoming more supportive of linked data and linked data structures. this may offer some hope for retrofitting ead data to be more suited for the task of meeting map-based discovery requirements. likewise, the data we have on hand does not suggest good support for event-based discovery, which would draw on well-structured dates, geographic subject terms, and topical subject terms (such as “battle of alma” or “great depression”). again, ead 2002 does not support the sort of encoding that would be necessary to serve event-based discovery, but ead3 may provide more appropriate structures. an optimum threshold for discovery? the picture for archival discovery and ead is decidedly mixed. on the one hand, we have elements that are in high or even complete use. on the other hand, we have many elements that are necessary for discovery interfaces that are in medium use; and even with elements that are in high or complete use, the contents of those tags are not always fit to purpose. this can be at least partly explained by ead’s history.  in the early days of ead the focus was largely on moving finding aids from typescript to sgml and xml. even with much attention given over to the development of institutional and consortial best practice guidelines and requirements, much work was done by brute force and often with little attention given to (or funds allocated for) making the data fit to the purpose of discovery. tag analyses such as the work described in this paper can help inform the development and implementation of the ead schema (indeed the work done by wisser and dean was considered in the development of ead3).  but our analysis suggests that the standard has most of the elements and attributes needed to effectively support discovery; what’s missing is agreement on and widespread application of best practices tied to supporting discovery. so, is the container list half empty? if the archival community continues on its current path then the potential of the ead format to support researchers or the public in discovery of material will remain underutilized. minimally, collection descriptions that are below the thresholds for discovery will hinder their discovery efforts and maximally will remain hidden from view. our paper provides suggestions for the elements where additional effort and investment are warranted to improve their utility for discovery systems. (we recognize that for some institutions, that additional effort may not be feasible or warranted; for their purposes they may find that html or pdf collection descriptions suffice.) or is the container list half full? perhaps with emerging evidence about the corpus of ead, continued discussion of practice, recognition of a need for greater functionality, and shared tools both to create new ead documents and improve existing encoding, we can look forward to further increasing the effectiveness and efficiency of ead encoding, and develop a practice of ead encoding that pushes collection descriptions across the threshold of discovery. tables table 1: (wisser table 1): general statistics for ead finding aids, using queries: /ead/*. element n n_uniq % [n_uniq/s] % [n_uniq/n=124009] % [(n_uniqk)/n=1136] diff eadheader 124009 124009 100.00 100.00 100.00 0.00 archdesc 124009 124009 100.00 100.00 100.00 0.00 frontmatter 46115 46115 37.19 37.19 24.60 12.59 eadgrp 0 0 0.00 0.00 0.00 0.00 archdescgrp 0 0 0.00 0.00 0.00 0.00 dscgrp 0 0 0.00 0.00 0.00 0.00 table 2: (wisser table 2): elements used within eadheader, using query /ead/eadheader/*. element n n_uniq % [n_uniq/s] % [n_uniq/n=124009] % [(n_uniqk)/n=1136] diff eadid 124445 124008 100.00 100.00 100.00 -0.00 filedesc 124009 124009 100.00 100.00 100.00 0.00 profiledesc 123103 123103 99.27 99.27 98.10 1.17 revisiondesc 42504 42501 34.27 34.27 32.70 1.57 table 3: (wisser table 3) attributes used with eadheader, using query //eadheader. element n n_uniq % [n_uniq/s] % [n_uniq/n=124009] % [(n_uniqk)/n=1136] diff countryencoding 107412 107412 86.62 86.62 89.50 -2.88 dateencoding 107377 107377 86.59 86.59 88.20 -1.61 findaidstatus 42910 42910 34.60 34.60 27.80 6.80 langencoding 117641 117641 94.86 94.86 95.00 -0.14 repositoryencoding 106370 106370 85.78 85.78 87.80 -2.02 scriptencoding 95230 95230 76.79 76.79 77.60 -0.81 table 4: (wisser table 4): attributes used with eadid, using query //eadid. element n n_uniq % [n_uniq/s] % [n_uniq/n=124009] % [(n_uniqk)/n=1136] diff countrycode 108668 108667 87.63 87.63 94.30 -6.67 mainagencycode 105351 105350 84.95 84.95 92.60 -7.65 publicid 45758 45758 36.90 36.90 31.10 5.80 url 38020 38020 30.66 30.66 42.30 -11.64 urn 2312 2312 1.86 1.86 3.90 -2.04 identifier 57260 57260 46.17 46.17 49.30 -3.13 table 5: (wisser table 8): elements within frontmatter, using query /ead/frontmatter/*. element n n_uniq % [n_uniq/s] % [n_uniq/n=46115] % [(n_uniqk)/n=279] diff titlepage 45726 45726 36.87 99.16 92.80 6.36 div 190 190 0.15 0.41 2.20 -1.79 table 6: (wisser table 9): values for @level within archdesc, using query //archdesc/@level. element n n_uniq % [n_uniq/s] % [n_uniq/n=124009] % [(n_uniqk)/n=1,136] diff collection 116957 116957 94.31 94.31 90.90 3.41 fonds 135 135 0.11 0.11 4.80 -4.69 class 9 9 0.01 0.01 0.30 -0.29 recordgrp 433 433 0.35 0.35 1.40 -1.05 series 2394 2394 1.93 1.93 0.60 1.33 subfonds 49 49 0.04 0.04 0.30 -0.26 subgrp 526 526 0.42 0.42 1.00 -0.58 subseries 46 46 0.04 0.04 0.00 0.04 file 2446 2446 1.97 1.97 0.40 1.57 item 987 987 0.80 0.80 0.30 0.50 otherlevel 25 25 0.02 0.02 0.10 -0.08 table 7: (wisser table 10): elements within archdesc/did, using query /ead/archdesc/did/*. element n n_uniq % [n_uniq/s] % [n_uniq/n=124009] % [(n_uniqk)/n=1,136] diff abstract 102792 98218 79.20 79.20 86.60 -7.40 container 5447 3471 2.80 2.80 0.40 2.40 langmaterial 112938 109232 88.08 88.08 89.90 -1.82 materialspec 41 41 0.03 0.03 1.60 -1.57 origination 113684 108853 87.78 87.78 89.00 -1.22 physdesc 135126 122402 98.70 98.70 97.20 1.50 physloc 53564 45620 36.79 36.79 27.80 8.99 repository 123343 123330 99.45 99.45 99.60 -0.15 unitdate 97247 90080 72.64 72.64 97.00 -24.36 unitid 119911 114898 92.65 92.65 90.10 2.55 unittitle 123959 123916 99.93 99.93 100.00 -0.07 table 8: (wisser table 11): elements within archdesc/did/physdesc, using query /ead/archdesc/did/physdesc/*. element n n_uniq % [n_uniq/s] % [n_uniq/n=124009] % [(n_uniqk)/n=1,136] diff dimensions 666 576 0.46 0.46 1.80 -1.34 extent 122613 87339 70.43 70.43 76.30 -5.87 physfacet 2000 1520 1.23 1.23 1.70 -0.47 table 9: (wisser table 12): elements within archdesc:above the dsc, using query /ead/archdesc/*. element n n_uniq % [n_uniq/s] % [n_uniq/n=124009] % [(n_uniqk)/n=1,136] diff accessrestrict 55751 55579 44.82 44.82 86.20 -41.38 accruals 694 694 0.56 0.56 7.10 -6.54 acqinfo 40668 40451 32.62 32.62 68.00 -35.38 altformavail 2293 2289 1.85 1.85 12.70 -10.85 appraisal 4613 4602 3.71 3.71 4.80 -1.09 arrangement 40979 40627 32.76 32.76 65.50 -32.74 bibliography 4573 4083 3.29 3.29 10.10 -6.81 bioghist 89103 87333 70.42 70.42 87.30 -16.88 controlaccess 92124 90390 72.89 72.89 85.00 -12.11 custodhist 8375 8366 6.75 6.75 14.10 -7.35 descgrp 67684 56446 45.52 45.52 32.00 13.52 fileplan 50 44 0.04 0.04 0.60 -0.56 index 1231 656 0.53 0.53 1.20 -0.67 odd 9594 8145 6.57 6.57 9.70 -3.13 originalsloc 988 973 0.78 0.78 3.40 -2.62 otherfindaid 6529 6271 5.06 5.06 11.90 -6.84 phystech 900 897 0.72 0.72 4.20 -3.48 prefercite 49015 48989 39.50 39.50 85.40 -45.90 processinfo 27249 26623 21.47 21.47 0.00 21.47 relatedmaterial 23932 23676 19.09 19.09 40.30 -21.21 runner 10822 10822 8.73 8.73 1.10 7.63 scopecontent 105384 104670 84.41 84.41 93.40 -8.99 separatedmaterial 5789 5691 4.59 4.59 14.80 -10.21 userestrict 41365 40749 32.86 32.86 68.30 -35.44 table 10: table 13: the inclusion of dsc in finding aids, using query //dsc. element n n_uniq % [n_uniq/s] % [n_uniq/n=124009] % [(n_uniqk)/n=1,136] diff < dsc > 98663 94473 76.18 76.18 90.30 -14.12 multiple < dsc > s 98663 2075 1.67 1.67 2.40 -0.73 table 11: (wisser table 14): dsc type attributes, using query //dsc/@type. element n n_uniq % [n_uniq/s] % [n_uniq/n=99023] % [(n_uniqk)/n=1,105] diff analyticover 3156 3149 2.54 3.18 5.10 -1.92 combined 49205 49184 39.66 49.67 66.50 -16.83 in-depth 36433 35876 28.93 36.23 16.70 19.53 othertype 1725 1572 1.27 1.59 3.50 -1.91 table 12: (wisser table 15): c-c12 tags, using query //c | //c01 | //c02 | //c03 | //c04 | //c05 | //c06 | //c07 | //c08 | //c09 | //c10 | //c11 | //c12. element n n_uniq % [n_uniq/s] % [n_uniq/n=96548] % [(n_uniqk)/n=1,053] diff c 4745698 14440 11.64 14.96 11.10 3.86 c01 1650659 78600 63.38 81.41 88.00 -6.59 c02 7432993 59217 47.75 61.33 72.50 -11.17 c03 6625963 29136 23.50 30.18 41.80 -11.62 c04 2927180 12819 10.34 13.28 20.60 -7.32 c05 1312217 5587 4.51 5.79 10.70 -4.91 c06 598647 2266 1.83 2.35 4.60 -2.25 c07 261648 922 0.74 0.95 2.00 -1.05 c08 90401 331 0.27 0.34 0.70 -0.36 c09 21514 110 0.09 0.11 0.30 -0.19 c10 3578 36 0.03 0.04 0.10 -0.06 c11 823 7 0.01 0.01 0.00 0.01 c12 96 2 0.00 0.00 0.00 0.00 table 13: (wisser table 16): values for level attribute on c, c/@level, using query //c/@level | //c01/@level | //c02/@level | //c03/@level | //c04/@level | //c05/@level | //c06/@level | //c07/@level | //c08/@level | //c09/@level | //c10/@level | //c11/@level | //c12/@level. element n n_uniq % [n_uniq/s] % [n_uniq/n=96548] % [(n_uniqk)/n=1,053] diff collection 13489 4782 3.86 4.95 2.10 2.85 fonds 418 95 0.08 0.10 0.70 -0.60 class 63134 2113 1.70 2.19 1.20 0.99 recordgrp 1535 193 0.16 0.20 0.70 -0.50 series 398727 58480 47.16 60.57 77.70 -17.13 subfonds 3210 637 0.51 0.66 1.70 -1.04 subgrp 5573 430 0.35 0.45 3.10 -2.65 subseries 466366 16974 13.69 17.58 35.30 -17.72 file 11419524 36262 29.24 37.56 56.90 -19.34 item 3480272 20415 16.46 21.14 24.20 -3.06 otherlevel 368942 6225 5.02 6.45 9.10 -2.65 table 14: (wisser table 17): c-c12/did elements, using query //c/did/* | //c01/did/* | //c02/did/* | //c03/did/* | //c04/did/* | //c05/did/* | //c06/did/* | //c07/did/* | //c08/did/* | //c09/did/* | //c10/did/* | //c11/did/* | //c12/did/*. element n n_uniq % [n_uniq/s] % [n_uniq/n=96548] % [(n_uniqk)/n=1,053] diff abstract 1421043 3850 3.10 3.99 2.50 1.49 container 24951558 72377 58.36 74.96 82.50 -7.54 langmaterial 46798 1127 0.91 1.17 6.10 -4.93 materialspec 22870 106 0.09 0.11 1.30 -1.19 origination 1308346 4090 3.30 4.24 8.10 -3.86 physdesc 3967094 37749 30.44 39.10 54.40 -15.30 physloc 1343791 5978 4.82 6.19 5.80 0.39 repository 34923 29 0.02 0.03 0.30 -0.27 unitdate 9613593 41894 33.78 43.39 90.60 -47.21 unitid 7167784 31035 25.03 32.14 46.20 -14.06 unittitle 25228059 92888 74.90 96.21 98.90 -2.69 table 15: (wisser table 18): c-c12/did/physcdesc elements, using query //c/did/physdesc/* | //c01/did/physdesc/* | //c02/did/physdesc/* | //c03/did/physdesc/* | //c04/did/physdesc/* | //c05/did/physdesc/* | //c06/did/physdesc/* | //c07/did/physdesc/* | //c08/did/physdesc/* | //c09/did/physdesc/* | //c10/did/physdesc/* | //c11/did/physdesc/* | //c12/did/physdesc/*. element n n_uniq % [n_uniq/s] % [n_uniq/n=96548] % [(n_uniqk)/n=1,053] diff dimensions 144079 1378 1.11 1.43 5.20 -3.77 extent 2401903 24495 19.75 25.37 36.60 -11.23 physfacet 164430 613 0.49 0.63 6.80 -6.17 table 16: (wisser table 19): other elements found in c-c12, using query //c/* | //c01/* | //c02/* | //c03/* | //c04/* | //c05/* | //c06/* | //c07/* | //c08/* | //c09/* | //c10/* | //c11/* | //c12/*. element n n_uniq % [n_uniq/s] % [n_uniq/n=96548] % [(n_uniqk)/n=1,053] diff accessrestrict 600069 4844 3.91 5.02 10.70 -5.68 accruals 12 11 0.01 0.01 0.00 0.01 acqinfo 68066 1477 1.19 1.53 4.50 -2.97 altformavail 252282 766 0.62 0.79 2.70 -1.91 appraisal 48 30 0.02 0.03 0.70 -0.67 arrangement 31945 5746 4.63 5.95 19.00 -13.05 bibliography 2067 48 0.04 0.05 1.50 -1.45 bioghist 12511 1132 0.91 1.17 4.60 -3.43 controlaccess 243134 2149 1.73 2.23 5.10 -2.87 custodhist 26224 181 0.15 0.19 2.20 -2.01 descgrp 2703 31 0.02 0.03 1.80 -1.77 index 386148 835 0.67 0.86 0.70 0.16 note 1180397 11265 9.08 11.67 20.30 -8.63 odd 242182 2663 2.15 2.76 7.20 -4.44 originalsloc 9959 211 0.17 0.22 1.00 -0.78 otherfindaid 1945 247 0.20 0.26 2.30 -2.04 phystech 8439 300 0.24 0.31 1.50 -1.19 prefercite 1995 264 0.21 0.27 0.10 0.17 processinfo 26332 1084 0.87 1.12 3.80 -2.68 relatedmaterial 16727 882 0.71 0.91 4.40 -3.49 runner 0 0 0.00 0.00 0.00 0.00 scopecontent 1852092 33483 27.00 34.68 61.30 -26.62 separatedmaterial 2784 208 0.17 0.22 0.00 0.22 userestrict 2993 580 0.47 0.60 3.20 -2.60 table 17: (wisser table 20): content tags in dsc, using query //dsc//*. element n n_uniq % [n_uniq/s] % [n_uniq/n=96548] % [(n_uniqk)/n=1,053] diff corpname 373402 6082 4.90 6.30 8.40 -2.10 famname 3644 914 0.74 0.95 1.70 -0.75 function 996 53 0.04 0.05 0.00 0.05 genreform 351956 6988 5.64 7.24 5.10 2.14 geogname 1023771 6653 5.36 6.89 6.30 0.59 name 34339 380 0.31 0.39 1.40 -1.01 occupation 25284 285 0.23 0.30 0.40 -0.10 persname 2610548 11970 9.65 12.40 12.90 -0.50 subject 1239139 2419 1.95 2.51 4.70 -2.19 references [1]  in april 2013, the archivegrid index contained 1,632,246 marc records, 119,984 ead records, 61,551 html records, and 4,532 pdf records.  the ead count in the index is lower than the set of documents analyzed, to avoid duplicating their display for certain contributors who supply corresponding marc records. [2] library of congress ead website: http://www.loc.gov/ead/index.html; eadiva: http://eadiva.com/. [3] e-mail to archives and archivists listserv, november 15, 2010. [4] wisser, katherine m, and jackie dean, ead tag usage: community analysis of the use of encoded archival description elements, article submitted for publication in the american archivist [5] smith-yoshimura, karen, catherine argus, timothy j. dickey, chew chiat naun, lisa rowlinson de ortiz, and hugh taylor. 2010. implications of marc tag usage on library metadata practices. about the authors marc bron is a researcher at the intelligent systems lab amsterdam, where he is about to complete his phd in information retrieval. his dissertation focused on improving accessibility to information stored in cultural heritage institutions by developing algorithms and interactive retrieval systems that support exploration and contextualization. during his phd marc has published over 20 papers at top tier conferences, journals, and workshops. his current research direction aims to develop new collaborative search methods for users of archival collections. bruce washburn is a consulting software engineer in oclc research.  he provides software development support for oclc research initiatives and participates as a contributing team member on selected research projects.  in addition, he provides software development support for selected oclc products and services. at oclc washburn has been a part of the product teams that developed and maintain camio,  archivegrid, the worldcat search api, and oaister. merrilee proffitt is a senior program officer in oclc research. she provides project management skills and expert support to institutions represented within the oclc research library partnership. merrilee has authored or co-authored articles, guidelines, and reports for a variety organizations and professional journals. she is frequently an invited speaker at international professional conferences and workshops on topics relating to digital libraries and special collections. her current projects and interests include: archival description, increasing access to special collections, looking at developing better relationships between wikipedia and cultural heritage institutions, and how massively open online courseware (moocs) may impact libraries. she is a member of the small but mighty archivegrid team. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – editorial introduction: on being on the code4lib journal editorial committee mission editorial committee process and structure code4lib issue 26, 2014-10-21 editorial introduction: on being on the code4lib journal editorial committee behind the scenes of the the code4lib journal… when faced with writing this introduction, i wondered how many people read introductory editorials in journals, but since you are here, some must. then i wondered what to write about. it eventually occurred to me that some of you might be interested in something of what goes on behind the scenes of the journal. i joined the code4lib journal editorial committee (ec) in 2009 with some trepidation. however, any fears i had were quickly alleviated when the other editors welcomed me and proved glad to answer any questions that i had. my orientation to the work of the ec was aided by the extensive documentation on our wiki (http://wiki.code4lib.org/category:code4lib_journal), which remains a valuable reference. in many ways, the ec is a microcosm of the code4lib community. it is a leaderless group composed of dedicated volunteers. the ec members take turns serving as the coordinating editor who shepherds a specific issue. the journey to each issue may be chaotic and take many twists and turns, but somehow it always comes together in the end. members of the ec participate in discussions on the ec’s closed list. the practices and policies of the journal are continually evolving. anyone can suggest a change. potential changes generate respectful discussion (and sometimes sharp disagreement). new editors often bring new perspectives and ideas. for example, one new editor felt strongly that a professional journal should have a set style guide for references for consistency and to make it easier to ensure proper citations. other members of the ec felt a set style imposed an unnecessary burden on authors and that the only requirement for citations should be that they are consistently formulated in a given article and findable. in this case, a compromise was reached and we adopted a recommended style. some questions reflect ongoing tensions. every time we issue a call for new editors, there are those who advocate for accepting a higher number of new editors so that we can publish more articles. we receive many high-quality proposals and the ec is often stretched very thin trying to publish as many as possible. more new editors would bring more enthusiasm and new ideas. others call for a smaller number to keep the ec more cohesive and focused, while also combating the diffusion of responsibility and temptation of social loafing that comes with a larger group. at some point, too many editors and articles would make the work of the coordinating editor unmanageable under the current model. there is a parallel discussion about how high to make the bar for acceptance and how much work it is reasonable to ask an editor to put into an article to get it to the point where it meets our standards for publication. the publishing cycle begins with a call for proposals on various email lists. the incoming proposals are put on a spreadsheet for voting and are discussed on our closed email list. we have adjusted the voting protocol over time, but seem to have settled on a satisfactory algorithm. we used to vote on article proposals in a rolling fashion, but recently changed to using a set deadline. the set deadline makes it easier to rank articles when we have too many and helps with matching editors to articles. a disadvantage is that many authors end up with a shorter time period between acceptance and the due date for a first draft. in most cases, voting gives a clear consensus. then there are those that we tussle over. there are disagreements over such questions as whether a proposal is in scope, is at the appropriate technical level or has a broad appeal for our audience. in some cases, editors see potential in a borderline proposal and want to work with authors to improve it. others say this has rarely worked out well and argue that it’s better to just reject such proposals up front. individual editors sign up to be the editor or second reader for articles that they are interested in and have the appropriate background for. editors contact the authors of their chosen articles and serve as the primary contact for those authors throughout the publishing process. articles go through two drafts. when each draft is submitted, it is reviewed by the editor and the second reader and by other members of the ec if they choose. they then provide feedback to authors on the drafts. a final draft is input into wordpress for review by the authors and the ec. the ec votes again on the articles and any final editorial changes are made. when the time comes, the coordinating editor takes a deep breath and hits the publish button. the first time i did this, it didn’t lead to the result i was trying to get. i was very grateful for the instructions on the wiki for backing out and republishing! the coordinating editor also uploads the issue to the directory of open access journals (http://doaj.org/). when the new issue is live, the ec begins publicizing it and the whole process starts again. having been on both the author side and the editorial side of journals using traditional blind peer review, as well as the peer-edited process used by the code4lib journal, i have found that for me the primary benefit of peer-editing is the interaction between editor and author. the articles i have written for the code4lib journal have been much improved by the iterative feedback from the editors. from the editorial side, i have seen many articles become clearer and more coherent as a result of the revision process. depending on the article, the revision process may involve a significant reworking of the article and a substantial time investment by both the editor and the author. in contrast, blind peer review is not so much a conversation as a one-way broadcast. my experience is that the feedback tends to be much briefer and it is not easy for either side to ask for clarification. of course, there are some dangers attached to an open process. despite our best intentions, the ec may be influenced by factors external to the content of the article. it also becomes harder to reject an article after having worked so closely with the author. articles do fail the final vote, but in the past, we have ended up publishing a few articles about which many of the ec had serious reservations. publishing in the code4lib journal may not be right for you if you are facing a tenure or promotion review by a conservative committee. however, if your goal is to communicate with a broad audience and make an impact on your peers and the field, the code4lib journal is the perfect vehicle. for a long time, i wasn’t sure if anyone had read my first article in a gated journal. i had no doubts that people were reading my first article in the code4lib journal as soon as it was published. the journal continues to explore new spaces. for issue 28 we are planning our first themed issue, on diversity and library technology. being a member of the code4lib journal editorial committee has been a lot of hard work, and the process of working with the other members of the ec and with authors has been both challenging and enjoyable. the reward for all this work is the knowledge that i am contributing to something that makes a significant and positive impact on our field. in this issue, we again have a slate of articles that have valuable things to say to our community. we have two articles related to the challenges of web-scale archiving. archiving the web: a case study from the university of victoria provides an overview of the tools for and challenges of archiving websites in the context of the university of victoria’s experience. technical challenges in developing software to collect twitter data explains how george washington university handled the technical challenges of turning its application for collecting social media data from twitter from a research prototype into something that is sustainable and useful for a wider audience. we also have two articles focused on using angularjs. exposing library services with angularjs provides an introduction to the javascript framework angularjs and introduces some angularjs modules for accessing library services. hacking summon 2.0 the elegant way shows how to reverse-engineer the angularjs-based summon 2.0 interface and explains how to use this knowledge to customize the interface. parsing and matching dates in viaf describes the challenges of interpreting the numerous ways of expressing dates found in authority records from various sources so they can be standardized and compared. mdmap: a tool for metadata collection and matching looks at a novel way of generating metadata to describe digitized books by evaluating and merging existing metadata that is gathered from many sources. using zapier with trello for electronic resources troubleshooting workflow shows how these free tools have been used to create a more efficient and effective process for tracking and responding to e-resource access problems. finally, developing applications in the era of cloud-based saas library systems uses ex libris’ developer network as an example of how vendors are addressing issues such as security, multi-tenancy, latency, and analytics in the cloud environment. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – adaptive digital library services: emergency access digitization at the university of illinois at urbana-champaign during the covid-19 pandemic mission editorial committee process and structure code4lib issue 51, 2021-06-14 adaptive digital library services: emergency access digitization at the university of illinois at urbana-champaign during the covid-19 pandemic this paper describes how the university of illinois at urbana-champaign library provided access to circulating library materials during the 2020 covid-19 pandemic. specifically, it details how the library adapted existing staff roles and digital library infrastructure to offer on-demand digitization of and limited online access to library collection items requested by patrons working in a remote teaching and learning environment. the paper also provides an overview of the technology used, details how dedicated staff with strong local control of technology were able to scale up a university-wide solution, reflects on lessons learned, and analyzes nine months of usage data to shed light on library patrons’ changing needs during the pandemic. by kyle r. rimkus, alex dolski, brynlee emery, rachael johns, patricia lampron, william schlaack, angela waarala background the covid-19 pandemic made itself felt across the university of illinois system on march 11, 2020 with the official announcement that its three campuses in chicago, springfield, and urbana-champaign would immediately migrate to online teaching and prohibit all work-related travel. [1] nine days later, illinois governor j.b. pritzker issued a statewide stay-at-home order. [2] in response, the libraries at the university of illinois at urbana-champaign closed their doors to patrons with the expectation of reopening on april 7, 2020. [3] however, as we write this article in may of 2021, all library buildings remain, with some restricted exceptions, shuttered to the public. the university has shifted the predominance of its operations to online teaching, learning, and remote work where possible (with the hope and expectation of easing these restrictions during summer 2021). in response to the university’s shift to distance learning, the library took a series of measures designed to extend services to remote patrons who sought access to collections materials. in early april, 2020, the university announced the summer semester would be fully remote [4], and the library announced its participation in the hathitrust emergency temporary access system, or etas [5] (more details on etas implementation to follow).  in june, a fully remote summer semester and an anticipated remote or hybrid fall semester looming, the library began devising a plan to meet the collection “fulfillment” needs of students. on june 2, the university allowed certain campus units to return to on-site work in a restricted capacity [6], with provisions for employee covid-19 testing [7]. while the library remained for the most part closed to the public, the university allowed staff members who maintained their testing protocol to work on-site in socially distanced conditions. on august 11, 2020, the library formally announced its policy for an “electronic-first” fulfillment strategy. [8] this policy prioritized the purchase of e-books over print when available, and specified that books available in the hathitrust etas would not circulate to patrons. it also meant that the library would digitize items unavailable through e-book vendors or etas and make the digital surrogates available to patrons in a restricted manner. for this reason, the library set to work beginning in june 2020 to introduce a new mode of providing patrons virtual access to books. our “emergency access” approach went live on july 20, 2020, and persists to the present (may, 2021). however, an end may be in sight. the university has suggested a return to a residential teaching and learning model for fall, 2021, and the library intends to follow suit by turning off the hathitrust etas and transitioning away from full-fledged emergency access policies and procedures in early summer, while keeping some on-demand digitization services in place for remote students and faculty. brief literature review the subject of this paper, one library’s approach to providing access to materials during a pandemic, is timely given that little has been published to date on the subject. a paper has been published on the topic in this journal [9], and we expect to see similar reports and analyses in the months and years to come as more librarians reflect on the services they altered or introduced in 2020. the subject of controlled digital lending, while not quite the same as what we in this article are calling emergency access digitization, did loom large in our minds as we planned for and implemented our solution. while we did not conduct a deep literature review on this subject, one white paper in particular served as an important reference for our understanding of laws related to providing restricted online access to works protected by copyright [10]. the emergency access workflow the university of illinois at urbana-champaign (uiuc) library’s approach to providing patrons with access to circulating print materials during the pandemic consisted, from a bird’s eye view, of the following: in the library’s online catalog (powered by alma/primo after a recent migration from voyager [11]), we enabled the hathitrust etas. if an item was available there, the print item would not circulate and patrons were obliged to “check it out” virtually to gain restricted online access. for items not available in the hathitrust etas, patrons would request access to books in the library’s online catalog and specify in their request form whether they preferred print or digital format. for print requests, the library placed items in lockers on site in the library for patrons to pick up in person. for digital requests, the library digitized items and made them available for online viewing in a restricted manner, prior to depositing them into the hathitrust digital library. as simple as this approach may sound, its execution required detailed planning and constant refinement. as this article focuses on digitization, we will outline our entire process while placing an emphasis on what went into initiating an on-demand emergency access digitization service. our process is best understood in two workflow diagrams that demonstrate what happened when patrons requested a book from our catalog. the first diagram (figure 1) breaks down the steps leading up to what will be the focus of this paper and is the subject of the second diagram (figure 2); that is, the workflow for items after they had been identified for digitization. figure 1. first steps of the library’s digital-first fulfillment strategy (adapted from a diagram created by mary laskowski). as figure 1 shows, the fulfillment process always began with a patron catalog request, and with staff from the item’s owning library verifying whether the item was already available in electronic format. if it was not, they pulled the item and routed it in a shipping bin to preservation services in the main library, where staff either routed items to lockers for in-person pickup or digitized them for online access. to the greatest extent possible, preservation staff sought to honor patrons’ access format preferences for print or digital, which was indicated in the patron’s request form in the “is digital or print access preferred? other info?” field. however, certain physical considerations made digitization impractical (e.g., books with very tight spines or an abundance of foldouts), and these items were routed directly to lockers. for more detail on this process, figure 2 below breaks down all of the steps taken in preservation services once we received a book: figure 2. triage, digitization, and access workflow. hathitrust etas access our goal was to deposit digitized books into the hathitrust digital library, where, if they were protected by copyright, patrons could gain limited and restricted access to them. according to its website (https://www.hathitrust.org/etas-description), the etas “permits special access for hathitrust member libraries that suffer an unexpected or involuntary, temporary disruption to normal operations, such as closure for a public health emergency, requiring the library to be closed to its patrons, or otherwise restrict print collection access services. the service makes it possible for member library patrons to obtain lawful access to specific digital materials in hathitrust that correspond to physical books held by their own library. the emergency temporary access service will enable many hathitrust member libraries to continue supporting the teaching, learning, and research mission of their institutions during said disruption in service.” in the uiuc library’s announcement of adopting a “digital-first” approach to library item fulfillment, we clarified that “students, faculty, and staff may log in to hathitrust and view all volumes that hathitrust has verified as being held in our print collection, even if in copyright.  users can read and search the book online, but they will not be able to download the book in full,” and assured patrons that, “through the etas, our faculty, staff, and students now have reading access to 46.65% of our print collection (based on overlap data generated by hathitrust staff from the most recent catalog update we supplied to them).” in practice, this meant that when patrons discovered the catalog record for an in-copyright book that was available in the hathitrust etas, they were unable to check it out in print format, but could click on a “full text available in hathitrust” link in order to gain restricted access to a digital surrogate of the item (figure 3). we made hathitrust links available in the catalog using a primo ad-on that matched the record’s oclc numbers with oclc numbers for items in the hathitrust digital library [12]. figure 3. links to non-circulating items available in hathitrust etas in the library catalog. on clicking the hathitrust link, the catalog routed users to the hathitrust etas, where they were prompted to “check out” the work for one hour at a time. figure 4. hathitrust etas item checkout prompt. once accessing the item in hathitrust, the patron could turn pages to view and read using the system’s native item viewer (figure 5). due to copyright restrictions, the etas viewer did not allow advanced features such as downloading page images. figure 5. the hathitrust etas item viewer. the end user experience the library’s goal was to fulfill all patron requests within 14 days. once an item arrived in preservation services, it generally took one day to route to a locker. digital requests required one to five days to digitize and one additional day to ingest into our access system and make it available to the patron. from there, it took one to three days to deposit into hathitrust and two additional days for hathitrust to ingest the item and for us to update our catalog with a link to the item for broader patron access. while the hathitrust etas was the end point for all digitized items, patrons were provided interim access to the digitized content in our local digital collections platform in order to meet their needs as swiftly as possible. as mentioned above, the process began with a request from the catalog (figure 6). figure 6. library catalog request form. after specifying a preference for digital format and clicking “request,” staff sent a courtesy email stating that the item had been accepted into the local digitization queue (figure 7). figure 7. email confirming item has been placed in digitization queue. upon digitization, patrons received another automated email, this one from our digital collections platform, notifying them of the availability of their requested item (figure 8). (to note: we will detail the technology we used to digitize and make items available in the digital collections platform in the section “digitization” below). figure 8. automated email confirming availability of digitized item. once authorized, the patron encountered a digital object page similar to others in the library’s digital collections platform [13], but stripped of page elements we provide for open access items such as a download feature and citation links (see figure 9 and, for implementation details, the section entitled digitization below). we granted patrons exclusive access to their requested items on this platform for 21 days, after which all items became broadly accessible to the requesting patron and others in the hathitrust etas. figure 9. patron access in the uiuc digital collections platform. communication and work tracking for local communication, we established an emergency access digitization space using an instance of microsoft teams (figure 10). here, we managed project communication, housed a manual that defined our policies and procedures, and tracked workflows. we also established a shared email list to simplify library-wide communication on the topic of emergency access digitization. figure 10. workflow tracking in microsoft teams. all items that came through preservation services were tracked using the “project” feature of our medusa digital preservation platform (figure 11). we developed this feature for queuing up items for preservation digitization prior to the pandemic, but were able to adapt many of its features to serve this project such as scanning item barcodes and importing catalog metadata. figure 11. tracking items queued for digitization using the “project” feature of the medusa digital preservation repository. digitization the uiuc library’s internet archive scanning center played a modest role in this project, digitizing 39 out-of-copyright items. the university library has hosted its internet archive scanning center on site since 2007. the number of scanning stations in the center fluctuates based on need, but currently the internet archive operates three scanning stations supported by three full-time staff. during fiscal year 2019, the scanning center digitized 1,179,650 pages and 4,841 items total. we digitized the majority of items for this project (764) in our own digitization services group, which is housed in the library’s preservation services unit. operationally, digitization services staff reformat visual materials, primarily in print and paper-based formats, for preservation and access. prior to the pandemic, the group was managed by one academic professional, with one full-time civil service digital imaging specialist, one half-time graduate assistant, and occasional hourly workers. over the course of the pandemic, the group was able to complete the long-planned hiring of a second digital imaging specialist. to meet demand for emergency access digitization, the group took on two temporary transfers of civil service library staff to serve as interim digital imaging specialists, and hired an extra help worker to provide imaging and workflow support. for emergency access digitization, digitization services staff relied on a variety of imaging equipment to complete their work. their work horse was an imaging station called the bc100, which had previously been used primarily for special collections materials like bound rare books, pamphlets, and historical theses and dissertations (see figure 12). the bc100 [14] is a dual-camera system with phase one p65 camera backs, northlight led lighting, schneider apo-digitar 90mm f/4.5 lenses, eizo color-calibrated monitors, and mac mini workstations. bc100 digitization operators use a pneumatic book cradle and foot pedals for platen control and remote triggers to make exposures. digitization specialists used capture one ch photo editing software to capture raw image files with digital cameras tethered to mac mini computers via firewire. they controlled camera settings remotely through a secondary application called dtdch shutter integration solution. autocrop functions and custom apple scripts in this software helped increase productivity and efficiency via automation of repetitive processes [15]. figure 12. the bc100 imaging station in action. the bc100 was not the only equipment we used for emergency access digitization. in the first months of the project, we borrowed a piece of equipment known as the knowledge imaging center bookeye (figure 13) from a public services unit in the library closed due to the pandemic. built as a walk-up digitization kiosk for library patrons unskilled in advanced digitization, we found the bookeye poorly suited to the sort of high-volume, high-quality production work we sought to do, primarily because its interface provided many barriers to control by an advanced operator, leading to cumbersome file processing, not to mention that the bookey’s page images were on the lower end of acceptable quality for our required specifications. we ceased using this machine as soon as we were able to procure additional imaging stations better suited to our needs. while we expect that the bookeye system will serve a valuable need as a walk-up digitization kiosk for library patrons, we cannot recommend it for use in a digitization lab environment. figure 13. the bookeye scanning kiosk, which we used briefly but concluded was unsuited for our needs. to meet the rising demand for digitized emergency access materials, we began planning in october, 2020 to procure a “do-it-yourself” imaging station built from equipment and materials available on the consumer market, and an additional station built from a combination of consumer-market and speciality-market components. while we hoped to have these two stations ready by december, 2020, it wasn’t until early 2021 that we had all of the equipment in place. this was due primarily to purchasing and shipping delays related to the pandemic, as well as the challenge of fitting new imaging stations into a room formerly used as office space. library facilities, it, and shipping were integral to adding new equipment, keeping staff safe, and solving logistical puzzles. after much rearranging of our office and lab spaces, we were able to install the two new imaging stations housed in box-shaped black canopy tents for control of lighting. the “diy” station (figure 14) features a kaiser book copy easel cradle on a  beseler copy stand canon 5dsr, a 24-70 2.8 canon lens, and rotolight anova pro led lights set on continuous mode at 100% and 5500 kelvin color temperature. we found these lights adequate for general collection image quality needs, but not powerful enough for higher-quality special collections workflows. in order to offset their low power output, they require a higher iso than is optimal for the camera. figure 14. the “diy” imaging station, built primarily of consumer-market components. the second additional station (figure 15) uses a digital transitions atom [16] copy stand equipped, based on image quality needs, with a canon 5dsr or a p40 phaseone medium format digital back we had on hand with a schneider lens. this station utilizes digital transitions photon led continuous lights tethered into capture one for phase one software [17]. figure 15. imaging station built of consumer-market and speciality-market photography equipment using a digital transitions atom copy stand. in order to simplify training for new staff, we decided to adapt local digitization and file packaging processes already in use in our special collections workflows for emergency access digitization. as ingest into the hathitrust digital library was our goal and we had no interest in taking up more server space with temporary files than was needed, we decided to photograph all page images to meet hathitrust specifications rather than our own higher standards for special collections preservation digitization. this meant producing files at 8-bit bit-depth and 400 dpi, as opposed to 16-bit images shot at 600 dpi or higher for special collections materials. furthermore, since long term preservation was not a priority for emergency access items, our staff streamlined unit quality control procedures to prioritize user accessibility over perfectionism. in practice, this meant occasionally compromising on technical standards such as exposure or white balance for the sake of providing a complete, readable copy of the material for users to access as quickly as possible. while we compromised on the in-house standards we use for imaging special collections materials, we made sure to meet hathitrust’s lower but still sufficient quality standards for digitized general collections. for the past three years, staff in preservation services have worked with digital library technical coordinator henry borchers to develop speedwagon [18], a gui-based software tool that pulls together a variety of file processing scripts used to generate properly packaged items for deposit into repository platforms (figure 16). our hathitrust packaging workflow included generating optical character recognition transcript text files for each page image, validating iptc and exif metadata to ensure these met hathitrust specifications, creating a yaml file to indicate the title page for each item, pulling marc records from the library catalog in marc xml format, and generating a checksum hash for each object. with speedwagon, we were able to efficiently generate these file packages. speedwagon was built using python 3, pyqt for generating its graphical user interface, google tesseract for generating ocr, exiv2 for examining embedded image metadata, and kakadu for creating jpeg2000 images. figure 16. user interface for speedwagon, a tool for processing packages of images post-digitization. after performing necessary quality control and post-processing on our digitized book file packages, we deposited our digitized files in our in-house-developed medusa preservation repository, where they received globally unique identifiers, regular backups, and integrity checks. from there, we ingested them into our companion digital collections platform, which incorporated them into structured digital objects. the digital collections platform leverages the open-source universalviewer [19] to provide convenient pagination through multi-page digital-objects and deep-zooming of individual pages. early conceptions of a delivery system for limited and restricted access to digital objects imagined a custom, purpose-built standalone application alongside our existing digital collections [20]. ultimately, we decided to build on the extensive infrastructure already in place in our digital collections platform related to the features mentioned previously. this minimized redundant development work and enabled staff to leverage their familiarity with our existing infrastructure. the first step in this direction was to modify the digital object entity in our digital library to add a boolean “restricted” flag. such objects are suppressed from browse, search, and harvesting results, and unavailable to unauthorized users in all other respects. next, we added an access control list of patron ids corresponding to access-expiration dates. after a book is ingested into the digital collections, it is set as restricted and the requesting patron is added to the allow list with an access-expiration date of 21 days in the future. when a patron id is added to an access list, the digital collections system sends an email to the corresponding email address containing the url of the item to which access has been authorized. attempting to dereference this url requires authentication via the campus single sign-on (sso) provider, which sends a patron id back to the digital collections platform and conditionally grants access depending on the access-expiration date. analysis we began gathering usage statistics on the emergency access service’s start date of july 20, 2021. we anticipate shutting down our emergency access protocols at some point in the summer of 2021, but for the purpose of bringing our paper to publication in a timely manner we present our data with a cutoff date of april 20. taken in all, during this nine month period we received 3,083 patron-requested items in preservation services. of these, based most importantly on whether the patron expressed a preference for print or digital: we sent 2,100 items to lockers for in-person pickup in the library (68% of all requests). we digitized 803 items for online access (26% of all requests). 39 items were in the public domain; these were digitized in our in-house internet archive scanning center and made freely available in the internet archive and then deposited in hathitrust. 764 items were still protected by copyright; these were digitized in-house by  digitization services and made available to requesting patrons in our own local digital collections platform, with eventual limited access in hathitrust emergency temporary access system (etas). these items total 191,372 pages digitized on-site. we withdrew 180 items from the digitization queue due to their prior availability in electronic format or removed them due to other errors in workflow routing (6% of all requests). figure 17. pie chart showing outcomes from nine months of emergency access services. table 1. emergency access production statistics by the month. month (“july” represents july 20-august 20) total books routed to preservation books sent to lockers books removed from digitization queue (already available in electronic format elsewhere) books digitized by internet archive (out of copyright) books digitized in-house (in copyright) total books digitized july 34 23 1 5 5 10 august 598 519 14 5 60 65 september 520 405 21 4 90 94 october 417 300 22 4 91 95 november 308 176 25 3 104 107 december 131 51 15 2 63 65 january 424 277 26 8 112 120 february 351 204 27 4 116 120 march 300 145 29 4 122 126 total 3,083 2,100 180 39 764 803 it is interesting to note that in the first three months (july-september) we found ourselves digitizing about 15% of all requested items. as we transitioned to the second half of the 2020 fall semester we expected to see a rising demand for digital access in response to the university administration’s request that students leave campus november 20 and not return for the remainder of the fall semester [21]. for this reason, we added new digitization staff and equipment to the project. indeed, from october to december we digitized 31% of all items to keep up with patron demand for remote access in digital format (of note all requesting features were shut off completely for a two week period in december for winter break, which explains the dip in production volume during that time). from january to march, we digitized 34% of all requested items to meet demand. figure 19. number of books digitized per month. over the entire nine-month period, the numbers average out to 26% of all requests evincing a preference for digital access, 68% specifying a preference for physical access in lockers, and 6% of items being located as ebooks already available elsewhere. there’s a bit of ambiguity in these numbers since not all patrons made it known which format they prefer, and in these cases we may have digitized items or sent them for locker pick-up based on our scanning capacity. however, since our process focuses on making sure to digitize at least everything people request for online access, to the best of our knowledge these numbers seem to reflect patron demand. figure 20. patron preference for locker pickup vs online access. given the circumstances of our largely remote teaching and learning environment, we were also curious to know if those taking advantage of the emergency access digitization service were doing so from far-flung locations. as we have google analytics running on our digital collections platform, we were able to take a list of all urls sent to library users of emergency access digitization and use the google analytics api to return a dereferenced list of their geographic locations. this data set is ambiguous due to the fact that some users may have been using campus-provided vpn technology in order to appear “on campus” and thus better able to access all university online services, and others may have been using vpn simply to obscure their location. also, google analytics may not track users who have an anti-tracking browser plugin installed. with these caveats in mind, the web analytics data seems to suggest that the majority of users who utilized this service were local to illinois (91%, with 75% in champaign-urbana), but that a meaningful (9%) portion of them from out of state, not to mention 1% out of the country, benefitted from remote access during the pandemic. figure 21. online access by patron ip address location. challenges, lessons learned, and next steps we were able to implement a robust emergency access digitization service within a month’s time due to several factors. most importantly: we have asserted control over our information technology infrastructure for many years. as a library, we have resisted the trend prevalent in many peer institutions of outsourcing information technology infrastructure to external vendors or to campus-level it units. (this is not to say we have not embraced technical change; we have become campus leaders, for instance, in implementing cloud services in favor of local server administration). in the case of implementing emergency access digitization, this allowed us to get to work straightaway on a needed solution without having to labor through the red tape and inflexibility of external partners. we have deliberately built flexible tools and services for managing digital content and production workflows. over the years, we have designed our repository architecture and workflow tools with adaptability in mind, so that with minor changes they can accommodate new needs as they arise. from a design perspective, this has meant giving simplicity right of way over complexity whenever possible for the sake of future adaptability to multiple uses. in many cases, this has meant eschewing popular open source repository platforms over the years after finding their design intractably complicated and instead choosing to build our own. we were willing to take on modified roles and challenging circumstances during the pandemic. throughout the library, multiple people took on modified work roles in order to implement this new service. this included two temporary library staff transfers to the preservation services unit to serve as digital imaging specialists. notably, this project pushed many people who had worked in a back-of-the-house technical services role into closer front-line contact with patrons than ever before, and required close collaboration with facilities and it to adapt work processes both on premise and remote. flexibility and cooperation were key in building a new team to implement the workflow, both for training new staff with little photography experience and adapting quickly. implementing emergency access digitization was certainly not without challenges. principally among these, we would note: disentangling individual digitization requests from needs related to course reserves and reading lists. certain instructors sought to use this process for course reserves. requests for digitization of portions of print materials were routed through the interlibrary loan and document delivery webpage [22]. the digitization of entire works for the purpose of course reserves was not recommended due to copyright restrictions; however, in preservation services, we were generally able to digitize full in-copyright works for limited access in hathitrust etas if we had enough advance notice. since these items would be accessed by multiple people, our goal was to get them all into the hathitrust etas in a timely fashion. communicating with patrons displeased with the library’s “digital first” approach.    some patrons preferred physical copies of books. while many patrons expressed their appreciation for gaining online access to library materials, there were others who vehemently complained about the library’s pandemic policy that gave preference to digital.    informal user feedback gathering. a method to aggregate feedback from our users would have been helpful from the start. in retrospect, it would have been wise to build a user feedback option into our interfaces in order to better understand how library users experienced our “digital-first” strategy during the pandemic. as it stands, we have received a variety of positive and negative feedback, but this has occurred in too scattershot and adhoc a manner to be useful. the pandemic made everything slower and more complicated than usual. while we were able to implement this service in short order, we had to work through significant slowdowns when ordering and shipping new equipment and furniture or fixing broken equipment. we also had to reorganize space to comply with safety precautions due to covid-19 and fit in two new equipment stations to support this work. taken in all, this paper gives an overview of emergency access digitization services as they existed from july 20, 2020 to april 20, 2021. looking ahead, we intend to turn off the hathitrust etas on june 13, 2021 and transition to some form of controlled digital lending for those still unable to come to campus and access items in print format. acknowledgments this was an all-hands-on-deck operation involving multiple people at the uiuc library. colleagues in our many departmental libraries adapted to alma and routing materials to preservation following this new procedure. william schlaack, with shelby strommer advising on collections care, came into the library while many people were working remotely and took the lead on triaging materials once they arrived in preservation services. rachael johns led digitization efforts in digitization services, aided from december on by tabby garbutt and christine willson. rachael’s work included overseeing daily operations and training two staff members from different areas of the library who worked temporarily in the preservation services unit: johna von behrens and diane griswell. they mastered digitization work using complicated equipment and processes, and did excellent work as digitization technicians. julie bumpus and preston wright of collection management services took on doing an additional review of whether items were available in digital format prior to digitization. tricia lampron in acquisitions and cataloging services (acs) ingested digitized books and metadata into our local access platform. digitization services graduate assistant brynlee emery performed quality review of materials and deposit with hathitrust. brian clark and later greta heng from acs updated our catalog with links to items once they become available in the hathitrust etas. from the scholarly communication and repository services group, alex dolski worked to modify our local digital collections platform so we could deliver items to one patron at a time in the restrictive fashion this project required. colleen fallaw fixed problems as they arose with our back-end medusa system, and seth robbins coordinated these efforts. in preservation services, henry borchers wrote scripts to allow us to retrieve marc metadata in xml format from alma for the proper packaging of our hathitrust submissions. in it, jon gorman helped henry understand the technical workings of alma to make this happen, while eric mosher, jason strutz, anthony stewart, rhonda jurinak, jim dohle, and lee galaway all helped make sure everything was properly in place with the many complex hardware, software, network, and storage components in play. staff in circulation have been in constant communication with staff in preservation to make sure patron needs are fully met, and janelle sanders helped those of us not familiar with alma to understand its intricacies. norris purdy, dillon brown, and brian agnoletti in shipping and receiving helped coordinate the routing of materials and followed up on special requests. timothy newman and lesli lundquist from facilities aided in coordinating new furniture and room arrangements. mary laskowski, cherie weible, chris prom, and jennifer teper did a fair amount of wrangling behind the scenes to put our high-level approach into place. taken in all, this project was a massive effort that involved committed collaboration from all areas of the library, and we salute everyone involved. about the authors kyle r. rimkus kyle rimkus is preservation librarian with rank of associate professor at the university of illinois at urbana-champaign library, where he leads efforts to establish policies, technologies, and practices to ensure that digital collections will persist into the future. orcid: https://orcid.org/0000-0002-9142-6677 alex dolski alex dolski is a research programmer at the university of illinois at urbana-champaign library, where he serves as lead developer on the digital collections platform and related software services. brynlee emery brynlee emery is the former graduate assistant for digitization services at the university of illinois at urbana-champaign, where she managed quality control and ingest workflows for digitized items. she now works in corporate digital asset management in salt lake city. rachael johns rachael johns works as a digital imaging specialist ii at the university of illinois at urbana-champaign, digitizing library materials and supporting the use and maintenance of digitization and photography tools. patricia lampron patricia lampron is a metadata services specialist at the university of illinois at urbana-champaign, specializing in organizing metadata for digital collections and making these collections accessible through the digital collections platform. william schlaack william schlaack is the digital reformatting coordinator at the university of illinois at urbana-champaign library. in this role he oversees general collections and vendor-based reformatting efforts. angela waarala angela waarala is the digital collections project manager at the university of illinois at urbana-champaign, where she oversees the digitization of on-demand requests and special collections. end notes [1] university of illinois office of the president. covid-19 policies. university of illinois massmail. 2020 mar 11 [accessed 2021 may 4]. https://massmail.illinois.edu/massmail/28768.html [2] university of illinois office of the chancellor. illinois massmail. 2020 mar 20 [accessed 2021 may 4]. https://massmail.illinois.edu/massmail/1741338.html [3] wilkin jp. covid-19: libraries closed to users. 2020 mar 21 [accessed 2021 may 4]. (libnews email list). https://wordpress.library.illinois.edu/staff/wp-content/uploads/sites/24/2020/03/covid-19_-libraries-closed-to-users-march-21.pdf [4] university of illinois office of the chancellor. illinois massmail. 2020 apr 2 [accessed 2021 may 4]. https://massmail.illinois.edu/massmail/9999819.html [5] prom c. hathitrust emergency temporary access. 2020 apr 1 [accessed 2021 may 4]. (libnews email list). https://wordpress.library.illinois.edu/staff/wp-content/uploads/sites/24/2020/04/hathitrust-emergency-temporary-access.pdf [6] university of illinois office of the chancellor. illinois massmail. 2020 jun 2 [accessed 2021 may 4]. https://massmail.illinois.edu/massmail/8421413.html [7] see university of illinois at urbana-champaign. covid-19 briefing series: shield – target, test, tell – covid-19. 2020 [accessed 2021 may 4]. https://covid19.illinois.edu/updates/covid-19-briefing-series-shield-target-test-tell/. to note, the university introduced a saliva-based covid-19 testing protocol for the university community, and required regular testing (once or twice a week, depending on infection rates in the university community) for affiliates seeking entry into campus buildings. [8] murphy h. back to on-site work weekly update. 2020 aug 11 [accessed 2021 may 4]. (libnews email list). https://wordpress.library.illinois.edu/staff/wp-content/uploads/sites/24/2020/08/back-to-on-site-work_-weekly-update-4.pdf [9] hennessey cl. customizing alma and primo for home & locker delivery. the code4lib journal. 2021 [accessed 2021 mar 1];(50). https://journal.code4lib.org/articles/15521 [10] hansen dr, courtney kk. a white paper on controlled digital lending of library books. lawarxiv; 2018 [accessed 2021 apr 30]. https://osf.io/7fdyr. doi:10.31228/osf.io/7fdyr [11] for more information on the library’s transition from alma to voyager, some resources are available here: alma training – staff website – u of i library. [accessed 2021 may 5]. https://www.library.illinois.edu/staff/alma/ [12] ex libris. add link to hathitrust. ex libris developer network. [accessed 2021 may 26]. https://developers.exlibrisgroup.com/appcenter/link-to-hathitrust/. note: we modified the code for this add-on to include both public domain as well as in-copyright items. [13] for an example, see https://digital.library.illinois.edu/collections/6ff64b00-072d-0130-c5bb-0019b9e633c5-2 [14] dt heritage. dt bc100 high resolution book scanning system. [accessed 2021 may 5]. https://heritage-digitaltransitions.com/dt-bc100/ [15] for more information on automating image capture using this software, view: peterson d. dt cropcontrol. dt photo. 2017 oct 24 [accessed 2021 may 5]. https://www.photo-digitaltransitions.com/dt-cropcontrol/ [16] dt heritage. dt atom entry level digitizer. [accessed 2021 may 5]. https://heritage-digitaltransitions.com/dt-atom-entry-level-digitizer/ [17] dt heritage. capture one ch imaging software for cultural heritage. [accessed 2021 may 5]. https://heritage-digitaltransitions.com/capture-one-ch/ [18] documentation available here: borchers h. speedwagon documentation. [accessed 2021 may 4]. https://www.library.illinois.edu/dccdocs/speedwagon/. source code available here: uiuclibrary/speedwagon. uiuc library; 2021 [accessed 2021 may 26]. https://github.com/uiuclibrary/speedwagon [19] universal viewer. [accessed 2021 may 4]. https://universalviewer.io/ [20] digital collections at the university of illinois at urbana-champaign library. [accessed 2021 may 4]. https://digital.library.illinois.edu/ [21] news bureau. university outlines fall plans including remote instruction after fall break. [accessed 2021 may 5]. https://news.illinois.edu/view/6367/1156975278 [22] university of illinois at urbana-champaign library. interlibrary loan and document delivery. [accessed 2021 may 5]. https://www.library.illinois.edu/interlibrary-loan/university-of-illinois-users/ subscribe to comments: for this article | for all articles one response to "adaptive digital library services: emergency access digitization at the university of illinois at urbana-champaign during the covid-19 pandemic" please leave a response below: hindi sex, 2025-07-16 https://smarthr.com.hk/companies/hindi-sex_ramona/ leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – manifold: a custom analytics platform to visualize research impact mission editorial committee process and structure code4lib issue 30, 2015-10-15 manifold: a custom analytics platform to visualize research impact the use of research impact metrics and analytics has become an integral component to many aspects of institutional assessment. many platforms currently exist to provide such analytics, both proprietary and open source; however, the functionality of these systems may not always overlap to serve uniquely specific needs. in this paper, i describe a novel web-based platform, named manifold, that i built to serve custom research impact assessment needs in the university of minnesota medical school. built on a standard lamp architecture, manifold automatically pulls publication data for faculty from scopus through apis, calculates impact metrics through automated analytics, and dynamically generates report-like profiles that visualize those metrics. work on this project has resulted in many lessons learned about challenges to sustainability and scalability in developing a system of such magnitude. by steven braun introduction assessment has become an essential and ubiquitous part of the operations of academic institutions. with increased competition for funding and the unspoken mandate to “publish or perish,” researchers and institutions are increasingly pressured to demonstrate the value and impact of their research scholarship against that of their peers. as the nature of scholarly communication continues to rapidly evolve, demonstrating such impact can come in many different forms, and as a result, both institutions and researchers alike have become burdened with the responsibility of managing all of the varied ways in which their value is represented and compared to that of others. in this environment, research impact metrics have become particularly popular and widely adopted across institutions. the appeal of these metrics is in their claim to precision and ease of calculation; for example, measures such as the h-index, which describes the minimum number of h citations to each of at least h publications authored by a researcher [1], as well as simpler measures like publication counts and total citation counts attempt to capture a snapshot of researchers’ careers in a singular point value. however, these metrics are built on assumptions that are often overlooked when used as a direct proxy for evidence of scholarly impact, resulting in comparisons between researchers and institutions that may be inaccurate. as a result, properly contextualizing these metrics continues to be a significant challenge to ensuring they are utilized responsibly. many platforms currently exist to provide research impact metrics in some form, and each attempts to meet the challenge of contextualization in its own way. some of these are proprietary extensions of online citation indexes, such as scopus, web of science, and google scholar; meanwhile, others are vendor-based or open source and adoptable into existing architectures, such as elsevier’s scival and pure platforms (formerly called experts), sciencv, and vivo. each of these attempts to solve the age-old problem of managing the numerous platforms across which researchers’ online scholarly identities may be distributed by providing a standardized and centralized architecture for integrating them all into one location. however, while they are all similar in this functionality, they differ slightly in their approaches, the kinds of scholarship metrics they provide, and the data sources used to calculate those metrics. as a result, it can still be difficult to identify a single platform that may serve all needs around institutional assessment at once. this problem was encountered at the university of minnesota in early 2014 when new needs in data-driven assessment arrived at the doorsteps of the university libraries. newly appointed in february 2014, the dean of the medical school was charged with the task of revitalizing faculty scholarship to bolster the medical school’s international reputation in education and research, attract and retain world-class researchers, and raise the institution’s national rankings. in pursuit of these objectives, the dean sought to programmatically recognize and encourage faculty research excellence through metrics of scholarly output and impact, based particularly on data about their peer-reviewed journal publications. the size and complexity of the medical school, which employs approximately 2,200 faculty, made this request largely unprecedented in its scope; however, conversations at the highest levels of medical school and university libraries administration suggested that the libraries might have the resources to provide the data needed. in its earliest iterations, specific medical school data requests centered around a handful of key data. delivered in regular reports for each medical school faculty member, these included 1. a full list of their peer-reviewed publications, 2. their h-index, and 3. metrics calculated about publications on which they were listed as first or last author. while the first two components were relatively straightforward in their acquisition, the third request for metrics on first or last authorships posed a new challenge. based on the assumption that having first or last author position indicates greatest contribution to a scholarly work, a newly defined metric named the h(fl)-index sought to capture the frequency and reach of publications substantially authored by faculty. a permutation of the h-index, the h(fl)-index is calculated in precisely the same way as the h-index but with a slightly different set of data: instead of rank-ordering all publications by a researcher, only the subset of first or last author publications by them is ranked by descending citation count and used to identify the highest threshold at which the citation count of a paper is equal to or greater than its rank. this metric, which was defined by the medical school, is not recorded in existing scholarly literature and as a result required specialized calculation. as with the h-index, determining the h(fl)-index of a researcher is feasible with complete data about their publication history, from which first or last authorships can be retrieved. since this prerequisite was already a component of the original data request, responding to the request depended upon a canonical source of publication data for medical school faculty. for several years, the university libraries have managed a local instance of elsevier’s experts [2] platform (named experts@minnesota), which produces profiles for researchers that include lists of publications and other scholarly works, calculations of h-indices, and data about collaboration patterns with other researchers. publication data are derived principally from scopus, which is well-representative of the physical, life, and health sciences, disciplines in which medical school faculty actively publish. since the majority of medical school full faculty have profiles in experts@minnesota, it seemed that a canonical source of data was found. however, as specifications around the requested data continued to be refined, further investigation revealed several limitations in using data from experts@minnesota alone. the first of these was the discovery of incompleteness in the medical school faculty list included in experts. in its entirety, the medical school faculty population is split roughly in half into two classes. one half includes those with full faculty status, those who functionally practice or work in the medical school campus proper. the other half, however, includes affiliate faculty, those with primary appointments in the medical school but who may functionally work at a satellite institution, such as the veterans affairs medical center. while most full-status faculty were included in experts, affiliate faculty were largely excluded, meaning that there was no centralized source for retrieving their publication data. this incompleteness meant that specialized faculty metrics, including the h(fl)-index and counts of first/last author publications, could not be derived exclusively from experts@minnesota data. an additional oddity was discovered in the h-index values provided by experts@minnesota; rather than being derived directly from h-index values provided on scopus author profiles, experts@minnesota calculations were based only on the limited subset of instances of experts at other academic institutions, making them unreliable. these limitations had significant implications for how the data the libraries endeavored to provide would be used: as these data were to be considered in performance review, promotion, and tenure processes, it was imperative that their reliability be prioritized. additionally, the data reports to be regularly provided were to be used for recognizing top-achieving faculty and tracking faculty publication activity, meaning stakes were high for faculty and the libraries alike. in recognition of these limitations, i developed a novel system that could be custom-built to the highly specific needs of the medical school. named manifold, this system dynamically generates data-driven profiles of scholarly output and impact metrics for all medical school faculty. these profiles include lists of publications, faculty metrics (including custom metrics), as well as visualizations of those metrics, with the aim of providing better context for their interpretation and use. publication data are principally drawn from scopus via the scopus api, and all analytics processes are carried out following a regular schedule to update all data stored internally. in this paper, i discuss the technical specifications of the system, challenges encountered in its development, and future directions in which development might continue. while the focus here is on the technological dimensions of this project from a systems perspective, a discussion of manifold from a services perspective will be provided elsewhere (braun, manuscript in preparation). manifold: technical specifications overview manifold is built on a standard lamp (linux/apache/mysql/php) framework, making it relatively easy to deploy on most architectures. all processes at both the front and back end are written in php, and data are stored in a mysql relational database. this distribution of functionality is based on a model-view-controller (mvc) design that seeks to make manifold lightweight and extensible. the full source code can be found on github at https://www.github.com/braunsg/manifold-impact-analytics. the creation of manifold was spurred principally by two broad needs specific to the medical school. in one capacity, manifold serves as the analytics platform needed to calculate research impact metrics and aggregate data about faculty publications for ongoing review purposes. in another capacity, manifold helps identify potential nominees for the medical school wall of scholarship, a physical installation on the university of minnesota – twin cities campus that recognizes and honors medical school faculty who have published first or last author papers that have received at least 1,000 citations in two out of three citation indexes (scopus, web of science, and google scholar) [3]. as a result, the manifold system workflow is built primarily with the unique needs of the medical school in mind. however, manifold is also built to maximize sustainability, especially in recognition that manifold data will continue to be used for the indefinite future. the manifold workflow thus attempts to ensure long-term sustainability by automating all data processes to the fullest extent possible, as described in brief below. 1. bibliographic data about medical school faculty, including important parameters such as tenure status, percent time, and job class, are ingested into the manifold database. on a regular basis, these data are updated for current faculty as well as instantiated for faculty who are newly employed at the university of minnesota. 2. using scopus author identifiers (henceforth “scopus ids”), manifold processes query the scopus api to pull all publications authored by each faculty member and store them in the database. these publications are subsequently updated at each new data pull to grab new bibliometric data, including citation counts. 3. research impact and productivity metrics are calculated for each faculty member and stored in the database. 4. data are output in profiles that are dynamically generated for individual faculty as well as whole departments and custom-defined subsets of faculty. figure 1 provides a schematic representation of this workflow. users only interact with the outputs of the manifold interface itself, i.e., manifold profiles. figure 1: schematic representation of manifold workflow. profile types and searches although manifold stores data for approximately 2,200 medical school faculty, profiles for only about half of these — those with full faculty status and who work full time (f.t.e. greater than or equal to 0.67) — are outwardly displayed and thus searchable. (data for the remaining faculty are stored for other internal review and assessment purposes.) profiles may be searched for and generated in three views: individual faculty, departments, and subsets of faculty across different departments or job classes that may be custom-defined by users. figure 2 provides a screenshot of an example search for an individual faculty member on the manifold search page. upon selecting a faculty member to profile, their metrics are previewed for quick reference. users may subsequently load their full profile for complete data. meanwhile, figure 3 shows an example of a custom search, based on selecting a subset of tenured faculty in a given department. figure 2: screenshot of an example search. figure 3: screenshot of an example custom search. profile components profiles consist of several components that are dynamically generated based on the entity profiled, each of which is described below. metrics each profile features an “overview” module that dynamically displays the following metrics for the profiled entity: 1. h-index, calculated internally based on the publications stored in the manifold database 2. h(fl)-index, a calculation of the h-index based only on first or last authorships, also calculated internally 3. total publication count, which may include both scopus publications (automatically harvested via api processes) as well as records manually imported by users from pubmed (discussed below) 4. first/last author publication count, which may also include both scopus and pubmed records 5. total sum of citations to all publications authored by the entity, based exclusively on scopus citation counts 6. total sum of citations to all first or last author publications, also based on scopus citation counts on individual faculty profiles, these metrics are displayed in conjunction with boxplots that depict the distribution of each respective metric for their home department (minimum, median, maximum, and interquartile ranges) as well as the faculty member’s relative position in that distribution. for departmental and custom profiles, displayed metrics represent median values across all profiled faculty affiliated with the department only, as mean values can be highly skewed by exceptionally prolific and highly cited faculty. figure 4 provides an example screenshot of these metrics for an individual faculty member. figure 4: an example screenshot of a faculty member’s metrics. publication lists each profile includes lists of publications authored by the entity in two forms, each of which is shown in the example profile illustrated in figure 4. the first of these lists the top ten most highly cited publications for the entity; the other is a year-delimited list of all publications by the entity, including both scopus and imported pubmed publications. users may change the year range of publications displayed, and each manifold record links back to the original scopus or pubmed record. for individual faculty, these lists include publications authored by them alone; for departments or custom-defined subsets of faculty, these include publications authored by any faculty affiliated in the defined population. all publication data may be exported to a spreadsheet that can be downloaded by users via the click of a button. individual profile publication lists additionally provide users with the option to output the listed publications in a citation format required for university of minnesota medical school curriculum vitae. users may copy this output and paste it directly into other documents. as faculty are being asked to provide metrics and publication data from manifold in their dossiers for review, manifold seeks to streamline and simplify this process as much as possible. faculty summaries department and custom profiles include a table providing summary data for all faculty affiliated with the entity (e.g., all faculty in a single department). this table includes impact and productivity metrics for all faculty and may be exported to a spreadsheet for further analyses. visualizations manifold is unique from existing systems utilized at the university of minnesota in its use of visualizations to graphically represent faculty metrics. these visualizations are generated dynamically using d3.js and seek to provide better context for manifold data that are used for review and assessment purposes. these include distributions of scopus publications by citation rank order (figure 5) and year of authorship (figure 6), with color distinctions for first/last authorships and links back to their respective original records in scopus; a chart that plots changes in h– and h(fl)-index over time for a faculty member (figure 7); and comparative distributions of metrics across departments (figure 8) and the medical school as a whole (figure 9). figure 5: distribution of scopus publications by citation rank order. figure 6: distribution of first/last authorships. figure 7: changes in in h– and h(fl)-index over time for a faculty member. figure 8: comparative distributions of metrics across departments. figure 9: comparative distributions of metrics across the medical school. data the majority of publication data stored in manifold are retrieved through the scopus api, using a registered api key that is credentialed by the university of minnesota’s licensing with scopus. additional data are stored as the result of manual import from pubmed. publications the university of minnesota manifold database currently stores publication data for approximately 2,200 faculty in the medical school. publications authored by this active faculty pool total over 68,000, which is composed most significantly of publication data harvested directly via the scopus api. since scopus may not index the entirety of a given researcher’s publication history, users also have the option to search for and import missing records into their manifold profile directly from pubmed, either by title or pubmed id. this import module is built directly into manifold profiles, and the process is authenticated via verification codes that are sent to users’ university e-mail accounts. scopus author ids faculty publication data are pulled by querying the scopus api with each faculty member’s scopus id; consequently, those ids must be known in advance. for most faculty in manifold, these ids are derived from experts@minnesota data, where they exist, using custom scripts that parse the data matched against medical school faculty. for others without experts@minnesota data, scopus author ids are harvested via a mixture of manual searching in scopus, automated scripts that attempt to query the scopus api based on name and affiliation, and discoveries by human users of manifold who report incorrect publication attributions or missing publications on their profiles. custom reports beyond being used to generate faculty profiles, manifold data have also been used to create custom reports for specific assessment purposes. for example, one such report used at the university of minnesota is a departmental breakdown of publication counts over time and measures that compare those counts to previous years. similar reports may be developed for other specific analytics as requested. database structure all manifold data are stored in a relational database. collectively, this database distributes faculty bibliographic data (e.g., name, tenure status, and percent time), faculty metrics, harvested publication records, and links between publications and faculty authorships in records across different tables, each of which holds one of these specific varieties of data.” they also include mappings that transform field codes used in the university central human resources query system, called the data warehouse, to internal manifold field names for interoperability. descriptions of these tables are provided below. table 1: database structure of manifold. table name table description affiliation_data data about (medical school) departments represented in manifold, including department codes used in the central university human resources system contact_messages data stored about messages sent through the manifold contact form, for tracking purposes data_sources index of data sources represented in manifold (i.e., scopus, pubmed) dw_field_mappings maps data warehouse query field names to corresponding field names in the manifold database events_master data on all processes executed (e.g., publication data update, calculation of metrics) faculty_affiliations departmental affiliations of medical school faculty represented in manifold, two per faculty: one is the primary departmental affiliation according to human resources, and the other is their functional affiliation in which they are considered active faculty (which may differ from human resources) faculty_data bibliographic data on all faculty represented in manifold, including name, sex, title, tenure status, job code, job class, percent time, and faculty status faculty_identifiers various identifiers for faculty (e.g., scopus id), one record for each unique id assigned faculty_metrics research metrics for all faculty, one record/row per unique faculty, including all metrics displayed in the overview module on profiles faculty_publications collation of faculty and their authorships; each record links a faculty member with a specific publication and stores data about their position in the author list for first/last author metrics publication_data all publication data, one record per unique publication, including all bibliometric data retrievable from scopus or pubmed reports data and parameters for loading custom reports displayed in manifold profiles temp_submissions temporary data stored from publications imported from pubmed, before authorized confirmation is made by a faculty user or their proxy visualizations data and parameters for loading metrics visualizations displayed in manifold profiles records in the manifold database are linked by shared keys between tables. these include identifiers taken directly from the data warehouse, such as faculty university internet ids and department codes, as well as internal proprietary manifold identifiers. no sensitive or private data about faculty or departments need to be stored as a result of these defined keys, enhancing the openness of the platform overall. challenges and roadblocks in building manifold the development of manifold in its current version occurred over several iterations spanning approximately 12 months from the summer of 2014 to the summer of 2015. throughout this course, several persistent challenges emerged at all levels of the manifold infrastructure, each with its own unique implications for long-term technical sustainability. here, i discuss a selection of these challenges and approaches to their solution. accurately defining a canonical faculty list of the many moving parts of manifold’s architecture, defining a list of faculty to be included in it would seem to be the most straightforward endeavor. however, much to our surprise, this task was the most persistent and resistant roadblock in manifold’s development. the reasons for this are varied, all related to conventions and idiosyncrasies in the university’s central human resources system that blur the lines between faculty and non-faculty status. as previously indicated, university of minnesota medical school faculty fall into two classes, one for full faculty and the other for affiliates. while full faculty status is defined in a relatively straightforward manner in university human resources, affiliate status is much more complicated, contingent upon many complex variables — including department of primary appointment, line of pay, and location of work. these variables may change rapidly, even over the course of a single pay period, and as a result, a persistent canonical faculty list that is accurate in both its qualifications for inclusion as well as its representation of faculty can be elusive. an illustration of this difficulty can be found in problems we encountered with the indeterminacy of faculty departmental affiliations. faculty may have multiple appointments and titles in several departments in the medical school; some of these may be formalities, related to responsibilities beyond faculty status (such as administrative positions like departmental chairs), and others may be related to the department through which a faculty member’s salary is paid. due to the transient nature of grant funding, these appointments may flip often, subsequently resulting in significant changes in faculty status. to complicate matters worse, these affiliations also may not always align properly with the department in which a faculty member may functionally work. since manifold is designed to run on automated processes, it has no built-in mechanism for detecting these misalignments if appointments are not recorded properly in the original data source (the data warehouse). in early faculty update processes, these inaccuracies were quickly revealed as a result of user feedback. several faculty reported being displayed in the incorrect department and thus felt inappropriately represented in manifold. however, these displayed affiliations were derived directly from human resources, meaning that reported inaccuracies needed to be corrected with human resources itself. given the impracticality of such a solution, manifold was designed to include a workaround allowing the database to hold two separate affiliation records for each faculty member: one record for their departmental affiliation formally defined by human resources, and one record for their department of functional appointment, to be displayed on manifold profiles and used for analytics purposes. additional problems emerged around even determining which medical school faculty should and should not be included in manifold. early attempts to specify rigid qualifications for faculty status illuminated many nuances in human resources data and sensitivities among faculty; some faculty who were excluded from display in manifold, perhaps due to their salary appointment, job title, or percent time, felt unjustly misrepresented, even though they objectively failed to meet the basic qualifications we defined. although making individual exceptions for such faculty was an option, it was not ideal for purposes of sustainability. to reduce the amount of manual curation needed to clean up faculty lists as a result of these issues, a standard workflow was established that maximizes interoperability with the data warehouse while retaining enough flexibility to also define functional departmental affiliations. at each time when the faculty roster requires an update, staff in the medical school payroll office query the data warehouse to produce two lists of faculty, one for full faculty and the other for affiliates. these lists are as rigorously defined as possible, minimizing overlap between them, and include in their consideration corrections for several known medical school idiosyncrasies that have been revealed due to past experiences. upon a faculty update, manifold ingests these lists through a customized series of scripts that update existing or instantiate new faculty records in the database by mapping data warehouse query fields to internal manifold fields. these scripts identify only the relevant fields to import and discard any extraneous data, flagging faculty manifold records with three fields that declare their status as currently active full faculty or affiliates. after this, additional scripts crosscheck manifold scopus author id records with those provided by experts@minnesota data. scopus data quality the quality of any data-driven platform is only as good as the quality of the data upon which it is built. in the case of manifold, this dependency falls principally upon scopus, and indeed, the development of manifold has resulted in a more intimate knowledge of the many different points at which scopus data quality may fail. the university of minnesota instance of manifold obtains most of its faculty scopus ids through data from experts@minnesota. however, these author ids may not always be accurate, resulting in errors that propagate forward into manifold. many of these errors are discovered by users of manifold, several of whom over the past year have reported that their manifold profile is linked to an author that is not them or lists publications that are misattributed to them. although scopus prides itself on its author disambiguation algorithms, in the majority of these cases, these problems were traced back to data provided in experts@minnesota, which occasionally assigns incorrect scopus author ids to faculty. in other cases, faculty have also reported publications entirely missing from their manifold profiles. upon deeper investigation, these absences have been frequently traced back to faculty publications being distributed across several different scopus author profiles and ids, requiring a faculty member to merge those profiles so changes in scopus can propagate forward into manifold. in yet other situations, scopus author profiles for faculty may be difficult to locate due to inaccuracies in scopus author affiliation records. errors such as these can only be identified and corrected via manual curation, and the manifold scopus author id base has iteratively improved as a result. additional oddities have been found at the level of individual publication records in scopus, particularly in the way that scopus handles articles in press. when scopus instantiates a new record for an article in press, that record is assigned a unique scopus electronic identifier which is subsequently captured and stored in manifold. over time, however, those in-press records may be overridden by new records for the full issue-assigned publication, causing the in-press records to disappear. in subsequent data updates, queries against the scopus api using in-press publication identifiers may thus return errors because the id has been rendered obsolete and invalid. when this in-press record remains in the manifold database while a new record for the fully assigned publication is also ingested, the result is a duplication of publication records for individual faculty which consequently affects the accuracy of metrics. manifold corrects for this problem by including a boolean flag in the faculty publications table that determines whether a given faculty publication authorship record in manifold is valid. at the beginning of each full publication update process, all of these flags are reset to zero (false), and as faculty are queried against the scopus api, only returned records that match against existing records in manifold are flagged as valid. after a data pull is complete, only valid faculty publication records are used in the calculation of metrics and displayed outwardly in manifold profiles. manifold data processes have thus been continuously refined over time to maximize automation and reduce the extent of manual cleanup required afterwards. working with the api like any api, the scopus api is proprietarily designed, and a full understanding of its construction, limitations, and fail points can only be developed with experience. since the foundational functionality of manifold is derived precisely from interaction with the scopus api, many challenges in working with it at many different levels have been encountered in the course of manifold’s development. one of the earliest challenges resulted from query limits imposed upon manifold’s registered api key. although the university libraries’ institutional contract with elsevier includes full access to scopus data for internal use, i encountered persistent blocks upon reaching thresholds around 20,000 query attempts. in the early stages of development, these blocks were a major impediment to the feasibility of the project as a whole; since query limits refreshed on a weekly basis, complete publication data pulls for all faculty — totaling almost 70,000 — would need to be distributed over several weeks. upon further conversation with representatives from scopus, these limits were temporarily removed. however, it is unclear whether such limits may be incurred once again in the future. additional problems stemmed from arbitrary server errors returned by the api itself. on occasion, queries to the api would arbitrarily fail for specific faculty or publication records due to a general-purpose http 500 error. these errors can only be resolved by halting the process for some time and then attempting the query again, which is impractical when updating a large database of records in sequence. to overcome this, catch mechanisms were implemented to pool failed queries for later retrieval or manual cleanup at the end of entire data processes. data load posed several challenges as well. since the university of minnesota instance of manifold currently stores nearly 70,000 publication records by 2,200 faculty, querying each faculty member and publication record in sequence through the api can take significant computing time when run in sequence. for example, in the most recent quarterly update carried out at the university of minnesota, the process for pulling all new publications for faculty completed in approximately 7 hours; meanwhile, the process to update all existing records completed in 11 hours. these estimates also assume that no errors were encountered along the way. in reality, however, procedural errors can emerge unexpectedly, and thus completing a full data update can span over several days. in the future, these updates may be parallelized across multiple api keys and processes to speed them up. prospects for further development in spite of the many challenges encountered along the way, manifold currently operates smoothly at the university of minnesota and faculty report fewer and fewer errors in the data it stores. this smoothness is owed to many months of iterative improvements, however, and continued experience working with manifold will reveal new challenges to be addressed. manifold is a work in progress, and there are a variety of directions in which it may be developed further to enhance its accuracy and reliability. some directions for improvements are currently being explored at the university of minnesota and may be implemented in new versions in the near future. as the use of manifold continues to increase in the medical school, interest in using the underlying data for more custom analytics purposes has continued to grow. one possible direction of growth thus may be the development of an api to allow external access to the underlying data in a controlled, secure connection. additionally, sources of scholarly works beyond scopus may be integrated into the manifold database, including book chapters, educational materials, and other research outputs that may not be captured by scopus. the resulting heterogeneity would place new constraints on manifold’s existing database architecture, requiring redesign to handle and represent highly variable modalities of research. manifold’s data processes may also be improved in several ways. as has been previously mentioned, publication data pull processes currently query the scopus api one-by-one, sequentially, resulting in a very long execution time. one possible improvement might be to distribute these processes over multiple simultaneous api connections, allowing for bulk data retrieval. also, more streamlined mechanisms could be implemented to handle the removal of incorrect data from manifold. for example, features could be implemented that allow users to manually remove misattributions from their publication lists directly in manifold, eliminating the need to contact the manifold developer to remove those data on the back end. the user interface of manifold may also be improved in several ways to enhance its user-friendliness. one such improvement may be the inclusion of additional custom user-defined metrics or other ways to visually represent those metrics, and in relation, features that allow users to export or print out those visualizations for other purposes. an additional change may involve providing users with the ability to report co-first authorship on their manifold publications, a capacity that has been increasingly requested by medical school faculty in connection to increased emphasis on first/last author metrics. by building in features to enhance flexibility, manifold can continue to grow and meet new needs as they emerge. conclusion manifold was custom-built to respond to the specific needs of the university of minnesota medical school. although manifold has been successful in doing so, the history of its development and challenges therein demonstrates that implementing a system of such magnitude requires careful consideration of the resources available to sustain it for the long term. as other academic institutions explore new platforms to meet their own analytics and assessment needs, it is hoped that manifold may serve as an example framework that may be adapted, extended, and improved upon. code the code for manifoldin its entirety is available open source on github at https://github.com/braunsg/manifold-impact-analytics and is protected under a gnu general public license (v2). acknowledgments i would like to acknowledge mary clay, director of special projects (university of minnesota medical school), medical school dean brooks jackson, and janice jaguszewski (director, health sciences libraries, university of minnesota) for their constant support throughout the course of manifold’s development. i would also like to acknowledge the efforts of university libraries web development, particularly michael berkowski and cody hanson, who have put forth tremendous effort to ensure that manifold can be maintained moving forward. manifold would not be successful without the support of all of these people in particular. notes [1] for more information about how the h-index is calculated, the reader may find the original citation useful: hirsch, j. e. (2005). an index to quantify an individual’s scientific research output. pnas 102 (46): 16569–16572. doi:10.1073/pnas.0507655102.(coins) [2] elsevier’s experts platform has recently migrated to their new pure platform; however, at the university of minnesota, the name of experts is still used for reference. [3] more information about the wall of scholarship can be found here: http://www.med.umn.edu/research/wall-scholarship about the author steven braun served as the informatics/data services specialist in the health sciences libraries at the university of minnesota. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – persistent identifiers for heritage objects mission editorial committee process and structure code4lib issue 47, 2020-02-17 persistent identifiers for heritage objects persistent identifiers (pid’s) are essential for getting access and referring to library, archive and museum (lam) collection objects in a sustainable and unambiguous way, both internally and externally. heritage institutions need a universal policy for the use of pid’s in order to have an efficient digital infrastructure at their disposal and to achieve optimal interoperability, leading to open data, open collections and efficient resource management. here the discussion is limited to pid’s that institutions can assign to objects they own or administer themselves. pid’s for people, subjects etc. can be used by heritage institutions, but are generally managed by other parties. the first part of this article consists of a general theoretical description of persistent identifiers. first of all, i discuss the questions of what persistent identifiers are and what they are not, and what is needed to administer and use them. the most commonly used existing pid systems are briefly characterized. then i discuss the types of objects pid’s can be assigned to. this section concludes with an overview of the requirements that apply if pids should also be used for linked data. the second part examines current infrastructural practices, and existing pid systems and their advantages and shortcomings. based on these practical issues and the pros and cons of existing pid systems a list of requirements for pid systems is presented which is used to address a number of practical considerations. this section concludes with a number of recommendations. by lukas koster persistent identifiers: the basics what is a persistent identifier? there are a number of definitions of persistent identifiers available that differ in two ways: the nature of the objects to be identified, and the extent of persistence. here, a persistent identifier is understood as a “unique universal persistent identifier”. an “identifier” is a string used to refer to an object. “unique” means that the identifier is only used for one object. “universal” entails that the identifier is valid for the whole of the world or world wide web. “persistent” means that the identifier remains available independently of individual institutions, systems or system implementations. in actual practice a pid has two functions: uniquely identifying an object making available an object and/or information about that object on the web. the process of generating pid’s is also known as “minting”. physical and digital objects? some definitions state that pid’s always refer to digital objects, such as the one in the digital preservation handbook by the digital preservation coalition: “a persistent identifier is a long-lasting reference to a digital resource.” [1] (also used by orcid [2]) and the dutch pidwijzer (pid guide) website: “a persistent identifier (pid) is a unique identification code attached to a digital object and registered at an agreed location.” [3]. however, a pid can also refer to physical objects and abstract concepts. english wikipedia [4] (contrary to dutch wikipedia [5]) states: “a persistent identifier (pi or pid) is a long-lasting reference to a document, file, web page, or other object. the term “persistent identifier” is usually used in the context of digital objects that are accessible over the internet”. doi’s (“digital object identifier”), a well known and much used pid system, can be used for “any entity — physical, digital or abstract” [6]. a doi is a “digital identifier of an object”, not an “identifier of a digital object” [7]. example: a pid can be assigned to a physical museum object like a vase (a unique item). a digital representation of the object can also have a pid assigned, but that pid does not refer to the original physical object. the physical object’s pid does not identify it’s digital representation, although usually that representation will be shown when the physical object’s pid resolves to a web page describing the physical object, including, among others, location metadata. universality a persistent identifier’s universality implies that the identifier is unique within the known universe, or the world. for practical reasons this means that the identifier is unique within a specific context. because the uniqueness of identifiers without context cannot be guaranteed, pid’s are assigned within specific “web namespaces” administered by an institution, such as uva.nl, doi.org, handle.net. this is called the pid-uri. a single string can occur more than once, but only once within a reserved namespace, which makes that string unique. in case several institutions have the right to assign identifiers within a specific namespace, then these institutions are assigned a code of their own that is also part of the total identifier string. the actual single code is unique for the institutional code within the namespace. the institutional code is often called a “prefix”, the actual code a “suffix”. this approach makes it possible for existing identifiers, for instance internal system identifiers or other pid’s, to be used as suffix. persistence “persistent” as a rule is defined as “permanent”, “everlasting”, “eternal”. in reality this is not possible. a pid only continues to be unique and usable as long as somebody assures that to be the case. usually it is an institution that administers identifiers and guarantees their uniqueness within a unique namespace on the web. for example: doi 10.1109/ccgrid.2018.00098 is unique within the doi.org namespace, and the referenced object can be found via the pid-uri https://doi.org/10.1109/ccgrid.2018.00098. if the international doi foundation would cease to continue the doi.org domain name/namespace, the object in question could no longer be found via the identifier. the actual identifier string 10.1109/ccgrid.2018.00098 would by itself no longer be guaranteed to be unique on the web, although it could still be. as mentioned before, english wikipedia defines a pid as “a long-lasting reference to a document, file, web page, or other object”, where “long-lasting” is a relative concept. other mentioned definitions have taken over the term “long-lasting”. alternatively, instead of “persistent”, the term “persistable” is used in order to indicate that an effort is needed to keep identifiers unique and usable. resolution persistence not only refers to the continued existence of the identifier itself, but also to the continued findability and accessibility (usability) of the identified object on the web. a location independent pid on the web also performs the function of resolution, the translation to the actual location on the web, the uniform resource locator (url [8]), where the object in question is described and findable. the aforementioned doi for instance refers to https://ieeexplore.ieee.org/document/8411085/. the technique of handling this resolution is called “redirection”. whenever an object is described on a different web address (url) than before, the pid-uri will have to redirect to the new location. this always requires administration activity. figure 1. redirection adjustment the standard method of resolving pid’s is to employ a mapping table, in which for every pid-uri a target url is recorded. alternatively “rule based mapping” can be used, where base url and the target url’s syntax are dynamically generated by applying a template, pattern and/or a regular expression and adding the distinctive identifier. this is only possible if the old identifier is available in the new system or if a mapping table of old to new system identifiers is available, and if this identifier can be used in a query. administration it should be clear that both persistence and resolution are only guaranteed by virtue of permanent administration. somebody (usually an institution) has to guarantee the continued existence of the identifiers and the maintenance of up to date resolutions to the actual locations of the objects in question. occasionally this can cause the implementation of major adjustments, for instance in the case of a system migration, where base url and system identifiers of all records are changed. permalinks “permalinks” are often mistakenly regarded as persistent identifiers. generally this term is used for the direct url’s pointing to metadata records in a specific installation of a specific system. a permalink is always a full url containing an internal system identifier generated by the system itself. the internal system identifier is only valid and unique within a specific system installation. after migration to a different system the old “permalink” will no longer work, unless redirection to the url’s in the new system is possible. for this some mapping between old and new system identifiers is required. types of pid systems an institution can choose to develop and administer its own pid system, or use a ready made, externally provided and administered system. the difference is that external pid systems are maintained and administered by a third party (usually for a fee), and that the third party guarantees the persistence and uniqueness of the identifiers. the correct redirection remains the responsibility of the institution itself, for instance in the case of system migration. local administration will still be necessary. pid systems have been around since the mid 1990’s. in everyday practice a limited number of external pid systems are used: nbn, handle, doi, ark, purl. see for instance hilse and kothe [2006]. the pid systems described below are characterized by type (pid/pid-uri/url), responsibility for minting, assigning, resolving and server administration, and rule based mapping options. the level of responsibilities is indicated. “local” means that the institutions are themselves fully responsible. “central” indicates the dependency of a global entity. in some cases there is a certain distribution of responsibilities between “central” and “local”. nbn type: pid syntax: urn:nbn:[country code]:[institutional code]-[pid-suffix] organisation registry: national (national libraries) minting: local assigning: local persistence: national resolving: national: pid-uri = [national resolver url]/[nbn] server admin: national rule based mapping: no nbn’s (national bibliographic number) are administered by a number of national libraries [9], such as the german national library, the royal library of sweden and the royal library of the netherlands. resolving of nbn’s is handled on the national level. nbn makes use of the urn (uniform resource name) mechanism [10]. an nbn consists of the protocol string “urn:nbn:” followed by an iso two character country code, an institutional code prefix and an “assigned nbn string”, the actual identifier string or suffix. existing identifiers can be used as assigned nbn strings. example: urn:nbn:nl:ui:29-8f66e0a8-b7c9-40a4-be28-54a7c01770, which identifies an article published in the repository of the university of amsterdam. in this case the suffix 8f66e0a8-b7c9-40a4-be28-54a7c01770 is identical to an internal system identifier (uuid) of the repository system. nbn policy differs per country. in the netherlands affiliated institutions can assign nbn’s themselves if they comply with certain conditions referring to the repositories that digital objects are stored in [11]. the repository system for instance has to be a “long term preservation depot” approved by the royal library registration agency. this means that in the netherlands nbn’s can only be assigned to digital objects. because a urn is not a url, redirection of nbn’s has to go through a special resolving server. in the netherlands this is taken care of by the dans [12] administered server at http://persistent-identifier.nl/. the example: https://www.persistent-identifier.nl/urn:nbn:nl:ui:29-8f66e0a8-b7c9-40a4-be28-54a7c0177061. for germany and switzerland that is https://nbn-resolving.org. in the netherlands nbn is a free service. handle system type: pid syntax: [naming authority code].[sublevel(s)]/[pid-suffix] organisation registry: central (dona) minting: local assigning: local persistence: central resolving: central/local: pid-uri = https://hdl.handle.net/[handle] server admin: local rule based mapping: yes handles are internationally unique identifiers within the handle system under responsibility of the dona foundation [13]. handles can be assigned and administered decentrally by naming authorities. a handle consists of a prefix indicating the naming authority, and a “local name” or suffix, the unique identifier string within the context of the naming authority. this suffix can be an existing identifier. handles can be administered by naming authorities in their own local handle servers for specific prefixes. the global handle registry serves as an intermediate conduit for these local servers. prefixes can have hierarchical levels for layered naming authorities, which are divided by a dot (“.”). handles are independent of the dns architecture and url syntax, but they can be resolved on the web via the global handle resolver hdl.handle.net. example: https://hdl.handle.net/11245.1/8f66e0a8-b7c9-40a4-be28-54a7c0177061, which refers to the same article as mentioned in the “nbn” paragraph before. again, in this case the “local name” is identical to the internal uuid identifier of the repository system. handles can be assigned to any type of object. a single handle is a mapping between a handle identifier and the target url of a specific object. in the case of a physical object that would be a “landing page”, containing at least a description of the object. handle supports an elementary form of “multiple redirection”, which entails conditionally selecting a url from a list, based on “id”, “country” and “weight” [14]. there are also “template handles”, for dynamically generating redirects based on regular expressions and patterns instead of one to one mappings for each handle. naming authorities are charged a one time and an annual fee per prefix. currently this is $50 [15] for both. doi system type: pid syntax: 10.[registration agency code].[sublevel(s)]/[pid-suffix] organisation registry: central (idf), nodes minting: nodes: registration agencies assigning: nodes persistence: central resolving: central: pid-uri = https://doi.org/[doi] server admin: nodes rule based mapping: no doi’s (digital object identifier) [16] are based on the handle system. the doi system is administered by idf (international doi foundation). doi’s are assigned by decentralized doi registration agencies (ra), like datacite (mainly focused on research datasets) [17], crossref (mainly focused on scholarly publications) [18] and zenodo (focused on open science scholarly output) [19]. a doi, like a handle, consists of a prefix for the “registrant” and a suffix as unique identifier within the prefix. and just like handles, prefixes can have a hierarchical subdivision. doi’s always start with an extra prefix “10” for idf as highest handle naming authority. registrants are represented by the second prefix level, after the “10”, for instance zenodo is 10.5281. existing identifiers can be used as suffixes. doi’s are resolved through the global doi server doi.org, for instance https://doi.org/10.14742/ajet.4926, which again refers to the above mentioned article, in this case as published in a specific journal. because doi’s are handles, they can also be resolved through hdl.handle.net: https://hdl.handle.net/10.14742/ajet.4926. and vice versa. doi’s can be assigned to any type of object: physical, digital, abstract. doi offers “multiple resolution” [20], making use of handle “multiple redirection”. the doi handbook explicitly states that this mechanism can be used for linked data content negotiation [21] (see below). institutions pay the registration agency in question a fee per registered doi or per prefix, depending on the ra’s conditions. for instance datacite doi’s are created by datacite members [22], such as figshare [23]. institutions can become a datacite member or cooperate with an existing datacite member. ark type: pid syntax: ark:/[name assigning authority number]/[pid-suffix] organisation registry: central (california digital library) minting: local: name assigning authorities (naa) assigning: local persistence: local resolving: local: pid-uri = [local resolver url]/[ark] / central: https://n2t.net/[ark] server admin: local rule based mapping: yes (via noid for instance) ark (archival resource key) [24] is a protocol for persistent identifiers including resolving for various types of “information objects”: digital, physical, abstract, people, institutions. ark is developed and maintained by the california digital library. currently a project is carried out to transfer administration to an international community [25]. a founding principle of the ark is that persistence is purely a matter of service (kunze and rodgers, 2018). this leads to three requirements: an identifier gives access to “a promise of stewardship” a description of the identified object (metadata) the object itself, or a copy of it, if possible in the ark protocol an identifier is an association between a string of data and an object, so not just a string. ark’s are assigned by name assigning authorities (naa), that all have a name assigning authority number (naan). these naa’s are maintained in a central registry [26]. an ark consists at least of three mandatory parts: the string “ark:/”, the naan, and the “name” (the actual identifier or pid within the naa context, the suffix). again, existing identifiers can be used for the name/suffix part. an example: ark:/12148/cb406766211. the “ark:/” part is necessary, because there is no global ark resolver (similar to nbn’s “urn:nbn:”). an implicit hierarchical subdivision can be applied, without using dots (“.”) as separators. this can be used to indicate for instance sub-naming authorities and specific repositories (peyrard et al. 2014). in practice up to the first five characters of the actual identifier suffix (“name”) are used for this purpose. this “sub-namespace” part is also called a “shoulder” [27]. there is no fixed syntax rule for this option. in the example ark given here, the “cb” part of the name/suffix indicates the shoulder. a “name mapping authority hostport” including “http(s)://” can be added to make the ark “actionable” (resolvable on the web), for example this catalogue record of the national library of france (bnf): https://catalogue.bnf.fr/ark:/12148/cb406766211. the nmah is not part of the ark identifier. the nmah is variable and can be changed. several nmah’s can be used for the same ark, for instance to present an object in different systems, such as a catalogue or a repository. example: https://catalogue.bnf.fr/ark:/12148/cb406766211 (in the bnf catalogue) https://data.bnf.fr/ark:/12148/cb406766211 (in the bnf linked data server) furthermore, a “qualifier” can be added following the “name”. qualifiers can indicate hierarchies by means of a slash (“/”) (for instance parts or pages), or variants by means of a dot (“.”) (for instance different formats, languages or versions). qualifiers can in theory be used for linked data content negotiation, for instance by adding variants like “.rdf”, “.ttl”, etc. (peyrard et al. 2014). the nmah should support the three basic ark requirements using thump (“the http url mapping protocol”). in this protocol appending a question mark (“?”) to a url means a request for the description (metadata) of the identified object, and appending two question marks (“??”) means a request for the “promise of stewardship”. example: object: http://texashistory.unt.edu/ark:/67531/metapth346793/ description: http://texashistory.unt.edu/ark:/67531/metapth346793/? “promise”: http://texashistory.unt.edu/ark:/67531/metapth346793/?? in practice this is not always possible, because a question mark is standard syntax for adding a query parameter to a url, which may cause conflicts in the web server used. in any case, the ark identifier has to resolve to at least the identified object. if there is no local ark resolver available, there is always the global name-to-thing (n2t) resolver n2t.net, for instance http://n2t.net/ark:/12148/cb406766211. there is a tool that can generate unique ark strings (but also other identifiers like handles and doi’s): noid (nice opaque identifier) [28]. noid can also be used as local resolver that can provide rule based mapping besides standard redirection [29]. it is also possible to manage and store ark’s using the ezid service [30] provided by cdl. depending on institution type an annual fee of $300 – $1500 is required. a financial contribution is not required for acting as a name assigning authority. purl type: pid-uri/url syntax: [purl resolver base url]/[name] organisation registry: central (internet archive) minting: local assigning: local persistence: central resolving: local/central server admin: local/central rule based mapping: partial url purl (persistent url) is a resolution and redirection service for web resources. it was originally developed and administered by oclc. in 2016 the maintenance was taken over by the internet archive [31]. purl uses standard http redirect services by means of standard http status codes [32]. purl’s are full url’s, of which the base-url can be purl.org, purl.net, purl.info, purl.com. institutions can create and administer their own purl’s for their own domain. the default and most used form of redirection is by means of the status code 302 (simple redirection to a target url). example: http://purl.org/dc is redirected to https://www.dublincore.org/. besides the http status code there is also a “partial purl” which will match the beginning of a url and append the remainder to the end of the resolved url, for instance http://purl.org/dc/elements/1.1/: http://purl.org/dc/elements/1.1/#terms-abstract, http://purl.org/dc/elements/1.1/#terms-title, etc. purl’s are not persistent identifiers by themselves, but they can be used as such. there is no rule based mapping other than the partial purl option. private pid’s type: pid/pid-uri/url organisation registry: local minting: local assigning: local persistence: local resolving: local server admin: local rule based mapping: optional institutions also have the option of creating their own private pid’s. this implies that an institution has a tool at its disposal to generate unique identifiers within a specific namespace as well as a tool for resolving and redirecting these private pid’s by means of pid-uri’s to the actual url’s of the identified objects. most heritage and scholarly collaboration platforms only advocate the existing global pid systems handle and doi and don’t mention private pid systems. the dutch pidwijzer site even discourages the use of private solutions [33] because these would be less robust, reliable and durable and more costly and laborious than existing global pid systems, but that is debatable. many heritage and educational institutions will very probably have a considerable future life before them, including their namespaces on the web. as far as costs and laboriousness are concerned, external pid systems also require a considerable maintenance effort, just like private systems. an advantage of private pid systems is more flexibility in policies, applications and redirect options. one option for generating private pid’s is using uuid’s as suffix. uuid (universally unique identifier) [34] is an identifier independent of a central authority. a uuid is a 128 bit number consisting of 32 hexadecimal digits in five groups separated by a hyphen. uuid’s are generated based on a specific algorithm. there are five versions, based on date/time/mac-address, namespace/name or random. a uuid by itself is not a pid. there is no prefix/namespace context. uuid’s are widely used, for instance in the pure repository system mentioned in the examples before. the suffix 29-8f66e0a8-b7c9-40a4-be28-54a7c0177061 is a uuid. a uuid is not resolvable as such, it is only an identifier. uuid’s can be made resolvable by using a resolver and redirection mappings, like in handles, doi’s, ark’s or any other method, for instance the museumid pid system [35]. the museumid has two urn components based on uuid’s, the museum namespace (mns) and the museum object identfier (moi). system type registry mint assign persistence resolve server admin rule based nbn pid (+ pid-uri) national local local national national national no handle pid (+ pid-uri) central local local central central/local local yes doi pid/pid-uri central/nodes nodes nodes central central nodes no ark pid central local local local local local yes (noid) purl pid-uri/url central local local central/local central/local central/local partial url private pid/pid-uri local local local local local local optional typology of pid systems object types eligible for pid’s unique items, scholarly output it is self-evident that institutions in principle only assign pid’s to objects they own or administer themselves. for these objects a distinction can be made between unique and non-unique items, and between institutional and external output. for all objects owned, administered and published elsewhere pid’s should be assigned by the appropriate institutions. unique items museum objects, manuscripts, research datasets etc. copies of editions in itself not unique, but made unique by coloring, annotations etc. non-unique items objects published in large editions (books, articles, etc.) institutional output scholarly output by the institutional staff (books, dissertations, articles, chapters, datasets, presentations etc.). because these objects are usually published in large quantities, they also count as “non-unique items”. figure 2. overlap of item types institutions themselves are the appropriate authority for assigning pid’s to their own unique objects. it is standard practice that pid’s are assigned to scholarly output published in institutional repositories too, even if they also have an externally assigned pid, like a doi for articles published in a journal. non-unique objects owned or administered by the institution can be referenced by already available identifiers. books for instance usually have an oclc worldcat number and/or isbn on the frbr manifestation level. physical/digital the basic rule is that a pid is assigned to the original object (“non-information resource” in linked data terms), either physical or digital. one or more digital copies or versions of a physical object may be available. as a rule these digital versions are not assigned their own pid’s. there should however be some kind of relationship defined between the original object and its digital copies. it is customary that a physical object’s pid resolves to a description of that object on the web (in linked data terms an “information resource”), including at least information regarding its current location and access options and optionally representations of the available digital versions. if possible also the web locations of these digital versions should be given (van de sompel et al. 2014). figure 3. source object and digital representations the europeana data model (edm) provides for this basic situation [36]. an original object is described as a “providedcho”, where “cho” stands for “cultural heritage object”. digital versions of it are “webresources”. both object types can have their own specific metadata and licenses. a providedcho and its webresources are related by means of an “aggregation” that can have its own metadata. figure 4. edm cultural heritage object model in traditional marc, still used in most library systems, this is a bit more problematic, because of the fixed flat record structure. one possibility is to use marc 856 tags (“electronic location and access”) with available indicators and subfields [37]. a disadvantage of this method is that it does not provide an opportunity for recording metadata (format, resolution, etc.) and license information for the digital versions. this problem can be countered by creating separate marc records for digital versions. in this case these digital object records must be linked in some way to the original object record. the transition to bibframe may solve this. special cases just a little bit unique the above mentioned distinction between unique and non-unique objects is in reality not as rigid as it seems. in this case it concerns objects published in various volume sizes, with or without small differences, for example etchings, engravings, lithographs, old maps. with this type of objects multiple prints are made from one source (plate, stone etc.), of various quality caused for instance by tear and wear. plates used for etchings and engravings can have multiple versions (“states”) with adjustments made on the original plates. this has happened a lot with old maps, which are in general copper engravings. the printed images were published and sold in various ways: separately, as part of a book or atlas, etc. the original prints are always in black and white, but individual copies were often hand colored. a single image can have many different copies. a heritage institution may have multiple identical or varying copies of a printed object in its possession. the question is how to catalogue these different versions and if they all deserve their own pid. are they to be regarded as unique works in their own right, or as different manifestations or even as multiple copies of one work or manifestation? the answer depends on the intended or actual usage of these objects. if the individual copies are treated as unique objects or individual works, then they deserve their own pid’s. how to resolve these pid’s is a different story. in any case there should be a web resource for the copy in question, for instance a descriptive page or a digital representation or similar. figure 5. a map and its possible versions the individual copies should be linked to each other in one way or another, as different versions of one source object. this can be achieved in a number of ways in catalogues and metadata. whole/part often objects are part of a bigger entity, or they can be divided into smaller parts themselves. examples are: archive – sections – objects book – chapters serial – volumes wall map – separate sheets similarly the answer to the question on what levels pid’s are appropriate depends on the actual or intended usage. digital representations in some cases it is necessary to use stable url’s for digital representations of the original object, for instance in case explicit references are made to a specific digital representation with a specific resolution, if they are used in iiif, or if the objects, or parts of them, are the subject of online annotation tools. versions of objects in some pid systems it is not allowed to change the objects that are referenced. usually this situation does not apply to physical objects, except possibly in the case of damage. but for texts or datasets for instance this is a real possibility. generally speaking every single version must be assigned a new pid. one option is to extend the original pid with a version numbering suffix. the original pid can refer to the most recent version. this would in fact mean that the pid does not refer to one and the same object in a persistent way. alternatively new pid’s can be assigned to every new version, so the original pid keeps pointing to the original object as it was at the time of assignment (van de sompel et al. 2013). figure 6. object versions and the pid versioning methods (“v1” – “v3” = “version 1” – “version 3”). timeline is from top to bottom. left: pid refers to most recent object version, older versions get new pid’s. right: pid refers to original object version, every new version gets new pid. pid’s and linked data linked data is a way of linking structured data in different datasets to each other by means of identifiers in order to present enriched information both for human use and for software processing (usually described as “machine readable”). the basic methodology consists of “triples”, where two things are related to each other, ideally presenting both things and the relationship as special pid-uri’s, http uri’s [38]. furthermore “content negotiation” is used to redirect these uri’s to either a human interface (like a web page) or a machine readable dataset in the rdf format, or to the object itself in the case of a digital object. redirection of a uri/url to a specific representation of the object is called “dereferencing”. an example of such a triple: figure 7. a triple https://hdl.handle.net/11245.1/8f66e0a8-b7c9-40a4-be28-54a7c0177061 (an article) http://purl.org/dc/elements/1.1/creator (written by) https://orcid.org/0000-0001-8795-6838 (a person) figure 8. content negotiation (van de sompel et al., 2014). http-uri-pid = resolvable form of the pid; http-uri-land = human readable web resource; http-uri-mach = machine actionable web resource; http-uri-loc = various end targets of the identified object uri’s a uri (uniform resource identifier) is a unique identifier [39]. uri’s exist in two forms: urn (uniform resource name) and url (uniform resource locator). urn a urn is a unique identifier in a specific namespace or uri scheme. urn’s do not have to comply to a specific protocol and do not have to be resolvable on the web. it is not a “locator”. urn’s can be monitored and guaranteed by central authorities, such as urn:issn and urn:isbn. some pid’s are in the form of urn’s: urn:nbn. urn’s can be made resolvable by dedicated resolvers such as is the case with nbn. url a url is a unique identifier resolvable on the web and always leading to a web resource on a specific web server. url’s use various protocols: http, ftp, mailto, etc. url’s always contain a physical location on the web in the form of a hostname. http uri’s are essential for linked data. the above mentioned pid systems that essentially don’t use the url syntax (except purl) can be expressed as url’s by prefixing the pid’s with “http://” or “https://” and the hostname of the resolver. for instance https://hdl.handle.net/11245.1/8f66e0a8-b7c9-40a4-be28-54a7c0177061 for the handle 11245.1/8f66e0a8-b7c9-40a4-be28-54a7c0177061. pid’s as linked data http uri’s in order to be used as a linked data identifier in rdf a uri has to comply with a second condition besides resolvability: content negotiation (so, selecting the representation to be delivered based on context). the described pid systems were not originally designed to do that. handle, doi and ark have mechanisms that might be used for it, but these appear to be too basic to configure this in a satisfactory manner. doi’s are not very suitable because the infrastructure is administered completely by an external party. in case the selected pid system cannot be used for content negotiation, additional functionality will have to be developed separately. with private pid systems content negotiation functionality can be part of the development. existing infrastructures and pid’s most heritage institutions will have at least one cataloguing system in which their collections’ items are described. these cataloguing systems either have an integrated or a separate end user interface used for presenting and making available the collection items on the web. depending on the nature of the institution and its collections these systems can be focused on publications, physical artefacts or archived items, all with their own specific features. many institutions have a combination of these collection types, possibly extended with repositories for digital objects. research institutions will also have research information systems with integrated or separate repositories for scholarly output and research datasets. many academic libraries will have all of these types of collections and systems to administer. in many cases items are described in more than one institutional system, either by copying or by duplicate cataloguing. metadata from many institutional systems are also harvested and published in all kinds of national and global aggregator portals. scholarly output in the form of articles are also published and described in e-journals and e-journal aggregators. across this variety of digital heritage infrastructures many different pid policies can be found. pid’s may or may not be assigned to all objects or to only a subset. pid’s may be available for objects in specific systems only, but not in other systems the objects are described in. pid’s can be managed in a full mapping table or dynamically generated using rule based mapping. pid’s based on existing global systems can’t be used for publishing collections as linked data. objects may have multiple pid’s (a local handle, a national nbn and a doi). pid’s may be assigned to digital versions only instead of their original physical source objects. an appropriate pid infrastructure based on the options of the described pid systems and the potential shortcomings of pid usage in existing infrastructures a list of requirements for pid systems can be drafted. the list can be used to formulate recommendations for specific activities regarding implementing pid’s. requirements for pid systems it must be possible to assign pid’s to objects of all kinds (physical, digital, abstract) institutions must be able to generate and assign pid’s themselves based on existing internal system identifiers based on existing pid’s as new automatic generation and assignment of pid’s for new objects must be possible batch generation and assignment of pid’s for existing objects must be possible resolving pid’s must be possible based on one to one mapping based on templates, patterns or rules it must be possible to adapt resolver mapping tables in batch pid’s must be usable as linked data uri’s by means of content negotiation it must be possible to store generated pid’s automatically in the object’s source system/database if possible as the default internal system identifier alongside the default internal system identifier it must be possible to copy pid’s from the source system/database to derivative systems/databases it must be possible to present pid’s in all relevant public end user interfaces it must be possible to publish pid’s in all machine readable interfaces it must be possible to retrieve objects and their metadata records by their pid’s in all public end user interfaces it must be possible to retrieve objects and their metadata records by their pid’s in all machine readable interfaces generating and assigning pid’s using more than one pid system does not have to be a problem. it just means more maintenance effort is required. some pid systems are even administered externally, notably doi’s for journal articles. if an institution already uses a specific pid system for a subset of its collections, it can extend the system to all of its objects, or decide to deploy a different system. in the later case the existing pid’s must still be maintained. when implementing a new pid system it has to be decided what the basis of the new pid’s will be: new or existing identifiers. new identifiers can be generated based on uuid’s, the default numbering of the pid system itself, other pid systems already in use or a private algorithm. these identifiers can be incorporated as suffix in the pid system of choice. as an example of using an existing pid system’s pid’s in another pid system: an institution could use an already used handle server to generate new handles, only to be used as suffixes in ark. the original handles should of course not be published. in this case there is no need for implementing and administering an additional separate pid generator. all information and cataloguing systems have unique internal system identifiers that can be used in pid’s. these internal system identifiers in a manner of speaking are promoted to pid suffix in the pid system in question and made universally unique. this is by far the easiest solution for implementing pids, because these pid’s do not have to be generated and stored separately in the metadata of the source system objects. this method only works if the pid’s can be generated dynamically on the fly using the internal system identifier as suffix by means of rules, patterns or templates. of course in the case of a migration this method implies that existing internal system identifiers have to be stored as additional queryable identifiers in the new system. if objects are available in more than one system/frontend, for instance the cataloguing system and a repository, and internal system identifiers are used as pid suffixes, then only one should be used for generating the pid. this is directly related to resolving pid’s (see below). it depends on the context of the institutional infrastructure which system will be leading. storing pid’s in the case of using internal system identifiers pid’s do not have to be stored in an object’s metadata at all. but when creating pid’s using new external identifiers (uuid’s, existing pid’s, private algorithms), these pid’s do have to be stored separately. it depends on the cataloguing systems if and how this is possible. library systems using marc can use the marc 024 tag (“other standard identifier”) [40] to store identifiers. 024 subfield $2 (“source of number or code”) can be used to record the type of identifier, for instance “doi” for doi’s, “hdl” for handles. there is a list of standard identifiers [41] available, but systems not in the list can be used too. ark could for instance be indicated by “$2ark”. if objects are available in more than one system and internal system identifiers are used as pid suffixes, complete pid’s must be stored in all other systems as well. resolving pid’s most eligible pid systems have their own local and/or central resolver servers. in the case of a private pid system a resolver mechanism has to be implemented as well. the standard way of resolving pid’s is the use of a mapping table containing a target url for every pid. this method is certainly usable for generating individual pid’s for new object registrations. the cataloguing workflow in this case should provide for automatic updating of the mapping table. batch generation of new pid’s for all records of an existing system should be reasonably easy to implement too. but when migrating to a new system that will be a lot harder to achieve. all existing target url’s in the mapping table will have to be replaced by url’s pointing to records in the new system. doing this in batch is only possible if a mapping list of all old system record identifiers and the new system record identifiers is available. in this case (but also in the other cases) it is much easier to achieve this by one rule based mapping line. for this to be possible the pid’s must be stored and queryable in the new system. something like: [pid base-url]/prefix.suffix => [system base-url]?pid_field=[prefix/suffix]. if objects are available in more than one system, only one of these can be configured as target in the resolver. if internal system identifiers are used as pid suffixes, then obviously that system will be the target. in this case links between the objects in all systems have to be available. ark provides an alternative: the pid can be prefixed with multiple nma’s (base-url’s), each requiring its own resolver of course. pid’s and content negotiation in order to be able to use pid’s as linked data http uri’s the pid system has to support content negotiation. the base-url/hostname for pid’s used as linked data uri’s must be immutable (persistent) just as the pid’s themselves. on top of that the base-url server must host a pid resolver that also manages content negotiation. this base-url must not be one used for a specific system/frontend. one strategy is to use a dedicated hostname for pid’s, like id.institution.org or similar. in this case there are two options. under this dedicated pid hostname a server can be hosted that will perform full content negotiation including standard pid redirection to a human interface. or one of the content negotiation options will be forwarding to an external pid system, such as handle, to perform the actual pid resolution. alternatively the external pid system’s syntax, for instance handle, will be used for everything. the pid resolver then redirects to the institutional content negotiation server, which handles also resolving of the pid to the human interface. of course it would be best if all external pid systems would offer content negotiation from the start. publishing pid’s presenting pid’s in various system end user interfaces should be relatively easy. pid-uri’s must be prominently presented as the proper way to refer to objects or records, instead of the url shown in the browser bar, which usually shows the system url with internal identifier. if internal system identifiers are used as pid suffix, then all records in these systems will have a pid, if they comply to the pid criteria used or not. in this case mechanisms can be applied that the pid’s are only published for objects that deserve pid’s. an additional benefit is that pid’s already exist when these criteria are extended. publishing pid’s for machine readable interfaces should also not be a problem, for instance in api’s and as rdf subject uri’s. an alternative method would be to use http headers in the frontends’ html pages, like it is done with the signposting approach [42]. here typed links with relation type “cite-as” are used, for example: <link href=”https://hdl.handle.net/11245.1/8f66e0a8-b7c9-40a4-be28-54a7c0177061″ rel=”cite-as”> figure 9. https://signposting.org/identifier/dspace/ version and representation pid’s it must be possible to show the relationships between the objects that have a pid assigned and their related versions and representations in the various frontends, using metadata and holdings data. it is important that these relationships are bidirectional, meaning that navigation is possible from object description to representations and vice versa. machine readable interfaces should also be able to capture these relationships. signposting can be used for this too, by means of “collection” and “item” as well as “cite-as” relation types. the html page describing the object contains a relation type “item” link for each version representation. the representation html page then contains a relation type “collection” link. additionally a “cite-as” link contains the pid-uri of the object itself. figure 10. https://signposting.org/publication_boundary/irrodl/ discussion what should be done? in order to improve internal and external interoperability, facilitate linked data and promote the reuse of digital collections, institutions must assign pid’s to all relevant objects. institutions must define a central policy for using pid’s. it is vital that already existing pid’s will keep working. existing pid’s can be integrated in new pid systems. pid’s must be available in all systems and interfaces (both for human end users and machine readable) used for describing the objects in question. institutions have to determine the default web representation of objects with pid’s. all other representations have to be findable from that default representation. if institutions want to publish collection data as linked data, they should make sure that the pid’s can be used as linked data uri’s. options and issues an easy way of implementing pid’s for all relevant physical and digital objects is to apply rule based mapping, generating pid-uri’s on the fly based on existing internal system identifiers. in practice this is possible with handle, ark or private pid’s. of course the internal system identifiers will have to be migrated to any new system in order for the pid’s to continue to be valid and resolvable. if there is any chance that existing internal system identifiers are subject to change, then this method is obviously not suitable. the alternative is that for every new record created for “pid worthy” objects in relevant systems an external pid is automatically generated, mapped and stored in the system. in this case rule based mapping can’t be applied. it depends on the system used if this is an option. home grown and open source systems can and should be extended with the necessary functionality by the institution or the community. commercial systems do not usually provide this functionality, but it should be or become standard practice. even if pid-uri’s are published prominently on web representation pages of objects, the browser address bars still present the actual target url’s of the redirected pid-uri’s. many people will store this url instead of the preferred pid-uri either by actively copying the browser bar url or by bookmarking a web page, thereby implicitly storing the url. it seems impossible to eliminate this problem. an alternative solution for bookmarking tools could be that the signposting “cite-as” relation type is used whenever available. the institution has to decide on the criteria used for defining which object types pid’s are assigned to, while considering the special cases regarding the various physical and digital representations. having said this, it would be very useful if institutions could implement existing standard profiles for dealing with pid’s for source objects (both physical and digital) and ways to identify the derived and related digital representations, as well as versions. to facilitate linked data using assigned pid’s, content negotiation must be available in one form or another, preferably without having to use separate linked data uri’s besides pid-uri’s. in most cases a separate tool will have to be developed or implemented for this. this may involve the decision to use an institutional namespace instead of an existing pid system namespace. this could be avoided if existing pid systems would support content negotiation as a built-in option. looking at current developments it is incomprehensible that for instance handle and doi do not support full blown content negotiation. conclusion a consistent policy for persistent identifiers for heritage objects is indispensable for institutions that want to streamline their internal digital infrastructure and publish their collections for reuse. the selected pid systems must support content negotiation in order to publish the collections as linked data. there is not one pid solution that caters for all institutions. there are many dependencies involved: collection management/cataloguing systems, vendors, institutional it expertise and staff, existing pid systems. many collection management systems, commercial and open source, do not provide integrated pid support (generating pid’s and resolvers). this is an issue that heritage institutions and their professional associations should ask vendors/developers to incorporate in their roadmaps. existing pid systems do not support content negotiation for linked data implementations. heritage institutions and their professional associations should encourage pid system administrators to implement this. institutions with the appropriate technical ict expertise, staff and funding can implement their own pid and content negotiation infrastructure, with or without commercial collection management systems. even then they have to decide on the best model or topology: which type of objects to assign pid’s to, how to implement the relationships between physical objects and digital representations, etc. heritage institutions and their professional associations should investigate the possibilities of establishing some kind of shared open repository of standard profiles for implementing pid’s. it is wise to test a number of different scenarios for various situations. you do not have to select one solution for all types of collections and systems. and it is not necessary to implement pid’s for all collections an systems at he same time. it is better to take small steps. in any case, institutions must implement the solution that can be administered with the least effort, preferably with as much automation as possible, because only then persistence is feasible. acknowledgements i am grateful to the following people for sharing valuable suggestions and feedback on earlier versions of this article. herbert van de sompel (dans, the hague, the netherlands) jasper bedaux (library of the university of amsterdam, the netherlands) remco van veenendaal (national archives, the hague, the netherlands) tim hasler (kobv, berlin, germany) about the author lukas koster msc (l.koster@uva.nl) is digital infrastructure coordinator at the library of the university of amsterdam (uva)/amsterdam university of applied sciences (hva). current projects: digital infrastructure architecture model, implementing object pid’s, survey digital object repositories, object and metadata licensing, resolvable controlled vocabularies. orcid: 0000-0003-0214-4721. notes [1] https://www.dpconline.org/handbook/technical-solutions-and-tools/persistent-identifiers [2] https://support.orcid.org/hc/en-us/articles/360006971013-what-are-persistent-identifiers-pids[3] https://www.ncdd.nl/en/pid/ [4] https://en.wikipedia.org/wiki/persistent_identifier (september 5, 2019) [5] https://nl.wikipedia.org/wiki/persistent_identifier (september 5, 2019) [6] https://www.doi.org/faq.html (6. what can a doi name be assigned to?) [7] http://www.doi.org/factsheets/doikeyfacts.html [8] https://nl.wikipedia.org/wiki/uniform_resource_locator [9] https://en.wikipedia.org/wiki/national_bibliography_number [10] https://en.wikipedia.org/wiki/uniform_resource_name [11] https://www.kb.nl/organisatie/onderzoek-expertise/informatie-infrastructuur-diensten-voor-bibliotheken/registration-agency-nbn [12] https://dans.knaw.nl/ [13] https://www.handle.net/ [14] https://www.handle.net/overviews/handle_type_10320_loc.html [15] https://www.handle.net/payment.html [16] http://www.doi.org/ [17] https://datacite.org/ [18] https://www.crossref.org/ [19] https://zenodo.org/ [20] http://www.doi.org/doi_handbook/3_resolution.html#3.3 [21] http://www.doi.org/doi_handbook/5_applications.html#5.4 [22] https://datacite.org/members.html [23] https://figshare.com/ [24] http://n2t.net/e/ark_ids.html [25] https://wiki.duraspace.org/display/arks/arks+in+the+open+project [26] https://n2t.net/e/pub/naan_registry.txt [27] https://ezid.cdlib.org/learn/id_concepts [28] http://n2t.net/e/noid.html [29] https://metacpan.org/pod/distribution/noid/noid#rule-based-mapping [30] https://ezid.cdlib.org [31] https://archive.org/services/purl [32] https://archive.org/services/purl/help [33] https://www.pidwijzer.nl/en/faq [34] https://en.wikipedia.org/wiki/universally_unique_identifier [35] http://museumid.net/ [36] https://pro.europeana.eu/resources/standardization-tools/edm-documentation [37] http://www.loc.gov/marc/bibliographic/bd856.html [38] https://www.w3.org/tr/ld-bp/#http-uris [39] https://en.wikipedia.org/wiki/uniform_resource_identifier [40] http://www.loc.gov/marc/bibliographic/bd024.html [41] http://www.loc.gov/standards/sourcelist/standard-identifier.html [42] https://signposting.org/ references ark. [accessed 2019 nov 21]. http://n2t.net/e/ark_ids.html. ark naan registry. [accessed 2019 nov 21]. https://n2t.net/e/pub/naan_registry.txt. arks in the open. [accessed 2019 nov 21]. https://wiki.lyrasis.org/display/arks/arks+in+the+open+project. crossref. [accessed 2019 nov 21]. https://www.crossref.org/. dans. [accessed 2019 nov 19]. https://dans.knaw.nl/nl. datacite. [accessed 2019 nov 21]. https://datacite.org/. datacite – members. [accessed 2019 nov 21]. https://datacite.org/members.html. digital object identifier system faqs. [accessed 2019 sep 17]. https://www.doi.org/faq.html. doi. [accessed 2019 nov 19]. https://doi.org. doi – linked data. [accessed 2019 nov 19]. http://www.doi.org/doi_handbook/5_applications.html#5.4. doi – multiple resolution. [accessed 2019 nov 19]. http://www.doi.org/doi_handbook/3_resolution.html#3.3. doi key facts. [accessed 2019 sep 17]. https://www.doi.org/factsheets/doikeyfacts.html. europeana data model. europeana pro. [accessed 2019 nov 21]. https://pro.europeana.eu/resources/standardization-tools/edm-documentation. ezid. ezid. [accessed 2020a jan 30]. https://ezid.cdlib.org/. ezid: identifier concepts and practices at the california digital library. ezid. [accessed 2020b jan 30]. https://ezid.cdlib.org/learn/id_concepts. figshare. [accessed 2019 nov 21]. https://figshare.com/. handle.net registry. [accessed 2019 nov 19]. https://www.handle.net/. handle.net registry – 10320/loc handle type. [accessed 2019 nov 19]. https://www.handle.net/overviews/handle_type_10320_loc.html. handle.net registry – payment. [accessed 2019 nov 19]. https://www.handle.net/payment.html. hilse h-w, kothe j. 2006. implementing persistent identifiers: overview of concepts, guidelines and recommendations. london/amsterdam: consortium of european libraries; european commission on preservation and access. [accessed 2019 sep 17]. http://xml.coverpages.org/ecpa-persistentidentifiers.pdf. klump j, huber r. 2017. 20 years of persistent identifiers – which systems are here to stay? data science journal. 16(0):9. doi:10.5334/dsj-2017-009. kunze j, rodgers r. 2008 may 22. the ark identifier scheme. [accessed 2019 aug 19]. https://n2t.net/ark:/13030/c7cv4br18. library of congress: marc 21 format for bibliographic data: 024: other standard identifier. [accessed 2019 nov 21]. http://www.loc.gov/marc/bibliographic/bd024.html. library of congress: marc 21 format for bibliographic data: 856: electronic location and access. [accessed 2019 nov 21]. http://www.loc.gov/marc/bibliographic/bd856.html. library of congress: standard identifier source codes: source codes for vocabularies, rules, and schemes. [accessed 2019 nov 21]. http://www.loc.gov/standards/sourcelist/standard-identifier.html. museumid. [accessed 2019 nov 21]. http://museumid.net/. national bibliography number. 2019a. in: wikipedia. [accessed 2019 sep 17]. https://en.wikipedia.org/w/index.php?title=national_bibliography_number. noid. [accessed 2019 nov 21]. http://n2t.net/e/noid.html. noid – rule based mapping. [accessed 2019 nov 21]. https://metacpan.org/pod/distribution/noid/noid#rule-based-mapping. persistent identifier. 2016. in: dutch wikipedia. [accessed 2019 sep 17]. https://nl.wikipedia.org/w/index.php?title=persistent_identifier. persistent identifier. 2019b. in: english wikipedia. [accessed 2019 sep 17]. https://en.wikipedia.org/w/index.php?title=persistent_identifier. persistent identifier wijzer – frequently asked questions. [accessed 2019 nov 21]. https://www.pidwijzer.nl/en/faq. persistent identifiers. 2015. in: digital preservation handbook. 2nd ed. digital preservation coalition. [accessed 2019 sep 17]. https://www.dpconline.org/handbook/technical-solutions-and-tools/persistent-identifiers. persistent identifiers. ncdd. [accessed 2019 sep 17]. https://www.ncdd.nl/en/pid/. peyrard s, tramoni j-p, kunze j. 2014 oct 1. the ark identifier scheme: lessons learnt at the bnf and questions yet unanswered. [accessed 2019 aug 15]. https://escholarship.org/uc/item/58d52295. purl. [accessed 2019 nov 21]. https://archive.org/services/purl/. purl help. [accessed 2019 nov 21]. https://archive.org/services/purl/help. registration agency nbn | koninklijke bibliotheek. [accessed 2019 nov 19]. https://www.kb.nl/organisatie/onderzoek-expertise/informatie-infrastructuur-diensten-voor-bibliotheken/registration-agency-nbn. signposting. [accessed 2019 nov 21]. https://signposting.org/. uniform resource identifier. 2019c. in: wikipedia. [accessed 2019 nov 21]. https://en.wikipedia.org/w/index.php?title=uniform_resource_identifier&oldid=925198387. uniform resource locator. 2019d. in: wikipedia. [accessed 2019 nov 19]. https://nl.wikipedia.org/w/index.php?title=uniform_resource_locator&oldid=54998398. uniform resource name. 2019e. in: wikipedia. [accessed 2019 sep 17]. https://en.wikipedia.org/w/index.php?title=uniform_resource_name. universally unique identifier. 2019f. in: wikipedia. [accessed 2019 nov 21]. https://en.wikipedia.org/w/index.php?title=universally_unique_identifier&oldid=927079784. van de sompel h, sanderson r, shankar h, klein m. 2014. persistent identifiers for scholarly assets and the web: the need for an unambiguous mapping. international journal of digital curation. 9:331–342. doi:10.2218/ijdc.v9i1.320. w3c best practices for publishing linked data – http uri’s. [accessed 2019 nov 21]. https://www.w3.org/tr/ld-bp/#http-uris. what are persistent identifiers (pids)? orcid. [accessed 2019 sep 17]. http://support.orcid.org/hc/en-us/articles/360006971013-what-are-persistent-identifiers-pids-. zenodo. [accessed 2019 nov 21]. https://zenodo.org/. update on may 13th 2020 following the author’s request we changed http://n2t.info to the canonical http://n2t.net in the urls of three ark links (the editor). subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – visualizing media archives: a case study mission editorial committee process and structure code4lib issue 6, 2009-03-30 visualizing media archives: a case study the wgbh media library and archives is piloting an online media archive for scholarly research. in conversation with users, we have discovered they want to quickly pinpoint items relevant to their work and get an overview of collections and their relationships to other materials. to demonstrate the size and complexity of our collection to users in a meaningful way, wgbh is employing data visualization techniques to provide an interactive, graphical representation of the various relationships between items. this article discusses the techniques employed in implementing our relationship map, emphasizes the cataloging techniques required for this effort, and offers code and examples to spark discussion about ways to improve or extend this effort. introduction the wgbh media library and archives has embarked on a project to design and pilot an online media archive for scholarly research. having launched http://openvault.wgbh.org in 2006 for educators and the general public, we are now focusing on the specific needs of academic researchers with our pilot site, http://openvaultresearch.wgbh.org. the goal of the project is to work with three prominent scholars to design a web-based system to address the needs of archive users in terms of searching, selecting, and accessing archival moving image materials. in conversation with our scholars, wgbh staff have discovered that academic researchers want to get a broad overview of our collections immediately and then quickly pinpoint items relevant to their work. in addition, they are extremely interested in learning how materials within the collection relate to each other. for example, scholars researching in media collections may be interested in a particular sequence of footage. they want to view and analyze the content, but they also seek information about its provenance: who shot it, who owns it, and how it was used (or not used) by producers for a particular documentary program or web site. successfully translating this information to a searchable and quickly scannable web site is our objective. one strategy we have employed in service to this end, has been to create an online visual representation of the relationships among and between items in the collection. while our scholars have struggled to become oriented to this visual method of surveying content, we are confident that further research and revision will prove our graph to be one useful device in the researchers’ toolkit. the challenge of moving image archives there are three major challenges for moving image archivists trying to serve the needs of researchers. the first is lack of exposure. media archives have traditionally been more difficult to access than paper-based archives because of obsolete formats and playback technologies. add to this copyright barriers, some scholars’ attitudes towards the use of media as a source, and the relative newness of history captured as moving images, and we find that media archives are only beginning to gain momentum in the research arena. the second challenge for moving image archivists relates to access strategies. the standards for cataloging moving image materials have traditionally mimicked bibliographic cataloging. they have focused on finished films with titles and authors. however, in order for researchers to find and access raw footage and film elements,  these materials must be cataloged at the item or sub-item level. this work is not only labor intensive, it also requires direct access to the materials via transfers, digitization, or the use of rare and aging playback equipment. for all of these reasons, moving image archives find themselves struggling to make their collections easily accessible to researchers and the public. finally, moving image archives face a number of technical challenges, due to the lack of existing applications and utilities for managing and exposing the repositories that are developed. consequently, there are few common or broadly applicable policies and frameworks for the legal issues, vocabulary development, and information discovery. the digital media files themselves pose problems for archivists as the file size of preservation quality video makes redundant copies of video expensive, but necessary in the volatile landscape of evolving compression formats. media archives themselves face obsolescence, however, if they do not find a way to make their collections accessible. given the rise of youtube, hulu and the current generation of “digital natives” who take video on the web for granted, moving image content must be exposed on the web or it may as well not exist. as the keepers of rich historical materials, moving image archivists find themselves at a turning point. with the exception of media studies, established humanities scholarship has yet to recognize film and television archives as a go-to source. yet, history is increasingly born digital and, just over the horizon, the younger generation expects their content to be delivered online and on-demand. background in order to bridge the gap between the seasoned generation of scholars who seldom use media and the emerging generation of scholars who demand it online, the andrew w. mellon foundation has been supporting projects which seek to make archival moving image content easily accessible. we launched our prototype in november of 2008 in order to solicit feedback from our scholar advisors. we continue to gather user input and plan to revise the site based on this data in spring 2009. established in 1978, the wgbh archives contains 750,000 asset level records representing the film, videos, still images and documentation gathered in the course of our productions. each of these components, gathered for one project initially, are constantly used and re-used by various productions. therefore the relationships between these assets are often highly complex. wgbh productions collect hundreds of assets in the course of their research, much of which never makes it into the final documentary films. for example, the american experience‘s 2 hour documentary on jimmy carter deposited 2040 assets into the archive on shutdown. while three minutes of an interview may be edited into a final documentary, the archives hold the entire hour-long interview conducted by the filmmakers. a five-minute sequence of news footage ordered from a network news station may be trimmed to 20 seconds for a production, but the archives hold the full five minutes for the next production to exploit. other examples include an interview with physicist david wilkinson filmed in 1998 for a science odyssey which was also used in nova‘s “origins: back to the beginning” in 2004. footage shot for a documentary on huckleberry finn was re-used by american experience for a film on kit carson. each of these uses are recorded in both our electronic databases and in legal documents also housed within the archives. one shot of moving image material, therefore, is represented across multiple databases and referenced by multiple related assets. our repository architecture attempts to record descriptive information about intellectual content (programs, series, etc), which allows us to create explicit relationships between digital objects from information once inferred from naming conventions in a physical assets database. this work will help tie together the loosely federated databases to provide an accurate and complete description of provenance. assets also share common themes by subject, location, and personalities. for example, wgbh producers have interviewed former national security advisor robert mcfarlane on four different occasions for four different projects. a search across the archives for reagan or mcfarlane reveals this wealth of oral history content. wgbh researchers working on a documentary on taiwan spent months gathering materials before they narrowed their selections for editing into their film. the archives, therefore, hold a wealth of material on taiwan which never made the final cut of the program.  our prototype seeks to illustrate these subject based relationships for researchers explicitly. for our descriptive metadata we are using several linked data sources which, we hope, provide sufficient information to construct broader terms or general categories of information, so additional data can be derived easily from external data sources (e.g. a map of locations covered by pulling in latitude/longitude information from resources like the getty thesaurus of geographic names or even the folksonomic freebase) and potentially re-integrated into the search index for a better search experience.  we also provide tagging suggestions from our controlled vocabularies for folksonomic data, which we presume will make re-integrating user-generated material easier.[1] our challenge is to make our content accessible not only in an orderly way, but also in a way that allows for quick surveyal and for “serendipitous discovery.”[2] while uni-dimensional data is quite easily expressed in lists and tables, the material in our archives rarely is so easily categorized. we are therefore experimenting with visualization as a new way for researchers to explore and browse our media collection and to see the relationships between assets based on shared subjects, persons and provenance. our current solution, a radial graph, also serves a secondary purpose: it creates a visually rich and interesting glimpse into our collections, themselves primarily visual. in this way, it helps acclimate the researcher to the types of materials they will find within our site. the cataloging behind the visualization as a media archive, our cataloging needs are very different from those of a library. we also do not fall into the traditional archival framework of the finding aid. we arrange as archivists but describe as librarians. that is to say we manage series and collections of materials whose provenance is of utmost importance, but, because of both the non-human-readable formats on which our collections exist, and because our content is so visually oriented, we must describe our materials to the item and even sub-item level. without thorough item level description, we have found that our media assets are of relatively little use because items of interest are difficult to uncover. at the same time, without collection level context, they lose much of their value to scholars.[3] to exploit the meaning and content of individual assets in a media archive, or any archive for that matter, the researcher must be able to see a larger picture. for our project, we have attempted to draw this picture by illustrating selected metadata-driven relationships between assets. our descriptive metadata is contained within a pbcore xml document, one document per digital asset. [4] wgbh has an internal digital asset management (dam) system from which we export metadata, which is converted to pbcore xml, and groomed as necessary. while pbcore describes media assets accurately,  it is a relatively new and underutilized standard, and we have found some areas for improvement. to supplement the pbcore vocabulary we have used xml namespaces to “mash” in, for example, the more precise date elements from the european broadcasting union’s ebucore. [5] for the purposes of visualizing the collection, the most important elements in the pbcore document are coverage (spatial), creator, contributor, and subject. we have struggled with the schema which defines contributor rather loosely and sometimes conflicts with our “people as subject” cataloging. for example, in an interview with jimmy carter’s national security advisor zbigniew brzezinski, should carter himself be a <contributor>, with other personalities, or should he be classified as a <subject>, with more abstract concepts? these are the kinds of cataloging kinks we are still working out. although the pbcore document serves as a high level summary of an archival asset, the real nitty-gritty details which greatly enrich the visualization come from the asset’s sub-item or content level description: a tei encoded transcript or shot log. using the “transcriptions of speech” profile of the text encoding initiative we tag people, places, dates, and subjects where applicable. we have made use of the library of congress authorities and subject headings as our cataloging authority wherever possible. as a back-up, we make use of wikipedia terms. for geography, we utilize the getty thesaurus of geographic names or tgn. below is a sample from an encoded transcript of our interview with zbigniew brzezinski about president jimmy carter: <keywords scheme="http://authorities.loc.gov"> <term xml:id="carter_presidency" xhtml:href="http://authorities.loc.gov/cgi-bin/pwebrecon.cgi?authrecid=55297&amp;amp;amp;amp;amp;amp;amp;amp;v1=3&amp;amp;amp;amp;amp;amp;amp;amp;hc=3&amp;amp;amp;amp;amp;amp;amp;amp;seq=20081021125716&amp;amp;amp;amp;amp;amp;amp;amp;pid=sikamkqj81yzyqevltutqwinzovun">united states. president (1977-1981 : carter)</term> <term xml:id="jimmy_carter" xhtml:href="http://authorities.loc.gov/cgi-bin/pwebrecon.cgi?authrecid=1886254&amp;amp;amp;amp;amp;amp;amp;amp;v1=1&amp;amp;amp;amp;amp;amp;amp;amp;hc=3&amp;amp;amp;amp;amp;amp;amp;amp;seq=20081021125716&amp;amp;amp;amp;amp;amp;amp;amp;pid=sikamkqj81yzyqevltutqwinzovun">carter, jimmy, 1924</term> <term xml:id="camp_david_accords" xhtml:href="http://authorities.loc.gov/cgi-bin/pwebrecon.cgi?authrecid=3527320&amp;amp;amp;amp;amp;amp;amp;amp;v1=1&amp;amp;amp;amp;amp;amp;amp;amp;hc=3&amp;amp;amp;amp;amp;amp;amp;amp;seq=20081027185602&amp;amp;amp;amp;amp;amp;amp;amp;pid=aq4q-fxob_mjtehktcoptplezkcyz">camp david agreements (1978)</term> <term xml:id="middle_east" xhtml:href="http://authorities.loc.gov/cgi-bin/pwebrecon.cgi?authrecid=4740719&amp;amp;amp;amp;amp;amp;amp;amp;v1=1&amp;amp;amp;amp;amp;amp;amp;amp;hc=3&amp;amp;amp;amp;amp;amp;amp;amp;seq=20080924182635&amp;amp;amp;amp;amp;amp;amp;amp;pid=o4ofiu_y0rauyowrztlezj99x7khl">middle east</term> <term xml:id="cold_war" xhtml:href="http://authorities.loc.gov/cgi-bin/pwebrecon.cgi?authrecid=4818518&amp;amp;amp;amp;amp;amp;amp;amp;v1=1&amp;amp;amp;amp;amp;amp;amp;amp;hc=2&amp;amp;amp;amp;amp;amp;amp;amp;seq=20080925121225&amp;amp;amp;amp;amp;amp;amp;amp;pid=uiw3m7iqs1lraot2ifffiznksijlc">cold war</term> </keywords> <keywords scheme="http://www.getty.edu"> <term xml:id="china" xhtml:href="http://www.getty.edu/vow/tgnfulldisplay?find=china&amp;amp;amp;amp;amp;amp;amp;amp;place=&amp;amp;amp;amp;amp;amp;amp;amp;nation=&amp;amp;amp;amp;amp;amp;amp;amp;prev_page=1&amp;amp;amp;amp;amp;amp;amp;amp;english=y&amp;amp;amp;amp;amp;amp;amp;amp;subjectid=1000111">china (nation)</term> <term xml:id="soviet_union" xhtml:href="http://www.getty.edu/vow/tgnfulldisplay?find=soviet+union&amp;amp;amp;amp;amp;amp;amp;amp;place=&amp;amp;amp;amp;amp;amp;amp;amp;nation=&amp;amp;amp;amp;amp;amp;amp;amp;prev_page=1&amp;amp;amp;amp;amp;amp;amp;amp;english=y&amp;amp;amp;amp;amp;amp;amp;amp;subjectid=6006211">soviet union (former nation/state/empire)</term> <term xml:id="poland" xhtml:href="http://www.getty.edu/vow/tgnfulldisplay?find=poland&amp;amp;amp;amp;amp;amp;amp;amp;place=&amp;amp;amp;amp;amp;amp;amp;amp;nation=&amp;amp;amp;amp;amp;amp;amp;amp;prev_page=1&amp;amp;amp;amp;amp;amp;amp;amp;english=y&amp;amp;amp;amp;amp;amp;amp;amp;subjectid=7006366">poland (nation)</term> </keywords> <keywords scheme="http://en.wikipedia.org"> <term xml:id="soviet_war_in_afghanistan" xhtml:href="http://en.wikipedia.org/wiki/soviet_war_in_afghanistan">soviet war in afghanistan</term> </keywords> ... <div type="qa" xml:id="q39"> <u who="#q"> <seg xml:id="para80" smil:begin="00:27:04.788" smil:end="00:27:20.138"> because he wasn't re-elected in <date when="1980">1980</date>, a lot of people thought his <name type="org" ref="#carter_presidency">presidency</name> had been a failure.  certainly on foreign policy we can look at some successes.  tell me about some of those.  what do you think <name type="person" ref="#jimmy_carter">carter</name> will be remembered for?  what are the great central accomplishments? </seg> </u> <u who="#a"> <seg xml:id="para81" smil:begin="00:27:20.138" smil:end="00:28:15.328"> well, .. i think, quite a few.  we in our conversations mentioned <name ref="#human_rights">human rights</name>.  i think that was a very major accomplishment.  it revived the american ideal worldwide.  <name ref="#camp_david_accords">camp david</name>, the first peace ever in the <name type="place" ref="#middle_east">middle east</name>.  normalization of relations with <name type="place" ref="#china">china</name> and a strategic relationship with <name type="place" ref="#china">china</name> at the high point of the <name ref="#cold_war">cold war</name>, when the <name type="place" ref="#soviet_union">soviet union</name> seemed to be cresting.  deterrence of the soviet invasion of <name type="place" ref="#poland">poland</name> in <date when="1980">1980</date>, when the soviets were about to march into <name type="place" ref="#poland">poland</name>, and <name type="person" ref="#jimmy_carter">carter</name> took a very strong stand.  tough-minded response to the soviets in <name ref="#soviet_war_in_afghanistan">afghanistan</name>, which bogged the soviets down for years.  there're many others.  strategic agreement with the russians.  although it didn't go terribly far, at least maintained that process of strategic normalization.  and i'm sure i have skipped some, but that's not a bad list already. </seg> </u> </div> to make this information useful, we need to add it to a search index. to do so, we create a master pbcore record that combines the basic descriptive information with technical metadata and the terms from the tei document using xslt documents and the fedora relationships. we developed a mapping for each keyword type to an appropriate pbcore element. figure 1: metadata aggregation for search indexing we are preserving the keyword urls for future use, with the idea that future linked data applications may be able to take advantage of them, or that data providers will offer a usable api to access additional contextual information. this, we hope, will provide a valuable entry point to the content for semantic web enabled technologies and also allow us to use inferencing to provide broader and related terms in our browsing and faceted search interfaces. as a result of our in-depth cataloging, we can do some very practical things, such as improving search results and providing relevant facets. perhaps more interestingly, we can also do new and creative mashups of the data. these mashups, we feel, may provide a method for “serendipitous discovery” of our materials for the general public and  encourage scholarly researchers to explore our material in a less structured manner. as a starting point, we decided to explore abstract graph-based visualizations to help demonstrate the depth of materials within the collection and the complexity of internal relationships between them. put another way, we are attempting to answer the  questions: “what is the scope of your collection?” and “i am interested in this item; what else in your collection might be worth my time?” the technology for our prototype, we experimented with two different types of visualizations: a force directed graph and a radial graph. because of the diversity and expanse of the collection, we wanted to work with algorithms that created the graph independent of editorial control. both layouts use the concepts of nodes and edges that connect them. the nodes are our descriptive records and the edges are the explicit metadata connections. to keep things simple, we have assumed there is a central node as a starting point, which also gives our users a starting point for their exploration. to quickly begin using these visualizations, we picked two javascript libraries, jsviz (http://jsviz.org) and the javascript information toolkit (jit) (http://thejit.org) , that accept json data to draw these graphs. to generate the dataset, we employed a very simple algorithm to gather up a set of related records: retrieve the metadata for the central record and create a list of keywords retrieve a new set of data that shares those keywords repeat step 2 until sufficient data has been gathered (from brief testing, we decided repeating this process twice provided a good balance between an overload of information and sufficient context to encourage discovery) this collection of records is serialized into json formats and used by the respective layout engines. force directed graphs in a force directed graph, algorithms are used to calculate layouts that conform to a set of rules. at their most basic level, these rules define a physical simulation of the attraction of connected nodes and the repulsion of others, often expressed as analogous to electrical forces and spring systems. in figure 2, the graph indicates that the central interview with robert mcfarlane (blue border) contains discussions of the vietnam war, the cold war, marxism, and terrorism. other interviews also discuss these topics, and the proximity of their icons to mcfarlane’s, indicate how many topics their interviews share in common. figure 2: force directed graph we use the jsviz javascript library to draw these graphs. this also gave us the ability to add the “strength” of a relationship as a determining factor of the layout, which affects the proximity of two records and visually indicates relevancy. the biggest technical weakness in force-directed graphs, however, is they do not perform well with larger datasets because of the calculations needed to display them. radial graphs radial graphs are much simpler because everything is positioned on spokes of a fixed length, so that the records are arranged on a series of concentric circles. the closer the record is to the center indicates its similarity to the central record. figure 3 contains the same content as figure 2, however, placement of related assets is restricted by the layout algorithm, resulting in a more structure appearance. figure 3: radial graph we are using the javascript information toolkit (jit) to draw these graphs. because the development of this graph is less computationally expensive, it is possible to add additional interactive features such as dynamic morphing (see figure 4), which has been shown to reduce confusion when the graph is re-oriented [6]. while proximity is no longer a dimension, user testing has shown the organizing structure makes visual scanning of the collection easier. figure 4: re-centered map conclusion and continued experimentation we are continually testing the utility of the relationship map with our users. in general, they are excited by the idea of visually browsing the relationships between items in the collection. our current instance allows them to browse by topic, people and context or source. we have found that upon first encountering the map, users generally understand the basic idea and purpose but require more information in order to make use of it. they have found an abstract visualization a bit foreign but quickly get acclimated once they start exploring. we have tried to integrate it into the search results function of our site but they are highly resistant to this. so far, they prefer using the map as a tool for browsing and exploration but do not trust or feel comfortable using it as a navigation tool to find specific items within collection. segmented assets our original conceptualization of the map included segmentation functionality within each asset. we hope to implement this functionality in the next revision of our site. this segmentation or “pie chart” functionality would indicate to the user the amount of content within a particular asset pertaining to a certain topic, place, etc. this will require the log or transcript of the asset to be segmented into sections and tagged at a higher level. figure 5: segmented assets additional cataloging is labor intensive but allows for additional functionality: this is a constant theme for the project. we are keeping our eye on whether the additional functionalities we experiment with are worth the effort in relation to the benefit to our users. finally, once we have invested the time to catalog the materials to the standards required for robust digital access, our developer can easily add new features in ways that are useable in other linked data applications.[7] for example, a user could possibly create a tag cloud from their search results in order to see the prominence of certain terms or places: figure 6: tag cloud users could also utilize the cataloging data to create a map of their found set and discover the locations where their archival items were created. cambridge community television in cambridge, massachusetts has built a similar function illustrating their neighborhood stories: figure 7: cctv media map[8] although we are just at the beginning of our project, we are hopeful that this exploration of visual expressions of media collections will continue to inspire other library and archives content providers to consider the value of their cataloging work beyond traditional search and discovery. our work so far has convinced us of the value of in-depth cataloging of media materials as a requirement for flexible user interfaces. as archivists move beyond the library catalog and the finding aid and into the semantic web, we must exploit the information we have already created to allow a rich diversity of access points into our collections. the code for this project is available under an mit license. it can be accessed at launchpad.net under the project name bvault on (http://launchpad.net/bvault). our prototype is currently closed to the public but readers can see and manipulate an example radial graph implementation on the homepage at http://openvaultresearch.wgbh.org. notes [1] we presented this work at the new england chapter of code4lib: “semi-controlled-folksonomic-tagging-vocabulary: encouraging useful metadata contributions” on december 9, 2008. see also an example of a controlled vocabulary tagging scheme using freebase’s api: http://mqlx.com/freebase-suggest/examples/suggest_demo_tagging.html [2] see “serendipitous information retrieval” by elaine g. toms in proceedings of the first delos network of excellence workshop on information seeking, searching and querying in digital libraries, http://www.ercim.org/publication/ws-proceedings/delnoe01/3_toms.pdf [3] for a discussion of the trade-offs between broad vs. item level description of moving image collections, see andrea leigh’s “context! context! context! describing moving images at the collection level” in the moving image, 6.1, spring 2006. [4] the pbcore schema is available at http://www.pbcore.org/index.html. [5] ebucore is available at http://tech.ebu.ch/lang/en/metadataebucore [6] yee, k. p., d. fisher, r. dhamija, and m. a. hearst (2001). animated exploration of dynamic graphs with radial layout. http://citeseer.lcs.mit.edu/yee01animated.html [7] much of this work was inspired by a presentation on data visualization at jcdl 2008: “shakespeare, god, and lonely hearts: transforming data access with many eyes” available at http://doi.acm.org/10.1145/1378889.1378914. the manyeyes project encourages users to play with, and discover new insights into often familar data. we hope our scholar researchers will be able to discover new paths into our content and gain a better understanding of our collection through interacting with our visualizations.” [8] cambridge community television, media map: http://www.cctvcambridge.org/mediamap acknowledgments this work was funded in part by the andrew w. mellon foundation and the institute for museum and library services. the authors also wish to thank karen cariani and peter pinch, co-principle investigators, and our advisors: drs. john dower, peter winn, james blight and janet lang. about the authors chris beer is a developer at wgbh interactive. he can be reached at chris_beer@wgbh.org. courtney michael is a project manager at the wgbh media library and archives. she can be reached at courtney_michael@wgbh.org. mayo todorovic is a designer at wgbh interactive. she can be reached at mayo_todorovic@wgbh.org subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – editorial introduction: the code4lib journal experiment, rejection rates, and peer review mission editorial committee process and structure code4lib issue 10, 2010-06-22 editorial introduction: the code4lib journal experiment, rejection rates, and peer review code4lib journal has been a successful experiment. with success, questions have arisen about the scholarly nature and status of the journal. in this editorial introduction we take a look at the question of code4lib journal’s rejections rates and peer review status. by edward m. corrado coordinating editor, issue 10 welcome to issue 10 of code4lib journal. when code4lib journal was first discussed a number of years ago, i thought it was unlikely it would ever come to fruition (even if i was among the first group of people to discuss the possibility during a breakout session at the first code4lib conference in february 2006). it took almost two years years after the journal was first discussed before issue 1 was published in late december 2007, with jonathan rochkind serving as coordinating editor. those of us who read code4lib journal owe jonathan a great deal of debt. while many of us talked about creating a journal, in the code4lib way, jonathan decided to do something about it; he oversaw the formation of an editorial committee, became our first coordinating editor, and put code4lib journal on the track to become what it is today. the code4lib journal experiment in his editorial introduction to issue 1, jonathan described code4lib journal as an experiment [1]. code4lib journal was indeed an experiment, and continues to be an experiment today. since the beginning of 2010, we have had over 21,000 unique visitors to our journal and approximately 56,500 page views from people located in over 140 countries/territories (as defined by google analytics). we are receiving more and more article proposals. articles are being cited in other journals. while code4lib journal continues to be an experiment, i think it is safe to say that thus far it has been a successful experiment. code4lib journal and journal rankings with this success, code4lib journal’s profile has grown and questions about where the journal fits in terms of scholarship have arisen. being a highly-ranked prestigious journal has never been a goal of the journal – at least not in my mind. we have always been more concerned with with being a useful resource and communication venue for the code4lib community than we have been about external rankings. however, with the promotion and tenure process at many academic libraries, i can appreciate why prospective authors would be concerned about this. the two most common questions i have seen in this regard relate to rejection rates and peer review status. these are difficult questions and depending on the context, are of limited, if any, usefulness when describing code4lib journal. rejection rates? the code4lib journal editorial committee has discussed rejection rates on more than one occasion. collectively we have decided not to attempt to calculate these rates. one reason is logistical. we haven’t necessarily kept track of all rejections in the past, making it an involved process to go back retroactively to calculated the percentage of rejections. however, there is a larger reason than this. there are difficulties and disagreements about what exactly constitutes a rejection and what exactly is a submission in the context of code4lib journal. there are two times in the publication process when an article is formally voted on (although this can also happen at other times as well). the first is when a proposal is submitted. the second is when the final article is being reviewed for publication. are rejections at either stage a rejection, or only the vote on a final article? the editorial committee does not ask for full articles, so what exactly are we rejecting in the proposal stage? if we reject a one or two sentence proposal, should that count as a rejection? if not, should it even count as a submission? is it a rejection if we ask an author to make changes that she does not agree with and thus decides to publish her article elsewhere? if someone sends us ten complete articles that are clearly off topic, is that ten more rejection submissions that we can use to pad our numbers? there is another important factor to consider with code4lib journal rejection rates. the editorial committee does everything we possibly can to not reject articles after we accept an initial proposal. while we have not published every article that made it past the proposal stage, some of the editors have gone way beyond the call of duty to work with authors to get an article to a point that we believed it was appropriate for our journal. i have seen some editors put so much work into articles published in past issues that they almost should have been listed as a coauthor. or put another way, unlike what traditional journal ranking metrics assume, our goal is to have a really low rejection rate, not a high one. peer reviewed? “is code4lib journal a peer reviewed journal?” is another difficult question to answer. according to code4lib journal guidelines, “we will generally use an editorial process, not a blind peer review process.” [2] code4lib journal obviously doesn’t follow a traditional double-blind peer review process. however, according to merriam-webster, peer review is defined as “a process by which something proposed (as for research or publication) is evaluated by a group of experts in the appropriate field.” [3] while i haven’t researched other journals, i think it is a fair assumption that the articles published in code4lib journal have had as many, if not more, “experts in the appropriate field” reviewing them compared to articles published in other library and information science (lis) journals. anecdotal evidence from some members of the code4lib journal editorial committee who have published in other peer reviewed lis journals suggest that typically there are two peer reviews and comments from the editor. an editor may also bring in a third reviewer if there are questions or disagreements about the article after receiving comments from the first two peer reviewers. this means in our experience there has been between two or three peer reviews and an editor evaluating any particular article. this is by no means conclusive, but i believe it is pretty typical of the field and perhaps true across other disciplines as well. code4lib journal currently has sixteen editorial committee members, fourteen of whom work on articles. while not all fourteen review every article, many articles are reviewed by over half the editorial committee – an editorial committee which is made up of experts in the field. so while code4lib journal does not have a double-blind peer review process and we claim that we use an “editorial process”, our process clearly fits the dictionary definition of peer review. will this match your institution’s definition of peer review? i can’t answer that, but you can point your promotion and tenure committee to our process and structure web page [4] for more details about our approach to journal publishing and let them decide. notes and links [1] rochkind, jonathan (2007). editorial introduction — issue 1. code4lib journal issue 1, december 2007, retrieved 16 june 2010 from http://journal.code4lib.org/articles/39 [2] see the “editorial process” section of: process and structure. (2010). in code4lib journal. retrieved 16 june 2010, from http://journal.code4lib.org/process-and-structure [3] peer review. (2010). in merriam-webster online dictionary. retrieved 19 june 2010, from http://www.merriam-webster.com/dictionary/peer review [4] process and structure. (2010). in code4lib journal. retrieved 16 june 2010, from http://journal.code4lib.org/process-and-structure subscribe to comments: for this article | for all articles 3 responses to "editorial introduction: the code4lib journal experiment, rejection rates, and peer review" please leave a response below: jakob, 2010-06-22 congratulations to the 10th code4lib journal! i strongly agree with your description of the rejection rates and review process. an addition you may want to experiment with an open proposal section and/or open submissions – but having a fairly open editorial committee is even better. thanks to all of you :-) laurie, 2010-11-11 i admit i am one of the tenure-track librarians who just found this page in a search for information for my dossier. aware that the journal is not ranked and not peer-reviewed, i’m looking for qualitative information – has the journal, as a whole, received praise from prominent leaders or publications? jodi schneider, 2010-12-20 laurie, this question received some discussion on the public listserv. that may be a good place to follow up! leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – piloting a homegrown streaming service with iaas mission editorial committee process and structure code4lib issue 42, 2018-11-08 piloting a homegrown streaming service with iaas bridgewater state university’s maxwell library has offered streaming film & video as a service in some form since 2008. since 2014 this has been done through the use of the infrastructure as a service (iaas) cloud provider amazon web services (aws) and their cloudfront content delivery network (cdn). this has provided a novel and low-cost alternative to various subscription and hosted platforms. however, with cloudfront’s reliance on external media players and flash via adobe’s real-time messaging protocol (rtmp) to stream content, the upcoming end of support for flash in 2020, and other security and accessibility concerns of library staff, an alternative method of delivery for this extremely popular and successful service was sought in summer and fall of 2017. with budget limitations, a flawed video streaming service currently in place, and university it’s desire to move much of its infrastructure to the iaas and cloud provider, microsoft azure, a pilot of a secure, multi-bitrate html5 streaming service via azure media services was conducted. this article describes the background of maxwell library’s streaming service, the current state of streaming services and technologies, azure iaas configuration, implementation, and findings. by robert t. wilson and ellen dubinsky introduction & background maxwell library serves approximately 10,000 students of bridgewater state university (bsu). in addition to the traditional library services offered to students, faculty, and staff of the university, the library began collaborating with the campus it services to provide streaming video content in the early 2000s. in the fall of 2008, the library rolled out maxwell video streaming (mvs), taking on full responsibility for administering the service. per the library director’s message, disseminated in the library’s fall 2008 newsletter, “[a]lthough not technically a new service, the library is now managing the entire process: from requests, to verifying copyright compliance, to streaming content, to providing links, to organizing directory structures, to cleanup and maintenance issues.” the library purchased a helix universal server, which was then administered by the campus it systems group, in order to stream realplayer (.rm) files. though the library provided the monies for the server, other campus entities, such as the radio station, jumped on the opportunity to use the helix server as well. students in classes with streaming content were required and instructed how to download realplayer to their computers. at that time students were required to have a laptop computer. the library’s acquisition department purchased video content with the appropriate streaming rights, taking care to follow legal guidelines. the circulation staff had to carefully monitor the streaming of personal videos (titles owned by faculty but not the library), limiting the viewing window to ten days, and not allowing reuse in subsequent semesters for the same course. the limitation to a ten-day window was based on the library director’s interpretation and consideration of both the teach act (2002) and the doctrine of fair use (17 u.s. code §107) [1]. that interpretation may have been faulty, but it was the policy adopted when the mvs service was started. the limitations of the short viewing window and re-use served as a catalyst for faculty to request the library purchase those video titles with appropriate streaming rights. streaming content from materials owned by the library were usually made available for the length of the semester. at the end of each semester, the staff moved these files from the helix server to another server for “archiving” and possible later reuse. streaming file versions of personal copies of films were removed from the helix server and were not archived. the library circulation staff was charged with transcoding video (vhs tapes and dvds) to the realplayer file format. the urls for streaming content were shared with faculty via email. the intention was that faculty would embed the urls within their learning management system (lms), blackboard or moodle, and class sites. in reality, faculty shared the urls via email, personal websites, and other channels. though the library staff was cognizant of copyright restrictions and the potential risk of infringing u.s. copyright law, there were no safeguards in place to restrict streaming video content access to just the registered and authenticated students in any class. additionally, faculty (especially part-time faculty who taught classes at multiple regional higher education institutions) were suspected of sharing the bsu library’s streaming content urls freely with non-bridgewater classes. it did not take long for staff within the library to express serious concern about potential copyright infringement and the risks that the mvs service presented. in april 2008, three academic publishers had brought suit against georgia state university for “pervasive, flagrant and ongoing unauthorized distribution of copyrighted materials” through the library’s e-reserve system (burtle [updated 2017]). while the georgia state case involved fair use and electronic reserves, fear spread throughout u.s. academic libraries that publishers and distributors would aggressively start pursuing infringement cases or focus on other library services (smith 2011) [2]. to fully protect the library (and the university) from a copyright infringement claim, we were concerned that: streaming content be restricted to only the authenticated students enrolled in a particular course; streaming content be available only during the applicable academic semester; streaming versions of personal-copy material be available only during a ten-day window and only for one semester; users not be able to download and save streaming video files; and urls leading to the streaming content not be discoverable or shared by users. any of the above conditions might put the library in the precarious position of having reproduced or distributed the copyrighted content, two of the explicit entitlements of a copyright holder per 17 u.s. code § 106 [3]. the provision of a streaming service itself (and the showing of films and videos in their entirety) could be defended as within fair use [4], but unrestricted distribution was indefensible. current state & need for change within two years of the launch of the mvs service, several members of the library staff, the circulation staff, the library director, systems librarian, and digital services librarian, were actively investigating improvements. usage was increasing as more faculty were requesting streaming content for their courses. concerns regarding copyright infringement were growing and the problems with emailing urls, restricting access to only students enrolled in a particular class, low quality video resolution, and management of the ten-day streaming windows for personal videos (not library-owned content) were not going away. at that time (2010-2011), the library and it tried to collaborate on the planning and implementation of a better streaming service – one with better copyright protections, better video quality, platform agnostic files, etc. however, at that time relations between the library and campus it were not always productive and meetings were often contentious. little progress was made during two plus years of meetings. in 2013, after some personnel changes within campus it, productive meetings resumed. campus it not only proposed using an outside vendor to offer solutions but also agreed to fund a pilot project. greystone solutions was contracted during the summer of 2013 and testing of several options began within months. by april 2014, a project plan was in place and testing began on a workflow using amazon web services (aws). mp4 video files were uploaded into an aws simple storage service (s3) bucket. staff could mark individual files as “public” and in the case of personal-copy video content, manually set start and end dates for public accessibility at the appropriate times. the files were then made available for playback through aws cloudfront distribution. at the end of the semester, files could be marked inactive and moved from active bucket into an archive bucket for possible use in subsequent semesters. in fall 2014, maxwell library went live with aws with many video files being re-transcoded at a higher quality. though an improvement over the previous service in terms of video quality and file management, copyright concerns remained. library staff had hoped to gain direct access to the courses within blackboard so that the urls could be directly embedded into the appropriate course sites. access to the video content would be limited to only the students enrolled in each particular class since the class rosters were automatically populated in blackboard at the start of each semester. in theory, url sharing by email would be unnecessary. however, direct embedding of the videos never occurred because the library staff never gained access to blackboard courses. many faculty used personal websites or another lms (such as canvas) for their courses and continued requesting the urls for the streaming content. even when faculty did embed the video urls into their blackboard course sites, an unforeseen problem arose: the default video player in blackboard allowed downloading of the video files and in some browsers allowed accessing the url of the video content. attempts to embed an alternative video player (such as jwplayer [5]) into blackboard were unsuccessful. with the arrival of a new emerging technologies and systems librarian in the spring of 2016, the inherent problems of the mvs streaming service were again investigated. to address this, urls for video content were placed behind the ezproxy server, thus partially restricting access to authenticated bsu students. additionally, in an effort to further obscure the urls, faculty were provided urls generated by the url shortener tinyurl [6]. lastly, the settings for the aws cloudfront cdn distribution server were adjusted to restrict access depending on geographic location of a user’s ip address, but those settings were not granular or customizable without additional customization and development which were resources not available to the library. these changes mitigated some of the issues and concerns, but did not fully resolve main concerns. to prompt a more earnest review of other providers, soon after these changes were introduced adobe announced an official end of life for flash in 2020 (warren 2017) which further affected the long-term viability of mvs configuration as streaming video content relied on an adobe real-time messaging protocol (rtmp) or adobe flash media server distribution through cloudfront [7]. depending on the browser and machine flash settings, streaming content through rtmp as well as with jwplayer or alternatives successfully was extremely inconsistent. with no indication from aws that it was developing an html5 media player or something less dependent on flash, seeking yet another alternative became necessary. identifying an alternative while so many solutions and acquisition models for streaming video now exist (ferguson and erdmann 2016), maintaining control and ownership of the service was still a goal. so, in seeking alternatives, that model was front and center. aws’s cloudfront has been the premier method of streaming video as far back as 2009 (bouthillier 2010), but plenty of competition has sprouted up in the years since. additionally, streaming technology has come a long way. flash as the dominant format of playback has been waning as devices used for video playback have changed and as html5 has become more widespread. standards and technologies utilizing html5 have become more prominent. in regards to streaming, dynamic adaptive streaming over http (dash) which is an iso standard for adaptive streaming intended to replace proprietary technologies like adobe’s rtmp (ozer 2011) was appealing.  identifying a service that made use of dash versus a proprietary technology or flash-reliant one would be an ideal solution. the requirements in regards to resolving copyright concerns were still clear on what the ideal service should do, but now there were additional caveats regarding technology that affected long-term viability and user experience. the configuration in aws only met our needs relating to its low cost and simple staff workflow. in reviewing comparable services, microsoft’s azure media services [8] and vimeo pro [9] stood out as the most viable in addressing library copyright and streaming technology needs and in having large enough communities to draw knowledge and support from if necessary. in testing, azure media services not only contained its own html5 media player supporting dash but allowed a more robust set of configurations and options for customization than vimeo pro. while azure media services right out of the box can be used in a similar manner to how vimeo pro functions, the storage, web services, and api possibilities as well as the potential of tying into the full range of azure services made it possible to create a vimeo-like platform if needed or desired. in the end, vimeo is a hosted video playback and streaming service while azure and aws are  iaas providers vimeo or similar services may use as their infrastructure to offer their services. serendipitously, within a month of looking into azure as an alternative, campus it expressed their intention to migrate the university’s infrastructure to azure where feasible. being able to rely on their expertise and assistance if ever necessary made azure more appealing and less of an unknown than vimeo pro or further developing services through aws. azure media services configuration & pilot in summer 2017, with support from it and in the brief testing conducted, steps were taken to get up and running with azure. the steps were similar to aws s3, cloudfront, and previous jwplayer configuration work. first, a media service had to be created. once an azure account was created, creating a media service is necessary to stream content. a storage component is created at this time as well for files which are referred to as assets in azure. however, other storage in azure can be used. in aws, this is the equivalent of the s3 and cloudfront services, but unlike them, media services is a dedicated space for preparing and managing media content. next, a user must activate the streaming endpoint that is created when the media service is turned on. as far as the uploading assets and encoding for multi-bitrate streaming, this is all that is needed. there are options to use external media players as well as using .net or rest for uploading and encoding assets. these were not explored in the pilot, but suggested how robust the environment could be. the final prerequisite was using an existing server of creating a cloud-instance server to host the html files for video playback. our particular server was a low specification linux ubuntu server with apache web services installed. access to the server’s http/https traffic was restricted to the library’s ezproxy server ip allowing the library staff to restrict access to the content to authorized faculty and students only by piggy-backing on the cas single-sign on (sso) authentication service ezproxy was using. as an added ux perk, blackboard was also using cas. so, user’s accessing content from a url in blackboard would already be logged into the authentication service. figure 1. comparison of aws cloudfront and azure media services configurations. once the services needed were in place, the manual workflow for adding and making content available was as follows:  upload an asset – in our case this was mp4 files.  encode the asset using the media encoding standard and adaptive streaming tool – this process depending on file size could take some time to complete. one can view the progress of in the jobs area of the media services dashboard. once the process is complete, a new asset will appear in the assets folder as multi-bitrate mp4.  upload captions – there are a few additional options for file management, but for accessibility and user experience purposes, being able to easily upload captions if the file was available was very appealing.  publish, select streaming, select start and end publish dates.  in the published urls section of the asset, view/copy the streaming url.  test playback – the embedded azure media player can be activated within the dashboard by selecting the asset and then play.  html file – once the video is playing as desired, library staff would then embed the copied streaming url into the video source area of an html file template provided below. the template could be customized similar to jwplayer comparable players with branding and sizing. azure media player html file example figure 2. example html for azure media player embedded streaming video. once the html file was named and saved, library staff would then upload to the mvs server via ftp/sftp. that url, once proxied, could be shared with faculty in whatever way the faculty member preferred. as previously mentioned, this server is only accessible via http/https from the ezproxy server ip. the result will be a file only accessible to users with valid institutional credentials and will look something like this: https://yourezproxyurl.edu/login?url=http://mvs.bridgew.edu/somevideo.html. for the pilot, it was determined that of the faculty utilizing the service, on average, they streamed five items each fall and spring semester. by using the average number of items streamed per faculty member (5) and the average number of faculty that use the service a semester (30), an estimate in costs per semester could be achieved. working with a faculty member teaching film studies courses who heavily uses mvs, five items of varying quality, length, and therefore file size, were converted to multi-bit streaming files for playback in the azure media player. working with one faculty member who is versed in the use of the service and familiar with past issues allowed for valuable qualitative feedback from the faculty member and their students regarding ease of access and user experience. findings & future work streaming via azure media services fully resolved security, access and management concerns by allowing for improved management of access by allowing start and end dates of when the file would be available which would control costs of streaming files and address copyright requirements for when and how long items are available as well as reduce workload on staff managing the service. as a bonus, being able to fully control access through ezproxy also had the added option to use groups to more granularly set user authorization either by course number or by a list of student usernames enrolled in a course. the system librarian checked in weekly with the faculty member who agreed to participate in the pilot to identify and address any issues experienced. there were no reports of issues in playback or access by faculty member or students, and according to the faculty member, the new method created a more comparable experience to consumer and subscription streaming services. no problems comparable to playback issues seen in aws related to flash or jwplayer were experienced. the items being available in multiple bitrates improved service for users with lower bandwidth available for video playback, and the service included an option to address always important accessibility needs like the ability to upload subtitle transcript files. while streaming through azure media service addressed many concerns of the library staff, uploading, converting, and creating an html file for playback added considerable time for processing each new video request where aws consisted of transcoding a file to mp4, uploading to s3, and making the url available in cloudfront. to fully implement the service, considerable upfront costs both in time, staffing, and fees generated by converting each file to multi-bitrate would be required. regarding fees, streaming costs were found to be much more expensive than current aws setup. the bulk of the cost appeared due to upload and encoding of the mp4 files to multi-bitrate mp4 files with the cost of streaming, asset storage, and our cloud web server for html files accounting for the rest. since the vast majority of our archive and any new requests would need to be uploaded and encoded, this suggested a very large upfront cost in time and money. it was not explored at the time of the pilot whether it was possible to encode the mp4 files to multi-bitrate mp4 files that were compatible for playback in azure media player before uploading into azure with third party tools like ffmpeg [10]. if possible, there would be an increased cost per upload as multi-bitrate files are larger in size than single-bitrate. however, this as an option would reduce the cost in encoding, and likely, the overall cost of the service. the total cost to operate the pilot was $295.00 for the fall 2017 semester streaming of five items. that number multiplied by the average number of faculty using the service was $8,850 per semester. that figure is multiple times costlier than the aws mvs configuration. with a large percentage of the cost due to uploading and converting the mp4 files, what would likely be seen is a shift in the cost breakdown from conversion to storage, but due to new items being purchased on a frequent basis, a significant reduction in cost would be unlikely. despite azure media services meeting our functional requirements and fully addressing long-held concerns, the cost projections as well as the additional cost in staff time per new item request were reason enough to cease the pilot and explore other options and services. though the cost relative to other streaming services is quite reasonable especially with how robust the service can be, it was a cost maxwell library was not prepared to fund at that time. shortly after the pilot was completed, a test plan was created to review vimeo pro which offers a flat subscription fee that, if found to provide the service, security, accessibility, and user experience required and demanded by faculty and students, may serve as a viable and perhaps more sustainable option due to the library’s budget and staffing. bibliography bouthillier l. 2010. how to get started with amazon cloudfront streaming. streamingmedia.com [internet]. [cited 2018 sep 1]. (coins) burtle l. [updated 2017 oct 29]. gsu library copyright lawsuit. [cited 2018 sep 1]. ferguson j, erdmann a. 2017. streaming video in academic libraries. american libraries magazine [internet]. cited 2018 aug 30]. (coins) ozer j. 2011. what is mpeg dash? streamingmedia.com [internet]. [cited 2018 aug 31]. (coins) smith, k. 2011. a nightmare scenario for higher education. scholarly communications @ duke [internet]. [cited 2018 sep 2]. (coins) warren, t. 2017. adobe will finally kill flash in 2020. the verge [internet]. [cited 2018 sep 5]. available from:. (coins) notes [1] for more information about the teach act, see http://www.gpo.gov/fdsys/pkg/bills-107s487es/pdf/bills-107s487es.pdf.  for more information about fair use, see https://www.law.cornell.edu/uscode/text/17/107. [2] for more information, see “what’s at stake in the georgia state copyright case.” the chronicle of higher education. [3] available here:  https://www.law.cornell.edu/uscode/text/17/106). [4] for more information, see the society for cinema and media studies’ statement of best practices for fair use in teaching for film and media educators. [5] available here: https://www.jwplayer.com/video-solutions/html5-video-player/ [6] available here:  https://tinyurl.com/ [7] for more information see: https://docs.aws.amazon.com/amazoncloudfront/latest/developerguide/distribution-rtmp.html [8] available here: https://azure.microsoft.com/en-us/services/media-services/ [9] available here: https://vimeo.com/ [10] available here: https://www.ffmpeg.org/ about the authors robert t. wilson (robert.wilson@mtsu.edu) is assistant professor, systems librarian at middle tennessee state university. he is formerly the emerging technologies & systems librarian at bridgewater state university’s maxwell library. http://rtwilson.info ellen dubinsky (edubinsky@email.arizona.edu) is the associate librarian, scholarly communications at the university of arizona. she is formerly digital services librarian at bridgewater state university’s maxwell library. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – the semantics of metadata: avalon media system and the move to rdf mission editorial committee process and structure code4lib issue 37, 2017-07-18 the semantics of metadata: avalon media system and the move to rdf the avalon media system (avalon) provides access and management for digital audio and video collections in libraries and archives. the open source project is led by the libraries of indiana university bloomington and northwestern university and is funded in part by grants from the andrew w. mellon foundation and institute of museum and library services. avalon is based on the samvera community (formerly hydra project) software stack and uses fedora as the digital repository back end. the avalon project team is in the process of migrating digital repositories from fedora 3 to fedora 4 and incorporating metadata statements using the resource description framework (rdf) instead of xml files accompanying the digital objects in the repository. the avalon team has worked on the migration path for technical metadata and is now working on the migration paths for structural metadata (pcdm) and descriptive metadata (from mods xml to rdf). this paper covers the decisions made to begin using rdf for software development and offers a window into how semantic web technology functions in the real world. by juliet l. hardesty and jennifer b. young introduction avalon media system (avalon) is “an open source system for managing and providing access to large collections of digital audio and video.” (the project [updated 2017]) in addition to playback capabilities and audio and video file management, avalon is a search and discovery interface. through metadata, audio and video items can be discovered and accessed based on people involved, subjects, dates, and genres along with any keyword-indexed content from titles, abstracts, and notes. this metadata has been managed as mods xml, a bibliographic xml standard that offers a more human-readable and shareable version of the marc library catalog record, and is a reflection of the libraries and archives focus of avalon media system as a software product. technical, structural, and some administrative metadata have also been managed using xml but not based on any specific xml standard. the xml created and used for these functional areas has been customized and used internally by avalon with the samvera community (formerly the hydra project) software stack using the fedora commons digital repository, version 3.x. [1] avalon’s move to update to the latest version of the fedora digital repository software, version 4.x, means a significant shift in working with metadata. fedora 4 is a major reworking of the digital repository software, particularly regarding increased linked data capabilities. instead of xml files containing different kinds of metadata information associated with a binary file (the “digital object”), rdf properties are stored together with the digital object (technically still in a text file associated with the digital object but available more directly through the repository software than in the previous environment). these properties can be a mix of ontologies that are needed to express necessary information about a digital object and they are stored together with the properties fedora tracks on its own about digital objects and relationships between objects. this significant change means multiple ontologies can be used including local ontologies. choices have to be made on how to use more complex ontologies that involve hierarchical rdf statements (where the object of a subject->predicate->object statement is an entire subject->predicate->object statement) since that can mean creating and managing objects in the repository that are not actual digital objects but rather a concept, like a subject or a person. for avalon the goal is to continue providing the same functionality and metadata as this shift occurs from xml to rdf. the following describes the process so far and the plans to move forward and enhance avalon’s metadata capabilities through the use of semantic web technologies. migrating avalon metadata from fedora 3 to fedora 4 and from xml to rdf is part of upgrading avalon versions so changes described here that have already occurred for technical and structural metadata and changes described here that will occur for descriptive metadata will be part of going through the avalon upgrade process. figure 1. diagram representing technical, structural, and descriptive metadata on avalon objects in fedora 3. each blue, purple, and red box represents a separate xml file accompanying the repository object. avalon technical and structural metadata in fedora 4 as part of the work that has already occurred for avalon release 6, avalon now works with fedora 4. technical metadata is now being stored as rdf properties on avalon objects instead of in an xml file. where custom xml defined file location, size, format, duration, checksum, and date digitized and was only usable within the context of avalon, that information is now handled as rdf properties defined using established ontologies for those properties (ebucore, premis, and library of congress identifier types) along with a few locally-defined properties for internal use. figures 2-6 show the process of storing technical and structural metadata as rdf properties instead of flat xml. <fields> <file_format>moving image</file_format> <file_location>/path/to/file/avalon_7089-1797.mov</file_location> <duration>43099</duration> <display_aspect_ratio>1.3333333333333333</display_aspect_ratio> <original_frame_size>720x486</original_frame_size> <poster_offset>8353</poster_offset> <thumbnail_offset>8353</thumbnail_offset> <file_size>735569993</file_size> <date_digitized>2014-12-08t19:03:59z</date_digitized> </fields> figure 2. a fedora 3 example of avalon technical metadata for a video object as custom xml. this is represented as the blue “tech” boxes in figure 1. @prefix fedoraobject: <http://[localfedorarepository]/>. @prefix ldp: <http://www.w3.org/ns/ldp#>. @prefix fedora: <http://fedora.info/definitions/v4/repository#>. @prefix ebucore: <http://www.ebu.ch/metadata/ontologies/ebucore/ebucore#>. @prefix avalon: <http://avalonmediasystem.org/rdf/vocab/master_file#>. @prefix premis: <http://www.loc.gov/premis/rdf/v1#>. fedoraobject:90/19/s2/51/9019s251h a ldp:rdfsource, ldp:container, fedora:container, fedora:resource; ebucore:aspectratio "1.3333333333333333"; ebucore:duration "43099"; avalon:posteroffset "8353"; avalon:thumbnailoffset "8353"; ebucore:filesize "735569993"; ebucore:filename "file:///path/to/file/avalon_7089-1797.mov"; ebucore:height 486; ebucore:width 720; ebucore:locator "/path/to/file/avalon_7089-1797.mov"; premis:hasformatname "moving image"; ebucore:datecreated "2014-12-08t19:03:59z". figure 3. the same technical metadata as in figure 2 rendered as rdf properties in fedora 4 avalon. please note that the ontology namespace prefixes are provided for easier reading. these technical properties can be mapped from avalon’s custom xml to simple rdf statements and do not require a modification in value ranges (i.e., properties that are defined with text values in xml can be mapped to properties that still use text values, also called literals, in rdf). many of these values (duration, aspect ratio, height, width) are audio and video specific so ebucore from the european broadcasting union was a useful ontology to express those properties. mapping recommendations for more general technical metadata properties have been made within the samvera community for common properties such as filename and file size and those recommendations were followed as closely as possible. structural metadata in avalon with fedora 3 used rdf/xml to express connections between objects based on the rels-ext relationships that fedora defines. <rdf:rdf xmlns:ns0="http://projecthydra.org/ns/relations#" xmlns:ns1="info:fedora/fedora-system:def/model#" xmlns:ns2="info:fedora/fedora-system:def/relations-external#" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"> <rdf:description rdf:about="info:fedora/avalon:22203"> <ns0:hasmodelversion>r3</ns0:hasmodelversion> <ns1:hasmodel rdf:resource="info:fedora/afmodel:masterfile"/> <ns2:ispartof rdf:resource="info:fedora/avalon:22202"/> </rdf:description> </rdf:rdf> figure 4. the rels-ext relationship in fedora 3 as rdf/xml for an avalon masterfile object (avalon:22203). this connects the masterfile to the avalon mediaobject (avalon:22202) that contains the masterfile. this is represented as the purple “rels-ext” box on the masterfile and the “ispartof” arrow in figure 1. <rdf:rdf xmlns:ns0="http://projecthydra.org/ns/relations#" xmlns:ns1="info:fedora/fedora-system:def/model#" xmlns:ns2="info:fedora/fedora-system:def/relations-external#" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"> <rdf:description rdf:about="info:fedora/avalon:22202"> <ns0:hasmodelversion>r4</ns0:hasmodelversion> <ns0:isgovernedby rdf:resource="info:fedora/avalon:966"/> <ns1:hasmodel rdf:resource="info:fedora/afmodel:mediaobject"/> <ns2:ismemberofcollection rdf:resource="info:fedora/avalon:966"/> </rdf:description> </rdf:rdf> figure 5. the rels-ext relationship defined on the avalon mediaobject (avalon:22202) connecting it to an avalon admin_collection (avalon:966) in fedora 3 as rdf/xml. this is represented as the purple “rels-ext” box on the mediaobject and the “ismemberofcollection” arrow in figure 1. <rdf:rdf xmlns:ns0="info:fedora/fedora-system:def/model#" xmlns:ns1="http://projecthydra.org/ns/relations#" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"> <rdf:description rdf:about="info:fedora/avalon:966"> <ns1:hasmodelversion>r3</ns1:hasmodelversion> <ns0:hasmodel rdf:resource="info:fedora/afmodel:admin_collection"/> </rdf:description> </rdf:rdf> figure 6. the admin_collection (avalon:966) is defined as not being part of anything else – it is the top level. the relationships defined in avalon by fedora 3 start from the lowest level of the object – the digital file – and point up to the highest level – the collection. the collection does not contain any structural definition of the objects contained within it, and the objects also do not contain any structural definition of the files that make up those objects. at the mediaobject level avalon creates custom xml to define the files, or sections, that make up a single avalon item so that those are available more easily and have a specified order when accessing the mediaobject: <fields> <section_pid>avalon:22203</section_pid> <section_pid>avalon:22204</section_pid> <section_pid>avalon:22205</section_pid> </fields> figure 7. sections xml metadata on an avalon mediaobject in fedora 3. this is represented as the purple “sections” box in figure 1. in fedora 4 those rels-ext relationships remain as rdf properties on the masterfile and the mediaobject, but they are no longer stored as rdf/xml (as shown in figure 1) and are augmented by other ontologies that allow for more and bidirectional structural definitions (see figure 8). a masterfile is still part of a mediaobject, which is still a member of an admin_collection using those rels-ext rdf properties. dublin core’s haspart property is used on the mediaobject to identify the masterfile objects that are part of that mediaobject. additionally, the mediaobject is able to use what is called a list_source object to define an ordered list of objects that serve as proxies for masterfiles contained within the mediaobject and places them in a certain order (important for tracks on an album or acts in movie). the linked data platform (ldp) vocabulary and the internet assigned numbers authority’s (iana) ordering properties are both ontologies that fedora 4 can use to establish these connections from a higher level like the mediaobject down to the masterfiles contained within instead of only the structural direction from lowest level upwards. @prefix fedoraobject <http://[localfedorarepository]/>. @prefix fedora: <http://fedora.info/definitions/v4/repository#>. @prefix ldp: <http://www.w3.org/ns/ldp#>. @prefix avalonmigration: <http://avalonmediasystem.org/ns/migration#>. @prefix fedoramodel: <info:fedora/fedora-system:def/model#>. @prefix dcterms: <http://purl.org/dc/terms/>. @prefix fedorarelsext: <info:fedora/fedora-system:def/relations-external#> @prefix iana: <http://www.iana.org/assignments/relation/>. fedoraobject:tq/57/nr/06/tq57nr067 a fedora:container, ldp:rdfsource, ldp:container, fedora:resource; avalonmigration:migratedfrom"avalon:22202"; fedoramodel:hasmodel"mediaobject" ; dcterms:haspart fedoraobject:dr/26/xx/45/dr26xx45j, fedoraobject:nv/93/52/90/nv9352905, fedoraobject:2v/23/vt/45/2v23vt451; fedorarelsext:ismemberofcollection  fedoraobject:dj/52/w4/73/dj52w4734; ldp:contains <http://[fedoraaggregation]/tq/57/nr/06/tq57nr067/list_source>; iana:first <http://[fedoraobjectproxy]/tq/57/nr/06/tq57nr067/list_source#g284616160>;     iana:last <http://[fedoraobjectproxy]/tq/57/nr/06/tq57nr067/list_source#g309531160>. figure 8. structural rdf properties on an avalon mediaobject in fedora 4 showing that it has three parts (avalon sections or masterfiles). this bidirectional structure will be further enhanced as avalon implements the portland common data model (pcdm). pcdm is a domain model meant to provide an interoperable framework for working with repository objects. its current focus is providing structural metadata and access controls for better interoperability between repository applications. (portland common data model [updated 2016 aug 9]) fedora 4 does not automatically apply pcdm properties and relationships to objects, however, in addition to ldp containers and indirect containers, pcdm allows the definition of collections, objects, and files, providing another option for representing the structure of avalon objects and collections of those objects. derivatives, or compressed audio and video files that can be streamed online at different bandwidths, produced off of each masterfile are also stored as objects in fedora with high and medium qualities created for audio and high, medium, and low qualities created for video. each derivative is connected back to its masterfile object using a rels-ext relationship defined on the derivative object only, so there is no bidirectional connection. this single relationship maps over to an rdf statement on the derivative object in fedora 4. custom xml in fedora 3 defined technical properties like encoding qualities and file locations. where possible in fedora 4 ebucore properties were used to map this information to rdf and all values were mapped as literals (strings of text). collections and units have come across from fedora 3 to fedora 4 but the mapping is not ideal. at the mediaobject level, the same rels-ext rdf property is used to connect to the collection, which is its own repository object and can be referenced using a uri. @prefix fedoramodel: <info:fedora/fedora-system:def/model#>. @prefix loc: <http://id.loc.gov/vocabulary/identifiers/>. @prefix dcterms: <http://purl.org/dc/terms/>. @prefix bf: <http://bibframe.org/vocab/>. <fedoraobject:dj/52/w4/73/dj52w4734> a ldp:container, fedora:resource, fedora:container, ldp:rdfsource;    fedoramodel:hasmodel    "admin::collection";    loc:local    "avalon:966";    dcterms:title    "jsom performance";    bf:heldby    "music library". figure 9. rdf statements stored on a collection object in fedora 4. the least ideal portion of figure 9’s mapping is the handling of avalon units. units are the top level of structure within avalon but collections are the top level of objects in the repository (see figure 1). so units are used for grouping and access, but not handled as repository objects. we can still manage units this way in fedora 4, but to map them to explain what they are semantically in rdf means they need to be defined by a property at the collection level that shows the collection is contained in, is part of, or is held by the unit. the level of unit in avalon can be equivalent to something that is identifiable as a thing in an authorized controlled vocabulary, such as lcnaf or viaf (for example, indiana university school of music or northwestern university archives). a property with that sort of semantic meaning but requiring a uri value will work, which is why the fedora 4 mapping is set to “heldby” from the bibframe ontology. units in avalon are based on hard-coded configurations at this point and changes are still needed to change the value to a uri for fedora storage. additionally, this bibframe property is still pointing to the 1.0 version of the ontology, which has been replaced by bibframe 2.0. this mapping is still considered a work in progress. one structural definition that neither fedora 3 nor fedora 4 provide for is defining structure within a single digitized file (within one masterfile). audio and video files often have multiple pieces of content on a single digital file (multiple songs, multiple acts of an opera or performance, etc.). <item label="tape 1"> <span begin="0:0:0" end="0:1:0" label="track 1. [ambience]"/> <span begin="0:1:0" end="0:16:19" label="track 2. ludwig van beethoven, twelve variations on a theme from judas maccabaeus (handel)"/> <div label="johannes brahms, sonata in f major, op. 99"> <span begin="0:16:19" end="0:23:10" label="track 3. i. allegro vivace"/> </div> </item> figure 10. custom structural xml defining structure within a masterfile object, used in fedora 3 and fedora 4. this is represented as the purple “structure” box in figure 1. to enable this structural definition avalon uses another set of custom xml that is stored in both fedora 3 and fedora 4 as an xml file with the masterfile object, using the same custom xml. there are no rdf properties that can express the needed structure with layers of labels, start times, and end times defined within a single file. this custom xml is closely based on xml used for the same purpose in the variations digital music library system, a digital audio and score access system developed at indiana university that in many ways is the predecessor to avalon media system. mapping from xml to rdf for technical metadata has not resulted in much that takes advantage of linked data, but we do now use common ontologies for the properties describing that technical metadata instead of customized xml that could only be understood and used in the context of avalon. our structural metadata makes the same use of linked data relationships as it did before through fedora’s rels-ext connections from files up to objects up to collections, but it is also expanded to allow connections in the opposite direction. an external triplestore might not require such explicit property statements if a semantic reasoner is available, but it is helpful in fedora 4 to provide statements at the object level connecting to the files that make up the object. hard-coded units have the potential for semantic mapping but the implementation is not yet complete. descriptive metadata and avalon our analysis of descriptive metadata began by looking at the mapping recommendations from the samvera mods and rdf descriptive metadata subgroup. avalon’s current move to fedora 4 has kept our descriptive metadata stored as a flat mods xml file without rdf properties. figure 1 represents this xml file as the red “mods” box on the mediaobject. changing this mods xml to rdf is another step in the overall metadata migration process. in preparation for allowing our descriptive metadata to be stored as rdf properties with the digital object we created a spreadsheet mapping our current mods xml elements in comparison to the mods and rdf subgroup’s recommendations. [2] while a modsrdf ontology from the library of congress does exist, it is still a work in progress and often results in complex hierarchical rdf (where the object of one rdf statement is another rdf statement). this complexity would make storing descriptive metadata in fedora about a single digital object more difficult than considering a use of mixed ontologies to express the same meaning of the mods xml elements as simple rdf statements. another consideration is that avalon mods does not currently allow for reference identifiers to be easily added to our fields. this reality means for migration purposes using a single ontology would likely require more reconciliation of text values over to uri values. considering properties that can allow for text values means more flexibility in how we are mapping and thus involves a variety of ontologies. the samvera community is developing a new “open-source hydra-powered [sic] repository front end” called hyrax. hyrax is the merger of two samvera gems – sufia and curationconcerns. while avalon only allows for audio and video content to be deposited, hyrax is meant to be customizable to allow for other types of content as needed. current avalon development is moving towards integration with hyrax, which is built on fedora 4 and uses rdf properties for descriptive metadata. another step in mapping our descriptive metadata to rdf then was to consider the rdf properties that hyrax is using. however, our analysis of hyrax’s generic type of work and descriptive metadata mapping shows that only 48%, or 12 out of 25 of avalon’s descriptive metadata mods xml elements match. [3] <?xml version="1.0" encoding="utf-8"?> <mods xmlns:xsi="http://www.w3.org/2001/xmlschema-instance" xmlns="http://www.loc.gov/mods/v3" xsi:schemalocation="http://www.loc.gov/mods/v3 http://www.loc.gov/standards/mods/v3/mods-3-4.xsd"> <titleinfo usage="primary"> <title>freshwater jellyfish</title> </titleinfo> <name type="personal" usage="primary"> <namepart>indiana university, bloomington. audio-visual center</namepart> <role> <roleterm authority="marcrelator" type="code">cre</roleterm> <roleterm authority="marcrelator" type="text">creator</roleterm> </role> </name> <name type="personal"> <namepart>flaten, clarence m., 1910-1974</namepart> <role> <roleterm authority="marcrelator" type="code">ctb</roleterm> <roleterm authority="marcrelator" type="text">contributor</roleterm> </role> </name> <name type="personal"> <namepart>vuke, george j.</namepart> <role> <roleterm authority="marcrelator" type="code">ctb</roleterm> <roleterm authority="marcrelator" type="text">contributor</roleterm> </role> </name> <name type="personal"> <namepart>lytle, charles f.</namepart> <role> <roleterm authority="marcrelator" type="code">ctb</roleterm> <roleterm authority="marcrelator" type="text">contributor</roleterm> </role> </name> <name type="personal"> <namepart>indiana university, bloomington. audio-visual center</namepart> <role> <roleterm authority="marcrelator" type="code">ctb</roleterm> <roleterm authority="marcrelator" type="text">contributor</roleterm> </role> </name> <typeofresource>moving image</typeofresource> <genre>nature films.</genre> <genre>science films.</genre> <genre>educational films.</genre> <genre>nonfiction films.</genre> <genre>short films.</genre> <origininfo> <dateissued encoding="edtf">1978</dateissued> <publisher>indiana university audio-visual center</publisher> </origininfo> <language> <languageterm type="code">eng</languageterm> <languageterm type="text">english</languageterm> </language> <physicaldescription> <internetmediatype>video/quicktime</internetmediatype> </physicaldescription> <abstract>examines the life cycle of the freshwater jellyfish and describes its habitats, collection procedures, feeding behavior, and methods of reproduction.</abstract> <note type="statement of responsibility" altrepgroup="00">produced by indiana university audio-visual center.</note> <note type="creation/production credits">producers, clarence m. flaten, george vuke; consultant, charles lytle.</note> <note type="additional physical form">also issued as videocassette.</note> <subject> <topic>craspedacusta sowerbyi</topic> </subject> <relateditem type="original"> <physicaldescription> <form authority="marccategory">motion picture</form> <form authority="marcsmd">film reel</form> <form type="media" authority="rdamedia">projected</form> <form type="carrier" authority="rdacarrier">film reel</form> <extent>1 film reel (13 min.) : sound, color ; 16 mm</extent> </physicaldescription> <identifier type="lccn">79700435</identifier> <identifier type="oclc">ocm05892454</identifier> <identifier type="oclc">5892454</identifier> <identifier type="local">7830578</identifier> </relateditem> <location> <url displaylabel="digitized version" usage="primary display”>http://purl.dlib.indiana.edu/iudl/media/653702wt32</url> </location> <location> <url access="object in context">http://purl.dlib.indiana.edu/iudl/media/653702wt32</url> </location> <recordinfo> <descriptionstandard>rda</descriptionstandard> <recordcontentsource authority="marcorg">indiana univ. audio-visual center.</recordcontentsource> <recordcreationdate encoding="edtf">790718</recordcreationdate> <recordchangedate encoding="iso8601">2015-10-04t09:05:13-04:00</recordchangedate> <recordorigin>converted from marcxml to mods version 3.5 using marc21slim2mods3-5.xsl (revision 1.106 2014/12/19), customized for the avalon media system (last revised 2014/12/26)</recordorigin> <languageofcataloging> <languageterm authority="iso639-2b" type="code">eng</languageterm> </languageofcataloging> <recordidentifier source="local">7830578</recordidentifier> <recordidentifier source="fedora">avalon:6696</recordidentifier> </recordinfo> </mods> figure 11. example of a mods xml record stored with an avalon item in fedora 3. it does not represent every possible mods element that avalon records, but offers a representation of the xml elements with which we are working. we will now provide details on evaluations and decision points for avalon’s descriptive metadata fields as we consider taking them from mods xml and making them rdf properties stored with digital objects in fedora 4. we want to have the ability to use uris as much as possible instead of literal text to allow our metadata to be used as linked data. names @prefix dc: <http://purl.org/dc/elements/1.1/> . @prefix relator: <http://id.loc.gov/vocabulary/relators/> . avalon mods mods & rdf subgroup hyrax avalon mods/rdf name@usage=”primary”/namepart (role/roleterm set to “creator”) relator:cre dc:creator dc:creator name/namepart (role/roleterm set to “contributor”) relator:ctb dc:contributor dc:contributor   in order to use the mods and rdf subgroup mapping recommendation for names, we would either have to resolve the names for creator and contributor fields to uris or create local uris for names. unfortunately many of the names entered by just iu and northwestern as examples will not easily be resolved so we made the decision to deviate from the recommended mods and rdf subgroup mapping of relator:cre and relator:ctb to dc:creator and dc:contributor to allow for text values. hyrax also maps creator to dc:creator and contributor to dc:contributor. dates @prefix dcterms: <http://purl.org/dc/terms/> . date fields avalon mods mods & rdf subgroup hyrax avalon mods/rdf publication date origininfo/dateissued dcterms:issued dcterms:created dcterms:issued creation date origininfo/datecreated@encoding=”edtf” dcterms:created none dcterms:created   avalon currently uses origininfo/dateissued within mods to store the publication date for an item. the mods and rdf subgroup recommends mapping to dcterms:issued while hyrax uses dcterms:created. we will have to change how our dates are stored as well as update documentation and the sorting of the facets if we change to using hyrax’s mapping of dcterms:created. we are likely to try matching this to the mods and rdf subgroup recommendation of dcterms:issued for publication date and dcterms:created for creation date. language @prefix dc: <http://purl.org/dc/elements/1.1/> . @prefix dcterms: <http://purl.org/dc/terms/> . avalon mods mods & rdf subgroup hyrax avalon mods/rdf language/languageterm dcterms:language dc:language dcterms:language   hyrax uses dc:language (from the dc element 1.1 set) while the mods and rdf subgroup recommends mapping to dcterms:language. both recommend use of a controlled vocabulary for the value but dcterms:language specifies that the value needs to be of the linguisticsystem class, so a uri value instead of a text value. the mods and rdf subgroup is mapping these to iso639-2 values and avalon uses marc language codes. the result is that avalon can migrate this content to a uri value (like http://id.loc.gov/vocabulary/languages/eng) so dcterms:language will work best for this mapping. physical description @prefix bibframe: <http://bibframe.org/vocab/> . avalon mods mods & rdf subgroup hyrax avalon mods/rdf relateditem@type=”original”/physicaldescription/extent bibframe:extent none bibframe:extent the mods and rdf subgroup is mapping to the “extent” property from bibframe 1.0 to allow for text values (bibframe 2.0’s “extent” property requires a uri value). another potential property that allows for the use of text values is bibframe lite’s “extent.” hyrax does not offer a field for physical description so this will be an addition or customization for avalon. there will have to be a decision made as to which property will be better to use, although there is no question that these extent values will have to remain as text. related items url/label @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> . avalon mods mods & rdf subgroup hyrax avalon mods/rdf relateditem@displaylabel/location/url none rdfs:seealso rdfs:seealso relateditem@displaylabel none none rdfs:label   the mods and rdf subgroup does not have a mapping for this particular case of a related item url. hyrax uses rdfs:seealso and the value will always be a uri so that mapping can work for a related item link. the text label for the link is a more difficult case to handle. in avalon mods xml, this text is encoded as a “displaylabel” attribute on <relateditem> so this mapping is not straightforward. the rdfs:label property is probably not specific enough to identify a label for the related item url but using rdfs:label with rdfs:seealso ties these pieces together, so even though the namespace is from a more general-use ontology it can work in the avalon/samvera context. there is still a problem, however, if there is more than one related item; which rdfs:label goes with which rdfs:seealso url? since related item links are really just links and not linked data objects themselves, there is no guarantee of being able to find a property with a text label at the resolved url for use in avalon. hyrax only shows the url of the link so it might be best, though not ideal, to migrate only the url and not the label text in fedora 4 with rdf. to maintain our current avalon functionality would mean possibly creating empty objects in fedora 4 to manage a text label for related item links. genre @prefix edm: <http://www.europeana.eu/schemas/edm/> . avalon mods mods & rdf subgroup hyrax avalon mods/rdf genre edm:hastype none edm:hastype   the mods and rdf subgroup recommends mapping genres to edm:hastype, which requires a uri for a value. to use this mapping will require avalon to reconcile any values stored in the genre field in fedora 3 before migrating to fedora 4. this is worth it to try reconciling. if text values fail to reconcile to uri values, they would need to be updated so they could reconcile before migration could complete. subjects @prefix bf2: <http://id.loc.gov/ontologies/bibframe/> . @prefix dc: <http://purl.org/dc/elements/1.1/> . @prefix dcterms: <http://purl.org/dc/terms/> . @prefix foaf: <http://xmlns.com/foaf/0.1/> . subject type avalon mods mods & rdf subgroup hyrax avalon mods/rdf topical subject/topic dcterms:subject dc:subject dc:subject geographic subject/geographic dcterms:spatial foaf:based_near dcterms:spatial temporal subject/temporal bf2:temporalcoverage none bf2:temporalcoverage   topical subjects: ideally we would like to use the mods and rdf subgroup recommendation of dcterms:subject and require uris, but that would mean external reconciliation, which is probably not realistic due the current practice of not requiring a particular controlled vocabulary. thus, subjects have been entered more like keywords. we would need to use the text-based dc:subject to make migration easier from this field. geographic subjects: similarly to topical subjects, we do not require a controlled vocabulary for geographic subjects. this field is much more likely to be reconcilable against a location vocabulary like the thesaurus of geographic names, so we think external reconciliation would be easier to manage. this will allow avalon to have uris available for the value and use the mods and rdf subgroup recommendation of dcterms:spatial. ideally we would also be able to provide a report of records that did not reconcile post-migration. hyrax maps geographic subject to foaf:based_near as location. the geonames web service is supposed to be behind this field, but regardless of entering text or a uri, only text is saved in fedora. work is happening to add controlled vocabulary selection to the questioning authority gem behind this feature, so even though this functionality is not working in hyrax out of the box, it is in progress. temporal subjects: bf2:temporalcoverage takes a literal value. we ask for extended date time format (edtf) values in this field but we have not enforced this in anyway. temporal subjects are not indexed for date faceting so the edtf format is not required for this field to be migrated to the bibframe 2.0 property. terms of use @prefix dc: <http://purl.org/dc/elements/1.1/> . @prefix edm: <http://www.europeana.eu/schemas/edm/> . avalon mods mods & rdf subgroup hyrax avalon mods/rdf accesscondition@type=”use and reproduction” dc:rights edm:rights dc:rights and edm:rights   this field contains text statements regarding rights instead of a uri value like that from rightsstatement.org. a separate field for a rights uri should be offered as part of the migration process and would make these items more externally shareable to places like the digital public library of america and europeana. but for the text values that need to migrate, dc:rights can take a text value. this might be an example of using both the mods and rdf subgroup recommendation and the hyrax property as separate rdf statements on the digital object. statement of responsibility @prefix skos: <http://www.w3.org/2004/02/skos/core#> . avalon mods mods & rdf subgroup hyrax avalon mods/rdf note@type=”statement of responsibility” skos:note none skos:note   mods records have notes elements of varying types. the mods and rdf subgroup’s simple mapping uses the skos:note property and appends the “type” attribute to the beginning of the text value of the note as a label. it is likely that avalon will follow this recommendation to bring over the statement of responsibility as well as other types of notes that are offered. notes @prefix skos: <http://www.w3.org/2004/02/skos/core#> . note fields avalon mods mods & rdf subgroup hyrax avalon mods/rdf note note skos:note none skos:note note type note@type skos:note none skos:note   if there is a “type” attribute for any note then that value becomes a label added to the note text with a colon: “[@type value]: [note text]” identifiers @prefix dc: <http://purl.org/dc/elements/1.1/> . @prefix edm: <http://www.europeana.eu/schemas/edm/> . @prefix loc: <http://id.loc.gov/vocabulary/identifiers/> . @prefix owl: <http://www.w3.org/2002/07/owl#> . @prefix prov: <http://www.w3.org/ns/prov#> . identifier fields mods & rdf subgrouphyraxavalon mods/rdf avalon:pid avalon:pid (migration only) none none prov:wasderivedfrom permalink (mediaobject and masterfile) location/url@access=”object in context” edm:isshownat dc:identifier dc:identifier bibliographic id relateditem@type=”original”/identifier loc:local none owl:sameas other identifier relateditem@type=”original”/identifier none none loc:local[19]   migration of avalon metadata from fedora 3 to 4 means we will have new local fedora identifiers in our avalon instances. to not lose the fedora 3-based unique identifiers (pids) we are mapping that data as a provenance property to contain a link to the fedora 3 object. even though it more than likely will not exist after migration completes, this fits the value expected from the prov-o ontology and acts as a record of the path on which the digital object has traveled. this only affects migrated items. fedora 3 pid: avalon:19920 <fedora 4 object> prov:wasderivedfrom <http://[fedora3url]/avalon:19920>. avalon has never created a unique identifier as part of the mods xml record stored for objects. we do, however, have a permalink field for all published items that creates a unique url for each mediaobject and for each masterfile. in avalon’s current release with fedora 4, this property is mapped to a custom samvera community property, “haspermalink.” hyrax uses dcterms:identifier for an optional identifier field (defined as: “a unique handle identifying the work. an example would be a doi for a journal article, or an isbn or oclc number for a book.”). we are choosing to use dc:identifier as avalon’s permalink field since we will treat the permalink as the unique identifier and it will not be the same property as hyrax’s optional identifier field. this is part of the proposal outlined for different avalon identifier properties that has yet to be implemented. [4]   avalon allows for the import of descriptive metadata by bibliographic id, such as through a z39.50 web service or other configured service. the mods and rdf subgroup mapped this type of identifier to loc:local. however, avalon’s connection from digital objects to library catalog records mean the avalon items are the same as the items described in the catalog, either in the same format or in a digitized format. additionally, there are different types of identifiers that can be connected to an avalon item (oclc numbers, music publisher numbers, video recording numbers, local record numbers). the loc:local identifier type from the library of congress makes more sense in this context reserved for those identifiers that are specifically local to a collection. for those reasons owl:sameas works better to connect a bibliographic identifier that is used to import library catalog information into avalon, and loc:local can then be used for local identifiers that do not match with the other library of congress identifiers. conclusion what avalon has managed to do most effectively so far in migrating from xml to rdf is make use of properties from common ontologies and keep rdf statements simple. technical metadata statements are producing values that are text-based instead of linked data uris. structural metadata is taking advantage of linked data values to connect fedora 4 objects to each other but structure within a single file remains as custom xml. descriptive metadata mapping shows the most possibility for engaging with linked data, but the transition will most likely be a process taken in steps. languages and related items can definitely become linked data statements using uris for values and offering potential semantic web connections. geographic subjects also seem likely to be reconcilable to linked data uris from a controlled vocabulary, making those terms usable as linked data. this could also provide a way to visualize location information for mapping visualizations. genre is another field that seems to need reconciliation (it will need a uri value) in order to be brought over as an appropriately mapped property in rdf. beyond those fields, however, the descriptive metadata stored in avalon is either not based on linked data objects, such as terms of use and notes, or there are not enough requirements on the data values to make it possible to reconcile text strings to uri values, like names and topical subjects. the reasons for moving avalon metadata to rdf are two-fold: conform to community standards and requirements of the software we are using, and the desire to enhance our metadata for better meaning and reuse through linked data. the results in mapping efforts so far have been mixed. most of the data already migrated has come across as rdf but not as linked data. descriptive metadata mapping promises more possibilities with linked data but implementing these mappings will require a stepped progression to move away from the way mods xml encoded values to a meaningful rdf statement that can be used to describe the digital object in avalon. notes [1] on may 24, 2017 the hydra project formally changed its name to the samvera community. https://projecthydra.org/2017/05/24/hydras-new-name-is-samvera/ [2] avalon mods comparison to mods and rdf subgroup mapping recommendations: https://goo.gl/erh5jr [3] avalon mods compared to hyrax rdf properties compared to mods and rdf subgroup mapping recommendations: https://goo.gl/rk17nb/a> [4] avalon identifiers proposal: https://goo.gl/fdmjee references portland common data model [internet]. [updated 2016 aug 9]. pcdm wiki; [cited 2017 may 25]. available from: https://github.com/duraspace/pcdm/wiki the project [internet]. [updated 2017]. bloomington (in): avalon media system; [cited 2017 may 25]. available from: http://www.avalonmediasystem.org/project about the authors juliet l. hardesty, jlhardes@iu.edu julie hardesty is the metadata analyst/librarian for the indiana university libraries where she manages metadata creation and use for digital library services and projects. she helps develop descriptive, structural, and technical metadata standards for the avalon media system with jennifer and has been part of the project since 2012. jennifer b. young, j-young2@northwestern.edu jennifer b. young is the metadata coordinator at northwestern university libraries managing the creation of metadata for nul’s digital repositories. as a metadata expert for the avalon media system, jennifer works with julie to make decisions regarding all aspects of metadata. she has been with the project since 2016. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – using php to parse ebook resources from drupal 6 to populate a mobile web page mission editorial committee process and structure code4lib issue 18, 2012-10-03 using php to parse ebook resources from drupal 6 to populate a mobile web page the ursula c. schwerin library needed to create a page for its mobile website devoted to subscribed ebooks. these resources, however, were only available through the main desktop website. these resources were organized using the drupal 6 content management system with contributed and core modules. it was necessary to create a solution to retrieve the ebook databases from the drupal installation to a separate mobile site. by junior tidal introduction when launching our mobile website, the new york city college of technology’s (cuny) ursula c. schwerin library needed to create a page devoted to our subscribed ebooks databases. these resources, however, were only available through our main website. it was then necessary to create a solution to retrieve the ebook databases from our drupal-based desktop website and onto a separate mobile site. as a solution, we used the php simple html dom parser. this free php framework allows administrators to parse, scrape, or extract, content from html pages. drupal’s core and extended modules the ebook resources are stored within the drupal content management system (cms). using the extended drupal modules, content creation kit (cck) and views, in conjunction with the core taxonomy module, allowed the development of a dynamically generated ebooks resource page. this type of e-resource management was selected because of its flexibility and ease of use with library website administrators. updating these resources only requires filling out a few form fields within the browser, and the parameters of drupal’s views module automatically organize and sort these resources for end-users’ consumption. the technique to store this information stems from the work of leo klein. klein has provided an online tutorial on the creation of a library database page using drupal [1]. using the cck we created a drupal 6 custom content type, aptly titled ebooks, to store information on our electronic resources. this includes the title of the resource, url, description, and subjects pertaining to that database. the taxonomy core module allows content types to be categorized with words and phrases, chosen by librarians. in this case, we assigned a category of “type” for electronic resources, designating which ones were specifically ebooks. this taxonomy is used by the views module to differentiate between resource types. the views module is used to display the page of ebook resources. views pulls the taxonomy types “electronic resources” and “ebooks” to render the page for end-users. the page displays the attributes of each resource in alphabetical order. a pull-down menu is also available to display resources for specific subject types. the subject categories are also populated from the taxonomy core module. figure 1. screenshot of the views configuration for our desktop a to z database page. figure 2. how end users see our databases a to z page, rendered by views. mobile website the library’s mobile website is based off of the jquery mobile framework (http://jquerymobile.com/). this framework was chosen because of its mass compatibility with a variety of mobile devices. at first, we developed a mobile site utilizing drupal themes, mirroring the library’s desktop site. this was not an ideal situation. the drupal theme that was used for mobile devices was overwhelming, providing all of the content of the desktop site in one huge menu. additionally, this iteration of the site was not compatible with a number of mobile devices and phones. some devices did not trigger the “switch” to alternate between the drupal mobile theme and the desktop theme. as a result, we developed a mobile site that displays only selected information and links from the main desktop site. this includes the cuny web opac, mobile friendly e-resources, contact information, and information such as library hours and directions. since many vendors are creating mobile versions of their resources, we felt it necessary to have a mobile specific page to collect them all. the pew research center noted that overdrive, a distributor of ebooks for libraries, saw a mobile device usage increase of 22% in 2011 [2]. noting this trend, we created a web page specifically for mobile electronic resources, ebooks, and ios specific apps. in creating this, we found that a lot of information was being recreated on the mobile site. this could lead to potential problems, such as inconsistencies between the desktop and mobile sites, as well as the unnecessary maintenance of updating two separate sites with the same information. the mobile framework is independent and separate from the library’s drupal site. the two platforms do not share the same database, making it difficult to share content. in order to populate the page containing ebook resources, we needed to pull that content from the desktop site. this would alleviate redundant maintenance and ensure a consistency between the two sites. this is where the php simple html dom parser comes into play. figure3. jquery mobile framework displaying ebook resources. php simple html dom parser the php simple html dom parser , available at http://simplehtmldom.sourceforge.net/, is a utility which can be used to extract html contents and elements from a web page. based on the parameters that are programmed into the php script, either html tags or the contents between tags may be extracted. once this information is scraped, it can be arranged at the administrator’s discretion. the parse library, a component of the php simple html dom parser, contains a number of examples on how to code a php parse page. using these as an example, and looking at how the views module creates html pages, it took some trial and error to program a page that would parse our drupal site. the process works like this: the program opens the live html page. it then identifies the specific div and span tags that indicate which elements are ebooks. once these ebooks are found, the program “scrapes” and stores specific attributes being used. this includes the title, url, and description. the final step is to display this stored content in jquery mobile-friendly html output. here is the source code of the scraper we wrote, using the php simple html dom parser library: <?php //this function takes the ebooks produced by drupal views, scrapes the page, and returns a jquery mobile friendly page. include_once('simple_html_dom.php'); //php simple html dom parser library function loadebooks(){ $html = file_get_html('http://library.citytech.cuny.edu/research/ebooks/index.php'); //source of ebooks $i = 0; $j = 1; //the following cycles through the html to find ebook content. foreach($html->find('span[class=field-content] a') as $ebook) { if($i%2==0){ $item['title'] = $html->find('span[class=field-content] a', $i)->plaintext; $item['url'] = $html->find('span[class=field-content] a', $j)->href; $ebooks[] = $item; } $j++; $i++; } $listbooks = '<ul data-role="listview" data-inset="true" data-filter="false" data-theme="b">'; for($k = 0; $k < ($i/2); $k++){ $listbooks .= '<li><a href="'.$ebooks&#91;$k&#93;&#91;'url'&#93;.'">'.$ebooks[$k]['title'].'</a></li>'; } $listbooks .= '</ul>'; echo $listbooks; //output linked ebook resources } ?> the following code is the process to collect the ebook resource information. basically, it opens the html file and stores the title and url of each resource into an array, aptly called $ebooks. the foreach loop cycles and finds anchor tags nested within span tags of class field-content. notice that the iteration only stores information at every even numbered cycle. iterations that ran through odd cycles collected extraneous and erroneous information. for example, it would output the title of a resource a second time, linked with a correct and incorrect url. this is due to the two anchor tags that views has produced (see below), with the first iteration of the loop harvesting the first anchor tag. the output of the double anchor tags is from the views configuration. the field’s value of this specific view displays the title as a link: <span class="field-content"><a href="/black%20thought%20and%20culture"><a href="http://citytech.ezproxy.cuny.edu:2048/login?url=http://bltc.alexanderstreet.com/" target="_blank"> also, as part of the php simple html dom parser, information scraped from a page can possess selected attributes. the title of the resource retrieves a plain text version of the html element, whereas the array container of the url contains the information found in the href attribute of the anchor tag. foreach($html->find('span[class=field-content] a') as $ebook) { if($i%2==0){ $item['title'] = $html->find('span[class=field-content] a', $i)->plaintext; $item['url'] = $html->find('span[class=field-content] a', $j)->href; $ebooks[] = $item; } $j++; $i++; the following is sample source html code of the page that views produces for our ebooks page. this is what the php script is “reading” from the live desktop version of the page. <div class="view-content"> <div class="views-row views-row-1 views-row-odd views-row-first"> <div class="views-field-field-url-url"> <span class="field-content"><a href="/black%20thought%20and%20culture"><a href="http://citytech.ezproxy.cuny.edu:2048/login?url=http://bltc.alexanderstreet.com/" target="_blank">black thought and culture</a></a><br></span> </div> <div class="views-field-field-description-value"> <div class="field-content">primary sources covering the non-fiction published works of leading african americans. also includes interviews, journal articles, speeches, essays, pamphlets, letters and other fugitive material. </p></div> the code that displays the ebooks entries on the mobile site is found below. this code creates a list item specific for the jquery mobile framework, utilizing its unique tag classes and attributes. it constructs a hyperlink to the ebook resource using the title as a link text out of the information found within the array container. $listbooks = '<ul data-role="listview" data-inset="true" data-filter="false" data-theme="b">'; for($k = 0; $k < ($i/2); $k++){ $listbooks .= '<li><a href="'.$ebooks&#91;$k&#93;&#91;'url'&#93;.'">'.$ebooks[$k]['title'].'</a></li>'; } applicability the technique to scrape and parse information from a html page has been long practiced. we believe that the compatibility of using a php scraper to collect information depends entirely on the framework in which the source material is presented. that is, if the resources to be read are displayed in a consistent layout, uniformly coded in html, and can be accessed through a standard, non-ssl or otherwise encrypted connection, then the script should be able to retrieve and re-display this information. there is an obstacle in deploying this technique. it may become difficult if there are numerous tags in the source html that the script must navigate, so it’s advised that content should use the least number of html tags possible. otherwise, it may be necessary to code a script that utilizes conditional statements to wade through the html file. in other words, if there are a limited number of classes or ids with simple naming conventions, then it would be easier to code a working script. additionally, it is necessary to have access to the html source code to properly code a script. having the names of the various div and span class names may not be enough information to create a parse script. it may also be possible for this parse script to be used with other websites. drupal 7 (or other content management systems) could be used as an html source to extract resource content. if resources are displayed using named span and div tags, then it should be possible to scrape the contents of that page. if the views configuration within drupal 7 is similar to the output of the views configuration of drupal 6, then this solution should be usable for these two different drupal versions. this has not been tested, however, so the results may not be the same. the jquery mobile framework also has parsing functionality built into its api. this may be used as an alternate solution to scrape external information for a mobile site. however, the author was unaware of the function in the framework. additionally, there exists drupal 7 modules that support the jquery framework and scraping (jquery mobile module and example web scraper, respectively), as well as a jquery mobile theme. however, these extensions are in alpha and beta development stages and may not be adequate for a live production site. conclusion this php parse solution exports the ebook resources from our library’s desktop site and displays it in a separate mobile site. this is extremely useful, as library administrators do not need to maintain two separate sites, reducing redundant maintenance. parsing can also be used for other platforms, and other cms-based sites. it should also be noted that the source information may not be restricted to just ebook resources, encouraging further use for a wide range of applications. code before using this code, php simple html dom parser must be installed on your server. ebookscraper resources php simple html dom parser sourceforge page – http://simplehtmldom.sourceforge.net/ notes 1. klein, l. (05/15/2010). screencast: creating a library database page with drupal. http://chicagolibrarian.com/node/262 2. libraries, patrons, and e-books [internet]. updated june 22, 2012]. washington dc: pew internet and american life project; [cited 2012 june 26]. available from http://libraries.pewinternet.org/files/legacy-pdf/pip_libraries_and_ebook_patrons%206.22.12.pdf about the author junior tidal (jtidal@citytech.cuny.edu) is the multimedia and web services librarian and assistant professor for the ursula c. schwerin library at the new york city college of technology of the city university of new york. his research interests include usability, web metrics, and information architecture. originally from whitesburg, kentucky, he has earned a mls and a master’s in information science from indiana university. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – predictable book shifting mission editorial committee process and structure code4lib issue 54, 2022-08-29 predictable book shifting there are many methods to carry out a library book shift but those methods allow for varying degrees of predictability. the bookshift.py script, when used in conjunction with accurate measurements of a library’s collection and shelving, provides library staff with predictability, flexibility, and the ability to shift in parallel. for every shelf, the script outputs a phrase such as the following, “the last book from this shelf goes 12.3 in/cm into shelf 776.” while complicated shifts can still create surprises, using bookshift.py or similar methods typically make those surprises easy to correct. by joshua lambert introduction libraries eventually must move to new buildings or renovate old ones. when they shift a collection, they must decide how to move the books into the new space. if the collection is large enough, then slight, but repeated misplacements during a shift can lead to many extra hours of work once library staff recognize there is a problem. the missouri state university libraries staff were faced with the task of moving thousands of shelves of books. to make the move more predictable, the staff wanted an answer to the question, “before a single book is moved, how can i determine which shelf each book should sit on after the shift is complete?” looking at library literature provided hints and good processes to follow, but those processes were not predictable or flexible enough. the following table (table 1) illustrates one possible outcome of the general goal, and the rest of this article explains how to obtain that outcome with the help of a python script, bookshift.py, in the github repository (lambert 2022). the repository also contains data from the actual msu shift described in this article. table 1. example shifting report shelf_num goal 1 the last book from this shelf goes 24.0 in/cm into shelf 1 2 the last book from this shelf goes 18.9 in/cm into shelf 2 … … 1478 the last book from this shelf goes 26 in/cm into shelf 1550 shifting library books is difficult for many reasons. books that were on the shelves during shift preparation may no longer be on the shelf during the shift. likewise, what was checked out may be back on the shelf. books must remain in their exact order due to the classification system. the number of shelves and shelving layout may change. librarians may want to leave more empty space in some book collections because they expect the number of books in that collection to grow faster than other collections. shifting inconsistencies cause unexpected problems. if someone shifting books puts one less inch of books on every shelf, that will be a big problem after 1000 shelves. yet, the process given in this article allows library staff to shift not only predictably, but also backward, forward, in an arbitrary number of places simultaneously, or all of the above, at the same time. the author acknowledges that real life does not fit nicely into mathematical formulas and computer algorithms. that said, the process put forward in this article provides acceptable and understandable trade-offs in many if not most cases. background information there have been discussions of moving books in north american library literature since at least melvil dewey’s library notes discussions in volume 2, 1887 (dewey 1887). from that time forward, people have published advancements and refinements that fit their goals and contexts. there is a large body of literature related to the intersection of operations research and librarianship from the late 1960s and early 1970s that applied mathematical techniques to library processes. there was even an operations research themed library conference in 1971 (morse 1972). the pairing of the library world and operations research was mostly short-lived, but some influence remains (reisman and xu 1994; warwick 2012). in 1983, donna lee kurkul, as a library practitioner, published an article with a page containing seven mathematical formulas (kurkul 1983). those formulas were not new to library literature, nor complicated, but kurkul presented them together in a more concise and understandable manner than previous publications. the ideas in those formulas are useful in understanding the python script which is the subject of this article. after kurkul’s article, most publications on the topic have contributed by explaining new tools or processes, as computers and spreadsheets have become integral parts of library operation. steven carl fortriede’s book and accompanying spreadsheets provide both information and the tools needed to make a library move successful (fortriede 2010). it contains many of the ideas presented in this article. importantly, his work uses the term “waypoint”, and describes it as “a defined spot in a collection designated by the call number of the book to be filed immediately after the waypoint.” the idea of a waypoint is very important, and it adds predictability to the shifting process. the author’s previous conference presentation provides the mathematical basis behind the bookshift.py script presented in this article (lambert 2016). the resulting algorithm is in the set_measurement_totals_after function. this article uses the following terminology to refer to library shelving. a shelf is the horizontal structure that books sit on. a section is a vertical group of shelves of similar size, stacked above each other, with vertical space between them for books. a range is a group of sections, attached to each other, forming a line. ranges are arranged in parallel with aisles for people to walk down and look at books. for the purposes of this article, and the script, a double-sided shelving unit consists of two ranges of shelving, sitting back to back, with books facing out on each side. figure 1. depiction of shelf, section and range. fortriede’s spreadsheets allowed library staff to split the shift into “collections” (fortriede 2010). each collection could have its own future fill ratio. for example, library staff might want shelves that hold books from collection one to have a fill ratio of 80%. that means that 80% of the shelf would have books on it. mathematically, that would be the measure of books on a shelf divided by the measure of books that could be on the shelf if it were full. after that collection, someone might want to leave more growth room for collection two and want those shelves to have a fill ratio of 60%. using fortriede’s method, shift coordinators must adjust the “growth factor” for each collection manually until your plan matches future shelving capacity. missouri state university shift coordinators did not want to have to specify fill ratios for all collections, but they wanted to for some. for this paper, an “imposed fill ratio” is a fill ratio chosen by library staff that the python script will not change. in fortriede’s spreadsheets each collection has an imposed fill ratio but bookshift.py requires that some collections not have a fill ratio imposed on them. note that the script will not work if any shelf or collection of shelves has a fill ratio of greater than 100%. consider the scale of a book shift and the effect that incorrect measurements and book circulation have on the overall process. the missouri state university library shift example provided in the github repository in the “files-to-import” directory is a good example. if people measuring the collection measured 2,833 too few inches, that would mean that each of the 3,864 shelves would have 0.73 inches more on them than expected. while annoying to find out, this will not require much, if any, extra labor. if a library uses the recommended one waypoint per range, someone shifting will get near the end of the range and realize they have too many books. the shifter can adjust by adding extra books to shelves near the end. it will make fill ratios in that range, and every range, uneven, but acceptable. the good thing is that the problem will not compound but can be addressed between each waypoint with small adjustments. mismeasuring 2,833 inches also seems highly unlikely if the shelving coordinator uses quality control measures. also, if a shift coordinator wants extra assurance, they can tell the measurers to round up to the nearest half inch or centimeter for each measurement. this creates a small amount of extra space between waypoints but it will be inconsequential to the overall shift. required data to create predictable results, bookshelf.py requires three input files with predictable data formats. the first file provides measurement data from before the shift. the second provides destination shelving data. the third file provides imposed fill ratios for collections of books. the following sections describe what should be in each file. data placement the script looks for a subdirectory by the name of “files-to-import” which should contain three files: current.csv, future.csv, and collections.csv. library staff can clone the github repository and work within the resulting directory. library staff can also download the bookshift.py file, create a subdirectory named “files-to-import” and place the three relevant files in that directory. the github repository contains data used during the missouri state university shift. measurements before the shift before the shift, there are at least four data points that must be collected for each shelf. three of those are implicitly or intermittently collected. the data points are: horizontal measurement of books on a shelf. the start of a new section. the call number of the first book on the first shelf of a range. collection number. paper-based tables, like table 2, imitate the look of shelves and make it simpler for student workers to comprehend the process. table 2 shows recorded measurements in inches, though someone may use any unit of measure, assuming they use the same unit the whole time. the top of every new column is the first shelf of a section, therefore section numbers do not need to be listed on the form because they are implicit in the table itself. each range is visually separated by a different paper table. the author’s experience is that using paper is the easiest, fastest, and most reliable method for student workers, but using this method will require transcribing the data into a spreadsheet before processing. the collection number need not be tracked on this paper form, but may be if library staff choose to do so. note that for the sake of this process, and bookshift.py, one shelf, before the shift, can only have one collection on it. if there are books on a shelf that library staff would like to be in two separate collections, it is easiest to ignore the fact and just assign it to one collection or the other. alternatively, shift the offending books one shelf over, so there are not books from two collections on a shelf. table 2. example shelving table callnum: a1.1: d35 rangenum: 1 24 23 24 22 1 24 21 25 26 21 29 25 25 25 26 25 26 25 27 26 24 24 26 25 8 23 26 26 23 25 26 23 25 9 26 25 the use of paper may be skipped entirely and a spreadsheet used instead. the spreadsheet is designed to reduce keystrokes and labor. library staff can first type in a range of measurements in a spreadsheet column, then go back to add new section increments and call numbers as shown in figure 3. library staff can quickly copy down collection numbers for each range or for entire collections at a time if all the measurements are present. the following table shows the measurements above in the required format. a spreadsheet is not required, but the end result must be a csv file, and many people will prefer to input data in a spreadsheet and then export to a csv file. table 3 shows the first lines of the “current.csv” file in the “files-to-import” directory of the github repository. table 3. example current.csv file layout measure sctn_increment call_num collection_num 24 x q1 .h68 1975 1 21 1 25 1 27 1 8 1 26 1 23 x 1 25 1 25 1 26 1 the data entry will take some time and probably be thousands if not tens of thousands of lines long. it is important that the data be correct for the process to provide accurate results. that said, the process is resilient enough that some measurement or data entry mistakes still allow shifters to easily adjust for the mistakes. the definition of “some” depends on the magnitude of the mistake. after the data is in the spreadsheet it should be exported to csv format with the name “current.csv”. destination shelf measurements while it does take extra time, recording shelving configurations and measurements for the future shelving locations allows library staff to carry out a predictable shift. shelf sizes are often similar, therefore one can quickly copy and paste a large section of shelf measurements in a spreadsheet. the data points required are: shelf measurement the start of a new section. the start of a new range. this data likewise should be exported as a csv file, and its name should be “future.csv”. the resulting file will look like the example missouri state university “future.csv” file in the repository. while the example shows “x” or “c” for section and range increments, the script accepts any character. if it is not the first shelf of a section or range, then the field must be empty. collection data this last file is not required if library staff do not intend to use imposed fill ratios. if they do, the data points needed for the file are: collection number, starting at one. imposed fill ratio. table 4. example collection data col_num imposed_fill_ratio 1 2 0.65 3 0.7 4 5 0.5 6 0.5 if library staff intends to use collections, and therefore wants to change the file, every collection number in the shift must be listed. only imposed fill ratios should be placed in the second column. this file may contain two or more rows of data if library staff choose to divide the shift into collections. the data should be exported as a csv file with the name of “collections.csv”. if library staff do not choose to use collections, then make sure there is no “collections.csv” file in the “files-to-import” directory available for the script to find. instead, the “no_collections.csv” file will be used. the repository contains such a file that you can place in the “files-to-import” directory. data proliferation the script converts each csv file into a python dictionary that can later be converted into a pandas dataframe. once the data is ingested from the csv files the script creates new columns. the new columns may be used for further column creation or for later error checking and human inspection. the script adds the columns to the original data and creates the “data_before.csv” file in the “export-data” directory. the add_calculated_data function creates most of the additional columns. it usually does so by calling one small function that creates the data for one column. the add_calculated_data function creates additional columns using data from both “current.csv” and “future.csv.” the dataframe headers created from the “current.csv” file, along with short descriptions, are as follows: measure: imported from csv file. sctn_increment: imported from csv file. call_num: imported from csv file. col_num: imported from csv file. shelf_num: calculated field. this is a simple index beginning with 1, the first shelf, and ending with the x shelf. section_num: calculated field. the first section is 1 and increments each time there is an “x” in the sctn_increment field. range_num: calculated field. the first range is 1 and increments each time there is a character in the call_num field. runsum: calculated field. this starts with the first value of “measure” and creates a running sum, sometimes known as a sequence of partial sums. col_num2: calculated field. this is validated and normalized data from col_num. shelf_per_section: calculated field. this displays the nth shelf per section. this is a convenience and error checking column. section_per_range: calculated field. this displays the nth section per range. it is a convenience and error checking column. way: calculated field. this displays the waypoint. units_into: calculated field. this displays the number of inches or centimeters into shelf x where the last book from the current shelf should end up. pretty_waypoint: calculated field. this is a convenience column constructed from “way” and “units_into,” for each row. the “future.csv” file gains a similar set of columns: shelf_measure sctn_increment range_increment shelf_num section_num range_num shelf_per_section section_per_range book_measure final_runsum way units_into pretty_waypoint analysis the following paragraphs explain the foundational ideas for getting useful output. most of this work happens in the function named set_measurement_totals_after. a first step toward analysis of the data is to calculate the shelf space occupied by each collection before the shift. python’s pandas library easily calculates such things using the groupby method. the “set_data.csv” file shows this calculation in the column called “books_measure” and figure 5 illustrates the final table. table 5. final table calculations col_num imposed_fill_ratio books_measure fill_ratio space_needed 1 34514.0 0.74476424049762 46342.18202654199 2 0.65 1723.0 0.65 2650.7692307692305 3 0.7 3437.0 0.7 4910.0 4 27076.0 0.74476424049762 36355.128949140955 5 0.5 1287.0 0.5 2574.0 6 0.5 901.0 0.5 1802.0 7 0.7 347.0 0.7 495.7142857142857 8 0.4 551 0.4 1377.5 9 12362.0 0.74476424049762 16598.541293739123 10 0.5 894.0 0.5 1788.0 11 0.6 1708.0 0.6 2846.666666666667 12 13033.0 0.74476424049762 17499.49754742776 once bookshift.py calculates the book measure for each collection, it needs to calculate the space needed for each collection. we can do so immediately for collections with imposed fill ratios, because the book measure and fill ratio are known. the “fill_ratio” column in table 5 shows the final calculation. note that the full table is given above but the script can calculate only rows with imposed fill ratios at the beginning. figure 2 shows how to calculate space needed. figure 2. space calculation formula. the set_measurement_totals_after function calculates the space needed for each imposed fill ratio. but, we do not yet know the fill ratio or space needed for collections that do not have imposed fill ratios. the script can calculate those using the subsequent steps. first, the amount of space left for all collections with no imposed fill ratio can be calculated using a second formula (figure 3). for the sample data this calculation is approximately 116795.35 = 135240 – 18444.65. figure 3. non-imposed collections space formula. using this formula, the fill ratio for all non-imposed collections is discovered, and once the script has that, it uses the first formula above to calculate space needed for each non-imposed collection. note that “space needed” at this point is synonymous with “space remaining” because we want to use the remaining shelf space for the rest of the collections. the script can now calculate the space_needed column for all collections. library staff can double-check that by finding the summation of the space needed for each collection and comparing that to the total available space. they should be identical except for minor rounding errors. to be useful, the script must provide actionable information that library staff can use to shift materials. one of the fields calculated above for the current books was the running sum value. the measure of books on each shelf allowed us to calculate that. we know the fill ratio and future shelving configuration, therefore we can now calculate the measure of books we want on each future shelf using a third formula (figure 4). figure 4. future shelf usage formula. in situations where books from two collections occupy the same shelf, the formula is more complicated, but the script takes that into account. once we know the measure of books on each shelf, we can calculate a running sum for the future locations. the function runsum_after accomplishes this in the script. using running sum values for before and after shifting, we can now calculate waypoints. a book at a running sum measure of 119 before the shift will still be at a running sum of 119 after the shift, as illustrated in table 6. the runsum_after column gives the last possible measure on a shelf where a book should be. after shifting, shelf one should hold books occupying measures 0 to 28. shelf five should hold books at measures greater than 102, but not exceeding 130. therefore, a book at measure 119 of runsum_before starts at shelf four and ends up on shelf five, as illustrated by the “x” in column book_after of table 6. table 6. before and after waypoints table (note that this figure is made for illustration only and does not come from the data listed elsewhere) shelf_num runsum_before book_before runsum_after book_after 1 34 28 2 64 56 3 92 84 4 124 x 102 5 149 130 x the function waypoint_calc calculates the waypoint for every shelf and creates a list. it also creates a list that tracks how many units into the shelf a waypoint is. this ultimately allows the script to create the last dataframe column, which takes the form of a sentence. that dataframe exports to the file, “data_before.csv”, and it will contain phrases such as “the last book from this shelf goes 24.0 in/cm into shelf 1.” following are two columns taken out of “data_before.csv” to illustrate the result. it is telling the book shifter that the last book from shelf 1 before the shift should be placed 24 inches into shelf 1 of its future location. table 7. final waypoints table shelf_num goal 1 the last book from this shelf goes 24.0 in/cm into shelf 1 1 the last book from this shelf goes 18.9 in/cm into shelf 2 1 the last book from this shelf goes 17.9 in/cm into shelf 3 1 the last book from this shelf goes 18.8 in/cm into shelf 4 there is also a file called “data_after.csv” that provides a look from the future shelving location back to the original. it will contain a phrase such as “the last book on this shelf comes from 7.1 in/cm into shelf 3.” the shift coordinator should only use one table as waypoints but they can choose which one best meets their needs. these phrases are the actionable output, created by the bookshift.py, that allows predictable shifting. discussion missouri state university shifted library of congress classified books on the q-z shelves, with data mentioned above. we moved the first books to two ranges of empty shelves, then we took apart and moved the newly vacated shelves, moved them to the new location, loaded them with different books, and repeated the process. we also needed to shift books from the r shelves before we shifted all of the books on the q shelves. during that time, we had people shifting books in two or three places simultaneously and the process worked well. subsequent smaller shifts outside of the q – z range allowed the author to refine the process presented here. using this method of measuring, calculating, and shifting provides great advantages. first, it provides a very high degree of predictability which aids staff planning. second, the method provides flexibility by accounting for varying shelf counts, varying shelf lengths, and the separation of different fill ratios for different collections. third, assuming there are not external factors that would negatively affect almost any shift, library staff can work in parallel without worrying that they will run into major surprises which would require re-shifting everything. if there are 10 student workers independently shifting at the same time, they can shift in different places and there is no worry of interfering with the others. this could also provide extra direction if the library hires a moving company whose personnel are not used to working in libraries. in addition, library staff can start shifting in the middle of the collection and work backward, or forward, as needed. the waypoints will keep book shifters on track, and they will give planners confidence things are moving as planned. this shifting process should involve someone proficient and experienced in shelving and shifting. shifting a collection in a library requires being exact but being tolerant of the unexpected. do not expect all measurements to be correct, and do not expect that every place staff shift to will have exactly the number of books needed. a three inch wide book may occupy the last 1 inch of a shelf and then add two extra inches that shouldn’t be there. that is ok, just move forward with the typical shelf length and there will be extra space somewhere else before the next waypoint. although the script provides waypoints for every shelf, the author highly suggests having shelver visible waypoints only once per range. the people doing the shelving, and their supervisors, must have the freedom to make small adjustments within those ranges to make things work. for instance, if there is a large encyclopedia set that is no longer published, the shifter may not need space between volumes. if the shifter puts three shelves worth of encyclopedias together, with a fill ratio of 100%, there will be extra space at the end of the range that will make the waypoint or shelving job seem wrong. the extra space here and there is tolerable, and it should not affect placement of books on any other range. the extra space will be available in the future when it is needed. the beauty of the method presented in this article is that mistakes or oddities are easily fixed within a range by shifting books a few shelves left or right. people who regularly shelve are often trained to make such shifts without consulting with a supervisor, so it takes less labor, specifically less supervisor labor, to address. the script described here does not account for cases in which library staff must combine book collections while simultaneously shifting. while out of scope for this article, there are some ways to accomplish that, but with fewer waypoints. there are some unusual situations where shifting books is probably never a good idea and the method presented here does not account for such situations. the script presented here will not work well if 50% of the collection is checked out when you measure and only 5% is checked out when you shift or vice versa. also, if a whole section of books is checked out and not initially measured, but returned before the shift, then there will be too many books in that area. it is the author’s experience that such checkout patterns are unusual and when they do occur, they are well known by the shelvers or other circulation staff. a shift coordinator can use imposed fill ratios to address such issues. that means, someone will have to make an educated guess, or use other evidence based processes outside the scope of this article, to determine the imposed fill ratio. if measurements are off by too much in one place, the described process will not work. an experienced shift coordinator should be able to recognize such problems and not start the shift process at all, revise the process, or review and correct incorrect data. it is always best to measure and shift in as short a time frame as possible and when the maximum number of books are on the shelf. that said, missouri state university circulation does not vary enough at any point in the year for it to matter for it to negatively affect the shift process. after considering the discussion above, there are a number of research directions people could follow. this method of shifting requires strict adherence to processes when creating csv files. can some of that strict process be relaxed to make data collection and processing easier? also, the fact that this shift method requires basic python knowledge, at least enough to execute a program, will mean that many library staff will not use it. it is likely that more people would adopt this method if they could do so using only spreadsheets. references dewey m. 1887. library notes: improved methods and labor-savers for librarians, readers and writers. library notes [internet]. [cited 2022 june 9]; 2(6):95-97. available from: http://hdl.handle.net/2027/hvd.hxh1dc fortriede sc. 2009. moving your library. ala publishing. [accessed 2022 jun 9]. https://www.ala.org/aboutala/offices/publishing/editions/webextras/fortriede09942/fortriede09942 fortriede sc. 2010. moving your library: getting the collection from here to there. chicago: american library association. kurkul dl. 1983. the planning, implementation, and movement of an academic library collection. college and research libraries. 44(4):220–234. lambert j. confident shifting for complex moves. brick and click: an academic library conference; 2016 nov 4 [accessed 2022 june 9]. https://static.sched.com/hosted_files/brickclickanacademiclibrary2016/3d/joshua%20lambert%20confident%20shifting.pptx lambert j. bookshift.py – predictable library book shifting. [accessed 2022 jun 9]. https://github.com/distichum/bookshift morse pm, leimkuhler, ff, buckland, mk et al. 1972. proceedings of the thirty-fifth annual conference of the graduate library school. the library quarterly. 42(1). reisman a, xu x. 1994. operations research in libraries: a review of 25 years of activity. operations research. 42(1):34–40. warwick j. 2012. library operational research: time for a new paradigm? in: higher education management and operational research. brill sense. p. 269–291. about the author joshua lambert is a librarian and associate professor at missouri state university. he manages government documents, provides reference assistance, teaches courses for credit, provides library instruction, and gathers and analyzes data for the library and other researchers. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – harnessing apache mahout to link content mission editorial committee process and structure code4lib issue 22, 2013-10-14 harnessing apache mahout to link content the national library board of singapore has successfully used apache mahout to link contents in several collections such as its infopedia collection of articles (http://infopedia.nl.sg). this article introduces apache mahout (http://mahout.apache.org) and focuses on its ability to link content through text analytic techniques. the article will run through the what, why, and the how. if there is a big collection of content that needs to be linked, apache mahout may just be the answer. by lim chee kiam, balakumar chinnasamy what is apache mahout? apache mahout [1] is a scalable open source machine learning library with implementations of a wide range of algorithms supporting use cases such as clustering, classification, collaborative filtering and frequent pattern mining. clustering takes a collection of documents and groups them into sets of related documents. classification learns from documents in existing groups and assigns new documents to an existing group. collaborative filtering recommends items based on user behavioural patterns, e.g. “people who borrowed this book also borrowed these other books”. finally, frequent set mining takes item groups (such as shopping cart contents) and identifies which items usually appear together. many commercial and academic organisations make use of apache mahout [2] . for example, foursquare uses apache mahout for its recommendation engine [3] . for this paper, we will only focus on its ability to allow us to link content together by finding other similar content. why use apache mahout? unstructured content, such as text articles, is a huge and growing portion of the data to which we all have access. however, unstructured content has always been a tad more challenging to work with. even for a small collection of a few hundred articles, the amount of time and effort needed to read and relate articles is already quite substantial. as the number of articles grows into the thousands and beyond, this gets exceedingly difficult. this difficulty is compounded when there is only minimal metadata on the articles in the collection. new articles and updates to existing articles pose the additional challenges of having to create a new recommendation list and update previously completed recommendation lists. it was precisely this problem that we first set out to address with our proof-of-concept on apache mahout. the singapore “infopedia” [4] (http://infopedia.nl.sg) is a popular electronic encyclopedia on singapore’s history, culture, people and events. with less than 2,000 articles, it attracted an average of over 200,000 page views a month in 2012. even for this small collection of articles, only a subset of the articles had recommendations to other related articles. (note: this is no longer the case now as we have successfully utilized apache mahout to supplement the manual recommendations from the librarians.) apache mahout generates recommendations for the full infopedia collection in less than 5 minutes. figure 1:infopedia article with manual librarian recommendations figure 1 is a sample article from infopedia. the 5 manual recommendations for the article are highlighted in a red box. figure 2: infopedia text analytics sample result the corresponding sample result in figure 2 shows that 3 of the manual recommendations were among the top 4 recommendations automatically generated by apache mahout. in addition, apache mahout unearths a whole new set of relevant related articles. our librarians sampled 50 random articles and their recommendation results. they noted that 48% of the recommendations were highly relevant and another 27% were of medium relevance. the other 25% were of low relevance. as of 13th june 2013, infopedia articles are now supplemented with apache mahout generated recommendations in production. figure 3: infopedia article supplemented with apache mahout recommendations the apache mahout results were generated by simply using the apache mahout command line. the next section delves into how this is done. how to use apache mahout apache mahout comes with a command line interface [5]. our proof-of-concept was accomplished simply through the use of the command line. the main steps are obtain the text files create the sequence files create tfidf weighted vectors get the similarity results integrate the results obtain the text files we wrote a simple program to extract our infopedia collection from our content management server running alfresco community edition. each infopedia article is extracted into its own file. the file is named with the article’s unique identifier and contains the article title and text. create the sequence files the next step is to create the sequence files to be used by apache mahout. to do that, we use the apache mahout command line as follows: mahout seqdirectory \ -c utf-8 \ -i datafiles/ \ -o seqfiles the above command simply takes in utf-8 encoded files from the datafiles folder as input and outputs the results into the seqfiles folder. create tfidf weighted vectors from the sequence files, we create tfidf weighted vectors. tfidf stands for term frequency inverse document frequency. tf measures the frequency of a term in a document while idf measures the rarity of the term across the collection. tfidf gives a high value for a highly occurring term in a document that rarely occurs in the collection. this effectively filters out common terms. mahout seq2sparse \ -i seqfiles/ \ -o vectors/ \ -ow -chunk 100 \ -x 90 \ -seq \ -ml 50 \ -n 2 \ -s 5 \ -md 5 \ -ng 3 \ -nv the above command reads the sequence files created in the previous step and creates the tfidf weighted vectors into the vectors folder. the created vectors are again sequence files that are used in the next step. the important options to note in the command are as follows: -x max document frequency percentage -s min support of the term in the entire collection -md min document frequency -ng maximum size of the n-grams the first 3 options listed above help to reduce the number of terms to be considered. the -ng option instructs apache mahout to look beyond single words and consider “phrases” of up to 3 words. in order to see all of the options and a brief note on their usage, execute the following command: mahout seq2sparse --help note: the –help option works for the other mahout commands as well. get the similarity results to get the similarity results, we first need to create a matrix from the vectors. each row in the matrix is a document’s vector representation. mahout rowid \ -i vectors/tfidf-vectors/part-r-00000 -o matrix the above command will create the matrix in the matrix folder. the actual computation is invoked with the following command. mahout rowsimilarity \ -i matrix/matrix \ -o similarity \ --similarityclassname similarity_cosine -m 10 -ess this command generates the 10 most similar articles for each article into the similarity folder. so how does apache mahout decide what is similar? in the above command, we instruct apache mahout to use cosine similarity as the distance measure between the various document vectors. cosine similarity is commonly used when handling text articles. the following command will extract the results into the similarity.txt file as text. mahout seqdumper -i similarity > similarity.txt the similarity.txt file contains records like the one below: key: 0: value: {14458:0.2966480826934176,11399:0.30290014772966095, 12793:0.22009858979452146,3275:0.1871791030103281, 14613:0.3534278632679437,4411:0.2516380602790199, 17520:0.3139731583634198,13611:0.18968888212315968, 14354:0.17673965754661425,0:1.0000000000000004} ... the above record says that the article with key 0 has the following similar articles: article with key 14458 scored 0.297, article 11399 scored 0.303, and so on. so how do we know which articles the key values are referring to? the next command extracts the document index into a text file named docindex.txt. mahout seqdumper -i matrix/docindex > docindex.txt the docindex.txt file contains records as follows: key: 0: value: datafiles/sip_254_2005-01-24.txt key: 1: value: datafiles/sip_690_2005-01-20.txt ... integrate the results the final step involves enhancing our infopedia website to process the similarity results and display the recommendations. coming next with the infopedia recommendations in production, we have moved on to applying what we have learnt to our other collections. this includes a collection of photographs, where apache mahout processes text files containing the metadata associated with the photographs. in addition, we have also experimented with cross-collection recommendations and will be pushing that out soon. conclusion the discoverability of a library’s collections increases with the number of links within and across collections. month pages jan 2013 227,241 feb 2013 217,004 mar 2013 243,081 apr 2013 226,199 may 2013 272,715 jun 20131 282,878 jul 2013 341,485 aug 2013 349,923 1infopedia with apache mahout recommendations launched on 13th june 2013 the table above shows the monthly page counts for infopedia in 2013. it may not be conclusive that the improved discoverability of the articles is the cause of the increase in the monthly page counts but the numbers are definitely encouraging. the ability to link content automatically is therefore essential for libraries to unearth the hidden values in their collections, especially for sizable collections that go beyond thousands of items. apache mahout is an example of a technology that seems esoteric and obscure but is now readily available and accessible. the time is ripe for libraries to leverage such technologies to bring out the enormous values in their collections. references [1]. apache mahout, “apache mahout,” [online]. available: http://mahout.apache.org/. [accessed 14 aug 2013]. [2]. apache mahout, “powered by mahout,” 30 july 2013. [online]. available: https://cwiki.apache.org/mahout/powered-by-mahout.html. [accessed 26 august 2013]. [3]. j. moore, “building a recommendation engine, foursquare style,” 22 march 2011. [online]. available: http://engineering.foursquare.com/2011/03/22/building-a-recommendation-engine-foursquare-style/. [accessed 26 august 2013]. [4]. national library board, singapore, “singapore infopedia,” [online]. available: http://infopedia.nl.sg. [accessed 15 august 2013]. [5]. p. ferrel, s. marthi. “quick tour of text analysis using the mahout command line,” 1 august 2013. [online]. available: https://cwiki.apache.org/confluence/display/mahout/quick+tour+of+text+analysis+using+the+mahout+command+line. [accessed 15 august 2013]. about the authors lim chee kiam (chee_kiam_lim@nlb.gov.sg) is a senior solution architect at the national library board in singapore. chee kiam is responsible for doing the initial proof-of-concept on apache mahout and continues to drive related projects in this area. balakumar chinnasamy (balakumar_chinnasamy@nlb.gov.sg) is a solution architect at the national library board in singapore. balakumar is the key person bringing the apache mahout cluster and related services into production. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – unveiling jangle: untangling library resources and exposing them through the atom publishing protocol mission editorial committee process and structure code4lib issue 4, 2008-09-22 unveiling jangle: untangling library resources and exposing them through the atom publishing protocol the jangle project intends to expose the data hidden in library systems by using the atom publishing protocol to provide simple, consistent access to content and resources. the lack of uniform access to the underlying data in library systems is a major impediment to library development. the jangle project has the potential to enable new development opportunities by leveraging simple to use and easy to understand processes. this article discusses the benefits of the atom publishing protocol and how jangle utilizes it, including a description of the current jangler reference implementation and case studies of the simplicity of developing within the framework. by ross singer and james farrugia introduction any library developer knows the difficulty of trying to write applications that interact with or use data out of library systems. the situation is even more problematic for outside developers entering a library-based integration project, since they are likely ignorant of the historical baggage, seemingly obsolete protocols, and arcane file formats that are involved when accessing and using library metadata. too often, costly library systems serve more as maximum security prisons than walled gardens. it can become impossibly difficult to integrate library systems into other local environments when the only ways to get data in or out require proprietary desktop clients or server-side command line scripts. if application programming interfaces (apis) exist, they vary from system to system with regards to access and functionality. jangle (just another next generation library environment) [1] is a specification (and accompanying reference implementation) employing simple, common and well supported web protocols to enable the creation, maintenance and distribution of resources among library-related data sources. the goal is to standardize and simplify the process of getting data in and out of library systems, allowing content to proliferate outside of standard library applications and providing a level playing field for external developers building applications that interact with library systems. this article will attempt to lay out the need for and role of jangle. it begins with an overview of the past efforts to enable open data in libraries and the evolution of the jangle project, including an introduction to the atom publishing protocol and the atom syndication format and a description of how jangle employs these specifications. then it describes the jangler reference implementation, the proof of concept unicorn connector and the dlf ils-di adapter, showcasing the simplicity of developing applications to both expose and consume jangle functionality. since the project is relatively new and the specification is somewhat volatile, we will finish by discussing the future plans and making some predictions about where the next steps may lead. a brief history of open data and libraries there have been plenty of attempts in the past to provide more standardized access to library data. the marc format itself was created solely for sharing records amongst libraries. z39.50 [2] has done an admirable job at providing access to data given its age and its pre-http protocol, but it generally only provides a search interface to bibliographic records; information about holdings is included in a limited percentage of libraries. limitations on the number of search results and available indexes make it impossible to “harvest” all records from an institution, and the protocol is almost impossible to explain to parties outside of the library world. ncip (niso circulation interchange protocol) [3], an attempt to provide circulation interoperability between libraries, suffers from a lack of implementation and support even within library systems. the ncip implementations that do exist are different enough to thwart the goal of universal borrowing between systems produced by different vendors. in 2004, niso began an initiative known as views (vendor initiative for enabling web services) to produce a standard for interoperable web services for library systems. the committee was to design a specification to enable standardized access to library data with metasearch applications as the major use case. eventually rebranded as the niso web services working group, the committee’s only legacy is a best practices document [4] for creating web services. the document lays out the ways this could be done, but prescribes nothing and offers no specifics on how the recommendations should be implemented in actual library systems. other groups (sru [5], niso metasearch initiative [6], and oasis search web services [7], for example) have defined standards to aid remote searching across institutions, but in many ways these are just incremental improvements to make z39.50 fit less awkwardly in a web-centric universe. there have been no real attempts to provide uniform access to library data outside of searching or to provide interfaces to create or update resources maintained within library systems (outside of the institutional repository space). it is this vacant space that jangle is intended to fill, providing a clear, straightforward api that uses the web’s de facto standards for creating, reading, updating, and deleting resources. the three major web standards that the jangle api utilizes are: persistent urls a restful design [8] that allows these urls to be used to create, read, update, and delete web resources a protocol designed to support feed readers that can periodically poll data for updates, with the ability to customize the kind of data that users want to be fed the evolution of jangle jangle began as a project within talis [9] as a means to promote use and accelerate adoption of open library data. its goal was to enable new development (local, open source or commercial) on existing library applications by providing an open framework and a consistent api. jangle was intended to be an open source service-oriented architecture application based loosely on talis’ keystone product, which is used to enable integration between library functions and other institutional applications [10]. keystone would remain a closed source application but would seed the restful web framework (known in keystone as talissoa) for jangle’s api. connectors, small applications that contain the business logic for interacting with specific back-end services, could then be reused between jangle and keystone. the talissoa framework, however, was complicated to set up; it had many dependencies and little documentation and a considerable amount of java expertise was needed to create even a simple connector. talis decided that this would prove to be too much of a burden to libraries that lacked the resources or skill set needed to develop connectors between their local systems and talissoa. in an effort to make a prototype quickly, the developers built early jangle designs using the ruby on rails web framework. jruby (a fully compliant ruby interpreter written completely in java) was chosen as the platform so local developers could build their connector classes in java, ruby or other java bytecode languages, such as jython and groovy. frustrations with jruby performance and use of rails as a solely web services interface began to raise doubts about this direction. questions arose as to whether or not a library-specific service-oriented architecture framework was even necessary. after an early jangle brainstorming session with the developers at equinox software incorporated, mike rylander, equinox’s vice president of research and design, raised the idea of using the atom publishing protocol (atompub) to provide a very simple rest interface for jangle. after lying dormant for several months, this proposal was revived, bringing new life to the project when jangle began to lose momentum due to a lack of clear direction in the face of other soa projects. the focus of jangle changed from creating a deliverable application providing web services to building an api that exposes resources within library systems using the atom publishing protocol. a reference implementation of the specification would be built concurrently to test, demonstrate and validate the api as well as provide a very basic framework for libraries to get started. talis is still heavily involved in the jangle community while the project tries to build its own momentum. the company is providing developer support and server resources in an effort to bootstrap the process. the jangle community, however, is external to talis and the decisions on direction and functionality of the project are made democratically and not subject to talis’ corporate interests. jangle is a project that has been released into the wild outside the control of any one organization. why atompub? when designing a web services interface, there are a variety of styles and protocols upon which the api could be based, including soap, xml-rpc, or custom rest api. when jangle was first being developed, the api was based on the rest style with no conformance to any particular syntax or specification. the xml responses it produced varied from resource to resource with no uniform format across the application. this meant that jangle would require custom schemas, clients and robust documentation in order to be usable. it was decided that utilizing a single standard protocol across jangle would immediately make it more accessible, creating a much shallower learning curve for developers not versed in the library domain or jangle. the atom publishing protocol (rfc5023) [11] is “an application-level protocol for publishing and editing web resources using http [rfc2616] and xml 1.0.” simply put, atompub advertises related groups of resources (collections), assigns them a uri and makes them available in the atom syndication format (rfc4287) [12]. the protocol also specifies how to add, update and delete resources within a collection (using put, post, and get requests). these collections are aggregated by distinct services known as workspaces. the specific resources within the collection are called entries. atompub announces the workspaces and collections available via atom service documents, xml documents that define the paths to collections and the appropriate mime-types allowed for creating and editing content. the atom syndication format was designed to syndicate all kinds of data, so while its native elements seem more suitable for applications such as weblogs, the content field can include anything, from marc xml [13] to ead [14] to pdfs to binary marc 21 [15]. the flexibility of the content field, as well as the ability to degrade gracefully for simple atom clients (they will display the fields they understand, such as title and author and ignore the fields they cannot work with) is what makes atom especially appealing. since atom was designed to be enhanced through domain-specific extensions, it can be quite flexible for diverse needs. an additional benefit to atompub is that it assigns unique, permanent and dereferenceable uris for each resource within the atom feeds. these uris are reusable within other applications, since they produce an atom feed consisting of just that resource. the most desirable attribute of atompub, however, is the broad base of support it is receiving from major web players. google has based its entire gdata api suite [16] upon the atom publishing protocol, using a combination of its own extensions and other commonly used extensions, such as opensearch [17]. microsoft has implemented atompub to enable access to its unified storage platform. the major blogging platforms, such as wordpress [18] and movabletype [19], have atompub interfaces to manage content remotely. with such a broad base of support from players that have a desire to make their content easily accessible everywhere, atompub has a wide client base, a growing user community, and expanding server support. the ability to draw upon cheap, commodity client libraries and learn from google or microsoft on how best to implement functionality makes the atom publishing protocol a very attractive standard for new web services. the anatomy of jangle jangle is comprised of two distinct pieces (fig. 1): the publicly accessible interface, known as the jangle core, which presents the atompub api to clients a federation of connectors, which take proxied http requests from the core, translate them using the business logic of a particular local library application and return the results serialized as a json [20] representation of an atom syndication document. the two pieces communicate via http, which means the core and connector can be written in two completely different languages. if multiple connectors are being used by the core, they could all be written in completely different languages. figure 1: a visual representation of jangle the core the jangle core is a very simple atom publishing protocol server. this means that if you have the jangle core running on a server, web clients can access atom syndication format feeds of data provided by the connectors in addition to adding, updating and deleting that data in the underlying library applications. the core’s purpose is to manage the http requests, proxy requests to the connectors and serialize the connectors’ json responses into xml. the jangle core contains no business logic and does not help clients negotiate the appropriate connector to call. it is the responsibility of the client interacting with the jangle core to understand the role and the context of the underlying library systems. the core can aid the client, however, by providing an atom service document with a brief, human readable description of the available applications being proxied. connectors external clients can access resources provided by individual jangle connectors using paths off of the jangle core url. each connector provides its own service document to define the collections it provides. an imaginary jangle core that is hosting both an integrated library system (ils) and an electronic resources management system might have connector paths of http://jangle-core.example.org/catalog/ and http://jangle-core.example.org/erms/. both of these would be able to return a service document (for the ils, this might be http://jangle-core.example.org/catalog/services/), which would explain what resources are exposed by the connector. the proxied connectors are completely independent of each other and feeds cannot be created across multiple connectors within a single request. however, since jangle provides permanent and reusable uris for every resource, it is possible to use these uris within the native library systems when describing entities in other applications. connectors require more knowledge of (and access to) the underlying library systems that they are exposing than the jangle core. a connector is intended to abstract inconsistencies between library applications and break the underlying data up into the jangle model. connectors provide simple restful interfaces with five base paths (/services, /actors, /resources, /items and /collections) and return results to the jangle core as json objects. the json objects also include the mime-type defining how the jangle core should serialize the response for the client. the connectors define the record formats available and the relationships between base entities (for example, how items are associated with actors) and manage the identifiers associated with any given resource. since a connector could potentially be used by multiple jangle cores, a convention was necessary to notify the connector of the location from which it was being called. for this reason, the jangle core sends an extra http request header, called “connector_base”, with the base url of the request. the connector then has the absolute uri in the event it needs to be embedded in its content payload. entities jangle specifies only four base data types, or entities, to define all of the data contained in a given library system. the entities are: actors: these are users of the underlying library system. for an integrated library system, these could correspond to borrowers. for an institutional repository, these might be the submitters. resources: these are the primary records central to the exposed application’s purpose. in an integrated library system, these would be bibliographic and authority records. in an interlibrary loan application, they would be loan requests. items: these are specific instances of resources, such as physical copies, holdings, or specific electronic formats. in the case of a link resolver, these could refer to the holdings of a given database with regards to a particular journal. collections: these are any grouping of any of the above entities, including groupings that pull from different entity types. collections could be used to separate bibliographic records from authority records or to identify course reserves in an integrated library system. collections have no inherent meaning of their own and a jangle client should make no assumptions about what collections exist. a fifth entity, events, has been proposed, but it is unclear if this would be necessary or would cause too much complexity as its own base data type. the entities map to corresponding base paths in the connector. for example, the url to retrieve all of the resources from a given connector would be http://jangle-connector.example.org/resources/ and the url to retrieve the actor with the local identifier 1234 would be http://jangle-connector.example.org/actors/1234. the output from these requests would be a json response (roughly resembling an atom feed) with the intended resources bundled in an array called “entry.” json was chosen over xml for this task because it is so much easier to produce and parse, especially for known data structures. the main ‘record’ being returned would be contained in the entry’s content element, along with the record’s mime-type (fig. 2). the metadata itself is served ‘as is’ within the content field rather than being transformed to json. in the case of a bibliographic record, for instance, the content element could contain marc xml or possibly marc21. the jangle core could then take this json response and serialize it into an xml document and serve the appropriate mime-type and http status (fig. 3). { "type":"application\/atom+xml", "extensions":{ }, "time":"2008-07-17t15:18:29-05:00", "entry":[ { "content":"<oai_dc:dc xsi:schemalocation=\"http:\/\/www.openarchives.org\/oai\/2.0\/oai_dc\/ \n http:\/\/www.openarchives.org\/oai\/2.0\/oai_dc.xsd\" xmlns:dc=\"http:\/\/purl.org\/dc\/elements\/1.1\/\" xmlns:oai_dc=\"http:\/\/www.openarchives.org\/oai\/2.0\/oai_dc\/\" xmlns:xsi=\"http:\/\/www.w3.org\/2001\/xmlschema-instance\"><dc:creator>conrad, joseph,<\/dc:creator><dc:title>chance<\/dc:title><\/oai_dc:dc>", "created":"2008-03-18t15:56:52-05:00", "updated":"2008-03-18t15:56:52-05:00", "author":{ "name":"conrad, joseph," }, "content_type":"application\/xml", "links":{ "alternate":[ { "type":"application\/atom+xml;format=marc", "href":"\/resources\/4107?record_format=marc" }, { "type":"application\/atom+xml;format=marcxml", "href":"\/resources\/4107?record_format=marcxml" }, { "type":"application\/atom+xml;format=dc", "href":"\/resources\/4107?record_format=dc" }, { "type":"text\/html", "href":"http:\/\/catalog.jangle.org\/shared\/biblio_view.php?bibid=4107&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;tab=opac" } ], "related":[ { "type":"application\/atom+xml", "href":"\/collections\/2" } ], }, "title":"chance", "id":"\/resources\/4107" } ], "request":"\/resources\/4107?record_type=oai_dc" } figure 2: the connector's json response <feed xmlns="http://www.w3.org/2005/atom"> <title>openbiblio</title> <link href="http://demo.jangle.org/openbiblio/resources/4107?record_format=oai_dc"/> <updated>2008-07-17t15:25:16z</updated> <id>http://demo.jangle.org/openbiblio/resources/4107?record_format=oai_dc</id> <entry> <title>chance</title> <link href="http://demo.jangle.org/openbiblio/resources/4107"/> <link type="application/atom+xml" href="http://demo.jangle.org/openbiblio/collections/2" rel="related"/> <link type="application/atom+xml;format=marc" href="http://demo.jangle.org/openbiblio/resources/4107?record_format=marc" rel="alternate"/> <link type="application/atom+xml;record_format=marcxml" href="http://demo.jangle.org/openbiblio/resources/4107?record_format=marcxml" rel="alternate"/> <link type="application/atom+xml;record_format=dc" href="http://demo.jangle.org/openbiblio/resources/4107?record_format=dc" rel="alternate"/> <link type="text/html" href="http://catalog.jangle.org/shared/biblio_view.php?bibid=4107&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;tab=opac" rel="alternate"/> <author> <name>conrad, joseph,</name> </author> <id>http://demo.jangle.org/openbiblio/resources/4107</id> <updated>2008-03-18t15:56:52-05:00</updated> <content type="application/xml"> <oai_dc:dc xsi:schemalocation="http://www.openarchives.org/oai/2.0/oai_dc/ http://www.openarchives.org/oai/2.0/oai_dc.xsd" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:oai_dc="http://www.openarchives.org/oai/2.0/oai_dc/" xmlns:xsi="http://www.w3.org/2001/xmlschema-instance"> <dc:creator>conrad, joseph,</dc:creator> <dc:title>chance</dc:title> </oai_dc:dc> </content> </entry> </feed> figure 3: a simple atom serialization from the jangle core with oai-dc metadata content the atom serialization while connectors provide json responses to the jangle core, the jangle core serializes this response into atom syndication format. jangle uses a few of these elements in a very specific manner. as mentioned earlier, connectors define the relationships between base entities. these relationships are handled via <link> elements within the atom entry. the url for the items associated with a particular resource (after being rewritten to atom by the jangle core) might look like http://jangle-core.example.org/example_service/resources/12345/items/ and would return an atom feed of the items associated with resource 12345. the reverse is also true, where http://jangle-core.example.org/catalog/items/987654/resources/ would return an atom feed with the resource record associated with item 987654. the benefit of these <link> elements is that users (people or programs) can discover related information, not just the information that was asked for. if an initial request was for resource (bibliographic record) x, the connector also makes available a link to all the item records associated with that resource (see orange highlighting in fig. 4). the atom entries also provide alternate representation links to other metadata formats available via jangle (see yellow highlighting in fig. 4). in an effort to help client applications identify the different metadata formats offered (and to comply with atom syndication specification), the mime-type of alternate links (application/atom+xml) is extended with the record_format parameter to declare what will be transported in the atom:content field. for example, if the atom feed is returning marc xml in its entry content fields, the mime-type returned would be application/atom+xml;record_format=marcxml. as these values are arbitrary strings, it would be helpful for jangle and other initiatives to produce a skos vocabulary to provide consistent identifiers ensuring agreement on what these strings actually mean. the vocabulary would not be exclusive to jangle, but available to any applications or specifications that require agreement on metadata formats, such as unapi [21]. <feed xmlns="http://www.w3.org/2005/atom"> <title>openbiblio</title> <link href="http://demo.jangle.org/openbiblio/resources/5975?record_format=marcxml"/> <updated>2008-07-17t15:27:57z</updated> <id>http://demo.jangle.org/openbiblio/resources/5975?record_format=marcxml</id> <entry> <title>the works of samuel johnson in nine volumes volume iv: the adventurer; the idler</title> <link href="http://demo.jangle.org/openbiblio/resources/5975"/> <link type="application/atom+xml" href="http://demo.jangle.org/openbiblio/resources/5975/items/" rel="related"/> <link type="application/atom+xml" href="http://demo.jangle.org/openbiblio/collections/2" rel="related"/> <link type="application/atom+xml;record_format=marc" href="http://demo.jangle.org/openbiblio/resources/5975?record_format=marc" rel="alternate"/> <link type="application/atom+xml;record_format=dc" href="http://demo.jangle.org/openbiblio/resources/5975?record_format=dc" rel="alternate"/> <link type="text/html" href="http://catalog.jangle.org/shared/biblio_view.php?bibid=5975&tab=opac" rel="alternate"/> <author> <name>johnson, samuel,&t;/name> </author> <id>http://demo.jangle.org/openbiblio/resources/5975</id> <updated>2008-03-18t15:57:00-05:00</updated> <content type="application/xml"> <record xmlns='http://www.loc.gov/marc21/slim'> <leader> z 22 4500</leader> <datafield tag='245' ind1='0' ind2='0'> <subfield code='a'>the works of samuel johnson in nine volumes volume iv: the adventurer; the idler</subfield> <subfield code='c'>by samuel johnson</subfield> <subfield code='h'>[electronic resource] /</subfield> </datafield> <datafield tag='100' ind1='0' ind2='z'> <subfield code='a'>johnson, samuel,</subfield> <subfield code='d'>1709-1784</subfield> </datafield> <datafield tag='42' ind1='z' ind2='z'><subfield code='a'>dc</subfield></datafield> <datafield tag='260' ind1='z' ind2='z'> <subfield code='b'>project gutenberg,</subfield> <subfield code='c'>2004</subfield> </datafield> <datafield tag='500' ind1='z' ind2='z'><subfield code='a'>project gutenberg</subfield></datafield> <datafield tag='506' ind1='z' ind2='z'><subfield code='a'>freely available.</subfield></datafield> <datafield tag='516' ind1='z' ind2='z'><subfield code='a'>electronic text</subfield></datafield> <datafield tag='830' ind1='z' ind2='z'> <subfield code='a'>project gutenberg</subfield><subfield code='v'>12050</subfield> </datafield> <datafield tag='856' ind1='z' ind2='z'> <subfield code='u'>http://www.gutenberg.org/etext/12050</subfield> </datafield> <datafield tag='856' ind1='z' ind2='z'> <subfield code='u'>http://www.gutenberg.org/license</subfield> <subfield code='3'>rights</subfield> </datafield> </record> </content> </entry> </feed> figure 4: atom serialization highlighting the link relationships and marcxml content (blue highlighting) since each collection of entities may contain many hundreds of thousands to millions of resources, feeds from the base entity urls (i.e. /actors/ or /items/) are paged according to rfc 5005 [22]. paging is also available when searching entities via opensearch. since one of the features of atom is the expectation that it will be extended, the json responses from each connector include a hash to specify namespaces and namespace prefixes used in the entries. it is possible the json objects will use a standardized xml style notation (such as badgerfish [23]) by the time of an official jangle api specification release, but discussions have only just begun of the possibilities. the jangle api itself makes no recommendations or requirements regarding the data that is available for any given system, in large part since the data may vary wildly between different kinds of library systems. an interlibrary loan system and an archival finding aid manager may have very few overlapping data points. instead, the expectation is that data requirements can be delegated to community profiles or best practices for a given application domain. this way, connector implementers working with integrated library systems can determine the appropriate responses and the most practical way of utilizing collections without worrying about how their decisions apply to other library systems. jangle’s job is merely to expose data. it is up to the communities to decide the best way to utilize jangle to meet their needs. adapters: since atom’s not for everybody the uniformity of jangle’s api, regardless of the variations in backend systems it provides access to, allows for a third tier, called adapters, to sit in front of a jangle core implementation and provide more traditional library interfaces, such as oai-pmh or ncip. if a single sru or ncip adapter can be built on top of jangle, every library system with a jangle connector can make use of this adapter. this would enable the implementation of traditional library standards across a far greater percentage of library systems. the reality, of course, may not be this quite this simple, but economies of scale make the development of particularly complicated standards-based protocols much more immediately beneficial. jangler: the jangle reference implementation since jangle is still very much a proof of concept and a work in progress, it was important to have a simple reference implementation to test api proposals, demonstrate jangle’s potential and provide a basic atompub server and connector framework for developers to build their own connectors against. the current jangle reference implementation is known as jangler. ruby was chosen for this application, mainly to provide a working prototype very quickly. a side benefit of this decision was the ability to build the reference core and connector architectures on the sinatra framework [24]. sinatra is well suited to rest interfaces because it allows the developer to set a particular action to happen at a specific route (or url path). different actions for the same route can be defined depending on the http method used to call the path. layers of path hierarchy are then quite simple, since they effectively just map to a particular method. since the logic of dealing with the requests is all handled by the framework, the core and connector frameworks are quite small, measuring about 150 lines and 360 lines of ruby, respectively. also, an example connector was needed for testing in the reference implementation. we chose to build a connector for the openbiblio open source integrated library system [25] because the openbiblio ils was simple; it ran on cheap, shared web hosting, required little maintenance or administration to run, and, most importantly, provided a fairly good representation of the data structures within the more sophisticated commercial and open source library systems without the overhead required to implement them. the database was seeded with marc records from project gutenberg to represent a library collection. the borrowers were populated with the names and addresses of the representatives and senators of the 110th congress of the united states. a script was written to assign a random number of item records to each bibliographic record (with the possibility of zero), set a current circulation status, and, if that status was “checked out”, assign the item to a borrower at random. the openbiblio connector uses the jangler connector framework. it defines get (read only) interfaces for accessing actors, resources, items and collections. actors return their data using the popular vcard format. resources may be returned as marc xml, marc 21 or oai dublin core. an opensearch interface is also available for resources, allowing clients to search on particular fields, such as title, author, subject and date modified. items are offered in the simpleavailability format [26] defined in the recommendation of the digital library federation’s integrated library system discovery interface (dlf ils-di) task group and in the document availability information api (daia) format [27]. other holdings and availability formats will be available in the future. collections for this connector are currently groups of resources as set in openbiblio itself (such as “adult non-fiction”). in an effort to show the relationship between jangle and the dlf ils discovery interface task group’s recommendation, an ils-di adapter providing level 1: basic discovery interfaces functionality [28] was also written as part of jangler. this illustrates the role of adapters within the jangle framework as, theoretically, this adapter should work on top of any jangle implementation with an integrated library system connector. the adapter combines open archives initiative protocol for metadata harvesting [29] (oai-pmh) to provide the harvestbibliographicrecords and harvestexpandedrecords functionality with simple cgi-style interfaces to provide the getavailability and gotobibliographicrequestpage methods. for getavailability, the adapter uses the item entity directly. for gotobibliographicrequestpage, jangle’s resource entity provides a <link> element to a text/html representation; the adapter parses the record and redirects the client to that url. the frameworks in jangler are released under the gnu public license version 2 (gplv2), meaning they are freely available and the source open for review or adaptation [30]. the connectors built using the framework do not have to conform to gplv2, since the connector logic exists in files outside of the framework itself. the strong copyleft license allows other developers to port the framework to other languages easily by ensuring that the code is always available to see. proof-of-concept unicorn connector (pcuc) connectors are simple restful interfaces that define five base paths (/services, /actors, /resources, /items and /collections) and return their results as json hashes. the json hashes also include the mime-type that the jangle core should serialize the response as. a proof-of-concept unicorn connector (pcuc) has been written by james farrugia at drew university. so far, only certain uri patterns have been implemented, based on three of the five base paths (/services, /resources, and /items). the pcuc was written to test the feasibility of implementing a jangle connector for unicorn (a commercial ils provided by sirsidynix), and it deals only with read access to certain bibliographicand item-level information. specifically, the following uri patterns have been implemented for accessing resources: /services – this uri indicates which services are offered. for now, services pertaining only to resources and items are offered – i.e., no services are currently offered for actors or collections. /resources – this uri is currently hard-coded to return a list of the first twenty bibliographic records. /resources/i – this uri returns the bibliographic record associated with unique record identifier i, which is a positive integer, ranging from 1 to the number of resources (bibliographic records) in the collection at drew university (roughly 570,000). if the integer i is larger than the number of resources in the catalog, the server returns a 404 error. /items/b – this uri returns item information for the item identified by the unique identifier “b” (for a book this is usually a barcode). /resources/i/items – this uri returns item information for all the items associated with a given resource (bibliographic record) uniquely identified by the integer i. examples of uri patterns that have not yet been implemented are: /resources/i1,i2 – given a list of bibliographic identifiers, return information about multiple bibliographic records. /resources/si1-i2 – given a range of bibliographic identifiers, return information about multiple bibliographic records. /items/b1,b2 – given a list of item identifiers, return information about multiple item records. /resources/b1,b2/items/ – given a list of bibliographic identifiers, return information about all the item records associated with the bibliographic records identified by b1 and b2. some details on how the pcuc is implemented we implemented the proof-of-concept unicorn connector using apache’s mod_rewrite and rewrite rules in .htaccess (fig. 5) to invoke particular php scripts on the unicorn server. these php scripts do two principal tasks: interact with the unicorn api to retrieve information relevant to the http get request, and then format this information into a json string that uses a jangle-specified syntax. the json string, with an appropriate mime type, is sent back to the jangle core, which then appropriately serializes the data to its proper xml representation (such as an atom feed or service document). # for one bib rec: http://myserver.mydomain/resources/12345 rewriterule ^resources\/([0-9]+)$ jangle/resourcesonebibid.php?entity=resource&idtype=resource&id=$1 # for one item rec: http://mysever.mydomain/items/3114005851463 rewriterule ^items\/([0-9]+)$ jangle/itemsoneitemid.php?entity=item&idtype=item&id=$1 figure 5: example rewrite rules from a pcuc .htaccess file conceptually, the connector implements a simple and straightforward process: it takes the parameters from the uri, uses those parameters to interact with unicorn, extracts information from unicorn, parses that information into jangle-specified json syntax, and sends everything back with a json mime type. in reality, it took a bit of doing to get everything working, but all the doing was done in about 2 weeks. for pragmatic and institutional reasons, we used php on a windows 2003 machine that is drew university’s unicorn test server. in addition to invoking a recent unicorn api tool (available as of release gl 3.1), we also needed a way to convert our marc records to marc xml. we chose the freely available yaz-marcdump program from indexdata [31]. then we needed two other capabilities: the ability to convert xml to json within php, and the ability to work with a json string as a php associative array (i.e., a hash). the first one was found in some off-the-web php software [32]; the second one is built in to php itself. finally, we needed to parse and reformat a variety of time stamps, ranging from the current system time, to the marc control field 005 time, to the form of the time returned by unicorn for item creation and due dates. getting the details correct for these different timestamps took a particularly long time. the following factors made writing the pcuc take more time than it otherwise might have: it took some time to understand the syntax and contents of the different json strings that the jangle core expects. although the overall syntax was similar for the different uri patterns, each implemented uri pattern required slightly different contents in its json string. the two authors spent a fair amount of time going back and forth with each other to clarify, test, and implement the expected json coming from the unicorn connector. the developer’s knowledge of jangle’s conceptual framework was very limited at the start of the project. the following factors made writing the pcuc take less time than it otherwise might have: the developer of the unicorn connector had experience working with the unicorn api from previous unicorn-related work that was done for the vufind project [33]. we were able think of the problem in easy conceptual terms: when a uri of this form hits your server, send back a json string, with mime type json, that looks like so. we had concrete examples from the openbiblio connector for the jangler reference implementation, which we could use to understand and model our json strings. we used apache and the rewriterules it makes available in mod_rewrite. having done this work, we have a basic proof-of-concept unicorn connector for jangle. furthermore, we are confident that it would be not be difficult to write additional unicorn connector scripts that would implement additional uri forms for resources and items. finally, if writing the unicorn connector components for actors and collections proves to be essentially no more difficult than writing the proof-of-concept for resources and items has been, then there is good reason to believe that a full-blown unicorn connector for jangle can indeed be written for get requests. as for put, post, and delete requests utilized for updating and deleting resources via atompub, a unicorn connector for resources, items, actors, and collections would need to access unicorn’s api in ways that were not done with the pcuc. specifically, api calls for put, post, and delete, which are different from the api calls used for get requests, would need to be able to change unicorn data, rather than simply read it. unicorn does have api calls that a delete request, for example, could invoke to delete a resource and all its associated items. unicorn also makes available other api calls that can be used to change data on the unicorn server. whether or not unicorn has available all api functionality necessary for any put, post, or delete request in the jangle framework has yet to be investigated. we should also point out that there are definitely security issues involved with http requests that can change data — for instance, we would not want to give permission to just any web service to delete resources. there may also be sirsidynix proprietary issues related to the use of api commands that do more than just read information from unicorn. possible jangle connectors for various kinds of systems the integrated library system certainly one of the most important and, potentially, common uses for jangle will be to expose the data from the integrated library system. this is useful for a variety of tasks, from providing bibliographic and availability data to discovery interfaces to exposing borrower account information to a campus portal or financial system. entity types of data included possible formats available actors borrowers vcard, foaf, finance formats for fines resources bibliographic records, subject and name authorities marc21, marc xml, dublin core, mods, mads items physical items, serials holdings iso 20775, marc 21 for holdings data, dlf simpleavailability, ncip collections bibliographic records, authorities, distinctions between item types (serials, monographs, government documents, etc.), course reserves defers to the entities in the collection a discovery system could stay up to date by polling the atom feed for resources at various intervals for changes. targeted discovery systems could be created by harvesting the feeds of distinct collections, allowing for very specific interfaces to resources such as serials or videos. changes to the borrower’s account in the ils could be propagated in real time as users update address information or pay fines in the central portal system or courseware. interlibrary loan system most of the uses surrounding a jangle interface to an interlibrary loan system would likely revolve around providing borrowers with status information regarding their requests or delivery of their requests in an electronic format. it would also be possible to provide more sophisticated reports of materials requested and, depending on the system and the library policies, even allow for the creation of requests from within other applications. entity types of data included possible formats available actors borrowers, lending institutions, borrowing institutions vcard, foaf, finance formats for fines resources interlibrary loan requests iso ill, openurl metadata formats, dublin core items loaned items circulation and location information, pdfs collections have a less obvious analog in this context and would probably be more usefully applied for local needs. because items would be associated with actors and resources, a feed of the status of the user’s request could be displayed in their account in other systems, such as the opac or central portal. institutional repository although a jangle interface to an institutional repository would probably overlap with the goals of the sword project (more on this in “how jangle gets along with its neighbors” section), sword does not define most of the activity involved with interacting with a repository. jangle would most likely defer to sword for dealing with submission to an archive and, instead, focus on making the data available. entity types of data included possible formats available actors submitters, authors vcard, foaf resources archived object metadata dublin core, bibtex, endnote, ris, biblioontology, premis items electronic formats available pdf, postscript, ms word, excel, scorm learning objects collections departments, subjects, special collections dublin core to describe the collection, defer to the entities within the collection feeds could then be provided by author or department to include in other applications. discovery systems could harvest the collections relevant to their domain. course reserves management system the most obvious use case for a jangle interface to a course reserves system would be to include feeds of class readings in courseware or a central portal. another scenario would be to enable instructors to add reserves from other interfaces, such as a link resolver, metasearch tool, or courseware. entity types of data included possible formats available actor instructors, students foaf resources journal articles, book chapters, annotations, instructions openurl metadata formats, dublin core, atom items scanned copies, circulating copies at reserves desk circulation and location information, openurl link, pdf collections course information atom to describe classes, defer to entities within the collection in the case of course reserves, it is easy to see actors related to collections, collections related to resources and resources related to items. circulation information for a physical reserve item could be pulled directly from a jangle connector for the ils if the uri is provided. the ils could also feed the reserves system as items are placed on reserves status in the catalog. how jangle gets along with its neighbors since there are other initiatives in place that share similarities with jangle, it seems appropriate to see how jangle fits in with them, if there are any conflicts or overlaps and how they may be resolved. sword sword [34] (simple web-service offering repository deposit) is a protocol developed to manage the submission of objects to repository systems. the project has the backing of the major open source repository platforms (dspace, fedora and e-prints) and uses the atom publishing protocol as the basis of its api. sword, however, only defines the post method for submitting objects since its purpose is to manage the ingestion of resources into repositories. retrieval, update and delete are out of scope for the initiative, so jangle could, theoretically, provide that functionality. jangle, in turn, could simply rely on sword for defining post behavior in a repository community profile and provide guidance for specifying the creation of resources across the entire specification. the sword and jangle projects can be mutually beneficial for one another. oai-ore oai-ore [35] (open archives initiative-object reuse and exchange) is a protocol for sharing collections of resources and the graph of the relationships between them for minting a uri to define that graph. oai-ore does this via an rdf graph of aggregated resources called a resource map. while oai-ore does define an atom syndication feed serialization, there are few other similarities between this protocol and jangle. jangle does not define aggregations in the same context as oai-ore and only defines the relationships between internal jangle entities in a very simplistic model. oai-ore has no capacity for creating, updating or deleting resources, only providing read access to data. these two initiatives appear to be addressing fairly different problems. digital library federation integrated library system discovery interface task group the digital library federation’s integrated library system discovery interface task group (ils-di) [36] produced a formal api recommendation describing methods to expose data from and interact with integrated library systems. the recommendation lays out a series of functions and defines their purpose and role. instead of being a turnkey protocol, the ils-di api takes a best of breed approach, suggesting protocols such as oai-pmh or openurl to meet the functional requirements. the recommendation makes no mandates as to how the functionality should be achieved. different implementations can use different protocols to provide the same functionality. the relationship between jangle and the ils-di is an unusual one. jangle has identified the ils-di api as a primary use case influencing its development. instead of directly competing, the role of jangle with regards to the api is that of the enabler. since the interfaces needed to meet the goals of the ils-di api do not currently exist in many integrated library systems, jangle could quite easily fill that role. since a jangle adapter should be able to work across many jangle implementations, a single ils-di adapter could establish a de-facto standard for how specific methods are implemented. in short, jangle is trying to provide the underlying api upon which the ils-di api can be built across all integrated library systems. the current state of jangle the jangle community decided to first specify read requests (http get) for all of the base entities before moving into the more complicated write functionality. the connector apis have been fairly stable although they need to be vetted against multiple implementations. since the connector api is limited to basic atom, a means of extending the api needs to be formalized before the first jangle specification is released. this way connector providers can tap into versatility of atom extensions, and if there is a standardized notation for how to parse them, the jangle core does not have to keep up with the semantics of each and every extension (especially ad-hoc ones) and can instead, directly translate the json into xml. once jangle is working satisfactorily to communicate between at least three different library systems and a consumer application (such as an opac replacement), the api will be written up as a formal specification document and released as 1.0. currently, there is the openbiblio connector and the proof-of-concept unicorn connector and work has begun on a talis alto [37] connector. development of the specification takes place on jangle’s website, the mailing list [38], and the project page for the jangle development repository [39]. the community votes on all major decisions and periodically does “approval polls” to make sure the project has not veered beyond the community’s vision or expectations. the process, as well as the community, is open to anyone interested in participating. when the specification is stable for the 1.0 release, a php implementation of the connector framework will also be released as open source. the php framework is considered critical to the success of the project, since it the language of choice in many libraries and is so easily deployed. tomorrow and beyond after get requests have stabilized and the first api specification document has been released, the community will vote on the next http method to focus on (post, put, delete, etc.). it is also possible that the community will decide to focus on implementing all methods for a particular underlying system. while development of the protocol has been quite swift during this iteration, it is expected the release cycles will slow considerably when changes to the local data stores are introduced. in the near future, the ideal would be to replace the jangle core with a more established atompub server application, such as apache’s abdera [40]. since jangle in its current state does nothing to address authentication or authorization (and the specification will only mention that security is at the discretion of the implementor), piggybacking on an existing project that already has these features would be pragmatic and relieve some of the development burden. since development of jangle really takes place at the connector level, it is desirable to have connector frameworks in a variety of major languages: java, perl and python (since ruby and php should already be addressed). it would also be desirable to supply a suite of connectors to various open source applications beyond integrated library systems: institutional repositories, interlibrary loan, course reserves, etc. to help guide the community profile development for various kinds of systems. this would also be the stage for soliciting vendor support for jangle by showing how easy it is to implement and how quickly the vendor could meet the demands of initiatives such as the dlf ils-di. in fact, by implementing jangle, it is quite possible that vendors could rely on their customers and competitors to provide them with the functionality needed to provide various protocol interfaces. and back to reality jangle is by no means a panacea for all of the woes of library interoperability. it cannot make free text holdings information magically machine parseable. it requires considerable agreement on compatible data formats and jargon. it cannot overcome politics and differences in policy that will still cause headaches for the developers tasked with designing a system that works across institutions. it does however make library data available using a simple mechanism that can be accessed with cheap, commodity client libraries. no knowledge of library data formats or transfer protocols is necessary. rather, once connectors are built (which would require knowledge of the underlying library applications), software clients outside the library world need only the knowledge of a commonly used programming language and standardized web protocols in order to access and make library data available. these same clients can be used to create, edit and delete resources. the whole universe of library data, liberated from the arcane world of proprietary library applications, will be available for developers outside the library domain. notes [1] http://jangle.org [2] http://www.loc.gov/z3950/agency/ [3] http://ncip.envisionware.com/ [4] http://www.niso.org/publications/rp/rp-2006-01.pdf [5] http://www.loc.gov/standards/sru/ [6] http://www.niso.org/apps/group_public/workgroup.php?wg_abbrev=mi [7] http://www.oasis-open.org/committees/tc_home.php?wg_abbrev=search-ws [8] for an introduction to restful design, see http://en.wikipedia.org/wiki/restful and http://www.infoq.com/articles/rest-introduction [9] http://www.talis.com/ [10] http://www.talis.com/integration/ [11] http://bitworking.org/projects/atom/rfc5023.html [12] http://www.atomenabled.org/developers/syndication/atom-format-spec.php [13] http://www.loc.gov/standards/marcxml/ [14] http://www.loc.gov/ead/ [15] http://www.loc.gov/marc/ [16] http://code.google.com/more/#products-dataapis-gdata [17] http://www.opensearch.org/home [18] http://codex.wordpress.org/atompub [19] http://www.movabletype.org/documentation/developer/api/atompub/ [20] http://json.org [21] http://unapi.info [22] http://www.ietf.org/rfc/rfc5005.txt [23] http://badgerfish.ning.com [24] http://sinatrarb.com [25] http://obiblio.sourceforge.net [26] http://onlinebooks.library.upenn.edu/schemas/dlfexpanded.xsd [27] http://www.gbv.de/wikis/cls/document_availability_information_api [28] http://diglib.org/architectures/ilsdi/ [29] http://www.openarchives.org/pmh/ [30] http://code.google.com/p/jangle/source/checkout [31] http://www.indexdata.com/yaz/doc/yaz-marcdump.tkl [32] http://www.ibm.com/developerworks/xml/library/x-xml2jsonphp/ and http://pear.php.net/pepr/pepr-proposal-show.php?id=198 [33] http://www.vufind.org [34] http://www.ukoln.ac.uk/repositories/digirep/index/sword [35] http://www.openarchives.org/ore/ [36] http://diglib.org/architectures/ilsdi/ [37] http://www.talis.com/alto/ [38] http://groups.google.com/group/jangle-discuss [39] http://code.google.com/p/jangle [40] http://incubator.apache.org/abdera/ about the authors ross singer is interoperability and open standards champion at talis. his blog can be found at http://dilettantes.code4lib.org/. james farrugia currently works as systems librarian at drew university. he can be reached at jfarrugi@drew.edu tags: api, app, atom, atompub, jangle subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – technical challenges in developing software to collect twitter data mission editorial committee process and structure code4lib issue 26, 2014-10-21 technical challenges in developing software to collect twitter data over the past two years, george washington university libraries developed social feed manager (sfm), a python and django-based application for collecting social media data from twitter. expanding the project from a research prototype to a more widely useful application has presented a number of technical challenges, including changes in the twitter api, supervision of simultaneous streaming processes, management, storage, and organization of collected data, meeting researcher needs for groups or sets of data, and improving documentation to facilitate other institutions’ installation and use of sfm. this article will describe how the social feed manager project addressed these issues, use of supervisord to manage processes, and other technical decisions made in the course of this project through late summer 2014. this article is targeted towards librarians and archivists who are interested in building collections around web archives and social media data, and have a particular interest in the technical work involved in applying software to the problem of building a sustainable collection management program around these sources. by daniel chudnov, daniel kerchner, ankushi sharma and laura wrubel introduction over the past two years, the george washington university libraries developed social feed manager, a python and django-based application for collecting social media data from twitter. [1] the project began as a research prototype to support researchers on the gwu campus studying twitter, replacing manual and time-consuming processes to collect and reformat data with an automated approach. over the past year, supported by an institute of museum and library services sparks! ignition grant, the libraries have refined and expanded the tool’s capacity and portability to make it useful for other institutions.  in doing so, the development team has encountered a number of technical challenges.  this paper will describe how the team handled changes in the twitter api; meeting researcher needs for groups or sets of data; management, storage, and organization of collected data; using supervisord to manage simultaneous streaming processes; improving documentation to facilitate other institutions’ installation and use of sfm; and upcoming work on the sfm application and the integration of social media data into archival collections, descriptive standards, and policies. the changing twitter api in early 2013 twitter changed its api in several important ways [2]: previously, some api queries had been allowed without authentication; now all requests require authentication access to the easy-to-consume syndication feed format for user timelines is no longer available new api request rate limits were implemented across all api calls this combination of changes might seem reasonable to a software developer. for the most part, the same information is available from the api after these changes, and adapting tools to abide by the new api was just a small matter of code.  however, to some researchers not experienced with software development, these changes made a once free-flowing source of research data much more difficult to access. on the gw campus, for example, a faculty researcher in the school of media and public affairs (smpa) had previously studied the use of twitter by major media outlets [3].  the researcher’s workflow used prior to the api changes was to: identify approximately three dozen twitter accounts of interest subscribe to the rss feeds for each using google reader [4] copy-and-paste individual tweets from google reader into spreadsheets although this painstaking process was suboptimal and not one the researchers wished to repeat, they were indeed able to complete their investigation with the twitter api v1.  however, the twitter api v1.1 deprecated the previously available rss feeds, and the new api shifted the security model so that all calls to the api required authentication, removing the ability to anonymously request query data. the libraries first began working with researchers in 2012, developing methods to automate the data collection process for the researcher.  this work resulted in a script that took a list of twitter handles as input, processed their rss feeds, and converted the results into a csv file for later use with excel, stata, or other applications familiar to social science researchers.  upon learning of the upcoming api changes, the development team switched to using a database-backed list of handles and the easy-to-use tweepy [5] python library for access to the twitter api.  among many python libraries evaluated, tweepy had excellent coverage of the old api and an active development branch for the new api well before twitter disabled the old api. the development team tested this work with a second smpa researcher studying congressional use of social media.  the tests consisted of feeding a list of hundreds of u.s. congress member twitter accounts to the application prototype.  the application immediately ran into the new twitter api rate limits.  every twitter api call has its own “window” of an allowable number of requests within a specific time interval; a volume of requests that exceeds this allowable number within that time interval will return rate limit exceeded errors from the twitter api.  the “get statuses/user_timeline” api [6] call, for example, allows up to 180 requests per 15 minutes per user and up to 300 requests per 15 minutes per authorized application.  because of this, we spread out calls to the api over time.  to make 535 calls to statuses/user_timeline, for example, we would need to spread those calls over at least a 45-minute period, or a minimum of three rate-limit windows. fortunately, the twitter api embeds rate limit information in its http response headers: x-rate-limit-remaining: how many calls remain in the current window x-rate-limit-reset: when the current window resets in sfm’s set_wait_time function we calculate a wait time between calls to spread requests across the remaining time in the current window, buffered by a few seconds to account for network delays and to keep our code’s use of the api gentle. def set_wait_time(last_response): """based on last tweepy api response, calculate a time buffer in seconds to wait before issuing next api call.""" wait_time = 0 try: remaining = int(last_response.getheader('x-rate-limit-remaining')) reset = int(last_response.getheader('x-rate-limit-reset')) except: remaining = reset_seconds = 1 # the out-of-calls-for-this-window case if remaining == 0: return reset_seconds + wait_buffer_seconds else: wait_time = (reset_seconds / remaining) + wait_buffer_seconds # #22: saw some negative ratelimit-reset/wait_times # so cushion around that too while wait_time < wait_buffer_seconds: wait_time += wait_buffer_seconds return wait_time the development team soon learned the importance of using this function all the time; for any process that makes multiple calls to a rate-limited api like twitter’s, skipping a step like adding a wait time between requests almost guarantees rapid exhaustion of available calls within a window.  because of this, every function in sfm that might make repeated calls to the api uses set_wait_time to spread calls over time.  a side benefit of spreading out calls means that any process run will determine a wait buffer for itself based on the rate limit information in the api response, so when multiple processes are run in parallel, each will slow itself down accordingly, making it less likely that one process will starve other processes of calls within a rate limit window. exporting and reformatting data for researchers in response to requests from campus researchers, social feed manager supports a number of different methods of exporting and working with the captured data.  the simplest use case is a csv or excel extract from raw json tweet data for a single twitter user account.  because this use case is so common, we have added csv and excel export links as features of the researcher ui.  every captured twitter account has its own page in sfm for browsing basic statistics and summaries of their captured tweets.  a campus user who has authenticated to sfm through twitter’s authentication system can use links on that page to export the data as csv or as excel.  sfm responds to the link with a streamed file in an http response that includes headers that should trigger a browser to open microsoft excel on most pcs.  each row in the export file contains a subset of the values contained in each tweet; the subset was selected based on researcher feedback and is documented in a data dictionary. [7] in response to additional researcher requests, sfm allows two variations on the export.  multiple twitter accounts may be grouped into sets — for example, members of the us congress, or individual journalists from the washington post, or gwu student groups — for which a single csv or excel extract can be generated.  however, these files can grow too large to efficiently stream via the browser, so we developed a command line function for this purpose, implemented as a django management command.  it allows a system administrator to generate a csv or excel export for a given user or a named set of users, and it also allows a date range to be specified to limit the extract.  for example, some gwu researchers have wanted a one-week sampling of tweets from a set of users; the command line function, “export_csv” [8], allows us to respond to those requests quickly. managing millions of tweets sfm also provides the capability to continuously collect live tweets using the twitter streaming api’s “statuses/filter” [9] and “samplestream” commands [10].  because this can quickly amass tweets, sfm writes these tweets to files rather than storing them in the database, and csv and excel are no longer feasible or useful export formats. for a computer science researcher, we extracted tweets that contained certain keywords for a period of several hours.  this generated millions of tweets, so rather than accumulating a small number of multi-gigabyte files, we developed a rolling file handler that starts new files at configurable intervals (defaulting to every 15 minutes).  these smaller files, with names including timestamps to differentiate them, can grow to over a dozen megabytes in size each, even when compressed, so handling storage and transfer of them requires thoughtful planning, like with any digital collection of a similar size. to arrange the files on disk, we use another command line function called “organizedata” to split the files out into a year/month/day/hour hierarchy of folders [11]. when we went to transfer this data to the researcher, the data comprised multiple terabytes in this instance.  to facilitate the transfer, we took a simple path of bundling up the hierarchy of compressed files of raw tweets in json into a bagit bag [12], with checksums for every individual file, packaged into a single file using the linux “tar” command. aside from using a usb drive over sneakernet — which would have worked — we found it a modest challenge to transfer this somewhat large dataset to the researcher using a network. none of the obvious cloud-hosted file sharing platforms supported file sizes over a few terabytes, and a simple web page or bittorrent file wouldn’t help because we only wanted to deliver the file to one person. we settled for a different simple solution, using the linux “split” command to cut the large tar file up into a handful of byte size-limited pieces, alongside additional checksums for each, and uploaded the set to cloud storage. we then shared access information, credentials, and basic instructions for file reassembly with the researcher, who found the result easy to receive and implement. it is possible that we will find this to be a common use case as we engage more often in sharing of large datasets. some recipients will not be comfortable following a readme file with sample linux commands to run and we will need to define a smoother path for sharing large files discreetly. for now, however, this has worked well. managing simultaneous streaming processes capturing a filtered stream of tweets from the twitter api has important considerations: a single authorized twitter user account may only hold one filter open at a time; a stream can generate a lot of data quickly; and streams may be interrupted. possible interruptions include network issues, a second simultaneous stream, or other system issues like upgrades and restarts. for this reason, streams need to be regularly monitored to ensure uptime. we sought a technical approach that allowed multiple streams to be managed in parallel using different authentication parameters, with a simple strategy for separating generated files and with a long-term monitoring solution in place that enabled automated starting, stopping, and restarting of streaming as appropriate. the development team considered writing a daemon manager suited to our particular needs, but reconsidered given the scope and scale of the project. we also considered using operating system-level tools for managing linux initscripts, but found that to be less than ideal for two reasons: we wanted an operating sfm user such as a librarian or archivist to be able to create, start, and stop stream filters through the sfm ui, and as a userspace application we did not want the sfm process to have privileges sufficient to write, edit, delete, and restart system-level files and processes. in between these two extremes, we found a working solution in supervisord [14], a process manager for users and applications, written in python. use of supervisord is not required for gathering sets of individual user timelines from twitter or even running a single filter stream. because of this, installing and configuring supervisord is optional for all sfm users. configuring, managing, and troubleshooting a linux process manager arguably requires the support of a staff person with experience administering linux systems, or at least willing to learn, because there are many things that can go wrong. using supervisord requires careful configuration of system scripts, users, and permissions, along with its own application-level configuration with scripts, users, and permissions.  even so, with careful attention to the documentation [15] many sfm users should be able to get up and running with supervisord and to use it to capture multiple parallel streams. when correctly configured, the supervisord-managed approach works well. a non-technical staff member using sfm can log in to sfm’s administration ui, add a new twitterfilter [16], and sfm will generate a supervisord configuration file for the filterstream command with the correct user information for authorization. supervisord will start a process immediately, storing streamed tweet data in a directory set aside for this twitterfilter using the sfm database id of the twitterfilter in the name of the configuration file and the name of the directory (e.g. “twitterfilter-12”) to contain captured tweet files. that same staff member can also deactivate, reactivate, or delete twitterfilters just as easily through the ui, triggering backend calls to supervisord to stop, restart, or remove processes and their configuration. these backend calls to supervisord are remote procedure calls made using the python xml-rpc implementation [17].  we expect that this feature will be particularly useful for rapidly responding to researchers’ requests to capture an ongoing flow of tweets in real-time. a potential benefit of having supervisord in place is to manage additional process types. we plan to add features to post-process media content such as images, videos, and web pages, as well as add other social media platforms, and anticipate that a supervisord-managed process will be a good fit for one or more of these new tasks. supporting other users and building community in addition to making the app available via github, we have worked to make it more readily usable by other cultural heritage organizations. at several points in the past year (e.g. a december 2013 grant-funded meeting with developers and librarians, a march 2014 code4lib preconference workshop, and in one-on-one support for potential users) we were able to  assist others installing the app and observe pitfalls in the installation process. these interactions have allowed us to identify problems that users may encounter in setting up the app and motivated us to expand and iteratively improve the documentation. other ways we have sought feedback on sfm have included structured interviews with current and potential users and interactions on our sfm-dev google group [18]. we also recognized that documentation serves a purpose in developing buy-in and understanding for sfm; collecting twitter data is a new activity for most cultural heritage organizations. the documentation enumerates the app’s possible uses and describes both staffing and technical considerations in embarking on collecting twitter data. we used the readthedocs.org as a platform for documentation: it is widely used, has a clean interface, and integrates with the code itself.  docs are formatted in restructured text (rest) [19] and stored in the git repository itself in a docs directory. anyone may fork the repository and submit pull requests suggesting improvements to any of the docs, just as one would submit pull requests with changes to the code. next steps over the next three years, supported by a grant from the national historic publications and records commission, [20] we will continue our work on sfm with overall goals of improving our service to the gwu community and in building collections in gw libraries.  in particular, we intend to add support for more platforms in addition to twitter, including at least flickr and tumblr, and others as demand requires and time allows. in addition, we plan to consider the integration of social media data into archival collections, including the full workflow of selection, transfer, accessioning, processing, describing, and enabling access. along this path we expect to evaluate opportunities for integration with existing descriptive standards and foundational strategies, as well as developing new policies as required. as part of this work we are looking to expand the notion of what we can collect from just one user or set of users’ tweets to a more robust notion of “correspondence”, including replies, mentions, comments, and other interactions present on different platforms. we seek to add the ability to capture more media types such as images, videos, and web pages referenced in tweets. at the same time, we will consider system strategies for accomplishing these tasks across platforms, integrating collected media with existing special collections, descriptive metadata environments, and media gathered via web archiving. throughout this work we will continue to manage issues and milestones for sfm using the github repository, and we will update documentation with each new release. we welcome feedback via comments, tickets, or pull requests in the github repository, or via email to the development list [21].   references [1]  social feed manager is a free and open source software application, available at https://github.com/gwu-libraries/social-feed-manager/ with documentation at  http://social-feed-manager.readthedocs.org/ [2] “changes coming in version 1.1 of the twitter api”, https://blog.twitter.com/2012/changes-coming-to-twitter-api [3] jesse holcomb, kimberly gross and amy mitchell. 2011. “how mainstream media outlets use twitter.”  report from project for excellence in journalism. http://www.journalism.org/2011/11/14/how-mainstream-media-outlets-use-twitter/ [4] google reader (https://www.google.com/reader/) was an rss/atom feed aggregator which was discontinued as of july 1, 2013.  although alternatives exist, the once-ubiquitous “feed reader” model popular for many years has now waned. [5] http://www.tweepy.org/ [6] https://dev.twitter.com/docs/api/1.1/get/statuses/user_timeline [7] http://social-feed-manager.readthedocs.org/en/m5_002/data_dictionary.html [8] http://social-feed-manager.readthedocs.org/en/m5_002/mgmt_commands.html#export-csv [9] https://dev.twitter.com/docs/api/1.1/post/statuses/filter [10] https://dev.twitter.com/streaming/reference/get/statuses/sample [11] http://social-feed-manager.readthedocs.org/en/m5_002/mgmt_commands.html#organizedata [12] http://en.wikipedia.org/wiki/bagit [13] http://social-feed-manager.readthedocs.org/en/m5_002/mgmt_commands.html#filterstream [14] http://supervisord.org/ [15] http://social-feed-manager.readthedocs.org/en/m5_002/supervisor_and_streams.html [16] in sfm, a twitterfilter contains the query predicates with which to call the twitter “statuses/filter” api [10]]; these predicates may include keywords, geospatial regions, and/or usernames. [17] https://docs.python.org/2/library/xmlrpclib.html [18] https://groups.google.com/forum/#!forum/sfm-dev [19] http://docutils.sourceforge.net/rst.html [20] nhprc award number nar14-di-50017-14: “building social media collections: tools and policy recommendations for collection development and management” [21] https://groups.google.com/forum/#!forum/sfm-dev acknowledgements this project was made possible in part by the institute of museum and library services sparks! ignition grant lg-46-13-0257. about the authors daniel chudnov is director, scholarly technology at the george washington university libraries, where he supervises library staff working in software and systems development and operations. he holds degrees from the university of michigan, and is pursuing an additional master’s degree in business analytics at george washington university. daniel kerchner is a senior software developer on the social feed manager project as well as other digital initiatives and open source projects.  he is a certified project management professional and holds degrees from cornell university and the university of virginia. ankushi sharma is a masters student in the department of computer science at the george washington university. she is a software developer on the social feed manager project, and was recently awarded a scholarship from the anita borg institute to attend the grace hopper celebration of women in computing. laura wrubel is the e-resources content manager at the george washington university libraries and is responsible for access to library e-resources. she holds an mls from the university of maryland. subscribe to comments: for this article | for all articles one response to "technical challenges in developing software to collect twitter data" please leave a response below: lina, 2014-10-22 way to go!!! leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – book review: semantic web for the working ontologist mission editorial committee process and structure code4lib issue 6, 2009-03-30 book review: semantic web for the working ontologist written by two of the leading authorities on the semantic web, the “semantic web for the working ontologist” is a timely and thorough introduction to the topic. covering rdf, rdfs, and owl, the book takes a logical, trainerly approach, with practical and illuminating examples. well worth a read. allemang, dean, and james a. hendler. 2008. semantic web for the working ontologist: modeling in rdf, rdfs and owl. boston: morgan kaufmann publishers. coins by tom keays a couple of years ago, i started reading shelley powers’ book, “practical rdf” [1]. published by o’reilly in 2003 while the rdf core working group was still working on revisions to the w3c’s rdf specification [2], this was one of the first attempts to bring the six documents of that specification together in one place and try to give them an understandable context along with practical examples and code. alas, i bogged down about halfway through and never did more than skim the last part. the book had a cursory treatment of alternate rdf serialization notations, including n-triples and notation 3 (n3), but most of the examples were written in xml. bulky, tedious rdf/xml. i already had a pretty good understanding of how to formulate valid xml but, despite the author’s best efforts, i found the syntax of rdf in xml to be extremely opaque. i was never able to gain any sort of proficiency translating rdf subject – predicate – object triples into xml and i never trusted my ability to construct any sort of proper rdf/xml statements from scratch. it didn’t help matters that powers, not being exactly sure how the draft rdf specification was going to play out, included several complicated use cases involving rdf containers and collections that were never actually put into practice. about a year ago, i started noticing that a number of new books were being written about the semantic web [3] and linked data [4] (as opposed to titling them as books about rdf). this signaled to me that a sea change had occurred in how the semantic web community was approaching the topic. i was also very aware that the then upcoming code4lib conference [5] had announced no less than a daylong pre-conference workshop, two keynote talks, and several presentations all dealing with linked data. from the various new books that were published in 2008, i decided to read dean allemang and jim hendler’s “semantic web for the working ontologist : effective modeling in rdfs and owl” [6], published by morgan kaufmann, singling it out as one of the more practical (as opposed to theoretical) treatments of the subject. both authors have academic backgrounds with excellent credentials in writing about, working with, and providing training in semantic web technologies. jim hendler [7] was co-author with tim berners-lee of the seminal 2001 paper introducing the idea of the semantic web to the world [8] and was co-chair of the w3c’s web ontology (owl) working group [9]. dean allemang [10] is the chief scientist at topquadrant, inc., a consulting, training and semantic web products company. allemang and hendler co-developed topquadrant’s semantic web training series. the “semantic web for the working ontologist,” is all about what you can build using rfd and why. by comparison, powers’ “practical rdf” book, as appropriate as it was for its time, was more about the parts and less about the structures. examining the difference in approaches is like comparing two books on the technology of woodworking: one describing different kinds of nails, screws, and planks, and the other showing you how to build furniture. the authors’ training experience shows through in the clear presentation of ideas and examples in this book. it is logically organized into 14 chapters, an faq appendix, and a subject index. although not explicitly divided into broad sections, the sequence of the chapters seemed to me to fall into seven rough subdivisions of content. the first 2 chapters start out gently, introducing general semantic web concepts including semantic modeling. chapters 3 through 5 then delve into the basic structures of rdf, including identities, uris, and the common namespaces of rdf, rdfs, and owl (along with several homebrewed namespace examples). all the examples in the book are framed in concise and clear n3 notation with some graph representations. n3, because of its compactness and simple rules, makes it much easier to see and understand the structure of rdf statements than is possible with xml (although chapter 3 briefly describes xml and n-triples as alternative serializations). the chapter also has an excellent example that, in showing how to translate more familiar data structures, such as a table of data, into rdf, provides much of the underpinnings needed for the rest of the book. this is the section where the reader really gets his feet wet and will probably make his decision to keep on reading or not. chapters 4 and 5 build on chapter 3, describing the idea of the rdf store, examples of rdf parsers and serializers for getting data in and out of stores, various forms of rdf query languages for retrieving information from stores, and application frameworks for putting it all together. along the way, the concept of inferencing, so essential for rdf queries to work, is introduced and described through a series of examples. chapter 6 introduces rdf schema language (rdfs) and explores how the addition of this schema structure allows you to begin to talk about sets and relationships. by itself, rdf is nothing more than a framework for describing things in terms of subjects, predicates, and objects. as such, there is very little that you can do with it that will get you much beyond describing nodes and graphs. rdfs, on the other hand, provides a vocabulary for describing rdf graphs, together with structures for defining relationships for members of graph nodes and a way to make more interesting queries against the data model. the examples in chapter 6 help the reader begin to understand the real power and intentions of rdf. owl is a topic that can make even someone experienced with semantic web principles quickly glaze over. there’s nothing in owl that can’t be also stated with rdfs, but using owl gives the rdf ontologist a set of rules that simplify and clarify semantic web structures. in chapter 7, the authors have chosen to ease the reader into the topic of ontologies by introducing selected owl properties and classes in a subset of elements they’ve dubbed rdfs-plus. they’ve found that this subset of owl increases the power of rdfs without requiring a large conceptual leap for the user. rdfs-plus is then used to provide a framework for introducing the reader to such notions as inverse properties, symmetric properties, transitivity, equivalence (both of classes and properties), and functional properties that allow merging of datasets. chapter 8 takes the concepts introduced in chapter 7 a bit further by exploring the real world applications of skos (simple knowledge organization system), a syntax that can be used to construct taxonomies, thesauri, and controlled vocabularies, along with foaf (friend of a friend), a simple framework for describing relationships between people and organizations. foaf, incidentally, served as a central example at the linked data pre-conference session [11] at code4lib 2009. once the reader is comfortable with rdfs-plus, chapters 9 through 11 explore owl more completely, covering the idea of restrictions in quite a lot of detail, before launching into a description of owl as a way to do set-specific analyses of relationships, along with the ideas of cardinality, counting, existence, contradictions, and unsatisfiable classes. chapter 11 describes instances of owl “in the wild”, using the federal enterprise architecture reference model (fea-rm) and the national cancer institute (nci) ontologies to extend the concepts previously introduced. the choice of these examples are not too surprising once you realize that the authors do a lot of training in the government sector. chapter 12 introduces suggested routines and procedures for constructing semantic models, including best practices and potential pitfalls of rule construction. chapters 13 and 14 wrap up the book, speculating on the future of the semantic web and summarizing and extending the ideas previously presented. the book has a short appendix in the form of a faq that indexes important concepts described in the book, as well as a more traditional alphabetic index of terms and sections. in addition, there is a small companion web site that has code examples and a short errata list [12]. conclusion the semantic web might not yet be widely regarded as a source of mainstream library data, but i sense that that day is coming. dublin core, frbr, and rda all have expressions in rdf. the library of congress is promising to give us back a sanctioned linked data version of lcsh and other sources of library authority data [13]. libris, the swedish union catalogue, is an example of an opac that has planned for the future and displays bibliographic records as linked data [14]. before reading this book, i managed to assemble a halting knowledge of rdf from various sources, sufficient enough that, together with some cribbed examples, i was able write a simple foaf profile, but not much more. after reading “semantic web for the working ontologist,” i feel like i understand rdfs and owl for the first time and, although far from expert, would be up to doing some real work. all in all, the “semantic web for the working ontologist” is a useful book. although some of the concepts are a little dense and not always perfectly stated, it is definitely a fine introduction to the topic and a good way to gain some real knowledge and skills. with a list price of $137, the publisher seems to be marketing the “semantic web for the working ontologist” as an academic textbook rather than a mass market computer science reference work. fortunately, amazon and other booksellers generally have new copies in the $45-$55 range and used books at $10 or so below that. this is a book that can be found in many academic libraries, so it ought to be fairly easy to examine a copy before deciding to buy it. references [1] powers, shelley. 2003. practical rdf. cambridge: o’reilly & associates, inc. oclc: 51527158. open library: ol3318069m. coins [2] rdf/xml syntax specification (revised). w3c recommendation 10 february 2004. http://www.w3.org/tr/rdf-syntax-grammar/ [3] w3c semantic web activity. http://www.w3.org/2001/sw [4] linked data. wikipedia. http://en.wikipedia.org/wiki/linked_data [5] code4lib conference. 2009. providence, ri. http://www.code4lib.org/conference/2009/ [6] allemang, dean, and james a. hendler. 2008. semantic web for the working ontologist: modeling in rdf, rdfs and owl. boston: morgan kaufmann publishers. oclc: 184925396. open library: ol16867827m. coins [7] james hendler’s website. http://www.cs.rpi.edu/~hendler/ [8] berners-lee, tim, james hendler, and ora lassila. “the semantic web.” scientific american 284, no. 5 (may 2001): 34-43. http://www.sciam.com/article.cfm?id=the-semantic-web coins [9] owl web ontology language reference. w3c recommendation 10 february 2004. http://www.w3.org/tr/owl-ref/ [10] dean allemang’s blog. http://dallemang.typepad.com/ [11] summers, ed. attendees list generated from linked foaf profiles. linked data pre-conference. code4lib conference. 2009. http://www.inkdroid.org/c4l2009/attendees [12] semantic web for the working ontologist companion website. http://workingontologist.org/ [13] id.loc.gov web service. library of congress. http://id.loc.gov/ [14] lindström, henrik and martin malmsten. “user-centred design and agile development: rebuilding the swedish national union catalogue.” the code4lib journal iss. 5 (december 2008). http://journal.code4lib.org/articles/561 about the author tom keays is a science librarian at syracuse university and a founding editor of the code4lib journal. subscribe to comments: for this article | for all articles 4 responses to "book review: semantic web for the working ontologist" please leave a response below: jrochkind, 2009-03-30 amazon and google both conveniently offer limited previews and search functionality. http://www.amazon.com/gp/sitbv3/reader/0123735564 http://books.google.com/books?id=rnfjztfpilcc&printsec=frontcover marissa, 2009-04-01 the $137 price was an amazon data error that has been fixed. the actual list price is $49.95 tomkeays, 2009-04-01 @marissa. thanks for that info. the price was wrong at amazon for quite a long time and the book itself didn’t have a price on it. $49.95 is expensive enough, however good the book. tomkeays, 2009-05-04 i haven’t read it yet, but another book that looks like it might be a good introduction is “the semantic web for dummies” by jeff pollock. it appears to have a lot of practical use cases from the description. i can recommend the related interview with the author done last month as part of the talis podcast series. http://blogs.talis.com/nodalities/2009/03/jeff-pollock-talks-about-his-new-book-the-semantic-web-for-dummies.php leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – implementing time travel for the web mission editorial committee process and structure code4lib issue 13, 2011-04-11 implementing time travel for the web this article discusses the challenges and solutions discovered for implementing the memento protocol in a variety of browser environments. it describes the design and deployment of the client technologies which have been developed: a web application that functioned as a browser, an add-on for firefox called mementofox, a plugin for internet explorer and an android-based client application. the design and technical solutions identified during the development will be of interest to those considering implementation of a memento based platform, especially on the client side, however the interactions are also important for building conformant server-side systems. by robert sanderson, harihar shankar, scott ainsworth, frank mccown, and sam adams introduction developers of digital libraries and archives built for digital preservation are very aware of the time dimension when it comes to their constituent resources; document models in digital libraries are well stocked with different versioning paradigms, and maintaining old versions of documents is the foundational aspect of digital archives. multiple timestamps, different creators and editors, and different publication statuses are just some of the aspects that need to be dealt with by any such repository of knowledge. it is well understood that access to prior versions of the documents is important for the scholarly record and authenticity of research. recent trends in digital libraries are towards integration with the architecture of the world wide web, either through web 2.0 functional behavior or with semantics via linked data and rdf. one feature of both is the use of http content negotiation [1] with “cool uris” [2] that provide access to the same resource in different formats, or provide embedded functionality that might previously have required multiple pages and search forms. these uris stay constant even though the representation delivered in the response changes. this is what makes the web possible in a distributed environment; if each different state had a different identifier, the links between pages that make up the web would break every time a new version was introduced. the memento project [3] proposes to combine the understanding of the importance of time and versions possessed by the digital library and preservation community with current thinking in the web domain. it should be possible for a browser to automatically navigate from the original resource to archived versions of its previous states using negotiation and other protocol level operations, rather than having to know in which archive (such as the internet archives [4] or the uk national archives [5]) to search for them by hand. this article first describes the protocol level details and then discusses the challenges of a client side implementation for the firefox web browser. some early work on an equivalent internet explorer plugin is presented, along with the implementation of an android application called memento browser. it is hoped that this research will provide insight into the practical issues of memento and motivation for further implementations that enable seamless access to the historical web. memento there are three main issues, hinted at above, with existing access to archived resources on the web: instead of being accessible directly or indirectly from their original identifier (their uri), the archived resources are available from new locations with no method to discover the most appropriate version, given their original uri. even if the user does know of one or more archives in which to search, the most appropriate version for their need may be in an archive that is unknown to them. each archive provides a different, manual search and selection interface that the user must navigate to discover the desired resource. however, given only the original resource’s uri and the desired time, the browser, if it understood the navigation process, could use this information to take the user straight to the appropriate resource without manual interaction. once the user has found an archived version of the resource, future navigation is inconsistent. either: (a) the navigation is locked into a single archive due to urls being rewritten, potentially to resources which do not exist or are not the most appropriate for the user’s requested timestamp; or (b) if the urls are not rewritten, the user will instantly be taken out of the past to the current state of a resource rather than an appropriate archived copy, as the browser does not know of the user’s desire to browse the past versions of the resources. memento approaches these issues with a protocol level solution that can be implemented by clients, servers and intermediate services. the currently evolving specification is available at the mementoweb site [6] and the details are summarized below. the first step in making previous states of resources discoverable at a protocol level is to make the archived resources indirectly available from the original resource’s uri, r. whenever the original resource is dereferenced, the web server can add an http link header [7] to the response which points to a new type of resource called a timegate, g. this response can be the same for any request, as the header makes no other impact on the transaction for clients unaware of memento. the request can be made as a head, as this link is the only information of interest in the response. the timegate provides the solution for the second issue, as it understands requests with a timestamp included in a request header, and can redirect the client to the most appropriate archived version that it is aware of without manual intervention. the client adds an accept-datetime header containing the timestamp of interest in rfc 1123 format [8], the mandated timestamp format for http, to the request. the server responds with a 302 status and a location header pointing at the appropriate archived resource, which we call a memento and is depicted as m in figure 1. the timegate also adds in links to the original resource, along with the first and last archived versions it knows of, and optionally to other mementos it thinks may be useful to the client, such as the next and previous in the sequence. the client then follows this redirection and ends up at the appropriate archived resource for the user’s request. the archival server includes in a memento-datetime header, the timestamp at which it knows the representation was served from the original resource’s uri. the archival server should also provide links to both r and g. this transactional process is applied to all requests made by the client and as such provides a solution to the third issue, as embedded resources, such as images or stylesheets, are resolved in the same time-aware fashion as the encapsulating html page without requiring the uris to be rewritten. in fact, having the uris rewritten is not necessarily the optimal strategy as the client stays locked within the current archive rather than potentially having access to resources across multiple archives. this series of interactions is depicted in figure 1 below. figure 1: memento http transactions r = original resource; g = timegate; m = memento memento also defines an api for interacting with archives in order to discover their holdings for a particular original resource. there are two complementary implementations of the api, one is aligned with linked data and based on oai ore [9], and the second uses the same csv-like format as the link header for ease of processing. the document retrieved is called a timemap and at a minimum includes the uris of the archived versions and the time for which they are appropriate. in the ore implementation, a timebundle is a specialization of an ore aggregation that aggregates all of the known archived versions of a given resource. the timebundle and timemap are referenced from the timegate via an entry in the link header in the response, and the client can thus choose which type is easier for it to process. the information from the timemap can then be used by third parties to create overlay services, timegates with access to the holdings of multiple archives, and to provide more detailed client/server interactions. figure 2: time map api r = original resource; b = timebundle; t and t’ = timemap server side memento client in order to demonstrate that the memento solution described above would work, it was necessary to implement a client capable of performing the request/response cycle of figure 1, and understanding and acting upon the headers involved in the transactions. the first attempt at building a client resulted in a web application,implemented and hosted at los alamos national laboratory, which reconstructed the requested page from known web archives. [10] using this client, the user starts by submitting the url and the time of the desired version to a simple web form. the client then performs all of the necessary http transactions for memento, following the link header and redirects in order to obtain and locally cache the archived representation. if the retrieved resource is an html document, then the client parses it and steps through each url, such as images, stylesheets and javascript, fetching the appropriate archived representation and caching it recursively to deal with frames. when there are no more urls to fetch, it returns the requested resource to the browser to be displayed to the user. this architecture is depicted below in figure 3. figure 3: server­side client architecture while informative and tolerable as an initial demonstration, the web application was slow and unreliable as it was necessary to cache all of the component resources locally before returning the page to the user, thereby making incremental rendering of the page impossible. not only was the user stuck inside a single, albeit distributed, solution, it exhibited all the problems associated with html parsing and url rewriting, from bad html to links being generated dynamically with javascript. on the other hand, it did make it easy to track usage and get a feel for how, or even if, people would use a truly integrated client. the more than 10,000 sessions logged in the two month lifetime of this little application gave us great motivation to create mementofox. memento state machine having already run into many unexpected issues in the server-side implementation, we were able to describe a state machine for all of the possibilities that a memento client would encounter. as depicted in figure 4, the logic requires access to the response status code and headers at each step to determine the next action. figure 4: memento client state diagram r = original resource; g:r = time gate for r; m = memento for r; x = another resource; g:x = time gate for x; dflt = default time gate starting with the original resource r, there may or may not be a link header with a rel attribute of "timegate". it is accepted that in the early days of memento adoption, the vast majority of web servers will not know of a web archive that contains mementos of its hosted resources, and hence will not include a link to a timegate. if the link header is not present, it is treated as an (expected) error condition and the client should use its default timegate dflt. if the link header does exist, the client should follow it to either a real timegate for the resource r, the node g:r, or in the case of a misconfigured server, to another resource x which is not a timegate for r. a timegate can be detected by inspecting the vary header of the response for the string “accept-datetime”. the resource that the timegate is for is given in the link header with a rel attribute of "original". if the link takes the client to a resource that is not a timegate, that resource will either issue a response with a 302 status code or some other status code. in the case of a non-302 response, an error condition is reached and the client should take over again by using its default timegate. on the other hand, if the response does have a 302 redirect, there are three possibilities: it can be to another resource that exhibits the same behavior as x, to a timegate g:x that in turn redirects to a memento of x rather than a memento of r, or to a timegate g:r for the original resource r. the first two 302 conditions result in the error state; however the admittedly unlikely final case would result in the client continuing its normal behavior. in the case where the link header from the original resource is to a real timegate for r, g:r, then it can either issue a non-302 response, resulting in an error condition, or issue a 302 with the vary header and a link header with rel attribute of “original” pointing back to r. this link header is necessary to determine the difference between g:r and g:x and the vary header is necessary in order to determine that the redirection is due to time-based content negotiation. the default timegate may also generate an error condition, at which point the client must either continue to try default timegates until one provides a correct response or stop its search. finally, either via dflt or g:r, the client will arrive at a memento, m. this may in turn redirect with a 3xx status to another resource, such as for a memento that has been moved to another archive. otherwise, the client has successfully reached the end state. mementofox initial approach mozilla’s firefox [11] was chosen as the first browser platform as building extensions for it is relatively straightforward. they can be implemented in javascript and distributed without requiring firefox be recompiled or the extension to be compiled for different operating systems. there are tens of thousands of registered add-ons which can be used as exemplars [12], and the documentation for the inner workings of the browser is clear, up to date and easily available [13]. the user interface for the extension is described in an xml schema called xul, which firefox interprets at run time. any other required files, such as images, are placed in an extension-specific directory. there is easy access to important internal functions, such as preferences, url and http protocol functionality via a javascript/c++ reflection layer called xpcom. once finished, the add-on is packaged with all of its resources in a zip file called an xpi. when firefox sees one of these files, it knows to install it rather than attempt to render the resource. our initial approach, implemented at old dominion university, created a sidebar for user interaction and relied on the obvious firefox events that monitor and change http requests. the sidebar allowed the user to enable and disable memento, specify the target date and time, and view the results of memento operations. figure 5 is a screenshot of the initial approach. to implement the memento protocol, two events were used. the http-on-modify-request event was used to add the accept-datetime header to the request. the sidebar and extension’s internal state were updated on the http-on-examine-response event. this approach was sufficient for use with memento-aware http servers but could not detect and respond to responses from services that were not yet memento-compliant. figure 5: initial approach screenshot monitoring the endloaddocument event allowed non-memento responses to be detected and acted upon. when a non-memento response was detected, the extension changed document.location to a timegate at los alamos national laboratory that searches a number of archives, thus satisfying the user’s request when the original server could not. unfortunately, endloaddocument is triggered after a page’s embedded and internal resources are fully loaded, resulting in a confusing user experience in which the embedded resources appeared in their current state and were then replaced with their archived versions. aside from discovering the limits of firefox’s http monitoring events, our initial approach contributed a method to apply memento independently for each firefox content window. this resulted in a working client, but one that provided a poor user experience. while this implementation was faster and closer to the desired result than the server side client, a solution to the display issue proved to be elusive. fireback elusive, at least, until jisc sponsored a competition to develop memento-related software at the developer happiness days event in london at the end of february 2010 [14]. the winning entry, called fireback, was a firefox add-on that demonstrated the way around the obstacle. instead of capturing the response, fireback used an interface called nsicontentpolicy in order to prevent firefox from making the request in the first place. this interface is normally used to block content such as unwanted advertising or resources unsafe for viewing by children, and was discovered by examination of the adblock plus add-on [15], which requires such behavior. the nsicontentpolicy interface is designed to allow an add-on to observe content as it is loaded into the browser and enable the add-on to block or permit access to particular uris. the interface provides two methods: shouldload(), which is called before a resource is loaded and allows the add-on to decide whether to start loading the resource, and shouldprocess() which allows the add-on to inspect a resource’s content and decide whether to render it. experimentation found that as well as being able to reject access to a requested resource from within the shouldload() method it was possible to direct the browser to load an alternative resource. this approach worked well, albeit with one usability issue: loading a page resulted in the ui locking up, with the window completely frozen to user input for ten or more seconds at a time. investigations found that firefox is a largely single-threaded application, and while the initial head request to the original resource is typically very fast, the get request to the timegate can be fairly slow. the synchronous nature of the get request was causing firefox’s main thread to block until a response was received from the timegate. it was thought that the delay could be minimized if only the head request were performed during the invocation of the shouldload() method, the browser then instructed to load the timegate uri, and the browser’s internal redirect handling mechanism relied on to handle the redirection from the timegate to the memento. unfortunately, while it is possible to direct the browser to load an alternative uri from within the shouldload() method, it is not possible to access the http headers of that request and add the accept-datetime header. the solution to add the headers was to employ the event observation technique of the initial approach. the header was added by processing the http-on-modify-request event, and cleanup performed on the corresponding http-on-examine-response. this considerably reduced the scope for the browser’s main thread to block, leading to a much more responsive ui. current version the fireback networking solution and the multiple tab/window handling abilities of the initial add-on were combined with improved internal logic and user interface to form the basis of the current version of mementofox. the majority of the mementofox logic takes place in shouldload(), which occurs immediately before the url is retrieved. if the resource should be loaded, the function returns accept, otherwise it returns one of four different reject flags. determining whether to accept or reject and rewrite a request is the fundamental issue for memento client development. the function has a series of cases that should always be accepted, for example, if the protocol is not http, if the resource is a memento, or if it has already been rewritten by the add-on. it then follows the logic described above in order to find the most appropriate archived version. as discovered in the implementation of fireback, it is possible to modify the url being processed in place, and this allows us to internally redirect the browser to the appropriate timegate for the top html page or to the memento for embedded resources such as images. firefox processes embedded resources in the context of the encapsulating html element (or the nsidomnode to be exact) that refers to them, rather than at the browser level in order to ensure the correct rendering. this means that the dom node must be modified internally to point to the memento, in a kind of client-side url rewriting operation, but one which occurs before the original resource is retrieved. if the memento comes from an archive which does not yet support memento natively, detectable via the lack of memento-datetime header, then in a pre-processing step, the system will first attempt to unrewrite any uris that have been rewritten to point to versions within the archive. current web archives will rewrite, for example, a link to google with a link to an archived version of the google homepage in order to provide an appropriate user experience. this is a temporary workaround in order to use non-compliant archives as they are currently, but is not possible in all situations as it relies on implementation-specific heuristics. the add-on must also implement an event observation method in order to capture the headers of the http responses. this http-on-examine-response event is what allows us to find the link and vary headers. this information lets us warn the user if the headers look suspicious and to provide additional navigation options, such as buttons to go to the first, last, previous and next mementos. other than adding the accept-datetime header on the outgoing requests, the observer is also used to update preferences in real time. after releasing the first public version, we received a lot of very positive and constructive feedback about the user interface. in particular, the time slider was well received and the menu bar rather than sidebar style of the first attempt was appreciated. however, users found our initial terminology confusing, and wanted even less of an interface footprint: vcr-style buttons for first, last, previous and next replaced a long drop down list, and the menu bar can be automatically hidden when the add-on is inactive. additional functionality was added, such as control for the behavior when an appropriate memento cannot be discovered: a preference determines if the current version or an error message is displayed. it was necessary to then rewrite a large proportion of the codebase in order to bring it up to the standards required by mozilla for the add-on to be endorsed as safe. variables in add-ons are not inherently encapsulated within the code, so any global variable can unexpectedly collide with another add-on’s variable of the same name. this was corrected by hiding all of the code inside a uniquely named object. all of the user interface strings needed to be exported to an external xml file in order to allow for localization, and further interface improvements were suggested. as of version 0.8.10 (september 2010), mementofox has been endorsed by mozilla as an add-on that is safe for widespread use [16]. the codebase has subsequently been published via the google code repository [17] to allow for distributed development. the screenshot in figure 6 is the result of dragging the time slider to october 2007 when the browser is displaying the current http://www.code4lib.org/ web page. the mementofox add-on takes over and discovers an appropriate version in the internet archive, via a multi-archive timegate chosen in the add-on’s user preferences. it retrieves the embedded images and other resources to generate an accurate representation of the historical version, without any manual interaction with web archives. figure 6: screenshot of mementofox internet explorer an internet explorer plugin, known as a browser helper object (bho), is a dynamic linked library (dll) module. the bho is loaded each time internet explorer starts up, runs in the same memory context as the browser and can perform any action on the available windows and modules. the bho components communicate with ie by implementing a binary interface that enables inter-process communication, called the component object model (com) interface. the proof-of-concept memento plugin for ie is written in c# and uses the .net technology. creating the bho turned out to be both difficult and time-consuming, particularly as documentation specific to the development task is sparse [18], and there are very few useful examples available. the memento bho implements the iobjectwithsite com interface to communicate with ie and attach to its typical events. in particular, the memento bho invokes the onbeforenavigate2 event, which fires before any navigation occurs in the web browser. the bho observes this event and then follows the logic described earlier to retrieve the memento. the logic for the memento bho varies from mementofox only when displaying the embedded resources of a memento. before ie navigates to the memento, the bho intercepts, fetches its dom, parses it, retrieves the memento for the embedded resources and renders it in the browser window. a screenshot of the bho and its preliminary ui is shown in figure 7. the bho currently has a very simple ui with a button to toggle memento mode on/off, a calendar to choose the date-time and a “navigate” button to start the memento navigation. to create the toolbar, the bandobjectlib library is used [19]. the calendar tool could not be directly incorporated into the toolbar. instead, the calendar from windows forms had to be wrapped by extending toolstripcontrolhost in order for it to be used in the ie toolbar. this meant that the calendar tool could not invoke the ie events directly and hence a separate “navigate” button was required to trigger the navigation after a date was picked. there are no buttons yet in the toolbar to navigate to previous and next mementos. installing the memento bho and the toolbar is simple as all of the necessary files are packaged for installation and stored as an msi file. installing or uninstalling this bho is similar to installing any other windows application and can be performed from the add/remove programs from the windows control panel. this plugin was tested to work in internet explorer 8 with windows xp/7. the current version has been presented to a team at microsoft research for review, but is not currently available for download. figure 7: screenshot of memento bho for ie android app: memento browser to showcase memento on a mobile platform, an android application was developed called the memento browser. android is a popular open source platform developed by google and governed by the open handset alliance. it is installed on a large number of smartphones from a variety of manufacturers. applications are written in java, and they can be downloaded from the android market or from any web page that exposes the application’s .apk file (a zip file composed of the executable code and other application files). memento browser was released as an open source project on google code in september 2010 [20]. a screenshot of memento browser is shown in figure 8 below. in the top image, a user is browsing the mobile version of cnn.com on sept. 16, 2010. to request a memento for a different date, the user clicks on the button that is titled “2010/09/16”. in the center image, the user has requested a new date, july 7, 2009, and is presented with the nearest memento as archived by webcite on july 8, 2009. the left and right arrow buttons on the upper-right side of the browser allow the user to scroll through older and more recent mementos. the user can also access a list of all known mementos, as shown in the bottom image, by pressing the android device’s menu button and selecting “list mementos”. figure 8: screenshots of memento browser there are a number of stock ui widgets that android developers may use, including the webview control which uses webkit [21], an open source web browser engine, to render web pages. the webview’s webviewclient provides the ability to override the web page’s url loading. the memento browser uses this mechanism in order to load the current version of a web page or a memento from a requested date. however, the webview control does not expose a mechanism to trap the http requests and responses for the embedded resources of a web page, like the images, style sheets, etc. therefore the current version of the browser does not use memento for each individual element of a web page but instead relies on the archive that supplies the html to also supply each element via regular http requests. when a user clicks on a link in a memento, the memento protocol is used again to find a web page that matches the requested date. the memento browser uses threading extensively to perform http requests without locking the ui thread. this allows the application to be responsive to user input while it is waiting for and processing http responses. for example, when the user requests a specific date when browsing a web page, the browser starts a new thread which performs the memento transactions as discussed earlier in figure 1. when the browser discovers previous and next mementos, it uses the android handler class to post a new runnable to the ui thread’s message queue. when this runnable is executed, it enables the ui’s previous and next buttons. memento browser uses a default memento timegate running at los alamos national laboratory. however, a user may enter a different default timegate that may expose different mementos. currently the memento browser does not offer many of the frills that other browsers offer like, for example, bookmarking. however, firefox launched a beta of their browser for android in october 2010 [22]. as their browser stabilizes and becomes more popular, a new mementofox add-on for the android firefox may offer a better alternative to a memento-only web browser. future strategies the approach described above is tested and known to work when memento is fully supported by the archive environment and by all browsers. however, an alternative strategy is being explored for environments in which this is not currently possible, such as the limited android webview control, and for the inevitable interim period in which memento is not natively supported by browsers and thus the uris must be rewritten for the archived pages to be viewable. it also supports a hybrid world in which some archives rewrite and some do not, as well as archives that rewrite some uris but not others, such as an institutional archive that only rewrites internal uris but leaves external ones pointing to the original resource. the experimental setup allows the archive to optionally rewrite the embedded urls, but for the client to be able to determine whether or not they have been rewritten. for limited-capability client platforms, this allows them to simply display the page as transmitted, but at the same time allows full clients to determine if the performance increase in simply accepting the html outweighs the accuracy of using timegates to potentially discover resources that are closer to the requested timestamp. the method being tested is for the archive to include a link header, with a rel attribute of original on every response served, even if the response is an error condition or internal redirect. combined with the memento-datetime header, served only on the actual memento response, this allows the client to decide if it should accept the response or use the original uri with a timegate to find a more appropriate representation. on the other hand, if the response does not include a link to the original, then the client can conclude that the resource is the original and the link has not been rewritten. in this case, the client should use a timegate to discover the appropriate memento. a full classification of the different possible types of resource is available at the memento site[23]. this setup would prevent the undesirable bifurcation of archives into those with rewritten pages and those with non-rewritten pages that can only be viewed with memento clients, and the equally undesirable duplication of uris to have two for each archived resource, one rewritten and one unrewritten. the archive can make the choice based on their own factors as to whether or not to rewrite, and memento clients will be able to distinguish and process both scenarios. it also allows clients to optimize for speed by not requiring all of the http requests and simply accepting “good enough” rather than as close to perfect as is available. conclusions development of the various memento clients is ongoing. mementofox is coordinated through the google code repository, and it is hoped that increased participation will occur as community engagement increases. it is also hoped that making the source code available will be encouraging to developers trying to implement the approach in other browsers and environments, such as safari, opera and chrome. some initial work has been performed on building a memento browser for the iphone; however, it is not yet ready for publication. more work is necessary on the user interface side as, although improvements have been made compared to previous versions, most of the effort so far has been in ensuring adherence to the memento protocol. acknowledging the archives that contributed resources to the view being displayed is an important user interface enhancement, although challenging to do without interfering with the rendering of the resources. further customization to allow the user to define a period of time in which resources are acceptable, customization of searching across multiple timegates, and a way to better handle error conditions are also prominent in the issues list. it is hoped that further memento development will be inspired by sharing this technical information, and time and frustration will be saved through learning from our experiences. access to the historical web has long been recognized as important; however, with the increased visibility of preservation of our digital cultural heritage, it has come to the forefront. memento offers a transparent method for giving access to that heritage, which is currently hidden in archives around the world. acknowledgements the authors would like to acknowledge the direct and indirect input of herbert van de sompel, michael nelson and luda balakireva, our co-collaborators in the memento project. we would also like to thank the library of congress for supporting the memento research, and jisc for providing the prize for the dev8d competition. references [1] rfc 2995: transparent content negotiation in http, march 1998 http://tools.ietf.org/html/rfc2295 [2] tim berners-lee. cool uris don’t change. 1998 http://www.w3.org/provider/style/uri [3] memento – adding time to the web http://www.mementoweb.org/ [4] internet archive – wayback machine http://web.archive.org/ [5] the national archives http://www.nationalarchives.gov.uk/ [6] internet draft: http framework for time-based access to resource states — memento. v01 2010-11-11 http://www.mementoweb.org/guide/rfc/id/ [7] rfc 5988: web linking, october 2010 http://tools.ietf.org/html/rfc5988 [8] rfc 1123: requirements for internet hosts — application and support, october 1989 http://www.ietf.org/rfc/rfc1123.txt [9] open archives initiative – object reuse and exchange http://www.openarchives.org/ore/ [10] memento web application demo, http://www.mementoweb.org/demo/client1/ (now defunct) [11] mozilla firefox http://www.mozilla.com/ [12] mozilla firefox add-ons https://addons.mozilla.org/ [13] mozilla development network doc center: extensions https://developer.mozilla.org/en/extensions [14] jisc developer days, 2010 http://dev8d.org/dev8d-2010/ [15] adblock plus http://adblockplus.org/ [16] mozilla firefox add-ons: mementofox 0.9.2, october 22, 2010 https://addons.mozilla.org/en-us/firefox/addon/100298/ [17] mementofox, firefox extension for memento http://code.google.com/p/mementofox/ [18] html and css, microsoft developer network library http://msdn.microsoft.com/en-us/library/aa902517.aspx [19] the code project: extending explorer with band objects using .net and windows forms, 29 apr 2002 http://www.codeproject.com/kb/shell/dotnetbandobjects.aspx [20] memento-browser, a web browser for android http://code.google.com/p/memento-browser/ [21] the webkit open source project http://webkit.org/ [22] the mozilla blog: firefox 4 beta for android and maemo is now available, posted by stuart parmenter, october 7th, 2010 http://blog.mozilla.com/blog/2010/10/07/firefox-4-beta-for-android-and-maemo [23] memento guide: determining resource type http://www.mementoweb.org/guide/resourcetype about the authors robert sanderson (rsanderson@lanl.gov), is a scientist in the research library at los alamos national laboratory. his current research is focused on web scale interoperability for annotations (oac) and for access to resource versions (memento). he obtained his phd from the university of liverpool, where he was subsequently a lecturer in computer science. harihar shankar (harihar@lanl.gov), is a research and development engineer in the prototyping team at the research library of los alamos national laboratory, working on the memento project. he has a masters degree in computer engineering from the university of new mexico. scott ainsworth (sainswor@cs.odu.edu), has an m.s. in computer science and has worked in the field for 35 years. he is currently a computer science research assistant and ph.d. student in the web science and digital libraries group at old dominion university where he works on the memento project. frank mccown (fmccown@harding.edu), is an assistant professor of computer science at harding university. his research interests include web archiving, web information retrieval, and mobile application development. sam adams (sea36@cam.ac.uk), has a phd in chemical informatics, and is currently a software developer at the unilever centre for molecular science informatics, university of cambridge, where he is working on a number of projects promoting the open publication of scientific data. subscribe to comments: for this article | for all articles one response to "implementing time travel for the web" please leave a response below: open access – do you really think it’s a good idea? « digitalcollaboration, 2011-04-12 […] implementing time travel for the web http://journal.code4lib.org/articles/4979 […] leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – assessing high-volume transfers from optical media at nypl mission editorial committee process and structure code4lib issue 51, 2021-06-14 assessing high-volume transfers from optical media at nypl nypl’s workflow for transferring optical media to long-term storage was met with a challenge: an acquisition of a collection containing thousands of recordable cds and dvds. many programs take a disk-by-disk approach to imaging or transferring optical media, but to deal with a collection of this size, nypl developed a workflow using a nimbie autoloader and a customized version of kbnl’s open-source iromlab software to batch disks for transfer. this workflow prioritized quantity, but, at the outset, it was difficult to tell if every transfer was as accurate as it could be. we discuss the process of evaluating the success of the mass transfer workflow, and the improvements we made to identify and troubleshoot errors that could occur during the transfer. a background of the institution and other institutions’ approaches to similar projects is given, then an in-depth discussion of the process of gathering and analyzing data. we finish with a discussion of our takeaways from the project. by michelle rothrock, alison rhonemus, nick krabbenhoeft background the digital archives program at the new york public library—a part of the digital preservation unit of the digital research division—works with the born-digital portions of archival acquisitions by the 11 curatorial units of the nypl research libraries. it’s responsible for accessioning and transferring digital materials to storage for further archival processing and maintaining the materials for long-term storage and access. this means the digital archives program works with storage media dating from the early years of personal computing—like floppy disks in all formats and sizes—to modern-day storage solutions. the program has the equipment, software, and processes for working with all of the more common storage media in nypl’s collections. a recent acquisition by the new york public library for the performing arts brought thousands of cd-rs and dvd-rs, containing mostly (99%) photos in jpg or raw files, to the digital archives lab. writable cds and dvds, like all optical media, are a risky format for preservation. faced with potential obsolescence and fragility, many institutions choose to migrate the contents of optical disks to more stable storage. many programs take a disk-by-disk approach to imaging or transferring optical media, but to deal with a collection of this size, nypl developed a workflow using a nimbie usb plus disc autoloader nb21-br and a customized version of kbnl’s open-source iromlab software to batch disks for transfer. this workflow allowed the disks to be migrated quickly with much less hands-on time for staff. but was quality being sacrificed in the name of efficiency? without reviewing every transfer individually—defeating the purpose of working in batches—the authors of this paper weren’t sure if the transfer process was capturing every file. we set out to answer that question by analyzing the metadata generated by the software used to transfer. in this paper, we describe the process of gathering that data and our findings, as well as the impact this had on nypl’s workflow and how this work touches on issues that apply to digital archives generally. approaches to working with optical media documentation of optical media migration workflows in digital archives typically approaches each disk individually. (wilsey, 2013; pendergrass, 2017) this is certainly in part due to the greater quality control that comes with examining a collection disk-by-disk—duryee’s intro to optical media preservation (kirschenbaum, 2010) recommends viewing every disk in isobuster to understand its structure prior to attempting migration—and the fact that failures are easy to identify and re-try when working with a single disk. however, a disk-by-disk approach is, as the british library observed in their 2013 article, extremely resource-intensive, requiring a lot of time both in terms of the archivists’ efforts and in terms of the overall project timeline. (dappert, 2013) the authors at the british library were faced with a project collection of ~18,000 cds and dvds, and needed to quickly migrate the disks to more stable storage—without handling each disk individually. like nypl, they chose a nimbie (though the british library used a different model—the nimbie nb11) to handle the disk loading, and used the software bundled with the machine to manage the process. the disk robot worked well, but the authors observed that error-checking was more difficult when working with disks en masse. their solution was fairly straightforward: by working in smaller batches, they were able to manually identify and deal with problem disks one-by-one. johan van der knijiff’s work at the dutch national library (or koninklijke bibliotheek van nederland, kbnl) provided another useful model for nypl. the kbnl’s collection includes up to 20,000 disks, so a hands-off workflow to speed up the migration process was key. following the british library, van der knijiff used a nimbie to handle the physical disks. however, the software bundled with the nimbie came up short, prompting the creation of the iromlab (image and rip optical media like a boss) application. designed to work with a collection that included cd-roms and audio cds, the application bundles software that operates the nimbie and works with the disks. first, a user enters the disks’ identifiers into the iromlab application before placing the physical disks into the nimbie, creating a queue. this is a key step in batch processing to keep track of each disk image’s relationship to its existing catalog entry and the physical disk. the software manages the nimbie via this queue, creating destination folders for each disk as it is loaded. the disk is identified with an application called cd-info, and then iromlab extracts either iso disk images or pcm audio data with isobuster or dbpoweramp, respectively. the disk is then ejected and the process repeated with the remaining disks. disk images have been the go-to migration process in digital archives for years. (kirschenbaum 2010) however, logical transfer has become a more appealing solution for many institutions. a 2017 presentation by keith pendergrass (pendergrass, 2017) dove into the ethical and sustainability issues that preserving disk images can present for archives, and pointed out that preserving low-level system data may have little research value. this sparked discussion amongst digital archivists and inspired policy change (as in the case of the university of california libraries, who mention pendergrass’s talk as justification for their recommendation to avoid disk imaging). (dundon, 2020) there are practical reasons some have moved away from creating disk images, too: some institutions found that logical transfer better suited their workflows and took up less of their limited storage space. (maches, 2020) nypl’s approach within nypl’s digital archives program, the practice has similarly evolved to prefer logical transfers. early removable media (like floppies or zip disks) often have complexities that might merit a more forensic approach, but later media can typically be regarded as a generic container. this is especially true for writable optical disks; many users treated these disks as a write-once, read-many backup system. nypl’s workflow departed from the british library’s and kbnl’s by transferring the files on each disk rather than imaging the disks. given the contents of the collection—photos, in well-known formats—it wasn’t likely that the low-level structural information on the disk would ever be necessary for interpreting the files. the digital archives’ process for migrating files from optical disks uses a disk loading device, the nimbie usb plus disc autoloader nb21-br, which can handle up to 100 disks per batch (or more, if a person is available to re-load the input bin as the disks are read). the nimbie is used with a local fork of the iromlab application. rather than imaging disks with isobuster, our local iromlab uses a modified version of bagit-python[1], which bags the files and transfers them to storage. (unmodified, bagit will only bag files in place.) gathering data with the workflow in place, we were now met with the same challenge that the british library faced: how successful were these transfers? though the process generated metadata about each disk, reviewing every log file would take a lot of time—and defeat the purpose of batching the transfers. we decided to evaluate the process before completing the transfer of the entire collection, working with just the first ~3500 transfers. application output at the end of each batch transfer, we have a batch folder, a manifest file with the status of each attempted transfer (including failed transfers), and folders with metadata about and files from each individual disk. the batch manifest file is relatively straightforward, containing basic info about each transfer like the date, id number, and, crucially, whether or not the transfer failed. here, a failure may mean that the software failed to complete a process for one reason or another—but doesn’t indicate the quality or completeness of the transfer. figure 1. a sample of some fields from a batch manifest file. each individual transfer folder contains a few metadata files, generated by different pieces of the application. three of the metadata files—bag-info, cd-info, and transfer—all contain information that provides clues about the transfer’s quality and completeness. figure 2. an example of the sub-folders and files found in the folder generated per disk. figure 3. cd-info.log the cd-info file records some details about the disk. most relevant here: the size of the raw/formatted data on the disk, given in mb. the difference between the raw and formatted sizes is due to the fact that the disks were originally intended to hold audio tracks, with a brief gap in playback between each. the raw size includes that gap, while the formatted size does not. figure 4. transfer.log the transfer.log file contains a report of each sequential step of the transfer process, generated by nypl’s localized version of iromlab. among other things, this file shows how long the disk transfer took (via the timestamps) and the success or failure of each attempt to transfer individual files from the disk. figure 5. bag-info.txt this metadata file includes the payload-oxum, indicating the size in bytes of the successfully transferred files and the file count. gathering data from the log files to get a batch view of our batch transfers, we needed to view the data from the various log files in bulk. we used python to write a script to gather information from each log file, store it with the associated media id, and output the results in a csv format. initially, we weren’t certain which information would tell us the most about the quality of the transfers. our first script pulled the following info: the size of the data on the disk, recorded in the cd-info file the bytecount and filecount of the transferred files in the bag-info file and the start and end times recorded in the transfer log file. this information was recorded with every transfer, so it was easy to identify from an initial look at individual log files. with this information, we compared the size of the data on the disk to the size of the data transferred, looking for discrepancies, and also noted any transfers with unusually long transfer durations. when we saw outliers, we did a closer check of the log files, the original disk (using isobuster), and the transferred files. through this process, we identified more information that would be useful to add to our analysis. some of the additional info we extracted turned out not to be terribly revealing in itself—for instance, the read time per disk didn’t provide a useful indicator that the disk had a problem—but, eventually, we found that some transfer.log files contained information about files that the software wasn’t able to retrieve from the disk. this turned out to be the single most useful piece of data, reliably identifying problematic transfers. the final version of the script gathered a count of the individual files that failed to transfer, along with the media id of the disk, and output the info into a csv. script walkthrough the full script, written in python, can be found on github. (pseudocode used below.) the requirements for the script were to: open each cd-info.log, transfer.log, and bag-info.txt file in the directory; identify the media id associated with each of those files; gather/create the necessary information from each of those files; provide output in the form of a csv for analysis. a few python modules helped us meet these requirements: os and os.path, to find files in subdirectories re for regular expressions to identify the metadata needed from each logfile, datetime to work with the timestamps in the transfer.log files csv for the output. first, the script searches the directory for files with the name cd-info.log. when it finds the file, it records the path to the file (which includes the media id of the transfer) and adds that info to a list for use in the next step. list_of_paths = [] for directory_path, directory_names, filenames: for filename in filenames that includes cd-info.log: join directory_path and filename add to list_of_paths next, an empty dictionary, dict, is opened to store the extracted info. the first section of this code sets up regular expressions to find the different pieces of information needed here: the media id and the raw and formatted data from cd-info. the media id is found in the list of filepaths and is stored in the dictionary as both a key and a value (a necessary workaround for output into the csv, since the csv writer only includes values). dictionary = {} mediaid_pattern = regular expression to identify the media id raw_pattern = regular expression to identify the raw data amount formatted_pattern = regular expression to identify the formatted data amount for each_item in list_of_paths: if the media id is found: add the media id to the dictionary, using the key “mediaid” with the cd-info.log file open: for each_line in cd-info.log: if the raw data amount is found: add the raw data amount to the dictionary</td> the process is repeated for the transfer.log and bag-info.txt files, searching for and storing information from each. finally, everything was written to a new csv file. with a new csv file open: create column headers using every key in the dictionary for the value of every item in the dictionary: add it to a new row in the csv file analyzing the transfer metadata at this point, we had a csv file containing the aggregated metadata from each individual disk transfer ready to analyze. some basic stats: 650,327 individual files transferred from 3,488 disks on average, a single disk took ~238 seconds (about four minutes) to transfer (though a few disks took much longer; some ran for hours) the transferred files consisted primarily of jpgs and cr2 files however, the fact that the software didn’t flag these disks as failed transfers didn’t mean 100% of the files were transferred perfectly. the data contains a column with a count of files not transferred per disk. checking out these partial transfers, we see: 2,259 untransferred files from 63 disks an average transfer time of ~3,209 seconds (53 minutes) and as for failed transfers, flagged in the batch manifest file—amazingly rare!—we found that only three disks out of the nearly 3,500 failed entirely. two disks were flagged with “bagging failed”; the last disk appeared to be blank. manually inspecting problem disks from this point, we had a manageable list of semi-successful (partial) transfers to review, to which we added the three failed transfers found in the batch manifest files. we reviewed these 46 disks and transfers one-by-one, inspecting: the physical disk for visible damage, deterioration, or dirt the files and filestrucure of each disk, using isobuster the transferred files, to ensure that these were intact and readable after inspection, we started troubleshooting. first, disks with visible scratches or dirt were gently cleaned with a lint-free cloth and a solution of alcohol and water. next, we attempted to transfer the files via isobuster’s extract function[2], using a different disk drive (not the nimbie). we chose isobuster for inspecting the contents of the disk and for a first attempt at recovering lost files due to its common use in other institution’s workflows. through troubleshooting, we discovered that the problem disks fell into three categories: damaged disks. these disks typically had long read times, and one or more files were unrecoverable; many of these were visibly scratched. the files that were transferred, however, were good (no half-visible photos or unopenable files). eighteen disks fell into this category. out of these, the nypl fork of iromlab successfully transferred 87.4% of files (a total of 3,360), and failed to transfer 12.6% (485 files). issues with files. some disks with partial or failed transfers were physically readable, but failed for other reasons—often related to file names or unreadable informational files left behind by software the donor used to organize and burn the photos. this accounted for thirty-four of the problem disks. [3] unknown issues. these four disks were not visibly damaged, and checking the disks with another tool or drive recovered all of the files. figure 6. reason for failure this graph shows the high proportion of file system issue-related failures compared to failures caused by damage to the physical disk. figure 7. damaged discs, successes versus failures disks with partial failures still had a high rate of recovery for individual files, as shown above. damaged disks we tested damaged disks with isobuster and ddrescue to find if these tools could recover additional information. in a sample of eighteen disks, isobuster recovered an additional 160 files, compared to the original transfer; ddrescue recovered an additional 141. successfully transferred files files that failed to transfer original transfer 3090 474 isobuster 3249 315 ddrescue 3240 324 while we didn’t record the total transfer time for these tests, both isobuster and ddrescue spent hours—often longer than an 8-hour workday—on many of these test disks. in our tests, isobuster performed a little better than ddrescue in recovering data. it’s also worth noting that isobuster provides more readable feedback on its success than ddrescue does, at least for our purposes. ddrescue offers a bytecount and percentage of total data recovered, while isobuster shows which specific files contained errors. file system issues for disks that failed during the batch transfer process for file system-related reasons, isobuster provided an easy fix. the extract function was able to recover 100% of the files from these disks. the isobuster software allows a user to inspect the disk through any of its available filesystems and bypasses any operating system-related issues with interpreting file names. it’s also easy to navigate within folders. these features made it possible to recover files with illegal characters in the filename and files tucked away within nested subfolders, two of the issues that seemed to affect iromlab’s transfers. discussion we had four key takeaways from our review of the batch transfer process: we confirmed that the batch transfer process was a good match for this collection; we were able to batch our troubleshooting and identify issues more holistically; we identified ways to improve the process; we discovered the key skills for making this project work apply to other projects in digital archives, too. impact on nypl’s workflow we found that the batch transfer process was, overall, very successful. the particular collection—consisting of cd-rs in good condition with relatively consistent contents and file structures—worked well with this hands-off approach, with over 99% of files successfully transferred. and, through our analysis, we were able to identify that missing <1% and troubleshoot these disks individually. inspecting the problem disks one by one, we discovered that most failures fell into one of two categories: file system problems or damaged disks. as discussed above, isobuster was an ideal fix for disks with unusual file systems or filenames. damaged disks, though, were a different story. isobuster and ddrescue did help recover some additional data, but this process often took hours of run time. the eighteen damaged disks had a combined run time of 41+ hours in the nimbie. while isobuster didn’t provide data about the time it took to extract, test disks often took more than eight hours, and one ddrescue test ran for over ten days before we decided to interrupt the process. ultimately, this extra processing time (and staff time, too) only yielded a hundred-odd additional photos—out of a collection of tens of thousands. this particular collection wasn’t made up of select photos; this was the total output of a working photographer, including every test shot, every angle, and every grimacing subject. a vast majority of these photos were likely never passed on to the client, put in print, or published on the web. given that, a few photos were deemed an acceptable loss. for the remainder of the collection, we chose not to try to save additional data from damaged disks, given the processing power and staff time necessary to run isobuster and ddrescue on each scratched disk. we also discovered that, like batch transfer, batch troubleshooting can be a big time-saver. when transferring disks one at a time, staff can troubleshoot one at a time, running through diagnostics and possible solutions before moving to the next disk. in this type of workflow, troubleshooting each problem disk as it appears doesn’t slow down the process, since troubleshooting needs to be done at some point regardless. when transferring disks in an automated batch process, stopping the batch to troubleshoot a single disk pauses processing for every other disk in the queue, so ignoring issues until the end of a batch makes procedural sense. and, by reviewing all of the problem disks at once, we were able to identify patterns of failure that might not have been as apparent working disk-by-disk—for example, the issues we encountered with file systems—and target our troubleshooting methods accordingly. moving forward nypl’s batch transfer process relies on free and open-source software. critical components of the workflow, and our workflow analysis, are open-source: iromlab, bagit-python, ddrescue, python, and more. the advantages of working with open-source software meant we could test out additional tools at no additional cost, combine tools into a program built for a specific purpose like iromlab, and modify those tools to fit even more specific purposes like nypl’s fork of iromlab. this last point is worth exploring further. even though iromlab was created for a very similar scenario—migrating large numbers of optical disks in a library setting—nypl had to change several features to meet its needs. these changes began with relatively less complex updates to the interface (such as changing the labels for text entry fields) and developed into adding additional fields, using the data from those new fields to change the behavior of the software, and finally incorporating bagit-python. because both iromlab and the data analysis script created for this project are written in python, work is underway to incorporate that data analysis directly into iromlab. once that analysis is part of the automated transfer process, it will be easier for staff to identify places where data might be missing and decide if it’s worthwhile to take steps to remedy it. this is key for nypl’s process moving forward, and could also be useful for other institutions looking to implement a similar process for large collections of optical media. it makes the batch transfer process less opaque, which is necessary for cultural heritage collections where a “pretty good” transfer won’t suffice. learning curves this project involved a lot of learning and a lot of trial and error. in this last section, we discuss some of the challenges that may be common to digital archives projects in general. digital archivists come from a range of technical backgrounds; we hope that by talking about some of the tech challenges, and tech skills used to help solve them, we can make this project description more accessible and more useful. limited documentation. documentation can never perfectly describe what a tool does. at points throughout this project, we struggled to understand what the software was doing, and if it was behaving as intended—especially when we brought together multiple open-source tools. reading official documentation closely was a crucial first step, but we often had to supplement the official documentation with blog posts, questions asked on forums, examining code, and testing. software, hardware, and os mismatches. this is a problem many digital archivists know well: software and hardware is often imperfectly compatible with all operating systems. at nypl, we have a variety of windows and mac machines. finding out that ddrescue didn’t work on our mac (for mysterious reasons) but ultimately did run on a windows machine in cygwin took a surprising amount of work and tinkering. for this challenge, it’s key to be comfortable working in different operating systems, and different shells—or at least be comfortable conducting a series of internet searches on things like “ddrescue install cygwin” “cygwin install procedures” “cygwin package mounting”. making metadata useful. as discussed above, a big piece of this project was writing code to bring together information from scattered log files and form a complete picture. this expands a bit on our limited documentation challenge; since we weren’t sure what metadata was being recorded, this part of the process was very iterative. our early code just grabbed a bit of metadata we knew was available; we then had to analyze the results (using pandas and excel) to help point us to outlier data that indicated problem disks. multiple rounds of checking results and rewriting the code were necessary to finally produce a useful set of metadata. end notes [1] the authors are in the process of contributing this feature to the bagit-python codebase. [2] the extract function was performed at the track level, which extracted each file individually rather than creating a disk image. [3] a handful of disks had a single file fail to transfer. in these cases, the file turned out to be a system file that had no research value to retain. since the desired files were successfully transferred, these disks were ultimately left out of the count of problem disks. references dappert, angela, andrew jackson, and akiko kimura. “developing a robust migration workflow for preserving and curating hand-held media,” 2013, 11. dundon, kate, laurel mcphee, elvia arroyo-ramirez, jolene beiser, courtney dean, audra eagle yun, jasmine jones, et al. “guidelines for efficient archival processing in the university of california libraries (version 4),” may 1, 2020. https://escholarship.org/uc/item/4b81g01z. kirschenbaum, matthew g., richard ovenden, gabriela redwine, and council on library and information resources. “digital forensics and born-digital content in cultural heritage collections. clir publication no. 149.” council on library and information resources, december 1, 2010. maches, tori. to image or not to image: implementing a staggered transition to logical capture by default. https://bitcuratorconsortium.org/session-5-imaging/. bitcurator consortium (bcc), october 16, 2020. pendergrass, keith. one does not simply keep disk images: ethical, risk tolerance, and sustainability issues with forensic disk image retention. https://docs.google.com/document/d/1bywee4p6mh0asq6mthloixjyfpt-jwv9nh-acrbkzq0/edit?usp=sharing. bitcurator users forum 2017. wilsey, laura, rebecca skirvin, peter chan, and glynn edwards. “capturing and processing born-digital files in the stop aids project records: a case study” 4, no. 1 (2013): 23 about the authors michelle rothrock is the 2020–21 digital archives fellow at the new york public library. the fellowship is hosted in collaboration with the school of information at pratt institute. alison rhonemus is a digital archives assistant at the new york public library. nick krabbenhoeft is the digital preservation manager at the new york public library. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – code4lib 2012 conference report mission editorial committee process and structure code4lib issue 17, 2012-06-01 code4lib 2012 conference report amy unger is one of the recipients of the gender diversity scholarships to attend the code4lib 2012 conference. the journal is pleased to present her conference report here. by amy unger making connections in seattle code4lib 2012 touched on an incredible number of topics in a short period of time. there were 22 presentations, 33 lightning talks, and a couple rounds of ask anything, where attendees ask a question to the audience and are connected with people who may have an answer, during the course of two and a half days. i felt lucky to be exposed to all the work that members of the code4lib community have been doing over the past years, and i think the quantity of technologies i learned about expanded my knowledge of the possible applications libraries and archives can develop for their data and resources. the variety of skills and technologies employed by code4lib members was impressive. each successive presentation brought up a new way to create or link together library technologies. meeting so many people involved in different technologies provided a great way to get a better sense of what the entire library programming landscape looks like. the presenters did an impressive job introducing topics in a way that allowed attendees with all technical backgrounds to grasp the problem at hand. although there were many talks outside my area of expertise, i never felt as though i lacked the technical grounding to understand the presentation. in this respect, and others, the conference certainly succeeded in being open and encouraging to newcomers. of course, given the uniqueness of every library’s technologies, i found that accessing the online resources greatly enhanced my understanding of each talk. the irc backchannel often provided an interesting take on the presentation, and for many talks, a few clicks could pull up the project’s documentation and code. further, i was impressed with the commitment to quality service and improved access for patrons. i develop primarily staff applications, which lately has involved a fair amount of data manipulation, so i do not currently spend much time focused on supporting usability and design for diverse groups. it was a lot of fun to see all the ways programmers are making our libraries and websites great resources for patrons of all ages and backgrounds. as one of my applications provides an interface for searching a solr index, i found the presentations on solr particularly useful. i was especially interested in solutions to some of the retrieval issues that occur as libraries increase the types of objects that can be found in indexes. for example, how do you boost a journal result that has a match on the title field more than a book result with a match on its title field? as someone who has been on the sidelines of discussions on how to structure an index and its fields, i found it fascinating to see when people chose to create separate indexes, when they chose to create alternate fields, and when we might still be waiting for better tools for tackling a particular challenge. code4lib was my first professional conference, and, combined with all the after-hours activities, it was certainly a bit of a whirlwind. seattle was a great place to unwind on a thursday afternoon, and i am happy to say from experience that the top floor of the seattle public library on a rainy afternoon is a wonderful place for putting in a few hours of uninterrupted programming. i want to thank the seattle organizing team and all the code4lib 2012 volunteers for an exciting and well-organized conference. i am immensely grateful to oregon state university and the digital library federation for the opportunity to meet members of the code4lib community and to participate in this year’s conference. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – collecting and describing university-generated patents in an institutional repository: a case study from rice university mission editorial committee process and structure code4lib issue 30, 2015-10-15 collecting and describing university-generated patents in an institutional repository: a case study from rice university providing an easy method of browsing a university’s patent output can free up valuable research time for faculty, students, and external researchers. this is especially true for rice university’s fondren library, a uspto-designated patent and trademark resource center that serves an academic community widely recognized for cutting edge science and engineering research. in order to make rice-generated patents easier to find in the university’s community, a team of technical and public services librarians from fondren library devised a method to identify, download, and upload patents to the university’s institutional repository, starting with a backlog of over 300. this article discusses the rationale behind the project, its potential benefits, and challenges as new rice-generated patents are added to the repository on a monthly basis. by scott carlson and linda spiro introduction a concrete measure of a research university’s success is the value of its patents. every year, the national academy of inventors and the intellectual property owners association publish a list of the top 100 worldwide universities that were granted united states utility patents — that is, patents based on a new and useful process, machine, manufacture, or composition of matter (or a new and useful improvement).[1] using data from the u.s. patent and trademark office (uspto), they select patents that list a university as the first assignee on the u.s. utility patent and post the information online (“top 100 worldwide universities granted u.s. utility patents in 2014”, 2014). the commercial value of patents to universities worldwide is a topic that has been explored by researchers for many years. in 1997, wallmark looked at how universities can encourage an increased output of inventions and patents. using patent data from chalmers university of technology in gothenburg, sweden, he estimated the economic value of patents on the basis of employment in spin-off companies from university patents (wallmark, 1997). in 2009, striukova from university college london discussed the impact of patenting not just on the university, but also on society and the economy. she stressed that “university patents are not only about creating financial market value,” but also “play an important role in creating knowledge spillovers, building networks with other academics and venture capitalists and catalyzing university-industry recognition”; however, she advises to patent judiciously so as not to waste resources on “unworthy patents” (striukova, 2009, p. 388). likewise in 2008, nicol from the university of tasmania underscored the value of knowledge dissemination as a key component of the university’s mission. in addition to having a social, academic, and practical value, she believed universities are increasingly recognizing the commercial value of university-created knowledge although she favors the release of raw research results into the public domain for some research over patenting (nicol, 2008). in 2014, while discussing how technology transfer offices might best manage university patents, cummings from ohio state and the university of utah emphasized the importance of universities’ patented inventions to the u.s. economy: “more and more, both our federal and state governments rely on top-tier research universities to improve our economy by providing the next generation of inventors and entrepreneurs who create groundbreaking inventions, high-growth startups, thousands of new jobs, and, ultimately be commercialized to drive a cycle of innovation, thereby securing a global leadership position for the u.s. economy.” (cummings, 2014, p. 1027) aware of the abundance of research detailing the contributions of patents to the economy as well as the prestige of originating universities, rice’s patent and trademark resource center staff discussed various ways of promoting rice patents. when jan comfort, patent librarian at clemson university, shared news of a project to add clemson-generated patents to their institutional repository, we realized this was the solution we were seeking. collecting rice patents in our own digital scholarship archive would have numerous potential impacts. first, the initiative would be a convenience, especially for novice patent searchers. using the popular google patent search to find patents assigned to rice university can be an exercise in frustration. even when enclosing rice university in quotation marks or using the “with the exact phrase” advanced search, google retrieves over 100,000 patents. meanwhile, the united states patent and trademark office’s patent assignment database is only searchable from 1980 onward, and depends on assignment data being free of errors — which we found was not the case for some rice patents. secondly, spotlighting rice patents in the digital scholarship archive could also raise rice’s standing locally, nationally and internationally. the repository would connect patent inventors with their other scholarly research in the archive, providing a more complete view of their respective bodies of work. increased visibility could also facilitate future collaborations between rice university and the technology or health sector, patent areas in which rice is strong. finally, collecting rice patents would also allow us to perform long-term tracking, such as tracing citations to determine if particular patents had long-term impacts on future scholarship. with these benefits in mind, a team of technical and public services librarians from rice university’s fondren library met to explore the feasibility of adding rice-generated patents to the university’s repository. members of the group included: linda spiro and siu min yu, government information librarians; scott carlson, metadata coordinator; monica rivero, digital curation coordinator; kathy weimer, head, kelley center for government information; lisa spiro, executive director of digital scholarship services; and shannon kipphut-smith, scholarly communications liaison. identifying & describing patents the project began with fondren’s government information librarians identifying an initial list of patents generated by the rice community. using the public web-based examiner search tool (pubwest), a research staple of patent and trademark resource centers, the librarians identified an initial 200 patents that listed variants of “rice university” or “william marsh rice university” as the assignee (a person or business receiving an assignment of ownership interest). this initial search showed that a sizeable amount of rice scholarship existed, but was not being represented in the repository, helping justifying the necessity of the project. before pressing on with the project, a handful of concerns about the initial dataset needed to be addressed. the first issue was identifying as many of the existing rice-generated patents as possible. keyword searching in pubwest using variations of the terms “rice” and “university” in the assignee field increased our search total to just over 400 results. a little more than 50 of these results were dismissed as false hits, usually patents relating to the production of rice crops. after meeting with rice’s office of technology transfer, which maintains a private list of rice-related patents dating back to 1978, we were able to identify several patents that were missed in the pubwest searches due to misspellings in the original text. (some of the assignee typos included “marshurice university”, “rich university”, “william rice marsh rice university” and “william marsh university.”) by the time the entire backlog was uploaded to the repository in late spring of 2015, the total number had topped off at 365 patents. the other major concern was the exportable metadata from pubwest. our application profile for the project defined 16 discrete pieces of metadata for each repository record, with almost half of the metadata directly referencing information about each specific patent: patent number, title, assignee(s), inventor(s), filing date, publishing date, and abstract. however, metadata exports from our pubwest searches did not provide all of the necessary information: only the first assignee and inventor names were included for each patent, while abstracts were missing entirely. we briefly considered data-mining downloaded copies of the patents to fill in the missing metadata, but this option was rendered moot when we came upon free patents online (fpo), a website that allowed us to perform complex searches and export metadata from the resulting searches, including the full assignee, inventor and abstract information. (this discovery also freed us from having to rely on the limited number of computer terminals able to access pubwest in fondren.) once acquired, the metadata from both pubwest and fpo were combined in a single spreadsheet and uploaded to google refine (now openrefine) for normalization — specifically, assignee and inventor metadata. both pieces of information appear in patents with appended geographic information; this geographic metadata caused us some degree of difficulty, as we soon found more than 50 instances of matching inventor names with inconsistent geographic information in our backlog. to counter these inconsistencies, as well as ward off future issues, we decided to expunge geographic information from all inventor and assignee metadata in the backlog and preclude it from any future submissions. acquiring patents an analysis of the metadata for rice patents showed that the rice community averaged 2 to 3 patents issued per month between january 2014 and march 2015. this suggested that the regular ongoing acquisition and submission of new patents into the repository could easily be a manual task; but with a starting backlog of more than 350 patents, it was obvious that we would need a batched process. the uspto website allows for the searching of individual patents, and has made bulk patent information available to the public, free of charge, through reed technology information services. while reed tech does offer full text versions of patents in multiple formats (such as tiff images and markup languages), downloads are only available in bundles of the entire weekly issuance of the uspto for each given year; in other words, acquiring a single patent would necessitate downloading all of the other patents issued that week. full text markup language bundles ranged between one and two gigabytes, while multi-page tiff archives could be anywhere between seven and 20 gigabytes each. due to time, bandwidth and hard drive space concerns, we decided to investigate automatically pulling pdf copies of patents directly from the uspto’s patent full-text and image database (patft). downloading full-text patent pdfs (“full documents”) is not a direct process on patft. searching for patents leads users to an html representation (full-text is available for patents issued from 1976 onward). clicking the “images” button at the top of each patent page brings the user to another section of the uspto website that contains embedded pdf images representing single pages of the patent. the pdf section is navigated with a toolbar on the left side of the page, which moves the user through the patent page by page; however, another button on the toolbar (“full pages”) will embed a pdf of the entire patent. after inspecting the source code on these embedded pages, we discovered patterns in the methods the uspto chose to store and label the pdf copies. because the navigation of the patents is by individual pages, the uspto stores each page as a separate pdf file in a directory specific to every patent. each file is named after its page number; however, the full pdf copy (from the “full pages” button) is named “0.pdf.” the directory path for each patent is also its own pattern: the seven-digit numerical portion patent identification number reversed, with a padding digit added to the beginning. for example, in the case of patent us8,880,707b2: us 8880707 b2 = http://pimg-fpiw.uspto.gov/fdd/07/807/088/ this means that the pdf containing the entire text of this particular patent would be at the url: http://pimg-fpiw.uspto.gov/fdd/07/807/088/0.pdf knowing this, we looked for a way to generate full-text pdf urls for all of our patents, using only the patent identifiers. to do this, we created a spreadsheet utilizing several formulas that broke the patent identifiers into the file path sectors and combined all of the pieces into a url using a concatenate formula: a b c d e f g h i us8880707b2 8880707 07 807 0 88 0.pdf 07/807/088/0.pdf http://pimg-fpiw.uspto.gov/fdd/07/807/088/0.pdf patent identifier, minus any spaces or punctuation isolated identifier [=mid(a1,3,7)] first url sector [=right(b1, 2)] second url sector [=mid(b1, 3, 3)] padding digit for third url sector [0] (boilerplate until the uspto moves into triple-digits) final url sector [=left(b1, 2)] pdf filename [0.pdf] (boilerplate) constructing the generated directory/filename [=concatenate(c1,”/”,d1,”/”,e1,f1,”/”,g1)] final constructed url [=concatenate(“http://pimg-fpiw.uspto.gov/fdd/”,h1)] once we generated a list of constructed pdf urls, we then wanted to automatically download them while renaming the pdfs from “0.pdf” to their unique patent identifiers. the easiest solution was to use gnu wget, a command line program. wget’s “-o” download option, normally used to concatenate multiple files into one written output file, can also be used as a simple renaming command when downloading one specific file. we set up another column in our url spreadsheet that transformed the patent identifiers (column a) and final constructed urls (column i) into wget commands: =concatenate(“wget -o “,a1,”.pdf “,i1) the final column of wget commands were then copied to a text file and saved as an executable .bat file, which successfully downloaded and renamed our entire list of patents. a sample of this spreadsheet is available for download. accessing patents after the patents were downloaded, the team began discussing strategies to make them searchable within rice’s digital scholarship archive. our digital scholarship archive — a dspace instance — utilizes the software’s pdf text extractor, which extracts recognized text from pdfs and stores it as a searchable text bitstream. unfortunately, the pdfs batch-downloaded from the uspto web site turned out to be image-only scans, with no optical character recognition (ocr) performed. this left us with the option to use adobe acrobat’s “recognize text” feature to manually ocr the files, but the potential textual errors left us wondering if this was our only option. we briefly flirted with an alternate plan: uploading the pdfs to the scholarship archive as image scans, but accompanying them with text-only versions, acquired by downloading html copies of the patents (using a modified form of the wget spreadsheet) and converting them to text-only. however, we assessed that this option would introduce complications into the future workflow for depositing new patents, and the error rate of the acrobat ocr method was deemed an acceptable risk. indeed, throughout the initial upload, our focus swiftly began to shift beyond the task at hand to dealing with monthly patent searches and deposits. once the payload for the initial batch of patents was described and ready for ingest, we turned our attention to devising a feasible, ongoing plan to add all new rice patents. the team eventually agreed that moving forward, the government information librarians would conduct a monthly search for all new rice-generated patents; then, following a set of instructions, they would acquire the pdfs, ocr them, and manually add them to the repository. because the fpo website allowed the librarians to not only conduct their monthly searches, but store the search results, export metadata, and download pdfs, much of the workflow was written with the web site in mind, though perhaps not on a long-term basis. (see the following section for a more detailed discussion). results and further work the initial batch of 365 patents was added to the rice digital scholarship archive in early may of 2015. as predicted (based on the number of monthly rice patents issued between 2014 and 2015), the number of patents issued in the ensuing months averaged between two and three new rice-related patents per month. besides being on-target with our workload estimations, two to three new patents per month provided our government information librarians with enough practice at depositing material into the digital scholarship archive — a relatively new experience for them — without being overwhelming. there were a handful of unresolved issues requiring further work. one such issue concerned the rights statement we applied to each patent deposited into the repository. patent descriptions for utility and plant patents are published into the public domain 18 months after the patent is filed, while design patent descriptions are made available when the patent is granted; accordingly, we assumed our rights description for all patent metadata would be “public domain.” but what about patents that contained copyrighted content? many patents cite proprietary journal articles which contain information relevant to the patent, but the articles are not included in a download of the patent, so that was not a concern. however, a patent author might elect to include copyrighted images to illustrate his or her point. we consulted sections 601.08 d and e of the manual of patent examining procedure (mpep) containing the rules and regulations governing patents (mpep 2014). section 601.08-d states that a “copyright or mask work notice may be placed in a design or utility patent application adjacent to copyright and mask work [2] material contained therein.” section 601.8-e then dictates that the notice must indicate that the owner “has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the patent and trademark office patent file or records, but otherwise reserves all (copyright or mask work) rights whatsoever.” because this statement is already included in a patent, there was  no need to post a specific copyright disclaimer; however, rather than classifying each repository entry as public domain, we dispensed with a standard rights statement and added text to the patent collection page encouraging patrons and researchers with questions to contact fondren library’s patent and trademark resource center. the other major issue concerned the source of our searches, metadata, and pdf downloads in the on-going workflow. though quite useful, the fpo website is, ultimately, a non-governmental, for-profit website; while neither of those descriptions are intended to disparage the site or condemn its contents, we as librarians are constantly mindful of becoming dependent on impermanent sources of information. in the future, we would like to pursue a coding project that would not only download new patents, but mine the text for metadata extraction, making the project completely in-house and independent of sources outside of the uspto. but for now, we will continue to make use of our workflow as-is. we will also use various methods to assess the success of the project. first, we will monitor usage statistics to determine future directions. the digital scholarship archive conveniently gathers statistics such as top 10 visited items, total visits per month, top country views, and top city views. if usage statistics are low, do they improve after a publicity campaign? if the national (or international) impact appears to be more than the local, to what groups do we need to advertise more? what departments on campus do we need to share usage statistics with to raise the profile of the university? finally, we will make use of our internal usage statistics, while also soliciting feedback from our users via a comments/suggestions request on the patent collection home page. with an eye to future improvements and data from the past and present, the rice patent collection has the potential to become and remain a vital part of the rice digital scholarship archive and a window to the world revealing the research conducted at rice university. acknowledgments the authors wish to acknowledge the efforts of the entire fondren library team that contributed to the success of this project: siu min yu, monica rivero, shannon kipphut-smith, kathy weimer, and lisa spiro. end notes [1] the united states patent and trademark office issues three kinds of patents: utility, design, and plant. design patents are granted to new, original, and ornamental designs for articles of manufacture; plant patents are awarded to researchers for the invention or discovery (and asexual reproduction) of distinct and new plant varieties. [2] under united states code title 17 § 901(a), a mask work is broadly defined as the topographic (layout) creation embodied in the design of an integrated circuit. title 17, sections 901 to 914 form the semiconductor chip protection act of 1984, which protected the intellectual property rights associated with circuit design. bibliography 17 u.s.c. § 901(a)(2) (2015).  available from: http://uscode.house.gov/ cummings, brian. 2014. “the changing landscape of intellectual property management as a revenue-generating asset for u.s. research universities.” george mason law review 21(4): 1027-1047. manual of patent examining procedure. section 601.08: detailed description and specification of the invention [r-11.2013] (9th ed. rev. march 2014) available from: http://www.uspto.gov/web/offices/pac/mpep/s608.html nicol, dianne. 2008. “strategies for dissemination of university knowledge.” health law journal annual 2008: 207-234. striukova, ludmila. 2009. “value of university patents as a determinant of technology transfer.” international journal of technology transfer and commercialisation 8(4): 379-391. “top 100 worldwide universities granted u.s. utility patents in 2014.” national academy of inventors and the intellectual property owners association. available from: http://www.academyofinventors.com/pdf/nai-ipo-top-100-universities-2014.pdf wallmark, j. torkel. 1997. “inventions and patents at universities: the case of chalmers university of technology.” technovation 17(3): 127-139. doi: 10.1016/s0166-4972(97)00094-1 about the authors scott carlson (http://orcid.org/0000-0002-6764-1238) is the metadata coordinator at fondren library, rice university. he received his mlis from dominican university in river forest, illinois, and an archives certificate in digital stewardship from simmons college. scott is also the co-founder of indie preserves, a website that provides practical preservation advice to independent music labels and bands. his twitter handle is @scottythered. linda spiro is a government information librarian at fondren library, rice university. she has served on the mentoring, strategic plan, and bylaws committees of the patent and trademark resource center association (ptrca). for the government documents roundtable (godort) of the american library association (ala), she has served as secretary and chair of many committees. she has master’s degrees from the university of texas at dallas (special education) and the university of north texas (library science). subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – simplifying subject indexing: a python-powered approach in kbr, the national library of belgium mission editorial committee process and structure code4lib issue 59, 2024-10-07 simplifying subject indexing: a python-powered approach in kbr, the national library of belgium this paper details the national library of belgium’s (kbr) exploration of automating the subject indexing process for their extensive collection using python scripts. the initial exploration involved creating a reference dataset and automating the classification process using marcxml files. the focus is on demonstrating the practicality, adaptability, and user-friendliness of the python-based solution. the authors introduce their unique approach, emphasizing the semantically significant words in subject determination. the paper outlines the python workflow, from creating the reference dataset to generating enriched bibliographic records. criteria for an optimal workflow, including ease of creation and maintenance of the dataset, transparency, and correctness of suggestions, are discussed. the paper highlights the promising results of the python-powered approach, showcasing two specific scripts that create a reference dataset and automate subject indexing. the flexibility and user-friendliness of the python solution are emphasized, making it a compelling choice for libraries seeking efficient and maintainable solutions for subject indexing projects. by hannes lowagie and julie van woensel introduction as the national library of belgium (kbr), we classify every book that comes in through legal deposit. classification and subject indexing crucial for organising our vast collection and facilitating information retrieval for our patrons. to streamline this process, we have explored various approaches, including using annif [1] and the microsoft category classification model [2]. each approach has its strengths and weaknesses, leading us to seek a solution that offers both ease of use and maintainability. this paper delves into our initial exploration of using python scripts to establish a reference dataset and automate the classification process. the reference dataset script uses a csv export of existing records that already have a manually assigned classification and generates a new csv file with associated words per classification category. the classification script takes marcxml files of records (the dataset where you want to add a classification, so for example the records that do not have a classification yet) as input and generates augmented marcxml documents as output, seamlessly integrating the assigned classifications. our primary objective is to demonstrate the practicality and adaptability of this method, with a particular emphasis on ensuring user-friendly interactions and transparent results. we made the workflow based on python scripts because of its well-earned reputation for user-friendliness. python’s syntax and readability make it particularly accessible, even for individuals with limited programming experience. this ease of use was a critical factor in our initial exploration as it allowed our team to quickly prototype and implement the scripts essential to our research without facing unnecessary complexities. it also aligned with our objective of creating a solution that could be easily adopted and adapted by a broad range of users. that is why we did not systematically assess or compare it against alternative programming languages in this particular context. the challenge of (automated) subject indexing subject indexing, a crucial aspect of organizing information, has traditionally relied on manual methods to categorize content based on its subject matter. the conventional approach involves human indexers carefully reading and analyzing documents to assign relevant subject headings or terms. while this method has been the cornerstone of information organization for years, it comes with inherent limitations. traditional subject indexing relies heavily on human expertise. the process demands skilled indexers who are knowledgeable in various domains. it can also be time-consuming and the dependence on human input affects the overall efficiency and accuracy of the indexing process. moreover, as the volume of information grows exponentially, manually indexing a vast array of documents becomes increasingly impractical. the traditional approach struggles to scale effectively to handle the sheer magnitude of digital content generated daily. recognising the limitations of traditional subject indexing, there is a growing need for a more streamlined and automated approach. an automated system can address these challenges and enhance the efficiency of information organization. first, automation can significantly accelerate the subject indexing process, allowing for the rapid categorization of large volumes of content. secondly, automated systems help ensure a consistent and standardized application of subject headings. this minimizes subjective variations and ensures a higher level of accuracy across the indexed content. thirdly, automated approaches are inherently scalable and well-suited for handling vast datasets. they can adapt to the increasing volume of information, providing a solution that is both efficient and sustainable. lastly, maybe the biggest advantage is that automated subject indexing can leverage advancements in technology, such as natural language processing and machine learning, to improve the accuracy and relevance of automatically assigned subject terms. this integration enables a more sophisticated and adaptable indexing process. the possibilities of artificial intelligence subject indexing has been a pioneering and widely adopted application of artificial intelligence (ai) in libraries, effectively streamlining the process of organizing and retrieving information [3]. however, its implementation is not without its challenges. two primary approaches to subject indexing using ai can be identified: using a self-trained model or employing a large language model. training models for subject indexing can be a challenging and time-consuming endeavor. while the customization and potential accuracy offered by self-trained models are appealing, the process of constructing a comprehensive training dataset and effectively monitoring the model’s performance can be quite demanding. one of the primary difficulties lies in building a sufficiently large and representative training dataset. this dataset should encompass a wide range of subject terms and the corresponding texts they represent. acquiring such a dataset can be a lengthy and labor-intensive process, often involving the manual extraction and compilation of data from various sources, such as library catalogs and online repositories. in our exploration of this option, we created a dataset using sparql queries to extract summaries from data.bnf.fr. this process entailed first identifying the subject terms we wished to include and then, for each term, locating at least ten associated summaries. this proved challenging, particularly for subject terms with a limited publication history or those specific to particular disciplines. even after establishing a comprehensive training dataset, the task of monitoring and refining the self-trained model remained a continuous process. as the model encounters new documents to index, it may occasionally generate inaccurate, biased or misleading suggestions. identifying the root cause of these misclassifications can be complex, as the model’s decision-making processes are often opaque and difficult to interpret. without a clear understanding of why a particular term was suggested and not another, it becomes challenging to effectively refine the training dataset and improve the model’s performance. this lack of transparency can hinder the development of a truly reliable and consistent subject indexing tool. in contrast, large language models like openai’s chatgpt or google’s bard offer a more automated and scalable approach to subject indexing. these models are trained on massive amounts of text data, enabling them to grasp the intricacies of human language and classify texts with greater accuracy. however, this versatility comes with the potential for inconsistency and a lack of control over the terminology used. in summary, while both self-trained models and large language models hold promise for automating subject indexing in libraries, each approach presents unique challenges. self-trained models offer greater customization and the potential for enhanced accuracy, but their training and monitoring can be demanding and opaque. large language models provide scalability and automation but may lack consistency and standardized terminology. the choice between these approaches ultimately depends on the specific needs and priorities of the library or organization. words matter: an explanation of our approach to better understand how human catalogers determined the subject of a text, we observed their work processes. there are many things that determine a classification such as the author’s biography, or the publisher’s name (one publisher published only law books, for example), as well as the cover (children’s books or cookbooks are easily recognised by the cover), but of course also the back cover text. it was noteworthy that some catalogers indicated that they did not read that entire text in depth, but rather “scanned” the text, looking for key words of that text, to then determine the classification based on those key words. human catalogers primarily rely on the presence of specific keywords and phrases within a text to infer its subject matter. this recognition inspired our approach to developing a system that places a central focus on the semantically significant words. in contrast to methods that often involve training sets with entire sentences, we opted for a distinctive system that centers around the extraction and analysis of specific words. our system operates on the premise that these individual words serve as the building blocks for comprehending and categorizing content. we have devised a mechanism that suggests terms based on the contextual use of these linguistic units. this departure from sentence-level training sets to a more granular emphasis on words not only aligns with the observed practices of human catalogers but also presents a pragmatic solution. our methodology prioritizes the semantically significant words, recognizing them as crucial elements in the intricate process of classification. this approach, while intuitively appealing, is not without its limitations. it can be subjective and prone to errors, especially when dealing with complex or nuanced topics. for instance, a text about the history of the middle ages might explicitly mention “knights,” “castles,” “crusades,” and “counts,” leading a human cataloger to confidently assign the subject category “middle ages”. however, a more nuanced analysis might uncover subtler themes related to social structure, political dynamics, or cultural developments, requiring a more comprehensive approach. but our primary objective was to establish a general classification for the belgian publications. for this purpose, a broad classification, such as “middle ages” would suffice. furthermore, we sought a system that would limit its subject suggestions to the specific terms defined within our classification system for the belgian bibliography. the system should not propose any term outside of our national bibliography’s classification framework. lastly, we also wanted a system that offers a more manageable and interpretable approach, allowing for greater control over the indexing process and enabling subject experts to refine and adapt the indexing rules as needed. our work aims to bridge the gap between the intuitive and practical approach of human catalogers and the more sophisticated capabilities of natural language processing (nlp) based methods. by combining these strengths, we hope to develop a practical and accurate solution for subject indexing in the context of belgian languages (primarily dutch and french), providing a valuable tool for improving the accessibility and discoverability of information in libraries and archives. while it may not achieve the same level of precision as self-trained models or large language models, it offers a more manageable and consistent approach for large-scale indexing projects. furthermore, our emphasis on the semantically significant words in the classification process aligns seamlessly with python’s prowess in handling textual information. python emerges as an ideal choice for automating tasks involving linguistic analysis, thanks to its extensive array of libraries and tools for data manipulation, analysis, and transformation. the intuitive syntax of python substantially reduces the learning curve, enabling users, even those without extensive programming backgrounds, to swiftly develop scripts for diverse tasks. the active and robust python community ensures continuous support and a wealth of documentation, facilitating troubleshooting and fostering collaboration [4]. case in point, although we have backgrounds in history and art history, we successfully authored python scripts that significantly aided us in improving and enriching our bibliographic and authority records. consequently, it is unsurprising that, in our pursuit of an automated subject solution, we naturally turned our attention to python. in our pursuit of an optimal workflow, we established several criteria. first, the creation of classification had to be clear and verifiable. equally significant was the ease of maintenance for this reference dataset. thirdly, transparency in the output of terms was a key priority: we sought to understand the rationale behind terms presented and, in the event of inaccuracies, an easy path for correction. accessibility was paramount, aiming for corrections to be effortlessly executed even by individuals with basic knowledge, opting for user-friendly tools such as excel rather than more complex formats like xml or json. lastly, the correctness of suggestions was imperative, ensuring that our workflow not only facilitated ease of use but also provided a tangible advantage in terms of accuracy. these criteria directed our developmental efforts, ensuring that the resulting workflow would be both effective and user-friendly. the workflow is illustrated in the following diagram : from exported bibliographic records, used for the creation of the reference dataset, to the generation of enriched bibliographic records ready for import into our cataloging system. our library management system (lms) enables us to export records, apply batch modifications externally to the lms, and subsequently re-import them to effect updates. this approach is adopted due to the inherent limitations of batch operations within our lms, compelling us to optimize efficiency through external modifications. our workflow unfolds as follows: we initiate the process by exporting records from our library management system (lms), each already assigned a classification. we export the pertinent information from these records, which is the title (245$a), remainder of the title (245$b), summary (520$a), and their corresponding classifications. opting for a csv export facilitates this operation, as we only require specific fields, and configuring an export profile tailored to these fields proves more manageable. subsequently, our first python script is executed to organize the texts by classification, with the associated words from the meaningful texts appended in the second column. the outcome is a csv file presenting individual classifications paired with their respective meaningful words, forming our reference dataset. afterwards, we further refined the reference dataset with batch ’replace all’ operations to eliminate less meaningful words and verbs. moving forward, we needed records awaiting classification, denoted as our input.xml. our second python script extracts the same crucial fields (245$a, 245$b, 520$a) and cross-references them with the words present in the reference dataset. subsequently, it suggests a classification and appends it to the bibliographic record (084$a). the script also incorporates print statements, to provide insights into the considered words and elucidate the reasoning behind a particular term’s prominence. this information proves valuable in manual corrections of the reference dataset by adding or removing specific words associated with a category. figure 1. representation let’s explore now the two specific scripts that played a pivotal role in establishing our automated subject indexing workflow. first script: creating the reference dataset creating the dataset is a pivotal step in our automated classification. initially, we extract existing records from our database, focusing on those already classified. this extraction results in a csv file, where classification terms (taken from our local classification system, that is stored in a local field 911$a were put in column ‘classifications’, while column ‘related terms’ includes relevant information such as title (245$a), subtitle (245$b), and summary (520$a). this script organises all records by classification term, and aggregates all associated words. in the process, stopwords (like articles) are deleted. this meticulous procedure culminates in the creation of our dataset, represented by the output file named “referencedataset.csv.” the code first imports the necessary libraries, including the natural language toolkit (nltk) [5], stopwords, and the csv library. it then downloads the nltk stopwords corpus, which contains a set of common words that are often filtered out for text processing. this is only necessary the first time. import nltk from nltk.corpus import stopwords import csv nltk.download(’stopwords’) # only necessary the first time next, we make a set of stopwords in english, french and dutch. the filter_stopwords() function takes a word as input and checks if it is in the set and thus a stopword. if it is, it will not be considered. if it is not a stopword, the function returns the word in lower caps. stopwords = set(stopwords.words(’english’) + \ stopwords.words(’french’) + \ stopwords.words(’dutch’)) def filter_stopwords(word): return word.lower() not in stopwords the function extract_unique_words() receives a text as its input and dissects it into individual words by employing the split() method. this results in a word list, disregarding any punctuation or whitespace characters in the text and eliminating stopwords. the isalpha() method is applied to verify that each word solely consists of letters (a-z and a-z, including letters with accents). words containing non-alphabetic characters are discarded, a deliberate choice to minimize meaningless data. consequently, words that are years or centuries are omitted, though it can be manually incorporated (e.g., as an associated term related to the ’history’ term). finally, the function appends the unique words to a set and returns the list of these distinctive words. def extract_unique_words(input_text): words = input_text.split() unique_words = set() for word in words: cleaned_word = word.strip(’.,’) if filter_stopwords(cleaned_word) and (cleaned_word.isalpha()): unique_words.add(cleaned_word.lower()) return list(unique_words) next, we define the names of input and output file. input_file_name = ’export.csv’ output_file_name = ’referencedataset.csv’ the code then creates an empty dictionary dict_classifications to store the extracted unique words for each classification. it iterates through the csv data, reading each row as a list using next() to skip the header row. for each row, it extracts the classification and input text, calls the extract_unique_words() function to get a list of unique words, and checks if the classification exists in the dict_classifications dictionary. if the classification is not found, it adds a new entry to the dictionary with the classification and the list of unique words. if the classification exists, it checks if any of the unique words are already present in the dictionary’s value for that classification. for each unique word, if it’s not already present in the classification’s value, it appends it to the list. finally, it iterates through the dict_classifications dictionary, creating a new row for each key-value pair and writing it to the output csv file using csv.writer(). with open(input_file_name, ’r’, encoding=’utf-8’) as infile, \ open(output_file_name, ’w’, encoding=’utf-8’, newline=’’) as outfile: listreader = csv.reader(infile, delimiter=’;’) listwriter = csv.writer(outfile, delimiter=’;’) dict_classifications = {} next(listreader) for row in listreader: classification = row[0] input_text = row[1] list_result = extract_unique_words(input_text) if classification not in dict_classifications: dict_classifications.update({classification: list_result}) for i in list_result: if i not in dict_classifications[classification]: dict_classifications[classification].append(i) else: pass for key, value in dict_classifications.items(): listwriter.writerow([key, ’, ’.join(value)]) once this script is executed, you’ll have a dataset ready for deployment in the subsequent script. however, we found it beneficial to perform some manual cleaning by deleting and eliminating extraneous words—those not essential or contributory to the classification (such as names, certain verbs, places, and occasionally dates (but some were kept because they might enhance the classification in categories like ’history’). table 1 : end result of script createdataset.py (after deleting non-significant words like author, monograph, book) export.csv (existing records: classification; 245$a+245$b+520$a referencedataset.csv (after script createdataset.py) biology;the selfish gene. this book explores evolution though the lens of genes, suggesting they frive behaviors that maximize their own survival. history;a people’s history of the united states. this study offers a perspective on american history from marginalized voices and social movements. biology;sapiens. the author charts the history of homo sapiens, examining the biological factors that shaped human evolution. sociology;bowling alone. this monograph explores the decline of social capital in modern society, highlighting the consequences of diminishing community engagement. history;the silk roads. this book traces the interconnected history of civilizations along the ancient trade routes, revealing cultural exchanges and influences. sociology;the sociological imagination. this book encourages individuals to connect personal experiences with broader social structures and historical context. history;states, perspective, american, movements, marginalized, offers, people’s, study, united, voices, history, social, civilizations, influences, traces, interconnected, revealing, along, roads, trade, ancient, cultural, routes, exchanges, silk biology;biological, examining, factors, homo, sapiens, human, charts, shaped, evolution, history sociology;bowling, capital, alone, modern, society, social, diminishing, explores, engagement, highlighting, community, consequences, decline, broader, connect, personal, structures, experiences, encourages, imagination, individuals, sociological, historical, context in this example, we use the ‘textual’ terms (biology, history, sociology), but our model was trained using identifiers from our classification terms rather than the textual terms themselves. this approach enables us to seamlessly integrate enriched bibliographic records into our catalog, associating them directly with the accurate thesaurus term or topical name authority rather than relying solely on a text field containing the term. second script: automated indexing the automated indexing script represents a crucial advancement in our subject indexing strategy, streamlining the intricate process and enhancing efficiency. at its core, this script aims to automate the assignment of a classification to bibliographic records based on their content. upon completing the referencedataset.csv, the next step involves exporting the bibliographic records that needed a classification. we opted for a method that exports the entire bibliographic record, appending the classification in a designated field. in this article it adds a marc21 field 084 (other classification number), but in our environment we use a local field 911, specifically created by our lms provider to add a thesaurus. so we export bibliographic data from our lms (=‘input.xml’). this file is treated by the python script ‘indexing.py’. the result is an enriched marcxml file that can be seamlessly re-imported into our lms to update the existing records (=’output.xml’). the script operates in two main phases. first, it analyzes the curated referencedataset.csv, organizing records by classification term and compiling associated words. the script then employs natural language processing techniques to establish connections between subject terms and the content of the bibliographic records. it counts how many times each associated word is used in the combined text. it sorts the related terms based on that count and then adds the top three words as classification suggestions. let’s delve into the key components of the script and explore its functionality: 1. import libraries the code first imports the necessary libraries and modules for the task, including csv, xml.etree.elementtree, and nltk. csv: to read and parse the csv file containing suggested terms and associated words. xml.etree.elementtree: to parse and modify the marcxml files. nltk: here we use the tokenize function from the nltk to perform nlp analysis on the text extracted from the marcxml files. import csv import xml.etree.elementtree as et from nltk.tokenize import word_tokenize 2. define function to extract text the getelementvalue() function is used to extract text from our marcxml file. it takes two arguments: an xml element and a separator string. the element can be a single element or a list of elements. the separator string is used to join the values of the elements. the function first checks if the element is none. if it is, then the function returns an empty string. if the element is not none, then the function checks if it is a list. if it is, then the function iterates over the list and appends the text of each element to a list. the function then joins the values of the list with the separator string and returns the joined string. if the element is not a list, then the function checks if it has a text attribute. if it does, then the function returns the value of the text attribute. if the element has neither of these properties, then the function returns an empty string. def getelementvalue(elem, sep=’;’): if elem is not none: if isinstance(elem, list): valuelist = list() for e in elem: if hasattr(e, ’text’): if e.text is not none: valuelist.append(e.text) return ’;’.join(valuelist) else: if hasattr(elem, ’text’): return elem.text return ’’ 3. parse csv file next, the code reads the referencedataset.csv file using the csv library and creates a dictionary term_associations using the csv data (the referencedataset.csv). each key represents a subject term and the corresponding value is a list of associated words. with open(’referencedataset.csv’, newline=’’, encoding=’utf-8’) as csvfile: csv_reader = csv.reader(csvfile, delimiter=’;’) term_associations = {rows[0]: rows[1].split(’, ’) for rows in csv_reader} 4. parse marcxml file and extract text the code parses the input.xml file using the et library and extracts the root collection node from the parsed xml document. it also defines the namespace for the marcxml. next, the code iterates through each record in the collection node. for each record, it creates an empty string combined_text to store the combined text from specific data fields. it processes each record and extracts the title (245$a), remainder of title proper (245$b), and summary note (520$a) from our marc21 record, using the pre-defined getelementvalue function and xpath expressions. it appends the values of marc21 fields 245$a, 245$b and 520$a to the combined_text string. it then prints the combined text (for debugging if necessary). et.register_namespace(’’, ’http://www.loc.gov/marc21/slim’) tree = et.parse(’input.xml’) collection = tree.getroot() all_ns = {’marc’: ’http://www.loc.gov/marc21/slim’} for record in collection: combined_text = "" field245a = getelementvalue(record.find(\ ’./marc:datafield[@tag="245"]/marc:subfield[@code="a"]’, all_ns)) field245b = getelementvalue(record.find(\ ’./marc:datafield[@tag="245"]/marc:subfield[@code="b"]’, all_ns)) field520a = getelementvalue(record.find(\ ’./marc:datafield[@tag="520"]/marc:subfield[@code="a"]’, all_ns)) combined_text = field245a + " " + field245b + " " + field520a print("combined text:", combined_text) 5. nltk word tokenization the next part of the script focuses on tokenizing and filtering the combined text for further analysis. the word_tokenize function from nltk is used to tokenize the combined_text into individual words. this function considers punctuation, contractions, and other special characters as separate tokens. then, the script iterates through the tokens and performs two filtering steps: lowercase conversion: converts all tokens to lowercase using a list comprehension. this helps normalize case differences and prepare for further analysis. alphanumeric check: keeps only tokens containing alphanumeric characters (letters and numbers) using the isalnum() method. this removes punctuation, symbols, and other non-word elements. a variable count is initialized to 0, for later use in counting occurrences of specific words or other analysis tasks. overall, this part of the script breaks down the text into individual words, converting them to lowercase, and filtering out non-alphanumeric characters. # nltk tokenization and analysis for french texts tokens = word_tokenize(combined_text) filtered_tokens = [word.lower() for word in tokens if word.isalnum()] count = 0 6. counting the occurrences of words this part of the script focuses on counting the occurrences of associated words in the filtered combined text. this part analyzes the filtered_tokens by counting how often each associated word (from the term_associations created in step 3) appears in it. it creates a word_counts dictionary to store these occurrences. this provides insights into the presence of relevant terms based on their associated words. for each associated word, it counts the number of occurrences in the text using the sum() function. based on the word_counts, the terms are sorted and saved in the sorted_terms dictionary. in the provided script, the occurrences of associated words in the combined text are counted individually. this means that if an associated word appears more than twice in the combined text, it will be counted as multiple matches. this is done to ensure that all instances of the associated words are considered when determining the relevance of a subject term. the script maintains a word_counts dictionary to track the occurrences of each associated word. this dictionary is used to sort the term associations based on the overall frequency of their associated words. as a result, subject terms with more associated words and higher occurrence counts are considered more relevant. the sorted_terms are limited to the top three suggestions. word_counts = {word: sum(1 for token in filtered_tokens \ if token.lower() == word) for word in set(word for words in term_associations.values() for word in words)} sorted_terms = sorted(term_associations.items(), \ key=lambda x: \ sum(word_counts[word] for word in x[1]), reverse=true)[:3] 7. extract subject terms next, we initialize an empty list subject_terms to store the potential subject terms. for each term and its associated words, it creates a list matched_words containing only associated words that appear in the filtered_tokens. in other words: the matched_words list is populated with words from the associated_words list that appear in the text. if matched_words is not empty: it counts the number of unique matched words using the len() function. it adds the term to the subject_terms list. it prints information about the matched term (useful for analysis and monitoring): the term itself. the list of matched words. the total number of occurrences (summed from word_counts)./li> the number of unique matched words. a newline for formatting. it prints the final list of extracted subject terms (subject_terms), followed by a separator (—-) for clarity. subject_terms = [] for term, associated_words in sorted_terms: matched_words = [word for word in associated_words \ if any(token.lower() == word \ for token in filtered_tokens)] if matched_words: unique_word_count = len(set(matched_words)) subject_terms.append(term) print(f"matched term: {term}") print(f"matched words: {matched_words}") print(f"total occurrences: {sum(word_counts[word] for word in associated_words)}") print(f"total unique word count: {unique_word_count}") print() print("final subject terms:", subject_terms) print("----") 8. add 084subfields the next part adds the subject term in marc21 084 subfields to the marcxml records for the top three extracted subject terms based on the analysis done previously. for term in subject_terms: datafield_084 = et.subelement(record, ’datafield’, tag="084", \ ind1=" ", ind2=" ") subfield_a = et.subelement(datafield_084, ’subfield’, code="a") subfield_a.text = term subfield_7 = et.subelement(datafield_084, 'subfield', code="7") subfield_7.text = 'automatically generated' 9. save modified xml finally, the code saves the modified marcxml record to the file output.xml. tree.write(’output.xml’, xml_declaration=true, encoding=’utf-8’, method="xml") case study let’s illustrate the functionality using an example. consider a bibliographic record with the following fields: <datafield tag="245" ind1="0" ind2="0"> <subfield code="a">societal shifts</subfield> <subfield code="b">a new approach</subfield> </datafield> <datafield tag="520" ind1=" " ind2=" "> <subfield code="a"> this book delves into the intricate fabric of contemporary communities. drawing on a rich tapestry of sociological theories and empirical studies, the book examines key themes such as the impact of globalization, technological advancements and shifting cultural norms on the structure of societies. through insightful analyses, the author dissects the evolving nature of interpersonal relationships, the emergence of new social inequalities, and the role of institutions in navigating these transformations. this sociological exploration goes beyond observation, offering readers a profound understanding of the complex web of relationships that define our modern social landscapes </subfield> </datafield> if we launch the script, we get the following output: combined text: societal shifts a new approach this book delves into the intricate fabric of contemporary communities. drawing on a rich tapestry of sociological theories and empirical studies, the book examines key themes such as the impact of globalization, technological advancements and shifting cultural norms on the structure of societies. through insightful analyses, the author dissects the evolving nature of interpersonal relationships, the emergence of new social inequalities, and the role of institutions in navigating these transformations. this sociological exploration goes beyond observation, offering readers a profound understanding of the complex web of relationships that define our modern social landscapes words (sorted by occurrence, and those with 0 occurrence excluded): sociological: 2 one of the associated words related to classification ‘sociology’, counted as 2 because the word appears two times in the combined text. social: 2 one of the associated words related to classification ‘sociology’, counted as 2 because the word appears two times in the combined text. also one of the associated words related to classification ‘history’, counted as 2 because the word appears two times in the combined text. modern: 1 one of the associated words related to classification ‘sociology’ personal: 1 one of the associated words related to classification ‘sociology’ cultural: 1 one of the associated words related to classification ‘history’ so the result of this small example is sociology: 5 occurrences of 3 matched words history: 3 occurrences of 2 matched words biology: 0 matches final subject terms: [’sociology’, ’history’]. the proposed terms are sorted by number of occurrences, as it is considered as best fitting. so that is why sociology is presented first. figure 2. outcome test certainly, the effectiveness of matching improves with a greater number of associated words related to the terms. if the results prove satisfactory, we have the flexibility to adjust and potentially limit the matching to just one term instead of three. one notable advantage lies in the ease with which we can improve our referencedataset.csv. this can be achieved effortlessly through manual additions or by incorporating words from other bibliographic records. notably, any changes or additions to associated words take immediate effect without the need for retraining. unlike systems relying on nlp that necessitates training based on new words, our approach involves a straightforward comparison of utilized words. this inherent flexibility ensures the system is easily adaptable and maintainable, accommodating dynamic adjustments to suit evolving requirements. conclusion in conclusion, our exploration of simplifying subject indexing at kbr, the national library of belgium, has unveiled a promising python-powered approach. we began by acknowledging the necessity of efficiently classifying our extensive collection through legal deposit. recognizing the growing need for a streamlined and automated approach, we explored two primary ai-based strategies: self-trained models like annif and large language models such as openai’s chatgpt of google’s bard. while each approach carries its own advantages and challenges, the choice ultimately depends on the specific needs and resources of each library. despite the advancements in ai-based subject indexing, we sought a third alternative. our focus shifted to python as a solution, emphasizing its power and simplicity in automating tasks across diverse domains. introducing two specific scripts—createdataset.py and indexing.py—we demonstrated the creation of a reference dataset and the subsequent automated subject indexing workflow. the creation of the dataset included taking the values of some fields, filtering out extraneous words, but also some manual cleaning was necessary. the indexing script automated the assignment of classifications to bibliographic records based on their content, using natural language processing and associating words from a curated dataset. our case study exemplified the script’s functionality, showing the extraction of subject terms based on the associated words. it ends with adding the classification to the marcxml records. the script’s flexibility allowed for customization, limiting the number of suggested terms (and control about which terms/identifiers can be suggested and thus imported) and offering immediate adaptability to changes in the reference dataset. we believe this method proves particularly useful in situations where there is limited text or summary available, such as when dealing solely with a title and/or subtitle because the approach relies on keywords (the ’associated words’ linked to each term). for instance, it can assist in classifying old and rare books, which often lack detailed summaries but have meaningful titles with significant words like ’biblia’, ’necrologia’, ’vita’ that can help to classify the works. an additional benefit is the clarity provided by the print statements, allowing a thorough analysis of why a particular classification term was suggested and identifying the words it matched. rapid correction of associated words can follow, enhancing the overall outcome. the user-friendly nature of this approach facilitates a clear comprehension of its workings. in our view, it strikes a good balance between automation and manual management of the reference dataset, contributing to improved results. in essence, our two scripts in python proved to be a potent solution for automation, providing unparalleled flexibility and simplicity in handling data processing tasks. the ease of use and adaptability of the approach make it a compelling choice for libraries working on subject indexing projects. we encourage readers to explore the possibilities of python in their own endeavors, leveraging its rich ecosystem and community support to enhance efficiency and maintainability in information organization. notes [1] https://annif.org/. see also osma suominen, ilkka koskenniemi, annif analyzer shootout: comparing text lemmatization methods for automated subject indexing, code4lib journal, 2022-08-29 (issue 54). [2] https://learn.microsoft.com/en-us/ai-builder/text-classification-overview [3] already two years ago, the cataloging & classification quarterly attributed a whole issue (issue 8, 20221, volume 59) to the topic of ‘artificial intelligence and automated processes for subject access. [4] recently, a clear overview of the status and advantages of using python was published in code4lib journal: collins m, song x, schon s. 2023. the use of python to support technical services work in academic libraries. code4lib journal [internet]. 58:2023-12-04. available from:https://journal.code4lib.org/articles/17701 [5] see https://github.com/nltk/nltk and https://www.nltk.org/ about the authors hannes lowagie (he/him) is head of the agency for bibliographic information in kbr, the national library of belgium. he is currently working on the transition to rda, linked open data and the use of ai-technologies in the cataloguing process. he has a phd in (medieval) history (university of ghent, 2012). julie van woensel (she/her) is a metadata librarian at kbr, the royal library of belgium, where she works on improving bibliographic and authority records and supporting the transition towards rda and linked data. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – onebutton: a link resolving application to guide users to optimal fulfillment options mission editorial committee process and structure code4lib issue 42, 2018-11-08 onebutton: a link resolving application to guide users to optimal fulfillment options like many consortia, institutional members of the private academic library network of indiana (palni) provide multiple fulfillment options to obtain requested items for their users. users can place on shelf holds on items, or they can request material that isn’t held by their institution through a group circulation resource sharing network (dubbed palshare) or through traditional interlibrary loan (ill) (through worldshare ill or illiad). all of these options can be confusing to users who may not understand the best or fastest way to get access to needed materials. a php application, onebutton, was developed that replaces multiple fulfillment buttons in institutional discovery interfaces with a single openurl link. onebutton looks up holdings and availability at a user’s home institution and across the consortium and routes the user to the optimal fulfillment option for them. if an item is held by and available at their institution, the user can be shown a stack map to help guide them to the item on the shelf; if an item is held by and available at the consortium, the user is routed to a group circulation request form; otherwise, the user is routed to an ill request form. all routing and processing are handled by the onebutton application – the user doesn’t need to think about what the ‘best’ fulfillment option is. this article will discuss the experiences of one institution using onebutton in production since fall 2017, analytics data gathered, and how other institutions can adopt the application (freely available on github: https://github.com/palni/onebutton). by lauren magnuson, karl stutzman, roger peters, noah brubaker introduction when the private academic library network of indiana (palni) migrated to oclc’s worldshare management services (wms) in 2014, it was a key opportunity to increase palni’s collaboration around resource sharing. although palni had offered a shared catalog and reciprocal borrowing agreement since its inception in 1992, patrons could not request delivery of materials between palni locations except via the traditional interlibrary loan (ill) system. this reliance on ill codes and practices as the mechanism for resource sharing stymied efforts to build collaboration within the palni consortium on collection development because libraries had no guarantee of being able to rely on palni partners for faster fulfillment or consistent, generous lending policies. this reliance on traditional ill for delivery also limited palni’s ability to distinctively brand its services for faculty and staff at its member institutions and brought into question the value of a catalog search that retrieved items across the palni consortium. palni libraries noticed a strategic opportunity in moving to a resource sharing system that would integrate closely with its discovery and circulation systems to deliver physical materials with fewer procedural hurdles than traditional ill. an integrated consortial resource sharing system offered options for limiting requests to available copies and implementing consistent loan periods among partner libraries. the palshare system was born as palni’s implementation of the oclc wms group circulation features. in addition to improving turnaround time for resources held within the palni consortium, palshare has become the “face” of palni to faculty and students at palni’s 22+ supported institutions spread across the state of indiana. because of the resource sharing policy guarantees in palshare, palni libraries have begun making purchasing and retention decisions based on whether other libraries in palni already hold a copy of an item, freeing up critical space and funds for the libraries to extend their services and realizing a long-held vision of collaborative collection development. still, even as it solved some long-standing issues for palni’s libraries, the successful implementation of palshare created new problems. of those issues, the most significant was that a new resource sharing option introduced potential user confusion: should they request an item from palshare or the traditional ill network? multiple request buttons appeared on the palni libraries’ search interfaces of worldcat local and worldcat discovery [see figure 1]. at the same time, other partnerships across the state of indiana began to emerge to increase resource sharing beyond traditional ill, including a separate direct borrowing system that was largely designed to serve smaller public libraries. complex user interfaces and multiple incompatible systems threatened to hinder the progress the palni libraries were making on resource sharing. expanding the resource sharing options was leading to button confusion. figure 1. multiple request buttons appear in an item record in the discovery system. the first get it button, if clicked, directs users to a consortial request form; the second get it button directs users to traditional ill (illiad or worldshare ill, depending on the institution). end users may not know why there are multiple request buttons, or which button would be the optimal request button to click. from this context of improved resource sharing options and the attendant button confusion, palni librarians began to dream of “one button” that would sensibly navigate the various resource sharing options for patrons. a single button that would intelligently direct users to their best resource sharing option was tempting because it offered some flexibility for integrating a variety of emerging resource sharing options without thoroughly confusing the user. application design the end goal of the application was to build an openurl resolving tool that handled user requests for material by choosing the best fulfillment option for a user without the user having to think about which fulfillment option would be best [see figure 2]. figure 2. onebutton concept. given data from an openurl string, onebutton takes available metadata about an item, such as the oclc number, looks up its availability in multiple fulfillment scenarios, and directs the user to the option that will get their material to them the fastest and easiest way possible. the application is configured as an openurl resolver for worldcat local or worldcat discovery. configuration options for the display of openurl resolver and place hold buttons for worldcat local and discovery are available in oclc service configuration, which enables libraries to choose openurl resolver links to display based on item format (e.g., book, article) and availability (held by palni, held by libraries worldwide, etc.). this enabled the application to be built with the understanding that onebutton would only display for monographs that have consortial holdings (in print), as all other resource sharing for other formats (articles, media, etc.) is currently handled only through traditional ill and never by consortial resource sharing. from the user’s perspective, no matter how many fulfillment options are technically available to them (retrieve the item from the shelf, place a palshare request, or place ill request) they will only see “one button” for requesting items [see figure 3]. the button also appears on items that are held by the user’s home institution. it was determined that the button should appear on items that are held by the user’s home institution so that users can potentially request items through consortial borrowing that are held by the home institution but not available there (checked out or non-circulating). some libraries in the consortia also enable users to place ‘on-shelf holds,’ effectively paging the item and triggering a workflow in wms to retrieve the item from the shelf and place it on hold for the user. it was theorized that some users, when looking at a print book held and available by the user’s own institution, may not know what the next step is in order to borrow the item – they may not know enough about how libraries usually work to know that they can go up to the item’s shelving location to retrieve the item themselves. figure 3. worldcat discovery interface showing onebutton link resolver in action as a button labeled “get it”. in this particular case, as the item is held and available through the consortium (“palni partner libraries”), if the button is clicked, the user will be automatically directed to a consortial place hold screen (provided the consortially-held items are available and lendable). the application is written in php and leverages the oclc wms availability api (available for libraries using the oclc worldshare management system (wms)), but the underlying conceptual design behind the application could be adapted for any library management system that has an api or another programmatic way to retrieve availability information about items from metadata available in an openurl. for example, ex libris’ alma library management system features search and retrieve via url (sru) integration that includes real-time availability information about items. work is currently underway to adapt onebutton for use with alma and primo. the conceptual design of onebutton was partially inspired by umlaut, which is a ruby on rails application that is also designed to route users to optimal fulfillment options by supplying “links that take the user in as few clicks as possible to the service listed, without ever listing ‘blind links’ that you first have to click on to find out whether they are available.” it was determined that umlaut had more functionality that was needed, as it features functionality to improve article-level fulfillment by querying catalog holdings and electronic resource knowledge bases (i.e., sfx). our use case was limited to evaluating monograph holdings held locally or across the consortium, and it was determined that a custom solution would be simpler to build and maintain for the onebutton use case. umlaut could have been adapted for our use case, and onebutton and umlaut have the same goal: to make it easier for users to get to the resources they need. development onebutton utilizes the oclc php authorization library (available on github) to send authorized queries to the oclc worldshare management system (wms)  availability api. libraries that use oclc’s wms library management system can request api key credentials from the oclc developer network. the oclc php authorization library and guzzle are invoked by onebutton (installed using composer) and are used to build and send authorized requests to the api. because onebutton is implemented in worldcat local or worldcat discovery as an openurl 1.0 resolver, well-formatted openurl data is appended to the url sent to the onebutton script when the onebutton link is clicked from the discovery interface. onebutton stores the complete openurl data (for use if the end user is ultimately routed to a traditional ill form that utilizes openurl data, such as an illiad request form) and also specifically retrieves the oclc number for the title from the openurl string. the oclc number is used as the primary identifier of the item and is included in the request sent to the wms availability api. an instance of the onebutton application is set up for each institution in the consortium with its own configuration settings, so each institution can determine which forms they want users to see for ill (e.g., illiad or worldshare ill forms). each institution configures their own local instance of onebutton in oclc’s service configuration as an openurl resolver baseurl and configures the button to display only for monographs. when a user clicks onebutton from a given institution’s discovery system, the user is routed to that institution’s specific instance of onebutton, which evaluates holdings and fulfillment options accordingly. the request url built for onebutton also includes a parameter (x-return-group-availability) to return holdings data for all members of the consortia (which is organized like a group circulation network). a complete url sent to the wms availability api looks like this: https://worldcat.org/circ/availability/sru/service?x-registryid=128807&query=no:ocm319501305 where 128807 is the institutionregistry id for the wms library and 3820802 is the oclc number of the item pulled from the openurl string when onebutton is clicked. note that this url will only retrieve results if wrapped in a complete header request including valid api credentials, but it can be tested with the wms availability api using oclc’s api explorer. bibliographic data, holdings, and availability information is returned in marcxml. an example of holdings data returned for an item by the wms availability api is shown below: <holdings> <holding> <typeofrecord>x</typeofrecord> <encodinglevel>z</encodinglevel> <format>zu</format> <receiptacqstatus>0</receiptacqstatus> <generalretention>8</generalretention> <completeness>4</completeness> <dateofreport>160421</dateofreport> <nuccode>ocpsb</nuccode> <locallocation>west</locallocation> <shelvinglocation>west-stacks</shelvinglocation> <callnumber>pr9199.4.j336 k55 2009</callnumber> <copynumber>2</copynumber> <volumes> <volume/> </volumes> <circulations> <circulation> <availablenow value="0"/> <itemid>3456789988</itemid> <renewable value="0"/> <onhold value="0"/> <reasonunavailable>dispatched</reasonunavailable> </circulation> </circulations> </holding> </holdings> the onebutton script retrieves each holding entry and builds an array[1] for each holding that includes the institution (stored as a coded value in the nuccode, which is the location from the marc holdings 852 $a), shelving location of the item, the call number, and the number of available items. the number of available items is stored in the value parameter in the availablenow element and represents whether the item is actually on the shelf (not checked out, withdrawn or otherwise unavailable). if an item is available, the value of the availablenow element will be 1; unavailable items have a value of 0. the barcode and the holding library (represented in the locallocation element) is also included in the array. from the example shown above, the array generated by onebutton would look something like this: $line = array( "inst"=>”ocpsb”, "loc"=>”west-stacks”, "callnum"=>”pr9199.4.j336 k55 2009”, "avail"=>0, "barcode"=>”3456789988”, "localloc"=>”west”); onebutton then loops through each array and evaluates the number of available items across all libraries. if the item is available at the user’s home institution and the home institution allows on-shelf holds (set in onebutton’s configuration file)[2], the user is automatically redirected to authenticate and place an on-shelf hold for the item. if on-shelf holds are not allowed but the item is available, the user is shown a screen indicating the holdings available for the item and directing the user to find the item on the shelf [see figure 4].[3] figure 4. screen shown to users when they click a request button for an item already held and available by the user’s home institution, and on-shelf holds are not configured in onebutton’s options. if institutional items are not available but consortial holdings exist, onebutton compares the shelving locations of each held item against a list of non-lendable locations in the consortia.[4] the list of non-lendable locations was obtained by examining lending rules for consortial borrowing in oclc service configuration and recording the locations that cannot be requested by any members of the consortia. including this evaluation of shelving locations into onebutton’s processing resolved a significant point of frustration for end users when requesting items through palshare, as the default behavior for placing consortial holds in worldcat discovery and local allowed users to place requests for items that didn’t lend consortially due to policies. those requests would never be filled and would ultimately be canceled, requiring users to place a second request in the ill system. instead of directing users to place a consortial request that cannot be filled, with onebutton the user is now automatically directed to an ill request form for a much better chance of receiving their requested item quickly. after evaluating each holding array, if there is at least one item held across the consortium that shows as available and is not in the list of non-lending locations, the user is directed to a consortial place hold screen (which requires authentication). in the example above, the item has an availability value of 0, indicating the item is not available; if that particular item was the only item held by the consortia, a user attempting to request the item would be automatically redirected to an ill request form. the ill request url would be constructed using the base url of the ill form indicated in onebutton configuration[5] and the full openurl string generated by the original onebutton request in worldcat local or discovery. implementation the application was available for beta implementation in november 2017, and one palni institution (concordia theological seminary in fort wayne, indiana) has been using the code since its beta release. each time onebutton is clicked, the result of the holdings evaluation (palshare request, ill request, or show institutional holdings / place hold on institutional holdings) and the resulting url to which the user is redirected are recorded in a log file for troubleshooting and usage analysis. usage of onebutton by concordia theological seminary since november 2017 to august 2018 is shown in figure 5. 1,471 onebutton clicks have been handled by onebutton since november 2017 at concordia. figure 5. data from the first institution to go live with onebutton from november 2017 to august 2018. as shown in figure 5, since the implementation of onebutton, palshare requests (consortial resource sharing requests) have made up the majority of those 1471 requests, while ill requests represent approximately 34% of all onebutton clicks. 4.3% of clicks were on items that were held and available by the user’s home institution (so the user was shown the availability screen indicating the item’s availability and instructions to retrieve the item from the shelf).  while that number is low, it is interesting that a not insignificant number of users clicked a button to request an item that displays as held and available, indicating that at least some users were either confused by the availability display in worldcat discovery, or that they simply needed more guidance to retrieve the item themselves from the shelf. only three requests (.2% of all requests) during this timeframe were routed to ill automatically due to the wms availability api being down (not responding). while we do not yet have sufficient data to determine whether the number of palshare requests has gone up since the implementation of onebutton, several more palni institutions have implemented onebutton since its beta release, and analysis is underway to assess the extent of the impact of onebutton implementation on resource sharing use. it is worth noting that prior to the development of onebutton, concordia was not participating in the palshare resource sharing network due to concerns about potential usability issues and button confusion; but since implementing onebutton, palshare now fulfills more user requests than ill at concordia. future development onebutton is available for any oclc worldshare management system (wms) and worldcat discovery or local library to implement, and development is currently underway to adapt the application design for use by alma / primo libraries. work is also underway to automate the aggregation and reporting of analytics from onebutton clicks and integrate those statistics into a dashboard that would enable libraries to compare onebutton click activity with resource sharing and ill statistics. development is also in progress to display detailed stack maps to users when they click onebutton for an item that is held and available for their institution. the stack map would display on the availability screen shown in figure 4 above and will include detailed directions regarding how to navigate to the item in the library. although onebutton is currently designed to work only with print monographic resource sharing, there is interest in exploring the development of functionality that would direct users to open access options where those options are available. for example, it may be possible to identify whether open access options are available by looking up a requested item using the open access button api. this possibility may also lead to further development adapting onebutton to handle article requests, particularly in order to help users who encounter problems accessing the full-text of an article through traditional openurl resolving. other future development possibilities including routing users to purchase request or copyright clearance options if items meet certain criteria. while these potential improvements are in the early stages of research and development, we are ultimately hopeful that onebutton will continue to resolve many problems users encounter when accessing and requesting library materials. endnotes [1] see https://github.com/palni/onebutton/blob/master/link.php#l115 [2] see https://github.com/palni/onebutton/blob/master/config.php.sample#l14 [3] the design and display of the page showing institutionally available items is configured in the showpage function in onebutton: https://github.com/palni/onebutton/blob/master/link.php#l202 [4] an example of this configuration file is available at: https://github.com/palni/onebutton/blob/master/nonlending.php.sample< [5] the ill form url is specified in onebutton configuration here: https://github.com/palni/onebutton/blob/master/config.php.sample#l18 about the authors lauren magnuson is the development coordinator for the private academic library network of indiana and the head of collections, delivery and access at california state university, san marcos in san marcos, california. karl stutzman is director of library services at anabaptist mennonite biblical seminary in elkhart, indiana. rev. roger peters is assistant to the director of library and information services at concordia theological seminary in fort wayne, indiana. noah brubaker is associate director of the private academic library network of indiana. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – automating authority control processes mission editorial committee process and structure code4lib issue 47, 2020-02-17 automating authority control processes authority control is an important part of cataloging since it helps provide consistent access to names, titles, subjects, and genre/forms. there are a variety of methods for providing authority control, ranging from manual, time-consuming processes to automated processes. however, the automated processes often seem out of reach for small libraries when it comes to using a pricey vendor or expert cataloger. this paper introduces ideas on how to handle authority control using a variety of tools, both paid and free. the author describes how their library handles authority control; compares vendors and programs that can be used to provide varying levels of authority control; and demonstrates authority control using marcedit. by stacey wolf so what is authority control? why do i want to do it, let alone automate it? according to ifla, “authority control (or access point control) refers to the [normalization] of controlled access points (headings) and the provision of alternative and related access points.” (zumer, 2009) for example, an authority record establishes a certain spelling and form of a name as preferred and lists alternate spellings, forms, etc. of that name, as well as related access points. the same can be true with subjects and titles. by using the same form of an access point (like the author or the subject or the series title), authority control helps users find everything by a certain author or on a specific subject. authority records contain information that can help users disambiguate names of people, corporate bodies, and titles; this is very helpful with common names, like john smith. authority control can also help users identify other subject terms to use by providing broader or narrower terms related to that subject. figure 1. name authority record for apple computer, inc. (source: screenshot of the marc record retrieved from classweb.org) here’s an example of an authority record. you have the authorized access point in box 1 (apple computer, inc.). other names of the company are apple (firm) or apple computer company, as shown in box 2. if you were to search by one of these names, the authority record would redirect you to the authorized form of apple computer, inc. this record also has extra information (in box 4) linking it to the founders of the company, steve jobs and steve wozniak. it also tells you that this company has changed its name to apple, inc., and redirects the user to that record. there is a lot more descriptive information in this record (see box 2), like the company’s headquarters, their fields of activity (like computer software, household electronics, macintosh, and ipod). the sources for this information are listed at the bottom of the record (here in box 5). how we handle authority control at unt the unt library catalog runs on iii’s sierra and we are members of oclc. we have about 3 million records, adding about 100 thousand records each year, with about 75% of those being e-resources from vendor batch records. unt libraries has decentralized cataloging, with media, music, government documents, and special collections doing their own cataloging. there are 7 catalogers in the main cataloging and metadata services department who all do their own authority work. one staff person is trained to do the authority work for the rest of the library, sending new names and subjects or problems to that specific department. when i started at unt in 2013, i worked with a librarian and another staff person to do all the library’s authority control. every day, we ran headings reports in sierra, which provide us with a list of names, subjects, and titles used for the first time, as well as other potential issues in the catalog. we looked up in oclc every new name and subject on the reports, but only downloaded the authority record into our catalog if it had been established with cross references or a see also reference. we validated new names and subjects and made sure the form was correct in the newly cataloged record as well as existing records in sierra. sometimes, if a name had changed, we needed to update the authority record and the bib records. in addition to keeping up with the headings reports, we also used the library of congress’s updated subject heading lists and oclc’s closed dates lists to update our authority records and bibliographic records. this all became difficult to maintain as rda adoption led to hundreds of thousands of authority records being updated to new standards and our library staffing changed. a three person team working on authority control became a team of one or two, depending on staffing at the time. also around this time, our library started looking at what other institutions our size do for catalog maintenance. with the addition of a new head of cataloging in our department, we proceeded to explore using a vendor for authority control, so we needed to re-prioritize our current authority work. figure 2. screenshot from sierra headings reports to illustrate “preceded by” and “followed by” now, instead of searching for every new name, we have started using the “preceded by” and “followed by” lines to see if there was a potential conflict and only investigate those in oclc connexion. if we search for a record in oclc and find one, we can download it to our ils if we want, even if there are no cross references. if we spent that time to look for the record, might as well use it; we are no longer concerned about space in our catalog. we are not loading new subject records. we are not bothering to update our authority records as the national files are updated. it took too much time to look up each listed heading in our catalog (in two different indexes for names). we figure the vendor will eventually provide us with subject records and updated records, so we won’t use precious staff time working on that now. our department of 7 catalogers now do their own authority control work. we still have one staff person doing the heading reports for the whole library, but using the new process, it takes only an hour or two to complete, instead of several hours each day. automate! the most time-effective way to handle authority work is to automate it as much as possible. working in large batches can often be quicker than individually trying to search for each new name or subject heading. there are a variety of ways to automate authority work. some tools include oclc connexion, marcedit, sierra (or your ils), notepad++, excel, and vendors. oclc connexion is where my library finds bibliographic and authority records to download into our ils. we use notepad++ and excel all the time to examine and manipulate our data. excel is helpful in many ways, especially with the text to column feature and conditional formatting where you can highlight duplicates. notepad++ is great to use with regular expressions and save data in a plain text format. sierra sierra is where i can run headings reports to identify new names and subjects, invalid uses, and duplicate records. we have a bibliographic maintenance feature that will automatically update bib records to a match new heading over night, but only if the heading in the bib record matches a cross reference in the authority record. marcedit marcedit is a free software that terry reese provides to catalogers. it will authorize and validate names and subjects for you so you know you have the right form before ever loading your records. it will provide a report for you to fix variants, or you can set it to automatically correct them. it will even add the uri in $0 for you if you are preparing for linked data in your catalog. the uri is also handy for batch-searching oclc later for the authority records to add to the catalog. it can even download the authority records that match the validated headings in the file. marcedit is a great tool for editing batches of records quickly before loading them into the ils where making mass changes is much more clunky, mainly because of its ability to use regular expressions. vendors vendors like backstage and marcive allow libraries to send a back file of all their records which are updated and validated. each vendor has their own special programming to fix field tags, indicator errors, and incorrect subfield coding; both vendors offer rda enrichment services, creating hybrid rda records out of older aacr2 records by adding certain fields. the library can decide on all kinds of customizations they want made to their processing profile, usually based on local practices and policies. once the records are cleaned up, they are run through as many authority files as you want to validate headings. the library gets the updated bibliographic records back as well as files of authority records. both vendors offer a current cataloging service so that you can send newly added bibliographic records to your profile. ongoing maintenance keeps your authority records up-to-date as national authority files are updated. this can be a wonderful option, if you have the money to spend on it. it can cost tens of thousands of dollars depending on the size of your catalog. the on-going service requires a yearly fee. and you will still need a staff person to run reports, create files, and reload records. so maybe you can afford to do a clean up project, sending your records once, but don’t want to subscribe to the on-going services. or you can’t afford to even get your records cleaned up. how can you maintain your catalog’s authority files and access points as efficiently as possible? batch-loading new authority records we’ve used a combination of headings reports, marcedit and oclc to batch load authority records for certain files of vendor records. once we load a new package of records into our ils, we run the headings reports and copy the new names to excel. this data is cleaned up so that just the name remains. we use marcedit to validate the names in our vendor file and embed the uri in $0 when possible. doing this tells us which names are actually in the national authority file and have an authority record to use. the uri provides a link to the authority record for that specific entry, which contains the library of congress control number (or lccn). copy the fields with $0 to excel to compare the two lists of names. isolate the library of congress control number from the link in the $0 for those names on the headings report. run a batch search in oclc using the lccn, add any local information you might need, and export the records either directly to your ils or into a batch file on your desktop to load with data exchange. this method could work the same way with subject headings too. ongoing maintenance as mentioned before, the name authority file and the subject authority files are always changing. subject headings get new cross-references or have fields deleted. fictional characters were once created in the subject authority file, but now are in the name authority file. once the fictional character is established in the name authority file, the subject version and all related records need to be deleted. names are updated to close death dates. some are changed to conform with today’s cataloging standard instead of a previous format. so how can you keep those authority records up-to-date? oclc provides a list of closed dates for names weekly. library of congress provides a list of updated subject records monthly. the problem is, you have no way of knowing for sure if you have these headings in your catalog without searching each name or subject heading one by one. this can be very time consuming, especially when the standards change and there are numerous computerized updates to the records, like when rda made a change to how dates were recorded. (http://www.loc.gov/aba/rda/pdf/lcnaf_rdaphase.pdf) it also doesn’t make much sense to just load all of these updated records unless you have a load table that will overlay existing records; you still need the system to change the headings bibliographic records to match the new authorized version and to delete blind references for records that don’t have a corresponding bibliographic record. we stopped updating our authority records when staffing changes made it difficult to keep up and the decision to move forward with a vendor was finalized. however, a few things have changed since then. i’ve learned more about regular expressions and have gotten rather comfortable writing search queries with them. our create list feature in sierra now allows for json queries as an input option instead of needing to enter each search command line by line. this makes it so much easier to search for more than a few names at once. with a few simple tweaks to the data in notepad++, it’s as easy as copying and pasting. we’ve also become more familiar with how to run batch searches for authority records in oclc. given all of these changes, i have developed a procedure for updating these records. essentially, turn the lists from library of congress and oclc into a json query in sierra to find any records that use the updated heading. take just those headings and use oclc to batch search for the new authority records to load. then update the headings in the bibliographic records to the new form. it can be a little bit time consuming to create the searches, but not nearly as much as searching for each heading one by one. subject headings library of congress releases lists of new and changed subject headings on their website: https://www.loc.gov/aba/cataloging/subject/weeklylists/. the lists show the subject heading, the control number, and then why that heading is on the list. options include changed heading, new heading, canceled heading, or field added or deleted. changed headings show both the old and new heading, as well as any other fields affected by the change. new headings show all of the cross-references, see also notes, call numbers, and a few other related fields from the new record. canceled headings have a note about why it was canceled, usually because it was replaced by another record. headings with added or deleted fields show those with a note about it. copy the entire list to excel and isolate the control number. figure 3. isolate the control numbers in excel use find and replace to insert a ^ where the [sp was and to replace the ] with nothing; then use text to columns to move everything after the delimiter ^ to its own column. figure 4. isolate the control numbers in excel using text to columns wizard copy this column of numbers to notepad++ where you will build the json query. remove empty lines and then use replace to add the necessary text around the numbers. figure 5. add the search query terms around the lccn using regular expressions find what: (^.*) replace with: {“target”: {“record”: {“type”: “authority”},”field”: {“marctag”: “010”}},”expr”: {“op”: “regex”,”operands”: [“sh\s*$1″]}},”or”, check “regular expression” click “replace all” this regular expression will find everything in a line; in this case, it is the lccn. it then replaces with a search query that will find authority records with 010 that have “sh[as many spaces as needed][one lccn]”. the “or” on the end of the line indicates that there are more searches to perform. add {“queries”: [ before your first search term and ]} after the last term. copy and paste the json query into a create lists line that holds a large amount of records, or at least the number of record number you have. store the record type as a authority. click search and work on something else while waiting for the search to finish. once the search is finished, export the control numbers in the 010 fields and save them to a text file. in oclc, under “batch,” select “enter authority batch search keys”. use the default index of “lccn (ln:)” and then hit “import.” find your text file of 010s and select it. the dialog box under the default search index should have populated with your search terms. hit “save” and then “close”. next, under “batch” select “process batch.” check the box to the left of “defaultauth” file name (as opposed to “defaultbib”). click “online searches” in the process box and then click “ok”. you should get a “batch search report” that shows you all of the records the search found. view these records by using the “authorities” – “show”— “by local save file status.” check the “in process” box under “workflow status” and click “show records.” select all of the records you need to export and hit f5 or “export”. if your load table is set up to overlay, they will overlay the old one. if not, just use the list you created earlier and delete those records in the “delete records” function of sierra. if you don’t have too many in your list, you could replace each one individually and check for local additions (hopefully coded with a $5) that you may not want to lose. which method you choose may depend on your past practices. check your headings reports for duplicate authority records and blind references. delete the old authority records that are duplicated by the upload. resolve blind references (which come from headings changing, like fictional characters). most of the time, cross-references and other fields are only deleted from or added to the record, not changing the main entry, so there shouldn’t be too many bibliographic records to fix. closed dates closed dates are a bit more complicated because they can occur in more places. names can occur in a 1xx main entry, as a 6xx in the subjects, or as a 7xx added entry. that means you have to search each name three times. there are two different levels of thoroughness you can work towards. the second level requires more manual work, so it is up to you to decide what is worth it in the end. option 1: you can just search the authority records for the closed dates (like we did with the updated subject headings). this will find the authority records of any names on the list in your catalog, but then you will still need to change the corresponding bib records. in our catalog, we did not load authority records for every name, so we may have used the name in a bib record, but not have the authority record. we would never know if we only used this option. option 2: you can search your bibliographic records for headings that match the old name. this does a pretty good job returning a name match. you just have to sort through many potentially false matches and there is more manual work to isolate the necessary control numbers. to build the json query for option 1 using just the control numbers: copy and paste the list from the oclc website into notepad++ deleted the header text and remove any empty lines (edit–line operations–remove empty lines) sort list so that the control numbers are all together at the end of the document. (edit—line operations—sort lines lexicographically ascending) delete everything except the control numbers. to search just name authority records in our ils, include the “n” part of the control number, but the spaces and other letters before the numbers may vary so make those wildcards in the search. use search -> replace (ctrl-h) find what: ^[^0-9]+([0-9]*) replace with: $1 check “regular expression” click “replace all” now that you have just control numbers, use search -> replace (ctrl-h) find what: (^.*)? replace with: {“target”: {“record”: {“type”: “authority”},”field”: {“marctag”: “010”}},”expr”: {“op”: “regex”,”operands”: [“n.*$1″]}},”or”,? check “regular expression” click “replace all” this will search for the 010 field of authority records to see if any of the control numbers on the closed dates list match authority records in our catalog. add {“queries”: [ before your first search term and ]} after the last term. copy and paste the json query into a create lists line that holds a large amount of records, or at least the number of control numbers you have. store the record type as a authority. click search and work on something else while waiting for the search to finish. once the search is finished, export the control numbers in the 010 fields and save them to a text file. in oclc, under “batch,” select “enter authority batch search keys”. use the default index of “lccn (ln:)” and then hit “import.” find your text file of 010s and select it. the dialog box under the default search index should have populated with your search terms. hit “save” and then “close”. next, under “batch” select “process batch.” check the box to the left of “defaultauth” file name (as opposed to “defaultbib”). click “online searches” in the process box and then click “ok”. you should get a “batch search report” that shows you all of the records the search found. view these records by using the “authorities” – “show”— “by local save file status.” check the “in process” box under “workflow status” and click “show records.” select all of the records you need to export and hit f5 or “export”. if your load table is set up to overlay, they will overlay the old one. if not, just use the list you created earlier and delete those records in the “delete records” function of sierra. if you don’t have too many in your list, you could replace each one individually and check for local additions (hopefully coded with a $5) that you may not want to lose. which method you choose may depend on your past practices. update names in bibliographic records. this could be done in global update or by the system with bibliographic maintenance program. check your headings reports for duplicate authority records and blind references. delete the old authority records that are duplicated by the upload. resolve blind references. to build your json query for option 2 using old name formats: copy and paste the list from the oclc website into notepad++ deleted the header text and remove any empty lines (edit—line operations—remove empty lines) sort list so that the record numbers are all together at the end of the document. (edit—line operations—sort lines lexicographically ascending) delete all of the new name entries and the record numbers. they should be grouped together at the beginning and the end. use search -> replace (ctrl-h) and regular expressions to remove the “old: 100 1” information. find what: …: 100 1. replace with: check “regular expression” click “replace all” use search -> replace (ctrl-h) and regular expressions to remove everything after the last comma or period, leaving just subfield a find what: (.*,.*)[,\.].* replace with: $1 check “regular expression” click “replace all” repeat step 6 as necessary until we are left with just the $a of the old formats of the names. search for the delimiter symbol to see if there are any that were missed and take care of those manually. because these names could be in the 100, 700 or even used as a subject in the 600 fields, we want to search in each index so we need three sets of the old names. highlight all of them and copy and paste twice. highlight the first set of names and use search -> replace (ctrl-h) find what: (^.*)? replace with: {“target”: {“record”: {“type”: “bib”},”field”: {“marctag”: “100”}},”expr”: {“op”: “regex”,”operands”: [“n.*$1″]}},”or”,? check “regular expression” click “replace all” highlight the second set of names and use search -> replace (ctrl-h) find what: (^.*)? replace with: ?{“target”: {“record”: {“type”: “bib”},”field”: {“marctag”: “700”}},”expr”: {“op”: “regex”,”operands”: [“n.*$1″]}},”or”,? check “regular expression” click “replace all” highlight the third set of names and use search -> replace (ctrl-h) find what: (^.*)? replace with: {“target”: {“record”: {“type”: “bib”},”field”: {“marctag”: “600”}},”expr”: {“op”: “regex”,”operands”: [“n.*$1″]}},”or”,? check “regular expression” click “replace all” add {“queries”: [ before your first search term and ]} after the last term. copy and paste the json query into a create lists line that holds a large amount of records, or at least the number of record number you have. store the record type as b bibliographic. click search and work on something else while waiting for the search to finish. look through the results and compare them to the original list of names. export the 100, 700, and 600 fields from sierra. copy the list to excel and alphabetize the list (or you can use notepad++). remove duplicates from the list. if using notepad++, remove duplicates with the following replace find what: ^(.*?)$\s+?^(?=.*^\1$)? replace with: check “regular expression” click “replace all” compare the names on the list to the list from oclc. copy the record number from the oclc list of matched names and use them to run a batch search in oclc. you may need to investigate some names to make sure that a name without a date in your ils isn’t the same person as the updated name. you can search by names instead of lccn in oclc to help gather the authority records quicker, but it will still take a lot of manual investigation. in oclc, under “batch,” select “enter authority batch search keys”. use the default index of “lccn (ln:)” (or “personal names (pn:) if you want to search by names, but this will result in more records than you need) and then hit “import.” find your text file of names and select it. the dialog box under the default search index should have populated with your search terms. hit “save” and then “close”. next, under “batch” select “process batch.” check the box to the left of “defaultauth” file name (as opposed to “defaultbib”). click “online searches” in the process box and then click “ok”. you should get a “batch search report” that shows you all of the records the search found. view these records by using the “authorities” – “show”— “by local save file status.” check the “in process” box under “workflow status” and click “show records.” select all of the records you need to export and hit f5 or “export”. if your load table is set up to overlay, they will overlay the old one. if not, just use the list you created earlier and delete those records in the “delete records” function of sierra. if you don’t have too many in your list, you could replace each one individually and check for local additions (hopefully coded with a $5) that you may not want to lose. which method you choose may depend on your past practices. update names in bibliographic records. this could be done in global update or by the system with bibliographic maintenance program. check your headings reports for duplicate authority records and blind references. delete the old authority records that are duplicated by the upload. resolve blind references. searching by just the control number, i found 12 instances where we already had an authority record for the name in either our subject or name index. when i searched by the old names (option 2), i found several more that we did not have authority records for. i found at least 18 names that seemed promising for further investigation. a few of them were indeed other people, but there were about 8 names that needed to be updated in the name index alone that didn’t have authority records. this was a small sample size, but i believe it is representative of the amount of names we might miss if only searching by the control number. a few parting tips google is your friend. if i do not know how to do something in excel or with regular expressions, i google it and can usually find a decent formula or two that can meet my needs. in create lists, build a search in the classic view, then switch over to the json view to find how to write the code. use $1 or “insert value here” as your value a and it’ll be that much easier to copy your own values. wait until you have several lists to combine and then set up one search all at once, instead of setting it up weekly try to avoid copying your regular expressions from word because the automatic-formatting might cause problems. for example, a quotation mark gets changed to a slanted quotation mark automatically, which the computer interprets differently. reuse! reuse! reuse! we figure out how to set up one json query and then reuse it whenever practical you will run into times where the code doesn’t want to work. sierra will prevent you from continuing your search with a dialog box. this box shows where the problem is but it’s hard to work with it in sierra. use notepad++ to find the line and column and see what might be wrong. usually you don’t have enough brackets or too many, or you might be missing a comma or quotation marks. experience will make this easier.? even i end up playing with my queries more than i’d care to admit just to get it to work. but once you do, reuse it! bibliography zumer, m. (2009). national bibliographies in the digital age. munchen: saur. https://www.loc.gov/aba/cataloging/subject/weeklylists/ https://www.oclc.org/authority-records.html about the author stacey wolf (stacey.wolf@unt.edu) is the cataloging and metadata librarian at the university of north texas. she specializes in batch editing e-resource records, juvenile cataloging, and authority control. she is currently leading the effort to join forces with vendor-provided authority control, cleaning up the catalog and getting the most out of the services offered subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – overly honest data repository development mission editorial committee process and structure code4lib issue 34, 2016-10-25 overly honest data repository development after a year of development, the library at the university of illinois at urbana-champaign has launched a repository, called the illinois data bank (https://databank.illinois.edu/), to provide illinois researchers with a free, self-serve publishing platform that centralizes, preserves, and provides persistent and reliable access to illinois research data. this article presents a holistic view of development by discussing our overarching technical, policy, and interface strategies. by openly presenting our design decisions, the rationales behind those decisions, and associated challenges this paper aims to contribute to the library community’s work to develop repository services that meet growing data preservation and sharing needs. by colleen fallaw, elise dunham, elizabeth wickes, dena strong, ayla stein, qian zhang, kyle rimkus, bill ingram, and heidi j. imker background while data have long underpinned research findings, routine and widespread access to and preservation of research data has not been emphasized until recently. the current interest in research data stems from the desire to enable greater research transparency, increase the return on investment of government-funded research through data reuse, and demonstrate by a number of long-standing, discipline-specific data archives where data can have significant long-term value within and beyond their domain interests. although the concept of research data as an important research output has gained traction in many disciplines, the state of data availability varies dramatically. today data may be made available as separate files that are supplemental to a journal article, which is a highly flexible, albeit wildly variable, sharing mechanism. however, supplements are not always allowed (maunsell 2010; borowski 2011). even when data are shared in supplemental files, the data are frequently entombed in pdfs, which are not an ideal format for reuse and do not lend themselves to easy extraction of the data. furthermore, supplemental files are not always persistently available (williams 2016), are subject to size limitations (kenyon and sprague 2014), and are rarely independently discoverable from the article itself (carpenter 2009, imker 2016). some communities have established discipline-specific repositories, such as the inter-university consortium for political and social research (icpsr; https://www.icpsr.umich.edu/icpsrweb/), genbank (https://www.ncbi.nlm.nih.gov/genbank/), and the protein data bank (pdb; http://www.rcsb.org/pdb/home/home.do). these resources are tailored to specific communities and data types and serve both extremely well through establishing and/or enforcing standards, providing quality control, and aggregating like-data. while discipline-specific repositories are a critical component of the data ecosystem, their specificity means they are unable to accommodate the extraordinary diversity of research data that exists. finally, in recent years general-purpose repositories such as figshare (https://figshare.com/) and zenodo (http://zenodo.org/) have emerged and represent a midpoint on the flexibility continuum. such repositories do require minimal, but not discipline-specific, metadata. on one hand, the required minimal metadata is an improvement over the unstructured nature of supplemental files, but advanced curation is not supported within these general-purpose repositories due to their scale and scope. furthermore, their long-term funding models are uncertain, suggesting that their missions will be more susceptible to change in the face of sustainability. therefore, although such resources are a welcome addition to the landscape, we can be less certain about their ability to retain and preserve deposited data. it’s clear that there are gaps in the current solutions for making data available, but it’s also clear that resources are emerging and evolving. the office of science and technology policy (ostp) memo of 2013 (holdren 2013) and subsequent implementation plans from federal agencies indicate that we can expect some development of data sharing mechanisms from federal agencies in the future (scholarly publishing and academic resources coalition 2016). however, murkiness abounds as to what those developments will look like; data inventories, a data “commons,” and/or partnerships with third party services were all mentioned in agency implementation plans. while the vagueness of the agency plans make it difficult to envision what the agencies may develop in the future, we note that agency-developed repositories were universally proposed for public access to research articles. despite years of library-led institutional repository (ir) development, at the time of writing, of the 16 federal agency plans, only the us department of energy mentioned irs as a solution for public access to research articles (us department of energy 2014). it is not clear if the agencies’ lack of acknowledgement toward irs signifies a barrier with regard to roles and perspectives about what modern libraries do or if agencies feel the need to show an independent, and possibly individualistic, commitment to the public access mandate. regardless, if history were to repeat itself, in five or ten years we can anticipate that agencies will develop their own data repositories with associated requirements that researchers deposit their federally-funded data within those specific agency-lead resources; thus, we have prepared ourselves for the chance that our efforts to build a data repository may be a short-term, stopgap solution. overall strategy while the national “big picture” remains nebulous, university of illinois at urbana-champaign researchers are coming forward in search of a data publishing platform that fills the gaps left by current options. this leaves us in a predicament. how do we support the needs that researchers have today, yet plan for an unpredictable future? at some level, the research prowess of an institution is proportional to its research output, and institutional support of such output is increasingly expected. as more research receives funding, more research publications and related data will be produced. therefore, an institutional data repository serves as a parallel to our other research collections and also provides a “home” for research data with an otherwise ambiguous fate. the univeristy library at university of illinois at urbana-champaign (referred to throughout as “illinois library”) already maintains several digital repositories, including the illinois digital environment for access to learning and scholarship (ideals; https://www.ideals.illinois.edu/). ideals primarily serves as a document repository for articles, reports, theses, dissertations, presentations, and in some cases, small datasets. ideals launched 10 years ago and has since amassed >85,000 items with >22,000,000 item downloads. accessions and downloads are steadily increasing over previous years, and we are comfortable saying that ideals is successful in the goals of being persistent, collecting a variety of scholarly materials, and being adopted across campus. since 2012, the library has worked to develop medusa, a preservation repository that centralizes preservation for all of the library’s digital collections (rimkus 2015). the medusa digital preservation repository provides an enduring storage and management environment for the digital content collected or created by the library. it also serves as a backbone infrastructure platform for building public-facing access to the library’s digital materials via various web applications. we considered many options during initial planning for data preservation and access and attempted to draw on our current repository development and operations experience to assess those options: customize our current dspace ir (ideals) to better support datasets adapt our preservation system (medusa) and build an integrated web application for dataset ingest and access adapt our preservation system (medusa) and integrate with a separate existing platform, e.g. dspace, fedora/hydra/sufia, ckan, dataverse, figshare for institutions, etc. each of these options has pros and cons. ultimately, our goal was to centralize as much of the repository functions as possible, as we have found that maintaining different repository systems is taxing on resources; thus we rejected option 3. additionally, because we are uncertain about data in terms of long-term expectations, value, preservability, and scalability, we also wanted to make sure to be pragmatic in our commitments and plan for change. while we could have tried to work within the dspace platform to create additional customization in ideals, it would have required a change in policies and functionality that would likely be orthogonal to ideals’ current success as a document repository originally intended for sustained, long-term preservation. building on dspace also would have meant customizing the underlying software, which is outside of our control and likely to change. over-customization is already a problem with our ideals instance of dspace; further customization to accommodate research data would exacerbate the difficulty. in light of this, we rejected option 1. therefore, we elected to go with option 2 and develop a web application that interacts directly with medusa in order to leverage medusa’s preservation functions. this approach would allow us to benefit from the long-term preservation functions of medusa while managing data-specific policies and commitments in a separate web application. by building the “data repository” as a front-end web application on top of the medusa preservation platform, smooth integration would enable robust management and preservation of data deposits and at the same time support the library’s ongoing approach to digital stewardship. we elected to name the application the “illinois data bank” because it is not an acronym [1], relays clarity of purpose, is easy to remember, and no one objected. funding model the illinois data bank (http://databank.illinois.edu) is developed and operated primarily by the research data service (rds; http://researchdataservice.illinios.edu), a campus-wide program that provides the illinois research community with the expertise, tools, and infrastructure necessary to manage and steward research data. in an ongoing commitment, the rds was funded in 2013 by the office of the vice chancellor for research at the university of illinois at urbana-champaign, but the rds has its home in the illinois library to leverage the established professional lis, digital preservation, and repository services expertise within the library. along with developing and operating a repository for public access to illinois research data, primary services offered directly through the rds include knowledge of policies and best practices for the preservation and sharing of research data, advising on data management planning and implementation, and providing data management training. rds funding supports four full-time staff: a director, two data curators, and a repository developer. while the developer is devoted entirely to the illinois data bank, the former three have responsibilities that cover the entirety of the rds’s services. team construction although the technology stack sometimes takes center stage, a repository encompasses much more than lines of code, a server, and some storage. indeed, policies, workflows, adoption and adaptation of a metadata schema, and user interface design are all critical components of repository development work. development of the illinois data bank therefore required the input from multiple people with expertise as summarized in table 1. all parties participated in regular brainstorming and discussion of requirements, features, challenges, and potential solutions, although participation ebbed and flowed depending on the stage of development. rds staff and their developer adopted an agile development process with weekly sprints documented in a jira ticketing queue. the developer, the rds director, data curators, and clir postdoc met weekly to discuss features and progress regarding the interface and curation functions, with others participating on an as-needed basis. for particularly challenging features, such as determining the complex workflows involved in supporting embargoes, the data curators worked to suss out the functional requirements ahead of time in an attempt to present the developer with a fairly mature vision of the feature. the team iterated upon most features throughout the process of articulation, implementation, and testing. this resulted in a tight coupling of the repository developer, who otherwise works directly within the library’s repository development team, with the rds team that would be primarily responsible for promoting and operating the illinois data bank as a service on campus. we found this overall structure to be effective because it did two things: 1) created an environment where everyone represented in table 1 was mutually invested in the success of the project but did not feel overly taxed with unnecessary meetings and 2) ensured that no one role was isolated from others. while occasional (but relatively few) issues surfaced with this team structure, they arose fairly quickly because of the interconnectedness, and were likewise resolved quickly by bringing needed parties together. table 1. staff breakdown during initial illinois data bank development (may 2015 – may 2016). staff % of time primary duties during illinois data bank development repository developer 100% programming, technical implementation, testing rds director 30% direction, develop cohesion, gather researcher input and feedback, testing, arbitration rds data curation specialist 30% metadata schema, policies, copyright, curation workflow rds data curation specialist 30% environmental scans, embargoing, ticketing, curation workflow repository services manager 10% direction, technical oversight metadata librarian 10% cross-walking of metadata schema, controlled vocabularies, metadata content format, implementing datacite metadata schema 3.1 clir postdoc 10% testing, documentation information design specialist 10% usability design, usability testing preservation librarian 5% direction, technical oversight technical infrastructure illinois repository suite overview in developing the suite of repository services within the library, the repository development team adopted the microservices architectural style of designing and building software applications to support nimble and sustainable infrastructure. a suite of services emerged over short, iterative development cycles, based entirely on the business needs of the librarians. this established an ongoing relationship between those at the library, including the rds team and repository development team, and the software in production continues to evolve through continuous refactoring as new user needs and requirements, known as “user stories” in agile parlance, arise. one user story in particular is most responsible for the overall design of repository services at the library. above all, the preservation librarians wanted file-level access to all content inside the repository. they rejected the common practice employed by most popular repository solutions of restricting access to content to a set of (often web-based) apis. likewise, they required that files ingested into the repository would not need to be renamed or obfuscated and that folder names and directory hierarchy would be persisted within the repository, unmolested. this requirement would allow preservation staff to use their existing suite of digital preservation tools directly on the files in the repository via standard portable operating system interface for unix (posix) apis; they would not be too beholden to the development team to build new, or adjust existing, custom apis in order to accommodate inevitable changes in preservation tooling. to accomplish this, the base of the medusa repository infrastructure is a general parallel file system (gpfs), which provides robust, high performance, petabyte-scale, clustered file storage that can be mounted by any ordinary computer and operated on by preservation staff without the need for additional tools or training. the decision to base our repository architecture on a regular posix file system led to an important overarching strategy of “meeting users where they are,” rather than requiring retraining or onerous adjustments to established workflows as a bar to adopting our software. this strategy has influenced several other design decisions, often characterized by tradeoffs between features and ease of use, but in most cases both goals were achieved. for us, the focus on meeting users where they are has resulted in a smoother rollout and wider adoption of medusa compared to other comparable library software products developed in recent years. digital preservation the heart of illinois’ repository infrastructure is the medusa digital preservation repository (https://medusa.library.illinois.edu/; https://github.com/medusa-project) which provides long-term retention and accessibility of its digital collection (rimkus and witmer 2016). in development of medusa, the library’s goal was to closely integrate the library’s digital production, preservation, and access services in order to establish a more efficient, sensible, and sustainable digital library program. medusa is developed and managed locally by the illinois library’s repository development team, and features a web-accessible interface for collection management and initiation and tracking of preservation actions, such as format tracking, on-demand integrity checking, and storage of preservation metadata along with depositor and illinois data bank-supplied content. medusa’s storage infrastructure, established through a partnership with illinois’ national center for supercomputing applications, consists of a copy of every file replicated daily across two distinct campus nodes, both on spinning disk, and a third copy of every file backed up and stored outside of illinois. while medusa content consists primarily of digitized and “born digital” materials such as books, manuscripts, photographs, audiovisual materials, scholarly publications from the library’s special collections, general collections, and repositories, it was determined research data entrusted into library care should also be ingested into the medusa digital preservation repository. as of 2016, illinois data bank constitutes a sixth source of content in medusa. illinois data bank construction the web interface component of illinois data bank is a ruby on rails application (fallaw 2016). the repository development team uses ruby on rails to support rapid development. functional prototypes provoked and inspired meaningful feedback within weeks of the developer’s start date.the interface continues to evolve responsively to feedback. externally, illinois data bank integrates with the datacite metadata store (datacite 2016) through purdue’s ezid service (purdue university 2016). when a depositor confirms an intention to publish, the web application requests a doi (digital object identifier). within a pre-registered prefix, the ezid api supports generation of a random doi or a specified doi. illinois data bank specifies a doi that is largely opaque, but encodes version information. after ezid returns a datacite doi, illinois data bank sends a message to medusa to initiate ingestion into the digital preservation system. building on an approach medusa uses for other functionality, the messages are sent using advanced message queuing protocol, specifically using a rabbitmq server. messaging supports effective integration, while allowing independent development. custom selection of files for zipping and downloading files in published datasets, which are stored in medusa, is supported with a distinct web service called medusa downloader. policy framework in order to effectively manage the research data stewarded by the illinois data bank, we developed a robust set of policies that articulate the obligations of the illinois data bank service and its variety of users. after nine months of drafting, revising, consulting, and fine-tuning, we debuted the set of policies, procedure, and guidelines when the illinois data bank soft-launched in may of 2016 (table 2). table 2. illinois data bank policies, procedures, and guidelines. policy title url access and use policy http://hdl.handle.net/2142/91041 accession policy http://hdl.handle.net/2142/91042 deposit agreement http://hdl.handle.net/2142/91043 preservation policy http://hdl.handle.net/2142/91044 preservation review, revision, retention, deaccession, and withdrawal procedure http://hdl.handle.net/2142/91045 withdrawal guidelines http://hdl.handle.net/2142/91616 preservation review guidelines http://hdl.handle.net/2142/91617 preliminary policy efforts to begin policy development work, we looked to the policies that govern ideals (shreeves 2014) as a solid starting point because they had been vetted by university stakeholders and cover the policy areas previously identified as being important when establishing a repository for university scholarly outputs. in addition, we looked to peer repositories’ policies (as acknowledged in each of the policies listed in table 2) and reviewed policy requirements for achieving data seal of approval certification (data seal of approval 2016). the data seal of approval (http://www.datasealofapproval.org/en/) policy requirements were useful to us even though we do not have an immediate goal of obtaining certification because they helped us guide our policy framework toward one that reflects that of a trusted repository. review of established policies and data seal of approval documentation led to the understanding that the illinois data bank needed policies that governed the deposit, preservation, access, preservation review, and withdrawal of datasets. the policies needed to dictate what is expected of three major players in the illinois data bank workflow: the depositor, the repository, and the accessor. ad hoc review group after developing initial drafts, we engaged with a wide variety of campus stakeholders to form an ad hoc review group for the policies (table 3). since the illinois data bank serves all of the urbana-champaign campus, researchers’ decisions to share data in the illinois data bank could potentially impact a number of offices and roles on campus, so it was important to see that many voices were represented in the development of the policy framework. the ad hoc review group undertook three rounds of revisions and met three times. the revision process involved soliciting in-line revisions and comments from each member. the feedback gathered ranged from suggested grammatical revisions to questions that challenged us to make operational and policy changes. we compiled feedback between each round of review and highlighted major questions and concerns as discussion points at subsequent ad hoc review group meetings. the perspective brought by the variety of representatives from all over campus was invaluable to seeing that the illinois data bank policies are adequately balanced in terms of the roles and responsibilities expected of depositors, the repository (and thus the university), and accessors. an additional benefit of working with the members of the ad hoc review group was that it provided the opportunity for in-depth discussions about the intentions and functionality of the illinois data bank with professionals on campus who are connected with university researchers and may be advising researchers regarding data sharing needs in the future. table 3. participating university staff and units in illinois data bank policy review. staff role campus unit director, research data service university library data curation specialist university library senior it security engineer technology services export compliance officer office of the vice chancellor for research research ethics officer office of the vice chancellor for research deputy cio of research it and research leader at national center for supercomputing applications technology services; national center for supercomputing applications director, institutional review board office for the protection of research subjects associate vice chancellor for research and director of sponsored programs office of the vice chancellor for research; office of sponsored programs director, records and information management services records and information management services senior associate university counsel university counsel senior technology manager (1) office of technology management senior technology manager (2) office of technology management assistant dean for the graduate college graduate college associate university librarian for research university library manager, scholarly communication and repository services university library engineering, physics and astronomy librarian university library interplay between policy and repository development the creation and refinement of the illinois data bank policies occurred in tandem with the development, testing, and refinement of the illinois data bank features and interface. the team was flexible and adaptable to the adjustments that ended up being required as the policy and functional elements of the illinois data bank changed together. for example, user testing of an early version of the illinois data bank, which loaded the deposit agreement as a big block of text, indicated that many depositors were not likely to read the deposit agreement. from a policy perspective, there were particular elements of the deposit agreement that we felt were important to highlight for the depositor: the fact that depositors must either be a creator of the dataset or acting on behalf of a creator of the dataset and that all sensitive information, if any, must be removed from the dataset. the team worked together to develop an interface solution that would streamline the deposit agreement and require depositors to address these specific elements. thanks to our ongoing communication with the ad hoc review group, the deposit agreement workflow was refined over time to incorporate a workflow block (i.e., a depositor cannot move forward with their deposit when “no” is selected for any question) and text that recommends depositors seek help from the rds when they have selected “no” for any question (figure 1). furthermore, these conversations inspired more sophisticated use of the deposit agreement by prompting us to design the system to create a plain a text version of the completed deposit agreement with the timestamp of submission and depositor information, which is both saved within the dataset metadata package and also linked in the deposit confirmation email sent to all authors. figure 1. deposit agreement showing page structure and an alert when a depositor has not indicated appropriate responses. another example of the ad hoc review group inspiring development work is that it became clear that it would be necessary to incorporate a mechanism for temporarily and permanently withdrawing access to files and/or dataset metadata. the conversations leading up to our decision to implement this capability centered on a number of potential problems that could emerge as the illinois data bank began to accept submissions. worst-case scenarios were articulated and listed as examples of compelling reasons for withdrawal in the illinois data bank preservation review, revision, retention, deaccession, and withdrawal procedure (see link in table 2 for details). while one of the goals of the rds is to circumvent many of these potential issues through educational efforts, it is likely that not every depositor will have had direct interaction with us before depositing. thus, having the means to withdraw a dataset is important in case any of the withdrawal scenarios do occur. with a new policy decision—the illinois data bank has the right to withdraw datasets for a compelling reason—new system capabilities had to be developed in response. the rds developer worked with others on the repository development team to implement support for the withdrawal of files and metadata in the illinois data bank. we developed a tiered approach to escalating withdrawal actions to include temporary suppression, permanent suppression, and deletion of files and metadata. to accomplish this, new dataset and file properties were added to control how elements of display presented to system users in various roles, such as curator, depositor, and guest. for the extreme cases identified as having a legal or ethical need to delete even archived (non-public) versions of files from the preservation system, the repository development team architected potential system changes to support automated withdrawal of files from the preservation system, but did not implement prior to launch because the expectation is that the need for complete deletion would be a rare-if-ever occurrence. instead, a manual process was identified, and we shelved system implementation of an automated process until more need arises. we then worked on developing the curator-only interface for managing file and metadata temporary and permanent suppression and internal procedures for determining when to implement such measures (see curator interface vide infra). the hope is that the need for this feature is likewise scarce but having it functional and available at launch was important to ensure that if a questionable deposit was identified, it could be immediately and effectively managed. policy maintenance recognizing that the data publishing and scholarly communication landscape are constantly in active evolution, we will review all illinois data bank policies on a quarterly basis during the illinois data bank’s first year (may 2016-may 2017). in subsequent years, the policies will be reviewed as needed but at least annually. staying attentive to the work of, and working in coordination with, other library teams, such as the units who manage ideals, the university archives, and medusa, and campus groups such as the records & information management services and technology services, the illinois data bank intends to maintain the currency and robustness of its policy framework well into the future. user interface design one of the driving goals of the interface design was simplicity. we wanted to make sure that people depositing their data clearly understood what was happening with their deposit at every stage, and that the depositors wouldn’t need to struggle through a difficult process regardless of how complex the behind-the-scenes decisions had been. the process of continually refining the interface while also adding new functionality meant that traditional usability assessment techniques needed some acceleration to fit smoothly into the rapid development cycle. jakob nielsen’s usability recommendations suggest batches of 5 users at a time, all experiencing the same interface, followed by modifications, and then another batch of 5 users (nielsen 2000). however, that process would have put an unacceptable delay in the development cycle. instead, the team began with heuristic analysis, generally performed one to two morae-based usability assessments (https://www.techsmith.com/morae.html) per week, and analyzed the feedback in a way that integrated how recently the issue was encountered along with how many people had encountered it and how critical we assessed it to be. (more recently encountered issues meant that either the problem had not been resolved or the problem had been introduced.) some other concerns were assessed with a/b click testing and term recognition assessments with the aid of the optimal workshop software suite (https://www.optimalworkshop.com/). there were three major usability assessment cycles between october 2015 and april 2016, with ongoing and significant interface changes (often implemented the same day or week as the test) in response to user feedback. sometimes the initial insights revealed by heuristic analysis and user testing were right on the mark. for example, many web-based software users have been trained to expect access to personalized views and controls by clicking on their name in the upper right quadrant of a screen. so many users displayed this behavior that we added a personalized view of their own datasets in that area, even though it hadn’t been in the original development plan. other times we came up with seemingly-great ideas that didn’t pan out in actual testing. the idea of the home page containing a visual display of the stages of deposit had appealed to us even in early stages of development; however, usability testing of a graphic designer’s rendering of that idea took an unexpected turn. all of the usability testers who saw that graphic on the homepage tried to click on the graphics as though they were interactive buttons despite conscious efforts to avoid a graphic that invited interaction. because the graphic design arrived very close to launch, the team decided it was better to fall back to a less-informative general data visualization rather than add new site functionality and interactions to accommodate the unintended trigger of expectations. in some cases, user behavior patterns required an interface change that wasn’t technically necessary but was behaviorally necessary to prevent confusion. the illinois data bank is configured to save a user’s progress at every step, so that closing a window would store an in-progress draft without content loss. however, despite the presence of a google drive-like message that the draft was saved, users never felt comfortable simply closing a window during testing; they needed the explicit reassurance of a clearly labeled “save” feature. while several other labels were tested over the course of development, “save and continue” and “save and exit” were the phrases that reassured data depositors that they knew what would be happening to their data at the next step. in other cases, fairly small interface adjustments had very large results. the words “continue” and “confirm” by themselves didn’t look as “clickable” to our users as “continue >>” and “confirm >>” did. similarly, items that users perceived as headlines were suddenly made visually interactive by adding “[+]” to the start of the phrase. we made several adjustments to the tooltip “(?)” icon in terms of size, color, and placement in order to make it noticeable but not too distracting. the interface transformation between october and april was dramatic. the policy agreement was streamlined from dozens of paragraphs to key checkpoints; the deposit process shrank from several pages to one; the upload system progressed from one file at a time to several methods of uploading, including from box, a university-supported file storage service that provided larger file upload methods than the web browser alone would reliably support. new and potentially complex features such as data embargo and funder identification were also added at the same time the interface was being simplified. the usability testers at the end of april had several more tasks to accomplish, but their tests were progressing more quickly because they experienced less confusion thanks to the usability improvements introduced throughout the development cycle. by adapting the usability testing size and frequency to the agile development cycle and scheduling small but frequent rounds of feedback, we were able to test ideas quickly and adjust the interface accordingly. the experience was almost like digital sculpture, carving away complexities and refining the visual interface, testing new ideas, and responding in near-real-time to the users’ experiences. metadata schema metadata schema development to establish the illinois data bank descriptive metadata, we assembled a “metadata sub-team” comprised of an rds data curator and a university library metadata librarian, with additional feedback from the rest of the library metadata services and the illinois data bank development teams. the metadata team used the datacite metadata schema 3.1 as the starting point for developing the illinois data bank metadata (ammann et al. 2011) because every dataset deposited into the illinois data bank is assigned a datacite doi. while researchers and journals may desire dois for the sake of persistency and/or branding, we also saw a major discoverability advantage in illinois data bank metadata being transmitted back to the datacite metadata store as a potential indexing source for data harvesters. a major shortcoming of an institution system can be isolation, and we deemed it imperative that data deposited into the illinois data bank be discoverable in the larger data ecosystem. in the process of determining what information would be captured when a researcher deposited a dataset, the metadata team sought to harmonize the mandatory datacite elements and local institutional priorities and also find a delicate balance between “pain-free” and “enough” metadata for depositors. the datacite metadata schema 3.1 defines eighteen properties, five of which are mandatory: identifier, creator, title, publisher, and publicationyear. the illinois data bank automatically generates several pieces of metadata: the doi, version number, and publisher, the latter always being “university of illinois at urbana-champaign.” in addition to the requisite datacite information, the illinois data bank also requires: license/rights information email address for each author contact person date available (i.e., when the dataset is available to the public) version all of the information collected by the illinois data bank is not necessarily sent to datacite. for example, the email addresses for each author are collected for institutional purposes only and are not sent to datacite. the illinois data bank sends the following information back to datacite:[2] author information, including the orcid, if applicable corresponding author title item identifier funding agency name, including the open funder registry identifier, if applicable publisher publication year keywords date available to the public in the illinois data bank version number rights information, including creative commons identifier, if applicable dataset description resource type (always dataset) related item information, including identifiers, relationship type, and the type of related resource, if applicable techniques such as drop down menus, auto-generated metadata, and orcid integration enable the collection of consistent, reliable, and robust metadata that adheres to best practices as well as laying the foundation for future work. the metadata team strove to keep the amount of free-text metadata entry to a minimum in an effort to simplify the process for depositors and mitigate errors. for example, the author list features an optional orcid id lookup search for each author name. if an author has an orcid, the lookup will auto-populate their name and orcid number once selected in the tool’s search results, thus limiting the possibility of typos. the illinois data bank requires a corresponding author to be designated for each dataset, which is indicated by selecting a radio button next to the author’s name in the deposit interface. other fields use drop down menus to collect information, including: license, publication delay, funding body name, and related resource type. the menu options for both the license and funder fields include terms from existing controlled vocabularies. a depositor can opt to publish their dataset under cc0 or cc-by licenses; selecting either of these options will populate the metadata using the appropriate creative commons identifier. a depositor may also choose the ‘other license’ option. however, the depositor must upload the license language as a .txt file, which is enforced by the system in order to complete a deposit. the ‘funder’ field is similar in that the menu includes a pre-populated list of options and an ‘other’ free-text option. the pre-populated list consists of the top ten most common grant funding agencies at the urbana-champaign campus. the terms from this list are drawn from the open funder registry taxonomy hosted by crossref (crossref 2016), and the metadata includes respective dois from the same registry. the implementation of related identifiers and our discontent at implementation has been described in a forthcoming publication (dunham et al. 2016) so the discussion here will be brief. of the twenty-five possible relationtype terms in datacite 3.1, we chose to use only six. this decision was based on analysis of current usage by other datacite data centers, close reading of the documentation and other supporting materials, and personal communication with members of the datacite community. due to the complexity of defining relationships between items, depositors indicate that a related item exists by selecting the type of related resource from a short list, the values of which are: article, code, dataset, presentation, thesis, or other. the choice of ‘other’ generates a free-text box for users to input their own resource type. when a depositor populates the related materials section, the data curators enhance the dataset record by selecting the appropriate relationtype and checking that the related identifier is entered properly. we decided that being able to link related items reliably is a true value-add capability of the illinois data bank, and therefore we are willing to commit to the curatorial time required. metadata views currently, there are three user-facing ways to view item level metadata in the illinois data bank. the first is the default html dataset landing page. it is also possible to view the dataset metadata as xml, by appending ‘.xml’ to the browser url, which dynamically generates the xml record. this valid datacite xml record contains only the information that is sent to datacite for inclusion in the datacite metadata store. finally, adding ‘.json’ to the browser url produces a json object. in addition to the metadata transmitted to datacite, the json view includes timestamps for any updates made to the descriptive metadata, but does not include the complete changelog information at this time. future metadata work many of the decisions made for the initial iteration of the illinois data bank lay the foundation for future work, including migration to datacite metadata schema 4.0. for example, we had already made the decision to record all author names in ‘name parts’, i.e., two separate boxes for the given and family names a researcher chooses to publish under and transmit a concatenated version to datacite[3]. name parts are not currently supported by datacite 3.1 but will be supported in 4.0 (datacite metadata working group 2016). the metadata team has finalized the current metadata application profile for the illinois data bank (dunham and stein 2016) and is monitoring the datacite dcap, mapping documents, and datacite spar ontology work. mid-term plans include making the illinois data bank metadata available as linked data. this may take shape by adding schema.org semantics to the html dataset landing pages as microdata, as well as by serializing the metadata into triples and making it available via a triple store. this linked data application profile is being designed to align with the university of illinois digital library metadata application profile and will integrate into library-wide rdf efforts. challenging features licensing we researched the current state of data licensing (ball 2014; carroll 2015), and ultimately put a great deal of thought and discussion time into developing the license section of the illinois data bank interface. complex questions threaded throughout our considerations of licensing datasets, such as whether data can be copyrighted, who owns data created by researchers at the university, and what role an institutional data repository should play in promoting open licensing and legal interoperability of research data. our research and discussion resulted in determining the following guiding principles for thinking about licensing in the illinois data bank: public datasets are more efficiently shared and reused when the rights of the accessor are clearly communicated. public datasets are more efficiently shared and reused when any rights associated with them are waived, by using, for example, a cc0 public domain dedication. attribution (i.e., dataset citation) would ideally be compulsory in scholarly communications and should therefore not need to be a legal requirement associated with a dataset. attribution is really, really important to scholars (kratz and strasser 2015) dataset creators, as rights-holders for any rights associated with their datasets, should be empowered to manage the rights associated with their datasets in whatever way they desire. the role of the illinois data bank is to enable data sharing, not dictate the ways in which datasets must be shared. dataset depositors, as potential data accessors and reusers themselves, must comply with licensing already placed on source data. many would-be depositors do not gain familiarity with issues related to copyright and licensing throughout their academic training. with a foundation in these principles, we came to the following decisions about the illinois data bank’s handling of licensing datasets: to support the clear communication of accessor rights, license field is required for datasets deposited in the illinois data bank. to balance our desire to promote barrier-free openness of public data and our commitment to respecting the intellectual property rights of dataset creators, the dropdown selections represent our preferred licenses and support the right of the depositor to assign his or her own terms on behalf of all dataset creators. the depositor is presented with a dropdown menu with three options: “cc0 1.0 universal public domain dedication (cc0 1.0),” “creative commons attribution 4.0 international (cc by 4.0),” and “other license (license.txt must be uploaded as part of dataset).” to support depositors throughout the self-deposit process, help content about licensing research data is provided and seeks to strike a balance between being informative and digestible for those new to issues associated with copyright and licensing (see figure 2 and https://databank.illinois.edu/help#license). figure 2. the licensing tooltip dialog box provides explanations of licensing options in the illinois data bank. click image for larger view. research data licensing is one of many areas that we will be consistently monitoring as norms and guidelines emerge across disciplines. additionally, we expect to adjust help content and data management training efforts in response to questions and feedback we receive from depositors. delayed release, a.k.a. embargoes the impetus of the illinois data bank, in response to changing expectations from funders and publishers, was to develop an open and fully public repository, and we had no interest in supporting data on an inaccessible dark server. not only did the idea of “dark data” seem misaligned with an increasingly open world, adding the ability to deposit a dataset not intended for immediate release would complicate workflows both for the depositor and the service. considering our very intentional effort to keep the system light and intuitive, adding this complexity ran counter to our objective to keep the illinois data bank simple to use and operate. as we grudgingly considered the embargo feature, a stream of immediate decisions to make quickly came to mind. how long could an embargo last? will depositors need to embargo the whole dataset, including the metadata, or just the data files? could datasets accidentally published prematurely be backed out into an embargoed state? are we going to police whether an embargo is warranted? is “embargo” even the right word? despite our desire to avoid the issue entirely, in the end adding embargo functionality within the illinois data bank was needed to support the following scenarios: timing of article publication vs. data publication: we wanted to support the ability to include a dataset doi in an eminent article publication, but understood that either the researcher or the publisher may not want the data to be publicly available prior to the article. [4] other externally determined timing issues: e.g., as an implication of a thesis embargo, external data provider contractual agreements, recalcitrant collaborator, etc. or, the worst scenario of all: a situation we didn’t anticipate and therefore couldn’t support. initially, we hoped we would be able to offer selection of a single date that would represent the point when the files would become public. this date would be permanent at the time of deposit, and an option to embargo the metadata would not be offered. from an interface standpoint, this was a simple and clean method of offering support. however, we knew that ideals offered both file-only and metadata and file embargos, and that both options were used. [5] furthermore, as we continued to attempt to define the specifications for the embargo function, it became obvious that we didn’t understand what our depositors needed, expected, or would even recognize, nor were there any external authorities or governing bodies that we could turn to for best practices for data specifically (in contrast to data licensing, which has been the focus of several high-level initiatives and working groups). therefore, we elected to carry out a (perhaps overly) short optimal workshop survey of the embargo interface design and associated wording. the response rate was small, with 7 respondents from an extremely convenient convenience sample. [6] all responded either positively (very likely – 3; somewhat likely – 2) or ambivalently (unsure – 2) that they would need to embargo deposited data, which verified the need for the functionality to begin with. respondents additionally reported that they would not be certain of the final release date for the embargo at the time of deposit. both file-only embargoes and metadata embargos (along with the files) were desired. respondents were also almost evenly split between use of the word ‘embargo’ (3 respondents) versus ‘release date’ (3 respondents) where we defined the word/phrase as indicating ‘a situation in which data is not publicly available until a certain time.’ these survey results, even though small and highly informal, revealed a fundamental issue with data publication at this time: researchers are unsure what to expect. indeed, we realized the following complications were at play: depositors are equally likely to relate to “embargo” and “delayed release” depositors may legitimately not know the exact data release date they need depositors may need to delay the release of just the files or both the files and the metadata with this feedback in mind, we endeavored to create a system that would account for those complications. because respondents were split between ‘embargo’ and ‘release date’ as the clearest wording, we chose a somewhat inelegant but straightforward approach: use both labels interchangeably (see figure 3). figure 3. embargo options available to data depositors in the illinois data bank. to combat the issue of an indeterminate embargo date, we allow depositors to adjust the release date of embargoed datasets, and they are allowed to change the embargo state to a more public state at any time (vide infra). we established a maximum embargo period of one year, as this was the most common time frame indicated in the agency public access plans. however, curators have the authority to select any date, so that internally we have the ability to accommodate edge cases should they arise. once we decided to include embargo functionality, and with an understanding of the dangers associated with inadvertent publication, keeping depositors well informed of what was happening with the release of their dataset was vital. similar to the licensing tooltip dialog box above, a table with core information was placed in a “what’s this?” help link. this allowed us to include more detailed help information while keeping the deposit interface clean (figure 4). the information provided in this area focuses strictly on the implications of embargo states from the perspective of the depositor and what may or may not be visible. as another way to mitigate any potential misunderstanding of embargo states, we also made sure that messaging within the deposit workflow was tailored to the specific embargo state the depositor selected (figure 5). figure 4. the embargoing tooltip dialog box provides explanations of embargo options and ramifications. figure 5. confirmation prompts for the final step of publication are tailored to specific embargo scenarios. panel a shows messaging if both the data files and the metadata record are embargoed. panel b shows messaging if only the data files are embargoed. panel c shows messaging if no embargos are selected. while we were able to sort out the user interface such that our usability testers were able to navigate selecting an embargo with relative ease, the complications did not end there. supporting detailed embargo options hit a classic stumbling point in design where the desired behavior seemed easy to explain while the technical implications were nearly the opposite. this was due to the nature of datacite doi states and the mixed nature of enabling embargo of just the files or of the files and the metadata. while the former renders a “published” doi, the latter results in a “reserved” doi. this is where a doi is issued to the depositor for use within a manuscript but without the metadata being made public. the internal names for each of these states were transient in our conversations and changed right up until soft-launch of the illinois data bank. we each differed in our personal preferences for these names and used things like invisible, partially visible, fully invisible, standard embargo, invisible embargo, public, etc. linguistic drift was evident among all team members at different stages until the meanings became so muddled that development conversations were internally incomprehensible. this was the point when a specific meeting was called to come up with the final wording. rather than attempting to assign code names or other single terms, team members opted to start with the most verbose options and trimmed them until the final labels remained. the final states are below: draft: when a record is saved within the illinois data bank, but nothing has been communicated to datacite. in this state: depositor may change the release date within the one year maximum and may also change the embargo state depositor may delete the dataset altogether at will file and metadata embargo: when the dataset has been formally deposited into the illinois data bank, a doi is reserved, but the illinois data bank dataset landing page and the doi redirect fail. the depositor is provided a doi suitable for inclusion in a publication, but they are told that link should be expected to fail. in this state: depositor may change the release date within the one year maximum or change the embargo state to either “no publication delay” or “file only publication delay” depositor may not delete or “unreserve” the dataset without contacting the rds file embargo: when the dataset has been formally deposited into the illinois data bank, and a datacite doi has been minted. while the landing page and the doi work, the list of files is replaced with a message stating that the files are under embargo until the specified release date. in this state: depositor may change the release date within the one year maximum or change the embargo state to “no publication delay” depositor may not delete dataset nor can they “unpublish” the metadata without contacting the rds metadata and files published: when the dataset is deposited and both the files and metadata are publicly available. this is the state where no embargos are in place. in this state: depositor cannot retroactively impose an embargo without contacting the rds depositor may not delete dataset nor can they “un-publish” either the metadata or the files corrections and versioning one rule that appears to remain universal regardless of research domain or perspective is that the word “final” almost never actually means that something will never change again. datasets are not immune to the need to make changes or clarifications. instead of viewing file changes to published deposits as anathema and attempting to avoid the conversation entirely, we took the stance that changes, updates, and additions need to be transparent and sustainable to support. indeed, as a stopgap method of hosting data until the illinois data bank was live, the library ingested small datasets into ideals. one of these datasets underwent two changes during the illinois data bank development, giving the team experience with the very real consequences of not building in support for these tasks. therefore, we understood that we would need to balance the need to enable corrections and the need to keep the integrity of the scholarly record. while support for versioning was marked as essential, a minimal curator-mediated workflow was adopted for the near term. an elegant, depositor-driven approach was something we decided could be tabled until we had better knowledge of the range of correction and versioning needs that would arise after the initial release. as we considered how to implement versioning, we attempted to align our curator-mediated practices with that of the rest of the repository community. a cursory review of existing repositories was conducted during december 2015 to determine archetypical methods of supporting dataset versioning. [7] the 70 top doi-issuing datacite data centers (by volume) were reviewed and informally coded on how they handled versions. only a few of these repositories seemed to represent dataset versioning on doi landing pages. we summarize a subset of the repositories in table 4 below. in short, we found that most repositories appeared to handle versioning slightly differently, but we noticed two overarching approaches emerged: one which focused on the representation of the version on the dataset landing page and one which focused representation of the version within the dataset’s metadata. both web and metadata aspects were important for the illinois data bank, but we believe if we focused on metadata first, we would be more readily able to improve the web interface in the longer term. table 4. a subset of datacite doi-minting repositories reviewed for versioning practices. repository version type? new doi for major? old versions available? version to datacite? uses version relation type? change log? cdl.digsci (figshare) major no yes no no no cdl.lternet major yes yes no no no cern.zenodo none; manual new entry yes; new entry yes no supports manual gesis.icpsr major yes (visible in doi) yes mostly no yes gesis.ubhd major on landing, major.minor to datacite no yes initial, but no updates no yes bl.oxdb major no dataset landing page only no no limited cdl.peerj (document repository) major yes, visible in doi yes yes no no bl.york – prospero (document repository) by date on dataset landing page, major in datacite no yes initial, but no updates no yes cdl.datavers major.minor no yes no no yes tib.tub major yes yes no no yes gesis.archiv major yes yes yes some yes ultimately, we decided that there were two distinct types of changes that we wanted the illinois data bank to handle: changes to files associated with a deposit and changes to metadata values once deposited. we were only concerned with tracking changes to the published versions of datasets and files. no changes to drafts are stored, and depositors are free to change everything up until the point of publication. in this context, publishing is considered to be the act of receiving a reserved or published datacite doi. changes to the files, including file level deletions and additions, are not allowed as soon as datacite receives information about the dataset. after attempting to combine what we knew of best and actual practices from the repository community, the team determined several operating principles for our initial launch: any changes to files within a published dataset require creation of a new version. new versions will receive their own dataset landing page. new versions will receive a new doi, and the version number will be present in the doi suffix. version numbers will start at 1 by default and be represented as whole numbers only with no additional granularity. [8] versions will be tracked in the metadata using the relatedidentifer element and transmitted to datacite. version numbers will be reported and easily discoverable on dataset landing pages. changes to the descriptive metadata within the illinois data bank do not result in creation of a new version. changes to the descriptive metadata will be tracked and reported on the dataset’s landing page within a publicly available changelog. the descriptive metadata record, while we consider it to be a part of a dataset, has the potential for many and ongoing changes, e.g., addition of keywords or related material as the dataset is discovered and reused. tracking changes to metadata values is straightforward for the system, so we determined that simply making the changes human readable and transparent was sufficient. operationalized, this meant adding a changelog table to the bottom of the dataset page to report any post-publication metadata changes. each copy of the updated metadata is also saved inside the system folder for each dataset, which can be made available upon request. because we store snapshots of the metadata record, a disaster recovery procedure can recreate the entire dataset package (metadata records and files) from information contained within the associated medusa folders. versioning of datasets for any needed file change has created more debate within our team. as a hard and fast rule, it is simple to implement, ensures absolute consistency in how file changes are handled, and provides a high-level of transparency to accessors. however, it does require that relatively minor changes, such as updating a readme file with the dataset’s official citation with the doi or correcting a url, escalates to creating an entirely new version of the dataset, which may not be ideal. curator functions within the illinois data bank interface, a curator-only view was developed prior to launch. currently, curator access is determined by a hard-coded list of identifiers in the application configuration files. within the curator-only view, curators are allowed to edit any metadata fields for a given record and are also allowed the ability to override the one year embargo limit. special curator-only fields are also displayed, which allow the ability to add a version number other than “1”, import a doi if the record was created elsewhere, as well as add and assign related identifiers. curator access also allows for the ability to toggle between a “depositor view” and a “curator view” to assist in demos and bug testing. perhaps most importantly, curators are able to control the visibility of dataset changelogs, metadata, and files such that access can be temporarily or permanently suppressed. based on policy discussions described above, we considered this feature to be important to have in place prior to launch. the interface for the suppression area provides verbose and explicit helper text to assist in (hopefully extremely) rare, “high-pressure” scenarios, when curators may need to make decisions quickly and confidently (figure 6). figure 6. suppression options are available only to illinois data bank curators. anticipating rare use, descriptive help text is provided to assist curators. click image for larger view. external to the illinois data bank, we have developed a series of guidelines and processes to curate deposited datasets, but these processes are not automated. we expect manual workflows will work sufficiently well enough during the initial rollout of the illinois data bank when a smaller number of datasets are submitted and curator workloads are still manageable. however, if deposit into the illinois data bank becomes routine for illinois researchers, we would be quickly overwhelmed considering our campus’ publication output exceeds 6,000 works a year (personal communication, w. mischo 2015). in that case, additional development will be required to improve the curation workflow. reception, known limitations, and future directions we launched the illinois data bank in may of 2016 and spent the following summer engaging with researchers to build awareness and initiate deposits. we also fixed inevitable bugs and attended to more urgent value-adds that emerged, such as the need for elegant reversion to read-only secondary copies when our primary storage goes down for maintenance. overall, reception to the platform has been positive. an interesting secondary use of the illinois data bank that we did not anticipate is that it serves as an educational tool. a faculty member and a technology transfer manager independently shared that they thought the illinois data bank has great utility to educate researchers about policy issues such as data ownership and data re-use. despite good feedback and having what we consider to be a relatively smoothly running and robust product, we know that there are several areas that could use additional refinement or implementation. we maintain a “parking lot” of features or improvements that have either occurred to us or we have already heard from depositors. some of these are alluded to above (e.g., metadata and versioning improvements), but a few others include better support for large data files [9] and folders; developing tools to query, subset, or explore data in place; and more flexible access control mechanisms to enable transfer of ownership or multiple dataset editors. our development process reinforced the value of evidence-informed decision-making and considering the ripples our work may have into the external systems we rely on. we would have been able to develop the illinois data bank more swiftly if we had not considered and (attempted to) address the potential consequences of how illinois data bank metadata would be represented in datacite—and to subsequent data indexers. given the grander mission to make the research data products discoverable, understanding the value of aligning our design choices to maximize discoverability has benefitted our end product and provided us with important examples of why the illinois data bank is valuable to our campus community. we understand our privilege in having the equivalent of several full time staff members dedicated to this development process, so we hope that by openly presenting the struggles and rationales behind some of our decisions, those facing data repository development projects will be able to see in our choices strategies to adopt, modify, or reject. we decided to share our experience in developing the illinois data bank in the spirit of the twitter hashtag #overlyhonestmethods, which documented researchers’ frank (and frequently funny and/or sad) admissions on the surprising hows and whys behind research. although we did this somewhat tongue-in-cheek, the truth is never far away, and transparency is an ideal we all should aspire to. if we were to be truly honest, we would need substantially more text, but we are already teetering on 10,000 words and can hardly expect readers to indulge us farther. [10] in closing, we’ll simply ask the question: “have we developed a resource that will serve our campus, researchers, and research communities well?” perhaps twitter user @luispedrocoelho answers best with “we have no idea, but this seems reasonable #overlyhonestmethods”. [11] acknowledgements we thank medusa developer howard ding for many helpful discussions and development of medusa components, and helenmary sheridan for discussion and sharing of information on ideals. notes [1] although internally we immediately lapsed into calling it the “idb.” [2] for example, see http://data.datacite.org/10.13012/b2idb-4900670_v1 [3] sometimes this changes, which is why we encourage the use of personal identifier schemes like orcid. [4] and data release upon publication is commonly mentioned in funding agency public access plans, including nih, dot, noaa, and nasa, etc. [5] although exactly to what extent was not something we knew at the time. [6] meaning we emailed a few of the rds-friendly researchers we knew and asked for their input and willingness to send along to others. therefore, we have no idea what our sample size actually was, but we did have representation of people with backgrounds in informatics, lis, gis, engineering, life sciences, and physical sciences. nearly all (6 of 7) had published a journal article before. [7] we note that in the time since figshare has changed its versioning practices, but we report here what we observed at the time of comparison with other repositories. [8] notably missing from this discussion is handling the version numbering system. an analysis of version numbers for datacite datasets indicated a lack of community consensus of the more advanced and detailed numbering systems (wickes 2016). [9] especially considering we’ve committed to 2tb per year per principal investigator (pi). [10] so feel free to contact us. [11] https://twitter.com/luispedrocoelho/status/288677624279093249 references ammann, noémie, lars holm nielsen, sebastian peters, and madeleine de smaele. 2011. “datacite metadata schema for the publication and citation of research data.” datacite. http://web.archive.org/web/20160825024134/http://schema.datacite.org/meta/kernel-3/doc/datacite-metadatakernel_v3.1.pdf. ball, alex. 2014. “how to license research data.” edinburgh: digital curation centre. http://web.archive.org/web/20160825025111/http://www.dcc.ac.uk/resources/how-guides/license-research-data. borowski, christine. 2011. “enough is enough.” the journal of experimental medicine 208: 1337–1337. doi:10.1084/jem.20111061. carpenter, todd. 2009. “supplementary materials: a pandora’s box of issues needing best practices.” against the grain 21: 84–85. carroll, michael w. 2015. “sharing research data and intellectual property law: a primer.” plos biol 13 (8): e1002235. doi:10.1371/journal.pbio.1002235. crossref. 2016. “open funder registry.” http://web.archive.org/web/20160825033231/http://www.crossref.org/fundingdata/registry.html. datacite. 2016. “metadata store.” http://web.archive.org/web/20160825020124/https://mds.datacite.org/. datacite metadata working group. 2016. “datacite metadata schema for the publication and citation of research data v4.0.” https://schema.labs.datacite.org/meta/kernel-4.0/doc/datacite-metadatakernel_v4.0.pdf. data seal of approval. 2016. “the guidelines 2014-2015.” data seal of approval. accessed august 25. https://web.archive.org/web/20160825023007/http://www.datasealofapproval.org/en/information/guidelines/. dunham, elise, and ayla stein. 2016. “illinois data bank metadata documentation v1.0.” university of illinois at urbana-champaign. https://www.ideals.illinois.edu/handle/2142/91020. dunham, elise, elizabeth wickes, ayla stein, colleen fallaw, and heidi imker. 2016. “pre-metadata counseling: putting the datacite relationtype attribute into action.” in curating research data, edited by lisa r. johnston. acrl. fallaw, colleen. 2016. databank: launch. zenodo. 10.5281/zenodo.51311. holdren, john p. 2013. “increasing access to the results of federally funded scientific research.” office of science and technology policy. http://web.archive.org/web/20160115125401/https://www.whitehouse.gov/sites/default/files/microsites/ostp/ostp_public_access_memo_2013.pdf. imker, heidi j. 2016. “overlooked and overrated data sharing: why so many scientists are confused and/or dismissive” in curating research data, edited by lisa r. johnston. acrl. kenyon, jeremy, and nancy r. sprague. 2014. “trends in the use of supplementary materials in environmental science journals.” issues in science and technology librarianship. doi:doi:10.5062/f40z717z. kratz, john ernest, and carly strasser. 2015. “researcher perspectives on publication and peer review of data.” plos one 10 (2): e0117619. doi:10.1371/journal.pone.0117619. maunsell, john. 2010. “announcement regarding supplemental material.” the journal of neuroscience 30: 10599–600. nielsen, jakob. 2000. “why you only need to test with 5 users.” nielsen norman group. march 19. https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/. purdue university. 2016. “learn about ezid.” http://web.archive.org/web/20160825020403/http://ezid.lib.purdue.edu/learn/. *rimkus, kyle. 2015. “medusa faq.” wiki. medusa digital preservation service. http://web.archive.org/save/https://wiki.illinois.edu//wiki/display/librarydigitalpreservation/medusa+faq. rimkus, kyle, and scott witmer. 2016. “managing file format endangerment in digital preservation repositories: an evidence-based approach.” in proceedings of the 13th international conference on digital preservation. bern, switzerland. scholarly publishing and academic resources coalition (sparc). 2016. “tracking and understanding federal open data plans.” sparcopen.org. http://sparcopen.org/our-work/research-data-sharing-policy-initiative/. shreeves, sarah. 2014. “policy documents – ideals.” university of illinois at urbana-champaign. https://www.ideals.illinois.edu/handle/2142/5. us department of energy. 2014. “statement on digital data management.” department of energy. http://science.energy.gov/funding-opportunities/digital-data-management/#requirements. wickes, elizabeth. 2016. “version values for datacite dataset records.” university of illinois at urbana-champaign. https://doi.org/10.13012/b2idb-4803136_v1. williams, sarah c. 2016. “practices, policies, and persistence: a study of supplementary materials in crop science journals.” journal of agricultural & food information 17 (1): 11–22. doi:10.1080/10496505.2015.1120213. about the authors colleen fallaw research programmer at the university of illinois at urbana-champaign library. http://orcid.org/0000-0002-0339-9809 mfall3@illinois.edu elise dunham data curation specialist with the research data service at the university of illinois at urbana-champaign library. http://orcid.org/0000-0002-7562-1923 emdunham@illinois.edu elizabeth wickes data curation specialist with the research data service at the university of illinois at urbana-champaign library. http://orcid.org/0000-0003-0487-4437 wickes1@illinois.edu dena strong senior information design specialist with technology services university of illinois at urbana-champaign. http://orcid.org/0000-0003-1834-9940 dlstrong@illinois.edu ayla stein metadata librarian at the university of illinois at urbana-champaign library. http://orcid.org/0000-0002-6829-221x astein@illinois.edu qian zhang clir postdoctoral researcher with the ischool and research data service at the university of illinois at urbana-champaign library. http://orcid.org/0000-0003-1549-7358 zqian1@illinois.edu kyle rimkus preservation librarian and assistant professor at the university of illinois at urbana-champaign library. http://orcid.org/0000-0002-9142-6677 rimkus@illinois.edu bill ingram manager of scholarly communication and repository services at the university of illinois at urbana-champaign library. http://orcid.org/0000-0002-8307-8844 wingram2@illinois.edu heidi imker director of the research data service (rds) and associate professor at the university of illinois at urbana-champaign. http://orcid.org/0000-0003-4748-7453 imker@illinois.edu subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – map it @ wsu: development of a library mapping system for large academic libraries mission editorial committee process and structure code4lib issue 10, 2010-06-22 map it @ wsu: development of a library mapping system for large academic libraries the wayne state library system launched its library mapping application in february 2010, designed to help locate materials in the five wsu libraries. the system works within the catalog to show the location of materials, as well as provides a web form for use at the reference desk. developed using php and mysql, it requires only minimal effort to update using a unique call number overlay mechanism. in addition to mapping shelved materials, the system provides information for any of the over three hundred collections held by the wsu libraries. patrons can do more than just locate a book on a shelf: they can learn where to locate reserve items, how to access closed collections, or get driving maps to extension center libraries. the article includes a discussion of the technology reviewed and chosen during development, an overview of the system architecture, and lessons learned during development. by paul gallagher introduction the wayne state library system (wsuls) has three main facilities, two special libraries (law and medical), an extension center library, and over 3.6 million items, creating a formidable challenge to locate materials. some collections are hidden from public view, and a large subset is available only on request. in august 2009, development began on a mapping system to help patrons locate materials from the library catalog, and in february 2010, the system became available to the public. the mapping application shows the locations of shelved materials and provides information for over three-hundred library collections; providing a wealth of information on how to use the wsu libraries. this article will be of interest to library software programming and systems staff interested in developing their own mapping application, and provides an overview of the architecture as well as the technologies reviewed during the research phase. additionally, some of the lessons learned are discussed. relevant examples numerous institutions have integrated stack maps into their catalogs, and the methods employed are as unique as the libraries themselves. some examples of catalog mapping systems include florida state university [1], and a koha based implementation by near east university [2], which contains animated arrows to point patrons to the right location. in 2007, wichita state published a paper providing details on how their system handles complex shelving situations using a custom data processing model [3]. stackmap, llc, a commercial entity that “provides comprehensive mapping software for libraries,” has links on their website to examples at stanford university and boston college [4], and canterbury christ church university has a document online describing the system used in their catalog [5]. in each case the approach is typically the same: use bibliographic information from the catalog to discern which map to display, and provide a method to view the map. general methods available for library mapping applications there are several options to create maps of library holdings, from using simple images to implementing complex systems utilizing gps, each having its own development pros and cons. while no single technique would work for all libraries, an overview of the technology surveyed will be useful for others considering a similar project. the simplest solution is to create a set of images for each call number range, then use catalog data to evaluate an item’s location and provide the correct image. this method is relatively easy to develop: the maps could be created with illustration software and updated by uploading new files to a web server. one additional benefit is the potential to use more sophisticated content, such as animated gif’s to display directional arrows or html to present additional information. however, this method can be difficult to sustain when hundreds of maps need updating as collections shift. a more complicated solution is to create maps using vector coordinates and employ programmatic methods to display material locations. in this model, coordinate data is held in a data source and program code converts the coordinates to a web graphic. however, creating vector maps is almost impossible without graphics software, as creating precise x/y coordinates would be difficult without a graphical means to create the shapes needed to define a map. while building maps completely from coordinates may have some benefits, it would duplicate readily available technologies, and overlap commercial software such as adobe illustrator and arcgis. converting hundreds of coordinates to web graphics on demand could be challenging to program, and could have processing requirements undesirable in a web environment. a similar approach using a hybrid method involves creating maps with readily available software, then programmatically adding the location of a requested item. for example, scalable vector graphics (svg) or a similar format could be used to implement this method by using illustration software to create the base map, and then modifying the svg xml to indicate the location of an item. however, svg is not a completely adopted format, and some browsers, most notably internet explorer, require the installation of a plug-in to view the file. while additional programming could convert the svg files into a more common format, it would entail additional image processing and add a level of complexity to the project. this approach also involves the creation of maps in vector format, a potentially large project for an institution with multiple facilities if such maps are not readily available. a more challenging solution involves the use of global positioning system (gps) coordinates to provide maps similar to those available with google earth. using the google earth api, maps could be overlaid on top of actual locations, and items displayed using coordinates derived from gps. a benefit to this approach is the possibility of leveraging mobile gps, and “pointing” patrons to the item using a navigational style display. however, the development of such a system would once again be a significant project, and the accuracy required to capture item locations would have to be extremely good – or the required shelf may end up on the wrong side of the library. during some initial testing at wsuls, capturing a gps signal indoors was unreliable, and the typical degree of accuracy (30-50 feet) was not sufficient to map item locations. another option, the one that wsuls selected, is a hybrid of the static and vector map designs. with this model, the maps are developed as “templates” which contain the basic floor layout, including the stacks and other navigational elements such as restrooms and staircases. the material locations are held in a data source, and the maps are dynamically created with a combination of the static template and the indicator areas using coordinate data. this provides a sharp reduction in the number of static images that require updates when collections shift. in the example shown in figure 1, this library floor would require eleven maps to indicate each shelving range, but the hybrid system requires only one background map – with specific areas highlighted using coordinate data. figure 1. template and dynamically created map. while this approach entails creating an interface to capture call number ranges and coordinate information for highlight areas, this was found to be a relatively easy proposition. this method provides a best of breed approach, using the simplicity of a static image for the background, and using coordinates to create the dozens of variants that specify specific call number ranges. in addition, this approach allows for the reuse of existing library floor maps, and not having to create entirely new maps exclusively for the system (as would be the case with vector maps). accessing the system there are two ways to access the system: either via the two wsuls catalogs (millennium “classic” and encore), or a form designed for use at the reference desk. in this sense, the application can be accessed in two tiers, the catalog tier, and the form tier. an example of both methods is shown in figures 2 and 3. figure 2. example of the reference form method of access. figure 3. example of the catalog method of access. reference form the reference form provides a simple method to find the location of a call number in any of the libraries, primarily at the reference desk. users simply select a library, enter a call number, and a map of the call number range is dynamically created and displayed to the user. an overview of the steps is shown in figure 4. figure 4. use of the reference form to access the mapping system. the reference form and the catalog link use the same code to produce the maps, but the form uses less processing as it does not harvest bibliographic information. however, this method has its shortcomings, as it only works with shelved materials that have an assigned call number, and is not aware of an item’s availability or if it is being mapped at the correct library. the catalog-based method can map a much larger variety of materials, including items without call numbers or in closed collections, and displays an item in the correct library, as opposed to only mapping a call number to a building. an additional feature allows the reference form to display maps based on library of congress (lc) subject terms. the functionality works the same as the call number form; select a library, select a subject, and the map is displayed (shown in figure 5). the dropdown menu used for the subject terms is a jquery plug-in that supports both pagination and searching, and was implemented to make browsing through the terms easier [6]. both reference forms are available on the wsuls website at: http://www.lib.wayne.edu/maps figure 5. reference form showing the map by subject feature. catalog mapping the mapping system is primarily used from the catalog, and is accessed by clicking the explore icon on the search results page, and then the “floor maps” icon. the system will then harvest the catalog data, and display the map or content. an overview of the process is shown in figure 6. figure 6. steps to use the mapping system from the catalog. this method of access has additional capabilities compared to the reference form, notably the ability to display items other than just shelved items with call numbers. the system checks the catalog’s location code, and if it is a shelved collection, displays the map; otherwise it displays static content that describes how to access that collection. the static content is html, and can be anything that describes a collection, including images, a redirect to another website, and embeddable content such as google maps. in figure 7, two examples are shown of non-shelved materials – one displaying a driving map to an extension center, the other a photograph of a hard to locate collection. figure 7. examples of information provided for non-shelved collections. in the left image, a driving map is provided using the google map api, and in the right, an image is used to show a hard to find collection. both methods use html to provide the content. architecture the system was designed using a tiered access model, as both methods use the same display code, but the catalog requires additional data gathering. figure 8 represents the overall architecture, with the top area representing access from the catalog, and the bottom the reference form and content display functionality. briefly, the catalog link forwards the oclc number to the index.php page, which queries the catalog and, if necessary, asks which item to map with the map_select.php page. once the library and call number data is obtained, the system forwards the call number and library to the map_display.php page, where the location is checked for which type of content to display, and finally the map or content is displayed. in the case of the reference form, the form forwards the library and call number to the map_display.php page directly, and the system displays the map. figure 8. system architecture. mapping from the catalog when using the link from the catalog, the process makes use of the millennium webbridge feature to add a link into the catalog containing the url shown in the first example in figure 9. while the exact steps are beyond the scope of this article, this involves setting webbridge to display a link with information, notably the oclc number, from the bibliographic record. figure 9. url processing steps used to harvest data from the millennium ils. next, the oclc number in the webbridge link is used to query the millennium xml server, returning the internal record number and conducting a final request, a “screen scrape” of the item detail page. this final step unlocks any of the data available on the millennium item detail page, including the number of copies, locations, and call numbers. while it would have been desirable to gather the data in single step (and not have to depend on screen scraping techniques), the millennium system presented some challenges to accessing its holdings information. the xml server does not reliably return the number of copies and locations, and conducting the screen scrape using the oclc number occasionally produces invalid results. this method – while somewhat complex – does consistently provide the data needed to map the item. there is one additional check for the number of copies, and if there are multiple copies, the map_select.php page prompts the user as to which item to map. this is a necessary step, as multiple copies of the same item may be located in different locations. in the case of a single item, the system forwards the request to the map_display.php page, which displays the map or content about that collection. outputting content / reference form once the library and call number have been obtained through the catalog or reference form, the data is forwarded to the map_display.php page for display. if the reference form is used, it simply submits the data directly from the form, and the content is prepared the same as if the request came from the catalog method of access. the collection locations and call number ranges are held in two tables in a mysql database. the tables employ a hierarchal relationship, the first containing the collection locations and the second table containing a list of call number ranges and coordinates. each location in the location table contains a field that specifies whether to use the overlay table and retrieve the coordinates to display a map, or just to display its content about that collection. an example of the structure is shown in figure 10. figure 10. relationship of two mysql tables used to hold location information and call number overlay coordinates. the system evaluates the location code in the location table, and has one of three options – query the overlay table and produce a map, redirect to another code, or display content. the output is determined by the docallnumberlookup and maptoanothercode fields in the location table that specify what content to display when queried by the system. an example query and response is shown below: select * from locationtable where locationcode = "pbstk" locationcode locationname docallnumberlookup maptoanothercode content pbstk purdy-kresge true null null the last three fields specify the next step, as either the docallnumberlookup, maptoanothercode, or content field will have data; some simple program logic determines which field is not null, and reacts accordingly. in the above example, docallnumberlookup would trigger a query of the overlay table, and get the x, y coordinates used for the overlay map. the specifics of each type of content are detailed below: 1.) static content (content) in this case, html is displayed to provide information about the location. this is used to provide details about locations that are not mappable, such as closed collections. other examples include directions to extension centers, links to other web resources, or redirect to an item request form. 2.) mapped to another code (maptoanothercode) for this case, the location code is mapped to another code. while at first this seems counterintuitive, it enables the ability to create “catch all” codes, such as the circulation desk. for example, reserved items are held at circulation desks; and reserve location codes are mapped to the circulation desk code, with the system displaying the information about how to find the circulation desk. 3.) call number lookup (docallnumberlookup) the third case is to use the call number system to show the appropriate map for shelved materials. the system conducts an additional query with the call number, and creates the appropriate map for that item. this step entails some additional logic to determine the library, floor, and call number range: 3.1) query the overlay table for the call number range the overlay table is queried to determine the correct call number range and return the coordinates used for the overlay. the query uses the library location code and a normalized version of the item’s call number to retrieve the correct call number range. this returns the information needed to produce the dynamic map, including the template, an x/y coordinate for the indicator, and the width and height to describe the size of the highlighted area. an example of this query is shown along with the returned data: select * from mapscall where locationcode = "pbstk" and callstart <="161900.3558" and callend >="161900.3558" locationcode template callstart callend width height x y pbstk pbstk 161900.015919 161900.35660 238 44 524 131 3.2) merge the template and highlight image the final step is to merge the template and overlay image using the php/gd image library [7]. the basic steps involve using two image files, one of the background map template, and another image containing only the colored area used as the item indicator. the gd imagecopymerge function combines the two image files and creates a composite image, using x, y coordinates for the highlighted area from the overlay table. additional logic during this step checks if the dynamic image has already been created for caching purposes. an example of the function used to merge the files is shown below. the php/gd function definition: //gets or creates the call number map image function displayoverlayimage($maptemplate, $dst_x­­­­, $dst_y, $src_w, $src_h) { // create image instances $img = 'maps/templates/'.$maptemplate.'.gif'; //template $imgoutput = 'maps/' . $maptemplate.$dst_x.$dst_y.$src_w.$src_h . '.gif'; //merged file //check for a cached file if (file_exists($imgoutput)) { return $imgoutput; } else { //create the image memory streams $dest = imagecreatefromgif($img); //get the template image $src = imagecreatefromgif('maps/templates/overlay.gif'); //get the overlay //merge the images imagecopymerge($dest, $src, $dst_x, $dst_y, 0, 0, $src_w, $src_h, 90); // output and free from memory $image = imagegif($dest,$imgoutput); imagedestroy($dest) or die (‘failed imagedestroy(1)’); imagedestroy($src) or die (‘failed imagedestroy(2)’); return $imgoutput; } } updating the system updating the system is completely web based, and as the map display design is based on two database tables holding both collection location and coordinate information, the updating system mirrors this structure. staff can modify the location code content using a grid view format for the location table, or update the call number ranges and indicator areas in the overlay table using a custom-built php form. the grid view tool used to update the location code content is simple in its implementation, and connects to the mysql table without any additional logic. all codes can have static content entered, be mapped to another code, or be assigned to do call number lookups. the grid view (shown in figure 11) used is a gnu/gpl application named php datagrid, available at http://phpbuilder.blogspot.com/. figure 11. data grid used to update location information. to update the call number range and indicators for the overlay table, a custom form was created that allows for the creation and modification of call number ranges. javascript is used to provide a two-click mechanism that overlays a red box over the appropriate area for that call number range, and the resultant x/y coordinates are stored in the database for use with the image merging function (shown in figure 12). users can select a library and floor, add the call numbers, and click on the appropriate area to draw a box over the location to be highlighted. figure 12. call number overlay update form. results and lessons learned initial results have been positive. usage is steadily increasing as the system gains more notice by patrons and staff, and growth will likely continue as other resources are integrated into the system. it is important to note the application’s dependence on staff to keep the ranges and location content updated. by collaborating with the wsuls access services early in the process, the system is now used during large material shifts, and is updated as a normal part of their collection shifting workflow. without having a simple means to manage the content, the data could quickly grow stale, and send patrons to the wrong location. one observation since the initial development has wsuls reevaluating how accurately the system maps call number ranges. the current implementation displays only large call number ranges, which may contain as many as forty individual shelving units. the expectation was that this would minimize updating, but shifting has not been as common as expected. additionally, range updates have to be recorded to produce new signage, creating a situation where the data is entered twice, once to create the new signage for the cases and again for the mapping system. an enhancement currently under development has the mapping update form update the mapping system and generate new signage all in one step, simplifying workflow. as a result, the system will provide more specific locations, and prevent having to enter the updated call number ranges twice. one challenge encountered was the integration with the encore catalog. when development started, the millennium webbridge feature was not yet available for the encore opac, and it was unclear how the mapping link could be added into the catalog interface. with webbridge implemented at wsu, the mapping link appears under the “explore” icon located for each item in the search results. this is not an optimal situation, as users must click on the explore link while viewing the search results, effectively hiding the link from plain view. even more problematic is that webbridge added links do not display on the item detail page, making the mapping link hidden to all but the most curious users. while future versions of encore may remedy this problem, for now visibility is limited. ongoing work to feature the reference form more prominently on the library website and the creation of an instructional video may help mitigate this problem. in addition, the older “millennium classic” catalog does contain the link on the item detail page, so for classic users this has not been an issue. for an example of the wsuls encore catalog, see http://encore.lib.wayne.edu/ overall, the system is a useful addition to the library catalog, and the ability to create maps for patrons is a welcome addition at the reference desk. future enhancements under consideration involve using the system for other applications, such as reserving study spaces. additionally, as mobile gps technology becomes more accessible, the system could provide a platform for experimenting with location awareness – and be able to point to specific item locations. the system is being considered for release as an open source application, based on demand and the work involved in refactoring the code to work as a standalone application. while there is not a timeframe for a possible release, updates about the availability of the application will be posted to the library website at http://www.lib.wayne.edu/maps/updates/. the project is considered a success by both staff and patrons, and while technology alone cannot make libraries more accessible, it can provide useful tools to help keep patrons from getting lost in the stacks. references [1] florida state library homepage. http://www.lib.fsu.edu [2] near east university, grand library catalog. http://library.neu.edu.tr/cgi-bin/koha/opac-main.pl [3] dynamic map display in web opac: an experiment at wichita state university libraries. http://www.white-clouds.com/iclc/cliej/cl24ldd.htm [4] stackmaps, llc. http://stackmap.com/public/ [5] ekins, andy. interactive map documentation. https://docs.google.com/doc?docid=0ayrdyxsoqkgfzgryd2pnmjvfmzfocw1iemhnag&hl=en_gb [6] jquery flexbox. http://flexbox.codeplex.com/php gd image library. http://php.net/manual/en/book.image.php [7] php gd image library. http://php.net/manual/en/book.image.php about the author paul gallagher (paul.gallagher@wayne.edu) is the developer librarian for the wayne state university library system with the digital library initiatives team. paul is a new librarian after recently completing his mlis at wsu’s school of library and information science. he has over ten years experience in the private sector including work in application development, technology instruction, and systems administration. subscribe to comments: for this article | for all articles 3 responses to "map it @ wsu: development of a library mapping system for large academic libraries" please leave a response below: mobile website development: from site to app | asis&t bulletin, 2011-10-08 […] [5] gallagher, p. (june 22, 2010). map it @ wsu: development of a library mapping system for large academic libraries. code{4}lib journal, 10. retrieved august 21, 2011, from http://journal.code4lib.org/articles/3072. […] you are here: rethinking the library map | antelope as document, 2013-07-31 […] would utilize the vector images to create finding paths and circle book locations, similar to the map it @ wsu project, should the library decide that they are interested in utilizing this technology. one of my […] using gis to improve library services | designer librarian, 2014-10-29 […] you do need to create a catalog mapping system for this to work though. check out what the wayne state library system did (to benefit their […] leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – using a raspberry pi as a versatile and inexpensive display device mission editorial committee process and structure code4lib issue 21, 2013-07-15 using a raspberry pi as a versatile and inexpensive display device this article covers the process by which a library took some unused equipment and added a cheap computing device to produce very inexpensive but effective electronic signage. hardware and software issues as well as a step-by-step guide through the process are included. by edward iglesias and arianna schlegel libraries are always short of money but rich in talent. as more recent graduates are not afraid to code–and perhaps even modify existing hardware–possibilities open. such was the case when the burritt library needed a display device and did not want to pay a large amount of money for a commercial solution. we looked at a few of the options available but decided to roll our own solution using a cheap computing device called the raspberry pi. the raspberry pi is a $35 computer running a stripped down version of linux. it is very bare bones and does not come with a keyboard, power supply or display device. fortunately we had all of these things lying around in the form of an old monitor, keyboard and mouse. a 5 volt power supply was scavenged from an old cell phone and in the end all we had to purchase was the pi itself, an hdmi cable for the monitor and an sd card for the operating system. figure 0. pi vs. pen setting up the operating system (os) is straightforward for anyone who has ever burned a linux distribution onto a cd. you will need a card reader. using the built in dd tool it was a simple matter to burn the os onto the card. using the raspberry pi and some custom php on a website, we came up with a solution that cost approximately $50.00 plus time. as a comparison, a typical commercial solution starts at around $900.00 for something like this which is set up at our local coffee house. figure 1. coffeehouse display the problem in the spring of 2012 the burritt library started checking out kindle readers for ebooks. this was only a pilot project with 5 readers but proved wildly successful. as a result students were often asking whether there were any kindles available. for some reason they did not think to check the online catalog first. the circulation department needed a way to display how many kindles were available for checkout. figure 2. initial raspberry pi setup the solution the newly released raspberry pi. this is a small $35 – $40 computer that can be loaded with a special distribution of the linux os (in this case raspian) and can be used as a normal computer or hooked up to a breadboard (a physical board for prototyping electronic projects) for use with physical sensors. since the distribution already came with a browser the idea emerged that by placing the computer hooked up to a monitor in a location at circulation the browser could be set to a web page that auto-updates. additionally, since the computer could be set to automatically log in and launch the browser in kiosk mode at startup there would be no need for a mouse or keyboard. figure 3. kindle counter display the process the project was divided between the two authors with iglesias taking care of the physical hardware and formatting of the pi while schlegel took care of the php scripting that would scrape the catalog to display the item status. raspberry pi customization in order for the display to be as low maintenance as possible, several settings had to be adjusted. the login sequence had to be changed so that no login was required. the boot sequence had to be changed so that upon startup the x environment (the graphical environment in linux that lets you run browsers and other software) was started. the default browser for raspian, midori, had to be invoked on startup and set to go to a particular site and stay there in full screen or kiosk mode. the energy saving sequence that blanked out the screen after a set period of time had to be disabled. at the time we worked on this project it was a long process to set up the pi so that it would start up and log into the graphical interface without a password. recent distributions of raspian have simplified this process. the raspberry pi configurator launches the first time you boot up and can be summoned at any time by opening a terminal window and typing in sudo raspi-config this allows you to simply select an option that boots the pi into the graphical interface under the section “boot_behaviour”. after this a very small script must be written that will invoke the browser and point it at the desired website. just open an editor at the command line and create a file called start.sh in your home directory such as nano /home/pi/start.sh in this file you need only two lines. the first tells the script where to call the command from and the second invokes midori in “fullscreen” mode and goes to the url you define. #!/usr/bin/sh midori -e fullscreen -a www.library.ccsu.edu/kindlecounter & after this make your script executable sudo chmod +x /home/pi/start.sh and add your script to your lxde autostart file sudo nano /etc/xdg/lxsession/lxde/autostart be sure to use sudo before trying to edit this file. at the bottom of the file add the line @sh /home/pi/start.sh finally there was the problem of the screen not blanking out. the solution was found on a discussion on stackexchange which involved the installation of the x11 server utilities. these are a group of tools which let you control among other things when a screen goes blank. to install these tools go to the command line and type in apt-get install x11-xserver-utils then it is simply a matter of editing the autostart file with sudo nano /etc/xdg/lxsession/lxde/autostart next, uncomment the @xscreensaver line in the following code block: @lxpanel --profile lxde @pcmanfm --desktop --profile lxde #@xscreensaver -no-splash @xset s off @xset -dpms @xset s noblank @sh /home/pi/start.sh more details can be found at http://raspberrypi.stackexchange.com/questions/752/how-do-i-prevent-the-screen-from-going-blank. at this point the pi should be able to be unplugged and replugged and automatically launch the browser with the specified target webpage. software the software part of this project is pretty painless and quick. additionally, it can be extrapolated to so many other uses! we are already considering setting up boards to display library announcements, notices, etc. figure 4. the original kindle counter screen for now, however, we have just what we call the “kindle counter.” to see a working example of the page, please visit: http://library.ccsu.edu/kindlecounter. (this soon will be changing to accommodate a split-screen view which incorporates a “laptop counter,” as well, but the basic work is all the same. the original page will be saved and perpetually available at: http://library.ccsu.edu/kindlecounter/justkindle.php.) to begin, we had to figure out how we wanted to read the information from the page. the library’s kindle holdings are listed at http://www.consuls.org/record=b3315628~s16, so we needed some way of scraping that page to read the current status of each item. the page updates immediately upon item processing, so we knew we could get current information from there, without having to figure out some way of getting access to our ils and trying to read the status from the database. (a reminder: do use your catalog’s persistent link functionality for this project!) we opted to use php for screen scraping, following the instructions at http://www.dmxzone.com/go/4402/page-scraping/. we relied heavily on this page as a jumping off point, ensuring we could successfully execute a scrape with the sample code before making the transition to our own site. we needed the code to indicate how many pis were available for checkout, so we decided that the best option would be to sum up the number of times that the two phrases which indicated availability – “available” and “recently returned” – occurred on the page: $numavail = substr_count($file, "available"); $numrr = substr_count($file, "recently returned"); originally, of course, we were simply displaying formatted text. but we wanted something that looked better – so we decided to create images of each number and store them on the web server. this was easy for us, because we currently only have 4 kindles – it might become quite time-intensive if you are looking at dozens or hundreds of items. we originally tried using google fonts, but found that automatic page refreshes caused the page to flicker too much. switching to images which resided on our web server resolved that issue. it was a simple matter of an if/else statement to check the current number of kindles available, then choose which image to display: $maxkindles = 4; if ($totnumavail <= $maxkindles) { print '<img alt="" src="img/kindle" . $totnumavail . ".jpg" />'; } else { print '<img alt="" src="img/kindle0.jpg" />'; } and lastly, because we wanted the kindle counter to be entirely autonomous, we had to make the page refresh itself – and do so regularly, so that we had a real-time display for anyone interested in whether there were any kindles available. we opted to go with this refresh: <head> <!-refresh every 5 seconds --> <meta http-equiv="refresh" content="5" > </head> and chose to set it to refresh every 5 seconds. this was enough that the update seemed almost immediate (we were told our circulation staff enjoyed scanning the kindles in and out to watch the number change when the counter was first set up at our desk!) but wasn’t too flickery (it does occasionally flash, but it doesn’t seem to be a problem). conclusion ths project provided numerous opportunities for the staff working on it as well as the library. by pushing ourselves to learn to make a solution rather than buy one the staff was able to acquire and use new skills that will doubtless prove useful in the future. additionally the cost and customizability would not have been possible otherwise. as we move forward with plans to create a makerspace in the library this has become a first step towards creating a maker culture in the library. finally the screen has proven wildly successful and we have had requests for more of these digital signs for other purposes in the library. now when the old argument comes up over whether it is better to develop a solution in-house or buy one, the answer is not always obviously to outsource. about the authors edward iglesias was born in laredo, texas and lived there much of his life. the environmental bias caused by living in a bilingual, bicultural society has permanently affected his outlook on life. as a result he is drawn to subjects that don’t easily fit description or are a blend of many things. his latest book robots in academic libraries is a good example as the field of library automation and technology is always in flux. after leaving laredo, mr. iglesias taught english at various colleges in houston before deciding to venture into the world of libraries by getting his mlis at the university of texas. from there he quickly settled into academic libraries and has worked in the field ever since. currently mr. iglesias is researching the role of maker spaces in libraries as a way for libraries to continue to be relevant and provide communities of creation for their users. arianna schlegel earned her bachelor’s degree in computer science. after several years in the software world, she decided she wanted her career to go in another direction entirely, and pursued her masters in library science. with these two wildly different but equally beloved fields, she began her work as a digital projects librarian at central connecticut state university (new britain, ct) in 2010. currently she splits her time between her geek-librarian job at central and her reference & instruction position at capital community college in hartford, ct. in her spare time, she knits obsessively, reads everything she can get her hands on, and dabbles in technological pursuits. subscribe to comments: for this article | for all articles 4 responses to "using a raspberry pi as a versatile and inexpensive display device" please leave a response below: shelly, 2013-07-18 http://screenlyapp.com i’ve just started using a raspberry pi and would like to let you know about this great app called screenly. it’s made for the pi. from thier site: digital signage made easy. full hd video and images on your tv screens powered by a single raspberry pi mini computer and the revolutionary screenly software stack. thank you for your article. dan scott, 2013-07-18 this (and screenly, to a certain extent) reminds me a lot of https://gitorious.org/el-cheapo-digital-signage – my quick and dirty means of turning a broken laptop + a tv into an electronic signage solution, albeit with more pain and moving parts than relying on the trusty little raspberry pi :) i like what you’re doing and hope you keep us updated on your progress with this project! mike green, 2014-01-15 good stuff. do you have all the code in github or somewhere? juan, 2014-01-23 i have setup my pi to work as a noc. i have made it boot to web page in full mode, though the webpage requires credentials. do you happen to know how to make the pi log in automatically to the web page, when it boots up? leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – conference reports: code4lib 2011 mission editorial committee process and structure code4lib issue 13, 2011-04-11 conference reports: code4lib 2011 conference reports from the 6th code4lib conference, held in bloomington, in, from february 7 to 10, 2011. the code4lib conference is a collective volunteer effort of the code4lib community of library technologists. included are two brief reports on the conference from some recipients of conference scholarships. by bohyun kim and elias tzoc the 6th code4lib conference was held in bloomington, in, from february 7 to 10, 2011. as in years past, the code4lib community was able to offer scholarships focused on gender diversity and minority representation, this year sponsored by oregon state university and the western north carolina library network. following are conference reports from two of the scholarship recipients. for more notes, many informal, by other code4lib community members, find content on the internet tagged “c4l11” or “#c4l11”, a label attendees were encouraged to use to aid collocation. for instance, archived twitter messages. from bohyun kim i attended this year’s code4lib conference as a first-time attendee thanks to the code4lib minority scholarship. the conference was tremendously informative and inspiring. as one of the attendees (michael j. giarlo @mjgiarlo) tweeted, it was exciting, exhausting, engaging, energizing, and edifying all at once. prepare for information overload all presentations were a short length of twenty minutes, and lightning talks were no longer than 5 minutes. this fast-paced format kept the conference lively and allowed many topics to be presented. however, since each presentation included technical details and made references to many systems and/or software tools, information overload was inevitable. i found the archived video of all talks extremely useful. things i have learned throughout the conference, i found many handy and useful software tools and tricks, such as the ajaxydialog jquery ui widget and the google chart api. i also learned interesting facts, such as opac usage statistics (bill dueber) and the user-interaction patterns on a touch-screen kiosk (andreas orphanides). but the greatest benefit of the conference for me was learning about new developments in the library programmers’ community and seeing how coding is applied to improve library services and products in various areas. things i did not expect while most presentations and lightning talks were technical in content, unarguably the strong point of the conference, there was also a healthy dose of discussion on library services beyond programming. the keynote address by diane i. hillmann and the closing talk by eric hellman addressed broader issues in libraries: metadata and rda (resource description and access) and room for libraries that is shrinking with the growth of the e-book market and the potential for library programmers to introduce and lead changes in the current library landscape. the code4lib conference was also surprisingly open to new attendees. five-minute lightning talks presented a very low barrier to participation. many social events organized via the code4lib wiki and the hospitality suite allowed new attendees to meet and socialize with peers, and to experience the conference at a more intimate and personal level. more offerings for novice coders? throughout the conference and also at the closing talk, the fact that many library programmers are self-taught and often ‘fractional’ coders in the sense that they can afford to spend only a fraction of their time on coding was widely acknowledged. however, during the conference, the focus on self-directed learning opportunities and resources was not strong, excepting beth sadler’s lightning talk, “a guide for the perplexed.” support for the growth and further development of novice library coders is something most libraries neglect to provide despite its extreme importance. while libraries often recruit programmers from non-library it sectors, it is just as important to encourage and support novice coders among existing library staff so they can grow into more experienced programmers. although enthusiastic, many novice coders often don’t realize how certain programming languages or software tools can be applied to current library services and systems and need guidance about which coding skills are most relevant and can be used to produce immediately useful results in the library context. i think that bringing out this theme of self-directed learning for novice library coders more prominently and strongly in future code4lib conferences would make a significant contribution to broadening the relatively small pool of library programmers. from elias tzoc in 1998 roy tennant, in his article the most important management decision: hiring staff for the new millennium, presented what he called “a laundry list” of qualities any candidate for a digital library position should have [1].  the first of eight qualities he mentioned was the capacity to learn constantly and quickly.  this year it was great to join a group of quick learners and coders (hackers) at the 2011 code4lib conference.  i found two main characteristics of this conference: a) the single-track method allows new participants to avoid the problems of choosing between breakout sessions; and b) many participants were not only taking notes or twitting about the sessions, but also coding.  perhaps the lesson here is that while some can talk about something new to implement, others (code4libbers) are actually implementing it.  i was also surprised to learn that many were presenting something that they started to think about at the previous conference; in short, code4lib seems to be a great place for brainstorming and initiating new projects that can produce deliverables for real-life projects in a matter of months. my own example of coding at the conference came on the second day, right after the presentation “a community-based approach to developing a digital exhibit at notre dame using the hydra framework” where rick johnson and dan brubaker horst (university of notre dame) presented their work using a combination of fedora, solr, and the hydra framework.  although we don’t use fedora, this presentation reminded me of a work in progress we had for a web-interface project for a dspace image theme (using xmlui); so i decided to open my vpn client and log into an ohiolink machine. i then opened a vim editor and began working.  after about 20 minutes of editing, saving, and restarting tomcat, a new gallery theme was correctly in place and working the way it should.  speaking of dspace, on the third day, it was great to hear about the “mendeley’s api and university libraries: three examples to create value” by ian mulvaney from mendeley.  among other things, he talked about a plug-in based on sword that will work with major repository systems -including dspacethat can potentially help both authors and ir administrators.  the plug-in will be available at the end of this summer. as for the lightning talks, i think it is an excellent method for newcomers to learn about a variety of topics in a short period of time. some of my favorites included the ajaxydialog jquery-ui widget, blacklight and hydra at the rock and roll hall of fame, open data and the biodiversity heritage library experience, mobile web apps for library exhibits, digital humanities and libraries, and a guide for the perplexed. the last one is a great example of the ongoing work of the code4lib community in providing information about areas of technology being used in libraries, background knowledge required to work effectively in those areas, and tips on how to acquire necessary skills. last but not least, the ask anything session was an interesting combination of basic, advanced, on-site, and online questions as well as short and long answers. for me, the question of the day was “what should we teach new librarians about coding/programming/hacking.”  the answers included the following: learn and try something simple, install linux, use the command line, be a good learner … and after this year, i would add one more to the list: join code4lib -which could be as simple as joining the listserv, attending a conference, checking the code4lib journal or maybe just making time to watch the archived videos of the 2011 conference. just for the sake of diversity, it was also absolutely great to see and talk to people from many parts of the world including japan, uk, france, etc.  thanks code4lib organizers! roy tennant, “the most important management decision: hiring staff for the new millennium”, library journal, 123(1998):102 . http://www.libraryjournal.com/article/ca156490.html about the authors bohyun kim is the digital access librarian at florida international university medical library in miami, fl. she can be contacted at kimb@fiu.edu and blogs at library hat (http://www.bohyunkim.net/blog/). elias tzoc, originally from guatemala, joined miami university libraries after completing his msis degree with a focus on digital libraries from the university of texas at austin in may 2007. as digital initiatives librarian, he assists the department head in providing access and management to the university libraries’ digital collections and the scholarly commons project. http://staff.lib.muohio.edu/~tzocea/ subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – openbook wordpress plugin: open source access to bibliographic data mission editorial committee process and structure code4lib issue 4, 2008-09-22 openbook wordpress plugin: open source access to bibliographic data openbook is a wordpress php plugin that implements the open library apis to insert book covers, titles, authors and publishers into web pages. the motive behind the development was to provide an easy alterative to the common practice of linking to amazon. open library was selected as a data source because it is both open source and open data.the plugin is useful for book reviewers, library webmasters, anyone who wants to put book covers and data on their wordpress blog or website. the plugin also allows users to add links to publisher websites, a feature that was considered significant to independent publishers. by john miedema part i – impetus and high level design choices impetus like many other book bloggers, i have had a long-standing practice of linking book reviews to their associated book data on amazon’s website. it was a convenience for readers seeking detailed bibliographic data or wishing to purchase the book. amazon was the de facto source. a year ago, an employee of random house stumbled across my blog. she offered occasional free books in exchange for a link to their website when i posted a review. this practice seemed fair. after all, a publisher puts more work and risk into a book than a distributor. i adopted a policy of linking to the publisher’s website for all the books i reviewed. recent aggressive moves by amazon compelled me to think deeper about the need to link to an alternate source. the wall street journal reported that amazon had “notified publishers who print books on demand that they will have to use its on-demand printing facilities if they want their books directly sold on amazon’s web site” (trachtenberg, 2008, march 28, p. b.4). the analysis indicated that the move would likely generate a significant profit for amazon. the new york times (carvajal, 2008, june 16) reported that amazon had disabled its “buy button” for print-on-demand titles published by the british unit of hachette livre. the publisher lost single-click purchase power.[1] readers would have to purchase these titles from third-party vendors. the announcement of amazon’s policy and reaction is being tracked at writer’s weekly (2008). the developments are bad news for independent publishing and thought; no doubt they made every librarian wince. as the “buy button” incident demonstrates, links are coin on the web. if anything is to be done to encourage diversity, it will have to be with links. the openbook wordpress plugin was developed to provide an easy way for non-technical web users to link to an alternate source of bibliographic data. the plugin was developed hacker style, and blogged while in process (miedema, 2008a). it got picked up by fellow book bloggers, and by library webmasters who build their website with wordpress. the ongoing interest is encouraging. this article describes the development process and technical details about the software. platform and language openbook is built for wordpress because it is one of the popular blogging platforms[2], and because wordpress is distributed under the gnu general public license (gpl) version 2 (wordpress, 2008). gpl is the most widely used free and open source (foss) license. among other things, this license disallows commercial exclusivity to code. it seemed the right place to start. since the platform was wordpress it followed that the programming language is php. with minor modifications, the code can quickly be adapted to any environment that supports php. data source the most significant decision was the choice of the bibliographic data source. the ideal source would provide an application programming interface (api) that returned bibliographic data on publisher websites in return for a book identifier, e.g., isbn. there is no single source of this kind, and publishers do not use a common standard across their websites. three other sources were evaluated: librarything (http://librarything.com/), a social cataloguing web application developed by tim spalding; open library (http://openlibrary.org/), a project of the internet archive and the open content initiative; and worldcat (http://worldcat.org/), the catalogue of oclc. data richness. of the three sources, librarything has the richest social software functions and data. worldcat has social software but it is hardly used. open library does not have social software but it does have the full text of public domain books that have been scanned.while rich book data is of value to book bloggers and readers, it was not considered a differentiator for the first version of openbook. profit status. the purpose of openbook is to provide an alternative to amazon because of its aggressive commercial moves. given that, a non-profit might seem a better choice for openbook than a private operation. librarything is a private operation, though it is not a large corporation. the internet archive and oclc are non-profits. however, it was recognized that neutrality is always difficult, and non-profits have agendas too. furthermore, there may be advantages to shared projects with private companies willing to invest resources in innovation. for these three sources, profit status alone was not a significant differentiator. open data. worldcat has the greatest number of titles but only for member libraries, with data accessible only to librarians. librarything and open library both have open data – anyone can add and modify titles. it may be argued that open data is more prone to error, but only librarians can correct errors in worldcat and the incentive for doing so is insufficient (beall, 2008). the ability to add outlier titles is central to the current purpose of encouraging diversity in the book industry. worldcat was ruled out. foss. librarything was built on open source software, and provides open source apis, but the core librarything code is proprietary. it must be so; otherwise competitors would steal their labour and put librarything out of business. the open source code at librarything may be called commercial open source software (coss) not foss (described earlier). any private business that uses coss has the ability and right to discontinue free use of its software at any time. open library’s code is licensed under gnu affero general public license (http://www.gnu.org/licenses/agpl.txt). should the day come that open library misuses its monopoly on bibliographic data, another operation could take its code, and compete with it on equal footing. open library was selected as the data source to replace amazon. since the deciding factors were open data and foss, the product was named openbook. it was recognized early on that open library is still in beta and has some bugs; this risk was considered acceptable, especially since the openbook plugin is itself an experimental piece, designed more for the argument it makes than for its technical quality. the source will be re-evaluated in the future for stability and enhanced features. it should be noted that each of the sources links to amazon, in addition to links to other sellers and worldcat. there is no intent to block access to amazon, if such a thing was even possible, but rather to inject some diversity into the book business. part ii – technical design openbook is a relatively small piece of software, implemented in 250 lines of code. the following sections highlight significant technical features. the source code can be downloaded from the wordpress server (miedema, 2008b). graphic design the graphic design for openbook is a simple combination of a cover image, title, author and publisher. figure 1 shows a sample of openbook used on my website, a review of the republic of nothing by lesley choyce (2007), published by goose lane. the cover image is an alternate cover that i added to open library. it comes from the publisher’s website where it is expressly offered as a download (goose lane, 2008). many publishers encourage the distribution of their cover images because of their promotional value. figure 1. sample openbook data light styling is added to float the text next to the image, and to distinguish the text items. the cover and title link to the detailed web page in open library, and the publisher text links to the publisher’s website. for now, openbook only displays data for the first set of tags it finds, and always displays it in the upper left corner of the page. the simplicity of the design was deliberately chosen, to avoid formatting problems in the first version of the plugin. the openbook tags and book identifier to display the bibliographic data, users need only insert two tags and a book identifier into their page, as in sample 1. [openbook]0864921535[/openbook] sample 1 the goal of open library is to have a web page for every book ever published (open library, 2007). every book has a webpage with a distinct url, as in sample 2. <a href="http://openlibrary.org/b/ol882707m">http://openlibrary.org/b/ol882707m</a> sample 2 sample 2 contains the open library key, "/b/ol882707m" [3]. the key is required to look up the item with the open library apis. openbook allows users to look up books in two ways. if a user provides an open library key (detected by the “/b/ol” pattern), openbook can look up the book data directly using the open library “get” api. users should not have to look up these proprietary numbers, so openbook also allows users to look up books by their ten or thirteen digit isbn. there can be multiple keys for a single isbn, so openbook must first use the open library “search” api to find a list of matching keys. by default, openbook uses the most recent record, but uses can pass an additional argument so select a particular version. once the key is selected, the “get” api can be used as above. see the examples in the following section. the ability to search by open library key serves two important purposes. first, during the development process, it was learned that open library does not automatically add titles to its search index if the titles have been added manually though the website. in this case, the title cannot be found with the search api until a scheduled manual routine is performed to add it to the search index. users cannot count on finding new titles by isbn, so they must use the open library key as a workaround. the folks at open library told me that a fix was high on their priority list. i have recently opened a ticket on the problem (2008). it has been marked as high importance and work is in progress, but the problem is not yet resolved. second, a main purpose of openbook was to provide access to independent publishers. some books and e-books do not have an isbn. the ability to search by open library key provides access to these titles. the open library apis openbook uses two of the four apis documented by open library (2007b)– the “get” api and the “search” api, as described above. the apis for open library are exposed through a browser url and querystring. the “get” api retrieves bibliographic data for a specific key. the api and the key are assembled into a url, such as the one in sample 3: <a href="http://openlibrary.org/api/get?key=/b/ol882707m">http://openlibrary.org/api/get?key=/b/ol882707m</a> sample 3 the “search” api looks up keys by its ten or thirteen digit isbn, and is formatted like the example in sample 4: http://openlibrary.org/api/search?q={"query":"(isbn_10:(0864921535) or isbn_13:(0864921535))"} sample 4 by default, the apis return an object. if you enter one of the above urls in a browser’s address bar, you will be prompted to save the results to disk, a file that you can read with a text editor. you can return results as text by appending "&text=true" to the querystring. you can format them for easier reading in your browser by appending "&prettyprint=true" . retrieving the data in php php provides two methods by which to call the urls and insert the results into a variable in order to work with them. in the first release of openbook i used the “file_get_contents” method, but user feedback indicated that many servers disallow this method. i switched to the curl method, as illustrated in sample 5: $ch = curl_init($url); $bookdata = curl_exec($ch); sample 5 this sample creates a curl handle, $ch, based on the url for the open library api. the handle is executed, and the book data is returned into a php variable, $bookdata. parsing the data in php the data returned by open library is in javascript object notation (json) format, an industry standard, text-based, language-independent data interchange format. it is easy for people to read and for machines to parse. php 5.2 comes with a json library that makes quick work of data extraction. my web host defaulted to php 4, but i was able to upgrade with a simple switch on my control panel (enetfirms, 2008). sample 6 shows the syntax for extracting bibliographic data into php variables from the $bookdata variable created in the last section. $obj = json_decode($bookdata); $result = $obj->{'result'}; $title = $result ->{'title'}; $subtitle = $result ->{'subtitle'}; $authors = $bookdataresult ->{'authors'}; $publishers= $result ->{'publishers'}; sample 6 the upgrade to php 5.2 was easy, but it caused a few stumbles for people when i released openbook. a subsequent update of openbook included the json library with the plugin, making it compatible with php 5.0. cover images and publisher links the $bookdata results do not typically contain a url for the book’s cover image, only for alternate covers added by users. a blog reader commented that i could construct the urls for the cover images because open library uses standard url and file naming conventions. logic was added to openbook to construct the cover url or use the alternate url if it was provided. the results did not contain links from the book to the publisher’s website. no known bibliographic data source provides this association. it was decided that for this first version of openbook, a link could be passed as an argument. more information about the arguments is provided in the readme file that comes with the plugin download. sample 7 illustrates how a publisher link can be passed. [openbook]0864921535,,,,http://www.gooselane.com/[/openbook] sample 7 assembling the html and hooking into wordpress once all the data was gathered, it was formatted into an html string according to the graphic design specification discussed above. wordpress allows for the creation of filter functions that can be used to search and replace content whenever a user saves a page. openbook searches for the tags, then replaces them with the html display of the bibliographic data. the completed code went into a function in a file with a “php” extension. the file was uploaded to the wordpress server where anyone can download it (miedema, 2008b). part iii – progressive software and future plans software that matters is shaped by progressive ideas, such as those of brewster kahle, founder of the internet archive. during the development of openbook, microsoft announced that it was ending its role in the open content alliance from which open library gets its scanned content (anderson, 2008). kahle made some noteworthy remarks. he thanked microsoft for it generous contributions, and for doing more than its share to facilitate a smooth ramp-down. he also viewed the development as a potential move forward: he’s a firm believer in the idea that corporations should not be the entities we trust to provide access to important cultural data stores. if people think that corporations are the right way to access the history of human discourse, kahle says they’re in for “a series of very rude shocks.” (anderson, 2008) kahle’s view is consistent with the purpose of openbook to provide an easy alternative linking practice to counter amazon’s near-monopoly on book sales, and more importantly, to encourage diverse media and thought. librarians are rapidly increasing their technological capabilities, but progressive ideas are what make this software relevant. openbook was drafted quickly as more of an argument piece than an enduring piece of software. there has been a strong positive response to the first release, and i have supported the plugin with enhancements based on user feedback. the latest version (1.6 beta) has new features, including a link to the worldcat record for a valid isbn, the ability to insert multiple covers in one post, and handling for open library down time. keep up on the latest features at the openbook support page (miedema, 2008a). it is expected that openbook will move out of beta stage by the end of 2008. references ^,^anderson, nate (2008). why killing live book search is good for the future of books. retrieved from http://arstechnica.com/news.ars/post/20080526-why-killing-live-book-search-is-the-right-thing-for-ms.html on august 7, 2008. ^beall, jeffrey (2008). oclc: a review, in roberto, k.r., eds. radical cataloging: essays at the front, pp. 85-93. mcfarland. retrieved from http://eprints.rclis.org/archive/00013701/ on august 7, 2008. ^carvajal, doreen (2008, june 16). small publishers feel power of amazon’s ‘buy’ button. the new york times. retrieved from http://www.nytimes.com/2008/06/16/business/media/16amazon.html on august 7, 2008. ^choyce, lesley (2007). the republic of nothing. fredericton, new brunswick: goose lane. gnu (2007). gnu affero general public license. retrieved from http://www.gnu.org/licenses/agpl.txt on august 7, 2008. goose lane (2008). the republic of nothing: reader’s guide edition. retrieved from http://www.gooselane.com/book/9780864924933 on august 7, 2008. (coins) ^, ^miedema, john (2008a). openbook wordpress plugin. retrieved from http://johnmiedema.ca/openbook-wordpress-plugin/ on august 7, 2008. ^, ^miedema, john (2008b). openbook book data – wordpress plugins. retrieved from http://wordpress.org/extend/plugins/openbook-book-data/ on august 7, 2008. ^netfirms (2008). how do i switch between php4 and php5? retrieved from http://support.netfirms.com/article.php?id=713 on august 7, 2008. ^open library (2007). about us. retrieved from http://openlibrary.org/about on august 7, 2008. ^open library (2007b). open library api. retrieved from http://openlibrary.org/dev/docs/api on august 7, 2008. ^open library (2008). bug #267853 in open library: manually added titles not added to search index. retrieved from https://bugs.launchpad.net/openlibrary/+bug/267853 on september 20, 2008. ^trachtenberg, jeffrey a. (2008, march 28). amazon tightens grip on printing. wall street journal (eastern edition), p. b.4. retrieved from abi/inform global database. (document id: 1452793061) on august 7, 2008. ^wordpress (2008). gpl. retrieved from http://codex.wordpress.org/gpl on august 7, 2008. ^writer’s weekly (2008). amazon booksurge information clearinghouse. retrieved from http://www.writersweekly.com/amazon.php on august 7, 2008. footnotes [1] subsequent investigation (august 2008) has been performed on amazon using the hachette livre title mentioned in the ny times article, “the hachette guide to french wine”. the buy-with-1-click button is currently active for this title (after signing in). it is unclear if amazon has back-tracked on the policy, perhaps due to the legal action reported at writer’s weekly (2008). [2] wordpress distinguishes between blog posts and pages. openbook works equally well in both. for simplicity this article just refers to pages. [3] the “b” prefix stands for book. other searchable objects have a different prefix, e.g., “a” for author. about the author john miedema is a library student at the university of western ontario and an it specialist with ibm. john balances out his busy it world with slow reading. he is the author of a forthcoming book on slow reading to be published by litwin books. contact john at mail@johnmiedema.ca. tags: book review resources, wordpress, wordpress plugins subscribe to comments: for this article | for all articles 4 responses to "openbook wordpress plugin: open source access to bibliographic data" please leave a response below: peter, 2009-11-04 i like openbook very much and i’m using it in my blog now, too. thanks for that great article, john. regards, peter from universalfernbedienung openbook article published in code4lib journal – publications and presentations, 2011-12-16 […] code4lib journal published my article on openbook. check out the article here. the full issue is available […] openbook article published in code4lib journal, 2012-10-27 […] the code4lib journal published my article on openbook. check out the article here. […] phim sex mới, 2025-06-29 https://muqing.uk.com/ leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – using zapier with trello for electronic resources troubleshooting workflow mission editorial committee process and structure code4lib issue 26, 2014-10-21 using zapier with trello for electronic resources troubleshooting workflow troubleshooting access problems is an important part of the electronic resources management workflow. this article discusses an opportunity to streamline and track troubleshooting using two web-based services: trello and zapier. by meghan finch after being hired as the digital assets librarian at oakland university, one of the first tasks i accepted was to tame the troubleshooting of access errors for our electronic resources. during my first year i received comments from our users regarding our resources through a newly set up online form and mailing list. data for an annual report was manually compiled from reviewing the emails stored in a labeled email folder, which indicated only the number of emails received each month and then the total for the year. the troubleshooting tasks became overwhelming for one individual to manage and i began to look to other staff to help resolve problems and respond to patrons. i wanted to make training as easy as possible and have a simple way to provide help and feedback without hovering. additionally, i was receiving a lot of information about the state of our electronic resources and where our problems might be, but i wasn’t easily able to sort and visualize any of this data. i began to investigate opportunities to: allow several team members from acquisitions, as well as myself, to coordinate efforts to resolve and respond to issues track the types of problems encountered to help determine where it might be worth making efforts to improve the system or when certain vendors may be more problematic than others visualize that information to see how much work was being done and in what areas in considering options, i looked at both the existing bug tracking system at oakland university, bmc footprints, as well as the potential for an electronic resource management system (erm) to help manage these tasks. bmc footprints was first considered as an option for tracking reported e-resources issues. footprints was already in place at the university and used primarily by central it to track projects, routine tasks, and issue resolution. footprints offered the ability to receive issues via email and an extensive hierarchy with which to assign responsibilities. however, the user interface for both the internal troubleshooters and patrons was too complex for the task. rather than making training easier, this looked likely to cause more confusion. having eliminated the existing bug tracking system from consideration, the next option was an erm. in winter of 2014 ou libraries investigated erms from ex libris, serial solutions, and ebsconet, but none had any features related to an error tracking workflow. trying out trello following the previous failed investigations, i then considered trello as an option for managing the workflow. trello is a workflow, task management, and project management tool. the service provides a web-based workspace for building bulletin boards for project task management. users create cards, modeled on index cards, which are organized on lists typically titled with a relevant category. the default list types populated when a new board is created are “to do,” “doing,” and “done,” though the user can add, delete, and edit lists on each board. trello provides a simple and flexible to do list or allows the user to design elaborate workflows with added features such as color-coded labels, comments, checklists, due dates, and attachments. these options are included in the freely-available version of trello, with additional features, such as stickers, background customization, and csv exports available in their gold and business class paid versions. other academic libraries have begun to experiment with using trello in their workflows. a 2013 presentation at the annual electronic resources and libraries conference by nancy beals and jennifer bazeley demonstrated trello use in an electronic resources workflow. trello was already being used by acquisitions and cataloging staff at ou libraries as a way to track projects in each department before it was considered for e-resources management. all technical services staff had accounts with trello and were familiar with the mechanics of card creation, notes, and moving notes to different lists. this familiarity with and acceptance of the tool made it an appealing option for working with electronic resource error resolution and tracking. incorporating zapier the next step was to investigate what options were available for automatically generating trello cards without the library patrons needing trello accounts. the online form was already in place and working well, so effort focused on incorporating trello into this reporting mechanism. oakland university uses google as its institutional email provider. both google apps and trello have documented apis that would allow the two applications to work together, making communication from ou email and trello an option. new tools designed to enable users to communicate between web services without extensive technical knowledge have recently developed, including ifttt (if this then that) and zapier. these web services allow users to use simple if-then statements to allow changes in one web application to result in actions taken in another web application. ifttt refers to these statements as recipes, while zapier calls them “zaps.” both ifttt and zapier have free-to-use versions and offer the ability to connect many popular online services. ifttt was not considered for use with this project, however, because it does not support trello integration. zapier’s support for trello and google docs integration led me to select it for use in this project. from form to trello card the form available to patrons for reporting errors is available from the ou libraries website in two locations. buttons are available on the serial solutions 360 link open url linking service as well as each page of the databases a-z list. the majority of errors reported originate from the serial solutions 360 link page. patrons encounter this page during an error regardless of whether they are coming from our summon discovery service or from any of our subscribed databases using 360 link to connect users. the button takes the patron to a simple web form. the form has no required fields, and asks the patron to provide their email, the name of the resource, and a description of the problem. in the background, the form also gathers the url where the “report an error” button was selected. this helps to provide additional information when a patron is vague or doesn’t provide enough information to get started on troubleshooting. once the form is submitted, it is sent to the email address of the e-resources mailing list. email + zapier + trello the zapier zap was set up to look for any email sent to the digital assets librarian from the address of the listserv. once identified, the information in the email is sent to trello creating a new card on the “to do” list in the electronic resources troubleshooting board. google to zapier mapping the zapier zap is designed to take fields from a google email and crosswalk that information to a field representing a feature in a trello card, as shown in table 1. for our troubleshooting emails, the subject and the email body are the only information items mapped to the trello card. google mail field trello card (email) subject name message (email body) description table 1: mapping fields to zapier cards additionally, the following settings for a trello card are set automatically by zapier (table 2): trello card setting trello card value list to do label none member (any member of the team, rotates weekly) due date tomorrow at midnight checklist name e-resources troubleshooting tier i table 2: fields automatically set in trello cards beyond creating cards from email information, several other presets are available. a predefined checklist stored in zapier is attached to each card to help troubleshooters get started reviewing a problem. this checklist includes questions such as: are you able to view the resource on-campus? do you receive a proxy error message page? this checklist is created in zapier and attached to each new card that is generated for the trello board. i update the zapier zap each week with a different member of the team assigned to the trello board. when a card is created, it is automatically assigned to that user for the week and that user is notified when a new trello card is created via trello’s notification system. each user can select settings to be alerted by email or just by the trello internal notification alerts. keeping everything in the same place the electronic resources troubleshooting trello board was set up with seven lists: to do tier ii waiting completed get done how to honey badger tips workflow lists: to do, tier ii, waiting, and done the primary workflow lists are the “to do,” “tier ii,” “waiting,” and “done” lists. cards typically move from “to do” to “done,” unless a problem requires more complex troubleshooting or a response from a third-party such as a vendor or service provider. cards with more complex needs are moved to “tier ii” where they are addressed by the digital assets librarian, while cards waiting on a third-party response are moved to “waiting.” (see figure 1) figure 1. troubleshooting workflow information lists: get done, how to, and honey badger tips many different variables are considered when troubleshooting e-resources. a troubleshooter may need to check proxy settings internally, email the knowledge base provider, or contact the support staff of a content provider for a problem that is reported. it was important to provide access to as much of this information as possible from within the trello board to minimize the need to hunt for outside information. the “get done” board helps to provide that information, listing important email addresses and technical support contact forms. the “how to” board provides the instructions for using the e-resources troubleshooting trello board. keeping documentation within the board provides quick access available within the environment, allows for easy updates, and connects easily with google docs for more complex documentation. when first approached to take on the task of working with e-resources, i asked the acquisitions team if they wanted a mascot to represent the group. they already had a mascot selected: the internet famous honey badger. the board is sprinkled with images of honey badgers to keep spirits up, and the “honey badger tips” board offers simple reminders on common customer service style responses and expectations. this includes example emails to help troubleshooters respond in a consistent way to patrons and reminders to stay away from internal jargon. tracking error types beyond resolving access problems, i was also interested in spotting trends in errors. this is where trello’s label system became useful. reviewing emails prior to trello implementation, i identified major areas of error: user errors errors originating with the discovery layer, knowledge base, or open url linker (all serial solutions in our case) errors originating with the resource provider (database or publisher) proxy errors problems associated with renewals (money not received in time, access removed even after paid in full, etc.) each of these error types was assigned to a trello label. once resolved, the troubleshooter adds the appropriate label to the card and then moves it to the “done” board. trello allows for multiple labels to be added to a single card, so we can add as many labels as necessary tracking in google docs completed cards can be viewed in trello even after they have been archived; however, it can be difficult to get a sense of the number of problems reported at different intervals and the types of problems being reported. to really understand this better, we needed to export the data from trello. zapier was again used to send the data from trello to a google spreadsheet. zapier was able to export: card name date of last activity labels id members (who was assigned to the card) each element was entered into their assigned column in a new row of the spreadsheet. once in a sortable google spreadsheet, information became much easier to read and quantify. visualizing data in google spreadsheets once the data is out of trello and into a spreadsheet, more can be done to sort and view it. errors can be sorted by date and type of error, and values such as the number of errors each month, the type of each kind of error, and the average time it takes to resolve an error can be quantified and visualized to present to colleagues and stakeholders. this information can help better inform decision-making related to electronic resource acquisition, renewal, and management. the example in figure 2 shows the type of graph that can be created from pivot tables of data exported from trello. this shows the distribution by month of the 50 recorded e-resources errors reported since trello’s implementation in march 2014. the colors are coded to correspond to the color of the labels in trello and show the distribution of the types of errors reported. figure 2. example graph of issue data future opportunities for ou libraries and trello trello and google api zapier zaps demonstrate that google and zapier work together to exchange data. trello’s drawback, however, is a limited number of fields in which values from an outside source, such as google, can be stored. a lot of information that could be more useful if sorted is lumped together in the “description” field of each trello card. working with the google and trello apis directly rather than through the zapier intermediary, it may be possible to parse information from this single field to be more useful in the final google spreadsheet. beyond the troubleshooting workflow we are beginning to expand our use of trello beyond just the troubleshooting workflow. ou libraries have just created a board dedicated to short-term loan and purchases from our dda program and are directing all email correspondence to this new trello board. trello gives us the ability to take a task that was previously only tracked via email to a dashboard that can be viewed by all team members and from which data can be gathered quickly. we will likely be exporting data from these boards as well, and the possibility of building an internal dashboard with this information is possible. some work performed by zapier, including the export of data, can be handled by trello with a business class subscription. while the free work-around is useful for ou libraries currently, it may be worth the investment to move to a business class account to take advantage of easy data exports. author meghan finch is the digital assets librarian at oakland university in rochester, mi. her professional interests include electronic resource management strategies, digital libraries, institutional repositories, and digital preservation. subscribe to comments: for this article | for all articles one response to "using zapier with trello for electronic resources troubleshooting workflow" please leave a response below: alison groves, 2014-10-31 this is so cool meghan! loved reading such an interesting use of zapier. :) leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – archiving an early web-based journal: addressing issues of workflow, authenticity, and bibliodiversity mission editorial committee process and structure code4lib issue 54, 2022-08-29 archiving an early web-based journal: addressing issues of workflow, authenticity, and bibliodiversity switch is a journal of new media art that has been published in an online-only format since 1995 by the cadre laboratory for new media at san josé state university (sjsu). the journal is distinctive in its commitment to presenting scholarship and criticism on new media art in a visual format that reflects and enhances its engagement with the subject. this approach, which includes the practice of redesigning the journal’s platform and visual presentation for each issue, raises significant challenges for the long-term preservation of the journal, as well as immediate issues related to indexing and discovery. this article describes the initial stages of a collaboration between the martin luther king, jr. library and the cadre laboratory at sjsu to archive and index switch and to host a copy of the journal on sjsu’s institutional repository, sjsu scholarworks. it will describe the process of harvesting the journal, share scripts used to extract metadata and modify files to address accessibility and encoding issues, and discuss an ongoing curricular project that engages cadre students in the process of augmenting metadata for switch articles. the process reflects the challenges of creating an authentic version of this journal that is also discoverable and citable within the broader scholarly communication environment. this effort is part of a growing multi-institutional project to archive the new media art community in the bay area in a 3d web exhibition format. by nick szydlowski, rhonda holberton, erika johnson introduction switch, a journal of new media art, has been published in an online-only format since 1995 by the cadre laboratory for new media (cadre) at san josé state university (sjsu). in 2021, the martin luther king, jr. library at sjsu began to work with cadre to archive the journal in sjsu scholarworks (https://scholarworks.sjsu.edu/), the university’s institutional repository, which is hosted on the digital commons platform. switch was one of the first online academic journals to embrace the visual possibilities of the world wide web (www).  the first electronic-only scholarly journal, postmodern culture, was founded five years earlier, in 1990 (amiran, 1991), but, like most of the earliest electronic journals, it was initially published in a text format that did not allow for graphics or specify the layout or design of the page. the year switch was published, the association of research libraries (arl) compiled a print directory of all online serials, which was not limited to academic journals or publications in english, listing only 675 journals and newsletters of any type (king and kovacs, 1995). the majority of these were published via email or gopher – less than half used the www. going back a year to 1994, the directory lists only 36 journals with urls (mogge, 1999). switch was truly a pioneering publication. the process of archiving switch presents a particular set of challenges due to the journal’s status as an early electronic journal coupled with its emphasis on design and visual presentation. one unique feature, referenced in the journal’s title, is the fact that each issue of the journal showcases a new visual design and layout. nearly all of the journal’s issues are still live on the web in their original formats, and the evolution of the journal represents a capsule history of web design strategies and site structures. however, these original versions are not visible to researchers as part of the broader scholarly literature, or easy to find using search engines or library discovery tools. because of the unique features of each issue, the journal can be challenging to browse and cite. for these reasons, it was very desirable to produce a version of the journal that would be easy to discover, browse, and cite, while maintaining as much of the authentic experience of the journal as possible. this article describes the development of a process to archive the journal in a pdf format and to create metadata for each article. we have automated as much of this process as possible using a modular, multi-step workflow. each step of the workflow is described separately, and while the entire workflow might be most applicable to journal-based projects, several of the modules can and have been used in targeted web archiving and in the archiving of digital humanities and other web-based scholarship projects. moreover, the collaborative nature of the project is an example of how libraries and other cultural heritage institutions can work more directly with communities of practice in a digital environment. the workflow also includes stages which are very much not automated, but instead use the archiving process to create unique curricular opportunities for students in the cadre program. students who are involved in the creation of current issues of switch have worked in class to write abstracts and create additional metadata for specific articles in the archived journal. these assignments have been incorporated into classes, allowing students to gain information literacy skills and understand the role of metadata in scholarly research, to encounter challenging scholarship in their field, and to bring their own experiences as scholars and artists to bear in making that scholarship discoverable. the students’ work is incorporated into the presentation of the archived journals, and students are publicly credited with their work on the collection’s metadata. the archiving of the journal and the integration of the process into the cadre curriculum are both part of a larger project to archive the new media art community in the bay area in a 3d web exhibition format. while the larger project involves a number of additional institutions and partners as well as a different technical approach, this initial project is an important step in creating and modeling active collaborations between the community of new media artists and cultural heritage institutions, and in building capacity to preserve complex digital artifacts that represent the early days of art on the www. a very brief history of switch switch was founded in 1995, only two years after the www made it feasible to create and share web pages using a graphical layout. the journal was edited by cadre students under the supervision of artist and cadre faculty member joel slayton. figure 1 shows the homepage for the first issue of switch, on the theme of information on the internet. figure 1. issue #1 of switch as it was originally published. screenshot from http://switch.sjsu.edu/archive/switch/switch_v1n1/switch.html. the first 16 issues of the journal each feature unique visual presentations and appear to consist of hand-coded html and css. with the 17th issue, published in 2001, the journal was relaunched on a platform which presented all of the previous issues in a standard visual format. that platform was only used for one additional issue (issue 18, on the topic interface: software as cultural production) before the journal returned to the tradition of introducing a new visual design for each issue. however, the re-launched versions of issues 1-16 came to be the most accessible and discoverable versions of those issues. the earlier versions were so difficult to find that it was only through the archiving process that our team became aware that the original versions of those issues were still live on the switch server. figure 2 shows version 2 of the first issue of switch. version 2 appears to be based on a linux, apache, mysql, and php (lamp) architecture. while in version 1, each article consisted of one or more html files stored directly on the switch server, for version 2 the switch editors stored the contents of each article in a mysql database, and the html of the page was generated via php when the user requested the page. while each issue has its own header image in version 2, the format of each issue and article is much more consistent that in the original version. figure 2. version 2 of issue #1. screenshot from http://switch.sjsu.edu/archive/nextswitch/switch_engine/front/front.php%3fcat=5.html the initial decision to begin the project by archiving the 18 issues on the version 2 platform was made before we became aware that version 1 of the journal was still live on cadre’s server, and it offered the primary advantage of providing a large set of structurally similar files on which to work out our automated workflow . we will discuss some of the other advantages and disadvantages of focussing on version 2 later in this article. process and workflow the process used to archive issues 1-18 and ingest them into sjsu scholarworks is as follows: first, httrack software (https://www.httrack.com/) was used to create a localized archive of the publicly available switch server second, scripts were used to address accessibility and encoding issues, simplify html structure, and extract basic metadata from each article the resulting simplified html files were batch-converted to pdf using adobe acrobat after the pdf files and basic metadata were published in sjsu scholarworks, cadre students augmented the metadata with abstracts and additional description, added alt text to images using adobe acrobat, and audited the extent to which version 2 articles faithfully represent the original versions we will discuss each of these steps below, with particular attention to methods that may be applicable to other attempts to perform targeted web archiving. harvesting switch to a local copy the project team created a local copy of switch using httrack, a gpl-licensed offline browser created by xavier roche and other contributors. while there are many methods available for crawling the web, httrack may be attractive to library practitioners due to its straightforward and stable graphical user interface and its options for controlling the behavior of the web crawler. the tool is largely absent from the library literature, but one member of our team has been using it in library practice for over a decade. we began our crawl on the switch archive page (http://switch.sjsu.edu/archive/archives/index.html) and allowed httrack to follow links on the site to archive the contents of the switch server. this approach yielded a local archive of html and other files with a folder structure that reflects the page urls. under this approach the articles from version 2 of issues 1-18 could all be found in a single local folder. for this project, the local copy was of particular use due to the complex and sometimes ambiguous url structure of the switch site. put simply, the server contains over 25 years of web development by cadre students and faculty. creating an offline copy allowed team members to browse the site through the file browser of their local operating system, providing a different view which clarified questions about the journal’s history and content. this method also effectively transforms the output of content management systems and similar database-backed sites into a set of html files, allowing for subsequent transformation and archiving. modifying harvested content and extracting metadata browsing the offline copy revealed 237 articles that had been published on the version 2 platform. the team chose to focus first on those articles first in order to establish a workflow that could be customized further to address the later issues. examining those 237 articles suggested three goals for the next phase of the workflow: addressing encoding problems in the version 2 articles, taking steps to enable the creation of accessible pdf files, and extracting metadata. all of these goals were addressed primarily using the python library beautiful soup 4 (https://www.crummy.com/software/beautifulsoup/bs4/doc/) using a working copy of the archived site. encoding issues visual inspection of both the archived and live versions of the version 2 files revealed a large number of missing special characters when the files were displayed in a web browser. after troubleshooting, it became apparent that while the header of the version 2 pages indicate text encoded according to the iso-8859-1 standard, the text that represents the articles is actually encoded using the windows-1252 standard. this was likely an artifact introduced during the process of creating version 2 of the journal. this was addressed by rewriting the files while replacing the encoding text. the very simple code below was used to address these issues. [1] filedata = filedata.replace('utf-8', 'windows-1252').replace('iso-8859-1', 'windows-1252') when the files specified the correct encoding, web browsers were able to render the special characters correctly. accessibility issues the version 2 site presented two categories of accessibility issues which, unless addressed, would be carried over to any subsequent versions. firstly, the layout features many areas with insufficient contrast between the text and background. secondly, all images lack alt text. the contrast issues could be addressed in an automated fashion by lightening the background of the content area and darkening text for certain html elements and classes. the initial plan for image alt text was for cadre students to add it to the pdf file for each article at the same time that they wrote the abstract for the article. adobe acrobat provides a reasonable set of tools for adding alt text to pdf files, but testing that process on initial batches of switch pdf files revealed that the complex html of the original files resulted in a pdf file that was incompatible with acrobat’s accessibility tools. switch’s version 2 platform, launched in 2001, uses nested tables to execute page layout, as was typical at the time. the article content is contained in a single cell of a table, which is the child element of another table, which is itself the child of a third table. a simplified and schematized version suggests why these pages might present challenges for tools like adobe’s that are expecting modern web page structures: <table><! -page layout table -—> <tbody>...<tr><td> <table><! -content area table -—> <tbody>...<tr><td> <table><! -v2 database output table -—> <tbody>...<tr><td> <! -article content html-—> </td></tr></tbody></table></td></tr></tbody></table></td></tr></tbody></table> in order to produce pdf files that could be made accessible using the available tools, the team chose to rewrite the html files to produce files that are visually identical to the original version 2 pages, but which use css and inline styles rather than tables to produce the layout. the contrast and page structure adjustments were combined into a single script which rewrites the files to a new folder. a simplified version is included below – the full script is available at https://github.com/nickszydlowski/switch/blob/main/rewrite-accessible-colors.py. because the version 2 articles follow a consistent structure, it is possible to extract the unique content from the page programmatically. article content and issue-level navigation were displayed in table cells with the classes “dbouput” for the article “specialobjs” for the navigation. the first three images on the page are also retrieved, as they make up the header image which changes from issue to issue. outside of these five elements, the remainder of the page is the same from article to article and can be replicated with greatly simplified static html. article = soup.find(class_='dboutput') special = soup.find(class_='specialobjs') images = soup.find_all("img") image1 = images[0] image2 = images[1] image3 = images[2] as an example, here is the code used to add the header images to the new simplified file: <div id="issueimage" style="background-color:#444444; clear:both; height:200px;"> {secondimage} {thirdimage} </div> the original html is too complex to easily reproduce. in it, these adjacent images are at the top two separate cells of a single table row. the special navigation is below the first image, in the same cell, and the article itself is below the second image, again in the same cell. in this case, simplifying the html structure of the page had accessibility benefits even when converting from html to pdf. a similar process could be used to extract the article content from the page in order to create a new html or xml version of a web-based publication. the same script addresses the accessibility issues present due to inadequate contrast on the version 2 site. because the original site uses a mix of inline styles and css, it was simplest to use a multi-pronged approach to addressing styles. here, as a last resort, background colors that were repeatedly specified in inline styles are replaced directly. # accessibility changes make changes here where inline styles and other issues make css difficult output=output.replace('bgcolor="#666666"','bgcolor="#888888"') these changes maintain the look and feel of the version 2 site while facilitating improved accessibility.  figure 3 shows screenshots of each version. figure 3. comparison of original and archived article. screenshot from from version 2 http://switch.sjsu.edu/archive/nextswitch/switch_engine/front/front.php%3fartc=30.html figure 3a. comparison of original and archived article. screenshot from archived pdf. https://scholarworks.sjsu.edu/cgi/viewcontent.cgi?article=1118&context=switch. note increased contrast for accessibility. web archiving approaches that are suitable at scale may struggle to address accessibility issues that are common in older websites. additionally, users of older sites may agree that character encoding problems leading to missing characters or the garbled text, sometimes called mojibake, are a particularly common issue for older sites. (mcnulty, alvarez, and langmayr, 2021) the practices described here show some of the benefits of a more hands-on approach to archiving early web content, in particular when that content is of importance to a community of practice or an institution. there are clear parallels here to the preservation and conservation practices that libraries apply to print materials: materials of local and community importance warrant hands-on conservation treatment as needed, while the much larger set of “general collection” materials can be treated with practices that are capable of scaling up to larger bodies of material.(e.g. boice, draper, lunde, and schwalm, 2017). metadata extraction the final script that was executed against the set of files used beautiful soup to parse out the article-level metadata available for each article and write that metadata to a csv file. available fields were title, author, issue number or issue description, date published, and filename. this module is an example of how close attention to the structure of these older files makes it possible to parse out data that is otherwise difficult to acquire programmatically. the pages do not include metadata in the header, and few elements have unique class names or ids. the code used to acquire the issue and date information is included below. issue =soup.find(class_='dboutput').find('table').find(align='right').find('a').get_text(":", strip=true) date =soup.find(class_='dboutput').tr.next_sibling.next_sibling.a.next_sibling.replace('\n', '').replace('\r','').replace(' on ','') even this minimal metadata, when added to a modern journal hosting platform, dramatically improved the discoverability of individual articles, as well as the journal as a whole. upon uploading to digital commons with minimal metadata, articles were almost immediately indexed in google scholar and easier to find on google. this information also provides a structure for the remaining steps of the project, in particular the curricular component which gives cadre students an opportunity to augment this metadata and grapple with the challenges of archiving this content. curricular component and metadata augmentation students in the cadre class art 104 – interdisciplinary seminar in digital media art are involved in the creation of new issues of switch and engaged academically and artistically with the themes addressed in the journal. being directly involved in archiving past issues of the journal is an opportunity for students to gain information literacy skills by learning more about how scholarship is created, shared, and archived. in particular, the challenge of writing abstracts for individual articles helped students understand how metadata influences the discoverability of scholarly articles. students each chose three articles to work on over the course of the semester. for each article, students wrote abstracts and assigned subjects and keywords, added alt text to images, and researched the author or authors of the piece. students also located the article on the original switch site and compared the archived version to the original. in many cases, students identified missing content or other issues that were introduced during the creation of version 2 in 2001. of the 98 articles students examined in spring 2022, 36 had missing content from version 1 to version 2, with the most common issue being articles where only the first few paragraphs of the original text were reproduced in version 2. this occurred when an article was published over multiple urls in the original version – many articles were laid out with a few paragraphs on one html page, with a next button or other navigation that brought the reader to a new url for the next part of the text. even in 2001, this pattern seems to have caused problems for the editors of switch as they migrated to a new platform. we expect to address this problem by adding the additional text to the downloaded html files and regenerating the pdfs of these articles. this is an example of the flexibility afforded by taking a more hands-on approach to digital preservation – while this stage of the process is likely to be labor intensive, it will restore the complete text of the articles that has been missing from the most visible version of the journal for two decades. conclusions as part of a pilot project for a much larger preservation effort focused on the digital art community in the bay area, we hoped that this project would help to establish ways of collaborating that could carry over to the broader project. additionally we believe some of the lessons from this project may also be applicable to similar contexts, especially those where relatively small research libraries, or other smaller cultural heritage institutions, aim to work with creative communities whose work involves technology. at our own institution, we believe the library’s work with the digital arts community provides a template for efforts to support digital scholarship in other fields. challenges there were some unanticipated challenges that we think will be instructive to those who archive web-based art and scholarship. to be straightforward, the closer we looked at either the original switch site, the 2nd version of the journal, or our archived surrogates, the more problems we found – problems with authenticity and missing content, problems with functionality and discoverability, problems with accessibility, and so on. this is not unique to switch by any means. both our previous experience with web archiving and existing studies (ayala, 2020) suggest that these challenges persist across web archiving projects. for our context, the lesson here is that, for content that is critical to the history of a particular institution or community of practice, we should expect to expend significant resources addressing these types of issues during the archiving process, and we should expect that hands-on, bespoke interventions will be required to make that content accessible to new users. in the context of the digital humanities, this project is particularly instructive. in switch, we have an example of web-based digital scholarship that is over 25 years old. the level of intervention that is required to archive this type of material is significant and is not likely to be reduced as the technical complexity of the scholarship increases. importantly, for this project, faithfulness to the published version of the journal was not sufficient to preserve and provide access to the content of the journal. this is not only because of the extensive missing content in version 2 of the published journal, but also because the published journal did not meet current accessibility standards or provide the necessary metadata or structure to make the publication visible as a journal in the modern scholarly communication environment.  web archiving at scale necessarily limits its goals to an accurate reproduction of the published site, but in this case that approach would not meet the goals of either the library or the community of practice that created the journal. more broadly, this project points to the need to consider the value of bibliodiversity to the scholarly communication environment, and the need for libraries to adopt an approach to technology that supports bibliodiversity. (shearer, chan, kuchma, and mounier, 2020) switch is distinctive among scholarly journals in ways that include its integration of subject matter and presentation and its close connection to a particular community of artistic practice, but it is challenging to retain the journal’s distinctive features while also increasing its visibility in the comparatively rigid world of scholarly publishing. this project points to a general approach for preserving both the visual and structural characteristics that make a journal like this distinctive. in particular, the close involvement of practitioners and students in the process of archiving the journal has produced a more authentic version of the publication than the library could have achieved on its own, while making the journal more visible and accessible to users. in response to these challenges, project participants adopted several specific practices that that we hope to carry forward and apply to future projects: flexible balance of hands-on and automated work making archived work visible and legible to the broadest possible audience close connection to communities of practice, including students flexible balance of hands-on and automated work this project began with the intention of automating as much of the process as possible through web crawling and scripting. the initial aspiration of the project was to archive version 2 of switch as it was published, but the input of various collaborators pointed the way toward a more ambitious project. making the archived journal accessible required manual creation of alt text as well as more complex scripts to address contrast issues and simplify the html structure of the harvested pages. student involvement created much richer metadata, grounded in the discipline, and also highlighted issues with version 2 which will require significant hand-work from the library, but will also make the archived switch a much more complete record of the journal. hands-on digital preservation work is often addressed from a position of scarcity, with the implicit assumption that the volume of potential digital content is simply too great to allow labor-intensive intervention on individual objects or files. (corrado, 2022) while library resources are necessarily limited, this project demonstrates how partnering with communities of practice can introduce a different dynamic, both by helping to clearly identify high-value material and by facilitating collaborative labor that can supplement work done within the library. making archived work visible and legible to the broadest possible audience this project treats switch as a journal, and in some cases it imposes a more conventional structure on the most experimental articles in the publication. while a traditional web archiving approach would capture the experience of the original publication more authentically – and would be an appropriate complement to this project – it would not make the journal’s contents easily discoverable and browseable by a general audience. this is another example where close collaboration facilitates a more interventionist approach. migrating the journal to a very different context involved numerous small decisions. on this project library workers and cadre faculty and students were able to discuss and work through those decisions, including during in-class discussions and workshops. this process allowed members of the digital arts community to have consistent input, shaping the archiving process to fit the expectations of the journal’s audience. close collaboration with communities of practice, including students as the previous examples underscore, this project revealed both expected and unanticipated benefits of close collaboration between communities that create content and institutions that take on the responsibility of archiving that content. the reference model for an open archival system (oais) defines the designated community of a digital archive as: an identified group of potential consumers who should be able to understand a particular set of information. the designated community may be composed of multiple user communities. a designated community is defined by the archive and this definition may change over time. (ccsds, 2012, 1-11) this case study suggests an alternate understanding of an archive’s designated community – as a community of collaborators in the archiving process, as well as one of potential consumers of archived information. for this project, the collaborative model influenced decisions large and small. this is an area where approaches and methods developed in the field of community archives might help inform the work of libraries who are active in archiving and facilitating digital scholarship created by faculty and students. (caswell, 2020). finally, this case study suggests that deeply collaborative partnerships between communities of practice and cultural institutions may be a viable model for digital preservation practice. it is important to expand the number and types of institutions that can practice complex digital preservation (blumenthal et al., 2020), and we believe this model is particularly valuable for smaller and less resourced libraries. this approach opens up opportunities to integrate digital preservation and digital scholarship practices into the curriculum, which are particularly valuable at teaching-intensive academic institutions. finding sustainable digital preservation models for small institutions is critical to maintaining the diversity and representativeness of archived digital collections, including the bibliodiversity represented by early electronic journals like switch. we hope that this case study helps to demonstrate the feasibility of this work in a variety of contexts. about the authors nick szydlowski is the scholarly communications & digital scholarship librarian at the martin luther king, jr. library at san josé state university. rhonda holberton is an oakland-based interdisciplinary artist and assistant professor of digital media at san josé state university. erika johnson is the institutional repository and digital scholarship coordinator at the martin luther king, jr. library at san josé state university. endnotes [1] partial code is included for explanatory purposes. the complete code for the project is available at https://github.com/nickszydlowski/switch. sources amiran e, unsworth j. 1991. postmodern culture: publishing in the electronic medium. the public-access computer systems review 2(1):67-76. available from: https://www.ideals.illinois.edu/handle/2142/201. ayala br. 2020. correspondence as the primary measure of quality for web archives: a grounded theory study. in: hall m, mer?un t, risse t, duchateau f, editors. digital libraries for open knowledge. vol. 12246. cham: springer international publishing. (lecture notes in computer science). p. 73–86. doi: 10.1007/978-3-030-54956-5_6. blumenthal k-r, griesinger p, kim jy, peltzman s, steeves v. 2020. what’s wrong with digital stewardship: evaluating the organization of digital preservation programs from practitioners’ perspectives. the journal of contemporary archival studies 7. available from: https://elischolar.library.yale.edu/jcas/vol7/iss1/13/. boice j, c. draper d, b. lunde d, m. schwalm a. 2017. preservation for circulating monographs: assessing and adapting practices for a changing information environment. technical services quarterly 34(4):369–387. doi: 10.1080/07317131.2017.1360640. caswell m. 2018. affective bonds: what community archives can teach mainstream institutions. in: flinn a, bastian ja, editors. community archives, community spaces: heritage, memory and identity. facet. p. 21–40. the consultative committee for space data systems (ccds). 2012. reference model for an open archival information system. washington, dc: ccds. available from: https://public.ccsds.org/pubs/650x0m2.pdf. corrado em. 2022. digital preservation is not just a technology problem. technical services quarterly. 39(2):143–151. doi:10.1080/07317131.2022.2045432. directory of electronic journals, newsletters, and academic discussion lists. 1994. association of research libraries, office of scientific and academic publishing. mcnulty j, alvarez s, langmayr m. 2021. detecting research from an uncurated html archive using semi-supervised machine learning. in: 2021 systems and information engineering design symposium (sieds). p. 1–6. doi: 10.1109/sieds52267.2021.9483725. mogge d. 1999. seven years of tracking electronic publishing: the arl directory of electronic journals, newsletters and academic discussion lists. library hi tech. 17(1):17–25. doi:10.1108/07378839910267154. shearer k, chan l, kuchma i, mounier p. 2020 apr 1. fostering bibliodiversity in scholarly communications: a call for action! doi: 10.5281/zenodo.3752923. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – affinity strings: enterprise data for resource recommendations mission editorial committee process and structure code4lib issue 5, 2008-12-15 affinity strings: enterprise data for resource recommendations the university of minnesota libraries have created a mylibrary portal, with databases and e-journals targeted to users, based on their affiliations. the university’s enterprise authentication system provides an “affinity string”, now used to personalize the mylibrary portal. this affinity string automates discovery of a user’s relationship to the university–describing a user’s academic department and degree program or position at the university. affinity strings also provide the libraries with an anonymized view of resource usage, allowing data collection that respects users’ privacy and lays the groundwork for automated recommendation of relevant resources based on the practices and habits of their peers. by cody hanson, shane nackerud, and kristi jensen the myu portal the myu portal is a campus-wide web portal running on the open-source metadot portal server. created by the university’s office of information technology and office of the vice provost for distributed education and instructional technology, the site is intended as “the first stop for university of minnesota information and applications” for students, staff, and faculty [1]. by bringing together feeds of data from many enterprise applications and filtering them based on a user’s identity, the myu portal provides a personalized dashboard. the interface to the myu portal presents several web pages, each accessible through a tabbed interface. the my courses tab lists details of academic courses a user may be taking or teaching. my work life provides payroll and benefits information for full-time or student employees. access to this sensitive personal information is provided on the basis of user authentication with the university’s enterprise central authentication hub. while the specific information on a tab can be personalized for a user, large parts of the myu portal interface can also be customized for categories of users. for example, incoming students accepted for fall 2008 enrollment have had access to the myu portal since spring. when they log in, they see a “view” of the myu portal that is heavily customized for the class of 2012. once those students choose a major, their portal view will be customized to their specific degree program. undergraduates who log in to the portal see information relevant to undergrads. graduate students see information relevant to grads. more specifically, grad students from the carlson school of management see information relevant to their studies at that particular school. there are many different views available on the portal, and all of them are geared towards specific audiences. this works through a unique piece of metadata, called an affinity string, which is assigned to each university of minnesota user. what is an affinity string? affinity strings are created by the office of information technology using data from the university’s peoplesoft system. everybody at the university of minnesota is assigned one or more affinity strings based on her area of study or work. for example, a ph.d. student in psychology may receive the affinity string: “tc.grad.gs.psy.phd.” figure 1: affinity string breakdown for a psychology grad student the segment “tc” indicates that the student is on the twin cities campus of the university. “grad” denotes that she is a graduate student, “gs” that she is enrolled in the graduate school, “psy” that she is in the psychology department, and “phd” that she is in the doctoral degree program. as another example, a faculty member in the carlson school of management may receive the affinity string “tc.fac.csom.327a.9403,” in which the last two segments correspond to peoplesoft job codes that indicate the faculty member’s department and rank, in this case assistant professor of accounting. there are over 15,000 affinity strings covering all the disciplines and roles at the university. the myu portal managers have mapped each string to a particular portal view, assuring that each user gets information relevant to her department, area of study, or affiliation with the university. mylibrary as acolytes of lorcan dempsey’s notion of placing library services “in the flow” of users, the university’s librarians recognized the traffic on the myu portal as a rich workflow in which it was natural to embed our resources. the libraries lobbied the portal managers for, and were granted, a dedicated tab, which we labeled “mylibrary.” responsibility for planning and development of the mylibrary tab was given to a group of librarians and library it professionals and was overseen by the director (and later the interim director) of the libraries’ digital library development lab. the design of the myu portal limits each tab to a narrow column on a single page. recognizing this constraint, and that users in the myu portal context would expect a high degree of personalization, the mylibrary team sought to prioritize content that would drive users efficiently to resources relevant to their research. in keeping with the modular design of the other tabs on the myu portal, the mylibrary team envisioned a layout that comprised a number of smaller cells or widgets, some of which would be universal to all users and others of which would be customized to particular affinity string groups. for example, all users would be presented with the libraries’ search box, which initially provided a box to search mncat (our aleph opac), our sfx e-journals catalog, and our metalib federated search. figure 2: the mylibrary tab on the myu portal. click to see full images of the undergraduate and graduate student mylibrary tabs affinity strings for resource recommendations to further personalize mylibrary, we took advantage of affinity strings. the myu portal publishing system does not allow live database queries, instead requiring that we publish flat, cached data. it quickly became clear that we would not be able to publish pages completely customized for each of the 15,000 affinity strings. instead, we created several templates: one for undergraduates; one for graduate students and faculty in the academic health center; one for most other graduate students and faculty; and one for users who were assigned affinity strings and were able to access the myu portal, but not authorized to access library databases. each template contained elements common to all users, and spaces for elements customized by affinity string. figure 3: interface for assigning affinity strings to research quickstart subjects in libdata long before the university libraries began work on mylibrary, we created libdata [2], an open-source web application that provides an authoring and publishing environment for subject pathfinders and other web pages. one feature of libdata is research quickstart, a set of database-driven subject guides maintained by subject specialists and published in both html and xml formats. the mylibrary team determined that the simplest and most sustainable method for providing relevant information in the myu portal would be to match a user’s affinity string to one of our approximately 250 existing subject guides. for example, by mapping the affinity strings for faculty and graduate students in psychology to our psychology research quickstart guide, we were able to present those users with a widget featuring recommended databases and indexes. figure 4: resource recommendation widget for a user with a psychology affinity string the process of matching affinity strings to templates and then to existing research quickstart subjects required significant work on the part of the mylibrary team. the libraries’ digital library development lab created an interface in libdata that made mapping all “.grad.” strings to a particular template relatively simple, but mapping every affinity string to subject content proved much more work-intensive. the earliest affinity strings available to the mylibrary team provided extensive information about graduate students at the university. although we analyzed almost 800 separate affinity strings assigned to more than 38,000 people, only a portion of those represented “active” graduate students. approximately 230 affinity strings represented the 12,000 individual graduate students then enrolled at the university. (note that this number did not represent graduate students in the academic health center and therefore excluded some graduate student groups.) the remaining strings represented students who had either completed their degree or those who had not finished a degree and were not currently enrolled. although this latter group did not have access to licensed library resources, they did have access to the portal and could be provided with a mylibrary tab. neither affinity strings, research quickstart guides, nor the myu portal were created with the use described in this article in mind. using these pre-existing systems to recommend resources to users has required some compromises. our mechanism for publishing content to the myu portal required us to build each possible page in advance and cache them on a server rather than building them dynamically when a user visited the portal. to minimize complexity and maximize the efficiency of this process, we limited ourselves to a one-to-one relationship between research quickstart guides and affinity strings, presenting each user with a single set of recommendations. unfortunately, not every degree program at the university matches cleanly with a single existing research quickstart subject. in some cases our subject guides were too broad to adequately reflect the specific focus of an academic department. also, some departments span more than one subject area. for example, the italian language program no longer exists by itself but was subsumed by the french & italian studies department. students and faculty in this department now share the same department designation in their affinity string. because we could only map a single research quickstart guide to this string, the libraries had to either create a combined subject guide, or fall back on a broader set of resource recommendations. the mylibrary team ultimately concluded that while affinity strings often provided a sufficient basis for determining the research needs of graduate students and faculty, they were a poor basis for providing resource recommendations to undergraduate students. until an undergraduate student has chosen a major, his affinity string reflects only his class year status, providing no insight into his areas of academic interest. likewise, even when an undergraduate student has chosen a major, it is safe to assume that he will continue to have coursework outside of that department. we therefore overhauled the undergraduate mylibrary template in the winter of 2008, and rather than recommendations of resources based on affinity string, mylibrary now offers recommendations based on each course in which an undergraduate is currently enrolled, showing on a semester-by-semester basis the resources which are most likely to be useful. presenting recommended resources based on course enrollment required a new mapping strategy. like we had done with affinity strings, the mylibrary team mapped nearly 600 course designators, e.g., acct for accounting or geog for geography, on a one-to-one basis to existing research quickstart pages. as with the affinity strings, we found that some of our existing research quickstart subjects were insufficiently specific for good matches to course designators. for example, we had no extant adult education research quickstart page to match with the aded course designator. in some cases we have created new research quickstart pages, and in others, as in the image below, we have simply mapped the designator to a more general research quickstart page, in this case, education. despite these limitations, user feedback on the resources provided has been positive. figure 5: course subjects mapped to research quick start subjects affinity strings for data gathering as the university libraries became better versed in the use of enterprise affinity strings as a means for providing users with a personalized view of our services, it became clear that these strings could also be used to aggregate electronic resource usage data. while we would be hesitant to track an individual’s use of databases or e-journals, we can look for her affinity string when she accesses one of our resources and generalize on that basis. the university of minnesota is fortunate to have a central system for online authentication. every student, faculty member, and staff member has a unique internet id which is used to access a variety of systems, ranging from email to course registration to licensed library resources. known as the central authentication hub (cah), this service has been adopted by the vast majority of campus systems requiring authentication. the cah features a common interface for entering a university internet id and password. when a user attempts to enter a system that requires authentication, he is first forwarded to a cah web page and asked for his internet id and password. upon authenticating, a secure, encrypted cookie is placed on the user’s browser. this secure cookie is a unique, session-specific hash string containing validation credentials. the user is then sent back to the original application, which uses the secure cookie to query the university’s validation service. this service decrypts the cookie and sends back any authorization data requested by the originating application. if the user has the proper validation credentials, he is allowed into the application. if the user does not close his browser or explicitly choose to log out of the system, the original cah cookie can be used repeatedly by university applications without requiring the user to log in again. the university libraries have long used the cah to control access to licensed resources. the libraries look for unique authorization data in the form of “library flags,” which indicate that a user meets the conditions to be granted borrowing privileges and access to our indexes and databases. the cah can also return other user-specific data, including address, status (i.e., staff, undergrad, grad, faculty), library card number, and departmental affiliation. in addition, the cah can return a user’s affinity string. each time a user with an active cah cookie accesses a library resource, we can capture his affinity string and associate it with the resource being requested. we thus gain valuable insight into which of our user populations are using (or not using) a given resource. by aggregating the activity of a number of users with the same affinity string, we can sufficiently anonymize our data to protect user privacy. figure 6: example of a report on the affinity strings that have accessed academic search premier most frequently affinity string tracking in practice the university libraries have been capturing resource usage by affinity strings since the end of march 2008. to date, we have captured over one million rows of data. these data do not, however, represent 100% of the usage of our electronic resources. we are able to capture an affinity string only when a user accessing our resources has a live cookie from the cah. many users on campus are able to access our licensed resources directly via ip address authentication, which does not require authentication through the cah. that said, many applications at the university require cah authentication and are not ip authenticated. we are able to read affinity strings on a cah cookie regardless of whether that cookie was issued when a user sought to access a library resource. by comparing our affinity string tracking data with other resource traffic data, we estimate that we are capturing 60-70% of the usage of our resources. in any analysis of the data we capture, we must consider this shortfall, as it may introduce significant bias. perhaps, for example, undergraduate users are more likely than other populations to access resources from campus ip addresses and as such may be underrepresented in our data. this tracking project is in its infancy, and the university libraries have not yet begun to analyze the data collected. however, we envision a broad range of potential applications: scenario 1: purchasing decisions the university libraries spend thousands of dollars licensing electronic material every year. until this year we could only track usage of these resources through hit counts. with tracking based on affinity strings we could determine if the intended university audience is using a resource. selectors could see at a glance which affinity strings are most heavily using a resource they have purchased. perhaps more importantly, a selector could see if the intended audience is not using a costly resource, perhaps indicating an opportunity for user education, or that those purchasing dollars would be better spent on another resource. scenario 2: resource recommendations the thousands of data points the university libraries collect on resource usage by affinity string could form the basis for a recommendation system. much as we currently present users with resource recommendations based on our research quickstart subject guides, we could recommend resources being heavily used by users’ colleagues. we could indicate to incoming graduate students the databases and e-journals most often used by the faculty and other graduate students in their departments. scenario 3: resource recommendations in other fields the university of minnesota, like many institutions, is focusing intently on interdisciplinary research. scholars working at the boundaries of the discipline in which they have been trained can find the prospect of researching in a new field daunting. our affinity string data could provide easy access to top resources used by researchers in that new field. affinity string data could also be used to provide undergraduates a quick look into the research ecosystem of a new subject. when an undergraduate is faced with the inevitable need to cite one or two scholarly sources in a paper, the libraries could provide direct links to scholarly journals that are heavily used by graduate students and faculty at the university in a particular field. scenario 4: interdisciplinary connections given a sufficiently large data set, affinity string data could be mined to reveal groups of affinity strings using the same or similar library resources. the university libraries could then receive early indications that researchers in different departments are working toward similar goals and, ideally, sow the seeds of interdisciplinary collaboration. while it is unlikely that affinity string data at the database and journal level will uncover many connections among disciplines that aren’t well known to researchers in those fields, the data could be used to create an “interdisciplinary map” of research at the university. scenario 5: marketing and promotion affinity string data can show librarians which affinity string groups are using the libraries the most, as well as which groups are using the libraries the least. with this data we could target our services more effectively to heavy library users or direct outreach to groups we may have failed to reach. scenario 6: library budget and staffing decisions affinity string tracking could for the first time measure what percentage of electronic resource usage is by a given user population. this has the potential to inform the libraries’ decisions on staffing and other budgeting. if, as our early data seem to indicate, use of our electronic resources is dominated by users from the university’s academic health center (ahc), can we justify investment in tools or development that would benefit ahc users exclusively? alternatively, can we presume that strong use of electronic resources by ahc users bespeaks comfort and self-sufficiency, and should we redirect staffing and instruction to other user populations in an attempt to increase their usage of our resources? before we made any such drastic decisions based on affinity string data, we would need to establish great confidence in the validity and significance of the data. privacy considerations monitoring library user behavior and the long-term storage of resulting data strike fear into the hearts of most librarians. we believe, however, that by capturing only a user’s affinity string and aggregating the usage data of all users with that string, we can ensure privacy. members of the mylibrary team and the university libraries’ associate university librarian for information technology met recently with a representative from the university’s office of general counsel to discuss the legal and privacy implications of affinity string tracking, and of the publication of affinity string data back to users. we were told that we could be justified in collecting affinity string data and in publishing back to users any data aggregated from an affinity string with at least five members. we are inclined at this point to use a significantly higher threshold, restricting our use of affinity string usage data to those strings that have perhaps fifteen or twenty members. we intend to make our decision based on our perception of the value we can derive from those strings with small user populations. future directions the university libraries have yet to extensively analyze the data we have collected using affinity strings. in addition to the possible uses outlined above, we expect that there are greater opportunities for deriving value from these data. though we’ve captured hundreds of thousands of data points, experience has taught us that we can never have too much data, especially as we look to automated recommendation services. in the future we hope to collaborate with other institutions in a position to collect similar information, so that we can standardize and pool our data. to date we have limited our data capture to the database and e-journal level. however, there are few, if any, technical barriers to extending our tracking to the opac and article level, where that data is available. tracking usage patterns for individual works would provide far richer data and fodder for recommendations than our current system. we’re aware that affinity strings are clumsy, if convenient, indicators of an individual’s true area of interest. research evolves far more rapidly than does a university’s administrative structure, and being paid through a given department is not necessarily a good indicator that a researcher’s work lies completely within the traditional bounds of that department. to provide truly useful recommendations we may need to collect more detailed data. we have discussed the idea of creating a system through which users could opt in to individualized tracking. we could then answer some of the questions we get from our users along the lines of “can you tell me the name of that book i checked out last june?” or, “i was reading an article last week; i think the authors name was something like…” the data we could collect through such a system would increase the fidelity of our recommendation systems, allowing us to base recommendations not on institutional affiliation, but on an identity derived from interaction with the libraries’ resources. conclusion we have just begun to explore the possibilities for data gathering and data mining by tracking usage of our electronic resources by affinity strings. we expect that aspects of our efforts will prove controversial with some of our users and colleagues. however, we believe that we have unprecedented opportunities to create a host of new data-driven library practices. affinity strings are a remarkable tool for segmenting our user base. by limiting our tracking to strings with a sufficient population, we can gather valuable data while protecting the privacy of our users. acknowledgements the tools, systems, and processes described in this article are the products of the hard work of many of our colleagues on the libraries’ portal team. we would especially like to acknowledge paul bramscher, john butler, and kevin o’rourke and our graduate student programmers tariq islam and ankur sharma. notes [1] https://umconnect.umn.edu/myuorient/ [2] libdata is a library oriented web based application which provides authoring environments for subject pathfinders (research quickstart), course related pages (courselib) and general purpose web pages (pagescribe). for more information, see the libdata.sourceforge.net. about the authors cody hanson is technology librarian in the coordinated educational services department of the university of minnesota libraries. he serves on the university libraries’ web services steering committee and is a member of the mylibrary team. shane nackerud is web coordinator and interim co-director of the digital library development lab at the university of minnesota libraries. he co-chairs the university libraries’ web services steering committee and chairs the mylibrary team. kristi jensen is the map librarian and head of the john r. borchert map library at the university of minnesota libraries. she is a member of the mylibrary team. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – streamlining book requests with chrome mission editorial committee process and structure code4lib issue 30, 2015-10-15 streamlining book requests with chrome this article starts by examining why a chrome extension was desired and how we saw it making the workflow for requesting new items both easier and more accurate. we then go on to outline how we constructed our extension, looking at the folder structure, third party scripts and services that combine to make this achievable. finally, the article looks at how the extension is regulated and plans for future development. by dr. rachel schulkins and joseph schulkins the chrome extension – an introduction the idea for the book request extension developed after a team meeting, in which several possibilities were raised to help standardise the format of book requests reaching acquisitions from in-house colleagues, primarily the academic liaison. the problem was that there was no clear format for sending book requests which resulted in disparate and insufficient information for purchasing. some of the emails reaching acquisitions contained unnecessary verbalism whereas others failed to provide basic information, all of which hindered the process. to address these, acquisitions needed a tool that would be able to provide us with all the information we needed to process requests as efficiently and quickly as possible, while also speeding the process for the librarians submitting the request. the initial idea was for librarians to fill out a pdf form and email it as an attachment to acquisitions. we concluded the meeting with the intention of looking into replacing the pdf with an online form. but it occurred to us that even the idea of an online form can be improved on. why not create some sort of an online mechanism that would automatically gather a book’s metadata, search that book against our current library holdings, and finally send the book details to the acquisitions team to purchase, thus making the process more efficient and accurate . we developed the chrome extension to specifically work with amazon.co.uk. we selected amazon.co.uk because amazon lists the majority of books published worldwide. whether it is an out of print book or a recently published manuscript, amazon will likely have the book’s metadata which can then be imported using the extension. the book request process established with the extension is quite simple. once a librarian downloads the extension onto his/her chrome browser, the extension becomes a primary point of contact for submitting requests. when a librarian wishes to submit a book request, s/he searches amazon.co.uk for the book. once the librarian locates the book, s/he clicks on the extension, which then gathers the book’s metadata and automatically populates the book request form with the following details: title, author, isbn, edition, publisher, year of publication. this leaves the librarian with the responsibility of providing the following criteria: format, department/fund, location, requesting party, and reservation details. the need for typing in any of the information is minimal as the majority of the fields are made up of dropdown menus. though designed with amazon.co.uk in mind, the extension is compatible with other online bookstores such as abe.com and blackwell’s (http://bookshop.blackwell.co.uk/) and it can even extract some data from publishers’ websites. the disadvantage of working with a different website other than amazon.co.uk is that the extension cannot extract the books’ metadata in a single click, which means the librarians will have to fill the form manually. nevertheless, the fact that the request form embedded in the extension is a mouse click away makes it easily accessible to users. another major advantage of working with the extension is related to its ability to check the book’s isbn against existing stock items in the library, prior to purchasing. whereas in the past, the librarians had to manually check every book request reaching their inbox against our library holdings, this task is now being performed by the extension. the librarians see a message alert notifying them whether a book is already available in the library or whether the extension could not perform the check (this is what usually happens when the extension is used on websites other than amazon.co.uk). the same message is displayed in the email sent to acquisitions, who can then follow the necessary process to complete the request depending on the book’s status. this has proven to be a great time saver for librarians, who now have a single point of contact where they can perform all the necessary steps prior to submitting a book request: check book requests against existing holdings; extract the books’ metadata and populate the book request; then submit their request to acquisitions. getting the librarians to test the extension was quite easy as they could see its immediate benefits. within three months the extension became the main tool for submitting book request across the library. it was adopted by the academic liaison team, collection management, as well as the customer service team. having the majority of book requests reaching acquisitions go through the extension allows for important data to be collected. prior to the extension there was no way of monitoring requests live. instead, we relied on our library management systems to run lists a few times a week according to different variables. now, the extension collects and analyzes incoming data live. every request submitted using the extension is stored on the wufoo data page and the information gathered is automatically analyzed and presented in various charts and graphs to reflect fund usage, location of material (across several sites on campus) and book format (print or ebook). the data gathered can then be used to analyze efficiency, best practices and to monitor trends across the academic year. for example, the data allows one to check the amount of requests submitted in a given time and to see how many orders were placed in the same amount of time. it allows us to see the time gap between request, execution and reception of material, and adjust and amend departmental procedures to maximize productivity and improve customer satisfaction. design and implementation a chrome extension is essentially a collection of javascript, css, and html that you can add to the chrome browser. these little pieces of software allow you to add functionality or change the way chrome behaves. extensions published to the chrome store can then be downloaded by either a select few that the developer allows or by the wider chrome community. given that the only requirements are an understanding of javascript, css, html and the subtleties of the development process, an extension can be manufactured relatively quickly and painlessly. that said, the hardest part in developing the extension was the flow of communication between the extension and the open window; however the official documentation [1] covers this well as do numerous excellent tutorials. indeed, the official documentation and the tutorial by gabe berke-williams ‘how to make a chrome extension’ [2] provided the basis upon which our extension was developed. our extension consists of: manifest.json — this has the settings of the extension such as location of background scripts and the default actions of the extension src/bg/background.js — used to handle messages returned from the active page to the extension: /** * a background listener waiting for the extension to be triggered * @param object message sent by src/inject/inject.js */ chrome.runtime.onmessage.addlistener(function(message,sender,sendresponse){ /** * set bibdetails as result of processing the returned message * the dataprocess function uses regular expressions to * chop up the message into the relevant bibliographic * parts * @type object */ var bibdetails = dataprocess(message); /** * function to check our catalogue against the submitted * bibliographic details and then populate the form using * the bibdetails object */ checkcat(bibdetails); }); /** * insert the inject scripts into the current window */ chrome.tabs.executescript(null, {file:"/src/inject/jquery.min.js"}); chrome.tabs.executescript(null, {file:"/src/inject/inject.js"}); figure 1: a snippet of our background.js showing how it injects scripts and handles the returned message. src/browser_action/browser_action.html — this html page controls what the user sees when they click on the extension and provides links to javascript and css resources that are to be used to process the data retrieved from the active page src/inject/inject.js — used to communicate between the extension and the active page using chrome’s sendmessage method: /** * set some basic variables to capture, taking a broad stab at where the * title is located. */ var title, url, pub, isbn; title = jquery('title').text(); pub = jquery('#detail_bullets_id ul li:contains("publisher")').text(); isbn = jquery('#detail_bullets_id ul li:contains("isbn-13")').text(); /** * get the current url. typically this will be amazon.co.uk, but having this means users * can request from elsewhere and someone in our acquisitions team * can go and view the item directly */ url = window.location.href; /** * send the details from the webpage to the handling script to populate the form */ chrome.runtime.sendmessage({method:'settitle', title: title, url: url, pub: pub, isbn: isbn}); figure 2: the inject script that sends information about the current page back to the extension. this is triggered when the extension button is clicked. src/inject/jquery.min.js — to add jquery functionality (if it doesn’t exist) to the active page making it easier to access elements css directory — has the bootstrap and alpacajs css files js directory — has alpacajs (a javascript form which provides excellent functionality) [3], bootstrap, isbnjs (a javascript library that validates isbns and allows conversion between 10-13 digit) [4] fonts directory — contains bootstrap glyphicons icons directory — contains png’s of the plugin’s icon at various sizes to be used by the browser a core part of the extension is the form it uses to capture the metadata and crucially we would then need to find a way to store any of the data captured by the form, develop some statistical reporting and set up email alerts of when new items were submitted. whilst other tools may be available, having previously used wufoo [5] forms joseph knew that it could handle all of this quite comfortably. when we created the form we did so exactly as we wanted it for the chrome extension with an idea that the form would be embedded. but as it turned out this approach was impractical and instead we made use of the wufoo api. knowing that we would use wufoo via their api, we settled on alpacajs as a suitable front end alternative for the form and it would be this that would be filled in when the user clicked on the extension. alpacajs uses jquery [6] and json to arrange the form elements and is extremely extensible; not only does it lend itself well to validation and conditional processing, but it will allow you to define custom functions and callbacks. whilst alpacajs is good in that it allows for great control of the form, there are a few pinch points that joseph encountered when using it and the official documentation was not very helpful. in the end it took several hours combing sites such as stackoverflow [7] to find help or solutions that joseph could work into our form. so with the front end form constructed and the wufoo form in place to receive entries, we needed something that would be able to send the information to the wufoo api and to receive back any response. as such we pointed the customised alpacajs form to a php script. this script has our api key, api endpoint and maps each entry from the form to a variable and, using curl, this information is sent to wufoo. if a ‘success’ message is received by the curl function, then the php script displays a success message in the chrome plugin, however should there be an error, the information regarding the curl error and the attempted values would be sent to joseph with a message saying that there had been a problem displayed to the user. before the plugin, it was standard practice for those both requesting and processing the requests to check each item in the catalogue to see if we already had a copy. if the item was found to be in the catalogue and the person requesting the item hadn’t checked or indicated the reasons why extra copies were needed, then this would spark a back and forth conversation (usually over email). as such joseph was asked whether an automatic catalog check could be built into the form. again, using php joseph was able to create a quick and simple script that received the isbn as a get request and queried our mobile catalogue for results. if the search failed (i.e. nothing with that isbn was found) an html page would be loaded that had content wrapped in a class of ‘errorarea’ and using domxpath() (since getelementsbyclassname doesn’t exist in domdocument), the script would be able to see whether the checked isbn existed or not. with the existence of this item now set to either ‘true’ or ‘false’, the php script would return this as json so that the extension could parse it. with a response of true, the extension would then inform the user that the item already existed and append this information to the notes field. because of this all parties are aware if there is an item being requested that is already in the catalogue and it allows the person submitting the request to add any additional comments. this early communication helps eliminate the unnecessary email conversations. one great piece of feedback we have received about this particular feature was from a librarian who whilst placing a request was alerted that we already had an item that he was ordering in stock. since he was fairly sure nothing with this title or authors existed in our catalogue he searched and found that we did have this particular item already, but that the title had changed slightly and so too had the authors. /** * an isbn is sent as a get request to this script and we will return json */ header('content-type: application/json'); $isbn = isset($_get['isbn'])?$_get['isbn']:'error'; /** * did we capture the isbn from the webpage or has it been set to false */ if($isbn == 'error' || $isbn == false){ echo '{"exists":"error"}'; } else{ /** * our mobile catalog will be used to search against as * this has less styling elements to navigate */ $url = 'http://m.library.liv.ac.uk/search/?searchtype=y&searcharg='.$isbn; /** * initialize a curl object */ $ch = curl_init(); /** * set url and other options */ curl_setopt($ch, curlopt_url, $url); curl_setopt($ch, curlopt_returntransfer, 1); /** * get the page contents */ $output = curl_exec($ch); /** * close curl resource to free up system resources */ curl_close($ch); /** * create a dom parser object */ $dom = new domdocument(); /** * the @ before the method call suppresses any warnings that * loadhtml might throw because of invalid html in the page. */ @$dom->loadhtml($output); /** * getelementsbyclassname doesn't exist in domdocument so use xpath * to find the error section if applicable */ $xpath = new domxpath($dom); $error_area = $xpath->query('//p[@class="errorarea"]'); /** * if there is an errorarea, then we don't have the item */ if($error_area->length > 0) { $exist = "false"; } else { $exist = "true"; } /* * return the exist variable wrapped in json */ echo '{"exists":"'.$exist.'"}'; } figure 3: our catalog lookup script. from the start we wanted this extension to behave like any other on chrome extension in that if/when updates were needed, we wouldn’t have to ask those who had installed it to re-install a new version (best case scenario) or have to re-install the new version ourselves on everyone’s chrome (worst case scenario). the only way to have an extension that updates automatically is to add it to the chrome web store. this has been a security change to prevent extensions from loading malicious software into the browser. you can still load extensions (albeit with the caveat that they won’t update) into your version of chrome by enabling ‘developer mode’ on the extensions page of chrome (chrome://extensions), and this will enable you to upload a zip file of your extension. to publish to the webstore [8] you will need to have an account and pay a one time fee. before publishing though, joseph found that we could set the extension as private (meaning it wouldn’t be listed in the web store) but we could also either a) specify individuals to grant access to (an individual gmail account would be required) or b) allow access to a group of users in google groups. for our situation the second one seemed ideal as it would mean we could administer access to the group independently of the extension and, since not all our staff have google accounts, provide an ingress to the extension via their university email. the submission process is relatively straightforward, first requiring you to upload an extension as a zip and then, through a form, supplying information about the plugin and its audience. once you have completed these steps and selected to publish to your private group/individuals, you are redirected to the landing page of your extension. copy the url as this is what you will need to send to your testers so they can load the extension. the extension will then take an hour or so to become available for download. one important thing to note, when you are creating the zip either for the first time or for subsequent updates, you will need to make sure that the folder is zipped without any extra or hidden files and folders as these can sometimes cause errors loading or running the extension. conclusion to conclude, the extension has proven to be popular and its implementation a success for standardising book requests reaching the acquisitions department. at the moment, it is an in-house tool to allow departments in the library to communicate their requests to the acquisitions team. however, the future objective is to develop the extension for students and academic staff outside the library. in the next academic year we will be developing the extension for other browsers and also explore the possibility of extending the concept to develop a mobile app, to help enhance user experience and ease accessibility to material staff and students require for their research. further, we will look to create specific webpage filters that will match the requesting url and adopt the appropriate filter to broaden the range of pages the extension can capture data from. in particular we will look at additional booksellers and high use publishers. to simplify the ordering process, we will add to the notification emails to the acquisitions team, links to our suppliers formed of openurls and the captured isbn(s) to directly link to the requested item. notes [1] what are extensions? [internet]: google [cited 2015 03/15]. available from: https://developer.chrome.com/extensions [2] how to make a chrome extension [internet]: berke-williams, gabe [cited 2015 03/15]. available from: https://robots.thoughtbot.com/how-to-make-a-chrome-extension [3] alpaca – easy forms for jquery [internet]: gitana software, inc. [cited 2015 03/18]. available from: http://www.alpacajs.org/ [4] isbnjs: an isbn javascript library [internet]. google project hosting [cited 2015 04/09]. available from: https://code.google.com/p/isbnjs/ [5] wufoo [internet]: wufoo [cited 2015 04/15]. available from: https://www.wufoo.com/ [6] jquery [internet]: jquery [cited 2015 04/10]. available from: http://jquery.com/ [7] alpaca javascript dom update [internet]: stackoverflow [cited 2015 06/02]. available from: http://stackoverflow.com/questions/24146694/alpaca-javascript-dom-update [8] publishing your app [internet]: google [cited 2015 06/09]. available from: https://developer.chrome.com/webstore/publish about the authors dr. rachel schulkins holds a phd in english literature from the university of liverpool and currently works as a principal library assistant. joseph schulkins (joesch@liv.ac.uk) has been the systems librarian at the university of liverpool since 2007. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – using vufind, xampp, and flash drives to build an offline library catalog for use in a liberal arts in prison program mission editorial committee process and structure code4lib issue 16, 2012-02-03 using vufind, xampp, and flash drives to build an offline library catalog for use in a liberal arts in prison program when grinnell college expanded its liberal arts in prison program to include the first year of college program in the newton correctional facility, the grinnell college libraries needed to find a way to support the research needs of inmates who had no access to the internet. the library used vufind running on xampp installed on flash drives to provide access to the libraries’ catalog. once the student identified a book, it would be delivered from the libraries to students on request. this article describes the process of getting vufind operating in an environment with no internet access and limited control of the computing environment. by julia bauder why build an offline library catalog? grinnell college, a highly selective liberal arts institution with a long social justice tradition, has been offering educational opportunities to inmates in local prisons for several years as part of its liberal arts in prison program. in 2009 the college decided to expand this initiative by offering the first year of college program in the newton correctional facility, a medium-security prison near the campus. selected inmates take a series of credit-bearing courses equivalent to those taken by first-year students at grinnell. the incarcerated students typically do not take a full load of courses per semester, so it takes each student about two years to finish the first year of college program. the program features small classes, many of them writing-intensive, across a range of disciplines in the sciences, social sciences, arts and humanities. the college’s decision to launch this program placed the grinnell college libraries in a difficult situation. first-year students in many classes at grinnell college are expected to do library research and to write research papers or annotated bibliographies, and now that online library catalogs and databases have replaced card catalogs and printed indexes, it is nearly impossible to do that sort of research without internet access. yet, the inmates at the prison are not allowed to access the internet.  the libraries needed to find some way of providing the incarcerated students with an authentic library research experience that did not rely on online databases, online library catalogs, or visits to the college’s physical libraries. replicating the online library catalog in an offline environment, and then delivering books from the libraries to the prison on request, seemed like the most feasible option. copyright law and licensing terms would have prevented us from creating offline facsimiles of our subscription databases, but the library catalog, full of records that we owned and could reuse in another environment, was fair game legally, and, with the many free and open-source options available, replicating it offline was plausible technologically and financially. [1] i originally developed the offline library catalog in the fall of 2009 for a biotechnology class that was to be taught in the spring of 2010. the original version of the catalog used the then-current versions of xampp and vufind: 1.7.2 and 1.0 release candidate (rc) 2, respectively. however, due to factors beyond the control of the libraries, the class was canceled less than two weeks before it was to begin. thus, the offline catalog was shelved for almost two years (literally, the external hard drive containing the fully-functional offline catalog sat on a shelf), until the fall of 2011 when another course was offered at the prison that required library research. in the summer of 2011, after i was informed about the class that would need to use the offline catalog, i upgraded the offline catalog to xampp 1.7.7. however, since the version of the catalog based on vufind 1.0 rc2 was already working well for our purposes, i did not upgrade it to the current version of vufind (1.2). i did subsequently experiment with modifying the current version of vufind to work in an offline environment, and it also seems to run well from a usb drive once modified. why vufind and xampp? xampp is a “web server in a box”: apache, mysql, and php (plus some other components), all in one convenient, easy-to-install, pre-configured, freely-licensed package. [2] it is often used as a sandbox for testing out software on one’s desktop computer. xampp often serves an easier and safer alternative to installing the software on an actual web server, but it can (with caution and after securing many options that are insecure by default) be used to run production software. xampp also offers a “usb lite” version (formerly just the “lite” version) designed for use on portable media, which was extremely helpful for our purposes. vufind was chosen over blacklight because most of the open-source software currently run by the grinnell college libraries is in php (and none of it is in ruby), so there was pre-existing staff experience with the language. also, i was confident that vufind would run on xampp (which was clearly going to be a critical component for carrying out this project with the minimum amount of work), because another vufind user, greg pendlebury, had already added advice on the vufind installation wiki about how to install vufind on xampp. [3] however, for institutions with a reason to prefer using blacklight, it should presumably be possible to modify that discovery layer to work in an offline environment in roughly the same way that we modified vufind. why flash drives? the original prototype of the offline catalog was created on an external hard drive, but we decided to copy the catalog to flash drives for use in the prison for cost-saving reasons: flash drives are cheaper, plus they are easier to transport and store. be aware that, depending on security settings, windows may not allow xampp to run from a flash drive, or even in some cases from an external hard drive. the security settings on the computers in the prison lab, luckily, allowed the catalog to run from flash drives. we originally decided to install the catalog on external drives for procedural rather than technological reasons. grinnell’s liberal arts in prison program does not administer the computer lab that it uses in the prison; the lab is managed by it staff from des moines area community college (dmacc), which runs other educational programs for prisoners. installing the offline catalog on external drives required less coordination with the it staff at dmacc, and it would allow the drives to be brought back to campus periodically for software and data updates. however, we have experimented with copying the catalog onto the desktop, since it seems to run slightly faster after doing so. the prison program was recently permitted to place several grinnell college-owned computers in the prison’s library; the offline catalog will be run from the desktop on these computers in future semesters. i briefly considered trying to create an offline catalog that could be burned to dvd-rom, which would have made creating multiple copies of the catalog even cheaper. it also would have been more robust, since we wouldn’t need to worry about students accidentally deleting files or otherwise altering the catalog. however, re-writing the vufind code to work when operated from read-only media would have required more time than i was able to devote to the project. how to do it the easy part: installing vufind the instructions on the vufind installation wiki can be followed mostly as written, with a few exceptions. instead of installing apache, mysql, and php, install xampp usb lite [4], which includes all three components and configures them correctly, on the external drive. smarty (the templating system used by vufind) can be installed as directed on the vufind installation wiki, following the advice there for installing smarty with xampp. the jdk (java development kit) can also be downloaded and installed as normal, with a few exceptions. be sure not to allow the installation script to install the public jre (java runtime environment). (letting the script install the public jre won’t affect the functioning of the catalog, but it will cause problems on your own computer later, when you’ve removed the external hard drive and the jre is no longer accessible to your computer.) also, the jdk should be installed to the flash drive or external hard drive rather than to the internal hard drive, and it is not necessary to set java_home as directed in the vufind installation instructions. (java_home and other environment variables will be set dynamically by the scripts used to launch vufind, since the drive letter for the flash drive may change each time the catalog is run.) the final step of the jdk installation instructions on the vufind wiki is adding the paths to php and pear to the system path variable. making this change is necessary for the vufind installation batch file to run as intended, since the batch file installs a number of pear packages. these paths should be removed from the system path variable after the installation script is run, since php and pear on the external drive will not be a permanent part of your system. the configuration files must be edited as directed on the vufind installation wiki, except they need to be relative (without drive letters), since the drive letter for the flash drive may change each time the catalog is run. launching and shutting down vufind since vufind will be launched by the user, and may be launched from a different location each time, some changes are needed to the scripts that launch it. i moved the script used to start the catalog (vufind.bat, which is created by the vufind installation script) out of the vufind folder and into the root directory of the flash drive to make it easier for users to find (and to allow us to make the vufind folder hidden) and renamed it “start vufind – run once to start vufind.bat.” i added four lines to this script: two to launch xampp and two to set the locations of vufind_home and java_home dynamically based on the current drive letter of the flash drive. this was done using batch parameters, a feature of windows batch files that will insert various pieces of environmental information into batch files when they are run.  using the batch parameters %~d0 (which will be replaced by the drive letter when the script is run; this works well if the offline catalog will only be run from external drives) or %~dp0 (which will be replaced by the full path to the folder from which the script is run; this is needed if the catalog may be copied onto the computer’s desktop), the script can set the paths to java and to vufind accordingly. [5] set vufind_home=%~d0\vufind // if the catalog will only be run from external drives set java_home=%~dp0jdk // gives the option of copying the catalog to the desktop the scripts used to import records also must be edited in a similar fashion to set vufind_home and java_home. interface changes to make the vufind interface work in a non-networked environment, most features that rely on internet connectivity to function, such as course reserve lists, new item lists, ask a librarian, book covers from external providers, email this search, text this, etc., must either be turned off in configuration files or links to them should be commented out in the web interface. other features such as creating an account, adding to favorites, adding tags, saving searches, etc. are also problematic in this environment—they will technically work, but since they only exist on one specific copy of the catalog, they are not very helpful—and should also be commented out or deleted. the hard part: call numbers and locations vufind normally pulls call number and location information from the catalog in real time, but simply commenting out the call number and location was not an ideal solution. we wanted the incarcerated students to be able to copy the call number and location information to the paging slips that the libraries created so that college staff would not have to look up the call number and location of each book when pulling the items. also, some of the reference and instruction librarians at grinnell (myself included) like to use the library of congress classification as a teaching tool, to talk about disciplinary differences in perspectives on a given topic and clues as to which disciplinary perspective a book is coming from. since we intended to go to the prison from time to time to provide information literacy instruction to the incarcerated students, i wanted the books’ call numbers to be visible in the offline catalog for this purpose as well. thus, i edited vufind’s code to force it to pull the call number and the location from the marc record, which is stored in its entirety in the solr index. in the vufind 1.0 rc2-based catalog that was actually used in the prison this semester, these changes were made in several different template files; in version 1.2, all of the changes can be made fairly easily in the marc record driver, although in the long run it would probably be easier to create and maintain a whole new pseudo-ils driver that connects with the solr index rather than with an ils. (the marc record driver is somewhat more likely to change in new versions of vufind in ways that would require re-writing the changed code, while the basic functions within ils drivers should not typically need to be modified when upgrading to new versions of vufind.) when using version 1.2, i set the sample driver as our driver. this driver, which was originally intended for non-production purposes only, supplies hard-coded dummy information for many possible requests. (e.g., it will always inform users that there are no new items and nothing on reserve.) for our intended uses of the offline-catalog, this dummy information was mostly sufficient, except for the above-noted call number and location information. to pull this information from the marc record, i copied the getstatus function from the sample driver to the marc record driver (allowing it to take advantage of the many marc-specific functions included in that class), and edited it as follows to pull the call number and the location from the marc record rather than returning hard-coded dummy information. then i edited the getrealtimeholdings function in the marc record driver to use that function, rather than trying to connect to the catalog via a catalog driver and the holdlogic class, to get the call number and location. call numbers: the solr index stores call numbers in three ways: a normalized call number field (with spaces removed) that is meant to be used only for behind the scenes purposes such as sorting; a field named callnumber-a that stores the first part of lc-style call numbers with spaces so that they can be used for display; and as part of the full marc record. the first two functions take the first call number found in fields 099, 090, and 050. the getcallnumber function simply retrieves the callnumber-a field for a given item. the code below then appends the second part of the call number by pulling it from the marc record. since sudoc call numbers are not included in the callnumber-a field, both halves of those call numbers need to be taken directly from the marc record. (the code below does not check the 099 field because the only campus collections with call numbers in the 099 fields are the media collections and parts of special collections, neither of which were indexed into the offline catalog because we did not intend to deliver items from those collections to the prison. in general, excluding the media collections and special collections, it can safely be assumed that if an item in our system has an lc number in the 090 field it is shelved under that number, if it has a sudoc number in the 086 field it is shelved under the sudoc number, and if it has neither it is shelved under the lc number in the 050 field, so the code makes that assumption.) locations: the location codes are stored in subfield a (a repeating sub-field) of a non-repeating 998 field in the records exported from our ils; this may vary depending on your particular set-up. these location codes are not human-readable, but vufind’s built-in translation capability makes it easy to translate these codes into human-readable english (or any other language) locations in the public display by simply adding the codes and their human-readable translations to the appropriate language translation file(s). original code from sample driver: public function getstatus($id) { $holding[] = array('availability' => 1, 'status' => 'available', 'location' => '3rd floor main library', 'reserve' => 'no', 'callnumber' => 'a1234.567', 'duedate' => '', 'number' => 1); return $holding; } edited code, moved into marc record driver: public function getstatus($id) { $callnumber = parent::getcallnumber(); if ($this->_getfirstfieldvalue('090', array('a')) != null) { $parttwo = $this->_getfirstfieldvalue('090', array('b')); $callnumber .= " " . $parttwo; } elseif ($this->_getfirstfieldvalue('086', array('a')) != null) { $callnumber = $this->_getfirstfieldvalue('086', array('a')); $parttwo = $this->_getfirstfieldvalue('086', array('b')); $callnumber .= " " . $parttwo; } elseif ($this->_getfirstfieldvalue('050', array('a')) != null) { $parttwo = $this->_getfirstfieldvalue('050', array('b')); $callnumber .= " " . $parttwo; } $locationstuff = $this->marcrecord->getfields('998'); $thislocation = $locationstuff[0]->getsubfields('a'); foreach ($thislocation as $thislocationa) { $holding[] = array('availability' => 1, 'status' => 'available', 'location' =>  $thislocationa->getdata(), 'reserve' => 'no', 'callnumber' => $callnumber, 'duedate' => '', 'number' => 1); } return $holding; } how it works from the students’ perspective when the student plugs in the flash drive, they see three files: start vufind – run once to start vufind.bat (discussed above), stop vufind.bat (a simple three-line script that stops xampp gracefully and kills vufind, allowing the flash drive to be removed safely without shutting down the computer when necessary), and a web shortcut to vufind at http://localhost/vufind. (the vufind, xampp and jdk folders are hidden.) (see figure 1.) figure 1: shortcuts for running vufind the instructions for the students are relatively easy. first, they need to double-click on “start vufind – run once to start vufind.” this launches xampp and solr in command prompt windows. solr needs anywhere from a few seconds to a minute before it is ready to handle searches; during this time, text will be scrolling in the command prompt window. (see figure 2.) students should be told to wait for the command prompt window to stop scrolling, then to double-click on the internet shortcut to vufind. students should also be instructed that, if they need to remove the flash drive, they should either shut down the computer or run stop vufind first. figure 2: dos screens while solr initiates the students in the one class that has used the offline catalog to date seemed to find it intuitive to search; very little instruction in how to navigate the offline catalog was needed. going further an offline library catalog is a niche product today, when the vast majority of library patrons in developed countries either have internet access at home or can go to a physical library to access the internet. yet those potential patrons who cannot easily access an online catalog—a category that is not limited to incarcerated students—are likely to be some of the most under-served patrons. for example, consider rural patrons without internet access who receive their library services through a non-internet-equipped bookmobile. an offline library catalog might be the only feasible way to provide such patrons with equitable access to resources that these patrons could request through their library system if they were aware that the resources existed. ideally, an offline library catalog should not just replicate the basic search and display functions of the online catalog, but should also contain extra features that would only make sense in the offline environment. for example, in our case, instead of having the incarcerated students fill out request forms by hand, it would be helpful to allow students to mark books of interest and then print out a request form with their list of desired books at the end of their search session, without having to create an account. to date, due to time constraints i have mostly been concerned with removing features that did not work in the offline environment and ensuring that the most critical functions worked, but in the future i hope to be able to add features that would improve the experiences of those patrons who must use an offline catalog but would not make sense in a typical online library catalog. notes: 1. ebsco does offer an offline periodical database on cd-rom that is intended for use in prisons and other institutions that lack internet access, but for various reasons it was decided that that database was not a good choice for this program at this time. 2. more information on xampp can be found on its web site, http://www.apachefriends.org/en/xampp.html. 3. the vufind installation wiki: http://vufind.org/wiki/installation_windows 4. http://www.apachefriends.org/en/xampp-windows.html#646 5. for more information on batch parameters, see http://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/batch.mspx?mfr=true. acknowledgements the author thanks emily guenther and micah bot-miller for all of their help in making the prison catalog a reality, as well as the code4lib journal editors and everyone else who provided feedback on drafts of this article about the author julia bauder is the data services librarian at the grinnell college libraries. she graduated with a ba from bard college at simon’s rock and an mlis from wayne state university, and can be reached at bauderj@grinnell.edu. subscribe to comments: for this article | for all articles 2 responses to "using vufind, xampp, and flash drives to build an offline library catalog for use in a liberal arts in prison program" please leave a response below: nada o'neal, 2012-02-06 great work! i hope you get time in the future to work on some of the offline-specific features you described, and that you release your code changes to github or some place similar. :) adalid ortiz, 2012-06-22 hey, this is great. why not make a portable ready tu use version to download like this ils http://framakey.org/webapp/pmbportable saludos, faoa leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – duepublica: automated bibliometric reports based on the university bibliography and external citation data mission editorial committee process and structure code4lib issue 37, 2017-07-18 duepublica: automated bibliometric reports based on the university bibliography and external citation data this paper describes a web application to generate bibliometric reports based on the university bibliography and the scopus citation database. our goal is to offer an alternative to easy-to-prepare automated reports from commercial sources. these often suffer from an incomplete coverage of publication types and a difficult attribution to people, institutes and universities. using our university bibliography as the source to select relevant publications solves the two problems. as it is a local system, maintained and set up by the library, we can include every publication type we want. as the university bibliography is linked to the identity management system of the university, it enables an easy selection of publications for people, institutes and the whole university. the program is designed as a web application, which collects publications from the university bibliography, enriches them with citation data from scopus and performs three kinds of analyses: 1. a general analysis (number and type of publications, publications per year etc.), 2. a citation analysis (average citations per publication, h-index, uncitedness), and 3. an affiliation analysis (home and partner institutions) we tried to keep the code highly generic, so that the inclusion of other databases (web of science, ieee) or other bibliographies is easily feasible. the application is written in java and xml and uses xsl transformations and latex to generate bibliometric reports as html pages and in pdf format. warnings and alerts are automatically included if the citation analysis covers only a small fraction of the publications from the university bibliography. in addition, we describe a small tool that helps to collect author details for an analysis. by eike t. spielberg the problem the increasing competition between universities and researchers with respect to funding has given rise to a huge demand to evaluate the output of scientists or groups of scientists. the scientific “productivity” of researchers is an especially important question and is usually measured by the scientific publications of a given author. the scientific field dealing with the statistical analysis of scientific publications, bibliometrics, offers a broad range of measures and metrics to quantify scientific output and to qualify the “impact” the publication has on the scientific community.[1] the use of bibliometric indicators, such as the impact factor for journals, h-index for authors, or just the number of received citations for a single paper are well established and widely used in the natural sciences, although several tripping wires exist.[2][3] although the applicability can be questioned in many cases and several effects (like kilo-papers with several hundreds of authors) strongly distort the findings of the analyses, they are often used to gain some insights into the complex scientific landscape. as more and more tools arise which allow an easy and automated ad-hoc analysis (scival, incites), even with hardly any knowledge about bibliometrics, there is a great risk that the number is seen as the final result. however, bibliometric reports should not answer questions, but rather open up questions and discussions. to prevent their misuse, several libraries take the opportunity to establish consulting services about metrics and their limitations. going even further, they can also offer to prepare bibliometric reports, paying attention to openly acknowledge the limitations and prevent over-interpretation of the data. the elementary prerequisite is a database listing all the publications and linking them to the referenced publications. such databases exist for specific subject fields (e.g., ieee xplore [4]) but for the general audience only two major databases exist: web of science [5] and scopus [6]. in addition, google scholar [7] offers the possibility to analyse the references and yields a similar functionality. however, as the coverage is not clearly defined, analyses from google scholar still suffers from some methodological uncertainties.[8] problems also arise when preparing analyses for scientists with the interdisciplinary databases . in principle two major problems must be faced: the coverage of citation and bibliographic databases the attribution of authors let us have a more detailed look at these two problems: coverage scopus and web of science cover mainly journal articles, reviews, and a growing number of conference proceedings and book chapters. however, their coverage is far from complete. while sometimes patents, books, and research data are also incorporated, there are a number of very subject-specific ways to communicate research, which are not included: presentations, video-uploads, blogs, etc. this affects not only the coverage of publications themselves, but also their citation in other media (e.g. html links). this problem has been recognized and a number of so-called altmetrics are trying to address these tasks.[9] especially in the case of partial coverage, bibliometric analyses give reasonable looking results, but can be strongly distorted. in such cases the reports need a strong warning from a specialist putting the report in context. attribution of publications when evaluating a certain institute, faculty, university or other groups of scientists, publications are usually collected by the name of the scientists and their affiliation. the author’s affiliations in a publication can give rise to a number of errors. the author might have given an incorrect address, might have given only his institute, but not the university, there might be spelling errors, errors upon transmission of data from the publisher to the database provider and so on. cleaning all these entries is tedious work and it can take quite a long time until corrections are incorporated into the citation database. to circumvent these two problems, we followed another approach, namely to use the in-house university bibliography [10] as a source and to only enrich the entries found in there with the necessary additional information such as citation counts. our approach to solve the problem the university asks all scientists to register their publications in the university bibliography. the university library offers several services that make the reporting of publication easier. publication lists can be sent to library staff who enter or import the entries into the system. entries from the internal bibliography of the university hospital (evaluna biblio) are imported as well. also, in order to reduce the efforts for the scientists, external sources such as the scopus database [11] are used to import new publications. and finally, the library maintains a web frontend so that every member of the university can submit their publications. the entries are then checked by library staff before being published in the university bibliography. additional metadata such as an attribution of institute and faculty via a local identifier from the campus management system is added as well. additionally, every scientist can check his entries and report inconsistencies. as all the data is present in the local system, errors can be quickly corrected. all entries are assigned to a section and single person via the connection to the internal identity management of the university. the use of the local identifiers and the curation within the universtity library is well established, as the identifier is the critical link to generate publication lists on the author’s institutional homepage via predefined html code. the bibliography is based on the open-source mycore-framework [12], a framework to build institutional repositories (mycore is an acronym for my content repository). it is java/xml based and offers many tools to generate, retrieve from external sources (via so-called resolvers), handle, and index xml-documents. the description of contents within the xml-documents is strictly based on the metadata object description scheme (mods) [13] defined by the library of congress, offering a well-defined description. at the university duisburg-essen two mycore-installations exist: firstly, our open-access publication server duepublico (duisburg-essen publications online) and secondly, the university bibliography. one of our library department heads (frank lützenkirchen) is one of the founders and current developers of the framework. changes, customizations, and enhancements can be included quite easily. if any scientist needs another type of publication within the university bibliography, it can be included. we solve the two problems mentioned in the previous section by including new publication formats, which increases the coverage (though not of the citation data, vide infra), and by adding institutional data and/or local personal identifiers, which helps to clarify author affiliations. we select the entries relevant for our task (for example the evaluation of a scientist) from the university bibliography and enrich them – if possible – with citation information from the citation databases. preferably, we make use of persistent identifiers, which are included in the entries as well. by following this approach, we avoid the extensive search for name-variants in the citation databases and time-consuming cleaning of the result list (which is, in fact, double work, as the university bibliography entries need to be checked either way). the report is then generated in an automated fashion assuring transparent calculations. figure 1. general workflow for the generation of bibliometric reports from the university bibliography. entries can be imported from a number of sources and are controlled locally both by library staff and the scientists themselves. upon report preparation additional citation information is added from a citation database. workflow steps as depicted in figure 2, duepublica comprises a user-specific workflow. if the authenticated user belongs to the group “analyst”, a form field is displayed allowing the user to select whether to perform a citation analysis or just generate basic statistics. if the “analyst” wants to perform a citation analysis, they have to choose which citation source is to be used. afterwards they enter a name or identifier, and run the analysis by clicking the button “run” (see screenshot 1). upon submission of a name or identifier, the report is prepared as an xml document and displayed via xsl transformation as an html page. in addition, a button is displayed for saving the analysis and preparing the reports. the “analyst” can therefore perform the analysis and check the results. if everything is evaluated properly and all publications are found, they can finalize the report. screenshot 1. the input fields as seen by a user with the role “analyst”. if the user is logged in with the role “guest”, they are only able to access a saved report as html in their web browser or to download a prepared pdf file. the submission button and the “save and create pdf” button are not shown in the “guest” view. figure 2. general workflow of duepublica. when an identifier (or name) is submitted through the input field of the web frontend, the type of identifier is determined programmatically and the knowledgebase is queried for a corresponding entry. to determine the type of identifier, several consecutive steps are performed (see figure 3): first of all, the program checks whether the provided input contains a “-” character. if this is the case, it can be either a name or an orcid. therefore, the program deletes all “-” characters and checks with regular expressions whether the remaining characters are all numbers. if yes, the input is identified as orcid, otherwise as name. if the provided input contained no “-“, the application checks whether all characters are numbers. again, if this is not the case, the input is assumed to be a name. in contrast, if all characters are numbers, the input can be assumed to be an identifier. the type of identifier can be differentiated according to the length, as the local identifiers are rather short (up to five digits), the orcid is exactly 16 digits and the scopus author id is in between. if more identifiers are to be included, one would need to replace this rather simple scheme with a more sophisticated one or allow for a manual selection of the identifier type. figure 3. workflow to distinguish between different identifiers used within the system. with the obtained author the university bibliography is queried for her or his publications. some general statistical measures, and, if selected, citation and affiliation statistics are calculated on this set of publications. figure 4 shows the process flow for a bibliometric report performed by duepublica. the blue boxes depict the individual internal modules responsible for the individual steps. the knowledge base holds information about the authors and their identifiers. the university bibliography collects all the publications. the general statistics, the citation, and affiliation statistics modules are part of the web application. the last two modules collect additional information for each entry from an external source (depicted in dark blue) such as scopus [14]. how these individual modules work is described in more detail in the next sections. figure 4. process flow for a bibliometric report performed by duepublica. the general steps are depicted as grey boxes, the internal data sources as blue boxes and the external as dark blue boxes. selection of authors – the “knowledgebase” for author data in general, a lot of information is needed to characterize an author and to make sure that all publications are found. to prepare the analysis, a collection of data about the author is helpful. for this purpose, we designed a small database collecting information about the author(s), such as name, local and remote identifier, web pages, or address. we designed the knowledgebase as an independent restful application. a postgresql database [15] is used, accessible via a restful web application. this is easily achievable by defining a plain old java object (pojo), holding just the individual fields (name, affiliation, etc.). the jparepository interface within the spring boot framework [16] builds a layer above the postgresql database that provides options to retrieve, search, save, modify and delete entries via http operations. very good guides on how to set up such repositories and use the spring boot framework can be found at https://spring.io/guides. a small web frontend comprised of form fields to enter the data offers an easy way to submit author details (see screenshot 2). the pre-definition of authors in the knowledgebase is optional. if no entries are found in the run of an analysis, a new author is created. screenshot 2. part of the web frontend to enter author details for the knowledgbase. the analysis tool itself, duepublica, offers a search field where the name, the local identifier, the scopus author id or the orcid can be entered (see screenshot 3). upon submission, the application determines the type of id entered and the knowledge base is queried. if the author is found, the corresponding object is retrieved. as stated above, if the author is not present in the database, a new object is generated and filled with information from available sources. for example, if a scopus author id is entered, the scopus api is used to fill the fields like author name, orcid (if present) [17] and so on. subsequently, the university bibliography is queried with this information. use of the local identifier as the query parameter offers the opportunity to easily collect the data for a specific author from already curated data (vide supra). screenshot 3. form fields to enter either a name or a personal identifier. collection of documents – the university bibliography as mentioned above, the internal format used within the mycore community is the metadata object description scheme (mods), therefore the obtained articles are in well-formed xml and several fields can be addressed easily. the university bibliography itself is based on xml files indexed by an apache solr core.[18] the mycore-api allows you to query the solr-api in a controlled way. upon querying the mycore-api for a set of entries, a so-called mods collection is returned in xml-format, holding the individual mods-entries. from the point of duepublica, all connections to external sources are handled by connectors. in the source code, two java-interfaces have been defined: the “connector”-interface and the “citationgetter”-interface. the first one describes methods to retrieve entries from bibliographic sources (the original publications, written by the people under evaluation), while the second one describes methods to retrieve data about the citation of these entries (the number of other publications, which cite the original publications). the purpose of these two connectors is quite different: while the first one gives a list of publications with bibliographic information (title, author etc.), the second one gives the number of citing articles (citation count) for each publication. the mycore-connector implements the “connector”-interface and is used to retrieve the publication entries for a given author from the mycore-based university bibliography. by implementing the “connector” interface with different connectors (for example a connector to a local current research information system – cris) and mapping the response to the corresponding mods-fields, other sources of bibliographic data can easily be implemented. connectors like the scopus-connector implement the “citationgetter”-interface and are used to retrieve citation data. by writing other connectors, which implement the “citationgetter” interface, other sources for citation data (web of science, ieee xplore) can also be used to prepare bibliometric reports. one of the mods fields contains identifiers related to this publication. the field “identifier” holds persistent identifiers such as the doi or the isbn, but also database specific identifiers, as the scopus id or the ut-code for web of science.[19] this might look like this: <mods:mods id="ubo_mods_00073321"> … <mods:identifier type="doi">10.1016/j.jallcom.2016.09.165</mods:identifier> <mods:identifier type="scopus">84989905431</mods:identifier> … </mods:mods> wherever possible these ids are added to the entries within the university bibliography, either directly when importing the entry (for example from the scopus database), subsequently by automated routines, or manually when preparing an analysis. using these ids, additional data can easily be retrieved from external sources. by means of this mechanism, we incorporate the cleaning of data necessary to select the publications from the citation database (or the export from the database) to the local university bibliography. for example, a study for a specific institute has to be performed. however, the authors do not always name the institute in the affiliation data within the publication. maybe they only mention the university, their own group, a common abbreviation, or something similar. if you want to search for the publications within a citation database, you would probably use the individual author names together with the some general affiliation data like city, university name, etc. also identifiers like the scopus author id show similar problems, especially if the author’s name is quite common. when authors have a very common name, you get a lot of publications with this approach that do not belong to the institute. so if you perform analyses, you have to spend a lot of time cleaning the information collected from the citation databases. all of these problems with name variants, institutional affiliations and identifiers can be directly and immediately tackled within our library. as the university bibliography acts as a kind of showcase for the publications of the university duisburg-essen, and also allows the incorporation of personalized publication lists on the personal homepages (vide supra), efforts have already been spent to curate the corresponding entries since the beginning. the attribution to an institute is already within the identity management of the university, the local identifiers are also incorporated (to allow features like the personal publication list). we can thus select the set of publications for an institute easily within the university bibliography. we get the local ids for an institute and collect the publications for these ids. so we have a good and clean list of publications, we just need to link them to the entries within the citation database and collect additional information such as citation counts directly during the analysis. we therefore do not need to set up new procedures and routines to clean data for bibliographic analyses, instead we have to extend the established procedures a little bit to include other kinds of identifiers like the link to the scopus database. this is particularly easy for those entries which have been imported from the citation database, as the identifiers are already included. so we have a clean set of publications and we have the links to the entries of the citation database. now we enrich the publications from the list with additional information such as citation counts and perform three types of analyses. general statistics the first step is the calculation of some basic statistics. this includes data such as the total number of publications, the distribution along the years and the types of publications. this analysis can also be performed if no external sources are used. the corresponding java-class generalstatistics holds all the information and offers methods to get well formed json-objects, which can be displayed directly. such a class can be instantiated by supplying a collection of mods entries. internally they are handled as a list of jdom2-elements. due to the well-defined structure of these entries, information such as publication year or type of publication can easily be accessed using xpath expressions.[20] by analysing the node contents of “<origininfo><datedateissued>” and “<genre>” with the help of string manipulations and regular expressions, values can be extracted in the desired format, counted, and saved as fields in a generalstatistics object. the methods “getpublicationsperyearasjson” and “getpublicationspertypeasjson” produce these values directly as json-objects, which can be directly displayed by the javascript-framework highcharts.[21]) citation statistics a basic citation analysis is performed with the citation data obtained from the citation database. parameters like the average number of citations per publication, the fraction of uncited papers and the h-index are calculated and stored within the java-class citationstatistics. again, this class is instantiated by supplying the list of mods entries. upon creation of the object, the parameters are calculated. a very important parameter is the coverage of the citation database. this value is calculated by dividing the number of entries in the citation database by the total number of entries in the university bibliography. it is expressed as percentage and used to display warnings or alerts if the analysis becomes unreliable due to an insufficient coverage of the entries within the citation database. as mentioned before, a common critique that we hear is: “my publications are not in this citation database, your analysis does not make any sense.” taking this point seriously, we try to establish trust in our analysis by incorporating corresponding warnings within our reports and openly acknowledging the limited applicability in certain cases. affiliation statistics where available, information from the external sources is used to generate a map of institutions the author has been affiliated with as the author of a publication as well as of institutions where co-authors were affiliated. again the list of mods entries is used to instantiate an affiliationstatistics object, which performs the analysis upon creation. for example, the scopus database contains information about the affiliation of authors. information about city and country are extracted from the scopus response and then the latitude and longitude are retrieved from the google maps geocoding api. the combined information are used to generate three geojson-objects: one that depicts the actual affiliation, one that holds all home affiliations of the investigated author and a third one that holds all affiliations of his co-authors. the use of the geojson format allows the data to be displayed by a number of programs and frameworks (see screenshot 4). output – generating the report in multiple formats all analyses are prepared as xml-documents (represented as jdom-elements within the java code). each analysis object (generalstatistics, citationstatistics, and affiliationstatistics) adds another node to the output (see figure 5). the parameters calculated within these analysis objects are then inserted as sub-nodes. also the author object, obtained either by querying the knowledgebase or newly generated, produces a node and sub-nodes for the name, identifiers and so on: <bibliometricreport> <author> <surname>…</surname> <orcid>…</orcid> … </author> <citationstatistics> <hindex>…</hindex> … </ citationstatistics > … </mods:mods> the json-data [22], produced by the affiliationstatistics and generalstatistics objects and used for the charts and maps, are not stored within the corresponding node, but are rather incorporated in a separate node. including the json-formatted information within the xml-document might look a little bit surprising at first glance. however, it offers some advantages with respect to the html rendering by xsl transformation (vide infra). by providing the data in the json format, no transformation is necessary and the data can be ingested directly by the javascript frameworks that produce the maps and figures. the collection of the data within one single node makes it easier to attach just one block of javascript code during the xsl transformations. figure 5. xml nodes and the corresponding content. if the generated report is accepted by the “analyst” by pressing the “save and prepare pdf” button, the xml file is saved on the server. a new guest user and a corresponding folder are generated, the figures are prepared and the pdf is produced. two links are generated and sent to the analyst in a short email, which can be forwarded to the recipient: one to view the report in guest mode (that is without the control buttons and form) and one to download the pdf-file. html after saving, the generated report can be retrieved as an html document by a web servlet (bibliometricreportdisplayservlet). this servlet loads the saved xml-file, renders a html page via xsl-transformations, and displays the report in the browser. using the generated link, the user is automatically authenticated with the role “guest”. the xsl-file responsible for the preparation of the html-page is the same as used before during the preparation of the report. however, depending on the role of the authenticated user, the form fields and the “save and create pdf” buttons are displayed (role: “analyst”) or not (role: “guest”). a “guest” user only sees the report, without the possibility to perform analysis himself. the html-report has a responsive design (based on bootstrap [23]) and is organised in a block-wise fashion (see screenshots 4-8). some of these blocks are omitted if the corresponding analysis has not been performed. for example, if no affiliationstatistics-node has been created, the complete block with the map would not be displayed. the report starts with the affiliation analysis (if performed) as an interactive map (see screenshot 4). it is followed by a general disclaimer (not depicted), and the overview of the types of publications (see screenshot 5 upper box). if a citation analysis was performed and if the coverage of publications in the citation database is below a threshold, a warning (coverage less than 80 %) or an alert box (coverage less than 60 %) is displayed (see screenshot 5, lower box). screenshot 4. affiliation statistics presented as an interactive map. screenshot 5. type of publications and warning box. if a citation analysis was performed, a general overview of basic metrics – e.g. number of publications in citation database, total number of citations, average citations per publication, percent of non-cited publications and h-index – is shown, first as a table block, followed by a detailed description of the basic metrics (see screenshot 6). screenshot 6. citation metrics and detailed description. finally, the distribution of publications per year is shown as an interactive chart (see screenshot 7) and the list of publications as a sortable, paginated table (see screenshot 8). in the case of journal articles, the title links to the corresponding publication, the name of the journal links to the availability information of our university library, and the number of citations links to the overview page in the citation database (for example scopus). screenshot 7. temporal evolution of publication activity. screenshot 8. list of publications. the representation as an html web page allows an interactive experience. charts and maps can be zoomed in, details analysed by sorting the table, and hover-over highlighting make it easy to read the data from tables. however, for several purposes a printed, more formal, type of report might be necessary. to this end, the xml file is also transformed into a pdf file. pdf to generate well-formatted reports as pdfs, the final xml-document is transformed into latex source code.[24] the relevant responsible part of the source code is depicted in listing 1. the “generatefile” method uses xsl 2.0 transformations (“bibliometricreport2latex.xsl”) to generate the plain text input file. the overall layout is based on the komascript package [25] and is included together with the text in the transformation file. up to now, only german output is produced. a localization to other languages could be achieved by adjusting the corresponding xsl file. similar to how the html is rendered, individual sections are generated. if analyses have not been performed, the corresponding sections are omitted. by using the package hyperref [26], a fully linked table of contents and several hyperlinks to external sources (such as scopus) can be included as well. to include the graphical representations within the pdf file, they need to be exported into some graphical format, preferably png. for the highcharts graphs a command line tool exists [27], which uses a node.js server [28] to produce svg or png files. in our case we choose png files as they can be included easily into the latex-document using the graphics package. to this end, the input-json files are generated from the analysis xml by xsl 2.0 transformations as well (“pubsperyear2json.xsl” and “pubspertype2json.xsl”). then the command line tool is called, producing the desired graph files. only the map cannot be exported (yet). after the graphs have been exported, the latex compiler (pdflatex) is called from the export servlet with the “nonstopmode” option generating the final pdf-file (see screenshot 9). as common with latex calls, two runs are required, the first one generating the references to figures and hyperlinks and the second including them into the output pdf. generatefile(reportxml.clone(),"xsl/bibliometricreport2latex.xsl",latexfile); generatefile(reportxml.clone(),"xsl/pubsperyear2json.xsl",pubsperyearfile); generatefile(reportxml.clone(),"xsl/pubspertype2json.xsl",pubspertypefile); // build the pdf by pdflatex processbuilder pbpubsperyear = new processbuilder("cmd","/c","{pathtoexportserver}highcharts-export server","-infile","pubsperyear.json","-outfile","pubsperyear.png"); pbpubsperyear.directory(reportfolder); pbpubsperyear.start().waitfor(); // build the pdf by pdflatex processbuilder pbpubspertype = new processbuilder("cmd","/c","{pathtoexportserver}highcharts-export-server", "-infile","pubspertype.json","-outfile","pubspertype.png"); pbpubspertype.directory(reportfolder); pbpubspertype.start().waitfor(); // build the pdf by pdflatex processbuilder pblatex = new processbuilder("pdflatex","-interaction=nonstopmode", "report.tex"); pblatex.directory(reportfolder); pblatex.redirectoutput(new file(reportfolder, "shell.log")); pblatex.start().waitfor(); pblatex.start().waitfor(); listing 1. programmatic generation of figures and call of latex compiler. the pdf is stored together with the source files and the xml-file and can be accessed by the generated download link. this link can be shared or forwarded, offering a nice way to deliver the report to the client. screenshot 9. first two pages of the bibliometric report (in german) as latex generated pdf file. next steps – inclusion of other sources and modularity although limited in the actual components, the web application has been designed to be highly flexible. up to now, only mycore based repositories and scopus as a citation source have been included as external sources. however, the structure of the program allows for extension to several other systems. we have decided to use the mods metadata scheme within the program, but transformation of other input via xsl transformations should be rather straight forward. the general functionality of the document loading has been defined as an interface, which can be implemented by a variety of connectors. in addition, an interface for a citation getter has been defined as well, defining the basic functions needed. other connectors can be written to implement this interface, giving access to a number of different sources for citation data. the three types of analyses (general analysis, citation analysis and affiliation analysis) are implemented by corresponding classes, which hold the results and offer methods to calculate the parameters and metrics. incorporating other metrics is possible by expanding these classes. our future development will focus on the inclusion of additional sources, both for the bibliographic information and for the citation data. the use of current research information systems (cris) as a bibliographic source would greatly enhance the applicability of our tool. the inclusion of other citation sources such as web of science [29], ieee xplore [30] or other subject specific databases might greatly enhance the acceptance of such reports within the investigated community. and last but not least, we would like to re-organise the code in order to provide a restful api, which can be used to include the generated data in other reports and/or systems. conclusions when preparing bibliometric analyses one faces two major problems: firstly, the coverage of the author’s output in the citation database and secondly, the collection of all relevant publications. we tackle these two problems by developing a web application to generate reports based on the university bibliography and the scopus citation database. provided with an author’s name or identifier (e.g. orcid, scopus author id) the tool collects the author’s publications from the university bibliography, enriches them with additional information gathered from the scopus database and applies up to three kinds of analyses: a general analysis (number and type of publications, publications per year etc.), a citation analysis (average citations per publication, h-index, uncitedness) and an affiliation analysis (home and partner institutions). especially the use of a local identifier, which is already curated within the university library, circumvents an extensive cleaning of data. the final report is available both as an html-version as well as in pdf-format. the general structure and workflow of this tool is designed to include many publication types. due to its modular structure, additional metrics or data sources can easily be included. technical details the web application is java/xml based. the mycore [31] framework was used to design the main servlet handling the requests. to allow user role specific views and options, authentication and authorization was performed using the apache shiro framework.[32] for the knowledgebase (see below) the spring boot framework [33] was used to generate the database backend and the web frontend. the general design of the web pages is bootstrap [34]) based, the charts are generated by the highcharts-package [35]) and the map is build using the lightweight leaflet package [36], based on openstreetmap [37] together with the mapbox tiles [38]. the source code is available at github: https://github.com/etspielberg/bibliometrics, https://github.com/etspielberg/knowledgebase-backend, and https://github.com/etspielberg/knowledgebase-frontend. acknowledgement special thanks goes to frank lützenkirchen for his support with the mycore-specific parts and to felix schmidt for helpful discussions. i would also like to thank beate baurmann for helping with some of the figures. references [1] todeschini r, baccini a. 2016. handbook of bibliometric indicators: quantitative tools for studying and evaluating research. 1. auflage. weinheim: wiley-vch verlag gmbh & co.kgaa. 483 p. isbn: 3527681957. eng. [2] bornmann l, mutz r, neuhaus c, daniel hd. 2008. citation counts for research evaluation: standards of good practice for analyzing bibliometric data and presenting and interpreting results. esep. 8:93–102. doi:10.3354/esep00084. [3] joshi ma, patil sg. 2014. bibliometric indicators for evaluating the quality of scientific publications. tjcdp. 15:258–262. doi:10.5005/jp-journals-10024-1525. [4][30] ieee. [accessed 2017 may 11]. ieee xplore digital library. http://ieeexplore.ieee.org/xplore/home.jsp. [5][19][29] clarivate analytics. [accessed 2017 jan 28]. web of science. http://wokinfo.com/. [6][11][14] elsevier. [accessed 2017 jan 28]. scopus: welcome to scopus. https://www.scopus.com/. [7] google inc. [accessed 2017 may 19]. google scholar. https://scholar.google.de/. [8] aguillo if. 2012. is google scholar useful for bibliometrics?: a webometric analysis. scientometrics. 91(2):343–351. doi:10.1007/s11192-011-0582-8. [9] priem j, taraborelli d, groth p, neylon c. 2010. altmetrics: a manifesto. [accessed 2017 may 19]. http://altmetrics.org/manifesto. [10] universität duisburg-essen. [accessed 2017 may 11]. welcome to bibliography of ude. [place unknown]: [publisher unknown]. https://bibliographie.ub.uni-due.de/lang=en. [12][31] mycore-community. [accessed 2017 jan 19]. mycore: das framework zur präsentation und verwaltung digitaler inhalte. de. http://mycore.de/. [13] the library of congress. [accessed 2017 may 11]. metadata object description schema. http://www.loc.gov/standards/mods/. [15] the postgresql global development group. [accessed 2017 may 11]. postgresql: the world’s most advanced open source database. https://www.postgresql.org/. [16][33] pivotal software. [accessed 2017 may 11]. spring boot. https://projects.spring.io/spring-boot/. [17] orcid i. [accessed 2017 may 11]. orcid: connectiong research and researchers. https://orcid.org/. [18] the apache software foundation. [accessed 2017 jun 2]. apache solr. http://lucene.apache.org/solr/. [20] w3c. [accessed 2017 jun 2]. xml path language (xpath). [place unknown]: [publisher unknown]. https://www.w3.org/tr/xpath/. [21][35] highcharts. [accessed 2017 jan 28]. interactive javascript charts for your webpage. http://www.highcharts.com/. [22] kotamraju j. 2013. java api for json processing: an introduction to json: the java api for json processing provides portable apis to parse, generate, transform, and query json. [accessed 2017 jan 19]. en. http://www.oracle.com/technetwork/articles/java/json-1973242.html. [23][34] @mdo, @fat. [accessed 2017 jan 28]. bootstrap: the world’s most popular mobile-first and responsive front-end framework. http://getbootstrap.com/. [24] the latex project. [accessed 2017 may 11]. latex: a document preparation system. https://www.latex-project.org/. [25] ctan comprehensive tex archive network. [accessed 2017 jun 2]. paket koma-script: a bundle of versatile classes and packages. https://www.ctan.org/pkg/koma-scriptlang=de. [26] ctan comprehensive tex archive network. [accessed 2017 jun 2]. paket hyperref: extensive support for hypertext in latex. https://www.ctan.org/pkg/hyperref. [27] highcharts. [accessed 2017 jun 2]. command line rendering. https://www.highcharts.com/docs/export-module/render-charts-serverside. [28] node.js foundation. [accessed 2017 jun 2]. node.js. https://nodejs.org/en/. [32] the apache software foundation. [accessed 2017 jan 19]. apache shiro: simple. java. security. https://shiro.apache.org/. [36] vladimir agafonkin. [accessed 2017 may 11]. leaflet – a javascript library for interactive maps. [place unknown]: [publisher unknown]. http://leafletjs.com/. [37] openstreetmap contributors. [accessed 2017 may 11]. openstreetmap. http://www.openstreetmap.org/. [38] mapbox. [accessed 2017 may 11]. mapbox: the location platform for developers and designers. https://www.mapbox.com/. about the author eike t. spielberg (orcid: 0000-0002-3333-5814) studied chemistry at the friedrich-schiller-unviersity in jena and now works as subject librarian and scientific co-worker at the university library duisburg-essen in germany since 2015. he develops tools for automation of library workflows and for new services within the context of bibliometrics. his email address is eike.spielberg@uni-due.de. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – automatic generation of printed catalogs: an initial attempt mission editorial committee process and structure code4lib issue 10, 2010-06-22 automatic generation of printed catalogs: an initial attempt printed catalogs are useful in a variety of contexts. in special collections, they are often used as reference tools and to commemorate exhibits. they are useful in settings, such as in developing countries, where reliable access to the internet—or even electricity—is not available. in addition, many private collectors like to have printed catalogs of their collections. all the information needed for creating printed catalogs is readily available in the marc bibliographic records used by most libraries, but there are no turnkey solutions available for the conversion from marc to printed catalog. this article describes the development of a system, available on github, that uses xslt, perl, and latex to produce press-ready pdfs from marcxml files. the article particularly focuses on the two xslt stylesheets which comprise the core of the system, and do the “heavy lifting” of sorting and indexing the entries in the catalog. the author also highlights points where the data stored in marc bibliographic records requires particular “massaging,” and suggests improvements for future attempts at automated printed catalog generation. by jared camins-esakov introduction since the advent of the opac, printed catalogs have become far less common than they once were. printed catalogs still have value, however. in some cases, printed catalogs are useful because they provide access to bibliographic information in places where electronic access is impractical, such as afghanistan, where i spent three months working as a librarian. although we did not have a printed catalog at the university where i worked, one would have been helpful on days when we were without electricity. at the kabul education university, they had a single partial manuscript catalog of their holdings, which served their several thousand students far better than our opac served our several hundred. in addition to offering 100% uptime, students seemed to find it much easier to understand how to use the manuscript catalog. in some cases, a printed catalog may be of particular value as a specialized bibliography suitable for browsing. this is particularly common in the world of rare books, where remarkably comprehensive collections are sometimes assembled. an example of this is mary ann o’brian malkin’s dancing by the book (new york: privately printed, 2003), a catalog of malkin’s collection of books relating to the history of dance notations, now used by scholars interested the subject. [1] another good example would be paul needham’s twelve centuries of bookbinding: 400-1600 (new york: pierpont morgan library, 1979), written to commemorate an exhibition at the morgan library, and now a well-respected exemplar for the description of bindings. in many cases, including the two just mentioned, printed catalogs that are well-respected as subject bibliographies were published at least partly to be keepsakes. private collectors often produce printed catalogs as a monument to their accomplishments as collectors. they commonly do this shortly before selling the collection, so they have something to remind them of what they had. both dancing by the book and the catalog discussed in this article could be considered keepsakes for private collectors. keepsakes are also produced by many museums and special collections units in university libraries. princeton university special collections, for example, publishes a catalog of every major exhibition they put on. [2] in early 2009, we were hired by a private collector to catalog his collection of ornithology books. we started the project expecting to prepare some sort of bibliographic database for his use on the computer. however, as the collector explained his desire for some sort of keepsake that would make it easy to check if he already owned a specific book, we realized that a printed catalog would probably be a better choice for him. since he was more concerned with costs than with a perfectly typeset product, and we had computer programming experience, we decided to develop a system to automatically produce a press-ready pdf for a catalog that met the collector’s requirements: title main entry (the collector felt this would be more useful than author main entry), with cross-references indexes for authors, subjects, geographical subjects (the collector wanted a separate index that included only geographic headings for faster reference), and series formatting reminiscent of catalog cards (i.e., isbd punctuation) a unique catalog number for each book so he would not need to copy the entire citation if he ever wanted to take notes on a specific book system architecture in the interest of having a finished product as soon as possible, we developed the system using whatever technology was easiest for each step. the result was a messy and heterogeneous (but effective) process that looks like this: a data file in marcxml format is run through an xslt stylesheet for preprocessing, which generates guaranteed unique id numbers for each record, and generates see references for titles in 246s. the result is sanitized by a perl script which converts marc-8 composed characters into the appropriate latex escapes, and corrects common misspellings. this is then run through the main catalog generating xslt stylesheet which produces the latex file. this includes the index entries, which are commented out so that they are not picked up by latex automatically. latex is run on the latex file to initialize makeindex. the index entries for the name, subject, geographical, and series indexes are extracted using standard *nix utilities (grep and sed), and illegal characters are removed before each is written directly into the raw index file initialized by latex. formatted indexes are generated using makeindex. latex is run twice more to incorporate the formatted indexes into the document and stabilize page number references in the table of contents (since the indexes are keyed to the entry numbers rather than page numbers, they are stabilized by step 3). the resulting pdf file is run through ghostscript to generate a pdf compatible with the rip processes used by print-on-demand services such as lulu.com. implementing the preprocessor the first, and possibly most thorny, problem we encountered in generating the catalog was dealing with title cross-references. the collector wanted to have access points for both the title proper (marc field 245) and alternative titles (marc field 246), but wanted only see references for alternative titles, so that each book was described only once in the catalog. this requirement presented two problems: based on our understanding of xslt, using <xsl:apply-templates/> with <xsl:sort/> requires every tree being sorted to be at the same level in the source tree sorting main entries and see references at run-time, we could not guarantee that see references would point to the correct main entry, nor limit numbering only to main entries a related problem came from the collector’s requirement that each record have an invariant catalog number associated with it. if we used <xsl:sort/> to sort both the main entries and the see references, the position() function would count see references as well as actual records, and using the preceding-sibling:: axis required far too much processing power for it to be a reasonable solution for catalogs of any significant length. the solution we developed for these related issues is embodied in the 49-line preprocessor xslt stylesheet. the preprocessor sorts the records by title main entry (marc field 245), assigns a unique ordinal id for each record, and generates a <see/> tag for each alternative title: <xsl:for-each select="marc:record"> <pre> <xsl:sort select="lower-case(substring(marc:datafield&#91;@tag='245'&#93;/marc:subfield&#91;@code='a'&#93;,marc:datafield&#91;@tag='245'&#93;/@ind2 + 1))" order="ascending"/> <xsl:variable select="position()"/> <record> <ref><xsl:value-of select="$pos"/></ref> <xsl:copy-of select="./*"/> </record> <xsl:for-each select="marc:datafield&#91;@tag='246'&#93;"> <xsl:variable select="marc:subfield&#91;@code='a'&#93;/text()"/> <xsl:if test="count(preceding-sibling::marc:datafield&#91;@tag='246'&#93;/marc:subfield&#91;@code='a' and text()=$subfield246a&#93;)=0 and not(contains($subfield246a,'&amp;')) and not(contains($subfield246a,'%26'))"> <see> <datafield tag="245" ind1="0"> <xsl:choose> <xsl:when test="matches(marc:subfield&#91;@code='a'&#93;,'^a ')"> <xsl:attribute>2</xsl:attribute> </xsl:when> <xsl:when test="matches(marc:subfield&#91;@code='a'&#93;,'^the ')"> <xsl:attribute>4</xsl:attribute> </xsl:when> <xsl:otherwise><xsl:attribute>0</xsl:attribute></xsl:otherwise> </xsl:choose> <xsl:for-each select="marc:subfield"> <xsl:copy-of select="."/> </xsl:for-each> </datafield> <title><xsl:copy-of select="preceding-sibling::marc:datafield&#91;@tag='245'&#93;/*"/></title> <ref><xsl:value-of select="$pos"/></ref> </see> </xsl:if> </xsl:for-each> </xsl:for-each> by using this preprocessor, we were able to provide a sort key in field 245 for both records and see references for use in the main xslt stylesheet, which is described in the next section. implementing the catalog generator once we had developed the preprocessor stylesheet, writing the main catalog generator was fairly straightforward. we chose to adapt the marc21slim2english.xsl stylesheet that the library of congress makes available on its website for our purposes. [3] the bulk of the adaptation involved inserting the appropriate latex markup around the output document, but there were a few specific parts of the catalog generator that presented problems. for example, although it is straightforward in hindsight, we had to account for non-filing characters when sorting the entries in the main loop: <xsl:apply-templates select="marc:collection/marc:record | marc:collection/marc:see"> <xsl:sort select="lower-case(substring(marc:datafield&#91;@tag='245'&#93;/marc:subfield&#91;@code='a'&#93;/text(),marc:datafield&#91;@tag='245'&#93;/@ind2 + 1))" order="ascending"/> </xsl:apply-templates> the principle challenge in the main catalog generation routine was creating the indexes. although we chose latex for the powerful indexing functionality provided by makeindex, we quickly discovered that the automatic indexing routines in latex would not meet our needs, since the index entries latex generated used page numbers as locators, and we wanted to use catalog numbers as locators. the solution we came up with was to generate the entries for the .idx files we needed directly, and extract those entries from the output file for use with makeindex. the following code snippet shows the xslt template that generated the index entries for general subjects (i.e., marc fields 650 and 651): <xsl:template> <xsl:if test="@code='a' or @code='v' or @code='x' or @code='y' or @code='z' or @code='b' or @code='c' or @code='d' or @code='p'"> <xsl:variable select="preceding-sibling::marc:subfield&#91;position()=1&#93;/@code"/> <xsl:if test="$lastsubfield='a' or $lastsubfield='v' or $lastsubfield='x' or $lastsubfield='y' or $lastsubfield='z' or $lastsubfield='b' or $lastsubfield='c' or $lastsubfield='d' or $lastsubfield='p'"> <xsl:text>!</xsl:text> </xsl:if> <xsl:value-of select="replace(replace(replace(text(),'(&#91;\s.&#93;&#91;a-za-z&#93;)\.&#91;\s,;:&#93;*$','$1#'),'&#91;\s.,:;&#93;+$',''),'#','.')"/> </xsl:if> </xsl:template> <xsl:template> <xsl:param/> <xsl:text>%subject:\indexentry{</xsl:text> <xsl:for-each select="marc:subfield"> <xsl:call-template/> </xsl:for-each> <xsl:text>}{</xsl:text><xsl:value-of select="$number"/><xsl:text>}&amp;#10;</xsl:text> </xsl:template> aside from those two challenges, however, the main stylesheet was not that difficult to develop. most of the changes we made to the stylesheet in the course of development were cosmetic, relating more to the latex preamble than to the actual generation of the catalog. the latex markup (aside from the markup contained in the catalog entries themselves, of course) is segregated into two <xsl:text/> tags. the first contains the preamble (i.e., style definitions, and definitions of custom commands for producing the catalog) and the preliminary material. we ultimately found that the following definitions were needed in the preamble to create a pdf suitable for use with a print-on-demand service: \documentclass[10pt]{book} \usepackage{geometry} \geometry{paperwidth=6in, paperheight=9in, textwidth=3.25in, textheight=7.3in, twoside, bottom=0.8in, left=1.25in} \geometry{pdftex} \usepackage[t5]{fontenc} \usepackage{pslatex} \usepackage{multind} \usepackage{tipa} \usepackage{sectsty} \usepackage{graphicx} \usepackage{tocvsec2} \usepackage[compact]{titlesec} \usepackage[pdftex]{hyperref} \makeindex{names} \makeindex{subject} \makeindex{geo} \makeindex{series} the second contains the markup for generating the indexes: \printindex{names}{name index} \printindex{subject}{subject index} \printindex{geo}{geographical index} \printindex{series}{series index} \end{document} our goal in formatting the catalog entries was to create a readable and attractive format reminiscent of a card from a traditional card catalog, while taking advantage of the additional space offered by pages. an example of the final result, which the collector was very happy with, can be seen here: figure 1. example record the name index (along with the subject and geographical indexes) used the default makeindex settings, resulting in a recognizably “latex-y” index, as the following example shows: figure 2. name index example the series index used a slightly modified style to make the volume numbers and catalog numbers easier to distinguish: figure 3. series index example future improvements and directions overall, our experiment in generating a printed catalog automatically from marc records was successful. after we finished, the collector had a press-ready pdf he could send off to a print-on-demand service, or read comfortably on his computer, and we had successfully prototyped and demonstrated a system for automatically generating printed catalogs from marc records. looking back on the project, however—success notwithstanding—what stands out are all the aspects that could have been done better. in many ways, our major complaint with the system is that it is inelegant. the system is developed in a combination of xslt, perl, and latex, with a healthy dose of bourne shell and standard *nix utilities to tie the whole thing together. in an ideal world, of course, we would not have needed to use latex (or perl, or *nix utilities) at all. a system that used xsl-fo for generating a pdf would reduce dependencies, simplify support for non-latin scripts, and, perhaps most importantly, make it easier for non-latex users to use the system (thanks to the ubiquity of xml, and particularly html, the syntax of xsl-fo seems, i think, less scary than the syntax of latex). unfortunately, our initial research indicated that index generation in xsl-fo would be very problematic, so—as we had a tight deadline—we had to shelve the idea of an entirely-xml stack. of the many inelegances, the most egregious is the use of the main xslt stylesheet to generate the entire latex document. in hindsight, a better way to do this would be to have a skeleton latex document include the results of the stylesheet’s processing. by doing this, global formatting changes could be tested without rerunning the xslt stylesheet. this would also simplify the production of multiple catalogs with identical content but differing preliminaries, fonts or page sizes (all issues that came up at one point or another in the process). taking this idea to its logical end, the system should use latex macros for all format-specific features. we were unable to meet this ideal with the positioning of the frames containing isbns, but aside from that, we used latex macros for all other formatting within the individual records. another example of inelegance is the heavy use of the copy-and-paste development methodology in the main catalog generation stylesheet. a more elegant solution would make better use of xslt parameters, and, for example, eliminate the almost-identical “index-*” templates in favor of a single “index” template that used a parameter to select the appropriate index. as with most of the other problems, this is the result of our compressed timeline for development. fortunately, this is a relatively simple change, and can (and probably will) be changed for the next time we need to generate a print catalog. aside from issues of elegance, there are a number of other particularly problematic areas in the catalog generation system. support for non-roman scripts is non-existent. there are two issues contributing to this. the first is the lack of character set support. in principle, lambda (the latex component of omega, a unicode-enabled version of tex) could be used with utf-8 encoded input, but we did not try this, and omega seems to have been more-or-less orphaned. another project, luatex, offers a unicode-compliant version of latex, but they anticipate remaining in beta for the next three years. for right-to-left scripts, there are additional issues, as latex treats right-to-left scripts differently than it treats left-to-right scripts. the second issue is that, as it currently stands, the system does no parsing of $6 (linkage) subfields and 880 (alternate graphic representation) fields. most marc records that include the original non-roman script also include romanized representations of non-roman scripts. these records store the non-roman script in an 880 field and link it to the standard marc fields with romanized data (e.g., 245) using subfield $6. the catalog generation system does not parse this in any way, and would only display the romanized representations. the series index could also be improved. at the moment, the series index is generated by using every single series entry (all 4xxs and all 8xxs) in each record. this results in numerous almost-identical entries. unfortunately, since the use of traced versus non-traced headings is inconsistent in cataloging copy (standards have changed repeatedly), we were unable to find an alternative way to deal with series entries. this problem affects non-printed catalogs too, as seen in a 2009 discussion of series headings on jonathan rochkind’s blog. [4] an algorithmic method for dealing with the peculiarities of series entries could be of use to a great many system designers, but, unfortunately, we are unable to suggest any such method. another flaw in the catalog generation system is that it cannot handle any organization other than title main entry. ideally, a general catalog generation system would offer the option to choose between title main entry and author main entry, depending on the project. fortunately, the implementation of this need not be complex. basically, all that would be needed is to have the two stylesheets sort on 1xx fields instead of 245 fields. in order to ensure that works with a title main entry were sorted properly, the preprocessor would need to be changed to copy the title from the 245 to a 1xx field when there is no 1xx field, but this would also be quite simple (see the code snippet from the preprocessor stylesheet, above, for an example of copying 246 fields into 245 fields for see references). the main challenge here would be creating a title index, which would need special formatting. should anyone be interested in tackling this problem, we hope that they will contribute their modifications back to the community. despite these technical complaints, our project was a success. not only did we generate a printable catalog, we developed a system that can generate a printed catalog out of any marcxml file we might want to throw at it. we also learned about some of the difficulties that one encounters when creating a printed catalog and, for the most part, overcame those difficulties. if our code is not the most brilliant code in the world (and we would be the first to acknowledge that it isn’t), it does work. we hope that anyone looking to create a printed catalog will be able to extend our system to do better than we could do in our initial attempt at generating a printed catalog. we have made our source code available to the public under the gnu general public license version 3 (gplv3) in our github repository at http://github.com/jcamins/cataloggen/. about the author jared camins-esakov is a freelance bibliographer and former computer programmer, currently employed by the american numismatic society. jared can be reached at jcamins@gmail.com or via his website at http://www.jaredcamins.com/ notes [1] personal communication with terry belanger, 4/3/2009 [2] for example, at the time of writing, a catalog of the current exhibition is available: liberty & the american revolution: selections from the collection of sid lapidus ’59: an exhibition catalogue (princeton: princeton university library, 2009). (coins) http://www.worldcat.org/oclc/378505880 [3] http://www.loc.gov/standards/marcxml/xslt/ [4] http://bibwild.wordpress.com/2009/09/24/a-reasonable-display-for-series-data-in-marc/ subscribe to comments: for this article | for all articles one response to "automatic generation of printed catalogs: an initial attempt" please leave a response below: zoltan tomory, 2010-06-24 good for you! assembling the list is the second lesson in marc reading; the first being actually pulling information out of marc records to a)pull info out to satisfy queries and b)format bib citations. eventually, you stand poised to make a bibliographic tool which can more or less resemble a catalog, depending on your tastes. the nastiest bit of the exercise is making it palatable to yourself and others. your output looks great, an important measure. thanks for supporting open source and sharing your code. leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – analyzing and normalizing type metadata for a large aggregated digital library mission editorial committee process and structure code4lib issue 47, 2020-02-17 analyzing and normalizing type metadata for a large aggregated digital library the illinois digital heritage hub (idhh) gathers and enhances metadata from contributing institutions around the state of illinois and provides this metadata to the digital public library of america (dpla) for greater access. the idhh helps contributors shape their metadata to the standards recommended and required by the dpla in part by analyzing and enhancing aggregated metadata. in late 2018, the idhh undertook a project to address a particularly problematic field, type metadata. this paper walks through the project, detailing the process of gathering and analyzing metadata using the dpla api and openrefine, data remediation through xsl transformations in conjunction with local improvements by contributing institutions, and the dpla ingestion system’s quality controls. by joshua d. lynch, jessica gibson, and myung-ja han background: problems with type metadata and existing quality controls introduction to the idhh and type metadata the illinois digital heritage hub (idhh) gathers and enhances metadata from contributing institutions around illinois and provides this metadata to the digital public library of america (dpla) for greater access. the idhh helps contributors shape their metadata to the standards recommended and required by the dpla. this paper describes the process for gathering and analyzing contributors’ type metadata, problems seen in the analysis of the metadata, and ways that contributors, the idhh, and the dpla remediated problems found in the analysis. the idhh is the illinois service hub [1] for the digital public library of america (dpla), and a collaboration among the illinois state library (isl), the consortium of academic and research libraries in illinois (carli), the chicago public library (cpl), and the university of illinois urbana-champaign (uiuc) library. as of july 2019, the idhh aggregated metadata from about 150 individual contributing institutions around the state. the metadata is harvested as qualified dublin core (qdc)[2], also known as dcmi metadata terms (dcterms), and provided as qdc records to the dpla. as of july 2019, the idhh harvested and exposed more than 310,000 records from 448 digital collections. through metadata assessment [3], a best practices document [4], and training; the idhh has made significant efforts so that contributors’ metadata conforms to the dpla’s standards and recommendations which ultimately improve discoverability of the idhh’s rich and unique digital resources. as in any aggregated environment, some fields, including type, have been difficult to standardize across the idhh’s contributing institutions. the dpla has robust quality controls for all its metadata and specific refinements for the type field [5]. however, its ingestion systems require a quality baseline that was not present in the idhh’s type metadata. the idhh type metadata is very diverse and the dpla requirements are strict; while all metadata from original records is retained and available in the dpla’s api, only dcmi type [6] values, such as ‘image’, ‘moving image’, ‘physical object’, ‘sound’, and ‘text’ facetable in the dpla’s api and catalog, as shown in figure 1. type is one of several data fields that the dpla uses to create facets by which to narrow search results and to link from an individual record to a list of items of a given type, as shown in figure 2. the dpla’s ingestion systems provide for mappings between a set list of non-conforming values and dcmi types, but mappings and subsequent catalog features that rely on dcmi type values seem to work best when valid dcmi type values are present in an original record. figure 1. type faceting options that appear with search results. figure 2. type facet within a record. clicking ‘image’ will link to other records in the dpla with the same type metadata. due to its importance for access, discovery, and interactivity, and the low rates of completion as seen in the dpla’s search interface [7], it was determined that enhancement of the type metadata contributed by idhh institutions was necessary. initial analysis revealed that there were 558 unique type values in all idhh records. after metadata analysis, a three-pronged approach to enhancing type metadata was undertaken. type metadata contributed by idhh institutions could be improved by: institutions submitting better quality metadata, more robust xsl transformations of the aggregated metadata by the idhh, and working with the dpla in order to improve their quality controls on idhh type metadata, which may in turn lead to improved quality controls on type and other fields for other hubs contributing metadata to the dpla. overview of idhh type metadata problems in-depth analysis discussed in detail below showed that, as of 2018-12-13, there were at least 558 unique type metadata values among about 327,000 values total. only about 176,000 of the 327,000 total values perfectly conformed to dcmi type. due to the large volume of unique values coming in and limitations in the dpla’s ingestion 1 [8] quality controls, only approximately 61,000 metadata values displayed and functioned as working facets in the dpla’s search interfaces, as shown in figure 3 below. this represented only one dcmi type value for every five records that idhh institutions had contributed to the dpla. figure 3. low rates of type completion in idhh metadata indicated by dpla analytics dashboard, 2018-12-14. some examples of problems in the type metadata originally provided by contributors included: a wide variation of capitalizations (“still image” vs. “still image” vs. “still image”) highly granular values with slight variations of the same essential type or format (“letter – carbon typescript”, “letter – typescript”, “letter”, etc.) large numbers of multiple delimited values in a field (e.g., “text; image”, “text; sound”, “text; still image”) superfluous delimiters appearing at the end of values or value sets (“still image;”, “text; image;”) a mix of semicolonand comma-separated values pluralization (books vs. book) values that were otherwise misspelled or were better expressed in fields other than the type field inspecting idhh type metadata initial analysis using the dpla analytics dashboard [9] and the dpla oai aggregation tools [10] originally developed by the north carolina digital heritage center and modified by the idhh, only scratched the surface of what appeared to be widespread issues with idhh-contributed type metadata. while the aggregation tools would reveal type metadata within a single collection, large-scale data analysis could not be easily accomplished with them due to their out-of-the-box design that evaluates one record set at a time. idhh and dpla staff made several attempts from 2017 to 2018 to analyze the metadata of the entire idhh record set with openrefine. due to the size of the set and possibly insufficient available memory on the machines facilitating this effort, openrefine failed to parse the dataset after multiple attempts. moreover, although openrefine supports gathering and analyzing xml through oai-pmh queries, it was unknown to idhh staff, even after extensive research, if and how openrefine can handle resumption tokens. thus, the only method known to idhh staff for dealing with huge oai-pmh sets with openrefine is the tedious process of manually entering resumption tokens. alternative methods were investigated in order to gather and analyze all the idhh type metadata, including bulk download [11] of json-ld data created by the dpla from the xml harvested from the idhh and calling the dpla’s api [12] for smaller samples of json-ld data. ultimately, the latter option was selected primarily because data analysis tools like openrefine work better with json data than with xml and because gathering data via the api would provide the most up-to-date data available from the dpla in a fairly small package. the download of a subset of metadata from all idhh records, including only each record’s type key-value pairs and four other fields discussed below was around 100mb, compared to the 1gb bulk download of all idhh metadata. a python script [13] was created to generate dpla api calls and to download all the json-ld data for each idhh data provider, as read from a utf-8 csv file. this script can run on python 2.7 or higher and requires the installation of the requests library [14], used for interacting with the dpla api. it may seem more obvious to use the api to harvest all records that contain the name or id of the hub in the provider metadata. the problem with this approach is that the api limits calls to 100 pages of 500 results each, or a maximum of 50,000 records. as only one of the idhh’s providers currently contributes more than 50,000 records and its metadata already conforms well to dcmi type, it was decided to gather collections by provider in order to work within the api’s limit. the following fields (see table 1) were selected from each record in every idhh-contributed collection, represented in the dpla api as a json-ld object: field name description dataprovider individual institution that provided record sourceresource.collection.title title of collection of which a given record is a part sourceresource.title title of the item an individual record describes isshownat url linking to the source instance of the record originalrecord.type original type value provided by idhh sourceresource.type dcmi type value created by the dpla table 1. json-ld fields selected with dpla api. the dataprovider and sourceresource.collection.title fields were selected in order to identify a given record’s contributing institution and collection. this allowed for faceting in openrefine by the institution that provided a record and by a record’s collection and thus, checking if problems with particular records represented widespread issues at the collection or institution level. similarly, sourceresource.title was chosen to identify an individual record, and isshownat was selected for easily linking to a record’s source instance. these fields allowed for greater refinement, especially in terms of faceting, grouping, and filtering the two type metadata fields available in the json-ld sample: the originalrecord.type, or the value provided by the idhh to the dpla, and the sourceresource.type, the value created by the dpla and available in its search interface if and when provider data has passed through the dpla’s quality controls, discussed above. for reference, here is an example json-ld object from the data gathered: "docs": [{ "sourceresource.title": "lt. col rhodes", "isshownat": "http://www.idaillinois.org/cdm/ref/collection/parkridg001/id/668", "sourceresource.collection.title": "mel tierney post servicemen file", "dataprovider": "park ridge public library", "originalrecord.type": "text", "sourceresource.type": "text" }] the script downloaded more than 900 json-ld data files containing the data from at least 94% of the idhh’s more than 300,000 records. the remaining 6% comprised of records provided by the single institution, discussed above, whose records number greater than 50,000 and thus exceed the dpla api’s call limits [15]. the json-ld files were then imported into openrefine 3.0 on the metadata specialist’s local machine and parsed in openrefine as json files, as shown in figure 4. figure 4. json-ld data fields were uploaded to openrefine using the ‘browse’ files feature. the initial openrefine table revealed that many records had two instances of the sourceresource.type or originalrecord.type fields with corresponding values due to the presence of two elements in the xml records originally provided to the dpla (see figure 5). figure 5. examples of two instances of type metadata values. further refinement was required in order to prioritize action items from the overwhelming results. the originalrecord.type facets and corresponding counts were downloaded from the original openrefine set and were imported into another openrefine sheet. facets were sorted alphabetically and case-sensitively. a blank down operation was performed to remove the duplicates that resulted from the two columns, multivalued cells were joined to get the total counts from the blank rows and the ones above them, and the multivalued cells were summed by creating a new column (see figure 6) based on the following google refine expression language code: foreach(value.split(','),v,v.tonumber()).sum() figure 6. openrefine’s “join multi-valued cells” feature the final product was a spreadsheet of 558 type values and their respective counts from all idhh collections. the number of unique type values revealed problems that could not be solved by the idhh through xsl transformation alone. in addition to mass-normalization of common values by the idhh, two other categories of issues were identified: those that would need to be redressed by the dpla and those that would simply need to be tackled by individual contributors refining the metadata they provide. solutions for data clean up idhh xsl enhancements much of the work of enhancing type metadata can be handled automatically through xsl stylesheet transformations applied in the idhh repox aggregation server. an xsl template was developed [16] for matching on the dc:type element and performing operations on the text node, e.g., the type metadata value the dc:type element contains. problems addressed with xsl many providers use commas or a combination of comma-separated values and, the more common delimiter, semicolons. occasionally, commas and semicolons appear in the same field. in the xsl templates for transforming type, a line of code utilizing the xpath replace() function converts commas to semicolons to normalize the delimiter. <xsl:variable name="typevalue" select="normalize-space(replace(., ',', ';'))"/> the tokenize() xpath2 expression is used to split multiple values and then, each token is inspected separately using a for-each loop which calls up a template for matching a token to the correct dcmi type. instead of relying on an exact match on a certain string value, the xsl transforms this lowercase string if it contains a certain keyword, using the xpath lower-case() and contains() functions. <xsl:when test="contains(lower-case($rawtype), 'physical') or contains(lower-case($rawtype), 'realia') or contains(lower-case($rawtype), 'object')"> <xsl:element name="dc:type"> <xsl:text>physical object</xsl:text> </xsl:element> </xsl:when> when the lower-case version of a record’s dc:type value contains a keyword such as “still,” as in the half dozen or so common variations of “still image”, this text node is to be transformed into ‘image’. <xsl:when test="contains(lower-case($rawtype), 'still')"> <xsl:element name="dc:type"> <xsl:text>image</xsl:text> </xsl:element> </xsl:when> xslt limitations xsl works on individual nodes. thus, if multiple dc:type elements are present, each will be passed through the normalization function. some records will have type values that will be transformed into the same value, for example: <dc:type>poster</dc:type> <dc:type>war poster</dc:type> will be transformed into: <dc:type>image</dc:type> <dc:type>image</dc:type> fortunately, the dpla can de-duplicate specific fields and has been asked to do so for the idhh’s type metadata. the keyword matches utilizing contains() described above will require continued careful attention to contributors’ type metadata to insure that the values are transformed properly. periodic tests, the latest of which at the writing of this paper was in 2019-05, show that the transformation has worked well for nearly half a year. however, there may eventually be newly-added type values that contain a certain keyword which will be converted to incorrect type values. clean up by individual institutions many problems boil down to unusual values appearing in the type metadata field that cannot be handled by xsl transformations and must be remediated by contributors. examples include phrases that appear to be more suited for an item description (dc:description) field or subject (dc:subject), urls that seem to be duplicate record identifiers, rights metadata, and odd misspellings of common type values. it is simply not practical to transform these often-unique values that nonetheless pervade several collections and severely impact their type metadata and discoverability. moreover, working with institutions’ staff allows them to improve the metadata at the local level and the usability of their digital collection environments. from the very first communications onward, the idhh respectfully engaged contributors, providing encouragement and instruction on remediating type and other metadata fields. this message included the purpose and significance of type metadata in the dpla, an overview of the dpla’s selected controlled vocabulary, dcmi type, as well as the steps the idhh is taking to ease at least some of the burden on contributors in conforming to the standards of dcmi type. the idhh took steps to help contributing institutions understand the issue and the best practices for type metadata. an online type metadata guide [17] for contributors was created and linked to on several other websites, including the idhh libguide [18]. regular office hours were held in order to offer the opportunity for contributors to discuss metadata remediation for type or other fields. deploying xslt and type metadata guide testing and deploying xslt the xslt was developed in test-and-go fashion on small xml data sets in oxygen editor 18.0 [19]. on 2019-01-24, immediately prior to the first dpla ingestion of 2019 for illinois, xslt was run for the first time on the idhh repox 2.3.7 server. key features of the transforms were tested by the harvest coordinator and metadata services specialist on several large datasets, including: the basic mapping from a provided metadata value to a dcmi type the transform’s effects on pluralized values, alternative spellings, white space, etc. the xslt’s handling of delimited values on comma and semicolon delimiters tests also confirmed the duplication of values predicted in the development of the xslt. however, no unexpected transformations occurred and the xslt was deployed for transformations on all records provided to the dpla on 2019-01-28. disseminating type metadata guide the type metadata guide for contributing institutions was disseminated through the first half of 2019. the idhh metadata best practices were updated to include the guide and it has also been posted on the idhh metadata working group website along with a list of values the idhh transforms to dcmi type. institutions are encouraged to contact the metadata services specialist at the idhh if they wish to recommend additions or other changes to the current mapping. measuring impact of the type metadata project the idhh type metadata project returned impressive results. before remediation efforts began, only 176,000 of 327,000 values (53.8%) conformed to dcmi type and the field held 558 unique values. of these values, only 20% of the idhh’s more than 300,000 records were showing up with type metadata in the dpla catalog. re-examinations of type metadata on 2019-02-15 and on 2019-05-20 using the python metadata downloader tool and openrefine showed remarkable improvements. the february check-up revealed that the number of unique metadata fields was reduced to 117. the number of dcmi conforming values as of 2019-02-15 was 300,985, whereas there were only 918 non-conforming values, meaning that 99.7% of idhh type metadata values conformed to dcmi, as shown in figure 7. figure 7. screen captures of type facets for all idhh items in the dpla, before (left) and after (right) the type metadata project. the february 2019 review also showed that there are still a few values that do not conform to dcmi type left over from the original list developed in 2018-11. these were generally facets which number fewer than 10 values each and thus were not made a priority in the project. in addition, there were a few new values that must have slipped in due to changes provider institutions had made to their metadata since it was initially downloaded and checked in openrefine in 2018-11 and 2018-12 and which had not been accounted for by the xsl. only two of these values had more than 100 instances. the may 2019 check-up again revealed high percentages of conformity to dcmi type. of approximately 308,000 originally-provided type metadata values, 307,000 conformed to dcmi type, maintaining the 99.7% conformity to dcmi type seen in the february review. there were 109 unique fields showing up in the analysis. there were only three non-conforming values with counts greater than 100 and the vast majority of these (over 86.5%) are being transformed by the dpla’s ingestion system. before project february 2019 may 2019 all type values = 327,000 301,903 = 308,000 conforming values = 176,000 300,985 = 307,000 % of conforming values = 53.8 99.7 = 99.7 unique values 558 117 109 % of idhh records with type metadata in dpla catalog 20 93 92 both reviews showed high percentages of completion of the dc:type metadata field and conformity to dcmi type. this may be allowing the dpla’s own quality control measures to do their best work. most of the values not transformed by the idhh’s xsl are transformed by the dpla’s ingestion system and, therefore, are no cause for concern. as of 2019-05-20, dpla analytics reports completion rates of the type field at 92%, a far cry from the 20% rates seen in december 2018 (see figure 8). this would likely be even higher if not for the fact that a number of contributors do not provide any type metadata. figure 8. type metadata completion rates, december 2018 (left) and may 2019 (right). note that type now has the highest rate of completion of any non-required field. it should also be noted that none of the problems foreseen by tokenizing multiple semicolon-separated type values manifested in either the february or may check-up. the dpla was able to de-duplicate type values, thus making normalizing delimiters and tokenizing them worthwhile. nevertheless, the idhh will continue to review the type metadata and transformations from time to time. next steps: new harvest and analysis systems and continued communication the extensive xsl transformations now in place on the idhh repox aggregation server make it impossible to analyze large samples of type metadata originally provided by contributors using json-ld data from the dpla api. before the xsl was deployed, the dpla received original values provided by idhh contributors and therefore, these same values were accessible in the api and through bulk download in the originalrecord.type field. after the xsl template for transforming type metadata was deployed, original values provided by contributors are transformed by the idhh before they are harvested by the dpla and thus are not available through the dpla api. therefore, in addition to existing tools that allow for the analysis of small samples of contributors’ metadata, a mode of analyzing large and, if possible, the complete idhh dataset of type and other metadata values directly from contributors will need to be developed. discussing new methods in depth is beyond the scope of this paper but may involve a change of harvesting platform from repox to one that provides more powerful data analysis, such as the michigan service hub’s solution now widely adopted by other dpla partners: combine [20]. type, like all metadata in a large-scale continuous aggregation, will be a moving target: current providers’ metadata may change and new collections with potentially new type metadata values are being added regularly. new collections’ metadata will continue to be thoroughly assessed. moreover, the idhh will continue to communicate with providers planning to improve their type metadata and other metadata fields. regular contact with the dpla will be essential. in addition to the ongoing discussions on the dpla’s role in normalizing provider data, it will be important to discuss any remaining or new issues involving the dpla’s recently deployed new harvesting system, ingestion 3 [21]. conclusions: a team effort the challenges of this project, particularly in gathering and analyzing the metadata, exposed some serious limitations in the idhh’s infrastructure and provided impetus and inspiration for new procedures and platforms. it also revealed the many uses of the dpla’s tools, especially its api and open apis in general. moreover, this project brought to light the growing importance of understanding multiple aspects of metadata work, including metadata harvest, analysis, and refinement, along with an ability to work with tools for each aspect. the project used several open source tools, including openrefine and the atom editor for developing python scripts. the type metadata analysis and refinement project was a testament to the challenges and rewards of working with aggregated metadata and required the cooperation of all parties involved: providers, the hub, and the dpla. about the authors joshua lynch is a metadata specialist at the university of illinois, urbana-champaign and the metadata manager and webmaster for the illinois digital heritage hub, a service hub of the digital public library of america. lynch’s professional experience and interests include metadata aggregation, curation, and remediation, web development, digital content creation, digital curation, and digital preservation. jessica gibson is the senior application support coordinator at the consortium of academic and research libraries in illinois (carli) and the harvesting coordinator for the illinois digital heritage hub. gibson’s current work finds her managing and supporting large, library-related systems, but she will never forget her cataloging roots. myung-ja k. han is the head of acquisitions and cataloging services at the university of illinois at urbana-champaign. her research interests include metadata in digital library environment, metadata interoperability, and linked open data. notes [1] for more information on service hubs and dpla’s structure, see the dpla’s documentation for prospective hubs: https://pro.dp.la/prospective-hubs [2] qualified dublin core (dcmi terms) specifications page: http://www.dublincore.org/specifications/dublin-core/dcmi-terms/ [3] idhh collection metadata assessment template: https://docs.google.com/document/d/1i46jjoehq5ki78vnwrbjr6a6dplzin_xv7iacd6lb3w [4] idhh metadata best practices: https://docs.google.com/document/d/1q1aorhoa0ey0fugotymhlvzncm6wq1qe9ddvfzsrpt0 [5] for example, dpla’s ingestion 3 module for type enrichment converts a growing list of currently 74 values to dcmi type: https://github.com/dpla/ingestion3/blob/develop/src/main/resources/types/dpla-type-normalization.csv. ingestion 3 performs similar operations on the language and format fields: https://github.com/dpla/ingestion3/tree/develop/src/main/resources. [6] dcmi type vocabulary specification: http://dublincore.org/documents/dcmi-type-vocabulary/ [7] dpla’s catalog interface: https://dp.la/search? [8] dpla’s ingestion 1 system: https://github.com/dpla/ingestion [9] dpla analytics: https://analytics-dashboard.dp.la [10] north carolina digital heritage center dpla oai aggregation tools: https://github.com/ncdhc/dpla-aggregation-tools [11] dpla bulk download: http://dpla-provider-export.s3.amazonaws.com/ [12] dpla api documentation: https://pro.dp.la/developers/api-codex [13] dpla api json downloader python script: https://github.com/jlynch2121/dpla-api-json-downloader [14] python requests library website: https://2.python-requests.org/en/master/ [15] this was not deemed a serious issue as 75% of this provider’s records were gathered from the api. analysis of the provider’s collections using the north carolina digital heritage center’s oai aggregation tools showed that they all conformed well to dcmi type. [16] xslt for type metadata enhancement: https://github.com/jlynch2121/idhh-xsl [17] type metadata guide for idhh contributors: https://ildplametadatawrkgrp.wordpress.com/documentation/type/ [18] idhh libguide originally created by hannah stitzlein, idhh metadata manager and metadata services specialist at the university of illinois, urbana-champaign from 2016-2018: http://guides.library.illinois.edu/idhh [19] the cost of oxygen may be prohibitive. the authors are also familiar with atom, a versatile free and open source editor for almost any language, including xsl. an atom plugin for transforming xml with xsl and testing transformations can be found at: https://atom.io/packages/atom-xsltransform. [20] combine documentation: https://combine.readthedocs.io/en/master/ [21] the illinois digital heritage hub metadata began to be harvested by ingestion 3 in 2019-07. dpla ingestion 3: https://github.com/dpla/ingestion3 subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – editorial — new name change policy mission editorial committee process and structure code4lib issue 53, 2022-05-09 editorial — new name change policy the code4lib journal editorial committee is implementing a new name change policy aimed to facilitate the process and ensure timely and comprehensive name changes for anyone who needs to change their name within the journal. by ron peterson with the publication of issue 53, the code4lib journal is implementing a new post-publication name change policy. drawing heavily on “a vision for a more trans-inclusive publishing world: guest article” and the sage publishing name change policy, the new policy seeks to lower the barriers for authors seeking to change their name or gender identity and decrease the risk of inadvertent disclosure within the code4lib journal. our goal in implementing this policy is to address the needs of all authors who seek to change their names for any reason, such as changes to gender identity, marriage status, religion and others. the editorial committee recognizes that individuals who are transgender, non-binary, and/or gender diverse are uniquely vulnerable to harassment and assault, as well as issues related to receiving proper credit for their work. our purpose with this policy is to not contribute to these issues and, where we are able to, provide relief. the code4lib journal will do its best to follow the 5 guiding principles and best practices laid out in the cope guest article: accessibility: name changes will be available to authors upon request without legal documentation, unnecessary barriers, burdens, or labor placed upon the author making the request. comprehensiveness: we will remove all instances of an author’s previous name from the records maintained and disseminated by the publisher.  invisibility: the journal will not draw attention to the gender identity of an author, nor create a clear juxtaposition between the current name and the previous name. expediency and simplicity: the journal will implement name changes in a timely manner, and with a minimum of bureaucratic overhead. recurrence and maintenance: the journal will regularly audit and correct new instances of changed names in order to prevent ongoing dissemination of incorrect information. the code4lib journal does not have the ability to update how these articles are represented in indexing and abstracting services, with the exception of the directory of open access journals (doaj) and must rely on the indexing services to pick up the changes to the publication. for doaj, new metadata will be submitted to update author information. if you would like to discuss this policy, or any other journal policy, i encourage you to start a discussion on the journal’s email discussion list, c4lj-discuss. we look forward to hearing from you. now for what you will find in this issue and as is typical we have something for everyone. for those of you interested in metadata we have articles like karen coyle’s article exploring the use of the library reference model’s (lrm) work, expression, manifestation, and instance concepts beyond the library and information science world. if that doesn’t meet your needs, maybe jenn randles and andrew bullen’s article, “citation needed: adding citations to contentdm records” is what you are looking for. there is also ross spencer’s article, “fractal in detail: what information is in a file format identification report?” about using the metadata in the file format identification report. we also have articles that will help you manage the workflow in your library. there’s “automated 3d printing in libraries” by brandon patterson, ben engel, and willis holle about responding to pandemic challenges by allowing users to request 3d printing jobs remotely. or if you need to automate referrals for reference assistance, stephen zweibel’s article, “automating reference consultations with javascript and a google form,” may be what you are looking for. chris diaz’s article, “lantern: a pandoc template for oer publishing” can help libraries looking to publish open educational resources in multiple formats. if your interests include digital content management, you might be interested in learning more about “strategies for preserving digital scholarship / humanities projects” at the university of toronto. kirsta stapelfeldt, sukhvir khera, natkeeran ledchumykanthan, lara gomez, erin liu, and sonia dhaliwal tell us about the work they are doing to plan for the long-term preservation of digital projects created at the university. you might also be interested in how shawn m. jones, himarsha r. jayanetti, alex osborne, paul koerbin, martin klein, michele c. weigle, and michael l. nelson are applying social media storytelling to a subset of a collection to facilitate collection understanding at a glance with their dsa toolkit. lastly for libraries needing to collect, manage, and curate information about the research activity produced at their campuses, daniel m. coughlin’s and cynthia hudson vitale’s article, “supporting open access, integrating distributed research platforms, and building a research information management platform” should give you some food for thought. i hope that you find these topics and articles as interesting and informative as i did and that they give you some ideas that you use to further the work that you are doing. have a great summer!   subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – emflix – gone baby gone mission editorial committee process and structure code4lib issue 33, 2016-07-19 emflix – gone baby gone enthusiasm is no replacement for experience. this article describes a tool developed at the emerson college library by an eager but overzealous cataloger. attempting to enhance media-discovery in a familiar and intuitive way, he created a browseable and searchable netflix-style interface. though it may have been an interesting idea, many of the crucial steps that are involved in this kind of high-concept work were neglected. this article will explore and explain why the tool ultimately has not been maintained or updated, and what should have been done differently to ensure its legacy and continued use. by netanel ganin introduction “what order is this in?” — patrons patrons love to browse. whether they’re at a library, retail establishment, or grocery store, serendipitous discovery always plays an important role in their selection. our classification schemes are designed to enhance this process; by placing similar items together, a patron can scan an entire section to find resources she didn’t even know she was looking for. unfortunately, some resource types, such as electronic resources, can’t be browsed at all on the shelves. videos and dvds simply aren’t suited to it. in march of 2014, i was working at emerson college, a small liberal arts institution with a strong communications program. the emerson library at that time held 100,000 books and other media, including roughly 3,000 dvds in the circulating collection. movies are often cataloged differently depending on the institutional policy. some institutions organize them by title, with no lc or dewey call number, others assign them to the pn1997 (plays, scenarios, etc.) section, then cutter them at title or director, and still others, including emerson college, class them in the pn1995.9.a-z (special topics in motion pictures). classing this way tied each film to a single genre/form/theme, even though films can and do belong to many categories. a foreign horror film had to be classed as foreign or as horror, it couldn’t be assigned both because it had to sit in a single physical location on the shelves. this struck me as a very outmoded “video-store” mentality. our patrons were much more accustomed to netflix, hulu, and amazon streaming services which, of course, allow for multiple access points in genre, form, and theme, as well as a visual browsing interface. i had an idea: not only would a new discovery interface allow for a better patron experience in dvd browsing, but it would also solve another problem that had stymied our patrons. students would regularly come to the desk expecting to be able to check out a specific movie only to discover that the movie they wanted was in the non-circulating collection. there was no way limit a search to the circulating collection only. there were an additional 4000 movies in the non-circulating collection all of which could only be watched in the library. furthermore, it was not possible to search for filmmakers within specific roles. a search for ‘steven spielberg’ would return movies he produced, executive produced, and wrote, whereas a patron may want steven spielberg as director. figure 1. screenshot of search results from emerson’s catalog. as figure 1 shows, there are 47 results for a search of ‘steven spielberg’ filtered to only include dvds and videos. of those 47, 30 were false positives (i.e. steven spielberg is in the record but not the director), and 22 of them required an additional click to determine if the item was in the circulating collection or not. i hoped and planned that my new interface could solve both of these issues as well. but as someone who not only hadn’t finished library school, but also hadn’t read any works about discovery interfaces, i ended up with a haphazard and erratic project. i didn’t follow any established methodology or framework, and after receiving a loose ‘go ahead’ didn’t consult with anyone at emerson about the work, it’s goals, or the process of achieving those goals. this is my story, learn from my mistakes. beginnings i want to point out that one of my stated goals was to save patrons from the frustration of getting hits from the non-circulating collection if they wanted to take the dvd home. that could’ve been solved by asking voyager to add ‘circulating media’ as a location along with ‘reserves’ and ‘reference’. that would’ve been a lot easier, but i didn’t think of that. instead, i began to build what i came to call emflix. i asked our systems librarian for the marc data from our catalog for the circulating collection. using microsoft access, she ran the following query: select mfhd_master.normalized_call_no, mfhd_master.display_call_no, utf8to16([bib_text].[title]) as title, utf8to16([bib_text].[title_brief]) as title_brief, utf8to16([bib_text].[uniform_title]) as uniform_title, utf8to16([bib_text].[author]) as author, bib_text.publisher, bib_text.begin_pub_date, bib_text.end_pub_date, utf8to16([bib_text].[pub_dates_combined]) as pub_dates_combined, bib_text.publisher_date, bib_text.language, getfieldall(getbibblob([bib_text].[bib_id]),"546") as 546, getfieldall(getbibblob([bib_text].[bib_id]),"500") as 500, getfieldall(getbibblob([bib_text].[bib_id]),"511") as 511, getfieldall(getbibblob([bib_text].[bib_id]),"520") as 520, bib_mfhd.bib_id, bib_mfhd.mfhd_id, mfhd_item.item_id, item_barcode.item_barcode from (((bib_text inner join bib_mfhd on bib_text.bib_id = bib_mfhd.bib_id) inner join mfhd_master on bib_mfhd.mfhd_id = mfhd_master.mfhd_id) inner join mfhd_item on mfhd_master.mfhd_id = mfhd_item.mfhd_id) inner join (item_barcode inner join item on item_barcode.item_id = item.item_id) on mfhd_item.item_id = item.item_id where (((mfhd_master.location_id)="373") and ((item.item_type_id) in ("19","225"))) order by mfhd_master.normalized_call_no; in hindsight, i should’ve asked questions such as what specifically was being retrieved by this query, or if i could get the full descriptive data. instead, i simply retrieved the download as an excel spreadsheet and saved it in xml. each entry in the resulting file looked something like this: <row> <heading0>pn 19959 c 55 s 68 2002</heading0> <heading1>[dvd] pn1995.9.c55 s68 2002</heading1> <heading2>1941 [videorecording] / universal pictures and columbia pictures ; a-team production ; screenplay by robert zemeckis and bob gale ; story by robert zemeckis, bob gale and john milius ; produced by buzz feitshans ; directed by steven spielberg.</heading2> <heading3>1941</heading3> <heading4></heading4> <heading5></heading5> <heading6>universal home video,</heading6> <heading7>2002</heading7> <heading8>1979</heading8> <heading9>2002-1979</heading9> <heading10>[2002]</heading10> <heading11>eng</heading11> <heading12>english dialogue, optional subtitles in french and spanish; closed-captioned.</heading12> <heading13>originally released as a motion picture in 1979. special features: restored footage not included in the original theatrical release; an original documentary on the making of 1941, including new video interviews with steven spielberg, bob gale, john miliu</heading13> <heading14>dan aykroyd, ned beatty, john belushi, lorraine gary, murray hamilton, christopher lee, tim matheson, toshiro mifune, warren oates, robert stack, treat williams.</heading14> <heading15>this comedy is set in los angeles days after the attack on pearl harbor, when the fear of a japanese invasion hung over the city.</heading15> <heading16>1336944</heading16> <heading17>1628403</heading17> <heading18>1461057</heading18> <heading19>0113503089311</heading19> </row> the mappings, drawn from the first row of the excel spreadsheet, are: <row> <heading0>normalized_call_no</heading0> <heading1>display_call_no</heading1> <heading2>title</heading2> <heading3>title_brief</heading3> <heading4>uniform_title</heading4> <heading5>author</heading5> <heading6>publisher</heading6> <heading7>begin_pub_date</heading7> <heading8>end_pub_date</heading8> <heading9>pub_dates_combined</heading9> <heading10>publisher_date</heading10> <heading11>language</heading11> <heading12>546</heading12> <heading13>500</heading13> <heading14>511</heading14> <heading15>520</heading15> <heading16>bib_id</heading16> <heading17>mfhd_id</heading17> <heading18>item_id</heading18> <heading19>item_barcode</heading19> </row> i then wrote an xslt stylesheet to transform the excel-as-xml data into an xml structure of my own design. the following two functions were important and complex parts of that stylesheet. name extraction function i used this function for extracting names from , which i determined was composed of most of the marc 245 field. the writer, creator, and director elements were constructed by this template in my home-grown xml structure. it calls a function “extract” with three parameters the text to be searched (in this case the value of ) the string markers to look for within the text the name of the element to construct (in this case ‘director’) <xsl:template match="heading2" mode="director"> <xsl:sequence select="functx:extract(normalize-space(.),('written, directed and edited by ','director/writer, ','directed &amp; produced by ','directors, ','direction by ','directed by ','director, ','directed and written by ','directed and produced by ','directed, written and produced by ','direction, '),'director')" /> </xsl:template> <!-extract function --> <xsl:function name="functx:extract" as="element()*"> <xsl:param name="input" as="xs:string"/> <xsl:param name="markers" as="xs:string*"/> <xsl:param name="element-name" as="xs:string"/> analyze-string joins all the markers together, separated by the regular expression ‘or’ operator and scans the input text. it stores all text which occurs after a marker phrase until it finds a space-semicolon (which in a 245 would indicate change of role), or the end of the input string. <xsl:analyze-string select="$input" regex="({string-join($markers, '|')})([^;]+)(\s;|$)"> the next analyze-string uses capture-group 2 (which would ideally be a person’s or multiple people’s names, following one of the marker phrases above) and would store any text preceding space-and-space, comma-space, or space-&-space. <xsl:matching-substring> <xsl:analyze-string select="regex-group(2)" regex="([^,]+?)(((\sand|,|\s&amp;)\s)|$)"> the element is constructed, and a sort attribute is added so that indexing could be done on the last name (you’ll note that it only builds the sort attribute if someone has a first and last name; if the person had three names, i looked it up myself to be sure which was their last name) <xsl:matching-substring> <xsl:element name="{$element-name}"> <xsl:if test="matches(regex-group(1),'^\w+\s\w+$')"> <xsl:attribute name="sort"> <xsl:value-of select="string-length(substring-before(regex-group(1),' ')) + 2" /> </xsl:attribute> </xsl:if> the name is inserted into the element and any trailing spaces or a period are stripped <xsl:value-of select="functx:substring-before-if-ends-with(normalize-space(regex-group(1)),'.')" /> </xsl:element> </xsl:matching-substring> </xsl:analyze-string> </xsl:matching-substring> <xsl:fallback> <xsl:element name="{$element-name}"/> </xsl:fallback> </xsl:analyze-string> </xsl:function> you may wonder why i didn’t simply extract names from the 700 fields. the answer is that i didn’t have any 700 fields in my data! this workaround, although complex, did teach me a lot about regex and its power for string manipulation. title capitalization function before calling the function, i removed trailing forward slashes, periods, and whitespace. <xsl:template match="heading3" mode="title"> <xsl:variable name="title" select="functx:substring-before-if-ends-with(functx:substring-before-if-ends-with((normalize-space(.)),' /'), '.')"/> the title is tokenized (broken down into pieces separated by a space) and each token is passed to the titlecase function. when it comes back it re-joins all the other tokens (in the same order) and the first letter of the entire title is capitalized. <xsl:sequence select="functx:capitalize-first(string-join(for $x in tokenize($title,'\s') return functx:titlecase($x),' '))" /> </xsl:template> the titlecase function takes a single parameter, in this case a token of the title. if the token (converted to lowercase) matches any of these lower-cased-title-words, it returns the token in lowercase. if the token is entirely uppercase it returns unchanged (based on my assumption that that word is being treated differently in the title). if neither of those are true, the first letter of the token is capitalized.[1] <xsl:function name="functx:titlecase" as="xs:string"> <xsl:param name="token" as="xs:string"/> <xsl:choose> <xsl:when test="lower-case($token)=('a','aboard','about','above','absent','across','after','against','alongside','amid','amidst','among','amongst','an','and','around','as','aslant','astride','at','athwart','atop','barring','before','behind','below','beneath','beside','besides','between','beyond','but','by','despite','down','during','except','failing','following','for','from','in','inside','into','like','mid','minus','near','next','nor','notwithstanding','of','off','on','onto','opposite','or','out','outside','over','past','per','plus','regarding','round','save','since','so','than','the','through','throughout','till','times','to','toward','towards','under','underneath','unlike','until','up','upon','via','vs.','when','with','within','without','worth','yet')"> <xsl:value-of select="lower-case($token)"/> </xsl:when> <xsl:when test="$token=upper-case($token)"> <xsl:value-of select="$token"/> </xsl:when> <xsl:otherwise> <xsl:value-of select="concat(upper-case(substring($token, 1, 1)), lower-case(substring($token, 2)))"/> </xsl:otherwise> </xsl:choose> </xsl:function> fun fact about titles: the way the data i had was structured, all non-filed articles were stripped. so i had to add back every ‘the’, ‘a’, and ‘an’. through the full xslt transformation (inevitably followed by some clean up), the example record above became my home-grown xml structure: <media id="1336944" datecreated="2014-07-18"> <title>1941</title> <director sort="8">steven spielberg</director> <actor sort="5">dan aykroyd</actor> <actor sort="5">ned beatty</actor> <actor sort="6">john belushi</actor> <actor sort="10">lorraine gary</actor> <actor sort="8">murray hamilton</actor> <actor sort="13">christopher lee</actor> <actor sort="5">tim matheson</actor> <actor sort="9">toshiro mifune</actor> <actor sort="8">warren oates</actor> <actor sort="8">robert stack</actor> <actor sort="7">treat williams</actor> <genrewrap> <genre>comedy</genre> <subgenre>spoofs and satire</subgenre> </genrewrap> <summary>this comedy is set in los angeles days after the attack on pearl harbor, when the fear of a japanese invasion hung over the city.</summary> <lcspecialtopics>comedy</lcspecialtopics> <writer sort="8">robert zemeckis</writer> <writer sort="5">bob gale</writer> <language>english</language> <year>1979</year> <callnumber href="http://endeavor.flo.org/vwebv/holdingsinfo?bibid=1336944">[dvd] pn1995.9 .c55 s68 2002</callnumber> <coverart href="pics/1941.jpg"/> </media> at this point i’d add the genres, subgenres and sub-subgenres to the film. i knew that genre analysis of over 3000 films would be the biggest challenge, and it certainly turned out to be the most tedious. rather than have to analyze them all myself, i decided to use a pre-existing database of films with genre analysis already complete: netflix. for every film that we owned, i’d enter its name into the netflix dvd database (not the streaming database) and manually entered the genres that netflix assigned into my structure. it was necessary to be logged in as a registered user to find this data. as our collection was fairly small relative to theirs i saw that i didn’t actually need all of netflix’s possible genres,[2] lest a single category be populated with 1 or even 0 items. to that end i made myself a thesaurus mapping all of netflix’s genres to the ones i’d actually be using. i also made many modifications to the netflix genre, some for hybrid-genre consistency, others for inclusion reasons. interface what i now had was a lot of movies in my own home-grown xml structure, and i needed to convert it into something patrons could search and browse. because i didn’t know how to use relational databases (and i didn’t even know that that’s what i should have been doing at this step), i decided to turn the entire structure into static html pages, one for each genre, subgenre, and sub-subgenre. xslt 2.0’s for-each-group functionality was invaluable at this point for grouping each set of films. i had struggled with trying to understand muenchian grouping [3] which was the technique used in xslt 1.0 for grouping sets. for the searching, i learned about php’s simplexml functions, and decided to separate results by where they were found in the record. searching a person’s name would return movies divided by their role in the film. the most important piece of code in the search functionality was another tokenizing function. by separating the inputted string into tokens i was able to use xpath’s contains() function to see if any given field contained the whole search phrase regardless of term order. $initialsearch = $_get["searchterm"]; $contains = array(); $str = $searchterm; $str_arr = explode(' ',$str); $i=0; $contains[0] = 'contains(translate(., "'.$upper.'","'.$lower.'"), "'.$str_arr[0].'")'; for($i=1; $i < count($str_arr);$i++){ $contains[$i] = ' and contains(translate(., "'.$upper.'","'.$lower.'"), "'.$str_arr[$i].'")'; } $csv_var = implode('', $contains); $directorcheck = $xml->xpath('/medialist/media[director['.$csv_var.']]'); $writercheck = $xml->xpath('/medialist/media[writer['.$csv_var.']]'); $actorcheck = $xml->xpath('/medialist/media[actor['.$csv_var.']]'); $mediacheck = $xml->xpath('/medialist/media['.$csv_var.']'); phase 4 – use testing and leaving emerson i sent a url of the site to the entire staff and student employees of the library to gather feedback. in my original implementation there were neither summaries of films nor actors of the films. both of these were heavily requested, so i went back to include them. before the testing was complete and the site was set to be launched, i accepted a position at another institution. i left my successor with a 23 page document detailing the idiosyncrasies of how to maintain and update emflix in my absence. many of the particulars and rules i had come up with were incredibly complicated. for example, because my thesaurus would deauthorize any genre, subgenre or sub-subgenre which had fewer than 5 films, as new films were added to the database, i maintained a separate spreadsheet marking how many films could fit in a deauthorized genre in case sufficient films bearing that genre were added to authorize it. another idiosyncrasy was the structure of the element itself. each had to hold one element and then as many elements as necessary. every element added to the element had to immediately follow a element. therefore if adding two elements from the same hierarchy, the element which was its broader term would have to be repeated. e.g., if a film were to be assigned ‘dramas based on bestsellers’ and ‘dramas based on contemporary literature’ the structure would have to look like: <genrewrap> <genre>drama</genre> <subgenre>dramas based on the book</subgenre> <subsubgenre>dramas based on bestsellers</subgenre> <subgenre>dramas based on the book</subgenre> <subsubgenre>dramas based on contemporary literature</subgenre> </genrewrap> despite my efforts to facilitate continuation of the project, in my heart i didn’t expect that anyone who hadn’t invested as much time as i had would be willing or able to maintain the project. every new entry required a certain amount of manual entry, and i’d estimate that it could have taken an average of 5 minutes per movie. at the rate at which movies were acquired, that would have meant an extra 8-10 hours of work a month. in truth, i did most of my entries at home, off-work time. not only is that bad work-life-balance, but it also created a false sense of time requirements to others who saw my progress. to date, no new films have been added, and the project has languished. lessons learned the primary take-away from this project was that i did too much alone. by this i mean that i wasn’t participating in any larger conversations in library communities. partially this is because i didn’t feel that i was ready to join those conversations as someone who was still a student in library-school. mostly however, it was that i’d been struck with an idea, and i didn’t think i needed to ask anyone else. i thought i’d invented the ‘simple’ solution that no one else had ever thought of for media discovery, and i ran with it. the more i worked on it and the more it grew, the less i felt i could stop and ask questions because i’d already invested so much time in what i’d done. if i’d researched vendor platforms, asked how other libraries were handling the issues i saw, or even asked peers if they wanted to work on it with me, i could have avoided many of the mistakes i made that led to the project’s current (and future) stagnation. i highlight some of those mistakes here. i should not have made my own xml structure. i spent a great deal of time designing and re-designing a schema when many metadata schemas already existed which would’ve served my purposes just fine. designing my own schema also meant that the data i was creating had interoperability with neither our own catalog, nor anyone else’s. it meant having to write lengthy documentation on the meaning of each attribute and element, and what values to enter into them. i’m particularly chagrined at how little thought i gave to enhancing the records already in our catalog. though i made thousands of genre assignments in my own xml, i never thought to explore the possibility of adding library of congress genre/form terms to our catalog’s records. emflix may not be in use anymore, but emerson’s catalog certainly is and if i’d focused on enhancing those records while working on the side-project, that data would still be useful today. i should have directly contacted the people responsible for our visual materials cataloging and collaborated with them to make the data i was generating reusable in the main library catalog. another area i wish i’d explored more was bringing in end-user (primarily student) opinions earlier. i think i spent so long building the site i lost sight of what it was supposed to be for. i wrote an xsd file to handle the basic validation of the xml, but because i didn’t know about any other xml validation languages, i also wrote xslt ‘error-checkers’ to validate some pieces that xsd cannot do but other languages like relaxng can. this created some code-bloat and increased the load for anyone hoping to take over the project. i also should have pressed harder to get the full bibliographic records in my dataset downloads. having access to the 7xx fields would have made retrieving the names of people associated with each film significantly easier. i never should have brought netflix into the equation. using a commercial database which requires a subscription to log-on is not a good model on which to sustain a library project. each genre (or subgenre or sub-subgenre) page of films was a static list that was manually generated by an xslt transformation and then uploaded to the server. i was expecting people to browse by genre and then drill down as needed, each subgenre nested inside its parent genre, etc. it was only after i learned some php (which wasn’t until i was working on the search functionality) that i realized that each genre ‘link’ could have actually passed a search term to the server to retrieve all the same films with that genre. it would have saved me a tremendous amount of time because i was actually duplicating all hybrid genres. for example, if a movie were assigned ‘action thriller’ as a subgenre, that movie’s structure would contain: <genrewrap> <genre>action and adventure</genre> <subgenre>action thrillers</subgenre> </genrewrap> <genrewrap> <genre>thrillers</genre> <subgenre>action thrillers</subgenre> </genrewrap> notice that ‘action thriller’ has to appear twice because i couldn’t know if a user would browse through ‘action and adventure’ or through ‘thrillers’ and i had to be sure that they’d find the same set of movies regardless. some sub-subgenres were actually repeated three times, e.g. ‘foreign classic dramas’. if i’d known how to use php when building the browse functionality, i would’ve written a simple line of code like: $genrecheck = $xml->xpath("/medialist/media[genrewrap[* = '$initialsearch']]"); so that when a user would click a genre (or sub-genre or sub-subgenre) it would retrieve all the films that had been assigned that term, regardless of its parent element. therefore i wouldn’t have had to add all those reciprocal subgenres. conclusion emflix may be dead, for all intents and purposes, and though i have many regrets — i’m not sorry that i embarked on the project. i learned a tremendous amount about xslt and its capabilities. i wrote my first lines of php, and more css than i ever had before. i’ve already been able to apply those skills to other projects. i’m not nearly ready to serve as a mentor to another eager cataloger, but i am equipped to provide advice and warnings. if i got wind of someone else jumping headlong into creation of an entirely new tool, i’d be there to tell them to slow down. having been through the experience myself, i’d make sure they spoke to peers, consulted the literature, and brought others into the work. it’s on this reflection that i see these as the largest places i went astray, the many important steps i missed or neglected to explore. but most importantly, i know that i can avoid those mistakes in the future, because the most valuable thing i learned over the course of the emflix project was that i had the ability to realize an idea. i imagined a ‘netflix-style’ discovery interface for emerson’s circulating media collection and, though imperfectly, i made one. [4] so on to the next–a little more slowly, and a lot more wisely. notes [1]list drawn from here: http://libraryguides.lanecc.edu/c.php?g=391380&p=2658274 [2]http://dvd.netflix.com/allgenreslist [3]for more on this method, see the brilliant jeni tennison, http://www.jenitennison.com/xslt/grouping/muenchian.html [4]full copy of site: http://netanelganin.com/projects/emflix/genrephp/index/index.php about the author netanel ganin netanel is the kind of person who would make an authority record for his cat, and he did. he can be reached at netganin@gmail.com and can be found at @oponions. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – choose your own educational resource: developing an interactive oer using the ink scripting language mission editorial committee process and structure code4lib issue 51, 2021-06-14 choose your own educational resource: developing an interactive oer using the ink scripting language learning games are games created with the purpose of educating, as well as entertaining, players. this article describes the potential of interactive fiction (if), a type of text-based game, to serve as learning games. after summarizing the basic concepts of interactive fiction and learning games, the article describes common interactive fiction programming languages and tools, including ink, a simple markup language that can be used to create choice based text games that play in a web browser. the final section of the article includes code putting the concepts of ink, interactive fiction, and learning games into action using part of an interactive oer created by the author in december of 2020. by stewart baker introduction “you have died of dysentery.” anyone who attended a u.s. elementary school in the 1980s or 1990s will likely recognize this as a game over message from the oregon trail, a computer game intended to teach students about the emigration of settlers to oregon in the 1850s. although the game presents indigenous peoples largely as props or obstacles to overcome and implicitly accepts the idea of manifest destiny (bigelow, 1997), it was hugely successful and remains iconic even today. interestingly, the earliest version of the oregon trail was a text-only game released in 1971 (bouchard, 2019), making it one of the earliest computer games designed for educational purposes and arguably a notable early example of interactive fiction (if), computer games where text, rather than advanced graphics or skill with a controller, are the main game element. this article will explain the basic concepts of interactive fiction and learning games and will show how to use interactive fiction scripting language ink to create a learning game that can be easily distributed as an open educational resource (oer). what is interactive fiction? interactive fiction, commonly abbreviated as if, is a type of video game “where the player’s interactions primarily involve text” (iftf, n.d.). although the idea of text-only video games seems quaint in the 2020s, if was big business in the 1980s, when the processing power of personal computers was limited. infocom’s 1979 fantasy exploration game zork, one of the most successful if games, sold over 150,000 copies in 1984 alone at roughly $50 a piece (carless, 2008). today, there is a thriving hobbyist community of if authors and players, with the interactive fiction database (https://ifdb.org/) describing more than 11,000 published games, 468 in the year 2020 alone. much if today is created as part of “comps” (competitions) and “game jams” (friendly competitions based around a theme), although there are commercial publishers as well. the biggest competition is ifcomp (https://ifcomp.org/), which offers prizes and reviews to more than 100 entrants annually, while independent game publishing site itch.io (https://itch.io/jams) hosts hundreds of jams each year. if is typically split into two distinct types: parser games are pieces of if where the player must type commands into a ‘parser’ to complete tasks in the game world. early if games were usually parser games. choice-based games are pieces of if played in a web browser or browser-like environment. in choice-based games, players use web design elements such as links to navigate a set of pages with varying degrees of sophistication and complexity. some purists argue that parser games are the only kind of games that can properly be considered if, with choice-based games instead being works of “hypertext” (montfort, 2003). in practice, however, most if designers and players today include both choice-based and parser games in definitions of if. this paper will primarily discuss a specific choice-based if scripting language, inkle studio’s ink (https://www.inklestudios.com/ink/), but otherwise uses the term “if” to refer to both choice-based and parser games. writing interactive fiction driving the boom in contemporary if is its strong sense of community. outside of the competitions and game jams, there are any number of thriving forums and discord servers dedicated to playing and writing if. an additional outcome of this community focus is that there are a number of scripting languages developed specifically to create new works of if, almost all of which have authoring tools that are freely available to writers. in fact, there are so many tools, as if developer emily short points out, that the first step to starting a new if project is to select which tool you will be using (short, n.d.). a few of the most popular languages and tools are listed below: twine (http://twinery.org/) is a choice-based authoring tool that creates if games which run in a web browser. twine is a good choice if you want to include javascript-based effects in your work. twine’s authoring tool can run in the browser or a stand-alone client can be downloaded for mac and windows. inform (http://inform7.com/) is a parser game programming language first created in 1993. the latest version, inform 7 uses natural language programming concepts. the authoring tool for inform is a stand-alone client that runs on mac, windows, and linux. players must have access to an “interpreter” program to run inform games, although authors can also host games in web-based interpreters on a web server. tads (http://www.tads.org/tads.htm) is a parser game programming language based on object oriented programming principles. the tads authoring tool is a stand-alone client that runs on mac, windows, and linux. players typically need access to an interpreter program, but the latest version (tads 3) also allows authors to export their games to web pages. tads 3 is notable also for its ability to integrate graphics and other web objects with games. ink (https://www.inklestudios.com/ink/) is a scripting language for creating choice-based games. created by inkle studios to produce games, ink is noteworthy for its ability to integrate with unity to serve as the dialogue engine for more complex video games. it can also be used to produce choice-based if that runs in a browser. ink’s authoring tool is a stand-alone client that runs on mac, windows, and linux. inkle studios also maintains a visual web-based tool for writing if in ink called inklewriter (https://www.inklestudios.com/inklewriter/), created with the aim of enabling writers without a programming background to write interactive stories more easily. choicescript (https://www.choiceofgames.com/) is the language used by commercial if publisher choice of games. choicescript is particularly useful for games where the player character’s “stats” or abilities affect the player’s choice. unlike other languages, choicescript’s license does have restrictions on creating and publishing games outside of choice of games’s official line of games; however, games written in choicescript can be distributed non-commercially without issue (fabulich, 2010). interactive fiction as learning game learning games (sometimes called “serious games,” a term this article will not use as it implies non-learning games are frivolous) are games designed to “help players develop new knowledge or skills or to reinforce existing knowledge or skills” (boller and kapp, 2017) rather than solely for entertainment purposes. although results vary, learning games can be effective ways to teach. one meta-analysis found that 29 out of 40 reviewed studies showed that learning games had a positive effect on learning (backlund and hendrix, 2013). likewise, studies have shown that roleplaying games and simulations can benefit students (kim, 2018; overland, 2017). studies of if learning games are scarce outside of composition studies, where the focus is often on students creating their own if rather than using it to learn about another topic. however, there is nothing that implies if should be any less effective than more visually-focused games at providing education as well as entertainment. indeed, if’s relatively low barriers to creation could well make it an attractive option for librarians and teachers without the extensive skillset required to produce well-designed full-featured video games or the budget to support a team of developers. if games also tend to be small in terms of file size and are more likely to be platform-agnostic, especially for choice-based games that typically run in a browser window. this means if has the potential to be distributed easily in oers, institutional repositories, and other web-based spaces teachers, librarians likely already use to provide instructional materials to students. of course, just because if can be considered a “low-tech” approach (harvey, 2018) doesn’t mean writing good if is easy. the effectiveness of a learning game depends on part in whether its players enjoy its narrative quality and ease of use, among other things (fokides, atsikpasi, kaimara and deliyannis, 2019), and if there aren’t splashy graphics or other elements associated with “proper” video games involved, it’s even more important to pay attention to the details of the prose and make sure your code is bug-free. although the majority of if produced today is written with entertainment, rather than education, in mind, examples of if intended to educate and entertain at the same time do exist: depression quest (https://depressionquest.com) – written in 2013 by game developers zoë quinn, patrick lindsey, and isaac schankler, depression quest simulates the experience of clinical depression by showing players choices they can’t take due to depression. the game was critically successful, with reviewers stating that the game helped them to understand what it was like to suffer from depression but was still fun to play (vasquez, 2014). depression quest is written in twine. when rivers were trails (https://indianlandtenure.itch.io/when-rivers-were-trails) – an oregon trail like game written in unity which centers indigenous peoples, intended to teach players about the impacts of colonization on their communities (lapensée and emmons, 2019). a bathroom myth (https://adeniro.itch.io/a-bathroom-myth) – although not a ‘learning game’ in the traditional sense, anya johanna deniro’s a bathroom myth is a game with a mission, using the trappings of a fantasy setting to build empathy and activism around anti-transgender bathroom laws. the game is written in twine. harmony square (https://harmonysquare.game/en) – developed by psychologists jon roozenbeek and sander van der linden in conjunction with the u.s. department of homeland security, harmony square teaches players about misinformation tactics by letting them roleplay as a “disinformation minister” hired to sow discord in the fictional community of harmony square. although the game has graphics, they are of secondary importance and the core of its interaction is handled by reading and responding to text. a study by the developers showed that the game improved people’s ability to spot misinformation and reduced the likelihood that they would share it on social media when they found it (roozenbeek and van der linden, 2020). these games and others like them show the potential of text-based games to reach a large audience and to educate and entertain. introduction to ink as noted above, there are many different authoring tools for creating if. this section of the article will take a look at inkle studio’s ink programming language, explaining its basic concepts and providing snippets of code. to reiterate, the different if languages and tools do very different things. ink is a useful language to write learning games in for several reasons: it is designed around a simple markup language, lowering the need for programming knowledge its games export to web pages, which can be hosted online and embedded in oers and instruction guides exported games can be styled using css without the need for additional coding knowledge inky (ink’s authoring tool) has good built-in error checking and outputs code as you write it inkle’s web-based inklewriter tool can be used to write games without code players only need to be able to read text and interact with links, and don’t need to learn parser commands on top of the game’s subject matter game content can easily be converted to a text-only, printable version more advanced developers can also take advantage of ink’s integration possibilities. the language has been used to create not just text-based games but critically acclaimed narrative games with heavy graphical components, including 80 days (https://www.inklestudios.com/80days/), where the water tastes like wine (https://www.wherethewatertasteslikewine.com/), and heaven’s vault (https://www.inklestudios.com/heavensvault/). like most if languages, ink has extensive online tutorials, including inkle’s official documentation (inkle, 2016a) and various community-authored guides (secchi, n.d.). ink basics technically, you can write ink files in any text editor, but use of inky (inkle’s standalone authoring tool for ink) is highly recommended as it makes debugging and playtesting significantly easier. inky is required to compile ink games into playable standalone files. to output text in ink, all you need to do is write that text into inky. it was a dark and stormy night. ink will parse this as content, outputting it immediately. as seen below, the end of an ink file will also produce an “end of story” notice to let players know they’ve run out of content. figure 1. a line of text in inklewriter. the left side of the screen shows code and the right side shows output. the core of any choice-based game is its choices, and ink is no exception. choices players make drive the story of the game they are playing, affecting everything from what their character does to the information they are able to learn from non-player characters (npcs). in ink, choices are represented by lines of code that start with an asterisk (*). it was a dark and stormy night. *what kind of storm? the rain fell in torrents. from this example, ink will output a line of content followed by a clickable hyperlink. figure 2. a choice in inklewriter. on the right side of the screen, the editor has rendered the choice into a clickable link. when the player clicks the hyperlink, its text will turn into plain text and the remainder of the content will be displayed. figure 3. inklewriter’s output changes after the user clicks the link, turning the choice into plain text and outputting the remainder of the content. multiple choices can be displayed at once by including multiple lines that begin with asterisks and placing the relevant response after each. ink ignores whitespace characters, so it can be a good idea to indent code to help you keep track of your game’s hierarchy. it was a dark and stormy night. *what kind of storm? the rain fell in torrents. *where are we? it is in london that our scene lies. this code will produce the following output: figure 4. multiple choices are rendered into multiple links. below each choice is the content inklewriter will output if that choice is selected. chapters (knots), scenes (stitches), and choices in ink any game of more than a few lines will need to take advantage of knots and stitches, ink’s two main “structural units” (inkle, 2016a). knots are the largest unit and can be thought of as chapters in a novel. stitches can be used to further subdivide knots, like scenes inside a chapter. knots and stiches are important not just because they make it easier to remember where things are in the code of larger games, but because they break game content up into sections that can be accessed by ink’s “divert” feature. like a “goto” line in other programming languages, diverts redirect the player to different parts of the code. knots are represented by lines of code that start with two or more equal signs. ink’s documentation uses three equal signs before and after knot names, but this is an aesthetic choice to make large games more legible and is not required. stitches are lines of code that start with a single equal sign. to add a divert, insert an arrow made up of a hyphen and less-than symbol (->) in front of the name of the stitch or knot you want ink to go to next. diverts can appear anywhere in a line. while they typically appear after choice text, they can be used to move players from the end of a knot or stitch, or any other arbitrary place, to elsewhere in the game. *[what kind of storm?]->storm if you want to direct the player to a stitch inside a knot other than the one where the divert is located, you must list both the knot name and the stitch name separated by a period. *[what kind of storm?]->story.storm knots, stitches, and diverts allow ink to create sophisticated games where the choices the player makes move them through a narrative and affect its outcomes. let’s take a look at a complete, albeit very short and uninteresting, game in ink. it was a dark and stormy night. *[what kind of storm?]->story.storm *where are we?->story.where ==story =storm the rain fell in torrents. ->conclusion =where it is in london that our scene lies. ->conclusion ==conclusion at occasional intervals, the rain was checked by a violent gust of wind. ->end this code will produce the following initial output: figure 5. the code view now shows knots and diverts. since no choice has been selected, the output shows only the first sentence and the first two choices available to the user. if the player selects “what kind of storm?” the output will append the content from the “storm” stitch in the “story” knot before appending the content from the “conclusion” knot along with the “end of story” notice. figure 6. after the user has selected “what kind of storm?” as a choice, inklewriter outputs the content in the story.storm stitch and then immediately outputs the content in the conclusion knot, followed by the “end of story” notice. ink also gives game designers options for how to display choice text. by default, choice text becomes text content when the player selects it. you can manipulate what displays by the use of a pair of square brackets in choice text, as shown in the example code above. choice text that comes before brackets will appear in both the choice text and as content after the player selects it, text inside brackets will only appear in the choice text, and text after the brackets will only appear as content after the choice has been selected by the player. in this case, because the “what kind of storm?” choice is entirely enclosed in brackets, the choice text does not appear at all in the outputted story after that choice has been selected by the player. one important thing to note is that knots and stitches must include a divert statement at the end of each choice they contain to avoid a “ran out of content error.” if no existing knot or stitch is appropriate, you can type ->done or ->end to indicate that ink should go to the end of the current knot or stitch or the end of the program, respectively. advanced ink it’s possible to write satisfying games in ink using nothing but choices and stitches as described in this article. ink can do much more than that, though. in addition to managing variables, functions, and logic, ink supports choices that appear more than once, random selection of items in a list, and many other things. ink’s official documentation contains a lot of detail, as well as numerous code snippets and examples (inkle, 2016a). if you’re the sort of person who learns by doing, you can copy a ready-to-play game from the official documentation and paste it into inky to get a better understanding of how the language works before starting on your own project (inkle, 2016a). inkle also maintains a document on how to run ink in the unity development engine or any other c# environment (inkle, 2106b), although this will only be useful to those with existing knowledge of how to create games in unity. rather than duplicating ink’s documentation on its advanced features, the remainder of this article will look at a finished ink game created as part of an oer. scholcom 202x: an interactive oer in late 2020, i learned about an open submission period for the scholarly communication notebook  (https://lisoer.wordpress.ncsu.edu/notebook/), a planned repository of oers that “reflect and encourage diversity in scholarly communication.” the aim of the scholarly communication notebook is to provide new and student librarians with education not just about scholarly communication concepts but about social justice concepts as they apply to a library setting. these requirements made interactive fiction a good fit for several reasons. since if is presented as a story, rather than a non-fiction resource, it offered the potential to build in diversity with a wide cast of non-player characters. likewise, if’s status as a narrative work could be used to offer a simulation-like experience for the player, allowing them to put scholarly communication principles into action rather than just reading about them. with this in mind, i proposed an interactive oer called “scholcom 202x.” rather than using the present-day, i decided to set the oer in a futuristic, “cyberpunk-lite” library, both because i thought a unique setting would make the experience more engaging and because it would let me draw on my decade of experience as an author of science fiction short stories. the main science fictional element is an “augment,” an imaginary mobile device the player can consult at key moments throughout the game and which provides them with an annotated bibliography and an overview of how busy they are. after my proposal was accepted, i spent several months researching, drafting and coding the game, as well as producing a non-interactive pdf version for use with in-person roleplay exercises or if a user would rather read static text than play a game for whatever reason. the final draft of scholcom 202x is an ink file which presents the player with ten distinct scenarios a scholarly communication librarian might encounter, each with its own set of choices to make and resources to consult. as players move through scholcom 202x, they make choices about how much time they want to spend helping the npc in each scenario. this adds a time management aspect to the game, tracked using variables and a simple function. i also wrote a function to let the player choose whether they wanted to see all ten scenarios or play a shorter game that only presents a few at random. in total, these scenarios and functions made up 13,637 words of content, with each scene reproduced in equivalent pdf files. a presentation about the process, given at the online northwest conference in march of 2021, contains more information (baker, 2021). the remainder of this article will look at several pieces of code from scholcom 202x to show how ink and if concepts can be used in practice to create an oer. tracking the player’s choices with variables and functions each time a player makes a choice in scholcom 202x, they receive reputation points based on how helpful that choice is. at the same time, though, high-reputation choices fill up more of the player’s work schedule. both numbers are tracked using variables declared at the very beginning of the ink file for the game, where they’re set to a default value of zero: var schedule = 0 var rep = 0 each time the player makes a choice at the end of a scenario, they receive between 0 and 2 points to both of these variables depending on how helpful they are being. for example, middling choices earn the player 1 reputation point and add one point to their schedule: ~rep++ ~schedule++ although the reputation points are invisible to the player until the end of the game, players can check how busy they are towards the end of each scenario by selecting a choice marked “check your calendar” before they decide how to respond to that scenario’s npc. although code for this could be reproduced in each section, it makes more sense to define a function that stores the code and call it in each relevant location. {checkcalendar()} this code calls the function defined below, which checks the value of the “schedule” variable and outputs text according to how much work the player has promised to do for people: === function checkcalendar() === { -schedule == 0: it's completely blank. <em>nice</em>! -schedule > 0 && schedule <= 3: there are a few things on there, but it's not too bad. plenty of time to pursue your own research on top of work. -schedule > 3 && schedule <= 7: hmm, it's starting to fill up a bit. you'd better not take on any more major projects, or you'll regret it. -schedule > 7 && schedule <= 10: <em>man</em>, there's a lot on there. so much for your lunch break, you guess. -schedule > 10: it's a mistake. you close the calendar again quickly. those blocks of time where you're double-booked were just an illusion, right? you hope you won't have to spend <em>too much</em> time working on the weekends and evenings to get everything done... } at the end of the game, two similar pieces of code give the player an idea of how helpful they’ve been and how busy their schedule is. because these are only used once, a function was not used and the code was simply inserted into its own stitch for use with a direct, like any other content that comes after a choice. a scholcom 202x scenario in one scholcom 20x scenario, an npc approaches the player to ask about oers. the code reproduced below is split into sections for readability but is a single 215-line piece of code in the game itself. the section begins with a stitch called oerstart (i didn’t use knots as the game was not long or complex enough to need them), then guides the player through a conversation with the npc that ends in a set of choices. =oerstart //9: a faculty member wants to find oers for use in a class <h2>scenario {days}: the case of the algebraic oer</h2> it's around noon, and you're cleaning up your lunch things in the staff lounge after a hard morning's work and a satisfying meal of riced with steamed vegetables. nobody's around — apparently you eat earlier than your new colleagues — and so when your augment buzzes, you go ahead and accept the call. it's from an adjunct faculty member in the math department named luis nunes, and they don't list pronouns or much other information on their public feed. "hi luis," you say. "what can i help you with?" * they speak, but no sound comes through.->oer2 =oer2 "sorry," you interrupt, "i think your mic is set to mute." they frown, concentrate for a minute, then try again. "how's that?" "much better." "great. i was told you could help me find free textbooks for an algebra class i'm teaching next term. is that correct?" "oers, you mean?" they shrug. "are those textbooks?" * "sometimes!->oer3 =oer3 the acronym stands for open educational resources. basically, you can think of them as any kind of resource that can be used in a classroom setting — that includes textbooks, but there are other options, too. anything from short chapters to videos with attached quizzes to, well, even educational games." luis looks skeptical. "and they're free? the department's supposed to be looking into reducing student costs for our basic classes, since they're required by so many majors and minors on campus." "that makes sense," you say, trying to get back into their good graces. "and yes, oers are definitely free. they're released under a special kind of license which gives people permission to read them." * that earns you a tight nod.->oer4 =oer4 "but are they good quality? what if i can't find one that works the way i want? what if something in it is <em>wrong</em>?" you smile. "they're written by professors and other experts, just like textbooks are. and if the way one approaches a subject is at odds with what you do in your class, you can usually modify it under the terms of the license. it depends, but..." you shrug, not wanting to overwhelm them with too much information. "there are lots of options for stuff like that, basically." another nod. "okay, sounds good. so what now?" what indeed? ->oerchoicesstart =oerchoicesstart * [check your augment.]->oersources * [check your schedule.]->oerschedule * [respond to their question.]->oerchoices note that each part of the conversation ends in a “choice” with only one option. this breaks the lengthy dialogue up into smaller, more digestible chunks. for added engagement, multiple dialogue options could be offered throughout the section to portray the back and forth of a conversation more realistically. at the end of this piece of code, the player must make their first real choice of the section: deciding whether to answer right away, check how busy they already are (using the function explained earlier), or consult a brief annotated bibliography on the topic. including the string “https://” in ink code gave me problems, so i inserted correctly formatted urls after the game was completed by directly modifying the javascript file ink creates during the last step of the export process. =oersources cox, g., & trotter, h. (2017). an oer framework, heuristic and lens: tools for understanding lecturers' adoption of oer. open praxis, 151-171. www.learntechlib.org/p/181421/ cox & trotter discuss necessary factors for the adoption of oers at any given institution, and by any given instructor. they suggest that approaching oer through a lens of institutional culture is the most effective way to encourage faculty to adopt oers for their courses if the infrastructure in the region supports their adoption. oer commons. (n.d.). open textbooks. oer commons. retrieved november 25, 2020, from www.oercommons.org/hubs/open-textbooks hundreds of textbooks released to the public under an open license. the larger oer commons site also has other types of resources. openstax. (n.d.). openstax. retrieved november 25, 2020, from openstax.org/ openstax textbooks are free to use, and also include integrations with blackboard, canvas, and other popular learning management systems. openstax is run by rice university, who also offers webinars on how to use their system. university of minnesota. (n.d.). open textbook library. retrieved november 25, 2020, from open.umn.edu/opentextbooks/ university of minnesota's open textbook library contains more than 800 openly licensable textbooks. ->oerchoicesstart =oerschedule you pull up your calendar on your augment. {checkcalendar()} ->oerchoicesstart note that both these stitches end in diverts back to the oerchoicesstart stitch which contains the three choices at the end of the preceding code snippet. this way, the player can explore all the sources and their calendar before deciding how to respond to the npc’s question. once they click the “respond” choice, they are directed to a new set of choices. =oerchoices you take a deep breath. time to decide. * "have you tried searching on g**gle for them yet?"[]<br /><br />->oerslacker * "let me send you some links."[]<br /><br />->oerresponsive * "do you have some time now? i can show you where to look and what to look for."[]<br /><br />->oeroverachiever =oerslacker luis signs off looking pretty offended, and you try not to laugh. but, really, this stuff is <em>super</em> easy to find. why shouldn't they do a little bit of legwork for their own classes? you do feel a little guilty, though, so you quickly email them a few links that might be useful, with an apology if you were rude. who knows if it'll make them less annoyed, but at least it should help out a bit. ->scenarioend =oerresponsive luis signs off with a brusque "thanks," and you try to avoid rolling your eyes. all the same, whether or not someone is a bit rude doesn't mean you shouldn't help them. and so many students have to take the core math classes that the knock-on effects will be huge if the department adopts oers instead of making everyone buy expensive textbooks. you pull together a few links and some quick explanations and send them luis's way, along with some suggestions on how they might modify any oers they like, or integrate them into their classes. if they have further questions, they'll just have to get in touch again. ~rep++ ~schedule++ ->scenarioend =oeroverachiever "i do," luis says. they interrupt you every few minutes as you're searching for oers and explaining them, to the point where what should have taken a few minutes lasts more than an hour. fortunately, you didn't have anything else scheduled, and at the end of the call luis seems pretty pleased with the oer you've helped them select. "i can't wait to show the rest of the department this," they tell you. "it's even better than our current texts in some ways, and the fact that we can just update it ourselves is going to be a gamechanger! their enthusiasm is catching, and you find yourself energized as well once they sign off, ready to move on to your next task. ~rep+=2 ~schedule+=2 ->scenarioend after reaching the end of the scenario, the player is diverted to the “scenarioend” stitch, which produces one of several messages suggesting the end of the workday and then moves them to another scenario at random, taking into account how long of a game they elected to play and which scenarios they have already completed. once the player completes enough scenarios, the code in the “scenarioend” stitch diverts them to the very end of the game where they are given a summary of their time as a scholarly communication librarian. conclusion learning games can be effective ways to deliver educational information to students. interactive text-based games, which require fewer technical skills to create and play and which can be directly hosted on websites and repositories for ease of distribution, may be particularly attractive for small teams of educator-developers or individual writers who lack advanced programming knowledge. this article has focused on the ink scripting language, showing how it can be used to create an interactive narrative game where player choices affect the outcome. although any if language has its own pros and cons, ink’s flexibility and relatively simple markup make it a good choice for anyone writing a text-based game for the first time. how helpful scholcom 202x will be to new and aspiring scholarly communication librarians is an open question, as is how effective text-based learning games are in general. i hope to explore these questions with future projects, although i think the research on graphic learning games also speaks to text-based games’ potential as educational entertainment. for now, anyone who wants to try out scholcom 202x for themselves can play the final version of the game online via the project’s github repository or the scholarly communication notebook blog. references baker s. choose your own education: interactive fiction and online instruction. online northwest. 2021. available from https://pdxscholar.library.pdx.edu/onlinenorthwest/2021/schedule/5/ backlund p and hendrix m. educational games – are they worth the effort? a literature survey of the effectiveness of serious games. 5th international conference on games and virtual worlds for serious applications (vs-games). ieee; 2013. available from https://curve.coventry.ac.uk/open/file/68df0eec-4e9b-4c70-8577-5cc2424b5856/1/educational%20games.pdf bigelow b. on the road to cultural bias: a critique of the oregon trail cd-rom. language arts. 1996; 74(2): 84-93. available from https://www.jstor.org/stable/41482844 boller s and kapp km. play to learn: everything you need to know about designing effective learning games. alexandria, va: american society for training and development; 2017. bouchard rp. a brief history of the oregon trail game. [internet.] 2019. available from http://www.died-of-dysentery.com/stories/brief-history.html carless s. 2008. great scott: infocom’s all-time sales numbers revealed. [internet.] 2008. available from https://web.archive.org/web/20080924074642/http://www.gamesetwatch.com/2008/09/great_scott_infocoms_alltime_s.php fabulich d. choicescript license v1.0. [internet.] 2010. available from https://github.com/dfabulich/choicescript/blob/master/license.txt fokides e, atsikpasi p, kaimara p, and deliyannis i. factors influencing the subjective learning effectiveness of serious games. journal of information technology education. 2019; 18. available from http://jite.org/documents/vol18/jitev18researchp437-466fokides5777.pdf harvey a. games as education in the united kingdom. journal of learning and teaching in higher education. 2018; 1(1). available from https://journals.le.ac.uk/ojs1/index.php/jlthe/article/view/2539 inkle. writing with ink. [internet.] 2016. available from https://github.com/inkle/ink/blob/master/documentation/writingwithink.md inkle. running your ink. [internet.] 2016. available from https://github.com/inkle/ink/blob/master/documentation/runningyourink.md interactive fiction technology foundation (iftf). n.d. frequently asked questions about interactive fiction. available from https://iftechfoundation.org/frequently-asked-questions/ kim e. effect of simulation-based emergency cardiac arrest education on nursing student’s self-efficacy and critical thinking skills: roleplay versus lecture. nurse education today. 2018; 61: 258-263. available from https://doi.org/10.1016/j.nedt.2017.12.003 lapensée e and emmons n.  indigenizing education with the game when rivers were trails. amerikastudien. 2019; 64(1): 75-93. available from https://www.academia.edu/download/61875819/indigenizingeducation_wrwt20200123-72845-1yf0b9m.pdf montfort n. twisty little passages. cambridge, ma: mit press; 2003. overland c. using roleplaying simulations and alternate reality gaming to develop professional behaviors in pre-service music teachers: a qualitative case study. contributions to music education. 2017; 42: 107-127. available from https://eric.ed.gov/?id=ej1141729 roozenbeek j and van der linden s. breaking harmony square: a game that “inoculates” against political misinformation. harvard misinformation review. 2020. available from https://misinforeview.hks.harvard.edu/article/breaking-harmony-square-a-game-that-inoculates-against-political-misinformation/ vasquez j. why depression quest matters. game revolution. [internet.] 2014. available from https://web.archive.org/web/20140819144011/http://www.gamerevolution.com/manifesto/why-depression-quest-matters-27657 secchi m. ink. [internet.] n.d. available from https://marcosecchi.github.io/resources/pages/gamedev_storytelling_ink.html short e. writing if. [internet.] n.d. available from https://emshort.blog/how-to-play/writing-if   about the author stewart baker is the systems & institutional repository librarian at western oregon university, as well as a science fiction and fantasy author and poet outside of work hours. he has been writing interactive fiction since 2015, and his interactive oer written in ink, “scholcom 202x,” is forthcoming in the scholarly communication notebook. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – improving the drupal user experience mission editorial committee process and structure code4lib issue 12, 2010-12-21 improving the drupal user experience drupal is a powerful, but complex, web content management system, being adopted by many libraries. installing drupal typically involves adding additional modules for flexibility and increased functionality. although installing additional modules does increase functionality, it inevitably complicates usability. at the university of houston libraries, the web services department researched what modules work well together to accomplish a simpler interface while simultaneously providing the flexibility and advanced tools needed to create a successful user experience within drupal. this article explains why particular modules were chosen or developed, how the design enhanced the user experience, how the cms architecture was created, and how other library systems were integrated into drupal. by rachel vacek, sean watkins, christina m. morris, and derek keller i. background the university of houston libraries [1] web services department first started using drupal [2] in spring of 2008 when we migrated our intranet from text-heavy, poorly organized and outdated static html pages to a more robust and flexible installation of drupal 6. as we were new to the platform and excited about its potential, we quickly began extending the code and adding requested functionality. the new system was well received, looked great, improved internal communication and met the needs of those wanting advanced features. however, as we gained experience in drupal development practices, some problems were solved with different solutions of varying quality. the system grew in complexity and as a result, the overall user experience began to suffer. content editors began consistently needing technical assistance from web services as the multitude of blocks, views, menus and custom content types were too confusing. many aspects of the system were so over-engineered that it became more and more difficult to know the purpose of many content types, much less how to output them on a page. in fall of 2008, the libraries began the initial stages of redesigning the main website. our existing cms was a custom system built on adobe coldfusion and microsoft sql server. although well liked by most library staff for being robust, flexible and modular, we were uncertain about adobe’s ongoing commitment to support coldfusion. the technology has aged and it has become difficult to find skilled coldfusion developers. within web services, one staff member was dedicated to full-time development and improvement of the entire system, thus not allowing for opportunities to work on other projects within the libraries. it was quickly apparent that we wanted a more open and vibrant cms platform with an active development community. after the favorable intranet migration experience and with the growing number of libraries in the drupal community [3], we chose to move forward and use drupal as the cms for our main website. we decided to learn from our initial mistakes and focus primarily on a system that was both powerful and easy to use. at the time, drupal 7 was still an unstable beta version, there was no definitive release date and many modules had not yet been ported. we decided to use drupal 6 for primary rollout of the website while minimizing (or at least simplifying) functionality so that a future upgrade to drupal 7 would be as easy as possible. our top priorities were to engineer a system that would be a much more pleasant experience for library staff to work with content, and could fluidly handle rapid changes to information architecture. ii. drupal user experience (ux) a critical concern arose during our initial experience with stock installations. users of all ranges of technical skill were confused and frustrated by unfamiliar drupal terminology, methodology and poor administrative interface elements. it quickly became apparent that we needed to adopt a new ideal while planning this second usage of drupal for the libraries. we needed to help users edit content quickly and easily, but not force them to think about drupal and its unfamiliar conventions. even the wider drupal community has acknowledged that drupal 6 doesn’t provide a very pleasant user experience, and there are many group initiatives that are attempting to improve it within drupal 7, such as the dux7 project [4]. the drupal project lead, dries buytaert [5] summarizes the need for improving the user experience within drupal: drupal’s steep learning curve filters out far too many smart, motivated people who could benefit from drupal. we see it all the time in the drupal.org forums, in my “state of drupal” surveys, on twitter, when talking to customers, and on the web. even though we’ve made significant progress with making drupal easier to use, a lot of work is left to be done. with other content management systems such as joomla! and wordpress making strides to catch up to drupal in terms of development flexibility, if we want drupal to remain competitive, we have a challenge we have to face: we need to create a user experience that makes it easier for people new to drupal to discover all of its richness and power. [6] during our research, we discovered one such project that seemed to exemplify a quality user experience: open atrium [7], a drupal installation profile for intranets. for users, any confusing drupal conventions are hidden behind a thoughtful, cleanly designed and organized interface while allowing developers access to the powerful back-end. this was exactly the idea we wanted to emulate. iii. building drupal in this section, we share what the most important modules are that we used, created, modified, and why we chose or built them. not all modules mentioned are referenced because they are either part of core or because they are custom built. modules are the building blocks to making our library staff and content creators’ user experience within drupal a successful experience. a. standard drupal modules admin module one of the bigger causes of confusion and frustration for end-users is drupal’s administrative back-end pages. the admin module [8], used with the clean administrative theme rubik, separates all administrative navigation into a clean and concise side menu that is much more familiar to the casual user familiar with other cmss. the rubik theme was set for all editing screens, giving a consistent interface through which to work with content. figure 1. drupal administration interface using the admin module book module the committee responsible for assembling content for the new website did so in an extremely structured, hierarchical way in order to ease maintenance and tracking over time in the future. drupal’s built-in book module specializes in hierarchically sequenced content, making it a natural fit to implement this defined tree-like outline. used alongside the menu breadcrumb [9] and pathauto modules, the book module provided a flexible way to manipulate content hierarchy while retaining auto-defined breadcrumb navigation trails with easily readable urls. the books module also was able to output the entire tree hierarchy as a linkable sitemap. it became trivial to create or delete subpages as well as move a page and its children to a different area of the structure. certain pages on the site were set to use the php input filter to run simple blocks of php code. the site map content, for example, was generated by a few lines of modified code that used the books module api functions to browse the site structure and build the map. content construction kit (cck) module another valuable drupal module is the content construction kit (cck) [10], which adds a flexible interface for creating custom fields attached to content nodes. for the libraries’ website, a right-hand side content area was needed on pages to communicate information such as related links or contact information. since we knew that library staff already struggled with creating and maintaining blocks on the intranet, we needed a simpler solution. we added a set of cck fields called “right block title” and “right block content” to the book page content type. drupal template files (tpl) were then created to customize the output of these field nodes and css was used to style appropriately. in short, users could create a content block on the right-hand side of the page without having to create, configure, and place the block on a page. figure 2. a second wysiwyg editor is added to the page so users can easily add content to the “right block” section without having to interact with the blocks module. figure 3. the “right block” feature is styled appropriately and displays on right side of the page. content locking module since we planned for the cms functionality to be as lightweight as possible to handle very structured content, there was no rigorous control or workflow system put in place. much of the content was to remain low maintenance and fairly static over time. however, during rush editing periods, it soon became apparent that we needed something in place to help protect users from accidentally interrupting or overwriting the work of others. the content locking module [11] was a small and simple solution to block concurrent page edits. any page edited by a user is then locked from others until changes are saved. this has worked well with our expected low volume of users and edits, but a higher volume would likely call for revisiting this system and inclusion of a more complete workflow system. taxonomy module only a small percentage of the library staff needs to create and manage content, and a web oversight committee must approve any major revisions. in lieu of a more traditional and complicated cms workflow where users are tightly controlled, we opted for flexibility similar to a wiki-type system where content editors have access to edit any page. with protection against deletions or errors in place via restricted drupal node permissions and revision control, we only needed an easy way to associate users with the pages for which they are responsible. a commonly used module provided in drupal core is taxonomy, which suited this purpose well. a list of content editing groups was defined as taxonomy terms, with some term names based on department and others on particular function. all book pages are assigned to any number of editing groups, and in turn each user is also manually assigned to a set of editing groups that are most appropriate. for example, someone who is responsible for a computer access policy would be added to the policy editing group. a “my content” administrative page was then created using php code to list all groups to which the user was a member and all pages belonging to those groups. figure 4. content creators see the groups they belong to and the pages owned by that group. as there is no cms-imposed access control being enforced for editors, using taxonomy in this fashion is merely a way to help them more easily navigate to the content they care about. this works in a low volume, structured editing environment but does allow for occasional editing accidents that must be reverted. cas module the uh libraries has pushed to more fully utilize a trusted single sign-on authentication system so that users only have to enter their username and password once to be logged in to all specified library applications. we successfully implemented cas, or central authentication service, a web-based system distributed by the jasig project [12]. drupal integrates seamlessly with cas through the phpcas library and drupal cas module [13]. diff module with the relative lack of editing restriction in the cms, it is important that we have dependable safety nets in place when things go wrong. the diff module [14] allows us to see nicely formatted differences in edits across revisions so that problematic edits may be more quickly spotted and reverted. recaptcha module on any website accepting user-submitted input, spam is always a problem. fortunately, drupal has many captcha [15] solutions available. the recaptcha module [16] syncs with the recaptcha web service to protect against web form spam submission, and offers an audio version for those with vision disabilities. this allows us to meet accessibility requirements from the state of texas. views module a single page was needed that would both list all past library events and grow as more were created. the views module [17], a highly valuable smart query builder, easily made it possible to utilize drupal events by automatically displaying the expanding list without requiring content editors to manually maintain the page content. pathauto module pathauto [18] is another essential module for drupal that automatically generates human-friendly url path aliases for content nodes. a path consisting of “node/123” is not very descriptive and would be more helpful if related to the page’s content. pathauto works with the book module so that the book hierarchy of the site is matched in the url without the content creator needing to think about how to create the url. for example, having a book page in the site called staff directory that is under the about section would result in pathauto generating the url path alias “about/staff-directory.” which makes more sense and is easier to remember than “node/43”. wysiwyg module the wysiwyg module [19] enhances usability by replacing the basic text-editing box with a rich text format box. this allows comfortable formatting functionality similar to that of common word processors. the tinymce editor was chosen for use with wysiwyg due to its clean and fast interface, stable performance and customizability. in addition, the add-on tinybrowser makes it easy for users to browse and upload any of multiple media files to be included on the page. the result is a familiar interface for the simultaneous management of text and media content for a page. figure 5. the editing window using wysiwyg and tinymce php code in content fields a drupal “developer” role was created and assigned permission to use php code within content node fields (although generally frowned upon and a potential security concern if not handled properly). if small bits of functionality are needed in a single area of the site that do not justify the time or work to create a custom module, php code is used when approved on a case-by-case basis. in the future, any further extension of that code will then likely be transferred into a proper drupal module. b. custom built drupal modules tiny gallery module library content editors needed a simple image gallery that would scroll through selected images uploaded by various content providers. many existing drupal gallery modules were either too limited or simplistic, while others provided so much functionality that the content creation process became confusing. as a result, we created the tiny gallery module to fit their needs simply and easily. figure 6. the admin part of the tiny gallery module lets users add additional galleries or edit existing ones. tiny gallery displays a transitioning slideshow with each image providing a title and url that, when clicked, links the user to a content page with related information. the module itself is created as a single drupal block which then displays on any number of a given set of url paths as defined in the tiny gallery configuration. the single block design keeps the administrative block page small and reduces overall block processing time. permalink module the usage of the book and pathauto modules often resulted in drupal url’s that were lengthy and cumbersome depending on the depth of location within the site hierarchy. these url’s were difficult to handle when printed in physical library materials, where readers need to type the addresses into their web browser via keyboard instead of a simple mouse-click or copy and paste. web services created the permalink module to add a custom, shortened path field to book pages based on the title of that page. the path links are auto-populated when content types are created, but can be manually changed afterward by the content editor. if a link has already been created with that title, an incremented number is simply appended to the end of the path. in addition, permalinks are available as fields and can be added to any content type, not just book pages. figure 7. when the page is created, users can either use the automatically generated permalink or can manually create one that is more appropriate for their needs. in this example, the employment page has a simple permalink that matches the page title. iv. integration with other library systems drupal may do many things well, but it doesn’t do everything. we have many other web-based applications in the libraries, and one of the challenges was seamlessly integrating all the various systems with the new cms for both application and end users. the first four of the five modules listed below are custom-built modules. homepage search system (hss) module along with the new main website, we launched a discovery platform: summon [20], branded as onesearch. onesearch was to become the primary search option on the main libraries’ homepage. however, we needed to display multiple search box options to users, not just onesearch, and do it in a way that wasn’t confusing to users by presenting too many search boxes. unfortunately, no existing drupal modules met our specific needs. web services created the hss module for the homepage and all the branches of the libraries to easily administer the homepage search box, allowing editors to create any number of separately ordered tabs, each with appropriate title, description and input form. where no html exists for a particular search form to be copied and pasted, drupal provides the ability to create those forms as drupal blocks that can then be simply chosen from within the hss module. while the submission of the summon search form currently redirects the user to the separately-hosted summon website, there are plans to utilize the summon api through ajax to return form results directly within the appropriate library or branch homepage. figure 8. the university of houston libraries homepage displaying the homepage search system module (left), as well as the tiny gallery module (right), and hours module (top right). electronic database system (edbs) module like most libraries, we needed electronic resources and their associated metadata from our integrated library system (ils) [21] to display on the website. a custom module called edbs was created to integrate with drupal so that any change in the cataloged databases would then be reflected within the drupal site with little staff interaction or redundant entry of data. the module reads a marc-format text file output from the ils consisting of all of the available databases and related metadata. it then automatically performs operations such as setting categories, marking databases for full-text and media formats, and the displaying of new databases within a specified timeframe. as the marc file lacked subject fields as part of the record, the module provides administrative pages for subjects to be manually created and assigned to databases. the edbs module [22] was an instant success with librarians and catalogers due to the easy user interface and integration between systems, which resulted in a much simplified workflow process and reduction of manual intervention. figure 9. edbs has a user-friendly interface for librarians to choose what databases they want listed under a particular subject. library hours module the drupal community provides a few modules to handle schedules and display hours of operation, but an existing system in the uh libraries already managed that information with an established workflow. the library hours module [23] was created to integrate with that system and make the entire process much more efficient. an hours text file is output on request from the ils, processed within drupal and then displayed on a page with a custom user interface to display hours of operation in a variety of formats. an advanced feature allows the creation of a block to be placed within the drupal theme to display the current day’s hours for a particular library branch. this automated integration has helpfully reduced the workload on the web services department for this previously tedious task, and has resulted in faster and more accurate display of scheduling for the libraries. staff directory system (sds) module the main goal of the sds module was to provide a means to sync staff information not only over other drupal installations, but other web services the uh libraries has implemented as well. the module is implemented with both server and client sub-modules. the sds server sub-module houses the main staff directory, editable by our human resources department to keep staff profile information current. individual staff members may also manage certain fields of their profile including contact information, background, subject area responsibilities, links to sites of interest and other related information. along with the libraries’ main site launch, research guides created with campusguides (the academic and more feature-rich version of libguides) [24] were introduced. links to librarian subject and course guides are dynamically pulled from the sds server and displayed on their individual profile pages. the sds client sub-module is for use on other drupal installations. the client uses the restful api component of the sds server to pull profile information for display within the staff directory page, caching this list for future use if the sds server somehow becomes unavailable. the sds module has greatly eased the process of distributing staff information, and library staff themselves have been much more happy with only needing to update their profiles in a single place. blogs blogs have become a popular and valuable means of communication between staff and patrons of the libraries. an installation of wordpress [25] has been in place with good reviews from content editors due to its pleasant usability experience, and many blog entries and comment discussions have populated the installation. one issue that caused some thought was whether to migrate this system from wordpress to drupal. after some research, we found the blog module included with drupal core severely lacks in functionality and finesse. we initially planned to recreate wordpress features within drupal, but after weighing the costs of time, effort and potential user disruption, we decided to simply integrate our wordpress content into drupal. the libraries’ blogs are updated frequently. while the content and editing process for our main site is rigid and the drupal backend is lightweight, we wanted the more structured workflow that wordpress offers which would more appropriately handle the volume and type of usage. content editors continue to manage blog content in their familiar interface, while drupal pulls certain entries such as news, emergency information or event announcements from wordpress using either rss or the wordpress api [26] to display the content on the main site. this architecture allows each application to take advantage of the power offered by the other with the least administrative work for the web services department. v. reflections & successes traditionally, throughout various iterations of the libraries’ website, different types of complex cms back-ends have been built to revisit the challenges of the last. none were unsuccessful at accomplishing their goals, but a common challenge among all was a lack of fluidity with regard to changing ideas toward content itself throughout the libraries. with this latest rewrite, the web services department decided to approach things from the standpoint that focus and priority of the entire project would be on the information architecture itself. we chose to use drupal due to its power, reputation and active developer community. we learned from the mistakes of our intranet project, and have designed a system using the “use only what you need” philosophy. while sacrificing fine-grained controls and restrictions, this lightweight foundation will allow us to expand flexibly with minimal need for rewriting functionality, and should flow much more smoothly with changes in information architecture as the libraries see fit. we have overcome a number of longstanding hurdles. there has been a marked increase in interest and participation among content editors as they feel more included in the user experience process, which we hope will continue as we further refine editing features. integration with other library systems has become even easier, and with automation and simplification freeing up time and budget for more projects, we are excited about further possibilities. while our implementation of drupal may not be altogether common in some aspects, it is entirely suited to this site rewrite. the beauty of drupal is its flexibility in solving various problems. the design we have used for this project would not be appropriate for an intranet installation, for example, but the drupal development methodology we have learned will definitely let us repurpose drupal exactly as we would need for upcoming projects. we have also been pleased with the outreach of interest and support from other libraries in the drupal developer communities, which was unexpected at this early of a stage in the process. we have received valuable support through other universities and drupal developer conferences, and have been contacted by other libraries asking for support on projects of their own. this new-found participation is extremely valuable and would have been unlikely had we chosen another software platform. some of the custom modules mentioned in this article will be available through the drupal site. while adopting drupal as a cms platform is no easy undertaking, the growing community of skilled and helpful developers is greatly narrowing the learning curve. drupal may be an intimidating system at introduction, but this can be overcome with patience, planning for what needs to be done, thinking less about the technology itself and more about project needs, and by being conservative with what functionality and features are used. there may be many different ways to solve a given problem with many different levels of complexity, but an initial investment in simplification and respect for user experience can yield an extremely powerful system run by satisfied, happy users. references [1] the university of houston libraries, in houston, tx, is comprised of the m.d. anderson library and three branches: the music library, optometry library, and the architecture & art library. available from: http://info.lib.uh.edu/ [2] drupal is a free and open source content management system written in php and distributed under the gnu general public license. [cited 2010 sept 27]. available from: http://drupal.org/ [3] libraries using drupal. [cited 2010 sept 3]. available from: http://groups.drupal.org/node/13473 4] drupal 7 user experience project. [cited 2010 sept 27]. available from: http://www.d7ux.org/ 5] dries buytaert’s personal website. he’s the original creator and project lead for the drupal open source web publishing and collaboration platform. [cited 2010 sept 27]. available from: http://buytaert.net/ [6] dries buytaert’s overview of why the d7ux project is essential. [cited 2010 sept 27]. available from: http://www.d7ux.org/about/ [7] development seed’s open atrium product is a drupal-based intranet and team portal package. [cited 2010 oct 15]. available from: http://developmentseed.org/product/open-atrium/ [8] admin module. [cited 2010 oct 15]. available from: http://drupal.org/project/admin [9] menu breadcrumb module. [cited 2010 oct 15]. available from: http://drupal.org/project/menu_breadcrumb [10] content construction kit (cck) module. [cited 2010 oct 15]. available from: http://drupal.org/project/cck [11] content locking module. [cited 2010 oct 15]. available from: http://drupal.org/project/content_lock [12] central authentication service (cas), provided by the jasig community. [cited 2010 oct 15]. available from: http://www.jasig.org/cas [13] cas module. [cited 2010 oct 15]. available from: http://drupal.org/project/cas [14] diff module. [cited 2010 oct 15]. available from: http://drupal.org/project/diff [15] captcha is a program that protects websites against bots by generating and grading tests that humans can pass but current computer programs cannot. [cited 2010 oct 3]. available from: http://www.captcha.net/ [16] recaptcha module. [cited 2010 oct 15]. available from: http://drupal.org/project/recaptcha [17] views module. [cited 2010 oct 15]. available from: http://drupal.org/project/views [18] pathauto module. [cited 2010 oct 15]. available from: http://drupal.org/project/pathauto [19] wysiwyg module. [cited 2010 oct 15]. available from: http://drupal.org/project/wysiwyg [20] serials solutions’ summon. [cited 2010 oct 15]. available from: http://www.serialssolutions.com/summon/ [21] innovative interfaces’ millennium ils. [cited 2010 oct 15]. available from: http://www.iii.com/products/millennium_ils.shtml [22] we are in the process of moving the edbs module to the drupal repository under the terms of the gnu general public license and expect it to be available by april, 2011. [23] we are in the process of moving the library hours module to the drupal repository under the terms of the gnu general public license and expect it to be available by april, 2011. [24] springshare’s campusguides. [cited 2010 oct 15]. available from: http://springshare.com/campusguides/ [25] up until wordpress 3.0 came out in july 2010, we had been using wordpress mu which supported multiple blogs and multiple authors. wordpress mu was integrated into version 3.0 and we were still able to have multiple blogs and offer the same functionality. [cited 2010 oct 15]. available from: http://www.wordpress.org/ [26] wordpress api. [cited 2010 november 17]. available from: http://codex.wordpress.org/wordpress_api’s about the authors rachel vacek is the head of web services at the university of houston libraries and manages the libraries’ web presence. she presents regularly at local and national conferences and was a 2007 ala emerging leader. she enjoys playing with drupal, experimenting with mobile technologies, thinking about the future of libraries, and gets teased frequently by colleagues about her social media and world of warcraft addictions. she can be contacted at revacek@uh.edu. sean watkins is a web developer at the university of houston libraries. he has been developing web application services since 2002 and spends most of his time typing php code. he enjoys building servers, creating new web services, and managing his world of warcraft guild. he can be contacted at slwatkins@uh.edu. christina m. morris is a web developer at university of houston libraries and has 14 years of experience in web and user interface development. she’s also proud to be a kingslayer on two different characters in world of warcraft. she can be contacted at cmmorris2@uh.edu. derek keller is a web developer at the university of houston libraries with 15 years of tech experience. he intelligently refuses to play world of warcraft but is somehow accepting of magic the gathering. he can be contacted at dkeller@uh.edu. subscribe to comments: for this article | for all articles 5 responses to "improving the drupal user experience" please leave a response below: cms usability: the overlooked part of digital productivity | j. boye, 2011-04-19 […] from a recent cms expert group meeting11 usability principles for cms products by james robertsonimproving the drupal user experience case study from the university of houston libraries who will also be speaking at in philadelphia […] jason, 2011-07-25 great article and comment, but link is dead: http://jboye.com/blogpost/cms-usability-the-overlooked-part-of-digital-productivity/ casos de estudio de uso drupal en bibliotecas « guilleten, 2011-10-24 […] ejemplo, vacek, en improving the drupal user experience, se refiere a un trabajo realizado con la herramienta en la red de bibliotecas de la universidad de […] casos de estudio de uso drupal en bibliotecas | wordpress 3.1.x, 2012-01-02 […] ejemplo, vacek, en improving the drupal user experience, se refiere a un trabajo realizado con la herramienta en la red de bibliotecas de la universidad de […] cms usability: the overlooked part of digital productivity j. boye, 2014-11-10 […] improving the drupal user experience case study from the university of houston libraries who will also be speaking at in philadelphia under the headline therapy for your cms: improving the user experience […] leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – editorial introduction – moving forward mission editorial committee process and structure code4lib issue 9, 2010-03-22 editorial introduction – moving forward welcoming new editors, and reflecting on the sustainability factor. by carol bean coordinating editor, issue 9 in the constantly changing landscape we are all familiar with in the technology world, the issue of sustainability surrounds us. it bubbles up in several of the articles in this issue. dr. mitchell speaks of it as an evaluative factor in using cloud services for library it infrastructure. the issue hovers in the background of querying oclc web services for name, subject, and isbn, which describes using pilot services at oclc. will what we have today be here next year? both creating an institutional repository for state government digital publications and a principled approach to online publication listings and scientific resource sharing are about those libraries’ efforts to establish sustainable and accessible respositories of digital content. wrangling electronic resources: a few good tools evaluates some tools for discovering evidence of no longer sustained resources, otherwise known as link rot. and sibyl schaefer confronts the issue head on in challenges in sustainable open source: a case study. sustainability haunts us as we seek to keep up with the rapidly changing technology environment. yet, the reaching for sustainability is also a magnet that draws us to code4lib, and the glue that holds us together. we share, join in, and participate, seeking to make things work. the code4lib conference grew from a tentative start in 2006 to a wildly successful annual event, despite, or perhaps because of, its unorthodox format (see conference reviews from this year’s scholarship recipients). the conference has reached sustainability without any organizational governance. clearly, the continuing success in this case is directly related to the code4lib community’s participation. issue 8 marked the second anniversary for the code4lib journal, which itself began on little more than enthusiasm and several code4libbers’ willingness to try to make it work. we have successfully navigated startup, made it over the initial adjustment hump, lost some editors, and added new ones. in response to our most recent call for new editors, we had an astounding response from a large number of very qualified applicants. we must be doing something right. but can we sustain it? we have sought to embrace the ideals of code4lib: openness, transparency, sharing, communication. but like many of the projects which have been written about in this journal, we face the challenge of sustainability in the face of changes, as we continue to find what works, what doesn’t, who our audience is and what they want, and what our place is in the larger world around us. with this issue, we are reluctantly waving goodbye to jodi schneider, one of the founding editors and a perpetually moving force in code4lib and on the editorial committee. with this issue we also welcome mj suhonos, tim mcgeary, tim lepczyk, tod olson, and gabriel farrell to the code4lib editorial committee. they bring to the journal fresh views, ideas, and, especially, renewing energy as we move forward. what will you bring to the journal? we have a stellar editorial committee. we continue to receive excellent articles. tom keays, in his editorial introduction to issue 7, noted the breadth of our readership. but sustainability in this world of technology requires more than one-way communication. we need input from our community. this is your cue to contribute to the journal with comments and feedback; to continue the communication; to move us forward. openness, transparency, sharing, communication. with your participation, we will be sustainable. thanks, on behalf of the editorial committee. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – a web service for file-level access to disk images mission editorial committee process and structure code4lib issue 25, 2014-07-21 a web service for file-level access to disk images digital forensics tools have many potential applications in the curation of digital materials in libraries, archives and museums (lams). open source digital forensics tools can help lam professionals to extract digital contents from born-digital media and make more informed preservation decisions. many of these tools have ways to display the metadata of the digital media, but few provide file-level access without having to mount the device or use complex command-line utilities. this paper describes a project to develop software that supports access to the contents of digital media without having to mount or download the entire image. the work examines two approaches in creating this tool: first, a graphical user interface running on a local machine. second, a web-based application running in web browser. the project incorporates existing open source forensics tools and libraries including the sleuth kit and libewf along with the flask web application framework and custom python scripts to generate web pages supporting disk image browsing. by sunitha misra, christopher a. lee and kam woods introduction with the increase in the acquisition of born-digital materials at collecting institutions, better tools are needed to manage, maintain and provide access to these complex resources. moving these materials from fixed and removable media into sustainable preservation environments is a common task in libraries, archives and museums (lams). this can involve media that are already in their holdings (e.g., disks stored in boxes alongside non-digital materials), as well as materials being acquired for the first time from individual donors or other producers. in an effort to use available resources and avoid re-inventing the wheel, digital forensics tools have found applications in digital curation efforts within lams. open source digital forensics software has been used to recover information from both removable media and fixed media (including hard disks). this paper introduces one such tool – called dimac (disk image access for the web) – which can be of use to end users of the materials – as well as lam professionals – to analyze the data contained in the digital media. background technical challenges in curating born-digital materials have given rise to several initiatives incorporating digital forensic techniques to assist with curation. bitcurator[1] integrates open source digital forensics tools to extract and analyze metadata from digital media, as well as supporting triage tasks and producing reports to support preservation decisions. bitcurator is led by the school of information and library science (sils) at the university of north carolina at chapel hill and maryland institute for technology in the humanities (mith) at the university of maryland; it addresses two core needs in collecting institutions: (1) integrating digital forensics tools and methods into the workflows and collection management environments of lams and (2) supporting mediated public access to forensically acquired data [5]. the bitcurator environment includes open source software including guymager, fiwalk, bulk_extractor, and the sleuth kit[9] to capture data from media, as well as newly developed code to generate reports describing the contents of disk images. it also provides a unique graphical user interface used to invoke many of these tools. early versions of bitcurator did not provide file-level access to the contents of disk images without requiring those images to be mounted as virtual drives. the tool described in this paper, dimac, fills this gap. it is intended to provide easy access to the contents of the files – both in a client-side application and within a web browser. this tool is likely to be useful for ongoing curation efforts conducted by lams including triage tasks, archival processing, preservation actions, and provision of public access. literature review recovering data from digital media has been discussed in library and archives literature for several years [21] [22]. this includes recent investigations of forensic techniques to acquire and process digital collections [23] [24]. the prometheus and perpos projects have developed software for data extraction, focusing on collecting contexts [25] [26] [27]. the project “computer forensics and born-digital content in cultural heritage collections” hosted a symposium and generated a report contributing to this discussion[3]. kirschenbaum, ovenden, and redwine (2010) [3] introduce the field of digital forensics to lam professionals, explaining the relevance of specific tools and principles. it also examines the convergences in the fields of archival study and law. a report in d-lib magazine [5] introduces the bitcurator project. it describes open source forensics software used by the project, and discusses relevant technologies including afflib (the advanced forensic format library) and dfxml (digital forensics xml). applying these concepts to practical applications in collecting institutions, woods and lee (2012)  [7] introduce topics in digital forensics applicable to collecting institutions, including write-blocking (preventing changes to source media), disk imaging (extracting sector-by-sector copies of disks), redaction and triage. all the tools presented in this paper assume that a disk image has been extracted from the relevant device. triage is the process by which items (file system objects and metadata in particular) are ranked in terms of relevance or priority. data triage may include identifying and analyzing deleted but recoverable data with potential value. these tasks often involve automating repetitive or technically challenging tasks during both appraisal and reprocessing after the ingest phase[5]. redaction is removal or masking of sensitive information before end users access the materials. woods and lee (2012)  [7] depict a potential archival workflow (beginning with media acquisition and ending with archival packaging) that includes triage and redaction tasks. analyzing disk images using fiwalk and bulk_extractor in bitcurator is dealt with in detail in woods, lee and misra (2013) [6]. woods and brown (2009) [1] describe the creation of a web-browsable cd-rom collection, with further technical detail in woods and brown (2009)[2]. woods and lee (2012)[7] investigate how collecting institutions incorporate forensics tools into digital curation workflows and discuss future needs, including provision of file level access to disk images. rationale the purpose of the prototype discussed here is to provide access to the contents and metadata of raw and forensically packaged[2] disk images via the web. users can browse the contents of disk images and reduce the time spent on analysis and making preservation decisions when working with born-digital assets. this project will also facilitate end-user access to collections via a web browser. the tool may also be used to “mask out” specific items from within disk images that need to be restricted from public access. for example, disk images may contain sensitive information which should be closed, redacted, or available only to users with the appropriate permissions. a future enhancement of this tool (outside the scope of this paper) could provide the ability to elide this information from the view of users without these permissions. several existing tools have some similarity to dimac. osfmount [18] is a commercial tool which allows the user to mount disk images in a windows host as a virtual drive. osfmount is proprietary but freely distributed. it provides read-only access to disk images, but requires the image to be mounted, and does not support web-based browsing. some digital preservation specific tools provide complementary functionality, but work on individual digital objects that have already been selected for preservation. one example is archivematica [17], an open source digital preservation system, which provides a browser-based interface for managing digital objects. dimac can help with the earlier stage of selecting or deselecting digital objects for inclusion in a repository. another relevant tool is gumshoe jr.[19], which provides searching and sorting on metadata of the files contained in an image. gumshoe jr. uses fiwalk to read disk images and to populate a solr[3] index with file-level metadata to support searching and sorting on metadata. however, it does not provide access to the contents of files or file systems. methodology dimac was developed as a web application that allows users to browse the contents of disk images, download files, and examine metadata within a web browser. a standalone application that can be run on a linux machine (including both graphical and command-line interfaces) was also developed to allow access to disk images accessible in local storage or on network drives. the standalone application and the dimac web application share many software dependencies (and some of the same code). they both allow users to treat disk images as serial objects that can be opened and parsed (for example, to view and download individual files and metadata items) without requiring mounting as a virtual device. the development of the standalone application and the dimac web service are described in detail in the following sections. evolution this work evolved from software developed for the bitcurator project. early releases of bitcurator provided tools to extract bitstreams from digital media and generate reports on both file systems and deleted or unallocated content. however, we found no open source tools providing a simple way for users to both browse the contents of disk images and extract files without mounting a disk image or creating a dedicated “case file”. even relatively sophisticated tools and frameworks such as autopsy and the digital forensics framework did not address this use case, long provided by the freely available (but non-open source) tool ftk imager. we first developed a command-line tool that accepted a disk image, file path, and inode as arguments and output the contents of that file. this tool used the pytsk[11] api to extract the byte runs within the disk image corresponding to the file. an appropriate list of file path and associated inodes was required for this to be useful, which we generated in the form of a digital forensics xml (dfxml) file using the fiwalk utility. this process was automated by modifying the bitcurator reporting tool with a new tab in the main graphical interface. the new tab brings the user to a new interface requiring a path to the dfxml file. using this dfxml file, the inode of a given file is identified and passed to code capable of extracting the file contents. the tab was further refined to use only the disk image, running fiwalk and then launching a second window displaying the file structure of the disk image. this allows the user to simply click on the files to select and download. future versions of the tool will incorporate code to render some file types in the interface itself, and display hexadecimal views. this graphical interface was useful, but depended on the creation of the dfxml file, which could be time-consuming for larger disk images. furthermore, not all of the file system metadata generated by fiwalk and written to the dfxml file was needed. as an alternative, we chose to call the sleuth kit application programming interface (api) directly to extract only the information required for navigation of disk content (also used by fiwalk when generating the dfxml file). dimac eliminates the need to run fiwalk, and is subsequently much more responsive. dimac stores disk image and related file system metadata in a database. modern disks often contain hundreds of thousands or millions of files. it is desirable to have near-instantaneous response in the web application, and a well-indexed database is better equipped to handle queries into such images rather than running forensics-specific utilities on the fly. functional specification and user interfaces dimac allows users to access disk images in a variety of contexts – online, in reading rooms or in other customized environments. the user does not have to mount a physical device or disk image in order to access it, nor is any local software required (beyond a web browser) to analyze the contents of the media. however, the tool expects a disk image as a starting point. the bitcurator environment provides several tools for generating disk images, including guymager, which has a graphical user interface. as noted previously, the dimac web application evolved from existing work conducted for the bitcurator environment. the original bitcurator tools included a graphical user interface invoked using the bitcurator reporting tool, and a command-line tool to export the contents of a specific file. the original bitcurator command-line and gui interfaces, along with the new dimac web application (separate from the bitcurator environment), are explained in detail in the subsequent sections. bitcurator (command-line and gui) interfaces a user can invoke the tool on the command-line in a terminal in the bitcurator environment by providing the path to the disk image and an output directory. executing the following command in the directory containing the python script bc_disk_access.py[4] launches the file-access gui. providing the argument dfxmlfile is optional. if not provided, it generates it internally by invoking the fiwalk command. if the image is large (multi-gigabyte or larger), generating the dfxml file may take some time. a spinning progress bar in the gui indicates that generation of this file is under way. python3 bc_disk_access.py –-image <image_name> [ --dfxmlfile <dfxmlfile>] --outdir <output directory> --listfiles another way to launch the gui is to use the bitcurator reporting tool and click on the “file access” tab and choose the image name and the export directory in the slots provided. figure 1 shows the bitcurator reporting tool when the disk access gui is successfully launched. figure 1. bitcurator reporting tool launching file access interface figure 2. gui based file access interface the file access gui displays a tree view showing the directories and files present in a given disk image. there is a checkable box next to each file-level item. a directory can be expanded by clicking on the arrow next to the directory name. contents of a given directory can be viewed by clicking on these arrows and expanding the depth of the tree, allowing full navigation of the disk images by expanding and collapsing subtrees as needed. this allows users working with disk images in digital collections to quickly navigate through file system as they would with “live” file systems mounted on their local machines. by clicking or double-clicking the check box associated with a file, a user can select or deselect the files he/she needs to view or export. all the files can be selected or deselected by selecting the “select all” or “deselect all” options respectively from the drop down menu from the window. clicking on the “export” button at the bottom of the tool downloads and displays the selected files. log messages appear on the text-edit window on the right side of the interface. alternate command-line tool to export files from disk images as an alternative in the bitcurator environment, a user can choose to display the contents of a specific file within the media, by providing the name of the file as a parameter in the command-line tool. following is an example of the command displaying the contents of the specified file and the output of the command: $ python3 bc_disk_access.py --image ~/aaa/charlie-work-usb-2009-12-11.aff --filename email/charlie_2009-12-04_0941_sent.txt --cat >> generating xml file /home/sunitha/aaa/dfxmlfile.xml >> invoking command for fiwalk = ['fiwalk', '-f', '-x', '/home/sunitha/aaa/dfxmlfile.xml', '/home/sunitha/aaa/charlie-work-usb-2009-12-11.aff'] >>> generated the file /home/sunitha/aaa/dfxmlfile.xml >> dumping contents of the file : email/charlie_2009-12-04_0941_sent.txt subject: i found something from: charlie <charlie@xxx.yyy> date: fri, 04 dec 2009 09:41:47 -0800 to: andy@xxx.com andy, lucky for me, i just happened to stumble .. web interface – disk image access (dimac) the dimac web application (independent of the bitcurator tool described in the sections above) provides a web interface allowing users to examine the contents of disk images (stored on the web host or in a network-accessible directory) within a browser. navigating into these disk images displays directories and the files they contain, along with relevant metadata contained within the file system. the start page displays disk images available for browsing (see figure 3). the directory containing the disk images can be specified before running the tool simply by entering a new path in the flask[10] configuration file. figure 3. dimac: access interface displaying disk image list clicking on the icon to the left of a specific disk image name generates a page showing some technical metadata associated with that disk image, including the date and time of capture, the md5 sum, and the size (see figure 4). figure 4. disk image metadata the main page (figure 3) provides two links to the right of each image: “browse” and “download”. clicking on the “download” link downloads the entire image to the client (useful in particular for disk images that contain executable or installable software). clicking on the “browse” option of a particular disk image generates a page showing the partition information along with the file system(s) contained on the given partition(s) (see figure 5). figure 5. dimac displaying partition information for a specific disk image clicking on a specific partition takes the user to new page showing the contents of the root directory for the file system in that partition (see figure 6). for each file listed, it displays the file type – whether it is a directory (indicated by the letter “d”) or a regular file (indicated by the letter “f”), size, modified time and whether it is deleted or allocated (indicated by “yes” or “no”). figure 6. file access interface displaying directory listing clicking on a directory displays the contents of that directory. clicking on a file prompts the user to download the contents of the file to their machine. future versions will display the contents of that file in a separate window, if appropriate rendering software is available on the user’s machine. design and implementation the design goals for dimac included using only open source applications and libraries, simplicity of setup and execution, and relatively low processing overhead. our original approach was to use apache and web.py to develop the web access component of dimac. however, the libtsk3 apis exhibited some integration issues with web.py[5], which would have delayed our design and implementation. we ultimately chose flask[10], a web application microframework that allows the developer to select extensions to handle the database abstraction layer and manage forms, among other tasks. because flask is lightweight, it allows for rapid prototyping of web applications. figure 7. software model the block diagram in figure 7 shows the software model used by the disk image analysis tool (dimac). the main web application, written in python, runs on linux server and provides a web service that allows users to navigate through web pages to view the metadata and the contents of the disk image. the web application uses the flask[10] web application microframework, and depends on the pytsk[11] python interface to the sleuth kit to manipulate disk images. the sleuth kit (tsk)[9] is a complete file-system analysis tool. it provides a general purpose library, libtsk3, which is written in c++. this software library provides apis to export functions including listing all the partitions of an image, files in a given directory, and dumping the contents of a specified file under a directory tree. pytsk3[11] is a python binding wrapped around libtsk3 shared object, and provides the same interface apis in python. disk image metadata is stored in a postgresql database and indexed to provide a high degree of responsiveness when interacting with the web application. note that in the current dimac implementation, the web server and database server are run on the same host machine for simplicity. flask[10] is a web application microservices framework for python. it uses the web server gateway interface (wsgi) and jinja2[12], a templating language for python, to enable rapid development of sophisticated web applications. the flask framework depends on three important concepts: an application object, a routing system, and a template engine. the dimac application defines one flask application object, which is an instance of the flask class. each invocation of the dimac tool creates an instance of this object. the routing system provides a framework to pass arguments and invoke a particular template. web pages are dynamically generated for the purpose specified in the routing routine. templates are written in the jinja2 templating framework, which mimics the python language and provides the flexibility to create dynamic web pages based on available content. the routing routine calls the api render_template with the relevant arguments to invoke the given template. web applications developed using flask are database-driven. in dimac, we use a postgresql database to store disk image and file level metadata to improve performance as the user navigates through a disk image within a browser. however, not all users may wish to use postgresql, particularly if they already have a database server running a different rdbms. to accommodate a range of back-end rdbms’s and provide a high degree of control over the relational components, we use sqlalchemy[14], a python sql toolkit that treats the database as a relational algebra engine. sqlalchemy includes an object-relational mapper that decouples the object model from the database schema. this allows the application to use simple, succinct commands to manipulate and query databases in a manner that is largely agnostic with respect to the rdbms. flask-sqlalchemy [15] is an extension for flask that adds support for sqlalchemy to flask applications (see figure 7). unit/feature test the three tools listed in the methodology section were tested using four disk images with one partition each and one image with four partitions[6]. for the bitcurator gui option, various combinations of files and directories were checked and were downloaded. dimac was tested to ensure the directory structure was being properly rendered at each level, files could be downloaded, and navigation options operated correctly in each dynamically rendered web page. downloaded files were examined to ensure the size of each file downloaded corresponded to the total size of the byte runs identified in the disk image, and whether the time taken to download the files was within reasonable limits. for the command-line option to extract individual files, files were chosen from different directories and tested to ensure the integrity of their contents. discussion and future work as indicated in a presentation at code4lib 2013 [16], processing and managing legacy born-digital materials can be extremely time consuming. the technical problems they pose may require significant intellectual and technical resources, and there is no guarantee that the process will go smoothly: one tool might work for recovering and analyzing a particular type of legacy medium, but not another. the reasons for this are multifaceted. for example, software tools built to mount disk images and raw devices may not be capable of mounting certain legacy devices. simply mounting the device or disk image can require significant technical expertise on the part of the user. applications such as dimac can alleviate some of these problems. dimac does not require the image to be mounted, and requires minimal technical expertise. it is a relatively simple tool (just a few hundred lines of code), is open source, and has been designed to be easy to use and modify. the dimac source code is currently hosted on github (master repository at https://github.com/kamwoods/dimac). two of the authors of this paper (sunitha misra and kam woods) are the primary contributors, and further development is planned beginning in fall 2014. dimac could be used in two distinct applications in lams. a typical use case could involve a curator making selection decisions about disk images extracted from physical media. selected images could be placed in an appropriate location for access (for example, on an existing server or network share). another use case could involve a patron browsing through the lam database looking for a specific file level item. the patron could examine metadata (and other manually generated finding aids) related to a particular disk image, browse the contents of that disk image, and download files of interest. this latter use case is similar to that in woods, kam and brown(2009)[1] where patrons use a web interface to access the contents of cd-rom disk images. similarly, lams could use standard virtual machine (vm) technologies in place of or as a supplement to existing workstations. as discussed earlier, dimac generates and stores metadata of the disk images in a postgresql database, which is queried when users click on relevant links. as a future enhancement, it would be beneficial to store the metadata information of individual files, which can be retrieved on demand (along with a downloaded file) or rendered in-browser before deciding to download a file. lam personnel could use this information in the curation process. dimac could also be used to carry out tasks that feed into other tools in later stages of the workflow. for example, dimac could be used to extract the files to be curated along with their metadata, at which point they could be prepared for ingest into an archiving and preservation platform such as archivematica. another potential enhancement of dimac is the addition of redaction capabilities – modules allowing the removal or masking of selected information from public view. using this feature, the content provider could mask out specific contents from the database. for example, certain files or a specific portion of a directory tree could be tagged to be masked. a redaction utility could also be invoked to create a redacted disk image prior to provision of access. this would automatically exclude files that carry sensitive information or mask off such information within the files. finally, database support could be extended to store data generated by tools such as bulk_extractor, identifying relevant features of interest (including potentially private and sensitive data) that appear on the disk. this information could then be browsed on demand from within the web browser, or redacted as necessary. conclusion a workflow in a lam is never completely finalized [8], and there is a constant stream of improved tools to handle artifacts and reprocess older materials. while born-digital materials have existed in collections for many years, digital curation processes are still in their infancy. tools like dimac, which demand relatively little technical expertise on the part of the user, will hopefully be simple to incorporate into workflows. the dimac tool provides remote access to the contents of disk images with very little overhead, could help with masking potentially sensitive information, and provides a generic interface to anyone familiar with web browsing. dimac is open-source software, which should further support adaptation to fit into the diverse workflows of lams. notes [1] the bitcurator project is funded by the andrew w. mellon foundation. [2] forensic disk image formats include the expert witness format and the advanced forensic format. forensic formats act as a secure, read-only wrapper for raw disk images and generally include substantial capture and technical metadata not available with raw images. [3]solr is an open source search platform from the apache lucene project [4] for example, in the bitcurator environment, $home/tools/bitcurator/python [5] web framework for python (www.webpy.org) [6] at the time of writing this paper, the feature is still being tested with an image with four partitions. bibliography [1] woods, kam and brown, geoffery, “creating virtual cd-rom collections”, international journal of digital curation , 2/no-4 (2009) [2] woods, kam and brown, geoffery, “from imaging to access – effective preservation of legacy removable media.“, archiving (2009): preservation strategies and imaging technologies for cultural heritage institutions and memory organizations: final program and proceedings, pages 213-218. [3] kirschenbaum, matthew g., ovenden, richard, redwine, gabriela and research assistance from rachel donahue, “digital forensics and born-digital content in cultural heritage collections.”, washington, dc: council on library and information resources (2010). [4] lee, christopher a., woods, kam, kirschenbaum, matthew and chassanoff, alexandra. “from bitstreams to heritage: putting digital forensics into practice in collecting institutions.”, a white paper on bitcurator project. (2013) [5] lee, christopher a., kirschenbaum, matthew, chassanoff , alexandra, olsen, porter, woods, kam. (2012). “bitcurator: tools and techniques for digital forensics in collecting institutions”, d-lib magazine may/june 2012 18/no 5/6. [6] woods, kam, lee, christopher a., misra, sunitha, “automatic analysis and visualization of disk images and file systems for preservation.”, proceedings of archiving 2013 (springfield, va: society for imaging science and technology, 2013), 239-244. [7] woods, kam and lee, christopher a., “acquisition and processing of disk images to further archival goals”, proceedings of archiving 2012, springfield, va, society for imaging science and technology, pg. 147-152. [8] gengenbach, martin j., “the way we do it here: mapping digital forensics workflows in collecting institutions” master’s paper, unc, chapel hill, 2012. [9] “the sleuth kit (tsk)” last modified march 21, 2014. http://www.sleuthkit.org/sleuthkit/ [10] “flask – web development, one drop at a time.” accessed february 14, 2014. http://flask.pocoo.org [11] “pytsk: python bindings for the sleuth kit.” last modified 28 may, 2012. https://code.google.com/p/pytsk/wiki/overview [12] “jinja2: template designer documentation” accessed february 14 2014. http://jinja.pocoo.org/docs/templates/ [13] the sleuth kit, “overview,” last modified 21 march 2014. http://www.sleuthkit.org/sleuthkit [14] sqlalchemy, “the python sql toolkit and object relational mapper” last modified february 8, 2014 http://www.sqlalchemy.org [15] “flask-sqlalchemy” last modified may 28, 2012. http://pythonhosted.org/flask-sqlalchemy [16] mennerich, don and matienzo, mark a, “pitfall! working with legacy born digital materials in special collections”, presented at code4lib 2013, chicago, illinois. [17] “archivematica” last modified february 19 2014. https://www.archivematica.org/wiki/main_page [18] passmark software, “osfmount” last modified february 9, 2014. http://www.osforensics.com/tools/mount-disk-images.html [19] “gumshoe jr.” accessed march 2014. https://github.com/anarchivist/gumshoe [20] matienzo, mark, “fiwalk with me: building emergent pre-ingest workflows for digital archival records using open source forensic software”, feb 09, 2011,last modified august 2, 2013. http://www.slideshare.net/anarchivist/fiwalk-with-me-building-emergent-preingest-workflows-for-digital-archival-records-using-open-source-forensic-software [21] woodyard, deborah. “data recovery and providing access to digital manuscripts.” paper presented at the information online 2001 conference, syndney, australia, january 16-18, 2001. [22] ross, seamus, and ann gow. “digital archaeology: rescuing neglected and damaged data resources.” london: british library, 1999. [23] john, jeremy leighton. “adapting existing technologies for digitally archiving personal lives: digital forensics, ancestral computing, and evolutionary perspectives and tools.” paper presented at ipres 2008: the fifth international conference on preservation of digital objects, london, uk, september 29-30, 2008. [24] garfinkel, simson, and david cox. “finding and archiving the internet footprint.” paper presented at the first digital lives research conference: personal digital archives for the 21st century, london, uk, february 9-11, 2009. [25] elford, douglas, nicholas del pozo, snezana mihajlovic, david pearson, gerard clifton, and colin webb.. “media matters: developing processes for preserving digital objects on physical carriers at the national library of australia.” paper presented at the 74th ifla general conference and council, québec, canada, august 10-14, 2008. [26] underwood, william e., and sandra l. laib. “perpos: an electronic records repository and archival processing system.” paper presented at the international symposium on digital curation (digccurr 2007), chapel hill, nc, april 18-20, 2007. [27] underwood, william, marlit hayslett, sheila isbell, sandra laib, scott sherrill, and matthew underwood. “advanced decision support for archival processing of presidential electronic records: final scientific and technical report.” technical report ittl/csitd 09-05. october 2009. about the authors: sunitha misra is a masters student in the school of information and library science at the university of north carolina at chapel hill. she is currently a software developer on the bitcurator project. christopher (cal) lee is an associate professor at the school of information and library science at the university of north carolina at chapel hill. his primary area of research is the long-term curation of digital collections. he is principal investigator for the bitcurator project and editor of i, digital: personal collections in the digital era published by the society of american archivists. kam woods is currently a postdoctoral research associate in the school of information and library science at the university of north carolina at chapel hill his research interests include long-term digital preservation, digital archiving, and the application of digital forensics tools and techniques to archival and preservation data analysis and management. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – integration of library services with internet of things technologies mission editorial committee process and structure code4lib issue 30, 2015-10-15 integration of library services with internet of things technologies the selida framework is an integration layer of standardized services that takes an internet-of-things approach for item traceability in the library setting. the aim of the framework is to provide tracing of rfid tagged physical items among or within various libraries. using selida we are able to integrate typical library services—such as checking in or out items at different libraries with different integrated library systems—without requiring substantial changes, code-wise, in their structural parts. to do so, we employ the object naming service mechanism that allows us to retrieve and process information from the electronic product code of an item and its associated services through the use of distributed mapping servers. we present two use case scenarios involving the koha open source ils and we briefly discuss the potential of this framework in supporting bibliographic linked data. by kyriakos stefanidis & giannis tsakonas introduction libraries are trying to identify potential applications for internet-of-things (iot) technologies and a recent survey by oclc (2015) highlighted that the anticipated uses of iot are mostly related to intelligent uses of space and facilities. the same survey revealed that the most familiar iot services to librarians were those designed for inventory purposes. such services would require the employment of radio frequency identification (rfid) tags as aids for increased visibility and unique identification. rfid have been envisaged as the appropriate technologies to fulfill the promise of iot in libraries. fortune (2012) describes a future state where books are tagged with rfid and “sensors are placed on the shelves to detect the removal of any item for consultation” making the shelves “active”. in doing so, libraries must be equipped with the right technology infrastructure, such as sensors, readers, services and software. one of the main technical challenges is that, apart from the lack of appropriate technologies to coordinate these components, any intervention has to be easily adjusted to key software infrastructure, such as the various integrated library systems (ils). the existing technologies in libraries cannot be ignored or skipped, but on they should be able to easily link to these solutions. in our case this is achieved with the selida framework, on which the main part of this article is focused. the selida framework has been developed in the frame of the self-titled project. its name stands for the greek equivalent for the acronym “printed material management using radio frequency identification technology” (“selida” in greek means “page”). selida is a public-private sector collaboration project, with the library and information center at the university of patras as host of the pilot implementation. the ils used in this prototype is the open source koha. a second section of the paper is dedicated to the exploration of the electronic product code (epc) tags’ potential to store information about library holdings. as the current linked data efforts in the bibliographic ecology have not substantially covered the issue of holdings information, we can exploit technologies from other domains. one of the main principles in the linked data domain is to reuse already operating or standardized components to avoid ambiguity (bizer & heath, 2011) and to this end epcs can be part of the linking of bibliographic records with holdings information. related technologies the key component of the selida framework is the use of rfid tags as an aid for identification of physical items, and epc as the underlying building block for standardized tracking services. rfid tags are now commonplace in libraries, with many advantages over other identification technologies, such as barcodes or qr-codes—namely in bulk detection, reading distance, alignment, capacity and so on. rfid tags are also preferred for supporting real time traceability and for being intrinsically connected to epcs. according to a recent survey of the gs1 us—the us chapter of gs global, an organization that promotes the adoption of its standards in industry—retail market stakeholders such as manufacturers and sellers are rapidly adopting rfid with epc to increase the visibility of their products (gs1 us, 2015). thus, the use of epc global, the gs1 suite of standards and specifications, enables the visibility and traceability of physical units. the epc standard is mostly used in the supply chain sector and in our case it is adjusted to the physical items of a library’s collections, such as books, discs, etc. the use of epcs together with the object naming services (ons) enables the circulation of uniquely tagged information; the items are notated by the tagging services of epc global and the information is exchanged with ons. an epc can be written on a physical rfid tag and the number of encoded digits is determined by its type (in our case 24 decimal digits). it encodes a serialized global trade item number (sgtin), which is a global trade item number (gtin) that is expected to be the same for a set of items, plus a serialization number for each physical unit. the epc is split into different, fixed-length sections, with each section providing different information. the first section is the header that reports on the coding scheme; the second is the company/institution prefix, which in our case represents a library; the third section defines the item reference; and the final section refers to its unique serial number. when an rfid reader captures an epc in binary format, it transmits it to a middleware layer and transforms it in a uri format. figure 1 presents the sections of a typical epc as a uri. figure 1. an example of an epc in uri format lastly, the object naming services are provided by a resolver, which translates a uri address to a domain name. by working in a way similar to dns, the ons resolver splits the binary format to different sections, each of which corresponds to a different server that holds the related information and services about its own section and a pointer to the next server in the uri sequence. architecture we deploy a three-layer architecture, outlined from bottom to top: 1. the bottom layer is the integration layer that manages seamlessly all the processes relevant to the library services workflows. by seamless we mean that the actual services (i.e. the source code) of the ils are not changed and the production system remains intact. therefore any change, such as a version update, in either part does not affect the other. the integration layer is injected upon page load as a javascript file. in our prototype, we use the mod_substitute directive on the ils’s dedicated apache web server. each time a module/page of interest is requested by the server (i.e., checking out an item), the web server adds a script tag that loads the additional functionality. this layer, in the form of a javascript application module (henceforth the selida module), adds the required user interface elements and handles all the necessary web service requests. 2. the middleware layer receives, analyses, processes and propagates the data collected by the rfid readers to the ils. the middleware hides the complexity of the actual rfid infrastructure and provides only the information about the workflow events. the middleware is agnostic of how the data is handled afterwards. 3. the upper layer, called the services layer, provides secure access to the ons infrastructure which consists of a mapping server and an ons resolver. the mapping server manages all provenance data which are relevant to a single physical item, as well as the mapping between an epc and its metadata. these data can contain information regarding item transactions within locations controlled by the owning library or any other interconnected library, as long as all those are part of the selida infrastructure. furthermore, through the services layer, the system accesses the ons resolver, which allocates specific services to specific types of users, such as inventory services for collection development librarians or history services for circulation librarians. this is accomplished by a web service layer that functions on top of the ons resolver, which provides authenticated users with the capability to query the whole ons infrastructure and discover the management services for any given epc and point them to the correct mapping server. figure 2. the layers of the selida framework architecture implementation in our prototype version, the framework supports two types of services: first is a subset of basic library services, namely checking-out, checking-in, adding, and deleting an item. secondly, the framework supports some item-related services, namely retrieving information about the history of an item (location of check-ins and check-outs across the interconnected libraries), as well as searching for the physical location of an item within a library. we describe one implementation case for each type of service. checking-out when a circulation librarian navigates to the check-out module of the ils by requesting the respective url, the selida module starts executing upon page load and adds the button “scan” next to the button “check-out” of the koha interface. when the user presses the button “scan”, a web service request is launched from the selida module, which starts up the rfid reader via the middleware services. the results, in the form of the items’ titles and codes that the reader captured, are sent back to the selida module. the selida module pops up a window that informs the user of these results. after this presentation, the regular check-out workflow resumes by sending the required post requests to the koha web server. since koha does not support (at the moment) multiple checkouts, the required multiple requests are sent via ajax and the results of these requests (namely the errors) are gathered by the selida module for subsequent presentation to the user. when the check-out process ends and the web server responds with the next web page, the selida module sends a second web service request to the middleware indicating that the check-out is complete. history services history services are expected to be used among interconnected libraries. information about items should adhere to the same api and should be easily stored by the networking parties for reviewing the required data. therefore a lightweight data exchange mechanism is required and as such the selida module services have been designed following the restful architectural style using javascript object notation (json). in our framework, historical data can include a list of transactions only based on the dates of interaction with the middleware, together with a geospatial representation of the location of the respective transaction, i.e. the library where this transaction takes place. any request for item status, like checking in items, can contain multiple epc tags at a time, as well as error warnings (0 in case of no error) and relevant metadata for the identified books, such as notes on the purpose of transfer, the owning and/or the target library, etc. the following code snippets in json exhibit the request and response for history services. request { "epclist":[ {"epc": "961012345678910012345678"}, {"epc": "961543210109876543210101"} ] } response { "errid":0, "epclist":[ {"epc":"961012345678910012345678", "barcode":"910012345678", "title":"macroeconomics", "callno":"339 pet", "status":"available", "owner":"gr-pauli", "location":{ "lat":"38.28923", "lon":"21.785369" } }, {"epc":"961543210109876543210101", "barcode":"876543210101", "title":"macroeconomics", "callno":"339 pet", "status":"loaned", "owner":"gr-atppv", "location":{ "lat":"37.959884", "lon":"23.719248" } } ] } table 1. request/response example can linked data and iot be real in the library? the framework could be a promising start for the extension of linked data models of the bibliographic ecosystem to enable the accommodation of information about holdings. whereas the linked data solutions are rapidly progressing in the library sector, thus creating a conceptual network of knowledge resources, iot proposals are relatively limited and as previously stated have reached a dead end in linking concepts to holdings. fast evolving linked data frameworks, such as bibframe, are currently missing mechanisms for global and unique visibility, identification and traceability of items. for instance, bibframe stores information about the holdings of an instance in annotation fields, but uses ambiguous and context-depended information, such as the instance’s call number, which might differ from one library to another. the following json example shows how minimal, but concise epc tagged information can be embedded in an annotation field of a bibframe record: { "type": "helditem", "id": "0123456789012helditem1", "bf-id": "0123456789012helditem1", "uri": "urn:epc:id:sgtin:96.1.012345.678.910012345678", "urn:epc:id:sgtin:96.1.543210.109.876543210101" "bf-holdingfor": "9600215006instance1" } by parsing the epc of an item in uri format, one could identify a specific physical unit from the collections of a library and retrieve information about its status and location. thus, there can be an effective and dynamic connection between different “things”—such as works and entities, which have demonstrated that they can be in uri format—and actual things, such as physical items. discussion the selida framework demonstrates the potential of using already globally working standards and protocols with existing technologies to increase the visibility of physical items with minimal cost and disruption. it also underlines that there is a lot to be done in the future. the penetration of rfid technologies is still slow, especially in academic libraries, where the large number of volumes requires significant investment, while the epc encoded tags are relatively unknown. furthermore, there are other kinds of investments and efforts, notably in the area of gs1 company prefixes, which would require the cooperation of authorities to assign global prefixes to libraries. finally, with the lack of wide application of the stgin in libraries, there have to be firm decisions about the form of the sections of the 24 digit code. for instance, if a library desires to migrate from an existing item bar-coding scheme, then the number of available item reference digits is limited and will have to refer to another item relation, such as a category (e.g., books or discs). however, if a more global scheme is required, then the item reference will have to be occupied by a worldwide common scheme, like an edition’s barcode, and the rest of the digits used to identify the serialized number of copies of this edition in the library’s collection. in this paper we described the architecture of the selida framework and presented two typical implementation cases. we also raised the issue of using parts of the framework to merge linked data with the iot efforts in the library in a more effective fashion. the services of the selida framework alone can be appealing to networks of libraries, such as cooperating libraries or branch structures that circulate items among them. it is encouraging that iot technologies are maturing and the potential they hold in identifying and tracing physical objects allow more confident visions for a global library catalog. appendix a: selida exhibition video acknowledgements the selida project (#09syn-72-646) was financially supported by the general secretariat for research and technology (gsrt) of the hellenic ministry of development in the frame of “cooperation” 2009 call. about the authors kyriakos stefanidis holds a phd in computer science and a bsc in computer engineering from university of patras (greece). he is a contracted researcher in university of patras and athena rc / industrial systems institute. since 2001 he is involved in a number of research and development projects both european and national. his research interests are mainly focused on the field of computer and network security, cyber-physical security and high performance computing. giannis tsakonas is acting library director in library & information centre, university of patras. he holds a ba in librarianship from the department of archives and library sciences, ionian university, greece and a phd in information science from the same department. more info at his website www.gtsak.info and/or at his twitter feed @gtsakonas. references bizer, c. and heath t. 2011. linked data: evolving the web into a global data space. synthesis lectures on the semantic web: theory and technology. new york, ny: morgan & claypool publishers. fortune m. 2012. can rfid save libraries? rfid arena [internet]. [cited 2015 august 11]. available: http://www.rfidarena.com/2012/11/8/can-rfid-save-libraries.aspx gs1 us. 2015. gs1 us survey shows manufacturers and retailers embrace rfid to enhance inventory visibility. [cited 2015 august 11] available: http://www.gs1us.org/about-gs1-us/media-center/press-releases/rfid-survey-findings. oclc 2015. libraries and the internet of things. nextspace [internet]. [cited 2015 august 11]; 24. available: http://www.oclc.org/en-europe/publications/nextspace/articles/issue24/librariesandtheinternetofthings.html. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – the planets testbed: science for digital preservation mission editorial committee process and structure code4lib issue 3, 2008-06-23 the planets testbed: science for digital preservation the preservation of digital objects requires specific software tools or services. these can be characterisation tools that abstract the essential characteristics of a digital object from a file, migration tools that convert digital objects to different formats, or emulation tools that render digital objects in their original context on a new infrastructure. until recently digital preservation has been characterised by practices and processes that could best be described as more art and craft than science. the planets testbed provides a controlled environment where preservation tools can be tested and evaluated, and where experiment results can be empirically compared. this paper presents an overview of the testbed application, an analysis of the experiment methodology and a description of the testbed's web service approach. by brian aitken, petra helwig, andrew jackson, andrew lindley, eleonora nicchiarelli, seamus ross introduction and research framework planets, preservation and long-term access through networked services, is a four-year project which began in june 2006. it is co-funded by the planets consortium and the european union under the sixth framework programme to address the core digital preservation challenges that libraries, archives and the digital preservation community are facing. the primary goal for planets is to build practical services and tools to help ensure long-term access to our digital cultural and scientific assets. planets is pursuing this goal with both a practical and an intellectual approach, and is addressing such urgent research issues as the establishment of an automated characterisation and validation framework for digital objects, the preservation of databases and the role of emulation in digital preservation. for more details and a list of planets publications see the planets website [1] and in particular the publications page [2]. as the information society and its knowledge economy continues to expand, the underlying information assets of our society and the processes that we deploy are susceptible to contextual, syntactical and semantic obsolescence. over the past 20 years there have been an increasing number of attempts to address these challenges. these have generally been haphazard and poorly co-ordinated, have relied on weak engineering methods and have proved difficult to replicate or deploy at any level of scale. until recently digital preservation has been characterised by practices and processes that could best be described as more art and craft than science; studies such as the delos/nsf research agenda on digital preservation [3, 4] and the digitalpreservationeurope research agenda [5] note that frameworks for experimentation must be central to the design and practice of digital preservation research. researchers such as gladney [6], ross [7], thibodeau [8], and most recently watry [9] have all argued that digital preservation investigations need to be more engineering focused. following on from work done by the national archives of the netherlands [10] and the digital preservation cluster of the delos network of excellence in digital libraries [11], planets identified that a key step in taking this agenda forward should be the development of a testbed framework to support the assessment of preservation approaches and tools. in order to perform research that has a scientific grounding, the digital preservation community needs to evaluate preservation approaches within a controlled environment. this environment must be able to simulate diverse real-life settings whilst avoiding duplication of work and maximising the use of invested resources. these are some of the issues that the planets testbed addresses. the testbed provides a dedicated research environment that allows the systematic execution of experiments by distributed actors, enabling the automated evaluation of experiment results, the reproducibility of experiments, the long-term availability of structured experiment documentation and shared access to the experiments themselves. the preservation of digital objects requires specific software tools or services. these can be characterisation tools that abstract the essential characteristics of a digital object from a file, migration tools that convert digital objects to different formats, or emulation tools that render digital objects in their original context on a new infrastructure. if we can gain an understanding about which tools best serve our needs we can compile preservation plans that indicate which of these preservation tools should be applied to a collection of digital objects under which circumstances. the goal of the planets testbed (consisting of a software application and a research framework) is to extend the already existing information on preservation tools with new information based on the outcomes of practical experimentation. this extended knowledge will enable digital preservation practitioners to make sensible, informed decisions regarding the applicability of the tools in various preservation settings. experiments can be made visible to other testbed users and be exported to publicly available registries, enabling the results from previous experiments to shape a common understanding of the best preservation approaches for specific types of data in specific situations. the testbed also provides the mechanisms to enable users to repeat existing experiments in order to validate the available results. the testbed concept is not a new one in the field of digital libraries; the development of testbeds was already a key component of the us digital library initiative (dli) which led to the development of the d-lib test suite [12]. other, more recent digital library initiatives include the open video digital library [13]; see the delos framework for testbed for digital preservation experiments [11] for a detailed introduction to this topic. in a report prepared at the request of the dutch government in 1999, the digital preservation expert jeff rothenberg recommended the establishment of a testbed project as part of a broader series of initiatives to address digital preservation ([10], [14]). as a result, the dutch digital preservation testbed was founded in october 2000. this research project tested the practical applicability of approaches to the preservation of governmental and other digital information. the national archives of the netherlands have utilised the dutch digital preservation testbed since 2003 to ensure that they select and implement the most appropriate and effective tools to preserve their electronic records over time. [10]. following on from this the delos digital preservation testbed [11] built upon the experiences of the dutch digital preservation testbed. the delos approach was to propose the abstraction of an operational retrieval environment that provides the means for researchers to explore the relative benefits of different preservation strategies in a laboratory setting. while at least one of the d-lib test suite collections was used to test the transformation of varied sgml formats into well-formed xml [15], the dutch and delos testbeds were the first serious attempts to develop a generic framework for the evaluation of digital preservation strategies. the planets testbed was in turn inspired by the work undertaken by the dutch and delos testbeds. however, the scope of the planets testbed is considerably broader, both in terms of the available services and the intended user community. the planets testbed’s web-service approach enables a large number of users to perform experiments on a wide variety of services with a focus on the workflow-based design of the experiments. consultations with the eight national memory institutions that are partners in planets has led to a number of refinements to the experiment methodology of the dutch and delos testbeds, resulting in a simplification of the experiment process. testbed users can execute single migration and characterisation experiments, but also load pre-existing or create new service workflows. a typical testbed workflow-based experiment could involve the following sequence of steps: invoking a characterisation service on the input data to determine their significant properties and appropriate migration tools; subsequently invoking a migration service for the execution of data migration; finally, invoking a second characterisation service to automatically assess the results of the migration. another typical scenario is that of large-scale experiments (thousands of files or files of extremely large size) which evaluate the non-functional characteristics of tools and services. this will be made possible by the establishment of the testbed corpora of test objects, as described below. the current functionality of the testbed the core functionality of the testbed enables a user to design, execute and evaluate an experiment on certain data with certain services using the web-based front-end of the testbed application. in general, every experiment follows a fixed experimentation workflow that consists of six steps, as figure 1 below demonstrates. figure 1: the testbed experiment workflow in the first three steps, the testbed user designs the experiment, specifying the focus and intended goals of the experiment and selecting the services and data on which the experiment will execute. the user also specifies how the outcomes of the experiment will be evaluated. following on from this a decision is made about whether experiment execution should proceed or not. the experiment is then carried out in step 5 and is then evaluated in step 6. for each of these steps there is a corresponding webpage in the testbed application that enables the experimenter to supply the required information. the user’s current position in the workflow and access to all previous stages is available in a panel on the left of the page, which allows the user to return and edit existing information if necessary. in the following section each of the experiment steps are described in more detail. step 1 – define basic properties in the first step, the basic properties for the experiment are defined. in addition to supplying general information such as an experiment name and a brief description, this stage enables the user to identify the purpose of the experiment, the pertinent research questions and references to any relevant research and literature. the experimenter must also indicate the type of experiment and the types of digital object the experiment will focus upon. figure 2: the testbed application showing step 1 of the experiment process step 2 – design experiment for step 2 the specific services on which the experiment will focus are selected and the input files that will be accessed by these services can be uploaded or selected. the list of services will include any characterisation, migration and emulation tools that have been wrapped as web services and have been made available within the testbed, as discussed in a following section. the input files for the experiment may consist of new files that the experimenter uploads, existing files that are stored in the testbed’s data registry or files taken from the digital preservation corpora. figure 3: step 2 of the experiment process step 3 – specify outcomes in the third step, a list of possible evaluation criteria is presented to the experimenter. the contents of this list are dependant on the types of object and experiment that have previously been chosen and the user can make a selection by ticking the corresponding criterion. the selections made here will define the evaluation options that are available in the final step of the experiment process. figure 4: step 3 of the experiment process step 4 – experiment approval after the first three steps a decision whether or not to proceed with the execution of the experiment is taken. this decision is initially made automatically by the testbed application; if the input data consists of a handful of files, the estimated intensity of the experiment is low or there are no currently executing experiments the application may automatically approve the experiment, in which case the user is informed that s/he may proceed to step 5. if this is not the case then the testbed administrator will need to manually approve and schedule the execution of the experiment. if this manual decision results in approval the user is then informed and may proceed with the next step. step 5 – run experiment in step 5 the experiment is executed. an experiment may take a considerable amount of time to execute depending on the number and size of the input files and the service that is called. during execution the webpage for this step will display run-time statistics about the experiment. in the current version of the testbed experiments are automatically executed following approval. however, as concurrent execution of multiple large-scale experiments will influence run-time and performance related benchmarks, additional scheduling and load balancing facilities will be added to subsequent testbed versions. step 6 – experiment evaluation after execution has completed the experimenter is presented with the results and may then evaluate the experiment. the evaluation criteria that were selected in step 3 are presented again, enabling the user to manually evaluate each criterion. for each criterion the user can enter values for the input and output files and may then assign a value for the evaluation based on the similarity of the input and output values. for example, if the value of “header spacing-left” was 1.5 cm in an input document stored as a microsoft word file and, after migration to pdf, the value of “header spacing-left” in the output file is 1.4 cm, the user can enter these values in the corresponding fields and rate the evaluation of this criterion. for more information on experiment evaluation please see below. figure 5: step 6 of the experiment process once the experimenter has completed step 6 the experiment process draws to a close and the results can be considered by other users, who are given the option of posting comments about the experiment. the results from the experiments are shared with other planets applications, enabling the output from testbed experiments to be used by decision support tools for the compilation of preservation plans. based on the testbed results, recommendations for refinements and enhancements for future experiments can be made and further experiments can be proposed. evaluation a digital object has syntactic and semantic properties or characteristics that can relate to content, context, appearance, structure or behaviour. a specific digital object type, such as text, image, sound or video, can be stored in many different file formats. the question when migrating a digital object from one file format to another is to what extent the intellectual characteristics of the object remain untouched. it should be possible to ask questions such as: how well is the font size preserved if we use tool x to migrate a text from a microsoft word format to a pdf format? for each digital object type the testbed contains a list of intellectual characteristics, or properties, which may be of interest to the experimenter. for example, properties for “text” include the number of pages, font size and background colour. these properties can then be used to assess the functional performance of the services. in characterisation experiments, experimenters may test whether the characterisation tool characterises these properties correctly: to what extent are the values of these properties in test objects, as characterised by the characterisation tool, the same as the “actual” values. in migration experiments, experimenters may test if these properties remain untouched when using the migration tool: to what extent are the properties of the migrated digital objects the same as the values of the original objects when using the migration tool. in emulation experiments, experimenters may test to what extent the values of the content, context, appearance, structure and behaviour of digital objects in the environment emulated by the emulation tool are the same as the values of the objects in their original environment. using the intellectual characteristics as evaluation criteria in this way it is possible to build up an overall picture of a tool’s performance over the course of several individual experiments. these results can be aggregated to give average information about the performance of tools on various types of digital objects and the more experiments that are carried out, the more trustworthy this kind of experimentally gained information will prove to be. within the testbed it is possible to both manually and automatically evaluate an experiment based on its experiment criteria. at the time of writing the automatic evaluation of experiments is currently being researched. in order to enable the automated extraction and evaluation of digital object characteristics the testbed will make use of the extensible characterisation definition language (xcdl) and extensible characterisation extraction language (xcel) [16], which are being developed by another part of the planets project. experiment search and retrieval a major goal of the testbed and its research framework is to ensure that the results of the experiments are comparable, retrievable and documented so that they can be repeated. the testbed’s database of existing experiments is a research tool of immense value to preservation officers and it is imperative that users can effectively search the experiments database in order to gain answers to their research questions. the testbed’s search facilities provide the necessary depth to gain answers to such questions as: which image migration tools are available that are known to preserve the colour correctly? which image migration tool is considered the best at preserving sharpness when migrating from tiff to jpeg2000? is there a characterisation tool that can automatically extract the values for font family and variant from word 2003 files? which tools are available for extracting textual information from an image representing a page of a printed book in uncial script? is there any tool that saves correctly undisclosed-format information (as maker notes) contained in jpeg exif tags? on which examples has it been benchmarked? testbed users may explore the database of completed experiments, comment on the design, methodology and outcomes of experiments and generate or browse knowledge trees based on experiment references. the architecture of the testbed in designing the testbed some key principles shaped the development of the software. firstly, the testbed application was designed to be platform independent, robust and scalable and for this reason development was undertaken using java enterprise edition. secondly, at the outset it was agreed that the testbed had to be able to execute experiments on as wide an array of preservation tools and services as possible, including legacy tools and tools that may not run natively on the platform on which the testbed itself is installed. a web service approach was considered to be the most flexible way of meeting this challenge; tools are wrapped as web services which can then be accessed by the testbed application by means of a platform-neutral uri. thirdly, the testbed application was designed to be a part of the overall planets software suite rather than as a standalone entity. a different planets sub-project, the interoperability framework, was tasked with providing many of the registries and components, such as the application server, storage systems and user management, which would be required by many of the various planets applications. this approach enabled the testbed developers to focus primarily on the components that were unique to the testbed. this sharing of common components across the entire project also makes it easier for the various planets applications to communicate with one another, allowing (for example) the results from testbed experiments on preservation tools to be aggregated within the appropriate registry and then fed into the preservation planning components. the design and development of the testbed began in september 2006 and since then the developers have followed a traditional structured and iterative methodology for defining the system’s functional range and the implementation of the software. this methodology, which is based on the rational unified process, included interviews, use case descriptions, a workflow checklist for functional and non-functional requirements, a detailed architectural system design document and several phases of development during which the design was refined and aspects of the system were enhanced. the planets testbed application was built around the jboss java enterprise edition 1.4 certified application server [17], and the testbed utilises and extends various features that jboss offers, such as messaging and transaction support. the application was designed and developed using the model-view-controller (mvc) architectural pattern, an approach which ensures that the business logic of an application is isolated from its user interface. by adopting this approach the developers were able to successfully divide the testbed development, enabling certain developers to focus on the complex business processes that the system required whilst other developers focused on the ways these processes could be best represented to users in the web-based front-end. within the testbed architecture the ‘view’ layer is realised as a web-based user interface, using the java server faces (jsf) standard [18] as the core technology and outputting valid xhtml / css web pages. the ‘controller’ layer, which processes all of the business logic, was developed using java enterprise edition (ee), and many of the enhancements offered through ee, such as enterprise java beans and session beans, have been utilised. the ‘model’ layer for the testbed has primarily been provided by the planets interoperability framework and consists of a number of registries shared by the planets applications where data and services are stored. figure 6 below demonstrates how the testbed mvc approach interoperates. figure 6: the testbed architecture built on planets overall, planets aims to build software components for digital preservation following a service-oriented architecture, as illustrated in figure 7 below. this approach allows complex preservation workflows to be built up as chains of simpler preservation services. for example, a migration workflow might start by retrieving a digital object from a repository via a data-access service. this object may then be passed through one or more services in order to characterise it, e.g. to determine the file format. based on this characterisation, the object may then be migrated to a new format and after this the new object may also be characterised to see how closely it retains the character of the initial form. if all is well, the object can then be saved back to the repository in the new format. such processes are easy to build using the planets infrastructure, and many different types of workflow can be created in order to meet any specific needs, such as those dictated by institutional policy. during the development of these preservation services and workflows, it must be possible to test them against any digital objects to which the institution has access. this is the purpose of the testbed, which leverages the planets infrastructure to allow preservation services and workflows to be discovered, modified, executed and evaluated. figure 7: simple schematic overview of the planets workflow model. working within jboss, each of the planets components provide classes, enterprise java beans, web services and web pages that can be used to build up a preservation system (see figure 8 below). the testbed is designed to be deployed as an enterprise archive within the planets application server, and as it leverages the same technologies as the wider framework, the shared components and code can be reused easily. the planets infrastructure provides user accounts, security management systems and administration pages, each of which the testbed utilises. user authentication is achieved by configuring the application server, so no explicit authentication code is required. for authorisation, the testbed defines the roles of experimenter, reader and administrator, but the management and assignment of these roles is handled externally to the testbed. the planets application server also provides a set of standard enterprise java beans which the testbed can use to access (for example) the user account information. figure 8: the different code layers of the planets infrastructure, with the two ‘code’ levels at the bottom and the two ‘user’ levels at the top. as indicated above, the planets project is attempting to standardise access to digital preservation data (via data registries), and to develop a common set of data types and preservation services (discoverable via service registries). the testbed is designed to interoperate with the planets infrastructure to access data on which experiments may be executed, and will be able to test and explore the preservation services that conform to the planets specifications. users of the testbed can discover services via the planets service registry and in addition to this may explore third-party services beyond this registry. the web service approach given the centrality of a service-oriented architecture approach to the planets project, it was fundamental to ensure that the preservation tools required for testbed experiments could be deployed and accessed as web services. the web service solution adopted by the project is built upon soap with attachments api for java (saaj) [19]. this api handles the creation, population and transmission of soap messages, and it was adapted by planets to execute generic wsi-compliant web services [20]. the main challenge in adopting a web service approach for the testbed was to ensure that a generic and extensible web service execution approach could be successfully integrated with the testbed’s workflow and user interface, could be easily understood by the testbed users and could enable access not only to planets services but also to third party services. in order to perform experiments on preservation tools these tools must first be wrapped as wsi-compliant web services. these services can then be registered within the testbed and service templates can be created. experimenters can then access these templates to simulate the specific usage of a tool. the steps involved in registering and configuring a service are handled by the testbed administrator. this enables experimenter users such as librarians and archivists to access and use the available services without requiring any knowledge of invocation details at the soap message level or complex service configuration options. different sets of service configuration options are currently presented to users as individual templates. however, user feedback has suggested that this simplified approach is insufficiently flexible; the fine-tuning of tools by end-users is one of the main scenarios of testbed use and the desire for a more flexible approach has been noted in the planets service requirements specification and will be incorporated into the testbed’s future releases. the service registration process consists of a five-step wizard based interface which guides the administrator through the tasks required to make a service available. the following screenshot demonstrates step one of the process. figure 9: step 1 of the service registration wizard during step 1 of the process the administrator must locate and analyse a web service endpoint in order to select the actual service operation that is to be registered with the testbed. step 2 involves creating a service request template with placeholders for supported service input. for step 3 the administrator must upload sample data in order to test that the service can be invoked successfully. in step 4 the administrator has to note where the service’s output (such as a migrated file) can be located and extracted from the xml service response message. in the final step the administrator may supply additional descriptive metadata for the service and apply restrictions on the minimum and maximum number of files the service can accept. the administrator may also supply some semantic metadata in the form of annotation tags which will help the experimenters to locate the service. once a service has been registered it can be experimented upon within the testbed, and all the information required by the testbed to execute an experiment will be available. this includes the information needed to re-factor the soap service request message, construct the actual service execution proxy, invoke the service upon the given experiment data and finally extract the actual service’s response (e.g. the migrated file) in order to store it within the application. the first internal release of the testbed incorporated three planets preservation services: openofficexmlmigration, imagemagictiff2jpeg and a simple characterisation action. however, the web service approach enables any service that adheres to the following constraints to be registered within the testbed’s infrastructure and used within an executable experiment: 1. the web-service must comply with the wsi basic profile v 1.1 2. the service’s wsdl endpoint description must be located and accessible at the uri: endpointaddresss?wsdl 3. service endpoints must identify a service uniquely 4. services and tools currently must be installed locally on one and the same machine 5. the service input parameters may only include a single file reference or an array of file references. other service configuration options are not currently supported and are assumed to be implicitly hidden behind a given service endpoint. next steps of development the first version of the planets testbed was released for internal use by the planets partners in march 2008. a new cycle of requirements gathering, design and implementation will integrate partners’ feedback and intermediate releases will follow, culminating in the release of the testbed to external institutions which is expected to take place in spring 2009. the public version of the testbed will be hosted in a central instance by hatii at the university of glasgow. planets partners and other users will be encouraged to use this central instance to ensure the seamless aggregation of experiment results. this will lead to a higher statistical significance and will make it possible for the community to create an experimental dataset in which they can look for patterns and processes that might otherwise be overlooked. a downloadable version of the testbed will also be made available. the central instance of the testbed will include a number of features that will make it a unique and invaluable reference for the evaluation of digital preservation tools and strategies: integration of digital preservation corpora in digital preservation research, a corpus can be defined as an annotated collection of digital objects, where the annotations should contain the criteria against which given algorithms will be evaluated. [21]. in the planets context, the algorithms will be those implemented by the tools that perform characterisation and preservation actions on the digital test objects. the planets testbed corpora will ensure that a sufficient knowledgebase is available for each experiment, which will avoid duplication of effort for experimenters who would otherwise have to find and upload their own data. the corpora will also facilitate the statistical analysis and aggregation of data. each corpus will contain individual “difficult” objects for the worst-case analysis of a given tool, and large sets of objects of a given type, or very large files, for average-case analysis. an exhaustive list of base types of digital objects used in testbed experiments will be established. the experience gained through the dutch testbed project [10] suggests that it is advisable to aim to collect simple objects with few properties each, and to keep the corpora as uniform as possible with respect to variation of significant properties. the management and maintenance of the testbed corpora will be carried out according to procedures established with the other planets partners; these procedures will formalise the coordination of additions of new objects and object types, of requests to incorporate pre-existing material and the management of access to the corpora. deployment of digital preservation tools to the testbed; certification of third-party tools one of the main results of the planets project will be the development of new digital preservation tools. these tools will be deployed within the testbed and will be evaluated using data from the testbed corpora or data supplied by testbed users. the availability of the corpora of digital objects is also a necessary basis for the certification of third-party tools, an additional feature that the planets testbed will offer in a future release. the testbed will not only make available existing, well-known digital preservation tools, it will also present a unique opportunity to software developers, enabling them to certify their tools against a collection of digital objects unsurpassable for its breadth, following a strict experiment process, and benefiting from previous experiment results collected in the testbed database. preservation plan assessment services the testbed corpora, together with the experiments database, will also be the basis on which preservation plans created by other planets components will be assessed. in the planets general workflow a preservation plan is devised by executing tool evaluation (following a model inspired by utility analysis) on a small number of files that should represent a sample of the collection that a preservation officer wants to preserve. this plan will consist of a general section that will take into consideration the institution’s policies and usage profiles, and an executable section that will detail the services needed for the actual execution of the plan itself. the planets testbed will allow the assessment of the executable plan by testing the services, or chains of services, on its corpora and comparing the results with those that can be found in the experiments database. evaluation of emulation approaches for digital preservation analysis of digital preservation strategies should not be restricted to migration [22]. the planets project is also conducting research on two different emulation approaches: the dioscuri modular emulator [23] and the ibm universal virtual computer [24]. dioscuri is an x86 computer hardware emulator written in java. it is being jointly developed by tessella support services and the national library of the netherlands and is managed by the national archives of the netherlands (the three partners are all members of the planets consortium). dioscuri is completely component-based; each hardware component is emulated by a software surrogate called a module. combining several modules allows the user to replicate the configuration of any computer system, as long as these modules are compatible. new or upgraded modules can be added to the software library, thus expanding the emulator’s capabilities. the ibm universal virtual computer (uvc) was initially conceived by raymond lorie of ibm research in almaden, california as a combined emulation/migration approach. ibm (also a planets partner) conducted proofs of concept with the national library of the netherlands to test the uvc approach in a library environment. within planets, work on the uvc is focussed on improving the performance of the uvc runtime model and expanding the development environment. with emulation, the planets testbed web-service architecture faces the challenge of evaluating tools and services that do not change the object, but rather the representation environment of the object, while leaving the object itself untouched. how to perform and evaluate emulation experiments conducted with dioscuri and uvc as web services is currently being investigated within planets by albert-ludwigs of the university of freiburg using the grate tool [25]; the results of this research will inform the next steps of the testbed development in this domain. conclusion it is widely acknowledged [4, 5] that the management of the long-term accessibility of digital materials depends upon the automation of the processes involved. automation of processes requires a rich understanding of them: their aims and objectives, the mechanisms needed to abstract and represent them, and often new technologies. in developing the testbed, planets has begun a first step in digital preservation to tackle this challenge both by providing tools to manage experimentation and also by creating an experimental evidence base to enable the analysis of the effectiveness of preservation services and strategies. this experimentation framework will foster the development of digital preservation solutions that will significantly increase automation in the daily practice of librarians and archivists, will make software developers more aware of the issues connected to obsolescence, and will ultimately change the way we all produce our digital content. it also ensures that digital preservation research is tackled as an engineering problem rather than as an art and craft. notes [1] preservation and long-term access through networked services (planets), viewed 23 may 2008 <http://www.planets-project.eu> [2] planets publications, viewed 23 may 2008 <http://www.planets-project.eu/publications/?l=1> [3] ross, seamus, and hedstrom, margaret (2005), ‘preservation research and sustainable digital libraries’, international journal on digital libraries, vol. 5/4, 317-324 [4] m. hedstrom and s. ross (eds.) (2003), invest to save: report and recommendations of the nsfâ€�delos working group on digital archiving and preservation, (national science foundation’s (nsf) digital library initiative & the european union under the fifth framework programme by the network of excellence for digital libraries (delos)) <http://eprints.erpanet.org/48/> [5] digitalpreservationeurope, dpe digital preservation research roadmap (2007), viewed 23 may 2008 <http://www.digitalpreservationeurope.eu/publications/dpe_research_roadmap_d72.pdf> [6] gladney, henry (2007), preserving digital information, springer verlag [7] ross, seamus (2007), digital preservation, archival science and methodological foundations for digital libraries, keynote address at the 11th european conference on digital libraries (ecdl), budapest, viewed 23 may 2008 <http://www.ecdl2007.org/keynote_ecdl2007_sross.pdf> [8] duranti, l and thibodeau, k (2006), ‘the concept of record in interactive, experiential and dynamic environments: the view of interpares,’ archival science, vol. 6, no. 1, 13â€�68 [9] watry, paul (2007). digital preservation theory and application: transcontinental persistent archives testbed activity, international journal of digital curation, vol 2, no 2, viewed 23 may 2008 <http://www.ijdc.net/ijdc/article/view/43/50> [10] potter, m (2002), ‘researching long term digital preservation approaches in the dutch digital preservation testbed (testbed digitale bewaring)’, rlg diginews, vol 6 no 3, viewed 23 may 2008 < http://digitalarchive.oclc.org/da/viewobjectmain.jsp?fileid=0000070519:000006287741&reqid=3550#feature2> [11] delos deliverable wp6, d6.1.1 (2004), framework for testbed for digital preservation experiments, november, viewed 23 may 2008 <http://www.dpc.delos.info/private/output/delos_wp6_d611_finalv2(0)_denhaag.pdf> [12] d-lib test suite, viewed 23 may 2008 <http://www.dlib.org/test-suite/research.html> [13] open video digital library, viewed 23 may 2008 <http://www.open-video.org/project_info.php> [14] rothenberg, j & bikson, t (1999), carrying authentic, understandable and usable digital records through time, report to the dutch national archives and ministry of the interior, viewed 23 may 2008 <http://www.digitaleduurzaamheid.nl/bibliotheek/docs/final-report_4.pdf> [15] cole, t. w., mischo. w. h., ferrer, r., & habing, t. g. (2001). “using xml and xslt to process and render online journals.” library hi tech, 19(3), 210-222. [16] thaller, m (2007), characterizing with a goal in mind: the xcl approach, viewed 23 may 2008 <http://www.kb.nl/hrd/congressen/toolstrends/presentations/thaller.pdf> [17] jboss, viewed 23 may 2008 <http://www.jboss.org/> [18] java server faces, viewed 23 may 2008 <http://java.sun.com/javaee/javaserverfaces/> [19] the soap with attachments api for java, viewed 23 may 2008 <https://saaj.dev.java.net/> [20] web services interoperability, viewed 23 may 2008 <http://www.ws-i.org/> [21] neumayer, r, kulovits, h, rauber, a, thaller, m, nicchiarelli, e, day, m, hofman, h & ross, s, on the need for benchmark corpora in digital preservation,viewed 23 may 2008 <http://www.ifs.tuwien.ac.at/~neumayer/pubs/neu07_delos.pdf> [22] emulation expert meeting statement, viewed 23 may 2008 <http://www.kb.nl/hrd/dd/dd_projecten/projecten_emulatie-eemstatement-en.html> [23] dioscuri, viewed 23 may 2008 <http://dioscuri.sourceforge.net/> [24] van diessen, r (2006), ibm’s uvc approach, viewed 23 may 2008 <http://www.kb.nl/hrd/dd/dd_projecten/slides/eem_ibm_rvdiessen.pdf> [25] hoeven, j, houtman, f, suchodoletz, d, lohman, b, schroder j (2007), next steps integration of pa/5 tools (emulation), viewed 23 may 2008 <http://gforge.planets-project.eu/gf/download/docmanfileversion/49/2190/pa5_next_steps_integration_v3.doc> about the authors brian aitken is a designer and developer of web based information management systems. he has been employed both as sole developer and as leader of a development team on a wide range of successful web based projects, most recently the digital curation centre, digital preservation europe and planets. petra helwig worked for several years for the dutch ministry of the interior where she specialised in software engineering and information management. she joined the dutch national archives in 2006 where she works as a digital longevity advisor. andy jackson worked for a number of years as an applications consultant at the edinburgh parallel computing centre. after a few years spent as a post-doctoral researcher in edinburgh and wellington (new zealand), andy joined the british library to work on the planets digital preservation project as a senior software developer. andrew lindley is currently working as a junior researcher (phd student) and software engineer for the austrian research centers gmbh – arc in vienna. he is part of the group’s digital preservation team and the main contact at arc for the eu fp6 ip planets testbed sub-project. eleonora nicchiarelli has been a researcher since june 2006 at the austrian national library in vienna, where she is responsible for the planets testbed subproject. seamus ross is professor of humanities informatics and digital curation and founding director of hatii (humanities advanced technology and information institute) (http://www.hatii.arts.gla.ac.uk) at the university of glasgow. he is an associate director of the digital curation centre in the uk (http://www.dcc.ac.uk), a co-principal investigator in the delos digital libraries network of excellence (http://www.dpc.delos.ac.uk), and principal director of digital preservation europe (dpe) (http://www.digitalpreservationeurope.eu). he was principal director of erpanet, a european commission activity to enhance the preservation of cultural heritage and scientific digital objects (http://www.erpanet.org), and a key player in the digital culture forum (digicult forum) which worked to improve the take-up of cutting edge research and technology by the cultural heritage sector (http://www.digicult.info). subscribe to comments: for this article | for all articles 2 responses to "the planets testbed: science for digital preservation" please leave a response below: digital preservation of science and art | supreme's blogs, 2011-05-08 […] by practices and processes that could best be described as more art and craft than science. the planets testbed provides a controlled environment where preservation tools can be tested and evaluated, and where […] format migration: more launching points for applied research | the signal: digital preservation, 2013-08-15 […] at scale you may want to access large corpora of data such as that provided by biomed central. the planets testbed was a very effective research environment hosted by the european planets project to facilitate […] leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – renewing upei’s institutional repository: new features for an islandora-based environment mission editorial committee process and structure code4lib issue 21, 2013-07-15 renewing upei’s institutional repository: new features for an islandora-based environment in october of 2012, the university of prince edward island (upei) launched an updated version of islandscholar, upei’s institutional repository. the repository, available from http://www.islandscholar.ca, is built on islandora 6 (http://islandora.ca). the repository includes a number of new features, including: csl integration for ingest, site display, and export of user-specific bibliographies; mads-based authority integration for departments and authors (with authorities created automatically using ldap); batch ingest from refworks (crosswalked to mods for storage in the repository); embargo and statistics functions. features from the first version of islandscholar were also migrated to the new site, including sherpa/romeo integration (which provides just-in-time information about open access policies). by donald moses and kirsta stapelfeldt the evolution of upei’s ir the provision of an institutional repository (ir) service has been a priority for the robertson library since 2008 with the release of university of prince edward island’s strategic research plan 2008?2018.  that document included as a strategic priority the development of, “a publicly accessible institutional repository which will contain citations for all research conducted at upei, as well as, in accord with copyright protocols, access to copies of our research documents, publications, and reports.” [1] the first version of islandscholar.ca was created shortly after and included the seeding of the repository with faculty citations harvested from refworks, tying-in authentication with upei’s ldap service, the ability for faculty to add their own citations and attach the related publication, and integration with sherpa/romeo [2], openurl and coins [3]. during february 2012 the university released a new policy entitled open access & dissemination of research output [4] which encourages: scholars to deposit their scholarly research in islandscholar, scholars to publish in journals that allow scholars to retain copyright in their work, scholars to consider open access and affordable / sustainable scholarly communication venues, scholars to deposit other types of research output in islandscholar (preprints, research data, etc.), students to deposit their honours thesis, master’s or phd thesis on acceptance. the policy specifies additional requirements such as ensuring that the content was harvestable and that deposited research data be linked to the final scholarly output.  to fulfill these needs and others, a set of requirements was drafted for an islandscholar v2 and the new service was launched in october 2012. as with the first islandscholar, the system was designed and is delivered using islandora software, which was first developed by the robertson library. summary of changes islandscholar v2 received a design update with a new, responsive theme that enables the site to be accessed from most mobile devices. the data structure in the repository was modified to include additional objects and relationships, with the aim of improving the granularity and accuracy of the system’s ontology, and facilitating new metrics in the future. other features include the ability for users to create and format citation lists, and access scholar-specific rss feeds. the data in the repository is harvestable using oai’s protocol for metadata harvesting (pmh). an administrative user can now set the embargo date for content, and expect the embargo to be automatically lifted. scholar deposit has also become simpler, and the project has recently begun writing code to expose alternative metrics, creating additional value for scholars seeking to understand the scope of their scholarly impact, ir administrators seeking to improve impact and guide researchers, and administrative groups. about the islandora architecture islandora is an open-source software framework designed to help institutions and organizations and their audiences collaboratively manage, and discover digital assets using a best-practices framework.  islandora was originally developed by the university of prince edward island’s robertson library, but is now implemented and contributed to by an ever-growing international community. islandora allows for digital assets to be stewarded following best practices for asset modelling by defining packages of data assets, derivatives, and xml-based administrative, technical, and descriptive metadata, with relationships to other data assets described in rdf. not only is islandora designed to serve best-practices for digital preservation, it also allows users to manage and discover assets, and for diverse user groups to create digital exhibitions, perform and collaborate on research, and enrich resources. a number of open source applications form the structure of the islandora framework which can be extended by integrating additional applications to meet new requirements. the front-end/interface for islandora is provided by the popular content management system, drupal [5]. elements of islandora are templated so that drupal’s theming layer can be extended to islandora elements (for example, displays and behaviors, search results displays, etc.). drupal’s form api is also used to provide form building and editing for any xml-based metadata schema, workflows can also be orchestrated, numerous roles and permissions can be assigned, and drupal permits integration with external identity stores using ldap and shibboleth. the islandora code layer provides integration with numerous third-party software as direct plug-ins (eg. imagemagick [6], fits [7], lame [8], tesseract [9], internet archive bookreader [10], open seadragon [11], etc.), or through the islandora micro services engine [12]. this software can be called upon to create derivatives and perform data transformations or extraction for data that is then stored in the digital object repository layer, built using fedoracommons [13] software. micro services in islandora leverage the jms server that comes packaged with fedoracommons. fedoracommons repository models and stores data assets and provides the repository’s rdf triple store (mulgara) as well as governing security policies (written in xacml [14]) for access and management of objects. in islandora 7 a new api, tuque [15], simplifies the process of interacting with and creating fedoracommons data. fedoracommons’ generic search service (gsearch) processes the data and passes it to solr where it is indexed by lucene. islandora includes a solr client module [16] that provides an administrative interface for configuring search, results, and facets.  for an overview of this framework, see the diagram below. figure 1. overview of islandora architecture. islandora releases  “solution packs” designed to serve particular populations or types of data. the robertson library’s approach to modeling data in islandora provides essential feedback for the ongoing development of an institutional repository solution pack. to learn more about the islandora project and plans for solution packs, visit the project’s website at http://islandora.ca. the following describes the way that islandscholar v2 leverages the islandora architecture to provide a number of ir related services. modelling metadata and relationships in islandscholar islandscholar contains collections of digital objects that adhere to fedora content models.  within fedora, content models define the composition of the digital objects (specifying what they contain — the “datastreams” —  and how they behave — the “methods”).  currently the following types of objects are modeled in islandscholar and the figure below illustrates their relationships: citations (both faculty publications and student theses) researchers departments grants figure 2. digital objects in islandscholar and their relationships. grant information has recently been added. citations islandscholar contains two content models related to citation-type metadata – faculty publications and student theses.  each of these objects, publications and theses are composed of files (or “datastreams” in fedora terminology). these files are described in figure three. datastream id description rels-ext metadata in rdf xml format that describes the relationships this object has to other objects in the repository. mods[17] metadata that describes the citation object in mods format. dc a transformed version of the mods stored as dublin core – a required fedora datastream. obj an optional datastream. this datastream is the document referenced by the citation and could be an original manuscript, a preor post-print, the published article, etc. pdf an optional datastream.  when an obj datastream is added to islandscholar (office type document or pdf), it is automatically normalized as a pdf/a file using islandora’s micro-services framework. zip an optional datastream. a zip package of any supplemental material that might be associated with the citation. e.g., an excel spreadsheet, spss datafile, questionnaire, etc. that has been packaged in a zip file. policy an optional datastream. an xacml [18] policy that governs access to the digital object or selected datastreams. for instance, for a citation object the citation data can be exposed, but the pdf that is part of the digital object may be embargoed, and so only those with appropriate rights can manage/view that particular datastream. figure 3. the structure of a digital citation object. the student thesis content model is similar in structure, but the mods metadata is extended with the <extension> element so that etd-ms [19] metadata can be included. etd-ms metadata is specific to electronic theses and dissertations and meets the metadata requirements for participation in library and archives canada’s e-thesis harvesting program [20]. researchers and departments recognizing that many citations would have authors entered in a variety of ways, we chose to create authority records for the people and organizations represented in the data. we describe these entities using the metadata authority description schema (mads), an xml schema that provides metadata about people and organizations (and other elements) [21].  mads is a complimentary schema to mods and shares some of the same elements.  having authority records for researchers allows us to associate citations with related researcher authority records – facilitating the creation of browse displays and search queries that return expected results. the figure below illustrates a researcher profile page for a faculty member and it is composed of data from the mads datastream, the thumbnail datastream from the researcher record, and data from objects that are related to the researcher. figure 4. a research profile page from islandscholar [22]. researcher records can be created in a number of ways within the islandscholar system: through an associated import from the university’s ldap system, adding a user through drupal’s users module, or by using islandora’s xml form/ingest framework. basic data about researchers is contained within the university’s ldap system; with the drupal ldap integration module [23] enabled and configured, we are able to pull data directly from the campus ldap server. when logged in with the role of administrator, an associate button is available for each citation. when selected it searches both the ldap directory and existing researcher records in the repository, looking for likely name matches between the citation and researchers and displaying those records. this process highlights existing associations within ldap and the repository and other possible associations. the results are then presented to the user (see figure five). appropriate associations can then be selected and created by the administrator, and are written in rdf to the rels-ext datastream of the citation object. figure 5. adding a researcher record using an ldap lookup. occasionally the researcher record will not be available via the campus ldap. in this case, to manually add a researcher record, an authorized user navigates to the collection containing the researcher records, selects add, selects the authority content model and proceeds to the data entry form. figures six and seven show a user’s view of how this process unfolds. figure 6. adding a researcher record using islandora figure 7. sample xml form view for a researcher record. grants grants have recently been added to the islandscholar framework – part of a desire to flesh out the researcher’s profile with additional data and a desire to associate citations with the grants that funded them.  currently this is a manual process whereby we map a csv export from upei’s research services to mods. we are actively investigating the casrai research activity profile [24] as a better match for modelling this data in the future. currently grants are associated with researchers and the data is displayed on researcher profile pages as a block. integration of citation style language in the first version of islandscholar, a pre-defined, generic citation display was created.  to provide options for citation formats for both display and export, one of the goals of this version of islandscholar was to integrate the citation style language as a method to reliably generate citations formatted in particular styles.  citation style language (csl) is “an open xml-based language to describe the formatting of citations and bibliographies.” [25]   there is a large collection of community contributed styles [26] available for download and these styles can be added to and used by islandscholar. figure eight  (below) illustrates the styles already added to islandscholar and the option for adding additional styles. a list of currently installed styles appears, and a user with the appropriate permissions can upload a new style, which then becomes available in the system. figure 8. administering csl styles in islandscholar the islandscholar administrator can set a default style for the entire system within the scholar administration panel (see figure nine). figure 9. setting the default style for islandscholara a csl editor [27] has recently been released that can be used to author your own custom styles if one from the community does not fit your use case. for example, at upei some researchers would like to be able to output their citations in the format required by particular funders and a custom style could accommodate this request. to generate formatted citations from the mods metadata in islandscholar,  developers on the project  first created php code [28] to convert the mods metadata stored in islandscholar into the csl json format. to accomplish this task the code parses the mods and outputs it in a format that conforms to csl – for example, a “journal article” (a local genre term in the mods metadata) is transformed to “article-journal” in the corresponding json. names and dates are also reformatted so that they match csl requirements. for many other fields xpath queries of the mods are used to populate csl variables. the resulting csl json is then handed to the citeproc-js [29] javascript module, the style is applied, and a formatted citation is the result. importing and exporting citation metadata importing/ingesting citations into the institutional repository islandscholar leverages a number of different technologies to provide users with a choice of formats to be used when adding citations into the repository.  the islandscholar application currently supports the import of citations in ris [30], endnote xml [31], refworks xml, or mods bibliographic formats. a ris, endnote xml, or refworks xml document can generate a single or multiple digital objects on ingest (one for each record within the ris, endnote xml, or refworks document uploaded to the repository). the islandora 7 version of the scholar solution-pack module (currently under development) provides additional support for single and batch ingest using unique identifiers such as digital object identifiers (dois) and pubmed ids (pmids). at the robertson library, the primary method used to manage and harvest citation metadata is refworks. citations are harvested into refworks and are then edited to include islandscholar specific information (identifiers for scholars and departments). once the citations have been enriched, they are then exported as refworks xml files.  figure ten illustrates the options for a user when ingesting a refworks file of citations into the citation collection. figure 10. citation import interface during refworks xml import, islandscholar reads the reference type <rt> element from the  refworks record, passes it through a matching xslt and transforms the refworks xml to mods. the relevant refworks xml is shown in figure eleven. <reference> <rt>journal article</rt> <sr>electronic(1)</sr> <id>11237</id> <a1>shiri,ali</a1> <a1>revie,crawford w.</a1> <t1>usability and user perceptions of a thesaurus-enhanced search interface</t1> <jf>journal of documentation</jf> <yr>2005</yr> <fd>10</fd> <vo>61</vo> <is>5</is> <sp>640</sp> <op>656</op> <sn>00220418</sn> <do>10.1108/00220410510625840</do> <u1>crevie</u1> <u2>health management</u2> <ol>unknown(0)</ol> </reference> figure 11. sample refworks record the refworks record is parsed and since <rt> = journal article the record is passed through the refworks_to_mods_journal.xsl [32] transformation (there are other transformations for other genres). figure twelve shows the resulting mods formatted record that is stored in islandscholar. <mods xmlns="http://www.loc.gov/mods/v3" xmlns:mods="http://www.loc.gov/mods/v3" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:xsi="http://www.w3.org/2001/xmlschema-instance"> <titleinfo> <title>usability and user perceptions of a thesaurus-enhanced search interface</title> <subtitle/> </titleinfo> <name type="personal"> <namepart type="given">ali</namepart> <namepart type="family">shiri</namepart> <role> <roleterm authority="marcrelator" type="text">author</roleterm> </role> </name> <name type="personal"> <namepart type="given">crawford w.</namepart> <namepart type="family">revie</namepart> <role> <roleterm authority="marcrelator" type="text">author</roleterm> </role> </name> <identifier type="u1">crevie</identifier> <identifier type="u2">health management</identifier> <typeofresource>text</typeofresource> <genre>journal article</genre> <identifier type="refworks">11237</identifier> <relateditem type="host"> <titleinfo> <title>journal of documentation</title> <subtitle/> <partnumber/> </titleinfo> <origininfo> <dateissued keydate="yes">2005</dateissued> <issuance>continuing</issuance> <publisher/> </origininfo> <part> <date>2005</date> <detail type="volume"> <number>61</number> </detail> <detail type="issue"> <number>5</number> </detail> <extent unit="page"> <start>640</start> <end>656</end> </extent> </part> <identifier type="issn">00220418</identifier> <identifier type="isbn"/> </relateditem> <identifier type="doi">10.1108/00220410510625840</identifier> </mods> figure 12. sample transformed mods record for files in the ris or endnote format, islandscholar utilizes bibutils [33] to transform those formats to a standardized mods metadata format.  bibutils is a set of command line tools used to transform various bibliographic formats to mods. exporting / sharing data bibliography a new feature added to islandscholar v2  is the ability to select citations and add them to a list. the resulting list can be managed, and bibliographies generated in user-specified styles can be exported in several file formats.  users can add citations to a list from any search result page or from a researcher’s profile. figure thirteen illustrates a search result with two citations that have been added to a list. figure 13. citations added to a list. when users wish to view / export the select citations, they select the my list option and choose the desired output citation format.  several export options are available including html, rtf, pdf, and ris. figure fourteen illustrates the options provided to a user. figure 14. exporting citations in the case illustrated in figure fourteen, the option for html export has been selected. this means that an html export passes the citations through csl and outputs the result to the screen (see figure fifteen) figure 15. html export of citations in the apa 6th style rtf and pdf exports are handled by bibutils and the selected styles are applied. bibutils also generates the tagged ris export [34]. oai-pmh another goal (and a requirement for e-thesis harvesting) in islandscholar v2 was the integration of the open archives initiative – protocol for metadata harvesting (oai-pmh).  this was achieved by enabling and configuring the islandora oai module [35], a module that exposes and transforms the mods datastream of digital objects in islandora to oai harvesters. once the oai module has been configured, a harvester can be pointed to particular collections. oai output is governed by the module and the mods datastream is passed through different stylesheets to return the requested metadata. for example, a mods to oai-dc stylesheet is applied to return results from the researchers’ citation collection as oai-dc. so, for example, an oai request like http://www.islandscholar.ca/oai2?verb=listrecords&metadataprefix=oai_dc&set=ir_citationcollection will provide results shown in figure sixteen. figure 16. top of a response provided by islandscholar v2 for an oai-dc request. to break down the oai request the harvester has requested a list of records (verb=listrecords) in the oai dc metadata format (metadataprefix=oai_dc) for the records contained in the repository’s ir:citationcollection collection (set=ir_citationcollecton). in order for our e-theses to be harvested, islandscholar must make etd-ms metadata available. to facilitate this, we configure the  oai module to include a transformation [36] of mods to etd-ms and expose the resulting metadata with the islandora oai module. for example, a request like http://www.islandscholar.ca/oai2?verb=listrecords&metadataprefix=oai_etdms&set=ir_thesiscollection will provide results shown in figure seventeen. figure 17. top of a response provided by islandscholar v2 for an etd-ms request. in this case the harvester has requested records in the etd-ms metadata format (metadataprefix=oai_etdms) from the repository’s ir:thesiscollection collection (set=ir_thesiscollecton). figures fifteen and sixteen illustrate the ways that separate metadata profiles can be exposed. the module that enables this function was developed generically and includes a variety of configuration options. see figure eighteen for a screenshot of one of the sections of the administration screen: figure 18. configuring etd-ms metadata for oai harvesting. rss islandscholar v2 also provides a feed of a researcher’s citations, updated when new publications are associated with the researcher. this has value for the researcher wishing to embed an rss feed in a faculty profile page or make similar use of an up-to-date list of publications (often required for tenure decisions, etc.). in islandscholar, a basic rss feed for researchers is generated by passing a solr query to an xslt. this is a feature that we hope to more fully develop in future work. figure nineteen shows this sample xml feed [37]. <?xml version="1.0" encoding="utf-8"?> <rss version="2.0"> <channel> <title>an islandscholar citation feed</title> <link>http://vre2.upei.ca:8080/solr</link> <description> a scholar's citation feed. </description> <language>en-us</language> <docs>http://vre2.upei.ca:8080/solr</docs> <item> <title>slow library</title> <link> http://islandscholar.ca/fedora/repository/ir:ir-batch6-5796</link> <description>leggott, mark; "slow library", , 2006, , </description> <pubdate>2012-10-19t12:06:20.848z</pubdate> <guid> http://islandscholar.ca/fedora/repository/ir:ir-batch6-5796</guid> </item> <item> <title>digital dumpster? preserving and accessing digital records</title> <link> http://islandscholar.ca/fedora/repository/ir:ir-batch6-5804</link> <description>leggott, mark; "digital dumpster? preserving and accessing digital records", , 2005, , </description> <pubdate>2012-10-19t12:06:30.322z</pubdate> <guid> http://islandscholar.ca/fedora/repository/ir:ir-batch6-5804</guid> </item> <!-snip ... more <item> elements follow --> </channel> </rss> figure 19. sample rss feed output a link to a researcher’s feed is available from their profile page. managing access to content one of the primary methods used to manage access to islandora content is by the inclusion of a policy datastream in digital objects.  policy datastreams are written in xacml, an xml based language used to define access control policies [38] in fedora. when combined with drupal’s existing user authentication and roles, fine grained access levels can be achieved at the collection, object, or datastream level.  figure twenty shows a snippet from a policy datastream of a citation object that illustrates the users (fedoraadmin, mleggott) and roles (administrator, robertson library) that can edit this object. <rule effect="deny" ruleid="denyapi-except-to-user"> <condition functionid="urn:oasis:names:tc:xacml:1.0:function:not"> <apply functionid="urn:oasis:names:tc:xacml:1.0:function:or"> <apply functionid="urn:oasis:names:tc:xacml:1.0:function:string-at-least-one-member-of"> <!-the drupal users that can edit/administer this object --> <subjectattributedesignator attributeid="urn:fedora:names:fedora:2.1:subject:loginid" datatype="http://www.w3.org/2001/xmlschema#string" mustbepresent="false"/> <apply functionid="urn:oasis:names:tc:xacml:1.0:function:string-bag"> <attributevalue datatype="http://www.w3.org/2001/xmlschema#string" >fedoraadmin</attributevalue> <attributevalue datatype="http://www.w3.org/2001/xmlschema#string" >mleggott</attributevalue> </apply> </apply> <!-the drupal roles that can edit/administer this record --> <apply functionid="urn:oasis:names:tc:xacml:1.0:function:string-at-least-one-member-of"> <subjectattributedesignator attributeid="fedorarole" datatype="http://www.w3.org/2001/xmlschema#string" mustbepresent="false"/> <apply functionid="urn:oasis:names:tc:xacml:1.0:function:string-bag"> <attributevalue datatype="http://www.w3.org/2001/xmlschema#string" >administrator</attributevalue> <attributevalue datatype="http://www.w3.org/2001/xmlschema#string">robertson library</attributevalue> </apply> </apply> </apply> </condition> </rule> figure 20. xacml snippet while all attempts are made to ensure that content is accessible within the repository, there are some use cases when associated content needs to be embargoed.  the full text of student theses can be embargoed and there is a global xacml policy that applies to records that contain an embargo date. xacml policies can be difficult to write (and can potentially lock your object or the entire repository) and while not used in the current version of islandscholar, a visual xacml editor [39] is now part of the islandora toolset. in the future version of islandscholar, the xacml editor will be used to simplify the application and management of embargo policies in islandscholar. scholar deposit as with the first iteration of islandscholar, the robertson library made the choice to ‘seed’ the repository with the collected citations of scholars at upei and encourage researchers to contribute whichever version of the article is permitted by the publishing policy of the journal in which they published. to help scholars decide which version of their work can be legally submitted, islandscholar v2 also maintained and updated the ability for researchers to review publisher copyright and self-archiving policies by integrating with the sherpa/romeo api. researchers are also permitted in islandscholar v2 to append any relevant associated research data to a citation by creating a .zip file of relevant materials and appending during the submission process. figure twenty-one illustrates the screen that is presented to a scholar appending documents to a citation. figure 21. scholars identify the version and appropriate license. islandora’s microservices transform the uploaded file to a pdf/a for long-term preservation. the system supports any office-type format, eg. .doc, .docx, .odt, .xls, etc.. the resulting files are stored as part of the citation object. metrics islandscholar v2 provides more advanced tools for measuring downloads of an article and page views, and has begun integrating altmetrics into the tool set. figure twenty-two shows how this appears for scholars. scholars log into the system and can view all citations, with a column showing the last date the citation was viewed, as well as total number of times the article was viewed. figure 22. a scholar can review the times a citation has been viewed. navigating directly to the record will show a user how many times an article has been downloaded, and when it was last downloaded (see figure twenty-three). figure 23. from the citation, users can see the number of times an article has been downloaded, and when it was last downloaded as the publishing platform for scholarly data, and the ways scholars share and interact with research has evolved, altmetrics has emerged as an approach that aims to account for those activities outside of the traditional peer-reviewed journal stream. [40] from the ciation view, islandscholar v2 provides altmetrics to scholars and visitors to the system. while currently in the development stage, we are optimistic about the potential for this application to show just-in-time altmetrics alongside more traditional measures of popularity and impact (such as downloads and views). if an article in the system has a doi, pmid, or arxiv, a badge appears in the citation’s full record view. scrolling over the badge shows a pop-up with altmetric statistics and a link to a more complete perspective on the impact of the article. this process is visible in figure twenty-four. figure 23. an altmetrics badge reveals altmetrics to visitors conclusion upei is proud of the recent upgrade to islandscholar and the number of new tools that have been integrated to address emerging best-standards practice and the needs of ir users. at the same time, the team is committed to ongoing iteration and development.  embracing an open and community-centric approach, islandscholar will continue to evolve to incorporate and develop new tools, and to take advantage of emerging tools. this work is also part of the ongoing development of islandora and islandora solution packs. members of the community are warmly welcomed to visit and explore islandscholar.ca and provide feedback to help guide development in the future. references and notes [1] strategic research plan 2008?2018 – http://research.upei.ca/files/research/v9%20senate%2022apr08.pdf [2] sherpa/romeo – http://www.sherpa.ac.uk/romeo/ [3] coins – http://ocoins.info/ [4] open access & dissemination of research output –  https://cab.upei.ca/sites/default/files/attachments/openaccessanddisseminationofresearchoutput.pdf [5] drupal – http://drupal.org [6] imagemagick – http://www.imagemagick.org/ – used in a variety of solution packs for image conversion [7] fits – https://code.google.com/p/fits/ integrated into the islandora framework with the islandora fits module – https://github.com/islandora/islandora_fits [8] lame – http://lame.sourceforge.net/ integrated into the islandora framework with the audio solution pack – https://github.com/islandora/islandora_solution_pack_audio [9] tesseract – https://code.google.com/p/tesseract-ocr/ integrated into the islandora framework with the islandora ocr module – https://github.com/islandora/islandora_ocr [10] islandora internet archive bookreader – https://github.com/islandora/islandora_internet_archive_bookreader [11] islandora open seadragon viewer – https://github.com/islandora/islandora_openseadragon [12] there are at least two ‘flavours’ of micro services used in the islandora context: python based – https://github.com/islandora/islandora_microservices; and php based – https://github.com/roblib/php_listeners [13] fedora commons – http://fedora-commons.org [14] fedora xacml policy writing guide – https://wiki.duraspace.org/display/fedora34/fedora+xacml+policy+writing+guide [15] build, access, modify and delete fedora objects with the tuque interface – https://github.com/islandora/islandora/wiki/build,-access,-modify-and-delete-fedora-objects-with-the-tuque-interface [16] islandora solr search module – https://github.com/islandora/islandora_solr_search [17] metadata object description schema (mods) – http://www.loc.gov/standards/mods/ [18] xacml (extensible access control markup language) is an xml-based policy language enforced by fedora at both the repository-wide and object level. read more about xacml and fedora at https://wiki.duraspace.org/display/fedora36/xacml+policy+enforcement. xacml is also discussed in more detail in a later section of this article. [19] etd-ms: an interoperability metadata standard for electronic theses and dissertations. http://www.ndltd.org/standards/metadata/etd-ms-v1.00-rev2.html [20] about electronic theses: harvesting program. http://www.collectionscanada.gc.ca/thesescanada/027007-9200-e.html [21] metadata authority description schema (mads). http://www.loc.gov/standards/mads/ [22] the researcher profile page for mark leggott. http://www.islandscholar.ca/fedora/repository/ir:mleggott [23] ldap integration module. http://drupal.org/project/ldap_integration [24] casrai research activity profile. http://dictionary.casrai.org/research-activity-profile-draft [25] citation style language. http://citationstyles.org/ [26] citation styles. https://github.com/citation-style-language/styles and http://citationstyles.org/”> [27] csl style editor. http://editor.citationstyles.org/about/ [28] islandscholar – converter.php. https://github.com/roblib/islandora_scholar_upei/blob/master/modules/citeproc/generators/converter.php [29] citeproc-js. https://bitbucket.org/fbennett/citeproc-js/wiki/home [30] ris file format. http://en.wikipedia.org/wiki/ris_%28file_format%29 [31] endnote. http://en.wikipedia.org/wiki/endnote [32] xslt used to transform refworks journal citation to mods https://github.com/roblib/islandora_scholar_upei/blob/master/xsl/refworks_to_mods_journal.xsl [33] http://sourceforge.net/p/bibutils/home/bibutils/ [34] islandora bibliography export.  https://github.com/roblib/islandora_scholar_upei/blob/master/citation/bibliography/export.inc [35] islandora oai module. https://github.com/islandora/islandora_oai [36] etd-ms xslt: https://github.com/islandora/islandora_oai/blob/7.x/transforms/mods_to_etdms.xsl [37] sample rss feed output: http://www.islandscholar.ca/rss/mleggott [38] https://wiki.duraspace.org/display/fedora36/fedora+xacml+policy+writing+guide [39] islandora xacml editor. https://github.com/islandora/islandora_xacml_editor [40] learn more about altmetrics at http://altmetrics.org/manifesto/ about the authors donald moses, mlis, is the digital initiatives and systems librarian at the university of prince edward island’s robertson library, participates in the islandora community, and is a member of the management team that oversees the virtual research environment (vre), including islandscholar, at upei. kirsta stapelfeldt, ma, mlis, is the manager of the islandora project at the upei’s robertson library, and has helped develop the vre framework at upei, and served as a subject matter expert on a variety of projects. the authors would like to acknowledge the programmers that have developed the islandscholar code: paul pound and richard wincewicz. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – machine learning and the library or: how i learned to stop worrying and love my robot overlords mission editorial committee process and structure code4lib issue 41, 2018-08-09 machine learning and the library or: how i learned to stop worrying and love my robot overlords machine learning algorithms and technologies are becoming a regular part of daily life – including life in the libraries. through this article, i hope to: * to introduce the reader to the basic terminology and concepts of machine learning * to make the reader consider the potential ethical and privacy issues that libraries will face as machine learning permeates society * to demonstrate hypothetical possibilities for applying machine learning to circulation and collections data using tensorflow/keras and open datasets through these goals, it is my hope that this article will inspire a larger, ongoing conversation about the utility and dangers of machine learning in the library (and concurrently society as a whole). in addition, the tripartite division of the article is meant to make the material accessible to readers with different levels of technical proficiency. in approaching the first two goals, the discussion is focused on high level terms and concepts, and it includes specific public cases of machine learning (ab)use that are of broad interest. for the third goal, the discussion becomes more technical and is geared towards those interested in exploring practical machine learning applications in the library. by charlie harper introduction computers may not be enjoying strawberries and cream yet (turing 1950), but they are learning to do a lot of things once thought to be uniquely human. tasks that were not so long ago difficult for a computer to complete accurately are now trivial. take, for example, the recognition of handwriting. try to think of a formal list of steps that you could give a computer in order for it to recognize handwriting. what formal steps do you follow when you read a handwritten document? how do you know that squiggle is a 3 or those two different looking scrawls are both 7s? it’s hard, if not impossible, to give a computer any formalized approach to this problem, let alone articulate how we as humans complete the process. still, we do it seamlessly every single day because we’ve learned by seeing many examples and our brains are built to learn this ability (although, not without error, as you may find when you try to read that doctor’s prescription!). this is what lies at the heart of machine learning, too. machine learning is not telling a computer how to, say, recognize handwriting in specific, formulaic steps; it’s programmatically building the structure necessary to learn this ability and then providing the examples from which to learn it. the concept of machine learning (henceforward ml) has existed since the early days of computing (turing 1950; rosenblatt 1958; samuel 1959). only over the past decades, though, has the realistic application of ml been widely realized. the drastic improvements in the capabilities of machines to learn and to perform complicated tasks accurately has been due, in large part, to three accompanying changes:   computational power has increased dramatically, with the rise of parallel computing, powerful gpus, and cloud computing of particular note the big data revolution has meant that datasets have ballooned in size and scope so that there are ample examples from which machines can learn programming libraries and approaches have been developed that abstract away difficult mathematical and computational problems, and as a result, they have democratized the implementation of ml the appearance of ml in our everyday lives (e.g. web searching, voice recognition, spam filtering, product recommendations, credit application approvals, red-light cameras, etc.) is occurring at a rapid pace, and its expansion across all aspects of human life will continue. what does this mean for the library? well, at a most fundamental level, ml is precipitating and will continue to precipitate major and unpredictable societal changes. as centers of society and long-standing advocates of democratic principles, the opportunities and challenges arising from ml’s growth will pervade the library in every regard (arlitsch and newell 2017); staffing, patrons, funding, resources, and the entire community dynamic, to name a few things, will be impacted by coming technological changes. to meet these changes, we need to educate ourselves on the basic concepts that underlie ml, actively discuss the unresolved (unresolvable?) ethical dilemmas that the growing use of ml is spawning, and be aware of how ml can be and is being applied within the library itself. what is machine learning? rather than programming a computer to follow a set of steps, ml is programming the computer to learn a set of steps based on examples or experience. when i say set of steps, what i’m properly talking about is an algorithm, or “a sequence of instructions that are carried out to transform the input to the output” (alpaydin 2016, 16). a simple example of an algorithm is a mathematical formula, like x + y – z. given this algorithm and a set of three numbers as input, a computer will produce a single number as output. for problems where there is a logical or mathematical solution, traditional algorithms leverage the immense power of computers to work wonders. there are many problems in life, though, where we can’t envision or create an algorithm. the example given above of recognizing handwriting is one. still, this is a task that we’d like machines to complete and to do so with high accuracy. this is where ml comes in. whereas a traditional algorithm gives fixed steps to take an input and produce some output, ml uses an initial algorithm which takes examples of what we’d like it to learn as input and uses these examples to output a new algorithm that it has learned from the data (figure 1). the process of feeding in examples to produce this learned algorithm is known as training. the learned algorithm, or what is called a model, can then be fed real world data, such as an image of a handwritten digit, as input and it will output a prediction, such as what it thinks this digit is. figure 1. a traditional algorithm (a) versus a machine learning algorithm (b).   the ml algorithms used to generate models are diverse, and certain algorithms are best suited for learning from particular types of data. without getting bogged down in the details, though, each ml approach can be broadly categorized as one of three types of learning: supervised, unsupervised, and reinforcement. supervised learning (alpaydin 2016, 38-54) trains with labeled data; in other words, each point of data in the dataset is already marked with the correct answer in advance. for our handwriting example, the training data could be a set of images of individual written characters, and each image includes a label with the correct character. the supervised learning algorithm will input each image, make a prediction, and check this against the label. as a result, the training process is able to measure (i.e. supervise) how well it’s performing and it will then attempt to iteratively improve the model over time as it sees more examples. unsupervised learning (alpaydin 2016, 111-118), in contrast, uses datasets that are unlabeled. in order to learn from an unlabeled dataset, the algorithm applies statistical techniques with the goal of uncovering hidden patterns within the data. a common type of unsupervised learning is clustering, in which the algorithm tries to form coherent groups from the data. then, when real world examples are fed into the trained model, a prediction will be made based on the group(s) to which the real world example is closest. clustering documents, such as news articles, to generate categories or topics based on statistical similarities is one practical example of unsupervised learning. reinforcement learning (alpaydin 2016, 125-132) relies on feedback from actions in order to learn. in reinforcement learning, instead of being fed a dataset, an agent acts within an environment, such as a game or a physical maze. within this environment, the agent is allowed to take different actions based on the state of environment, such as moving left or right in a maze. to learn which actions it should take over time, a reward function is used. in a game, the agent might receive a reward for scoring points or, in the case of the maze, for reaching the end.  the goal is for the agent to learn a set of actions that maximizes this reward. whereas supervised learning is compared to learning with a teacher (i.e. being told if your answer was right or wrong immediately using labeled data), this approach has been compared to learning with a critic, who only tells you whether you did well or poorly (i.e. using the reward function) after you take a series of actions (alpaydin 2016, 127). a deeper look at a supervised learning example will illustrate a bit more about how ml works in practice. as an example, let’s take a common problem presented to ml beginners: train a model using passenger data to predict who survived the titanic disaster. in itself, this trained model has no real world application, but it’s easy to see how something very similar, such as a model predicting survival rates in car accidents based on driver and car characteristics, does have practical implications. the titanic dataset (titanic: machine learning from disaster) contains 1,309 passenger records and each record has 10 attributes or features (e.g. age, sex, fare, ticket class, etc.). one of these is a binary label indicating whether the passenger survived (=1) or not (=0), so we can apply a supervised ml approach to this data.   in practice, big datasets can have billions of records and thousands of features. using all of these features would increase the computational time needed to train a model and might undermine your model’s predictive power by introducing irrelevant data. in any dataset, you need to identify only the important features relying on domain knowledge, statistical analysis, intuition, and experimentation. this process is known as feature selection. for the titanic dataset, i know that women and children were given preference in lifeboats, so my domain knowledge tells me that sex and age are likely important features to select. on the other hand, name seems unimportant. meanwhile, passenger class and ticket fare are probably strongly correlated. do i need to keep both of these features? only by training a model and verifying its accuracy will you discover whether you selected good features or not. the process of applying ml to data is truly both science and art. it’s important to test the accuracy of your model and to avoid the problem of overfitting. if we were to feed all 1,309 passenger records into the ml algorithm, our model could reach  100% accuracy, but this is really misleading. it would have seen every example of data and therefore adjusted itself to account for all the nuances. we couldn’t then objectively test whether our model was actually good at predicting unseen data. in supervised learning, this is overcome by splitting data into a training set and a testing set (figure 3). a general rule-of-thumb is to use a random 80/20 split of the original dataset. the training set is exclusively used to train the model. afterwards, the testing set is fed into the trained model and its predictions are compared to the testing labels. this provides an objective measure of model accuracy and shows if your model is generalizable, or good with data it hasn’t seen before. this goal of model generalization is essential for real world ml applications. at that point, if the accuracy is insufficient, then other choices must be made with the design of the ml algorithm or feature selection, and a new model must be trained. figure 3. training and testing in supervised machine learning.   machine learning (ab)uses the power of ml algorithms to learn from large datasets and then generate predictive models is exciting and frightening. it’s opening up new pathways for scientific and humanistic exploration, and raising pressing ethical questions at the same time. to address these concerns requires cross-disciplinary discussion and this naturally places the library in a key position to facilitate and enable such discussions. the past years have witnessed a number of remarkable applications of ml. a few important examples are discussed below and their implications for ethical debates are raised. it’s worth noting that these examples are not meant to reflect the entirety of ml’s impact on society and i don’t discuss overtly positive examples, such as the great benefit ml has provided for medical diagnostic imaging; the below are examples that raise deep questions about the future role of ml in society and the examples may be unfamiliar to many. to underscore the points i’d like to draw from each, i discuss the examples under four headings: garbage in, garbage out; knowing the unknown; anonymized doesn’t mean anonymous; and redefining reality. the lines between each of these headings and the ensuing examples are fuzzy, though, and the groupings should be taken as a starting point for discussion and nothing more.   garbage in, garbage out it’s an old adage and easy to understand: inputting garbage data outputs garbage results. with ml, the problem of bad data and bad results has taken on new importance and created new difficulties because what constitutes “bad” data and results isn’t always clear or objectively definable. data input into ml algorithms can often be bad unintentionally. this is particularly troubling when ml models are put into action before the validity of their results has been rigorously tested, and even if properly tested there’s no guarantee that models will continue to produce valid results. that is to say, ml doesn’t remove human subjectivity and error from decision making and prediction. instead, it can actually reify human behavior in complex and unpredictable ways through the data on which a model is trained. image recognition technologies are a great example of how poorly selected training data can produce problematic models. in a study of ibm, microsoft, and face++ image gender classification products, buolamwini and gebru (2018) found significant discrepancies in the models’ accuracy based on race and gender. these products performed between 8.1% to 20.6% worse at assigning gender to images of women than men. for darker skinned women, the difference in error rate increased up to 34.4% when compared to images of lighter skinned males (buolamwini and gebru 2018, 8, 10). as a result, the authors argued that image training datasets need to include broader phenotypic and demographic representations of people (buolamwini and gebru 2018, 12). in other words, if your training data skews towards white males, then the accuracy of your model will also skew that way. even if done unintentionally, poorly sampled, biased, or just plain bad training data will bake itself into your model. on the other hand, when it comes to training with purposefully bad data, there’s no better example than chatbot tay. in a move that can be hailed as only slightly more brilliant than the windows phone, microsoft designed tay to interact with and learn (i.e. train itself) from unfiltered tweets, including giving tay the special ability to tag photos that were tweeted at it. tay’s first moments of life on the internet were angelic and innocent. within hours, however, the crush of twitter trolls had converted poor, ignorant tay, the ‘teenage’ ai bot, into a sexist, racist, genocidal supporter of hitler (vincent 2016). microsoft quickly unplugged tay (and as if to compound our collective heartbreak, the windows phone was officially euthanized only months later). tay is an extreme example, but bad actors can alter ml models purposefully by manipulating training data. determining how, when, and by whom a large dataset has been manipulated for possibly ulterior motives isn’t a straightforward process. for this reason, documenting and publicizing the source of a dataset and its “chain of custody,” so to speak, are more important than ever. someday that dataset might end up training an ml model! simultaneously, vetting the provenience of a dataset and thoroughly understanding its composition is a must for anyone building an ml model that may end up in real world use.   knowing the unknown the ability of ml to reveal hidden traits about ourselves and others is striking. from a medical perspective, revealing what we as humans cannot see provides an amazing tool for diagnosing and treating disease. concurrently, ml’s penetrating gaze is altering the nature of privacy and diminishing our individual power to keep things about ourselves hidden from public view. it also has the potential to reveal things about ourselves that we did not know and did not want to know. a recent study found that facial images contain much more information about sexual orientation than a human can recognize (wang and kosinski 2018). when given five images of an individual, an ml model was able to correctly predict sexual orientation 91% of the time for males and 83% of the time for females. with only a single image, its accuracy was still a staggering 81% for males and 71% for females. take a moment to consider this level of accuracy and just how easy it is to acquire a mere five images of someone. what else might be accurately predictable from data that was once thought benign? how will such predictive power be deployed in good and bad ways? another major group of ml models being used to reveal the unknown are found in criminal risk-assessment tools. one such tool is compas (correctional offender management profiling for alternative sanctions), a product of a corporation called northpointe. the tool uses a model trained to predict recidivism. a propublica study showed that the assessment tool predicts recidivism correctly 61% of the time (an accuracy which already strikes me as particularly heinous given its use), but when broken down by race, black defendants are twice as likely as white defendants to be labeled as at a high risk for recidivism, but not actually re-offend (angwin et al. 2016). these predictions by compas are especially consequential because they’re used by judges during criminal sentencing. as of 2016, propublica noted that the results of these risk-assessments were given to judges in arizona, colorado, delaware, kentucky, louisiana, oklahoma, virginia, washington, and wisconsin when they considered how to sentence a defendant (angwin et al. 2016). meanwhile, the durham (uk) police have been using hart (harm assessment risk tool) to predict a suspect’s risk of committing a crime in the next two years (burgess 2018, oswald et al. 2018). hart utilizes a number of categories, including age and gender, to rank an individual as at low, moderate, or high risk of reoffending. hart also originally utilized postcode information, a spatial variable that risked creating a feedback loop in which police target certain areas for more scrutiny and therefore generate more arrests in those areas. the higher volume of arrests in these postcodes then feedbacks into the model and iteratively draws further police scrutiny (oswald et al. 2018, 228-229). in a positive example of external pressure, the publicization of this feedback danger has led the durham police to remove the postcode variable from hart (burgess 2018). another notable feature of hart, which is in tandem with the prediction of sexual orientation from facial features, is that there’s a significant discrepancy between what hart predicts and what a human predicts (burgess 2018). in the case of both compas and hart, decisions that significantly impact the lives of individuals are being guided by ml models. this raises serious questions not only about the data with which these models are being trained, but about the role of predictive technologies in criminal justice. models generated by ml algorithms are opaque to human scrutiny; the whole point of ml is that humans can’t formulate algorithmic steps to make these predictions, so instead the ml algorithms generate complex models to make predictions. as a result, it’s very difficult to know how the model is working internally. models are tested for accuracy after training, but this procedure is again limited by the dataset available to you and it’s directly impacted by the choices you’ve made in feature selection, such as choosing sex or postcode as features. to what extent should technologies that cannot be easily scrutinized by defendants, especially when the technologies are proprietary, be drawn into criminal justice? propublica cites at least one objectionable example of compas’ current impact: an already agreed upon plea deal between defendant and prosecutor was explicitly rejected by a judge as a result of the defendant’s high-risk score produced by compas (burgess 2018). the judge then sentenced the defendant to two years in state prison rather than the agreed upon one year in county jail.   anonymized doesn’t mean anonymous large datasets are at the heart of ml and modern analytics. to protect individuals’ privacy on matters such as health, criminal victimization, or academic performance, datasets are traditionally anonymized before analysis, but anonymous and private have never been the same thing. can you guess where i’m going with this? now, in the era of big data and ml, anonymized and anonymity are no longer the same thing either as it’s increasingly possible to re-identify supposedly anonymous data. textually, this has manifested in the field of stylometry, which statistically analyzes characteristics of writers’ language. in the digital humanities, stylometry appears extensively for literary studies, but its applications are wider ranging. for example, it can also be used to deanonymize article submissions. payer et al. (2015) found that once an author had published at least three articles, it was possible to begin attributing anonymous submissions to that author. like all ml approaches, this type of deanonymization relies on existing data, but as individuals post written content to social media, they’re also unwittingly building a dataset that ml algorithms can consume for future textual analysis. meanwhile, social networks themselves can be de-anonymized (ma et al. 2017). what impact will this power of deanonymization have on the future of free and open discussion (what if you couldn’t check out books anonymously)? another example of deanonymization that is frighteningly futuristic mixes ml and genomic data. the danger and ability to identify individuals by name from anonymized genomic data without the use of ml has already existed for a number of years (sweeney, abu, and winn 2013). using ml techniques, genomic re-identification has reached a new, inconceivable level. instead of compiling disparate datasets to assign names to genomic samples, genomic samples alone are now providing identifying information. with a trained ml model, lippert et al. (2017) recreated the faces of individuals from their genomic sequences. the recreations are not perfect, but they are striking in their similarity (figure 4). what does it mean that you may be leaving your name or even an image of your face everytime you leave behind dna? obviously, this could have great impact on criminal forensics, but it’s a slippery slope. if this and similar technologies become readily available, how else might they be deployed and what limits can/should we place on their use? figure 4. actual faces, left, and ml facial reconstructions from genomic data, right (lippert et al. 2017, fig. 2)   redefining reality perhaps the most awesome power of ml that has emerged recently is its ability to alter and create media that challenges our sense of reality. this ability has been driven by the invention of generative adversarial networks (gans). gans work by harnessing the power of two neural networks, which compete against each other. one network generates (hence generative) something, such as a fake image. the other network fights (hence adversarial) to separate out the faked content from real content. as each network improves, their pairing drives the generative network to create progressively better content in order to better fool the other neural network. the result is an internal arms race. these powerful gans have created some shockingly realistic content, including astonishing images of fake celebrities (figure 5; karras et al. 2018; metz and collins 2018) and art in various styles (elgammal, liu, elhoseiny, and mazzone 2017). figure 5. images of fake celebrities generated by a gan (karras et al. 2018, fig. 10)   deepfakes (wikipedia: deepfakes, 2018) and similar spin-offs, though, have been the most extreme example of gans. they have also been called, “one of the cruelest, most invasive forms of identity theft invented in the internet era” (foer 2018). not limited to still images, deepfakes creates realistic videos of existing individuals. in april 2018, buzzfeed and comedian jordan peele released a “real fake” psa video of obama, who gave his opinion of donald trump and warned of the dangers of fake news. the technology has also notoriously been used to transfer the faces of celebrities, such as emma watson, taylor swift, and emilia clarke, into pornography (dold 2018; lee 2018). right now this technology is mostly limited to celebrities because of the number of images required for it to work well in video, but this is not true for all gans (and it won’t be true of deepfakes forever). lyrebird (https://lyrebird.ai/) provides a similar functionality for audio. it claims that with only a single minute of recorded audio, it can create a voice that sounds like anyone (mark 2018). like the five images required to accurately predict sexual orientation, one minute of recorded audio isn’t much. in light of such powerful gans, existing discussions of “fake news” are quaint. what happens when the news really is fake in the truest sense of the word? as it stands, ml means that our images, voices, and mannerism are copyable and mimicable. this presents the very real possibility for a loss of ownership of our own identities and it eviscerates existing notions of reality by undermining what our eyes and ears tell us is “real.” the dystopian possibilites are palpable.   the role of the library there are many benefits arising from ml, but also great potential for unintended consequences and outright wrongdoing. recognizing our own ignorance of ml’s future developments and impacts, and avoiding the all-to-common fallacy that computer results are tantamount to an absolute truth is fundamental. the library needs to participate in shaping this future, and there are particular areas of focus that it should consider. offering guidance on information literacy, data documentation, and source vetting all fall within the traditional realm of a library’s activities. these issues are exceedingly more important than in the past and are becoming increasingly complicated. traditional models of instruction and collaboration on these topics need continuous improvement to match the changing times; they also require the activate role of technically-minded individuals whose education falls outside the scope of traditional lis degrees. the privacy (or really lack thereof) and ethics of data collection and dissemination should become an integral part of information literacy services. now that images, video, and audio can be faked in staggering ways, the reality of source origin is becoming increasingly messy, too. facilitating and promoting critical thinking and awareness within the community is a must. since ml is a cross-disciplinary and cutting-edge field, libraries should act to bring together diverse groups for ongoing discussions that focus on ethical guidelines and the societal implications of ml. humanists, scientists, community members, and elected officials each have differing stakes in the outcomes of ml and the library can provide the forum and guidance necessary to mediate their views in a meaningful way. one particular point of exploration should be the increasing primacy of metrics that reduces human complexity to a mathematical exercise. the same metrical slant that has led to the creation of dubious predictors of recidivism is spreading, and its growth is particularly clear across all levels of academia and education. assessments and rankings that are easily measurable have become the norm (muller 2018, 65-101). in higher education, libraries are particularly in danger of being overly drawn into this academic ranking mania. because of its centrality to education, there are many calls to begin aggregating library data with larger university data in order to effectively create university-wide data centers. matthews (2012) pointed out that despite anecdotal and gate count evidence, almost all academic libraries really have no idea of who is actually using the library and for what purposes. he argued that, as a result, the library should look beyond its own data and examine the larger picture of university outcomes by aggregating internal data with university-wide data in an accessible data warehouse. equally, matthews argued that a large amount of data should be collected within the library for the purpose of assessment and demonstrating library value to the university. this intensive collection and aggregation scheme would be “the enabling tool that will allow a library to prepare credible analysis of the library’s impact in the lives of its students, faculty, and researchers” (matthews 2012, 400). renaud et al. (2015) implemented such a system in limited form by aggregating data sets to measure the correlation of gpa and library use. again, the argument was made that this was a good way to show library impact on university wide goals. in light of growing datasets, such library records are just now starting to be subjected to ml. for example, litsey and mauldin (2017) generated ml predictions of ill usage and dreamed of an eventual system that would “[monitor] all library transactions from circulation, the holdings, the ill transactions, the reserves items and even the ejournals” (litsey and mauldin 2017, 143). the growing collection of library data, aggregation with university-wide data, and nascent application of ml to these dataset are all done with good intentions, but they are extremely precarious. as we know apparently anonymized datasets are not necessarily private, and as data is united in more complex ways it becomes increasingly more powerful. the same mentality of assigning a recidivism score during a criminal sentencing is easily extended to assigning a “success” score to individual students. this possibility may be too attractive for administrators to resist, and libraries should by all means responsibly analyze their own data, but the walls of the library shouldn’t be breached, and collection management shouldn’t be solely driven by machine predictions. an additional point to make is that every single dataset has the potential to become a source for predictive ml models. seemingly benign library data can equally reveal amazing and unexpected things about people. libraries are now acting as massive stores of data, including personal information, borrowing histories, reference questions, entrance/exit data, security surveillance records, program and meeting attendance, room rentals, and web surfing habits. this data is so extensive it can even be considered “big data” (berman 2018). concurrently, there’s the growing problem of 3rd parties who offer access to e-resources, and collect and retain information on patrons (rubel 2014). these 3rd parties aren’t necessarily held to the same standards of privacy as libraries and librarians must carefully review their policies and advocate against 3rd party data collection. fister raised an excellent question when she asked, “do the new capabilities of data mining offer libraries valuable opportunities to peer into individual students’ experiences to improve learning and demonstrate value – or does it pose a significant challenge to one of librarians’ most revered values, patron privacy?” (fister 2015). although the two terms have overlap, we could easily replace the words ‘data mining’ with ‘machine learning’ and ask the same question. the problems of privacy, intellectual freedom, and censorship that arise from big data and ml, including the emerging conflicts with traditional library ethics and values, are just beginning to be addressed (jones and salo 2018). ultimately, libraries need to be attentive to these fast-paced socio-technical changes and to keep a taut finger on the mechanical pulse of machine learning. i’ll readily admit that i don’t know where this is all going, but i know that we’re all going along for the ride willingly or not, and the library must do so with a watchful eye.   references alpaydin e. 2016. machine learning: the new ai. cambridge (ma): mit press. angwin j, larson j, mattu s, kirchner l. 2016. machine bias. propublica [internet]. [cited 6 june 2018] available from: https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing arlitsch k, newell b. 2017. thriving in the age of accelerations: a brief look at the societal effects of artificial intelligence and the opportunities for libraries. journal of library administration 57:789-798. berman, e. 2018. big brother is watching you: the ethical role of libraries and big data [internet]. [cited 13 july 2018] available from: https://chooseprivacyeveryday.org/the-ethical-role-of-libraries-and-big-data/ buolamwini j, gebru t. 2018. gender shades: intersection accuracy disparities in commercial gender classification. proceedings of machine learning research 81:1-15. burgess m. 2018. uk police are using ai to inform custodial decisions–but it could be discriminating against the poor. wired [internet]. [cited 7 june 2018] available from: http://www.wired.co.uk/article/police-ai-uk-durham-hart-checkpoint-algorithm-edit dold, c. 2018. face-swapping porn: how a creepy internet trend could threaten democracy. rolling stone [internet]. [cited 10 june 2018] available from: https://www.rollingstone.com/culture/features/face-swapping-porn-how-creepy-trend-could-threaten-democracy-w518929 elgammal a, liu b, elhoseiny m, mazzone m. 2017. can: creative adversarial networks generating “art” by learning about styles and deviating from style norms [internet]. [cited 10 june 2018] available from: https://arxiv.org/abs/1706.07068 fister b. big data or big brother? data, ethics, and academic libraries [internet]. [cited 12 july 2018] available from: https://barbarafister.net/libigdata.pdf foer f. 2018. the era of fake video begins. the atlantic [internet]. [cited 7 june 2018] available from: https://www.theatlantic.com/magazine/archive/2018/05/realitys-end/556877/ jones kml, salo d. 2018. learning analytics and the academic library: professional ethics commitments at a crossroads. college & research libraries 79(3):304-323. karras t, aila t, laine s, lehtinen j. 2018. progressive growing of gans for improved quality, stability, and variation. icrl 2018 [internet]. [cited 11 june 2018] available from: http://research.nvidia.com/sites/default/files/pubs/2017-10_progressive-growing-of/karras2018iclr-paper.pdf lee d. 2018. deepfakes porn has serious consequences [internet]. [cited 10 june 2018] available from: https://www.bbc.com/news/technology-42912529 lippert c, sabatini r, maher mc, kang ey, lee s, arikan o, harley a, bernal a, garst p, lavrenk v, et al. 2017. identification of individuals by trait prediction using whole-genome sequencing data. pnas [internet]. [cited 10 june 2018] available from:  https://doi.org/10.1073/pnas.1711125114 litsey r, mauldin w. 2017. knowing what the patron wants: using predictive analytics to transform library decision making. journal of academic librarianship 44:140-144. ma j, giao y, hu g, huang y, sangaiah ak, zhang c, wang y, zhang r. 2017. de-anonymizing social networks with random forest classifier. ieee access 6: 10139-10150. mark t. 2018. can you believe your own ears? with new ‘fake news’ tech, not necessarily. npr [internet]. [cited 10 june 2018] available from: https://www.npr.org/2018/04/04/599126774/can-you-believe-your-own-ears-with-new-fake-news-tech-not-necessarily marsland s. 2015. machine learning, 2nd ed. boca raton (fl): crc press. matthews jr. 2012. assessing library contributions to university outcomes: the need for individual student level data. library management 33(6/7):389-402. metz c, collins k. how an a.i. 2018. ‘cat-and-mouse game’ generates believable fake photos. new york times, 2 january 2018 [internet]. [cited 10 june 2018] available from: https://www.nytimes.com/interactive/2018/01/02/technology/ai-generated-photos.html muller, jz. 2018. the tyranny of metrics. princeton (nj): princeton university press. oswald m, grace j, urwin s, barnes gc. 2018. algorithmic risk assessment policing models: lesson from the durham hart model and ‘experimental’ proportionality. information & communications technology law 27(2):223-250. payer m, huang l, gong nz, borgolte n, frank m. 2015. what you submit is who you are: a multimodal approach for deanonymizing scientific publications. iee transactions on information forensics and security 10(1):200-212. renaud j, britton s, wang d, ogihara m. 2015. mining library and university data to understand library use patterns. the electronic library 33(3):355-372. rosenblatt f. 1958. the perceptron: a probabilistic model for information storage and organization in the brain. psychology review 65(6):386-408. rubel, a. 2014. libraries, electronic resources, and privacy: the case for positive intellectual freedom. the library quarterly: information, community, policy 84(2):183-208. rubel a. 2014. libraries, electronic resources, and privacy: the case for positive intellectual freedom. the library quarterly 84(2):183-208. samuel al. 1959. some studies in machine learning using the game of checkers. ibm journal of research and development 3(3):210-229. sweeney l, abu a, winn j. 2013. identifying participants in the personal genome project by name (a re-identification experiment). data privacy lab [internet]. [cited 11 june 2018] available from: https://privacytools.seas.harvard.edu/publications/identifying-participants-personal-genome-project-name titanic: machine learning from disaster [internet]. 2018. kaggle. available from: https://www.kaggle.com/c/titanic. turing am. 1950. computing machinery and intelligence. mind lix(236):433-460. vincent j. 2016. twitter taught microsoft’s ai chatbot to be a racist asshole in less than a day. the verge [internet]. [cited 6 june 2018] available from: https://www.theverge.com/2016/3/24/11297050/tay-microsoft-chatbot-racist wang y, kosinski m. 2018. deep neural networks are more accurate than humans at detecting sexual orientation from facial images. journal of personality and social psychology 114(2):246-257. wikipedia, the free encyclopedia, s.v. “deepfake,” (accessed august 6, 2018), https://en.wikipedia.org/wiki/deepfake about the author(s): charlie harper is the digital scholarship librarian at case western reserve university, where he collaborates on and manages a variety of digital projects. much of his work currently centers on the interdisciplinary applications of machine learning, text analysis, and gis. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – countering stryker’s punch: algorithmically filling the black hole mission editorial committee process and structure code4lib issue 37, 2017-07-18 countering stryker’s punch: algorithmically filling the black hole two current digital image editing programs are examined in the context of filling in missing visual image data from hole-punched united states farm security administration (fsa) negatives. specifically, photoshop’s content-aware fill feature and gimp’s resynthesizer plugin are evaluated and contrasted against comparable images. a possible automated workflow geared towards large scale editing of similarly hole-punched negatives is also explored. finally, potential future research based upon this study’s results are proposed in the context of leveraging previously-enhanced, image-level metadata. by michael j. bennett figure 1. roy e. stryker, head of the historical section (information division) of the u.s. farm security administration. photograph by russell lee, august 1938 (lee, 1938) i. introduction the visual record left behind by the united states farm security administration (fsa) photographers has, according to new york times critic charles hagen, come to “represent one of the most ambitious attempts ever made to depict a society in photographs (hagen, 1985).” leading the fsa’s historical section was roy stryker, a columbia-trained economist. though stryker was not a photographer himself, he understood the power that images could have in economic argument and the general sway of ideas. in fact, new deal propaganda on the plight and needed social assistance for the rural impoverished was one of the historical section’s main aims. early during his tenure, stryker would hire a small but remarkable group of photographers who would go on to express new documentary styles of depicting rural america during the great depression. walker evans, dorothea lange, and ben shahn, among other notable artists, all served under stryker, who in many ways was a pivotal support to them during times of acute economic hardship, and was a strong advocate towards their subsequent post-fsa careers. as his staff would send back their negatives from the field, it was stryker who would determine which images would be printed. those that did not pass his visual inspection he would “kill” by punching a hole in the transparency with a hole punch. to this day, articles continue to lament this past practice, which has certainly affected stryker’s legacy and the resulting historical visual record (taylor, 2017)(arbuckle, 2016)(banks, 2010). figure 2. untitled photo, possibly related to: girl in washington, d.c. slum area. photograph by john vachon, december 1937 (vachon, 1937b). however, with today’s advanced editing tools it is interesting to ask if these black holes might be revisited and perhaps algorithmically filled in just a few mouse clicks? and if such a workflow was consistently successful, could it be automated across a large quantity of hole-punched images? ii. gathering test images and comparing tools to begin it was deemed best to start with a clearer sense of the total number of negatives that were killed yet were still archived. luckily the library of congress’ prints and photographs online catalog (ppoc) hosts the farm security administration/office of war information black-and-white negatives as a digital collection (“farm security administration/office of war information black-and-white negatives – about this collection – prints & photographs online catalog (library of congress),” n.d.). within this collection, each punched negative has a note in its bibliographic record that states, “negative has a hole punch made by fsa staff to indicate that the negative should not be printed.” the note entry is an indexed field of the descriptive record, so its contents are searchable. a quick look-up, then, for “hole punch” resulted in 2,472 total images. examples with either complex detail surrounding the punch or in cases where the hole has completely removed significant image context were excluded from this initial study. figure 3. killed image with significant lost detail. untitled photo, possibly related to: start of the three-legged race. fair at albany, vermont. photograph by carl mydans, september 1936 (mydans, 1936a). two of the main criteria used in the author’s subsequent examination of photo editing tools included usability, and scalability. in turn, elaborate selecting, layering, masking, and individualized cloning techniques were not studied. regardless of the proven power of these traditional but highly manual methods, the focus of this inquiry was to gauge how effective newer, more automated tools have become. content aware-fill first became available in photoshop cs5 in 2010 (goldman, shechtman, barnes, belaunde, & chien, n.d.). foundational investigations that went into the tool’s development came from studies done on the completion of space-time holes in video sequences (wexler, shechtman, & irani, 2007), and work towards a randomized algorithm for quickly finding approximate nearest neighbor matches between image patches (barnes, shechtman, finkelstein, & goldman, 2009). these efforts were conducted by researchers at the weizmann institute of science, princeton university, university of washington, and adobe’s creative technologies lab. with new software releases over time, adobe often makes iterative optimizations to their tools and the algorithms that these features depend upon. the workflow tests that follow were conducted by the author with photoshop cc 2017.0.1 on an apple macbook pro (os x 10.11.6, 16gb ram, 3.1 ghz intel core i7 processor, 1tb ssd). here is an example of how content aware-fill was used on one of the fsa negatives: figure 4. untitled photo, possibly related to: arkansas tenant farmer. photograph by ben shahn, october 1935 (shahn, 1935). using photoshop’s elliptical marquee tool, a tight selection was made around the hole’s edge. figure 5. from photoshop’s edit menu > fill > content-aware > ok. figure 6. deselect. done… figure 7. before and after photoshop content-aware fill that the entire process takes four clicks and roughly 10 seconds was a promising revelation. equally encouraging was the seamlessness of the resulting fill. here is a 100% detail of the mend from the previous image: figure 8. detail of content-aware fill the following are examples of other images that were similarly filled using photoshop: figure 9. before and after photoshop content-aware fill. untitled. photograph attributed to walker evans (evans, n.d.). figure 10. before and after photoshop content-aware fill. untitled photo, possibly related to: sharecropper and sharecropper’s dog. north carolina. photograph by john vachon, april 1938 (vachon, 1938a). figure 11. before and after photoshop content-aware fill. floyd burroughs, jr., and othel lee burroughs, called squeakie. son of an alabama cotton sharecropper. photograph by walker evans, summer 1936 (evans, 1936). figure 12. before and after photoshop content-aware fill. untitled photo, possibly related to: spectators at county fair, central ohio. photograph by ben shahn, august 1938 (shahn, 1938). it is interesting to note that besides photoshop’s content-aware fill, gimp’s resynthesizer plugin also offers similar functionality within a piece of image editing freeware (david, 2012). though photoshop’s tool gets much of the attention for being the swiss-army knife of quick and effective image filling, resynthesizer’s code actually pre-dates the release of content-aware fill (harrison, 2005). gimp’s latest v.2.8.18 loaded with the resynthesizer plugin was used by the author to test usability and effectiveness against content-aware fill on comparable images. one of the details that immediately became apparent when employing gimp was how ingeniously designed the built-in fine selection guides were within the software’s ellipse select tool. as a result, very precise selections could be much more easily made than with photoshop’s elliptical marquee tool: figure 13. fine-tuning gimp’s ellipse select tool after selecting the hole, the subsequent workflow steps, though nominally unique to gimp, were in essence the same as photoshop’s. in the gimp, filters > enhance > heal selection were chosen: figure 14. using gimp’s heal selection this was then followed by a dialog box where the sampling radius around the selection could be tweaked. the radius setting affects the region of texture sampling that is used to inform the resynthesizer plugin’s algorithm on how best to heal the selection: figure 15. setting radius parameter, then running gimp resynthesizer plugin on selection once initiated, the plugin ran noticeably slower than photoshop’s content-aware fill command. based upon further sampling from four test images, each run in both programs, the following trend emerged. what required photoshop on average 10 seconds to accomplish, from selection to fill, gimp took 34 seconds to finish the same set of operations. and though the resulting visual analyses were subjective, the healed holes in gimp were consistently more conspicuous and not blended as well as the ones that photoshop filled. however, the differences between programs in this regard were subtle on average. figure 16. untitled photo, possibly related to: man with homemade pipe, washington, d.c. photograph by john vachon, april 1937 (vachon, 1937a). figure 17. detail of photoshop content-aware fill result vs. gimp resynthesizer plugin heal result iii. testing automated selection and filling the last question examined by the author was whether such image processing could be automated, from selection to final image filling. photoshop is a flexible piece of software that allows for several different approaches to similar editing issues. however, it is important to remember that one of the guiding principles of this investigation was to find simple strategies that were effective and repeatable. the biggest challenge in attempting to remove all manual steps from the established content-aware fill workflow was to automate the initial hole selection. for this, photoshop’s color range selection option was tested. color range allows for a selection to be drawn around areas within an image that have a particular color. the color can be sampled from within a given image and be set with an eyedropper tool. as the fsa images are all grayscale, the color range setting for the selection tool was pegged at a black level found in one of the image’s holes and was then used thereafter across all the batch processed images. once the color range selection tool settings were tweaked to satisfaction, the tool’s deployment along with the content-aware fill steps were recorded into a photoshop action, which was then run across a source folder of hole-punched images in an automated batch. the initial action step then was select > color range. this brought up a dialog box where the following choices were made and the black eyedropper was used to sample from the hole: figure 18. select > modify > expand to give the content-aware fill tool some additional image data from around the hole to sample from: figure 19. untitled photo, possibly related to: sharecropper and sharecropper’s dog. north carolina. photograph by john vachon, april 1938 (vachon, 1938b). edit > fill > content-aware figure 20. before and after color range auto selection and content-aware fill using photoshop action as the example shows, the auto selection process was not exact enough to limit the resulting fill to just the punched hole. film edge perforations were also selected through the color range process and as a result underwent extraneous content-aware fill. though somewhat promising, further automation trials also exposed additional problems with imprecise selection. beyond edge perforations, other random areas of the image that have similar black values to the punched hole can also be included in the selection. these extraneous areas often gave content-aware fill confusing information to work with and led to mixed results. figure 21. before and after color range auto selection and content-aware fill using photoshop action. untitled photo, possibly related to: vermont farmer and sheep near north troy. photograph by carl mydans, august 1936 (mydans, 1936b). iv. summary as the results of this study indicate, automated image filling currently has limitations of scale when faced with attempting to rebuild highly complex image detail or to fill in the loss of entire objects (e.g. faces). however, when used in conjunction with more manual methods, automation can potentially be employed in a hybrid manner to achieve enhanced post-processing throughput on large image aggregations. an area of potential future research on the examples in this study includes the use of neighboring negatives. though all the hole-punched negatives are technically untitled, the library of congress, through manual visual analysis, has enhanced the catalog records of these images to include information on other known, titled, and related un-killed images (note the phrase, possibly related to: in the killed images’ title fields). in the case of the 35mm images, the related killed and un-killed images most likely were shot on the same roll of film. a browse by call number link below a given killed negative’s thumbnails brings the user to a screen where related images can be viewed in a grid. figure 22. browse neighboring items by call number link. in the example depicted in figure 23, the two neighboring images were most likely shot in rapid succession on the same film roll. figure 23. neighboring items grid view could such un-killed images be used in the future to check the accuracy of related filled images? could these un-killed images be used in some way as source data by a fill tool to better inform the tool on how to more accurately perform its work? (michel, 2017) finally, it is interesting to step back and consider the nature of the hole-punched fsa negatives from their original creation on mostly 1930s-era 35mm black and white film. if it were not for their recognized visual value, safe transfer to the library of congress in 1944 (as requested by stryker), initial digitization in the 1990s, and subsequent re-digitization between 2010-2014, there would be limited image data to work with today (“farm security administration/office of war information black-and-white negatives – digitizing the collection – prints & photographs online catalog (library of congress),” n.d.). that such image repurposing as described in this study is even possible is largely based upon a compelling confluence of transcendent photography, collection building foresight, careful digitization decisions and execution, open online access, and clever software algorithms that continue to be refined by intelligent minds. yet, what are we to make of these surrogate negatives? though they are not based upon standard content replication like their hole-punched brethren, the software-filled versions still hold a certain spell and feel of visual symmetry. from an archival standpoint we may even regard them as born-digital objects in their own right. perhaps in the end they may simply be best considered informed guesses on fragments of displaced history. as time goes on, and software tools are further enhanced, and perhaps someday the negatives are re-digitized once again to even higher standards, it will be fascinating to re-run similar fill routines on the images to determine if our best visual estimates inch any closer to a truth that we will never be able to fully perceive. about the author michael j. bennett is head of digital imaging and conservation at the university of connecticut. there he oversees the digital capture and conservation operations for the university’s archives and special collections. previously he has served as project manager of digital treasures, a digital repository of the cultural history of central and western massachusetts and as executive committee member for massachusetts’ digital commonwealth portal. his research interests include technologies and techniques that focus on digitization, post-processing, and 2d and 3d data formats. he holds a ba from connecticut college and an mlis from the university of rhode island. bibliography arbuckle, a. (2016, march 26). why some great depression photos were punched full of holes. retrieved march 3, 2017, from http://mashable.com/2016/03/26/great-depression-killed-photos/ banks, eric. (2010, october 4). william e. jones: punctured – the paris review. retrieved may 4, 2017, from https://www.theparisreview.org/blog/2010/10/04/william-e-jones-punctured/ barnes, c., shechtman, e., finkelstein, a., & goldman, d. b. (2009). patchmatch: a randomized correspondence algorithm for structural image editing. acm transactions on graphics (proc. siggraph), 28(3), 24. david, p. (2012, august 27). pat david: getting around in gimp – heal selection (resynthesizer). retrieved april 6, 2017, from https://patdavid.net/2012/08/getting-around-in-gimp-heal-selection.html evans, w. (1936). floyd burroughs, jr., and othel lee burroughs, called squeakie. son of an alabama cotton sharecropper. retrieved march 13, 2017, from http://www.loc.gov/pictures/collection/fsa/item/fsa1998017017/pp/ evans, w. (n.d.). [untitled]. retrieved march 13, 2017, from http://www.loc.gov/pictures/collection/fsa/item/fsa1998016897/pp/ farm security administration/office of war information black-and-white negatives – about this collection – prints & photographs online catalog (library of congress). (n.d.). retrieved march 20, 2017, from http://www.loc.gov/pictures/collection/fsa/ farm security administration/office of war information black-and-white negatives – digitizing the collection – prints & photographs online catalog (library of congress). (n.d.). retrieved april 12, 2017, from http://www.loc.gov/pictures/collection/fsa/digitizing.html goldman, d., shechtman, e., barnes, c., belaunde, i., & chien, j. (n.d.). adobe research?» content-aware fill. retrieved april 8, 2017, from https://research.adobe.com/project/content-aware-fill/ hagen, c. (1985). american photographers of the depression farm security administration photographs: 1935-1942. new york: pantheon books. harrison, p. (2005). image texture tools. retrieved from http://www.logarithmic.net/pfh-files/thesis/dissertation.pdf lee, r. (1938, august). roy e. stryker, photograph chief of the u.s. farm security administration standing in street, probably in washington, d.c. retrieved march 13, 2017, from https://www.loc.gov/pictures/item/fsa1997023350/pp/ michel, p. (2017, april 12). digital conversion coordinator, library of congress, private communication. mydans, c. (1936a, september). [untitled photo, possibly related to: start of the three-legged race. fair at albany, vermont]. retrieved june 5, 2017 from http://www.loc.gov/pictures/collection/fsa/item/fsa1997002469/pp/ mydans, c. (1936b, august). [untitled photo, possibly related to: vermont farmer and sheep near north troy]. retrieved march 13, 2017, from http://www.loc.gov/pictures/collection/fsa/item/fsa1997002541/pp/ shahn, b. (1935, october). [untitled photo, possibly related to: arkansas tenant farmer]. retrieved march 13, 2017, from http://www.loc.gov/pictures/collection/fsa/item/fsa1997016285/pp/ shahn, b. (1938, august). [untitled photo, possibly related to: spectators at county fair, central ohio]. retrieved march 13, 2017, from http://www.loc.gov/pictures/collection/fsa/item/fsa1997018930/pp/ taylor, a. (2017, february 28). holes punched through history – the atlantic. retrieved march 3, 2017, from https://www.theatlantic.com/photo/2017/02/holes-punched-through-history/518115/ vachon, j. (1937a, april). [untitled photo, possibly related to: man with homemade pipe, washington, d.c.]. retrieved march 13, 2017, from http://www.loc.gov/pictures/collection/fsa/item/fsa1997003067/pp/ vachon, j. (1937b, december). [untitled photo, possibly related to: girl in washington, d.c. slum area]. retrieved march 13, 2017, from http://www.loc.gov/pictures/collection/fsa/item/fsa1997002977/pp/ vachon, j. (1938a, april). [untitled photo, possibly related to: sharecropper and sharecropper’s dog. north carolina]. retrieved march 13, 2017, from http://www.loc.gov/pictures/collection/fsa/item/fsa1997003141/pp/ vachon, j. (1938b, april). [untitled photo, possibly related to: sharecropper and sharecropper’s dog. north carolina]. retrieved march 13, 2017, from http://www.loc.gov/pictures/collection/fsa/item/fsa1997003138/pp/ wexler, y., shechtman, e., & irani, m. (2007). space-time completion of video. ieee transactions on pattern analysis and machine intelligence, 29(3). retrieved from http://www.wisdom.weizmann.ac.il/~vision/videocompletion/spacetimecompletion_pami07.pdf subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – how we built a spatial subject classification based on wikidata mission editorial committee process and structure code4lib issue 51, 2021-06-14 how we built a spatial subject classification based on wikidata from the fall of 2017 to the beginning of 2020 a project had been carried out to upgrade spatial subject indexing in north rhine-westphalian bibliography (nwbib) from uncontrolled strings to controlled values. for this purpose, a spatial classification with around 4,500 entries was created from wikidata and published as skos (simple knowledge organization system) vocabulary. the article gives an overview over the initial problem and outlines the different implementation steps. by adrian pohl background the north rhine-westphalian bibliography (nwbib) is a regional bibliography that records literature about the german state of north rhine-westphalia (nrw), its regions, places and people. as of now, nwbib comprises around 440,000 resources. nwbib collects monographs, articles, as well as other media like maps, dvds, etc. the cataloging takes place in the north rhine-westphalian library service centre (hbz) union catalog. the nwbib’s public interface for end users is the bibliography’s website at https://nwbib.de. the hbz union catalogue contains more than 21 million cooperatively maintained bibliographic titles including the german union catalogue of serials (zdb) [1] plus holding information from libraries in north rhine-westphalia and rhineland-palatinate. aleph [2] is used as cataloging system, a migration to alma [3] is underway. cologne-based libraries and the library centre of rhineland-palatinate (lbz) in cooperation with the hbz started an open data initiative in 2010 being the first to publish their catalog data under an open license. by now, the whole hbz union catalogue is published under cc0 public domain dedication in different ways and formats: as linked open usable data (loud) via the user interface and api at https://lobid.org/resources and as full dumps both in linked open data format and in marc21/xml format with mab2 [4] field numbering. from the fall of 2017 to the beginning of 2020 we carried out a project to upgrade spatial subject indexing in nwbib from uncontrolled strings to controlled values from a spatial classification that is created from wikidata [5]. this article gives an overview over what we have achieved and how. nwbib as part of the hbz union catalogue before diving deeper into the actual project, a short explanation on how the technical and editorial work is divided between different people and institutions. nwbib editors are working at the university and state libraries düsseldorf and münster while the whole underlying technical infrastructure (from cataloging to the web presence of nwbib) is managed by hbz which mostly means: the open infrastructure team. [6] nwbib editors use the aleph union catalog for cataloging, which means that nwbib records are also part of lobid-resources, a lod-based web api for the union catalogue, at its core consisting of json-ld indexed in elasticsearch [7]. within lobid-resources, all nwbib titles – see for example https://lobid.org/resources/ht019035846.json – are marked like this: { "incollection" : [ { "id" : "http://lobid.org/resources/ht014176012#!", "type" : [ "collection" ], "label" : "nordrhein-westfälische bibliographie (nwbib)" } ] } this information enables to easily query nwbib data via lobid-resources by attaching incollection.id:"http://lobid.org/resources/ht014176012#!" to a lobid-resources query. this filter is utilized to offer the web interface for nwbib at https://nwbib.de which is, at its core, a web application built on the lobid-resources api. spatial subject indexing without controlled values for years, subject indexing in nwbib has been done mainly by adding to a record the text strings of locations or regions a resource is about. it looked like this in the source data which is marcxml with mab2 field numbering (in a marcxml representation, the information would be in field 084): <datafield tag="700" ind1="n" ind2="1"> <subfield code="a">99</subfield> <subfield code="b">duisburg</subfield> </datafield> <datafield tag="700" ind1="n" ind2="1"> <subfield code="a">99</subfield> <subfield code="b">essen</subfield> </datafield> the strings “duisburg” and “essen” refer to the cities in nrw with this name. around three quarters of nwbib records (~300,000 records) contained one or more of these place name strings. in 2017, we had 8,800 distinct strings that denoted locations, mostly referring to administrative areas but also to monasteries, principalities, natural regions etc. enriching strings with geo coordinates for years (2014-2019), we have been using wikidata to enrich nwbib data with geo coordinates by matching those place strings to wikidata items and pulling the geo coordinates from there. we had chosen a rather naive matching approach which resulted in poor or no matches for some resources. however, it was good enough to enable map-based filtering of results at nwbib.de. besides the sub-optimal matching there were other drawbacks with this approach. of course the well-known problem occured which everybody encounters after some time when using strings instead of controlled values: you get lots of different strings for the same place because of different recording practices, omissions or typos. as said, in 2017 nwbib included around 8,800 distinct spatial strings which roughly referred to only 4,500 different places. taking a look at a 2017 list containing all the distinct place name strings found in nwbib including an occurrence count, you find for example five strings referring to wiesdorf, a part of leverkusen: – “wiesdorf” – “wiesdorf <niederrhein>” – “wiesdorf, niederrhein” – “leverkusen-wiesdorf” – “leverkusenwiesdorf (niederrhein)” another drawback of the string-based approach was that we could not provide users with a hierarchical overview of all the places. to address these drawbacks, we started thinking about using controlled values from a spatial classification. doing it the right way using controlled values you do not have to convince librarians that the best way to do spatial subject indexing is by using a classification rather than textual strings. soon we all agreed to go this way, the main goals of the approach being: no multiple values referring to the same place a hierarchical view of all places by nrw’s administrative structure: government region, rural districts, urban and rural municipalities as well as city district. here is a mockup of the envisaged classification nwbib editors created in 2017: figure 1. a mockup of a spatial classification created with a spreadsheet requirements for reuse of spatial authority data because of limited resources, nwbib editors would not be able to create a classification from scratch and maintain it all by themselves. thus, we we were looking for existing authority data to be used for our purpose. we identified the following requirements towards the authority data to be reused for the spatial classification: coverage: as many places as possible that were used in spatial tagging must be covered. this included some historical places. hierarchy: hierarchical relations between places must be included. extensibility: nwbib editors must be able to add new places. furthermore, there should be one common entry for a place before and after incorporation into another geographic entity (e.g. parts of a town that had once been standalone administrative entities). why wikidata rather than gnd? as the discussion happened in the german university library context, nwbib editors naturally tended to use the integrated authority file (gnd) [8] which is the main authority file in german-speaking countries, being used and maintained by hundreds of institutions. as nwbib titles already had been indexed with gnd subjects for some years (including spatial subjects), gnd looked like the obvious candidate. however, gnd did not cover most of the requirements. only few hierarchical relations then existed in the data, and with the changes during the move to resource description and access (rda) [9], more and more entries had been split into separate entries for an entity before and after its incorporation into a administrative superior entity, see e.g. these two gnd entries for wiesdorf: 4108828-1 (independent city between 1921 and 1930) & 4099576-8 (part of leverkusen since 1930). the next candidate we looked at was wikidata as we had some experiences with it implementing the above mentioned geodata enrichment. wikidata already had good coverage of place entries, geo coordinates and hierarchical information. as with the gnd, wikidata comes with a technical infrastructure for maintaining the authority data, the difference with wikidata being that the editing community encompasses virtually anybody and not only catalogers. the implications are twofold: the bigger the community, the easier it is to keep the data up to date which means less work for nwbib editors. with anybody being able to edit the data, nwbib editors justifiably worried about unwanted changes and vandalism. furthermore, while the free infrastructure of wikidata is great to start working on a project, it might lead to problems in the long run if you solely rely on wikimedia to keep this infrastructure running. we guaranteed nwbib editors that we’d mitigate 2.) and would also mitigate the lack of control over the infrastructure by creating a solution that adds some kind of buffer between wikidata and the nwbib classification so that we could identify and fix unwanted changes in wikidata before deploying them in nwbib. implementation as this project had to be carried out alongside other projects and in parallel to different maintenance duties by a team of three persons working part-time, the whole duration of the project covered more than two years. this fitted quite well with the constant need for review and feedback cycles which sometimes could take some time due to nwbib editors residing in two organizations different from the hbz. match strings as said above, we already had been matching place strings with wikidata items since 2014 in order to enrich nwbib data with geo coordinates. we had learned early on that using the wikidata api is not sufficient for matching strings of places with wikidata items. one reason being that different levels of administrative areas do have very similar names which led to systematic errors. so we created a custom elasticsearch index from the query which was used in the matching process. as there was no single property for filtering all the places in north rhine-westphalia out of wikidata and as we only wanted to get out those types of items we needed to match the strings with, we had to work with a sparql query that would be improved over several iterations, see the evolution of this sparql query in https://git.io/jovs9 & https://git.io/jovs7. for an optimal matching result, we indexed the german name and alternative names (label, aliases) along with the geo coordinates (geo) for the enrichment. as place strings in nwbib data often contained the name of the superior administrative body for disambiguation, we also added this name to the index (locatedin). here is a full example of the information from wikidata we loaded into elasticsearch: { "spatial":{ "id":"http://www.wikidata.org/entity/q897382", "label":"ehrenfeld", "geo":{ "lat":50.9464, "lon":6.91833 }, "type":[ "http://www.wikidata.org/entity/q15632166" ] }, "aliases":[ { "language":"de", "value":"köln/ehrenfeld" }, { "language":"de", "value":"köln-ehrenfeld" } ], "locatedin":{ "language":"de", "value":"köln-ehrenfeld" } } it then took quite some manual work in wikidata (adding items and/or alternative names) as well as adjustment of field boosting in elasticsearch (see https://git.io/j3lfi) to reach a successful matching for around 99% of all nwbib records, which covered approximately 92% of all place strings we had in the data. to get to a 100% coverage, nwbib editors adjusted catalog records and made more than 6000 manual edits to wikidata adding aliases and type information or creating new entries. update nwbib data having reached the milestone of a 100% matching rate for all the place strings, we could move on to the next step: updating nwbib data with information from wikidata. we did this by including the information in a spatial json object (the json key being mapped to the rdf property http://purl.org/dc/terms/spatial). to avoid being susceptible to vandalism or other unwanted wikidata edits, we refrained from directly using wikidata uris. instead, we minted custom uris in the nwbib.de namespace and used the simple knowledge organization system (skos) [10], a world wide web consortium (w3c) recommendation for publishing classifications, thesauri or other controlled vocabularies as linked data using the resource description framework (rdf) [11]. in a subsequent step, we’d create a skos classification from these skos concept uris (see below). to link the skos concepts for those places to the corresponding wikidata entities, the foaf:focus (http://xmlns.com/foaf/0.1/focus) property is used. [12] besides the uri (id), the type information and geodata from wikidata is also embedded in the focus object, see for example the spatial object from the record ht017710656: { "spatial":[ { "focus":{ "id":"http://www.wikidata.org/entity/q365", "geo":{ "lat":50.942222222222, "lon":6.9577777777778 }, "type":[ "http://www.wikidata.org/entity/q22865", "http://www.wikidata.org/entity/q707813", "http://www.wikidata.org/entity/q200250", "http://www.wikidata.org/entity/q2202509", "http://www.wikidata.org/entity/q42744322", "http://www.wikidata.org/entity/q1549591" ] }, "id":"https://nwbib.de/spatial#q365", "type":[ "concept" ], "label":"köln", "notation":"05315000", "source":{ "id":"https://nwbib.de/spatial", "label":"raumsystematik der nordrhein-westfälischen bibliographie" } } ] } with this setup, it becomes possible to query lobid for nwbib resources based on a place’s q-identifier from wikidata, for example: spatial.focus.id:"http://www.wikidata.org/entity/q365" having reached a state where controlled values for all the places are in nwbib/lobid data, we were still lacking a hierarchical overview of the places to be browsed by users. to enable this, we created a skos classification with broader/narrower relations. create & present skos classification as nwbib editors did not want nwbib to directly rely on wikidata’s infrastructure and data, we had early on decided to represent the classification in an intermediate skos file that we control. the nwbib web application would then rely on this file and not directly on wikidata. as an additional requirement, it became clear that we would not be able to maintain 100% of the classification in wikidata for several reasons: some parts of the classification were quite nwbib-specific and it wasn’t feasible to include them in wikidata. also, we could not claim authority over the preferred names used in wikidata so that we had to think about a solution when a label in wikidata would differ from the label we wanted to be used in nwbib. here is the setup we came up with: a skos turtle file covering the core spatial nwbib classification with (currently around 40) concepts not covered in wikidata or where we need to use another label than wikidata: nwbib-spatial-conf.ttl the complete classification as a skos file created from 1.) and wikidata: nwbib-spatial.ttl the html classification that is rendered from the classisication skos file can be viewed and browsed at https://nwbib.de/spatial (where you can also download the skos version under https://nwbib.de/spatial.ttl). here is a screenshot showing a part of the html classification: figure 2. a part of the nwbib spatial classification showing different hierarchical levels you can directly link to each element in the html classification by using the concept’s hash uri which is based on its wikidata id – e.g. https://nwbib.de/spatial#q365. displayed in parenthesis behind each classification entry is the number of bibliographic resources in nwbib that refer to this place. this number is linked to a result view listing these resources. update wikidata with links to nwbib with the concept uris resolving to their classification entry, the next step could be implemented: adding links from wikidata to nwbib. prerequisite is a wikidata property that enables adding those links to wikidata items. we proposed such a property in the beginning of june 2019, see https://www.wikidata.org/wiki/wikidata:property_proposal/nwbib_id. after getting a few affirmative votes, the wikidata property nwbib id (p6814) was created quickly one or two days later. we then batch loaded the nwbib ids with quickstatements [13], see https://github.com/hbz/nwbib/issues/469 for details. after this step, we could get all relevant wikidata entries by querying for all items that have a p6814 statement. until then, we were getting information about hierarchy (superordinate administrative entity) from p131 (located in the administrative territorial entity statements. as for some places multiple superordinate administrative entities were listed, we had problems to identify the one we would use for creating our classification’s hierarchy. that’s where the wikidata property p4900 (broader concept) came into play. we used this property to batch load “broader” links to other wikidata entries with a qualifier to the p6814 (nwbib id) statements, see the corresponding issue for implementation details. the result, e.g. for cologne (q365), looks like this in wikidata: figure 3. nwbib id entry in wikidata example with p4900 (broader concept) qualifier the wikidata entries which are used in nwbib can thus be identified by usage of the nwbib id property p6814, e.g. by using this sparql query on the wikidata query endpoint at https://query.wikidata.org: # get all places used in the nwbib spatial classification select ?item ?itemlabel where { ?item wdt:p6814 ?nwbibid. service wikibase:label { bd:serviceparam wikibase:language "[auto_language],en". } } support aleph cataloging with concept uris by now, we had also replaced all spatial strings in the underlying aleph system with skos concept uris. to enable subsequent cataloging with those concept uris, we somehow had to establish a process for cataloguers to easily add the controlled values into the bibliographic records. for this, we added a hidden copy button behind every classification entry to get the needed aleph format with one click. one has to hover over the space behind a classification entry to make the button and an explaining tooltip visible, then click the button, select the correct field in aleph and paste the entry there. figure 4. hidden button with tooltip in the classification page for copying snippets into aleph for example, a click on the button as shown in the screenshot adds the following string to the clipboard which can directly be pasted into the right field in the aleph cataloging client: köln$$0https://nwbib.de/spatial#q365 the resulting catalog data looks like this (in the marc21/xml export format that uses mab2 field numbering): <datafield tag="700" ind1="n" ind2="1"> <subfield code="a">köln</subfield> <subfield code="0">https://nwbib.de/spatial#q365</subfield> </datafield> monitoring and fixing synchronization issues setting up the new infrastructure for the spatial classification, we noticed two problems that might and did occur with regard to the synchronization between nwbib data and wikidata: spatial concepts used in nwbib and derived from wikidata might not have a p6814 (nwbib id) statement in wikidata – either because they were newly added to nwbib or someone removed the statement from wikidata. wikidata entries with a p6814 (nwbib id) statement miss an equivalent concept in nwbib. to be notified about a potential asynchronicity, we set up a monitoring process that sends out an email whenever one of these cases occurs. usually, a reported error can easily be resolved by adding/removing a p6814 (nwbib id) from wikidata. establish editorial process for maintaining the classification having set up this whole infrastructure we had to think about an editorial process for maintaining the classification. as noted, an up-to-date skos classification is created from: the manually maintained skos config file nwbib-spatial-conf.ttl which includes a) top-level concepts, b) hierarchical relationships that can’t be recorded in wikidata with p4900 (broader concept) and c) concepts whose desired display label differs from the label in wikidata, all wikidata items with p6814 nwbib id statements including – if existent – the qualifier p4900 (broader concept). in general, statements from the config file (1.) overwrite those from wikidata (2.). [14] having this technical update process in place, with a few iterations we established the following editorial process: nwbib editor decides to change an entry or to include a new place by adding an nwbib id (p6814) statement. nwbib editor triggers a new build of the classification in the test system by clicking a button “update from wikidata” and checks whether the desired change is achieved. editor can now already use the test classification for copying the entry into aleph as described above. if everything is fine on test, editor notifies the lobid team via email and requests a rebuild of the classification in production. lobid team updates the classification in the production system and notifies editors about possible undesired changes coming in from wikidata. assessment summarizing the result of the project, the nwbib’s spatial classification – which can be browsed at https://nwbib.de/spatial – is now comprising controlled entries for around 4,500 places or geographic areas. the underlying structured data is stored as an rdf/turtle file using the simple knowledge organization system (skos): https://nwbib.de/spatial.ttl. this skos file in turn is for the utmost part derived from wikidata. an editorial process for updating the classification has been set up. both the nwbib editors who are responsible for cataloging and the development team are very pleased with the results of the project and the achieved possibilities of maintaining and using a larger spatial classification. here are some of the benefits: common sources of errors like typos or missing parentheses with qualifiers are now impossible. with regard to the cataloging workflow, we got the feedback that the new method of browsing and filtering the classification and clicking the hidden button to copy the field entry into aleph would be much better and easier than the previous method of adding a textual stirng by hand. especially, new staff or staff that is not so familiar with nwbib is very happy about the change. the previous process of deriving geo coordinates from wikidata by matching strings resulted in some errors. directly using wikidata-derived controlled values for spatial indexing eliminates this source of error. the editorial process along with the monitoring tools results in a small and easily manageable maintenance burden for the team maintaining the technical infrastructure. nwbib users benefit from the hierarchical, browsable and clickable overview of the whole spatial classification. finally, the connections to wikidata enable us to utilize even more information from wikidata in the future, e.g. by setting up a profile page for each place in the classification with a small description, a picture and the coat of arms. acknowledgements thanks to the nwbib editors holger flachmann, stefanie von gumpert-hohmann, elisabeth lakomy, irmgard niemann, ute pflughaupt, doris ritter-wiegand, and cordula tetzlaff for being open to this wikidata-based approach and for the sincere, constructive and friendly communication which was crucial for the success of this project. thanks to the open infrastructure team, especially fabian steeg and pascal christoph, for the good team work in general and especially in this project. finally, thanks to magnus sälgö, péter király and osma suominen for showing their interest in the project during swib20 (see the mattermost thread archived by me) which led me to document the project in this code4lib article. further resources slides from wikidatacon 2019: https://slides.lobid.org/nwbib-wikidatacon/ github project wiki (german): https://github.com/hbz/nwbib/wiki project issues (mostly in german): https://github.com/hbz/nwbib/issues?q=is%3aissue+project%3ahbz%2f3 notes [1] for information on zdb and how to query it, see pohl, adrian (2018): accessing zdb via lobid-resources. url: https://web.archive.org/web/20210125181012/https://blog.lobid.org/2018/09/04/zdb.html [2] https://librarytechnology.org/product/aleph/ [3] https://librarytechnology.org/product/alma [4] “maschinelles austauschformat für bibliotheken”, a data format similar to marc21 used as exchange format in the german-speaking library world until 2013. see https://de.wikipedia.org/wiki/maschinelles_austauschformat_f%c3%bcr_bibliotheken for more background. [5] https://www.wikidata.org [6] https://lobid.org/team-en [7] for details, see the blog posts about lobid-resources at https://blog.lobid.org/tags/lobid-resources [8] https://gnd.network [9] http://www.rda-rsc.org/content/about-rda [10] https://www.w3.org/2004/02/skos/ [11] https://www.w3.org/rdf/ [12] for a detailled account on foaf:focus and related properties see johnston, pete (2011): things & their conceptualisations: skos, foaf:focus & modelling choices. url: https://efoundations.typepad.com/efoundations/2011/09/things-their-conceptualisations-skos-foaffocus-modelling-choices.html [13] https://www.wikidata.org/wiki/help:quickstatements [14] as a sidenote: in the beginning, we hoped to directly extract skos data from wikidata using a sparql construct query. unfortunately, wikidata does not support construct queries for such a large amount of data, see the respective issue. about the author adrian pohl (pohl@hbz-nrw.de) has been working at the north rhine-westphalian library service centre (hbz) in cologne, germany, since 2008. he is leading the open infrastructure team whose work focuses on open standards, tools and processes for the publication of structured data on the web. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – 3d adaptive virtual exhibit for the university of denver digital collections mission editorial committee process and structure code4lib issue 29, 2015-07-15 3d adaptive virtual exhibit for the university of denver digital collections while the gaming industry has taken the world by storm with its three-dimensional (3d) user interfaces, current digital collection exhibits presented by museums, historical societies, and libraries are still limited to a two-dimensional (2d) interface display. why can’t digital collections take advantage of this 3d interface advancement? the prototype discussed in this paper presents to the visitor a 3d virtual exhibit containing a set of digital objects from the university of denver libraries’ digital image collections, giving visitors an immersive experience when viewing the collections. in particular, the interface is adaptive to the visitor’s browsing behaviors and alters the selection and display of the objects throughout the exhibit to encourage serendipitous discovery. social media features were also integrated to allow visitors to share items of interest and to create a sense of virtual community. by shea-tinn yeh, jeff rynhart, thomas dressler and fernando reyes introduction digital collection exhibits have been promoted by museums, historical societies, and libraries around the world to provide public access to cultural heritage, hidden collections, or artefacts. these exhibits are presented with hyperlinks on web pages within a two dimensional (2d) interface. items of interest are generally sought using a query search mechanism or by browsing in a sequential fashion. however, toms (2000) experimentally validated the concept that people find information through “accidental, incidental or serendipitous discoveries.” she urged the development of creative approaches to trigger serendipitous retrieval that complements searching and browsing for information acquisition in a digital library. in this work of adaptive virtual exhibit (ave), the authors proposed a proof-of-concept project to develop a three-dimensional (3d) exhibit in which visitors are guided by an adaptive interface to view selected 2d digital images from the university of denver libraries’ collections. we have also integrated social media features to promote a sense of virtual community. the proposal in the amount of $2,898.45 was awarded in march 2014 by the university of denver’s faculty research fund (frf). the concept of the ave is attributed to the research by bonis et al. (2013) in which the authors presented a framework for “designing and implementing multiuser virtual exhibitions that adapt visitor’s preferences and goals and foster the emergence of communities with common interests.” two primary architecture elements were proposed: (1) provide a 3d space for the presentation of objects in a thematic museum area, and (2) update presentation content to reflect visitors’ interests obtained from both initial assumption and the sequence of events between the visitors and the objects with which they interact. bonis et al. validated their framework by using vrml, java, java3d, and tcp sockets as well as designing a semantic graph with calculations to accommodate adaptivity. our prototype successfully advanced bonis’s framework by utilizing more readily available and agile technologies including a simple 3d javascript library, api, and the neo4j graphing database. to date, we have found no extant model that applies this innovation with any university’s digital collections. this article describes the systems design architecture and programming techniques in detail. we hope to generate interest in the library application development community to form collaborative efforts for further ave development. systems design architecture four components are required in the ave architecture: authentication, 3d virtual exhibit client, graph database for adaptivity, and the display of metadata for each respective object. after logging into the exhibit with a valid gmail account (figure 1), the visitor is presented with a preference form to select a topic of interest (figure 2). once selected, the exhibit is populated with items related to this topic. there are four rooms in the 3d virtual exhibit designed as a museum space simulation; each room contains three object display slabs (figure 3a & 3b). visitors can move from room to room with the directional keys on the keyboard, viewing the objects within (figure 3c). they may interact with any object by clicking to open a magnified image, viewing its descriptive metadata, or sharing the object with others on their google+ profile. these interactions are sent to the database real time, which in turn, increases the attribute of “user interest” to generate adaptivity. the content of the exhibits in the 3d space are subsequently updated based on the adaptivity. figure 4 illustrates this project’s system design architecture. the metadata contains creator, description, topic(s), and identifier as elements and are stored in the csv file format. the data is ingested into the graph database with a php script that utilizes fgetcsv() function. the script is stored on bitbucket, a source code management and collaboration solution in the cloud. figure 1. visitor login screen (enlarge) figure 2. visitor topic selection form (enlarge) figure 3a. virtual exhibit room (enlarge) figure 3b. virtual exhibit room (enlarge) figure 3c. descriptive item view (enlarge) figure 4. systems design architecture programming techniques authentication user authentication is accomplished through the google auth authentication service. when the visitor’s gmail credentials are posted to the service and validated, google auth returns the gmail address along with a temporary access token. the address is stored in the database as a userid serving as the persistent identifier while the token is stored as a session id, used to verify a current active session and to share objects of interest. 3d virtual exhibit client the 3d interface was created with the three.js javascript 3d library/api (http://threejs.org/) for three reasons. first, it creates three dimensional representations of data without needing to install browser plugins. one may argue that browser plugins are beneficial and extensible. on the other hand, designing systems to integrate with third party software is associated with higher maintainability and complexity. secondly, three.js allows for the definition, placement, and rendering of 3d shapes with minimal mathematical computation. thirdly, it assigns texture images to shapes. the library also contains a camera property which renders the view based on the camera’s current coordinates and the angle of the view. this camera vector can be updated during runtime to move the visitor’s vantage point, to allow navigation and interaction with the objects within the 3d space. the application communicates with the server via ajax calls by transmitting json-encoded data, and utilizes a rest api to perform specific actions. three main programming tasks are required to emulate a 3d exhibit space and implement the front-end application: (1) 3d exhibit space map construction, (2) digital object display and interaction reporting, and (3) visitor movement. as shown in figure 5, the exhibit structure is a single square subdivided into four rooms by a cross, with doorways connecting each room to its adjacent rooms. these rooms are created in the browser canvas with walls comprised of 3d cubes. rooms are designed with various wallpapers so that the visitor is aware of room changes and re-entry of visited areas. figure 5. exhibit room structure a. 3d exhibit space map construction the floor layout was implemented with a 2d array, which allows the rendering of all 3d cubes via a looped iteration of the array elements. the draw function contains a doubly nested for loop that iterates over each element, drawing an object based on the stored numeric value. this value is a 2 digit integer that describes what should be drawn at the corresponding location. the first digit defines the object to be drawn, and the second digit represents an attribute such as orientation or texture. for instance, a value of 13 draws a wall block at the corresponding location, using texture number 3; and a value of 92 places exhibit tables along a wall in direction number 2 (east). a value of 0 in any array position draws nothing and becomes empty space. after the entire array is iterated over, the 2d wall layout is complete, and all objects that appear in the map are inserted as shown in figure 6. figure 6. 2d array b. digital object display and interaction reporting all objects that appear in the exhibit space besides the walls are interactive. this means that they can be clicked by the visitor to initiate an event, or trigger an event as the visitor comes within proximity of the object. locations of interactive objects must be registered or stored with the map object during runtime. these locations are based on a virtual coordinate system superimposed on a map array as illustrated above. for interaction reporting, three.js contains a useful function that projects a ray reaching into the 3d canvas space directly outward from the point of observation; this vector would appear to the visitor as a dot in the center of the screen. any object behind the visitor’s click is intersected and identified to the application based on the intersection point and the object’s stored coordinates, which are useful for attaching click events to an object or determining proximity to the camera. if the visitor attempts to walk through a wall, the proximity is detected, the current motion cancelled, and the wall appears to be solid. visitor interaction with items is essential to the adaptivity of the exhibit explained in the upcoming adaptivity techniques section . c. visitor movement the first person vantage point is created with the camera that renders 3d objects using the browser window as a first person perspective. the camera’s location and heading can be updated during runtime, and the objects are drawn to the screen from any resulting viewpoint when the three.js render function is called. to create the illusion of movement of the vantage point within the 3d space, an animation loop was created to execute this function continuously. this loop is infinite, terminating only when the visitor exits the exhibit or closes the browser window. the position of the camera is altered when the visitor presses the directional keys, and this update happens between the render “frames.” movement is created by calculating a vector based on the current camera heading and an established unit of movement, then adding this vector to the current camera position. this is accomplished with a simple and efficient algorithm which computes an x component and a y component for the new vector. for example, if one movement unit is 10 pixels, and the y axis points to 0 degrees in the positive direction, the visitor (at heading 0) moving one unit forward produces components of y:10 and x:0. this advances the camera 10 pixels to the positive y direction, and creates the illusion that the visitor has taken one step forward in the direction of 0 degrees. a step in the heading of 45 degrees produces the components y:5 and x:5, a step along heading 225 will produce x:-5, y:-5 and so forth (figure 7). figure 7. movement components adaptivity techniques according to kobsa et al. (2001), the adaptation process involves three tasks: acquiring information about users’ characteristics and behavior, representing data in a system that draws assumptions about user preferences, and producing personalized content. for this prototype, we acquire the visitors’ characteristics by having them select the initial topic of interest. we track the visitors’ browsing behavior, and we represent data in terms of their relationships. to store the exhibit data and to implement an adaptive design, the popular graph database neo4j is used instead of a relational database. neo4j, sponsored by neo technology, is an open source nosql database implemented in java and scala (what is a graph … [updated 2015]). graph databases are supportive of “relationships” as entities eliminating the need for foreign-keys relationships. each node in the graph database contains the relationship-records that represent its relationships to other nodes. therefore the expensive join operation is not necessary in a graph database. in our prototype, if a creator is represented as a node, the creator can have relationships to other data with different entities and attributes. with such a relationship design, this graph database is particularly suited for exploring an adaptive technique. two levels of nodes are designed in the database: categories and objects. each object is a child of at least one category node and the category describes some elements of the object. siblings are nodes which share at least one parent category node. in this design, objects can be related to each other by their common categories, and connections between objects can be determined by tracing their relationships to shared category nodes. the visitor is also represented by a node which is connected to individual category and object nodes with a special type of relationship, one containing an ‘interest’ attribute. the database is adapted to a specific visitor by updating these interest values each time they view an object. after continued use of the exhibit, each visitor will develop a unique set of interest relationships to the objects they have viewed, as well as to objects that are adapted to their interest via common categories. figure 8 illustrates the relationships between the category and object. figure 8. representation of category, object, and their relationships to create adaptivity, an algorithm was developed to analyze a visitor’s unique interest values. the interest amount increases in three situations: (1) when the visitor comes in close proximity to a specific object, (2) when the visitor opens an object’s metadata view dialog for more than 10 seconds (the average time needed to read through item descriptions of an object), and (3) when the visitor shares the object to their google+ profile. after the object and time data are posted to the server via an ‘interest increase’ request, the interest value of the viewed object is updated in the database, along with the interest values of all objects related to it. based on these current values, the database utilizes algorithms implemented with the phalcon php framework to gather a set of objects to return to the client, which then refreshes the objects displayed in the 3d exhibit space. the assumption is that the visitor is likely to have interest in other items that share attributes with the item of current interest. for example, a visitor who views various portraits of chancellor ritchie may be delighted to discover the carillon in the bell tower in the ritchie center on the university of denver campus. figure 9 illustrates the interest value design in the database and figure 10 describes the adaptivity techniques. figure 9. interest value design in the neo4j graph database figure 10. adaptivity technique conclusions due to limited availability of funding, this study has a few limitations. first of all, we prototyped the concept rapidly and experimented with one direction of presenting adaptivity, i.e. visitor’s interaction with the objects. there are other possible ways to promote serendipity. for example, toms (2000) suggested the exploring of “role of chance” or “blind luck” as well as “reasoning by analogy” which are complex, yet intriguing concepts. secondly, we required authentication with a gmail account to enable social media features. this can be redesigned to utilize other methods without authentication requirements. thirdly, we did not factor in privacy concerns nor implement responsive design as part of the development scopes. fourth, the amount of fitting objects and their metadata obtained from the targeted collections was small, thus difficult to guarantee relatedness. this limitation can be overcome with enriched and quantified metadata. finally, the prototype is not currently available outside of the university firewall. when we identify a team of developers to further implement ave, we will propose a university sponsored development environment for external collaboration and usability testing. 3d interfaces are a paradigm shift away from 2d representations that are based on icons, menus, and pointers. an exhibit presented in a 3d environment simulates the visitor’s experience with a feeling of being present within an exhibit space. adaptive computing applied in interface design seeks to personalize the behavior of a system to reflect the user’s particular search goals, attitudes, and interests. both of these technological advances have been widely adopted in the video gaming industry; however, this is the first work that explored these techniques with a university’s digital assets. accordingly, there is also a lack of literature about these technologies in the library and information science discipline. the authors hope to generate interest in the library application development community and form an alliance to make the ave available for general deployment. what we envision is to not only address the current limitations but also build a 3d adaptive interface for 3d objects. acknowledgement this project was funded by a $2,898.45 faculty research fund (frf) from the office of research and sponsored programs at the university of denver. the authors would like to thank andrew darby for his valuable comments and suggestions to improve the quality of the paper. references bonis, b., vosinakis, s., andreou, i., panayiotopoulos, t. (2013). adaptive virtual exhibitions. journal of library & information technology 33(3): 183-198. kobsa, a., koenemann, j., phol, w. (2001). personalized hypermedia presentation techniques for improving online customer relationships. the knowledge engineering review 16(2): 111-155. toms, e. (2000). serendipitous information retrieval. in proceedings of the first delos network of excellence workshop: information seeking, searching and querying in digital libraries, zurich, 11-12 december, 2000. available from: http://www.ercim.eu/publication/ws-proceedings/delnoe01/3_toms.pdf what is a graph database? [internet]. [updated 2015]. available from: http://neo4j.com/developer/graph-database/ about the authors shea-tinn yeh library digital infrastructure and technology coordinator, university of denver libraries email: sheila.yeh@du.edu jeff rynhart systems programmer ii, university of denver libraries email: jeff.rynhart@du.edu thomas dressler programmer, webroot email: thomas.dressler1@gmail.com fernando reyes senior systems analyst, university of denver libraries email: fernando.reyes@du.edu subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – wordpress as a content management system for a library web site: how to create a dynamically generated subject guide mission editorial committee process and structure code4lib issue 3, 2008-06-23 wordpress as a content management system for a library web site: how to create a dynamically generated subject guide this article explains a method of generating dynamic subject guides through the wordpress content management system. this method includes the use of the exec-php wordpress plugin and additional php code to create a new category-based loop within the preexisting wordpress loop. example code and screenshots are provided. by joshua dodson introduction i am working with the lincoln memorial university’s carnegie-vincent library on a redesign of their web site, based on the wordpress (wp) framework. (the new site is slated to launch in the summer of 2008.) the director asked me to look into libdata, a library oriented, web-based application that offers an authoring environment tailored to creating resource guides and course pages. libdata organizes library database listings, bibliographic records, and external links through its server back end. the director had used libdata at the last university library she worked at to organize subject guides for individual majors. they used it in such a way that every english major, for example, would go to the english subject guide and find all of the resources related to their subject area on one page, as arranged for them by the librarians. the most relevant journals, books, and relevant web sites they may need for research or further studies are contained on one page for their convenience. theoretically this one digital hub is the only place the english majors would need to to begin locating appropriate resources for required assignments. this eases the daunting task of research.since i am primarily a wp theme designer and am already in the process of creating the new library site to run on wp, i decided to centralize the entire process and do all of this in wp. i have had experience with the ease of use that wp allows, as well as its strength and capability of handing difficult tasks, so using wp to handle both the general web site and the subject guides seemed the logical choice. i looked into casey bisson’s work on scriblio, the wp-based opac, and knew that it was possible to transform the functionality of wp into a robust solution for library needs. in this spirit i set out to use wp for the subject guide system. wordpress i needed wp to store database links and descriptions with metadata so that i could pull every database entry that meets given criteria and display results in a designated area within a page. as an example, if academic search premier database is listed as a multidisciplinary database, every page that contains a call for multidisciplinary databases would display the entry for academic search premier as one of the databases. to accomplish this i entered each database as a new post within wp. the database name is entered as the post title. the database link and description is entered in the post body. i created a “connect” image that users can click on to connect to the database. i put every database entry into at least two subject categories. each database receives the category of “database” (for the a-z listing) as well as the specific subject category of whatever page it needs to appear on. if jstor needs to be on the english subject guide, it will receive the “englishdb” category. note that i added db to the end of the subject guide categories since the web site is used as a general content management system (cms) with other pages and areas. i did not want a separate entry for an english-related topic to be misinterpreted and included in a subject guide when it is not a database. thus, db at the end of the category name designates the use of that category within a subject guide as a database entry.an additional requirement for creating the subject guide was to have a method of pulling each entry within a category and displaying it on a given page. i could have used the regular category link to display each englishdb page, each historydb page, and so on, but did not want the “db” displayed at the end of the category name; those characters were meant as labels only, not for user display. i also wanted to pull multiple categories into a single page. for instance, every page needed to have the “multidisciplinary categorydb” and the “reference toolsdb” category within it. i could have hard-coded the individual php files to do this, updating them all when necessary. however, putting such categorizations in the code would make it more difficult for a non-technical guide editor to update the site. figure 1. sample wordpress entry for jstor. a tool i found invaluable to simplifying this task was the amazing exec-php plugin, a free wp plugin requiring minimal configuration that will execute php entered into a wp post or page. after activating that plugin, i was ready to start working on some php within the wp framework. at this point i also created a new wp user with the username “phpuser” that would be the “author” of the php-based pages within wordpress. since phpuser is an admin account, only a couple people have access to it. this minimizes chances of changing the code, which might result in the web site malfunctioning. wp already uses a php loop to call posts under given criteria for its general structure. i needed to do this multiple times within the pre-established loop since we are working inside of the wp framework. extending wordpress a helpful resource that i thought contained the solution was perishable press’s triple loop. in an early effort the triple loop worked just as i needed it to, but i had to create individual php pages (outside of the wp dashboard) for each subject guide. i further needed it to work within the wp dashboard and use the category name, rather than number, to specify which category should be displayed. there is still a lot to learn from the triple loop, but i needed something else.i ran across an article at devlounge (“customizing wordpress: advanced“) on customizing wp that made everything fit together perfectly. though the article was along the lines of what perishable press offered, it took a different approach. it utilized the category name, rather than number, and worked within the wp dashboard with the help of exec-php. using the following code within the wp dashboard “write page,” i was able to dynamically pull every post within the englishdb category for display on the page. <?php $my_query = new wp_query('category_name=englishdb&amp;orderby=title&amp;order=asc&amp;showposts=100');while ($my_query->have_posts()) : $my_query->the_post(); $do_not_duplicate=$post->id;?><?php the_title();?><?php the_content();?><?php endwhile; ?> the code requests 100 posts within the database that are categorized as “englishdb” and returns them sorted by title in ascending alpha-numeric order.this method of retrieving posts enables the library staff to specify a category name rather than a category id. though the category id has the benefit of always staying the same if the category name changes, it is also more confusing since the id’s numeric value is not as semantically meaningful. i believe that using the category name instead of the id makes it easier for everyone to use, now and in the future. figure 2. php queries in the write page wordpress panel. the complete template to bring it all together, the following is an example of a page. to create it i log into the wp dashboard as phpuser, click write, then click page, title it “english subject guide,” and paste the following code into the post body: <div id="wrap"> <h4>english-specific</h4> <?php $my_query = new wp_query('category_name=englishdb&amp;orderby=title&amp;order=asc&amp;showposts=100'); while ($my_query->have_posts()) : $my_query->the_post(); $do_not_duplicate = $post->id; ?> <div> <span> <?php the_title(); ?> </span> <?php the_content(); ?> </div> <?php endwhile; ?> <h4>other useful databases</h4> <?php $my_query = new wp_query('category_name=englishotherdb&amp;orderby=title&amp;order=asc&amp;showposts=100'); while ($my_query->have_posts()) : $my_query->the_post(); $do_not_duplicate = $post->id; ?> <div> <span> <?php the_title(); ?> </span> <?php the_content(); ?> </div> <?php endwhile; ?> <h4>multidisciplinary</h4> <?php $my_query = new wp_query('category_name=multidisciplinarydb&amp;orderby=title&amp;order=asc&amp;showposts=100'); while ($my_query->have_posts()) : $my_query->the_post(); $do_not_duplicate = $post->id; ?> <div> <span> <?php the_title(); ?> </span> <?php the_content(); ?> </div> <?php endwhile; ?> <h4>reference tools</h4> <?php $my_query = new wp_query('category_name=referencetoolsdb&amp;orderby=title&amp;order=asc&amp;showposts=100'); while ($my_query->have_posts()) : $my_query->the_post(); $do_not_duplicate = $post->id; ?> <div> <span> <?php the_title(); ?> </span> <?php the_content(); ?> </div> <?php endwhile; ?> </div> as shown in the above example, the technique can be modified and added upon to pull content from multiple categories (englishotherdb, multidisciplinarydb, referencetoolsdb, etc.). the extra <div> and <span> tags allow for styling with css and additional javascript functionality. this makes for a rather long page, so i added a bit of javascript to only show the titles of the databases and the links. to see their descriptions one has to click on the title.the addition of the dynamically generated subject guides has made the task of updating the web site much easier. the librarians simply have to write a post and give it the relevant category to display it on the related subject page. as with any new method, there is a learning curve, but the usability of wp and the way the process streamlines updates makes it well worth the effort.â the library staff has worked to create the subject guides in preparation of the new web site launch, due later in the summer. we will evaluate the efficacy of the subject guides after the launch and adapt them as student needs arise. adapting the subject guides will be simple, and most likely only the addition and removal of various categories will need to occur. in the future, i would like to make it simpler still for the staff to add additional subject guides, possibly through the use of wp administration options to select categories instead of the minimal use of code that is currently required. all in all, it was not too difficult to create a dynamically generated subject guide for a library web site. this method can, of course, be adapted for other purposes that require a loop within a loop.to view a live demo, visit http://experiment.fragmentist.com/?page_id=44. (as of the date of publication, the library’s site has not been launched; that is anticipated later this summer.) download a free wordpress theme with example code at http://www.fragmentist.com/wordpress-themes/ulysses/. about the author joshua dodson is a web designer living in cumberland gap, tennessee. he is currently working within the lincoln memorial university library systems. he creates custom wordpress themes and is constantly learning new skills. he is available for web design and questions at www.fragmentist.com where he occasionally blogs and publishes. subscribe to comments: for this article | for all articles 6 responses to "wordpress as a content management system for a library web site: how to create a dynamically generated subject guide" please leave a response below: bhushan, 2008-08-04 dear joshua dodson this is very good article for me, anyway there good number of content management around, but wordpress has its own features found by design, 2008-08-14 are you finding the code to work with the newest version of wordpress? i used a verison of this code below and it showed all posts, not just the one category… have_posts()) : $my_query->the_post(); $do_not_duplicate = $post->id; ?> i am using wp2.6. thanks found by design, 2008-08-14 ok, so my last comment did not post all the code, but it is using the $my_query = new wp_query('category_name=multidisciplinarydb method… just not working for me… is there something else that needs to be added? joshua dodson, 2008-09-09 i submitted this a while back, but it must not have gone through. please accept my apologies. found by design, i have had great luck with wp 2.6. to double check, you are using a variant of the following code: <h4>multidisciplinary</h4> <?php $my_query = new wp_query('category_name=your category name here&orderby=title&order=asc&showposts=100'); while ($my_query->have_posts()) : $my_query->the_post(); $do_not_duplicate = $post->id; ?> <?php the_title(); ?> <?php the_content(); ?> <?php endwhile; ?> the library web site i was working on when i wrote this article is now up. the subject guides can be found at the following url: http://library.lmunet.edu/databases jodi schneider, 2008-11-02 karen coombs has also written recently about using wordpress for databases: http://www.librarywebchic.net/wordpress/2008/09/28/creating-database-lists-with-wordpress-link-tool/ mehmet hakan çolpan, 2011-11-05 i also work libray. next time sharing. leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – using cloud services for library it infrastructure mission editorial committee process and structure code4lib issue 9, 2010-03-22 using cloud services for library it infrastructure cloud computing comes in several different forms and this article documents how service, platform, and infrastructure forms of cloud computing have been used to serve library needs. following an overview of these uses the article discusses the experience of one library in migrating it infrastructure to a cloud environment and concludes with a model for assessing cloud computing. by erik mitchell, ph.d. introduction this article examines some of the key issues related to the use of different forms of cloud computing in libraries and discusses the experience of one library in moving to cloud-based infrastructure. cloud computing refers to the abstraction of information technology (it) software and services from the hardware they run on. the national institute of standards and technology (nist) expands this definition by examining specific characteristics (e.g. self-service, resource pooling, and elasticity), management models (e.g. service, platform, or infrastructure focus), and deployment models (e.g. public, private) (nist, 2009). cloud platforms enable organizations to use external expertise and resources to deliver complex services, remove the need for organizations to invest in server infrastructure, and lower the cost for organizations seeking elastic computing resources. libraries have been adopting cloud-based solutions for different services including electronic journal access management, statistics tracking, digital library hosting, and even integrated library system (ils) hosting. this has allowed libraries to make strategic choices about the allocation of resources and to offer better service than would be possible if relying on in-house solutions. while much of the focus on cloud computing in libraries has been on subscription service or platform (e.g. ils hosting) there are cases where libraries need computing resources for requirements that are not provided by service or platform providers. this article looks specifically at the experience of one library in moving its it infrastructure to cloud-based environments. the article seeks to address how well these systems fill library it needs, asks what other elements define the success of the use of cloud-based infrastructure and concludes with a case study discussion of one experience. the state of cloud computing in libraries cloud computing can be divided into three categories: software-as-a-service (saas), platform-as-a-service (paas), and infrastructure-as-a-service (iaas). in a saas environment, organizations use an application via a hosted service. they do not have access to the underlying infrastructure (i.e. network or server elements) and are not responsible for managing the underlying software. a common library example is electronic journal subscription management systems. paas solutions focus on providing a hosted platform on which a specific application can be deployed. this platform is often some provisioned space and computing resources from a hosting company running a pre-configured set of tools. organizations can deploy a locally developed or managed application on the platform but do not manage the underlying server infrastructure. platform management is often done using an application such as cpanel or plesk. iaas environments allow users to provision servers, storage space, and networking components to meet their computing needs. in an iaas environment, the organization is responsible for starting and sizing a server, managing its network access, and ensuring that the core server components (e.g. operating system, web server, firewall) are configured correctly. libraries have quietly been on the forefront of cloud computing technology for a number of years. the use of saas in libraries reaches back into early 2000 with the establishment of companies like serialssolutions (http://serialssolutions.com). much of the work in migrating to electronic journals has focused on an saas platform and recent companies such as libguides (http://www.libguides.com) have shown that libraries are willing to invest in saas solutions. in the iaas arena, amazon elastic computing cloud (ec2) offers it infrastructure for organizations to launch differently sized servers using a variety of operating systems, including several flavors of linux and windows. ec2 provides organizations with essentially unlimited storage using their s3 service, the ability to take snapshots of both data and servers, and the ability to include ec2 servers in an organization’s private network. a full catalog of ec2 features is available on the ec2 website (http://aws.amazon.com/ec2/). wheeler and waggener (2009) use this classification (saas, paas, and iaas) as a launching pad to discuss ways in which they can be used to enable collaboration or ‘sourcing’ between institutions and consortia. marshall breeding (2009) places these three types of services within the context of other infrastructure and hosting options such as co-location (the duplication of specific it resources in multiple places), shared and dedicated hosting (licensing a shared or distinct portion of a server for use), and cloud computing (abstracting the hardware, software, and service layers to provide an extensible computing environment). embedded within these classifications are needs and use arguments, organizational goals, and institutional priorities. for example, wheeler and waggener point out that organizations have different goals for adopting cloud computing and suggest that these goals are embedded in a complex web of cost, needs, priorities and organizational culture. libraries are in a unique position to experiment with cloud computing given their service-oriented mission and need to find appropriate solutions using limited resources. fox observes that the goals of the organization have an impact on their use of cloud solutions (2009). for example, he observes that one of the key pressures that both pushes libraries to cloud solutions and proves to be an impediment to the migration is the availability of it support services. libraries are often supported by external or organization level it services and do not have internal expertise on advanced it management. further, fox observes that libraries may be governed by policies and regulations that dictate how they can use cloud-based solutions. both of these factors make saas and paas approaches appealing, and make iaas approaches difficult to consider. despite this, many libraries have been active in investigating innovative uses of cloud computing (kroski, 2009), including new ways of using infrastructure services. kroski’s article mentions the use of amazon ec2 services by both the dc public library system and ohiolink to provide library it services using iaas techniques. critics of cloud computing observe that the transition to remote and subscription based resources opens up organizations to new legal and operational challenges. critics also ask whether or not the major cloud-computing platforms are open enough to trust (truitt, 2009) and suggest that new government regulations and policies are required to ensure openness and sustainability (nelson, 2009). in addition to these high-level concerns, a number of technical concerns exist such as how to include cloud-based applications within the organizational network, how to back up and archive information located in the cloud, and how to manage services in a decentralized environment. cloud-hosting organizations have taken steps to address some of these concerns. for example, amazon has added features to their core elastic computing cloud (ec2) service, including private connectivity between cloud-servers and an organization’s network (virtual private cloud); elastic block store (ebs) based storage which offers bit-level snapshots and persistence across server instances; and server-monitoring and management tools. amazon also established service level agreements (sla) that apply to all services running on their cloud platform (http://aws.amazon.com/ec2-sla/). these slas define not only uptime but also address security and legal issues. regardless of the concerns or skepticism of the authors, the literature on cloud computing and the expanding use of cloud platforms indicates that cloud computing is still a growing field. both the gartner hype report on cloud computing (2009) and the educause horizon report (2009) point to the expansion of cloud services in the coming years. libraries and especially academic organizations have largely followed suit, having already migrated key services such as openurl providers, and federated and pre-indexed search engines. fox (2009), for example, discusses the growing trend of hosted ilses. libraries who take this approach using a platform or infrastructure approach are doing so with some risk, however, as paas and iaas solutions tend to offer less support for these types of applications. case study in infrastructure services in the cloud for the last two years, the z. smith reynolds library (http://zsr.wfu.edu) has been focusing on migrating key it services to cloud-based or hosted environments. as part of this migration we have also focused on moving our key systems to open source options. we have found that these two goals are not always in sync with each other and have some applications which are decidedly not open source but work well in hosted environments and vice versa. the table below shows which applications have already been migrated or are in the process of being migrated to cloud environments. over time, we expect to migrate paas based services to iaas platforms. services which are not yet in the cloud include our library website, digital library applications, and institutional repository. platform applications saas openurl resolver, journal listing service, instructional guides, reserves statistics, im/chat service paas ils, archives management software, initial website applications iaas institutional repository discovery layer, ils discovery layer table 1: inventory of current cloud-based solutions as we continued to transition to cloud services our discussion centered on key it issues: service level agreements – what assurances do we have for uptime, legal protection, and security? uptime and reliability – how does this provider compare to being able to locally host and manage our resources? cost and affordability – what personnel and technology resources are involved with a hosted versus local solution?  how does this cost model look over time? legal and organizational issues – what organizational and legal issues do we need to consider?  are we dealing with patron data?  are we sure that the platform and our connection to it are secure? staff knowledge – how would migrating to this platform impact staff knowledge and competency?  do we know everything that we need to know? an initial needs analysis indicated that while we enjoyed good university-wide infrastructure, it was becoming increasingly difficult to launch and manage specific applications. for this reason we decided to examine cloud computing as a way of delivering our core library applications including our ils, website, and digital library systems. in late 2008 we licensed our first hosted server that was in essence a paas solution. we chose a virtual private server with a leading hosting company. the server was fairly small (20 gb of storage space, 512 mb of dedicated ram) but offered us total flexibility over the system in addition to an easy to use control panel (plesk). over the course of the year we used the platform for experimenting with remote access and application management. in 2009 we implemented the archivists’ toolkit on the system as a test application and found that it served well although the archivists reported slower system response times than they would like. further, the server had no easy way to expand drive space and we were looking to begin storing larger amounts of data. in early 2009 we began investigating hosting options for our ils. we examined several options but decided to use the ils vendor given their familiarity with the system and sla. we completed the migration to their environment in the summer of 2009 and began looking at where our other applications including the website, discovery services, and digital library services could live. after a survey of available providers we chose to focus on amazon’s ec2 service. our initial investigation proved interesting but not overwhelmingly positive. for example, although the infrastructure was robust, the command line tools required to manage the instances would not be easy for everyone to use and both the servers and the data stored on the primary partition disappeared when stopped. during the course of our investigation, however, these concerns were addressed with new developments and we decided to bring up our first server in the fall of 2009. amazon proved to be a good choice given their scalable server solutions and easy to use management consoles. for example, there were a number of base server images available to work with including open source and proprietary platforms. further, the ability to easily mount large amounts of disk space via their ebs service gave us the ability to implement data-rich services. finally, the amazon management console allowed us to easily start, stop, and manage our servers. recent developments in creating ebs based volumes provides users the ability to stop instances without losing them. in addition to this the ebs snapshot tools are available to create server images. this allowed us to simplify our images as we did not have to find a way to store server logs on a separate ebs volume in case the instance died unexpectedly. amazon also had long-standing features such as the ability to set a firewall at the ec2 service layer and the ability to swap ip addresses on servers easily. these features simplified our implementation and made it easy to switch between development and production environments. in addition to these features amazon offers a number of other options including automatic load balancing, connection with campus networks via virtual private cloud and auction-style access to server time. evaluating cloud computing in order to assess the success of our initial migration to amazon ec2 services, we assessed three central areas; 1) quality and stability of service, 2) impact on our ability to provide library services, and 3) comparison of cost with local technology solutions. quality of service our initial experience with the quality of the ec2 service has been very positive. amazon’s legal agreements and slas adequately addressed our concerns for data security and uptime. like many libraries we have become comfortable using saas-style services for other solutions and are increasingly reliant on internet connectivity, so moving our core application infrastructure outside the campus network proved to be a non-issue. further, the quality of the company and community documentation (http://aws.amazon.com/documentation/) has made finding solutions to common problems such as server configuration, backups, and archives fairly easy. impact on library services one reason that saas and paas solutions are difficult to implement in libraries is that our core applications often require specialized software or configurations which are either localized or simply application specific. by using an infrastructure level service we were able to bring our library applications online without having to find a service that supported the correct versions of the underlying technology. further, having the ability to mount new applications quickly without having to focus on identifying available server space has meant that we can offer technology-based library services much more quickly than when using locally-based hardware. this separation of machine images and data from the hardware they are currently running on provides us the ability to minimize downtime in the case of hardware failures. granted, moving our key application infrastructure outside of the campus network means that during internet connectivity issues we will have none of our key services (e.g. website, ils, discovery layer). while a full plan has not been put in place to handle this contingency, the library is currently configuring a local machine that will be capable of serving up a scaled-down version of the library website and a snapshot of the library catalog running on a locally installed version of our discovery layer (vufind). cost comparison calculating the total cost of ownership (tco) of technology can be difficult, and there seems to be more research which questions the validity of these measures than research comparing cloud and traditional solutions. for example, golden (2009) found that when including electricity and administrative costs amazon’s pricing was comparable with the cost of in-house managed servers. other research, however, indicates that simply swapping in-house for virtual servers does not result in a lowering of tco (leong, 2009). further, the impact of cost comparisons on cloud computing indicate that minimal cost savings may not be a sufficient inducement to change. a recent gartner research article points to larger issues such as security and operational concerns as key factors (harris & smith, 2009). our own cost-benefit analysis indicated that ec2 had similar costs but provided operational benefits. we were facing the need to replace our two servers and large disk array, and in comparing the cost of the hardware projected over five years against the cost of ec2 and ebs data volumes for the same time period we found that the costs were comparable. the initial experience of running our discovery layer on a single small instance server showed actual costs of approximately $100 per month including cpu time, data storage, and i/o fees. after analyzing the actual used capacity of our servers, we found that most of our applications could run on two amazon ec2 small servers (1 cpu, 1.7 gb ram) (http://aws.amazon.com/ec2/#instance). as we add our digital objects to the system we will see increased data storage and backup fees. observations and next steps while we can call our initial transition to cloud computing a success we have had to work through several challenges. first, we found that working with ec2 required some additional training for our staff. while there is good documentation offered by amazon, understanding system architecture in a cloud environment and working out the security issues required careful attention. second we found that while amazon was the most mature of the services on the market when we started, the market is changing rapidly and we will need to remain tuned in to changes to ensure that we make use of new options. for example, the establishment of persistent images was a significant step forward in easing our transition to ec2. finally, we found that while ec2 made it possible for us to mount and configure new servers quickly, it took coordination with campus it services to bring these servers online in a way that was seamless for our patrons. placing our applications on iaas platforms provided us with a flexibility which we had not previously enjoyed with local servers. as a result, the library has been positioned to be more responsive to new developments in the coming years. as the iaas market matures we are looking for services to become more affordable and standardized. further, as more libraries become comfortable with using applications on iaas platforms it becomes possible to share and mount appliance-style servers that serve distinct purposes. about the author erik mitchell (http://erikmitchell.info) is the assistant director for technology services at the z. smith reynolds library in winston salem, nc. works cited 2009 horizon report | educause. (2009, january 20). educause. retrieved january 25, 2010, from http://www.educause.edu/eli/2009horizonreport/163616. breeding, m. (2009). the advance of computing from the ground to the cloud. computers in libraries, 29(10), 22-25. (coins) fox, r. (2009). library in the clouds. oclc systems & services, 25(3), 156-161. doi: 10.1108/10650750910982539. (coins) golden, b. (2009, february 19). the case against cloud computing, part four – cio.com – business technology leadership. cio.com. retrieved january 25, 2010, from http://www.cio.com/article/480595/the_case_against_cloud_computing_part_four harris, m., & smith, d. m. (2009, september 8). higher education q&a: cloud computing. gartner research. retrieved january 25, 2010, from http://my.gartner.com/portal/server.pt?open=512&objid=260&mode=2&pageid=3460702&resid=1169112&ref=quicksearch&sthkw=cloud+computing+cost. hype cycle for cloud computing, 2009. (2009, july 16). gartner research. retrieved january 25, 2010, from http://my.gartner.com/portal/server.pt?open=512&objid=260&mode=2&pageid=3460702&resid=1078112&ref=quicksearch&sthkw=cloud+computing+cost. leong, l. (2009, june 16). software on amazon’s elastic compute cloud: how to tell hype from reality. gartner research. retrieved january 25, 2010, from http://my.gartner.com/portal/server.pt?open=512&objid=260&mode=2&pageid=3460702&resid=1022715&ref=quicksearch&sthkw=cloud+computing+tco. nelson, m. r. (2009). the cloud, the crowd, and public policy. issues in science & technology, 25(4), 71-76. (coins) nist. (2009). nist.gov – computer security division – computer security resource center. retrieved february 12, 2010, from http://csrc.nist.gov/groups/sns/cloud-computing/index.html. truitt, m. (2009). editorial: computing in the “cloud”. information technology & libraries, 28(3), 107-108. (coins) wheeler, b., & waggener, s. (2009). above campus services: shaping the promise of cloud computing for higher education. educause review, 44(6), 52-66. (coins) subscribe to comments: for this article | for all articles 9 responses to "using cloud services for library it infrastructure" please leave a response below: suresh k chauhan, 2010-10-19 hi erik, thanks for putting across a wonderful piece of infomation on cloud computing and its impact on libraries. i have been in search of good article on the topic but while reading “using cloud services for library it infrastructure”, got clearity about the concept. thanks once again for an informative article. regards, suresh k chauhan amanda goodman » blog archive » cloud computing in libraries and library school, 2011-04-18 […] mitchell, e. d. (2010, march 22). using cloud services for library it infrastructure. retrieved november 5, 2010, from code4lib journal: http://journal.code4lib.org/articles/2510 […] prof. nandkishor gosavi, 2011-11-30 hi i am prof. nandkishor gosavi i am working as an librarian in dr. d. y. patil knowledge city , pune ,india . i want work on cloud computing. so please halp me rakesh mohindra, 2011-12-14 very clear information dealing with many conceptual terms of cloud computing in respect of applying in library context dr d.p. singh,, 2012-01-10 dear eric i have just gone through your article. i sure this will be more informative for me beacuse concept of cloud computing is very new for me. dr d.p. singh asstt. librarian gbpuat pantnagar dr s m rokade, 2012-02-20 article is quite useful for researchers being an new idea dr s m rokade librarian anca, anandwan, warora, ms dr. kanchan kamila, 2014-06-27 dear eric, thank you so much for your valuable article. hope this article will be helpful for academic community. dr. kanchan kamila university librarian the university of burdwan sneh lata, 2014-11-10 dear eric, thank you so much for such a informative article. regards, sneh lata professional development – kevin at lita national forum, 2016-07-26 […] cloud computing. from saturday’s general session, roy tennant discussed how the cost of innovation is approaching zero, that the model “easy-come-easy-go” enables a greater flexibility and lower risk to experiment, and cited erik and his code4lib article. […] leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – the use of python to support technical services work in academic libraries mission editorial committee process and structure code4lib issue 58, 2023-12-04 the use of python to support technical services work in academic libraries technical services professionals in academic libraries are firmly committed to digital transformation and have embraced technologies and data practices that reshape their work to be more efficient, reliable, and scalable. evolving systems, constantly changing workflows, and management of large-scale data are constants in the technical services landscape. maintaining one’s ability to effectively work in this kind of environment involves embracing continuous learning cycles and incorporating new skills – which in effect means training people in a different way and re-conceptualizing how libraries provide support for technical services work. this article presents a micro lens into this space by examining the use of python within a technical services environment. the authors conducted two surveys and eleven follow up interviews to investigate how python is used in academic libraries to support technical services work and to learn more about training and organizational support across the academic library community. the surveys and interviews conducted for this research indicate that understanding the larger context of culture and organizational support are of high importance for illustrating the complications of this learning space for technical services. consequently, this article will address themes that affect skills building in technical services at both a micro and macro level. by maria collins, xiaoyan song, and sherri schon 1.0 introduction 1.1 the importance of a digital mindset technical services professionals in academic libraries are firmly committed to digital transformation and have embraced technologies and data practices that reshape their work to be more efficient, reliable, and scalable. evolving systems, constantly changing workflows, and management of large-scale data are constants in the technical services landscape. maintaining one’s ability to effectively work in this kind of environment involves embracing continuous learning cycles and incorporating new skills – which in effect means training people in a different way and re-conceptualizing how libraries provide support for technical services work. this article presents a micro lens into this space by examining the use of python within a technical services environment. in addition, the surveys and interviews conducted for this research indicate that understanding the larger context of culture and organizational support were also of high importance for illustrating the complications of this learning space for technical services. consequently, this article will address themes that affect skills building in technical services at both a micro and macro level. in their book, the digital mindset, leonardi and neeley (2022)[1] discuss the concept of a digital mindset, which “is the set of approaches we use to make sense of, and make use of, data and technology,” and what it means to function in today’s information environment (p.11). the idea of a digital mindset is a useful frame for understanding the larger context of this research as there are implications in a library environment beyond the application of the skill, including gaining administrative support, providing an education framework, and re-structuring the organization to better align technical skills with data expertise. successful skill-building for individual staff includes deepening their technical understanding of their work even if they are not the person writing actual code. this is explained broadly by leonardi and neeley (2022) when they note that “skills give you the vocabulary, knowledge and intuition to see the bigger picture – to ask important questions” (p. 10). it also involves cultivating a supportive culture that allows for experimentation across all levels of an organization. leonardi and neeley (2022) observe that “if employees don’t feel comfortable suggesting experiments, managers and departments cannot benefit from insights of those closest to the data” (p.162). from a theoretical perspective, it’s easy to recognize that harnessing insights of those embedded in the work is a critical component of process improvement. this is often trickier in practice especially in larger organizations with many roles and many moving parts. one intent of this article is to elevate and make visible the evolving skill sets that are of increasing importance and utility in technical services work. 1.2 nc state context and motivation for this work experimentation has been a key element of the culture of north carolina state university (nc state) libraries, which has led to a willingness to build custom solutions to assist with managing technical services when market solutions are unavailable. this approach has provided opportunities for technical services staff in acquisitions & discovery (a&d) to gain more technical skills by participating in the development of solutions that support electronic resource management and automation of technical services work. prioritizing staff development was critical in developing the knowledge needed for staff to participate in this kind of work. these experiences have also been critical in developing a digital mindset for a&d staff. in 2012, the a&d department became increasingly interested in introducing additional technical skill sets, such as light-weight scripting using python, after the creation of a data-centered unit in the a&d department called data projects & partnerships (dpp). the formation of this unit was in direct response to the need for more technical and metadata expertise. dpp staff have learned to use openrefine, regular expressions, and some programming languages (such as python) in order to write queries and create data workflow efficiencies. given the growing interest in using python in a&d as well as across several other library departments, a&d’s former electronic resources librarian, xiaoyan song, organized a python interest group to examine the use of python in library work. this interest group has served as a learning cohort for sharing python-related tips, ideas, and strategies. sharing sessions of the group also revealed increased interest and application of python by the broader library community. these ad hoc efforts have proven useful in exposing staff to the use of python to support their work, however evolving these kinds of technical skills in a&d has not occurred without navigating numerous barriers including gatekeeping of resources and limited institutional understanding of the evolving requirements for technical services work. as a&d staff embarked on their learning journey, cultivating and acquiring organizational support emerged as a critical component for the successful adoption of these evolved skills. consequently, in addition to learning about examples of how python is used to perform technical services work, our team was interested in learning more about training and organizational support across the academic library community to support growing this skill set among technical services staff. 1.3 research objectives in order to explore this further, the authors conducted two surveys and eleven follow up interviews to investigate the following three areas: use of python to support technical services work including how often python is used, areas of support, examples, and strategies for organizing and communicating code; education including skills and competencies, formal and informal training options, and suggested resources; and organizational support and acknowledgment including positions in the organization performing the work, types of organizational support such as time, increased pay, or formal training, transition of existing positions, and inclusion of related skills in position descriptions. this research represents conversations among a limited number of professionals in the field, and these findings signal trends and areas of potential growth in technical services work. beyond providing ad hoc examples of python work, these conversations point to the value of growing technical skills more broadly to support technical services work. 2.0 literature review 2.1 changing landscape of work since the approach to this article is both broad and narrow in scope, the literature review topics explored were likewise. starting with the bigger picture context of the changing nature of work, leonardi and neeley (2022) observe that transitions in a digital environment are continuous and require constant change management. they state “leading in a digital environment is no longer about navigating change when it happens, it means helping your coworkers, employees, partners, and customers to continuously prepare for what’s coming next and to embrace that they live and work in inexorable transition” (p. 168). this concept as it relates to skills training indicates a different kind of learning environment that allows for constant introduction of new skills and a system of training that is iterative and not staged in phases. this has implications related to formal vs informal training as well as the ability to independently learn new skills. more specific to libraries, adam (2018)[2] discusses “the shifting paradigm of libraries in the digital age” and advocates in his essay that api training is an essential skill for all librarians (p. 180). although this article is not python-related, adam’s call to action points to the shift in the kinds of works that librarians perform in the digital age and highlights the division of labor that exists in many libraries between information technology professionals and librarians. there are increasingly blurred lines between these two groups. adam (2018) argues that “this division, though, is not tenable in the information world in which librarians now work, and it is certainly not going to be tenable in the future. as libraries focus increasingly on digital content, and as the realm of the information libraries collect and provide access to expands beyond the traditional “published” forms of monographs, reference works, and serials, anyone who seeks to provide patrons access to information is going to have to be familiar with pulling that information from a range of sources” (p.182). this shift in the way library professionals work goes both ways as the information skills applied by technical services staff are increasingly useful for supporting broader library functions. this is especially true for metadata support. madden (2020)[3] discusses this idea of broadening the application of technical services support for areas like accessibility work for websites, for instance, by discussing how technical services staff are helpful in creating alternative text for images on a library’s website. 2.2 shifting core competencies expanding the technical skills that librarians need in a digital environment indicates a need for broader core competencies related to technology for areas of librarianship. the literature in this area focuses on developing competencies for library science more broadly as well as specific areas of librarianship where there is increasing demand for programming skills. in a 2022 article, yadav[4] seeks out alumni of india’s 10 library and information schools to determine essential competencies that should be taught. “most of the professionals opined that lis schools should integrate computing or it skills in the curriculum, including digital library skills, library automation, discovery services, web publishing and database management. there should be intensive training on digital tools, statistical tools, programming languages, content management software and so on” (yadav, 2022, p. 853). beyond library school, the literature also points to a need to evolve competencies for professionals working in academic libraries. raju (2017)[5] describes the changing skill sets required by academic librarians and notes that their “south african study concurs with findings from the international literature that librarians in the academic library environment, which has seen perhaps the greatest impact of technology compared to other library sectors, require it knowledge and skills to a significant extent” (p.753). in respect to core competencies for specific areas of librarianship, read and cox (2020)[6] explore technical competencies for scholarly communications librarianship. their take is more about creating connective tissue between stakeholders interested in technical innovation rather than having scholarly communication librarians learn coding. they conclude that the necessary competency related to technology should be “soft skills and certain positive orientations to technology rather than practical technical skills” (p. 1). this speaks to leonardi and neelley’s (2022) ideas about establishing a baseline knowledge related to technical work that allows for and enhances collaboration. in other words, increasing understanding of these skill set areas has a positive benefit even if you are not actually programming. another article by white and powell (2019)[7] focuses on the need for evolved core competencies to better support gis librarianship noting the increasing “importance of coding and programming languages” (p. 65). these authors discuss the utility of “applying a coding approach to solve practical problems in the library” and note, “in particular, r and python, two programming languages used widely due to their accessibility and ease of learning” (p. 47). this call for updated technical skills to perform gis work mirrors a similar need for shifts in core competencies for technical services work. related to technical services core competencies specifically, frank (2013)[8] discusses not only the changes in technical services work, but also the collaborative benefits between it and technical services professionals. as batch editing of bibliographic marc records remains an important aspect of a cataloger’s job, frank (2013) observes that “a cataloger’s ability to add to their arsenal additional tools and skills to automate workflows by analyzing and correcting marc records in bulk is becoming more critical than ever.” frank points to the tension that often exists between those with programming skills and those waiting on programming skills and observes that these tensions are often best handled through collaborative synergy. frank observes that “the increase in collaborations between catalogers and coders, as well as the increase in catalogers who are coders, is proving to be the next logical direction for the field of bibliographic description and access.” shifting competencies in this way directly corresponds to the changing demands of the job. mitchell (2013)[9] discusses this idea of evolving competencies by framing programming competencies as part of metadata literacy. mitchell states that “the ability to work with metadata programmatically both allows [a metadata professional] to be more effective at a type of work that they do and have greater overall control over the flow and use of metadata in their collections. while there is considerable movement in the library community toward cloud-based and linked data models for metadata these new formats require greater, not less, familiarity with programmatic tools (p. 309).” this last reference to understanding metadata initiatives through a programmatic lens is central to the idea of a digital mindset. likewise, adam’s take on librarians learning to use technology, such as apis, further establishes the idea that evolved core competencies in this area are a necessity to perform the job. adam (2018) states that “if librarians want to continue to play a key role in the supply chain of information, they will take this opportunity seriously and learn these new skills” (p.187). the common thread across these articles that are focused on developing technical core competencies is the critical need to evolve work practices to align with the changing nature of the job. 2.3 why python – its utility and value to libraries so, why python? there are several reasons including general agreement in the literature that python is an accessible and easy to use coding language for library work. walton states that python is “one of the best languages to start with…python is simple and powerful” (thomsett-scott, 2016, p.7)[10]. he also addresses the accessibility of python noting “a person reading the script can easily see what the script is trying to do” (thomsett-scott, 2016, p.7). white and powell (2019) concur that “one big advantage to learning python is that it is easy” (p.54). other authors comment specifically on coding libraries offered for python including pymarc. magnuson (2016, july 5)[11] comments that “python is a great programming language to know if you work in a library:… its syntax is fairly clear and intuitive, and it has great, robust libraries for doing routine library tasks like hacking marc records and working with delimited data, csv files, json and xml.” pymarc, which is a “python library for working with bibliographic data encoded in marc21…and provides an api for reading, writing and modifying marc records,” is referenced by several authors as a benefit for libraries, especially for catalogers (python software foundation, 2023).[12] frank (2013, april 17) explains that the pymarc python library is a result of an open-source project and “provides common methods of manipulating field indicators, subfields, and field content.” in terms of utility, hill, frank and pernotto further explain that “this ability to programmatically work with data can save a large amount of time, and while tools such as marcedit can do large amounts of batch processing, learning pymarc both gives greater freedom and, as mentioned, allows calling from batch scripts, such that it can be one of a number of things you automate together” (thomsett-scott, 2016, p. 17).[13] the consequence of this automation is dramatic savings in time and improved, efficient workflows. this is critical, especially for technical services departments that continue to decrease in staff support. numerous articles emphasized that the scale of automation using python can be quite small and still provide value for daily tasks. this limited frame or concept of scale helps to make python more accessible and less overwhelming to staff with limited programming expertise. yelton (2015. april)[14] comments that “you don’t need to write large-scale, polished, reusable software in order to get big benefits from learning to code. automating a task with a few dozen lines of code can save you many hours in a year” (p.29). python’s accessibility allows for these interactive improvements. this is especially true for areas of growth in the collection, such as ebooks, where staff are having to manage high-volume workflows on a daily basis. for instance, thompson and traill (2017, oct 18)[15] observe that “as ebook collections grow (and libraries’ staff shrink), management processes must also be scalable… in the current environment with its complex implementation choices and demanding maintenance workflows, automation is essential to make these choices and tasks manageable.” another growing category of technical services work involves improving inclusive description. dong, et al. (2017)[16] note that python could also prove useful for managing remediation projects (p.144). another important factor in helping technical services staff learn and use python is the community of practice that supports this tool. having a community to lean on when learning something technical is a game changer, especially for new or inexperienced staff. enis (2013, march 1)[17] reports that the python community is one “that [does] a lot of outreach to newbies, and [is] explicitly pro-diversity.” the value of the python community is also commented on by hill, frank and pernotto in their chapter from the librarian’s introduction to programming (thomsett-scott, 2016). they comment on the support provided stating that “the community is highly robust, creating packages and offering help so that any questions you might have will quite possibly already be answered” (p.17). they also talk about the general philosophy of the community in terms of ease of use and openness. “the language and the vibrant, active community that uses it place great value on code being readable and as obvious as possible; clarity is prized over cleverness and openness over concealment. this has led to the community releasing large amounts of open-source code” (thomsett-scott, 2016, p.17). 2.4 examples of python given the openness of the python community, it’s no surprise that the literature abounds with examples of strategies, tips, and techniques for working with python. many of these examples are one-off descriptions of projects that share code or scripts. one common area of interest was collections analysis. examples from the literature include how to use python scripts to assist with reviewing duplicate titles to facilitate gift processing (maguson, 2016), how to assist with a journal review project through subject heading evaluation (mcdavid, mcdavid, das, 2021)[18], and ways to query gobi to help with collections reporting (frazier, 2020)[19]. all of these collection examples realized time savings and efficiencies. for instance, frazier (2020) noted that the script she used to help compile reports reduced time and effort from 10-15 hours a week to just 30 minutes. several of the articles mention useful python libraries for data analysis in addition to pymarc including the beautiful soup library and the pandas library; in particular, these articles call out the ability to process large amounts of data (maguson, 2016; zavalin & miksa, 2021)[20]. another exploration of python libraries offers a comparison for readers. in harrington’s (2019)[21] discussion of a pilot project to generate geographic terms for a streaming video site, he uses three different python libraries to generate terms and then compares these methods for quality. in general, these articles are a useful introduction to the various python libraries and how they might be used. other technical services examples in the literature have a bibliographic focus either for data clean up, updates, or remediation. davis (2020)[22] outlines a project that utilized python to update 1,487 marc records with hierarchical place names from a rare materials cataloging aid. another example discussed by doug, glerum, and fenichel (2017) used python to help with data remediation of duplicate series data. these authors emphasized how “catalogers can build on their traditional knowledge and also use data analysis, scripting, and batch manipulation when performing large-scale remediation” (doug, glerum, & fenichel, 2017, p.143). another article by thomson & traill (2017) not only discusses how staff at the university of minnesota libraries use python to improve the quality of metadata for ebooks, but also ways to use python to evaluate marc record sources from vendors, the alma community zone, and worldshare collection manager. 2.5 training & organizational support in respect to support for python work, the literature does provide training resources as there is “little formal pipeline for teaching librarians to code” (yelton, 2015. p.15). in addition to a list of resources at the end of yelton’s (2015) chapter on learning to code, she also provides several strategies for learning that include “ • find a project • rely on google and existing code • write documentation • persevere • find a mentor” (p.26). helping academic librarians find training opportunities to learn to code is also a central focus of enis’ (2013) article “cracking the code.” enis notes a variety of options including “classes at the schools where they work [and] free online lectures and tutorials.” from a broader perspective, leonard & neeley (2022) point out several key ideas related to education and training. the learning experiences need to be understood as continuous and should be customized to the needs of the learner, and there needs to be a supportive environment. they observe that “tailored curricula are the core of continuous learning initiatives – designed for specific roles and functions based on the skills and credentials necessary for each” (p.193). in general, leonard & neeley (2022) acknowledge that this kind of learning, which is self-driven, can be isolating and that it’s good to have a community to reach out to that supports this space ( p.193). this brings us to the importance of having support from your organization to pursue this kind of work. there is limited discussion of this topic in the literature as most of the resources explored were pragmatic in nature, providing examples of code and discussion of projects. but this idea of organizational support was addressed in some resources in a few different ways. in raju’s (2017) discussion of the blurring lines between information professionals and it professionals, he posits that this evolution in skill sets for library science professionals should be acknowledged and taught to library professionals as part of the domain. he states that “it knowledge and skill sets should be taught not as stand-alone or it-serviced courses but should be firmly embedded within lis epistemology, demonstrating the intellectual claim on this broadened disciplinary space resulting from a natural evolution of the lis discipline in response to a technology-driven information environment” (raju, p.754). this is not a direct recommendation for organizational support, but it is at least an acknowledgement that the foundational concepts for library science should include these kinds of knowledge and skills. yelton (2015) addresses organizational support in another way by pointing out barriers or a lack of support in some institutions that may prevent this kind of learning. yelton comments that “unfortunately, some librarians have no support, or even face active hostility… [yelton shares a story about] two managers [who] were unable to secure coding skills development for interested supervisees because their institutions did not want to reclassify them into higher salary categories reflecting those skills” (p.29). this lack of support often goes beyond salary concerns. yelton also points out how organizational structures can prove to be intractable and lead to gatekeeping where certain skill sets are either exclusive to a particular unit in an organization or subject to approval when these technical skills are needed to perform work in other units. yelton (2015) illustrates this with an example of “one librarian who spends a significant amount of time coding and is doing so without upper management’s knowledge; in that institution, only people belonging to other, explicitly technical, units are allowed to code” (p. 29). these ideas are discussed further by leonardi and neeley (2022) within the context of organizational structure and who is allowed to experiment, specifically departmentalizing or centralizing certain skill sets. they explain that “existing organizational structures can make it difficult to democratize experimentation and, in many cases, they lead to the death of experimentation” (p.158). leonardi and neeley note that it is more helpful to the organization to have people closest to the services and data participate in innovation (p.157). within the context of academic libraries, this means allowing staff with data expertise in specific domains the opportunity to develop technical skills that support that work. managers can help support these endeavors in several ways. yelton (2015) highlights several methods of support including time, resources, workshops and conferences, code review, and internships (p.29). much of the focus of the literature in this space centers on specific resources or techniques that the community can re-use or borrow. other aspects of the literature include discussions of blurring lines and core competencies for technical services work. the authors believe that this research effort is unique in its broad assessment of this space using interviews and surveys to touch on not only examples and resources, but also on the types of organizational change and initiatives that support the development of technical skills needed to support evolving practices in technical services in academic libraries. this research also highlights the value of teaching technical skills to staff not trained as developers as a means to bridge gaps and provide a better understanding of the work and systems that support it. this holistic approach brings to the surface ideas and strategies to transition organizational structures, understand barriers that require active management, and to shape continuing learning and development opportunities in this space. this article also consolidates resources shared among the participants to share with the larger community. 3.0 methodology to investigate the use of python in supporting technical services work within academic libraries, the authors conducted two surveys and a series of eleven interviews, which received approval from the institutional review board at nc state university. the first survey was originally developed by sherri schon, an electronic resources intern in nc state’s a&d department, as part of her internship project. the survey featured open-ended questions focusing on participants’ experiences using python for technical services in academic libraries, as well as their training and learning journeys related to python. the survey questions can be found in appendix a. in july 2021, the first survey was distributed through the code4lib listserv, a community that intersects with libraries and possesses technical expertise. within three weeks, the authors received nineteen responses. these responses provided insights into how python is employed in technical services work, encompassing aspects such as the duration and frequency of python usage, strategies for managing python-related tasks, practical examples of python’s applications, the concurrent use of other programming languages, and details about formal and informal training. upon review of the initial survey results, the authors were intrigued by the influence of organizational structure on python-related work, and this topic was added to the scope of this project. survey participants were also asked if they would be willing to participate in follow-up interviews. out of the nineteen survey respondents, eleven expressed interest in interviews. subsequently, in september 2021, the authors sent interview invitations to these individuals, all of whom graciously accepted. the interviews, conducted in hour-long sessions via zoom, were organized either as group sessions or individual interviews based on participants’ preferences. each focus group consisted of two or three participants. the interview questions were categorized into three sections: experience, training and education, and organizational structure. for the experience category, participants were asked about the management of python work, the specific tasks for which python is employed, and how python scripts are handled. training and education inquiries revolved around formal and informal training experiences, their benefits, important prerequisites for learning python, and training responsibilities within the organization. under organizational structure, the authors sought insights into the roles responsible for python work, the evolution of these roles, and the organizational support provided. the interview questions can be found in appendix b. following the interviews, the authors recognized that participants’ python experiences could vary depending on the size of the library and the department. subsequently, additional information regarding the library’s size and department was collected from the participants via email. in september 2022, the authors decided to conduct a second survey (see appendix c) that more directly focused on the technical services community. the objective was to explore how python was used to support electronic resources work within academic libraries, with a particular emphasis on technical services. three listservs, namely electronic resources in libraries (eril-l), acqnet, and autocat, were selected for the second survey. this survey revisited some of the same questions from the first survey regarding participants’ python experience and introduced new inquiries aimed at understanding the ways in which organizational support was provided. the authors received 27 responses within three weeks. to distill the insights from both the surveys and interviews, the authors reviewed and synthesized the information in several ways. they sought to identify valuable knowledge to share with the community, including examples of python’s practical applications, similarities and differences in responses to discern potential patterns or indicators, and qualitative examples or stories illustrating how this work is supported. 4.0 results and discussion this section delves into the findings derived from the surveys and interviews, presenting them under three categories: python experience, training and learning, and organizational structure. 4.1 python experience this section scrutinizes various dimensions of the respondents’ experience with python in the context of supporting technical services work. it includes an exploration of the duration for which respondents have been utilizing python for their work, the frequency of its use, the management of python-related tasks within specific institutions, encompassing the individuals responsible for writing python scripts and the workflow for soliciting such work, as well as an examination of the specific tasks for which staff utilize python scripts. 4.1.1 length of time using python the question of how long staff have been using python in support of technical services work was asked in both surveys. similar results were seen across all listservs as well as some differences in different communities. over 60% (20 out of 33) of participants from the code4lib listserv have used python within the five-year range. for the same five-year range, almost 50% (11 out of 22) of participants from other listservs have used python (see figures 1a and 1b). about 6% (2 out of 33) code4lib participants started using it within a year and a bigger percentage, slightly over 30% (7 out of 22), from other listservs fall into this category. from the code4lib responses, 33% (11 out of 33) of participants have used python for more than five years and a smaller percentage, less than 22% (5 out of 22) from other listservs have used it for more than 5 years. the results indicate that the code4lib community had started to use python earlier than respondents from the other listservs that we surveyed. this is likely due to the fact that the code4lib community comprises many developers who do development for libraries while other listservs focus more on traditional technical service work such as acquisition, cataloging, and electronic resource work. the general trend among all listserv responses is that python has been picked up by academic library workers within the last five years to assist in technical services work.   figure 1a. number of years using python. response from code4lib listserv survey. figure 1b. number of years using python. response from other listservs survey. 4.1.2 frequency of python use when examining the frequency of python usage among survey participants, the responses revealed a clear division between heavy users and light users, as depicted in figures 2a and 2b. among code4lib participants, 30% employ python on a daily basis, 33% do so weekly, while 24% utilize it at least once a month. the remaining 12% employ python much less frequently, with intervals exceeding one month. for respondents from other listservs, the distribution shifts slightly, with monthly users representing the largest portion, exceeding one third of the total. these findings underscore the pivotal role python plays in supporting day-to-day technical service work for the surveyed participants. figure 2a. how often do you use python in your work? from code4lib survey. figure 2b. how often do you use python in your work? from other listservs survey. 4.1.3 assigning or distributing python-related work the survey also inquired about the methods used by participants to manage their python-related tasks and how they received requests for such work. in terms of the assignment of python-related tasks, participants outlined a diverse set of approaches. some requests were formal and channeled through a ticketing system (occurring once or twice a week) for technical support. requests also originated directly from their supervisors or spontaneously from colleagues in different departments on campus. regardless of the source of these work requests, participants highlighted a common theme: they typically encounter a problem (rather than a python-specific request) and are entrusted with the autonomy to select the most suitable solution. python is frequently one of the options they consider. these responses underscore the significance of being closely connected to or integrated within the problem spaces that staff support, as well as the staff’s capacity to independently explore a variety of solutions, including the use of python. 4.1.4 types of work supported by python work the survey findings demonstrated that python is a versatile tool employed by staff to facilitate a wide range of functions within the library, spanning acquisitions, circulation, marc records, institutional repositories, system api integration, reporting, data transformation, data analysis, and language translation. although the nature of these tasks varies, a common thread is the involvement of batch processes or process automation in supporting these areas. a glimpse into the types of work for which python is utilized is presented in figure 3, which derives from the second survey. the top five applications of python among respondents encompass system api integration (56.5%), task automation (56.5%), data cleanup (52.5%), batch processing of marc records (34.8%), and reporting (30.4%). for more comprehensive insights into the specific work categories that survey participants employ python to support, please refer to appendix d. figure 3. types of work that python supports. from code4lib survey. 4.1.5 other programs, apis, and programming languages used with python the surveys and interviews revealed that python is employed both independently and in tandem with other software tools. respondents from the first survey and interviewees disclosed instances where they combined python with other applications, apis, or programming languages, such as integrated library system (ils) apis, java, javascript, sql, php, and more. this underscores python’s versatility as a general-purpose language, making it a versatile and widely applicable tool. 4.1.6 managing python code and documentation during the interviews, participants shared various approaches for managing python scripts and documentation. many recommended utilizing github as a platform for storing and sharing code, both within and outside their organizations. additionally, a few of them adapted python code authored by others from their github repositories. some participants opted to store their code and documentation on local machines or internal documentation systems, such as confluence wiki, while others found value in using google colab. notably, one participant using google colab mentioned creating and sharing scripts with colleagues in their department. the choice of which system or tool to employ predominantly hinged on availability, convenience, or the necessity to share with others. one noteworthy best practice highlighted by several participants was the importance of commenting on the code, enhancing its comprehensibility and usability for others. 4.2 training and learning the second aspect investigated in the surveys and interviews pertained to training and learning. participants were queried about the methods they employed to acquire the skills needed to integrate python into their technical services responsibilities. this inquiry encompassed both informal and formal training experiences, as well as the obstacles encountered in the learning process. 4.2.1 acquiring python skills is a computer science degree a prerequisite for acquiring python skills? our participants unequivocally asserted that it is not. the majority of them developed their python proficiency through self-guided learning and learning on the job. among the eleven interview participants, only two possessed a background in computer science. some participants even had humanities backgrounds, including one with a degree in english. the results of the second survey, involving participants from other listservs, unveiled a diverse array of approaches to acquiring python skills (see figure 4). notably, none of the survey respondents learned python through formal education during their undergraduate or graduate studies. instead, 64% engaged in python workshops or courses, either in-person or online, while 41% pursued self-guided learning without formal training. thirty-two percent turned to youtube as a valuable resource for learning python, and some found python classes within library schools, constituting 9% of the participants. additional avenues for acquiring python skills included utilizing books and attending conference presentations. figure 4. methods for learning python. from the code4lib survey. 4.2.2 value of formal training when asked about the potential advantages of formal python training, the responses from over half of the interview participants indicated that its value hinged on an individual’s preferred learning style. while some individuals might find formal training beneficial, others may not. participants also acknowledged the crucial role that library schools can play in providing formal training. they suggested that library schools should offer classes on the fundamentals of programming, as a strong foundation in these principles can facilitate the acquisition of proficiency in any programming language. additionally, some participants emphasized the importance of tailored training opportunities, concentrating on python within a specific library domain, as these might be more advantageous than generic python courses. 4.2.3 other learning strategies during the interviews, participants were asked about their responsibilities regarding training their colleagues in python usage. in general, none of the interviewees held such responsibilities. this was mainly because they either operated as the sole python user in their respective roles or were among a very limited number of individuals within their departments or libraries who utilized python. nevertheless, one interviewee recounted receiving training in a different programming language from their supervisor in a previous job. another participant highlighted the potential benefit of informal or formal mentorship programs to foster skill development among their peers. this underscores the potential value of internal training and mentorship as a means of organizational support. numerous interviewees emphasized the importance of having a prior understanding of command lines and apis before embarking on python-related tasks. furthermore, they identified additional beneficial knowledge areas, such as git, github, regular expressions, and marcedit. previous experience with programming languages also proved advantageous, as it facilitated comprehension of python’s logic and structure. participants further contributed a wealth of valuable resources for python learning and work, which can be found in appendix e. 4.2.4 python learning challenges when confronting the challenge of acquiring new knowledge, the primary obstacle, as reported by both interview and survey participants, was the constraint of time. for instance, a survey participant expressed, “the hardest part was learning the basics; i had to make time for that outside of work and it took multiple weeks. it can be hard to convince fellow staff to make the effort, especially if they feel overworked already, and it’s easier to just use excel or openrefine to do the same thing.” in summary, python proved to be beginner-friendly, with no necessity for a computer science degree to embark on its learning journey. individuals cultivated their python skills through on-the-job tasks and capitalized on professional development opportunities. online learning emerged as a popular choice due to its accessibility and flexibility allowing individuals to learn at their own pace from virtually anywhere. while formal training may not always be deemed essential, the emphasized relevance of knowledge by participants, such as tools openrefine and marcedit, underscores the value of gaining a foundational understanding of wrangling with metadata in library technical services. 4.3 organization structure in addition to exploring participant experiences with learning and training python, this study also looked into how organization structure impacted or influenced the use of python in technical services. four aspects were considered: location of positions within an organization, evolution and transformation of current positions, position descriptions, and organizational support and barriers. 4.3.1 location of positions in the organization regarding the organizational placement of positions utilizing python for technical services, the interview findings revealed a diverse range of department locations. these positions were not exclusively housed within traditional technical services departments. interviewees held roles in various departments, including four systems librarians in library system departments, three programmers in information technology (it) departments, one archivist in a digital strategy unit, another archivist in a special collections technical services unit, one librarian responsible for metadata and assessment in a resource discovery services department, and one metadata analyst in a technical services department. of all the interview participants, only three positions – the metadata librarian, metadata analyst, and one archivist—resided within conventional technical services departments. their duties encompassed batch data manipulation to support institutional repositories, metadata management, cataloging, authority control, and working with dublin core elements schema. two of the positions located in technical services, the metadata analyst and the archivist, mentioned that python was not utilized in their department until after their hire. for those situated outside traditional technical service departments, their responsibilities primarily revolved around technical support tasks such as batch loading, report generation, or serving as library system administrators to assist colleagues in technical services. additionally, these positions contribute to data science initiatives within their libraries. in cases where positions were located outside of technical services, python usage varied, with some work tasks incorporating use of python and others not using it. one factor to consider regarding the placement of positions performing python-related work for technical services is the size of the institution. the libraries where interviewees were employed varied in staff size, ranging from five to over five hundred individuals. two specific examples stand out in this context: one institution with a substantial technical services staff retained the position responsible for python-related work within the technical services department. this institution had twelve staff members in their cataloging department, and the individual responsible for python work transitioned from a specialized language cataloger to a more technically oriented role that supported batch processing, automation, report generation, and linked data projects. conversely, in another institution with a smaller overall library staff of 18, the position handling similar python-related technical services work transitioned from a cataloging role but was relocated to the it department. their duties encompassed reporting, batch data cleanup, and extracting data from various systems like illiad and alma. the results from the second survey, which targeted listservs closely associated with technical services, did not explicitly indicate departmental location. however, respondents did specify their roles at their respective institutions. many of these roles, such as acquisitions librarians and electronic resources librarians, are conventionally located within technical services departments. according to the second survey, the top five roles for staff performing python-related work included metadata librarians (47.8%), catalogers (30.4%), electronic resources librarians (26.1%), acquisitions librarians (13%), and system librarians (13%). please refer to figure 5. notably, all of these position titles, with the exception of system librarians, are typically associated with technical services functions. figure 5. respondents’ roles. 4.3.2 evolution and transformations of current positions regarding the inquiry into whether organizations are transitioning traditional technical services roles into more system-oriented positions, the responses were mixed. among the eleven interviewees, only three had experienced such a transition from traditional technical services positions to more technically focused roles. one interviewee’s role shifted from cataloging to systems librarian, while another transitioned from a technician in technical services to a programmer and system administrator position. similarly, a metadata librarian had evolved from a conventional cataloging librarian position. interview participants held diverse perspectives on whether the transformation of technical services positions into more technical roles represented a potential trend. the metadata analyst interviewed expressed optimism about seeing this transition occur more frequently, as they believed that utilizing python for batch work was a progressive step. the metadata librarian emphasized the increasingly data-centric nature of technical service work, suggesting that technical services staff should concentrate on data analysis and visualization, indicating a potential need for staff with enhanced technical skills. on the other hand, some interviewees stated that their libraries had no intentions of converting traditional technical service positions into more system-oriented roles. one reason cited was the presence of a robust library systems group within their libraries. nonetheless, one interviewee observed that such transitions might occur organically as personnel retire. 4.3.3 position descriptions during the interviews, participants were also asked about whether python skills were explicitly mentioned in their job descriptions. the responses varied depending on the participants’ job titles. two system librarians stated that python was not specifically listed but that programming and reporting were integral parts of their job descriptions. one interviewee mentioned that position descriptions for faculty librarian roles were challenging to update, making it unlikely for something as specific as python to be included. both the metadata analyst and the metadata librarian noted that python was not listed as a requirement but that programming skills were either expected or necessary. the two archivists interviewed indicated that python was not mandatory but was considered a valuable addition in the preferred category for their roles. several system librarians and programmers shared that python was initially not mentioned in their job descriptions but was subsequently added, alongside other coding languages, when their position descriptions were revised. when asked whether their departments would continue to seek candidates with python skills or if the use of these skills was more of a personal preference, interview participants once again provided a variety of responses. some interviewees expressed their intention to keep hiring individuals with these skills, while others stated that it wouldn’t be a strict requirement. a few participants observed that the use of python was a matter of personal choice. one participant mentioned that although functions or tasks might be slower without python, staff would still be able to complete their work. another interviewee saw the demand for python skills as a trend, even though it hadn’t fully manifested in their libraries yet. generally, the use of python in technical services was considered a current trend by the interview participants. 4.3.4 organizational support and barriers the authors were also interested in exploring the different methods organizations employ to facilitate python-related work. in the second survey, participants were queried about the mechanisms their organizations employ to support python skills development (see figure 6). the findings revealed that 85% of the organizations, as indicated by the survey participants, allocated time for independent learning among their staff. additionally, 40% offered financial assistance for training, 10% delivered structured training programs, and another 10% recognized and rewarded staff who used python to enhance their work with increased compensation. figure 6. organizational support for python learning. however, information gained from the interviews as well as the literature review highlighted numerous organizational barriers to learning python or similar technical skills. these barriers included: gatekeeping: this refers to when others determine your level of access or need for a particular skill. one interview participant mentioned that they hid the fact that they used python to assist with their work as they were not allowed to complete their work in this way by their institution. imposter syndrome: during one interview discussion a participant admitted that if python had been listed in the job ad, they probably would not apply. this comment was made after the participant acknowledged how useful they found python to be on the job. formal acknowledgement: this relates to lack of formal acknowledgement by the organization that python is useful and necessary when performing technical services work as well as a lack of monetary compensation. informal adoption of coding skills can result in lack of systemic support where training and funding are left up to the individual on a case-by-case basis. limited knowledge of changes in technical services: for behind the scenes library work like technical services, there is often limited understanding by others in the library of changing workflows and processes. this lack of knowledge by others in the library of the many transitions that have occurred in technical services can lead to a form of benign neglect where administrators are simply out of touch with changing requirements and competencies. these kinds of barriers and concerns reveal that it’s simplistic to just ask a talented staff member to learn a new technical skill and apply it. there are management and infrastructure strategies that are required to better support staff and ensure they have the resources needed to effectively perform their job. when these barriers are recognized and addressed, staff will adjust to their changing work environment. when these needs are not met, staff will leave. the authors discussed these barriers during several presentations following the completion of the survey work. shortly, thereafter, one of the interview participants reached out to the authors and commented that if these barriers had been addressed at their library, they would have stayed in their current position. they have since left the library to pursue a different position at the university where their use of these kinds of technical skills is accepted and supported. 5.0 conclusion establishing a digital mindset encompasses much more than just learning a new skill to perform a task, it also involves understanding how technology fundamentally changes how and what we do. learning a new technical skill is woven into a broader organizational context that includes continuing education, administrative support, and evolving practice. leonardi and neeley (2022) observe that “new technologies directly facilitate change in the way people perform their tasks” (p.176). consequently, part of developing a digital mindset is being responsive to new technologies and being willing to make adjustments in how one works. the changing nature of technical services work in academic libraries illustrates this idea well. with increasing data pipelines and a plethora of systems supporting discovery, access, and management of thousands of library resources, scalable approaches to work are required. library staff are increasingly using tools like python to better manage this work. within this context of a changing environment, this study explored the use of python to assist with technical services work as well as the organizational support provided. interviews and survey data revealed several key insights. first and foremost, python has proven to be valuable for technical services tasks, especially for creating efficiencies and automating data workflows. the participants’ experiences demonstrated that python has become an essential tool to support a variety of work in recent years. python’s accessibility as a programming language makes it an ideal entry point for individuals without prior coding experience. a variety of positions in academic libraries within and outside of technical services are writing or using python scripts to support technical services work. interview participants noted that python-related work assignments always started with a specific problem, with python consistently emerging as one of numerous potential solutions, frequently complementing other skills like sql and apis. training for python is mostly informal and self-directed; support is provided on an as-needed basis, with a wealth of resources available to aid in learning the language, including ample examples within the library literature. the choice to use python is often a matter of personal preference. although this ad hoc approach was identified as preferable to formal training by many of the interview participants, this can be indicative of a lack of structure within organizations that formally acknowledges the importance of and need for this skill. this could be due to a variety of reasons: newness of using these skills, lack of awareness of changing work requirements, or rigidity in the organizational culture. however this has occurred, formal acknowledgment is needed to ensure that staff are consistently trained and adequately compensated for the skills that are needed on the job. the fact that several interview participants noted that programming/coding work was added to position descriptions in at least the preferred category is a possible indicator that formal acknowledgement is beginning to happen. in respect to barriers and concerns, it is also worth noting that certain organizations still maintain gatekeeping practices, dictating who is permitted to utilize python in their work. this is problematic when trying to create an organizational culture that embraces change and supports innovation. leonardi and neeley (2022) address the problematic nature of centralizing and gatekeeping experimental practices. they observe that centralized experimentation “mak[es] it difficult for the people closest to the products and the data to have the legitimacy to run their own experimentation” (p.160), which is critical to success. even if technical services staff are not the ones using python, it’s still important in establishing a digital mindset for staff to recognize when coding is an option to automate or improve their workflows. knowledge of programming languages helps staff to support their work at scale more directly. scalable practices are especially needed given the changing nature of technical services work and the decrease in technical services staff. experience in this space also allows technical services staff to better communicate with technical stakeholders in other parts of the library including system administrators and web team staff. so how might we formalize support for this kind of work within a technical services environment? this might look like adding coding and scripting skills as a core skill or competency for performing technical services work, creating infrastructure to support training and growth, providing equitable access to resources and systems, providing equitable pay for evolving skills required for the job, and providing time for learning. these strategies allow for staff to stretch and adjust to keep up with evolving demands on the job. this kind of support also provides the potential to enhance collaboration across library departments. the use of coding to support technical services work is commonplace as evidenced by the numerous examples of how-to-articles in the literature, but formalizing support for transitioning to and developing technical skills is less common. establishing a digital mindset is a critical piece of this skills building equation as it will help staff (and the organizations that support them) better understand how technology can evolve their work, especially as new technologies appear on the horizon. for instance, generative artificial intelligence (ai) has the potential to transform technical services work including assisting staff with writing code, automating repetitive tasks, improving metadata quality, and streamlining acquisitions. further investigation is required to explore the potential impacts of generative ai. however, library staff with a digital mindset will be better able to understand, adjust, and integrate these technologies into their work. references [1]leonardi, p., & neeley, t. 2022. developing a digital mindset: how to lead your organization into the age of data, algorithms, and ai. harvard business review [cited 2023 october10]. available from: https://www.hbs.edu/ris/publication%20files/developing%20a%20digital%20mindset_81f3f69d-e28d-483e-8d1e-ce0ee159c0bb.pdf. [2]adam, r. m. jr. 2018. overcoming disintermediation: a call for librarians to learn to use web service apis. library hi tech 36 (1):180-190. [3]madden, l. 2020. a new direct for library technical services: using metadata skills to improve user accessibility. serials review. [cited 2023 october 10]: 46 (2). available from: https://doi.org/10.1080/00987913.2020.1782648. [4]yadav, a. k. 2022. an evaluation of library and information science curricula and professional perspectives in india. international information & library review 54 (3): 242-254. [5]raju, j. 2017. information professional or it professional? the knowledge and skills required by academic librarians in the digital library environment. portal: libraries and the academy. [cited 2023 october 10]; 17(4): 739-757. available from: https://doi.org/10.1080/07317131.2013.785802. [6]read, a. & cox, a. 2020. underrated or overstated? the need for technological competencies in scholarly communication librarianship. the journal of academic librarianship, [cited 2023 october 10]: 46 (4). available from: https://doi.org/10.1016/j.acalib.2020.102155. [7]white, p. and powell, s. 2019. code-literacy for gis librarians: a discussion of languages, use cases, and competencies. journal of map & geography libraries 15:45–67. [8]frank, h. 2013. augmenting the cataloger’s bag of tricks: using marcedit, python, and pymarc for batch-processing marc records generated from the archivists’ toolkit. code4lib journal [internet]. [cited 2023 october 10]; 20. available from: http://journal.code4lib.org/articles/8336. [9]mitchell, e. 2013. trending tech services: programmatic tools and the implications of automation in the next generation of metadata. technical services quarterly. [cited 2023 october 10]; 30 (3): 296-310. available from: https://doi.org/10.1080/07317131.2013.785802. [10]walton d. introduction. in: thomsett-scott b, editor. the librarian’s introduction to languages: a lita guide. maryland: rowman & littlefield; 2016. p. 1-10. [cited 2023 october 13]. available from: https://books.google.com/books?hl=en&lr=&id=q9lldaaaqbaj&oi=fnd&pg=pr7&dq=python+pymarc&ots=g4sb91aqdy&sig=oxevqzgsuvw8vbujn5lfqkcqra0#v=onepage&q=python%20pymarc&f=false. [11]magnuson, l. 2016. do library stuff faster with python. techconnect. available from: https://acrl.ala.org/techconnect/post/do-library-stuff-faster-with-python/. [12]pymarc 5.1.0 [internet]. 2023. python software foundation; [cited 2023 october 13]. available from: https://pypi.org/project/pymarc/. [13]hill, c.e., frank, h., pernotto, m. python. in: thomsett-scott b, editor. the librarian’s introduction to languages: a lita guide. maryland: rowman & littlefield; 2016. p. 11-26. [cited 2023 october 13]. available from: https://books.google.com/books?hl=en&lr=&id=q9lldaaaqbaj&oi=fnd&pg=pr7&dq=python+pymarc&ots=g4sb91aqdy&sig=oxevqzgsuvw8vbujn5lfqkcqra0#v=onepage&q=python%20pymarc&f=false. [14]yelton, a. 2015. chapter 6: learning to code. in: library technology reports 51(3). ala techsource. [cited on 2023 october 13]. available from: https://journals.ala.org/index.php/ltr/issue/view/506. [15]thompson, k. and traill, s. 2017. leveraging python to improve ebook metadata selection, ingest, and management. code4lib journal [internet]. [cited 2023 october 10]; 38. available from: https://journal.code4lib.org/articles/12828. [16]dong, e., glerum, m. a., & fenichel, e. 2017. notes on operations, using automation and batch processing to remediate duplicate series data in a shared bibliographic catalog. libraries resources & technical services. [cited 2023 october 10]: 61 (3). available from: https://journals.ala.org/index.php/lrts/article/view/6395/8442. [17]enis, m. 2013. cracking the code. library journal 138 (4). [18]mcdavid, s. r., mcdavid, e., das, n. e. 2021. leveraging a custom python script to scrape subject headings for journals. code4lib journal [internet]. [cited 2023 october 10]; 52. available from: https://journal.code4lib.org/articles/16080. [19]frazier, k. 2020. automated collections workflows in gobi: using python to scrape for purchase options. code4lib journal [internet]. [cited 2023 october 10]; 49. available from:https://journal.code4lib.org/articles/15334. [20]zavalin, v. i., & miksa, s. d. 2021. collecting and evaluating large volumes of bibliographic metadata aggregated in the worldcat database: a proposed methodology to overcome challenges. the electronic library. [cited 2023 october 13]; 39(3):486-503. available from: https://www.emerald.com/insight/content/doi/10.1108/el-11-2020-0316/full/html. [21]harrington, p. 2019. generating geographic terms for streaming videos using python: a comparative analysis. code4lib journal [internet]. [cited 2023 october 10]; 49. available from: https://journal.code4lib.org/articles/14676. [22]davis, k.k. 2020. leveraging the rbms/bsc latin place names file with python. code4lib journal [internet]. [cited 2023 october 10]; 48. available from: https://journal.code4lib.org/articles/15143. about the authors maria collins is the head of acquisition and discovery at north carolina state university libraries. she is a former editor-in-chief for the journal serials review. her research interests include management practices in technical services and electronic resources management (erm) systems. xiaoyan song previously served as the electronic resources librarian at north carolina state university. since july 2023, she transitioned to duke university libraries, assuming the role of electronic resources acquisitions and licensing librarian. with a passion for enhancing productivity, she leverages tools and programming languages like python to optimize work processes. sherri schon received her master’s degree in library and information science from simmons university in 2021. she currently works in advancement services at hampshire college.   appendix a:  python questionnaire questions sent to code4lib have you or someone on your staff ever used python to complete any library technical services work? if so, how long have you or your staff been using python? less than a year 1 year – 5 years more than 5 years how is the work managed? for example, who writes the code and who uses the script, etc.? (no need to list any names, just job title or position) what kinds of tasks are accomplished with python? how often do you use python in your work? at least once a day. at least once a week. at least once a month. less than once a month. do you use other programs, apis, or programming languages in conjunction with python? if so, what are they? how long have you been using them? have you participated in any training (formal or informal) with python scripting? if so, can you describe the training? if you have not used python for your work yet, are you planning to use it in the future? for what purposes? are you willing to be contacted for further discussion about python use in your library? (if so, please provide your email address below).   appendix b: python interview questions introduction thank you for taking the time to share your python experience. we will start with the purpose of the project, which is to find out how python is used in academic libraries to support technical services work. it’s for our own education to see what we can do in our own department and may potentially write up for publications. we also want to make it clear that the interview will not be recorded and that you have the option to skip any question. your name, institution and email address will not be used in any of the written or presented reports. organizational structure discussion of position: what is the name of your position title? what department are you in? what kind of work do you typically do (system, metadata, cataloging, ir support)? what is the history of doing python-related work in your current department? if your position is not in a traditional technical services department, describe your intersection with positions in technical services and the work that is performed there. is your organization considering transforming traditional technical services positions into more systems-focused technical positions? are python skills listed as a formal part of your job description? would your department continue to hire someone with python skills or is the use of these skills more a personal preference training/education how did you acquire the skills needed for python work? have you done any formal or informal training? if you acquired your training organically through self-teaching, do you think people would benefit from a more formal training approach? are you responsible for training others in your department on python-related work? can you think of skills or experience that would be important to have before learning or beginning python-related work (prerequisites)? experience how is python work managed or assigned in your department? can you talk more about your experience with python that you mentioned in the survey, especially the technical services-related work? what other kinds of tasks do you hope to address or approach using python-related skills? how do you manage the scripts that you and your colleagues have written (github, internal documentation system)? is there anything else from your experience working with python that we should know? wrap-up potential follow up questions for us thanks again for sharing your python experience with us!   appendix c. python questionnaire sent to other listservs have you or someone on your staff ever used python to complete any library technical service work?  if your answer is yes, please proceed with the remaining questions. if your answer is no, click next and you’ll be asked to submit the survey. what kinds of task do you use python for? please select all that apply. can you provide more details about the tasks you perform with python? how long have you or your staff been using python? are you writing your own python scripts or someone else writes for you? how often do you use python in your work? have you participated in any training (formal or informal) with python scripting? select all that apply. are there any python workshop or courses you’d like to recommend to others? if so, please share them here. in what ways has your organization supported you in learning more about python/coding to improve or enhance your work? what’s your role at your institution? is there anything else you’d like to share about your python use?   appendix d. types of work supported with python circulation batch searching against gobi for lost items to have them replaced send out overdue notice use pymarc, come up with a local tag 912 to store circ info from old to the new ils clean up user records marc records work use pymarc combining the oclc metadata api to clean up bib records and then have them ready for import bulk edit a marc field across all records in a manner that isn’t supported by local ils bulk record modification function cataloging copy look up program – batch search cataloging copy in worldcat based on isbn, lccn, and oclc number. marc record generator for internet archives records, books clean up call numbers after migration to a new ils batch adding of a list of urls from a spreadsheet as 856 tags to the corresponding bib records. batch encoding holdings information stored in a csv file in a particular ils marc holdings format for respective bib records. retrieving and processing marc files from oclc collection manager, marcive for government documents, and marcive authority work processing and submitting marc output files from sierra to ebsco for discovery service on-going bibliographic maintenance, fixing errors via sierra api, identifying duplicate or merged records submitting serial holdings from sierra to oclc via data sync acquisition work batch update renewal dates and renewal periods for po lines transform data from one format to another from ead xml to human readable pdfs from tiff to pdf and apply ocr support ir work check bad links make global changes system interaction and integration via apis pulling data from different systems or sources calling apis to grab and parse data writing plugin modules to integrate with folio ils reporting a report on how many folks are in google scholar, used to maintain databases on course reserve system; number of citations and publications in combination with scopus api and python report on new items, high demand holds, lost and paid, missing items support language translation convert regional language to a standard script apply ocr for regional languages other data analysis analyzing order records, import logs parsing mod xml files or spreadsheets, csv files oa content evaluation other automation file renaming inventory of born-digital content scripting to enable students, faculty, and staff to play legacy computer games   appendix e. python resources recommended by participants https://docs.google.com/document/d/1bvzlrr34-33svrxbt6fe37kddgo7i3bsv1ncyryje2a/edit?usp=sharing   subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – identifying frbr work-level data in marc bibliographic records for manifestations of moving images mission editorial committee process and structure code4lib issue 5, 2008-12-15 identifying frbr work-level data in marc bibliographic records for manifestations of moving images the library metadata community is dealing with the challenge of implementing the conceptual model, functional requirements for bibliographic records (frbr). in response, the online audiovisual catalogers (olac) created a task force to study the issues related to creating and using frbr-based work-level records for moving images. this article presents one part of the task force’s work: it looks at the feasibility of creating provisional frbr work-level records for moving images by extracting data from existing manifestation-level bibliographic records. using a sample of 941 marc records, a subgroup of the task force conducted a pilot project to look at five characteristics of moving image works. here they discuss their methodology; analysis; selected results for two elements, original date (year) and director name; and conclude with some suggested changes to marc coding and current cataloging policy. by kelley mcgrath and lynne bisko introduction in 1998, the international federation of library associations and institutions (ifla) issued a report on the functional requirements for bibliographic records (frbr) [1] that presents a conceptual, entity-relationship model of the information represented in existing library catalog records for bibliographic objects, such as books or videos. the frbr report discusses four bibliographic entities ranging from the most abstract to most concrete: work, expression, manifestation and item. these terms are defined as follows: work: a distinct intellectual or artistic creation expression: the intellectual or artistic realization of a work manifestation: the physical embodiment of an expression of a work item: a single exemplar of a manifestation (frbr report, p. 13 [1]) current standard marc bibliographic records [2] generally describe a specific manifestation. however, they often include information about the expression or work mixed in with manifestation-level information. current marc authority records generally represent works or expressions, but tend to include only enough information to uniquely identify the work or expression rather than a more complete description. existing frbr-inspired implementations consist mostly of what jonathan rochkind calls “grouping records into work sets” or “collocating records for the work” [3] rather than a complete implementation of the frbr model. presentation of work sets is useful in that most users are probably primarily interested in finding works and then identifying specific expressions or manifestations that are accessible or desirable to them. however, work sets fail to realize the full potential of the frbr model. if separate work records were created and maintained, more accurate information could be explicitly recorded rather than whatever happens to surface from automated clustering, such as that used by worldcat local. creating and maintaining work-level records is also more likely to reap economic benefits as this information can be recorded and maintained in only one place rather than reproduced in multiple manifestation-level records. in the case of moving images, explicit work-level records are likely to provide two main benefits: since film and video are often re-issued, there are economic and efficiency incentives for making it easy to re-use this data, especially as there is often extensive information common to all versions of a moving image. library catalogs currently do a poor job of providing consistent and useful access to much of the subset of information common to all versions of a given moving image (e.g., original language, country of production, date of original release or broadcast), despite the fact that users often express interest in this information [4], [5] and this type of information tends to be given prominently in reference sources for moving images. currently we do not have work-level records for moving images, except for a relatively small number of uniform title authority records, which usually contain only title information. however, information about moving image works is often embedded in our manifestation-level bibliographic records. if we wish to move to an environment where we are creating and sharing work-level records for moving images, it would be helpful if we could use automated means to extract data from existing bibliographic records to populate provisional work-level records. these provisional records could later be verified, corrected, and enhanced by human review. in early 2008, the cataloging policy committee (capc) of online audiovisual catalogers (olac) created a task force to investigate and make recommendations on issues related to frbr-based work-level records for moving image materials [6]. one part of the task force was charged with identifying places in marc manifestation-level bibliographic records where work-level information may be encoded and examining a sample of marc records to see how reliably this information might be extrapolated from data in existing records. we also examined whether there are better ways to embed this work-level information in marc bibliographic records for easier automated extraction or updating. we conducted a pilot project to look at five characteristics of moving image works. in this paper, we will discuss our methodology, analysis and selected results for two elements, original date (year) and director name, and conclude with some suggested changes to current cataloging policy. some fields that we examined are omitted here as the lessons learned are the same or very similar to those derived from the fields we do discuss. a detailed, field-by-field report of our analysis will be available on the olac website [6]. first, we will discuss one possible scenario for using the work-level data extracted by these methods. one possible scenario for work-level records for moving images before discussing how we attempted to identify and extract work-level data from existing records, we would like to briefly discuss one possible scenario for using work-level records populated with extracted data. one possible approach to moving image cataloging is to record the reusable data in one record (what we refer to here as a work-level record and discussed in the olac task force’s report, parts 1-2 [6], as a work/primary expression record), the manifestation-specific data in machine-comprehensible form in another record, and to link the two (or for more traditional systems, to merge them in some form; if this data is machine-analyzable, the parts that don’t vary from the original could easily be suppressed). most of the time, it is unclear that explicit expression-level records offer any advantages. the exception is what might be called “named” expressions, e.g., director’s cut or unrated versions, which cannot be reduced to exhaustive, controlled vocabularies and may require cross-references that cannot be anticipated prior to the creation of additional manifestations. it would be more practical to record most characteristics that may vary at the expression-level (e.g., color, duration, language access) in machine-readable form in the manifestation-level record and program the computer interface to offer this information as navigation options. in particular, for moving images in which given expressions tend to be multifaceted, it probably is not time-saving to try to locate or create an expression-level record that reflects a specific combination of options. below we give an example of how this combination of workand manifestation-level records could be presented to an end user. this is not intended to be a comprehensive example nor an ideal display, but merely to present a possible idea. limiters (from manifestation-level records) work available at: ball state university libraries muncie public library format: dvd blu-ray vhs spoken language: english spanish french chinese subtitle/caption language: english spanish thai accessibility options: audio-described captioned aspect: fullscreen widescreen (1.37 : 1) color: black and white colorized publisher/distributor: warner home video special features: commentary track carrotblanca (cartoon starring bugs bunny) who holds tomorrow? (1955 television adaptation of casablanca) casablanca (screen guild players radio production) outtakes gallery title: casablanca date: 1942 director: curtiz, michael, 1888-1962. producer: wallis, hal b., 1899-1986. writers: epstein, julius j., 1909; epstein, philip g. ; koch, howard. production company: warner bros. pictures cast: bogart, humphrey, 1899-1957 ; bergman, ingrid, 1915-1982 [additional creators and contributors could be included] summary: during world war ii, rick’s café in casablanca is a center for war refugees awaiting visas for america. rick abandons his cynicism to help an old love escape the nazis with her underground leader husband. genre: drama ; romance ; adaptation setting: casablanca, morocco, africa time period: world war ii language: english country of production: united states run time: 102 min. color: black and white sound: mono. aspect ratio: 1.37 : 1 awards: academy award (best picture ; best director ; best writing, screenplay) based on: everyone comes to rick’s (play) authors of play: burnett, murray ; alison, joan if the data in the work-level display on the right were recorded in a separate record, mechanisms currently exist to extract most of the data on the left from related marc bibliographic records, assuming full and accurate records. the notable exceptions are that there is no reliable way to extract aspect ratio or special features in the form given here. missing or mistaken data will have some impact on implementation, but could be improved retrospectively. although it seems desirable to many to store data for bibliographic materials in a multi-record, frbr-based structure, the transition by the diverse and under-funded library world to a new structure is likely to be difficult and to proceed at different paces in different institutions. creation of work-based records that can be linked to and used both with existing manifestation records and future, leaner manifestation records created in a more robust model would provide one way of easing this transition. methodology overview we identified a representative sample of work-level information for moving images and used our knowledge of cataloging rules and practices to identify all possible fields and subfields where this information might occur in marc records. we then evaluated possible marc data sources, based on how commonly they are used and how amenable they are to reliable automatic extraction, and selected the most promising for processing. in order to test the usefulness of our selected data sources, we acquired from a variety of types of institutions a sample of marc bibliographic records that describe a range of moving images, including features, television programs, and nonfiction films. we extracted from these marc records the fields and subfields from which we wished to extract data, as well as those deemed useful for evaluating the accuracy of the extracted data. we wrote brief programs and queries to automatically extract the values of interest and then manually reviewed the results. the manual review was useful in that it allowed us to identify patterns of problems. this will enable us to improve future iterations of our program and also possibly to proactively identify records that are more likely to need manual intervention. the manual review also allowed us to make more accurate assessments of the relative usefulness and reliability of data from the various sources. our analysis has enabled us to suggest two types of improvements to enhance our ability to more effectively record and identify this type of data in the future. the first is to recommend the use of specific cataloging practices that are possible in the current infrastructure and that would support the machine-manipulable recording of data in which we are interested. the second is that, when we have identified areas where it is not possible to record useful data in machine-manipulable form, we can create proposals to expand the marc format to support this type of data input. location of data in marc records we selected the fields from which to extract data based on the estimated accuracy of the data for our purposes and our perception of how often these fields are used. we decided to limit our data sources to those that have a high probability of containing correct data in a form that can be extracted without manual review. selection of records for sample testing we obtained a sample of 941 marc records from six institutions, primarily via z39.50. these included records from a public library, two medium-sized academic libraries, two large academic libraries and a film archive, all of whom do at least some local editing of their records. the records we selected included well-known feature films likely to have multiple releases, non-english language films in languages using both roman and non-roman alphabets, and a general keyword search (“sleep”) intended to retrieve a variety of records, including nonfiction, television shows and features. processing and review of sample records once we obtained the records, we used marcedit, a free windows-based marc editing tool [7], to export the relevant data in tab-delimited form. marcedit lets users specify data for export at the field and subfield level. we exported the fields/subfields we wanted to extract data from, as well as any fields/subfields that we thought necessary to assess the results of automated processing. we imported the data into microsoft access and used queries and visual basic for applications (vba) to manipulate it. we used access primarily because one of our group members was familiar with it and it seemed likely to suffice for our proof-of-concept project. we normalized the data by removing diacritics, most punctuation and some subfield coding. we retained punctuation that we intended to use, such as semi-colons in statements of responsibility. we used queries and text processing to extract the relevant data. this process is described in more detail in the individual review sections. following this, we reviewed our results manually to determine if the information had been correctly extracted and to identify any patterns of problems. at this point, we have only been able to examine whether or not the data was correctly extracted. in the future, we plan to assess at least a subset of our data against external sources for accuracy. other issues we do not believe that we can accurately extract data from multi-work records (e.g., records for a set of all the james bond movies or a collection of animated shorts). the various pieces of information that pertain to the individual works in a multi-work marc record are not linked in any way, so it is impossible for a machine to identify, for example, which titles go with which dates or genres. it might be possible, once we have a set of provisional work-level records, to identify which works are contained in a given manifestation by matching information in the provisional work-level records to information in the manifestation records. this is an area that will require more manual intervention. we did attempt to see how accurately we can identify the multi-work records in our dataset by looking for the presence of things like non-collective titles and analytical titles. we were able to identify almost all of the multi-work records through the presence of information such as contents note (505) fields in the record, but we did have a fairly high level of false drops (31%). based on manual review, 79% of our records represent single works and an additional 6% are records for a main work that mention subsidiary work(s) not likely to interfere with extraction of data about the main work. we are not sure what the threshold should be for reasonable reliability of this information. it is clear that information derived from manifestation-level bibliographic records will be incomplete and at times incorrect so we will eventually have to decide on an acceptable level of accuracy. for works that have been issued in many versions, our results may be improved with clustering of manifestation-level records for the same work. analysis of individual characteristics: original date fields and areas of the marc record examined we attempted to extract the original date from existing marc bibliographic records for moving images via a number of methods, including: date2 008 (part of marc 008 control field). when present in the record, this date is the most reliable method of determining the original date for moving image works. for many videos, “type of date/publication status” is coded “p” for “date of distribution/release/issue and production/recording session when different,” the original motion picture date is given in date2 and the publication date of the video is given in date1. date/time and place of an event (033). this field includes a formatted date/time of creation, capture or broadcast associated with an event in the form yyyymmddhhmm+-hmm. it seems to be more commonly used by archives. uniform title (130). the original date is sometimes found here when needed to distinguish between two moving images with the same title. general note (500). these notes were parsed to look for years in 18xx, 19xx, or 20xx format in combination with a limited set of keywords (e.g., broadcast, produced) that often indicate that the note refers to the original date of the work. we also examined data from the date/time and place of an event note (518) field, but the issues we encountered are similar to those for the general note (500) field so we have not discussed it here. the original date may exist in other fields in the record, but we deemed the ones we examined to be the most likely sources for reliable information about the original date. the most common place the original date may be found, other than those described above, is in date1 of the 008 field. however, we did not include date1 in our project because there is no automated means to distinguish between the following scenarios: the date of publication of the video and the date of the work are the same so there is only one date to put in the fixed fields and it is in date1 the date in date1 is the date of publication of the video and there is no date in date2 because: a. the cataloger forgot or chose not to do the research to determine the original date b. the cataloger is following newer policies in which changes or additions (e.g., subtitle tracks, making-of featurettes) to the content of the original moving image work make the dvd a new publication with a single date. the publication date subfield of the publication, distribution, etc. (260$c) field has similar problems. process we created queries in access to identify records with data in the fields/subfields we intended to examine. in some cases, data identification was fairly straightforward. for example, date2 is part of the marc 008 field (fixed-length data elements, general information) in which meaning is determined by the character’s position in the string. in the case of date2, we identified a desirable date type in the 06 position in 008 and then extracted the date in positions 11-14. we recorded the extracted data in a separate access field using field names such as marc500 for the original general note (500) field data and date500 for the year(s) extracted from the general note (500) field. the process for the date/time and place of an event (033) field was similar to that for date2 of the 008 except that when multiple dates existed we used subfield coding to identify individual date strings and then extracted each year separately. in the case of the uniform title (130) field for moving images, the date is unfortunately not specified by a separate subfield code. we relied on a standard citation format (“title (motion picture/television program : [date] : [director’s last name])”) that assumed a qualifier of “motion picture” or “television program” and the use of a colon before the date. as it turned out, most, but not all, moving image uniform titles include a qualifier and some included non-standard qualifiers, such as cartoon. we would have been better off removing the standard qualifiers and using the opening parenthesis to identify the location of the date. original dates are also often given in free-text form in the general note (500) field. we used vba to loop through each note field. although records may contain multiple general note (500) fields, we were able to evaluate each note separately because fields exported from marcedit are in tab-delimited form with a semi-colon separating each instance of a given field. we first used the instr function to determine if any of our keywords (e.g., produced, release) existed. we then looked for a year or years in 18xx, 19xx, or 20xx format and recorded all that we found. before beginning this process, we also removed the letter c from in front of any five-character sequence that appeared to represent a year. analysis we examined our sample of 941 records from six sources. at this point we have only looked at whether we can extract dates that might be the original date. we have not assessed the extent to which these dates actually do represent the correct original date. we found that 72% of the records had some date that potentially could be identified as the original date, while 28% did not contain any information that we could leverage. some adjustments to our vba extraction program would improve our results slightly. however, about one quarter of the records would still not contain information useful for automatic extrapolation of an original date, as these records include no identifiable dates in any of the fields we examined. the two methods that worked best for extracting potential original dates were the 008 date2 field (present in 41% of records) and the general note (500) field (present in 39% of records). the other methods, date/time and place of an event (033) and uniform title (130), were each present in less than 10% of the records and these two fields were disproportionately represented in records from the film archive, which may indicate a difference between archival and standard library cataloging. original date overview 008date2 general note (500) date/time and place of an event (033) uniform title (130) overall any date blank field or no identifiable date in field 556 407 829 846 265 28% correctly-identified data 385 368 89 57 676 72% multiple dates 0 137 23 17 0 missing keyword associated with presence of date (e.g., “produced”) 0 29 0 21 0 minimum presence of data** 30% 0% 16% 0% 53% maximum presence of data** 81% 26% 70% 6% 91% ** minimum and maximum show variations in the availability of data by institution. that is, the number of records that contained useful data in 008date2 ranged from 30% in the institution with the lowest use of this field to 81% in the institution with the highest use. these variations can reflect differences in the types of material collected, but also show the effects of local cataloging practices on the availability of data. some particular problems encountered in our data sample: some general note (500) fields in our record set refer to the date associated with an external verification source, such as the publication year of the american film institute catalog or the date the cataloger checked the internet movie database. our program cannot distinguish between these dates and relevant dates and incorrectly uses the verification date as the original date. this could be resolved in many cases by having a hierarchy of date sources and preferring the more reliable fields. records in which general note (500) fields contains multiple dates; one is the release date, but the earliest date refers to an event other than the release. for example, one record included the note “based on the novel dracula by bram stoker first published in london in 1897.” different or inconsistent dates in the date/time and place of an event (033) and uniform title (130) fields for the same video. for example, a record may contain a uniform title of “simpsons (television program : 1989),” qualified by the date the show began airing, as well as a date/time and place of an event (033) field of 19920507 that represents the date of a particular episode. incorrect cataloging practice for the 008 date1 and date2 fields, in which the dates were reversed so that the original date was in date1 and the manifestation date was in date2. date1 is supposed to contain the publication date of the manifestation in hand and date2 may contain the original release date under certain circumstances. so, this example is a non-standard use of marc coding often employed to achieve a desired end in opac displays, i.e., sorting by original release rather than publication date. or it could be that the cataloger did not know how to correctly code these fields. keywords that signal dates in general note (500) fields that were not included in our original program, e.g., “filmed,” “copyright,” “recorded.” in the uniform title (130) field, we also missed dates in titles that did not include the phrase “motion picture” or “television program,” but our program could be revised to pick up those dates. recommendations there should be a field in the marc record that specifically contains the original date of a moving image work. it is probably sufficient to record the year, but may be useful to include an option for recording exact dates, particularly for episodes of television programs. perhaps the formatted date/time and place of an event (033) field could be expanded to incorporate this use. analysis of individual characteristics: director fields and areas of the marc record examined our goal was the correct identification of an added entry-personal name (700) field containing the authorized, standardized form of the director’s name. it is possible that the director’s name might occur in a main entry-personal name (100) field, but this is relatively rare and we did not account for this possibility in our sample. director can also be traced in the added entry-corporate name (710) field. we found this type of added entry in the case of the director team the brothers quay in our sample. we attempted to extract the director’s name from existing marc bibliographic records for moving images via a number of methods, including: statement of responsibility, etc. (245$c). many records contain a transcribed statement of responsibility that includes the director’s name and function, usually as they appear on the title frames. moving images often list multiple functions in the statement of responsibility, with each distinct function separated by specific punctuation, i.e., space-semicolon-space. we used this prescribed punctuation to parse each function and attempt to match it with its associated authority-controlled name entry. added entry-personal name (700) with marc relator code ($4) subfield of drt. some added entry-personal name (700) fields include a marc relator code of “drt” in 700 $4 identifying that person as the director. we also examined data in the creation/production credits note (508) field and the relator term (700$e) subfield. these results are not discussed here. the director’s name may exist in other places in the record, such as in contents note (505) fields in multi-work records, but we chose to focus on the most commonly-occurring instances. process we created queries to identify which records had data in each of the fields/subfields we intended to examine. in the case of the added entry-personal name (700) field that included a relator code ($4) subfield, the identification of the authorized form of the director’s name was simple. we just extracted the names that included the string “$4drt”. for the free text statement of responsibility, etc. (245$c) subfield we split out each separate statement of function using the existing semi-colons. we identified each statement of function that included the letter sequence “direct” to pick up variations such as “director,” “directed, “direction,” etc. we did not attempt to account for non-english terms for director or directing in our test run nor did we attempt to distinguish other types of directors, such as directors of photography. many libraries do not trace the other types of directors so there often is not a matching added entry-personal name (700) field in the record, which cuts down on the number of false drops. it is unlikely that we can achieve 100% accuracy discriminating between primary directors and other types of directors and directing functions in practice. within each statement of responsibility, we then identified the individual words. since we had no way to automatically identify names as opposed to other types of information, we processed all words occurring in a given directing function statement. we attempted to match from the statement of responsibility, etc. (245$c) subfield either 1) two consecutive words or 2) two words separated by a single word with words occurring in an added entry-personal name (700) field. the latter helped with transcribed names with middle initials in the statement of responsibility, but not in the matching authorized form in the 700 field. we looped through all added entry-personal name (700) fields in each record and looked for the presence of the two words identified in the statement of responsibility, etc. (245$c) in the added entry-personal name (700) field. for example, if the credits said “directed by clint eastwood,” we used the instr function to look independently for each pair of words (e.g., “directed by,” “directed clint,” “by clint,” etc.) , in all added entry-personal name (700) fields. the words did not have to occur consecutively or in any particular order in the added entry-personal name (700) field. on the whole, this method worked well, but did lead to a few false hits. these generally involved names with initials, which more sophisticated programming could eliminate. this does not work with one-word names, but those are relatively rare. if we did not use the two-word minimum, we would have had a lot more false matches. analysis as with the original date, we examined 941 records from six sources. we found that we could identify at least one added entry-personal name (700) field representing a director in 62% of the records. the vast majority of these (84%) were derived by matching the statement of responsibility, etc. (245$c) subfield with an added entry-personal name (700) field. use of the relator code ($4) identified directors in about 15% of the records. the use of relator code ($4) subfields varied widely among institutions and ranged between 0-83% for a given institution. this reflects the impact of local cataloging practices on the usability of data for our purposes. most of the directors identified by relator code ($4) were also identified by matching added entry-personal name (700) fields with statement of responsibility, etc. (245$c) subfields, but the use of relator codes has the advantage of eliminating the hard matching problems (e.g., accounting for foreign language terms for director and variations in spelling, transliteration and form of name). on the other hand, a quarter of the records did not include identifiable director information in the fields we examined. another 10% lacked an added entry-personal name (700) field that corresponded to the director(s) identified in the statement of responsibility, etc. (245$c) subfield. less than 10% of the records with no director information included director in a contents note (505) field. the remainder either had no director information, used a different form (e.g., “a film by…”) or the cataloger omitted that information. more sophisticated programming could improve the match rate on director information found in the statement of responsibility, etc. (245$c) subfield. for example, thirty names (3%) in the statement of responsibility, etc. (245$c) subfield failed to match because we did not look for non-english director functions such as “regie” or “kantoku.” however, accounting for all variations would be time-consuming vis-à-vis the number of affected records. this problem is somewhat mitigated by the fact that not all libraries transcribe original language credits; many prefer to use english language credits from another source. some names failed to match because of variations in spelling or transliteration between the transcribed and authorized forms (e.g., “pierre schoendorffer” vs. “schoendoerffer, pierre” and “andrei tarkovsky” vs. “tarkovskii, andrei arsenevich”). in some cases the name was traced under a different form entirely (e.g., “t. c. frank” vs. “laughlin, tom”). some match failures could be resolved by using both the official added entry-personal name (700) field form of name and the forms of name in the cross-references in the relevant authority record. director overview statement of responsibility, etc. (245$c) added entry-personal name with relator code (700$4) overall overall% blank field or no identifiable relevant information 492 797 237 25% correctly-identified data 310 144 584 62% problem with matching algorithm and initials; fixable with better programming 4 0 3 <1% director is corporate body (710) 1 0 2 <1% no matching authorized name (700) for transcribed name 84 0 90 10% non-english term for director 30 0 9 1% difference in spelling or transliteration between transcribed and authorized forms of name 16 0 11 <1% other difference between transcribed and authorized form of name (e.g., use of variant names or pseudonyms) 4 0 4 <1% minimum presence of data** 44% 0% maximum presence of data** 69% 83% ** minimum and maximum show variations in the availability of data by institution. that is, the number of records that contained useable data in the statement of responsibility, etc. (245$c) subfield ranged from 44% in the institution with the lowest rate of success to 69% in this institution with the highest rate of success. these variations can reflect differences in the types of material collected, but also show the effects of local cataloging practices on the availability of data. recommendations although the matching algorithm found authorized names in added entry-personal name (700) fields for most directors transcribed in corresponding statement of responsibility, etc. (245$c) subfields, a certain number of matches will inevitably be missed due to variations in form of name or non-english terms for director. accuracy is still unlikely to reach 100%, even if we take into account authority record cross-references and include additional non-english director keywords. the automated extraction will produce “good enough” work-level records which provide a starting place for developing these records from existing metadata. the process of matching transcribed and authorized forms after the fact is inherently more complex than indicating during cataloging that this particular authorized form accurately identifies the director. the use of the marc relator code ($4) subfield (or, alternatively, the marc relator term subfield, $e) is more reliable and amenable to machine-based processing than even the most sophisticated matching algorithm and it is recommended that one of these options be used whenever possible. this is particularly useful for moving image records, which usually record a variety of functions. conclusion it would be desirable to be able to easily extract work-level information from existing marc bibliographic records. it would also be useful, so long as current marc bibliographic records are used, to be able to automatically insert previously verified work-level information into a new marc bibliographic manifestation record or to update existing marc manifestation records with corrected or expanded work-level information. the logistical challenges associated with moving from our current environment to one utilizing related frbr-based records are significant. the creation of work-level records that can be linked to existing manifestation-level records is a practical first step in this process that will also have worthwhile benefits for libraries and their users. it is also likely that institutions will follow different migration timetables to what rda (resource description and access, the upcoming new cataloging rules) calls a “relational/object-oriented database structure,” [8] requiring a need for parallel implementations over a long time period. if the marc bibliographic format is updated to contain more machine-identifiable information about the work record, it is possible that these traditional manifestation-level bibliographic records could be updated with new information from collectively-maintained work records. institutions using related work and expression and/or manifestation records would not need to repeat the redundant data and could rely on the shared collection of work-level records. to do this, it is necessary to be able to easily and accurately identify the location of work-level data in the marc bibliographic record. our preliminary assessment of a sample of records suggests that varying amounts of work-level data can be extracted from existing marc bibliographic records. about 20% of our sample consisted of multi-work records, from which we are unlikely to automatically extract work-level data due to the difficulties of matching disparate pieces of information with the correct work. we extracted potential work-level data for original date in 72% and for director in 62% of the records we examined. it is likely that we could improve our success rate by looking at patterns in clusters of records for the same work. also, more sophisticated programming would help with better data extraction. we have identified some cataloging practices that can be implemented in current marc bibliographic records that will ensure that work-level data is available for machine processing, such as using relator codes or terms after the controlled form of names (e.g., marc fields 100 and 700) whenever possible. there are also ways in which the marc bibliographic format could be modified to enable machine-readable encoding of data for elements that currently do not have such fields. for example, a field could be created in the marc bibliographic record where the original date of a moving image work could be unambiguously recorded. it is unlikely that complete, accurate work-level records could reliably be derived from existing marc bibliographic records, but it is possible that “good-enough” provisional records could be created and then revised and manually upgraded. we think this approach bears further investigation and testing. olac has produced a report providing a practical definition of a moving image work and a list of relevant data elements [6]. we are currently in the process of creating operational definitions and a list of information sources for five representative data elements, as well as completing the analysis of the location and potential extractability of these five elements from existing marc bibliographic records. this paper is based on that analysis. if this work is deemed useful, we plan to expand it to include operational definitions, sources, and marc-based analysis for additional data elements. we hope to eventually construct a publicly-available database of moving image work records based on and linked to existing marc bibliographic records which will enable the creation, correction, maintenance and enhancement of work-level data at a collective level. this should also enable the development of an interface that allows users to search for work-level characteristics and to identify and obtain particular expressions or manifestations that they would like to use. notes [1] the original 1998 and current versions of the frbr report are available at http://www.ifla.org/vii/s13/frbr/. quotes are from the pdf version of the current 2008 text. [2] detailed information about the fields and subfields in the marc bibliographic format is available at http://www.loc.gov/marc/bibliographic/ecbdhome.html. [3] jonathan rochkind. “‘frbrization’ is not frbrization.” bibliographic wilderness, http://bibwild.wordpress.com/2008/07/31/frbrization-is-not-frbrization, 31 july 2008. [4] jeannette ho. “faculty and graduate student search patterns and perceptions of videos in the online catalog,” cataloging & classification quarterly 33.2 (2001): 69-88. (coins) [5] margaret hume. “searching for media in the online catalog: a qualitative study of media users.” mc journal 3.1 (1995): http://wings.buffalo.edu/publications/mcjrnl/v3n1/hume.html. [6] the olac capc moving image work-level records task force page with the task force’s reports is at http://www.olacinc.org/capc/movingimagework.html. [7] information about marcedit is available at http://oregonstate.edu/~reeset/marcedit/html/index.php. [8] tom delsey. “rda database implementation scenarios,” http://www.collectionscanada.gc.ca/jsc/docs/5editor2.pdf, 14 january, 2007. about the authors kelley mcgrath is cataloging & metadata services librarian (audiovisual) at ball state university. lynne bisko is non-print librarian at elon university. they are members of the online audiovisual catalogers (olac) moving image work-level records task force. they can be reached at kmcgrath at bsu dot edu and lbisko at elon dot edu. subscribe to comments: for this article | for all articles 6 responses to "identifying frbr work-level data in marc bibliographic records for manifestations of moving images" please leave a response below: kelley mcgrath, 2009-01-21 the nitty-gritty report on this is now available at http://www.olacinc.org/test/?q=node/28 (as part 4) kelley mcgrath, 2009-02-11 the correct current url for the olac reports is http://www.olacinc.org/drupal/?q=node/27 a regra do jogo « a imagem, o som, o tempo, 2012-01-11 […] kelley; bisko, lynne. identifying frbr work-level data in marc bibliographic records for manifestations of moving images. code4lib journal, [s.l.], n. 5, p. 12-15, […] re: [rda-l] cataloging matters no. 16 | my great wordpress blog, 2014-06-26 […] i am involved in a project that is trying, among other things, to retrospectively add role information to authorized names in records for moving image materials. we have an article in the code4lib journal about our preliminary test, which including figuring out which 700(s), if any, were for the director: http://journal.code4lib.org/articles/775.&nbsp; […] re: [rda-l] cataloging matters no. 16 | first thus, 2014-06-26 […] i am involved in a project that is trying, among other things, to retrospectively add role information to authorized names in records for moving image materials. we have an article in the code4lib journal about our preliminary test, which including figuring out which 700(s), if any, were for the director: http://journal.code4lib.org/articles/775.&nbsp; […] identifying frbr work-level data in marc bibliographic records for manifestations of moving images | documentação audiovisual, 2015-10-07 […] link para download: http://journal.code4lib.org/articles/775 […] leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – ref2ris: importing word-processed bibliographies into bibliographic management software mission editorial committee process and structure code4lib issue 16, 2012-02-03 ref2ris: importing word-processed bibliographies into bibliographic management software many who would benefit the most from timesaving bibliographic managers hesitate to adopt the technology due to the difficulties in importing legacy bibliographies developed over years. existing shortcuts rely on manual reformatting or on re-searching online databases for the records – often almost as laborious as retyping the references. ref2ris was developed to automate the task of converting a bibliography in specific citation styles from common word processing document formats into the widely used ris format. it uses the unix stream editor sed and the conversion options of apple’s textutil. it can be invoked as a series of simple shell commands on any linux terminal, or more simply as a drag-and-drop applescript application on macos 10.4 or higher. by deborah fitchett introduction bibliographic management software helps users download and organise large amounts of citation data and quickly format references and bibliographies that would take hours to write manually. academic libraries commonly encourage students and academics to use such software – the university of canterbury library, for example, provides particular support for endnote although some users instead use bibtex, zotero, or mendeley among others. other users have manually maintained (sometimes for decades) bibliographies in a typed word document, and are unwilling to make the shift to endnote if it means individually retyping or downloading citations for hundreds of references.  they need a user-friendly tool to convert their existing bibliography from a word processing format into a format that can be imported to bibliographic management software.  this article will look at some tools that partially fill this need and introduce ref2ris, including sample scripts and output. existing tools various approaches have been taken to importing legacy bibliographies into dedicated bibliographic management software. instructions are available to remap data from excel spreadsheets (how can i transfer… [2010]) or access databases (gear 2002).  while not novice-friendly, these processes are relatively quick and produce good results as the data comes pre-structured.  more complex is the process is to remap data from a text bibliography (garret 2008): this requires manual reformatting of each citation to separate elements of the citation with tabs and multiple authors with slashes, and may not save much time over simply retyping the data. parsing the citations is a second approach, essentially automating the remapping process. the challenge is to identify whether a given text string is an author, article/chapter title, or journal/book title, or a given number refers to a year, issue, or page number, since citation styles vary wildly in how they order and delimit these fields. erik hetzner’s method (hetzner 2008) uses hidden markov models to powerful effect, identifying fields within a citation regardless of citation style, but the software is more suitable for use in digital libraries than by casual users.  two other useful parsing tools work regardless of citation style, using conditional random fields, but these also suffer from the lack of a user-friendly interface. freecite (freecite) can work with either a single reference copy-and-pasted into a web interface or with multiple references through an api; in either case the xml output is not directly importable into bibliographic managers.  parscit (kan et al. 2004) is available for linux, macos and windows and outputs to ris format among others, but requires installation and knowledge of unix to use; its web interface is not reliably available for more than one reference at a time. more user-friendly is makebib (takaki [date unknown]), which converts to bibtex format.  besides a downloadable perl script, makebib has a web interface which accepts copy-and-pasted text containing multiple references separated by blank lines.  the sample conversions work well, but it’s not clear which citation styles are supported – testing it with apa produced garbled results and results from clinica chimica acta are cut off in the middle of the journal title. a third approach is to automate the process of looking the citations up in one or more online databases and download the bibliographic data from there.  lookup tools will usually include some parsing of citation data to facilitate this, but are less affected by variations in citation style as they need only identify key fields such as authors and titles. similarly, typos in input or errors in parsing are less likely to scramble results – as long as a result can be found, the data downloaded is clean. two tools that look up references in multiple databases are lemon8 (suhonos 2009) and paracite ([2004]).  lemon8 is an editing tool that converts from word processing formats into xml-based publishing layout formats, including citation editing facilities; it requires installation on a web server using apache, and being intended for use in journal publishing it doesn’t allow export of citation data to a bibliographic manager.  paracite consists of a web form that accepts a single reference at a time and searches for the article in various databases from which the user would need to download citation data manually. with other tools, apis may limit the number of queries that can be made, which can be a problem for a large bibliography.  two examples of such a limitation are the crossref simple text query (crossref [updated 2010]), which retrieves a doi for each reference searched, but requires registration and limits users to 1000 references per month; and wizfolio, which searches pubmed and can be used to convert to ris or bibtex (brekhus [updated 2010]), but is limited to 50 references per month. the most user-friendly lookup tool is hubmed citation finder which runs as a web form accepting copy-and-pasted text (transferring a word bibliography…[date unknown]) and converts to ris and bibtex formats among others.  some initial editing may be required to delimit records in the input, but the main limitation is that it searches only in pubmed, so is only suitable in some subject areas.  this is a general weakness with lookup tools:  even tools that search a large number of databases will perform poorly on bibliographies that contain significant amounts of older or local material, grey literature, or titles that remain unindexed for other reasons. outside of disciplines indexed by pubmed, then, there is a lack of text-based bibliography conversion tools suitable for students, academics, and library staff who are not expert computer users. ref2ris was developed to fill this need. ref2ris ref2ris takes the citation parsing approach, allowing it to work with citations regardless of where they might be indexed.  it is fundamentally a series of find/replace commands, searching on patterns specific to a given citation style and replacing with the appropriate ris tags (ris format…[updated 2011]) to create a new file that can be imported directly into most bibliographic managers. the core functions are packaged as an applescript application, letting the user either drag-and-drop the bibliography or bibliographies to be converted onto the ref2ris icon or run ref2ris and be prompted to select the file(s) to convert.  after testsystem() checks that all necessary components are available, the user is prompted to select the citation style which the bibliographies were written in, and convertbib() converts the bibliographies. finally cleanup() removes files created as intermediate stages of the conversion process, leaving only the original bibliography and the final ris file. ref2ris applescript (version 1.2) property thelocation : "" property existsbib : 0 set existsbib to 0 property theconverted : "nothing." if existsbib is equal to 0 then set allsources to (choose file with prompt "select the bibliographies to convert:" default location (path to documents folder) with multiple selections allowed) open allsources end if on open allsources testsystem() set existsbib to 1 set theconverted to "nothing." set thesed to (choose file with prompt "select the style to convert from (in ref2ris/conversionfiles/):" default location (thelocation)) repeat with thesource in allsources convertbib(thesource, thesed) end repeat display alert "successfully converted:" message theconverted buttons {"thank you!"} default button 1 end open on testsystem() try do shell script "echo no | sed s:no:yes:" on error display alert "cannot continue" message "a key component (sed) is not installed in your system (/bin/sh)." as warning buttons {"oh dear..."} default button 1 cancel button 1 end try try do shell script "textutil -help" on error display alert "cannot continue" message "a key component (textutil) is not installed in your system (/bin/sh)." as warning buttons {"oh dear..."} default button 1 cancel button 1 end try if thelocation is equal to "" then tell application "system events" to set thelocation to (posix path of container of (path to me) & "/conversionfiles/") end if try do shell script "/bin/ls " & quoted form of thelocation on error tell application "system events" to set thelocation to (posix path of container of (path to me) & "/conversionfiles/") try do shell script "/bin/ls " & quoted form of thelocation on error display alert "ref2ris can't find the folder conversionfiles." message "click find to locate it, or download ref2ris again." as warning buttons {"cancel", "find"} default button 2 cancel button 1 set thelocation to posix path of (choose folder with prompt "select the folder conversionfiles:") end try end try try do shell script "/bin/ls " & quoted form of thelocation & "markup.sed" on error display alert "ref2ris can't find markup.sed" message "click find to copy it into the folder conversionfiles, or download it again." as warning buttons {"cancel", "find"} default button 2 cancel button 1 set tomove to posix path of (choose file with prompt "select the file markup.sed:") do shell script "cp " & quoted form of tomove & " " & quoted form of thelocation end try end testsystem on convertbib(thesource, thesed) set thebibliography to quoted form of posix path of (thesource as text) set thestyle to quoted form of posix path of (thesed as text) set theprepared to quoted form of thelocation & "prepared.rtf" set themarkerup to quoted form of thelocation & "markup.sed" set themarkedup to quoted form of thelocation & "markedup.rtf" set theinput to quoted form of thelocation & "markedup.txt" set theoutput to quoted form of (do shell script "echo " & thebibliography & " | sed s:\\.[^.]*$:\\.ris:") repeat 1 times set thebibname to name of (info for thesource) tell application "finder" to set theextension to (name extension of thesource) set goodfiletypes to {"txt", "html", "htm", "rtf", "doc", "docx", "wordml", "odt"} if (goodfiletypes does not contain theextension) then display alert "cannot convert " & thebibname message "sorry, " & thebibname & " is not in a supported format." as warning buttons {"cancel", "continue with other files"} default button 2 cancel button 1 giving up after 30 exit repeat end if try do shell script "textutil -convert rtf -encoding utf-8 " & thebibliography & " -output " & theprepared do shell script "sed -f " & themarkerup & " < " & theprepared & " > " & themarkedup do shell script "textutil -convert txt -encoding utf-8 " & themarkedup do shell script "lang=c sed -f " & thestyle & " < " & theinput & " > " & theoutput on error errmsg number errnum cleanup() display alert "error converting " & thebibname message "error number " & errnum & ": " & errmsg as warning buttons {"cancel", "continue with other files"} default button 2 cancel button 1 giving up after 30 exit repeat end try cleanup() if theconverted is equal to "nothing." then set theconverted to (thebibname) else set theconverted to (theconverted & return & thebibname) end if end repeat end convertbib on cleanup() try do shell script "rm " & quoted form of thelocation & "prepared.rtf" do shell script "rm " & quoted form of thelocation & "markedup.rtf" do shell script "rm " & quoted form of thelocation & "markedup.txt" end try end cleanup the bulk of the process is done by the four shell commands executed in the convertbib() subroutine. these commands accomplish the reference conversion in 4 simple steps: convert the incoming file to rtf convert the rtf formatting markup to html convert the marked up file to txt convert the txt citations into tagged ris format 1. convert the incoming file to rtf the first shell command deals with the fact that the user’s bibliography may be in any of a number of file formats.   for the final conversion the file will need to be in .txt format, but converting directly into txt would lose formatting needed to parse the citation – such as italics, bold, and underlines – as well as formatting used to add meaning in specific disciplines – such as italics for species names and superscripts and subscripts for chemical formulae. to avoid this, ref2ris first makes use of apple’s textutil utility (included since macos 10.4) (textutil(1)…[updated 2004]) to convert from txt, html, rtf, rtfd, doc, docx, wordml, odt, or webarchive into rtf format: textutil -convert rtf -encoding utf-8 inputfilename -output outputfilename 2. convert the rtf formatting markup to html the second shell command then executes a sed script (markup.sed) to standardise the bibliography by converting relevant rtf format tags into html-style tags, and smart quotes into plain quotes. sed is a unix stream editor (pement 2003), which searches and replaces strings line by line through an inputted file.  commands use regular expressions and take the basic form: s/oldtext/newtext/ more complex commands are available: for example to search globally on a line, to delete lines, to find a string in the presence or absence of another string, or to create loops.  a sed script consists of a series of these commands and is invoked with the command: sed -f scriptfilename < inputfilename > outputfilename markup.sed (version 1.3) # italics s/\\f[0-9]*\\i /\/g s/\\f0\\i0 /\<\/i\>/g s/{\\i \([^}]*\)}/\\1\<\/i\>/g s/\\i /\/g s/\\i0 /\<\/i\>/g # bold s/\\f[0-9]*\\b /\/g s/\\f0\\b0 /\<\/b\>/g s/{\\b \([^}]*\)}/\\1\<\/b\>/g s/\\b /\/g s/\\b0 /\<\/b\>/g # underline s/{\\ul \([^}]*\)}/\\1\<\/u\>/g s/\\ul /\/g s/\\ulnone /\<\/u\>/g # superscript s/{\\super \([^}]*\)}/\\1\<\/sup\>/g s/\\super \([^\]*\)\\nosupersub /\\1\<\/sup\>/g # subscript s/{\\sub \([^}]*\)}/\\1\<\/sub\>/g s/\\sub \([^\]*\)\\nosupersub /\\1\<\/sub\>/g # smartquotes etc s/\\\'91/'/g s/\\\'92/'/g s/\\\'93/"/g s/\\\'94/"/g s/\\\'96/-/g s/\\\'97/-/g s/\\page/\\par/g 3. convert the marked up file to txt the third shell command uses textutil again to convert to .txt format: textutil -convert txt -encoding utf-8  inputfilename 4. convert the txt citations in tagged ris format the fourth shell command uses whichever sed script the user has selected as appropriate (according to the citation style the bibliography was written in) to convert the marked up txt file into the final ris file.  unicode characters (such as with accented letters) in the bibliography can cause problems here, but this is easily prevented by setting the lang environment variable to the c locale: lang=c sed -f clinicachimicaacta.sed < inputfilename > outputfilename the clinicachimicaacta sed conversion script is included as an example below. clinicachimicaacta.sed (version 1.3) # tidy up tags, punctuation, spurious spacing, blank lines, and urls s/\<[^/>]*\>$// s/^\<\/[^>]*\>// s/\(\<[^/>]*\>\)\([^0-9a-za-z<>]*\)/\2\1/g s/\([^0-9a-za-z\<\>]*\)\(\<\/[^\>]*\>\)/\2\1/g s/\<\([^>]*\)\>\<\/\1\>//g s/\( \)\([ ,.]*\)/\1/g s/^[ ]*// s/[ ]*$// /^$/d s/\([^/]\)www/\1http:\/\/www/g # deliminate records (adding a copy of the full record to the notes field) s/[., ]*$// s/\(.*\)$/\1@@n1 \1@@er / # strip numbers from the start s/^[[0-9]*][ ]*// # deal with the format author. title1. s/^\([^@.]*\)\. \([^@.?]*\?*\)\.* */@@a1 \1@@t1 \2@@t2 / # deal with journal year;vol:sp-ep. format s/@@t2 \([^@;0-9]*\)\([0-9]*;\)/@@jf \1@@y1 \2/ s/\(@@y1 [^@;]*\); *\([^@:]*:\)/\1@@vl \2/ s/\(@@y1 [^@;]*\); *\([0-9]* *-\)/\1@@sp \2/ s/\(@@vl [^@:]*\): *\([0-9]* *-*\)/\1@@sp \2/ s/\(@@sp [^@-]*\) **\([0-9]*\)/\1@@ep \2/ # deal with in: editor a, editor. title2. s/@@t2 in:* \([^@.]*\), editors*\. */@@a2 \1@@t2 / # deal with title. city: publisher, year: sp-ep. format s/\(@@t[12] [^@.]*\)\. /\1@@cy / s/\(@@cy [^@:]*\): */\1@@pb / s/\(@@pb [^@0-9]*\), */\1@@y1 / s/\(@@y1 [^@:]*\): */\1@@sp / # deal with editions s/\(@@t[12] [^@]*\), \([0-9a-z]*\) ed@@/\1@@vl \1@@/ # keep italicised species names etc in t1 : sp_loop s/\\([^<]*\)\<\/i\>\(.*@@t2\)/\\1\<\/species\>\2/ s/\\([^<]*\)\<\/i\>\(.*@@jf\)/\\1\<\/species\>\2/ t sp_loop # get rid of unnecessary html tags (bold, italics, underline) s/\<\/*b\>//g s/\<\/*i\>//g s/\<\/*u\>//g s/species\>/i\>/g # classify records /@@jf/s/^/@@ty jour/ /@@ty/!s/^\(.* [tt]hesis\)/@@ty thes\1/ /@@ty/!s/^\(.* [dd]issertation\)/@@ty thes\1/ /@@ty/!s/^\(.* [cc]onference\)/@@ty conf\1/ /@@ty/!s/^\(.* [cc]ongress\)/@@ty conf\1/ /@@ty/!s/^\(.* [ww]orkshop\)/@@ty conf\1/ /@@ty/!s/^\(.* [ss]eminar\)/@@ty conf\1/ /@@ty/!s/^\(.*@@t2\)/@@ty chap\1/ /@@ty/!s/^/@@ty book/ # appropriately split information in the author field s/\(@@[^@]*\), et al\([^@]*@@[^e]\)/\1\2/g : au_loop s/\(@@a[12] \)\([^@,]*\), /\1\2\1/g t au_loop s/\(@a[12] [^@]*\) \([a-z]*@\)/\1, \2/g # finally replace placeholder with linebreak. s/@@/\ /g the user can import the resulting ris file using the normal process in their bibliographic software.  to import into endnote, for example, the user chooses file > import and selects the file and the import option utf8.  if the records contain formatting such as italicised species names or superscripts or subscripts in chemical formulae, these will display with html tags.  a “convert html tags” plug-in can be downloaded from the endnote forums (rollins 2011) to convert these to ordinary formatting. results then need to be checked for any errors caused either by the conversion process or by typos in the input.  the notes field of each record contains a copy of the original input for comparison. sample output three examples of references to journal articles in clinica chimica acta style, each followed by the ris output, show ref2ris working at its best, and some of the limitations and glitches that users would still need to edit manually. [1]         bedford jj, harper jl, leader jp, et al. betaine is the principal counteracting osmolyte in tissues of the elephant fish, callorhincus millii (elasmobranchii, holocephali). comp. biochem. physiol. b 1998;119:521-526. ty jour a1 bedford, jj a1 harper, jl a1 leader, jp a1 <i>et al</i> t1 betaine is the principal counteracting osmolyte in tissues of the elephant fish, <i>callorhincus millii</i> (elasmobranchii, holocephali) jf comp. biochem. physiol. b y1 1998 vl 119 sp 521 ep 526 n1 [1] bedford jj, harper jl, leader jp, et al. betaine is the principal counteracting osmolyte in tissues of the elephant fish, callorhincus millii (elasmobranchii, holocephali). comp. biochem. physiol. b 1998;119:521-526 er here the user will need to edit the author fields to add the names abbreviated by et al, and the journal field to expand the abbreviation. [2]         sizeland, pcb, chambers st, lever m et al. organic osmolytes in human and other mammalian kidneys. kidney int. 1993;43:448-453. ty jour a1 sizeland a1 pcb a1 chambers, st a1 lever m <i>et al</i> t1 organic osmolytes in human and other mammalian kidneys jf kidney int. y1 1993 vl 43 sp 448 ep 453 n1 [2] sizeland, pcb, chambers st, lever m et al. organic osmolytes in human and other mammalian kidneys. kidney int. 1993;43:448-453 er the extraneous comma after the first surname in the original reference, and the missing comma after the third author, both affect the output. [3]         chance dh, kalas ta. a biological perspective on the use of tandem mass spectrometry for newborn screening and clinical testing. clin. biochem. 2005;38:296-309. ty chap a1 chance, dh a1 kalas, ta t1 a biological perspective on the use of tandem mass spectrometry for t2 n1 [3] chance dh, kalas ta. a biological perspective on the use of tandem mass spectrometry for er ty jour a1 newborn screening and clinical testing t1 clin jf biochem. y1 2005 vl 38 sp 296 ep 309 n1 newborn screening and clinical testing. clin. biochem. 2005;38:296-309 er here an accidental newline in the middle of a reference – effectively invisible to a human editor – results in it being split into two malformed references. scrambled output can also result when a field (most common in titles) contains punctuation which the script reads as delimiting other fields or more generally when the script is written such that it matches a common pattern of citation formatting at the expense of a less common pattern.  thus while ref2ris can do the bulk of the conversion work, the user will always need to inspect and manually edit the results. conclusion and future directions ref2ris can accept bibliographies saved as txt, html, rtf, rtfd, doc, docx, wordml, odt, or webarchive.  it accepts unicode format, allowing for citations that include accented characters, and retains italics, superscripts, and subscripts in article titles.  it can easily deal with large bibliographies – one file containing 5964 references was processed in less than two minutes.  most importantly, it is easy to use for people without expertise in programming or compiling code. although ref2ris was developed to help students and researchers import their own bibliographies into bibliographic managers, it can as readily be used to import bibliographies from published papers.  this opens up possibilities for bibliographic analysis; importing references into a bibliographic manager can greatly speed up sorting by author, date, source, or format compared to manual analysis of the raw citations. conversion scripts have been written for apa and clinica chimica acta.  more citation styles can be supported by creating additional sed scripts.  however with the proliferation of highly diverse citation styles it would be more ideal to create a process that could parse any inputted citation style.  an initial attempt at the problem (see generic.sed in appendix 1) works for some styles and fails spectacularly for others.  even identifying a year is difficult when it can commonly appear in first, second or last place or elsewhere in a given style, and when a given four-digit number might be a page, document, or issue number, or part of a title or even corporate author.  a simple series of find/replace commands is unlikely to ever be sufficient for this task. even with the improved accuracy of citation style-specific conversions, the problems that can be introduced by typos and unexpected punctuation mean that output should always be reviewed by users manually. while the current applescript version of ref2ris is easy to use, it can only be used on macos 10.4 or later.  it would also be desirable to package the core sed scripts into applications for windows, linux, earlier macos, and/or web environments.  however while versions of sed are available on most systems, textutil is not, so another solution for converting from diverse word processing formats into rtf and txt would need to be found. other useful refinements could include: supporting input from pdf supporting conversion into bibtex additional code to convert in-text citations into a format recognised by the user’s bibliographic software note ref2ris is available from http://deborahfitchett.com/toys/ref2ris/ under a gpl license. about the author deborah fitchett is a subject liaison librarian at the university of canterbury eps library. she can be contacted at deborah.fitchett@canterbury.ac.nz references [1] brekhus r. converting bibliographies from text to endnote using wizfolio [internet]. [updated 2010 mar 25]. columbia (mo): university of missouri, mu libraries; [cited 2011 nov 22]. available from: http://libraryguides.missouri.edu/content.php?pid=89799&sid=671253 [2] crossref – simple text query form [internet]. [updated 2010 may 5]. oxford: crossref; [cited 2011 nov 29]. available from: http://www.crossref.org/simpletextquery/ [3] freecite – open source citation parser [internet]. [date unknown]. providence (ri): brown university; [cited 2011 nov 22]. available from: http://freecite.library.brown.edu/ [4] garret p. 2008. importing a bibliography in text format into endnote [internet]. [place of publication unknown]: crevilles.org; [cited 2011 nov 17]. available from: http://crevilles.org/mambo/textes/tutoriels/importendnote_eng.pdf [5] gear rw. 2002. exporting endnote records to ms access and importing endnote records from access using tab delimited files in both directions [internet]. [place of publication unknown]; [cited 2011 nov 17]. available from: http://research.mbs.ac.uk/hsi/portals/0/docs/alanboyd/gear-endnote-access-transfer.pdf [6] hetzner e. 2008. a simple method for citation metadata extraction using hidden markov models. proceedings of the joint conference on digital libraries. pittsburgh, pa. [7] how can i transfer references into endnote from an excel spreadsheet? [internet]. [2010]. sydney: university of technology sydney; [cited 2011 nov 17]. available from: http://www.lib.uts.edu.au/question/5952/how-can-i-transfer-references-endnote-excel-spreadsheet-or-access-database [8] kan m-y, councill ig, giles cl, luong m-t, ng yk, nguyen td, do hnh. 2004. parscit: an open-source crf reference string and logical document structure parsing package [internet]. [updated 2011 may 7]. singapore: national university of singapore; [cited 2011 nov 29]. available from: http://wing.comp.nus.edu.sg/parscit/ [9] paracite [internet]. [2004]. southampton: university of southampton; [cited 2011 nov 29]. available from: http://paracite.eprints.org/ [10] pement e. frequently asked questions about sed, the stream editor: basic sed [internet]. [updated 2003 mar 10]. [place of publication unknown]; [cited 2011 nov 21]. available from: http://sed.sourceforge.net/sedfaq2.html [11] ris format specifications [internet]. [updated 2011 oct 6]. carlsbad (ca): reference manager; [cited 2011 nov 21]. available from: http://www.refman.com/support/risformat_intro.asp [12] rollins j. 2011. convert html tags sample plug-in [internet]. [place of publication unknown]; [cited 2011 nov 21]. available from: http://community.thomsonreuters.com/t5/endnote-api/convert-html-tags-sample-plug-in/td-p/18467 [13] suhonos mj. 2009. semi-automatic citation correction with lemon8-xml. code4lib journal [internet]. [cited 2012 jan 16]; 6. available from: http://journal.code4lib.org/articles/1011 [14] takaki m. generate bibtex entry from plain text [internet]. [date unknown]. [place of publication unknown]; [cited 2011 nov 22]. available from: http://www.snowelm.com/~t/doc/tips/makebib.en.html [15] textutil(1) mac os x manual page [internet]. [updated 2004 sep 9]. [place of publication unknown]: mac os x developer library; [cited 2011 nov 21]. available from: http://developer.apple.com/library/mac/#documentation/darwin/reference/manpages/man1/textutil.1.html [16] transferring a word bibliography to endnote  [internet]. [date unknown]. brisbane: university of queensland library; [cited 2011 nov 22]. available from: http://www.library.uq.edu.au/faqs/endnote/convert_bibliography.html appendix generic.sed (version 2.0) # tidy up tags, punctuation, spurious spacing, blank lines, and urls s/\<[^/>]*\>$// s/^\<\/[^>]*\>// s/\(\<[^/>]*\>\)\([^0-9a-za-z<>]*\)/\2\1/g s/\([^0-9a-za-z<>]*\)\(\<\/[^>]*\>\)/\2\1/g s/\<\([^/>]*\)\>\([^a-za-z0-9<>]*\)\<\/\1\>/\2/g s/\<\/\([^>]*\)\>\([^a-za-z0-9<>]*\)\<\1\>/\2/g s/ **/-/g s/\( \)\([ ,]*\)/\1/g s/^[ ]*// s/[ ]*$// s/[ ][ ]*/ /g /^$/d s/\([^/]\)www/\1http:\/\/www/g # deliminate records (adding a copy of the full record to the notes field) s/\(.*\)$/@@xx \1@@n1 \1@@er / # strip numbers from start s/@@xx \[0-9]*\<\/sup\>[ ]*/@@xx / s/@@xx [[( ]*[0-9]*[]). ]*/@@xx / # find info in square brackets : info_loop s/\[\([^]]*\)\]\(.*@@n1\)/@@xx \1@@xx \2/ t info_loop # find doi and webpages s/[dd][oo][ii]:\([^ <>"]*\)\(.*@@n1\)/@@do \1@@xx \2/ s/\(https*:[^ <>"]*\)\(.*@@n1\)/@@ur \1@@xx \2/ s/\(ftp:[^ <>"]*\)\(.*@@n1\)/@@ur \1@@xx \2/ # find labelled page, issue, and volume numbers s/\(.*\)[^a-za-z]pp*[^a-za-z0-9]*\([0-9][-0-9]*\)\(.*@@n1\)/\1@@sp \2@@xx \3/ s/\(.*\)[^a-za-z][nn]os*[^a-za-z0-9]*\([0-9][-0-9]*\)\(.*@@n1\)/\1@@is \2@@xx \3/ s/\(.*\)[^a-za-z][vv]ols*[^a-za-z0-9]*\([0-9][-0-9]*\)\(.*@@n1\)/\1@@vl \2@@xx \3/ # start identifying records /@@do/s/^/@@ty elec/ /^@@xx .*[tt]hesis/s/^/@@ty thes/ /^@@xx .*[dd]issertation/s/^/@@ty thes/ /^@@xx .*conference/s/^/@@ty cpaper/ /^@@xx .*workshop/s/^/@@ty cpaper/ /^@@xx .*congress/s/^/@@ty cpaper/ /^@@xx .*seminar/s/^/@@ty cpaper/ /^@@xx .*symposium/s/^/@@ty cpaper/ /^@@xx .*meeting/s/^/@@ty cpaper/ /^@@xx .*forum/s/^/@@ty cpaper/ /^@@xx .*[rr]eport/s/^/@@ty rprt/ /^@@xx .*[pp]atent/s/^/@@ty pat/ /^@@xx .*@@ur/s/^/@@ty elec/ /@@ty/!s/^/@@ty gen/ # find identifiable titles s/\(@@xx [^@"]*\)"\([^@"]*\)"\(.*@@n1\)/\1@@t1 \2@@xx \3/ s/\(@@xx [^@<]*\)\<[iu]\>\([^@<]*\)\<\/[iu]\>\(.*@@n1\)/\1@@tt \2@@xx \3/ s/\(@@xx [^@<]*\)in:* \(.*@@n1\)/\1@@xx \2/ # split at editor/edition s/@@xx [^a-za-z0-9@]*[ee]ds*[^a-za-z0-9@][^a-za-z0-9<@]*\(.*@@n1\)/@@xx \1/g s/\([^a-za-z0-9@][ee]ds*[^a-za-z0-9@][^a-za-z0-9<@]*\)\(.*@@n1\)/\1@@xx \2/g # rewrite dates : date_loop s/\([a-z][a-z][a-z]\)[a-z]* \([0-9][-0-9]*\),* \([1-2][0-9][0-9][0-9]\)\(.*@@n1\)/\3-\1-\2@@xx \4/ s/\([0-9][-0-9]*\)[snrt]*[tdh]* \([a-df-z][a-z][a-z]\)[a-z]*,* \([1-2][0-9][0-9][0-9]\)\(.*@@n1\)/\3-\2-\1@@xx \4/ s/\([1-2][0-9][0-9][0-9]\)[^-a-za-z0-9@]*\([a-z][a-z][a-z]\)[a-z]*[^-a-za-z0-9@]*\([0-9][-0-9]*\)\(.*@@n1\)/\1-\2-\3@@xx \4/ t date_loop s/-jan-/-01-/g s/-feb-/-02-/g s/-mar-/-03-/g s/-apr-/-04-/g s/-may-/-05-/g s/-jun-/-06-/g s/-jul-/-07-/g s/-aug-/-08-/g s/-sep-/-09-/g s/-oct-/-10-/g s/-nov-/-11-/g s/-dec-/-12-/g s/\([12][0-9][0-9][0-9]\)-\([0-1][0-9]\)-\([0-9][-0-9]*\)/@@dd \1\/\2\/\3\//g s/\(@@dd [12][0-9][0-9][0-9]\/[0-1][0-9]\/\)\([0-9][-/]\)/\10\2/g s/@@xx [^@]*[aa]ccess[^@]* @@dd /@@da / s/@@xx [^@]*[rr]etriev[^@]* @@dd /@@da / s/@@xx [^@]*[cc]ite[^@]* @@dd /@@da / # find years s/\(@@xx [^@]*\)\\([1-2][0-9][0-9][0-9]\)\<\/b\>\(.*@@n1\)/\1@@y1 \2@@xx \3/ /@@y1/!s/\(@@xx [^@]*\)[[(]\([1-2][0-9][0-9][0-9][^])0-9]*\)[])]\(.*@@n1\)/\1@@y1 \2@@xx \3/ /@@y1/!s/\(@@xx [^@]*[^-0-9]\)\([1-2][0-9][0-9][0-9]\)\([^-0-9].*@@n1\)/\1@@y1 \2@@xx \3/ s/\(@@xx [^@]*\)n\.d\.\(.*@@n1\)/\1@@y1 n\.d\.@@xx \2/ s/\(@@xx [^@]*\)in press\(.*@@n1\)/\1@@y1 in press@@xx \2/ /@@y1/!s/@@dd /@@y1 / /@@dd/!s/\(@@ty cpaper.*\)@@y1/\1@@da/ # find unlabelled page numbers, volumes and issues /@@sp/!s/\(.*[ .:p]\)\([0-9][0-9]*-[0-9][0-9]*\)\([^-0-9].*@n1\)/\1@@sp \2@@xx \3/ /@@is/!s/\(.*[^/0-9]\)\([0-9][-0-9]*[a-z]*\)\<\/[ibu]\>[^a-za-z0-9]*\([0-9][-0-9]*\)\([^a-za-z0-9]*@@sp\)/\1@@vl \2@@is \3\4/ /@@is/!s/\(.*[^/0-9]\)\([0-9][-0-9]*[a-z]*\)@@xx [^a-za-z0-9]*\([0-9][-0-9]*\)\([^a-za-z0-9]*@@sp\)/\1@@vl \2@@is \3\4/ /@@is/!s/\(.*[^/0-9]\)\([0-9][-0-9]*[a-z]*\)[^a-za-z0-9][^a-za-z0-9]*\([0-9][-0-9]*\)\([^a-za-z0-9]*@@sp\)/\1@@vl \2@@is \3\4/ /@@vl/!s/\(.*[^/0-9]\)\([0-9][-0-9]*[a-z]*\)\<\/[ibu]\>\([^a-za-z0-9]*@@sp\)/\1@@vl \2\3/ /@@vl/!s/\(.*[^/0-9]\)\([0-9][-0-9]*[a-z]*\)@@xx \([^a-za-z0-9]*@@sp\)/\1@@vl \2\3/ /@@vl/!s/\(.*[^/0-9]\)\([0-9][-0-9]*[a-z]*\)\([^a-za-z0-9]*@@sp\)/\1@@vl \2\3/ s/\(@@vl [^@]*\)@@xx \([^@]*@@sp \)/\1@@is \2/ s/@@y1 @@vl [^a-za-z0-9@]*\([1-2][0-9][0-9][0-9]\)[^a-za-z0-9@]*@@is /@@y1 \1@@vl / # split at end of titles s/\. *\. *\./@@ellipsis/ : split_loop s/\(@@xx [^@]*[^a-z]\)\. \(.*@@n1\)/\1@@xx \2/ t split_loop s/@@ellipsis/\.\.\./ # strip italics, bold, underline : html_loop s/\<\/*[^>]\>\(.*@@n1\)/\1/ t html_loop # clean up empty fields s/@@.. [^a-za-z0-9]*@@/@@/g s/@@.. [^a-za-z0-9]*@@/@@/g # identify editions vs editors s/@@xx \([^@]*[^a-za-z0-9][^a-za-z0-9]*\)[ee]ditors*\(.*@@n1\)/@@a2 \1\2/ s/@@xx [^a-za-z0-9]*\([0-9][^@]*\)[ee]dition\(.*@@n1\)/@@et \1\2/ s/@@xx [^a-za-z0-9]*\([0-9][^@]*\)[ee]d[^a-za-z0-9@]*\(.*@@n1\)/@@et \1\2/ s/@@xx [^a-za-z0-9]*\(rev[^@]*\)[ee]dition\(.*@@n1\)/@@et \1\2/ s/@@xx [^a-za-z0-9]*\(rev[^@]*\)[ee]d[^a-za-z0-9@]*\(.*@@n1\)/@@et \1\2/ s/@@xx \([^@]*[^a-za-z0-9][^a-za-z0-9]*\)eds*\([^a-za-z0-9].*@@n1\)/@@a2 \1@@xx \2/ # identify city:publishing press (and vice versa) s/@@xx \([^@]*\):\([^@]*press[^@]*\)/@@cy \1@@pb \2/ s/@@xx \([^@]*\):\([^@]*publi[sc][ha][^@]*\)/@@cy \1@@pb \2/ s/@@xx \([^@]*press[^@]*\):\([^@]*\)/@@pb \1@@cy \2/ s/@@xx \([^@]*publi[sc][ha][^@]*\):\([^@]*\)/@@pb \1@@cy \2/ # identify city, state:publisher (and vice versa) /@@pb/!s/@@xx \([^@]*[a-z][a-z][^a-za-z0-9@]*\):\([^@]*\)/@@cy \1@@pb \2/ /@@pb/!s/@@xx \([^@]*\):\([^@]*[a-z][a-z][^a-za-z0-9@]*@@\)/@@pb \1@@cy \2/ # identify author fields s/@@xx \([^@]*et al[^@]*\)/@@aa \1/ s/@@xx \([^@]*, \)\.\.\.\([^@]*\)/@@aa \1 et al, \2/ s/@@xx \([^@]*&[^@]*\)/@@aa \1/ s/@@xx \([^@]* [js]r[^@]*\)/@@aa \1/ s/@@xx \(['a-za-z][-' a-za-z]*[, ]*[a-z][-. a-z]*[,.;@]\)/@@aa \1/ s/@@xx \([a-z][-. a-z]* [a-za-z][-' a-za-z]*[,.;@]\)/@@aa \1/ # identify more record types /@@a2/s/@@ty [a-z]*/@@ty chap/ /@@vl/s/@@ty gen/@@ty jour/ /@@vl/s/@@ty elec/@@ty ejour/ # guess at city:publisher /@@ty e*jour/!s/@@xx \([^@]*, [^@]*\):\([^@]*\)/@@cy \1@@pb \2/ /@@ty e*jour/!s/@@xx \([^@]*\):\([^@]*, [^@]*\)/@@pb \1@@cy \2/ # identify miscellaneous fields s/@@xx \([^@]*[tt]hesis[^@]*\)/@@m3 \1/g s/@@xx \([^@]*[dd]issertation[^@]*\)/@@m3 \1/g s/@@xx \([^@]*\) [pp]atent [^@0-9]*\([0-9]*\)/@@cy \1@@sn \2/g # identify title fields s/@@xx \([^@]* a [^@]*\)/@@tt \1/g s/@@xx \([^@]* an [^@]*\)/@@tt \1/g s/@@xx \([^@]* in [^@]*\)/@@tt \1/g s/@@xx \([^@]* on [^@]*\)/@@tt \1/g s/@@xx \([^@]* and [^@]*\)/@@tt \1/g s/@@xx \([^@]* for [^@]*\)/@@tt \1/g s/@@xx \([^@]*[rr]eport[^@]*\)/@@tt \1/g # final guess at publisher /@@pb/!s/@@xx \([^@]*:[^@]*\)/@@pb \1/ /@@pb/s/@@ty gen/@@ty book/ # assign authors s/@@xx \([^@]*@@y1\)/@@aa \1/ /@@a1/!s/@@aa/@@a1/ /@@a2/!s/@@aa/@@a2/ s/@@aa/@@a3/ s/@@aa/@@xx/g # split authors s/\(@@a. \)\([^@]*\) et al */\1\2\1et al\1/ : au_loop s/\(@@a. \)\([^@]*\); /\1\2\1/ s/\(@@a. \)\([^@]*\)& /\1\2\1/ s/\(@@a. \)\(['b-za-z][-' a-za-z]*\),* \([a-z][.a-z]*\), /\1\2, \3\1/ s/\(@@a. \)\([a-z][.a-z]* ['a-za-z][-' a-za-z]*\), /\1\2\1/ t au_loop s/\(@@a. \)\(['b-za-z][-' a-za-z]*\),* \([a-z][.a-z]*\)\([a-z][a-z]\)/\1\2, \3@@tt \4/g s/\(@@a. -\)\([^@]*\)and /\1\2\1 /g s/\(@@a. \)\([a-z][.a-z]*\) \(['a-za-z][-' a-za-z]*\)/\1\3, \2/g # assign titles /@@t1/!s/@@tt/@@t1/ /@@t1/!s/@@t2/@@t1/ /@@t2/!s/@@tt/@@t2/ /@@t3/!s/@@tt/@@t3/ s/@@tt/@@xx/ # assign dates /@@y1/!s/@@dd/@@y1/ /@@y2/!s/@@dd/@@y2/ /@@da/!s/@@dd/@@da/ s/@@dd/@@xx/g # clean punctuation from start and end of fields s/\(@@.. \)[^a-za-z0-9@<>]*/\1/g s/[^a-za-z0-9<>]*\(@@.. \)/\1/g # clean up empty fields s/@@.. [^a-za-z0-9]*@@/@@/g s/@@.. [^a-za-z0-9]*@@/@@/g # guess at, or collapse, unidentified fields s/\(@@ty [a-z]*\)@@xx \([^@]*\)\(@@.. \)/\1\3\2\. / /@@t1/!s/@@xx /@@t1 / /@@t2/!s/@@xx /@@t2 / /@@ty e*jour/s/@@t2/@@jf/ /@@t3/!s/@@xx /@@t3 / s/@@xx /. /g # standardise field names s/@@a1/@@au/g s/@@t1/@@ti/ s/@@y1/@@py/ # finally replace placeholder with linebreak. s/@@/\ /g subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – where do we go from here: a review of technology solutions for providing access to digital collections mission editorial committee process and structure code4lib issue 47, 2020-02-17 where do we go from here: a review of technology solutions for providing access to digital collections the university of toronto libraries is currently reviewing technology to support its collections u of t service. collections u of t provides search and browse access to 375 digital collections (and over 203,000 digital objects) at the university of toronto libraries. digital objects typically include special collections material from the university as well as faculty digital collections, all with unique metadata requirements. the service is currently supported by iiif-enabled islandora, with one fedora back end and multiple drupal sites per parent collection (see attached image). like many institutions making use of islandora, utl is now confronted with drupal 7 end of life and has begun to investigate a migration path forward. this article will summarise the collections u of t functional requirements and lessons learned from our current technology stack. it will go on to outline our research to date for alternate solutions. the article will review both emerging micro-service solutions, as well as out-of-the-box platforms, to provide an overview of the digital collection technology landscape in 2019. note that our research is focused on reviewing technology solutions for providing access to digital collections, as preservation services are offered through other services at the university of toronto libraries. by kelli babcock, sunny lee, jana rajakumar, andy wagner introduction the university of toronto libraries (utl) system consists of over 40 libraries located on three university campuses. it serves over 14,000 faculty members and more than 90,000 undergraduate and graduate students [1]. the utl information technology services (its) department provides digital library services to the utl system, including the collections u of t service. collections u of t provides search and browse access to u of t digital collections. it currently contains 375 digital collections and over 197,000 digital images, and typically includes archives or special collections material from libraries and archives across the library system, as well as digital collections owned by university departments, faculty, or external project partners. each collection often has unique metadata requirements as well as custom requests for presentation of web content. the collections u of t service is currently supported by a single static html site at the main url. this static html site links out to multiple international image interoperability framework (iiif) enabled islandora 7 websites that act as the point of access for each top-level collection in the repository. digital objects for each collection are stored in a single fedora repository. images are served by a loris iiif image server that retrieves them from fedora and provides a caching layer. iiif manifests are generated through customized islandora ingest scripts and stored in a mongodb instance. like many institutions using islandora 7, the collections u of t service is now confronted with drupal 7 end of life by november 2021. the department has begun to investigate a migration path forward, with research efforts coordinated by kelli babcock, digital initiatives librarian. this article summarises the first phase of our research, including a brief history of collections u of t, lessons learned from experience with our current technology stack, and our selected path forward after reviewing various technology options. the first phase of our research has included investigating both out-of-the-box platforms and emerging “pluggable” technology solutions for digital collections. note that our research is focused on reviewing technology solutions for providing access to digital collections, and not preservation. we have decoupled the long-term preservation of our digital objects from their discovery and access with the launch of project canopus, a preservation focused digital repository at the university of toronto libraries. where we started: our initial context collections u of t began as an initiative to create a web portal to support the discovery of multiple u of t digital collections. in 2013, the collections u of t web portal was created in response to increasing requests from faculty members, external partners, u of t archives and library departments – “collection owners” – for technology support to make their digital collections available online. we refer to “collection owners” throughout this article to identify collections u of t users that are either the archive or library that cares for the digital collection, or the faculty or department that sponsored the collection’s publication to the web. collection owners often request similar features for technology support, including: presentation; search; collection theming; the ability to customise metadata per collection; a wysiwyg editor interface to manage the contextual web content surrounding collection objects; and “preservation” functionality. within the library, there was also an interest in providing users with the ability to search both across collections and within a specified collection. prior to 2013, its used the coldfusion programming language to create digital collection websites. [2] each of these digital collections existed in its own siloed coldfusion site. the idea, at the time, was to unite the collections through a central access point for users. a review of proprietary and open source repository platforms was conducted in 2013. islandora 7 was identified as the best fit for the requirements and context of collections u of t at the time: islandora 7 could be set up as a multi-site environment: one “parent” site to search across collections with separate “child” sites per collection out of the box, islandora 7 provided support for mods and dublin core metadata. the use of both standards would meet most descriptive needs of the proposed collections islandora was also flexible enough to accommodate other metadata schemas if needed in the future metadata search and display could be customised through the user interface by collection owners islandora 7 had been, or was being, adopted by many canadian and international academic libraries. this meant islandora came along with a strong open source community (with incredibly helpful volunteers, as it turned out) utl its already maintained many drupal websites. staff in the its department had existing expertise in building drupal sites. the department had also gained experience with islandora 6 while creating the heritage u of t site in 2013, [3] a project that preceded and somewhat inspired the collections u of t portal though its had not yet hired a digital preservation librarian at the time of the 2013 platform review, there was some attraction to islandora because it was thought that fedora could be a candidate for supporting some of the workflows and infrastructure of future preservation services. since 2013, other technology solutions were selected to provide preservation services at utl [4] after the platform selection, its set up an islandora instance for collections u of t in 2014 (initially set up as one fedora 3 with multiple islandora/drupal 7 sites). a number of collection owners were eager to get their collections online by joining collections u of t upon its initial set up. the repository grew quickly. there were many lessons learned following the initial set up. most importantly, we realized that adjustments to the architecture of our islandora instance were required to improve internal development workflows for a digital collections portal at this scale. [5] next, we created custom ingest scripts combined with mediated ingest – its now runs ingest scripts upon request by collection owners, as opposed to collection owners submitting gb of digital object data through their web browsers. this mediated workflow has become common practice because the custom ingest scripts also provide more control over the permanent identifiers (pids) assigned to digital objects. for example, it is not possible to control the full pid of digital objects in islandora 7 if they are ingested through the batch ingest user interface (only the namespace can be assigned, leaving the second half of the pid to be a random number). our ingest scripts enabled us to create meaningful pids – from collection objects down to page-level objects. this meant that we could maintain legacy identifiers for objects migrated from coldfusion and also ensure each object had a meaningful pid in its corresponding url. mediated ingest also meant that developers could resolve any ingest issues more efficiently. collection owners were, and are still, required to prepare the images and metadata for ingest, but they no longer experience the wait times of ingesting through islandora’s batch ingest user interface, or the burden of reporting failed ingests for developers to triage and resolve. we also found that using a cms like drupal provided some opportunity to distribute the work of content creation and site maintenance to collection owners, but not in all cases. for many collection owners, there was a big learning curve in navigating drupal 7, let alone having to learn the search and metadata configuration possibilities in islandora 7. initially, frequent training was provided to collection owners alongside detailed documentation for support. eventually, its took on most of this work for the sake of efficient site set up and to ensure search and browse worked well for users. collection owners now interact with their sites primarily through drupal wysiwyg editors to modify page content. shortly after collections u of t was created, the exhibits u of t service was set up in its by digital scholarship librarian, leslie barnes. exhibits u of t uses the omeka platform. compared to drupal, omeka has proved to be a truly diy solution for enabling collection owners to create and manage their own web content. generally, collections u of t (islandora) is used when search facets and querying metadata are requirements of the collection owner. exhibits u of t is preferred for simple exhibit building that does not require search features. in terms of building a parent site to search across collections, usage statistics from google analytics revealed that very few users were searching across collections using the collections u of t “parent” portal site. the parent collections u of t site was retired in 2017 and replaced with a static html site directing to each child collection. in the future, we will instead be focusing on promoting discovery of collections through our library catalog, search engine optimization, and connecting with consortial discovery portals and digital collection aggregators. collections u of t also evolved from simply being a portal for utl digital collections to becoming a full service provided by its to support the management of u of t digital collections. the service now includes consultations to those interested in creating digital collections, including tools and templates for metadata creation, advice on technology selection, and – when the technology fits the needs of the project – development support for setting up and configuring a collections u of t site for collections owners. from 2015 to 2016, iiif was integrated into our islandora instance while collaborating on the french renaissance paleography project. [6] during this project we also implemented mirador [7] as a viewer for islandora book and page objects. eventually, iiif became seamless with our collections u of t ingest workflows – all image objects and collections in the repository came to have a iiif manifest. the islandora mirador viewer and iiif manifest generation functionalities are implemented as a custom module called islandora mirador. manifests can be generated and posted to mongodb (part of the iiif presentation api) via a “generate iiif manifest” button in the islandora user interface when “managing” an object, or through command-line interface (cli) using a custom drush (drupal shell scripts) command. we create collection-level iiif manifests per collection and also keep a top-level collections u of t collection manifest containing all collections and objects. [8] as described in figure 1, iiif is now core to the collections u of t service to enable the delivery of digital objects. figure 1. diagram of the current technology supporting the collections u of t service, described by sunny lee, digital initiatives programmer analyst. collections u of t’s web server component uses drupal, with iiif ingestor and mirador viewer, combined with solr for search, and fedora for object management. loris iiif image server fetches the images from fedora and a microservice provides iiif manifests which are stored in mongodb. iiif manifests are created through customized tuque ingest scripts and posted to mongodb via the microservice. collections u of t digital objects are no longer limited to being accessible through only one islandora site. digital objects can now be embedded in an iframed mirador viewer on any faculty or library website, regardless of platform, while simultaneously being shared with any iiif aggregator. iiif has drastically changed how and where collections u of t digital objects can be presented. this has led its to investigate alternate presentation opportunities for digital collections. for example, we are currently testing a simple solution for embedding archives and special collection digital objects from collections u of t into utl’s archival description platform, discover archives, [9] an access to memory (atom) instance. in early 2017 its began work on project canopus, a two year partnership with scholars portal [10] to implement a preservation platform for utl. after an evaluation of extant solutions and ongoing community initiatives it was decided that no existing solutions would provide an implementation that would scale to accommodate the university of toronto libraries’ needs, let alone a multi-tenanted offering from scholars portal for the larger ontario council of university libraries (ocul) community. with the understanding that all technological systems will have a lifetime shorter than the life of the preserved objects, a number of small microservices were built to provide an api for a uniform and consistent experience for interacting with the underlying modular components of the platform, and allows them to be transparently swapped out for new versions when needed without any impact to users or software. currently the main components used in canopus are marklogic, kafka, and the elastic stack. put into production in april 2019, canopus has grown to contain information on 240 million objects representing 220 terabytes of data. further work is planned for 2020 and beyond to extend the implementation to allow multi-tenancy. the combination of iiif opening up presentation possibilities, canopus removing the need for preservation capabilities, and drupal 7 end of life fast approaching in november 2021 has initiated our investigation into new technology solutions to support the collections u of t service. november 2021 may seem some time away, but migrating 197,000 digital images plus 14 parent collection sites will be no small task. the first phase of our research, a scan of technology options, is now complete. preceding this first phase was an initial broad review of digital collection platforms in anticipation of a library services platform procurement process. this was completed in early 2019 by our its colleagues leslie barnes, rachel di cresce, and mariya maistrovskaya. their review recommended that vital, portfolio, and alma-d were not suitable for collections u of t. it highlighted a custom samvera hyrax or turnkey samvera hyku, blacklight/spotlight, omeka-s, and collective access as solutions to continue monitoring. since may 2019 the department has continued to scan digital collection platform options, including further assessment of islandora 8, the emerging archipelago commons platform, resourcespace, and adam matthew digital’s quartex solution. [11] throughout the spring and summer of 2019 we were also inspired by institutions publishing and presenting on “microservices”-like digital collection solutions, such as project electron from the rockefeller archive centre. [12] these ideas led to initiating an exploration of what we have been calling a “pluggable pieces” solution for collections u of t. it should be noted that the “pluggable pieces” solution has relied heavily on the expertise of its developers and network services staff in the department, while the packaged platform solutions have been more easily explored by librarians. framing all of this research is a set of functional requirements of the collections u of t service in its current context. the high level functional requirements break down into 5 categories: collection management and object storage indexing search and discovery user management and authentication statistics (collection and usage) related to these high-level functional requirement categories, there is an interest in developing linked open data and annotation tools for digital collection objects. both of these goals have been on our minds while conducting research but are ultimately “nice to haves” and outside of the scope of requirements for this first phase of research. another important consideration during the course of our research has been utl’s pending migration to alma and primo in 2020. we have been learning about alma and primo while concurrently researching other digital collection technology solutions. additionally and importantly, accessibility is a priority for any technology we move forward with and will be assessed in greater detail during the next testing phase of our research process. as a result of our research to date, we have proposed that its does not pursue any implementation of the packaged platforms that have been reviewed thus far. instead, we have recommended that its focus on a “pluggable pieces solution” to prepare for migration away from islandora 7/drupal 7 and fedora 3. we will not be investigating digital preservation repository solutions, as this service is now performed by project canopus. we will be replacing fedora 3 with a simpler digital object storage solution. for the sake of brevity in this article, our notes on each platform reviewed from may 2019 to date are available in a public google document. [13] within this article, we have chosen to include our islandora 8 considerations alongside our proposed “pluggable pieces” path forward, along with brief mentions of the institutions that have inspired this approach. islandora 8 islandora 8 [14] is an obvious option to consider first for migration. for us, the benefits of using islandora 8 (i.e. drupal 8, solr, and a self-described “bento box” of other technology) include continuing to leverage the drupal expertise we have in its. we have already begun to use drupal 8 for other digital library projects in the department. using drupal 8 for collections u of t would beneficially build on that existing knowledge and infrastructure. islandora 8 is more closely tied to drupal 8 than islandora 7 was to drupal 7, especially in terms of managing collections and object storage. for example, metadata is no longer just an xml file in fedora that gets indexed in solr. with islandora 8, metadata is actually stored and managed as fields in the drupal database. the introduction to islandora 8 webinar [15] notes that “metadata profiles are created via drupal content types” and “it can be serialized into other formats (xml, json, json-ld) on demand”. batch metadata updates are made possible through drupal’s views bulk operations by a “modify field values” operation. metadata display is configurable in the drupal view associated with the object type. islandora 8 promises to make good use of drupal this time around. islandora 8 community developers are also proposing interesting ways of integrating iiif, such as generating iiif manifests from drupal views results. [16] the downside of continuing to use drupal for collections u of t is that we will still be using technology that has a steep learning curve for collection owners – possibly even steeper for islandora 8, since the level of configuration possibilities has increased dynamically compared to 7, upon first look. continuing down the drupal path means that the collections u of t service must continue to offer collection owners with sufficient drupal 8 training and support. the question that hangs on drupal 8 is: will it complicate or simplify collection building for our collection owners? overall, islandora 8 does a lot. it is an impressive and powerful repository platform, but it might do a lot more than we need – especially considering all of the preservation functionality it comes along with. we have considered that drupal is the primary data store and islandora 8 can be configured so that you don’t have to use fedora or any preservation functions. avoiding it is possible through islandora 8’s use of the drupal context module [17] and editing the field containing the file for a media content type. for example, within the image content type, field_media_image’s field settings can be set to either drupal’s public file system or fedora. other preservation workflows, such as generating derivatives when original files are ingested, are also configurable through the context module. out of the box islandora 8 is set up to perform these actions. [18] this makes sense, since most institutions use islandora for preservation. for collections u of t, preservation is not a functional requirement. the depth of configuration necessary to remove preservation from islandora leads us to wonder how much work will be required by our team to avoid the preservation functions baked into islandora 8 per top-level collection in collections u of t (a reminder that, for us, each top-level collection is equal to one site/discovery point – meaning that, if we were to use islandora 8, each site would require this level of customisation). finally, at the time of our review, islandora 8 did not yet have the functionality to support some requirements that we needed for a migration test – though many of these requirements have since been completed or are on the islandora 8 roadmap [19] and may become available with time. more anecdotally, a migration to islandora 8 feels like it would boil down to replacing one large repository platform (islandora 7) with pieces and functions that we no longer need with the new version of that large repository platform (islandora 8). most reviews of “packaged platforms” left us with this feeling and so our interest turned to colleagues publishing and presenting on an alternate route forward. pluggable pieces investigating options for pluggable pieces of technology to support collections u of t has not been as straightforward as researching packaged platform solutions. we celebrated the publication of “bc digitized collections: towards a microservices-based solution to an intractable repository problem” from boston college libraries in issue 44 of code4lib [20] and are also very interested in the work of the university of denver libraries in building a digital collections “ecosystem.” [21] we have spent some time over the past few months discussing what a similar approach would look like for collections u of t. it might have been our senior application programmer analyst, bilal khalid, who said to make it “pluggable”. while the rockefeller archive center proposed microservices as a visual of stitching seams together, we have run with bilal’s visual of plugging pieces in and out based on the functions and needs of the collections u of t service. this approach appeals because it feels more flexible than implementing an entirely new platform that might expire in under 5 years, and seems resource efficient because it allows us to consider existing technology infrastructure in our large department that we can potentially re-use. it is early days and more exploration is needed, but the ideas we are considering are described below. collections management and object storage our current practice for creating collections starts with images and metadata being prepared for ingest by collection owners. in most cases, collection owners compile their image files along with corresponding xml files (usually mods) and give us access to these files through shared network drives or other media storage devices. file structure follows islandora ingest requirements (note that the ingest process is preceded by one, or many, consultations on the needs and structure of the collection). our islandora ingest scripts then create the digital objects in our collections u of t infrastructure, which includes depositing jp2 copies of the images (we do not store the original tiffs) plus metadata into fedora. the ingest scripts also generate iiif manifests that are stored in mongodb. in a “pluggable” scenario, its developers have proposed simplifying this ingest process by untangling our ingest scripts from islandora and fedora and instead creating a platform-agnostic ingest process. in this scenario, what we are calling a “collections manager” interface would be built upon existing apis to enable collection owners to create and manage collections and deposit digital objects and corresponding metadata files through a simple user interface. this will make use of some components (independent and reusable bits of code) which will be built in-house – and can potentially be reused for other services or digital projects. an “uploader” component will take in raw files and upload them to a remote location. as soon as the uploader finishes a process, a “converter” component will verify and convert the files ready for ingesting. using these two components, the collections manager will upload the media and metadata files to the object storage and populate a database while also sending metadata to an indexing service. upon requests from the public, iiif manifests will be generated by the presentation api based on the relationships specified in the database. images will be served by the iiif image server downloading jp2s from the object storage. to replace fedora, we are looking at using minio for digital object storage. figure 2. diagram of proposed “pluggable pieces” to support the collections u of t service, described by sunny lee, digital initiatives programmer analyst. we are also considering elasticsearch for search and indexing, as this is used for other services in our department, in combination with static html pages for presentation. we recently developed a small bit of javascript that will direct each pid of a digital object in collections u of t to a permanent url that displays the object on a static page – for example, https://collections.library.utoronto.ca/view/storynations:2197711. the static page currently only contains a mirador embedded viewer, but we will be expanding this page to include the metadata text of the object. we also plan to review how other static site tools might work for this approach, such as gatsbyjs,[22] or static site generators for digital collections, including the university of idaho’s collectionbuilder-gh,[23] columbia university libraries’ wax,[24] and a react discovery tool maintained by christopher johnson. [25] plugging in to existing discovery points after creating workflows for collection management, we then want to explore existing discovery points to present digital objects. in september 2019, utl announced a migration to the ex libris library services platform, alma, including the primo resource discovery service. in the coming weeks, we will begin to investigate connecting iiif digital collections to primo for discovery. for some time we have also been interested in connecting iiif archives digital objects to discover archives, our access to memory (atom) instance. this will require considerations around performing item level discovery within an archival finding aid database and close consultation with archivists and users performing research on primary documents described hierarchically. we will also pursue other opportunities for distributing utl collections to iiif aggregators, such as the biblissima manuscripts & rare books aggregator,[26] and research into surfacing digital collection objects and their associated iiif urls through wikidata.[27] testing the path forward layered on top of this research is our desire to build on the lessons learned after running the collections u of t service for six years. part of that desire is acknowledging that the context in which the department initially reviewed technology to support the service has shifted – and it will continue to shift. for example, iiif has offered new presentation and access possibilities that we could not have anticipated. iiif has allowed us to move beyond the goal of building one “portal” site to instead pursue multiple points of discovery – even moving between technology platforms by embedding digital objects into any number of existing or yet-to-be-created discovery access points. this both simplifies and complicates our goal of selecting technology to provide access to digital collections because there may not be one technology solution for access. moving forward with our research, we want to choose an approach to building infrastructure that will allow us to be more flexible and pluggable. recognizing that these terms are contextual, our next phase of research will consist of in depth testing and pilot projects to explore the pluggable pieces solution we are proposing – both in our department, as well as facilitated testing with collection owners and users. part of this testing will be strategizing how to best “plug into” existing technology infrastructure to deliver the collections u of t service and meet our users where they want to find content. it was thought that others in the code4lib community facing drupal 7 end of life may find some value in the shifting context of the current collections u of t islandora 7 instance and the initial technology review we have conducted. while we move forward with the next testing phase of our research we are looking for opportunities to share our findings. as a start, we hope this article is helpful in contributing to the public discussions, in code4lib as well as associated communities, around the technologies our libraries select to support access to digital collections. references and notes [1] quick facts | university of toronto. [accessed 2019 dec 12]. available from: https://www.utoronto.ca/about-u-of-t/quick-facts [2] all of these coldfusion sites have since been migrated, most were migrated to islandora 7. we used archive-it to capture the original coldfusion sites – they are listed among utl’s digital collections. university of toronto digital collections | archive-it. [accessed 2019 dec 12]. available from: https://archive-it.org/collections/6473 [3] heritage u of t. [accessed 2019 dec 12]. available from: https://heritage.utoronto.ca/ [4] while splitting up technology solutions for access and preservation might not be possible or desirable for all institutions, it has worked well for the utl its context. anecdotally, not all collections intended for discovery within collections u of t require long term preservation. removing preservation as a functional consideration has freed our current collections u of t technology review from being locked into considering only the, often weight-y, technology solutions that do both preservation and access [5] read more about chris crebolder and andy wagner’s work on this in their “from an island to islets: reimagining islandora in a microservices world” presentation – goo.gl/r4txcn [6] view the project at https://paleography.library.utoronto.ca/ and learn more about iiif at https://iiif.io/ [7] the work is available at https://github.com/utlib/islandora_mirador_bookreader. more information about mirador is available at https://projectmirador.org/ [8] the collections u of t iiif manifest is available at https://iiif.library.utoronto.ca/presentation/collections [9] https://discoverarchives.library.utoronto.ca/ [10] scholars portal is a service of the ontario council of university libraries (ocul). founded in 2002, scholars portal provides shared technology infrastructure and shared collections for all 21 university libraries in the province of ontario – https://scholarsportal.info/ [11] a very helpful starting point for anyone embarking on a digital collection platform reviews is ashley blewer’s spreadsheet: https://bits.ashleyblewer.com/blog/2017/08/09/collection-management-system-collection/ [12] arnold h. project electron update: building microservices for integration. 2018 dec 13 [accessed 2019 dec 12]. available from: https://blog.rockarch.org/project-electron-update-building-microservices-for-integration [13] our may 2019 research omitted platforms that do not support iiif, since iiif is core to the collections u of t service. all phase 1 – collections u of t technology review research and notes are publicly available at http://bit.ly/2shxnfv [14] islandora 8. [accessed 2019 dec 12]. available from: https://islandora.ca/islandora8 [15] https://www.youtube.com/watch?v=zaeqccs_9fy [16] generate a iiif manifest from views results · issue #1263 · islandora/documentation. [accessed 2019 dec 12]. available from: https://github.com/islandora/documentation/issues/1263 [17] context. islandora/documentation. [accessed 2019 dec 12]. available from: https://islandora.github.io/documentation/user-documentation/context/ [18] “by default, derivatives are generated from media with the “original files” term selected. when an original file is uploaded, if the node that owns it has an “image” model, image derivatives are created.” – media – derivatives. islandora/documentation. [accessed 2019 dec 12]. available from: https://islandora.github.io/documentation/user-documentation/media/#derivatives [19] proposed technical roadmap. islandora/documentation. [accessed 2019 dec 12]. available from: http://bit.ly/2eaisqy [20] mayo c, jazairi a, walker p, gaudreau l. bc digitized collections: towards a microservices-based solution to an intractable repository problem. 2019 [accessed 2019 dec 12];(44). available from: https://journal.code4lib.org/articles/14445 [21] pham k, clair k, reyes f, rynhart j. 2019. in a third space: building a horizontally connected digital collections ecosystem. proceedings of open repositories; 2019 june 10-12; hamburg. available from: https://lecture2go.uni-hamburg.de/l2go/-/get/v/24782. see more information on the university of denver library system’s front-end and back-end is available at https://github.com/dulibrarytech [22] fast in every way that matters. [accessed 2019 dec 12]. available from: https://www.gatsbyjs.org/ [23] collectionbuilder. [accessed 2019 dec 12]. available from: https://collectionbuilder.github.io/collectionbuilder-gh/ [24] wax. [accessed 2019 dec 12]. available from: https://minicomp.github.io/wax/ [25] ubleipzig. react discovery. 2019 sep 5 [accessed 2019 dec 12]. available from: https://github.com/ubleipzig/react-discovery [26] biblissima. iiif collections of manuscripts and rare books. [accessed 2019 dec 12]. available from: https://iiif.biblissima.fr/collections/ [27] poulter m. build your own digital bodleian with iiif and sparql. 2019 may 30 [accessed 2019 dec 12]. available from: https://blogs.bodleian.ox.ac.uk/digital/2019/03/13/build-your-own-digital-bodleian-with-iiif-and-sparql/ about the authors kelli babcock has been the digital initiatives librarian with the university of toronto libraries since 2013. she coordinates the collections u of t and discover archives services, in addition to a number of other digital projects at utl. sunny lee is a digital initiatives application programmer analyst at the university of toronto libraries. she is currently working on the development and support of web-based applications for a wide range of its projects, with a particular focus on digital collections. jana rajakumar is an application programmer analyst at the university of toronto libraries. his recent efforts include building the iiif presentation api used by collections u of t and other digital projects, revamping the mymedia service (an in-house media streaming service) and containerizing existing and new applications using docker. andy wagner works as a senior software engineer and the repository architect at the university of toronto libraries and is involved in helping the library and the university preserve its growing body of digital content. andy also coordinates the utl mymedia service. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – subject guides & more: creatively transforming an open source management system mission editorial committee process and structure code4lib issue 12, 2010-12-21 subject guides & more: creatively transforming an open source management system this article describes the implementation of subjectsplus to manage the subject guides at the wichita state university libraries. the decision to implement an open source solution, the implementation process, and customizations to the software are discussed. in addition to the subject guides, subjectsplus is also used to manage course specific and miscellaneous topic guides, the library staff directory, and database links. the article also covers the reception of subjectsplus by the librarians and teaching faculty. by gemma blackburn and mary walker introduction during the summer of 2010 the library at wichita state university, a medium sized academic library, implemented the subjectsplus subject guide management system during a complete overhaul of the library’s web presence. subjectsplus is an open source content management system developed by andrew darby at ithaca college, and is based on the pirate source system developed at east carolina university [1]. subjectsplus was chosen over other systems in part because its open source nature allowed us to modify the system to best fit the needs of our library. subjectsplus is also able to manage other online library resources, including course specific and miscellaneous topic guides, the library staff directory, and database links. it has proven to be a valuable time-saver in all three areas, but most especially in the management of database links. implementation of the system took approximately three months of part time work, with an additional two months for librarians to enter content and build the guides. a new system a new way of delivering subject guide material was long overdue. librarians were maintaining subject web pages and paper subject guides. the collection development department kept track of the user names and passwords for those pages so that if a librarian should leave, the next person could access and edit the pages. staffing in collection development has been reduced and the responsibility of tracking the access information had become an added burden.  the database listings, both alphabetic and subject-oriented, needed to be kept up-to-date but we needed a more efficient way of doing this. since this was done manually, and database information changes frequently,  it was incredibly time consuming for the electronic resources librarian to stay on top of these updates. because each librarian has his or her own style and html/css skill level, the look of the subject pages and the type of information found on the pages varied greatly. in addition, some librarians were not comfortable adding multimedia content to their pages.  the engineering subject page, which was created in adobe flash by a graduate student, created a further complication.  after the student left, the expertise and support files were no longer available, making updating the guide nearly impossible. we also needed a better way to connect the electronic databases to their subject areas and a better way of maintaining the lists of database links on the library website. previously, the library maintained two lists of links (an alphabetical list of database by title, and a list of database by subject area) in static html pages, as well as 49 subject guides with sections for primary and additional database listings. if a database was added, removed, or had updated information, the two lists required modification, along with all the pertinent subject pages. we needed a better way to handle all of this information. we desired a solution that had a uniform appearance, was easy to maintain, and was discoverable by our students. we knew we wanted a system that would be more user-friendly for librarians, and be web 2.0 compatible, while also bringing consistency to our current environment for a smoother transition for both the students and the librarians. subjectsplus budget concerns at wsu made the cost of an open source solution more attractive than a commercial solution. the university computing and telecommunications department (where the library’s servers are maintained) already managed several lamp (linux apache mysql php) applications for the library, so additional hardware and expertise was not required for the mysql/php driven subjectsplus. the open source library a la carte system from oregon state university [2] was also considered, but was not chosen because there are no other ruby on rails applications currently used at wsu, and therefore no available expertise to install and maintain the system. the implementation of library a la carte would have taken more resources (especially time) to achieve the library’s goals. the popular libguides system [3] was also considered, but the cost of the system, and the additional addons that we would have required from springshare, would be an additional budget concern for the library. we also found the appearance of libguides to be cluttered. because visual customizations to this hosted system are limited it would be impossible to make the guides consistent with the rest of the library’s website. the ease of using subjectsplus was beneficial to librarians with busy schedules and varying skill levels.  multimedia elements are especially easy to embed.  minimal training was required, and librarians felt confident in using subjectsplus with the majority of guides being finished within 90 days. the biggest advantage of subjectsplus is the ease with which it can be extended to meet the needs of any library. expanding subjectsplus is only limited by the expertise available at wsu. also, the subjectsplus community, organized through a google group, has proven to be a helpful source of support and information in implementing and expanding the system [4]. database access before subjectsplus, the electronic resources librarian maintained static web pages with links to all library databases, organized by title and by subject on the library website. included were fifty databases by subject pages which displayed the database name, access type (off campus, open, library only, or limited access), a link to the database description page, the major subject area, the vendor (some with links to the vendor’s site), and indicators for “full text”, “some full-text”, and/or a “link to the wsu catalog”. the electronic resources librarian maintained these pages using microsoft frontpage.  using the find and replace feature to update url changes or delete databases from the lists is not too difficult.  however, care must be taken to update only relevant urls found on the current database pages.  archive copies of the alphabetic list were made before major changes were implemented. adding a new database was more time consuming. not only did the database and all related information need to be added to the alphabetical page, but it also needed to be added to each appropriate subject page either as a primary database or as an “other useful database.” an email would be sent out to the subject librarians informing them that the database was now available and soliciting their input on what subject guides to include it on either as a primary or secondary resource. updates would be made as responses were received. using subjectsplus has streamlined this process.  now we have only  one record to add, delete, or modify and have reduced the number of pages requiring updates by at least 50%.  web of science, a popular general database, appeared in 32 of the static html database pages, requiring 32 files to be updated if the information about this database changed.  now it is changed in only one place. adding a new database is done by filling out a form (figure 1); knowledge of html is no longer necessary. updating a record is as simple as logging in, retrieving the database record, making the necessary updates, and clicking on the ‘save’ button. deleting is just as easy; once the record is returned, just click the ‘delete’ button. figure 1. the new record form when adding a record, an email is still sent by the electronic resources librarian to the subject librarians but this is only to inform them that the record is now available for them to add to their own subject pages. subject librarians add the resource to individual subject pages by using the newly created record. they associate their subject guide with that record by selecting their subject from the list of subjects which allows the record to display on their subject page.  they also select a source type to define that record in their guide (choosing either databases or databases – supplemental), and can also choose to override the default description of the database with their own. the benefit of entering, updating, or removing information from one place was long overdue.  a process that took 45 minutes per record to add, now only takes about twenty.  updating takes only a few minutes and deleting takes seconds. the main downside to using subjectsplus instead of a static website is that archiving older versions of our database listings for historical or practical purposes is not a built-in feature. however, it is possible to keep expired records but not have them display publicly. customizing subjectsplus while subjectsplus will work fine out-of-the-box, its openness allows libraries to build upon and customize the system as much or as little as desired.  some alterations made to subjectsplus at wsu were based on the need of the library, and others were intended to add value to already existing library services by integrating them into the subject guides. some of the customizations were straightforward to implement and are covered in depth in the subjectsplus documentation or had previously been implemented by other subjectsplus libraries, and some modifications required changes to the database structure and additional php files. adding pluslets the guides within subjectsplus are built by dragging and dropping boxes, called pluslets, onto a guide canvas (figure 2). there are numerous pluslets readily available in subjectsplus, including twitter, flickr, del.icio.us, and generic rss feed displays, a base pluslet containing the fckeditor for custom web content, and many others. figure 2. the blank guide canvas because of the open source nature of subjectsplus there are limitless possibilities for additional pluslets. the subjectsplus wiki [5] provides an example of an added pluslet to embed a meebo chat box along with instructions to create it.  this pluslet allows a subject librarian to add live chat to any guide, giving point-of-need service to students. the top priority custom pluslet for the library was an online catalog search box. the catalog search and the meebo box pluslets were both achieved by creating a new php file incorporating static html, and linking the new pluslet to the guide editor so it becomes available to drop onto the canvas. new pluslets are made available in the guide editor by adding them to the get_pluslets.php file. another custom pluslet created was a search box for the libraries’ federated search engine.  this concept proved to be more complex than the catalog search box as we desired more than a simple static form. the wsu libraries use the serials solutions 360 search federated search engine, which was locally branded harvester article finder. harvester gives patrons the option to search most of our databases (104 of our 110 databases) all at one time, to search a specific subject grouping of our databases as defined by our subject librarians, or to select specific databases to cross-search. in order to make the harvester search box pluslet as relevant to each subject guide as possible, it was preferred that each guide would contain a box to search only those databases relevant to that guide. for example, the harvester search box on the mathematics & statistics subject guide would search only those databases defined as relevant to mathematics & statistics. the subject-specific harvester search boxes can be created using the 360 search catid – a five digit code used to define each subject group. in order to create the harvester search box pluslet, first we added a new field (harvester_id) to the subjectsplus database to hold the catid.  then we created a new pluslet php file to build the search box and make an sql call to the database to retrieve the correct catid for the appropriate subject guide.  the final step was to add input fields to the staff interface for the creating new guide form (figure 3). now the librarian may choose to add the harvester catid to each guide as it is created in the create new guide form. a lookup screen with each subject and the corresponding catid is linked to the form for reference. once the harvester pluslet is added to the guide it will use the selected catid to build the subject-specific search box (figure 4). the harvester pluslet: <?php ///////////////////// // harvester info // grabs 360 search catid and throws it into search box ///////////////////// global $proxyurl; global $check_this; if (isset($_get&#91;"subject_id"&#93;)) { if (is_int($_get&#91;"subject_id"&#93;)) { $subject_id= $_get&#91;"subject_id"&#93; ; } else { $subject_id = 1; } } elseif (isset($check_this)) { $q1 = "select harvester_id from subject where shortform = '" . $check_this . "'"; $r1 = mysql_query($q1); $subject_id = mysql_fetch_row($r1); $subject_id = $subject_id&#91;0&#93;; } else { $pluslet .= "<p>you will see results when you return to this guide later.</p>"; return; } $harvester .= "<form action=\"http://proxy.wichita.edu/form?qurl=http://dn3kg6nn2s.cs.serialssolutions.com/resultframeset.jsp\" name=\"searchform\" method=\"post\" target=\"_blank\" /> <input type=\"hidden\" name=\"ss_libhash\" value=\"dn3kg6nn2s\" /> <input type=\"hidden\" name=\"catgrouplist\" value=\"default\" /> <input type=\"hidden\" name=\"searchby\" value=\"database\" /> <input type=\"hidden\" value=\"keyword\" name=\"field\" /> <input type=\"hidden\" name=\"catid\" value=\"$subject_id\" /> <input type=\"text\" name=\"term\" id=\"articlebox\" size=\"28\" value=\"enter search terms\" onclick=\"document.searchform.term.value='';\" /> <input type=\"submit\" name=\"submit\" value=\"search\" /></form>"; $pluslet .= $harvester; ?> figure 3. create new guide form with harvester id field figure 4. the harvester search box wsu uses several user-built extensions for the voyager ils, including the new books list feature created by michael doran at the university of texas at arlington library which allows users to search only the most recent books added to our collection [6]. jim robinson, of the tarrant county college library, later extended this feature with rss feeds of the items within the new books list, sorted by library of congress class [7]. a new books list rss feed pluslet was built for subjectsplus to integrate these rss feeds into the subject guides. this pluslet allows the librarian to enter lc call number classes and choose the number of items to display. the pluslet will retrieve the latest items from the new books list that fall into those classes, and display them in the subject guide as links to the catalog records. the new books pluslet was created differently than the other custom pluslets, as a new php file was not needed. there are several files in subjectsplus that handle the preexisting rss pluslets (general rss, twitter, del.icio.us, and flickr). additions were made to these files by copying and modifying the code for the other rss pluslets for the new books pluslet (figure 5). additions made to guide_functions.php for new books pluslet: case "newstuff":    print $pluslet;    print "<br /><input type=\"text\" id=\"$this_instance\" name=\"new-$itemtype-body-$idnum\" value=\"$body\" class=\"required_field\" size=\"$title_input_size\" /><br />;    print "<p style=\"font-size: 11px;padding-top: 3px;\"> $vrss1 <input type=\"text\" id=\"\" name=\"displaynum-$idnum\" value=\"$num_items\" size=\"1\"  style=\"font-size: 8px;\" /></p>       <div>\n";    print $close_pluslet;    return;    break; case "newstuff": $feed = trim($body);    $pluslet .="<p class=\"find_newstuff\" name=\"http://libcat.wichita.edu/vwebv/new_books_subjects.cgi?sort=week&subject1=$feed|$extra\">$vrssloading</p>";    break; additions made to guide.php for new books pluslet: $(".find_newstuff").each(function(n) {    var feed = $(this).attr("name").split("|");    $(this).load("includes/feedme.php", {type: 'rss', feed: feed[0], count: feed[1], show_desc: feed[2], show_feed: feed[3]}); }); additions made to local.js for new books pluslet: $(".find_newstuff").each(function(n) {    var feed = $(this).attr("name").split("|");    $(this).load("../../subjects/includes/feedme.php", {type: 'rss', feed: feed[0], count: feed[1], show_desc: feed[2], show_feed: feed[3]}); }); figure 5. new books pluslet visual customizations one of the benefits of a locally hosted, open source system is the ability to make that system look exactly like the library website and conform to campus-wide visual standards. this was easy to accomplish in subjectsplus. using the firebug addon for firefox to test new styles and compare our library website with subjectsplus side-by-side, the visual customizations were uncomplicated and quick to achieve. the most significant visual alteration was creating a visual separation of the right-hand sidebar column from the main column content (figure 6). the pluslets in each column contain the same css classes, so separating them was achieved by using descendant selector rules [8]. the sidebar pluslets exist within <div id=”sidebar”>, so a second set of rules was added by defining all classes of the pluslets within the sidebar separately (figure 7). example of css variations: titlebar_text { background: transparent; color: #fff; font-weight: 700; } figure 6. main column and sidebar before css variations #sidebar .titlebar_text  { background: #eee; color: #000; font-weight: bold; } figure 7. main column and sidebar after css variations databases by subject page since  wsu decided to use subjectsplus to manage our electronic database links, it was essential to maintain an experience in subjectsplus similar to the end user’s experience using the database web pages. subjectsplus produces an alphabetical list of all databases, including their descriptions and a search box for locating a database.  however, out-of-the-box subjectsplus does not contain a way to browse databases by subject – something the patrons at wsu are used to. a database by subject page was created using a three step process: an array of parent subject areas (humanities, business, social sciences, etc.) was defined and added to the config.php file. the values in the array were added as a drop down box option to the create new guide form (with minor changes made to the database structure to handle the new field). a new page was created to display a direct link to the databases section of each subject guide (using an anchor link common to each guide), sorted by the array values. this page, which was created by copying and modifying the main subjectsplus index page, filters out the topic and course guides, displaying only links to the databases by subject area. ‘area’ array added to config.php: $area = array("business", "education", "engineering", "fine arts", "general", "government", "health sciences", "humanities", "sciences", "social sciences"); creating the links by subject area: foreach ($area as $value) {  //display parent subject areas $options = ""; $leftcol .= " <div class=\"pluslet\"> <div class=\"titlebar\"><div class=\"titlebar_text\">$value</div></div> <div class=\"pluslet_body\">"; // call all subject guides (not topic or course guides) that are live for each parent subject $q = "select subject, subject_id, shortform from subject where active = '1' and area = '$value' and type= 'subject' order by subject"; $r = mysql_query($q); while($myrow =  mysql_fetch_array($r)) { //create a link to the databases section (anchor #3) for each subject guide   if ($mod_rewrite == 0) {       $link = "subjects/" . $myrow[shortform] . '#3';    } else {       $link = "guide.php?subject=" . $myrow[shortform] . '#3';    } //display the links with the subject guide name  $leftcol .="    <a href=\"$link\">$myrow[0]</a>&nbsp;\n "; } reactions to the change the librarians were excited about the potential that subjectsplus pages held. they were apprehensive about the time it would take to learn the new software and complete the task at hand. once they received training and began using subjectsplus, they realized it was not as daunting as they had first perceived it to be. they also realized that teaching bibliographic instruction sessions would be easier because all the information for a particular subject would be located on one page rather than in two places on the library website; on a subject page and on a databases by subject page. when the idea of course guides was introduced to a group of non-library faculty members at a blackboard training session, an education professor was very eager to work with a librarian on developing a guide for her course. since then eight course guides have been created: bio 210, hp 330, chem 532, psy 328, me 251, engl 102, comm 111, and lasi 170. these guides have eliminated the need for lengthy handouts and paper copies. the librarians created each course guide for its own purpose.  some teaching faculty have a specific need that they want addressed by the library resulting in highly structured guides that complement a class and contain class assignments. other course guides are more general and contain only a list of resources to aid students with research projects. for example, the bio 210 (general biology 1) course guide was created to help with a specific library assignment created by the instructor for that class. there were nine sections of the course being taught in fall 2010, with a total of 213 students. the guide was created to reflect the previous paper copy of the assignment with links to resources that could be used to answer the questions. it also included additional help and tips. having the assignment in a course guide instead of on paper was more convenient for the students. librarians have created several topic guides, including children’s literature, reader’s advisory & leisure reading, sac & sag movie collection, and style manuals, to name a few. the librarian for juvenile literature has received positive feedback from a curriculum and instruction professor who said, “i love it!” the professor is also recommending her class bookmark the course guide and is planning to make assignments from this guide in a future course. the sac & sag movie collection and reader’s advisory & leisure reading guides have also received positive feedback. plans for the future overall the wsu libraries is happy with the outcome of subjectsplus, but there is still room to grow. many of the future plans hinge on the addition of new features in upcoming releases of subjectsplus. a recent survey of the subjectsplus community suggested the following enhancements:  a link checker, a mobile interface, and better integration with course management systems such as blackboard [9]. all of these would be useful in meeting the current initiatives of our library, including expanding the ‘embedded librarian’ concept by creating more library content that can be easily embedded into blackboard, and developing more mobile-friendly services. other plans include the creation of more custom pluslets to enhance services, such as a ‘resource of the month’ pluslet to promote specific databases, and additional collaboration with teaching faculty to create more course-specific guides. library administration desires that a course guide be created for each class with a high dwif (drop/withdrawl/incomplete/fail) rate to aid the university in raising success in these courses. you can view wsu’s new library guides at http://libraries.wichita.edu/subsplus/ acknowledgments we wish to acknowledge kathy downes, nan myers, and stephanie sauls who were also members of the subjectsplus implementation team at wsu libraries, as well as andy speagle, systems & storage administrator in the university computing & telecommunications department. references [1] darby, a. 2006. implementing an open source application in a college library: ecu’s pirate source. college & undergraduate libraries 13(1): 41-52 [2] library a la carte – http://alacarte.library.oregonstate.edu/ [3] libguides – http://springshare.com/libguides/ [4] the subjectsplus community group – http://groups.google.com/group/subjectsplus [5] the subjectsplus wiki – http://www.subjectsplus.com/wiki/ [6] michael doran’s new books list for voyager – http://rocky.uta.edu/doran/newbooks/ [7] jim robinson’s new books rss feeds for voyager – http://lib-serv.tccd.edu/code/webvoyage/new_books_subjects.php [8] more information about descendant selectors (previously called contextual selectors) in css can be found here – http://www.w3.org/tr/css2/selector.html#descendant-selectors [9] this survey was sent to subjectsplus users in september, 2010. the results are available here: http://www.subjectsplus.com/wiki/index.php?title=survey_results about the authors gemma blackburn is assistant professor & library systems developer at wichita state university (gemma.blackburn@wichita.edu) mary walker is assistant professor & electronic resources librarian at wichita state university (mary.walker@wichita.edu) subscribe to comments: for this article | for all articles one response to "subject guides & more: creatively transforming an open source management system" please leave a response below: ashutosh mishra, 2023-05-03 when i install subjectsplus i found this error the requested url sp-3/control/install php was not found on this server apache/2.4.9 (win64) php/5.3.12 server at localhost port 80 leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – comparing the librarything, oclc, and open library isbn apis mission editorial committee process and structure code4lib issue 21, 2013-07-15 comparing the librarything, oclc, and open library isbn apis librarything, oclc, and the open library project all provide isbn services that take an isbn and return information about related works for that isbn. this article compares the license terms, quality of the data returned, documentation available, and details of the apis for the three products. the article focuses on ease of use for developers, restrictions on types of use (and assumptions about types of use built into the license or terms of use), and cost of the services. by david fiander introduction oclc, librarything, and the open library all provide web services for finding basic information about books based on their isbns, but the services all have different apis, provide different types of information, and have different license terms which may restrict the ways that they may be used. this work arose from a research project involving comparing the private collections of faculty on campus to the holdings of the university library, to determine what the service gap was between what faculty required for their research and what the library was providing. in order to reduce the amount of manual labour involved in the comparison, we collected isbns from the faculty’s books (for those books that had isbns) and want to compare those isbns to the library’s collection. however, the fact that even the “same” book might have multiple isbns associated with it (e.g., cloth vs paper bindings, or british vs american publishers) complicates this comparison. so we wanted to use a service like oclc’s xisbn service to determine the set of “related” isbns and check those against the library’s collection, if the isbn gathered from the faculty collection was not in the collection directly. looking for the best way to do this turned into an exploration of the balance between usability of the api, the services’ terms of use, and the quality and amount of the bibliographic data available from each of the services. the question of what editions of books are “related” to each other is a complicated one. for all the examples in this paper, i am using the 15th edition of the chicago manual of style as my starting point. at one level, all the different editions of the chicago manual are related, since they are all the chicago manual, but they are also all different books, and it is not necessarily the case that one edition can be substituted for another (for example, the 13th edition contains information about the typesetting process that is omitted from later editions, and the 15th edition includes extensive revision to citation style related to citing online journals an other works). for the purposes of this study, what matters is consistency in the treatment of editions: either all editions of the manual should be treated as “the same”, or they should all be distinct. basics of the apis librarything librarything (http://librarything.com) is a social platform for book lovers that allows them to catalog their personal libraries. according to librarything’s “zeitgeist” page, it has over 1.5 million members, and has cataloged over 7.5 million unique items. librarything provides seven apis, each of which serves a distinct function and has separate license terms and conditions of use (librarything, 2012). librarything is not a general purpose bibliographic database; most of the apis that it provides are designed to provide information about an individual member’s personal collection, rather than from the full librarything catalog. i will be focusing on the “thingisbn” api in this article. the thingisbn api takes a given isbn and returns an xml-formatted set of “related isbns”, which are the isbns of other formats or editions of the book that has the given isbn. the output is identical to that of the original oclc test xisbn service (described below). it provides only isbn information, and there is no way to get other bibliographic data from librarything in general, although it is possible to find out information about the books in an individual user’s personal collection. $ curl -s "http://www.librarything.com/api/thingisbn/0226104036" <?xml version="1.0" encoding="utf-8"?> <idlist> <isbn>0226104036</isbn> <isbn>0226103897</isbn> <isbn>0226103900</isbn> <isbn>0226104206</isbn> <isbn>0226104044</isbn> <isbn>0226770087</isbn> <isbn>0226104052</isbn> <isbn>0226104176</isbn> <isbn>9562913961</isbn> <isbn>1434102831</isbn> </idlist> the thingisbn api will accept isbns either with or without dashes in them (0-226-10403-6 and 0226104036 are both acceptable), but the isbns returned will never have dashes. it also accepts both isbn-10 and isbn-13, but the list returned always consists of isbn-10s (once there are books that have isbn-13s that cannot be converted back to isbn-10s, this will obviously have to change). if the identifier passed to the api is not a valid isbn, or is not in the librarything database, then it is flagged as an “unknownid”. oclc oclc launched its xisbn service as research project in january of 2004 (oclc research, 2004). at that time it was a very basic service that took an isbn and returned an xml-formatted list of “related” isbns. this is the model on which librarything’s “thingisbn” service was based. in 2007, the research project service was downgraded and the production service was launched (hellman, 2007). the production service provides not just the list of related isbns, but also options for retrieving other bibliographic data about each of the items returned. the api allows the user to request just those fields that they require. the data may be returned formatted as csv, text (i.e. tab delimited), xml, xhtml, json, python, php, or ruby. the default format returned is xml (oclc, n.d.). $ curl -s "http://xisbn.worldcat.org/webservices/xid/isbn/0226104036? method=geteditions" <?xml version="1.0" encoding="utf-8"?> <rsp xmlns="http://worldcat.org/xid/isbn/" stat="ok"> <isbn >0226104036</isbn> </rsp> the api will accept isbns with or without dashes. the isbns returned will be in the same format as in the request. similarly, the length of the isbns returned will match that in the request: a request for an isbn-10 will return a list of isbn-10s; a request for an isbn-13, a list of isbn-13s. if the identifier passed to the api is not a valid isbn, then the “stat” attribute in the response (as shown above) will have the value “invalidid”; if the isbn is valid but not in the oclc database, then the “stat” attribute will have the value “unknownid”. while the oclc xisbn api will return basic bibliographic information about the item identified by the given isbn, oclc also provides a separate, more complete, “oclc search api”, which is governed by a separate license agreement and requires a separate registration process before it can be used. the complete metadata available through the xisbn interface is { "stat":"ok", "list":[{ "url":["http://www.worldcat.org/oclc/183927955?referer=xid"], "publisher":"university of chicago press", "form":["ba", "da"], "lccn":["2003001860", "20031860", "92037475"], "lang":"eng", "city":"chicago", "author":"[john grossman]", "ed":"15th ed.", "year":"2003", "isbn":["0226104036"], "title":"the chicago manual of style.", "oclcnum":["183927955", "186032987", "254766064", "255224549", "265518898", "270770568", "300224813", "316300181", "441827088", "465067945", "51553085", "636071725", "642131460", "696100286", "716977182", "731426594", "731438337", "732317662", "733751229", "748774915", "760775753", "779265103", "781185293", "781436125", "782550638", "783145177", "787879695", "789898520" ] }] } open library the open library api was designed as the primary programming interface to the open library’s catalogue of bibliographic data, including related works. because the api is primarily a catalog api, it takes two steps to find the complete set of related isbns: find out the bibliographic information about the given isbn (that is, we are doing a “known item” search of the catalogue based on the isbn). this search returns the isbns of other formats of the specific edition (such as cloth, paper, or electronic), plus the open library “work identifier” for the over-arching “work” that encompasses the item we began with. this api can return all of the bibliographic data known about the item identified by the isbn, but in this case, i am only asking for the work identifier. $ curl -s "http://openlibrary.org/query.json?type=/type/edition &isbn_10=0226104036&works=" [ { "works": [{"key": "/works/ol15979229w"}], "key": "/books/ol3672021m" } ] use the work identifier discovered in step 1 to retrieve the list of isbns linked to the work, including all the various editions. $ curl -s 'http://openlibrary.org/query.json?type=/type/edition &works=/works/ol15979229w&isbn_10=&isbn_13=' [ { "isbn_13": null, "isbn_10": [ "0226104036", "0226104052", "0226104044" ], "key": "/books/ol3672021m" } ] the open library query api, used above, is very rich, with several options for specifying the item for which one is searching, and the type of data that should be returned, but it seems that it only returns data in json format. the open library also has a simple restful api that allows one to get information about a known item based on its open library identifier, whether that’s a “book”, “work”, or even “author”. this interface does provide support for returning data in json, rdf, and yaml. the query api provides far more information about an item than the restful api does. for example, the full set of information available from the query api for the work “ol1597922w” is [ { "identifiers": { "librarything": ["7335"], "goodreads": ["103362", "4657649", "332548"] }, "subtitle": "the essential guide for writers, editors, and publishers", "weight": "3.3 pounds", "covers": [6868096, 140171, 140169], "lc_classifications": ["z253 .u69 2003"], "latest_revision": 11, "uri_descriptions": [ "publisher description", "contributor biographical information", "table of contents" ], "languages": [{"key": "/languages/eng"}], "genres": ["style manuals."], "source_records": [ "marc:marc_records_scriblio_net/part14.dat:135267575:1040", "marc:marc_cca/b10621386.out:42100111:1631", "marc:collingswoodlibrarymarcdump10-27-2008/collingswood.out:59082004:2424", "marc:marc_ithaca_college/ic_marc.mrc:195612531:1390", "marc:marc_ithaca_college/ic_marc.mrc:195613921:1390" ], "title": "the chicago manual of style", "edition_name": "15th ed.", "subjects": [ "printing -style manuals", "authorship -style manuals" ], "publish_country": "ilu", "type": {"key": "/type/edition"}, "uris": [ "http://www.loc.gov/catdir/description/uchi051/2003001860.html", "http://www.loc.gov/catdir/bios/uchi051/2003001860.html", "http://www.loc.gov/catdir/toc/uchi051/2003001860.html" ], "physical_dimensions": "9.5 x 6.4 x 2.4 inches", "revision": 11, "publishers": ["university of chicago press"], "description": { "type": "/type/text", "value": "includes bibliographical references and index." }, "physical_format": "harbcover", "last_modified": { "type": "/type/datetime", "value": "2011-07-29t04:01:51.327875" }, "key": "/books/ol3672021m", "publish_places": ["chicago, usa"], "lccn": ["2003001860"], "pagination": "xvii, 956 p.", "classifications": {}, "created": { "type": "/type/datetime", "value": "2008-04-01t03:28:50.625462" }, "url": [ "http://www.loc.gov/catdir/description/uchi051/2003001860.html", "http://www.loc.gov/catdir/bios/uchi051/2003001860.html", "http://www.loc.gov/catdir/toc/uchi051/2003001860.html" ], "notes": "includes bibliographical references (p. [863]-879) and index.", "number_of_pages": 956, "dewey_decimal_class": ["808/.027/0973"], "isbn_10": ["0226104036", "0226104052", "0226104044"], "publish_date": "2003", "copyright_date": "2003", "works": [{"key": "/works/ol15979229w"}] } ] but the information provided by the work itself, at the url http://openlibrary.org/works/ol15979229w.json, is merely { "title": "the chicago manual of style.", "created": { "type": "/type/datetime", "value": "2011-07-29t04:01:15.259393" }, "last_modified": { "type": "/type/datetime", "value": "2011-07-29t04:01:15.259393" }, "latest_revision": 1, "key": "/works/ol15979229w", "authors": [ { "type": {"key": "/type/author_role"}, "author": {"key": "/authors/ol3236644a"} } ], "type": {"key": "/type/work"}, "revision": 1 } data quality the idea that different editions of a book are merely manifestations of a platonic “work” that encompasses all the editions is a core part of the functional requirements for bibliographic records (frbr), and is essential for my research. the apis give us a simple way to determine the quality of the bibliographic data held by each of the services, relative to this idea of clustering editions. as a simple test of the quality of the data provided by each of the services, i explored what related works they each provided for the hardcover 15th edition of the chicago manual of style, isbn 0-226-10403-6, (chosen primarily because it’s convenient to me, common, and many libraries will hold several editions). librarything because librarything’s data is created by individuals cataloging their own collections, the coverage is somewhat idiosyncratic. for example, librarything only records one copy of the ieee posix 1994 standard, while worldcat documents over sixty libraries that own it. on the other hand, librarything has a record for the chapbook “oracle gretel” by julia rios, but worldcat has no record of the book at all. librarything actively encourages its community to perform a form of crowdsourced authority control that ensures that authors whose names are recorded in a variety of ways, or who write under pseudonyms, are correctly linked together, and that related works are gathered together. this latter project makes librarything’s data a good fit for my research project. librarything returns the following isbns as all being related to the 15th edition of the chicago manual: 1434102831 (1st ed, reproduction) 9562913961 (9th ed) 0226770087 (12th ed) 0226103900 (13th ed) 0226103897 (14th ed, cloth) 0226104044 (15th ed, cd-rom only) 0226104176 (15th ed, windows cd-rom) 0226104036 (15th ed, cloth) 0226104052 (15th ed, cloth+cd-rom) 0226104206 (16th ed, cloth) oclc when searching oclc for the initial isbn, 0226104036, it reports that there are no other related isbns. searching oclc separately for each of the isbns returned by librarything, oclc reports that, in general, there are no related isbns for any of them, except for two cases: the isbn 9562913961, for the 9th edition, and the isbn 9568209492 are related. according to worldcat, the second isbn is for the book kiwala va a la selva, published in chile by editorial amanuta. the isbn 0226103900, for the 13th edition, and the isbn 0226104206, for the 16th edition, are related. the first case is probably a data entry error by an oclc member library; the second case is just odd. why are these two editions related, but none of the other ones? open library open library supports two levels of work relations: the different editions of the given title, such as cloth, paper, or electronic, as well the broader related works, which includes different editions, and translations. for this case, it returns the following isbns as being different formats of the 15th edition: 0226104036 0226104052 (cloth+cd-rom) 0226104044 (cd-rom only) like oclc, it does not return other editions of the manual as related works to the 15th edition. once again, starting with the librarything dataset of related isbns, open library reports that the isbn 1434102831 (the reproduction of the 1st edition) is unknown, and that all other editions are unique and unrelated to each other. documentation the librarything api documentation consists of one or two paragraphs for each api, but this is reasonable, given that the apis in question are all very simple with few options. the oclc api documentation is much richer, although the examples are somewhat monotonous, with all examples, except for one demonstrating the json callback format, using the xml format. the open library api is the richest one available, with several different interfaces, each having a variety of options related to the format and type of data being returned, but it is the api with the weakest documentation. this lack of documentation for open library means that developers must spend a great deal of time experimenting and exploring to find out what is available and exactly how it works. licensing librarything the librarything license dictates that it may only be used for non-commercial purposes, that an application can only access the api once per second, and that if a user is going to access the api more than 1,000 times in a day, that the user must contact librarything for approval. every api that librarything provides has a slightly different license, although they all conform to the basic outline above. the more problematic restrictions are those on the apis that provide access to information about the items in an individual librarything user’s collection. the licenses for these apis, the json books and json works apis, state that the application using the api “must be run as javascript on user’s browser, not fetched by a server; cannot be stored, except for browser caching” (librarything, 2012). oclc the oclc xisbn api is part of its family of “affiliate services”, and is regulated by the terms and conditions for those services (oclc, 2009). according to those terms and conditions, users of the xisbn api may only use it in applications to accomplish acceptable “purposes”, which are defined as, g. “purposes” means managing and enabling access to: (i) library services, library materials, library resources and information related thereto; and/or (ii) services, materials, resources or information of interest to library patrons. as well, section 3 of the terms and conditions gives a list of ways that we are not permitted to use the api, including “(iv) use of bots, spiders, or other automated information-gathering devices or programming routines to “mine” or harvest material amounts of data.” while prohibiting most of the activities listed in section 3 is perfectly reasonable, the conditions listed in clause (iv) are troubling, since for many researchers, including me, the point of the api is to develop an “automated information-gathering device[sic]”, and often to mine or “harvest” data. one can only wonder how much data oclc might consider to be “a material amount”. oclc also provides a list of “frequently asked questions on rights and responsibilities,” which partially addresses questions related to research at an institution by non-library faculty: 6. may an oclc member make available or transfer extracted worldcat data that represents or enriches the records for the member’s own holdings to an individual scholar for the purposes of supporting the scholar’s personal academic or scientific research or study? yes. please note, however, that the policy asks that in making extracted worldcat data available or transferring it, members consider such actions and subsequent uses of the transferred worldcat data (including copying, displaying, publishing, modifying, reformatting and/or creating works or services from) in the context of oclc member community norms, oclc’s public purpose and this policy’s intent. but again, oclc is assuming that access to their data will be mediated by the library, thus implicitly denying faculty researchers the option of building their own tools for data extraction and analysis. the definition of acceptable purposes and the restrictions on the activities that may be performed by developers combine to make many types of bibliographic research, and potential research areas in the newly growing field of digital humanities of borderline acceptability. the xisbn service is free for non-commercial use, without registering, for up to 1,000 requests per day. oclc cataloguing members are granted free access after registering for a subscription. other users (commercial use at any level, or non-oclc cataloguing members making more than 1,000 requests in a day) may subscribe to the service (oclc, 2007). the libx browser plugin uses the xisbn service to provide its “related titles” catalogue search feature, interpreting these terms to apply to the user running the plugin, rather than to the developer writing the plugin. that is, when i use the libx plugin, i am the one using the xisbn service for non-commercial personal research, not the libx developers. open library it’s difficult to find a license agreement on the open library website. the main page for the api documentation says, “please note that this api should not be used for bulk downloads” (open library, 2013), but then goes on to say that the site makes data dumps available on a regular basis, and that those dumps should be used if a copy of the entire data set is required. there is a licensing page; the entire content of that page is: the internet archive does not assert any new copyright or other proprietary rights over any of the material in the open library database. there may be existing rights issues on some contributions and in some jurisdictions. when it comes to community projects, the legal issues are, frankly, very confusing, but we are attempting to make a database that can be openly used for a wide variety of purposes. we appreciate all that have contributed. (open library, 2010) the footer of every page on the open library website says that users are subject to the internet archives terms of use. those terms of use say that access is granted for “scholarship and research purposes only”, and that users certify that their use will be “noncommercial and will be limited to noninfringing or fair use under copyright law” (internet archive, 2001). conclusions the web apis available for basic isbn searching are all reasonably comparable in terms of functionality, with librarything being the most minimal, and the open library being the richest, but the api by itself is almost the least important aspect of the selection process. each api is associated with a unique bibliographic database that has its own quirks and flaws, and is of a certain size, and the use of each api is governed by specific license terms that may rule out the use of the api, regardless of the suitability of the interface and the data. librarything has a strong database, that is constantly growing, and is being maintained by a large, active, user community, but the api returns almost too little data, and the terms under which one may use the api, especially the limits on the number of times it may be called daily and the speed with which one may call it, severely limit its use for extensive research projects, even if it is suitable for the type of bibliographic search functionality for which the libx browser plugin uses it. the open library has the most liberal license, and the most flexible api, but the dataset is small, inconsistent, error-prone, and no longer being maintained. oclc’s api fits into a good middle ground in terms of the metadata available via the api, and it has the largest bibliographic database available, but the terms and conditions placed on the use of the api bar certain users, and certain uses, from taking advantage of it. while the database is large and, for the most part, free of errors, it is not very consistent in its identification of related works. because my library is an oclc cataloging member, many of the books that i will be examining may be very specialized or obscure, and i will be processing potentially tens of thousands of isbns, i opted to use the oclc api, even though i don’t need the support for the additional bibliographic data. the ideal situation would be the oclc database, with librarything’s level of quality and linking between related works, and the open library’s terms and conditions. unfortunately, the real world dictates that developers for every project will have to assess for themselves which system is most suitable for their work. references hellman, eric. 2007. “oclc xisbn service is moving.” code4lib archives [cited 2013 jul 11]; feb 12. available from https://listserv.nd.edu/cgi-bin/wa?a2=ind0702&l=code4lib&p=r14329&1=code4lib&9=a&j=on&k=3&d=no+match%3bmatch%3bmatches&z=4 internet archive. 2001. “internet archive terms of use.” internet archive [cited 2013 jul 11]; march 10. available from http://archive.org/about/terms.php. librarything. 2012. “librarything apis.” librarything [cited 2013 april 30]; december 3. available from: https://www.librarything.com/wiki/index.php/librarything_apis. oclc. n.d. “worldcat web service: xisbn [oclc – worldcat affiliate tools]: api.” oclc worldcat [cited 2013 april 30]. available from: http://xisbn.worldcat.org/xisbnadmin/doc/api.htm. oclc. 2007. “worldcat web service: xisbn [oclc – worldcat affiliate tools]: home.” oclc worldcat [cited 2013 april 30]. available from: http://xisbn.worldcat.org/xisbnadmin/index.htm. oclc. 2009. “oclc affiliate services terms and conditions.” worldcat [cited 2013 july 11]; april 3. available from: http://www.worldcat.org/ldap/terms.jsp. oclc. n.d. “worldcat rights and responsibilities for the oclc cooperative—frequently asked questions” [cited 2013 july 11]. available from: https://www.oclc.org/en-ca/worldcat/community/record-use/policy/questions.html oclc research. 2004. “experimental service accepts an isbn, returns set of isbns associated with same frbr work; bookmarklet then uses to query library holdings.” oclc research [cited 2013 july 11]; january 26. available from: http://www.oclc.org/research/news/2004/01-26.html. open library. 2010. “developers / licensing.” open library [cited 2013 july 11]; march 10. available from: http://openlibrary.org/developers/licensing. open library. 2013. “open library restful api (open library).” open library [cited 2013 july 11]; february 28. available from: http://openlibrary.org/dev/docs/restful_api. about the author david fiander is the web services librarian at the university of western ontario where he is responsible for overseeing development of the western libraries website. he has been actively involved in open source development in libraries for over a decade, including developing for the evergreen ils project. before becoming a librarian, david was a software developer, focusing on networking protocols and standards-based software portability. subscribe to comments: for this article | for all articles 3 responses to "comparing the librarything, oclc, and open library isbn apis" please leave a response below: dan scott, 2013-07-18 on openlibrary: 1) i find it odd that the author omits any mention of the openlibrary read api, as it seems most appropriate for this use case and has been available for use since june 2011. see http://openlibrary.org/dev/docs/api/read for the official documentation on its usage. for example, http://openlibrary.org/api/volumes/brief/isbn/0226104036.json gives you the related isbns for the work in a single call. 2) under the licensing discussion, the author implies that the request to use data dumps is at odds with the request not to use the api to perform bulk downloads. rather than being a licensing matter, this is clearly a request for developers to minimize the processing resources required to support the api. therefore, to support the work of developers interested in using the openlibrary data, they make bulk data dumps available for use. 3) most seriously, the author states in the “conclusions” section that “the [openlibrary] dataset is small, inconsistent, error-prone, and no longer being maintained” – but the body of the article offers no supporting evidence for that rather severe conclusion. beyond that, there is a possible typo in oclc example #2: should “author”:”[john grossman]” be “author”:[“john grossman”]? jessamyn west, 2013-07-25 at open library we’d love some help with the api documentation if anyone would like to pitch in. many of the pages on the open library docs part of the site are editable by anyone with a login. not excusing the lack of documentation, just saying it would be great to have help. i, too, am curious about the relative sizes of the datasets. jrochkind, 2013-08-05 jessamyn, from my reading, the biggest documentation gap in this ol service is lack of clarify in the terms of use. that’s not something a volunteer can fix, it needs someone at ol to officially make things clear. for instance, is it really intended that ol api’s can only be used for “scholarship and research purposes only”? if that’s intentional, some guidance/examples as to what ol considers “scholarship and research purposes” would be helpful. leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – the tools we don’t have: future and current inventory management in a room reservation system mission editorial committee process and structure code4lib issue 41, 2018-08-09 the tools we don’t have: future and current inventory management in a room reservation system fondren library at rice university has numerous study rooms which are very popular with students. study rooms, and equipment, have future inventory needs which require a visual calendar for reservation. traditionally libraries’ manage reservations through a booking module in an integrated library system (ils), but most, if not all, booking modules lack a visual calendar which allows patrons to pick out a place and time to create a reservation. the it department at fondren library was able to overcome this limitation by modifying the open source booked scheduling software so that it did all of the front end work for the ils, while still allowing the ils to manage the use of the rooms. by denis galvin, mang sun, and hanjun lee background fondren library at rice university, like many other academic libraries, has done extensive renovation of its library space. it has modernized and renovated its building, and it has become a very popular place with students. part of the renovation has been to add more study rooms, and to equip them with attractive and functional furniture and state-of-the-art technology. as a result, the study rooms are some of the most in demand spaces in the library. the library has historically let students make advanced reservations on study rooms. the initial scheduling system was a paper ledger which was maintained by the circulation department. students made reservations by visiting the library and staff members would input them into the ledger. when the reservation time came, students would come to the circulation desk; staff would consult the ledger and checkout the key which corresponded to the room which had been reserved. in 2010 the library purchased an online room reservation system from evanced which was software as a service (saas). for the most part this system worked, but the keys which unlocked the rooms were still checked out through the ils. the library discovered numerous downsides to having an online reservation system which is not connected to the ils. when a student would show up to get the key for their room, circulation staff needed to check the reservation system to determine what room they had reserved. alternatively, if a user walked in, staff would have to look through the reservation system to check what rooms were available. they would then move to the ils and check the key to the room out for the length of the reservation. if a student did not show up for a reservation, the staff member would need to remove it from the online system or it would look like someone was in the room. sometimes reserved rooms were checked out accidentally to a walk-in user when the room had an existing reservation which belonged to another patron. there was nothing built into the system to stop this from happening because the system used to book the room had no relationship to the system used to check the key for the room out to the user. it’s entirely up to staff to match the ils with the online reservation system, they must maintain two systems. policy when we were evaluating systems, the important thing was for students to be able to make their own reservations online, but staff also had to be able to block rooms out for various reasons ranging from repairs to special programs. this meant having a system which allowed for mediated and unmediated reservations. the library wanted students to be able to make only one reservation per day, with reservations to be made no more than three days in advance. rooms have always been checked out to students through the integrated library system. rooms are represented by the key associated with them. when a student comes to the circulation desk, staff view their reservation and the key is checked out. students can have rooms for up to four hours. they can only have one reservation per day, but once the reservation is complete they may make another one. they are only allowed to make reservations up to three days in advance. the library has maintained this policy through every iteration of reservation system; paper through electronic. in an ideal world software used to manage library collections would have a reservation system that had a working booking module complete with an online calendar. it is necessary to have a visual representation of when materials are reserved, when they are in use and when they are available. if there are multiple materials, such as rooms, they need to be shown simultaneously so users can pick something that is available at the time they want it. there is no library software built at this time that does this. many booking modules that come with the ils do not have usable calendars. in sirsidynix’s symphony, the way the public books a resource does not show when that resource is in use. this means a patron selects a resource, chooses the date and time they want to use it, they click a button and the ils determines availability. if the item is in use or reserved at the time selected, the system displays a message that the material is not available at that time. there is no way to tell before selecting the resource whether it is available or not. essentially this means it is a guessing game. if a resource is highly used it might mean selecting dates and times over and over only to find out the item is not available. this is frustrating and unacceptable. the online “calendar” for bookings also doesn’t show related items at the same time; you need to view each item individually and attempt to create a booking. for most patrons this would not seem to be a workable system. literature review when ball state university decided they needed a study room reservation system, they built it from scratch as is covered by faust, hafner and seaton.[1] prior to building their room reservation system it was difficult for students to book a room. like a lot of libraries, they used many ways to book rooms prior to an online reservation system. they discuss pros and cons of buying versus building, and they come to the conclusion that nothing on the market would satisfy their needs. they walk the reader through the different iterations of the software and they discuss problems they encountered, such as long page loads when the schedule filled up. they choose to develop their software, openroom, around a combination of php and mysql with either iis or apache acceptable as a web server. doherty and white discuss the general issues that go into the selection of an online room reservation system.[2] they chose the aforementioned open source system “openroom” which was developed out of ball state university. the article looks at the problems that create the need for a reservation system such as equitable use and policy decisions. of the literature reviewed, this paper is the only one that looked at booking rooms through the ils. they choose not to because of the cumbersome nature of booking in the system. this article offers a good look at standard policy decisions which are relevant to any room reservation system. the university of virginia first evaluated open source tools before deciding they would be better off going with a commercial solution. in the article “selecting and implementing a patron-driven resource reservation system,” sallans, et al. chose a system that was in use by a lot of other academic libraries.[3] in their implementation, they found challenges which stemmed from the complexity of their organization, differing usage policies based on rooms/materials, and issues related to the local it environment. williams and hada discovered many of the same problems that other libraries have found when they started investigating room reservations systems for newly renovated spaces at queens college libraries.[4] after evaluating numerous tools that would need to fit their specific environment they chose the open source system openroom. the system was chosen because they were able to modify the underlying source code. they immediately ran into problems with deprecated code due to the fact that they were on a newer os which had updated code libraries. this resulted in a consultation between the team at queens college library and ball state university who had ceased active development of the code. it was agreed that queens college would take over the project, which they did, and with some modifications they were able to launch a successful prototype. (interestingly, at the time of this writing the openroom system has not had its code updated so it is once again carrying deprecated code; if you tried install it on an up-to-date operating system the libraries the code depends on would not match.) in the broadest context outside of a library, hohpe and woolf introduce and elaborate on enterprise integration patterns that can be adopted and used in projects.[5] they are based on messaging technologies that have been widely used in joining and connecting applications and services. unlike web services technology which relies on synchronous calls, messaging is a technology that drives asynchronous yet reliable interactions among programs by consuming data called message. in a typical messaging architecture, a program that sends message is called publisher while other programs that act on received messages are called consumers. as a practical example, the high volume messaging and streaming solution apache kafka is used at linkedin as a way to integrate different internal systems.[6] message-based application integration is the ideal tool to hook systems together. architectural considerations interesting applications rarely live in isolation.[7] it was a longstanding goal of the library it department to try to front the academic reservation system of the symphony ils with an open source calendaring reservation system. from the choices available, the most robust and familiar to the department was booked (formerly phpscheduleit). the library had been using phpscheduleit for a number of years as a standalone reservation system for the library’s computer equipment, and it was also in use by several campus units. it is one of the more popular choices as it has a large user community, is being actively developed, and is consistently publishing new versions. when choosing open source software it is wise to look for projects that are widely in use so that you don’t wind up an orphan on a piece of software with few users and no developers. booked met that criteria so it seemed like a safe choice. the major challenge of this project was the integration between the ils and booked. in order to accomplish this, the systems need to maintain two-way communications. reservation related transactions originating from the web interface must be sent to symphony and the ones performed in symphony must be sent back to the web interface. the channels would be running at a standard http (80) or https port (443). the key component of this integration was data synchronization. the first option we looked at was doing synchronization between symphony’s oracle database tables and booked’s mysql database. this idea was dropped because there were several potential complications. the first concern was that if for some reason the connection to booked’s underlying mysql database hung, the symphony databases might also hang, and anything that had the potential to slow down or hang the ils was not acceptable. the other concern was that this was probably against the terms of service in our contract with our ils vendor. the second thing we took some time looking at was messaging technologies to make data synchronization happen in real time. messaging technology uses queues to accept, store and dispatch messages. for this project, messages could be used to carry reservation related transaction data from the symphony ils to the booked system. the biggest challenge was how to capture in real time the reservation related transactions originating in the symphony database. there were several options, one of which was buying and implementing oracle golden gate, but the cost was too high, so it was quickly removed from consideration. another choice was to write our own oracle triggers to capture reservation transactions and send them into a messaging queue that would be deployed and running in parallel with symphony on the same server. this option still required tampering with the underlying oracle database which was also likely to violate our terms of service. passing out data captured by triggers was also a complicated procedure which required installation of additional oracle jar libraries. we were not very confident that we could figure this out within the defined project deadline or that those jar libraries would work reliably in the long run. the project team, somewhat reluctantly, agreed not to spend time on this solution, but it may be revisited in the future. the third option was to use web apis to do ils-to-booked data synchronization; we would implement this by developing web apis to do two-way data synchronization. there was concern that transactions would be missed since it seemed like a lot of back and forth between servers. everytime the ils updated it would need to send the new information to the room reservation system, and everytime the room reservation system updated it would need to send that information back to the ils, but that was the solution we went with. we customized the code on booked to call web apis that would be developed to wrap around symphony apis associated with the reservation module to sync the transaction data initiated from booked. the challenging part is synchronizing the ils transaction data, such as checked out room keys and booking information, to booked because there is nothing native to the ils which triggers external transactions. the workaround was somewhat primitive but works; set up a synchronization mechanism on booked to call the web apis driven by user need/events. synchronization and reverse synchronization of transactions is triggered any time a user opens the booked pages which display the reservation schedules of the study rooms. the web apis on ils side based on our synchronization strategy, one of the first things we did was define and implement web programming interfaces (web apis) with the ils. ed-douibi defines web apis as “offering one or more publicly exposed endpoints to a defined request-response message system (typically expressed in json or xml) for either a web server or a web browser.”[8] this is a simple and efficient definition of web apis, though it might be more appropriate to use the term web application over the term web server. in the diagram below, booked is the web application consuming the room reservation web apis we developed on the ils side. the apis provide the following five services, as determined by the value of query parameter “qt” or “op” in the returned json messages. qt=1: list reservations qt=2: create a reservation qt=3: list a user’s reservations. qt=4: remove a user’s reservations. qt=6: modify a user’s reservation figure 1. component diagram of web service apis in order to implement these five web apis, we used the perl programming language as it is already installed and in use on our ils server. to design the apis, we had two sides to handle. one is the web interface, booked, which takes requests and sends back replies. the other is the interaction with our symphony system. our approach on the ils side is to avoid intrusive operations that might go against the terms of service with our ils vendor. in order to achieve this design goal, we didn’t use any create, update, delete (cud) sql statements or its variant form like an object relational mapping (orm) tool in the api code to interact with the ils’ underlying oracle database. symphony provides a rich set of api commands and has advanced transaction architecture. the transaction architecture specifically allows some interesting operations to be performed without the need for the graphical client – symphony workflows. virtually any command that you can run through the ils graphical interface can be formatted and run through the api server of symphony. as a result, we didn’t need to tamper with the underlying oracle database and could avoid running into customer support/warranty issues with the ils vendor. in this application architecture, the ils api acts as middleware to relay requests to the database, and this is acceptable use of the system in the eyes of the ils vendor. within symphony, we are able to view the logged transactions for the following operations: create reservation, delete reservation, and modify reservation. we captured those lines of code and fed them back to the symphony api server. those captured lines were analyzed, abstracted and turned into parameters to be used as templates in a custom perl module we created, which we called symdrive. the following code snippet shows the class method function for adding a booking record via the symphony api server. this module is one of the key components of the web apis for moving reservation transaction data back and forth between booked and the ils. it can take dates, room ids, reservations ids, user ids, and generate corresponding transaction lines which are fed into the symphony api server in real time to create, delete or update reservations in the ils. based on the status of the code returned from the symphony api server, the web apis set request status to 0 (success) or 1 (failure) in returned json messages. although we don’t run any operations directly with the ils’ underlying oracle database, we do run read only sqlplus queries in order to achieve the quickest query result with the simplest coding logic. we are able to do this because read is not an intrusive operation. sub symaddbk { my $err_code=0;#0success, 1failure. my ($self,%val)=@_; my $userid=$val{'userid'}; my $start_time=$val{'start_time'}; my $end_time=$val{'end_time'}; my $cat_key=$val{'cat_key'};#system number of a room key record my $itemid=$val{'itemid'};#barcode of a room key my $resvdsk=$val{'resvdsk'}; #the following two lines are mystified to observe code ip of the ils vendor my $apiline1='...$userid...$start_time...$end_time...$cat_key...$itemid...$resvdsk...'; my $apiline2='...$userid...$cat_key...$itemid...$start_time...$end_time...$start_time...$end_time...$resvdsk...'; my $apistmt='echo -e "'.$apiline1.' \n '. $apiline2.'"|...sent to symphony api server';#the last part of the pipe is also mystified open(api_result,"$apistmt 2&gt;&amp;1 1&gt;/dev/null |") or die "can't run '$apistmt'\n$!\n";#only output standard error while() { chomp $_; if ($_=~ /\s+2\s+\$\(1463/) { $err_code=1; } } return $err_code; } the web application interface – booked fondren library’s room reservation system is based on the open source booked software. the original software’s code has been customized to allow for synchronization and integration with sirsidynix’s symphony ils. the systems use two way communications to send reservation information that is made through the web interface to the ils. anytime a user creates, edits or deletes a reservation via the web interface, the custom code built into the system sends reservation information (reservation begin/end time, user id, and study room number) to symphony via a custom built reservation api. when this reservation is made, both booked and symphony validate the reservation request. all general validation processes such as maximum time duration, duplicated reservation, maximum quota per user, or room availability at requested time are performed upfront by the web system (booked). once the general validations are passed by booked, the reservation request is sent to the symphony system for further validation such as a user’s ability to create a booking based on a qualifying user profile or status, library open/close time, or a room’s availability. if all these condition are met, the ils creates the reservation which includes a booking confirmation number. this booking confirmation number is valid in both booked and the ils and once it is created it will be passed back to the online reservation system. data synchronization schema there are two major synchronization processes as shown below. one is writing ils data into the room reservation system (booked) database, and another is vice versa writing reservation data into the ils system. from ils to room reservation system the following data will be synced from ils to room reservations: reservations made directly via ils system edited/deleted reservation directly via ils system room key checkout data released room key data from checkout released room key data from non-checkout after specified period of time from room reservation system to ils the f ollowing data will be synced from room reservations to ils: reservations made via room reservation system edited/deleted reservation via room reservation system figure 2. flow chart for data synchronization overview. data synchronization and the booking id the booking id is a concept which exists in the symphony ils. for every reservation made, a unique booking id is created. a synchronization script on the booked server requests data from the ils via the restful api, and returns data from the ils in the form of json metadata. the booking id is the functional key between the two systems. if a reservation comes from the ils to booked with no booking id the script recognizes the reservation as newly made and creates it. if a reservation has a booking id that already exists in the database, the script compares the beginning and ending time of the reservation. if the beginning and ending times are different from the existing reservation, the script recognizes the reservations as an edit. if the booking id only exists in the room reservation system database, but not in the ils system, the script recognize this as a deleted reservation. for (begin of metadata data from ils) until (end of metadata from ils) { if (reservation in ils does not exist in the room reservation system database){ insert the reservation into room reservation system } else if (reservation in ils exist in the room reservation system database) { if (begin & end time is different){ edit the reservation begin and end time in the system database } } } foreach ($booking as $key =&gt; $value) { $key_id = $value['keyid']; $bk_id = $value['bkgid']; $date_start = date('y-m-d h:i:s',(strtotime($value['begtime']) + $dst_adjust)); $date_end = date('y-m-d h:i:s',(strtotime($value['endtime']) + $dst_adjust)); $query = "select res.*, rev.*, inst.* from resources as res , reservation_resources as rev , reservation_instances as inst where res.resource_id = rev.resource_id and inst.series_id = rev.series_id and inst.reference_number = '".$bk_id."'"; $result = mysql_query($query); $row_length = mysql_num_rows($result); $sync_run = false; if($row = mysql_fetch_array($result)){ $ref_query = "select inst.*, series.* from reservation_instances as inst, reservation_series as series where inst.series_id = series.series_id and reference_number = '".$value['bkgid']."'"; $ref_result = mysql_query($ref_query); if($ref_row = mysql_fetch_array($ref_result)){ $bgt_time = $row['start_date']; $end_time = $row['end_date']; $status_id = $ref_row['status_id']; if($date_start != $bgt_time || $date_end != $end_time || $status_id == 2){ $update_query = "update reservation_instances set start_date = '".$date_start."', end_date = '".$date_end."' where reference_number = '".$value['bkgid']."'"; if ( ! @mysql_query ($update_query) ){ die ( mysql_error() ); } if($status_id == 2){ $extra_update_query = "update reservation_series set status_id = '1' where series_id = ".$ref_row['series_id']; if ( ! @mysql_query ($extra_update_query) ){ die ( mysql_error() ); } } } } else{ $query_resource = "select res.* from resources as res where res.room_id = ".$key_id; $resource_result = mysql_query($query_resource); if($res_row = mysql_fetch_array($resource_result)){ $username = strtolower($value['userid']); $uid = 2; $uid_query = "select * from users where username = '".$username."'"; $uid_result = mysql_query($uid_query); if($uid_row = mysql_fetch_array($uid_result)){ $uid = $uid_row['user_id']; } $create_series = "insert into reservation_series (date_created, title, description,type_id,status_id,owner_id,repeat_type,repeat_options) values ('".date('y-m-d h:i:s',strtotime("+5 hours"))."','reservation desk','reservation made by reservation desk', '1','1','" .$uid."','none','')"; mysql_query($create_series); $series_id = mysql_insert_id(); $res_rev_query = "insert into reservation_resources (series_id,resource_id,resource_level_id) values ('".$series_id."','".$res_row['resource_id']."','1')"; mysql_query($res_rev_query) or die(mysql_error()); $instance_query = "insert into reservation_instances (start_date,end_date,reference_number,series_id,booking_id) values ('".$date_start."','".$date_end."','".$value['bkgid']."','" . $series_id."','".$value['bkgid']."')"; mysql_query($instance_query) or die(mysql_error()); } } synchronizing the systems a key checked out through the ils is very similar to the process for reservation synchronization from the booked system to the ils. comparing ils data and room reservation system data is based on checkin and checkout time. the algorithm is as follows: for (begin of metadata data from ils) until (end of metadata from ils) { if (key checkout end time is past){ delete the reservation from database } else { if (the room is not checked out record does not exist during the specified begin and end time){ insert the new checkout record in the system } elseif(the room is already checked out but with different begin and end time) edit the existing checkout record in the system } } foreach ($checkouts as $key =&gt; $value){ $key_id = $value['keyid']; $user_id = $value['userid']; $checkout_id = $key_id.strtotime($value['timedue']).$user_id; $date_end = date('y-m-d h:i:s',(strtotime($value['timedue']) + $dst_adjust)); $query_resource = "select res.* from resources as res where res.room_id = ".$key_id; $resource_result = mysql_query($query_resource); if(strtotime("now") &lt; strtotime($value['timedue']) &amp;&amp; $row = mysql_fetch_array($resource_result)){ $resource_id = $row['resource_id']; $ex_query = "select inst.*, series.*, res.* from blackout_instances as inst , blackout_series as series, blackout_series_resources as res where series.blackout_series_id = inst.blackout_series_id and series.blackout_series_id = res.blackout_series_id and res.resource_id = ".$resource_id." and inst.end_date = '".$date_end."'"; $ex_result = mysql_query($ex_query); $ex_length = mysql_num_rows($ex_result); if($ex_length == 0){ $create_series = "insert into blackout_series( date_created, title, description, owner_id) value('".date('y-m-dh:i:s',strtotime("+".$conf['settings']['time_delay']." hours"))."','checked out','".$user_id."','2')"; mysql_query($create_series); $series_id = mysql_insert_id(); $create_series = "insert into blackout_series_resources(blackout_series_id,resource_id) values ('".$series_id."','".$resource_id."')"; mysql_query($create_series); $instance_query = "insert into blackout_instances(start_date,end_date,blackout_series_id) values ('".date('y-m-d h:i:s',strtotime("+".$conf['settings']['time_delay']." hours"))."','".$date_end."','".$series_id."')"; mysql_query($instance_query) or die(mysql_error()); } } } synchronizing reservations made in booked to the ils the reverse data synchronization happens when user creates a reservation in booked. the first validation happens in booked and is to verify if the reservation request is valid based on library policy. reservation duration cannot exceed 4 hours user can have one reservation per day user can request a reservation 3 days in advance if it passes validation, the reservation created in booked is sent to the ils for further validation. if the user status is valid (fines, overdues, etc.) if the reservation time is within the library open/close time if the room is available during the requested reservation time once the ils records the reservation, the api signals booked to create it. if (ils is ready) { if ( the requested reservation time duration exceed 4 hr) { throw error and stop process } else if ( the user requesting reservation has another reservation on same day) { throw error and stop process } else if ( the requested reservation begin time is beyond 3 days ahead) { throw error and stop process } if ( the reservation request is valid){ variable if_valid = return send_to_ils (reservation record) if (if_valid is true){ proceed the room reservation process. } } } $ils_initiate = true; if ($startperiod == null || !$startperiod-&gt;isreservable() || !$startperiod-&gt;begindate()-&gt;equals($startdate)) { $errors-&gt;appendline(resources::getinstance()-&gt;getstring('invalidstartslot')); $ils_initiate = false; } if ($begin_date &lt; $current) { $errors-&gt;appendline(resources::getinstance()-&gt;getstring('pasttime')); $ils_initiate = false; } if (strtotime($begin_date) &gt; strtotime($three_day_ahead)) { $errors-&gt;appendline(resources::getinstance()-&gt;getstring('threedays')); $ils_initiate = false; } if ($endperiod == null || !$endperiod-&gt;begindate()-&gt;equals($enddate)) { $errors-&gt;appendline(resources::getinstance()-&gt;getstring('invalidendslot')); $ils_initiate = false; } if ($user_has_reserv){ $errors-&gt;appendline(resources::getinstance()-&gt;getstring('userhasreservation')); $ils_initiate = false; } blackouts and checkouts the initial concept of blackout times in the original booked software has been modified so it also can represent when a key is checked out in the ils system. the booked systems queries the ils for both reservations and checkouts. when a user makes a reservation, the system marks the cell as reserved. the reservation can still be modified or deleted by the reservation owner or an administrator. once a user checks out the key, the system blocks the room for the duration of the reservation. at this point, the room is considered occupied and is blocked. from time to time rooms need to be made inaccessible for reservation for various reasons such as renovations, repairs or events. it is easier to block the rooms in booked than it is in the ils. figure 3. sample scheduling interface; blackouts and checkouts. automatically delete reservations by policy, reservations are forfeit if they are not picked up within 30 minutes of the start of their reservation time, and this is where having integrated systems is a plus. the booked system automatically deletes the reservation in both the ils and the room reservation system if thirty minutes have passed and the key has not been checked out. this function is enacted through user-driven events. when a user opens the booked schedule page, a script runs that deletes all reservations that are greater than 30 minutes if they have not been checked out. once this is complete, the schedule is shown. this is far more efficient than having staff members remove reservations which have exceeded their pickup time. conclusion in an ideal world there would be no need to develop such a complex system to place reservations on materials which have future use needs. the tools libraries use to manage these collections should have graphical interfaces that allow patrons to easily determine future availability across all or selected resources (rooms) and dates. unfortunately at this time no such tools exist. libraries’ services have shifted over the past 20 years, but the tools that we use to manage them have not kept up. we have seen a decline in the checkout of physical materials like books, but that has been supplanted by a surge in the usage of our space and equipment. many libraries seem to take a hands-off approach to how these materials are used, leaving rooms unlocked and offering a first come, first serve approach to equipment checkout, but that is probably because we don’t have tools that would help us manage these collections. there is a need for systems that have graphical interfaces which offer our customers detailed information about current and future availability of items/spaces and allow them to make reservations on those resources. it would be better and more efficient to have systems that were created specifically for this task rather than write complex code that has to be maintained and updated. that said, we have shown that it can be done. over the course of four years we have had over 100,000 reservations made and fulfilled through these two systems. it is better to manage a reservation system through an online calendar which is backed by an ils. bibliography [1] faust, b. d., hafner, a. w., & seaton, r. l. (2010). openroom: making room reservation easy for students and faculty. the code4lib journal, (10). retrieved from http://journal.code4lib.org/articles/2941 [2] doherty, m., & white, e. (2012). room reservations at vcu libraries: how we coped with rapid growth and overwhelming demand for student study space. college & research libraries news, 73(3), 142-146. retrieved from https://doi.org/10.5860/crln.73.3.8722 [3] sallans, a., soule, k., & gilbert, j. (2010). selecting and implementing a patron-driven resource reservation system. library hi tech news, 27(8), 5-6. https://doi.org/10.1108/07419051011104222 [4] williams, d. j., & hada, k. (2017). the best tool for the job: revisiting an open source library project. d-lib magazine, 23(7/8). https://doi.org/10.1045/july2017-williams [5][7] hohpe, g., & woolf, b. (2004). enterprise integration patterns: designing, building, and deploying messaging solutions. boston: addison-wesley. [6] humphrey, p. (2017, april 26). understanding when to use rabbitmq or apache kafka. retrieved from https://content.pivotal.io/blog/understanding-when-to-use-rabbitmq-or-apache-kafka [8] ed-douibi, h. (2016, january 2). modeling web apis: your best choices. retrieved from https://modeling-languages.com/modeling-web-api-comparing/ about the authors denis galvin (dgalvin@rice.edu) is director of library technology for rice university. he has worked with systems in academic libraries for 17 years. prior to working at rice he worked for carnegie mellon university. mang sun (ms20@rice.edu) works for fondren library at rice university where he is the systems librarian. his main job duties include administering several library systems including the ils system and developing in house applications and solutions when needed. prior to rice he worked for saskatchewan provincial library in canada as a systems librarian (information resources specialist). his research interests cover a wide spectrum of library and it technologies. he got his mlis degree from dalhousie university in 2006. hanjun lee (hanjun.lee@rice.edu) works for fondren library at rice university where he is the library web application developer. he has developed many library related web applications using various open source software and has integrated web based software to the ils and various commercial web platforms. prior to rice he worked for online dakota information system as a web application developer. his current interests are mobile based application development and integration of various web platforms into unified library web & mobile applications.   subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – metadata analytics, visualization, and optimization: experiments in statistical analysis of the digital public library of america (dpla) mission editorial committee process and structure code4lib issue 33, 2016-07-19 metadata analytics, visualization, and optimization: experiments in statistical analysis of the digital public library of america (dpla) this paper presents the concepts of metadata assessment and “quantification” and describes preliminary research results applying these concepts to metadata from the digital public library of america (dpla). the introductory sections provide a technical outline of data pre-processing, and propose visualization techniques that can help us understand metadata characteristics in a given context. example visualizations are shown and discussed, leading up to the use of “metadata fingerprints” — d3 star plots — to summarize metadata characteristics across multiple fields for arbitrary groupings of resources. fingerprints are shown comparing metadata characteristics for different dpla “hubs” and also for used versus not used resources based on google analytics “pageview” counts. the closing sections introduce the concept of metadata optimization and explore the use of machine learning techniques to optimize metadata in the context of large-scale metadata aggregators like dpla. various statistical models are used to predict whether a particular dpla item is used based only on its metadata. the article concludes with a discussion of the broad potential for machine learning and data science in libraries, academic institutions, and cultural heritage. by corey a. harper introduction metadata quality, metadata completeness, and metadata assessment. these are topics that have been discussed at length on and off for many years. recently, the digital library federation’s assessment community formed a metadata assessment working group, which has begun a bibliography of articles related to metadata quality and metadata assessment. [1] many of these articles contain very useful best practices and recommendations gleaned from years of experience aggregating metadata from diverse digital library projects. some of these articles date back almost 15 years. in 2002, jewel ward conducted a quantitative analysis of dublin core element usage in registered oai-pmh endpoints. [2] a year later, in 2003, naomi dushay and diane hillmann used early commercial data visualization tools to identify patterns in metadata submitted to the national science digital library and discussed the use of these visualizations to prioritize and manage remediation efforts. [3] in many ways, these studies were ahead of their time. data visualization tools were in their infancy at the turn of the century — there was no d3 or tableau, and tools that might help with the quantitative measurement of metadata impact were non-existent. although completed shortly after the term “data science” was coined in 2002 [4], these early quantitative analyses and data visualization efforts foreshadowed the application of data science techniques to library data. in the past few years, there have been some very preliminary conversations about the relationships between data science and libraries. in a recent cni presentation, eric mitchell, vicky steeves, and jenny muilenburg discussed the application of data science to the library field. [5] their talk touched on a variety of areas, including data science education and outreach to scholars and on the idea of bringing machine learning, probabilistic modeling, and related techniques to bear on library problems and library decision making. all three presenters are librarians, some embedded in data science projects and research institutes, and all are involved in the moore-sloan data science environment. some library and information science programs are now offering data science certificates, and the harvard library has recently started offering “data scientist training for librarians“. in a recent computers in libraries article, dan chudnov introduces librarians to the underlying notions and techniques utilized by practitioners of data science. [6] chudnov notes that the goal of data science programs is “to make better decisions, based on data, despite uncertainty.” this goal exactly summarizes the topic of the research conducted for this article. there are decades worth of recommendations and best practices around metadata quality and shareable metadata, but the tools to precisely measure the impact of “quality” metadata — to assess the outcomes afforded by our metadata, and make better decisions based on these data — are brand new. this decision-making component is essential, and it is telling that the preliminary planning for the project discussed here fulfilled the requirements of an assignment for a fall 2014 mba course titled “data driven decision making.” this paper presents the concepts of metadata assessment and “quantification”, provides a technical outline of data pre-processing, and proposes some visualization techniques that can help us understand metadata characteristics in a given context. additionally, the closing sections introduce the concept of metadata optimization and explore the use of machine learning techniques to optimize metadata in the context of large-scale metadata aggregators like dpla. finally, the article considers the broad potential for machine learning and data science in libraries, academic institutions, and cultural heritage. quantifying metadata — project proposal & data preparation the project proposal set out to provide some initial analysis of dpla metadata and its impact on collection usage. the proposal introduction read, the intention is to explore distribution of metadata field population across dpla partners and identify some initial data visualization techniques to understand these patterns. distributions of metadata usage across different collections and resource types, as well as correlations between fields will be analyzed. additionally, if relevant information can be generated from google analytics, analysis will be conducted to ascertain relationships between metadata terms and user queries, as well as to build regression models using metadata field use and content as predictors of item usage statistics. according to a recent survey by crowdflower, in data science, data preparation is 80% of the work. [7] this project was no exception. dpla publishes a bulk download of their data in json. effective visualization and quantitative analysis of these data first required counting field occurrences. python code was developed for this research to parse the “sourceresource” of the dpla data and count occurrences of each field in the dpla metadata application profile (map). the sourceresource section of the map contains 19 fields drawn primarily from dublin core, dublin core terms, and the europeana data model. the output of this script is a pipe delimited file with one row per record, where each column contains the count of occurrences of a specific field. figure 1. wide data, 1 record per row in addition to this “wide” data format, some analyses are more effective with data in a “long” format, that is effectively de-normalized. the long format for dpla data, shown in figure 2, puts each field on its own row and adds a column to indicate which field is shown. this pattern is so common that many data management tools (such as r, and the “pandas” python module) offer tools for reshaping and converting between these formats. utilities to convert data from wide to long are often called “melt” functions, though rather than melting the data here, the data creation script has been modified to support both the wide and long formats. the long format also has the advantage of adding a “binary” column, which contains a binary value of whether or not that field appears in that record. we will see later how this additional data point can be useful, though it is worth noting that this binary data can also be added after the fact. figure 2. long data, 1 field per row, 21 rows per record it is also worth noting that this data probably doesn’t qualify as “big data,” but could almost certainly be called “medium data.” the wide data format is only 1.5 gb, but as of december 2015 contained 11,557,180 rows. the “molten” format is 25 gb and contains 242,477,520 rows. the original, raw json format is a 59 gb file. metadata visualization once our data is in these formats, we can easily start doing visualizations of field counts, distributions, and variation across providers. unless noted otherwise, all of the following charts and graphics are produced using tableau 9.2. figure 3 shows the average number of subject fields per record, broken down by provider. figure 3. average subject count by provider another view of this data, shown in figure 4, highlights the binary values mentioned earlier. rather than average count of subjects per record, we can graph the percentage of records for each provider that have at least one subject. notice how the us government publishing office and the university of virginia jump from the middle of the pack to the top three. this tells us that these providers have on average fewer subjects per record than many other providers, but are much more consistent in having at least one subject. figure 4. percentage of records having subjects it is also possible to look at the distribution of outliers — records with exorbitantly large numbers of occurrences of particular fields. figure 5, below, looks at the distribution of subjects in our new york state hubs: the new york public library and the empire state digital network. figure 5. distribution of nypl and esdn subject counts looking at these outliers has already identified mapping problems that dpla has since fixed, such as a record with 1,476 subject headings. a record with 3,800 identifiers, one with 480 occurrences of the “spatial” metadata element, and one with 24 languages also appear in the dataset. initial versions of these visualizations, and the work to report these outliers, was conducted shortly prior to dpla 2015. at almost exactly the same time, mark phillips from university of north texas was blogging the same findings, though using a completely different toolkit. his work involved loading the complete json data into solr and using solr’s built-in analytics to determine means, standard deviations, outliers, and other summary statistics of metadata to field distributions, with an initial focus on subject. [8] it was fascinating to note the parallels between the phillips analysis and the findings reported here, with both processes even finding the same subject heading count outlier. however, this shouldn’t be a surprise given that phillips’ earlier post on “metadata quality, completeness, and minimally viable records” were a partial inspiration for this work. [9] bar graphs, and to an extent line charts are useful mechanisms for viewing single metadata fields, but are poorly suited for looking at variance across multiple fields and across multiple providers. this is too much information to try to put on a single chart, and using “small multiples” of bar graphs is a bit cluttered in this context. it also does not allow at-a-glance comparison of multiple characteristics across providers, or some other differentiating feature. this kind of comparison can be effective when viewed in a table, as shown in figure 6 below, but the significant differentiating features aren’t as eye-catching or readily apparent when shown as tabular data. figure 6. tabular view of cross-field statistics this paper introduces the concept of “metadata fingerprints” as a mechanism to provide an easy to read, quick visualization of multiple quantitative characteristics of arbitrary sets of metadata records. figure 7 provides an example, comparing average “field counts” for multiple fields for content hubs (ch), service hubs (sh), and the overall averages for dpla as a whole. content hubs were the original dpla contributors, and represent cases where the hub is a single organization and owns all of its content. service hubs have since become much more common, and represent mini-aggregators of their own, helping funnel the content of smaller libraries, archives, and museums into the dpla. figure 8 shows fingerprints for a selection of dpla providers. figure 7. metadata fingerprints: content hubs, service hubs, all dpla these are simple d3 star plots, rendered using nodejs and a “reusable chart generator” provided by kevin schaul of the washington post. [10] in these plots, each radius represents a metadata field, and the extent of the spike on that radius represents the average count of occurrences for that field for the given subset. it’s important to note that the relative shaded area of these fingerprints is not meaningful, but much information can be gleaned quickly from comparing the overall shape and the length of the spikes. in figure 7, you can clearly see multiple characteristics of the target datasets in one glance. for example, you can see that service hubs overall have almost twice as many subjects per record as content hubs. this data may be skewed slightly, since there is an uneven distribution of content hub vs service hub data, but for the sake of a rough comparison, the visualization should be sufficient because it is an average per record. figure 8. metadata fingerprints: selected hubs similarly, figure 8 makes it very easy to see that the david rumsey collection has far more “spatial” metadata than other collections, and more than the hub-level aggregations in figure 7. this makes sense, given that the rumsey collection represents resources digitized from a map library. figure 9, which compares the binary occurrences rather than the average numbers, presents even more stark contrasts. this is because the scale is flattened out to percentages rather than representing counts. in the ‘averages’ prints, the length of each axis (radius) represents a range of 0-5, whereas in the binary version, the range is 0-1. here, you can still definitely see the disproportionate representation of spatial data in the david rumsey collection. it is also clear here that getty and rumsey records are both far more likely to have “description” elements than many other data sets. again, this makes intuitive sense, since both of these collections are largely image based and image metadata often has more free-text description than other data formats. it is still useful to be able to ascertain this information quickly and visually. figure 9. binary fingerprints: selected hubs these fingerprints provide a simple and intuitive visualization to quickly compare characteristics of metadata across different levels of aggregation. if it were possible to view these data for sets representing “records that have been viewed in the dpla user interface” and “records that have not been viewed,” this might give us information about the metadata qualities that help to make resources discoverable. google analytics recall that the initial project proposal noted that, “if relevant information can be generated from google analytics, analysis will be conducted to ascertain relationships between metadata terms and user queries, as well as to build regression models using metadata field use and content as predictors of item usage statistics.” phase two of this research project involved collecting data from the dpla google analytics account and merging it with these statistical descriptions of dpla metadata. it is important to note that google analytics may not offer a comprehensive view of usage. users can opt out of google analytics, disable javascript, or otherwise browse anonymously, and these users are not counted. there are other factors that may result in undercounting, such as link design decisions discussed below and the fact that dpla does not currently count api usage. additionally, spam referrers and other bots could inflate these counts, though preliminary analysis shows that these are not a statistically significant component of the hits analyzed for this research. google analytics dashboards provide a tool to do some preliminary analysis and to browse statistical information about pageviews, searches, and other characteristics of web traffic. for example, figure 10 shows the top 10 most viewed pages in dpla, filtered to only show those with the “/item/” pattern that denotes an item record. this is useful, but it does not give us the capacity to visualize these data, sorting and manipulating them in interesting ways. more importantly, it is difficult to combine analytics data from the dashboards with data from other sources, such as our metadata characteristics. it is also challenging that for some queries the statistics presented are a summary and google analytics will perform sampling to reduce the volume of rows returned. figure 10. top 10 items by pageview to overcome some of these limitations, it is possible to harvest data from the google analytics api, avoid throttling, and store that data offline for various additional forms of analysis. figure 11 shows a bar graph representing the same top 10 most viewed items, and makes it far easier to see the difference in scale between the top viewed item and number 10. note that these counts are based on a query limited to the 2013 launch of dpla through feb 6, 2016. there are now additional page views, including a spike during the lead up to dpla 2016, but those additional data were not available at the time this analysis was completed. figure 11. top 10 items by pageview, bar graph harvesting from google analytics was initially completed through one-off harvesting scripts through the google analytics api. those scripts have now been generalized into a simple utility that allows bulk harvesting. this python google analytics utility, pyganalytics, is available on github along with straightforward installations instructions. the code takes a yaml configuration file containing a google analytics “view” (to which you must have access), a single “metric,” and a set of up to 7 “dimensions.” additional command line arguments include a start and end date and an output file. the script will retrospectively query the api, using 1 week intervals to avoid sampled data, and output a single pipe delimited csv file. it is also possible to switch harvest intervals to one day if throttling does occur. additionally, and perhaps more importantly, harvesting data from google analytics allows us to combine these data with other datasets, such as the metadata field counts that drive our “fingerprints.” with this additional data, it becomes possible to examine if, and to what extent, the metadata fingerprints differ for used and not used dpla items. figure 12 shows our average and binary field counts split for approximately 550,000 of the dpla items that have at least 1 hit registered in google analytics and a random sample of 550,000 of those with 0 hits. note that the shape of these two fingerprints is almost identical, but the records that show evidence of use are just slightly “bigger” than those that are not used. the subject spike for used is 3.65, while the spike for unused is 2.95. this isn’t conclusive, but seems to imply that slightly more metadata is going to slightly increase your likelihood of being seen in the dpla aggregation. it would be worth looking at how this changes if analyzing fingerprints of items with more than two, or more than five hits. additionally, it could be interesting to compare used versus not used fingerprints for specific formats or item types. figure 12. fingerprints by usage status there is one caveat in the definition of “use” employed by this analysis. this definition is almost certainly under representing use of dpla items. anecdotally, many users probably don’t hit the dpla item page, but instead go directly from the search results list to the source collection via the “view object” link and therefore never land on the dpla item page. figure 13. dpla search results unfortunately, it is not possible at this time to track outbound links from dpla to constituent data providers via the dpla google analytics. we can, however, look at referrals from dpla in those cases where a hub or other partner institution has its own google analytics setup. figure 14 shows the distribution of referring page paths for dpla referrals to one of the service hubs for 2015. this shows that there is an additional referral directly from the “search” page for every two referrals from the “item” page. this seems to support the hypothesis that our research methodology under-reports the usage of materials in the dpla. figure 14. dpla referring page paths nonetheless, the question of whether this usage is influenced by the shape and other characteristics of an item’s metadata in general bears further exploration, and could be used to inform policies and recommendations about metadata assessment. during dplafest 2016, gretchen gueguen made the very important point that “[metadata] quality is contextual.” [11] in the subsequent discussion, an audience member suggested that the concept of metadata assessment might be better described as metadata optimization. this conversation was the source of the title of this article, and these findings apply to optimization in the context of discoverability in the dpla. word counts another phase of this research project looked closely at word counts, natural language processing, topic models, bi-gram and tri-gram frequency analysis, and a variety of other aspects of the language of metadata in the dpla. findings in these areas are out of scope for this article, though hopefully future articles or blog posts will address some of these topics. however, raw word counts per field are in scope, and inform some of the final steps of this preliminary analysis. another long format data set was generated containing all the words per field per record in all of the dpla. this is a 25 gb file, containing some 150 million rows of textual data. this allows us to easily segment data by field and provider and analyze word counts. once again, mark phillips’ work in the weeks before dpla fest 2016 paralleled this research almost exactly. figure 15 shows the average total number of words in all description fields for a given record, broken down by provider. as was true with description binary field presence percentages, we can see that our primarily image-based providers have generally more verbose description fields than any other hubs. the same weekend that this figure was generated, phillips posted “dpla descriptive metadata lengths: by provider/hub,” which performs a very similar analysis, only by character length rather than word count. [12] figure 15. average word count for descriptions as before, with field occurrences, we can generate fingerprints based on word counts and compare the general shape of our metadata across different providers. figure 16, below, shows the log of average word count for selected fields for selected providers. logs are taken in order to flatten out the ranges and allow us to see more of the differentiation between hubs. without taking logs, we have fields like description, which could average as many as 150 words, and other fields like type, which might average one word. given these uneven distributions, fingerprints of raw word counts, shown in figure 17, don’t allow us to differentiate between providers at all. figure 16. log word count fingerprints figure 17. word count fingerprints machine learning and predictive analytics the stretch goal of the initial project proposal was to determine whether these metadata characteristics presented any capacity as predictors of usage as counted by dpla’s google analytics. a variety of regression analysis and machine learning techniques were tested, with varying degrees of success. the remainder of this paper focuses on the most promising set of techniques: variations on decision tree induction. for an excellent visual introduction to decision trees, and to machine learning in general, read the blog post by stephanie jyee and tony hschu. [13] additional techniques that showed less promise included linear regression, logit regression, support vector machines, and gaussian naive bayes classification. an analysis of the results of experiments with these techniques can be found in a jupyter notebook published concurrently with this article. for a more comprehensive look at all of these approaches, provost and fawcett’s data science for business provides both an excellent overview of many machine learning and data science techniques for both technical and non-technical audiences. [14]. please note: all of the following results are preliminary and should be validated by independent application of similar techniques before being taken as indicative of any predictive ability of these models. using scikit-learn and pandas, a number of classification algorithms were run to try to predict a binary analytics target variable, representing used or not used. the classifiers built their model from a feature vector composed of the following fields: featurelist = ['collection_x', 'contributor_x', 'creator_x', 'date_x', 'description_x', 'extent_x', 'format_x', 'identifier.1', 'ispartof_x', 'language_x', 'publisher_x', 'relation_x', 'rights_x', 'spatial_x', 'subject_x', 'temporal_x', 'title_x','type_x', 'collection_y', 'contributor_y', 'creator_y', 'date_y', 'description_y', 'extent_y', 'format_y', 'identifier_y', 'ispartof_y', 'language_y', 'publisher_y', 'relation_y', 'rights_y', 'spatial_y', 'subject_y', 'temporal_y', 'title_y', 'type_y', 'total'] all fields with an “x” suffix represent occurrence counts of that field, while “y” fields are word counts per field, and total is the total word count of the record. initially, models were trained on a sample dataset of 1.1 million rows, 550k used and 550k not used items. models were trained on 70% of those records and evaluated on 30% which were held out and not used when training the model. the use of holdout evaluation provides a more accurate picture of how well the model performance generalizes. the first non-linear model tested was a gaussian naive bayes classifier. a naive bayes (nb) classifier is one that naively assumes complete independence between features. the gaussian nb further assumes that the probability distributions of the features are normal (gaussian) distributions. that assumption means this is a poor choice here since features are unlikely to be distributed normally, but as a basic starting point it has the benefit of providing a baseline performance to which we can compare other classifiers. the gaussian nb gave us an overall prediction accuracy of nearly 90%, but an area under receiver operating characteristic (roc) curve of under 0.55. the area under roc curve, often abbreviated auc, is a slightly more nuanced evaluation metric than straight accuracy. fawcett and provost define an roc curve as “a two-dimensional plot of a classifier with a false positive rate [fpr] on the x axis against a true positive rate [tpr] on the y axis.” this essentially means that the curve is showing the tradeoffs between recall as measured by true positives, and incorrect false positives. a diagonal line on the plot would show an auc of 0.5, which is the equivalent of guessing. the top right corner represents classing every point as positive, which means 100% of your true positive instances are predicted as positive, but so are 100% of true negatives. at any given point on the curve, you will have a correspondence. for example, an 80% true positive rate may corresponds to a 40% false positive rate. figure 18 shows the roc curve for the gaussiannb model to help illustrate this concept. figure 18. gaussiannb area under roc curve based on these metrics it is clear that, while accurate overall, there are some serious problems with the gaussian nb. its precision, which is the ratio of true positive predictions to total predicted positives (tp / (tp+fp)) is skewed. in other words, despite its overall predictive accuracy, it falsely predicts a large number of unused resources as used. the next non-linear classifier used was a basic decision tree. simple decision trees are rarely used in production predictive analytics, but its output can give a sense of whether there is potential for predictive modeling via tree induction. it performed surprisingly well on the evaluation data set. tree induction is a technique that builds a decision tree to classify data based on the features it is given. the tree building is tuned by a number of criteria about the size and depth of the resulting tree. this preliminary tree was tuned to never have less than 20 data samples in a split and to stop before exceeding 100 leaf nodes (final endpoints on the tree). this tree performed admirably, with auc of approximately 0.67. figure 19 shows the auc for this initial decision tree. based on this metric, the decision tree induction used here was significantly better than a coin toss for trying to predict whether a dpla item had been used or not used. figure 19. decision tree area under roc curve so what decision points was the induction basing these predictions on? figure 20 shows the full tree, which is a bit daunting. however, figure 21 shows the first split, and it is informative. this cut splits off approximately 24,000 records containing fewer than 16.5 words total, and tells you that almost none of these records will ever be found. this makes intuitive sense, but it is useful to be able to actually demonstrate it. the only surprise here is that there are so many records in dpla with so little metadata. the second split is focused specifically on the description field, though the “gini impurity” values tell you that the result of this split is not much better than a coin toss. gini impurity tells you how often a prediction would be wrong if randomly assigned based on predicted distribution. [15] still, as you go further down the tree, you eventually end up with a classifying algorithm that predicts with approximately 67% accuracy whether or not a dpla item has ever been seen by a user through the dpla interface. these predictions are evaluated on data that has not been seen by the training process. figure 20. full decision tree chart figure 21. first splits for emphasis, it bears repeating that this decision tree is seldom used in serious predictive analytics, but has the significant advantage of enabling tree visualization. it is far more likely for tree-based predictions to rely on ensemble methods, which take into account multiple classifiers — in this case trees — and average their predictions through a voting procedure. ensembles tend to be more accurate, but they are also much harder to interpret and understand, since they make their predictions based on the average predictions of a large number of trees. experimentation with ensemble methods began with a basic bagging classifier that coordinated predictions across 30 randomly generated trees, tuned using a “min_samples_leaf” parameter set to 50. tests were run using a set of 7 bagging classifiers, with varying parameter settings, but min_samples_leaf=50 turned out to be the most effective. this model was initially trained and tested on balanced data, with a 50/50 split of used and not used items. it performed admirably, with an auc of over 72. further evaluation was done using a random sample of unbalanced data — approximately 5% used, resulting in an auc of almost 73 percent, but with an overall accuracy of only about 70%. additional evaluation metrics and ensembles of trees these additional details about model performance raised a few questions. why would this model have a strong auc, but a relatively low accuracy? what metrics should we be using to evaluate and tune this particular model? is this result unfavorably influenced by the decision to train on a balanced data sample? to answer these questions, it is necessary to dig into a wider array of evaluation metrics. figure 22 discusses a range of these model evaluation measures. as noted previously, accuracy (acc) gives us a summary of our overall accuracy, but does not provide any information about precision or recall. looking at auc adds information about the relationship between the true positive rate, which is another name for recall, and the false positive rate, which is sometimes called “fall out”. auc doesn’t give us information about precision (the percentage of your predicted positives that are actually positive). there are a few options for getting a sense of both precision and recall in one metric. one approach is to look the precision and recall scores separately. another is to review the f1 score, or f-measure, which is the harmonic mean of both precision and recall. finally, it can be helpful to look directly at the confusion matrix, and interpret the preponderances of true positives, false positives, true negatives, and false negatives. figure 22. accuracy measures there are overall precision, recall, and f-measure scores as well as scores for each individual class. a useful way of trying to understand all parts of a model’s accuracy is to view the confusion matrix, as well as the complete set of overall and class specific precision, recall, and f-measure values. below, you will find those reports for our bagging classifier. confusion matrix: [[334012 141841] | tn | fp | [ 5883 18264]] | fn | tp | f1 score: 0.198250222521 classification report: precision recall f1-score support 0 0.98 0.70 0.82 475853 1 0.11 0.76 0.20 24147 avg / total 0.94 0.70 0.79 500000 this confirms our suspicion that this model sacrifices a lot of precision in order to get good recall numbers. it’s able to identify over 75% percent of our used items, as shown by the 18k true positives vs less than 6k false negatives in our confusion matrix. however, doing so mislabels a huge number of points — over 141,000 — as positive that are not. the f-measure of 0.2 for our negative class also tells us that the combination of recall and precision here has much room for improvement. there are many ways to try to adjust, or tune, our modeling to try to account for our strongly unbalanced data and to improve the precision scores of our results. options include starting with the representative, unbalanced data; using random forests or extra trees rather than bagging classifiers; specifying class-weight when training the model; using grid search to tune parameters; or experimenting with changes to prediction probabilities. the choice depends on the specific question asked and what outcome we are optimizing to achieve. after a lot of experimenting with these options and tunings, additional models began to offer different and potentially more useful results. notably, working with deliberately balanced data resulted in models that, while they generalize well, tend to have little precision and overestimate positive cases to a large degree. so instead of a artificially balanced data set, a random sample of approximately 1 million records was produced, and split into 3 groups for training, tuning, and testing. using a random forest classifier provided an alternative ensemble method to bagging, which allows control over more useful parameters. most notably, this approach allows you to set a number of features for each tree in the ensemble to use. convention in the machine learning community is to limit this number of features to the square root of the number of features available. in this example, we have 37 features so we will work with a max_features per tree of 6. we continued with 30 estimators, though this can also be tuned. there are also class-weight options that we can use to tell the model to give priority to predicting one class over another. this helps us mitigate for the relative scarcity of our used items. the default behavior is “unweighted”, which could contribute to poor recall when training a model on data that matches the distributions in the data set as a whole. a basic change to this is to use the “balanced” parameter, which “automatically adjusts weights inversely proportional to the class frequencies in the input data as n_samples / (n_classes * np.bincount(y))“. [16] in this data set, since in the holdout data used to evaluate the models we have 475,853 false and 24,147 true, we end up with weights of 0.52 for our false class and 10.35 for our true class. if we wanted to further experiment, we could do hyperparameter tuning to fluctuate these weights, either randomly or using a systematic approach like grid search, and see if we can further tune our model. finally, since our default prediction functions are based on whichever class is more probable, it uses a probability estimation threshold of 50%. this means that we predict used if we are more than 50% confident in this prediction. we can further weight our model toward a specific class by looking at different thresholds for prediction and estimating various evaluation metrics for each probability. for example, perhaps we want to optimize our prediction threshold for the best possible f-measure. we can iterate through possible thresholds to determine where we hit our maximum, and we can graph the results. figure 23 shows a graph of the f-measures for every possible prediction threshold for our negative class. this graph shows that our model maximizes the f-measure at a threshold of 32%. figure 23. prediction threshold f-measure optimizations the threshold indicates that the “best” predictions, where best is measured by f-measure, come if the model predicts 0 whenever it is 32% or more sure that the value is false. this amounts to a deliberate over-predicting not used, which may seem counter-intuitive, but in the context it makes sense. we know there to be far more false than true values, and we’re telling the predictive function to explicitly compensate for that. it is important to note that this kind of threshold setting should be done on data that is not the data used to train the model, but is also not the model used to evaluate. otherwise, information will “leak” from phase to phase of the process, and the results will be less generalizable. this is part of why the models here were trained on “training” data, tuned on “tuning” data, and evaluated on “test” data. a 4th random sample of data was used for further evaluation on a larger data set. you can learn more about “leakage” on the kaggle wiki, which has an excellent discussion of the topic. [17] as shown in the metrics below, our model is now accurate almost 91% of the time. this is a significant improvement over the previous 70% accuracy. the auc is slightly lower than the prior model (0.61 as opposed to 0.72), but the f-measure is up from 0.19 to 0.24. most notably, we’ve clearly traded a significant amount of recall for precision in our ability to class positives. there are almost 80% fewer false positives, though our false negative rate has tripled. accuracy: 0.909846 auc: 0.619666986052 confusion matrix: [[447716 28137] [ 16940 7207]] f1 score: 0.242288749559 classification report: precision recall f1-score support 0 0.96 0.94 0.95 475853 1 0.20 0.30 0.24 24147 avg / total 0.93 0.91 0.92 500000 the statistician george box famously wrote that, “all models are wrong, but some are useful.” [18] in this case, we have a few different models that are “wrong” in distinctive and interesting ways. this raises the question of whether we want to err on the side of precision, recall or somewhere in between? and do we want to focus on more accurate prediction of not used items or used items. our tendency here is to lean towards optimizing based on f-measure and precision at the expense of recall in the used (true) cases. this optimization helps us better understand the characteristics of dpla metadata for items that have been used. for example, it gives more weight to the idea that a used item tends to be one that has at least 16.5 words total. further discussions of the use case for this information with stakeholders at dpla or the dpla hubs could lead to a refinement of these optimization decisions. this raises one more caveat in this research. though the ensemble methods, and especially random forests, performed more accurately in these experiments, they are also much harder to interpret. this is another tradeoff that must be considered when making decisions about what model to use. the simple decision tree had the advantage that the tree could be visualized. digging into its depths was tedious, but if the goal is to help advise collection curators and metadata experts on best practices for metadata production, interpretability is important. imagine now trying to interpret the collection of 30, or 50 such trees that inform an ensemble. however, the random forest classifier does offer us a simple list of what features are “most important” across all the constituent trees. in this case, this provides a bit of information and more importantly, it seems to confirm the general split points seen in the initial decision tree classifier. total words (0.113245) desc words (0.086540) rights words (0.063029) title words (0.057273) subject words (0.056207) collection words (0.053083) subject count (0.052737) the decimal values across all 37 features add to one and represent relative importance in the forest. the first 7, listed here, add up to almost 50%. the next 20 have values between 0.01 and 0.05 and the remaining 10 show importance of less than 1%. it is notable that “subject count” is the only non-word-based feature with an importance greater than 0.05. conclusions and next steps the quantitative metadata visualization described in this paper, as well as in work by mark phillips and péter király [19], represent a significant contribution to the areas of metadata assessment and optimization. the metadata fingerprints are especially novel, in that they allow digital collection managers, aggregators such as dpla, and metadata specialists to compare groups of metadata across multiple facets at a quick glance. fingerprints make it relatively easy to investigate metadata variance across groupings, and also to identify common metadata patterns. the investigations here offer comparisons across dpla providers and between items that have been used and not used in the dpla user interface as identified by dpla’s google analytics. other segmentations of this data are possible, depending on the question being asked. for example, it is possible to group metadata by format or genre, perhaps to look for differences in the general shape of images, text resources, and audio visual materials. it would also be possible to compare the used and not used fingerprints for a specific format, such as images, to see if some of the variance disappears. if a similar analysis were done on europeana data, building on király’s work, a comparison could be conducted on the subset of europeana data model (edm) properties common to both dpla and europeana. the predictive modeling described here is very preliminary, and there are a number of next steps that could be taken. to begin with, further conversations with dpla and the dpla hubs can help refine the use case for this kind of analysis. while the initial research question was rooted in metadata assessment, recent discussions with dpla staff have suggested that the word count findings could indicate that google’s harvesting or search algorithm prefers records with more words. dpla hypothesized that there are some search engine optimizations that could be applied to increase referrals from google to dpla. additionally, these findings could inform decision making about tuning the indexes on dpla’s internal search engine. addressing these new questions could lead to the development of alternate models, with slightly different parameters. discussions of google crawling also led to questions about the potential impact of term uniqueness. there is a possibility that having many records with very similar, template-based or collection-based metadata records may be penalized by search engines. another possible future development would be to capture some measures of term or phrase uniqueness for each record, either through hapaxes (terms or phrases that appear only once in a corpus) or the average of a measure like tf-idf, which weights document term frequencies by the relative frequency of the term in a corpus of documents. this is a good example of “feature engineering,” which is another useful area of continued exploration. the 37 features used in this model include representations of metadata field counts and word counts, but there are a number of other metadata characteristics that could produce a correlation with increased usage. categorical or factor variables could also be included, such as “collection” or “format.” thumbnails could also provide information to help predict item usage. features could be extracted that represent “whether there is a thumbnail,” “whether the thumbnail is unique or generic,” or even “what colors or hues are in the thumbnail.” the last of these could be derived from chad nelson’s work on dpla color browsing. [20] these are just a few examples of features that could be engineered from this data set and exposed to predictive models. predictive modeling of digital library usage is only one potential application of these techniques to library data and metadata practice. other possibilities include modeling of cost per use of print, subscription electronic, or local digital collections; comparison of electronic book / journal downloads to print circulation to quantify the effect of electronic materials purchasing on print usage; or use of clustering algorithms and collaborative filtering to build recommendations and suggestions for specific users, topics, or disciplines. additionally, collection development or digitization prioritization and decision-making could be informed by probabilistic groupings of latent topic models in metadata, full-text documents, or even user queries. these models, along with other natural language processing techniques, particularly as applied to full text and to query logs, could also help identify emerging areas of research, inform the development of taxonomies and controlled vocabularies, and even contribute to the automated generation of provisional metadata records. these probabilistic, machine learning, and data science techniques are increasingly used in other domains, and are starting to gain traction with library systems vendors. academic libraries are in a unique position to partner with data science institutes and centers to advance research of this kind, and the profession should be further exploring these areas. libraries, archives, and museums should also look more closely at data about the use of our collections, systems and services. this data can be used to inform decisions about staffing, resourcing, strategic directions, and prioritization. this form of data driven decision making has long been employed in other domains, but is only beginning to catch on in our community. there are few examples of the application of predictive modeling to library decision making in the literature. the most notable example is the work of daniel coughlin at penn state university to use linear regression and anova to model the relationship between cost and usage of electronic journal subscriptions. [21] [22]. additionally, libraries should be using algorithms and probabilistic modeling to deliver appropriate and timely resources and services to the researchers we serve. vendors like elsevier are already engaged in this space, and we run the risk of being left out of the conversation if we don’t develop core competencies in these areas. elsevier, for example, has a research intelligence division, and it has been suggested that elsevier’s purchase of the social science research network (ssrn) was due to ssrn’s value as a repository of data about the value of social science research, whose work influences whom, and what trends exist in particular fields. [23] university libraries have both the data and the credibility to do this in an open access context, but to do so we must collectively develop the skills and muster the interest in this space. libraries need to start applying these data science techniques. our profession should be experimenting with using natural language processing and machine learning to generate provisional controlled vocabularies and apply them to corpuses of text. this is demonstrated on a specific, targeted domain in a forthcoming article by raphael hubain. [24] as these modeling techniques evolve, we should also be able to cluster resources in meaningful ways, build recommender systems, and use data visualization techniques to create new kinds of end-user interfaces. data science techniques are rapidly maturing in the private sector, where the collection and analysis of data allows for more informed decision-making. through the adoption of similar techniques, libraries can improve our approach to the management and assessment of metadata practice. the techniques discussed in this article are very preliminary examples of how data science can help transform the role of the library and ensure that our organizations make better decisions to support the communities we serve. acknowledgements i would like to graciously thank dan cohen, gretchen gueguen, tom johnson, and mark matienzo of dpla for making bulk json data publicly available, providing me with access to google analytics information for this research, and allowing me to use dplafest as a forum for feedback on this work. additionally, i would like to thank andreas mueller for offering advice on scikit-learn evaluation metrics and model selection techniques. finally, i am deeply grateful to dan chudnov, gloria gonzalez, tom johnson, and reed shadgett for providing substantive and helpful comments on various drafts of this paper. references [1] digital library federation (dlf) assessment interest group (aig) metadata working group [internet]. updated 2016 jul 06. [cited 2016 jul 07]. available from: http://dlfmetadataassessment.github.io/ [2] ward jh. 2002. a quantitative analysis of dublin core metadata element set (dcmes) usage in data providers registered with the open archives initiative (oai) [thesis]. [chapel hill, nc]: university of north carolina. [cited 2016 jul 07]. available from: http://ils.unc.edu/mspapers/2816.pdf [3] dushay n, hillmann di 2003 sep 28. analyzing metadata for effective use and re-use. international conference on dublin core and metadata applications. [cited 2016 jul 07]. available from: http://dcpapers.dublincore.org/pubs/article/view/744 [4] press g. 2013 may 28. a very short history of data science [internet]. forbes. [cited 2016 jul 07]. available from: http://www.forbes.com/sites/gilpress/2013/05/28/a-very-short-history-of-data-science/ [5] mitchell e, steeves v, muilenburg j. 2015 nov 19. organizational implications of data science environments in education, research, and research management in libraries [internet]. cni fall 2015 membership briefing. [cited 2016 jul 07]. available from: https://www.cni.org/topics/digital-curation/organizational-implications-of-data-science-environments-in-education-research-and-research-management-in-libraries [6] chudnov d. 2016 jun. the intentional data scientist: three ways to get started [internet]. computers in libraries. [cited 2016 jul 07]. available from: http://www.infotoday.com/cilmag/jun16/chudnov–the-intentional-data-scientist–three-ways-to-get-started.shtml [7] press g. 2016 mar 23. cleaning big data: most time-consuming, least enjoyable data science task, survey says [internet]. forbes. [cited 2016 jul 08]. available from: http://www.forbes.com/sites/gilpress/2016/03/23/data-preparation-most-time-consuming-least-enjoyable-data-science-task-survey-says [8] phillips, me. 2015 feb 24. dpla metadata analysis: part 1 – basic stats on subjects [internet]. [cited 2016 jul 08]. available from: http://vphill.com/journal/post/5553/ [9] phillips, me. 2015 jan 05. metadata quality, completeness, and minimally viable records [internet]. [cited 2016 jul 08]. available from: http://vphill.com/journal/post/4075/ [10] schaul k. 2014 jan. d3-star-plot [internet]. [cited 2016 jul 08]. available from: https://github.com/kevinschaul/d3-star-plot [11] gueguen g. 2016 data quality at the scale of aggregation [internet]. [cited 2016 jul 04]. available from: https://drive.google.com/folderview?id=0bzpqesfqkizkvwrhynr3uevjqm8 [12] phillips, me. 2016 apr 12. dpla descriptive metadata lengths: by provider/hub [internet]. [cited 2016 jul 08]. available from: http://vphill.com/journal/post/5908/ [13] jyee s, hschu t. 2015. a visual introduction to machine learning [internet]. [cited 2016 jul 04]. available from: http://www.r2d3.us/visual-intro-to-machine-learning-part-1/ [14] provost f, fawcett t. 2013. data science for business. cambridge (ma): o’reilly media. [15] wikipedia contributors. decision tree learning [internet]. wikipedia, the free encyclopedia; 14 may 2016, 07:04. utc [cited 2016 july 5]. available from: https://en.wikipedia.org/wiki/decision_tree_learning#gini_impurity. [16] scikit-learn developers. 2014. sklearn.svm.svc [internet]. [cited 2016 july 5]. available from: http://scikit-learn.org/stable/modules/generated/sklearn.svm.svc.html [17] kaggle contributors. 2013 dec 04. leakage [internet]. [cited 2016 july 8]. available from: https://www.kaggle.com/wiki/leakage [18] box gep. 1979. robustness in the strategy of scientific model building. new york: academic press. launer rl, wilkinson, gn. robustness in statistics; p. 201–236. [19] király p. 2016 apr 09. report number three (cassandra, spark and solr) [internet]. [cited 2016 july 8]. available from: http://pkiraly.github.io/2016/04/09/third-report/ [20] nelson c. mar 2015. color browse [internet]. [cited 2016 july 8]. available from: https://dp.la/apps/33 [21] coughlin dm. 2016 mar 03. a web analytics approach for appraising electronic resources in academic libraries. journal of the association for information science and technology. 67(3):518-534. doi: 10.1002/asi.23407 [22] coughlin dm. 2015 sep 07. modeling journal bibliometrics to predict downloads and inform purchase decisions at university research libraries. journal of the association for information science and technology [internet]. [cited 2016 jul 08]. available from: http://dx.doi.org/10.1002/asi.23549 [23] kelty c. 2016. it’s the data, stupid: what elsevier’s purchase of ssrn also means [internet]. [cited 2016 jul 04]. available from: http://savageminds.org/2016/05/18/its-the-data-stupid-what-elseviers-purchase-of-ssrn-also-means/ [24] hubain r. 2016. automated skos vocabulary design for the pharmaceutical industry [internet]. [cited 2016 jul 04]. available from: http://hubain.be/lemuridae/index.php/automated-skos-vocabulary-design/ about the author for almost 15 years, corey has built digital libraries, administered library systems, and managed, migrated, and analyzed library metadata. first at the university of oregon, and then at new york university, corey led data analysis for an ils migration and administered systems including aleph, contentdm, dspace, hydra, metalib, millennium, and primo. corey is also deeply involved in the community around linked open data in libraries, co-founding the ala lita/alcts linked library data interest group and leading the igelu / eluna linked data special interest working group for users of ex libris products. corey recently completed his mba at nyu’s stern school of business, with a focus on business analytics and is now transitioning to a career focused on business intelligence and data science in academic research library and cultural heritage environments. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – best practices for a university laptop lending program mission editorial committee process and structure code4lib issue 15, 2011-10-31 best practices for a university laptop lending program the university of arizona libraries currently circulates over three hundred pieces of equipment including laptops, netbooks, projectors and ipads. this article describes the best practices and workflows we have developed since 2003 to create a laptop/equipment lending program that is efficient and mindful of financial resources and that our student body loves and continues to support. by pamela c. buzzard and travis s. teetor introduction the university of arizona libraries has 336 total laptops, netbooks and projectors in circulation, including 260 netbooks. the program started in 2003 with only 16 full sized laptops. after that, the program took off and we continued to purchase new equipment to keep up with the demand from the students. laptops and equipment are now purchased with student fees, so only currently enrolled students can check them out. the students love the library services and laptop lending so they continue to support the student fee. circulation netbooks circulate for 72 hours and students have full administrative privileges on them. full sized laptops (pc and mac) circulate for 4 hours, cannot stay out overnight, and have administrative privileges restricted by deep freeze. initially, all laptops had a 4 hour loan period because we didn’t have enough available for everyone to check out and we were concerned that allowing them to circulate for longer would increase the likelihood that they would be damaged.  as this lending program became more popular, our students provided feedback that they would at least like to be allowed to keep laptops over the weekend.  when we started circulating netbooks, which were considerably less expensive than full sized laptops, we decided to pilot a 72 hour loan period, which turned out to be extremely popular.  since we are never able to install all of the software that every student requests and none of our other library computers allow students to install software, we also decided to give students full administrative privileges on netbooks. all laptops circulate with a power adapter and a storage bag, but we also have usb cables, apple dvi to vga adapters, and computer mice available for checkout separately. when a student wants to check out a piece of equipment, we ask them to sign a laptop loan agreement form. students only have to sign this form once a semester and we record their agreement in a field in their patron record. to see up to date policies and equipment information, and to view our laptop agreement form, visit our laptop policies page. all of our laptops are stored in cabinets purchased from american locker.  this allows them to be stored in a secure location and allows full sized laptops to charge while stored.  our netbooks are stored four to a slot to maximize space; however, because each slot only allows one computer to be plugged in at a time, this does mean that we need to recharge the battery as we are reimaging them.  in order to quickly see how many laptops are available at any given moment, we have created cards that we place on the outside of each storage space to indicate which are ‘in’ or ‘out.’  the cards are laminated card-stock inserted into leftover book pockets and affixed to the cabinets with double-sided tape. figure 1: in/out cards on outside of the netbook storage locker. netbooks are stored four to a slot. [view full-size image] in order for students to have a sense as to the availability of equipment, we have created records in our catalog for each item.  we have also created separate circulation accounts for the repair and reimaging processes, and we check out laptops to these accounts while they are being repaired or reimaged.  while this allows us to accurately reflect the number available for checkout, most students and staff tend to just call the various service sites to see how many laptops are available rather than check our website. damaged equipment the hardest part of evaluating damaged equipment is making the determination between normal wear and tear and user damage. we inspect all of the equipment before it gets checked out to the student so that when it is returned, we can be fairly certain that any damage was caused by the most recent user. we even turn the netbooks on to see if the screen is cracked since the cracks on those screens cannot be seen when the computers are off. we also check all of the ports, the bags, and the cords for damage. on full sized laptops, we turn them on and check the memory to make sure it is still intact. if we find any of the types of damage listed below, we leave the item checked out to the student and wait to hear back from our it staff on whether or not to charge him or her. if the damage cannot be fixed by our it staff and is not covered by manufacturer’s warranty, we charge the student. our policy is that if desk staff are unsure whether the student will be responsible for the damage, they leave the equipment checked out to the student and wait for the it staff to make the final determination. types of damage that students may be charged for: damaged screen damaged ports (headphone, usb etc.) missing bag or adapter hardware damage the most common types of damage are cosmetic problems such as dings and cracks to the external case and lost hinge covers. cosmetic damage that does not impede the use of the equipment gets noted in the item record in the integrated library system (ils) and the item continues to circulate. other common problems are software problems that can be fixed with our reimaging process or hardware problems that can be fixed by our it staff or the manufacturer’s warranty (such as a key that has fallen off). often, in those cases, we do not charge the student for the damage. if we have determined that the student is not responsible for the damage, we check out equipment to the repair circulation account in the ils while it is being repaired in order to keep our inventory accurate. this prevents the laptops from being shown as “in library” in our catalog if they are actually being repaired. in january 2011, we decided to have one person from our desk staff serve as the laptop manager and report damaged equipment to one person in the it department. this allows for easier communication between it staff and desk staff, faster turnaround of damaged equipment due to it staff scheduling, and less equipment ”loss” due to miscommunication of who has a piece of equipment.  with this change, we also created a spreadsheet for inventory purposes. use of this spreadsheet creates a paper trail when equipment is moved and allows us to notice ongoing problems more quickly. our inventory spreadsheet contains data such as: location, barcode, serial number, whether or not the item is in circulation, and reason not in circulation.  some reasons why things might not be in circulation are: repair, warranty, or withdrawn. for all items sent to our it department, we also include: name of person picking it up, date picked up, date returned, and problem/fix—if any. including this information in an inventory spreadsheet outside the ils allows the laptop manager to see how long items have been in the it department. since most items are still checked out to the student when they are in the it office, this is the only way to track this information. additionally, the laptop manager keeps notes in the spreadsheet of the problem and fix that would otherwise clutter up the item record in the ils. this way we are able to see if there are continuing problems with a piece of equipment. we limit the notes in the item record in the ils to information on the physical condition of the equipment so our student employees and desk staff are able to tell if physical/cosmetic damage is new. reimaging unlike the full sized laptops, students have full administrative privileges while they have the netbooks, which allows them to install additional software and make system changes. each time a netbook is returned, it is reimaged to restore all default settings. each semester, our it staff prepares a sample netbook for each specific model in circulation by installing/configuring pre-selected software. once that netbook is configured, a copy of the operating system is uploaded as an image to a dedicated pxe server. this server stores a separate image for each of the different brands or models of netbooks we use. to reimage, we use networking cables to connect each netbook to the server to retrieve the appropriate image from a menu list. once the image process has started, it can take up to 30 minutes to complete. each of our service sites can reimage up to 10 netbooks at one time. the reimaging process re-formats the hard drive and erases all previously loaded content and reinstalls the operating system and programs, restoring settings back to our defaults. it also downloads any program updates at that time. the reimaging process was handled by it staff only when we began circulating netbooks, but as demand increased, we realized we were being overly cautious. the process was simple enough to be handled by our student employees—even those who were not particularly tech-savvy. now all of our desk staff are trained on the check in, check out and reimaging processes, but our student employees handle most of the work including reimaging netbooks once an hour as needed. ipads we are currently piloting the circulation of ten ipad 2s. we looked at stanford university’s list of ipad apps in order to help determine which apps we would purchase and load onto the ipads. in addition to the ipads, we also purchased five bluetooth keyboards, spare usb chargers, smart covers and protective cases. nytimes pulp (news) icircuit the elements astronomy picture of the day wsj (wall street journal) mit technology review wolfram alpha modality star walk pop mechanics convert units graphing calculator solar system g – gravitation simulator pop sci pi cubed numbers orbit architect s&t skyweek nasa ted kindle inkling (digital textbooks) pages iannotate good reader lightning talk evernote dropbox keynote ua iphone app flashcards deluxe scanner pro splashtop table 1: list of apps included on circulating ipads the ipads will circulate with a smart cover, an xgear snapon case as a protective cover, usb and wall charger, and a timbuk2 carrying case for 72 hours. during that time, students will be able to sync their own itunes accounts and upload any of their own apps. when the ipads are returned, we will restore the ipad to factory settings and then sync our itunes library so all of our apps are installed again and the student’s information is no longer stored on the device. ipads can be restored to factory settings by going into the settings feature on the actual device and selecting “erase all content and settings” under “restore.” it only takes a few moments and then the ipad’s hard drive is empty and ready to have apps installed. the next step is to connect the ipad to our syncing computer to install our purchased apps. this process takes around 15 minutes due to the size of some of the apps. our syncing computer is one of our netbooks. we have installed itunes and our purchased apps on the netbook so that our student employees can install the apps on the ipad while still being available at the service desk. with the release of ios5, we should be able to install the apps wirelessly instead of having to connect to the netbook. as part of our pilot, we will include a link on the ipad to an instructional page we have created on using the ipad and transferring purchased or created content for students who may be unfamiliar with using them. additionally, we will include a link to a survey created in google docs to get feedback on the apps and use of the ipad to improve the process in the future. conclusion we hope that you can use the information that we’ve cultivated over the last several years to start a new laptop lending program or make changes to your existing processes.  we have evolved these best practices to work efficiently for the university of arizona libraries staff to provide a service that our students are excited about. if you are interested in any details about our program which are not covered in this article, please feel free to contact us. about the authors pamela is a library information associate at the university of arizona libraries. she currently oversees the laptop and equipment circulation program. email: buzzardp@u.library.arizona.edu travis is a library operations supervisor at the university of arizona libraries.  he was directly involved with establishing the ua’s initial laptop lending policies/practices and has worked to make improvements to the library’s process over the years. email: teetort@u.library.arizona.edu subscribe to comments: for this article | for all articles 5 responses to "best practices for a university laptop lending program" please leave a response below: thomas la foe, 2012-02-20 thank you for this article. it is a great example of a system that works and we would like to steal some of your ideas! we are starting to look at checking out the ipads and have run into an issue with the icloud backups. you mentioned that you might be able to do this wirelessly and i would like to know if that is, in fact, working for you. it makes me nervous because it only keeps the last three backups as far as i can tell. we are still restoring the ipad with a pc version of itunes in order to make sure all of our apps are there. could you let me know if that process has changed at all for you with ios5? thanks so much, thomas la foe tlafoe@library.msstate.edu laptops in libraries « ualr library tech blog, 2012-07-18 […] a_wireless_latop_lending program_univ_of_akron_experience an_assessment_of_student_satisfaction_with_a circulating_laptop_service arl spec kit exec summary arl spec kit survey questions and responses best practices for a university laptop lending program […] laptop lending articles | ualr library tech blog, 2013-03-04 […] best practices for a university laptop lending program […] marielle knispel, 2013-07-18 i found the same file but without the trojan at keygenpen. rajendra mohan dev sarma, 2015-03-12 this paper is very useful for me for my phd work as reference please send the same to my email with best regards rm devsarma leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – rasmuson library dvd browser: fun with screen scraping and drupal mission editorial committee process and structure code4lib issue 5, 2008-12-15 rasmuson library dvd browser: fun with screen scraping and drupal the dvd browser is a simple application that lets library patrons browse movie covers, titles, and reviews. it works by screen scraping the the rasmuson library catalog for dvd movies and dumps the data into a drupal mysql database. this paper describes the process of setting up the dvd browser. by ilana kingsley and mark morlino introduction the rasmuson library dvd movie browser was developed because our library patrons were unhappy with the search function of our sirsi/dynix library catalog. patrons told us they wanted to be able to browse movie covers and titles, and the library catalog did not meet their needs. the dvd browser is a simple application that screen scrapes the rasmuson library catalog for dvd movies and dumps the data into a drupal mysql database. [1] drupal is a content management system (cms) which runs on mysql/php. the dvd browser application could have been built without drupal, but since our library web site runs on drupal and because drupal has many useful features, such as rss for new items, tagging, and user comments, we decided to store the data within the drupal cms. our first version of the dvd browser ran on drupal 4.7 and used php to screen scrape the catalog. we found that the easiest way to port the dvd browser to drupal 6.5 was to rewrite the entire script. the jump from 4.7 to 6.5 was big, and there were some significant changes within drupal, specifically the change for custom templates to use the cck module instead of the flexinode module.  we found that the flexinode to cck converter did not work well, and decided to re-create the entire system. the script was rewritten in perl. this paper describes the process of setting up the dvd browser.  the screen scraping script provided may be repurposed for your needs. if your library uses a different library catalog vendor, the concepts presented below should work with minor modifications to the code. what the application does a perl script screen scrapes the library catalog for new dvds. the script temporarily stores the data in a text file and gathers additional information about the movie, such as the movie cover and genre, from other web sites. the data is dumped into a drupal mysql database. the end user is able to browse and search for dvds. requirements perl mysql drupal 6.x drupal modules: cck, filefield, imagefield, imageapi, link php 5.2 (required for imageapi) gd (configured to support jpg) or imagemagick drupal configuration the rasmuson library dvd browser uses drupal 6.5 for its backend. besides the core drupal modules, which are automatically added when installing drupal, we installed the following modules. cck (content construction kit) module and its child modules: filefield imagefield link imageapi views the dvd browser requires configuration of five areas within drupal. they are content types, taxonomy, views, blocks, and theme templates. each section is described below. content type creation drupal content types are used to represent a specific type of content and provide a content-input template so that content providers can add, edit, and delete data without knowing how to program or code. drupal automatically installs two basic content types, page and story. in order to modify these existing content types or create your own, the cck module must be installed. we created a content type named “movie,” which contains the following settings and fields: initial settings identification name: movie the human-readable name for this content type. the name is required and must be unique. type: movie the machine-readable name for this content type. the name is required and must be unique. description: a dvd for the dvd browser. the description is not required. it is used to help content providers select an appropriate template. submission form settings title field label: title the default name for this required field is title. body field label: the default name for this field is body; however, when creating a custom content type, it’s often best to omit this field by leaving it blank. workflow settings default options: * published promoted to front page the default options for published and promoted to front page are checked. we want to keep the published option checked, so that when a new movie is added to the database it’s automatically published. we’ve deselected the promoted to front page option; however, since we’re using the views module to present data, checking or unchecking this option has no visible effect to the end-user. fields label field name type description taxonomy taxonomy module form menu settings menu module form title node module form the main title of the dvd. note, the title field is required and is not stored in drupal’s content type table (e.g, moviecontent_type_movie). it is stored in the node table. alternative title field_dvdb_alt_title text if the title is not in english, then the english title of the movie, if known. sort title field_dvdb_sort_title text the title of the movie without words like “a”, “the”, “le”, “la.” this field is used for sorting the movies. record id field_dvdb_record_id text the unique id number associated with the item in the library catalog. call number field_dvdb_call_number text the call number of the dvd. short summary field_dvdb_short_summary text information about the movie, taken from the screen scrape. long summary field_dvdb_long_summary text information about the movie, taken from the screen scrape. imdb code field_dvdb_imdb_code text if applicable, the imdb number for the movie. rt code field_dvdb_rotten_tomatoes_code text if applicable, the rotten tomatoes number for the movie. cover field_dvdb_cover image a thumbnail of the dvd cover. note: items in gray were automatically created by drupal. items in black were fields that we created. taxonomy creation the dvd browser’s perl script grabs the genre classification of a movie from the internet movie database (imdb) and dumps this information into the drupal database. in order for the script to input data into the database, we needed to create a vocabulary using drupal’s taxonomy module. when setting up a drupal vocabulary, you can associate content types with a specific vocabulary. we created a vocabulary named genre and associated it with the movie content type. view and block creation the views module allows you to pull data from the drupal database and output it in different ways. the dvd browser uses views to output movies to two sidebar blocks, the genre block, the most recent block,and the #-z header block. theme templates much documentation has been written about theming drupal sites. whether you use an out-of-the box theme (e.g., garland), a third party theme (e.g., newsflash), or create your own theme, you’ll probably need to customize the node.tpl.php template. an out-of-the-box node.tpl.php template usually outputs the content of a node with the following statement: <div class="content"> <?php print $content; ?> </div> the statement doesn’t include <div /> tags that can be used for formatting content with css (cascading style sheets). for example, the following content is displayed using the zen out-of-the box node template, node.tpl.php. figure 1: dvd listing: zen template for the dvd browser, we used zen starter theme as the basis of our own theme. we modified node.tpl.php to use <div /> tags for most of the drupal database fields. below is a screen shot of the same movie, but using the modified file. figure 2: dvd listing: uaf modified zen theme the screen scrape the dvd browser gathers the call number, title, short summary, and long summary by screen scraping the rasmuson library catalog. it then gathers additional information, the long summary, genre, and dvd cover from the internet movie database (imdb), rotten tomatoes, and freecovers.net. this section describes what the perl script is looking for when gathering data from the catalog and other web pages. initial population of the database takes several days due to the large number of videos owned by the library and because our catalog only displays ten items on a page. the perl script calls the catalog url, processes ten items, and begins again, starting where it left off. each time the script is called it scrapes the html source code the catalog for information about newly added dvds and stores the information in a .txt file on the server, for example dvd-22.txt. the script looks at the source code for the case sensitive string details. <tr> <td class="itemlisting2" rowspan="2" width="10%"> <input value="details" name="view^1" id="view1" class="itemdetails" type="submit"> </td> <td class="itemlisting2"> <a href="/uhtbin/cgisirsi.exe/rtbet1khpc/uafras/204850051/20/dvd-22%20videodisc/1/x1002522166/"> <!-current hit; bold it --> <strong> dvd-22 videodisc <!-current hit; unbold it --> </strong> </a> <br>billy's hollywood screen kiss [videorecording] / trimark pictures presents a revolutionary eye production ; co-producers, meredith scott lynn and irene turner ; produced by david moseley ; written and directed by tommy o'haver. <br>&amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;nbsp;o'haver, tommy. </td> <td class="defaultstyle" rowspan="2" width="100" align="left"> <img src="/webcat_images/english/special/link/spacer.gif" alt="" width="100" border="0" height="1"> </td> </tr> it then looks for the call number, which occurs three lines after the word details. in the screen shot above, the call number is dvd-22. the script searches for the title, which occurs just before the case sensitive string [videorecording]. the text after [videorecording] / (in this example, “trimark pictures presents….”), will be stored in the drupal database in the short summary field. to obtain the long summary, the script calls the specific movie (e.g., http://goldmine.uaf.edu/uhtbin/cgisirsi.exe/x/uafras/x/20/dvd-1/1/x1002522166) and looks for the case sensitive string summary:. after getting the title and summary from the library catalog, the script attempts to gather the genre, long summary, and cover from the internet movie database (imdb), the summary and cover from rotten tomatoes, and the cover from freecovers.net. in cases where the program is able to gather multiple summaries, or multiple images for a single dvd we establish an order of preference and only insert one summary or image into drupal per movie but the extra ones are saved to files in case we want to use them later. conclusion our library catalog runs on an oracle database; however, due to vendor restrictions, instead of easily accessing and repurposing data stored in the oracle database, we needed to find a work-around to create a useful dvd browsing tool for our patrons. feedback about the dvd browser has been positive. it is a heavily used tool and our patrons find it much easier to use that the library catalog. future plans for the dvd browser include turning on comments and ratings, so that our patrons can add user-generated content. code there are two files available. the dvd_browser_screen_scraper.pl file continues to be under active development at uaf; future versions will use subroutines and error checking will be refined. node.tpl.php – our modified template file used for formatting the display of nodes in the dvd browser. dvd_browser_screen_scraper.pl – the perl script used for screen scraping the library catalog and for gathering data from imdb, rotten tomatoes, and freecovers.net. a word on dvd covers the dvd browser stores thumbnail images of dvd covers on a locally hosted server. the perl script gathers images from freecovers.net, the internet movie database, or rotten tomatoes. if an image can not be found for a movie, a “no image is available” picture is displayed. we believe that the use of thumbnail images is covered under the doctrine of fair use. besek’s (2003) gives a layman’s overview of fair use. she discusses the four factors of fair use as outlined in the copyright law of the united states of america, section 107. limitations on exclusive rights: fair use. these factors are: “purpose and character of the use;” “nature of the copyrighted work;” “amount and substantiality of the portion used in relation to the copyrighted work as a whole;” and “effect of the use upon the potential market for or value of the copyrighted work.” applying these factors to the dvd browser, we believe that: the “purpose and character of the use” is educational. we are an academic library that has built an internet search application that helps our students search dvd movies owned by the library. the “the nature of the copyrighted work” is creative. creative works, compared with factual works, generally have a stronger case of copyright infringement. however, similar to the kelly v. arriba soft corp. case analysis by donohue (2002), “works that have been previously published, lend themselves more readily to their fair use.” movie art work is published in a variety of formats such as posters, dvd covers, animations, and commercials. the “amount and significance of the portion used in relation to the copyrighted work as a whole” is insignificant. we are providing a low resolution copy of an image. the “effect of the use upon the potential market for or value of the copyrighted work” is insignificant. as in the kelly v. arriba soft case and the perfect 10 v. google case, the images are transformative in nature. the images are being used as a tool to help students select and search for information in an academic setting. note [1] sirsi/dynix has a command line tool, as well as a reporting tool, that could have allowed us to export the marc records from our catalog, however due to administrative reasons, this option was not available to us. references ayazi, sara. “search engines score another perfect 10: the continued misuse of copyrighted images on the internet.” north carolina journal of law and technology 7.2(2006): 367. 2 oct. 2008 < http://jolt.unc.edu/abstracts/volume-7/ncjltech/p367>. besek, june m. “copyright: what makes a use “fair”?” educause review 38.6 (2003): 12-13. 2 oct. 2008 < http://connect.educause.edu/library/educause+review/copyrightwhatmakesausefai/40446>. donohue, kelly. “court gives thumbs-up for use of thumbnail pictures online.” duke law and technology review 0006 (2002). 2 oct. 2008 <http://www.law.duke.edu/journals/dltr/articles/2002dltr0006.html>. u.s. copyright office. copyright law of the united states of america and related laws contained in title 17 of the united states code. 2 oct. 2008 <http://www.copyright.gov/title17/92chap1.html#107>. acknowledgements we’d like to thank eugeniy kalin for his programming efforts in creating the first version of the dvd browser using drupal 4.7. thanks to jim hassel and david basham for coming up with the idea for the dvd browser and for the prototype coding. about the authors ilana kingsley, web librarian university of alaska fairbanks rasmuson & biosciences libraries ilana.kingsley@uaf.edu mark morlino, systems administrator/programmer university of alaska fairbanks rasmuson & biosciences libraries mrmorlino@alaska.edu subscribe to comments: for this article | for all articles 4 responses to "rasmuson library dvd browser: fun with screen scraping and drupal" please leave a response below: jonathan rochkind, 2008-12-20 i would love more information about how you retrieve the information from imdb, rottentomatoes, and freecovers.net. you are just matching on title? do you have much trouble with false positives or false negatives matching on title keyword? do these three services offer any kind of an api, or are you screen-scraping to find content? jonathan rochkind, 2008-12-20 oh, it also occurs to me that i’m not sure how to identify _which_ bib records in my catalog represent movies. how are you determining that, by call number including ‘dvd’ at your library? mark morlino, 2008-12-22 hi jonathan, thanks for your interest in our article. we are just matching by title. imdb and rotten tomatoes are both usually pretty good. if there are multiple matches, the first one is very often the correct one. the development and testing was time consuming because when i encountered false positives or false negatives i modified the code to prevent them, and that would require dumping all of the data and starting from the first dvd to make sure the change to the screen scraping code did not affect the processing of any of the other dvds. freecovers.net is the only one that offers a public api. basically, the api offers the same searching functionality as the web page but returns an xml document rather than an html document. so we can parse the xml into a data structure easily and consistently to look for matches. it is considerably easier to code and less error prone than the screen scraping. unfortunately, the perl module we are using to parse the xml stores the list of possible title matches as a hash, which means we do not maintain the order in which they were sent, and freecovers appends various information to the titles (to indicate a release, a language, a region, or sometimes a video game by the same name) so there are many dvds that have cover images available from freecovers that the program doesn’t because it cannot determine which one to use. the dvds in our catalog all have call numbers that begin with “dvd” the rest of the call number is based on the order in which the dvd was cataloged and whether or not it is part of a multiple dvd set. so it is fairly easy for us to search for them in the catalog by number. the browser can just search for the most recent dvd that it knows about and keep processing results until it finds a call number that does not begin with dvd. i hope this helps. -mark mark411, 2009-08-31 hi! is it possible to retrieve information from covers.hearsay24.com ? leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – wrangling electronic resources: a few good tools mission editorial committee process and structure code4lib issue 9, 2010-03-22 wrangling electronic resources: a few good tools there are several freely available tools today that fill the needs of librarians tasked with maintaining electronic resources, that assist with tasks such as editing marc records and maintaining web sites that contain links to electronic resources. this article gives a tour of a few tools the author has found invaluable as an electronic resources librarian. by brandy klug. introduction the position of electronic resources librarian is one that many libraries have added to their staff in the last decade. a librarian who accepts this position quickly discovers how varied the duties are, and how quickly they can change due to new technologies or staffing changes. the responsibilities of this position will vary depending on the library, but may include web design and maintenance, creating instructional or technical support materials, and maintaining electronic resources. there are numerous free tools available to assist librarians working with electronic resources in various tasks and to help them get started on large projects fairly quickly and easily. this article examines two categories of tools, marc editors and link checkers, that i have found helpful over the last few years when it comes to completing these tasks. the tools have the added benefit of being available at no cost. marcedit: a tool for editing marc records one of the biggest challenges of working with electronic resources is editing marc records. marcedit allows you to do bulk editing in a snap, and the interface is extremely intuitive and user-friendly. marcedit is developed by terry reese and is available for microsoft windows and other operating systems that can use the 2.0 version of the .net framework. the marcedit-l email list hosted by george mason university at http://listserv.gmu.edu/cgi-bin/wa?a0=marcedit-l is a valuable resource for solving tricky problems when using marcedit. marcedit may be used to customize marc records provided by vendors by adding an ezproxy prefix to the link in the 856 field. there are a few ways to do this using marcedit. once the file is opened in marcedit, under tools there is an option to edit subfield data. to edit the subfield using a regular expression, enter the field (856), the subfield (u), ^b for field data, the ezproxy information in the replace with line, select use regular expression, and click on replace text. using a regular expression, a caret followed by the letter b will automatically append the proxy information to the beginning of the 856 $u subfield. figure 1: marcedit: using edit subfield to insert ezproxy prefix the basic replace function located under edit can also be used to complete this task. the electronic resource url stem can be entered in the find what field, and the url stem with the ezproxy in front of it can be entered in the replace with field. clicking the replace all option will make the change. figure 2: marcedit find & replace marcedit may also be used to edit the call number field in electronic book records. libraries may choose to add something like *(online) instead of the letters eb which are usually included at the end of electronic book call numbers. using marcedit, *(online) can replace eb and be added to the end of the call number fields very easily. this can be done by clicking on tools and edit subfield data. to edit the call number subfields enter the field (050), the subfield (b), eb for field data, *(online) in the replace with line, and click on replace text. for ilss that generate call numbers based on 050, when the electronic book records are loaded and the holdings are created, *(online) will be included at the end of the call numbers. figure 3: marcedit: using edit subfield to change call number marcedit is also very useful for adding fields to records. many libraries have added directory of open access (doaj) journals and other free resources to their catalogs. a 500 field denoting free access, for example, can be easily added to these records. this can be done by clicking on tools and add field. enter the field (500), the field data (\\$afree access.), and click on add field. figure 4: marcedit: add/delete field utility if there is a more complicated project where each marc record may need to be examined individually, marcedit is still a huge time-saver. rather than having to pull up the records one at a time in the cataloging module, you can run through the file in marcedit and make edits much faster. marcedit is such a powerful editing tool that i haven’t found anything comparable. however, another tool that might work for making simple marc edits in a pinch is writer from openoffice.org. writer is word processing software that is included as part of the openoffice.org suite of products, and a great tool to use to produce documentation. openoffice.org is available for a variety of platforms including microsoft windows, macintosh os/x, and linux. once the openoffice.org suite has been installed, open the marc file in writer. a simple find/replace can be done by clicking on edit, and find & replace. if there were a group of marc records with a 500 field that needed to be edited, for example, it would be possible to do a blanket find/replace using this tool. the file would then be saved as a text file and named with an .mrc extension. tools for checking links many electronic resources librarians will find themselves in the position of maintaining web pages, lists, or databases containing links to various electronic resources. when maintaining these resources, it is essential to make sure that all web page links are functional. there are few things more frustrating to patrons than dead links. as we all know, online content is volatile. an electronic resource that is available today may not necessarily be available tomorrow. xenu’s link sleuth (often shortened to xenu) is software designed for microsoft windows that requires the user to download the application to their computer. xenu is particularly helpful for checking links on library web resource pages. in order to check links on a particular page, open up xenu, click on file, and select check url. enter the url of the page and check off the option to check external links. click on more options to set the maximum level to 0. selecting maximum level 0 means that xenu will check all links on that page only. higher levels will go deeper and check links on pages that your page links to, and so on. you can also select what to include on the report, such as broken links, redirected urls, and orphaned files. when the process is complete, the report can be viewed as a webpage, or exported to a tab-separated file. the tab-separated file is a text file that can be pasted or imported into excel or another spreadsheet application. once the data is in excel, it can be sorted by status and the urls that may have moved or are no longer available can be checked. xenu can be particularly helpful for checking open access resources such as directory of open access journal (doaj) titles. many libraries are choosing to add marc records for open access resources to their online catalogs. xenu can be used to check these links on a regular basis in order to make sure they are working properly. how you use xenu to check links in the library catalog may depend on your ils and whether or not you have hosted links. if you do not have hosted links, you can use a tool like access to generate a list of titles and urls to check. our electronic journal links are hosted by a vendor. the link in the marc record links to the e-journal a-z list and not directly to the e-journal. to check hosted links using xenu, first a spreadsheet of all electronic journal links needs to be generated from the vendor client center. this spreadsheet has a resource column that can be filtered in excel, or similar spreadsheet application such as openoffice.org’s calc, to get a list of doaj titles and links by selecting the sort & filter option and filter. in the resource column click on the filter and deselect everything. once everything is deselected, click on directory of open access journals. copy and paste the filtered data into a new spreadsheet and save it. the column with the urls can be copied and pasted into a text file. open xenu and select file and check url list. browse to the text file and click open. xenu will begin checking links immediately. figure 5: xenu showing list of urls to check once the check is complete, a tab delimited report can be exported from xenu and saved. it can be imported into a new excel spreadsheet or copied directly from the tab separated file. figure 6: xenu export to tab function the exported information can now be pasted into the saved spreadsheet so the url and status information from xenu will match up with the url and e-journal title in the excel spreadsheet. the status column can then be sorted to pinpoint those urls with potential access problems. when using xenu, it is helpful to test a few different methods to find what works best. i found that reading the documentation created by consortium of academic and research libraries in illinois (carli) was incredibly helpful when it came to finding what worked best for us. [1] two other options for web page link checking are w3c link checker and link valet. you do not need to download any software to use these tools. to use w3c link checker, go directly to the w3c link checker web site and enter the url. when link checking is complete, a report will be generated with a list of redirects and possible broken links. w3c link checker generates a very thorough report. it provides the line number where the link is found in the html code, the link status, and notes about the link status. the report cannot be downloaded, but it can be copied and pasted into a tool like writer or word. to use link valet, go directly to the web site, enter the url, and select ok. a full report provides a list of every link on the page on the left-hand side and color coded messages on the right that indicate whether or not the page is ok or has been moved either temporarily or permanently. this report cannot be downloaded, but as with w3c link checker it can be viewed as a web page or copied and pasted into excel. conclusion the number of electronic resources available continues to grow and so does the need for librarians to find tools to manage them. with many library budgets being cut, it is important for librarians to seek out tools that are available at no cost. most libraries employ only one electronic resources librarian, therefore this person must be proactive when it comes to seeking opportunities for collaboration with librarians in similar positions. there are several ways electronic resources librarians, or any library staff member working with electronic resources, can stay informed and up-to-date on free tools. joining and monitoring e-mail discussion lists is essential and a great way to communicate with other librarians and find out about tools your colleagues are using to complete projects [2]. conferences, library publications, and blogs are also great ways to find out about free tools that will be invaluable when navigating through the exciting world of electronic resources. about the author the author currently works at texas woman’s university library as an electronic resources librarian. she received an mls from texas woman’s university in 2005 and an m.s.ed. from university of nebraska-kearney in 2007. the author may be reached via e-mail at bklug@mail.twu.edu. list of tools link valet – http://htmlhelp.com/tools/valet/ marcedit – http://people.oregonstate.edu/~reeset/marcedit/html/index.php w3c link checker – http://validator.w3.org/checklink writer (part of the open office suite) – http://www.openoffice.org/product/writer.html xenu’s link sleuth – http://home.snafu.de/tilman/xenulink.html reference/notes [1] consortium of academic and research libraries in illinois. (2006). url checking with xenu software. retrieved january 11, 2010, from http://www.carli.illinois.edu/mem-prod/i-share/cat/using_xenu.pdf [2] relevant mailing lists will vary by library, but the ones i have found very helpful would be our ils listserv (voyager-l), autocat, the ezproxy listserv, the marcedit support in technical and instructional matters listserv (marcedit-l), refworks administrators listserv (rwlist), medical libraries discussion list (medlib-l), and the serials in libraries discussion forum (serialst). subscribe to comments: for this article | for all articles 26 responses to "wrangling electronic resources: a few good tools" please leave a response below: david bigwood, 2010-03-22 the problem with xenu and the other link checkers is that they will not work on a file of marc records. one option is to use the free tool from tom tyler, marcxgen. it turns a marc file into an html page that can then be checked by the link checkers you mentioned. http://tinyurl.com/2odo4m another link checker i find very useful is the firefox plugin from oclc, link evaluator. http://evaluator.oclc.org/ brandy klug, 2010-03-22 david, thanks for the information about marcxgen. i look forward to trying it out. brandy klug, 2010-03-22 david, thanks for your comments. i am looking forward to trying marcxgen. a step by step guide to modern broken link building, 2015-08-19 […] “file” menu option is what you’ll use the […] a step by step guide to modern broken link building | seo nyc, 2015-08-19 […] “file” menu option is what you’ll use the […] a step by step guide to modern broken link building michael j. goldrich, 2015-08-19 […] “file” menu option is what you’ll use the […] a step by step guide to modern broken link building | seo nyc & digital marketing, 2015-08-19 […] “file” menu option is what you’ll use the […] a step by step guide to modern broken link building | business trends nyc, 2015-08-19 […] “file” menu option is what you’ll use the […] a step by step guide to modern broken link building | greggsdomains, 2015-08-19 […] “file” menu option is what you’ll use the […] a step by step guide to modern broken link building -, 2015-08-19 […] “file” menu option is what you’ll use the […] a step by step guide to modern broken link building | jvweeklybonus, 2015-08-19 […] “file” menu option is what you’ll use the […] a step by step guide to modern broken link building | kyean.org, 2015-08-19 […] “file” menu option is what you’ll use the […] a step by step guide to modern broken link building | marketing force, 2015-08-20 […] “file” menu option is what you’ll use the […] a step by step guide to modern broken link building | short accountancy ltd, 2015-08-20 […] “file” menu option is what you’ll use the […] a step by step guide to modern broken link building | mortice and green, 2015-08-20 […] “file” menu option is what you’ll use the […] a step by step guide to modern broken link building | tax twerk, 2015-08-20 […] “file” menu option is what you’ll use the […] a step by step guide to modern broken link building | push button marketing, 2015-08-20 […] “file” menu option is what you’ll use the […] a step by step guide to modern broken link building | developing power to succeed, 2015-08-20 […] “file” menu option is what you’ll use the […] a step by step guide to modern broken link building | my blog, 2015-08-20 […] “file” menu option is what you’ll use the […] a step by step guide to modern broken link building | seanmorrisonn, 2015-08-20 […] “file” menu option is what you’ll use the […] a step by step guide to modern broken link building | paulbarletta, 2015-08-20 […] “file” menu option is what you’ll use the […] a step by step guide to modern broken link building | anthony lewis, 2015-08-20 […] “file” menu option is what you’ll use the […] a step by step guide to modern broken link building | ds domination online, 2015-08-20 […] “file” menu option is what you’ll use the […] a step by step guide to modern broken link building | websetnet, 2015-08-20 […] “file” menu option is what you’ll use the […] a step by step guide to modern broken link building | online digital profits, 2015-08-23 […] “file” menu option is what you’ll use the […] a step by step guide to modern broken link building | profit maxim tips&tricks, 2015-09-20 […] “file” menu option is what you’ll use the […] leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – on the nature of extreme close-range photogrammetry: visualization and measurement of north african stone points mission editorial committee process and structure code4lib issue 51, 2021-06-14 on the nature of extreme close-range photogrammetry: visualization and measurement of north african stone points image acquisition, visualization, and measurement are examined in the context of extreme close-range photogrammetric data analysis. manual measurements commonly used in traditional stone artifact investigation are used as a starting point to better gauge the usefulness of high-resolution 3d surrogates and the flexible digital tool sets that can work with them. the potential of various visualization techniques are also explored in the context of future teaching, learning, and research in virtual environments. by michael j. bennett i. introduction figure 1. artifact 2005.002.007 bag 5a point 425, max height 30.13mm. max width 18.95mm. gsd 9.52 μm/pix, https://skfb.ly/6x9tx from the caldwell collection of north african points, https://skfb.ly/6vr9d as libraries, archives, and museums continue to explore and implement the digital capture and rendering of their 3d collection objects, it is important to understand and anticipate both the near-term and future uses of the resulting data sets from such activities. novel repurposed data needs can, in turn, form a dynamic feedback loop to help guide appropriate levels of original acquisition fidelity as analysis methods of rendered outputs advance through time. unlike the 2d imaging ecosystem, guidelines and terminology for assessing the accuracy and resulting quality of image-based 3d reconstructions lack normalized use. standardized performance measurements like the fundamental 2d imaging resolution metric, sfr (iso 12233, 15529), have yet to be similarly codified in the 3d domain (webb, 2020) (saidi, 2020). though this gap and the challenges of evaluating data in three dimensions are real, there exist measurements from other areas of remote sensing that can have direct application with objects from cultural heritage. alongside these issues of metrology and geometric accuracy and precision, there remain the more qualitative aspects of captured and processed 3d visual information that can be texture mapped on top of fine polygonal information. in fields such as archaeology and anthropology, visual cues found in both the millimeter and submillimeter levels can shed new investigative light on things like the taxonomic classification of artifacts and are critical to rapidly evolving movements in teaching, learning, and scholarship in the digital and online realms. ii. background during spring 2020, the author conducted initial experiments with small objects and extreme close-range photogrammetry through focus stacking to create 3d recordings with full object views (bennett, 2020a). these tests were made to better determine whether this technique could help alleviate the problems of blurring and the loss of spatial resolution that close focus and narrow depth of field present photogrammetric 3d reconstructions. focus stacking involves the acquisition of a bracket of images with varying focal points. these are subsequently focus stacked through computational means into final rendered images that possess an increased depth of field. in addition, the author’s early examination explored similar methods to those described by gallo et al (2014) and their work on partial 3d object views with the idea of extending these foundational techniques towards more comprehensive, full 3d object views of small lithics. overall, these initial efforts showed promise and subsequently led to the forming of an interdisciplinary, cross-campus working group made up of faculty from the university of connecticut’s anthropology department, and staff from the connecticut state museum of natural history, office of the state archaeologist, and the uconn library. its goal was to run 10 – 20 stone artifacts from the state museum’s caldwell collection of north african points through the extreme close-range photogrammetric workflow that the author had begun to refine in the spring. with the sudden reliance on digital and virtual collections brought on by covid-19 lockdowns, it was felt that members of the anthropology department and archaeology educators and scholars at large might find new pedagogic and research use of such new high quality assets. concurrent with these deliverables, the author also wished to explore the boundaries of extreme close-focus 2d image capture in the context of feasible and consistent photogrammetric reconstructions. in this way it was hoped that a better understanding could be achieved on the relationships among acquisition quality, downstream geometric accuracy, discernible visual detail, and general production time and data archiving costs. as the initial photogrammetry/focus stacking experiments were conducted at the author’s home studio during the early days of the covid-19 shutdown, efforts towards the working group’s aims could be seamlessly continued there and gradually improved upon as time went on. key iterative changes made beyond the early spring experimentation phase included upgrading to a faster and higher capacity 13 inch apple macbook pro (macos 10.15.7, 32gb ram, 2.3 ghz quad-core intel core i7, 4tb ssd). also, helicon focus pro v7.6.4 substituted for photoshop to complete needed focus stacking from the large focus-bracketed image stacks. helicon focus’ ability to split and batch process multiple stacks to go along with its selectable rendering methods made it well suited to more elegantly complement the large image sets that resulted from the increasingly close-range photogrammetry that evolved as the project progressed. figure 2. photogrammetry/focus stacking rig. the original surplus tripod was also swapped out for a manfrotto 057 carbon fiber unit with geared column. this greatly assisted in the raising and lowering of the heavy camera, cognisys focus rail, and manfrotto 3-way geared pan and tilt head combination between successive look angles on the stone points as they were photographed. a programmable cognisys turntable was also eventually added. finally, an amd radeon pro wx 8200 graphics card and razer core x thunderbolt 3 graphics expansion chassis were also slotted into the existing hardware chain to better optimize both agisoft’s metashape pro and helicon focus’ abilities to leverage available gpu power (figure 2). the amd wx 8200 card was judged to be a good balance of size and speed (8gb, 512 gb/s bandwidth) paired with the computational stability of the amd pro series’ overall design. also, when not in use in the external enclosure, this graphics card could easily be swapped into the uconn library’s digital production lab’s automated 3d capture workstation (bennett, 2018) once operations there returned to pre-pandemic levels. iii. rendered data and performance metrics with few exceptions, the majority of the study’s stone points were captured using four total look angles (two per object orientation). a canon eos 6d dslr camera with 35.8mm x 23.9mm, 5,472 x 3,648px cmos sensor was alternately used with either a zeiss milvus 50mm f/2m ze or zeiss milvus 100mm f/2m ze macro lens. in addition, the 100mm zeiss lens was fitted with two stacked canon ef 12 ii extension tubes to further shorten its minimum focus distance. both of these lenses are manual-focus-only and have long and heavy focus throws which are very stable once set. as a result, they are well-suited for subsequent structure from motion (sfm) camera calibration, alignment, and sparse point cloud generation which are foundational to overall high quality photogrammetric data creation (“structure from motion,” 2021). a kneadable eraser base was used to mount objects in either up or down orientation on top of the turntable (figure 2) as described by porter, et al (2016). based upon the size and shape of a given object, focus brackets of 8 to 48 images per camera position were employed to increase depth of field in order to keep the object in focus at each position throughout a given 360° circuit. turntable rotation was 10° per camera position (and focus bracket). these image groups were then focus-stacked in helicon focus pro v7.6.4 using its depth map rendering method and split stack batch process. helicon focus’ depth map algorithm was deemed optimally suited among the software’s methods due to its ability to better mitigate direct reflections and glare that often appeared on lithics based upon their rock type and given angles to camera and lights. final processed images were then imported into agisoft metashape pro v1.6.4 where they were aligned (cultural heritage imaging | photogrammetry, 2021; mallison, 2015; schroer et al., 2014), and masked following the “up/down” workflow established by porter (2016). dense point clouds were initially generated with metashape’s high quality parameter for the first nine models captured using the zeiss 50mm lens. however, after installation of the amd external graphics card into the hardware chain, the remaining lithics were shot with the zeiss 100mm lens and processed with the ultra high quality dense cloud parameter. a typical final image network can be seen in figure 3. figure 3. final image network for the stone artifact 2005.002.007 bag 5a point 425, from the caldwell collection of north african points. ground sampling distance (gsd) is the resolution of a surface model which is governed by the area on the real-world subject represented by the pixels in the images from which the surface model was generated (cultural heritage imaging | photogrammetry, 2021). gsd can also be thought of as the size of an object or feature in object space represented by a single pixel in the image space. it can be estimated from the camera distance from the object, the focal length of the lens, and the pixel size of the camera. gsd can also be used to understand the parameters required for recording the smallest detail that needs to be resolved through image-based 3d reconstructions (andrews et al., 2015; matthews, 2008; remondino et al., 2013; verhoeven, 2018, 2019; webb, 2020). in many ways it is analogous to the 2d imaging world’s idea of sampling frequency. however, on their own, neither sampling frequency nor gsd completely describe 2d or 3d systems’ performance, since no system operates at 100% efficiency. instead these measurements more accurately describe a given system’s potential. spatial frequency response (sfr) is used in 2d to describe an imaging system’s ability to maintain the relative contrast of features within a given spatial proximity (iso 12233, iso 16067-1, iso 16067-2, iso 15524). sfr takes into consideration the effects that levels of optical “blur” from imaging optics, hardware, imaging processing software, and environmental factors can have on captured images (williams, 2010). as a result, sfr can more clearly determine a 2d imaging system’s overall capacity for quality. but is there a similar comprehensive metric to gauge the measurement performance of image-based 3d reconstructions? for cultural heritage recording, the registration of individual 3d point clouds is generally achieved using the object’s own geometry (natural points, distinctive surface features) (luhmann et al., 2014; “structure from motion,” 2021). more specifically metashape pro employs a combination of proprietary sfm algorithms that are similar in nature to scale-invariant transform (sift) for such initial feature detection (“scale-invariant feature transform,” 2021; semyonov, 2011). for a given sparse point cloud, the program can in turn calculate its root mean square (rms) reprojection error. the rms reprojection error is the distance between the point on the image where a reconstructed 3d point can be projected and the original projection of that 3d point detected on the photo and used as a basis for the 3d point reconstruction procedure (agisoft metashape user manual – professional edition, version 1.6, 2020). though an insightful indicator of initial camera alignment accuracy, rms reproduction error does not factor in subsequent photogrammetric processing where inaccuracies of dense point cloud creation, mesh generation, and texture mapping can potentially occur. in turn, its use is naturally limited. unlike 2d digitization with its maturing performance metrics and test targets (rieger, 2016; van dormolen, 2012), image-based 3d reconstruction for cultural heritage lacks standardized test objects, analysis protocols, and reliable reference data which make accuracy assessment challenging (webb, 2020). however, rules of thumb on determining the smallest resolvable detail of acquired 3d geometry have been posited based upon nyquist–shannon sampling theorem (“nyquist–shannon sampling theorem,” 2021), experimentation in remote sensing, and known optical aberrations present in imaging systems. for example, in aerial photography applications, verhoeven (2018, 2019) has suggested that gsd should be at least three times smaller than the required limiting spatial resolution. with regard to image-based 3d reconstructions, remondino et al (2013) have similarly advised that gsd should be at least 2-3 times smaller than the smallest geometric detail to be captured. in turn, the author has multiplied the gsd value for each of the study’s reconstructed stone objects by 3 to roughly estimate their smallest resolvable detail in mm (table 1). it should be noted that image-based reconstructions can vary in quality across 3d surfaces due to problems that photogrammetric processing algorithms can suffer as a result of an object’s physical properties. these properties can include occlusions, reflectance, translucence, and low frequency features. however, the estimated figures presented here, which range from 0.02577mm to 0.06870mm among sampled objects, do provide an initial glimpse into the effectiveness that focus stacking can have on the creation of resulting photogrammetric geometries. table 1. stone artifact metrics sorted by ground sampling distance (gsd). such accuracy, however, does come with certain costs. these include very large image data sets and lengthy capture times (table 1). nevertheless both capture and focus stacking/photogrammetry pipelines have been designed by the author to be largely automated through programmable hardware and optimized software that are geared for large batch processing. as an illustration, beyond mounting an object, adjusting the camera look angle, and programming the focus rail to an appropriate focus bracket range, subsequent photographic acquisition can be done completely unattended (bennett, 2020a). at the time of writing, the complete data sets of each of the study’s objects are in the process of being archived in the connecticut digital archive open repository (caldwell collection of north african points data sets, 2021). iv. measurement and visual analysis figure 4. artifact 2005.002.007 bag 5a point 424, from the caldwell collection of north african points. max height 19.14mm. max width 12.02mm. common measurements of archaeological artifacts include maximum height and width (banning, 2006). mostly done with hand-held tools either in situ or in a lab environment, similar measurements can be digitally conducted with high precision using scaled, image-based 3d reconstructions and open source software (bennett, 2020b). meshlab is one such application (cignoni et al., 2020). figure 5 illustrates a simple maximum height and width measurement of the euclidean geometry of a sample 3d surrogate within the program. figure 5. maximum height and width measurements in millimeters of artifact 2005.002.007 bag 5a point 424 using meshlab. angle measurements (figure 6) are also possible through virtual protractor tools such as onscreenprotractor (straffi, 2016). figure 6. angle calculation of artifact 2005.002.007 bag 5a point 424 using onscreenprotractor. precise cross-section geometric analyses can also be achieved in software from such high quality rendered data. these non-destructive manipulations, which are impractical to conduct on the physical artifacts themselves, can open up new lines of inquiry with relative ease. for example, cross-sections can be deftly sliced, measured, and exported for further examination from meshlab (figures 7-9). from these, automated artifact classification based on 3d shape or profile can be accomplished. figure 7. artifact 2005.002.007 bag 5a point 424 centered to xyz world coordinates in meshlab. figure 8. artifact 2005.002.007 bag 5a point 424 rotated 90° along z axis in meshlab. top of point facing viewer. figure 9. cross-section of artifact 2005.002.007 bag 5a point 424 measured in meshlab. geometric measurements made of cross-sections in meshlab can also be combined with various angle calculations using onscreenprotractor (figure 10). figure 10. combining measured cross-section in meshlab with angle calculations made with onscreenprotractor tool. lithics can also be illuminated in meshlab by various virtual light sources from different directions (figure 11). as a result light can be raked by the user across surface features to help bring out fine textured geometry. figure 11. raking light from a single light source in meshlab. these visual details, in turn, can be further highlighted by meshlab’s unsharp mask geometry filter. features can also be easily measured to sub-millimeter levels of precision from the rich 3d data acquired and processed in this study (figure 12). figure 12. visual sub-millimeter detail of artifact 2005.002.007 bag 5a point 424 measured in meshlab and enhanced with raking light and the application of the program’s unsharp mask geometry filter. sketchfab (figure 13) also offers similar dynamic, user-controlled lighting within the browser for models processed with shading. figure 13. animated detail of user-controlled raking illumination in sketchfab of artifact 2005.002.007 bag 5a point 447 – shaded version, https://skfb.ly/6zgsk additionally, untextured, solid model visualizations can be further rendered through meshlab, metashape pro, and sketchfab shaders as well. these can bring out extremely detailed characteristics of negative surface scarring left by the original manufacturing process and ripples in the rock for close surface analysis (figures 14-16). figure 14. untextured, solid model of artifact 2005.002.007 bag5a point 459 in metashape pro. figure 15. untextured, solid model of artifact 2005.002.007 bag5a point 459 in meshlab. figure 16. untextured, solid model of artifact 2005.002.007 bag5a point 459 in sketchfab, https://skfb.ly/6yaa6 finally, simply zooming and panning into and around the textured models can give deep visual insight into the various sub-millimeter inclusions in the rock itself (figure 17). such details can help investigators better answer questions on the types of raw materials that were used in lithic industries through time. figure 17. detail from artifact 2005.002.007 bag 5a point 447, gsd 0.012 mm/pix, as visualized in metashape pro. model is tilted backwards resulting in a look up the front surface face towards the tip. model may also be viewed on sketchfab https://skfb.ly/6xgup v. conclusion the results of this study corroborate the initial feasibility assessments of gallo et al (2014) on focus stacking techniques coupled with photogrammetric processes for cultural heritage objects. resolvable detail down to 26µ has been conservatively estimated with the author’s modified use of this combined workflow for complete 3d views of small north african stone points. the accuracy of this method may be even further improved through the use of machine-readable targets and scale bars. according to sapirstein (2018), targets allow camera orientations and calibrations to be refined in software beyond the estimates obtained by sfm alone. however, fitting such coded targets into the extremely close-focus and limited depth of field scenes of this study would have increased the amount of needed source images beyond feasible time and storage capacities. additionally, with ongoing advances in computer vision, target-free camera calibration and fully automated orientation via feature-based matching techniques have been reported to achieve similar sub-pixel accuracy compared to targeted networks (stamatopoulos & fraser, 2014). in turn, neither coded targets nor scale bars were used by the author. object scale was instead applied to resulting 3d data based upon digital caliper measurements of the artifacts’ maximum height and width values. future comparative research on the deployment of higher resolution cameras is thus warranted. new mirrorless options (blair, 2021; “fujifilm gfx100s,” 2021) could theoretically be less prone to mirror-slap vibrations (and the need for mirror lockup), and allow for faster image acquisition, longer shooting distances, wider depth of field, and larger captured scene sizes. in combination, these factors could potentially allow for the use of coded targets and scale bars while gathering data at similar gsd values as described in this inquiry (table 1). how these different acquisition parameters would affect focus stacking needs, resulting throughput rates, and storage requirements remain open questions for further inquiry. regardless, the lack of consistency in applied terminology, metrics, and test practices of 3d spatial (and colorimetry) data remains. this absence of functional standardization, particularly within cultural heritage, makes comparative studies of both the existing and future literature difficult and ambiguous as a result. in turn, well-understood, applicable measurement guidelines and the greater availability of open access 3d research data sets are needed for optimal coherence and transparency within the field. finally, smoothing of small areas of reconstructed geometry can be common during mesh creation of sharp-edged and often reflective artifacts like stone points. however, fine details that are difficult to discern from the mesh can often be found in the photographic texture (sapirstein, 2016, 2018). in this way, the importance that extreme close-focus image acquisition can have not only with conveying overall visual detail but also with how it can in certain instances compensate for local errors in reconstructed geometry adds additional merit to the process outlined in this study. as it has been noted throughout the discussion, high resolution image capture can thus serve multiple functions towards the creation and rendering of 3d data and enrich its educational and scholarly potential. vi. acknowledgements the author wishes to thank university of connecticut colleagues, professor christian tryon, department of anthropology, for selecting the study’s original stone points, and dr. jacqueline veninger-robert, office of state archaeology, for facilitating the viewing and lending of artifacts from the caldwell collection. the author would also like to express his appreciation to rhonda kauffman, metadata management librarian, for her assistance in creating and deploying the project’s 3d data sets’ metadata schema, and to michael kemezis, repository manager, connecticut digital archive, for his help with the study’s data sets uploads to the ctda. bibliography agisoft metashape user manual—professional edition, version 1.6. (2020). 160. andrews, d., bedford, j., & bryan, p. (2015). metric survey specifications for cultural heritage. banning, e. b. (2006). the archaeologist’s laboratory: measurement and error. http://homes.chass.utoronto.ca/~banning/arh%20312/312lab02.htm bennett, m. j. (2018, august 1). automating original image capture for photogrammetry—tundra graphics. tundra graphics. https://tundragraphics.com/blog/2018/08/automating-3d-capture/ bennett, m. j. (2020a, june 26). small object photogrammetry through focus stacking—tundra graphics. tundra graphics. https://tundragraphics.com/blog/2020/06/small-object-photogrammetry-through-focus-stacking bennett, m. j. (2020b). examining the geometric and visual details of high resolution 3d lithics. university of connecticut library white paper. https://opencommons.uconn.edu/libr_pubs/66 blair, c. (2021, march 24). canon to release a 100mp eos r system camera next year. canon rumors. https://www.canonrumors.com/canon-to-release-a-100mp-eos-r-system-camera-next-year-cr2/ caldwell collection of north african points data sets. (2021). http://hdl.handle.net/11134/20006:caldwell cignoni, p., callieri, m., corsini, m., dellepiane, m., ganovelli, f., & ranzuglia, g. (2020). meshlab (2020.12) [computer software]. https://www.meshlab.net/ cultural heritage imaging | photogrammetry. (2021). http://culturalheritageimaging.org/technologies/photogrammetry/ fujifilm gfx100s. (2021). in wikipedia. https://en.wikipedia.org/w/index.php?title=fujifilm_gfx100s&oldid=1012601450 gallo, a., muzzupappa, m., & bruno, f. (2014). 3d reconstruction of small sized objects from a sequence of multi-focused images. journal of cultural heritage, 15(2), 173–182. https://doi.org/10.1016/j.culher.2013.04.009 luhmann, t., kyle, s., & boehm, j. (2014). close-range photogrammetry and 3d imaging (2nd ed.). de gruyter. mallison, h. (2015, october 11). photogrammetry tutorial 11: how to handle a project in agisoft metashape (photoscan) | dinosaurpalaeo. https://dinosaurpalaeo.wordpress.com/2015/10/11/photogrammetry-tutorial-11-how-to-handle-a-project-in-agisoft-photoscan/ matthews, n. a. (2008). aerial and close-range photogrammetric technology: providing resource documentation, interpretation, and preservation. bureau of land management technical note 428. http://www.blm.gov/pgdata/etc/medialib/blm/wo/planning_and_renewable_resources/coop_agencies/paleontology_library/paleo_publications.par.69972.file.dat/technote428.pdf nyquist–shannon sampling theorem. (2021). in wikipedia. https://en.wikipedia.org/w/index.php?title=nyquist%e2%80%93shannon_sampling_theorem&oldid=1002056076 porter, s. t., roussel, m., & soressi, m. (2016). a simple photogrammetry rig for the reliable creation of 3d artifact models in the field: lithic examples from the early upper paleolithic sequence of les cottés (france). advances in archaeological practice, 4(1), 71–86. https://doi.org/10.7183/2326-3768.4.1.71 remondino, f., menna, f., koutsoudis, a., & chamzas, c. (2013). design and implement a reality-based 3d digitisation and modelling project. 2013 digital heritage international congress (digitalheritage), 137–144. https://doi.org/10.1109/digitalheritage.2013.6743723 rieger, t. (2016). fadgi technical guidelines for digitizing cultural heritage materials, september 2016. http://www.digitizationguidelines.gov/guidelines/fadgi%20federal%20%20agencies%20digital%20guidelines%20initiative-2016%20final_rev1.pdf saidi, k. s. (2020, april 6). the need for 3d imaging performance standards for robotic assembly | 2020-04-06 | quality magazine. quality magazine. https://www.qualitymag.com/articles/96011-the-need-for-3d-imaging-performance-standards-for-robotic-assembly sapirstein, p. (2016). accurate measurement with photogrammetry at large sites. journal of archaeological science, 66, 137–145. https://doi.org/10.1016/j.jas.2016.01.002 sapirstein, p. (2018). a high-precision photogrammetric recording system for small artifacts. journal of cultural heritage, 31, 33–45. https://doi.org/10.1016/j.culher.2017.10.011 scale-invariant feature transform. (2021). in wikipedia. https://en.wikipedia.org/w/index.php?title=scale-invariant_feature_transform&oldid=1011461449 schroer, c., mudge, m., & lum, m. (2014). photogrammetry training: practical, scientific use of photogrammetry in cultural heritage. semyonov, d. (2011, may 3). algorithms used in photoscan. https://www.agisoft.com/forum/index.php?topic=89.0 stamatopoulos, c., & fraser, c. s. (2014). automated target-free network orientation and camera calibration. isprs annals of photogrammetry, remote sensing and spatial information sciences, ii–5, 339–346. https://doi.org/10.5194/isprsannals-ii-5-339-2014 straffi, p. (2016). on-screen protractor (0.5) [computer software]. http://osprotractor.sourceforge.net/protractor.html structure from motion. (2021). in wikipedia. https://en.wikipedia.org/w/index.php?title=structure_from_motion&oldid=1000495782 van dormolen, h. (2012). metamorfoze preservation imaging guidelines. 44. verhoeven, g. j. (2018). resolving some spatial resolution issues – part 1: between line pairs and sampling distance. https://doi.org/10.5281/zenodo.1465017 verhoeven, g. j. (2019). resolving some spatial resolution issues – part 2: when diffraction takes over. https://doi.org/10.5281/zenodo.3518178 webb, e. k. (2020). optimising spectral and 3d imaging for cultural heritage documentation using consumer imaging systems [doctor of philosophy, university of brighton]. https://research.brighton.ac.uk/files/21915605/webb_final_thesis_redacted.pdf williams, d. (2010, may 5). making sense of digital imaging performance in a cultural heritage environment. image science associates seminar, library of congress. about the author michael j. bennett is head of digital imaging and conservation at the university of connecticut. there he oversees the digital capture and conservation operations for the university’s archives and special collections. his research interests include technologies and techniques that focus on digitization, post-processing, and 2d and 3d data formats. he holds a ba from connecticut college and an mlis from the university of rhode island. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – improving access to archival collections with automated entity extraction mission editorial committee process and structure code4lib issue 29, 2015-07-15 improving access to archival collections with automated entity extraction the complexity and diversity of archival resources make constructing rich metadata records time consuming and expensive, which in turn limits access to these valuable materials. however, significant automation of the metadata creation process would dramatically reduce the cost of providing access points, improve access to individual resources, and establish connections between resources that would otherwise remain unknown. using a case study at oregon health & science university as a lens to examine the conceptual and technical challenges associated with automated extraction of access points, we discuss using publically accessible api’s to extract entities (i.e. people, places, concepts, etc.) from digital and digitized objects. we describe why linked open data is not well suited for a use case such as ours. we conclude with recommendations about how this method can be used in archives as well as for other library applications. by kyle banerjee and max johnson introduction archival collections contain a wealth of knowledge in finding aids, oral histories, letters, and other materials. however, subject access to these resources is minimal, and it’s difficult for researchers to identify information relevant to their needs in these resources or in supplementary materials such as blog entries, reports, photos, exhibit text, and other materials created by archivists, special collections librarians, and their staff. manual assignment of access points is too time consuming and expensive to be practical for large numbers of items or when staff are under time pressure — providing access points to a single oral history might take a skilled professional two to three hours. in late 2014, staff at oregon health & science university (ohsu) initiated an experiment to see if cloud-based entity extraction services could help with this problem. if the entities — such as people, places, and concepts — within archival resources could be identified automatically, then new access points could be created more efficiently. valuable linkages could be created among items that might otherwise never be discovered, allowing researchers to search collections more quickly and comprehensively. we found that staff assisted with automated identification of entities (i.e. people, places, concepts, etc.) can provide access points to an oral history within a few minutes, demonstrating that such tools could help improve access to archival collections. our experiment focused on oral histories because this class of documents is finite and currently has minimal access at ohsu, but we also attempted to extract entities from other types of documents including encoded archival description (ead) finding aids, drug approval documents, and web pages. we briefly examined a number of services including aylien, meaningcloud, opencalais, semantria, textrazor, and zemanta before deciding that alchemyapi met our specific needs best. choosing an api many of the text extraction services we explored were not compatible with our needs because they either appeared optimized for processing particular types of documents such as news (aylien, opencalais, textrazor) or short communications like tweets (datatxt, semantria), or they are designed to support particular services such as content based ads (zemanta) or competitive intelligence (meaningcloud). we had no budget, so we tested api keys for free tiers of service. the restrictions associated with the accounts were usually documented clearly — typically a charge against a quota. however, we sometimes received inconsistent results in repeated tests against the same document with some services. this behavior repeated itself too frequently to be attributable to changes in the service itself. vendor documentation provided no explanation as to why this might happen, so we speculate it could possibly reflect the service returning whatever it can within a given amount of time. despite the limitations we encountered, we felt the free accounts were permissive enough to allow us to determine the suitability of individual services for our needs. while more than one of the services could have met our needs adequately, we selected alchemyapi because it performed better with some of the specific types of entities that interested us, less postprocessing of output was necessary, and our use case fit more easily within their free use tier. issues with entity extraction simply extracting entities is of limited utility, even if this task can be performed at 100% accuracy because documents often mention entities that are unimportant. for example, every oral history in ohsu’s archives references ohsu. the vast majority of archival documents in our collection reference oregon, and there are a number of other entities that commonly appear in documents and records. within the context of ohsu collections, mentioning oregon, ohsu, and major departments within ohsu is not significant so these entities should be used as access points only for documents that actually are about these things. otherwise, the sheer number of entries makes it harder to find resources relevant to these entities. our project aims to identify access points to be presented as simple web links that help people find related items. in other words, we want meaningful tags to add to metadata records. surrogate (i.e. metadata) searches support different use cases than full text searches, so the key is to identify a small number of relevant entities that represent the “aboutness” of a resource rather than a large number of entities that are mentioned in a resource. identifying relevance is difficult within the context of entity extraction. frequent mention of ohsu, the school of nursing, bone cancer, or fluoxetine (the generic name for prozac) in a document is not by itself a reliable indicator that an entity is relevant. likewise, the fact that a term appears only once does not mean it is not highly important — to make textual content less redundant, people often reference things obliquely, incompletely (e.g. “dr. beals,” “school of medicine,” etc.) or using terms representing various levels of abstraction. we found that extracting as much relevant information as possible, but not so much that working with the extracted information takes more time than it would for a human operator to manually scan materials and identify access points, was challenging. text extraction software relies on heuristics and pattern matching to determine which entities are present and what type of entities they are. this is a complex task because names for people, places, and things are often identical — consider how the entity “washington” could be any one of many people, geographic or political entities, roads, bridges, buildings, etc. this task is further complicated in that information about different types of objects may be structured in ways that the heuristic algorithms may not anticipate. our entity extraction tests proved much more successful on unstructured documents than on structured and semistructured metadata such as ead. we hoped alchemyapi or other services could extract relevant entries from extensive ead documents as these frequently reference people, organizations, and other entities of interest. however, data in ead and marc records are structured according to library conventions — for example names are presented in inverted order, geographic and corporate entities are frequently qualified, and the information is generally presented in a terse manner. such conventions led to a dramatic reduction in accuracy from tested entity extractors which assume documents are expressed in natural language. for specialized applications such as archives, documents follow specific structures and most important entities are domain specific. however, the cloud-based services we tested cannot be trained with local data and documents. they consult predetermined knowledgebases and their algorithms are optimized for extracting information from news and social media. nonetheless, these services can be used to extract terms from a representative corpus of documents which can then be curated so that specific access points could be targeted or rejected with the aid of scripts. a simple example the three line script below illustrates how alchemyapi can extract entities from an oral history served as a pdf on the web. all services we tested offered a simple rest based api that can be integrated into any process using the language of one’s choosing. in addition to being accessible via direct http get and post calls, alchemyapi provides sdk’s for python, php, ruby, and node.js as well as a particularly easy-to-use command line client. depending on the parameters used in api calls, alchemyapi can return entities in a number of formats including json, rdf, delimited text, and xml. it can also return linked open data (lod) — i.e. information about the entities themselves as well as the data sources for that information. the following lines download a pdf from the web, convert it to text, and extract entities which are described in xml using alchemyapi: ~$ wget 'http://digitalcommons.ohsu.edu/cgi/viewcontent.cgi?article=1000&context=hca-oralhist' -o temp.pdf ~$ pdftotext temp.pdf ~$ alchemycmd --mode entity -s text -f temp.txt -o xml that returns the following output: <?xml version="1.0" encoding="utf-8"?> <results> <status>ok</status> <warningmessage>truncated-oversized-text-content</warningmessage> <usage>by accessing alchemyapi or using information generated by alchemyapi, you are agreeing to be bound by the alchemyapi terms of use: http://www.alchemyapi.com/company/terms.html</usage> <url></url> <language>english</language> <entities> <entity> <type>person</type> <relevance>0.923889</relevance> <count>72</count> <text>dr. rodney k. beals</text> </entity> <entity> <type>organization</type> <relevance>0.460871</relevance> <count>22</count> <text>medical school</text> </entity> <entity> <type>stateorcounty</type> <relevance>0.417948</relevance> <count>18</count> <text>oregon</text> <disambiguated> <name>oregon</name> <subtype>location</subtype> <subtype>politicaldistrict</subtype> <subtype>administrativedivision</subtype> <subtype>governmentaljurisdiction</subtype> <subtype>usstate</subtype> <subtype>wineregion</subtype> <website>http://www.oregon.gov</website> <dbpedia>http://dbpedia.org/resource/oregon</dbpedia> <freebase>http://rdf.freebase.com/ns/m.05kj_</freebase> <opencyc>http://sw.opencyc.org/concept/mx4rvvi3ypwpebgdrcn5y29yca</opencyc> <census>http://www.rdfabout.com/rdf/usgov/geo/us/or</census> <yago>http://yago-knowledge.org/resource/oregon</yago> <geonames>http://sws.geonames.org/5744337/</geonames> </disambiguated> </entity> ...[most of output omitted]... </entities> </results> if pdf to text conversion is unnecessary, a browser can be used to explore alchemyapi simply by pasting url’s containing supported parameters, e.g.: http://access.alchemyapi.com/calls/url/urlgetrankednamedentities?url=[url]&apikey=[secret] working with api output all entity extractors return far more data than is useful — services that don’t truncate output due to parameters passed to the api or service limitations returned hundreds of entities for a single oral history, many of which were not useful for purposes of creating access points. in addition, most extractors also returned a great deal of information about many of the entities. for example, the phrase “oregon health & science university” frequently appears in documents at oregon health & science university. when alchemyapi sees that word, it returned: <entity> <type>organization</type> <relevance>0.230005</relevance> <count>2</count> <text>ohsu</text> <disambiguated> <name>oregon health &amp; science university</name> <subtype>location</subtype> <subtype>collegeuniversity</subtype> <subtype>hospital</subtype> <subtype>university</subtype> <geo>45.498333333333335 -122.68555555555555</geo> <website>http://www.ohsu.edu</website> <dbpedia>http://dbpedia.org/resource/oregon_health_&amp;_science_university</dbpedia> <freebase>http://rdf.freebase.com/ns/m.05l0dl</freebase> <opencyc>http://sw.opencyc.org/concept/mx4rvwpdd5wpebgdrcn5y29yca</opencyc> <yago>http://yago-knowledge.org/resource/oregon_health_&amp;_science_university</yago> </disambiguated> </entity> alchemyapi recognizes that “oregon” and “oregon health & science university” are not the same thing and that “oregon health & science university” and “ohsu” are the same thing. however, simple identification of this entity does not make it relevant, and the additional information returned by the api is not useful even if all of it is correct. users must find navigation of linkages and external information intuitive for identification of these entities to be helpful. the challenge with providing links to external information is that they must be aligned with the user’s workflow to be useful. just as it is unhelpful to include numerous links in this article that break the reader’s flow of thought in the name of providing more information, it is not helpful to include links in metadata records which distract the user from having his or her information need satisfied. users have different needs, but interfaces must be optimized to meet use cases appropriate for the service being provided to be predictable and useful. including information simply because someone might conceivably have a use for it undermines the entire premise for organizing information. looking up terms in a knowledgebase frequently led to incorrect identifications. for example, when a document referenced a letter written by a person with a last name of “beck,” alchemyapi identified the entry as a potential composer, musical artist, award nominee, award winner, broadcast artist, celebrity, film music contributor, guitarist, lyricist, musical group member, recording engineer, record producer, songwriter, and tv actor. it identified the website for “beck” the musician and drew information from dbpedia, freebase, and musicbrainz. in fact, we found that linkages returned by services were outright detrimental because many people, places, organizations, and other entities have nonunique or ambiguous names. in general, we found the linked open data (lod) returned by various services was reliable for common entities but was almost always incorrect for lesser known ones. for example, there are over 100,000 people with the last name of “beck” in the united states [http://howmanyofme.com/]. lod sources containing names may know of the famous musician in addition to other well-known people named “beck.” however, they won’t know about obscure individuals with that name. many individuals referenced in archival documents have common names and are not famous, so the lod returned for them is inevitably erroneous. even when the information returned for common entities is correct (oregon, ohsu school of medicine, willamette valley, etc.), there is little need for linkages to such entities which are better researched in other environments. for lod to be useful in an archival context, the knowledgebase needs to reflect information in the archives being searched and to support archival research use cases. given that the lod sources cloud-based entity extractors utilize are optimized for general use cases such as parsing social media and news, they are best disabled for specialized needs such as archives. disabling disambiguation and lod functionality in our queries via parameters in the api calls dramatically reduced the amount of spurious information. for example, doing so caused the “beck” entity to be identified only as a person, which is both interesting and correct. alchemyapi detects hundreds of types of entities such as time expressions, shopping malls, body parts, comic book publishers, ski areas, quantities, radio programs, and concepts. by focusing on things we were specifically interested in such as people, organizations, drugs, and health conditions, we were able to reduce the information returned to a manageable level. removing the verbose xml display and unhelpful lod data while limiting to those specific domains is trivial and can be accomplished several ways including the following: ~# wget 'http://digitalcommons.ohsu.edu/cgi/viewcontent.cgi?article=1000&context=hca-oralhist' -o temp.pdf ~# pdftotext temp.pdf ~# alchemycmd --mode entity -s text -f temp.txt | \ > grep -e ‘,(anatomy|healthcondition|organization|person|drug),' this returns entities in relevance order in the form of: dr. rodney k. beals,person,0.923889 medical school,organization,0.460871 jim kronenberg,person,0.384462 professor emeritus rodney k. beals,person,0.371191 dr. dillehunt,person,0.297205 division of orthopedics,organization,0.290363 frenchy chuinard,person,0.289277 shriners,organization,0.28133 oregon medical association,organization,0.257537 ohsu,organization,0.251646 … [rest of output omitted] ... as observed earlier, people, places, and things frequently share names so the same text was sometimes identified as different types of entities in the same document. for example, in one finding aid, the phrase a. d. mackenzie papers was separately identified as a “facility” (one occurrence) and as an “organization”(two occurrences). elsewhere, in that same document, the phrase “alexander donald mackenzie papers” was identified as a person. in this particular case, alchemyapi correctly identified the presence of an entity three times, but was wrong all three times about what type of entity it was. likewise, societies were consistently identified as facilities rather than organizations. we found this strange because the name “society” appeared in them and there were no sources of linked data associated with these that would have fooled alchemyapi (e.g. oregon state medical society, yamhill county medical society, portland city and county medical society). however, entity type identification was reliable enough that we found it worthwhile to ignore entities that did not have the right type despite the fact that doing so causes a small but significant quantity of desirable entities to be ignored. at some point in the future, we intend to generate lists of entities that are inherently interesting and then use post processing to transform them into the more accurate form. in an archival context, alchemyapi can help staff identify significant terms and indicate their unique roles in documents and related archival materials. we found access points extracted from oral histories can help identify collections or materials that were previously unlinked. this increases the value of the oral histories as well as that of related archival materials across different records groups, administrative units and bodies — even if they are spread across institutions with divergent record creation practices, storage policies, and procedures. one practical use of alchemyapi is to compare the list of terms it returns with current collection indices, subject headings, creator names, and institutional functions found within other archival materials and then manually create links between the entity output and the archival resource — these linkages can be presented in the form of traditional see also, see and statements. for example, we start with the output from above: medical school,organization,0.460871 jim kronenberg,person,0.384462 professor emeritus rodney k. beals,person,0.371191 dr. dillehunt,person,0.297205 division of orthopedics,organization,0.290363 frenchy chuinard,person,0.289277 shriners,organization,0.28133 oregon medical association,organization,0.257537 ohsu,organization,0.251646 and then we use a process we call “uniquity triangulation” to establish relationships which commonly occur across our collections. “uniquity triangulation” could be considered a broad concept used to describe manually sifting through terms to gather and target specific sets of ohsu-specific terms that assist in elevating the document to a state of uniquity useful in aiding researchers with the decision of whether to dig deeper into any one document. this process could yield: jim kronenberg,person professor emeritus rodney k. beals,person dr. dillehunt,person, see also richard b. dillehunt photograph album (2004-001) division of orthopedics,organization frenchy chuinard,person shriners,organization oregon medical association,organization see also oregon medical association: oral history project (2004-005) by limiting output to desired information sources and entity types, reducing the frequency of common terms within the context of our collections (e.g. “ohsu” is not a unique term in ohsu oral histories, for example), and focusing on terms that appear with a high enough frequency to suggest relevance, our “uniquity triangulation” process combines automated extraction with staff judgment to identify useful access points. conclusions commercial cloud-based entity extractors appear to be optimized to analyze social media, news, and shorter documents to support general knowledge use cases. as such, we do not find them suitable for fully automated generation of semantic data for special collections descriptions even if we do think they are potentially a valuable tool that can help archival staff work more effectively. for all api’s that we tested, we found the following to be true: each api appears to be optimized for certain types of documents performance was significantly better on unstructured text than on structured documents such as encoded archival description (ead) finding aids so many entities were extracted that filters must be applied for the output to be manageable inherent ambiguities in the way concepts are expressed caused significant entity pollution, and this problem was amplified considerably when linked open data (lod) was enabled the general knowledge lod sources consulted by the api’s created associations that were too numerous and erroneous to be useful all of the services we tested are fast, and they extract potentially useful terms far quicker than a human manually scanning items. as such, they can greatly speed the process of cataloging materials such as oral histories, letters, articles, blogs, and theses. while the services we tested miss important entities and return spurious ones, they also quickly provide a list that a human operator can scan for purposes of selecting a few relevant tags. as such, the method drastically speeds up subject analysis and allows libraries to provide more complete metadata than they would otherwise be able to. output from cloud-based services can be combined using automated and manual means with information with other sources such as indices and authority lists to provide much richer linkages across a wide range of archival materials. our process can logically be extended to associate unique identifiers such as doi’s, orcid id’s, and create appropriate semantic linkages with entities occurring in a wide array of document types. references alchemyapi [internet]. [cited 2015 apr 7]. available from http://www.alchemyapi.com aylien [internet]. [cited 2015 apr 7]. available from http://aylien.com datatxt. [internet]. [cited 2015 apr 7]. available from https://dandelion.eu/products/datatxt howmanyofme.com. [internet] [cited 2015 jun 8]. available from http://howmanyofme.com meaningcloud’s topics extraction api. [internet]. [cited 2015 apr 7]. available from http://www.meaningcloud.com/products/topics-extraction opencalais. [internet]. [cited 2015 apr 7]. available from http://www.opencalais.com semantria. [internet]. [cited 2015 apr 7]. available from https://semantria.com textrazor. [internet]. [cited 2015 apr 7]. available from https://www.textrazor.com zemanta. [internet]. [cited 2015 apr 7]. available from http://www.zemanta.com about the authors kyle banerjee (banerjek@ohsu.edu) is the digital collections and metadata librarian at oregon health & science university. max johnson (johnsmax@ohsu.edu) is the university archivist at oregon health & science university. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – recommendations for the application of schema.org to aggregated cultural heritage metadata to increase relevance and visibility to search engines: the case of europeana mission editorial committee process and structure code4lib issue 36, 2017-04-20 recommendations for the application of schema.org to aggregated cultural heritage metadata to increase relevance and visibility to search engines: the case of europeana europeana provides access to more than 54 million cultural heritage objects through its portal europeana collections. it is crucial for europeana to be recognized by search engines as a trusted authoritative repository of cultural heritage objects. indeed, even though its portal is the main entry point, most europeana users come to it via search engines. europeana collections is fuelled by metadata describing cultural objects, represented in the europeana data model (edm). this paper presents the research and consequent recommendations for publishing europeana metadata using the schema.org vocabulary and best practices. schema.org html embedded metadata to be consumed by search engines to power rich services (such as google knowledge graph). schema.org is an open and widely adopted initiative (used by over 12 million domains) backed by google, bing, yahoo!, and yandex, for sharing metadata across the web it underpins the emergence of new web techniques, such as so called semantic seo. our research addressed the representation of the embedded metadata as part of the europeana html pages and sitemaps so that the re-use of this data can be optimized. the practical objective of our work is to produce a schema.org representation of europeana resources described in edm, being the richest as possible and tailored to europeana’s realities and user needs as well the search engines and their users. by richard wallis, antoine isaac, valentine charles, and hugo manguinhas introduction this article captures the core output of a project between the authors in which richard wallis, as an external consultant, was engaged to advise europeana on the potential for, and ramifications of, introducing detailed schema.org structured data into europeana websites. europeana provides access to more than 54 million cultural heritage objects through its portal europeana collections[1]. the portal is fueled by metadata describing digitized cultural objects from cultural institutions, which results from a long curation and aggregation process that leads to representing it into the europeana data model[2] (edm). it is therefore crucial for europeana to be recognized as a trusted repository of cultural heritage on the web and as an authoritative resource by search engines. indeed, even if the portal is considered as the main entry point for searching for cultural heritage, most of the europeana users come to it via search engines, especially google. the schema.org vocabulary[3] provides a way to embed metadata into web pages for direct consumption by search engines such as google to power rich user services (such as the google knowledge graph). schema.org is an open initiative, backed by google, bing, yahoo!, and yandex that has established a widely adopted practice (used by over 12 million domains) and vocabulary for sharing metadata across the web. its broad adoption underpins the emergence of new techniques, such as so-called semantic seo. europeana started to research the benefits of schema.org for its services, primarily the ability of the schema.org vocabulary to enable external organizations in general, and search engines in particular, to crawl and add that data into their knowledge graphs, thus enhancing the discoverability of cultural resources. our research lead us to investigate, review, and propose ways forward for the utilization of the schema.org vocabulary and associated technologies in europeana data services. we addressed first the representation of europeana resources using the schema.org vocabulary. this work resulted in a set of data mapping recommendations taking into account various concerns such as the representation of literals versus uris. we also addressed the publication of the embedded schema.org data as part of the europeana dynamic html pages and sitemaps using json-ld serialization. our recommendations detail the practical techniques that need to be considered to optimize the sharing of this data via the europeana websites and services. we propose several approaches to identify the effects of adding schema.org to the sites and produce analytics on fine-grained visiting patterns, which can be relevant for both europeana and its data providers. for these different areas, recommendations are made that correspond to the current situation as well as a more ideal situation where data for concepts, persons and other associated entities will be available as part of europeana data services. even though these recommendations have yet to be implemented in europeana, we think they can be useful to institutions interested in using schema.org. 1. data semantics and structure the practical objective of our work is to produce a schema.org representation of europeana resources described in edm, being as rich as possible and tailored to europeana’s realities and user needs as well as the search engines and their users. our mapping recommendations (see appendix 1) identify schema.org types and properties and the source edm term entities that could be used to derive data values, or suggest appropriate schema.org types. for example schema:name would be used for data derived from dc:title. instead of elaborating on each individual mapping decision, the remainder of this section discusses the more general challenges, design decisions and recommendations for extending the coverage of available data. at first view there are three major mapping challenges between edm, as used by europeana, and the more generic schema.org vocabulary. note: in the examples of this article the output format for schema.org data is assumed to be json-ld[4]. the reasoning for this choice is detailed in the json-ld output section in the json-ld output section below. challenge 1: flatter hierarchy the first challenge is to map the edm metadata structure to more generic (and flatter) schema.org classes. edm metadata is organized as a set of resources describing a cultural heritage object (i.e., edm:providedcho and ore:proxy that represent different data sources for objects), one or more digital representations (i.e, edm:webresource) and “contextual” resources (places, persons, concepts, timespans), in compliance with the one-to-one principle[5]. these resources are grouped together by an edm:aggregation into one bundle recording also some information on the aggregation process (e.g., the providers of the metadata). this package of data contains many internal relationships linking resources to each other. the schema.org approach is to define a schema:creativework (equivalent to the cho) plus links to access the resource. the inevitable result of mapping to schema.org will be a flatter representation of the data. in both cases, however, there are still relationships with associated contextual entities (in the edm sense, schema:person, schema:organization, schema:place, etc. are contextual entities). most of the ‘flattening effect’ would be apparent for proxies and aggregations. figure 1. the four main classes of the edm metadata structure and their relations. challenge 2: identifying the type of thing the core entity in edm is the edm:providedcho, broadly equivalent to the schema.org class schema:creativework, with the type of object being defined via properties such as dc:type. however, using schema.org it is preferable wherever possible to identify the specific type of ‘thing’ (schema:thing being the base class in schema.org which all other inherit from) being described – schema:book, schema:painting, schema:sculpture, schema:imageobject, etc. often this data is available within edm, via properties such as dc:type and references to concepts from, e.g., the getty art and architecture thesaurus[6]. web resources represented as schema:mediaobjects also ideally should be identified by more specific subtypes of schema:mediaobject (schema:imageobject, schema:audioobject, schema:videoobject). again this will require mapping between mimetypes, file extensions, etc. to ascertain the correct type. these mapping rules, the basis for which are referenced in the data mapping recommendations appendix below, will need to be evolved and established over time. challenge 3: (identified) things vs strings record vs. entity view edm is a step towards a semantic approach to data modeling, but the europeana data currently available provides a record oriented view. references to entities, and their data, are included in a single output, and not all entities (organizations, persons, concepts etc.) have separate uri identifiers. ideally for schema.org, output should be in an entity based form, where each entity is described as such with its own uri, and references to other entities (creator of an object, for example) provided only as a uri. this approach is guided by linked data principles.[7] satisfying this need imposes both a data and a technical challenge. firstly, the data for each entity should be available in a form resembling an rdf concise bounded description (cbd)[8], providing a graph based view of the data for an individual entity and its relationship(s) with other entities.in simple terms this restricts the properties output to those directly associated with the entity being described (formed from the rdf triples having the entity’s uri as their subject). edm object records, currently served by europeana’s api which seeks to reflect the type of records usually produced and consumed in the cultural sector[9], are a good base. however further refinement may be needed focusing on separate description of individual entities (person, place, creative work, etc) as against aggregating them into a single record.edm proxies, which are essentially “derived” resources, should receive specific attention while assembling the data to be published. secondly, the systems need to be able to resolve and display each entity based upon its uri, supplying the embedded schema.org data for it. it is recognized that moving to fully entity based data output will be an evolutionary process for europeana (especially in terms of web identifiers and web pages associated with resources). we recommend in the interim that standard rdf blank node techniques (cf. point on blank nodes below) be used to separate out the notional entities within the combined data payload. expanding string literals into entities wherever possible literals should be expanded to an entity description of the appropriate type. for example the following in json-ld: "creator": "j .m. smith" should be expanded to: "creator": { "@type": "person" "name": "j. m. smith" } this will explicitly inform those consuming the data of related entity types. (see note below about blank nodes for preferred json-ld syntax) entity data, blank nodes, uris in json-ld output (as in some other rdf syntaxes) there are three formatting options in a combined payload for describing entities, such as creator in the examples below, related to a resource. implicit blank nodes (nested output) – this is where the data for the associated entity is provided inline as part of the resource description data: { "@id": " http://data.europeana.eu/item/2021618/internetserver_details_kunst_25027", "@type": ["creativework", "visualartwork"], "name": "mona lisa | leonardo da vinci", "creator": { "@type": "person", "name": "luigi calamatta" } } explicit blank nodes – a blank node reference to the associated entity is provided in the resource description with the entity description provided in the same data payload thus: "@graph":[ { "@id": "http://data.europeana.eu/item/2021618/internetserver_details_kunst_25027", "@type": ["creativework","visualartwork"], "name": "mona lisa | leonardo da vinci", "creator": "_:p0", }, { "@id": "_:p0", "@type": "person", "name": "luigi calamatta" } ] entity reference – a resolvable uri is provided for the associated entity. it is for a consuming application or service to access that uri to obtain the entity description: { "@id": " http://data.europeana.eu/item/2021618/internetserver_details_kunst_25027", "@type": ["creativework","visualartwork"], "name": "mona lisa | leonardo da vinci", "creator": "http://data.europeana.eu/item/p12345", } the recommended ideal for this is option 3 – entity reference, but is achievable only when entity descriptions with resolvable uris have been created. europeana is currently working on a “semantic entity collection” that acts as a centralised point of reference and access to data about contextual entities. this entity collection will provide resolvable uris (at http://data.europeana.eu) for all the entities described in europeana. in the meantime pption 2 – explicit blank nodes – is used. this will facilitate easier adoption for option 3 at a later date. “duplicating” statement objects both as uris and labels when separate entity uris are provided as values for properties such as dc:creator, the question arises as to whether the label associated with these entities should be represented next to the uri of the separate entity. this would provide “more readable” data, also accomodating machine clients who couldn’t ‘follow their nose’ from the creator uri to the creator name in the separate (linked data) description. as in the following: { "@id": "http://data.europeana.eu/item/2021618/internetserver_details_kunst_25027", "@type": ["creativework", "visualartwork"], "name": "mona lisa | leonardo da vinci", "creator": ["http://data.europeana.eu/item/p12345", "luigi calamatta"] } this approach is not recommended as it can introduce maintenance problems, as changes to the (remotely accessible) linked data available for the entity being referenced may invalidate the name/label that has been reproduced locally. mapping to external resources it is preferable wherever possible to map internal europeana entities, and concept identifiers, to external equivalents. identifying an object, organization, person, concept, etc. to be schema:sameas a wikidata or similar identifier. to a certain extent the ability to identify such references is constrained by the amount and quality of data contributed to europeana by its partners. however, especially when processing (enriching) data to identify entities for persons, organizations, etc. this need should be given high consideration. 1.4 uri design decisions choice of identifying uris the identifying uri for a europeana resource should be of the form: http://data.europeana.eu/.......... this differs from the current url form for web pages: http://www.europeana.eu/portal/.......... this approach separates the provision of canonical europeana identifiers from the structure of current and future web site structure and implementation, which allows unambiguous knowledge to be published about each of them. note that requests for human-readable html representations of a resource with a data uri are already redirected by europeana services to the appropriately addressed www addressed page. data references between web page and resource in line with the above decision, it is recommended that consideration be given to outputting a representation of the relationship between a resource and the web page displaying it. for example: "@graph":[ { "@id": "http://www.europeana.eu/portal/record/2021618/internetserver_details_kunst_25027.html", "@type": "webpage", "mainentity": "http://data.europeana.eu/item/2021618/internetserver_details_kunst_25027" }, { "@id": "http://data.europeana.eu/item/2021618/internetserver_details_kunst_25027", "@type": ["creativework", "visualartwork"], "name": "mona lisa | leonardo da vinci", "creator": ["http://data.europeana.eu/item/p12345", "luigi calamatta"] "mainentityofpage": "http://www.europeana.eu/portal/record/2021618/internetserver_details_kunst_25027.html" } ] note: in this example the property schema:mainentityofpage is used to reference the page on which the resource is displayed. this is optional as it may be difficult to maintain, especially when a resource is displayed on more than one page, in themed portals etc. 1.5 data coverage recommendations annotations europeana represents annotations (such as users’ tags, links to related objects) with an extension of edm based on the recommendations from w3c web annotation model[10]. when processing entities for mapping to schema.org information in these annotations may provide valuable information to be shared using schema.org properties. for example annotations with a ‘motivation’ of oa:commenting could be mapped to schema:comment; oa:tagging to schema:about or possibly schema:keywords; oa:linking to schema:sameas or schema:exampleofwork, etc. it was beyond the scope of the project to analyze the current, and potential future, coverage of annotations for comments and provenance etc. in europeana data. it was recommended however that this coverage should be taken into account when building the mapping rules and processes. this recommendation carries the caveat that the processing required to do this may impact performance for on-the-fly data creation (as described in the following technology section of this report) and therefore may have to be deferred until there are batch mode capabilities available. description of europeana and providers as organizations it is important in placing europeana resources into context on the web that they are related to well schema.org-described descriptions of associated entities such as for europeana as an organization and its data providing partners as organizations. to this end, it is recommended to embed a rich schema.org description of europeana as an organization in the http://europeana.eu page. equivalent efforts should also be made for pages about each of the providing partners, when possible, and appropriate references to uris of europeana and providers should be made in the (object) data. note: in the longer term, to boost the ‘schema.org reputation’ of the europeana organization resource, snippets of mark-up “powered by europeana” should have schema.org mark-up in them, referring to http://europeana.eu. 3. technology the purpose of defining europeana resources using the schema.org vocabulary is to enable external organizations in general, and search engines in particular, to consume the data into their knowledge graphs of resources on the web. search engines consume such data from sites on the web as part of their standard web crawl processes. they parse the totality of the contents of crawled html pages identifying structured data, in schema.org form, marked-up using either microdata, rdfa, or json-ld syntax[11]. the technical requirements are to provide this schema.org data embedded within the html pages of europeana websites without detrimentally impacting the primary purpose of those pages in supporting human interaction. 3.1 separation of interface concerns over time, it is highly likely that the user interface design requirements of europeana websites will need to change independently of the underlying data structures. this could be because of aesthetic changes or the introduction of themed views. equally as the schema.org vocabulary and associated industry practices evolve, and the modeling and quality of data stored by europeana evolves, there will be need to change the structured data embedded in a page without changing the visual representation. a standard, and the recommended, approach to solve the separation of these needs is to ‘bolt-on’ the structured data to the page construction. this technique, as implemented by oclc on worldcat.org, involves inserting a section in the page source code, containing the structured data, that does not impact on its visual output. 3.2 json-ld output if schema.org output is to be ‘bolted-on’ to page source, the recommended approach is to use json-ld format inserted into a html script tag: <script type="application/ld+json"> { "@context": "http://schema.org", "@graph":[ { "@id": "http://data.europeana.eu/item/2021618/internetserver_details_kunst_25027", "@type": ["creativework", "visualartwork"], "name": "mona lisa | leonardo da vinci", } ] } </script> this format provides the smallest payload, compared with rdfa and microdata; json-ld is supported by many tools, and is already used in some europeana api services. 3.3 json-ld usage significant consumers of schema.org, especially those consuming json-ld do not share which tools they are using to parse the data. therefore the implementation of data format within the json-ld should be as simple as possible and follow established guidelines (see the section on multilingual strings in appendix 1). 3.4 dynamic loading of json-ld one technically attractive option for embedding json-ld is to enable the dynamic loading of the json-ld contents into the page. this is achieved by calling a script, potentially hosted on a different server to that serving the web page. this script either will contain the json-ld code or will use javascript code to insert it into the of the document. for example: <script src="http://api.europeana.eu/schemainsert?id=2011618" type="application/ld+json"> it has been documented[12] that google indexing tools recognize such data. this approach is therefore recommended. some experimentation would be required to ascertain the most appropriate implementation technique for europeana. performance the introduction of script tags and possible calls to external services could potentially negatively impact the in-browser performance of pages. experimentation with the placement of script tags (probably near the end of the page) and the use of asynchronous calls should be undertaken to optimise display performance. supporting api to support the dynamic loading of json-ld, it is recommended that a simple api be implemented on europeana systems, as a stand-alone tool or as an extension of the current europeana apis. it will accept the uri identifier of a resource as a parameter and will return the code to embed json-ld into the page it is called from. the call to this api will be added to a <script> tag in each page, and the script will be called by browsers displaying the page. the output format of this api should be generic in nature, not requiring specific html formats within the page. a recommended common technique is for data, being dynamically inserted into the page dom, to be placed in the element. cors support it is in the nature of the api and javascript oriented proposals above that under some circumstances default browser security constraints may block calls to servers from which the page being displayed did not originate. to prevent this cors[13] support should be added to the http headers output by these services. 3.5 generation of schema.org data there are various options for creating schema.org data, as described previously, from source edm data. on-the-fly potentially the simplest option is to create schema.org as and when requested, the source data being read as edm from storage and then being passed through a mapping/conversion process. such a process would be invoked each time a resource is displayed. the advantage of such an approach is that no extra data is stored to support schema.org; also changes to mapping rules are instantly available. disadvantages include system loading, and difficulty in supporting complex dependencies in data mapping. batch creation an alternative is that the resource data is batch processed, to produce additional schema.org data that is then stored alongside the edm data. this has the advantage of not needing processing to extract data for display. however there are disadvantages around coping with mapping changes and re-indexing of databases. data creation recommendation a combination of on-the-fly and batch creation is used over time. initial implementation should use the on-the-fly approach providing a lightweight flexible approach. standard web caching techniques may be applied to limit loading requirements of such a service. this would require some understanding of data changes to ensure freshness of cached data. however future plans for europeana systems should take into account potential benefits of evolving the current data store into a europeana knowledge graph in which batch processes following complex relationships within the data could infer and add value to the overall data. 3.5 sitemap a major end objective is to get search engines to crawl and consume data from the pages describing europeana resources. to help them in that task, sitemaps[14] need to be provided and well maintained for all pages that contain schema.org data on the europeana websites. it is important that these sitemaps are regularly updated to indicate new and updated pages. this needs to take into account pages that visually may not have changed, but have data output that has changed. note: as seen earlier, the uri for an object (http://data.europeana.eu/...) will differ from the url of the page(s) that display information about that object (http://www.europeana.eu/portal/...). this introduces the potential for a data change to result in more than one url being identified in the sitemap as being updated. as the data.europeana.eu uris redirect to a default page for an entity, it may be sufficient to use those in the sitemap as a surrogate for the actual page. it is recommended that search engine webmaster tools are used to monitor and analyze crawler coverage over time. search engine crawlers tend to be most efficient when they can use sitemaps to gain a view of the real state of a site. not indicating changed pages, or wrongly indicating that pages have changed, can result in a site not being fully crawled and data not being consumed. nb: in accordance with the above recommendation “description of europeana and providers as organizations”, the sitemap should also include reference to http://europeana.eu and provider pages that europeana could publish in the future. 4. analytics the purpose of analytics in the context of schema.org is to identify changes in the numbers and or visiting patterns of users to europeana websites. by definition, to identify changes comparisons need to be made with current patterns. future pattern traffic predictions include assumptions that there may be an increase of visits directly to individual detail pages. the detailed schema.org structured data embedded in the pages enabling search engines to more accurately target them as being representative of the resource they describe. it could be expected that this increase in traffic to individual detail pages would be balanced by a decrease in visits to general pages, such as the home and search pages of a site. it is recommended that current traffic analysis be tuned to identify visits to such pages and the routes taken by users to reach these pages. then comparison made to identify trends as schema.org markup is introduced to the site. such, more detailed, analysis will help in analyzing increases or decreases in gross site traffic to identify the success, from a user point of view, of visits. one potential scenario that this could help diagnose would be an overall drop in traffic yet an increase in successful visits. with the recommended introduction of json-ld data using html <script> tags, there is an opportunity to enable the added schema.org terms to be used by google tag manager[15]. we recommend the use of schema.org terms to provide tag manager enabled enhanced analytics in the future. this could facilitate detailed analytics based upon the type, subject, and/or format of resources, for example.[16] it is not recommended that this step is undertaken immediately. however it is recommended that a simple step is taken to enable its future use. introducing an id property to the <script> tag is enough to enable its contents to be identified by tag manager. <script id="schema" type ="application/ld+json"> 5. summary in brief this project has identified that it is possible to represent europeana data resources using the schema.org vocabulary. the following points will need to be addressed as part of the project to implement schema.org output: mapping recommendations – examples of which are in appendix 1 below – form the basis to establish processing required to produce schema.org output for europeana.eu websites. there is a mismatch between the edm providedcho core entity, with object types being defined using properties such as dc:type and the schema.org model of defining schema:creativework subtypes such as schema:book, schema:painting, schema:visualartwork, etc. this will require some processing to identify types as part of the mapping. web resources represented as schema:mediaobjects also ideally need to be identified by schema.org types (schema:imageobject, schema:audioobject, schema:videoobject). again this will require mapping between mimetypes, file extensions, etc., to ascertain the correct type. the recommended mappings and processing should be refined over time as experience is gained, europeana entity definitions for context entities (concepts, places, etc.) are introduced, and vocabularies evolve. output should be via json-ld embedded in html <script> tags in the resource detail pages. dynamically loading json-ld, facilitated by a new europeana api should be considered. initial mapping processing should be carried out on-the-fly as pages are visited. over time, as the data evolves, other entities are introduced, and mapping becomes more complex to add value and quality to the data, batch processing and subsequent storage of terms should be considered. current analytics should be tuned to identify traffic to resource detail pages and the paths taken. over time, after the introduction of schema.org output, comparison of traffic patterns to detail pages should be compared with current patterns. in preparation for possible enhanced analytics using google tag manager, an id property should be added to the json-ld <script> tag. about the authors richard wallis (richard.wallis@dataliberate.com) is an independent consultant, with a long history in library and semantic web technology, at the forefront of promoting, explaining, and applying new and emerging web, semantic web, and linked data technologies. he works with google and others in the support, extension and application of the schema.org vocabulary. also chairing and participating in several w3c community groups focused on the development of the vocabulary in areas such as bibliographic & cultural heritage resources, tourism, and finance. this background informing his work with many clients helping them introduce schema.org into their web presence and supporting infrastructure. antoine isaac (antoine.isaac@europeana.eu) is r&d manager at europeana. he works and coordinates work on various aspects of the interoperability and quality of cultural collections on the web, focusing on linked data and semantic web technologies. he has served various w3c efforts, like skos, library linked data, and data on the web best practices. he is also a guest researcher at the web & media group in the free university amsterdam. valentine charles (valentine.charles@europeana.eu) is data r&d coordinator at europeana. she consults the europeana network on data practices and standards and communicates europeana’s r&d activities to the broader cultural heritage sector. she also coordinates the further development and adoption of the europeana data model (edm). valentine is also co-chair of the dublin core metadata initiative (dcmi) technical board and community specification committee. hugo manguinhas (hugo.manguinhas@europeana.eu) has recently taken on the role of product owner for apis at europeana, but was working before as the technical coordinator for research & development for the same institution. he advised and coordinated the scientific work on europeana-related projects and participated in the development of europeana’s linked open data, metadata enrichment, vocabulary services and alignment, data quality and crowdsourcing infrastructure. notes [1] http://europeana.eu/ [2] http://pro.europeana.eu/edm-documentation [3] http://schema.org [4] http://json-ld.org/ [5] http://wiki.dublincore.org/index.php/glossary/one-to-one_principle [6] http://www.getty.edu/research/tools/vocabularies/lod/ [7] https://www.w3.org/designissues/linkeddata.html [8] https://www.w3.org/submission/cbd/ [9] http://labs.europeana.eu/api/record-jsonld [10] https://www.w3.org/tr/annotation-model/ [11] https://schema.org/docs/gs.html [12] http://www.centrical.com/test/google-json-ld-and-javascript-crawling-and-indexing-test.html [13] cross-origin resource sharing (cors) http:/enable-cors.org/, the server needs to add the following http header to its response: access-control-allow-origin: * [14] https://en.wikipedia.org/wiki/site_map [15] https://www.google.com/analytics/tag-manager/ [16] so that we can compare e.g. traffic towards books with traffic towards paintings. this could be useful to measure the efficiency of the mapping. especially for advanced mappings, like the ones to specializations of creativework, or the ones with links to specific collections, like pinboards. [17] http://www.europeana.eu/portal/record/2021618/internetserver_details_kunst_25027.html [18] http://www.europeana.eu/portal/record/09102/_uedin_214.html [19] http://tools.ietf.org/html/bcp47 [20] https://en.wikipedia.org/wiki/iso_8601 [21] http://schema.org/docs/gs.html#schemaorg_expected [22] http://schema.org/docs/gs.html#advanced_enum [23] https://github.com/schemaorg/schemaorg/issues/894 appendix 1 – data mapping recommendations two approaches were taken in recommending mapping from current edm based resources to schema.org. the attached pdf documents the recommendations thus far. firstly, resources are described directly in schema.org drawing data from available edm, but not being constrained by those structures. two example files, appended to this report, (mona-lisa.jsonld & buccin.jsonld) are supplied as examples. source for these examples were drawn from their associated web pages: mona lisa[17] & buccin[18] and their debug=json outputs. inspecting the mona-lisa.jsonld example: "@graph":[ { "@id": "http://data.europeana.eu/item/2021618/internetserver_details_kunst_25027", "@type": ["creativework", "visualartwork"], "name": "mona lisa | leonardo da vinci", "creator": ["_p:0", "_p:1"], "about": ["mona lisa", "_:p1"], "artform": "grafiek", ... } ] note: the examples and recommendations supplied here can only provide guidance towards the process of building a mapping structure for terms to be described using schema.org. it is inevitable with the significant range of resource types that some anomalies will occur when following a basic recommendation. these will need to be addressed by experience over time. a1.1 creativework properties @id this indicates the uri identifier of the resource itself as to be shared with the web, not a proxy or other internal identifier. this uri should resolve to a description of the resource (which is already the case for data.europeana.eu uris). @type it is assumed that all europeana providedcho resources can be described as creativeworks. it is normal practice to provide multiple schema types when appropriate. wherever possible a more accurate type should be provided. this introduces challenges in the mapping process as some of the information required to calculate the resource type are currently held in properties such as dc:type and ebucore:hasmimetype. some physical resources (such as the trombone in the buccin example) can be defined as type schema product, which introduces properties such as manufacturer and itemcondition to further aid description. name the name/title of the resource. (see notes below about multilingual strings) description the description of the resource. (see notes below about multilingual strings) creator in this example "creator": ["_p:0","_p:1"], is indicating blank-node identifiers for two creators linking to person entity descriptions later in the output. eventually as europeana entity descriptions become available, these should be replaced by fully resolvable uris. see also discussion on person/organization. about this often repeated property contains text and or uris for related entities that the resource is related with. in this example it includes a reference to the name of the artwork that this represents, “mona lisa”, and the identifier of the original creator _:p1 (leonardo da vinci). equally it could include references to places, types of object (e.g., in dc:type and edm:hastype), and other concepts. ideally, types references for schema:about would be filtered, avoiding especially the types that lead to assigning a specific @type value. rules that do not select book, painting, etc., could be applied. also types that are local would ideally be related to reference types, such as the wikidata ones (for example a trombone would be: thing sameas https://www.wikidata.org/wiki/q8350) artmedium/artform/artworksurface these are properties of schema:visualartwork indicating the physical type of artwork such as sculpture, painting, drawing, etc. in this example, "grafiek" was drawn from dc:type. note: these properties are only available for a visualartwork type. if used, the resource type should be defined as schema:visualartwork in addition to schema:creativework etc. ideally the uri of an authoritative term, either europeana supplied or external (such as getty aat, wikidata, etc.) should be used here if available. note: given the high variability in the corresponding edm fields, recognizing appropriate values for mapping to these properties will be quite difficult in an initial stage. height/width/depth if this information is available it should be provided using a schema:quantitativevalue description to indicate the value and (un/cefact) unitcode. for example: "height": { "@type": "quantitativevalue", "value": "768", "unitcode": "mmt" }, note: this is probably more for webresources as values for these properties is expected to be hard to get from the dcterms:extent of the providedcho, which often includes all this information in one (language-specific) text value. license this should be the uri of a license description that applies to the resource, preferably, as in the example, an authoritative uri. in accordance to europeana’s focus on standardized rights statements, schema:license should be used only with edm:rights. other, less constrained rights-related properties like dc:rights, of the providedcho being described, could be mapped to another description string. note: for a creativework, or a subtype of it, license information may be derived from a providedcho, an aggregation, or webresource. potentially there could be differences between the [dc: or edm:] rights statements on an individual, or one of the related, resources that are being used to populate the creativework. also some dc:rights statements may not be informative. in these cases it is recommended to identify at least the creative works that are in the public domain. in this case we could have a rule saying “map it from edm:rights when the value is public domain.” thumbnailurl url of an openly viewable preview image. provider name or, preferably, uri of providing organization(s) or person(s). encoding/associatedmedia reference to the mediaobject representation of the creativework. schema:encoding and schema:associatedmedia are synonyms for each other. it is recommended that associatedmedia is used as it has greater usage. potentialaction this provides a description of ways that a user could potentially interact with the resource. for europeana resources it is expected that this would be an instance of listenaction, readaction, viewaction, or watchaction (depending on the value of edm:type). viewaction should be the default if the detailed required action type cannot be ascertained from edm:type, for example. in the example, the viewaction has been given a name “view at teylers museum.” there is an expectation that a search engine may use this to annotate a link they would offer to a user. this string would have to be created as part of a mapping process. note: the target value of an action, in europeana’s case, should be the europeana proxy url for a partner’s web page. this will enable the tracking of traffic. however, potentially this could be seen by the search engine as internal linking, decreasing its importance in their indexing. only traffic analysis over time will make these effects clear. if the information is available, a schema:offer could be provided as a schema:expectsacceptanceof value, describing costs/requirements needed to enable the action to be carried out. a1.2 media the buccin.jsonld example identifies some of the issues in relating media representations to resource descriptions. { "provider": ["_:o0", "_:or1", "university of edinburgh"], "audio": "_:mo0", "image": "_:mo1", "video": "_:mo2", "license": "http://creativecommons.org/licenses/by-nc-sa/3.0/" }, { "@id": "_mo0", "@type": "audioobject", "name": "buccin trombone. nominal pitch: b?", "contenturl": "http://www.mimo-db.eu/media/uedin/audio/0032195s.mp3", "encodescreativework": "http://www.europeana.eu/portal/record/09102/_uedin_214", "duration": "30334", "bitrate": "128000", "encodingformat": "mp3", "contentsize": "485349", ... } audio/image/video the basic approach is to provide the access uri for the media files, as properties of the resource description. however europeana contains rich information about these media representations. it is recommended that these should be described using schema:mediaobject or appropriate subtypes. note that using specialized properties (schema:audio, schema:image, schema:video) is only possible when a specific subtype of mediaobject has been recognized. mediaobject/audioobject/imageobject/videoobject mapped by analyzing mimetype, file suffix, dc:format and or possibly dc:type, the appropriate type should be chosen. if exact type is not ascertainable mediaobject should be used. encodescreativework relates back to the resource description. duration/bitrate/encodingformat/contentsize/height/width as much detailed technical information as possible should be included. name/provider/description although potentially a repeat of information in the main resource description this also needs to be included within the mediaobject description. license license for media can differ from that of the providedcho so should be explicitly supplied for a media object. a1.3 person/organization in all cases that reference persons or organizations – creator, provider, etc. – need to be defined as a schema:person or schema:organization. there is no schema.org equivalent of edm:agent. the specific type should be known when the resource derives from prior entification done by europeana. if the type is unknown, they should be defined as a schema:thing. as future work the following hints may be also used: (i) use of personor organization-specific attributes on the resource to be typed (ii) use of the resource to be typed as object in a statement where the property is expected to be used with persons or organizations in the europeana context. a1.4 sameas wherever possible schema:sameas references should be provided between europeana resources & entities, and external equivalents. this is particularly relevant for concept, person, organization, and place entities. note that schema:sameas has broader scope than owl:sameas: it can cover skos:exactmatch or even skos:closematch (up to the data publisher’s preference): "@id": "_:p1", "@type": "person", "name": "leonardo da vinci", "sameas": "http://dbpedia.org/resource/leonardo_da_vinci" a1.5 multilingual strings multilingual values for names, etc., should be provided as a language tagged array of strings as per the example: note: do not mix language tagged strings with normal strings in a single array. a1.6 text values and data types – bcp 47 / iso6801 etc. schema.org recommends bcp 47[19] for language definitions; iso 8601[20] for date / time references; and expected types for schema properties. ideally these requirements should be adhered to wherever possible. however under the spirit of any data is better than no data and schema.org guidance[21], it is acceptable to include the representation available from edm source data. this includes dates such as “circa. 1850”, and text only representations for a resource where an entity representation can not be supplied. the same applies for similar recommendations for data types for properties in schema.org. it’s up to the data consumer to do something with these recommendations. not implementing them will mean missing opportunities, but it should not be too detrimental to item discovery. a1.7 concept entities within the schema.org vocabulary there is type for ‘concept’. in principle a set of concepts, or authoritative values (subject headings) would be represented as enumerations[22] in schema.org. these are reserved in the vocabulary for simple small sets of data such as for schema:bookformattype. it is recommended that when defining a concept entity it is given a schema.org type of thing. when defining concepts it is important to provide sameas relationships to external authoritative sources for the same concept – getty, wikidata, lcsh, etc. note: there is currently a proposal[23] to the schema.org community to enable the description of concepts, described as a schema:categorycode. if that proposal is accepted europeana should adopt that markup form. appendix 2: schema.org json-ld example : mona lisa appendix 3: schema.org json-ld example: buccin <script type="application/ld+json"> { "@context": "http://schema.org", "@graph":[ { "@id": "http://data.europeana.eu/item/09102/_uedin_214", "@type": ["creativework","product"], "name": "buccin trombone. nominal pitch: b?", "description": "technical description: brass; ligature fitting on bell section at joint; stockings on main slides. bell with one coil, angled to face forwards. repair history: main slide possibly not original (tenon of slide section of joint is tapered, bell section joint for cylindrical tenon).", "datecreated": "circa 1840", "about": [ "http://www.mimo-db.eu/hornbostelandsachs/356", "http://www.mimo-db.eu/instrumentskeywords/4378", "http://www.mimo-db.eu/hornbostelandsachs/245", "http://www.mimo-db.eu/hornbostelandsachs/333" ], "about": [ { "@language": "de", "@value": "buccin" }, { "@language": "sv", "@value": "buccin" }, { "@language": "en", "@value": "buccin" }, { "@language": "it", "@value": "buccin-trombone" }, { "@language": "fr", "@value": "buccin" }, { "@language": "nl", "@value": "buccin" }, ], "potentialaction": { "@type": "viewaction", "name": "view at mimo musical instrument museums online", "target": "http://www.europeana.eu/api/rbwpaapq6/redirect?shownat=http%3a%2f%2fwww.mimo-db.eu%2fuedin%2f214&provider=mimo+-+musical+instrument+museums+online&id=http%3a%2f%2fwww.europeana.eu%2fresolve%2frecord%2f09102%2f_uedin_214&profile=full" }, "provider": ["_:o0","_:or1","university of edinburgh"], "audio": "_:mo0", "image": "_:mo1", "video": "_:mo2", "license": "http://creativecommons.org/licenses/by-nc-sa/3.0/" }, { "@id": "_mo0", "@type": "audioobject", "name": "buccin trombone. nominal pitch: b?", "contenturl": "http://www.mimo-db.eu/media/uedin/audio/0032195s.mp3", "about": [ "http://www.mimo-db.eu/hornbostelandsachs/356", "http://www.mimo-db.eu/instrumentskeywords/4378", "http://www.mimo-db.eu/hornbostelandsachs/245", "http://www.mimo-db.eu/hornbostelandsachs/333" ], "encodescreativework": "http://www.europeana.eu/portal/record/09102/_uedin_214", "duration": "30334", "bitrate": "128000", "encodingformat": "mp3", "contentsize": "485349", "provider": "_:or1", "license": "http://creativecommons.org/licenses/by-nc-sa/3.0/" }, { "@id": "_:mo2", "@type": "videoobject", "name": "buccin trombone. nominal pitch: b?", "contenturl": "http://www.mimo-db.eu/media/uedin/video/0032195v.mpg", "about": [ "http://www.mimo-db.eu/hornbostelandsachs/356", "http://www.mimo-db.eu/instrumentskeywords/4378", "http://www.mimo-db.eu/hornbostelandsachs/245", "http://www.mimo-db.eu/hornbostelandsachs/333" ], "encodescreativework": "http://www.europeana.eu/portal/record/09102/_uedin_214", "duration": "40411", "bitrate": "128000", "height": "576", "width": "720", "encodingformat": "mpg", "contentsize": "19480576", "provider": "_:or1", "license": "http://creativecommons.org/licenses/by-nc-sa/3.0/" }, { "@id": "_:mo1", "@type": "imageobject", "name": "buccin trombone. nominal pitch: b?", "contenturl": "http://www.mimo-db.eu/media/uedin/image/0032195c.jpg", "about": [ "http://www.mimo-db.eu/hornbostelandsachs/356", "http://www.mimo-db.eu/instrumentskeywords/4378", "http://www.mimo-db.eu/hornbostelandsachs/245", "http://www.mimo-db.eu/hornbostelandsachs/333" ], "encodescreativework": "http://www.europeana.eu/portal/record/09102/_uedin_214", "encodingformat": "jpg", "provider": "_:or1", "license": "http://creativecommons.org/licenses/by-nc-sa/3.0/" }, { "@id": "_:or0", "@type": "localbusiness", "name": "europeana", "url": "http://europeana.eu" }, { "@id": "_:or1", "@type": "localbusiness", "name": "mimo musical instrument museums online", "url": "http://www.mimo-db.eu" }, { "@id": "_:or2", "@type": "localbusiness", "name": "university of edinburgh" } ] } subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – using an agile-based approach to develop a library mobile website mission editorial committee process and structure code4lib issue 12, 2010-12-21 using an agile-based approach to develop a library mobile website this article discusses how the uc san diego libraries developed and implemented a mobile website by giving a small collaborative group decision-making authority for all of the library stakeholders. the group used rapid development and testing cycles with an understanding that delivering a fast and “good enough” service was preferable over slow and seemingly perfect development. by matt critchlow, lia friedman, and dan suchy introduction the uc san diego campus (ucsd) is comprised of six colleges and a student body of nearly thirty thousand.  within the nine campus libraries there are over 300 staff across dozens of departments. as is probably common with large, distributed academic units, projects in this environment traditionally have a long period of development and testing prior to the release of anything functional, including a well-defined system of gaining feedback and direction from library management and administration. this traditional approach, similar to the waterfall model [1], looks to have ideas complete, approved, and set in stone before moving forward.  in creating a new mobile site we looked at employing a different type of model: responsive, ‘good enough”, collaborative and fast-paced and with an eye towards experimentation. the mobile site was a pilot project to help examine the advantages of employing this model in future projects. a fine example of a traditional, or waterfall, development project is the website redesign recently completed by ucsd libraries in order to keep in line with the campus-wide website look and feel. this project involved forming several committees with representatives from each of the nine libraries as well as various other stakeholder groups (interlibrary loan, reserves, it, administration, etc). the entire process spanned several months, and near the end of the project additional groups were formed to provide feedback and discuss ongoing issues. just one of these ancillary groups was made up of the web managing editor, the libraries’ director of communications, a public services member-at-large, a  representative from bibliographers’ council, an instruction representative from the instruction & outreach committee, an outreach representative from the instruction & outreach committee, a representative from the reference & information services committee and a representative from the document delivery service /inter-library loan committee. while the outcome and public response to this project was positive, a project of this scope and scale required a protracted timeline. in contrast, the mobile advisory group for the mobile site project consisted of just 4 representatives from selected user services groups, a facilitator that straddled both user services and it, and the web technical manager. working within a two month timeframe, this pilot group was empowered to address the broad needs of patrons with mobile devices and to provide input from a user services perspective. we employed three two-week cycles, each with one week of site development followed by a week of feedback and testing, allowing us to move quickly on to a new aspect of the site as the process was repeated each cycle. while understanding that this model will not work for all projects, the receptiveness and collaborative nature of the participants, the skill of those working on the technical development, and the freedom to work largely without having to report back to departments for input have made this an approach we look to employ in the future for similar projects, particularly when timeliness and technology hype cycles are likely to impact project implementation. project approach our development method was based in part on the tenets of agile software development.  much has been written concerning the agile methodology [2], but for our purposes we particularly embraced the notion of short, iterative development cycles with an emphasis on in-person communication between library web developers and user services representatives. our project included the following structure and assumptions: a short, well-structured timeline successfully managing projects of this nature requires a commitment to a short development timeline with well-defined project milestones.  this is especially the case when designing a new library service from scratch (like a mobile website), where is it fairly easy to let unforeseen details sidetrack the entire project. in our example, we began with an initial ‘kickoff’ meeting to ensure everyone knew their role and the established project parameters (as detailed in gathering requirements below). we then employed 3 two-week cycles as follows: week 1, monday-friday: web development team creates a website section, and distributes the changes to the group. week 2, monday-wednesday: the 4-person mobile advisory group tests for functionality and accuracy, and submits changes using the project wiki. week 2, thursday-friday: development team incorporates feedback, and releases the section to production. week 3, monday: advisory group meets with the development team to discuss the next upcoming cycle.  as with the kickoff meeting, an in-person gathering is a good opportunity to set mutual work expectations and reinforce the short timeline.  it also allows the developers to communicate any technical challenges that would be out of scope for the next cycle. weeks 3-6: two additional cycles are completed. open and in-person communication channels while an essential part of any project, the development of our mobile site featured a greater degree of communication than is usually found in technology projects at the ucsd libraries. representatives from user services and it frequently met in person to discuss changes and feedback. outside of meetings, an effort was made to be easily accessible via chat and social media. an awareness that a traditional approach would miss the entire arc of mobile library use a traditional long-term approach involving multiple stakeholders and committees can at times be an ideal method. however, when addressing rapidly evolving technologies the slow and steady approach guarantees you will implement the service after it is most relevant. the dramatic rise of smartphones, with a pew study showing that nine out of ten 18 to 29 year olds own a cellphone and that 65% of them access the internet using these devices [3], perhaps best illustrates the need for libraries to act quickly to meet their users’ needs in this area. a small “super group” given authority to decide and design on behalf of the entire organization for certain projects, decision-making via committee quickly resembles design via committee: the outcome can be overly complex, unwieldy, and lack cohesion. in contrast, our project group consisted of just five user services representatives and one it staff member. while most projects of this scope require oversight and input from many committees (administration, reference services, instruction, etc.), members of the “super group” were considered representatives of several committees and were given the authority to make decisions on behalf of the entire library. an understanding that sometimes fast and “good enough” is better than slow and perfect along with recognizing the need to act quickly, we made the decision to create a website that was useful enough for most user needs, but did not meet every need. thus our initial launch included the key items that were present in other library mobile sites, and intentionally excluded the bells and whistles that might have slowed progress. instead, we plan to add new services incrementally as they are developed. gathering requirements the initial requirements gathering for the project began with an environmental scan of what library mobile websites currently offer. in total, 25 academic libraries were included in the scan, and all available services were tallied inclusively. [4] this environmental scan has since been expanded as part of the cdl mobile device user research project. [5] it was reviewed thoroughly by it and the mobile advisory group and was used as the foundation for prioritizing our initial mobile services. the resulting services we chose to implement initially were: hours catalog ask a librarian research tools and databases maps and directions contacts view full site since february of 2009, the ucsd campus has been using the hannon hill cascade server content management system. [6] the library’s public website and the library’s intranet website were developed and are maintained using cascade server. therefore, in order to follow campus standards and leverage existing resources, the mobile website development occurred within cascade server as well. as a result of this requirement, external mobile frameworks such as the open source mit solution [7] were not considered. the mobile device platforms targeted by our initial offering can be summarized as mid to high level devices with high market penetration and a solid user base. [8] our first priority was to reach as many clients as possible by using the web as a platform rather than developing a mobile application for specific devices. by developing a mobile website we were also able to leverage the skills and knowledge of our web team in it. had we taken the application approach we would have needed time and resources to train staff in new programming languages and apis or contract the development work out to another company. given the current fiscal situation at ucsd, neither of those options was acceptable. a final benefit to this approach is that changes or updates to the website are published and available immediately, while a mobile application approach would require updates submitted to various application stores for approval and then pushed out to users’ phones. the final requirement was a solution not only for displaying the hours for all nine libraries on the mobile website, but to also create a new hours data entry and storage mechanism within cascade server. the current mechanism for entering, storing and displaying the ucsd public website hours is rooted in a legacy software application which is not only cumbersome to use, but is becoming increasingly difficult to maintain. therefore, in addition to creating a library hours interface, we also developed a data entry solution in cascade server which will eventually be used as a central repository for all library hours data. design process figure 1. contrasted displays of the ucsd full website vs. the mobile website designed for smartphones. it was critical early on in the design process to set expectations and perspective in a mobile context. we designed our website with the understanding that mobile users have different goals when accessing the internet than users coming from the traditional desktop or laptop interface, and we were careful to develop templates and content which required limited scrolling and used appropriate font and user interface element sizes. we continually referenced proven mobile best practice guides during the design process. we used the safari web content guide [9] while designing many of our user interface elements such as lists, buttons and icons. we also consulted the yahoo style guide [10] for content formatting in a mobile context. css3 was used, when appropriate, for progressive enhancement styling of the user interface. the interface is designed to be as intuitive as possible given the mobile context. for example, since users with a mobile device do not have the ability to hover over a link to see where it might take them, it was critical to not make them guess what the destination might be. an html link to send an email or text message, launch an application store such as itunes, make a phone call, or even load an external website is a much more alarming experience on a mobile device than on a desktop browser. with that in mind, we were careful to design our link template to allow developers to specify the type of link, and included icons to inform the user what action will be taken when a link is chosen. navigation is also a unique experience on a mobile device. today’s monitors with high resolutions of well over a thousand pixels per axis have no problem cleanly displaying multiple levels of navigation depth via breadcrumbs or lists.  however, mobile devices do not have this real estate, with a width of generally either 480 pixels or 320 pixels for an hvga screen (depending on how the device is oriented). our solution was to leverage the icon set used on our main page as section breadcrumbs in navigation. as the user navigates the site, we offer a consistent icon-based breadcrumb trail for returning back to previously navigated levels of the site. using fixed width icons rather than variable length page titles as breadcrumb hyperlinks ensures a consistent and intuitive experience for our users. figure 2. navigation was simplified through icon based breadcrumbs. keeping in mind that we were the first group on campus to develop a mobile website using cascade server, we designed our templates so that they could be scaled to become a ucsd-wide solution. one of the key design decisions that made this possible was our icon-based breadcrumb trail. by simply adding a ucsd parent icon at the beginning of the trail, our mobile templates could be leveraged across the entire campus. this would give all ucsd users a consistent mobile experience, while ensuring content editors could use the same familiar cascade server template system. we are currently in discussion with the ucsd campus web development group to explore this possibility. in order to provide the ability to redirect users browsing the web with a mobile device to our new website, we have taken advantage of the open source mobile browser detection script hosted at detectmobilebrowser.com. this script directs users accessing libraries.ucsd.edu with a mobile browser to libraries.ucsd.edu/m. we do, however, allow users to navigate to the full website if they desire, and we provide a link on the home page of the mobile site for this purpose. performance was another important consideration. mobile networks are often considerably slower than fixed data connections. mobile devices and browsers also do not have the same processing and memory caching abilities as desktop browsers. while steve souders [11] and ryan grove [12] have recently shown that the cache size limit for a single stylesheet or script has improved lately, the maximum cache size is still an area of concern and study. we have currently addressed this issue by compressing [13] our stylesheets and scripts, keeping images as few and compressed as possible, and setting far apache future-expires headers for applicable content. we recognize, however, that this is a fluid area of study and will be paying close attention to any new solutions or developments within the web community that we can take advantage of. future challenges creation of a new library service does not mark the completion of a project, but instead is one phase in the overall life cycle of the service.  this is particularly the case for mobile devices, where rapid change and shifting platform support is the norm rather than the exception. looking forward, one immediate challenge is to routinize the process for updates and additions to site content.  since portions of our mobile website have the same content as our regular website, ideally staff could update both at the same time, and using the same tool.  thus critical content such as library hours and contact information would not get out of sync.  another challenge is to build in regular usability testing that is tailored specifically for cross-platform devices and the information needs of a mobile user.  lastly, we need to prioritize future additions to the website based on their usefulness and the amount of development work involved.  continually working towards these goals will ensure our mobile website evolves to meet the needs of library users. conclusion overall, this project and process have garnered positive feedback from users and library administration, and we envision employing a similar approach to development projects in the future. however, it is important to note that this is obviously not a panacea for project management in general, as certain characteristics limit what can be accomplished. this model is most appropriate for projects with: small to medium scope key staff available for a short duration of significant workload support of library administration in allowing the representatives to make decisions on behalf of all the stakeholders. timeliness and rapid development as a crucial factor for success conversely, it might not be a good model for projects with: enterprise level scope dependencies on campus or consortia-wide involvement the need to include and consider feedback from every stakeholder a lack of firm administrative support to illustrate the above differences, we found the agile-based approach is well suited for the development of a mobile website (with its small scope and technology timeliness), but might be the wrong choice for the creation of an entire library website. as stated earlier, with our organization there is a broad and layered hierarchy to most project planning and implementation, so short timelines, quick communication practices and overall agility would be challenging to implement with our organization. as with any project, proper evaluation of the scope and timeline is a key factor in deciding whether a rapid development model will be successful, and it is key to embrace systems of development that can be modified to meet the needs and breadth of a project. endnotes [1] http://www.editlib.org/p/12509 [2] james shore’s the art of agile development is an excellent introduction and practical guide to the agile method: (coins). [3] pew internet: mobile access 2010: http://pewinternet.org/reports/2010/mobile-access-2010/summary-of-findings.aspx. [4] institutions were identified using the library success: a best practices wiki: http://www.libsuccess.org. [5] cdl mobile device user research project: https://confluence.ucop.edu/display/cmdur/resources#resources-comparativeanalysisresources. [6] hannon hill cascade server: http://www.hannonhill.com/. [7] mit mobile web framework: http://sourceforge.net/projects/mitmobileweb/. [8] admob mobile metrics: http://metrics.admob.com/2010/06/may-2010-country-data/ statcounter global stats: http://gs.statcounter.com/#mobile_browser-us-monthly-201006-201008-bar. [9] apple safari web content guide: http://developer.apple.com/library/safari/#documentation/appleapplications/reference/safariwebcontent/introduction/introduction.html. [10] yahoo style guide: http://styleguide.yahoo.com/. [11] high performance web sites blog: http://www.stevesouders.com/blog/2010/07/12/mobile-cache-file-sizes/. [12] yahoo! user interface blog: http://www.yuiblog.com/blog/2010/07/12/mobile-browser-cache-limits-revisited/. [13] yahoo! developer network: http://developer.yahoo.com/yui/compressor/. about the authors matt critchlow is web technical manager at the university of california, san diego libraries. matt holds a bachelor’s of information and computer science from the university of california, irvine. in his current position he works with ucsd library staff on a variety of web projects and his group is responsible for the entire web presence of the libraries. his professional interests include project management, user experience, digital preservation and staying out of trouble whenever possible. matt tweets as @mattcritchlow. lia friedman is head of public services, instruction & outreach librarian, arts library uc san diego. lia is also staff librarian for make/shift magazine and a member of radical reference. contact lia at lia.friedman (at) gmail.com dan suchy is user services technology analyst at the university of california, san diego libraries.  he works with a broad range of technologies to improve the library user experience.  you can email him at dsuchy (at) ucsd (dot) edu, or visit his badly outdated website at: http://sandiegotransplant.com. subscribe to comments: for this article | for all articles 4 responses to "using an agile-based approach to develop a library mobile website" please leave a response below: emily lynema, 2011-01-07 i am interested in your thoughts about what types of project agile methodologies work well for. in my thinking, one of the great things about agile is increased transparency and open communication. you mention that at your institution, agile techniques might not work well for the creation of an entire library website. but to me, that sounds like a great project for agile! with iterative timelines, decisions can be made in small pieces, and staff and committees can have products like wireframes or mockups to react to even before anything has actually been developed. this kind of iterative feedback loop helps move along large projects where it’s really difficult to get all decisions made ahead of time. so i guess my thought is that agile could work well with large projects. perhaps the issues that you anticipate are institution-specific, however. matt critchlow, 2011-01-21 hi emily, great post. my thoughts are the following: yes you’re absolutely right that agile could be the perfect project methodology to use for building an entire library website, or any other large project. agile has been around for years, and there have been plenty of successful large projects completed using the methodology or it’s variants (e.g. scrum). like any project methodology, it is really only able to be applied successfully in the right environment. this includes project sponsor and champion support, as well as all stakeholders. our environment is not currently at the point where building an entire library website would have the support necessary to use an agile methodology, and that’s what we were trying to convey in the article. if your library/institution environment is different, you may be in a better place to pursue a large agile project. so all that to say, i think your environment, resources, as well as support from management/administration are going to be the indicating or limiting factors of whether agile will work for a particular project. i hope that helps clarify where we were coming from. if you’d like to discuss this more offline, both dan and i will be at the code4lib conference or feel free to contact any of us on email anytime! jodi, 2011-09-19 so a bit more complicated than just downloading the mobile theme plugin on wordpress then ;) when diy = sigh. | info-mational, 2013-06-06 […] in libraryland that listing them is a challenge. one well-documented tech-focused effort is an agile mobile site design project undertaken by cross-departmental staff at the ucsd library – in essence, lia friedman, matt […] leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – keeping up with ebooks: automated normalization and access checking with normac mission editorial committee process and structure code4lib issue 20, 2013-04-17 keeping up with ebooks: automated normalization and access checking with normac cataloging ebooks is difficult to do well, as they are often purchased in large collections, sometimes with only low-quality cataloging copy available. marc records may be provided upfront in a large batch, or trickle in one at a time as they become available. records may contain links that point nowhere, to the wrong book, or to an offer to sell you the book you already own. loading records sight unseen may introduce inconsistency or overlay good print records with poor electronic ones, making the catalog much more difficult to search. this article describes in more detail the major challenges in ebook cataloging, record normalization and access checking, and introduces normac: an open source web-based tool for processing marc records. by kathryn lybarger introduction cataloging ebooks is difficult to do well, as they are often purchased in large collections, sometimes with only low-quality cataloging copy available. marc records may be provided upfront in a large batch, or trickle in one at a time as they become available. records may contain links that point nowhere, to the wrong book, or to an offer to sell you the book you already own. loading records sight unseen may introduce inconsistency or overlay good print records with poor electronic ones, making the catalog much more difficult to search. traditional tools for dealing with marc may not easily handle the challenges of large batches with complicated conditional field edits, especially when faced with multiple cataloging standards present in a single batch. checking not only that a link points somewhere, but that it points to the book in question is also beyond the standard features offered by cataloging software or web site link checkers. to help solve these problems, i wrote normac: a new software package for editing batches of marc records to bring them up to local standards and checking ebook access. once the system is configured, a new batch of marc records can be quickly processed through a web interface or from the command line. some background – marc editing marc is a flexible file format for data encoding and interchange, but in its native binary format it is not easy to edit: figure 1: marc record in its native format most ils and cataloging software (such as ex libris voyager) include a friendly editor for record-by-record marc editing: figure 2: ils marc editor many also include batch editing functionality, such as ex libris’s global data change and millennium’s global update. there are also software libraries available for manipulating marc records, including the python library pymarc [1] and the perl library marc/perl [2]. perhaps most popular among librarians is terry reese’s software marcedit [3]. it provides batch conversion of marc records between several formats, including binary marc, mnemonic text marc and marcxml, and does character set conversion between marc-8 and unicode. it also includes a powerful yet friendly marc editor for applying common edits (including regular expressions) to mnemonic marc files. marcedit also includes a gui script wizard which creates vbscript programs to perform common operations; those scripts may be edited to perform arbitrary operations on marc batches. challenges in normalization when adding ebook records to our catalog, we would like to do so efficiently and expediently, with consistent quality in the records, and only including working links. one challenge stems from the variability in size of record batches. often when we add a new collection to the catalog, marc records for all titles are available upfront from the vendor. we may have thousands of records to deal with at once, which is not itself a problem; marcedit can handle large numbers of records, and many powerful checks and modifications can be done through its marceditor interface. that is an efficient way to handle large batches (spending relatively little time per record) but the efficiency falls off if we do that same marceditor procedure frequently on smaller batches of files, such as an update to that collection. marcedit addresses this need in some instances with its script wizard add-in, which allows some of its more common functions to be saved as a script, but more complex edits may be desired. it may be quicker to modify small batches one record at a time, but this sacrifices consistency. it may be more efficient to wait until we have a larger batch of titles before performing the batch edits, but we prefer to make the titles available to our patrons as soon as possible. it would be nice to perform complex edits to batches of marc records efficiently and with consistent results regardless of batch size. another challenge stems from the variability of cataloging standards that may be present in a single batch of records. records for a single collection gathered from a collective like oclc may have been created at different times by different catalogers following different standards. some cataloger judgment is allowed in what makes a record acceptable quality: records may vary in their inclusion of call numbers, authorized subject headings, and contents notes. some records even contain content specific to an individual institution (and not useful to others) such as urls modified to restrict access with a proxy prefix. records in a collection may vary not only in details, but also may just use different codified standards. prior to 2009, ebooks were cataloged as reproductions of print books, with each record specific to the vendor who had “digitized” it (even if the book was born digital). most fields in the record were identical to those in the corresponding print record, with a few fields and subfields to indicate its electronic nature: 245 04 $a the developer's guide to debugging $h [electronic resource] / $c by thorsten gro?tker, ulrich holtmann, holger keding, markus wloka. 300 __ $a xix, 224 p. : $b ill. ; $c 23 cm. 533 __ $a electronic reproduction. $b new york : $c springer, $d 2008. $f (computer science). $n mode of access: world wide web. $n system requirements: web browser. $n access may be restricted to users at subscribing institutions. in 2009, the recommended standard changed to provider-neutral records [4]; that is, one record describes all potential electronic versions of a title and includes links to all known versions. this change seemed to raise the overall quality of ebook records (since all improvements to a title’s metadata were concentrated on one record), but the new standard brought other issues with it as well. notes that would have been included with provider-specific records, such as: 538 __ $a mode of access: world wide web. 506 __ $a restricted to subscribers. were no longer appropriate, because they would not apply to an open access version available via ftp. the format of the physical description is also different: 300 __ $a 1 online resource (xix, 224 p.) : $b ill. and added entries for ebook collections or vendors that would previously have aided the cataloger to collocate all ebooks in a package would be removed. patrons using your catalog may not notice these differences, but they would certainly be confused by the appearance of multiple links (one for each vendor), most of which did not work for them. for consistency and utility, catalogers may want to standardize formats, restore absent notes, and remove inappropriate links. again, in 2013, we are seeing a mix of standards in record batches as some cataloging agencies are switching to the new cataloging standard rda (resource description and access) [5] while others are continuing to catalog in aacr2. many differences between the two standards do not necessitate different processing procedures, though a couple of them might, depending on your local cataloging environment. for example, rda records include three new physical description fields (33x), indicating the content, media and carrier types of the resource: 245 __ $a for whom the bell tolls. 336 __ $a two-dimensional moving image $2 rdacontent 337 __ $a computer $2 rdamedia 338 __ $a online resource $2 rdacarrier however, many ils’s are not prepared to handle these new fields. while it may be straightforward enough to add 33x fields to the tag table so they don’t cause errors when used, it is quite another to make the ils display them in a way that is as clear to patrons as the gmd: figure 3: opac record displaying gmd until their ils software catches up with the change in standards, libraries are handling rda records in various ways. some are loading them as-is, but others are making changes such as deleting the 33x fields, applying a static gmd to batches known to be of a certain type (electronic resource, video recording, etc) or copying data from 33x fields to form a new gmd, such as: 245 __ $a for whom the bell tolls $h [two-dimensional moving image : electronic resource] with a mix of record standards and quality, the cataloger doing batch editing may want to apply unusual logic, such as “if the record is coded rda in the 040, and there is no 245$h, concatenate the first instances of 336 and 338 and add that string as a 245$h following its $a.” such needs are unlikely to go away over time. when ils software can handle these new fields effectively, catalogers may wish to add them even to aacr2 records; oclc is already allowing such changes to its master records [6]. using normac’s normalizer, you can write a simple configuration file to perform common functions (such as adding and deleting fields) and run pre-defined functions to perform more complicated functions (such as converting a record to the provider-neutral standard, or moving oclc numbers to a 776 field). you can also write (and share) arbitrarily complicated functions to be run on your marc records. challenges in access checking there are many popular solutions for the problem of web site access checking, including the w3c link checker [7] and xenu’s link sleuth [8]. unfortunately, many link checkers work by checking http status codes [9]: some common status codes: 200 ok request successful 403 forbidden server refuses to fulfill your request 404 not found server could not find anything matching your request if the software queries the web server with the url and receives a good code like 200, it will consider the link to be good. if it finds a bad code like 403 or 404, it will report the link as bad. when the web pages in question are ebooks, the answers are not that simple. an ebook may appear fine to a link checker, even when you don’t really have access to its content. many ebook links point to a metadata page with a table of contents; that page is a valid web page even if you don’t have access to individual chapters. if you have your link checker go to a greater depth (not only check the link, but the links on the page it returns), some chapters such as front matter may be available and others may not. even if the content is not available to you, the link checker can still be fooled; rather than throwing up an unfriendly 404 page, the site may redirect you to a web form where you can login with different credentials or purchase the content, and that web form may be a valid web page with a 200 status. similarly, an ebook may appear bad to a link checker when it is just fine: a 403 status may be returned for all links if the ebook platform just does not allow link checking by search engine robots. an ebook may instead have a broken link which is relatively easy to fix. many ebooks have dois, which are great for inclusion in ebook records; in the case of a platform migration or other change, the vendor only needs to register the change with crossref to effect the change in all catalogs at once. if the doi is not registered properly however, the catalog link will return a page indicating a bad doi. when this situation is detected, you can just replace the doi with the direct link in your catalog, but it is preferable to report the doi as broken, fixing the broken link for libraries that have already loaded the record with the broken link. using normac’s access checker, you can check a list of ebooks for actual functional access based on the specific ebook platform. with some programming, the software may be configured to search for key phrases from that platform (such as “not available to your institution”) or perform arbitrarily complex actions, such as searching the table of contents for the first non-frontmatter chapter and confirming that the page returned by that link is in pdf format. after all checks are done for a batch, different types of access errors (no vendor access, broken doi) are identified for reporting to the appropriate vendors. normac normac is web-based software that normalizes marc records according to institutional policy and vendor specifics, and then verifies access to any linked online resources. it can be easily configured to add, delete and modify fields in common ways, and can perform arbitrarily complex checks and edits with additional programming. once normac is installed and configured, you can process marc batches from the command line, or by feeding them to a web page and selecting settings for the batch. normalization and customization finish quickly, prompting only for edits that require manual intervention, and then the modified file is made available for download. figure 4: normac if you would also like to do access checking for links in the marc record batch, you can then submit the modified file to the access checking queue. if you wish to check access for a collection already in your catalog, you can also just submit a list of those links. link checking is a slower process, mainly for consideration to the vendor; requests will only be sent to a given vendor approximately every ten seconds so as not to be a nuisance. after links are checked, the marc batch is split into two files: one with working links and one where links have problems. a report is generated indicating the types of errors, such as broken dois, deleted resources or lack of institutional access. the good file may be then loaded into the catalog, while problems are investigated. after problems are addressed, the problem file may be re-submitted for link checking before loading. customizing normalization for institutions and vendors to simplify creation of workflows for new vendors, normac accepts vendor profiles describing changes that should be made to records from that vendor. this profile includes static fields that should be added, patterns of fields that should be deleted, and functions that should be applied to the record at various times. profiles are encoded in an ini-style file (example: uk.ini) with functions implemented in a sibling php file (example: uk.php). multiple profiles can be applied to a batch, allowing an institution’s main rules to be encoded in a primary profile and having smaller, simpler profiles specific to individual vendors. for example, an institution’s profile may have the form: [description] base behavior for ebooks processed at uk; additional behavior should be added to individual vendor profiles [variables] proxyprefix = "http://ezproxy.uky.edu/login?url=" publicnote = " -click here for internet access to title" goodgmd = "electronic resource", "videorecording" [initial] [delete] =029 =506 =533 =538 =6.. .[^02] =938 [middle] removeinvalidlinks add856label add856proxyprefix add856publicnote neutralize300 addgmd [add] =099 \\$asee internet access =538 \\$amode of access: world wide web. =506 \\$arestricted to subscribers. [final] that institution’s profile for wiley ebooks might look like: [description] extra processing for wiley ebooks [variables] validurlhosts = dx.doi.org/10.1002, onlinelibrary.wiley.com label856 = wiley online library [initial] specialwileystuff [delete] =710.*wiley interscience [middle] [add] =730 0\$awiley online library. [final] with these two profiles defined and applied (in the order uk, wiley), the following occurs: 1. uk profile variables are set. 2. wiley profile’s variables are set, possibly overwriting uk’s 3. uk’s initial functions are run. 4. wiley’s initial functions are run. 5. uk’s delete lines are deleted. 6. wiley’s delete lines are deleted. … (and so on) given the modular nature of vendor profiles and functions defined in them, code for small specific tasks can be compartmentalized, allowing them to be quickly re-ordered within a procedure, easily modified, and shared between institutions. customizing access checking for platforms normac accepts platform profiles that distinguish between a good ebook link and bad ones of various types. this profile is a php file containing a function valid_url that accepts a url and returns 0 for a good url and other numbers for different types of problems. the function will be run on each url queued for access checking. within a profile, links may be used as objects with some useful methods available, such as httpstatus (to check for 404 not found or 403 forbidden), contains (to check if the contents contain a string), and ispdf (to confirm an expected content type). <?php require_once "macprofile.php"; class sciencedirect extends macprofile { function verify_url( $url ) { if( is_string( $url ) ) $url = new url( $url ); if( $url->httpcode() != "200" ) return self::bad_http_code; if( ! $this->verify_url_access_string( $url ) ) return self::no_access; if( ! $this->verify_url_has_working_pdf( $url ) ) return self::no_pdf; return self::ok; } private function verify_url_access_string( $url ) { return true; return $url->contains("you are entitled to access the full text of this document" ); } private function verify_url_has_working_pdf( $url ) { foreach( $url -> gethtmldocument() -> getlinks() as $link ) { if( substr( $link->geturl(), -9 ) === "-main.pdf" ) return $link->ispdf(); } return false; } } implementation details normac is open source software written by me in collaboration with my husband jack schmidt, and the source code is available from the github repository: https://github.com/zemkat/normac . the code is written in php/mysql. marc records may be submitted as binary marc or as mnemonic text marc (.mrk). within normac’s code, marc record representation is object-oriented, making available functions straightforward to read and modify. the command line version of the normalizer does not require a database and can be run on unix, mac os x, or windows (tested under cygwin). the web version has been tested on a linux / apache server and major web browsers. the access checker requires a database for link queuing, and currently runs on mysql under linux. future plans the current version of the software requires some installation of software either on a server or desktop computer. if this requirement is prohibitive, the normalizer may be rewritten in javascript, allowing the process to run completely by visiting a web page, with no need to upload files to a server. your contributions? i have included functions that i have found to be common types of edits so that they may be easily shared and modified. i welcome contributions of functions for new kinds of edits, or suggestions of ones that would be useful. i have also included platform profiles for detecting access issues for some vendors with which i have encountered them. if you find ebooks whose access is being incorrectly detected, please let me know. i also welcome submissions of new vendor profiles to the repository. notes: [1] pymarc [internet]. available from: http://pymarc.sourceforge.net/ [2] marc/perl [internet]. available from: http://marcpm.sourceforge.net/ [3] marcedit homepage [internet]. available from: http://marcedit.reeset.net/ [4] provider-neutral e-monograph marc record guide [internet]. library of congress. available from: http://www.loc.gov/aba/pcc/bibco/documents/pn-guide.pdf [5] rda toolkit: resource description & access [internet]. available from: http://www.rdatoolkit.org/ [6] oclc rda policy statement [internet]. available from http://www.oclc.org/rda/new-policy.en.html [7] w3c link checker [internet]. world wide web consortium. available from: http://validator.w3.org/checklink [8] xenu’s link sleuth [internet]. available from: http://home.snafu.de/tilman/xenulink.html [9] http/1.1: status code definitions [internet]. world wide web consortium. available from: http://www.w3.org/protocols/rfc2616/rfc2616-sec10.html about the author kathryn lybarger (kathryn.lybarger@uky.edu) is head of cataloging and metadata at the university of kentucky libraries. her interests in libraries include metadata of all sorts (including traditional cataloging and fancy new standards), preservation, digitization, and automation / computer-assisted workflows. subscribe to comments: for this article | for all articles one response to "keeping up with ebooks: automated normalization and access checking with normac" please leave a response below: tim mccarthy, 2015-03-13 very interesting article. since it’s from 2013, i’d love to hear how people are using it and what platform they’re running it on. leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – editorial mission editorial committee process and structure code4lib issue 50, 2021-02-10 editorial resuming our publication schedule by eric hanson after an interruption in our quarterly production schedule, the code4lib journal has returned for its 50th issue. sadly, the pandemic has been going on long enough that a phrase like “unprecedented times” feels almost trite and clichéd as we enter the one-year anniversary of the period in which much of the world entered lockdown. even though it feels overused, it is hard to find a better descriptor for something that has disrupted and destroyed the lives of so many across the world. as pervasive as covid-19 has been in all of our lives, it should come as no surprise that its influence is felt in many of the article submissions. six of the nine articles in this issue mention covid-19 and several focus directly on initiatives taken in response to the pandemic. while it is hard to find any silver lining in the pandemic’s devastation, it has been encouraging to see people find creative solutions to pandemic-related challenges and begin to share those solutions with the wider community. despite skipping our fall issue, the editorial committee was not entirely idle during the production gap. we sent out a call for editors, reviewed the applications, and selected three new editors to join the committee from a pool of many qualified candidates. this is the first issue for three new editors: gustavo candela from university of alicante, kirstien kroeger, and angela j.a. kent. the editorial committee also unfortunately lost a member with rebecca hirsch stepping down. we are grateful for her service on the committee since issue 37. the articles of issue 50 managing an institutional repository workflow with gitlab and a folder-based deposit system by whitney r. johnson-freeman, mark e. phillips, and kristy k. phillips details the transition at the university of north texas libraries to a new gitlab-based workflow that was paired with an existing folder-based deposit workflow. customizing alma and primo for home & locker delivery by christina l. hennessey discusses the implementation of new services in response to california state university, northridge’s physical locations becoming inaccessible to patrons. ganch: using linked open data for georgia’s natural, cultural and historic organizations’ disaster response by cliff landis christine wiseman, allyson f. smith, matthew stephens describes the efforts at the atlanta university center robert w. woodruff library to create and maintain a directory of natural, cultural, and historic organizations to aid emergency planners. archive this moment d.c.: a case study of participatory collecting during covid-19 by julie burns, laura farley, siobhan c. hagan, paul kelly, and lisa warwick details the effort to create a local archival collection as historical events unfolded for the people’s archive at the dc public library. advancing arks in the historical ontology space by mat kelly, christopher b. rauch, jane greenberg, sam grabus, joan boone, john kunze and peter m. logan lays out the process and lessons learned in applying arks to a historical ontology at the nineteenth-century knowledge project. considered content: a design system for equity, accessibility, and sustainability by erinn aspinall, amy drayer, gabe ormsby, and jen neveau describes the transition at university of minnesota libraries to a more intentional and ethically-guided method of system design. robustifying links to combat reference rot by shawn jones, martin klein, and herbert van de sompel explains the concepts and practices underlying the robust links approach. machine learning based chat analysis by christopher brousseau, justin johnson, curtis thacker details the implementation of a machine learning tool to analyze chat interactions between patrons and staff at brigham young university’s harold b. lee library. always be migrating by elizabeth mcaulay presents a different way of thinking about migration based on her experiences at university of california, los angeles. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – book review: 3 titles from a book apart mission editorial committee process and structure code4lib issue 14, 2011-07-25 book review: 3 titles from a book apart three recently published books by a book apart, the book-publishing arm of the website a list apart, offer concise, high-impact introductions to three tools that can be employed in facing this challenge: html5, css3, and content strategy. this article reviews the books “html5 for web designers” by jeremy keith; “css3 for web designers” by dan cederholm; and “the elements of content strategy” by erin kissane. by nathan mealey introduction faced with a diverse range of content and devices to model it for, and an array of users with unique information-seeking needs, web-developers face a significant challenge in creating highly usable, content-rich websites. three recently published books by a book apart offer concise, high-impact introductions to three tools that can be employed in facing this challenge: html5, css3, and content strategy. a book apart is the print-publishing venture from a list apart, the web-design site that was founded in 1998. from the outset the site’s creators adopted unique approach and voice, developing a standards-based design for its site (remember that this was 1998) and writing articles that blended timely topics, advocating best practices, and voicing provocative opinions on the state of web design. since then, a list apart has published over 300 articles, and in the process has established itself as one of the most prominent voices in the web-design world. in 2010 they launched a book apart with the goal of publishing “highly detailed and meticulously edited examinations of single topics…the goal of every title in our catalog is to shed clear light on a tricky subject, and do it fast, so you can get back to work.” [1]  a book apart’s first title, html5 for web designers, came out last fall, and was quickly followed by css3 for web designers. the third title, the elements of content strategy, was published this past spring. each book is written by an expert in the field who explores their subject through well-written text and step-by-step examples, and each provides a range of compelling reasons for why you should be designing your content and sites using these tools. html for web designers by jeremy keith in this book, jeremy keith deftly turns a potentially dry subject into compelling reading. he kicks off the book by laying out the genesis of html5, a saga which makes an interesting story all on its own. the path leading to the creation of the html5 standard has been a circuitous one that reflected two different visions of web-development’s future: one based on defining a whole new, non-backwards-compatible, xml-based standard versus one based on proven practices in html/xhtml and the assurance of backwards-compatibility. in the end, html5 definitively embodied the latter, as the book sets out to demonstrate. keith begins by delving into the broad strokes of html5, first illustrating the design principles that it embodies: “do not reinvent the wheel” and “pave the cowpaths”. [2] ostensibly, with the design of html5, the goal was to take established ways of accomplishing development tasks and codify them in the html5 standard. at the same time, in order to support existing content a priority was placed on making html5 backwards-compatible. keith states that, “on the one hand, the specification needs to be powerful enough to support the creation of web applications. on the other hand, html5 needs to support existing content…it’s a delicate balancing act that requires a pragmatic, level-headed approach.” [3]  ultimately, this is an approach that will depend on the efforts of browser makers implementing the standards, and designers using them. with this in mind, keith begins illustrating the key aspects of html5, starting with some of the basic elements that are new or changed, such as the simplified syntax of the html5 doctype declaration (<!doctype html>) or stylesheet call (<link rel=’stylesheet’ href=’file.css’>); elements that have been identified as obsolete in the new specification, including the frame, frameset, font, big, strike, bgcolor, cellspacing, valign, and others; [4] and the new javascript apis that are built into the specification. the new javascript apis are a particularly exciting aspect of html5, as they should result in common javascript tasks being both well documented and consistent from browser-to-browser. keith then goes on to tackle the html5’s support for “rich media”, specifically how audio and video content can be delivered using the html5 <canvas> element. he gives a number of examples for safely using this new element in current browsers, and really demonstrates the potential inherent in this powerful new element. his examples cover the ability to conditionally deliver content based on a device’s capabilities, to include native playback controls, and for styling embedded media. this should be particularly relevant for developers whose sites deliver a range of content that will be accessed on myriad devices (some of which will not support flash). following this, keith next looks at html5’s new features for web forms. html5 gives developers a range of exciting tools for improving forms, such as the ability to add placeholder text using the placeholder attribute for input tags, requiring validation via the required attribute, and the new datalist element that merges together the functions of an input element with a select element. he also touches on some of the new input types, including contact-related types (email, url, and tel), dates, times, sliders, color pickers and even regular-expression based patterns (using the pattern attribute). in each case, he provides clear, relevant examples that illustrate putting these new types into practice, and what to expect from different browsers. keith devotes a whole chapter to the role of semantics in html5. at first, this area of html5 may seem somewhat arcane, until you get to his discussion of using the new semantics for structuring your pages by using the <section>, <header>, <footer>, <aside>, and <nav> elements. while these are not required replacements for our typical use of divs to mark up content (e.g. <div id=”header”>), their benefit lies in the ability of a browser to recognize these tags and thus understand the page and its content semantically. as a result, browsers would be able to display a page to different interfaces (desktop, screen reader, mobile browser) more effectively by structuring the content appropriately for each type of device. keith’s examples throughout this discussion do a great job of illustrating the impact that these new concepts could have. keith wraps up the book with a look at which html5 features are currently implemented in browsers. since the book’s release in early fall 2010, all of the major browsers have released new versions that incorporate more html5 support than those versions available at the time the book was written (even microsoft, with ie9, has arrived at the html5 standards table!). the wide support of html5 means that the code examples in the book enable a developer to hit the ground running. this ultimately is the greatest strength of keith’s book: he weaves together new concepts with relevant examples that enable us – as web developers – to put html5 to work today. css for web designers by dan cederholm the title of this book’s first chapter, “using css3 today” clearly indicates the author’s goal of explaining how developers can begin using css 3 today in a practical way.the book begins with a few pages discussing when to push the envelope with css3, but by page 6 he’s discussing a list of high-impact css3 properties that are already supported by browsers, and that the rest of the book will cover.  each of the next four chapters is devoted to a practical demonstration of these properties. [5] chapter 2 looks at css3 transitions, a feature that had been unevenly supported by browsers, therefore necessitating a range of javascript solutions. with the growing adoption of css3 by browser makers, this feature is pretty commonly supported (even in ie’s latest version). with this chapter, cederholm initiates the structure for the remaining chapters. using his demo website (“things we left on the moon” [6]), he focuses on a specific set of css3 elements and incorporates them into the site’s design, including code examples all along the way. being able to see the code on the page, and then in practice on the live site, goes a long way towards making these new css3 techniques readily graspable. chapter 3 and 4 follow this same approach while transitioning to hover-effects and image transformations.  he demonstrates how to make your text and image-based links behave dynamically through user interactions; how to transform images using native css3 properties for zooming, skewing; and changing the orientation of images. again these are features that web-designers have historically needed to use various sleights of hand to accomplish, and for which css3 provides native properties. in both cases, the result is less code, and more browser compatibility than was previously possible. chapter 5 demonstrates how to use css3 layers to build rich backgrounds for your websites and, most importantly, how to gracefully fall-back to a simple background in the case of unsupported browsers. lastly, chapter 6 looks at forms properties that are new to css3. this chapter is particularly full of useful examples that illustrate such tricks as creating stylish submit buttons, adding styles to labels and text inputs, and using gradients for your background colors. in contrast to html5 for web designers, css3 for web designers was more immediately satisfying. the code examples and best practices that this book contains will prove immediately useful in designing your websites. you’ll be able to readily replace javascript snippets and hover-replaced images, and quickly put the form properties to use in your designs. at the same time, the book is (by design) limited in its scope. while other new and powerful aspects of css3 were left out, including media queries [7], multi-column layout [8], and web fonts [9], the book is very focused on experience-layer properties in css3 that you can use in designing sites right now. but then, what it does cover, it covers very well, and in a very practical, helpful manner. the elements of content strategy by erin kissane where the previous two books looked at the tools and standards used for developing websites, erin’s kissane’s concise primer on content strategy illustrates the overarching framework within which this work takes place. the book covers the what and how , of content strategy, and serves as a great companion to the previous two books. in the chapter on “basic principles”, kissane sets the tone for the whole book. in this one, simple chapter, kissane walks through the seven premises that lie at the core of her definition of content strategy [10]: good content is appropriate: publish content that is right for the user and for the business good content is useful: define a clear, specific purpose for each piece of content; evaluate content against this purpose good content is user-centered: adopt the cognitive frameworks of your users good content is clear: seek clarity in all things good content is consistent: mandate consistency, within reason good content is concise: omit needless content good content is supported: publish no content without a support plan for each of the above, she provides a compelling explanation that illustrates a common way in which web content regularly fails its purpose. this section of the book is a must read for anyone who manages a website and its content, and is one that you will want to return to for a regular refresher. following these basic principles, kissane dives into the heart of content strategy, going into great depth on the different approaches that the content strategist brings together, namely that of the editor, curator, marketer, and information scientist. for each of these approaches she provides a detailed explanation and relevant examples of putting the ideas and motivations inherent within each approach into practice. as a result, this section of the book has a lot of content to convey. all the same, kissane does a good job of not overwhelming the reader while successfully illustrating the complexity and enormity of the task that faces anyone responsible for managing a website’s content. i suspect that each person who reads this will find a specific approach that they identify with more than the others. yet, the opportunity to see that one approach in the context of the others that she illustrates will provide a valuable frame of reference. kissane’s discussion on the tools and techniques of content strategy draws on the approaches described in the above discussion. this includes tools such as project definitions aimed at convincingly answering the question of “what are we doing and why?”; research that uncovers who users are and what they want; and the development of guidelines for creating content, including plans for maintaining and nurturing content once it has been created. this is very much the nuts and bolts section of the book, and covers a wide range of methods that you can use when managing content. this is perhaps best summed up by her list of all various types of deliverables a content strategist might produce – a list of 34 different items, including accessibility guidelines, benchmarks, channel strategy, cms requirements, communication plans, and many more. these tools and techniques she explicates are all ultimately embodied by one of the deliverables. the chapter on basic principles alone makes the book valuable, but the remaining chapters provide significant ideas, methods, and tools that anyone engaging in a website or content development project should consider. the book’s content may not be quite as accessible as the ones on html5 and css3, owing in part to the lack of demonstrations and code examples that make the latter immediately gratifying while breaking up the text at the same time. but this difference is unavoidable, given the very different nature of the two topics – development standards and examples versus content strategy, planning, and guidelines. yet, this difference results in the elements of content strategy proving to be very complementary to the previous two books. why review all three of these books at once? each of these three books serves as an excellent primer on their subjects. they give you a number of tools that you can use right now, while also pointing towards additional topics and associated resources that you can turn to for more depth. while these are not definitive, one-stop guides to the full breadth of html5, css3, or content strategy, each makes for a perfect jumping off point for those new to the topics. both html5 for web designers and css3 for web designers blend theory, explanations, and examples to great effect, and in both cases you’ll regularly find yourself stopping your reading to test a new html or css tool. each contains a range of ideas that you’ll be able to put into great effect today. the same can be said of the elements of content strategy, which provides an great illustration of all that goes into the craft of content strategy, while also providing specific ideas and tools that you can turn to in developing content now. most importantly, the themes of these three books work well together, providing a complementary range of ideas and tools that developers and content creators can immediately benefit from. references [1] a book apart, http://www.abookapart.com/pages/about [accessed july 11, 2011] [2] keith, jeremy. 2010. html5 for web designers. new york (ny): a book apart, p. 6.  [coins] [3] keith, jeremy. 2010. html5 for web designers. new york (ny): a book apart, p. 11.  [coins] [4] for a complete list of obsolete elements, see http://www.w3.org/tr/html5/obsolete.html [accessed july 11, 2011] [5] cederholm, dan. 2011. css3 for web designers. new york (ny): a book apart. [coins] [6] things we left on the moon, http://css3exp.com/moon [accessed july 11, 2011] [7] media queries, http://www.w3.org/tr/css3-mediaqueries [accessed july 11, 2011] [8] css multi-column layout module, http://www.w3.org/tr/css3-multicol/ [accessed july 11, 2011] [9] css fonts module level 3, http://www.w3.org/tr/css3-fonts/ [accessed july 11, 2011] [10] kissane, erin. 2011. the elements of content strategy. new york (ny): a book apart, p. 7-12. [coins] about the author nathan mealey (nathan.mealey@pdx.edu) is the manager of library technologies at portland state university in oregon. prior to this, he was the information technology & systems librarian at simmons college in boston, ma until october 2010. subscribe to comments: for this article | for all articles 2 responses to "book review: 3 titles from a book apart" please leave a response below: the code4lib journal – book review: 3 titles from a book apart | reviewtica, 2011-07-25 […] see original book review here: the code4lib journal – book review: 3 titles from a book apart […] code4lib journal, n° 14 « pintiniblog, 2011-07-26 […] book review: 3 titles from a book apart […] leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – generating metadata on a shoestring sans programmer, with our good friend, excel (or any spreadsheet) mission editorial committee process and structure code4lib issue 5, 2008-12-15 generating metadata on a shoestring sans programmer, with our good friend, excel (or any spreadsheet) how to use excel to generate metadata for any encoded filename or identifier for any digital object whose attributes can be expressed in an abbreviated form. by jill strass the challenge the tool we are describing came into existence as part of an effort to digitize, and make keyword-searchable, a student-written newspaper at a small college. the paper started in 1887 as a literary magazine, nearly standard-letter in size. the issues were bound into books. the plan was to scan the pages of the books, save them as tiff files, and then upload them into a database. to do the upload, we needed to create a tab-delimited file in a very specific format for the database. this specialized database (oclc’s contentdm) allows for the display of images and text. it also has a module that takes scanned text and makes it keyword searchable (ocr). the format of the spreadsheet instructs the database on the display of images in a digital-book format. this means that images of pages of text, like a printed book, are displayed in a specific order. when the user does a search, items appear in a hit list, and there is a finding aid that shows the user what issue and volume a page appeared in. we selected our metadata for the project with the supposition that our users would wish to browse by date, issue, volume, and year. we also imagined they would like to be able to search by keyword. we opted for both dublin core and oai-compliant metadata. we had no programmers to assist us, and there were approximately 8000 pages we could digitize given the scanner we had. we had student labor for scanning, but didn’t wish to put the onus of creating metadata into student hands. we decided that we needed a single point of data entry, as this would increase the efficiency of workflow and reduce possibility for data-entry errors. we figured out that this single point of data entry could be the filename of the scanned image. with the filename containing all our metadata, we needed to develop a way to generate metadata automatically from the filename. from the metadata generated from the filename, we needed to find a way to display metadata so it was automatically arranged into the oclc-specified, non-marc format. here is the main browse page from volume 1, issue 1 of the manitou messenger: figure 1: the main browse page from the manitou messenger a list of all issues of the newspaper can be selected from https://contentdm.stolaf.edu/cdm4/browse.php?cisoroot=/mess. the tools in a perfect world, we would have used a scripting tool to generate the specially formatted tab-delimited file that contentdm demands for hierarchical documents. the only tool we had available — and had sufficient knowledge of — was excel. our choice was reinforced when, after sending out a plea on a listserv, jane cullinane and glenn sherman at the connecticut state library were kind enough to share their spreadsheet knowledge . they answered our plea for how to convert an encoded filename into a serials-type statement. by serials-type, we mean that the statement was translated from the encoded filename, into plain english to indicate the volume, issue, year, month, and page number.  and their answer to the problem of how to generate metadata from an encoded filename was to use excel. the right/left command was the crucial bit of knowledge we needed to extract values from filenames and generate all kinds of metadata from the filename. what we did we decided to encode our metadata with each filename and extract the information from the filename into an excel spreadsheet. the tool we made with excel is a spreadsheet which has a bunch of formulas on one worksheet. on the second worksheet, the extracted data is rearranged into the correct format for upload into contentdm. to generate actual metadata, the end-user only has to cut and paste a list of filenames into a column in the first worksheet, then cut and paste using the paste special command, and select values only for pasting into a new spreadsheet. this final sheet is saved as a tab-delimited text file and uploaded into contentdm. screen shots below illustrate how the spreadsheet looks as it is in use. figure 2: spreadsheet before filenames are input figure 3: spreadsheet after filesnames are input figure 4: resulting metadata generating the metadata with excel: extracting values from the metadata our file naming convention evolved into this: v001i01m01y1901p001 this stands for volume 1, issue 1, january 1901, page one. now comes the fun part. we needed a way to convert the information from the filename into fields that could be exported as metadata. the first step to doing this was to convert the information encoded in the filename into plain values. for example, v001 stands for volume 1 or it could stand for 001, or 01, whatever we thought would be best for the format of the metadata. to extract metadata from the encoded filename, we used the right/left command. this allows you to tell the computer exactly which characters you want to take from a field. what we did was take parts of the filename, which represented bits of metadata, and extracted each bit into a separate field. =right(left(a2;4);3) to figure out how this command is constructed, let’s go back to algebra and go into the parenthesis first. left(a2;4) means that we go into cell a2, which is where we put our filenames. then inside a2, we go to the far left of the cell. then we take the first 4 characters, reading left to right, which would be v001. the rest of the command tells us to take the 4 chosen characters, and only take the 3 characters starting from the right to the left, which leaves us with 001. it is this value of 001 that is recorded in the spreadsheet and used as metadata to represent the volume number. we repeated this process for all parts of the filename, so we had numerical values for: volume issue month year page number generating the metadata with excel: generating metadata from the values we extracted we obtained the values from the filenames, which we could use as actual data. we also needed to express some of them in a way that would be useful for display purposes. one example is the title field. in contentdm, this field is used as the top level for a book-like collection of images. we used the book-like configuration for the newspaper as it allows us to display by volume, issue and page. when the newspapers are displayed online as volumes, we needed to show serial-like information so users would know which volume or issue they are accessing from a display list: we needed to express the title, volume, issue, month and year. we used this command in excel, which is entered by hand: =”manitou messenger: volume “&b2&”, issue “&d2&”, “&k2&” “&f2&”” in this command, we were able to display the title, manitou messenger. this command allowed us to add the title information without data from the filename: the title itself is generated from text from the command itself. the quotation of text is combined with numerical data extracted from the filename. using the &b2& command, we were able to take the volume information from cell b2 and display it, same with issue, month and year. we strung all these values together in the command above, which allowed us to generate a title to represent the newspaper at the issue level in an online environment. other tricks for generating values getting a textual name for a month to display to get the name of the month to display from the m02 (month is february), we used the scheme from the aforementioned connecticut state library spreadsheet, which consists of an if statement: =if(i65=”01″;”january”;if(i65=”02″;”february”;if(i65=”03″;”march”;if(i65=”04″;”april”;if(i65=”05″;”may”;if(i65=”06″;”june”;j65)))))) this statement (which divides the year in half, the rest of the year appears in a separate formula) takes the value, which we have extracted from the filename into separate fields, and then assigns the textual name based on the value in the field for month. getting an original publication date field (yyyy-mm) to display: take the value for the year and month and display in a single field with a hyphen to separate them. =g2&”-“&j2 getting to display the decade for an object take the year field, only display the first three characters from the year, and add a zero at the end of the field. =””&g120&”0″ generating the tab-delimited file we generated metadata by pasting a list of filenames into fields which then populated  all the fields we needed for metadata. this happened as the filenames were transformed by formulas using the right/left command and the aforementioned tricks for stringing together text, characters and extracted metadata. see figures 1, 2, and 3 above for a visual on how this worked. when we had irregular numbering and oddities, we had to hand-edit the metadata. this was done during the qa process when proofing image filenames. proofing the filenames was easier than we anticipated; mis-named files revealed themselves as we generated metadata. it was very easy to see where there were inconsistencies when the filenames were translated into numerical fields. we generate a tab-delimited file from the spreadsheet by taking the filenames of images, copying and pasting them into a column of the spreadsheet, and then selecting a worksheet within the spreadsheet which contains the manipulated raw data which has been rearranged to suit the needs of the tab-delimited file. the worksheet is then copied and the paste-special command is used to paste values-only into a new spreadsheet. the new spreadsheet is saved as a .txt file, in a tab-delimited format. we use this .txt file to upload metadata and filename info into contentdm. contentdm then takes the tab-delimited file, generates an index, and links between images and text, and creates a flat-file database that allows users to see images and text together or just plain text. contentdm also has an ocr function, that takes a .tiff file and converts it into a keyword-searchable file. in our case, we used excel to generate the tab-delimited file used for book-like documents (hierarchical). this format allows for images to be bundled together so “pages” follow each other in sequence when browsing, and groups of pages are linked together to give a searcher context when selecting pages out of a hit list from a search. possible improvements this process could have been more elegantly handled with any scripting language. one of the things that needed automation was the numbering scheme for when special pages appeared that didn’t get ordinary page numbers, like the table of contents, advertisements, plates (fancy pictures) and other irregular paging. the naming scheme for these pages was irregular due to the complexities of the pages themselves and to human error. it would have been nice to automate this aspect of page naming, but we couldn’t find an elegant way to do so, and so special page-naming issues were done by hand, albeit they were easy to identify as the scanning folks were instructed to give them letters after the page number, which tipped off whomever was data-wrangling. another issue that might be problematic in other computing environs is our file-naming convention. since the filenames are more than 8 characters long, they might present problems on different platforms and servers. at the moment, all we are using is a windows environment, but it is something to consider if you have a multi-platform situation. future applications this tool will enable you to generate metadata from any digital object. we have used a filename as the point of encoding, but it could be a unique id or any field. the encoding for this project was to reflect serial-like information – (title is given elsewhere in the metadata), issue, date, page number, but this method can be used for any situation where you have an object that has different facets that you’d like to express as potential access points for a search or browse function. it can also be used to generate metadata for any purpose. summary we were able to get by without a programmer and generate metadata for over 8000 images, using one person, on the fly. this technique can be used for any set of data where the data is highly patterned and represents anything that can be expressed numerically. it uses the right/left command in excel, or any spreadsheet, to extract specific parts of a filename and expresses that part as a numerical value. it uses different ways of stringing data and text together for creating online display information. the results were converted into a tab-delimited file and used to batch-upload text and images into a database. and it all started with the kindness of strangers, jane cullinane and glenn sherman. about the author jill strass has been in computing and libraries for 15 years. she’s worked in corporate, public, and academic libraries as a reference, cataloging and tinkering librarian. she’s managed projects involving the transference of print to online platforms. she’s developed and applied taxonomies. she’s worked with databases on both ends (both querying and building) and is especially interested in how systems fit together. visit the manitou messenger, a student newspaper for st. olaf college. it’s still fresh out of the contentdm box and is not a finished product. jill wrangled with metadata standards and contentdm to figure out how to get the newspaper online. she then created a tool to generate the metadata, supervised the scanning, and uploaded the records. author email: jillmeister@hotmail.com tags: excel, excel macros, metadata subscribe to comments: for this article | for all articles 3 responses to "generating metadata on a shoestring sans programmer, with our good friend, excel (or any spreadsheet)" please leave a response below: jennifer parsons, 2008-12-29 first of all, i enjoyed this article a great deal– thank you. secondly, i wonder if something like the metadata harvesting-and-parsing technique used here could be employed to help library and archival students better understand metadata. using a familiar program like excel can help ease people into the idea of how it is created, used, and reused. otherwise, too often metadata can be seen as a “black box”– a mysterious, hypertechnical discipline that is only peripherally related to what a librarian does. monica rivero, 2009-02-19 these are great tips. we use excel for data processing quite a bit as well. i find it very flexible especially for processes that are somewhat manual or ad hoc in nature and when a full script would be overkill. as an example of excel’s diversity, another way do to parse out data is to use the data>text to columns or use the vlookup formula to match up designated values for your parsed data. enjoyed reading it! andrew marlow, 2009-02-22 what you have done is very interesting. it confirms my belief that there is a desperate need to amend the metadata standards such as dublin core so that collections can be organised by journal, volume, issue and article. furthermore, this organisation is for the collections own sake whether or not there are any citations. in the few cases i have found where others have expressed this need the metadata is unfortnately expressed as citation data. i think journals, volumes, issues and articles are independent of citation. leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – deciphering journal abbreviations with jabbr mission editorial committee process and structure code4lib issue 7, 2009-06-26 deciphering journal abbreviations with jabbr jabbr is an online tool developed at cornell university to help users decipher journal title abbreviations. this article discusses why these abbreviations are so problematic, and how traditional tools are often insufficient, and then describes the novel approach used by jabbr. given an abbreviation, jabbr creates a regular expression for fuzzy matching, tests it against a list of serial titles extracted from the library catalog, and returns a list of possible matches to the user. jabbr is available as a web site and as a web service. by keith jenkins background at cornell university’s mann library, there is a species of question that inevitably finds its way to the reference desk: the journal title abbreviation. in order to find a cited article, one must first find the journal in which it appeared. for any given journal, there are several ways that its title might have been abbreviated in footnotes, bibliographies, and hand-written notes. it is often trivial to expand these abbreviations back to the full journal title, especially when the journal is well known. for example, most people would recognize “natl. geog.” as the popular national geographic, and medical researchers would immediately recognize jama as the journal of the american medical association. but researchers who are not already familiar with a particular title — maybe it’s a journal outside of their specialty — can have a difficult time deciphering the odd jumble of letters that comprise an abbreviation. it can be tempting to guess at unfamiliar journal titles. for example, one might suppose that “proc. am. soc. psych. res.” expands to “proceedings of the american society of psychological research“. however it may take some time and frustration to discover that the actual title uses the word “psychical” rather than “psychological”. foreign languages present additional challenges. for example, “rev. biol.” could be a reference to the portuguese-language revista de biologia, the spanish-language revista biología (published in cuba), or the french-language revue de biologie (published in romania). after helping a patron decrypt a journal abbreviation, the next question a reference librarian typically hears is “why couldn’t they have just spelled out the whole title?” indeed — why didn’t they? saving space seems to be the primary motive for abbreviating titles. a century ago, the publishers of large multi-volume print indexes such as the agricultural index were trying to use less paper. later, online databases were trying to reduce data storage costs, and speed up searches. the abbreviation habit continues today. a good description of what an abbreviation tries to achieve is provided by frank rogers and thelma charen: the ideal journal title abbreviation is first of all brief; then it is clear, unique, informative as to the language of the original, indicative as to scope of the publication, easily remembered and used, and universally applicable. the actual must often fall considerably short of the ideal; the various elements are at war with each other [1]. despite the inherent difficulties, several attempts at standardization have been made, including the z39.5 american standard for periodical title abbreviations [2]; the iso 4 international code for the abbreviation of titles of periodicals [3]; and the iso 833 international list of periodical title word abbreviations [4]. such standardization certainly provides guidance to those doing the abbreviating, but it doesn’t necessarily help those trying to interpret the abbreviations. as rogers and charen note, an abbreviation should be self-explanatory, but “it need not, however, be in a form which can serve as a base for verbatim reconstruction of the title” [1]. furthermore, these abbreviation standards are routinely ignored, especially in the cases of acronyms and hand-written notes. there is no shortage of publications that refer to themselves by the decidedly non-unique name “cq”: caribbean quarterly, communication quarterly, concrete quarterly, congressional quarterly, ad infinitum. and i know of no scholar who consults the relevant iso standards when hastily jotting down the citation of an newly-discovered article. over the years, various lists of journal title abbreviations have been compiled. mary kinney [5] has noted that such compilations are more common in the scientific and technical fields, in which titles are abbreviated more often than in the social sciences and humanities. kinney goes on to list numerous sources for the identification of abbreviated titles, helpfully organized by subject. many of these sources can also be found in the library catalog by searching for the subject subdivision, “abbreviations of titles”. for a collection of online sources, there is gerry mckiernan’s all that jas: journal abbreviation sources [6], which links to over one hundred separate abbreviation lists, organized by discipline. such tools are useful, but the idea of sequentially searching several different sources to find one abbreviation is not an appealing one. jabbr in late 2004, having repeatedly encountered the journal abbreviation problem at the reference desk, i started thinking about a different approach to the problem of deciphering journal abbreviations. what patrons really seemed to want was a webpage where they could simply type in their perplexing journal abbreviation, click a button, and get the full title. how might something like this work? given an abbreviation input by the user, a program could iterate through a list of journal titles, apply some sort of fuzzy matching, and return a relatively short list of possible matches, from which the patron could select the best match. this would take the patron to the full catalog record for the item, where publication dates and volume numbers could be used to verify the title. being unaware at the time of anything with a similar name, i decided to call the program “jabbr”, which is, of course, an abbreviation for “journal abbreviations”. below is a screenshot of the jabbr website [7]: screenshot of jabbr the first step was to find a list of every journal ever published anywhere. since that was clearly impossible, i settled for extracting all the serials records in the cornell university library catalog. this amounted to nearly 300,000 marc records representing not just journals, but also proceedings and other serials whose titles might often be abbreviated. title information can be found in many different fields of the marc record, including the 210 field, which is defined as the “abbreviated title”. (while this field sounds quite promising, not all serials records contain this field. and even when it is present, it doesn’t necessarily include every possible abbreviated form of the title.) the current version of jabbr uses titles extracted from the following marc fields: 210 “abbreviated title” (subfield a) 245 “title statement” (subfields a, b, n, p) 246 “varying form of title” (subfield a) 247 “former title” (subfield a) the data from these fields has been extracted, normalized, and reorganized into a tab-delimited file structure that is more efficient for pattern-matching. each record is represented on a separate line containing the main title (from the marc 245) followed by various alternate, normalized titles that come from any subfields 245a, 210a, 246a, or 247a that are present in the marc record. thus, the pattern is: main title [tab] normalized 245a [tab] normalized 210a [tab] normalized 246a [tab] etc. the normalization function performs the following steps: remove diacritics replace periods, commas, semi-colons, exclamation points, slashes, and underscores with spaces remove other non-alphanumeric characters collapse multiple spaces and trim whitespace from the beginning and end convert to lowercase multiple records for the same journal (for print, online, etc.) are merged whenever their main titles are identical. thus the 300,000 marc records (~352 mb) were reduced to about 240,000 lines (~27 mb) in the tab-delimited text file. the number of alternate titles that appear on each line varies depending on the number of title fields present in the original marc records, as shown in the following example: journal of applied behavior analysis [tab] journal of applied behavior analysis [tab] j appl behav anal [tab] jaba journal of applied biobehavioral research [tab] journal of applied biobehavioral research [tab] j appl biobehav res journal of applied biochemistry [tab] journal of applied biochemistry jabbr takes the user’s input string, transforms it into a complex regular expression, and then tests it against each line of this tab-delimited file. if the pattern matches, then the title is displayed to the user. if the match occurs in an alternate title but not in the main title, then the matching alternate title is also displayed. it took some experimentation to develop good regular expressions. when testing the effect of various adjustments, i found that relaxing the strictness of the match too much could easily result in a flood of false hits, and an overly complex expression could make the matching process intolerably slow. acronyms needed to be recognized, even where multiple words in the title have been dropped, such as when foreign agricultural trade of the united states is abbreviated as fatus. and, although most words get abbreviated by having letters dropped from their ends, jabbr also needed to account for cases such as “natl” and “engr”, where the final letter is retained. a simplified version of the matching function, written in php, appears below. (code can be requested from the author via the jabbr website [7].) function findtitles($q) { $x = normalize($q); # acronym? if (preg_match("#^\w+$#", $x, $m)) { $acronym = chop(preg_replace("#(\w)#", "\b$1\s*( \w*){0,2} ?", $x)); } else $acronym = ""; # look for terms only at beginning of words $x = preg_replace("#(\b\w+)\s#", "$1[^\\t]* ", $x); # but accommodate natl, engr, boln, etc. $x = preg_replace("#(\b\w+)(\w)\b#", "$1\w*$2", $x); $matches = array(); foreach (file("jabbrdata.txt") as $line) { if ( preg_match("#\t({$x}[^\t\n]*)#i", $line, $m) || $acronym &amp;&amp; preg_match("#\t({$acronym}[^\t\n]*)#i", $line, $m) ) { $title = preg_replace("#\t.*$#", '', $line); $matches[] = array('title'=>$title, 'alttitle'=>$m[1]); } } return $matches; } as an example, let us assume that that the user has typed “j ap beh”. the findtitles() function manipulates the input string $q and assigns to $x the following regular expression: \bj[^\t]* a\w*p[^\t]* be\w*h the foreach loop tries to match this expression against each tab-delimited title in each line of the data file, and stores any matches in the $match array. in this particular example, the results contain five possible matches, including the likely titles journal of applied behavioral science and journal of applied behavior analysis. improving jabbr’s performance if the user query looks like it might be an acronym (i.e., it has no spaces) then an additional regular expression $acronym is generated and tested as well. for example, a search for “jacs” assigns to $acronym the following regular expression: \bj\s*( \w*){0,2} ?\ba\s*( \w*){0,2} ?\bc\s*( \w*){0,2} ?\bs\s*( \w*){0,2} ? this regular expression allows for up to two words (such as “of the”) to go missing between each word that is actually represented in the acronym. thus we can find such titles as journal of the american chemical society, journal of the american college of surgeons, journal of the australian ceramic society, etc. testing complex regular expressions against every line in a 27 mb file can take several seconds to run. while this is obviously much faster than manually checking multiple print or online sources, an even shorter response time was desired. in order to make jabbr faster, the tab-delimited text file was split into separate files for each letter of the alphabet. in this way, a query that begins with the letter “a” would only have to churn through the subset of data for journals that begin with that letter, effectively cutting down the average search time to about 1/26th of the original. as a result, most searches now retrieve results in a fraction of a second. for particularly cantankerous abbreviations, the current version of jabbr also has a “deep search” option that will look through all the titles, regardless of the initial letter. the deep search also allows for fuzzier matches (doubling the number of non-abbreviated words in acronyms, for example). the method of constructing regular expressions has changed slightly over time, as jabbr-stumping examples have been discovered and reported by reference staff. as result, a few special exceptions have been made, such as “jrnl” or “jrl” as abbreviations for “journal”, and “bks” for “books”. in these cases, implementing a more general change in the regular expressions resulted in far too many false hits. so instead, these specific abbreviations are handled before the rest of the regular expression is constructed. as more exceptional cases are encountered, they too can be written into the code. one recent enhancement was to boost exact matches to the top of the results list. for example, a search for “pnas” exactly matches an alternate title that comes from the marc 210 field (“abbreviated title”). recognizing this exact match, jabbr places proceedings of the national academy of sciences of the united states of america first in the results list. note that, sometimes, there can be more than one “exact” match, as in the case of cq mentioned earlier. jabbr web service since its beginnings in 2004, the jabbr website has featured a match-as-you-type capability, displaying matches even while the user is still typing. initially, this was accomplished with hand-coded javascript xmlhttprequests and a local text-based web service, but the javascript code for the jabbr website was recently rewritten to use the jquery javascript library [8], which has greatly simplified the matter of fetching results from the server and updating the page. the website now uses jquery.getjson() to fetch matches from the json [9] web service available at http://supportingcast.mannlib.cornell.edu/jabbr/json. although designed specifically for use within the jabbr website, which links results into the cornell library catalog, the json web service provides a generic response that may be of benefit to other developers. the web service accepts the following querystring parameters: parameter required? description q yes the abbreviation to be deciphered max no maximum number of results (defaults to 20) deep no set to 1 for a deep search (defaults to 0) as an example, the request: http://supportingcast.mannlib.cornell.edu/jabbr/json?q=javma … returns the following json object: { "q": "javma", "max": 20, "deep": 0, "count": 4, "matches": [ { "alttitle": "javma", "exact": 1, "title": "journal of the american veterinary medical association" }, { "alttitle": "journal of the american holistic veterinary medical association", "title": "journal" }, { "title": "journal of anhui vocational college of metallurgy and technology" }, { "title": "journal of the arab veterinary medical association" } ] } the values of the q, deep, and max members simply echo the parameters that were requested, and count reports how many matching titles were found. the value of count may be greater than max, but no more than max matches will be returned. the value of the matches member is an array of objects. every object in this array will contain a title, and some objects may also include alttitle or exact members: the alttitle is only included when it was used to make the match, and the presence of the exact member indicates an exact match, as described above. because jabbr uses the cornell library catalog as the source of its titles, it can currently only find matches when the titles are held by cornell. although cornell does have extensive serial holdings, this is clearly a limitation, and it would be useful to augment the list of titles. possible sources include existing serial titles lists and other library catalogs. (any interested parties are invited to contact the author.) conclusion inevitably, jabbr does return some false hits. in fact, it almost always does. however, the goal is not to provide a single result that is guaranteed to be correct. rather, jabbr aims to provide a short list of titles that can be quickly scanned to find the title that the abbreviation most likely represents. thus, it is not intended to be used for programmatically converting a list of cited abbreviations, but is designed to assist a human user to interpret a single abbreviation. human intervention is required. in january 2006, we added a “journal title abbreviation” option to the main search page of our voyager-based library catalog. when a user selected this option, the search would be diverted to a customized version of jabbr that was disguised to look like a regular catalog page. clicking a title linked back into the local catalog, with voyager session information intact. in june 2009, worldcat local became our library’s official public catalog, so the results on the jabbr website now link to a serials search within worldcat. we have been using jabbr in the libraries of cornell university for several years now, and it has repeatedly proved its usefulness. over the past three years, we have seen an annual average of over 25,000 jabbr searches originating from the library catalog. and back at the reference desk of mann library, which focuses on the life sciences, jabbr has been especially helpful for tracking down obscure foreign journals and conference proceedings in the biological and agricultural sciences that otherwise could not be found. references [1] rogers, frank b., and thelma charen. “abbreviations for medical journal titles”. bulletin of the medical library association 1962 july; 50(3): 311-352. http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=197853 [2] z39.5-1963. american standard for periodical title abbreviations. approved november 20, 1963 by the american standards association. [3] iso 4-1972. documentation: international code for the abbreviation of titles of periodicals. switzerland: international organization for standardization. [4] iso 833-1974. documentation: international list of periodical title word abbreviations. switzerland: international organization for standardization. (this standard has since been withdrawn.) [5] kinney, mary r. the abbreviated citation: a bibliographical problem. acrl monograph no. 28. chicago: american library association, 1967. [6] all that jas: journal abbreviation sources. http://www.abbreviations.com/jas.asp [7] jabbr: decipher your journal abbreviations. http://supportingcast.mannlib.cornell.edu/jabbr/ [8] jquery: the “write less, do more” javascript library. http://jquery.com/ [9] json (javascript object notation). http://json.org/ about the author keith jenkins is currently the gis/geospatial applications librarian at cornell university’s albert r. mann library. he previously held the position of metadata librarian at the time he originally developed jabbr. subscribe to comments: for this article | for all articles 3 responses to "deciphering journal abbreviations with jabbr" please leave a response below: yitzchak schaffer, 2009-06-28 great technique. how much usage does this tool get? did you do any scalability testing? keith jenkins, 2009-06-30 hi, yitzchak. jabbr has averaged about 25,000 searches annually, in small bursts of activity every few minutes, which is pretty light traffic by web standards. at that rate, scalability hasn’t yet been an issue. but i’m continuing to track usage, and will contemplate further optimizations if (and only if) necessary. keith jenkins, 2009-06-30 source code is now linked from the bottom of this page: http://supportingcast.mannlib.cornell.edu/jabbr/about leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – toward element-level interoperability in bibliographic metadata mission editorial committee process and structure code4lib issue 2, 2008-03-24 toward element-level interoperability in bibliographic metadata this paper discusses an approach and set of tools for translating bibliographic metadata from one format to another. a computational model is proposed to formalize the notion of a ‘crosswalk’. the translation process separates semantics from syntax, and specifies a crosswalk as machine executable translation files which are focused on assertions of element equivalence and are closely associated with the underlying intellectual analysis of metadata translation. a data model developed by the authors called morfrom serves as an internal generic metadata format. translation logic is written in an xml scripting language designed by the authors called the semantic equivalence expression language (seel). these techniques have been built into an oclc software toolkit to manage large and diverse collections of metadata records, called the crosswalk web service. by carol jean godby, devon smith and eric childress 1. who uses interoperable records? the winters are gloomy in columbus, ohio, so susan decides to take an online course on memoir writing. after reviewing one of her homework assignments, her instructor emails her late one night with the recommendation that she take a close look at how homer hickam, in his celebrated book rocket boys, manages to craft gritty but lyrical descriptions of his boyhood home, a coal mining town in a remote valley of west virginia in the 1950s. susan needs this book tomorrow morning, so she wonders if she can get the book from a nearby library. she goes to worldcat.org [1] and types the title into the search box. the top-ranked result shows that the book is available from worthington public library, just ten miles away from her house. this paper describes one component in a complex process that enables worldcat.org to deliver information like this to library patrons. how does this service, which can be accessed from the internet anywhere in the world, recognize that a library close to susan has the book she wants? at the start of the process, third-party jobbers obtain orders from a collection of libraries, which they fill by placing a bulk order to an online bookseller. the worthington library’s copy of rocket boys and the other books in the same order are located and shipped from the bookseller’s warehouse using the information in a record coded in onix [2], the relatively new but emerging de-facto standard for descriptions of materials handled by the publishing industry. the jobber’s order is also sent to oclc to determine whether it can be matched with the corresponding marc [3] records in the worldcatâ® database [4]. a software module translates some of the elements in the onix record to marc and searches for it. since rocket boys has already been cataloged, the worldcat record is updated with a flag indicating that the book will soon be held by the worthington public library. worldcat.org now has the essential information required to connect susan with her book. 2. translating bibliographic metadata: a short progress report the above scenario is an example of how the library community interacts with publishers to deliver value to patrons. the workflow isn’t particularly complex, nor is the relationship between libraries and publishers especially problematic because the negotiations are about well-understood resources that must simply be transferred from one community to another. nevertheless, the achievement of this goal requires that two descriptive standards be synchronized. both are sophisticated. both are entrenched in their respective communities of practice and both are necessary because they annotate the outcomes of events involving the book in two different contexts. the onix standard originated from the need to identify and track books as they are bought and sold. the transaction typically involves a bookseller web site such as amazon.com, which links a physical item to a machine-readable description and associates it with ancillary materials, including web sites, study guides, cover art, tables of contents, reviews, or digitized pages, all used to create a rich context for interacting with the book before and after a purchase. the marc standard evolved from the need to create an authoritative description establishing the authorship, subject, and provenance of a work that can be interpreted as an archival record with many anticipated uses in the library community. the marc record has relatively few links to commercial transaction data or to elements required to populate a given web site, while the onix record is less rich in links to name and subject authorities. but the two descriptions are not incompatible. 2.1. the crosswalk. in fact, there is considerable overlap because the onix and marc descriptions of rocket boys refer to the same book. but it takes the effort of an expert familiar with the formal specifications as well as the normal use cases of both standards to tease out the commonalities and record them, typically in a table of correspondences called a crosswalk [5], such as a sample of the onix-to-marc mapping maintained by the library of congress [6] shown in table 1: title composite <title> the title composite contains the text of a title, including a subtitle when necessary. <b202> <titletype> (used to set field 246 2nd indicator: if b202 = 00, 246 i2 = #; if b202 = 01, 246 i2 = 2) <b276> <abbreviatedlength>   <b203> <titletext> 246 $a <b030> <titleprefix> 245 $a <b031> <titlewithoutprefix> 245 $a <b029> <subtitle> 245 $b; 246 $b table 1. a fragment of an onix to marc crosswalk if the goal is to share records between libraries and publishers, clear correspondences between onix and marc must be established among elements that identify authors, titles, publishers, dates of publication, numeric identifiers, editions, physical formats, summaries, tables of contents, or subjects. it is less clear what to do with descriptions of the bundled items, such as associated web sites and user reviews that can have their own complete bibliographic descriptions. but the easy correspondences are good enough to conduct much of the day-to-day work in digital libraries and library support organizations, including record creation, normalization, and enhancement; database searches and retrievals; and the automatic generation of rough drafts for archival records that would be completed by human experts. nevertheless, we have discovered that even the slightest incompatibilities among record formats produce huge backlogs that translate to unfulfilled queries from users. consider a few elements from some of the major standards for representing bibliographic information that name primary book titles. in the marc to dublin core crosswalk, 245 $a maps to dublin core (dc) title, a relationship that persists through xml, asn.1/ber[7], rdf [8], and various standard or non-standard human-readable displays. it also persists through multiple versions and editions of both standards, such as oai-pmh dublin core [9], marc-xml [10], marc 2709 [11], and versions 1.1 and 1.2 of unqualified dublin core (or dc-simple) [12], as well as dublin core terms [13] (or qualified dc). the mods standard can also be reasonably added to this mix. mods [14] is a slight simplification of marc designed for the description digital library resources, whose title concept, <titleinfo><title>, is also mapped to 245 $a and dc:title, according to crosswalks available from the library of congress. marc 245 $a also maps to several onix elements, among them <distinctivetitle>. in addition, crosswalks are being developed between dublin core and onix, which maps dc:title to <distinctivetitle>. in sum, crosswalks exist between marc and dublin core, onix and marc, onix and dublin core, mods and marc, and mods and dublin core. for a single concept, perhaps corresponding to an educated layman’s understanding of title, the relationships among the standards are consistent and easy to implement. but in its current form, this information is difficult to use for automated processing. it requires corresponding technical resources such as xml schemas and xslt scripts, which are all too often unavailable, broken, or obsolete. the relative scarcity of machine-processable files implies that it is difficult to transform this content into an executable format and maintain it. but perhaps we need to acknowledge that the antecedent step of intellectual analysis is also difficult. it is a nontrivial task to determine whether existing crosswalks are sufficient to perform a translation, or how they can be modified or enhanced. if crosswalks need to be developed, they are often created from scratch even though they might be algorithmically derived. these problems imply a need for a better formal model of the crosswalk: a model that makes it easier to do the intellectual analysis, one that more closely associates the executable files with the analysis, and one that encodes a realistic proposal for managing different versions and encodings of standards. 2.2. toward schema-level interoperability. using the terms defined in chan and zeng’s progress report on interoperability [15,16], the problem we have defined is in the scope of schema-level interoperability because we are trying to identify the common ground among formally defined metadata schemas independently of the short-term need to process instance records. but our goal is a conceptual framework that will make this job easier because we want a solution that can be reused for different kinds of tasks and applications. chan and zeng identify several conceptual objects that have emerged from the effort to achieve this kind of interoperability. in addition to the crosswalk, which is the focus of our work, they also describe derivations, switching-across schemas, and application profiles, all of which have a place in the solution outlined in subsequent sections of this article. a derivation results when a new schema is created from an antecedent, as when mods was derived from marc to create a standard that is more appropriate for software systems that manage descriptions in digital library collections. a switching-across schema is a common format, which we call an interoperable core in a hub-and-spoke model [17], into which multiple schemas can be mapped to create a highly efficient processing model. and an application profile is a hybrid schema that is created by combining elements from more than one schema, observing certain restrictions on how they are used [18]. application profiles achieve interoperability by reusing elements defined in existing schemas and creating new elements only when necessary. for example, dublin core terms is an application profile that extends the dublin core element set with a rich set of vocabulary for describing relationships, dates, and audience levels, among other concepts. crosswalks, derivatives, hub-and-spoke models, and application profiles respond to the need to identify common ground in the complex landscape of resource description. but these objects also imply an unresolved tension between the need to minimize the proliferation of standards and the need to create useful machine-processable descriptions of resources. we concentrate on the crosswalk, partly because of the institutional and technical context of our work, but also because we are primarily interested in devising reusable solutions by identifying and exploiting the common ground in metadata standards. but we also believe that the crosswalk is more fundamental because it underlies two of the objects that chan and zeng describe: the derivative, which requires a crosswalk to map elements from the parent schema to the child; and the hub-and-spoke model, which requires a crosswalk to map related schemas to an interoperable core. but the crosswalk is built up from an even more fundamental unit of analysis: a single element defined in a standard, such as title in dublin core, marc, mods, and onix. like words in a natural language, metadata elements can be extracted from one context and combined into new ones, forming application profiles. they can also enter into semantic relations, such as the equivalences defined by crosswalks, which resemble synonymy. when the focus is on the element, chan and zeng’s schema-level interoperability reduces to the problem of element-level interoperability, whose technical infrastructure is a baseline on which more comprehensive systems can eventually be built. and as our opening scenario shows, commercially useful work can be done here, work that is now being done redundantly and wastefully, if at all. here we describe a generic computational model that permits someone with an expert’s knowledge of the metadata standards used in libraries, museums, education, the publishing industry, or other contexts that have well-defined data models, to develop executable translations by creating a list of equivalences that looks like the crosswalk shown in table 1. the model is structured so that the standards expert can concentrate on mapping equivalent elements, letting the software handle the messy details that emerge from multiple structural realizations, such as those that come up when the title concept is identified in mods, marc, dublin core, and onix. because the model closely resembles the human-produced mappings, relatively little programming is required to make this data actionable—in fact, the process of converting crosswalks to executable code can be largely automated. once the system is seeded with the most important crosswalks, it can run with minimal human intervention. only minor tweaks are required to add new standards or accommodate new versions or translations. the system described below is the core logic of the crosswalk web service, which represents an outcome of the research program first reported in godby, childress and smith [17], and childress, godby and young [19] and is currently being used to power the metadata crosswalking facility in oclc connexionâ®, as well as many offline services that process larges sets of records. a publicly accessible demo will soon be available from oclc’s researchworks page [20]. the crosswalk web service is part of a larger effort to re-engineer oclc’s products as a collection of modular, configurable services for harvesting, validating, enhancing, and manipulating descriptions of resources, many of which are engineered from open source software and exposed through web service interfaces. the result is an architecture that reduces internal duplication of effort and exposes critical functionalty to any partner, client, or member that recognizes a common set of standards and can be accessed through an internet connection. 3. a process model for record translation figure 2 depicts the major steps in the crosswalk web service process model. it is designed to separate the semantics from the syntax of the translation task, from which many desirable consequences follow, including reusable components and simplified translation scripts, as we argue below. but the most important consequence is that the task of creating and interpreting crosswalks remains abstract, so the metadata subject matter expert can concentrate on defining mappings without getting bogged down by complex structural detail. we will illustrate the process by walking through the translation of a simple description of rocket boys that starts out as an onix record (shown in figure 5). figure 2. the translation model [view full-size image] 3.1. a brief overview. input records conforming to the source metadata standard at state a are converted to a common xml structure at step 1 and are then translated to the semantics of the target metadata standard at step 2. the result of the translation is a set of normalized xml records at state c, which are converted to the native structure of the target metadata schema at step 3, producing output records at state d. only the first and last steps of the translation model have access to the syntactic details of the native standards. in the internal step 2, the translation operates on a common, locally defined structure. 3.2. step 1: reading the input. a utility program, informally called a reader, converts an input metadata record to the internal structure required by the translation software in our model. this structure is an xml container we have named morfrom, whose document type definition contains four elements: record, the top-level element; header, which identifies a default namespace for the record and links to a resource containing a definition of the elements that populate it; field, whose name attribute identifies an element in the native standard; and value, which defines a path to the data. the field elements preserve the element order, sequence, and hierarchical structure, while the name attributes on field preserve the names of the elements in the native record. finally, the value elements enclose the unique data in the record. figure 3 shows the result of running the onix reader on our sample record. as shown, this record contains author, title, and subject elements that will be translated. by convention, morfrom records are designated with the .mfm suffix. figure 3. the input record in native onix syntax and morfrom [view full-size image] 3.3. step 2: the translation. the key component of the processing model is step 2 in figure 2, where the translation takes place. the translation process has two inputs: the translation logic, and a morfrom-encoded record in the source metadata standard, i.e., the record containing onix elements shown in the bottom portion of figure 3. the translation logic is written in an xml scripting language we have designed and named seel, an acronym for semantic equivalence expression language. it is processed by a custom-written interpreter, which performs the essential function of xslt or other easily written scripting languages, but represents a computational model of the crosswalk that keeps the defining feature, the assertions of equivalence, in the foreground. the seel language specification and interpreter bear some resemblance to the jumper package [21], but seel has more features for expressing fine-grained details of conditional translations, some of which we describe below. figure 4 shows a complete seel script that can process everything but the subject elements in the sample input, which are treated in figure 5. a seel script is enclosed in the <translation> element, which has two children. the <header> specifies the metadata standards in the source and target of the translation. one or more <map> elements follow, each of which contains roughly the same information as a row in a crosswalk: an element defined in the source metadata standard and the equivalent element from the target. like a row in a crosswalk, a seel map is modular, self-contained, and abstracted away from the native syntax of the standard. and like a crosswalk, a seel script can either be lightweight or comprehensive, depending on how many maps it has. the script in figure 4 maps onix <title><titletext> to marc 245 $a and the first occurrence of onix <contributor><personname> to marc 100 $a. <?xml version="1.0" encoding="iso-8859-1"?> <!-dc:source = http://www.loc.gov/marc/onix2marc.html --> <translation> <header> <sourceschema name="onix" namespace="uri:ns:onix-book:2.1"/> <targetschema name="marc" namespace="uri:ns:marc:21"/> </header> <map id="1"> <source> <mainpath> <branch><step name="title"/><step name="titletext"/></branch> </mainpath> </source> <target> <mainpath> <branch><step name="245"/><step name="a"/></branch> </mainpath> </target> </map> <map id="2"> <source> <mainpath> <branch><step name="contributor" position="1"/><step name="personname"/></branch> </mainpath> </source> <target> <mainpath> <branch><step name="100"/><step name="a"/></branch> </mainpath> </target> </map> </translation> figure 4. a simple but complete seel script in more technical terms, seel maps the elements that might correspond to a concept of title in the two standards using the information in the seel <mainpath><branch> element, which identifies a path in the source record and constructs an equivalent path in the target. once the paths are aligned, the data involved in the translation is transferred from the markup of the source to that of the target. to accomplish this task, seel has a path language that can refer to any location in a source record encoded in morfrom. each seel element <step name=’abc’> picks out a morfrom element <field name=’abc’> and defines one step on a path from the record root to the data. for example, the effect of the first seel map in figure 4 is to locate the fragment <field name=’title’><field name=’titletext’> in an onix-tuned morfrom record and construct the path <field name=’245′><field name=’a’> in the corresponding morfrom marc record. figure 5 shows the seel map for the subject elements. it is separated out here for the sake of discussion, but in our production environment it would be appended to the other maps in figure 4 to create a single onix-to-marc script. this map is more complex than the maps in figure 5 because it has a conditional. its effect is to map the onix <subject><subjectheadingtext> element to a library of congress subject heading in marc 650 $a if the onix source has a sibling <subjectschemeidentifier> whose value is 4, the onix code for lcsh in this context. this map also builds a marc i2 field and populates it. all of this work is done by the <context> element, which has two essential sub-elements: a <path>, which describes a path relative to path described in the corresponding <mainpath> element; and a <value>, which contains data. in the <source>, the <context> element implements the ‘if’ part of the statement given above by locating the subtree containing the ‘4.’ the ‘from’ attribute on the <path> uses the standard unix subdirectory idiom to address any path in the record, and here specifies that <subjectschemeidentifier> is a sibling of <subject><subjectheadingtext>, which can be verified by checking the morfrom record shown in the bottom portion of figure 3. a <context> can also appear in the <target>, where it is interpreted as an instruction to build an additional path, a child of the root defined in <mainpath>. in the map shown in figure 5, the <context> element is used to create an indicator element and assign appropriate data to it, resulting in the <field name=’650′> element shown in the morfrom record in the top portion of figure 6. the map shown in figure 5 is actually a fragment of a larger map that would build marc subject elements with different tag names and internal structures depending on whether the onix subjectschemeidentifier code is lcsh, mesh, or a publisher’s unique scheme. in the full script, there would be multiple <branch> and <context> elements, which would be bound to one another with the branch id (‘bid’) attribute. but a single branch is sufficient to translate the sample record. <map id="3"> <source> <mainpath> <branch bid='1'><step name="subject"/><step name='subjectheadingtext'/></branch> </mainpath> <context bid='1'> <equals> <path from= '..'><step name='subjectschemeidentifier'/></path> <value>4</value> </equals> </context> </source> <target> <mainpath> <branch bid='1'><step name='650'/><step name='a'/></branch> </mainpath> <context bid='1'> <equals> <path><step name='i2'/></path> <value>0</value> </equals> </context> </target> </map> figure 5. a seel map with a context 3.4. step 3: writing the output. the output of the translation is a second morfrom record containing elements defined in the target metadata standard. the morfrom marc version of rocket boys is shown in the top portion of figure 6. to complete the translation process, the record must be converted to the syntax of the output by passing through a utility program called a writer. this happens at step 3 in the translation model shown in figure 2. because the client requests iso 2709 syntax in our example, the record is sent to the iso marc writer. the result is the record shown in the bottom portion of figure 6. the leader elements shown in the iso record are defaults generated by our production system, but they can also be created by seel maps when the need arises. figure 6. a morfrom record written to iso marc [view full-size image] 3.5. readers and writers: a recap. we have walked through a simple example showing the translation of an onix xml input to a marc iso output to demonstrate that the crosswalk web service model is flexible enough to handle xml, as well as non-xml inputs that are formally defined and consistent, a necessity in an unstable resource description environment such as oclc’s. the readers and writers invoked at steps 1 and 3 require custom programming, but they repay the effort by delivering much-needed abstraction. these programs, typically written in java, hide the complex and sometimes idiosyncratic details of hierarchical structure, element order, and internal element syntax from the rest of the model. and since these are the very details that are least likely to change as standards evolve, the readers and writers can be written once and reused extensively. the crosswalk web service now has a collection of readers and writers that operate on or generate morfrom marc for iso 2709 marc, marc-xml, marc-8 [22], and locally designed marc-like formats. since these programs also make java‘s library of character sets available to the translation model, the service performs hundreds of distinct conversions from one syntax and character encoding to another, a nontrivial accomplishment that, left unaddressed, results in a large backlog of unprocessed records in our production environment. the syntactic normalization step also simplifies the translation. as figures 3 and 6 show, the morfrom representation eliminates many superficial differences between onix and marc and hence the complexity of the rules required to map the two standards. and only one set of maps is needed because the readers and writers handle all structural variation. thus the same map would work for marc xml or marc iso-2709 to onix or dublin core terms. this reusability comes at the expense of some extra processing when the native record syntax is xml, but the cost is minor, given the benefits. 3.6. the translation: why not do it in xslt? this is a good place to contrast the seel translation engine with the the most popular alternative, an xslt stylesheet that translates directly from the source to the target. figure 7 shows a fragment of an xslt stylesheet that constructs marc 100 1# $a from onix <contributor><personname> and does some of the same work as the seel script shown in figure 6 [23]. this example is derived from the onix-to-marc stylesheet maintained by the library of congress [24], which we have simplified slightly for the sake of discussion. <xsl:for-each select="contributor&#91;personname&#93;"> <xsl:call-template name="datafield"> <xsl:with-param name="tag">100</xsl:with-param> <xsl:with-param name="ind1">1</xsl:with-param> <xsl:with-param name="ind2">#</xsl:with-param> <xsl:with-param name="subfields"> <subfield code="a"> <xsl:value-of select="."/> </subfield> </xsl:with-param> </xsl:call-template> </xsl:for-each> figure 7. part of an xslt stylesheet for the onix-marc relationship this stylesheet is a set of instructions for building a marc xml record that contains properly formatted fields and subfields in the required marc sort order. accordingly, the instructions for building and populating the 100 field shown here are located in the full script between those that create the control fields and those for the higher-ranking variable fields in a valid marc record. since the record-building task is performed simultaneously with the record-translation task, the semantic elements that would be isolated and brought together in a seel script are interleaved with statements that specify structural details. compared to our model, an xslt script like that shown in figure 7 is far less abstract because it combines the work performed by a seel script with that done by the readers and writers. of course, an additional process would be required to produce non-xml output in a model that depends on xslt to translate metadata records. and unlike the corresponding seel script, the xslt stylesheet is inherently asymmetric because the source and the target paths to the data have qualitatively different encodings. the source path is defined using xsl xpath [25] elements, which identifies the onix tag of interest in line 1 and transfers the data at that node to the target record in line 8. but the target path is defined more simply by a set of hard-coded xml elements obtained from the target record specification. for example, line 3 establishes the root of the target path by passing a value to an xsl template that produces the element <datafield tag=’100′>. line 7 extends this path with the element <subfield code=’a’>, a relationship that is not explicitly labeled, as it would be in a seel path statement, but is inferred from the fact that this line is executed immediately after line 3. by contrast, a seel map is unambiguous because everything is fully labeled with elements defined in the seel document type definition. because a seel map is explicitly labeled and symmetric, it can be algorithmically reversed by transposing the contents of the <source> and <target> elements. in the one-to-one maps shown in figure 4, this relationship is perhaps obvious, but it is also true of the more complex map in figure 5. in the reverse translation, a marc 650 $a containing a second indicator with the value ‘0′ triggers the construction of the onix <subject><subjectheadingtext> and builds a sibling <subjectschemeidentifier> element with the value ‘4.’ we are still fine-tuning the reversibility feature of seel, but it has already been helpful in the development and evaluation of round-trip translations. 4. fine-tuning the translation step our walkthrough of the crosswalk web service translation model in the previous sections featured a translation from onix to iso 2709 marc. this is one path through our system, but there are many others. to promote reusability, the crosswalk web service implements a hub-and-spoke model, which sometimes requires two translation steps: one to map the input to a core format, and a second to map the core format to the desired output. in our model, a core format has no literal existence either as a software implementation or as a data structure. it is nothing more than an agreed-upon convention to write one set of seel scripts with a common target (the to-scripts), and a second set with a common source (the from-scripts). in a service designed to meet the needs of the library community, a core format based on marc produces certain efficiencies. these savings are achieved because the translations are divided into to-marc inputs and from-marc outputs. when a new to-marc input such as mods is introduced, it has automatic access to the existing collection of all from-marc outputs. as a result, once the collection of inputs and outputs grows even moderately, the hub-and-spoke model eliminates considerable programming effort. for example, the baseline version of the crosswalk web service has the following outputs: four syntactic encodings of marc and locally defined marc-like extensions, five versions and two syntactic encodings of dublin core, two versions of mods, and one version each of onix-books and onix-serials. when we added the mods-to-marc input, all of these outputs became available, even translations from mods to onix and mods to dublin core terms, for which metadata experts have attempted to write crosswalks, perhaps unnecessarily, because many of the relevant mappings can be inferred. this model can easily cut through the problems surrounding the translation of title we discussed in section 2. figure 8 shows a hypothetical hub-and-spoke translation model that maps a single element and depicts the syntactic encodings, represented as black tabs, that would be handled by the readers and writers. in all, just six maps would be required: three to-marc maps and two from-marc maps. but the model can handle 49 different combinations of inputs and outputs that could be found in a request to translate a set of metadata records to a target standard and encoding. of course, the full crosswalk web service has a collection of comprehensive scripts that map hundreds of elements. but figure 8 would accurately depict the mapping relationships in the full system if all references to the ‘title’ elements were omitted. figure 8. a hub-and-spoke translation model for the ‘title’ concept. [view full-size image] inefficiencies result from thinking about the metadata mapping problem in terms of pairwise translations that would have to be written for every input and output format, each of which requires a corresponding crosswalk—for example, mods to marc 2709, mods to marc-xml, mods to onix books, and so on. in this model, software cannot be reused as the collection of translations grows. our model is more efficient because it separates the semantics of the problem, the assertions of equivalence between pairs of elements—from the syntax, a potentially large variety of structural realizations that must be decoded at the start of the process and reassembled at the end. 5. seel maps in a persistent resource a set of seel maps renders a crosswalk into a computable form through statements in a declarative language. the relationships of interest are assertions of semantic equivalence, which represent the judgment by metadata standards experts that pairs of author, title, and other elements in two standards are synonymous, or at least close enough that their differences don’t affect the outcome of a translation task. since a seel map is fully labeled, it can be interpreted either as a set of executable instructions or as static data, enabling it to function as a record in a database of relationships among metadata standards. over time, these crosswalk records can be collected, searched, dynamically assembled, and even mined for new information. figure 9 shows a schematic representation of a database of seel maps that can be accessed from a utility associated with the crosswalk web service. figure 9. seel maps in a persistent resource [view full-size image] a metadata standards expert enters a new line into a machine-processable form of a crosswalk, such as an excel spreadsheet. because a seel map closely resembles an entry in a crosswalk, a seel map can be automatically generated in most cases. the standards expert and the software developer need only observe a few common conventions for coding the source, target, and special conditions, such as indicator values, that appear in seel <context> elements. once these conventions are established, a small set of changes may be the only effort required to upgrade a crosswalk to a new version of a standard because many of the maps in the database can be reused. once the new maps have been generated and added to the database, the expert interested in the onix-to-marc translation can use the search interface to retrieve all maps with an onix source and a marc target. this is a typical standard-to-standard translation, but in our model it is conceptually similar to other outputs. for example, a search query could specify dublin core or onix sources and marc targets, producing a translation that mimics an application profile. yet another search could retrieve all seel maps whose target is marc 245 $a. if the source in one map in the result set identifies the onix path <distinctivetitle>, while the source in another identifies the mods path <titleinfo><title>, an automated process could safely produce an implied translation between the title fields in onix and mods, using marc 245 $a as an intermediary. as we said in the introduction, metadata standards experts have not yet published a crosswalk for onix and mods, but none is required when a collection of maps from related standards can be intelligently exploited. the seel maps database acknowledges some important realities about the metadata crosswalk as a conceptual object for achieving chan and zeng’s element-level interoperability. the maps corresponding to a published crosswalk between a pair of standards can be retrieved from the database with a straightforward search query, as we explained. more than likely, though, the translation will need to be customized in some way by adding, deleting, or overriding individual maps to accommodate the needs of a local task. even so, the logic of decomposing the translation into maps and storing them emerges from the observation that most maps are likely to be persistent. thus the seel maps database enables the user to map the relationship between dublin core title and marc 245 $a once, store it, reuse it in a variety of translations and structural encodings, or mine information from it. 6. using the translation service the translation logic is available to oclc applications through a call to the crosswalk web service, or to third parties through a link on oclc’s researchworks page [20]. the service appears to the client application as a black box that processes a work order expressed in the web services description language [26], where the user supplies a triple that specifies the metadata standard, the syntactic encoding, and the character encoding for the input and output of the translation. the service is called from many contexts, ranging in complexity from batch processes that translate only a few elements to an editing interface that premits a cataloger to enter a detailed record and view it in multiple formats, in real time. crosswalks are obtained from public sources such as the library of congress, and from oclc’s own metadata subject matter experts. internally created crosswalks are submitted as spreadsheets. though oclc’s subject matter experts coded crosswalks as spreadsheets long before the crosswalk web service was developed, the task is arduous, so we make every effort to leverage this work. for example, to obtain the relationship between dublin core and marc, the subject matter expert mapped dublin core terms to marc, from which a seel script was automatically generated. since dublin core terms is an application profile, the ‘dumb-down’ principle [18], which collapses extensions such as dcterms:title:alternative to dc:title, can be used to generate the crosswalk to unqualified or simple dublin core. the same intellectual input produces two different translations and many more when the different structural encodings are taken into account, and still more when these translations are used as the source or destination in relationships involving marc and mods, gem, or onix. these savings are possible because every component in our translation model is designed for reuse. in the future, we plan to streamline the writing of crosswalks by developing a user interface that accepts inputs required by the seel map structure and generates either human-readable crosswalks or seel scripts. so far, our work has focused on bibliographic records but discussion is underway to develop additional sets of translations that represent descriptions of authorities, holdings, and museum data. our goal is to create an environment that makes it as easy as possible to exploit the intelligence embedded in crosswalks for any kind of markup. through repeated use, these crosswalks will be vetted and revised and the most popular ones will emerge as de-facto standards. 7. the status of schema-level interoperability to summarize, we have described a translation model (shown in figure 2), a data model (morfrom), a language specification (seel), a software toolkit (the crosswalk web service), and, soon, a publicly accessible demo designed for the large-scale management of diverse streams of metadata. this work proceeds from the hypothesis that beneficial consequences follow if the atomic unit of analysis is a map, or a single equivalence between two elements—not a complete translation between two standards, the more typical solution proposed in the metadata translation problem space. in short, the crosswalk web service answers chan and zeng’s call for tools that make it easier to implement element-level interoperability and has permitted us to normalize the records encountered in the day-to-day conduct of oclc’s business. these records represent about a dozen formally defined metadata standards and a difficult-to-enumerate span of structural diversity. but because we serve a single community of practice, the semantics of the relationships required to execute a large number of transactions are generally well understood. nevertheless, this solution addresses only the easy cases. at a level lower than the element is a set of issues that we informally refer to as field normalization—i.e, the standardization of data contained in elements that are especially critical to the management of bibliographic records such as names, dates, and citations. in these cases, as in the typical bibliographic record that passes through oclc’s systems, syntactic expression may be diverse but the semantics are transparent. as a result, we are exploring the potential of a scripting language similar to seel that operates at the sub-element level and can be written as a persistent set of instructions retrieved from a database to add, delete, or rearrange data. but at a level higher than the element is the more difficult problem of enhancing the interoperability of two standards by making their abstract models more consistent. unlike the problems we addressed in the crosswalk web service, which are largely in the realm of software engineering, reconciling differences among abstract models is a considerably fuzzier sociological problem that requires intellectual analysis, consultation with the gatekeepers of the standards, and bilateral compromise. for example, doerr, et al.[28], a conceptual reference model used in cultural heritage communities to describe museum objects, with controlled vocabularies that have been developed for broader application in digital library communities, including the authors’ own harmony [29]. but the authors reported only partial success. to resolve the problem for good requires an ontology of everything that has been deemed important to a description of a broad range of resources and genres. and the ontology of descriptions continues to evolve, as leading-edge work is being done to define the essential components in descriptions of intellectual property rights, or of educational materials that are deployed in complex technical and institutional contexts. since we have not yet worked with metadata for museum objects, the problems described by doerr, et al. might seem remote to our concerns. but this impression is misleading because a similar problem lies at the core of onix and marc, a relationship that we have examined closely. despite the fact that many pairs of elements from the two standards can be mapped and made accessible through the crosswalk web service, as we have discussed in this paper, many more await the resolution of a problem whose major symptom is that the authoritative crosswalk maintained by the library of congress is too ambiguous to use as direct input to our system. for example, about thirty elements map to marc 245 $a (two of which crop up in table 1), despite the fact that marc allows only one instance of this element per record. the underlying problem is that the onix books standard defines a record that describes a foreground object and an optional set of ancillary objects—reviews, tables of contents, web sites, promotional materials, and even pieces of clothing, each of which may be accompanied by a detailed description that includes a title—while a marc record is designed as a description of a single item. collaborative work by two high-profile organizations is required to resolve the differences between the abstract models that underly the two standards. but when this work matures, more elements could be mapped in the crosswalk web service. notes [1] “worldcat.” <http://worldcat.org>. accessed 11/01/07. [2] “onix for books.” <http://www.editeur.org/onix.html>. accessed 11/01/07. [3]”marc standards.” <http://www.loc.gov/marc/>. accessed 11/01/07. [4]”oclc worldcat (the oclc online union catalog) (worldcat).” <http://www.oclc.org/support/documentation/firstsearch/databases/dbdetails/details/worldcat.htm>. accessed 11/01/07. [5]we use crosswalk in the sense defined in the glossary of “introduction to metadata: pathways to digital information,” (n.d.): “a chart or table that represents the semantic mapping of fields or data elements in one data standard to fields or data elements in another standard that has a similar function or meaning. crosswalks enable heterogeneous databases to be searched simultaneously with a single query as if they were a single database (semantic interoperability) and to effectively convert data from one metadata standard to another. see also metadata mapping below. also known as field mapping.” <http://www.getty.edu/research/conducting_research/standards/intrometadata/glossary.html#c>. accessed 02/08/08. [6] “onix to marc 21 mapping.” <http://www.loc.gov/marc/onix2marc.html>. accessed 11/01/07. [7] “introduction to asn.1.” <http://asn1.elibel.tm.fr/introduction/index.htm>. accessed 11/01/07. [8] “resource description framework (rdf).” <http://www.w3.org/rdf/>. accessed 11/01/07. [9] “open archives initiative.” <http://www.openarchives.org/>. accessed 11/01/07. [10] “marcxml: marc 21 schema.” <http://www.loc.gov/standards/marcxml/>. accessed 11/01/07. [11] “iso: 2709:1996.” <http://www.iso.org/iso/iso_catalogue/catalogue_tc/catalogue_detail.htm?csnumber=7675>. accessed 11/01/07. [12] “dublin core metadata element set, version 1.1.” <http://dublincore.org/documents/dces/>. accessed 11/01/07. [13] “dcmi metadata terms.” <http://dublincore.org/documents/dcmi-terms/>. accessed 11/01/07. [14] “mods: metadata object description schema.” <http://www.loc.gov/standards/mods/>. accessed 11/01/07. [15] chan, l.m. and zeng, m.l. (2006). “metadata interoperability and standardization – a study of methodology. part i: achieving interoperability at the schema level.” d-lib magazine.12:6 (june). <http://www.dlib.org/dlib/june06/chan/06/chan.html>. accessed 11/01/07. [16] zeng, m.l. and chan, l.m. (2006). “metadata interoperability and standardization – a study of methodology. part ii: achieving interoperability at the record and repository levels.” d-lib magazine.12:6 (june). <http://www.dlib.org/dlib/june06/zeng/06/zeng.html>. accessed 11/01/07. [17] godby, c.j., smith, d., and childress, e. (2003). “two paths to interoperable metadata.” presented at dublin core (dc)-2003: supporting communities of discource and practice–meadata research& applications. seattle, washington (usa), september 28-october 2. <http://www.oclc.org/research/publications/archive/2003/godby-dc2003.pdf>. accessed 11/01/07. [18] heery, r. and patel, m. (2000). “application profiles: mixing and matching metadata schemas.” ariadne. 25 (september). <http://www.ariadne.ac.uk/issue25/app-profiles/>. accessed 11/01/07. [19] godby, c.j., young, j.a, and childress, e. (2004). “a repository of metadata crosswalks.” d-lib magazine. 10:12 (december). <http://www.dlib.org/dlib/december04/godby/12godby.html>. accessed 11/01/07. [20] “researchworks: things to play with and think about.” <http://www.oclc.org/research/researchworks/default.htm>. accessed 11/01/07. the public demo of the oclc crosswalk web service was originally intended to be available in time for publication of this article. it was not quite ready in time, but we anticipate it being available soon. [21] “jumper.” <http://www.jumpernetworks.com/>. accessed 11/01/07. [22] “marc 21 specifications for record structure, character sets, and exchange media.” <http://www.loc.gov/marc/specifications/specchartables.html>. accessed 11/01/07. [23] strictly speaking, the scripts are not analogous because this script inserts indicators, while the corresponding seel map in figure 6 does not. but the seel map shown in figure 7 shows how it would be done. [24] “onix to marcxml stylesheet.” <http://www.loc.gov/standards/marcxml/xslt/onix2marc21slim.xsl>. accessed 11/01/07. [25] “xml xpath language (xpath).” <http://www.w3.org/tr/xpath>. accessed 11/01/07. [26] “web services description language (wsdl).” <http://www.w3.org/tr/wsdl20/>. accessed 11/01/07. [27] doerr, m., hunter, j. and lagoze, c. (2003). “towards a core ontology for information integration.” journal of digital information. 4:1, article no. 169, 2003-04-09. <http://jodi.tamu.edu/articles/v04/i01/doerr/ >. accessed 11/01/07. [28] the cidoc conceptual reference model. <http://cidoc.ics.forth.gr/>. accessed 11/01/07. [29] lagoze, c. and hunter, j. (2001). “the abc ontology and model.” journal of digital information.” 2:2, article no. 77, 2001-11-06. <http://jodi.tamu.edu/articles/v02/i02/lagoze/>. accessed 11/01/07. about the authors carol jean godby is a research scientist in oclc research. her research interests focus on automatic metadata extraction and creation. godby at oclc.org devon smith is a consulting software engineer in oclc research. he works primarily on metadata translation. smithde at oclc.org eric r. childress is a consulting project manager in oclc research. he provides project management support for oclc research initiatives and participates as a contributing team member on selected research projects. eric_childress at oclc.org subscribe to comments: for this article | for all articles 2 responses to "toward element-level interoperability in bibliographic metadata" please leave a response below: will, 2012-06-22 thanks for the great notes toward element-level interoperability in bibliographic metadata | john's ls 566 blog, 2015-02-06 […] http://journal.code4lib.org/articles/54 […] leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – an open-source strategy for documenting events: the case study of the 42nd canadian federal election on twitter mission editorial committee process and structure code4lib issue 32, 2016-04-25 an open-source strategy for documenting events: the case study of the 42nd canadian federal election on twitter this article examines the tools, approaches, collaboration, and findings of the web archives for historical research group around the capture and analysis of about 4 million tweets during the 2015 canadian federal election. we hope that national libraries and other heritage institutions will find our model useful as they consider how to capture, preserve, and analyze ongoing events using twitter. while twitter is not a representative sample of broader society – pew research shows in their study of us users that it skews young, college-educated, and affluent (above $50,000 household income) – twitter still represents an exponential increase in the amount of information generated, retained, and preserved from 'everyday' people. therefore, when historians study the 2015 federal election, twitter will be a prime source. on august 3, 2015, the team initiated both a search api and stream api collection with twarc, a tool developed by ed summers, using the hashtag #elxn42. the hashtag referred to the election being canada's 42nd general federal election (hence 'election 42' or elxn42). data collection ceased on november 5, 2015, the day after justin trudeau was sworn in as the 42nd prime minister of canada. we collected for a total of 102 days, 13 hours and 50 minutes. to analyze the data set, we took advantage of a number of command line tools, utilities that are available within twarc, twarc-report, and jq. in accordance with the twitter developer agreement & policy, and after ethical deliberations discussed below, we made the tweet ids and other derivative data available in a data repository. this allows other people to use our dataset, cite our dataset, and enhance their own research projects by drawing on #elxn42 tweets. our analytics included: breaking tweet text down by day to track change over time; client analysis, allowing us to see how the scale of mobile devices affected medium interactions; url analysis, comparing both to archive-it collections and the wayback availability api to add to our understanding of crawl completeness; and image analysis, using an archive of extracted images. our article introduces our collecting work, ethical considerations, the analysis we have done, and provides a framework for other collecting institutions to do similar work with our off-the-shelf open-source tools. we conclude by ruminating about connecting twitter archiving with a broader web archiving strategy. by nick ruest and ian milligan introduction during the 2015 canadian federal elections, we captured 3,918,932 tweets written using the #elxn42 hashtag: thoughts on the nature and stature of political candidates or parties, live running commentary during leader debates, exhortations to vote, and witty ripostes or jokes to liven up the long campaign. political scientists, journalists, and other researchers can use these tweets as evidence of sentiment amongst a certain slice of the electorate: did a policy go over well? did it not? what tweets get re-tweeted, or further shared, and which ones do not? if these are questions that resonate amongst contemporary researchers, historians are also interested in the long-term preservation of digital material. tweets, as well as the much broader scope of archived webpages and born-digital data, are the primary sources of tomorrow. tweets present considerable advantages in that they represent the preservation of material representing the voices of everyday people that might not otherwise be saved, but also considerable challenges in the collection and use of data on such a large scale. if the norm until the digital era was to have human information vanish, “now expectations have inverted. everything may be recorded and preserved, at least potentially” (gleick, 2012). useful historical information is being preserved at mind-boggling rates that continue to accelerate. ibm research, for example, notes that “every day, we create 2.5 quintillion bytes of data — so much that 90% of the data in the world today has been created in the last two years alone.” (ibm research, 2016) this data has the potential to reshape multiple avenues of historical research. in the case of the #elxn42 hashtag, we have access to the tweets of some 318,176 unique users (which would include some bots and spam accounts, of course). consider what the scale of this dataset means. social and cultural historians will have access to the thoughts, behaviours, and activities of everyday people, the sorts of which are not generally preserved in the record. military historians will have access to the voices of soldiers, posting from overseas missions and their bases at home. and political historians will have a significant opportunity to see how people engaged with politicians and the political sphere, during both elections and between them. the scale boggles. modern social movements, from the canadian #idlenomore protest focusing on the situation of first nations peoples to the global #occupy movement that grew out of new york city, leave the sorts of interactions that would rarely, if ever, have been recorded by previous generations. during the #idlenomore protest, for example, twitter witnessed an astounding 55,334 tweets on 11 january 2013. if we were to take the median length of a tweet (60 characters), the average length of a word (5 characters plus a space), and think about 300 words per page, we’re looking at over 1,800 pages. this for a single day of a single social movement in the relatively small country of canada. while twitter is certainly not a representative sample of broader society – pew research shows in their study of us users that it skews young, college-educated, and affluent (above $50,000 household income). we need to keep the demographic limitations of this source base in mind, as we do with all source bases. this is not a random sample of canadian society, but a self-selecting portion of it (as with many non-digital archival collections as well). as a record of society, twitter certainly suffers from selection bias. yet, twitter – and other web archives – will still represent an exponential increase in the amount of information generated, retained, and preserved by everyday people. therefore, when historians study the 42nd federal election, we believe that twitter will be an important source. recognizing twitter’s significance, it calls out for active preservation. once an event has happened, if a small window of time has passed – 7 to 9 days – the tweets become largely inaccessible on a large scale without considerable monetary resources. while the library of congress archives tweets, it remains unclear how their access regime will work. yet using a combination of several open-source tools, librarians, archivists and other researchers can do the following: create their own twitter archives using twarc; analyse tweets using twarc-report and twarc-utilities; visualize the material; use twitter as a launchpad for further web archiving activities; and share tweet ids with an eye to sharing collections in accordance with the twitter developer agreement & policy. this article walks users through these five steps, with an eye to presenting this as a model for other forms of analysis. libraries, spread across the world, can collect hashtags of local or national significance, taking a step towards the more widespread preservation of today’s cultural record. creating your own twitter archive: data collection the web archives for historical research group began capturing #elxn42 tweets on august 3, 2015 with twarc. “twarc is a command line tool and python library for archiving twitter json data. each tweet is represented as a json object that is exactly what was returned from the twitter api. tweets are stored as line-oriented json. twarc runs in three modes: search, stream and hydrate. when running in each mode twarc will stop and resume activity in order to work within the twitter api’s rate limits.” (summers, et al, 2015) on august 3, the team initiated both a search api and stream api collection with twarc using the hashtag #elxn42. the search api was used to gather any tweets with the #elxn42 hashtag before initial collection date. the stream collection mode was initiated with the intention to gather #elxn42 tweets for the entirety of the election. however, we noticed that twarc had silently failed during september, and the research team did not notice. we believe the failure here was because of an issue with the twitter api or network connection issues, but it is not clear, and we are not confident as to why we had a silent failure. as a result we lost 27 days in total. upon realization of the collection failure, the research team immediately began collecting via the stream api and began search api collection (allows collection back 7-9 days) simultaneously. data collection was stopped on november 5, 2015, the day after justin trudeau was sworn in as the 42nd prime minister of canada. a total of 102 days, 13hrs and 50 minutes. in retrospect, the research team recommends using a combination of collection via the search and streaming api. we would use a streaming api collection over the period of the capture, as well as weekly search api collections. then, at the end of data collection and concatenating all the files together, we would deduplicate the entire dataset. library and archives canada (lac) also collected the #elxn42 hashtag, using the search api, during a similar time period; august 11, 2015 – october 28, 2015. the team made use of the lac #elxn42 capture by downloading their tweet id dataset (library and archives canada, 2015), and hydrating it. once the lac dataset was hydrated, the team combined their original dataset (ruest, 2015) with the lac dataset, and deduplicated it (ruest et al, 2015). $ twarc.py --hydrate elxn42-tweets-lac.txt > elxn42-tweets-lac.json $ cat elxn42-tweets.json elxn42-tweets-lac.json > elxn42-tweets-combined.json $ python ~/git/twarc/utils/deduplicate.py elxn42-tweets-combined.json > elxn42-tweets-combined-deduplicated.json this does not necessarily mean that between lac and our research group that we captured all tweets. driscoll and walker (2014) have shown substantial differences in what is captured using twitter’s commercial gnip service versus the streaming api. while the #elxn42 hashtag never exceeded the hard limit of 1% of all tweets enacted using the streaming api – which comes into play if the volume of tweets you are capturing exceeds 1%, common in cases such as high-profile events (the paris shootings or an american presidential debate) – there is still a chance that some content was not collected. how do you collect? collecting tweets is very straightforward. once you install and configure twarc, you can collect tweets using the twitter stream and search apis. as noted below, syntax changed slightly with twarc 0.5.0 so we have provided both as an example: search api: twarc.py --search "#elxn42" > elxn42-search.json stream api ( < v0.5.0 twarc): twarc.py --stream "#elxn42" > elxn42-stream.json stream api ( > v0.5.0 twarc): twarc.py --track "#elxn42" > elxn42-stream.json these two apis complement each other well. the search api provides historical search on a given query, such as #elxn42, stretching back somewhere between six and nine days of tweets. their api cautions that “the search api is focused on relevance and not completeness. this means that some tweets and users may be missing from search results.” given our project goals, this makes the search api insufficient. for completeness, then, we can turn to the streaming api. this gives “developers low latency access to twitter’s global stream of tweet data,” up to the aforementioned 1% volume. whereas search api goes back into past tweets, streaming only captures tweets as they happen. to put this into context, we could begin the search api on #elxn42 on 5 september 2015 and still get tweets from 3 september 2015, for example; streaming api cannot retroactively gather content. it is more complete, however. a combination of the two is a recommended approach: the streaming api for the bulk collection, and the search api to fill in any gaps that may have happened when using the system. once collected, tweets can be shared with other people through the tweet ids, which can be rehydrated using twarc. as twarc’s readme notes: the twitter api’s terms of service prevent people from making large amounts of raw twitter data available on the web. the data can be used for research and archived for local use, but not shared with the world. twitter does allow files of tweet identifiers to be shared, which can be useful when you would like to make a dataset of tweets available. you can then use twitter’s api to hydrate the data, or to retrieve the full json for each identifier. this is particularly important for verification of social media research. the command: twarc.py --hydrate elxn42-tweet-ids.txt > elxn42-tweets.json will recreate the original tweet(s) in json format, provided the content is still available on twitter. if you wanted to use our dataset, for example, it could be download in scholars portal dataverse. if a user deleted their tweet between the time of our collection and the time of your rehydration, you would not gain access to that tweet. should you collect? ethical considerations beyond the technical question of how to collect tweets comes the ever-important question of should you, and if so, how to handle the question of consent? strictly speaking, we have legal permission thanks to the twitter developer agreement & policy. we can only capture public tweets, and given the tweets are public, we interpret that as consent in the broadest form to archive and preserve this material. consent is not perpetual, as users may decide to make their account “private” after collection. accordingly, when tweet ids are hydrated, only publicly accessible tweets are hydrated (indeed, as deleted or private tweets are not made available via the api, this is unavoidable – one cannot get data about a deleted tweet from twitter). so, if a tweet is deleted in the period between our capture and hydration, the tweet will not be hydrated. similarly, if an account is public, and set to private in the period between our capture and hydration, the tweet will not be hydrated. we discuss this further in our section below on deleted tweets. george washington university’s library has been exploring, as part of their work with the social feed manager, a platform to collect social media data from twitter, the legal and ethical implications of twitter archiving. in a recent presentation at web archives 2015: capture, curate, analyze, seemantani sharma, vakil smallen, and daniel chudnov (2015) explored the three primary legal areas of concern: copyright, privacy, and access. while in the united states, the issues surrounding fair dealing largely would not see tweets as copyrighted content, they accordingly focus much of their attention on the murkier area of the ethical concerns of privacy and access. securing consent at the collection stage is largely unworkable, as sharma, smallen, and chudnov note – making this a far trickier question. as they note, and as we know, legal does not equal ethical, though. as aaron bady (2014) has noted, “[t]he act of linking or quoting someone who does not regard their twitter as public is only ethically fine if we regard the law as trumping the ethics of consent.” as researchers at the university of southern california discovered with their “black twitter project,” many are uncomfortable with the prospect of their online content being harnessed without consent for research projects. (o’neil, 2014) yet, if we do not archive this material, it could be lost forever: invaluable, diverse perspectives on unfolding events like the 2015 canadian federal election. collecting these tweets raises the prospect of a historical record not dominated by the mainstream media. we thus collect the material with the proviso that it needs to be ethically used by researchers. as dorothy kim and eunsong kim (2014) put it in their “#twitterethics manifesto,” academics and those using this material in their work need to rethink their approach: in the end, the work, the credit, the compensation, and the view need to be a shared, collaborative process. twitter and new media journalism, the internet and technology involves all of us. the voices on the platform are multiple, collective, dissenting, singular, and loud. you don’t need to speak for us–we are talking. cite us, ask us to write, get our permission. we collect the material so that it can be used. researchers need to be ethically aware. when distributing the tweet ids, we encourage them to use this material with respect. approach to analysis to analyze the data set, we took advantage of command line utilities, a number of utilities that are available with twarc and twarc-report, as well as jq. twarc-report is a set of utilities “for generating reports from twarc collections using tools such as d3.js.” (binkley, 2015) the timeline graphs above were created with twarc-report. the command is as follows: ~/git/twarc-report/d3times.py elxn42-tweets-combined-deduplicated.json -a -o embed -t local -i 24h > elxn42-times.html the flags do the following: -a aggregates output; -o specifies we wanted embedded output, -t specifies the timezone to use (local, or est, in our case), -i sets the interval, in our case every 24 hours. upon completion of capturing #elxn42, the team immediately began aggregating their dataset into a single file. the team began with 12 different line oriented json object files totaling 22gb and 4,117,753 undeduplicated tweets. these 12 files were aggregated into a single file: cat *json > elxn42-tweets.json. once aggregated, the dataset was validated with validate.py (ensuring that each line was a valid json object), and deduplicated (we have to dedupe given the combination of search api and stream api collection modes with twarc) using deduplicate.py. once deduplicated, we were able to come up with the number of tweets collected. since each tweet is a single json object representing a single line in the file, we were able to quickly calculate with simple command line utilities: $ cat elxn42-tweets-combined-deduplicated.json | wc -l since twitter automatically shortens urls, the team also unshortened every url in the dataset so that we would be able create a canonical list of urls tweeted for further analysis. we were able to create this using a combination of tools; unshorten.py and unshrtn (“a small leveldb backed url unshortening microservice written for node”). $ sudo docker build --tag unshrtn:dev . $ sudo docker run -p 80:3000 -d -t unshrtn:dev $ cat elxn42-tweets-combined-deduplicated.json | ~/git/twarc/utils/unshorten.py > elxn42-tweets-combined-deduplicated-unshortened.json with the urls, we were able to run subsequent analysis: from creating a subsequent web crawl using the corpus in order to launch further explorations of an #elxn42 web crawl, to comparing coverage within the #elxn42 url corpus with the broader internet archive, and beyond. this sort of derivative dataset can be very useful, especially given the url-centric nature of the wayback machine. data analysis and results text using jq, we extracted all of the plain text of every tweet: $ cat elxn42-tweets.json | jq -c '.text' | cat > elxn42-tweets-text.txt this was useful for working with text analysis software, such as custom scripts written in r, python, mathematica, or even using the accessible online platform voyant-tools. we were also interested in contrasting twitter data by day, to see how it evolved. to do so, we used this following script: #!/usr/bin/env python #cc0 1.0 universal from __future__ import print_function import sys import json import fileinput import dateutil.parser import dateutil.rrule import pytz import pandas as pd import datetime import io eastern = pytz.timezone('us/eastern') start_date = dateutil.parser.parse("25-july-2015") start_date = eastern.localize(start_date) end_date = dateutil.parser.parse("06-november-2015") end_date = eastern.localize(end_date) dates = pd.date_range(start_date, end_date).tolist() for date in dates: date_plus_one = date + pd.dateoffset(1) pretty_print = date.to_pydatetime().strftime('%y%m%d') filename = 'elxn42-tweets-' + pretty_print + '.json' f = io.open(filename, 'w', encoding='utf-8') for line in fileinput.input(): tweet = json.loads(line) created_at = dateutil.parser.parse(tweet["created_at"]) created_at = created_at.astimezone(eastern) if ((created_at >= date) and (created_at < date_plus_one)): f.write(unicode(json.dumps(tweet, ensure_ascii=false) + '\n')) f.close() once broken into dates, we could run further analysis. built into twarc is the ability to generate word clouds of tweets, using the following command, for example (using the 18 october 2016 data): $ python ~/git/twarc/utils/wordcloud.py elxn42-tweets-18-oct-2016.json > wordcloud-18-oct-2016.html while word clouds have considerable limitations, especially in the occlusion of context around a given keyword, the simplicity of the visualization – where the more a word appears the larger it is – can surface overall trends. the ensuing results can be seen below: here we can see the following transition in the tweets: 17 october 2015: we see the keyword “harper” is the most prominent one, as it was throughout much of the election. as the incumbent was a politically polarizing individual, the election was largely a referendum on his leadership. 18 october 2015: the day before election day. “vote” becomes the most prominent, as people want to exhort people to be ready for the polls. no one political party dominates, but the word “conservative” remains the most frequent word. 19 october 2015: election day. we see “vote” dominate, as well as the word “liberal.” this was mostly reflecting the widely retweeted announcement of the liberal party of canada’s victory that evening. 20 october 2015: the new prime minister trudeau is the topic of the day, as well as his first name: “justin.” at a glance, we are seeing a major narrative within the tweets. you can see all of the wordclouds yourself here, or animated here. this could be useful for a researcher wanting an overall birds-eye-view of content, or as a teaser to further investigations. it also speaks to how researchers could use more sophisticated textual analysis software or programming languages, such as r, python, mathematica, or beyond, to extract meaningful information from this soup of knowledge. retweets retweets can tell us quite a bit, mostly around which tweets were collectively deemed to be the most significant: whether because retweeters agreed with them, disagreed with them, or wanted to share in a pivotal moment. for example, the most retweeted tweet was justin trudeau and his wife declaring that they were “ready” after winning the election. the most retweeted tweets can be seen below: using retweets.py from twarc utilities: $ python ~/git/twarc/utils/retweets.py elxn42-tweets-combined-deduplicated.json > elxn42-tweets-retweets.json $ python ~/git/twarc/utils/tweet_urls.py elxn42-tweets-combined-deduplicated.json > elxn42-tweets-retweets.txt retweets tweet 1. 5483 https://twitter.com/justintrudeau/status/656342399854223360 2. 2104 https://twitter.com/globalnews/status/655983013168336897 3. 2104 https://twitter.com/cbcalerts/status/656283780152479744 4. 1999 https://twitter.com/ctvnews/status/656283368863223808 5. 1808 https://twitter.com/22_minutes/status/655902459769004032 6. 1760 https://twitter.com/vancityreynolds/status/656355980997881856 7. 1541 https://twitter.com/pmharper/status/655828288594669569 8. 1456 https://twitter.com/theadamchristie/status/656228806118789120 9. 1421 https://twitter.com/west_ender/status/656295500765761537 10. 1417 https://twitter.com/justintrudeau/status/655912460101152768 geographic information 5,370 out of #elxn42 3,918,932 tweets (0.14%) had geographic information associated with them. we were able to determine this by utilizing geo.py and simple command line utilities: using geo.py from twarc utilities: $ python ~/git/twarc/utils/geo.py elxn42-tweets-combined-deduplicated.json > elxn42-tweets-with-geo.json $ cat elxn42-tweets-with-geo.json | wc -1 5370 we were also able to create a geojson file of all the tweets with geographic information associated with them. with this geojson file, we were then able to map the tweets fairly simply with leaflet.js. using geojson.py from twarc utilities: $ python ~/git/twarc/utils/geojson.py elxn42-tweets-combined-deduplicated.json > elxn42-tweets.geojson using the geojson file, we can put them on an interactive map with leaflet.js with some simple html and javascript boilerplate: <!doctype html> <html> <head> <title>#elxn42 tweets with leaflet.js</title> </head> <body> <script src="http://ruebot.net/d3.v3.min.js"></script> <script src="http://ruebot.net/d3.layout.cloud.js"></script> <link rel="stylesheet" href="http://cdn.leafletjs.com/leaflet-0.7/leaflet.css" /> <script src="http://cdn.leafletjs.com/leaflet-0.7/leaflet.js"></script> <script src="http://ruebot.net/files/elxn42-tweets.geojson" type="text/javascript"></script> <link rel="stylesheet" href="http://leaflet.github.io/leaflet.markercluster/dist/markercluster.css" /> <link rel="stylesheet" href="http://leaflet.github.io/leaflet.markercluster/dist/markercluster.default.css" /> <script src="http://leaflet.github.io/leaflet.markercluster/dist/leaflet.markercluster-src.js"></script> <script src="http://d3js.org/d3.v3.min.js"></script> <script src="https://raw.github.com/jasondavies/d3-cloud/master/d3.layout.cloud.js"></script> <div id="map" style="width: 720px; height: 450px;border: 1px solid #ccc;"></div> <style type="text/css"> .leaflet-popup-content-wrapper .leaflet-popup-content { width:250px !important; } </style> <script type="text/javascript"> var tiles = l.tilelayer('http://{s}.tile.osm.org/{z}/{x}/{y}.png', { maxzoom: 18, attribution: '&copy; <a href="http://osm.org/copyright">openstreetmap</a> contributors' }); var map = l.map('map').addlayer(tiles); var markers = l.markerclustergroup(); var geojsonlayer = l.geojson(elxn42, { oneachfeature: function (feature, layer) { layer.bindpopup('<img src="'+ feature.properties.profile_image_url +'" /> <b>@'+ feature.properties.screen_name +'</b> <a href="'+ feature.properties.url +'" target="_blank">'+ feature.properties.text +'</a> '); } }); markers.addlayer(geojsonlayer); map.addlayer(markers); map.fitbounds(markers.getbounds()); </script> </html> github also supports rendering geojson files. for example, the geojson file above is rendered here with a simple gist. users we are able to create a list of the unique twitter usernames in our dataset by using users.py, and additionally sort them by the number of tweets: using users.py from twarc utilities: $ python ~/git/twarc/utils/users.py elxn42-tweets-combined-deduplicated.json > elxn42-tweets-users.txt $ cat elxn42-tweets-users.txt | sort | uniq -c | sort -n > elxn42-tweets-uniq-users.txt $ cat elxn42-tweets-uniq-users.txt | wc -l $ tail elxn42-tweets-uniq-users.txt using jq: $ cat elxn42-tweets-combined-deduplicated.json | jq -r '[.user.name, .user.screen_name] | @csv' | elxn42-tweets-users.txt $ cat elxn42-tweets-users.txt | sort | uniq -c | sort -n > elxn42-tweets-uniq-users.txt $ cat elxn42-tweets-uniq-users.txt | wc -l $ tail elxn42-tweets-uniq-users.txt from the above, we can see that there are 318,176 unique users in the dataset, and the top 10 accounts were as follows: tweets username 1. 21423 davidmorrison17 2. 15527 p_wog 3. 10812 chuddles11 4. 10051 444_nal4b 5. 8871 joannecangal 6. 8346 littleshasta 7. 8316 madeincanada56 8. 8114 lucmatte9 9. 7360 frazzling 10. 7019 stopharpertoday two of the accounts, stopharpertoday and 444_nal4b (a spam account), no longer exist. we discuss this in our deletion section below. the other users all tweeted a large amount on this hashtag, either as individuals or on behalf of organizations. hashtags we were able to create a list of the unique tags using in our dataset by using tags.py. while our original collecting was focused on the #elxn42 hashtag, many tweets use multiple hashtags: tweeting about the new democratic party of canada with #ndp, for example, in addition to the larger #elxn42 tag. we did so by using tags.py from twarc utilities: $ python ~/git/twarc/utils/tags.py elxn42-tweets-combined-deduplicated.json > elxn42-tweet-tags.txt $ cat elxn42-tweet-tags.txt | wc -l $ head elxn42-tweet-tags.txt from the above, we can see that there were 70,112 unique hashtags used. the top 10 hashtags used in the dataset were: tweets hashtag 1. 3,685,885 #elxn42 2. 1,390,783 #cdnpoli 3. 164,339 #ndp 4. 139,070 #cpc 5. 129,082 #lpc 6. 89,303 #elxn2015 7. 68,387 #polcan 8. 64,718 #realchange 9. 62,282 #polqc 10. 61,700 #globedebate urls we are able to create a list of the unique urls tweeted in our dataset by using urls.py, after first unshortening the urls as described in the “approach to analysis” section. we did so by using urls.py from twarc utilities: $ python ~/git/twarc/utils/urls.py elxn42-tweets-combined-deduplicated-unshortened.json > elxn42-tweets-urls.txt $ cat elxn42-tweets-urls.txt | sort | uniq -c | sort -n > elxn42-tweets-urls-uniq.txt $ cat elxn42-tweets-urls.txt | wc -l $ cat elxn42-tweets-urls-uniq.txt | wc -l $ tail elxn42-tweets-urls-uniq.txt from the above, we can see that there were 1,988,693 urls tweeted, representing 50.75% of total tweets, and 334,841 unique urls tweeted. the top 10 urls tweeted were as follows: tweets url 1. 11956 http://www.cbc.ca/includes/federalelection/dashboard/index.html 2. 9712 http://www.conservative.ca/ 3. 4562 http://www.votetogether.ca/ 4. 3983 http://www.cbc.ca/news/politics/macleans-debate-leaders-2015-1.3182000 5. 3926 http://www.elections.ca/scripts/vis/finded?l=e&qid=-1&pageid=20 6. 3104 http://www.elections.ca/home.aspx 7. 2812 http://www.theglobeandmail.com/try-it-now/?articleid=26875323 8. 2808 https://www.facebook.com/abu.nawaf.581/posts/10206977713713332?pnref=stor 9. 2757 http://dont-be-a-fucking-idiot.ca/ 10. 2707 https://www.mypayingads.com/index.php?ref=51826 we were also curious how many domains were tweeted. this required two steps. first, taking a text file (see elxn42-tweets-urls.txt in our dataset) of the url list and then extracting only the domain: #!/bin/bash while read p; do echo $p | awk -f/ '{print $3}' done < elxn42-tweets-urls-fixed.txt > domains-all.txt and then subsequently normalizing by removing sub-domains, so that m.youtube.com and youtube.com were both simply recorded as youtube.com. #!/bin/bash while read l; do (sed 's/.*\.\(.*\..*\)/\1/' <<< ${l%/*}) done < domains-all.txt > normalized-domains-all.txt and generating sorted frequency lists with: sort normalized-domains-all.txt | uniq -c | sort -nr > normalized-domains-all-sorted.txt the top 10 domains that were tweeted were as follows: tweets domain 1. 615421 twitter.com 2. 143941 cbc.ca 3. 66886 youtube.com 4. 66758 huffingtonpost.ca 5. 63401 theglobeandmail.com 6. 53051 thestar.com 7. 49295 ctvnews.ca 8. 46488 globalnews.ca 9. 39989 twimg.com 10. 35280 macleans.ca from this we can get a sense of how social media shapes what people share, although legacy media was surprisingly well-represented in the canadian context: the canada broadcast corporation (especially their election day dashboard), the two highest-circulation newspapers the globe and mail and toronto star, and popular television networks ctv and global news. while the huffington post‘s canadian edition made an appearance, we were surprised by the degree to which traditional media dominated. embedded images we are able to create a list of images tweeted in our dataset by using image_urls.py. using image_urls.py from twarc utilities: $ python ~/git/twarc/utils/image_urls.py elxn42-tweets-deduped.json > elxn42-tweets-images.txt $ cat elxn42-tweets-images.txt | sort | uniq -c | sort -n > elxn42-tweets-images-uniq.txt $ cat elxn42-tweets-images.txt | wc -l $ cat elxn42-tweets-images-uniq.txt | wc -l $ tail elxn42-tweets-images-uniq.txt using jq: $ cat elxn42-tweets-deduped.json | jq -r '.entities | select(.media != null) | .media[].media_url_https' | cat > elxn42-tweets-images.txt $ cat elxn42-tweets-images.txt | sort | uniq -c | sort -n > elxn42-tweets-images-uniq.txt $ cat elxn42-tweets-images.txt | wc -l $ cat elxn42-tweets-images-uniq.txt | wc -l $ tail elxn42-tweets-images-uniq.txt from the above, we can see that there were 1,203,867 total images tweets, representing 30.72% of total tweets, and 176,513 unique images. the top 10 images tweeted were as follows: tweets image 1. 5111 2. 2247 3. 1975 4. 1968 5. 1895 6. 1478 7. 1376 8. 1357 9. 1346 10. 1206 deleted tweets as mentioned above, twarc has a mode called “hydrate”. hydrate allows a user to a take a set of tweet ids — in this case you can use the data set we are working with here — and hydrate the tweets ids with the full tweet from the twitter api. this process can be slow since, “twitter limits users to 180 api requests every 15 minutes. each request can hydrate (twitter’s term for turning tweet ids into tweet objects) at a rate of up to 100 tweet ids using the statuses/lookup rest api call. so 80 requests * 100 tweets = 18,000 tweets/15 min = 72,000 tweets/hour.” (summers, 2015) in our case, we began hydrating on november 21, and finished on november 23. the process took a little over 39 hours. in the end, we had a total of 2,832,270 tweets. which means that 207,534 tweets deleted, giving us a 7.33% tweet churn. the twitter developer agreement & policy prevents us from going into much detail on the deleted users, but several significant users were deleted. one, stopharpertoday, no longer exists as of writing. and another major account, 444_nal4b, appears to be a spammer account that extensively tweeted on the #elxn42 hashtag. while twitter’s user experience is arguably enhanced by the loss of spam tweets, they are an essential part of the twitter experience and it is worth nothing that they may be significantly reduced in rehydrated twitter databases. future historians may have difficulty studying the online advertisements – annoying as they can be – of our day, unless the original data is deposited somewhere where it can be studied (the library of congress, perhaps?). but this, as noted in our reflection on ethics, is one of the key components of working responsibly with twitter. as ed summers (2015a) has put it: but if you squint right, twitter is taking an ethical position for their publishers to be able to remove their data: to exercise their right to be forgotten, allowing them to remove a teensy bit of what maciej ceg?owski calls informational toxic waste. people may be deleting their tweets because they were spam, or inflammatory, or something they regretted, especially in the aftermath of a heated election. summers (2015b) and ruest noted much the same with tweets in the aftermath of the charlie hebdo massacre. ultimately, archives are always full of large gaps and omissions: at least in this one we know that people in many cases could make their own informed decision to be removed. integrating twitter archiving with web archiving there are also fruitful opportunities for integrating this form of twitter archiving and analysis with other approaches to web archiving. our team has a complementary undertaking, the webarchives.ca portal, which enables citizen access to large canadian political web archives. the canadian political parties and political interest groups (cpp) collection is a key example of these sorts of collections. the cpp collection is of national interest in canada, covering some fifty groups ranging from major and minor canadian political parties to an assortment of political interest groups. collected quarterly, and occasionally more frequently during federal elections, it is an invaluable record of public and political life. in the lead up to the 2015 federal election, we received almost 30,000 page views and some 3,000 individual distinct users. it also received significant media attention in the canadian broadcasting corporation, including on several national programs. this collection, however, has a significant downside: its limited seed list. of the 263,708 unique urls, we checked each against the list of fifty domains to see which of the urls tweeted would have been included in the cpp collection. 46,778, or 17.7%, were part of the fifty top-level domains. on the #elxn42 hashtag then, 82.3% of urls that were tweeted would not have been included in the formal cpp collection. the domains shared also affected their permanent archiving within the global web archive. by comparing this list of unique urls to the internet archive’s cdx availability api, which takes a user-provided url and determines whether there is an archived, accessible copy in the main wayback machine, we found that of the 334,841 unique urls, only 20.34% or 68,112 existed at all in the wayback machine. of those 68,112 urls, only 33,685 had been archived relatively recently, between august and december 2015. this is largely due to the domains that are largely excluded from the wayback machine: twitter, facebook, and youtube, for example. this speaks to the importance of social media crawls. 334,841 links were shared by everyday people during the election, and only roughly one in five would appear in the global wayback machine. twitter archiving is a useful complement to a broader web archiving strategy. social media is an essential primary source for gathering and aggregating content that matters to people, for both contemporary analysis and for the future historical record. we currently have a separate paper comparing this analysis in detail as a method of web archive seed list generation, building on previous international internet preservation consortium (iipc)-funded work such as twittervane. (milligan et al, 2016) conclusion this article has outlined a light-weight and open-source method of collecting and analyzing twitter events. the case study of the 2015 canadian federal election hashtag, #elxn42, is roughly analogous to other medium-scale, longitudinal events: it lacked the severe spikes and pitfalls of an event such as the paris shootings or an american election (in which case a commercial approach would be necessary for full scoping). yet it is a perfect fit for many events of interest to libraries, archives, and special collections. beginning by identifying a hashtag of interest, twarc can be used to assemble a full dataset of tweets. twarc-report, twarc‘s utilities, and other tools discussed here can all give users a rough sense of what happened within the collection. these distant reading approaches could help isolate particular days, users, or popular tweets for researchers to study. they could not read all four million tweets, but they could use these tools to find the right ones to investigate further. while twitter’s developer agreement & policy prevents the wholescale sharing of the collected data itself, rosters of tweet ids can be easily shared using institutional repositories or other sharing platforms, allowing other users to “rehydrate” their own tweets. while this has the downside of removing tweets deleted until the moment of rehydration, this allows one to continually monitor “churn” within a collection. others may be interested in using this project to either continue their own work on the #elxn42 corpus – political scientists studying an election, for example – or as an illustrative model for other unfolding events. for those interested in using the tweet data, it can be downloaded here. in an era where web archiving and twitter collection can be seen as expensive luxuries, this article shows how, for a relatively small investment of computing power, bandwidth, and storage, people can create and analyze their own twitter archives. while aimed at historians and librarians, the open-source and free model outlined here really does open up the realm of citizen scholars being able to do their own work along these lines. as social movements unfold, both those who study events as well as those who are participating are able to collect their own archives. we hope that our #elxn42 experience can serve as an illustrative model to all of these disparate groups. acknowledgements we’d like to graciously thank the support of the social sciences and humanities research council of canada, which has supported this work with an insight grant (435-2015-0011), as well as russell white and tom smyth from library and archives canada for collecting and sharing #elnx42 tweets, and ed summers for creating twarc. thanks as well to jason colditz, zack macdonald, shawn graham, ed summers, peter murray, john fink, and peter binkley. references bady, a., #notallpublic, heartburn, twitter, 10 june 2014, http://thenewinquiry.com/blogs/zunguzungu/notallpublic-heartburn-twitter/, last accessed 16 june 2015 binkley, peter, twarc-report readme.md, https://github.com/pbinkley/twarc-report/blob/master/readme.md, last accessed 24 apr 2015 driscoll, k. and s. walker, big data, big questions| working within a black box: transparency in the collection and production of big twitter data, international journal of communication, vol. 8, p. 20, jun. 2014. gleick, james, the information: a history, a theory, a flood, 2012. ibm research, what is big data? http://www-01.ibm.com/software/data/bigdata/what-is-big-data.html, 2016. library and archives canada, #elxn42 tweets (42nd canadian federal election), hdl:10864/11310 v3 [version], 2015 kim, dorothy and eunsong kim, the #twitterethics manifesto, 7 april 2014, https://modelviewculture.com/pieces/the-twitterethics-manifesto. milligan, ian; nick ruest and jimmy lin, “content selection and curation for web archiving: the gatekeepers vs. the masses,” proceedings of the acm/ieee joint conference on digital libraries 2016. forthcoming. o’neil, lauren, university’s ‘black twitter’ study generates controversy, 4 september 2014, http://www.cbc.ca/newsblogs/yourcommunity/2014/09/universitys-black-twitter-study-generates-controversy.html. ruest, nick, #elxn42 tweets, http://hdl.handle.net/10864/11270 v2 [version], 2015. ruest, nick; library and archives canada, 2015-12-07, #elxn42 tweets (42nd canadian federal election), hdl:10864/11311 v2 [version], 2015 sharma, seemantani; vakil smallen; and daniel chudnov, “social feed manager,” presented at web archives 2015: capture, curate, analyze, ann arbor, mi, 13 november 2015. http://www.lib.umich.edu/webarchivesconference/webarchives-schedule summers, ed; hugo van kemenade; peter binkley; nick ruest; recrm; stefano costa; eric phetteplace; et al. twarc: v0.3.4. zenodo, 2015. doi:10.5281/zenodo.31919. summers, ed, on forgetting and hydration, https://medium.com/on-archivy/on-forgetting-e01a2b95272, 2015a. summers, ed, tweets and deletes, 14 april 2015, http://inkdroid.org/2015/04/14/tweets-and-deletes/, 2015b. about the authors nick ruest is the digital assets librarian at york university, and co-principal investigator of the sshrc grant “a longitudinal analysis of the canadian world wide web as a historical resource, 1996-2014“. at york university, he oversees the development of data curation, asset management and preservation initiatives, along with creating and implementing systems that support the capture, description, delivery, and preservation of digital objects having significant content of enduring value. he is also active in the islandora and fedora communities, serving as project director for the islandora claw project, member of the islandora foundation’s roadmap committee and board of directors, and contributes code to the project. in the past he has served as the release manager for islandora, the moderator for the ocul digital curation community, the president of the ontario library and technology association, and president of mcmaster university academic librarians’ association. ian milligan is an assistant professor of digital and canadian history at the university of waterloo. he is also principal investigator of the web archives for historical research group. he serves as a co-editor of the programming historian. his favourite service roles at waterloo involve working with the university of waterloo library and he also serves on library and archives canada‘s acqusitions advisory committee. his new book, exploring big historical data: the historian’s macroscope (co-authored with shawn graham and scott weingart), appeared in late 2015. he has published on web and digital archives in histoire sociale/social history, the canadian historical review, the international journal of arts and humanities computing, and the journal of the canadian historical association. his 2013 article “mining the internet graveyard” won the journal of the canadian historical association’s best article award that year. these complement publications in other canadian academic journals and a 2014 monograph with the university of british columbia press entitled rebel youth: 1960s labour unrest, young workers, and new leftists in english canada. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – a method for visualizing transaction logs of a faceted opac mission editorial committee process and structure code4lib issue 12, 2010-12-21 a method for visualizing transaction logs of a faceted opac the authors introduce a method for visualizing user transaction logs from a library catalog application. simple visualization supporting intuitive or qualitative analysis to quickly make sense of complicated patterns can be a useful supplement or alternative to more common quantitative analysis. to this end, a visual flowchart is created illustrating an individual user session. this visualization can be used to qualitatively grasp user behavior within the application, possibly as an aid to identifying patterns or clusters of use. these flowcharts are created by automatically pre-processing apache transaction logs into an xml representation of meaningful user actions, which are then converted via javascript in a web browser to html table based flowcharts. the particular toolkit introduced is named visualization for understanding transaction logs (vutl), and is available with an open source license. the toolkit has been prototyped with logs from the catalog applications of several academic and one public library. by xi niu and bradley m. hemminger background transaction log analysis (tla) has been widely used as a data harvesting and analysis method when researching users’ interaction with an information system. tla is an unobtrusive way for using a large amount of data to understand users’ interactive behavior with a system. examples of information systems on which tla has been used to examine users’ searching behavior are 1) web search engines (jansen & spink, 2006; jörgensen & jorgensen, 2005); 2) online databases (e.g., davis, 2004; wildemuth, 2004); and 3) library opacs (e.g., chen & cooper,2002; lown, 2008). these empirical studies have explored the benefits and limitations of transaction log analysis techniques, however most exploration of tla has concentrated on statistical analysis of a large volume of data. gaining an understanding of the typical flow of search actions during a session is difficult from only quantitatative statistical analysis. it can be helpful to instead intuitively or qualitatively explore large collections of data, to discover hidden patterns, or to identify and eliminate undesirable outlier log records. a visualization method that takes advantage of the human visual system to quickly understand complicated patterns could help. such a visualization must also facilitate high-quality human judgment with a limited investment of the analyst’s time and enable diverse exploratory analytical tasks, like understanding the data, discerning patterns and errors, or determining indicators of the intent of an action or an individual (thomas & cook, 2005). tools for mining transaction logs and presenting generic visual charts and graphs have been previously described (for example, gray, badre & guzdial, 1996; hochheiser & shneiderman,1999). automatic visualization tools have also been developed for several other areas including computer monitoring logs which regularly record system errors, application errors and other details of computer performance. example of these studies are the visual log browser by takada and koike (2002), graphical analysis by eick, nelson and schmidt (1993), and experimental systems by girardin and brodbeck(1998). these visualization tools relieve system administers from manually scanning sequences of log entries and could handle a large amount of data at one time. there also exist commercial tools for mining logs for business process, such as aris and hp bpi (grigori, casati, dayal & shan, 2001). this paper introduces vutl, a method for graphic visualization of transaction logs for understanding users’ interactions, facilitating further analysis. xml is employed to store the log data while html is used to display the data with desired styles. javascript is applied to parse xml and load it into html. in this paper, the transaction logs are from the faceted web catalog of the library at the university of north carolina at chapel hill (unc). this application combines free text search and facet navigation features. our data contains over 2.5 million transaction lines logging patron interaction from dec 2008 to aug 2009. however, the visualization method described is flexible enough to accommodate other similarly structured transaction logs, and has been tested with logs from several other academic libraries, as well as the phoenix (arizona) public library. technical methods data preparation in this paper, the scripts for data extracting and processing are adapted from the scripts written by lown and hemminger (2009) for logs from the nc state library catalog because both are standard apache server logs. in order to accommodate the features and functions specific to the unc catalog, we’ve made modifications for action coding. details about analyzing the data using perl scripts and mysql database are elaborated elsewhere (niu, lown, & hemminger, 2009; lown, 2008). an “action” refers to a user’s interaction with the system. in most cases, a transaction represents a single action. related to “actions” are the codes used to indicate generic categories of requests. there are a finite number of things that the user can manipulate when interacting with the system and therefore a finite number of codes representing actions. this study uses a combination of fine-grained and coarse-grained coding to allow flexibility in analysis. for the unc library catalog, 23 possible fine-grained codes and 10 possible coarse-grained codes are identified. the name and description of each action code are illustrated in table 2 below. fine-grained code of action coarse-grained code of action description beginsimpletext textsearch open the simple text search tab to begin a simple text search singletermtext textsearch submit a single word query in the simple text search box multipletermtext textsearch submit a multiple-word query in the simple text search box multiplefieldstext textsearch submit a query with words in multiple fields, typically in the advanced search page booleantextsearch textsearch submit a query in the boolean search box emptytextsearch textsearch submit an empty query in whatever search mode switchtextfield textsearch switch the field (e.g. anywhere to author) under the same query in the simple search box beginadvancedsearch advsearch open the advanced search tab to begin a multiple fields text search openfacet showhidefacets click the “+” next to a facet to show the values of the facet closefacet showhidefacets click the “-” next to a facet to hide the values of the facet showmorefacet showhidefacets click “show more” under a list of values for a facet to show more values addfacet modifyfacets click a facet value to incorporate it into the query removefacet modifyfacets click “x” next to the already chosen query term to remove it refineyears modifyfacets under “publication year” facet group, manually type the starting and ending years and submit nextpage nextpage click a page number or “next” in the result page to view what in next page sortresults sortresults click “sort by” to sort the results in result list refresh refresh click refresh button in the browser viewrecord viewrecord click a record link to view details followupaction followupaction click a link within a record to find the related records beginnewtitlessearch othersearch open the browse new titles tab to begin a new search selectnewtitles othersearch select new titles options to search begincallnumbersearch othersearch open the browse by call number tab to begin a new search selectcallnumberrange othersearch select call numbers (subject headings) to search table 2. action codes of log data under this coding scheme, each transaction is assigned to one of the fine-grained codes and one of the coarse-grained codes. coding work is done automatically by a perl script ( action_code.pl in supplementary materials) which codes each transaction by comparing two successive transactions (i.e. their url addresses which contain the search request). the difference between the two addresses allows us to determine what happened in the current action. an additional 10 data elements are either extracted or calculated based on the original transaction logs (log_pro.txt in supplementary materials). the name and description of the 12 data elements are illustrated in table 3 below. no. parameter calculated/extracted description 1 action_fine-grained calculated fine-grained action coding 2 action_coarse-grained calculated coarse-grained action coding 3 searchterm extracted text string used as a query. extracted from “ntt” 4 field extracted text field within which text string was searchedextracted from “ntk” 5 facet extracted facet values selected. extracted from “ntk” 6 refine extracted beginning and end years specified. extracted from “nf” 7 expand extracted facet group opened or closed. extracted from “ne” 8 more extracted facet group for which the user clicked “show more”. extracted from “more” 9 record extracted the viewed item. extracted from “r” 10 time calculated dwell time (in second) between the current action and the next action. extracted from date& time stamp 11 visit_num calculated session number 12 visit_step calculated step number within a session table 3. the 12 extracted and calculated data elements to illustrate the data elements, one example of the logs is given as below: 71.70.185.34 [09/mar/2009:00:13:53 -0400] "get /search?ntk=keyword&ne=2+200043+206475+206590+11&n=206432&ntt =boston+globe http/1.1" 200 40035 "http://search.lib.unc.edu/search?nty=1&ntk=keyword&ntt=boston+globe" "mozilla/5.0 (macintosh; u; ppc mac os x 10.4; en-us; rv:1.9.0.7) gecko/2009021906 firefox/3.0.7" the log example parsed into 12 data elements is summarized in table 4: action_fine-grained addfacet action_coarse-grained modifyfacets searchterm boston+globe field keyword facet 206432 (format:online) refine – expand 2+200043+206475+206590+11 (availability+location+format+subject+publication year) more – record – time 9 visit_num 154495 visit_step 2 table 4. single line of server log with 12 data elements data visualization this paper proposes using a combination of html and xml representations for visualization. the data is first stored in xml, then transformed to html for graphical presentation. javascript is used to transform an xml document to a html web page. the visualization is implemented in three steps, as illustrated in figure 1. figure 1.visualization process of transaction logs the first step of the visualization process is described above. the following steps in the process are discussed in the sub-sections below. conversion of transaction logs into a well-formed xml the processed logs with 12 data elements are converted by a perl script to a hierarchically structured xml document, in which the root element is log. each log line from the processed logs is mapped to a transaction element. the transaction element contains all the data elements in the processed logs except the coarse-grained action code, since the fine-grained code is used for the action name due to its specificity. thus, the transaction element has 11 total child elements. figure 2 illustrates the aforementioned example in xml format. figure 2.tabular and xml data of logs *the action element refers to action_fine-grained parsing xml using javascript the next stage is to read and parse the xml document, which is done with javascript executed in a web browser, using the standard xml dom api. graphic representation with html the next step is to define the graphic representation of the xml data. a good graphical representation can help even an inexperienced user to navigate a tree structure without additional knowledge about the graphical notation used (jelinek & slavik, 2004). we use flowcharts to express a process simply. we avoid using too many shapes or complex layouts to keep the notations concise and easily understandable. every transaction element in the xml document is represented by a colored rectangle in a flowchart. different colors denote different coarse-grained actions (described in table 5). actions are colored at the coarse-grained level because of the large number of fine-grained actions which make distinguishing them by individual colors difficult. for example, a green rectangle represents a transaction coded as textsearch and a red rectangle represents a transaction of viewrecord.. then the 11 child elements of the transaction element are used for additional textual information displayed in each rectangle. not all of the 11 elements are displayed for every action, because only some elements are applicable for certain kinds of actions. for example, a textsearch transaction needs to display four elements: action, field, searchterm, and time whereas a viewrecord transaction displays three elements: action, record, and time. figure 3 below is an example of the textsearch and viewrecord rectangles. table 5 illustrates mapping between actions and their graphics. figure 3. graphic visualization of textsearch and viewrecord action code shape variables displayed in the rectangle textsearch green rectangle action, field, searchterm, time advsearch light green rectangle action, time showhidefacets light blue rectangle action, expand, time modifyfacets blue rectangle action, facet, time nextpage yellow rectangle action, time sortresults white rectangle action, time refresh gray rectangle action, time viewrecord red rectangle action, record, time followupaction light pink rectangle action, time othersearch dark green rectangle action, time table 5. mapping between coarse-grained action codes and flowchart shapes and colors next, the colored rectangles are laid out to reflect the user’s navigational path. rectangles could have been laid out completely vertically or completely horizontally according to chronological sequence. here a combination of vertical and horizontal layouts is employed to indicate continuing versus new searches. if the action is refining the search, which operates on the previous result set, the action is placed horizontally to the right of its predecessor. if the action is starting a new search, which will generate a new result set, it is placed in a new row. the layout example in figure 4 illustrates this. as shown in the figure, the second action addfacet is placed horizontally next to the first action singletermtext because it was refining the search results under the same keyword “biomechanics” used in the first action. the same reasoning applies to the placement of removefacet. the next action singletermtext, using a new keyword “kinetics” to generate a new result set, was instead used to start a new line to indicate a new search. figure 4. horizontal layout vs. vertical layout in addition to the basic graphical representation, some features were added to aid understanding. first, an index, which links to each session, is built at the top of the visualization file. the index also contains some meta information about each session, including the session number and how many steps there are in that particular session. second, for transactions with dwell time longer than 60 seconds, a black bar is appended to the edge of the rectangles to indicate that. for transactions longer than 300 seconds and longer than 600 seconds, double black bars and triple black bars are appended, respectively. and finally, some visual effects were added through css style sheet language to improve the aeshtetics of the display. for example, shadowing is added to each rectangle as well as 3d visual effects. the flowchart is implemented as an html table and each rectangle is mapped to a table cell. by using html, the visualizations can be viewed with a standard web browser. figure 5 shows a randomly chosen search session. this is a whole search session visualized using this technique. the action sequence, types of actions, and their duration are all displayed visually. note again that the black bars displayed for some actions indicate longer duration. for more examples, please visit http://ils.unc.edu/bmh/pubs/searchloganalysis_unc2010/log_pro.html. there are three basic steps to constructing such graphics from raw log data: preparation of raw logs into processed log data transformation of processed log data into xml format visualization of the xml document. for more details, you can visit http://ils.unc.edu/bmh/pubs/searchloganalysis_unc2010/instructions.html. figure 5. example of a graphic visualization of a search session practical benefits through visually scanning a large amount of search sessions using the vutl, the overall sense of the search is gained and clusters of similar sessions can be identified qualitatively. to confirm the qualitatively detected clusters, some quantitative methods, such as cluster analysis, markov modeling and maximal repeating patterns, can be employed (niu, lown, & hemminger, 2009). then the vutl can be used to visualize search sessions for each cluster detected by quantitative methods. for example, one common pattern of interactions that cluster analysis identifies is something that we could call “facettextsearch,” where users formulate their queries by first typing in a text search and then applying facets. three sessions showing this type of interaction are visualized in figure 6. a more complete set of examples for different groups can be seen at http://ils.unc.edu/bmh/pubs/searchloganalysis_unc2010/clusters.html. figure 6. example search sessions in “facettextsearch” group as an alternative to displaying session visualizations one by one, a large number of search sessions could be visualized on the same html page, allowing researchers to scroll through visualizations to quickly browse large numbers of sessions, and look for patterns across sessions. vutl supports the exploration and analysis of large amounts of log data produced by library catalog interfaces, so that individual and group behaviors can be studied by both librarians and researchers. by visually scanning the color distribution, the layout of rectangles, and the length of each search session, librarians and researchers will gain an intuitive overview of users’ behaviors. for example, a large amount of green rectangles and relatively fewer blue rectangles suggest a higher usage of the text search compared to the facet search. librarians and researchers may also discover hidden patterns when browsing the search sessions. for example, in our unc data it could easily be seen that users took longer actions at the end of rows. the implication is that users tend to think for a longer period of time before deciding to try a new track in their search process (which we are verifying with statistical tests). using this visual analytics approach allows for discovering hypotheses about the search action behavior which can then be tested quantitatively from the log data, but which would not have been self evident from just examining the log data. this method of visualization can also provide a convenient tool for studying query reformulations. in the field of information retrieval (ir), there are a series of studies focused on query formulation and query reformulation. in vutl, each row of actions maps to a query formulation or reformulation. by focusing on only the initial actions on each row and looking at the query actions used by users and the time they take, librarians and researchers can learn users’ strategies for modifying queries, either by narrowing, broadening, or paralleling. knowing this is particularly helpful for improving the system through incorporating some query-reformulation assistance for its users. by presenting action sequences, this visualization technique may help librarians and researchers discern some naturally segregated clusters of users whose action sequences are similar, and to further analyze the search tactics and strategies adopted by searchers belonging to a particular cluster. identified user clusters, in turn, will help customize library catalogs catering to different populations. finally, this kind of visualization may help identify “unexpected” or atypical behaviors, so that they can be discarded from quantitative analyses – perhaps because they are likely to be web spider robot accesses, for example. in our analysis, if there are too many actions in a short time frame to be human behavior in one search session, the session is assumed to be conducted by a web spider and therefore should be removed from the data set. if there are always repeating actions (actions which are exactly the same) in one search session, it might be due to the server’s duplicate records of one action. these duplicates should also be removed. to examine vutl’s practical significance, we discussed the technique and the visualization result with unc librarians in charge of the user interface. they expressed significant interest in it. they indicated that the most obvious usage for the visualization tool is for exploring patterns of navigational behavior taken by users in a qualitative manner, which would be combined with additional quantitative methods to uncover sets of recurring patterns in overall usage. some of the librarians we talked with also considered using this technique for an end-user interface: if incorporated into the actual end-user interface as a tool for power users to see their complete past work flow, users will gain a visually organized version of their query history. software availability the source code of the software is available at http://ils.unc.edu/bmh/pubs/searchloganalysis_unc2010/. when first visiting this address, please read the provided instructions. final visualization for more examples using this technique are also available at the address above. this source code can be adapted to other apache server logs for information retrieval systems. future possibilities and limitations our approach is novel in providing open source software for generating a graphical representation of users’ interactions from apache search logs customized to a specific search system, in this case a faceted library catalog. automation of the visualization process allows it to be used with a much higher volume of data compared to manual creation of visualization undertaken by, for example, wildemuth (2004) and lown (2008). while some insight can be gained from visualizing a few sessions, the ability to see patterns and understand the big picture was limited in these studies which could only produce a small number of visualizations. although vutl is derived from log data of a library catalog, it is flexible enough to be adapted to many other information systems, such as web search engines, online databases, other opacs, or any task-based system which provides apache server logs or similar user transactional logs. by storing intermediate data in an xml format, alternate visualizations could be switched in as well. there are some limitations to the html rendering method we used. the size of the xml file loaded to the browser is highly dependent on the memory of the host computer. for example, for a computer with 2-gigabytes of memory, the browser takes several minutes to load an xml file of 10 megabytes. in this study with several million records, the size of the total xml file generated from the raw logs is several gigabytes, which can be too large for browsers in many environments. currently this problem is addressed by segmenting the xml files into different parts according the time window and then importing them into the browser separately. secondly, vutl is not a fully integrated process. currently the process contains two sub-processes, each of which is automatic. the first sub process is the conversion of log data to xml data and the second is the visualization of xml data with html language. integration between the two processes is not seamless at this point. we hope to integrate the process as part of our future work. there is still great potential for improvements on the visualization effects and styles depending on different research needs. for example, support for semantic zooming with a layered presentation providing information at different levels of detail would be helpful. finally, the most important issue is how to quantify the interaction patterns revealed by this visualization. the researcher still has to look at all, or a larger number of, sessions to glean information. it still remains a challenge to view and comprehend many thousands of sessions to identify patterns. references chen, h. m., & cooper, m. d. (2002). stochastic modeling of usage patterns in a web-based information system. journal of the american society for information science and technology, 53(7), 536-548. coins davis, p.m. (2004). information-seeking behavior of chemists: a transaction log analysis of referral urls. journal of the american society for information science & technology, 55(4), 326-332. coins eick, s. g., nelson, m. c., & schmidt, j. d. (1993). graphical analysis of computer log files. communication of the acm, 37(12). 50-56. coins girardin, luc; brodbeck, dominique. (1998) a visual approach for monitoring logs. in proceedings of 12th usenix system administration conference, boston, massachusetts, 299308. coins gray, m., badre, a., & guzdial, m. (1996). visualizing usability log data. in proceedings of the 1996 ieee symposium on information visualization, 93. coins grigori, d., casati, f., dayal, u., & shan, m. c. (2001). improving business process quality through exception understanding, prediction, and prevention. in proceedings of the international conference on very large data bases, 159-168. coins hochheiser, h., & shneiderman, b. (1999). understanding patterns of user visits to web sites: interactive starfield visualization of www log data. technical report. coins jansen, b. j., & spink, a. (2006). how are we searching the world wide web? a comparison of nine search engine transaction logs. information processing and management, 42(1), 248-263. coins jelinek, j., & slavik, p. (2004). xml visualization using tree rewriting. in proceedings of the 20th spring conference on computer graphics, 65-72. coins jörgensen, c., & jörgensen, p. (2005). image querying by image professionals. journal of the american society for information science & technology, 56(12), 1346-1359. coins lown, c. (2008). a transaction log analysis of ncsu’s faceted navigation opac. master thesis. university of north carolina at chapel hill. coins lown, c., & hemminger, b. (2009). extracting user interaction information from the transaction logs of a faceted navigation opac. code4lib journal, 7. http://journal.code4lib.org/articles/1633 niu, x., lown, c., & hemminger, b. m. (2009). log based analysis of how faceted and text based search interact in a library catalog interface. in proceedings of third workshop on human-computer interaction and information retrieval, 62-65. coins takada, t. & koike, h. (2002). mielog: a highly interactive visual log browser using information visualization and statistical analysis. in proceedings of lisa xvi sixteenth systems administration conference, 133–144. coins thomas. j & cook, k.a. (ed.) (2005). illuminating the path: the r&d agenda for visual analytics. national visualization and analytics center, 3–33. coins wildemuth, b. m. (2004). the effects of domain knowledge on search tactic formulation. journal of the american society for information science and technology, 55(3), 246-258. coins about the authors xi niu is a phd student in school of information and library science at university of north carolina at chapel hill. her research interests are information seeking behavior, faceted search systems, and empirical research methods. xi can be contacted at xiniu at email-unc-edu. brad hemminger is an associate professor in the school of information and library science (sils) at the university of north carolina at chapel hill, with a joint appointment in the center for genome sciences. he directs the informatics and visualization research laboratory in sils and leads the center for research in digital libraries at unc. his primary research interests are in scholarly communications, digital libraries, human computer interaction, information searching, information visualization and bioinformatics. his current interest is in studying ways to globally share collaborative annotations and reviews for scientific research in the neonote project. http://ils.unc.edu/bmh/ subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – special report: creating conference video mission editorial committee process and structure code4lib issue 5, 2008-12-15 special report: creating conference video capturing video at a conference is easy. doing it so the product is useful is another matter. many subtle problems come into play so that video and audio obtained can be used to create a final product. this article discusses what the author learned in the two years of shooting and editing video for code4lib conference. by noel f. peden introduction capturing video at a conference is easy. doing it so the product is useful is another matter. many subtle problems come into play so that video and audio obtained can be used to create a final product. in this paper i discuss the things learned in the two years of shooting and editing video for the code4lib conference. history in 2006, the first year we did video we captured the talks and the projection screen with no previous planning. the keynote speaker, karen schneider, had brought a video camera to record her own speech. dan scott had brought a camera as well. in typical code4lib fashion, the video taping spontaneously continued after the keynote for the remainder of the conference, through volunteers. for a last minute save, this worked pretty well. several problems surfaced, both during and after the conference: insufficient tapes: we were not prepared with enough tapes, so all of the video had to be imported onto a laptop at the end of each day. since we were not prepared with software, we resorted to windows movie maker and the wmv format. this had later implications. one camera: with only one camera, we lost content while switching tapes if the speaker was not willing to wait. timing these switches is fairly easy, but it is easy to forget, especially if you are listening to the talks. no microphone: we were not tied into the pa system, so the audio was weak (particularly for soft speakers). the video camera mic was omnidirectional, so typing and other conversations were in the background, or worse. no slides: the camera was usually focused on the speaker, but if the projection screen was essential, the camera was focused on it. inevitably, some key slides were missed, and there were some weak transitions. since the projection screen was only occasionally visible, there was no way to embed actual slides during editing. wmv format: due to the wmv capture format, i used windows movie maker for editing. this worked well enough as we had no fancy editing to do, other than adjusting final audio levels and adding titles and credits. output was limited to wmv or avi, however. so, for a final output of mp4, the movie was output in avi and converted to mp4 using quicktime pro. this was done manually, and took quite some time. scripting could have been done, as i later found out. posting: ryan eby did most of the work posting the mp4 files to google video and linking it into code4lib.org. a year later, i ran all avi files through adobe premier elements to create mpeg2 versions, which i then uploaded to archive.org. for the second year of the conference, i really wanted a picture-in-picture arrangement, where the presenter desktop fills the screen and the presenter appears in the lower right corner. after searching for software, i decided upon wirecast 3, which allowed captures from multiple sources to disk, and live compositing for picture-in-picture and other transitions. with jeremy frumkin’s help, oregon state university paid for the wirecast license. since wirecast stored video on disc we would not need to remember to change tapes. wirecast supported a remote desktop capture over the network; desktops could be captured for clear pictures of slides or demos on both windows and mac machines. using two cameras, we could still capture the desktop for those with linux. finally, we could tie into the pa system for direct audio as well. this plan required quite a bit of equipment, however. for this time around we used: powermac tower, monitor, keyboard, mouse, ups, power strip, cables (since two cameras meant two firewire ports, a desktop machine was required) two video cameras, two tripods, firewire cables, lots of tapes (minidv, my own, and many contributed by dan scott) errata: 20+foot audio cable, ethernet hub & cables, dongles, two extra laptops, mac & windows, a nice big rubbermaid tote i drove all this equipment to portland in february, stored it in my room at night, and set it up each morning. the hotel provided a table for me near the front, which i placed so cameras on either side could be pointed at the speaker and projection screen. the hotel tech also provided me a mono input from their pa system. here are the problems that arose: network and desktop capture: the network was completely unreliable for connecting remote desktops. in fact, of all attempts, it worked only once. hotel networks have not been up to our demanding use of bandwidth as a group. i would not ever plan on this again. even if the network was reliable, it was not reasonable to expect every presenter to install this software, particularly for short 5 minute presentations. set up time: i got locked out of my room on the first morning. allow for plenty of setup time. preferably do a test run the day before, not the morning of the event. recording audio for questions: i was not prepared to record questions. i relied at first on the camera microphones, then on a mic on a stand which people would seldom come to, and then on a portable mic carried by ed corrado. it is nearly impossible to record audio for questions without a portable mic, and someone to carry it around. however, if a camera is recording, it will have some audio as well, and this can be coaxed with some computational amplification and noise removal into passable audio. software: wirecast had some quirks: audio: the program would not remember audio settings. each time i started the program i had to change audio to the mic input, and turn off the camera audio on each camera input. this resulted in the loss of several minutes of audio for our main keynote when wirecast crashed and, upon restart, i did not realize the audio was not being captured. cameras: wirecast did not do well with a camera’s entering sleep mode. occasionally it would crash, which led to a reset of audio settings. a few times this left us without any video or audio. most cameras have a recording or capture mode that does not enter sleep. one camera we used had to be set to be photo mode where it acted like a web cam. record mode will usually go to sleep if the camera is paused for any extended period of time. transitions: in wirecast, multiple camera views could be configured to zoom in on certain areas and position themselves in areas of the screen. i tried to use this to zoom into detailed slides to make things readable. it was not really possible to tell when a speaker was going to change slides, however. usually i ended up zooming in the transition to the next slide. in editing, nearly all of these attempts turned into problems and had to be fixed. the only consistent transition was bringing the speaker to full view during questions. editing: when editing finally came around, several more things became apparent: file format: wirecast output in a quicktime format. this ostensibly should have been easy to load into imovie or the like. it turned out no version of imovie i could find, even on a new machine, could load the captured video. i ended up buying a license to adobe premier elements and doing all video editing on a pc. though it was occasionally unstable, this worked well and i was very pleased with the program. the process of finding something that would work took a large amount of time. tape backup: not having a backup recording on tape was a real drawback. partway through the conference i realized this and i started recording when i could. this saved our bacon on a few recordings that crashed. it would have been helpful for the ten minutes of lost audio from our main keynote. transitions: as mentioned above, most live transitions did not come at expected times with unrehearsed talks. when recreating slides by inserting their static images in the video, these transitions only created trouble spots that had to be finessed. for example, suppose the slide has not changed for a long time, and you transition to the speaker so they fill the screen. the speaker moves on to the next slide, and before you can catch it, they move on again. you have no recording of the missed slide. now you are faced with inserting the slide and transitioning the speaker video out and in around it. audio volume: any steps to improve audio will help. however, nothing will change speakers moving back and forth, or turning away to the screen. as a result, one has to go through each video from beginning to end to make audio levels consistent and clear. audio channels: due to the nature of the hotel pa system, we had audio only on the left channel of a stereo recording. with any passable video editor this can be fixed, but it adds one more item that must be remembered for every video, and so it will be forgotten at least a few times. lightning talk information: lightning talks are informal 5 minute talks given in bunches of 10 to 20. they are informally organized during the conference. many of the presenters are not used to presenting. it is best to do the simplest recording possible here, with no attempt at transitions, and to record to tape as well. it is of utmost necessity that someone record titles, presenters, and content while these presentations are happening. it is almost impossible to find the information later. uploading: google video turned out to be more trouble than it was worth. after uploading several gigabytes worth of video, none of it appeared, with no explanation for the failures. i could not tell if it failed due to the format, the encoding, the size, or some other issue. in the end we chose a quicktime format that was smaller, and ryan eby battled with the uploads until they were recognized. format is not an easy issue, as there are so many. lessons learned looking forward, these are my recommendations for the third year: divide and conquer: do not have one person do this. ryan eby helped with posting in both 2006  and 2007, but i easily spent over 120 hours capturing, solving problems, and editing. editing will always be the lion’s share. however, things could be divided as follows: capture: one person uses wirecast and does all the capture during the conference. editing: one person receives all the raw footage on a hard drive and does all the editing. uploading: one or more persons uploads these files to archive.org and/or google video, entering all the metadata. code4lib.org: one person links all files into the code4lib.org website. use three cameras: since a recording camera usually goes to sleep a short while after it reaches the end of a tape, and a sleeping camera crashes wirecast, use a third camera to record to tape (a second person could do this.) use two cameras for wirecast; one to capture the projection screen, and one to capture the presenter. the tape recordings provide the means to recover material that is lost or unusable through capture. transitions: only do one transition for all presentations. start with the slides and the presenter in the lower right, and end with the presenter only for questions, if there are any. anything else should be done during the editing phase where tape footage and slides can be used. wireless mic for questions: plan from the beginning for one person to walk around with a wireless mic. people will still start talking without it, but as long as one video camera is recording, its omni directional mic will provide something. release agreements: have these ready so presenters can fill them out before or after they speak. we completely forgot this the second year. thankfully the people involved with code4lib are quite nice, and will tell us to take down any of their material they don’t want posted. obtain slides during the conference by any means, including begging: although it takes more time to overlay slides into the video, the end product is incomparable. but it is difficult to chase these down afterwards. what can be obtained during the conference should be saved. announce the need at the beginning of the conference, and assign and name one person to save the files. presenters can give the files to them before or after their presentation. they should probably have a few flash drives handy. be prepared to record lightning talk information: information on informal talks is very hard to reconstruct later. you can’t get it all from the video. editing software: good editing software will cover a multitude of mistakes. adobe premier elements worked very well for me, after i learned its tricks. it has many of the features of premier pro, without being too complicated. export format: we found a smaller format, such as quicktime, worked best for google video. archive.org suggests mpeg2 format with mp3 audio. premier elements supports this format, though mono mp3 audio must be specified. these are large files, being roughly 1gb for a 20 minute presentation. archive.org accepted and converted these just fine. uploading: i found the google video uploader buggy. hopefully it will be better next year. on archive.org, do not use the creative commons publisher tool for the large mpeg2 files. use the direct ftp route for files over 100mb. when the cc tool fails, which happened fairly often, it leaves you with a metadata mess to clean up. the ftp route separates the ftp upload from the metadata creation. conclusion first, plan ahead. be sure you know what the hotel provides, and what equipment you need to bring. always provide enough time for set up, and test ahead so you are not in a last minute rush. for a successful output (and a sane editor), quality raw footage is a requirement. while capture software will provide much of that footage, raw camera recordings and slide files are indispensable. with some planning, division of labor, and the right software and equipment, the conference can be preserved and enjoyed by many. both those who are not able to attend and those who did will revisit the content, in some cases many times. since the conference this year, many have expressed thanks at the work done. i do not have exact numbers, but even videos that have merely been placed on archive.org, and not linked anywhere else, have been downloaded 20 to 50 times. if the information you are recording is important to preserve and make available to others who could not attend, the effort is worth it. resources: adobe elements premier: http://www.adobe.com/products/premiereel/ vara software wirecast: http://varasoftware.com/products/wirecast/ code4lib conference videos on archive.org: http://www.archive.org/search.php?query=code4lib about the author noel peden , has worked at pierce library at eou for several years, creating web applications and managing data. he graduated with a physics degree in 1997 from eou. author email: noel@peden.biz author url: http://www.peden.biz subscribe to comments: for this article | for all articles one response to "special report: creating conference video" please leave a response below: harold williams, 2010-03-18 thank you for this report mr. peden. i am using this as a guide for my video shoot of our women’s retreat. your honesty has made this an easy reading. i, too, have had some blunders in the field and should document them but i would rather forget them. thanks again for being so courageous. harold williams leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – open source library software development in a small rural library system mission editorial committee process and structure code4lib issue 19, 2013-01-15 open source library software development in a small rural library system using the crawford county federated library system’s development of an open source web kiosk management system, as an example, this article will illustrate how an open source library project is defined, specified, written, tested and rolled out. the article will also discuss how the project was released as an open source project and future development of the project. the web kiosk project is called libki and was written to authenticate users and allow access to the internet kiosks based on time limits. libki is a completely open source project and is now used by multiple libraries across the us. the client side of libki is cross platform and supports multiple operating systems including microsoft windows and linux. the administrative side of the program allows access to user logs, controls time and access and allows the librarian to log a patron off the system in real time. libki was completely developed and written by staff members of the crawford county federated library system. by kyle hall, cindy murdock ames, and john brice introduction how does a rural pennsylvania county library system develop open source solutions that are used by other libraries? a more important question is why does the crawford county federated library system, with slightly less than 100,000 citizens and nine libraries develop open source solutions? the long version of the above two answers is presented in the article below. the short answer is that we can. actually, any library can. american libraries embraced computers as soon as the technology was practicable for use. unfortunately, the computers at the time were large, required engineers from mit to write the software, and were quite expensive. the necessary know-how to setup, manage, and support such systems was best left to large corporations with libraries purchasing their services. even the biggest libraries in the country could not develop their own circulation systems. even when libraries formed consortiums to develop software it was very expensive to market and distribute circulation systems. for all these reasons, circulation systems, which eventually became integrated library systems (ils’s), were eventually spun off to commercial interests. today, technology has become ubiquitous; software languages are accessible to individuals with the proper training, and programming is taught in every reasonably sized university. communities of like minded libraries can collaborate using the internet around the world for free. once an open source project has been developed and released it can be distributed, again for free, and it can be updated and changed as more libraries give their input. eventually successful open source projects develop vibrant communities around them. all of these trends have resulted in libraries now having the tools they need to develop, maintain and distribute their own software, as well as take advantage of the work of others. for example, any library using the apache web server to host their website is benefiting from the open source methodology. as you read the rest of the article we ask you to think about what impediments are in your organization’s way to do what we do. we would hazard to guess that most of the impediments are how things are done traditionally, not necessarily what can be done now with current technology and open source methodologies. the road to libki the crawford county federated library system has been using oss (open source software) in its it operations since around 1999, beginning with some basic floppy-based routers used to share a dialup internet connection among each library’s computers. (please see http://meadvillelibrary.org/os for a full listing of all of our oss projects.) following the success of that project we recognized the potential of oss, integrating it increasingly into our infrastructure. from that first project onward we decided that whenever possible any future new it solution we offered our member libraries would utilize open source software. we made this decision for a number of reasons. first was reliability. we were frustrated with commercial products that were a closed book to us, relying on reboots, fresh installs, and/or phone-based tech support to attempt to resolve problems, often unsuccessfully. we found the hands-on approach to resolving issues with oss to be a refreshing alternative; generally with a bit of web searching or queries on appropriate mailing lists or irc channels we could find solutions to most of the problems we encountered. secondly, developing our own solutions with open source software allowed us to gain control of hardware purchases. numerous times we had to allocate precious dollars to replace perfectly fine computers because they no longer met the specifications of the latest commercial software. by choosing to use oss software on desktops and servers we were able to extend the life of much of our hardware. the third issue was customization; we could customize open source software to meet our needs, from locking down public desktops to creating custom websites and services for our libraries. finally, the last reason was the cost. in addition to saving money by being able to extend the life of our hardware through the use of oss, we were able to save considerable funds with oss since it is largely free, unlike most commercial software, and doesn’t require the payment of per-user or per-seat licence fees. of course, no solution is completely free; with oss you might save funds on software and license fees, but there are costs associated with hiring staff capable of installing & managing it. however, we have found from our first oss project onward that the costs of such employees can be considerably less than many organizations pay for commercial software solutions. furthermore, our it staff were able to apply the skills and knowledge gained implementing oss solutions in our libraries to a wide variety of projects, since oss projects are often built using many of the same tools. our first oss project, shared dial-up internet access for our libraries, was done in desperation because ccfls had written a grant based on what became vaporware software. back in the late 1990’s before broadband was available in rural communities, we needed to construct dialup routers for our seven rural libraries in order to provide staff & public internet access. we estimated the cost of using a commercial software and hardware setup at about $3,150 per library or about $22,000 total, money which ccfls did not have. through a local contact (one of our rural librarians was married to a college professor who had learned linux to build his own weather station) we found a linux expert for free. the consultant found the linux router project (lrp), a floppy-based linux distribution that could convert an old computer into a router and firewall. we purchased several old 486s on ebay, spending about $40 each. we removed the hard drives, installed 56k modems, and simple as that we had shared network internet access and a firewall. what was great about this solution is the systems were relatively hackproof (the floppies were set up for read only) and very easy to maintain. total cost of the project was less than a $1,000. the last of the dial-up servers was retired after five years of trouble-free service. realising that we could save a considerable amount of money using open source software and understanding that we couldn’t get free help all the time we then hired this same linux expert to train cindy murdock ames, our it director, to use linux. the total cost of the training was less than $2,000. using the strategy of using our own internal resources for open source solutions we started replacing our proprietary solutions. the next project after the floppy-based router project was to replace our microsoft iis web server with a linux-based one running apache. not long after we replaced our public windows desktops with a linux-based thin client solution called ltsp (the linux terminal server project, http://ltsp.org), in january 2000. the ltsp-based thin clients provide patron internet access as well as an office suite (currently libreoffice). we also use them as public catalog computers within the library. of course reading all of this on paper makes it sound like it was a snap. it was not! on just the ltsp project alone the development work took over six months, mainly because it was our first large scale project and the learning curve was quite steep. however, we were able to take the skills gained from implementing ltsp and apply them to many other useful projects, such as building oss-based routers, filtering public internet terminals, customizing staff & public desktops, building websites, fileservers, and so on. as for the cost savings of the ltsp project we saved a considerable amount of money over the lifetime of the project. instead of purchasing new public computers we purchased new hardware to upgrade our old computers, repurposing existing cases & power supplies. the cost of upgrading the hardware for our seven original thin clients plus the ltsp server totaled $3,779.25 (the server cost $1,485, the upgraded thin clients were $327.75 each). our average cost per seat for 15 thin clients plus the server was $426.75; we priced the same number of windows-based thin clients and server at $1,649 per seat; according to cnet the average cost for a windows-based pc in january 2000 was $873. however, if we had chosen windows-based pcs we would additionally have had the cost of software to lock down the computers for public use, which we were able to accomplish inherently with linux. we used those upgraded computers for over ten years before they were replaced by newer hardware, though we did replace the server twice during that ten-year period. not having to purchase new computers every three years and new software every three to five years saved just one library in our system at least $31,000 (two replacement cycles for hardware and three for software and licensing for 15 terminals) over a 10 year period. during this time we did not increase staff size, and development work for oss projects was done around the normal routine of system maintenance and training. we culminated the switchover to oss by migrating from the winnebago circ/cat ils to the koha ils in 2005. the actual migration cost from winnebago to koha was slightly more than $45,000 for 9 libraries (two medium sized libraries and 7 rural libraries). this cost was in line with what other proprietary vendors quoted us for the migration. the cost of migration for a library working entirely on their own to implement koha would not likely be this high today–most of our cost was hiring one of the companies that supported koha to integrate indexdata’s zebra indexing system into koha so that it could support libraries with collections of our size or larger. our initial costs included the purchase of a server and staff programming and development time; we developed our own staff front end for koha as well as many custom reports, since we had full access to our data with koha. at the time there were no current costs associated with winnebago circ/cat, since it was no longer supported by spectrum, the company that currently owned it. we had looked at spectrum’s current offerings but felt that with our past successes with oss we could get more out of a migration to koha than going with a commercial vendor again. since koha is entirely web-based we would not be limited to using windows on circulation and catalog computers, so we were also able to convert our staff computers and public catalogs to linux desktops or thin clients as well, saving us both future windows license fees and freeing staff from supporting a windows desktop environment. furthermore, since we could support koha in-house there are no annual support fees involved. the only ongoing cost with koha is in planned hardware upgrades and staff. with our decision to migrate to koha, we also decided to hire kyle hall as an addition to our it staff. since kyle has a masters in information technology we could use his skills as a software developer to guide the direction of development to meet our organization’s needs. in essence, we leveraged the cost savings of choosing free software into hiring staff instead. one interesting aspect of using an open source ils is that we were able to install it as a proof of concept system, exporting the data from our current ils into a test koha installation. using these proof of concept servers our library staff could test drive koha as much as desired without having to purchase any software. we were also able to make repeated test runs of the migration process to ensure that all our data was migrated as expected. we were also able to migrate each of our libraries from their individual installations of winnebago to our centralized koha server on our own schedule, over the course of several months. if we had hired a company to migrate us to a proprietary ils we might not have had that flexibility. when we migrated each library we would shut down their old system on friday or saturday at closing, migrated all their data, and then they would open on monday using koha with all their marc, patron, and transaction data intact. once we had a developer on staff the question became how to best use his time. in the beginning a great amount of mr. hall’s time was in developing, improving, and customizing koha for our own use. once those projects were finished our librarians suggested we find a better way to manage patron time on our public access computers. since ccfls had been using open source software for several years on most of its public computers, it was a logical choice to use oss software for patron time management on those computers. almost all the public computers in our nine libraries are either standalone linux workstations or linux-based thin clients. at first we tried to use existing oss-based solutions for patron time management, but none of them met our needs, so we made the decision to develop it ourselves. how was libki developed? the inspiration the development of libki was inspired by kyle hall’s (the developer) idea to automate our member libraries’ signup process for public internet computers. in the beginning, the library simply used a signup sheet for patrons to sign in with their name and the time that they logged in. a librarian would need to constantly check this sheet to see if someone had used up his or her allotted time. when a patron’s time was over, the librarian would have to approach the patron and ask them to log off the computer so the next patron could use it. this method of time control served only to complicate the librarian’s job. the search libki was born out of a search for a kiosk management system that fulfilled our requirements. though many such systems exist, none met the library system’s basic requirements. those requirements were that, a) the software be cross-platform and run on both windows and linux, b) the system be completely free open source software, and c) would use the existing barcode number and password used on our koha ils for patron authentication. in addition the librarians required that the administration functions of the program allow them to add time to a patron’s account, view who is on the public computers at any given time, send messages to patrons, and kick patrons off the system if necessary. we first evaluated existing open source solutions to see if they would meet our needs. the first candidate was openkiosk. this software seemed to fit the requirements exactly. the source code was released under the gnu public license, and the client would run on both linux and windows. however, further review discovered that though the author may have adhered to letter of the license, the spirit had been violated. the client half of the system would compile under linux, but to compile it on windows required a special file that was not made available. to obtain a windows compatible version of the client required payment for each release. to quote the author’s website: the windows client uses the same source code from the unix client but it is not linked to any gpl’d qt library. however, it is interfaced to a qt-like c++ win32 layer. this way, source code between the unix and windows clients can be shared without any major changes.[1] this combined with the fact that the author had announced he would be ceasing development lead to the abandonment of openkiosk as an option. the second candidate was far more promising, outkafe. very similar in features to openkiosk, outkafe was indeed completely free open source software, with no catches to be found. in testing, the software appeared to work without issues. the design of outkafe included a database of users names, password and statuses, the client software that ran on top of an operating system (windows or linux) and the administration software. however, it was discovered that the administration software, though compatible with our linux operating system in use at the time, was not capable of running correctly on the ltsp server it would be deployed on. the seed so with no viable existing open source options, the decision was made to tailor outkafe to meet our needs. the project started out as simply a web-based replacement of the outkafe administration program, while retaining the rest of the outkafe application. the outkafe client and server were designed not to interact directly with each other, but via database access using the postgres dbms. both sides of the software would connect directly to outkafe’s database and make updates to it that the other could then read. though not the best design, it made writing an alternative administration interface easy. this new administrative interface was released as free open source software using the gpl under the title outkafe-web. so at this point our kiosk management system consisted of two components (client and database) written as outkafe and one component (administration) written by kyle hall. soon it became apparent that the process in outkafe that managed the decrementing of time used had to be rewritten. then came a request to customize the client interface. these customizations were not possible with the client as written, and kyle was not familiar with the programming language (pascal) in which the client had been written. the birth of libki since our evolved solution could not meet the demands of our librarians we decided to create our own solution which included all three components (administration, client, and database management) written by kyle. after a period of about nine months of on and off development all of the original parts of outkafe were replaced. the new replacements were packaged up by the developer and released as libki, a free open source kiosk management system. libki was first installed at the system headquarters library and went through a three plus month testing period to work out all of the kinks. training was needed on the staff side and the program had a few bugs that needed to be fixed. after an appropriate amount of sorting things out we then presented libki to the rest of the libraries in ccfls. most of them chose to use it. though not perfect, the system worked, and continues to work for libraries in the crawford county federated library system and elsewhere. though the number of institutions using libki is unknown, a few have contacted us to thank us and some have contributed in support of libki development. the advancement though functional enough to be used by many libraries over the years since its release, the developer realized that there were a number of areas in the software that could be improved. both the client and server required direct database access, a holdover from being a replacement for a different piece of software. the administration web interface required full page refreshes to update the data about the patrons and clients. in other words, if you left the administration page open for any period of time the data would quickly become stale. in addition, the client software was slow to load. these deficiencies in the current version of libki have led to a complete ground up rewrite of the entire system, libki 2.0. first, the client was rewritten in c++/qt4 a language that allows for much faster startup times and better integration with the host operating system. a ‘shim’ was written for for the existing server to allow it to communicate directly with the server, rather than the previous method of direct database access. currently, at the time this article is being written, a new administration system is in development. web-based, like the previous iteration, it is vastly improved in nearly every aspect, both visually and mechanically. this new server leverages ajax to allow for constantly up to date information to appear on the page. it additionally can display a history of client usage, and also has a permissions system to limit staff from changing system settings. another big improvement for libki 2.0 is support for the sip2 protocol. the current version of libki integrates with the county’s ils by leveraging the open source nature of koha. kyle was able to modify koha to send alerts to the libki servers across the county in order to add, update and delete accounts as they exist in the koha ils. by adding sip2 support to the system, libki can simply make a request to the ils to authenticate a given user and act on the results accordingly. this advancement also allows libki to have single logon integration with any ils that supports sip2. the current status as of november 2012, of libki 2.0 is that it is up and running on a test server getting ready for beta testing at system headquarters. it will probably take about a month of beta testing before being released for production. the current code of libki 2.0 is available at sourceforge.net via libki.org. at this point future possible enhancements include reservation of kiosks, as well as print management and payment. discussion on ccfls oss strategy and how it relates to the rest of the world. if you do the math the total amount invested in the project is slightly over $15,000 including labor and hardware costs. the biggest cost of the libki project was kyle hall’s time. ccfls is located in northwest pennsylvania, which is an area of the country with a very low cost of living compared to much of the us. therefore development staff cost here is actually on par with what a large urban library would pay a professional librarian i. the process we have followed with oss projects and development is that we identify a need, investigate the oss solutions available, determine if development is necessary, develop the software, create a proof of concept system, get feedback from our librarians, develop a beta test version and launch it in one of our libraries, and then finally roll it out to all of the libraries interested in the product. the actual development time is not budgeted, it is just one of kyle’s duties. kyle, at the time of the libki project, not only wrote code but also installed and maintained computer systems throughout the county libraries including tasks such as running network cables, installing security systems, and researching possible software for adoption in our libraries. development was done outside of other routine tasks or during times when it was not practicable to visit other libraries such as in bad weather. currently he does development work for ccfls part-time while working for bywater solutions as a koha developer full-time. when analyzing this situation it is easy to say that what ccfls does cannot be replicated at other locations. ccfls hired a programmer at far below market rates, used him for routine it chores and does development when it suits the schedule. however, if you analyse the situation further we figured out how to do open source development even though we have a very small staff, very limited funding, and difficulty recruiting people to northwest pennsylvania due to our low pay and long winters. the strategy we followed was to train our existing staff on oss operating systems, use all of the available sources of free support available, hire smart individuals from local state universities who were content to stay in our area, and develop one project at a time with no firm due date for roll out. will this exact technique work for every library in the country? no. what we wish to say here in crawford county is that oss tools and individuals trained in their use are more readily available now than ever before. by mixing and matching the latest tools and resources an effective oss development or adoption strategy can be developed by any library. it just takes a fresh perspective, a willingness to think in nontraditional ways and full management backing in order to formulate an effective plan for developing and/or adopting oss for libraries. conclusion overall the development of libki has been spread out over five years with an estimated 500 hours of development time. for that effort ccfls has had the use of an efficient and free kiosk management program. the program is available for free to anyone wishing to use it and we do know that numerous libraries have benefitted from its use. a representative from coos county libraries shares his experience: we utilize libki in 5 public libraries and 1 academic library in our consortium. libki has enabled library staff to remove themselves from distractive public computer management chores while allowing them to concentrate their efforts on circulation and reference services. with an easy-to-use staff interface, and support for oss as well as windows, libki is a perfect solution for public computer time management in a public library. did ccfls save money with this project? a closed source solution for nine libraries over the same period of time would come close to the same amount of money that we spent developing our own solution. however, we now have kiosk management that exactly meets our needs. if there are issues that arise we can fix them ourselves, there is no waiting for the next release to be made available in order to fix bugs and no reliance on external support services. in addition, as the product matures and other libraries help contribute to the development of the libki project, our yearly investment decreases. so if you look at the total value of libki to ccfls in the first five years it was a zero gain proposition; however, going into the future we save money. this is typical of other open source projects we have done. the original cost is similar to commercial solutions; however, the savings downstream can be tremendous and can be invested into development of other projects plus the library community gets a dividend of a new open source program. it can be said that the libki project shows what can go wrong with open source development. a simple project to implement time management using open source software was not possible. by changing one aspect of an existing program we could make it work. once we had it working we found we needed additional capability. so, we had to do further development to the point where we eventually had to rewrite the rest of the program. once we got that second solution up and running we decided to make it even better and include the latest software (ajax) and standards (sip2) to the program. so, once again, we redid libki to make it better. so to get there we had to do basically three versions of the program. would we have been better off writing a fresh program to begin with? maybe, but chances are we wouldn’t have gotten it right the first time anyway. the end result is that ccfls now has an up to date kiosk management program that seamlessly ties into our koha ils, all for the cost of a similar commercial system that wouldn’t necessarily meet all our requirements. the upside to all of the work and treasure spent so far is that not only does ccfls have free access to libki, but so does the rest of the library community. this is exactly the type of collaboration that all libraries should consider. we know the excuse will be given that not all libraries can afford to hire a developer. however, if we all have to hire accountants for the finance department or catalogers for technical services shouldn’t we consider that programmers are a vital component of a library as well? shouldn’t a developer’s skills and abilities be included in a library organization? we here in the rural forests of pennsylvania believe that the answer to that question is yes. end notes [1] http://openkiosk.sourceforge.net/dw.html author information kyle hall is the project manager of libki and a software developer at crawford county federated library system. cindy murdock ames is the director of it at crawford county federated library system. john brice is the executive director of the meadville public library and the ceo of crawford county federated library system. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – editorial introduction: new year resolutions mission editorial committee process and structure code4lib issue 31, 2016-01-28 editorial introduction: new year resolutions while new year’s day came and went with very little fanfare at my house (well, if you don’t count our star wars marathon), i think i’d be remiss if i didn’t take the time to mark the passing of the new year, with a look ahead to the future.  and i think it is fitting, then, […] while new year’s day came and went with very little fanfare at my house (well, if you don’t count our star wars marathon), i think i’d be remiss if i didn’t take the time to mark the passing of the new year, with a look ahead to the future.  and i think it is fitting, then, that this issue includes an article focusing on the distribution and ease of adoption of open source software in libraries.  this is a topic that i’ve written about in the past[1] and has been recently weighing a lot on my mind as i’ve been working with a handful of cultural heritage organizations struggling as they navigate these waters.  the members of the code4lib community do tremendous things.  they create innovative projects and develop solutions to complex issues that move their institutions forward leaps and bounds.  this community is selfless and idealistic – they share their code, their time, and their expertise – all of which will be on display in philadelphia during the annual meeting.  but we have a bit of a blind spot when it comes to making the tools and solutions that we build easily adoptable for others.  we put up unintentional barriers that can lead potential users down the winding roads of dependency hell or place them in a never ending purgatory of difficult data migrations between application versions.  we embrace the idea of open source as “free as in kittens,” underlying the care and feeding that will be required to maintain and engage with a particular solution.  but for a significant number of cultural heritage organizations, this idea is closer to “free as in mountain lions” because sure – you catch one – but you are likely going to end up a little bloody and in a lot of pain… fortunately, we can do something about it.  i have been really excited to see the thoughtful approach related to ease of adoption around efforts like hydra-in-a-box, and the number of projects working with vagrant and puppet to simplify the initial process of installation and evaluation.  we have communities around projects like archivesspace, dspace, hydra, koha, kuali ole; thinking hard about how to lower the barriers not just for adoption, but around the long-term support of a project.  but there is so much left to be done.  we have some exciting work taking place in this community and some big problems yet to be tackled (linked data, cough) but i’d like echo davidson and casden’s suggestion, that we resolve in 2016 to “extend the open source software values of collaboration and transparency to include the wide and affordable distribution of software.”[2]  the community does great work – let’s make it easier for everyone to use it. issue summary: the editorial committee is pleased to submit issue 31 to the community.  this issue includes 7 articles, covering issues related to data manipulation, migration, and reconciliation – as well as discussions around best practices and feature development.  we encourage you to explore this issue, engage in the comments, and reach out to the many fine authors that contributed their work to this work. articles: bret davidson and jason casden, beyond open source: evaluating the community availability of software michael sutherland, rss feed 2.0: the crux of a social media strategy ruth tillman, extracting, augmenting, and updating metadata in fedora 3 and 4 using a local openrefine reconciliation service gregory wiedeman, practical digital forensics at accession for born-digital institutional records keith gilbertson and liz mcvoy, video playback modifications for a dspace repository katherine perdue, bringing our internet archive collection back home: a case study from the university of mary washington rainer simon, leif isaksen, elton barker, and pau de soto cañamares, peripleo: a tool for exploring heterogeneous data through the dimensions of space and time   –tr references [1] reese, terry, purposeful development: being ready when your project moves from ‘hobby’ to mission critical  http://journal.code4lib.org/articles/6393 [2] davidson, bret and casden, jason.  beyond open source: evaluating the community availability of software.  http://journal.code4lib.org/articles/11148 subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – the icap (interactive course assignment pages) publishing system mission editorial committee process and structure code4lib issue 2, 2008-03-24 the icap (interactive course assignment pages) publishing system the icap publishing system is an open source custom content management system that enables librarians to easily and quickly create and manage library help pages for course assignments (icaps), without requiring knowledge of html or other web technologies. the system’s unique features include an emphasis on collaboration and content reuse and an easy-to-use interface that includes in-line help, simple forms and drag and drop functionality. the system generates dynamic, attractive course assignment pages that blend web 2.0 features with traditional library resources, and makes the pages easier to find by providing a central web page for the course assignment pages. as of december 2007, the code is available as free, open-source software under the gnu general public license. by margaret mellinger and kim griggs icap project[1] background this project originated in a 2005 needs assessment conducted by oregon state university libraries (osul) to determine the best approach for connecting students with the wealth of information available to them through the library. we were particularly interested in lowering the barriers to library use for undergraduates searching for topical information. initially, we considered building undergraduate portals based on colleges or subjects, but as we gathered data, read about undergraduate research activities and habits, and listened to students, we changed the direction of our subsequent project.[2, 3] undergraduates, especially lower division students, identify more with courses they are currently taking than with a specific college or subject. they prefer clear starting points for library research, need to save time, and are very interested in resources that have been recommended by their instructors. rather than developing generic subject or college portals, we focused on library help pages that are tightly aligned with courses and assignments, include some interactive or web 2.0 features, and are also simple for subject librarians to create and maintain. the internal name we have been using for the project is “interactive course assignment pages,” or icaps. we investigated a variety of ways to deliver the course-centric, interactive web pages: library portal software, campus portal software, course management software, content management systems, and web page templates. because we wanted to lower barriers to participation, we were wary of implementing library portal software that would require students to log on and create profiles. we were interested in the possibility of developing a tool that integrated with campus portal software, but it was not an option, as our institution evaluated and declined to purchase a campus portal system. because we wanted to integrate library resources with particular courses, we carefully considered the campus course management system, blackboard, but there were several challenges with a potential blackboard integration. blackboard is not used universally across our institution, it is not as flexible and customizable as we envisioned our pages, and it was not apparent that the blackboard team had the resources to work with us on developing a new service. osul has a long-term plan to move the library web pages into a content management system, but it was not in place when we started this project. we tested static html templates as an early prototype of the course assignment pages, but they did not allow librarians to easily publish or re-use content. when we reached this stage of the project, we realized we needed to hire a programmer, and before we did so, we wrote use case scenarios and distilled from those the baseline project requirements. project requirements the project requirements fell into two large categories: end user requirements library course pages that are easily found on the osu libraries home page pages that are discoverable by course number pages that have content recommended by instructors as well as librarians a quick search box (federated search tool) modules that help students find books, articles, the best web sites, style guides, course reserves, news feeds link to chat service for live librarian help. modules to provide greater interactivity functional requirements generate course assignment pages with a consistent look and feel enable librarians without technical knowledge to create pages provide a web interface for content input provide the ability to choose which modules appear in the web page provide the ability to choose which databases to search in the federated search provide the ability to view rss feeds on the page provide the ability to generate an rss feed for the page allow the addition of learning objects (small movies that instruct students on research concepts or library tools ) provide the ability for librarians to change the content and headers of the modules manually manage the publishing of the web pages to the libraries’ web server provide permissions and authorization control support content re-use among course assignment pages provide a framework to support further interactive features and content re-use the programmer came on board in september 2006 and explored available proprietary and open source solutions that might meet the project requirements. the programmer reviewed the university of minnesota’s libdata and courselib [4] but found that the complexity and comprehensiveness of the software was not a match for this project. courselib requires a significant amount of set up, and we wanted instead to allow librarians to contribute and manage the content, without heavy administrative overhead. also, the interface seemed complicated and required users to have some html knowledge to format text. university of rochester’s library course builder [5] met the requirement of focusing on the course, but as the tool is built in coldfusion, osul would have had to buy the software, set up a special server, and find coldfusion programmers. this did not seem to be a good way to leverage server space and programmer capacity when the development environment at osul favors mysql, php and ruby on rails. while ithaca college’s subjects plus [6] uses php and mysql and would have fit into osul’s current development infrastructure, it would have needed to be extended to meet our project requirements, as it focuses on the broader subject level and is set up to build lists of recommended resources from their library catalog and database lists. after getting a clear picture of the computing environment at our institution and inventorying the available open source and proprietary software, we chose to build a custom solution. about the icap publishing system the icap publishing system is a custom content management system that enables librarians to easily and quickly create and manage course assignment pages (icaps). icap is a dynamic system; rather than forcing users to write code, it takes care of those details and lets librarians focus on the actual content to be displayed. this makes it easy for librarians to quickly add or change content, without requiring knowledge of html or other web technologies. the system’s unique features include an emphasis on collaboration and content reuse and an easy-to-use interface that includes in-line help, simple forms and drag and drop functionality. the system generates dynamic, attractive course assignment pages that blend web 2.0 features with traditional library resources, and makes the pages easier to find by providing a central web page for the course assignment pages. the publishing system includes a user interface that allows librarians to create, edit, copy, share, and publish course assignment pages. the tool also automates the task of sending the url of a page to the instructor. in the current workflow, an administrator registers a librarian and provides them with a temporary login / password. a librarian then logs in and can create new pages or edit their pages, modules, and account. once a page is created, librarians can add content, in what we call modules, to the page. currently, there are modules for a librarian and instructor profile, an assignment module, a rss module, a federated search module, course reserves module, custom content module and a widget module. the librarian profile can include an image and a chat box. both the librarian and instructor module provide contact information and links to related websites. the assignment module includes links to the assignment, syllabus and custom content. the librarian, instructor, and assignment modules are designed to communicate to the student that the pages are recommended by their instructor and to increase student awareness of the subject librarians. the custom content module includes a wysiwyg (what you see is what you get) text editor. the wysiwyg editor enables librarians to easily add links, images, lists and formatted text without using html or wiki mark-up. the widget module includes a code editor that can be used to embed 3rd party widgets, like youtube, google custom search box, and librarything. the module for osul’s federated search engine, libraryfind [7], can be customized to search selected databases. the course reserves module automatically pulls information from our catalog and the rss module aggregates and displays data from a rss feed. users can share their pages with other system users and publish their pages to the osul website through the interface. librarian interface: custom content interface: the course assignment pages the system creates are netvibes-inspired, with modules that chunk the content under helpful headers. the pages maintain a consistent look and feel that harmonizes with the rest of the library web site and ensure web and accessibility standards through the use of css and templates. we wanted the pages to have a consistent look and feel, so the system controls the design of the pages, but lets librarians choose from a one or two column layout. both content within the page and the layout of that content can be customized to a certain extent. the interfaces of both the icap system and course pages were carefully designed with all of our users in mind. example of icap page: designing the icap system the icap team was committed to employing user-centered design techniques while developing the project. user-centered design (ucd) is a process in which the user’s needs, wants, and limitations are considered at every stage of the development cycle. it was necessary to involve our three user groups, librarians, students and faculty, to increase buy-in, facilitate using web 2.0 applications, avoid feature creep, and most importantly, improve the user interface. usability testing is an iterative process; it is important to conduct usability testing throughout the development cycle. by designing for our users and then testing with our users, we acquired built-in early adopters, quality assurance, and an invaluable knowledge base. the first phase of the project was to develop a design and define the basic functionalities of the interfaces based on the use-case scenarios for each user group. the student interface (the course assignment pages) was designed with the guiding principles of saving students’ time and helping them discover quality resources for their course work. we developed an html template that was used with students for three terms beginning in spring 2006. with the help of course instructors in five classes, we surveyed 130 students about their experience with the course assignment pages [8]. the results of this survey helped us identify what was working and what changes needed to be made to the pages. we found that some students were not using the pages because they did not know about them. we needed to make the pages more findable from the library home page and embed links and information about the pages in the course web pages and syllabuses. we also found that students needed help navigating through the many resources libraries offer and to address that we created pages that better supported both browsing and searching. students wanted to be able to connect to the reference librarian outside of the library, so we added more communication options to the pages. most importantly, the majority of the students responded positively to the pages and indicated that after they learned about them they did use them. we have continued to solicit feedback from students in the form of online surveys, chats, and emails and will soon undertake a more in-depth assessment of the pages and their effectiveness for students. next, we developed the icap tool. working with a graphic design student, we first developed a paper prototype of the icap tool’s interface. we started with a paper prototype because we did not want to commit to any one solution or development framework before we had a good idea of the librarians’ needs. the prototype demonstrated only the core functionalities, which allowed for more interaction with the system, such as interacting with error pages. we first tested the paper prototype with five librarians who had varied technical skills. we asked the librarians to perform five core tasks (create, edit, clone, share, email url) with the prototype while thinking aloud [8]. we also evaluated the tool on librarians’ success rate based on the time it took to perform tasks and the number of errors made. at the end of each task the librarians rated levels of satisfaction and ease of use. after completing the test, participants rated their overall impression of the tool and offered suggestions and feedback. the results from that initial study indicated that since osul librarians had been using dreamweaver and contribute, their mental models of the process of creating web pages did not match the tool’s interface. this resulted in confusion over terminology and workflow while editing and cloning (copying) a page. we changed the terms and workflow to align more closely with the familiar microsoft model of file operations (open, copy and save as). because the librarians could not find what they needed to complete some of the tasks, we reworked the interface to make navigation easier. one main goal of the tool was to alleviate technical gaps among the librarians. we hoped that we could use wiki-style editing to give the librarians control over the format of the custom content they create for the pages. but we found that the wiki-editing was too difficult for some and that the librarians preferred formatting text in the manner of microsoft word. using all these results, we built a prototype of the tool in ruby on rails and had three different librarians conduct the same test on the interactive interface, which uncovered more terminology confusion and additional workflow issues. after addressing these issues with iterative testing, we released the system in spring 2007. knowing when to stop testing and release the project is never easy. we used our use-case, goals and deadlines to define the initial beta release, knowing we were going to continue to develop the tool. during the beta release, we set up a forum for librarians to provide feedback and to facilitate open communication between the users and the development team. from the forum feedback and other conversations, we improved the workflow to make the tool easier to use and added new functionalities based on the needs and requests from the librarians. we have rolled out new feature sets at the end of every term since the first release and periodically offer training. a word of caution about following the “release early and often” mantra: when releasing beta projects it is important to manage the user group’s expectations and perceptions through clear communication. releasing without the full feature set can turn potential users off and decrease faith in the project, unless users trust that the beta product will evolve to suit their needs. even though we employed user-centered design techniques and iteratively conducted user testing, users still doubted that the tool would meet their expectations. we learned quickly that we needed to increase communication and user involvement. feedback is solicited in a variety of ways, including user testing, user training, a discussion forum, surveys, email, meetings, and other forms of personal communication. releasing new feature sets in response to user feedback restored faith in the project and created a sense of ownership in the product which built an unanticipated level of adoption. icap system architecture the icap system technology stack consists of ruby on rails, mysql, mongrel and apache. the icap system is implemented in ruby on rails (rails), an open source web application development framework.[9] rails provides all the components necessary to build complete database-backed web applications. rails is an mvc (model, view, controller) framework. the model-view-controller (mvc) design pattern separates the data access (model) and business logic (controller) from the data presentation and user interaction (view). rails provides ‘out of the box’ scaffolding which can construct most of the user interface needed for a basic website. rails targets a fast development cycle with zero turnaround time, which enabled us to iteratively test and immediately refine our prototypes. the icap system is mounted on an apache web server in conjunction with mongrel, an open-source http library and web server. the system uses mysql as the database and is locally hosted on unix servers. the database consists of 20 relational tables and rich complex relationships. for example, the resource table is a polymorphic table that uses rails :through clause for the associations of the different modules to the page they belong to. the use of a polymorphic table enables us to easily add new module types to the system without having to build multiple near-identical join tables and complex insertion methods to maintain the relationships. the following code snippet demonstrates the polymorphic relationships and the helper methods that ease the selection and insertion process of a page’s modules. /** * resource table migration * builds a polymorphic table that associates the different module types * (librarian, instructor, course reserves) to the page that they belong to. * the table uses the page id and the * module id and the module type to make the associations. **/ class createresources < activerecord::migration def self.up create_table "resources", :force => true do |t| t.column "page_id", :integer t.column "module_id", :integer t.column "module_type", :string ... end end /** * resource model * tell the class it has a virtual association by setting polymorphic to true. this * creates an interface that the modules can then associate with. we must also tell the * resources that they belong to a page. * **/ class resource < activerecord::base belongs_to :page belongs_to :resource, :polymorphic =>true ... /** *page model *tell the pages about its resources **/ class page < activerecord::base has_many :resources ... /** * assignment module model * associate the modules to resources as a resource so that it uses the module id * instead of the assignment id in the polymorphic table. * tell the module it belongs to a page through resources **/ class assignment < activerecord::base has_many :resources, :as => :resource, :dependent => :destroy has_many :pages, :through => :resources ... /** * page model * methods to add and get a page's modules **/ // get the collection of associated modules. each module is automatically cast to its model class def modules mods = resources.collect {|a| a.resource } return mods end //create the resource and build the association of the page to a module through resources def link(mod) resource = resource.create(:resource => mod) self.resources << resource end /** * what we can do now... **/ //get a page using rails dynamic finder page = page.find_by_title("english comp") #-> //create and add some modules to a page page.link(assignment.create(:name => "course assignment")) page.link(librarian.create(:name => "course librarian")) //get the names of the modules on this page page.modules.each do |module| module.name end #-> "course assignment", "course librarian" one requirement of the tool was to provide authorization and permission controls. there are currently two types of users in the system: authors and admins. authors can create new pages and modules and edit only the pages and modules belonging to them. an admin user can register new users, create new pages and modules, edit all the pages and modules, and grant ownership of any page to any user within the system. to secure a user’s account, the password runs through a sha1 encryption with a randomly generated salt value. once a user is authorized in the system, the user id is stored in a session variable. this session value is used in a rails before_filter that constrains the dynamic finder with the criteria that the owner of the page or module must match the current user. this prevents a user from editing pages and modules they do not own. other precautions exist in the code to protect against security risks. one particular security challenge we faced was how to allow the addition of 3rd party widgets to the system. to allow widgets, we have to let users embed html tags that can expose the database to cross-site scripting. we are in the process of enhancing our widget module with a custom sanitization method that uses complex regexs to strip out and rebuild the code snippets from popular widget sources. adding the sanitization process secures our database from malicious code while still allowing librarians to add some 3rd party widgets to their pages. another requirement of the tool was to provide interoperability with other applications. for example, we wanted the tool to be able to interact with our osul federated search engine, libraryfind, and oasis, the osul library catalog. to interact with libraryfind, we use its api to target the selected databases chosen by the librarian and to send an http request to the web service. the search results are displayed in a new window. pulling course reserves from oasis was not as easy. since we do not have access to the oasis database and it does not provide an api, we could not simply query it for the data. instead, we have implemented a screen scraper to accomplish the data transfer from oasis to the icap database. rails’ web scraping library, net/http, provided an excellent means for developing our screen scraping program. because each icap page is required to have a subject and course number (the required attributes of an oasis course reserve search) this made it easier to process the resulting display output from the catalog, extract the desired data, and pass it on to the icap system for use in a course assignment page. since rails provides so much help with building the back end, we were able to place a high priority on the user interface for the icap system. ajax and css are well integrated into the rails framework, which made adding features like drag and drop, inline help, simple forms, and helpful feedback to the tool’s interface faster and easier. the first release of the icap publishing tool did not allow the librarians to arrange the modules on a page. we quickly found out that this was a flaw in the design and we needed to implement a way for them to do so. one option was to add a position field in the module database and have the user enter a number in a form to indicate the position on the page. this can be confusing, especially when working in a grid format, and did not test well. based on feedback, we decided to add the ability to drag a module around on a page to arrange it. the first code snippet below is the portion of the view that lists a page’s modules. the second snippet is what we added to the code to provide the drag and drop functionality: /** * edit modules view * list of a page's modules in a two column layout * without drag and drop **/ <ul id = "right"> <%@mods_right.each do |mod| %> <li class = "list-item"> <span class="list-title" ><%=truncate(h(mod.module_title) ,30) %></span> </li> <% end %> </ul> <ul id = "left"> <%@mods_left.each do |mod| %> <li class = "list-item"> <span class="list-title" ><%=truncate(h(mod.module_title) ,30) %></span> </li> <% end %> </ul> /** * edit modules view * list of a page's modules in a two column layout * with drag and drop **/ <ul id = "right"> <%@mods_right.each do |mod| %> <li id = "item_<%=mod.id%>" class = "list-item"> //add a unique id for each module <span class="list-title" ><%=truncate(h(mod.module_title) ,30) %></span> <span class = 'handle'> drag & drop </span> </li> <% end %> </ul> <ul id = "left"> <%@mods_left.each do |mod| %> <li id = "item_<%=mod.id%>" class = "list-item"> //add a unique id for each module <span class="list-title" ><%=truncate(h(mod.module_title) ,30) %></span> <span class = 'handle'> drag & drop </span> </li> <% end %> </ul> /**create a sortable element out of each list. this makes each item in the list (through the unique ids) * draggable and each list area droppable. * it also highlights an item when it is moved to provide feedback and calls a method * that saves the new order of the module. **/ <%= sortable_element 'left', :url => {:action => "save_order" }, :complete => visual_effect(:highlight, 'left'), :handle => 'handle', :containment => ['left', 'right'], :constraint => false %> <%= sortable_element 'right', :url => { :action => "save_order" } :complete => visual_effect(:highlight, 'right'), :handle => 'handle', :containment => ['left', 'right'], :constraint => false %> example of drag and drop interface: in addition to reducing development time, effort, and lines of code, rails also lends itself nicely to an agile development cycle. as expressed by the agile alliance [10], agile development emphasizes: “individuals and interactions over processes and tools. working software over comprehensive documentation. customer collaboration over contract negotiation. responding to change over following a plan.” agile development can work well in the library setting. software companies have staff dedicated to specific software development roles, such as business analysts, designers, programmers, project managers, etc. but most libraries, osul included, have small technology departments where staff must fill more than one role in the software development process. agile development methods lessen staff overhead needed for software development by reducing processes and documentation. the icap project team is small, consisting of two reference librarians and a programmer, and has fairly broad decision-making ability that helped streamline the development cycle. although the team was small, agile development also allowed us to involve all stakeholders, and respond quickly to their feedback. challenges early feedback exposed design flaws that could have kept the tool from being adopted by the librarians. three refinements were particularly important: the ability to move modules around on the page, the ability to choose one or two column layouts, and the ability to forgo an added chat box or to replace a generic chat box with their own. after we changed the design, the tool has gained wide acceptance with the librarians, evidenced by the large number of pages created in the first term following the changes. as librarians used the creation tool more, minor bugs and workflow issues were found and addressed. later feedback led to further refinements, such as the ability to add third-party widgets to the pages, and changes to functionalities of the wysiwyg editor. there were some critical junctures in the project. the first involved getting the right skills and personnel in place to get the project off the ground. the second was the push-back about how the tool would work, what features would be included, and how it would change librarians’ workflow and level of control over course assignment web pages after the initial release. the team learned firsthand that people need time to adopt new software, and that a high level of user involvement is crucial in building acceptance for a new tool. the third was when libguides [11] became available. osul took a careful look at whether or not to continue developing the icap publishing system. we decided to go forward because we had a functioning tool in place, and also because we planned to share our open source code with the library community. conclusion in the coming months we will add features from our original use case, such as a feedback/comments module, and extend the tool’s functionality to include creating subject research guides. we will also build in search-engine optimization features and supply a means to access web analytics from within the icap system. some of these features will then be added to the open source code for this project. additionally, we will begin a new round of usability testing with students and integrate the feedback into the icap system. while we continue to evaluate this project and the tool, the icap publishing system has met our initial project goals. the primary goal of this project is to better serve students, and the assessment data collected from students via surveys, email and use statistics suggests that we are on the right track. we will continue to gather evaluative feedback from student users, learning more about their searching preferences and research habits as we do. we also met our goal of providing librarians with an easy-to-use publishing system. librarians have adopted the tool, enjoy using it, appreciate that it helps them incorporate web 2.0 features, and find that it saves them time when creating course assignment pages. the icap publishing system gives osu libraries an updated way to connect students with relevant, quality resources at their point of need. an additional benefit is that the new tool and the course pages it generates gives librarians something to showcase with faculty, who have responded positively, collaborating with librarians on pages, using the pages in class, and requesting more pages. the library as a whole built new capacities as a result, gaining experience with user-centered design, usability testing, software development cycles, and project management. the cumulative effects of the icap project go beyond the tool: osul’s future software development projects will benefit from these new capacities. notes [1] icap publishing system: http://ica.library.oregonstate.edu/about/index.html [2] nichols, jane and margaret mellinger. (2007) portals for undergraduate subject searching: are they worth it? portal: libraries and the academy 7(4) 481-490. [3] reeb, brenda and susan gibbons. (2004). students, librarians, and subject guides: improving a poor rate of return portal: libraries and the academy 4(1):123-130 [4] libdata. http://libdata.sourceforge.net/ [5] library course builder. libcb http://sourceforge.net/projects/libcb/ [6] subjectsplus. http://www.ithacalibrary.com/subsplus/ [7] libraryfind: http://libraryfind.org [8] see: icap system wiki. http://wiki.library.oregonstate.edu/confluence/display/icateam/usability and http://wiki.library.oregonstate.edu/confluence/display/icateam/assessment [9] ruby on rails. http://www.rubyonrails.org/ [10] agile alliance. http://www.agilealliance.org/ [11] libguides. http://www.springshare.com/libguides/ author biographies margaret mellinger is an engineering librarian at oregon state university libraries, and the lead of the icap team. (margaret dot mellinger at oregonstate dot edu) kim griggs is a programmer/analyst at oregon state university libraries, and the programmer of the icap publishing system. (kim dot griggs at oregonstate dot edu) subscribe to comments: for this article | for all articles 3 responses to "the icap (interactive course assignment pages) publishing system" please leave a response below: tomkeays, 2009-04-06 i noticed that icap is now called library à la carte and that the first link in the references is now broken. the new url is http://alacarte.library.oregonstate.edu pi (weekly) « pi en second life, 2012-05-26 […] the code4lib journal – the icap (interactive course assignment pages) publishing system […] alusión (weekly) | alusión…llamada virtual, 2012-05-26 […] the code4lib journal – the icap (interactive course assignment pages) publishing system […] leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – conspectus: a syllabi analysis platform for leganto data sources mission editorial committee process and structure code4lib issue 52, 2021-09-22 conspectus: a syllabi analysis platform for leganto data sources in recent years, higher education institutions have implemented electronic solutions for the management of syllabi, resulting in new and exciting opportunities within the area of large-scale syllabi analysis. this article details an information pipeline that can be used to harvest, enrich and use such information. by david massey, thomas sødring introduction in recent years, higher education institutions have implemented electronic solutions for the management of syllabi [1]. this opens up for the development of new student experiences and workflows, as well as possibilities for integration with learning management systems as well as other systems. from a research perspective, new and exciting opportunities within the area of large-scale syllabi analysis will materialize once a significant portion of reading lists are digital and publicly available. one such opportunity will be the possibility to analyze and compare syllabi from both intraand inter-institutional perspectives, as well as intraand inter-domain perspectives. the conspectus project has a goal to develop an open source etl-style information pipeline for syllabi analysis, and has three overarching functionalities: harvesting, enrichment, and dissemination. there are a number of systems for managing syllabi available, including leganto [2] from exlibris, aspire [3] from talis, keylinks [4] from cla and kortext, and bluecloud lists [5] from sirsidynix. many of these have a published api that may allow for the automated extraction of data. conspectus is currently limited to using metadata extracted from a server hosting the exlibris leganto syllabus management system, but it should be possible to extract metadata from the other systems as well. the article is organized into four main sections. in the first section, we introduce the high-level architecture of conspectus. in the second section, we describe the leganto api and our harvesting approach. in the third section, we describe an enrichment process using a selection of data sources. in the final section, we discuss issues regarding dissemination, transformation of the data to rdf, and the general use of the pipeline. conspectus architecture conspectus is an information pipeline to harvest, enrich, use and disseminate digital syllabi information. figure 1 details an architectural overview of conspectus. the pipeline consists of three stages. the harvesting stage downloads data from any number of leganto servers. currently, we only have access to the local oslomet (oslo metropolitan university, https://www.oslomet.no/en) leganto server, however once data processing agreements can be reached with other institutions we expect the collection of data to increase significantly. in the context of norwegian higher level institutions, each institution has its own leganto server. the enrichment stage makes use of additional sources, that include open data sources available on the internet, as well as information procured from local administrative systems at the university. finally, the dissemination and use stage makes the processed data available to data consumers. figure 1. conspectus architecture. the leganto system provides access to course syllabi, proposed reading lists, instructors, and syllabus comments. figure 2 depicts the relationship between these resources. the leganto api exposes endpoints for multiple entities, however, only the course, reading list and citation entities are deemed to be relevant. figure 2. resources available from the leganto api. harvesting conspectus currently only has support for the leganto api and can download syllabus and related information from a leganto server that complies with version 1 of the api [6]. a specially purposed harvester retrieves syllabi metadata, that is subsequently stored both as json text files as well as in a relational database. citation data is also aggregated in a single large csv file. conspectus employs this approach in order to support various tools and expectations of researchers. course data the highest level of the api provides a list of courses. listing 1.1 shows an example of a curl command to retrieve information on courses. the command consists of a straight forward http request with a hostname, context-path (almaws/v1) and entity type. the only additional requirement is the identification of an api key to interact with the api. the api is well-documented and relatively easy to use. the api supports result filtering using query parameters and allows users to retrieve results in both json and xml formats. note, all urls corresponding to internal resources have been replaced with the hostname https://example.com. curl "https://example.com/almaws/v1/courses?apikey=[redacted]&format=json" listing 1.1. curl command to get all courses from the api in json format. listing 1.2 shows a subset of the metadata belonging to a course. the information that is stored here includes details about the instructors, the department the course belongs to, the year the course occurs, a leganto identifier for the course, an identifier about the previous instance of the course, the official name of the course and identifiers used for search. the link field is a url that points to the current course. { "searchable_id ": [ "mbib4600", "ua_215_mbib4600_1_2021_vår_1" ], "name": "mbib4600 interactive information retrieval (2021 -v)", "academic_department ": { "desc": "institutt for arkiv -, bibliotek og informasjonsfag", "value": "215 _15_2" }, "year": "2021" , "id": "5798071290002212" , "link": "https :// example.com/almaws/v1/courses/5797973330002212" , "instructor ": [ { "last_name ": "doe", "first_name ": "john" }], "rolled_from ": "4729892920002212" } listing 1.2. abridged version of a payload describing a particular course from the oslomet leganto instance. some values included here are imported from other systems. these include information about the academic department, searchable identifiers and the official name of the course. there is also a field called primary_id (not shown) that is associated with an instructor. for brevity, some information has been excluded. this includes fields that appear to have relevance for internal system processing, fields indicating processing history, as well as fields that appear to not be in use. an example of a field that appears to not be in use is a field called campus. it is always empty. reading lists leganto supports the management of reading lists on a per-course basis, where multiple reading lists can be associated with a course. a reading list is a list of required, recommended or additional reading resources defined for a course. syllabus citations are associated with a reading list, so a course must have at least one reading list if it has a syllabus. listing 1.3 shows an example of a curl command to get the sole reading list for the spring 2021 mbib4600 course. the command shows how a reading list identifier (5797973440002212) is explicitly associated with a course (5798071290002212), and a request for a reading list must include the course identifier as well as the reading list identifier. an http endpoint is available that identifies all reading lists associated with a course. curl "https://example.com/almaws/v1/courses/5797973330002212/reading-lists/5797973440002212?apikey=[redacted]" listing 1.3. curl command to get the only reading list for the course mbib4600 (spring 2021). listing 1.4 shows the returned payload for a request for a reading list. from a bibliographic analysis perspective, there is little value in the data here. the contents of the description field is mostly a description of the study-program and the year the class (cohort) started, and does not provide any insight into the contents of the reading list. very few reading lists provide any information about the purpose of the reading list, while a few reading lists provide an overview of the number of pages that the reading list consists of. { "id": "5797973440002212" , "name": "mbib4600 interactive information retrieval (2021 -v)", "due_back_date": "2021-07-31z", "description": "mandatory and optional literature for mbib4600 (iir)", "link": "https://example.com/almaws/v1/courses/5797973330002212/reading-lists/5797973440002212", "status": { "desc": "complete", "value": "complete" }, "visibility": { "desc": "anyone restricted", "value": "open_to_world" }, } listing 1.4. abridged version of a payload describing an identified reading list from the oslomet leganto instance. (contents translated to english, for clarity) there are 5,759 courses registered in the oslomet leganto server (as of june 2021). these courses cover seven semesters of teaching (autumn and spring). new information about a course is generated each time a course is scheduled to be run. the course mbib4600 is, for example, registered three times. a course may change name, be retired and replaced by a similar course, retired without replacement, and new courses will be introduced. as such, the details presented here are aggregate data. 121 courses use more than one reading list. 109 courses have more than two reading lists. the fact that most courses have a single reading list, while only a small percentage (approx 2%) use more than one reading list, tells that the functionality is not widely used. it is unclear why so few courses have multiple reading lists. it may be an intentional approach by the course instructors, or it could be that they are unaware that the functionality is available. another reason could be that the practices employed in previous approaches to managing syllabus citations, e.g., citations in office document formats, led to a similar practice being continued in leganto. 812 courses (approx. 14%) have no reading list associated with them. a reading list is required for syllabus citations, so these courses have no citations associated with them in the leganto server. citations citations are references to a source of information used as part of a reading list. typically, citations make up part of the information requirements that provide students with the necessary skills and knowledge in order to successfully pass the assessment requirements when studying a course. from a bibliographic analysis point of view, the citations’ endpoint offers a trove of interesting information. listing 1.5 shows the curl command to get a particular citation used in the syllabus of the spring 2021 mbib4600 course. the command shows that each citation has a unique identifier (5933190900002212), that is explicitly associated with both the course (5798071290002212) and reading list identifier (5797973440002212). there is an http endpoint available that identifies all citations associated with a reading list. curl "https://example.com/almaws/v1/courses/5798071290002212/reading-lists/5797973440002212/citations/5933190900002212?apikey=[redacted]" listing 1.5. curl command to get a particular citation that is part of the reading for the course mbib4600 (spring 2021). the payload response to the request described in listing 1.5 is presented separately in listing 1.6 and listing 1.7. listing 1.6 describes the administrative context of the citation, while 1.7 details bibliographic information about the citation. administrative information includes details about whether the citation describes an article or book and that it is either physical or electronic. the secondary_type object details the type of object the citation is. this can, for example, be an audio recording, video, or manuscript. a peculiarity about the payload (not shown for brevity) is that there are ten fields named source1 to source10. from a data modelling point of view, it appears that such a source field is a multi-valued field and should be modelled as a separate entity in a 0:m relationship to the citation. { "id": "5933190900002212" , "link": "https://example.com/almaws/v1/courses/5797973330002212/reading-lists/5797973440002212/citations/5933190900002212" , "type": { "desc": "physical article", "value": "cr" }, "status": { "desc": "complete", "value": "complete" }, "secondary_type": { "desc": "article", "value": "cr" }, "section_info": { "visibility": false , "description": "supplementary", "section_tags": { "link": "", "total_record_count": 0 }, "section_locked": false , "id": "5797973690002212" , "name": "optional literature" }, "note": [], "copyrights_status": { "desc": "not determined", "value": "notdetermined" } } listing 1.6. an abridged example of a citation belonging to the reading list presented earlier. each citation has a metadata object that contains detailed information about the citation. listing 1.7 shows an example of a metadata object. "metadata": { "author": "taylor , robert s", "issn": "00100870" , "pmid": null , "start_page": "251" , "publication_date": "2015-03-01" , "end_page": "267" , "doi": "10.5860/crl.76.3.251" , "issue": "3" , "title": null , "isbn": null , "journal_title": "college & research libraries", "chapter": null, "article_title": "question negotiation and information seeking in libraries", "note": null , "part": null , "volume": "76" , "publisher": null , "year": "2015-03-01" , "edition": null , "pages": "251-267" , "source": "https://example.com/10.5860/crl.76.3.251" } listing 1.7. an example of a metadata object belonging to a particular citation. most of the fields in the metadata object are self-explanatory. there are, however, some data quality issues present. the publication_date field does not follow a standardized date format. the variations here include “yyyymmdd”, “yyyy-mm-dd”, “yyyymm”, “yyyy-mm”, “yyyy”, “[yyyy]”, “cop. yyyy”, “cyyyy” and a text value indicating that the year is unknown. there are other variations that make the field difficult to search. the year field also does not appear to adhere to a particular standardized description, with a number of variations. the year field can be a date field with similar variations of “yyyymmdd” (found in publication_date), as well stating the month in textual form. there are two title fields, title and article_title. if a citation is identified as an article, then article_title is used, otherwise title is used. there are 98,549 citations registered in the oslomet leganto server (as of june 2021). the maximum number of citations registered for a reading list is 142. 31 reading lists have more than 100 citations registered. 295 reading lists have only one citation. in some cases, the citation is a single note stating that there is no mandatory syllabus. in other cases, the syllabus consists of a single book. the average number of citations per reading list is 21. there are 72 notes associated with citations. that equates to approximately 0.07% of citations having a note attached to them. the notes appear to be comments to students, but some are messages to the library staff. the harvester the harvester is a python (version 3.6) script and is distributed as free and open source software. it is available from gitlab [7]. the harvester requires the use of a mysql relational database to store information for processing. the domain model implemented in the database reflects the json payloads returned from the api, where array results are implemented as 0:m relationships. the code repository also includes the ddl to create the database structure in mysql. the leganto api makes limited use of http-linking and, as such, it can be seen as an implementation of a level 2 hateoas api [8]. any client interacting with the api must themselves build http links to the various objects based on provided identifiers. a level 3 hateoas api would allow a user to traverse the api endpoints using built-in links in the returned payloads. the harvester works by first downloading all courses available from the api. any request to retrieve courses will have a count field detailing the number of courses available via the api. this allows for an upper bound in relation to the number of courses to retrieve. listing 1.8 shows an example of a command using the harvester to download all course data from the leganto api. the option of setting the institution allows the script to download from multiple leganto servers belonging to various institutions. python main.py --institution=oslomet --what=course --start=1 --end=5760 listing 1.8. command to run harvester to retrieve courses. the ordering of results is deterministic based on the values in code and section. as such, they are not related to time and new courses will not appear at the end of the list of courses when querying the api. it is possible, however, to add a query parameter called order_by to assign a particular ordering to the result set. this has not been implemented in the current version of the api. the harvester retrieves courses, ten at a time, until all courses are downloaded. this is a courtesy approach, to reduce server side processing. the harvester downloads course information and persists it to a local mysql database. it also saves the retrieved json data as a json files using the course identifier as the file name. leganto can also provide the data as xml, but the harvester only requests json. once the required course information has been downloaded, the harvester can be re-run to download reading lists. listing 1.9 shows the command to collect reading list data. python main.py --institution=oslomet --what=reading-list --start=1 --end=5760 listing 1.9. command to run harvester to retrieve reading lists. reading lists are downloaded in the same order courses are persisted in the courses table in the database. the retrieved reading lists are also stored as the harvester can be re-run to download related citations. listing 1.10 shows the command to download citations. recall that the citations are only retrievable using a combination of the course and reading list identifiers. python main.py --institution=oslomet --what=citation --start=1 --end=5760 listing 1.10. command to run harvester to retrieve citations. the harvester avoids a download-everything approach, as potential problems (for example, server unavailability) will likely require the script to download everything repeatedly. an incremental approach is more robust and makes it easier to download subsequent data when the leganto server is updated with new course information at the start of a semester. with more experience from working on multiple leganto apis, it may be worth considering alternative approaches. in norway, higher education institutions have a unique code available in the leganto data. this can be seen in listing 1.2 where the value in the academic_department field is 215_15_2. such unique identifiers may not be in use in other countries, so the script allows additional tagging of institutional information. enrichment the syllabus metadata are enriched using both internal (university administrative system) and external data sources using various protocols, apis and file downloads. the information pipeline addresses some general interoperability problems faced when using heterogeneous data sources. the external data sources currently used are crossref, microsoft academic graph (mag), directory of open access journals (doaj), the norwegian register for scientific journals, series and publishers (nsd), and the bibsys union catalog. the purpose of the enrichment data is twofold. firstly, to improve existing data by replacing semi-structured data (e.g., author strings) with structured, machinereadable data. secondly, to add additional properties, such as language and subject descriptors, and thus enabling enhanced analysis. the results of the enrichment stage, presented here, is based on a subset of the downloaded data and limited to 25,333 citations from the 2020-21 academic year. identifiers a successful enrichment process depends on the reliable interlinking of metadata descriptions across data sources, and requires the use of shared identifiers. table 1 lists the four relevant identifiers available in the leganto citation metadata that can be used in an enrichment process. the metadata management system (mms) identifier differs from the other three identifiers, in that it is an internal identifier, linking the syllabus citation metadata to the corresponding metadata description in the university library management system. note that the mms_id field is only found in the metadata object of the citation when a link to the library system is made. the syllabus metadata schema also allows for a pubmed (https://pubmed.ncbi.nlm.nih.gov/) identifier, but no values are found in the retrieved metadata. table 1. identifiers found in the leganto metadata. identifier acronym identifies international standard book number isbn books international standard serial number issn journals metadata management system mms id books and journals digital object identifier doi articles and books we limit the current pipeline to citations that describe either books or articles. together, they account for almost 90% of the citations. 57.6% of the citations are books and 30% are articles. the remaining 13% include web pages and other document types. the issn field identifies journals rather than individual articles. the pipeline has an assumption that journal metadata may be able to provide additional relevant information about an article. for example, the language accepted by the journal can be used to assign a language code to an article that is published in the journal. approximately 90% of the article citations contain an issn, whereas only approximately 60% contain a doi. the dois and issns complement each other, enabling the use of a greater number of enrichment sources. seven percent of the articles contain neither identifier, and therefore cannot be linked to enrichment sources. a similar identifier coverage is found for book citations, where approximately 90% of the citations contain one or more isbn. the next question the pipeline explored is which sources should be used to enrich the leganto metadata? this leads to a series of additional questions: which sources can be queried using the identifiers (issns, dois and isbns)? which sources offer high levels of coverage for the target articles and books? which sources contain relevant metadata? our current selection of sources are listed in table 2. this selection is only an example of possible sources, and shows which identifier is used to query the source and the access method used. the table also shows the level of coverage, that is, what proportion of the syllabus identifiers that are found in the source. the low level of coverage for doaj suggests that only a limited number of the articles are published in open access journals. the final overall coverage column shows how many of the books or articles that can be linked using the source. the ceiling for this value is the identifier coverage discussed earlier. the crossref and mag coverage for dois is high, but the overall coverage is lower because fewer article citations contain dois than issns. table 2. current external enrichment sources. source publication type identifier method coverage overall coverage bibsys book isbn sru 94.7 91.2 bibsys journal issn sru 96.8 81.9 doaj journal issn download 11.3 14.8 crossref journal issn api 85.3 71.2 nsd journal issn download 92.9 80.4 crossref article doi api 94.4 63.8 mag article doi api 94.5 63.8 we continue with a closer examination of the enrichment sources. how they are accessed, and what data do they provide? bibsys union catalogue oslomet is a part of the bibsys union catalog. the catalog contains bibliographic metadata describing the holdings of 80 norwegian higher education and special libraries. the source can be queried using the sru (search/retrieval via url) protocol (http://www.loc.gov/standards/sru/). isbn, issn and mms identifiers can be queried using separate search indexes [9]. for example, an sru query for the example article’s issn is shown in listing 1.11. curl "https://bibsys.alma.exlibrisgroup.com/view/sru/47bibsys_network?version=1.2&operation=searchretrieve&recordschema=marcxml&query=alma.issn=00100870" listing 1.11. url for issn lookup in the bibsys sru service. results can be returned in a variety of standard formats including marc21 xml [10], dublin core [11] and metadata object description schema (mods) [12]. we use the marc21 format as this provides the most detailed data. an abridged version of the returned data is shown in listing 1.12. <?xml version="1.0" encoding="utf-8" standalone="no"?> <recorddata> <record xmlns="http://www.loc.gov/marc21/slim"> <leader>01443 cas a2200385 c 4500</leader> <controlfield tag="001">999101706734702201</controlfield> <controlfield tag="005">20180613120321.0</controlfield> <controlfield tag="007">ta</controlfield> <controlfield tag="008">150417d19392013xx#b||p||||||||||||0eng|d</controlfield> <datafield ind1=" " ind2=" " tag="022" > <subfield code="a">0010-0870</subfield> </datafield> <datafield ind1=" " ind2=" " tag="080" > <subfield code="a">027.7(05)</subfield> </datafield> <datafield ind1="0" ind2=" " tag="130" > <subfield code="a">college &amp; research libraries (trykt utg.)</subfield> </datafield> <datafield ind1="1" ind2="0" tag="210" > <subfield code="a">coll.res.libr.</subfield> </datafield> <datafield ind1="1" ind2="0" tag="245" > <subfield code="a">college &amp; research libraries :</subfield> <subfield code="b">c& amp;rl</subfield> </datafield> <datafield ind1=" " ind2="7" tag="650" > <subfield code="a">bibliotekvesen</subfield> <subfield code="v">tidsskrifter</subfield> <subfield code="2">tekord</subfield> </datafield> </record> </recorddata> listing 1.12. an abridged marc21 xml record returned by a bibsys sru query. directory of open access journals the directory of open access journals (doaj, https://doaj.org/) is a database with metadata descriptions of over 15,000 open access journals. the database is available for download (https://doaj.org/public-data-dump/journal) in both csv and json formats. doaj is a rich source of information about open access journals. it also provides subject descriptors, as is shown in listing 1.13 of metadata on our example article. { "bibjson":{ "keywords":[ "library science", "information literacy", "higher education" ], "subject":[ { "code":"z", "scheme":"lcc", "term":" bibliography . library science . information resources" } ], "eissn":"2150-6701" , "language":[ "en" ], "title":"college and research libraries", "pissn":"0010-0870", "license":[ { "nc":true, "nd":false, "by":true, "type":"cc by-nc", "sa":false } ], } } listing 1.13. abridged version of data from doaj. crossref crossref (https://www.crossref.org/) is a doi registration agency that was launched in early 2000 as a cooperative effort among publishers to enable persistent cross-publisher citation linking of online academic journals. the crossref rest api allows doi lookup [13]. a lookup of the example article’s doi is shown in listing 1.14. curl "https://api.crossref.org/works/10.5860/crl.76.3.251" listing 1.14. doi lookup in crossref api. the returned metadata is primarily a source of structured reference and citation data. however, as listing 1.15 shows, the returned json data also offers structured information on article authors. { "message" : { "type": "journal-article", "created": { "date time": "2015-03-13t14:06:42z" }, "page": "251-267", "title": [ "question negotiation and information seeking in libraries" ], "volume":"76", "issue":"3", "author": [ { "given":"robert s.", "family":"taylor", "sequence":"first", "affiliation ":[] } ] "container-title": [ "college & research libraries" ] } } listing 1.15. abridged version of data from crossref. the norwegian register for scientific journals, series and publishers the norwegian register for scientific journals, series and publishers (nsd, https://dbh.nsd.uib.no/publiseringskanaler/forside.action?request_locale=en) is available for download as a csv file. the register contains metadata on approximately 35,000 journals. the csv file contains 36 columns. to improve readability, listing 1.16 uses a column name colon value format. for clarity, the column headings have been translated from norwegian. relevant columns are language, npi subject area (https://npi.nsd.no/fagfeltoversikt), npi subject field and level 2021. the level is a classification of journals which is used in publishing metrics. level 2 journals are deemed the most prestigious. nsd journal_id : 439235 original title : college & research libraries international title : college & research libraries print issn : 0010 -0870 online issn : 2150 -6701 open access : doaj npi subject area : social sciences npi subject field : library and information science level 2021: 2 itar_id : 7656 nsd publisher_id : publishing house : publisher : american library association publication country : usa language : conference report : 0 established : 1939 listing 1.16. abridged version of data from the nsd register for scientific journals (contents translated to english, for clarity). microsoft academic graph the microsoft academic graph (mag) “is a heterogeneous graph containing scientific publication records, citation relationships between those publications, as well as authors, institutions, journals, conferences, and fields of study” [14]. microsoft announced in may 2021 that the project will be discontinued from the end of 2021 [15]. ourresearch have recently announced [16] that they will reuse the mag data set in the openalex service. conspectus uses the microsoft academic knowledge rest api [17] to query the graph for dois. the evaluate method [18] is used to query the graph. the entity attributes to return are listed in a commaseparated string of codes [19]. for example, ’f’ contains data on the field of study. listing 1.17 shows a query for the doi of our example article. curl "https://api.labs.cognitive.microsoft.com/academic/v1.0/evaluate?subscription-key=[redacted]&attributes=id,ti ,y,cc,aa.aun,aa.auid,f.dfn,j.jn&expr=doi==’10.5860/crl.76.3.251’" listing 1.17. doi lookup in teh mag api. the list of field of study names, shown in listing 1.18, is a possible source of subject descriptors for the article. {"expr":"doi==’10.5860/crl.76.3.251’", "entities":[ "ti":"question negotiation and information seeking in libraries", "y":1967, "cc":676, "aa":[{ "aun":"robert s taylor", "auid":2140052427 }], "f":[ {"dfn":"reference interview"}, {"dfn":"information seeking"}, {"dfn":"negotiation"}, {"dfn":"computer science"}, {"dfn":"search theory"}, {"dfn":"social communication"}], "j":{ "jn":"college & research libraries" } } ]} listing 1.18. abridged version of data from microsoft academic graph. case: analysing the languages of syllabi citations the leganto citation metadata does not include information about the language of the books or articles. an analysis of languages used in the university’s syllabi is therefore dependent on enrichment from external data sources. figure 3 shows language codes gathered from the five external enrichment sources for our example article. successfully assigning a language code to a citation depends on four factors. firstly, that the citation includes one or more identifiers. secondly, that sources can be queried using the identifier. thirdly, that the sources contain metadata on the citation. finally, that the sources contain information on the language of the citation. this illustrates how different forms of completeness – schema, property, and population – can negatively affect interoperability across sources. see [20, p. 103] for definitions of the three forms of completeness. population completeness is also related to the concept of coverage, that was discussed earlier. figure 3. article, journal and language field in external enrichment sources. the example article is identified with a doi, and both mag and crossref contain metadata on the article. the mag schema, however, does not include a language field. thus, enrichment is hampered by inadequate schema completeness. the crossref schema does include language information, but 18.8% of the matched crossref records, including our example article, have no language field. this is an example of an issue concerning property completeness. at the journal level, the journal is also identified with an issn. the nsd csv file includes a language column, but 46.1% of the journals have no language assigned. all the doaj records contain one or more language codes. finally, almost two percent of the bibsys records have und (undefined) as a language code. it is clear that the property completeness varies across the sources. the example article and journal from listing 1.7 are present in all five of the sample data sources, but as we saw earlier (see table 2), this is not the case for all articles and journals. population completeness is therefore a data quality issue that can impede effective enrichment. interoperability issues also arise at the data value level. our sample sources express language in a variety of ways. bibsys uses marc21 to describe journals and books. marc21 stores the document’s language in position 35-37 of the 008 control field (https://www.loc.gov/marc/bibliographic/bd008.html). the library of congress maintains a registry of the three letter language codes (https://www.loc.gov/marc/languages/language_code.html). documents that are written in multiple languages are given mul. the value for our example article, as shown in line 6 of listing 1.12, is eng. languages in the nsd data file are specified with norwegian names rather than codes, for example, engelsk (eng. english). the documentation of the crossref metadata api json format [21] does not include a language field. the returned data for doi lookup, however, includes a language field for many of the documents. the value is a two-letter code which appears to correspond with the iso 639-1 standard, e.g., en. the doaj json file contains a language field with a list of language codes. the value is a two-letter code which appears to correspond with the iso 639-1 standard. several language codes can be assigned to a journal, for example: "language": ["nb", "da", "en", "nn"], we map the language codes and names to a common coding system. finally, we select one of the codes to represent the language of the citation in our analysis. factors affecting the choice of code include: codes for articles (dois) override codes for journals (issns); agreement between sources; and single codes overriding multiple codes. figure 4 uses the language codes to display the proportion of articles, chapters and books written in norwegian at different academic levels. not surprisingly, we see a fall in the proportion of norwegian language citations at the master and phd levels. a similar pattern is shared across the university’s four faculties [22]. the academic level of the course is not contained within the leganto data. internal enrichment data from the university administrative systems is therefore combined with the language codes to generate this analysis. figure 4. proportion of norwegian language citations for five academic levels across the university’s faculties. dissemination and use a project that is very relevant to our pipeline is open syllabus (https://opensyllabus.org/). the open syllabus project has collected syllabi from numerous institutions across the world and provides interesting insights. in many ways, the open syllabus project is a great example showing how such data can be disseminated and used. conspectus differs here in that our approach is more fine-grained and tries to deal with data quality and other enrichment issues to provide a platform for syllabus analysis. we expect, however, that the conspectus pipeline will retrieve relevant citation information from leganto servers and submit them to the open syllabus project. the conspectus approach takes existing syllabi metadata that are currently semi-available and aims to make them more openly available in machine-readable sources for others. this builds upon the development of a generic data model to allow for the publication of data using standardized approaches. currently, the project has explored rdf (https://www.w3.org/rdf/) to make data available to other users. an example use-case is to facilitate libraries undertaking bibliometric research of syllabi. semantic representation of the data we have considered two ontologies found in the literature. firstly, the curriculum course syllabus ontology (ccso, https://vkreations.github.io/ccso/#d4e3502) discussed in [23]. secondly, ontosyllabus (https://jachicaiza.github.io/ontologydoc/) described in [24]. [25] and [26] also discuss curonto: an ontological model for curriculum representation, but we have been unable to locate and further assess this ontology. we decided to combine ccso with schema.org (https://schema.org/). ccso is used to model the non-bibliographic entities, such as courses, departments and syllabi. schema.org is used to model the bibliographic data, such as articles, chapters and books. table 3 shows the mapping between the merged leganto and enrichment data and the classes and properties from the chosen ontologies. the initial codes, for example cit.m in cit.m.author, in the leganto field column refer to the resource (citation) and the object (metadata) where the field is located. the codes used in the source column are l for leganto, i for enrichment data from the internal administrative system, and e for external enrichment data. the prefixes used in the ontology columns are c for ccso,s for schema.org, and x for local example properties. figure 5 shows the classes we used in transforming the data to rdf. the schema.org model uses publicationvolume and publicationissue to link articles to journals. currently, we simplify this by linking the article directly to the journal. this results in the omission of the leganto volume and issue fields. we also omit the ccso programofstudy class, and use a locally minted predicate (ex:academiclevel) to express the academic level the course is taught at (bachelor, master or phd level). the relationship between the article and the journal, or between the chapter and the book, is not explicitly expressed in the leganto data. this is identified by the value of the secondary_type sub-fields (see lines 12-15 of listing 1.6) and the presence of additional leganto fields in the metadata object, such as journal_title and article_title. in addition to facilitating secure interlinking between data sources, identifiers are crucial for transforming textual labels to rdf entities. for example, the lack of identifiers for authors means we use schema:author for a textual label, i.e., a data property, rather than its intended use, to link to an independent entity, i.e., an object property. we hope that enrichment sources can provide the author identifiers that will allow the automated creation of author entities. alternatively, other reconciliation techniques could be employed to create the entities. richard wallis identified entity reconciliation as the key missing piece in the process of transforming legacy cultural heritage data to linked data in a recent blog post [27]. the code of the course is not available as a separate field in the leganto data, but the course name follows a common pattern [28] that allows the code to be extracted. such pattern-based extractions are marked with regex in the notes column of the mapping table. the course code plays an important role as it is a shared identifier between leganto and the university administrative system. the leganto api returns the reading lists that are associated with a course. the links between course, reading list and citations can be added when retrieving data. alternatively, the citation metadata (see line six of listing 1.6) contains a link field that contains the leganto identifiers for the linked course and reading list. figure 5. rdf classes and object properties. the about property in schema.org is used to assign a subject to a citation. it is an object property that links a thing or the appropriate sub-class of thing to a citation. again, this depends on the creation of an entity to represent the concept. we show a simple example of subjects from mag listed as strings. a successful mapping and merging of subjects from a variety of controlled and uncontrolled vocabularies across languages is a major challenge that goes beyond the simple mapping that is utilized to merge language codes. the rdf turtle data for the example article, and related reading list, course and other entities are shown in listing 1.19. @prefix schema : <https://schema.org/> . @prefix ccso : <https://w3id.org/ccso/ccso/> . @prefix ex: <https://example.org/> . <http://example.org/institutions/4fc1875ac49b43fe5ab83121e27b8df5> a ccso:university ; schema:name "oslo metropolitan university" . <http://example.org/schools/a1c4253c96c559930e24f9f5349fd5a9> a ccso:school ; schema:name "faculty of social sciences" ; ccso:belongsto <http://example.org/institutions/4fc1875ac49b43fe5ab83121e27b8df5> . <http://example.org/departments/764639ea184bb49ed19de5b968a9ea12> a ccso:department ; schema:name "institutt for arkiv-, bibliotekog informasjonsfag" ; schema:identifier "215 _15_2" ; ccso:belongsto <http://example.org/schools/a1c4253c96c559930e24f9f5349fd5a9> ; ccso:offerscourse <http://example.org/courses/b4a9e82caa7f257b67f8834534fd2558> . <http://example.org/courses/b4a9e82caa7f257b67f8834534fd2558> a ccso:course ; ccso:csname "interactive information retrieval" ; ccso:code " mbib4600" ; ex:academiclevel "master" ; ccso:hassyllabus <http://example.org/syllabi/20b9736c7ff1df5e2001149abb201dc0> . <http://example.org/syllabi/20b9736c7ff1df5e2001149abb201dc0> a ccso:syllabus ; ccso:academicyear "2021-spring" ; ccso:haslm <http://example.org/articles/d73b25120b396a3cfefa1aa8a88003f0> . <http://example.org/articles/d73b25120b396a3cfefa1aa8a88003f0> a schema:article; schema:name "question negotiation and information seeking in libraries" ; schema:author "taylor, robert s." ; schema:datepublished "2015" ; schema:identifier [ a schema:propertyvalue ; schema:propertyid "doi" ; schema:value "10.5860/crl.76.3.251" ] ; schema:pagestart "251" ; schema:endpage "267" ; schema:pagination "251 -267" ; schema:about "reference interview", "information seeking", "negotiation", "computer science", "search theory", "social communication" ; schema:ispartof <http://example.org/journals/44a073d72c91b4aa5030553ae18d66fb> . <http://example.org/journals/44a073d72c91b4aa5030553ae18d66fb> a schema:periodical; schema:name "college & research libraries" ; schema:issn "00100870" ; schema:inlanguage "eng" ; ex:level "2" . listing 1.19. rdf turtle data for the example article. listing 1.20 shows a sparql query that returns the names of courses that have a syllabus that contains an article from the journal college & research libraries. the titles of the reading list articles are also returned. a similar query could be the basis of a simple network analysis where courses are nodes and shared journals are weighted edges. prefix schema : <https://schema.org/> prefix ccso : <https://w3id.org/ccso/ccso/> prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> select ?course_name ?article_title where { ?course rdf:type ccso:course ; ccso:hassyllabus ?syllabus ; ccso:csname ?course_name . ?syllabus ccso:haslm ?citation . ?citation schema:ispartof ?journal ; schema:name ?article_title . ?journal rdf:type schema:periodical ; schema:name "college & research libraries" . } listing 1.20. example sparql query to find course and article information of all articles published by ’college & research libraries’. summary and next steps conspectus is an information pipeline that harvests data on courses, reading lists and citations from the oslomet leganto syllabus management system using a published api. harvested citation metadata is enriched using data retrieved from both internal administrative systems from the university, and a sample of external data sources. the chosen external data sources provide an insight into some interoperability challenges faced when combining data from heterogeneous sources. issues relating to completeness and to mapping are highlighted. we also describe how the enriched data can be transformed to rdf and identify issues that must be faced when undertaking such transformations. an example of how enriched data can expand the possible scope of analysis is also detailed. conspectus is an exploratory tool for syllabus research that will mature as the project progresses. the current approach has focused on understanding the process of creating a pipeline and identifying potential enrichment sources. the next steps will see an increased focus on dissemination and use, and may provide relevant feedback to data owners about the data they manage. data quality issues and potential api improvements are obvious candidates to provide feedback on. conspectus is built upon an open source approach, however, currently only the code for the harvester has been released (a copy is attached to this article). as the approach and coding matures, the other stages of the pipeline will also be published. one of the future goals for conspectus is to provide an alternative to the current data-silo approach where syllabus information is institutionalized in proprietary formats in order to make data available for independent analysis. the next step for harvesting is to increase the volume of data by importing data from additional norwegian universities that also use leganto. after that, we aim to investigate how to integrate data from various syllabus management systems. that is, to go beyond leganto. this will require the development of a common data model that encompasses data returned from the different apis. the rdf data model presented under the section “semantic representation of the data” will form the basis for such a model. regarding enrichment, further exploration is necessary to identify other relevant sources. currently, enrichment is limited to bibliographic entities – journals, articles and books – that have global identifiers. a next step is to expand the enrichment stage to include additional entities (for example, authors and subjects) and use fuzzy string-based matching techniques to link to relevant data sources, such as wikidata. the next steps for dissemination require discussions with stakeholders within the university library, and academic departments on potential analysis projects. the conspectus pipeline can form the basis of a more innovative approach to the way syllabi are viewed and managed by a university library and may provide new ways to undertake bibliometric analysis across courses, departments and institutions. table 3. mapping between leganto fields, enrichment sources and ccso and schema.org ontologies. leganto field src. ontology class ontology property note – i c:university s:name – i c:school s:name – i c:school c:belongsto cou.ad.desc l c:department s:name cou.ad.value l c:deparment s:identifier – i c:deparment c:belongsto cou.academic_department l c:deparment c:offerscourse cou.name l c:course c:csname regex cou.name l c:course c:code regex – i c:course e:academiclevel rea.link l c:course c:hassyllabus regex rea.name l c:syllabus c:academicyear regex cit.link l c:syllabus c:haslm regex cit.m.article_title l s:article s:name cit.m.author l s:article s:author string, not person or org. cit.m.year l s:book s:datepublished regex cit.m.doi l s:article s:identifier blank node of type s:propertyvalue cit.m.start_page l s:article s:pagestart cit.m.end_page l s:article s:pageend cit.m.pages l s:article s:pagination – e s:article s:inlanguage – e s:article s:about string, not thing – l s:article s:ispartof no publicationvolume or publicationissue cit.m.journal_title l s:periodical s:name cit.m.issn l s:periodical s:issn – e s:periodical s:inlanguage – e s:periodical x:level nsd publication level cit.m.chapter_title l s:chapter s:name cit.m.chapter_author l s:chapter s:author string, not person or org. cit.m.chapter l s:chapter s:position check cit.m.start_page l s:chapter s:pagestart cit.m.start_end l s:chapter s:pageend – l s:chapter s:ispartof cit.m.title l s:book s:name cit.m.author l s:book s:author string, not person or org. cit.m.year l s:book s:datepublished regex cit.m.isbn l s:book s:isbn cit.m.edition l s:book s:bookedition cit.m.pages l s:book s:numberofpages string, not integer cit.m.publisher l s:book s:publisher string, not person or org. – e s:book s:inlanguage – e s:book s:about string, not thing cit.m.place_of_publication l s:pubilcationevent s:location string, not location notes [1] ken chad. the rise of library centric reading list systems. helibtech briefing paper no, 5:1–13, 2018. [2] leganto course resource list management. [internet]. available from: https://exlibrisgroup.com/products/leganto-reading-list-management-system/ [3] talis aspire. [internet]. available from: https://talis.com/talis-aspire/ [4] introducing keylinks: the new reading list platform. [internet]. available from: https://www.cla.co.uk/news-introducing-keylinks [5] bluecloud course lists. [internet]. available from: https://www.sirsidynix.com/bluecloud-course-lists/ [6] courses – exlibris developer network. [internet]. available from: https://developers.exlibrisgroup.com/alma/apis/courses/ [7] syllabus harvesting. [internet]. available from: https://gitlab.com/alvis-project/bibliometrics/development/syllabus-harvesting [8] roy thomas fielding. architectural styles and the design of network-based software architectures. university of california, irvine, 2000. [9] bibsys sru details. [internet]. available from: https://bibsys.alma.exlibrisgroup.com/view/sru/47bibsys_network?version=1.2&operation=explain [10] marc21 xml schema. [internet]. available from: https://www.loc.gov/standards/marcxml/ [11] dc schema for sru. [internet]. available from: https://www.loc.gov/standards/sru/recordschemas/dc-schema.html [12] mods: metadata object description schema. [internet]. available from: https://www.loc.gov/standards/mods/v3/mods-3-5.xsd [13] crossref rest api. [internet]. available from: https://github.com/crossref/rest-api-doc [14] microsoft academic graph. [internet]. available from: https://www.microsoft.com/en-us/research/project/microsoft-academic-graph/ [15] microsoft academic. may 4, 2021. next steps for microsoft academic – expanding into new horizons. [internet]. available from: https://www.microsoft.com/en-us/research/project/academic/articles/microsoft-academic-to-expand-horizons-with-community-driven-approach/ [16] ourresearch blog. june 13, 2021. mag replacement update: meet openalex!. [internet]. available from: https://blog.ourresearch.org/openalex-update-june/ [17] microsoft. project academic knowledge. [internet]. available from: https://www.microsoft.com/en-us/research/project/academic-knowledge/ [18] microsoft. evaluate method. [internet]. available from: https://docs.microsoft.com/en-us/academic-services/project-academic-knowledge/reference-evaluate-method [19] microsoft. paper entity attributes. [internet]. available from: https://docs.microsoft.com/en-us/academic-services/project-academic-knowledge/reference-paper-entity-attributes [20] carlo batini and monica scannapieco. data and information quality: dimensions, principles and techniques. springer, 2016. [21] crossref metadata api json format. [internet]. available from: https://github.com/crossref/rest-api-doc/blob/master/api_format.md [22] lui: faculty of education and international studies; sam: faculty of social sciences; tkd: faculty of technology, art and design; hv: faculty of health sciences. [23] evangelos katis, haridimos kondylakis, giannis agathangelos, and kostas vassilakis. developing an ontology for curriculum and syllabus. in european semantic web conference, pages 55–59. springer, 2018. [24] mariela tapia-leon, carlos aveiga, janneth chicaiza, and mari carmen suárez-figueroa. ontological model for the semantic description of syllabuses. in proceedings of the 9th international conference on information communication and management, pages 175–180, 2019. [25] ghadeer ashour, ahmad al-dubai, imaed romdhani, and naif aljohani. an ontological model for courses and academic profiles representation: a case study of king abdulaziz university. in 2020 international conference engineering technologies and computer science (ent), pages 123–126. ieee, 2020. [26] maha al-yahya, auhood al-faries, and remya george. curonto: an ontological model for curriculum representation. in proceedings of the 18th acm conference on innovation and technology in computer science education, pages 358–358, 2013. [27] richard wallis. may 14, 2019. library metadata evolution: the final mile. data liberate. [internet]. available from: https://www.dataliberate.com/2019/05/14/library-metadata-evolution-final-mile/ [28] for example, mbib4600 – interactive information retrieval (2021-v). pattern: course code (mbib4600) – course name (interactive information retrieval) (year (2021) – semester (v). about the authors david massey is an assistant professor at the department of archivistics, library and information science at oslo metropolitan university. he is a member of the information systems based on metadata (metainfo) research group. thomas sødring is an associate professor at the department of archivistics, library and information science at oslo metropolitan university. he is a member of the information systems based on metadata (metainfo) research group. subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license. the code4lib journal – an xml-based migration from digital commons to open journal systems mission editorial committee process and structure code4lib issue 52, 2021-09-22 an xml-based migration from digital commons to open journal systems the oregon library association has produced its peer-reviewed journal, the ola quarterly (olaq), since 1995, and olaq was published in digital commons beginning in 2014. when the host institution undertook to move away from bepress, their new repository solution was no longer a good match for olaq. oregon state university and university of oregon agreed to move the journal into their joint instance of open journal systems (ojs), and a small team from osu libraries carried out the migration project. the osu project team declined to use pkp’s existing migration plugin for a number of reasons, instead pursuing a metadata-centered migration pipeline from digital commons to ojs. we used custom xslt to convert tabular data exported from bepress into pkp’s native xml schema, which we imported using the ojs native xml plugin. this approach provided a high degree of control over the journal’s metadata and a robust ability to test and make adjustments along the way. the article discusses the development of the transformation stylesheet, the metadata mapping and cleanup work involved, as well as advantages and limitations of using this migration strategy. by cara m. key introduction content migrations are a fact of life for library technologists. for a variety of reasons, things need to get moved from old systems to new systems. in between the old and new systems is often where the interesting work takes place. in that space between, great pains may be taken to make sure that what one ends up with is essentially the same as what one started with – that everything looks at least as good and functions at least as well, and hopefully better. it isn’t just the digital assets that are relocated; whatever descriptions they possess that allow users to find and understand them must also be brought along. a pdf migrated from one repository to another may not require much special care, but differences in platform-specific metadata schemas can make it quite challenging to preserve the meaning of the accompanying description. migration solutions that try to serve all use cases may not give sufficient attention to the transformation or even inclusion of metadata, or the preservation of its meaning. this article describes a migration project in which the available tools did not meet our needs, so a custom solution was built from scratch. in this project we focused on moving the metadata to the new system, and brought the content along as well. background the ola quarterly (olaq) is the journal of the oregon library association. first published in 1995, the journal had been hosted and managed by pacific university within their commonknowledge repository since 2014. the full archival run of olaq was available digitally. at the time of the migration, there were 93 published issues. commonknowledge is pacific university’s institutional repository, hosted on bepress’ digital commons software until 2020. olaq was a slightly odd fit within commonknowledge since it was not a product of the university, but in digital commons the journal had a functionally standalone space separate from the rest of the repository. pacific decided they would migrate their institutional repository from bepress to hyku, a turnkey version of the open source hyrax repository platform (meetz and laird 2020). they recognized in the planning phase that olaq would not be able to keep its dedicated space on the new platform. the ola board contacted various universities in search of a new host institution, and oregon state university’s bid was the preferred choice. oregon state university libraries (osul) maintains a long-established open journal systems (ojs) site, in partnership with the university of oregon library under the name “ojs at oregondigital.org.” ojs is a widely used open source software product from public knowledge project (pkp) for publishing open access scholarly journals. it would be able to provide a dedicated site for olaq, and costs would be minimal. at the time, ojs at oregondigital.org was operating on an older version of ojs (2.4.8), with plans to upgrade but resources not yet allocated. the adoption of olaq into ojs at oregondigital.org would be just the nudge needed to complete the upgrade. project planning the project plan was drafted and approved by the ola board in june 2019, and an initial project team was formed at osu libraries. drafted to the team were ojs product owners for osu and uo, the scholarly communication & publishing services librarian at pacific university, the communications coordinator for olaq, one osu libraries software developer, and two osu library technicians, with potential student assistance if needed. the target completion date was september 2019, ahead of the deadline set by pacific university’s expected completion date in november. the plan was uncomplicated: upgrade ojs at oregondigital to version 3.x; decide on the design template for olaq; use the existing bepress to ojs plugin to migrate the content; test the migration for completeness and accuracy; update dois and indexing; train editors to use the new software. the project reality did not, however, follow the project plan. in october, the ojs2 to ojs3 upgrade had not measurably progressed, and early investigations had raised concerns about errors and data loss from the existing migration plugin offered by pkp. several shortcomings with results of the plugin were clearly acknowledged in the plugin’s documentation, including articles being out of order, irregularities with sections, and cover art files not being imported (felczak 2017; felczak [updated 2020]). additional problems were identified, which were amplified by the particulars of our content and situation: osu did not have administrator access to the source content in digital commons; the csv to xml conversion script recommended by pkp to prepare data for their plugin introduced more potential errors and data loss (vilhuber … [updated 2018]); we had missing and incomplete metadata in ojs required fields; we had metadata, especially at the issue level, that did not align perfectly with ojs but which we wanted to somehow preserve. this paper’s author, osu’s metadata librarian, had been very new to the job at the project’s inception, but was called in at this point to consult. we initially aimed to identify strategies for handling the fallout from an imperfect but extant solution, but we realized that we had all the tools we needed to try something entirely different. ojs 3.x includes an xml import tool, the native xml plugin, and the ojs/pkp native xml schemas are publicly available online. we also had the exported metadata, and experience writing xsl transformations. we were already facing a time investment for correcting introduced errors and possibly modifying the code, so deciding to change course was not destabilizing. we formulated a new plan: to create custom xslt, transform the digital commons data in-house, and import xml via the native xml plugin. the team was reduced to the metadata librarian and software developer, with oversight by our shared department head and consultations available with the journal stakeholders. figure 1. digital commons to ojs migration workflow. shortly after this pivot, we also concluded that migrating in tandem with upgrading our existing ojs 2.x system was too ambitious. the upgrade was more difficult than anticipated, and was unlikely to be completed before the digital commons repository was sunsetted. we resolved to launch olaq as a standalone journal, temporarily, on a separate ojs 3.x site while the existing ojs at oregondigital site remained on ojs 2. this change did not have a significant negative impact on the outcome of the olaq migration, which would still be moved to ojs 3.x as intended, and it freed up the developer’s time for immediate needs. data preparation pacific university provided an up-to-date export of olaq metadata from digital commons, delivered as an excel spreadsheet. for ease of reviewing and collaborating, the osu team used google sheets to work with the metadata. some data cleanup, normalization, and enhancements were required before converting the data to xml format. key spreadsheet cleanup tasks that impacted the rest of the migration process included: replacing xml reserved characters in the data with character references (`&amp;`, `&lt;`, `&gt;`); verifying the correct order of contents (the ordering of contents in ojs follows the order of the xml import files); correcting publication date formats to `yyyy-mm-dd`, and ensuring all items had publication dates (more on this later). the cleaned metadata was exported from google sheets as a csv file. we lightly modified a python script found online to convert the csv to flat xml (fb26 2010). the flat xml structure has a root element `<csv_data>`; each row of data is a record within a `<row>` element; and each item in the header row becomes an element name, as seen in this excerpt: ``` <row> <title>communicating with library donors</title> <keywords>library, library marketing, library communications[...]</keywords> <disciplines>archival science; cataloging and metadata[...]</disciplines> <document_type>article</document_type> <doi>http://dx.doi.org/10.7710/1093-7374.1829</doi> <volnum>21</volnum> <issnum>4</issnum> <vol_iss>21(4)</vol_iss> <fpage>5</fpage> <lpage>9</lpage> <distribution_license></distribution_license> [...] </row> ``` snippet 1: xml output from csv_to_xml.py. a row in the csv source document is represented by the row container element, and the csv column headers become individual child elements. this flat xml was used as the source data for the xsl transformation into the ojs/pkp native schema. metadata mapping and xslt development in the next stage, the digital commons metadata fields were mapped to ojs elements according to the pkp/ojs native xml schema, which could then be used to write xslt that would produce a valid xml import file. appendix a shows the fields that were present in the digital commons export, and the xpaths used for their element and attribute mappings in ojs. for unmapped fields, either no data was populated for that field in our dataset, there was no straightforward element match in the ojs schema, or the field is not meaningful in the ojs context (such as a digital commons system field). some of the information contained in unmapped fields – specifically publisher, issn, and journal id – is provided in the ojs’s journal-level configuration. over the course of the project, additional fields were added to the dataset to fill in gaps and facilitate processing, as more was learned and the transformation became more finely tuned. the data import and export page of the pkp administrator’s guide lets intrepid readers know that the native xml plugin could be used for “[m]oving a large number of back issues and articles into ojs,” but cautions that one will need “[a] basic understanding of xml” (pkp [n.d.]). links to the schema xsl files and sample xml files are provided, along with several tips. we were unable to find any publicly available supporting documentation for the schema beyond these resources. xsd files are readable enough to experienced xml practitioners, and the overall field mapping was not complicated. however, definitions and guidance detailing how xml data values connect to software functionality would have been tremendously appreciated (especially for brand new ojs 3.x users). as a basic example, the `published` attribute on the `<issue>` element is stated in the schema to have a type `integer`; but the schema does not state that an attribute value of `1` will publish the issue immediately upon import while a `0` value means the issue will be imported in an unpublished state. furthermore, our testing revealed that the import plugin has stricter requirements for import files than the schema, and therefore xml files that validate may still fail to import. an xslt stylesheet for converting the flat xml into ojs native xml was built based on the field mapping. oxygen xml editor software was used to write the xslt and run transformations. the style of the xsl code is somewhat explicit, naming each ojs destination element in order and pulling from the source data to populate the element, as seen in this excerpt populating the keywords field. ``` <xsl:if test="keywords/text()"> <keywords> <xsl:for-each select="tokenize(keywords, ', ')"> <keyword> <xsl:value-of select="."/> </keyword> </xsl:for-each> </keywords> </xsl:if> ``` snippet 2: section of xsl stylesheet handling keywords. the mapping and xslt development happened in tandem with the ojs 3 installation, and credentials for the development site were given out just in time for the first fully valid output to be produced. we proceeded with testing the xml output using the native xml plugin, experimenting to determine connections between metadata and system functionality, and iterating until we ultimately had results we wanted. metadata challenges given the vast differences between the origin and destination systems and our spotty understanding of schema details, troubleshooting and revising were ongoing throughout the project. of the larger metadata challenges to solve, some were caused by a lack of complete data, and others by an abundance of it. still other frustrations arose out of the design choices within the native xml schema and the logic of the import tool. a few notable cases are described below. one compromise in the move to ojs was the loss of metadata granularity for journal issues. in digital commons, especially in more recent years, olaq editors had described issue-level objects similarly to article-level objects, and digital commons issue records included disciplines, keywords, and guest editor biographies. ojs issue-level metadata fields are limited to issue identification (numeration, year, and title), publication date, and description. we were interested in preserving the issue-level metadata, so we accommodated as well as possible by placing all of the above information into description fields so that the information would be available, even if some of the utility was lost. in working to address the issue description problem, we found that, while valid per the schema, repeated descriptive xml elements are not preserved on import. so while we originally mapped several of the issue fields to multiple `<description>` elements, each successive instance of `<description>` appeared to overwrite the previous one when the file was imported. re-exporting the issue confirmed that only the last-occurring instance of the element was saved. we revised the xslt to combine multiple values in a single `<description>` element, adding html markup for headers and newlines within the field text to replicate the appearance of distinct fields for readers. overall, olaq’s metadata was high quality, but there were a number of holes in the data including some that required consulting stakeholders to approve any solution. two fields in particular were required for every ojs submission but incomplete in the digital commons dataset. (“submission” is ojs terminology for an individual component of a journal issue, predominantly but not exclusively articles.) the first was author(s), including subfields for first name, last name, and affiliation. authorship was diligently noted for all articles except editorials, but olaq also had separate submission objects for tables of contents and back matter components, and these pieces did not have stated authorship. we agreed to assign the journal name as the default author if none was given. (because ojs enforces a given name + family name rule that does not accommodate corporate authors well, we ended up with given name = “oregon library,” family name = “association,” and affiliation = “ola”). this was handled within the xslt script. the second was a full `yyyy-mm-dd` publication date. in the source data prior to 2014, when olaq was moved into digital commons, dates were year-only. issue titles indicated the season, as in fall 2010. to fill in the gaps, we agreed to assign retroactive dates based on the season, where fall issues were now recorded as being published on 09-01, winter on 12-01, spring on 03-01, and summer on 06-01 of their given year. we altered the dates in the spreadsheet using a google sheets function. processing and import the native xml plugin provides means to import xml journal content into ojs. it is included “out of the box” and does not require separate installation. there is a command-line import option, but in this project we exclusively imported through the admin dashboard user interface. xml import files may contain one or multiple issues and/or articles. file size limitations do apply to imported xml files, though these can be adjusted in the system configuration. for our use case, the ability to easily delete, edit, and re-import content was essential, so our xslt was designed to process the complete dataset and output one xml file per issue. content imported using the plugin results in fully structured journal issues, which may be either immediately published or set to pre-publication, depending on attribute values. ideally, little to no additional work is needed post-import. we tested the transform-and-import workflow using three olaq issues representing distinct time periods and thus differing descriptive practices. this test set was sufficient to identify and resolve the metadata problems previously described. when the initial batch was successful, we generated three more issues and continued to review; then finally executed the xslt across the full corpus to import all 93 issues. the ojs infrastructure allowed us to fetch the pdfs remotely from digital commons and insert them as article galley files with the simple usage of a `<href src=”{url}”>` reference in the xml. these pdf urls were not present in the digital commons export data – the field exists, but was blank in our dataset. we were not successful in retrieving them with the xslt due to the bepress html being reportedly ill-formed. however, google sheets was more permissive, so we used the importxml function to add the pdf urls to the source data. this process worked very reliably for the 873 submission objects. however, for unclear reasons, while submission galley files can be fetched remotely, issue galley files may only be added via the xml import file if embedded in base-64 encoding. the same is true for cover art files. olaq had both full-issue pdfs and cover art for each of the 93 issues. we decided to create local copies of the full-issue pdfs and manually add them to the issue objects post-import. we took the opportunity to regenerate the cover art files from the pdfs to replace the lower-resolution gifs used in digital commons; these were added manually post-import as well. if we had been working with a larger body of content, we would likely have investigated encoding the pdfs and adding them to the import files. in our case, the admittedly tedious ui-button-clicking time was almost certainly shorter than the development time would have been. this is an area for potential future work, if the xslt tools are to be carried forward. finishing details we created custom css to apply on top of the default ojs theme, which mimics the look of the former olaq site at pacific university, which in turn had been designed to resemble the print journal’s cover art style. populating the static site content, such as about pages and policy information, was a manual copy-paste operation, in part because we only had normal public access to that static content in digital commons. since our project had only a single journal to configure, this was not excessively laborious. with the content in place, we did a soft launch of the production site and invited the ola stakeholders to review the work. while there were a few tweaks needed, happily no problems were found that required reprocessing any of the content. as they shut down their digital commons repository, pacific university implemented a base redirect to our ojs site. setting identifiers in the metadata that resembled the original paths allowed for a simple redirect rule on our end for all submission-level items, though issue objects and static pages needed explicit redirects. the bulk of the project work took place between november 2019 and february 2020; the early majority went toward installation of the software and development of the transform, while the execution of the final transformation and import of the full journal’s metadata and content – the “actual migration” – required less than a week. pacific university’s timeline for sunsetting digital commons was extended at least twice, which was to our project team’s benefit as we were consistently operating behind schedule. the official launch suffered from unfortunate timing, taking place in march 2020 just as covid-19 sent us all out of our offices, but the journal’s site has enjoyed strong readership and three new olaq issues have been published post-migration. reflections in addition to the previously described metadata solutions and the manual process for uploading issue files, the overall project encountered plenty of other challenges and made plenty of other compromises. the biggest overall challenge was that osu’s upgrade from ojs 2 to ojs 3 could not be completed before the deadline to move olaq, so we are running two separate ojs instances, which is not an ideal situation. work on the upgrade has progressed, but carving out time and resources from an established workcycle rhythm with many competing projects is more difficult without the looming deadline of loss of access to the content. there were plenty of minor irritations along the way, too; for instance, discovering that dois imported with the xml metadata could only be saved in the system if the doi plugin is turned on prior to import. while there is much to like about publishing in ojs, the move from digital commons meant some compromises for the journal as well. olaq’s usage statistics from digital commons were lost, and all items’ views were reset to zero in ojs. a few digital commons features are not available in ojs, such as the readership map and hidden keywords, and browsing functionality and analytics are less robust. the ojs 3.x software was new to the project team, and there was an associated learning curve. the lack of official documentation for trying to migrate journals from outside systems into ojs or for understanding the xml schema indicated this is not a process that the developer feels strongly about supporting – we often had the sense that we were not using it the way it was meant to be used. there was a great deal of trial and error and fixing and retrying; but that’s exactly the spirit of work that our approach fostered. designing in-house transformation tools that were customized for our particular needs lent itself perfectly to a testing and iterating workflow. the metadata-forward approach meant we had precise control over the results, at least within the confines of the system. output could be easily destroyed and remade with improvements. at the conclusion of the project, while the results might not have been perfect, we were able to feel confident that they were as close to perfect as we could reasonably have gotten. our xslt pipeline was not designed with the intent to cover every possible digital commons to ojs migration, but worked well for our own use case. the main advantage of the xml-based, metadata-forward approach is the degree of control it provides over the data. other institutions working to migrate content from digital commons to ojs may be able to repurpose these tools; adjustments may be needed to the xslt code, but the majority of the work is likely to be in data preparation. since the olaq migration has concluded, osu libraries does not currently have a use case for further development of the digital commons to ojs migration tools. in addition to supporting olaq editors with the transition, we are continuing to work toward moving our existing ojs at oregondigital.org journals from our ojs 2.x instance onto the ojs 3.x, alongside olaq. that said, the digital commons to ojs xslt pipeline would benefit from additional stylesheet versions corresponding to xml schema updates in recent ojs releases, since adopters going forward are less likely to be installing ojs 3.1 versus a later version. another possible enhancement, as previously mentioned, would be automating the incorporation of base-64 encoded content files within the xml output, to avoid having to upload issue galleys post-import; this would also provide flexibility for including article galleys without relying on a live remote source location. potential adopters of this toolset may also have special circumstances that suggest other directions for development not anticipated here. the project to migrate olaq from digital commons to ojs involved building a custom solution despite the existence of established and tested tools. this was a bold decision, and we ultimately can’t measure how different the results would have been if we had used the existing tools instead. we can say with certainty that the migration was successful, and that we gained familiarity with ojs 3.x that will be useful as we continue to support olaq and other journals on the platform. acknowledgements this project leaned heavily on the hard work of osu libraries analyst programmer ryan wick, and the coaching of our department head, margaret mellinger. we are also grateful for the patience of our associates at ola, especially charles wood and elaine hirsch. finally we would like to extend our congratulations to pacific university on their beautiful new commonknowledge repository. read ola quarterly on ojs at https://journals3.oregondigital.org/olaq. find the code on github at https://github.com/osulp/bepress-ojs-xslt. appendix a. bepress digital commons to ojs 3.1 field mapping this table shows the fields included in the olaq metadata export from digital commons and the destination ojs fields to which they are mapped. for olaq, objects with document type “full_issue” became ojs issues, and all other document types were migrated as ojs submissions. data transformations were applied to many of the fields during the course of both spreadsheet cleanup and xsl processing, which are not specified in the table. readers are invited to visit this project’s github repository for those details. it should also be noted that the xpaths given in this table are specific to ojs 3.1, and that more recent versions of the schema are known to deviate for some of these.   bepress field ojs xpath – submissions ojs xpath – issues title //article/title //issue/issue_identification/title keywords //article/keywords //issue/description disciplines //article/disciplines //issue/description document_type //article/@section_ref doi //article/id[@type=”doi”] //issue/id[@type=”doi”] volnum //issue/issue_identification/volume issnum //issue/issue_identification/number fpage //article/pages lpage //article/pages distribution_license //article/licenseurl abstract //article/abstract //issue/description comments fulltext_url peer_reviewed publication_date //article/@date_published //issue/date_published //issue/issue_identification/year rights //article/copyrightyear //article/copyrightholder license_statement filename erratum publisher issn journal_id author[#1-5]_fname //article/authors/author/givenname author[#1-5]_mname //article/authors/author/givenname author[#1-5]_lname //article/authors/author/familyname author[#1-5]_suffix author[#1-5]_email //article/authors/author/email author[#1-5]_institution //article/authors/author/affiliation calc_url //article/id[@type=”public”] context_key issue //issue/id[@type=”public”] ctmtime   for unmapped fields, either no data was populated for that field in our dataset, there was no straightforward element match in the ojs schema, or the field is not meaningful in the ojs context (such as a digital commons system field). some of the information contained in unmapped fields – specifically publisher, issn, and journal id – is provided in the ojs’s journal-level configuration. references fb36. 2010. convert csv to xml (python code). activestate code: recipes; [cited 2021 july 15]. available from: https://code.activestate.com/recipes/577423-convert-csv-to-xml/ felczak m. [internet]. [updated 2020 july 23]. ojs bepress digital commons import plugin (readme); [cited 2021 july 15]. available from: https://github.com/mfelczak/bepress felczak m. 2017. migrating bepress digital commons journals to ojs (blog post). public knowledge project; [cited 2021 july 15]. available from: https://pkp.sfu.ca/2017/12/07/migrating-bepress-digital-commons-journals-to-ojs/ meetz j, baird l. 2020. migrating an ir at a small, liberal arts university [presentation]. northeast institutional repository day (nird); 2020 december 3; virtual.  https://doi.org/10.13028/0q6z-2g81. pkp. [internet]. [no date]. pkp administrator’s guide: data import and export; [cited 2021 july 15]. available from: https://docs.pkp.sfu.ca/admin-guide/en/data-import-and-export vilhuber l. [internet]. [updated 2018 february 26]. migration scripts bepress (csv) -> ojs (readme); [cited 2021 july 15]. available from:  https://github.com/journalprivacyconfidentiality/jpc-migration about the author cara m. key (she/her) is the metadata librarian at oregon state university libraries & press, where she supports open access publishing and digital repository services.  contact: cara.key@oregonstate.edu subscribe to comments: for this article | for all articles leave a reply name (required) mail (will not be published) (required) website δ issn 1940-5758 current issue issue 60, 2025-04-14 previous issues issue 59, 2024-10-07 issue 58, 2023-12-04 issue 57, 2023-08-29 issue 56, 2023-04-21 older issues for authors call for submissions article guidelines log in this work is licensed under a creative commons attribution 3.0 united states license.